11institutetext: Fraunhofer IKS, Munich, Germany 22institutetext: Technical
University of Munich, Munich, Germany

# Finding Dino: A plug-and-play framework for unsupervised detection of out-
of-distribution objects using prototypes

Poulami Sinhamahapatra 11 2 2 Franziska Schwaiger 11 Shirsha Bose 11 2 2 Huiyu
Wang 11 Karsten Roscher 11 Stephan Guennemann 22

###### Abstract

Detecting and localising unknown or Out-of-distribution (OOD) objects in any
scene can be a challenging task in vision. Particularly, in safety-critical
cases involving autonomous systems like automated vehicles or trains.
Supervised anomaly segmentation or open-world object detection models depend
on training on exhaustively annotated datasets for every domain and still
struggle in distinguishing between background and OOD objects. In this work,
we present a plug-and-play generalised framework - PRototype-based zero-shot
OOD detection Without Labels (PROWL). It is an inference-based method that
does not require training on the domain dataset and relies on extracting
relevant features from self-supervised pre-trained models. PROWL can be easily
adapted to detect OOD objects in any operational design domain by specifying a
list of known classes from this domain. PROWL, as an unsupervised method,
outperforms other supervised methods trained without auxiliary OOD data on the
RoadAnomaly and RoadObstacle datasets provided in SegmentMeIfYouCan (SMIYC)
benchmark. We also demonstrate its suitability for other domains such as rail
and maritime scenes.

###### Keywords:

OOD object detection Anomaly segmentation Prototype learning

##  1 Introduction

Figure 1: Sample results for zero-shot unsupervised OOD object detection with
PROWL across multiple domains: road driving (test set of RoadAnomaly data from
SMIYC benchmark [7]), rail (inpainted OOD on RailSem19 [33]) and maritime
scene (marine obstacle detection dataset [4]). All the detected OOD objects
are marked as ‘unknown’ in red.

Artificial Intelligence (AI) has become a cornerstone of autonomous systems -
especially in the perception of the surroundings. Systems operating in the
real world must dynamically adapt to any situation in open-world settings.
This means that in any given scene: the system should be able to understand
its context, usually through the detection and localisation of relevant
objects. To this end, AI models are typically trained extensively based on
object categories in the given operational design domain (ODD). With high-
quality publicly available datasets, such as Cityscapes [10], RailSem19 [33]
and MODD [4], several State-of-the-Art (SOTA) deep neural networks (DNNs)
provide outstanding performances for a closed set of object categories.
However, they are unable to identify and categorise unknown objects, i.e.
objects that do not belong to any of the training classes. One can encounter
obstacles that were never learned in the training data like a random animal on
the road or unknown floating obstacles in front of unmanned surface vehicles
(USVs) in maritime applications. Due to the open world setting, there can be
numerous such unknown or out-of-distribution (OOD) objects at any time and it
is almost impossible to train DNNs exhaustively with annotated datasets on all
possible known object categories and object variations, especially in complex
domains such as autonomous driving.

In contrast to image classification, where OOD detection is a well-defined and
widely researched topic, the main challenge in the context of camera-based
object detection is the explicit distinction between an unknown object and the
common _background_ , i.e. anything in the scene that is not relevant.
Therefore, most existing approaches rely on supervised anomaly segmentation
and the use of selected OOD samples during training [28, 8, 1, 30, 22].
Especially the latter is a severe limitation when trying to detect things that
were not known at the training time of a model. Open world object detection
[34, 17, 14] on the other hand gained some attention recently but is
struggling with the application-dependent understanding what a relevant object
is.

In this work, we propose a novel framework for the detection of unknown
objects in an image: PRototype-based zero-shot OOD detection Without Labels
(PROWL). It can detect an arbitrary number of unknown objects in scenes from
any domain in a zero-shot plug-and-play manner, i.e. without any additional
model training on the target domain, as illustrated in Fig. 1. It leverages
the rich and diverse features from pre-trained foundation models such as
DINOv2 [27] to robustly capture the known object categories as prototypes in a
prototype feature bank. The similarity to those prototypes is then used to
calculate a pixel-level similarity score for each given test image. This score
can be thresholded to detect OOD pixels. Additionally, we propose to combine
foreground masks provided by unsupervised segmentation with those pixel-level
scores to refine them into high-quality masks for individual instances of OOD
or unknown objects. Making use of pre-trained visual features from foundation
models which provide robust representations of almost any object, PROWL can
easily generalize across application domains by simply specifying the list of
ODD classes to determine the corresponding prototype feature bank. It can
unknown objects without additional model training in just simple inference
steps and performs comparably to existing supervised methods. To the best of
our knowledge, PROWL is the first end-to-end framework for unsupervised zero-
shot OOD detection in any scene without any training in ground truth (GT)
classes. Within the framework of PROWL, we compare the performance of several
different unsupervised segmentation and detection methods based on the quality
of foreground masks generated. In the absence of a direct baseline for
unsupervised anomaly segmentation in a multi-object scene, we further compare
our results with other supervised segmentation methods from the established
SMIYC [7] benchmark based on given metrics and datasets. Further, we show the
applicability of PROWL for finding unknowns in diverse scenes like road
driving, on railroad tracks, or even maritime.

To summarise, the key contributions of our framework PROWL are:

  * •

PROWL is the first unsupervised OOD object detection and segmentation
framework that can sufficiently and reliably distinguish OOD objects from
background

  * •

PROWL is a zero-shot OOD object detection framework that relies on pre-trained
features from foundation models without additional training on domain data

  * •

PROWL can be applied as an adaptable plug-and-play module generalised to any
scene in a new domain without domain-specific training. We demonstrate this by
applying to new domains beyond road driving such as rail and maritime domains.
For the rail domain, we additionally contribute by creating datasets with
inpainted OOD objects

  * •

PROWL outperforms supervised methods trained without auxiliary OOD datasets on
the validation sets of RoadAnomaly and RoadObstacle datasets of road driving
SMIYC [7] benchmarks as well as shows comparable performance wih other
supervised methods in Fishyscapes [3] benchmark

##  2 Related Work

In vision tasks, detecting the unknown/OOD object has been formulated under
different banners.

Open World Object Detection (OWOD): As compared to standard closed-world
object detection, OWOD poses several challenges such as generating quality
candidate proposals on potentially unknown objects or distinguishing the
unknown objects from the background. Recent methods [34, 17, 14] have explored
probabilistic models based on objectness score of the unknown object and
learning novel classes via incremental learning and retraining. But they still
have much room for improvement in distinguishing the unknown class from the
background.

Anomaly Detection: Anomaly or Out-of-Distribution (OOD) detection was
initially conducted in the context of image classification. OOD detection has
been widely used for finding deviations from in-distribution (ID), i.e.
training data. It encompasses both deviations in distribution shift such as
perturbations, weather, or lighting conditions as well as changes in semantic
classes previously not seen during training. Methods originating from image
classification focused on developing techniques that aimed to quantify
uncertainty in confidence values produced by classification outputs of DNNs
(e.g., Maximum Softmax Probability [19], Mahalanobis distance [22]). Other
methods find anomalies by estimating the likelihood with generative models
[29] or training discriminatively with negative or auxiliary data from OOD
samples like ODIN [23], Outlier exposure [20]. Another line of work is finding
anomalies or defective parts in an object, such as defective part detection in
Industrial Anomaly Detection [25]. Although developed as image-level anomaly /
OOD detection, most of these methods can be applied to anomaly segmentation by
finding potential anomalies based on the confidences of each pixel.

Anomaly Segmentation: The goal is to predict anomaly probabilities for each
pixel in an image. Different works use discriminative or generative approaches
[12, 24]. Most methods rely on auxiliary out-of-distribution (OOD) data during
training. For example, Max Entropy [8] predicts high entropy in anomalous
regions and reduces false positives using a meta-classifier on OOD data.
DenseHybrid [15] combines discriminative and generative modeling. However,
pixel-based reasoning often produces noisy anomaly scores, especially for
border pixels and poorly localized anomalies. Recent approaches focus on mask-
based methods that capture anomalies as whole objects. These methods predict
regions instead of pixels, resulting in fewer false predictions [9]. RbA [26],
EAM [16], Maskomaly [1], and Mask2anomaly [28] utilize mask-based
classification. However, all these methods are trained with supervised labels
and sometimes with OOD data. Our proposed approach eliminates the need to
distinguish between ID and OOD object masks. We learn every object as a
foreground mask without the notion of OOD/ID object class. We perform
unsupervised foreground segmentation and OOD detection separately.

##  3 Method

In this section, we provide an overview of our framework PROWL. As illustrated
in the architecture diagram in Fig. 2, PROWL comprises three modules - a plug-
and-play prototype matching module (Sec. 3.2), a refinement module that is
used to generate foreground masks using SOTA unsupervised segmentation methods
(Sec. 3.1) and lastly OOD detection module (Sec. 3.3).

###  3.1 Preliminaries: Foreground Mask Generation

In the refinement module, we rely on the generation of foreground masks for
every object in the image without considering their respective classes. We
find that SOTA unsupervised segmentation methods such as STEGO or CutLER can
provide such masks with high quality. Let the foreground masks generated using
these methods be denoted as $M=\\{m_{1},m_{2},...,m_{N}\\}$. STEGO[18] is an
unsupervised semantic segmentation model that distills pre-trained
unsupervised visual features from DINO [6] into semantic clusters using
contrastive loss, thus discovering and segmenting semantic objects without
human supervision for each dataset. CutLER [31] is an approach for training
unsupervised object detection and segmentation models. It is trained
exclusively on unlabeled ImageNet [11] data without any additional in-domain
data. Using pre-trained self-supervised features, it uses MaskCut strategy to
discover multiple coarse object masks. Further, it trains a detector through
several rounds of self-training to detect multiple foreground objects and
corresponding instance segmentation masks. Thus, unlike STEGO, CutLER does not
provide dense segmentation as output, rather it provides a zero-shot object
detection and instance segmentation for detected foreground objects in the
image. Upon varying the detector threshold, CutLER can detect a range of small
to large objects.

Figure 2: Overview of our proposed framework PROWL. Firstly, in the plug-and-
play prototype matching module, prototype feature bank is created by
extracting features from pre-trained foundation models corresponding to few
segmented object samples for specified list of ODD object classes. Using this
feature bank, prototype matching is performed for the given test image to
generate corresponding heatmaps for each object class. Heatmaps show maximum
activation (in yellow) wherever the given object is found in test image. In
the OOD detection step, the object pixels not satisfying given similarity
thresholds are detected as OOD or ‘unknown’ (in red). For less noisy and
precise OOD detection, we combine an additional refinement step with prototype
heatmaps where foreground masks for every objects in the scene are first
extracted in an unsupervised manner. Finally, these foreground masks are
detected as either an ODD class or an OOD.

###  3.2 Plug-and-play Prototype Matching Module

The first step in our pipeline is to create the feature bank with prototypes
for every class specified in the ODD. After creation of the prototype, every
pixel is assigned a class via prototype matching.

Creating the prototype feature bank: We aim to create an offline ‘prototype
feature bank’ which are global feature space representation of a specific
object class or part of an image. Pre-trained features from foundation models
like DINO [6] use knowledge distillation via a teacher-student network for
learning in a self-supervised approach. Authors of DINO observe that a self-
supervised Vision Transformer (ViT) can learn to a great extent the underlying
perceptual grouping of image patches and semantic correspondences across
images and image domains. This property is even strengthened for DINOv2
features [27] which are trained on much larger image corpora and distilled to
smaller models. Thus, we utilize the robust general-purpose visual features
from DINOv2 for every known object class in ODD using a minimal number of
samples from the train split. We assume a list of ODD object categories
(expert-specified) that one can expect in a given scene. Let it consist of $K$
known classes $C=\\{c_{1},c_{2},...,c_{K}\\}$ and a list of prototype vectors
for each class $P=\\{p_{1},p_{2},...,p_{K}\\}$. Let us assume $L$ number of
samples contribute to each prototype vector $p_{k}$ for each class $c_{k}$.
Let the features predicted from a DINOv2 model ($g$) be $z=g(x)$ for an image
$x$. For each object instance $o_{l}\in c_{k}$ class, the segmentation mask
$s_{o_{l}}$ for the instance is extracted, such that each prototype sample
image contains only the given object instance with a zeroed-out background,
shown in Fig. 2. The features from the region of interest for the object
instance are thus calculated as $z_{o_{l}}=mean(z*s_{o_{l}})$. Finally,
prototype vector $p_{k}$ for object class $c_{k}$ is updated as:
$p_{k}=\\{z_{o_{l}}\\};o_{l}:l\in\\{1,L\\}$, i.e. the prototype vector list
$p_{k}$ is extended until $L$ object instances of $c_{k}$ class is added. This
process is repeated for all the $K$ ODD object classes. Depending on the
complexity of the dataset, a number of prototype samples can suffice
$L\in\\{5,20\\}$ for each object (see Sec 5.3) as compared to supervised
training methods which require large training samples.

Prototype matching: For each input image $x$, the inference output $z$ is
calculated based on the feature vectors from DINOv2. Now, each pixel is
classified into one of the K classes based on cosine similarity between
feature vectors of the input image and the prototype vectors $P$, resulting in
K heatmaps one for each class. Thus, the prototype heatmap for class $c_{k}$
is calculated by taking the maximum of cosine similarity between the prototype
vectors ${p_{k}}$ for respective class and features $z$ given as -

$\vspace{-1mm}h_{k}=max(z\cdot p_{k})$ (1)

The list of prototype heatmaps for all $K$ ODD classes is given as,
$H=\\{h_{1},h_{2},...,h_{K}\\}$. For each pixel $[i,j]$ in $x$, the assigned
class label ($y$) and score ($v$) is given as:

$\vspace{-3mm}y_{[i,j]}=argmax(H)$ (2)

$\vspace{-3mm}v_{[i,j]}=max(H)$ (3)

###  3.3 OOD Detection

The next step consists of the detection of OOD pixels or object masks
following the prototype-based classification of every pixel in Eq 2. This is
done by comparing the cosine similarity scores obtained in Eq 3 with a given
threshold $t,t\in[0,1]$. For every pixel we calculate an inverse normalised
cosine similarity score as:

$\vspace{-1mm}w_{[i,j]}=1-\mbox{norm}(v_{[i,j]}),\mbox{norm}(x)=\frac{x-min(x)}{max(x)-min(x)}$
(4)

Thus, pixels where $w_{[i,j]}>t$ is designated as an OOD pixel. We have
investigated the per-pixel OOD detection performance directly on the prototype
heatmaps obtained in according to Equation 1. Although OOD detection based on
DINOv2 features can be quite reliable, the resulting OOD detection tends to be
noisy, and often the OOD pixels might not precisely localise the relevant
object. Thus, the output of PROWL can be further refined by introducing
instance-level foreground masks obtained in Sec. 3.1 in combination with the
prototype heatmaps. Since these models were trained on huge generic datasets
in a self-supervised manner, they can reliably detect multiple foreground
object masks in a scene without the notion of known or unknown object class.
These masks tend to capture the objectness of every entity in the scene
without learning it as background as in supervised detection cases. Thus,
every mask $m$ where the majority of pixels is designated as OOD is now
considered to be OOD for the entire mask. As shown in Fig. 2, the OOD objects
dinosaur and passenger car were correctly detected using prototype heatmaps in
PROWL as they did not match with any of the classes listed in the prototype
bank, however, additional pixels were also spuriously detected as OOD. While
PROWL in combination with foreground masks from CutLER, correctly detected and
precisely localised the exact OOD object masks as ‘unknown’.

##  4 Implementation Details

###  4.1 Datasets

We demonstrate the plug-and-play performance of our methods by evaluating them
in three different operational design domains (ODD).

Road driving scene - We generated prototypes for the urban driving road ODD
scene from Cityscapes [10]. It is a benchmark suite with both pixel and
instance-level semantic scene understanding. For evaluation on datasets with
real OOD objects in the road scene, we refer to the datasets and benchmark
suite for anomaly segmentation in SegmentMeIfYouCan (SMIYC) [7]. It provides
two novel real-world datasets: RoadAnomaly21 and RoadObstacle21. RoadAnomaly21
consists of $100$ test and $10$ validation images with real objects or animals
as OOD appearing anywhere in the scene. In contrast, RoadObstacle21 has OOD
objects (or obstacles) appearing on the road or ego track. The authors of
SMIYC withhold the GT for the test set, where the scores are only accessible
by submitting the method to the official benchmark. Further, we also evaluate
on FS Static subset of Fishyscapes [2] anomaly segmentation benchmark. It
consists of $30$ validation images with generic objects taken from PASCAL VOC
[13] synthetically overlayed on Cityscapes images.

Rail scene - For rail ODD, we use RailSem19[33] dataset which provides diverse
images taken from an ego-perspective of a rail vehicle (trains or trams) along
with extensive semantic annotations including rails. Regarding OOD detection,
there exist no publicly available datasets with OOD objects in the rail scene
yet. Thus, we create our validation data consisting of $10$ images. We have
used Inpaint Anything [32] which uses a combination of SAM, LaMa, and Stable
Diffusion, for inpainting potential OOD objects like animals, cars, carts etc
on to the RailSem19 images. More details are given in Suppl.

Maritime scene - Obstacle detection and segmentation in maritime ODD (MODD)
scenario is prevalent for autonomous operations of unmanned surface vehicles
(USVs). Here, the task is to detect and segment all obstacles beyond sky, sea
classes as ‘unknown’. We formulate this as an OOD detection task, where
prototypes are generated from train split of MaStr1325 (Maritime Semantic
Segmentation Training Dataset) [4]. For OOD detection evaluation, we evaluate
on corresponding test dataset. We do not compare with MODS benchmark [5] as we
do not include evaluations taking into account multi-modal data with IMU
calibrations for shoreline detection.

###  4.2 Experimental Setup

PROWL is a zero-shot method, which means it does not rely on any training but
merely some inference steps. The ViT-S/14 of DINO-v2[27] pre-trained on a
dataset of 142 million images is taken as the feature extractor of PROWL. We
observe that with a default inverse cosine similarity threshold (Eq. 4) at
$0.55$ and CutLER detector threshold of $0.2$, we obtain a general
satisfactory performance across all the datasets. We further report that
depending upon the size of the OOD objects and complexity of the OOD datasets,
this optimal threshold tends to suffice even at higher values as shown in the
ablation study for OOD datasets for Cityscapes (Sec 5.3).

###  4.3 Evaluation metric

To the best of our knowledge, there exists no prior work on unsupervised OOD
detection from multiple objects in a scene. Thus, there are no established
benchmarks and evaluation metrics for the task. The Area under the Receiver
Operating Characteristic (AUROC) curve is a standard metric for OOD detection
for image classification. However, in the case of multiple OOD object
detection and segmentation tasks with a highly imbalanced number of OOD
objects, the Area under the Precision-Recall Curve (AUPRC) is a more suitable
choice. Thus, to compare with Anomaly Segmentation metrics provided with the
SMIYC benchmark, we use AUPRC and False Positive Rate at True Positive Rate of
$95\%$ (FPR) metrics for pixel-wise evaluation. Further, for evaluation as a
binary segmentation task, we select a fixed inverse cosine similarity
threshold for all the datasets and predict $1$ (positive) for pixels
determined as OOD otherwise $0$. This binary prediction mask is compared with
the GT mask and evaluated using mean Intersection over Union (mIoU) and F1
score.

##  5 Results and Discussion

In this section, we show different experiments to evaluate the performance of
PROWL for OOD detection. In Sec. 5.1, we present results comparing with State-
of-the-Art (SOTA) supervised methods in the road driving scene reported in the
existing anomaly segmentation benchmarks SMIYC [7]. In Sec. 5.2, we show the
applicability of PROWL into other domains such as rail and maritime scenes
where there are no directly comparable benchmarks. Lastly, in Sec 5.3, we
provide ablation for different hyper-parameters and strategies used in our
implementation.

We report results for our method PROWL in two scenarios as explained in Sec.
3.3: (a) PROWL (Without foreground masks) - OOD pixels are directly obtained
by thresholding the scores from prototype heatmaps, (b) PROWL + refinement
module using foreground masks - The foreground masks generated by unsupervised
segmentation methods (STEGO [18], CutLER [31] as applicable) are combined with
prototype heatmaps, such that individual masks (not pixels) are detected as
OOD objects or not.

###  5.1 Comparison with the State-of-the-Art on road driving scene

In the road driving scene, we evaluate on datasets with real OOD objects like
RoadAnomaly21 andRoadObstacle21 provided by SMIYC benchmark [7] and the
synthetic OOD objects in FS-Static given in Fishyscapes [3] benchmark. We
created prototype bank for road scene using $19$ ODD classes from Cityscapes,
in accordance with the classes recommended in SMIYC following [10].

|  |  | RoadAnomaly | FS Static | RoadObstacle
---|---|---|---|---|---
Method | Seg. type | Aux. data | AUPR | FPR | AUPR | FPR | AUPR | FPR
Supervised |  |  |  |  |  |  |  |
PEBAL | Pixel | ✓ | 45.1 | 44.6 | 92.1 | 1.5 | - | -
Max. Entropy | Pixel | ✓ | 79.7 | 19.3 | 76.3 | 7.1 | - | -
DenseHybrid | Pixel | ✓ | 63.9 | 43.2 | 60.0 | 4.9 | - | -
M2A | Mask | ✓ | 79.7 | 13.5 | - | - | - | -
EAM | Mask | ✗ | 66.7 | 13.4 |  87.3 |  2.1 | - | -
Maskomaly | Mask | ✗ |  70.9 |  11.9 | 69.5 | 14.4 | 0.96* | 92.10*
Unsupervised |  |  |  |  |  |  |  |
PROWL | Pixel | ✗ | 45.98 | 26.58 |  61.08 |  20.28 | 11.10 | 46.97
PROWL + STEGO | Mask | ✗ |  53.97 |  13.52 | 45.6 | 26.25 |  40.15 |  14.11
PROWL + CutLER | Mask | ✗ | 75.25 | 1.75 | 73.29 | 1.46 | 75.43 | 1.84

Table 1: Results on RoadAnomaly, FS Static and RoadObstacles on the validation
set. We separate the methods based on supervision needed during training. The
best results are marked in bold, and second best are underlined for each
category. * indicates reproduced results with official code and checkpoints.
PROWL+ CutLER outperforms unsupervised variants in all datasets as well as
supervised methods trained without auxiliary data on RoadAnomaly and
RoadObstacle.

To the best of our knowledge, there are no other unsupervised OOD object
detection methods comparable as a baseline to our method. Thus, it should be
noted that all the methods used for comparison with SOTA are supervised, i.e.
these models are trained on semantic annotations of the $19$ classes of
Cityscapes, whereas we rely on few inference steps based on prototype features
from DINOv2.

In Table 1, we compare results reported by other supervised methods on the
validation set of respective OOD datasets. The metrics used are threshold-
independent metrics such as AUPR and FPR at the best TPR (or $95\%$). We note
that while the other reported metrics are based on probabilistic confidence
scores, we calculate positives (OOD pixels) by varying the inverse cosine
similarity threshold in Eq. 4. Thus, we are unable to compare with the test
set of the benchmark due to non-compatibility with expected probabilistic
inputs on the official benchmark code. We indicate the segmentation type of
each method as pixel or mask-based. Further, many SOTA methods achieve
superior results by discriminatively training with auxiliary data, i.e.,
exposed to few OOD samples as negative data. We include those SOTA methods
with backbones having similar predictive power in evaluation and are ranked
higher in the SMIYC benchmark. For comparison purposes, we assume Maskomaly
[1] as our direct supervised baseline as it is also a zero-shot inference
based approach like PROWL, depending on pre-trained Mask2Former [9] model on
Cityscapes. Unlike other methods, it provided SOTA results on RoadAnomaly
amongst the methods not trained with auxiliary data.

We show that PROWL with CutLER performs best compared to all the other
variants amongst the unsupervised methods. In comparison to supervised methods
trained without auxiliary data, PROWL outperforms on the RoadAnomaly and
RoadObstacle datasets. It provides overall best FPR for RoadAnomaly across all
methods and $+4.35\%$ gain in AUPR. For FS Static, our unsupervised methods
perform quite comparable to supervised methods (second best) and performs
better than Maskomaly as well. We note that most supervised methods do not
report results on validation set for RoadObstacles and the reproduced results
for Maskomaly show poor performance, also shown in Fig 3.

Figure 3: Qualitative comparison of different unsupervised methods with PROWL
as compared to supervised baseline Maskomaly. Results generated using fixed
thresholds on the validation set of RoadAnomaly, FS Static and RoadObstacles
datasets. Detected OOD pixel segmentations shown in red. PROWL+CutLER provides
qualitatively superior OOD detection and segmentation across all datasets.

|  | RoadAnomaly | FS Static | RoadObstacle
---|---|---|---|---
Method | Supervised | mIoU | F1 | mIoU | F1 | mIoU | F1
Maskomaly* | Yes | 46.86 | 56.76 | 23.97 | 29.92 | 9.31 | 11.29
PROWL | No | 38.46 | 54.68 | 39.43 | 50.75 | 6.79 | 11.21
PROWL + STEGO | No | 56.24 | 69.81 | 39.21 | 49.9 | 29.92 | 35.10
PROWL + CutLER | No | 75.22 | 85.25 | 64.79 | 72.18 | 49.16 | 53.31

Table 2: Performance of our unsupervised methods compared to supervised
baseline for binary segmentation with fixed thresholds on the validation set
of OOD datasets. * indicates reproduced results with official code and
checkpoints with a threshold of 0.9. PROWL + CutLER outperforms across all
datasets.

In Table 2 and Fig. 3, we present the comparative performance of our methods
against supervised SOTA baseline (Maskomaly [1]) as a binary segmentation
task. For this, we compare the binary GT masks with OOD pixels given as $1$
with the binary masks generated from the pixels predicted as OOD by different
methods. We evaluate mIoU and F1 scores with fixed inverse cosine similarity
thresholds on the best threshold for validation set of respective OOD datasets
for Cityscapes (Fig. 5(c)). The authors of Maskomaly [1] report results on the
validation set by calculating best threshold per-image by checking best F1
score using GT. However, for fair comparison we reproduce all the results for
Maskomaly with their reported fixed confidence threshold of $0.9$.

In Table 2, we show that PROWL with CutLER outperforms other unsupervised
variants as well as supervised baseline in terms of both mIoU and F1 scores
with a significant margin. We observe that there’s an overall decreasing trend
of performance in most methods across OOD datasets from RoadAnomaly, FS-Static
to RoadObstacle. This can be attributed also to the difficulty of the dataset
and the size of OOD object within the image. We can correlate this with the
qualitative results provided in Fig. 3. RoadAnomaly dataset provides real OOD
objects which are comparably big like the carriage or helicopter but also in a
scene different than the city roads in Cityscapes. Thus, while bigger OOD
objects are easier to detect, the background objects might also be often
detected as OOD. We observe this trend for results corresponding to prototype
heatmaps in PROWL and STEGO with PROWL. Although Maskomaly reports good
results on SMIYC benchmark, we still observe that the masks detected for OOD
are highly insufficient with only partial detections. In contrast, since
CutLER detects foreground object boxes and then provides semantic masks, the
entire mask is precisely detected as OOD object using PROWL. For FS-Static,
the OOD objects are synthetic images which often have a matching texture with
the background of Cityscapes test images. So, although most methods do not
falsely predict too many background pixels as OOD, we see some false
predictions in the background for Maskomaly and PROWL in the image of dog
crossing the road. The primary difficulty in this dataset could be missing the
OOD object due to similar texture as background. This is apparent for the
Maskomaly image with dog sitting on road. Overall, STEGO with PROWL performs
quite nicely, as STEGO has learnt via contrastive clustering to find semantic
correspondences in foreground masks on Cityscapes and thus nicely finds small
semantic masks which are significantly different rather than bigger deviations
in the scene as in RoadAnomaly. We observe that CutLER with PROWL precisely
detects the OOD masks in most cases. For the RoadObstacle dataset, the OOD
objects are meant as real obstacles of small sizes and varying numbers placed
on the ego-track at different distances. Due to smaller sizes with increasing
distance, all methods show worse performance as compared to other datasets.
The reproduced Maskomaly results falsely captures most of the road. PROWL
finds the OOD objects in most cases but also falsely predicts many background
pixels as OOD. We observe the superior qualitative performance of object masks
provided by STEGO and CutLER which helps in localising the objects as entire
masks and then detecting them as OOD via PROWL. We note although STEGO
localises all the OOD object masks, the semantic masks are somewhat
discontinuous. In comparison, CutLER with PROWL precisely localises and
segments the OOD object masks, justifying the overall better performance.
Although vanilla PROWL is quite close to detecting the OOD pixels, however
with above examples we show that the additional unsupervised foreground masks
helping in refining the OOD object masks avoiding spurious false predictions.

Figure 4: Qualitative results for zero-shot OOD detection in other ODD domains
- rail and maritime scene. Detected OOD pixel segmentations are depicted in
red colour.

###  5.2 Evaluation of plug and play application to other domains

To demonstrate zero-shot plug-and-play application of PROWL to other domains,
we extend our evaluation to rail and maritime ODD scene with respective OOD
datasets given in Sec. 4.1. For rail ODD, we use the dataset created with
inpainted OOD objects. For simplicity, we consider the following $6$
predominant classes in RailSem19 [33] for creating prototype bank of known
classes - train car, platform, rail, fence, person, pole. Similarly, for
maritime we refer to MODD [4] and create prototype bank with sky, sea as known
ODD classes. Every other object is OOD. Since there exists no benchmark on
RailSem19 OOD detection and the MODS [5] uses separate evaluation strategy, we
provide evaluation using our zero-shot methods with PROWL. Since CutLER models
were trained without labels only on ImageNet [11], it can easily provide zero-
shot inference on any domain, whereas STEGO still needs to train on the
datasets of new domain although without labels. Similarly, our supervised
baseline Maskomaly [1] relies on Mask2Former which still needs to train with
the labels of the respective training data, thus can not be readily used for
comparison.

|  | Rail Inpainted OOD | MODD
---|---|---|---
Method | Supervised | mIoU | F1 | mIoU | F1
PROWL | No | 8.01 | 14.33 | 62.35 | 76.36
PROWL + CutLER | No | 83.29 | 90.38 | 73.30 | 84.08

Table 3: Performance of our unsupervised methods in other domains such as rail
and maritime scenes for binary segmentation with fixed thresholds on the
validation datasets. * indicates reproduced results with official code and
checkpoints with a threshold of 0.9.

In Table 3, we show the performance of PROWL vs CutLER with PROWL as a binary
segmentation task using fixed similarity threshold for detection of OOD
pixels. We find that CutLER with PROWL performs better as compared to vanilla
PROWL in both cases. In Fig. 4, we show corresponding qualitative results. We
observe that PROWL is significantly worse in case Rail Inpainted OOD data as
compared to MODD where it performs quite comparably as CutLER with PROWL. This
could be attributed to the fact that prototype heatmaps in PROWL provide per-
pixel outputs, causing false positives to the background objects like
vegetation, buildings that are not in the assumed ODD class list. CutLER
provides an advantage here, where the relevant object masks (including OOD)
can be filtered based on foreground score while irrelevant background objects
can be ignored. However, in MODD dataset, the task is relatively simple as
everything other than sea, sky should be detected as OOD or obstacle, thus
both methods perform well on this dataset.

###  5.3 Ablations

(a) Average OOD object size

(b) Thresholds for CutLER

(c) Thresholds for PROWL

Figure 5:  Ablation Study: a) Variation of OOD size, b) Performance of PROWL
with variation of Cutler Detector Threshold, c) Performance of PROWL with
variation of Inverse Similarity Threshold, across all OOD datasets for
Cityscapes.

In Fig.5, we show ablation study for different hyperparameters used in the
experiments with PROWL depending upon complexity of different OOD datasets,
such as variation of average OOD object size in pixels Fig. 5(a) where
RoadAnomaly shows much bigger objects as compared to RoadObstacle with smaller
objects. Fig.5(b) shows CutLER thresholds for generating optimal number of
foreground masks for PROWL with the inverse cosine similarity threshold set at
$0.55$. Optimal mIoU is achieved for CutLER thresholds of $0.7$, $0.5$, and
$0.2$ for RoadAnomaly, FS Static, and RoadObstacle datasets respectively. This
trend is justifiable as smaller objects require lesser thresholds so as to
detect maximum object masks, whereas bigger objects can be reliably detected
even with higher detector thresholds. In Fig. 5(c), we fix the CutLER
thresholds given above and find the optimum inverse cosine similarity
threshold for best OOD detection based on mIoU. We observe FS Static peaks at
relatively high threshold of 0.7 as the dataset is quite similar to ID
Cityscapes data, whereas RoadAnomaly and RoadObstacle dataset peak at $0.55$,
$0.60$ respectively close to the default threshold at $0.55$.

Figure 6: Performance of PROWL with variation in the number of prototype
samples

Lastly Fig. 6, shows the performance of PROWL with varying number of prototype
samples used to create the prototype feature bank for the Cityscapes dataset
in road driving domain. We show already quite good AUPRC is achieved with as
less as $5$ prototypes and it starts to saturate from $15$ and above. For our
experiments, $20$ samples per ODD class were used for Cityscapes. For
RailSem19 and MODD datasets with less complexity and lesser number of ODD
classes, approximately $10$ prototype samples were found to be sufficient.

##  6 Conclusion

In this work, we proposed the first unsupervised OOD object detection and
segmentation framework - PROWL. It is a zero-shot plug-and-play framework
which can be easily transferred to different domains without further training
on domain specific data. Since it relies on extracting prototype features from
foundation models, it stands as a practical approach towards many open world
settings. We show that PROWL combined with CutLER outperforms all the
unsupervised as well supervised methods (trained without auxiliary OOD data)
on the SMIYC validation set in road driving scenes. With different OOD
datasets, we show that it can detect real OOD objects of different sizes
(RoadObstacle) as well as diverse scenes (RoadAnomaly). By applying it to rail
and maritime applications, we demonstrate that it can be easily adapted to
other domains. Even with a limited number of classes and prototypes defined in
our ODD setting, PROWL performs reliably on the available benchmarks. However,
due to the limited availability of diverse and high quality data sets for
evaluation, a next step would be to put PROWL to the test in diverse scenarios
with extensively defined ODD . Further, we clearly identified a need for
harmonized evaluation metrics and settings to enable a fair comparison of
approaches in the future.

## References

  * [1] Ackermann, J., Sakaridis, C., Yu, F.: Maskomaly:Zero-Shot Mask Anomaly Segmentation (Aug 2023). https://doi.org/10.48550/arXiv.2305.16972
  * [2] Blum, H., Sarlin, P.E., Nieto, J., Siegwart, R., Cadena, C.: Fishyscapes: A Benchmark for Safe Semantic Segmentation in Autonomous Driving. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). pp. 2403–2412. IEEE, Seoul, Korea (South) (Oct 2019). https://doi.org/10.1109/ICCVW.2019.00294
  * [3] Blum, H., Sarlin, P.E., Nieto, J., Siegwart, R., Cadena, C.: The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation. Int J Comput Vis 129(11), 3119–3135 (Nov 2021). https://doi.org/10.1007/s11263-021-01511-6
  * [4] Bovcon, B., Muhovic, J., Pers, J., Kristan, M.: The MaSTr1325 dataset for training deep USV obstacle detection models. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 3431–3438. IEEE, Macau, China (Nov 2019). https://doi.org/10.1109/IROS40897.2019.8967909
  * [5] Bovcon, B., Muhovič, J., Vranac, D., Mozetič, D., Perš, J., Kristan, M.: MODS – A USV-oriented object detection and obstacle segmentation benchmark. IEEE Trans. Intell. Transport. Syst. 23(8), 13403–13418 (Aug 2022). https://doi.org/10.1109/TITS.2021.3124192
  * [6] Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging Properties in Self-Supervised Vision Transformers. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 9630–9640. IEEE, Montreal, QC, Canada (Oct 2021). https://doi.org/10.1109/ICCV48922.2021.00951
  * [7] Chan, R., Lis, K., Uhlemeyer, S., Blum, H., Honari, S., Siegwart, R., Fua, P., Salzmann, M., Rottmann, M.: SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation
  * [8] Chan, R., Rottmann, M., Gottschalk, H.: Entropy Maximization and Meta Classification for Out-of-Distribution Detection in Semantic Segmentation. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 5108–5117. IEEE, Montreal, QC, Canada (Oct 2021). https://doi.org/10.1109/ICCV48922.2021.00508
  * [9] Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention Mask Transformer for Universal Image Segmentation. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1280–1289. IEEE, New Orleans, LA, USA (Jun 2022). https://doi.org/10.1109/CVPR52688.2022.00135
  * [10] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The Cityscapes Dataset for Semantic Urban Scene Understanding. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3213–3223. IEEE, Las Vegas, NV, USA (Jun 2016). https://doi.org/10.1109/CVPR.2016.350
  * [11] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. pp. 248–255 (Jun 2009). https://doi.org/10.1109/CVPR.2009.5206848
  * [12] Di Biase, G., Blum, H., Siegwart, R., Cadena, C.: Pixel-wise Anomaly Detection in Complex Driving Scenes. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 16913–16922. IEEE, Nashville, TN, USA (Jun 2021). https://doi.org/10.1109/CVPR46437.2021.01664
  * [13] Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The Pascal Visual Object Classes (VOC) Challenge. Int J Comput Vis 88(2), 303–338 (Jun 2010). https://doi.org/10.1007/s11263-009-0275-4
  * [14] Fontanel, D., Tarantino, M., Cermelli, F., Caputo, B.: Detecting the unknown in Object Detection (Aug 2022)
  * [15] Grcić, M., Bevandić, P., Šegvić, S.: DenseHybrid: Hybrid Anomaly Detection for Dense Open-set Recognition (Jul 2022)
  * [16] Grcić, M., Šarić, J., Šegvić, S.: On Advantages of Mask-level Recognition for Outlier-aware Segmentation. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 2937–2947. IEEE, Vancouver, BC, Canada (Jun 2023). https://doi.org/10.1109/CVPRW59228.2023.00295
  * [17] Gupta, A., Narayan, S., Joseph, K.J., Khan, S., Khan, F.S., Shah, M.: OW-DETR: Open-world Detection Transformer. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 9225–9234. IEEE, New Orleans, LA, USA (Jun 2022). https://doi.org/10.1109/CVPR52688.2022.00902
  * [18] Hamilton, M., Zhang, Z., Hariharan, B., Snavely, N., Freeman, W.T.: UNSUPERVISED SEMANTIC SEGMENTATION BY DISTILLING FEATURE CORRESPONDENCES (2022)
  * [19] Hendrycks, D., Gimpel, K.: A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks (Oct 2018). https://doi.org/10.48550/arXiv.1610.02136
  * [20] Hendrycks, D., Mazeika, M., Dietterich, T.: Deep Anomaly Detection with Outlier Exposure (Jan 2019). https://doi.org/10.48550/arXiv.1812.04606
  * [21] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollár, P., Girshick, R.: Segment Anything. In: 2023 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 3992–4003. IEEE, Paris, France (Oct 2023). https://doi.org/10.1109/ICCV51070.2023.00371
  * [22] Lee, K., Lee, K., Lee, H., Shin, J.: A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks (Oct 2018). https://doi.org/10.48550/arXiv.1807.03888
  * [23] Liang, S., Li, Y., Srikant, R.: Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks (Aug 2020). https://doi.org/10.48550/arXiv.1706.02690
  * [24] Lis, K., Nakka, K.K., Fua, P., Salzmann, M.: Detecting the Unexpected via Image Resynthesis. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 2152–2161. IEEE, Seoul, Korea (South) (Oct 2019). https://doi.org/10.1109/ICCV.2019.00224
  * [25] Liu, J., Xie, G., Wang, J., Li, S., Wang, C., Zheng, F., Jin, Y.: Deep Industrial Image Anomaly Detection: A Survey. Mach. Intell. Res. 21(1), 104–135 (Feb 2024). https://doi.org/10.1007/s11633-023-1459-z
  * [26] Nayal, N., Yavuz, M., Henriques, J.F., Güney, F.: RbA: Segmenting Unknown Regions Rejected by All. In: 2023 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 711–722. IEEE, Paris, France (Oct 2023). https://doi.org/10.1109/ICCV51070.2023.00072
  * [27] Oquab, M., Darcet, T., Moutakanni, T., Vo, H.V., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.Y., Li, S.W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., Bojanowski, P.: DINOv2: Learning Robust Visual Features without Supervision
  * [28] Rai, S.N., Cermelli, F., Fontanel, D., Masone, C., Caputo, B.: Unmasking Anomalies in Road-Scene Segmentation
  * [29] Ren, J., Liu, P.J., Fertig, E., Snoek, J., Poplin, R., Depristo, M., Dillon, J., Lakshminarayanan, B.: Likelihood Ratios for Out-of-Distribution Detection. In: Advances in Neural Information Processing Systems. vol. 32. Curran Associates, Inc. (2019)
  * [30] Tian, Y., Liu, Y., Pang, G., Liu, F., Chen, Y., Carneiro, G.: Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation on Complex Urban Driving Scenes (Sep 2022). https://doi.org/10.48550/arXiv.2111.12264
  * [31] Wang, X., Girdhar, R., Yu, S.X., Misra, I.: Cut and Learn for Unsupervised Object Detection and Instance Segmentation. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3124–3134. IEEE, Vancouver, BC, Canada (Jun 2023). https://doi.org/10.1109/CVPR52729.2023.00305
  * [32] Yu, T., Feng, R., Feng, R., Liu, J., Jin, X., Zeng, W., Chen, Z.: Inpaint Anything: Segment Anything Meets Image Inpainting (Apr 2023). https://doi.org/10.48550/arXiv.2304.06790
  * [33] Zendel, O., Murschitz, M., Zeilinger, M., Steininger, D., Abbasi, S., Beleznai, C.: RailSem19: A Dataset for Semantic Rail Scene Understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 0–0 (2019)
  * [34] Zohar, O., Wang, K.C., Yeung, S.: PROB: Probabilistic Objectness for Open World Object Detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11444–11453 (2023)


# Electron correlations and charge density wave in the topological kagome
metal FeGe

Chandan Setty⊕,† Department of Physics & Astronomy, Rice Center for Quantum
Materials, Rice University, Houston, Texas 77005,USA  Christopher A. Lane ⊕
Theoretical Division, Los Alamos National Laboratory, Los Alamos, New Mexico
87545 USA  Lei Chen  Department of Physics & Astronomy, Rice Center for
Quantum Materials, Rice University, Houston, Texas 77005,USA  Haoyu Hu
Department of Physics & Astronomy, Rice Center for Quantum Materials, Rice
University, Houston, Texas 77005,USA  Jian-Xin Zhu  Theoretical Division, Los
Alamos National Laboratory, Los Alamos, New Mexico 87545 USA  Qimiao Si
Department of Physics & Astronomy, Rice Center for Quantum Materials, Rice
University, Houston, Texas 77005,USA

###### Abstract

Charge order in kagome metals is of extensive current interest. Recently, a
charge density wave was discovered in the magnetic binary kagome metal FeGe
Teng  _et al._ (2022). In analogy to its predecessor, the non-magnetic
$A$V3Sb5 ($A$=K, Cs, Rb), the in-plane ordering occurs at the $M$ point. In
contrast, however, the system manifestly shows effects of substantial
correlations. Here we identify the topological bands crossing the Fermi energy
(EF) in FeGe and characterize the correlation-induced renormalization of these
bands. We then derive a charge order from an effective model comprising
topological kagome ‘flat’ bands in the presence of a magnetic order. We
demonstrate edge states as well as excess out-of-plane magnetic moment
associated with the charge order; both are fingerprints of non-trivial band
topology and consistent with recent experimental observations. Our results
point to FeGe as an ideal platform to realize and elucidate  correlated
topological physics.

Introduction: Charge order phases in kagome metals have garnered much recent
attention owing to their highly unconventional properties. Especially in the
vanadium based non-magnetic ternary kagome materials ($A$V3Sb5; $A$=K, Cs, Rb)
Ortiz  _et al._ (2019, 2020, 2021), charge order has been found Jiang  _et
al._ (2021); Zhao  _et al._ (2021); Zhou  _et al._ (2021); Liang  _et al._
(2021); Uykur  _et al._ (2021); Li  _et al._ (2021a); Hu  _et al._ (2021) and
shows several non-trivial characteristics.  These include the breaking of
time-reversal (T) symmetry au2  _et al._ (2021); Yu  _et al._ (2021); Wu  _et
al._ (2021), anomalous Hall transport Yang  _et al._ (2020), nematicity Li
_et al._ (2021b); Wu  _et al._ (2021) and other accompanying/competing orders
Oey  _et al._ (2021); Liang  _et al._ (2021); Li  _et al._ (2021a); Wu  _et
al._ (2022), and possible coexistence with unconventional superconductivity
Chen  _et al._ (2021).

Another prominent class of transition element kagome metals is the binary TmXn
(T= Fe, Co, Ni and X= Ge, Sn). They have magnetic ground states Ye  _et al._
(2018); Liu  _et al._ (2020); Lin  _et al._ (2018); Liu  _et al._ (2018); Yao
_et al._ (2018) and possess flat bands near the Fermi level (EF) Kang  _et
al._ (2020a, b); Liu  _et al._ (2020). Recently hexagonal FeGe (SG 191,
P6/$mmm$) was shown to be the first member in this family ($T_{m}\sim$ 400 K
Watanabe and Kunitomi (1966)) where charge order is identified, with a
transition temperature of $T_{CO}\sim 110$ K Teng  _et al._ (2022); Yin  _et
al._ (2022); et al (2022). The spins are aligned (anti-)ferromagnetically
(between) within the kagome planes and point in a direction perpendicular to
the layers Watanabe and Kunitomi (1966).

Much of the currently known phenomenology of the charge order is analogous to
the ternary compounds – the in-plane ordering occurs at $M$ point ($2\times 2$
order) and the STM peaks are sensitive to the directionality of the applied
perpendicular ($c$-axis) magnetic field. However, in contrast with the ternary
family, here the charge order emerges from a magnetic parent phase Teng  _et
al._ (2022), and is accompanied by edge states Yin  _et al._ (2022) and  an
excess contribution to the ordered magnetic moment Teng  _et al._ (2022) below
$T_{CO}$.

In this work we develop a model of charge order in FeGe. We identify two
minimal ingredients – the presence of flat bands crossing EF and their non-
trivial topology – that can broadly capture much of the observed data. We
justify the relevance of these two ingredients to FeGe using first principle
calculations. To isolate the salient physics from the complexities of the
real-material  electronic structure, we begin from an extended Hubbard model
on a kagome lattice. We then seek a low energy model for the topological flat
bands by working in the limit where Coulomb interactions are comparable to the
bandwidth of the flat bands. By solving the low-energy model non-
perturbatively using the slave-spin method, we analyze the nature of the
resulting charge order phase with focus on the role played by the parent
magnetic phase. We find that the non-trivial band topology of the kagome
lattice, and its interplay with charge order and magnetism give rise to
topological edge states in the charge order gap. Another key result we uncover
is an excess magnetic moment, which points along the existing magnetic order,
arising from the intertwining of charge and spin degrees of freedom – a
property unique to correlated phases derived from topological bands Setty  _et
al._ (2021). Thus we identify previously unknown fingerprints of non-trivial
band topology manifest in correlated phases.

Our strategy is to isolate a pair of flat bands crossing EF with chern numbers
$\pm 1$ that are split by the internal field of the parent magnetic phase. To
write an effective model with interactions for this pair of ‘active’ bands, we
follow the framework developed by several of us in Ref. Setty  _et al._ (2021)
where such a charge order was already predicted for binary kagome metals.
Wannier construction of exponentially localized wannier orbitals (WOs)
Soluyanov and Vanderbilt (2011, 2012); Marzari  _et al._ (2012); Kuneš  _et
al._ (2010) allows for tight binding description of the low energy topological
bands, from which we can study the correlation effect.

Figure 1:  (Top row) First principles band structure of FeGe in the
nonmagnetic (left) and magnetic (right) phases with SOC. (Bottom row) Same as
top row but with  Coulomb interactions (U = 4 eV) accounted for through DMFT.

Topological bands: Before describing the minimal interacting model, we discuss
the electronic structure of FeGe and its corresponding topological
characteristics. Fig. 1 (top row) shows the spin-orbit coupled bands,
calculated by density-functional theory (DFT), in the nonmagnetic  (NM) (left
panel) and magnetic (right panel) phases respectively. In both the nonmagnetic
phase (SG P6/$mmm$) and the magnetic one (SG
P6/$m^{\prime}m^{\prime}m^{\prime}$), the flat bands directly cross EF.

Fig. 2 (top left) highlights three topological bands near EF in the  NM phase
with spin-orbit coupling (SOC) in green, red and purple. The topological
indices for these  band pairs appear in Fig. 2 (top right).

Electron correlations and DMFT calculations: FeGe shows the effect of
substantial electron correlations Teng  _et al._ (2022). It has a relatively
large room temperature resistivity, $\sim 160\mu\Omega\cdot$cm,  which reaches
the Mott-Ioffe-Regel value, implicating FeGe as a bad metal. The electronic
bands also show substantial correlation-induced renormalization Teng  _et al._
(2022); et al (2022). We characterize this correlation effect using dynamical
mean field theory (DMFT). The DFT$+$DMFT Haule  _et al._ (2010); Blaha  _et
al._ (2020) results are shown in the bottom row in Fig. 1 . The flat bands
become very narrow and the spectral weight is highly concentrated at EF in
both the  NM and magnetic phases. See also Supplemental Material (SM) Fig. 5.

Importantly, the topologically non-trivial flat bands still cross EF. The
density of states (DOS) near EF are completely dominated by the flat bands in
both the NM and magnetic phases. This is highlighted by black arrows in Fig. 2
(bottom left). The conclusion stands in the presence of interactions as seen
in DMFT (bottom right panel of Fig. 2).

Figure 2:  (Top left) The three topological bands near EF in the NM phase
where the gaps are shaded in blue and red, (top right) $Z_{2}$ number for each
of the six time reversal invariant planes and resulting $Z_{2}$ topological
index for the (green, red) and (violet,red) pair of bands as shown in the top
left panel. (Bottom left) DOS in the NM and magnetic phases without
interactions and (Bottom right) DOS results from DMFT. The black arrows
indicate flat band contributions to DOS.

Effective model for correlated topological bands: Having provided evidence for
the existence of topological flat bands crossing EF, we now utilize the
minimal ingredients to study correlation-driven charge order. As a proof-of-
principle demonstration, we focus on the intralayer couplings within a kagome
layer. The minimal total Hamiltonian in the physical electron basis (UV limit)
is given as $\mathscr{H}=\mathscr{H}_{0}+\mathscr{H}_{I}$. The kinetic part
$\mathscr{H}_{0}$ is described by electrons hopping on a kagome lattice in the
presence of a Zeeman field of the magnetic order within the plane and is
written as

$\displaystyle\mathscr{H}_{0}$ $\displaystyle=$ $\displaystyle-t\sum_{\langle
ij\rangle\alpha\beta}c_{i\alpha}^{\dagger}c_{j\beta}+i\lambda_{1}\sum_{\langle
ij\rangle\alpha\beta}\left(\textbf{E}_{ij}\times\textbf{R}_{ij}\right)\cdot
c_{i\alpha}^{\dagger}\sigma c_{j\beta}$ (1) $\displaystyle-
t_{2}\sum_{\langle\langle
ij\rangle\rangle\alpha\beta}c_{i\alpha}^{\dagger}c_{j\beta}+i\lambda_{2}\sum_{\langle\langle
ij\rangle\rangle\alpha\beta}\left(\textbf{E}_{ij}\times\textbf{R}_{ij}\right)\cdot
c_{i\alpha}^{\dagger}\sigma c_{j\beta}$
$\displaystyle+h\sum_{i\alpha}c_{i\alpha}^{\dagger}\sigma_{z}c_{i\alpha}.$

Here $c_{j\alpha}^{\dagger}$ creates an electron at site $j$ and internal
quantum number $\alpha$. The internal quantum numbers include both sub-lattice
and spin indices. $t,t_{2}$ and $\lambda_{1,2}$ are the hopping and SOC
parameters for the nearest neighbor and next-nearest neighbor lattice sites
respectively. The displacement from site $i$ to site $j$ is $\textbf{R}_{ij}$,
and the electric field experienced by the electrons along $\textbf{R}_{ij}$ is
$\textbf{E}_{ij}$. The last term arises due to the Zeeman field experienced by
the electrons from the magnetic moments pointed out-of-plane in a quasi 2D
kagome plane. We assume this field to be uniform and proportional to $h$. Fig.
3 (left panel) describes the lattice structure and magnetically ordered
moments.

Figure 3:  (Left) The kagome lattice of FeGe in the magnetic phase with
ferromagnetic in-plane and anti-ferromagnetic out-of-plane moments. (Right)
Band structure in the presence of spin-orbit coupling with the ’active’
topological bands near EF (dashed line) marked in blue.

The interaction part, $\mathscr{H}_{I}$, is given by the extended Hubbard term
on the kagome lattice

$\displaystyle\mathscr{H}_{I}$ $\displaystyle=$
$\displaystyle\sum_{ij\alpha\beta}U_{j,\alpha\beta}n_{i\alpha}n_{i+j\beta}\quad.$
(2)

It contains the onsite Hubbard interaction $U$, and nearest and next-nearest
neighbor density interactions $V$ and $V^{\prime}$ respectively. For
convenience, these parameters are succinctly expressed in terms of
$U_{j,\alpha\beta}$ and the corresponding number operators are denoted by
$n_{i\alpha}$. We use the primitive vectors of the triangular lattice
$\textbf{n}_{1}=2a(\frac{1}{2},\frac{\sqrt{3}}{2})$,
$\textbf{n}_{2}=2a(-\frac{1}{2},\frac{\sqrt{3}}{2})$, $\textbf{n}_{3}=2a(1,0)$
where $a$ is the nearest neighbor inter atomic spacing, to write the minimal
band structure from the kinetic Hamiltonian Eq. 1. The non-interacting bands
of the minimal model are shown in Fig. 3 (right panel). There are three pairs
of bands separated in energy and each is split by the Zeeman field. Each pair
has chern numbers $C=\pm 1,0,\pm 1$ from highest to lowest energy. Note that
the total chern number for each pair of bands sums to zero even the presence
of the Zeeman field Foo .

To mimic the  realistic electronic structure of FeGe, we place the chemical
potential in the lowest pair of flat bands with $C=\pm 1$ as highlighted in
the blue region in Fig. 3 (right panel). We seek a low energy model describing
this pair of bands. To this end, we wannierize the bands to derive
exponentially localized WOs Soluyanov and Vanderbilt (2011, 2012); Setty  _et
al._ (2021). Denoting $b_{i\mu}^{\dagger}$ as the creation operator in the WO
basis on site $i$, the low energy (IR) Hamiltonian obtained after projecting
on the bands crossing EF is written as $H=H_{0}+H_{I}$ where

$\displaystyle H_{0}$ $\displaystyle=$
$\displaystyle\sum_{ij}\sum_{\mu\nu}b_{i\mu}^{\dagger}t_{ij}^{\mu\nu}b_{j\nu}$
$\displaystyle H_{I}$ $\displaystyle=$
$\displaystyle\sum_{ijkl}\sum_{mu\mu^{\prime}\nu\nu^{\prime}}u_{\mu\mu^{\prime}\nu\nu^{\prime}}(j,k,l)b_{i\mu}^{\dagger}b_{i+j\mu^{\prime}}b_{i+k\nu}^{\dagger}b_{i+l\nu^{\prime}}\quad.$
(3)

Here $t_{ij}^{\mu\nu}$ and $u_{\mu\mu^{\prime}\nu\nu^{\prime}}(j,k,l)$ are
respectively the hopping parameters  (see SM Table 1 and SM Fig. 8) and
interaction matrix elements. The interaction parameters are listed in the SM
(Tables I-III) of Ref. Setty  _et al._ (2021).  It is noteworthy that the
interaction parameters $u_{\mu\mu^{\prime}\nu\nu^{\prime}}(j,k,l)$ are
strictly non-local with no onsite interactions. This is because the
wannierization procedure splits the WO centers in real space (See Fig.1(b) of
Setty  _et al._ (2021) ). This lack of a common wannier center arises from the
$C=\pm 1$ chern distribution and is hence an intrinsic topological property. A
key consequence of the wannier center split is the coupling of charge and spin
degrees of freedom and is ultimately responsible for the enhanced $T$-symmetry
breaking upon entry into the charge order state Setty  _et al._ (2021).

Figure 4:  Edge states with the chemical potential located in the upper sub-
band and a gapless bulk spectrum. The color scale marks the spectral weight of
the states. (Left panel) Edge states in the parent magnetic order. (Right
panel) Edge states in the charge order phase derived from magnetic order.

A full analysis of the Hamiltonian in Eq. 3 without the magnetic order appears
in Ref. Setty  _et al._ (2021). We have explicitly checked that a similar
analysis in the presence of a Zeeman term due to the magnetic order yields an
analogous phase diagram with the phase boundaries modified only
quantitatively. Above a critical next nearest neighbor interaction
$V^{\prime}$, a correlation driven charge order phase at the $M$ point in the
Brillouin zone (BZ) is stable.  This order is interesting since, for trivial
bands, the two WOs share the same Wannier center and a minimization of the
onsite Coulomb repulsion would have yielded a spin order. Instead, in our
case, the non-local $V^{\prime}$ interaction dominates and drives a charge
order at $M$.

The charge order also couples to an applied perpendicular magnetic field due
to the topology driven mixing between the charge and spin order consistent
with recent STM data Yin  _et al._ (2022).  For further details regarding
these conclusions, we refer the reader to Ref. Setty  _et al._ (2021). For the
purposes of the current work, we focus on the edge states and enhanced moment
in the charge order phase. Here, the Hamiltonian in the
$c_{\textbf{k}\alpha\sigma}$ basis takes the form

$\displaystyle H$ $\displaystyle=$
$\displaystyle\sum_{\textbf{k},\alpha,\gamma}\bigg{[}c_{\textbf{k},\alpha,\uparrow}^{\dagger}[h_{0,\textbf{k}}]_{\alpha\gamma}c_{\textbf{k},\gamma,\uparrow}+c_{\textbf{k},\alpha,\downarrow}^{\dagger}[h^{*}_{0,-\textbf{k}}]_{\alpha\gamma}c_{\textbf{k},\gamma,\downarrow}\bigg{]}$
$\displaystyle+$
$\displaystyle\sum_{\textbf{k},\alpha}h_{0}(c_{\textbf{k},\alpha,\uparrow}^{\dagger}c_{\textbf{k},\alpha,\uparrow}-c_{\textbf{k},\alpha,\downarrow}^{\dagger}c_{\textbf{k},\alpha,\downarrow})$
$\displaystyle+$
$\displaystyle\sum_{\textbf{k},\alpha,\gamma,\sigma,\sigma^{\prime}}\bigg{[}\sum_{a}\phi_{\textbf{Q}}(u_{\textbf{k}+\textbf{Q},\alpha,\sigma}^{a})^{*}u_{\textbf{k},\gamma,\sigma^{\prime}}^{a}\bigg{]}c^{\dagger}_{\textbf{k}+\textbf{Q},\alpha,\sigma}c_{\textbf{k},\gamma,\sigma^{\prime}}$

where $\alpha,\gamma$ denotes sublattice indices. $h_{0,\textbf{k}}$ is the
hopping matrix of spin up electron with bare dispersions
$\epsilon_{\textbf{k}},\epsilon_{\textbf{k}+\textbf{Q}}$ and $h_{0}$ is the
magnetic field. $\phi_{\textbf{Q}}$ denotes the charge order parameter in the
wannier-orbital basis which is a real number.
$u_{\textbf{k},\alpha,\sigma}^{a}$ is the wavefunction of WO $a$, defined as
$b_{a,\textbf{k}}=\sum_{\alpha,\sigma}u_{\textbf{k},\alpha,\sigma}c_{\textbf{k},\alpha,\sigma}$,
and is related to the Haldane matrix as described in Ref. Setty  _et al._
(2021).

The terms that mix two opposite spins $\sigma$ and $\bar{\sigma}$ (Eq.
Electron correlations and charge density wave in the topological kagome metal
FeGe, with a finite momentum Q) couple charge and spin. They would not have
existed for charge order from topologically trivial bands when the SOC is
directed out of the kagome plane. The uniqueness of such terms to correlated
orders derived from topological bands is due to wannier center splitting
discussed above Setty  _et al._ (2021). This effect will be crucial for
understanding the charge order-induced magnetic moment appearing later.

We can now obtain edge states by solving Eq. Electron correlations and charge
density wave in the topological kagome metal FeGe on a slab. Fig. 4 shows a
plot of the edge state spectrum along the $x=0$ edge in the reduced BZ. The
spectral intensities are shown in a color scale. The left (right) panel
corresponds to the edge state spectrum in the parent magnetic order (charge
order) phase. Whereas the edge states connecting the top and bottom sub-bands
are overwhelmed by the gapless bulk bands in the magnetic parent phase, they
completely dominate in the charge order phase. Additional edge state analysis
when the bulk is fully gapped by the Zeeman field appears in the SM Fig. 7.
The origin of the edge state behavior is tied to the non-zero chern numbers of
the kagome flat bands crossing EF and a non-trivial coupling between the
magnetic and charge orders. The out-of-plane field of the magnetic order
splits the non-interacting bands with equal and opposite chern numbers and the
low energy (non-interacting) model written in the original $c$ electron basis
can be mapped to the Haldane bands. Hence the suppression of the density of
states at the chemical potential due to the charge order enables the
observation of edge states as observed in STM Yin  _et al._ (2022).

We now illustrate our topological mechanism for the enhanced magnetic moment,
at the in-plane $\textbf{q}=0$, along the positive $c$-axis in the charge
ordered phase. We do this by expanding Eq. Electron correlations and charge
density wave in the topological kagome metal FeGe in a Ginzburg-Landau type of
free energy in terms of the charge order parameters
$\Delta_{\textbf{Q}}\sim\sum_{\textbf{k}}\langle
c_{\textbf{k}\alpha\uparrow}^{\dagger}c_{\textbf{k}+\textbf{Q}\alpha\downarrow}\rangle$
and $\Delta^{\prime}_{\textbf{Q}}\sim\sum_{\textbf{k}}\langle
c_{\textbf{k}+\textbf{Q}\alpha\uparrow}^{\dagger}c_{\textbf{k}\alpha\downarrow}\rangle$.
For a single sub-lattice $\alpha$, this is given as

$\displaystyle F[\Delta_{\textbf{Q}_{i}},\Delta_{\textbf{Q}_{i}}^{\prime}]$
$\displaystyle=$ $\displaystyle
F_{0}[\Delta_{\textbf{Q}_{i}},\Delta_{\textbf{Q}_{i}}^{\prime}]+\Delta
F[\Delta_{\textbf{Q}_{i}},\Delta_{\textbf{Q}_{i}}^{\prime}]$ (5)
$\displaystyle\Delta
F[\Delta_{\textbf{Q}_{i}},\Delta_{\textbf{Q}_{i}}^{\prime}]$ $\displaystyle=$
$\displaystyle-h^{\prime}\gamma\sum_{i}\left(\Delta_{\textbf{Q}_{i}}\Delta_{-\textbf{Q}_{i}}+\Delta_{\textbf{Q}_{i}}^{\prime}\Delta_{-\textbf{Q}_{i}}^{\prime}\right).$

Here $F_{0}$ is independent of a ‘probe’ field $h^{\prime}$ (SM, Eq. 10), and
$\gamma\simeq\frac{3h_{0}}{2T_{CO}^{4}}\left[\frac{-1+\tilde{h}_{0}^{-1}\sinh\tilde{h}_{0}}{\tilde{h}_{0}^{2}(1+\cosh\tilde{h}_{0})}\right]$
with $\tilde{h}_{0}=h_{0}/T_{CO}$ in the nested, flat band limit of
$\epsilon_{\textbf{k}}=\epsilon_{\textbf{k}+\textbf{Q}_{i}}\sim 0$.
$\textbf{Q}_{i}$ are the $M$ point wave vectors corresponding to the $2\times
2$ charge order. The $\Delta F$ term (see SM Fig. 9) couples charge and
$\textbf{Q}=0$ magnetic orders and is odd in the static and probe fields
$h_{0}$ and $h^{\prime}$. The induced magnetic moment becomes

$\displaystyle\delta M=-\frac{\partial F[\Delta_{\textbf{Q}_{i}}]}{\partial
h^{\prime}}\propto\gamma(\Delta_{\textbf{Q}_{i}}\Delta_{-\textbf{Q}_{i}}+\Delta^{\prime}_{\textbf{Q}_{i}}\Delta^{\prime}_{-\textbf{Q}_{i}}).$
(6)

From the expression of $\gamma$ above, the induced $\delta M$ is along the
direction of the static magnetic field ($h_{0}$) and rises quadratically in
the order parameter upon entry into the charge order phase. This result is
consistent with neutron scattering data Teng  _et al._ (2022).

As alluded to earlier, the terms proportional to $\gamma$ in Eqs. 5, 6 have a
topological origin. From the definitions of $\Delta_{\textbf{Q}}$ and
$\Delta_{\textbf{Q}}^{\prime}$, it is evident that the order parameters have a
finite wave vector and mix opposite spins, hence coupling spin and charge.
This is absent for charge order derived from trivial bands with an out-of-
plane SOC. In fact, the finite wave vector non-spin flip terms of the type
$\sum_{\textbf{k}}\langle
c_{\textbf{k}+\textbf{Q}\alpha\sigma}^{\dagger}c_{\textbf{k}\alpha\sigma}\rangle$
do not pin the induced magnetic moment to $h_{0}$. Hence the induced $\delta
M$ along the static $h_{0}$ is a fingerprint of topology of the parent
magnetic phase.

Discussion: Several points are in order. While the suppression of the DOS
close to the chemical potential due to the charge order enables the
observation of edge states in STM, it should be noted that the bulk states
need not be fully gapped or even nodal for the observation of edge states.
Even a modest suppression of the spectral weight at EF – 30 %- 40% of the high
temperature value with a large residual density of states – is sufficient for
the edge spectrum to be discernible. This is consistent with our calculations
in Figs 4 where edge states have a large spectral weight despite the lack of a
full gap in the charge order phase.

In principle, such topological edge states should exist in the purely magnetic
phase as well ($T_{CO}$$<T<T_{m}$). However, both DFT and DMFT demonstrate
large density of states of the bulk bands near EF in this regime.
Correspondingly, edge states are expected to be  smeared by the bulk bands
crossing EF. This picture is supported by our calculations in Fig. 4. In fact,
topological surface states are expected to exist even in the normal state
above $T_{m}$ due to the non-trivial topology of the bands, albeit mixed with
the more dominant bulk states. Recently such states were observed in
photoemission measurements of the ternary kagome compounds Hu  _et al._ (2021)
in both the normal and charge ordered phases.

We have delineated the role of van Hove singularities in the electronic
structure. They are located $\sim-0.6eV$ below EF in the nonmagnetic phase,
and are pushed closer to EF in the magnetic phase. However, the flat bands
always dominate the DOS. As seen in the bottom panels Fig. 1, and further
highlighted in SM (Fig. 6), the density of states (DOS) from the van Hove
singularities is essentially completely obscured by the topological flat
bands. This renders the possibility of a strong coupling topological mechanism
for the charge order more viable and robust Setty  _et al._ (2021). Finally,
we have illustrated the topological mechanism by focusing on an effective
model with only intra-layer coupling. Generalization of our analysis to  the
more realistic 3D case is left for future investigations.

In conclusion, the recent discovery of charge order in the topological flat
band magnetic kagome metal FeGe opens up a new platform for exploring flat
band correlated phases and their topological properties.  While the charge
order in FeGe shares many features with its ternary counterparts $A$V3Sb5 ($A$
= K, Cs, Rb), it is also distinct in important ways. The strong correlation
driven band renormalizations, active topological flat bands, appearance of
edge states, and enhancement of the magnetic moments below the charge ordering
temperature are a few key aspects unique to FeGe.  Beyond what was discussed
in the nonmagnetic setting Setty  _et al._ (2021), namely correlation driven
charge order, ordering at the wavevector $M$ in the BZ, and breaking of time-
reversal symmetry,  we have shown here that our minimal model in a magnetic
setting provides a qualitative understanding of  the most salient and puzzling
features of the charge order in FeGe: (i) the existence of edge states and
(ii) development of an excess magnetic moment with the in-plane
$\textbf{q}=0$. The origin of all these exotic properties can be traced to the
non-trivial band topology of the magnetic parent phase. As such, our results
will likely have general implications on the interplay of topology and
correlations in a broad range of correlated systems.

Acknowledgements: We thank P. C. Dai, M. Yi, J. X. Yin for sharing their
results with us and for useful discussions. This work has in part been
supported by the AFOSR under Grant No. FA9550-21-1-0356 (C.S. and Q.S.) and
the Robert A. Welch Foundation Grant No. C-1411 (L. C. and H.H.). Work at Los
Alamos was supported by LANL LDRD Program, UC Laboratory Fees Research Program
(Grant Number: LFR-20-653926), and in part by Center for Integrated
Nanotechnologies, a U.S. DOE BES user facility. One of us (Q.S.) acknowledges
the hospitality of the Aspen Center for Physics, which is supported by NSF
grant No. PHY-1607611.

$\oplus$ These authors contributed equally to this work.
$\dagger$ Email<EMAIL_ADDRESS>

## References

  * Teng  _et al._ (2022) Xiaokun Teng, Lebing Chen, Feng Ye, Elliott Rosenberg, Zhaoyu Liu, Jia-Xin Yin, Yu-Xiao Jiang, Ji Seop Oh, Zahid Hasan, Kelly Neubauer, Bin Gao, Yaofeng Xie, Makoto Hasimoto, Donghui Lu, Chris Jozwiak, Aaron Bostwick, Eli Rotenberg, Robert Birgeneau, Jiun-Haw Chu, Ming Yi, and Pengcheng Dai, “Discovery of charge density wave in a correlated kagome lattice antiferromagnet,” Submitted (2022).
  * Ortiz  _et al._ (2019) Brenden R Ortiz, Lídia C Gomes, Jennifer R Morey, Michal Winiarski, Mitchell Bordelon, John S Mangum, Iain WH Oswald, Jose A Rodriguez-Rivera, James R Neilson, Stephen D Wilson, _et al._ , “New kagome prototype materials: discovery of kv 3 sb 5, rbv 3 sb 5, and csv 3 sb 5,” Physical Review Materials 3, 094407 (2019).
  * Ortiz  _et al._ (2020) Brenden R Ortiz, Samuel ML Teicher, Yong Hu, Julia L Zuo, Paul M Sarte, Emily C Schueller, AM Milinda Abeykoon, Matthew J Krogstad, Stephan Rosenkranz, Raymond Osborn, _et al._ , “Cs v 3 sb 5: A z 2 topological kagome metal with a superconducting ground state,” Physical Review Letters 125, 247002 (2020).
  * Ortiz  _et al._ (2021) Brenden R Ortiz, Paul M Sarte, Eric M Kenney, Michael J Graf, Samuel ML Teicher, Ram Seshadri, and Stephen D Wilson, “Superconductivity in the z 2 kagome metal kv 3 sb 5,” Physical Review Materials 5, 034801 (2021).
  * Jiang  _et al._ (2021) Yu-Xiao Jiang, Jia-Xin Yin, M Michael Denner, Nana Shumiya, Brenden R Ortiz, Junyi He, Xiaoxiong Liu, Songtian S Zhang, Guoqing Chang, Ilya Belopolski, _et al._ , “Unconventional chiral charge order in kagome superconductor kv3sb5,” Nature Materials 20, 1353 (2021).
  * Zhao  _et al._ (2021) He Zhao, Hong Li, Brenden R Ortiz, Samuel ML Teicher, Taka Park, Mengxing Ye, Ziqiang Wang, Leon Balents, Stephen D Wilson, and Ilija Zeljkovic, “Cascade of correlated electron states in a kagome superconductor csv3sb5,” Nature (2021), https://doi.org/10.1038/s41586-021-03946-w.
  * Zhou  _et al._ (2021) Xiaoxiang Zhou, Yongkai Li, Xinwei Fan, Jiahao Hao, Yaomin Dai, Zhiwei Wang, Yugui Yao, and Hai-Hu Wen, “Origin of the Charge Density Wave in the Kagome Metal CsV3Sb5 as Revealed by Optical Spectroscopy,” arXiv e-prints , arXiv:2104.01015 (2021), arXiv:2104.01015 [cond-mat.supr-con] .
  * Liang  _et al._ (2021) Zuowei Liang, Xingyuan Hou, Wanru Ma, Fan Zhang, Ping Wu, Zongyuan Zhang, Fanghang Yu, J-J Ying, Kun Jiang, Lei Shan, _et al._ , “Three-dimensional charge density wave and robust zero-bias conductance peak inside the superconducting vortex core of a kagome superconductor csv $\\_3$ sb $\\_5$,” arXiv preprint arXiv:2103.04760 (2021).
  * Uykur  _et al._ (2021) E Uykur, BR Ortiz, SD Wilson, M Dressel, and AA Tsirlin, “Optical detection of charge-density-wave instability in the non-magnetic kagome metal kv $\\_3$ sb $\\_5$,” arXiv preprint arXiv:2103.07912 (2021).
  * Li  _et al._ (2021a) HX Li, TT Zhang, Y-Y Pai, C Marvinney, A Said, T Yilmaz, Q Yin, C Gong, Z Tu, E Vescovo, _et al._ , “Observation of unconventional charge density wave without acoustic phonon anomaly in kagome superconductors av3sb5 (a= rb, cs),” arXiv preprint arXiv:2103.09769 (2021a).
  * Hu  _et al._ (2021) Yong Hu, Samuel ML Teicher, Brenden R Ortiz, Yang Luo, Shuting Peng, Linwei Huai, JZ Ma, NC Plumb, Stephen D Wilson, J-F He, _et al._ , “Charge-order-assisted topological surface states and flat bands in the kagome superconductor csv $\\_3$ sb $\\_5$,” arXiv preprint arXiv:2104.12725 (2021).
  * au2  _et al._ (2021) C. Mielke III au2, D. Das, J. X. Yin, H. Liu, R. Gupta, C. N. Wang, Y. X. Jiang, M. Medarde, X. Wu, H. C. Lei, J. J. Chang, P. Dai, Q. Si, H. Miao, R. Thomale, T. Neupert, Y. Shi, R. Khasanov, M. Z. Hasan, H. Luetkens, and Z. Guguchia, “Time-reversal symmetry-breaking charge order in a correlated kagome superconductor,” (2021), arXiv:2106.13443 [cond-mat.mtrl-sci] .
  * Yu  _et al._ (2021) Li Yu, Chennan Wang, Yuhang Zhang, Mathias Sander, Shunli Ni, Zouyouwei Lu, Sheng Ma, Zhengguo Wang, Zhen Zhao, Hui Chen, Kun Jiang, Yan Zhang, Haitao Yang, Fang Zhou, Xiaoli Dong, Steven L. Johnson, Michael J. Graf, Jiangping Hu, Hong-Jun Gao, and Zhongxian Zhao, “Evidence of a hidden flux phase in the topological kagome metal csv3sb5,” (2021), arXiv:2107.10714 [cond-mat.supr-con] .
  * Wu  _et al._ (2021) Qiong Wu, Z. X. Wang, Q. M. Liu, R. S. Li, S. X. Xu, Q. W. Yin, C. S. Gong, Z. J. Tu, H. C. Lei, T. Dong, and N. L. Wang, “The large static and pump-probe kerr effect with two-fold rotation symmetry in kagome metal csv3sb5,” (2021), arXiv:2110.11306 [cond-mat.supr-con] .
  * Yang  _et al._ (2020) Shuo-Ying Yang, Yaojia Wang, Brenden R Ortiz, Defa Liu, Jacob Gayles, Elena Derunova, Rafael Gonzalez-Hernandez, Libor Šmejkal, Yulin Chen, Stuart SP Parkin, _et al._ , “Giant, unconventional anomalous hall effect in the metallic frustrated magnet candidate, kv3sb5,” Science advances 6, eabb6003 (2020).
  * Li  _et al._ (2021b) Hong Li, He Zhao, Brenden R Ortiz, Takamori Park, Mengxing Ye, Leon Balents, Ziqiang Wang, Stephen D Wilson, and Ilija Zeljkovic, “Rotation symmetry breaking in the normal state of a kagome superconductor kv3sb5,” arXiv preprint arXiv:2104.08209 (2021b).
  * Oey  _et al._ (2021) Yuzki M. Oey, Brenden R. Ortiz, Farnaz Kaboudvand, Jonathan Frassineti, Erick Garcia, Samuele Sanna, Vesna Mitrovi?, Ram Seshadri, and Stephen D. Wilson, “Fermi level tuning and double-dome superconductivity in the kagome metals csv3sb5-xsnx,” (2021), arXiv:2110.10912 [cond-mat.supr-con] .
  * Wu  _et al._ (2022) Shangfei Wu, Brenden R. Ortiz, Hengxin Tan, Stephen D. Wilson, Binghai Yan, Turan Birol, and Girsh Blumberg, “Charge density wave order in kagome metal av3sb5 (a= cs, rb, k),” (2022), arXiv:2201.05188 [cond-mat.str-el] .
  * Chen  _et al._ (2021) Hui Chen, Haitao Yang, Bin Hu, Zhen Zhao, Jie Yuan, Yuqing Xing, Guojian Qian, Zihao Huang, Geng Li, Yuhan Ye, _et al._ , “Roton pair density wave and unconventional strong-coupling superconductivity in a topological kagome metal,” arXiv preprint arXiv:2103.09188 (2021).
  * Ye  _et al._ (2018) Linda Ye, Mingu Kang, Junwei Liu, Felix Von Cube, Christina R Wicker, Takehito Suzuki, Chris Jozwiak, Aaron Bostwick, Eli Rotenberg, David C Bell, _et al._ , “Massive dirac fermions in a ferromagnetic kagome metal,” Nature 555, 638–642 (2018).
  * Liu  _et al._ (2020) Zhonghao Liu, Man Li, Qi Wang, Guangwei Wang, Chenhaoping Wen, Kun Jiang, Xiangle Lu, Shichao Yan, Yaobo Huang, Dawei Shen, _et al._ , “Orbital-selective dirac fermions and extremely flat bands in frustrated kagome-lattice metal cosn,” Nature communications 11, 1–7 (2020).
  * Lin  _et al._ (2018) Zhiyong Lin, Jin-Ho Choi, Qiang Zhang, Wei Qin, Seho Yi, Pengdong Wang, Lin Li, Yifan Wang, Hui Zhang, Zhe Sun, _et al._ , “Flatbands and emergent ferromagnetic ordering in fe 3 sn 2 kagome lattices,” Physical review letters 121, 096401 (2018).
  * Liu  _et al._ (2018) Enke Liu, Yan Sun, Nitesh Kumar, Lukas Muechler, Aili Sun, Lin Jiao, Shuo-Ying Yang, Defa Liu, Aiji Liang, Qiunan Xu, _et al._ , “Giant anomalous hall effect in a ferromagnetic kagome-lattice semimetal,” Nature physics 14, 1125–1131 (2018).
  * Yao  _et al._ (2018) M Yao, H Lee, N Xu, Y Wang, J Ma, OV Yazyev, Y Xiong, M Shi, G Aeppli, and Y Soh, “Switchable Weyl nodes in topological Kagome ferromagnet Fe3Sn2,” arXiv:1810.01514 (2018).
  * Kang  _et al._ (2020a) Mingu Kang, Linda Ye, Shiang Fang, Jhih-Shih You, Abe Levitan, Minyong Han, Jorge I Facio, Chris Jozwiak, Aaron Bostwick, Eli Rotenberg, _et al._ , “Dirac fermions and flat bands in the ideal kagome metal fesn,” Nature materials 19, 163–169 (2020a).
  * Kang  _et al._ (2020b) Mingu Kang, Shiang Fang, Linda Ye, Hoi Chun Po, Jonathan Denlinger, Chris Jozwiak, Aaron Bostwick, Eli Rotenberg, Efthimios Kaxiras, Joseph G Checkelsky, _et al._ , “Topological flat bands in frustrated kagome lattice cosn,” Nature communications 11, 1–9 (2020b).
  * Watanabe and Kunitomi (1966) Hiroshi Watanabe and Nobuhiko Kunitomi, “On the neutron diffraction study of fege,” Journal of the Physical Society of Japan 21, 1932–1935 (1966).
  * Yin  _et al._ (2022) Jia-Xin Yin, Yu-Xiao Jiang, Xiaokun Teng, Shafayat Hossain, Sougata Mardanya, Tay-Rong Chang, Zijin Ye, Gang Xu, Michael Denner, Titus Neupert, Benjamin Lienhard, Han-Bin Deng, Chandan Setty, Qimiao Si, Guoqing Chang, Zurab Guguchia, Bin Gao, Nana Shumiya, Qi Zhang, Tyler Cochran, Daniel Multer, Maksim Litskevich, Zi-Jia Cheng, Xian Yang, Ming Yi, Pengcheng Dai, and Zahid Hasan, “Microscopic evidence for topological charge order in kagome magnet fege,” Submitted (2022).
  * et al (2022) M. Yi et al, in preparation (2022).
  * Setty  _et al._ (2021) Chandan Setty, Haoyu Hu, Lei Chen, and Qimiao Si, “Electron correlations and $t$-breaking density wave order in a $\mathbb{Z}_{2}$ kagome metal,” (2021), arXiv:2105.15204 [cond-mat.str-el] .
  * Soluyanov and Vanderbilt (2011) Alexey A Soluyanov and David Vanderbilt, “Wannier representation of z 2 topological insulators,” Physical Review B 83, 035108 (2011).
  * Soluyanov and Vanderbilt (2012) Alexey A Soluyanov and David Vanderbilt, “Smooth gauge for topological insulators,” Physical Review B 85, 115415 (2012).
  * Marzari  _et al._ (2012) Nicola Marzari, Arash A Mostofi, Jonathan R Yates, Ivo Souza, and David Vanderbilt, “Maximally localized wannier functions: Theory and applications,” Reviews of Modern Physics 84, 1419 (2012).
  * Kuneš  _et al._ (2010) Jan Kuneš, Ryotaro Arita, Philipp Wissgott, Alessandro Toschi, Hiroaki Ikeda, and Karsten Held, “Wien2wannier: From linearized augmented plane waves to maximally localized wannier functions,” Computer Physics Communications 181, 1888–1895 (2010).
  * Haule  _et al._ (2010) Kristjan Haule, Chuck-Hou Yee, and Kyoo Kim, “Dynamical mean-field theory within the full-potential methods: Electronic structure of ceirin 5, cecoin 5, and cerhin 5,” Physical Review B 81, 195107 (2010).
  * Blaha  _et al._ (2020) Peter Blaha, Karlheinz Schwarz, Fabien Tran, Robert Laskowski, Georg KH Madsen, and Laurence D Marks, “Wien2k: An apw+ lo program for calculating the properties of solids,” The Journal of Chemical Physics 152, 074101 (2020).
  * (37) A key point to note, however, is that despite each pair of bands having a net chern number zero, there is no guarantee that they form a physical elementary band representation (EBR). This is because despite the existence of exponentially localizable wannier functions for pairs with $C=\pm 1$, the wannier centers are off-site and break T-symmetry. This lack of adiabatic connectivity to the atomic limit renders the bands topological. .

##  I Supplemental Material

In this Supplemental Material, we further expand on the computational aspects
of the electronic structure and edge state analysis.

Role of Coulomb interactions: Fig. 5 shows the DMFT DOS (top row) and spectral
intensities (middle and bottom rows) as a function of increasing coulomb
interaction $U$ (columns from left to right correspond to $U=4eV,5eV,6eV$
respectively). Many of the important low energy features such as location of
the flat bands near Ef and Dirac cones remain intact with increasing
correlations both with (bottom row) and without magnetic order (middle row).
However, both the flat bands and Dirac cones at $M$ become narrower and more
localized as is natural when the electrons become strongly correlated. The
spectral intensities are also suppressed indicating loss of quasiparticle-like
behavior.

Figure 5:  Top row: DOS calculated from DMFT in the NM and magnetic phases.
The plots are over a larger energy window than shown in the main text. (Middle
row) DMFT bands with spectral intensities in the NM phase with SOC. (Bottom
row) DMFT bands with spectral intensities in the magnetic phase with SOC. The
columns from left to right correspond to $U=4eV,5eV,6eV$ respectively.

Role of vHove singularities: Fig. 6 (left) marks different high intensity
peaks and satellites features in the DOS and their relationship to various
properties of the electronic structure. Fig. 6 (right) shows the same plot
zoomed into a narrower energy window. Like in the main text, the short black
arrows indicate peaks in the DOS that are determined by the flat bands. The
long green arrow highlights the contribution from the vHove singularity. From
a comparison of the relative intensities, it is evident that the role of vHove
singularities is obscured by the more dominant flat band effects. This is
unlike the 135 ternary kagome compounds where the flat bands are buried deep
below the fermi energy and vHave singularities have a greater influence on the
low energy properties.

Figure 6:  Flat band vs vHove singularity contribution to the DOS in the NM
and magnetic phases as calculated in DMFT. The black short arrows denote the
dominant flat bands while the green long arrow denotes the v-Hove
contribution. The right panel is a zoomed in version of the left for clarity.

Figure 7:  Edge states in the charge order phase when the magnetic order is
strong enough to fully gap out the bulk. The color scale marks the spectral
weight of the states. In the left panel, the charge order parameter is
comparable to the parent magnetic order. In the right panel, the charge order
parameter is much smaller than the magnetic order parameter.

Large magnetic order: Here we consider the case where magnetic order
completely polarizes the chern bands and gaps out the bulk at half filling.
Fig. 7 (left) shows the edge states when the charge order parameter is
comparable to the parent magnetic order. Edge states persist at both quarter
filling and half filling. A similar scenario follows when the charge order
parameter is much smaller than the magnetic order parameter (Fig. 7 (right)).
However, note that in this scenario, the bulk state contribution to the STM
DOS is non-existent at half filling. Whereas at quarter filling, there are
both bulk and edge state contribution to the DOS.

Projected Hamiltonian in the Wannier basis: In this section, we derive the
effective model in the Wannier basis. As shown in Ref. Setty  _et al._ (2021),
the lowest two bands of the extended Hubbard model with Coulomb interaction on
the kagome lattice can be projected into an effective Hamiltonian defined on
the WOs. The projected creation and annihilation operators in the kagome
lattice is connected with WOs with a sequence of unitary transformations:

$Pc_{{\bf k}}P=\sum_{\mu}(V_{{\bf k}}^{\dagger}U_{H,{\bf k}})_{\mu\nu}b_{{\bf
k}\nu},$ (7)

where $V_{{\bf k}}$ diagonalizes the hopping matrix, and $U_{H,{\bf
k}}^{\dagger}$ is the Haldane matrix Soluyanov and Vanderbilt (2012). The
kinetic part of effective Hamiltonian reads:

$H_{0}=\sum_{ij,\mu\nu}b^{\dagger}_{i\mu}t_{ij}^{\mu\nu}b_{j\nu},$ (8)

where

$t^{\mu\nu}_{ij}=\sum_{\mu^{\prime}\nu^{\prime}i^{\prime}j^{\prime}}T^{\mu^{\prime}\nu^{\prime}}_{i^{\prime}j^{\prime}}\omega^{*}_{\mu^{\prime}\mu,i^{\prime}-i}\omega_{\nu^{\prime}\nu,j^{\prime}-j},$
(9)

with $\omega_{\alpha,\nu,i}=\frac{1}{N}\left(V_{{\bf k}}^{\dagger}U_{H,{\bf
k}}^{\dagger}\right)_{\mu\nu}e^{i{\bf k}\cdot{\bf r_{i}}}$ the real space wave
functions of the WOs.

The hopping parameters and the Zeeman field of the original kagome lattice
are: $t_{1}=1$, $t_{2}=-0.3$, $\lambda_{1}=0.28$, $\lambda_{2}=0.2$ and
$h=0.02$. We truncate the hoppings to the third nearest neighbor which are
listed in Table. 1, where $\mu/\nu$ denote the orbitals and $i/j$ denote the
sites. To fix the particle number to be half-filled, we introduce the chemical
potential $\mu=0.067$. Fig. 8 shows a comparison between the bands of the
original kagome lattice and the effective model.

(m, n) | (0, 1) | (1,0) | (1,1) | (m, n) | (0, 1) | (1,0) | (1,1)
---|---|---|---|---|---|---|---
$t^{00}_{m{\bf n_{1}}+n{\bf n_{2}}}$ | -0.0007+$i$0.00319 | -0.0007+$i$0.00319 | 0.0007-$i$0.00319 | $t^{11}_{m{\bf n_{1}}+n{\bf n_{2}}}$ | 0.0007-$i$0.00319 | 0.0007-$i$0.00319 | -0.0007+$i$0.00319
(m, n) | (2, 1) | (1,2) | (1,-1) | (m, n) | (2, 1) | (1,2) | (1,-1)
$t^{00}_{m{\bf n_{1}}+n{\bf n_{2}}}$ | -0.003 | -0.003 | -0.003 | $t^{11}_{m{\bf n_{1}}+n{\bf n_{2}}}$ | -0.003 | -0.003 | -0.003
(m, n) | (0, 2) | (2,0) | (2, 2) | (m, n) | (0, 2) | (2,0) | (2, 2)
$t^{00}_{m{\bf n_{1}}+n{\bf n_{2}}}$ | -0.0019-$i$0.00046 | -0.0019-$i$0.00046 | 0.0019+$i$0.00046 | $t^{11}_{m{\bf n_{1}}+n{\bf n_{2}}}$ | 0.0019+$i$0.00046 | 0.0019+$i$0.00046 | -0.0019-$i$0.00046
(m, n) | (0, 0) | (0,1) | (-1, 0) | (m, n) | (0, 0) | (1,0) | (0, -1)
$t^{01}_{m{\bf n_{1}}+n{\bf n_{2}}}$ | 0.01 | 0.01 | 0.01 | $t^{10}_{m{\bf n_{1}}+n{\bf n_{2}}}$ | 0.01 | 0.01 | 0.01
(m, n) | (1, -1) | (1,1) | (-1, -1) | (m, n) | (-1, 1) | (-1,-1) | (1, 1)
$t^{01}_{m{\bf n_{1}}+n{\bf n_{2}}}$ | -0.0038 | -0.0038 | -0.0038 | $t^{10}_{m{\bf n_{1}}+n{\bf n_{2}}}$ | -0.0038 | -0.0038 | -0.0038
Table 1: Table of hopping parameters of the effective Hamiltonian Eq. 3
appearing in the main text.  Figure 8:  Comparison of dispersions along high
symmetry lines of the lowest degenerate doublet of kagome bands. Dashed line
is the original kagome band and solid line is from the effective model with
WOs.

Ginzburg-Landau (GL) free energy: The GL free energy to fourth order in
perturbation is given by

$\displaystyle F[\Delta_{\textbf{Q}_{i}}]$ $\displaystyle=$
$\displaystyle\alpha\sum_{i}|\Delta_{\textbf{Q}_{i}}|^{2}+\beta_{1}\sum_{i}|\Delta_{\textbf{Q}_{i}}|^{4}$
(10) $\displaystyle+$
$\displaystyle\beta_{2}\sum_{i<j}|\Delta_{\textbf{Q}_{i}}|^{2}|\Delta_{\textbf{Q}_{j}}|^{2}-h^{\prime}\gamma\sum_{i}\Delta_{\textbf{Q}_{i}}\Delta_{-\textbf{Q}_{i}}$
$\displaystyle+$
$\displaystyle\alpha^{\prime}\sum_{i}|\Delta^{\prime}_{\textbf{Q}_{i}}|^{2}+\beta_{1}^{\prime}\sum_{i}|\Delta^{\prime}_{\textbf{Q}_{i}}|^{4}$
$\displaystyle+$
$\displaystyle\beta_{2}^{\prime}\sum_{i<j}|\Delta^{\prime}_{\textbf{Q}_{i}}|^{2}|\Delta^{\prime}_{\textbf{Q}_{j}}|^{2}-h^{\prime}\gamma\sum_{i}\Delta^{\prime}_{\textbf{Q}_{i}}\Delta^{\prime}_{-\textbf{Q}_{i}}$
$\displaystyle+$ $\displaystyle...\quad.$

Here the parameters
$\alpha,\alpha^{\prime},\beta_{i},\beta_{i}^{\prime},\gamma$ are constants
that depend on details of the electronic structure. The expression for
$\gamma$ in the flat band limit is given as

$\displaystyle\gamma$ $\displaystyle=$
$\displaystyle\sum_{k_{n}}\frac{6h_{0}}{(h_{0}^{2}+k_{n}^{2})^{2}}.$ (11)

where $k_{n}$ is the Fermionic Matsubara frequency and $h_{0}$ is the static
external field. The Matsubara sum can be evaluated to give

$\displaystyle\gamma$ $\displaystyle\simeq$
$\displaystyle\frac{3h_{0}}{2T_{CO}^{4}}\left[\frac{-1+\tilde{h}_{0}^{-1}\sinh\tilde{h}_{0}}{\tilde{h}_{0}^{2}(1+\cosh\tilde{h}_{0})}\right]$
(12) $\displaystyle\tilde{h}_{0}$ $\displaystyle\equiv$ $\displaystyle
h_{0}/T_{CO}.$ (13)

This expression for $\gamma$ is used in the main text of the Hamiltonian.

Figure 9:  Lowest order Feynman diagrams that couple the charge order
fluctuations $\Delta_{\textbf{Q}_{i}}$ and $\Delta_{\textbf{Q}_{i}}^{\prime}$
(zig-zag lines) to fluctuations of the broken time reversal field (wavy lines)
$h^{\prime}$. The solid lines are the electron Green functions in a static
external field $h_{0}$. These terms drive the enhanced magnetic moment in the
charge order phase.