# OVeNet: Offset Vector Network for Semantic Segmentation

Stamatis Alexandropoulos  National Technical University of Athens  Christos
Sakaridis  ETH Zürich  Petros Maragos  National Technical University of Athens

###### Abstract

Semantic segmentation is a fundamental task in visual scene understanding. We
focus on the supervised setting, where ground-truth semantic annotations are
available. Based on knowledge about the high regularity of real-world scenes,
we propose a method for improving class predictions by learning to selectively
exploit information from neighboring pixels. In particular, our method is
based on the prior that for each pixel, there is a seed pixel in its close
neighborhood sharing the same prediction with the former. Motivated by this
prior, we design a novel two-head network, named Offset Vector Network
(OVeNet), which generates both standard semantic predictions and a dense 2D
offset vector field indicating the offset from each pixel to the respective
seed pixel, which is used to compute an alternative, seed-based semantic
prediction. The two predictions are adaptively fused at each pixel using a
learnt dense confidence map for the predicted offset vector field. We
supervise offset vectors indirectly via optimizing the seed-based prediction
and via a novel loss on the confidence map. Compared to the baseline state-of-
the-art architectures HRNet and HRNet$+$OCR on which OVeNet is built, the
latter achieves significant performance gains on two prominent benchmarks for
semantic segmentation of driving scenes, namely Cityscapes and ACDC. Code is
available at https://github.com/stamatisalex/OVeNet.

##  1 Introduction

Semantic segmentation is one of the most central tasks in computer vision. In
particular, it is the task of assigning a class to every pixel in a given
image. It has lots of applications in a variety of fields, such as autonomous
driving [7, 32], robotics [71, 3], and medical image processing [64, 1], where
pixel-level labeling is critical.

(a) Image

(b) Semantic annotation

(c) HRNet [74] prediction

(d) Offset vectors of OVeNet

(e) OVeNet prediction

Figure 1: Real-world scenes have a high degree of regularity. We propose a
semantic segmentation method which can exploit this regularity, by learning to
selectively leverage information from neighboring _seed_ pixels. Our proposed
Offset Vector Network (OVeNet) can improve upon state-of-the-art architectures
[74] by estimating the offset vectors to such seed pixels and using them to
refine semantic predictions.

The adoption of convolutional neural networks (CNNs) [54] for semantic image
segmentation has led to a tremendous improvement in performance on
challenging, large-scale datasets, such as Cityscapes[19], MS COCO [49] and
ACDC [65]. Most of the related works [15, 74, 85, 58, 12] focus primarily on
architectural modifications of the employed networks in order to better
combine global context aggregation and local detail preservation, and use a
simple loss that is computed on individual pixels. The design of more
sophisticated losses [35, 86, 56] that take into account the structure which
is present in semantic labelings has received significantly less attention.
Many supervised techniques utilize a pixel-level loss function that handles
predictions for individual pixels independently of each other. By doing so,
they ignore the high regularity of real-world scenes, which can eventually
profit the final model’s performance by leveraging information from adjacent
pixels. Thus, these methods misclassify several pixels, primarily near
semantic boundaries, which leads to major losses in performance.

Based on knowledge about the high regularity of real scenes, we propose a
method for improving class predictions by learning to selectively exploit
information from neighboring pixels. In particular, the general nature of this
idea is applicable on SOTA models like HRNet [74] or HRNet $+$ OCR [86] and
can extend these models by adding a second head to them capable of achieving
this goal.

The architecture of our Offset Vector Network (OVeNet) is shown in Fig. 2. In
particular, following the base architecture of the backbone model (e.g HRNet
or HRNet $+$ OCR), the first head of our network outputs the initial pixel-
level semantic predictions. In general, two pixels $\mathbf{p}$ and
$\mathbf{q}$ that belong to the same category share the same semantic outcome.
If the pixels belong to the same class, using the label of $\mathbf{q}$ for
estimating class at the position of $\mathbf{p}$ results in a correct
prediction.

We leverage this property by learning to identify _seed pixels_ which belong
to the same class as the examined pixel, whenever such pixels exist, in order
to selectively use the prediction at the former for improving the prediction
at the latter. This idea is motivated by a prior which states for every pixel
$\mathbf{p}$ associated with a 2D semantically segmented image, there exists a
seed pixel $\mathbf{q}$ in the neighborhood of $\mathbf{p}$ which shares the
same prediction with the former. In order to predict classes with this scheme,
we need to find the regions where the prior is valid. Furthermore, in order to
point out the seed pixels in these regions, we must predict the offset vector
$\mathbf{o(p)}=\mathbf{q}-\mathbf{p}$ for each pixel $\mathbf{p}$.

As a result, we design a second head that generates a dense offset vector
field and a confidence map. The predicted offsets are used to resample the
class predictions from the first head and generate a second class prediction.
The outcomes from the two heads are fused adaptively using the learnt
confidence map as fusion weights, in order to down-weigh the offset-based
prediction and rely primarily on the basic class prediction in regions where
the prior is not valid. Thanks to using seed pixels for prediction, our
network classifies several pixels with incorrect initial predictions, e.g.,
boundary pixels, to the correct classes. Thus, it improves the shape as well
as the form of the corresponding segments, leading to more realistic results.
Last but not least, we propose a confidence loss which supervises the
confidence map explicitly and further improves performance. An illustrative
example of this concept is depicted in Fig. 1, where OVeNet outperforms the
baseline HRNet [74] model, since it enlarges correctly the road and the car
segment (red and yellow frame correspondingly) and reduces the total number of
misclassified pixels.

We evaluate our method extensively on 2 primary benchmarks for semantic
segmentation of driving scenes: Cityscapes [19] and ACDC [65]. We implement
our offset vector branch both on HRNet [74] and HRNet$+$OCR [87]. Our approach
significantly improves the initial models’ output predictions by achieving
better mean and per-class results. We conduct a thorough qualitative and
quantitative experimental comparison to show the clear advantages of our
method over previous SOTA techniques.

##  2 Related Work

Semantic segmentation architectures. Fully convolutional networks [67, 66]
were the first models that re-architected and fine-tuned classification
networks to direct dense prediction of semantic segmentation. They generated
low-resolution representations by eliminating the fully-connected layers from
a classification network (e.g AlexNet [38], VGGNet [68] or GoogleNet [69]) and
then estimating coarse segmentation maps from those representations. To create
medium-resolution representations [11, 42, 12, 13, 85], fully convolutional
networks were expanded using dilated/atrous convolutions, which replaced a few
strided convolutions and their associated ones. Following, in order to restore
high-resolution representations from low-resolution representations an
upsample process was used. This process involved a subnetwork that was
symmetric to the downsample process (e.g VGGNet [68]), and included skipping
connections between mirrored layers to transform the pooling indices (e.g.
DeconvNet [58]). Other methods include duplicating feature maps, which is used
in architectures like U-Net [64] and Hourglass [57, 18, 82, 33, 80, 20, 6, 70]
or encoder-decoder architectures [2, 61]. Lastly, the process of asymmetric
upsampling [5, 16, 78, 48, 72, 60, 91, 30] has also been extensively
researched.

The models’ representations were then enhanced to include multi-scale
contextual information [92, 8, 9]. PSPNet [93] utilized regular convolutions
on pyramid pooling representations to capture context at multiple scales,
while the DeepLab series [13, 12] used parallel dilated convolutions with
different dilation rates to capture context from different scales. Recent
research [27, 81, 40] proposed extensions, such as DenseASPP [81], which
increased the density of dilated rates to cover larger scale ranges. Other
studies [44, 15, 25] used encoder-decoder structures to exploit the multi-
resolution features as the multi-scale context. Here belongs the HRNet [74],
the baseline model of our method. HRNet connects high-to-low convolution
streams in parallel. It ensures that high-resolution representations are
maintained throughout the entire process and creates dependable high-
resolution representations with accurate positional information by repeatedly
merging the representations from various resolution streams. Applying
additionally the OCR [86] method, HRNet $+$ OCR is one of the leading models
in the task of semantic segmentation.

Lately, transformers have been successful in computer vision tasks
demonstrating their effectiveness. ViT [22] was the first attempt to use the
vanilla transformer architecture [73] for image classification without
extensive modification. Unlike later methods, such as PVT [76] and Swin [53],
that incorporated vision-specific inductive biases into their architectures,
the plain ViT suffers inferior performance on dense predictions due to weak
prior assumptions. To tackle this problem, the ViT-Adapter [17] was
introduced, which allowed plain ViT to achieve comparable performance to
vision-specific transformers and achieves the SOTA performance on this task.

Semantic segmentation loss functions. Image segmentation has highly correlated
outputs among the pixels. Converting pixel labeling problem into an
independent problem can lead to problems such as producing results that are
spatially inconsistent and have unwanted artifacts, making pixel-level
classification unnecessarily challenging. To solve this problem, several
techniques [37, 14, 95, 52] have been developed, such as integrating
structural information into segmentation. For instance, Chen et al. [12]
utilized denseCRF [37] for refining the final segmentation result. Following,
Zheng et al. [95] and Liu et al. [52] made the CRF module differentiable
within the deep neural network. Other methods that have been used to encode
structures include pairwise low-level image cues like grouping affinity [55,
51] and contour cues [4, 10]. GANs [63] are another alternative for imposing
structural regularity in the neural network output. However, these methods may
not work well in cases where there are changes in visual appearance or may
require expensive iterative inference procedures. Thus, Ke et. al. [35]
introduced AAFs, which are easier to train than GANs and more efficient than
CRF without run-time inference. There has been also proposed another loss
function [56] suitable for in real time applications that pulls the spatial
embeddings of pixels belonging to the same instance together.

Offset Vector-Based methods are essential for image analysis tasks that
involve adjacent pixels. In particular, they can effectively exploit the
information contained in neighbouring pixels, handle image distortions and
noise, and improve the accuracy of various image analysis tasks. They can be
used in applications such as depth estimation [59] or semantic segmentation
[88, 56]. Based on knowledge about the high regularity of real 3D scenes,
P3Depth [59] is a method used for 3D depth estimation learns to selectively
leverage information from coplanar pixels to improve the predicted depth. In
semantic segmentation, SegFix [88] is a model-agnostic post-processing scheme
that improved the boundary quality for the segmentation result. Motivated by
the empirical observation that the label predictions of interior pixels are
more reliable, SegFix replaced the originally unreliable predictions of
boundary pixels by the predictions of interior pixels. OVeNet provides another
perspective to use offset vectors and structure modeling by matching the
relations between neighbouring pixels in the label space. Although our
approach is inspired by the P3Depth idea, we focus on a different task in the
2D world which is semantic segmentation. Moreover, the key difference between
our method and SegFix is the timing of when they are applied. Essentially,
OVeNet integrates the offset vector learning process into the model training,
while SegFix applies the offset correction as a separate post-processing step.

##  3 Method

In this section, we will analyze our method shown in Fig. 2. Firstly, in Sec.
3.1 we give some basic notation and terminology of semantic segmentation. As
we mentioned before in Sec. 1, our network estimates semantic labels by
selectively combining information from each pixel and its corresponding seed
pixel. The intuition and the advantages of using seed pixels to improve the
initial prediction of a model are described analytically in Sec. 3.2. Lastly,
in Sec. 3.3 we introduce an additional confidence loss, which further enhances
our method.

Figure 2: Overview of OVeNet. OVeNet is a two-headed network. The first head
outputs semantic logits ($\mathbf{C}$), while the second head outputs a dense
offset vector field ($\mathbf{o}$) identifying positions of seed pixels along
with a confidence map ($F$). The logits are passed through a softmax function
and output the initial class prediction ($S_{i}$) of the model. Then, the
offset vectors are used to resample the logits from the first head and
generate a second class prediction ($S_{s}$). The two predictions are
adaptively fused using the confidence map resulting in the final prediction
$S_{f}$. For the visualization of the offset vectors we use the optical flow
color coding from [29]. Smaller vectors are lighter and color represents the
direction.

###  3.1 Terminology

Semantic segmentation requires learning a dense mapping
$f_{\theta}:I(u,v)\rightarrow S(u,v)$ where $I$ is the input image with
spatial dimensions $H\times W$, $S$ is the corresponding output prediction map
of the same resolution, $(u,v)$ are pixel coordinates in the image space and
$\theta$ are the parameters of the mapping $f$. In supervised semantic
segmentation, a ground-truth semantically segmented map $H$ is available for
each image during training. The aim is to optimize the function parameters
$\theta$ such that the predicted output map is as close as possible to the
ground-truth map across the entire training set $T$. This can be achieved by
minimizing the difference between the predicted and ground-truth images:

$\min_{\theta}\sum_{(I,H)\epsilon T}\mathcal{L}(f_{\theta}(I),H)$ (1)

where $\mathcal{L}$ is a loss function that penalizes variations between the
prediction and the ground truth.

###  3.2 Seed Pixel Identification

Let us assume we have one pixel $\mathbf{p}$ which belongs to segment of a
semantically segmented image. By definition, every other pixel on this segment
has the same class value. Thus, ideally, in order to get all of the class
values accurate, the network only has to predict the class at one of these
pixels, $\mathbf{q}$. This pixel can be interpreted as the seed pixel that
describes the segment-class. Finally, we let the network find this seed pixel
and the corresponding region.

Let us define a prior which is a relaxed version of the previous idea.

###### Definition 1

For every pixel $\mathbf{p}$ associated with a 2D semantically segmented
image, there exists a seed pixel $\mathbf{q}$ in the neighborhood of
$\mathbf{p}$ which shares the same prediction with the former.

In general, there may be numerous seed pixels for $\mathbf{p}$ or none at all.
Given that the Definition 1 holds, semantic segmentation task for $\mathbf{p}$
can be solved by identifying $\mathbf{q}$. For this reason, we let our network
predict the offset vector $\mathbf{o(p)}=\mathbf{q}-\mathbf{p}$. Thus, we
design our model so that it features a second, offset head and let this offset
head predict a dense offset vector field $\mathbf{o}(u,v)$. The two heads of
the network share a common main body and then they follow different paths. We
resample the initial logits $\mathbf{C}$, being predicted by the first head,
using the estimated offset vector field via:

$\mathbf{C_{s}(p)=C(p+o(p))}$ (2)

To manage fractional offsets, bilinear interpolation is used. The resampled
logits are then used to compute a second semantic segmentation prediction:

$\displaystyle S_{s}(u,v)=h(\mathbf{C_{s}}(u,v),u,v)$ $\displaystyle\implies
S_{s}(\mathbf{p})=S_{i}(\mathbf{p+o(p)})$ (3)

based on the seed locations. In our experiment, $h=softmax$.

Due to the fact that the prior is not always correct, the initial semantic
prediction $S_{i}$ may be preferred to the seed-based prediction $S_{s}$. To
account for such cases, the second head additionally predicts a confidence map
$F(u,v)\epsilon[0,1]$, which represents the model’s confidence in adopting the
predicted seed pixels for semantic segmentation via $S_{s}$. By adaptively
fusing $S_{i}$ and $S_{s}$, the confidence map is used to compute the final
prediction:

$S_{f}(\mathbf{p})=(1-F(\mathbf{p}))S_{i}(\mathbf{p})+F(\mathbf{p})S_{s}(\mathbf{p})$
(4)

We apply supervision to each of $S_{f}$, $S_{s}$, and $S_{i}$ in our model, by
optimizing the following loss:

$\mathcal{L}_{\text{semantic}}=\mathcal{L}(S_{f},H)+\kappa\mathcal{L}(S_{s},H)+\lambda\mathcal{L}(S_{i},H)$
(5)

with $\kappa$ and $\lambda$ being hyperparameters and H denotes the GT train
ids of each class for each pixel. In this way, we encourage the initial
model’s head to output an accurate representation across all pixels, even when
they have a high confidence value, and the offset vector head to learn high
confidence values for pixels for which the definition 1 holds and low
confidence values for pixels for which the prior does not.

This formulation, however, comes with a drawback. The model is not supervised
directly on the offsets. In fact, it could just predict zero offsets
everywhere and yet give valid $S_{s}$ and $S_{f}$ predictions that are equal
to $S_{i}$. Since the initial predictions $S_{i}$ are erroneously smoothed
around semantic boundaries due to the regularity of the mapping $f_{\theta}$
in the case of neural networks, this undesirable behavior is avoided in
practice. As a result, predicting a non-zero offset that points away from the
boundary provides a lower value for $\mathcal{L}_{\text{semantic}}$ for pixels
on either side of the boundary. This happens due to the fact that such an
offset utilizes a seed pixel for $S_{s}$ that is further away from the border
and suffers from reduced inaccuracy due to smoothing. These non-zero offsets
are also transmitted from the boundaries to the inner sections of areas with
smooth segments, helping the network in predicting non-trivial offsets due to
the regularity of the mapping that forms the offset vector field.

###  3.3 Confidence-Based Loss

Our confidence loss is based on the concept that given a pixel coordinate, its
surrounding pixels should be in the same segment. For each pixel $\mathbf{p}$,
we define the confidence loss as follows:

$\displaystyle\mathcal{L}_{f}(\mathbf{p})$
$\displaystyle=-\mathbbm{1}_{H(\mathbf{p})=H(\mathbf{p+o(p)})}\log
F(\mathbf{p})$ $\displaystyle\quad-\mathbbm{1}_{H(\mathbf{p})\neq
H(\mathbf{p+o(p)})}\log(1-F(\mathbf{p}))$ (6)

This idea is motivated by the fact that confidence should have large value
,for those pixels whose offset vector points to seed pixels with the same
class. Similarly, confidence should have small value, for those pixels whose
offset vector points to seed pixels with different class.

To sum up, the complete loss is:

$\mathcal{L}_{final}=\mathcal{L}_{\text{semantic}}+\mathcal{L}_{f}$ (7)

##  4 Experiments

To evaluate the proposed method, we carry out comprehensive experiments on
Cityscapes and ACDC dataset. Experimental results demonstrate that our method
achieved the highest performance, outperforming all other baselines.
Following, we first introduce the datasets, evaluation metrics and
implementation details in Sec. 4.1. We then compare our method to SOTA
approaches in Sec. 4.2. Finally, we perform a series of ablation experiments
on Cityscapes dataset in Sec. 4.3.

###  4.1 Experimental Setup

In this section, we present Cityscapes and ACDC datasets, which are used to
evaluate our approach. Evaluation on these datasets is performed using
standard semantic segmentation metrics explained below.

Cityscapes [19] is a challenging dataset, tasked for urban scene
understanding. It includes semantic, instance-wise, and dense pixel
annotations for 30 classes divided into eight groups (flat surfaces, humans,
vehicles, constructions, objects, nature, sky, and void). From these 30
classes only 19 classes are used for parsing evaluation. Around 5000 high
quality pixel-level finely annotated images and 20000 coarsely annotated
images make up the collection. During multiple months, daytimes, and ideal
weather circumstances, data was collected in 50 cities. The finely annotated
5000 images are divided into 2.975, 500, 1.525 images for training, validation
and testing respectively.

ACDC [65] is a demanding dataset, used for training and testing semantic
segmentation methods on adverse visual conditions. It consists of a big
collection of 4006 images divided evenly between four frequent unfavorable
conditions: fog, dark, rain, and snow. It consists of 19 semantic classes,
coinciding exactly with the evaluation classes of the Cityscapes dataset. ACDC
is manually split into four sets corresponding to the examined conditions:
1000 foggy, 1006 nighttime, 1000 rainy and 1000 snowy images. Each set of each
adverse condition is split into 400 training, 100 validation and 500 test
images, except the nighttime set with 106 validation images. This results in a
total of 1600 training and 406 validation images with public annotations and
2000 test images with annotations withheld for benchmarking purposes.

Evaluation Metrics. The mean of class-wise intersection over union (mIoU) is
adopted as the evaluation metric. In addition to the mean of class-wise
intersection over union (mIoU), we report other three scores on the test set:
IoU category (cat.), iIoU class (cla.) and iIoU category (cat.)

Implementation Details. Our network consists of two heads. The first head
outputs 19 channels, one for each class. The second head outputs three
channels: one for each coordinate of the offset vectors and one for
confidence. These two heads follow the structure of HRNet. Both OVeNet and the
baseline HRNet are initialized with pre-trained ImageNet [38] weights. This
initialization is important to achieve competitive results as in [74].
Following the same training protocol as in [74], the data are augmented by
random cropping (from 1024 × 2048 to 512 × 1024 in Cityscapes and from 1080 x
1920 to 540 x 960 in ACDC), random scaling in the range of [0.5, 2], and
random horizontal flipping.We use the SGD optimizer with the base learning
rate of 0.01, the momentum of 0.9 and the weight decay of 0.0005. The number
of epochs used for training is 484. For lowering the learning rate, a poly
learning rate policy with a power of 0.9 is applied. The offset vectors are
restricted via a tanh layer to have a maximum length of $\tau$ in normalized
image coordinates. After an ablation study shown in Sec. 4.3, we set $\tau$ to
0.5 by default and the branch will occur on the last ($4^{th}$) stage of
HRNet. The confidence map is predicted through a sigmoid layer. For $S_{i}$,
$S_{s}$ and $S_{f}$ predictions, Ohem Cross Entropy Loss is used. It is
remarkable that, at first, the model computed the $S_{i}$, $S_{s}$, $S_{f}$
predictions directly but that led to low mIoU results. On the other hand, the
final model with the best results outputted the $S_{i}$, $S_{s}$, $S_{f}$
logits. The loss weights $\lambda$ and $\mu$ are set to 0.5. In addition, the
confidence based loss is applied using the final semantic prediction $S_{f}$.

###  4.2 Comparison with the State of the Art

Cityscapes. The results on Cityscapes which is the major database focusing on
semantic understanding of urban street scenes are shown below. We achieve
better results on Cityscapes than both the initial HRNet and HRNet $+$ OCR
model under similar training time, outperforming prior SOTA methods. Table 1
compares our method with SOTA methods on the Cityscapes test set. All the
results are with six scales and flipping. Two cases w/o using coarse data are
evaluated: One is about the model learned on the train set, and the other is
about the model learned on the train $+$ val set. In both cases our offset
vector - based model achieves the superior performance. To become more
specific, we achieve a superior relative performance gain of $1.4\%$ in mIoU,
$2.4\%$ in iIoU cla., $0.4\%$ in IoU cat. and $1.4\%$ in iIoU cat when the
model is learned only on train set and a gain of $0.5\%$ in mIoU when the
model is learned on both train $+$ val set. We can also observe that OVeNet
built on HRNet and learned only on the training set, outperforms initial
HRNet’s mIoU learned only on training set as well as both the training and
validation sets. Table 2 analytically compares our approach with HRNet’s per
class results. As we can see our method achieves better results in the
majority of classes. Our offset vector-based model learns an implicit
representation of different objects which can benefit the overall semantic
segmentation estimation capability of the network. Regarding val set results,
HRNet achieves $81.8\%$ mIoU while OVeNet built on it surpasses it reaching
$82.4\%$ mIoU .

Qualitative results on Cityscapes support the above findings, as shown in Fig.
3. To be more specific, from left to right, we depict the RBG input image, GT
image, CCNet’s [28], DANet’s [24], HRNet’s [87] and our model’s prediction .
Specifically, our model demonstrates successful classification of false
predicted pixels (identified by a red frame) in the first row of the image.
Furthermore, the model accurately eliminates the terrain segment that had been
previously incorrectly classified in the image (blue frame). In the second
row, OVeNet exhibits superior performance compared to previous models, as it
accurately expands the sidewalk and the pole depicted in the blue and yellow
frames, respectively. Furthermore, not only it successfully eliminates
discontinuities in the pink frame, but also results in a better representation
of the bicycle’s shape. As far as the third set of images is concerned, our
model reduces the false prediction made by HRNet in the sidewalk segment
achieving a better final result. Last but not least, in the last set of
images, our model effectively reduces the pole (pink frame) in the
segmentation output. Despite the presence of some errors that were not present
in the initial HRNet, our model significantly increases the number of pixels
that correctly belong to the ’bus’ category, resulting in a superior
prediction compared to the baseline. To sum up, OVeNet surpasses the HRNet’s
performance.

(a)

(b)

(c)  (d)

(e)

(f)

(g)

(h)

(i)  (j)

(k)

(l)

(m)

(n)

(o)  (p)

(q)

(r)

(s)

(t)

(u)  (v)

(w)

(x)

Figure 3:  Qualitative results of selected examples on Cityscapes.From left to
right: RGB Input Image, GT, CCNet [28], DANet [24], HRNet [74], OVeNet. Best
viewed on a screen and zoomed in. Table 1: Semantic segmentation results on
Cityscapes test set.. We compare our method against SOTA methods as in
[74].D-ResNet-101 = Dilated-ResNet-101. By default, OVeNet is built on HRNet,
unless stated otherwise.

| backbone | mIoU | iIoU cla. | IoU cat. | iIoU cat.
---|---|---|---|---|---
_Model learned on the train set_ |  |  |
PSPNet [92] |  D-ResNet-$101$ | $78.4$ | $56.7$ | $90.6$ | $78.6$
PSANet [94] |  D-ResNet-$101$ | $78.6$ | - | - | -
PAN [39] |  D-ResNet-$101$ | $78.6$ | - | - | -
AAF [34] |  D-ResNet-$101$ | $79.1$ | - | - | -
HRNet [74] |  HRNetV$2$-W$48$ | ${80.4}$ | ${59.2}$ | ${91.5}$ | ${80.8}$
OVeNet | HRNetV2-W48 | $\mathbf{81.8}$ | $\mathbf{61.6}$ | $\mathbf{91.9}$ | $\mathbf{82.2}$
_Model learned on the train+val set_ |  |  |
GridNet [23] | - | $69.5$ | $44.1$ | $87.9$ | $71.1$
LRR-4x [26] | - | $69.7$ | $48.0$ | $88.2$ | $74.7$
DeepLab [8] |  D-ResNet-$101$ | $70.4$ | $42.6$ | $86.4$ | $67.7$
LC [41] | - | $71.1$ | - | - | -
Piecewise [47] |  VGG-$16$ | $71.6$ | $51.7$ | $87.3$ | $74.1$
FRRN [62] | - | $71.8$ | $45.5$ | $88.9$ | $75.1$
RefineNet [46] |  ResNet-$101$ | $73.6$ | $47.2$ | $87.9$ | $70.6$
PEARL [31] |  D-ResNet-$101$ | $75.4$ | $51.6$ | $89.2$ | $75.1$
DSSPN [43] |  D-ResNet-$101$ | $76.6$ | $56.2$ | $89.6$ | $77.8$
LKM [60] |  ResNet-$152$ | $76.9$ | - | - | -
DUC-HDC [75] | - | $77.6$ | $53.6$ | $90.1$ | $75.2$
SAC [90] |  D-ResNet-$101$ | $78.1$ | - | - | -
DepthSeg [36] |  D-ResNet-$101$ | $78.2$ | - | - | -
ResNet38 [77] | WResNet-38 | $78.4$ | $59.1$ | $90.9$ | $78.1$
BiSeNet [83] |  ResNet-$101$ | $78.9$ | - | - | -
DFN [84] |  ResNet-$101$ | $79.3$ | - | - | -
PSANet [94] |  D-ResNet-$101$ | $80.1$ | - | - | -
PADNet [79] |  D-ResNet-$101$ | $80.3$ | $58.8$ | $90.8$ | $78.5$
CFNet [89] |  D-ResNet-$101$ | $79.6$ | - | - | -
Auto-DeepLab [50] | - | $80.4$ | - | - | -
DenseASPP [92] |  WDenseNet-$161$ | $80.6$ | $59.1$ | $90.9$ | $78.1$
SVCNet [21] |  ResNet-$101$ | $81.0$ | - | - | -
ANN [96] |  D-ResNet-$101$ | $81.3$ | - | - | -
CCNet [28] |  D-ResNet-$101$ | $81.4$ | - | - | -
DANet [24] |  D-ResNet-$101$ | $81.5$ | - | - | -
HRNet [74] |  HRNetV$2$-W$48$ | $81.6$ | $61.8$ | $92.1$ | $82.2$
HRNet + OCR [87] |  HRNetV$2$-W$48$ | $81.9$ | $\mathbf{62.0}$ | $\mathbf{92.0}$ | $\mathbf{82.5}$
OVeNet (_HRNet $+$ OCR_) | HRNetV2-W48 | $\mathbf{82.4}$ | 61.6 | 91.9 | $82.2$

Table 2: Per Class Results on Cityscapes test set

Method |  road |  sidew. |  build. |  wall |  fence |  pole |  light |  sign |  veget. |  terrain |  sky |  person |  rider |  car |  truck |  bus |  train |  motorc. |  bicycle | mIoU
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
HRNet [74] | $98.73$ | $\mathbf{87.49}$ | $93.65$ | $56.48$ | $61.57$ | $\mathbf{71.57}$ | $78.76$ | $81.81$ | $93.99$ | $74.11$ | $95.68$ | $87.95$ | $73.72$ | $96.35$ | $69.94$ | $82.52$ | $76.93$ | $70.88$ | $\mathbf{78.02}$ | $80.40$
OVeNet | $\mathbf{98.74}$ | $87.41$ | $\mathbf{93.79}$ | $\mathbf{61.65}$ | $\mathbf{64.00}$ | $71.35$ | $\mathbf{78.98}$ | $\mathbf{81.65}$ | $\mathbf{94.00}$ | $\mathbf{73.42}$ | $\mathbf{95.81}$ | $\mathbf{87.99}$ | $\mathbf{74.36}$ | $\mathbf{96.42}$ | $\mathbf{74.76}$ | $\mathbf{87.70}$ | $\mathbf{82.83}$ | $\mathbf{71.77}$ | $77.86$ | $\mathbf{81.82}$
HRNet $+$ OCR | $98.77$ | $\mathbf{87.85}$ | $93.72$ | $57.75$ | $63.92$ | $\mathbf{71.74}$ | $\mathbf{78.56}$ | $\mathbf{81.77}$ | $\mathbf{94.06}$ | $\mathbf{73.69}$ | $95.68$ | $88.04$ | $74.64$ | $96.46$ | $76.40$ | $88.78$ | $84.63$ | $71.79$ | $\mathbf{78.63}$ | $81.90$
OVeNet (_HRNet $+$ OCR_) | $\mathbf{98.79}$ | $87.47$ | $\mathbf{93.86}$ | $\mathbf{62.97}$ | $\mathbf{64.41}$ | $70.80$ | $78.45$ | $81.13$ | $93.99$ | $73.31$ | $\mathbf{95.72}$ | $\mathbf{88.08}$ | $\mathbf{74.90}$ | $\mathbf{96.47}$ | $\mathbf{76.95}$ | $\mathbf{89.95}$ | $\mathbf{88.48}$ | $\mathbf{71.87}$ | $78.43$ | $\mathbf{82.42}$

ACDC. The results on ACDC, which targets semantic understanding of driving
scenes in adverse visual conditions, are shown below. We also achieve far
better results than the initial HRNet under similar training time,
outperforming prior SOTA methods. Table 4 compares our approach with SOTA
models not only on all methods but also on different conditions of the ACDC
test set. OVeNet improves the initial HRNet model by $2.5\%$ mIoU in ”All”
conditions. Additionally, we can observe from the per class results shown on
Table 3 on different conditions, that our approach outperforms the HRNet’s one
in the vast majority of classes as well as in the total mIoU score.
Specifically, on foggy days, performance on classes containing small-instance
instances, such person, rider, and bicycle, is very low. This is likely
because of the combined effect of contrast reduction and poor resolution for
examples of these classes that are far from the camera. There is also a huge
performance boost of approximately $10\%$ and $15\%$ on the ”bus” and ”truck”
class respectively. Moreover, it is more difficult to separate classes at
night that are often dark or poorly lit, such as buildings, vegetation,
traffic signs, and the sky. This behaviour is observed also in offset vector
performance as they have small values when the visibility is limited. Lastly,
during night and snow conditions, road and sidewalk performance is at its
lowest, which can be attributed to misunderstanding between the two classes as
a result of their similar look. As for val set results, HRNet achieves
$75.5\%$ mIoU while our OVeNet surpasses it reaching $75.9\%$ mIoU.

Qualitative results on ACDC support the above findings, as shown in Fig. 4. To
be more specific, from up to down, we depict the input RBG image, the GT, the
initial HRNet’s output and our model’s output. In the first column, we can
underline that our model tries to enlarge correctly the sidewalk segments in
both red and green frames and reduces the erroneously terrain segment
predicted by the initial model. As far as the second set of images is
concerned, HRNet classifies incorrectly the sign of the house into the traffic
sign class (red frame). On the contrary, our model corrects not only this
mistake, but also a discontinuity occurred in the yellow frame. Last but not
least, regarding the last set of materials, our offset vector - based model
eliminates correctly the sidewalk area (red framework), as in the initial
image there is no such area. To sum up, OVeNet surpasses HRNet’s performance.

Table 3: Per Class Results on ACDC test set. OVeNet is built on HRNet.

Condition | Method |  road |  sidew. |  build. |  wall |  fence |  pole |  light |  sign |  veget. |  terrain |  sky |  person |  rider |  car |  truck |  bus |  train |  motorc. |  bicycle | mIoU
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Fog | HRNet | $95.3$ | $81.1$ | $89.7$ | $54.2$ | $48.1$ | $60.3$ | $76.2$ | $76.0$ | $89.1$ | $78.1$ | $98.3$ | $\mathbf{58.8}$ | $60.1$ | $83.3$ | $50.5$ | $47.9$ | $\mathbf{86.4}$ | $49.8$ | $36.3$ | $69.3$
OVeNet | $\mathbf{95.7}$ | $\mathbf{82.1}$ | $\mathbf{90.6}$ | $\mathbf{55.5}$ | $\mathbf{50.4}$ | $\mathbf{63.9}$ | $\mathbf{79.3}$ | $\mathbf{78.2}$ | $\mathbf{89.6}$ | $\mathbf{78.8}$ | $\mathbf{98.4}$ | $56.5$ | $\mathbf{63.8}$ | $\mathbf{85.7}$ | $\mathbf{60.3}$ | $\mathbf{63.8}$ | $85.5$ | $\mathbf{52.7}$ | $\mathbf{39.4}$ | $\mathbf{72.1}$
Night | HRNet | $95.4$ | $78.1$ | $86.8$ | $46.5$ | $41.6$ | $59.1$ | $65.8$ | $65.2$ | $77.2$ | $42.0$ | $86.2$ | $66.2$ | $40.1$ | $80.5$ | $\mathbf{16.6}$ | $31.0$ | $86.2$ | $37.1$ | $49.2$ | $60.6$
OVeNet | $\mathbf{95.6}$ | $\mathbf{79.4}$ | $\mathbf{87.4}$ | $\mathbf{48.5}$ | $\mathbf{42.3}$ | $\mathbf{61.1}$ | $\mathbf{68.2}$ | $\mathbf{69.9}$ | $\mathbf{78.1}$ | $\mathbf{43.8}$ | $\mathbf{86.4}$ | $\mathbf{68.6}$ | $\mathbf{44.4}$ | $\mathbf{82.0}$ | $10.9$ | $\mathbf{44.3}$ | $\mathbf{87.3}$ | $\mathbf{43.3}$ | $\mathbf{51.3}$ | $\mathbf{62.8}$
Rain | HRNet | $95.7$ | $83.9$ | $93.6$ | $60.2$ | $62.7$ | $70.1$ | $80.9$ | $79.7$ | $94.4$ | $51.9$ | $98.7$ | $72.4$ | $18.7$ | $92.5$ | $67.0$ | $85.3$ | $87.3$ | $50.9$ | $66.4$ | $74.5$
OVeNet | $\mathbf{96.4}$ | $\mathbf{86.5}$ | $\mathbf{94.3}$ | $\mathbf{65.2}$ | $\mathbf{64.6}$ | $\mathbf{72.6}$ | $\mathbf{83.6}$ | $\mathbf{82.2}$ | $\mathbf{94.6}$ | $\mathbf{54.3}$ | $\mathbf{98.7}$ | $\mathbf{76.4}$ | $\mathbf{21.8}$ | $\mathbf{93.4}$ | $\mathbf{75.7}$ | $\mathbf{88.9}$ | $\mathbf{89.1}$ | $\mathbf{50.9}$ | $\mathbf{66.4}$ | $\mathbf{76.6}$
Snow | HRNet | $95.0$ | $79.6$ | $91.4$ | $49.6$ | $58.2$ | $69.7$ | $86.6$ | $78.6$ | $92.9$ | $59.4$ | $97.9$ | $77.2$ | $24.5$ | $91.7$ | $53.4$ | $56.2$ | $90.1$ | $39.8$ | $66.8$ | $71.5$
OVeNet | $\mathbf{95.7}$ | $\mathbf{81.9}$ | $\mathbf{92.4}$ | $\mathbf{55.3}$ | $\mathbf{59.8}$ | $\mathbf{71.9}$ | $\mathbf{88.1}$ | $\mathbf{80.5}$ | $\mathbf{93.4}$ | $\mathbf{60.7}$ | $\mathbf{98.0}$ | $\mathbf{79.8}$ | $\mathbf{25.7}$ | $\mathbf{92.7}$ | $\mathbf{59.6}$ | $\mathbf{65.4}$ | $\mathbf{91.6}$ | $\mathbf{45.2}$ | $\mathbf{70.3}$ | $\mathbf{74.1}$
All | HRNet | $95.3$ | $80.3$ | $90.5$ | $52.0$ | $53.1$ | $65.1$ | $78.2$ | $74.2$ | $89.2$ | $68.4$ | $96.7$ | $70.6$ | $36.1$ | $88.2$ | $55.9$ | $54.3$ | $88.0$ | $43.8$ | $58.9$ | $70.5$
OVeNet | $\mathbf{95.8}$ | $\mathbf{82.1}$ | $\mathbf{91.3}$ | $\mathbf{55.8}$ | $\mathbf{54.6}$ | $\mathbf{67.6}$ | $\mathbf{80.5}$ | $\mathbf{77.3}$ | $\mathbf{89.7}$ | $\mathbf{69.5}$ | $\mathbf{96.8}$ | $\mathbf{73.4}$ | $\mathbf{39.1}$ | $\mathbf{89.5}$ | $\mathbf{61.9}$ | $\mathbf{65.0}$ | $\mathbf{89.4}$ | $\mathbf{47.2}$ | $\mathbf{60.6}$ | $\mathbf{73.0}$

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Figure 4:  Qualitative results of selected examples on ACDC. Up to down: RGB,
GT, HRNet [74], OVeNet. Best viewed on a screen and zoomed in.

###  4.3 Ablation study

In order to experimentally confirm our design choices for the offset vector -
based model, we performed an ablation study, as shown in Table 5. We trained
and evaluated 7 different variations on Cityscapes. The performance of each
model variation in relation to the ground truth images was calculated by means
of the mIoU ( as defined above). At first, we initialized both heads of the
network with the pre-trained Imagenet weights and set the offset vector length
equal to $0.5$. Secondly, we freezed both main body’s and initial head’s
weights. The freezed part of our model was initialized with the corresponding
Cityscapes final pre-trained weights. The only part trained was the second
head, which was initialized with pre-trained ImageNet weights. As shown in
Table 5, although the performance of our model is higher than the initial
single head model’s one, it still remains lower than the case were both heads
are trained simultaneously. Then, we deactivated the ”Freezed” feature and
changed the offset vector’s length logarithmically, setting both the values
$1$ and $0.2$. We observed that in both cases the performance is lower than
that of $0.5$. This is due the fact that larger offset vectors point to more
distant objects that may affect erroneously the final prediction, while
smaller vectors do not exploit too much information from neighbouring classes.
Furthermore, we deactivated the OHEM Cross Entropy Loss and enabled the simple
Cross Entropy Loss. As expected, the performance of the model was lower. OHEM
penalizes more high loss values and leads to a better training of the model.
Lastly, HRNet [74] consists of $4$ stages. In all the previous cases, the
branch occured on the last ($4^{th}$) stage so as not to overload the new
network with many extra parameters. When the branch occured on the $3^{rd}$
stage the performance did not became better.

Table 4: Comparison of the models on different conditions of ACDC. Method | Fog | Night | Rain | Snow | All
---|---|---|---|---|---
RefineNet [45] | $65.7$ | $55.5$ | $68.7$ | $65.9$ | $65.3$
DeepLabv2 [12] | $54.5$ | $45.3$ | $59.3$ | $57.1$ | $55.3$
DeepLabv3+ [15] | $69.1$ | $60.9$ | $74.1$ | $69.6$ | $70.0$
HRNet [74] | $69.3$ | $60.6$ | $74.5$ | $71.5$ | $70.5$
OVeNet | $\mathbf{72.1}$ | $\mathbf{62.8}$ | $\mathbf{76.6}$ | $\mathbf{74.1}$ | $\mathbf{73.0}$
Table 5: Ablation study of components of our method. ”Fr:” Freezed main body’s and initial head’s weights and initiliaze them with Cityscapes final pre-trained weights, ”Br:” Branch, ”$\boldsymbol{\tau}$”: Offset Vector Length, ”OHEM”: OHEM Cross Entropy. Fr | Br | $\boldsymbol{\tau}$ | OHEM | mIoU
---|---|---|---|---
|  |  | ✓ | $81.83$
| $4$ | $0.5$ | ✓ | $\mathbf{82.40}$
✓ | $4$ | $0.5$ | ✓ | $82.01$
| $4$ | $1$ | ✓ | $81.79$
| $4$ | $0.2$ | ✓ | $82.30$
| $4$ | $0.2$ |  | $81.96$
| $3$ | $0.2$ | ✓ | $82.20$

##  5 Conclusion

All in all, we have presented a supervised method for semantic segmentation
which selectively exploits information from neighboring pixels. OVeNet has
enhanced by far the initial model’s output predictions, since it has achieved
not only better total results but also better per class results in the
majority of classes under similar training time. It has learned an implicit
representation of different objects which benefits the overall semantic
segmentation estimation capability of the network. In fact, not only our idea
has achieved highly satisfactory results close to GT images, but also has
outperformed the predictions outputted by the initial model’s head.
Classifying some false predicted pixels in the correct classes, it has
eliminated discontinuities and improved the shape as well as the form of the
corresponding segments, leading to more realistic results. This is a strong
contribution which opens new pathways for real world applications, such as
self-driving cars or medical imaging and diagnostics.

## References

  * [1] Matthew Amodio, Feng Gao, Arman Avesta, Sanjay Aneja, Lucian V Del Priore, Jay Wang, and Smita Krishnaswamy.  Cuts: A fully unsupervised framework for medical image segmentation.  arXiv preprint arXiv:2209.11359, 2022.
  * [2] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12):2481–2495, 2017.
  * [3] Jonathan C Balloch, Varun Agrawal, Irfan Essa, and Sonia Chernova.  Unbiasing semantic segmentation for robot perception using synthetic data feature transfer.  arXiv preprint arXiv:1809.03676, 2018.
  * [4] Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani.  Semantic segmentation with boundary neural fields.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3602–3610, 2016.
  * [5] Adrian Bulat and Georgios Tzimiropoulos.  Human pose estimation via convolutional part heatmap regression.  In ECCV, pages 717–732, 2016.
  * [6] Adrian Bulat and Georgios Tzimiropoulos.  Binarized convolutional landmark localizers for human pose estimation and face alignment with limited resources.  In ICCV, pages 3726–3734, 2017.
  * [7] Senay Cakir, Marcel Gauß, Kai Häppeler, Yassine Ounajjar, Fabian Heinle, and Reiner Marchthaler.  Semantic segmentation for autonomous driving: Model evaluation, dataset generation, perspective comparison, and real-time capability.  arXiv preprint arXiv:2207.12939, 2022.
  * [8] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE Trans. Pattern Anal. Mach. Intell., 40(4):834–848, 2018\.
  * [9] Liang-Chieh Chen, Yi Yang, Jiang Wang, Wei Xu, and Alan L. Yuille.  Attention to scale: Scale-aware semantic image segmentation.  In CVPR, pages 3640–3649, 2016.
  * [10] Liang-Chieh Chen, Jonathan T Barron, George Papandreou, Kevin Murphy, and Alan L Yuille.  Semantic image segmentation with task-specific edge detection using cnns and a discriminatively trained domain transform.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4545–4554, 2016.
  * [11] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Semantic image segmentation with deep convolutional nets and fully connected crfs.  arXiv preprint arXiv:1412.7062, 2014.
  * [12] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848, 2017.
  * [13] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  arXiv preprint arXiv:1706.05587, 2017.
  * [14] Liang-Chieh Chen, Alexander Schwing, Alan Yuille, and Raquel Urtasun.  Learning deep structured models.  In International Conference on Machine Learning, pages 1785–1794. PMLR, 2015.
  * [15] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In Proceedings of the European conference on computer vision (ECCV), pages 801–818, 2018.
  * [16] Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun.  Cascaded pyramid network for multi-person pose estimation.  CoRR, abs/1711.07319, 2017.
  * [17] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao.  Vision transformer adapter for dense predictions.  arXiv preprint arXiv:2205.08534, 2022.
  * [18] Xiao Chu, Wei Yang, Wanli Ouyang, Cheng Ma, Alan L. Yuille, and Xiaogang Wang.  Multi-context attention for human pose estimation.  In CVPR, pages 5669–5678, 2017.
  * [19] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016.
  * [20] Jiankang Deng, George Trigeorgis, Yuxiang Zhou, and Stefanos Zafeiriou.  Joint multi-view face alignment in the wild.  CoRR, abs/1708.06023, 2017.
  * [21] Henghui Ding, Xudong Jiang, Bing Shuai, Ai Qun Liu, and Gang Wang.  Semantic correlation promoted shape-variant context for segmentation.  In CVPR, June 2019.
  * [22] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [23] Damien Fourure, Rémi Emonet, Élisa Fromont, Damien Muselet, Alain Trémeau, and Christian Wolf.  Residual conv-deconv grid network for semantic segmentation.  In BMVC, 2017.
  * [24] Jun Fu, Jing Liu, Haijie Tian, Zhiwei Fang, and Hanqing Lu.  Dual attention network for scene segmentation.  CoRR, abs/1809.02983, 2018.
  * [25] Jun Fu, Jing Liu, Yuhang Wang, Yong Li, Yongjun Bao, Jinhui Tang, and Hanqing Lu.  Adaptive context network for scene parsing.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6748–6757, 2019.
  * [26] Golnaz Ghiasi and Charless C. Fowlkes.  Laplacian pyramid reconstruction and refinement for semantic segmentation.  In ECCV, pages 519–534, 2016.
  * [27] Junjun He, Zhongying Deng, Lei Zhou, Yali Wang, and Yu Qiao.  Adaptive pyramid context network for semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7519–7528, 2019.
  * [28] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu.  Ccnet: Criss-cross attention for semantic segmentation.  CoRR, abs/1811.11721, 2018.
  * [29] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox.  Flownet 2.0: Evolution of optical flow estimation with deep networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2462–2470, 2017.
  * [30] Md. Amirul Islam, Mrigank Rochan, Neil D. B. Bruce, and Yang Wang.  Gated feedback refinement network for dense image labeling.  In CVPR, pages 4877–4885, 2017.
  * [31] Xiaojie Jin, Xin Li, Huaxin Xiao, Xiaohui Shen, Zhe Lin, Jimei Yang, Yunpeng Chen, Jian Dong, Luoqi Liu, Zequn Jie, Jiashi Feng, and Shuicheng Yan.  Video scene parsing with predictive feature learning.  In ICCV, pages 5581–5589, 2017.
  * [32] Çağrı Kaymak and Ayşegül Uçar.  Semantic image segmentation for autonomous driving using fully convolutional networks.  In 2019 International Artificial Intelligence and Data Processing Symposium (IDAP), pages 1–8, 2019.
  * [33] Lipeng Ke, Ming-Ching Chang, Honggang Qi, and Siwei Lyu.  Multi-scale structure-aware network for human pose estimation.  CoRR, abs/1803.09894, 2018.
  * [34] Tsung-Wei Ke, Jyh-Jing Hwang, Ziwei Liu, and Stella X. Yu.  Adaptive affinity fields for semantic segmentation.  In ECCV, pages 605–621, 2018.
  * [35] Tsung-Wei Ke, Jyh-Jing Hwang, Ziwei Liu, and Stella X Yu.  Adaptive affinity fields for semantic segmentation.  In Proceedings of the European conference on computer vision (ECCV), pages 587–602, 2018.
  * [36] Shu Kong and Charless C. Fowlkes.  Recurrent scene parsing with perspective understanding in the loop.  In CVPR, pages 956–965, 2018.
  * [37] Philipp Krähenbühl and Vladlen Koltun.  Efficient inference in fully connected crfs with gaussian edge potentials.  Advances in neural information processing systems, 24, 2011.
  * [38] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems, 25, 2012.
  * [39] Hanchao Li, Pengfei Xiong, Jie An, and Lingxue Wang.  Pyramid attention network for semantic segmentation.  In BMVC, page 285, 2018.
  * [40] Ke Li, Bharath Hariharan, and Jitendra Malik.  Iterative instance segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3659–3667, 2016.
  * [41] Xiaoxiao Li, Ziwei Liu, Ping Luo, Chen Change Loy, and Xiaoou Tang.  Not all pixels are equal: Difficulty-aware semantic segmentation via deep layer cascade.  In CVPR, pages 6459–6468, 2017.
  * [42] Zeming Li, Chao Peng, Gang Yu, Xiangyu Zhang, Yangdong Deng, and Jian Sun.  Detnet: Design backbone for object detection.  In Proceedings of the European conference on computer vision (ECCV), pages 334–350, 2018.
  * [43] Xiaodan Liang, Hongfei Zhou, and Eric Xing.  Dynamic-structured semantic propagation network.  In CVPR, pages 752–761, 2018.
  * [44] Di Lin, Dingguo Shen, Siting Shen, Yuanfeng Ji, Dani Lischinski, Daniel Cohen-Or, and Hui Huang.  Zigzagnet: Fusing top-down and bottom-up context for object segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7490–7499, 2019.
  * [45] Guosheng Lin, Anton Milan, Chunhua Shen, and Ian Reid.  Refinenet: Multi-path refinement networks for high-resolution semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1925–1934, 2017.
  * [46] Guosheng Lin, Anton Milan, Chunhua Shen, and Ian D. Reid.  Refinenet: Multi-path refinement networks for high-resolution semantic segmentation.  In CVPR, pages 5168–5177, 2017.
  * [47] Guosheng Lin, Chunhua Shen, Anton van den Hengel, and Ian D. Reid.  Efficient piecewise training of deep structured models for semantic segmentation.  In CVPR, pages 3194–3203, 2016.
  * [48] Tsung-Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie.  Feature pyramid networks for object detection.  In CVPR, pages 936–944, 2017.
  * [49] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [50] Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan Yuille, and Li Fei-Fei.  Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation.  arXiv preprint arXiv:1901.02985, 2019.
  * [51] Sifei Liu, Shalini De Mello, Jinwei Gu, Guangyu Zhong, Ming-Hsuan Yang, and Jan Kautz.  Learning affinity via spatial propagation networks.  Advances in Neural Information Processing Systems, 30, 2017.
  * [52] Ziwei Liu, Xiaoxiao Li, Ping Luo, Chen-Change Loy, and Xiaoou Tang.  Semantic image segmentation via deep parsing network.  In Proceedings of the IEEE international conference on computer vision, pages 1377–1385, 2015.
  * [53] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021.
  * [54] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * [55] Michael Maire, Takuya Narihira, and Stella X Yu.  Affinity cnn: Learning pixel-centric pairwise relations for figure/ground embedding.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 174–182, 2016.
  * [56] Davy Neven, Bert De Brabandere, Marc Proesmans, and Luc Van Gool.  Instance segmentation by jointly optimizing spatial embeddings and clustering bandwidth.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8837–8845, 2019.
  * [57] Alejandro Newell, Kaiyu Yang, and Jia Deng.  Stacked hourglass networks for human pose estimation.  In ECCV, pages 483–499, 2016.
  * [58] Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han.  Learning deconvolution network for semantic segmentation.  In Proceedings of the IEEE international conference on computer vision, pages 1520–1528, 2015.
  * [59] Vaishakh Patil, Christos Sakaridis, Alexander Liniger, and Luc Van Gool.  P3depth: Monocular depth estimation with a piecewise planarity prior.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1610–1621, June 2022.
  * [60] Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, and Jian Sun.  Large kernel matters - improve semantic segmentation by global convolutional network.  In CVPR, pages 1743–1751, 2017.
  * [61] Xi Peng, Rogério Schmidt Feris, Xiaoyu Wang, and Dimitris N. Metaxas.  A recurrent encoder-decoder network for sequential face alignment.  In ECCV (1), volume 9905, pages 38–56, 2016.
  * [62] Tobias Pohlen, Alexander Hermans, Markus Mathias, and Bastian Leibe.  Full-resolution residual networks for semantic segmentation in street scenes.  In CVPR, pages 3309–3318, 2017.
  * [63] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  arXiv preprint arXiv:1511.06434, 2015.
  * [64] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
  * [65] Christos Sakaridis, Dengxin Dai, and Luc Van Gool.  Acdc: The adverse conditions dataset with correspondences for semantic driving scene understanding.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10765–10775, 2021.
  * [66] Pierre Sermanet, David Eigen, Xiang Zhang, Michaël Mathieu, Rob Fergus, and Yann LeCun.  Overfeat: Integrated recognition, localization and detection using convolutional networks.  arXiv preprint arXiv:1312.6229, 2013.
  * [67] Evan Shelhamer, Jonathan Long, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(4):640–651, 2017.
  * [68] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [69] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015.
  * [70] Wei Tang, Pei Yu, and Ying Wu.  Deeply learned compositional models for human pose estimation.  In ECCV, September 2018.
  * [71] Maria Tzelepi and Anastasios Tefas.  Semantic scene segmentation for robotics applications.  In 2021 12th International Conference on Information, Intelligence, Systems & Applications (IISA), pages 1–4. IEEE, 2021.
  * [72] Roberto Valle, José Miguel Buenaposada, Antonio Valdés, and Luis Baumela.  A deeply-initialized coarse-to-fine ensemble of regression trees for face alignment.  In ECCV, pages 609–624, 2018.
  * [73] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [74] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al.  Deep high-resolution representation learning for visual recognition.  IEEE transactions on pattern analysis and machine intelligence, 43(10):3349–3364, 2020.
  * [75] Panqu Wang, Pengfei Chen, Ye Yuan, Ding Liu, Zehua Huang, Xiaodi Hou, and Garrison W. Cottrell.  Understanding convolution for semantic segmentation.  In WACV, 2018.
  * [76] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 568–578, 2021.
  * [77] Zifeng Wu, Chunhua Shen, and Anton van den Hengel.  Wider or deeper: Revisiting the resnet model for visual recognition.  CoRR, abs/1611.10080, 2016.
  * [78] Bin Xiao, Haiping Wu, and Yichen Wei.  Simple baselines for human pose estimation and tracking.  In ECCV, pages 472–487, 2018.
  * [79] Dan Xu, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe.  Pad-net: Multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing.  In CVPR, pages 675–684, 2018.
  * [80] Jing Yang, Qingshan Liu, and Kaihua Zhang.  Stacked hourglass network for robust facial landmark localisation.  In CVPR, pages 2025–2033, 2017.
  * [81] Maoke Yang, Kun Yu, Chi Zhang, Zhiwei Li, and Kuiyuan Yang.  Denseaspp for semantic segmentation in street scenes.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3684–3692, 2018.
  * [82] Wei Yang, Shuang Li, Wanli Ouyang, Hongsheng Li, and Xiaogang Wang.  Learning feature pyramids for human pose estimation.  In ICCV, pages 1290–1299, 2017.
  * [83] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang.  Bisenet: Bilateral segmentation network for real-time semantic segmentation.  In ECCV, pages 334–349, 2018.
  * [84] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang.  Learning a discriminative feature network for semantic segmentation.  In CVPR, pages 1857–1866, 2018.
  * [85] Fisher Yu and Vladlen Koltun.  Multi-scale context aggregation by dilated convolutions.  In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
  * [86] Yuhui Yuan, Xiaokang Chen, Xilin Chen, and Jingdong Wang.  Segmentation transformer: Object-contextual representations for semantic segmentation.  arXiv preprint arXiv:1909.11065, 2019.
  * [87] Yuhui Yuan, Xilin Chen, and Jingdong Wang.  Object-contextual representations for semantic segmentation.  CoRR, abs/1909.11065, 2019.
  * [88] Yuhui Yuan, Jingyi Xie, Xilin Chen, and Jingdong Wang.  Segfix: Model-agnostic boundary refinement for segmentation.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16, pages 489–506. Springer, 2020.
  * [89] Hang Zhang, Han Zhang, Chenguang Wang, and Junyuan Xie.  Co-occurrent features in semantic segmentation.  In CVPR, June 2019.
  * [90] Rui Zhang, Sheng Tang, Yongdong Zhang, Jintao Li, and Shuicheng Yan.  Scale-adaptive convolutions for scene parsing.  In ICCV, pages 2050–2058, 2017.
  * [91] Zhenli Zhang, Xiangyu Zhang, Chao Peng, Xiangyang Xue, and Jian Sun.  Exfuse: Enhancing feature fusion for semantic segmentation.  In ECCV, pages 273–288, 2018.
  * [92] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In CVPR, pages 6230–6239, 2017.
  * [93] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881–2890, 2017.
  * [94] Hengshuang Zhao, Yi Zhang, Shu Liu, Jianping Shi, Chen Change Loy, Dahua Lin, and Jiaya Jia.  Psanet: Point-wise spatial attention network for scene parsing.  In ECCV, pages 270–286, 2018.
  * [95] Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, and Philip HS Torr.  Conditional random fields as recurrent neural networks.  In Proceedings of the IEEE international conference on computer vision, pages 1529–1537, 2015.
  * [96] Zhen Zhu, Mengde Xu, Song Bai, Tengteng Huang, and Xiang Bai.  Asymmetric non-local neural networks for semantic segmentation.  CoRR, abs/1908.07678, 2019.

##  Appendix A Network Instantiation

OVeNet follows the same general architecture as [74]. Since it is built on
HRNet [74], our network contains four stages and two heads, as shown in Table
6. The branch occurs on the 4${}^{\text{th}}$ stage. The semantic head of our
model is constructed from modularized blocks, which are repeated a specific
number of times in each of the four stages. In particular, the blocks are
repeated $1$, $1$, $4$, and $3$ times, respectively, following the same
configuration as the original HRNet model. However, in the offset head, we had
to make a modification to the number of block repetitions in order to address
memory constraints. As a result, the blocks are repeated 2 times in the
4${}^{\text{th}}$ stage of the offset head. Each modularized block in our
network consists of $1$, $2$, $3$, or $4$ branches, depending on the stage in
which it is located (1${}^{\text{st}}$, 2${}^{\text{nd}}$, 3${}^{\text{rd}}$,
or 4${}^{\text{th}}$). Each branch is associated with a different resolution
and is composed of four residual units and one multi-resolution fusion unit.

Table 6:  The architecture of the OVeNet (main body). Within each cell of our
network, there are three distinct components. The first component, represented
by $[\cdot]$, refers to the residual unit. The second component is a numerical
value that specifies the number of times the residual unit is repeated. The
final component is another numerical value that indicates how many times the
modularized block is repeated within the cell. In each residual unit, the
variable $C$ is used to represent the number of channels.

Head | Resolution |  Stage $1$ |  Stage $2$ |  Stage $3$ |  Stage $4$
---|---|---|---|---|---
Semantic | $4\times$ | $\left[\begin{array}[]{c}{1\times 1,64}\\\\[-0.70004pt] {3\times 3,64}\\\\[-0.70004pt] {1\times 1,256}\end{array}\right]$ $\times$$4\times 1$ |  $\left[\begin{array}[]{c}{3\times 3,C}\\\\[-0.70004pt] {3\times 3,C}\end{array}\right]\times 4\times 1$ |  $\left[\begin{array}[]{c}{3\times 3,C}\\\\[-0.70004pt] {3\times 3,C}\end{array}\right]\times 4\times 4$ |  $\left[\begin{array}[]{c}{3\times 3,C}\\\\[-0.70004pt] {3\times 3,C}\end{array}\right]\times 4\times 3$
$8\times$ |  |  $\left[\begin{array}[]{c}{3\times 3,2C}\\\\[-0.70004pt] {3\times 3,2C}\end{array}\right]\times 4\times 1$ |  $\left[\begin{array}[]{c}{3\times 3,2C}\\\\[-0.70004pt] {3\times 3,2C}\end{array}\right]\times 4\times 4$ |  $\left[\begin{array}[]{c}{3\times 3,2C}\\\\[-0.70004pt] {3\times 3,2C}\end{array}\right]\times 4\times 3$
$16\times$ |  |  |  $\left[\begin{array}[]{c}{3\times 3,4C}\\\\[-0.70004pt] {3\times 3,4C}\end{array}\right]\times 4\times 4$ |  $\left[\begin{array}[]{c}{3\times 3,4C}\\\\[-0.70004pt] {3\times 3,4C}\end{array}\right]\times 4\times 3$
$32\times$ |  |  |  |  $\left[\begin{array}[]{c}{3\times 3,8C}\\\\[-0.70004pt] {3\times 3,8C}\end{array}\right]\times 4\times 3$
Offset Vector | $4\times$ |  |  |  |  $\left[\begin{array}[]{c}{3\times 3,C}\\\\[-0.70004pt] {3\times 3,C}\end{array}\right]\times 4\times 2$
$8\times$ |  |  |  |  $\left[\begin{array}[]{c}{3\times 3,2C}\\\\[-0.70004pt] {3\times 3,2C}\end{array}\right]\times 4\times 2$
$16\times$ |  |  |  |  $\left[\begin{array}[]{c}{3\times 3,4C}\\\\[-0.70004pt] {3\times 3,4C}\end{array}\right]\times 4\times 2$
$32\times$ |  |  |  |  $\left[\begin{array}[]{c}{3\times 3,8C}\\\\[-0.70004pt] {3\times 3,8C}\end{array}\right]\times 4\times 2$

##  Appendix B Additional Qualitative Results

We provide additional qualitative comparisons of OVeNet to its HRNet baseline
for the two examined datasets: Cityscapes [19] and ACDC [65]. More
specifically, we provide sets of successful segmentations in Fig. 5 and 6 and
sets of challenging and failure cases in Fig. 7 and 8 correspondingly.

In Fig. 5 we depict the successful results on Cityscapes. We also present an
additional comparison of the default instance of OVeNet built only on HRNet
[74] with the instance of OVeNet built on HRNet$+$OCR [87]. As we can see,
OVeNet (_HRNet_) reduces correctly the number of misclassified terrain pixels
on the right side of the road in the $3^{rd}$ row of Fig. 5. On the other
hand, OVeNet (_HRNet $+$ OCR_) achieves a better prediction since it enlarges
the sidewalk segment and it eliminates the terrain. Regarding the $6^{th}$
row, OVeNet (_HRNet_) expands the terrain on the right, while the HRNet $+$
OCR-based model increases additionally the sidewalk segment in the background.
Overall, the latter generally achieves better predictions than the former.

In Fig. 6 we observe the results of successful segmentations on ACDC, where
HRNet [74] is compared to OVeNet built only on it. Specifically, in the
$2^{nd}$ as well as the $7^{th}$ row of Fig. 6, OVeNet enhances the prediction
made by HRNet in the sidewalk on the right. A similar result happens with the
terrain segment in the $5^{th}$ row, as it covers correctly all the more
space. Thus, our model surpasses baseline’s performance on adverse conditions.

As for Fig. 7, we provide some challenging cases on Cityscapes. To be more
specific, we can see that in both rows, HRNet [74] results in a better
prediction than OVeNet (_HRNet_) (e.g car on the left in the $1^{st}$ row,
terrain on the right in the $2^{nd}$ one). On the other hand, OVeNet (_HRNet
$+$ OCR_) outperforms both the HRNet and HRNet-based model, leading to a
better total outcome.

Regarding Fig. 8, we see some some failure cases of our model built on HRNet
[74] on ACDC. Specifically, in both rows, there are more correctly predicted
labels in the vegetation segment on the right outputted by HRNet than by
OVeNet.

All in all, we observe that OVeNet not only produces results that are very
faithful to ground-truth annotations, but its predictions also surpass the
predictions made by the initial HRNet in terms of quality. By correctly
classifying several pixels which are misclassified by the HRNet baselines,
OVeNet (_HRNet $+$ OCR_) eliminates inconsistencies and enhances the shape and
appearance of respective segments, resulting in more realistic outputs.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

(ae)

(af)

(ag)

(ah)

(ai)

(aj)

(ak)

(al)

(am)

(an)

Figure 5: Additional qualitative results of selected examples on Cityscapes.
From left to right: input image, ground-truth annotation, and prediction with
HRNet [74], OVeNet (_HRNet_), and OVeNet (_HRNet $+$OCR_). Best viewed on a
screen and zoomed in.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

(ae)

(af)

Figure 6: Additional qualitative results of selected examples on ACDC. From
left to right: input image, ground-truth annotation, and prediction with HRNet
[74] and OVeNet. Best viewed on a screen and zoomed in.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

Figure 7: Challenging cases on Cityscapes. From left to right: input image,
ground-truth annotation, and prediction with HRNet [74], OVeNet (_HRNet_), and
OVeNet (_HRNet $+$OCR_). Best viewed on a screen and zoomed in.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 8: Failure cases on ACDC. From left to right: input image, ground-truth
annotation, and prediction with HRNet [74] and OVeNet. Best viewed on a screen
and zoomed in.