# DejaVu: Conditional Regenerative Learning to Enhance Dense Prediction

Shubhankar Borse  Debasmit Das These authors contributed equally to this work.
Hyojin Park 11footnotemark: 1 Hong Cai  Risheek Garrepalli  Fatih Porikli
Qualcomm AI Research
{sborse, debadas, hyojinp, hongcai, rgarrepa<EMAIL_ADDRESS>
Qualcomm AI Research, an initiative of Qualcomm Technologies, Inc.

###### Abstract

We present DejaVu, a novel framework which leverages conditional image
regeneration as additional supervision during training to improve deep
networks for dense prediction tasks such as segmentation, depth estimation,
and surface normal prediction. First, we apply redaction to the input image,
which removes certain structural information by sparse sampling or selective
frequency removal. Next, we use a conditional regenerator, which takes the
redacted image and the dense predictions as inputs, and reconstructs the
original image by filling in the missing structural information. In the
redacted image, structural attributes like boundaries are broken while
semantic context is largely preserved. In order to make the regeneration
feasible, the conditional generator will then require the structure
information from the other input source, i.e., the dense predictions. As such,
by including this conditional regeneration objective during training, DejaVu
encourages the base network to learn to embed accurate scene structure in its
dense prediction. This leads to more accurate predictions with clearer
boundaries and better spatial consistency. When it is feasible to leverage
additional computation, DejaVu can be extended to incorporate an attention-
based regeneration module within the dense prediction network, which further
improves accuracy. Through extensive experiments on multiple dense prediction
benchmarks such as Cityscapes, COCO, ADE20K, NYUD-v2, and KITTI, we
demonstrate the efficacy of employing DejaVu during training, as it
outperforms SOTA methods at no added computation cost.

##  1 Introduction

Figure 1: Training within the DejaVu framework enables dense prediction models
to improve their initial predictions using our proposed loss. The segmentation
results are for the same OCR [87] model with and without DejaVu. The surface
normal results are for SegNet-XTC [43].

Dense prediction tasks produce per-pixel classification or regression results,
such as semantic or panoptic class labels, depth or disparity values, and
surface normal angles. These tasks are critical for many vision applications
to better perceive their surroundings for XR, autonomous driving, robotics,
visual surveillance, and so on. There has been significant success in adopting
neural networks to solve dense prediction tasks through innovative
architectures, data augmentations and training optimizations. For example,
[46] addresses pixel level sampling bias and [7] incorporates boundary
alignment objectives. Orthogonal to existing methods, we explore novel
regeneration-based ideas to understand how additional gradients from
reconstruction tasks complement the established training pipelines for dense
prediction tasks and input representations.

There are works [69, 94] in classification settings that leverage
reconstructions and likelihood-based objectives as auxiliary loss functions to
enhance the quality of feature representations and also improve Open-set/OOD
Detection [55, 48, 49, 25]. The core intuition is that, for discriminative
tasks the model needs a minimal set of features to solve the task and any
feature which does not have discriminative power for the target subset of data
are ignored. Another line of work for dense predictions [54] focuses on depth
completion and leverages reconstruction-based loss to learn complementary
image features that aid better capture of object structures and semantically
consistent features. Following such intuitions, we can see that
reconstruction-based auxiliary loss should capture more information in
representation than discriminative-only training.

Here, we introduce a novel training strategy, DejaVu111In training, DejaVu
redacts the input image and constructs its regenerated versions, in a way,
these regenerated versions are ”already seen” yet not exactly the same due to
initial redaction., for dense prediction tasks with an additional, conditional
reconstruction objective to improve the generalization capacity of the task-
specific base networks as illustrated in Fig. 1. We redact the input image to
remove structure information (e.g., boundaries) while retaining contextual
information. We adopt various redaction techniques that drop out components in
spatial or spectral domains. Then, we enforce a conditional regeneration
module (CRM), which takes the redacted image and the base network’s dense
predictions, to reconstruct the missing information. For regeneration
feasibility, the CRM will require structure information from the dense
predictions. By including this conditional regeneration objective during
training, we encourage the base network to learn and use such structure
information, which leads to more accurate predictions with clearer boundaries
and better spatial consistency, as shown in the experimental section. In
comparison, the supervised loss cannot capture this information alone since
the cross-entropy objective (for segmentation, as an example) looks at the
probability distribution of every pixel. In this sense, DejaVu can implicitly
provide cues to the dense prediction task from the reconstruction objective
depending on the type of redaction we select. We also note that using the same
number of additional regenerated images as a data augmentation scheme does not
provide the performance improvements that DejaVu can achieve (as reported in
the Appendix). This shows that DejaVu conditions the training process more
effectively than any data augmentation technique.

Our DejaVu loss can be applied to train any dense prediction network and does
not incur extra computation at test time. When it is feasible to leverage
additional computation, DejaVu can be extended where we incorporate an
attention-based regeneration module within the dense prediction network,
further improving accuracy. An advantage of regenerating the original image
from predictions is that we can additionally use other losses including text
supervision and cyclic consistency, as described in Section 3.4.

Our extensive experiments on multiple dense prediction tasks, including
semantic segmentation, depth estimation, and surface normal prediction, show
that employing DejaVu during training enables our trained models to outperform
the latest state of the art on several large-scale benchmarks.

Our main contributions are summarized as follows:

  * •

We devise a novel learning strategy, DejaVu, that leverages conditional image
regeneration from redacted input images to improve the overall performance on
dense prediction tasks. (Sec. 3.3)

  * •

We propose redacting the input image to enforce the base networks to learn
accurate dense predictions such that these tasks can precisely condition the
regenerative process. (Sec. 3.1)

  * •

We devise a novel shared attention scheme, DejaVu-SA, by incorporating the
regeneration objective into the parameters of the network. (Sec. 3.4)

  * •

We further provide extensions to DejaVu, such as the text supervision loss
DejaVu-TS and Cyclic consistency loss DejaVu-CL, further improving performance
when additional data is available. (Sec. 3.5)

  * •

DejaVu is a universal framework that can enhance the performance of multiple
networks for essential dense prediction tasks on numerous datasets with no
added inference cost. (Sec. 4)

##  2 Related Work

Dense Prediction with Supervised Learning: Deep learning has been successfully
applied for various dense prediction tasks such as semantic segmentation [53],
with hierarchical branches [10, 11, 95], attention mechanisms [79, 24, 88, 33,
5, 67], and auxiliary losses [4, 7], to point out a few. There are also
extensions to panoptic and instance segmentation [84, 39, 15, 44], plug and
play modules [91, 6], and universal architectures [17, 16]. To improve
performance, many works utilize extra image/text datasets or use test-time
augmentation such as multi-scale inference [77, 68, 78, 23]. Depth estimation
is another dense prediction task where most works usually adopt self-
supervised training paradigms [27] in the stereo setting or [29, 28, 98, 59,
35] in monocular videos, exploiting geometric transformation and consistency
between views to train the network. Many works leverage semantics [30, 42, 8]
and consistency [60, 61, 93], spherical regression [45], uncertainty [1] and
spatial rectifiers [20]. Some efforts simultaneously solve these tasks within
multi-task settings, exploiting inter-task information [50, 13, 38, 66, 43,
41, 92]. In our work, we present a framework and training objective that can
enhance the performance of such dense prediction tasks individually or in a
multi-task setting.

Image Reconstruction as an Auxiliary Task: Autoencoders [32] and their
variants [72] are among popular techniques for reconstruction-based
unsupervised representation learning. Recent works incorporate different
masking [73, 58, 12] and channel removal [89] schemes. Inspired by success in
NLP [12], masking-inspired ideas are exploited to learn image representations,
for instance, [21, 3] via image tokens, [31] in pretraining, [36], [36, 81] in
self-supervised segmentation, and [19] in detection. Reconstruction-based
objectives are also used for domain adaptation [85]. In comparison, our work
aims at making the best use of image reconstruction to improve supervised
image segmentation.

Image Generation & Translation: Pixel-to-pixel image translation [51, 57, 76]
aims to improve the quality of image synthesis conditioned on segmentation
maps. In contrast, we use reconstruction to enhance the dense prediction
quality. There are also image stylization methods that blend two images into a
new one, keeping the content of the one while changing the style according to
the other. [71, 65, 37]. More recently, diffusion-based models have
demonstrated remarkable reconstruction performance [64, 2]. In our work, we
also consider an extension motivated by denoising diffusion. However, instead
of training within the diffusion paradigm, we design an iterative generator
module that reduces the degree of masking across its timesteps using dense
predictions and redacted images.

Figure 2: DejaVu consists of image redaction, dense conditioning from a base
network (such as class-wise masks for semantic segmentation), and a
conditional regeneration module (CRM) that constructs a regenerated version of
the input image, from a redacted image and dense conditioning. The DejaVu loss
backpropagates to update the base network with the supervised loss.

##  3 DejaVu Framework

In this Section, we discuss the DejaVu framework in detail, along with all its
extensions.

Overview: As illustrated in Fig. 2, consider that we train a dense prediction
network (called the base net) $h$, which inputs an image $I$ to generate dense
outputs $C=h(I)$. We first apply redaction to the input $I$, generating the
redacted image $I_{R}$. The redaction methods are explained in Section 3.1.
Next, we pass the redacted image $I_{R}$ and dense outputs $C$ as inputs to a
conditional regeneration module (CRM), as described in Section 3.2. The CRM
outputs a regenerated image $I_{G}$, which is then compared with the original
image to provide a loss for training, as explained in Section 3.3. We also
present an optional shared attention module, DejaVu-SA, which integrates the
regeneration operation from the DejaVu loss into the base net in inference
(Section 3.4). Finally, we elaborate on further extensions of DejaVu that
incorporate vision-language training and consistency regularization (Section
3.5).

In the following subsection, we describe image redaction options.

###  3.1 Image Redaction

Figure 3: Sample redactions in spectral and spatial domains.

When applying redaction to the input image, we intentionally remove the type
of information that would be desired for the dense prediction base network to
learn to generate. Selecting an appropriate redaction style is critical to
ensure the image regeneration is feasible and the dense prediction network
receives useful feedback to learn better features. We consider information
redaction in two domains, i.e., spatial and spectral, which target different
image attributes.

When performing spatial redaction, we mask out specific pixels. The original
pixel values can be removed randomly or in a structured fashion. Figure 3
shows typical examples of random and structured spatial redaction. For
instance, in random spatial redaction, we randomly mask pixels with a fixed
probability $t$. Alternatively, we generate a checkerboard redaction by
setting a block size of $b$. We also extend the checkerboard redaction by
randomly shifting the grids, to generate the random blocks redaction. The
values of $t$ and $b$ are considered as hyperparameters. As visible,
structural motifs and details, such as object silhouettes and semantic class
boundaries, are partially removed with spatial redaction. As a result, the CRM
will enforce and thus facilitate the dense prediction network to embed such
removed and missing information in the predictions for the CRM to regenerate
the original image as closely as possible.

As for spectral redaction, we first transform the original image to the
frequency domain, e.g., Discrete Cosine Transform (DCT). We mask out DCT
components and then apply the inverse transform to obtain the redacted image.
As shown in Fig. 3, lowpass spectral redaction implies masking out high-
frequency DCT components of the input image, while highpass spectral redaction
implies masking out low-frequency components of the input image. We also
experiment with bandstop redactions, by masking out a band from the middle.
Filtering out high-frequency coefficients causes finer image patterns to be
distorted, and applying bandstop filters smears object-level details at
certain scales. When providing such redacted images for regeneration, the
dense prediction network will be required to embed the corresponding
information (e.g., distorted patterns, edges, object details) in its
predictions.

Based on our analysis shown in Section 4, we empirically observe that spatial
redaction performs better on segmentation tasks as the DejaVu loss penalizes
inaccurate class-wise predictions (similar to the one observed in Fig. 1)
heavily. On the other hand, spectral redaction works well on depth and surface
normal estimation tasks as their outputs are penalized heavily if they contain
blurred boundaries.

In the following subsection, we describe the Conditional Regeneration Module
architecture.

###  3.2 Conditional Regeneration Module

The Conditional Regeneration Module (CRM) takes the redacted image $I_{R}$ as
well as the dense conditioning $C$, and regenerates the input image $I$. We
use two types of regeneration modes: 1) Forward mode CRM-F, and 2) Recursive
mode CRM-R, as shown in Fig. 4. Both modules take the dense condition and the
redacted image as an input and produce perceptually similar images to the
original input image at the output. CRM-F, illustrated in Fig. 4(a), simply
consists of stacked Conv-BatchNorm-ReLU blocks, inspired by the related work
on conditional image translation [57].

As the CRM-F performs an auxiliary reconstruction task, there exists a trade-
off between its model complexity and the dense prediction task accuracy. We
study this trade-off in the Appendix. CRM-R, illustrated in Fig. 4(b),
consists of a single Convolution-BatchNorm-ReLU block, which recursively
produces a residual for the reconstructed image, inspired by cold diffusion
[2]. The number of steps is treated as a hyperparameter over which we perform
a search. Hyperparameter choices for both CRM architectures are provided in
the Appendix. Empirically, we find that the CRM-R is more effective for random
occlusions and CRM-F produces better results on structured occlusions.

Inputs to the CRM are the dense condition $C\in\mathbb{R}^{N\times H\times W}$
and redacted input image $I_{R}\in\mathbb{R}^{3\times H\times W}$, where $H$
and $W$ are height and width, and $N$ is the number of predicted channels. We
use two operations to combine the two inputs: 1) multiplication and 2)
concatenation. For the multiplication operation, we average the input image to
a single channel, $\overline{I_{R}}\in\mathbb{R}^{1\times H\times W}$,
broadcast to $N$ channels, and perform element-wise channel multiplication
with the dense condition. Likewise for the concatenation operation, $I_{R}$
and $C$ are concatenated along the channel dimension and fed as input to the
generation module, and thus the size of input channels is $3+N$.

###  3.3 DejaVu Loss to Update Base Network

Figure 4: Architecture of the CRM. Regardless of the dense prediction task,
each mode can cope with redactions effectively, the Forward (one-step) mode
for structured redactions and the Recursive mode for random redactions.

The DejaVu loss, as illustrated in Fig. 2, is computed by comparing the
regenerated image $I_{G}$ to the input image $I$. We add this loss to the
original task loss term during training. Specifically, when comparing the
original and regenerated images, we use the Mean Squared Error (MSE) and LPIPS
losses [90] in order to attain supervision on both local (pixel-level) and
global (content-level) feedback. The total training loss is given as follows:

$\mathcal{L}=\mathcal{L}_{\text{base}}+\mathcal{\gamma}\cdot\mathcal{L}_{\text{regen}},\vspace{-5pt}$
(1)
$\mathcal{L}_{\text{regen}}=\mathcal{\gamma}_{1}\cdot\mathcal{L}_{\text{lpips}}+{\gamma}_{2}\cdot\mathcal{L}_{\text{mse}},$
(2)

where $\mathcal{L}_{\text{base}}$ is the loss from the base training
procedure, e.g., cross-entropy loss for semantic segmentation,
$\mathcal{L}_{1}$ loss for depth estimation. $\mathcal{L}_{\text{regen}}$ is
the loss from our proposed conditional image regeneration. $\gamma$,
$\gamma_{1}$, and $\gamma_{2}$ are hyperparameters that blend these loss
terms.

###  3.4 DejaVu Shared Attention Module (DejaVu-SA)

Figure 5: Folding the CRM into a shared attention mechanism. For dense
supervision, a multi-head attention block consumes dense predictions as
queries while inputs as keys & values. For regeneration, the same block
consumes spectrally redacted inputs as queries and dense predictions as keys &
values.

The DejaVu loss provides improvements at no added inference cost. However, in
cases when we have room for increasing the computational complexity of our
base network, we propose the DejaVu Shared Attention scheme, DejaVu-SA, which
incorporates the DejaVu operation into the parameters of our base network. The
inputs to DejaVu-SA are the image $I$ and dense predictions $C$, and the
outputs are enhanced predictions $\tilde{C}$. The attention module is
illustrated in Fig. 5.

During training, we perform two passes through the attention module. We first
push the input image $I$ through the spectral redaction operation to obtain
the redacted input $I_{R}$. We compute queries
$Q^{s}_{I_{R}}=q^{\prime}(I_{R})$, keys $K^{s}_{C}=k^{\prime}(C)$ and values
$V^{s}_{C}=v^{\prime}(C)$, where $q^{\prime}$, $k^{\prime}$ and $v^{\prime}$
denote patch embedding operations in Fig. 5. Through a multi-head attention
(MHA) operation followed by rearrangement, we pass the outputs through
regeneration decoder $D_{r}$ to obtain the regenerated image
$I_{G}=D_{r}(\text{MHA}(Q^{s}_{I_{R}},K^{s}_{C},V^{s}_{C}))$. This regenerated
image is now supervised by with the original image $I$. In the second pass, we
obtain queries $Q^{s}_{C}=q(C)$, keys $K^{s}_{I}=k(I)$ and values
$V^{s}_{I}=v(I)$ where $q$, $k$ and $v$ denote patch embedding operations
shown in Fig. 5. Through the same MHA operation followed by rearrangement, we
pass the outputs through enhancement decoder $D_{e}$, to obtain enhanced
predictions $\tilde{C}=D_{e}(\text{MHA}(Q^{s}_{C},K^{s}_{I},V^{s}_{I}))$. The
enhanced predictions $\tilde{C}$ are used as the final predictions and are
supervised by ground truth for training the whole network. The intermediate
channel dimension after patch embedding and embedding dimension in the MHA
module are treated as constants and used as a hyperparameter. In Fig. 8, we
provide results for the trade-off between accuracy and complexity, after
scaling these intermediate dimensions.

###  3.5 Extending the Dejavu framework

In this section, we describe extensions of the DejaVu framework that can
produce further enhancements to dense predictions after regenerating the input
image.

Regenerated Text Supervision (DejaVu-TS): After regenerating the input image
from the dense prediction, we propose to perform a novel text-based
supervision objective. More specifically, we match CLIP [62] features between
the original image $I$ and the regenerated image $I_{G}$. Essentially, we can
obtain the CLIP features $f_{G}=\text{CLIP}(I_{G})$ and $f_{I}=\text{CLIP}(I)$
for the reconstructed image and the input image respectively. Optionally, the
CLIP model can be conditioned by only the tokenized input of the class names
for segmentation tasks. The matching loss is the mean squared error between
the features and can be defined as

$\mathcal{L}_{text}=(1/D)||f_{G}-f_{I}||^{2}_{2}$ (3)

where $D$ is the number of elements in the feature vectors.

Cyclic Consistency loss (DejaVu-CL): Another benefit of regenerating the input
image is our proposed Cyclic Consistency Loss. Once the regenerated image
$I_{G}$ is produced, we propose to pass it through the base network $h$ to
produce the regenerated predictions $C_{G}$. We apply the MSE loss to match
the outputs $C_{G}$ with the dense predictions $C$. The detailed structure and
results for this loss term are illustrated in the Appendix.

Figure 6: Visualization of (c) baseline prediction, (d) our enhanced
prediction and the (f) regenerated image given the (a) input image, (b) ground
truth and (e) redacted input. This is shown for three dense prediction tasks:
semantic segmentation on Cityscapes using OCR (top) and surface normal
estimation on NYUD-v2 using SegNet-XTC (middle and bottom).

##  4 Experimental Setup

In this section (and the Appendix), we perform comprehensive analysis and
experiments using DejaVu.

###  4.1 Implentation Details

Datasets: For Semantic Segmentation we consider Cityscapes [18], and ADE20K
[97] datasets. Cityscapes consists of 5,000 annotated images of size
$1024\\!\times\\!2048$, divided into 2,975 training, 500 validation, and 1,525
test images. It covers 11 stuff and 8 thing classes. For Panoptic Segmentation
we consider COCO [47], COCO consists of 118,000 training, 5,000 validation,
and 20,000 test images. There are 53 stuff and 80 thing classes in COCO.
ADE20K contains 20,210 training, 2,000 validation, and 3,000 test images, with
35 stuff and 115 thing classes. For Monocular Depth Estimation we consider
KITTI[26] and use data split from [22]. For the multi-task learning setup, We
also consider NYU-Depth-v2 [56] dataset which has annotations/ground truth for
semantic segmentation, Depth and surface-normals. We use the original 795
train and 654 test images for the NYUD-v2 dataset.

Networks and Training: We train multiple existing baselines using the DejaVu
loss term for various dense prediction tasks. For semantic segmentation with
Cityscapes dataset, we apply DejaVu to HRNet [75], OCR [80] and HMS [70]. For
semantic segmentation with ADE20K dataset, we apply DejaVu loss to Semantic
FPN [40], UperNet [82] and DenseCLIP [63]. Semantic FPN uses PoolFormer [86]
backbone while the rest use ViT backbone [21]. For panoptic segmentation, we
train Mask2Former [16] with Swin [52] backbones using the DejaVu loss. For
depth estimation, we apply DejaVu loss to self-supervised training of
Monodepth2 [28]. For fully and partially supervised multi-task learning
setting, we apply DejaVu loss to MTL [9] and XTC [43] baseline models. For all
base methods, we apply the original loss function ie. cross-entropy loss for
semantic segmentation, $\mathcal{L}_{1}$ norm loss for depth estimation,
cosine similarity loss for surface normal estimation, etc. in addition to the
Déjà vu loss described in Eq. (1). Training details are all discussed in the
Appendix.

Evaluation Metrics: For semantic segmentation tasks, we evaluate using mean
intersection-over-union mIoU. For the panoptic segmentation task, we report
panoptic quality PQ [40]. We also report PQ scores for things and stuff,
denoted as PQth and PQst, respectively. In the multi-task learning setup [50],
depth estimation performance is evaluated using absolute relative error aErr,
and surface normal estimation performance is evaluated using mean error mErr
in the predicted angles. For monocular depth estimation, we use absolute
relative error Abs Rel and squared relative error Sq Rel. Furthermore, the
classification metric $\delta_{1}$ measures whether the ratio between ground
truth and estimated depth values is within a certain range around 1. We also
report GMacs to measure efficiency.

Backbone | Method | mIoU$\uparrow$ | GMacs$\downarrow$
---|---|---|---
HRNet18 |  HRNet [75] | 77.6 | 19
+DejaVu | 78.8 | 19
HS3[4] | 78.1 | 19
HS3-Fuse[4] | 81.4 | 39
OCR [80] | 80.7 | 39
+DejaVu | 82.0 | 39
MiT-B5 |  Segformer [83] | 84.0 | 362
Swin-L [52] |  Mask2Former [16] | 83.3 | 251
SeMask [34] | 84.0 | 258
ViT |  ViT Adapter [14] | 84.9 | 1089
HRNet48 | HRNet | 84.7 | 175
+DejaVu | 85.4 | 175
OCR | 86.1 | 348
+DejaVu | 86.5 | 348
HMS [70] | 86.7 | 893
+DejaVu | 87.1 | 893
Table 1: Comparing with SOTA methods on Cityscapes val.

###  4.2 Experiments using the DejaVu Loss

In this subsection, we perform experiments by training various baseline models
using our proposed DejaVu loss function.

Semantic Segmentation: In Table 1, we report results of comparing our proposed
framework with respect to semantic segmentation baselines on the Cityscapes
val dataset. As observed, our method can produce boost of more than 1.3 pts in
mIoU when using HRNet18 backbones. When using heavier HRNet48 versions,
training with the DejaVu loss still improves, but the relative improvement
reduces closer to SOTA scores. We also train HRNet-OCR [87] and HRNet-OCR-HMS
[70] backbones using the DejaVu loss. The HMS backbone trained with DejaVu
loss acheives the SOTA score on Cityscapes val. In Table 3, we show
improvement over existing semantic segmentation baselines on the ADE20K val
dataset. Specifically, adding DejaVu loss on top of Semantic FPN, UPerNet and
DenseCLIP produces consistent improvements in mIoU.

Method | Backbone | PQ$\uparrow$ | PQst$\uparrow$ | PQth$\uparrow$
---|---|---|---|---
MaX-Deeplab [74] | Max-S | 48.4 | 53.0 | 41.5
MaskFormer [17] | Swin-T | 47.7 | 51.7 | 41.7
Mask2Former [16] | Swin-T | 53.2 | 59.3 | 44.0
+DejaVu | Swin-T | 54.3 | 60.5 | 44.9
MaX-Deeplab [74] | Max-L | 51.1 | 57.0 | 42.2
K-Net [91] | Swin-L | 54.6 | 60.2 | 46.0
MaskFormer [17] | Swin-L | 52.7 | 58.5 | 44.0
Mask2Former [16] | Swin-L | 57.6 | 64.2 | 47.5
+DejaVu | Swin-L | 58.0 | 64.4 | 48.3
Table 2: Comparison with SOTA methods on COCO Panoptic Segmentation val.

Panoptic Segmentation: In Table 2, we show results for panoptic segmentation
on the COCO val dataset. We trained the previous SOTA method, Mask2former
[16], using our DejaVu loss function and observed consistent improvements in
PQ on two different Swin backbones.

Method | Backbone | mIoU$\uparrow$
---|---|---
Semantic FPN [40] | PoolFormer-M48 | 42.4
+DejaVu | PoolFormer-M48 | 43.3
UperNet [82] |  ViT-B [21] | 47.4
+DejaVu | ViT-B | 48.2
SETR-MLA-DeiT [96] | ViT-B | 46.2
Semantic FPN [40] | ViT-B | 48.3
DenseCLIP [63] | ViT-B | 49.8
+DejaVu | ViT-B | 50.3
Mask2Former [16] | Swin-L | 56.0
+DejaVu | Swin-L | 56.5
Table 3: Comparison on ADE20K Semantic Segmentation val.

Self-Supervised Monocular Depth Estimation: In Table 4, we show comparison
results on monocular depth estimation after training the Monodepth2-R50
baseline using the DejaVu loss. Results show improved performance in terms of
lower Abs Rel and Sq Rel error and higher $\delta_{1}$ compared to other
competitive baselines. Interestingly, Monodepth2 produces much higher Sq Rel
error compared to the highly competitive PackNet [29]. However, training with
DejaVu reduces the Sq Rel error by a large margin, successfully outperforming
PackNet. This shows that DejaVu can also work well with fully self-supervised
training schemes, in addition to conventional supervised training.

Method | Abs Rel$\downarrow$ | Sq Rel $\downarrow$ |  $\delta_{1}$$\uparrow$
---|---|---|---
PackNet-SfM [29] | 0.111 | 0.785 | 0.878
Guizilini [30] | 0.113 | 0.831 | 0.878
Klingner [42] | 0.112 | 0.833 | 0.884
DiPE [35] | 0.112 | 0.875 | 0.880
Patil [59] | 0.111 | 0.821 | 0.883
Monodepth2 [28] | 0.110 | 0.903 | 0.883
+DejaVu | 0.108 | 0.769 | 0.885
Table 4: Comparison after training with DejaVu loss on the KITTI eigen split
for self-supervised monocular depth estimation.

Multi-Task Learning: We also report results after training multi-task learning
models using DejaVu loss, in Tables 5 and 6. We add the DejaVu loss to the
training objectives of SegNet-MTL and SegNet-XTC baselines provided in [43].
Adding the DejaVu loss shows increased semantic segmentation accuracy (mIoU)
and decreased depth estimation and surface normal estimation errors in all
dense tasks for both the fully supervised setting in Table 6 and partially
supervised setting in Table 5.

Labels | Method | Seg.(mIoU)$\uparrow$ | Depth(aErr)$\downarrow$ | Norm(mErr)$\downarrow$
---|---|---|---|---
Random |  MTL [9] | 28.30 | 0.6488 | 32.89
+DejaVu | 30.13 | 0.6072 | 31.97
XTC [43] | 34.26 | 0.5787 | 31.06
+DejaVu | 35.72 | 0.5665 | 29.82
Single |  MTL [9] | 26.32 | 0.6482 | 33.31
+DejaVu | 28.07 | 0.6264 | 32.02
XTC [43] | 30.36 | 0.6088 | 32.08
+DejaVu | 31.02 | 0.5959 | 32.15

Table 5: Comparison after training with DejaVu loss on NYUD-v2 for Partially
Supervised Multi-Task Learning.

Method | Seg.(mIoU)$\uparrow$ | Depth(aErr)$\downarrow$ | Norm(mErr)$\downarrow$
---|---|---|---
MTL [9] | 36.95 | 0.5510 | 29.51
+DejaVu | 37.40 | 0.5426 | 28.74
DWA [50] | 36.46 | 0.5429 | 29.45
GradNorm [13] | 37.19 | 0.5775 | 28.51
MTAN [50] | 39.39 | 0.5696 | 28.89
MGDA [66] | 38.65 | 0.5572 | 28.89
XTC [43] | 41.00 | 0.5148 | 28.58
+DejaVu | 42.69 | 0.4996 | 27.49

Table 6: Comparison after training with DejaVu loss on NYUD-v2 for Fully
Supervised Multi-Task Learning.

###  4.3 Ablation Studies and Analyses

Visualization: Figure 6 shows qualitative results with and without our DejaVu
loss. Each row corresponds to different images and tasks, for separate
baselines trained with and without DejaVu loss. The first row shows semantic
segmentation on Cityscapes. Column (c) shows the baseline prediction without
our DejaVu loss while column (d) shows the prediction with the DejaVu loss. We
observe that our proposed framework produces better quality semantic and
panoptic segmentation masks as it can better perceive the structure of the
pavement. Furthermore, the regenerated image (f) structurally resembles the
input image (a) with some error margin. In the second and third rows, we
report visual results for surface normal estimation on NYUD-v2 where we obtain
better quality predictions (d) compared to the baseline (c). Also, the
regeneration module has sharpened the spectrally redacted input image (e) to
produce the regenerated image (f).

Varying the types of Redaction: In Table 8, we study the effect of applying
the different types of redaction as explained in Section 3.1, to various dense
tasks. Results are shown on the NYUD-v2 dataset for the single-task learning
setup. We pick the ”Random Blocks” spatial redaction and apply a bandstop
spectral redaction selected for various tasks. For all tasks, redaction
improves performance. However, we observe that the semantic segmentation
performance has better improvement for the spatial redaction technique while
depth and normals perform better when clubbed with spectral redaction. We
conclude that this is because segmentation is a pixel-wise classification task
and hence best complements a spatial in-painting task for regeneration. On the
other hand, depth and normal estimation is a regression task and requires
texture information (as described by its corresponding spectrum) such that it
learns accurate shapes to produce a sharp regeneration.

Baseline | DejaVu | DejaVu-TS | mIoU$\uparrow$
---|---|---|---
HRNet18 | ✗ | ✗ | 77.6
✓ |  ✗ | 78.8
✓ |  ✓ | 79.2
HRNet18-OCR | ✗ | ✗ | 80.7
✓ |  ✗ | 82.0
✓ |  ✓ | 82.3
Table 7: Effect of applying CLIP supervision (extra data) over the regenerated
images to train various baseline models on Cityscapes.

In Fig. 7, we study the effect of redacting various bands of spectra in the
spectral redaction (DCT) block for depth estimation. We find that the error is
lowest when we use a middle band frequency for redaction. This is because most
of shape information is contained in the middleband of spectra. Highband
components contain grainy artifacts, and lowband components contain textures,
both which are not expected to be present in the given condition (depth map).

###  4.4 Extending DejaVu

DejaVu Text Supervision: In Table 7, we show the effect of applying DejaVu-TS,
described in Section 3.5, as an additional loss. We observe that adding text
supervision on top of the DejaVu loss can improve semantic segmentation
performance. This is because text supervision involves matching CLIP features
between the original input image and the regenerated image. The CLIP model,
trained on COCO, embeds additional knowledge based on textual context. Through
the DejaVu regenerated image, this model is able to provide textual ques to
the base network.

DejaVu Shared Attention Module: Figure 8 shows accuracy vs. complexity (left)
and size (right) analysis for our proposed shared attention module DejaVu-SA,
when evaluated on the Cityscapes semantic segmentation task. The red plot
simply scales HRNet base model using different width multipliers. The blue
plot scales the DejaVu-SA module by increasing its intermediate dimensions. It
is important to note that HRNet18 model with DejaVu-SA can produce a higher
mIoU score over HRNet20, with lower GMacs and parameters. This implies that
the enhanced performance is not due to added model complexity but due to the
additional embedded regeneration-based context.

Method | Redaction | Seg.(mIoU)$\uparrow$ | Depth(aErr)$\downarrow$ | Norm(mErr)$\downarrow$
---|---|---|---|---
Baseline | ✗ | 37.25 | 59.70 | 26.30
Ours | Spatial | 38.38 | 58.34 | 26.07
Ours | Spectral | 38.21 | 56.76 | 25.75

Table 8: Studying the performance of spatial v/s spectral redaction on NYUD-v2
on three tasks, using the single-task-learning setting. Figure 7:  Redacting
various bands of frequencies to observe the performance with DejaVu loss on
NYUD-v2 for depth estimation.  Figure 8: Performance of the proposed DejaVu-SA
shared attention module, folding our regeneration operation into the network.
DejaVu-SA module shows better performance at the same GMacs, as compared to
simply scaling up the baseline architecture.

##  5 Conclusion

We proposed the DejaVu framework for enhancing the performance on various
dense prediction tasks. DejaVu consists of adding a conditional regeneration
module to reconstruct original inputs from redacted inputs and dense
predictions. The reconstruction loss serves as an additional objective to
produce a more generalizable dense prediction network. Additionally, we
extended our framework to include an attention mechanism, text supervision,
and cycle consistency losses. We performed experiments using DejaVu on various
backbones on different dense prediction tasks such as segmentation, depth
estimation, and surface normal estimation. Results show that our framework
produces significant improvement in performance over existing baselines both
quantitatively and qualitatively. We also conducted various additional
analyses and ablations to study the efficacy of different design choices in
our framework.

## References

  * [1] Gwangbin Bae, Ignas Budvytis, and Roberto Cipolla.  Estimating and exploiting the aleatoric uncertainty in surface normal estimation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13137–13146, 2021.
  * [2] Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Cold diffusion: Inverting arbitrary image transforms without noise.  arXiv preprint arXiv:2208.09392, 2022.
  * [3] Hangbo Bao, Li Dong, and Furu Wei.  Beit: Bert pre-training of image transformers.  arXiv preprint arXiv:2106.08254, 2021.
  * [4] Shubhankar Borse, Hong Cai, Yizhe Zhang, and Fatih Porikli.  Hs3: Learning with proper task complexity in hierarchically supervised semantic segmentation.  arXiv preprint arXiv:2111.02333, 2021.
  * [5] Shubhankar Borse, Marvin Klingner, Varun Ravi Kumar, Hong Cai, Abdulaziz Almuzairee, Senthil Yogamani, and Fatih Porikli.  X-align: Cross-modal cross-view alignment for bird’s-eye-view segmentation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3287–3297, 2023.
  * [6] Shubhankar Borse, Hyojin Park, Hong Cai, Debasmit Das, Risheek Garrepalli, and Fatih Porikli.  Panoptic, instance and semantic relations: A relational context encoder to enhance panoptic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1269–1279, 2022.
  * [7] Shubhankar Borse, Ying Wang, Yizhe Zhang, and Fatih Porikli.  Inverseform: A loss function for structured boundary-aware segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5901–5911, 2021.
  * [8] Hong Cai, Janarbek Matai, Shubhankar Borse, Yizhe Zhang, Amin Ansari, and Fatih Porikli.  X-distill: Improving self-supervised monocular depth via cross-task distillation.  In Proceedings of the British Machine Vision Conference, 2022.
  * [9] Rich Caruana.  Multitask learning.  Machine learning, 28(1):41–75, 1997.
  * [10] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs.  PAMI, 2018.
  * [11] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  arXiv:1706.05587, 2017.
  * [12] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever.  Generative pretraining from pixels.  In International conference on machine learning, pages 1691–1703. PMLR, 2020.
  * [13] Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich.  Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks.  In International conference on machine learning, pages 794–803. PMLR, 2018.
  * [14] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao.  Vision transformer adapter for dense predictions.  arXiv preprint arXiv:2205.08534, 2022.
  * [15] Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang-Chieh Chen.  Panoptic-DeepLab: A simple, strong, and fast baseline for bottom-up panoptic segmentation.  In CVPR, 2020.
  * [16] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar.  Masked-attention mask transformer for universal image segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1290–1299, 2022.
  * [17] Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov.  Per-pixel classification is not all you need for semantic segmentation.  In NeurIPS, 2021.
  * [18] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The Cityscapes dataset for semantic urban scene understanding.  In CVPR, 2016.
  * [19] Giancarlo Di Biase, Hermann Blum, Roland Siegwart, and Cesar Cadena.  Pixel-wise anomaly detection in complex driving scenes.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16918–16927, 2021.
  * [20] Tien Do, Khiem Vuong, Stergios I Roumeliotis, and Hyun Soo Park.  Surface normal estimation of tilted images via spatial rectifier.  In European Conference on Computer Vision, pages 265–280. Springer, 2020.
  * [21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [22] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  Advances in neural information processing systems, 27, 2014.
  * [23] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao.  Eva: Exploring the limits of masked visual representation learning at scale.  arXiv preprint arXiv:2211.07636, 2022.
  * [24] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu.  Dual attention network for scene segmentation.  In CVPR, 2019.
  * [25] Risheek Garrepalli.  Oracle analysis of representations for deep open set detection.  arXiv preprint arXiv:2209.11350, 2022.
  * [26] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  The International Journal of Robotics Research, 32(11):1231–1237, 2013.
  * [27] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow.  Unsupervised monocular depth estimation with left-right consistency.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 270–279, 2017.
  * [28] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow.  Digging into self-supervised monocular depth estimation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3828–3838, 2019.
  * [29] Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raventos, and Adrien Gaidon.  3d packing for self-supervised monocular depth estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2485–2494, 2020.
  * [30] Vitor Guizilini, Rui Hou, Jie Li, Rares Ambrus, and Adrien Gaidon.  Semantically-guided representation learning for self-supervised monocular depth.  In International Conference on Learning Representations, 2019.
  * [31] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022.
  * [32] Geoffrey E Hinton and Richard Zemel.  Autoencoders, minimum description length and helmholtz free energy.  Advances in neural information processing systems, 6, 1993.
  * [33] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu.  CCNet: Criss-cross attention for semantic segmentation.  In ICCV, 2019.
  * [34] Jitesh Jain, Anukriti Singh, Nikita Orlov, Zilong Huang, Jiachen Li, Steven Walton, and Humphrey Shi.  Semask: Semantically masking transformer backbones for effective semantic segmentation.  arXiv, 2021.
  * [35] Hualie Jiang, Laiyan Ding, Zhenglong Sun, and Rui Huang.  Dipe: Deeper into photometric errors for unsupervised learning of depth and ego-motion from monocular videos.  In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10061–10067. IEEE, 2020.
  * [36] Srivallabha Karnam.  Self-Supervised Learning for Segmentation using Image Reconstruction.  Rochester Institute of Technology, 2020.
  * [37] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110–8119, 2020.
  * [38] Alex Kendall, Yarin Gal, and Roberto Cipolla.  Multi-task learning using uncertainty to weigh losses for scene geometry and semantics.  In CVPR, 2018.
  * [39] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár.  Panoptic feature pyramid networks.  In CVPR, 2019.
  * [40] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár.  Panoptic feature pyramid networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6399–6408, 2019.
  * [41] Marvin Klingner, Shubhankar Borse, Varun Ravi Kumar, Behnaz Rezaei, Venkatraman Narayanan, Senthil Yogamani, and Fatih Porikli.  X3kd: Knowledge distillation across modalities, tasks and stages for multi-camera 3d object detection.  arXiv preprint arXiv:2303.02203, 2023.
  * [42] Marvin Klingner, Jan-Aike Termöhlen, Jonas Mikolajczyk, and Tim Fingscheidt.  Self-supervised monocular depth estimation: Solving the dynamic object problem by semantic guidance.  In European Conference on Computer Vision, pages 582–600. Springer, 2020.
  * [43] Wei-Hong Li, Xialei Liu, and Hakan Bilen.  Learning multiple dense prediction tasks from partially annotated data.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18879–18889, 2022.
  * [44] Yanwei Li, Hengshuang Zhao, Xiaojuan Qi, Yukang Chen, Lu Qi, Liwei Wang, Zeming Li, Jian Sun, and Jiaya Jia.  Fully convolutional networks for panoptic segmentation with point-based supervision.  arXiv preprint arXiv:2108.07682, 2021.
  * [45] Shuai Liao, Efstratios Gavves, and Cees GM Snoek.  Spherical regression: Learning viewpoints, surface normals and 3d rotations on n-spheres.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9759–9767, 2019.
  * [46] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In ICCV, 2017.
  * [47] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [48] Si Liu, Risheek Garrepalli, Thomas Dietterich, Alan Fern, and Dan Hendrycks.  Open category detection with PAC guarantees.  In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3169–3178. PMLR, 10–15 Jul 2018\.
  * [49] Si Liu, Risheek Garrepalli, Dan Hendrycks, Alan Fern, Debashis Mondal, and Thomas G Dietterich.  Pac guarantees and effective algorithms for detecting novel categories.  J. Mach. Learn. Res., 23:44–1, 2022.
  * [50] Shikun Liu, Edward Johns, and Andrew J Davison.  End-to-end multi-task learning with attention.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1871–1880, 2019.
  * [51] Xihui Liu, Guojun Yin, Jing Shao, Xiaogang Wang, et al.  Learning to predict layout-to-image conditional convolutions for semantic image synthesis.  Advances in Neural Information Processing Systems, 32, 2019.
  * [52] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  arXiv:2103.14030, 2021.
  * [53] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In CVPR, 2015.
  * [54] Kaiyue Lu, Nick Barnes, Saeed Anwar, and Liang Zheng.  From depth what can you see? depth completion via auxiliary image reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11306–11315, 2020.
  * [55] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan.  Hybrid models with deep and invertible features.  In International Conference on Machine Learning, pages 4723–4732. PMLR, 2019.
  * [56] Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus.  Indoor segmentation and support inference from rgbd images.  In ECCV, 2012.
  * [57] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2337–2346, 2019.
  * [58] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544, 2016.
  * [59] Vaishakh Patil, Wouter Van Gansbeke, Dengxin Dai, and Luc Van Gool.  Don’t forget the past: Recurrent depth estimation from monocular video.  IEEE Robotics and Automation Letters, 5(4):6813–6820, 2020.
  * [60] Xiaojuan Qi, Renjie Liao, Zhengzhe Liu, Raquel Urtasun, and Jiaya Jia.  Geonet: Geometric neural network for joint depth and surface normal estimation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 283–291, 2018.
  * [61] Xiaojuan Qi, Zhengzhe Liu, Renjie Liao, Philip HS Torr, Raquel Urtasun, and Jiaya Jia.  Geonet++: Iterative geometric neural network with edge-aware refinement for joint depth and surface normal estimation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020\.
  * [62] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [63] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu.  Denseclip: Language-guided dense prediction with context-aware prompting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18082–18091, 2022.
  * [64] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  * [65] Artsiom Sanakoyeu, Dmytro Kotovenko, Sabine Lang, and Bjorn Ommer.  A style-aware content loss for real-time hd style transfer.  In proceedings of the European conference on computer vision (ECCV), pages 698–714, 2018.
  * [66] Ozan Sener and Vladlen Koltun.  Multi-task learning as multi-objective optimization.  Advances in neural information processing systems, 31, 2018.
  * [67] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid.  Segmenter: Transformer for semantic segmentation.  In ICCV, 2021.
  * [68] Weijie Su, Xizhou Zhu, Chenxin Tao, Lewei Lu, Bin Li, Gao Huang, Yu Qiao, Xiaogang Wang, Jie Zhou, and Jifeng Dai.  Towards all-in-one pre-training via maximizing multi-modal mutual information.  arXiv preprint arXiv:2211.09807, 2022.
  * [69] Xin Sun, Zhenning Yang, Chi Zhang, Keck-Voon Ling, and Guohao Peng.  Conditional gaussian distribution learning for open set recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13480–13489, 2020.
  * [70] Andrew Tao, Karan Sapra, and Bryan Catanzaro.  Hierarchical multi-scale attention for semantic segmentation.  arXiv:2005.10821, 2020.
  * [71] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky.  Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6924–6932, 2017.
  * [72] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol.  Extracting and composing robust features with denoising autoencoders.  In Proceedings of the 25th international conference on Machine learning, pages 1096–1103, 2008.
  * [73] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou.  Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.  Journal of machine learning research, 11(12), 2010.
  * [74] Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen.  MaX-DeepLab: End-to-end panoptic segmentation with mask transformers.  In CVPR, 2021.
  * [75] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao.  Deep high-resolution representation learning for visual recognition.  PAMI, 2019.
  * [76] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8798–8807, 2018.
  * [77] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al.  Image as a foreign language: Beit pretraining for all vision and vision-language tasks.  arXiv preprint arXiv:2208.10442, 2022.
  * [78] Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al.  Internimage: Exploring large-scale vision foundation models with deformable convolutions.  arXiv preprint arXiv:2211.05778, 2022.
  * [79] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In CVPR, 2018.
  * [80] Zian Wang, David Acuna, Huan Ling, Amlan Kar, and Sanja Fidler.  Object instance annotation with deep extreme level set evolution.  In CVPR, 2019.
  * [81] Xide Xia and Brian Kulis.  W-net: A deep model for fully unsupervised image segmentation.  arXiv preprint arXiv:1711.08506, 2017.
  * [82] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun.  Unified perceptual parsing for scene understanding.  In ECCV, 2018.
  * [83] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo.  Segformer: Simple and efficient design for semantic segmentation with transformers.  In NeurIPS, 2021.
  * [84] Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, and Raquel Urtasun.  Upsnet: A unified panoptic segmentation network.  In CVPR, 2019.
  * [85] Jinyu Yang, Weizhi An, Sheng Wang, Xinliang Zhu, Chaochao Yan, and Junzhou Huang.  Label-driven reconstruction for domain adaptation in semantic segmentation.  In European conference on computer vision, pages 480–498. Springer, 2020.
  * [86] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan.  Metaformer is actually what you need for vision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10819–10829, 2022.
  * [87] Yuhui Yuan, Xilin Chen, and Jingdong Wang.  Object-contextual representations for semantic segmentation.  In European conference on computer vision, pages 173–190. Springer, 2020.
  * [88] Yuhui Yuan, Lang Huang, Jianyuan Guo, Chao Zhang, Xilin Chen, and Jingdong Wang.  OCNet: Object context for semantic segmentation.  IJCV, 2021.
  * [89] Richard Zhang, Phillip Isola, and Alexei A Efros.  Colorful image colorization.  In European conference on computer vision, pages 649–666. Springer, 2016.
  * [90] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.
  * [91] Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy.  K-net: Towards unified image segmentation.  In NeurIPS, 2021.
  * [92] Yizhe Zhang, Shubhankar Borse, Hong Cai, and Fatih Porikli.  Auxadapt: Stable and efficient test-time adaptation for temporally consistent video semantic segmentation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2339–2348, 2022.
  * [93] Yizhe Zhang, Shubhankar Borse, Hong Cai, Ying Wang, Ning Bi, Xiaoyun Jiang, and Fatih Porikli.  Perceptual consistency in video segmentation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2564–2573, 2022.
  * [94] Yuting Zhang, Kibok Lee, and Honglak Lee.  Augmenting supervised neural networks with unsupervised objectives for large-scale image classification.  In International conference on machine learning, pages 612–621. PMLR, 2016.
  * [95] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In CVPR, 2017.
  * [96] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al.  Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6881–6890, 2021.
  * [97] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Scene parsing through ade20k dataset.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633–641, 2017.
  * [98] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe.  Unsupervised learning of depth and ego-motion from video.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1851–1858, 2017.


# Virtual Power Grid Reinforcement
via Coordinated Volt/VAr Control

Jonas G. Matt, Lukas Ortmann, Saverio Bolognani, Florian Dörfler Jonas G.
Matt, Lukas Ortmann, Saverio Bolognani, and Florian Dörfler are with the
Automatic Control Laboratory at ETH Zurich, Switzerland. Emails:
<EMAIL_ADDRESS>
###### Abstract

The rise in residential photovoltaics (PV) as well as other distributed energy
sources poses unprecedented challenges for the operation of distribution
grids. The high active power infeed of such sources during times of peak
production is a stress test which distribution grids have usually not been
exposed to in the past. When high amounts of active power are injected into
the grid, the overall power flow is often limited because of voltages reaching
their upper acceptable limits. Volt/VAr methods aim to raise this power flow
limit by controlling the voltage using reactive power. This way, more active
power can be transmitted safely without physically reinforcing the grid. In
this paper, we use real consumption and generation data on a low-voltage CIGRÉ
grid model and an experiment on a real distribution grid feeder to analyze how
different Volt/VAr methods can virtually reinforce the distribution grid. We
show that droop control and machine-learning-improved droop control virtually
reinforce the grid but do not utilize the reactive power resources to their
full extent. In contrast, methods which coordinate the usage of reactive power
resources across the grid, such as Online Feedback Optimization (OFO), can
reinforce the grid to its full potential. The simulation study performed on
data of an entire year suggests that OFO can enable another 9% of maximum
active power injections. To achieve that, OFO only requires voltage magnitude
measurements, minimal model knowledge and communication with the reactive
power sources. A real-life experiment provides a demonstration of how OFO acts
at the level of a single device, and proves the practical feasibility of the
proposed approach.

##  I Introduction

Electrifying the transportation sector and building heating will lead to much
higher peak electric power consumption in distribution grids. At the same
time, the shift towards decentralized energy resources and residential PV in
particular will increase the peak electric power generation in distribution
grids. While shifting power consumption in time and storing power generation
in local batteries can reduce the peak power flows, some parts of the
distribution grids will still reach or have already reached their capacity
limit, which is determined both by the maximal permissible current as well as
voltage deviation. Which of these two contraints becomes violated first
depends on the topology of the grid. On electrically shorter lines, the
maximum current constraints are usually reached before the voltage deviates
too much. On electrically longer lines however, the voltages reach their lower
or upper limit before the current reaches its limit. Today’s distribution
grids typically consist of electrically long lines which means that the
voltage deviations determine the grid’s capacity. A study found that 75% of
the necessary grid reinforcement in the low voltage distribution grids in
Germany is due to voltage problems [1, page 172]. Therefore, controlling the
voltage to enforce voltage limits can allow more active power to flow and
hence virtually reinforce the grid. One of the most economic ways to control
the voltage is Volt/VAr control, i.e. controlling the amount of reactive power
injected or absorbed at certain nodes in the grid. Conveniently, the inverters
of photovoltaic systems, heat pumps, or electric vehicle chargers can
typically choose their reactive power setpoints rather freely and thus
actively participate in the mitigation of any voltage deviations that their
active power generation or consumption might cause. Most modern grid codes
hence stipulate some form of Volt/VAr control to be done, e.g. by PV inverters
[2, 3, 4]. The state-of-the-art is droop control, a scheme in which every
inverter selects its reactive power setpoint according to a predefined
function of the local voltage measurement.

We consider the virtual reinforcement, provided by Volt/VAr control, to be
optimal when the voltage limits are enforced whenever the available reactive
power resources allow to do so as this maximizes the possible active power
flows and hence the grid capacity. This feasibility problem can be phrased as
an Optimal Reactive Power Flow (ORPF) problem. It was shown that local
controllers do not necessarily find the solution of this ORPF problem and that
coordination is needed for optimal virtual reinforcement [5]. Fortunately, the
advance of communication capabilities in distribution grids render
coordination-based controllers a promising alternative to conventional local
methods. However, a communication infrastructure makes the control setup more
complicated. We are therefore interested in how much more virtual
reinforcement can coordinated Volt/VAr methods provide in comparison to local
Volt/VAr methods.
The local methods we analyze are droop control, as the state-of-the-art, and
an approach based on machine learning (ML), called ML-tuned droop. This
approach attempts to learn the ORPF solution from data [6] and therefore tries
to achieve optimal virtual reinforcement without using a communication
channel. It solves the ORPF problem offline to generate data points and a ML
approach then takes these to synthesize droop curves for each reactive power
source trying to mimic the ORPF solution.
The coordinated methods we present also aim at solving the ORPF problem. The
first one is a standard ORPF solver. However, it needs an accurate grid model,
is not robust to model mismatch, and needs to measure or estimate all active
and reactive consumption and production in the grid. This means not only
communication with the reactive power resources is needed but also with
sensors everywhere in the grid that currently do not exist. These requirements
are unrealistic in a distribution grid setup and therefore the ORPF solver can
only serve as the benchmark for optimal virtual grid reinforcement. The second
coordinated method is called OFO and circumvents the problems of a standard
ORPF solver by using measurements from the grid as feedback. It uses this to
iteratively update the reactive power setpoints until it has converged to the
solution of the ORPF problem. The method is characterized by being nearly
model-free, robust and only voltage magnitude measurements are needed. It
comes with theoretical convergence guarantees, strictly respects constraints
in steady-state and has already been tested experimentally on power system
testbeds [7, 8, 9]. OFO controllers can be implemented as centralized or
distributed controllers. A centralized controller gathers measurements from
and sends setpoints to the reactive power sources [7]. A distributed
controller can be implemented in the reactive power resources with distributed
communication between them [8].

Overall, we analyze the virtual reinforcement capabilities of three Volt/VAr
control methods. Namely, droop control, ML-tuned droop control, and OFO. We
are interested in how much more active power can be transported through the
grid when using each of these three controllers. A special focus is on the
difference between local and coordinated methods. We implement the three
controllers on a CIGRÉ benchmark grid and quantify their performance using
real consumption and PV generation data. Finally, we show results from a
proof-of concept experiment of OFO on a real distribution grid feeder to
demonstrate its viability in a real-life setup and its capability to achieve
grid reinforcement.

The remainder of this paper is structured as follows: In Section II, we
describe the setup of the numerical study. The three different Volt/VAr
controllers are introduced in Section III. We present the simulation results
in Section IV and the experimental results in Section V. Finally, we conclude
the paper in Section VI.

##  II Simulation and data setup

The comparison of the different control algorithms is conducted on a benchmark
distribution grid, and using real load and PV generation data to obtain
meaningful time-series results. The simulation framework is based on Python
and the open-source package pandapower [10]. All code has been made available
here [11].

###  II-A Benchmark Low Voltage Grid

At the foundation of the simulation framework lies the benchmark low voltage
distribution grid with European layout, as proposed by CIGRÉ in 2014 [12]. It
is available as a native implementation in pandapower and has been found to be
a good testbed for the analysis of voltage constraint violations. As shown in
Fig. 1, only the residential subnetwork is used, which is characterized by a
radial structure, underground cable transmission, a line-to-line voltage of
400 V and a system frequency of 50 Hz. A transformer connects the network to
the external MV grid, modeled as a constant voltage source at 1 p.u. The grid
layout comprises 18 measurement buses, 5 of which are the connection points of
loads with local PV generation. Each load bus aggregates a small neighborhood
of buildings with nominal load values ranging from 15 to 55 kW per bus.

Figure 1: The CIGRÉ low-voltage distribution grid with European layout. The
residential subnetwork is highlighted and was used for this work. Figure
adapted from [12].

###  II-B Data

To quantify how the different Volt/VAr methods behave, real data was used for
both household electricity consumption and PV generation. Among the limited
range of publicly available datasets suited for this purpose, Dataport by
Pecan Street Inc. [13] was identified as the best one according to the
following criteria: Large number of different households at one single
location with matching load and PV generation data, high temporal resolution
and a total duration which is sufficient to reveal seasonal effects. A more
detailed overview of all reviewed datasets can be found in [14]. The dataset
available under free academic licensing is from the year 2018, and contains
data from 25 different households in Austin, USA with a temporal resolution of
1 sec and a total duration of 1 year. It was downsampled to a resolution of 1
min to deal with the high cost of computing the power flows at each time step.
The houses available in Dataport were assigned to the 5 load buses of the
benchmark grid such that the peak demand and generation values were in
accordance with the nominal values stated by CIGRÉ. Fig. 2 shows the resulting
total load and generation at each bus for one exemplary day.

Figure 2: One exemplary day (July 2, 2018) from the resulting dataset, after
assigning the available Dataport data to the load buses in the CIGRÉ LV grid.

###  II-C Scenarios for PV integration

To investigate the impact of increasing power injections from DERs into
distribution grids, three scenarios have been designed which differ in the
amount of installed PV capacity. Starting from the base scenario given by the
actual PV data from the year 2018, PV capacity (both active and reactive power
resources) is increased by the factors 2 and 3.5, to create the PV integration
scenarios 2030 and 2035, respectively. These values agree with current average
predictions for the global increase in PV installations [15, 16, 17, 18].

##  III Reviewed Methods

This work aims at comparing three different controllers which are suited for
this purpose and are therefore benchmarked with the proposed simulation
framework. Particular attention is given to the importance of communication
for achieving optimal control and constraint satisfaction. The current state-
of-the-art, local droop control with standard parameter tuning, serves as the
lower baseline, whereas an ORPF defines the theoretical limit of what can be
achieved using reactive power resources for voltage control. Relative to these
two benchmarks, the performance of two recently proposed methods is evaluated.
On the one hand, a controller implemented according to [6] that utilizes a
data-driven procedure to learn optimal local droop curves, using the outputs
of the ORPF as training data. On the other hand, OFO which relies on full
communication but is essentially model-free and utilizes only voltage
magnitude measurements to steer the reactive power setpoints to the solution
of the ORPF problem.

###  III-A Droop Control

Droop control is a purely local control scheme in which every PV inverter is
required to follow a predefined droop curve that maps the bus voltage at its
own point of connection to a reactive power setpoint. Usually, these so-called
droop curves take the form of a piece-wise linear function as the one shown
below which is characterized by a deadband around the nominal voltage and
maximum injection/consumption of reactive power whenever the voltage
constraints are violated. In control language, droop control behaves like a
nonlinear P-controller, whose gain is given by the slope of the droop curve.

$\displaystyle q_{h}\\!=\\!\begin{cases}q_{h,\text{max}}&v_{h}\\!<\\!v_{1}\\\
q_{h,\text{max}}\dfrac{v_{2}\\!-\\!v_{h}}{v_{2}\\!-v_{1}}&v_{1}\\!\leq\\!v_{h}\\!\leq\\!v_{2}\\\
0&v_{2}\leq v_{h}\\!\leq\\!v_{3}\\\
q_{h,\text{min}}\dfrac{v_{h}\\!-\\!v_{3}}{v_{4}\\!-\\!v_{3}}&v_{3}\\!\leq\\!v_{h}\\!\leq\\!v_{4}\\\
q_{h,\text{min}}&v_{4}\\!<\\!v_{h}.\\\ \end{cases}$

###  III-B Optimal Reactive Power Flow

ORPF is the most obvious, and in theory ideal, candidate for an optimization-
based Volt/VAr controller. Assuming full communication, it centrally solves an
optimization problem to determine and communicate back the reactive power
setpoints for all PV inverters in the grid. However, it relies on perfect
knowledge of the grid topology, cable parameters and on measurements of the
active and reactive power consumption and production at each bus, all of which
are in practice not achievable. Furthermore, solving an ORPF problem is
computationally expensive, which can render it infeasible for real-time
settings.

Nevertheless, in the simulation framework a flawless model of the grid is
available, and an ORPF approach can be implemented. It represents the
theoretical limit of what can be achieved by any controller and will hence
serve as the upper benchmark. Its objective is to find the vector of reactive
power setpoints $q$ to solve the constrained optimization problem

$\displaystyle\min_{q}$ $\displaystyle\ \dfrac{1}{2}q^{T}q$ (1) s.t.
$\displaystyle\ v_{\text{min}}\leq v_{h}(q,w)\leq v_{\text{max}}\quad$
$\displaystyle\text{at all inverters}\;h$ $\displaystyle\ q_{\text{min}}\leq
q_{h}\leq q_{\text{max}}$ $\displaystyle\text{at all inverters}\;h.$

Hence, the ORPF approach attempts to keep all voltages within the constraints
while using a minimum amount of squared reactive power. If at some point the
combined reactive power capabilities of all PV inverters are insufficient to
achieve constraint satisfaction, the ORPF problem becomes infeasible. In that
case, the reactive power resources operate at the reactive power limit, and by
that keep the voltages as close to the constraints as possible. In practice, a
secondary control scheme such as the curtailment of active power would need to
become active under these circumstances. This is to ensure that the voltage
constraints are never violated but comes at the expense of a higher associated
cost which is why in this paper, we focus on reactive power compensation only.

###  III-C ML-tuned Droop

As an improved local controller, a data-driven droop control based on [6] has
been added to the comparison. The idea is to learn optimized droop curves by
using the setpoints generated by the ORPF approach as training inputs. In
principle, the resulting controller should then approximate the ORPF solution
through local control. The proposed algorithm consists of fitting a piece-wise
linear function to the training data, i.e. the Volt/VAr pairs generated by the
ORPF controller, for each inverter separately and offline. During operation,
the resulting mappings are used in a manner analogous to standard droop
control.

As the behavior of ORPF can be described as bang-bang, this procedure would
result in the deployment of droop curves with very steep slopes. However, it
has been shown that droop curves which are too steep lead to system
instability [19] and the set of allowed functions has hence been limited. To
still enable a larger negative slope and by this leave more freedom to the
algorithm when fitting the data, the output of the ML-tuned droop controller
has been low-pass filtered according to

$\displaystyle q_{h}(t)=(1-\beta)q_{h}(t-1)+\beta f_{h,\textrm{ml-
droop}}(v_{h}(t)),$ (2)

with $f_{h,\textrm{ml-droop}}$ being the learned, piecewise linear droop curve
for bus $h$ and with a smoothing factor $\beta=0.8$. Overall, the resulting
lower limit for the slope is the marginal value for which the closed-loop
system remains stable under application of the low-pass filter and has been
determined to be -4.5 VAr/V, both experimentally and theoretically according
to [19]. Furthermore, the droop curves are forced to exhibit only non-positive
slopes and to go through the points $(v_{min},q_{max})$ and
$(v_{max},q_{min})$. The latter ensures that the reactive power source is
utilizing its full reactive power when the measured voltage is outside the
limits.

The resulting droop curves are shown in Figure 3. Depending on the amount of
training data, the algorithm can become computationally intense, leading to
significant runtimes. However, once the curves have been determined offline,
the controller works just like standard droop and does not require any further
online computations. Nonetheless, if there are any changes to the system, the
complete procedure must be run again. The droop curves must be updated for
every reactive power source and under the premise of being a local control
method, there might not be communication in place to do so.

Figure 3: Training points for the ML-tuned droop and the optimized droop
curves.

###  III-D Online Feedback Optimization

OFO is a novel control method which can steer a system to the solution of a
constrained optimization problem. In the presented case, the ORPF problem (1)
is used as the steady-state specification for the OFO design. However, we
remark that compared to the ORPF controller, the resulting feedback law does
not require a detailed model of the system. While the ORPF approach typically
calculates the model-based solution in one step and operates the system in an
open-loop manner, OFO exploits voltage measurements to gradually update the
inputs towards the optimum. In this paper, we use the OFO controller from [7]
which employs a dual ascent strategy to deal with constraints. Two dual
variables per inverter integrate constraint violations at each time step
according to

$\displaystyle\lambda_{\textrm{min}}(t+1)$
$\displaystyle=[\lambda_{\textrm{min}}(t)+\alpha(v_{\textrm{min}}-v)]_{\geq
0}$ (3) $\displaystyle\lambda_{\textrm{max}}(t+1)$
$\displaystyle=[\lambda_{\textrm{max}}(t)+\alpha(v-v_{\textrm{max}})]_{\geq
0}.$ (4)

The new reactive power setpoints are then determined based on the current
values of the dual variables and a sensitivity matrix H as follows:

$\displaystyle\begin{split}q_{\textrm{unc}}&=H^{T}(\lambda_{\textrm{min}}(t+1)-\lambda_{\textrm{max}}(t+1))\\\
q(t+1)&=\arg\min_{q\in\mathcal{Q}}\;(q-q_{\textrm{unc}})^{T}(q-q_{\textrm{unc}}),\end{split}$
(5)

where $\mathcal{Q}=\\{q\;|\;q_{\textrm{min}}\leq q\leq q_{\textrm{max}}\\}$.
More specifically, the matrix H captures how the bus voltages change for a
change in reactive power at each bus. It represents a linearization of the
power flow equations and can be computed analytically based on an approximate
grid model [20], or experimentally. In practice, OFO controllers are typically
robust against an inaccurate choice of H. It may be determined for a nominal
and known operating point such as the one without any load and generation as
was done in the presented case. However, it has been shown that the algorithm
performs well even ignoring any prior information on the grid topology (that
is, setting H equal to the identity matrix) [7].

The OFO algorithm is further characterized by the step size $\alpha$ which
determines the size of the dual ascent update and functions as an integral
control gain. Larger values of $\alpha$ increase the convergence speed whereas
too large values can render the system unstable. Overall, this yields a
tradeoff and tuning the step size is hence required to obtain good
performance. However, given that it is a single scalar value, this is not a
very challenging task in practice. In this work, $\alpha$ has been chosen to
be 4.0. Finally, we note that OFO is computationally cheap and can hence be
run at small time intervals and on any low power microcontroller.

##  IV Numerical study on
Virtual Grid Reinforcement

This section presents the results of the numerical study. First, the
characteristic behavior of the different control methods is illustrated based
on intraday simulations of a particular summer day in the 2035 scenario.
Subsequently, the performance of the controllers is compared across the three
PV integration scenarios, pointing out the importance of improving the
optimality of Volt/VAr control. Finally, a different perspective on the matter
is presented: We quantify by how much the PV generation fed into distribution
grids can be increased by applying each of the reviewed Volt/VAr control
methods. This is to reveal the economic impact that optimal Volt/VAr control
can have for grid operators.

###  IV-A Controller Characteristics

In 2035, the active PV power fed into the grid is expected to reach levels at
which overvoltages occur frequently, despite the larger availability of
reactive power. In fact, none of the proposed Volt/VAr controllers is able to
keep the voltages within bounds at all times. However, the amount of
constraint violations varies drastically for different controllers. This makes
this scenario well-suited to observe how the different methods cope with
voltages approaching and exceeding the constraints and at the same time, shows
the future need for better Volt/VAr control methods.

Figure 4 shows the intra-day evolution of the voltage and reactive power
injection at each bus. Negative values of the latter correspond to consumption
of reactive power by the PV inverters. The amounts of reactive power which are
injected or absorbed at each bus are commanded by droop control (Figure 4(b)),
ML-tuned droop control (Figure 4(c)), OFO (Figure 4(d)) and ORPF control
(Figure 4(a)), respectively.

(a) ORPF

(b) Droop

(c) ML-tuned droop

(d) OFO

Figure 4: Voltages and reactive power injections throughout a sunny summer day
for the 2035 scenario for each of the considered controllers.

The irradiation and active power consumption data is from July 2, 2018 and
represents a typical warm and sunny day. This is the data shown in Figure 2,
except that PV generation was scaled by 3.5 in the 2035 scenario. Around noon,
the irradiance peak leads to high PV active power production, causing
overvoltages to occur across the grid. In the evening, high power consumption
and low irradiance give rise to marginal undervoltages. In the following, we
will focus on the controller behavior with respect to overvoltages.

Figure 4(a) shows the results for the ideal ORPF approach which corresponds to
the best performance achievable by any Volt/VAr controller under perfect
knowledge of the grid model and all active and reactive consumption and
generation. This is unachievable in practice, rendering the ORPF approach an
idealized upper benchmark for the other control methods. The ORPF approach
solves problem (1) at each timestep and hence uses the reactive power
resources optimally. When no constraints are violated, no reactive power is
used. Once the voltages exceed the constraints, the ORPF approach uses the
minimum amount of reactive power that is required to keep them right at the
constraints. Once the optimization problem becomes infeasible, i.e. constraint
violations become unavoidable, OPF control uses the full reactive power
resources to keep the voltages as close to the constraints as possible.

Standard droop control applies control inputs based solely on the local bus
voltage. Maximum reactive power is used only if the local voltage constraints
at a bus are violated. However, the inverters which are connected to a bus
that is close to the external grid do never experience such high voltages.
Hence, they do not absorb much reactive power even though there might be
overvoltages occurring further into the grid. Thus, constraint violations
persist even if problem (1) is feasible, meaning the voltage violation could
be mitigated using other reactive power sources differently. Additionally,
some reactive power is used unnecessarily at times when all voltages are
within bounds and would have been admissible using lower or even no control
effort.

Using ML-tuned doop control, the local controllers aim to emulate the global
behavior of ORPF but there is still no coordination between the participants.
By adapting the local droop curves based on the inputs that an ideal ORPF
approach would apply, the inverters close to the external grid are instructed
to contribute slightly more than under standard droop control. Furthermore,
the waste of reactive power at uncritical times is smaller compared to
standard droop control. However, ML-tuned droop is not able to come close to
the optimality of the ORPF approach whose behavior it tries to imitate.
Rather, its performance with respect to the duration and size of constraint
violations is not significantly better than the one of standard droop.

Lastly, the OFO controller exhibits a performance which is comparable to the
one of the ideal ORPF approach, as shown in Figure 4(d). Importantly, it
achieves this by using only the sensitivity matrix $H$ and voltage magnitude
measurements. Active and reactive power consumption and generation do not need
not be measured. As with the ORPF approach, the control inputs are
coordinated, yielding near-optimal performance. Due to being an integral-like
controller, the closed-loop convergence of OFO is not immediate, as constraint
violations must be integrated for at least one time-step to initiate an update
of the set points. Hence, high-frequency changes in the grid can cause
temporary constraint violations until OFO has converged and then satisfies all
constraints. In our experiments, the OFO controller was operated every 10
seconds, yielding convergence after 1 minute at most.

###  IV-B Overall Controller Performance

This section analyzes the performance of the controllers over a complete
yearly cycle. For this purpose, all the available data is used, corresponding
to a full year of household power consumption and PV power production. This is
to reveal the actual implications which different Volt/VAr control strategies
have for grid operation. Figure 5 summarizes the results. In the 2020
scenario, corresponding to the PV capacity which is present in distribution
grids today, no significant overvoltages occur over the entire year. This
indicates why local droop control, currently the state-of-the-art Volt/VAr
control method, has been sufficient until now. However, as PV capacity
increases towards the 2030 and 2035 scenarios, the size and duration of
constraint violations increase drastically. Even though the reactive power
capacities and hence the control capabilities increase by the same factor as
active power production, they no longer suffice to mitigate the overvoltages.
This shows the importance of moving toward better Volt/VAr control methods in
the future and to use the available reactive power resources as effectively as
possible.

(a) Results for the 2020 scenario.

(b) Results for the 2030 scenario.

(c) Results for the 2035 scenario.

Figure 5: Visualization of the yearly reactive power usage, voltage
distributions, and constraint violations for droop, ML-tuned droop, OFO, and
ORPF for all three scenarios.

Standard droop control yields the worst performance in all three scenarios,
allowing the most constraint violations while using the largest amount of
reactive energy over the yearly period. ML-tuned droop achieves slightly more
efficient control with respect to the reactive energy used, but allows about
the same amount of constraint violations as standard droop control.

The proposed OFO controller outperforms both local droop controllers
significantly in terms of both control effectiveness and efficiency. It
achieves near-optimal constraint satisfaction that is almost equal to the one
of the ideal ORPF approach.

###  IV-C Virtual Grid Reinforcement

In this section, we approach the analysis of the different Volt/VAr
controllers from a different perspective. The goal is to evaluate for each of
the three controllers, how much power can be fed into the grid without
exceeding the upper voltage limit at any bus. Active power production is
assumed to be equal at all load buses and increased linearly until the
critical point and beyond. Figure 6 presents the results. In the absence of
any Volt/VAr control, the first overvoltages occur at a maximum infeed of 407
kW. Using local droop control (both standard or ML-tuned) the critical amount
is about 490 kW. Both the ideal ORPF approach and OFO are able to increase the
infeed limit to 535 kW. Hence, OFO optimally virtually reinforces the grid.
Compared to local droop control, OFO increases the maximum grid capacity by 45
kW or more than 9%. This emphasizes the potential economic value of
establishing communication to achieve better Volt/VAr control in distribution
grids.

Figure 6: Simulation-based virtual reinforcement capabilities of droop, ML-
tuned droop, OFO, and ORPF.

##  V Experimental Results on
Virtual Grid Reinforcement

The virtual reinforcement capabilities of droop control and OFO were also
experimentally tested on a distribution grid in Roskilde, Denmark. A detailed
description of the experimental setup and the OFO implementation can be found
in [7]. In short, an active power source was connected at the end of a long
feeder, see Figure 7.

Figure 7: Topology of the real distribution grid feeder used for the
experimental validation. The battery acts as renewable generation, PV2 is able
to provide reactive power injections, and PV1 and a static load mimic an
electric vehicle. The figure is taken from [7].

Its active power injections lead to a voltage increase that limits the active
power that can be injected. This power source and two reactive power sources
along the feeder perform Volt/VAr control to regulate the voltage. Figure 8
shows the voltage at the end of the feeder for different active power
injections. The lines correspond to the different control methods used for the
Volt/VAr control. The Figure shows that without any Volt/VAr control, the
voltage limit of 1.05 p.u. is reached at 8.77 kW. Using droop control, the
limit is reached at 10.16 kW. When OFO is used for control, the grid can
virtually be reinforced to 11.23 kW which is another 10.5% on top of the
capacity achieved using droop control. Finally, we note that the pink line,
corresponding to the ideal ORPF approach, violates the voltage constraint.
This is due to a model mismatch that occurred even though the cable types and
parameters were known and all generation and consumption was measured. This
showcases that feed-forward methods fail to enforce constraints in real
systems even under excellent model knowledge. Robust optimization methods can
be used to guarantee constraint satisfaction but they would not utilize the
resources to the full extent as they must leave some slack for any potential
model mismatch. In contrast, OFO guarantees constraint satisfaction even under
model mismatch and virtually reinforces the grid to its full potential using
only limited model information.

02468NCDPOFO$1$$1.02$$1.04$$1.06$$1.08$Active Power Injection [kW]Voltage
Magnitude [p.u.]ORPF approachDroop (DP)OFO controllerNo control (NC)

Figure 8: Experimental results of virtual reinforcement through Volt/VAr
control on a real distribution grid feeder.

##  VI Conclusion

The results of the experiment and simulations show that droop control, as
currently used in many grid codes, helps to control the voltage. This enables
more power to be transmitted before voltage constraints are violated. However,
reactive power is often used even if the voltages are within their limits
leading to unnecessary losses. The simulations further reveal that the grid
can be virtually reinforced by another 9% when communication is added.
Furthermore, we showed that OFO can reach this maximum level of virtual
reinforcement using only minimal model information, voltage magnitude
measurements and communication channels for these measurements. Also, OFO only
uses reactive power when really needed helping to minimize losses. Finally, we
demonstrated OFO’s practical viability on a real distribution grid feeder.

Overall, our analysis suggests that OFO can extend the capacity of a
distribution grid in the range of 10% beyond the state-of-the-art which would
enable distribution grid operators to mitigate or postpone physical grid
reinforcements that will be required in the future.

## References

  * [1] B. Höflich, P. Richard, J. Völker, C. Rehtanz, M. Greve, B. Gwisdorf, J. Kays, T. Noll, J. Schwippe, A. Seack _et al._ , “dena-verteilnetzstudie ausbau-und innovationsbedarf der stromverteilnetze in deutschland bis 2030,” _Deutsche Energie-Agentur: Berlin, Germany_ , 2012\.
  * [2] IEEE 1547-2018, “Standard for interconnection and interoperability of distributed energy resources with associated electric power systems interfaces,” Tech. Rep., 2018.
  * [3] “VDE-AR-N 4105: Generators connected to the LV distribution network – Technical requirements for the connection to and parallel operation with low-voltage distribution networks,” 2018.
  * [4] “Commission Regulation (EU) 2016/631 of 14 April 2016 establishing a network code on requirements for grid connection of generators,” _OJ_ , vol. L112, pp. 1–68, 2016.
  * [5] S. Bolognani, R. Carli, G. Cavraro, and S. Zampieri, “On the need for communication for voltage regulation of power distribution grids,” _IEEE Trans. Control Netw. Syst._ , vol. 6, no. 3, pp. 1111–1123, 2019\.
  * [6] S. Karagiannopoulos, P. Aristidou, and G. Hug, “Data-driven local control design for active distribution grids using off-line optimal power flow and machine learning techniques,” _IEEE Transactions on Smart Grid_ , vol. 10, no. 6, pp. 6461–6471, 2019.
  * [7] L. Ortmann, A. Hauswirth, I. Caduff, F. Dörfler, and S. Bolognani, “Experimental validation of feedback optimization in power distribution grids,” _Electric Power Systems Research_ , vol. 189, p. 106782, 2020.
  * [8] L. Ortmann, A. Prostejovsky, K. Heussen, and S. Bolognani, “Fully distributed peer-to-peer optimal voltage control with minimal model requirements,” _Electric Power Systems Research_ , vol. 189, p. 106717, 2020.
  * [9] B. Kroposki, A. Bernstein, J. King, and F. Ding, “Good grids make good neighbors,” _IEEE Spectrum_ , vol. 57, no. NREL/JA-5D00-78521, 2020.
  * [10] L. Thurner, A. Scheidler, F. Schäfer, J.-H. Menke, J. Dollichon, F. Meier, S. Meinecke, and M. Braun, “pandapower - an open source python tool for convenient modeling, analysis and optimization of electric power systems,” vol. 33, no. 6, pp. 6510–6521, 2018.
  * [11] J. Matt, “Gitlab repository,” https://gitlab.ethz.ch/ortmannl/code-for-publication-virtual-power-grid-reinforcement-via-coordinated-voltvar-control, 2023\.
  * [12] _Benchmark systems for network integration of renewable and distributed energy resources_. CIGRÉ, 2014.
  * [13] O. Parson, G. Fisher, A. Hersey, N. Batra, J. Kelly, A. Singh, W. Knottenbelt, and A. Rogers, “Dataport and NILMTK: A building data set designed for non-intrusive load monitoring,” in _IEEE Global Conference on Signal and Information Processing (GlobalSIP)_ , 2015, pp. 210–214.
  * [14] J. G. Matt, “Online feedback optimization for minimal curtailment of renewable generation with real data,” Bachelor Thesis, ETH Zurich, Zurich, 2021.
  * [15] “Renewable energy statistics 2022,” The International Renewable Energy Agency, Tech. Rep., 2022.
  * [16] “World energy outlook 2021,” IEA, Tech. Rep., 2021. [Online]. Available: https://www.iea.org/reports/world-energy-outlook-2021
  * [17] S. Philipps and W. Warmuth, “Photovoltaics report,” Fraunhofer Institute for Solar Energy Systems, ISE, Tech. Rep., 2020.
  * [18] J. Moore and N. Bullard, “Executive factbook - power, transport, buildings and industry, commodities, food and agriculture, capital,” BloombergNEF, Tech. Rep., 2021.
  * [19] A. Eggli, S. Karagiannopoulos, S. Bolognani, and G. Hug, “Stability analysis and design of local control schemes in active distribution grids,” pp. 1–1, 2020\.
  * [20] S. Bolognani and F. Dörfler, “Fast power system analysis via implicit linearization of the power flow manifold,” in _53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton)_. IEEE, 2015, pp. 402–409.

  *[PV]: photovoltaics
  *[OFO]: Online Feedback Optimization
  *[ORPF]: Optimal Reactive Power Flow
  *[ML]: machine learning