11institutetext: Institute of Autonomous Systems Technology, University of the
Bundeswehr Munich 11email<EMAIL_ADDRESS>

# YOLOPoint: Joint Keypoint and Object Detection††thanks: This research is
funded by dtec.bw – Digitalization and Technology Research Center of the
Bundeswehr. dtec.bw is funded by the European Union - NextGenerationEU.

Anton Backhaus  Thorsten Luettel  Hans-Joachim Wuensche

###### Abstract

Intelligent vehicles of the future must be capable of understanding and
navigating safely through their surroundings. Camera-based vehicle systems can
use keypoints as well as objects as low- and high-level landmarks for GNSS-
independent SLAM and visual odometry. To this end we propose YOLOPoint, a
convolutional neural network model that simultaneously detects keypoints and
objects in an image by combining YOLOv5 and SuperPoint to create a single
forward-pass network that is both real-time capable and accurate. By using a
shared backbone and a light-weight network structure, YOLOPoint is able to
perform competitively on both the HPatches and KITTI benchmarks.

###### Keywords:

Deep learning Keypoint detection Autonomous driving

##  1 Introduction

Keypoints are low-level landmarks, typically points, corners, or edges that
can easily be retrieved from different viewpoints. This makes it possible for
moving vehicles to estimate their position and orientation relative to their
surroundings and even perform loop closure (i.e., SLAM) with one or more
cameras. Historically, this task was performed with hand-crafted keypoint
feature descriptors such as ORB [17], SURF [2], HOG [5], SIFT [14]. However,
these are either not real-time capable or perform poorly under disturbances
such as illumination changes, motion blur, or they detect keypoints in
clusters rather than spread out over the image, making pose estimation less
accurate. Learned feature descriptors aim to tackle these problems, often by
applying data augmentation in the form of random brightness, blurring and
contrast. Furthermore, learned keypoint descriptors have shown to outperform
classical descriptors. One such keypoint descriptor is SuperPoint [6], a
convolutional neural network (CNN) which we use as a base network to improve
upon.

SuperPoint is a multi-task network that jointly predicts keypoints and their
respective descriptors in a single forward pass. It does this by sharing the
feature outputs of one backbone between a keypoint detector and descriptor
head. This makes it computationally efficient and hence ideal for real-time
applications.

Furthermore, after making various adjustments to the SuperPoint architecture,
we fuse it with YOLOv5 [11], a real-time object detection network. The full
network thus uses one shared backbone for all three tasks. Moreover, using the
YOLOv5 framework we train four models of various sizes: nano, small, middle
and large. We call the combined network YOLOPoint.

Our motivation for fusing the keypoint detector with the object detector is as
follows: Firstly, for a more accurate SLAM and a better overview of the
surrounding scene, using a suite of multi-directional cameras is beneficial.
But processing multiple video streams in parallel means relying on efficient
CNNs. In terms of computational efficiency, the feature extraction part does
most of the heavy lifting. This is why it has become common practice for
multiple different tasks to share a backbone. SuperPoint’s architecture is
already such that is uses a shared backbone for the keypoint detection and
description, hence it was an adequate choice for fusion with another model.
Secondly, visual SLAM works on the assumption of a static environment. If
keypoints are detected on moving objects, it would lead to localization errors
and mapping of unwanted landmarks. However, with classified object bounding
boxes we simply filter out all keypoints within a box of a dynamic class
(e.g., pedestrian, car, cyclist). Finally, although it seems that keypoint and
object detection are too different to jointly learn, they have been used in
conjunction in classical methods (e.g., object detection based on keypoint
descriptors with a support vector machine classifier [27]).

Figure 1: Example output of YOLOPointM on a KITTI scene with keypoint tracks
from 3 frames and object bounding boxes.

Our contributions can be summarized as follows:

  * •

We propose YOLOPoint, a fast and efficient keypoint detector and descriptor
network which is particularly well suited for performing visual odometry.

  * •

We show that object and keypoint detection are not mutually exclusive and
propose a network that can do both in a single forward pass.

  * •

We demonstrate the efficacy of using efficient cross stage partial network
(CSP) blocks [25] for point description and detection tasks.

Code will be made available at https://github.com/unibwtas/yolopoint.

##  2 Related Work

Classical keypoint descriptors use hand-crafted algorithms designed to
overcome challenges such as scale and illumination variation [20, 17, 2, 14,
5, 22] and have been thoroughly evaluated [16, 21]. Although their main
utility was keypoint description, they have also been used in combination with
support vector machines to detect objects [27].

Since then, deep learning-based methods have dominated benchmarks for object
detection (i.e., object localization and classification) and other computer
vision tasks [27, 26, 28]. Therefore, research has increasingly been dedicated
to finding ways in which they can also be used for point description. Both
using CNN-based transformer models COTR [10] and LoFTR [24] achieve state-of-
the-art results on multiple image matching benchmarks. COTR finds dense
correspondences between two images by concatenating their respective feature
maps and processing them together with positional encodings in a transformer
module. Their method, however, focuses on matching accuracy rather than speed
and performs inference on multiple zooms of the same image. LoFTR has a
similar approach, the main difference being their ”coarse-to-fine” module that
first predicts coarse correspondences, then refines them using a crop from a
higher level feature map. Both methods are detector-free, and while achieving
excellent results in terms of matching accuracy, neither method is suitable
for real-time applications. Methods that yield both keypoint detections and
descriptors are generally faster due to matching only a sparse set of points
and include R2D2 [19], D2-Net [7] and SuperPoint [6]. R2D2 tackles the
matching problems of repetitive textures by introducing a reliability
prediction head that indicates the discriminativeness of a descriptor at a
given pixel. D2-Net has the unique approach of using the output feature map
both for point detection and description, hence sharing all the weights of the
network between both tasks. In contrast, SuperPoint has a shared backbone but
seperate heads for the detection and description task. What sets it apart from
all other projects is its self-supervised training framework. While other
authors obtain ground truth point correspondences from depth images gained
from structure from motion, i.e., using video, the SuperPoint framework can
create labels for single images. It does this by first generating a series of
labeled grayscale images depicting random shapes, then training an
intermediate model on this synthetic data. The intermediate model subsequently
makes point-predictions on a large data set (here: MS COCO [13]) that are
refined using their ”homographic adaptation” method. The final model is
trained on the refined point labels.

While there exist several models that jointly predict keypoints and
descriptors, there are to our knowledge none that also detect objects in the
same network. Maji et al’s work [15] comes closest to ours. They use YOLOv5 to
jointly predict keypoints for human pose estimation as well as bounding boxes
in a single forward pass. The main differences are that the keypoint detection
training uses hand-labelled ground truth points, the object detector is
trained on a single class (human), and both tasks rely on similar features.

##  3 Model Architecture

Figure 2:  Full model architecture exemplary for YOLOPointS. The two types of
bottlenecks, C3 block (left) and a sequence of convolution, batch
normalization and SiLU activation form the main parts of YOLO and by extension
YOLOPoint. $k$: kernel size, $s$: stride, $p$: pad, $c$: output channels,
$bn$: bottleneck, _SPPF_ : fast spacial pyramid pooling [11].

Our keypoint detection model is an adaptation of SuperPoint with CSPDarknet
[25] elements. SuperPoint uses a VGG-like encoder [23] that breaks down a
grayscale image of size $H\times W\times 1$ into a feature vector of size
$H/8\times W/8\times 128$. In each decoder head the feature vector is further
processed and reshaped to the original image size. The keypoint decoder branch
thus produces a heat map representing the pixel-wise probability of ”point-
ness”, which is passed through a non-maximum suppression method that produces
the final keypoints. The output of the descriptor branch is a normed
descriptor vector linearly up-scaled to $H\times W\times 256$. Both outputs
are combined into a list of keypoints and corresponding descriptor vectors,
with which keypoints can be matched from frame to frame.

Our version of SuperPoint substitutes the VGG-like backbone with part of
CSPDarknet and has a CSP bottleneck with additional convolutions in each head.
CSP bottlenecks are the cornerstone of YOLO and provide a good speed to
accuracy trade-off for various different computer vision tasks but have so far
not been used in keypoint detection for visual landmarks. We keep the same
final layers as SuperPoint, i.e., softmax and reshape in the detector head and
2D convolution in the descriptor head. Using YOLOv5’s scaling method we create
four networks, each varying in width, depth and descriptor length. The
descriptors $\mathcal{D}$ are sized at 256, 196, 128 and 64 for models
YOLOPointL, -M, -S and -N respectively. A shorter descriptor reduces the
computational cost of matching at the cost of accuracy. For comparison: SIFT’s
descriptor vector has 128 elements and ORB’s has only 32. Since reducing the
vector size comes at an accuracy trade-off, the smallest vector is left at 64.
Furthermore, in order for the descriptor to be able to match and distinguish
between other keypoints, it requires a large receptive field [24]. This,
however, comes with down-sampling the input image and loosing detail in the
process. In order to preserve detail, we enlarge the low-resolution feature
map with nearest-neighbor interpolation and concatenate it with a feature map
higher up in the backbone before performing further convolutions. The full
model fused with YOLOv5 is depicted in fig. 2.

##  4 Training

To generate pseudo ground truth point labels, we follow the protocol of
SuperPoint by first training the point detector of YOLOPoint on the synthetic
shapes dataset, then using it to generate refined outputs on COCO using
homographic adaptation for pre-training. Pre-training on COCO is not strictly
necessary, however it can improve results when fine-tuning on smaller data
sets, as well as reduce training time. Thus, the pre-trained weights are later
fine-tuned on the KITTI dataset [8]

For training the full model, pairs of RGB images that are warped by a known
homography are each run through a separate forward pass. The model
subsequently predicts “point-ness” heat maps, descriptor vectors and object
bounding boxes. When training on data sets of variable image size (e.g., MS
COCO) the images must be fit to a fixed size in order to be processed as a
batch. A common solution is to pad the sides of the image such that $W=H$,
also known as letterboxing. However, we found that this causes false positive
keypoints to be predicted close to the padding due to the strong contrast
between the black padding and the image, negatively impacting training.
Therefore, when pre-training on COCO, we use mosaic augmentation, that
concatenates four images side-by-side to fill out the entire image canvas,
eliminating the need for image padding [4]. All training is done using a batch
size of 64 and the Adam optimizer [12] with a learning rate of $10^{-3}$ for
pre-training and $10^{-4}$ for fine-tuning.

For fine-tuning on KITTI we split the data into 6481 training and 1000
validation images resized to $288\times 960$. To accommodate the new object
classes we replace the final object detection layer and train for 20 epochs
with all weights frozen except those of the detection layer. Finally, we
unfreeze all layers and train the entire network for another 50 epochs.

###  4.1 Loss Functions

The model outputs for the warped and unwarped image are used to calculate the
keypoint detector and descriptor losses. However, only the output of the
unwarped images is used for the object detector loss, as to not teach strongly
distorted object representations.

The keypoint detector loss $\mathcal{L}_{\text{det}}$ is the mean of the
binary cross-entropy losses over all pixels of the heatmaps of the warped and
unwarped image of size $H\times W$ and corresponding ground truth labels and
can be expressed as follows:

$\mathcal{L}_{\text{det}}=-\frac{1}{HW}\sum^{H,W}(y_{ij}\cdot\log
x_{ij}+(1-y_{ij})\cdot\log(1-x_{ij}))$ (1)

where $y_{ij}\in\\{0,1\\}$ and $x_{ij}\in[0,1]$ respectively denote the target
and prediction at pixel $ij$.

The original descriptor loss is a contrastive hinge loss applied to all
correspondences and non-correspondences of a low-resolution descriptor
$\mathcal{D}$ of size $H_{c}\times W_{c}$ [6] creating a total of
$(H_{c}\times W_{c})^{2}$ matches. This, however, becomes computationally
unfeasible for higher resolution images. Instead, we opt for a sparse version
adapted from DeepFEPE [9] that samples $N$ matching pairs of feature vectors
$d_{ijk}$ and $d_{i^{\prime}j^{\prime}k}$ and $M$ non-matching pairs of a
batch of sampled descriptors $\tilde{\mathcal{D}}\subset\mathcal{D}$ and their
warped counterpart $\tilde{\mathcal{D}^{\prime}}\subset\mathcal{D}^{\prime}$.
Using the known homography, each descriptor $\mathbf{d}$ of pixel $ij$ of the
$k$th image of a mini-batch can be mapped to its corresponding warped pair at
$i^{\prime}j^{\prime}$. $m_{p}$ furthermore denotes the positive margin of the
sampled hinge loss.

$\mathcal{L}_{\text{desc}}=\mathcal{L}_{\text{corr}}+\mathcal{L}_{\text{n.corr}}$
(2)

where

$\mathcal{L}_{corr}=\frac{1}{N}\sum_{i,j,k,i^{\prime},j^{\prime}}^{\tilde{\mathcal{D}},\tilde{\mathcal{D^{\prime}}}}\max(0,m_{p}-\mathbf{d}_{ijk}^{T}\mathbf{d}_{i^{\prime}j^{\prime}k}^{\prime})$
(3)
$\mathcal{L}_{n.corr}=\frac{1}{M}\sum_{i,j,k}^{\tilde{\mathcal{D}}}\sum_{o,p,q}^{\tilde{\mathcal{D^{\prime}}}}(\mathbf{d}_{ijk}^{T}\mathbf{d^{\prime}}_{opq}),\quad(o,p,q)\neq(i^{\prime},j^{\prime},k)$
(4)

The object detector loss $\mathcal{L}_{\text{obj}}$ is a linear combination of
intermittent losses based on the objectness, class probability and bounding
box regression score and is identical to the loss function used in YOLOv5.

The final loss is calculated as the weighted sum of the keypoint detector,
descriptor and object loss.

$\mathcal{L}=\mathcal{L}_{\text{det}}+\mathcal{L}_{\text{det,warp}}+w_{\text{desc}}\cdot\mathcal{L}_{\text{desc}}+w_{\text{obj}}\cdot\mathcal{L}_{\text{obj}}$
(5)

##  5 Evaluation

Figure 3: HPatches matches between two images with viewpoint change estimated
with YOLOPointS. Matched keypoints are used to estimate the homography matrix
describing the viewpoint change.

In the following sections we present our evaluation results for keypoint
detection and description on HPatches [1] and using all three task heads for
visual odometry (VO) estimation on the KITTI benchmark. For evaluation on
HPatches the models trained for 100 epochs on MS COCO are used, for VO the
models are fine-tuned on KITTI data.

###  5.1 Repeatability and Matching on HPatches

The HPatches dataset comprises a total of 116 scenes, each containing 6
images. 57 scenes exhibit large illumination changes and 59 scenes large
viewpoint changes. The two main metrics used for evaluating keypoint tasks are
repeatability which quantifies the keypoint detector’s ability to consistently
locate keypoints at the same locations despite illumination and/or viewpoint
changes and homography estimation which tests both repeatability and
discrimination ability of the detector _and_ descriptor. Our evaluation
protocols are kept consistent with SuperPoint’s where possible.

####  5.1.1 Repeatability

Repeatability is computed at $256\times 320$ resolution with up to 300 points
detected per image using a non-maximum suppression of 8 pixels. A keypoint
counts as repeated if it is detected within $\epsilon=3$ pixels in both
frames. The repeatability score determines the ratio of repeated keypoints to
overall detected keypoints [6]. Table 1 summarizes the repeatability scores
under viewpoint and illumination changes. SuperPoint outperforms YOLOPoint in
both illumination and viewpoint changes by a small margin. Surprisingly, all
four versions of YOLOPoint perform virtually the same, despite varying
strongly in number of parameters. A likely explanation for this is that the
keypoint detector branch has more parameters than it actually needs with
regard to the simplicity of the task.

Table 1:  Repeatability score on HPatches in scenes of strong illumination and viewpoint changes. SuperPoint outperforms YOLOPoint by a small margin in both scenarios. Best results are boldfaced.  | Repeatability
---|---
Illumination | Viewpoint
YOLOPointL | .590 | .540
YOLOPointM | .590 | .540
YOLOPointS | .590 | .540
YOLOPointN | .589 | .529
SuperPoint | .611 | .555

####  5.1.2 Homography Estimation

Homography estimation is computed at $320\times 480$ resolution with up to
1000 points detected per image using a non-maximum suppression of 8 pixels. By
using matched points between two frames (cf. fig. 3) a homography matrix that
describes the transformation of points between both frames is estimated. The
estimated homography is then used to transform the corners of one image to
another. A homography counts as correct if the l2 norm of the distance of the
corners lies within a margin $\epsilon$ [6].

Table 2 shows the results of the homography estimation on the viewpoint
variation scenes. Overall, both models are roughly on par, although SuperPoint
yields slightly better results within larger error margins $\epsilon$.
SuperPoint’s descriptors have a significantly higher nearest neighbor mean
average precision (NN mAP), indicating good discriminatory ability, however a
worse matching score than YOLOPoint. The matching score is a measure for both
detector and descriptor, as it measures the ratio of ground truth
correspondences over the number of proposed features within a shared viewpoint
region [6].

Table 2: Homography estimation on HPatches. YOLOPoint and SuperPoint perform comparably to each other. Additional descriptor metrics are included. Best results are boldfaced. |  | Homography
---
estimation
| Descriptor
---
metrics
$\epsilon=1$ | $\epsilon=3$ | $\epsilon=5$ | NN mAP |  | Matching
---
Score
YOLOPointL | .390 | .739 | .841 | .650 | .459
YOLOPointM | .410 | .712 | .817 | .625 | .449
YOLOPointS | .420 | .729 | .817 | .592 | .436
YOLOPointN | .292 | .668 | .803 | .547 | .389
SuperPoint | .312 | .742 | .851 | .756 | .409

###  5.2 Visual Odometry Estimation on KITTI

Figure 4: Translation and rotation RMSE over all KITTI sequences plotted
against mean VO estimation time for YOLOPointL (YPL), M, S and N with filtered
points as well as SuperPoint and classical methods for comparison (lower left
is better). VO estimation was done with $376\times 1241$ images, NVIDIA RTX
A4000 and Intel Core i7-11700K.

The KITTI odometry benchmark contains 11 image sequences of traffic scenes
with publicly available ground truth camera trajectories captured from a
moving vehicle from the ego perspective.

Odometry is estimated using only frame-to-frame keypoint tracking (i.e., no
loop closure, bundle adjustment, etc.), in order to get an undistorted
evaluation of the keypoint detectors/descriptors. In our tests we evaluate
different versions of YOLOPoint with filtering out keypoints on dynamic
objects using object bounding boxes and compare them to SuperPoint and other
real-time classical methods. Fig. 4 plots the RMSE over all sequences against
the mean VO estimation time (i.e., keypoint detection, matching and pose
estimation) for several models. Despite the overhead of detecting objects,
YOLOPoint provides a good speed-to-accuracy trade-off with a full iteration
taking less than $100\text{\,}\mathrm{ms}$. The pure inference times of
YOLOPoint (i.e. without pose estimation) are 49, 36, 27 and
$25\text{\,}\mathrm{ms}$ for L, M, S and N, respectively. Note that the VO
time using YOLOPointN is longer despite having a faster inference since its
keypoints have more outliers and noisier correspondences. Consequently, the
pose estimation algorithm needs more iteration steps to converge, resulting in
a slower VO.

Figure 5: Sequence 01: Driving next to a car on a highway. Top: All keypoints.
Bottom: Keypoints on car removed via its bounding box. Figure 6: Sequence 01
trajectories of YOLOPointM (YPM) using filtered and unfiltered points and
other keypoint detectors for comparison.

The efficacy of filtering out dynamic keypoints can best be demonstrated on
sequence 01 as it features a drive on a highway strip where feature points are
sparse and repetitive, i.e., hard to match. Consequently, points detected on
passing vehicles are no longer outweighed by static points (fig. 5), causing
significant pose estimation error. Fig. 6 shows some trajectories including
YOLOPointM with and without filtered points. Without filtering, YOLOPoint
performs comparatively poorly on this sequence due to it detecting many points
on passing vehicles. Finally, although all points on vehicles are removed,
regardless of whether or not they are in motion, we do not find that this has
a significant negative impact on pose estimation accuracy in scenes featuring
many parked cars.

##  6 Conclusion and Future Work

In this work, we propose YOLOPoint, a convolutional neural network that
jointly predicts keypoints, their respective descriptors and object bounding
boxes fast and efficiently in a single forward pass, making it particularly
suitable for real-time applications such as autonomous driving. Our tests show
that the keypoint detector heads that use a Darknet-like architecture perform
similarly to SuperPoint in matching and repeatability tasks featuring strong
view-point and illumination changes. On the KITTI visual odometry dataset we
compare our model’s pose estimation performance to SuperPoint and some
classical methods. Using predicted bounding boxes to filter out non-static
keypoints, YOLOPoint shows the best accuracy to speed trade-off of all tested
methods.

Future work will concentrate on incorporating YOLOPoint into our SLAM
framework [3] by using keypoints and static objects as landmarks and
increasing the robustness of object tracking [18] by matching keypoints.

## References

  * [1] Balntas, V., Lenc, K., Vedaldi, A., Mikolajczyk, K.: Hpatches: A benchmark and evaluation of handcrafted and learned local descriptors. In: CVPR (2017)
  * [2] Bay, H., Tuytelaars, T., Van Gool, L.: Surf: Speeded up robust features. pp. 404–417. Springer Berlin Heidelberg, Berlin, Heidelberg (2006)
  * [3] Beer, L., Luettel, T., Wuensche, H.J.: GenPa-SLAM: Using a General Panoptic Segmentation for a Real-Time Semantic Landmark SLAM. In: Proceedings of IEEE Intelligent Transportation Systems Conference (ITSC). pp. 873–879. Macau, China (Oct 2022). https://doi.org/10.1109/ITSC55140.2022.9921983, ©IEEE
  * [4] Bochkovskiy, A., Wang, C., Liao, H.M.: Yolov4: Optimal speed and accuracy of object detection. CoRR abs/2004.10934 (2020)
  * [5] Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. vol. 1, pp. 886–893 vol. 1 (2005). https://doi.org/10.1109/CVPR.2005.177
  * [6] DeTone, D., Malisiewicz, T., Rabinovich, A.: SuperPoint: Self-Supervised Interest Point Detection and Description (2018), http://arxiv.org/abs/1712.07629
  * [7] Dusmanu, M., Rocco, I., Pajdla, T., Pollefeys, M., Sivic, J., Torii, A., Sattler, T.: D2-Net: A Trainable CNN for Joint Detection and Description of Local Features. In: Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019)
  * [8] Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: The kitti dataset (2013)
  * [9] Jau, Y.Y., Zhu, R., Su, H., Chandraker, M.: Deep keypoint-based camera pose estimation with geometric constraints. pp. 4950–4957 (2020). https://doi.org/10.1109/IROS45743.2020.9341229
  * [10] Jiang, W., Trulls, E., Hosang, J., Tagliasacchi, A., Yi, K.M.: COTR: correspondence transformer for matching across images. CoRR abs/2103.14167 (2021)
  * [11] Jocher, G.: YOLOv5 by Ultralytics (May 2020). https://doi.org/10.5281/zenodo.7347926, https://github.com/ultralytics/yolov5, v7.0
  * [12] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [13] Lin, T.Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C.L., Dollár, P.: Microsoft coco: Common objects in context (2014), http://arxiv.org/abs/1405.0312
  * [14] Lowe, D.: Object recognition from local scale-invariant features. In: Proceedings of the Seventh IEEE International Conference on Computer Vision. vol. 2, pp. 1150–1157 vol.2 (1999). https://doi.org/10.1109/ICCV.1999.790410
  * [15] Maji, D., Nagori, S., Mathew, M., Poddar, D.: Yolo-pose: Enhancing yolo for multi person pose estimation using object keypoint similarity loss (2022). https://doi.org/10.48550/ARXIV.2204.06806
  * [16] Mikolajczyk, K., Schmid, C.: A performance evaluation of local descriptors. IEEE transactions on pattern analysis and machine intelligence 27(10), 1615–1630 (2005)
  * [17] Mur-Artal, R., Montiel, J.M.M., Tardós, J.D.: ORB-SLAM: a versatile and accurate monocular SLAM system. CoRR abs/1502.00956 (2015), http://arxiv.org/abs/1502.00956
  * [18] Reich, A., Wuensche, H.J.: Fast Detection of Moving Traffic Participants in LiDAR Point Clouds by using Particles augmented with Free Space Information. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Kyoto, Japan (Oct 2022), ©IEEE
  * [19] Revaud, J., Weinzaepfel, P., De Souza, C., Pion, N., Csurka, G., Cabon, Y., Humenberger, M.: R2d2: Repeatable and reliable detector and descriptor (2019). https://doi.org/10.48550/ARXIV.1906.06195
  * [20] Rosten, E., Drummond, T.: Machine learning for high-speed corner detection. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) Computer Vision – ECCV 2006. pp. 430–443. Springer Berlin Heidelberg, Berlin, Heidelberg (2006)
  * [21] Schmid, C., Mohr, R., Bauckhage, C.: Evaluation of interest point detectors. International Journal of computer vision 37(2), 151–172 (2000)
  * [22] Schweitzer, M., Wuensche, H.J.: Efficient Keypoint Matching for Robot Vision using GPUs. In: Proc. 12th IEEE Int’l Conf. on Computer Vision, 5th IEEE Workshop on Embedded Computer Vision (Oct 2009). https://doi.org/10.1109/ICCVW.2009.5457621
  * [23] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2014). https://doi.org/10.48550/ARXIV.1409.1556
  * [24] Sun, J., Shen, Z., Wang, Y., Bao, H., Zhou, X.: Loftr: Detector-free local feature matching with transformers (2021). https://doi.org/10.48550/ARXIV.2104.00680
  * [25] Wang, C.Y., Mark Liao, H.Y., Wu, Y.H., Chen, P.Y., Hsieh, J.W., Yeh, I.H.: Cspnet: A new backbone that can enhance learning capability of cnn. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 390–391 (2020)
  * [26] Wang, M., Leelapatra, W.: A review of object detection based on convolutional neural networks and deep learning. International Scientific Journal of Engineering and Technology (ISJET) 6(1), 1–7 (2022)
  * [27] Xiao, Y., Tian, Z., Yu, J., Zhang, Y., Liu, S., Du, S., Lan, X.: A review of object detection based on deep learning. Multimedia Tools and Applications 79(33), 23729–23791 (2020)
  * [28] Zhao, Z.Q., Zheng, P., Xu, S.t., Wu, X.: Object detection with deep learning: A review. IEEE transactions on neural networks and learning systems 30(11), 3212–3232 (2019)


# Using Geographic Load Shifting
to Reduce Carbon Emissions

Julia Lindberg, Bernard C. Lesieutre and Line A. Roald  University of
Wisconsin-Madison, United States, Contact<EMAIL_ADDRESS>
###### Abstract

An increasing focus on the electricity use and carbon emissions associated
with computing has lead to pledges by major cloud computing companies to lower
their carbon footprint. Data centers have a unique ability to shift computing
load between different geographical locations, giving rise to geographic load
flexibility that can be employed to reduce carbon emissions. In this paper, we
present a model where data centers shift load independently of the ISOs. We
first consider the impact of load shifting guided by locational marginal
carbon emissions, denoted by $\lambda_{\text{CO}_{2}}$, a sensitivity metric
that measures the impact of incremental load shifts. Relative to previous
models for data center load shifting, the presented model improves accuracy
and include more realistic assumptions regarding the operation of both data
centers and the electricity market. Further, we introduce a new benchmark
model in which data centers have access to the full information about the
power system and can identify optimal shifts for the current time period. We
demonstrate the efficacy of our model on the IEEE RTS GMLC system using 5
minute load and generation data for an entire year. Our results show that the
proposed accuracy improvements for the shifting model based on
$\lambda_{\text{CO}_{2}}$ are highly effective, leading to results that
outperform the benchmark model.

###### Index Terms:

load shifting, carbon reduction

##  I Introduction

The recent technology revolution has infiltrated most aspects of modern day
life, and has lead to an increase in demand for computing resources. Between
2010 and 2018 there was an estimated 550% increase globally in the number of
data center workloads and computing instances [1]. This increased computing
demand has commanded a shift from smaller data centers to large-scale, highly
optimized and efficient facilities. These facilities are referred to as
_hyper-scalar_ data centers and are operated by large technology companies
such as Amazon, Facebook, Google, Microsoft and Alibaba. These companies
operate vast networks of these data centers which are dispersed geographically
throughout the world [2, 3]. Data centers currently consume around 1-2% of
electricity

globally [1]. This low percentage is largely due to the increased efficiency
of hyper-scalar data centers, which have processed the increased computing
demand while only increasing electricity use by an estimated 6% [1]. However,
as the scope for further efficiency gains is largely exhausted, it is
projected that electricity use for computing will increase rapidly in the
future.

Hyper-scalar data centers are large loads on electric power networks with
unique characteristics. For example, data center loads can defer when
computing tasks are processed or process them at different locations. These
properties equip data center operators with the unique ability to participate
in both geographic and temporal load shifting. Motivated by recent pledges
made by technology companies to reduce their carbon emissions [4, 5, 6], we
are interested in understanding how these hyper-scale data centers can
effectively interact with electricity markets in a way that minimizes the
carbon emissions caused by their computing loads. Start-ups such as Lancium
consider computing that adapts to the operation of the electric grid [7], an
idea that plays a critical role in the vision of zero carbon cloud computing
[8, 9].

When shifting load, data centers interact with electricity markets operated by
independent system operators (ISOs). Previous research has examined the impact
of integrating data centers and demand response [10, 11, 12, 13, 14, 15, 16]
or have considered geographical load shifting to reduce electricity costs [17,
18, 19], including load shifting between different electricity markets [20].
Other work has modelled data center flexibility through the use of virtual
links in time and space [21]. Much of this body of work considers the shifting
of computing load to reduce the carbon emissions of data centers [22, 23, 19]
and increase absorption of renewable energy [24, 25]. There are, however,
several major challenges to effective collaboration between the ISO and data
centers, such as barriers to the exchange of possibly sensitive information.
Another challenge is that ISOs and data centers have different objectives,
namely, ISOs focus on minimizing electricity cost while data centers would
like to minimize carbon emissions. While it is a common assumption that those
two objectives are well aligned, our previous research [26] found that if data
centers are specifically targeting the objective of reducing carbon emissions
(rather than reducing cost) they can achieve better results by doing the load
shifting outside of the electricity market rather than collaborating with the
ISO. Since ISOs are unlikely to change the market to directly minimize carbon
emissions in the near future, we focus on a method to identify geographical
load shifts that reduce carbon emissions within the existing electricity
market.

To effectively shift load geographically, we require information about the
locational variation in the carbon footprint of electricity, similar to the
way locational marginal prices describe the locational variations in the cost
of electricity. While data on electricity prices is easily available, similar
information on the carbon emissions associated with electricity usage is not.
As a result, it is less straight forward to develop metrics that provide good
guidance for shifting load to reduce carbon emissions. Previous work in this
vein has assumed that prices are directly tied to the fraction of non-
renewable energy [27], or considered average carbon emissions for electricity
in a region and/or renewable energy curtailment [28, 29, 25]. A benefit to
these metrics is that several companies provide the necessary information
needed to compute them [30, 31, 32]. A downside is that they all fail to
consider necessary aspects of electric grid operation, such as the impact of a
marginal change in load and transmission capacity limits. Specifically, there
is no consideration for which generators in the system will be providing the
additional generation or if is there is sufficient transmission capacity
available to deliver the generation. To effectively guide load shifting, we
need a measure of marginal emissions [26]. The use of marginal emissions to
assess energy efficiency or the potential impact of renewable energy has been
considered in [33, 34], proposed to guide renewable energy investments [35,
36, 33], or considered the impact of marginal emissions rates at cogeneration
facilities [37]. More recently, a measure of locational marginal carbon
emissions was proposed for data center geographical load shifting in [38]. The
superiority of this metric over others such as average carbon emissions,
curtailment or LMPs for data center geographic load shifting was demonstrated
in [26].

While the previous load shifting models in [38, 26] presented promising
results, it included several unrealistic assumptions such as the electricity
market being solved twice in each time step. Here, we present a model that
addresses these issues and also leverages regularization to increase the
accuracy of the model. Further, due to the fact that the shifting metric
derived in [38] is a linear sensitivity, we expect that at each time step this
metric is not necessarily finding the best load shift. For this reason, we
present a new benchmark model for optimal load shifting. This bilevel program
finds the best load shift a data center can perform, given the current
electricity market. In summary, the three main contributions of this paper are
the following: _1) Realistic model for data center load shifting:_ We address
the unrealistic assumptions in our previous model by implementing load
shifting in a cumulative fashion and incorporating a regularization parameter
to increase the accuracy of shifting.

_2) Benchmark model for optimal data center shifting:_ We develop a new
benchmark model that computes the optimal data center load shift, given access
to all system information. This bilevel optimization method determines the
optimal load shift to reduce carbon emissions, subject to the operation of the
current electricity market.

_3) Computational Analysis:_ We compare both models using the RTS-GMLC system
with one year of 5 minute load and generation data, and observe that the
shifting model based on locational marginal carbon emissions performs quite
well.

The remainder of the paper is organized as follows. In Section II we review
the existing data center driven shifting model, while Section III presents the
proposed improvements to this model. In Section IV, we present a new benchmark
model for optimal load shifting. Section V demonstrates the efficacy of our
model in a case study, and Section VI concludes.

##  II Review of Existing Load Shifting Model

Previous work has considered ISO independent load shifting to reduce carbon
emissions via a value called the locational marginal carbon emissions,
$\lambda_{\text{CO}_{2}}$ [38]. This value is calculated as a linear
sensitivity around the optimal solution to the DC OPF. We give a brief outline
of this load shifting model here, but suggest [38] as a more detailed
reference.

Step 1: This model begins by assuming the ISO solves a DC OPF [39, 40]. The DC
OPF is a linear optimization problem that seeks to minimize generation costs
subject to network and demand constraints with decision variables $x=[\theta\
P_{g}]$ where $P_{g}$ are the generation variables, $\theta$ are the voltage
angles at each node. Let $\mathcal{N}$ be the set of all nodes, $\mathcal{L}$
the set of lines and $\mathcal{G}$ the set of generators. The DC OPF is
defined as

$\displaystyle\min_{\theta,P_{g}}\leavevmode\nobreak\ \leavevmode\nobreak\ $
$\displaystyle c^{T}P_{g}$ (1a) s.t.
$\displaystyle\textstyle\sum_{\ell\in\mathcal{G}_{i}}\\!P_{g,\ell}-\\!\textstyle\sum_{\ell\in\mathcal{D}_{i}}P_{d,\ell}=$
$\displaystyle\qquad\quad\textstyle\sum_{j:(i,j)\in\mathcal{L}}\\!\\!\\!\\!-\beta_{ij}(\theta_{i}\\!-\\!\theta_{j}),$
$\displaystyle\forall i\in\mathcal{N}$ (1b) $\displaystyle-$ $\displaystyle
P^{lim}_{ij}\\!\leq\\!-\beta_{ij}(\theta_{i}\\!-\\!\theta_{j})\\!\leq\\!P^{lim}_{ij},$
$\displaystyle\forall(i,j)\in\mathcal{L}$ (1c) $\displaystyle
P^{min}_{g,i}\leq P_{g,i}\leq P^{max}_{g,i},$ $\displaystyle\forall
i\in\mathcal{G}$ (1d) $\displaystyle\theta_{ref}=0.$ (1e)

The cost function (1a) minimizes the cost of generation, where $c_{i}$ is the
cost of generation at generator $i$. Constraints (1b)-(1d) constrain the nodal
power balance, transmission line and generator capacity constraints
respectively. Finally, (1e) sets the voltage angle at the reference node to
zero.

Step 2: Independent of any ISO collaboration, data center operators shift
their load to minimize carbon emissions. To guide this effort, a metric known
as the locational marginal carbon emissions was proposed in [38]. In [26] the
authors demonstrated the superiority of this metric to other more commonly
studied metrics such as the average carbon emissions or excess low carbon
power.

To calculate the locational marginal carbon emissions, we first consider a
basic optimal solution $x^{*}=[\theta^{*},P_{g}^{*}]\in\mathbb{R}^{n}$ to the
DC OPF (1). From sensitivity analysis in linear optimization theory [41], a
basic optimal solution can be written as $Ax^{*}=b$, where
$A\in\mathbb{R}^{n\times n}$ is a full rank matrix consisting of all the
active constraints of (1) at the optimal solution $x^{*}$. In the case of the
DC OPF, the rows of $A$ consist of the equality constraints (1b) and (1e) as
well as a subset of the inequality constraints (1c), (1d) that are satisfied
at equality for $x^{*}$.

A small change in load can be represented as a small change in the right hand
side $b$, given by $\Delta b=\begin{bmatrix}\Delta P_{d}&0\end{bmatrix}^{T}$.
Assuming that the change is small enough as not to alter the set of active
constraints, we compute the optimal change in generation as $A\cdot\Delta
x=\Delta b$ where $\Delta x=[\Delta\theta\ \Delta P_{g}]$. This gives rise to
the linear relationship

$\displaystyle\begin{bmatrix}\Delta\theta\\\ \Delta P_{g}\end{bmatrix}$
$\displaystyle=A^{-1}\cdot\begin{bmatrix}\Delta P_{d}\\\ 0\end{bmatrix}$ (2)

If we denote the matrix consisting of the last $|\mathcal{G}|$ rows and first
$|\mathcal{N}|$ columns of $A^{-1}$ by $B$, this gives the linear relationship
between load and generation changes, $\Delta P_{g}=B\cdot\Delta P_{d}$.

Consider the cost vector $g\in\mathbb{R}^{|\mathcal{G}|}$ that measures the
carbon emissions of each generator per MW. Specifically, the $i$th component
of $g$, is the carbon intensity of generator $i$. Multiplying each side of
$\Delta P_{g}=B\cdot\Delta P_{d}$ on the left by $g$ gives us the change in
carbon emissions

$\displaystyle\Delta CO_{2}=g\cdot\Delta P_{g}=g\cdot B\cdot\Delta
P_{d}=\lambda_{\text{CO}_{2}}\Delta P_{d}.$ (3)

where $\lambda_{\text{CO}_{2}}=g\cdot B$. Intuitively, the $k$th component of
$\lambda_{\text{CO}_{2}}$ measures how an increase of $1$ MW of load at node
$k$ will affect the total carbon emissions of the system.

We let $\mathcal{C}$ denote the set of data center loads that can
geographically shift load and consider optimization variables that denote the
change in load at data center $i$, $\Delta P_{d,i}$, and the shift in load
from data center $i$ to $j$, $s_{ij}$. The geographic load shifting
optimization problem is given by:

$\displaystyle\min_{\Delta P_{d},s}\ \ $
$\displaystyle\sum_{i\in\mathcal{C}}\lambda_{\text{CO}_{2},i}\Delta P_{d,i}$
(4a) s.t. $\displaystyle\Delta
P_{d,i}=\textstyle\sum_{j\in\mathcal{C}}s_{ji}-\textstyle\sum_{k\in\mathcal{C}}s_{ik}\quad$
$\displaystyle\forall i\in{C}$ (4b) $\displaystyle\sum_{i\in\mathcal{C}}\Delta
P_{d,i}=0$ (4c) $\displaystyle-\epsilon_{i}\cdot\text{Cap}_{i}\leq\Delta
P_{d,i}\leq\epsilon_{i}\cdot\text{Cap}_{i}\quad\\!\\!$ $\displaystyle\forall
i\in\mathcal{C}$ (4d) $\displaystyle 0\leq\Delta
P_{d,i}+P_{d,i}\leq\text{Cap}_{i}\quad$ $\displaystyle\forall i\in\mathcal{C}$
(4e) $\displaystyle 0\leq s_{ij}\leq M_{ij}\quad$ $\displaystyle\forall
ij\in\mathcal{C}\\!\times\\!\mathcal{C}.$ (4f)

The objective value (4a) minimizes the change in carbon emissions as a
function of the change in load, (4b) enforces that the change in load at a
given data center is equal to the total load shifted in minus the total load
shifted out, while (4c) says the sum of all load shifts must be zero.
Constraint (4d) limits the amount each data center can shift as a fraction
$\epsilon$ of the data center capacity Cap. Constraint (4e) puts an upper
bound $M_{ij}$ on the total capacity at each data center and (4f) limits how
much load data center $i$ can send to data center $j$.

Step 3: Finally, the ISO resolves the DC OPF (1) with new load profile,
$P_{d,i}^{\prime}=P_{d,i}+\Delta P_{d,i}^{*}$, where $\Delta P_{d,i}^{*}$ is
the optimal solution to (4) for all $i\in\mathcal{N}$.

##  III Realistic Data Center Load Shifting Model

The above model has several drawbacks. First, it is unrealistic to assume that
the ISO resolves the market clearing twice, once before and once after the
shifting has happened. Second, since the model is linear, we tend to see large
load shifts even with small differences in $\lambda_{\text{CO}_{2}}$ between
data center locations. Since $\lambda_{\text{CO}_{2}}$ is a local sensitivity
factor that is only accurate near the previous optimal solution, these large
shifts lead to inaccurate results that sometimes increase carbon emissions. To
address these issues, we introduce two improvements to the model: cumulative
load shifting and regularization.

###  III-A Cumulative load shifts

We refine the model defined in [38] by considering _cumulative load shifts_.
Instead of resolving the DC OPF in Step $3$ of the above model, the load shift
is applied to the market clearing in the next time step. Specifically, the
algorithm runs as follows:

Step 1: At time $t$, the ISO solves the DC OPF (1) with data center load set
to $P_{d}^{t}$.

Step 2: Given information about $\lambda_{\text{CO}_{2}}$ as described above,
the data center operator computes a load shift $\Delta P_{d}^{t}$ according to
(4). Then, the data center load for time $t+1$ is set to
$P_{d}^{t+1}=P_{d}^{t}+\Delta P_{d}^{t}$, and the algorithm proceeds to Step
$1$ of the next time step.

While the cumulative load shifting model more accurately reflects the current
market set up, it introduces an additional inaccuracy in our model. The
locational marginal carbon emission value $\lambda_{\text{CO}_{2}}$ at each
data center is derived as a linearization from the operating point at time
$t$, but the internal data center shifting optimization will only affect the
market clearing at time $t+1$. The expectation is that since operating
conditions remain similar between time steps, shifting with respect to
$\lambda_{\text{CO}_{2}}$ will still lead to a decrease in total system carbon
emissions. We also note that cumulative load shifting can increase accuracy
relative to the existing model, particularly the load shift allowed in each
time step is small (i.e., only a small fraction $\epsilon$ can be shifted). In
this case, changes in the data center load build up slowly over time. This is
in contrast to our previous model, where the data center load was reset to the
original value $P_{d}$ in each time step.

###  III-B Regularizing load shifts

To discourage large load shifts which can cause oscillations and increased
emissions, we propose to use a regularization term (i.e., a quadratic penalty)
that discourages large shifts. Specifically, this model replaces the objective
value (4a) with

$\sum_{i\in\mathcal{C}}\lambda_{\text{CO}_{2},i}\Delta P_{d,i}+\gamma\|\Delta
P_{d,i}\|_{2}^{2}$

where $\gamma\in\mathbb{R}$ is a regularization parameter. The goal in using
this regularization term is to discourage large shifts that lead to an
increase in carbon emissions as well as increase the accuracy of the data
center driven shifting model. However, the regularization term can also be
interpreted as a quadratic cost on load shifting. This ensures that while
small shifts are cheap and frequent, we only shift a large amount of load when
there will be a large reduction in carbon emissions.

Throughout the rest of this paper we refer to the model outlined in this
section as ($\lambda_{\text{CO}_{2}}-$shift).

##  IV Benchmark model for optimal shifting

Our next contribution is to introduce a new model to benchmark the data center
driven shifting model. Since the shifts provided by $\lambda_{\text{CO}_{2}}$
are calculated by a linear sensitivity, they can be inaccurate, even giving
shifting profiles that increase carbon emissions.

The problem of identifying the optimal load shift data center operators should
employ to minimize carbon emissions can be modelled as a bilevel linear
program. The upper level problem identifies the optimal choice of load shift
$\Delta P_{d}$ to minimize carbon, i.e.,

$\displaystyle\min_{\Delta P_{d},s,P_{g}^{*}}$ $\displaystyle\ g^{T}P_{g}^{*}$
s.t. $\displaystyle P_{g}^{*}=\arg\min\eqref{opt:dcopf_with_flex}$ (Opt-shift)
$\displaystyle(\Delta P_{d},s)\in\mathcal{P}$

Here, the last constraint represents the set of feasible load shifts from the
data center perspective, i.e., $\mathcal{P}$ is the polytope of permissible
load shifts defined by the constraints in (4).. The first constraint states
that the generation values $P_{g}^{*}$ is the solution to the lower level
optimization problem (DC-shift). This problem is a version of the standard DC
OPF (1) where the nodal balance constraints include the change in demand.
Formally we write it as

$\displaystyle\min_{P_{g},\theta}\ c^{T}P_{g}\quad\text{subject to}$
$\displaystyle\text{Constraints
}\eqref{lineineq},\eqref{genineq},\eqref{refnode}$ (DC-shift)
$\displaystyle\sum_{\ell\in\mathcal{G}_{i}}\\!P_{g,\ell}-\\!\\!\\!\sum_{\ell\in\mathcal{D}_{i}}\\!(P_{d,\ell}+\Delta
P_{d,\ell})=\\!\\!\\!\\!\\!\\!\\!\sum_{j:(i,j)\in\mathcal{L}}\\!\\!\\!\\!\\!\\!\\!\\!-\beta_{ij}(\theta_{i}\\!-\\!\theta_{j}),$
$\displaystyle\\!\\!\\!\forall i\\!\in\\!\mathcal{N}$

As in $(\lambda_{\text{CO}_{2}}$-shift), we also consider cumulative load
shifting in (Opt-shift). Specifically, at each data center
$\ell\in\mathcal{C}$, at time step $t$ we assume the load $P_{d,\ell}$ in (DC-
shift) reflects the sum of new load $P_{d,\ell}$ from time $t$ and the load
shift $\Delta P_{d,\ell}$ from time $t-1$. Herein out when we refer to the
model (Opt-shift) we assume it is employed with this cumulative load shifting.

##  V Computational Results

We next analyze the efficacy in carbon reduction of
$(\lambda_{\text{CO}_{2}}$-shift) versus (Opt-shift).

###  V-A Accuracy of $(\lambda_{\text{CO}_{2}}$-shift)

An important aspect of $(\lambda_{\text{CO}_{2}}$-shift) that needs to be
considered is the accuracy of the predicted change to carbon emissions and
generation cost. Specifically, using the relation $\Delta P_{g}=B\cdot\Delta
P_{d}$, we can get a predicted change in generation that will result as an
effect of the load shift $\Delta P_{d}$. Using this predicted generation
change, we can derive a value for the predicted change in cost and carbon
emissions to the system as a result of the load shift. We compare this
predicted change to the true change in carbon emissions.

###  V-B Test Case

We perform an extensive year long analysis of carbon reduction methods
mentioned above using the RTS-GMLC system [42]. This system has $73$ buses,
$158$ generators and $120$ lines. Since the original system does not designate
data center loads, we assign data centers at buses $103,107,204$ and $322$. We
assume that cumulatively, the four data centers consume a fixed power of
$1000$ MW at each time step throughout the year, although the distribution of
that power among the four data centers varies. We assume at time step $0$,
each data center starts with $250$ MW of load. For all other loads and
renewable generation, we use the real time, i.e. $5$ minute, load and
generation data provided with [42]. Over the course of the year, this system
serves $526,220,000$ MW of load, and $105,408,000$ MW or roughly $20.03\%$ of
it is data center load.

Adding these large data center loads to the network greatly increases the
total system load and results in time steps where the original DC OPF is
infeasible. To remedy this we change the generation limits by setting
$P_{g}^{\text{min}}=0$ for all $g\in\mathcal{G}$, and increase the maximum
generation limits by $50\%$. At each time step we allow each data center to
shift up to $20\%$ of its total capacity, i.e. $50$ MW, and enforce that data
center capacities remain between $0$ and $300$ MW. Further, we put no
limitations on how much load each data center can shift to one another.

###  V-C The effect of regularization

We first investigate the effect of the regularization parameter $\gamma$. The
effect of various regularization parameters on generation cost, total system
carbon emissions and total load shift is shown in Figure 4, where the orange
and blue lines represent the predicted and actual values, respectively. Figure
4(a) show that the minimum total system carbon emissions occurs when the
regularization parameter $\gamma=1.5$ is used. In addition, we see in both
Figure 4(a) and (b) that as the regularization parameter $\gamma$ increases,
the difference between the predicted carbon emissions and generation cost and
the actual carbon emissions and generation costs decreases. This indicates
that including a regularization term helps not only in the efficacy of the
data center driven shifting model, but also in the accuracy.

The reason regularization is considered is to discourage load shifting in
cases where it is not predicted to make large differences. Figure 4(c) shows
how as the regularization parameter increases, the total load shifted
throughout the year decreases. We see that when the regularization parameter
is set at $\gamma=1.5$, the total amount of load shifted is less than half of
the amount of load shifted when $\gamma=0$. Considering that the carbon
emissions and generation cost when $\gamma=1.5$ are lower than when
$\gamma=0$, this demonstrates that shifting less load, more strategically can
lead to a larger reduction in carbon emissions and a smaller increase in
generation costs.

Figure 1: Change in carbon emissions.

Figure 2: Change in generation cost.

Figure 3: Change in total load shift. Figure 4: Change in carbon emissions, generation cost and load shift as the regularization parameter $\gamma$ varies. | DC OPF | (Opt-shift) |  $\lambda_{\text{CO}_{2}}-$shift: $\gamma=0$ |  $\lambda_{\text{CO}_{2}}-$shift: $\gamma=1.5$
---|---|---|---|---
Generation Cost | $3,802,706,000$ | $4,981,076,000$ | $3,847,332,000$ | $3,843,847,000$
CO2 Emissions | $164,402,000$ | $110,444,000$ | $161,522,000$ | $161,427,000$
Total Shifts | 0 | $1,048,000$ | $6,199,000$ | $2,245,000$
Table I: Summary of results from all models

###  V-D Comparison with Opt-Shift and Original DC OPF solution

We next compare the solutions for $(\lambda_{\text{CO}_{2}}$-shift) with
regularization parameters $\gamma=0$ and $\gamma=1.5$ with the original DC OPF
solution and the solution obtained using our benchmark model (Opt-shift).
These results are given in Table I. We see that when considering
($\lambda_{\text{CO}_{2}}$-shift) with no regularization, carbon emissions
relative to the original DC OPF decreases by around 2.8 million tons or
$1.75\%$. This reduction is achieved while shifting around $6.2$ million MW of
load. Conversely, once the regularization term $\gamma=1.5$ is added, we
achieve an even greater reduction in carbon emissions, namely $2,975,000$ tons
or $1.81\%$ while only shifting around $2.25$ million MW of load. In addition,
when considering regularization, total system generation costs only increased
by $1.08\%$ while without regularization it increased by $1.17\%$.

In contrast to the above results, we see a dramatic carbon savings when using
the benchmark (Opt-shift). In this case we save $53,958,000$ tons of carbon,
i.e. $32.82\%$. This occurs while only shifting a little over $1$ million MW.
This dramatic savings occurs at a major increase to generation costs. Namely,
(Opt-shift) results in an increase in $\$1,178,370,000$ to generation costs or
$30.99\%$ over the original DC OPF. This benchmark model suggests that
dramatic reductions in carbon emissions are possible even with limited data
center flexibility, but come at a large increase to generation costs.

###  V-E Carbon Emissions vs Generation Costs

As seen above, minimizing carbon emissions can lead to an increase in
generation cost. To better understand the trade-off between carbon emissions
and cost, we consider the benchmark model (Opt-shift) with objective function

$(\alpha c^{T}+(1-\alpha)g^{T})P_{g}^{*}$

and $(\lambda_{\text{CO}_{2}}-$shift) with objective function

$(\alpha\text{LMP}+(1-\alpha)\lambda_{\text{CO}_{2}})\Delta
P_{d}+1.5\cdot\|\Delta P_{d}\|_{2}^{2}$

in place of (4a) where $\alpha\in[0,1]$ is a trade off parameter that allows
us to weight the emphasis on minimizing carbon emissions versus generation
costs and LMP is a vector of the locational marginal prices at each node. The
trade off between minimizing carbon emissions and generation cost is shown
graphically in Figure 5.

Figure 5: Trade off between carbon emissions and generation cost.

When considering ($\lambda_{\text{CO}_{2}}-$shift), shown in yellow, we see a
small variation in the overall system generation cost and carbon emissions
that remains close to the carbon emissions and generation cost of the original
DC OPF. This is consistent with the results shown above, and is due to the
fact that this model considers small shifts away from an operating point that
minimizes generation costs. The benchmark model (Opt-shift) produces a much
larger variation in operating points as we change the trade-off parameter
$\alpha$. As $\alpha$ increases, the model produces a large increase in carbon
emissions for only a moderate cost savings. In addition, we see that for (Opt-
shift) to achieve lower carbon emissions than the DC OPF and the
$(\lambda_{\text{CO}_{2}}-$shift), a major increase to generation cost is
needed. This demonstrates that even with limited geographic load shifting
flexibility, a large reduction in carbon emissions is possible but it comes at
the price of significantly higher generation costs.

Figure 5 also demonstrates an interesting phenomenon, namely the greedy nature
of (Opt-shift). When only trying to minimize carbon emissions, (Opt-shift) is
able to reduce total system carbon emissions by roughly $33\%$ but this comes
at a significant increase to total system generation cost. However, for the
same generation cost, (Opt-shift) gives a solution with higher carbon
emissions than the DC OPF or $(\lambda_{\text{CO}_{2}}$-shift). This
demonstrates that the greedy nature of (Opt-shift) is not necessarily an
optimal way to shift load over a long time span. Specifically, (Opt-shift)
finds a load shift that gives the largest reduction in carbon emissions at
that time step, with no consideration to how the load shift will affect the
carbon emissions of the system at the next time step. Using forecasts of
future load and generation information to aid in a long term load shifting
strategy is left as future work.

###  V-F Data Center Operating Load

Finally, we show consider the impact of each model on the data center
operating load. We consider $(\lambda_{\text{CO}_{2}}$-shift) with
regularization parameter $\gamma=1.5$ and (Opt-shift), and two different
limits on the amount of load that can be shifted in each time step,
$\epsilon=0.01$ and $\epsilon=0.2$.

In Figure 6 we see the operating conditions of each data center over the
course of the first day when using $(\lambda_{\text{CO}_{2}}$-shift) when
$\epsilon=0.01$ (left) and $\epsilon=0.2$ (right). In both cases we see
similar overall trends in operating load. However, $\epsilon=0.2$ leads to
much quicker changes and also dramatic oscillations in the load at data
centers $1$ and $3$ towards the end of the day. Similarly, in Figure 7 we see
the operating conditions of each data center over the course of the first day
using (Opt-shift) when $\epsilon=0.01$ (left) and $\epsilon=0.2$ (right).
Again, we see similar trends data center load for both values of $\epsilon$,
but for $(\lambda_{\text{CO}_{2}}$-shift), $\epsilon=0.2$ leads to more
oscillations in data center operating load.

Interestingly, there are some differences between the
$(\lambda_{\text{CO}_{2}}$-shift) and (Opt-shift). In both cases we see an
initial pull for data center $4$ to operate at maximum capacity while the
other data centers operate at lower capacities. This implies that the
$\lambda_{\text{CO}_{2}}$ value for data center $4$ is accurately dictating
that it is the most carbon neutral data center. In contrast, we see that when
shifting with respect to $(\lambda_{\text{CO}_{2}}$-shift), data center $2$ is
also operating at maximum capacity. This is in contrast to shifting when using
(Opt-shift). In this instance data center $2$ initially drops to be the data
center operating at the lowest load. This discrepancy highlights the
inaccuracy when shifting with respect to $\lambda_{\text{CO}_{2}}$.

Figure 6: Load at each data center during the first $24$ hours using
$\lambda_{\text{CO}_{2}}$-shift with $\gamma=1.5$ and $\epsilon=0.01$ (left)
and $\epsilon=0.2$ (right).

Figure 7: Load at each data center during the first $24$ hours using (Opt-
shift) with $\epsilon=0.01$ (left) and $\epsilon=0.2$ (right).

Finally, we investigate how using different $\epsilon$ values impacts the
overall effect on carbon emissions. In Figure 8 we see the change in total
system carbon emissions as $\epsilon$ varies for both
$(\lambda_{\text{CO}_{2}}$-shift) with $\gamma=1.5$ as well as (Opt-shift). In
both cases we see only a very mild decrease in total carbon emissions as we
allow $\epsilon$ to increase. Further, for $(\lambda_{\text{CO}_{2}}$-shift),
we see that as $\epsilon$ increases, the accuracy of the model decreases and
once $\epsilon>0.1$, the carbon emissions starts to increase. This indicates
that allowing small shifts not only is more desirable from an operational
stand point to avoid rapid changes and oscillations in data center loading,
but it leads to similar carbon savings as allowing larger shifts.

Figure 8: Predicted and actual change in carbon emissions using
$\lambda_{\text{CO}_{2}}-$shift (left) and the change in carbon emissions
using (Opt-shift) (right) for varying epsilon values.

##  VI Conclusion

In this paper we presented an improved model for data center load shifting to
reduce carbon emissions. This model shifts load independently of ISO
collaboration via a measure known as locational marginal carbon emissions. We
built on existing work, but made several improvements to increase realism and
accuracy of our model. We also proposed a new benchmark model which gives the
best load shift at each time step, and compared the results. The main
conclusion from the paper is that smaller load shifts, limited by
regularization and shifting caps, are quite effective in reducing carbon
emissions. Larger load shifts tend to decrease accuracy of the model and
produce less carbon savings. Further, while our benchmark model is able to
achieve large carbon reductions, it also significantly increases cost.

This paper demonstrates many natural directions for future work. First, this
work shows that shifting load in a greedy way, i.e. shifting to reduce the
maximal amount of carbon at each time step, is not necessarily the best
approach if current load shifts will impact the load profile in future time
steps. This demonstrates that forecasting future load and generation patterns
is important to obtain better solutions. Finally, the information needed to
calculate the locational marginal carbon emissions is currently not made
publicly available. Therefore, finding ways to infer and predict
$\lambda_{\text{CO}_{2}}$ at the data center nodes from publicly available
data is needed in order to implement $(\lambda_{\text{CO}_{2}}$-shift) in
practice.

## References

  * [1] Eric Masanet, Arman Shehabi, Nuoa Lei, Sarah Smith, and Jonathan Koomey.  Recalibrating global data center energy-use estimates.  Science, 367(6481):984–986, 2020.
  * [2] Pierre Delforge and Josh Whitney.  Data center efficiency assessment-scaling up energy efficiency across the data center industry: Evaluating key drivers and barriers.  Natural Resources Defense Council, 2014.
  * [3] Yevgeniy Sverdlik.  Analysts: There are now more than 500 hyperscale data centers in the world, October 2017.  https://bit.ly/3jx8lsz.
  * [4] Google.  Google environmental report 2019.  Technical report, Google, 2019.  https://services.google.com/fh/files/misc/google_2019-environmental-report.pdf.
  * [5] Justine Calma.  Amazon boosts climate commitments and greenhouse gas emissions, June 2020\.  https://bit.ly/2Sr4Vfc.
  * [6] Ana Radovanovic.  Our data centers now work harder when the sun shines and wind blows, April 2020.  Google Official Blog, https://bit.ly/3cXPAMu.
  * [7] Lancium.  With 5th patent, lancium powers ahead in fast-ramping data center innovation.  https://lancium.com/lancium-smart-response-patent-press-release/.
  * [8] Fan Yang and Andrew A Chien.  Zccloud: Exploring wasted green power for high-performance computing.  In IEEE Int Parallel and Distributed Processing Symposium (IPDPS), pages 1051–1060. IEEE, 2016.
  * [9] Andrew A Chien.  Zero-carbon compute project.  2019\.  https://bit.ly/34m68dh.
  * [10] Adam Wierman, Zhenhua Liu, Iris Liu, and Hamed Mohsenian-Rad.  Opportunities and challenges for data center demand response.  In Int. Green Comp. Conf., pages 1–10, 2014.
  * [11] Zhenhua Liu, Adam Wierman, Yuan Chen, Benjamin Razon, and Niangjun Chen.  Data center demand response: avoiding the coincident peak via workload shifting and local generation.  In ACM Int. Conf. on Measurement and Modeling of Computer Systems, 2013.
  * [12] Zhenhua Liu, Iris Liu, Steven Low, and Adam Wierman.  Pricing data center demand response.  In ACM Int. Conf. on Measurement and Modeling of Computer Systems, 2014.
  * [13] Zhi Zhou, Fangming Liu, and Zongpeng Li.  Bilateral electricity trade between smart grids and green datacenters: Pricing models and performance evaluation.  IEEE Journal on Selected Areas in Communications, 34(12):3993–4007, 2016.
  * [14] Zhi Zhou, Fangming Liu, Shutong Chen, and Zongpeng Li.  A truthful and efficient incentive mechanism for demand response in green datacenters.  IEEE Trans. Parallel and Distributed Systems, 31(1):1–15, 2020\.
  * [15] Zhi Zhou, Fangming Liu, Zongpeng Li, and Hai Jin.  When smart grid meets geo-distributed cloud: An auction approach to datacenter demand response.  In IEEE Conf. on Computer Communications (INFOCOM), pages 2650–2658, 2015.
  * [16] Shutong Chen, Lei Jiao, Lin Wang, and Fangming Liu.  An online market mechanism for edge emergency demand response via cloudlet control.  In IEEE Conf. on Computer Communications (INFOCOM), pages 2566–2574, 2019.
  * [17] Jie Li, Zhen Bao, and Zuyi Li.  Modeling demand response capability by internet data centers processing batch computing jobs.  IEEE Trans. Smart Grid, 6(2):737–747, 2014.
  * [18] Lei Rao, Xue Liu, M. D. Ilic, and Jie Liu.  Distributed coordination of internet data centers under multiregional electricity markets.  Proceedings of the IEEE, 100(1):269–282, 2011.
  * [19] Hui Dou, Yong Qi, Wei Wei, and Houbing Song.  Carbon-aware electricity cost minimization for sustainable data centers.  IEEE Trans. Sustainable Computing, 2(2):211–223, 2017.
  * [20] Lei Rao, Xue Liu, Le Xie, and Wenyu Liu.  Minimizing electricity cost: optimization of distributed internet data centers in a multi-electricity-market environment.  In 2010 Proc. IEEE INFOCOM, pages 1–9, 2010.
  * [21] Weiqi Zhang, Line A. Roald, Andrew A. Chien, John R. Birge, and Victor M. Zavala.  Flexibility from networks of data centers: A market clearing formulation with virtual links.  21st Power Systems Computation Conference, 2020.
  * [22] Ínigo Goiri, William Katsak, Kien Le, Thu D Nguyen, and Ricardo Bianchini.  Parasol and greenswitch: Managing datacenters powered by renewable energy.  ACM SIGPLAN Notices, 48(4):51–64, 2013.
  * [23] Zhenhua Liu, Yuan Chen, Cullen Bash, Adam Wierman, Daniel Gmach, Zhikui Wang, Manish Marwah, and Chris Hyser.  Renewable and cooling aware workload management for sustainable data centers.  In 12th ACM SIGMETRICS/PERFORMANCE Joint Int. Conf. on Measurement and Modeling of Computer Systems, pages 175–186, 2012.
  * [24] Kibaek Kim, Fan Yang, Victor M. Zavala, and Andrew A. Chien.  Data centers as dispatchable loads to harness stranded power.  IEEE Trans. Sustain. Energy, 8(1):208–218, 2016.
  * [25] Jiajia Zheng, Andrew A Chien, and Sangwon Suh.  Mitigating curtailment and carbon emissions through load migration between data centers.  Joule, 2020.
  * [26] J Lindberg, YAbdennadher, J Chen, B C. Lesieutre, and L Roald.  A guide to reducing carbon emissions through data center geographical load shifting.  In ACM Int. Conf. on Future Energy Systems, pages 430–436, 2021\.
  * [27] Z. Liu, M. Lin, A. Wierman, S. Low, and L. L. H. Andrew.  Greening geographical load balancing.  IEEE/ACM Trans. Networking, 23(2):657–671, 2015.
  * [28] Andrew A Chien, Richard Wolski, and Fan Yang.  The zero-carbon cloud: High-value, dispatchable demand for renewable power generators.  The Electricity Journal, 28(8):110–118, 2015.
  * [29] Fan Yang and Andrew A Chien.  Large-scale and extreme-scale computing with stranded green power: Opportunities and costs.  IEEE Trans. Parallel and Distributed Systems, 29(5):1103–1116, 2017\.
  * [30] Tomorrow.  Data driven climate action.  Technical report, https://www.electricitymap.org, 2020.
  * [31] CAISO.  Today’s outlook: Emissions, 2021.
  * [32] CAISO.  Managing oversupply, 2021.
  * [33] Kyle Siler-Evans, Ines Lima Azevedo, and M Granger Morgan.  Marginal emissions factors for the us electricity system.  Environmental science & technology, 46(9):4742–4748, 2012.
  * [34] D S Callaway, M Fowlie, and G McCormick.  Location, location, location: The variable value of renewable energy and demand-side efficiency resources.  J. of the Association of Environmental and Resource Economists, 5(1):39–75, 2018.
  * [35] P. A. Ruiz and A. Rudkevich.  Analysis of marginal carbon intensities in constrained power networks.  In 43rd Hawaii Int. Conf. on System Sciences, 2010.
  * [36] A. Rudkevich, P. A. Ruiz, and R. C. Carroll.  Locational carbon footprint and renewable portfolio policies: A theory and its implications for the eastern interconnection of the us.  In Hawaii Int Conf on System Sciences (HICSS), 2011.
  * [37] Richard Tabors, Aleksandr Rudkevich, Hua He, Ninad Kumthekar, Xindi Li, Robert Bland, Garret Quist, and Joshua LaPenna.  Methodology for calculation of the marginal emission rates from a complex cogeneration facility compared with that of the co-located ny iso bus.  Hawaii Int Conf on System Sciences (HICSS), 2021.
  * [38] J. Lindberg, B. C. Lesieutre, and L. Roald.  The environmental potential of hyper-scale data centers: Using locational marginal CO2 emissions to guide geographical load shifting.  In Hawaii Int Conf on System Sciences (HICSS), pages 1–10, 2021\.
  * [39] R D Christie, B F Wollenberg, and I Wangensteen.  Transmission management in the deregulated environment.  Proceedings of the IEEE, 88(2):170–195, 2000.
  * [40] E. Litvinov.  Design and operation of the locational marginal prices-based electricity markets.  IET Gen., Trans. and Distr., 4:315–323(8), February 2010.
  * [41] D. Bertsimas and J. Tsitsiklis.  Introduction to linear optimization.  1997\.
  * [42] C. Barrows, E. Preston, A. Staid, G. Stephen, J.-P. Watson, A. Bloom, A. Ehlen, J. Ikaheimo, J. Jorgenson, D. Krishnamurthy, J. Lau, B. McBennett, and M. O’Connell.  The ieee reliability test system: A proposed 2019 update.  IEEE Trans. Power Systems, 35(1):119–127, 2020.