# RobustCalib: Robust Lidar-Camera Extrinsic Calibration with
Consistency Learning

Shuang Xu1, Sifan Zhou1,2, Zhi Tian1, Jizhou Ma1, Qiong Nie1, Xiangxiang Chu1
1 Meituan Inc. 2 Southeast University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>Work done as an intern at
Meituan.Corresponding author.

###### Abstract

In autonomous vehicles and robotic systems, 3D LiDARs and 2D cameras are
widely used for environment perception. The fusion of LiDAR and camera has
shown impressive performance gains in various perception tasks. One important
basis of LiDAR-camera fusion is the accurate extrinsics between them.

Current traditional methods for LiDAR-camera extrinsics estimation depend on
offline targets and human efforts, while learning-based approaches resort to
iterative refinement for calibration results, posing constraints on their
generalization and application in on-board systems. In this paper, we propose
a novel approach to address the extrinsic calibration problem in a robust,
automatic, and single-shot manner. Instead of directly optimizing extrinsics,
we leverage the consistency learning between LiDAR and camera to implement
implicit re-calibartion. Specially, we introduce an appearance-consistency
loss and a geometric-consistency loss to minimizing the inconsitency between
the attrbutes (e.g., intensity and depth) of projected LiDAR points and the
predicted ones. This design not only enhances adaptability to various
scenarios but also enables a simple and efficient formulation during
inference. We conduct comprehensive experiments on different datasets, and the
results demonstrate that our method achieves accurate and robust performance.
To promote further research and development in this area, we will release our
model and code.

##  1 Introduction

Robust and accurate perception is a crucial issue in self-driving and
robotics. Due to the complexity and high dynamic of the self-driving
environment, using a single sensor can not guarantee robust perception ability
in various scenes. Conversely, multi-sensor fusion (such as LiDAR and camera)
can effectively improve the accuracy and robustness of perception in real-
world environments with the complementary nature of the two sensors [2, 17,
35, 16].In detail, LiDAR provides 3D geometrics and reflection intensity
information within sparse point cloud representation, which allows for
accurate distance measurements. Besides, LiDAR is more robust against
illumination change. Differently, cameras capture abundant color and texture
features within dense RGB images, which provides more fine-grained details.

Figure 1: Projection of lidar points onto image colored with intensity
information. (Top:) Uncalibrated extrinsic. (Middle:) Corrected extrinsic
using our proposed method. (Bottom:) Ground truth. The image with ID 003837 is
in the 00 sequence of the KITTI dataset. Best viewed in color.

The combination of LiDAR and camera provides the feasibility to overcome the
inherent limitations of each sensor. One important basis of fusing these two
heterogeneous sensors lies in the precise extrinsic parameter estimation
(relative rigid body transformation between them).

Early target-based calibration methods [8, 21, 22] primarily rely on
checkboard or specific calibration plates towards accurate results. However,
these methods endure notorious drawbacks, such as time-consuming, labor-
intensive, environment-depended, etc. As a result, these methods are not
suitable for large-scale industrial applications. Unlike target-based methods,
some target-less methods [36, 14] have been explored for online automatic
calibration. However, these methods often have high requirements for geometric
features in the environment and may lack generalizability.

Considering the remarkable success archived by neural networks in computer
vision tasks, some deep learning-based methods are introduced for multi-sensor
extrinsics estimation. RegNet [23] is the first introduced learning-based
method, which employs a multi-scale iterative refinement formulation.
Following RegNet, various methods [18, 10] are built upon this formulation
during training and inference. This complex formulation inevitably poses
challenges for deployment on autonomous vehicles as well as robotic systems.
In addition, current methods mostly evaluate their performance on KITTI scenes
using only one camera, leaving room for improvement in the performance and
generalization ability.

In this paper, we introduce a novel paradigm to address the challenge of
extrinsic calibration between LiDAR and camera systems. Different from
previous methods that depend on specific scene features or improve calibration
performance through iterative refinement, our proposed method does not impose
assumptions on environmental priors, and generates the corrected extrinsics in
an end-to-end, single shot manner. For the first time, we transform the direct
extrinsics optimization problem into a consistency learning problem between
LiDAR and cameras. By minimizing the inconsistency between the projected point
cloud attributes (_e.g._ , appearance attributes as intensity, geometric
attributes as depth) and the predicted ones, our PoseNet is implicitly
enforced to predict accurate extrinsic parameters. Furthermore, this novel
design enables a concise and deployment-friendly subnetwork during inference.
Our method demonstrates that with just the input of paired LiDAR and camera
data, along with the initial, uncalibrated extrinsics, our proposed method
automatically outputs the recalibrated extrinsics.

To summarize, our main contributions are three-fold:

  * •

We propose a consistency constraint to enforce the appearance-consistency and
geometric-consistency of LiDAR and camera, leading to an automatic extrinsic
calibrator across multiple sensors.

  * •

We unveil a novel paradigm for sloving the SE(3) pose problem. The proposed
RobustCalib is concise in its formulation, requiring only a subnetwork during
inference phase and eliminating the need for iterative refinement. This
characteristic is particularly beneficial for industrial applications, where
efficiency and reliability are paramount.

  * •

Extensive experiments on the KITTI, nuScenes and MT-ADV datasets demonstrate
the superior performance and robust generalization ability of our RobustCalib.
These results affirm its applicability and effectiveness in diverse real-world
scenarios.

##  2 Related Work

###  2.1 Target-based Methods

Traditional target-based calibration methods [8, 21, 22, 32, 13, 1] need
certain environment and equipment, such as checkerboards or custom-made
markers placed on the walls. Mishra et al. [19] proposed a LiDAR-camera
extrinsic estimation method on a marker-less planar target by utilizing the
geometric constraints between planes and lines. Toth et al. [29] presented an
automatic calibration method using spheres whose contours can be accurately
detected on LiDAR and camera. Recently, Yan et al. [34] located the LiDAR pose
by adding circular holes around the checkerboard, then using the reprojection
constraint to joint calibrate camera intrinsic and LiDAR-camera extrinsic.
Target-based methods are accurate and commonly used. However, these methods
require tuning a lot of parameters and iterative optimization, which are time-
consuming and costly to implement regularly on a vehicle.

###  2.2 Target-less Methods

Target-less methods [26, 27, 20, 28, 12] usually use general geometric
features (such as lines or edges) from natural scenes to calibrate the
extrinsic parameters based on geometric constraints. Yuan et al. [36] adopted
point cloud voxel cutting and plane fitting for LiDAR edge extraction to
achieve automatic extrinsic calibration in targetless environments. Recently,
Koide et al. [14] presented a fully automatic LiDAR-camera calibration toolbox
without targets. They first estimate the LiDAR camera transformation via
RANSAC [6], and then refine the transformation based on the normalized
information distance. These target-less methods are automated and can be
computed online. However, they still rely on computation-heavy optimization
and require feature-rich environments.

Figure 2: The detailed architecture of the proposed RobustCalib. (a) The
training pipeline. (b) The inference pipeline. Once the training process is
completed, we only use PoseNet for inference to obtain the calibration
extrinsic parameters, which enables point cloud and image alignment. Best
viewed in color.

###  2.3 Learning-based Methods

Considering the recent success of deep learning, learning-based methods [23,
10, 4, 37, 33, 38, 15] often take advantage of CNN to estimate the extrinsic
parameters between LiDAR and cameras. RegNet [23] is a pioneering work, which
leverages CNN to predict the 6-DoF extrinsic parameters. CalibNet [10] is
another significant work, which maximizes the geometric and photometric
consistency between the input image and the point cloud to estimate the 6-DoF
extrinsic parameters in real-time. After that, RGGNet [37] proposes a deep
generative model to construct a tolerance-aware loss considering the
Riemannian geometry. NetCalib [24] optimizes the geometric error between the
depth map from stereo cameras and the depth map from LiDAR projected to
achieve more accurate performance. Besides, Zhu et al. [38] formulated
extrinsic calibration as an optimization problem. They proposed a calibration
quality metric based on semantic features to align pairs of camera and LiDAR
frames. LCCNet [18] is a milestone work, which adopts the cost-volume [25] as
its backbone and achieves good performance via multi-scale refinement.
Recently, Alexander et al. [30] achieved extrinsic calibration by registering
image and point cloud with semantic labels after semantic segmentation. These
methods achieve accurate calibration but leave room for improvement in
accuracy, performance, and reliability in unstructured environments. In this
paper, we make the first attempt to explore the consistency-learning to
achieve more accurate and robust online LiDAR-camera extrinsic calibration.
Different from previous works, the porposed method can be generalized to
various practical application scenarios with a simple yet efficient inference
structure.

##  3 Method

###  3.1 Method Overview

Our goal is to train a learning-based network for online lidar-camera
extrinsic calibration. Leveraging constrainting the consistency (i.e.,
appearance-consistency and geometric-consistency) between lidar and camera,
the proposed method is totally end-to-end, avoiding multiple refinements
during inference. The pipeline of RobustCalib is illustrated in Fig. 2,
RobustCalib consists of three sub-networks: PoseNet, which predicts the
extrinsic $T_{pred}$ between the LiDAR and camera; IntensityNet, which
predicts pseudo intensity image; DepthNet, which predicts depth. Given an RGB
image $I$, a point cloud $P$, and initial extrinsic $T_{init}$ as input.
Firstly, we project the point cloud into the pixel coordinate system to get a
unified pseudo-image representation with 7 channels (i.e., RGB channels
$r,g,b$, camera coordinates $x_{i},y_{i},z_{i}$ and intensity $I_{i}$).
Secondly, we input this pseudo-image to PoseNet, IntensityNet, and DepthNet.
In PoseNet, we input the pseudo-image and get the delta extrinsic prediction.
After that, we multiply the extrinsic prediction with the initial $T_{init}$
to obtain the calibrated extrinsic $T_{pred}$. In IntensityNet, after getting
the predicted pseudo-intensity image, we respectively project the predicted
intensity using the $T_{pred}$ and $T_{gt}$ to construct constraints based on
correspondence and learning appearance-consistency between liDAR and camera.
Similarly, in DepthNet, after getting the predicted pseudo-depth image, we
respectively project the predicted depth using the $T_{pred}$ and $T_{gt}$ to
construct constraints based on geometric-consistency between liDAR and camera.
In the following sections, we discuss the data preparation and representation,
the core idea of lidar-camera consistency learning, and the details of network
architecture.

###  3.2 Data Preparation and Representation

The performance of data-driven learning-based methods improve with the quality
and quantity of data. In this case, we follow previous methods (i.e., RegNet
[23], LCCNet [18] and UniCal [5]) which add random de-calibration within a
reasonable range to the ground truth extrinsic calibration parameters
$T_{LC}$. Thus generating the initial extrinsic $T_{init}=\Delta T\cdot
T_{LC}$ and mitigating insufficiency of training data. In practical autonomous
driving operation scenarios, ranging $\pm 10cm$ for translation and $\pm
1^{\circ}$ for rotation is an empirical estimate of the levels of
perturbation, which is also reported in UniCal [5]. Thus we choose the same
translation and rotation range $\left\\{\pm 10cm,\pm 1^{\circ}\right\\}$.

Considering the inherent different characteristics of multimodal data (image
and point cloud in our case), the efficient representation and fusion of them
is an inevitable challenge. In this paper, we formulate the image input and
the point cloud input into a unified pseudo-image representation.
Specifically, let $I\in\mathbb{R}^{H\times W\times 3}$ be an RGB image and
$P\in\mathbb{R}^{N\times 4}$ be a point cloud. And
$P_{i}=\left[X_{i},Y_{i},Z_{i},I_{i}\right]$, the first three dimensions
represent spatial coordinates of a point, the last dimension $I_{i}$
represents intensity. To establish correspondences between multimodal inputs,
we first project lidar point onto camera’s coordinate system
$c_{i}=\left[x_{i},y_{i},z_{i}\right]$ and pixel coordinate system
$p_{i}=\left[u_{i},v_{i}\right]$, using the initial calibration ${T}_{init}$
and the camera intrinsic $K$. The projection process can be formulated as
follows:

${{c}_{i}^{T}}={{T}_{init}}\cdot{{P}_{i}^{T}}$ (1)

${{z}_{i}\cdot{p}_{i}^{T}}={K\cdot{T}_{init}}\cdot{{P}_{i}^{T}}$ (2)

To obtain the projection of point cloud, we generate an $N$ channel pseudo-
image as network input. In our implementation, $N$ is set to 7, which contains
3 channels for RGB information, and 4 channels for lidar information (i.e.,
intensity $I_{i}$ and camera coordinates $x_{i},y_{i},z_{i}$).

###  3.3 Lidar-Camera Consistency Learning

As mentioned above, we design the consistency constraint (i.e., appearance-
consistency and geometric-consistency) to ensure alignment between the lidar
and camera, which implicitly optimizes the extrinsic between them. Here, we
present the core idea and advantages of our design.

Figure 3: (a) Architecture of the transformer decoder layers of the PoseNet.
(b) Architecture of the IntensityNet. (c) Architecture of the DepthNet.

Appearance-consistency Loss. Takes the unified representation as input, we
first feed it into an encoder-decoder like IntensityNet. The detailed
architecture of IntensityNet will be introduced in the subsection 3.4. After
feature encoding, the IntensityNet aims to predict a pseudo intensity image.
In order to simplify optimization and accelerate convergence, we transform the
intensity regression task into a binary classification task. That is, the
intensity is mapped to 1 or 0 depending on whether the value is greater than a
threshold. The setting of the intensity threshold will be specifically
elaborated in Section 4.2. Correspondingly, the predicted pseudo intensity
image is shaped as $C\times H\times W$ (C is set as 2), which is obtained by
an intensity binary classifier.

The PoseNet parallelly takes the unified representation as input, and outputs
extrinsic in the format of translation in meters and rotation in Euler angles.
Then the output is transformed into a rotation translation matrix $T_{delta}$,
multiplied with $T_{init}$, and obtained the $T_{pred}$. The detailed
architecture of PoseNet is illustrated in subsection 3.4. In order to
construct correspondence and learning appearance-consistency between lidar and
camera, we respectively project the point cloud into the pixel coordinate
system using the $T_{pred}$ and $T_{gt}$, and sample values from the predicted
pseudo intensity image. The sampled intensity denoted as $I^{pred}_{s}$ and
$I^{gt}_{s}$ respectively and supervised by the binary intensity label
$I_{gt}$ via cross-entropy loss. This process can be formulated as:

${L_{I}}=L_{CE}(I^{pred}_{s},I_{gt})+L_{CE}(I^{gt}_{s},I_{gt})$ (3)

Geometric-consistency Loss. Formulated as a similar paradigm of the
appearance-consistency learning, the geometric-consistency learning takes
depth as the medium. The DepthNet, which is also an encoder-decoder structure,
predicts a one-channel pseudo depth image. Then the LiDAR points are projected
using $T_{pred}$ and $T_{gt}$, and the corresponding depth values are sampled
by the pixel coordinates, denoted as $I^{pred}_{s}$ and $I^{gt}_{s}$,
respectively. We take the depth label (denoted as $D_{gt}$) obtained by the
extrinsic ground truth projection as the learning target and formulate the
objective function as:

${L_{D}}=L_{1}(D^{pred}_{s},D_{gt})+L_{1}(D^{gt}_{s},D_{gt})$ (4)

User-friendly Inference Model. Notably, in the training process, we use the
appearance-consistency and geometric-consistency to constrain the extrinsic
prediction. However, in the inference stage, IntensityNet and DepthNet are not
needed. The inference pipeline is concise and formulated in a single-branch
pipeline. Our inference process only takes the lidar-camera information and
initial extrinsic $T_{init}$ as input, and automatically outputs the corrected
extrinsic $T_{pred}$ in a single-shot manner, which shows advantages in
application compared with previous iterative refinement methods [23, 18].

###  3.4 Network Architecture

PoseNet. The PoseNet parallelly takes the unified representation as input, and
outputs extrinsic in the format of translation in meters and rotation in Euler
angles. For PoseNet, we formulate it as a ResNet-50 [9] based encoder and
transformer-based [31] decoder architecture. The decoder part takes the
encoded feature ($B\times C\times h\times w$, and reshaped as $B\times
N_{1}\times C$) from lidar and image, and the randomly initialized queries
(shaped as $B\times N_{2}\times C$) as inputs. We first compute self-attention
score of features and queries separately, as well as cross-attention score
between them. Then the queries equipped with calibration information are
decoded into query features (shaped as $B\times N_{1}\times C$) by a feed-
forward network (FFN). Two parallel Linear layers are adopted to transform the
query feature to three-channel outputs that respectively represent rotation in
Euler angles and translation in meters. Finally, the rotation and translation
are represented as a rotation translation matrix $T_{delta}$, which is
multiplied with the initial $T_{init}$ to obtain the corrected extrinsic
$T_{pred}$. The architecture of PoseNet, IntensityNet, and DepthNet can be
seen in Fig. 3.

IntensityNet. The IntensityNet is an encoder-decoder architecture. The encoder
part is based on ResNet-18 [9], and the first convolutional layer is adjusted
to adapt the unified input with $N$ channels. The decoder part is composed of
convolutional-based up-sampling layers, gradually restoring the encoded
features to the original resolution. The final classifier is configured to
output two-channel intensity prediction ($B\times 2\times h\times w$).

DepthNet. The DepthNet enjoys a similar architecture as the IntensityNet, the
only difference is that the final classifier is adjusted to predict one-
channel depth ($B\times 1\times h\times w$).

Figure 4: Illustrations of intensity distribution in both the KITTI and the
MT-ADV dataset. The first row is data collected from the KITTI dataset, with
frames from sequence 18, 21, and 02 arranged from left to right. The second
row is data sampled from the MT-ADV dataset.

##  4 Experiments

In this section, we evaluate our proposed calibration approach on the KITTI
odometry dataset, nuScenes dataset and MT-ADV dataset. We detail the data
processing, training details, quantitative experiments, ablation study and
visualization results.

###  4.1 Dataset

##### KITTI dataset.

KITTI [7] is widely used as a reference dataset in current mainstream
calibration methods. In this paper, we use the odometry branch of the KITTI
dataset to verify the proposed method. KITTI Odometry dataset consists of 21
sequences from different scenarios using 64-line Velodyne 3D LiDAR (10 HZ) and
camera with a resolutions of 1392 $\times$ 1032, and provides accurate ground
truth extrinsic calibration parameters. Following LCCNet [18] setting, we
split the KITTI dataset as follows: 01-21 for training (39011 frames) and 00
for test (4541 frames).

##### nuScenes dataset.

We also conducted experiments on the nuScenes dataset [3], which is a large-
scale autonomous driving dataset. Specifically, the perception system in
nuScenes consists of a 32-line LiDAR and 6 surround-view cameras with a
resolutions of 1600 $\times$ 900 and provides accurate ground truth extrinsic
calibration parameters. nuScenes dataset contains 1640310 frames for training
and 351006 frames for test from different scenarios.

##### MT-ADV dataset.

To further evaluate the generalization ability and robustness of the proposed
method, we also conducted experiments on the Meituan autonomous delivery
vehicles (MT-ADV) dataset. Specifically, the perception system in MT-ADV
consists of a 64-line LiDAR and 6 surround-view cameras. MT-ADV dataset
contains 543057 frames for training and 87904 frames for test. The ground
truth extrinsics between LiDAR and cameras are generated from Meituan’s
calibration infrastructure. Unlike the KITTI dataset, MT-ADV dataset includes
more complex real-word scenarios and a larger number of data samples, which
also brings more difficulties and challenges. Besides, MT-ADV dataset
encompasses multiple cameras, which more meets the requirements of the
practical application scenarios.

###  4.2 Implementation Details.

##### Training details.

The proposed method is trained totally end-to-end. We use SGD Optimizer during
training with the parameter weight decay set as $0.0001$ and momentum set as
$0.9$. We train the network with an initial learning rate $0.01$ and adopt the
consine decay strategy. For the KITTI dataset, the network is trained for
$240,000$ iterations.

##### Intensity threshold setting.

In the section of apperance-consistency learning, we formulate the intensity
regression problem as a binary classification problem. Therefore, there needs
a reasonable threshold to classify the LiDAR points depending on their
intensity value. In our experiment, we set this threshold based on the
statistic of the intensity distribution in different datasets. Here we present
several examples intensity distribution in the KITTI dataset and the MT-ADV
dataset in Fig. 4. The first row is collected from the KITTI dataset, the
second row is collected from the MT-ADV dataset. Based on the intensity
distribution of LiDAR points, we can see that 30 is a reasonable threshold for
split in KITTI, as well as 10 for the MT-ADV dataset.

Method | Translation absolute Error (cm) | Rotation absolute Error (∘)
---|---|---
mean | X | Y | Z | mean | Roll | Pitch | Yaw
Regnet [23] | 6.00 | 7.00 | 7.00 | 4.00 | 0.28 | 0.24 | 0.25 | 0.36
Calibnet [10] | 4.34 | 4.20 | 1.60 | 7.22 | 0.41 | 0.18 | 0.90 | 0.15
NetCalib [24] | 1.69 | 1.20 | 2.77 | 1.10 | 0.09 | 0.04 | 0.16 | 0.09
LCCNet [18] | 1.292 | - | - | - | 0.173 | - | - | -
DXQ-Net [11] | 1.425 | 0.754 | 0.476 | 1.091 | 0.084 | 0.049 | 0.046 | 0.032
RobustCalib (ours) | 0.806 | 0.820 | 0.740 | 0.860 | 0.046 | 0.034 | 0.037 | 0.068
Table 1: Comparison with current state-of-the-art deep learning-based calibration methods on KITTI dataset. Figure 5: Visualization of calibration results from different scenes of KITTI dataset. The first column shows the projecion of LiDAR points onto the image using the uncalibrated extrinsics. The second column shows the calibrated extrinsics produced by our method. The third column shows the ground truth results. The green box show the mis-alignmnet. Better view zoom in with color. Method | Translation absolute Error (cm) | Rotation absolute Error (∘)
---|---|---
mean | X | Y | Z | mean | Roll | Pitch | Yaw
LCCNet [18] | 2.218 | 2.430 | 2.176 | 2.048 | 0.154 | 0.149 | 0.124 | 0.189
RobustCalib (ours) | 1.600 | 2.000 | 1.530 | 1.270 | 0.114 | 0.103 | 0.085 | 0.155
Table 2: Comparison with state-of-the-art learning-based calibration methods
on nuScenes dataset. Figure 6: Visualization of calibration results under
multiple views from MT-ADV dataset. The first column shows the projecion of
LiDAR points onto the image using the uncalibrated extrinsics. The second
column shows the calibrated extrinsics produced by our method. The third
column shows the ground truth results. The green box show the mis-alignmnet.
Better view zoom in with color.

###  4.3 Evaluation Metrics

In this section, we introduce the evaluation metrics for calibration
parameters. The metric of mean absolute error is used for translation and
rotation. For translation, we compute the translation verctor’s absolute error
in X, Y, Z directions and the mean value. For rotation, we transform the
rotation metric to the formulation of Euler angles and compute the angle error
of Roll, Pitch, Yaw axes.

###  4.4 Overall Results

####  4.4.1 Results on KITTI.

The calibration results for our proposed RobustCalib on KITTI are shown in
Table 1. We can observe that RobustCalib achieves a mean translation error of
0.806cm (x, y, z: 0.820cm, 0.740cm, 0.860cm), a mean angle error of
$0.046^{\circ}$ (roll, pitch, yaw: $0.034^{\circ}$, $0.037^{\circ}$,
$0.068^{\circ}$), which is a new state-of-the-art performance in both
translation and rotation metrics. Compared to the state-of-the-art method
DXQNet [11], our method achieves a nearly 50% reduction in both mean
translation error and mean angle error. Compared to LCCNet [18], the mean
angle error of RobustCalib is reduced by approximately 3 times. It is worth
noting that our RobustCalib favorably outperforms other learning-based methods
with a single shot manner, while other methods (i.e., RegNet [23], LCCNet [18]
and UniCal [5]) rely on iterative refinement during training and inference
process.

As shown in Fig.5, we visualize the calibration results of different scenarios
on KITTI dataset. Here, the LiDAR points are colored with intensity values. It
can be seen that our proposed RobustCalib is capable of providing re-
calibrated extrinsic parameters that align the LiDAR point cloud projection
onto the corresponding camera images. Particularly, for the obvious
uncalibrated results (green box in 1st column), after applying our
RobustCalib, achieving the almost same results as the ground truth (2nd and
3rd columns in Fig.5).

Method | Translation absolute Error (cm) | Rotation absolute Error (∘)
---|---|---
mean | X | Y | Z | mean | Roll | Pitch | Yaw
w/o geometric-consistency | 1.040 | 1.020 | 0.840 | 1.260 | 0.047 | 0.034 | 0.042 | 0.066
w/o appearance-consistency | 3.796 | 4.980 | 5.030 | 1.380 | 0.197 | 0.055 | 0.035 | 0.502
RobustCalib (ours) | 0.806 | 0.820 | 0.740 | 0.860 | 0.046 | 0.034 | 0.037 | 0.068
Table 3: Ablation study results of RobustCalibe on KITTI dataset.

####  4.4.2 Results on nuScenes.

The calibration results for our proposed RobustCalib on nuScenes are shown in
Table 2. We can observe that RobustCalib achieves a mean translation error of
1.600cm (x, y, z: 2.000cm, 1.530cm, 1.270cm), a mean angle error of
$0.114^{\circ}$ (roll, pitch, yaw: $0.103^{\circ}$, $0.085^{\circ}$,
$0.155^{\circ}$). Compared to LCCNet [18], the mean translation and angle
error of RobustCalib are reduced by 0.618cm and $0.04^{\circ}$.

####  4.4.3 Results on MT-ADV.

To further demonstrate the performance of the proposed method, we conducted
experiments on the MT-ADV dataset. Compared with KITTI dataset, this dataset
is more challenging. In KITTI dataset, our RobustCalib and other high-
performance methods only calibrated the LiDAR and the left color camera (i.e.,
one-to-one). However, in the MT-ADV dataset, in actual operational scenarios,
we calibrate one LiDAR with multiple cameras, which better meets the
requirements of the practical application scenarios. The proposed method shows
general abilities in calibrating different cameras. We achieve a mean
translation error of 1.38cm (x, y, z: 1.55cm, 1.45cm, 1.16cm), a mean angle
error of $0.103^{\circ}$ (roll, pitch, yaw: $0.126^{\circ}$, $0.107^{\circ}$,
$0.077^{\circ}$).

Except KITTI dataset, we also visualize the qualitative calibration results on
MT-ADV dataset, which requires to align one LiDAR to multiple cameras. As
shown in Fig. 6, the proposed RobustCalib shows robust and accurate re-
calibration ability under different viewpoints. Specifically, for objects such
as trees and poles, the uncalibrated projection results present large offsets.
However, our method achieves effective alignment between point clouds and
images from different viewpoints.

###  4.5 Ablation Study

In this section, the influence of appearance-consistency learning and
geometric-consistency learning on the calibration results is explored. We
independently disable the appearance-consistency loss and the geometric-
consistency loss in the experimental setting. These experiments are conducted
on KITTI and the corresponding results are listed in Table. An observation is
that when disabling appearance-consistency learning in the calibration
process, there appears an obvious performance drop. When the geometric-
consistency learning is disabled, there is a slight performance loss. The main
reason is that the intensity from LiDAR sensor provides more helpful visual
features and patterns, which is more informative than that of depth. When
enabling the appearance-consistency learning and geometric-consistency
learning together, our method achieves better calibration results.

##  5 Conclusion

In this paper, we present a novel approach to estimate extrinsic calibration
between LiDAR and cameras. During the training process, we design an
appearance-consistency loss and a geometric-consistency loss for implicitly
optimizing the extrinsic parameters. During the inference phase, our pipeline
exhibits a more concise architecture within a subnetwork. Given a pair of
LiDAR point cloud and camera images, as well as the uncalibrated extrinsic
between them, our method directly infers the 6 DOF of the extrinsics
calibration. On KITTI dataset, our method yields a mean calibration error of
0.806cm for translation and $0.046^{\circ}$ for rotation, which achieves a new
state-of-the-art. Furthermore, the proposed RobustCaliob still demonstrates
superior performance on the MT-ADV dataset, which involves the calibration
between LiDAR and multiple surround-view cameras with a larger number of
samples. We believe our study will provide valuable insights and contribute to
extrinsic calibration for the community.

##  6 Acknowledgement

This work is supported by National Key R&D Program of China (No.
2022ZD0118700).

## References

  * An et al. [2020] Pei An, Tao Ma, Kun Yu, Bin Fang, Jun Zhang, Wenxing Fu, and Jie Ma.  Geometric calibration for lidar-camera system fusing 3d-2d and 3d-3d point correspondences.  _Optics express_ , 28(2):2122–2141, 2020.
  * Bai et al. [2022] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai.  Transfusion: Robust lidar-camera fusion for 3d object detection with transformers.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 1090–1099, 2022.
  * Caesar et al. [2020] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 11621–11631, 2020.
  * Cattaneo et al. [2019] Daniele Cattaneo, Matteo Vaghi, Augusto Luis Ballardini, Simone Fontana, Domenico Giorgio Sorrenti, and Wolfram Burgard.  Cmrnet: Camera to lidar-map registration.  In _2019 IEEE Intelligent Transportation Systems Conference (ITSC)_ , pages 1283–1289, 2019.
  * Cocheteux et al. [2023] Mathieu Cocheteux, Aaron Low, and Marius Bruehlmeier.  Unical: a single-branch transformer-based model for camera-to-lidar calibration and validation.  _arXiv preprint arXiv:2304.09715_ , 2023.
  * Fischler and Bolles [1981] Martin A Fischler and Robert C Bolles.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.  _Communications of the ACM_ , 24(6):381–395, 1981.
  * Geiger et al. [2012a] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In _CVPR_ , pages 3354–3361. IEEE, 2012a.
  * Geiger et al. [2012b] Andreas Geiger, Frank Moosmann, Ömer Car, and Bernhard Schuster.  Automatic camera and range sensor calibration using a single shot.  In _2012 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 3936–3943. IEEE, 2012b.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  pages 770–778, 2016.
  * Iyer et al. [2018] Ganesh Iyer, R Karnik Ram, J Krishna Murthy, and K Madhava Krishna.  Calibnet: Geometrically supervised extrinsic calibration using 3d spatial transformer networks.  In _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 1110–1117. IEEE, 2018.
  * Jing et al. [2022] Xin Jing, Xiaqing Ding, Rong Xiong, Huanjun Deng, and Yue Wang.  Dxq-net: differentiable lidar-camera extrinsic calibration using quality-aware flow.  In _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 6235–6241. IEEE, 2022.
  * Kang and Doh [2020] Jaehyeon Kang and Nakju L Doh.  Automatic targetless camera–lidar calibration by aligning edge with gaussian mixture model.  _Journal of Field Robotics_ , 37(1):158–179, 2020.
  * Kim and Park [2020] Eung-su Kim and Soon-Yong Park.  Extrinsic calibration between camera and lidar sensors by matching multiple 3d planes.  _Sensors_ , 20(1):52, 2020.
  * Koide et al. [2023] Kenji Koide, Shuji Oishi, Masashi Yokozuka, and Atsuhiko Banno.  General, single-shot, target-less, and automatic lidar-camera extrinsic calibration toolbox.  2023\.
  * Kuangyi Chen and Xia [2022] Wen Yang Lei Yu Sebastian Scherer Kuangyi Chen, Huai Yu and Gui-Song Xia.  I2D-Loc: Camera Localization via Image to LiDAR Depth Flow.  In _ISPRS Journal of Photogrammetry and Remote Sensing_ , pages 209–221, 2022.
  * Liang et al. [2018] Ming Liang, Bin Yang, Shenlong Wang, and Raquel Urtasun.  Deep continuous fusion for multi-sensor 3d object detection.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 641–656, 2018.
  * Liu et al. [2023] Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela L Rus, and Song Han.  Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 2774–2781. IEEE, 2023.
  * Lv et al. [2021] Xudong Lv, Boya Wang, Ziwen Dou, Dong Ye, and Shuo Wang.  Lccnet: Lidar and camera self-calibration using cost volume network.  In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 2888–2895, 2021.
  * Mishra et al. [2020] Subodh Mishra, Gaurav Pandey, and Srikanth Saripalli.  Extrinsic calibration of a 3d-lidar and a camera.  In _2020 IEEE Intelligent Vehicles Symposium (IV)_ , pages 1765–1770, 2020.
  * Pandey et al. [2015] Gaurav Pandey, James R McBride, Silvio Savarese, and Ryan M Eustice.  Automatic extrinsic calibration of vision and lidar by maximizing mutual information.  _Journal of Field Robotics_ , 32(5):696–722, 2015.
  * Park et al. [2014] Yoonsu Park, Seokmin Yun, Chee Sun Won, Kyungeun Cho, Kyhyun Um, and Sungdae Sim.  Calibration between color camera and 3d lidar instruments with a polygonal planar board.  _Sensors_ , 14(3):5333–5353, 2014.
  * Pusztai and Hajder [2017] Zoltan Pusztai and Levente Hajder.  Accurate calibration of lidar-camera systems using ordinary boxes.  In _ICCV_ , pages 394–402, 2017.
  * Schneider et al. [2017] Nick Schneider, Florian Piewak, Christoph Stiller, and Uwe Franke.  Regnet: Multimodal sensor registration using deep neural networks.  In _2017 IEEE intelligent vehicles symposium (IV)_ , pages 1803–1810. IEEE, 2017.
  * Shan [2020] Amnir; Vivet Damien; Prabhakar Yadu Shan, Wu; Hadachi.  Netcalib: A novel approach for lidar-camera auto-calibration based on deep learning.  In _Proceedings of the 25th International Conference on Pattern recognition 2020_. IEEE, 2020.
  * Sun et al. [2018] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.  Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8934–8943, 2018.
  * Tamas and Kato [2013] Levente Tamas and Zoltan Kato.  Targetless calibration of a lidar-perspective camera pair.  In _ICCV_ , pages 668–675, 2013.
  * Tamas et al. [2014] Levente Tamas, Robert Frohlich, and Zoltan Kato.  Relative pose estimation and fusion of omnidirectional and lidar cameras.  In _ECCV_ , pages 640–651. Springer, 2014.
  * Taylor et al. [2015] Zachary Taylor, Juan Nieto, and David Johnson.  Multi-modal sensor calibration using a gradient orientation measure.  _Journal of Field Robotics_ , 32(5):675–695, 2015.
  * Tóth et al. [2020] Tekla Tóth, Zoltán Pusztai, and Levente Hajder.  Automatic lidar-camera calibration of extrinsic parameters using a spherical target.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 8580–8586. IEEE, 2020.
  * Tsaregorodtsev et al. [2022] Alexander Tsaregorodtsev, Johannes Muller, Jan Strohbeck, Martin Herrmann, Michael Buchholz, and Vasileios Belagiannis.  Extrinsic camera calibration with semantic segmentation.  In _2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC)_ , pages 3781–3787. IEEE, 2022.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2017] Weimin Wang, Ken Sakurada, and Nobuo Kawaguchi.  Reflectance intensity assisted automatic and accurate extrinsic calibration of 3d lidar and panoramic camera using a printed chessboard.  _Remote Sensing_ , 9(8):851, 2017.
  * Wang et al. [2020] Weimin Wang, Shohei Nobuhara, Ryosuke Nakamura, and Ken Sakurada.  Soic: Semantic online initialization and calibration for lidar and camera.  _arXiv preprint arXiv:2003.04260_ , 2020.
  * Yan et al. [2023] Guohang Yan, Feiyu He, Chunlei Shi, Pengjin Wei, Xinyu Cai, and Yikang Li.  Joint camera intrinsic and lidar-camera extrinsic calibration.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 11446–11452. IEEE, 2023.
  * Yoo et al. [2020] Jin Hyeok Yoo, Yecheol Kim, Jisong Kim, and Jun Won Choi.  3d-cvf: Generating joint camera and lidar features using cross-view spatial feature fusion for 3d object detection.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVII 16_ , pages 720–736. Springer, 2020.
  * Yuan et al. [2021] Chongjian Yuan, Xiyuan Liu, Xiaoping Hong, and Fu Zhang.  Pixel-level extrinsic self calibration of high resolution lidar and camera in targetless environments.  _IEEE Robotics and Automation Letters_ , 6(4):7517–7524, 2021.
  * Yuan et al. [2020] Kaiwen Yuan, Zhenyu Guo, and Z Jane Wang.  Rggnet: Tolerance aware lidar-camera online calibration with geometric deep learning and generative model.  _IEEE Robotics and Automation Letters_ , 5(4):6956–6963, 2020.
  * Zhu et al. [2020] Yufeng Zhu, Chenghui Li, and Yubo Zhang.  Online camera-lidar calibration with sensor semantic information.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 4970–4976. IEEE, 2020.


Amplitude-Based Sequential Optimization of Energy Harvesting with Reconfigurable Intelligent Surfaces

This paper was supported by Digital Futures.

Morteza Tavana^*, Meysam Masoudi^†, and Emil Björnson^*

^*School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, Stockholm, Sweden

^†Ericsson, Global AI Accelerator (GAIA) unit, Sweden

E-mail: ^*{morteza2<EMAIL_ADDRESS><EMAIL_ADDRESS>
Reconfigurable Intelligent Surfaces (RISs) have gained immense popularity in recent years because of their ability to improve wireless coverage and their flexibility to adapt to the changes in a wireless environment. These advantages are due to RISs’ ability to control and manipulate radio frequency (RF) wave propagation. RISs may be deployed in inaccessible locations where it is difficult or expensive to connect to the power grid. Energy harvesting can enable the RIS to self-sustain its operations without relying on external power sources. In this paper, we consider the problem of energy harvesting for RISs in the absence of coordination with the ambient RF source. We consider both direct and indirect energy harvesting scenarios and show that the same mathematical model applies to them. We propose a sequential phase-alignment algorithm that maximizes the received power based on only power measurements. We prove the convergence of the proposed algorithm to the optimal value  under specific circumstances. Our simulation results show that the proposed algorithm converges to the optimal solution in a few iterations and outperforms the random phase update method in terms of the number of required measurements.

Energy harvesting, reconfigurable intelligent surface, phased array, zero-energy devices.

§ INTRODUCTION

Future wireless networks should provide seamless connectivity for the rapidly growing number of devices and services. If wireless networks are implemented in the same manner as before, the energy consumption would keep increasing dramatically with the traffic volume. From both carbon footprint and energy consumption perspectives, it is a necessity to develop energy-saving techniques that can be implemented in the network nodes including low-power and zero-energy devices <cit.>.

Traditional wireless networks  have no control over the radio propagation environment.
Providing connectivity for regions with low *snr comes at cost of deploying more sophisticated transmission schemes, more radio resources such as antennas and spectrum and consequently consuming more energy <cit.>.

With the emergence of the *ris, several limiting factors associated with the propagation environment can be eliminated. A *ris  can manipulate the propagation environment to increase the signal strength in the desired direction and guide the electromagnetic waves toward the receiver via engineered reflections <cit.>. The *ris is primarily envisioned for providing coverage for regions that are blocked by objects<cit.>.
For instance, a *ris can be deployed in a city with dense buildings and poor *los conditions, or it can also be deployed in homes, where the walls obstruct the signal path.

In the presence of a wired power supply, *ris may not have clear benefits compared to relays and small cells, and the respective advantages are debatable. However, if the *ris is self-sustainable, it opens up new possibilities for deployment with no competition. This is where energy harvesting comes into play. By harvesting energy from *rf signals that are already present in the environment, the *ris can operate without relying on external power sources <cit.>. This is particularly beneficial in situations where the *ris is deployed in remote or inaccessible locations where it can be difficult or expensive to provide a continuous power source. In <cit.>, the authors developed a model for the *ris power consumption that is based on the number of elements and their phase resolution. Higher phase resolution in the *ris increases the complexity and the power consumption <cit.>.

The study <cit.> considers a hybrid-relaying scheme empowered by a self-sustainable *ris to simultaneously improve the downlink energy transmission and uplink information transmission. The authors proposed time-switching and power-splitting schemes for *ris operation. The paper <cit.> proposes a novel framework for *wpt system using a *ris to improve power transfer efficiency. The proposed framework employs independent beamforming to replace conventional joint beamforming, which results in higher efficiency.

The state-of-the-art techniques consider perfect *csi at the *ris, which is obtained by coordination between the transmitters and the *ris. The existing solutions require extra hardware (i.e., multiple *rf receiver to measure both amplitude and phase to obtain *csi) and signaling, which in turn increases the energy consumption and cost of the *ris.

In this paper, we consider a different scenario, where the *ris must configure itself without *rf receivers. The proposed method makes power measurements in the energy harvesting units and uses them to maximize the harvesting power.

Hardware structure of a RIS.

The main contributions of this paper as compared to the existing works are as follows.

* We propose a sequential phase-alignment algorithm to maximize the received power at the harvesting unit based on power measurements.

* We prove the optimality of the proposed algorithm analytically.

* Our simulation results show that the proposed algorithm greatly outperforms the benchmark random phase-update method in terms of the number of required measurements to achieve the optimum.

The remainder of this paper is organized as follows. Section <ref> describes the *ris hardware architecture, two different energy harvesting schemes in the *ris, and the phase alignment problem. Section <ref> describes the proposed abstract model of the *ris operation and the proposed solution. Simulation results are presented in Section <ref>, while Section <ref> provides our conclusions.

Notations: We denote sets by upper-case script letters, e.g., $\mathcal{S}$. The only exceptions are the sets of natural numbers, real numbers, and complex numbers that we  represent with $\mathbb{N}$, $\mathbb{R}$, and $\mathbb{C}$, respectively.  The cardinality of a set $\mathcal{S}$ is represented by $|\mathcal{S}|$. Vectors are indicated by lower-case bold-face letters, e.g., $\bm{x}$, and $x_{i}$ denotes the $i$th element of $\bm{x}$. We represent matrices by upper-case bold-face letters, e.g., $\bm{A}$. We also use $\delequal$ to indicate an equal by definition sign. With $\left(\cdot\right)^{\mathsf{T}}$, we denote the transpose operator. We denote the imaginary unit by $j\delequal\sqrt{-1}$. We represent the conjugate of a complex number $z$ with $z^{*}$. However, we denote the optimal solution with the superscript ${\star}$, e.g., $x^{\star}$. Also, the operation $\operatorname{Arg}{\left(z\right)}$ returns the single-valued argument of $z$ that lies within the interval $(-\pi, \pi]$, while $\arg{\left(z\right)}$ returns the set of all possible values of the argument of $z$.

§ PROBLEM DESCRIPTION

This section presents 1) the *ris device architecture, 2) the energy harvesting schemes, and 3) the phase alignment problem for *ris-assisted energy harvesting.

§.§ RIS Hardware Architecture

Fig. <ref> illustrates the hardware architecture of a typical *ris. The front layer consists of several metal elements that are printed on a dielectric substrate and arranged in a two-dimensional array to reflect the incoming signals. For each patch element, there is a controllable circuit that is used to adjust the phase of the reflected signals and direct the signal in a desired direction.
There is a control circuit board that can control the reflection amplitudes and phase shifts of the patch elements. The *ris controller can communicate with the network components such as *bs via a connectivity interface <cit.>.  Finally, the *ris requires a power supply to adjust the phases and then maintain the desired reflection state.

§.§ Energy Harvesting Schemes

In general, an *ris can harvest *rf energy directly or indirectly from an ambient *rf source. We will describe these scenarios below and later show that they lead to system models of the same kind.

§.§.§ Direct Energy Harvesting

The *ris elements are capable of reflecting and receiving the incident *em waves. During the harvesting phase, the *ris operates in the reception mode, where it combines the received signals from each element with some phase shifts.[There is a type of metasurface implementation called holographic beamforming that can pass the incident EM waves from one side to the other. The metamaterial can add adjustable phase shifts, similar to a phased array, but with a different implementation <cit.>.] The *ris can use the harvested energy to fully sustain its operation or, if the energy is insufficient, it can decrease the consumption from other energy sources, as a first step toward achieving a zero-energy *ris system. The abstract model of the direct energy harvesting operation is shown in Fig. <ref>.

§.§.§ Indirect Energy Harvesting

The *ris is generally deployed and designed to reflect the *em waves towards a desired location (typically a receiver location). Inspired by this principle, the energy harvesting device can alternatively be deployed in front of the *ris (at a short distance) and the phases of the *ris elements can be aligned so they combine constructively at the location of the energy harvesting device. This device then can return the energy to the *ris via a cable. The concept of indirect energy harvesting is demonstrated in Fig. <ref>.

Energy harvesting schemes: (a) direct (b) indirect.

§.§ Phase Alignment Problem for the Energy Harvesting

We consider a scenario of *eh from an ambient *rf source by an *ris with no prior  *csi, and there is no coordination between the *ris and the *rf transmitter. We assume that the transmit power and the location of the transmitter are unknown.[This scenario is more practical (compared to a scenario with coordination between the transmitter and the *ris) as the transmitter may not be designed to coordinate the *csi with the *ris or only do it when it requests that the *ris is supporting its data transmissions.]

For the direct (indirect) scheme, we assume the measured *rf power by the *ris (*eh device) has the following expression <cit.>

\begin{equation}\label{eq:Power_func}
y =\left|\sum_{n=1}^{N} z_{n}e^{j\vartheta_{n}}\right|^{2},
\end{equation}

where $z_{n}\in\mathbb{C}$ for each $0\leq n\leq N$. In (<ref>), $z_{n}$ not only includes all channel gains between transmitters and the energy harvester (except the adjustable phase shift $\vartheta_{n}$), but also takes into account the transmission power. Also, $\vartheta_{n}\in[0, 2\pi)$ is the adjustable phase shift of the $n$th element of the *ris. Also $y$ represents the measured received power that is obtained from the reading of the power in the *eh device<cit.>.[Power measurements can be obtained from the input of the energy harvesting unit using a circuit such as a voltmeter. Alternatively, power measurements can be applied at the output of the energy harvesting unit by compensating for the nonlinear harvesting conversion efficiency.]

For direct and indirect energy harvesting schemes, and for all $1\leq n\leq N$, we have

\begin{equation} \label{eq:z_definition}
P_{\text{t}}h_{n}, \qquad &\text{Direct EH}\\
P_{\text{t}}h_{n}g_{n}, \qquad &\text{Indirect EH},
\end{cases}
\end{equation}

where $h_{n}\in \mathbb{C}$ and $g_{n}\in \mathbb{C}$ are the channel gains from the transmitter to the *ris element $n$ and from the *ris element $n$ to the *eh device in the indirect case, respectively. Also, $P_{\text{t}}$ is the transmit power.

We assume no *csi is available at the *ris controller (i.e., $z_{n}$ is unavailable to the *ris). This scenario is more general compared to a scenario with coordination between the transmitters and the *ris, since the transmitter might lack a protocol to reach the *ris and can even be unaware of its existence.

The values of $\left\{z_{n}\right\}_{n=1}^{N}$ depend on the geometry and propagation environment and can be estimated by measuring amplitude and phase using *rf receiver circuits. Since the *ris lack such *rf chains, the *ris cannot estimate the amplitudes and phases. Hence, the values of $\left\{z_{n}\right\}_{n=1}^N$ will remain unknown to the *ris. On the other hand, with power measurement at the *eh device, the *ris can measure the combined power from all elements for any feasible phase configuration.

The general optimization problem is to maximize the received *rf power[Note that the harvested *rf power is a nonlinear function of the received *rf power that is called the conversion efficiency function. We consider the problem of choosing optimal phase shift $\bm{\vartheta}^{\star}$ for the *ris elements to maximize the harvesting *rf power. However, since the conversion efficiency function is generally an increasing function, the optimal solution for maximization of the harvesting *rf power is equivalent to the one for the maximization of the measured received *rf power.] by finding proper phase shifts (i.e.,  $\bm{\vartheta}{\delequal}\left[\vartheta_{1},\dots,\vartheta_{N}\right]^{\mathsf{T}}$) for the *ris array.

§ PROPOSED PHASE ALIGNMENT SCHEME

In this section, we investigate the problem of received *rf power maximization using a dynamic sequence of power measurements. We propose a model for the RIS energy harvesting operations and an algorithm to find the optimal phase of *ris elements.

§.§ An Abstract Model of the Proposed RIS Operation

A simplified model of the operation of an *ris with an energy harvesting module is shown in Fig. <ref>. The network entity manager can assess the network environment, including but not limited to network demand, *snr, and power measurements from the *ris, and determine if the *ris should function to provide connectivity or to harvest energy. Within the *ris controller, the preferred phase of each element can be determined based on functionality, and the controller can adjust the phase of the elements accordingly. The energy harvester can be located either outside or inside the *ris surface. It sends power measurements to a phase alignment algorithm implemented in the network entity manager module. These values can be used by the network entity manager to decide whether the *ris should be in energy harvesting or data transmission mode. After reaching an appropriate phase configuration, the harvested energy can be fed back to the power supply to be used by the *ris.

The abstract model of the operation of a *ris with an energy harvesting module.

§.§ Proposed Algorithm

We consider an ideal scheme, where the *ris is capable of adding continuous phase shifts to the incident *em waves.
First, we start with stating Lemma <ref> that provides a mechanism to find the optimal phase shifts in the case of known *csi, then based on that, we develop our proposed algorithm that require no a priori *csi.

Let $f:\mathbb{R}^{N}\rightarrow \mathbb{R}$ be the function

\begin{equation}\label{eq:def_f}
f{\left(\bm{\vartheta}\right)}\delequal\left|\sum_{n=1}^{N} z_{n}e^{j\vartheta_{n}}\right|^{2},
\end{equation}

where $z_{n}\in \mathbb{C}$ for all $1\leq n\leq N$. The optimal variable $\bm{\vartheta}^{\star}\delequal\left(\vartheta_{1}^{\star}, \dots, \vartheta_{N}^{\star}\right)$ that maximizes $f{\left(\cdot\right)}$ is given by

\begin{equation}
\vartheta_{n}^{\star}=\vartheta_{0}-\arg\left(z_{n}\right),
\end{equation}

where $\vartheta_{0}\in \mathbb{R}$, and

\begin{equation}
f{\left(\bm{\vartheta}^{\star}\right)} = \left(\sum_{n=1}^{N}\left|z_{n}\right|\right)^{2}
\end{equation}

is the maximum value of $f{\left(\cdot\right)}$.

Let $f:\mathbb{R}\rightarrow \mathbb{R}$ be a function of the form

$$f{\left(\vartheta\right)}=\left|z_{0}+ ze^{j\vartheta}\right|^{2},$$

where $z_{0}, z\in \mathbb{C}$. Without knowing the explicit values of $z_{0}$ and $z$, the optimal variable $\vartheta^{\star}$ that maximizes $f{\left(\cdot\right)}$ can be computed as

\begin{equation}
\vartheta^{\star}=\arg{\left(x_{2}+j x_{3}\right)},
\end{equation}

where $\bm{x}\delequal \bm{A}^{-1}\bm{y}$. The matrix $\bm{A}$ is defined as

\begin{equation} \label{eq:A}
\bm{A} \delequal \begin{bmatrix}
1 & \cos\left(\varphi_{1}\right) &  \sin\left(\varphi_{1}\right)\\
1 & \cos\left(\varphi_{2}\right) &  \sin\left(\varphi_{2}\right)\\
1 & \cos\left(\varphi_{3}\right) &  \sin\left(\varphi_{3}\right)
\end{bmatrix},
\end{equation}

and $\bm{y}\delequal\left[f{\left(\varphi_{1}\right)}, f{\left(\varphi_{2}\right)}, f{\left(\varphi_{3}\right)}\right]^\mathsf{T}$ is the measurement vector. Note that $\varphi_{1}, \varphi_{2}, \varphi_{3}\in \left[0, 2\pi\right)$ are selected such that $\det{\left(\bm{A}\right)}\neq 0$.

The proof is provided in Appendix <ref>.

In general, Theorem <ref> allows for the computation of the optimal phase shift using only three measurements, without requiring knowledge of the explicit expression of the function.
Specifically, for the measurement phases $\varphi_{1}=0$, $\varphi_{2}=\pi/2$, and $\varphi_{3}=\pi$, a simple expression for the optimal phase shift can be obtained as follows:

\begin{equation}\label{eq:contin_phase_update}
\vartheta^{\star}=\arg{\left(y_{1}-y_{3}+j\left(2y_{2}-y_{1}-y_{3}\right)\right)}.
\end{equation}

Comment/*  */
The proposed phase-alignment algorithm for power maximization
Input: The number of RIS elements $N$ and the number of iterations $M$

Output: Near-optimal phase vector $\bm{\vartheta}^{\star}$ that maximizes the received power.

Initialize: $\bm{\vartheta}\gets \bm{\vartheta}_{0}$, $m\gets 0$, and $\bm{e}_{n}$ is a vector, where the component $n$ is $1$ and all other components are $0$.

$m < M$
$m \gets m + 1$
$n \gets 1$
$n \leq N$
$y_{1}\gets$ the measured power for the phase configuration $\bm{\vartheta}$
$y_{2}\gets$ the measured power for the phase configuration $\bm{\vartheta}+\frac{\pi}{2}\bm{e}_{n}$
$y_{3}\gets $ the measured power for the phase configuration $\bm{\vartheta}+\pi\bm{e}_{n}$
$\bm{\vartheta} \gets \bm{\vartheta}+\arg\!\left(y_{1}-y_{3}+j\left(2y_{2}-y_{1}-y_{3}\right)\right)\bm{e}_{n}$
$n \gets n + 1$

$\bm{\vartheta}^{\star}\gets \bm{\vartheta}$

Algorithm <ref> is a sequential, iterative phase update algorithm. At each iteration, adjusting the phase of each element requires measuring the received power for three different phase configurations. The algorithm updates the phase of one element using (<ref>), then proceeds to the next element until all $N$ elements have had their phases updated. This process is repeated $M$ times.

startstop = [rectangle, rounded corners, minimum width=1.5cm, minimum height=0.8cm,text centered, draw=black, fill=mycolor1!20, inner sep=-0.3ex]
startstop_blue = [rectangle, rounded corners, minimum width=1.5cm, minimum height=0.8cm,text centered, draw=black, fill=mycolor2!20, inner sep=-0.3ex]
startstop_green = [rectangle, rounded corners, minimum width=1.5cm, minimum height=0.8cm,text centered, draw=black, fill=mycolor3!20, inner sep=-0.3ex]

decision = [diamond, aspect=2,  text centered, draw=black, fill=mycolor1!20, inner sep=-0.3ex]
arrow = [thick,->,>=stealth]

[cross/.style=path picture=
(path picture bounding box.south east) – (path picture bounding box.north west) (path picture bounding box.south west) – (path picture bounding box.north east);

[arrow, color = mycolor2, line width = 1.2] (0,0) – (1,0.2);
[arrow, color = mycolor2, line width = 1.2] (1,0.2) – (0.9,0.9);
[arrow, color = mycolor2, line width = 1.2] (0.9,0.9) – (0.1,0.3);
[mycolor2] at (1, 0) 0.9$z_{1}$;
[mycolor2] at (1.12, 0.85) 0.9$z_{2}$;
[mycolor2] at (0.14, 0.58) 0.9$z_{3}$;

[arrow, color = mycolor2, dashed, line width = 1.2] (+0,0) – (-1.0136,0.1126);
[arrow, color = mycolor2, line width = 1.2] (-1.0136,0.1126) – (-1.1136,0.8126);
[arrow, color = mycolor2, line width = 1.2] (-1.1136,0.8126) – (-1.9136,0.2126);
[arrow, color = mycolor1, dotted, line width = 1.2] (-1.0136,0.1126) – (-1.9136,0.2126);
[mycolor2] at (-0.58, 0.38) 0.9$z_{1}e^{j\vartheta^{1}_{1}}$;
[mycolor2] at (-0.89, 0.78) 0.9$z_{2}$;
[mycolor2] at (-2, 0.4) 0.9$z_{3}$;
[mycolor1] at (-1.8, -0.03) 0.9$z_{2}{+}z_{3}$;

[arrow, color = mycolor2, dashed, line width = 1.2] (+0,+0) – (-0.6829,- 0.1835);
[arrow, color = mycolor2, line width = 1.2] (-0.6829,- 0.1835) – (-1.6964,- 0.0709);
[arrow, color = mycolor2, line width = 1.2] (-1.6964,- 0.0709) – (-2.4964,- 0.6709);
[arrow, color = mycolor1, dotted, line width = 1.2] (-0.6829,- 0.1835) – (-2.4964,- 0.6709);
[mycolor2] at (-1.4, +0.18) 0.9$z_{1}e^{j\vartheta^{1}_{1}}$;
[mycolor2] at (-0.42, +0.10) 0.9$z_{2}e^{j\vartheta^{1}_{2}}$;
[mycolor2] at (-2.5, -0.39) 0.9$z_{3}$;
[mycolor1] at (-2, -0.9) 0.9$z_{1}e^{j\vartheta^{1}_{1}}{+}z_{3}$;

[arrow, color = mycolor2, line width = 1.2] (+0,+0) – (-0.6829,- 0.1835);
[arrow, color = mycolor2, line width = 1.2] (-0.6829,- 0.1835) – (-1.6964,- 0.0709);
[arrow, color = mycolor2, dashed, line width = 1.2] (-1.6964,- 0.0709) – (-2.6956,- 0.1127);
[arrow, color = mycolor1, dotted, line width = 1.2] (+0,+0) – (-1.6964,- 0.0709);
[mycolor2] at (-1.4, -0.36) 0.9$z_{1}e^{j\vartheta^{1}_{1}}$;
[mycolor2] at (-0.42, -0.30) 0.9$z_{2}e^{j\vartheta^{1}_{2}}$;
[mycolor2] at (-2.52, -0.32) 0.9$z_{3}e^{j\vartheta^{1}_{3}}$;
[mycolor1] at (-1.3, +0.22) 0.9$z_{1}e^{j\vartheta^{1}_{1}}{+}z_{2}e^{j\vartheta^{1}_{2}}$;

[arrow, color = mycolor2, dashed, line width = 1.2] (+0,+0) – (-1.0108,- 0.1354);
[arrow, color = mycolor2, line width = 1.2] (-1.0108,- 0.1354) – (-1.6937,- 0.3189);
[arrow, color = mycolor2, line width = 1.2] (-1.6937,- 0.3189) – (-2.6928,- 0.3606);
[arrow, color = mycolor1, dotted, line width = 1.2] (-1.0108,- 0.1354) – (-2.6928,- 0.3606);
[mycolor2] at (-0.72, +0.15) 0.9$z_{1}e^{j\vartheta^{2}_{1}}$;
[mycolor2] at (-1.4, -0.5) 0.9$z_{2}e^{j\vartheta^{1}_{2}}$;
[mycolor2] at (-2.52, -0.6) 0.9$z_{3}e^{j\vartheta^{1}_{3}}$;
[mycolor1] at (-2.25, -0.01) 0.9$z_{2}e^{j\vartheta^{1}_{2}}{+}z_{3}e^{j\vartheta^{1}_{3}}$;

[arrow, color = mycolor2, dashed, line width = 1.2] (+0,+0) – (-0.7044,- 0.0621);
[arrow, color = mycolor2, line width = 1.2] (-0.7044,- 0.0621) – (-1.7152,- 0.1975);
[arrow, color = mycolor2, line width = 1.2] (-1.7152,- 0.1975) – (-2.7143,- 0.2392);
[arrow, color = mycolor1, dotted, line width = 1.2] (-0.7044,- 0.0621) – (-2.7143,- 0.2392);
[mycolor2] at (-1.4, -0.4) 0.9$z_{1}e^{j\vartheta^{2}_{1}}$;
[mycolor2] at (-0.42, +0.22) 0.9$z_{2}e^{j\vartheta^{2}_{2}}$;
[mycolor2] at (-2.52, -0.5) 0.9$z_{3}e^{j\vartheta^{1}_{3}}$;
[mycolor1] at (-2.1, +0.15) 0.9$z_{1}e^{j\vartheta^{2}_{1}}{+}z_{3}e^{j\vartheta^{1}_{3}}$;

[arrow, color = mycolor2, line width = 1.2] (+0,+0) – (-0.7044,- 0.0621);
[arrow, color = mycolor2, line width = 1.2] (-0.7044,- 0.0621) – (-1.7152,- 0.1975);
[arrow, color = mycolor2, dashed, line width = 1.2] (-1.7152,- 0.1975) – (- 2.7086,- 0.3118);
[arrow, color = mycolor1, dotted, line width = 1.2] (+0,+0) – (-1.7152,- 0.1975);
[mycolor2] at (-1.4, +0.15) 0.9$z_{1}e^{j\vartheta^{2}_{1}}$;
[mycolor2] at (-0.42, +0.24) 0.9$z_{2}e^{j\vartheta^{2}_{2}}$;
[mycolor2] at (-2.52, +0.05) 0.9$z_{3}e^{j\vartheta^{2}_{3}}$;
[mycolor1] at (-1.2, -0.5) 0.9$z_{1}e^{j\vartheta^{2}_{1}}{+}z_{2}e^{j\vartheta^{2}_{2}}$;

[black] at (0.6, -1) 0.9Initial Vectors;
[black] at (3.1, -1) 0.9Iter 1: Step 1;
[black] at (5.8, -1) 0.9Iter 1: Step 2;
[black] at (8.6, -1) 0.9Iter 1: Step 3;

[black] at (1.7, -4.6) 0.9Iter 2: Step 1;
[black] at (5.2, -4.6) 0.9Iter 2: Step 2;
[black] at (8.6, -4.6) 0.9Iter 2: Step 3;

Visualization of the proposed algorithm in different steps for a toy example with $N=3$.

Figure <ref> presents a toy example that demonstrates the different steps of the proposed algorithm. The vector representation shows that initially, the vectors are misaligned, leading to a relatively small amplitude of their sum compared to a scenario where they are aligned in the same direction. The algorithm updates the phase of each vector to match with the sum of the others. As the algorithm progresses and reaches the end of the second iteration, the vectors become almost aligned in the same direction, resulting in a nearly maximum amplitude of their sum.

The proposed Algorithm <ref> converges to the maximum value of the function $f{\left(\bm{\vartheta}\right)}=\left|\sum_{n=1}^{N} z_{n}e^{j\vartheta_{n}}\right|^{2}$ as $M\to\infty$.

The proof is provided in Appendix <ref>.

§.§ A Random Algorithm

We use a random algorithm as a benchmark for the proposed algorithm. The basic concept is that, at each step, the algorithm picks a single element of the *ris sequentially and assigns a random phase value from a uniform distribution over the interval $[0, 2\pi)$. The new power measurement is then compared to the previous one. If the power increases, the algorithm updates the phase and moves on to the next element. If the measured power decreases, the algorithm maintains the previous phase and proceeds to the next element.

§ PERFORMANCE EVALUATION

In this section, we evaluate the performance of our proposed algorithm and compare it to the random phase update method. We use a *ris with $100$ elements and generate random complex Gaussian distributed values with unit variance for $\left\{z_n\right\}_{n=1}^N$. We conducted Monte-Carlo simulations and compared the results with those obtained from the random phase update method.

Fig. <ref> illustrates the normalized achieved power versus the number of power measurements for the proposed and random methods. The proposed method uses three measurements per phase update of each element and converges to its final value after $300$ measurements (i.e., $3N$), while the random one requires ten times more measurements as it has a slow convergence rate. Even in the presence of the noise, at the SNR of $10$ dB the proposed algorithm reaches the $93$% of the maximum achievable power <cit.>.

Normalized achieved power versus the number of measurements for the proposed and random methods.

The CDF of the normalized achieved power for the proposed and random algorithms.

Fig. <ref> shows the *cdf of the relative achieved power for both the proposed and random algorithms, compared to the optimum. We conducted $1000$ simulations with randomly generated channels. The results indicate that the proposed algorithm after the first iteration performs significantly better than the random one after ten iterations. Although the proposed algorithm still performs slightly better than the random one after fifty iterations, the latter appears to be slightly more stable.

§ CONCLUSIONS

This paper has presented a method for energy harvesting at a *ris from an ambient *rf source in the absence of coordination with the source. The objective was to maximize the received power by adjusting the phases of the *ris elements based only on power measurements, obtained without having a *rf receiver. The proposed sequential algorithm is proved to converge to the optimum and outperformed the random phase update method in terms of convergence rate.
In future work, we will extend this study to  more general scenarios, including the case with discrete phase shifts.

§ PROOF OF THEOREM <REF>

Let us define

\begin{equation}\label{eq:x_def_proof}
\bm{x}\delequal \left[\left|z_{0}\right|^{2}+\left|z\right|^{2}, 2\,\operatorname{Re}\!{\left(z_{0}z^{*}\right)}, 2\,\operatorname{Im}\!{\left(z_{0}z^{*}\right)}\right]^{\mathsf{T}}.
\end{equation}

From Lemma <ref> and (<ref>), we have

\begin{align*}
\vartheta^{\star}&=\arg{\left(z_{0}\right)}-\arg{\left(z\right)}\\
&=\arg{\left(x_{2}+j x_{3}\right)}.
\end{align*}

Therefore, we can compute the optimal phase shift from $\bm{x}$. We show that one can compute $\bm{x}$ using the received power from three different measurements. By expanding the function $f{\left(\varphi_{l}\right)}$, we get

\begin{align}
f{\left(\varphi_{l}\right)}&\!=\!\left|z_{0}\right|^{2}\!\!+\! \left|z\right|^{2}\!\!+\!2\operatorname{Re}\left(z_{0}z^{*}\right)\!\cos{\left(\varphi_{l}\right)}\!\!+\!2\operatorname{Im}\left(z_{0}z^{*}\right)\!\sin{\left(\varphi_{l}\right)}\nonumber\\
\end{align}

where $\bm{a}_{l}\delequal\left[1, \cos{\left(\varphi_{l}\right)},  \sin{\left(\varphi_{l}\right)}\right]^{\mathsf{T}}$, for $1\leq l\leq 3$.

Assuming $\bm{A}\delequal \left[\bm{a}_{1}, \bm{a}_{2}, \bm{a}_{3}\right]^\mathsf{T}$, and $\bm{y}=\left[f{\left(\varphi_{1}\right)}, f{\left(\varphi_{2}\right)}, f{\left(\varphi_{3}\right)}\right]^{\mathsf{T}}$, we have $\bm{A}\bm{x}=\bm{y}$, or $\bm{x}=\bm{A}^{-1}\bm{y}$ for $\det{\left(\bm{A}\right)}\neq 0$.

§ PROOF OF THEOREM <REF>

We denote the phase-shift vector generated by the algorithm at iteration $m$ up to the element $n$ with $\bm{\vartheta}^{k}$, where $k=mN+n$. We prove that for a given $N$, $\lim_{k\to\infty} f{\left(\bm{\vartheta}^{k}\right)}=\max_{\bm{\vartheta}}f{\left(\bm{\vartheta}\right)}$.

For any integer $1\leq n\leq N$ and an integer $k\geq 1$, we define

\begin{equation}
w_{n}^{k}\delequal \sum_{\substack{i=1\\i\neq n}}^{N} z_{i}e^{j\vartheta^{k}_{i}}.
\end{equation}

Hence, we have $f{\left(\bm{\vartheta}^{k}\right)} = \left|w^{k}_{n}+z_{n}e^{j\vartheta^{k}_{n}}\right|^2$.

At the phase update $k+1$, the algorithm only updates the phase of the element $n$ ($w^{k+1}_{n}=w^{k}_{n}$) as $f{\left(\bm{\vartheta}^{k+1}\right)} = \left|w^{k}_{n}+z_{n}e^{j\vartheta^{k+1}_{n}}\right|^2$.

According to the Theorem <ref>, we have

\begin{equation}
f{\left(\bm{\vartheta}^{k+1}\right)} \!=\! \left|\!w^{k}_{n}\!+\!z_{n}e^{j\vartheta^{k+1}_{n}}\!\right|^2\!\geq\! \left|w^{k}_{n}\!+\!z_{n}e^{j\vartheta^{k}_{n}}\right|^2\!=\!f{\left(\bm{\vartheta}^{k}\right)}.
\end{equation}

Therefore, $f{\left(\bm{\vartheta}^{1}\right)}, f{\left(\bm{\vartheta}^{2}\right)}, \dots$ form an increasing sequence. Moreover, according to the Lemma <ref>, the set $\mathcal{F}\delequal\left\{f{\left(\bm{\vartheta}^{k}\right)}; k\in\mathbb{N}\right\}$ is upper bounded by $\left(\sum_{n=1}^{N}\left|z_{n}\right|\right)^{2}$. Therefore, according to the monotone convergence theorem, we have

\begin{equation}
\lim_{k\to\infty}f{\left(\bm{\vartheta}^{k}\right)} = \sup{\mathcal{F}}\leq \left(\sum_{n=1}^{N}\left|z_{n}\right|\right)^{2}.
\end{equation}

If we show that $\sup{\mathcal{F}}=\left(\sum_{n=1}^{N}\left|z_{n}\right|\right)^{2}$, the proof is complete. Lets define $\bm{\vartheta}^{\star}\delequal\lim_{k\to\infty}\bm{\vartheta}^{k}$. For the phase shift vector $\bm{\vartheta}^{\star}$, any further phase update will not increase $f{\left(\cdot\right)}$. In other words, for updating element $n$, we should apply Theorem <ref> to the following function

\begin{equation}
f{\left(\bm{\vartheta}^{\star}\right)} = \left|w_{n}^{\star}+z_{n}e^{j\vartheta^{\star}_{n}}\right|^2,
\end{equation}

where $w_{n}^{\star}\delequal \sum_{\substack{i=1\\i\neq n}}^{N} z_{i}e^{j\vartheta^{\star}_{i}}$. Since any further update will not increase the value of $f{\left(\cdot\right)}$, therefore for all $1\leq n\leq N$, we have $\operatorname{Arg}{\left(z_{n}e^{j\vartheta^{\star}_{n}}\right)}=\operatorname{Arg}{\left(w_{n}^{\star}\right)}$. Using Lemma <ref>, we have

\begin{align}
\operatorname{Arg}{\left(z_{1}e^{j\vartheta^{\star}_{1}}\right)}=\operatorname{Arg}{\left(z_{2}e^{j\vartheta^{\star}_{2}}\right)}=\dots = \operatorname{Arg}{\left(z_{N}e^{j\vartheta^{\star}_{N}}\right)},
\end{align}

\begin{align}
\operatorname{Arg}{\left(z_{1}\right)}+\vartheta^{\star}_{1}=\dots= \operatorname{Arg}{\left(z_{N}\right)}+\vartheta^{\star}_{N} = \vartheta_{0} \!\mod 2\pi.
\end{align}

Therefore, we have $\vartheta_{n}^{\star}=\vartheta_{0}-\arg{\left(z_{n}\right)}, \ \text{for all}\ 1\leq n\leq N$,

that are according to the Lemma <ref>, the phase shifts that maximize $f{\left(\cdot\right)}$.

Assume for each $1\leq n\leq N$, $u_{n}\delequal \sum_{\substack{i=1\\i\neq n}}^{N}z_{i}$, if
$\operatorname{Arg}{\left(z_{n}\right)}=\operatorname{Arg}{\left(u_{n}\right)}$ for all $1\leq n\leq N$, then

For $1\leq m, n\leq N$ and $m\neq n$, we have $\operatorname{Arg}{(z_{m})}=\operatorname{Arg}{\left(u_{m}\right)}$ and $\operatorname{Arg}{\left(z_{n}\right)}=\operatorname{Arg}{\left(u_{n}\right)}$, therefore, for some real positive $c_{m}$ and $c_{n}$, we have $z_{m}=c_{m}u_{m}$ and $z_{n}=c_{n}u_{n}$ .
Assuming $u_{m,n}\delequal \sum_{\substack{i=1\\i\neq m,n}}^{N}z_{k}$, we have

\begin{align}
\end{align}

After some algebraic manipulations, we obtain

\begin{equation}
\end{equation}

Hence, $\operatorname{Arg}{\left(z_{m}\right)}=\operatorname{Arg}{\left(z_{n}\right)}$ and the proof is complete.