# 2D-3D Pose Tracking with Multi-View Constraints

Huai Yu∗, , Kuangyi Chen∗, Wen Yang, , Sebastian Scherer, , and Gui-Song Xia
∗These authors contribute equally to this work. This work has been submitted
to the IEEE for possible publication. Copyright may be transferred without
notice, after which this version may no longer be accessible.Huai Yu, Kuangyi
Chen and Wen Yang are with the School of Electronic Information, Wuhan
University, Wuhan 430072, China (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>yangwen@whu.edu.cn)Sebastian Scherer is with the
Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA
(e-mail: basti@andrew.cmu.edu)Gui-Song Xia is with the School of Computer
Science, Wuhan University, Wuhan 430072, China (e-mail:
<EMAIL_ADDRESS>
###### Abstract

Camera localization in 3D LiDAR maps has gained increasing attention due to
its promising ability to handle complex scenarios, surpassing the limitations
of visual-only localization methods. However, existing methods mostly focus on
addressing the cross-modal gaps, estimating camera poses frame by frame
without considering the relationship between adjacent frames, which makes the
pose tracking unstable. To alleviate this, we propose to couple the 2D-3D
correspondences between adjacent frames using the 2D-2D feature matching,
establishing the multi-view geometrical constraints for simultaneously
estimating multiple camera poses. Specifically, we propose a new 2D-3D pose
tracking framework, which consists: a front-end hybrid flow estimation network
for consecutive frames and a back-end pose optimization module. We further
design a cross-modal consistency-based loss to incorporate the multi-view
constraints during the training and inference process. We evaluate our
proposed framework on the KITTI and Argoverse datasets. Experimental results
demonstrate its superior performance compared to existing frame-by-frame 2D-3D
pose tracking methods and state-of-the-art vision-only pose tracking
algorithms. More online pose tracking videos are available at
https://youtu.be/yfBRdg7gw5M.

###### Index Terms:

Camera localization, LiDAR maps, multi-view geometry, 2D-3D matching

††publicationid: pubid: 0000–0000/00$00.00 © 2021 IEEE

##  I Introduction

Camera localization in 3D LiDAR maps has attracted more and more attention due
to the convenient system setup and high application prospects for mobile
robots and autonomous driving. On the one hand, it only leverages lightweight
and low-cost cameras, similar to visual SLAM, while offering significant
potential in mitigating pose drift issues. On the other hand, LiDAR maps can
be effortlessly constructed at a large scale and remain unaffected by changes
in illumination, thanks to LiDAR SLAM or registration techniques. However, the
existence of cross-modal gaps between camera images and LiDAR point clouds
hampers the establishment of robust 2D-3D correspondences, consequently
challenging the robustness and accuracy of camera localization.

Figure 1: Illustrative examples of the proposed 2D-3D pose tracking and visual
odometry on the KITTI 00 sequence. _Top:_ The visualization of the LiDAR
projection with initial pose and predicted pose. _Bottom:_ The top view of the
final trajectories of the proposed method and the visual odometry algorithm.

To alleviate the issue of cross-modal gaps, traditional methods mainly use
geometric consistency in 3D or 2D space[1, 2, 3, 4, 5]. The localization
performance highly depends on the accuracy of intermediary “products” such as
reconstructed sparse points or 3D line segments, which lack robustness and
generalization capability. With the help of deep learning, the correspondences
between LiDAR points and images are established by a cross-modal flow
network[6, 7], and then the camera poses can be inferred using the PnP solver
in a RANSAC loop. Besides, some methods also utilize a neural network to
regress the camera poses directly[8, 9, 10]. Despite their remarkable
performance, these methods solely estimate the camera pose frame by frame,
disregarding the multi-view constraints between adjacent frames, thereby
resulting in an unstable pose tracking.

We notice that the bundle adjustment of multi-view image matching can ensure
the smoothness of camera pose estimation in classical visual odometry
algorithms, and 2D-2D image matching is already a mature technique with
traditional handcraft features and learning-based optical flow. Therefore, our
intuition is to fuse the LiDAR-image correspondences and image-image flow to
establish multi-view visual constraints for further smoothing and improving
localization performance.

In this paper, we propose a novel 2D-3D pose tracking framework that leverages
multi-view constraints to achieve accurate pose estimation for consecutive
frames. The proposed framework formulates pose estimation as a local
optimization problem with an initial pose. Specifically, the initial pose of
each frame is from the previous frame during pose tracking. We impose a random
disturbance on the ground truth poses to obtain the initial poses during
network training. The 3D point clouds are projected onto the 2D plane with
rough initial poses to obtain the synthetic depth maps. Then we utilize a
hybrid flow estimation network to estimate the image-to-LiDAR depth flows
between the adjacent camera frames and a synthetic depth map, as well as the
optical flow between the adjacent camera frames. Especially, a cross-modal
consistency loss function is devised to incorporate multi-view constraints
between the estimated image-to-LiDAR depth flows and the optical flow into the
learning process. During pose tracking, we define an objective function
consisting of the reprojection error and the cross-modal consistency error,
and then optimize the estimated camera poses under the paradigm of a non-
linear least square problem. An illustrative example of the proposed framework
and visual odometry is shown in Fig. 1.

Figure 2: The proposed 2D-3D pose tracking framework. It consists of two main
components: the front-end hybrid flow estimation network, and the back-end
pose optimization module.

Our main contribution can be summarized as:

  * •

We propose an effective hybrid flow estimation network that simultaneously
estimates image-to-LiDAR depth flow and image-to-image flow between
consecutive video frames. We further devise a cross-modal consistency loss
function to incorporate multi-view constraints into the learning process.

  * •

We introduce a back-end optimization algorithm to smooth and improve
localization performance of camera pose tracking in LiDAR maps. The estimated
poses corresponding to the consecutive frames are refined together under the
paradigm of a non-linear least square problem based on cross-modal
consistency.

  * •

We conduct extensive experiments on two public datasets (i.e., KITTI and
Argoverse) to evaluate the performance. The experimental results demonstrate
that the proposed 2D-3D pose tracking framework can achieve more accurate and
robust localization than other frame-by-frame pose tracking methods.

##  II Related Work

###  II-A Visual-only Pose Tracking

Cameras provide a wealth of spectral and degenerated geometrical information
at a significantly lower cost compared to other competing sensor options. As a
result, visual localization plays an important role in modern localization
systems for autonomous vehicles. The paradigm of visual localization is to
fetch 2D-2D correspondences between two or more 2D images, typically employing
techniques such as optical flow or feature-based matching, and then leverage
the epipolar geometry to compute the relative motion. To overcome problems
such as scale ambiguity and error accumulation, Triggs [11] introduces a joint
optimization procedure that minimizes the re-projection error between the
observed feature points and the estimated points, which is called bundle
adjustment optimization. It adjusts the bundle of rays passing through the
camera center and the feature points in the 3D world to minimize this error
over multiple frames. In addition, some methods [12, 13, 14, 15] propose to
incorporate global information, such as GPS or inertial measurement units
(IMUs), to help resolve scale ambiguity and reduce error accumulation during
pose tracking. In recent years, with the advent of machine learning and
particularly deep learning, learned visual pose tracking methods [16, 17, 18,
19] have also emerged. These methods leverage Convolutional Neural Networks
(CNNs) to learn features and their associations between frames, thereby
estimating camera poses. However, visual-only pose tracking has inevitable
pose drift and lacks robustness over challenging conditions such as
illumination, weather, and season changes over time.

###  II-B 2D-3D Pose Tracking

Unlike cameras, LiDAR is less susceptible to those visual factors. Therefore,
pose tracking based on 2D-3D correspondences between 2D images and offline 3D
LiDAR maps holds greater potential in achieving robust localization. Existing
approaches focus on solving the cross-modal gaps between 2D images and 3D
LiDAR points. Some researchers propose to extract repeatable points or line
features from images and LiDAR points for 2D-3D matching [20, 21, 4, 5], which
achieve remarkable performance in scenes with rich geometric information.
Other approaches transform data into the same modal at first. For example, in
certain approaches [1, 2, 3], sparse 3D points are reconstructed from
consecutive camera frames or stereo disparity images captured by a stereo
camera. Subsequently, these reconstructed points are matched with the global
LiDAR map. The localization performance of these approaches relies on the
accuracy of the reconstructed 3D points. In addition, the 3D reconstruction
process is time-consuming and the reconstructed 3D points may not correspond
to any points in the LiDAR map. Other approaches [6, 22, 23, 8, 9, 7, 10]
first project the LiDAR map onto the 2D plane to obtain synthetic depth maps
and then match them with camera images. However, these methods predict the
camera pose frame by frame and ignore the constraints between adjacent frames.
As a result, they are prone to jitter and encounter error accumulation
problems during pose tracking. [24] implicitly utilizes the temporal features
between consecutive frames, but still predicts the camera poses frame by
frame. In this paper, we propose a novel 2D-3D pose tracking framework which
integrates the multi-view constraints between consecutive frames during the
network training and inference. Unlike the aforementioned methods, we regard
the poses of multiple frames as a whole and optimize them together under the
proposed cross-modal consistency, thus can well handle the cross-modal
differences and multi-view constraints.

##  III Proposed Method

The general frame-by-frame 2D-3D pose tracking framework adheres to the
paradigm of local optimization based on an initial pose, which can be
formulated as follows: In the beginning, the camera pose of the first frame is
initialized using GPS. For each subsequent camera frame, the pose of the
previous frame serves as the initial estimation. Based on this initial pose, a
fixed-size point cloud is segmented from the global LiDAR map. Then, the 2D-3D
correspondences between the camera frame and the segmented point cloud are
estimated. Finally, the corresponding camera pose is calculated using the PnP
solver. Previous work[8, 9, 7] has confirmed the feasibility of this paradigm
for pose tracking. However, it has also highlighted the limited robustness in
some scenarios, such as environments with extreme degeneracy. The unstable
camera localization motivates us to incorporate multi-view constraints between
adjacent camera frames into the 2D-3D pose tracking framework.

Our proposed 2D-3D pose tracking framework is shown in Fig. 2. Our key insight
is to simultaneously estimate the 6-DoF poses for two adjacent frames based on
a hybrid flow estimation network. For each group of two adjacent frames, we
initialize the poses as the same and from the estimation of previous
continuous frames. Then the global LiDAR maps are cropped with a fixed size
centered at the initial poses. By using the hybrid 2D-3D and 2D-2D flow
network, we can obtain stable 2D-3D correspondences. PnP solver is then
utilized to get the estimated poses. Finally, these poses are further refined
using a back-end optimization algorithm that incorporates multi-view
constraints. Subsequently, the initial pose of the current time step is
updated with the estimated pose of the current frame, while the estimated pose
of the next frame serves as the initial pose for the next time step.

In the following, we first introduce the front-end flow estimation network,
and then describe the details of the proposed cross-modal consistency loss
function. Finally, we present the back-end optimization algorithm.

###  III-A Hybrid Flow Estimation Network

####  III-A1 Depth Map Generation

To obtain the image-to-LiDAR depth flow using the flow estimation network, we
first need to project the 3D LiDAR point clouds to the 2D plane to generate
the synthetic depth maps. The projection process follows the pinhole camera
model which can be described as:

$\left(\begin{array}[]{l}u\\\ v\\\
1\end{array}\right)=\frac{1}{Z}\left(\begin{array}[]{ccc}f_{x}&0&c_{x}\\\
0&f_{y}&c_{y}\\\ 0&0&1\end{array}\right)\left(\begin{array}[]{l}X\\\ Y\\\
Z\end{array}\right)\triangleq\frac{1}{Z}\bm{K}P$ (1)

where $(X\quad Y\quad Z)^{T}$ and $P$ represent the coordinates of the 3D
points in the camera coordinate system. $(u\quad v\quad 1)^{T}$ represents the
homogeneous coordinates of the projection in the pixel coordinate system. The
intrinsic camera parameters, $f_{x},f_{y},c_{x},c_{y}$, form the camera matrix
$\bm{K}$. Additionally, the occlusion removal scheme proposed by [25] is
applied to eliminate occluded 3D points on the generated depth map.

####  III-A2 Flow Estimation

After obtaining the synthetic depth map, we utilize a hybrid flow estimation
network to predict the image-to-LiDAR depth flow and the optical flow between
the consecutive camera frames simultaneously. The hybrid network mainly
consists of three parts: the image-to-LiDAR depth flow estimation networks
$F_{c}$ and $F_{n}$, and the optical flow estimation network $F_{i}$. We use
I2D-Loc[7] as the backbone to predict the image-to-LiDAR depth flow and
RAFT[26] as the backbone to predict the optical flow because they both excel
at their respective tasks.

The ground truth image-to-LiDAR depth flow is generated by computing the
distance between the LiDAR depth maps projected based on the initial pose and
the ground truth pose. Given the initial pose $\bm{T}_{\text{init}}$, the
ground truth image-to-LiDAR depth flow for the current frame can be computed
using the following formulation:

${[\Delta u,\Delta
v]_{\text{cur2depth}}=h\left(P_{w},\hat{\bm{T}}_{\text{cur}}\right)-h\left(P_{w},\bm{T}_{\text{init}}\right)}$
(2) $h(P,\bm{T})\triangleq\bm{KT}P$ (3)

where $[\Delta u,\Delta v]_{\text{cur2depth}}$ is the calculated ground truth
image-to-LiDAR depth flow between the current camera frame and the synthetic
depth map. $\hat{\bm{T}}_{\text{cur}}$ and $\bm{T}_{\text{init}}$ are the
ground truth pose and the initial pose, respectively. $P_{w}$ is the
coordinate of the corresponding 3D point cloud in the world coordinate system.
$h$ represents the camera projection function that first transforms 3D points
from the world coordinate system to the camera coordinate system and then
projects them to the 2D plane. Besides, the ground truth image-to-LiDAR depth
flow for the next frame can be calculated based on a similar formulation:

${[\Delta u,\Delta
v]_{\text{next2depth}}=h\left(P_{w},\hat{\bm{T}}_{\text{next}}\right)-h\left(P_{w},\bm{T}_{\text{init}}\right)}$
(4)

where $[\Delta u,\Delta v]_{\text{next2depth}}$ represents the calculated
ground truth image-to-LiDAR depth flow between the next camera frame and the
synthetic depth map. $\hat{\bm{T}}_{\text{next}}$ is the ground truth pose of
the next camera frame.

###  III-B Loss Function

Regarding the network supervision, we first consider the masked average
endpoint error (EPE) between the predicted image-to-LiDAR depth flow and the
ground truth, defined as follows:

$L_{\text{epe}}=\frac{\sum
g(u,v)\left\|f_{\text{pre}}(u,v)-f_{\text{gt}}(u,v)\right\|_{2}}{\sum g(u,v)}$
(5) $g(u,v)=\left\\{\begin{array}[]{l}1,f_{\text{gt}}\neq 0\\\
0,\text{otherwise}\end{array}\right.$ (6)

where $f_{\text{pre}}$ and $f_{\text{gt}}$ represent the predicted and ground
truth image-to-LiDAR depth flow, respectively. $g(u,v)$ identifies the valid
pixels in the ground truth image-to-LiDAR depth flow. Due to the superior
accuracy achieved by the same modal image optical flow network compared to the
image-to-LiDAR depth flow network, we load a pre-trained optical flow model
and keep its parameters fixed during network training.

Figure 3: Diagram of the cross-modal consistency. Figure 4: Visualization of
the flow estimation. _a:_ LiDAR projection overlay on the current camera
image. _b:_ LiDAR projection overlay on the next camera image. _c:_ Predicted
image-to-LiDAR depth flow between the current image and the LiDAR projection.
_d:_ Projected image-to-LiDAR depth flow between the next image and the LiDAR
projection. _e:_ Warped difference matrix between the predicted image-to-depth
flows. _f:_ Predicted optical flow between the two camera images.

Additionally, to incorporate the multi-view constraints between two adjacent
camera frames during the network training, we propose a cross-modal
consistency-based loss function, which is formulated as follows:

$L_{\text{consist}}=\left\|w\left(f_{\text{pre}}^{\text{n2d}}-f_{\text{pre}}^{\text{c2d}},f_{\text{pre}}^{\text{c2d}}\right)-f_{\text{pre}}^{\text{c2n}}\right\|_{2}$
(7)

where $f_{\text{pre}}^{\text{n2d}}$ represents the predicted image-to-LiDAR
depth flow between the next camera frame and the synthetic depth map.
Similarly, $f_{\text{pre}}^{\text{c2d}}$ represents the predicted image-to-
LiDAR depth flow for the current camera frame. $f_{\text{pre}}^{\text{c2n}}$
is the predicted optical flow between the adjacent camera frames. $w$
represents the warping operation.

The cross-modal consistency, as shown in Fig. 3, explicitly describes the
relationship between the adjacent camera frames and the synthetic depth map.
Specifically, the image-to-depth flow represents the 2D-3D correspondences
between the camera frame and depth map. Fig. 4c and 4d visualize the predicted
flows corresponding to the current and next frame, respectively. The image-to-
image flow (i.e. optical flow) represents the 2D-2D correspondences between
two adjacent camera frames, which is shown in Fig. 4f. By calculating the
difference between the predicted image-to-depth flows, we will obtain the
equivalent image-to-image flow between the camera frames. As a result, the
error between the equivalent and predicted image-to-image flow constructs the
multi-view constraints between these predictions. In addition, it’s important
to note that the predicted image-to-LiDAR depth flow represents the
displacement field from the synthetic depth map to the image. Hence, the
difference between the image-to-LiDAR depth flows of two adjacent frames is
not equivalent to the image-to-image flow between them. We need to warp the
calculated difference matrix based on the image-to-LiDAR depth flow of the
first frame to obtain the final equivalent optical flow.

The final loss function is the sum of the above loss functions:

$L=L_{\text{epe}}^{\text{cur}}+L_{\text{epe}}^{\text{next}}+L_{\text{consist}}$
(8)

where $L_{\text{epe}}^{\text{cur}}$ is the average endpoint error of the
predicted current image-to-depth flow. $L_{\text{epe}}^{\text{next}}$ is the
average endpoint error of the predicted next image-to-depth flow.

In summary, we employ two loss functions to guide the network training: the
average endpoint error and the cross-modal consistency loss function. The
average endpoint error serves as a conventional constraint commonly used in
optical flow estimation tasks. The cross-modal consistency error introduces
the multi-view constraints between adjacent camera frames, capturing the
relationship between different views.

###  III-C Multi-View Constraints-based Back-End Optimization

During pose tracking, as previously mentioned, we utilize the trained hybrid
flow estimation network to simultaneously predict the image-to-LiDAR depth
flows between the adjacent camera frames and the synthetic depth map, as well
as the optical flow between the camera frames. Then the camera poses
corresponding to the two camera frames are solved using the PnP solver
according to the prediction. Besides, we define an energy function to further
optimize the solved poses, as:

$\bm{T}_{\text{cur}}^{*},\bm{T}_{\text{next}}^{*}=\operatorname*{arg\,min}_{\bm{T}_{\text{cur}},\bm{T}_{\text{next}}}(E_{\text{consist}}+E_{\text{reproj}}^{\text{cur}}+E_{\text{reproj}}^{\text{next}}),$
(9)

where $\bm{T}_{\text{cur}}$ and $\bm{T}_{\text{next}}$ represent the predicted
camera poses corresponding to the current and next frames, respectively.
$E_{\text{consist}}$ is the cross-modal consistency error and formulated as
follows:

$E_{\text{consist}}=\left\|h(P_{w},\bm{T}_{\text{next}})-h(P_{w},\bm{T}_{\text{cur}})-f^{\text{c2n}}_{\text{pre}}\right\|_{2}$
(10)

where $h$ is the camera projection function and defined as Eq. (3).
$f^{\text{c2n}}_{\text{pre}}$ is the predicted optical flow. Based on this
formula, we begin by projecting the 3D points onto the 2D plane using the
predicted poses. Subsequently, we compute the difference matrix between the
projections to derive the equivalent optical flow. Therefore,
$E_{\text{consist}}$ represents the distance between the equivalent and
predicted optical flows. Additionally, $E_{\text{reproj}}^{\text{cur}}$ and
$E_{\text{reproj}}^{\text{next}}$ both represent the reprojection error of the
solved camera poses:

$E_{\text{reproj}}^{\text{cur}}=\left\|h(P,\bm{T}_{\text{cur}})-X_{\text{cur}}\right\|_{2}$
(11)
$E_{\text{reproj}}^{\text{next}}=\left\|h(P,\bm{T}_{\text{next}})-X_{\text{next}}\right\|_{2}$
(12)

where $X_{\text{cur}}$ and $X_{\text{next}}$ represent the 2D coordinates of
pixels with valid depth values in the depth maps that have been warped using
the predicted image-to-LiDAR depth flows.

Based on the formulated energy function, we employ the least square method to
determine the optimal camera poses $\bm{T}_{\text{cur}}^{*}$ and
$\bm{T}_{\text{next}}^{*}$ that minimize the function value. The obtained
$\bm{T}_{\text{cur}}^{*}$ replaces the initial pose $\bm{T}_{\text{init}}$ as
the final pose for the current frame, while $\bm{T}_{\text{next}}^{*}$ is
utilized to initialize the subsequent localization process.

##  IV Experiments

In this section, we conduct extensive experiments on two public datasets to
evaluate the performance of our proposed method.

TABLE I: Performance comparison of the flow estimation network under different setups on the KITTI dataset. “C”, “N”, “T”, and “L” represent the current image-to-LiDAR depth flow estimation network $F_{c}$, the next image-to-LiDAR depth flow estimation network $F_{n}$, additional training, and the cross-modal consistency loss function, respectively. In (e) and (f), the first value represents the performance of the network $F_{c}$, while the second value represents the performance of the network $F_{n}$. Case | Module | Mean Error | Median Error | Fail[%] $\downarrow$
---|---|---|---|---
C | N | T | L | Rot.[∘] $\downarrow$ | Transl.[cm] $\downarrow$ | Rot.[∘] $\downarrow$ | Transl.[cm] $\downarrow$
Initial pose |  |  |  |  | $\approx 9.6726$ | $\approx 182.8381$ | $\approx 9.9033$ | $\approx 187.3079$ | -
(a) | ✓ |  |  |  | 0.8619 | 24.6955 | 0.6900 | 17.6687 | 1.81
(b) |  | ✓ |  |  | 2.5284 | 75.4198 | 1.8364 | 63.5510 | 16.65
(c) | ✓ |  | ✓ |  | 0.8375 | 22.4241 | 0.6777 | 15.3426 | 1.61
(d) |  | ✓ | ✓ |  | 2.4225 | 74.0713 | 1.8184 | 62.6881 | 16.67
(e) | ✓ | ✓ | ✓ |  | 0.8640/2.4801 | 23.7909/75.5822 | 0.6978/1.8511 | 16.0583/63.5162 | 1.61/16.93
(f) | ✓ | ✓ | ✓ | ✓ | 0.9051/2.4842 | 26.5121/71.8964 | 0.7168/1.7970 | 17.8525/61.0808 | 1.78/15.90

###  IV-A Dataset and Evaluation Metrics

####  IV-A1 Dataset

We conduct experiments on KITTI[27] and Argoverse[28] datasets. KITTI is
commonly used in the field of autonomous driving. It consists of 22 sequences.
We use sequences 03, 05, 06, 07, 08, and 09 as the training set and sequences
00 and 10 as the validation set. Argoverse offers more complex driving
scenarios, albeit with shorter odometry sequences. We use the sequences
train1, train2, and train3 as the training set, and sample several sequences
with less noise as the validation set.

To acquire the global LiDAR map, unavailable in the datasets, we begin by
aggregating all scans based on the ground truth poses. Subsequently, to
conserve storage space, we down-sample the aggregated map at a resolution of
0.1m. During training, we segment the point cloud for each frame by centering
it around the ground truth pose and extending it 100m forward, 10m backward,
and 25m to the left and right. During pose tracking, the coverage of the
segmented point cloud remains fixed, but it is centered on the initial pose of
the current frame being tracked.

####  IV-A2 Evaluation Metrics

We evaluate the performance of the proposed method in two ways. The first one
is to evaluate the localization accuracy frame by frame using the mean and
median errors of the predicted poses. Following [6], [7], we assume the
position errors larger than four meters as a _failure_. The second one is to
evaluate the pose tracking performance on the global LiDAR maps using the
average trajectory error (ATE) and the relative pose error (RPE), which are
defined in [29] and commonly used for quantitative trajectory evaluation.

###  IV-B Experimental Setup

Accounting for the varying sizes of RGB images across different sequences, we
uniformly crop all RGB images to a resolution of $960\times 320$ during
network training. As mentioned above, the proposed hybrid network comprises
three components: the current image-to-LiDAR depth flow estimation network
$F_{c}$, the next image-to-LiDAR depth flow estimation network $F_{n}$, and
the optical flow estimation network $F_{i}$. We employ I2D-Loc[7] as the
backbone for $F_{c}$ and $F_{n}$, while for $F_{i}$, we utilize RAFT-S[26] as
the backbone due to memory limitations. In particular, we initialize $F_{c}$
and $F_{i}$ by loading the pre-trained models provided by [7, 26]. As for
$F_{n}$, we first train it singly for 100 epochs based on the same
experimental setup in [7]. Subsequently, we load all the pre-trained model
weights and conduct joint training for additional 50 epochs on $F_{c}$,
$F_{n}$, and $F_{i}$ using the proposed cross-modal consistency loss function.
The learning rate, weight decay, and training batch size are set to $4\times
10^{-6}$, $1\times 10^{-4}$, and 2, respectively. We employ the _MultiStepLR_
learning rate scheduler, which reduces the learning rate to one-tenth of the
previous value at the 10th and 30th epochs. The joint training is performed
using two NVIDIA RTX 3090 Ti GPUs, while the other experiments are conducted
on a single NVIDIA RTX 3090 Ti GPU.

TABLE II: Pose tracking performance comparison on KITTI sequence 00. “Complete” indicates that the pose tracking process ends without any interruptions caused by localization failure. The localization errors are calculated until being interrupted. Case | Method | Translation Error | Rotation Error | Complete | Param[M] | Time[ms]
---|---|---|---|---|---|---
Mean.[cm] $\downarrow$ | Std.[cm] $\downarrow$ | Mean.[∘] $\downarrow$ | Std.[∘] $\downarrow$
(a) | CMRNet[8] | 205 | 107 | 126 | 39 | ✓ | 37.1 | 25
(b) | I2D-Loc[7] | 21 | 10 | 0.46 | 0.30 |  | 6.3 | 176
(c) | VO | 162 | 63 | 4.95 | 2.34 | ✓ | - | 219
(d) | I2D-VO | 15 | 21 | 0.48 | 0.64 | ✓ | 6.3 | 180
(e) | Ours w/o Optim | 14 | 17 | 0.50 | 0.53 |  | 13.6 | 374
(f) | Ours w/ Optim | 14 | 17 | 0.49 | 0.54 | ✓ | 13.6 | 588
Figure 5: 2D-3D pose tracking framework that loosely couples 2D-3D
correspondences and 2D-2D correspondences.

###  IV-C Ablation Study

In this section, we evaluate the effectiveness of the multi-view constraints
during the training process. The experimental results are shown in Table I,
where we report the mean and median errors of the calculated poses based on
the predicted flows as evaluation metrics.

The performance of the single image-to-LiDAR depth flow estimation network for
current and next frames using the official I2D-Loc model is displayed in Table
I(a) and (b). The error range of the initial poses for the network $F_{n}$ is
greater than that for the network $F_{c}$, resulting in a relatively higher
localization accuracy for the network $F_{c}$. After conducting an additional
50 epochs of training for each network individually in Table I(c) and (d), the
pose errors of the two networks both decrease by a little margin. Then when
the networks $F_{c}$ and $F_{n}$ are trained jointly for another 50 epochs,
Table I(e) shows that the localization accuracy for each network hasn’t been
improved. The above experiments imply an inherent conflict between the network
$F_{c}$ and $F_{n}$ due to their varying error ranges for initial poses. In
Table I(f), we utilize the proposed cross-modal consistency loss function to
incorporate the multi-view constraints during the training process. The
experimental result shows that the localization performance of the network
$F_{c}$ degrades a little, but the pose error of the network $F_{n}$ decreases
by a large margin. This outcome demonstrates that the proposed cross-modal
consistency-based loss bridges the gaps between the predicted 2D-3D
correspondences of adjacent frames.

###  IV-D Results Analysis

Table II gives the quantitative results of our method evaluated on KITTI
sequence 00. In addition to the frame-by-frame 2D-3D pose tracking methods
CMRNet[8] and I2D-Loc[7], we also compare our method with the traditional
visual odometry algorithm and a devised 2D-3D pose tracking framework that
loosely couples the visual odometry algorithm with I2D-Loc, referred to as
I2D-VO. As illustrated in Fig. 5, this framework utilizes the visual odometry
algorithm and I2D-Loc to generate pose candidates based on 2D-2D and 2D-3D
correspondences, respectively. Subsequently, the final pose is selected from
these candidates based on predefined thresholds.

Figure 6: LiDAR projection overlaid on the next frame before and after
optimization. TABLE III: Performance comparison on KITTI dataset.

Method | KITTI_00 (1861m) | KITTI_10 (459m) | Arg_2c07 (44m) | Arg_2595 (54m)
---|---|---|---|---
Transl.[m] $\downarrow$ | Rot.[∘] $\downarrow$ | Transl.[m] $\downarrow$ | Rot.[∘] $\downarrow$ | Transl.[m] $\downarrow$ | Rot.[∘] $\downarrow$ | Transl.[m] $\downarrow$ | Rot.[∘] $\downarrow$
Mean | Std | Mean | Std | Mean | Std | Mean | Std | Mean | Std | Mean | Std | Mean | Std | Mean | Std
VINS-Fusion[30] | 16.70 | 9.40 | 4.99 | 2.51 | 2.34 | 1.13 | 1.75 | 0.74 | 3.09 | 2.24 | 22.87 | 6.77 | 19.02 | 11.04 | 176.67 | 1.11
I2D-VO | 0.24 | 0.26 | 0.54 | 0.94 | 0.61 | 0.91 | 2.74 | 15.61 | 0.43 | 0.23 | 0.45 | 0.42 | 0.34 | 0.22 | 4.86 | 0.18
Ours | 0.13 | 0.17 | 0.49 | 0.55 | 0.45 | 0.63 | 1.04 | 1.24 | 0.17 | 0.15 | 0.58 | 0.34 | 0.14 | 0.11 | 2.96 | 0.20

(a) KITTI_00

(b) KITTI_10

(c) Argoverse_2c07

(d) Argoverse_2595

Figure 7: Estimated trajectories of three tested methods on four sampled LiDAR
maps.

According to Table II(a) and (b), CMRNet efficiently tracks the entire LiDAR
map without any interruption, albeit with certain limitations in its
localization accuracy. Conversely, I2D-Loc boasts superior localization
precision, yet it falls short of completing the entire map. This difference
arises due to the heavy dependence of I2D-Loc on accurate 2D-3D
correspondences, while CMRNet directly regresses camera poses. Additionally,
as shown in Table II(c), the traditional visual odometry (VO) algorithm can
track the entire map seamlessly. However, as shown in Fig. 1, the drift of VO
is serious. By integrating I2D-Loc with the visual odometry algorithm in a
loosely-coupled manner, our proposed framework, I2D-VO, accomplishes an
enhancement in localization accuracy and successfully overcomes previous
shortcomings. In Table II(e) and (f), our proposed 2D-3D pose tracking
framework, which incorporates multi-view constraints, achieves minimal
translation and rotation errors upon map completion. Furthermore, as
demonstrated in Fig. 6, the accuracy of 2D-3D correspondences noticeably
improves the following optimization in various challenging scenarios. In these
cases, a single image-to-depth flow estimation network will fail to predict
reliable 2D-3D correspondences due to homogeneous depth projections or
extremely dark RGB images.

To highlight the outstanding performance of our proposed 2D-3D pose tracking
methodology, we conducted a comprehensive comparison with the state-of-the-art
visual-only pose tracking system, VINS-Fusion[30]. The evaluation was
conducted on multiple sequences, including sequences 00 and 10 from the KITTI
dataset, as well as sequences 2c07- and 2595- from the Argoverse dataset. The
estimated trajectories of these sampled sequences are depicted in Fig. 7,
demonstrating the excellent alignment of our proposed method with the ground
truth trajectories.

Quantitative analysis, presented in Table III, provides clear evidence of the
superiority of our proposed 2D-3D pose tracking method, which incorporates
multi-view constraints. Across all four LiDAR maps, our method consistently
outperforms the other tested methodologies, delivering remarkable performance.

Figure 8: Comparison of ATE between VINS-Fusion and our proposed methods.
Figure 9: Comparison of RPE between VINS-Fusion and our proposed methods on
the KITTI 00 sequence.

Additionally, we have also conducted a comparison between our proposed method
and VINS-Fusion, utilizing Absolute Trajectory Error (ATE) and Relative Pose
Error (RPE) as the key performance metrics. The quantitative outcomes are
presented in Fig. 8 and Fig. 9. These figures emphatically demonstrate that
our proposed method consistently achieves superior localization accuracy.
Moreover, it exhibits reduced drift during pose tracking compared to both the
carefully engineered I2D-VO and the current visual-only VINS-Fusion.

###  IV-E Discussion

The iterative update module in I2D-Loc[7] improves the matching accuracy by
solving the large displacement problem. However, when the initial pose is
already accurate enough, the motion tends to be small. This hypothesis holds
in our pose tracking pipeline. Consequently, we conduct additional experiments
to validate the impact of reducing the number of iterative updates. The
results of these experiments are presented in Table IV.

TABLE IV: Performance comparison of the proposed method with different
iterative numbers.

Num | Translation Error | Rotation Error | Time[ms]
---|---|---|---
Mean.[cm] $\downarrow$ | Std.[cm] $\downarrow$ | Mean.[∘] $\downarrow$ | Std.[∘] $\downarrow$
24 | 14 | 16 | 0.49 | 0.53 | 588
20 | 14 | 17 | 0.50 | 0.56 | 562
16 | 14 | 17 | 0.51 | 0.57 | 540
12 | 15 | 17 | 0.51 | 0.56 | 520

The experimental results demonstrate that reducing the number of iterative
updates can speed up the localization process without causing a significant
decrease in accuracy. However, despite the improved efficiency achieved by
reducing the number of iterations, our pose tracking method currently operates
at a top speed of 3-4 frames per second, which is mainly due to inefficient
point cloud cutting and projection.

##  V Conclusions and Future Work

In this study, we propose a novel 2D-3D pose tracking framework that tightly
integrates 2D-3D correspondences through 2D-2D matching. Our approach
incorporates a cross-modal consistency-based loss function to enable effective
network supervision under multi-view constraints. Additionally, we introduce a
non-linear least square problem for joint optimization of adjacent camera
frame poses during the pose tracking process. Furthermore, we present a
comparative analysis by incorporating a pose tracking framework that loosely
couples 2D-3D and 2D-2D correspondences. Extensive experiments demonstrate
that our proposed method significantly improves the smoothness and accuracy of
the pose tracking. In the future, we will further extend our method to various
scenarios and improve the efficiency.

## References

  * [1] T. Caselitz, B. Steder, M. Ruhnke, and W. Burgard, “Monocular camera localization in 3D LiDAR maps,” in _IEEE Int Conf Intell Robot Syst_ , 2016, pp. 1926–1931.
  * [2] L. Ding and G. Sharma, “Fusing structure from motion and LiDAR for dense accurate depth map estimation,” in _Proc IEEE Int Conf Acoust Speech Signal Process_ , 2017, pp. 1283–1287.
  * [3] X. Zuo, P. Geneva, Y. Yang, W. Ye, Y. Liu, and G. Huang, “Visual-inertial localization with prior LiDAR map constraints,” _IEEE Robot Autom Lett_ , vol. 4, no. 4, pp. 3394–3401, 2019.
  * [4] Y. Liu, T. S. Huang, and O. D. Faugeras, “Determination of camera location from 2-D to 3-D line and point correspondences,” _IEEE Trans Pattern Anal Mach Intell_ , vol. 12, no. 1, pp. 28–37, 1990.
  * [5] H. Yu, W. Zhen, W. Yang, and S. Scherer, “Line-based 2-D–3-D registration and camera localization in structured environments,” _IEEE Trans Instrum Meas_ , vol. 69, no. 11, pp. 8962–8972, 2020.
  * [6] D. Cattaneo, D. G. Sorrenti, and A. Valada, “CMRNet++: Map and camera agnostic monocular visual localization in LiDAR maps,” in _IEEE Int Conf Robot Autom Workshop_ , 2020.
  * [7] K. Chen, H. Yu, W. Yang, L. Yu, S. Scherer, and G.-S. Xia, “I2D-Loc: Camera localization via image to LiDAR depth flow,” _ISPRS J Photogramm Remote Sens_ , vol. 194, pp. 209–221, 2022.
  * [8] D. Cattaneo, M. Vaghi, A. L. Ballardini, S. Fontana, D. G. Sorrenti, and W. Burgard, “CMRNet: Camera to LiDAR-map registration,” in _Proc IEEE Conf Intell Transp Syst_ , 2019, pp. 1283–1289.
  * [9] M.-F. Chang, J. Mangelson, M. Kaess, and S. Lucey, “HyperMap: Compressed 3D map for monocular camera registration,” in _IEEE Int Conf Robot Autom_ , 2021, pp. 11 739–11 745.
  * [10] X. Lv, B. Wang, Z. Dou, D. Ye, and S. Wang, “LCCNet: LiDAR and camera self-calibration using cost volume network,” in _Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit_ , 2021, pp. 2894–2901.
  * [11] B. Triggs, P. F. McLauchlan, R. I. Hartley, and A. W. Fitzgibbon, “Bundle adjustment—a modern synthesis,” in _Int Worksh Vis Algorithms_. Springer, 2000, pp. 298–372.
  * [12] A. J. Davison, I. D. Reid, N. D. Molton, and O. Stasse, “MonoSLAM: Real-time single camera SLAM,” _IEEE Trans Pattern Anal Mach Intell_ , vol. 29, no. 6, pp. 1052–1067, 2007.
  * [13] J. Xiao, D. Xiong, Q. Yu, K. Huang, H. Lu, and Z. Zeng, “A real-time sliding-window-based visual-inertial odometry for MAVs,” _IEEE Trans Industr Inform_ , vol. 16, no. 6, pp. 4049–4058, 2019.
  * [14] J. Henawy, Z. Li, W.-Y. Yau, and G. Seet, “Accurate IMU factor using switched linear systems for VIO,” _IEEE Trans Ind Electron_ , vol. 68, no. 8, pp. 7199–7208, 2020.
  * [15] L. Yu, J. Qin, S. Wang, Y. Wang, and S. Wang, “A tightly coupled feature-based visual-inertial odometry with stereo cameras,” _IEEE Trans Industr Inform_ , vol. 70, no. 4, pp. 3944–3954, 2022.
  * [16] R. Li, S. Wang, and D. Gu, “DeepSLAM: A robust monocular SLAM system with unsupervised deep learning,” _IEEE Trans Ind Electron_ , vol. 68, no. 4, pp. 3577–3587, 2020.
  * [17] R. Li, S. Wang, Z. Long, and D. Gu, “Undeepvo: Monocular visual odometry through unsupervised deep learning,” in _Proc IEEE Int Conf Robot Autom_. IEEE, 2018, pp. 7286–7291.
  * [18] R. Clark, S. Wang, H. Wen, A. Markham, and N. Trigoni, “Vinet: Visual-inertial odometry as a sequence-to-sequence learning problem,” in _Proc AAAI Conf Artif Intell_ , vol. 31, no. 1, 2017.
  * [19] S. Wang, R. Clark, H. Wen, and N. Trigoni, “Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks,” in _Proc IEEE Int Conf Robot Autom_. IEEE, 2017, pp. 2043–2050.
  * [20] M. Feng, S. Hu, M. H. Ang, and G. H. Lee, “2D3D-Matchnet: Learning to match keypoints across 2D image and 3D point cloud,” in _Proc IEEE Int Conf Robot Autom_ , 2019, pp. 4790–4796.
  * [21] B. Wang, C. Chen, Z. Cui, J. Qin, C. X. Lu, Z. Yu, P. Zhao, Z. Dong, F. Zhu, N. Trigoni _et al._ , “P2-net: Joint description and detection of local features for pixel and point matching,” in _IEEE Int Conf Comput Vis Workshops_ , 2021, pp. 16 004–16 013.
  * [22] R. W. Wolcott and R. M. Eustice, “Visual localization within LIDAR maps for automated urban driving,” in _IEEE Int Conf Intell Robot Syst_ , 2014, pp. 176–183.
  * [23] A. D. Stewart and P. Newman, “LAPS-localisation using appearance of prior structure: 6-DoF monocular camera localisation using prior pointclouds,” in _Proc IEEE Int Conf Robot Autom_ , 2012, pp. 2625–2632.
  * [24] A. D. Nguyen and M. Yoo, “Calibbd: Extrinsic calibration of the lidar and camera using a bidirectional neural network,” _IEEE Access_ , vol. 10, pp. 121 261–121 271, 2022.
  * [25] R. Pintus, E. Gobbetti, and M. Agus, “Real-time rendering of massive unstructured raw point clouds using screen-space operators,” in _Int Conf Virtual Real Archaeol Cult Herit_ , 2011, pp. 105–112.
  * [26] Z. Teed and J. Deng, “RAFT: Recurrent all-pairs field transforms for optical flow,” in _Comput Vis ECCV_ , 2020, pp. 402–419.
  * [27] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The KITTI dataset,” _Int J Rob Res_ , vol. 32, no. 11, pp. 1231–1237, 2013.
  * [28] M.-F. Chang, J. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan _et al._ , “Argoverse: 3D tracking and forecasting with rich maps,” in _Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit_ , 2019, pp. 8748–8757.
  * [29] Z. Zhang and D. Scaramuzza, “A tutorial on quantitative trajectory evaluation for visual (-inertial) odometry,” in _IEEE Int Conf Intell Robot Syst_. IEEE, 2018, pp. 7244–7251.
  * [30] T. Qin, P. Li, and S. Shen, “VINS-Mono: A robust and versatile monocular visual-inertial state estimator,” _IEEE Trans Robot_ , vol. 34, no. 4, pp. 1004–1020, 2018.


where we note that $w_{gh}$ is a martingale difference sequence with respect
to the filtration
$\mathcal{F}_{gh}=\sigma(\\{s_{mn}\\}_{m=1,\ldots,g-1,n=1,\ldots,M_{m}},\\{s_{gn}\\}_{n=1,\ldots,h})$,
i.e. ${\rm E}(w_{gh}|\mathcal{F}_{g,h-1})=0$. Then A.2 follows from the
martingale central limit theorem (e.g., Brown, 1971, Theorem 2) if

$\displaystyle\operatorname{Var}(\theta)^{-\lambda}\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}{\rm
E}|w_{gh}|^{2\lambda}\longrightarrow 0\quad\textrm{for some }\lambda>1,$ (A.5)
$\displaystyle\operatorname{Var}(\theta)^{-1}\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}{\rm
E}(w_{gh}^{2}|\mathcal{F}_{g,h-1})\overset{P}{\longrightarrow}1.$ (A.6)

We first prove the Lyapunov condition in A.5. We find ${\rm
E}|s_{gh}|^{2\lambda}\leq cN_{gh}^{2\lambda}$ by Lemma A.1. We also find that

${\rm E}\Big{|}\sum_{j=1}^{h-1}s_{gj}\Big{|}^{2\lambda}\leq c{\rm
E}\Big{|}\sum_{j=1}^{h-1}s_{gj}^{2}\Big{|}^{\lambda}\leq
c\Big{|}\sum_{j=1}^{h-1}({\rm
E}s_{gj}^{2\lambda})^{1/\lambda}\Big{|}^{\lambda}\leq
c\Big{|}\sum_{j=1}^{M_{g}}(N_{gj}^{2\lambda})^{1/\lambda}\Big{|}^{\lambda}\leq
cN_{g}^{\lambda}\sup_{g,h}N_{gh}^{\lambda},$ (A.7)

where the first inequality is Marcinkiewicz-Zygmund, the second is Minkowski,
and the third is due to Lemma A.1. Thus, we obtain the bound

${\rm E}|w_{gh}|^{2\lambda}\leq 2^{2\lambda}{\rm E}|s_{gh}|^{2\lambda}{\rm
E}\Big{|}\sum_{j=1}^{h-1}s_{gj}\Big{|}^{2\lambda}\leq
cN_{gh}^{2\lambda}N_{g}^{\lambda}\sup_{g,h}N_{gh}^{\lambda},$ (A.8)

and hence

$\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}{\rm E}|w_{gh}|^{2\lambda}\leq
c\sup_{g,h}N_{gh}^{3\lambda-1}\sup_{g}N_{g}^{\lambda}N.$ (A.9)

Combining A.3 and A.9, the Lyapunov condition in A.5 is satisfied by the
second condition of Assumption 5.

We next prove convergence of the conditional variance in A.6. Because
$\operatorname{Var}(\theta)$ equals $\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}{\rm
E}(w_{gh}^{2})$, we decompose ${\rm E}(w_{gh}^{2}|\mathcal{F}_{g,h-1})-{\rm
E}(w_{gh}^{2})=q_{1,gh}+q_{2,gh}$, where
$q_{1,gh}=\sigma_{gh}^{2}\sum_{j=1}^{h-1}(s_{gj}^{2}-\sigma_{gj}^{2})$ and
$q_{2,gh}=\sigma_{gh}^{2}\sum_{j_{1}=1}^{h-1}\sum_{j_{2}\neq
j_{1}}^{h-1}s_{gj_{1}}s_{gj_{2}}$. Then A.6 follows if

$\operatorname{Var}(\theta)^{-1}\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}q_{m,gh}\overset{P}{\longrightarrow}0\quad\textrm{for
}m=1,2.$ (A.10)

For $m=1$, we reverse the summations and find that
$\sum_{h=1}^{M_{g}}q_{1,gh}=\sum_{h=1}^{M_{g}}r_{1,gh}$, where
$r_{1,gh}=(s_{gh}^{2}-\sigma_{gh}^{2})\sum_{j=h+1}^{M_{g}}\sigma_{gj}^{2}$ is
mean zero and independent across both $g$ and $h$. We prove convergence in
$L_{\lambda}$-norm. We find ${\rm E}|r_{1,gh}|^{\lambda}\leq c{\rm
E}|s_{gh}|^{2\lambda}(\sum_{j=h+1}^{M_{g}}\sigma_{gj}^{2})^{\lambda}\leq
cN_{gh}^{2\lambda}(\sum_{j=1}^{M_{g}}\sigma_{gj}^{2})^{\lambda}=cN_{gh}^{2\lambda}\sigma_{g}^{2\lambda}$
using Lemma A.1 and $\sum_{j=1}^{M_{g}}\sigma_{gj}^{2}=\sigma_{g}^{2}$. By the
Marcinkiewicz-Zygmund and Minkowski inequalities we find that ${\rm
E}\big{|}\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}r_{1,gh}\big{|}^{\lambda}\leq
c\big{(}\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}({\rm
E}|r_{1,gh}|^{\lambda})^{2/\lambda}\big{)}^{\lambda/2}$, and hence

${\rm E}\Big{|}\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}r_{1,gh}\Big{|}^{\lambda}\leq
c\Big{(}\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}N_{gh}^{4}\sigma_{g}^{4}\Big{)}^{\lambda/2}\leq\sup_{g,h}N_{gh}^{3\lambda/2}\sup_{g}N_{g}^{\lambda/2}\Big{(}\sum_{g=1}^{G}\sigma_{g}^{4}\Big{)}^{\lambda/2}.$

Combining this with the bound A.3, the result A.10 for $m=1$ follows if

$\sup_{g,h}N_{gh}^{3\lambda/2}\sup_{g}N_{g}^{\lambda/2}\Big{(}\sum_{g=1}^{G}\sigma_{g}^{4}\Big{)}^{-\lambda/2}\longrightarrow
0,$

which is satisfied by the first condition of Assumption 5.

For $m=2$, we use symmetry and reverse the summations to find
$\sum_{h=1}^{M_{g}}q_{2,gh}=\sum_{h=1}^{M_{g}}r_{2,gh}$, where
$r_{2,gh}=2s_{gh}\sum_{j_{1}=h+1}^{M_{g}}\sigma_{gj_{1}}^{2}\sum_{j_{2}=1}^{h-1}s_{gj_{2}}=w_{gh}\sum_{j=h+1}^{M_{g}}\sigma_{gj}^{2}$
is a martingale difference sequence with respect to $\mathcal{F}_{gh}$. We
prove convergence in mean square. By A.8 with $\lambda=1$ the variance is
${\rm E}(r_{2,gh}^{2})\leq{\rm E}(w_{gh}^{2})\sigma_{g}^{4}\leq
cN_{gh}^{2}N_{g}\sup_{g,h}N_{gh}\sigma_{g}^{4}$, and hence

${\rm
E}\Big{(}\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}r_{2,gh}\Big{)}^{2}=\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}{\rm
E}(r_{2,gh}^{2})\leq
c\sup_{g,h}N_{gh}^{2}\sup_{g}N_{g}^{2}\sum_{g=1}^{G}\sigma_{g}^{4}.$

Combining this with the bound A.3, the result A.10 for $m=2$ follows by the
first condition of Assumption 5. This completes the proof of A.6 and hence of
A.2.

It remains to show the second part of Theorem 1. This follows directly from
Lemma A.4 by application of Assumption 5 to the remainder terms.

###  A.2 Proof of Theorem 2

As in the proof of Theorem 1, we give the proof for $\tau_{\sigma}$ only, and
we ignore the asymptotically irrelevant factors $m_{\rm c}$ and $m_{\rm f}$.
Under the conditions of Theorem 2, and specifically under Assumption 6, we
find from 8 that ${\bm{\Sigma}}_{\rm c}=\sum_{g=1}^{G}\sigma_{g}^{2}$.
However, it is important to note that, under the conditions of Theorem 2,
$\sigma_{g}^{2}=\operatorname{Var}(s_{g})\neq\sum_{h=1}^{M_{g}}\sigma_{gh}^{2}$.

We decompose the test statistic as follows:

$\frac{\hat{\theta}}{\widehat{\operatorname{Var}}(\hat{\theta})^{1/2}}=\frac{\sum_{g=1}^{G}\sigma_{g}^{2}}{\widehat{\operatorname{Var}}(\hat{\theta})^{1/2}}\bigg{(}\frac{\hat{\theta}-\theta}{\sum_{g=1}^{G}\sigma_{g}^{2}}+\frac{\theta-{\rm
E}(\theta)}{\sum_{g=1}^{G}\sigma_{g}^{2}}+\frac{{\rm
E}(\theta)}{\sum_{g=1}^{G}\sigma_{g}^{2}}\bigg{)},$

where we note that ${\rm
E}(\theta)/\sum_{g=1}^{G}\sigma_{g}^{2}=({\bm{\Sigma}}_{\rm
c}-{\bm{\Sigma}}_{\rm f}){\bm{\Sigma}}_{\rm c}^{-1}$ is non-zero in the limit
under the alternative hypothesis, H1 in 9. Thus, it suffices to prove that

$\frac{\theta-{\rm
E}(\theta)}{\sum_{g=1}^{G}\sigma_{g}^{2}}\overset{P}{\longrightarrow}0,\quad\frac{\hat{\theta}-\theta}{\sum_{g=1}^{G}\sigma_{g}^{2}}\overset{P}{\longrightarrow}0,\quad\textrm{and}\quad\frac{\widehat{\operatorname{Var}}(\hat{\theta})^{1/2}}{\sum_{g=1}^{G}\sigma_{g}^{2}}\overset{P}{\longrightarrow}0.$
(A.11)

For the first result in A.11, we prove convergence in mean square. The second
moment of the numerator is

$\displaystyle{\rm E}\big{(}\theta-{\rm
E}(\theta)\big{)}^{2}=\operatorname{Var}(\theta)$
$\displaystyle=\sum_{g=1}^{G}\operatorname{Var}\Big{(}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}s_{gh_{1}}s_{gh_{2}}\Big{)}=\sum_{g=1}^{G}\operatorname{Var}\Big{(}s_{g}^{2}-\sum_{h=1}^{M_{g}}s_{gh}^{2}\Big{)}$
$\displaystyle\leq c\sum_{g=1}^{G}N_{g}^{4}\leq c\sup_{g}N_{g}^{3}N,$ (A.12)

where the second equality is by Assumption 6 and the penultimate inequality is
by Lemma A.1 (applying the Cauchy-Schwarz inequality to the covariance terms).
Hence, $\theta-{\rm E}(\theta)$ is $O_{P}(\sup_{g}N_{g}^{3/2}N^{1/2})$, which
proves the first result in A.11 by Assumption 7. The second result in A.11
follows directly from Lemma A.2(ii) and Assumption 7. Finally, by the same
methods as applied in the proof of A.43, we find that

$\widehat{\operatorname{Var}}(\hat{\theta})=2\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}\hat{s}_{gh_{1}}^{2}\hat{s}_{gh_{2}}^{2}=O_{P}(\sup_{g,h}N_{gh}^{2}\sup_{g}N_{g}N),$

which proves the third result in A.11 by Assumption 7.

###  A.3 Proof of Theorem 3

As in the proofs of Theorems 1 and 2, we give the proof for $\tau_{\sigma}$
only. The proof for $\tau_{\Sigma}$ is essentially the same but with slightly
more complicated notation. The bootstrap probability measure is denoted
$P^{\ast}$, and expectation under this measure is denoted ${\rm E}^{\ast}$. We
define the bootstrap contrast
$\theta^{\ast}=\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}s_{gh_{1}}^{\ast}s_{gh_{2}}^{\ast}$, and similarly the bootstrap
variance estimator, and so on.

We prove the bootstrap analog of Theorem 1, but under the conditions of
Theorem 3, which will establish the required result. Specifically, for all
$x\in\mathbb{R}$ and all $\epsilon>0$, we prove that

$P^{\ast}\bigg{(}\frac{\hat{\theta}^{\ast}}{\sqrt{\operatorname{Var}^{\ast}(\theta^{\ast})}}\leq
x\bigg{)}\overset{P}{\longrightarrow}\Phi(x)\quad\textrm{and}\quad
P^{\ast}\bigg{(}\kern
0.83328pt\bigg{|}\frac{\widehat{\operatorname{Var}}(\hat{\theta}^{\ast})}{\operatorname{Var}^{\ast}(\theta^{\ast})}-1\bigg{|}>\epsilon\kern-0.83328pt\bigg{)}\overset{P}{\longrightarrow}0,$
(A.13)

as $N\to\infty$, where $\Phi(x)$ denotes the cumulative distribution function
of the standard normal distribution. Clearly, A.13 implies that
$P^{\ast}(\tau_{\sigma}^{\ast}\leq x)\overset{P}{\longrightarrow}\Phi(x)$.
From Corollary 1 we have the result that $P_{0}(\tau_{\sigma}\leq
x)\to\Phi(x)$. Because $\Phi(x)$ is everywhere continuous, the desired result
then follows by application of the triangle inequality and Polya’s Theorem.

Thus, we need to prove A.13. We first note that, even though Assumption 1 is
not imposed, it nonetheless holds by construction that, under the bootstrap
probability measure $P^{\ast}$, the bootstrap data are clustered according to
the fine structure in Assumption 1. Therefore, the proof of A.13 largely
follows that of Theorem 1. One main difference is that
$\sigma_{g}^{2}=\operatorname{Var}(s_{g})\neq\sum_{h=1}^{M_{g}}\sigma^{2}_{gh}$
because Assumption 1 is not imposed in Theorem 3.

We first establish the bootstrap equivalent of the lower bound in A.3,

$\operatorname{Var}^{\ast}(\theta^{\ast})\geq
c\big{(}1+o_{P}(1)\big{)}\sum_{g=1}^{G}\Big{(}\sum_{h=1}^{M_{g}}\sigma_{gh}^{2}\Big{)}^{\\!2},$
(A.14)

where we have used the fact that $\sigma_{g}^{4}$ in A.3 needs to be replaced
by $(\sum_{h=1}^{M_{g}}\sigma_{gh}^{2})^{2}$ under the assumptions of Theorem
3. To prove A.14, we first use $s_{gh}^{\ast}=\hat{s}_{gh}v_{gh}^{\ast}$,
where $v_{gh}^{\ast}$ is independent across both $g$ and $h$, such that, c.f.
18 and 19,

$\operatorname{Var}^{\ast}(\theta^{\ast})=\operatorname{Var}^{\ast}\\!\Big{(}\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}s_{gh_{1}}^{\ast}s_{gh_{2}}^{\ast}\Big{)}=2\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}\hat{s}_{gh_{1}}^{2}\hat{s}_{gh_{2}}^{2}=\widehat{\operatorname{Var}}(\hat{\theta}).$

The result in A.14 now follows from Lemma A.4 by application of Assumption 5
to the remainder terms.

We next prove the following four results, which imply A.13. For all
$x\in\mathbb{R}$ and all $\epsilon>0$,

$\displaystyle P^{\ast}\bigg{(}\kern
0.83328pt\bigg{|}\frac{\hat{\theta}^{\ast}-\theta^{\ast}}{\sqrt{\operatorname{Var}^{\ast}(\theta^{\ast})}}\bigg{|}>\epsilon\kern-0.83328pt\bigg{)}\overset{P}{\longrightarrow}0,$
(A.15) $\displaystyle
P^{\ast}\bigg{(}\frac{\theta^{\ast}}{\sqrt{\operatorname{Var}^{\ast}(\theta^{\ast})}}\leq
x\bigg{)}\overset{P}{\longrightarrow}\Phi(x),$ (A.16) $\displaystyle
P^{\ast}\bigg{(}\kern
0.83328pt\bigg{|}\frac{\widehat{\operatorname{Var}}(\theta^{\ast})}{\operatorname{Var}^{\ast}(\theta^{\ast})}-1\bigg{|}>\epsilon\kern-0.83328pt\bigg{)}\overset{P}{\longrightarrow}0,$
(A.17) $\displaystyle P^{\ast}\bigg{(}\kern
0.83328pt\bigg{|}\frac{\widehat{\operatorname{Var}}(\hat{\theta}^{\ast})-\widehat{\operatorname{Var}}(\theta^{\ast})}{\operatorname{Var}^{\ast}(\theta^{\ast})}\bigg{|}>\epsilon\kern-0.83328pt\bigg{)}\overset{P}{\longrightarrow}0,$
(A.18)

as $N\to\infty$. The proofs of A.15 and A.16 are nearly identical to the
corresponding proofs of A.1 and A.2. Similarly, the proofs of A.17 and A.18
are nearly identical to the corresponding proofs of A.39 and A.38. We
therefore merely highlight the differences.

First, A.15 follows by Markov’s inequality and application of Lemma A.3, the
lower bound A.14, and Assumption 5.

Consider now A.16. Under the bootstrap probability measure,
$w_{gh}^{\ast}=2s_{gh}^{\ast}\sum_{j=1}^{h-1}s_{gj}^{\ast}$ is a martingale
difference sequence with respect to the filtration

$\mathcal{F}_{gh}^{\ast}=\sigma\big{(}\\{v_{mn}^{\ast}\\}_{m=1,\ldots,g-1,n=1,\ldots,M_{m}},\\{v_{gn}^{\ast}\\}_{n=1,\ldots,h}\big{)}.$

To verify the bootstrap equivalent of the Lyapunov condition, we apply the
same proof as for A.5. Replacing ${\rm E}$ with ${\rm E}^{\ast}$, the bounds
A.7–A.9 hold under the bootstrap measure with the right-hand sides being
$O_{P}$ of the indicated order by A.24, A.29, A.30, and Lemma A.1. Thus, in
particular,

$\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}{\rm
E}^{\ast}|w_{gh}^{\ast}|^{2\lambda}=O_{P}\big{(}\sup_{g,h}N_{gh}^{3\lambda-1}\sup_{g}N_{g}N\big{)},$

which together with A.14 and Assumption 5 verifies the Lyapunov condition for
A.16. For the proof of convergence of the conditional variance, we apply the
same proof as for A.6 with $r_{1,gh}^{\ast}=(s_{gh}^{\ast
2}-\hat{s}_{gh}^{2})\sum_{j=h+1}^{M_{g}}\hat{s}_{gj}^{2}$ and
$r_{2,gh}^{\ast}=w_{gh}^{\ast}\sum_{j=h+1}^{M_{g}}\hat{s}_{gj}^{2}$. For both
terms we prove mean square convergence (because $\lambda\geq 2$). The
arguments are nearly identical to those in the proof of A.6, with all bounds
being $O_{P}$ of the indicated order, using A.24, A.29, A.30, and Lemma A.1.
This completes the proof of A.16.

For the proof of A.17, we follow the proof of A.39 and obtain
$q_{3,gh}^{\ast}=(s_{gh}^{\ast
2}-\hat{s}_{gh}^{2})\sum_{j=1}^{h-1}s_{gh}^{\ast 2}$. We then apply the same
proof as for $q_{3,gh}$ with $\lambda=2$. Specifically, we find that there
exists a set $\mathcal{A}^{\ast}$ with
$P^{\ast}(\mathcal{A}^{\ast})\overset{P}{\longrightarrow}1$, and on this set
we have

$\operatorname{Var}^{\ast}(q_{3,gh}^{\ast})=O_{P}\bigg{(}\\!N_{gh}^{4}\Big{(}\sum_{h-1}^{M_{g}}\sigma_{gh}^{2}\Big{)}^{\\!2}\bigg{)},$

where we used again A.24, A.29, A.30, and Lemma A.1. Because $q_{3,gh}^{\ast}$
is a martingale difference sequence, the proof of A.17 is concluded in the
same way as that of A.39.

Finally, we prove A.18. As in A.43–A.47, we write
$\widehat{\operatorname{Var}}(\hat{\theta}^{\ast})-\widehat{\operatorname{Var}}(\theta^{\ast})$
as

$\displaystyle 2$
$\displaystyle(\hat{\beta}_{1}^{\ast}-\hat{\beta}_{1})^{2}\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}(\hat{s}_{gh_{1}}^{\ast}\hat{s}_{gh_{2}}^{\ast}+s_{gh_{1}}^{\ast}s_{gh_{2}}^{\ast})\sum_{i=1}^{N_{gh_{1}}}x_{gh_{1}i}^{2}\sum_{j=1}^{N_{gh_{2}}}x_{gh_{2}j}^{2}$
(A.19)
$\displaystyle+8(\hat{\beta}_{1}^{\ast}-\hat{\beta}_{1})\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}s_{gh_{1}}^{\ast
2}s_{gh_{2}}^{\ast}\sum_{i=1}^{N_{gh_{2}}}x_{gh_{2}i}^{2}$ (A.20)
$\displaystyle-4(\hat{\beta}_{1}^{\ast}-\hat{\beta}_{1})^{2}\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}\Big{(}s_{gh_{1}}^{\ast}\sum_{i=1}^{N_{gh_{2}}}x_{gh_{2}i}^{2}+s_{gh_{2}}^{\ast}\sum_{j=1}^{N_{gh_{1}}}x_{gh_{1}j}^{2}\Big{)}s_{gh_{1}}\sum_{\ell=1}^{N_{gh_{2}}}x_{gh_{2}\ell}^{2}$
(A.21)
$\displaystyle-4(\hat{\beta}_{1}^{\ast}-\hat{\beta}_{1})^{2}\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}s_{gh_{1}}^{\ast}\Big{(}\sum_{j=1}^{N_{gh_{1}}}x_{gh_{1}j}^{2}\Big{)}\Big{(}\sum_{i=1}^{N_{gh_{2}}}x_{gh_{2}i}^{2}\Big{)}^{\\!2}.$
(A.22)

For A.19, A.21, and A.22, we use A.30 and A.36 together with Lemma A.1, and
find that

${\rm
E}^{\ast}|\hyperref@@ii[BSproof3a]{\textup{\tagform@{\ref*{BSproof3a}}}}|=O_{P}\bigg{(}N^{-1}\sup_{g,h}N_{gh}\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}N_{gh_{1}}^{2}N_{gh_{2}}^{2}\bigg{)}=O_{P}\big{(}\sup_{g,h}N_{gh}^{3}\sup_{g}N_{g}\big{)}.$

By the same argument, we also find the same bound for A.21 and A.22. Using
A.14 and the first condition of Assumption 5 shows the required result for
these terms. For A.20, we apply the Cauchy-Schwarz inequality as in A.48,

$\hyperref@@ii[BSproof4a]{\textup{\tagform@{\ref*{BSproof4a}}}}^{2}\leq
64(\hat{\beta}_{1}-\beta_{1,0})^{2}\Big{(}\sup_{g,h}\sum_{i=1}^{N_{gh_{1}}}x_{gh_{1}i}^{2}\Big{)}^{\\!2}\bigg{(}\sum_{g=1}^{G}\Big{(}\sum_{h_{1}=1}^{M_{g}}s_{gh_{1}}^{\ast
2}\Big{)}^{\\!2}\bigg{)}\bigg{(}\sum_{g=1}^{G}\Big{(}\sum_{h_{2}=1}^{M_{g}}s_{gh_{2}}^{\ast}\Big{)}^{\\!2}\bigg{)}.$
(A.23)

The first two factors on the right-hand side satisfy A.36 and A.30,
respectively. The third factor is non-negative, and, under the bootstrap
probability measure, it has a mean which is
$O_{P}\big{(}\sum_{g=1}^{G}(\sum_{h=1}^{M_{g}}N_{gh}^{2})^{2}\big{)}=O_{P}(\sup_{g,h}N_{gh}\sup_{g}N_{g}N)$
using A.24, A.29, A.30, and Lemma A.1. The last factor is non-negative and,
under the bootstrap probability measure, it has a mean which is
$O_{P}\big{(}\sum_{g=1}^{G}(\sum_{h=1}^{M_{g}}\sigma_{gh}^{2})^{2}\big{)}$
using again A.24, A.29, A.30, and Lemma A.1. The proof for A.23 is now
completed in the same way as that of A.48. This completes the proof of A.18
and hence of Theorem 3.

###  A.4 Proof of Theorem 4

The result that $P(\hat{m}\leq m_{0}-1)\to 0$ is a direct consequence of
Theorem 2 for the asymptotic tests and of Corollary 2(ii) for the bootstrap
tests. In case (ii), where $m_{0}=p$, there is nothing more to prove. In case
(i) we have $m_{0}\leq p-1$. Because $P(\hat{m}\leq m_{0}-1)\to 0$, the
sequential procedure will reach the test of the null hypothesis $m=m_{0}$ with
probability converging to one. This is a test of a true null, so we find from
Corollary 1 and Corollary 2(i) that $P(\hat{m}=m_{0})\to 1-\alpha$, which
proves the theorem.

###  A.5 Auxiliary Lemmas

######  Lemma A.1.

Let Assumption 2 be satisfied. Then

$\sup_{g,h}N_{gh}^{-\xi}{\rm
E}\|{\bm{s}}_{gh}\|^{\xi}=O(1)\quad\textrm{and}\quad\sup_{g}N_{g}^{-\xi}{\rm
E}\|{\bm{s}}_{g}\|^{\xi}=O(1)\quad\textrm{for }1\leq\xi\leq 2\lambda.$

###### Proof.

This is Lemma A.2 of Djogbenou, MacKinnon, and Nielsen (2019). ∎

######  Lemma A.2.

Let Assumptions 2, 3 and 4 be satisfied. Let $\hat{\bm{\theta}}$ be defined by
15 and also define
${\bm{\theta}}=\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}\operatorname{vech}({\bm{s}}_{gh_{1}}{\bm{s}}_{gh_{2}}^{\top})$;
c.f. 16. Then

  * (i)

Under Assumption 1,
$\|\hat{\bm{\theta}}-{\bm{\theta}}\|=O_{P}(\sup_{g,h}N_{gh}\sup_{g}N_{g})$.

  * (ii)

Under Assumption 6,
$\|\hat{\bm{\theta}}-{\bm{\theta}}\|=O_{P}(\sup_{g}N_{g}^{2})$.

###### Proof.

We give the proof in the scalar case only. The proof for the multivariate case
is nearly identical but with more complicated notation.

First note that

$\hat{s}_{ghi}=x_{ghi}\hat{u}_{ghi}=x_{ghi}(u_{ghi}-x_{ghi}(\hat{\beta}-\beta_{0}))=s_{ghi}-x_{ghi}^{2}(\hat{\beta}-\beta_{0}).$
(A.24)

From 12, 14, and 16, we then find the difference

$\displaystyle\hat{\theta}-\theta={}$
$\displaystyle\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}\Big{(}\sum_{i=1}^{N_{gh_{1}}}x_{gh_{1}i}^{2}(\hat{\beta}-\beta_{0})\\!\Big{)}\Big{(}\sum_{i=1}^{N_{gh_{2}}}x_{gh_{2}i}^{2}(\hat{\beta}-\beta_{0})\\!\Big{)}$
(A.25) $\displaystyle-2\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}s_{gh_{1}}\sum_{i=1}^{N_{gh_{2}}}x_{gh_{2}i}^{2}(\hat{\beta}-\beta_{0}).$
(A.26)

Using 2 we find that
$\hat{\beta}-\beta_{0}=({\bm{x}}^{\top}{\bm{x}})^{-1}\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}s_{gh}$.
Under Assumption 1 we have

$\operatorname{Var}\Big{(}\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}s_{gh}\Big{)}=\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}\operatorname{Var}(s_{gh})\leq
c\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}N_{gh}^{2}\leq cN\sup_{g,h}N_{gh}$ (A.27)

using Assumption 2 and Lemma A.1. Similarly, under Assumption 6,

$\operatorname{Var}\Big{(}\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}s_{gh}\Big{)}=\sum_{g=1}^{G}\operatorname{Var}\Big{(}\sum_{h=1}^{M_{g}}s_{gh}\Big{)}=\sum_{g=1}^{G}\operatorname{Var}(s_{g})\leq
c\sum_{g=1}^{G}N_{g}^{2}\leq cN\sup_{g}N_{g}.$ (A.28)

Hence, using also Assumption 3,

$\displaystyle|\hat{\beta}-\beta_{0}|$
$\displaystyle=O_{P}\big{(}N^{-1/2}\sup_{g,h}N_{gh}^{1/2}\big{)}\quad\textrm{under
\lx@cref{creftypecap~refnum}{as:cluster}},$ (A.29)
$\displaystyle|\hat{\beta}-\beta_{0}|$
$\displaystyle=O_{P}\big{(}N^{-1/2}\sup_{g}N_{g}^{1/2}\big{)}\quad\textrm{under
\lx@cref{creftypecap~refnum}{as:clusteralt}}.$

We also need the simple bounds

$\sup_{g,h}N_{gh}^{-1}\sum_{i=1}^{N_{gh}}x_{ghi}^{2}=O_{P}(1)\quad\textrm{and}\quad\sup_{g}N_{g}^{-1}\sum_{h=1}^{M_{g}}\sum_{i=1}^{N_{gh}}x_{ghi}^{2}=O_{P}(1),$
(A.30)

which follow from the uniform moment bound in Assumption 3. Using A.30, we
find that the absolute value of the right-hand side of A.25 is bounded by

$(\hat{\beta}-\beta_{0})^{2}\sum_{g=1}^{G}\Big{(}\sum_{h=1}^{M_{g}}\sum_{i=1}^{N_{gh}}x_{ghi}^{2}\Big{)}^{\\!\\!2}=(\hat{\beta}-\beta_{0})^{2}O_{P}\Big{(}\sum_{g=1}^{G}N_{g}^{2}\Big{)}=(\hat{\beta}-\beta_{0})^{2}O_{P}\big{(}N\sup_{g}N_{g}\big{)},$
(A.31)

which proves the result for A.25 using A.29.

Next, we write A.26 as

$\displaystyle\hyperref@@ii[diffB]{\textup{\tagform@{\ref*{diffB}}}}={}$
$\displaystyle-2\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}s_{gh_{1}}\sum_{h_{2}=1}^{M_{g}}\sum_{i=1}^{N_{gh_{2}}}x_{gh_{2}i}^{2}(\hat{\beta}-\beta_{0})$
(A.32)
$\displaystyle+2\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}s_{gh}\sum_{i=1}^{N_{gh}}x_{ghi}^{2}(\hat{\beta}-\beta_{0}).$
(A.33)

Under Assumption 1, A.27, A.29, and A.30 show that
$|\hyperref@@ii[diffB1]{\textup{\tagform@{\ref*{diffB1}}}}|=O_{P}(\sup_{g,h}N_{gh}\sup_{g}N_{g})$
and that
$|\hyperref@@ii[diffB2]{\textup{\tagform@{\ref*{diffB2}}}}|=O_{P}(\sup_{g,h}N_{gh}^{2})$.
Under Assumption 6, A.28, A.29, and A.30 show that
$|\hyperref@@ii[diffB1]{\textup{\tagform@{\ref*{diffB1}}}}|=O_{P}(\sup_{g}N_{g}^{2})$
and that
$|\hyperref@@ii[diffB2]{\textup{\tagform@{\ref*{diffB2}}}}|=O_{P}(\sup_{g,h}N_{gh}\sup_{g}N_{g})$.
This proves the result for A.26. ∎

######  Lemma A.3.

Let Assumptions 2, 3, 4 and 6 be satisfied. Let
$\hat{\bm{\theta}}^{\ast}=\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}\operatorname{vech}(\hat{\bm{s}}_{gh_{1}}^{\ast}\hat{\bm{s}}_{gh_{2}}^{\ast\top})$
and ${\bm{\theta}}^{\ast}=\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}\operatorname{vech}({\bm{s}}_{gh_{1}}^{\ast}{\bm{s}}_{gh_{2}}^{\ast\top})$.
Then

${\rm
E}^{\ast}\|\hat{\bm{\theta}}^{\ast}-{\bm{\theta}}^{\ast}\|=O_{P}\big{(}\sup_{g,h}N_{gh}\sup_{g}N_{g}\big{)}.$

###### Proof.

The proof is very similar to that of Lemma A.2. Again, we give the proof in
the scalar case only since the multivariate case is nearly identical but with
more complicated notation. We first write

$\displaystyle\hat{\theta}^{\ast}-\theta^{\ast}={}$
$\displaystyle\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}\Big{(}\sum_{i=1}^{N_{gh_{1}}}x_{gh_{1}i}^{2}(\hat{\beta}^{\ast}-\hat{\beta})\\!\Big{)}\Big{(}\sum_{i=1}^{N_{gh_{2}}}x_{gh_{2}i}^{2}(\hat{\beta}^{\ast}-\hat{\beta})\\!\Big{)}$
(A.34) $\displaystyle-2\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}s_{gh_{1}}^{\ast}\sum_{i=1}^{N_{gh_{2}}}x_{gh_{2}i}^{2}(\hat{\beta}^{\ast}-\hat{\beta}).$
(A.35)

As in A.27, we find that

$\operatorname{Var}^{\ast}\Big{(}\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}s_{gh}^{\ast}\Big{)}=\operatorname{Var}^{\ast}\Big{(}\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}\hat{s}_{gh}v_{gh}^{\ast}\Big{)}=\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}\hat{s}_{gh}^{2}=O_{P}\big{(}N\sup_{g,h}N_{gh}\big{)},$

where the second equality uses independence of $v_{gh}^{\ast}$ across $g$ and
$h$ and the third equality uses A.24, A.29, A.30, and Lemma A.1. It follows
that

$\operatorname{Var}^{\ast}(\hat{\beta}^{\ast}-\hat{\beta})=O_{P}\big{(}N^{-1}\sup_{g,h}N_{gh}\big{)}.$
(A.36)

Using A.30 and A.36, we find that ${\rm
E}^{\ast}|\hyperref@@ii[BSdiffA]{\textup{\tagform@{\ref*{BSdiffA}}}}|=O_{P}(\sup_{g,h}N_{gh}\sup_{g}N_{g})$
as in A.31. By the same argument, see also A.32–A.33, we find that
$\operatorname{Var}^{\ast}\hyperref@@ii[BSdiffB]{\textup{\tagform@{\ref*{BSdiffB}}}}=O_{P}(\sup_{g,h}N_{gh}^{2}\sup_{g}N_{g}^{2})$.
∎

######  Lemma A.4.

Let Assumptions 2, 3 and 4 be satisfied and let
$\widehat{\operatorname{Var}}(\hat{\bm{\theta}})$ be given by 21. Suppose also
that either (i) Assumption 1 or (ii) Assumption 6 and $\lambda\geq 2$ is
satisfied. Then, for an arbitrary conforming, non-zero vector ${\bm{\delta}}$
and
${\bm{\delta}}^{\top}{\bm{H}}_{k}={\bm{b}}^{\top}=[{\bm{b}}_{1}^{\top}\otimes{\bm{b}}_{2}^{\top}]$,

$\displaystyle\widehat{\operatorname{Var}}({\bm{\delta}}^{\top}\hat{\bm{\theta}})-2\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}{\bm{b}}_{1}^{\top}{\bm{\Sigma}}_{gh_{1}}{\bm{b}}_{1}{\bm{b}}_{2}^{\top}{\bm{\Sigma}}_{gh_{2}}{\bm{b}}_{2}=O_{P}\big{(}N^{1/\lambda}\sup_{g}N_{g}\sup_{g,h}N_{gh}^{3-1/\lambda}\big{)}$
$\displaystyle\quad+O_{P}\big{(}\sup_{g,h}N_{gh}^{2}\sup_{g}N_{g}^{2}\big{)}+O_{P}\bigg{(}\\!\sup_{g,h}N_{gh}\sup_{g}N_{g}\Big{(}\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}=1}^{M_{g}}{\bm{b}}_{1}^{\top}{\bm{\Sigma}}_{gh_{1}}{\bm{b}}_{1}{\bm{b}}_{2}^{\top}{\bm{\Sigma}}_{gh_{2}}{\bm{b}}_{2}\Big{)}^{\\!\\!1/2}\,\bigg{)}$

and

$\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}{\bm{b}}_{1}^{\top}{\bm{\Sigma}}_{gh_{1}}{\bm{b}}_{1}{\bm{b}}_{2}^{\top}{\bm{\Sigma}}_{gh_{2}}{\bm{b}}_{2}\geq
c\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}=1}^{M_{g}}{\bm{b}}_{1}^{\top}{\bm{\Sigma}}_{gh_{1}}{\bm{b}}_{1}{\bm{b}}_{2}^{\top}{\bm{\Sigma}}_{gh_{2}}{\bm{b}}_{2}.$

###### Proof.

We give the proof of the first result for the univariate case, where
$\widehat{\operatorname{Var}}(\hat{\theta})$ is given by 19, and we show that

$\displaystyle\widehat{\operatorname{Var}}(\hat{\theta})-2$
$\displaystyle\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}\sigma^{2}_{gh_{1}}\sigma^{2}_{gh_{2}}=O_{P}\big{(}N^{1/\lambda}\sup_{g}N_{g}\sup_{g,h}N_{gh}^{3-1/\lambda}\big{)}$
(A.37)
$\displaystyle+O_{P}\big{(}\sup_{g,h}N_{gh}^{2}\sup_{g}N_{g}^{2}\big{)}+O_{P}\bigg{(}\\!\sup_{g,h}N_{gh}\sup_{g}N_{g}\Big{(}\sum_{g=1}^{G}\Big{(}\sum_{h=1}^{M_{g}}\sigma^{2}_{gh}\Big{)}^{\\!2}\,\Big{)}^{\\!\\!1/2}\,\bigg{)}.$

The proof for the multivariate case is nearly identical, but with more
complicated notation.

We decompose the left-hand side of A.37 as

$\displaystyle 2\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}(\hat{s}_{gh_{1}}^{2}\hat{s}_{gh_{2}}^{2}-s_{gh_{1}}^{2}s_{gh_{2}}^{2})$
(A.38) $\displaystyle+2\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}(s_{gh_{1}}^{2}s_{gh_{2}}^{2}-\sigma^{2}_{gh_{1}}\sigma^{2}_{gh_{2}}).$
(A.39)

We first prove the result for A.39 under Assumption 1. We use 18 and 19 to
write

$\hyperref@@ii[proof2a]{\textup{\tagform@{\ref*{proof2a}}}}=4\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}(q_{1,gh}+q_{3,gh}),$
(A.40)

where $q_{1,gh}=\sigma_{gh}^{2}\sum_{j=1}^{h-1}(s_{gj}^{2}-\sigma_{gj}^{2})$
and $q_{3,gh}=(s_{gh}^{2}-\sigma_{gh}^{2})\sum_{j=1}^{h-1}s_{gj}^{2}$. Under
Assumption 1 we have already proven in A.10 that
$\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}q_{1,gh}=O_{P}(\sup_{g,h}N_{gh}^{3/2}\sup_{g}N_{g}^{1/2}(\sum_{g=1}^{G}\sigma_{g}^{4})^{1/2})$.
The sequence $q_{3,gh}$ is a martingale difference with respect to the
filtration $\mathcal{F}_{gh}$ defined just below A.4. When $1<\lambda<2$, we
prove convergence in $L_{\lambda}$-norm. By the von Bahr-Esseen inequality,
${\rm E}\big{|}\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}q_{3,gh}\big{|}^{\lambda}\leq
2\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}{\rm E}|q_{3,gh}|^{\lambda}$, where ${\rm
E}|q_{3,gh}|^{\lambda}\leq{\rm E}|s_{gh}|^{2\lambda}{\rm
E}\big{|}\sum_{j=1}^{h-1}s_{gj}^{2}\big{|}^{\lambda}$, which was analyzed in
A.7. The remainder of the proof for $q_{3,gh}$ with $1<\lambda<2$ is identical
to that of the Lyapunov condition in A.5, showing that
$\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}q_{3,gh}=O_{P}(N^{1/\lambda}\sup_{g}N_{g}\sup_{g,h}N_{gh}^{3-1/\lambda})$.

Next, suppose $\lambda\geq 2$. We find that
$\sum_{j=1}^{h-1}s_{gj}^{2}\leq\sum_{j=1}^{M_{g}}s_{gj}^{2}$ is a non-negative
random variable, and hence is of order $O_{P}({\rm
E}\sum_{j=1}^{M_{g}}s_{gj}^{2})=O_{P}(\sum_{h=1}^{M_{g}}\sigma_{gh}^{2})$.
That is, there exists a constant $K<\infty$ and a set $\mathcal{A}$ with
$P(\mathcal{A})\to 1$ on which $\sum_{j=1}^{h-1}s_{gj}^{2}\leq
K\sum_{h=1}^{M_{g}}\sigma_{gh}^{2}$. Then, on the set $\mathcal{A}$,

${\rm
E}(q_{3,gh}^{2}|\mathcal{F}_{g,h-1})=\operatorname{Var}(s_{gh}^{2})\Big{(}\sum_{j=1}^{h-1}s_{gj}^{2}\Big{)}^{\\!2}\leq
K^{2}\Big{(}\sum_{h=1}^{M_{g}}\sigma_{gh}^{2}\Big{)}^{\\!2}\operatorname{Var}(s_{gh}^{2}),$
(A.41)

and therefore

$\operatorname{Var}(q_{3,gh})\leq
cN_{gh}^{4}\Big{(}\sum_{h=1}^{M_{g}}\sigma_{gh}^{2}\Big{)}^{\\!2}$ (A.42)

by Lemma A.1. Using A.42 and the fact that $q_{3,gh}$ is a martingale
difference sequence, it follows that, on the set $\mathcal{A}$,

$\operatorname{Var}\bigg{(}\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}q_{3,gh}\bigg{)}=\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}\operatorname{Var}(q_{3,gh})\leq
c\sup_{g,h}N_{gh}^{3}\sup_{g}N_{g}\Big{(}\sum_{h=1}^{M_{g}}\sigma_{gh}^{2}\Big{)}^{\\!2}.$

This shows the required result for $q_{3,gh}$ on the set $\mathcal{A}$ when
$\lambda\geq 2$. Because $P(\mathcal{A})\to 1$, this completes the proof for
A.39 under Assumption 1.

We now prove the result for A.39 under Assumption 6 and $\lambda\geq 2$. We
again apply the decomposition in A.40. Define
$Q_{m,g}=\sum_{h=1}^{M_{g}}q_{m,gh}$ for $m=1,3$, which are both independent
across $g$ by Assumption 6. For $Q_{1,g}$ we note that
$\sum_{j=h+1}^{M_{g}}\sigma_{gj}^{2}\leq\sum_{j=1}^{M_{g}}\sigma_{gj}^{2}$ and
apply the Cauchy-Schwarz inequality such that

${\rm
E}(Q_{1,g}^{2})\leq\Big{(}\sum_{h=1}^{M_{g}}\sigma_{gh}^{2}\Big{)}^{\\!2}{\rm
E}\Big{(}\sum_{h=1}^{M_{g}}|s_{gh}^{2}-\sigma_{gh}^{2}|\Big{)}^{\\!2}\leq\Big{(}\sum_{h=1}^{M_{g}}\sigma_{gh}^{2}\Big{)}^{\\!2}\Big{(}\sum_{h=1}^{M_{g}}({\rm
E}(s_{gh}^{2}-\sigma_{gh}^{2})^{2})^{1/2}\Big{)}^{\\!2},$

where last factor on the right-hand side is
$O(\sup_{g,h}N_{gh}^{2}\sup_{g}N_{g}^{2})$ by Lemma A.1. Because $Q_{1,g}$ has
mean zero and is independent across $g$, it follows that

$\operatorname{Var}\Big{(}\sum_{g=1}^{G}Q_{1,g}\Big{)}=\sum_{g=1}^{G}{\rm
E}(Q_{1,g}^{2})\leq c\kern
0.83328pt\sup_{g,h}N_{gh}^{2}\sup_{g}N_{g}^{2}\sum_{g=1}^{G}\Big{(}\sum_{h=1}^{M_{g}}\sigma_{gh}^{2}\Big{)}^{\\!2},$

which proves the result for $Q_{1,g}$. For $Q_{3,g}$ we note that there exists
a constant $K<\infty$ and a set $\mathcal{A}$ with $P(\mathcal{A})\to 1$ such
that, on $\mathcal{A}$, it holds that $\sum_{j=1}^{h-1}s_{gj}^{2}\leq
K\sum_{j=1}^{M_{g}}\sigma_{gj}^{2}$. We can then apply the same proof as for
$Q_{1,g}$. This completes the proof for A.39 under Assumption 6.

To prove the result for A.38, we use 19 and A.24 to write

$\displaystyle\hyperref@@ii[proof2b]{\textup{\tagform@{\ref*{proof2b}}}}={}$
$\displaystyle 2\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}(\hat{s}_{gh_{1}}\hat{s}_{gh_{2}}+s_{gh_{1}}s_{gh_{2}})(\hat{s}_{gh_{1}}\hat{s}_{gh_{2}}-s_{gh_{1}}s_{gh_{2}})$
$\displaystyle={}$ $\displaystyle
2(\hat{\beta}-\beta_{0})^{2}\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}(\hat{s}_{gh_{1}}\hat{s}_{gh_{2}}+s_{gh_{1}}s_{gh_{2}})\sum_{i=1}^{N_{gh_{1}}}x_{gh_{1}i}^{2}\sum_{j=1}^{N_{gh_{2}}}x_{gh_{2}j}^{2}$
(A.43)
$\displaystyle+4(\hat{\beta}-\beta_{0})\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}(\hat{s}_{gh_{1}}\hat{s}_{gh_{2}}+s_{gh_{1}}s_{gh_{2}})s_{gh_{1}}\sum_{i=1}^{N_{gh_{2}}}x_{gh_{2}i}^{2}.$
(A.44)

By another application of A.24 followed by straightforward application of
A.29, A.30, and Lemma A.1, it follows that A.43 is of order
$O_{P}(\sup_{g,h}N_{gh}^{2}\sup_{g}N_{g}^{2})$.

For A.44, we apply again A.24 and write

$\displaystyle\hyperref@@ii[proof3b]{\textup{\tagform@{\ref*{proof3b}}}}={}$
$\displaystyle
8(\hat{\beta}-\beta_{0})\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}s_{gh_{1}}^{2}s_{gh_{2}}\sum_{i=1}^{N_{gh_{2}}}x_{gh_{2}i}^{2}$
(A.45)
$\displaystyle-4(\hat{\beta}-\beta_{0})^{2}\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}\Big{(}s_{gh_{1}}\sum_{i=1}^{N_{gh_{2}}}x_{gh_{2}i}^{2}+s_{gh_{2}}\sum_{j=1}^{N_{gh_{1}}}x_{gh_{1}j}^{2}\Big{)}s_{gh_{1}}\sum_{\ell=1}^{N_{gh_{2}}}x_{gh_{2}\ell}^{2}$
(A.46)
$\displaystyle-4(\hat{\beta}-\beta_{0})^{3}\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}\neq
h_{1}}^{M_{g}}s_{gh_{1}}\Big{(}\sum_{j=1}^{N_{gh_{1}}}x_{gh_{1}j}^{2}\Big{)}\Big{(}\sum_{i=1}^{N_{gh_{2}}}x_{gh_{2}i}^{2}\Big{)}^{\\!2}.$
(A.47)

Direct application of A.29, A.30, and Lemma A.1 shows that A.46 is
$O_{P}(\sup_{g,h}N_{gh}^{2}\sup_{g}N_{g}^{2})$ and that A.47 is
$O_{P}(N^{-1/2}\sup_{g,h}N_{gh}^{2}\sup_{g}N_{g}^{5/2})=O_{P}(\sup_{g,h}N_{gh}^{2}\sup_{g}N_{g}^{2})$.
Finally, for the right-hand side of A.45, we apply the Cauchy-Schwarz
inequality,

$\displaystyle\hyperref@@ii[proof4a]{\textup{\tagform@{\ref*{proof4a}}}}^{2}$
$\displaystyle\leq
64(\hat{\beta}-\beta_{0})^{2}\bigg{(}\sum_{g=1}^{G}\Big{(}\sum_{h_{1}=1}^{M_{g}}s_{gh_{1}}^{2}\sum_{i=1}^{N_{gh_{1}}}x_{gh_{1}i}^{2}\Big{)}^{\\!2}\bigg{)}\bigg{(}\sum_{g=1}^{G}\Big{(}\sum_{h_{2}=1}^{M_{g}}s_{gh_{2}}\Big{)}^{\\!2}\bigg{)}$
$\displaystyle\leq
64(\hat{\beta}-\beta_{0})^{2}\Big{(}\sup_{g,h}\sum_{i=1}^{N_{gh}}x_{ghi}^{2}\Big{)}^{\\!2}\bigg{(}\sum_{g=1}^{G}\Big{(}\sum_{h_{1}=1}^{M_{g}}s_{gh_{1}}^{2}\Big{)}^{\\!2}\bigg{)}\bigg{(}\sum_{g=1}^{G}\Big{(}\sum_{h_{2}=1}^{M_{g}}s_{gh_{2}}\Big{)}^{\\!2}\bigg{)}.$
(A.48)

As in A.41, we find that the penultimate factor on the right-hand side of A.48
is bounded by a constant times
$\sum_{g=1}^{G}(\sum_{h=1}^{M_{g}}\sigma_{gh}^{2})^{2}$ on a set $\mathcal{A}$
with $P(\mathcal{A})\to 1$. The last factor on the right-hand side of A.48 is
a non-negative random variable and hence is of order $O_{P}({\rm
E}\sum_{g=1}^{G}(\sum_{h=1}^{M_{g}}s_{gh})^{2})=O_{P}(\sum_{g=1}^{G}{\rm
E}s_{g}^{2})=O_{P}(N\sup_{g}N_{g})$ by Lemma A.1. Combining these results and
using A.29 and A.30, we find that

$\hyperref@@ii[proof4a]{\textup{\tagform@{\ref*{proof4a}}}}=O_{P}\bigg{(}\\!\sup_{g}N_{g}\sup_{g,h}N_{gh}\Big{(}\sum_{g=1}^{G}\big{(}\sum_{h=1}^{M_{g}}\sigma_{gh}^{2}\big{)}^{\\!2}\Big{)}^{\\!\\!1/2}\bigg{)},$

which proves the required result for A.45, and hence for A.44 and A.38.

To prove the second result of the lemma we write the left-hand side as

$\displaystyle 2\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}=1}^{M_{g}}$
$\displaystyle{\bm{b}}_{1}^{\top}{\bm{\Sigma}}_{gh_{1}}{\bm{b}}_{1}{\bm{b}}_{2}^{\top}{\bm{\Sigma}}_{gh_{2}}{\bm{b}}_{2}-2\sum_{g=1}^{G}\sum_{h=1}^{M_{g}}{\bm{b}}_{1}^{\top}{\bm{\Sigma}}_{gh}{\bm{b}}_{1}{\bm{b}}_{2}^{\top}{\bm{\Sigma}}_{gh}{\bm{b}}_{2}$
$\displaystyle=2\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}=1}^{M_{g}}{\bm{b}}_{1}^{\top}{\bm{\Sigma}}_{gh_{1}}{\bm{b}}_{1}{\bm{b}}_{2}^{\top}{\bm{\Sigma}}_{gh_{2}}{\bm{b}}_{2}\bigg{(}\\!1-\frac{\sum_{h=1}^{M_{g}}{\bm{b}}_{1}^{\top}{\bm{\Sigma}}_{gh}{\bm{b}}_{1}{\bm{b}}_{2}^{\top}{\bm{\Sigma}}_{gh}{\bm{b}}_{2}}{\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}=1}^{M_{g}}{\bm{b}}_{1}^{\top}{\bm{\Sigma}}_{gh_{1}}{\bm{b}}_{1}{\bm{b}}_{2}^{\top}{\bm{\Sigma}}_{gh_{2}}{\bm{b}}_{2}}\bigg{)}$
$\displaystyle\geq
2\sum_{g=1}^{G}\sum_{h_{1}=1}^{M_{g}}\sum_{h_{2}=1}^{M_{g}}{\bm{b}}_{1}^{\top}{\bm{\Sigma}}_{gh_{1}}{\bm{b}}_{1}{\bm{b}}_{2}^{\top}{\bm{\Sigma}}_{gh_{2}}{\bm{b}}_{2}\bigg{(}\\!1-\frac{\sup_{h}{\bm{b}}_{2}^{\top}{\bm{\Sigma}}_{gh}{\bm{b}}_{2}}{\sum_{h=1}^{M_{g}}{\bm{b}}_{2}^{\top}{\bm{\Sigma}}_{gh}{\bm{b}}_{2}}\bigg{)},$

where the inequality is due to
$\sum_{h=1}^{M_{g}}{\bm{b}}_{1}^{\top}{\bm{\Sigma}}_{gh}{\bm{b}}_{1}{\bm{b}}_{2}^{\top}{\bm{\Sigma}}_{gh}{\bm{b}}_{2}\leq(\sup_{h}{\bm{b}}_{2}^{\top}{\bm{\Sigma}}_{gh}{\bm{b}}_{2})\sum_{h=1}^{M_{g}}{\bm{b}}_{1}^{\top}{\bm{\Sigma}}_{gh}{\bm{b}}_{1}$.
The result follows because
$\sup_{g,h}{\bm{b}}_{2}^{\top}{\bm{\Sigma}}_{gh}{\bm{b}}_{2}/({\bm{b}}_{2}^{\top}\sum_{h=1}^{M_{g}}{\bm{\Sigma}}_{gh}{\bm{b}}_{2})\leq\sup_{g,h}\omega_{\max}({\bm{\Sigma}}_{gh}(\sum_{h=1}^{M_{g}}{\bm{\Sigma}}_{gh})^{-1})<~{}1$
by Assumption 4. ∎

## References

  * Abadie et al. (2023) Abadie, A., S. Athey, G. W. Imbens, and J. Wooldridge (2023).  When should you adjust standard errors for clustering?  Quarterly Journal of Economics 138, 1–35.
  * Athey and Imbens (2017) Athey, S. and G. W. Imbens (2017).  The econometrics of randomized experiments.  In A. V. Banerjee and E. Duflo (Eds.), Handbook of Economic Field Experiments, pp. 73–140. Elsevier.
  * Berger and Sinclair (1984) Berger, R. L. and D. F. Sinclair (1984).  Testing hypotheses concerning unions of linear subspaces.  Journal of the American Statistical Association 79, 158–163.
  * Bertrand et al. (2004) Bertrand, M., E. Duflo, and S. Mullainathan (2004).  How much should we trust differences-in-differences estimates?  Quarterly Journal of Economics 119, 249–275.
  * Bester et al. (2011) Bester, C. A., T. G. Conley, and C. B. Hansen (2011).  Inference with dependent data using cluster covariance estimators.  Journal of Econometrics 165, 137–151.
  * Brown (1971) Brown, B. M. (1971).  Martingale central limit theorems.  Annals of Mathematical Statistics 42, 59–66.
  * Cai (2022) Cai, Y. (2022).  A modified randomization test for the level of clustering.  ArXiv e-prints 2105.01008v2, Northwestern University.
  * Cameron et al. (2008) Cameron, A. C., J. B. Gelbach, and D. L. Miller (2008).  Bootstrap-based improvements for inference with clustered errors.  Review of Economics and Statistics 90, 414–427.
  * Cameron et al. (2011) Cameron, A. C., J. B. Gelbach, and D. L. Miller (2011).  Robust inference with multiway clustering.  Journal of Business & Economic Statistics 29, 238–249.
  * Cameron and Miller (2015) Cameron, A. C. and D. L. Miller (2015).  A practitioner’s guide to cluster-robust inference.  Journal of Human Resources 50, 317–372.
  * Carter et al. (2017) Carter, A. V., K. T. Schnepel, and D. G. Steigerwald (2017).  Asymptotic behavior of a $t$ test robust to cluster heterogeneity.  Review of Economics and Statistics 99, 698–709.
  * Chiang et al. (2022) Chiang, H. D., K. Kato, and Y. Sasaki (2022).  Inference for high-dimensional exchangeable arrays.  Journal of the American Statistical Association, to appear.
  * Cho and Phillips (2018) Cho, J. S. and P. C. B. Phillips (2018).  Pythagorean generalization of testing the equality of two symmetric positive definite matrices.  Journal of Econometrics 202, 45–56.
  * Conley et al. (2018) Conley, T. G., S. Gonçalves, and C. B. Hansen (2018).  Inference with dependent data in accounting and finance applications.  Journal of Accounting Research 56, 1139–1203.
  * Davezies et al. (2021) Davezies, L., X. D’Haultfœuille, and Y. Guyonvarch (2021).  Empirical process results for exchangeable arrays.  Annals of Statistics 49, 845–862.
  * Davidson and Flachaire (2008) Davidson, R. and E. Flachaire (2008).  The wild bootstrap, tamed at last.  Journal of Econometrics 146, 162–169.
  * Davidson and MacKinnon (1992) Davidson, R. and J. G. MacKinnon (1992).  A new form of the information matrix test.  Econometrica 60, 145–157.
  * Davidson and MacKinnon (2000) Davidson, R. and J. G. MacKinnon (2000).  Bootstrap tests: How many bootstraps?  Econometric Reviews 19, 55–68.
  * de Chaisemartin and Ramirez-Cuellar (2022) de Chaisemartin, C. and J. Ramirez-Cuellar (2022).  At what level should one cluster standard errors in paired experiments, and in stratified experiments with small strata?  Working Paper 27609, revised, National Bureau of Economic Research.
  * Djogbenou et al. (2019) Djogbenou, A. A., J. G. MacKinnon, and M. Ø. Nielsen (2019).  Asymptotic theory and wild bootstrap inference with clustered errors.  Journal of Econometrics 212, 393–412.
  * Eicker (1963) Eicker, F. (1963).  Asymptotic normality and consistency of the least squares estimators for families of linear regressions.  Annals of Mathematical Statistics 34, 447–456.
  * Esarey and Menger (2019) Esarey, J. and A. Menger (2019).  Practical and effective approaches to dealing with clustered data.  Political Science Research and Methods 7, 541–559.
  * Finn and Achilles (1990) Finn, J. D. and C. M. Achilles (1990).  Answers and questions about class size: A statewide experiment.  American Educational Research Journal 27, 557–577.
  * Hansen (2022) Hansen, B. E. (2022).  Jackknife standard errors for clustered regression.  Working paper, University of Wisconsin.
  * Hansen and Lee (2019) Hansen, B. E. and S. Lee (2019).  Asymptotic theory for clustered samples.  Journal of Econometrics 210, 268–290.
  * Harville (1997) Harville, D. A. (1997).  Matrix Algebra From a Statistician’s Perspective.  New York: Springer-Verlag.
  * Hausman (1978) Hausman, J. A. (1978).  Specification tests in econometrics.  Econometrica 46, 1251–1272.
  * Ibragimov and Müller (2010) Ibragimov, R. and U. K. Müller (2010).  t-statistic based correlation and heterogeneity robust inference.  Journal of Business & Economic Statistics 28, 453–468.
  * Ibragimov and Müller (2016) Ibragimov, R. and U. K. Müller (2016).  Inference with few heterogeneous clusters.  Review of Economics and Statistics 98, 83–96.
  * Imbens and Kolesár (2016) Imbens, G. W. and M. Kolesár (2016).  Robust standard errors in small samples: Some practical advice.  Review of Economics and Statistics 98, 701–712.
  * Krueger (1999) Krueger, A. B. (1999).  Experimental estimates of education production functions.  Quarterly Journal of Economics 114, 497–532.
  * Leeb and Pötscher (2005) Leeb, H. and B. M. Pötscher (2005).  Model selection and inference: Facts and fiction.  Econometric Theory 21, 21–59.
  * Liu (1988) Liu, R. Y. (1988).  Bootstrap procedures under some non-I.I.D. models.  Annals of Statistics 16, 1696–1708.
  * MacKinnon (2013) MacKinnon, J. G. (2013).  Thirty years of heteroskedasticity-robust inference.  In X. Chen and N. R. Swanson (Eds.), Recent Advances and Future Directions in Causality, Prediction, and Specification Analysis, pp. 437–461. Springer.
  * MacKinnon (2019) MacKinnon, J. G. (2019).  How cluster-robust inference is changing applied econometrics.  Canadian Journal of Economics 52, 851–881.
  * MacKinnon et al. (2021) MacKinnon, J. G., M. Ø. Nielsen, and M. D. Webb (2021).  Wild bootstrap and asymptotic inference with multiway clustering.  Journal of Business and Economic Statistics 39, 505–519.
  * MacKinnon et al. (2022) MacKinnon, J. G., M. Ø. Nielsen, and M. D. Webb (2022).  Leverage, influence, and the jackknife in clustered regression models: Reliable inference using summclust.  QED Working Paper 1483, Queen’s University.
  * MacKinnon et al. (2023a) MacKinnon, J. G., M. Ø. Nielsen, and M. D. Webb (2023a).  Cluster-robust inference: A guide to empirical practice.  Journal of Econometrics 232, 272–299.
  * MacKinnon et al. (2023b) MacKinnon, J. G., M. Ø. Nielsen, and M. D. Webb (2023b).  Fast and reliable jackknife and bootstrap methods for cluster-robust inference.  Journal of Applied Econometrics, to appear.
  * MacKinnon and Webb (2017) MacKinnon, J. G. and M. D. Webb (2017).  Wild bootstrap inference for wildly different cluster sizes.  Journal of Applied Econometrics 32, 233–254.
  * MacKinnon and Webb (2018) MacKinnon, J. G. and M. D. Webb (2018).  The wild bootstrap for few (treated) clusters.  Econometrics Journal 21, 114–135.
  * MacKinnon and Webb (2020) MacKinnon, J. G. and M. D. Webb (2020).  Clustering methods for statistical inference.  In K. F. Zimmermann (Ed.), Handbook of Labor, Human Resources and Population Economics. Cham, Switzerland: Springer.
  * MacKinnon and White (1985) MacKinnon, J. G. and H. White (1985).  Some heteroskedasticity consistent covariance matrix estimators with improved finite sample properties.  Journal of Econometrics 29, 305–325.
  * Menzel (2021) Menzel, K. (2021).  Bootstrap with cluster-dependence in two or more dimensions.  Econmetrica 89, 2143–2188.
  * Mosteller (1995) Mosteller, F. (1995).  The Tennessee study of class size in the early school grades.  The Future of Children 5, 113–127.
  * Moulton (1986) Moulton, B. R. (1986).  Random group effects and the precision of regression estimates.  Journal of Econometrics 32, 385–397.
  * Racine and MacKinnon (2007) Racine, J. S. and J. G. MacKinnon (2007).  Simulation-based tests that can use any number of simulations.  Communications in Statistics–Simulation and Computation 36, 357–365.
  * Roodman et al. (2019) Roodman, D., J. G. MacKinnon, M. Ø. Nielsen, and M. D. Webb (2019).  Fast and wild: Bootstrap inference in Stata using boottest.  Stata Journal 19, 4–60.
  * Webb (2022) Webb, M. D. (2022).  Reworking wild bootstrap based inference for clustered errors.  Canadian Journal of Economics, to appear.
  * White (1980) White, H. (1980).  A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity.  Econometrica 48, 817–838.
  * White (1982) White, H. (1982).  Maximum likelihood estimation of misspecified models.  Econometrica 50, 1–25.
  * Wu (1986) Wu, C. F. J. (1986).  Jackknife, bootstrap and other resampling methods in regression analysis.  Annals of Statistics 14, 1261–1295.