11institutetext: University of Illinois at Urbana Champaign 22institutetext:
University of California, Santa Cruz 33institutetext: Saint Francis
Preparatory High School 44institutetext: Palo Alto High School

# Fusing Pretrained ViTs with TCNet for Enhanced EEG Regression

Eric Modesitt  11 Haicheng Yin  22 Williams Huang Wang  33 Brian Lu  44

###### Abstract

The task of Electroencephalogram (EEG) analysis is paramount to the
development of Brain-Computer Interfaces (BCIs). However, to reach the goal of
developing robust, useful BCIs depends heavily on the speed and the accuracy
at which BCIs can understand neural dynamics. In response to that goal, this
paper details the integration of pre-trained Vision Transformers (ViTs) with
Temporal Convolutional Networks (TCNet) to enhance the precision of EEG
regression. The core of this approach lies in harnessing the sequential data
processing strengths of ViTs along with the superior feature extraction
capabilities of TCNet, to significantly improve EEG analysis accuracy. In
addition, we analyze the importance of how to construct optimal patches for
the attention mechanism to analyze, balancing both speed and accuracy
tradeoffs. Our results showcase a substantial improvement in regression
accuracy, as evidenced by the reduction of Root Mean Square Error (RMSE) from
55.4 to 51.8 on EEGEyeNet’s Absolute Position Task, outperforming existing
state-of-the-art models. Without sacrificing performance, we increase the
speed of this model by an order of magnitude (up to 4.32x faster). This
breakthrough not only sets a new benchmark in EEG regression analysis but also
opens new avenues for future research in the integration of transformer
architectures with specialized feature extraction methods for diverse EEG
datasets.

##  1 Introduction

The analysis of Electroencephalogram (EEG) signals is a cornerstone in the
advancement of Brain-Computer Interfaces (BCIs), offering profound insights
into the intricate neural processes of the human brain. EEG regression, in
particular, stands as a pivotal tool in both neuroscience and medical
diagnostics, gaining prominence for its ability to decode complex neural
dynamics. This technique plays a crucial role in a myriad of applications,
ranging from pinpointing brain damage locations to monitoring cognitive
activities and deciphering the neural basis of seizures (Subasi and Ercelebi,
2005; Sabbagh et al., 2020; Teplan, 2002). The essence of EEG regression lies
in its capacity to transform raw EEG data into interpretable and meaningful
information, thus providing an invaluable perspective into the brain’s
operations.

In the realm of machine learning, the advent of Transformer models has marked
a revolutionary shift in EEG regression analysis. Initially celebrated for
their breakthroughs in natural language processing, these models have been
adeptly modified to cater to EEG data analysis, substantially elevating both
the precision and efficiency of the analysis (Liu et al., 2022). A significant
stride in this field is the adaptation of pre-trained Vision Transformers
(ViTs) for EEG datasets, such as ImageNet Deng et al. (2009). The application
of ViTs in EEG regression has demonstrated exceptional results, surpassing
traditional methods across various benchmarks (Yang and Modesitt, 2023).

Concurrently, Temporal Convolutional Networks (TCNet) have emerged as a
formidable force in the field of EEG signal processing. Exhibiting outstanding
capabilities in feature extraction, TCNets excel in identifying intricate
patterns and nuances in EEG data (Bai et al., 2018; Ingolfsson et al., 2020;
Altaheri et al., 2022). Their robustness in capturing temporal dynamics and
their efficacy in EEG signal handling render them an indispensable component
in neural signal analysis.

This study delves into the synergistic integration of ViTs and TCNet, aiming
to harness their combined strengths to substantially augment the accuracy and
reliability of EEG regression. This innovative approach seeks to leverage the
detailed feature extraction of TCNet and the contextual interpretation prowess
of ViTs, hypothesizing a significant enhancement in EEG analysis.

Our research presents a comprehensive evaluation of this hybrid model,
juxtaposing it against previous methodologies to underscore its superiority in
EEG regression. We meticulously examine the performance of the ViT-TCNet
combination, elucidating the contribution of each component to the overall
effectiveness of the model. The implications of our findings extend beyond the
confines of EEG analysis, potentially influencing a broad spectrum of data
interpretation tasks in various scientific and AI-related fields.

In addition to the aforementioned aspects, a notable facet of this study is
the emphasis on the processing speed of the integrated ViT-TCNet model. Speed
is a critical parameter in EEG analysis, especially for real-time applications
in Brain-Computer Interfaces (BCIs) where rapid response times are essential
By optimizing the architecture and employing advanced techniques in model
training and inference, we have successfully accelerated the processing speed
of the EEG analysis. This advancement is particularly significant in scenarios
where real-time data processing is crucial, such as in neurofeedback systems
or in clinical settings where prompt decision-making is imperative. The
increase in processing speed, achieved without compromising the model’s
performance, marks a substantial leap forward in making EEG-based BCIs more
viable and user-friendly.

In the ensuing sections, we will outline the methodology utilized in our
study, present our empirical findings, and discuss the broader implications
and future research directions stemming from our work. This research not only
enriches the existing literature in EEG regression but also sets the stage for
future explorations into the amalgamation of advanced machine learning
architectures for refined neural data analysis.

In summary, the contributions of this work can be articulated in three primary
areas:

1\. Innovative Combination of ViTs and TCNet for Advanced EEG Regression: This
research marks a significant advancement in EEG regression analysis through
the novel integration of pretrained Vision Transformers (ViTs) with Temporal
Convolutional Networks (TCNet). This fusion harnesses ViTs’ exceptional
capability in processing sequential data and TCNet’s robust feature extraction
techniques, culminating in a notable improvement in EEG regression accuracy.In
addition, we open source all of our methods and results at
github.com/ModeEric/EEGViT-TCNet

2\. Enhancement of Model Processing Speed and Efficiency: A key contribution
of this study is the substantial improvement in the processing speed of the
EEG analysis model. Recognizing the importance of swift data processing in
real-time applications such as Brain-Computer Interfaces, the research
introduces optimizations that significantly accelerate the model’s
performance.

3\. Ablation Studies and Future Research Directions: The research undertakes
comprehensive ablation studies to understand the individual and combined
contributions of ViTs and TCNet to the model’s performance. These studies
offer valuable insights into the mechanics of the model, paving the way for
further optimizations.

##  2 Related Work

###  2.1 Deep Learning in EEG

The evolution of EEG signal processing has been significantly influenced by
the emergence of deep learning techniques. Traditional machine learning
methods, while effective, often fall short in capturing the high-dimensional
and complex nature of EEG data. The introduction of deep learning models,
particularly convolutional neural networks (CNNs) and recurrent neural
networks (RNNs), revolutionized this field. These models brought enhanced
capabilities in handling large datasets, extracting relevant features, and
recognizing intricate patterns in EEG signals (Subasi and Ercelebi, 2005;
Sabbagh et al., 2020; Teplan, 2002). This shift not only improved the accuracy
of EEG analyses but also expanded the potential applications in neurological
research and clinical diagnostics.

Deep learning’s impact on EEG signal processing is profound, offering new
perspectives in understanding brain activity. The ability of these models to
learn from data autonomously, without the need for extensive feature
engineering, has opened avenues for more nuanced and detailed analyses of
neural signals. This advancement is crucial in fields where EEG data plays a
pivotal role, such as in the study of cognitive processes, sleep patterns, and
brain-computer interfaces. The integration of advanced deep learning
architectures in EEG analysis heralds a new era of innovation and discovery in
neuroscience.

###  2.2 ViTs in Non-Image Data Analysis

Vision Transformers (ViTs), originally designed for image recognition, have
demonstrated remarkable versatility by extending their application to various
other domains, including EEG data analysis (Dosovitskiy et al., 2020; Wu et
al., 2020; Han et al., 2022). The cornerstone of their success, the self-
attention mechanism, enables ViTs to efficiently manage sequential data, a
feature crucial in interpreting EEG signals Vaswani et al. (2017). This
characteristic of ViTs facilitates an understanding of the complex, temporal
relationships inherent in EEG data, making them an ideal choice for this type
of analysis.

The adaptation of ViTs to non-image data, such as EEG signals, signifies a
major shift in the approach to data analysis across disciplines. It
underscores the potential of transformer models to handle diverse types of
data beyond their initial scope. This cross-domain applicability of ViTs not
only enriches the toolkit available for EEG analysis but also inspires
innovative approaches to data interpretation. The flexibility and
effectiveness of ViTs in handling sequential data pave the way for their
broader adoption in various scientific and analytical fields.

Figure 1: EEGViT architecture, SOTA on EEGEYENET (Yang and Modesitt, 2023).

###  2.3 Temporal Convolutional Networks (TCNet)

:

Temporal Convolutional Networks (TCNet) have gained significant attention for
their ability to process time-series data, particularly in EEG signal
analysis. The architecture of TCNet, with its focus on capturing temporal
dependencies through convolutional layers, makes it exceptionally suited for
extracting detailed features from EEG data (Farha and Gall, 2019; Hewage et
al., 2020). The efficacy of TCNet in identifying subtle patterns and temporal
features in complex datasets has established it as a leading tool in the field
of neural signal processing.

The role of TCNet in EEG data interpretation extends beyond mere feature
extraction. It involves a deeper understanding of the temporal dynamics and
inherent structures within the EEG signals. This understanding is vital in
applications where precise timing and sequence of neural events are critical,
such as in epilepsy research or brain-computer interface development. The
combination of TCNet with other advanced models like ViTs presents a promising
avenue for enhancing EEG analysis, potentially leading to more accurate and
insightful interpretations of neural data.

##  3 Methods

Our study employs an innovative approach by integrating pre-trained Vision
Transformers (ViTs) with Temporal Convolutional Networks (TCNet) to enhance
EEG regression analysis. This section outlines the dataset utilized, the
specifics of the proposed model, and the methodology for evaluating its
effectiveness.

###  3.1 EEGEyeNet Dataset

The data presented here is derived from the EEGEyeNet dataset Kastrati et al.
(2021). The EEGEyeNet dataset encompasses recordings from 356 healthy adults,
including 190 females and 166 males, aged 18 to 80 years. All individuals in
this study provided written informed consent, compliant with the Declaration
of Helsinki, and were compensated monetarily.

The EEG recordings in the EEGEyeNet dataset were obtained using a high-density
128-channel EEG Geodesic Hydrocel system, operating at a sampling rate of 500
Hz with a central recording reference. Eye positions were concurrently
recorded using an EyeLink 1000 Plus system at the same sampling rate. This
setup maintained electrode impedances below 40 kOhm and ensured accurate eye
tracker calibration. Participants were positioned 68 cm from a 24-inch
monitor, with their head stabilized using a chin rest.

EEG data, as recorded in the EEGEyeNet dataset, are prone to various
artifacts, including environmental noise and physiological interferences such
as eye movements and blinks. To address this, the dataset underwent rigorous
preprocessing with two levels: minimal and maximal. The minimal preprocessing
involved identifying and interpolating faulty electrodes, along with applying
a high-pass filter at 40 Hz and a low-pass filter at 0.5 Hz. The maximal
preprocessing, aimed at neuroscientific analyses, further incorporated
Independent Component Analysis (ICA) and IClabel for artifact component
removal.

The EEGEyeNet dataset also includes synchronized EEG and eye-tracking data,
facilitating time-locked analyses relative to event onsets. This
synchronization was stringently verified to ensure a maximum error margin of 2
ms.

The Absolute Position Task, a key component of the EEGEyeNet dataset, involved
participants fixating on sequentially displayed dots at various screen
positions. Each dot appeared for 1.5 to 1.8 seconds, located at one of 25
distinct screen positions. The central dot was presented thrice, resulting in
27 trials per block. This setup, covering the entire screen area, captured a
broad range of gaze positions. Adapted from Son et al. (2020) for fMRI
studies, modifications were made for EEG compatibility, including stimulus
duration and repetition adjustments. The dot presentation followed a pseudo-
randomized sequence across five experimental blocks, repeated six times,
totaling 810 stimuli per participant.

The Absolute Position task is particularly relevant for our research as it
provides a comprehensive dataset for analyzing eye movement patterns and gaze
positions. The variety in dot positions and the high number of trials allow
for a complete assessment of the participants’ gaze behavior, which is crucial
for our objective of determining the exact XY-coordinates of a participant’s
gaze using EEG data.

###  3.2 EEGViT-TCNet Model Architecture

Intending to advance EEG signal analysis, we developed the EEGViT-TCNet model,
a novel architecture that combines Temporal Convolutional Networks (TCNet)
with a pre-trained Vision Transformer (ViT). This model was meticulously
designed to decipher the temporal dynamics and spatial characteristics
embedded within EEG signals.

####  3.2.1 Temporal Convolutional Network (TCNet) Component:

The EEGViT-TCNet model begins with the TCNet component, tailored to embrace
the complexities of EEG data. This component is characterized by:

  * •

Input Layer: Accepting EEG signals, the TCNet is prepared to handle an input
dimensionality of 129, corresponding to the number of recorded EEG channels +1
for including grounding information (as done in the original EEGEyeNet paper).

  * •

Sequential TCNet Layers: The architecture encompasses three layers, with the
number of channel dimensions expanding progressively to 64, 128, and 256. This
hierarchy is instrumental in capturing a comprehensive spectrum of temporal
dependencies inherent in the EEG signals.

    * –

Kernel Size: A kernel size of 3 is uniformly applied across the TCNet layers.

    * –

Dropout: To counteract the potential for overfitting, a dropout rate of 0.75
is employed.

    * –

Causality and Normalization: The layers incorporate weight normalization
alongside the ReLU activation function to enhance the model’s stability and
performance.

Figure 2: An outline of the TCNet functionality (Bai et al., 2018).

####  3.2.2 Convolutional and Batch Normalization Layers:

The pathway pathway we designed from the TCNet layers to the ViT involves:

  * •

Convolutional Layers: Two convolutional layers connect the TCNet and ViT. The
first layer, equipped with 256 filters and a kernel size of (1, 36), is
succeeded by batch normalization and ReLU activation. This configuration
initiates the spatial feature extraction. Subsequently, the second layer
amplifies the channel dimension to 768, aligning with the ViT’s input
specifications. In addition, this layer employs a kernel size of (256, 1) to
effectuate a spatial compression conducive to the subsequent transformer
analysis.

####  3.2.3 Vision Transformer (ViT) Component:

The culmination of the EEGViT-TCNet model’s preprocessing lies in the ViT
component:

  * •

EEG Data Adaptation: Leveraging the "google/vit-base-patch16-224" model from
Huggingface, the configuration is modified to fit the unique format of EEG
data.

  * •

Patch Embeddings Projection: This layer is reimagined as a 1D convolutional
layer, directly accommodating the output from preceding stages BS ensuring the
input’s integration into the transformer architecture.

  * •

Classifier Head: The model ends with a randomly initialized classifier layer,
transitioning through a linear layer, followed by a dropout layer (p=0.1),
culminating in a linear layer that predicts the gaze XY coordinates.

###  3.3 Training and Evaluation Procedure

For training, we employed a supervised learning approach. The model was
trained on a split of the EEG dataset, with 70% used for training and 30% for
validation. During training, we employed a mean squared error loss function,
optimized using the Adam optimizer with a learning rate of 1e-4. To prevent
overfitting, we implemented early stopping based on the validation loss, with
a patience of 10 epochs.

The primary metric for evaluating our model’s performance was the Root Mean
Square Error (RMSE). This metric provides a clear indication of the model’s
accuracy in predicting the gaze coordinates. A lower RMSE value indicates a
closer approximation to the actual gaze positions. To ensure the robustness of
our findings, we conducted five independent runs for each model configuration
and reported the mean and standard deviation of these runs.

Figure 3: An outline of our addition to EEGViT, demonstrating our distinct
feature extraction methodology Yang and Modesitt (2023)Bai et al. (2018).

##  4 Results

Through rigorous testing and comparison, our model has demonstrated its
capability to predict gaze positions with state-of-the-art precision,
outperforming a spectrum of both conventional and advanced methodologies.

###  4.1 Performance Benchmarking

Our EEGViT-TCNet model achieved a Root Mean Square Error (RMSE) of 51.8mm,
marking a significant advancement over existing models. This performance
showcases a 6.5% enhancement in precision over standalone ViT models Yang and
Modesitt (2023). The stark contrast is further accentuated when juxtaposed
with traditional approaches such as Linear Regression and Random Forest, where
the RMSE figures exceed 115mm.

Our model’s evaluation, focusing on a single data partition divided by subject
as done in Kastrati et al. (2021), demonstrates its ability to generalize
effectively. This approach, where the testing data consisted of completely
unseen, new groups of data, highlights the model’s robustness and
adaptability. Despite being tested on different subsets of subjects than what
the model was trained on, it maintained consistent RMSE metrics, underscoring
its sophisticated architecture’s capability to handle the complexities of EEG
data. This consistency in performance across various subject-based segments is
crucial for real-world applications, affirming the model’s potential for
effective generalization.

Model | Absolute Position RMSE (mm)
---|---
Naive Guessing | 123.3 $\pm$ 0.0
KNN | 119.7 $\pm$ 0
RBF SVR | 123 $\pm$ 0
Linear Regression | 118.3 $\pm$ 0
Random Forest | 116.7 $\pm$ 0.1
CNN | 70.4 $\pm$ 1.1
EEGViT (Pre-trained) | 55.4 $\pm$ 0.2
EEGViT-TCNet | 51.8 $\pm$ 0.6

Table 1: Comparative analysis of Root Mean Squared Error (RMSE) across various
models, highlighting the superior performance of the EEGViT-TCNet model. The
values represent the mean $\pm$ standard deviation over five independent runs,
illustrating the model’s consistency and accuracy in the Absolute Position
Task.

###  4.2 Computational Efficiency and Speed Enhancement

One of the features of our EEGViT-TCNet model is its computational efficiency.
In comparison to the original EEGViT architecture, our enhanced model achieves
a substantial speed-up, being 4.32 times faster during inference. This
improvement is not merely a testament to the model’s optimized architecture
but also underscores its practicality for real-time gaze position prediction
applications.

The speed enhancement is achieved through key optimizations in the model’s
architecture, mainly through an optimization of the patch size accuracy-
efficiency tradeoff. These optimizations ensure that our model not only
maintains high accuracy but also operates with greater efficiency, making it
well-suited for deployment in computationally expensive environments.

###  4.3 Ablation Studies

To assess the individual contributions of various components within our
EEGViT-TCNet model, we conducted a series of ablation studies. These studies
aimed to isolate the effects of specific elements of the model, such as the
convolutional layers, dropout rates in the TCNet, and the use of a pretrained
Vision Transformer (ViT). Each variation of the model was evaluated using the
same dataset and metrics, allowing us to directly compare their performance.

####  4.3.1 Impact of Convolutional Layers

We first examined the impact of the additional convolutional layers that
bridge the gap between the TCNet and the ViT on the model’s performance. In
particular, we analyze the results after removing all possible combinations of
1 convolutional layer (spatial, temporal, and pointwise convolution).

By removing the pointwise layer, we observed a slight increase in the Root
Mean Square Error (RMSE) from 51.8 ± 0.6mm to 52.5 ± 0.8mm. This suggests that
the pointwise convolutional layer plays a modest role in feature extraction
and spatial representation, contributing to the model’s overall accuracy.

For both the spatial and temporal layers we observed a significant increase in
the Root Mean Square Error (RMSE) from 51.8 ± 0.6mm to 55.1 ± 0.6mm and 55.0 ±
0.5mm respecitvely. This suggests that both sptial and temproal layers plays a
major role in feature extraction and representation, contributing to the
model’s overall accuracy.

####  4.3.2 Influence of Dropout Rates in TCNet

The role of dropout rates in TCNet was another focus of our study. By varying
the dropout rates, we investigated their effect on the model’s capability to
generalize and prevent overfitting. The original model with a 0.75 dropout
rate achieved an RMSE of 51.8 ± 0.6mm. Reducing the dropout rate to 0
increased the RMSE to 54.1 ± 0.6mm, indicating a higher propensity for
overfitting. Conversely, lower dropout rates of 0.25 and 0.5 yielded RMSEs of
52.5 ± 0.4mm and 52.1 ± 0.4mm, respectively. These findings illustrate a
nuanced balance between dropout rate and model performance, with moderate
dropout rates contributing positively to the model’s accuracy and
generalizability.

####  4.3.3 Contribution of Pretrained ViT

Finally, we evaluated the contribution of using a pretrained ViT in our model.
By replacing the pretrained ViT with a non-pretrained counterpart, the RMSE
increased to 53.2 ± 0.5mm. This increase underscores the significance of
pretraining in enhancing the model’s feature recognition capabilities,
particularly in the context of EEG data analysis.

Model Variation | Absolute Position RMSE (mm)
---|---
EEGViT-TCNet (Ours) | 51.8 $\pm$ 0.6
No Pointwise Conv Layer | 52.5 $\pm$ 0.8
No Temporal Conv Layer | 55.0 $\pm$ 0.5
No Spatial Conv Layer | 55.1 $\pm$ 0.6
0% Dropout | 54.1 $\pm$ 0.6
25% Dropout | 52.5 $\pm$ 0.4
50% Dropout | 52.1 $\pm$ 0.4
No Pretrained ViT | 53.2 $\pm$ 0.5

Table 2: Ablation study results comparing RMSE across various EEGViT-TCNet
model configurations. The values represent the mean $\pm$ standard deviation
over five runs.

These ablation studies reveal the delicate interplay of different
architectural components in optimizing the EEGViT-TCNet model for EEG
regression analysis. The presence of the second convolutional layer, the
calibration of dropout rates in TCNet, and the incorporation of a pretrained
ViT each contribute uniquely to the model’s performance. Our findings
highlight the importance of these components in achieving high precision in
EEG regression tasks, providing valuable insights for future enhancements and
applications of the model.

##  5 Discussion

Our novel approach has not only showcased a marked improvement in regression
accuracy but also set new benchmarks in processing speed and efficiency. The
results obtained from this study reflect the significant potential of
leveraging the strengths of both ViTs and TCNet, underscoring the profound
impact that such hybrid models can have on understanding and interpreting
complex neural dynamics.

The success of the EEGViT-TCNet model in reducing the Root Mean Square Error
(RMSE) to its current levels emphasizes the model’s capability to provide a
more accurate interpretation of EEG data. This breakthrough is particularly
relevant in the development of BCIs, where the precision of signal
interpretation directly correlates to the effectiveness and user-friendliness
of the interface. In clinical settings, the enhanced accuracy and speed of EEG
analysis facilitated by the EEGViT-TCNet model could lead to more timely and
accurate diagnoses of neurological conditions, potentially transforming
patient care.

Throughout the research, adapting ViT to the unique nature of EEG data
highlighted the complexities inherent in neural signal processing. The
preprocessing of EEG signals, essential for maintaining the integrity of
temporal features, posed significant challenges. This process is critical in
ensuring the model’s adaptability and generalizability across different
subjects and experimental conditions, a vital aspect for the practical
application of such technologies.

Looking forward, the field beckons for further exploration into the
scalability of hybrid models like EEGViT-TCNet, particularly in handling
larger datasets and assessing performance in diverse real-world scenarios. A
key area of interest lies in enhancing the interpretability of these deep
learning models. Improved interpretability is crucial for clinical acceptance
and can lead to advancements in personalized medicine, where EEG analysis can
be tailored to individual patients for monitoring or therapeutic purposes.

Moreover, the exploration of other hybrid architectures and their efficacy
across various domains of neural data presents an exciting avenue for
research. The integration of multimodal data sources, alongside the
application of transfer learning techniques, could further refine the accuracy
and applicability of EEG signal analysis methods. Such advancements could pave
the way for the development of more sophisticated BCIs, offering improved
interaction mechanisms between humans and machines.

##  6 Conclusion

This study represents a significant advancement in EEG regression analysis,
underscoring the role of meticulous feature extraction in the efficacy of
sophisticated computational models like Vision Transformers (ViT). The
integration of Temporal Convolutional Networks (TCNet) with pretrained ViTs
has unveiled the vast potential of harmonizing specialized feature extraction
techniques with advanced deep learning frameworks. This synergy not only
elevates the accuracy of EEG analysis but also establishes a new benchmark in
the field, showcasing the profound benefits of refined feature representation.

Our findings highlight the criticality of nuanced feature extraction in
interpreting complex EEG data, with the EEGViT-TCNet model demonstrating
notable performance enhancements. This indicates that features, often
overlooked by traditional models, can be captured and leveraged for more
accurate regression analysis, suggesting a broad array of applications, from
clinical diagnostics to enhanced brain-computer interfaces.

For future research, the horizon of EEG analysis and deep learning promises
continued expansion and innovation. The development of increasingly
sophisticated models capable of navigating the complexities inherent in EEG
data is anticipated. A pivotal challenge will be enhancing the
interpretability of these models, ensuring they not only perform optimally but
also offer actionable insights for practitioners. For example, the performance
of nearly all effective models in EEG analysis currently relies heavily on the
subject-dependent nature on which they are trained, leading to the possibility
of poor generalization in some instances. Moreover, integrating these advanced
models into real-world applications will be crucial, extending the benefits of
this research to society at large.

In sum, the fusion of TCNet and pretrained ViTs within the EEG regression
domain exemplifies the transformative power of targeted feature extraction and
advanced data processing. This study not only redefines the standards for EEG
analysis but also lights the way for future endeavors in the realms of deep
learning and neural data interpretation. As we delve deeper into the
complexities of the human brain, the significance of innovative computational
models grows ever more evident, harboring the potential for groundbreaking
discoveries in neuroscience and artificial intelligence.

## References

  * Altaheri et al. (2022) Hamdi Altaheri, Ghulam Muhammad, and Mansour Alsulaiman.  Physics-informed attention temporal convolutional network for eeg-based motor imagery classification.  _IEEE Transactions on Industrial Informatics_ , 19(2):2249–2258, 2022.
  * Bai et al. (2018) Shaojie Bai, J. Zico Kolter, and Vladlen Koltun.  An empirical evaluation of generic convolutional and recurrent networks for sequence modeling.  _arXiv preprint arXiv:1803.01271_ , 2018.
  * Deng et al. (2009) Jia Deng et al.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_. IEEE, 2009.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Farha and Gall (2019) Yazan Abu Farha and Jurgen Gall.  Ms-tcn: Multi-stage temporal convolutional network for action segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019.
  * Han et al. (2022) Kai Han et al.  A survey on vision transformer.  _IEEE transactions on pattern analysis and machine intelligence_ , 45(1):87–110, 2022.
  * Hewage et al. (2020) Pradeep Hewage et al.  Temporal convolutional neural (tcn) network for an effective weather forecasting using time-series data from the local weather station.  _Soft Computing_ , 24:16453–16482, 2020.
  * Ingolfsson et al. (2020) Thorir Mar Ingolfsson et al.  Eeg-tcnet: An accurate temporal convolutional network for embedded motor-imagery brain–machine interfaces.  In _2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC)_. IEEE, 2020.
  * Kastrati et al. (2021) Ard Kastrati et al.  Eegeyenet: a simultaneous electroencephalography and eye-tracking dataset and benchmark for eye movement prediction.  _arXiv preprint arXiv:2111.05100_ , 2021.
  * Liu et al. (2022) Jiyao Liu et al.  Spatial-temporal transformers for eeg emotion recognition.  In _Proceedings of the 6th International Conference on Advances in Artificial Intelligence_ , 2022.
  * Sabbagh et al. (2020) David Sabbagh et al.  Predictive regression modeling with meg/eeg: from source power to signals and cognitive states.  _NeuroImage_ , 222:116893, 2020.
  * Son et al. (2020) Jake Son, Lei Ai, Ryan Lim, Ting Xu, Stanley Colcombe, Alexandre Rosa Franco, Jessica Cloud, Stephen LaConte, Jonathan Lisinski, Arno Klein, et al.  Evaluating fmri-based estimation of eye gaze during naturalistic viewing.  _Cerebral Cortex_ , 30(3):1171–1184, 2020.
  * Subasi and Ercelebi (2005) Abdulhamit Subasi and Ergun Ercelebi.  Classification of eeg signals using neural network and logistic regression.  _Computer methods and programs in biomedicine_ , 78(2):87–99, 2005.
  * Teplan (2002) Michal Teplan.  Fundamentals of eeg measurement.  In _Measurement Science Review_ , volume 2, 2002.
  * Vaswani et al. (2017) Ashish Vaswani et al.  Attention is all you need.  In _Advances in neural information processing systems_ , volume 30, 2017.
  * Wu et al. (2020) Bichen Wu et al.  Visual transformers: Token-based image representation and processing for computer vision.  _arXiv preprint arXiv:2006.03677_ , 2020.
  * Yang and Modesitt (2023) Ruiqi Yang and Eric Modesitt.  Vit2eeg: Leveraging hybrid pretrained vision transformers for eeg data.  _arXiv preprint arXiv:2308.00454_ , 2023.


x_{1}}\big{[}\operatorname*{\mathcal{N}}(\boldsymbol{\ell}^{m}_{\boldsymbol{\ell}_{1},\boldsymbol{\ell}_{2}}\,|\,\mathbf{W}^{\star}_{m+3})-\boldsymbol{\ell}_{1:2}^{\mathbf{1}_{2}}\big{]}\boldsymbol{\ell}_{3:4}^{\mathbf{1}_{2}}\big{]}\Big{\|}_{\infty}^{[0,h]^{4}}$
$\displaystyle=$ $\displaystyle 2^{-m}+2^{-m}=2\cdot 2^{-m},$

where we have used Lemma 2.1 of the main text and the chain rule. Thus, we can
conclude that

$\displaystyle\left\|\frac{\partial}{\partial
x_{1}}[\mathcal{N}_{\boldsymbol{\ell}^{m}}(\boldsymbol{\ell}_{1:4}\,|\,\mathbf{W}^{\star}_{m+3})-\boldsymbol{\ell}_{1:4}^{\mathbf{1}_{4}}]\right\|_{\infty}^{[0,h]^{4}}\leq
3\cdot 2^{-m}.$

Repeat the above procedure as in the second step of this lemma. Then we are
able to conclude the validity of the third result. The proof is now completed.
$\blacksquare$

Proof of Lemma 2.2:

Without loss of generality, we assume all elements of $\boldsymbol{\alpha}$
are greater than 0. If one element is 0 (say $\alpha_{1}=0$), we just
rearrange $\boldsymbol{\ell}_{\mathbf{x}\,|\,{\boldsymbol{\alpha}}}$ as
follows:

$\displaystyle\boldsymbol{\ell}_{\mathbf{x}\,|\,{\boldsymbol{\alpha}}}=(x_{2}^{\alpha_{2}},\ldots,x_{r}^{\alpha_{r}},\mathbf{1}_{q}^{\top},x_{1}^{\alpha_{1}})^{\top}=(x_{2}^{\alpha_{2}},\ldots,x_{r}^{\alpha_{r}},\mathbf{1}_{q}^{\top},1)^{\top},$

which has no impact on the proof at all.

Apparently, we have

$\displaystyle\mathbf{x}^{\boldsymbol{\alpha}}=\boldsymbol{\ell}_{\mathbf{x}\,|\,{\boldsymbol{\alpha}}}^{\mathbf{1}_{2^{\lceil\log_{2}r\rceil}}}=\prod_{i=1}^{r}x_{i}^{\alpha_{i}},$

which in connection with Lemma A4 immediately yields the three results of this
lemma. Here the third result follows from the chain rule of the derivative.
The proof is now completed. $\blacksquare$

Proof of Lemma 2.3:

Recall that we have defined the necessary monomials in Section 1. By Lemma A1,
we can find a $p_{\vartheta}(\mathbf{x}\,|\,\mathbf{x}_{0})$ admitting a form
of
$p_{\vartheta}(\mathbf{x}\,|\,\mathbf{x}_{0}):=\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{x}\,|\,\mathbf{x}_{0})^{\top}\boldsymbol{\beta}_{\star}$
such that

$\displaystyle\sup_{C_{\mathbf{x}_{0},h}}|f_{\star}(\mathbf{x})-p_{\vartheta}(\mathbf{x}\,|\,\mathbf{x}_{0})|=O(h^{p}),$

where $\boldsymbol{\beta}_{\star}$ is an $r_{\vartheta}\times 1$ vector
depending on $\mathbf{x}_{0}$.

To proceed, we note that as $h\to 0$, it is guaranteed that

$\displaystyle\mathbf{x}-\mathbf{x}_{0}\in[0,h]^{r}\subseteq[0,1]^{r}$

in the definition of $C_{\mathbf{x}_{0},h}$. Also, recall that we have defined
$\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{x}|\mathbf{x}_{0})$ in the end of
Section 1. Then we write

$\displaystyle\sup_{C_{\mathbf{x}_{0},h}}|f_{\star}(\mathbf{x})-\mathbf{N}(\mathbf{x}\,|\,\mathbf{x}_{0})^{\top}\boldsymbol{\beta}_{\star}|$
$\displaystyle\leq$
$\displaystyle\sup_{C_{\mathbf{x}_{0},h}}|f_{\star}(\mathbf{x})-p_{\vartheta}(\mathbf{x}\,|\,\mathbf{x}_{0})|+\sup_{C_{\mathbf{x}_{0},h}}|p_{\vartheta}(\mathbf{x}\,|\,\mathbf{x}_{0})-\mathbf{N}(\mathbf{x}\,|\,\mathbf{x}_{0})^{\top}\boldsymbol{\beta}_{\star}|$
$\displaystyle\leq$ $\displaystyle
O(h^{p})+\|\boldsymbol{\beta}_{\star}\|\cdot\sup_{C_{\mathbf{x}_{0},h}}\|\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{x}\,|\,\mathbf{x}_{0})-\mathbf{N}(\mathbf{x}\,|\,\mathbf{x}_{0})\|$
$\displaystyle\leq$ $\displaystyle
O(h^{p})+\|\boldsymbol{\beta}_{\star}\|\cdot\sqrt{r_{\vartheta}}\cdot
3^{\lceil\log_{2}\vartheta\rceil-1}2^{-m}=O(h^{p}+2^{-m}),$

where the second inequality follows from Lemma A1, and the third inequality
follows from Lemma 2.2 and the definitions of
$\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{x}\,|\,\mathbf{x}_{0})$ and
$\mathbf{N}(\mathbf{x}\,|\,\mathbf{x}_{0})$.

The proof is now completed. $\blacksquare$

Proof of Theorem 2.1:

By construction of $C_{\mathbf{x}_{\mathbf{i}},h}$’s, for
$\forall\mathbf{x}\in[-a,a]^{r}$, we can always find a
$C_{\mathbf{x}_{\mathbf{i}},h}$ to ensure

$\displaystyle\mathbf{x}\in C_{\mathbf{x}_{\mathbf{i}},h}.$ (B2.37)

Also, we note that $I_{\mathbf{i}}(\mathbf{x})\cdot
I_{\mathbf{j}}(\mathbf{x})=0$ when $\mathbf{i}\neq\mathbf{j}$. Therefore, for
any $\mathbf{x}\in[-a,a]^{r}$, there is only one
$C_{\mathbf{x}_{\mathbf{i}},h}$ to ensure (B2.37). Finally, invoking Lemma
2.3, the result follows. $\blacksquare$

Proof of Lemma 3.1:

Write

$\displaystyle
Q_{T}(\boldsymbol{\theta},\mathbf{B})=\frac{1}{T}\sum_{t=1}^{T}[y_{t}-\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,,\mathbf{B})]^{2}$
$\displaystyle=$
$\displaystyle\frac{1}{T}\sum_{t=1}^{T}\big{[}f_{\star}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star})-\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star}\,|\,\mathbf{B}_{\star})+\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star}\,|\,\mathbf{B}_{\star})-\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{B}_{\star})$
$\displaystyle+\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{B}_{\star})-\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{B})+\varepsilon_{t}\big{]}^{2}$
$\displaystyle=$
$\displaystyle\frac{1}{T}\sum_{t=1}^{T}[f_{\star}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star})-\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star}\,|\,\mathbf{B}_{\star})]^{2}+\frac{1}{T}\sum_{t=1}^{T}[\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star}\,|\,\mathbf{B}_{\star})-\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{B}_{\star})]^{2}$
$\displaystyle+\frac{1}{T}\sum_{t=1}^{T}[\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{B}_{\star})-\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{B})]^{2}+\frac{1}{T}\sum_{t=1}^{T}\varepsilon_{t}^{2}$
$\displaystyle+\frac{2}{T}\sum_{t=1}^{T}[f_{\star}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star})-\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star}\,|\,\mathbf{B}_{\star})][\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star}\,|\,\mathbf{B}_{\star})-\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{B}_{\star})]$
$\displaystyle+\frac{2}{T}\sum_{t=1}^{T}[f_{\star}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star})-\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star}\,|\,\mathbf{B}_{\star})][\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{B}_{\star})-\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{B})]$
$\displaystyle+\frac{2}{T}\sum_{t=1}^{T}[f_{\star}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star})-\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star}\,|\,\mathbf{B}_{\star})]\varepsilon_{t}$
$\displaystyle+\frac{2}{T}\sum_{t=1}^{T}[\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star}\,|\,\mathbf{B}_{\star})-\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{B}_{\star})][\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{B}_{\star})-\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{B})]$
$\displaystyle+\frac{2}{T}\sum_{t=1}^{T}[\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star}\,|\,\mathbf{B}_{\star})-\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{B}_{\star})]\varepsilon_{t}$
$\displaystyle+\frac{2}{T}\sum_{t=1}^{T}[\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{B}_{\star})-\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{B})]\varepsilon_{t}$
$\displaystyle:=$ $\displaystyle Q_{T,1}+\cdots+Q_{T,10},$

where the definitions of $Q_{T,j}$ for $j\in[10]$ are obvious. In what
follows, we consider these terms one by one.

Among $Q_{T,j}$’s, some of them can be studied quite easily. For example, by
Theorem 2.1, it is straightforward to obtain

$\displaystyle Q_{T,1}=O(h^{2p}+2^{-2m}),$

which in connection with Cauchy-Schwarz inequality yields that

$\displaystyle\sup_{\boldsymbol{\theta},\mathbf{B}}|Q_{T,j}|=O_{P}(h^{p}+2^{-m})\quad\text{for}\quad
j=5,6,7.$

Also, by Assumption 2, it is obvious

$\displaystyle Q_{T,4}=\sigma_{\varepsilon}^{2}+o_{P}(1).$

We next consider
$\frac{1}{T}\sum_{t=1}^{T}\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{B})\varepsilon_{t}$.
Note that by Theorem 2.1 we can write

$\displaystyle\frac{1}{T}\sum_{t=1}^{T}\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{B})\varepsilon_{t}$
$\displaystyle=$
$\displaystyle\sum_{\mathbf{i}\in[M]^{2}}\boldsymbol{\beta}_{\mathbf{i}}^{\top}\frac{1}{T}\sum_{t=1}^{T}I_{\mathbf{i}}(\mathbf{z}_{t})\,\mathbf{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{x}_{\mathbf{i}})\varepsilon_{t}$
$\displaystyle\asymp$
$\displaystyle\sum_{\mathbf{i}\in[M]^{2}}\boldsymbol{\beta}_{\mathbf{i}}^{\top}\frac{1}{T}\sum_{t=1}^{T}I_{\mathbf{i}}(\mathbf{z}_{t})\,\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{x}_{\mathbf{i}})\varepsilon_{t},$

where the last step follows from Lemma 2.2, and the proof of Lemma 2.3. Here,
it is easy to show that

$\displaystyle\sup_{\boldsymbol{\theta}}\left\|\frac{1}{T}\sum_{t=1}^{T}I_{\mathbf{i}}(\mathbf{z}_{t})\,\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{x}_{\mathbf{i}})\varepsilon_{t}\right\|=o_{P}(1)$

using the facts that $r_{\vartheta}$ is fixed, and
$\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{z}\,\boldsymbol{\theta}\,|\,\mathbf{x}_{\mathbf{i}})$
is formed by
$(\mathbf{z}\,\boldsymbol{\theta}-\mathbf{x}_{\mathbf{i}})^{\mathbf{J}}$ with
$0\leq|\mathbf{J}|\leq\vartheta$. Thus, simple algebra shows that

$\displaystyle\sup_{\boldsymbol{\theta},\mathbf{B}}\frac{1}{M^{r}}|Q_{T,j}|=o_{P}(1)\quad\text{for}\quad
j=9,10.$

With the above results in hand, we only need to pay attention to $Q_{T,2}$,
$Q_{T,3}$ and $Q_{T,8}$. To study $Q_{T,3}$, we note that

$\displaystyle\frac{1}{h^{r}}\sum_{\mathbf{i}\in[M]^{r}}E\left[I_{\mathbf{i}}(\mathbf{z}_{t})\mathbf{H}^{-1}\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{x}_{\mathbf{i}})\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{x}_{\mathbf{i}})^{\top}\mathbf{H}^{-1}\right]$
(B2.38) $\displaystyle=$
$\displaystyle\frac{1}{h^{r}}\sum_{\mathbf{i}\in[M]^{r}}\int_{\mathbf{w}\in
C_{\mathbf{x}_{\mathbf{i}}}}\mathbf{H}^{-1}\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{w}\,|\,\mathbf{x}_{\mathbf{i}})\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{w}\,|\,\mathbf{x}_{\mathbf{i}})^{\top}\mathbf{H}^{-1}\phi_{\boldsymbol{\theta}}(\mathbf{w})\mathrm{d}\mathbf{w}$
$\displaystyle=$
$\displaystyle\sum_{\mathbf{i}\in[M]^{r}}\phi_{\boldsymbol{\theta}}(\mathbf{x}_{\mathbf{i}})\int_{[0,1]^{r}}\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{w})\,\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{w})^{\top}\mathrm{d}\mathbf{w}\cdot(1+o(1)),$

where the first equality follows from the construction of $C_{\mathbf{i}}$ in
(3.2), and the second equality follows from integration by substitution and
Assumption 2. Then we can write

$\displaystyle Q_{T,3}$ $\displaystyle=$
$\displaystyle\frac{1}{T}\sum_{t=1}^{T}[\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{B}_{\star})-\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{B})]^{2}$
(B2.39) $\displaystyle\asymp$
$\displaystyle\sum_{\mathbf{i}\in[M]^{r}}\frac{1}{T}\sum_{t=1}^{T}I_{\mathbf{i}}(\mathbf{z}_{t})\left[\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{z}_{t}\,\boldsymbol{\theta}\,|\,\mathbf{x}_{\mathbf{i}})^{\top}\mathbf{H}^{-1}\mathbf{H}(\boldsymbol{\beta}_{\star\mathbf{i}}-\boldsymbol{\beta}_{\mathbf{i}})\right]^{2}$
$\displaystyle\asymp$ $\displaystyle
h^{r}\sum_{\mathbf{i}\in[M]^{r}}\|\mathbf{H}(\boldsymbol{\beta}_{\star\mathbf{i}}-\widehat{\boldsymbol{\beta}}_{\mathbf{i}})\|^{2}\asymp\frac{1}{M^{r}}\sum_{\mathbf{i}\in[M]^{r}}\|\mathbf{H}(\boldsymbol{\beta}_{\star\mathbf{i}}-\widehat{\boldsymbol{\beta}}_{\mathbf{i}})\|^{2},$

where the second step follows from Lemma 2.2, Theorem 2.1 and
$I_{\mathbf{i}}(\mathbf{z}\,\boldsymbol{\theta})I_{\mathbf{j}}(\mathbf{z}\,\boldsymbol{\theta})=0$
for $\mathbf{i}\neq\mathbf{j}$, the third step follows from (B2.38) and
Assumption 2, and the last step follows from the definition of $h$.

Note further that by (B2.39), $Q_{T,2}+Q_{T,3}+Q_{T,8}$ admits a quadratic
form using matrix notation which reaches its minimum value (i.e., 0) at
$(\boldsymbol{\theta}_{\star},\mathbf{B}_{\star})$. From here, using
Assumption 2, it is easy to see that
$\|\widehat{\boldsymbol{\theta}}-\boldsymbol{\theta}_{\star}\|=o_{P}(1)$ and
$\frac{1}{M^{r}}\sum_{\mathbf{i}\in[M]^{r}}\|\mathbf{H}(\boldsymbol{\beta}_{\star\mathbf{i}}-\widehat{\boldsymbol{\beta}}_{\mathbf{i}})\|^{2}=o_{P}(1)$
must be satisfied. Otherwise, in view of the continuity of the quadratic form,
it is easy to know that $Q_{T,2}+Q_{T,3}+Q_{T,8}=c>0$ for some positive
constant $c$ in probability one. The proof is now completed.$\blacksquare$

Proof of Lemma A5:

We consider the two asymptotic distributions one by one below.

(1). For notational simplicity, we let

$\displaystyle\widetilde{\mathbf{A}}_{T}$ $\displaystyle=$
$\displaystyle\frac{1}{\sqrt{T}}\sum_{t=1}^{T}\varepsilon_{t}\cdot\mathbf{f}^{(1)}_{\star}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star})\cdot\widetilde{\mathbf{z}}_{t}\cdot
I_{a,t}:=\frac{1}{\sqrt{T}}\sum_{t=1}^{T}\varepsilon_{t}\mathbf{A}_{t},$
(B2.40)

where the definition of $\mathbf{A}_{t}$ is obvious.

We now proceed to prove the asymptotic normality for
$\frac{1}{\sqrt{T}}\sum_{t=1}^{T}\varepsilon_{t}\mathbf{A}_{t}$. Obviously, we
have

$\displaystyle E[\widetilde{\mathbf{A}}_{T}\widetilde{\mathbf{A}}_{T}^{\top}]$
$\displaystyle=$
$\displaystyle\frac{1}{T}\sum_{t,s=1}^{T}E[\varepsilon_{t}\varepsilon_{s}\,I_{a,t}I_{a,s}\,\mathbf{f}^{(1)}_{\star}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star})\,\widetilde{\mathbf{z}}_{t}\widetilde{\mathbf{z}}_{s}^{\top}\,\mathbf{f}^{(1)}_{\star}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star})]$
(B2.41) $\displaystyle\to$
$\displaystyle\boldsymbol{\Sigma}_{11}+\boldsymbol{\Sigma}_{12}+\boldsymbol{\Sigma}_{12}^{\top}.$

Below, we use small-block and large-block to prove the normality. To employ
the small-block and large-block arguments, we partition the set
$\\{1,\ldots,T\\}$ into $2k_{T}+1$ subsets with large blocks of size $l_{T}$
and small blocks of size $s_{T}$ and the last remaining set of size
$T-k_{T}(l_{T}+s_{T})$, where $l_{T}$ and $s_{T}$ are selected such that

$\displaystyle s_{T}\to\infty,\quad\frac{s_{T}}{l_{T}}\to
0,\quad\frac{l_{T}^{1+\nu}}{T^{\frac{\nu}{2}}}\to 0,\quad\text{and}\quad
k_{T}\equiv\left\lfloor\frac{T}{l_{T}+s_{T}}\right\rfloor,$

and $\nu$ is defined in Assumption 2.

For $j=1,\ldots,k_{T}$, define

$\displaystyle\boldsymbol{\xi}_{j}=\sum_{t=(j-1)(l_{T}+s_{T})+1}^{jl_{T}+(j-1)s_{T}}\varepsilon_{t}\mathbf{A}_{t},\quad\boldsymbol{\eta}_{j}=\sum_{t=jl_{T}+(j-1)s_{T}+1}^{j(l_{T}+s_{T})}\varepsilon_{t}\mathbf{A}_{t},\quad\boldsymbol{\zeta}=\sum_{t=k_{T}(l_{T}+s_{T})+1}^{T}\varepsilon_{t}\mathbf{A}_{t}.$

Note that $\alpha(T)=o(1/T)$ and $k_{T}s_{T}/T\to 0$. By direct calculation,
we immediately obtain that

$\displaystyle\frac{1}{T}E\left\|\sum_{j=1}^{k_{T}}\boldsymbol{\eta}_{j}\right\|^{2}\to
0\quad\text{and}\quad\frac{1}{T}E\left\|\boldsymbol{\zeta}\right\|^{2}\to 0.$

Therefore,

$\displaystyle\frac{1}{\sqrt{T}}\sum_{t=1}^{T}\varepsilon_{t}\mathbf{A}_{t}=\frac{1}{\sqrt{T}}\sum_{j=1}^{k_{T}}\boldsymbol{\xi}_{j}+o_{P}(1).$

By Proposition 2.6 of Fan and Yao (2003), we have as $T\to 0$

$\displaystyle\left|E\left[\exp\left(\frac{iw}{\sqrt{T}}\sum_{j=1}^{k_{T}}\boldsymbol{\xi}_{j}\right)\right]-\prod_{j=1}^{k_{T}}E\left[\exp\left(\frac{iw\boldsymbol{\xi}_{j}}{\sqrt{T}}\right)\right]\right|$
$\displaystyle\leq$ $\displaystyle 16(k_{T}-1)\alpha(s_{T})\to 0,$

where $i$ is the imaginary unit. In connection with (B2.40) and (B2.41), the
Feller condition is fulfilled as follows:

$\displaystyle\frac{1}{T}\sum_{j=1}^{k_{T}}E[\boldsymbol{\xi}_{j}\boldsymbol{\xi}_{j}^{\top}]\to\boldsymbol{\Sigma}_{11}+\boldsymbol{\Sigma}_{12}+\boldsymbol{\Sigma}_{12}^{\top}.$

Also, we note that

$\displaystyle E[\|\boldsymbol{\xi}_{1}\|^{2}\cdot
I(\|\boldsymbol{\xi}_{1}\|\geq\epsilon\sqrt{T})]$ $\displaystyle\leq$
$\displaystyle\\{E\|\boldsymbol{\xi}_{1}\|^{2\cdot\frac{2+\nu}{2}}\\}^{\frac{2}{2+\nu}}\left\\{E[I(\|\boldsymbol{\xi}_{1}\|\geq\epsilon\sqrt{T})]\right\\}^{\frac{\nu}{2+\nu}}$
$\displaystyle\leq$
$\displaystyle\\{E\|\boldsymbol{\xi}_{1}\|^{2+\nu}\\}^{\frac{2}{2+\nu}}\left\\{\frac{E\|\boldsymbol{\xi}_{1}\|^{2+\nu}}{\epsilon^{2+\nu}T^{\frac{2+\nu}{2}}}\right\\}^{\frac{\nu}{2+\nu}}$
$\displaystyle=$
$\displaystyle\frac{1}{\epsilon^{\nu}T^{\frac{\nu}{2}}}\left\\{E\|\boldsymbol{\xi}_{1}\|^{2+\nu}\right\\}^{\frac{1}{2+\nu}\cdot(2+\nu)}$
$\displaystyle=$ $\displaystyle
O(1)\frac{l_{T}^{2+\nu}}{\epsilon^{\nu}T^{\frac{\nu}{2}}}E\|\varepsilon_{1}\mathbf{A}_{1}\|^{2+\nu}$
$\displaystyle=$ $\displaystyle O(1)\frac{l_{T}^{2+\nu}}{T^{\frac{\nu}{2}}},$

where the first inequality follows from Hölder inequality, the second
inequality follows from Chebyshev’s inequality, and the second equality
follows from Minkowski inequality. Consequently,

$\displaystyle\frac{1}{T}\sum_{j=1}^{k_{T}}E[\|\boldsymbol{\xi}_{j}\|^{2}\cdot
I(\|\boldsymbol{\xi}_{j}\|\geq\epsilon\sqrt{T})]=O\left(\frac{k_{T}l_{T}^{2+\nu}}{TT^{\frac{\nu}{2}}}\right)=O\left(\frac{l_{T}^{1+\nu}}{T^{\frac{\nu}{2}}}\right)=o(1),$

where the last step follows from the choice of $l_{T}$ as specified above.
Therefore, the Lindberg condition is justified. Using a Cramér-Wold device,
the first result follows immediately by the standard argument.

(2). Without loss generality, we suppose that $\mathbf{x}_{0}\in
C_{\mathbf{i}}$ for some $\mathbf{i}$. For notational simplicity, we further
let

$\displaystyle\widetilde{\mathbf{C}}_{T}$ $\displaystyle=$
$\displaystyle\frac{1}{\sqrt{Th^{r}}}\sum_{t=1}^{T}\varepsilon_{t}I_{\mathbf{i}}(\mathbf{z}_{t})\mathbf{H}^{-1}\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star}\,|\,\mathbf{x}_{\mathbf{i}}):=\frac{1}{\sqrt{Th^{r}}}\sum_{t=1}^{T}\varepsilon_{t}\mathbf{C}_{t},$
(B2.42)

where the second equality follows from Lemma 2.2, and the definition of
$\mathbf{C}_{t}$ is obvious.

We now proceed and write

$\displaystyle
E\left[\left(\frac{1}{\sqrt{Th^{r}}}\sum_{t=1}^{T}\varepsilon_{t}\mathbf{C}_{t}\right)\left(\frac{1}{\sqrt{Th^{2}}}\sum_{t=1}^{T}\varepsilon_{t}\mathbf{C}_{t}\right)^{\top}\right]$
(B2.43) $\displaystyle=$
$\displaystyle\frac{1}{Th^{r}}\sum_{t=1}^{T}\sigma_{\varepsilon}^{2}E[\mathbf{C}_{t}\mathbf{C}_{t}^{\top}]+\frac{1}{h^{r}}\sum_{t=1}^{T-1}(1-t/T)E[\mathbf{C}_{1+t}\mathbf{C}_{1}^{\top}\varepsilon_{1}\varepsilon_{1+t}]$
$\displaystyle+\frac{1}{h^{r}}\sum_{t=1}^{T-1}(1-t/T)E[\mathbf{C}_{1}\mathbf{C}_{1+t}^{\top}\varepsilon_{1}\varepsilon_{1+t}],$

where the last two terms are the same up to a transpose operation.

The term
$\frac{1}{h^{r}}\sum_{t=1}^{T-1}(1-t/T)E[\mathbf{C}_{1}\mathbf{C}_{1+t}^{\top}\varepsilon_{1}\varepsilon_{1+t}]$
on the right hand side can be bounded as follows.

$\displaystyle\|E[\mathbf{C}_{1}\mathbf{C}_{1+t}^{\top}\varepsilon_{1}\varepsilon_{1+t}]\|$
$\displaystyle\leq$ $\displaystyle
O(1)\alpha(t)^{\nu/(2+\nu)}\left\\{E\|\mathbf{C}_{1}\varepsilon_{1}\|^{2+\nu}\right\\}^{2/(2+\nu)}$
$\displaystyle\leq$ $\displaystyle
O(1)\alpha(t)^{\nu/(2+\nu)}\left\\{E\|\mathbf{C}_{1}\varepsilon_{1}\|^{2+\nu}]\frac{1}{h^{r}}\right\\}^{2/(2+\nu)}\cdot(h^{r})^{2/(2+\nu)}$
$\displaystyle=$ $\displaystyle
O((h^{r})^{2/(2+\nu)})\alpha(t)^{\nu/(2+\nu)}.$

Thus, we have

$\displaystyle\left\|\frac{1}{h^{r}}\sum_{t=1}^{T-1}(1-t/T)E[\mathbf{C}_{1}\mathbf{C}_{1+t}^{\top}\varepsilon_{1}\varepsilon_{1+t}]\right\|$
$\displaystyle\leq$ $\displaystyle
O(1)\frac{1}{h^{r}}\sum_{t=1}^{d_{T}}\left\|E[\mathbf{C}_{1}\mathbf{C}_{1+t}^{\top}\varepsilon_{1}\varepsilon_{1+t}]\right\|+O(1)\frac{1}{h^{r}}\sum_{t=d_{T}+1}^{T}\left\|E[\mathbf{C}_{1}\mathbf{C}_{1+t}^{\top}\varepsilon_{1}\varepsilon_{1+t}]\right\|$
$\displaystyle\leq$ $\displaystyle
O(1)h^{r}\sum_{t=1}^{d_{T}}\left\|E[\frac{1}{h^{r}}I_{\mathbf{i}}(\mathbf{z}_{1})\frac{1}{h^{r}}I_{\mathbf{i}}(\mathbf{z}_{1+t})\varepsilon_{1}\varepsilon_{1+t}]\right\|$
$\displaystyle+O(1)\frac{1}{h^{r}}\sum_{t=d_{T}+1}^{T}\left\|E[\mathbf{C}_{1}\mathbf{C}_{1+t}^{\top}\varepsilon_{1}\varepsilon_{1+t}]\right\|$
$\displaystyle\leq$ $\displaystyle
O(1)h^{r}d_{T}+O(1)\frac{(h^{r})^{2/(2+\nu)}}{h^{r}}\sum_{t=d_{T}+1}^{T}\alpha^{\nu/(2+\nu)}(t)$
$\displaystyle=$ $\displaystyle
O(1)h^{r}d_{T}+O(1)\frac{1}{h^{\frac{r\nu}{2+\nu}}}\sum_{t=d_{T}+1}^{T}\alpha^{\nu/(2+\nu)}(t)=o(1),$

where $\nu>0$ is defined in Assumption 2, and the last step follows from
Assumption 2.1 that we can choose $d_{T}$ to ensure

$\displaystyle d_{T}h^{r}\to
0\quad\text{and}\quad\sum_{t=d_{T}+1}^{T}\alpha^{\nu/(2+\nu)}(t)=o\left(h^{\frac{r\nu}{2+\nu}}\right),$
(B2.44)

which can be achieved by choosing $d_{T}=\lfloor T^{c_{1}}\rfloor$,
$h=T^{-c_{2}}$ and $\alpha(t)=t^{-c_{3}}$ for some suitable $c_{1}>0$,
$c_{2}>0$ and $c_{3}>0$, for example.

Thus, we can conclude that

$\displaystyle
E\left[\left(\frac{1}{\sqrt{Th^{r}}}\sum_{t=1}^{T}\varepsilon_{t}\mathbf{C}_{t}\right)\left(\frac{1}{\sqrt{Th^{r}}}\sum_{t=1}^{T}\varepsilon_{t}\mathbf{C}_{t}\right)^{\top}\right]$
(B2.45) $\displaystyle=$
$\displaystyle\frac{1}{Th^{r}}\sum_{t=1}^{T}\sigma_{\varepsilon}^{2}E[\mathbf{C}_{t}\mathbf{C}_{t}^{\top}]+o(1)$
$\displaystyle\to$
$\displaystyle\sigma_{\varepsilon}^{2}\phi_{\boldsymbol{\theta}_{0}}(\mathbf{x}_{0})\int_{[0,1]^{r}}\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{w})\,\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{w})^{\top}\mathrm{d}\mathbf{w},$

where the last line follows from the development of (B2.38) and the continuity
of $\phi_{\boldsymbol{\theta}_{0}}$ by Assumption 2.

Below, we further use small-block and large-block to prove the normality. To
employ the small-block and large-block arguments, we partition the set
$\\{1,\ldots,T\\}$ into $2k_{T}+1$ subsets with large blocks of size $l_{T}$
and small blocks of size $s_{T}$ and the last remaining set of size
$T-k_{T}(l_{T}+s_{T})$, where $l_{T}$ and $s_{T}$ are selected such that

$\displaystyle s_{T}\to\infty,\quad\frac{s_{T}}{l_{T}}\to
0,\quad\frac{l_{T}^{1+\nu}}{(Th^{r})^{\frac{\nu}{2}}}\to
0,\quad\text{and}\quad
k_{T}\equiv\left\lfloor\frac{T}{l_{T}+s_{T}}\right\rfloor,$

and $\nu$ is defined in Assumption 2.

For $j=1,\ldots,k_{T}$, define

$\displaystyle\boldsymbol{\xi}_{j}=\sum_{t=(j-1)(l_{T}+s_{T})+1}^{jl_{T}+(j-1)s_{T}}\frac{1}{\sqrt{h^{r}}}\mathbf{C}_{t}\varepsilon_{t},\quad\boldsymbol{\eta}_{j}=\sum_{t=jl_{T}+(j-1)s_{T}+1}^{j(l_{T}+s_{T})}\frac{1}{\sqrt{h^{r}}}\mathbf{C}_{t}\varepsilon_{t},$
$\displaystyle\boldsymbol{\zeta}=\sum_{t=k_{T}(l_{T}+s_{T})+1}^{T}\frac{1}{\sqrt{h^{r}}}\mathbf{C}_{t}\varepsilon_{t}.$

Note that $\alpha(T)=o(1/T)$ and $k_{T}s_{T}/T\to 0$. By direct calculation,
we immediately obtain that

$\displaystyle\frac{1}{T}E\left\|\sum_{j=1}^{k_{T}}\boldsymbol{\eta}_{j}\right\|^{2}\to
0\quad\text{and}\quad\frac{1}{T}E\left\|\boldsymbol{\zeta}\right\|^{2}\to 0.$

Therefore,

$\displaystyle\frac{1}{\sqrt{Th^{r}}}\sum_{t=1}^{T}\mathbf{C}_{t}\varepsilon_{t}=\frac{1}{\sqrt{T}}\sum_{j=1}^{k_{T}}\boldsymbol{\xi}_{j}+o_{P}(1).$

By Proposition 2.6 of Fan and Yao (2003), we have as $T\to 0$

$\displaystyle\left|E\left[\exp\left(\frac{iw}{\sqrt{T}}\sum_{j=1}^{k_{T}}\boldsymbol{\xi}_{j}\right)\right]-\prod_{j=1}^{k_{T}}E\left[\exp\left(\frac{iw\boldsymbol{\xi}_{j}}{\sqrt{T}}\right)\right]\right|$
$\displaystyle\leq$ $\displaystyle 16(k_{T}-1)\alpha(s_{T})\to 0,$

where $i$ is the imaginary unit. In connection with (B2.43)-(B2.45), the
Feller condition is fulfilled as follows:

$\displaystyle\frac{1}{T}\sum_{j=1}^{k_{T}}E[\boldsymbol{\xi}_{j}\boldsymbol{\xi}_{j}^{\top}]\to\sigma_{\varepsilon}^{2}\phi_{\boldsymbol{\theta}_{0}}(\mathbf{x}_{0})\int_{[0,1]^{r}}\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{w})\,\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{w})^{\top}\mathrm{d}\mathbf{w}.$

Also, we note that

$\displaystyle E[\|\boldsymbol{\xi}_{1}\|^{2}\cdot
I(\|\boldsymbol{\xi}_{1}\|\geq\epsilon\sqrt{T})]$ $\displaystyle\leq$
$\displaystyle\\{E\|\boldsymbol{\xi}_{1}\|^{2\cdot\frac{2+\nu}{2}}\\}^{\frac{2}{2+\nu}}\left\\{E[I(\|\boldsymbol{\xi}_{1}\|\geq\epsilon\sqrt{T})]\right\\}^{\frac{\nu}{2+\nu}}$
$\displaystyle\leq$
$\displaystyle\\{E\|\boldsymbol{\xi}_{1}\|^{2+\nu}\\}^{\frac{2}{2+\nu}}\left\\{\frac{E\|\boldsymbol{\xi}_{1}\|^{2+\nu}}{\epsilon^{2+\nu}T^{\frac{2+\nu}{2}}}\right\\}^{\frac{\nu}{2+\nu}}$
$\displaystyle=$
$\displaystyle\frac{1}{\epsilon^{\nu}T^{\frac{\nu}{2}}}\left\\{E\|\boldsymbol{\xi}_{1}\|^{2+\nu}\right\\}^{\frac{1}{2+\nu}\cdot(2+\nu)}$
$\displaystyle=$ $\displaystyle
O(1)\frac{l_{T}^{2+\nu}}{\epsilon^{\nu}T^{\frac{\nu}{2}}}E\|\mathbf{C}_{1}\varepsilon_{1}\|^{2+\nu}\cdot\frac{1}{h^{\frac{r}{2}\cdot(2+\nu)}}$
$\displaystyle=$ $\displaystyle
O(1)\frac{l_{T}^{2+\nu}}{\epsilon^{\nu}(Th^{r})^{\frac{\nu}{2}}}\cdot\frac{1}{h^{r}}E\|\mathbf{C}_{1}\varepsilon_{1}\|^{2+\nu}$
$\displaystyle=$ $\displaystyle
O(1)\frac{l_{T}^{2+\nu}}{(Th^{r})^{\frac{\nu}{2}}},$

where the first inequality follows from Hölder inequality, the second
inequality follows from Chebyshev’s inequality, and the second equality
follows from Minkowski inequality. Consequently,

$\displaystyle\frac{1}{T}\sum_{j=1}^{k_{T}}E[\|\boldsymbol{\xi}_{j}\|^{2}\cdot
I(\|\boldsymbol{\xi}_{j}\|\geq\epsilon\sqrt{T})]=O\left(\frac{k_{T}l_{T}^{2+\nu}}{T(Th^{r})^{\frac{\nu}{2}}}\right)=O\left(\frac{l_{T}^{1+\nu}}{(Th^{r})^{\frac{\nu}{2}}}\right)=o(1),$

where the last step follows from the choice of $l_{T}$ as specified above.
Therefore, the Lindberg condition is justified. Using a Cramér-Wold device,
the CLT follows immediately by the standard argument.

The proof is now completed.$\blacksquare$

Proof of Theorem 3.1:

We now start the investigation.

By the first order condition, we have

$\displaystyle 0$ $\displaystyle=$ $\displaystyle\frac{\partial
Q_{T}(\boldsymbol{\theta},\mathbf{B})}{\partial\boldsymbol{\theta}}\big{|}_{(\boldsymbol{\theta},\mathbf{B})=(\widehat{\boldsymbol{\theta}},\widehat{\mathbf{B}})}$
(B2.46) $\displaystyle=$
$\displaystyle-\frac{2}{T}\sum_{t=1}^{T}[y_{t}-\mathscr{N}(\mathbf{z}_{t}\,\widehat{\boldsymbol{\theta}}\,|\,\widehat{\mathbf{B}})]\sum_{\mathbf{i}}I_{\mathbf{i}}(\mathbf{z}_{t})\,\frac{\partial\mathscr{N}(\mathbf{z}_{t}\,\widehat{\boldsymbol{\theta}}\,|\,\widehat{\boldsymbol{\beta}}_{\mathbf{i}})}{\partial\boldsymbol{\theta}}$
$\displaystyle\asymp$
$\displaystyle-\frac{2}{T}\sum_{t=1}^{T}[y_{t}-\mathscr{N}(\mathbf{z}_{t}\,|\,\widehat{\boldsymbol{\theta}},\widehat{\mathbf{B}})]\,\mathbf{f}^{(1)}_{\star}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star})\widetilde{\mathbf{z}}_{t}I_{a,t}$
$\displaystyle:=$
$\displaystyle-2\mathbf{A}_{T,\boldsymbol{\theta}}(\widehat{\boldsymbol{\theta}},\widehat{\mathbf{B}}),$

where the third line follows from Lemma 2.2 and Lemma 3.1. Similarly, we have

$\displaystyle 0$ $\displaystyle=$ $\displaystyle\frac{\partial
Q_{T}(\boldsymbol{\theta},\mathbf{B})}{\partial\boldsymbol{\beta}_{\mathbf{i}}}\big{|}_{(\boldsymbol{\theta},\mathbf{B})=(\widehat{\boldsymbol{\theta}},\widehat{\mathbf{B}})}$
(B2.47) $\displaystyle=$
$\displaystyle-\frac{2}{T}\sum_{t=1}^{T}[y_{t}-\mathscr{N}(\mathbf{z}_{t}\,\widehat{\boldsymbol{\theta}}\,|\,\widehat{\mathbf{B}})]I_{\mathbf{i}}(\mathbf{z}_{t})\,\frac{\partial\mathscr{N}(\mathbf{z}_{t}\,\widehat{\boldsymbol{\theta}}\,|\,\widehat{\boldsymbol{\beta}}_{\mathbf{i}})}{\partial\boldsymbol{\beta}_{\mathbf{i}}}$
$\displaystyle\asymp$
$\displaystyle-\frac{2}{T}\sum_{t=1}^{T}[y_{t}-\mathscr{N}(\mathbf{z}_{t}\,\widehat{\boldsymbol{\theta}}\,|\,\widehat{\mathbf{B}})]\,I_{\mathbf{i}}(\mathbf{z}_{t})\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{z}_{t}\,\widehat{\boldsymbol{\theta}}\,|\,\mathbf{x}_{\mathbf{i}})$
$\displaystyle:=$
$\displaystyle-2\mathbf{A}_{T,\boldsymbol{\beta}_{\mathbf{i}}}(\widehat{\boldsymbol{\theta}},\widehat{\boldsymbol{\beta}}_{\mathbf{i}}),$

where we have used Lemma 2.2 again to obtain the third equality.

To proceed, we further label $\boldsymbol{\beta}_{\mathbf{i}}$’s and
$C_{\mathbf{i}}$’s as

$\displaystyle\boldsymbol{\beta}_{1},\ldots,\boldsymbol{\beta}_{M^{r}}\quad\text{and}\quad
C_{1},\ldots,C_{M^{r}}$

respectively. Thus, we define

$\displaystyle\mathbf{V}(\boldsymbol{\theta},\mathbf{B})=(\boldsymbol{\theta}^{\top},\boldsymbol{\beta}_{1}^{\top},\ldots,\boldsymbol{\beta}_{{M^{r}}}^{\top})^{\top},$
$\displaystyle\mathbf{R}_{T}=\operatorname*{\textrm{diag}}\\{\sqrt{T}\mathbf{I}_{d},\sqrt{Th^{r}}\mathbf{H}\\},$
$\displaystyle\overline{\mathbf{I}}_{\mathbf{x}_{0}}=\operatorname*{\textrm{diag}}\\{I(\mathbf{x}_{0}\in[-a,a]^{r})\mathbf{I}_{d},\mathbf{I}_{\mathbf{x}_{0}}\\},$
$\displaystyle\mathbf{I}_{\mathbf{x}_{0}}=\left(I(\mathbf{x}_{0}\in
C_{1})\mathbf{I}_{r_{\vartheta}},\ldots,I(\mathbf{x}_{0}\in
C_{M^{r}})\mathbf{I}_{r_{\vartheta}}\right),$
$\displaystyle\widetilde{\mathbf{I}}_{\mathbf{x}_{0}}=\operatorname*{\textrm{diag}}\\{I(\mathbf{x}_{0}\in[-a,a]^{r})\mathbf{I}_{d},I(\mathbf{x}_{0}\in
C_{1})\mathbf{I}_{r_{\vartheta}},\ldots,I(\mathbf{x}_{0}\in
C_{M^{r}})\mathbf{I}_{r_{\vartheta}}\\},$
$\displaystyle\mathbf{A}_{T}(\boldsymbol{\theta},\mathbf{B})=(\mathbf{A}_{T,\boldsymbol{\theta}}(\boldsymbol{\theta},\mathbf{B})^{\top},\mathbf{A}_{T,\boldsymbol{\beta}_{1}}(\boldsymbol{\theta},\mathbf{B})^{\top},\ldots,\mathbf{A}_{T,\boldsymbol{\beta}_{M^{r}}}(\boldsymbol{\theta},\mathbf{B})^{\top})^{\top},$
$\displaystyle\mathbf{C}_{T}(\boldsymbol{\theta},\mathbf{B})=\begin{pmatrix}\frac{\partial\mathbf{A}_{T,\boldsymbol{\theta}}(\boldsymbol{\theta},\mathbf{B})}{\boldsymbol{\theta}^{\top}}&\frac{\partial\mathbf{A}_{T,\boldsymbol{\theta}}(\boldsymbol{\theta},\mathbf{B})}{\boldsymbol{\beta}_{1}^{\top}}&\frac{\partial\mathbf{A}_{T,\boldsymbol{\theta}}(\boldsymbol{\theta},\mathbf{B})}{\partial\boldsymbol{\beta}_{2}^{\top}}&\cdots&\frac{\partial\mathbf{A}_{T,\boldsymbol{\theta}}(\boldsymbol{\theta},\mathbf{B})}{\partial\boldsymbol{\beta}_{M^{r}}^{\top}}\\\
\frac{\partial\mathbf{A}_{T,\boldsymbol{\beta}_{1}}(\boldsymbol{\theta},\mathbf{B})}{\partial\boldsymbol{\theta}^{\top}}&\frac{\partial\mathbf{A}_{T,\boldsymbol{\beta}_{1}}(\boldsymbol{\theta},\mathbf{B})}{\partial\boldsymbol{\beta}_{1}^{\top}}&\mathbf{0}&\cdots&\mathbf{0}\\\
\frac{\partial\mathbf{A}_{T,\boldsymbol{\beta}_{2}}(\boldsymbol{\theta},\mathbf{B})}{\partial\boldsymbol{\theta}^{\top}}&\mathbf{0}&\frac{\partial\mathbf{A}_{T,\boldsymbol{\beta}_{2}}(\boldsymbol{\theta},\mathbf{B})}{\partial\boldsymbol{\beta}_{2}^{\top}}&\cdots&\mathbf{0}\\\
\vdots&\vdots&\vdots&\ddots&\vdots\\\
\frac{\partial\mathbf{A}_{T,\boldsymbol{\beta}_{M^{r}}}(\boldsymbol{\theta},\mathbf{B})}{\partial\boldsymbol{\theta}^{\top}}&\mathbf{0}&\mathbf{0}&\cdots&\frac{\partial\mathbf{A}_{T,\boldsymbol{\beta}_{M^{r}}}(\boldsymbol{\theta},\mathcal{B})}{\partial\boldsymbol{\beta}_{M^{r}}^{\top}}\end{pmatrix}.$
(B2.48)

We establish a joint CLT. By Taylor expansion, simple algebra shows that

$\displaystyle\widetilde{\mathbf{I}}_{\mathbf{x}_{0}}\mathbf{A}_{T}(\boldsymbol{\theta},\mathbf{B})\big{|}_{(\boldsymbol{\theta},\mathbf{B})=(\boldsymbol{\theta}_{\star},\mathbf{B}_{\star})}$
$\displaystyle=$
$\displaystyle\widetilde{\mathbf{I}}_{\mathbf{x}_{0}}\mathbf{C}_{T}(\boldsymbol{\theta},\mathbf{B})\big{|}_{(\boldsymbol{\theta},\mathbf{B})=(\widetilde{\boldsymbol{\theta}},\widetilde{\mathbf{B}})}(\mathbf{V}(\widehat{\boldsymbol{\theta}},\widehat{\mathbf{B}})-\mathbf{V}(\boldsymbol{\theta}_{\star},\mathbf{B}_{\star})),$

where $(\widetilde{\boldsymbol{\theta}},\widetilde{\mathbf{B}})$ lie between
$(\widehat{\boldsymbol{\theta}},\widehat{\mathbf{B}})$ and
$(\boldsymbol{\theta}_{\star},\mathbf{B}_{\star})$. In connection with the
fact that $[-a,a]^{r}=\cup_{i=1}^{M^{r}}C_{i}$ and $C_{i}\cap C_{j}=\emptyset$
for $i\neq j$, it is easy to see that

$\displaystyle\mathbf{R}_{T}\overline{\mathbf{I}}_{\mathbf{x}_{0}}(\mathbf{V}(\widehat{\boldsymbol{\theta}},\widehat{\mathbf{B}})-\mathbf{V}(\boldsymbol{\theta}_{\star},\mathbf{B}_{\star}))$
$\displaystyle=$
$\displaystyle\left(\mathbf{R}_{T}^{-1}\overline{\mathbf{I}}_{\mathbf{x}_{0}}\cdot\mathbf{C}_{T}(\boldsymbol{\theta},\mathbf{B})\big{|}_{(\boldsymbol{\theta},\mathbf{B})=(\widetilde{\boldsymbol{\theta}},\widetilde{\mathbf{B}})}\cdot\overline{\mathbf{I}}_{\mathbf{x}_{0}}^{\top}\mathbf{R}_{T}^{-1}\right)^{-1}\cdot\mathbf{R}_{T}^{-1}\cdot\overline{\mathbf{I}}_{\mathbf{x}_{0}}\mathbf{A}_{T}(\boldsymbol{\theta},\mathbf{B})\big{|}_{(\boldsymbol{\theta},\mathbf{B})=(\boldsymbol{\theta}_{\star},\mathbf{B}_{\star})}.$

Note that the evaluation of
$\mathbf{R}_{T}^{-1}\overline{\mathbf{I}}_{\mathbf{x}_{0}}\cdot\mathbf{C}_{T}(\boldsymbol{\theta},\mathbf{B})\big{|}_{(\boldsymbol{\theta},\mathbf{B})=(\widetilde{\boldsymbol{\theta}},\widetilde{\mathbf{B}})}\cdot\overline{\mathbf{I}}_{\mathbf{x}_{0}}^{\top}\mathbf{R}_{T}^{-1}$
is straightforward and similar to (B2.41) and (B2.43) after invoking Weak Law
of Large Numbers and noting that
$\mathbf{C}_{T}(\boldsymbol{\theta},\mathbf{B})$ is continuous with respect to
$(\boldsymbol{\theta},\mathbf{B})$. Therefore, we focus on
$\mathbf{R}_{T}^{-1}\overline{\mathbf{I}}_{\mathbf{x}_{0}}\cdot\mathbf{A}_{T}(\boldsymbol{\theta},\mathbf{B})\big{|}_{(\boldsymbol{\theta},\mathbf{B})=(\boldsymbol{\theta}_{\star},\mathbf{B}_{\star})}$
below.

Write

$\displaystyle\sqrt{T}\mathbf{A}_{T,\boldsymbol{\theta}}(\boldsymbol{\theta},\mathbf{B})\big{|}_{(\boldsymbol{\theta},\mathbf{B})=(\boldsymbol{\theta}_{\star},\mathbf{B}_{\star})}$
(B2.49) $\displaystyle=$
$\displaystyle\frac{1}{\sqrt{T}}\sum_{t=1}^{T}[f_{\star}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star})+\varepsilon_{t}-\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star}\,|\,\mathbf{B}_{\star})]\cdot\mathbf{f}^{(1)}_{\star}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star})\cdot\widetilde{\mathbf{z}}_{t}\cdot
I_{a,t}$ $\displaystyle=$
$\displaystyle\frac{1}{\sqrt{T}}\sum_{t=1}^{T}\varepsilon_{t}\cdot\mathbf{f}^{(1)}_{\star}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star})\cdot\widetilde{\mathbf{z}}_{t}\cdot
I_{a,t}$
$\displaystyle+\frac{1}{\sqrt{T}}\sum_{t=1}^{T}[f_{\star}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star})-\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star}\,|\,\mathbf{B}_{\star})]\cdot\mathbf{f}^{(1)}_{\star}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star})\cdot\widetilde{\mathbf{z}}_{t}\cdot
I_{a,t}.$

Here, it is easy to see that the bias term converges to

$\displaystyle c_{\boldsymbol{\theta}}$ $\displaystyle=$
$\displaystyle\sum_{\mathbf{i}}\int_{C_{\mathbf{i}}}\phi(\mathbf{z})\sum_{\|\mathbf{J}\|_{1}=\vartheta}\frac{\vartheta(\mathbf{z}\,\boldsymbol{\theta}_{\star}-\mathbf{x}_{\mathbf{i}})^{\mathbf{J}}}{\mathbf{J}!}F_{\mathbf{i}}(\mathbf{z}\,\boldsymbol{\theta}_{\star})\mathbf{f}^{(1)}_{\star}(\mathbf{z}\,\boldsymbol{\theta}_{\star})\cdot\mathbf{z}^{\top}\mathbf{1}_{r}d\mathbf{z},$

where

$\displaystyle
F_{\mathbf{i}}(\mathbf{x})=\int_{0}^{1}\left[(1-w)^{\vartheta-1}f^{(\mathbf{J})}(\mathbf{x}_{\mathbf{i}}+w(\mathbf{x}-\mathbf{x}_{\mathbf{i}}))-f^{(\mathbf{J})}(\mathbf{x}_{\mathbf{i}})(\mathbf{x}-\mathbf{x}_{\mathbf{i}})^{\mathbf{J}}\right]\mathrm{d}w.$

Without loss of generality, we suppose that $\mathbf{x}_{0}\in
C_{\mathbf{i}}$, and then write

$\displaystyle\sqrt{Th^{r}}\mathbf{H}^{-1}\mathbf{A}_{T,\boldsymbol{\beta}_{\mathbf{i}}}(\boldsymbol{\theta},\boldsymbol{\beta}_{\mathbf{i}})\big{|}_{(\boldsymbol{\theta},\mathbf{B})=(\boldsymbol{\theta}_{\star},\mathbf{B}_{\star})}$
(B2.50) $\displaystyle=$
$\displaystyle\frac{1}{\sqrt{Th^{r}}}\mathbf{H}^{-1}\sum_{t=1}^{T}[y_{t}-\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star}\,|\,\mathbf{B}_{\star})]I_{\mathbf{i}}(\mathbf{z}_{t})\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star}\,|\,\mathbf{x}_{\mathbf{i}})$
$\displaystyle=$
$\displaystyle\frac{1}{\sqrt{Th^{r}}}\sum_{t=1}^{T}\varepsilon_{t}I_{\mathbf{i}}(\mathbf{z}_{t})\mathbf{H}^{-1}\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star}\,|\,\mathbf{x}_{\mathbf{i}})$
$\displaystyle+\frac{1}{\sqrt{Th^{r}}}\sum_{t=1}^{T}[f_{\star}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star})-\mathscr{N}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star}\,|\,\mathbf{B}_{\star})]I_{\mathbf{i}}(\mathbf{z}_{t})\mathbf{H}^{-1}\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star}\,|\,\mathbf{x}_{\mathbf{i}}).$

Here, the bias term converges to

$\displaystyle c_{f}$ $\displaystyle=$
$\displaystyle\sum_{\mathbf{i}}\frac{I(\mathbf{x}_{0}\in
C_{\mathbf{X}_{\mathbf{i}}})}{h^{r}}\int_{C_{\mathbf{X}_{\mathbf{i}}}}\phi_{\boldsymbol{\theta}}(\mathbf{x})\sum_{\|\mathbf{J}\|_{1}=\vartheta}\frac{\vartheta(\mathbf{x}-\mathbf{x}_{\mathbf{i}})^{\mathbf{J}}}{\mathbf{J}!}F_{\mathbf{i}}(\mathbf{x})\mathbf{H}^{-1}\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{x}\,|\,\mathbf{x}_{\mathbf{i}})d\mathbf{x}.$

In addition, we note that

$\displaystyle\left\|\frac{\sqrt{h^{r}}}{Th^{r}}\mathbf{H}^{-1}\frac{\partial\mathbf{A}_{T,\boldsymbol{\beta}_{\mathbf{i}}}^{(1)}(\boldsymbol{\theta},\mathbf{B})}{\partial\boldsymbol{\theta}^{\top}}\big{|}_{(\boldsymbol{\theta},\mathbf{B})=(\widetilde{\boldsymbol{\theta}},\widetilde{\mathbf{B}})}\right\|$
(B2.51) $\displaystyle=$
$\displaystyle\left\|\mathbf{H}^{-1}\frac{\sqrt{h^{r}}}{Th^{r}}\sum_{t=1}^{T}I_{\mathbf{i}}(\mathbf{z}_{t})\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star}\,|\,\mathbf{x}_{\mathbf{i}})\widetilde{\mathbf{z}}_{t}^{\top}\mathbf{f}^{(1)}_{\star}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star})\right\|$
$\displaystyle\leq$ $\displaystyle
O(1)\frac{\sqrt{h^{r}}}{Th^{r}}\sum_{t=1}^{T}I_{\mathbf{i}}(\mathbf{z}_{t})\|\mathbf{H}^{-1}\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star}\,|\,\mathbf{x}_{\mathbf{i}})\|\cdot\|\mathbf{f}^{(1)}_{\star}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star})\|$
$\displaystyle=$
$\displaystyle\sqrt{h^{r}}\cdot\phi_{\boldsymbol{\theta}_{\star}}(\mathbf{x}_{0})\cdot\|\mathbf{f}^{(1)}_{\star}(\mathbf{x}_{0})\|\int_{[0,1]^{r}}\|\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{w})\|\mathrm{d}\mathbf{w}\cdot(1+o_{P}(1))$
$\displaystyle=$ $\displaystyle O_{P}(\sqrt{h^{r}}),$

where the inequality follows from $\mathbf{z}_{t}\in C_{\mathbf{i}}$ (i.e.,
belonging to a bounded set by design), and the second equity follows from the
integration by substitution and Assumption 2.

Finally, invoking Lemma A5 and in view of (B2.49)-(B2.51), the result follows
immediately. $\blacksquare$

Proof of Corollary 3.1:

In view of the development of Theorem 3.1, we consider the following term
only:

$\displaystyle\frac{1}{\sqrt{T}}\sum_{t=1}^{T}\varepsilon_{t}\cdot\mathbf{f}^{(1)}_{\star}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star})\cdot\widetilde{\mathbf{z}}_{t}\cdot
I_{a,t}\cdot\eta_{t}:=\frac{1}{\sqrt{T}}\sum_{t=1}^{T}\varepsilon_{t}\mathbf{A}_{t}\eta_{t}.$

For the rest of the terms, the development can be done similarly but much
simpler.

Using the Cramér-Wold device, let $\boldsymbol{\ell}$ be a $d\times 1$ vector
and $\|\boldsymbol{\ell}\|=1$. Thus, we consider

$\displaystyle
B^{*}=\frac{1}{\sqrt{T}}\sum_{t=1}^{T}\varepsilon_{t}\boldsymbol{\ell}^{\top}\mathbf{A}_{t}\eta_{t}.$

The goal is to show that

$\displaystyle
B^{*}\to_{D^{*}}N(0,\boldsymbol{\ell}^{\top}(\boldsymbol{\Sigma}_{11}+\boldsymbol{\Sigma}_{12}+\boldsymbol{\Sigma}_{21}^{\top})\boldsymbol{\ell}),$
(B2.52)

which in connection with Theorem 3.1 immediately yields the result.

We now consider

$\displaystyle E^{*}[(B^{*})^{2}]$ $\displaystyle=$
$\displaystyle\frac{1}{T}\sum_{t,s=1}^{T}\varepsilon_{t}\varepsilon_{s}\boldsymbol{\ell}^{\top}\mathbf{A}_{t}\mathbf{A}_{s}\boldsymbol{\ell}E[\eta_{t}\eta_{s}]$
$\displaystyle=$
$\displaystyle\frac{1}{T}\sum_{t,s=1}^{T}\varepsilon_{t}\varepsilon_{s}\boldsymbol{\ell}^{\top}\mathbf{A}_{t}\mathbf{A}_{s}\boldsymbol{\ell}+\frac{1}{T}\sum_{t,s=1}^{T}\varepsilon_{t}\varepsilon_{s}\boldsymbol{\ell}^{\top}\mathbf{A}_{t}\mathbf{A}_{s}\boldsymbol{\ell}(E[\eta_{t}\eta_{s}]-1)$

Note that

$\displaystyle
E\left|\frac{1}{T}\sum_{t,s=1}^{T}\varepsilon_{t}\varepsilon_{s}\boldsymbol{\ell}^{\top}\mathbf{A}_{t}\mathbf{A}_{s}\boldsymbol{\ell}(E[\eta_{t}\eta_{s}]-1)\right|$
$\displaystyle\leq$
$\displaystyle\frac{1}{T}E\left|\sum_{t=1}^{d_{T}}\sum_{s=1}^{T-t}\varepsilon_{t}\varepsilon_{s}\boldsymbol{\ell}^{\top}\mathbf{A}_{t}\mathbf{A}_{s}\boldsymbol{\ell}(E[\eta_{t}\eta_{s}]-1)\right|+\frac{1}{T}E\left|\sum_{t=d_{T}+1}^{T}\sum_{s=1}^{T-t}\varepsilon_{t}\varepsilon_{s}\boldsymbol{\ell}^{\top}\mathbf{A}_{t}\mathbf{A}_{s}\boldsymbol{\ell}(E[\eta_{t}\eta_{s}]-1)\right|$
$\displaystyle=$ $\displaystyle
O(1)\sum_{t=1}^{d_{T}}|a(t/\ell)-a(0)|+\sum_{t=d_{T}+1}^{T}E|\varepsilon_{0}\varepsilon_{t}|$
$\displaystyle\leq$ $\displaystyle
O(1)d_{T}^{2}/\ell+\sum_{t=d_{T}+1}^{T}E|\varepsilon_{0}\varepsilon_{t}|=o(1),$

where the second inequality follows from $a(w)$ being Lipschitz continuous on
$[-1,1]$, and the last equality holds by letting $d_{T}^{2}/\ell\to 0$ and
$d_{T}\to\infty$. In addition, using the property of $\ell$ dependent time
series, we know that $E|(B^{*})^{2}-E[(B^{*})^{2}]|=o(1)$ by the development
of Theorem 1 of Hansen (1992). Then we can further obtain that
$E^{*}[(B^{*})^{2}]=\boldsymbol{\ell}^{\top}(\sigma_{\varepsilon}^{2}\boldsymbol{\Sigma}_{1}+\boldsymbol{\Sigma}_{3}+\boldsymbol{\Sigma}_{3}^{\top})\boldsymbol{\ell}+o_{P}(1)$.

We now rewrite $B^{*}$ as follows.

$\displaystyle B^{*}=\sum_{j=1}^{K}\nu_{j}^{*}+\sum_{j=1}^{K}\varpi_{j}^{*},$
(B2.53)

where

$\displaystyle\nu_{j}^{*}=\sum_{t=B_{j}+1}^{B_{j}+r_{1}}\frac{1}{\sqrt{T}}\varepsilon_{t}\boldsymbol{\ell}^{\top}\mathbf{A}_{t}\eta_{\boldsymbol{\theta},t},\quad\varpi_{j}^{*}=\sum_{t=B_{j}+r_{1}+1}^{B_{j}+r_{1}+r_{2}}\frac{1}{\sqrt{T}}\varepsilon_{t}\boldsymbol{\ell}^{\top}\mathbf{A}_{t}\eta_{\boldsymbol{\theta},t}.$

Moreover, $B_{j}=(j-1)(r_{1}+r_{2})$, and without loss of generality we
suppose that $K=T/(r_{1}+r_{2})$ is an integer for simplicity. Otherwise, one
needs to include the remaining terms in (B2.53) which are negligible for an
obvious reason. In addition, we let

$\displaystyle(r_{1},r_{2})\to(\infty,\infty),\quad\left(\frac{r_{2}}{r_{1}},\frac{r_{1}}{T}\right)\to(0,0),\quad
r_{1}\geq\ell,$ (B2.54)

so the blocks $\varpi_{j}^{*}$’s are mutually independent by the construction
of $\xi_{t}$’s. Note that by $\frac{r_{2}}{r_{1}}\to 0$ of (B2.54),

$\displaystyle\frac{Kr_{2}}{T}\to 0\quad\text{and}\quad\frac{Kr_{1}}{T}\to 1.$

By construction, a direct calculation on the small blocks shows that

$\displaystyle
EE^{*}\left[\left(\sum_{j=1}^{K}\varpi_{j}^{*}\right)^{2}\right]=\sum_{j=1}^{K}EE^{*}[(\varpi_{j}^{*})^{2}]=O(1)\frac{Kr_{2}}{T}=o(1).$

Therefore, the term $\sum_{j=1}^{K}\varpi_{j}^{*}$ of (B2.53) is negligible.

Next, we employ the Lindeberg CLT to establish the asymptotic normality of
$\sum_{j=1}^{K}\nu_{j}^{*}$. Recall that we have shown that
$E^{*}[(B^{*})^{2}]=\boldsymbol{\ell}^{\top}(\sigma_{\varepsilon}^{2}\boldsymbol{\Sigma}_{1}+\boldsymbol{\Sigma}_{3}+\boldsymbol{\Sigma}_{3}^{\top})\boldsymbol{\ell}+o_{P}(1)$
and $\sum_{j=1}^{K}\varpi_{j}^{*}$ of (B2.53) is negligible, so it is easy to
know that

$\displaystyle
E^{*}(\sum_{j=1}^{K}\nu_{j}^{*})^{2}=\boldsymbol{\ell}^{\top}(\sigma_{\varepsilon}^{2}\boldsymbol{\Sigma}_{1}+\boldsymbol{\Sigma}_{3}+\boldsymbol{\Sigma}_{3}^{\top})\boldsymbol{\ell}+o_{P}(1).$

Similar arguments can be seen in (A.8)-(A.9) of Chen et al. (2012). That said,
we just need to verify that for $\forall\epsilon>0$

$\displaystyle\sum_{j=1}^{K}E^{*}\left[(\nu_{j}^{*})^{2}\cdot
I\left(|\nu_{j}^{*}|>\epsilon\right)\right]=o_{P}(1).$ (B2.55)

Before proceeding further, we point out that the series
$\frac{1}{\sqrt{T}}\varepsilon_{t}\boldsymbol{\ell}^{\top}\mathbf{A}_{t}\eta_{t}$
is in fact a mixingale sequence mentioned in Definition 1 of Hansen (1991),
where the term
$|\frac{1}{\sqrt{T}}\varepsilon_{t}\boldsymbol{\ell}^{\top}\mathbf{A}_{t}\eta_{t}|$
is equivalent to $c_{i}$ in the notation of Hansen (1991). This is not hard to
justify given $\\{\xi_{t}\\}$ is an $\ell$-dependent series. When $m$ in the
notation of Hansen (1991) is greater than $\ell$, all the requirements of
Definition 1 of Hansen (1991) are fulfilled. Thus, it allows us to invoke the
asymptotic properties associated to the mixingale sequence in the following
development.

Write

$\displaystyle\sum_{j=1}^{K}\mathbb{E}^{*}[(\nu_{j}^{*})^{2}\cdot
I(|\nu_{j}^{*}|>\epsilon)]$ $\displaystyle\leq$
$\displaystyle\sum_{j=1}^{K}\\{\mathbb{E}^{*}|(\nu_{j}^{*})^{2}|^{\delta/2}\\}^{2/\delta}\cdot\\{\mathbb{E}^{*}[I(|\nu_{j}^{*}|>\epsilon)]\\}^{(\delta-2)/\delta}\leq\sum_{j=1}^{K}\\{\mathbb{E}^{*}|(\nu_{j}^{*})^{2}|^{\delta/2}\\}^{2/\delta}\left\\{\frac{\mathbb{E}^{*}|\nu_{j}^{*}|^{\delta}}{\epsilon^{\delta}}\right\\}^{(\delta-2)/\delta}$
$\displaystyle=$
$\displaystyle\epsilon^{\delta-2}\sum_{j=1}^{K}\mathbb{E}^{*}|\nu_{j}^{*}|^{\delta}=\epsilon^{\delta-2}\sum_{j=1}^{K}\left\\{\mathbb{E}^{*}\left[\left(\sum_{t=B_{j}+1}^{B_{j}+r_{1}}\frac{1}{\sqrt{T}}\varepsilon_{t}\boldsymbol{\ell}^{\top}\mathbf{A}_{t}\eta_{t}\right)^{\delta}\right]\right\\}^{\frac{1}{\delta}\cdot\delta}$
$\displaystyle\leq$ $\displaystyle
O(1)\epsilon^{\delta-2}\sum_{j=1}^{K}\left\\{\sum_{t=B_{j}+1}^{B_{j}+r_{1}}\left(\frac{1}{\sqrt{T}}\varepsilon_{t}\boldsymbol{\ell}^{\top}\mathbf{A}_{t}\eta_{t}\right)^{2}\right\\}^{\frac{1}{2}\cdot\delta}$
$\displaystyle\leq$ $\displaystyle
O(1)\epsilon^{\delta-2}\sum_{j=1}^{K}r_{1}^{\delta/2-1}\sum_{t=B_{j}+1}^{B_{j}+r_{1}}\left(\frac{1}{\sqrt{T}}\varepsilon_{t}\boldsymbol{\ell}^{\top}\mathbf{A}_{t}\eta_{t}\right)^{\delta}$
$\displaystyle\leq$ $\displaystyle
O(1)\epsilon^{\delta-2}\frac{r_{1}^{\delta/2-1}}{T^{\delta/2-1}}\cdot\frac{1}{T}\sum_{t=1}^{T}\left(\varepsilon_{t}\boldsymbol{\ell}^{\top}\mathbf{A}_{t}\eta_{t}\right)^{\delta}$
$\displaystyle=$ $\displaystyle
O_{P}(1)\frac{r_{1}^{\delta/2-1}}{T^{\delta/2-1}}=o_{P}(1),$

where the first inequality follows from the Hölder inequality, the second
inequality follows from the Chebyshev’s inequality, the third inequality
follows from Lemma 2 of Hansen (1991), and the last equality follows from
$r_{1}/(Th)\to 0$ and $\delta>2$ (say, letting $\delta=4$). Thus, we can
conclude the validity of (B2.55).

Based on the above development, we are readily to conclude that (B2.52) holds.
The proof is now completed. $\blacksquare$

Proof of Lemma 4.2:

Write

$\displaystyle\widehat{\boldsymbol{\Lambda}}\,\boldsymbol{\Sigma}_{\mathbf{X}}$
$\displaystyle=$
$\displaystyle\frac{1}{nT}(\boldsymbol{\Lambda}\mathbf{Z}_{2}^{\top}+\mathbf{V})(\boldsymbol{\Lambda}\mathbf{Z}_{2}^{\top}+\mathbf{V})^{\top}\widehat{\boldsymbol{\Lambda}}$
(B2.56) $\displaystyle=$
$\displaystyle\frac{1}{nT}\boldsymbol{\Lambda}\mathbf{Z}_{2}^{\top}\mathbf{Z}_{2}\boldsymbol{\Lambda}^{\top}\widehat{\boldsymbol{\Lambda}}+\frac{1}{nT}\boldsymbol{\Lambda}\mathbf{Z}_{2}^{\top}\mathbf{V}^{\top}\widehat{\boldsymbol{\Lambda}}+\frac{1}{nT}\mathbf{V}\mathbf{Z}_{2}\boldsymbol{\Lambda}^{\top}\widehat{\boldsymbol{\Lambda}}+\frac{1}{nT}\mathbf{V}\mathbf{V}^{\top}\boldsymbol{\Lambda}.$

Consider the terms on the right hand side of (B2.56) one by one below.

First, write

$\displaystyle\frac{1}{\sqrt{n}}\left\|\frac{1}{nT}\boldsymbol{\Lambda}\mathbf{Z}_{2}^{\top}\mathbf{V}^{\top}\widehat{\boldsymbol{\Lambda}}\right\|_{2}\leq
O(1)\frac{1}{\sqrt{n}T}\|\mathbf{Z}_{2}^{\top}\mathbf{V}^{\top}\|_{2},$

where the inequality follows from the facts
$\frac{1}{n}\widehat{\boldsymbol{\Lambda}}^{\top}\widehat{\boldsymbol{\Lambda}}=\mathbf{I}_{d_{2}}$
and
$\frac{1}{n}\boldsymbol{\Lambda}^{\top}\boldsymbol{\Lambda}\to_{P}\boldsymbol{\Sigma}_{\boldsymbol{\Lambda}}$.
Note further that

$\displaystyle
E\|\mathbf{Z}_{2}^{\top}\mathbf{V}^{\top}\|^{2}=E\left\|\sum_{t=1}^{T}\mathbf{z}_{2t}\mathbf{V}_{t}^{\top}\right\|^{2}=\sum_{i=1}^{n}\sum_{t=1}^{T}\sum_{s=1}^{T}E[v_{it}v_{is}\mathbf{z}_{2t}^{\top}\mathbf{z}_{2s}]=O(nT),$
(B2.57)

where the third equality follows from Assumption 4.1. Thus, we have

$\displaystyle\frac{1}{\sqrt{n}}\left\|\frac{1}{nT}\boldsymbol{\Lambda}\mathbf{Z}_{2}^{\top}\mathbf{V}^{\top}\widehat{\boldsymbol{\Lambda}}\right\|_{2}=O_{P}\left(\frac{1}{\sqrt{T}}\right).$

Similarly, we obtain that

$\displaystyle\frac{1}{\sqrt{n}}\left\|\frac{1}{nT}\mathbf{V}\mathbf{Z}_{2}\boldsymbol{\Lambda}^{\top}\widehat{\boldsymbol{\Lambda}}\right\|_{2}=O_{P}\left(\frac{1}{\sqrt{T}}\right).$

Next, we write

$\displaystyle E\left\|\frac{1}{nT}\mathbf{V}\mathbf{V}^{\top}\right\|^{2}$
$\displaystyle=$
$\displaystyle\frac{1}{n^{2}T^{2}}\sum_{i=1}^{n}\sum_{j=1}^{n}\sum_{t=1}^{T}\sum_{s=1}^{T}E[v_{it}v_{is}v_{jt}v_{js}]$
(B2.58) $\displaystyle=$
$\displaystyle\frac{1}{n^{2}T^{2}}\sum_{t=1}^{T}\sum_{s=1}^{T}\left(\sum_{i=1}^{n}\mathbb{E}[v_{it}^{2}v_{is}^{2}]+\sum_{i=1}^{n}\sum_{j\neq
i}\mathbb{E}[v_{it}v_{is}v_{jt}v_{js}]\right)$ $\displaystyle=$
$\displaystyle\frac{1}{n^{2}T^{2}}\sum_{t=1}^{T}\left(\sum_{i=1}^{n}\mathbb{E}[v_{it}^{4}]+\sum_{i=1}^{n}\sum_{j\neq
i}\mathbb{E}[(v_{it}v_{jt}-\sigma_{v,ij})^{2}]\right)$
$\displaystyle+\frac{1}{n^{2}T^{2}}\sum_{t=1}^{T}\sum_{s\neq
t}\left(\sum_{i=1}^{n}\mathbb{E}[v_{it}^{2}v_{is}^{2}]+\sum_{i=1}^{n}\sum_{j\neq
i}\mathbb{E}[(v_{it}v_{jt}-\sigma_{v,ij})(v_{is}v_{js}-\sigma_{v,ij})]\right)$
$\displaystyle+\frac{1}{n^{2}}\sum_{i=1}^{n}\sum_{j\neq i}\sigma_{v,ij}^{2}.$

By Assumption 4.1, we can obtain that

$\displaystyle|E[(v_{it}v_{jt}-\sigma_{v,ij})(v_{is}v_{js}-\sigma_{v,ij})]|$
(B2.59) $\displaystyle\leq$ $\displaystyle
O(1)\alpha(|t-s|)^{\nu/(2+\nu)}\\{E|v_{it}v_{jt}-\sigma_{v,ij}|^{2+\nu}\\}^{2/(2+\nu)}$

using the mixing condition along time dimension. Bring (B2.59) into (B2.58),
it is obvious that

$\displaystyle
E\left\|\frac{1}{nT}\mathbf{V}\mathbf{V}^{\top}\right\|^{2}=O\left(\frac{1}{n}+\frac{1}{T}\right).$
(B2.60)

Therefore, we can conclude that

$\displaystyle\frac{1}{\sqrt{n}}\left\|\widehat{\boldsymbol{\Lambda}}\,\boldsymbol{\Sigma}_{\mathbf{X}}-\frac{1}{nT}\boldsymbol{\Lambda}\mathbf{Z}_{2}^{\top}\mathbf{Z}_{2}\boldsymbol{\Lambda}^{\top}\widehat{\boldsymbol{\Lambda}}\right\|_{2}=O_{P}\left(\frac{1}{\sqrt{n\wedge
T}}\right),$ (B2.61)

which yields that

$\displaystyle\frac{1}{n}\left\|\boldsymbol{\Lambda}^{\top}\widehat{\boldsymbol{\Lambda}}\,\boldsymbol{\Sigma}_{\mathbf{X}}-\frac{1}{T}\mathbf{Z}_{2}^{\top}\mathbf{Z}_{2}\boldsymbol{\Lambda}^{\top}\widehat{\boldsymbol{\Lambda}}\right\|_{2}=O_{P}\left(\frac{1}{\sqrt{n\wedge
T}}\right),$

and

$\displaystyle\left\|\boldsymbol{\Sigma}_{\mathbf{X}}-\frac{\widehat{\boldsymbol{\Lambda}}^{\top}\boldsymbol{\Lambda}}{n}\cdot\frac{\mathbf{Z}_{2}^{\top}\mathbf{Z}_{2}}{T}\cdot\frac{\boldsymbol{\Lambda}^{\top}\widehat{\boldsymbol{\Lambda}}}{n}\right\|_{2}=O_{P}\left(\frac{1}{\sqrt{n\wedge
T}}\right).$

We now proceed, and note

$\displaystyle\widehat{\mathbf{z}}_{2t}$ $\displaystyle=$
$\displaystyle\frac{1}{n}\,\widehat{\boldsymbol{\Lambda}}^{\top}\mathbf{X}_{t}=\frac{1}{n}\,\widehat{\boldsymbol{\Lambda}}^{\top}(\boldsymbol{\Lambda}\mathbf{z}_{2t}+\mathbf{V}_{t}),$

which in connection with (B2.56) yields that

$\displaystyle\frac{1}{T}\sum_{t=1}^{T}\left(\widehat{\mathbf{z}}_{2t}-\frac{1}{n}\,\widehat{\boldsymbol{\Lambda}}^{\top}\boldsymbol{\Lambda}\mathbf{z}_{2t}\right)$
$\displaystyle=$
$\displaystyle\frac{1}{n^{2}T^{2}}\sum_{t=1}^{T}\boldsymbol{\Sigma}_{\mathbf{X}}^{-1}\left(\widehat{\boldsymbol{\Lambda}}^{\top}\boldsymbol{\Lambda}\mathbf{Z}_{2}^{\top}\mathbf{Z}_{2}\boldsymbol{\Lambda}^{\top}+\widehat{\boldsymbol{\Lambda}}^{\top}\mathbf{V}\mathbf{Z}_{2}\boldsymbol{\Lambda}^{\top}+\widehat{\boldsymbol{\Lambda}}^{\top}\boldsymbol{\Lambda}\mathbf{Z}_{2}^{\top}\mathbf{V}^{\top}+\boldsymbol{\Lambda}^{\top}\mathbf{V}\mathbf{V}^{\top}\right)\mathbf{V}_{t}$
$\displaystyle=$
$\displaystyle\boldsymbol{\Sigma}_{\mathbf{X}}^{-1}\frac{\widehat{\boldsymbol{\Lambda}}^{\top}\boldsymbol{\Lambda}}{n}\cdot\frac{\mathbf{Z}_{2}^{\top}\mathbf{Z}_{2}}{T}\cdot\frac{\sum_{t=1}^{T}\boldsymbol{\Lambda}^{\top}\mathbf{V}_{t}}{nT}+\boldsymbol{\Sigma}_{\mathbf{X}}^{-1}\frac{\widehat{\boldsymbol{\Lambda}}^{\top}\mathbf{V}\mathbf{Z}_{2}}{nT}\cdot\frac{\sum_{t=1}^{T}\boldsymbol{\Lambda}^{\top}\mathbf{V}_{t}}{nT}$
$\displaystyle+\boldsymbol{\Sigma}_{\mathbf{X}}^{-1}\frac{\widehat{\boldsymbol{\Lambda}}^{\top}\boldsymbol{\Lambda}}{n}\cdot\frac{\sum_{t=1}^{T}\mathbf{Z}_{2}^{\top}\mathbf{V}^{\top}\mathbf{V}_{t}}{nT^{2}}+\boldsymbol{\Sigma}_{\mathbf{X}}^{-1}\frac{\boldsymbol{\Lambda}^{\top}}{\sqrt{n}}\cdot\frac{p\mathbf{V}\mathbf{V}^{\top}}{nT}\cdot\frac{\sum_{t=1}^{T}\mathbf{V}_{t}}{\sqrt{n}T}.$

Using Assumption 4.1, it is easy to show

$\displaystyle\left\|\boldsymbol{\Sigma}_{\mathbf{X}}^{-1}\frac{\widehat{\boldsymbol{\Lambda}}^{\top}\boldsymbol{\Lambda}}{n}\cdot\frac{\mathbf{Z}_{2}^{\top}\mathbf{Z}_{2}}{T}\cdot\frac{\sum_{t=1}^{T}\boldsymbol{\Lambda}^{\top}\mathbf{V}_{t}}{nT}\right\|=O_{P}\left(\frac{1}{\sqrt{nT}}\right).$

Note further that

$\displaystyle\left\|\boldsymbol{\Sigma}_{\mathbf{X}}^{-1}\frac{\widehat{\boldsymbol{\Lambda}}^{\top}\mathbf{V}\mathbf{Z}_{2}}{nT}\cdot\frac{\sum_{t=1}^{T}\boldsymbol{\Lambda}^{\top}\mathbf{V}_{t}}{nT}\right\|\leq
O_{P}(1)\frac{\|\widehat{\boldsymbol{\Lambda}}\|}{\sqrt{n}}\cdot\frac{\|\mathbf{V}\mathbf{Z}_{2}\|}{\sqrt{n}T}\cdot\frac{\|\sum_{t=1}^{T}\boldsymbol{\Lambda}^{\top}\mathbf{V}_{t}\|}{nT}=o_{P}\left(\frac{1}{\sqrt{nT}}\right),$
$\displaystyle\left\|\boldsymbol{\Sigma}_{\mathbf{X}}^{-1}\frac{\widehat{\boldsymbol{\Lambda}}^{\top}\boldsymbol{\Lambda}}{n}\cdot\frac{\sum_{t=1}^{T}\mathbf{Z}_{2}^{\top}\mathbf{V}^{\top}\mathbf{V}_{t}}{nT^{2}}\right\|\leq
O_{P}(1)\frac{\|\mathbf{Z}_{2}^{\top}\mathbf{V}^{\top}\|}{\sqrt{n}T}\cdot\frac{\|\sum_{t=1}^{T}\mathbf{V}_{t}\|}{\sqrt{n}T}=O_{P}\left(\frac{1}{T}\right),$
$\displaystyle\left\|\boldsymbol{\Sigma}_{\mathbf{X}}^{-1}\frac{\boldsymbol{\Lambda}^{\top}}{\sqrt{n}}\cdot\frac{\mathbf{V}\mathbf{V}^{\top}}{nT}\cdot\frac{\sum_{t=1}^{T}\mathbf{V}_{t}}{\sqrt{n}T}\right\|=O_{P}(1)\frac{\|\widehat{\boldsymbol{\Lambda}}\|}{\sqrt{n}}\cdot\frac{\|\mathbf{V}\mathbf{V}^{\top}\|}{nT}\cdot\frac{\|\sum_{t=1}^{T}\mathbf{V}_{t}\|}{\sqrt{n}T}=O_{P}\left(\frac{1}{\sqrt{T(n\wedge
T)}}\right),$

where we have used Assumption 4, (B2.57), and (B2.60).

Putting everything together, the result follows. $\blacksquare$

Proof of Theorem 4.1:

The consistency can be proved in exactly the same way as in Lemma 3.1, and we
focus on the asymptotic distribution below. Similar to the proof of Theorem
3.1, we start with

$\displaystyle 0$ $\displaystyle=$
$\displaystyle\frac{\partial\widehat{Q}_{T}(\boldsymbol{\theta},\mathbf{B})}{\partial\boldsymbol{\theta}}\big{|}_{(\boldsymbol{\theta},\mathbf{B})=(\widecheck{\boldsymbol{\theta}},\widecheck{\mathbf{B}})}$
$\displaystyle=$
$\displaystyle-\frac{2}{T}\sum_{t=1}^{T}[y_{t}-\mathscr{N}(\widehat{\mathbf{z}}_{t}\,\widecheck{\boldsymbol{\theta}}\,|\,\widecheck{\mathbf{B}})]\sum_{\mathbf{i}}I_{\mathbf{i}}(\widehat{\mathbf{z}}_{t})\,\frac{\partial\mathscr{N}(\widehat{\mathbf{z}}_{t}\,\widecheck{\boldsymbol{\theta}}\,|\,\widecheck{\boldsymbol{\beta}}_{\mathbf{i}})}{\partial\boldsymbol{\theta}}$
$\displaystyle=$
$\displaystyle-\frac{2}{T}\sum_{t=1}^{T}\varepsilon_{t}\sum_{\mathbf{i}}I_{\mathbf{i}}(\widehat{\mathbf{z}}_{t})\,\frac{\partial\mathscr{N}(\widehat{\mathbf{z}}_{t}\,\widecheck{\boldsymbol{\theta}}\,|\,\widecheck{\boldsymbol{\beta}}_{\mathbf{i}})}{\partial\boldsymbol{\theta}}$
$\displaystyle-\frac{2}{T}\sum_{t=1}^{T}[f_{\star}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star})-\mathscr{N}(\widehat{\mathbf{z}}_{t}\,\widecheck{\boldsymbol{\theta}}\,|\,\widecheck{\mathbf{B}})]\sum_{\mathbf{i}}I_{\mathbf{i}}(\widehat{\mathbf{z}}_{t})\,\frac{\partial\mathscr{N}(\widehat{\mathbf{z}}_{t}\,\widecheck{\boldsymbol{\theta}}\,|\,\widecheck{\boldsymbol{\beta}}_{\mathbf{i}})}{\partial\boldsymbol{\theta}}$
$\displaystyle=$
$\displaystyle-\frac{2}{T}\sum_{t=1}^{T}\varepsilon_{t}\sum_{\mathbf{i}}I_{\mathbf{i}}(\widehat{\mathbf{z}}_{t})\,\frac{\partial\mathscr{N}(\widehat{\mathbf{z}}_{t}\,\widecheck{\boldsymbol{\theta}}\,|\,\widecheck{\boldsymbol{\beta}}_{\mathbf{i}})}{\partial\boldsymbol{\theta}}$
$\displaystyle-\frac{2}{T}\sum_{t=1}^{T}[f_{\star}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star})-f_{\star}(\widehat{\mathbf{z}}_{t}\mathbf{W}_{\boldsymbol{\Lambda}}^{-1}\boldsymbol{\theta}_{\star})]\sum_{\mathbf{i}}I_{\mathbf{i}}(\widehat{\mathbf{z}}_{t})\,\frac{\partial\mathscr{N}(\widehat{\mathbf{z}}_{t}\,\widecheck{\boldsymbol{\theta}}\,|\,\widecheck{\boldsymbol{\beta}}_{\mathbf{i}})}{\partial\boldsymbol{\theta}}$
$\displaystyle-\frac{2}{T}\sum_{t=1}^{T}[f_{\star}(\widehat{\mathbf{z}}_{t}\mathbf{W}_{\boldsymbol{\Lambda}}^{-1}\boldsymbol{\theta}_{\star})-\mathscr{N}(\widehat{\mathbf{z}}_{t}\,\widecheck{\boldsymbol{\theta}}\,|\,\widecheck{\mathbf{B}})]\sum_{\mathbf{i}}I_{\mathbf{i}}(\widehat{\mathbf{z}}_{t})\,\frac{\partial\mathscr{N}(\widehat{\mathbf{z}}_{t}\,\widecheck{\boldsymbol{\theta}}\,|\,\widecheck{\boldsymbol{\beta}}_{\mathbf{i}})}{\partial\boldsymbol{\theta}}.$

Note that

$\displaystyle\frac{1}{T}\sum_{t=1}^{T}[f_{\star}(\mathbf{z}_{t}\,\boldsymbol{\theta}_{\star})-f_{\star}(\widehat{\mathbf{z}}_{t}\mathbf{W}_{\boldsymbol{\Lambda}}^{-1}\boldsymbol{\theta}_{\star})]\sum_{\mathbf{i}}I_{\mathbf{i}}(\widehat{\mathbf{z}}_{t})\,\frac{\partial\mathscr{N}(\widehat{\mathbf{z}}_{t}\,\widecheck{\boldsymbol{\theta}}\,|\,\widecheck{\boldsymbol{\beta}}_{\mathbf{i}})}{\partial\boldsymbol{\theta}}=o_{P}\left(\frac{1}{\sqrt{T}}\right),$

where
$\mathbf{W}_{\boldsymbol{\Lambda}}=\operatorname*{\textrm{diag}}\\{\mathbf{I}_{d_{1}},\frac{1}{n}\,\widehat{\boldsymbol{\Lambda}}^{\top}\boldsymbol{\Lambda}\\}$,
and the rate $o_{P}\left(\frac{1}{\sqrt{T}}\right)$ follows from Taylor
expansion and Lemma 4.2.

Then the rest development of the first result is similar to those in Theorem
3.1 by conducting Taylor expansion at
$(\mathbf{W}_{\boldsymbol{\Lambda}}^{-1}\boldsymbol{\theta}_{\star},\mathbf{B}_{\star})$.
$\blacksquare$

Proof of Corollary 4.1:

By Lemma A1, we can find a $p_{\vartheta}(\mathbf{x}\,|\,\mathbf{x}_{0})$
admitting a form of
$p_{\vartheta}(\mathbf{x}\,|\,\mathbf{x}_{0}):=\boldsymbol{\psi}_{r_{\vartheta}}(\mathbf{x}\,|\,\mathbf{x}_{0})^{\top}\boldsymbol{\beta}_{\star}$
such that

$\displaystyle\sup_{C_{\mathbf{x}_{0},h}}|f_{\star}^{(\boldsymbol{\delta})}(\mathbf{x})-p_{\vartheta}^{(\boldsymbol{\delta})}(\mathbf{x}\,|\,\mathbf{x}_{0})|=O(h^{p-\|\boldsymbol{\delta}\|_{1}}),$

where $\boldsymbol{\beta}_{\star}$ is an $r_{\vartheta}\times 1$ vector
depending on $\mathbf{x}_{0}$.

To proceed, we note that as $h\to 0$, it is guaranteed that

$\displaystyle\mathbf{x}-\mathbf{x}_{0}\in[0,h]^{r}\subseteq[0,1]^{r}$

in the definition of $C_{\mathbf{x}_{0},h}$. Also, for notational simplicity,
define

$\displaystyle\boldsymbol{\psi}_{r_{\vartheta}}^{(\boldsymbol{\delta})}(\mathbf{x})=(\psi_{1}^{(\boldsymbol{\delta})}(\mathbf{x}),\ldots,\psi_{r_{\vartheta}}^{(\boldsymbol{\delta})}(\mathbf{x}))^{\top}.$

Then we write

$\displaystyle\sup_{C_{\mathbf{x}_{0},h}}|f_{\star}^{(\boldsymbol{\delta})}(\mathbf{x})-\mathbf{N}_{\boldsymbol{\delta}}(\mathbf{x}\,|\,\mathbf{x}_{0})^{\top}\boldsymbol{\beta}_{\star}|$
$\displaystyle\leq$
$\displaystyle\sup_{C_{\mathbf{x}_{0},h}}|f_{\star}^{(\boldsymbol{\delta})}(\mathbf{x})-p_{\vartheta}^{(\boldsymbol{\delta})}(\mathbf{x}\,|\,\mathbf{x}_{0})|+\sup_{C_{\mathbf{x}_{0},h}}|p_{\vartheta}^{(\boldsymbol{\delta})}(\mathbf{x}\,|\,\mathbf{x}_{0})-\mathbf{N}_{\boldsymbol{\delta}}(\mathbf{x}\,|\,\mathbf{x}_{0})^{\top}\boldsymbol{\beta}_{\star}|$
$\displaystyle\leq$ $\displaystyle
O(h^{p-\|\boldsymbol{\delta}\|_{1}})+\|\boldsymbol{\beta}_{\star}\|\cdot\sup_{C_{\mathbf{x}_{0},h}}\|\boldsymbol{\psi}_{r_{\vartheta}}^{(\boldsymbol{\delta})}(\mathbf{x}\,|\,\mathbf{x}_{0})-\mathbf{N}_{\boldsymbol{\delta}}(\mathbf{x}\,|\,\mathbf{x}_{0})\|$
$\displaystyle\leq$ $\displaystyle
O(h^{p-\|\boldsymbol{\delta}\|_{1}})+\|\boldsymbol{\beta}_{\star}\|\cdot\sqrt{r_{\vartheta}}\cdot
3^{\lceil\log_{2}\vartheta\rceil-1}2^{-m}$ $\displaystyle=$ $\displaystyle
O(h^{p-\|\boldsymbol{\delta}\|_{1}}+2^{-m}),$

where the second inequality follows from Lemma A1, and the third inequality
follows from Lemma 2.2 and the definitions of
$\boldsymbol{\psi}_{r_{\vartheta}}^{(\boldsymbol{\delta})}(\mathbf{x}\,|\,\mathbf{x}_{0})$
and $\mathbf{N}_{\boldsymbol{\delta}}(\mathbf{x}\,|\,\mathbf{x}_{0})$.

The proof is now completed. $\blacksquare$

Proof of Lemma 4.1:

(1). By Theorem 9.42 of Rudin (2004), we may take derivative under the
integral. Note that $\phi_{s}(x)=s\phi(sx)$, and

$\displaystyle\sup_{u}|\sigma_{s}(u)-\sigma(u)|$ $\displaystyle=$
$\displaystyle\sup_{u}\left|\int\sigma(x)\phi_{s}(x-u)dx-\sigma(u)\right|$
$\displaystyle\leq$
$\displaystyle\sup_{u}\int|\sigma(x+u)-\sigma(u)|\phi_{s}(x)dx$
$\displaystyle\leq$ $\displaystyle
c\int|x|\phi_{s}(x)dx=\frac{c}{s}\int|x|\phi(x)dx.$

(2). Note that we can always write

$\displaystyle\sigma_{s}(u)-\sigma(u)$ $\displaystyle=$
$\displaystyle\int[\sigma(x+u)-\sigma(u)]\phi_{s}(x)dx$ $\displaystyle=$
$\displaystyle\int[\sigma(x+u)-\sigma(u)]s\phi(sx)dx$ $\displaystyle=$
$\displaystyle\int[\sigma(x/s+u)-\sigma(u)]\phi(x)dx,$

where the third equality follows from integration by substitution.

In what follows, we consider two cases: (i) $u\geq 0$ and (ii) $u<0$. For case
(i), write

$\displaystyle\sigma_{s}(u)-\sigma(u)$ $\displaystyle=$
$\displaystyle\int_{-1}^{-su}[0-u]\phi(x)dx+\frac{1}{s}\int_{-su}^{1}x\phi(x)dx$
$\displaystyle=$
$\displaystyle-u\int_{-1}^{-su}\phi(x)dx+\frac{1}{s}\int_{-su}^{1}x\phi(x)dx,$

where the first equality follows from the definition of $\sigma(\cdot)$. Note
further that if $su\geq 1$ (i.e., $u\geq s^{-1}$), we obtain that

$\displaystyle\sigma_{s}(u)-\sigma(u)=0$

by the definition of $\phi(\cdot)$. Therefore, it remains to consider the case
$0\leq su\leq 1$ (i.e., $0\leq u\leq\frac{1}{s}$), then it is obvious that

$\displaystyle\sigma_{s}(u)-\sigma(u)$ $\displaystyle=$
$\displaystyle-u\int_{-1}^{-su}\phi(x)dx+\frac{1}{s}\int_{-su}^{su}x\phi(x)dx+\frac{1}{s}\int_{su}^{1}x\phi(x)dx$
$\displaystyle=$
$\displaystyle-u\int_{-1}^{-su}\phi(x)dx+\frac{1}{s}\int_{su}^{1}x\phi(x)dx$
$\displaystyle=$ $\displaystyle
u\int_{1}^{su}\phi(x)dx+\frac{1}{s}\int_{su}^{1}x\phi(x)dx$ $\displaystyle=$
$\displaystyle\int_{su}^{1}\frac{x-su}{s}\phi(x)dx,$

where the second equality follows from $\phi(x)$ being symmetric, and the
third equality follows from integration by substitution. As $\phi(x)$ is
nonnegative by Assumption 3, it immediately yields that

$\displaystyle 0\leq\sigma_{s}(u)-\sigma(u)\leq O(1)\frac{1}{s}.$

For case (ii), it is easy to know that

$\displaystyle\sigma_{s}(u)-\sigma(u)$ $\displaystyle=$
$\displaystyle\frac{1}{s}\int_{-su}^{1}x\phi(x)dx.$

For $su\leq-1$ (i.e., $u\leq-s^{-1}$), we have

$\displaystyle\sigma_{s}(u)-\sigma(u)=0$

by the definition of $\phi(\cdot)$. Thus, it remains to consider $-1\leq
su\leq 0$ (i.e., $-\frac{1}{s}\leq u\leq 0$), then it is obvious that

$\displaystyle 0\leq\sigma_{s}(u)-\sigma(u)\leq O(1)\frac{1}{s}.$

Collecting the results for both cases (i) and (ii), we conclude that for
$|u|\leq s^{-1}$

$\displaystyle 0\leq\sigma_{s}(u)-\sigma(u)\leq O(1)s^{-1},$

and for $|u|>s^{-1}$

$\displaystyle\sigma_{s}(u)-\sigma(u)=0.$

The proof is now completed. $\blacksquare$