#  Learning at the Speed of Wireless: Online Real-Time Learning for AI-Enabled
MIMO in NextG

Jiarui Xu, Shashank Jere, Yifei Song, Yi-Hung Kao, Lizhong Zheng, and Lingjia
Liu J. Xu, S. Jere, Y. Song, Y. Kao, and L. Liu are with Wireless@Virginia
Tech, ECE Department at Virginia Tech. L. Zheng is with the EECS Department at
the Massachusetts Institute of Technology. The corresponding author is L. Liu
(ljliu@ieee.org).

###### Abstract

Integration of artificial intelligence (AI) and machine learning (ML) into the
air interface has been envisioned as a key technology for next-generation
(NextG) cellular networks. At the air interface, multiple-input multiple-
output (MIMO) and its variants such as multi-user MIMO (MU-MIMO) and
massive/full-dimension MIMO have been key enablers across successive
generations of cellular networks with evolving complexity and design
challenges. Initiating active investigation into leveraging AI/ML tools to
address these challenges for MIMO becomes a critical step towards an AI-
enabled NextG air interface. At the NextG air interface, the underlying
wireless environment will be extremely dynamic with operation adaptations
performed on a sub-millisecond basis by MIMO operations such as MU-MIMO
scheduling and rank/link adaptation. Given the enormously large number of
operation adaptation possibilities, we contend that online real-time AI/ML-
based approaches constitute a promising paradigm. To this end, we outline the
inherent challenges and offer insights into the design of such online real-
time AI/ML-based solutions for MIMO operations. An online real-time AI/ML-
based method for MIMO-OFDM channel estimation is then presented, serving as a
potential roadmap for developing similar techniques across various MIMO
operations in NextG.

##  I Introduction

The concept of an artificial intelligence (AI)-enabled cellular network [1]
has attracted much attention from both academia and industry. As a critical
step, the 3rd Generation Partnership Project (3GPP) has initiated the
exploration of AI in the 5G-Advanced air interface. This trend of
standardizing and deploying AI at the air interface is anticipated to continue
and evolve through NextG networks.

The growing interest in this domain mainly arises from the intrinsic issues of
_network complexity_ , _model deficit_ , and _algorithm deficit_ as discussed
in [1], but tailored towards the air interface of the NextG. Specifically, the
air interface of the NextG (e.g., 6G and beyond) is expected to be
increasingly sophisticated with complex network topologies/numerologies, non-
linear device components, and high-complexity processing algorithms.
Therefore, it becomes exceedingly challenging to utilize conventional model-
based approaches in a scalable and efficient manner. Meanwhile, AI/ML-based
data-driven approaches can effectively resolve these issues, providing an
appealing alternative for the design of the NextG air interface.

MIMO technology has been a prominent enabler for enhanced system performance
across successive generations of cellular networks and remains a cornerstone
in 5G-Advanced and NextG. Massive MIMO techniques, particularly in the
emerging context of MU-MIMO and distributed MIMO, introduce new grand
challenges for conventional model-based approaches. AI/ML can offer promising
solutions to achieve attractive performance–complexity trade-offs for various
MIMO operations at the air interface: attaining quality of service (QoS)
requirements while maintaining acceptable inference complexity. Recognizing
the need for AI/ML integration into MIMO operations, we pose a foundational
question: What types of AI/ML-enabled MIMO operations are suitable at the
NextG air interface?

##  II Types of AI-enabled MIMO operations

The deployment of AI/ML-enabled methods for MIMO operations at the air
interface must ensure reliable and low latency transception while meeting QoS
requirements across a wide range of system configurations, scenarios, and
possible operation adaptations.

A direct approach to meet these requirements is to employ offline learning.
Specifically, offline learning refers to the process in which the AI/ML model
is trained with a large amount of offline field-collected data and then
deployed for real-time inference. Many existing works have focused on
developing methods using offline learning. For example, deep residual learning
is exploited in [2] to design an offline channel estimation approach. However,
offline-trained AI/ML models often encounter the well-known “uncertainty in
generalization” issue [1], which can be caused by a combination of the
following mismatch between offline training and online deployment: i) _system
configuration_ , ii) _scenario_ , and iii) _operation adaptation_.
Specifically, system configuration refers to antenna configurations, antenna
numbers, and network numerologies. Scenarios include outdoor versus indoor,
urban versus rural, and macro versus micro settings, which can change more
frequently than system configurations. In 5G NR and NextG, operations such as
link/rank adaptation and scheduling are expected to be performed on a slot
basis, which makes operation adaptations occur on a sub-millisecond level [3].
Furthermore, the number of possible operation adaptations within each slot can
be enormously large. For example, for a single-cell network with $18$ subbands
and $10$ active users, the number of scheduling options for single-user MIMO
(SU-MIMO) operation can be as large as $10^{18}$, while the same for MU-MIMO
is even larger. Therefore, it is extremely challenging if not impossible to
ensure the robust generalization ability of offline learning methods across
all possible cases.

Another option involves offline learning followed by online adaptation, which
is a hybrid approach. Specifically, this hybrid approach starts from an
offline AI/ML model that is trained with substantial offline collected data
and then adapted online using the limited online training data. The online
adaptation approach is developed to fine-tune the offline-trained model to
different scenarios, which can mitigate poor generalization caused by the
scenario mismatch. Due to the dynamically changing operation environment
(channel and/or scheduling), it is desirable for the AI/ML model updated
online within the scheduling granularity utilizing over-the-air (OTA)
reference signals (RS). For instance, the scheduling granularity in 5G NR is a
slot. However, the online OTA RS (training data) within the scheduling
granularity is extremely limited due to the system overhead constraint. For
cases with substantial deviation between the offline and online scenarios that
induce different underlying distributions in the corresponding datasets, the
AI/ML model obtained from this hybrid approach may be heavily biased toward
the offline dataset leading to degraded performance during online deployment.
Recent attempts have been made at developing possible hybrid learning
approaches using model-agnostic meta-learning (MAML). The MAML algorithm aims
to learn a general model initialization for fast adaptation to new tasks
utilizing only a few training samples and is independent of the model
architecture. Examples of the MAML algorithm being adopted for hybrid learning
include the downlink beamforming approach at the transmitter side [4] and the
channel prediction method at the receiver side [5]. These meta-learning-based
methods have been effective in mitigating _scenario mismatch_ with a reduced
amount of online training samples compared to the offline dataset. However,
the mismatch caused by the system configuration and the operation adaptation
between offline and online stages, which are major mismatches, will still
impede the smooth deployment of such a transfer or meta-learning-inspired
hybrid approach in practice.

As an alternative, we introduce another hybrid approach that starts with: i)
identifying an AI/ML-based approach that is completely online and real-time,
i.e., relies only on the OTA RS and is updated based on their scheduling
granularity, followed by ii) adding “offline components” to improve its
overall efficiency and resiliency. Such an online real-time AI/ML-based
approach avoids the generalization issue caused by all three aforementioned
mismatches due to the fact that the training and testing data share almost the
same features within the scheduling granularity. Subsequently, as introduced
in our previous work, the additional “offline component” can be trained using
data-driven methods or directly extracted from the available domain knowledge,
e.g., exploiting the pre-trained nearest neighbor binary classifier [6], or
utilizing available channel statistics [7], to further improve the performance
and learning efficiency of the online learning. Although the hybrid learning
approach that starts with a completely offline-trained model and appended with
online adaptation may cater to specific MIMO operations at the air interface,
in general, we posit that the alternate hybrid approach starting with a
completely online and real-time trained model supplemented with offline
components can mitigate generalization challenges much more effectively.
Recognizing the inherent challenge of developing a one-size-fits-all offline
learning-based solution and the need to adopt a hybrid learning paradigm in
general, we envision that designing a purely online and real-time AI/ML
solution catering to specific MIMO operations is a crucial step towards a
robust hybrid learning paradigm for the NextG air interface.

In this article, we focus on online real-time AI/ML-based approaches, which
can serve as potential enablers in fulfilling the vision of AI/ML-enabled MIMO
operations at the NextG air interface. Specifically, this approach entails: i)
_Online learning:_ Training AI/ML models with only the OTA reference signals.
ii) _Real-time learning:_ Enabling AI/ML models to complete the training
procedure within the scheduling granularity. Online learning demands sample-
efficient AI/ML algorithms capable of utilizing extremely limited standard-
compliant RS. Real-time learning further requires AI/ML models to have
efficient training processes to meet stringent latency constraints. For
instance, real-time learning in 5G NR refers to slot-based learning (sub-
millisecond basis).

##  III Online Real-Time Learning under Channel Uncertainty

Figure 1: Downlink CSI acquisition and data transmission in an FDD system.

The development of online real-time learning-based methods at the air
interface of NextG networks has two notable common challenges: i) _limited OTA
training data_ , and ii) _real-time training constraint_. For MIMO operations
where the OTA training data does not directly contain the target (e.g.,
channel estimation), an additional challenge is iii) _lack of ground truth_.
In the context of online real-time learning, the OTA training data in the form
of standard-mandated RS is notably limited. The requirement of model updates
based on the scheduling granularity calls for fast, low-complexity, and
efficient procedures for real-time training. Moreover, in certain MIMO
operations, the absence of explicit ground truth during the online stage
exacerbates the complexity of the problem.

In this section, we narrow our focus to MIMO operations within the downlink
channel state information (CSI) acquisition and subsequent downlink data
transmission processes in a frequency division duplexing (FDD) system. We
explore the necessity, challenges, and existing works related to online real-
time learning-based approaches for these specific operations. In the remainder
of this article, AI/ML-enabled methods are referred to as “learning-based
approaches”.

###  III-A Overview

Knowledge of the wireless channel between the transmitter and the receiver (in
both downlink and uplink) is critical for ensuring overall end-to-end QoS.
Considering FDD systems as an example as depicted in Fig. 1, downlink CSI
acquisition involves a UE estimating the CSI with the CSI RS (CSI-RS). The
estimated CSI is compressed and sent as CSI feedback by the UE to the BS
through the uplink control channel. Upon reception of the compressed CSI
feedback, the BS performs decompression, precoding, and beam management for
efficient downlink data transmission. The transmitted data is recovered
through symbol detection at the UE. However, a time delay in the CSI
acquisition process can lead to outdated feedback, a phenomenon termed
“channel aging”, which can negatively impact the downlink QoS. While the CSI
acquisition process can be simplified in the time-division duplex (TDD) system
by leveraging channel reciprocity to avoid CSI feedback, the imperfect channel
reciprocity due to high user mobility or hardware impairments may also cause
outdated CSI [8, 9]. Therefore, channel prediction has been envisioned as a
crucial step to address the channel aging issue in NextG networks.

###  III-B Online Real-Time Channel Estimation

The accuracy of channel estimation is pivotal for subsequent operations, such
as CSI feedback, transmit precoding, beam management, and multi-user
scheduling. These operations collectively determine whether the link-level and
system-level QoS metrics are satisfactorily met. Conventional channel
estimation approaches, such as the least squares (LS) and the linear minimum
mean square error (LMMSE) methods, either suffer from inferior estimation
performance or require accurate channel statistics. Due to the limitations of
conventional methods, learning-based approaches have started to receive
significant attention. To enhance generalization capability, learning-based
approaches with online and real-time capabilities are preferable, particularly
in highly dynamic channel environments.

Applying online real-time learning to the channel estimation task has certain
key challenges. Firstly, explicit ground truth in terms of perfect channel
knowledge is not available. The OTA training data for channel estimation
refers to the transmitted and received CSI-RS, instead of the actual channel.
Secondly, the OTA training data in the form of CSI-RS is extremely limited due
to the overhead constraint, as mandated by 3GPP standards. Finally, updating
the AI/ML model within the scheduling granularity necessitates extremely low
complexity and efficient training procedures.

Recent efforts have aimed at achieving online channel estimation solely using
the OTA training data, eliminating the need for perfect channel knowledge.
Specifically, existing approaches showcase the possibility of employing
innovative loss functions [10] and utilizing deep reinforcement learning (DRL)
[11] for online channel estimation without knowing the perfect channel.
However, these methods fall short of meeting real-time learning requirements,
necessitating an extensive number of subframes for convergence. To address
this, we introduce StructNet-CE in [12], which is an innovative online real-
time learning approach that leverages the symmetry of modulation
constellations. This significantly enhances neural network (NN) training
efficiency, facilitating online subframe-based channel estimation. Given the
pivotal role of channel estimation, further exploration in this direction is
essential in advancing online real-time learning solutions.

###  III-C Online Real-Time Symbol Detection

Symbol detection is a critical operation within contemporary communication
receivers, aimed at recovering the transmitted information symbol from the
received signal. Conventional symbol detection approaches usually rely on
explicit system modeling and accurate knowledge of CSI, which may not always
be applicable in real-world scenarios. Therefore, learning-based approaches
offer a promising alternative by circumventing these assumptions. The dynamic
nature of channel environments further calls for an online and real-time
symbol detection approach with robust generalization ability.

Designing such a symbol detection method can be challenging due to the
scarcity of OTA training data and the real-time training constraint. Different
from channel estimation, the OTA training data for symbol detection is
available in the form of the demodulation reference signal (DM-RS), which can
serve as ground truth labels. In a multi-user downlink setting with the BS
employing rate-splitting multiple access (RSMA), the model-based learning
approach in [13] provides a promising solution for online user-specific symbol
detection by designing the algorithm based on successive interference
cancellation. However, its performance with 3GPP-compliant OTA training data
and real-time training constraints remains to be seen. Our prior work [6]
achieves online real-time symbol detection by leveraging reservoir computing
(RC) and embedding structural knowledge in its design. RC is a unique family
of recurrent neural networks (RNN) characterized by having only a few
trainable weights, while the majority of the weights are randomly initialized
and remain fixed, facilitating a fast and efficient training procedure with
limited training data. In our most recent work [7], we introduce an approach
to initialize the untrained weights of RC based on domain knowledge to further
improve its performance for online real-time symbol detection. We envision
that RC can play an instrumental role in enabling online real-time learning
solutions, especially for the symbol detection task.

###  III-D Channel Prediction

Channel aging drives the need for channel prediction, especially in scenarios
with moderate to high user mobility in MU-MIMO systems. However, deriving
closed-form solutions for the prediction of non-stationary channel processes
is challenging using traditional signal processing and optimization methods.
Learning-based approaches, on the other hand, can present promising
alternatives for CSI prediction. Online real-time learning methods are
preferable over offline approaches for their better generalization ability in
dynamic wireless channels encountered during practical deployments.

Designing online real-time channel prediction approaches faces challenges
similar to those in the channel estimation task. However, the hurdle of lack
of ground truth becomes more pronounced in channel prediction, when historical
channels are adopted as input to a NN for prediction. Without perfect channel
knowledge, the accuracy of estimated historical channels can significantly
influence prediction performance. For instance, the shallow RNN in [8]
addresses limited training data by using a single hidden layer but faces
performance loss with inaccurately estimated historical channels during online
stages. In [9], DRL is employed for joint channel prediction and beamforming,
which optimizes the transmission sum rate to implicitly address ground truth
channel absence. While the above works enable online learning, their extended
training times make adhering to real-time training constraints impractical in
real-world systems. The design of online real-time channel prediction is still
an active area to explore.

###  III-E CSI Compression

CSI compression plays a crucial role in ensuring robust downlink transmission
and achieving high CSI reconstruction accuracy in practical wireless systems.
In massive MIMO systems, where the BS is equipped with a large number of
antennas, the resulting CSI dimension becomes extensive, leading to
substantial feedback overhead. Therefore, there is a need to explore learning-
based approaches for CSI compression, especially with online real-time
characteristics to address the common generalization issue.

Enabling online real-time learning for CSI compression faces challenges
similar to those encountered in channel estimation and prediction. Besides
these challenges, the inherent difficulty in updating the end-to-end NN from
UE to BS presents an additional obstacle in achieving online real-time CSI
compression. Recent efforts explore adapting offline-trained autoencoders
online. For instance, an unsupervised method is introduced in [14] by adding
an extra encoder after the pre-trained decoder at the BS to perform online
adaptation. While this approach provides a potential solution to address the
issue of lack of perfect channel knowledge, the extended processing time for
multiple training rounds may be impractical in real-world systems,
particularly when CSI feedback occurs every few milliseconds. The design of NN
models for CSI compression that can work in an online real-time learning
fashion requires further exploration.

###  III-F Beam Management

A beam management (BM) framework has been introduced by 3GPP that relies on
two sets of downlink beams labeled ‘Set A’ and ‘Set B’, as defined in 3GPP TR
38.843. This framework utilizes the measurements of reference signal received
power (RSRP) resulting from ‘Set B’ (drawn from a low-resolution codebook) to
predict the best beam from ‘Set A’ (drawn from a high-resolution codebook) for
downlink data transmission. This is a challenging sparse signal recovery
problem, with the only ground truth being the limited RSRP measurements
corresponding to the low-resolution codebook. Although iterative compressive
sensing (CS)-based optimization methods can be utilized, their computational
complexity becomes prohibitive in massive MIMO systems. On the other hand,
learning-based methods can potentially provide computationally efficient
solutions. The vulnerability of BM to user mobility in the mmWave bands due to
narrower beam widths requires the NN model weights at the BS updated upon
receiving the periodic RS (e.g., CSI-RS). Therefore, learning-based approaches
with online and real-time characteristics are necessary to tackle high user
mobility for BM, especially in mmWave systems.

The common challenges of online and real-time learning are also applicable to
BM. Namely, the training data in the form of periodic RSRP reports from the
served UEs is extremely scarce, while explicit ground truth labels (drawn from
the high-resolution codebook) are not available. Additionally, the real-time
constraint of executing the beamforming decision is highly stringent.
Therefore, for robust adaptation to user mobility, the online learning
approach must be able to use extremely limited OTA training data and have a
short update time to facilitate standards-compliant implementations.

As an example, a DRL-based beam training framework that mitigates the
challenge of lack of explicit ground truth data and misaligned scenarios is
introduced in [15]. Although such DRL-based approaches can meet online
learning requirements, they are typically not amenable to real-time training
due to long convergence times, potentially limiting their applicability in
highly non-stationary channel scenarios caused by rapid user mobility. Thus,
developing online learning-aided methods for BM with low computational
complexity remains an active area of investigation.

##  IV Design Insights for Online Real-Time Learning

###  IV-A Design Insights and Guidelines

Figure 2: Exploiting domain knowledge to aid online learning.

In light of the preceding challenges, we foresee three possible approaches to
achieve online real-time learning: i) Reduce dimensionality of search space: A
possible approach is to reduce the dimensionality of the search space based on
domain knowledge, ensuring the target is in the search space. By reducing the
dimensionality of the search space, a smaller NN model with fewer trainable
parameters can be employed to facilitate online real-time learning in a
sample-efficient manner. This is illustrated in Fig. 2, where exploiting
domain knowledge permits confining the search space to a lower dimensional
function space $\mathcal{P}$ instead of the original function space
$\mathcal{Q}$ of much larger dimensionality. As an example, the symmetric
structure of modulation constellations can be exploited to transform the
multi-class classification problem of symbol detection into tasks that can be
solved by a single binary classifier, as shown in our previous work [6]. ii)
Improve initialization: A supplementary strategy is to expedite online
learning by initializing NN weights based on domain knowledge. This ensures
that the initial starting point for model training in the reduced search space
is drawn from a neighborhood that is much closer to the target than an
arbitrary random model initialization in $\mathcal{P}$, thus enabling faster
convergence to the optimum target. This is conceptually depicted in Fig. 2,
where domain knowledge is exploited further to initialize training within the
neighborhood $\mathcal{P}_{\mathrm{opt}}$ around the target model $f^{*}$. For
example, for the task of online real-time symbol detection, a novel procedure
was introduced in our most recent work [7] to initialize the untrained weights
of an RC-based symbol detector based on domain knowledge, specifically
knowledge of the channel statistics, thereby resulting in a vast improvement
in performance over an RC-based detector with randomly initialized untrained
weights, i.e., without domain knowledge utilization. iii) Select proper
learning objectives: Under circumstances where the ground truth label is not
available, the learning objectives can be designed based on domain knowledge
to enable the NN to estimate the target without knowing the ground truth.
Specifically, instead of designing the objective function to explicitly
estimate the target, the learning objective can be designed to aim at another
task that is related to the target based on domain knowledge, making the
target estimation an implicit optimization goal. For instance, as demonstrated
in our prior work [12], the channel can be implicitly estimated by optimizing
over the symbol detection task and incorporating the symmetry of modulation
constellations into the NN architecture. As another example, the channel
prediction problem can be solved by maximizing the received downlink sum rate
through DRL [9], instead of explicitly minimizing the prediction error.

###  IV-B Case Study: Online Real-Time MIMO Channel Estimation

Figure 3: Structure of StructNet-CE.

Given the aforementioned insights, we present an online real-time MIMO-OFDM
channel estimation approach named StructNet-CE [12]. StructNet-CE leverages
the symmetric structure of the QAM constellation to reduce the dimensionality
of the search space, which equips it with a fast and efficient learning
procedure. With the reduced search space dimensionality and improved sample
efficiency, StructNet-CE can learn with only the OTA training data and conduct
channel estimation on a subframe basis, achieving the online and real-time
characteristics. Furthermore, StructNet-CE is optimized over the symbol
detection task while achieving channel estimation as an implicit optimization
goal, addressing the challenge of lacking perfect channel knowledge during the
online stage. The training data utilized for learning StructNet-CE is the
transmitted and received OTA RS.

TABLE I: Experimental settings

Item | Offline | Online
---|---|---
Receive antenna number | 2 | 2
Transmit antenna number | 1 | 2
Channel scenario | Rural macro-cell non-line-of-sight (NLOS) | Urban macro-cell NLOS
User speed | 50 km/h | 5 km/h
Subcarrier number | 1024
OFDM symbols per subframe | 14
Data symbols per subframe | 10
Pilot symbols per subframe | 4

As illustrated in Fig. 3, StructNet-CE consists of a channel layer for channel
estimation, an interference invariant layer to capture inter-stream
interference in single-user MIMO, and a binary classifier for symbol
detection. This architecture leverages the symmetry of the modulation
constellation, allowing the transformation of a multi-class classification
problem into a series of binary detection tasks. The channel layer and the
binary classifier are designed to facilitate this conversion. Moreover, the
detection of symbols in the target data stream should remain unaffected by the
symbols transmitted in interference data streams. The interference invariant
layer is designed to capture this property. More details about the design are
provided in [12]. The input to StructNet-CE is the transmitted pilot along
with the received signal, and the output is the detected symbol. By embedding
the constellation symmetry, StructNet-CE forces the weights of the channel
layer to be aligned with the MIMO channel coefficients through
backpropagation. This allows for the direct acquisition of the estimated
channel coefficients from the channel layer weights without perfect channel
knowledge.

To demonstrate the advantages of online real-time learning, we conduct an
experiment where a large NN model, specifically ReEsNet [2], is considered for
channel estimation, with both channel scenarios and system configurations used
for offline training and online testing being misaligned. Subsequently, its
performance is compared with StructNet-CE, which is trained completely online.
Note that to the best of our knowledge, online adaptation of the offline-
trained ReEsNet without the ground truth channel being available is still a
challenging problem, and therefore is not considered here. The simulation
settings during the offline and online stages are summarized in Tab. I. More
detailed settings are provided in [12]. The channel realizations are generated
by QuaDRiGa using the 3GPP-3D channel model specified in 3GPP TR 36.873. The
adopted pilot pattern complies with the 3GPP standard, which follows Tab.
7.4.1.1.2-4 in 3GPP TS 38.211. Orthogonal pilots are employed for conventional
methods, while non-orthogonal pilots are exploited for learning-based
approaches, following the discussion of pilot choice in [6]. Note that when
adopting ReEsNet trained offline under the $1\times 2$ SIMO-OFDM system to the
$2\times 2$ MIMO-OFDM online scenario, the $2\times 2$ MIMO-OFDM channel is
estimated via two $1\times 2$ SIMO-OFDM systems without considering the inter-
stream interference.

Fig. 4 compares the normalized mean square error (NMSE) of different
approaches under various signal-to-noise ratios (SNRs). The SNR is defined as
the ratio of the received signal power to the noise power. Compared with
ReEsNet without such misalignment (labeled “ReEsNet”), ReEsNet with mismatched
offline-online channel scenarios and system configurations, denoted as
“ReEsNet (mismatch)”, exhibits performance degradation due to the “uncertainty
of generalization” issue. While ReEsNet without any misalignment outperforms
StructNet-CE, such performance is an unachievable error lower bound for
ReEsNet. This is because it is extremely difficult in practice to ensure the
online deployment scenario matches exactly with the scenario encountered
during offline training, due to the dynamic channel environment.

In contrast, the purely online and real-time trained StructNet-CE outperforms
ReEsNet with mismatches as well as conventional approaches, such as LS and
empirical LMMSE (labeled “em-LMMSE”), while utilizing only the OTA pilots for
training and adhering to real-time constraints. More importantly, in the high
SNR regime, StructNet-CE can reach the practically unachievable error lower
bound of ReEsNet, demonstrating the effectiveness of StructNet-CE in online
real-time learning. In [12], the CPU run time is furnished to demonstrate the
real-time properties of StructNet-CE.

In summary, StructNet-CE overcomes the bottleneck of poor generalization seen
with purely offline training by effectively incorporating domain knowledge
into its design. With purely online and real-time learning, StructNet-CE
outperforms conventional approaches and achieves the error lower bound that
cannot be obtained by ReEsNet in the high SNR regime.

##  V Conclusion and Outlook

Figure 4: NMSE of channel estimation methods.

The futuristic use cases envisioned in NextG will bring about the need for
sophisticated MIMO techniques, where conventional model-based solutions may
fall short. AI/ML techniques can come to the rescue to realize such
applications. In this article, a pivotal role of AI/ML-based solutions is
envisioned, particularly of online real-time learning methods to implement
critical MIMO operations at the air interface. The challenges involved in
designing online real-time AI/ML-based methods for these tasks are outlined,
and supported with crucial insights that can serve as guiding principles in
constructing such approaches. Building on these insights, an online real-time
AI/ML-based solution is introduced for channel estimation. Looking ahead, we
believe that effective utilization of domain knowledge in the design of the
structure of AI/ML models, which remains an exploratory endeavor at present,
will be central in making both online real-time learning and general hybrid
learning approaches a reality at the NextG air interface.

## References

  * [1] R. Shafin, L. Liu, V. Chandrasekhar, H. Chen, J. Reed, and J. C. Zhang, “Artificial Intelligence-Enabled Cellular Networks: A Critical Path to Beyond-5G and 6G,” _IEEE Wireless Commun._ , vol. 27, no. 2, pp. 212–217, 2020.
  * [2] L. Li, H. Chen, H.-H. Chang, and L. Liu, “Deep Residual Learning Meets OFDM Channel Estimation,” _IEEE Wireless Commun. Letters_ , vol. 9, no. 5, pp. 615–618, 2020.
  * [3] _3rd Generation Partnership Project; Technical Specification Group Radio Access Network; Study on New Radio (NR) access technology (Release 17)_ , 3GPP Std. TR 38.912, Rev. 17.0.0, 2022.
  * [4] Y. Yuan, G. Zheng, K.-K. Wong, B. Ottersten, and Z.-Q. Luo, “Transfer Learning and Meta Learning-Based Fast Downlink Beamforming Adaptation,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 3, pp. 1742–1755, 2021.
  * [5] Y. Yang, F. Gao, Z. Zhong, B. Ai, and A. Alkhateeb, “Deep transfer learning-based downlink channel prediction for FDD massive MIMO systems,” _IEEE Trans. Commun._ , vol. 68, no. 12, pp. 7485–7497, 2020\.
  * [6] J. Xu, L. Li, L. Zheng, and L. Liu, “Detect to Learn: Structure Learning with Attention and Decision Feedback for MIMO-OFDM Receive Processing,” _IEEE Trans. Wireless Commun._ , pp. 1–1, 2023.
  * [7] S. Jere, K. Said, L. Zheng, and L. Liu, “Towards Explainable Machine Learning: The Effectiveness of Reservoir Computing in Wireless Receive Processing,” in _2023 IEEE Military Commun. Conf. (MILCOM)_ , 2023, pp. 667–672.
  * [8] Y. Zhu, X. Dong, and T. Lu, “An Adaptive and Parameter-Free Recurrent Neural Structure for Wireless Channel Prediction,” _IEEE Trans. Commun._ , vol. 67, no. 11, pp. 8086–8096, 2019.
  * [9] M. Chu, A. Liu, V. K. N. Lau, C. Jiang, and T. Yang, “Deep Reinforcement Learning Based End-to-End Multiuser Channel Prediction and Beamforming,” _IEEE Trans. Wireless Commun._ , vol. 21, no. 12, pp. 10 271–10 285, 2022\.
  * [10] X. Zheng and V. K. N. Lau, “Online Deep Neural Networks for MmWave Massive MIMO Channel Estimation With Arbitrary Array Geometry,” _IEEE Trans. Signal Process._ , vol. 69, pp. 2010–2025, 2021.
  * [11] M. S. Oh, S. Hosseinalipour, T. Kim, C. G. Brinton, and D. J. Love, “Channel Estimation via Successive Denoising in MIMO OFDM Systems: A Reinforcement Learning Approach,” in _ICC 2021 - IEEE Int. Conf. on Commun._ , 2021, pp. 1–6.
  * [12] L. Li, S. S. Rayala, J. Xu, L. Zheng, and L. Liu, “Learning to estimate: A real-time online learning framework for MIMO-OFDM channel estimation,” _arXiv preprint arXiv:2305.13487_ , 2023.
  * [13] R. Cerna Loli, O. Dizdar, B. Clerckx, and C. Ling, “Model-Based Deep Learning Receiver Design for Rate-Splitting Multiple Access,” _IEEE Trans. Wireless Commun._ , vol. 22, no. 11, pp. 8352–8365, 2023.
  * [14] Y. Cui, J. Guo, C.-K. Wen, S. Jin, and S. Han, “Unsupervised Online Learning in Deep Learning-Based Massive MIMO CSI Feedback,” _IEEE Commun. Lett._ , vol. 26, no. 9, pp. 2086–2090, 2022.
  * [15] J. Zhang, Y. Huang, J. Wang, X. You, and C. Masouros, “Intelligent interactive beam training for millimeter wave communications,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 3, pp. 2034–2048, 2021.

Jiarui Xu is a Ph.D. student at Virginia Tech, USA.
---
Shashank Jere is a Ph.D. student at Virginia Tech, USA.
---
Yifei Song is a Ph.D. student at Virginia Tech, USA.
---
Yi-Hung Kao is a Ph.D. student at Virginia Tech, USA.
---
Lizhong Zheng is a Professor in Electrical Engineering and Computer Science at
Massachusetts Institute of Technology, USA.
---
Lingjia Liu is a Professor & Bradley Senior Faculty Fellow in Electrical and
Computer Engineering at Virginia Tech, USA.
---


# A proof of Sanov’s Theorem via discretizations

Rangel Baldasso  mail<EMAIL_ADDRESS>Mathematical Institute,
Leiden University, P.O. Box 9512, 2300 RA Leiden, The Netherlands. Roberto I.
Oliveira<EMAIL_ADDRESS>IMPA, Estrada Dona Castorina 110, 22460-320 Rio de
Janeiro, RJ - Brazil. Alan Pereira<EMAIL_ADDRESS>Instituto de
Matemática, Universidade Federal de Alagoas, Rua Lorival de Melo Mota s/n,
57072970 Maceió, AL - Brazil. Guilherme Reis<EMAIL_ADDRESS>Fakultät
für Mathematik, Technische Universität München, Boltzmannstraße 3, 85748
Garching bei München, Germany.

###### Abstract

We present an alternative proof of Sanov’s theorem for Polish spaces in the
weak topology that follows via discretization arguments. We combine the
simpler version of Sanov’s Theorem for discrete finite spaces and well chosen
finite discretizations of the Polish space. The main tool in our proof is an
explicit control on the rate of convergence for the approximated measures.

_Keywords and phrases:_ large deviations; Sanov’s Theorem

MSC 2010: 60F10

##  1 Introduction

Sanov’s Theorem is a well know result in the theory of large deviations
principles. It provides the large deviations profile of the empirical measure
of a sequence of i.i.d. random variables and characterizes its rate function
as the relative entropy. This short note provides an alternative proof of this
fact, by exploring the metric structure of the weak topology with the
variational formulation of the relative entropy.

Formally, let $(M,\operatorname{d})$ be a Polish space and let
$\big{(}X_{n}\big{)}_{n\in\mathbb{N}}$ be a sequence of independent $M$-valued
random elements identically distributed according to $\mu\in\mathcal{P}(M)$,
where $\mathcal{P}(M)$ is the set of Borel probability measures on $M$. We
denote by $\delta_{x}$ the probability measure degenerate at $x\in M$, and
define the empirical measure of $X_{1},\dots,X_{n}$ by

$L_{n}:=\frac{1}{n}\sum_{i=1}^{n}\delta_{X_{i}}.$ (1.1)

Also, given $\upsilon,\mu\in\mathcal{P}(M)$, the relative entropy between
$\upsilon$ and $\mu$ is defined as

$H(\upsilon|\mu):=\sup\left\\{\int f\textnormal{d}\upsilon-\log\int
e^{f}\textnormal{d}\mu;\,f\text{ is measurable and bounded}\right\\}.$ (1.2)

Sanov’s Theorem is given by the following statement.

######  Theorem 1.1 (Sanov).

Let $\big{(}X_{n}\big{)}_{n\in\mathbb{N}}$ be a sequence of i.i.d. random
variables taking values in a Polish space $(M,\operatorname{d})$ with
distribution $\mu\in\mathcal{P}(M)$. The sequence of empirical measures
$\big{(}L_{n}\big{)}_{n\in\mathbb{N}}$ of
$\big{(}X_{n}\big{)}_{n\in\mathbb{N}}$ (defined in Equation (1.1)) satisfies a
large deviations principle on the space $\mathcal{P}(M)$ with rate function
$H(\,\cdot\,|\mu)$.

When the space $M$ is finite, the theorem above is proved in an elementary and
elegant way (see Den Hollander [3, Theorem II.2], and Dembo and Zeitouni [2,
Theorem 2.1.10]). In this work, we prove the theorem for general Polish metric
spaces by extending this elementary proof via sequences of discretizations of
the space. We split the set $M$ in a finite number of subsets which belong to
one of two distinct categories. The well-behaved sets are the ones with small
diameter, while the badly behaved sets will have small $\mu$-measure. We
remark though that when the space $M$ is compact, no badly behaved sets are
necessary. These partitions define natural projections on the space and allow
us to approximate the sequence $\big{(}X_{n}\big{)}_{n\in\mathbb{N}}$ by
variables in the discretized spaces, and by consequence provide approximations
for its empirical measures. The main technical observation is that the
discretized relative entropy converges to the relative entropy (1.2) as we
take thinner partitions (see Lemma 4.1) and the relative entropy is well
approximated in balls (see Lemma 4.3).

Some ideas used to prove Lemma 4.3 are roughly inspired by the proof of the
upper bound in Csiszár [1]. His work presents a proof of Sanov’s Theorem for
the $\uptau$-topology, a stronger topology than that of weak convergence, with
an approach that differs greatly from more classical ones that can be found,
for example, in [2, Theorem 6.2.10].

There are two proofs of Sanov’s Theorem in [2], one by means of Cramér’s
Theorem for Polish spaces and the other following a projective limit approach.
Although we strongly use the metric structure of the space, our proof does not
require profound knowledge of large deviations theory or general topology.

Organization of the paper. In the next section we collect some preliminary
notation and results that are used during the text. Section 3 introduces the
discretization considered here. Section 4 contains the statement of the main
lemmas used in the proof. We also show how Sanov’s Theorem is proved in
Section 4. Sections 5 and 6 contain the proofs of Lemmas 4.1 and 4.3,
respectively.

Acknowledgments. RB is supported by the Mathematical Institute of Leiden
University for support. RIO counted on the support of CNPq, Brazil via a Bolsa
de Produtividade em Pesquisa (304475/2019-0) and a Universal grant
(432310/2018-5). GR was partially supported by a Capes/PNPD fellowship
888887.313738/2019-00 while he was a post doc at Federal University of Bahia
(UFBA).

Data Availability Statement . Data sharing is not applicable to this article
as no datasets were generated or analysed during the current study.

##  2 Preliminaries

In this section we review some basic concepts. We provide the definition of
large deviations principle and weak topology, and collect some properties of
the relative entropy.

######  Definition 2.1 (Large deviation principle).

A sequence $\big{(}\mathbb{P}_{n}\big{)}_{n\in\mathbb{N}}$ of probabilities
over a metric space $(\mathfrak{X},\textnormal{d}_{\mathfrak{X}})$ satisfies a
large deviation principle with rate function $I$ if

  1. 1.

(Lower bound) For any open set $\mathcal{O}\subset\mathfrak{X}$,

$\liminf_{n\to\infty}\frac{1}{n}\log\mathbb{P}_{n}(\mathcal{O})\geq-\inf_{x\in\mathcal{O}}I(x);$
(2.1)

  2. 2.

(Upper bound) For any closed set $\mathcal{C}\subset\mathfrak{X}$,

$\limsup_{n\to\infty}\frac{1}{n}\log\mathbb{P}_{n}(\mathcal{C})\leq-\inf_{x\in\mathcal{C}}I(x).$
(2.2)

The weak topology on $\mathcal{P}(M)$ is defined as the topology generated by
the functionals

$\upsilon\mapsto\int\varphi\,\textnormal{d}\upsilon,$ (2.3)

where $\varphi\in C_{b}(M)$ is a continuous bounded function.

A metric compatible with the weak topology is the bounded Lipschitz metric

$\textnormal{d}_{\rm BL}(\mu,\upsilon)=\sup\left\\{\left|\int
f\,\textnormal{d}\upsilon-\int f\,\textnormal{d}\mu\right|:f\in
BL(M)\right\\},$ (2.4)

where $BL(M)$ is the class of $1$-Lipschitz functions $f:M\to\mathbb{R}$
bounded by one.

For the next lemma, let $x\wedge y$ denote the minimum between $x$ and $y$.

######  Lemma 2.2.

Let $(X,Y)$ be a coupling of two distributions $\mu$ and $\upsilon$. Then

$\textnormal{d}_{\rm BL}(\mu,\upsilon)\leq\mathbb{E}(\textnormal{d}(X,Y)\wedge
2).$ (2.5)

###### Proof.

Let $x,y\in M$ and notice that, for each $f\in BL(M)$,

$|f(x)-f(y)|\leq\textnormal{d}(x,y)\wedge 2,$ (2.6)

since $f$ is 1-Lipschitz and bounded by one. The proof is now complete by
noting that

$\left|\int f\,\textnormal{d}\upsilon-\int
f\,\textnormal{d}\mu\right|=\big{|}\mathbb{E}\big{(}f(X)-f(Y)\big{)}\big{|}\leq\mathbb{E}(\textnormal{d}(X,Y)\wedge
2).$ (2.7)

∎

Equation (1.2) is called the variational formulation of entropy, and it
readily implies the so-called entropy inequality

$\int f\textnormal{d}\upsilon\leq H(\upsilon|\mu)+\log\int
e^{f}\textnormal{d}\mu,$ (2.8)

for any measurable bounded function $f$. We will also make use of the integral
formulation of the relative entropy, provided in the next lemma. This
formulation will be the key result used to approximate relative entropies in
the discrete case to the general case.

######  Lemma 2.3.

The variational formula of the relative entropy (1.2) is equivalent to the
following integral formulation of the entropy

$H(\upsilon|\mu)=\begin{cases}\int\frac{\textnormal{d}\upsilon}{\textnormal{d}\mu}\log\frac{\textnormal{d}\upsilon}{\textnormal{d}\mu}\textnormal{d}\mu,&\text{
if }\upsilon\ll\mu,\\\ +\infty,&\text{ otherwise}.\end{cases}$ (2.9)

We refrain from presenting the proof of the lemma above, and refer the reader
to [4, Theorem 5.2.1].

##  3 Discretization

In this section we present the discretization procedure used for the space $M$
and related constructions for measures and random variables.

We start by discretizing the space. Let $\mu\in\mathcal{P}(M)$ and recall
that, since $M$ is a Polish space, there exists, for each $m\in\mathbb{N}$, a
compact set $K_{m}$ with

$\mu(K_{m}^{\complement})\leq\frac{e^{-m^{2}-1}}{m}.$ (3.1)

The support of the measure $\mu$ is contained in the closure of the union of
the compacts $K_{m}$. Notice that the collection of probability measures
supported on the closure of $\cup_{m=1}^{\infty}K_{m}$ forms a closed subset
of $\mathcal{P}(M)$, and thus it is enough to prove a large deviations
principle for this subspace (see [2, Lemma 4.1.5]). We assume from now on that

$M=\overline{\bigcup_{m=1}^{\infty}K_{m}}.$ (3.2)

Given a sequence of partitions
$\big{(}\mathcal{A}_{m}\big{)}_{m\in\mathbb{N}}$, let $\mathcal{F}_{m}$ and
$\mathcal{F}_{\infty}$ denote the $\sigma$-algebras generated by
$\mathcal{A}_{m}$ and by the union $\cup_{m=1}^{\infty}\mathcal{A}_{m}$,
respectively. We write $\mathcal{B}(M)$ for the Borel $\sigma$-algebra in $M$.

######  Lemma 3.1.

There exists a sequence of nested partitions
$\big{(}\mathcal{A}_{m}\big{)}_{m\in\mathbb{N}}$ such that
$\mathcal{A}_{m}=\\{A_{m,1},\dots,A_{m,\ell_{m}}\\}$ and

  * •

$\operatorname{diam}(A_{m,i})<\frac{1}{m}$, if $i=1,\dots,\tilde{\ell}_{m}$,
for some $\tilde{\ell}_{m}\leq\ell_{m}$.

  * •

$K_{m}^{\complement}=\bigcup_{i=\tilde{\ell}_{m}+1}^{\ell_{m}}A_{m,i}$.

  * •

$\mathcal{F}_{\infty}=\mathcal{B}(M)$.

###### Proof.

Notice that if we can construct partitions $\mathcal{A}_{m}$ for each $m$ that
satisfy the three first requirements of the lemma without requiring them to be
nested, it is possible to take refinements in order to obtain a nested
sequence.

Recall the definition the compact set $K_{m}$ in (3.1). By compactness, it is
possible to partition $K_{m}$ into subsets
$\\{C_{m,1},\dots,C_{m,\bar{\ell}_{m}}\\}$ of diameter at most $\frac{1}{m}$,
so that
$\mathcal{C}_{m}=\\{K_{m}^{\complement},C_{m,1},\dots,C_{m,\bar{\ell}_{m}}\\}$
defines a partition of $M$.

Consider an enumeration $\big{(}B^{i}\big{)}_{i\in\mathbb{N}}$ of balls of
rational radius and centered in a countable dense subset of $M$. We now define
the partition $\mathcal{A}_{m}$ via the intersections of sets in
$\mathcal{C}_{m}$ with $B^{m}$ and its complement. We write

$\mathcal{A}_{m}=\\{A_{m,1},\dots,A_{m,\ell_{m}}\\},$ (3.3)

where $A_{m,i}$, $i\leq\tilde{\ell}_{m}$, denotes the sets contained in
$K_{m}$ and $A_{m,i}$, $\tilde{\ell}_{m}+1\leq i\leq\ell_{m}$, indicates the
sets contained in $K_{m}^{\complement}$.

Notice that the first two statements about the partition $\mathcal{A}_{m}$ are
immediately verified. To check the last claim, notice that
$B^{i}\in\mathcal{F}_{i}$, and thus $B^{i}\in\mathcal{F}_{\infty}$, for all
$i\in\mathbb{N}$, which implies $\mathcal{F}_{\infty}=\mathcal{B}(M)$ and
concludes the proof. ∎

We select a subset $M_{m}:=\\{a_{m,1},\dots,a_{m,\ell_{m}}\\}\subset M$ such
that $a_{m,i}\in A_{m,i}$ for $i=1,\dots,\ell_{m}$ and turn
$(\mathcal{A}_{m},M_{m})$ into a tagged partition. We will furthermore assume
that $M_{m}\subset M_{m+1}$.

For each $m\in\mathbb{N}$, the tagged partition $(\mathcal{A}_{m},M_{m})$
defines a natural projection $\pi^{m}:M\to M_{m}$ via

$\pi^{m}(x)=a_{m,i},\text{ if }x\in A_{m,i}.$ (3.4)

This allows us to define, for any measure $\upsilon\in\mathcal{P}(M)$, its
discretized version $\upsilon^{m}\in\mathcal{P}(M)$ as the probability measure
supported in $M_{m}$ given by the pushforward of $\upsilon$ via the map
$\pi^{m}$, i.e.

$\upsilon^{m}(a_{m,i})=\upsilon\big{(}(\pi^{m})^{-1}(a_{m,i})\big{)}=\upsilon(A_{m,i}),\text{
for }i=1,\dots,\ell_{m}.$ (3.5)

Random elements are also discretized with the aid of the projection maps
$\pi^{m}$. If $\big{(}X_{i}\big{)}_{i\in\mathbb{N}}$ is an i.i.d. sequence of
random elements with distribution $\mu\in\mathcal{P}(M)$, then
$X_{i}^{m}=\pi^{m}(X_{i})$ yields an i.i.d. sequence of random elements
distributed according to $\mu^{m}$.

The empirical measure for the discretized elements is given by

$L_{n}^{m}:=\frac{1}{n}\sum_{i=1}^{n}\delta_{X_{i}^{m}}.$ (3.6)

Since, for each $m\in\mathbb{N}$, the elements $X_{i}^{m}$ take values on the
finite space $M_{m}$, we know that the sequence of empirical measures
$\big{(}L_{n}^{m}\big{)}_{n\in\mathbb{N}}$ satisfies a large deviations
principle on the space $\mathcal{P}(M_{m})$ with rate function
$H(\,\cdot\,|\mu^{m})$. Via [2, Lemma 4.1.5], we can extend these large
deviation principles to the whole space $\mathcal{P}(M)$ with rate function
also given by $H(\,\cdot\,|\mu^{m})$ (note that $H(\upsilon|\mu^{m})$ is
infinite if $\upsilon\notin\mathcal{P}(M_{m})$).

Lemma 2.3 yields the following expression for the rate function
$H(\upsilon|\mu^{m})$, when $\upsilon\in\mathcal{P}(M_{m})$:

$H(\upsilon|\mu^{m})=\sum_{a\in
M_{m}}\upsilon(a)\log\frac{\upsilon(a)}{\mu^{m}(a)}.$ (3.7)

##  4 Proof of Theorem 1.1

In this section we present our approach to the proof of Sanov’s Theorem. Our
goal is to deduce that the empirical measures $L_{n}$ given by (1.1) satisfy a
large deviations principle from the information that the sequences
$\big{(}L_{n}^{m}\big{)}_{n\in\mathbb{N}}$ satisfy large deviations
principles, for all $m\in\mathbb{N}$. Since the rate function given by Sanov’s
Theorem (Theorem 1.1) is the relative entropy, the following two lemmas that
relate the entropies in discrete and Polish spaces are the central pieces in
our proof.

######  Lemma 4.1.

For any two probability measures $\mu,\upsilon\in\mathcal{P}(M)$,

$\sup_{m}H(\upsilon^{m}|\mu^{m})=\lim_{m}H(\upsilon^{m}|\mu^{m})=H(\upsilon|\mu).$
(4.1)

Furthermore, if $\sup_{m}H(\upsilon^{m}|\mu^{m})<\infty$ then there exists a
positive constant $c\geq 0$ such that

$\textnormal{d}_{\rm BL}(\upsilon^{m},\upsilon)\leq\frac{c}{m}.$ (4.2)

######  Remark 4.2.

Notice that the lemma above provides an alternative expression for the rate
function in Sanov’s Theorem, depending on the sequence of partitions chosen.
Even more is true: the supremum in (4.1) can be taken over all finite
partitions of $M$. In fact, simply notice that, if $(\mathcal{A},\mathcal{M})$
is a dotted partition of $M$ with projection map $\pi:M\to\mathcal{M}$, then
any function $f:\mathcal{M}\to\mathbb{R}$ can be extended to $M$ via
$\tilde{f}=f\circ\pi$. To conclude, apply the variational formulation of
relative entropy to obtain $H(\upsilon^{\mathcal{A}}|\mu^{\mathcal{A}})\leq
H(\upsilon|\mu)$.

######  Lemma 4.3.

For any $\mu,\upsilon\in\mathcal{P}(M)$ and $m_{0}\in\mathbb{N}$,

$\sup_{m\geq
m_{0}}\inf_{\sigma\in\bar{B}_{\frac{1}{\sqrt{m}}}(\upsilon)}H(\sigma|\mu^{m})=H(\upsilon|\mu).$
(4.3)

We prove Lemma 4.1 in Section 5 and Lemma 4.3 in Section 6.

The next lemma is a result of exponential equivalence.

######  Lemma 4.4.

Let $L_{n}$ and $L_{n}^{m}$ as defined in Equations (1.1) and (3.6),
respectively. Then

$\mathbb{P}\bigg{(}\textnormal{d}_{\rm
BL}(L_{n},L_{n}^{m})>\frac{3}{m}\bigg{)}\leq\exp\big{(}-mn\big{)}.$ (4.4)

###### Proof.

Observe that if $X_{i}\in A_{m,j}$ for some $j=1,\dots,\tilde{\ell}_{m}$ then
$d(X_{i},X_{i}^{m})\leq 1/m$. In particular,

$\textnormal{d}_{BL}(L_{n},L_{n}^{m})\leq\frac{1}{n}\sum_{i=1}^{n}\textnormal{d}(X_{i},X_{i}^{m})\wedge
2\leq\frac{1}{m}+\frac{2}{n}\sum_{i=1}^{n}1_{\\{X_{i}\in
K_{m}^{\complement}\\}}.$ (4.5)

This implies

$\mathbb{P}\bigg{(}\textnormal{d}_{BL}(L_{n},L_{n}^{m})>\frac{3}{m}\bigg{)}\leq\mathbb{P}\bigg{(}\frac{1}{n}\sum_{i=1}^{n}1_{\\{X_{i}\in
K_{m}^{\complement}\\}}>\frac{1}{m}\bigg{)}$ (4.6)

In order to bound the last probability, we use union bound and independence to
obtain

$\begin{split}\mathbb{P}\bigg{(}\frac{1}{n}\sum_{i=1}^{n}1_{\\{X_{i}\in
K_{m}^{\complement}\\}}>\frac{1}{m}\bigg{)}&\leq\sum_{A\subset[n]:|A|=\frac{n}{m}}\mathbb{P}\big{(}X_{i}\in
K_{m}^{\complement},\text{ for all }i\in A\big{)}\\\
&\leq\binom{n}{\frac{n}{m}}\Big{(}\frac{e^{-m^{2}-1}}{m}\Big{)}^{\frac{n}{m}}\\\
&\leq\bigg{(}em\frac{e^{-m^{2}-1}}{m}\bigg{)}^{\frac{n}{m}}\\\
&\leq\exp\big{(}-mn\big{)},\end{split}$ (4.7)

concluding the proof. ∎

We are now ready to work on the proof of Theorem 1.1. It is proved in [2,
Lemma 6.2.6] that the sequence $\big{(}L_{n}\big{)}_{n\in\mathbb{N}}$ is
exponentially tight. In particular, there exists a subsequence
$\big{(}L_{n_{k}}\big{)}_{n_{k}}$ that satisfies a large deviations principle
with rate function $I$.

From now on, we drop the subscript $k$ in $n_{k}$. Notice that

$-I(\upsilon)=\lim_{\varepsilon\to
0}\lim_{n\to\infty}\frac{1}{n}\log\mathbb{P}(L_{n}\in
B_{\varepsilon}(\upsilon)).$ (4.8)

Even though the rate function $I$ might depend on the subsequence, our goal is
to prove that this is not the case. In fact, we prove that
$I(\,\cdot\,)=H(\,\cdot\,|\mu)$. In Proposition 4.5, we prove that
$H(\,\cdot\,|\mu)\geq I(\,\cdot\,)$, while the opposite inequality is
established in Proposition 4.6. This concludes the proof of Theorem 1.1, since
any possible subsequence $\big{(}L_{n_{k}}\big{)}_{n_{k}}$ that satisfies a
large deviations principle does so with the same rate function
$H(\,\cdot\,|\mu)$, which implies that the whole sequence also satisfies a
large deviations principle.

######  Proposition 4.5.

The function $I$ in Equation (4.8) satisfies $H(\,\cdot\,|\mu)\geq
I(\,\cdot\,)$.

###### Proof.

Fix $\upsilon\in\mathcal{P}(M)$ and notice we can assume that
$H(\upsilon|\mu)$ is finite since the statement is trivially verified if
otherwise.

Due to Lemma 4.1, we have

$\textnormal{d}_{\rm BL}(\upsilon,\upsilon^{m})\leq\frac{c}{m},$ (4.9)

for some positive constant $c>0$. In particular, this implies

$\mathbb{P}(L^{m}_{n}\in
B_{\varepsilon}(\upsilon^{m}))\leq\mathbb{P}\Big{(}L_{n}\in\bar{B}_{\varepsilon+\frac{3+c}{m}}(\upsilon)\Big{)}+\mathbb{P}\Big{(}\textnormal{d}_{\rm
BL}(L_{n},L_{n}^{m})>\frac{3}{m}\Big{)},$ (4.10)

which yields

$\begin{split}\frac{1}{n}&\log\mathbb{P}(L^{m}_{n}\in
B_{\varepsilon}(\upsilon^{m}))\leq\frac{\log 2}{n}\\\
&\qquad\qquad+\max\left\\{\frac{1}{n}\log\mathbb{P}\Big{(}L_{n}\in\bar{B}_{\varepsilon+\frac{3+c}{m}}(\upsilon)\Big{)},\frac{1}{n}\log\mathbb{P}\Big{(}\textnormal{d}_{\rm
BL}(L_{n},L_{n}^{m})>\frac{3}{m}\Big{)}\right\\}.\end{split}$ (4.11)

Lemma 4.4 gives

$\frac{1}{n}\log\mathbb{P}\Big{(}\textnormal{d}_{\rm
BL}(L_{n},L_{n}^{m})>\frac{3}{m}\Big{)}\leq-m,$ (4.12)

and, by taking the limit as $n$ grows in (4.11),

$-\inf_{\sigma\in
B_{\varepsilon}(\upsilon^{m})}H(\sigma|\mu^{m})\leq\max\bigg{\\{}-\inf_{\sigma\in\bar{B}_{\varepsilon+\frac{3+c}{m}}(\upsilon)}I(\sigma),-m\bigg{\\}}.$
(4.13)

We now take the limit as $\varepsilon$ goes to zero to obtain

$-H(\upsilon^{m}|\mu^{m})\leq\max\bigg{\\{}-\inf_{\sigma\in\bar{B}_{\frac{3+c}{m}}(\upsilon)}I(\sigma),-m\bigg{\\}},$
(4.14)

which readily implies, via Lemma 4.1,

$H(\upsilon|\mu)=\sup_{\bar{m}}H(\upsilon^{\bar{m}}|\mu^{\bar{m}})\geq\min\bigg{\\{}\inf_{\sigma\in\bar{B}_{\frac{3+c}{m}}(\upsilon)}I(\sigma),m\bigg{\\}},$
(4.15)

for all $m\in\mathbb{N}$.

Since the function $I$ is lower semicontinuous,

$\lim_{m\to\infty}\inf_{\sigma\in\bar{B}_{\frac{3+c}{m}}(\upsilon)}I(\sigma)=I(\upsilon).$
(4.16)

In particular,

$H(\upsilon|\mu)\geq I(\upsilon),$ (4.17)

concluding the proof. ∎

######  Proposition 4.6.

We have $H(\,\cdot\,|\mu)\leq I(\,\cdot\,)$.

###### Proof.

Fix $\upsilon\in\mathcal{P}(M)$ and observe once again that

$\begin{split}\frac{1}{n}&\log\mathbb{P}\big{(}L_{n}\in
B_{\varepsilon}(\upsilon)\big{)}\leq\frac{\log 2}{n}\\\
&\qquad\qquad+\max\bigg{\\{}\frac{1}{n}\log\mathbb{P}\big{(}L_{n}^{m}\in\bar{B}_{\varepsilon+\frac{1}{\sqrt{m}}}(\upsilon)\big{)},\frac{1}{n}\log\mathbb{P}\big{(}\textnormal{d}_{\rm
BL}(L_{n},L_{n}^{m})>\tfrac{1}{\sqrt{m}}\big{)}\bigg{\\}}.\end{split}$

Taking $n\to\infty$ and $\varepsilon\to 0$, we obtain with the aid of Lemma
4.4

$-I(\upsilon)\leq\max\bigg{\\{}-\inf_{\sigma\in\bar{B}_{\frac{1}{\sqrt{m}}}(\upsilon)}H(\sigma|\mu^{m}),\,-m\bigg{\\}}.$
(4.18)

Let us now split the discussion in whether $H(\upsilon|\mu)$ is finite or not.
Assume first that this relative entropy is infinite and notice that Lemma 4.3
implies

$\sup_{m}\inf_{\sigma\in\bar{B}_{\frac{1}{\sqrt{m}}}(\upsilon)}H(\sigma|\mu^{m})=\infty,$
(4.19)

which readily implies $I(\upsilon)=\infty$, when combined with (4.18).

If on the other hand we have $H(\upsilon|\mu)<\infty$, we combine (4.18) and
Lemma 4.3 with $m_{0}\geq H(\upsilon|\mu)$ to obtain

$I(\upsilon)\geq\min\bigg{\\{}\inf_{\sigma\in\bar{B}_{\frac{1}{\sqrt{m}}}(\upsilon)}H(\sigma|\mu^{m}),m\bigg{\\}}\geq\inf_{\sigma\in\bar{B}_{\frac{1}{\sqrt{m}}}(\upsilon)}H(\sigma|\mu^{m}),$
(4.20)

for every $m\geq m_{0}$. Taking the supremum in $m$ concludes the proof. ∎

##  5 Proof of Lemma 4.1

In this section we prove Lemma 4.1. We start with the following preliminary
lemma, which in particular implies the second part of Lemma 4.1. We prove the
first part afterwards.

######  Lemma 5.1.

If $\sigma\in\mathcal{P}(M)$ is such that $H(\sigma|\mu)\leq\alpha$, then, for
any $\theta>0$, we have

$\textnormal{d}_{\rm
BL}(\sigma^{m},\sigma)\leq\frac{1}{m}+2\frac{\alpha}{\theta}+2\frac{e^{-m^{2}-1+\theta}}{m\theta}.$
(5.1)

In particular,

$\textnormal{d}_{\rm BL}(\sigma^{m},\sigma)\leq\frac{3+2\alpha}{m},\text{ for
all }m\in\mathbb{N}.$ (5.2)

###### Proof.

Consider $X\sim\sigma$ and notice that $X^{m}=\pi^{m}(X)$ has distribution
$\sigma^{m}$. Therefore,

$\textnormal{d}_{\rm
BL}(\sigma^{m},\sigma)\leq\mathbb{E}(\textnormal{d}(X^{m},X)\wedge 2).$ (5.3)

Splitting on whether $X\in K_{m}$ or not, we obtain

$\textnormal{d}_{\rm
BL}(\sigma^{m},\sigma)\leq\frac{1}{m}+2\sigma(K_{m}^{\complement}).$ (5.4)

We now combine the entropy inequality with the bound $\log(1+x)\leq x$ to
obtain, for $\theta>0$,

$\begin{split}\sigma(K_{m}^{\complement})&=\frac{1}{\theta}\mathbb{E}_{\sigma}[\theta
1_{K_{m}^{\complement}}]\leq\frac{1}{\theta}\Big{(}H(\sigma|\mu)+\log\mathbb{E}_{\mu}[e^{\theta
1_{K_{m}^{\complement}}}]\Big{)}\\\
&=\frac{1}{\theta}\Big{(}\alpha+\log\big{(}1-\mu(K_{m}^{\complement})+e^{\theta}\mu(K_{m}^{\complement})\big{)}\Big{)}\\\
&\leq\frac{\alpha}{\theta}+\frac{(e^{\theta}-1)}{\theta}\mu(K_{m}^{\complement})\\\
&\leq\frac{\alpha}{\theta}+\frac{(e^{\theta}-1)}{\theta}\frac{e^{-m^{2}-1}}{m},\end{split}$
(5.5)

by the choice of $K_{m}$ in (3.1). Combining the equation above with (5.4)
concludes the proof of (5.1).

Choose now $\theta=m$ in (5.1) to obtain

$\textnormal{d}_{\rm
BL}(\sigma^{m},\sigma)\leq\frac{1}{m}+2\frac{\alpha}{m}+2\frac{e^{-m^{2}+m-1}}{m^{2}}\leq\frac{3+2\alpha}{m},$
(5.6)

concluding the proof. ∎

Second, we provide a martingale that will be useful during the proof.

######  Lemma 5.2.

Assume either that $H(\upsilon|\mu)$ or $\sup_{m}H(\upsilon^{m}|\mu^{m})$ is
finite. Then

$S_{m}=\frac{\textnormal{d}\upsilon^{m}}{\textnormal{d}\mu^{m}}\circ\pi^{m}$
(5.7)

is an uniformly-integrable martingale in the probability space
$\big{(}M,\mathcal{B}(M),\mu\big{)}$ with respect to the filtration
$\big{(}\mathcal{F}_{m}\big{)}_{m\in\mathbb{N}}$.

###### Proof.

Assume first that $H(\upsilon|\mu)<\infty$. In this case,
$\tfrac{\textnormal{d}\upsilon}{\textnormal{d}\mu}$ exists and

$\hat{S}_{m}=\mathbb{E}_{\mu}\Big{[}\frac{\textnormal{d}\upsilon}{\textnormal{d}\mu}\Big{|}\mathcal{F}_{m}\Big{]}$
(5.8)

is a uniformly-integrable martingale. It follows directly from the definition
of conditional expectation and Radon-Nikodyn derivative that
$\hat{S}_{m}=S_{m}$ almost surely for every $m\in\mathbb{N}$, concluding the
proof of the first case.

Assume now that $\sup_{m}H(\upsilon^{m}|\mu^{m})<\infty$ and observe that this
implies that $S_{m}$ is well defined for all $m\in\mathbb{N}$, has expectation
one, and is non-negative. We first have to verify that
$\mathbb{E}[S_{m+1}|\mathcal{F}_{m}]=S_{m}$. Take an element
$A_{m,k}\in\mathcal{F}_{m}$, with $0\leq k\leq\ell_{m}$, so that

$\mathbb{E}_{\mu}[S_{m}\cdot
1_{A_{m,k}}]=\frac{\upsilon(A_{m,k})}{\mu(A_{m,k})}\mathbb{E}_{\mu}[1_{A_{m,k}}]=\upsilon(A_{m,k}).$
(5.9)

Now, let $B_{1},\cdots,B_{j}$ elements of $\mathcal{F}_{m+1}$ such that
$\cup_{i=1}^{j}B_{i}=A_{m,k}$, then

$\begin{split}\mathbb{E}_{\mu}\left[\mathbb{E}_{\mu}[S_{m+1}|\mathcal{F}_{m}\right]\cdot
1_{A_{m,k}}]&=\mathbb{E}_{\mu}\left[\mathbb{E}_{\mu}\left[S_{m+1}\left.\sum_{i=1}^{j}1_{B_{i}}\right|\mathcal{F}_{m}\right]\right]\\\
&=\mathbb{E}_{\mu}\left[\mathbb{E}_{\mu}\left[\left.\sum_{i=1}^{j}\frac{\upsilon(B_{i})}{\mu(B_{i})}1_{B_{i}}\right|\mathcal{F}_{m}\right]\right]\\\
&=\sum_{i=1}^{j}\frac{\upsilon(B_{i})}{\mu(B_{i})}\mu(B_{i})\\\
&=\upsilon(A_{m,k})=\mathbb{E}[S_{m}1_{A_{m,k}}],\end{split}$ (5.10)

which implies $\mathbb{E}[S_{m+1}|\mathcal{F}_{m}]=S_{m}$ , concluding our
first statement.

In order to verify uniform integrability of $S_{n}$, observe that

$\mathbb{E}_{\mu}[S_{m}\log
S_{m}]=H(\upsilon^{m}|\mu^{m})\leq\sup_{m}H(\upsilon^{m}|\mu^{m}):=K<\infty.$
(5.11)

Now, for each $M>0$ we have, uniformly in $m\in\mathbb{N}$,

$E_{\mu}[S_{m}1_{\\{S_{m}\geq
M\\}}]\leq\mathbb{E}_{\mu}\left[S_{m}1_{\\{S_{m}\geq M\\}}\dfrac{\log
S_{m}}{\log M}\right]\leq\frac{K}{\log M}.$ (5.12)

Therefore, $S_{m}$ is a uniformly-integrable martingale, concluding the proof
of the lemma. ∎

We are now in position to prove the first part of Lemma 4.1.

###### Proof of Lemma 4.1.

We first observe that, via the variational definition of entropy and the fact
that $M_{m}\subset M_{m+1}$, for all $m\in\mathbb{N}$, we obtain that
$H(\upsilon^{m}|\mu^{m})$ is monotone increasing in $m$ (following the steps
pointed out in Remark 4.2 or directly as a consequence of [4, Corollary
5.2.2]). In particular,

$\sup_{m}H(\upsilon^{m}|\mu^{m})=\lim_{m\to\infty}H(\upsilon^{m}|\mu^{m})$
(5.13)

and thus it suffices to verify that

$\lim_{m}H(\upsilon^{m}|\mu^{m})=H(\upsilon|\mu).$ (5.14)

Again from the variational definition of relative entropy we have
$H(\upsilon^{m}|\mu^{m})\leq H(\upsilon|\mu)$, for all $m\in\mathbb{N}$, so
that

$\limsup_{m\to\infty}H(\upsilon^{m}|\mu^{m})\leq H(\upsilon|\mu).$ (5.15)

We now work on the proof of the reverse inequality. The strategy of the proof
is as follows. If at least one of the two quantities of interest is finite, we
have access to the uniformly-integrable martingale $S_{m}$ given by the Radon-
Hikodyin derivative of $\upsilon^{m}$ with respect to $\mu^{m}$. As we will
see, this martingale converges in $L^{1}$ and almost surely to
$\tfrac{\textnormal{d}\upsilon}{\textnormal{d}\mu}$, which will yield the
result when combined with Fatou’s Lemma.

Assume that either $H(\upsilon|\mu)<\infty$ or
$\sup_{m}H(\upsilon^{m}|\mu^{m})<\infty$. The martingale $S_{m}$ introduced in
(5.7) is uniformly integrable and thus converges almost surely and in $L^{1}$
to a random variable $X$.

In the case $H(\upsilon|\mu)<\infty$, we have

$X=E_{\mu}\Big{[}\frac{\textnormal{d}\upsilon}{\textnormal{d}\mu}\Big{|}\mathcal{F}_{\infty}\Big{]}=\frac{\textnormal{d}\upsilon}{\textnormal{d}\mu},$
(5.16)

since $\mathcal{F}_{\infty}=\mathcal{B}(M)$ (see Lemma 3.1). If we are in the
case $\sup_{m}H(\upsilon^{m}|\mu^{m})<\infty$, the above also holds precisely
because elements in $\mathcal{B}(M)$ can be approxiamted by elements in
$\cup_{m=1}^{\infty}\mathcal{F}_{m}$.

We now note that, since $x\log x\geq-e^{-1}$, Fatou’s Lemma implies

$\liminf_{m}\mathbb{E}_{\mu}\big{[}S_{m}\log
S_{m}\big{]}\geq\mathbb{E}_{\mu}\Big{[}\frac{\textnormal{d}\upsilon}{\textnormal{d}\mu}\log\frac{\textnormal{d}\upsilon}{\textnormal{d}\mu}\Big{]},$
(5.17)

which verifies (4.1) (see also Lemma 2.3) and concludes the proof of the
lemma.

∎

##  6 Proof of Lemma 4.3

In this section we prove Lemma 4.3. We fix $m_{0}\in\mathbb{N}$ and denote by

$I^{0}(\upsilon):=\sup_{m\geq
m_{0}}\inf_{\sigma\in\bar{B}_{\frac{1}{\sqrt{m}}}(\upsilon)}H(\sigma|\mu^{m}).$
(6.1)

Our goal is to show that $I^{0}(\upsilon)=H(\upsilon|\mu)$. We will prove this
in two steps, by checking that $I^{0}(\upsilon)\leq H(\upsilon|\mu)$ and
$I^{0}(\upsilon)\geq H(\upsilon|\mu)$. The first inequality is verified in the
next paragraph. The reverse inequality is more delicate and we dedicate the
rest of the section to verify it.

Let us check that $I^{0}(\upsilon)\leq H(\upsilon|\mu)$. Indeed, the
inequality is trivial if $H(\upsilon|\mu)=\infty$. If on the other hand this
entropy is finite, we have, in view of Lemma 4.1,

$\textnormal{d}_{\rm
BL}(\upsilon,\upsilon^{m})\leq\frac{c}{m}\leq\frac{1}{\sqrt{m}},$ (6.2)

for $m$ large enough, from which our claim follows by noting that
$\upsilon^{m}\in\bar{B}_{\frac{1}{\sqrt{m}}}(\upsilon)$ and applying Lemma
4.1.

We now focus on the proof of the inequality

$I^{0}(\upsilon)\geq H(\upsilon|\mu).$ (6.3)

Once again we assume that $I^{0}(\upsilon)<\infty$, since the alternative case
is trivial.

The first observation we make is that (6.3) follows if, for any $\alpha>0$,

$I^{0}(\upsilon)<\alpha\text{ implies }H(\upsilon|\mu)\leq\alpha.$ (6.4)

This follows directly from the following lemma together with the lower
semicontinuity of the relative entropy $H(\,\cdot\,|\mu)$.

######  Lemma 6.1.

If $I^{0}(\upsilon)<\alpha$, then, for every $\varepsilon>0$, there exists
$\rho\in B_{\varepsilon}(\upsilon)$ such that

$H(\rho|\mu)\leq\alpha.$ (6.5)

###### Proof.

Our goal will be find $\rho$ such that $H(\rho|\mu)\leq\alpha$ and
$\textnormal{d}_{\rm BL}(\upsilon,\rho)<\varepsilon$. Fix $m\geq m_{0}$ large
enough such that

$\frac{3+2\alpha}{m}+\frac{1}{\sqrt{m}}<\varepsilon,$ (6.6)

Recall from (6.1) that $I^{0}(\upsilon)<\alpha$ implies that there exists
$\sigma\in\bar{B}_{\frac{1}{\sqrt{m}}}(\upsilon)$ such that
$H(\sigma|\mu^{m})\leq\alpha$. Notice that, since this entropy is finite, we
have $\sigma=\sigma^{m}$.

Define

$\rho(F):=\sum_{i=1}^{\ell_{m}}\frac{\sigma(A_{m,i})}{\mu(A_{m,i})}\mu(F\cap
A_{m,i})=\sum_{i=0}^{\ell_{m}}\mu(F|A_{m,i})\sigma(A_{m,i}).$ (6.7)

Via direct substitution it follows that $\rho^{m}=\sigma$. We claim that
$H(\rho|\mu)=H(\sigma|\mu^{m})\leq\alpha$ and $\textnormal{d}_{\rm
BL}(\upsilon,\rho)<\varepsilon$.

In order to verify that $H(\rho|\mu)=H(\sigma|\mu^{m})$, observe that

$\begin{split}H(\rho^{m+j}|\mu^{m+j})&=\sum_{i=0}^{\ell_{m+j}}\rho(A_{m+j,i})\log\frac{\rho(A_{m+j,i})}{\mu(A_{m+j,i})}\\\
&=\sum_{i=0}^{\ell_{m}}\sum_{k:A_{m+j,k}\subset
A_{m,i}}\rho(A_{m+j,k})\log\frac{\rho(A_{m+j,k})}{\mu(A_{m+j,k})}.\end{split}$
(6.8)

Furthermore, if $A_{m+j,k}\subset A_{m,i}$, then, from (6.7),

$\rho(A_{k+j,m})=\frac{\sigma(A_{m,i})}{\mu(A_{m,i})}\mu(A_{m+j,k}).$ (6.9)

Therefore,

$\begin{split}H(\rho^{m+j}|\mu^{m+j})&=\sum_{i=0}^{\ell_{m}}\sum_{k:A_{m+j,k}\subset
A_{m,i}}\frac{\sigma(A_{m,i})}{\mu(A_{m,i})}\mu(A_{m+j,k})\log\frac{\sigma(A_{m,i})}{\mu(A_{m,i})}\\\
&=H(\sigma|\mu^{m}),\end{split}$ (6.10)

since

$\sum_{k:A_{m+j,k}\subset A_{m,i}}\mu(A_{m+j,k})=\mu(A_{m,i}).$ (6.11)

In particular, from Lemma 4.1, $H(\rho|\mu)=H(\sigma|\mu^{m})\leq\alpha$.

Finally, we now prove that $\textnormal{d}_{\rm
BL}(\rho,\upsilon)<\varepsilon$ by estimating

$\begin{split}\textnormal{d}_{\rm BL}(\rho,\upsilon)&\leq\textnormal{d}_{\rm
BL}(\rho,\rho^{m})+\textnormal{d}_{\rm
BL}(\rho^{m},\sigma)+\textnormal{d}_{\rm BL}(\sigma,\upsilon)\\\
&\leq\frac{3+2\alpha}{m}+\frac{1}{\sqrt{m}}\leq\varepsilon,\end{split}$ (6.12)

where the last line uses Lemma 5.1, since $H(\rho|\mu)$ is bounded by $\alpha$
and recalling that $\rho^{m}=\sigma$. This concludes the proof. ∎

## References

  * [1] Imre Csiszár.  A simple proof of Sanov’s theorem.  Bulletin of the Brazilian Mathematical Society, 37(4):453–459, 2006\.
  * [2] Amir Dembo and Ofer Zeitouni.  Large deviations techniques and applications.  Stochastic Modelling and Applied Probability. Springer Berlin Heidelberg, 2009.
  * [3] Frank Den Hollander.  Large deviations.  American Mathematical Society, 2008.
  * [4] Robert M Gray.  Entropy and information theory.  Springer Science & Business Media, 2011.
  * [5] Srinivasa RS Varadhan.  Probability theory.  New York University Courant Institute of Mathematical Sciences, New York, 2001.