# Sensitivity-Aware Mixed-Precision Quantization and Width Optimization of
Deep Neural Networks Through Cluster-Based Tree-Structured Parzen Estimation

Seyedarmin Azizi*, Mahdi Nazemi*, Arash Fayyazi, and Massoud Pedram
*Seyedarmin Azizi and Mahdi Nazemi contributed equally to this work.
Department of Electrical & Computer Engineering, University of Southern
California, Los Angeles, CA, USA
<EMAIL_ADDRESS>
###### Abstract

As the complexity and computational demands of deep learning models rise, the
need for effective optimization methods for neural network designs becomes
paramount. This work introduces an innovative search mechanism for
automatically selecting the best bit-width and layer-width for individual
neural network layers. This leads to a marked enhancement in deep neural
network efficiency. The search domain is strategically reduced by leveraging
Hessian-based pruning, ensuring the removal of non-crucial parameters.
Subsequently, we detail the development of surrogate models for favorable and
unfavorable outcomes by employing a cluster-based tree-structured Parzen
estimator. This strategy allows for a streamlined exploration of architectural
possibilities and swift pinpointing of top-performing designs. Through
rigorous testing on well-known datasets, our method proves its distinct
advantage over existing methods. Compared to leading compression strategies,
our approach records an impressive 20% decrease in model size without
compromising accuracy. Additionally, our method boasts a 12$\times$ reduction
in search time relative to the best search-focused strategies currently
available. As a result, our proposed method represents a leap forward in
neural network design optimization, paving the way for quick model design and
implementation in settings with limited resources, thereby propelling the
potential of scalable deep learning solutions.

##  I Introduction

Deep neural networks (DNNs) have emerged as highly powerful and versatile
tools for tackling real-world problems across various domains, including
computer vision [1, 2, 3, 4] and natural language processing [5, 6, 7]. The
remarkable performance of DNNs can be attributed to their capacity to learn
intricate patterns and representations from large-scale data, involving
millions or even billions of arithmetic operations and parameters. However,
this success comes at the expense of high compute cycles, memory footprint,
I/O bandwidth, and energy consumption, resulting in an increased carbon
footprint and limitations in deploying them on resource-constrained platforms.

To address these challenges, quantization and structured pruning techniques
have emerged as promising strategies, offering the potential to mitigate the
computational and memory burden of DNNs while maintaining satisfactory
accuracy [8, 9, 10, 11]. These techniques have proven effective in making DNNs
more efficient, enabling their deployment on edge devices, and accelerating
inference in cloud environments. However, the successful application of
quantization and structured pruning heavily relies on finding the optimal bit-
width and layer-width for each layer of the DNN, which presents a formidable
challenge due to the exponential growth of the search space as the number of
layers increases.

The challenges in optimizing the bit-width for each layer of a DNN are further
compounded by the inherent diversity in weight distributions across different
layers (see Fig. 1 in addition to the sensitivity of a DNN’s predictions to
each layer’s weights.

Figure 1: Distribution of weights in three representative layers of the
MobileNetV1 architecture trained on the CIFAR-100 dataset.

The weight distributions and sensitivity values can vary significantly, and as
a result, different layers may benefit from different bit-widths to achieve
maximum gains in terms of memory and computation cost savings without
compromising accuracy.

Additionally, widening layers can profoundly impact the accuracy of the model
[2]. Therefore, the search process should not be confined to reducing the
layer-width only; it must also identify situations where widening a layer,
combined with a sufficiently reduced bit-width, leads to potential accuracy
gains without compromising cost savings.

The search process for optimizing the bit-width and layer-width of DNNs should
extend beyond traditional metrics such as FLOPs and memory footprint when
evaluating cost savings. While these metrics are useful, they may not directly
reflect the real-world performance and efficiency of the model. Therefore, it
is imperative to consider factors like real latency, throughput, and energy
consumption when comparing models in terms of cost savings. Hence, devising an
algorithm that effectively explores the bit-width and layer-width search space
while considering the diverse weight distributions and sensitivity values and
the varying impact of layer-width on accuracy and real-world performance
becomes critical. This necessitates an approach that can intelligently adapt
the bit-width and layer-width for each layer, striking a balance between
latency, throughput, or energy consumption and the preservation of model
accuracy.

This paper presents a novel model-based optimization method based on tree-
structured Parzen estimators (TPEs) [12] to address the challenge of
simultaneously searching for the optimal bit-width and layer-width values for
DNN layers. The major innovations of the presented optimization method are as
follows:

  * •

Exploiting Second-Order Derivatives: In addition to leveraging the
distribution of layer weights, our method incorporates second-order
derivatives of the loss function with respect to layer weights. This
exponential pruning of the search space is particularly effective when dealing
with pre-trained models, resulting in enhanced optimization efficiency.

  * •

Hardware-Aware Objective Function: Our approach takes into account essential
information about the target hardware that will execute the optimized DNN. We
construct latency, throughput, and energy consumption models, which, combined
with model accuracy, define a composite objective function guiding the
optimization process.

  * •

Handling Flat Loss Landscapes: DNNs often exhibit flat loss landscapes, which
pose challenges for conventional optimizers. To address this, we fit surrogate
distributions to both desirable and undesirable observations of the objective
function. This adaptation enables us to achieve comparable or better objective
values compared to state-of-the-art optimizers while significantly reducing
convergence time, typically by a factor of at least 10x.

  * •

Joint Optimization of Bit-Width and Layer-Width: Unlike conventional
approaches that treat bit-width and layer-width independently, we search for
joint optimal configurations. This novel approach enables us to discover
configurations that yield higher-quality results, which would have otherwise
been challenging to find with independent optimization.

Through an extensive series of experiments, we demonstrate the effectiveness
and efficiency of our method. The results showcase substantial improvements in
DNN processing efficiency without sacrificing predictive accuracy.

##  II Related Work

Mixed-precision quantization exploits the fact that not all model parameters
require the same level of precision to maintain model accuracy. Quantization-
aware training techniques simulate the quantization effects during the
training process, allowing the model to adapt to lower precision, while post-
training quantization techniques quantize DNN models after they have been
trained at full precision. The former typically achieves higher accuracy at
the expense of increased training time.

In sensitivity-based mixed-precision quantization, first- or second-order
gradient statistics, or other sensitivity metrics, are employed as a proxy to
layer importance, which is subsequently used to assign an appropriate bit-
width to each layer [13, 14, 15]. However, sensitivity-based quantization
comes has some shortcomings. First, it does not take into account the
influence of quantized weights and input activations on output activations,
which are inputs to other layers of a DNN including batch normalization layers
commonly found after convolutional layers. In other words, the batch
statistics of batch normalization layers or weights of succeeding
convolutional layers are trained with full-precision input activations, and
any change in those input activations due to quantization may hamper model
accuracy. Second, gradient-based methods utilize gradients calculated on the
full-precision model, thus overlooking the impact of quantization on
gradients, which is a significant effect when using ultra-low bit-width
quantization. Third, the sensitivity values are often calculated based on the
weights of layers, and thus, do not provide any insight into the proper bit-
width of input activations. As a result, sensitivity-based quantization
approaches often fall short of achieving high model size compression ratios
and/or maintaining accuracy levels, leaving ample space for further
optimization.

In reinforcement learning (RL)-based mixed-precision quantization, an agent
interacts with a quantized DNN environment, adjusting the bit-width
configurations for various layers using RL algorithms, such as policy
gradients, which provide rewards based on model accuracy and resource
consumption, enabling the agent to discover optimal quantization strategies
through iterative exploration and exploitation [16, 17, 18]. Despite their
potential benefits, these RL-based techniques are confronted with a
significant challenge: the considerable search time involved in the RL
training process. As a consequence, achieving favorable results within
specified GPU-hour constraints becomes challenging.

Differentiable search-based approaches build upon the concept of
differentiable architecture search [19] and apply it to mixed-precision
quantization. In these approaches, a super-network is trained to replace each
layer (or activation function) of a DNN, such as a convolutional layer, with
parallel branches where each branch implements a quantized version of the
layer (or activation function) [20]. These approaches have significant
drawbacks. First, they suffer from large training times and high GPU RAM
requirements due to the very large size of the super-network. Second, the
distribution defined over each group of parallel branches, which is found
during training, may not converge to a uni-modal distribution, rendering the
selection of a single quantized version of a layer infeasible.

Lastly, sequential model-based mixed-precision quantization approaches employ
surrogate models that define a mapping between the search space and an
objective function to guide the exploration of the search space [21]. Tree-
structured Parzen estimator (TPE) is a powerful tool that has shown great
success in hyperparameter tuning [22]. However, its naive application to
mixed-precision quantization ignores the flat loss landscapes that are
prevalent in DNNs. This greatly increases the search time and yields inferior
objective values. Our TPE-based optimization achieves a $12\times$ average
search time speedup and a 20% reduction in model size compared to [21] while
preserving the model accuracy.

##  III Proposed Method

This section details the three main components of our model-based bit-width
and layer-width optimization framework, i.e., Hessian-based search space
pruning, $K$-Means TPE, and hardware-aware performance modeling.

###  III-A Hessian-Based Search Space Pruning

The size of the search space grows exponentially as the number of DNN layers
increases. As a result, pruning the search space by eliminating the bit-width
choices that are likely to hamper the model accuracy or cause unnecessary
computations is of paramount importance due to its exponential reduction in
the size of the search space. The Hessian of the loss function with respect to
each layer’s weights provides an excellent starting point to evaluate the
criticality of the bit-width for each layer. We first prove an important
result.

######  Lemma 1.

The maximum error induced in a DNN’s output by unit perturbation in a layer’s
parameters is bounded by the trace value of the Hessian matrix of the loss
with respect to that layer’s parameters.

###### Proof.

Assume we freeze all parameters of a DNN except those of a single
convolutional filter or neuron in layer $l$. Let $\mathbold{w}_{l}$ and
$\mathbold{w}_{l}^{\mathrm{q}}$ denote the parameters of layer $l$ and their
corresponding quantized values, respectively. Taylor’s Theorem implies that
the output loss may be approximated around $\mathbold{w}_{l}^{\mathrm{q}}$ as
follows:

$\begin{split}\mathcal{L}(\mathbold{w}_{l})\approx\mathcal{L}(\mathbold{w}_{l}^{\mathrm{q}})&+(\mathbold{w}_{l}-\mathbold{w}_{l}^{\mathrm{q}})^{\mathrm{T}}\mathbold{\nabla}\mathcal{L}_{\mathbold{w}_{l}}\\\
&+\frac{1}{2}(\mathbold{w}_{l}-\mathbold{w}_{l}^{\mathrm{q}})^{\mathrm{T}}\mathbold{H}_{\mathbold{w}_{l}}(\mathbold{w}_{l}-\mathbold{w}_{l}^{\mathrm{q}}),\end{split}$

where $\mathbold{\nabla}\mathcal{L}_{\mathbold{w}_{l}}$ is the gradient vector
of the loss function with respect to parameters $\mathbold{w}$ evaluated at
$\mathbold{w}_{l}$, which is nearly zero on a trained model, and
$\mathbold{H}_{\mathbold{w}_{l}}$ denotes the Hessian matrix, whose entries
are the second derivatives of the loss function with respect to parameters
$\mathbold{w}$ evaluated at $\mathbold{w}_{l}^{\mathrm{q}}$. Therefore, we
have:

$\mathcal{L}(\mathbold{w}_{l})-\mathcal{L}(\mathbold{w}_{l}^{\mathrm{q}})\approx\frac{1}{2}(\mathbold{w}_{l}-\mathbold{w}_{l}^{\mathrm{q}})^{\mathrm{T}}\mathbold{H}_{\mathbold{w}_{l}}(\mathbold{w}_{l}-\mathbold{w}_{l}^{\mathrm{q}}).$
(1)

By writing the spectral decomposition of the Hessian matrix as
$\mathbold{H}_{\mathbold{w}_{l}}=\mathbold{U}\mathbold{D}\mathbold{U}^{T}$,
where $\mathbold{U}$ is a unitary matrix and $\mathbold{D}$ is a diagonal
matrix, and plugging this expression back into (1), we obtain:

$\Delta\mathcal{L}(\mathbold{w}_{l})\approx\frac{1}{2}(\mathbold{U}^{\mathrm{T}}\Delta\mathbold{w}_{l})^{\mathrm{T}}\mathbold{D}(\mathbold{U}^{\mathrm{T}}\Delta\mathbold{w}_{l}).$

with $\Delta\mathbold{w}_{l}=\mathbold{w}_{l}-\mathbold{w}_{l}^{\mathrm{q}}$.
Denoting $\mathbold{U}^{T}\Delta\mathbold{w}_{l}$ with $\mathbold{a}$, we
have:

$\begin{split}\Delta\mathcal{L}(\mathbold{w}_{l})\approx\frac{1}{2}\mathbold{a}^{\mathrm{T}}\mathbold{D}\mathbold{a}&=\frac{1}{2}\sum\limits_{i}a_{i}^{2}\lambda_{i}\\\
&\leq\frac{1}{2}\max_{i}(a_{i}^{2})\sum\limits_{i}\lambda_{i}.\end{split}$

Since $\mathbold{U}$ is a unitary matrix and the lemma assumes unit
perturbation, $\max_{i}(a_{i}^{2})\leq 1$. Therefore,

$\Delta\mathcal{L}(\mathbold{w}_{l})\leq\frac{1}{2}\mathrm{Tr}(\mathbold{H}_{\mathbold{w}_{l}}).$

∎

The significance of this lemma lies in its implications for training and
quantizing DNNs. For example, a small trace value of the Hessian matrix
indicates that the loss function’s surface is relatively flat around the
current parameter values, making the model less sensitive to perturbations and
potentially more stable during training. On the other hand, a large trace
value suggests that the loss function’s surface has steeper curves, indicating
higher sensitivity to parameter changes and potentially making training more
challenging.

The Hessian-based search space pruning algorithm starts by normalizing the
trace of the Hessian of the loss function with respect to each layer’s weight
by the number of weights in that layer to find an estimate of the relative
importance of layers in a DNN. It then applies $k$-means clustering to the
normalized trace values, sorts clusters in non-increasing order of their
centroid values, and assigns candidate bit-widths to the layers within each
cluster according to these centroid values by assigning higher bit-widths to
layers that are within clusters with larger trace values. We use the same bit-
width for weights and input activations of a DNN layer for improved hardware
performance. For example, when $k=4$, possibly overlapping subsets of
candidate bit-widths $B=\\{8,6,4,3,2\\}$ can be considered for different
clusters, e.g., $B_{1}=\\{8,6\\}$, $B_{2}=\\{6,4,3\\}$, $B_{3}=\\{4,3,2\\}$,
and $B_{4}=\\{3,2\\}$, where the bit-widths of layers within the cluster with
the largest centroid value are selected from $B_{1}$, and those for the second
largest centroid value are chosen from $B_{2}$, and so on.111The part of the
search space defined by the layer-width values is not pruned, and layer-width
is always taken from the set $S=\\{0.75,0.875,1,1.125,1.25\\}$.

###  III-B $K$-Means TPE

An integral part of the presented framework is a novel sequential model-based
optimization methodology that is based on the TPE methods. The main idea
behind TPE methods is to recursively partition the data space into smaller
regions (nodes) and estimate the probability density function within each
region separately. In particular, the TPE methods use Bayesian reasoning to
propose configurations from the search space that are likely to improve an
objective function as explained next. After drawing a few random
configurations $\chi=\\{x^{(1)},x^{(2)},\ldots,x^{(k)}\\}$ from the search
space and observing their corresponding objective function values
$Y=\\{y^{(1)},y^{(2)},\ldots,y^{(k)}\\}$, the TPE methods define a threshold
$\hat{y}$ that is equal to the largest of $q$-quantiles of $Y$, and creates a
surrogate distribution $l(x)$ for configurations with desirable objective
function values ($y^{(i)}\geq\hat{y}$ in a maximization problem) and a
surrogate distribution $g(x)$ for configurations with undesirable objective
function values ($y^{(i)}<\hat{y}$.) The candidate configuration to be
evaluated next is an $\tilde{x}$ that maximizes the $\frac{l(x)}{g(x)}$ ratio.
The chosen $\tilde{y}$ leads to an update in $\hat{y}$, $l(x)$, and $g(x)$
until a stopping criterion is met.

Due to the flat loss landscapes that are prevalent in DNNs, widely different
configurations from the search space may yield very close objective values.
This becomes problematic when objective values of configurations from
promising parts of the search space fall slightly below $\hat{y}$, which puts
those configurations in $g(x)$. This effectively discourages exploring those
parts of the search space, which, in turn, can yield configurations with
inferior objective values.

To address this problem, we introduce a novel dual-threshold TPE method that
incorporates $k$-means clustering in the threshold definition process (which
we name $k$-means TPE.) The dual-threshold optimizer starts with an initial
$k$ (where $k\geq 3$) for clustering the elements of $Y$, sorts clusters in
decreasing order of their centroid values ($C_{1},\ldots,C_{k}$), and defines
the surrogate distributions as follows:

$p(x|y)=\begin{cases}l(x),&\text{if $y\in C_{1}$ }\\\ g(x),&\text{if $y\in
C_{k}$ .}\end{cases}$

After every few iterations of the search process, we increase $k$, which
tightens the criteria of being a desirable or undesirable configuration. This
effectively implements an annealing process that initially allows for large
moves in the search space to explore distant regions of the search space, and
gradually reduces the move size so that search is narrowed to nearby regions
of the search space (i.e., those regions that are close to the currently found
promising solutions.)

###  III-C Hardware-Aware Objective Function

The problem addressed in this work is mathematically formulated in its general
form as follows:

$\displaystyle\max_{B,S}$
$\displaystyle\mathrm{accuracy}(\mathbold{\Theta},\mathcal{D},B,S)$ s.t.
$\displaystyle\mathrm{modelSize}(\mathbold{\Theta},B,S)\leq\mu$
$\displaystyle\mathrm{latency}(\mathbold{\Theta},\mathcal{H},B,S)\leq\tau$
$\displaystyle\mathrm{energy}(\mathbold{\Theta},\mathcal{H},B,S)\leq\epsilon$
$\displaystyle\mathrm{throughput}(\mathbold{\Theta},\mathcal{H},B,S)\geq\pi,$

where $\mathbold{\Theta}$ denotes the parameters of the target DNN to be
optimized, $\mathcal{D}$ comprises samples from a target dataset, $B$ and $S$
denote the sets of candidate bit-widths and layer-widths multipliers,
respectively, $\mathcal{H}$ characterizes the target hardware, $\mu$, $\tau$,
and $\epsilon$ represent the model size, latency, and energy consumption upper
bounds, and $\pi$ denotes the throughput lower bound.

The optimization problem can be addressed by solving its Lagrangian dual
problem, where large Lagrangian multipliers are assigned to various
constraints. In practical applications, one or more of the said constraints
may be relaxed. The focus of this work is on the model size and latency
constraints.

Exemplary target hardware used in this work is a Xilinx FPGA in which the chip
layout includes columns of DSPs, BRAMs, and CLBs. Our design for the hardware
architecture of the accelerator that processes a given DNN is comprised of

  * •

a 2D systolic array $M\times N$ of processing elements (PEs) where each PE
contains one DSP and a companion BRAM, and

  * •

a memory hierarchy that encompasses off-chip DRAM, on-chip URAMs and BRAMs,
and register files.

When processing some DNN layer, a first set of $N$ input activations are
loaded into the first row of the systolic array, and each input activation is
multiplied with a corresponding weight for the first output filter, residing
in the BRAM associated with each of the PEs in that row. In the next cycle,
the first set of input activations is passed down the second row of PEs, while
a second set of input activations are loaded into the first row. The second
set of input activations is multiplied with a second set of weights associated
with the first filter while the first set of activations is multiplied with
the first set of weights associated with the second output filter. Evidently,
this process is repeated multiple times (i.e., $N^{\prime}/N$ where $N$ is the
number of entries in the input feature patch, which is commonly equal to
$3\times 3\times I$ with $I$ denoting the input channel count) to produce the
filter results for $M$ output channels of the layer. The partial products
computed in different cycles are accumulated within each PE in the 2D array.
In the end, all partial accumulation results stored locally in the PEs of each
row are passed onto a tree adder structure (which we call a processing unit,
of which there are $M$) to produce the final scalar convolution result for
each of the output channels. We point out that when the number of output
channels $M^{\prime}$ is larger than $M$, the systolic array must be invoked
$M^{\prime}/M$ times. Each DSP block can perform a $27\times 18$-bit
multiplication followed by a 48-bit accumulation. A crux of our design lies in
packing multiple low-bit-width operands in each line of memory in addition to
utilizing each DSP to efficiently perform multiple multiplications and
additions.

We extend the idea of HiKonv [23], which introduces packed operations for 1D
convolutions, to support 2D convolutions with arbitrary bit-widths. More
specifically, our operand and operation packing approach is capable of
performing two multiplications for eight- or six-bit operands, six
multiplications and two additions for four- or three-bit operands, and 15
multiplications and eight additions for two-bit operands all while only using
a single DSP. Fig. 2 illustrates an example of operand and operation packing
for four-bit operands.

Figure 2: Four-bit operand and operation packing. The design yields the
computations required for two rows of convolutional kernels every two cycles.

As a result of our architecture design and packing scheme, weight quantization
yields linear weight size reduction as a function of the bit-width selected
for a layer while latency reduction is a function of the number of operations
that can be packed as explained above. Considering the total number of layers
and the number of weights/operations per layer, the overall model size
reduction and speedup can be easily calculated. Alg. 1 summarizes different
steps of the search process presented in this work.

Algorithm 1 $k$-means-based TPE

1:

2:$\mathbold{\Theta}$ //DNN to be optimized

3:$\mathcal{D}$ //dataset

4:$\mu$, $\tau$ //model size limit and latency constraint

5:$B,S$ //available bit-widths and layer-width multipliers

6:${n_{0}}$ //no. of random configurations to evaluate before building
surrogates

7:${n}$ //total number of configurations to evaluate

8:$\alpha=0.98,\mathrm{maxiters}=100,c=0.25$

9:

10:$x^{*}$ //the optimal bit-widths and layer-widths configurations

11:$\texttt{sensitivities}=\texttt{analyze\\_hessian}(\mathbold{\Theta},\mathcal{D})$

12:$\texttt{space}=\texttt{create\\_search\\_space}(\texttt{sensitivities},B,S)$

13:$\chi=\texttt{sample\\_randomly}(\texttt{space},n_{0})$

14:

15:$\chi_{l}=\emptyset$ //desirable configurations

16:$\chi_{g}=\emptyset$ //undesirable configurations

17:$\texttt{objectives}=[]$ //initializing objective values list

18:for each $x$ $\in\chi$ do

19:
${\texttt{objectives.add}(\texttt{evaluate}(x,\mathbold{\Theta},\mathcal{D},\mu,\tau))}$

20:end for

21:do

22: $k=\lceil\frac{1}{c}\rceil$

23: $\texttt{clusters}=\texttt{k\\_means\\_and\\_sort}(\texttt{objectives},k)$

24: $\chi_{l}=C_{1},\chi_{g}=C_{k}$

25: $l(x)=\texttt{fit\\_gaussian}(\chi_{l})$

26: $g(x)=\texttt{fit\\_gaussian}(\chi_{g})$

27: $\tilde{x}=\operatorname*{argmax}_{x}l(x)/g(x)$

28: $\chi=\chi\bigcup\\{\tilde{x}\\}$

29:
${\texttt{objectives.add}(\texttt{evaluate}(\tilde{x},\mathbold{\Theta},\mathcal{D},\mu,\tau))}$

30: $c=c\times\alpha$

31:while $\mathrm{iter}\leq(n-n_{0})$

32:$x^{*}=\max_{x}\texttt{objectives}$

33:return $x^{*}$

##  IV Results & Discussion

This section presents the results of experiments that evaluate the
effectiveness of our methodology, which involve hyperparameter tuning in
addition to mixed-precision quantization and layer-width scaling through
neural architecture search.

###  IV-A Convergence of $K$-Means TPE

This section presents the speedup in convergence for three types of machine
learning models and three datasets. The first experiment involves applying
random forest regression to the Iris dataset. The variables that define the
search space are the number of trees in the forest, the maximum depth of each
tree, and the minimum number of samples required to split an internal node.

The second experiment involves training a gradient boosting classifer on the
Titanic dataset. The variables that define the search space are the learning
rate, number of boosting stages, maximum depth of the estimator, minimum
number of samples required to split, minimum number of samples needed to be at
a leaf node, and number of features to consider when looking for the best
split. For the first two experiments, $n_{0}=20$, $n=100$, $k=4$, and
$\alpha=0.98$.

Finally, the third experiment involves mixed-precision quantization and layer-
width scaling of a ResNet-18 model on the CIFAR-100 dataset. For this
experiment, $n_{0}=40$, $n=160$, $k=4$, and $\alpha=0.98$. Figure 3 showcases
the superior convergence speed of the presented $k$-means TPE over the
traditional TPE for all experiments.

Figure 3: Comparison of the convergence speed of TPE and $k$-means TPE for
different machine learning algorithms and on Iris, Titanic, and CIFAR-100
datasets.

It is observed that $k$-means TPE converges to superior or same-quality
results at about two to three times fewer evaluations of proposed
configurations compared to TPE.

###  IV-B Mixed-Precision Quantization and Layer-Width Scaling

To show the effectiveness of $k$-means TPE on a variety of DNNs and datasets,
we integrate it into the HyperOpt [22] library, and use the PyTorch library
[24] to train and quantize the models. For all experiments, we only use a very
small number of epochs to evaluate different configurations during the search
process, similar to [19]. More specifically, for models trained on the
CIFAR-10 and CIFAR-100 datasets, the number of epochs to train is set to four,
and for the more extensive ImageNet dataset [25], this number is set to one.
As shown in Table I for an exemplary model and dataset, such an approximation
does not make a noticeable difference in the final results compared to a
scenario where each configuration is trained for a much larger number of
epochs.

TABLE I: Impact of the number of epochs to train each configuration on final
accuracy, model size, and speedup (compared to FiP-16)

Dataset | Architecture | Approach | Accuracy | Model | Speedup
---|---|---|---|---|---
(Epochs per Config.) | (%) | Size (MB)
CIFAR-10 | ResNet-20 | 90 | 91.94 | 0.097 | 10.86$\times$
4 | 91.90 | 0.088 | 11.14$\times$

After finding the best configuration, we train the models for 90 epochs using
the Adam [26] optimizer with a weight decay equal to $10^{-4}$. We apply
OneCycleLR learning rate scheduling with maximum learning rate of 0.01.

Table II compares the accuracy, model size, and speedup (in terms of latency)
of different DNNs trained on different datasets and quantized using a variety
of approaches.

TABLE II: Comparison of accuracy, model size, and speedup on different
datasets and DNN architectures

Dataset | Architecture | Approach | Accuracy | Model | Speedup
---|---|---|---|---|---
(W/A) | (%) | Size (MB)
ImageNet | ResNet-18 | Baseline (FiP16/FiP16) | 71.0 | 23.38 | 1.00$\times$
PACT [8] (3/3) | 68.1 | 4.38 | -
AutoQ [18] (4MP/4MP) | 68.2 | 5.80 | -
[14] (3MP/3MP) | 69.7 | 4.38 | -
EvoQ [15] (?MP/?MP) | 68.5 | 5.85 | -
Ours (2MP/2MP) | 70.8 | 4.01 | 10.90$\times$
MobileNetV2 | Baseline (FiP16/FiP16) | 72.6 | 6.8 | 1.00$\times$
PACT [8] (4/4) | 61.4 | - | 5.09$\times$
HAQ [16] (4MP/4MP) | 67.0 | - | 5.13$\times$
[27] (4/4) | 71.2 | 1.79 | -
EMQ [28] | 71.0 | 1.45 | -
EvoQ [15] (?MP/?MP) | 68.9 | 1.78 | -
Ours (2MP/2MP) | 72.0 | 1.50 | 7.74$\times$
ResNet-50 | Baseline (FiP16/FiP16) | 77.3 | 51.3 | 1.00$\times$
PACT [8] (3/3) | 75.3 | 9.17 | -
[27] (3/3) | 75.1 | 9.36 | -
HAQ [16] (3MP/3MP) | 75.3 | 9.22 | -
HAWQ [29] (?MP/?MP) | 75.5 | 7.96 | -
HAWQ-V2 [13] (?MP/?MP) | 75.8 | 7.99 | -
EMQ [28] (?MP/?MP) | 76.0 | 8.26 | -
EvoQ [15] (?MP/?MP) | 75.5 | 12.81 | -
[14] (3MP/4MP) | 76.9 | 7.97 | -
Ours (2MP/2MP) | 76.7 | 7.15 | 13.56$\times$
CIFAR-100 | ResNet-18 | Baseline (FiP16/FiP16) | 76.1 | 23.38 | 1.00$\times$
Ours (2MP/2MP) | 76.1 | 2.09 | 22.55$\times$
MobileNetv1 | Baseline (FiP16/FiP16) | 65.5 | 8.4 | 1.00$\times$
Ours (2MP/2MP) | 66.09 | 1.66 | 11.09$\times$
CIFAR-10 | ResNet-20 | Baseline (FiP16/FiP16) | 91.5 | 0.54 | 1.00$\times$
ReLeQ [17] | 91.38 | 0.101 | -
Ours | 91.9 | 0.088 | 11.14$\times$

####  IV-B1 ImageNet

ResNet-18 [1] The ResNet-18 model trained with bit-widths and layer-widths
returned by the search process achieves 70.8% validation accuracy with model
size reduced to 4.01MB and latency improved by a factor of 10.9 compared to a
16-bit fixed point baseline. The trained model outperforms PACT [8], AutoQ
[18], EvoQ [15], and the mixed-precision quantization presented in [14] in
terms of both accuracy and compression (latency numbers are not reported in
these references). Compared to the best prior work on ResNet-18 [14], our
ResNet-18 achieves 1.1% higher accuracy at about 9% smaller model size (please
note that [14] uses mixed-precision quantization with a minimum of three bits
per weight, but it does not report model size, so we underestimate their model
size by assuming all their weights are mapped to three bits).

MobileNetV2 [30] The $k$-means TPE can compress the MobileNetV2 model to only
1.5 MB while causing a 0.6% accuracy drop compared to the baseline floating
point model. These results advance state-of-the-art on the quantization of
MobileNetV2: the optimized MobileNetV2 model has an only 3% higher model size
compared to EMQ [28] while it achieves 1% higher accuracy on the difficult
ImageNet dataset.

ResNet-50 [1] The optimized ResNet-50 has a 7.15 MB model size while it only
causes a 0.6% accuracy degradation. To the best of our knowledge, our method
is the first to compress the network to this level with an acceptable accuracy
drop. The work in [14] achieves a similar accuracy drop but obtains a model
with 11.4% larger model size.

####  IV-B2 CIFAR-100

Using $k$-means TPE, we successfully compressed ResNet-18 and MobileNetV1 on
CIFAR-100 datasets by factors of 11.18$\times$ and 5.06$\times$, respectively,
without any loss in accuracy. The inference latency also experienced a
significant improvement, as summarized in Table II. Figure 4 showcases some of
the samples explored by the search engine for ResNet-18, alongside the best
configuration returned by the algorithm. The effective performance of
$k$-means TPE in discovering high-performing samples is evident.

Figure 4: The space for ResNet-18 compression and the output model

####  IV-B3 CIFAR-10

For the CIFAR-10 dataset, we conducted experiments with ResNet-20 and compared
our results to those of ReLeQ [17]. As demonstrated in Table II, $k$-means TPE
outperformed ReLeQ in terms of compressing to ultralow storage size while
preserving accuracy levels. Our approach showcased its superiority in
achieving highly efficient models without compromising performance.

To gain deeper insights into the efficiency of our $k$-means TPE search, we
conducted a comparison with the recent work BOMP-NAS [21], which achieved
state-of-the-art results in terms of GPU-hours of search and compression
level. As depicted in Table III, our ResNet-20 model achieved nearly the same
level of accuracy as BOMP-NAS, while being 31.5% smaller in size. Furthermore,
the search cost for $k$-means TPE was 9.23$\times$ less than that of BOMP-NAS.
Additionally, our ResNet-18 model was 40% more compressed than BOMP-NAS, with
a search time that was 14.63$\times$ less than the reported GPU-hours for
BOMP-NAS. These results highlight the significant advantages of our $k$-means
TPE approach in terms of search efficiency and model compression.

TABLE III: Comparison with BOMP-NAS

Dataset | Approach | Accuracy | Model Size | Speedup | Search Cost
---|---|---|---|---|---
(%) | (MB) |  | (GPU-hour)
CIFAR-10 | BOMP-NAS [21] | 88.67 | 0.076 | - | 12.00
Our ResNet-20 | 88.60 | 0.052 | 18.8$\times$ | 1.30
CIFAR-100 | BOMP-NAS [21] | 75.84 | 4.199 | - | 30.00
Our ResNet-18 | 76.10 | 2.090 | 8.5$\times$ | 2.05

Table IV shows some configurations found by $k$-means TPE. For each model and
dataset, the first and second rows of configurations contain the assigned bit-
width for each layer and the assigned layer-width scaling factor. As the table
demonstrates, to be able to quantize some layers to ultralow precisions (e.g.,
2 or 3 bits), the method may strategically decide to scale up the number of
filters in that layer. By doing so, the algorithm effectively mitigates
quantization errors, achieving a favorable trade-off between precision
reduction and layer-width scaling. This demonstrates the effectiveness of the
joint optimization of bit-widths and layer-widths through $k$-means TPE.

TABLE IV: Configurations returned by $k$-means TPE for representative DNN
architectures

Model | Dataset | Configuration
---|---|---
ResNet-18 | ImageNet | 8, 6, 6, 4, 4, 6, 6, 4, 4, 4, 4, 2, 2, 3, 3, 2, 2, 6
1.25, 1.25, 1.25, 1.25, 1.25, 0.875, 0.875, 0.875, 0.875, 1, 1, 1, 1, 1, 1, 1,
1, 1
ResNet-20 | CIFAR-10 | 8, 3, 3, 3, 3, 3, 3, 3, 3, 2, 2, 2, 2, 3, 3, 2, 2, 3, 3, 8
1, 1, 1, 1, 1, 1, 1, 0.75, 0.75, 0.75, 0.75, 0.75, 0.75, 0.75, 0.75, 0.75,
0.75, 0.75, 0.75, 0.75
MobileNetV1 | CIFAR-100 | 8, 8, 8, 6, 6, 4, 4, 4, 4, 4, 4, 4, 4, 3, 3, 3, 3, 4, 4, 3, 3, 4, 4, 3, 3, 2, 2, 2
1, 1.25, 1.25, 0.875, 0.875, 0.875, 0.875, 1.125, 1.125, 0.875, 0.875, 1.25,
1.25, 1, 1, 1.125, 1.125, 1.25, 1.25, 1.25, 1.25, 1.125, 1.125, 0.75, 0.75, 1,
1, 0.875

##  V Conclusions

We presented a search-based approach including a Hessian-based pruner and a
tree-structured dual-threshold Parzen estimator for automatic optimization of
DNNs bit-width and layer-width configurations to enable their efficient
deployment. Through extensive experiments on benchmark datasets, we showed a
$12\times$ average search time speedup and a 20% reduction in model size
compared to the state-of-the-art compression techniques while preserving the
model’s output accuracy. Such an advancement facilitates rapid model
development and deployment in resource-constrained environments, unlocking new
possibilities for scalable deep learning systems.

## References

  * [1] K. He, X. Zhang _et al._ , “Deep residual learning for image recognition,” in _Conference on Computer Vision and Pattern Recognition_ , 2016.
  * [2] S. Zagoruyko and N. Komodakis, “Wide residual networks,” in _British Machine Vision Conference_ , 2016.
  * [3] A. Dosovitskiy, L. Beyer _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” in _International Conference on Learning Representations_ , 2021.
  * [4] I. O. Tolstikhin, N. Houlsby _et al._ , “MLP-Mixer: An all-MLP architecture for vision,” in _Advances in Neural Information Processing Systems_ , 2021.
  * [5] A. Vaswani, N. Shazeer _et al._ , “Attention is all you need,” in _Advances in Neural Information Processing Systems_ , 2017.
  * [6] J. Devlin, M. Chang _et al._ , “BERT: pre-training of deep bidirectional transformers for language understanding,” in _Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2019.
  * [7] H. Touvron, T. Lavril _et al._ , “LLaMA: Open and efficient foundation language models,” _CoRR_ , vol. abs/2302.13971, 2023.
  * [8] J. Choi, Z. Wang _et al._ , “PACT: parameterized clipping activation for quantized neural networks,” _CoRR_ , vol. abs/1805.06085, 2018.
  * [9] S. Zhou, Z. Ni _et al._ , “DoReFa-Net: Training low bitwidth convolutional neural networks with low bitwidth gradients,” _CoRR_ , vol. abs/1606.06160, 2016.
  * [10] H. Li, A. Kadav _et al._ , “Pruning filters for efficient convnets,” in _International Conference on Learning Representations_ , 2017.
  * [11] J. Luo, J. Wu, and W. Lin, “ThiNet: A filter level pruning method for deep neural network compression,” in _International Conference on Computer Vision_ , 2017.
  * [12] J. Bergstra, R. Bardenet _et al._ , “Algorithms for hyper-parameter optimization,” in _Advances in Neural Information Processing Systems_ , 2011\.
  * [13] Z. Dong, Z. Yao _et al._ , “HAWQ-V2: Hessian aware trace-weighted quantization of neural networks,” in _Advances in Neural Information Processing Systems_ , 2020.
  * [14] C. Tang, K. Ouyang _et al._ , “Mixed-precision neural network quantization via learned layer-wise importance,” in _European Conference on Computer Vision_ , 2022.
  * [15] Y. Yuan, C. Chen _et al._ , “EvoQ: Mixed precision quantization of DNNs via sensitivity guided evolutionary search,” in _International Joint Conference on Neural Networks_ , 2020.
  * [16] K. Wang, Z. Liu _et al._ , “HAQ: hardware-aware automated quantization with mixed precision,” in _Conference on Computer Vision and Pattern Recognition_ , 2019.
  * [17] A. T. Elthakeb, P. Pilligundla _et al._ , “ReLeQ : A reinforcement learning approach for automatic deep quantization of neural networks,” _IEEE Micro_ , 2020.
  * [18] Q. Lou, F. Guo _et al._ , “AutoQ: Automated kernel-wise neural network quantization,” in _International Conference on Learning Representations_ , 2020.
  * [19] H. Liu, K. Simonyan, and Y. Yang, “DARTS: differentiable architecture search,” in _International Conference on Learning Representations_ , 2019\.
  * [20] Z. Cai and N. Vasconcelos, “Rethinking differentiable search for mixed-precision neural networks,” in _Conference on Computer Vision and Pattern Recognition_ , 2020.
  * [21] D. van Son, F. de Putter _et al._ , “BOMP-NAS: Bayesian optimization mixed precision NAS,” in _Design, Automation & Test in Europe Conference & Exhibition_, 2023.
  * [22] J. Bergstra, D. Yamins, and D. D. Cox, “Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures,” in _International Conference on Machine Learning_ , 2013\.
  * [23] X. Liu, Y. Chen _et al._ , “HiKonv: High throughput quantized convolution with novel bit-wise management and computation,” in _Asia and South Pacific Design Automation Conference_ , 2022.
  * [24] A. Paszke, S. Gross _et al._ , “PyTorch: An imperative style, high-performance deep learning library,” in _Advances in Neural Information Processing Systems_ , 2019.
  * [25] J. Deng, W. Dong _et al._ , “ImageNet: A large-scale hierarchical image database,” in _Computer Society Conference on Computer Vision and Pattern Recognition_ , 2009.
  * [26] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _International Conference on Learning Representations_ , 2015.
  * [27] S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural network with pruning, trained quantization and Huffman coding,” in _International Conference on Learning Representations_ , 2016.
  * [28] Z. Liu, X. Zhang _et al._ , “Evolutionary quantization of neural networks with mixed-precision,” in _International Conference on Acoustics, Speech and Signal Processing_ , 2021.
  * [29] Z. Dong, Z. Yao _et al._ , “HAWQ: Hessian aware quantization of neural networks with mixed-precision,” in _International Conference on Computer Vision_ , 2019.
  * [30] M. Sandler, A. G. Howard _et al._ , “MobileNetV2: Inverted residuals and linear bottlenecks,” in _Conference on Computer Vision and Pattern Recognition_ , 2018.


# The von Bahr-Esseen type inequality under sub-linear expectations and
applications††thanks: Supported by the National Natural Science Foundation of
China (12201079), the Natural Science Foundation of Anhui Province
(2108085MA06, 2108085QA15), and the Excellent Scientific Research and
Innovation Team of Anhui Colleges (2022AH010098).

Yi Wua, Xuejun Wangb
a School of Big Data and Artificial Intelligence, Chizhou University, Chizhou,
247000, P.R. China
b School of Big Data and Statistics, Anhui University, Hefei, 230601, P.R.
China Corresponding author. Email address<EMAIL_ADDRESS>
###### Abstract

Moment inequalities play important roles in probability limit theory and
mathematical statistics. In this work, the von Bahr-Esseen type inequality for
extended negatively dependent random variables under sub-linear expectations
is established successfully. By virtue of the inequality, we further obtain
the Kolmogorov type weak law of large numbers for partial sums and the
complete convergence for weighted sums, which extend and improve corresponding
results in sub-linear expectation space.

Keywords: von Bahr-Esseen type inequality; weak law of large numbers; complete
convergence; sub-linear expectations

Mathematics Subject Classification 2020: 60E15; 60F15

##  1 Introduction

It is known that the moment inequalities are very powerful tools in classical
probability theory and statistics. Due to these inequalities, many elegant
results in probability theory and statistical applications were established
successively in the past decades. For a sequence $\\{X_{n},n\geq 1\\}$ of
independent and identically distributed random variables with mean zero and
$E|X_{n}|^{p}<\infty$ for some $1\leq p\leq 2$, von Bahr and Esseen (1965)
proved that, for each $n\geq 1$,

$E\left|\sum_{i=1}^{n}X_{i}\right|^{p}\leq C_{p}\sum_{i=1}^{n}E|X_{i}|^{p},$
(1)

where $C_{p}$ is a positive constant depending only on $p$. The aforementioned
inequality is called the $p$-th von Bahr-Esseen moment inequality. In view of
the significance of this powerful tool, many scholars generalized it to
various dependence settings. For example, Chatterji (1969) showed that (1)
holds for martingale difference sequence; Bryc and Smolenski (1993) proved it
for weakly correlated random variables; Shao (2000) extended it to negatively
associated random variables; Asadian et al. (2006) established (1) for
negatively orthant dependent random variables; Wang et al. (2014) generalized
it to widely orthant dependent random variables; Chen et al. (2014)
innovatively proved (1) with $1<p<2$ by that with $p=2$ for pairwise
independent as well as pairwise negatively quadrant dependent random
variables. Later on, Wu et al. (2021) obtained (1) for asymptotic negatively
associated random variables as well as $m$-asymptotic negatively associated
random variables by adopting the method in Chen et al. (2014).

Most of moment inequalities including (1) are based on the linearity of
expectations and probability measures. However, such an additivity assumption
is not feasible in many areas of applications because many uncertain phenomena
cannot be well modelled by using additive probabilities or additive
expectations.

In recent years, the sub-linear expectations receive a lot of attention from
statisticians. There are many entitative differences between the classical
probability space and the sub-linear expectation space. For example, in the
classical probability space, the probability measures and mathematical
expectations are always assumed to be additive. The linear expectation and
determinant statistics are built on the distribution-certainty or model-
certainty. However, in practical fields such as risk measure and super-hedging
in finance, this assumption is not always plausible. For related references,
we refer the readers to El Karoui et al. (1997), Artzner et al. (1997), Chen
and Epstein (2002), and Föllmer and Schied (2004) among others. Without
distribution-certainty, the resulting expectation is usually nonlinear.
Therefore, the theory and methodology of nonlinear expectation have been well
developed and received much attention in some application fields such as
finance risk measure and control in the past decades. A typical example of the
nonlinear expectation, called $g$-expectation, was introduced by Peng (1997)
in the framework of backward stochastic differential equations. Peng (2006)
proposed the $G$-expectation and its related versions as a further
development. For more details about the applications of the nonlinear
expectation, one can refer to Denis and Martini (2006), Gilboa (1987),
Marinacci (1999), and Peng (1999, 2008a) for instance. Peng (2006, 2008a,
2008b) firstly introduced a reasonable framework of the sub-linear expectation
in a general function space by relaxing the linear property of the classical
expectation to the sub-additivity and positive homogeneity.

Let $(\Omega,\mathcal{F})$ be a given measurable space and $\mathcal{H}$ be a
linear space of real functions defined on $(\Omega,\mathcal{F})$ such that if
$X_{1},\cdots,X_{n}\in\mathcal{H}$ then
$\varphi(X_{1},\cdots,X_{n})\in\mathcal{H}$ for each $\varphi\in
C_{l,Lip}(\mathbb{R}^{n})$, where $C_{l,Lip}(\mathbb{R}^{n})$ denotes the
linear space of (local Lipschitz) functions $\varphi$ satisfying

$\displaystyle|\varphi(\mbox{\boldmath{$x$}})-\varphi(\mbox{\boldmath{$y$}})|\leq
C(1+|\mbox{\boldmath{$x$}}|^{m}+|\mbox{\boldmath{$y$}}|^{m})|\mbox{\boldmath{$x$}}-\mbox{\boldmath{$y$}}|,\textrm{
for any }\mbox{\boldmath{$x$}},\mbox{\boldmath{$y$}}\in\mathbb{R}^{n},$

for some $C>0$, $m\in\mathbb{N}$ depending on $\varphi$. $\mathcal{H}$ can be
considered as a space of “random variables”. In this case we denote
$X\in\mathcal{H}$. We also denote $C_{b,Lip}(\mathbb{R}^{n})$ to be the space
of bounded Lipschitz functions $\varphi$ satisfying

$\displaystyle|\varphi(\mbox{\boldmath{$x$}})|\leq
C,~{}~{}|\varphi(\mbox{\boldmath{$x$}})-\varphi(\mbox{\boldmath{$y$}})|\leq
C|\mbox{\boldmath{$x$}}-\mbox{\boldmath{$y$}}|,\textrm{ for any
}\mbox{\boldmath{$x$}},\mbox{\boldmath{$y$}}\in\mathbb{R}^{n},$

for some $C>0$ depending on $\varphi$.

Definition 1.1. A sub-linear expectation $\hat{\mathbb{E}}$ on $\mathcal{H}$
is a function $\hat{\mathbb{E}}:\mathcal{H}\rightarrow\bar{\mathbb{R}}$
satisfying the following properties: for all $X,Y\in\mathcal{H}$, we have

(a) Monotonicity: If $X\geq Y$, then
$\hat{\mathbb{E}}[X]\geq\hat{\mathbb{E}}[Y]$;

(b) Constant preserving: $\hat{\mathbb{E}}[c]=c$, where $c$ is a constant;

(c) Sub-additivity:
$\hat{\mathbb{E}}[X+Y]\leq\hat{\mathbb{E}}[X]+\hat{\mathbb{E}}[Y]$ whenever
$\hat{\mathbb{E}}[X]+\hat{\mathbb{E}}[Y]$ is not the form $+\infty-\infty$ or
$-\infty+\infty$;

(d) Positive homogeneity: $\hat{\mathbb{E}}[\lambda
X]=\lambda\hat{\mathbb{E}}[X]$, $\lambda>0$.

Here $\bar{\mathbb{R}}=[-\infty,+\infty]$. The triple
$(\Omega,\mathcal{H},\hat{\mathbb{E}})$ is called a sub-linear expectation
space. For a given sub-linear expectation $\hat{\mathbb{E}}$, the conjugate
expectation $\hat{\varepsilon}$ of $\hat{\mathbb{E}}$ is denoted by

$\displaystyle\text{\Large$\hat{\varepsilon}$}[X]:=-\hat{\mathbb{E}}[-X],\textrm{
for any }X\in\mathcal{H}.$

It is easy to verify that $\hat{\varepsilon}$$[X]\leq\hat{\mathbb{E}}[X]$,
$\hat{\mathbb{E}}[X+c]=\hat{\mathbb{E}}[X]+c$ and
$\hat{\mathbb{E}}[X-Y]\geq\hat{\mathbb{E}}[X]-\hat{\mathbb{E}}[Y]$ for all
$X,Y\in\mathcal{H}$ with $\hat{\mathbb{E}}[Y]$ being finite. Furthermore, if
$\hat{\mathbb{E}}[|X|]$ is finite, then both $\hat{\varepsilon}$$[X]$ and
$\hat{\mathbb{E}}[X]$ are finite.

Let $(\Omega,\mathcal{H},\hat{\mathbb{E}})$ be a sub-linear expectation space.
Peng (2010) initiated and proved that there exists a family of finitely
additive probabilities $\\{P_{\theta}\\}_{\theta\in\Theta}$ defined on
$(\mathbb{R},\mathcal{B}(\mathbb{R}))$ such that for each $X\in\mathcal{H}$,

$\displaystyle\hat{\mathbb{E}}[X]=\sup_{\theta\in\Theta}E_{P_{\theta}}[X].$

Let $\mathcal{P}$ be a family of probability measures defined on the
measurable space ($\Omega,\mathcal{F}$). Define an upper expectation
$\hat{\mathbb{E}}[\cdot]$ by
$\hat{\mathbb{E}}[X]=\sup_{P\in\mathcal{P}}E_{P}[X],$ where $E_{P}$ denotes
the classical expectation under probability $P$. Then $\hat{\mathbb{E}}$ is a
sub-linear expectation.

Peng (2008b) put forward the definitions of independence and identical
distribution through the sub-linear expectation. A random vector
$\mbox{\boldmath{$Y$}}=(Y_{1},\cdots,Y_{n})$, $Y_{i}\in\mathcal{H}$ is said to
be independent to another random vector
$\mbox{\boldmath{$X$}}=(X_{1},\cdots,X_{n})$, $X_{i}\in\mathcal{H}$ under
$\hat{\mathbb{E}}$, if for each test function $\varphi\in
C_{l,Lip}(\mathbb{R}^{m}\times\mathbb{R}^{n})$,

$\hat{\mathbb{E}}[\varphi(\mbox{\boldmath{$X$}},\mbox{\boldmath{$Y$}})]=\hat{\mathbb{E}}\left[\hat{\mathbb{E}}[\varphi(\mbox{\boldmath{$x$}},\mbox{\boldmath{$Y$}})]|_{\mbox{\boldmath{$x$}}=\mbox{\boldmath{$X$}}}\right]$
(2)

whenever
$\bar{\varphi}(\mbox{\boldmath{$x$}}):=\hat{\mathbb{E}}[|\varphi(\mbox{\boldmath{$x$}},\mbox{\boldmath{$Y$}})|]<\infty$
for all $x$ and
$\hat{\mathbb{E}}[|\bar{\varphi}(\mbox{\boldmath{$X$}})|]<\infty$. A sequence
$\\{X_{n},n\geq 1\\}$ of random variables is said to be independent if

$\displaystyle\hat{\mathbb{E}}[\varphi(X_{1},\cdots,X_{n},X_{n+1})]=\hat{\mathbb{E}}\left[\hat{\mathbb{E}}[\varphi(x_{1},\cdots,x_{n},X_{n+1})]|_{x_{1}=X_{1},\cdots,x_{n}=X_{n}}\right]$

for all $n\geq 1$ and $\varphi\in C_{l.Lip}(\mathbb{R}^{n+1})$.

Let $X_{1}$ and $X_{2}$ be two $n$-dimensional random vectors defined,
respectively in sub-linear expectation spaces
$(\Omega_{1},\mathcal{H}_{1},\hat{\mathbb{E}}_{1})$ and
$(\Omega_{2},\mathcal{H}_{2},\hat{\mathbb{E}}_{2})$. They are called
identically distributed, if

$\hat{\mathbb{E}}_{1}[\varphi(\mbox{\boldmath{$X_{1}$}})]=\hat{\mathbb{E}}_{2}[\varphi(\mbox{\boldmath{$X_{2}$}})],\textrm{
for any }\varphi\in C_{l,Lip}(\mathbb{R}^{n}),$

whenever the upper expectations are finite. Under this framework, many
extensions of dependence structures and limit theorems have been investigated
successively. We refer the readers to Peng (2008b, 2010) for the central limit
theorem and weak law of large numbers; Chen and Hu (2014), and Zhang (2016a)
for the law of the iterated algorithm; Zhang (2015) for the small derivation
and Chung’s law of the iterated logarithm; Zhang (2016b) for the the moment
inequalities of the maximum partial sums and the Kolomogov type strong law of
large numbers; Wu and Jiang (2018) for strong law of large numbers and
Chover’s law of the iterated logarithm; Wu et al. (2018) for the asymptotic
approximation of inverse moment; Tang et al. (2019) for some exponential
inequalities; Xi et al. (2019) for complete convergence and its application in
nonparametric regression model; Zhang et al. (2020) for some results on
convergence of the sums of independent random variables; Kuczmaszewska (2020)
for complete convergence; Wu and Jiang (2020) for complete convergence and
complete moment convergence of negatively dependent random variables, and so
on. Since many basic properties or powerful tools such as various inequalities
in classical probability space are no longer available in sub-linear
expectation space, the study of limit theorems under sub-linear expectations
is still a challenging work.

Zhang (2022) introduced the concept of extended negative dependence as
follows.

Definition 1.2. In a sub-linear expectation space
$(\Omega,\mathcal{H},\hat{\mathbb{E}})$, a sequence $\\{X_{n},n\geq 1\\}$ of
random variables is said to be upper (resp. lower) extended negatively
dependent if there exists some dominating constant $K\geq 1$ such that

$\displaystyle\hat{\mathbb{E}}\left[\prod_{i=1}^{n}f_{i}(X_{i})\right]\leq
K\prod_{i=1}^{n}\hat{\mathbb{E}}[f_{i}(X_{i})],\textrm{ for any }n\geq 1,$

whenever the non-negative functions $f_{i}\in C_{b,Lip}(\mathbb{R})$,
$i=1,2,\ldots$, are all non-decreasing (resp. all non-increasing). They are
called extended negatively dependent (END, for short) if they are both upper
extended negatively dependent and lower extended negatively dependent.

Next, we recall the capacities corresponding to the sub-linear expectations.
Let $\mathcal{G}\subset\mathcal{F}$. A function
$V:\mathcal{G}\rightarrow[0,1]$ is called a capacity if

$\displaystyle V(\phi)=0,V(\Omega)=1,~{}and~{}V(A)\leq V(B),\textrm{ for any
}A\subset B,~{}A,B\in\mathcal{G}.$

It is said to be sub-additive if $V(A\bigcup B)\leq V(A)+V(B)$ for all
$A,B\in\mathcal{G}$ with $A\bigcup B\in\mathcal{G}$.

In the sub-linear space $(\Omega,\mathcal{H},\hat{\mathbb{E}})$, a pair
$(\mathbb{V},\mathcal{V})$ of capacities can be denoted by

$\displaystyle\mathbb{V}(A)=\inf\\{\hat{\mathbb{E}}[\xi]:I_{A}\leq\xi,\xi\in\mathcal{H}\\},\mathcal{V}(A)=1-\mathbb{V}(A^{c}),\textrm{
for }A\in\mathcal{F},$

where $A^{c}$ is the complement set of $A$. Obviously, $\mathbb{V}$ is sub-
additive, and

$\displaystyle\hat{\mathbb{E}}[f]\leq\mathbb{V}(A)\leq\hat{\mathbb{E}}[g],~{}\text{\Large$\hat{\varepsilon}$}[f]\leq\mathcal{V}(A)\leq\text{\Large$\hat{\varepsilon}$}[g],\textrm{
if }f\leq I_{A}\leq g\textrm{ and }f,g\in\mathcal{H}.$

The inequalities above can conclude Markov’s inequality in the same form of
the classical probability space since $I(|X|\geq
x)\leq|X|^{p}/x^{p}\in\mathcal{H}$. Also, the Choquet integrals/expecations
$(C_{\mathbb{V}},C_{\mathcal{V}})$ can be defined by

$\displaystyle C_{V}[X]=\int_{0}^{\infty}V(X\geq
t)dt+\int_{-\infty}^{0}[V(X\geq t)-1]dt$

with $V$ being replaced by $\mathbb{V}$ and $\mathcal{V}$, respectively.

As stated above, the study under sub-linear expectations is quite challenging,
one reason for which is the lack of powerful tools. For a sequence
$\\{X_{n},n\geq 1\\}$ of random variables where $X_{i}$ is negatively
dependent to $(X_{i+1},\ldots,X_{n})$ for each $i=1,\ldots,n-1$, Zhang (2016b)
proved that for any $1\leq p\leq 2$,

$\hat{\mathbb{E}}\left[\left|\max_{1\leq k\leq
n}\sum_{i=1}^{k}X_{i}\right|^{p}\right]\leq
2^{2-p}\sum_{i=1}^{n}\hat{\mathbb{E}}[|X_{i}|^{p}],$

provided $\hat{\mathbb{E}}[X_{n}]\leq 0$ for each $n\geq 1$. However, as
pointed out in Zhang (2022), both the independence in (2) and the negative
dependence raised in Zhang (2016b) have directions, i.e., $Y$ is independent
(resp. negatively dependent) relative to $X$ does not imply that $X$ is
independent relative to $Y$ (resp. negatively dependent). The two concepts are
restrictive and difficult to verify, that is why Zhang (2022) put forward the
concepts of extended independence and extended negative dependence. Suppose
that $\\{X_{n},n\geq 1\\}$ is a sequence of END random variables under sub-
linear expectations with $\hat{\mathbb{E}}[X_{n}]\leq 0$ for each $n\geq 1$,
Zhang (2022) established the following inequality

$\mathbb{V}\left(\sum_{i=1}^{n}X_{i}>x\right)\leq(1+Ke)x^{-2}\sum_{i=1}^{n}\hat{\mathbb{E}}[X_{i}^{2}].$
(3)

It is not known whether the von Bahr-Esseen inequality (1) holds for this
dependence structure under sub-linear expectations since the concept of
extended negative dependence is quite different from negative dependence. Our
intention is that, maybe the exponent in (3) can also be extended from 2 to
$1<p<2$. Of course, it holds trivially for $0<p\leq 1$. However, it is also a
quite challenging work. First, the method used in Chen et al. (2014) does not
work since the expression $E|X|=\int_{0}^{\infty}P(|X|>t)dt$ is unavailable
for the sub-linear expectation $\hat{\mathbb{E}}[|X|]$. Second, if we adopt
the method in Zhang (2022), the monotony of the function
$F(x)=\frac{e^{x}-1-x}{|x|^{p}}$ can not be guaranteed for $1<p<2$. Therefore,
in this work, we will proceed it with two steps. The von Bahr-Esseen type
inequality is established for bounded END random variables by a function
concerning the hyperbolic cosine borrowed from Hu et al. (2015), and then it
is proved for unbounded END random variables under sub-linear expectations. By
virtue of the inequality, we obtain the Kolmogorov type weak law of large
numbers for partial sums of END random variables. Another application concerns
two results on complete convergence for weighted sums, which improve some
existing ones in sub-linear expectation space.

Throughout this paper, let $C$ be a positive constant not depending on $n$,
which may not be the same in various places. $a_{n}=O(b_{n})$ means $a_{n}\leq
Cb_{n}$ for all $n\geq 1$. Denote $x_{+}=xI(x>0)$, where $I(A)$ is the
indicator function of the event $A$.

##  2 Main results

In this section, we first establish the von Bahr-Esseen type inequality for
bounded END random variables under sub-linear expectations as follows.

Theorem 2.1. Let $1<p\leq 2$ and $\epsilon>0$. If $\\{Z_{n},n\geq 1\\}$ is a
sequence of upper END random variables in
$(\Omega,\mathcal{H},\hat{\mathbb{E}})$ with $\hat{\mathbb{E}}[Z_{n}]\leq 0$
and $|Z_{n}|\leq\epsilon$ for each $n\geq 1$, then

$\displaystyle\mathbb{V}\left(\sum_{i=1}^{n}Z_{i}>\epsilon\right)\leq
Ke\epsilon^{-p}\sum_{i=1}^{n}\hat{\mathbb{E}}[|Z_{i}|^{p}].$ (4)

If $\\{Z_{n},n\geq 1\\}$ is a sequence of END random variables in
$(\Omega,\mathcal{H},\hat{\mathbb{E}})$ with
$\hat{\mathbb{E}}[Z_{n}]=\text{\Large$\hat{\varepsilon}$}[Z_{n}]=0$ and
$|Z_{n}|\leq\epsilon$ for each $n\geq 1$, then

$\displaystyle\mathbb{V}\left(\left|\sum_{i=1}^{n}Z_{i}\right|>\epsilon\right)\leq
2Ke\epsilon^{-p}\sum_{i=1}^{n}\hat{\mathbb{E}}[|Z_{i}|^{p}].$ (5)

Applying Theorem 2.1, the von Bahr-Esseen type inequality for unbounded END
random variables under sub-linear expectations can be obtained as follows.

Theorem 2.2. Let $1<p\leq 2$. If $\\{X_{n},n\geq 1\\}$ is a sequence of upper
END random variables in $(\Omega,\mathcal{H},\hat{\mathbb{E}})$ with
$\hat{\mathbb{E}}[X_{n}]\leq 0$ for each $n\geq 1$, then for any $x>0$,

$\displaystyle\mathbb{V}\left(\sum_{i=1}^{n}X_{i}>x\right)\leq
4^{p}(1+Ke)x^{-p}\sum_{i=1}^{n}\hat{\mathbb{E}}[|X_{i}|^{p}].$ (6)

If $\\{X_{n},n\geq 1\\}$ is a sequence of END random variables in
$(\Omega,\mathcal{H},\hat{\mathbb{E}})$ with
$\hat{\mathbb{E}}[X_{n}]=\text{\Large$\hat{\varepsilon}$}[X_{n}]=0$ for each
$n\geq 1$, then for any $x>0$,

$\displaystyle\mathbb{V}\left(\left|\sum_{i=1}^{n}X_{i}\right|>x\right)\leq
2^{2p+1}(1+Ke)x^{-p}\sum_{i=1}^{n}\hat{\mathbb{E}}[|X_{i}|^{p}].$ (7)

By virtue of Theorem 2.2, we can obtain the following Kolmogorov type weak law
of large numbers for END random variables under sub-linear expectations.

Theorem 2.3. Let $\\{X_{n},n\geq 1\\}$ be a sequence of END random variables
in $(\Omega,\mathcal{H},\hat{\mathbb{E}})$. If
$\lim_{c\rightarrow\infty}\sup_{i\geq 1}\hat{\mathbb{E}}[(|X_{i}|-c)_{+}]=0$,
then for any $\varepsilon>0$,

$\displaystyle\mathbb{V}\left(\left\\{\frac{\sum_{i=1}^{n}(X_{i}-\text{\Large$\hat{\varepsilon}$}[X_{i}])}{n}<-\varepsilon\right\\}\bigcup\left\\{\frac{\sum_{i=1}^{n}(X_{i}-\hat{\mathbb{E}}[X_{i}])}{n}>\varepsilon\right\\}\right)\rightarrow
0\textrm{ as }n\rightarrow\infty.$ (8)

Moreover, if $\hat{\mathbb{E}}[X_{n}]=\bar{\mu}$ and
$\text{\Large$\hat{\varepsilon}$}[X_{n}]=\underline{\mu}$, then

$\displaystyle\mathcal{V}\left(\underline{\mu}-\varepsilon\leq\frac{\sum_{i=1}^{n}X_{i}}{n}\leq\bar{\mu}+\varepsilon\right)\rightarrow
1\textrm{ as }n\rightarrow\infty.$

Remark 2.1. Zhang (2022) obtained the result where $\\{X_{n},n\geq 1\\}$ is
identically distributed. Hence, Theorem 2.3 extends the corresponding result
of Zhang (2022) to the case of non-identical distribution.

Recall that a real valued function $l(x)$, positive and measurable on
$\mathbb{R}^{+}$, is said to be slowly varying if for each $\lambda>0$,

$\lim_{x\rightarrow\infty}\frac{l(\lambda x)}{l(x)}=1.$

Another application of Theorem 2.2 is the following result on complete
convergence for weighted sums of END random variables.

Theorem 2.4. Let $0<p<2$, $\alpha>0$ and $\alpha p>1$. Let $\\{X;X_{n},n\geq
1\\}$ be a sequence of identically distributed END random variables defined on
$(\Omega,\mathcal{H},\hat{\mathbb{E}})$. Assume further that
$\hat{\mathbb{E}}[X]=0$ and $\hat{\mathbb{E}}[|X|^{p}]<\infty$ if $p>1$. Let
$l(x)$ be a slowly varying function and $\\{a_{ni},1\leq i\leq n,n\geq 1\\}$
be an array of nonnegative real numbers satisfying
$\sum_{i=1}^{n}a_{ni}^{q}=O(n)$ for some $q>p$. If
$C_{\mathbb{V}}\left[|X|^{p}l(|X|^{1/\alpha})\right]<\infty$, then for any
$\varepsilon>0$,

$\displaystyle\sum_{n=1}^{\infty}n^{\alpha
p-2}l(n)\mathbb{V}\left(\sum_{i=1}^{n}a_{ni}X_{i}>\varepsilon
n^{\alpha}\right)<\infty.$ (9)

If we further assume $\text{\Large$\hat{\varepsilon}$}[X]=0$ when $p>1$, then
for any $\varepsilon>0$,

$\displaystyle\sum_{n=1}^{\infty}n^{\alpha
p-2}l(n)\mathbb{V}\left(\left|\sum_{i=1}^{n}a_{ni}X_{i}\right|>\varepsilon
n^{\alpha}\right)<\infty.$ (10)

For $\alpha p=1$, we can obtain the following conclusion under some slightly
different assumptions.

Theorem 2.5. Let $0<p<2$ and $\\{X;X_{n},n\geq 1\\}$ be a sequence of
identically distributed END random variables defined on
$(\Omega,\mathcal{H},\hat{\mathbb{E}})$. Assume further that
$\hat{\mathbb{E}}[X]=0$ and
$\lim_{c\rightarrow\infty}\hat{\mathbb{E}}[(|X|^{p}-c)_{+}]=0$ if $p\geq 1$.
Let $l(x)$ be a slowly varying function and $\\{a_{ni},1\leq i\leq n,n\geq
1\\}$ be an array of nonnegative real numbers satisfying
$\sum_{i=1}^{n}a_{ni}^{q}=O(n)$ for some $q>p$. If
$C_{\mathbb{V}}\left[|X|^{p}l(|X|^{p})\right]<\infty$, then for any
$\varepsilon>0$,

$\displaystyle\sum_{n=1}^{\infty}n^{-1}l(n)\mathbb{V}\left(\sum_{i=1}^{n}a_{ni}X_{i}>\varepsilon
n^{1/p}\right)<\infty.$ (11)

If we further assume $\text{\Large$\hat{\varepsilon}$}[X]=0$ when $p\geq 1$,
then for any $\varepsilon>0$,

$\displaystyle\sum_{n=1}^{\infty}n^{-1}l(n)\mathbb{V}\left(\left|\sum_{i=1}^{n}a_{ni}X_{i}\right|>\varepsilon
n^{1/p}\right)<\infty.$

Remark 2.2. Zhong and Wu (2017) also obtained the corresponding results on
complete convergence for weighted sums. It is easy to see that the assumption
$\sum_{i=1}^{n}a_{ni}^{2}=O(n)$ is improved to $\sum_{i=1}^{n}a_{ni}^{q}=O(n)$
for some $q>p$. Moreover, they assumed that $l(x)$ is a monotone nondecreasing
function which is not needed here. For $\alpha p>1$ and $1<p<2$, the
hypothesis $\hat{\mathbb{E}}[|X|^{p}l(|X|^{1/\alpha})]\leq
C_{\mathbb{V}}\left[|X|^{p}l(|X|^{1/\alpha})\right]$ in their result is
stronger than $\hat{\mathbb{E}}[|X|^{p}]<\infty$ in Theorem 2.4. Therefore,
our theorems improve the related results in Zhong and Wu (2017).

##  3 Some lemmas

In this section, we will present some lemmas which are essential in proving
the results on complete convergence. The first lemma concerning the slowly
varying function can be seen in Bai and Su (1985).

Lemma 3.1. If $l(x)$ is a slowly varying function, then

(i) $\lim_{k\rightarrow\infty}\sup_{2^{k}\leq
x<2^{k+1}}\frac{l(x)}{l(2^{k})}=1$;

(ii) for every $r>0$, $C_{1}2^{kr}l(2^{k})\leq\sum_{j=1}^{k}2^{jr}l(2^{j})\leq
C_{2}2^{kr}l(2^{k})$;

(iii) for every $r<0$,
$C_{1}2^{kr}l(2^{k})\leq\sum_{j=k}^{\infty}2^{jr}l(2^{j})\leq
C_{2}2^{kr}l(2^{k})$.

The following lemma comes from Zhong and Wu (2017).

Lemma 3.2. Suppose that $X\in\mathcal{H}$, $p>0$, and $\alpha>0$, $l(x)$ is a
slowly varying function.

(i) For any $c_{0}>0$,

$C_{\mathbb{V}}\left[|X|^{p}l(|X|^{1/\alpha})\right]<\infty\Leftrightarrow\sum_{n=1}^{\infty}n^{\alpha
p-1}l(n)\mathbb{V}(|X|>c_{0}n^{\alpha})<\infty.$

(ii) If $C_{\mathbb{V}}\left[|X|^{p}l(|X|^{1/\alpha})\right]<\infty$, then for
any $\theta>1$ and $c_{0}>0$,

$\sum_{k=1}^{\infty}\theta^{k\alpha
p}l(\theta^{k})\mathbb{V}(|X|>c_{0}\theta^{k\alpha})<\infty.$

Since the discontinuous function $I(x)\notin C_{l,Lip}(\mathbb{R})$, it should
be transformed under the sub-linear expectations. Therefore, before presenting
the last lemma, some notations are indispensable.

For $0<\mu<1$, let $g(x)\in C_{l,Lip}(\mathbb{R})$ be an even and non-
increasing function on $[0,\infty)$ such that $0\leq g(x)\leq 1$ for all $x$
and $g(x)=1$ if $|x|\leq\mu$, $g(x)=0$ if $|x|>1$. Hence,

$\displaystyle I(|x|\leq\mu)\leq g(x)\leq I(|x|\leq 1),~{}~{}I(|x|>1)\leq
1-g(x)\leq I(|x|>\mu).$ (12)

Let $g_{j}(x)\in C_{l,Lip}(\mathbb{R}),~{}j\geq 1$ such that $0\leq
g_{j}(x)\leq 1$ for all $x$ and $g_{j}(\frac{x}{2^{j\alpha}})=1$ if
$2^{(j-1)\alpha}<|x|\leq 2^{j\alpha}$, $g_{j}(\frac{x}{2^{j\alpha}})=0$ if
$|x|\leq\mu 2^{(j-1)\alpha}$ or $|x|>(1+\mu)2^{j\alpha}$. Thus,

$\displaystyle g_{j}\left(\frac{x}{2^{j\alpha}}\right)\leq I\left(\mu
2^{(j-1)\alpha}<|x|\leq(1+\mu)2^{j\alpha}\right),$ (13)

and

$\displaystyle|x|^{t}g\left(\frac{x}{2^{k\alpha}}\right)\leq
1+\sum_{j=1}^{k}|x|^{t}g_{j}\left(\frac{x}{2^{j\alpha}}\right),\textrm{ for
any }t>0.$ (14)

Lemma 3.3. Suppose that $X\in\mathcal{H}$, $0<\mu<1$, $\alpha>0$ and $p>0$. If
$C_{\mathbb{V}}\left[|X|^{p}l(|X|^{1/\alpha})\right]<\infty$, then for any
$s>p$,

$\sum_{n=1}^{\infty}n^{\alpha p-\alpha
s-1}l(n)\hat{\mathbb{E}}\left[|X|^{s}g\left(\frac{\mu
X}{n^{\alpha}}\right)\right]<\infty.$

Proof. It follows from (13), (14), Lemmas 3.1 and 3.2 that

$\displaystyle\sum_{k=0}^{\infty}\sum_{2^{k}\leq n<2^{k+1}}n^{\alpha p-\alpha
s-1}l(n)\hat{\mathbb{E}}\left[|X|^{s}g\left(\frac{\mu
X}{n^{\alpha}}\right)\right]$ $\displaystyle\leq$
$\displaystyle\sum_{k=0}^{\infty}2^{\alpha(p-s)k}l(2^{k})\hat{\mathbb{E}}\left[|X|^{s}g\left(\frac{\mu
X}{2^{\alpha(k+1)}}\right)\right]$ $\displaystyle\leq$ $\displaystyle
C\sum_{k=0}^{\infty}2^{\alpha(p-s)k}l(2^{k})\hat{\mathbb{E}}\left[1+\sum_{j=0}^{k}|X|^{s}g_{j+1}\left(\frac{\mu
X}{2^{\alpha(j+1)}}\right)\right]$ $\displaystyle\leq$ $\displaystyle
C\sum_{k=0}^{\infty}2^{\alpha(p-s)k}l(2^{k})+C\sum_{k=0}^{\infty}2^{\alpha(p-s)k}l(2^{k})\sum_{j=0}^{k}\hat{\mathbb{E}}\left[|X|^{s}g_{j+1}\left(\frac{\mu
X}{2^{\alpha(j+1)}}\right)\right]$ $\displaystyle\leq$ $\displaystyle
C+C\sum_{j=0}^{\infty}\hat{\mathbb{E}}\left[|X|^{s}g_{j+1}\left(\frac{\mu
X}{2^{\alpha(j+1)}}\right)\right]\sum_{k=j}^{\infty}2^{\alpha(p-s)k}l(2^{k})$
$\displaystyle\leq$ $\displaystyle C+C\sum_{j=0}^{\infty}2^{\alpha
pj}l(2^{j})\mathbb{V}(|X|>\mu^{2}2^{j\alpha})<\infty.$

The proof is completed. $\Box$

##  4 The proofs of the main results

Proof of Theorem 2.1. Note that for any $t>0$, the non-negative function
$f(x)=e^{tx}\in C_{b,Lip}(\mathbb{R})$ since $f^{\prime}(x)=te^{tx}\leq
te^{t\epsilon}$ for any $|x|\leq\epsilon$. It follows from Markov’s inequality
and Definition 1.2 that

$\displaystyle\mathbb{V}\left(\sum_{i=1}^{n}Z_{i}>\epsilon\right)\leq
e^{-t\epsilon}\hat{\mathbb{E}}\left[\exp\left(t\sum_{i=1}^{n}Z_{i}\right)\right]=e^{-t\epsilon}\hat{\mathbb{E}}\left[\prod_{i=1}^{n}e^{tZ_{i}}\right]\leq
Ke^{-tx}\prod_{i=1}^{n}\hat{\mathbb{E}}[e^{tZ_{i}}].$ (15)

Define the function $H(x):\mathbb{R}\rightarrow\mathbb{R}$ by

$H(x)=\begin{cases}{{\cosh x-1}\over{|x|^{p}}},&if~{}x\neq 0,\\\
{{1}\over{2}},&if~{}x=0~{}and~{}p=2,\\\ 0,&if~{}x=0~{}and~{}0<p<2.\end{cases}$

It is easy to verify that $H(x)$ is a nonnegative, continuous, even function,
and is increasing when $x\geq 0$. Thus, for all $|Z_{i}|\leq\epsilon$, we have
$H(tZ_{i})\leq H(t\epsilon)$, $1\leq i\leq n$. Therefore, we obtain by
$e^{x}-1-x\leq 2(\cosh x-1)$ for any $x\in\mathbb{R}$ that

$\displaystyle e^{tZ_{i}}$ $\displaystyle=$ $\displaystyle
1+tZ_{i}+\frac{e^{tZ_{i}}-1-tZ_{i}}{|tZ_{i}|^{p}}|tZ_{i}|^{p}$ (16)
$\displaystyle\leq$ $\displaystyle
1+tZ_{i}+\frac{2\big{(}\cosh(tZ_{i})-1\big{)}}{|tZ_{i}|^{p}}|tZ_{i}|^{p}$
$\displaystyle\leq$ $\displaystyle
1+tZ_{i}+\frac{2\big{(}\cosh(t\epsilon)-1\big{)}}{\epsilon^{p}}|Z_{i}|^{p}.$

Note that $\hat{\mathbb{E}}[Z_{i}]\leq 0$ and $1+x\leq e^{x}$ for any
$x\in\mathbb{R}$. A combination of (15) and (16) gives that

$\displaystyle\mathbb{V}\left(\sum_{i=1}^{n}Z_{i}>\epsilon\right)$
$\displaystyle\leq$ $\displaystyle
Ke^{-t\epsilon}\prod_{i=1}^{n}\left[1+t\hat{\mathbb{E}}[Z_{i}]+\frac{2\big{(}\cosh(t\epsilon)-1\big{)}}{\epsilon^{p}}\hat{\mathbb{E}}[|Z_{i}|^{p}]\right]$
(17) $\displaystyle\leq$ $\displaystyle
Ke^{-t\epsilon}\prod_{i=1}^{n}\left[1+\frac{2\big{(}\cosh(t\epsilon)-1\big{)}}{\epsilon^{p}}\hat{\mathbb{E}}[|Z_{i}|^{p}]\right]$
$\displaystyle\leq$ $\displaystyle
Ke^{-t\epsilon}\prod_{i=1}^{n}\exp\left\\{\frac{2\big{(}\cosh(t\epsilon)-1\big{)}}{\epsilon^{p}}\hat{\mathbb{E}}[|Z_{i}|^{p}]\right\\}$
$\displaystyle=$ $\displaystyle
K\exp\left\\{-t\epsilon+\frac{2\big{(}\cosh(t\epsilon)-1\big{)}}{\epsilon^{p}}\sum_{i=1}^{n}\hat{\mathbb{E}}[|Z_{i}|^{p}]\right\\}.$

Since $t>0$ is arbitrary, we take
$t=\epsilon^{-1}\ln\big{(}1+\frac{\epsilon^{p}}{\sum_{i=1}^{n}\hat{\mathbb{E}}[|Z_{i}|^{p}]}\big{)}$
in (17) to obtain that

$\displaystyle\mathbb{V}\left(\sum_{i=1}^{n}Z_{i}>\epsilon\right)$
$\displaystyle\leq$ $\displaystyle
K\exp\left\\{-\ln\left(1+\frac{\epsilon^{p}}{\sum_{i=1}^{n}\hat{\mathbb{E}}[|Z_{i}|^{p}]}\right)+\frac{\epsilon^{p}}{\epsilon^{p}+\sum_{i=1}^{n}\hat{\mathbb{E}}[|Z_{i}|^{p}]}\right\\}$
$\displaystyle\leq$ $\displaystyle
Ke\frac{\sum_{i=1}^{n}\hat{\mathbb{E}}[|Z_{i}|^{p}]}{\epsilon^{p}+\sum_{i=1}^{n}\hat{\mathbb{E}}[|Z_{i}|^{p}]}\leq
Ke\epsilon^{-p}\sum_{i=1}^{n}\hat{\mathbb{E}}[|Z_{i}|^{p}],$

which proves (4). Replacing $Z_{i}$ with $-Z_{i}$ for each $1\leq i\leq n$ in
(4) and noting that

$\mathbb{V}\left(\left|\sum_{i=1}^{n}Z_{i}\right|>\epsilon\right)\leq\mathbb{V}\left(\sum_{i=1}^{n}Z_{i}>\epsilon\right)+\mathbb{V}\left(\sum_{i=1}^{n}(-Z_{i})>\epsilon\right),$

we get (5) immediately. The details are omitted. $\Box$

Proof of Theorem 2.2. Since the desired result holds trivially if
$4^{p}(1+Ke)x^{-p}\sum_{i=1}^{n}\hat{\mathbb{E}}[|X_{i}|^{p}]>1$, we only need
to prove it under the assumption
$4^{p}(1+Ke)x^{-p}\sum_{i=1}^{n}\hat{\mathbb{E}}[|X_{i}|^{p}]\leq 1$ in what
follows. Define for each $1\leq i\leq n$ and any $x>0$ that

$\displaystyle X_{i}(1)$ $\displaystyle=$
$\displaystyle-\frac{x}{4}I\big{(}X_{i}<-\frac{x}{4}\big{)}+X_{i}I\big{(}|X_{i}|\leq\frac{x}{4}\big{)}+\frac{x}{4}I\big{(}X_{i}>\frac{x}{4}\big{)};$
$\displaystyle X_{i}(2)$ $\displaystyle=$
$\displaystyle\big{(}X_{i}+\frac{x}{4}\big{)}I\big{(}X_{i}<-\frac{x}{4}\big{)}+\big{(}X_{i}-\frac{x}{4}\big{)}I\big{(}X_{i}>\frac{x}{4}\big{)}.$

It is easy to see that $X_{i}=X_{i}(1)+X_{i}(2)$ for each $1\leq i\leq n$ and
$\\{X_{i}(1)-EX_{i}(1),i\geq 1\\}$ is still a sequence of END random variables
since $f_{\frac{x}{4}}(x)=\max\\{-\frac{x}{4},\min(x,\frac{x}{4})\\}\in
C_{l,Lip}(\mathbb{R})$. Note that
$|X_{i}(1)-\hat{\mathbb{E}}[X_{i}(1)]|\leq\frac{x}{2}$ and
$\hat{\mathbb{E}}[X_{i}(1)-\hat{\mathbb{E}}[X_{i}(1)]]=0$. It follows from
Theorem 2.1, $|X_{i}(1)|\leq|X_{i}|$, $C_{r}$-inequality and Jensen’s
inequality that

$\displaystyle\mathbb{V}\left(\sum_{i=1}^{n}(X_{i}(1)-\hat{\mathbb{E}}[X_{i}(1)])>\frac{x}{2}\right)$
$\displaystyle\leq$ $\displaystyle
Ke\big{(}\frac{x}{2}\big{)}^{-p}\sum_{i=1}^{n}\hat{\mathbb{E}}[|X_{i}(1)-\hat{\mathbb{E}}[X_{i}(1)]|^{p}]$
(18) $\displaystyle\leq$ $\displaystyle
Ke\big{(}\frac{x}{2}\big{)}^{-p}2^{p-1}\sum_{i=1}^{n}\big{(}\hat{\mathbb{E}}[|X_{i}|^{p}]+(\hat{\mathbb{E}}[|X_{i}|])^{p}\big{)}$
$\displaystyle\leq$ $\displaystyle
Ke\big{(}\frac{x}{2}\big{)}^{-p}2^{p}\sum_{i=1}^{n}\hat{\mathbb{E}}[|X_{i}|^{p}]$
$\displaystyle=$ $\displaystyle
4^{p}Kex^{-p}\sum_{i=1}^{n}\hat{\mathbb{E}}[|X_{i}|^{p}].$

By Markov’s inequality we can easily obtain that

$\displaystyle\mathbb{V}\left(\sum_{i=1}^{n}X_{i}(2)>\frac{x}{4}\right)\leq\mathbb{V}\left(\bigcup_{i=1}^{n}\Big{\\{}|X_{i}|>\frac{x}{4}\Big{\\}}\right)\leq\sum_{i=1}^{n}\mathbb{V}\left(|X_{i}|>\frac{x}{4}\right)\leq
4^{p}x^{-p}\sum_{i=1}^{n}\hat{\mathbb{E}}[|X_{i}|^{p}].$ (19)

Furthermore, observing that
$(|X_{i}|-\frac{x}{4})I(|X_{i}|>\frac{x}{4})\leq|X_{i}|I(|X_{i}|>\frac{x}{4})\leq(\frac{4}{x})^{p-1}|X_{i}|^{p}$
and $4^{p}(1+Ke)x^{-p}\sum_{i=1}^{n}\hat{\mathbb{E}}[|X_{i}|^{p}]\leq 1$, we
have

$\displaystyle\sum_{i=1}^{n}\big{(}\hat{\mathbb{E}}[X_{i}(1)]-\hat{\mathbb{E}}[X_{i}]\big{)}$
$\displaystyle\leq$
$\displaystyle\sum_{i=1}^{n}\hat{\mathbb{E}}\big{[}|X_{i}(1)-X_{i}|\big{]}$
(20) $\displaystyle\leq$
$\displaystyle\sum_{i=1}^{n}\hat{\mathbb{E}}\Big{[}\big{(}|X_{i}|-\frac{x}{4}\big{)}I\big{(}|X_{i}|>\frac{x}{4}\big{)}\Big{]}$
$\displaystyle\leq$
$\displaystyle\big{(}\frac{4}{x}\big{)}^{p-1}\sum_{i=1}^{n}\hat{\mathbb{E}}[|X_{i}|^{p}]$
$\displaystyle=$ $\displaystyle
4^{p}x^{-p}\sum_{i=1}^{n}\hat{\mathbb{E}}[|X_{i}|^{p}]\cdot\frac{x}{4}\leq\frac{x}{4}.$

Recall that $\hat{\mathbb{E}}[X_{i}]\leq 0$ for each $1\leq i\leq n$. It
follows from (18)-(20) that

$\displaystyle\mathbb{V}\left(\sum_{i=1}^{n}X_{i}>x\right)$
$\displaystyle\leq$
$\displaystyle\mathbb{V}\left(\sum_{i=1}^{n}(X_{i}(1)-\hat{\mathbb{E}}[X_{i}(1)])+\sum_{i=1}^{n}\big{(}\hat{\mathbb{E}}[X_{i}(1)]-\hat{\mathbb{E}}[X_{i}]\big{)}+\sum_{i=1}^{n}X_{i}(2)>x\right)$
$\displaystyle\leq$
$\displaystyle\mathbb{V}\left(\sum_{i=1}^{n}(X_{i}(1)-\hat{\mathbb{E}}[X_{i}(1)])>\frac{x}{2}\right)+\mathbb{V}\left(\sum_{i=1}^{n}X_{i}(2)>\frac{x}{4}\right)$
$\displaystyle\leq$ $\displaystyle
4^{p}(1+Ke)x^{-p}\sum_{i=1}^{n}\hat{\mathbb{E}}[|X_{i}|^{p}],$

which implies (6). Replacing $X_{i}$ with $-X_{i}$ for each $1\leq i\leq n$ in
(6), we can easily obtain (7). The proof is completed. $\Box$

Proof of Theorem 2.3. Define for each $1\leq i\leq n$ that

$\displaystyle X_{i}(3)$ $\displaystyle=$ $\displaystyle-
cI(X_{i}<-c)+X_{i}I(|X_{i}|\leq c)+cI(X_{i}>c),$ $\displaystyle X_{i}(4)$
$\displaystyle=$ $\displaystyle(X_{i}+c)I(X_{i}<-c)+(X_{i}-c)I(X_{i}>c),$

where $c>0$ is sufficiently large such that $\sup_{i\geq
1}\hat{\mathbb{E}}[(|X_{i}|-c)_{+}]\leq\varepsilon/3$. It is easy to see that
$X_{i}=X_{i}(3)+X_{i}(4)$ for each $1\leq i\leq n$ and
$\\{X_{i}(3)-\hat{\mathbb{E}}[X_{i}(3)],i\geq 1\\}$ is a sequence of END
random variables. Moreover, we can also easily check that

$\displaystyle
n^{-1}\sum_{i=1}^{n}\big{(}\hat{\mathbb{E}}[X_{i}(3)]-\hat{\mathbb{E}}[X_{i}]\big{)}$
$\displaystyle\leq$ $\displaystyle
n^{-1}\sum_{i=1}^{n}\hat{\mathbb{E}}\big{[}|X_{i}(3)-X_{i}|\big{]}$
$\displaystyle\leq$ $\displaystyle
n^{-1}\sum_{i=1}^{n}\hat{\mathbb{E}}\big{[}(|X_{i}|-c)_{+}\big{]}$
$\displaystyle\leq$ $\displaystyle\sup_{i\geq
1}\hat{\mathbb{E}}[(|X_{i}|-c)_{+}]\leq\varepsilon/3.$

Hence, we obtain by Theorem 2.2 and Markov’s inequality that

$\displaystyle\mathbb{V}\left(\sum_{i=1}^{n}(X_{i}-\hat{\mathbb{E}}[X_{i}])>\varepsilon
n\right)$ $\displaystyle=$
$\displaystyle\mathbb{V}\left(\sum_{i=1}^{n}(X_{i}(3)-\hat{\mathbb{E}}[X_{i}(3)])+\sum_{i=1}^{n}\big{(}\hat{\mathbb{E}}[X_{i}(3)]-\hat{\mathbb{E}}[X_{i}]\big{)}+\sum_{i=1}^{n}X_{i}(4)>\varepsilon
n\right)$ $\displaystyle\leq$
$\displaystyle\mathbb{V}\left(\sum_{i=1}^{n}(X_{i}(3)-\hat{\mathbb{E}}[X_{i}(3)])>\frac{\varepsilon
n}{3}\right)+\mathbb{V}\left(\sum_{i=1}^{n}X_{i}(4)>\frac{\varepsilon
n}{3}\right)$ $\displaystyle\leq$
$\displaystyle\big{(}\frac{3}{\varepsilon}\big{)}^{2}n^{-2}\sum_{i=1}^{n}\hat{\mathbb{E}}\big{[}(X_{i}(3)-\hat{\mathbb{E}}[X_{i}(3)])^{2}\big{]}+\mathbb{V}\left(\sum_{i=1}^{n}|X_{i}(4)|>\varepsilon
n/3\right)$ $\displaystyle\leq$
$\displaystyle\big{(}\frac{3}{\varepsilon}\big{)}^{2}n^{-2}\sum_{i=1}^{n}\hat{\mathbb{E}}\big{[}(X_{i}(3))^{2}\big{]}+\frac{3}{\varepsilon}n^{-1}\sum_{i=1}^{n}\hat{\mathbb{E}}[|X_{i}(4)|]$
$\displaystyle\leq$
$\displaystyle\big{(}\frac{3}{\varepsilon}\big{)}^{2}n^{-1}c^{2}+\frac{3}{\varepsilon}\sup_{i\geq
1}\hat{\mathbb{E}}[(|X_{i}|-c)_{+}]\rightarrow 0$

by letting $n\rightarrow\infty$ first and then $c\rightarrow\infty$. Hence,
the desired result (8) has been proved. $\Box$

Proof of Theorem 2.4. It follows from Hölder’s inequality that
$\sum_{i=1}^{n}a_{ni}^{\gamma}=O(n)$ for any $0<\gamma<q$. The proof will be
conducted under the following two cases.

Case 1. $0<p<1$.

It is easy to check that

$\displaystyle\sum_{n=1}^{\infty}n^{\alpha
p-2}l(n)\mathbb{V}\left(\sum_{i=1}^{n}a_{ni}X_{i}>\varepsilon
n^{\alpha}\right)$ $\displaystyle\leq$
$\displaystyle\sum_{n=1}^{\infty}n^{\alpha
p-2}l(n)\mathbb{V}\left(\sum_{i=1}^{n}a_{ni}X_{i}I(|X_{i}|\leq
n^{\alpha})>\varepsilon n^{\alpha}\right)$
$\displaystyle+\sum_{n=1}^{\infty}n^{\alpha
p-2}l(n)\mathbb{V}\left(\bigcup_{i=1}^{n}\\{|X_{i}|>n^{\alpha}\\}\right)$
$\displaystyle=:$ $\displaystyle I_{1}+I_{2}.$

Denote $q^{\prime}=\min\\{1,q\\}$. Note by (12) that $I(|X_{i}|\leq
n^{\alpha})\leq g\left(\frac{\mu X_{i}}{n^{\alpha}}\right)$. Hence, it follows
from Markov’s inequality and Lemma 3.3 that

$\displaystyle I_{1}$ $\displaystyle\leq$ $\displaystyle
C\sum_{n=1}^{\infty}n^{\alpha p-\alpha
q^{\prime}-2}l(n)\sum_{i=1}^{n}a_{ni}^{q^{\prime}}\hat{\mathbb{E}}\left[|X_{i}|^{q^{\prime}}g\left(\frac{\mu
X_{i}}{n^{\alpha}}\right)\right]$ $\displaystyle\leq$ $\displaystyle
C\sum_{n=1}^{\infty}n^{\alpha p-\alpha
q^{\prime}-1}l(n)\hat{\mathbb{E}}\left[|X|^{q^{\prime}}g\left(\frac{\mu
X}{n^{\alpha}}\right)\right]<\infty.$

On the other hand, by (12) and Lemma 3.2 we can obtain that

$\displaystyle I_{2}$ $\displaystyle\leq$
$\displaystyle\sum_{n=1}^{\infty}n^{\alpha
p-2}l(n)\sum_{i=1}^{n}\mathbb{V}\left(|X_{i}|>n^{\alpha}\right)$
$\displaystyle\leq$ $\displaystyle\sum_{n=1}^{\infty}n^{\alpha
p-2}l(n)\sum_{i=1}^{n}\hat{\mathbb{E}}\left[1-g\left(\frac{X_{i}}{n^{\alpha}}\right)\right]$
$\displaystyle=$ $\displaystyle\sum_{n=1}^{\infty}n^{\alpha
p-1}l(n)\hat{\mathbb{E}}\left[1-g\left(\frac{X}{n^{\alpha}}\right)\right]$
$\displaystyle\leq$ $\displaystyle\sum_{n=1}^{\infty}n^{\alpha
p-1}l(n)\mathbb{V}\left(|X|>\mu n^{\alpha}\right)<\infty.$

Case 2. $1\leq p<2$.

Define for each $1\leq i\leq n$, $n\geq 1$ that

$\displaystyle X_{ni}=-n^{\alpha}I(X_{i}<-n^{\alpha})+X_{i}I(|X_{i}|\leq
n^{\alpha})+n^{\alpha}I(X_{i}>n^{\alpha}).$

It is easy to see that $\\{X_{ni},1\leq i\leq n,n\geq 1\\}$ is still an array
of END random variables under sub-linear expectations. We will firstly show

$\displaystyle
n^{-\alpha}\left|\sum_{i=1}^{n}a_{ni}\hat{\mathbb{E}}[X_{ni}]\right|\rightarrow
0\textrm{ as }n\rightarrow\infty.$ (21)

Noting that $\hat{\mathbb{E}}[X]=0$ if $p>1$, and

$(|X|-n^{\alpha})_{+}\leq|X|I(|X|>n^{\alpha})\leq
n^{\alpha(1-p)}|X|^{p}I(|X|>n^{\alpha})\leq n^{\alpha(1-p)}|X|^{p},$

we derive by $\alpha p>1$ and $\hat{\mathbb{E}}[|X|^{p}]<\infty$ that

$\displaystyle
n^{-\alpha}\left|\sum_{i=1}^{n}a_{ni}\hat{\mathbb{E}}[X_{ni}]\right|$
$\displaystyle=$ $\displaystyle
n^{-\alpha}\left|\sum_{i=1}^{n}a_{ni}(\hat{\mathbb{E}}[X_{ni}]-\hat{\mathbb{E}}[X_{i}])\right|$
$\displaystyle\leq$ $\displaystyle
n^{-\alpha}\sum_{i=1}^{n}a_{ni}\hat{\mathbb{E}}[|X_{i}-X_{ni}|]$
$\displaystyle\leq$ $\displaystyle
Cn^{1-\alpha}\hat{\mathbb{E}}[(|X|-n^{\alpha})_{+}]$ $\displaystyle\leq$
$\displaystyle Cn^{1-\alpha p}\hat{\mathbb{E}}[|X|^{p}]\rightarrow 0\textrm{
as }n\rightarrow\infty.$

If $p=1$, we also have by $\alpha>1$ that

$\displaystyle
n^{-\alpha}\left|\sum_{i=1}^{n}a_{ni}\hat{\mathbb{E}}[X_{ni}]\right|\leq
n^{-\alpha}\sum_{i=1}^{n}a_{ni}\hat{\mathbb{E}}[|X_{ni}|]\leq
Cn^{1-\alpha}\hat{\mathbb{E}}[|X|]\rightarrow 0\textrm{ as
}n\rightarrow\infty.$

Hence, (21) has been proved, from which we can conclude that

$\displaystyle\sum_{n=1}^{\infty}n^{\alpha
p-2}l(n)\mathbb{V}\left(\sum_{i=1}^{n}a_{ni}X_{i}>\varepsilon
n^{\alpha}\right)$ $\displaystyle\leq$ $\displaystyle
C\sum_{n=1}^{\infty}n^{\alpha
p-2}l(n)\mathbb{V}\left(\sum_{i=1}^{n}a_{ni}\big{(}X_{ni}-\hat{\mathbb{E}}[X_{ni}]\big{)}>\frac{\varepsilon}{2}n^{\alpha}\right)$
$\displaystyle+\sum_{n=1}^{\infty}n^{\alpha
p-2}l(n)\mathbb{V}\left(\bigcup_{i=1}^{n}\\{|X_{i}|>n^{\alpha}\\}\right)$
$\displaystyle=:$ $\displaystyle I_{3}+I_{2}.$

Since $I_{2}<\infty$ has been proved in Case 1, we only need to show
$I_{3}<\infty$. Take $\beta=\min\\{2,q\\}$. Note by (12) that

$|X_{ni}|^{\beta}=|X_{i}|^{\beta}I(|X_{i}|\leq
n^{\alpha})+n^{\alpha\beta}I(|X_{i}|>n^{\alpha})\leq|X_{i}|^{\beta}g\left(\frac{\mu
X_{i}}{n^{\alpha}}\right)+n^{\alpha\beta}\left[1-g\left(\frac{X_{i}}{n^{\alpha}}\right)\right].$

Therefore, we obtain by (12) again, Theorem 2.2, $C_{r}$-inequality, Jensen’s
inequality, Lemma 3.2 and Lemma 3.3 that

$\displaystyle I_{3}$ $\displaystyle\leq$ $\displaystyle
C\sum_{n=1}^{\infty}n^{\alpha
p-\alpha\beta-2}l(n)\sum_{i=1}^{n}a_{ni}^{\beta}\hat{\mathbb{E}}\big{[}|X_{ni}-\hat{\mathbb{E}}[X_{ni}]|^{\beta}\big{]}$
$\displaystyle\leq$ $\displaystyle C\sum_{n=1}^{\infty}n^{\alpha
p-\alpha\beta-2}l(n)\sum_{i=1}^{n}a_{ni}^{\beta}\hat{\mathbb{E}}\big{[}|X_{ni}|^{\beta}\big{]}$
$\displaystyle\leq$ $\displaystyle C\sum_{n=1}^{\infty}n^{\alpha
p-\alpha\beta-2}l(n)\sum_{i=1}^{n}a_{ni}^{\beta}\left\\{\hat{\mathbb{E}}\left[|X_{i}|^{\beta}g\left(\frac{\mu
X_{i}}{n^{\alpha}}\right)\right]+n^{\alpha\beta}\hat{\mathbb{E}}\left[1-g\left(\frac{X_{i}}{n^{\alpha}}\right)\right]\right\\}$
$\displaystyle\leq$ $\displaystyle C\sum_{n=1}^{\infty}n^{\alpha
p-\alpha\beta-1}l(n)\hat{\mathbb{E}}\left[|X|^{\beta}g\left(\frac{\mu
X}{n^{\alpha}}\right)\right]+C\sum_{n=1}^{\infty}n^{\alpha
p-1}l(n)\hat{\mathbb{E}}\left[1-g\left(\frac{X}{n^{\alpha}}\right)\right]$
$\displaystyle\leq$ $\displaystyle C\sum_{n=1}^{\infty}n^{\alpha
p-\alpha\beta-1}l(n)\hat{\mathbb{E}}\left[|X|^{\beta}g\left(\frac{\mu
X}{n^{\alpha}}\right)\right]+C\sum_{n=1}^{\infty}n^{\alpha
p-1}l(n)\mathbb{V}\left(|X|>\mu n^{\alpha}\right)$ $\displaystyle<$
$\displaystyle\infty.$

Therefore, (9) is proved. Replacing $X_{i}$ with $-X_{i}$ for each $1\leq
i\leq n$ in (9), we can obtain (10) without much effort. The proof is hence
completed. $\Box$

Proof of Theorem 2.5. We only give the proof of (11). In view of the proof of
Theorem 2.4, we only need to verify that (21) holds under the assumption
$\lim_{c\rightarrow\infty}\hat{\mathbb{E}}[(|X|^{p}-c)_{+}]=0$ when $1\leq
p<2$ with $\alpha p=1$. Actually, it is easy to check that

$\displaystyle(|X|-n^{1/p})_{+}$ $\displaystyle\leq$
$\displaystyle|X|I(|X|>n^{1/p})$ $\displaystyle\leq$ $\displaystyle
n^{(1-p)/p}|X|^{p}I(|X|^{p}>n)$ $\displaystyle\leq$ $\displaystyle
2n^{(1-p)/p}(|X|^{p}-n/2)I(|X|^{p}>n)$ $\displaystyle\leq$ $\displaystyle
2n^{(1-p)/p}(|X|^{p}-n/2)_{+}.$

Hence, we obtain by the assumption $\hat{\mathbb{E}}[X]=0$ for $p\geq 1$ that

$\displaystyle
n^{-1/p}\left|\sum_{i=1}^{n}a_{ni}\hat{\mathbb{E}}[X_{ni}]\right|$
$\displaystyle\leq$ $\displaystyle
n^{-1/p}\sum_{i=1}^{n}a_{ni}\hat{\mathbb{E}}[|X_{i}-X_{ni}|]$
$\displaystyle\leq$ $\displaystyle
Cn^{1-1/p}\hat{\mathbb{E}}[(|X|-n^{1/p})_{+}]$ $\displaystyle\leq$
$\displaystyle C\hat{\mathbb{E}}[(|X|^{p}-n/2)_{+}]\rightarrow 0\textrm{ as
}n\rightarrow\infty.$

The proof is completed. $\Box$

## References

  * [1] Artzner Ph., Delbaen F., Eber J.M., Heath D., 1997. Thinking coherently. RISK, 10, 68-71.
  * [2] Asadian N., Fakoor V., Bozorgnia A., 2006. Rosenthal’s type inequalities for negatively orthant dependent random variables. Journal of the Iranian Statistical Society, 5, 69-75.
  * [3] Bai Z.D., Su C., 1985. The complete convergence for partial sums of i.i.d. random variables. Science in China, Series A, 12, 31-47.
  * [4] Bryc W., Smolenski W., 1993. Moment conditions for almost sure convergence of weakly correlated random variables. Proceedings of the American Mathematical Society, 119(2), 629-635.
  * [5] Chatterji S.D., 1969. An $L^{p}$-convergence theorem. The Annals of Mathematical Statistics, 40, 1068-1070.
  * [6] Chen P.Y., Bai P, Sung S.H., 2014. The von Bahr-Esseen moment inequality for pairwise independent random variables and applications. Journal of Mathematical Analysis and Applications, 419(2), 1290-1302.
  * [7] Chen Z.J., Epstein L., 2002. Ambiguity, risk and asset returns in continuous time. Econometrica, 70(4), 1403-1443.
  * [8] Chen Z.J., Hu F., 2014. A law of the iterated logarithm under sublinear expectations. Journal of Financial Engineering, 1(2), Article ID 1450015, 23 pages.
  * [9] Denis L., Martini C., 2006. A theoretical framework for the pricing of contingent claims in the presence of model uncertainty. The Annals of Applied Probability, 16(2), 827-852.
  * [10] El Karoui N., Peng S.G., Quenez M.C., 1997. Backward stochastic differential equation in finance. Mathematical Finance, 7(1), 1-71.
  * [11] Follmer H., Schied A., 2004. Statistic Finance, An Introduction in Discrete Time (2nd Edition), Walter de Gruyter.
  * [12] Gilboa I., 1987. Expected utility theory with purely subjective non-additive probabilities. Journal of Mathematical Economics, 16(1), 65-88.
  * [13] Hu T.-C., Wang K.L., Rosalsky A., 2015. Complete convergence theorems for extended negatively dependent random variables. Sankhyā A: The Indian Journal of Statistics, 77(1), 1-29.
  * [14] Kuczmaszewska A., 2020. Complete convergence for widely acceptable random variables under the sublinear expectations. Journal of Mathematical Analysis and Applications, 484(1), Article ID 123662.
  * [15] Marinacci M. 1999. Limit laws for non-additive probabilities and their frequentist interpretation. Journal of Economic Theory, 84(2), 145-195.
  * [16] Peng S.G., 1997. Backward SDE and related $g$-expectations, in Backward Stochastic Differential Equations, Pitman Research Notes in Math. Series, No. 364, El Karoui Mazliak edit. 141-159.
  * [17] Peng S.G.,1999. Monotonic limit theorem of BSDE and nonlinear decomposition theorem of Doob-Meyer type. Probability Theory and Related Fields, 113(4), 473-499.
  * [18] Peng S.G., 2006. $G$-expectation, $G$-Brownian motion and related stochastic calculus of Itos type. The Abel Symposium 2005, Abel Symposia 2, Edit. Benth et. al., 541-567, Springer-Verlag, 2006.
  * [19] Peng S.G., 2008a. Multi-dimensional $G$-Brownian motion and related stochastic calculus under $G$-expectation. Stochastic Processes and Their Applications, 118(12), 2223-2253.
  * [20] Peng S.G., 2008b. A new central limit theorem under sublinear expectations. arXiv preprint arXiv:0803.2656.
  * [21] Peng S.G., 2010. Nonlinear expectations and stochastic calculus under uncertainty. arXiv preprint arXiv:1002.4546.
  * [22] Shao Q.M., 2000. A comparison theorem on moment inequalities between negatively associated and independent random variables. Journal of Theoretical Probability, 13(2), 343-356.
  * [23] Tang X.F., Wang X.J., Wu Y., 2019. Exponential inequalities under sub-linear expectations with applications to strong law of large numbers. Filomat, 33(10), 2951-2961.
  * [24] von Bahr B., Esseen C.G., 1965. Inequalities for the $r$th absolute moment of a sum of random variables, $1\leq r\leq 2$. The Annals of Mathematical Statistics, 36(1), 299-303.
  * [25] Wang X.J., Xu C., Hu T.-C., Volodin A., Hu S.H., 2014. On complete convergence for widely orthant dependent random variables and its applications in nonparametric regression models. TEST, 23, 607-629.
  * [26] Wu Q.Y., Jiang Y.Y., 2018. Strong law of large numbers and Chover’s law of the iterated logarithm under sub-linear expectations. Journal of Mathematical Analysis and Applications, 460(1), 252-270.
  * [27] Wu Q.Y., Jiang Y.Y., 2020. Complete convergence and complete moment convergence for negatively dependent random variables under sub-linear expectations. Filomat, 34(4), 1093-1104.
  * [28] Wu Y., Wang X.J., Shen A.T., 2021. Strong convergence properties for weighted sums of $m$-asymptotic negatively associated random variables and statistical applications. Statistical Papers, 62, 2169-2194.
  * [29] Wu Y., Wang X.J., Zhang L.X., 2018. On the asymptotic approximation of inverse moment under sub-linear expectations. Journal of Mathematical Analysis and Applications, 468, 182-196.
  * [30] Xi M.M., Wang X.J., Wu Y., 2019. Complete convergence for arrays of rowwise END random variables and its statistical applications under sub-linear expectations. Journal of the Korean Statistical Society, 48(3), 412-425.
  * [31] Zhang L.X., 2015. Donsker’s invariance principle under the sub-linear expectation with an application to Chung’s law of the iterated logarithm. Communications in Mathematics and Statistics, 3(2), 187-214.
  * [32] Zhang L.X., 2016a. Exponential inequalities under sub-linear expectations with applications to laws of the iterated logarithm. Science China Mathematics, 59(12), 2503-2526.
  * [33] Zhang L.X., 2016b. Rosenthal’s inequalities for independent and negatively dependent random variables under sub-linear expectations with applications. Science China Mathematics, 59(4), 751-768.
  * [34] Zhang L.X., 2020. The convergence of the sums of independent random variables under the sub-linear expectations. Acta Mathematica Sinica, English Series, 36(3), 224-244.
  * [35] Zhang L.X., 2022. Strong limit theorems for extended independent random variables and extended negatively dependent random variables under sub-linear expectations. Acta Mathematica Scientia, 42B(2), 467-490.
  * [36] Zhong H.Y., Wu Q.Y., 2017. Complete convergence and complete moment convergence for weighted sums of extended negatively dependent random variables under sub-linear expectation. Journal of Inequalities and Applications, Volumn 2017, Article ID 261, 14 pages.