# Joint Frame Structure, Beamwidth, and Power Allocation for UAV-Aided
Localization and Communication

Tianhao Liang, , Tingting Zhang, , Sheng Zhou, , Wentao Liu, Dong Li, , Qinyu
Zhang Part of this work will be presented in the IEEE WCNC 2024 [1] Tianhao
Liang, Tingting Zhang and Qinyu Zhang are with School of Electronics and
Information Engineering, Harbin Institute of Technology (Shenzhen), Shenzhen,
China (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>zqy@hit.edu.cn).
Sheng Zhou is with Department of Electronic Engineering, Tsinghua University,
Beijing, China (e-mail: sheng.zhou@tsinghua.edu.cn). Wentao Liu and Dong Li
are with the School of Computer Science and Engineering, Macau University of
Science and Technology, Macau, China (e-mail:
<EMAIL_ADDRESS>dli@must.edu.mo).

###### Abstract

In wireless sensors networks, integrating localization and communications
techniques is crucial for efficient spectrum and hardware utilizations. In
this paper, we present a novel framework of unmanned aerial vehicle
(UAV)-aided localization and communication for ground node (GN), where the
average spectral efficiency (SE) is used to reveal the intricate relationship
among frame structure, channel estimation error, and localization accuracy. In
particular, we first derive the lower bounds for channel estimation error and
the three dimensional location prediction error. Leveraging these
comprehensive analysis, we formulate a problem to maximize the average SE in
UAV-GN communication, where the frame structure, beamwidth and power
allocation are jointly optimized. Subsequently, we propose an efficient
iterative algorithm to address this non-convex problem with closed-form
expressions for beamwidth and power allocation. Numerical results demonstrate
that the performance of our proposed method can approach the upper bound with
much lower complexity, and achieve over 70% performance gain compared to non-
localization benchmarks. Additionally, the analysis highlights the dominated
impacts from the Doppler effect over noise on the average SE.

###### Index Terms:

Wireless sensor networks, channel estimation, spectral efficiency, location
and communication.

##  I Introduction

###  I-A Background and Motivation

Accurate localization and reliable communication are two fundamental aspects
of mobile wireless sensor networks [2]. Localization typically involves
determining the spatial coordinates of mobile sensors, while communication
facilitates the exchange of information between sensor nodes, enabling data
transmission, remote control, and collaboration. However, the current design
in sensor networks often treats localization and communication as separate
entities, which can result in high hardware costs and low spectral efficiency
(SE).

Prompted by advanced technologies of the millimeter-wave (mmWave) and massive
multiple-input multiple-output (MIMO) in the upcoming 6G, integrated
localization and communication systems are driving innovation and research
holds in many spectrum and cost limited scenarios [3]. However, obstacles on
the ground may prevent the establishment of line-of-sight (LoS) links, leading
to severe attenuation of mmWave signals and indicating poor performance in
localization and communication. Obtaining reliable connected links of
infrastructures is crucial to support high-quality localization and
communication services. Consequently, in areas lacking infrastructures, such
as deep canyon environment monitoring, disaster life detection, and border
target tracking, unpredictable difficulties may arise in localization and
communication [4].

Multiple fascinating advantages of unmanned aerial vehicles (UAVs), including
high mobility, low cost, and fast deployment, have attracted substantial
attention from both academia and industry [5]. UAVs can provide higher
reliable LoS connected links and flexible topologies, presenting a cost-
effective alternative to traditional infrastructures. Additionally, UAVs have
been proven to be significantly important in supporting localization and
communication [6] [7]. Nevertheless, the high mobility also inevitably
introduces non-negligible Doppler effects, indicating a strong time variant of
the channel.

The bi-static channel estimation for communication in multi-antenna systems
can be regarded as a standard integrated localization and communication
process, where localization parameters, such as distance, time of arrival
(TOA) [8], and angle of arrival (AOA) [9], can be obtained simultaneously from
the channel. However, the increased pilot overhead will lead to a lower SE,
especially in UAV-involved networks. Therefore, investigating the frame
structure design in UAV-aided localization and communication system to achieve
a high system SE is a particularly challenging task.

###  I-B Related Work

Many efforts have been dedicated to optimizing the time frame structure for
accurate channel estimation and data transmission. The authors in [10]
investigated the channel estimation accuracy with respect to pilot length and
base stations density. In [11], the authors optimized the channel estimation
time to improve data rates, where the duration of channel estimation was
determined by comparing the predicted and the current data rate. Furthermore,
they extended this investigation to the multi-user scenario in [4]. However,
these investigations were limited to static communication conditions. For
dynamic scenarios, the authors in [12] investigated frame structures to
improve the achievable rate in both block and continuous fading, revealing the
relationship of channel estimation error and frame structure. Building on this
study, the authors in [13] proposed a time frame structure optimization
problem to maximize the system throughput while guaranteeing the latency, and
provided two frame structures for different scenarios. However, these studies
focused exclusively on simple single-antenna systems. To address this
limitation, the authors in [14] enhanced the sum SE and maximum error
probability in the MIMO system by establishing a function between channel
coefficients, precoders, error probabilities, and achievable SE. Nevertheless,
this study assumed prefect channel state information (CSI). Despite the frame
structure and resource allocation strategies designed in the above studies
from different perspectives, localization operations are rarely discussed, and
three-dimensional (3D) scenarios are not investigated.

Based on the fact that the antenna phased shifts can be expressed as a
function of receiver location [15], the authors in [16] proposed a position-
aided channel estimation method for high speed railways, where the antenna
arrays was divided into two part for localization and communication,
respectively. They also derived the relationship between mobility and
throughput. However, this work required large scale antennas arrays, making it
unsuitable for the low-cost wireless sensor networks. In [17], the authors
optimized the pilot overhead by the cooperative beamforming and power
allocation. However, this framework was designed for scenarios with sufficient
infrastructures. Additionally, the authors in [18] utilized a full-duplex UAV
as the aerial relay to increase the communication capacity under a 3D channel
model. The UAV location was optimized together with the CSI-based beamforming
and power control. However, the UAV was assumed to be fixed in the air.

In practice, the impacts of mobility in UAV-involved network cannot be
ignored. In such scenarios, CSI-based beamforming becomes computationally
intensive and inaccurate due to the rapidly evolving channel conditions. To
handle the channel uncertainty, it will incur a substantial overhead on
channel estimation and localization. Location-aware beamforming can be adopted
to reduce the beam swapping time and enhance the accuracy of beam alignment.
The authors in [19] investigated a location-aided beam alignment for fast link
establishment and provided a robust scheme by Bayesian team decision, but they
only considered this framework in a static scenario. In [20], the authors
proposed a location-aware beamforming method for UAV communications by
utilizing a spatial estimation method. However, they also assumed perfect CSI.

###  I-C Main Contributions

Motivated by the aforementioned issues, we aim to reveal the relationship
among channel estimation, localization, and communication in UAV aided
localization and communication systems. Additionally, we seek to develop an
optimization framework based on the findings. The contributions of this paper
are summarized as follows.

  * •

We first elaborate on the settings and process of the UAV-aided localization
and communication framework. Then, we derive the error bounds for channel
estimation and location predication in terms of communication signal-to-noise
ratio (SNR) and time frame structure. Moreover, we formulate a joint frame
structure, beamforming, and power optimization problem, to maximize the
average SE.

  * •

Building on the comprehensive analysis presented, we propose an efficient
iterative method to solve the proposed optimization problem. This method
employs relaxation and approximation strategies for determining transmission
and pilot duration. Remarkably, the beamwidth and power allocation are
expressed in closed-form.

  * •

Lastly, we conduct numerical simulations to demonstrate the advantages of our
proposed approach compared to four benchmarks. The results showcase that our
integrated localization and communication method performs close to the upper
bound and achieves over 70% performance gain compared to the communication-
only method.

The rest of this paper is organized as follows. Section II introduces the
detailed system model and formulates the problem of the UAV-aided localization
and communication network. Section III presents an efficient iterative
algorithm to solve this optimization problem. Section IV demonstrates the
simulation results. Finally, Section V provides the conclusion.

##  II System Model

Figure 1: UAV aided localization and communication in dynamic scenario.

We consider a point-to-point UAV-aided localization and communication process
depicted in Fig. 1. This involves a UAV equipped with uniform planar array
(UPA) employing $N_{\text{t}}=M\times N$ antennas to assist the localization
and communication of the ground node (GN), which is equipped with a single
antenna. Without loss of generality, we consider downlink communication in the
three-dimensional Cartesian coordinate system. The location and velocity of
the UAV at time $t$ are denoted by
${\bf{x}}_{\text{u}}(t)=\left[p_{\text{u}}^{x}(t),p_{\text{u}}^{y}(t),p_{\text{u}}^{z}(t)\right]^{\text{T}}$
and
${\boldsymbol{\nu}}_{\text{u}}(t)=\left[\nu_{\text{u}}^{x}(t),\nu_{\text{u}}^{y}(t),\nu_{\text{u}}^{z}(t)\right]^{\text{T}}$,
respectively. Similarly, let
${\bf{x}}_{\text{n}}(t)=\left[p_{\text{n}}^{x}(t),p_{\text{n}}^{y}(t),p_{\text{n}}^{z}(t)\right]^{\text{T}}$
and
${\boldsymbol{\nu}}_{\text{n}}(t)=\left[\nu_{\text{n}}^{x}(t),\nu_{\text{n}}^{y}(t),\nu_{\text{n}}^{z}(t)\right]^{\text{T}}$
denote the location and velocity of the GN, respectively.

###  II-A Communication Model

In our considered system, the UAV transmits signal ${\bf{s}}(t)={\bf
w}(t)s(t)$ to the GN with power $P(t)$, where ${\bf
w}(t)\in\mathbb{C}^{N_{\text{t}}\times 1}$ is the transmitted beamforming
vector, and $\mathbb{E}(|{\bf{s}}(t)|^{2})=1$. Thus, the received signal of
this multiple-input single-output (MISO) system during the $t$-th time slot
can be expressed by

${{r}}(t)=\sqrt{P(t)}{\bf{H}}(t){\bf{s}}(t)+{n}(t),$ (1)

where ${\bf H}(t)\in\mathbb{C}^{1\times N_{\text{t}}}$ is the channel vector
between the UAV and the GN, and ${n}(t)$ is the white Gaussian noise at the
receiver having zero mean and power $\sigma_{0}^{2}$. The current channel
vector can be expressed by

${\bf{H}}(t)=\sqrt{\beta(t)}h(t){\bf{a}}_{\text{t}}^{\text{H}}\left(\theta(t),\varphi(t)\right),$
(2)

where $\beta(t)=\beta_{0}/d(t)^{2}$ is the large scale channel fading with the
distance between UAV and GN
$d(t)=\sqrt{{\left(p_{\text{u}}^{x}(t)-p_{\text{n}}^{x}(t)\right)^{2}+\left(p_{\text{u}}^{y}(t)-p_{\text{n}}^{y}(t)\right)^{2}+\left(p_{\text{u}}^{z}(t)-p_{\text{n}}^{z}(t)\right)^{2}}}$.
The term $\beta_{0}$ represents the channel power at the reference distance 1
m. $h(t)$ is small scale Rician fading channel gain111The channel fading
between GN and each antenna element at the UAV are highly correlated.
Therefore, we use $h(t)$ to describe the LoS channel., and
${\bf{a}}_{\text{t}}\left(\theta(t),\varphi(t)\right)\in\mathbb{C}^{N_{\text{t}}\times
1}$ is the array steering vectors of azimuth angle of departure (AoD)
$\varphi$ and elevation AoD $\theta$.

In this paper, we assume the UAV is equipped with a directional antenna with
adjustable beamwidth, which can be expressed by a simplified antenna gain
model as [21]

$\displaystyle\\!\\!\\!\\!\\!\\!G\left(\theta(t),\varphi(t)\right)\\!\\!=\\!\\!\left\\{\begin{array}[]{cc}\frac{G_{0}}{\Phi(t)^{2}},\\!\\!\\!&\varphi-\Phi(t)\\!<\\!\varphi<\\!\varphi+\Phi(t)\\\
&\text{and}\quad\\!\\!\\!\\!\theta-\Phi(t)\\!<\\!\theta<\\!\theta+\Phi(t)\\\
0,&\text{otherwise}\end{array}\right.,$ (6)

where $2\Phi(t)\in(0,\pi)$ is the same half-power beamwidth of the antenna on
azimuth and elevation, and $G_{0}$ is a gain constant. The antenna in the GN
is omnidirectional with unit gain. Then, the instantaneous SNR can be computed
by

$\gamma(t)=\frac{P(t)G_{0}\beta_{0}}{\Phi(t)^{2}d(t)^{2}\sigma_{0}^{2}}.$ (7)

Figure 2: Time frame structure of GN for uplink transmission.

The time frame structure is illustrated in Fig. 2, where the UAV takes
$T_{\text{s}}$ to estimate the CSI during the sensing phase, which comprises
$k$ orthogonal pilots with the minimum time slot $T_{0}$. Subsequently, the
UAV employs beamforming to build up a communication link with the GN for data
transmission. The term $T_{\text{d}}$ represents the duration of data
transmission. The channel is assumed to be correlated in a frame within
channel coherence time duration $T_{f}$, which is influenced by the Doppler
effect of the connection link.

###  II-B Channel Estimation

Due to the time-varying characteristics of the proposed scenario caused by the
mobility of the UAV, the CSI will soon become outdated, resulting in the high
estimation error. Typically, the channel estimation error is caused by two
factors [22]. The fist is the noise, and the second is the rapid temporal
variation of channel, reflected in the Doppler effect.

We denote ${\bf h}_{1:t}=\left[h(1),\cdots,h(t)\right]^{\text{T}}$ as the true
channel vector for time 1 to time $t$, and $\widehat{\bf h}_{1:t}$ as our
estimated channel. Then, the channel estimation error can be expressed as

$\displaystyle\Delta{\bf h}_{1:t}=\widehat{\bf h}_{1:t}-{\bf
h}_{1:t}=\Delta{\bf h}_{1:t}^{\text{n}}+\Delta{\bf h}_{1:t}^{\text{D}},$ (8)

where $\Delta{\bf h}_{1:t}^{\text{n}}$ is the channel mismatch caused by
noise, and $\Delta{\bf h}_{1:t}^{\text{D}}$ is the error due to temporal
variation.

Proposition 1 : Given the pilot length $T_{\text{s}}$ with $k$ orthogonal
symbols, the channel estimation minimum mean square error (MMSE) at time
$T_{f}$, which corresponds to the $(k+n)$-th symbol, can be expressed as [13]

$\delta_{\textbf{h}}^{2}=\underbrace{\frac{1}{1+k\gamma}}_{\text{noise}}+\underbrace{\frac{2k\gamma\left(1-\alpha_{k,k+n}\right)}{1+k\gamma}}_{\text{Doppler}},$
(9)

where $\alpha_{k,k+n}$ is the correlation function between the channel at
symbol $k$ and symbol $k+n$ with
$\alpha_{k,n}=\kappa^{\frac{nf_{d}}{0.423B}}$. The term $\kappa\in(0,1)$ is
the level of correlation, $f_{d}$ and $B$ are the Doppler frequency shift and
system bandwidth, respectively.

Proof : See Appendix A

###  II-C Localization Estimation and Prediction

The motion model of UAV can be expressed as

$\displaystyle{\bf{x}}_{\text{m}}(t)={\bf{x}}_{\text{m}}(t-1)+\boldsymbol{\nu}_{\text{m}}(t)T_{0}+\boldsymbol{\omega}(t),$
(10)

where $\text{m}=\\{\text{u},\text{n}\\}$ is the index of UAV or GN, and
$\boldsymbol{\omega}(t)=[\omega_{x}(t),\omega_{y}(t),\omega_{z}(t)]^{\text{T}}$
is the measurement noise vector along the x-axis, y-axis and z-axis direction,
which are assumed to be independent Gaussian distribution with zero mean and
variance $\sigma_{x}^{2}$, $\sigma_{y}^{2}$ and $\sigma_{z}^{2}$.

According to our settings, the estimation parameters for localization at time
$t$ include time delay $\tau(t)$, elevation angle $\theta(t)$ and azimuth
angle $\varphi(t)$. Therefore, the localization parameter space can be defined
as

$\displaystyle\boldsymbol{\Theta}_{t}=\left[\tau(t),\theta(t),\varphi(t)\right]^{\text{T}}.$
(11)

Let $\widehat{\bf x}_{\text{m}}$ denote the estimated location of the UAV and
GN. In this paper, we use the localization MSE
$L_{e}^{2}=\mathbb{E}\left\\{\|{\bf x}_{\text{m}}(t)-\widehat{\bf
x}_{\text{m}}(t)\|^{2}\right\\}$ to evaluate the localization performance,
which is determined by the on-broad inertial measurements and the signal
measurements between the UAV and GN.

We derive the Posterior Cramér-Rao bound (PCRB) as the lower bound of the
estimator variance in dynamic scenarios [23]. This metric is valid for a large
number of measurements. The equivalent Fisher information matrix (EFIM) of
$\boldsymbol{\Theta}_{t}$ can be expressed as a block diagonal matrix by chain
rules from time 1 to time $t$ with222We assume prediction only related to
adjacent time slots, indicating ${\bf C}_{i-2,i}=0$.

$\displaystyle\mathbf{J}_{\text{m}}^{e}=\\!\\!\\!\left[\\!\\!\\!\begin{array}[]{ccccc}\mathbf{J}_{1,1}&\mathbf{C}_{1,2}&0&\cdots&0\\\
\mathbf{C}_{1,2}^{\text{T}}&\mathbf{J}_{2,2}&\mathbf{C}_{2,3}&\cdots&0\\\
\vdots&\ddots&\ddots&\ddots&\vdots\\\
0&\cdots&\mathbf{C}_{t-2,t-1}^{\text{T}}&\mathbf{J}_{t-1,t-1}&\mathbf{C}_{t-1,t}\\\
0&\cdots&0&\mathbf{C}_{t-1,t}^{\text{T}}&\mathbf{J}_{t,t}\\\
\end{array}\\!\\!\\!\right].$ (17)

The element of $\mathbf{J}_{i,i},i\in[1,t]$ is calculated by the FIM
$\mathbf{J}_{i,i}^{\text{p}}$ of measurement pilot signals or information
$\mathbf{J}_{i,i}^{\text{I}}$ from the on-broad inertial measurement unit.
Then, we can obtain the expression as follows

$\displaystyle\mathbf{J}_{i,i}=\left\\{\begin{array}[]{cc}\mathbf{J}_{i,i}^{\text{p}}+\mathbf{J}_{i,i}^{\text{I}},&i\leq
T_{\text{s}},\\\
\mathbf{J}_{i,i}^{\text{I}},&T_{\text{s}}<i,\end{array}\right.$ (20)

where the 3D measurement matric can be characterized by [9]

$\displaystyle\mathbf{J}_{i,i}^{\text{p}}=\lambda_{i}^{r}\mathbf{q}_{i,r}\mathbf{q}_{i,r}^{\text{T}}+\lambda_{i}^{\theta}\mathbf{q}_{i,\theta}\mathbf{q}_{ij,\theta}^{\text{T}}+\lambda_{i}^{\varphi}\mathbf{q}_{i,\varphi}\mathbf{q}_{i,\varphi}^{\text{T}}.$
(21)

The terms $\lambda_{i}^{r}$, $\lambda_{i}^{\theta}$ and
$\lambda_{i}^{\varphi}$ are the ranging information of intensity and angle
information from the signal, and $\mathbf{q}_{i,r}$, $\mathbf{q}_{i,\theta}$
and $\mathbf{q}_{i,r}$ are the related angular vector. The ranging information
can be obtained by
$\lambda_{i,r}=\frac{8\pi^{2}\zeta^{2}(1-\chi^{2})}{c^{2}}\gamma(i)$,
$\lambda_{i,\theta}=\frac{8\pi^{2}(\zeta\chi+f_{c})^{2}}{c^{2}}\gamma(i)\sum\limits_{n=1}^{N_{t}}A_{n,i}$,
and
$\lambda_{i,\varphi}=\frac{8\pi^{2}(\zeta\chi+f_{c})^{2}}{c^{2}}\gamma(i)\sum\limits_{n=1}^{N_{t}}A_{n,i}$,
where $\zeta$, $\chi$, $f_{c}$ are the related signal parameters, $A_{n,i}$ is
the parameter related with the geometry of antenna array [24].

Proposition 2: The direction vectors of distance, elevation and azimuth are
given by

$\displaystyle\mathbf{q}_{i,r}=\left[{\begin{array}[]{*{20}{c}}{\cos\theta({i})\cos\varphi({i})}&{\sin\theta({i})\cos\varphi({i})}&\sin\varphi({i})\end{array}}\right]^{\text{T}},$
(23) $\displaystyle{{\bf{q}}_{i,\theta}}$
$\displaystyle=\left[\begin{array}[]{*{20}{c}}{-\frac{\cos{\varphi({i})}\sin{\theta({i})}}{d({i})}}&{-\frac{\sin{\theta({i})}\sin{\varphi({i})}}{d({i})}}&{\frac{\cos{\theta({i})}}{d({i})}}\end{array}\right]^{\text{T}},$
(25) $\displaystyle{{\bf{q}}_{i,\varphi}}$
$\displaystyle=\left[{\begin{array}[]{*{20}{c}}{-\frac{{\sin{\varphi({i})}}}{{{d({i})}\cos{\theta({i})}}}}&{\frac{{\cos{\varphi({i})}}}{{{d({i})}\cos{\theta({i})}}}}&0\end{array}}\right]^{\text{T}}.$
(27)

Proof 2: See Appendix B

The second term in (20) can be expressed as
${\mathbf{J}_{i,i}^{\text{I}}}={\rm{diag}}\left\\{\frac{1}{\sigma_{x}^{2}},\frac{1}{\sigma_{y}^{2}},\frac{1}{\sigma_{z}^{2}}\right\\}$,
and the cross information matrix $\mathbf{C}_{i-1,i}$ is expressed as

$\displaystyle\mathbf{C}_{i-1,i}=-{\rm{diag}}\left\\{\frac{1}{\sigma_{x}^{2}},\frac{1}{\sigma_{y}^{2}},\frac{1}{\sigma_{z}^{2}}\right\\}.$
(28)

According to the matrix chain and Schur complement rules, we can obtain the
localization FIM at time $t$ by

$\displaystyle{\bf J}({{\bf
x}_{\text{m}}(t)})=\\!\mathbf{J}_{t,t}\\!-\\!\mathbf{C}_{t-1,t}({\bf J}({{\bf
x}_{\text{m}}(t-1)})\\!+\\!\mathbf{C}_{t-1,t-1})^{-1}\mathbf{C}_{t-1,t}^{\text{T}},$
(29)

in which ${\bf J}({{\bf x}_{\text{m}}(1)})=\mathbf{J}_{1,1}$. Consequently,
the localization error can be bounded by

$\displaystyle\mathbb{E}\left\\{\|{\bf x}_{\text{m}}(t)-\widehat{\bf
x}_{\text{m}}(t)\|^{2}\right\\}\geq\text{tr}\left\\{\left[{\bf J}^{-1}({{\bf
x}_{\text{m}}(t)})\right]\right\\}.$ (30)

###  II-D Problem Formulation

The careful design of frame structure, beamwidth and power allocation has
significant impacts on the localization accuracy and channel estimation error,
which will contribute to the performance of SE. Given the frame structure and
resource allocation, the ergodic achievable SE can be expressed as [25]

$\displaystyle\eta_{\text{SE}}=\left(1-\frac{T_{\text{s}}}{T_{f}}\right)\log_{2}\left(1+\gamma_{e}\right),$
(31)

where
$\gamma_{e}=\frac{\gamma\left(1-\delta_{\textbf{h}}^{2}\right)}{1+\gamma\delta_{\textbf{h}}^{2}}$
is the effective SNR under current instantaneous SNR $\gamma$ with channel
estimation error $\delta_{\textbf{h}}^{2}$.

In this paper, we aim to maximize the average SE by optimizing the pilot and
data transmission durations, antenna beamwidth, and transmitted power. Denote
${\bf
T}_{\text{s}}=\left[T_{\text{s}}(1),T_{\text{s}}(2),\cdots,T_{\text{s}}(N)\right]^{\text{T}}$,
${\bf
T}_{\text{d}}=\left[T_{\text{d}}(1),T_{\text{d}}(2),\cdots,T_{\text{d}}(N)\right]^{\text{T}}$,
$\boldsymbol{\Phi}=\left[\Phi(1),\Phi(2),\cdots,\Phi(N)\right]^{\text{T}}$ and
${\bf P}=\left[P(1),P(2),\cdots,P(N)\right]^{\text{T}}$ as the variable vector
of $N$ frames of pilot length, transmission length, beamwidth and power
allocation, respectively. Then, this optimization problem can be formulated as

$\displaystyle\underset{{\bf T_{\text{s}}},{\bf
T_{\text{d}}},\boldsymbol{\Phi},{\bf P}}{\max}$
$\displaystyle\quad\frac{1}{N}\sum_{i=1}^{N}\eta_{\text{SE}}(i)$ (32) s.t.
$\displaystyle\text{C1}:\gamma_{e}(i)\geq\gamma_{\text{th}},\forall i$ (32a)
$\displaystyle\text{C2}:T_{\text{s}}(i)>0,T_{\text{d}}(i)>0,\forall i$ (32b)
$\displaystyle\text{C3}:\frac{T_{\text{s}}(i)}{T_{0}},\frac{T_{\text{d}}(i)}{T_{0}}\in\mathbb{Z},\forall
i$ (32c)
$\displaystyle\text{C4}:\Phi_{\text{min}}\leq\Phi(i)\leq\Phi_{\text{max}},\forall
i$ (32d) $\displaystyle\text{C5}:P(i)>0,\sum_{i=1}^{N}P(i)\leq
P_{\text{max}},\forall i$ (32e)

where C1 is the communication transmission constraint, C2 and C3 are
constraints on pilot and transmission durations, and C4 shows the constraint
on beamwidth, limited by the array size. The constraint C5 ensures that the
transmitted powers are nonnegative and cannot exceed a maximum value, where
$P_{\text{max}}$ is the total transmitted powers of the UAV for $N$ frames.
Due to the non-convex objective function and the presence of mixed discrete
and continuous constraints, this optimization problem can not be optimally
solved using current optimization tools.

##  III Efficient Iterative Algorithm

Since the frame structure, beamwidth and power are highly coupled, obtaining a
global optimal solution is challenging. In the section, we aim to find a sub-
optimal solution using an efficient iterative algorithm by optimizing one
variable while keeping others fixed. Specifically, we first relax the
constraints and approximate the objective function to optimize transmission
duration. Subsequently, the pilot length is optimized by a specifical solver.
Then, the closed-form expression of beamwidth is given using the geometric
theory. Next, the optimal power allocation is provided using the Karush-Kuhn-
Tucker (KKT) conditions. Finally, we analyze the complexity of overall
algorithm.

###  III-A Transmission Duration Determination

In this subsection, we will determine the transmission duration with fixed
other variables. Obviously, the system average SE is maximum when each frame
$i$ is maximized. Consequently, we will analyze the optimization method in one
frame, and we omit the index $i$ for simplicity.

Given the pilot length $T_{\text{s}}$, beamwdith $\Phi$ and power allocation
$P$, the $\gamma$ is fixed and the SE of current frame is calculated by

$\eta_{\text{SE}}=\frac{T_{\text{d}}}{T_{\text{s}}+T_{\text{d}}}\log_{2}\left(\frac{A_{1}}{A_{2}-2k\gamma^{2}\kappa^{T_{\text{d}}A_{3}}}\right),$
(33)

where $A_{1}=1+k\gamma+\gamma+k\gamma^{2}$,
$A_{2}=1+k\gamma+\gamma+2k\gamma^{2}$ and $A_{3}=\frac{f_{d}}{0.423BT_{0}}$.

To tackle the non-convexity of (33), we employ the successive convex
optimization method to approximate it with a convex form using the first-order
Taylor expansion around a given local point $T_{\text{d}}^{r}$. Therefore, the
approximated objective function can be expressed as

$\tilde{\eta}_{\text{SE}}^{r}=A_{4}^{r}\left(T_{\text{d}}-T_{\text{d}}^{r}\right)+A_{5}^{r},$
(34)

where

$\displaystyle A_{4}^{r}=$
$\displaystyle-\frac{T_{\text{d}}^{r}}{T_{\text{s}}+T_{\text{d}}^{r}}\log_{2}e\left(\frac{2k\gamma^{2}\ln\kappa
A_{3}\kappa^{T_{\text{d}}^{r}A_{3}}}{A_{2}-2k\gamma^{2}\kappa^{T_{\text{d}}^{r}A_{3}}}\right)$
(35)
$\displaystyle+\frac{T_{\text{s}}}{T_{\text{s}}+T_{\text{d}}^{r}}\log_{2}\left(\frac{A_{1}}{A_{2}-2k\gamma^{2}\kappa^{T_{\text{d}}^{r}A_{3}}}\right),$
(36)

$A_{5}^{r}=\frac{T_{\text{d}}^{r}}{T_{\text{s}}+T_{\text{d}}^{r}}\log_{2}\left(\frac{A_{1}}{A_{2}-2k\gamma^{2}\kappa^{T_{\text{d}}^{r}A_{3}}}\right).$
(37)

Moreover, there exists a maximum transmission duration
$T_{\text{d}}^{\text{max}}$, which is calculated by C1 as follows

$\displaystyle\gamma_{\text{th}}\leq\frac{\gamma\left(1-\delta_{\textbf{h}}^{2}\right)}{1+\gamma\delta_{\textbf{h}}^{2}},\quad\delta_{\textbf{h}}^{2}\leq\frac{\gamma-\gamma_{\text{th}}}{\gamma+\gamma\gamma_{\text{th}}}.$
(38)

Plugging (9) into the equation (38), we can obtain the maximum transmission
duration by

$T_{\text{d}}^{\text{max}}=\frac{\ln\left(\frac{2k\gamma+1}{1+k\gamma}-\frac{\gamma-\gamma_{\text{th}}}{\gamma+\gamma\gamma_{\text{th}}}\right)A_{4}}{\ln(\kappa)f_{d}},$
(39)

where $A_{4}=\frac{1+k\gamma}{2k\gamma}T_{0}$.

The constraint C3 renders the integer limitations of $T_{\text{d}}^{r}$. To
address this, we relax this constraint into continuous space. Consequently,
the optimization problem for one frame structure can be replaced by the
following problem

$\displaystyle\underset{{T_{\text{d}}}}{\max}$
$\displaystyle\quad\tilde{\eta}_{\text{SE}}^{r}$ (40) s.t. $\displaystyle
0<T_{\text{d}}\leq T_{\text{d}}^{\text{max}}.$

In this way, the optimization problem becomes a standard convex optimization
problem, enabling us to solve it via CVX solver. Then, we can obtain the
transmission duration
$T_{\text{d}}^{*}=T_{0}\lfloor\frac{T_{\text{d}}}{T_{0}}\rfloor$

###  III-B Pilot Length Optimization

Similar to determining transmission duration, the system’s average SE is
maximized by ensuring each frame’s SE reaches its peak. Therefore, we will
also optimize the pilot length in one frame while keeping other variables
fixed. The average SE can be expressed as

$\eta_{\text{SE}}=\frac{n}{k+n}\log_{2}\left(\frac{k\gamma^{2}+k\gamma+B_{1}}{2k\gamma^{2}B_{2}+k\gamma+B_{1}}\right),$
(41)

where $n$, $k$ are the symbol numbers of transmission and pilot, respectively.
The term $B_{1}=\gamma+1$ and $B_{2}=1-\alpha_{n,k+n}$. Maximizing
$\eta_{\text{SE}}$ is equivalent to minimizing $-\eta_{\text{SE}}$. Then this
subproblem can be reformulated as

$\displaystyle\underset{T_{\text{s}}}{\min}$
$\displaystyle\quad-{\eta}_{\text{SE}}$ (42) s.t. $\displaystyle
T_{\text{s}}>0,$ $\displaystyle k=\frac{T_{\text{s}}}{T_{0}},k\in\mathbb{Z}.$

Due to the complicated objective function (41), this is a standard nonlinear
integer programming problem (NLIP), which is difficult to solve directly. In
this paper, we address this NLIP problem using NOMAD solver in MATLAB OPTI
Toolbox, which can find solutions in several iterations. After that, we can
obtain the optimal pilot length $T_{\text{s}}^{*}$.

###  III-C Beamwidth Design

When the pilot and transmission duration are fixed, the channel estimation
error decreases by increasing the SNR. According to our antenna model in (6),
a narrower beamwidth corresponds to higher antenna gain. However, if the
$\theta$ and $\varphi$ are out of the beam, the antenna gain trends toward
zero. That indicates that high localization accuracy offers the high beam
alignment and small beamwidth, which contributes the high performance on the
SNR of the connected link. Similar to the pervious descriptions, the average
SE is maximized when the SE of each frame is maximized. Here, we provide a
Proposition to demonstrate the relationship between beamwidth and localization
accuracy.

Proposition 3 [7]: Given data transmission duration $T_{\text{d}}$, pilot
length $T_{\text{s}}$ and transmitted power $P$, we can obtain the
localization accuracy of UAV and GN by (30), denoted by $L_{\text{u}}^{2}$ and
$L_{\text{n}}^{2}$, respectively. Denote horizontal distance between UAV and
GN as
$d_{h}=\sqrt{{\left(p_{\text{u}}^{x}(t)-p_{\text{n}}^{x}(t)\right)^{2}+\left(p_{\text{u}}^{y}(t)-p_{\text{n}}^{y}(t)\right)^{2}}}$.
Then, the optimal beamwidth can be calculated in the following two cases.

1\. Case 1: When $d_{h}\geq l_{\text{u}}+2l_{\text{n}}$

$\displaystyle\Phi^{*}=$
$\displaystyle\max\left\\{\arctan\left(\frac{d_{1}}{d_{z}}\right)-\arctan\left(\frac{d_{2}}{d_{z}}\right),\Theta_{\text{min}}\right\\},$
(43)

where $d_{1}={d_{h}-l_{\text{u}}+2l_{\text{n}}}$,
$d_{2}=d_{h}-l_{\text{u}}-2l_{\text{n}}$ and
$d_{z}=p_{\text{u}}^{z}-p_{\text{n}}^{z}$.

2\. Case 2: When $d_{h}\leq l_{\text{u}}+2l_{\text{n}}$

$\displaystyle\Phi^{*}=$
$\displaystyle\max\left\\{2\arctan\left(\frac{2l_{\text{n}}}{p_{\text{u}}^{z}-p_{\text{n}}^{z}}\right),\Theta_{\text{min}}\right\\}.$
(44)

Proof: The similar proof illustration can be found in the Appendix B of our
pervious investigation [7]. However, unlike the proof in our previous
research, we consider the imperfection in UAV location in this paper.
Therefore, the optimal beamwidth should introduce the uncertainty of UAV
location.

###  III-D Power Allocation

Given other variables, a higher transmitted power will lead to a higher SNR,
better localization accuracy, and a lower channel estimation error. Thus, the
average SE monotonically increases with the transmitted power. Therefore, the
maximum SE is achieved when the constraint C5 becomes an equality. To ensure
the constraint, we can obtain the lower bound of transmitted power
$P_{\text{min}}(i)$ at each frame $i$ by solving following equation.

$D_{1}\gamma(i)^{2}+\gamma_{\text{th}}\left(k+1\right)\gamma(i)+\gamma_{\text{th}}=0,$
(45)

where $D_{1}=2kB_{2}\gamma_{th}+2kB_{2}-k$. This equation can be easily proven
to have one positive root. Then, this non-linear convex subproblem can be
formulated as an equivalent form

$\displaystyle\underset{\bf P}{\min}$
$\displaystyle\quad-\sum_{i=1}^{N}\eta_{\text{SE}}(i)$ (46) s.t.
$\displaystyle P_{\text{min}}(i)-P(i)\leq 0,\forall i$
$\displaystyle\sum_{i=1}^{N}P(i)-P_{\text{max}}=0,\forall i$

Under this formulation, the power allocation of each frame can be determined
using Lagrangian multiplier method with KKT conditions. The Lagrangian
function for this problem is expressed as

$\displaystyle\mathcal{L}({\bf
P},{\boldsymbol{\lambda}},\mu)=-\sum_{i=1}^{N}\eta_{\text{SE}}(i)+$
$\displaystyle\sum_{i=1}^{N}\lambda(i)\left(P_{\text{min}}(i)-P(i)\right)$
$\displaystyle+\mu\left(\sum_{i=1}^{N}P(i)-P_{\text{max}}\right),$ (47)

where $\lambda(i)\geq 0$ and $\mu$ are the lagrangian multipliers. According
to the KKT conditions, we have

$\displaystyle\left\\{\begin{aligned}
&-\eta_{\text{SE}}^{\prime}(i)-\lambda(i)+\mu=0,\quad\forall i\\\
&\lambda(i)\left(P_{\text{min}}(i)-P(i)\right)=0,\quad\forall i\\\
&\sum_{i=1}^{N}P(i)-P_{\text{max}}=0,\\\ \end{aligned}\right.$ (48)

where
$\eta_{\text{SE}}^{\prime}(i)=\frac{\partial\eta_{\text{SE}}(i)}{\partial
P(i)}$ is the first derivative of $\eta_{\text{SE}}(i)$. To guarantee the
solvability of this problem, the maximum power should satisfy
$P_{\text{max}}\geq NP_{\text{min}}$.

In the second equation of (48), if $P_{\text{min}}(i)\neq P(i)$, then
$\lambda(i)=0$. By substituting $\lambda(i)$ into the first equation and
combining it with the equations of the first condition in (48) and the third
condition, we can solve for the $i+1$ variables with $i+1$ equations. Then, we
can obtain the power allocation result of each frame with $P(i)^{\dagger}$.
Consequently, the optimal power of each frame $i$ is given by

$P(i)^{*}=\max\left\\{P(i)^{\dagger},P_{\text{min}}\right\\}.$ (49)

###  III-E Overall Algorithm and Complexity Analysis

The overall iterative algorithm is summarized in Algorithm 1. Here, we present
the analysis of the complexity for our proposed method. Denote $I$ as the
number of iteration for convergency. For optimizing the transmission duration,
the major complexity is solving $N$ approximated convex problems with
complexity $N\mathcal{O}(c(1))$, where $C(1)$ represents the computational
complexity by employing interior-point method with 1 variable [26]. For
optimizing the pilot length, the major complexity lies in solving the NLIP
with complexity $N\mathcal{O}(\log_{2}k(i))$, where $k(i)$ is the pilot length
of frame $i$. The beamwidth and power allocation are determined by the closed-
form. Consequently, the total complexity of our proposed algorithm is
$N\mathcal{O}(c(1)+\log_{2}k(i))$.

Algorithm 1 Proposed Efficient Iterative Algorithm

1: Set the initial state$\left({\bf T}_{\text{d}}^{0},{\bf
T}_{\text{s}}^{0},\boldsymbol{\Phi}^{0},{\bf P}^{0}\right)$ and the iteration
number $I=1$.

2: Repeat

3: for $i=1:N$ do

4: Given $\left({\bf T}_{\text{s}}^{I-1},\boldsymbol{\Phi}^{I-1},{\bf
P}^{I-1}\right)$, solve problem (40) for frame $i$ to obtain the optimal
transmission duration $T_{\text{d}}^{*}$, update ${\bf
T}_{\text{d}}^{I}(i)=T_{\text{d}}^{*}$.

5: end for

6: for $i=1:N$ do

7: Given $\left({\bf T}_{\text{d}}^{I},\boldsymbol{\Phi}^{I-1},{\bf
P}^{I-1}\right)$, solve problem (42) for frame $i$ to obtain the optimal pilot
length $T_{\text{s}}^{*}$, update ${\bf
T}_{\text{s}}^{I}(i)=T_{\text{s}}^{*}$.

8: end for

9: for $i=1:N$ do

10: Given $\left({\bf T}_{\text{d}}^{I},{\bf T}_{\text{s}}^{I},{\bf
P}^{I-1}\right)$, obtain the optimal beamwidth $\Phi^{*}(i)$ of frame $i$ by
(43) (44), update $\boldsymbol{\Phi}^{I}(i)=\Phi^{*}(i)$.

11: end for

12: Given $\left({\bf T}_{\text{d}}^{I},{\bf
T}_{\text{s}}^{I},\boldsymbol{\Phi}^{I}\right)$, obtain the optimal power
allocation result by (49), update into ${\bf P}^{I}$.

13: Until the objective function $\sum_{i=1}^{N}\eta_{\text{SE}}(i)$
converges.

##  IV Numerical Simulations

In this section, we thoroughly evaluate the proposed UAV aided localization
and communication method via numerical simulations. The size of the simulation
scenario is $1000m$$\times$$200m$$\times$$100m$. Unless otherwise specified,
the main simulation parameters are presented in Table I.

TABLE I: Main Simulation Parameters Parameters | Value
---|---
Symbol period |  $T_{0}=66.7$ [$\mu\text{s}$]
Carrier frequency |  $f_{c}=4.9$ [GHz]
Effective bandwith |  $\zeta=1$ [MHz]
Antenna reference gain |  $G_{0}=2.28$ [21]
Baseband carrier correlation |  $\chi=\sqrt{0.32}$[27]
Noise power |  $\sigma_{0}^{2}=-110$ [dBm] [28]
The channel gain at 1 $\rm{m}$ |  $\beta_{0}=-80$ [dB]
Threshold of SNR |  $\gamma_{\text{th}}=3$ [dB]
Number of antenna | $N_{\text{a}}=8$
Minimum beamwidth | $\Phi_{\text{min}}=5^{\circ}$
Motion noise |  $\sigma_{x}=\sigma_{y}=\sigma_{z}=0.005$ $\rm{m}$
Channel correlation level | $\kappa=0.8$

To highlight the advantages of our framework, we compare the proposed
efficient iterative method with a lower bound and three benchmark schemes in
the following.

  * •

Upper bound: In this scheme, we employs an exhaustive method for this joint
optimization problem, where the beamwidth and power allocation are executed
using closed-form expressions in (43) (44) and (49).

  * •

Benchmark 1: In this scheme, we optimize the transmission duration, pilot
length and power allocation without considering the localization. This
optimization problem is also solved using the iterative algorithm.

  * •

Benchmark 2: This scheme combines localization operation with a fixed frame
structure. The beamwidth and power allocation are also given by our
derivations.

  * •

Benchmark 3: In this scheme, we adopt the heuristic algorithm of particle
swarm optimization (PSO) to find suboptimal solutions for the optimization
method. The related parameters of this algorithm can be found in [29].

Figure 3: Convergence performance of proposed method for various maximum total
transmitted power.

We first analyze the convergence behavior of our proposed iterative algorithm
for various maximum transmitted power settings. This analysis is conducted
with $N=100$ and a UAV-GN relative velocity of $50\rm{m/s}$. As illustrated in
Fig. 3, the average SE of GN always converges within 10 iterations. Higher
transmitted power levels, indicating the higher SNR, lead to faster
convergence rates. Specifically, under the high SNR condition with
$P_{\text{max}}=8\rm{W}$, our method converges within 3 iterations.

We also conducted a comparative evaluation of the computational time between
the proposed method and PSO-based method with 200 particles. Using the same
simulation platform, the proposed method only takes about 5 seconds, while the
PSO-based method and the exhaustive research require over 50 and 300 seconds,
respectively. This demonstrates the notable computational efficiency of our
method.

Figure 4: Average SE performance of different conditions with fixed pilot
length versus transmission duration. Figure 5: Average SE performance of
different methods versus UAV-GN relative velocities. Figure 6: PTR of
different methods versus UAV-GN relative velocities.

Fig. 4 demonstrates the performance of the average SE with a fixed pilot
length ($k=5$) under different power limitations and relative velocities. As
shown in this figure, higher power limitations lead to better performance, and
there exists an optimal transmission duration $T\text{d}^{*}$ for each
condition. When $T_{\text{d}}<T_{\text{d}}^{*}$, the average SE increases with
transmission duration due to the increase in the first term
($\frac{T_{\text{s}}}{T_{\text{s}}+T\text{d}}$) of (31). When
$T_{\text{d}}\geq T_{\text{d}}$, the average SE decreases with transmission
duration due to the worsened channel estimation and location prediction
results. An interesting observation is that as the transmission duration
increases, different power limitations converge in the same level. That can be
explained by the effective SNR equation
$\gamma_{e}=\frac{\gamma\left(1-\delta_{\textbf{h}}^{2}\right)}{1+\gamma\delta_{\textbf{h}}^{2}}$,
which can be rewritten as
$\gamma_{e}=\frac{\left(1-\delta_{\textbf{h}}^{2}\right)}{\frac{1}{\gamma}+\delta_{\textbf{h}}^{2}}$.
Therefore, the effective SNR is highly dependent on the significant channel
estimation error $\sigma_{\textbf{h}}$. As observed in this figure (colored
straight dotted-line), higher maximum power provides a greater range for
convergence.

In Fig. 5, we evaluate the average SE performance across different UAV-GN
relative velocities, comparing it against the upper bound and other
benchmarks. Here, we set the maximum transmitted power
$P_{\text{max}}=4\rm{W}$. The results demonstrate that our proposed method
perform close to the upper bound, meaning that it is near-optimal, and
achieves a remarkable over 75% performance gain compared to benchmark 1,
highlighting the advantages of integrated localization and communication,
which also aligns with the motivation of our investigation. Furthermore, our
method shows significant performance gain compared to other benchmarks. As
velocities increase, our results also display the greater robustness than
benchmark 3, which fails to achieve reliable performance under lower effective
SNR conditions.

Fig. 6 shows the pilot and transmission duration ratio (PTR) of different
methods with the same conditions in Fig. 5. From the upper bound and our
proposed method, we can find that our integrated localization and
communication system increases the PTR compared to the communication-only
method. This improvement is attributed to the fact that more pilot overhead
can provide better localization accuracy, leading to a narrower beamwidth and
higher SNR. There exists a change of the upper bound at the low velocity
region. The PTR decreases at relative velocity $10\rm{m/s}$. This is due to at
this situation the SE is mainly influenced by the first term
$\frac{T_{\text{s}}}{T_{\text{s}}+T\text{d}}$ of (31). That also verifies the
performance gap in Fig. 5 between our method and the upper bound. As the
velocity increasing, the channel estimation error becomes grievous. The PTR
should increase to improve the performance localization and channel
estimation. The PTR of Our method perform close to the upper bound at the high
Doppler effect regions. In comparison to the fixed frame method, our proposed
method can reduce the pilot overhead to guarantee high SE when the Doppler
effect is small, while increasing the pilot ratio to compensate for the
channel estimation mismatch when the Doppler effect is rigorous.

Figure 7: Average SE performance of different methods versus maximum
transmitted powers. Figure 8: PTR of different methods versus maximum
transmitted powers.

In Fig. 7, we evaluate the average SE of different methods with various
transmitted power limitations. Here, we set the relative velocity at
$10\rm{m/s}$, and the PTR of fixed frame is adjusted to $1/30$ in the
benchmark 2. As expected, the average SE exhibits a positive correlation with
the total transmitted power, and our proposed method consistently maintains a
small gap with the upper bound. The benchmark 2 also demonstrates the good
system performance, highlighting the importance of localization operation. As
can be observed, the performance gap between our method and other benchmarks
is smaller with the increasing SNR. That phenomenon showcases the
effectiveness of our proposed method in low SNR regions.

In Fig. 8, we analyze the PTR for the simulation results in Fig. 7. Different
from the result in Fig. 6, where increasing the pilot overhead mitigates the
Doppler effect, this result reveals that it also needs to add the pilot ratio
or reduce the duration of transmission as transmitted power increases. A
shorter data transmission time implies reduced channel time-variant
characteristics, indicating that the channel estimation interval has a
significant impact compared to SNR in channel estimation. At high SNR levels,
the time frame structure tends to become fixed.

##  V Conclusion

In this paper, we explored a UAV-aided localization and communication system.
We formulated a joint frame structure, beamwidth, and power allocation
optimization problem, to maximize the average SE of UAV-GN link. Given this
non-convex problem, we developed an efficient iterative algorithm to obtain
sub-optimal solutions. Base on the numerical results, we can draw following
key conclusions (i) The proposed method closely approaches the upper bound
with remarkably low complexity, outperforming the communication-only method by
over 70% in average SE. (ii) The Doppler effect significantly influences
channel estimation error, and the PTR should increase in both high dynamic and
high SNR scenarios. (iii) For a specifical scenario, there exists a unique
frame structure to achieve optimal average SE. These findings provide valuable
insights for system design. In the future, the multi-user, computing analysis,
and trajectory management can be investigated.

## Appendix A: Proof of Proposition 1

The channel estimation is executed by the receiving pilot symbols. Denote
${\bf x}_{k}=\left[x(1),x(2),\cdots,x(k)\right]^{\text{T}}$ and ${\bf
y}_{k}=\left[y(1),y(2),\cdots,y(k)\right]^{\text{T}}$ be the input training
and received symbol vectors, respectively. According to the MMSE rule [12],
the estimated channel can be expressed as

$\widehat{h}_{k}=\sqrt{\gamma}(1+k\gamma)^{-1}{\bf x}_{k}^{\text{H}}{\bf
y}_{k},$ (50)

where $(\cdot)^{\text{H}}$ is the Hermitian operation. According to the
received signal model in (1), $y(k)=\sqrt{\gamma}{{h}}(k){x}(k)+{n}(k)$. In
this paper, we assume $x(k)$ has unit mean square, which means $|{\bf
x}_{k}|^{2}=k$. Then, the estimated channel can be rewritten

$\widehat{h}_{k}=\frac{k\gamma h(k)}{1+k\gamma}+\frac{\sqrt{\gamma}{\bf
x}_{k}^{\text{H}}{\bf n}_{k}}{1+k\gamma},$ (51)

where ${\bf n}_{k}=\left[n(1),n(2),\cdots,n(k)\right]^{\text{T}}$ is the noise
vector.

In this paper, the temporal channel correlation between two symbols can be
modeled as a first-order Markov process [13]

${h}_{n}=\prod_{i=k+1}^{n}\alpha_{k,i}h_{k}+\sqrt{1-\prod_{i=k+1}^{n}\alpha_{k,n}^{2}}\varsigma_{n},$
(52)

where $\varsigma_{n}$ is a Guassian random variable with zero mean and unit
variance, and $\alpha_{k,n}=\kappa^{\frac{f_{d}}{0.423B}}$ is the correlation
factor between $h_{k}$ and $h_{n}$. The term $\kappa$ is the level of
correlation, $f_{d}$ and $B$ are the Doppler frequency shift and system
bandwidth, respectively.

Therefore, given $k$ pilot symbols, the channel estimation error at the symbol
$n$ can be calculated by

$\displaystyle\delta_{\bf h}$
$\displaystyle={h}_{n}-\widehat{h}_{k}=\prod_{i=k+1}^{n}\alpha_{k,i}h_{k}$
$\displaystyle+\sqrt{1-\prod_{i=k+1}^{n}\alpha_{k,n}^{2}}\eta_{n}-\frac{k\gamma
h(k)}{1+k\gamma}+\frac{\sqrt{\gamma}{\bf x}_{k}{\bf n}_{k}}{1+k\gamma}.$ (53)

We take the MSE of (Appendix A: Proof of Proposition 1) to obtain
$\mathbb{E}\left\\{|\delta_{\bf h}|^{2}\right\\}$ with

$\displaystyle\delta_{\bf h}^{2}$
$\displaystyle=\left(\prod_{i=k+1}^{n}\alpha_{k,i}-\frac{k\gamma}{1+k\gamma}\right)\mathbb{E}\left\\{|h_{k}|^{2}\right\\}$
$\displaystyle+\left({1-\prod_{i=k+1}^{n}\alpha_{k,i}^{2}}\mathbb{E}\left\\{|\eta_{n}|^{2}\right\\}\right)+\frac{{k\gamma}}{\left(1+k\gamma\right)^{2}}$
$\displaystyle=\frac{1}{1+k\gamma}+{\frac{2k\gamma\\!\\!\left(1-\\!\\!\prod\limits_{i=k+1}\limits^{n}\\!\\!\alpha_{k,i}\right)}{\left(1+k\gamma\right)}}$
$\displaystyle=\frac{1}{1+k\gamma}+{\frac{2k\gamma\\!\\!\left(1-\alpha_{k,k+n}\right)}{\left(1+k\gamma\right)}}.$
(54)

Thus the proof of Proposition 1 is completed.

## Appendix B: Proof of Proposition 2

We convert the parameter space ${\boldsymbol{\Theta}}_{t}$ to the position
coordinates, and show their relationships as follows.

$\left\\{{\begin{array}[]{*{20}{l}}c\tau_{t}=\sqrt{\left(p_{\text{u}}^{x}(t)-p_{\text{n}}^{x}\right)^{2}+\left(p_{\text{u}}^{y}(t)-p_{\text{n}}^{y}\right)^{2}+\left(p_{\text{u}}^{z}(t)-p_{\text{n}}^{z}\right)^{2}}\\\
\varphi_{t}=\arctan\left(\frac{p_{\text{u}}^{y}(t)-p_{\text{n}}^{y}(t)}{p_{\text{u}}^{x}(t)-p_{\text{n}}^{x}(t)}\right)\\\
\theta_{t}=\arctan\left(\frac{p_{\text{u}}^{z}(t)-p_{\text{n}}^{z}(t)}{\sqrt{\left(p_{\text{u}}^{x}(t)-p_{\text{n}}^{x}\right)^{2}+\left(p_{\text{u}}^{y}(t)-p_{\text{n}}^{y}\right)^{2}}}\right)\end{array}}\right..$
(55)

We omit $t$ for simplicity. Without loss generality, we derive the direction
vector for localization of GN as follows.

${\bf q}_{r}=\left[\begin{array}[]{c}\frac{\partial\tau}{p_{\text{n}}^{x}}\\\
\frac{\partial\tau}{p_{\text{n}}^{y}}\\\
\frac{\partial\tau}{p_{\text{n}}^{z}}\end{array}\right]=\left[\begin{array}[]{c}-\frac{p_{\text{u}}^{x}-p_{\text{n}}^{x}}{d}\\\
-\frac{p_{\text{u}}^{y}-p_{\text{n}}^{y}}{d}\\\
-\frac{p_{\text{u}}^{z}-p_{\text{n}}^{z}}{d}\end{array}\right]=\left[\begin{array}[]{c}-\cos\varphi\cos\theta\\\
-\sin\varphi\cos\theta\\\ -\sin\theta\end{array}\right],$ (56)
$\\!\\!\\!\\!\\!\\!\\!\\!{\bf
q}_{\theta}=\left[\begin{array}[]{c}\frac{\partial\theta}{p_{\text{n}}^{x}}\\\
\frac{\partial\theta}{p_{\text{n}}^{y}}\\\
\frac{\partial\theta}{p_{\text{n}}^{z}}\end{array}\right]\\!\\!=\\!\\!\left[\begin{array}[]{c}\frac{\left(p_{\text{u}}^{x}-p_{\text{n}}^{x}\right)\left(p_{\text{u}}^{z}-p_{\text{n}}^{z}\right)}{\sqrt{\left(p_{\text{u}}^{x}-p_{\text{n}}^{x}\right)^{2}+\left(p_{\text{u}}^{y}-p_{\text{n}}^{y}\right)^{2}}d}\\\
\frac{\left(p_{\text{u}}^{y}-p_{\text{n}}^{y}\right)\left(p_{\text{u}}^{z}-p_{\text{n}}^{z}\right)}{\sqrt{{\left(p_{\text{u}}^{x}-p_{\text{n}}^{x}\right)^{2}+\left(p_{\text{u}}^{y}-p_{\text{n}}^{y}\right)^{2}}}d}\\\
-\frac{{\left(p_{\text{u}}^{x}-p_{\text{n}}^{x}\right)^{2}+\left(p_{\text{u}}^{y}-p_{\text{n}}^{y}\right)^{2}}}{d}\end{array}\right]\\!\\!=\\!\\!\left[\begin{array}[]{c}\frac{\cos\varphi\sin\theta}{d}\\\
\frac{\sin\varphi\sin\theta}{d}\\\ -\frac{\cos\theta}{d}\end{array}\right],$
(57) $\\!\\!\\!\\!\\!\\!\\!\\!{\bf
q}_{\varphi}=\left[\begin{array}[]{c}\frac{\partial\varphi}{p_{\text{n}}^{x}}\\\
\frac{\partial\varphi}{p_{\text{n}}^{y}}\\\
\frac{\partial\varphi}{p_{\text{n}}^{z}}\end{array}\right]\\!\\!=\\!\\!\left[\begin{array}[]{c}\frac{p_{\text{u}}^{y}-p_{\text{n}}^{y}}{\sqrt{\left(p_{\text{u}}^{x}-p_{\text{n}}^{x}\right)^{2}+\left(p_{\text{u}}^{y}-p_{\text{n}}^{y}\right)^{2}}}\\\
-\frac{p_{\text{u}}^{x}-p_{\text{n}}^{x}}{\sqrt{{\left(p_{\text{u}}^{x}-p_{\text{n}}^{x}\right)^{2}+\left(p_{\text{u}}^{y}-p_{\text{n}}^{y}\right)^{2}}}}\\\
0\end{array}\right]\\!\\!=\\!\\!\left[\begin{array}[]{c}\frac{\sin\varphi}{d\cos\theta}\\\
-\frac{\cos\varphi}{d\cos\theta}\\\ 0\end{array}\right].$ (58)

Then, the proof of proposition 2 is completed.

## References

  * [1] T. Liang, Z. Yu, T. Zhang, S. Zhou, W. Liu, D. Li, and Z. Niu, “Joint frame structure and beamwidth optimization for integrated localization and communication,” in _WCNC 2024 - IEEE Wireless Communications and Networking Conference_ , 2024, pp. 1–1.
  * [2] J. Liu, P. Tong, X. Wang, B. Bai, and H. Dai, “UAV-aided data collection for information freshness in wireless sensor networks,” _IEEE Transactions on Wireless Communications_ , vol. 20, no. 4, pp. 2368–2382, Dec. 2021.
  * [3] F. Liu, Y. Cui, C. Masouros, J. Xu, T. X. Han, Y. C. Eldar, and S. Buzzi, “Integrated sensing and communications: Toward dual-functional wireless networks for 6g and beyond,” _IEEE Journal on Selected Areas in Communications_ , vol. 40, no. 6, pp. 1728–1767, 2022.
  * [4] J. Kang, H. Wymeersch, C. Fischione, G. Seco-Granados, and S. Kim, “Optimized switching between sensing and communication for mmwave MU-MISO systems,” in _2022 IEEE International Conference on Communications Workshops (ICC Workshops)_ , 2022, pp. 498–503.
  * [5] Q. Wu, J. Xu, Y. Zeng, D. W. K. Ng, N. Al-Dhahir, R. Schober, and A. L. Swindlehurst, “A comprehensive overview on 5g-and-beyond networks with uavs: From communications to sensing and intelligence,” _IEEE Journal on Selected Areas in Communications_ , vol. 39, no. 10, pp. 2912–2945, 2021.
  * [6] Y. Liu, Y. Wang, X. Shen, J. Wang, and Y. Shen, “Uav-aided relative localization of terminals,” _IEEE Internet of Things Journal_ , vol. 8, no. 16, pp. 12 999–13 013, 2021.
  * [7] T. Liang, T. Zhang, Q. Wu, W. Liu, D. Li, Z. Xie, D. Li, and Q. Zhang, “Age of information based scheduling for UAV aided localization and communication,” _IEEE Transactions on Wireless Communications_ , pp. 1–1, 2023.
  * [8] Y. Shen and M. Z. Win, “On the accuracy of localization systems using wideband antenna arrays,” _IEEE Transactions on Communications_ , vol. 58, no. 1, pp. 270–280, 2010.
  * [9] T. Liang, T. Zhang, J. Yang, D. Feng, and Q. Zhang, “UAV-aided positioning systems for ground devices: Fundamental limits and algorithms,” _IEEE Internet of Things Journal_ , vol. 9, no. 15, pp. 13 470–13 485, 2022.
  * [10] F. Mirhosseini, A. Tadaion, and S. M. Razavizadeh, “Spectral efficiency of dense multicell massive MIMO networks in spatially correlated channels,” _IEEE Transactions on Vehicular Technology_ , vol. 70, no. 2, pp. 1307–1316, 2021.
  * [11] J. Kang, N. Garcia, H. Wymeersch, C. Fischione, G. Seco-Granados, and S. Kim, “Optimizing the mmwave channel estimation duration by rate prediction,” _IEEE Communications Letters_ , vol. 25, no. 2, pp. 555–559, 2021.
  * [12] M. Mousaei and B. Smida, “Optimizing pilot overhead for ultra-reliable short-packet transmission,” in _2017 IEEE International Conference on Communications (ICC)_ , 2017, pp. 1–5.
  * [13] J. Cao, X. Zhu, Y. Jiang, Y. Liu, Z. Wei, S. Sun, and F.-C. Zheng, “Independent pilots versus shared pilots: Short frame structure optimization for heterogeneous-traffic urllc networks,” _IEEE Transactions on Wireless Communications_ , vol. 21, no. 8, pp. 5755–5769, 2022.
  * [14] J. Choi and J. Park, “Mimo design for internet of things: Joint optimization of spectral efficiency and error probability in finite blocklength regime,” _IEEE Internet of Things Journal_ , vol. 8, no. 20, pp. 15 512–15 521, 2021\.
  * [15] B. Zhu, Z. Zhang, and J. Cheng, “Outage analysis and beamwidth optimization for positioning-assisted beamforming,” _IEEE Communications Letters_ , vol. 26, no. 7, pp. 1543–1547, 2022.
  * [16] T. Li, X. Wang, P. Fan, and T. Riihonen, “Position-aided large-scale mimo channel estimation for high-speed railway communication systems,” _IEEE Transactions on Vehicular Technology_ , vol. 66, no. 10, pp. 8964–8978, 2017.
  * [17] G. Kwon, A. Conti, H. Park, and M. Z. Win, “Joint communication and localization in millimeter wave networks,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 15, no. 6, pp. 1439–1454, 2021.
  * [18] L. Zhu, J. Zhang, Z. Xiao, X. Cao, X.-G. Xia, and R. Schober, “Millimeter-wave full-duplex uav relay: Joint positioning, beamforming, and power control,” _IEEE Journal on Selected Areas in Communications_ , vol. 38, no. 9, pp. 2057–2073, 2020.
  * [19] F. Maschietti, D. Gesbert, P. de Kerret, and H. Wymeersch, “Robust location-aided beam alignment in millimeter wave massive mimo,” in _GLOBECOM 2017 - 2017 IEEE Global Communications Conference_ , 2017, pp. 1–6.
  * [20] A. Mohammadi, M. Rahmati, and H. Malik, “Location-aware beamforming for mimo-enabled uav communications: An unknown input observer approach,” _IEEE Sensors Journal_ , vol. 22, no. 8, pp. 8206–8215, 2022.
  * [21] Z. Yang, C. Pan, M. Shikh-Bahaei, W. Xu, M. Chen, M. Elkashlan, and A. Nallanathan, “Joint altitude, beamwidth, location, and bandwidth optimization for UAV-enabled communications,” _IEEE Communications Letters_ , vol. 22, no. 8, pp. 1716–1719, 2018.
  * [22] R. K. Mallik, M. R. Bhatnagar, and S. P. Dash, “Fractional pilot duration optimization for SIMO in rayleigh fading with MPSK and imperfect CSI,” _IEEE Transactions on Communications_ , vol. 66, no. 4, pp. 1732–1744, 2018\.
  * [23] T. Wang, H. Zhao, and Y. Shen, “High-accuracy localization using single-anchor ultra-wide bandwidth systems,” in _2019 IEEE/CIC International Conference on Communications in China (ICCC)_ , 2019, pp. 59–63.
  * [24] Y. Wang, Y. Wu, and Y. Shen, “On the resolution limits for MIMO localization,” _IEEE Communications Letters_ , vol. 23, no. 3, pp. 462–465, 2019.
  * [25] H. Ren, C. Pan, Y. Deng, M. Elkashlan, and A. Nallanathan, “Joint pilot and payload power allocation for massive-mimo-enabled urllc iiot networks,” _IEEE Journal on Selected Areas in Communications_ , vol. 38, no. 5, pp. 816–830, 2020.
  * [26] S. Boyd and L. Vandenberghe, _Convex Optimization_. Cambridge University Press, 2004.
  * [27] T. Zhang, A. F. Molisch, Y. Shen, Q. Zhang, H. Feng, and M. Z. Win, “Joint power and bandwidth allocation in wireless cooperative localization networks,” _IEEE Transactions on Wireless Communications_ , vol. 15, no. 10, pp. 6527–6540, 2016.
  * [28] D. Li, “Ergodic capacity of intelligent reflecting surface-assisted communication systems with phase errors,” _IEEE Communications Letters_ , vol. 24, no. 8, pp. 1646–1650, 2020.
  * [29] T. Liang, W. Liu, J. Yang, and T. Zhang, “Age of information based scheduling for UAV aided emergency communication networks,” in _ICC 2022 - IEEE International Conference on Communications_ , 2022, pp. 5128–5133.


# Uncertainty Aware Learning for Language Model Alignment

Yikun Wang, Rui Zheng,22footnotemark: 211footnotemark: 1 Liang Ding, Qi
Zhang,22footnotemark: 2 Dahua Lin, Dacheng Tao
{yikunwang19, rzheng20<EMAIL_ADDRESS><EMAIL_ADDRESS>
{liangding.liam<EMAIL_ADDRESS>
Yikun and Rui contributed equally.School of Computer Science, Fudan
University, China.The University of Sydney, Austrilia. Corresponding
author.The Chinese University of Hong Kong, Hong Kong.Shanghai Artificial
Intelligence Laboratory, China.Nanyang Technological University, Singapore.

###### Abstract

As instruction-tuned large language models (LLMs) evolve, aligning pretrained
foundation models presents increasing challenges. Existing alignment
strategies, which typically leverage diverse and high-quality data sources,
often overlook the intrinsic uncertainty of tasks, learning all data samples
equally. This may lead to suboptimal data efficiency and model performance. In
response, we propose uncertainty-aware learning (UAL) to improve the model
alignment of different task scenarios, by introducing the sample uncertainty
(elicited from more capable LLMs). We implement UAL by a simple fashion –
adaptively setting the label smoothing value of training according to the
uncertainty of individual samples. Analysis shows that our UAL indeed
facilitates better token clustering in the feature space, validating our
hypothesis. Extensive experiments on widely used benchmarks demonstrate that
our UAL significantly and consistently outperforms standard supervised fine-
tuning. Notably, LLMs aligned in a mixed scenario have achieved an average
improvement of 10.62% on high-entropy tasks (i.e., AlpacaEval leaderboard),
and 1.81% on complex low-entropy tasks (i.e., MetaMath and GSM8K).

Uncertainty Aware Learning for Language Model Alignment

Yikun Wang,††thanks: Yikun and Rui contributed equally.††thanks: School of
Computer Science, Fudan University, China. Rui Zheng,22footnotemark:
211footnotemark: 1 Liang Ding,††thanks: The University of Sydney,
Austrilia.††thanks:  Corresponding author. Qi Zhang,22footnotemark: 2 Dahua
Lin,††thanks: The Chinese University of Hong Kong, Hong Kong.††thanks:
Shanghai Artificial Intelligence Laboratory, China. Dacheng Tao††thanks:
Nanyang Technological University, Singapore. {yikunwang19, rzheng20,
<EMAIL_ADDRESS><EMAIL_ADDRESS>{liangding.liam,
<EMAIL_ADDRESS>

##  1 Introduction

Figure 1: Illustration of feature clustering. Compared to SFT, UAL-based
models show more convergence in the feature space, which we detailed our
exploration in Section 4.

Large language models (LLMs), represented by GPT-4 OpenAI (2023), Claude, and
Llama2 Team (2023), have recently achieved significant success in a series of
natural language generation and understanding tasks Qin et al. (2023); Zhong
et al. (2023a); Peng et al. (2023b); Lu et al. (2023). The emergence of
alignment methods has further enhanced the capabilities of LLMs, for example,
the ability to follow human instructions or to achieve better zero-shot
performance. The current popular alignment approaches including RLHF Li et al.
(2023); Zheng et al. (2023), consider supervised fine-tuning (SFT) an
essential part that contributes significantly.

It is undeniable that the current SFT paradigm achieves considerable success.
Some previous studies find that high-quality, complex, and diverse data
contribute significantly to alignment Zhou et al. (2023); Liu et al. (2023).
However, the data can exhibit different levels of uncertainty. For data points
in highly technical, scientific, or specialized settings, context may have
little or no ambiguity and a limited set of correct answers. In contrast,
other dialogues might feature varied and dynamic social contexts with
idiomatic language uses. Appendix A presents a pair of such examples.
Nevertheless, the common SFT paradigm applies the same level of supervision to
all samples in the training set, overlooking the intrinsic uncertainty of the
data.

Furthermore, due to the phenomenon of catastrophic forgetting Luo et al.
(2023); Ramasesh et al. (2022); Aleixo et al. (2023), SFT-aligned models may
perform worse than their foundational models. The Deepseek technical report
reveals that the SFT model often underperforms compared to its base model on
several benchmarks Team (2024), while LLMs generally show diminished
performance on general tasks when forging their agent capability Zeng et al.
(2023). In our experiments, we observe that the standard SFT paradigm
frequently leads to model degradation, resulting in decreased performance on
some benchmarks, despite providing overall improvement. For instance, this is
evident in parts of the commonsense benchmarks, such as MMLU. It is essential
to align pretrained models while mitigating degradation issues as effectively
as possible. Therefore, we propose:

Uncertainty Hypothesis. In an ideal paradigm, to further enhance alignment
performance, the model should attend to samples differently based on their
properties during alignment. Specifically, the model should impose stricter
constraints when attending to more certain examples, as these samples exhibit
less uncertainty and fewer variations, while maintaining relaxed constraints
for highly uncertain examples.

To design our uncertainty-aware learning (UAL) method and address potential
model degradation caused by SFT, we propose measuring uncertainty through a
coarsely-grained approach that incorporates an autonomous judge (e.g., GPT-4)
to assess the uncertainty of each sample in the training set. Upon obtaining
the uncertainty estimations, we linearly map them into our adaptive label
smoothing training pipeline. Further details of our algorithm are presented in
Section 2.

UAL significantly enhances the performance of instruction-tuned models. We
apply this method across various model architectures and alignment datasets,
observing that the UAL paradigm consistently outperforms the vanilla SFT
paradigm on prominent benchmarks, including MMLU and TruthfulQA Lin et al.
(2022), among others. As Figure 2 shows, UAL also helps the model improve in
different scenarios. Moreover, compared to its foundational model, as Figure 1
presents, the SFT-aligned model brings same-class tokens closer together in
the feature space, and UAL enhances this tendency even further. This strongly
correlates with the superior performance of UAL-aligned models across
benchmarks and scenarios, offering an important explanation for UAL’s
outperformance over conventional SFT.

#### Contributions.

Our contributions are summarized threefold:

  * •

Due to the limitations of the current supervised finetuning (SFT) paradigm, we
propose the uncertainty-aware learning (UAL) approach to mitigate the
alignment degradation problem and improve model performance in high-entropy
and low-entropy scenarios.

  * •

Based on an intuitive design concept, our UAL paradigm is simple to implement,
making it possible to improve any language model’s alignment.

  * •

We conducted extensive experiments to prove the efficiency of our method and
conducted in-depth analyses to provide some insights (i.e. feature clustering)
into the mechanism.

Figure 2: Mistral-7B Jiang et al. (2023) aligned with UAL on the mixed
scenario dataset has an improvement of 10.62% on the high entropy scenario and
1.81% on the low entropy scenario, compared with SFT. Evaluation results from
models solely SFT aligned with high-entropy data (i.e., HE aligned with
ShareGPT-6k) or low-entropy data (i.e., LE aligned with MetaMath-6k) are also
provided for reference.

##  2 Uncertainty Approach for Characterizing Data

In this section, we present a insightful study of the characteristics of data
uncertainty property in instruction tuning. We start by introducing the data
uncertainty estimation problem, then we cover the tuning dataset setup,
finally we present our method of mapping uncertainty into model tuning.

###  2.1 Uncertainty Estimation

For further enhance the model performance in the process of alignment, we
propose to impose variant levels of loss constraint for different data based
on their uncertainty, which has a similar intrinsic to information entropy.
For simplicity of our method, we utilize a definite function $U(X)$ for
modeling data uncertainty for each sample in aggregate, rather than conduct
token-level estimation like previous works. More specifically speaking, for a
tuning dataset $X=\\{x_{1},x_{2},...,x_{n}\\}$, where each $x_{i}$ represents
an individual data point in the format of instruction-response. Our goal is to
collect uncertainty $U(x_{i})$ of each sample $x_{i}$.

Some researchers have found that LLMs could express uncertainty accurately
Xiong et al. (2023), referred to as confidence elicitation, which is essential
in ensuring reliable and trustworthy decision-making processes. In our
following empirical study, we will employ a more capable model (w.r.t. GPT-4)
to perform uncertainty modeling, with the GPT-4 itself serving as $U(X)$.

###  2.2 Tuning Dataset Setup

Dataset | Source | Size
---|---|---
LIMA | - | 1030
Deita | ShareGPT, UltraGPT | 6k
ShareGPT-6k | ShareGPT | 6k
MetaMath-6k | MetaMathQA | 6k
Mixed-6k | ShareGPT, MetaMathQA | 6k

Table 1: Statistics of the alignment datasets employed in the empirical study,
LIMA and Deita are from Zhou et al. (2023) and Liu et al. (2023) respectively,
and the others are sampled and synthesized by employing the data selection
technique DEITA Liu et al. (2023).

The large language model research community has developed a consensus: a small
amount of human-collected and high-quality data such as LIMA containing around
1k examples, can already achieve satisfying alignment performance Zhou et al.
(2023); some researchers have even demonstrated that a limited size of high-
quality, complex and diverse dataset could reach or even surpass the
effectiveness of large scale datasets such as Alpaca-52k Liu et al. (2023). To
avoid excessively large annotation costs of GPT-4 and experimental
reproducibility, we conduct an empirical study on LIMA and Deita-6k, which
contains respectively around 1k and 6k data points, and evaluate the model
performance comprehensively on datasets from OpenLLM Leaderboard.

To further demonstrate that our method could simultaneously improve the
model’s capability in both high-entropy and low-entropy scenarios, and to
prove the superiority of our method compared to SFT, we will sample several
following datasets from different scenarios: High Entropy: we employ the DEITA
method to sample high-quality, complex and diverse examples from ShareGPT,
which mainly contains various kinds of dialogues between users and GPT,
resulting a high-quality and high-entropy dataset ShareGPT-6k containing 6k
examples; Low Entropy: we employ DEITA Liu et al. (2023) to sample from
MetaMathQA Yu et al. (2023) with 395k examples, resulting in a high-quality
math question and answer datasets with 6k examples; Mixed Scenario: we aim to
compare compare the superiority of our method in a comprehensive setting
against standard SFT, and compare it with both the high-entropy and the low-
entropy models that respectively tuned with the high-entropy dataset and low-
entropy dataset, while ensuring a fair comparison. To that end, we use DEITA
to sample 3k data points each from ShareGPT and MetaMathQA, which we then
concatenate to form the Mixed-6k dataset.

###  2.3 Uncertainty Mapping

Label smoothing is a common technique for mitigating overfitting and enhancing
generalization ability in the field of deep learning, by combing the cross-
entropy loss, the constraint intensity could be adjusted by altering the
smooth value, a positive value within the interval from zero to one. Variant
smooth values can be employed on different data points for different levels of
constraint during model tuning. To satisfy our uncertainty hypothesis,
ensuring low-uncertainty samples are subject to stronger constraints and
meanwhile high-uncertainty samples are allowed looser constraints, the data
uncertainty values $U$ are mapped with a function $\mathcal{F}:U\rightarrow V$
to label smooth values $V$, every sample has a unique smooth value
$v_{i}=\mathcal{F}(u_{i})$ during training.

$\mathcal{F}(x_{i})=\min(h(u_{i}),v_{t})$ (1)
$v_{i}=\mathcal{F}(x_{i})\quad\text{s.t.}~{}\mathcal{E}(v_{i})=\alpha$ (2)

Model | Data Size / Alignment | ARC-c | HellaSwag | MMLU | TruthfulQA | Average
---|---|---|---|---|---|---
Llama-2 based Models
Llama-2-7B | \- / - | 63.21 | 75.12 | 60.61 | 53.37 | 60.08
Vicuna-7B-v1.5 | 125K / SFT | 71.23 | 89.32 | 67.40 | 52.43 | 70.10
Alpaca-2-7B | 52K / SFT | 64.54 | 87.04 | 63.68 | 46.26 | 65.38
LIMA-7B | 1K / SFT | 55.51 | 79.61 | 60.42 | 64.01 | 64.89
LIMA-7B (Ours) | 1K / UAL | 58.89 | 79.87 | 65.70 | 66.16 | 67.66
Deita-7B | 6K / SFT | 67.22 | 74.24 | 64.40 | 57.77 | 65.91
Deita-7B ($\text{Ours}^{-}$) | 6k / UAL | 66.88 | 75.01 | 63.81 | 60.18 | 66.47
Deita-7B (Ours) | 6K / UAL | 69.55 | 76.77 | 67.64 | 64.06 | 69.51
Mistral based Models
Mistral-7B | \- / - | 77.25 | 75.63 | 68.97 | 33.78 | 63.91
Deita-7B | 6K / SFT | 71.57 | 71.73 | 62.11 | 38.88 | 61.07
Deita-7B ($\text{Ours}^{-}$) | 6k / UAL | 78.92 | 76.86 | 66.08 | 33.29 | 63.79
Deita-7B (Ours) | 6K / UAL | 78.26 | 78.93 | 66.16 | 36.23 | 64.90

Table 2: The Open LLM Leaderboard benchmark evaluation results of Llama-2 Team
(2023) and Mistral-based Jiang et al. (2023) models employing inference (Best-
of-4, zero-shot) strategy. Bold indicates the models performing the best in
its counterparts from the same foundation model and tuned on the same
instruction-response dataset, with Underline indicates the models better than
its corresponding foundation models. - indicates the models aligned with the
PPL uncertainty.

Assume a linearly scaling function $h:U\rightarrow V$ with the form of
$h(u)=\beta u$, where $\beta$ is a scaling factor. To reflect our uncertainty
hypothesis and the simplicity of our approach, we directly employ the
truncated linear mapping $\mathcal{F}(U)$ to transform the uncertainty of an
instruction tuning example $u_{i}$ into a label smoothing value $v_{i}$.
Truncation means that the label smoothing value cannot exceed 1, hence The
form of $\mathcal{F}$ is indicated in equation 1, where $v_{t}$ represents the
maximum permissible label smoothing value, which is set to 0.99 in subsequent
experiments. To regulate the overall level of constraints during training, the
average smoothing value across all samples is set to a fixed value $\alpha$
(e.g. a common value of $0.1$).

#### Approach Outline.

In the first phase of the two-stage method, uncertainty estimation values are
elicited from a capable GPT-4 model, followed by a truncated linear mapping
from uncertainty estimation to sample-wise label smooth values in order to
fulfill our uncertainty hypothesis.

##  3 Experiments

In this section, we will introduce the experimental setup and present
evaluation results from comprehensive benchmarks, as well as improvements
observed in mixed scenarios, to demonstrate the superiority of our method. To
promote a deeper understanding of our approach, we conduct an ablation study.
Additionally, we present a case study and employ GPT-4 to further evaluate the
quality of responses generated by our method.

###  3.1 Experimental Setup

For uncertainty estimation, we utilize GPT-4 for the collection of aligned
datasets to estimate uncertainty, with the corresponding prompts displayed in
the Appendix C. Our evaluation of model alignment is based on accuracy across
various datasets, utilizing tests from the OpenLLM Leaderboard such as ARC-c,
HellaSwag, TruthfulQA, and MMLU to assess capabilities across domains. In
order to evaluate performance in both high-entropy and low-entropy scenarios,
we employ AlpacaEval Dubois et al. (2023) for general dialogue and GSM8k /
MetaMath for low-entropy contexts. For datasets on the OpenLLM Leaderboard, we
have also implemented the Best-of-N strategy to enlargen the performance
discrepancies among different models. The training settings are detailed in
the Appendix D.

###  3.2 Evaluation on Comprehensive Benchmarks

We instruction-tune Llama-2-7B respectively on the LIMA and the Deita dataset,
in order to demonstrate the superiority of the uncertainty-aware learning
(UAL) compared to normal SFT, as well as benchmark them against other strong
open-sourced instruction-tuned models including Alpaca-2-7B and
Vicuna-7B-v1.5. Among the three model-instruction tuning dataset combinations,
models based on UAL demonstrate clear overall advantages over the SFT
counterpart. Specifically for Llama-2-7B, model degradation due to SFT (w.r.t.
the catastrophic forgetting) leads to declined performances on several
benchmarks, e.g. LIMA-7B (Llama-2-7B, SFT) on ARC-c and Deita-7B (Llama-2-7B,
SFT) on HellaSwag. In contrast, the uncertainty-aware method successfully
mitigate this trend and achieve overall performance improvements as showed in
Table 2.

#### PPL Uncertainty.

In previous studies, researchers have attempted to model a model’s uncertainty
at the token level Zhu et al. (2023); Peng et al. (2023a); Zhong et al.
(2023b), with one significant approach being the use of Perplexity (PPL) that
calculated from log probability Zhou et al. (2020). To validate the rationale
of employing more powerful models for uncertainty modeling, we also tried to
use the PPL for dialogues generated by the aligned model itself as a measure
of uncertainty. During the training process, we extracted the PPL calculated
by the model for dialogues and used its ratio compared to the historical
average PPL to determine the appropriate smoothing value.

Figure 3:  Training dynamics of Llama-2 and Mistral based models using SFT and
UAL. The performance is evaluated on four datasets of OpenLLM Leaderboard,
i.e., MMLU, HellaSwag, TruthfulQA and ARC-c. The UAL approach employing
uncertainty estimation from GPT-4 and PPL shows consistently better and more
robust performance, compared with its vanilla SFT counterpart.

Modeling uncertainty with PPL can also aid in model alignment. Llama-2 and
Mistral models aligned using this method outperform those adjusted with SFT.
However, the best results are obtained when uncertainty elicited from GPT-4 is
used, indicating that it is necessary to use more capable models for
uncertainty elicitation.

Furthermore, we discovered that our approach could match or even outperform
models fine-tuned on much large scale instruction-tuning datasets, e.g.,
LIMA-7B (UAL-SFT) is able to achieve performance on par with Vicuna-7B-v1.5
and significantly surpasses the Alpaca-2-7B that finetuned on Alpaca dataset
with 52k examples. Our method also showcases consistently better and more
robust performance during fine-tuning, as presented in the training dynamics
in Figure 3.

###  3.3 Enhancement in the Mixed Entropy Scenario

Model | MetaMath / Gsm8k | Average
---|---|---
Base Model | 34.17 / 24.63 | 29.40
High Entropy (SFT) | 40.67 / 35.86 | 38.27
Low Entropy (SFT) | 45.63 / 47.38 | 46.51
Mixed (SFT) | 42.33 / 43.06 | 42.70
Mixed (Ours) | 44.66 / 44.35 |  44.51 (+1.81)
AlpacaEval
---
-
68.81
37.05
55.08
65.70 (+10.62)

Table 3: The zero-shot evaluation results (/wo Best-of-n strategy) on math
problem solving and general dialogue (i.e., AlpacaEval) scenarios. All the
models are instruction-tuned from Mistral-7B with an identical data scale. The
performance of Mistral-based model tuned in the mixed scenario with UAL is
improved in tasks of different levels of entropy.

In order to gain a deeper understanding of the uncertainty-aware learning’s
capability to enhance model performance in both low-entropy and high-entropy
scenarios, the High Entropy and Low Entropy datasets are employed to fine-tune
the Mistral-7B model separately via SFT, thus obtaining models specialized for
High Entropy and Low Entropy scenarios mentioned in 2.2. It is noteworthy that
this extreme data settings are used to contrast the superiority of our method,
particularly considering that the Low Entropy dataset comprised solely of
mathematics-related dialogue data, which is rarely used for training models
intended for general conversational purposes.

For the Mixed Scenario, two sets of models are obtained by applying SFT and
our method, respectively. Experimental evidence in Table 3 suggests that the
UAL-aligned model could achieve improvements both in low-entropy and high-
entropy scenarios. Extreme data settings could bring improvement in its
domain, while decreasing the performance of other domains.

Figure 4:  Example responses of UAL and SFT Mistral-7B models.

Figure 5:  Visualizations of features from (LEFT) Llama-2-7B, (Middle)
LIMA-7B-SFT (Llama-2) and (right) LIMA-7B-UAL (Llama-2) for a pair of tokens
(i.e., #I, #To). The features are extracted from the penultimate layer Müller
et al. (2019) of the Llama-2 model and then mapped into 2d feature space with
the PCA technique.  Figure 6: GPT-4-turbo evaluation results on model response
quality. Top: SFT (/w Label Smoothing, $\alpha=0.1$) and common SFT aligned
Llama-2-7B response quality comparison. Middle: UAL and SFT aligned Mistral-7B
response quality comparison. Bottom: UAL and SFT aligned Llama-2-7B response
quality comparison.

###  3.4 Ablation Study

LS | Uncertainty | Silhouette | OpenLLM Benchmarks (AVG)
---|---|---|---
✘ | ✘ | 0.406 | 64.89
✔ | ✘ | 0.427 | 64.25
✔ | ✔ | 0.446 | 69.16

Table 4: Ablation Study. LS indicates label smoothing technique. Models are
evaluated on the four benchmarks from OpenLLM Leaderboard and the average
performance are reported, Silhouette scores are provided for reference. Label
smoothing alone does not lead to better performance, while UAL yields better
feature convergence and higher performance.

UAL is, in essence, an alignment approach employing label smoothing that takes
uncertainty into account. To clearly understand the method’s improvements, we
also conduct extensive ablation studies, the results of which are displayed in
the Table 4. The findings indicate that while label smoothing alone can
achieve a higher Silhouette Coefficient compared to SFT (as shown in the first
row), it does not inherently enhance the model’s overall performance across
various popular benchmarks. Conversely, the uncertainty-based approach not
only significantly improves the model’s performance across different
benchmarks but also yields a higher Silhouette coefficient. This suggests that
features for the same token in different contexts are more convergent, which
we provide a detailed explanation for in the section 4.

###  3.5 Case Study of Model Quality

The superiority of our models can be seen in the examples in Figure 4. Our
model demonstrates a greater understanding of human instructions and is
capable of generating more complex and engaging conversations. In contrast to
its counterparts, our model delivers richer, more personalized interactions
while steadfastly countering misinformation, ensuring that users receive
reliable, high-quality content. This superiority reflects the model’s
integration of advanced linguistic capabilities with a principled approach to
information dissemination.

###  3.6 GPT-4 Evaluation of Response Quality

In order to arrive at more generalized conclusions, we conduct a larger-scale
evaluation of response quality on the LIMA dataset’s test set, which comprises
300 instructions, encompassing a variety of challenging questions across
different scenarios. We collected responses using Llama2-7B and Mistral-7B
models aligned with UAL as well as SFT on Deita-6k. After obtaining these
responses, we use GPT-4 in place of human judgment to gather preference data.
Although some studies suggest that LLMs-as-judge may exhibit a certain degree
of bias Zhao et al. (2023); Sottana et al. (2023); Xu et al. (2024), strong
proprietary LLMs, e.g., GPT-4, are capable of making preference determinations
that are highly consistent with those of human annotators Dubois et al.
(2023), thus proving to be well-suited for the task of evaluating model
response quality.

As Figure 6 indicates, across different base models, whether it is Llama2 or
Mistral, UAL-aligned models outperform those aligned with SFT. Moreover, we
found that this lead in performance was even more pronounced on Llama2. In
addition, we compare the response quality of models utilizing uncertainty-
aware learning with those using label smoothing under equivalent $\alpha$
conditions, i.e. $\alpha=0.1$, finding that the latter did not perform as well
as UAL or even SFT.

##  4 One Alternative View of Uncertainty-Aware Learning

Feature clustering has been an interesting view in machine learning and deep
learning Huang et al. (2022); Müller et al. (2019). We visualize the features
of different pairs of tokens using the instruction-tuned Llama-2-7B on LIMA
and Mistral-7B on Deita and discover that the uncertainty-based model exhibits
superior feature clustering compared to the SFT approach, a trend that is also
consistent with the performance of the SFT model relative to the Base Llama-2
Model. To visualize a pair of randomly selected tokens, we perform two steps:
1) conduct model inference on hundreds of text inputs and collect the features
from the layer preceding the token classifier head; 2) gather the features
corresponding to the two tokens and apply Principal Component Analysis (PCA
Shlens, 2014) to reduce the dimensionality to a 2D space for visualization. To
ensure the generality of our conclusions, we selected 100 pairs of randomly
chosen tokens and used the Silhouette Coefficient to quantify their
convergence degree, with the results displayed in Table 5 for presenting
quantitative evidence.

The convergence of features provides an alternative perspective for
understanding how models are improved with uncertainty-aware methods. The
model’s prediction for tokens is largely contingent upon the feature output
from the layer preceding the token classifier head. The texts from which we
collected features originate from the test set of the LIMA Zhou et al. (2023)
dataset, which encompasses a wide range of different scenarios. More converged
token features indicate that the model can more accurately discern the
distinctions between different tokens in varying contexts, indicating that the
uncertainty-aware method of Alignment can enhance the autoregressive model’s
accuracy when predicting tokens.

Model | Base Model | SFT | UAL
---|---|---|---
LIMA-7B (Llama-2) | 0.354 | 0.406 | 0.446
Deita-7B (Mistral) | 0.053 | 0.061 | 0.084

Table 5: Average Silhouette Coefficient scores for features on one hundred
randomly selected pairs of tokens. Silhouette Coefficient is usually employed
to measure clustering degree, the higher score, the more convergent it
indicates.

##  5 Related Work

#### LLM Alignment.

The LLM research community has discovered that a relatively small, diverse,
and high-quality dataset is sufficient for language model alignment Zhou et
al. (2023); Liu et al. (2023). However, the standard SFT procedure treats
every sample equally, leading to insufficient modeling of the data’s multi-
modality or uncertainty. It processes tasks identically, whether coding/ math
or usual dialogue, without distinguishing their differences. RLHF has proposed
distinguishing between reward modeling and supervised fine-tuning Li et al.
(2023); Zheng et al. (2023), but this process requires additional data
collection, which is complex to conduct and introduces additional human bias.
Some research has suggested dividing SFT into different stages based on
principles of continuous learning Wang et al. (2023a), but this approach
deteriorates the forgetting phenomenon in large language models.

Additionally, due to the catastrophic forgetting phenomenon Luo et al. (2023);
Ramasesh et al. (2022); Aleixo et al. (2023); Wang et al. (2023b), some
studies have mentioned that when aligning large language models, the DeepSeek
technical report reveals that the SFT model often underperforms compared to
its base model on several benchmarks Team (2024); Decreased performance is
observed on general tasks when forging the agent capability of LLMs Zeng et
al. (2023). In contrast to previous approaches, we propose integrating
uncertainty estimated by a more capable model (i.e., GPT-4 OpenAI (2023)) into
alignment to achieve considerably better performance.

#### Uncertainty-Aware Learning.

Estimating uncertainty to establish prediction confidence is vital for deep
learning, and it has been studied thoroughly Lakshminarayanan et al. (2017);
Maddox et al. (2019); Malinin and Gales (2018). Uncertainty has played an
important role in Bayesian Neural Networks Gal and Ghahramani (2015), semi-
supervised learning, and self-learning Xu et al. (2022); Rizve et al. (2021).
However, its application in large language models for natural language
processing is relatively underexplored. Previous work has focused on improving
neural machine translation Zhou et al. (2020) and code generation Zhu et al.
(2023), but these efforts have been limited to token-level uncertainty or
perplexity (PPL). With the emergence of large-scale language models, some
researchers have begun to elicit uncertainty from these models and have found
that they can express their uncertainty well Xiong et al. (2023). This insight
provides inspiration for subsequent research.

##  6 Conclusion

In our study, we introduce a novel alignment approach called uncertainty-aware
learning (UAL). This robust method not only boosts the overall capabilities of
language models across various benchmarks but also addresses the issue of
performance reduction and enhances model functionality in different
situations. Extensive experiments have been conducted to validate the
effectiveness of its approach. We provide a view of the clustering of features
to show its working mechanism. Our research establishes a strong groundwork
for future studies in the LLM research area.

## Limitations

Our approach centers on enhancing the alignment procedure of LLMs through the
introduction of a novel uncertainty-aware method. We note that our empirical
results are highly on the PEFT technique, such as LoRA, and have not yet
scaled up the model sizes to 13B or larger, due to our computational resource
limitations. Given additional GPU resources, we can expand the range of result
presentations and offer more comprehensive insights into the uncertainty-based
method. Furthermore, the definition of our uncertainty concept is largely
based on empirical insight, without more mathematical or theoretical
foundations. Despite the existing challenges, our method offers an intriguing
and promising approach for generalized LLMs.

## Ethics and Reproducibility Statements

#### Ethics.

We take ethical considerations very seriously and strictly adhere to the ACL
Ethics Policy. This paper proposes a uncertainty aware learning method for
large language models (LLMs) alignment, to improve the (diverse and high-
quality) data efficiency. All employed models and datasets in this paper are
publicly available and have been widely adopted by researchers. All
experimental results upon these open models and datasets are reported
accurately and objectively. Thus, we believe that this research will not pose
any ethical issues.

#### Reproducibility.

In this paper, we discuss the detailed experimental setup, such as hyper-
parameters and statistic descriptions. More importantly, we release our code
to better help our readers to reproduce the experimental results:
github.com/ekonwang/UAL4Alignment.

## Acknowledgments

We are grateful to the anonymous reviewers and the area chair for their
insightful comments and suggestions.

## References

  * Aleixo et al. (2023) Everton Lima Aleixo, Juan Gabriel Colonna, Marco Cristo, and Everlandio Fernandes. 2023.  Catastrophic forgetting in deep learning: A comprehensive taxonomy.  _arXiv preprint_.
  * Dubois et al. (2023) Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Alpacafarm: A simulation framework for methods that learn from human feedback.  _arXiv preprint_.
  * Gal and Ghahramani (2015) Yarin Gal and Zoubin Ghahramani. 2015.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  _ICML_.
  * Huang et al. (2022) Dong Huang, Chang-Dong Wang, and Jian-Huang Lai. 2022.  Fast multi-view clustering via ensembles: Towards scalability, superiority, and simplicity.  _arXiv preprint_.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7b.  _arXiv preprint_.
  * Lakshminarayanan et al. (2017) Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017.  Simple and scalable predictive uncertainty estimation using deep ensembles.  In _NeurIPS_.
  * Li et al. (2023) Zihao Li, Zhuoran Yang, and Mengdi Wang. 2023.  Reinforcement learning with human feedback: Learning dynamic choices via pessimism.  _arXiv preprint_.
  * Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022.  Truthfulqa: Measuring how models mimic human falsehoods.  In _ACL_.
  * Liu et al. (2023) Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. 2023.  What makes good data for alignment? A comprehensive study of automatic data selection in instruction tuning.  _arXiv preprint_.
  * Lu et al. (2023) Qingyu Lu, Baopu Qiu, Liang Ding, Kanjian Zhang, Tom Kocmi, and Dacheng Tao. 2023.  Error analysis prompting enables human-like translation evaluation in large language models: A case study on chatgpt.  _arXiv preprint_.
  * Luo et al. (2023) Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023.  An empirical study of catastrophic forgetting in large language models during continual fine-tuning.  _arXiv preprint_.
  * Maddox et al. (2019) Wesley J. Maddox, Pavel Izmailov, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. 2019.  A simple baseline for bayesian uncertainty in deep learning.  In _NeurIPS 2019_.
  * Malinin and Gales (2018) Andrey Malinin and MarkJ.F. Gales. 2018.  Predictive uncertainty estimation via prior networks.  _NeurIPS_.
  * Müller et al. (2019) Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton. 2019.  When does label smoothing help? In _NeurIPS 2019_.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 technical report.  _arXiv preprint_.
  * Peng et al. (2023a) Keqin Peng, Liang Ding, Qihuang Zhong, Yuanxin Ouyang, Wenge Rong, Zhang Xiong, and Dacheng Tao. 2023a.  Token-level self-evolution training for sequence-to-sequence learning.  In _ACL_.
  * Peng et al. (2023b) Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023b.  Towards making the most of chatgpt for machine translation.  In _Findings of EMNLP_.
  * Qin et al. (2023) Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023.  Is chatgpt a general-purpose natural language processing task solver? _arXiv preprint_.
  * Ramasesh et al. (2022) Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. 2022.  Effect of scale on catastrophic forgetting in neural networks.  In _ICLR 2022_.
  * Rizve et al. (2021) Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S. Rawat, and Mubarak Shah. 2021.  In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning.  In _ICLR 2021,_.
  * Shlens (2014) Jonathon Shlens. 2014.  A tutorial on principal component analysis.  _arXiv preprint_.
  * Sottana et al. (2023) Andrea Sottana, Bin Liang, Kai Zou, and Zheng Yuan. 2023.  Evaluation metrics in the era of GPT-4: reliably evaluating large language models on sequence to sequence tasks.  _arXiv preprint_.
  * Team (2024) Deepseek Team. 2024.  Deepseek LLM: scaling open-source language models with longtermism.  _arXiv preprint_.
  * Team (2023) Llama Team. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint_.
  * Wang et al. (2023a) Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. 2023a.  A comprehensive survey of continual learning: Theory, method and application.  _arXiv preprint_.
  * Wang et al. (2023b) Yikun Wang, Rui Zheng, Haoming Li, Qi Zhang, Tao Gui, and Fei Liu. 2023b.  Rrescue: Ranking LLM responses to enhance reasoning over context.  _arXiv preprint_.
  * Xiong et al. (2023) Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2023.  Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms.  _arXiv preprint_.
  * Xu et al. (2022) Yi Xu, Jie Hu, Zhiqiao Gao, and Jinpeng Chen. 2022.  UCL-AST: active self-training with uncertainty-aware clouded logits for few-shot text classification.  In _ICTAI 2022_.
  * Xu et al. (2024) Ziyang Xu, Keqin Peng, Liang Ding, Dacheng Tao, and Xiliang Lu. 2024.  Take care of your prompt bias! investigating and mitigating prompt bias in factual knowledge extraction.  In _LREC-COLING_.
  * Yu et al. (2023) Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023.  Metamath: Bootstrap your own mathematical questions for large language models.  _arXiv preprint_.
  * Zeng et al. (2023) Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. 2023.  Agenttuning: Enabling generalized agent abilities for llms.  _arXiv preprint_.
  * Zhao et al. (2023) Jiaxu Zhao, Meng Fang, Shirui Pan, Wenpeng Yin, and Mykola Pechenizkiy. 2023.  GPTBIAS: A comprehensive framework for evaluating bias in large language models.  _arXiv preprint_.
  * Zheng et al. (2023) Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi, Nuo Xu, Wenbin Lai, Minghao Zhu, Cheng Chang, Zhangyue Yin, Rongxiang Weng, Wensen Cheng, Haoran Huang, Tianxiang Sun, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, and Xuanjing Huang. 2023.  Secrets of RLHF in large language models part I: PPO.  _arXiv preprint_.
  * Zhong et al. (2023a) Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023a.  Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert.  _arXiv preprint_.
  * Zhong et al. (2023b) Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023b.  Self-evolution learning for discriminative language model pretraining.  In _Findings of ACL_.
  * Zhou et al. (2023) Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023.  LIMA: less is more for alignment.  _arXiv preprint_.
  * Zhou et al. (2020) Yikai Zhou, Baosong Yang, Derek F. Wong, Yu Wan, and Lidia S. Chao. 2020.  Uncertainty-aware curriculum learning for neural machine translation.  In _ACL_.
  * Zhu et al. (2023) Yuqi Zhu, Jia Allen Li, Ge Li, Yunfei Zhao, Jia Li, Zhi Jin, and Hong Mei. 2023.  Improving code generation by dynamic temperature sampling.  _arXiv preprint_.

##  Appendix A Low-Entropy and High-Entropy Examples

Figure 7: Examples of low-entropy and high-entropy example.

##  Appendix B Uncertainty Statistics

Figure 8: Uncertainty estimation by GPT-4. Left: Uncertainty scores of
Deita-6k alignment dataset. Right: Uncertainty scores of LIMA dataset.

##  Appendix C Prompt Examples

Figure 9: Examples of low-entropy and high-entropy example.

##  Appendix D Hyperparameters setting

The foundation models employed are Llama2-7B and Mistral-7B, which are aligned
using the LoRA technique if without any other specifications. We set the bias
to none, with LoRA parameters $r=8$ and $\alpha=16$. Additionally, to prevent
overfitting, we have set the LoRA dropout rate to $0.1$.

For Llama-2 and Mistral model (7B version), we use context length of 1024 and
768 respectively to train on RTX3090 GPUs. For Deita-6k dataset we apply
$\alpha=0.1$ for UAL-aligned models and larger $\alpha$ (i.e. $0.3$) for LIMA
dataset. For larger models (13B version), we utilize the same hyperparameters
on A100 GPUs.