crafted prompts, hierarchical classification strategies, and adjusted model
temperatures. Meanwhile, Pratt et al. [129] also demonstrate that using LLM’s
knowledge can immediately improve zero-shot accuracy on a variety of image
classification tasks, saving considerable manual effort. In addition, LLMs can
also describe and summarize the image content for further classification,
documentation, and processing, and an example is given in [17] by generating
the accident report of a car crash.

Finally, Fig.9 presents an example of using LLM models for image
classification and object detection in radio access networks. In particular,
the cameras attached to the BS can capture environmental images, and then the
image data will be sent to the LLM at the network edge by wired backhaul. The
LLM can use computational resources at the edge cloud for image processing,
classification, and detecting object locations such as vehicles, users, and
blockage buildings. After that, the edge cloud can send back the
classification and detection results to BSs, and then the BS can adjust the
beamforming and hand-off decisions accordingly.

Figure 9: Framework of LLM-aided computer vision in wireless networks.

###  V-E Encrypted Traffic Classification

Network traffic classification is an essential technique in telecom network
management, which aims at identifying the category of traffic from various
applications [150]. Specifically, the widespread utilization of traffic
encryption plays a significant role in protecting data and user privacy.
However, it also presents challenges in capturing implicit and robust patterns
within encrypted traffic, which is essential for network management. To tackle
these challenges, conventional methods [132] usually extract features within
encrypted traffic such as certificates to create fingerprints for
classification through fingerprint matching, while these methods fall short
with the advent of advanced encryption techniques. Additionally, existing ML-
based studies [151] can automatically extract complex and abstract features to
analyze encrypted traffic, resulting in notable performance improvement.
However, these methods are heavily dependent on the amount and distribution of
labelled training data, leading to limited generalization ability due to model
bias.

Recently, pre-training-based methods have achieved great breakthroughs across
a wide range of application fields. In particular, pre-trained models are
designed to learn data representations from unlabelled data, allowing these
representations to be effectively applied to downstream tasks through fine-
tuning models on labelled data. In the context of encrypted traffic
classification, Ma et al. [130] capture long-distance contextual relations
within traffic sequence through BERT, and then integrate packet-level token
semantic features at the forward and backward positions of BiLSTM, enhancing
the BiLSTM attention to packet-level features. BERT-BiLSTM is evaluated to
identify the types of network communication application activities using the
ISCX VPN dataset [152], which includes various pcap files corresponding to
different application activities. The dataset is comprised of 17 label
categories, with each label representing a distinct type of application
activity, including Email, Facebook, Gmail, Netflix, SCP, Skype, Youtube, and
Spotify, among others. BERT-BiLSTM effectively distinguishes each application,
achieving an overall accuracy of 99.70%, precision of 99.34%, recall of
99.51%, and F1-score of 99.43%, thereby surpassing the performance of
traditional ML methods. The performance enhancement further indicates the
advantages of BERT-BiLSTM in encrypted traffic classification: (1) BiLSTM
captures the relevant feature of front and rear token sequences after BERT
extracts general features of encrypted traffic, learning the long-distance
relations within token sequences. (2) BiLSTM captures packet-level features
and contextual relations by simultaneously integrating packet-level token
semantic features at both forward and backward starting positions of BiLSTM.

Figure 10: Framework of LLM-based encrypted traffic classification[131].
”BURST” refers to a set of time-adjacent network packets originating from the
request or the response in a single session flow, and therefore a group of
BURSTs can characterize the network flow transmission patterns.

Moreover, Lin et al. [131] introduce Encrypted Traffic BERT (ET-BERT), as
shown in Fig. 10, which aims to learn generic traffic representations from
large-scale unlabelled encrypted traffic. Concretely, Datagram2Token is first
utilized to convert traffic flow into word-like tokens, through three steps:
(1) BURST Generator extracts BURST time-adjacent network packets representing
the session information. (2) BURST2Token applies a bi-gram model to convert
the datagram of each BURST into token embeddings and divides a BURST into two
segments for subsequent pre-training tasks. (3) Token2Embedding merges the
token embeddings, position embeddings, and segmentation embeddings of each
token as the input representations for pre-training. To demonstrate the
effectiveness and generalization of ET-BERT, the authors conduct experiments
across several encrypted traffic classification tasks, e.g., general encrypted
application classification [132], encrypted malware classification, encrypted
traffic classification on VPN [133], with the remarkable improvements over
existing state-of-the-art methods by 5.4%, 0.2%, and 5.2%.

Although ET-BERT exhibits a strong generalization capability across various
tasks, the assumption of clean pre-training data presents challenges in secure
traffic classification. This vulnerability is exposed particularly when
attackers craft a poisoned model with backdoors by maliciously inserting low-
frequency words as toxic embeddings. Such manipulation allows attackers to
deceive the normally fine-tuned model during specific classification tasks.
Hence, how to construct toxic tokens within encrypted traffic can be
potentially investigated as a promising direction in the field of LLM-based
encrypted traffic classification.

TABLE VII: Summary of LLM-enabled classification for telecom.

LLM-based forecasting techniques |  Main features |  Prompt and fine-tuning requirements |  Advantages compared with conventional approaches |  Network classification application opportunities
---|---|---|---|---
Attack detection |  Attack detection is vital for maintaining the security and reliability of telecom networks. LLMs have showcased a strong ability to capture discriminative information within multi-modal and heterogeneous network data. |  Fine-tuning LLMs on security-specific language emerges as a promising approach for attack detection. This method allows fine-tuned LLM models to maintain their proficiency in processing general English vocabulary while excelling at achieving specific security objectives. |  LLM techniques have the strong advantages of processing both numerical traffic loads and descriptive security-related textual contents, e.g., ransomware and keylogger, achieving better performance than existing ML and DL algorithms. |  LLM models can be effectively employed for detecting cyber attacks [121, 124] and contributing to the mitigation and recovery strategies against such attacks [124].
Text classification |  Text classification and processing are very useful for the telecom industry. LLM models have shown great promise in understanding and processing text and languages. |  Fine-tuning LLM models on telecom language is required, e.g., network trouble report datasets and 3GPP technical specifications. There are no specific requirements for prompts. |  Automatic text classification and processing will greatly save human efforts on many document-related tasks, e.g., automated troubleshooting report generation and ranking. |  The telecom applications include user enquiries and intent classification and analyses[127], automated trouble report classification[144], standard specification classification[30].
Image classification |  Computer vision is a very useful technique for 6G networks, enabling 3D sensing for the environment. Some LLMs have shown impressive capabilities in image and vision-related tasks. |  The study in [26] shows that carefully crafted prompts are critical to improving the classification performance of LLM models, e.g., ”Fill in the blank, this is a picture of {…} ”. However, fine-tuning LLM models on telecom-image datasets can improve classification accuracy. |  LLM’s zero-shot learning capability can avoid the complexity of dedicated model training. For instance, [26] achieved a satisfactory performance by pure prompt engineering without any fine-tuning. |  LLM-aided image classification can be used for blockage prediction[146], proactive beamforming and hand-off[25], user localization [128], etc.
Traffic classification |  Network traffic classification is an essential technique in network management and security, which aims at identifying the category of traffic from various applications. LLM techniques have demonstrated remarkable performance in encrypted traffic classification. |  Fine-tuning LLM models on labeled network data is crucial for ensuring their adaptability across various traffic classification scenarios, such as single packet and single flow classification. |  LLMs are capable of learning generic traffic representations from extensive amounts of unlabeled, encrypted traffic without plaintext, resulting in extracting valuable insights from encrypted traffic for downstream traffic classification. |  Traffic analyses and classification are very common tasks in telecom networks. LLMs can be effectively applied for encrypted traffic classification [131, 130].

###  V-F Discussions and Analyses

Table VII summarized LLM-enabled classification techniques in terms of the
main features, prompt and fine-tuning requirements, advantages, and network
classification application opportunities. It shows LLM’s versatile
capabilities on different classification tasks, ranging from textual security
logs and customer comments to images and network traffic files.

In particular, Section V-B demonstrates that LLM techniques have great
potential for telecom network security. Security is an important topic for
telecom operations, and LLM models can contribute through their classification
and detection capabilities. In particular, the LLM can handle multi-modal and
heterogeneous network data, e.g., CSI, traffic load level, network device logs
and network configurations, and then extract useful network security
information from these correlated inputs. Additionally, some LLMs can also
recommend response and recovery strategies for network incidents[124]. This
indicates the potential of building an end-to-end telecom security system,
from status monitoring and attack detection to incident response and recovery.

Meanwhile, LLM can serve as a zero-shot classifier. Telecom networks indicate
a complicated dynamic environment, leading to various tasks. Existing methods
are usually task-specific, with dedicated designs for each incoming request.
By contrast, some LLM models have shown zero-shot classification
capabilities[26]. For instance, they can be directly used to classify images
captured by the cameras on the BS, or analyze customer comments without prior
training. Such a feature can be very useful in handling diverse tasks in
complicated telecom systems such as object detection and user localization. In
addition, LLM models have outstanding capabilities in processing text-related
tasks, including both natural languages, such as customer comments and system
language like network log files. These textual tasks are usually performed
manually, but LLM models can easily handle different classification and
detection tasks with much less human intervention.

Finally, LLM models can contribute to vision-aided telecom. Sensing is
increasingly important for wireless networks, and computer vision is an
important approach to capturing wireless environment dynamics. With pre-
trained real-world knowledge, LLM models can be directly used for image and
vision-related tasks, such as image description, image-text transformation,
object detection, and image classification. In addition, LLM technologies also
have advantages over conventional algorithms in terms of generalization
capabilities. This means that LLMs can process various telecom tasks without
extra training, e.g., blockage detection and prediction by using BS
cameras[146], proactive beamforming and hand-off[25], and user localization
[128].

##  VI LLM-enabled Optimization Techniques for telecom

Optimization techniques are of paramount importance to telecom network
management, and this section presents LLM-enabled optimization techniques. It
first analyzes the motivations and optimization capabilities of LLMs, and then
introduces LLM-aided reinforcement learning, black-box optimizer, convex
optimization, and heuristic algorithms along with network optimization
applications. Finally, we analyze and summarize the key findings.

###  VI-A Motivations and Optimization Capabilities of LLM

Optimization problems have been widely investigated in the communication field
due to their critical importance. Existing optimization techniques can be
categorized into several approaches[2]: ML-based, convex optimization,
heuristic algorithms, and black-box optimization. For instance, reinforcement
learning is a widely considered ML algorithm to solve optimization
problems[4]. Meanwhile, fractional programming is a well-known convex
optimization technique in wireless networks, e.g., decoupling signal strength
with interference and noise to maximize the data rate[153]. Heuristic
algorithms are particularly useful for solving problems with integer control
variables[154], and black-box optimization is also a useful method to handle
problems with unknown objective function structure [155].

However, applying these techniques to telecom is not straightforward. For
instance, the reward function is an important part of implementing
reinforcement learning, but the corresponding design can be difficult without
professional knowledge of telecom. Moreover, the reward function may be
related to multiple network metrics such as delay, throughput, and packet drop
rate, incorporating these metrics into the reward function usually follows a
time-consuming trial-and-error manner[156]. Similarly, although there have
been many commercial convex optimization solvers, e.g., CPLEX and LINDO[157],
it is worth noting that optimization problems have to be formulated in
standard form, i.e., relaxing specific constraints for convexity or
continuity, which are considered as obstacles for the application of convex
optimization. To this end, existing studies have shown that LLM may offer new
opportunities to overcome the theory-application gap between existing
optimization techniques and real-world telecom applications. There are
multiple advantages to exploiting LLM-enabled optimize techniques for telecom:

Firstly, LLMs demonstrate a strong ability to follow human instructions.
Specifically, the LLM agent has the potential to formulate problems, design
algorithms, select models, and finally optimize the system performance based
on human preferences and language instructions[158]. With LLM-enabled
intelligence, operators can easily manage the network operation using simple
natural language input, and then LLM can automatically select proper ML
algorithms to implement tasks with minimum human intervention.

TABLE VIII: Summary of LLM-aided Optimization Techniques Studies.

Refer- ences |  Proposed LLM-aided optimization techniques |  Key findings & Conclusion |  Telecom application opportunities
---|---|---|---
[34] |  A LLM framework with a self-refinement mechanism for automated reward function design, where LLM can formulate an initial reward function based on natural language inputs. |  LLM-designed reward functions can rival or even surpass manually designed reward functions in 9 robot control tasks. |  Reinforcement learning is very useful for network optimization, and automatic reward design /universal proxy reward function is an appealing approach to lower the difficulty of applying reinforcement leaning techniques to various network management scenarios.
[35] |  It considers a universal reward design by prompting the LLM as a proxy reward function, where the user provides a textual prompt with a few examples or a description of the desired behavior. |  The generated rewards are well-aligned with the user’s objectives and outperform supervised learning approaches. |
[36] |  An LLM-aided reward design system with zero-shot generation, code-writing, and in-context improvement capabilities. It performs evolutionary optimization over reward code. |  It outperforms human experts on 83% of the tasks, leading to an average normalized improvement of 52%. |
[159] |  It proposed a novel framework to reinforce language agents through linguistic feedback. The agent verbally reflects on task feedback signals, maintaining the reflective text in an episodic memory buffer to induce better decision-making. |  The proposed framework achieves a 91% accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80%. |  LLM models have self-improvement capability, which means they can work as an agent to receive network environment feedback and improve the policies based on textual input.
[21] |  LLM generates new solutions from the prompt that contains previously generated solutions with their values, then the new solutions are evaluated and added to the prompt for the next optimization step. |  The proposed prompt-design scheme outperforms human-designed prompts by up to 8% on GSM8K[160], and by up to 50% on Big-Bench Hard tasks[161]. |
[162] |  Evaluating the optimization capabilities of LLM models across diverse tasks and data sizes, including gradient descent, hill-climbing, grid-search, and black-box optimization. |  1) The LLM show strong optimization capabilities; 2) LLM models perform well in small-size samples; 3) They exhibit strong performance in gradient-descent; 4) LLMs are black-box optimizers. |  Black-box optimizer is a promising approach to estimating the unknown loss function, which is especially useful since telecom networks become more complicated.
[163] |  A natural language-based system that engages in interactive conversations about infeasible optimization models. It provides natural language descriptions of the optimization model itself, identifies potential sources of infeasibility, and offers suggestions to make the model feasible. |  The proposed system can assist both expert and non-expert users in improving their understanding of the optimization models, enabling them to quickly identify the sources of infeasibility. |  Convex optimization is a commonly used technique for network optimization, and integrating LLM with convex optimization can bring promising changes to network optimization.
[164] |  An LLM-aided system that can develop mathematical optimization models, write and debug solver code, develop tests, and check the validity of generated solutions. |  It achieves nearly 0.8 success rate in 41 linear programming and 11 mixed integer linear programming problems. |
[165] |  Using LLMs such as GPT-4 to generate novel hybrid swarm intelligence optimization algorithms. |  Generated a novel meta-heuristic algorithm with pseudo-code by using 5 existing algorithms. |  Heuristic algorithms are naturally compatible with LLM techniques, since many heuristic rules can be easily described by textual language. It offers new opportunities for selecting and design new heuristic network optimization methods.
[166] |  Using general LLM serves as a black-box search operator for decomposition-based multi-objective evolutionary optimization in a zero-shot manner. |  The LLM operator only learned from a few instances can have robust generalization performance on unseen problems with quite different patterns and settings. |

Secondly, LLMs can lower the training and fine-tuning difficulties of ML-based
network optimization. Algorithm training is considered one of the main
obstacles to realizing AI-enabled wireless networks, which is usually very
time-consuming. By contrast, LLM has shown impressive few-shot or even zero-
shot learning capabilities in many fields[167]. In particular, LLM models can
learn in context from few or zero network management and optimization examples
and then generalize to incoming new tasks. By providing a handful of examples,
the LLM agent can quickly learn the hidden patterns without any extra model
training and fine-tuning, saving considerable time and effort for algorithm
training in network management. In addition, such fast learning capability is
critical to make rapid responses to network dynamics. This means that network
optimization decisions can be efficiently adjusted based on traffic patterns,
user types, and operator demands.

Finally, the rich real-world knowledge of LLM will contribute to network
optimization algorithm modelling and design. LLM is equipped with rich
internalized knowledge about the world in the pre-training stage[53]. Such
diverse knowledge can contribute to comprehending user preferences, task
requirements, and even optimization algorithm modelling and design. For
instance, LLM can already understand the fundamental concepts of reinforcement
learning and linear programming without any extra training, and both
techniques are very useful in optimizing telecom networks. This real-world
knowledge eliminates the gap between real-world network optimization demands
and problem modelling and design. Table VIII summarizes existing studies on
LLM-aided optimization techniques, including proposed methods, key findings,
and telecom application opportunities. Given these motivations and the
benefits of applying LLM to telecom optimization, we will introduce state-of-
the-art LLM-aided optimization techniques along with telecom network
optimization applications.

###  VI-B LLM-aided Reinforcement Learning for Network Optimization

Reinforcement learning is one of the most important techniques for network
optimization. It explores various sequential action combinations, e.g.,
network resource allocation strategies and signal transmission power level, to
maximize the long-term reward, such as higher data rate or lower transmission
delay[168]. Many network optimization problems can be transformed into a
unified MDP, and then using reinforcement learning to improve network metrics
dynamically. For instance, resource allocation is a very common problem in
many telecom scenarios, in which allocation decisions, desired network
performance metrics, and network dynamics are usually defined as actions,
rewards, and states, respectively[6]. However, it is worth noting that these
definitions are usually intuitive and require expert knowledge of
reinforcement learning techniques and telecom. Especially, most reward
functions are manually designed using trial-and-error approaches, and the
algorithm performance is affected by the hyperparameter selection, e.g.,
learning rate, batch size, and number of hidden layers. Fortunately, LLM
techniques provide new opportunities to overcome these bottlenecks. This
section will introduce two LLM-aided reinforcement learning techniques:
automatic reward function design and verbal reinforcement learning.

####  VI-B1 Using LLM for reward function design

A recent survey in [27] shows that 92% reinforcement learning researchers use
manual trial-and-error reward function design and 89% indicate that the
designed rewards lead to unintended behaviour during algorithm training[169].
Such issues become more difficult in complicated telecom scenarios since
various network elements are involved, e.g., users with diverse requirements,
limited available resources, and dynamic network environments. To this end,
LLM shows the capability of developing a universal approach for reward design,
which will significantly lower the difficulty of using reinforcement learning.
For instance, [34] proposed a self-refined LLM model for automated reward
function design in robotics, achieving a comparable performance as manually
designed functions. [35] applies LLM as a proxy reward function, where the
user provides a textual prompt with a few examples or a description of the
desired behaviour. In the following, we will introduce how the automatic
reward function is designed.

An MDP can be defined as a tuple $<S,A,R,T>$, where $S$ and $A$ are the set of
environment states $s\in S$ and actions $a\in A$, respectively. $T$ is the
transition probability with $T(s,s^{\prime})=Pr(s^{\prime}|s,a)$, indicating
the probability of taking action $a$ under state $s$ and reaching the next
state $s^{\prime}$. $R$ is the reward with $R=\mathcal{F}(s,a)$, where
$\mathcal{F}$ is the reward function that maps the states and action selection
to an immediate reward[170]. The reward feedback $R$ will further affect the
action selection policy $\pi$ with $a=\pi(s)$, which means the action
selection is under the current state $s$. However, the definition of such a
reward function $\mathcal{F}$ is not straightforward since mapping the state
$s$ and action $a$ to a specific value requires considerable experience and
trial-and-error tests. Therefore, the objective of reward design is to use LLM
as a proxy reward function or generate a reward function automatically[35,
34]. Given the above MDP fundamentals, extra prompt input is required as
textual input for LLM. Consider a set of prompt string $l\in\mathcal{L}$ and a
mapping function $\mathcal{M}$, we need to define:

  * •

Task description $l_{1}$: The string or environment code to describe the
target task[36];

  * •

Objective description $l_{2}$: The optimization objective or desired final
states of the task;

  * •

States and actions description $l_{3}$: It explains the definition of states
and actions in the target task;

  * •

Examples description $l_{4}$: It provides a trajectory or examples of the
episode. Note that a trajectory usually serves as a demo, but it is not
required in zero-shot learning.

  * •

A mapping function $\mathcal{M}$ that maps the textual output of LLM to a
binary value, e.g., ”good” or ”bad”. This binary value feedback indicates the
quality criteria of the generation, and then the LLM model easily understands
the overall feedback.

Given these definitions, the LLM-aided MDP definition becomes
$<S,A,R,T,\mathcal{L},\mathcal{M}>$, where $\mathcal{L}$ is a set of prompts
$l_{1}$ to $l_{4}$, and $\mathcal{M}$ is the mapping function. The reward
function $\mathcal{F}$ in $R=\mathcal{F}(s,a)$ is defined by
$\mathcal{F}:=\mathcal{G}(\mathcal{L},\mathcal{M})$, where $\mathcal{G}$ is
the inference of LLM models.
$\mathcal{F}:=\mathcal{G}(\mathcal{L},\mathcal{M})$ shows that the design of
the reward function $\mathcal{F}$ depends on prompt input $\mathcal{L}$ and
the mapping function $\mathcal{M}$. Based on the LLM-aided MDP framework,
using LLM for reward function design can be summarized as the following steps:

  * •

Step 1: Description input. Using language to describe the task, objective,
states, and actions. If necessary, providing possible trajectory examples to
LLM models. Here an alternative approach is to feed the environment code to
the LLM agent, and then using natural language to describe the task, which has
been used in [36].

  * •

Step 2: Initial reward function design, which will use Step 1 as input, and
produce initial reward function designs.

  * •

Step 3: Reward function implementation. Using the reward function produced in
Step 2 to train the reinforcement learning agent.

  * •

Step 4: Evaluation and feedback. Evaluating the reinforcement learning
training output and providing feedback to LLM models.

  * •

Step 5: Self-improvement. Sending the feedback and evaluation results to the
LLM agent, and then LLM will produce a new reward function design. Repeating
from Step 3 until the algorithm has the desired performance or reaches the
maximum iteration number.

Figure 11: LLM for reward design in network optimization.

To better explain how LLM-aided reward design can be used for network
optimization. Fig.11 shows the procedure of solving a simple resource slicing
problem[4]. In particular, it considers resource allocation as an example with
two types of users. Group 1 indicates URLLC users that desire lower latency
and higher reliability, and group 2 represents enhanced Mobile Broad Band
(eMBB) users with high throughput demands. As shown in Fig.11, the resource
allocation task is described by natural language as input for LLM, including
task description and user features, objectives, states and actions, and reward
design rules. Note that we use ”group 1” and ”group 2” instead of ”eMBB” and
”URLLC” to lower LLM understanding difficulty. In addition, the features of
the two groups have been clearly defined. Then, LLM will generate an initial
reward function design and send the initial design to the reinforcement
learning framework for evaluation. After that, we will receive and analyze the
evaluation results, e.g., convergence and system metrics. For instance, the
evaluation results in Fig.11 show that the 5% drop rate of group 1 users is
much higher than the predefined threshold 1%, and therefore the overall
evaluation for this design is ”bad”. It is worth noting that the final
evaluation of ”good” or ”bad” depends on the user’s predefined criteria, which
varies between different scenarios.

In Fig.11, if the evaluation result is ”bad”, then a detailed feedback summary
is provided with possible suggestions, e.g., the drop rate of group 1 users is
too high. Given this feedback, the LLM agent will redesign the reward function
and repeat the process from Step 3. On the other hand, if the evaluation
result is ”good”, the system will output the final reward function design. The
bottom module also shows an example that the reward function is improved by
iterations. e.g., the coefficient of $Drop\\_rate\\_group1$ is increased from
1 to 10, preventing dropping group 1 users. The coefficient of
$Throughput\\_avg\\_group2$ is also improved to balance the latency and
through metrics of two groups.

Reward function design is a prerequisite for applying reinforcement learning
to telecom, and LLM-aided automatic reward function design significantly
lowers the difficulty. However, it is worth noting that some reward functions
can be very complicated in the telecom field, which may include transformation
functions like $arctan$ or $sigmoid$ and diverse network metrics. These design
problems can be more complicated if multiple network elements are
simultaneously involved, such as vehicle networks and RISs [2]. The
simulations in [34, 35, 36] have demonstrated LLM’s capabilities in reward
design for robotics and logic games, but the application in the telecom field
is still an open issue.

Figure 12: LLM-aided verbal reinforcement learning for network optimization.

####  VI-B2  Verbal reinforcement learning via LLM

Section VI-B1 proves that LLM can use the feedback to improve previous
solutions. Given this self-improvement capability, a promising optimization
technique is to consider LLM as an agent, interacting with the environment to
explore optimal policy. Verbal reinforcement learning is proposed in [159],
and achieved satisfied performance across diverse tasks, including sequential
decision-making, coding, and language reasoning. Fig.12 shows an example of
using verbal reinforcement learning for radio access network optimization, and
the agent consists of the following modules:

  * •

Actor: The actor is built upon an LLM model, which is specifically prompted to
generate actions, e.g., network control and optimization strategies. Based on
short-term and long-term memories, the actor can apply various methods to
produce actions, such as CoT [33] and ReAct [171]. These advanced prompt
techniques can improve the actor’s capability of reasoning and planning, which
can better adapt to the complicated decision-making of network optimization
problems.

  * •

Evaluator: The evaluator is a critical module to assess the performance of the
actor. In particular, it takes the short-term trajectories as input and
produces a reward score that shows the action quality. The evaluator can be
defined in various approaches, e.g., a specified reward function or heuristic
criteria. For instance, in resource allocation problems, the evaluator can be
defined by a reward function with network metrics, or a heuristic like ”all
the users’ requirements have been fulfilled”. We still consider the radio
access network as an example. The evaluator’s internal feedback could be ”The
average latency of network edge users is too high, and 10% edge users’
communication demand is dropped. The overall performance of this trajectory is
bad.”

  * •

Self-reflection: The self-reflection module is the most important part of the
verbal reinforcement learning scheme, providing useful feedback instructions
to the actor. Specifically, with external feedback from the environment and
internal feedback from the evaluator, the self-reflection module can generate
more detailed feedback to the actor, which is far more informative than a pure
reward value in conventional reinforcement learning. A feedback example could
be ”Cell edge users should have more resources if available, and cell edge
means users that are far away from the BS than other users.”

  * •

Short-term and long-term memories: The memory mechanism consists of short-term
and long-term memories. The long-term memory indicates important lessons
learned from previous experience, while the short-term memory shows recent
decisions and performance. This is an intuitive approach that is similar to
the human brain with fine-grain recent details and important lessons from
long-term memory. With the self-reflection mechanism, the long-term memory
will automatically learn important rules, e.g., ”Cell edge users should have
more available resources; Type 1 users are delay sensitive, they should have
higher priority.”

Compared with conventional reinforcement learning, the LLM-aided verbal
learning technique has multiple advantages for telecom optimization: 1)
Lowering the difficulty of implementing network optimization. Verbal
reinforcement learning avoids the difficulty of tuning hyperparameters like
learning rate, batch size, and training frequency. This will significantly
lower the difficulty of applying artificial intelligence to network
optimization. 2) Allowing for language instructions to guide network
optimization policies. Specifically, the LLM-aided system allows for language
instructions to guide the agent exploration, which is much more efficient than
existing strategies such as $\epsilon$-greedy policy. Experienced network
operators can provide language instructions to the LLM models directly, and no
ML knowledge is required. 3) Reasoning and interpretable explanations for
algorithm performance. One crucial advantage of LLM-aided systems is that they
provide interpretable explanations of the algorithm and telecom system
performance, and these experiences can further help understand network
management policies.

Despite the advantages, LLM-aided reinforcement learning is still at a very
early stage, and there are very few studies that apply this technique to the
telecom field. In addition, specific telecom domain knowledge may be required
to let the LLM better understand user demand. Therefore, professional wireless
knowledge datasets such as TeleQnA in [110] may be required to fine-tune the
LLM model.

###  VI-C LLM as a Black-box Optimizer

Black-box optimizer is also an appealing approach for network optimization
problems. It refers to the task of optimizing an objective function
$f:X\rightarrow R$ without access to any other information about $f$, e.g.,
gradients or the Hessian[172]. Telecom networks will become more and more
complicated in the 6G era, and black optimization can avoid the complexity of
building dedicated optimization models. Existing studies have shown that LLM
has the black-box optimization capability to fit an unknown loss
function[162]. Fig.13 shows an example of using LLM in a black-box manner. It
starts by describing the optimization task, and then LLM will generate an
initial solution. The generated solution will be evaluated by the objective
function evaluator, e.g., average or sum data rate, average latency, etc. If
the evaluated score is satisfied or it is the maximum iteration number, then
the system will output the final solution. Otherwise, the current solution is
sent to a solution-score pairs pool, and a new prompt will be generated
accordingly for LLM as input. Here the solution-score pair pool includes past
experience and corresponding scores. By comparing the similarities of high-
score solutions, the LLM can generate better solutions iteratively with few-
shot learning capabilities. To better understand how LLM can be used as a
black optimizer for network optimization, we provide an example of BS power
control[173]:

  * •

Initial task description module:

BS power control task description “We have an interference control task
related to wireless network management. We need to control the power level of
two BSs to maximize the average data rate. We need you to provide the
transmission power of these two BSs, and adjust the power based on provided
feedback”.

  * •

Prompt inputs module for black-box optimization:

Prompt input for black-box optimization “Below are some previous power levels
and the corresponding data rate, which are arranged in descending order.
Input: $P\\_level\\_1$: 14 dBm, $P\\_level\\_2$: 17 dBm; Output $Avg\\_rate$:
1.1 Mbps; … … … … … … Input: $P\\_level\\_1$: 22 dBm, $P\\_level\\_2$: 15 dBm;
Output: Average data rate is 1.8 Mbps; Input: $P\\_level\\_1$: 25 dBm,
$P\\_level\\_2$: 22 dBm; Output: Average data rate is 2.5 Mbps. Give me a new
power level input that is different from all the traces above and has a higher
average data rate than any of the above”.

After the above prompts input, one can use the output to update the candidate
solutions and then repeat this process as shown in Fig.13 until obtaining a
satisfactory solution. The main advantage of black-box optimization is that it
avoids the complexity of defining dedicated optimization models, which have
been used to automatically construct the wireless network optimization model
in [174], and to optimize cellular network coverage and capacity in [175]. For
the LLM-aided black-box optimizer, the existing example quality may affect the
output results, and the algorithm performance cannot be guaranteed. However,
telecom management usually has stringent requirements on solution qualities to
guarantee the service level, which can be an obstacle to using LLM techniques.

Figure 13: LLM-as a black-box optimizer for telecom. Figure 14: LLM-aided
Convex Optimization Problems.

###  VI-D LLM-aided Convex Optimization for Telecom

Convex optimization is a crucial technique for telecom networks, and it is
commonly used in many scenarios[2]. For instance, fractional programming is
especially useful for wireless network optimization due to the fractional
terms in communication systems such as signal-to-interference-plus-noise ratio
(SINR) and energy efficiency, which is applied to wireless power control and
beamforming[153]. Convex optimization can provide stable and efficient
solutions, especially when closed-form solutions are achieved. However,
deploying convex optimization techniques usually requires dedicated problem
modelling, transformation, and relaxation since the original problems may be
non-convex. Therefore, the requirement for expert knowledge may prevent the
application of convex optimization techniques. To improve the accessibility of
convex optimization, the authors in [163] propose to use the LLM to diagnose
the infeasibility of optimization problems, aiming to relax or remove some
infeasible constraints, and LLM is used for convex optimization problem
modelling, code generation and solving in [164]. The experiments in [163] and
[164] have demonstrated that LLM has the potential to improve convex
optimization techniques.

Figure 15: LLM-aided meta-heuristic algorithm generation.

Fig.11 shows the key steps of using LLM to solve network convex optimization
problems with the following modules:

  * •

Problem modelling and description: Transforming the network optimization
problem into a standard form is the first step of automatic problem modelling.
Fig.11 presents some key elements of defining the problem, including problem
type, problem information, input and output format, objective and solvers.
Specifically, problem type specifies the type of this problem, e.g., linear
programming, mixed-integer linear programming, quadratic programming, etc.
Problem information includes the core description of the problem, which
defines the relationship between input and output variables. Then, input and
output variables show the expected input and output variables along with
definitions, i.e., network decision variables and output metrics. Objectives
and solvers give the optimization objective and applied solvers. Such a
standard form and description will lower the difficulty of LLM understanding.

  * •

Telecom knowledge and formulation templates: Telecom optimization requires
professional network knowledge. LLM has learned fundamental knowledge in the
pre-training period such as calculating information capacity using Shannon’s
formula. However, using state-of-the-art telecom knowledge and formulation
templates to fine-tune the LLM model can better improve the modelling
accuracy. For instance, a dataset named TeleQnA is defined in [110], and it
includes nearly 10000 communication field questions from both standards and
research articles.

  * •

LLM and Solvers: Existing studies have shown that LLM can use the advanced
features of existing solvers such as Gurobi and cvxpy to solve the
problems[176]. For instance, [164] observed that LLM can use the function
$gurobi.abs$ to model $\mathcal{L}$1-norm objective instead of adding
auxiliary constraints and variables. It demonstrates that LLM has the
potential to take advantage of existing solvers to address complicated
optimization problems. In addition, if the implementation fails, code-fix
templates can also be included to address the issues automatically and rerun
the test.

In summary, Fig.14 shows an example of solving network optimization problems
in an end-to-end manner. Given a proper problem description, the LLM-aided
system can automatically model the problem, generate code, and call the server
to solve and debug the problem. Such a scheme has been used in [164] to solve
41 linear programming and 11 mixed-integer linear programming problems and
achieved a nearly 0.8 success rate for small-scale problems using GPT-4. The
study in [164] also observed that the success rate could be further improved
by adding supervised tests and data augmentation.

Despite the great potential, it is worth noting that telecom networks have
become more and more complicated, and there are many complicated large-scale
and non-convex optimization tasks. For example, RIS-related optimization
problems usually include multiple control variables, e.g., RIS phase-shift
control and BS passive beamforming, which are usually optimized in an
alternating approach. It still requires dedicated human effort to transform
the problems into standard forms[99]. However, LLM-aided automatic convex
optimization is still a promising approach that will save human time and
effort on network optimization problem modelling and solving.

###  VI-E LLM-based Heuristic Algorithm Design

Heuristic algorithms are very useful techniques for network management and
optimization. Specifically, they apply diverse heuristic rules to select near-
optimal solutions with low design and computational complexity[177]. Heuristic
algorithms are particularly useful for solving optimization problems with
integer control variables, which are very frequently formulated in telecom.
For instance, the phase-shift optimization of RISs is considered as a very
difficult problem with integer control variables and large solution space, and
genetic algorithms and particle swarm optimization are used in [178] and [179]
to solve this problem. In addition, heuristic algorithms are intuitively
compatible with LLM models, since heuristic rules can be easily described by
natural language and instructions. For example, swarm-based methods are very
widely used heuristic algorithms, e.g., genetic algorithm, particle swarm
optimization, and grey wolf optimizer, providing near-optimal solutions by
iteratively searching for better solutions. However, the number of these
algorithms has grown significantly in the past decade, and selecting the
proper algorithm to solve specific network optimization problems has become
more difficult. Given the reasoning and understanding capabilities, LLM offers
promising changes for selecting and designing novel meta-heuristic algorithms.

Fig.15 presents an example of using LLM to design novel swarm-based meta-
heuristic algorithms for network optimization, which consists of 5 tasks. Such
a decomposition and CoT approach can considerably lower the prompt difficulty
and improve output performance[165]. The first step is to identify the key
requirements of optimization tasks. For example, RISs consist of hundreds of
small units, and each requires dedicated phase-shift control, leading to a
large solution space. Therefore, Prompt 1 in Fig.15 requires the candidate
algorithm to have “good exploration capabilities”, and the first instruction
is to “Please list 5 candidate algorithms”. Then, the next task is to identify
the key components of these algorithms. For instance, inertia weight and local
and global best mechanisms are two key components in particle swarm
optimization, and then LLM can better understand the functionality of each
unique heuristic rule. After that, Tasks 3 and 4 will generate the step-by-
step design and pseudo-code of a new swarm-based meta-heuristic optimization
algorithm. Most importantly, Task 5 will take full advantage of LLM’s
reasoning capability, and explain how this novel algorithm is designed with
step-by-step motivations.

In summary, Fig.15 presents an automatic approach for novel meta-heuristic
algorithm design, which can be very useful for telecom network control and
optimization. For instance, many network control scenarios require rapid
responses for environment dynamics such as traffic load level and user demand
changes, and LLM-aided systems in Fig.15 have the potential to generate novel
heuristic algorithms with fast convergence and low computational complexity.
Additionally, such a scheme can also be used to generate new heuristic network
protocols or management policies, significantly saving human efforts in terms
of creation and design[30].

TABLE IX: Summary of LLM-based Optimization Techniques for Telecom.

LLM-based optimization applications |  Main features |  Prompt/ Input requirements |  Advantages compared with existing approaches |  Potential issues |  Network optimization application opportunities
---|---|---|---|---|---
LLM-aided reward function design |  Reward function is a crucial part of reinforcement learning-enabled network optimization, and LLM provides automatic reward function design by using its self-improvement and understanding capabilities. |  Task/environment description; Objective description; States and actions; Examples or demos. A mapping function/criteria to evaluate the design to ”good”/”bad”. |  Automatic reward function design can significantly save human effort in applying reinforcement learning to network optimization tasks. Automatic reward function design has produced comparable performance as human manual design. |  1) Automatic reward design is still at a very early stage, and there are few applications that explore such a novel technique in the telecom field; 2) The prompt has to be carefully designed to describe the target task, which is known as prompt engineering. |  Reinforcement learning is a very useful technique for telecom network management, and automatic reward design is a promising technique to enable artificial general intelligence, which is particularly useful for small-scale optimization problems to save human effort.
Verbal reinforcement learning |  It considers LLM as an agent, exploring the environment and accumulating experiences. Using the self-improvement capability to improve previous solutions and obtain a higher reward. |  1) Self-evaluator will provide critical feedback to the actor for improved performance; 2) Short-term and long-term memories are crucial for the actor to distinguish between good and bad actions. |  1) Avoiding the difficulty of tuning hyperparameters like learning rate, batch size, and training frequency; 2) Allowing for language instructions to guide network optimization policies; 3) Providing Reasoning and interpretable explanations for algorithm performance. |  1) The evaluator and self-reflection modules have to be carefully designed to generate useful experience; 2) It may have exploration-exploitation difficulty, since the agent relies on previous experience to produce new solutions. |  Verbal reinforcement learning can be very useful for solving problems that have been well-defined with small action spaces and immediate rewards, which is very common in telecom networks, e.g., resource allocation and association.
LLM as a black-box optimizer |  Black-box optimization is a useful approach for network optimization, and LLM has been demonstrated to have the black-box optimization capability to fit an unknown loss function. |  1) Task description; 2) Previous input and output examples, and then asking for a better solution based on previous input and output. |  Black-box optimization avoids the complexity of building dedicated optimization models and transformations, which can be very time-consuming in complicated telecom network environments. |  The performance of using an LLM black-box optimizer cannot be guaranteed, which relies on the quality of the provided input and output examples. |  Black-box optimization is a promising technique for telecom network, but it may have difficulty providing stable and reliable results. The reasoning capability of LLM may shed light on solving this problem.
LLM-enabled convex optimization |  LLM provides end-to-end automatic solutions for convex optimization techniques, including problem modelling, code generation, and solver implementation. |  1) The problem has to be defined in standard form, so then the LLM can understand and model it; 2) Telecom knowledge and formulation template are required; 3) Existing solvers have to be specified for the LLM to solve the problem. |  1) Automatic problem modelling is an especially promising technique, significantly saving human effort; 2) It enables automatic problem-solving in an end-to-end manner, requiring minimum human intervention. |  Some convex optimization problems in the telecom field are extremely complicated with coupled control variables and highly non-convex objectives and constraints. These problems can be very difficult to solve automatically. |  Many network control problems can be formulated as convex optimization problems, and LLM-aided convex optimization has great potential to solve these problems efficiently with much less human effort.
LLM for heuristic algorithms |  Heuristic algorithms are inherently compatible with LLM, since many heuristic rules can be easily described by natural language. LLM offers opportunities for heuristic algorithm selection and design for specific network optimization tasks. |  It may require a series of prompts in a CoT manner, including candidate algorithm selection, analyses, new algorithm and pseudo-code code generation, and reasoning. |  Automatic heuristic algorithm selection and design will considerably save human time on algorithm analyses and design. It can also provide reasoning and analyses of the generated results. |  The generated heuristic algorithms still need to be tested and verified. Such an automatic design cannot guarantee the performance of the algorithm that was produced. |  Heuristic algorithms are widely used for telecom network optimization and management, and LLM has promising potential for heuristic algorithm selection and design, producing novel algorithms that can better serve telecom networks.

###  VI-F Discussions and Analyses

Subsections VI-B to VI-E have introduced various LLM-aided optimization
techniques along with telecom applications. Table.IX summarizes various LLM-
aided optimization techniques, including main features, prompt and fine-tuning
requirements, advantages compared with existing approaches, and network
optimization application opportunities. In the following, we summarize our key
findings and analyses.

Firstly, task description is crucial for network optimization. Task
description is the first step of using LLMs, which requires accurate and
standard input, e.g., input and desired output format, objective and specific
rules. In addition, these tasks are usually closely related to telecom domain
knowledge, and LLM may have difficulty understanding some professional
concepts. For example, in Section VI-E, the LLM may already have some general
knowledge of RIS technology, but they are unable to directly understand the
difficulty of RIS phase-shift control, which is a very professional domain-
specific knowledge. Therefore, the task description has to be carefully
designed, which will directly affect the LLM output. Meanwhile, prompt design
is the key to network optimization problems. Previous sections have
demonstrated that prompt is one of the most important approaches to take
advantage of LLM’s capabilities, and there have been various prompt
engineering techniques, e.g., CoT[33], ReAct [171], zero-shot instruction
[180], etc. Therefore, understanding the function of prompt engineering is
crucial for applying LLM to solve optimization problems. For instance, in
reward design problems, the feedback prompt is critical to improve the reward
design step by step. In the heuristic algorithm design problem in Section
VI-E, the output completely depends on the user prompt input to the LLM agent.

In addition, balancing LLM’s self-improvement and exploration-exploitation
efficiency. It is straightforward that most previous techniques rely on the
self-improvement capability of LLM models. For example, in verbal
reinforcement learning, based on environmental feedback, the LLM agent can
adjust the action selections to obtain a higher reward. On the other hand,
exploration-exploitation difficulty is a common obstacle for many optimization
techniques. This problem becomes severe when the action space becomes larger,
which is very common in telecom networks. [36] proposed a solution by
generating multiple solutions in each iteration and then applying evolutionary
strategies to select better strategies. [159] showed a similar idea by using
previous good trajectories as parents to generate new solutions. Therefore,
how to use the LLM’s self-improvement capability and meanwhile balance the
exploration-exploitation is very important.

Finally, although LLM already has fundamental knowledge of wireless networks
in the pre-training period, fine-tuning LLM models with specific domain
knowledge can still benefit telecom applications. In particular, a telecom-
specific LLM model can better handle the domain-specific description, prompt,
tasks, and solutions. For example, in reward function design and convex
optimization problems in Sections VI-B1 and VI-D, we can directly use network
terminologies to define the optimization problem, and the LLM can take full
advantage of the domain knowledge to understand the questions.

##  VII Time Series LLM for Prediction Problems

Prediction tasks are crucial in telecom networks that involve predicting
future trends, demands, and behaviours based on historical data, e.g.,
predicting network traffic, customer demand, equipment failures, and service
usage. This section will introduce time series models for prediction problems
in wireless networks, including pre-training foundation models, frozen pre-
trained, fine-tuning, and multi-modality LLMs.

###  VII-A Motivations

Conventional prediction algorithms in the telecom domain rely on statistical
and time-series analysis to estimate the output. However, telecom data is
usually non-linear, non-stationary, and influenced by various external
factors, leading to challenges in capturing complex patterns and
relationships. While these traditional methods have been effective to some
extent, they may struggle with the complexity and dynamic nature of telecom
data. Recently, LLM technologies have shown promise in addressing the
challenges of time-series prediction due to their ability to handle complex
data structures and adapt to changing patterns.

Firstly, LLM models provide a universal and generalizable model for telecom
network prediction. Given historical data, conventional prediction approaches
must train a new model to adapt to incoming target tasks. These methods
usually require extensive feature engineering and manual tuning, which can be
time-consuming and may not generalize well across different scenarios. By
contrast, the versatility of LLMs makes them suitable for processing diverse
forms of time-series data, and such adaptability is crucial given the vast
volumes of data generated in telecom. Moreover, LLM’s capability to
continuously learn and adapt to new data patterns helps mitigate the concept
drift problem, ensuring that the models remain relevant and effective over
time. As a result, the integration of LLM techniques in time-series prediction
offers a promising avenue for developing more robust and generalizable models
that can better handle the complexities of data in telecom.

Meanwhile, LLM models have excellent ICL capabilities, which means that they
can perform new tasks by leveraging contextual information in demonstrations.
In particular, it means that the LLM can directly learn from the provided
examples, and map the input-output relationships without extra model training.
Such a prediction method is much more efficient than conventional prediction
methods. Meanwhile, it is also more accessible since no professional knowledge
of model training/fine-tuning is required. In addition, multi-modal LLM-
enabled prediction can also be combined with sensing in telecom networks. In
particular, multi-modal LLM models can process and integrate information from
various data types, such as text, images, audio, and time-series data. In
addition, sensing is an important part of envisioned 6G networks, aiming to
integrate environmental information into communication networks, e.g., the
image captured by street cameras or satellites, 3D LiDAR maps and WiFi
sensing. In the context of telecom prediction, multi-modal LLMs can combine
sensing data with numerical time-series data to generate more accurate
context-aware prediction, which can be particularly useful in future 6G
networks.

Given the great potential, it is important to investigate time series LLM
techniques and applications in telecom networks. In the following, we will
introduce various LLM-based prediction methods and applications to telecom
networks.

Figure 16: Encoder-decoder-based TimeGPT for prediction problems in telecom
networks[181].

###  VII-B Pre-training Foundation Models for Zero-shot Prediction

The pursuit of training a general-purpose foundation model for time-series
data is driven by the desire to address the inherent challenges associated
with diverse and dynamic data. Traditional time-series methods may struggle to
adapt to the non-stationary properties of real-world data, where the
statistical characteristics of the series can change over time due to evolving
patterns and trends. For instance, the network traffic load level can be
affected by many factors, including time, area, environment buildings, service
types, etc. It usually requires dedicated model design and training for each
target task, and then extracts the underlying patterns from history
datasets[182]. By contrast, a general-purpose foundation model aims to
overcome these challenges by leveraging the advancements in LLM technologies.
The following will first formulate the problem of training a foundational LLM
model for time-series prediction, and then discuss different tokenization
mechanisms and model architectures.

####  VII-B1 Problem formulation

The primary goal of a foundation model is to design a zero-shot forecasting
scheme that utilizes the past $t$ time points of a time series as input to
predict the future $h$ time points. Let the input context be
$y_{1:L}:=\\{y_{1},\ldots,y_{L}\\}$ and the prediction horizon be
$y_{L+1:L+H}$. The model, denoted as $f_{\theta}$ (parameterized by $\theta$),
aims to map the context to the horizon, i.e.,
$f_{\theta}:(y_{1:L})\rightarrow\hat{y}_{L+1:L+H}$. In this setting, the
prediction model $f_{\theta}$ maps the feature space $\mathcal{X}$ to the
dependent variable space $\mathcal{Y}$. The spaces are defined as
$\mathcal{X}=\\{y_{[0:t]},x_{[0:t+h]}\\}$ and
$\mathcal{Y}=\\{y_{[t+1:t+h]}\\}$, where $h$ is the prediction horizon, $y$ is
the target time series, and $x$ are exogenous covariates. The prediction task
is to estimate the conditional distribution:

$\mathbb{P}(y_{[t+1:t+h]}|y_{[0:t]},x_{[0:t+h]})=f_{\theta}(y_{[0:t]},x_{[0:t+h]})$
(1)

####  VII-B2 Tokenization mechanisms

Motivated by ViT [183], many existing works use patching to convert the raw
input sequences to tokens. In particular, each time series $x_{[0:t]}$ is
segmented into a series of patches, which may overlap or be distinctly
separate. The patch length is denoted as $P$, and the stride, representing the
non-overlapping interval between consecutive patches, is denoted as $S$.
Consequently, this patching technique produces a sequence of patches
$x_{p}\in\mathbb{R}^{P\times N}$, where $N$ denotes the number of patches,
calculated by $N=\left\lfloor\frac{L-P}{S}\right\rfloor+2$. Before patching,
$S$ repetitions of the final value $x_{t}$ are appended to the sequence’s end.
This tokenization mechanism effectively reduces the number of input tokens
from $L$ to roughly $L/S$, which significantly diminishes the memory space
consumption and computational intensity.

####  VII-B3 Model architecture

Most existing works employ either encoder-decoder or decoder-only architecture
as the backbone model to train a time-series foundation model.

Encoder-decoder: The encoder-decoder transformer architecture stands out for
its remarkable efficiency and efficacy, primarily attributed to its self-
attention mechanism [41]. Fig.16 shows an example named TimeGPT that
exemplifies the application of the encoder-decoder transformer for prediction
problems[181]. In particular, TimeGPT inputs a historical sequence of data
points to predict future values. The inputs are added relative positional
embedding, which demonstrates higher capability to handle long sequences than
the original absolution positional embedding of the transformer [41]. Its
encoder captures temporal dependencies within the historical context, encoding
it into a latent space, while the decoder utilizes this encoded information to
predict future values. As shown in Fig.16, with its specialized architecture,
TimeGPT can address the intricacies of time-series data, such as trends and
seasonality, which makes it an ideal model for telecom time-series predicting
such as network traffic load, channel state, user mobility, etc. Once pre-
trained, such a universal model can be used for various prediction tasks
without extra training. By contrast, conventional methods such as RNN and DNN
are usually task-specific, and training a new model from scratch for each
incoming new task is time-consuming.

Decoder-only: Even though encoder-decoder models exhibit impressive
effectiveness for handling sequences, decoder-only models become more popular
in recent years. The essential difference between encoder-decoder and decoder-
only models is that the bidirectional attention is used by encoder [41], which
means each token is attending to all other tokens. In contrast, the decoder-
only model employs casual attention, where each token cannot attend to tokens
after it, but can only look at tokens before it. As shown by Fig.17, TimesFM
[184] employs decoder-only architecture to train a time-series prediction
model. Unlike traditional LLM techniques, which predict one element at a time,
TimesFM is designed to predict extended future sequences in a single step,
enhancing accuracy for long-term predictions. This flexibility also extends to
inference; given a series, the model can predict its immediate future in fewer
steps than a model with equal-length input and output segments would require.
Such fast inference could be an appealing feature for telecom applications,
because many prediction tasks require rapid response to network dynamics, such
as channel state, short-term traffic changes, and indoor user locations.
Conventional prediction methods usually take a long training time to adapt to
such network environment changes. By contrast, TimesFM has the potential to
capture short-term patterns instantly, which aligns with the fast-response
requirements of telecom networks.

In summary, the key differences between encoder-decoder and decoder-only
architecture can be found by comparing Fig. 16 and Fig. 17. In particular, the
encoder-decoder design in Fig. 16 includes an encoder to encode the raw
features into latent representations by using bidirectional attention. In
contrast, the decoder-only scheme in Fig. 17 illustrates causal attention,
e.g., the first token is attended by all other tokens, and the second token is
attended by all except the first token.

With a backbone model and an effective tokenization mechanism, one can train a
time-series prediction foundation model for telecom with a mixture of
different datasets. Meanwhile, understanding the tokenization and model
architecture differences is crucial for designing and pre-training a time
series LLM for telecom applications. For instance, the tokenization mechanism
introduced in the previous Section VII-B2 is very useful for telecom
applications, since telecom networks are associated with a large number of
network devices and end users, generating a huge number of datasets, such as
historical CSI, traffic load level[185], and network performance metrics[186].
Therefore, reducing the number of input tokens can lower the pre-training
difficulty of LLM models, especially considering that network edge devices
usually have limited computational resources.

Figure 17: A decoder-only model named TimesFM for time-series prediction
proposed in [184].

###  VII-C Frozen Pre-trained LLM for Prediction

Rather than developing a specific LLM model for prediction, frozen pre-trained
LLM refers to approaches that directly adapt a general-domain LLM to
prediction tasks. This section delves into using a pre-trained LLM for
prediction tasks without the necessity for further fine-tuning. There are two
primary approaches: prompting-based and preprocessing-based methods.
Specifically, the prompting-based methods include hard and soft prompts. Hard
prompts employ rigid and predefined textual structures to present time-series
information in a format that is intuitive for the language model. Conversely,
soft prompts adopt a more nuanced strategy by integrating trainable embeddings
within the input that subtly guide the language model’s predictions.
Meanwhile, preprocessing-based methods aim to reformat the time series
numerical values into a representation that aligns more seamlessly with LLM’s
tokenization process, rather than introducing extra template tokens or
trainable embeddings.

TABLE X: Three hard prompt examples for prediction tasks in telecom.

Network traffic load prediction | Input prompt (source) |  From $\\{t_{1}\\}$ to $\\{t_{obs}\\}$, network $\\{U_{m}\\}$ experienced $\\{x_{{t_{1}}:t_{obs}}\\}$ GB of traffic each hour.
---|---|---
Question |  What will the data traffic be on $\\{t_{obs+1}\\}$?
Output (target) |  The network will experience $\\{x_{t_{obs}+1}\\}$ GB of traffic.
Network users prediction | Input prompt (source) |  From $\\{t_{1}\\}$ to $\\{t_{obs}\\}$, the BS had $\\{x_{{t_{1}}:t_{obs}}\\}$ active connections each day.
Question |  What will the BS utilization be on $\\{t_{obs+1}\\}$?
Output (target) |  The BS will have $\\{x_{t_{obs}+1}\\}$ active connections.
Customer service prediction | Input prompt (source) |  From $\\{t_{1}\\}$ to $\\{t_{obs}\\}$, customer service received $\\{x_{{t_{1}}:t_{obs}}\\}$ calls each week.
Question |  How many service calls will be received in the week of $\\{t_{obs+1}\\}$?
Output (target) |  There will be $\\{x_{t_{obs}+1}\\}$ service calls received.

1) Prompting-based methods In leveraging prompt engineering, two predominant
prompting strategies are utilized: hard prompts [187] and soft prompts [188].

Hard prompts ($P_{\text{hard}}$) involve pre-pending a fixed textual
instruction or query to the input data sequence. Specifically, the model input
for a time-series $x_{[1:T]}$ with a hard prompt is thus formalized as the
concatenation $[P_{\text{hard}};x_{[1:T]}]$, which directs the model to
generate a prediction in response to the prompt. When designing hard prompts
for time-series prediction with language models, the general guideline is to
transform numerical data into a format that mimics natural language constructs
[187]. This involves two main components: input prompts and output prompts.
Input prompts provide historical context and highlight the target time step
for prediction, while output prompts focus on the desired prediction value,
serving as the ground truth label for training or evaluation. Table X presents
several telecom examples of designing hard prompts for specific tasks, such as
network traffic load prediction, network user number prediction, and customer
service prediction. The process mirrors the source/target structure common in
machine translation tasks or can be likened to a question-answering setting,
with the context as background information and the question seeking future
insights. Then the output prompt becomes the answer to this query, such as
”the number of active users, traffic load level at specific times, and
predicted customer calls next week”. This approach effectively bridges the gap
between raw numerical sequences and language-based data representations,
facilitating the use of language models for prediction tasks traditionally
handled by numerical methods.

Conversely, soft prompts ($P_{\text{soft}}$) introduce trainable embeddings
that are optimized during training to influence the model’s prediction subtly
[189]. The input for a soft prompt is represented as
$[P_{\text{soft}};x_{[1:T]}]$, where $P_{\text{soft}}$ constitutes a series of
parameters that are fine-tuned to enhance the predictive capability of the
model. This adjustable approach allows the model to internalize and apply
nuanced guiding signals without the rigidity of fixed textual cues. Fig. 18
shows an example of using soft prompts in [189]. Upon receiving a time series
input, it undergoes tokenization and embedding through a process called
patching, complemented by a specialized embedding layer (patch reprogram).
Subsequently, all parameters of the pre-trained LLM are frozen, and it
requires training solely for the tailored embedding layer to bridge the gap
between time series and textual data. When designing soft prompts for time-
series prediction using a pre-trained LLM, there are a couple of guiding
principles and design choices. Unlike hard prompts, soft prompts require no
explicit textual additions to the input data. The general approach involves
encoding time series data into a format that the LLM can process, harnessing
its underlying capabilities to discern patterns and generate predictions. To
utilize soft prompts prediction for telecom efficiently, one might consider
the specific characteristics of the telecom time-series data, such as traffic
patterns or usage trends, to design the transformation and reprogramming steps
that align the time-series data with the model’s language understanding
capabilities.

Figure 18:  The model framework of TIME-LLM with soft prompt[189].

2) Preprocessing-based Methods The preprocessing-based method leans on the
LLM’s inherent ability to detect and follow patterns within generalized
sequences, devoid of reliance on any specific language structure. In
particular, when numerical values are adeptly transformed into textual
strings, prediction with the model adheres to standard language model sampling
methods. Therefore, tokenization plays a pivotal role because it shapes the
model’s perception of numerical patterns. LLMTIME [37] proposes two ways to
preprocess the raw data:

  * •

Introducing extra space: For example, GPT-$3$’s tokenizer might dissect the
number $42235630$ into $[422,35,630]$, which complicates arithmetic
operations. To address this, a preprocessing step is introduced where digits
are separated by spaces, and time steps by commas, ensuring uniform
tokenization of each digit: ”4 2 2 3 5 6 3 0”. With this small change, the
tokenizations are completely different. Each digit now is processed by the
model individually.

  * •

Eliminating decimal points and rescaling Given a fixed precision, the decimal
points are redundant and unnecessary. Decimal points are excluded under fixed
precision to optimize context length, transforming a series
$"0.123,1.23,12.3,123.0"$ into $"12,123,1230,12300"$. It provides a
straightforward approach to processing the inputs.

In terms of telecom application potentials, these two preprocessing techniques
provide a simple but efficient approach to using LLM techniques for
prediction. They eliminate the need for careful designs of prompts, which can
better adapt to various prediction tasks in telecom. Preprocessing-based
methods have the potential to generate prediction results instantly based on
given raw network data input.

###  VII-D Fine-tuned LLM Prediction

Fine-tuning pre-trained LLM models presents a significant advancement for
time-series prediction, offering a powerful alternative to traditional
prediction approaches [190, 191]. General-domain LLM models, initially pre-
trained on extensive linguistic data, can be fine-tuned to capture the unique
temporal patterns inherent in time-series data. This process equips LLMs with
the ability to effectively prediction in domains where data scarcity or
specificity presents challenges to conventional deep learning models. In the
pursuit of efficiency and practicality, most recent works have shifted towards
parameter-efficient fine-tuning methods like Low-Rank Adaptation (LORA) [192]
and Layer Normalization Tuning (LNT) [193]. In particular, LORA adapts pre-
trained models to new tasks by modifying the weight matrices of the model’s
layers. Given a weight matrix $W\in\mathbb{R}^{d\times m}$ in a pre-trained
model, LORA fine-tuning introduces two low-rank matrices
$A\in\mathbb{R}^{d\times r}$ and $B\in\mathbb{R}^{r\times m}$, where $r$ is
the rank and $r\ll\min(d,m)$. The weight matrix $W$ is updated as:

$W^{\prime}=W+\Delta W,\quad\text{where}\quad\Delta W=AB.$ (2)

The matrices $A$ and $B$ are the parameters learned during fine-tuning while
the original weights $W$ are kept frozen. This results in a model that is
fine-tuned for the task at hand with only a small increase in the number of
parameters. Many works of fine-tuning LLMs proposed applying the technique to
the query (Q) and value (V) matrices in attention layers, showing notable
results without extending it to all parameters within the attention or feed-
forward layers. In the context of time-series prediction, however, this
selective fine-tuning may require adjustment. As shown in Fig. 19, LLM4TS
[190] applies LORA fine-tuning to the query (Q) and key (K), achieving state-
of-the-art performance. Using LORA allows for retaining the general
capabilities of the LLM while imbuing it with domain-specific knowledge,
ensuring that the time-series prediction model is both specialized and robust.

Figure 19:  The model framework of LLM4TS framework[190]. $Q$, $K$, $V$ are
the query, key, value vectors respectively. $W_{q}$, $W_{k}$, $W_{v}$ are the
matrices used for generating query, key and value vector.

On the other hand, LNT offers a focused approach to adapt pre-existing
parameters in transformer blocks to specific tasks. LNT specifically targets
the affine transformation parameters within the layer normalization components
of a transformer model. These parameters, such as scale and shift, originally
set to ensure standardized input distribution across network layers, become
trainable to allow the model to retain its learned representations while fine-
tuning the time-series prediction. As shown in Fig. 19, LLM4TS [190] employs
both LNT and LORA fine-tuning for the query and key. A similar strategy can be
found in [191], which freezes all attention and feed-forward layers, and only
fully fine-tunes the embedding layers and applies the LNT. Incorporating LNT
in the fine-tuning process, in the context of adapting pre-trained LLM models
for time-series prediction, provides a mechanism for the model to adjust its
internal normalization to better fit the dynamics and scale of the time-series
data.

These parameter-efficient fine-tuning methods, such as LoRA and LNT, are
crucial for the practical deployment of LLM models in telecom. Lin et al.
claims that applying LoRA to GPT-3 can reduce the number of trainable
parameters from 175.2 billion to 37.7 million[16], and combining LoRA with
federated split learning can significantly reduce computing and communication
latency at the mobile edge.

Figure 20: Multi-modality LLM for prediction problems in wireless networks.

###  VII-E Multi-modal LLM for Telecom Prediction

Multi-modal learning is a promising feature of LLM techniques, aiming to
process related information from multiple modalities, such as text, audio,
image, video, 3D maps, graphs, etc [194]. A multi-modal LLM can use diverse
encoders to extract features from different modalities into desired outputs,
indicating a more comprehensive and flexible approach to process information.
Such multi-modal capabilities can be particularly useful for integrating
sensing and communication, which is a crucial technique in 6G networks. In
particular, as shown in Fig. 20, LLMs can include multiple inputs with diverse
modalities, e.g., the image captured by satellite or street cameras, 3D LiDAR
maps and videos collected by vehicles. Sensing has become a critical part of
envisioned 6G networks, and multi-modal LLMs are capable of making the most of
the collected sensing information. On the other hand, LLMs can also include
conventional tabular-based numerical input, and further consider textual input
and prompt instructions. With multi-modal inputs, LLM agents can better
understand the surrounding environment and then make more accurate predictions
for network dynamics.

1) Channel state information (CSI) prediction: CSI plays an increasingly vital
role in wireless networks, enabling the transmitter to adjust the transmission
parameters based on current channel conditions and, therefore, achieve better
performance. Prediction-based methods are appealing methods to obtain
instantaneous CSI. For instance, Jiang et al. [195] applied deep learning for
CSI prediction using generated or historical data. Most existing studies
consider single modality input, which usually consists of tabular-based
numerical data such as historical CSI. However, the real-world environment can
be more complicated, and CSI may be affected by many other factors such as
weather conditions and dense buildings [5, 196]. These multi-modal inputs,
such as weather maps and building distributions, can provide a more
comprehensive understanding of the signal transmission environment, but
jointly processing these inputs is beyond the capabilities of existing
techniques. Multi-modal LLMs offer promising solutions by jointly considering
diverse modalities and data sources, producing more accurate CSI prediction
results. In addition, users can provide textual prompt instructions, which are
easy-accessible and user-friendly for non-researcher users.

2) Prediction-based mmWave/THz beamforming: The increasing traffic demand and
limited bandwidth resources make mmWave and THz communications promising
techniques. However, these high-frequency transmissions are highly directional
and vulnerable to signal blockages. Consequently, efficient beamforming and
alignment are required to achieve reliable mmWave and THz networks. For
instance, Ke et al. [197] applied a Gaussian process-based ML scheme for UAV
position prediction and UAV-mmWave beam-tracking, and Shah et al. [198]
deployed LSTM networks to predict multiple mmWave beams from multiple cells.
These predictions usually consider numerical input, especially historical data
[197]. Charan et al. [25] introduced computer vision-aided techniques for
signal blockage prediction, using cameras on BSs to capture possible blockages
and then initiate user hand-off beforehand. However, it is still limited to a
single image modality with limited environmental information. By contrast,
multi-modal LLMs can take holographic input from the environment, and jointly
consider historical tracks, instant images, and text instructions, etc. These
comprehensive inputs can produce more accurate and reliable prediction
results, contributing to efficient mmWave and THz beamforming.

3) Traffic load prediction: Accurate traffic load prediction is the
prerequisite of efficient network management, which is related to user
numbers, service types, time periods, and so on. Similar to CSI prediction,
most existing studies take numerical datasets as single modal input [199,
200]. For instance, Alekseeva et al. [199] compared the performance of seven
ML algorithms (including Bagging, Random Forest, Gradient Boosting, Linear
Regression, Bayesian Regression, Huber Regression, and SVM Regression) on the
task of traffic load prediction. Their findings indicate that Boosting-based
methods demonstrate superior performance when handling large volumes of load
data, yet incurring significant training costs. Hu et al. [200] integrated a
sequence of AutoEncoders to extract multiple sets of latent temporal features
from historical load data for load prediction, which ensures that extracted
feature sets are representative of the entire load data. Most existing studies
mainly consider two factors: the spatial correlation between nearby BSs and
the temporal dynamics captured in historical data. However, Abdullah et al.
[201] suggested that various meteorological factors, such as rain, wind, and
temperature, can also significantly influence the volume of traffic loads.
Consequently, multi-modal LLMs may be used to harness diverse information
streams, including spatial BS corrections, temporal historical traffic loads,
and environmental factors, facilitating accurate load prediction and providing
effective network management and service delivery.

TABLE XI: Summary of LLM-based Prediction for Telecom.

LLM-based prediction techniques |  Main features |  Input and fine-tunning requirements |  Advantages compared with conventional approaches |  Telecom prediction application opportunities
---|---|---|---|---
Pre-training foundation models |  Leverages pre-trained models on diverse time series datasets to capture general temporal patterns. It means training LLM models from scratch specifically for prediction purposes. |  Requires a large corpus of time-series data for initial pre-training, but collecting these datasets may be difficult in telecom; fine-tuning may be needed for specific telecom tasks. |  The model has zero-shot prediction capabilities, and quickly adapts to new tasks with minimal fine-tuning; captures a wide range of temporal dynamics. |  A prediction foundation model for telecom can handle various short-term or long-term prediction tasks, such as traffic load prediction, CSI prediction, user number estimation, and so on.
Frozen pre-trained LLM for prediction |  Using pre-trained LLM models without fine-tuning their parameters, including prompting-based and preprocessing-based methods. |  Tokenization and embedding of time series data; For prompt-based methods, the prompt format must be carefully designed; No fine-tuning for LLM models is required. |  Low computational cost and design complexity; Leveraging the generalization capabilities of a pre-trained LLM directly. |  This technique is particularly useful for short-term prediction, such as short-term traffic load and network performance prediction. The low computational cost can also adapt to network edge, even mobile applications.
Fine-tuned LLM prediction |  Adapts a pre-trained LLM to telecom-specific prediction tasks through fine-tuning techniques such as LoRA and LNT. |  It requires time series data for fine-tuning; may require parameter-efficient fine-tuning like LoRA and LNT to improve the efficiency. |  Fine-tuning can incorporate telecom domain knowledge into LLM models, improving the accuracy and specificity of telecom tasks. |  Fine-tuning LLMs can better adapt to specific tasks in telecom, e.g., collecting specific datasets to fine-tune an LLM for user localization. It is more flexible than pre-trained models from scratch, and more reliable than pure prompting-based methods.
Multi-modality prediction |  Using LLM models to jointly consider multi-modal environment information, e.g., tabular data, text, and image, aiming to provide more accurate prediction results. |  It requires multi-modal input and proper prompt to predict desired output; LLM models can be specifically pre-trained/fine-tuned to further improve the performance and generalization capabilities. |  Multi-modal can take advantage of inputs with various modalities. With these comprehensive inputs, LLMs can better predict the network dynamics than existing methods. |  Sensing is an important part of 6G networks, and multi-modal sensing can provide more comprehensive input for LLM models, producing more accurate prediction results by jointly considering various inputs, e.g., more accurate traffic load prediction and beam steering.

4) Quality of Experience (QoE) prediction: QoE is a measure of the customer’s
experiences of specific services, which is a useful metric in diverse mobile
scenarios, such as mobile edge computing[135], edge caching [202], and
resource allocation [203]. QoE is closely related to the user’s natural
language comments. An example is given by [135]: “I am having the same issues
as everyone else…Phone shows 5 bars on 4G.” Makes calls and texts just fine
but no imessage or internet (safari as well as any other apps that require
connectivity). Right now the two things I have noticed are that I’m more
likely to have it work late at night (11pm-2am) and more likely to have it
work when I’m outdoors…” Most existing studies predict QoE by extracting the
key attributes of users, devices, applications, and networks for modeling and
measuring. However, this comment indicates a specific network issue ”no
imessage or internet at midnight”. Extracting such an informative and specific
user experience to several attributes could lead to considerable information
loss, and therefore the service provider cannot fully understand the user’s
demand. With multi-modal LLMs, user’s textual comments and network numerical
metrics can be jointly evaluated, providing a comprehensive evaluation of the
network performance and user experiences. In addition, multi-modal LLMs can
also be used to generate and predict user experience using LLM’s comprehension
and reasoning capabilities.

###  VII-F Discussions and Analyses

Table XI summarized the LLM-enabled prediction techniques in terms of main
features, input and fine-tuning requirements, advantages, and telecom
prediction application opportunities. We summarize the key findings as
follows.

Firstly, large-scale time-series datasets are important for building Time-LLM
for telecom. Previous sections have demonstrated LLM’s potential for solving
time-series prediction problems. However, it is worth noting that time-series
datasets are prerequisites of pre-training Time-LLM, and then the LLM can
understand and capture the hidden patterns of the input data. Despite the
importance, collecting such datasets can be difficult in telecom due to
various data formats and sources, different network operators, customer
privacy, etc.

Secondly, prompting and preprocessing-based methods are the most efficient
approaches to using LLM for prediction tasks. Compared with pre-training and
fine-tuning, the discussions in Section VII-C demonstrate that prompting is
one of the most straightforward methods of using an LLM for prediction tasks.
Such an advantage can still be explained by LLM’s impressive zero-shot
learning capabilities. In addition, preprocessing input data is another simple
method. Transforming numerical values into textual strings can make the most
of LLM’s capabilities in processing standard language tasks. These two methods
are particularly useful for short-term prediction problems in telecom with
instant responses.

In addition, previous sections also show that parameter-efficient fine-tuning
methods are critical for LLM deployment in telecom. Section VII-D introduced
two parameter-efficient methods, LoRA and LNT, to fine-tune LLM for prediction
tasks. Efficient fine-tuning methods can improve overall computing efficiency,
lowering the demand for computational resources. These features are very
useful for processing various tasks ranging from generation and classification
to prediction problems in the telecom field such as in Section VII-D,
especially considering limited computational and storage resources at the
network edge.

Finally, multi-modal LLM has great potential for telecom applications.
Incorporating multi-modality LLM into telecom has been discussed in multiple
existing studies[30], and Section VII-E investigates the potential for telecom
prediction problems such as CSI prediction, prediction-based beamforming, and
QoE prediction. The key motivation is that multi-modal information from
multiple sources can contribute to prediction accuracy, and such enhancement
can further improve the network operator’s decision-making.

##  VIII Challenges and Future Directions of LLM-empowered telecom

This section will introduce the challenges of realizing LLM-empowered telecom,
including telecom-domain LLM training, practical LLM deployment in telecom,
and prompt engineering for telecom applications. Then, we identify several
future directions, e.g., LLM-enabled planning, model compression and fast
inference, overcoming hallucination problems, retrieval augmented-LLM, and
economic and affordable LLMs.

###  VIII-A Challenges of Applying LLM Techniques to Telecom

####  VIII-A1 Telecom-domain LLM training

Previous sections have shown the importance of building telecom-specific LLMs,
e.g., telecom-domain question answering[100], telecom troubleshooting[23], and
standard specification classification[13]. Despite the great potential,
training an LLM model specifically for telecom presents unique challenges due
to the complex nature of large-scale communication networks. In the following,
we will analyze this challenge in detail.

Sufficient domain-specific datasets are prerequisites for training a telecom
LLM. Unlike general-domain LLM models, which can leverage large-scale text
corpora, obtaining a sizable dataset exclusively focused on communication
networks can be challenging. Existing studies usually focus on one specific
task and then build the corresponding dataset, e.g., the trouble report
dataset [23], 3GPP specification dataset [13], and telecom question answering
dataset [100]. However, these datasets are usually small-scale and task-
specific, and a comprehensive large-scale dataset should include network-
related documents, standard specifications, protocols, textbooks, research
papers, and other relevant sources. Maatouk et al. started the exploration in
[110] by building a dataset with 10000 telecom-related questions and answers,
including around 25000 pages and 6 million words. More efforts are needed to
provide more comprehensive and diverse datasets for telecom LLM training.

Meanwhile, it is worth noting that telecom networks involve a large number of
various concepts such as network protocols, routing algorithms, network
topologies, network security, etc. Therefore, teaching an LLM to comprehend
and reason about these complex concepts requires a robust training strategy.
An effective approach is to pre-train the LLM on a large-scale general
language corpus and then fine-tune it on specific communication network
datasets, e.g., datasets for the BS services, historical datasets for
prediction, or datasets for edge computing-related tasks. In addition,
balancing model size and performance is crucial. LLM models trained on large-
scale datasets tend to be computationally expensive and memory-intensive.
Appropriate model size can reduce the burden on energy and computation
resources during pre-training and fine-tuning phases. In addition, balancing
model size and performance is crucial to ensure practical usability,
especially considering scenarios with limited computational capacity such as
vehicles and mobile phones. Therefore, techniques like model compression,
knowledge distillation, or utilizing specialized hardware accelerators can be
explored to reduce the model’s size and enhance its efficiency without
compromising its understanding of communication networks.

In summary, training an LLM specifically for telecom is of great importance.
However, multiple challenges must be addressed, including domain-specific
training data, complex networking concepts, and balancing model size and
performance. Overcoming these challenges will enable the development of a
robust LLM capable of effectively applying networking domain knowledge to
various telecom tasks.

####  VIII-A2 Practical LLM deployment in telecom

To leverage the benefits of LLM techniques, the models should be properly
deployed in telecom networks. Specifically, LLM models can be deployed at
different levels, including central cloud, network edge, or user devices. We
have introduced the features of each approach in Section III-F. However, the
related studies are still in very early stages, and the proposed schemes
mainly focus on system-level design and definitions. The following will
discuss the key challenges and difficulties for practical LLM deployment in
telecom networks.

Firstly, many real-world wireless applications have stringent requirements for
service delay, e.g., autonomous driving and robotic control. With such time
constraints, using a central cloud-based LLM to process these latency-critical
tasks can be inappropriate, since the task uploading and solution downloading
may increase the service delay. Additionally, if the task involves image and
video processing, the uploading and downloading process will significantly
increase the latency, especially considering the limited backhaul capacity.
For instance, the image classification tasks introduced in Section V-D require
rapid responses for signal blockage prediction and autonomous driving[25]
[146], and processing these requirements on cloud can be impractical due to
high service latency. In addition, the LLM inference time will also contribute
to system latency, ranging from 0.58 to 90 seconds. Therefore, the service
time should be very carefully evaluated before using a LLM for latency-
critical applications.

Network edge provides an efficient approach for computational task processing,
and edge intelligence has become an appealing direction to deploy ML
algorithms in telecom networks, especially for distributed algorithms such as
federated learning. However, network edge servers have limited computational
or storage capacity, and LLMs are usually computationally intensive with large
model sizes, which may prevent edge-LLM deployment. By contrast, hybrid
deployment approaches, combining central cloud, edge, and user device
deployments, can provide a balance between scalability, low latency, and
privacy. For example, a hierarchical architecture can be used, where various
LLMs are deployed at the central cloud for resource-intensive tasks, at the
network edge for latency-sensitive tasks, and on user devices for personalized
or offline tasks. However, coordinating these different deployment levels can
be challenging and requires efficient protocols and algorithms. Xu et al.
proposed a hybrid deployment approach in [17], but this is still an open
challenge for the communication society. Deploying LLM models at different
levels, including the central cloud, network edge, or user devices, offers
unique opportunities and challenges in telecom applications. Each deployment
methods has its considerations regarding latency and computational resources.
Hybrid deployment can be an ideal solution, leveraging the advantages of
multiple deployment levels, but the coordination may lead to potential issues.

####  VIII-A3 Prompt engineering for telecom applications

Prompt engineering is a crucial aspect of utilizing LLM techniques
effectively, as it plays a significant role in guiding the model’s behaviour
and generating desired outputs. However, designing prompts for telecom
applications presents unique challenges due to the domain complexity. This
subsection will discuss the challenges of prompt engineering for telecom
networks.

Telecom networks encompass a wide range of concepts, protocols, and
technologies, making it challenging to distill the necessary information into
a concise prompt. The diverse nature of the domain requires a deep
understanding of networking principles and the ability to capture specific
nuances related to network architectures, protocols, performance optimization,
and security. To design effective prompts, researchers must identify the most
relevant components and provide concise yet comprehensive instructions to
LLMs.

Meanwhile, prompt designs should strike a balance between being specific
enough to guide the LLM in generating accurate and contextually appropriate
responses, while also remaining general enough to handle a wide range of
network-related queries or tasks. Achieving this balance is crucial as overly
specific prompts may limit the model’s ability to generalize, while a general
prompt may lead to irrelevant responses. Fine-tuning the prompt language and
incorporating domain-specific keywords or phrases can help achieve the desired
balance for telecom applications.

Moreover, telecom tasks often require the LLM to consider contextual
information and situational variables. For example, network troubleshooting
may involve analyzing network logs, diagnosing performance issues, or
identifying security vulnerabilities. Designing prompts that take into account
the relevant context and guide the model to consider appropriate factors can
significantly enhance the accuracy and relevance of the generated responses.
Techniques like providing explicit context cues or utilizing conditional
generation can be explored.

To summarize, prompt design of LLMs for telecom applications poses a
significant challenge due to the intricate and constantly evolving nature of
the domain. Crafting effective prompts necessitates a profound comprehension
of networking principles, the capacity to strike a balance between specificity
and generality, and an awareness of contextual factors. Collaboration and
input from researchers and industry experts play a crucial role in developing
robust prompt engineering strategies that empower LLM agents to deliver
precise, pertinent, and impartial responses in telecom applications.

###  VIII-B Future Directions

This subsection will identify possible future directions for LLM-empowered
telecom.

####  VIII-B1 LLM-enabled planning in telecom

Telecom networks are complicated large-scale systems, and many tasks require
multi-step thinking and planning. For instance, the previous Section IV-C2
introduced an example of coding wireless projects with multi-step planning,
and many optimization problems with multiple control variables have to been
solved step-by-step[2]. However, recent benchmarks have shown that LLM models
struggle with tasks requiring complex planning and sequential decision-making.
Some existing studies such as [14] and [18] propose to improve the multi-step
planning capabilities by step-by-step and CoT prompting. Despite the
satisfactory performance in [14] and [18], they require dedicated analyses to
manually decompose a complicated task into multiple sub-tasks. Therefore,
future studies should aim at developing better algorithms for planning that
can be integrated into LLM models, and such multi-step planning capability is
crucial for solving telecom-domain tasks. This might involve incorporating
structured reasoning and problem-solving frameworks into the models, enabling
them to break down tasks into smaller and more manageable sub-tasks.
Therefore, automated task decomposition can be an attractive solution to
improve the planning performance of LLM models. However, automatically
decoupling one complicated task into multiple sub-tasks is still very
challenging in the telecom field. Additionally, another solution could be
integrating simulation environments directly within the training process,
allowing models to practice and refine their planning skills in a controlled
setting before applying them to real-world tasks. It allows the LLM models to
improve the planning performance by trial-and-error before applying it to
telecom tasks.

####  VIII-B2 Model compression and fast inference for network edge and mobile
applications

The model size is one of the key bottlenecks of applying LLMs to the telecom
domain, leading to stringent requirements for computational and storage
capacities. Therefore, compressing the model size to adapt to network edge and
mobile applications becomes a promising direction. In addition, it will also
contribute to the fast inference of LLM models, since many wireless
applications require rapid response time and low latency. For instance, Xu et
al. proposed an on-device inference model specifically designed for efficient
generative natural language processing tasks[97], achieving a $9.3\times$
faster generation speed. Such a technique can be very promising for LLM-
enabled mobile applications in telecom, enabling faster response time for user
inquiries. Meanwhile, it is worth noting that compressing the model size may
degrade the LLM performance, and how to balance the model size and performance
requires more research efforts. It calls for novel model compression and
pruning techniques to reduce the storage and computation burdens at the
network edge; on the other hand, standard metrics must be defined to evaluate
the performance of LLM models in the telecom domain, e.g., accuracy and
hallucination probability as we introduced in previous Section III-E.

####  VIII-B3 Overcoming hallucination problems in telecom applications

Hallucination, or the generation of factually incorrect or nonsensical
information, remains a significant issue for LLM applications. Specifically,
it means that the LLM may generate some nonsensical answers or solutions for
the given telecom task. Hallucination can severely undermine the reliability
and credibility of LLM-generated content, degrading the performance on many
downstream tasks. For instance, a nonsensical answer may be generated when
using LLM for telecom question answering. Overcoming these issues is critical
for telecom applications to guarantee network service quality and reliability.
To this end, future research should focus on developing methods to reduce
hallucination and improve the factual accuracy of model outputs. This could
include enhancing the training datasets with more verified and reliable
sources, implementing post-generation verification steps, or incorporating
cross-referencing mechanisms within the model. Additionally, exploring the use
of external knowledge bases and real-time fact-checking during the generation
process could help mitigate this issue. Recently, it has been demonstrated
that under specific evaluation conditions, LLMs exhibit exceptional zero-shot
capabilities in assessing factual consistency [204]. This underscores their
potential to become leading evaluators of hallucination in various contexts.
Moreover, techniques such as adversarial testing can also help in assessing
their susceptibility to hallucination, where models are deliberately presented
with complex or misleading inputs.

####  VIII-B4 Retrieval augmented-LLM for telecom

Retrieval augmentation is an important direction for LLM development, which
retrieves facts from an external knowledge base to ground LLM models on the
most up-to-date information. Telecom networks are constantly evolving and
updating, and retrieval augmentation has great potential for telecom
applications. In particular, retrieval-augmented LLM can improve the quality
and relevance of the generated responses since the LLM has access to more
accurate and relevant information. However, current retrieval-augmented
generation models increase the context length, which in turn decreases the
efficiency of the model due to the added computational cost, which may lead to
severe slow-response issues. Such slow response may increase the overall
network latency and degrade the service quality. It may prevent the
application of some scenarios with tight delay budget, which is very common in
wireless networks. Therefore, future research could focus on improving the
efficiency of retrieval-augmented generation by optimizing retrieval
mechanisms to balance context relevance and length. This could involve
developing more advanced indexing and search algorithms that require less
memory. Additionally, dynamically adjusting the amount of retrieved
information based on the query’s complexity could help maintain or improve
efficiency without sacrificing the quality of the output.

####  VIII-B5 Economic and affordable LLM models

Despite the great potential and advantages, training an LLM model can be
financially expensive. For instance, the training expenses for GPT-4 exceeded
$ 100 million, and the LLaMa2 70B model was trained on 2048 GPUs A100 for 23
days with $ 1.7 million estimated cost [205]. Although training some smaller
models such as LLaMa2 7B can be much cheaper, the affordability of LLM
techniques is still one of the main concerns. For instance, the study in [206]
shows that using GPT-4 to support customer service can cost more than $ 21,000
per month for a small business. Meanwhile, there are many LLM APIs with
various prices, including the prompt cost proportional to the prompt length,
generation cost related to the generation length, and a possible fixed cost
per query. For example, it costs $30 for 10M tokens using OpenAI’s GPT-4,
while only $ 0.2 for GPT-J hosted by Textsyth[207]. The financing cost of
training, fine-tuning, and deploying LLM models will significantly affect the
application in telecom networks. Given the heterogeneous prices and service
quality, it is of great importance to evaluate the financing cost of deploying
LLM models in telecom networks, e.g., balancing the possible performance
improvement and the LLM deployment cost, and using LLM models in an economic
manner for telecom applications. However, this direction has limited existing
studies, and it still requires more research efforts.

##  IX Conclusion

Recently, large language model (LLM) have shown great promise in many fields,
specifically for language-related tasks such as summarization and question and
answering. LLM-based solutions have also been primarily investigated in the
telecom field. In this work, we aim to present a comprehensive survey on LLM
for Telecom. In particular, we first introduced the LLM fundamentals. We
present a comprehensive overview of the model architecture, pre-training,
fine-tuning, inference and utilization, evaluation, and deployment of LLM-
based solutions. Then, a comprehensive survey of existing works on the key
techniques and applications in terms of generation, classification,
optimization, and prediction problems is presented. These investigations and
analyses have proven that LLMs have outstanding potential to bring artificial
general intelligence to the telecom field using in-context and zero-shot
learning capabilities. Finally, we discussed the key challenges, such as data
sets and cost, as well as future research opportunities of LLM-empowered
telecom. We hope this work can serve as a good reference for researchers and
engineers to better understand the existing works, potentials, challenges, and
opportunities of applying LLM for the telecom field.

## References

  * [1] Z. Zhang, Y. Xiao, Z. Ma, M. Xiao, Z. Ding, X. Lei, G. K. Karagiannidis, and P. Fan, “6G wireless networks: Vision, requirements, architecture, and key technologies,” _IEEE vehicular technology magazine_ , vol. 14, no. 3, pp. 28–41, 2019.
  * [2] H. Zhou, M. Erol-Kantarci, Y. Liu, and H. V. Poor, “A survey on model-based, heuristic, and machine learning optimization approaches in RIS-aided wireless networks,” _IEEE Communications Surveys & Tutorials (Early access)_, Dec. 2023.
  * [3] H. Zhou, M. Erol-Kantarci, and V. Poor, “Knowledge transfer and reuse: A case study of AI-enabled resource management in RAN slicing,” _IEEE Wireless Communications_ , vol. 30, no. 5, pp. 160–169, Oct. 2022.
  * [4] H. Zhou, M. Elsayed, and M. Erol-Kantarci, “RAN resource slicing in 5G using multi-agent correlated Q-learning,” in _Proc. of 2021 IEEE PIMRC_ , Sep. 2021, pp. 1179–1184.
  * [5] C. Luo, J. Ji, Q. Wang, X. Chen, and P. Li, “Channel state information prediction for 5G wireless communications: A deep learning approach,” _IEEE Trans. on Network Science and Engineering_ , vol. 7, no. 1, pp. 227–236, 2018.
  * [6] H. Zhang, H. Zhou, and M. Erol-Kantarci, “Federated deep reinforcement learning for resource allocation in O-RAN slicing,” in _Proc. of IEEE 2022 GLOBECOM Conf._ , 2022, pp. 958–963.
  * [7] K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, L. Hou, K. Clark, S. Pfohl, H. Cole-Lewis, D. Neal _et al._ , “Towards expert-level medical question answering with large language models,” _arXiv preprint arXiv:2305.09617_ , 2023.
  * [8] P. Colombo, T. P. Pires, M. Boudiaf, D. Culver, R. Melo, C. Corro, A. F. Martins, F. Esposito, V. L. Raposo, S. Morgado _et al._ , “Saullm-7b: A pioneering large language model for law,” _arXiv preprint arXiv:2403.03883_ , 2024.
  * [9] S. Wu, O. Irsoy, S. Lu, V. Dabravolski, M. Dredze, S. Gehrmann, P. Kambadur, D. Rosenberg, and G. Mann, “Bloomberggpt: A large language model for finance,” _arXiv preprint arXiv:2303.17564_ , 2023.
  * [10] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, _et al._ , “A survey of large language models,” _arXiv:2303.18223_ , 2023.
  * [11] J. Wei, L. Hou, A. Lampinen, X. Chen, D. Huang, Y. Tay, X. Chen, Y. Lu, D. Zhou, T. Ma _et al._ , “Symbol tuning improves in-context learning in language models,” _arXiv preprint arXiv:2305.08298_ , 2023.
  * [12] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 1877–1901, 2020.
  * [13] L. Bariah, H. Zou, Q. Zhao, B. Mouhouche, F. Bader, and M. Debbah, “Understanding telecom language through large language models,” _arXiv:2306.07933_ , 2023.
  * [14] Y. Du, S. C. Liew, K. Chen, and Y. Shao, “The power of large language models for wireless communication system development: A case study on fpga platforms,” _arXiv:2307.07319_ , 2023.
  * [15] Y. Shen, J. Shao, X. Zhang, Z. Lin, H. Pan, D. Li, J. Zhang, and K. B. Letaief, “Large language models empowered autonomous edge AI for connected intelligence,” _IEEE Communications Magazine_ , 2024.
  * [16] Z. Lin, G. Qu, Q. Chen, X. Chen, Z. Chen, and K. Huang, “Pushing large language models to the 6G edge: Vision, challenges, and opportunities,” _arXiv:2309.16739_ , 2023.
  * [17] M. Xu, N. Dusit, J. Kang, Z. Xiong, S. Mao, Z. Han, D. I. Kim, and K. B. Letaief, “When large language model agents meet 6G networks: Perception, grounding, and alignment,” _arXiv:2401.07764_ , 2024.
  * [18] Q. Xiang, Y. Lin, M. Fang, B. Huang, S. Huang, R. Wen, F. Le, L. Kong, and J. Shu, “Toward reproducing network research results using large language models,” in _Proc. of the 22nd ACM Workshop on Hot Topics in Networks_ , 2023, pp. 56–62.
  * [19] L. Li, Y. Zhang, and L. Chen, “Prompt distillation for efficient llm-based recommendation,” in _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management_ , 2023, pp. 1348–1357.
  * [20] Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou _et al._ , “The rise and potential of large language model based agents: A survey,” _arXiv preprint arXiv:2309.07864_ , 2023.
  * [21] C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. Chen, “Large language models as optimizers,” _arXiv:2309.03409_ , 2023.
  * [22] D. Wu, X. Wang, Y. Qiao, Z. Wang, J. Jiang, S. Cui, and F. Wang, “Large language model adaptation for networking,” _arXiv:2402.02338_ , 2024.
  * [23] N. Bosch, “Integrating telecommunications-specific language models into a trouble report retrieval approach,” Master’s thesis, KTH, School of Electrical Engineering and Computer Science (EECS), 2022.
  * [24] Y. Xu, Y. Chen, X. Zhang, X. Lin, P. Hu, Y. Ma, S. Lu, W. Du, Z. Mao, E. Zhai _et al._ , “Cloudeval-yaml: A practical benchmark for cloud configuration generation,” _arXiv preprint arXiv:2401.06786_ , 2023.
  * [25] G. Charan, M. Alrabeiah, and A. Alkhateeb, “Vision-aided 6G wireless communications: Blockage prediction and proactive handoff,” _IEEE Trans. on Vehicular Technology_ , vol. 70, no. 10, pp. 10 193–10 208, 2021.
  * [26] M. Matsuura, Y. K. Jung, and S. N. Lim, “Visual-LLM zero-shot classification,” 2023. [Online]. Available: https://www.crcv.ucf.edu/wp-content/uploads/2018/11/Misaki-Final-report.pdf
  * [27] S. Booth, W. B. Knox, J. Shah, S. Niekum, P. Stone, and A. Allievi, “The perils of trial-and-error reward design: misdesign through overfitting and invalid task specifications,” in _Proc. of the AAAI Conf. on Artificial Intelligence_ , vol. 37, no. 5, 2023, pp. 5920–5929.
  * [28] S. Tarkoma, R. Morabito, and J. Sauvola, “AI-native interconnect framework for integration of large language model technologies in 6G systems,” _arXiv:2311.05842_ , 2023.
  * [29] Z. Chen, Z. Zhang, and Z. Yang, “Big AI models for 6G wireless networks: Opportunities, challenges, and research directions,” _arXiv:2308.06250_ , 2023.
  * [30] L. Bariah, Q. Zhao, H. Zou, Y. Tian, F. Bader, and M. Debbah, “Large language models for telecom: The next big thing?” _arXiv:2306.10249_ , 2023.
  * [31] S. Xu, C. K. Thomas, O. Hashash, N. Muralidhar, W. Saad, and N. Ramakrishnan, “Large multi-modal models (LMMs) as universal foundation models for AI-native wireless systems,” _arXiv:2402.01748_ , 2024.
  * [32] W. Wang, C. Zhou, H. He, W. Wu, W. Zhuang, and X. Shen, “Cellular traffic load prediction with LSTM and gaussian process regression,” in _Proc. of 2020 IEEE Intl. Conf. on communications (ICC)_ , 2020, pp. 1–6.
  * [33] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou _et al._ , “Chain-of-thought prompting elicits reasoning in large language models,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 24 824–24 837, 2022.
  * [34] J. Song, Z. Zhou, J. Liu, C. Fang, Z. Shu, and L. Ma, “Self-refined large language model as automated reward function designer for deep reinforcement learning in robotics,” _arXiv:2309.06687_ , 2023.
  * [35] M. Kwon, S. M. Xie, K. Bullard, and D. Sadigh, “Reward design with language models,” _arXiv:2303.00001_ , 2023.
  * [36] Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar, “Eureka: Human-level reward design via coding large language models,” _arXiv:2310.12931_ , 2023.
  * [37] N. Gruver, M. Finzi, S. Qiu, and A. G. Wilson, “Large language models are zero-shot time series forecasters,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [38] Y. Li, Z. Li, K. Zhang, R. Dan, S. Jiang, and Y. Zhang, “Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge,” _Cureus_ , vol. 15, no. 6, 2023.
  * [39] Z. Xu, Y. Zhang, E. Xie, Z. Zhao, Y. Guo, K. K. Wong, Z. Li, and H. Zhao, “Drivegpt4: Interpretable end-to-end autonomous driving via large language model,” _arXiv preprint arXiv:2310.01412_ , 2023.
  * [40] J. Zhang, Y. Hou, R. Xie, W. Sun, J. McAuley, W. X. Zhao, L. Lin, and J.-R. Wen, “Agentcf: Collaborative learning with autonomous language agents for recommender systems,” _arXiv preprint arXiv:2310.09233_ , 2023.
  * [41] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in Neural Information Processing Systems_ , vol. 30, 2017.
  * [42] N. Shazeer, “Fast transformer decoding: One write-head is all you need,” _arXiv:1911.02150_ , 2019.
  * [43] J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai, “Gqa: Training generalized multi-query transformer models from multi-head checkpoints,” 2023.
  * [44] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv:1810.04805_ , 2018.
  * [45] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” _arXiv:1907.11692_ , 2019.
  * [46] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “Albert: A lite bert for self-supervised learning of language representations,” _arXiv:1909.11942_ , 2019.
  * [47] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” _Journal of machine learning research_ , vol. 21, no. 140, pp. 1–67, 2020.
  * [48] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” _arXiv:1910.13461_ , 2019.
  * [49] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann _et al._ , “Palm: Scaling language modeling with pathways,” _Journal of Machine Learning Research_ , vol. 24, no. 240, pp. 1–113, 2023.
  * [50] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar _et al._ , “Llama: Open and efficient foundation language models,” _arXiv:2302.13971_ , 2023.
  * [51] A. Zeng, X. Liu, Z. Du, Z. Wang, H. Lai, M. Ding, Z. Yang, Y. Xu, W. Zheng, X. Xia _et al._ , “Glm-130b: An open bilingual pre-trained model,” _arXiv:2210.02414_ , 2022.
  * [52] M. Guo, Z. Dai, D. Vrandečić, and R. Al-Rfou, “Wiki-40b: Multilingual language model dataset,” in _Proceedings of the 12th Language Resources and Evaluation Conference_ , 2020, pp. 2440–2452.
  * [53] I. Beltagy, K. Lo, and A. Cohan, “SciBERT: A pretrained language model for scientific text,” _arXiv:1903.10676_ , 2019.
  * [54] F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn, “A systematic evaluation of large language models of code,” in _Proc. of the 6th ACM SIGPLAN Intl. Symposium on Machine Programming_ , 2022.
  * [55] H. Laurençon, L. Saulnier, T. Wang, C. Akiki _et al._ , “The bigscience roots corpus: A 1.6 TB composite multilingual dataset,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 31 809–31 826, 2022.
  * [56] T. Kudo, “Subword regularization: Improving neural network translation models with multiple subword candidates,” _arXiv:1804.10959_ , 2018.
  * [57] Z. Li, S. Zhuang, S. Guo, D. Zhuo, H. Zhang, D. Song, and I. Stoica, “Terapipe: Token-level pipeline parallelism for training large-scale language models,” in _Proc. of 38th Intl. Conf. on Machine Learning (ICML)_ , 2021, pp. 6543–6552.
  * [58] Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu _et al._ , “Gpipe: Efficient training of giant neural networks using pipeline parallelism,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [59] M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” _arXiv:1909.08053_ , 2019.
  * [60] S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He, “Zero: Memory optimizations toward training trillion parameter models,” in _Proc. of SC20: Intl. Conf. for High Performance Computing, Networking, Storage and Analysis_ , 2020, pp. 1–16.
  * [61] J. Wei, M. Bosma, V. Y. Zhao _et al._ , “Finetuned language models are zero-shot learners,” _arXiv preprint arXiv:2109.01652_ , 2021.
  * [62] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray _et al._ , “Training language models to follow instructions with human feedback,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 27 730–27 744, 2022.
  * [63] OpenAI, “Gpt-4v(ision) system card,” _OpenAI_ , 2023.
  * [64] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma _et al._ , “Scaling instruction-finetuned language models,” _arXiv:2210.11416_ , 2022.
  * [65] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving, “Fine-tuning language models from human preferences,” _arXiv preprint arXiv:1909.08593_ , 2019.
  * [66] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” _Advances in Neural Information Processing Systems_ , vol. 30, 2017.
  * [67] C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu _et al._ , “Lima: Less is more for alignment,” _arXiv:2305.11206_ , 2023.
  * [68] J. Wei, X. Wang, D. Schuurmans, M. Bosma _et al._ , “Chain-of-thought prompting elicits reasoning in large language models,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 24 824–24 837, 2022.
  * [69] J. Liu, D. Shen, Y. Zhang, B. Dolan, L. Carin, and W. Chen, “What makes good in-context examples for gpt-$3$?” _arXiv preprint arXiv:2101.06804_ , 2021.
  * [70] Y. Lu, M. Bartolo, A. Moore, S. Riedel, and P. Stenetorp, “Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity,” _arXiv preprint arXiv:2104.08786_ , 2021.
  * [71] N. Wies, Y. Levine, and A. Shashua, “The learnability of in-context learning,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [72] J. Von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, and M. Vladymyrov, “Transformers learn in-context by gradient descent,” in _Proc. of the 40th ICML_ , 2023, pp. 35 151–35 174.
  * [73] J. Wei, J. Wei, Y. Tay, D. Tran, A. Webson, Y. Lu, X. Chen, H. Liu, D. Huang, D. Zhou _et al._ , “Larger language models do in-context learning differently,” _arXiv:2303.03846_ , 2023.
  * [74] S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [75] J. Qian, H. Wang, Z. Li, S. Li, and X. Yan, “Limitations of language models in arithmetic and symbolic induction,” _arXiv preprint arXiv:2208.05051_ , 2022.
  * [76] D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bousquet, Q. Le _et al._ , “Least-to-most prompting enables complex reasoning in large language models,” _arXiv preprint arXiv:2205.10625_ , 2022.
  * [77] L. Wang, W. Xu, Y. Lan, Z. Hu, Y. Lan, R. K.-W. Lee, and E.-P. Lim, “Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models,” _arXiv preprint arXiv:2305.04091_ , 2023.
  * [78] C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in _Text summarization branches out_ , 2004, pp. 74–81.
  * [79] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “Bertscore: Evaluating text generation with bert,” _arXiv preprint arXiv:1904.09675_ , 2019.
  * [80] M. Gao and X. Wan, “Dialsummeval: Revisiting summarization evaluation for dialogues,” in _Proc. of the 2022 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2022, pp. 5693–5709.
  * [81] Y. Zha, Y. Yang, R. Li, and Z. Hu, “Alignscore: Evaluating factual consistency with a unified alignment function,” _arXiv preprint arXiv:2305.16739_ , 2023.
  * [82] S. Samsi, D. Zhao, J. McDonald, B. Li, A. Michaleas, M. Jones, W. Bergeron, J. Kepner, D. Tiwari, and V. Gadepally, “From words to watts: Benchmarking the energy costs of large language model inference,” in _2023 IEEE High Performance Extreme Computing Conference (HPEC)_. IEEE, 2023, pp. 1–9.
  * [83] A. Faiz, S. Kaneda, R. Wang, R. Osi, P. Sharma, F. Chen, and L. Jiang, “Llmcarbon: Modeling the end-to-end carbon footprint of large language models,” _arXiv preprint arXiv:2309.14393_ , 2023.
  * [84] J. McDonald, B. Li, N. Frey, D. Tiwari, V. Gadepally, and S. Samsi, “Great power, great responsibility: Recommendations for reducing energy for training language models,” _arXiv preprint arXiv:2205.09646_ , 2022.
  * [85] Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang _et al._ , “A survey on evaluation of large language models,” _ACM Transactions on Intelligent Systems and Technology_ , 2023.
  * [86] C. Ziems, W. Held, O. Shaikh, J. Chen, Z. Zhang, and D. Yang, “Can large language models transform computational social science?” _Computational Linguistics_ , pp. 1–55, 2024.
  * [87] P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar _et al._ , “Holistic evaluation of language models,” _arXiv preprint arXiv:2211.09110_ , 2022.
  * [88] N. Ding, Y. Qin, G. Yang, F. Wei, Z. Yang, Y. Su, S. Hu, Y. Chen, C.-M. Chan, W. Chen _et al._ , “Parameter-efficient fine-tuning of large-scale pre-trained language models,” _Nature Machine Intelligence_ , vol. 5, no. 3, pp. 220–235, 2023.