# A Robust Pedestrian Detection Approach for Autonomous Vehicles

Bahareh Ghari  Department of Computer Engineering
University of Guilan
Rasht, Iran
<EMAIL_ADDRESS>Ali Tourani  Interdisciplinary Centre for
Security,
Reliability, and Trust (SnT)
University of Luxembourg
Luxembourg, Luxembourg
<EMAIL_ADDRESS>Asadollah Shahbahrami  Department of Computer Engineering
University of Guilan
Rasht, Iran
<EMAIL_ADDRESS>
###### Abstract

Nowadays, utilizing Advanced Driver-Assistance Systems (ADAS) has absorbed a
huge interest as a potential solution for reducing road traffic issues.
Despite recent technological advances in such systems, there are still many
inquiries that need to be overcome. For instance, ADAS requires accurate and
real-time detection of pedestrians in various driving scenarios. To solve the
mentioned problem, this paper aims to fine-tune the YOLOv5s framework for
handling pedestrian detection challenges on the real-world instances of
Caltech pedestrian dataset. We also introduce a developed toolbox for
preparing training and test data and annotations of Caltech pedestrian dataset
into the format recognizable by YOLOv5. Experimental results of utilizing our
approach show that the mean Average Precision (mAP) of our fine-tuned model
for pedestrian detection task is more than $91$ percent when performing at the
highest rate of 70 FPS. Moreover, the experiments on the Caltech pedestrian
dataset samples have verified that our proposed approach is an effective and
accurate method for pedestrian detection and can outperform other existing
methodologies.

###### Index Terms:

pedestrian detection; deep learning; object detection; autonomous vehicles;

##  I Introduction

As a well-known issue, many people get injured or lose their lives in traffic
accidents every day. There are several causes for these accidents, such as
driver tiredness, speeding, and drunk driving. Therefore, autonomous systems
have been utilized in modern vehicles to provide reliable and safe driving and
decease the risk of accidents [1]. These systems use various sensors to
analyze the surroundings in real-time and provide automatic decision-making
while driving. In this regard, pedestrian detection is a vital component in
automotive safety and Advanced Driver-Assistance Systems (ADAS) domains, as it
can guarantee safe vehicle passage on the roads [2].

The process of pedestrian detection in autonomous vehicles can lie under
object detection tasks. Object detection has recently matured and overwhelmed
any other hand-craft or classic detection solutions [3]. Currently, there is a
vast number of object detection algorithms, including Histogram of Oriented
Gradients (HOG) [4], Scale-invariant Feature Transform (SIFT) [5], Speeded up
Robust Features (SURF) [6], Haar feature-based methods [7], and Artificial
Neural Networks [8]. Object detection contributes to the Convolutional Neural
Networks [9] in the computer vision domain, providing robust outcomes compared
to the classic image processing approaches [10]. In this regard, many deep
learning-based applications for object detection are divided into either ”one-
stage” -e.g., You Only Look Once (YOLO) [11]\- or ”two-stage” -e.g., Faster
Region-based CNN (Faster R-CNN) [12]\- methodologies. While in the two-stage
methods a set of samples is used to find the candidate objects, one-stage
techniques predict the locations as a regression problem.

Figure 1: The network architecture of YOLOv5, which includes three modules:
(1) Backbone: CSPDarknet, (2) Neck: PANet, and (3) Head: YOLOv3 (anchor-
based). The input data includes mosaic data augmentation and anchor box
calculation are first input to CSPDarknet for feature extraction, and then fed
to PANet to boost information flow. Finally, the Head consumes features from
the Neck and outputs the results such as bounding boxes and class predictions.
Hereby, SPP stands for Spatial Pyramid Pooling and Concat refers to
concatenation.

Among a wide range of algorithms, YOLO is a one-stage object detector that
utilizes a real-time end-to-end approach to predict bounding boxes and class
labels at once [11]. At the time of writing this paper, many different
versions and variations have been introduced. YOLOv2 has improved the
performance of the first version, including low recall and small object
detection, by using batch normalization on convolutional layers, improved
classifier, and anchor boxes [13]. As the fast approach introduced in YOLOv2
decreased the accuracy, YOLOv3 appeared with a robust backbone, allowing it to
detect features at three scales [14]. Although later versions of YOLO are not
considered official, their developers have changed the architecture of YOLOv3
to improve performance and accuracy. In YOLOv4 [15], the primary focus is on
enhancing the object detection stage, resulting in ten and twelve percent
gains in accuracy and performance, respectively. In $2021$, the fifth version
was introduced as compound-scaled object detection models trained on the COCO
dataset [16]. YOLOv5[17] is implemented with the aid of the Ultralytics
PyTorch framework and Python programming language, making it a super fast
methodology to train.

In this paper, a pedestrian detection application based on YOLOv5 is
introduced. The main contributions of the paper are described below:

  * •

Fine-tuning of YOLOv5 for handling the pedestrian detection challenges,

  * •

Implementing a toolbox for preparing Caltech pedestrian dataset training and
test data and annotations into proper YOLOv5 format.

The rest of the paper is organized as follows: Section II reviews recent
pedestrian detection approaches. In Section III, we discuss our proposed
method and contributions for pedestrian detection in detail. Section IV
presents the experimental results and discussions, and finally, the paper
concludes in Section V.

##  II Related Works

Many recent publications have recommended deep learning-based solutions for
pedestrian detection and tracking. Most of these researches have focused on
the overall architecture of the detector module for a reliable pedestrian
detection and tracking. For instance, Boyuan et al. [18] have proposed a
detection model that combines a new type of Spatial Pyramid Pooling (SPP)
network and K-means clustering algorithm with YOLOv4 model for easier feature
extraction. In [19], DeepParts is proposed, consisting of forty-five fine-
tuned part detectors to tackle the occlusion challenge. Each part detector can
cover the entire body of pedestrians at different positions and scales with
partial occlusion. Similarly, Noh et al. [20] adopted a set of part-based
detectors learned and integrated into single-stage networks to solve the
occlusion problem. For domain adaptation, Zeng et al. [21] have proposed a new
approach to automatically learn domain invariant features and visual patterns
in static video surveillance with no manual labels from the target scene. In
[22], weighted combination layer and depth-wise separable convolution were
combined with YOLO to improve the accuracy and speed of detecting pedestrians
in hazy weather.

Moreover, based on the recent advances in the anchor-free detection networks,
Center and Scale Prediction (CSP) [23] is proposed to detect the center and
scale of pedestrians with a fixed aspect ratio. In [24], Recurrent Rolling
Convolution (RRC) architecture is introduced to gradually aggregate relevant
contextual information among the multi-scale feature maps. Based on the
channel-wise attention mechanism, Zhang et al. [25] added an attention network
to the Faster R-CNN architecture to handle different occlusion patterns in
pedestrian detection. Some other studies addressed the role of data and tuning
in pedestrian detection applications. For instance, the Boosting-like Deep
Learning (BDL) framework [26] is used to prevent the over-fitting problem
while training by reducing the average miss rate of the detector.
Additionally, and as a baseline, Tumas et al. [27] introduced a dataset for
detecting pedestrians in challenging weather conditions.

Unlike the mentioned studies, our method fine-tunes the robust architecture of
YOLOv5 object detector for the pedestrian detection task. The fundamental goal
of this paper is to provide accurate pedestrian detection in different
scenarios, such as profound environmental changes, while keeping the
architecture simple and fast. We also aim to provide detection in real-time so
that our approach can be used in driverless vehicles.

##  III Proposed Method

Due to the splendid features introduced in YOLOv5, including a lower training
time, auto-learning anchor bounding boxes, and mosaic data augmentation, we
have utilized this version in our approach.

###  III-A Architecture

The network structure of YOLOv5 consists of three primary parts, as shown in
Figure1. Accordingly, the backbone is a CNN that aggregates and forms image
features at different granularities. YOLOv5 adopts Cross-Stage Partial
Networks [28] as its backbone to formulate image features. The CSPNet
addresses duplicate gradient problems in deeper CNNs, resulting in fewer
parameters and Floating-point Operations-per-Second (FLOPS) of the model,
improving the inference speed and accuracy and reducing the model size. The
network also adds an SPP block [29] after the CSP to remove the fixed-size
input image constraint. The SPP block is used to compute the feature maps from
the entire image only once, and then pool features in arbitrary regions (sub-
images) to increase the receptive field and generate fixed-length
representations for training the detectors. The Neck in the model is a series
of layers to combine image features to pass them forward for the detection
stage. It employs a Path Aggregation Network (PANet) [30] to boost the
information flow process.

Specifically, the feature pyramid is enhanced with accurate localization
signals in lower layers by bottom-up path augmentation, which shortens the
information path between the lower layers and the top feature. At the same
time, adaptive feature pooling is employed, which links the feature grid and
all feature levels to propagate helpful information in each level directly to
the following proposal sub-networks. The Head of the model is mainly used to
perform the final detection part. It applied feature anchor boxes and
generated final output vectors with class probabilities, objectness scores,
and bounding boxes regression. YOLOv5 implements the same YOLOv3 (anchor-
based) head for prediction. In addition YOLOv5including four models ranging
from small to large according to the memory storage size (parameters): YOLOv5s
(the smallest, which we have used in this paper), YOLOv5m (medium), YOLOvl
(large), and YOLOv5x (extra large, the most prominent). All four models were
trained on the MS COCO [16] training dataset.

###  III-B Dataset

We have utilized the Caltech pedestrian dataset[31] in this work, which
contains more than ten hours of real-world videos taken from a vehicle driving
through regular traffic in an urban environment. The resolution of the videos
is $640\times 480$ and the total number of frames exceeds $250,000$. Also,
challenges in the Caltech pedestrian dataset such as variation in lighting
conditions, pedestrian size, and occlusion, making this dataset a proper
choice for the training of pedestrian detectors. This dataset contains more
than $350,000$ bounding boxes and $2,300$ unique pedestrians annotated.
Figure2 shows some instances of the dataset.

###  III-C Data preparation for YOLOv5

As YOLO does not recognize the Caltech’s training data and annotations, we
needed to implement a conversion toolbox for it. The mentioned toolbox
contains different tools to prepare proper feed for YOLO architecture and is
publicly accessible in a GitHub repository
111https://github.com/GuilanITS/Caltech-Pedestrian-YOLO. Using our toolbox, we
processed Caltech pedestrian datasettraining .seq files and generated squared
$640\times 640$ .png images. Additionally, the .vbb annotation files are
converted into .txt files according to the YOLOv5 standard.

###  III-D Training Process

We picked the most miniature version of the YOLOv5 framework for the
experiment, titled YOLOv5s. After fine-tuning the architecture of YOLOv5s
based on the application requirements, the network has been trained on $3,000$
instances of the dataset in $600$ epochs with a learning rate of $0.01$, a
momentum of $0.937$ and a batch size of eight. The number of images in the
validation and test sets was $280$ and $1006$, respectively. The total number
of parameters in the network was $7,012,822$, shaped into $213$ layers. Pre-
trained weights on COCO dataset are used to initialize the detection model.
Among SGD, Adam and AdamW optimization functions implemented in YOLOv5, we
chose the Adam algorithm for the optimization of parameters. With this
settings, $600$ epochs completed in $10.902$ hours. Moreover, YOLOv5 can
easily trade-off between speed and accuracy by changing the size of input
image, without retraining. For instance, at the resolution of $1280$, YOLOv5s
runs in $4.9ms$ at $83.3$ mAP. So, larger image sizes usually lead to better
results, but obviously, take longer to process. It should also be noted that
we used only person class in the dataset and ignored the people class,
enabling us to detect individuals correctly even if they are in a crowd.

##  IV Evaluation

Figure 2: Some instances of the Caltech pedestrian dataset, the baseline for
training and testing in this paper.

The machine for conducting experiments was equipped with an NVIDIA GeForce
RTX3090 Graphics Processing Unit (GPU) with $24,576$ MegaBytes of memory and
an Intel(R) Xeon(R) Gold 6248R processor. The codes were implemented in Python
v3, and we used CUDA v.11.6.55 computing platform and PyTorch framework.

Fig. 5 demonstrates the pedestrian detection results of the proposed approach
in various scenarios. We can see that even people walking in the throngs have
been detected and tracked correctly using our fine-tuned network.

Figure 3: Fluctuations of key indicators while training in different epochs.
The top five curves are related to network training and the bottom five curves
are related to network validation. In all graphs, the x-axis indicates the
epochs. The y-axis refers to the loss or accuracy values, based on the graph.

###  IV-A Evaluation Metrics

We have employed precision, recall, Average Precision (AP), and mAP metrics to
evaluate our proposed method. Precision refers to the ratio of all correctly
predicted instances (i.e., pedestrians) among all predictions. Recall is used
to indicate the number of correctly classified samples in the total number of
ground-truth data. In other words, it shows how many sample was the model able
to detect out of the total number of pedestrians in the input image. The
mentioned metrics are calculated using the equations 1 and 2, respectively.

$precision=\frac{TP}{TP+FP}$ (1)

$recall=\frac{TP}{TP+FN}$ (2)

Where TP is True-Positive, FP refers to False-Positive, and FN refers to
False-Negative. A True-Positive is an outcome where the model correctly
predicts the positive class. A False-Positive is an outcome where the model
incorrectly predicts the positive class. And similarly, a False-Negative is an
outcome where the model incorrectly predicts the negative class. The value of
precision and recall depends on how many True-Positives were detected by the
model. A precision-recall curve plots the value of precision against recall
for different confidence threshold values. To simplify using precision and
recall measure, we use F1-score, which shapes a single metric based on their
harmonic mean. Equation 3 shows calculation of F1-score, where Pr and Rc refer
to precision and recall, respectively.

$F1-score=2\times\frac{(Pr\times Rc)}{Pr+Rc}$ (3)

AP represents the area under the precision-recall curve and summarizes the
precision-recall curve to one scalar value. The range for AP is between 0 to
1. The metric AP can be calculated from Equation 4, where n is the number of
thresholds.

$AP=\sum_{k=0}^{k=n-1}(Rc(k)-Rc(k+1))\times Pr(k)$ (4)

Additionally, mAP is another metric to evaluate object detection models
indicating the average AP for each category. The formula for calculating mAP
is shown in Equation 5, in which k refers to a specific class and n refers to
the total number of classes.

$mAP=\frac{1}{n}\sum_{k=1}^{k=n}AP_{k}$ (5)

###  IV-B Evaluation Results

Figure 4: The Precision-Recall curve of the proposed method. Figure 5: Detection results using the fine-tuned YOLOv5s model proposed in this paper. TABLE I: Experimental results of different models. Model | F1-Score |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>: .95 | Frame-rate (fps) | Inference (ms)
---|---|---|---|---|---
YOLOv5s (proposed) | 0.885 | 0.918 | 0.663 | 69.4 | 14.4
Improved YOLOv4 [18] | 0.80 | 0.847 | - | 36.4 | -

Figure3 demonstrates ten curves of changes in key indicators according while
training during the defined epochs. The top five curves are related to network
training and the bottom five curves are related to network validation. Here,
YOLOv5’s loss function is composed of three parts: $box_{l}oss$ is a bounding
box regression loss that measures how tight the predicted bounding boxes are
w.r.t. to the ground truth object (loss criteria in use is CIoU()).
$obj_{l}oss$ is the objectness loss that represent the confidence of object
presence (Binary Cross Entropy()). And finally, $cls_{l}oss$ is a
classification loss that measures the correctness of the classification of
each predicted bounding box. This loss is usually called Cross Entropy loss().
Since our dataset has one class only, the classification error is constantly
zero. AP is the average over multiple Intersection over Unions, and
mAP@[.5:.95] corresponds to the average AP for IoU from 0.5 to 0.95 with a
step size of 0.05.

In this regard, the precision-recall curve is shown in Figure4. As mentioned
above, the curve is obtained by plotting the model’s precision and recall
values as a function of the model’s confidence score threshold. The smaller
the confidence threshold, the higher the recall. On the other hand, the higher
the confidence threshold, higher the precision. Therefore, there exists a
trade-off between precision and recall based on the value of the confidence
threshold. With the precision-recall curve, we can see that YOLOv5 model has
the high precision and recall values at the pedestrian detection task.

Table I presents the accuracy of the proposed pedestrian detection system.
Accordingly, the system could achieve $0.935$ in precision and $0.84$ in the
recall. Also, the mean average precision at $50$% IoU (cut-off value of 5) was
around $91.8$%, which indicates something.

Moreover, to assess the system’s performance, we noticed that the detection
speed of the YOLOv5s architecture is $14.4ms$, which indicates a real-time
performance. It can also work properly for pedestrian detection when the frame
rate is $~{}69$ fps. The rate of frame per second can increases to $277$ fps
at the test when a batch size of $8$ is considered.

###  IV-C Discussion

This paper uses YOLOv5s as the pedestrian detector, which can meet the
requirements of real-time detection. It can be seen in Table I that, compared
with the other pedestrian detectors, the fine-tuned YOLOv5s model has obvious
advantages in terms of accuracy (F1-score) and frame rate. As a result of
comparing, the mAP of YOLOv5s model is $7$% higher than YOLOv4 [18]. Also, the
frame rate of YOLOv5s is $69.4$ fps, which is approximately twice the value of
YOLOv4 ($36.4$ fps).

For further work, we plan to evaluate our system in an end-to-end application
with real-world data and improve the object detection module using other
robust approaches, such as transformers.

##  V Conclusions

Pedestrians are among the paramount objects that autonomous vehicles must
detect. Considering the current challenges in pedestrian detection and the
prominent features introduced in YOLOv5, including a lower training time,
auto-learning anchor bounding boxes, and mosaic data augmentation, this paper
presented a pedestrian detection application based on YOLOv5s. The framework
is written in the Ultralytics PyTorch framework, making it very fast to train.
In this approach, we have utilized the Caltech pedestrian datasetand
implemented a conversion toolbox to prepare proper feed for the YOLO
architecture. The mentioned toolbox converted images and annotations of the
Caltech pedestrian datasetto items recognizable by YOLOv5. According to the
experiments, the mAP value of the proposed system is 91.8%, while the frame-
rate is 69.4 frames-per-second.

## Acknowledgment

The training and evaluation processes of the proposed work could not been done
without the supports of the Amirkabir University of Technology for providing
the Simorgh Supercomputer under Contract No. ISI-DCE-DOD-Cloud-900808-1700.

## References

  * [1] N. Adnan, S. M. Nordin, M. A. bin Bahruddin, and M. Ali, “How trust can drive forward the user acceptance to the technology? in-vehicle technology for autonomous vehicle,” _Transportation research part A: policy and practice_ , vol. 118, pp. 819–836, 2018.
  * [2] S. Zhang, R. Benenson, M. Omran, J. Hosang, and B. Schiele, “Towards reaching human performance in pedestrian detection,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 40, no. 4, pp. 973–986, 2017\.
  * [3] M. Hnewa and H. Radha, “Object detection under rainy conditions for autonomous vehicles: A review of state-of-the-art and emerging techniques,” _IEEE Signal Processing Magazine_ , vol. 38, no. 1, pp. 53–67, 2020.
  * [4] N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in _2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05)_ , vol. 1. Ieee, 2005, pp. 886–893.
  * [5] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” _International journal of computer vision_ , vol. 60, no. 2, pp. 91–110, 2004\.
  * [6] H. Bay, T. Tuytelaars, and L. V. Gool, “Surf: Speeded up robust features,” in _European conference on computer vision_. Springer, 2006, pp. 404–417.
  * [7] P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” in _Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001_ , vol. 1. Ieee, 2001, pp. I–I.
  * [8] I. N. Da Silva, D. H. Spatti, R. A. Flauzino, L. H. B. Liboni, and S. F. dos Reis Alves, “Artificial neural networks,” _Cham: Springer International Publishing_ , vol. 39, 2017.
  * [9] J. Gu, Z. Wang, J. Kuen, L. Ma, A. Shahroudy, B. Shuai, T. Liu, X. Wang, G. Wang, J. Cai _et al._ , “Recent advances in convolutional neural networks,” _Pattern recognition_ , vol. 77, pp. 354–377, 2018.
  * [10] L. Jiao, F. Zhang, F. Liu, S. Yang, L. Li, Z. Feng, and R. Qu, “A survey of deep learning-based object detection,” _IEEE access_ , vol. 7, pp. 128 837–128 868, 2019.
  * [11] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 779–788.
  * [12] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” _Advances in neural information processing systems_ , vol. 28, 2015.
  * [13] J. Redmon and A. Farhadi, “Yolo9000: better, faster, stronger,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 7263–7271.
  * [14] ——, “Yolov3: An incremental improvement,” _arXiv preprint arXiv:1804.02767_ , 2018.
  * [15] A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, “Yolov4: Optimal speed and accuracy of object detection,” _arXiv preprint arXiv:2004.10934_ , 2020.
  * [16] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [17] G. Jocher, A. Stoken, J. Borovec, A. Chaurasia, L. Changyu, A. Laughing, A. Hogan, J. Hajek, L. Diaconu, Y. Marc _et al._ , “ultralytics/yolov5: v5. 0-yolov5-p6 1280 models aws supervise. ly and youtube integrations,” _Zenodo_ , vol. 11, 2021.
  * [18] W. Boyuan and W. Muqing, “Study on pedestrian detection based on an improved yolov4 algorithm,” in _2020 IEEE 6th International Conference on Computer and Communications (ICCC)_. IEEE, 2020, pp. 1198–1202.
  * [19] Y. Tian, P. Luo, X. Wang, and X. Tang, “Deep learning strong parts for pedestrian detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1904–1912.
  * [20] J. Noh, S. Lee, B. Kim, and G. Kim, “Improving occlusion and hard negative handling for single-stage pedestrian detectors,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 966–974.
  * [21] X. Zeng, W. Ouyang, M. Wang, and X. Wang, “Deep learning of scene-specific classifier for pedestrian detection,” in _European Conference on Computer Vision_. Springer, 2014, pp. 472–487.
  * [22] G. Li, Y. Yang, and X. Qu, “Deep learning approaches on pedestrian detection in hazy weather,” _IEEE Transactions on Industrial Electronics_ , vol. 67, no. 10, pp. 8889–8899, 2019.
  * [23] W. Liu, S. Liao, W. Ren, W. Hu, and Y. Yu, “High-level semantic feature detection: A new perspective for pedestrian detection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 5187–5196.
  * [24] J. Ren, X. Chen, J. Liu, W. Sun, J. Pang, Q. Yan, Y.-W. Tai, and L. Xu, “Accurate single stage detector using recurrent rolling convolution,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 5420–5428.
  * [25] S. Zhang, J. Yang, and B. Schiele, “Occluded pedestrian detection through guided attention in cnns,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2018, pp. 6995–7003.
  * [26] L. Wang and B. Zhang, “Boosting-like deep learning for pedestrian detection,” _arXiv preprint arXiv:1505.06800_ , 2015.
  * [27] P. Tumas, A. Nowosielski, and A. Serackis, “Pedestrian detection in severe weather conditions,” _IEEE Access_ , vol. 8, pp. 62 775–62 784, 2020.
  * [28] C.-Y. Wang, H.-Y. M. Liao, Y.-H. Wu, P.-Y. Chen, J.-W. Hsieh, and I.-H. Yeh, “Cspnet: A new backbone that can enhance learning capability of cnn,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , 2020, pp. 390–391.
  * [29] K. He, X. Zhang, S. Ren, and J. Sun, “Spatial pyramid pooling in deep convolutional networks for visual recognition,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 37, no. 9, pp. 1904–1916, 2015\.
  * [30] S. Liu, L. Qi, H. Qin, J. Shi, and J. Jia, “Path aggregation network for instance segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8759–8768.
  * [31] P. Dollar, C. Wojek, B. Schiele, and P. Perona, “Pedestrian detection: An evaluation of the state of the art,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 34, no. 4, pp. 743–761, 2011.

  *[ADAS]: Advanced Driver-Assistance Systems
  *[mAP]: mean Average Precision
  *[HOG]: Histogram of Oriented Gradients
  *[SIFT]: Scale-invariant Feature Transform
  *[SURF]: Speeded up Robust Features
  *[YOLO]: You Only Look Once
  *[Faster R-CNN]: Faster Region-based CNN
  *[SPP]: Spatial Pyramid Pooling
  *[CSP]: Center and Scale Prediction
  *[RRC]: Recurrent Rolling Convolution
  *[BDL]: Boosting-like Deep Learning
  *[CNN]: Convolutional Neural Network
  *[CNNs]: Convolutional Neural Network
  *[FLOPS]: Floating-point Operations-per-Second
  *[PANet]: Path Aggregation Network
  *[GPU]: Graphics Processing Unit
  *[AP]: Average Precision
  *[IoU]: Intersection over Union


#  * Towards Fast Multilingual LLM Inference:
Speculative Decoding and Specialized Drafters

Euiin Yi1 Taehyeon Kim1∗ Hongseok Jeung2 Du-Seong Chang2 Se-Young Yun1
1KAIST AI 2KT
{euiin_mercyii, potter32<EMAIL_ADDRESS>
https://github.com/Kthyeon/Multilingual-SpecBench Equal contribution.

###### Abstract

Large language models (LLMs) have revolutionized natural language processing
and broadened their applicability across diverse commercial applications.
However, the deployment of these models is constrained by high inference time
in multilingual settings. To mitigate this challenge, this paper explores a
training recipe of an assistant model in speculative decoding, which are
leveraged to draft and-then its future tokens are verified by the target LLM.
We show that language-specific draft models, optimized through a targeted
pretrain-and-finetune strategy, substantially brings a speedup of inference
time compared to the previous methods. We validate these models across various
languages in inference time, out-of-domain speedup, and GPT-4o evaluation.

* Towards Fast Multilingual LLM Inference:
Speculative Decoding and Specialized Drafters

Euiin Yi1††thanks: Equal contribution. Taehyeon Kim1∗ Hongseok Jeung2 Du-Seong
Chang2 Se-Young Yun1 1KAIST AI 2KT {euiin_mercyii, potter32,
<EMAIL_ADDRESS>https://github.com/Kthyeon/Multilingual-SpecBench

##  1 Introduction

Large language models (LLMs) such as Gemini Team et al. (2023), GPT Achiam et
al. (2023), and Llama Touvron et al. (2023a) have remarkable success across
various natural language processing tasks. Their deployment in commercial
settings has expanded to include applications such as coding assistance,
writing support, conversational interfaces, and tools for search Reid et al.
(2024). Despite their potential, the practical deployment of these models is
often limited by prohibitively high inference time, particularly in
multilingual contexts Ahia et al. (2023). For example, character-level and
byte-level models exhibit encoding length discrepancies exceeding fourfold for
certain language pairs, resulting in significant disparities in cost and
inference time available to different language communities Petrov et al.
(2024). These challenges present substantial huddles to scalable and cost-
efficient applications of LLMs.

Speculative decoding, utilizing assistant models, has emerged as a promising
strategy to improve LLM inference efficiency Leviathan et al. (2023); Chen et
al. (2023); Xia et al. (2024), inspired by speculative execution Burton
(1985). This method drafts potential future tokens by leveraging a smaller
model for the initial predictions. In parallel, these tokens are verified by
the target LLM, ensuring only outputs aligned with the target LLM’s
predictions are accepted. Recent efforts are focused on aligning these initial
predictions with the target LLM’s outputs Liu et al. (2023); Zhou et al.
(2023). This involves advancing the training methods and modifying the
architectural design of drafters Miao et al. (2024); Li et al. (2024).

Figure 1:  Speedup ratio222Evaluated on a single RTX3090 GPU with a batch size
1. relative to the standard autoregressive greedy decoding on various
multilingual datasets. Target model is Vicuna 7B v1.3 and the drafter is
Vicuna 68M. Speculative greedy sampling is implemented with the drafter by
Yang et al. (2024) (green) and our specialized drafter (pretrain-and-finetune)
(red).

Although speculative decoding has garnered considerable hype recently, the
adaptation of this approach to multilingual scenarios common in real-world
applications remains largely unexplored. Prevailing methods Cai et al. (2024);
Li et al. (2024); Yang et al. (2024) use small drafters simply trained on
datasets such as ShareGPT ShareGPT (2023) which is often used for instruction
tuning of LLMs to learn a pattern of target LLM’s language modeling. However,
our investigations reveal that such approach are insufficient for multilingual
translation (footnote 2). This research also raises concerns regarding the
capacity of such small drafters with simple tuning to comprehend the nuances
of all target languages, thus questioning the feasibility of employing such
models for universal speculative decoding. This paper aims to shed light upon
the behaviors of drafters in speculative decoding within multilingual tasks
and to explore their efficacy. Our contributions are as follows:

Figure 2: Speedup comparison of various speculative decoding methods on WMT16
De-En dataset Bojar et al. (2016) with greedy settings ($T$=$0.0$) across
various hardwares. Target model is Vicuna-7B.

  * •

We demonstrate that the strategy of pretrain-and-finetune significantly
improves the alignment of drafter models, achieving the highest speedup ratio
among the baselines (Figure 2).

  * •

We find that the speedup ratio increases as the number of tokens specific to
the target task used in training increases. This speedup is logarithmically
proportional to the scale of token count in drafter training.

  * •

In multilingual translation, we observe that input languages consistent with
the training set result in notable speedup, whereas outputs aligned with the
training domain do not necessarily lead to improved performance. Additionally,
our results are corroborated by GPT-4o judgment scores and qualitative
analyses.

##  2 Method

###  2.1 Preliminaries: speculative decoding

Speculative decoding employs a draft-verify-accept paradigm for fast
inference. This method leverages a simpler assistant model ($M_{p}$) to
predict easy tokens, thereby addressing memory bandwidth constraints in LLM
inference Shazeer (2019):

  1. 1.

Draft: An assistant model $M_{p}$, which is less computationally intensive
than the target LLM $M_{q}$, drafts the future tokens
$\\{x_{t_{1}},\ldots,x_{t_{K}}\\}$ based on the input sequence
$x_{1},\ldots,x_{t}$.

  2. 2.

Verify: The target LLM $M_{q}$ assesses each token $x_{t_{i}}$ regarding
whether it is aligned with its own predictions:
$p_{i}=M_{p}(x_{t_{i}}|x_{1},\ldots,x_{t},x_{t_{1}},\ldots,x_{t_{i-1}})$,
$q_{i}=M_{q}(x_{t_{i}}|x_{1},\ldots,x_{t},x_{t_{1}},\ldots,x_{t_{i-1}})$.

  3. 3.

Accept: Tokens meeting the validation criteria (e.g., rejection sampling)
aligned with $M_{q}$’s outputs are retained. Tokens failing verification are
either discarded or corrected, and the draft-verify cycle is repeated.

In this paper, the verification process employs rejection sampling Leviathan
et al. (2023); Li et al. (2024) when the temperature parameter is above zero
to accept only tokens that closely match $M_{q}$’s predictions. For greedy
decoding with a temperature of zero, tokens are accepted if they are identical
to $M_{q}$’s predictions.

Figure 3:  Speedup444Evaluated on a single RTX3090 GPU with a batch size 1.
comparison across categories containing multi-turn conversation (MT-Bench)
Zheng et al. (2024), math reasoning (GSM8K) Cobbe et al. (2021), and
translation (WMT16 De-En). Target model is Vicuna-7B with speculative greedy
sampling.

###  2.2 Motivation

Our evaluation of various speculative models, including SpS Chen et al.
(2023), Medusa Cai et al. (2024), Eagle Li et al. (2024), as shown in footnote
4, demonstrates that speedup ratios significantly differ by task domain. While
these models excel in English tasks such as multi-turn conversations and
mathematical reasoning, where they achieve notable speed improvements, they
underperform in translation tasks (red dotted box in footnote 4). This result
confirms that the effectiveness of these models is not universal but may
highly language-specific. The consistent underperformance in translation tasks
highlights a key weakness and drives our study towards developing specialized
drafters.

###  2.3 Training specialized assistant models

At the core of our approach is the recognition that smaller models, due to
their inherent limited capacity, struggle to capture the diverse token
distributions across languages. To address this challenge, we present
specialized drafter models tailored to each language. Our strategy consists
of:

  1. 1.

Pretrain (P): Assistant models are pretrained on a part of C4 Raffel et al.
(2019) and ShareGPT dataset ShareGPT (2023) for language modeling.

  2. 2.

Finetune (F): The models are finetuned on the target lingual task with
instructions to refine their responses to non-English inputs.

While the practices of pretraining and finetuning are well-established
paradigms in language model training, applying these steps to drafter models
represents a novel adaptation within the field. Traditionally, assistnat
models have been trained from scratch with little strategic rationale or clear
justification for dataset selection.

Figure 4 shows that the pretrain-and-finetune strategy significantly boosts
the speedup ratio as the number of training tokens increases. Our ‘P-F’
approach outperforms models that are only finetuned (F), and even surpasses
the speedup rates by Yang et al. (2024), which stood at 1.12.

Figure 4: Speedup with speculative greedy sampling on the WMT16 De-En dataset
as the training token for finetune (F) count varies, displayed on a
logarithmic x-axis. ‘P-F’ represents our strategy and ‘F’ involves training
solely on De-En without pretrain step (P).

#### Dataset with self-distillation

The training dataset for our assistant models is generated through self-
distillation from the target LLM, ensuring alignment with its outputs Kim and
Rush (2016); Zhou et al. (2023); Cai et al. (2024). To capture the full range
of the target’s output variability, we generate multiple responses at a range
of temperatures—{0.0, 0.3, 0.7, 1.0}.

##  3 Experiment

###  3.1 Experimental setup

#### Models

We utilize Vicuna 7B Chiang et al. (2023), Gemma-Instruct 7B Team et al.
(2024), and Llama2-chat Touvron et al. (2023b) as target LLMs. The drafter
models employed include Vicuna 68M Yang et al. (2024), a custom Gemma 250M
drafter and Llama 68M Miao et al. (2024). Training configurations are outlined
in Appendix F.

#### Number of drafts

For speculative sampling (SpS), we use a single draft candidate Chen et al.
(2023). In contrast, Medusa and Eagle models are evaluated using multiple
drafts via tree-attention mechanism by following their original settings.

#### Training and evaluation

Training datasets for each target model are generated via self-distillation
and comprise five datasets: German (De)→English (En), French (Fr)→En, Russian
(Ru)→En, Japanese (Ja)→En and Chinese (Zh)→En, each with 4 million (M)
conversations ($\sim$ 1.3 billion (B) tokens) sourced from WMT14 Fr-En Bojar
et al. (2014), WMT16 De-En, and Ru-En Bojar et al. (2016), and JParaCrawl-v3.0
Morishita et al. (2022). Evaluations are conducted using a single NVIDIA 3090
GPU, under both greedy settings ($T$=$0.0$) and for diversity at $T$=$1.0$
with three different seeds. The details are in Appendix F.

Figure 5: Speedup with speculative greedy sampling on various out-of-domain
dataset as the drafters for ‘Ours (P-F)’ and ‘F’ are trained on WMT16 De-En
dataset. Table 1: Speedup comparison of different methods for Vicuna 7B v1.3.
Results are provided for $T$=$0.0$ and $T$=$1.0$ across various translation
tasks. For our approach, each drafter is finetuned with the corresponding
dataset.

Method | T=0.0 | T=1.0
---|---|---
De→En | Fr→En | Ru→En | Ja→En | Zh→En | Avg | De→En | Fr→En | Ru→En | Ja→En | Zh→En | Avg
Sps - Yang et al. (2024) | 1.19±0.06 | 1.14±0.05 | 1.11±0.04 | 1.23±0.03 | 1.22±0.00 | 1.18±0.04 | 1.07±0.03 | 1.06±0.02 | 1.04±0.01 | 1.15±0.02 | 1.11±0.02 | 1.09±0.02
Lookahead Fu et al. (2024) | 1.03±0.01 | 1.01±0.02 | 0.98±0.01 | 1.00±0.01 | 0.96±0.00 | 1.00±0.01 | 1.03±0.03 | 1.04±0.03 | 0.99±0.00 | 0.98±0.05 | 0.98±0.00 | 1.01±0.02
PLD Saxena (2023) | 1.13±0.06 | 1.05±0.04 | 1.03±0.00 | 1.09±0.05 | 0.99±0.07 | 1.06±0.05 | - | - | - | - | - | -
Medusa Cai et al. (2024) | 1.58±0.05 | 1.57±0.01 | 1.52±0.01 | 1.55±0.01 | 1.43±0.00 | 1.53±0.02 | 1.61±0.03 | 1.69±0.01 | 1.62±0.00 | 1.72±0.01 | 1.60±0.01 | 1.65±0.01
Eagle Li et al. (2024) | 1.90±0.05 | 1.88±0.00 | 1.67±0.05 | 1.88±0.01 | 1.75±0.01 | 1.81±0.02 | 1.57±0.00 | 1.61±0.01 | 1.45±0.02 | 1.63±0.01 | 1.51±0.03 | 1.55±0.01
Sps - pretrain-and-finetune (Ours) | 2.42±0.02 | 2.05±0.04 | 1.74±0.02 | 1.71±0.01 | 1.52±0.01 | 1.89±0.02 | 1.99±0.01 | 1.86±0.03 | 1.58±0.00 | 1.67±0.01 | 1.44±0.00 | 1.71±0.01

###  3.2 Main result

#### Overall

Table 1 shows that our specialized drafter (pretrain-and-finetune) for
targeted languages demonstrates superior performance across multiple
translation tasks, recording the highest speedup in both deterministic
($T$=$0.0$) and diverse ($T$=$1.0$) settings. At $T$=$0.0$, our model
outperforms all competitors with an average speedup ratio of 1.89. Similarly,
at $T$=$1.0$, it maintains robust performance with an overall speedup ratio of
1.71.

Table 2: Examples of speculative decoding on WMT16 De-En dataset. Black
indicates standard decoded output and magenta indicates accepted draft tokens.
Input
---
Translate German to English: So wie er gestartet ist , wird es nicht lange
dauern , bis er auf der „ Pferd des Jahres “ -Schau ist – und ich bin mir
sicher , dass er gut abschneiden wird.
SpS with a drafter by Yang et al. (2024)
As he started, it won’t take long until he’s on the "Horse of the Year" show,
and I’m sure he’ll do well.
Eagle Li et al. (2024)
As he started, it won’t take long until he’s on the "Horse of the Year" show,
and I’m sure he’ll do well.
SpS with our specialized drafter (pretrain-and-finetune)
As he started, it won’t take long until he’s on the "Horse of the Year" show,
and I’m sure he’ll do well.

#### Speedup on out-of-domain translation tasks

As Figure 5 shows, our analysis reveals variability applying the drafter,
trained on the WMT16 De-En dataset, across diverse translation pairs. Speedups
are consistently higher when translating from German to other languages,
highlighting the importance of input domain consistency with the training
data. Conversely, translations involving non-German languages with English and
English-German pairings show limited gains. This result emphasizes that
effective speculation relies more on matching the translation task’s input
domain with the training data than on the output domain.

#### Qualitative analysis on responses

Table 2 provides examples of speculative inference on the WMT16 De-En dataset.
Both Eagle and our method incorporate a significant number of accepted tokens
from drafts. However, our model achieves this with $\sim 75\%$ fewer
parameters, leading to reduced latency and faster inference time (Table 1).
Speculation typically takes place at critical junctions of the sentence such
as transitions between clauses and phrases.

Figure 6: GPT-4o judgment scores following the Zheng et al. (2024) on various
multilingual translation dataset. The score is evaluated random sampling with
$T$=$1.0$.  Table 3: Ablations with speedup as the training stages continue on
WMT19 Zh→En.

Target LLM - Drafter | P | \+ F
---|---|---
Gemma-Instruct 7B - Gemma 250M | 0.92±0.01 | 1.04±0.02
Llama2-chat 7B - Llama 68M | 1.47±0.00 | 1.95±0.01

#### GPT-4o judgment analysis

Figure 6 depicts the GPT-4o judgment scores Zheng et al. (2024) generated
using a temperature of 1.0. Our drafter closely matches the target Vicuna LLM
across multiple datasets. The setup and further results are in Appendix F and
Appendix G.

#### Ablation study

Table 3 presents the ablation results, illustrating the progressive impact of
the pretrain-and-finetune approach on the performance of Gemma and Llama2-chat
models.

##  4 Conclusion

This paper has demonstrated that the pretrain-and-finetune strategy for
training drafters significantly enhances speedup ratio relative to standard
autoregressive decoding in multilingual translation tasks. This gain grows
logarithmically with the increase in the number of training tokens. Supported
by qualitative analysis, out-of-domain analysis, and GPT-4o evaluation, this
strategy substantially outperforms the state-of-the-art methods in various
language pairs. Our study uncovers approaches to maximize the benefits from
drafter models, thereby setting a new benchmark in this area.

## Limitations

Despite the improvement, our approach, requiring separate drafters for each
language, introduces complexities in deployment, especially in multilingual
settings. For instance, in environments where multiple languages are
frequently interchanged, such as multinational corporations or global customer
service platforms, the lack of an automated drafter selection system could
hinder operational efficiency. Currently, our study focuses on independent
drafters; however, examining systems that utilize interdependent models,
similar to Eagle and Medusa, might offer insights into more interesting
strategies. Additionally, while our findings are promising for translation
tasks, expanding this methodology to other multilingual applications, like
real-time multilingual generation or summarization, is essential to understand
its broader applicability and uncover additional constraints.

This work primarily presents no direct ethical concerns. Further discussions
are detailed in Appendix B and Appendix H.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Ahia et al. (2023) Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R Mortensen, Noah A Smith, and Yulia Tsvetkov. 2023.  Do all languages cost the same? tokenization in the era of commercial language models.  _arXiv preprint arXiv:2305.13707_.
  * Bae et al. (2023) Sangmin Bae, Jongwoo Ko, Hwanjun Song, and Se-Young Yun. 2023.  Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding.  _arXiv preprint arXiv:2310.05424_.
  * Bhendawade et al. (2024) Nikhil Bhendawade, Irina Belousova, Qichen Fu, Henry Mason, Mohammad Rastegari, and Mahyar Najibi. 2024.  Speculative streaming: Fast llm inference without auxiliary models.  _arXiv preprint arXiv:2402.11131_.
  * Bojar et al. (2014) Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al. 2014.  Findings of the 2014 workshop on statistical machine translation.  In _Proceedings of the ninth workshop on statistical machine translation_ , pages 12–58.
  * Bojar et al. (2016) Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al. 2016.  Findings of the 2016 conference on machine translation (wmt16).  In _First conference on machine translation_ , pages 131–198. Association for Computational Linguistics.
  * Burton (1985) F Warren Burton. 1985.  Speculative computation, parallelism, and functional programming.  _IEEE Transactions on Computers_ , 100(12):1190–1193.
  * Cai et al. (2024) Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. 2024.  Medusa: Simple llm inference acceleration framework with multiple decoding heads.  _arXiv preprint arXiv:2401.10774_.
  * Chen et al. (2023) Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023.  Accelerating large language model decoding with speculative sampling.  _arXiv preprint arXiv:2302.01318_.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_.
  * Elhoushi et al. (2024) Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. 2024.  Layer skip: Enabling early exit inference and self-speculative decoding.  _arXiv preprint arXiv:2404.16710_.
  * Fan et al. (2021) Yimin Fan, Yaobo Liang, Alexandre Muzio, Hany Hassan, Houqiang Li, Ming Zhou, and Nan Duan. 2021.  Discovering representation sprachbund for multilingual pre-training.  _arXiv preprint arXiv:2109.00271_.
  * Fu et al. (2024) Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2024.  Break the sequential dependency of llm inference using lookahead decoding.  _arXiv preprint arXiv:2402.02057_.
  * Gloeckle et al. (2024) Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. 2024.  Better & faster large language models via multi-token prediction.  _arXiv preprint arXiv:2404.19737_.
  * Gu et al. (2023) Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2023.  Minillm: Knowledge distillation of large language models.  In _The Twelfth International Conference on Learning Representations_.
  * Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel. 2016.  Gaussian error linear units (gelus).  _arXiv preprint arXiv:1606.08415_.
  * Kim et al. (2024) Taehyeon Kim, Ananda Theertha Suresh, Kishore Papineni, Michael Riley, Sanjiv Kumar, and Adrian Benton. 2024.  Exploring and improving drafts in blockwise parallel decoding.  _Preprint_ , arXiv:2404.09221.
  * Kim and Rush (2016) Yoon Kim and Alexander M Rush. 2016.  Sequence-level knowledge distillation.  _arXiv preprint arXiv:1606.07947_.
  * Ko et al. (2024) Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. 2024.  Distillm: Towards streamlined distillation for large language models.  _arXiv preprint arXiv:2402.03898_.
  * Leviathan et al. (2023) Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023.  Fast inference from transformers via speculative decoding.  In _International Conference on Machine Learning_ , pages 19274–19286. PMLR.
  * Li et al. (2024) Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024.  Eagle: Speculative sampling requires rethinking feature uncertainty.  _arXiv preprint arXiv:2401.15077_.
  * Li et al. (2020) Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gonzalez. 2020.  Train big, then compress: Rethinking model size for efficient training and inference of transformers.  In _International Conference on machine learning_ , pages 5958–5968. PMLR.
  * Liu et al. (2023) Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Ion Stoica, Zhijie Deng, Alvin Cheung, and Hao Zhang. 2023.  Online speculative decoding.  _arXiv preprint arXiv:2310.07177_.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * Miao et al. (2024) Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. 2024.  Specinfer: Accelerating large language model serving with tree-based speculative inference and verification.  In _Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3_ , pages 932–949.
  * Morishita et al. (2022) Makoto Morishita, Katsuki Chousa, Jun Suzuki, and Masaaki Nagata. 2022.  Jparacrawl v3. 0: A large-scale english-japanese parallel corpus.  _arXiv preprint arXiv:2202.12607_.
  * OpenAI (2024) OpenAI. 2024.  Hello GPT-4o.  Accessed: Insert the current date.
  * Pan (2023) Jiayi Pan. 2023.  Tiny-vicuna 1b.  https://huggingface.co/Jiayi-Pan/Tiny-Vicuna-1B.
  * Patterson (2004) David A Patterson. 2004.  Latency lags bandwith.  _Communications of the ACM_ , 47(10):71–75.
  * Petrov et al. (2024) Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. 2024.  Language model tokenizers introduce unfairness between languages.  _Advances in Neural Information Processing Systems_ , 36.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv e-prints_.
  * Reid et al. (2024) Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024.  Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.  _arXiv preprint arXiv:2403.05530_.
  * Saxena (2023) Apoorv Saxena. 2023.  Prompt lookup decoding.
  * Schuster et al. (2022) Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. 2022.  Confident adaptive language modeling.  _Advances in Neural Information Processing Systems_ , 35:17456–17472.
  * ShareGPT (2023) ShareGPT. 2023.  Sharegpt: Vicuna unfiltered dataset.  https://huggingface.co/datasets/Aeala/ShareGPT_Vicuna_unfiltered.  Accessed: 2024.
  * Shazeer (2019) Noam Shazeer. 2019.  Fast transformer decoding: One write-head is all you need.  _arXiv preprint arXiv:1911.02150_.
  * Stern et al. (2018) Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018.  Blockwise parallel decoding for deep autoregressive models.  _Advances in Neural Information Processing Systems_ , 31.
  * Team et al. (2023) Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_.
  * Team et al. (2024) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024.  Gemma: Open models based on gemini research and technology.  _arXiv preprint arXiv:2403.08295_.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  _Preprint_ , arXiv:2307.09288.
  * Varshney et al. (2023) Neeraj Varshney, Agneet Chatterjee, Mihir Parmar, and Chitta Baral. 2023.  Accelerating llama inference by enabling intermediate layer decoding via instruction tuning with lite.  _arXiv e-prints_ , pages arXiv–2310.
  * Xia et al. (2024) Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. 2024.  Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding.  _arXiv preprint arXiv:2401.07851_.
  * Xiao et al. (2023) Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023.  Smoothquant: Accurate and efficient post-training quantization for large language models.  In _International Conference on Machine Learning_ , pages 38087–38099. PMLR.
  * Yang et al. (2023) Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023.  Inference with reference: Lossless acceleration of large language models.  _arXiv preprint arXiv:2304.04487_.
  * Yang et al. (2024) Sen Yang, Shujian Huang, Xinyu Dai, and Jiajun Chen. 2024.  Multi-candidate speculative decoding.  _arXiv preprint arXiv:2401.06706_.
  * Zhang et al. (2023) Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. 2023.  Draft & verify: Lossless large language model acceleration via self-speculative decoding.  _arXiv preprint arXiv:2309.08168_.
  * Zheng et al. (2024) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _Advances in Neural Information Processing Systems_ , 36.
  * Zhou et al. (2023) Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean-François Kagy, and Rishabh Agarwal. 2023.  Distillspec: Improving speculative decoding via knowledge distillation.  _arXiv preprint arXiv:2310.08461_.

##  Appendix A Overview of appendix

This appendix provides supplementary material that expands on the main
contents. Each section is designed to complement the research presented:

  * •

Appendix B: Broader impact \- Examines the wider implications of our findings
on speculative decoding.

  * •

Appendix C: Future work \- Outlines possible directions for future research,
building upon the current study’s findings to explore new avenues and
improvements.

  * •

Appendix D: Related works \- Provides a comprehensive review of literature and
previous research that relate to the speculative decoding techniques discussed
in the paper.

  * •

Appendix E: Algorithm \- Details the algorithms used in the speculative
decoding processes, providing pseudocode and explanations to support
reproducibility.

  * •

Appendix F: Implementation details \- Offers an in-depth look at the practical
implementation of the speculative decoding methods, including baselines, self-
distillation, training, and GPT-4o evaluation.

  * •

Appendix G: Additional experimental results \- Presents extra experimental
data and analyses that were not included in the main sections due to space
constraints.

  * •

Appendix H: Discussions \- Engages in discussions on results, such as
foundational beliefs that underpin our research approach, the number of drafts
used, and drafter size.

Each appendix is intended to provide clarity and additional context to the
research.

##  Appendix B Broader impact

Implementing language-specific drafters significantly enhances the speed of
large language models tailored to diverse linguistic environments. For
instance, a system could leverage heuristic analysis of input prompt token
distributions to automatically select an optimal drafter, streamlining
processing efficiency. Moreover, if a user interface allows individuals to
choose their preferred language, the system can instantly apply the
corresponding drafter, thereby accelerating response times considerably. Such
advancements not only reduce computational load but also enrich the user
experience by providing rapid and culturally relevant responses in
multilingual contexts.

##  Appendix C Future work

Future projects will explore broadening the scope of our speculative decoding
framework to cover general multi-task environments, extending beyond
multilingual translation to include varied domains such as legal and medical
text processing. A significant challenge lies in developing an efficient
method for selecting the appropriate drafter among multiple options when
direct user input is unavailable or when inputs consist of mixed languages.
This issue becomes more complex as the ambiguity of language indicators
increases. To alleviate this, designing an advanced router capable of
intelligently assigning tasks to the most suitable drafter based on the nature
of the input presents a promising direction. Training such a router involves
leveraging advanced techniques to understand and predict the optimal drafter
based on contextual representations. This approach aims to improve the overall
efficiency and accuracy of language model applications across diverse and
dynamically changing content landscapes.

10: : Target LLM $\mathcal{M}_{q}$, a small assistant model $\mathcal{M}_{p}$,
initial prompt sequence $x_{1},\ldots,x_{t}$ and target sequence length $T$.

21: Initialize $t\leftarrow 1$

2: while $t<T$ do

3: for $k\leftarrow 1,\ldots,K$ do

4:
$x_{t_{k}}\sim\mathcal{M}_{p}(x|x_{1},\ldots,x_{t},x_{t_{1}},\ldots,x_{t_{k-1}})$

5: end for

36: In parallel, compute $K+1$ sets of logits drafts
$x_{t_{1}},\ldots,x_{t_{K}}$ with the target LLM $\mathcal{M}_{q}$:

$\mathcal{M}_{q}(x|x_{1},\ldots,x_{t}),\mathcal{M}_{q}(x|x_{1},\ldots,x_{t},x_{t_{1}}),\ldots,\mathcal{M}_{q}(x|x_{1},\ldots,x_{t},x_{t_{1}},\ldots,x_{t_{K}})$

7: for $j\leftarrow 1,\ldots,K$ do

8: Sample $r\sim U[0,1]$ from a uniform distribution

9: if
$r<\min(1,\frac{\mathcal{M}_{q}(x|x_{1},\ldots,x_{t+j-1})}{\mathcal{M}_{p}(x|x_{1},\ldots,x_{t+j-1})})$
then

10: Set $x_{t+j}\leftarrow x_{t_{j}}$ and $t\leftarrow t+1$

11: else

12: Sample
$x_{t+j}\sim(\mathcal{M}_{q}(x|x_{1},\ldots,x_{t+j-1})-\mathcal{M}_{p}(x|x_{1},\ldots,x_{t+j-1}))_{+}$
and exit for loop.

13: end if

14: end for

415: If all tokens $x_{t+1},\ldots,x_{t+K}$ are accepted, sample extra token

$x_{t+K+1}\sim\mathcal{M}_{q}(x|x_{1},\ldots,x_{t},x_{t+K})$ and set
$t\leftarrow t+1$

16: end while

Algorithm 1 Speculative sampling

##  Appendix D Related works

###  D.1 Speculative decoding

Speculative decoding, advancing from blockwise parallel decoding introduced by
Stern et al. (2018), adopts a draft-then-verify paradigm to enhance LLM
inference efficiency. This method addresses latency issues in autoregressive
decoding, which stem from the extensive memory transfers required for each
token generation, leading to computational underutilization Xia et al. (2024);
Patterson (2004). To further advance this paradigm, Leviathan et al. (2023)
and Chen et al. (2023) introduced speculative decoding and sampling, which
includes the lossless acceleration of various sampling methods. These methods
utilize smaller models from the same series, such as T5-small, to accelerate
inference for larger counterparts like T5-XXL without additional training.

Recent advancements in speculative decoding, exemplified by models like EAGLE
Li et al. (2024) and Medusa Cai et al. (2024), have significantly refined the
efficiency of LLMs by integrating lightweight feedforward neural network (FFN)
heads directly into their architecture. These FFN heads facilitate the early
drafting of token sequences, enhancing throughput and reducing latency.
Similarly, approaches such as the self-speculative model Zhang et al. (2023)
and Elhoushi et al. (2024) incorporate early exiting and layer skipping
strategies, allowing for a reduction in computational load by prematurely
terminating decoding processes or bypassing less impactful neural layers.
Another line of research explores the blockwise parallel language models with
multiple softmax heads pretrained from scratch presented by Stern et al.
(2018) by either refining its drafts Kim et al. (2024) or scaling up the model
size Gloeckle et al. (2024).

###  D.2 Inference acceleration of LLM

As LLMs continue to evolve rapidly, enhancing their inference speed has become
a focal area of research. Traditional techniques such as knowledge
distillation Gu et al. (2023); Ko et al. (2024), model compression Li et al.
(2020), and quantization Xiao et al. (2023) aim to optimize these models but
often require extensive training adjustments or significant architectural
modifications. More recent strategies have shifted towards applying early
exiting mechanisms, particularly within series like T5 Schuster et al. (2022);
Bae et al. (2023) and decoder-only architectures Varshney et al. (2023), to
streamline inference processes. Although early exiting can significantly
hasten model responses by truncating computational sequences, this method
typically introduces a trade-off with performance degradation Schuster et al.
(2022).

##  Appendix E Algorithm: speculative sampling

By referring to Chen et al. (2023), algorithm 1 demonstrates the speculative
sampling process. Initiating with an initial prompt, an assistant model is
utilized to generate multiple prospective continuations at each step, which
are concurrently verified against the target LLM’s predictions.

Each candidate token’s acceptance probability is calculated based on the
target LLM’s relative confidence compared to the assistant model’s suggestion
(i.e., rejection sampling). If a value, randomly drawn from a uniform
distribution, falls below this threshold, the token is accepted and
incorporated into the ongoing sequence. If not, the algorithm recalibrates,
adjusting the speculative path by directly sampling from the differences in
predictions, enhancing accuracy and contextual relevance.

##  Appendix F Implementation details

###  F.1 Baselines

Following the Spec-Bench settings Xia et al. (2024), we have selected 5
speculative decoding methods, all open-source and rigorously tested for
reliability. Each method represents a unique approach to improving LLM
inference speeds:

  1. 1.

SpS Chen et al. (2023): SpS employs a smaller LM from the same model series as
the drafter. In the verification, this method corrects the last token with
residual probability if the token is rejected.

  2. 2.

Medusa Cai et al. (2024) and Eagle Li et al. (2024): Both methods enhance the
target LLM by integrating additional lightweight FFN heads. These heads are
designed to efficiently draft potential token sequences depending on the
penultimate representations from the target LLM.

  3. 3.

Lookahead Fu et al. (2024): This method appends multiple special tokens to the
end of the input prompt. These tokens are used for parallel drafting, with the
resultant drafts transformed into n-gram candidates for efficient prediction.

  4. 4.

PLD Saxena (2023): Serving as the practical code implementation of Yang et al.
(2023), PLD selects text spans directly from the input to serve as drafts,
optimizing the relevance and accuracy of the initial predictions.

###  F.2 Self-distillation

We follow the self-distillation pipeline as described by Cai et al. (2024).
Initially, a public dataset, such as WMT 16 De-En, is selected as the training
dataset. The target model’s responses are then generated using the OpenAI API
server, with input prompts derived directly from the training dataset.

#### Install prerequisites

For software dependencies, CUDA 12.1 and PyTorch 2.1.2 are required. To start
the server, install the necessary dependencies:

⬇

vllm==0.4.0, openai==0.28.0

#### Use of vLLM

We utilize the vLLM library for self-distillation, executing the following
command:

⬇

python -m

vllm.entrypoints.openai.api_server

–model lmsys/vicuna-7b-v1.3

–port 8000 –max-model-len 2048

#### Input prompt

For instance, when self-distillation the WMT14 Fr-En dataset using the
Vicuna7b v1.3 model, the input prompt consists of a system prompt and a user
prompt. In the user prompt, we prepend "Translate French to English: ".

A chat between a curious user and an artificial intelligence assistant. The
assistant gives helpful, detailed, and polite answers to the user’s questions.
USER: Translate French to English: Madame la Présidente, c’est une motion de
procédure. ASSISTANT:

###  F.3 Details on training setup

For the shared settings across all training drafters, we employ the
Fastchat555https://github.com/lm-sys/FastChat/tree/main framework. We utilize
a cosine learning rate scheduler with a warmup ratio of 0.03 and the AdamW
Loshchilov and Hutter (2017) optimizer. The drafter is trained using the ‘P-F’
strategy (ours) for 3 epochs, and using the ‘F’ strategy (without the
pretraining step ‘P’) for 5 epochs to ensure sufficient learning. The model’s
maximum length is set to 2048 tokens. The training is conducted using 4 GPUs
with a batch size of 2 per GPU.

For finetuning the Vicuna 68M drafter Yang et al. (2024), the learning rate is
set to 2e-5. Similarly, for finetuning the Llama 68M model Miao et al. (2024),
the learning rate is set to 3e-5.

As a drafter for Gemma-Instruct 7B model, we newly design a Gemma 250M model
as a drafter (Table 4). We use the same training recipe with Vicuna 68M and
Llama 68M.

Table 4: Custom Gemma 250M model configuration.

Configuration | Value
---|---
Activation function | GeLU Hendrycks and Gimpel (2016)
Hidden size | 768
Intermediate size | 6144
Number of attention heads | 16
Number of hidden layers | 2
Number of key-value heads | 2
RMS epsilon | 1e-06
Vocabulary size | 256000

###  F.4 Details on GPT-4o evaluation

We follow LLM-as-a-Judge framework Zheng et al. (2024) to evaluate the model’s
answers. The GPT-4o model is utilized as a judge, which has greater
performance on both English and non-English than GPT-4 Turbo OpenAI (2024).
For Single answer grading, used prompt is followed:

[System] You are a helpful assistant. Please act as an impartial judge and
evaluate the quality of the response provided by an AI assistant to the user
question displayed below. Your evaluation should consider factors such as the
helpfulness, relevance, accuracy, depth, creativity, and level of detail of
the response. Begin your evaluation by providing a short explanation. Be as
objective as possible. After providing your explanation, you must rate the
response on a scale of 1 to 10 by strictly following this format:
"[[rating]]", for example: "Rating: [[5]]". [Question] {question} [The Start
of Assistant’s Answer] {answer} [The End of Assistant’s Answer]

The detail implementation of LLM-as-a-judge is in the following GitHub
repository666https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge.

(a) T=0.8

(b) T=0.9

Figure 7: GPT-4o evaluation scores following the Zheng et al. (2024) on
various multilingual translation dataset. Each figure denotes the score of
random sampling with different temperature on the output whose target LLM is
Vicuna 7B v1.3.

(a) Drafter trained on Ru-En

(b) Drafter trained on Ja-En

(c) Drafter trained on Zh-En

Figure 8: Speedup with speculative greedy sampling with the same settings in
Figure 5.

##  Appendix G Additional experimental results

###  G.1 Out-of-domain speedup

Building on the findings discussed in the main body, this subsection further
explores the speedup variations achieved by employing a drafter trained on
each dataset across a range of translation tasks. Figure 8 depicts the speedup
results using speculative greedy sampling for drafters trained on different
datasets: Ru-En, Ja-En, and Zh-En.

Most observations align with those discussed in section 3. Notably, drafters
trained on the Ja-En (Figure 8 (8(b))) and Zh-En (Figure 8 (8(c))) datasets
consistently outperform Yang et al. (2024)’s drafter, even on out-of-domain
tasks. We hypothesize these into two folds. Firstly, this suggests that
certain intrinsic properties of the Japanese and Chinese languages may improve
the efficacy of speculative decoding when applied to unrelated language pairs,
possibly due to specific syntactic or lexical features that are effectively
captured during training. In another scenario, the target LLM does not work
well on those tasks, and thus drafters are easier to catch the target token
distribution. More precisely, for instance, in Zh-Ru task, Vicuna 7B should
translate the Chinese to Russian, but to English, and thus the speedup seems
to happen for us due to English generation.

In the case of the Ru-En (Figure 8 (8(a))) trained drafter, translations from
Russian to other languages generally surpass Yang et al. (2024)’s results.
Interestingly, translations from French to English and German to English
exhibit unexpectedly high speedups. This could hint at underlying linguistic
similarities or shared grammatical structures between Russian, French, and
German that the Ru-En drafter is particularly adept at handling, thereby
facilitating more efficient speculative decoding. While Fan et al. (2021)
demonstrates that Russian belongs to another cluster from En / Fr / De,
perhaps our results provide a different perspective in lens of speculative
decoding.

###  G.2 GPT-4o judgments

Figure 7 show additional GPT-4o evaluation scores for various multilingual
translation datasets. The graphs display the comparative performance across
different language pairs under two sampling conditions, at temperatures
$T$=$0.8$ and $T$=$0.9$, respectively. Each data point reflects the quality of
translations produced by the target model (orange circle), SpS with the
instruction tuned model using ShareGPT Yang et al. (2024) (green pentagon),
and SpS with our specialized drafter (pretrain-and-finetune) (red square). For
the red points, each drafter is trained with the corresponding dataset. For
instance, when the red point specify De-En, it indicates that the drafter has
been fine-tuned with the De-En dataset.

The results demonstrate negligible differences in quality among the three
methods, underscoring the efficacy of speculative decoding in delivering
translations with lossless quality. Both temperature settings show that our
speculative decoding strategy closely matches the performance of the
established target model across various language pairs. This consistent
performance across different settings and language pairs illustrates that
speculative decoding effectively maintains high-quality outputs without
compromising accuracy due to increased randomness in sampling.

Table 5: Speedup comparison of speculative greedy sampling across different
drafter sizes on WMT16 De-En dataset.

Drafter | Vicuna 68M Yang et al. (2024) | Vicuna 68M (pretrain-and-finetune; Ours) | Tiny-Vicuna 1B Pan (2023)
---|---|---|---
Speedup | 1.19 | 2.42 | 0.75
Mean of accepted tokens | 1.47 | 3.03 | 3.06

##  Appendix H Discussion

###  H.1 Why is pretrain-and-finetune better in small-size LM drafter?

Drafting in speculative decoding has been treated akin to n-gram prediction
Bhendawade et al. (2024), often relying on straightforward pretraining using
datasets designed to replicate target LLM behaviors, such as the ShareGPT
dataset Yang et al. (2024). This approach posits that generating a limited
sequence of future tokens suffices for speculative inference.

Contrary to this belief, our empirical result presents a different narrative.
Figure 5 illustrates that even in seemingly straightforward translation tasks,
such as from German to English, outcomes are not as effective. This suggests
that drafting requires a broader array of language modeling capabilities to
manage complex linguistic structures and context variations effectively.

Drafters, therefore, benefit significantly from a robust pretrain-and-finetune
approach, where they are first exposed to a wide array of linguistic contexts
and then finely tuned to specific tasks. This training regimen transforms them
into compact, yet comprehensive, language models capable of handling diverse
and challenging speculative decoding scenarios with better alignment.

###  H.2 Number of drafts

This study primarily explores the speculative decoding process utilizing a
single draft. In contrast, advanced baseline methods such as EAGLE and Medusa
deploy multiple drafts, leveraging tree-attention mechanisms to enrich draft
selection. This technique allows for a broader exploration of multiple draft
candidates at each decoding step, potentially increasing the rate and quality
of accepted drafts.

Adapting our approach to incorporate multiple drafts with tree-attention could
significantly enhance performance, suggesting an untapped potential in our
method. Experimenting with this expanded setup could lead to notable
improvements in the speculative sampling’s effectiveness, particularly in
increasing the mean number of high-quality tokens accepted per sequence. This
prospect opens a critical path for future research, where deeper explorations
could elevate the capabilities of our specialized drafters.

###  H.3 Is scaling up drafter size better for SpS?

Evaluating the efficacy of increasing drafter size reveals nuanced insights
into speculative decoding performance. Table 5 compares three versions of
drafters: the Vicuna 68M by Yang et al. (2024), our pretrain-and-finetune
Vicuna 68M, and Tiny-Vicuna 1B Pan (2023)—a larger model with 1B parameters
that has been instruction-tuned.

Despite Tiny-Vicuna 1B’s substantial parameter count, it achieves a lower
speedup of 0.75 compared to 2.34 by our optimized Vicuna 68M. Both models show
similar mean accepted tokens, suggesting that increasing size does not
proportionally enhance computational efficiency. This is due to speculative
decoding’s reliance on minimizing memory bottlenecks to exploit parallel
computation effectively. Larger models like Tiny-Vicuna 1B exacerbate these
bottlenecks, diminishing the potential speed gains from increased parallelism.

Conversely, our pretrain-and-finetune Vicuna 68M demonstrates that strategic
training and optimization of a smaller model can achieve high efficiency and
speed, highlighting the importance of model configuration over mere size
increase. This balance between model size and computational dynamics is
crucial for optimizing speculative decoding, suggesting that enhancing model
capabilities through targeted training may be more effective than scaling
size.