Update README.md
Browse files
README.md
CHANGED
@@ -17,6 +17,13 @@ language:
|
|
17 |
|
18 |
FastMTP is a simple yet effective method that enhances Multi-Token Prediction (MTP) for speculative decoding during inference. Our approach fine-tunes a single MTP head with shared weights across multiple causal draft steps, enabling it to capture longer-range dependencies and achieve higher acceptance rates in speculative decoding. By incorporating language-aware vocabulary compression, we further reduce computational overhead during draft generation. Experimental results across diverse benchmarks demonstrate that FastMTP achieves an average of 2.03× speedup over vanilla next token prediction while maintaining lossless output quality. With low training cost and seamless integration into existing inference frameworks, FastMTP offers a practical and rapidly deployable solution for accelerating LLM inference.
|
19 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
20 |
## What's Included
|
21 |
|
22 |
This repository contains the model checkpoints for FastMTP, and the processed compressed vocabulary.
|
|
|
17 |
|
18 |
FastMTP is a simple yet effective method that enhances Multi-Token Prediction (MTP) for speculative decoding during inference. Our approach fine-tunes a single MTP head with shared weights across multiple causal draft steps, enabling it to capture longer-range dependencies and achieve higher acceptance rates in speculative decoding. By incorporating language-aware vocabulary compression, we further reduce computational overhead during draft generation. Experimental results across diverse benchmarks demonstrate that FastMTP achieves an average of 2.03× speedup over vanilla next token prediction while maintaining lossless output quality. With low training cost and seamless integration into existing inference frameworks, FastMTP offers a practical and rapidly deployable solution for accelerating LLM inference.
|
19 |
|
20 |
+
<!-- {width=50%} -->
|
21 |
+
<img src="./assets/mtp-overview.png" width="50%">
|
22 |
+
|
23 |
+
Speedup comparison of different methods across subtasks, evaluated on a single A10 GPU:
|
24 |
+
|
25 |
+
<img src="./assets/radar_chart.png" width="30%">
|
26 |
+
|
27 |
## What's Included
|
28 |
|
29 |
This repository contains the model checkpoints for FastMTP, and the processed compressed vocabulary.
|