|
|
--- |
|
|
license: apache-2.0 |
|
|
language: |
|
|
- en |
|
|
- zh |
|
|
--- |
|
|
|
|
|
# FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Prediction |
|
|
|
|
|
<p align="left"> |
|
|
<strong>Technical report (coming soon)</strong> · |
|
|
<a href="https://github.com/Tencent-BAC/FastMTP"><strong>Github</strong></a> · |
|
|
<a href="https://huggingface.co/TencentBAC/FastMTP"><strong>HuggingFace</strong></a> · |
|
|
<a href="https://modelscope.cn/models/TencentBAC/FastMTP"><strong>ModelScope</strong></a> |
|
|
</p> |
|
|
|
|
|
## Overview |
|
|
|
|
|
FastMTP is a simple yet effective method that enhances Multi-Token Prediction (MTP) for speculative decoding during inference. Our approach fine-tunes a single MTP head with shared weights across multiple causal draft steps, enabling it to capture longer-range dependencies and achieve higher acceptance rates in speculative decoding. By incorporating language-aware vocabulary compression, we further reduce computational overhead during draft generation. Experimental results across diverse benchmarks demonstrate that FastMTP achieves an average of 2.03× speedup over vanilla next token prediction while maintaining lossless output quality. With low training cost and seamless integration into existing inference frameworks, FastMTP offers a practical and rapidly deployable solution for accelerating LLM inference. |
|
|
|
|
|
<!-- {width=50%} --> |
|
|
<img src="./assets/mtp-overview.png" width="75%"> |
|
|
|
|
|
Speedup comparison of different methods across subtasks, evaluated on a single A10 GPU: |
|
|
|
|
|
<img src="./assets/radar_chart.png" width="55%"> |
|
|
|
|
|
## What's Included |
|
|
|
|
|
This repository contains the model checkpoints for FastMTP, and the processed compressed vocabulary. |
|
|
|
|
|
## Links |
|
|
|
|
|
- Technical report (coming soon) |
|
|
- Training & inference code: [GitHub Repository](https://github.com/Tencent-BAC/FastMTP) |
|
|
|
