|
|
---
|
|
|
license: other
|
|
|
license_name: tencent-hunyuan-community
|
|
|
license_link: https://huggingface.co/Tencent-Hunyuan/HunyuanDiT/blob/main/LICENSE.txt
|
|
|
language:
|
|
|
- en
|
|
|
---
|
|
|
|
|
|
# HunyuanDiT TensorRT Acceleration
|
|
|
|
|
|
English | [中文](https://huggingface.co/Tencent-Hunyuan/TensorRT-libs/blob/main/README_zh.md)
|
|
|
|
|
|
We provide a TensorRT version of [HunyuanDiT](https://github.com/Tencent/HunyuanDiT) for inference acceleration
|
|
|
(faster than flash attention). One can convert the torch model to TensorRT model using the following steps.
|
|
|
|
|
|
## 1. Download dependencies from huggingface.
|
|
|
|
|
|
```shell
|
|
|
cd HunyuanDiT
|
|
|
# Use the huggingface-cli tool to download the model.
|
|
|
huggingface-cli download Tencent-Hunyuan/TensorRT-libs --local-dir ./ckpts/t2i/model_trt
|
|
|
```
|
|
|
|
|
|
## 2. Install the TensorRT dependencies.
|
|
|
|
|
|
```shell
|
|
|
sh trt/install.sh
|
|
|
```
|
|
|
|
|
|
## 3. Build the TensorRT engine.
|
|
|
|
|
|
|
|
|
### Method 1: Use the prebuilt engine
|
|
|
|
|
|
We provide some prebuilt TensorRT engines.
|
|
|
|
|
|
| Supported GPU | Download Link | Remote Path |
|
|
|
|:----------------:|:---------------------------------------------------------------------------------------------------------------:|:---------------------------------:|
|
|
|
| GeForce RTX 3090 | [HuggingFace](https://huggingface.co/Tencent-Hunyuan/TensorRT-engine/blob/main/engines/RTX3090/model_onnx.plan) | `engines/RTX3090/model_onnx.plan` |
|
|
|
| GeForce RTX 4090 | [HuggingFace](https://huggingface.co/Tencent-Hunyuan/TensorRT-engine/blob/main/engines/RTX4090/model_onnx.plan) | `engines/RTX4090/model_onnx.plan` |
|
|
|
| A100 | [HuggingFace](https://huggingface.co/Tencent-Hunyuan/TensorRT-engine/blob/main/engines/A100/model_onnx.plan) | `engines/A100/model_onnx.plan` |
|
|
|
|
|
|
Use the following command to download and place the engine in the specified location.
|
|
|
|
|
|
```shell
|
|
|
huggingface-cli download Tencent-Hunyuan/TensorRT-engine <Remote Path> --local-dir ./ckpts/t2i/model_trt/engine
|
|
|
```
|
|
|
|
|
|
### Method 2: Build your own engine
|
|
|
|
|
|
If you are using a different GPU, you can build the engine using the following command.
|
|
|
|
|
|
```shell
|
|
|
# Set the TensorRT build environment variables first. We provide a script to set up the environment.
|
|
|
source trt/activate.sh
|
|
|
|
|
|
# Method 1: Build the TensorRT engine. By default, it will read the `ckpts` folder in the current directory.
|
|
|
sh trt/build_engine.sh
|
|
|
|
|
|
# Method 2: If your model directory is not `ckpts`, you need to specify the model directory.
|
|
|
sh trt/build_engine.sh </path/to/ckpts>
|
|
|
```
|
|
|
|
|
|
4. Run the inference using the TensorRT model.
|
|
|
|
|
|
```shell
|
|
|
# Run the inference using the prompt-enhanced model + HunyuanDiT TensorRT model.
|
|
|
python sample_t2i.py --prompt "渔舟唱晚" --infer-mode trt
|
|
|
|
|
|
# Close prompt enhancement. (save GPU memory)
|
|
|
python sample_t2i.py --prompt "渔舟唱晚" --infer-mode trt --no-enhance
|
|
|
```
|
|
|
|
|
|
|
