README.md

---
language:
- en
- zh
tags:
- fp8
- quantization
- dynamic
- vision-language
- multimodal
- vllm
- llm-compressor
- internvl3.5
base_model: OpenGVLab/InternVL3_5-1B
base_model_relation: quantized
pipeline_tag: image-text-to-text
inference: false
license: mit
---
# 🔥 InternVL3_5-1B-FP8-Dynamic 🔥
This is a **fp8 dynamic (w8a8)** version of [OpenGVLab/InternVL3_5-1B](https://huggingface.co/OpenGVLab/InternVL3_5-1B), optimized for high-performance inference with vLLM. 
The model utilizes **fp8 dynamic (w8a8)** for optimal performance and deployment.

## Just Run It (vLLM serve)

You can serve the model using vLLM's OpenAI-compatible API server.

```bash
vllm serve brandonbeiler/InternVL3_5-1B-FP8-Dynamic \
    --quantization compressed-tensors \
    --served-model-name internvl3_5-1b \
    --reasoning-parser qwen3 \
    --trust-remote-code \
    --max-model-len 32768 \
    --tensor-parallel-size 1 # Adjust based on your GPU setup
```
**Notes**
- 32k max context length
- reasoning parser ready to go, requires system prompt to run in thinking mode
- still investigating tool calling


## 🚀 Key Features
- **FP8 Dynamic Quantization**: No calibration required, ready to use immediately
- **Vision-Language Optimized**: Specialized quantization recipe that preserves visual understanding
- **vLLM Ready**: Seamless integration with vLLM for production deployment  
- **Memory Efficient**: ~50% memory reduction compared to FP16 original
- **Performance Boost**: Significant faster inference on H100/L40S GPUs
## 📊 Model Details
- **Original Model**: [OpenGVLab/InternVL3_5-1B](https://huggingface.co/OpenGVLab/InternVL3_5-1B)
- **Source Model**: OpenGVLab/InternVL3_5-1B
- **Quantized Model**: InternVL3_5-1B-FP8-Dynamic
- **Quantization Method**: FP8 Dynamic (W8A8)
- **Quantization Library**: [LLM Compressor](https://github.com/vllm-project/llm-compressor) v0.7.1
- **Quantized by**: [brandonbeiler](https://huggingface.co/brandonbeiler)

## 🏗️ Technical Specifications
### Hardware Requirements
- **Inference**: ? VRAM (+ VRAM for context)
- **Supported GPUs**: H100, L40S, A100 (80GB), RTX 4090 (2x for tensor parallelism)
- **GPU Architecture**: Latest NVIDIA GPUs (Ada Lovelace, Hopper and later) and latest AMD GPUs. Recommended for NVIDIA GPUs with compute capability >=9.0 (Hopper and Blackwell)
### Quantization Details
- **Weights**: FP8 E4M3 with dynamic per-tensor scales
- **Activations**: FP8 E4M3 with dynamic per-tensor scales  
- **Preserved Components**: Vision tower, embeddings, mlp1
## 🔬 Package Versions
This model was created using:
```
llmcompressor==0.7.1
compressed-tensors==0.10.2
transformers==4.55.0
torch==2.7.1
vllm==0.10.1.1
```

*Quantized with ❤️ using LLM Compressor for the open-source community*