|
|
--- |
|
|
license: apache-2.0 |
|
|
base_model: unsloth/Meta-Llama-3.1-8B-Instruct |
|
|
tags: |
|
|
- diffusion |
|
|
- language-model |
|
|
- llama |
|
|
- text-generation |
|
|
library_name: transformers |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
# Llama-3.1-8B Diffusion Model (LAD) |
|
|
|
|
|
This is a **Language Autoregressive Diffusion (LAD)** model based on Llama-3.1-8B-Instruct. |
|
|
|
|
|
## Features |
|
|
- 🎯 Dual mode: Autoregressive + Diffusion generation |
|
|
- 🚀 Cosine noise schedule with 1000 timesteps |
|
|
- 🧠 LoRA fine-tuning (rank 32) |
|
|
- ⚡ Custom diffusion components |
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
|
|
|
model = AutoModelForCausalLM.from_pretrained("rootxhacker/llama3-diffusion") |
|
|
tokenizer = AutoTokenizer.from_pretrained("rootxhacker/llama3-diffusion") |
|
|
|
|
|
# Generate text |
|
|
inputs = tokenizer("The future of AI", return_tensors="pt") |
|
|
outputs = model.generate(**inputs, max_length=100) |
|
|
print(tokenizer.decode(outputs[0])) |
|
|
``` |
|
|
|
|
|
## Training Details |
|
|
- Base: Meta-Llama-3.1-8B-Instruct |
|
|
- Dataset: PatrickHaller/cosmopedia-v2-1B |
|
|
- Framework: Unsloth + Custom Diffusion |
|
|
- Context: 256 tokens |
|
|
- Training: 60% AR + 40% Diffusion |
|
|
|
|
|
Uploaded: 2025-06-08 23:13 |
|
|
|
