README.md

---
license: apache-2.0
base_model: meta-llama/Llama-3.2-1B-Instruct
tags:
- dpo
- lora
- peft
- llama-3.2
- iterative-dpo
- self-rewarding
library_name: peft
---

# Iterative DPO Fine-Tune of Llama-3.2-1B (Iteration 2)

This repository contains the LoRA adapters from the **second and final iteration** of a Direct Preference Optimization (DPO) fine-tuning process on the `meta-llama/Llama-3.2-1B-Instruct` model.

This model represents a further refinement of the Iteration 1 model, demonstrating a self-improvement loop where the model learns from preferences on its own generated outputs. This work was inspired by the "Self-Rewarding Language Models" paper.

- **Repository for Iteration 1:** [NilayR/llama32-iterative-dpo-iter1](https://huggingface.co/NilayR/llama32-iterative-dpo-iter1)

## Model Details

### Model Description

This model is the result of the second fine-tuning cycle in an iterative DPO pipeline. The process began with the model from Iteration 1 generating a new set of responses. These responses were then evaluated by an LLM Judge (GPT-3.5-Turbo) to create a fresh preference dataset. This new dataset was used to further fine-tune the model, resulting in the adapters contained in this repository.

The goal of this iteration was to demonstrate that the model could continue to improve its alignment with desired behaviors (accuracy, helpfulness, clarity) using its own outputs as a foundation for learning.

- **Developed by:** NilayR
- **Model type:** Causal Language Model
- **Language(s):** English
- **License:** apache-2.0
- **Finetuned from model:** `meta-llama/Llama-3.2-1B-Instruct` (with adapters from Iteration 1)

## How to Get Started with the Model

To use these LoRA adapters, load the base model (`meta-llama/Llama-3.2-1B-Instruct`) and then apply the adapters from this repository.

```python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

# Set base model ID and adapter path
base_model_id = "meta-llama/Llama-3.2-1B-Instruct"
adapter_id = "NilayR/llama32-iterative-dpo-iter2"

# Configure BitsAndBytes for 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

# Load the base model with quantization
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
tokenizer.pad_token = tokenizer.eos_token

# Load and apply the PEFT adapters
model = PeftModel.from_pretrained(base_model, adapter_id)

# --- Generate a response ---
prompt = "What are the key benefits of meditation?"
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": prompt}
]

input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    input_ids,
    max_new_tokens=200,
    do_sample=True,
    temperature=0.7,
    top_p=0.95
)

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response.split("assistant")[-1].strip())
```

## Training Details

### Training Data

The model was trained on a preference dataset generated by the **Iteration 1 model** (`NilayR/llama32-iterative-dpo-iter1`).

  * **Data Generation Process:**
    1.  **Instructions:** The model from Iteration 1 generated responses to 20 instructions from the LIMA dataset.
    2.  **Preference Labeling:** A custom LLM Judge powered by `GPT-3.5-Turbo` evaluated pairs of the new responses, creating a dataset of **57 chosen/rejected pairs**.

### Training Procedure

The model was trained for one epoch using the TRL library's `DPOTrainer`.

#### Training Hyperparameters

  * **Framework:** `trl.DPOTrainer`
  * **Epochs:** 1
  * **Batch Size:** 1
  * **Gradient Accumulation Steps:** 2 (Effective Batch Size: 2)
  * **Optimizer:** `paged_adamw_8bit`
  * **Learning Rate:** 2e-5
  * **DPO Beta (β):** 0.1
  * **Max Steps:** 50
  * **Final Training Loss:** `0.6343`

#### LoRA Configuration

  * **Rank (`r`):** 16
  * **Alpha (`lora_alpha`):** 32
  * **Target Modules:** `q_proj`, `k_proj`, `v_proj`, `o_proj`
  * **Dropout:** 0.05

### Compute Infrastructure

  * **Hardware:** 1x NVIDIA A100 40GB GPU
  * **Cloud Provider:** Google Colab
  * **Software:** `transformers`, `peft`, `trl`, `bitsandbytes`

-----


```
```