README.md

---
library_name: transformers
license: llama3.1
base_model:
- meta-llama/Llama-3.1-8B-Instruct
tags:
- nlrl
---


# Model Card for Llama-3.1-8B-Instruct-NLRL-Breakthrough-Value

## Model Details

### Model Description

- **Developed by:** NLRL Team
- **Model type:** Language Value Function Model for Breakthrough
- **Language(s):** English
- **License:** MIT
- **Finetuned from model:** LLaMA-3.1-8B-Instruct

This model serves as a language value function in Natural Language Reinforcement Learning (NLRL) framework, specifically trained for the Breakthrough game. It evaluates the state through natural language description and provides value assessment.

## Uses

### Direct Use
This model can be used as a Breakthrough position evaluator that explains its evaluation through natural language before providing the final assessment. The model generates both reasoning chains and final value judgments.

### Out-of-Scope Use
This model is specifically trained for Breakthrough board state evaluation and should not be used for other games or value assessment tasks.

## Training Details

### Training Data
Training data consists of TD data collected through NLRL value learning process, with language-based TD estimates serving as training targets for the value function.

### Training Procedure
- Trained using FSDP (Fully Sharded Data Parallel) across 4 H100 GPUs
- Learning rate: 2e-5
- Training epochs per iteration: 2
- Max sequence length: 1024

## Evaluation
- Demonstrates consistent evaluation capabilities across different game states

## Model Architecture
- Base model: LLaMA-3.1-8B-Instruct
- Input: Text description of Breakthrough board state
- Output: Chain-of-thought evaluation followed by value assessment

## Citation
```bibtex
@misc{feng2024naturallanguagereinforcementlearning,
      title={Natural Language Reinforcement Learning}, 
      author={Xidong Feng and Ziyu Wan and Haotian Fu and Bo Liu and Mengyue Yang and Girish A. Koushik and Zhiyuan Hu and Ying Wen and Jun Wang},
      year={2024},
      eprint={2411.14251},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2411.14251}, 
}
```

## Model Card Contact
[email protected]