File size: 19,511 Bytes
356892d 6c936e6 eb43613 87eba90 356892d 30156e8 87eba90 356892d eb43613 87eba90 eb43613 a814f06 87eba90 eb43613 87eba90 028f607 87eba90 028f607 87eba90 356892d 87eba90 356892d 87eba90 356892d 87eba90 356892d 87eba90 356892d 87eba90 356892d 87eba90 a814f06 87eba90 356892d 87eba90 b83312b 87eba90 356892d 87eba90 356892d 87eba90 356892d 87eba90 356892d 87eba90 356892d b83312b 87eba90 356892d b83312b 356892d 87eba90 356892d 87eba90 356892d 87eba90 356892d 87eba90 356892d 87eba90 356892d 87eba90 356892d 87eba90 356892d b83312b 87eba90 eb43613 356892d 87eba90 356892d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 |
---
license: mit
language:
- en
pipeline_tag: fill-mask
---
# Ettin: an Open Suite of Paired Encoders and Decoders
[](https://opensource.org/licenses/MIT)
[](https://github.com/jhu-clsp/ettin-encoder-vs-decoder)
[](https://huggingface.co/jhu-clsp)
[](https://huggingface.co/datasets/jhu-clsp)
[](https://github.com/jhu-clsp/ettin-encoder-vs-decoder)
> 🎯 **TL;DR**: State-of-the-art paired encoder and decoder models (17M-1B params) trained identically for fair comparison with open data. Encoders beat ModernBERT. Decoders beat Llama 3.2/SmolLM2.
📄 [Paper (Coming Soon)](https://github.com/jhu-clsp/ettin-encoder-vs-decoder) | 🚀 [GitHub Repository](https://github.com/jhu-clsp/ettin-encoder-vs-decoder)
This model is part of the Ettin suite - the first collection of paired encoder-only and decoder-only models trained with identical data, architecture, and training recipes. Ettin enables fair comparisons between encoder and decoder architectures across multiple scales, providing state-of-the-art performance for open-data models in their respective size categories.
## Table of Contents
- [Performance Highlights](#performance-highlights)
- [Quick Start](#quick-start)
- [Model Description](#model-description)
- [Training Data](#training-data)
- [Model Family](#model-family)
- [Encoder Models](#encoder-models)
- [Decoder Models](#decoder-models)
- [Cross-Objective Models](#cross-objective-models)
- [Accessing Training Checkpoints](#accessing-training-checkpoints)
- [Usage Examples](#usage-examples)
- [Research Applications](#research-applications)
- [Training Details](#training-details)
- [Model Architecture](#model-architecture)
- [Citation](#citation)
## 📊 Performance Highlights
### Encoder Tasks (vs. ModernBERT)
- **GLUE Average**: 88.9 vs 88.4 (Base), 90.8 vs 90.4 (Large)
- **MTEB v2 English Retrieval**: 45.7 vs 43.9 (Base), 48.4 vs 47.0 (Large)
- **Code Search and Long Context**: Superior performance on CodeSearchNet and MLDR
### Decoder Tasks (vs. SmolLM2 & Llama 3.2)
- **Average Score**: 46.2 vs 45.2 (SmolLM2-135M)
- **1B Model**: 59.0 vs 56.6 (Llama 3.2-1B)
- **Generative Tasks**: Competitive across all model sizes
### Key Finding
**Architecture-specific advantages persist**: A 400M encoder outperforms a 1B decoder on classification tasks, while a 400M decoder outperforms a 1B encoder on generation tasks.
## 🚀 Quick Start
### Installation
```bash
pip install torch>=1.9.0 transformers>=4.21.0
```
### 30-Second Examples
**Encoder for Classification/Embeddings:**
```python
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/ettin-encoder-150m")
model = AutoModel.from_pretrained("jhu-clsp/ettin-encoder-150m")
```
**Decoder for Text Generation:**
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/ettin-decoder-150m")
model = AutoModelForCausalLM.from_pretrained("jhu-clsp/ettin-decoder-150m")
```
## Model Description
Ettin models are designed to provide a foundation for comparing encoder-only and decoder-only architectures. Unlike previous comparisons that were limited by different training data, architectures, and recipes, Ettin models use:
1. **Identical training data** - Same high-quality mixture across all models
2. **Open Training Data** - Data is available now with batch-level training data for each of the 250+ checkpoints
3. **Matched architectures** - Only differing in attention patterns (bidirectional vs causal) and training objectives (MLM vs CLM)
4. **Consistent training recipe** - Three-phase training with 2T tokens
5. **Multiple scales** - From 17M to 1B parameters
This approach allows for true apples-to-apples comparisons between encoder and decoder models, revealing the inherent strengths of each architecture.
## Training Data
The training data is publicly available and split across different phases:
- **Pre-training Data**: [jhu-clsp/ettin-pretraining-data](https://huggingface.co/datasets/jhu-clsp/ettin-pretraining-data) - 1.7T tokens of diverse data mixture
- **Mid-training/Extension Data**: [jhu-clsp/ettin-extension-data](https://huggingface.co/datasets/jhu-clsp/ettin-extension-data) - 250B tokens of higher-quality filtered data
- **Decay Phase Data**: [jhu-clsp/ettin-decay-data](https://huggingface.co/datasets/jhu-clsp/ettin-decay-data) - 100B tokens of premium data sources
- **Training Data Order**: [jhu-clsp/ettin-data-order](https://huggingface.co/datasets/jhu-clsp/ettin-data-order) - Batch-level training order (columns: input_ids, step)
## Model Family
### Encoder Models
| Size | Model | Parameters | Best For | Download |
|:-----|:------|:-----------|:---------|:---------|
| XXS | [ettin-encoder-17m](https://huggingface.co/jhu-clsp/ettin-encoder-17m) | 17M | Mobile/Edge devices | [](https://huggingface.co/jhu-clsp/ettin-encoder-17m) |
| XS | [ettin-encoder-32m](https://huggingface.co/jhu-clsp/ettin-encoder-32m) | 32M | Fast inference | [](https://huggingface.co/jhu-clsp/ettin-encoder-32m) |
| Small | [ettin-encoder-68m](https://huggingface.co/jhu-clsp/ettin-encoder-68m) | 68M | Balanced performance | [](https://huggingface.co/jhu-clsp/ettin-encoder-68m) |
| Base | [ettin-encoder-150m](https://huggingface.co/jhu-clsp/ettin-encoder-150m) | 150M | Standard use cases | [](https://huggingface.co/jhu-clsp/ettin-encoder-150m) |
| Large | [ettin-encoder-400m](https://huggingface.co/jhu-clsp/ettin-encoder-400m) | 400M | High accuracy needs | [](https://huggingface.co/jhu-clsp/ettin-encoder-400m) |
| XL | [ettin-encoder-1b](https://huggingface.co/jhu-clsp/ettin-encoder-1b) | 1B | Best performance | [](https://huggingface.co/jhu-clsp/ettin-encoder-1b) |
### Decoder Models
| Size | Model | Parameters | Best For | Download |
|:-----|:------|:-----------|:---------|:---------|
| XXS | [ettin-decoder-17m](https://huggingface.co/jhu-clsp/ettin-decoder-17m) | 17M | Lightweight generation | [](https://huggingface.co/jhu-clsp/ettin-decoder-17m) |
| XS | [ettin-decoder-32m](https://huggingface.co/jhu-clsp/ettin-decoder-32m) | 32M | Quick prototyping | [](https://huggingface.co/jhu-clsp/ettin-decoder-32m) |
| Small | [ettin-decoder-68m](https://huggingface.co/jhu-clsp/ettin-decoder-68m) | 68M | Efficient generation | [](https://huggingface.co/jhu-clsp/ettin-decoder-68m) |
| Base | [ettin-decoder-150m](https://huggingface.co/jhu-clsp/ettin-decoder-150m) | 150M | Standard generation | [](https://huggingface.co/jhu-clsp/ettin-decoder-150m) |
| Large | [ettin-decoder-400m](https://huggingface.co/jhu-clsp/ettin-decoder-400m) | 400M | Quality generation | [](https://huggingface.co/jhu-clsp/ettin-decoder-400m) |
| XL | [ettin-decoder-1b](https://huggingface.co/jhu-clsp/ettin-decoder-1b) | 1B | Best generation | [](https://huggingface.co/jhu-clsp/ettin-decoder-1b) |
### Cross-Objective Models
These models demonstrate what happens when you continue training encoders as decoders (and vice versa). **Important**: Load these models using the architecture they were *converted to*, not their original architecture.
#### Encoders Trained from Decoders (Decoder → MLM)
**Load as encoders** using `AutoModel` or `AutoModelForMaskedLM`:
| Size | Model | Parameters | Description | Download |
|:-----|:------|:-----------|:------------|:---------|
| XXS | [ettin-encoder-from-decoder-17m](https://huggingface.co/jhu-clsp/ettin-encoder-from-decoder-17m) | 17M | Decoder → MLM continued training | [](https://huggingface.co/jhu-clsp/ettin-encoder-from-decoder-17m) |
| XS | [ettin-encoder-from-decoder-32m](https://huggingface.co/jhu-clsp/ettin-encoder-from-decoder-32m) | 32M | Decoder → MLM continued training | [](https://huggingface.co/jhu-clsp/ettin-encoder-from-decoder-32m) |
| Small | [ettin-encoder-from-decoder-68m](https://huggingface.co/jhu-clsp/ettin-encoder-from-decoder-68m) | 68M | Decoder → MLM continued training | [](https://huggingface.co/jhu-clsp/ettin-encoder-from-decoder-68m) |
| Base | [ettin-encoder-from-decoder-150m](https://huggingface.co/jhu-clsp/ettin-encoder-from-decoder-150m) | 150M | Decoder → MLM continued training | [](https://huggingface.co/jhu-clsp/ettin-encoder-from-decoder-150m) |
| Large | [ettin-encoder-from-decoder-400m](https://huggingface.co/jhu-clsp/ettin-encoder-from-decoder-400m) | 400M | Decoder → MLM continued training | [](https://huggingface.co/jhu-clsp/ettin-encoder-from-decoder-400m) |
| XL | [ettin-encoder-from-decoder-1b](https://huggingface.co/jhu-clsp/ettin-encoder-from-decoder-1b) | 1B | Decoder → MLM continued training | [](https://huggingface.co/jhu-clsp/ettin-encoder-from-decoder-1b) |
#### Decoders Trained from Encoders (Encoder → CLM)
**Load as decoders** using `AutoModelForCausalLM`:
| Size | Model | Parameters | Description | Download |
|:-----|:------|:-----------|:------------|:---------|
| XXS | [ettin-decoder-from-encoder-17m](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-17m) | 17M | Encoder → CLM continued training | [](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-17m) |
| XS | [ettin-decoder-from-encoder-32m](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-32m) | 32M | Encoder → CLM continued training | [](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-32m) |
| Small | [ettin-decoder-from-encoder-68m](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-68m) | 68M | Encoder → CLM continued training | [](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-68m) |
| Base | [ettin-decoder-from-encoder-150m](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-150m) | 150M | Encoder → CLM continued training | [](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-150m) |
| Large | [ettin-decoder-from-encoder-400m](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-400m) | 400M | Encoder → CLM continued training | [](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-400m) |
| XL | [ettin-decoder-from-encoder-1b](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-1b) | 1B | Encoder → CLM continued training | [](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-1b) |
**Example Usage for Cross-Objective Models:**
```python
# Encoder-from-decoder: Load as encoder
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/ettin-encoder-from-decoder-150m")
model = AutoModel.from_pretrained("jhu-clsp/ettin-encoder-from-decoder-150m")
# Decoder-from-encoder: Load as decoder
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/ettin-decoder-from-encoder-150m")
model = AutoModelForCausalLM.from_pretrained("jhu-clsp/ettin-decoder-from-encoder-150m")
```
## Accessing Training Checkpoints
Beyond the final models listed above, we provide access to intermediate training checkpoints for research and analysis purposes. These checkpoints allow you to study model behavior and performance throughout the training process. You can get the checkpoints either in HF format or raw for continued pre-training (e.g. Composer format).
#### Raw Checkpoints
All raw training checkpoints are available in the [jhu-clsp/ettin-checkpoints](https://huggingface.co/datasets/jhu-clsp/ettin-checkpoints) dataset.
#### HuggingFace Format Checkpoints
Each model repository contains multiple tagged versions representing different training stages:
- **`step{number}`** - Pretraining phase checkpoints (e.g., `step599525`, `step596528`)
- **`ext{number}`** - Extension/mid-training phase checkpoints (e.g., `ext1000`, `ext2000`)
- **`decay{number}`** - Decay phase checkpoints (e.g., `decay100`, `decay500`)
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
# Load a specific pretraining checkpoint
model = AutoModelForCausalLM.from_pretrained(
"jhu-clsp/ettin-decoder-400m",
revision="step590532" # Specific checkpoint tag
)
# Load an extension phase checkpoint
model = AutoModelForCausalLM.from_pretrained(
"jhu-clsp/ettin-decoder-400m",
revision="ext1000"
)
# Load a decay phase checkpoint
model = AutoModelForCausalLM.from_pretrained(
"jhu-clsp/ettin-decoder-400m",
revision="decay100"
)
```
This checkpoint availability enables detailed analysis of training dynamics, loss curves, and capability emergence across the complete 2T token training process.
## Usage Examples
### Encoder: Masked Language Modeling
<details>
<summary>Click to expand <strong>encoder</strong> usage examples</summary>
```python
from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch
# Load MLM model
tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/ettin-encoder-150m")
model = AutoModelForMaskedLM.from_pretrained("jhu-clsp/ettin-encoder-150m")
def predict_masked_token(text):
inputs = tokenizer(text, return_tensors="pt")
with torch.no_grad():
outputs = model(**inputs)
# Get predictions for [MASK] tokens
mask_indices = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)
predictions = outputs.logits[mask_indices]
# Get top 5 predictions
top_tokens = torch.topk(predictions, 5, dim=-1)
return [tokenizer.decode(token) for token in top_tokens.indices[0]]
# Example
masked_text = "The capital of France is [MASK]."
predictions = predict_masked_token(masked_text)
print(f"Predictions: {predictions}")
```
</details>
### Decoder: Text Generation
<details>
<summary>Click to expand <strong>decoder text generation</strong></summary>
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/ettin-decoder-150m")
model = AutoModelForCausalLM.from_pretrained("jhu-clsp/ettin-decoder-150m")
# Set pad token if needed
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
def generate_text(prompt, max_length=100, temperature=0.7):
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
outputs = model.generate(
inputs.input_ids,
max_length=max_length,
temperature=temperature,
do_sample=True,
pad_token_id=tokenizer.eos_token_id,
num_return_sequences=1
)
return tokenizer.decode(outputs[0], skip_special_tokens=True)
# Example usage
prompt = "The future of artificial intelligence is"
generated = generate_text(prompt)
print(generated)
```
</details>
## 🔬 Research Applications
### What Makes Ettin Unique
Ettin provides the first **controlled comparison** of encoder vs. decoder architectures:
- **Identical Training Data**: Same 2T token mixture across all models
- **Matched Architectures**: Only attention patterns and objectives differ
- **Open Everything**: Training data, model weights, and batch-level training order
- **Multiple Scales**: Fair comparison from 17M to 1B parameters
- **250+ Checkpoints**: Complete training trajectory analysis
### Key Research Findings
1. **Architecture Specialization Persists**:
- Encoders excel at classification/retrieval even vs. larger decoders
- Decoders excel at generation even vs. larger encoders
- A 400M encoder beats a 1B decoder on MNLI (89.2 vs 88.2)
2. **Cross-Training Limitations**:
- Converting decoder→encoder or encoder→decoder underperforms
- 50B tokens of continued training insufficient to close gaps
- Native training objective remains superior
3. **Scaling Insights**:
- Performance gaps between architectures widen with size
- Decoder-from-encoder adaptation scales particularly poorly
### Use Cases for Researchers
- **Architecture Studies**: Compare encoder vs decoder capabilities fairly
- **Training Dynamics**: Analyze 250+ checkpoints with batch-level data ordering
- **Scaling Laws**: Study how architectural advantages change with scale
- **Transfer Learning**: Investigate cross-objective training effectiveness
- **Replication Studies**: First open replication of ModernBERT training recipe
### Reproducibility
All training artifacts are publicly available:
- Training data with exact batch ordering
- Model checkpoints every 8.5B tokens
- Complete hyperparameter configurations
- Training code and evaluation scripts
## Training Details
**Data:** High-quality mixture including DCLM, Dolma v1.7, scientific papers, code, and curated sources totaling 2T+ tokens
**Architecture:** Transformer with RoPE, GLU activations, and prenorm layers
**Training Phases:**
- **Pre-training**: 1.7T tokens with diverse data mixture
- **Mid-training**: 250B tokens with higher-quality filtered data and context extension to 8K
- **Decay phase**: 100B tokens with premium data sources
**Key Features:**
- Context length: Up to 8K tokens
- Vocabulary: 50,368 tokens (ModernBERT tokenizer)
- Deep but efficient architectures following MobileLLM principles
## Model Architecture
| Parameter | 17M | 32M | 68M | 150M | 400M | 1B |
|:----------|:----|:----|:----|:-----|:-----|:---|
| Layers | 7 | 10 | 19 | 22 | 28 | 28 |
| Hidden Size | 256 | 384 | 512 | 768 | 1024 | 1792 |
| Intermediate Size | 384 | 576 | 768 | 1152 | 2624 | 3840 |
| Attention Heads | 4 | 6 | 8 | 12 | 16 | 28 |
## Citation
If you use Ettin models in your research, please cite our work:
```bibtex
@misc{weller2025seqvsseq,
title={Seq vs Seq: An Open Suite of Paired Encoders and Decoders},
author={Orion Weller and Kathryn Ricci and Marc Marone and Antoine Chaffin and Dawn Lawrie and Benjamin Van Durme},
year={2025},
note={Paper coming soon},
url={https://github.com/jhu-clsp/ettin-encoder-vs-decoder},
}
``` |