|
|
--- |
|
|
datasets: |
|
|
- togethercomputer/RedPajama-Data-V2 |
|
|
language: |
|
|
- de |
|
|
library_name: transformers |
|
|
license: other |
|
|
pipeline_tag: feature-extraction |
|
|
tags: |
|
|
- masked-lm |
|
|
- long-context |
|
|
base_model: |
|
|
- LSX-UniWue/LLaMmlein_1B |
|
|
--- |
|
|
|
|
|
# LLäMmlein2Vec 1B |
|
|
|
|
|
LLäMmlein2Vec 1B is a German encoder language model derived from our German decoder-only model [LLäMmlein 1B](https://huggingface.co/LSX-UniWue/LLaMmlein_1B) via [LLM2Vec](https://github.com/McGill-NLP/llm2vec). |
|
|
Find more details in our [preprint](https://arxiv.org/abs/2505.13136)! |
|
|
|
|
|
|
|
|
We provide three transformed models: |
|
|
|
|
|
|
|
|
* [LLäMmlein 7B](https://huggingface.co/LSX-UniWue/LLaMmlein2Vec_7B) |
|
|
|
|
|
* [LLäMmlein 1B](https://huggingface.co/LSX-UniWue/LLaMmlein2Vec_1B) ← You are here |
|
|
|
|
|
* [LLäMmlein 120M](https://huggingface.co/LSX-UniWue/LLaMmlein2Vec_120M) |
|
|
|
|
|
|
|
|
### Usage |
|
|
You can use LLäMmlein2Vec with the `llm2vec` library. |
|
|
|
|
|
```python |
|
|
import torch |
|
|
from llm2vec import LLM2Vec |
|
|
|
|
|
model_id = "LSX-UniWue/LLaMmlein2Vec_1B" |
|
|
l2v = LLM2Vec.from_pretrained( |
|
|
model_id, |
|
|
device_map="cuda" if torch.cuda.is_available() else "cpu", |
|
|
torch_dtype=torch.bfloat16, |
|
|
) |
|
|
``` |
|
|
|
|
|
### License |
|
|
|
|
|
We release the ModernGBERT models under a research-only RAIL-M license. See [license.md](./license.md) for details. |
