README.md

metadata

datasets:
  - togethercomputer/RedPajama-Data-V2
language:
  - de
library_name: transformers
license: other
pipeline_tag: feature-extraction
tags:
  - masked-lm
  - long-context
base_model:
  - LSX-UniWue/LLaMmlein_1B

LLäMmlein2Vec 1B

LLäMmlein2Vec 1B is a German encoder language model derived from our German decoder-only model LLäMmlein 1B via LLM2Vec.
Find more details in our preprint!

We provide three transformed models:

• LLäMmlein 7B

• LLäMmlein 1B ← You are here

• LLäMmlein 120M

Usage

You can use LLäMmlein2Vec with the llm2vec library.

import torch
from llm2vec import LLM2Vec

model_id = "LSX-UniWue/LLaMmlein2Vec_1B"
l2v = LLM2Vec.from_pretrained(
    model_id, 
    device_map="cuda" if torch.cuda.is_available() else "cpu",
    torch_dtype=torch.bfloat16,
)

License

We release the ModernGBERT models under a research-only RAIL-M license. See license.md for details.