File size: 4,178 Bytes
22df496 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 |
---
base_model: Alpha-VLLM/Lumina-Image-2.0
library_name: diffusers
license: apache-2.0
instance_prompt: a puppy, yarn art style
widget:
- text: a puppy in a pond, yarn art style
output:
url: yarn_lora.png
- text: a puppy in a pond, yarn art style (dark env)
output:
url: yarn_lora_You_are_an_assistant_designed_to_generate_superior_images_with_a_dark_overall_theme.png
- text: a puppy in a pond, yarn art style (shiny env)
output:
url: yarn_lora_You_are_an_assistant_designed_to_generate_superior_images_with_a_bright_and_shiny_overall_.png
tags:
- text-to-image
- diffusers-training
- diffusers
- lora
- lumina2
- lumina2-diffusers
- template:sd-lora
---
# Lumina2 DreamBooth LoRA - trained-lumina2-lora-yarn
<Gallery />
## Model description
These are `trained-lumina2-lora-yarn` DreamBooth LoRA weights for [Alpha-VLLM/Lumina-Image-2.0](https://hf.co/Alpha-VLLM/Lumina-Image-2.0).
The weights were trained using [DreamBooth](https://dreambooth.github.io/) with the [Lumina2 diffusers trainer](https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_lumina2.md).
## Trigger words
You should use `yarn art style` to trigger the image generation.
The following `system_prompt` was also used used during training (ignore if `None`): None.
## Download model
[Download the *.safetensors LoRA]({repo_id}/tree/main) in the Files & versions tab.
## Use it with the [🧨 diffusers library](https://github.com/huggingface/diffusers)
```py
import torch
from diffusers import Lumina2Text2ImgPipeline
pipe = Lumina2Text2ImgPipeline.from_pretrained(
"Alpha-VLLM/Lumina-Image-2.0", torch_dtype=torch.bfloat16
).to("cuda")
pipe.load_lora_weights("trained-lumina2-lora-yarn")
prompt = "a puppy in a pond, yarn art style"
image = pipe(
prompt,
negative_prompt="bad quality, worse quality, degenerate quality",
guidance_scale=6,
num_inference_steps=35,
generator=torch.manual_seed(0)
).images[0]
```
For more details, including weighting, merging and fusing LoRAs, check the [documentation on loading LoRAs in diffusers](https://huggingface.co/docs/diffusers/main/en/using-diffusers/loading_adapters).
## Results
The model benefits from `system_prompt`. Here is a comparison across different system prompts:
<table>
<thead>
<tr>
<th>No system prompt</th>
<th>"Dark surrounding"<br>system prompt</th>
<th>"Sunny surrounding"<br>system prompt</th>
</tr>
</thead>
<tbody>
<tr>
<td><img src="yarn_lora.png" alt="No system prompt image" width="200"></td>
<td><img src="yarn_lora_You_are_an_assistant_designed_to_generate_superior_images_with_a_dark_overall_theme.png" alt="Dark surrounding image" width="200"></td>
<td><img src="yarn_lora_You_are_an_assistant_designed_to_generate_superior_images_with_a_bright_and_shiny_overall_.png" alt="Sunny surrounding image" width="200"></td>
</tr>
<tr>
<td colspan="3">
<div style="text-align: center; font-weight: bold;">
<b>Original prompt</b>: <i>a puppy in a pond, yarn art style</i>
</div>
</td>
</tr>
</tbody>
</table>
<details>
<summary>Code</summary>
```py
import torch
from diffusers import Lumina2Text2ImgPipeline
pipe = Lumina2Text2ImgPipeline.from_pretrained(
"Alpha-VLLM/Lumina-Image-2.0", torch_dtype=torch.bfloat16
).to("cuda")
system_prompts = [
None,
"You are an assistant designed to generate superior images with a dark overall theme.",
"You are an assistant designed to generate superior images with a bright and shiny overall theme."
]
pipe.load_lora_weights("trained-lumina2-lora-yarn")
prompt = "a puppy in a pond, yarn art style"
for sp in system_prompts:
filename = "yarn_lora"
image = pipe(
prompt,
negative_prompt="bad quality, worse quality, degenerate quality",
system_prompt=sp,
guidance_scale=6,
num_inference_steps=35,
generator=torch.manual_seed(0)
).images[0]
if sp:
filename += "_" + "_".join(sp.split(" ")).replace(",", "").replace(".", "")
filename = filename[:100]
image.save(f"{filename}.png")
```
</details> |