Article
Training Design for Text-to-Image Models: Lessons from Ablations
•
57
Cédric
deltheil
AI & ML interests
None yet
Recent Activity
new activity
9 days ago
finegrain/finegrain-image-enhancer:Your app is paused
upvoted
an
article
9 days ago
Training Design for Text-to-Image Models: Lessons from Ablations
updated
a Space
12 days ago
finegrain/finegrain-image-enhancer
Organizations
Your app is paused
1
#16 opened 9 days ago
by
learrist
upvoted
an
article
9 days ago
updated
a
Space
12 days ago
updated
3
Spaces
24 days ago
zeroing and reshaping the text-related cross-attentions into self-attentions
It's actually narrowing, not zeroing (even though strategy="zeros" is used in the StateDictAdapter()).
For instance, the logs show:
Adapting down_blocks.0.attentions.0.transformer_blocks.0.attn2.to_k.weight by narrowing from shape torch.Size([320, 768]) to torch.Size([320, 320])
So the extra weights are just discarded in this case. Zero-filling is only used when expanding tensors to larger shapes.
Corresponding code: link.
reacted to
piercus's
post with 🔥
4 months ago
Post
3975
Starts erasing! 🎉 🎉 🎉
This is made with a one-step SD1.5 LBM [1] eraser !
Data is open. Data pipeline is open. Training code is open.
On our LBM fork : https://github.com/finegrain-ai/LBM
[1] LBM: Latent Bridge Matching for Fast Image-to-Image Translation (2503.07535)
This is made with a one-step SD1.5 LBM [1] eraser !
Data is open. Data pipeline is open. Training code is open.
On our LBM fork : https://github.com/finegrain-ai/LBM
[1] LBM: Latent Bridge Matching for Fast Image-to-Image Translation (2503.07535)
- 1 reply
reacted to
piercus's
post with 👍
4 months ago
Post
850
🚧 Reproducing LBM-Eraser… in the open [1] !
Today we have trained a LBM [2] promptless inpainter using
We use a subset of 1.25M images with
2 takeaways :
🖼 Inpainting is better compared to our RORD experiments [5]
🦶 "4 steps" outperforms single-step
[1] Finegrain LBM Fork : https://github.com/finegrain-ai/LBM
[2] LBM: Latent Bridge Matching for Fast Image-to-Image Translation (2503.07535)
[3] supermodelresearch/Re-LAION-Caption19M
[4] Resolution-robust Large Mask Inpainting with Fourier Convolutions (2109.07161)
[5] https://huggingface.co/posts/piercus/778833977889788
cc @supermodelresearch @presencesw
Today we have trained a LBM [2] promptless inpainter using
Re-LAION-Caption19M[3].We use a subset of 1.25M images with
aesthetic_score > 5.6 and pwatermark < 0.2 and LaMa [2] mask generation.2 takeaways :
🖼 Inpainting is better compared to our RORD experiments [5]
🦶 "4 steps" outperforms single-step
[1] Finegrain LBM Fork : https://github.com/finegrain-ai/LBM
[2] LBM: Latent Bridge Matching for Fast Image-to-Image Translation (2503.07535)
[3] supermodelresearch/Re-LAION-Caption19M
[4] Resolution-robust Large Mask Inpainting with Fourier Convolutions (2109.07161)
[5] https://huggingface.co/posts/piercus/778833977889788
cc @supermodelresearch @presencesw
reacted to
piercus's
post with 🧠
4 months ago
Post
1861
🚧 Reproducing LBM-Eraser… in progress! [1]
When repurposing a T2I model into a pure I2I model, there’s always that orphaned text path — what do we do with it? 🤔
You can reuse it as learnable embeddings in multi-task setups [2], freeze an empty text prompt, distillate or prune the corresponding part.
In LBM, they take a clever route — zeroing [3] and reshaping [4] the text-related cross-attentions into self-attentions.
This gives you fresh weights for I2I computation, nicely integrated into your SD architecture.
📎 References
[1] Our LBM Fork: https://github.com/finegrain-ai/LBM
[2] OmniPaint: OmniPaint: Mastering Object-Oriented Editing via Disentangled Insertion-Removal Inpainting (2503.08677)
[3] LBM Zeroing: https://github.com/gojasper/LBM/blob/cafebc46a9ac16dcc61691d289cc4676b5c75380/examples/training/train_lbm_surface.py#L147-L148
[4] LBM Reshaping: https://github.com/gojasper/LBM/blob/cafebc46a9ac16dcc61691d289cc4676b5c75380/examples/training/train_lbm_surface.py#L100
When repurposing a T2I model into a pure I2I model, there’s always that orphaned text path — what do we do with it? 🤔
You can reuse it as learnable embeddings in multi-task setups [2], freeze an empty text prompt, distillate or prune the corresponding part.
In LBM, they take a clever route — zeroing [3] and reshaping [4] the text-related cross-attentions into self-attentions.
This gives you fresh weights for I2I computation, nicely integrated into your SD architecture.
📎 References
[1] Our LBM Fork: https://github.com/finegrain-ai/LBM
[2] OmniPaint: OmniPaint: Mastering Object-Oriented Editing via Disentangled Insertion-Removal Inpainting (2503.08677)
[3] LBM Zeroing: https://github.com/gojasper/LBM/blob/cafebc46a9ac16dcc61691d289cc4676b5c75380/examples/training/train_lbm_surface.py#L147-L148
[4] LBM Reshaping: https://github.com/gojasper/LBM/blob/cafebc46a9ac16dcc61691d289cc4676b5c75380/examples/training/train_lbm_surface.py#L100
-
- 2 replies
upvoted
an
article
4 months ago
Article
SOTA OCR with Core ML and dots.ocr
•
62
reacted to
piercus's
post with 🔥
4 months ago
Post
3150
We have trained a LBM-Eraser with RORD-Dataset in the open 🔥
🚀 1-step only inference, no distillation
🪶 Light backbone :SD1.5
🧠 Light training : converge in 6k steps
Now let's improve this, especially the inpainting capabilities. Stay tuned for more :-)
LBM paper : LBM: Latent Bridge Matching for Fast Image-to-Image Translation (2503.07535)
Our LBM fork : https://github.com/finegrain-ai/LBM
🚀 1-step only inference, no distillation
🪶 Light backbone :SD1.5
🧠 Light training : converge in 6k steps
Now let's improve this, especially the inpainting capabilities. Stay tuned for more :-)
LBM paper : LBM: Latent Bridge Matching for Fast Image-to-Image Translation (2503.07535)
Our LBM fork : https://github.com/finegrain-ai/LBM
upvoted
an
article
4 months ago
Article
Swift 🧨Diffusers - Fast Stable Diffusion for Mac
•
5
reacted to
piercus's
post with 👍
4 months ago
Post
2942
We've just forked LBM to reproduce the LBM eraser results
Our fork : https://github.com/finegrain-ai/LBM
LBM paper: LBM: Latent Bridge Matching for Fast Image-to-Image Translation (2503.07535)
LBM relighting demo : jasperai/LBM_relighting
Our fork : https://github.com/finegrain-ai/LBM
LBM paper: LBM: Latent Bridge Matching for Fast Image-to-Image Translation (2503.07535)
LBM relighting demo : jasperai/LBM_relighting
- 2 replies
updated
a
Space
4 months ago
upvoted
2
articles
4 months ago
Article
Make your ZeroGPU Spaces go brrr with ahead-of-time compilation
- +2
•
75
Article
Small Language Models (SLM): A Comprehensive Overview
•
130
upvoted
an
article
5 months ago
Article
Model Quality: Hugging Face Is All You Need
•
23
published
an
article
5 months ago
Article
Model Quality: Hugging Face Is All You Need
•
23