(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: Individual Researcher 22institutetext: Individual Researcher

# DivCon: Divide and Conquer for Progressive Text-to-Image Generation

Yuhao Jia 11 Wenhan Tan 22

###### Abstract

Diffusion-driven text-to-image (T2I) generation has achieved remarkable
advancements. To further improve T2I models’ capability in numerical and
spatial reasoning, the layout is employed as an intermedium to bridge large
language models and layout-based diffusion models. However, these methods
still struggle with generating images from textural prompts with multiple
objects and complicated spatial relationships. To tackle this challenge, we
introduce a divide-and-conquer approach which decouples the T2I generation
task into simple subtasks. Our approach divides the layout prediction stage
into numerical & spatial reasoning and bounding box prediction. Then, the
layout-to-image generation stage is conducted in an iterative manner to
reconstruct objects from easy ones to difficult ones. We conduct experiments
on the HRS and NSR-1K benchmarks and our approach outperforms previous state-
of-the-art models with notable margins. In addition, visual results
demonstrate that our approach significantly improves the controllability and
consistency in generating multiple objects from complex textural prompts.

###### Keywords:

Text-to-Image Generation Large Language Models Diffusion Models

https://github.com/DivCon-gen/DivCon

Figure 1: Layouts and images generated by our DivCon. DivCon enhances the
ability of text-to-image diffusion models to understand complex numerical and
spatial relationships in the text.

##  1 Introduction

Large-scale text-to-image generation (T2I) models demonstrate exceptional
zero-shot capacity in efficiently generating small quantities and varieties of
objects [30, 7, 3, 21, 29, 31]. However, these methods still suffer from
limited capability in generating images from text prompts containing specified
object counts, varying sizes, rich details, and complicated spatial
relationships [2, 7, 14, 8, 35]. Recently, combining Large Language Models
(LLMs) and layout-to-image diffusion models has significantly improved T2I
models’ ability in numerical and spatial reasoning [15, 26]. LLMs have strong
reasoning and visual planning ability to predict objects’ layouts from the
text prompt, which provides more informative conditions for diffusion models
to produce higher-quality images.

Despite the significant advancement, existing methods still suffer from
limited accuracy when the text prompt contains multiple objects and
complicated spatial relationships [2, 26]. Intuitively, the difficulty of
synthesizing objects with different characteristics varies a lot.
Specifically, LLMs encounter great challenges when conducting simultaneous
reasoning and visual planning from complicated text prompts (e.g. "five mice
are scurrying around a wooden table, each nibbling on one of the five ripe
apples that are laid out on top."). Besides, layout-to-image diffusion models
exhibit varying levels of generative proficiency for objects with different
sizes, shapes, and details. However, previous methods generate all objects
simultaneously without taking their differences into consideration. As a
result, objects with higher difficulties cannot be well generated in the
images (e.g. toaster and pizza, Fig. 2).

To address this issue, we propose DivCon, a training-free approach that
applies the divide-and-conquer strategy to both layout prediction and image
generation phases. Our approach divides complicated tasks into multiple
subtasks according to their difficulties and then address these subtasks in an
iterative manner. During layout prediction, the number of objects and their
spatial relationships are first reasoned from the text prompt. Then, visual
planning is performed to predict object bounding boxes. During layout-to-image
generation, the diffusion model is first executed to process the layout. Then,
the consistency between the synthetic objects and the text prompt is
calculated. Next, low-fidelity objects’ layouts are highlighted and fed to the
diffusion model to encourage the network to focus on objects with higher
difficulties. Extensive experiments have demonstrated that DivCon has
significantly improved the quality of generated and achieves state-of-the-art
performance on the HRS[2] and NSR-1K[15] benchmark datasets.

Our main contributions can be summarized as follows:

  * •

We propose DivCon, a divide-and-conquer approach for layout-based text-to-
image generation by dividing complicated tasks into simple subtasks.

  * •

To obtain accurate layouts from the text prompt, we divide the layout
prediction stage into numerical & spatial reasoning and bounding box planning.

  * •

To generate high-fidelity images from the layout, we divide layout-to-image
generation into two iterations to synthesize objects with different levels of
difficulty.

  * •

Experimental results show that our approach achieves significant performance
gains, surpassing current state-of-the-art models.

Figure 2: A comparison of generation difficulty between "toaster" and "pizza".
Current methods tend to generate pizza but fail to generate enough toasters or
completely miss the category.

##  2 Related work

In this section, we review three main categories of text-to-image generation
methods: text-to-image generation (T2I) models, layout-conditioned T2I models,
and text-to-layout-to-image generation models (Fig. 3).

####  2.0.1 Text-to-Image (T2I) Generation Models

The evolution of large-scale T2I synthesis demonstrates a significant leap
forward in the realm of generative models, enabling the creation of detailed
and coherent images directly from textual descriptions [3, 17, 29, 21, 31,
38]. Milestone methods in this area are Generative Adversarial Networks (GANs)
[21, 32], Variational Autoencoders (VAEs) [6, 12, 29, 38], and diffusion
models [3, 19, 25, 28, 31]. GANs often struggle with training stability and
mode collapse, whereas VAEs may sometimes produce overly smooth or blurry
outputs due to their reconstruction loss. Recently, diffusion models are
highly valued for their robustness and ability to generate detailed and
realistic images. Particularly, Stable Diffusion[30], which utilizes open
vocabulary encoder[27] for encoding text to embeddings as conditional inputs,
has demonstrated a remarkable capability for open-world image synthesis,
pushing the boundaries of generative models. Despite promising results, a text
prompt is usually difficult to provide fine-grained control of the generated
images.

Figure 3: Three main categories of text-to-image generation methods.

####  2.0.2 Layout-conditioned T2I Models

To improve the controlability of T2I models, prompts like layouts as employed
as additional conditions for T2I models. Existing layout-conditioned
generation models can be categorized into three branches. One branch of
methods trains external networks on layout-images pairs [22, 39, 25, 1, 37].
For instance, GLIGEN incorporates a trainable gated self-attention layer that
transforms bounding box information into conditional input for Stable
Diffusion. Another branch of methods, including attention-refocusing [26],
layout-predictor [34], direct-diffusion[24], layout-guidance[9] and BoxDiff
[36], directly manipulate attention maps to optimize the sampling process.
These methods generate images with multiple objects through a single forward
pass of the diffusion model, resulting in limited fidelity for objects with
complicated appearances or spatial relationships. To remedy this, the last
branch of methods including Mixture-of-Diffusion[5], MultiDiffusion [4], and
LLM-grounded Diffusion [23] perform denoising on each object (_i.e._ ,
bounding box) separately before fusing the results. However, these methods
introduce much higher computational costs. Moreover, the requirements of
additional layout prompts hinder their practicality.

####  2.0.3 Text-to-Layout-to-Image Generation Models

Current LLMs have demonstrated strong numerical and spatial reasoning
abilities in visual planning and layout generation. Inspired by this, several
efforts are made to integrate LLMs to generate layouts from texts, and then
utilize layout-conditioned diffusion models for image generation. Attention-
refocusing [26] and LLM-grounded Diffusion [23] explore in-context learning of
GPT-4 for layout prediction. LayoutGPT [15] searches similar samples from
their database as input examples for the LLMs. LLM-layout-generator [10]
leverages Chain-of-Thought (CoT) prompting [33] to improve the layout
prediction accuracy. However, LLMs are expected to simultaneously perform
numerical & spatial reasoning and visual planning of bounding boxes in these
methods. Contrary to these methods, we propose a divide-and-conquer approach
to conduct reasoning and visual planning separately, which improves the
accuracy of layout predictions and substantially advances the prompt fidelity
of generated images. In addition, during layout-to-image generation, the
objects are synthesized in an iterative manner according to their
difficulties, which achieves not only superior fidelity but also significant
computational cost savings.

Figure 4: The proposed DivCon framework. The layout prediction stage is
divided into numerical & spatial reasoning and bounding box planning. The
layout-to-image generation is divided into two iterations to generate objects
with different difficulty levels separately.

##  3 Method

###  3.1 Overview

As illustrated in Fig. 4, our framework comprises two stages. Given a text
prompt, LLMs first predict layouts from it. Then, the resultant layout is
employed as an additional condition to achieve text-to-image generation. More
specifically, each stage is further divided into two steps to handle simple
subtasks.

###  3.2 Text-to-Layout Prediction

Intuitively, to predict the layout from a text prompt, the number & spatial
relations of the objects are first required to be reasoned, and then bounding
boxes are generated to describe the coordinates & sizes of the objects.
However, previous methods commonly conduct these two steps simultaneously
without considering their inherent correlation, thereby hindering them from
understanding complicated text prompts well. To remedy this, we introduce the
idea of divide-and-conquer to improve the accuracy of predicted layouts.
Specifically, leveraging the state-of-the-art large language model GPT-4 [],
numerical and spatial reasoning is first performed, and then bounding boxes
are predicted.

####  3.2.1 Numerical and Spatial Reasoning

Given a text prompt $C$ describing an image, an LLM is adopted to predict a
set of strings $O=\\{s_{j}|j=1,2,...,n\\}$. If the prompt primarily focused on
counting relationships among objects, each string $s_{j}$ includes both the
name and quantity information of each object category $j$. If the prompt
mainly describes spatial relationships, each string $s_{j}$ will contain the
object name and its spatial position in the image. This method allows LLMs to
conclude complex numerical and spatial relationships.

####  3.2.2 Bounding Box Prediction

After numerical and spatial reasoning, both the meta prompt $C$ and the
prediction result $O$ are fed to the LLM again to predict a list of name-box
pairs $P=\\{p_{i}|i=1,2,...,n\\}$. Each pair
$p_{i}=\\{o_{i}:(l_{i},t_{i},r_{i},b_{i})|i=1,2,...,n\\}$ consists of the
object name $o_{i}$ and its bounding box’s coordinates
$(l_{i},t_{i},r_{i},b_{i})$. The in-context exemplars include a process of
correcting the features of generated bounding boxes to ensure the quality of
the output layouts.

###  3.3 Layout-to-Image Generation

After layout prediction, the resultant layout is employed as an additional
condition to generate the image. As objects in the layout have different
levels of difficulty to synthesize, a divide-and-conquer approach is adopted
to reconstruct objects from easy to hard in an iterative manner, as shown in
Fig. 4. First, the layout is fed to the diffusion model to synthesize an
image. Then, the consistency between the synthetic objects and the text prompt
is calculated. Afterward, objects with low fidelity are highlighted and passed
to the diffusion model for a second time, encouraging the diffusion model to
focus on these difficult objects to achieve higher quality.

####  3.3.1 First-Round Generation

We employ a layout-to-image diffusion model with sampling optimization [26] as
the generative model to generate an image conditioned on the text prompt and
the predicted layout prompt. During the denoising process, random noise
$z_{T}$ is fed to the diffusion model to gradually denoise it to an image
$z_{0}$ after $T$ iterations.

####  3.3.2 Consistency Evaluation

With the generated image $x$, each object is cropped using the bounding boxes
in the layout, which are denoted as $x_{i}\in\mathbb{R}^{4\times h\times w}$,
where $h\times w$ represents the height and width of the bounding box. Then,
the CLIP similarity $S_{x_{i},o_{i}}\in(-1,1)$ is employed to evaluate the
consistency between the cropped patch and the object name $o_{i}$ in the text
prompt. The higher the similarity is, the higher the fidelity of the
corresponding object is. Consequently, the objects with similarities higher
than a specified threshold can be considered as easy samples that are "good
enough”. In contrast, other objects with low similarities are hard samples
that are more difficult to reconstruct (e.g. "sheep" and "horse" in Fig. 5).
Next, a binary mask $M$ is calculated with bounding boxes of easy objects
being set to 1 and the rest regions being set to 0.

Figure 5:  Comparison of easy and hard samples in a generated image.

####  3.3.3 Second-Round Refinement

After consistency evaluation, different objects are of various levels of
fidelity. Our objective is to preserve the “good enough” objects while
reproducing the remaining ones to ease the difficulty of diffusion model. To
this end, we retrieve the latent representation during the denoising process
of the first-round generation. Specifically, $z_{0.7T}$ and $z_{0.3T}$ are
first retrieved. Then, the binary mask $M$ is adopted to crop high-fidelity
regions in $z_{0.7T}$ and $z_{0.3T}$. Next, the starting noise
$z^{\prime}_{T}$ for the second-round diffusion is obtained as:

$\displaystyle z^{\prime}_{T}=z_{0.7T}\circ
M+\mathcal{N}(\mu,\sigma^{2})\circ(1-M)$ (1)

where $\circ$ is element-wise multiplication and $\mathcal{N}(\mu,\sigma^{2})$
is random noise.

The second-round denoising process consists of three stages. First, the input
noise $z^{\prime}_{T}$ is denoised for $0.3T$ iterations to obtain
$z^{\prime}_{0.3T}$. Second, the high-fidelity regions in $z_{0.7T}$ are
cropped and used to replace the corresponding regions in $z^{\prime}_{0.3T}$
and fed to the diffusion model for anthoer $0.4T$ denoising iterations. Note
that, the text prompt and the full layout prompt are employed as conditions in
these two stages. Moreover, the regions marked by $M$ is frozen to preserve
these high-fidelity regions. During optimization, Cross-Attention Refocusing
(CAR) and Self-Attention Refocusing (SAR) losses [26] are employed to update
only regions of low fidelity, encouraging the model to focus on these hadr
samples. Finally, all regions are optimized for the last $0.3T$ iterations to
smoothly blend the high-fidelity regions with other regions.

##  4 Experiments

In this section, we provide an evaluation of our method and compare it with
state-of-the-art T2I models. Ablation experiments are conducted to demonstrate
the effectiveness of our divide-and-conquer approach in layout prediction and
layout-to-image generation stages.

###  4.1 Experiment Setup

####  4.1.1 Benchmarks

To evaluate the performance of spatial and numerical reasoning in the text-to-
image generation tasks, we utilize benchmarks HRS[2] and NSR-1K[15]. The HRS
dataset contains 3,000 prompts (labeled with objects’ names and specific
counts) for numerical relationships and 1,002 prompts (labeled with objects’
names and corresponding positions) for spatial relationships. The NSR-1K
dataset consists of 762 prompts for numerical relationships and 283 prompts
for spatial relationships, all labeled with names, counts, bounding boxes, and
ground truth images. We also utilize NSR-1K for quality evaluation. Compared
to NSR-1K, HRS exclusively employs natural language prompts, alongside a
larger quantity of objects per sample and more intricate spatial relations.
This elevates HRS as a more sophisticated and challenging benchmark.

####  4.1.2 Evaluation Metrics

We evaluate the images produced by our methods in terms of both quality and
fidelity.

(1) Quality To assess the quality of generated images, we use FID as the
metric[18]. Lower FID represents higher similarity between generated images
and ground truths images. We report FID in NSR-1K [15] as each prompt has a
corresponding ground truth image for FID calculation.

(2) Fidelity To evaluate the fidelity of the generated images, grounding
accuracy between layouts/images and the text prompt is calculated.
Particularly, YOLOv8 [20] is employed to obtain the bounding boxes of the
objects in the generated images. In counting, the number of bounding boxes is
compared to the ground truths to calculate the precision, recall, F1 score,
and accuracy [13, 16]. In spatial relationships, the relative position between
two bounding boxes is adopted following PaintSkills[11] and LayoutGPT [15].
Specifically, if the horizontal distance between two bounding boxes exceeds
the vertical distance, they are denoted as a left-right relationship.
Otherwise, they are denoted as a top-bottom relationship.

####  4.1.3 Baselines

Stable Diffusion [30], Attend-and-Excite[7], and Attention-Refocusing [26] are
employed for comparison. Stable Diffusion and Attend-and-Excite directly
generate images from the text prompt. Meanwhile, Attention-Refocusing employs
layout as an intermedium for image generation.

Table 1: Comparison of our method with baselines on HRS benchmark. Text in bold shows the best results of each metric.  Methods |  Numerical Layout Pre. Rec. F1 Acc. Image Pre. Rec. F1 Acc. |  Spatial Layout Image Acc. Acc.
---|---|---
Stable Diffusion 2.1[30] |  - - - - 73.97 53.93 62.38 16.20 |  - 7.96
Attend-and-Excite[7] |  - - - - 76.57 55.94 64.65 18.37 |  - 9.23
Attention-Refocusing[26] |  88.62 91.91 90.44 77.05 77.43 61.78 68.72 25.60 |  89.57 48.29
DivCon (Ours) |  94.17 89.53 91.80 80.73 79.13 63.53 70.48 29.62 |  91.87 53.87
Table 2: Comparison of our method with baselines on NSR-1K benchmark. Text in bold shows the best results of each metric.  Methods |  Numerical Layout Pre. Rec. F1 Acc. Image Pre. Rec. F1 Acc. |  Spatial Layout Image Acc. Acc.
---|---|---
Stable Diffusion 2.1[30] |  - - - - 78.94 75.42 77.41 33.99 |  - 22.97
Attend-and-Excite[7] |  - - - - 79.21 78.62 78.91 39.51 |  - 29.61
Attention-Refocusing[26] |  98.51 99.11 98.81 94.31 84.61 85.64 85.12 56.10 |  95.05 69.28
DivCon (Ours) |  98.47 98.35 98.41 94.39 85.31 87.54 86.41 55.80 |  96.11 72.95
Figure 6: Qualitative comparison of numerical reasoning between Stable Diffusion, Attend-and-excite, Attention -Refocusing, and our DivCon.  Figure 7: Qualitative comparison of spatial reasoning between Stable Diffusion, Attend-and-excite, Attention -Refocusing and our DivCon.  Table 3: FID results achieved by different methods in NSR-1K.  Methods |  FID ($\downarrow$)
---|---
Stable Diffusion 2.1[30] |  20.77
Attention-Refocusing[26] |  21.05
DivCon (Ours) |  20.89

###  4.2 Evaluation Results

####  4.2.1 Quantitative Results

We present the results on the HRS benchmark [2] in Tab. 1. As we can see,
DivCon outperforms previous T2I models by over 10% in numerical accuracy and
over 45% in spatial accuracy, demonstrating its superior numerical and spatial
reasoning performance. In addition, DivCon surpasses Attention Refocusing [26]
on most metrics. Specifically, our layout prediction improves numerical
accuracy by 3% and spatial accuracy by 2%, respectively. For image generation,
the F1 score and accuracy achieve 2% and 4% gains, while the spatial accuracy
is boosted by up to 6%. By dividing the T2I task into simple subtasks, our
DivCon can better understand the text prompt to achieve higher fidelity.

The results on the NSR-1K benchmark are presented in Tab. 2. It can be
observed that DivCon outperforms other T2I models by over 15% in numerical
accuracy and over 45% in spatial accuracy. As compared to Attention Refocusing
[26], our DivCon produces competitive results in terms of layout accuracy.
Meanwhile, DivCon achieves approximately 2% superiority in precision, recall,
and F1 score for numerical reasoning, and surpasses by nearly 4% in terms of
spatial accuracy.

Quantitative results in terms of image quality are shown in Tab. 3. As we can
see, our DivCon produces compatible FID scores as compared to stable diffusion
and outperforms attention-refocusing with notable margins. This demonstrates
that our approach is able to preserve image quality while enhancing prompt
fidelity.

####  4.2.2 Qualitative Results

Fig. 6 and Fig. 7 illustrate the qualitative comparison of DivCon and
baselines. In all cases, our DivCon can accurately predict the object
quantities and the spatial relations from the text prompts.

(1) Large Quantity of Objects. Given a prompt containing more than one object
category (Fig. 6), Stable Diffusion [30] and Attend-and-Excite [7] often
generate an insufficient quantity of objects, while Attention-Refocusing [26]
sometimes omits one category of objects or generates an excessive number of
objects. For instance, the prompt in Fig. 6(b) contains five spoons and four
oranges. However, Stable Diffusion and Attend-and-Excite cannot synthesize
sufficient spoons or oranges. Meanwhile, Attention-Refocusing generates an
excess number of objects, with spoons appearing incomplete due to missing
parts. As compared to these methods, our DivCon achieves superior numerical
accuracy.

(2) Complicated Spatial Relation. Given a prompt containing complex spatial
relationships among more than two object categories (Fig. 7, previous T2I
models struggle to generate correct spatial relations. Meanwhile, Attention-
Refocusing tends to produce distortion or blending artifacts around the
boundaries of objects. In Fig. 7 (b), Stable Diffusion and Attend-and-Excite
miss certain object categories, while Attention-Refocusing merges the color of
the banana with the person. In contrast, our DivCon inference accurate object
categories and their spatial relationships, exhibiting superior performance in
these challenging cases.

Figure 8: Comparison of layouts generated from Attention-Refocusing and our
DivCon.  Figure 9: Ablation study of our image generation.  Figure 10:
Comparison of transmit one or two latent representations.

###  4.3 Ablation Study

In this sub-section, we conduct ablation experiments to evaluate the
effectiveness of our divide-and-conquer approach in both layout prediction and
image generation. Results are presented in Tab. 4 and Attention-Refocusing
[26] is used as the baseline.

####  4.3.1 Divide-and-Conquer in Layout Prediction

To demonstrate the effectiveness of our divide-and-conquer approach in the
layout prediction stage, we develop a variant by conducting numerical &
spatial reasoning and bounding box prediction in a single step. As we can see,
our divide-and-conquer approach produces consistent improvements across all
metrics, with an increase of 4% in both numerical and spatial accuracy. We
further compare the layouts predicted by our DivCon and Attention-Refocusing
in Fig. 8. Compared to Attention-Refocusing, layouts generated by our DivCon
not only exhibit higher reasoning accuracy (Fig. 8 (a), (b)) but also are more
organized with less overlap (Fig. 8 (c), (d)).

####  4.3.2 Divide-and-Conquer in Image Generation

To study the effectiveness of our divide-and-conquer approach in the image
generation stage, we develop a variant to synthesize the image in a single
forward pass. It can be observed from Tab. 4 that the divide-and-conquer
approach facilitates our method to notable performance gains in terms of all
metrics. We further compare the results produced by our method using prompts
containing multiple objects in Fig. 9. As we can see, the divide-and-conquer
approach facilitates our DivCon to better reconstruct the objects, especially
for small-sized ones, while avoiding generating objects outside the designated
bounding boxes. For example, in Fig. 9(a), DivCon accurately reconstructs the
small objects. In Fig. 9(d), DivCon successfully avoids generating the dog
outside boxes.

Overall, with our divide-and-conquer approach in both layout prediction and
image generation stages, our DivCon produces the best accuracy and surpasses
Attention-Refocusing by approximately 2% on numerical f1, 4% on numerical
accuracy, and 6% on spatial accuracy.

####  4.3.3 Fractiles in Image Generation

During image generation of our DivCon, two fractiles (_i.e._ , $z_{0.3T}$ and
$z_{0.7T}$) are used to preserve “good enough” results in the first-round
generation. We conduct ablation experiments to study these two fractiles. The
second fractile $z_{0.7T}$ marks the end of optimization and is non-
adjustable. We observe that transmitting only $z_{0.7T}$ to the next phase
results in instability in image generation instability, specifically
manifesting as noticeable boundary demarcations of bounding boxes in the
images (Fig. 10).

We further compare the performance achieved by our model with different values
for the first fractile. Experiment results are presented in Tab. 5. It can be
observed that the model suffers a notable performance drop if only the second
fractile is employed. With an additional fractile, our DivCon yields a notable
improvement. Additionally, the selection of the first fractile’s value
exhibits a marginal impact on the efficacy of the model, with the 30%
threshold demonstrating optimal performance.

Table 4: Ablation study of our method’s image generation performance on HRS benchmark. "Layout w/DivCon": Layout prediction from LLMs with divide-and-conquer prompting. "Image w/DivCon": Layout-to-Image generation with divide-and-conquer strategy.  |  Layout w/DivCon |  Image w/DivCon |  Numerical Precision Recall F1 Acc. |  Spatial Acc.
---|---|---|---|---
(1) |  - |  - |  77.43 61.78 68.72 25.60 |  48.29
(2) |  ✓ |  - |  78.26 62.61 69.57 29.46 |  52.74
(3) |  - |  ✓ |  78.18 62.36 69.38 27.83 |  53.17
(4) |  ✓ |  ✓ |  79.13 63.53 70.48 29.62 |  53.87
Table 5: Ablation study of two fractiles used to preserve results in the first-round generation.  First Fractile |  Second Fractile |  Numerical Precision Recall F1 Acc. |  Spatial Acc.
---|---|---|---
- |  ✓ |  76.82 60.69 67.80 24.31 |  48.53
$0.1T$ |  ✓ |  78.44 61.61 69.01 29.12 |  52.30
$0.3T$ |  ✓ |  79.13 63.53 70.48 29.62 |  53.87
$0.5T$ |  ✓ |  78.13 63.25 69.90 28.71 |  51.46

##  5 Limitations and Discussions

As illustrated in Fig. 11, our method still struggles with generating objects
from certain pattern layouts. When the text prompt describes objects
overlapping in space, the diffusion model may fail to generate all objects
accurately. This limitation stems from the inherent capacity of layout-
conditioned image generation models to handle overlapping boxes. Despite these
challenges, our approach still demonstrates a reduced frequency of overlapping
in layout prediction compared to the base model.

Figure 11: Comparison of transmit one or two latent representations.

##  6 Conclusion

In this work, we introduce DivCon, a novel divide-and-conquer approach that
divides the text-to-image generation task into simple subtasks to achieve
higher image generation fidelity. Specifically, the layout prediction is
divided into two steps to conduct numerical & spatial reasoning and bounding
box prediction, respectively. Then, layout-to-image generation is achieved via
an iterative manner to synthesize objects in an easy-to-hard fashion.
Comprehensive experiments demonstrate that our DivCon outperforms existing
state-of-the-art grounded text-to-image models in terms of both image quality
and prompt fidelity.

## References

  * [1] Avrahami, O., Hayes, T., Gafni, O., Gupta, S., Taigman, Y., Parikh, D., Lischinski, D., Fried, O., Yin, X.: Spatext: Spatio-textual representation for controllable image generation. In: CVPR (2023)
  * [2] Bakr, E.M., Sun, P., Shen, X., Khan, F.F., Li, L.E., Elhoseiny, M.: Hrs-bench: Holistic, reliable and scalable benchmark for text-to-image models. In: ICCV. pp. 20041–20053 (2023)
  * [3] Balaji, Y., Nah, S., Huang, X., Vahdat, A., Song, J., Zhang, Q., Kreis, K., Aittala, M., Aila, T., Laine, S., Catanzaro, B., Karras, T., Liu, M.Y.: ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 (2022)
  * [4] Bar-Tal, O., Yariv, L., Lipman, Y., Dekel, T.: Multidiffusion: Fusing diffusion paths for controlled image generation (2023)
  * [5] Álvaro Barbero Jiménez: Mixture of diffusers for scene composition and high-resolution image generation. arXiv preprint arXiv:2302.02412 (2023)
  * [6] Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.H., Murphy, K., Freeman, W.T., Rubinstein, M., Yuanzhen, L., Dilip, K.: Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704 (2023)
  * [7] Chefer, H., Alaluf, Y., Vinker, Y., Wolf, L., Cohen-Or, D.: Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG) 42(4), 1–10 (2023)
  * [8] Chen, M., Laina, I., Vedaldi, A.: Training-free layout control with cross-attention guidance. arXiv preprint arXiv:2304.03373 (2023)
  * [9] Chen, M., Laina, I., Vedaldi, A.: Training-free layout control with cross-attention guidance. arXiv preprint arXiv:2304.03373 (2023)
  * [10] Chen, X., Liu, Y., Yang, Y., Yuan, J., You, Q., Liu, L.P., Yang, H.: Reason out your layout: Evoking the layout master from large language models for text-to-image synthesis. arXiv preprint arXiv:2311.17126 (2023)
  * [11] Cho, J., Zala, A., Bansal, M.: Dall-eval: Probing the reasoning skills and social biases of text-to-image generative transformers. arXiv preprint arXiv:2202.04053 (2022)
  * [12] Ding, M., Zheng, W., Hong, W., Tang, J.: Cogview2: Faster and better text-to-image generation via hierarchical transformers. arXiv preprint arXiv:2204.14217 (2022)
  * [13] El-Nouby, A., Sharma, S., Schulz, H., Hjelm, D., El Asri, L., Ebrahimi Kahou, S., Bengio, Y., Taylor, G.W.: Tell, draw, and repeat: Generating and modifying images based on continual linguistic instruction. In: ICCV (2019)
  * [14] Feng, W., He, X., Fu, T.J., Jampani, V., Akula, A., Narayana, P., Basu, S., Wang, X.E., Wang, W.Y.: Training-free structured diffusion guidance for compositional text-to-image synthesis. In: ICLR (2023)
  * [15] Feng, W., Zhu, W., jui Fu, T., Jampani, V., Akula, A., He, X., Basu, S., Wang, X.E., Wang, W.Y.: Layoutgpt: Compositional visual planning and generation with large language models. In: NeurIPS (2023)
  * [16] Fu, T.J., Wang, X.E., Grafton, S., Eckstein, M., Wang, W.Y.: Sscr: Iterative language-based image editing via self-supervised counterfactual reasoning. In: Conference on Empirical Methods in Natural Language Processing (EMNLP) (2020)
  * [17] Gafni, O., Polyak, A., Ashual, O., Sheynin, S., Parikh, D., Taigman, Y.: Make-a-scene: Scene-based text-to-image generation with human priors. In: ECCV. pp. 89–106. Springer (2022)
  * [18] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: NeurIPS. vol. 30 (2017)
  * [19] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: NeurIPS. pp. 6840–6851 (2020)
  * [20] Jocher, G., Chaurasia, A., Qiu, J.: Ultralytics YOLO (Jan 2023), https://github.com/ultralytics/ultralytics
  * [21] Kang, M., Zhu, J.Y., Zhang, R., Park, J., Shechtman, E., Paris, S., Park., T.: Scaling up gans for text-to-image synthesis. In: CVPR. pp. 10124–10134 (2023)
  * [22] Li, Y., Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., Li, C., Lee, Y.J.: Gligen: Open-set grounded text-to-image generation. In: CVPR (2023)
  * [23] Lian, L., Li, B., Yala, A., Darrell, T.: Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint arXiv:2305.13655 (2023)
  * [24] Ma, W.D.K., Lewis, J., Kleijn, W.B., Leung, T.: Directed diffusion: Direct control of object placement through attention guidance. arXiv preprint arXiv:2302.13153 (2023)
  * [25] Nichol, A.Q., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., Mcgrew, B., Sutskever, I., Chen, M.: GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In: ICML. pp. 16784–16804. PMLR (2022)
  * [26] Phung, Q., Ge, S., Huang, J.B.: Grounded text-to-image synthesis with attention refocusing. arXiv preprint arXiv:1905.00538 (2023)
  * [27] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748–8763. PMLR (2021)
  * [28] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 (2022)
  * [29] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I.: Zero-shot text-to-image generation. In: ICML. pp. 8821–8831. PMLR (2021)
  * [30] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR. pp. 10684–10695 (2022)
  * [31] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S.K.S., Ayan, B.K., Mahdavi, S.S., Lopes, R.G., Salimans, T., Ho, J., Fleet, D.J., Norouzi, M.: Photorealistic text-to-image diffusion models with deep language understanding. In: NeurIPS. vol. 35, pp. 36479–36494 (2022)
  * [32] Sauer, A., Karras, T., Laine, S., Geiger, A., Aila, T.: Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis. arXiv preprint arXiv:2301.09515 (2023)
  * [33] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. In: NeurIPS. vol. 35, pp. 24824–24837 (2022)
  * [34] Wu, Q., Liu, Y., Zhao, H., Bui, T., Lin, Z., Zhang, Y., Chang, S.: Harnessing the spatial-temporal attention of diffusion models for high-fidelity text-to-image synthesis. arXiv preprint arXiv:2304.03869 (2023)
  * [35] Xiao, G., Yin, T., Freeman, W.T., Durand, F., Han, S.: Fastcomposer: Tuning-free multi-subject image generation with localized attention. arXiv preprint arXiv:2305.10431 (2023)
  * [36] Xie, J., Li, Y., Huang, Y., Liu, H., Zhang, W., Zheng, Y., Shou, M.Z.: Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion. arXiv preprint arXiv:2307.10816 (2023)
  * [37] Yang, Z., Wang, J., Gan, Z., Li, L., Lin, K., Wu, C., Duan, N., Liu, Z., Liu, C., Zeng, M., et al.: Reco: Region-controlled text-to-image generation. arXiv preprint arXiv:2211.15518 (2022)
  * [38] Yu, J., Xu, Y., Koh, J.Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B.K., Hutchinson, B., Wei, H., Parekh, Z., Li, X., Zhang, H., Baldridge, J., Wu, Y.: Scaling autoregressive models for content-rich text-to-image generation. Transactions on Machine Learning Research (2022)
  * [39] Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: ICCV. pp. 3836–3847 (2023)


# RuCLIP - new models and experiments:
a technical report

Alex Shonenkov1, Andrey Kuznetsov13, Denis Dimitrov14, Tatyana Shavrina2,
Daniil Chesakov1,
Anastasia Maltseva1, Alena Fenogenova1, Igor Pavlov1, Anton Emelyanov1, Sergey
Markov2,
Daria Bakshandaeva1, Vera Shybaeva1 and Andrey Chertok1  1 Sber AI, 2 Sber
Devices, 3 Samara National Research University, 4 Lomonosov Moscow State
University
Email: {AVShonenkov, AVladimirKuznetsov<EMAIL_ADDRESS>
###### Abstract

In the report we propose six new implementations of ruCLIP model trained on
our 240M pairs. The accuracy results are compared with original CLIP model
with Ru-En translation (OPUS-MT) on 16 datasets from different domains. Our
best implementations outperform CLIP + OPUS-MT solution on most of the
datasets in few-show and zero-shot tasks. In the report we briefly describe
the implementations and concentrate on the conducted experiments. Inference
execution time comparison is also presented in the report.

###### Index Terms:

ruCLIP, zero-shot learning, few-shot learning

##  I Introduction

The problem of the lack of expensive labeled data for training new deep
architectures is becoming more and more acute every year. Due to the lack of
resources to perform large-scale data collection and labeling, scientists in
the modern world are moving towards the development of self-supervised
learning solutions that will allow them to receive information about new
objects, focusing on previously acquired knowledge.

Another problem is that deep learning models are often developed for a
specific domain (and one modality - text, code, sound, image or time series).
At the same time, as practice shows, human learning is multimodal in nature
(and teaching a person to solve a problem in one modality helps the process of
teaching the same person to solve some other problem in another modality). In
this regard, we assume that in the case of deep learning models, such an
effect can also take place: models working in specific modalities can extract
missing knowledge from other modalities or even when solving some specific
problems.

In 2021, language and visual transformers were especially actively developed,
which led to the emergence of new developments in the field of multimodal data
analysis. A striking example was the DALL-E[4] model for image synthesis from
text descriptions from OpenAI. We also successfully broke into this race of
transformers and could not ignore another useful application of language and
visual transformers, the CLIP (Contrastive Language - Image Pre-training)[3]
model from OpenAI, whose task is to determine the ’semantic’ proximity of
texts in natural (English) language and images. Successful training of the
model by the OpenAI team made it possible to solve such computer vision
problems as zero-shot classification and zero-shot object detection. As a
result, we set ourselves the goal of CLIP model Russification so that the
model can be used directly in Russian and bypass the use of translators in
conjunction with the original English CLIP model. We made the first steps in
this direction at the beginning of 2021 - we developed the ruCLIP Small model.
Now we want to describe six more versions of the ruCLIP model that we have
trained during this time.

##  II RuCLIP Models

We releases the following versions of the ruCLIP model, which differ in the
number of layers of the ViT encoder, the size of the patch used (14×14, 16×16,
32×32) and the sizes of input images — 224×224, 336×336 and 384×384 pixels.
The semantics of the model name is as follows:

  * •

ruCLIP Base [vit-base-patch16-224];

  * •

ruCLIP Base [vit-base-patch32-224];

  * •

ruCLIP Base [vit-base-patch32-384];

  * •

ruCLIP Large [vit-large-patch14-224];

  * •

ruCLIP Base [vit-base-patch16-384];

  * •

ruCLIP Large [vit-large-patch14-336].

All new versions of the ruCLIP model were trained on open datasets, as well as
on data from the Sber ecosystem. In total, we managed to collect about 240
million unique ’image-description’ pairs. We trained the model for 12 days on
the SberCloud ML Space platform and the Christofari Neo supercomputer using
distributed training on 256 Tesla A100 GPUs, which significantly exceeds the
resources spent during the first ruCLIP Small training. It is important to
emphasize that in the new versions, the batch size 32768 was used during
training, which corresponds to how the original CLIP model was trained (in the
ruCLIP Small version, the batch size was 16).

The new versions of the ruCLIP model are still based on two components:

  1. 1.

Image Encoder is an encoder that generates a vector representation of images.
The well-known ViT remains at the heart of our model.

  2. 2.

Text Encoder is an encoder that generates a vector representation of text
descriptions. Unlike the ruCLIP Small model, we did not use RuGPT3Small, but
took a text transformer with the following parameters:

     * •

Base versions:

       * –

Context Length: 77

       * –

Transformer Layers: 12

       * –

Transformer Width: 512

       * –

Transformer Heads: 8

     * •

Large versions:

       * –

Context Length: 77

       * –

Transformer Layers: 12

       * –

Transformer Width: 768

       * –

Transformer Heads: 12

##  III Training Procedure

###  III-A Data

As we mentioned earlier to train the ruCLIP models we used a specifically
collected dataset, consisting of 240M pairs of ’image-description’ pairs. It
should be noted that our dataset is one of the largest Russian-language
datasets containing ’image-description’ pairs, but in comparison to the
existing English-language datasets our set is far from the leaders of the list
in terms of volume, which can be seen in Table I.

TABLE I: Dataset size comparison Model | ruCLIP Small | ruCLIP | CLIP | ALIGN | BASIC
---|---|---|---|---|---
Num. of samples | 3M | 240M | 400M | 1.8B | 6.6B

The data volume leader belongs to Google with their ALIGN[1] and BASIC[2]
models. In the first of them, data is collected on the basis of Conceptual
Captions with simplified data filtering. In the second model, the dataset for
ALIGN is supplemented with a private JFT-3B dataset.

###  III-B Training Details

The training process was quite challenging, so we provide some details in this
section. First, we started with a model based on ViT-Large. At the first
stage, ruCLIP Large [vit-large-patch14-224] was trained from scratch based on
the collected dataset. This model was trained for 380K iterations. Further, by
changing the size of the input data to $336\times 336$ pixels and training for
another 15K iterations, the ruCLIP Large [vit-large-patch14-336] exclusive
model was obtained. As for the ViT-Base versions of the model, they were
trained by fine-tuning the English model for 140K iterations. As a result, the
ruCLIP Base [vit-base-patch16-224] and ruCLIP Base [vit-base-patch32-224]
models were obtained. By changing the size of the input images to $384\times
384$ pixels and performing fine-tuning for another 20K iterations, the ruCLIP
Base [vit-base-patch32-384] and ruCLIP Base [vit-base-patch16-384] exclusive
models were obtained.

The training process is visualized below in Fig. 1 as a dependency of the loss
value from the number of train iterations. Also, for each training process we
set a specific learning rate scheme (right column in Fig. 1).

Figure 1: Training process results for ruCLIP architectures.

##  IV Experiments

During the experiments, we pursued several goals. The first is to see how good
different ruCLIP models are on data of different nature, and the second is to
compare the quality of our solution with a combination of the OPUS-MT Ru-En
translator and the original CLIP model. In the following part of the section
we provide the results of our research.

###  IV-A Quality Estimation

The quality of the model is assessed using the standard cosine metric. Text
and image descriptions are transformed into their separate embeddings, then we
calculate the similarity and choose the best ”image-description” pair.

Further we take the largest of our open models ruCLIP Large [vit-large-
patch14-224] and check its quality on a number of ”image-description” pairs
used for evaluation in the original CLIP[3] paper. The cosine similarity
between text prompts and images for the ruCLIP Large model [vit-large-
patch14-224] is shown in Fig. 2.

Figure 2: Image and text cosine similarity comparison.

In Table IV-A we compare the implementations of the ruCLIP model within the
zero-shot classification accuracy for different domains datasets. The table
shows the results of comparing the first trained ruCLIP Small model, all 6
implementations of the ruCLIP model, the CLIP original model with the Ru-En
OPUS-MT translator, and the CLIP original model. In 8 out of 18 datasets, the
ruCLIP Large [vit-large-patch14-336] exclusive model shows the best result
among all compared, and in those where it loses, the difference is not
significant (perhaps, except for MNIST and FGVC Aircraft - there is still
something to work on hard). It is worth noting that on the MNIST dataset, even
the largest ALIGN and BASIC models cannot outperform CLIP in terms of quality.
For most datasets (11 out of 18), the models we trained outperform the CLIP
[vit-base-patch16-224] original + OPUS-MT model. The advantage of the CLIP
original model (right column) can be explained by the two times larger
training sample size (400M vs 240M).

TABLE II: Zero-shot classification accuracy for different models

|  ruCLIP Small |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---
rugpt3-small |  ruCLIP Base |  |  |  |  |  |  |  |
vit-base-patch32-224 |  ruCLIP Base |  |  |  |  |  |  |  |
vit-base-patch16-224 |  ruCLIP Large |  |  |  |  |  |  |  |
vit-large-patch14-224 |  ruCLIP Base |  |  |  |  |  |  |  |
vit-base-patch32-384 |  ruCLIP Large |  |  |  |  |  |  |  |
vit-large-patch14-336 |  |  |  |  |  |  |  |  |
exclusive |  ruCLIP Base |  |  |  |  |  |  |  |
vit-base-patch16-384 |  |  |  |  |  |  |  |  |
exclusive |  CLIP |  |  |  |  |  |  |  |
vit-base-patch16-224 |  |  |  |  |  |  |  |  |
\+ OPUS-MT |  CLIP |  |  |  |  |  |  |  |
vit-base-patch16-224 |  |  |  |  |  |  |  |  |
Food101, acc |  0.137 |  0.505 |  0.552 |  0.597 |  0.642 |  0.712 |  0.689 |  0.664 |  0.883
CIFAR10, acc |  0.808 |  0.818 |  0.810 |  0.878 |  0.862 |  0.906 |  0.845 |  0.859 |  0.893
CIFAR100, acc |  0.440 |  0.504 |  0.496 |  0.511 |  0.529 |  0.591 |  0.569 |  0.603 |  0.647
Birdsnap, acc |  0.036 |  0.115 |  0.117 |  0.172 |  0.161 |  0.213 |  0.195 |  0.126 |  0.396
SUN397, acc |  0.036 |  0.452 |  0.462 |  0.484 |  0.510 |  0.523 |  0.521 |  0.447 |  0.631
Stanford Cars, acc |  0.023 |  0.433 |  0.487 |  0.559 |  0.572 |  0.659 |  0.626 |  0.567 |  0.638
DTD, acc |  0.169 |  0.380 |  0.401 |  0.370 |  0.390 |  0.408 |  0.421 |  0.243 |  0.432
MNIST, acc |  0.137 |  0.447 |  0.464 |  0.337 |  0.404 |  0.242 |  0.478 |  0.559 |  0.559
STL10, acc |  0.910 |  0.932 |  0.932 |  0.934 |  0.946 |  0.956 |  0.964 |  0.967 |  0.970
PCam, acc |  0.484 |  0.501 |  0.505 |  0.520 |  0.506 |  0.554 |  0.501 |  0.603 |  0.573
CLEVR, acc |  0.104 |  0.148 |  0.128 |  0.152 |  0.188 |  0.142 |  0.132 |  0.240 |  0.240
Rendered SST2, acc |  0.483 |  0.489 |  0.527 |  0.529 |  0.508 |  0.539 |  0.525 |  0.484 |  0.484
ImageNet, acc |  – |  0.375 |  0.401 |  0.426 |  0.451 |  0.488 |  0.482 |  0.392 |  0.638
FGVC Aircraft, mean-per-class |  0.020 |  0.033 |  0.043 |  0.046 |  0.053 |  0.075 |  0.046 |  0.220 |  0.244
Oxford Pets, mean-per-class |  0.462 |  0.560 |  0.595 |  0.604 |  0.587 |  0.546 |  0.635 |  0.507 |  0.874
Caltech101, mean-per-class |  0.590 |  0.786 |  0.775 |  0.777 |  0.834 |  0.835 |  0.835 |  0.792 |  0.883
Flowers102, mean-per-class |  0.063 |  0.401 |  0.388 |  0.455 |  0.449 |  0.517 |  0.452 |  0.357 |  0.697
Hateful Memes, roc-auc |  0.527 |  0.564 |  0.516 |  0.530 |  0.537 |  0.519 |  0.543 |  0.579 |  0.589

In Fig. 3 and Fig. 4 we show the results of conducted experiments on zero-shot
and few-shot classification by 10 models on 16 datasets. Just like in the
original article, linear classifiers were trained on the features that CLIP
extracts for images using 1, 2, 4, 8, and 16 images for each class. This is a
fairly simple and understandable transfer learning approach, called a linear
probe, in which the trained model is used to extract features, and then the
”head” is trained on the required number of classes with a limited set of
images for each class. Since the features that are extracted from the CLIP
original and CLIP original + OPUS-MT models are the same, there is no separate
few-shot classification plot for CLIP original + OPUS-MT. We also calculated
the average few-shot graph for the best ruCLIP Large exclusive model without
taking into account three datasets - PCam, Oxford Pets and FGVC Aircraft, on
which the model loses significantly, and you can see (dotted line) that the
average quality even slightly exceeds ruCLIP Small. The asterisks show the
average zero-shot accuracy values for the models.

Figure 3: Average the results of zero-shot (asterisks) and few-shot (lines)
experiments for various models for all datasets used in the quality
assessment.

Figure 4: The results of the zero-shot (asterisks) and few-shot (lines)
experiments for various models for each of the datasets used in the quality
assessment.

It should be mentioned that when implementing ruCLIP Small, an Image Encoder
with frozen weights of the CLIP original model was used as the backbone and
two additional linear layers were added and trained. Due to the fact that only
the visual component of the model was used for the linear probe experiment,
the quality of ruCLIP Small expected to be very close to CLIP original. This
is why ruCLIP Small scores are high and outperform several architectures.

###  IV-B Inference Time Estimation

We did not limit experiments to the qualitative only, but also evaluated the
performance of our ruCLIP model implementations on the CIFAR100 dataset using
Nvidia-V100 GPU. The results of inference time are shown in Table IV-B.

TABLE III: Inference time (iter/sec) of the ruCLIP model implementations on
the CIFAR100 dataset

ruCLIP Base |  |  |  |  |
---|---|---|---|---|---
vit-b-patch32-224 |  ruCLIP Base |  |  |  |
|  ruCLIP Large |  |  |  |
|  ruCLIP Base |  |  |  |
|  ruCLIP Large |  |  |  |
vit-large-patch14-336 |  |  |  |  |
|  ruCLIP Base |  |  |  |
vit-base-patch16-384 |  |  |  |  |
| exclusive |  |  |  |
308.84 | 155.35 | 49.95 | 147.26 | 22.11 | 61.79

##  V Conclusion

We managed to train several different versions of ruCLIP, which on a number of
datasets successfully outperformed the original CLIP model with a Ru-Eng
translator. All the training was based on our dataset of 240M pairs and took
12 days on 256 Tesla GPU A100 of the Christofari Neo supercomputer. Extensive
research on datasets from various domains shown the applicability of the
ruCLIP model in zero-shot and few-shot classification problems. An example of
the ruCLIP model is presented in Fig. 5).

In the zero-shot and few-shot classification tasks, the ruCLIP model showed
lower quality in comparison to the original CLIP model due to the fact that a
smaller dataset was used for training our ruCLIP implementations. It should
also be noted that the quality of the models increases with the increase in
the volume of training samples, which is confirmed by the advantage of ALIGN
and BASIC over CLIP due to the significantly larger amount of data used.

Figure 5: RuCLIP model evaluation example.

## References

  * [1] Chao Jia et al.  “Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision”, 2021  arXiv:2102.05918 [cs.CV]
  * [2] Hieu Pham et al.  “Combined Scaling for Zero-shot Transfer Learning”, 2021  arXiv:2111.10050 [cs.LG]
  * [3] Alec Radford et al.  “Learning Transferable Visual Models From Natural Language Supervision”, 2021  arXiv:2103.00020 [cs.CV]
  * [4] Aditya Ramesh et al.  “Zero-Shot Text-to-Image Generation”, 2021  arXiv:2102.12092 [cs.CV]