11institutetext: CVC, Universitat Autonóma de Barcelona, Bellaterra 08193,
Spain 22institutetext: MICC, University of Florence, Florence 50134, Italy
22email<EMAIL_ADDRESS>
22email<EMAIL_ADDRESS>
# Multimodal Transformer for Comics Text-Cloze

Emanuele Vivoli corresponding authors1122 0000-0002-9971-8738 Joan Lafuente
Baeza ⋆ 11 0009-0007-9736-9048 Ernest Valveny Llobet 11 0000-0002-0368-9697
Dimosthenis Karatzas 11 0000-0001-8762-4454

###### Abstract

This work explores a closure task in comics, a medium where visual and textual
elements are intricately intertwined. Specifically, text-cloze refers to the
task of selecting the correct text to use in a comic panel, given its
neighbouring panels. Traditional methods based on recurrent neural networks,
have struggled with this task due to limited OCR accuracy and inherent model
limitations. We introduce a novel Multimodal Large Language Model (Multimodal-
LLM) architecture, specifically designed for text-cloze, achieving a 10%
improvement over existing state-of-the-art models in both its easy and hard
variants. Central to our approach is a Domain-Adapted ResNet-50 based visual
encoder, fine-tuned to the comics domain in a self-supervised manner using
SimCLR. This encoder delivers comparable results to more complex models with
just one-fifth of the parameters. Additionally, we release new OCR annotations
for this dataset, enhancing model input quality and resulting in another 1%
improvement. Finally, we extend the task to a generative format, establishing
new baselines and expanding the research possibilities in the field of comics
analysis.

###### Keywords:

comics panels text-cloze VL-T5SimCLR ResNet

##  1 Introduction

Figure 1: Two instances of the text-cloze task: a question panel (with masked
balloons) is provided together with three candidate answers. Three context
panels are provided to help choose the correct answer.

In the evolving landscape of multimodal data analysis, the study of comics
presents a unique challenge, intertwining visual and textual elements. This
fusion forms a distinct narrative structure, where the synergy between images
and text is complex and integral. This is particularly evident in the text-
cloze task, where models must predict missing text in a comic panel, given the
panel with obscured text (question panel) and the previous three panels
(context panels). The task is framed as a multiple-choice question, offering a
number of distractor texts and the correct text originally on the panel
(Figure 1).

Prior work in this area has mainly utilized recurrent neural networks [12].
Soykan et al. [23] underscored the impact of OCR accuracy on model
effectiveness, pointing out that subpar OCR transcriptions are a limiting
factor. However, as we show in this work, advancements in OCR technology,
while important, are insufficient by themselves to advance this task
significantly. Considering this, progress also necessitates enhancements in
the underlying model architecture.

In this work, addressing the text-cloze task in comics, we make the following
contributions:

  1. 1.

We introduce a novel Multimodal-LLM based architecture specifically designed
for the comics text-cloze task, outperforming existing models by 10% in both
easy and hard variants of the task.

  2. 2.

Compare various image representations and demonstrate that finetuning
ResNet-50 to the domain of the comic in a self-supervised manner (SimCLR)
yields comparable results to advanced Multimodal LLM image encoders, whilst
having one-fifth of the parameters.

  3. 3.

We release new OCR data (Textract) for the original dataset.

  4. 4.

We formulate a new version of the task addressing generation, marking a
significant step forward in this area of research.

##  2 Related works

###  2.1 Comics Tasks in Vision and Language

Comics, as a medium, is thought to be multimodal from its inception. It
consists of multiple panels on a page, each featuring text in balloons and
recurrent characters with varying appearances. This complexity is amplified by
animated objects and onomatopoeias, adding depth to the narrative. Among all
datasets available [11, 10, 8, 23, 9], the most common task is the detection
of the various elements just described. The differences in style from one
artist to another make difficult also the detection of text, a task that is
largely explored in Computer Vision [31, 33, 21]. As so, many works in the
field of Japanese comics (manga) propose tasks of onomatopoeia detection and
their linking when an onomatopoeia is split into multiple pieces across the
pages [15, 16, 1].

However, one important feature of comics is understanding the storyline, which
cannot emerge from training detectors. The understanding has been designed as
a collection of “closure tasks” in comics [17]. These tasks are designed to
probe the intricate relationship between text and images, given a context of 3
panels and the text in those. The key tasks include text-cloze, which focuses
on identifying the appropriate text for a given image (Figure 1); visual-
cloze, which involves selecting the matching image for a given text; and
character-coherence, which assesses the continuity of character actions across
panels.

Our research focuses on the text-cloze task, characterized by its unique
challenges. This task is observed to be demanding, as reflected in its
performance scores. In terms of computational approaches, earlier LSTM-based
methods [12, 1] have shown limited success in this area. This underscores the
task’s complexity, particularly in integrating visual and textual elements in
comics effectively.

###  2.2 Multimodal Large Language Models

Recent years have witnessed a surge in the prominence of Language Models,
marked by significant advancements [18, 19, 20, 24, 5, 25]. A pivotal
development in this domain has been the T5 model [20], which introduced a
unified framework for diverse NLP tasks, employing a text-to-text format.

Transformers have introduced substantial progress in text processing,
primarily by overcoming the limitations associated with long-term memory
states in previous models [26]. While speed enhancement is a notable benefit,
the key contribution lies in their ability to handle long-range dependencies
more effectively. In image processing, transformers have also shown remarkable
capabilities, as demonstrated by the Vision Transformer (ViT) which surpasses
conventional convolutional architectures in certain aspects of image analysis
[7]. The effectiveness of transformers in processing both textual and visual
data has naturally extended their application to multimodal contexts. This
progression has led to the development of various multimodal vision and
language Large Language Models (LLMs) [27, 6, 32, 3].

Building on these developments, our research applies the VL-T5 model [6] to
the text-cloze task in comics, selected for its balanced encoder-decoder
architecture. This choice addresses the limitations of encoder-only
architectures, as explored in our ablation studies. While models like
MiniGPT-4 [32], combining a Vision Transformer (ViT) with a linear layer and a
decoder-only LLM, were considered, their larger parameter scale prompted us to
defer their use to future work. The VL-T5 model, consisting of approximately
224M parameters (112M each in encoder and decoder), presents a more manageable
scale compared to the combined parameter sizes of larger models such as the
1.2B-parameter ViT from BLIP2 [14] and the smallest 7B-parameter Vicuna [5].
This approach allows us to maintain a focus on efficiency while tackling the
unique challenges of the text-cloze task in a multimodal context.

##  3 Method

This section provides an overview of the fundamental elements pertaining to
the architecture depicted in Figure 2. It begins with a formal definition of
the text-cloze task, followed by an explanation of the text and vision
pipelines. Each of these pipelines is composed of two key modules: the
extractor and the encoder.

Figure 2: Architecture of VL-T5 with text pipeline (OCR system extractor and
VL-T5 embeddings), and image pipeline (custom extractors and image encoders).

###  3.1 Task Definition

Let $P=\\{p_{1},p_{2},\cdots,p_{n}\\}$ represent a sequence of comic panels,
where each panel $p_{i}$ is composed of two modalities: the visual component
$I_{i}$ referring to the image representation of a panel, and the text
component $T_{i}$, which is the text that appears in the different balloons of
the panel. $T_{i}$ comprehends the text up to three balloons, linearised using
the order in which it is extracted by the OCR. The task is to predict the
occluded text $T_{\text{target}}$ for a given panel $p_{\text{target}}$,
utilizing the visual context $I_{target}^{\star}=M(I_{target})$ where $M$ is
the masking process and $I_{target}^{\star}$ is the result of masking the text
inside the panel image. For the task, three preceding panels are provided as
context. Finally, as for the original definition of the task [12], the
probability distribution is also conditioned on the possible answer options
$O=\\{O_{0},O_{1},\cdots,O_{n}\\}$ which includes the target text $O_{t}$ and
distractors text $O_{j}$ with $t,j\in[0,n]$, $j\neq t$, and $n$ is either $3$
or $10$, making it a multiple-choice question task. The final task consists of
predicting the index $i$ of the correct answer and not the correct answer
itself. The final task is expressed by the following equation:

$\hat{t}=\underset{i}{\mathrm{argmax}}\;P(T_{i}\mid
O,I_{\text{target}}^{\star},C)$

If the task is tackled as a generation task, where tokens are generated one
after the other, the previous formula can be expressed as:

$T_{\text{predicted}}=\bigoplus_{k=1}^{K}\underset{T_{k}}{\mathrm{argmax}}\;P(T_{k}\mid
T_{<k},I_{\text{target}}^{\star},C)$

where $T_{\text{predicted}}$ is the generated text sequence, $T_{k}$ is the
token generated at step $k$, $T_{<k}$ represents all previously generated
tokens, and $K$ is the length of the generated sequence. The context $C$
includes the content from the three preceding panels. The operation
$\bigoplus_{k=1}^{K}$ is used to denote the aggregation of individual tokens
into a continuous sequence.

###  3.2 Vision Pipeline

As can be seen from Figure 2, there are two blocks involved in creating the
image representation (right pipeline, purple): the Image Extractor and the
Image Encoder.

####  3.2.1 Extractor.

The Image Extractor module processes each panel image and employs one of two
approaches: either retaining the original image in its entirety or applying
region extraction to create a series of patches that depict the panel’s
contents. In the case of region extraction, these patches, representing
various objects within the panel, collectively form a comprehensive
representation of the panel. We use two different region extraction methods:

##### Faster R-CNN.

We utilize a convolutional-based object detector, Faster R-CNN [22], for
detecting and identifying objects in the panel images. Our implementation,
adapted from the Detectron2 [28] repository, employs a ResNet-101 backbone
trained on the Visual Genome dataset. This approach allows us to represent a
panel with the most confidently detected objects. The rationale behind this
object-focused representation stems from the intricate nature of comic scenes,
where a mere image-based representation might fail to capture the full
complexity of a comic panel.

##### Segment Anything Model (SAM).

As an alternative to the CNN-based method, we employ SAM, a transformer-based
foundational model specifically designed for segmentation tasks [13]. SAM
utilizes a ViT backbone along with a prompted masking light network for mask
generation. Post mask generation, the segmented objects are extracted using
their bounding boxes. The decision to use bounding boxes, as opposed to
polygonal masks, is twofold: it maintains consistency and avoids potential
alterations to object representations that could arise from cutting objects
and filling backgrounds with solid colors, which might introduce unwanted
noise.

#####

The details about Faster R-CNN and SAM hyperparameters are reported in Section
4.2. The elements in Fig. 2, denoted as
[$\\{img_{1}\\},\\{img_{2}\\},\\{img_{3}\\},\\{img_{4}\\}$], are either the
whole panel image or a collection of segmented elements from the panel with
one of the approaches described before.

####  3.2.2 Encoder.

The Encoder works to extract features from images, which can be sourced from
the entire panel or individual patches produced by the Extractor. In the case
of panel-level representation, this process yields a single token that
represents the entire image. Conversely, when representing the panel through
objects, a token is produced for each object. Subsequently, these images are
mapped into the VL-T5 space. To facilitate this, a linear projection layer is
employed, serving the essential purpose of aligning the output dimensions of
the Encoder with the input dimensions required by the VL-T5 model. We analyze
several alternatives for the architecture of the encoder.

##### ResNet.

A ResNet-50 backbone, pre-trained on ImageNet.

##### Comics ResNet.

The ResNet-50 model, initially trained on the ImageNet dataset, required
adjustments to effectively process comic book imagery, due to the significant
differences between these two visual domains. To bridge this gap, we adapted
ResNet-50 to better align with the unique features of comic images. This
adaptation was essential, particularly in the absence of labeled comic
datasets, leading us to adopt a self-supervised learning approach. Within this
framework, we chose the SimCLR framework [4], recognized for its efficacy in
domain adaptation through contrastive learning. This approach was applied in
two distinct ways: fine-tuning an existing ImageNet-trained ResNet-50 model
and training a new ResNet-50 model from scratch. The use of the SimCLR method
is particularly apt given the large volume of varied, unlabeled comic data
available. The primary goal is to enable the network to accurately recognize
and understand the visual nuances of comic imagery. Although SimCLR can be
applied to entire pages or panels, our focus was on the object level. This
decision was made because defining similar objects is relatively easier,
allowing the model to more effectively learn and represent objects with
similar visual characteristics close to each other. Instead, the concept of
what constitutes two similar pages or panels remains unclear when viewed at
varying levels of granularity. To this end, we have created a dataset of
around 730k objects extracted from comic images using the Segment Anything
Model (SAM), drawn from the first 250 books in our collection. This dataset
has been instrumental in training the two versions of the ResNet-50 model.
This training approach enabled the models to learn a feature space
meaningfully representing different comic elements, significantly enhancing
their feature extraction capabilities. More details on our methodology and
findings can be found in the supplementary materials.

##### BLIP2-Qformer.

The Bidirectional Language-Image Pre-training for Vision Transformer, or BLIP2
[14], is a model that has been pre-trained on a diverse set of images and
associated textual descriptions. Its design is inherently multimodal,
combining the strengths of Vision Transformers (ViT), and the new proposed
Q-former, all with language supervision. This pretraining enables
BLIP2-Qformer to understand complex visual scenes and their narrative
contexts, which is particularly advantageous for interpreting the rich visual
language of comics.

#####

In our study, we compare the above three encoders. ResNet-50, with less than a
third of the parameters of BLIP2-Qformer, serves as a reliable baseline. The
fine-tuned ResNet-50 aims to capture domain-specific nuances of comics,
potentially matching the performance of more complex models. BLIP2-Qformer,
distinguished by its extensive multimodal pre-training, is included for its
potential in interpreting complex visual narratives. This selection allows us
to evaluate the impact of model size and domain-specific tuning on
performance, as we will demonstrate in the experiments and results section.

###  3.3 Text pipeline

####  3.3.1 Extractor and encoder.

In our methodology, the text extraction process is fundamental, serving as the
initial step for handling textual information. We utilize Optical Character
Recognition (OCR) systems [2, 21] to retrieve text from comic panels. These
OCR systems are responsible for both detecting and recognizing text, and they
additionally determine the sequence in which the text appears within each
panel. OCR is instrumental in dealing with two types of text: contextual
images (C) and multiple-choice answer options (O). For context images, the
extracted text is used to enrich the text pipeline with explicit
representation, complementing the visual data processed through the vision
pipeline. We evaluated the efficacy of two OCR systems:

##### Original OCR

[12]: For the initial task definition, text extraction was performed using an
open-source OCR system. We have adopted these provided transcripts as a
benchmark, enabling us to compare with the original methodologies and assess
the impact of OCR quality by juxtaposing it with more advanced OCR
technologies.

##### Amazon Textract

111https://docs.aws.amazon.com/textract

: This is a high-precision, commercial OCR solution available through an API.
Its adaptability for general purposes enables us to test its effectiveness in
comic texts as well. As part of our contribution, we have generated new OCR
transcriptions for every panel in our dataset, which we will make available
for further research.

Figure 3: Representation of textboxes (balloon) separator tokens. Every group
of tokens referred to textbox (balloon) has two tokens prefix that represents
the context (light-green) and the textbox id (colored tokens). The orange
tokens are the balloon text tokens.

The process of extracting text involves identifying its position within each
panel and grouping it with the corresponding speech balloon. After extracting
text from the context panels, the VL-T5 tokenizer breaks down the text into a
series of individual tokens $T=\\{t_{1},t_{2},\cdots,t_{m}\\}$. These tokens
are then assigned to specific embeddings via the embedding layer.
Subsequently, tokens from context panels are combined and introduced with
custom learnable tokens at the beginning. An example of these custom tokens is
depicted in Figure 3. A similar approach is applied to the answer options.

###  3.4 Proposed model

We base our model on the VL-T5 architecture [6], a multimodal transformer
model that has been effectively employed in a variety of vision and language
tasks [30, 29]. VL-T5 extends the T5 model to integrate visual data, making it
suitable for multimodal tasks. In our adaptation of VL-T5 for the comic text-
cloze task, we maintain the core architecture but introduce task-specific
modifications, and add the prefix-token text-cloze. Our implementation of
VL-T5 features two distinct configurations: a classification setup and a
generation setup. In the classification setup, we either employ an encoder-
only approach, where the encoder’s final layer output is pooled into a single
token for classification within a $[0,n]$ range (refer to “pooling” in Fig.
2), or we use the full encoder-decoder model to generate the id of the
selected option. The generation setup, on the other hand, prompts the model to
produce the complete text of the option it deems correct.

##  4 Experiments and Results

Our model variants are built by combining different textual and visual
extractors and encoders. A series of experiments are run to establish the best
combination, as follows:

  * •

Visual Extractors comparison. Utilizing VL-T5 with the original OCR system and
various Image Extractors to establish which visual extractor works better to
represent the comic panels.

  * •

Visual Encoders comparison. After choosing the Image Extractor, benchmark the
vision encoder components within the VL-T5 framework, to explore the
synergistic effects of multimodal inputs.

  * •

Textual comparison. Employing VL-T5 with different OCR systems using the same
visual encoders, to gauge the impact of text quality on performance.

By employing these configurations, we aim to systematically dissect the
contributions of each component within the VL-T5 architecture. This
methodology allows us to evaluate the choice of image extractor and encoder,
the importance of high-quality OCR, and the overall effectiveness of the VL-T5
model in integrating visual and textual information for the comic text-cloze
task.

###  4.1 Data

Our experimental dataset, sourced from [12], encompasses the text-cloze task’s
training, validation, and test partitions across both easy and hard variants.
It includes the OCR of the text within balloons, images of complete comic
pages, and individual panels that have been automatically cropped.
Furthermore, we have utilized a state-of-the-art OCR system to generate an
updated OCR dataset, enhancing the text quality. Additionally, we have
accurately mapped the panel coordinates corresponding to the cropped panels
provided. A summarization of the dataset statistics used throughout our
experiments is provided in Table 1, offering an overview of the comprehensive
nature of the data.

Table 1: “Comics” dataset (left) and text-cloze task (right) statistics.

Number of objects
---
Books | 3,948
Pages | 198,657
Panels | 1,229,664
Textboxes | 2,498,657

text-cloze instances
---
Train | 142,952
Validation | 22,490
Test | 11,909

####  4.1.1 Easy vs. Hard variants.

The text-cloze task presents two distinct levels of difficulty: easy and hard.
The distinction between these levels lies in the source of the distractor
options. For the easy variant, distractors are drawn from a broad pool of
balloon texts across the dataset, whereas for the hard variant, distractors
are chosen from balloons on pages close to the target panel. This implies that
the distractors are more likely to be contextually similar to the correct
answer, thus demanding a more nuanced understanding of the context for
accurate prediction. In our experiments, each model is evaluated in both test
sets, regardless of the training data. We define tests as Independent and
Identically Distributed (iid) when the training and testing difficulties match
(easy with easy, hard with hard). Conversely, tests are categorized as Out Of
Distribution (ood) when there is a cross-evaluation (training on easy, testing
on hard, and vice versa).

###  4.2 Impact of Panel Representation

Our investigation into visual information extraction compares the efficacy of
global image representation (entire panel) versus cropped object
representation. We utilized two distinct neural networks for object detection
in this comparison: Faster R-CNN (from Detectron2) and SAM (Segment Anything
Model).

For object extraction with Faster R-CNN, we selected up to 10 objects per
panel based on confidence scores, with a minimum confidence threshold of 0.2.
With SAM, we limited the number to 36 higher stability score objects per panel
when more were available. We adjusted SAM’s default parameters to optimize
mask proposal and aggregation, setting points_per_side to 20, pred_iou_thresh
to 0.85, stability_score_thresh to 0.90, and crop_n_layers to 0.

Table 2: Performance comparison of different Image Extractors using the VL-T5 model, reported in Accuracy (%). White rows indicate full image (panel-level) representation, while gray rows denote object-level representation. model specifics $\downarrow$ | training data $\rightarrow$ |  |
---|---|---|---
Image | Text | easy | hard
Extractor | Encoder |  | Extractor
---
(OCR)
| easy
---
(iid)
| hard
---
(ood)
| easy
---
(ood)
| hard
---
(iid)
Faster R-CNN | CRN scratch | original | 77.28 | 62.52 | 75.42 | 68.13
SAM | CRN scratch | original | 75.69 | 61.39 | 75.80 | 67.32
✗ | CRN scratch | original | 78.55 | 64.84 | 77.00 | 69.91
SAM | CRN finetuned | original | 76.98 | 62.70 | 76.59 | 68.89
✗ | CRN finetuned | original | 78.60 | 64.74 | 78.18 | 69.28

Table 2 outlines the performance of two image encoders, “CRN scratch” and “CRN
finetuned” (where CRN stands for Comics ResNet-50 ). The former is a ResNet-50
trained from scratch using the self-supervised SimCLR method, and the latter,
a domain-adapted ResNet-50 pre-trained on ImageNet and fine-tuned with the
same SimCLR approach. Original text transcriptions were used for comparison.
Results suggest that models using the full panel for representation
outperformed those with object-level images, indicating that a comprehensive
view of the panel is advantageous for narrative understanding.

###  4.3 Effect of visual encoder

Focusing on panel-level representation, which yielded the best results in the
previous experiment, we evaluated different image encoders. As we can see from
Table 3, results indicate a clear advantage of “scratch” and “finetuned”
Domain Adapted ResNet-50 models over the standard ImageNet-trained version.
Despite being similar in performances, the “CRN scratch” and “CRN finetuned”
obtain the highest accuracy in the hard and easy ood test sets. However, when
compared with a bigger model such as ViT, our versions of ResNet-50 hold the
head with 3% to 4% in iid settings and above 3% for the ood. Notably, even
though BLIP2-Qformer emerged as the top-performing model with its $1.3B$ total
parameters in our ComicVT5 configuration, the custom models “CRN scratch” and
“CRN finetuned” achieve comparable accuracies with a fifth of the total
weights.

Table 3: Comparison of different Image Encoders using the VL-T5 model. The visual representation is panel-level and the OCR transcriptions the original version. Values are reported in terms of Accuracy (%). | model specifics $\downarrow$ | training data $\rightarrow$ | easy | hard
---|---|---|---|---
| Image | Text
| total
---
params
Extractor | Encoder |  | Extractor
---
(OCR)
| easy
---
(iid)
| hard
---
(ood)
| easy
---
(ood)
| hard
---
(iid)
250M | ✗ | ResNet-50 | original | 74.99 | 59.99 | 69.98 | 62.45
250M | ✗ | CRN scratch | original | 78.55 | 64.84 | 77.00 | 69.91
250M | ✗ | CRN finetuned | original | 78.60 | 64.74 | 78.18 | 69.28
313M | ✗ | ViT-b-16 | original | 74.34 | 58.56 | 74.60 | 64.97
1.3B | ✗ | BLIP2-Qformer | original | 79.44 | 64.08 | 78.63 | 70.38

###  4.4 Effect of text extractor

The quality of the OCR-generated text is a crucial element in our text-cloze
task, with poor text quality potentially leading to narrative
misinterpretations. An analysis of the original OCR output revealed numerous
errors, such as frequent misinterpretations of the word ”amsmet” in various
contexts, and minor errors like single-word mistakes or punctuation issues
significantly altering the intended meaning. Consequently, we hypothesized
that using updated OCR technology for context text would improve narrative
understanding and model performance. Therefore, we incorporated Amazon
Textract as our new generation OCR, with comparative transcription examples
available in the supplementary materials.

Our investigation included a preliminary experiment without OCR context text.
As Table 4 illustrates, the significance of OCR was evident across all image
encoders. Subsequently, we assessed the impact of employing the new generation
OCR. This change resulted in enhanced performance for the ”easy” task with the
Comics ResNet-50 as the image encoder, but not with BLIP2-Qformer , indicating
its lesser reliance on context text. For more challenging tasks, both models
showed improved performance, underscoring the heightened importance of context
panel text in complex scenarios. These results, reflecting the variation in
performance with different OCR inputs, underscore the critical influence of
OCR quality on model effectiveness.

Table 4: Comparison between no OCR (pixel-only setting), original OCR, and new generation OCR. Values are reported in terms of Accuracy (%). Gray rows correspond to new generation OCR. model specifics $\downarrow$ | training data $\rightarrow$ |  |
---|---|---|---
Image | Text | easy | hard
Extractor |  | Encoder
---
| Extractor
---
(OCR)
| easy
---
(iid)
| hard
---
(ood)
| easy
---
(ood)
| hard
---
(iid)
✗ | CRN scratch | ✗ | 67.70 | 64.24 | 67.94 | 65.62
✗ | CRN scratch | original | 78.55 | 64.84 | 77.00 | 69.91
✗ | CRN scratch | textract | 79.1$\uparrow$ | 67.3$\uparrow$ | 77.94$\uparrow$ | 70.46$\uparrow$
✗ | BLIP2-Qformer | ✗ | 65.72 | 61.66 | 69.46 | 66.13
✗ | BLIP2-Qformer | original | 79.44 | 64.08 | 78.63 | 70.38
✗ | BLIP2-Qformer | textract | 78.28$\downarrow$ | 66.23$\uparrow$ | 78.63$=$ | 71.31$\uparrow$

###  4.5 Comparison with other methods

Up to now, we have explored many configurations of the proposed ComicsVT5
architecture defining the top-performing model as the combination of panel-
level features, BLIP2-Qformer as Image Encoder, trained with the new
generation OCR data (called ComicVT5-blip). The second best model, with just a
fifth of the previous model’s parameters, is the Comic Domain-Adapted
ResNet-50 as Image Encoder, trained from scratch with SimCLR framework on
comics (indicated as ComicVT5-scratch). We compare our ComicVT5 models with
the previous state-of-the-art model in Table 5. As we can see from the table,
we obtain up to a 10% improvement over previous models which use original OCR
and recurrent network architecture [12]. Human benchmarks, cited from prior
work [12], show 84% accuracy on hard tasks and an assumed 100% on easier ones
(“trivial”), serving as an approximate upper bound.

Table 5: Comparison of our models with previous state-of-the-art. |  | training data
---|---|---
|  | easy | hard
Models | params |  | easy
---
(iid)
| hard
---
(ood)
| easy
---
(ood)
| hard
---
(iid)
random | - | 33.3 | 33.3 | 33.3 | 33.3
iyyer [12] | 145M | 68.6 | - | - | 61.0
ComicVT5-scratch | 250M | 79.1 | 67.3 | 77.94 | 70.46
ComicVT5-blip | 1.3B | 78.28 | 66.23 | 78.63 | 71.31
human* | - | 100 | 84 | 100 | 84

###  4.6 Qualitative Analysis

Our qualitative analysis examines the model’s predictions for both easy and
hard tasks. Figure 4 showcases three scenarios, each illustrating a different
aspect of our model’s capabilities. The figure includes the context panels
used by the model (the first three), the query panel with its text masked (a
grey translucent box is used for visualization instead of an actual black
box), and the possible text choices (with the correct answer in green). The
model’s selection is represented by  ✗ (error) and  ✓ (correct).

Figure 4: Examples of our ComicVT5-blip model. For every row: (i) context
(first three panels), (ii) query panel (fourth panel) and (iii) options (on
the right). For the generative text-cloze task the model is trained with text
generation objective.

In the first instance depicted in Figure 4, our model accurately navigates a
“hard” task sample, successfully identifying the correct narrative
continuation amidst distractors sourced from adjacent pages. The first
distractor is rendered implausible by its narrative incoherence, notably the
incongruous inclusion of ”robots”, while the third distractor’s lack of
logical sentence structure and irrelevant mention of ”girl” renders it
infeasible. The second instance in the figure exemplifies a prevalent error
trend stemming from OCR inaccuracies, where flawed or incomplete
transcriptions significantly alter the intended meaning, leading to erroneous
selections by the model. The third instance showcases the model’s proficiency
in generating plausible, though not exact, narrative continuations, as
evidenced by a response that, while divergent from the original text,
maintains contextual relevance. This highlights the model’s capability to
generate contextually coherent but not necessarily accurate predictions.

These examples collectively demonstrate the various factors influencing the
performance of our model, including the accuracy of OCR transcriptions and the
model’s contextual understanding in the text-cloze task.

####  4.6.1 Further Experimental Details

Additional exploratory studies, including those focusing on encoder-only
architectures and the incorporation of increased distractors, have been
detailed in the supplementary material due to space constraints in the main
paper.

###  4.7 Dialogues generation

To add complexity and realism to the task, we propose a variation of text-
cloze which involves dialogue generation. The new task aims at generating the
target dialogue either from the three possible sentences provided as input or
without any option provided.

####  4.7.1 With provided options.

For evaluating model performance, we utilized various natural language
processing metrics, reported in the supplementary material. At inference time,
for text generation, we employed beam search with 2 beams and a limit of 35
tokens, avoiding random sampling since the target output has been used as
input of the model. To facilitate comparison with earlier methods we also
measured accuracy, as shown in Table 6. To calculate the accuracy, we
identified the model prediction among the three options that most closely
matched the meaning of the generated text. More in detail: (i) we generated
embeddings for each sentence using a transformer
encoder222https://huggingface.co/sentence-transformers/all-mpnet-base-v1 (a
model fine-tuned on 1 billion sentence pairs), which provides contextualized
embeddings for each sentence token; (ii) we compute a singular embedding for
each sentence through average pooling; (iii) we compared these embeddings
(generated one and the three options) using cosine similarity; (iv) the most
similar option sentence is chosen for calculating accuracy. Qualitatively, as
reported in the fourth row of Figure 4, it’s apparent that the model has
learned to generate the correct answers with minor modifications, such as
eliminating extra white spaces before special characters (e.g., !, ?).

Table 6: Comparison of different VL-T5 Multimodal architecture. The “Input options” column specifies whether the model gets the three options as input. The “Output task” can be either classification (cls) or generation (gen). Gray rows correspond to generative models whose accuracy has been calculated with sentence similarity. |  |  | training data
---|---|---|---
|  |  | easy | hard
Models |  | Input
---
options
| Output
---
task
| easy
---
(iid)
| hard
---
(ood)
| easy
---
(ood)
| hard
---
(iid)
ComicVT5-scratch | ✓ | cls | 79.1 | 67.35 | 77.04 | 70.46
ComicVT5-scratch | ✓ | gen | 72.84 | 62.12 | 72.95 | 66.32
ComicVT5-scratch | ✗ | gen | 48.12 | 44.80 | 48.12 | 44.80
ComicVT5-blip | ✓ | cls | 78.28 | 66.23 | 78.63 | 71.31
ComicVT5-blip | ✓ | gen | 74.58 | 62.14 | 73.33 | 67.06
ComicVT5-blip | ✗ | gen | 49.93 | 46.77 | 49.93 | 46.77

####  4.7.2 Without input options.

In our exploration of more sophisticated dialogue generation for comics, we
experimented without predefined answer options, focusing solely on context
text and panel images. Details and examples of the model-generated dialogues
are provided in the supplementary material. Evaluating this approach required
a shift from traditional natural language metrics due to the challenge of
having only a single reference dialogue. To address this, we assessed the
models using accuracy, similar to our previous methods and also experimented
with feeding different proportions of the dialogue’s starting tokens into the
transformer decoder. For text generation, we utilized sampling with a
temperature of 0.6 and a maximum length of 35 tokens. Results indicate a
decline in accuracy compared to our previous dialogue generation experiments,
reflecting the increased complexity of this task. Finally, we evaluated NLP-
specific metrics for the generation task (Rouge, Bleu, Meteor, and Rouge-l) in
both generation settings (with and without input options). The detailed
results of these experiments, including the variations in performance with
different percentages of target decoder tokens, are presented in the
supplementary material.

##  5 Conclusions

In this paper, we introduced a novel Multimodal Large Language Model
architecture (Multimodal-LLM) tailored for the comics text-cloze task. Our
investigation spanned various image representations and OCR technologies. We
established that a self-supervised domain adaptation of the ResNet
architecture to comic imagery achieves performance on par with recent
Multimodal LLM image encoders, with significantly fewer parameters. This
Domain-Adapted image encoder not only achieves state-of-the-art results in the
encoder-decoder framework but also excels as a standalone component.
Furthermore, we identified that the conventional text-cloze task, also thanks
to accurate OCR, can be relatively straightforward. To address this, we
designed more challenging variants of the task by introducing a generative
approach. Our extensive results and ablations contribute valuable benchmarks
for advanced prediction tasks in this field.

To foster further research and facilitate the development of novel
architectures, we are releasing our comprehensive OCR dataset and model code,
underlining our commitment to research reproducibility and innovation in
comic-based language modeling.

## References

  * [1] Baek, J., Matsui, Y., Aizawa, K.: COO/ Comic Onomatopoeia Dataset for Recognizing Arbitrary or Truncated Texts. arXiv (Jul 2022). https://doi.org/10.48550/arXiv.2207.04675
  * [2] Baek, Y., Lee, B., Han, D., Yun, S., Lee, H.: Character Region Awareness for Text Detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 9357–9366. IEEE, Long Beach, CA, USA (Jun 2019). https://doi.org/10.1109/CVPR.2019.00959
  * [3] Chen, J., Zhu, D., Shen, X., Li, X., Liu, Z., Zhang, P., Krishnamoorthi, R., Chandra, V., Xiong, Y., Elhoseiny, M.: MiniGPT-v2: Large language model as a unified interface for vision-language multi-task learning (Nov 2023)
  * [4] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A Simple Framework for Contrastive Learning of Visual Representations (Jun 2020)
  * [5] Chiang, W.L., Zhuohan, L., Zi, L., Ying, S., Zhanghao, W., Hao, Z., Lianmin, Z., Siyan, Z., Yonghao, Z., Gonzales, J.E., Stoica, I., Xing, E.P.: Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. https://lmsys.org/blog/2023-03-30-vicuna (Jun 2023)
  * [6] Cho, J., Lei, J., Tan, H., Bansal, M.: Unifying Vision-and-Language Tasks via Text Generation (May 2021). https://doi.org/10.48550/arXiv.2102.02779
  * [7] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: ViT: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (Jun 2021). https://doi.org/10.48550/arXiv.2010.11929
  * [8] Dunst, A., Hartel, R., Laubrock, J.: The Graphic Narrative Corpus (GNC): Design, Annotation, and Analysis for the Digital Humanities. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR). vol. 03, pp. 15–20 (Nov 2017). https://doi.org/10.1109/ICDAR.2017.286
  * [9] Dutta, A., Biswas, S., Das, A.K.: BCBId: First Bangla comic dataset and its applications. International Journal on Document Analysis and Recognition (IJDAR) 25(4), 265–279 (Dec 2022). https://doi.org/10.1007/s10032-022-00412-9
  * [10] Fujimoto, A., Ogawa, T., Yamamoto, K., Matsui, Y., Yamasaki, T., Aizawa, K.: Manga109 dataset and creation of metadata. In: Proceedings of the 1st International Workshop on coMics ANalysis, Processing and Understanding (Manpu). pp. 1–5 (Dec 2016). https://doi.org/10.1145/3011549.3011551
  * [11] Guérin, C., Rigaud, C., Mercier, A., Ammar-Boudjelal, F., Bertet, K., Bouju, A., Burie, J.C., Louis, G., Ogier, J.M., Revel, A.: eBDtheque: A representative database of comics. In: 2013 12th International Conference on Document Analysis and Recognition. pp. 1145–1149 (Aug 2013). https://doi.org/10.1109/ICDAR.2013.232
  * [12] Iyyer, M., Manjunatha, V., Guha, A., Vyas, Y., Boyd-Graber, J., Daumé III, H., Davis, L.: The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives. In: Proc. - IEEE Conf. Comput. Vis. Pattern Recognit, CVPR. vol. 2017-January, pp. 6478–6487. arXiv (May 2017). https://doi.org/10.1109/CVPR.2017.686
  * [13] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollár, P., Girshick, R.: SAM: Segment Anything (Apr 2023). https://doi.org/10.48550/arXiv.2304.02643
  * [14] Li, J., Li, D., Savarese, S., Hoi, S.: BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (Jan 2023)
  * [15] Louis, J.B., Burie, J.C.: Detection of Buried Complex Text. Case of Onomatopoeia in Comics Books. In: Coustaty, M., Fornés, A. (eds.) Document Analysis and Recognition – ICDAR 2023 Workshops. pp. 177–191. Lecture Notes in Computer Science, Springer Nature Switzerland, Cham (2023). https://doi.org/10.1007/978-3-031-41498-5
  * [16] Louis, J.B., Burie, J.C., Revel, A.: Can Deep Learning Approaches Detect Complex Text? Case of Onomatopoeia in Comics Albums. In: Rousseau, J.J., Kapralos, B. (eds.) Pattern Recognition, Computer Vision, and Image Processing. ICPR 2022 International Workshops and Challenges. pp. 48–60. Lecture Notes in Computer Science, Springer Nature Switzerland, Cham (2023). https://doi.org/10.1007/978-3-031-37742-6
  * [17] McCloud, S.: Understanding Comics: The Invisible Art. IEEE Transactions on Professional Communication 41(1), 66–69 (Mar 1998). https://doi.org/10.1109/TPC.1998.661632
  * [18] Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving Language Understanding by Generative Pre-Training (Jun 2018)
  * [19] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language Models are Unsupervised Multitask Learners (2019)
  * [20] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Jul 2020)
  * [21] Raisi, Z., Naiel, M.A., Fieguth, P., Wardell, S., Zelek, J.: Text Detection and Recognition in the Wild: A Review (Jun 2020). https://doi.org/10.48550/arXiv.2006.04305
  * [22] Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks (Jan 2016). https://doi.org/10.48550/arXiv.1506.01497
  * [23] Soykan, G., Yuret, D., Sezgin, T.M.: A Comprehensive Gold Standard and Benchmark for Comics Text Detection and Recognition (2022). https://doi.org/10.48550/ARXIV.2212.14674
  * [24] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., Lample, G.: LLaMA: Open and Efficient Foundation Language Models (Feb 2023). https://doi.org/10.48550/arXiv.2302.13971
  * [25] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C.C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P.S., Lachaux, M.A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E.M., Subramanian, R., Tan, X.E., Tang, B., Taylor, R., Williams, A., Kuan, J.X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., Scialom, T.: Llama 2: Open Foundation and Fine-Tuned Chat Models (Jul 2023). https://doi.org/10.48550/arXiv.2307.09288
  * [26] Vaswani, Ashish, Shazeer, Noam, Parmar, Niki, Uszkoreit, Jakob, Jones, Llion, Gomez, Aidan N., Kaiser, Lukasz, Polosukhin, Illia: Attention Is All You Need (Dec 2017). https://doi.org/10.48550/arXiv.1706.03762
  * [27] Wang, J., Yang, Z., Hu, X., Li, L., Lin, K., Gan, Z., Liu, Z., Liu, C., Wang, L.: GIT: A Generative Image-to-text Transformer for Vision and Language (Dec 2022)
  * [28] Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R.: Detectron2. https://github.com/facebookresearch/detectron2 (2019)
  * [29] Yang, Z., Gan, Z., Wang, J., Hu, X., Ahmed, F., Liu, Z., Lu, Y., Wang, L.: Crossing the Format Boundary of Text and Boxes: Towards Unified Vision-Language Modeling. ArXiv (2021)
  * [30] Ye, F., Long, Y., Feng, F., Wang, X.: Whether you can locate or not? Interactive Referring Expression Generation. Proceedings of the 31st ACM International Conference on Multimedia pp. 4697–4706 (Oct 2023). https://doi.org/10.1145/3581783.3612214
  * [31] Ye, Q., Doermann, D.: Text Detection and Recognition in Imagery: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 37(7), 1480–1500 (Jul 2015). https://doi.org/10.1109/TPAMI.2014.2366765
  * [32] Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models (Apr 2023). https://doi.org/10.48550/arXiv.2304.10592
  * [33] Zhu, Y., Yao, C., Bai, X.: Scene text detection and recognition: Recent advances and future trends. Frontiers of Computer Science 10(1), 19–36 (Feb 2016). https://doi.org/10.1007/s11704-015-4488-0

Multimodal Transformer for Comics Text-Cloze Supplementary Materials

## Structure overview

This supplementary material complements our main paper by delving deeper into
specific aspects of our research that were not extensively covered due to
length constraints. The content is organized as follows: in Section 6 we
present a nuanced analysis of the Comics domain-adapted ResNet, including
t-SNE visualizations and a detailed exploration of some clusters. In Section
7, we illustrate a comparison between the original OCR and the advanced new-
generation OCR used in our study. In Section 8, we present two different
experiment results: (i) an investigation of the impact of integrating more
distractors during the training phase; and (ii) the assessment of performance
differences between the encoder-only and encoder-decoder versions of our
architecture. Finally, in Section 9 we provide an in-depth analysis
(quantitative and qualitative results) of the proposed generative text-cloze
task.

##  6 Analysis of Comics Domain-Adapted ResNet

Figure 5: Reduced dimensional distribution of test objects via t-SNE on Comics
ResNet, featuring K-means detected clusters. Figure 6: Representative objects
from each identified cluster in Figure 5, maintaining consistent cluster
labeling.

In Figure 5, we present a visual representation of 10,000 test objects
analyzed through a ResNet-50;trained from the ground up using SimCLR. To
achieve a dimensional reduction from 2048 to 2, we employed the t-SNE
algorithm. This figure also delineates 8 clusters, ascertained using the
K-means clustering technique. Figure 6 displays a selection of five objects
from each cluster identified in Figure 5, adhering to the same cluster
numbering as indicated in the figure’s legend. Clusters 2, 4, and 6
predominantly include segmented balloons, words, and letters, respectively.
While Cluster 3 exhibits a less pronounced trend, it predominantly features
faces. The other clusters group objects with visual similarities, though they
lack the distinct object type trends observed in the earlier mentioned
clusters.

##  7 Comparison of OCR Performance

Figure 7 showcases examples contrasting the original balloon image with
transcriptions from two different OCR technologies. The comparison clearly
demonstrates the superior quality of the new generation OCR (shown on the
right).

Figure 7: Contrast between the original OCR and Amazon Textract
transcriptions.

##  8 Ablation Studies

###  8.1 Exploring Number of distractors

In an effort to increase the task’s complexity, we introduced a revision of
the task where the number of distractor choices was expanded from two to nine.
This modification aimed to create a more challenging environment for the
model’s decision-making process. We constructed two variants: an “easy”
version with distractors chosen from the entire comic dataset, and a “hard”
version with distractors selected from pages near the target, heightening the
presence of contextually significant distractors. These tasks’ versions follow
the same recipe as the original sets. The model’s performance in this
heightened complexity is detailed in Table 7.

Table 7: Model performance when trained and tested with ten distractors. model specifics $\downarrow$ | training data $\rightarrow$ | easy | hard
---|---|---|---
Image | Text
Extractor | Encoder |  | Extractor
---
(OCR)
| easy
---
(iid)
| hard
---
(ood)
| easy
---
(ood)
| hard
---
(iid)
✗ | CRN scratch | original | 73.61 | 37.10 | 67.15 | 56.26

Notably, the task difficulty rises with more distractors, especially within
the “hard” setting where the model is tasked with distinguishing finer
narrative details. The data reflects a significant enhancement in task
complexity with more distractors, while also showing that in the “easy”
variant, the model’s performance remains relatively stable even with the
expanded distractor pool.

###  8.2 Comparing Architectural Impact

In our study, we investigated the performance impact of utilizing a
transformer decoder compared to a Multilayer Perceptron (MLP) classification
head. To this end, we modified the VL-T5 encoder-decoder architecture by
incorporating an MLP head that classifies among three options, following the
average pooling of the encoder’s output tokens. Table 8 presents the results
of this ablation study.

Table 8: Comparison of different ComicVT5 Multimodal architecture. Gray rows correspond to VL-T5 encoder-decoder (standard VL-T5 ) while white rows to encoder-only architecture. |  |  | training data
---|---|---|---
|  |  | easy | hard
Models | Params |  | Multimodal
---
architecture
| easy
---
(iid)
| hard
---
(ood)
| easy
---
(ood)
| hard
---
(iid)
ComicVT5-scratch | 137M | enc | 77.37 | 65.72 | 77.38 | 69.95
ComicVT5-scratch | 250M | enc-dec | 79.1 | 67.35 | 77.04 | 70.46
ComicVT5-blip | 1.2B | enc | 76.63 | 63.61 | 77.23 | 70.69
ComicVT5-blip | 1.3B | enc-dec | 78.28 | 66.23 | 78.63 | 71.31

It reveals a marginal performance reduction when omitting the VL-T5 decoder,
despite the encoder-only variant having fewer learnable parameters. On the
other hand, this also suggests the complete encoder-decoder framework offers a
slight advantage, especially for the more intricate ’hard’ version of the
task.

##  9 Generative text-cloze Task

###  9.1 Quantitative Results

Table 9: Metrics for the dialogue generation models giving the three possible answers of the text-cloze task in the input of the encoder. |  | easy (iid) |  |
---|---|---|---|---
Models | Params | bleu1 | bleu4 | rouge1 | rouge2 | meteor | rouge-l
ComicVT5-scratch | 250M | 0.6543 | 0.6198 | 0.7216 | 0.7075 | 0.6699 | 0.7203
ComicVT5-blip | 1.3B | 0.6663 | 0.6334 | 0.7380 | 0.7249 | 0.6854 | 0.7370
|  | easy (ood) |  |
|  | bleu1 | bleu4 | rouge1 | rouge2 | meteor | rouge-l
ComicVT5-scratch | 250M | 0.6525 | 0.6181 | 0.7227 | 0.7085 | 0.6702 | 0.7216
ComicVT5-blip | 1.3B | 0.6599 | 0.6249 | 0.7267 | 0.7120 | 0.6740 | 0.7251
|  | hard (ood) |  |
|  | bleu1 | bleu4 | rouge1 | rouge2 | meteor | rouge-l
ComicVT5-scratch | 250M | 0.5834 | 0.5439 | 0.6268 | 0.6039 | 0.5808 | 0.6243
ComicVT5-blip | 1.3B | 0.5718 | 0.5329 | 0.6274 | 0.6044 | 0.5817 | 0.6246
|  | hard (iid) |  |
|  | bleu1 | bleu4 | rouge1 | rouge2 | meteor | rouge-l
ComicVT5-scratch | 250M | 0.6130 | 0.5746 | 0.6657 | 0.6450 | 0.6169 | 0.6633
ComicVT5-blip | 1.3B | 0.6131 | 0.5738 | 0.6721 | 0.6520 | 0.6238 | 0.6706

Table 9 displays the natural language processing (NLP) metrics for dialogues
generated with the text-cloze; task’s three options fed into the transformer
encoder. The results showcase the proficient performance of the two models
trained for this task in accurately recalling the correct dialogue from the
three options.

Table 10: Metrics for the dialogue generation models without seeing any of the possible answers of text-cloze task in the encoder input. The percentage specified for each evaluation corresponds to the percentage of the target tokens used on the transformer decoder, these tokens are not considered on the evaluation. |  |  | 30% |
---|---|---|---|---
Models | Params | bleu1 | bleu4 | rouge1 | rouge2 | meteor | rouge-l
ComicVT5-scratch | 250M | 0.1600 | 0.0038 | 0.1783 | 0.0302 | 0.1332 | 0.1716
ComicVT5-blip | 1.3B | 0.1577 | 0.0044 | 0.1777 | 0.0310 | 0.1336 | 0.1707
|  |  | 50% |
|  | bleu1 | bleu4 | rouge1 | rouge2 | meteor | rouge-l
ComicVT5-scratch | 250M | 0.1717 | 0.0035 | 0.1983 | 0.0329 | 0.1501 | 0.1946
ComicVT5-blip | 1.3B | 0.1699 | 0.0032 | 0.1978 | 0.0337 | 0.1491 | 0.1939
|  |  | 75% |
|  | bleu1 | bleu4 | rouge1 | rouge2 | meteor | rouge-l
ComicVT5-scratch | 250M | 0.2389 | 0.0034 | 0.2661 | 0.0422 | 0.2151 | 0.2646
ComicVT5-blip | 1.3B | 0.2363 | 0.0036 | 0.2653 | 0.0454 | 0.2171 | 0.2638

Similar to before, Table 10 presents the NLP metrics for dialogues generated
without the inclusion of possible answers in the encoder input. The table
shows results calculated based on the proportion of the target dialogue
provided to the transformer decoder. The results indicate a trend where
providing a larger percentage of the target dialogue leads to a sentence
completion more aligned with the target itself. Interestingly, these results
highlight a lower performance relative to the previous approach, reflecting
the increased task complexity when no option is provided as input.

###  9.2 Qualitative results

This section showcases three dialogue examples generated by the Comics-
VT5-blip model, utilizing textract OCR for context text and without providing
predefined answer options. The generation of these dialogues employed a beam
search approach with 2 beams and a cap of 35 tokens. The examples demonstrate
the model’s capabilities in dialogue generation, including two instances of
contextually appropriate dialogues (Figures 8(a) and 8(b)) and one instance
where the generated dialogue, while coherent, does not align well with the
ongoing narrative (Figure 8(c)).

(a)

(b)

(c)

Figure 8: (a) Example of a contextually appropriate generated dialogue; (b)
Another instance of a feasible dialogue generation; (c) An example of a
repeated dialogue that does not match the current story context.