# iColoriT: Towards Propagating Local Hint to the Right Region in Interactive
Colorization by Leveraging Vision Transformer

Sanghyeon Lee*, Jooyeol Yun*, Minho Park*, Jaegul Choo
Korea Advanced Institute of Science and Technology (KAIST)
Daejeon, Korea
{shlee6825, blizzard072, m.park<EMAIL_ADDRESS>
###### Abstract

Point-interactive image colorization aims to colorize grayscale images when a
user provides the colors for specific locations. It is essential for point-
interactive colorization methods to appropriately propagate user-provided
colors (_i.e_., user hints) in the entire image to obtain a reasonably
colorized image with minimal user effort. However, existing approaches often
produce partially colorized results due to the inefficient design of stacking
convolutional layers to propagate hints to distant relevant regions. To
address this problem, we present iColoriT, a novel point-interactive
colorization Vision Transformer capable of propagating user hints to relevant
regions, leveraging the global receptive field of Transformers. The self-
attention mechanism of Transformers enables iColoriT to selectively colorize
relevant regions with only a few local hints. Our approach colorizes images in
real-time by utilizing pixel shuffling, an efficient upsampling technique that
replaces the decoder architecture. Also, in order to mitigate the artifacts
caused by pixel shuffling with large upsampling ratios, we present the local
stabilizing layer. Extensive quantitative and qualitative results demonstrate
that our approach highly outperforms existing methods for point-interactive
colorization, producing accurately colorized images with a user’s minimal
effort.

##  1 Introduction

Unconditional image colorization [13, 33, 45, 36, 12, 43] has shown remarkable
achievement in restoring the vibrance of grayscale photographs or films in a
fully-automatic manner. Interactive colorization methods [7, 40, 44, 47, 16,
41] further extend the task to allow users to generate colorized images with
specific color conditions. These approaches can dramatically reduce the user
effort for producing specific colorized images. It can also serve as an
effective way of editing photos by re-coloring existing images to have a new
color theme. Among different types of interactions provided by users (_e.g_.,
a reference image or a color palette), point- or scribble-based interactions
[47, 16, 41] are designed to progressively colorize images when a user
provides the colors at specific point locations.

Figure 1: Example results of various point-interactive colorization
approaches. Previous approaches often produce partially colorized results even
where the grayscale values are persistent (_e.g_., water, floor, and grass),
which indicates that the user hints did not properly propagate to the relevant
regions.

Practical point-interactive colorization methods assist the user to produce a
colorized image with minimal user interaction. Thus, accurately estimating the
regions relevant to the user hint can be beneficial for reducing the amount of
user interactions. For example, using hand-crafted filters [16, 41] to
determine the region a user hint should fill in was an early approach for
colorizing simple patterns within the image. Recently, Zhang  _et al_. [47]
proposed a learning-based model trained on a large-scale dataset [28] which
produces colorized images with a simple U-Net architecture. However, existing
methods tend to suffer from partially colorized results even in obvious
regions where the grayscale values are persistent, as seen in Figure 1. This
is due to the inefficient design of stacking convolutional layers in order to
propagate hints to distant relevant regions. In other words, propagating hints
to large semantic regions can only be done in the deep layers, which makes
colorizing larger semantic regions more challenging than colorizing smaller
regions. To overcome this hurdle, we leverage the global receptive field of
self-attention layers [35] in Vision Transformers [4], enabling the model to
selectively propagate user hints to relevant regions at each single layer.

Learning how to propagate user hints to other regions aligns well with the
self-attention mechanism. Specifically, directly computing the similarities of
features from all spatial locations (_i.e_., the similarity matrix) can be
viewed as deciding where the hint colors should propagate in the entire image.
Thus, in this work, we present iColoriT, a novel point-interactive
colorization framework utilizing a modified Vision Transformer for colorizing
grayscale images. To the best of our knowledge, this is the first work to
employ a Vision Transformer for point-interactive colorization.

Furthermore, promptly displaying the results for a newly provided user hint is
essential for assisting users to progressively colorize images without delay.
For this reason, we generate color images by leveraging the efficient pixel
shuffling operation [30], an upsampling technique that reshapes the output
channel dimension into a spatial resolution. Through the light-weight pixel
shuffling operation, we are able to discard the conventional decoder
architecture and offer a faster inference speed compared to existing
baselines. Despite its efficiency, pixel shuffling with large upsampling
ratios tends to generate unrealistic images with missing details and notable
boundaries as seen in Figure 2. Therefore, we present the local stabilizing
layer, which restricts the receptive field of the last layer, to mitigate the
artifacts caused by pixel shuffling. Our contributions are as follows:

  * •

We are the first work to utilize a Vision Transformer for point-interactive
colorization enabling users to selectively colorize relevant regions.

  * •

We achieve real-time colorization of images by effectively upsampling images
with minimal cost, leveraging the pixel shuffling and the local stabilizing
layer.

  * •

We provide quantitative and qualitative results demonstrating that iColoriT
highly outperforms existing state-of-the-art baselines and generates
reasonable results with fewer user interactions.

##  2 Related Work

Interactive Colorization Learning-based methods for image colorization [13,
36, 33, 45, 12, 14, 43, 48, 14] have proposed fully-automated colorization
methods, which generate reasonable color images without the need of any user
intervention. Interactive colorization methods [44, 7, 16, 47, 41, 22, 40, 38,
42, 17, 18] are designed to colorize images given a user’s condition which
conveys color-related information. A widely-studied condition type for
interactive colorization methods are reference images [7, 17, 44, 38, 22, 40,
18, 42], which are already-colored exemplar images. Using reference images can
be convenient since the user can provide the overall color tones with a single
image. However, it is difficult for the user to further edit specific regions
in the colorized image since a new reference image is likely to produce a
different colorization result.

Figure 2: Images generated with large upsample ratios [5] tends to suffer from
evident borders between image patches.

Point-interactive Colorization Point-interactive colorization models [16, 47,
41] allow the user to progressively colorize images by specifying colors
(_i.e_., user hints) at different point locations in the input grayscale
image. Since commonly used point sizes for specifying the spatial locations
range from $2\times 2$ to $7\times 7$ pixels, the user hints only cover a
small portion of the entire image. Thus, a point-interactive colorization
model is required to propagate user hints to the entire image in order to
produce a reasonable result with minimal user interaction. Early approaches
[16, 41] utilized hand-crafted image filters to determine the propagation
region of each hint by detecting simple patterns. The colors of the user hints
are then propagated within each region using optimization techniques.
Recently, Zhang  _et al_. [47] proposed a learning-based method by extending
an existing unconditional colorization model [45] to produce color images
given a grayscale image and user hints. Although these methods use user hints
as a condition for generating color images, common failure cases presented in
Figure 1 indicate that the models often propagate hints incompletely. Stacking
convolutional layers to propagate user hints indicates that propagating hints
to distant relevant regions can only be done in the deeper layers, which makes
colorizing larger semantic regions more challenging than nearby regions. Thus,
we utilize the self-attention layer to enable user hints to propagate to any
relevant regions at all layers.

Figure 3: The overall workflow of iColoriT. We first obtain input $X$ by
concatenating the grayscale image $I_{g}$ and the user hint $I_{\text{hint}}$
containing color conditions. The input is reshaped into input patches for the
Transformer encoder. The output features from the Transformer encoder are
passed through the local stabilizing layer and the pixel shuffling layer to
obtain the final colors $I_{\text{pred}}^{\text{ab}}$.
$I_{\text{pred}}^{\text{ab}}$ is then concatenated with $I_{g}$ to produce the
colorized image.

Image Colorization with Transformers Unlike the widely-used convolution-based
approach for image synthesis, recent studies [13, 5, 39, 15] made efforts to
synthesize images by only utilizing the Transformer architecture. Colorization
Transformer (ColTran) [13] proposes an autoregressive model for unconditional
colorization which uses the Transformer decoder architecture [35] in order to
generate diverse colorization results. Despite its outstanding performance for
unconditional colorization, the excessively slow inference speed of
autoregressive models hinders its application to user-interactive scenarios.
Specifically, it takes 3.5-5 minutes to colorize a batch of 20 images of size
$64\times 64$ images even with a P100 GPU. In this work, we leverage the
Transformer encoder to generate the colors of a grayscale image. The multi-
head attention of the Transformer encoder enables our approach to generate
color images with a single forward pass which reduces the inference time of
our model compared to autoregressive colorization.

Upsampling via Pixel Shuffling Pixel shuffling [30] is an upsampling operation
that rearranges a $(H,W,C\times P^{2})$ sized feature map into a shape of
$(H\times P,W\times P,C)$ where each channel in the original feature map is
reshaped into a $P\times P$ image patch. This can be viewed as _upsampling via
reshaping_ , and is often used in super-resolution approaches to effectively
upsample an image with minimal computational overhead. ViTGAN [15] is a
pioneering approach for using Transformer encoders for image generation and
synthesizing image patches by pixel shuffling the output feature map. However,
the usage of pixel shuffling was limited to a small upscale factor of 4 or 8,
restricting the model to generate images in small resolutions (_e.g_.,
$32\times 32\text{ and }64\times 64$). A known issue [5] with pixel shuffling
with larger upsampling ratios ($P>8$) was that output images tend to contain
evident borders between image patches as seen in Figure 2. This is due to
upsampling different image patches from different locations in the feature
map. To overcome this hurdle, we present a local stabilizing layer, which
promotes neighboring image patches to have coherent colors, allowing iColoriT
to effectively upsample images to higher resolutions (_i.e_., $224\times 224$)
without such artifacts.

##  3 Proposed Method

###  3.1 Preliminaries

We first prepare the grayscale image $I_{g}\in\mathbb{R}^{H\times W\times 1}$
and the simulated user hints $I_{\text{hint}}\in\mathbb{R}^{H\times W\times
3}$ to be used as our training sample. A grayscale image $I_{g}$ can be
acquired from large-scale datasets by converting the color space from RGB to
CIELab [31] and taking the L or lightness value. Similarly, the color
condition $I_{\text{hint}}$ provided by the user can be expressed with the
remaining a, b channel values $\tilde{I}_{\text{hint}}\in\mathbb{R}^{H\times
W\times 2}$ by filling the a,b channel values of all non-hint regions with 0.
The user hint $I_{\text{hint}}\in\mathbb{R}^{H\times W\times 3}$ is
constructed by adding a third channel to $\tilde{I}_{\text{hint}}$ that marks
hint regions with 1 and non-hint regions with 0.

During training, we simulate the user hints by determining the hint location
and the color of the hint. We sample hint locations from a uniform
distribution since a user may provide hints anywhere in the image. Once the
hint location is decided, the color of the user hint is obtained by
calculating the average color values for each channel within the hint region
since a user is expected to provide a single color for a single hint location.
Finally, given the grayscale image $I_{g}\in\mathbb{R}^{H\times W\times 1}$
and the simulated user hints $I_{\text{hint}}\in\mathbb{R}^{H\times W\times
3}$, we obtain our input $X\in\mathbb{R}^{H\times W\times 4}$ by

$X=I_{g}\oplus I_{\text{hint}},$

where $\oplus$ is the channel-wise concatenation.

###  3.2 Propagating User Hints with Transformers

We utilize the Vision Transformer [4] to achieve a global receptive field for
propagating user hints across the image as shown in Figure 3. We first reshape
our input $X\in\mathbb{R}^{H\times W\times 4}$ into a sequence of tokens
$X_{p}\in\mathbb{R}^{N\times(P^{2}\times 4)}$, where $H,W$ are the height and
width of the original image, $P$ is the patch size, and $N=HW/P^{2}$ is the
number of input tokens (_i.e_., sequence length). Thus, a $P\times P\times 4$
size image patch from the original input $X$ is used as a single input token.
These sequence of input tokens are passed through the Transformer encoder,
which computes the input as,

$z_{0}=X_{p}+E_{pos},\quad E_{pos}\in\mathbb{R}^{N\times d}$ (1)
$z^{\prime}_{l}=\text{MSA}(\text{LN}(z_{l-1}))+z_{l-1},$ (2)
$z_{l}=\text{MLP}(\text{LN}(z^{\prime}_{l}))+z^{\prime}_{l},$ (3)
$y_{p}=\text{LN}(z_{L}),$ (4)

where $E_{pos}$ denotes the sinusoidal positional encoding [4],
$\text{MSA}(\cdot)$ indicates the multi-head self-attention [35],
$\text{LN}(\cdot)$ indicates the layer normalization [2], $d$ denotes the
hidden dimension, $l$ denotes the layer number, and
$y_{p}\in\mathbb{R}^{N\times d}$ denotes the output of the Transformer
encoder. Since self-attention does not utilize any position-related
information, we add positional encoding $E_{pos}$ to the input and relative
positional bias [19, 27, 9, 10] in the attention layer. Thus, the attention
layer is computed as,

$\text{Attention}(Q,K,V)=\text{softmax}(QK^{T}/\sqrt{d}+B)V,$ (5)

where $Q,K,V\in\mathbb{R}^{N\times d}$ are the query, key and value matrices,
$B\in\mathbb{R}^{N\times N}$ is the relative positional bias. The colors of
the user hints are able to propagate to any spatial location at all layers due
to the global receptive field of the self-attention mechanism.

###  3.3 Pixel Shuffling and the Local Stabilizing Layer

The output features of the Transformer encoder $y_{p}\in\mathbb{R}^{N\times
d}$ can be viewed as a feature map $y\in\mathbb{R}^{H/P\times W/P\times d}$ of
the original image. The spatial resolution of the output feature map $y$ is
smaller than the resolution of the input image by a factor of $P$ since image
patches of size $P\times P$ consists of a single input token. Therefore, the
output feature map $y$ needs to be upsampled in order to obtain a full-
resolution color image. While previous approaches [47, 33] leverage a decoder
for upsampling, we utilize pixel shuffling [30] which is an upsampling
technique rearranging a ($H/P,W/P,C\times P^{2}$) feature map into a shape of
($H,W,C$) to obtain a full-resolution image.

However, as mentioned in Section 2, large upsampling ratios (_e.g_., $P>8$)
may lead to images with visible artifacts along the image patch boundaries as
seen in Figure 4. Thus, in order to promote reasonable generation of colors,
we propose a local stabilizing layer, which restricts the model to generate
colors utilizing neighboring features, and place the layer before pixel
shuffling. We provide experiments in Section 4.2 with various design choices
for the local stabilizing layer (_e.g_., linear, convolutional layer, and
local attention) and select a simple yet effective convolutional layer as our
final model. To sum up, our upsampling process can be written as,

$I_{\text{pred}}^{\text{ab}}=\mathcal{PS}(\text{LS}(y)),$ (6)

where $\mathcal{PS}(\cdot)$ is the pixel shuffling operation, LS($\cdot$) is
the local stabilizing layer, and
$I_{\text{pred}}^{\text{ab}}\in\mathbb{R}^{H\times W\times 2}$ is the ab color
channel outputs. The predicted color image
$I_{\text{pred}}\in\mathbb{R}^{H\times W\times 3}$ is obtained by

$I_{\text{pred}}=I_{g}\oplus I_{\text{pred}}^{\text{ab}},$

which is the concatenation of the given grayscale input $I_{g}$ (L channel)
and $I_{\text{pred}}^{\text{ab}}$ (ab channel). Through pixel shuffling and
the local stabilizing layer, we can effectively obtain a full-resolution color
image _without_ an additional decoder, allowing real-time colorization for the
user (Section 4.1).

Figure 4: Example images of inconsistent colorization results observed in
images produced by without the local stabilizing layer.

###  3.4 Objective Function

We train our model with the Huber loss [11] between the predicted image and
the original color image in the CIELab color space,

$\begin{split}L_{recon}=&\frac{1}{2}(I_{\text{pred}}-I_{GT})^{2}\mathbbm{1}_{\left|I_{\text{pred}}-I_{GT}\right|<1}\\\
&+(\left|I_{\text{pred}}-I_{GT}\right|-\frac{1}{2})\mathbbm{1}_{\left|I_{\text{pred}}-I_{GT}\right|\geq
1}.\end{split}\vspace{-0.2cm}$ (7)

Figure 5: Average PSNR and LPIPS of the test images according to the number of
provided hints. Hint locations are sampled from a uniform distribution and
$2\times 2$ hints are revealed to the model. Yin  _et al_. ∗ [41] denotes the
results evaluated with $2\times 2$ hints and Yin  _et al_. [41] denotes the
results evaluated with $7\times 7$ hints. iColoriT outperforms existing
approaches by a large margin as the number of provided hints increases.

##  4 Experiments

Implementation Details We follow the configurations of ViT-B [4] for the
Transformer encoder blocks. For the local stabilizing layer, we use a single
layer with a receptive field of 3. We experiment with two types of layers
(Section 4.2), the local attention and the convolutional layer, and use the
simple yet effective convolutional layer as the default local stabilizing
layer. For training, we resize images to a $224\times 224$ resolution and use
a patch size of $P=16$ which also becomes the upsampling ratio. Thus, the
sequence length $N$ is 196 and the last output dimension $d$ is 512. We sample
hint locations uniformly across the image and sample the number of hints from
a uniform distribution $\mathcal{U}(0,128)$. We provide experiments on
different model sizes, patch sizes, the local stabilizing layer, and the
number of hints in Section 4.2 and the supplementary material.

We use the AdamW optimizer [21] with a learning rate of $0.0005$ managed by
the cosine annealing scheduler [20]. The model is trained for 2.5M iterations
with a batch size of 512. The codes for iColoriT implemented with the Pytorch
library [25] will be available.

Datasets For training, we use the ImageNet 2012 train split [28] which
consists of 1,281,167 images. We do not use the classification labels during
training since our model is trained in a self-supervised manner. We evaluate
our method on three datasets from different domains, all of which are colorful
validation datasets suitable for evaluating colorization approaches. Note that
we do not additionally finetune the model for each validation dataset. The
ImageNet ctest [14] is a subset of the ImageNet validation split used as a
standard benchmark for evaluating colorization models. ImageNet ctest excludes
any grayscale image from ImageNet and consists of 10,000 color images. We also
evaluate on the Oxford 102flowers dataset [24] and the CUB-200 dataset [37]
which provide 1,020 colorful flower images from 102 categories and 3,033
samples of bird images from 200 different species, respectively.

Baselines We compare the performance of iColoriT with existing interactive
colorization methods [47, 41]. We also extend a recent unconditional
colorization model by Su  _et al_. [33], which utilizes an off-the-shelf
object detector [6] to individually color multiple instances, to a point-
interactive colorization model. Since the model proposed by Su  _et al_. [33]
employs the same model architecture and objective function as the point-
interactive colorization model by Zhang  _et al_. [47], we are able to
effortlessly extend the approach to a point-interactive colorization method by
conditioning the model with user hints in the same manner. The extended model
is trained under the configurations provided by Zhang  _et al_. [47] and Su
_et al_. [33] using ImageNet [28]. Note that although the model proposed by Su
_et al_. [33] is trained with the ImageNet [28] dataset, this approach is
assisted by an off-the-shelf object detector pre-trained on a large-scale
object detection dataset [3]. All baselines are trained and evaluated with the
publicly available official codes.

Figure 6: Qualitative results of point-interactive colorization methods given
1, 5, 10, and 100 user hints. iColoriT is able to produce reasonable color
images by appropriately propagating user hints.

###  4.1 Comparison with Existing Approaches

Quantitative Evaluation of iColoriT We plot the average peak signal-to-noise
ratio (PSNR) and the learned perceptual image patch similarity (LPIPS) [46] of
the test images according to the number of provided hints in Figure 5. For
evaluating the point-interactive colorization models, we simulate user hints
with the ground-truth colors from the image, considering a situation where the
user intends to colorize the grayscale image into the original color image.
User hints are simulated by randomly selecting hint locations from a uniform
distribution. The hint sizes are set to $2\times 2$ and the hint color is
given as the average color within each hint region in the original color image
following the protocol of Zhang  _et al_. [47]. We empirically find that
smaller hint sizes are usually beneficial for both the colorization model and
the user in terms of receiving and giving accurate color conditions. However,
the method proposed by Yin  _et al_. [41] assumes that a user provides an
abundant amount of user hints. Thus, we further evaluate this method by
revealing larger hints of size $7\times 7$ which is the result we report for
all following evaluations.

We empirically find that methods proposed by Zhang  _et al_. [47] and Su  _et
al_. [33] tend to arbitrarily colorize images without reflecting user hints.
While this may be helpful for achieving a relatively higher initial PSNR when
the arbitrarily colorized color is the ground-truth color, it hinders further
control for the user to achieve a high PSNR in subsequent stages of
colorization. As seen in Figure 5, iColoriT quickly reflects the user hints
and aids the user to efficiently colorize grayscale images with minimal
interaction. The PSNR in the early stages of colorization notably increases
with each additional hint. The results indicate that iColoriT highly
outperforms existing baselines for generating colorized images a user
specifically has in mind.

Qualitative Results of iColoriT We provide qualitative results produced by the
baselines and iColoriT in Figure 6 when given an original grayscale image and
the simulated user hints. iColoriT is able to produce realistic images that
closely resemble the ground-truth image indicating that a user can colorize
images as they please. Also, as seen in the colorized results in Figure 1 and
Figure 6, iColoriT is capable of appropriately colorizing large areas even
with a small number of user hints while other approaches leave most regions
uncolored or incorrectly colored. iColoriT can also colorize detailed regions
when given a sufficient number of hints as shown in the last row of Figure 6.

iColoriT is also suitable for producing _diverse_ colorized images when given
various user hints as seen in Figure 7. Instead of the simulated user hints
from the ground-truth image, we provide multiple sets of hand-picked user
hints to colorize a single grayscale image. We fix the hint locations for an
image and alter the user-provided colors to observe the colorized results.
iColoriT can produce various realistic colorization results that reflect the
intention of the user. We provide uncurated qualitative results and a demo
video in the supplementary material. Also, we will release the iColoriT demo
including the graphical user interface, providing a powerful tool for image
colorization.

Figure 7: Images colorized with different colors provided by the user. The images from the ImageNet ctest [14] are colored by hand-picking hint locations and changing the hint colors.  Methods |  PSNR@10 |  LPIPS@10
---|---|---
iColoriT-T |  28.86 |  0.084
iColoriT-S |  29.67 |  0.073
iColoriT |  30.63 |  0.062
Table 1: Scalability of iColoriT to lightweight models. PSNR and LPIPS given
10 user hints (PSNR@10 and LPIPS@10) on the ImageNet ctest [14] are reported
for each model.

Scaling to Lightweight Models iColoriT can easily scale to smaller models and
still achieve high performance. We train iColoriT in smaller scales using the
configurations of the ViT-S and the ViT-Ti [32] for our Transformer encoder.
We report the PSNR and the LPIPS given 10 hints (PNSR@10 and LPIPS@10) for
ImageNet ctest and compare them against other models in Table 1. We were able
to train iColoriT-S and iColoriT-T with only a slight performance drop and
still maintain a high performance. We believe that the Transformer
architecture and the self-attention mechanism are central for propagating
hints to larger semantic regions, achieving a high PSNR even in small-scale
models.

Real-time Inference The inference speed (_i.e_., latency) of point-interactive
models is important for providing a satisfying user experience. Thus, we
measure the time required for a single forward pass and compare it with the
latency of baseline models in Table 2. We report the speed on both CPU and GPU
using a commercial AMD Ryzen 5 PRO 4650G and a single NVIDIA RTX 3090. We also
provide the number of floating-point operations (FLOPs) and the number of
parameters required for each model. We were not able to measure GPU latency,
FLOPs, and the number of parameters for Yin  _et al_. [41] since the method is
not a learning-based model. The model proposed by Su  _et al_. [33] operates
in two stages, an initial object detection stage and an instance-wise
colorization stage. We only report the latency for the second stage which
still exhibits a slow inference speed since the colorization model needs to
color multiple objects individually. Due to the efficient pixel shuffling for
upsampling images, iColoriT enjoys a short latency of 540ms and 14ms on a CPU
and GPU device respectively, providing real-time colorization results for the
user. iColoriT-T and iColoriT-S show an exceptionally fast inference speed on
a CPU-only device (_i.e_., 177ms and 253ms, respectively), which makes the
model an appealing option when considering applications to real-world
scenarios where accelerators may not be available.

Methods |  CPU Latency |  GPU Latency |  GFLOPs
---|---|---|---
Zhang  _et al_. [47] |  881ms |  24ms |  58.04
Yin  _et al_. [41] |  15,248ms |  - |  -
Su  _et al_. [33] |  1,389ms |  45ms |  123.48
iColoriT-T |  177ms |  13ms |  1.43
iColoriT-S |  253ms |  14ms |  4.95
iColoriT |  540ms |  14ms |  18.22
Table 2: Inference speed of iColoriT and each baseline model. We provide the
latency of each model in a CPU device and a GPU device along with the
computational cost measured in FLOPs and number of parameters.

###  4.2 Ablation Study

Designing the Local Stabilizing Layer We provide an ablation study on the
local stabilizing layer by replacing it with different operations such as the
linear layer and the local self-attention layer [26]. Using a linear layer can
be viewed as eliminating the local stabilizing layer since a linear layer does
not utilize neighboring features for generating the final output. In order to
quantify the inconsistent color generation among image patches seen in Figure
4, we measure the mean squared error (MSE) for each image patch and report the
variance of the errors within an image. We denote this measure the patch error
variance (PEV). A high PEV implies that the model has varying accuracy
depending on the image patch. The local stabilizing layer resolves this issue
in a simple yet effective manner by predicting the ab channel values of an
image patch from neighboring output features as illustrated in Figure 3. We
also measure the PSNR near the image patch boundaries (_i.e_., one pixel from
the patch borders) to observe the accuracy in the regions containing
inconsistent color generation. As seen in Table 3, adding an operation with a
limited receptive field (_i.e_., convolution and local self-attention) lowers
the PEV and increases the PSNR along the patch boundaries, indicating that the
model generates colors with consistent accuracy across the image. The
convolutional layer serves as a simple yet effective approach for reducing
artifacts caused by pixel shuffling and generating realistic colorized images.

Methods |  PSNR@10 |  B-PSNR@10 |  PEV$\downarrow$
---|---|---|---
Linear |  28.78 |  28.71 |  39.39
Local Attention |  28.85 |  28.77 |  38.82
Convolution |  28.86 |  28.80 |  38.81
Table 3: Ablation study on the local stabilizing layer. PSNR@10, PSNR along the boundary (B-PSNR@10), and PEV on the ImageNet ctest [14] are reported for each model. All models are trained with the iColoriT-T configuration. Patch Size |  PSNR@10 |  CPU Latency
---|---|---
$8\times 8$ |  29.17($+$0.31) |  373ms($+$196ms)
$32\times 32$ |  28.32($-$0.54) |  147ms($-$30ms)
$16\times 16$ |  28.86 |  177ms
Table 4: iColoriT different upsampling ratios. PSNR@10, LPIPS@10, and CPU
latency are reported for each model on the ImageNet ctest [14] test set. All
models are trained with the iColoriT-T configuration.

Changing the Upsampling Ratio We experiment on various patch sizes $P$
(_i.e_., $P=8,16,$ and $32$), which also becomes the upscaling ratio for pixel
shuffling. While smaller patch sizes may allow fine-grained calculation of the
similarity matrix, the computational cost escalates _biquadratically_ , since
the computational complexity for the self-attention follows
$\mathcal{O}(N^{2})$ and $N=HW/P^{2}$ is the sequence length. Thus, we were
not able to train our base model with a smaller patch size due to the
prohibitive computational overhead. Instead, we compare the results on the
smaller iColoriT-T model and report the average PSNR@10 and CPU latency on
Table 4. While using a smaller patch size may be beneficial for achieving a
higher PSNR, the increased computational cost hinders scaling to larger models
for an additional performance gain and increases the CPU latency. We choose a
patch size of $16\times 16$ since it can obtain both a short latency and a
high PSNR while also being scalable to larger models (_i.e_., iColoriT-S and
iColoriT).

###  4.3 Visualizing the Internal Representation

We further provide analysis on the self-attention mechanism to examine how our
model is propagating user hints to other regions. We use the attention rollout
method [1] to interpret the attention weights from the Transformer encoder for
specific spatial locations. We visualize the attention maps for the input
tokens which contain a user hint in Figure 8. Attention maps for hint
locations can be directly interpreted as how the hint is propagating to other
locations since tokens with high similarities are likely to be colorized with
similar color as the color of the user hint. The self-attention mechanism
enables iColoriT to selectively colorize relevant locations, even for regions
with spatially complicated structures. These visualization aligns well with
our qualitative and quantitative results demonstrating that iColoriT can
effectively aid users to colorize images with minimal interaction.

Figure 8: Visualization of the self-attention mechanism employing the
attention rollout [1] method. iColoriT appropriately attends the user hint to
relevant locations even for complex structures.  Figure 9: A common failure
case for point-interactive colorization models in detailed regions.

##  5 Conclusion and Limitations

In this paper, we present iColoriT, a novel real-time point-interactive
colorization framework capable of selectively propagating colors of the user
hints to relevant regions. Through the Transformer encoder, pixel shuffling
and the local stabilizing layer, iColoriT highly outperforms existing
baselines, being able colorize images with minimal user interaction. Also,
qualitative results indicate that iColoriT can generate diverse and realistic
results when given various user hints. We justify our novel design through
extensive experiments and ablation studies.

Although iColoriT shows its strength even in detailed regions as shown in both
quantitative and qualitative results, iColoriT may not be able to colorize
small objects or distinguish close objects with the same grayscale intensity,
since it does not leverage any semantic labels. This is a common drawback of
point-interactive colorization approaches as seen in Figure 9 since models are
trained in a self-supervised manner. Directly utilizing segmentation labels
for training a point-interactive colorization model can be a promising future
work. Nonetheless, we believe that the iColoriT is a practical application for
real-world scenarios, effectively assisting the user to colorize images.

## References

  * [1] Samira Abnar and Willem Zuidema.  Quantifying attention flow in transformers.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190–4197, 2020.
  * [2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [3] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari.  Coco-stuff: Thing and stuff classes in context.  In Proc. of the IEEE conference on computer vision and pattern recognition (CVPR), 2018.
  * [4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2020.
  * [5] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  arXiv preprint arXiv:2111.06377, 2021.
  * [6] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [7] Mingming He, Dongdong Chen, Jing Liao, Pedro V Sander, and Lu Yuan.  Deep exemplar-based colorization.  ACM Transactions on Graphics (TOG), 37(4):1–16, 2018.
  * [8] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Proc. the Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
  * [9] Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei.  Relation networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3588–3597, 2018.
  * [10] Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin.  Local relation networks for image recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3464–3473, 2019.
  * [11] Peter J Huber.  Robust estimation of a location parameter.  In Breakthroughs in statistics, pages 492–518. Springer, 1992.
  * [12] Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa.  Let there be color!: Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification.  Proc. the ACM Transactions on Graphics (ToG), 35, 2016.
  * [13] Manoj Kumar, Dirk Weissenborn, and Nal Kalchbrenner.  Colorization transformer.  In ICLR, 2020.
  * [14] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich.  Learning representations for automatic colorization.  In European Conference on Computer Vision (ECCV), 2016.
  * [15] Kwonjoon Lee, Huiwen Chang, Lu Jiang, Han Zhang, Zhuowen Tu, and Ce Liu.  Vitgan: Training gans with vision transformers.  In International Conference on Learning Representations, 2021.
  * [16] Anat Levin, Dani Lischinski, and Yair Weiss.  Colorization using optimization.  Proc. the ACM Transactions on Graphics (ToG), 23:689–694, 2004\.
  * [17] Bo Li, Yu-Kun Lai, Matthew John, and Paul L Rosin.  Automatic example-based image colorization using location-aware cross-scale matching.  IEEE Transactions on Image Processing, 28(9):4606–4619, 2019.
  * [18] Haoxuan Li, Bin Sheng, Ping Li, Riaz Ali, and CL Philip Chen.  Globally and locally semantic colorization via exemplar-based broad-gan.  IEEE Transactions on Image Processing, 30:8526–8539, 2021.
  * [19] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In ICCV, 2021.
  * [20] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  In International Conference on Learning Representations, 2017.
  * [21] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In International Conference on Learning Representations, 2018.
  * [22] Peng Lu, Jinbei Yu, Xujun Peng, Zhaoran Zhao, and Xiaojie Wang.  Gray2colornet: Transfer more colors from reference image.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 3210–3218, 2020.
  * [23] David L. MacAdam.  Visual sensitivities to color differences in daylight$\ast$.  J. Opt. Soc. Am., 32(5):247–274, May 1942.
  * [24] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In Indian Conference on Computer Vision, Graphics and Image Processing, 2008.
  * [25] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Proc. the Advances in Neural Information Processing Systems (NeurIPS), 32:8026–8037, 2019.
  * [26] Jack Rae and Ali Razavi.  Do transformers need deep long-range memory?  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, July 2020. Association for Computational Linguistics.
  * [27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  Journal of Machine Learning Research, 21:1–67, 2020.
  * [28] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  International Journal of Computer Vision (IJCV), 115(3):211–252, 2015.
  * [29] Gaurav Sharma.  Color fundamentals for digital imaging.  Digital color imaging handbook, 20:1–114, 2003.
  * [30] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang.  Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1874–1883, 2016.
  * [31] Thomas Smith and John Guild.  The cie colorimetric standards and their use.  Transactions of the optical society, 33(3):73, 1931.
  * [32] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer.  How to train your vit? data, augmentation, and regularization in vision transformers.  arXiv preprint arXiv:2106.10270, 2021.
  * [33] Jheng-Wei Su, Hung-Kuo Chu, and Jia-Bin Huang.  Instance-aware image colorization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7968–7977, 2020.
  * [34] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision. arxiv.  arXiv preprint arXiv:1512.00567, 2015.
  * [35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [36] Patricia Vitoria, Lara Raad, and Coloma Ballester.  Chromagan: Adversarial picture colorization with semantic class distribution.  In WACV, 2020.
  * [37] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona.  Caltech-ucsd birds 200.  Technical Report CNS-TR-2010-001, California Institute of Technology, 2010\.
  * [38] Chufeng Xiao, Chu Han, Zhuming Zhang, Jing Qin, Tien-Tsin Wong, Guoqiang Han, and Shengfeng He.  Example-based colourization via dense encoding pyramids.  In Computer Graphics Forum, volume 39, pages 20–33. Wiley Online Library, 2020.
  * [39] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu.  Simmim: A simple framework for masked image modeling.  arXiv preprint arXiv:2111.09886, 2021.
  * [40] Zhongyou Xu, Tingting Wang, Faming Fang, Yun Sheng, and Guixu Zhang.  Stylization-based architecture for fast deep exemplar colorization.  In Proc. of the IEEE conference on computer vision and pattern recognition (CVPR), pages 9363–9372, 2020.
  * [41] Hui Yin, Yuanhao Gong, and Guoping Qiu.  Side window filtering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8758–8766, 2019.
  * [42] Wang Yin, Peng Lu, Zhaoran Zhao, and Xujun Peng.  Yes,” attention is all you need”, for exemplar based colorization.  In Proceedings of the 29th ACM International Conference on Multimedia, pages 2243–2251, 2021.
  * [43] Seungjoo Yoo, Hyojin Bahng, Sunghyo Chung, Junsoo Lee, Jaehyuk Chang, and Jaegul Choo.  Coloring with limited data: Few-shot colorization via memory augmented networks.  In Proc. of the IEEE conference on computer vision and pattern recognition (CVPR), 2019.
  * [44] Bo Zhang, Mingming He, Jing Liao, Pedro V Sander, Lu Yuan, Amine Bermak, and Dong Chen.  Deep exemplar-based video colorization.  In Proc. of the IEEE conference on computer vision and pattern recognition (CVPR), 2019.
  * [45] Richard Zhang, Phillip Isola, and Alexei A Efros.  Colorful image colorization.  In European conference on computer vision, pages 649–666. Springer, 2016.
  * [46] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proc. of the IEEE conference on computer vision and pattern recognition (CVPR), 2018.
  * [47] Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S Lin, Tianhe Yu, and Alexei A Efros.  Real-time user-guided image colorization with learned deep priors.  ACM TOG, 2017.
  * [48] Jiaojiao Zhao, Jungong Han, Ling Shao, and Cees Snoek.  Pixelated semantic colorization.  International Journal of Computer Vision, 128, 04 2020.

Supplementary Materials

This supplementary material presents quantitative results on unconditional
colorization (Section 6), the effect of the number of hints sampled during
training (Section 7), quantitative analysis of the hint propagation range
(Section 8), and additional qualitative results (Section 9). We also attach a
demo video demonstrating the use case of iColoriT.

##  6 Unconditional Colorization Using iColoriT

Similar to recent point-interactive colorization approaches [47, 33], iColoriT
is capable of generating colorized images without any user hints. We compare
the Fréchet inception distance (FID) score [8] of the colorization results
generated without any user hints with a pre-trained Inception V3 [34] network.
We exclude the conventional filter-based approach by Yin  _et al_. [41] since
the method does not operate without user hints. The FID score is widely used
to evaluate the how realistic the generated images are compared to the
original color images. Our approach achieves a low FID score and generates
realistic colors even without user hints.

Methods |  FID
---|---
Zhang  _et al_. [47] |  7.29
Su  _et al_. [33] |  6.56
Yin  _et al_. [41] |  -
iColoriT |  4.89
Table 5: Fréchet inception distance (FID) score of unconditional results of
iColoriT and baselines. iColoriT is able to generate realistic colors even
without user hints.

##  7 Number of Hints Sampled During Training

The number of simulated user hints sampled during training may alter the
performance since iColoriT directly learns how to propagate the color hints.
We train our model provided with different number of hints sampled from
various uniform distributions (_e.g_., $\mathcal{U}(0,16)$,
$\mathcal{U}(0,32)$, $\mathcal{U}(0,64)$, $\mathcal{U}(0,128)$,
$\mathcal{U}(0,256)$). To our surprise, the number of hints provided during
training did not have an immense effect on the final performance. As plotted
in Fig. 10, the PSNR measured on ImageNet ctest [14] are similar for most
models. We empirically find that sampling the number of hints from
$\mathcal{U}(0,128)$ demonstrates a high performance in most regions including
PSNR given 10 user hints (PSNR@10) which we choose as a representative
indicator. We also observe that the performance of models given a relatively
small number of hints (_i.e_., $\mathcal{U}(0,16)$) does not necessarily
outperform other models when evaluated with a small number of hints (_e.g_.,
PSNR@5). We presume that the model learns to appropriately propagate hints to
relevant regions when trained with a more diverse number of hints. Future work
focusing on how to sample the simulated hints is an interesting subject.

Figure 10: PSNR achieved by models trained by sampling different number of
simulated user hints. The numbers on the legend indicate the range of the
uniform distribution. The PSNR is measured in the ImageNet ctest [14]
validation set. Figure 11: Average hint propagation range (HPR) and PSNR gain
($\Delta$PSNR) when given an additional hint. All scores are measured in the
ImageNet ctest [14] dataset. iColoriT shows both high HPR and a high PSNR gain
at all stages of the colorization process.  Figure 12: Visualization of the
colorized region $\mathcal{C}_{1}$ for different baselines. iColoriT is able
to propagate the hint to further regions if needed, while other approaches do
not necessarily colorize all relevant regions.

##  8 Measuring the Hint Propagation Range

Not only can iColoriT accurately reflect user hints and achieve a higher PSNR,
iColoriT is capable of propagating user hints to longer distances if needed.
In order to measure how far a hint propagates to further regions, we present
the hint propagation range (HPR) measure. Given an image
$I_{\text{pred}}^{t-1}$ colorized with $t-1$ number of color hints, HPR@ $t$
indicates the average distance from the newly provided $t$-th hint to the
pixels that have been colorized by the $t$-th hint. We define a pixel to have
been colorized if the mean squared error (MSE) between the initial value and
the altered value is larger than 2.3 in the CIELab color space [31], which is
the just-noticeable-difference (JND) perceived by the human eye [29]. Given a
set of coordinates $(x_{i},y_{i})\in\mathcal{C}_{t}$ of pixels colorized by
hint $h_{t}$ and the coordinate of $h_{t}$ $(x_{h},y_{h})$, we calculate HPR
with,

$\mathcal{C}_{t}=\\{(x,y)\;|\;\text{MSE}(I^{t}_{xy},I^{t-1}_{xy})>\text{JND}\\},$
(8)
$\text{HPR}@\,t=\frac{1}{|\mathcal{C}_{t}|}\sum_{(x_{i},y_{i})\in\mathcal{C}_{t}}\sqrt{(x_{i}-x_{h})^{2}+(y_{i}-y_{h})^{2}},$
(9)

which is the average Euclidean distance from $h_{t}$ to all locations in
$\mathcal{C}_{t}$.

We measure the HPR across ImageNet ctest [14] at different stages of the
colorization process and plot the results in Figure 11. Note that the HPR
measure itself does not assess whether a model is appropriately reflecting the
color hints provided by the user. However, when examined together with PSNR
gain ($\Delta$PSNR) in Figure 11 where the PSNR hugely improves at earlier
stages, we can conclude that iColoriT reflects the color hints to further
regions in a constructive manner. The optimization-based method proposed by
Yin  _et al_. [41] often overly propagates the initial color hint to the
entire image, which does not contribute to improving the PSNR or the
perceptual quality compared to the original grayscale image. Learning-based
baselines [47, 33] on the other hand, tend to locally colorize images which
hinders further PSNR gain. iColoriT takes advantage of both aspects and
propagates color hints to further regions while hugely improving the PSNR.

Figure 13: The MacAdam ellipse from MacAdam’s paper [23] illustrating the
just-noticeable-difference in the xy chromaticity diagram.

We also visualize $\mathcal{C}_{1}$, the pixel coordinates colorized by the
initial hint $h_{1}$, in Figure 12 to intuitively understand the regions of
which the hint alters the image. We believe that the self-attention mechanism
is central for selectively colorizing the relevant regions regardless of the
distance from the user-provided color hint. Qualitative results supporting
this claim are provided in Section 9.

For a clearer understanding, we provide an illustration of the MacAdam ellipse
which visualizes the just-noticeable-difference of colors. The colors within
the same ellipse indicate that the colors are indistinguishable to the human
eye. Although the areas of the ellipses are not uniform in the xy chromaticity
diagram, the JND in the CIELab color space is known to be roughly constant.

##  9 Additional Qualitative Results

In this section and the remaining pages of the supplementary, we provide
additional qualitative results. Figs. 14, 15, 16 and 17 compares the results
from different baseline models and iColoriT. Furthermore, uncurated results
from the ImageNet ctest [14] of iColoriT are provided in Figs. 18, 19, 20 and
21. The figures are sorted according to the number of hints given to the
model. We can observe that most images do not contain color-bleeding artifacts
or partially colorized images unlike the baselines which produce partially
colorized images. Finally, we attach a demo video demonstrating an interactive
colorization scenario and a use case of iColoriT.

Figure 14: Additional qualitative results compared with baseline approaches. A
single hint location is sampled from a uniform distribution.  Figure 15:
Additional qualitative results compared with baseline approaches. 5 hint
locations are sampled from a uniform distribution.  Figure 16: Additional
qualitative results compared with baseline approaches. 10 hint locations are
sampled from a uniform distribution.  Figure 17: Additional qualitative
results compared with baseline approaches. 100 hint locations are sampled from
a uniform distribution.  Figure 18: Uncurated images produced with a single
hint where the hint location is randomly sampled from a uniform distribution.
Figure 19: Uncurated images produced given five hints where the hint locations
are randomly sampled from a uniform distribution. Figure 20: Uncurated images
produced given ten hints where the hint locations are randomly sampled from a
uniform distribution. Figure 21: Uncurated images produced given hundred hints
where the hint locations are randomly sampled from a uniform distribution.


# Listen Then See: Video Alignment with Speaker Attention

Aviral Agrawal∗, Carlos Mateo Samudio Lezcano∗, Iqui Balam Heredia-Marin∗,
Prabhdeep Singh Sethi∗
Carnegie Mellon University
{avirala, csamudio, iquibalh<EMAIL_ADDRESS>
###### Abstract

Video-based Question Answering (Video QA) is a challenging task and becomes
even more intricate when addressing Socially Intelligent Question Answering
(SIQA). SIQA requires context understanding, temporal reasoning, and the
integration of multimodal information, but in addition, it requires processing
nuanced human behavior. Furthermore, the complexities involved are exacerbated
by the dominance of the primary modality (text) over the others. Thus, there
is a need to help the task’s secondary modalities to work in tandem with the
primary modality. In this work, we introduce a cross-modal alignment and
subsequent representation fusion approach that achieves state-of-the-art
results (82.06% accuracy) on the Social IQ 2.0 dataset for SIQA. Our approach
exhibits an improved ability to leverage the video modality by using the audio
modality as a bridge with the language modality. This leads to enhanced
performance by reducing the prevalent issue of language overfitting and
resultant video modality bypassing encountered by current existing techniques.
Our code and models are publicly available at [1].

**footnotetext: Denotes equal contribution. Names are arranged in alphabetical
order.

##  1 Introduction

Video Question Answering (VQA) is a challenging field aiming to bridge the gap
between visual understanding and natural language processing. In VQA, the goal
is to develop systems that can accurately answer questions about the content
of videos, requiring a deep understanding of both visual elements and their
temporal relationships. This task involves recognizing objects and actions
along with comprehending complex narratives and interactions depicted in the
video. As a result, VQA requires multimodal integration techniques to
effectively combine visual, textual, and auditory information. The complexity
of VQA is further amplified by the diversity of video content and the
questions, making it a vibrant area of research with wide-ranging applications
in entertainment, education, and human-computer interaction.

Figure 1: Speaking Turn Sampling (STS) and Vision-Language Cross
Contextualization (VLCC) in action. In the top dotted rectangle, the audio
modality is used to obtain the speaking turn intervals, contributing to our
STS. These intervals are used to obtain the lower dotted rectangle, which
contain the corresponding video frames and transcript excerpts. These are used
in tandem in the model, to obtain jointly contextualized vision-language
embeddings.

Artificial social intelligence (ASI) [38, 2, 5, 11, 7] requires the ability to
perceive, interpret, and generate combinations of social cues to operate
effectively within human-agent teams. In this work, we adopt the definition of
social intelligence as outlined in [43], where the questions and answers in
the Social IQ dataset are designed to measure specific criteria of social
intelligence.

Training a SIQA system is challenging. Firstly, there is a strong need for
multimodal understanding to answer questions of this nature. For example, a
board meeting with some happy stakeholders and some angry stakeholders having
a discussion, or two friends having a sarcasm filled conversation, no single
modality can capture the whole situation. Large Language Models (LLMs) are a
powerful tool for capturing the intricacies of human language constructs, but
they face limitations when questions require information beyond language.
Incorporating multiple modalities into the input is essential, but fine-tuning
LLMs to use these inputs efficiently and avoid language bias presents
challenges. Furthermore, current methods for aligning language and vision
modalities often rely on trivial assumptions, highlighting the need for more
efficient alignment techniques. LLMs have very strong language priors, and to
incorporate other modalities, we need to align the corresponding elements of
the multiple modalities and subsequently fuse these aligned elements followed
by fine-tuning the LLM for using these inputs efficiently. In works like [23],
we have seen the challenges that one can face in fine-tuning LLMs.
Furthermore, the LLM might also end up short-cutting the secondary and
subsequent modality inputs and rather focus only on the language modality
inputs [33]. This is exacerbated by the fact that the LLM is unaware that it
is hallucinating and should use other modalities to improve performance.

In addition to using multiple modalities to enhance the quality of LLM
answers, we also need to consider the very nature of human interactions. When
a person talks, there is an inherent ‘attention’ between the speaker and the
listener. There is a need to facilitate efficient alignment between modality
elements which we capture through the above-described method of inherent
attention in human interactions, called ‘speaking turns’ [9, 15].

These high-level challenges translate to harder representation learning and
context addition. To perform well in this task, the system will need to learn
nuanced representations from the multimodal data that are relevant to the
query and will also need to capture any contextual cues from different
temporal frames that are necessary to understand causal aspects and
interactions that lead to a certain social situation.

Our approach to this challenge, as seen in Fig. 1, can be briefly described as
follows.

  * •

Vision and language modality elements’ explicit alignment, with the help of
audio modality based speaking turn sampling as a bridge between the video and
text inputs.

  * •

Vision and language cross-contextualization for better representation fusion,
by fusing the CLIP embeddings for vision and language modality elements and
projecting the consequent contextualized embedding to language space.

##  2 Related Work

With a rise in multi-modal applications of Large Language Models (LLMs) [35,
12, 4, 34], there is an increased need for a deep dive into the performance in
human interaction settings and to make sure that the model is not falling prey
to mode collapse. However, there have been few attempts in the literature to
improve the performance of these models in tasks that require social
intelligence [45].

In the context of ASI, as stated in [38], difficulties remain in mapping
natural language processing as symbols in an agent model to objects and
situations in their environment. Proper alignment between vision and language
in multimodal LLMs (MLLM) is a known shortcoming of even the state of the art
MLLMs [33, 44, 27, 31, 17].

The multimodal nature of ASI. Various research approaches rely on the
importance of the interaction between modalities. For example, to explain a
scene we see the importance to jointly predict interactions between all
characters in movies [18]. These predictions are based on both visual cues
(from video scenes) and dialog cues (from character conversations). Joint
modeling in [18] seeks to explain how social situations can only be achieved
by modeling interactions and relationships jointly. For instance, characters
in a romantic movie might evolve from being strangers to friends to lovers,
similarly, some interactions are visually expressed (e.g., running together),
while others are driven by dialog (e.g., confessing feelings). Although this
study is relevant to the ASI task, the Social-IQ 2.0 dataset is more
challenging than video from movies, since it is built from in the wild video
interactions, as well as scenes from movies.

Importance of visual cues for ASI. The importance of event detection that
comes from different modalities than language can make a difference between
success and failure to capture the right information for the task. For
instance if a person or character is not appearing on screen, then the vision
modality would not be present when trying to capture relevant information [19,
6]. This problem can be exacerbated with random frame sampling from video,
where the character might appear at times different than the ones fed into the
multimodal LLM as input. In addition, cross-modal interactions are
particularly important in video question answering tasks, with several
approaches having been tested in the past [22, 13, 14]. Furthermore, datasets
like VoxCeleb[24], MAV-Celeb[26] have studied the importance of obtaining
useful cues from audio for complementing the vision modality.

SOTA models for Social-IQ. Previous models have attempted to solve the task
set forth by the previous (first) version of the Social-IQ dataset. In
previous work by [36], each clip in the Social-IQ dataset was first divided
into speaking turns, and each speaking turn was encoded into a fully connected
graph through contrastive learning. The speaking turn representations were
then used in a supervised fine-tuning phase on question answering, achieving
an accuracy of 72%. In [25], external common sense is introduced to the model
through the COMET architecture, and an accuracy of 84.83% is achieved, also on
the first version of the Social-IQ dataset. It is important to note that these
results are for the previous iteration of the dataset, and that the current
version has been extended, and the task has been made more difficult. For
example, more videos have been added into Social-IQ 2.0, and only one choice
anwser among four is considered correct in the new version, whereas in the
first version, approximately half the choices were consireded correct.

There are also recent works that use the Social-IQ 2.0 dataset. In [39], they
introduce the Multi-Modal Temporal Correlated Network with Emotional Social
Cues (MMTC-ESC), which uses contrastive learning by using emotional social
cues and achieves. In [28], the Just Ask Plus model, a transformer based
architecture combining transcripts and video is trained on Social-IQ 2.0
dataset for zero-shot inference. In another work [40], they address an issue
more relevant to our work, which is efficient video chunk retrieval. Given a
question (query) and a long video, this model attempts to identify the most
relevant K video chunks and uses their associated visual tokens for training.
Lastly, in [20], to solve the problem of excessive visual tokens in long video
VQA, an Interactive Visual Adapter is used, which contains a lightweight
temporal frame selector and a spatial feature interactor within the internal
blocks of LLMs to capture instruction-aware and fine-grained visual signals.

Novelties of current work. In this work, we introduce speaking turn informed
video frame sampling with contextualization of image embeddings with text. We
use the Frozen BiLM architecture for video question answering. Here, the video
frames are passed through a video decoder (CLIP architecture), and projected
to language space through a linear projection layer. Instead of using the CLIP
video embeddings from equidistant video frames, we introduce two
modifications: first, we sample the video frames from speaking turns, with
sampling weights proportional to their duration. The hypothesis is that these
frames are more relevant for SIQA tasks. Second, we extract the transcripts
from these speaking turns and use the corresponding CLIP embeddings to
contextualize the video frame embeddings. The speaking turn information is
extracted from audio using the PyAnnote speaker diarization library [3].

##  3 Methodology

In this section, we shall describe the data analysis and the baseline model
used for the SIQA task followed by our approach that leads to better multi-
modality alignment and subsequent fusion.

###  3.1 Dataset and Analysis

In this section, we present the details of the SocialIQ 2.0 [37] dataset which
follows the guidelines for measuring social intelligence. The Social-IQ 2.0
dataset consists of 1,400 social in-the-wild videos annotated with 8,076
questions and 32,304 answers (4 answers per question, 3 incorrect, 1 correct).
We use this dataset which is the only dataset that captures social
intelligence in the VQA setup according to [45].

Modalities:  The dataset, includes videos (mp4), audio (mp3, wav) and
transcripts (vtt) .

Split:  We use the original provided split, which is:

  1. 1.

Train 5599 annotated questions.

  2. 2.

Validation 876 annotated questions.

  3. 3.

Test 1577 annotated questions.

We use the validation set to report all our evaluation metrics since the test
set is a part of the Social-IQ2.0 challenge and the corresponding ground truth
labels have not been released publicly.

Figure 2: Example videos and questions in Social-IQ 2.0 dataset [37], a video
contains multiple questions, four options where one is correct and three are
incorrect.

The average length of the questions is 10.87 words (some examples are
mentioned in Fig. 2) and that of answers is 10.46 words. The long average
length of answers enables researchers to capture more nuances.

We further investigate the visual modality of the data to understand it in
depth. We used Deepface [32] to analyze dominant emotions and ethnicity in all
videos. We found that the range of emotions shows a bias towards sad emotions.
We also find that the most dominant ethnicity is Caucasian, which represents
approximately 85% of the total representation of all individuals, which calls
for a more inclusive dataset.

To further understand the requirement of visual modality, we undertook a human
analysis on a random subset of videos to see if video is required to answer
the given questions or if the model behaves correctly by overfitting on
language. A group of 4 human annotators manually went through a total of 100
unique videos (20 per member and 20 common across members) and attempted the
Question-Answering task. The detailed process is as follows :

  1. 1.

Sample a total of 100 unique videos from the train and validation set at
random

  2. 2.

Assign 20 unique videos to each member along with 20 common videos to assess
agreement amongst the team members

  3. 3.

Use the audio, vision, and language modality to answer 2 questions per video
(a total of 200 unique questions)

  4. 4.

Report predicted answer and the need to use video to answer the question
metrics

In Tab. 1, we show the level of agreement achieved between the 4 members of
our group when answering the sampled questions. The table also reports the
need for video modality to answer that question. We can see a high level of
50% agreement on the answer. However, when evaluating the need for video
modality, full agreement is 52.2% and 85.4% level of agreement of half of the
members. This shows that, at a qualitative level, human analysis deems most
question-answering tasks in this dataset need the video component to be
answered correctly.

Aspect |  Full Agr. |  50% Agr.
---|---|---
Answers |  0.475 |  0.95
Video Use |  0.522 |  0.854
Table 1: Human Analysis to find out the requirement of videos in the dataset.
The reported number corresponds to the Full Aggrement and the 50% Aggrement,
respectively. A higher agreement means that video usage is more needed to
answer the questions.

###  3.2 SIQA Foundational model

Social intelligence cues are best obtained from a combination of verbal and
visual cues. One can further bifurcate verbal cues into the literal spoken
content and the way it was spoken, specifically natural language and audio. We
build on FrozenBiLM [42] in our work and address issues that not only affect
FrozenBiLM in particular, but rather the task of SIQA itself.

Base model design. FrozenBiLM is a model that builds on frozen Bidirectional
Language Models (BiLM) such that it can be used for zero-shot VQA as well as
fine-tuned VQA for a variety of downstream tasks. The model uses Masked
Language Modeling (MLM) [10] to map the output of the constituent LLM to a
vector of logits that represent a categorical distribution over the
Vocabulary. The language model is assumed to be pre-trained over a large set
of textual data from the web, based on the standard MLM objective. For
providing the visual modality as input to a language model, the pre-trained
frozen video encoder CLIP [30] is used to generate text-contextualized video
embeddings. Subsequently, a linear mapping adapter was used to project visual
embeddings into the text token embedding space.

Putting together, the input to the LLM is the text token embeddings of the
question, of a candidate answer followed by the answer mask, of the
transcript, and the linear projected visual encodings. Since the SIQ 2.0
dataset deals with multiple choice question answering, we train the model to
predict the ‘answer-mask’ as yes/no. Thus, we train the model with the
following loss :

$\mathcal{L}_{\mu}(x,y)=\frac{1}{M}\sum_{m}log(p_{\mu}(\tilde{x},y)_{m}^{x_{m}})$
(1)

where $\tilde{x}$ is the corrupted text sequence, $y$ is the sequence of video
frames, $p_{\mu}(\tilde{x},y)_{m}^{x_{m}}$ is the probability that the masked
token $m$-th token in $\tilde{x}$ is $x_{m}$, and $M$ is the number of masks
in the sequence $\tilde{x}$. Note that $\mu$ refers to the trainable
parameters of the model. This loss is for cross-modal training. The text data
(transcript) are altered by masking some tokens, and the models have to
predict these tokens based on the surrounding words and video inputs.

A fitting choice for SIQA. The benefit of using a model that is largely frozen
is having a modular architecture resulting in design flexibility and allowing
for multiple downstream tasks such as multiple-choice QA, open-ended QA, zero-
shot predictions, etc. Furthermore, this design choice also handles the
problem of Catastrophic Forgetting [8] since the majority of our model is
frozen and certain components, such as the adapters, can be fine-tuned for
task/data-specific objectives. To facilitate task/data-based fine-tuning, the
model is interspersed with multi-layer perceptron adapters along with fine-
tuning-enabled normalization layers in the BiLM so that the vast majority of
the model can be frozen but still be domain-adaptable.

Prompt Engineering. A roadblock to using language models for long video and
textual inputs is the limited context size that can be fed to the model.
However, to capture the dynamics of social interactions, we require the
ability to feed long transcripts in the SIQ2 dataset as context to the model.
For this purpose, we create overlapping bifurcated context prompts and pass
them to the model in subsequent feed-forward passes and collate the result
from these multiple feed-forward passes.

###  3.3 Multi-modal alignment: Listen to the video

The need for better multimodal alignment. We define multimodal alignment [21]
as identifying the connections between the different elements across
modalities. These elements can be discrete or continuous. We discretize the
visual and language modalities with the help of corresponding audio and the
consequent speaking turns [36].

Video contains a lot of information such as active characters, passive
background characters, camera panning, actions, etc. Not all the information
might be relevant for identifying the social dynamics of the scene. However,
current methods to extract information from the video are to either (1) use a
pre-trained frozen video encoder or (2) a fine-tuned video encoder to generate
video embeddings and then sample the video features at a specific granularity,
such as regular intervals. However, there are no guarantees of how much usable
information for a specific question would be present in the sampled video
embeddings.

Speaking Turn Sampling for SIQA. As mentioned earlier, our strategy of
discretizing various modality elements would be the speaking turns. Hence, we
use a third modality, audio, to provide attention to the visual and language
modality discretization process. We use an off-the-shelf speaker diarization
module [29] to identify the time stamps with conversations between the active
characters in the video. We then map the speaker diarization information to
the transcript to identify the language modality element for a given speaking
turn. Similarly, we find the corresponding visual modality element for the
given speaking turn. Note that this method is guaranteed to obtain visual
elements with conversation information, if present in the view, between at
least two individuals. Through this proposed approach, we perform explicit
cross-modality alignment.

Furthermore, the corresponding language modality element for each selected
visual modality element will always be present in the input to the model. This
is in contrast with methods in existing works, wherein no guarantees can be
made for a visual modality element always being present in tandem with a given
language modality since the sampling strategy for the visual modality might
simply be disconnected from the language modality.

To illustrate the concept better, let us consider an example interaction setup
in a sparsely occupied diner. To capture the interactions happening across the
diner, the camera would have to pan from one position to the other quite a
lot. Thus, the dialogues would be highly interspersed with camera panning.
Equidistant frame sampling would probabilistically end up capturing more of
such ‘empty’ dialogue frames. In contrast, our approach, Speaking Turn
Sampling (STS) would find the active conversation intervals within the video
duration and would return sampled frames only from such locations.

Formally, let $I^{(i)}_{k}$ be the $k^{th}$ speaking turn interval in the
audio of video $i$, and $K^{(i)}$ be the number of speaking turns in the
$i^{th}$ video. The video frame sampling and contextualization pre-processing
steps would take a certain number of equidistant frames $V^{(i)}_{k}$ from
this interval, as well as the transcript $L^{(i)}_{k}$ of what is spoken in
that speaking turn. The number of frames sampled from $I^{(i)}_{k}$ is

$N^{(i)}_{k}=\frac{I^{(i)}_{k}}{\sum_{j=1}^{j=K^{(i)}}I^{(i)}_{j}}M$ (2)

where $M$ is the total number of video frames that are fed to the LLM (M is a
hyperparameter). Whenever no speaking turns are detected in a video, we
default back to equidistant video frame sampling. Given a limit M of video
frames that can be fed to the LLM, we move from the original, temporally
equidistantly sampled video frames $\\{V^{(i)}_{0},...,V^{(i)}_{N}\\}$, to
pairs of frames and context transcripts
$\bigcup_{k=1}^{k=K}\\{(V^{(i)}_{k,t},L^{(i)}_{k})_{t=1}^{t=N^{(i)}_{k}}\\}$
that are achieved through our sampling strategy. Note that $L^{(i)}_{k}$ does
not have the $t$ subscript, since we use the transcript for the entire
$k^{th}$ speaking turn in every video frame-transcript pair belonging to the
same speaking turn.

Figure 3: Speaking Turn Informed Video Frame Sampling Strategy: We focus the
sample of the frames only where the people is speaking.

###  3.4 Modality Fusion: See the video

Figure 4: The figure displays the proposed architecture. We run the Speaking
Turn Sampling (STS) module to the aligned $frame_{i}$ from the speaking turn k
and the corresponding subtitle from the transcript. We pass this pair to the
frozen CLIP encoder to obtain the visual and text encodings respectively. The
resultant encodings are passed through the Vision Language Cross
Contextualization (VLCC) module and subsequently passed through the projection
layer to generate one of the inputs to the LLM. Simultaneously, we generate
the text embeddings of size U for each question-answer pair, and the text
embeddings of size V for the video subtitles.

Need for better Representation Fusion We define multimodal representation
fusion [21] as the task of creating a joint representation of the individual
modality elements post-successful alignment such that the joint representation
integrates cross-modal interaction information. Existing methods typically
focus on adapter creation to project one modality to another modality’s
domain. In the case of LLMs, existing methods project visual modality to
language modality and subsequently feed the projected embeddings to the LLMs
for inference.

Existing methods, such as [42] use CLIP [30] to generate video embeddings and
subsequently pass them through an adapter to project them into the language
domain. Finally, these projected embeddings are given as input to the LLM
along with the question, answer-mask, and transcript as context. However, from
our experiments (see Tab. 3), we observe that not only is the SIQ2 fine-tuned
FrozenBiLM model not using the visual modality to make any inference, but in
fact is getting rather confused by vision input. Using this result, along with
other experimentation in Tab. 3, we deduce that the pre-trained LLM along with
the learnings from fine-tuning on the given dataset, the language foundational
model develops strong priors on the questions’ domain and can use that
information to answer a majority of the test questions correctly. However,
these priors can also lead to hallucinations in downstream tasks, resulting in
reduced reliability on the system. Thus, our approach proposes ways to
mitigate the dependence on the priors and encourage the language foundational
model to refer to the vision modality for more accurate inferences.

Vision-Language Cross Contextualization for SIQA Our alignment strategy,
Speaking Turn Sampling (STS), can translate into better performance as well as
prior-dependency mitigation benefits with the help of successful
representation fusion, as shown in Fig. 4. We propose a Vision-Language Cross
Contextualization (VLCC) module that encapsulates vision embeddings within
language embeddings such that the LLM is made to give equal attention to
vision modality as well, which it typically in existing works, learns to
ignore.

The VLCC setup is now formally defined. Let
$f_{\theta}:(V,L)\mapsto\mathbb{R}^{d}\times\mathbb{R}^{d}$ be the frozen CLIP
architecture, which takes video frame input $V$ and text input $L$ as a pair,
and outputs a pair of contextualized embeddings in $\mathbb{R}^{d}$, where $d$
is the embedding length hyperparameter. In our setting, we take a speaking
turn’s video frame and transcript and pass it through the frozen CLIP module
to get

$f_{\theta}(V^{(i)}_{k,t},L^{(i)}_{k})\mapsto(A_{k,t}^{(i)},B_{k}^{(i)})$ (3)

where $A_{k,t}^{(i)}$ is the visual embedding of $V^{(i)}_{k,t}$, and
$B_{k}^{(i)}$ is the text embedding of $L^{(i)}_{k}$. After these embeddings
are generated by the frozen CLIP module, they are linearly combined and passed
through a trainable adapter $g_{\phi}:\mathbb{R}^{d}\mapsto\mathbb{R}^{d}$, to
get

$g_{\phi}(A_{k,t}^{(i)},B_{k}^{(i)})\mapsto C_{k,t}^{(i)}$ (4)

With $C_{k,t}^{(i)}\in\mathbb{R}^{d}$ projected to the language space by the
linear adapter, it is subsequently fed to the DeBERTa-v2 [16] LLM
architecture.

##  4 Results

###  4.1 Primary evaluation metrics

Our proposed approach aims to improve (1) achieved performance on the SIQA
task, and (2) improvement in cross-modal alignment and representation fusion
for a more comprehensive multimodal system. For the former, we compute the
performance of our approach on the SIQ 2.0 evaluation dataset split, and for
the latter, we measure the difference in achieved performance on the SIQ 2.0
evaluation dataset split for the correct set of input vs. various types of
deformed vision/language inputs.

Figure 5: The question asked in this video is ”What is the tone of the people
speaking?”. This example shows that our method (in the green box) uses more
relevant frames where people are speaking. In contrast, the baseline (in the
red box) samples frames that do not contain relevant information for the task.
In this example, our model predicts the correct answer, whereas the baseline
does not.

From our results in Tab. 2 we show that we improve on other existing works by
3.89%. Our results, 82.06 % accuracy, show that we achieve state-of-the-art
performance on Social-IQ 2.0, outperforming previous attempts to solve the
SIQA problem, as is demonstrated in the example in Fig. 5.

Model |  Accuracy
---|---
Just-Ask [41] |  52.12 %
Just-Ask-Plus [28] |  53.35%
IVA [20] |  68.0%
MMTC-ESC [39] |  75.94%
R-VLM [40] |  65.65%
FrozenBiLM [42] |  78.17%
STS & VLCC (Ours) |  82.06%
Table 2: Comparison with the other video-language models evaluated on Social-
IQ 2.0.

###  4.2 Ablation experiments

Based on our second measure for performance improvement described before, we
present our comparison in Tab. 3 where we have values that represent the
change in accuracy from the actual performance if the modalities are altered.
Here, $\Delta_{1}$ is the difference in accuracy when videos have defaced
characters, $\Delta_{2}$ is the difference in accuracy when we remove the
video from the LLM input, and $\Delta_{3}$ is the difference in accuracy when
the transcript is set to the word ‘gibberish’. Two of these metrics,
$\Delta_{1}$ and $\Delta_{2}$, enable us to evaluate the dependency of our
model on the visual modality. Note that a higher $\Delta_{1}$ and $\Delta_{2}$
will mean more visual attention since the predictions then rely more on the
video inputs. $\Delta_{3}$ helps us understand the contribution of video
subtitles as language input and similarly a higher $\Delta_{3}$ value
indicates higher reliance of the model on the transcripts.

|  Correct Video Inputs $\uparrow$ |  Defaced Video Inputs |  $\Delta_{1}$ $\uparrow$ |  Blank Video Embeddings |  $\Delta_{2}$ $\uparrow$ |  Gibberish Transcript |  $\Delta_{3}$ $\uparrow$
---|---|---|---|---|---|---|---
Frozen-BiLM: Fine-tuned |  78.17% |  76.57% | 1.60% |  78.40% | -0.23% |  74.29% | 3.88%
STS & VLCC (Ours) |  82.06% |  78.97% | 3.09% |  76.34% | 5.72% |  76.68% | 5.38%
Table 3: Ablation experiments, comparison of the FrozenBiLM zero shot,
FrozenBiLM finetuned on Social-IQ 2.0, and STS & CC (Ours). The experiments
include Correct Video Input, Defaced Video Inputs, where images are faces are
anonymized on the videos; Blank Video EEmbeddings,where frames are zeroed; and
Gibberish Transcript, where a single word, ”Gibberish”, is given as
transcript.

####  4.2.1 Increasing the model dependency on the visual modality

The dependency on video inputs, as indicated by $\Delta_{1}$ and $\Delta_{2}$
is intentional and achieved through strategic modifications in the model
architecture and training process, emphasizing the importance of the visual
modality.

The $\Delta_{1}$ value of 3.09% for our approach vs. 1.60% for the baseline
model indicates that our model has an increased ability to decipher and
utilize facial expressions within the video content, which has bolstered its
performance on the SIQA task, as expected. Similarly, a $\Delta_{2}$ value of
5.72% for our approach vs. -0.23% for the baseline proves the overall
capability of our approach to effectively use multimodal data. For the
baseline, we see that the model is utilizing just the language priors and
hence the negative $\Delta_{2}$ value.

####  4.2.2 Increasing the model dependency on language modality

The $\Delta_{3}$ value of 5.38% for our approach vs. 3.88% for the baseline
shows that our approach is beneficial for increasing the usage of the language
inputs. This exhibits a clear indication that our approach helps reduce the
over-dependency of the LLM on question-answer priors. By deliberately
increasing the model’s dependency on the transcripts, we aim to enhance the
ability to contextualize and interpret the subtitles of social interactions
that are often conveyed through dialogue and textual cues. This increased
dependency is achieved through advanced natural language processing techniques
and training strategies that prioritize the integration of linguistic
information with visual data.

Thus, our approach is a balanced and effective multimodal system capable of
leveraging both visual and linguistic inputs to achieve superior performance
on the SIQA task.

##  5 Discussion

###  5.1 Limitations

To reproduce our approach on a different dataset, the dataset needs to have
paired audio and video information. The audio is required to extract the
Speaking Turn information and perform both sampling and contextualization.
Furthermore, having such paired information is mostly helpful only in a social
interaction setting.

###  5.2 Overall improvement

The advancement in model performance, as demonstrated in our results section
Tab. 2, particularly a remarkable 3.89% increase in accuracy over the existing
state-of-the-art, underscores the effectiveness of our approach in addressing
the challenge of Social-IQ 2.0. The STS & VLCC model, our novel contribution,
achieves an accuracy of 82.06%, setting a new milestone for the SIQA task.
This leap in performance is attributed to our model’s enhanced cross-modal
alignment and representation fusion capabilities, enabling it to interpret and
integrate visual and linguistic inputs more effectively than prior models. Our
approach not only excels in accurately understanding social interactions
depicted in video content but also demonstrates a significant improvement in
handling complex multimodal data. This overall enhancement is crucial for
applications that require nuanced understanding of human social behaviors,
paving the way for more sophisticated and context-aware AI systems.

###  5.3 Anticipating unintended consequences of our work

An SIQA system, while designed to improve human-computer interaction and
provide contextually relevant responses, could have certain potential
unintended impacts. One such consequence is the propagation of the inherent
biases in the training data, Sec. 3.1, leading to potentially unfair or
discriminatory responses in downstream tasks. Furthermore, our ablations, Sec.
4.2, attempt to explain the various motivations and consequences of our
approach. However, more comprehensive methods should be employed before
deploying the approach in systems to instill more confidence in the prediction
outcomes.

##  6 Conclusions

In conclusion, our novel approach of STS & VLCC improved task performance of
social question answering as well as increased the dependence on both language
and video modalities. There is still work to be done in better using visual
and audio modality and removing dependence of LLM from question and answer
pairs which we feel falls in the future directions

## References

  * [1] Sts & vlcc official repository.  https://github.com/sts-vlcc/sts-vlcc.git.
  * Bainbridge et al. [1994] William Sims Bainbridge, Edward E Brent, Kathleen M Carley, David R Heise, Michael W Macy, Barry Markovsky, and John Skvoretz.  Artificial social intelligence.  _Annual review of sociology_ , 20(1):407–436, 1994.
  * Bredin et al. [2020] Hervé Bredin, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, and Marie-Philippe Gill.  Pyannote. audio: neural building blocks for speaker diarization.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 7124–7128. IEEE, 2020.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Castelfranchi [1998] Cristiano Castelfranchi.  Modelling social action for ai agents.  _Artificial intelligence_ , 103(1-2):157–182, 1998.
  * Chung and Zisserman [2017] Joon Son Chung and Andrew Zisserman.  Out of time: automated lip sync in the wild.  In _Computer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13_ , pages 251–263. Springer, 2017.
  * Dautenhahn [2007] Kerstin Dautenhahn.  _A paradigm shift in artificial intelligence: why social intelligence matters in the design and development of robots with human-like intelligence_.  Springer, 2007.
  * Delange et al. [2021] M. Delange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pages 1–1, 2021.
  * Denny [1985] Rita Denny.  Marking the interaction order: The social consitution of turn exchange and speaking turns.  _Language in Society_ , 14(1):41–62, 1985.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Dhelim et al. [2021] Sahraoui Dhelim, Huansheng Ning, Fadi Farha, Liming Chen, Luigi Atzori, and Mahmoud Daneshmand.  Iot-enabled social relationships meet artificial social intelligence.  _IEEE Internet of Things Journal_ , 8(24):17817–17828, 2021.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
  * Gao et al. [2019] Peng Gao, Haoxuan You, Zhanpeng Zhang, Xiaogang Wang, and Hongsheng Li.  Multi-modality latent interaction network for visual question answering.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019.
  * Han et al. [2021] Dezhi Han, Shuli Zhou, Kuan Ching Li, and Rodrigo Fernandes de Mello.  Cross-modality co-attention networks for visual question answering.  _Soft Computing_ , 25:5411–5421, 2021.
  * HARRIGAN [1985] JINNI A. HARRIGAN.  Listeners’ body movements and speaking turns.  _Communication Research_ , 12(2):233–250, 1985.
  * He et al. [2021] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen.  Deberta: Decoding-enhanced bert with disentangled attention.  In _International Conference on Learning Representations_ , 2021.
  * Koh et al. [2023] Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried.  Grounding language models to images for multimodal inputs and outputs.  2023\.
  * Kukleva et al. [2020] Anna Kukleva, Makarand Tapaswi, and Ivan Laptev.  Learning interactions and relationships between movie characters.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9849–9858, 2020.
  * Lei et al. [2020] Jie Lei, Licheng Yu, Tamara L. Berg, and Mohit Bansal.  What is more likely to happen next? video-and-language future event prediction, 2020.
  * Li et al. [2024] Yunxin Li, Xinyu Chen, Baotain Hu, and Min Zhang.  Llms meet long video: Advancing long video comprehension with an interactive visual adapter in llms.  _arXiv preprint arXiv:2402.13546_ , 2024.
  * Liang et al. [2022] Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency.  Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions.  _arXiv preprint arXiv:2209.03430_ , 2022.
  * Liu et al. [2023] Yang Liu, Guanbin Li, and Liang Lin.  Cross-modal causal relational reasoning for event-level visual question answering.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * Luo et al. [2023] Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang.  An empirical study of catastrophic forgetting in large language models during continual fine-tuning, 2023.
  * Nagrani et al. [2017] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman.  Voxceleb: a large-scale speaker identification dataset.  _arXiv preprint arXiv:1706.08612_ , 2017.
  * Natu et al. [2023] Sanika Natu, Shounak Sural, and Sulagna Sarkar.  External commonsense knowledge as a modality for social intelligence question-answering.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3044–3050, 2023.
  * Nawaz et al. [2021] Shah Nawaz, Muhammad Saad Saeed, Pietro Morerio, Arif Mahmood, Ignazio Gallo, Muhammad Haroon Yousaf, and Alessio Del Bue.  Cross-modal speaker verification and recognition: A multilingual perspective.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 1682–1691, 2021.
  * Peng et al. [2023] Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei.  Kosmos-2: Grounding multimodal large language models to the world.  _arXiv preprint arXiv:2306.14824_ , 2023.
  * Pirhadi et al. [2023] Mohammad Javad Pirhadi, Motahhare Mirzaei, and Sauleh Eetemadi.  Just ask plus: Using transcripts for videoqa.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3082–3085, 2023.
  * Plaquet and Bredin [2023] Alexis Plaquet and Hervé Bredin.  Powerset multi-class cross entropy loss for neural speaker diarization.  In _Proc. INTERSPEECH 2023_ , 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Rasheed et al. [2023] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Erix Xing, Ming-Hsuan Yang, and Fahad S Khan.  Glamm: Pixel grounding large multimodal model.  _arXiv preprint arXiv:2311.03356_ , 2023.
  * Serengil and Ozpinar [2021] Sefik Ilkin Serengil and Alper Ozpinar.  Hyperextended lightface: A facial attribute analysis framework.  In _2021 International Conference on Engineering and Emerging Technologies (ICEET)_ , pages 1–4. IEEE, 2021.
  * Tong et al. [2024] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie.  Eyes wide shut? exploring the visual shortcomings of multimodal llms.  _arXiv preprint arXiv:2401.06209_ , 2024.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Vaswani et al. [2023] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need, 2023.
  * Wilf et al. [2022] Alex Wilf, Martin Q. Ma, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency.  Face-to-face contrastive learning for social intelligence question-answering, 2022.
  * Wilf et al. [2023] Alex Wilf, Leena Mathur, Sheryl Mathew, Claire Ko, Youssouf Kebe, Paul Pu Liang, and Louis-Philippe Morency.  Social-iq 2.0 challenge: Benchmarking multimodal social understanding.  https://github.com/abwilf/Social-IQ-2.0-Challenge, 2023.
  * Williams et al. [2022] Jessica Williams, Stephen M Fiore, and Florian Jentsch.  Supporting artificial social intelligence with theory of mind.  _Frontiers in artificial intelligence_ , 5:750763, 2022.
  * Xie and Park [2023] Baijun Xie and Chung Hyuk Park.  Multi-modal correlated network with emotional reasoning knowledge for social intelligence question-answering.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3075–3081, 2023.
  * Xu et al. [2023] Jiaqi Xu, Cuiling Lan, Wenxuan Xie, Xuejin Chen, and Yan Lu.  Retrieval-based video language model for efficient long video question answering.  _arXiv preprint arXiv:2312.04931_ , 2023.
  * Yang et al. [2021] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid.  Just ask: Learning to answer questions from millions of narrated videos.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 1686–1697, 2021.
  * Yang et al. [2022] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid.  Zero-shot video question answering via frozen bidirectional language models.  _Advances in Neural Information Processing Systems_ , 35:124–141, 2022.
  * Zadeh et al. [2019] Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency.  Social-iq: A question answering benchmark for artificial social intelligence.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8799–8809, 2019.
  * Zhao et al. [2023] Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, and Bingyi Kang.  Bubogpt: Enabling visual grounding in multi-modal llms.  _arXiv preprint arXiv:2307.08581_ , 2023.
  * Zhong et al. [2022] Yaoyao Zhong, Junbin Xiao, Wei Ji, Yicong Li, Weihong Deng, and Tat-Seng Chua.  Video question answering: Datasets, algorithms and challenges.  _arXiv preprint arXiv:2203.01225_ , 2022.