# ParaColorizer: Realistic Image Colorization using Parallel Generative
Networks

Himanshu Kumar*, Abeer Banerjee*, Sumeet Saurav, and Sanjay Singh *These
authors contributed equally to this work.The authors are with the Intelligent
Systems Group of Central Electronics Engineering Research Institute (CSIR-
CEERI) Pilani, Rajasthan<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Grayscale image colorization is a fascinating application of AI for
information restoration. The inherently ill-posed nature of the problem makes
it even more challenging since the outputs could be multi-modal. The learning-
based methods currently in use produce acceptable results for straightforward
cases but usually fail to restore the contextual information in the absence of
clear figure-ground separation. Also, the images suffer from color bleeding
and desaturated backgrounds since a single model trained on full image
features is insufficient for learning the diverse data modes. To address these
issues, we present a parallel GAN-based colorization framework. In our
approach, each separately tailored GAN pipeline colorizes the foreground
(using object-level features) or the background (using full-image features).
The foreground pipeline employs a Residual-UNet with self-attention as its
generator trained using the full-image features and the corresponding object-
level features from the COCO dataset. The background pipeline relies on full-
image features and additional training examples from the Places dataset. We
design a DenseFuse-based fusion network to obtain the final colorized image by
feature-based fusion of the parallelly generated outputs. We show the
shortcomings of the non-perceptual evaluation metrics commonly used to assess
multi-modal problems like image colorization and perform extensive performance
evaluation of our framework using multiple perceptual metrics. Our approach
outperforms most of the existing learning-based methods and produces results
comparable to the state-of-the-art. Further, we performed a runtime analysis
and obtained an average inference time of 24ms per image.

###### Index Terms:

Information Restoration, Realistic Colorization, Parallel GANs, ResUNet, Image
Fusion

##  I Introduction

Grayscale image colorization is a well-known problem in the field of computer
vision. Due to the fundamentally ill-posed nature of this problem, the
solutions can be multimodal [1], i.e., there might not be a particular
solution for this task. There can be multiple plausible output images for a
single input grayscale image (e.g., a hat can be red, blue, green, etc.).
Potential applications of colorization can range from reviving the authentic
significance from historical images and colorizing old black-and-white movies,
to information compression. Although there have been multiple approaches for
user-guided image colorization, fully automatic image colorization still
remains a challenging research problem. Few of the results achieved using our
approach are shown in Figure 1.

Figure 1: Samples of colorized outputs using our method. The outputs are
displayed as obtained without any further enhancements. Image courtesy:
shorturl.at/ckpK0 and shorturl.at/lzY29.

Earlier work in this field used a human-level interpretation-guided method [2,
3, 4, 5, 6, 7] to colorize the whole image. Primarily scribbles are applied on
a grayscale image as baseline structured and the whole image is colorized from
it. Using scribbles over an image is not a convenient task and it requires
guidance from expert artists. In the advancement of guided methods, the
reference-image based method [8, 9, 1, 10, 11, 12] is being used which removes
the dependency of expert artists by providing the reference image as a
baseline idea to colorize the whole image. Recent development in deep learning
[13] accelerates the predictive modelling-based image colorization by
generating the two missing channels. Generally, an image consists of a
foreground and a background, where the foreground consists of object(s) of
interest, and the background is the remaining image. Earlier deep-learning-
based methods [12, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25] performed
learning and colorization on the entire image which usually led to poor
performance in images with multiple objects but without clear figure-ground
separation. This problem was addressed by instance-aware colorization [26].
Although this method colorizes the objects properly in most cases, in some
cases, the context is missed and the background is left desaturated because a
single model for colorization of the whole image is insufficient at covering
the diverse data modes.

Figure 2: The selected images have cluttered/confusing backgrounds which make
the colorization task difficult even for existing state-of-the-art methods.
DeOldify [25] suffered from color bleeding and IAIC [26] had desaturated
background and both frameworks failed to capture the data diversity.

In this paper, we aim at the automatic colorization of grayscale images while
preserving contextual information using a novel deep-learning-based
adversarial training framework. Unlike the existing SOTA method [26] that uses
cropped objects from a large-scale common objects dataset to fine-tune a pre-
trained model on full images to obtain the object-level semantic information,
we deployed two parallel networks to colorize foreground and background
separately and an adaptive fusion network which results in the final colorized
output. Our main intuition was to employ multiple generators, instead of using
a single one which has been proven to be extremely effective at covering
diverse data modes while the whole image information is preserved [27]. Our
whole colorization framework consists of the following salient steps. First,
inspired by the unbalanced GAN approach, we pre-train the generators of both
the GAN pipelines before the adversarial training to prevent the discriminator
from winning too easily [28]. Second, adversarial training of a foreground
network using an off-the-shelf pre-trained object detector to detect the
potential objects corresponding to the images from the main dataset. Parallel
training of a background network for which, common real-world scene-based
examples are appended to the main dataset. Third, a deep-learning-based fusion
network inspired by DenseFuse [29], trained on the main dataset, is deployed
which adaptively merges the generated outputs of the foreground and background
network to generate the final colorized image. Our main contributions are:

  * •

A parallel adversarial training framework for edge-preserved image
colorization using Unbalanced GANs.

  * •

Separately reconfigured Residual-UNets specializing in foreground and
background colorization which preserves contextual information.

  * •

Design of a DenseFuse-based fusion network for feature-based adaptive fusion
of the generated parallel outputs.

  * •

Achieving realistic colorization results comparable to state-of-the-art with
substantially lesser training data.

The remaining contents of the paper are organized as follows: Section II
provides a quick overview of some of the most prominent ways for image
colorization using various methods like user-guided, example-based, and
learning-based image colorization. Section III elaborates our motivation and
methods for achieving realistic colorization. Section IV discusses the
different types of experiments performed to train a model that can produce
results close to the state-of-the-art. This section also discusses the
fundamental problems with the existing evaluation frameworks for image
colorization models. Section V provides an extensive ablation study of the
individual models, the loss functions used, and the fusion model. Section VI
provides an analysis of the average inference time of our model and the
computational resources used to reproduce the colorization results. Section
VII discusses the strengths and concerns of our method and states the failure
cases of our model. Finally, in Section VIII, we conclude this paper by
highlighting our key observations and we further discuss the direction of
future research on image colorization.

##  II Related Works

###  II-A User-guided colorization

These methods [2, 3, 4, 5, 6, 27] use human interaction to assign the initial
strokes or color points on a grayscale image that become the initial
parameters for colorizing the whole image. A method was proposed by Levin et
al. [2] in which the nearby pixels accounted for the same color based on the
similar luminance by solving a Markov random field to propagate scribble
colors. Scribble-based work continues with the advancement in similarity
metrics, for instance, An et al. [30] proposed an energy-optimization method
in which the initially coarse edits propagate to refined edits based on the
policy that similar edits are applied to spatially-close regions. Some methods
[3] involve edge-detection to prevent color bleeding problems or color
propagation based on color texture similarity [5, 6]. The hybrid method
proposed by Zhang et al. [17] in which both deep learning and scribble are
used that drastically reduced the human effort just by assigning the color
hints.

###  II-B Example-based colorization

Unlike the previous works, they used reference image-guided methods to
colorize grayscale images. The guided methods [8, 9, 1, 10, 11, 12, 20] are
broadly divided into global and local transfer methods. In the global transfer
method [31, 32, 5], the targeted result is matched with the reference image
using the global features like mean, variance and histogram. Since these
methods missed the low-level semantic pixel information the generated output
seems unrealistic. Local transfer methods [33, 34, 35] on the other hand,
preserves the low-level information at pixel level [8, 11], super-pixel level
[10, 21] and semantic segments level [9, 1]. Example-based methods rely on the
quality of the reference image that includes manual annotation of image region
[9, 12] which is a tedious process. To address this issue, learning-based
methods are used where the model learns the mapping from a large dataset [20]
to colorize the image and the video [36].

###  II-C Learning-based colorization

These methods [10, 37, 14, 15, 16, 17, 18, 19] are fully autonomous in which
the whole image is generated from the pixel level. Deep learning algorithm
used as the backbone in which end-to-end learning involves a large-scale
dataset such as e.g., ImageNet [38]. To colorize a grayscale image, broadly
two kinds of problems are addressed by the recently proposed method: semantics
and multi-modality [1]. The semantics-based method proposed by Iizuka et al.
[14] and Zhao et al. [19] uses two branched architectures to learn local image
features and global priors and fuse them to colorize the complete image.
Further, Zhang et al [16] proposed a cross-channel encoding scheme to preserve
the semantics. Larsson et al. [15] also achieved semantic interpretability by
pre-training the model for a classification task. To address the multi-
modality of the problem, recent works [15, 16, 19] proposed color distribution
at per pixel level and achieved plausible results in most cases but missed the
object level semantics.

To address this problem, Su et al. (IAIC) [26] proposed a method in which an
object detector is used for instance-based fine-tuning of the baseline.
Although the instances are colorized properly, the backgrounds are left
desaturated and the excellence of the model depends on the object detector’s
performance. We observed that obtaining the semantic information at the object
level while maintaining the foreground and background context is difficult to
achieve using a single deep-learning network. Thus, we employ parallel
networks specifically tailored at maintaining the foreground semantics and the
background scene separately.

###  II-D Image Colorization with GANs

Attempts have been made to colorize grayscale images using GAN-based
frameworks [39]. Vitoria et al. [40] proposed a self-supervised adversarial
model that performs colorization by using the perceptual and semantic
understanding of color, while DeOldify [25] uses progressive GANs for
colorization of historical images. Both of these methods have been included
for comparison with our approach. Style transfer-based anime sketch
colorization [41] has been performed using a Residual-UNet-based generator.
Parallel networks have been previously used by Johari et al. [42] for
generating images with similar color themes. Our method is unique and advanced
in several manners. First, we parallelly train two different GAN architectures
to cover the data diversity. (a) Foreground pipeline: Consists of a Residual-
UNet based generator with self-attention and is trained using an edge-aware
loss function. (b) Background pipeline: Works as a compensator to the
foreground model and is separately modified for colorizing common backgrounds.
Secondly, using an unbalanced training method [28] by pre-training the
generator to prevent the early convergence of the discriminator. Thirdly,
using a DenseFuse-inspired fusion network to obtain a final fused image that
performs the feature-based adaptive fusion of the parallelly generated
outputs.

Figure 3: The adversarial training pipeline. Generator $G_{1}$ and generator
$G_{2}$ are Res-UNets (Generator $G_{1}$ has self-attention enabled) that
accept L-channel at the input and predict the $(a,b)$ channels. The L-channel
is combined with the predicted $(a,b)$ channels and passed to PatchGAN
discriminators $D_{1}$ and $D_{2}$ in each forward pass. $L_{MSE}$,
$L_{EDGE}$, $L_{L1}$, $L_{ADV}$ denote the mean-squared loss, the edge-map
loss, the L1 loss, and the adversarial loss respectively.

##  III Method

The task of image colorization has been divided into two subtasks of
foreground and background colorization. The basic idea is to employ two
independently trained separately modified specialized models which parallelly
aim at colorizing the input grayscale image and the results are fused by a
fusion network to obtain the final colorized output. The complete training
pipeline is shown in Figure 3.

###  III-A Foreground Colorization

A subtask that mainly aims at effectively colorizing the objects in the image
is achieved by training the network on the full image along with their
corresponding object proposals. The main objective of doing this is to
simulate object attention which makes the network specialize at colorizing the
foreground. In order to simulate object attention, the object proposals were
obtained from the full images in the main dataset by making use of an off-the-
shelf pre-trained network, Mask R-CNN [43], as done in IAIC [26]. The full
images $x_{i}$ were passed through the MaskRCNN framework with ResNet50
backbone to detect the object bounding boxes and the corresponding cropped
object proposals denoted by $x_{j}$ were obtained.

Colorization Backbone Architecture: We use a GANbased framework for colorizing
the foreground which is heavily inspired by the Pix2Pix framework [18]. The
generator model employs a modified version of UNet presented in Pix2Pix. Our
generator is a UNet that uses a pre-trained ResNet-18 network as its encoder
for simplifying the training process. We have also enabled a self-attention
mechanism to model the long-range dependencies across image regions [44]. The
discriminator network is a PatchGAN as proposed in Pix2Pix which classifies
patches of the total image instead of classifying the whole image as real or
fake which helps to capture the local style statistics.

Loss Function and Training: We use a weighted combination of three loss
functions for training the foreground colorizer network. Firstly, the standard
mean-squared error loss function described by (enter equation) as described in
Least-Squares GAN [45] for stable learning and better quality of output
images. Secondly, the standard L1 loss function encourages less blurring of
output [18]. Thirdly, we implement an edge-map loss inspired by Xia et al.
[46] to reduce color bleeding and blurs around the image edges. The edge map
is computed using the Canny edge detector (using Otsu’s thresholding method).
We compute the edge maps of the ground truth image and the foreground
colorized image and the edge map loss is defined as the mean of the difference
between these two edge maps.

Consider $x$ as the input grayscale image, $z$ as the input noise for the
generator, $y$ as the ground truth color image, $G$ as the generator, and $D$
as the discriminator. For paired image-to-image translation, $x,y$ pair
denotes the paired domain of grayscale and color images. Then the loss for our
conditional GAN is:

$\displaystyle\begin{aligned} L_{ADV}(G,D)&=E_{x,y}[\log D(x,y)]\\\
&\qquad+E_{x,y}[\log(1-D(x,G(x,z)))]\end{aligned}$ (1)

The edge-map loss function involves the Canny edge detector which is denoted
by $C$. The basic idea is to compute the difference between the edge maps of
the ground truth image $C(y)$ and the colorized image $C(G(x))$. If the
colorized output is suffering from color bleeding artifacts, the edge map of
the colorized output will be different from the ground truth, thus the edge-
loss will be more. The value of the edge loss will be less only if the edge
maps come out to be similar to each other. So, the loss functions being used
are:

$L_{EDGE}(x,y,G)=E_{x,y,z}[\mid\mid C(y)-C(G(x,z))\mid\mid]$

$L_{L1}(G)=E_{x,y,z}[\mid\mid y-G(x,z)\mid\mid]$

Therefore, the final loss function is:

$\displaystyle\begin{aligned} G^{*}&=arg\>min_{G}\>max_{D}\>L_{ADV}(G,D)\\\
&\qquad+\lambda_{1}L_{EDGE}(G)+\lambda_{2}L_{L1}(G)\end{aligned}$ (2)

###  III-B Background Colorization

In this case, the complexity of the training pipeline is reduced because
background colorization does not generally involve colorizing sharp edges
since only the foreground stays in focus. The background colorized image is
obtained by combining the predicted channels with the luminance channel $L$ to
form $(L,a_{B},b_{B})$. Image datasets with more examples of common scenes,
places, and landscapes can serve as better examples of background. Public
datasets containing the same are appended to the main dataset. This time the
network is not provided with object proposals.

Colorization Backbone Architecture: The background colorizer network is simply
a slightly modified version of the Pix2Pix framework. The generator model is a
simple Residual UNet (with the same pre-trained ResNet-18 encoder) without
self-attention. This is chosen to reduce the model complexity as colorizing
backgrounds is a relatively easier task. The same PatchGAN discriminator is
being used here as proposed in Pix2PIx.

Loss Function and Training: A combination of MSE loss function and L1 loss
function is used to optimize the background colorizer. This model should act
as a compensator to the foreground colorizer model so the edge map loss is not
significant in this case.

The overall loss function for the background model is, therefore:

$G^{*}=arg\>min_{G}\>max_{D}\>L_{ADV}(G,D)+\lambda L_{L1}(G)$

###  III-C Fusion Network

The fusion network was inspired by the DenseFuse [29] framework that was used
for the fusion of visible and infrared images. Here, we redesigned and
repurposed the network to merge the outputs of the foreground and background
colorizer models. It is to be noted that the foreground and background
colorizer models are self-sufficient models at full colorization of an image
and they are named such that their specialization area is identified. This
network is required at the testing pipeline for the adaptive fusion of the
colorized foreground-background pair, so this can be trained independently.

Training and Network Architecture: The fusion network is a modified version of
the DenseFuse [29] framework that was actually made for infrared thermal
images. The network is modified to work with color images. It is an encoder-
decoder framework with a three-layer dense convolutional block at the encoder.
The network takes single color image as its input and is trained using a
weighted combination of mean-squared error (MSE) loss and structural
similarity index (SSIM) loss function because the aim of the network is to
generate an identical copy of the input image. The training was performed on
the COCO dataset [47] for nine epochs.

Fusion Process: In the testing pipeline, the pre-trained encoder of the fusion
network receives a pair of three-channel colorized images and the features are
computed. The features are added using feature fusion strategies that are
employed in DenseFuse. The pre-trained decoder receives the combined features
as its input and produces a three-channel color output as the final image.

##  IV Experiments

To obtain the ideal behavior for image colorization using a parallel method,
we conducted extensive experiments using different versions of generator
architecture. The detailed experimental settings corresponding to each version
have been discussed in the Ablation Study section. All the different versions
of the parallel GAN framework had the same skeleton but were trained with
different network settings to efficiently capture the contextual information
of the image. We used the COCO and Places dataset to train the colorizer
pipelines independently. We show significant counterexamples where the metric
of PSNR fails to match human visual assessment, so we present a quantitative
comparison report against state-of-the-art colorization models using multiple
perceptual and non-perceptual evaluation techniques.

###  IV-A Training Details

Our whole training procedure is divided into three stages. The first two
stages correspond to the parallel adversarial training of the colorizer
frameworks. The third stage is for training the fusion framework. The adopted
training methodologies for training the colorizer models include the pre-
training of generators inspired by the unbalanced GANs [28] framework. For the
final model, we used generator and discriminator learning rates of $2e-4$ with
the Adam optimizer ($\beta_{1}=0.5,\beta_{2}=0.999$). Using the pre-trained
generator, the adversarial training had to be performed for 15 epochs to
obtain stable outputs. The fusion network was trained for 20 epochs with a
learning rate of $1e-4$ using the Adam optimizer
($\beta_{1}=0.9,\beta_{2}=0.999$). All the training was performed on two 16GB
NVIDIA Tesla V100 GPU nodes.

Pre-training the Generators: The unbalanced GANs [28] framework showed faster
convergence of the generator and the discriminator with a better image quality
at much earlier epochs compared to ordinary GANs. The main objective of this
step was to prevent the discriminator from winning too easily during the
adversarial training and to maintain a stabilized learning process. We found
that pre-training the generator even for just a few epochs can be a good start
for the adversarial training process. The generators of the final version of
the parallel GAN framework were pre-trained using the COCO Training Split
dataset with a learning rate of $1e-4$ for 15 epochs using the Adam optimizer
($\beta_{1}=0.9,\beta_{2}=0.999$).

###  IV-B Evaluation Metrics

Since plausible colorization results probably diverge a lot from the original
color image, pixel-wise evaluation metrics like PSNR fail to model the human
visual assessment. An image elaborating the issue is shown in Figure 4.

Figure 4: The ground truth (left) image is recolorized using IAIC [26]
(middle) and our method (right). The PSNR of the middle image is 23.93 and the
PSNR obtained using our method (left) is 16.80

The first image being the ground truth, the second image is having a much
higher PSNR compared to the third even though one can observe that the third
image is better colorized with pronounced color details in both object and
background. But, since PSNR does not account for multimodality, the color
accuracy becomes the only determining factor.

Figure 5: Examples where the PSNR metric fails to identify a better colorized
output. The middle column has a better PSNR compared to the right column.

The dependence of PSNR on the ground truth makes the metric a good choice for
producing accurate colors but for tasks like image restoration, even if the
ground truth itself is corrupted, PSNR compels the model to produce a
corrupted image that is similar to the ground truth that generates a higher
PSNR as shown in the figure 5. That’s why perceptual metrics like FID, LPIPS,
etc. are better suited for the quantitative comparison of image quality and
fidelity. FID, instead of comparing pixel by pixel, compares the mean and
standard deviation of one of the deeper layers in Inception model. We computed
the FID using the official implementation of FID by PIQ [48] using 5000 images
of COCO validation split dataset resized to resolution 256x256. Inspired by Wu
et al. [49], we use Colorfulness score [50] to evaluate the vividness of the
colorized outputs.

###  IV-C Comparison Report

We used the COCO validation split dataset with 5000 images resized to 256*256
for the quantitative comparison. While the state-of-the-art models mainly used
large-scale datasets like ImageNet with 1.3M images for training, we used a
much more compact and manageable COCO Training Split dataset and scene samples
from the Places dataset [51]. Even with a substantially lesser amount of
training data, our model was able to compete and achieve quantitative scores
close to state-of-the-art methods. We evaluated our results using perceptual
metrics like FID and LPIPS which are well-suited for evaluating the
performance of generator outputs. FID specifically compares the similarity of
the generated data distribution with that of the real data distribution making
it a significant evaluation metric for judging the realism of the generated
data. We also evaluate the PSNR and SSIM values for reference but it has been
previously established that these metrics are not reliable for evaluating
colorization results [49]. The results are discussed in Table I.

Method |  $\Delta$Colorful$\downarrow$ | FID$\downarrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$
---|---|---|---|---|---
CIC [16] | 23.40 | 33.73 | 0.234 | 21.83 | 0.895
LRAC. [15] | 31.57 | 31.66 | 0.183 | 25.06 | 0.930
DeOldify [25] | 8.12 | 23.09 | 0.180 | 23.69 | 0.920
ChromaGAN [40] | 5.84 | 30.51 | 0.159 | 23.60 | 0.911
IAIC [26] | 9.60 | 17.66 | 0.125 | 27.77 | 0.940
Ours | 4.90 | 17.60 | 0.151 | 23.96 | 0.925
TABLE I: Quantitative comparison of colorization methods. Our method was
trained on COCO (Training Split) and Places only.

In order to provide additional comparison results with ColTran [52], a recent
learning-based method who have used FID for evaluating their results, we
randomly selected 5k images from the ImageNet Testing Split dataset as
performed by ColTran, using which our model was able to achieve better FID
scores compared to state-of-the-art models. (see Table II).

Method | ColTran [52] | IAIC [26] | Ours
---|---|---|---
FID$\downarrow$ | 22.0613 | 18.6942 | 16.8273
TABLE II: Comparison with state-of-the-art methods on ImageNet Testing Split
dataset.

###  IV-D Visual Comparison

Figure 6: This grid compiles the visual comparison results of existing methods
and ours. Our method outperforms the existing methods by accurately colorizing
the foreground and background while maintaining the edges and the long-range
contextual dependencies. RTUG [17] DeOldify [25] IAIC [26]

The visual comparison report is presented as a grid of images with a varied
range of foreground-background combinations to illustrate the strength of our
pipeline. The results are provided in Figure 6. The first column consists of a
dominant object in the foreground with a moderately cluttered background. To
make the colorized images appear realistic, context preservation with accurate
colorization of the subject in the frame is of utmost importance. The second
column of images has a cluttered but prominent background (the stadium crowd)
and a foreground with multiple objects (the baseball players). Our method is
able to colorize the players, the stadium crowd while colorizing the field.
Other popular methods suffered context confusion (DeOldify) and desaturated
backgrounds (IAIC). The third column requires the model to account for long-
range spatial dependencies (achieved by using the self-attention mechanism) so
that the left and the right side of the wooden door have the same color. Most
methods fail to maintain the long-range color consistency, thus colorizing the
left and the right side differently (IAIC). The fourth column has a prominent
object with saturated colors. Our method is able to colorize the foreground
while maintaining the edge details, whereas other methods showed a yellow bias
(CIC, ChromaGAN). The fifth column represents natural images where most of the
listed approaches performed flawlessly, the reason being clear figure-ground
separation, whereas in the final column, even under a low illumination level,
our model could colorize the complete image. An improved view of selected
images is highlighted in Figure 7.

Figure 7: This grid compiles the visual comparison results of the SOTA and
ours. The first row shows the object has been completely colorized without any
bleeding artifacts. The second row displays that our model could capture the
long range spatial dependencies.

####  IV-D1 Human Evaluation

To support our claims of realism, we performed human evaluation based the
whether our colorized outputs looked real or not. Traditional human evaluation
tests ask the subjects to rate the colorization quality of a given method.
Whereas, we believe a blind photorealism evaluation of the generated images
would provide a better estimate of whether a method can generate images that
look natural. We prepared 4 batches of recolorized images, each with 5 pairs
of recolorized images and their corresponding ground truths. The pairs were
displayed to the test subjects and they had to identify the recolorized image
generated by our colorization framework and provide a confidence score at the
end of each batch. To make the subjects accustomed to the test, a trial run
was prepared where the answers were shown to the subject. We chose 20 subjects
from diverse educational backgrounds who were shown 20 images each, totalling
400 decisions. All the recolorized images and their corresponding ground truth
images have been shown in Figure 8.

Figure 8: The top section corresponds to the ground truth images and the
bottom section is for the recolorized images.

The results were obtained by performing the human evaluation test. The
subjects had four options corresponding to each image that enabled them to
classify the image as fake or real and to say whether they were confident
about their prediction. The scoring strategy for the evaluation of our model
was prepared on the basis of these four options that generated four possible
cases after the subject has classified the image as real or fake: Case 1, the
subject is confident about their prediction and they are right. Our score is
0. Case 2, the subject is not confident about their prediction but still, they
are right. Case 3, the subject is not confident and is wrong. Here we can
choose our scores to be 0.25 and 0.75 respectively. Case 4, the subject is
confident but still is wrong. We get a score of 1.

For the first case, our model failed to fool the human subject, so it gets the
minimum score and for the last case, our model was able to fool the subject
completely i.e. it might have generated a better quality of colorized output
compared to the real image, so it gets the maximum score. For the second case,
the subjects were not sure but still were able to guess correctly, here, the
model couldn’t fool the subjects but at least produced results that could
confuse the subjects, so it get 0.25. The third is when the model fools the
subject but there is confusion, therefore we set the score as 0.75 since the
model could deceive the subjects anyway. The fooling score $F$ was simply
calculated from the table by the given equation where $V(I_{i},C_{i})$ is the
entry corresponding to the $i$th row and $j$th column and $K_{i}$ is the score
corresponding to the $i$th case; $N$ is the number of subjects participating
in the test; $n$ is the maximum number of cases (=4 in this case); $b$ is the
number of batches (=4 in this case)

$F=\frac{\sum_{i=1}^{n}\sum_{j=1}^{b}V(I_{i},C_{j})*K_{i}}{b\times N}$

The human evaluation score obtained by considering the mentioned scoring
strategy is 0.5375.

Batch | Case 1 | Case 2 | Case 3 | Case 4
---|---|---|---|---
Batch 1 | 6 | 37 | 52 | 5
Batch 2 | 12 | 40 | 40 | 8
Batch 3 | 6 | 29 | 55 | 10
Batch 4 | 7 | 34 | 49 | 10
TABLE III: Table corresponding to the human evaluation test performed to
evaluate the photorealism of the generated colorized images.

##  V Ablation Study

This section has been divided into three subsections where we perform an
extensive ablation study of the individual models in the foreground and
background colorization pipeline, the edge-loss function, and the fusion
model. The foreground colorized output has a desaturated background but the
objects (the donut, the woman, and her clothes) in the foreground were
properly colorized, while the background colorized output compensates for the
desaturated portions of the former by properly colorizing the complementary
regions. The merged output finally combines the best of both the images to
produce foreground-background colorized output.

###  V-A Ablation study of the Individual Models

Different versions (shown in Figure 9) of the pipeline are tested with
different experimental settings to obtain generators specific to foreground
and background colorization tasks. For the foreground colorizer pipeline, we
find that adding self-attention to the generator Res-UNet improves the
performance by throwing consistent colors across the spatial range of the
image to account for long-range dependencies. Including the edge-map loss is
also found to reduce the color-bleeding problem in the final colorized image
thus increasing the FID score of the model. For the background colorizer,
isolated performance decreases compared to the foreground colorizer.

####  V-A1 UNet Generator (V1)

We implemented a basic Pix2Pix based end-to-end pipeline for object and scene
colorization independently where we used the COCO dataset and its cropped
object proposals to train the UNet based generator for the foreground
colorizer pipeline. The PatchGAN based discriminator was employed for
adversarial training which could distinguish between the ground truth and the
generated colorized output. The background colorizer pipeline was trained on
full image features derived from the COCO dataset. Both pipelines used L1 loss
for updating the generator. The model was trained for 15 epochs with generator
and discriminator learning rates of 2e-4 using the Adam optimizer (
$\beta_{1}=0.5,\beta_{2}=0.999$).

####  V-A2 Res-UNet Generator (V2)

We used a ResNet-18 network as the encoder of the UNet based generator to
capture the object information more effectively. To make the generators more
robust, we used unbalanced training [28] in this version so that the
discriminator does not win too easily over the generator. We are using the
ResNet18 network as the encoder of the Residual-UNet architecture of both the
pipelines and we train the generator only to condition it for a particular
task. Other experimental settings were kept the same. Since the previous
version was unable to colorize the scene-based examples, we appended Places
dataset samples in the background colorization pipeline and it produced
acceptable results.

####  V-A3 Self-Attention Res-UNet Generator (V3)

Deployment of unbalanced training with Res-UNet architecture drastically
improved the convergence rate of the model and the addition of the samples of
the Places dataset resulted in a better performance in complicated test cases.
We further modified our model to incorporate the long-term dependency by
introducing self-attention to the network due to which color artifacts were
reduced and color consistency was maintained in the colorized outputs.

Figure 9: Results of ablation study with different versions of colorizer
frameworks.

We identified the problem of color bleeding in the previous versions so an
edge loss function was incorporated with the L1 and MSE loss functions in the
generator of the foreground colorizer pipeline. The final modified version
addressed all the problems discussed in the above experimental settings and
efficiently dealt with the diverse data modes and generated the foreground and
background colorized proposals for the fusion network.

###  V-B Ablation Study of Edge Loss

The edge loss ablation study has been performed to study the effect of edge
loss on color bleeding. The factor $\lambda_{1}$ decides the weight of edge
loss in the overall loss function given by the equation 2.

Since color bleeding has adverse effects on the structural similarity between
the ground truth and the colorized image, we chose the perceptual metric of
Structural Similarity Index (SSIM) to evaluate the performance of our method
with varying values of $\lambda_{1}$. Since the edge loss function was applied
only to the foreground colorizer, the SSIM results have been obtained on the
individual colorized outputs of the foreground colorizer. The training
pipeline provided in Figure 3 was followed for each version of the model with
different values of $\lambda_{1}$ and the results are discussed in Table IV.

$\lambda_{1}$ | 0.00 | 0.25 | 0.50 | 0.75 | 1.00
---|---|---|---|---|---
SSIM $(\uparrow)$ | 0.863 | 0.894 | 0.900 | 0.915 | 0.924
TABLE IV: Edge loss ablation study: Performed by varying $\lambda_{1}$ to
obtain the SSIM results on the outputs of the foreground colorizer.

###  V-C Ablation Study of Fusion Model

The colorization performance achieved by the individual colorizer models has
been computed to study the effect of the fusion model on the colorized image.
The colorization performance of the fusion model has also been compared with
an approach where we simply perform a weighted overlay of the foreground and
background proposals to obtain the final output. The weighting factor $\alpha$
determines the proportion of foreground colorized output in the final image.
We have used the COCO Validation split dataset to report the FID obtained by
each approach and the results are discussed in Table V.

$\alpha$ | 0.00 | 0.50 | 1.00 | Fusion
---|---|---|---|---
FID $(\downarrow)$ | 22.59 | 19.45 | 21.37 | 17.60
TABLE V: Fusion Model Ablation study: Comparison with weighted overlay of
foreground and background proposals.

##  VI Runtime Analysis

The weight parameter file size of our complete colorization framework is less
than 250 MB which is significantly lesser than the available state-of-the-art
learning-based models for grayscale image colorization. We perform extensive
runtime analysis of our approach by using NVIDIA GTX 1080 GPU with 8GB of
graphics memory.

Figure 10: Testing Pipeline

Since our method consists of two stages, namely, the parallel colorization
step and the feature-based adaptive fusion stage, we report the average
inference time computed using a thousand randomly selected images from the
COCO Validation Split dataset for each stage. It is to be noted that the
object detector is not required in the testing pipeline since the model
performs colorization based on full image features only. The parallel
colorization stage takes an average time of 0.015 seconds to execute and the
fusion pipeline reports an average of 0.009 seconds. Therefore the complete
testing pipeline represented in Figure 10 takes an average time of around
0.024 seconds to produce final colorized outputs of resolution 256x256.

##  VII Discussion

The problem of image colorization and addressing it using a learning-based
algorithm provides important insight toward designing color-predictive models
that understand the context of the scene. Earlier works on image colorization
have identified the problem of multi-modality but have failed to evaluate
their methods using metrics that account for multi-modality. We observed that
preserving the semantic information and maintaining the balance between
foreground and background color appearance using a single network is hard to
achieve even in the state-of-the-art. This became a major reason we came up
with an approach that leverages parallel training to preserve the foreground-
background information. We separately modified the networks and trained them
parallelly. We successfully merged their outputs to retain the best features
to produce a balanced image. The image shown in Figure 11 is a case where the
foreground colorizer has performed well and assigned proper colors to the
output, but the background colorizer has left a lot of blue artifacts that
have been retained even after the fusion. The probable solution to this could
be training the fusion model using the derived recolorized images from the
colorizer pipelines.

Figure 11: Left image is the object colorized output, middle image is the
background colorized output and the right image is the fused output.

##  VIII Conclusion

When the problem of image colorization is subdivided into two tasks of
foreground and background colorization, we observed that a significant
improvement in visual performance can be obtained with a substantially lesser
amount of training data. Employing multiple models helps address the diverse
data modes that alleviate the problem of the desaturated background while
colorizing the foreground without leaving color-bleeding artifacts. Parallel
GANs adversarially trained to produce the foreground and background colorizer
models act as compensators for each other so the fusion network can
efficiently combine the specialization areas of both the colorizer models thus
obtaining a balanced final image. Since our main focus was to produce
colorized images that look realistic, we performed a blind human evaluation of
our results based on the visual realism of the colorized images. The testing
pipeline instantly produces colorized images of resolution 256x256. After an
extensive performance evaluation against existing state-of-the-art methods, we
showed that our results achieved quantitative scores comparable to the state-
of-the-art with significantly less training data.

## Acknowledgements

This work has been performed using the GPU resources provided by the AI
Computation Facility of Intelligent Systems Group. The authors would like to
thank the Knowledge Resource Centre at Central Electronics Engineering
Research Institute (CSIR-CEERI), India for providing the required information
resources for their research.

## Compliance with ethical standards

Conflict of interest The authors declare that they have no conflict of
interest.

## References

  * [1] G. Charpiat, M. Hofmann, and B. Schölkopf, “Automatic image colorization via multimodal predictions,” in _European conference on computer vision_. Springer, 2008, pp. 126–139.
  * [2] A. Levin, D. Lischinski, and Y. Weiss, “Colorization using optimization,” in _ACM SIGGRAPH 2004 Papers_ , 2004, pp. 689–694.
  * [3] Y.-C. Huang, Y.-S. Tung, J.-C. Chen, S.-W. Wang, and J.-L. Wu, “An adaptive edge detection based colorization algorithm and its applications,” in _Proceedings of the 13th annual ACM international conference on Multimedia_ , 2005, pp. 351–354.
  * [4] L. Yatziv and G. Sapiro, “Fast image and video colorization using chrominance blending,” _Trans. Img. Proc._ , vol. 15, no. 5, p. 1120–1129, may 2006\. [Online]. Available: https://doi.org/10.1109/TIP.2005.864231
  * [5] Y. Qu, T.-T. Wong, and P.-A. Heng, “Manga colorization,” _ACM Trans. Graph._ , vol. 25, no. 3, p. 1214–1220, jul 2006. [Online]. Available: https://doi.org/10.1145/1141911.1142017
  * [6] Q. Luan, F. Wen, D. Cohen-Or, L. Liang, Y.-Q. Xu, and H.-Y. Shum, “Natural image colorization,” in _Proceedings of the 18th Eurographics Conference on Rendering Techniques_ , ser. EGSR’07. Goslar, DEU: Eurographics Association, 2007, p. 309–320.
  * [7] D. Sýkora, J. Dingliana, and S. Collins, “Lazybrush: Flexible painting tool for hand‐drawn cartoons,” _Computer Graphics Forum_ , vol. 28, 2009\.
  * [8] T. Welsh, M. Ashikhmin, and K. Mueller, “Transferring color to greyscale images,” _ACM Trans. Graph._ , vol. 21, no. 3, p. 277–280, jul 2002. [Online]. Available: https://doi.org/10.1145/566654.566576
  * [9] R. Ironi, D. Cohen-Or, and D. Lischinski, “Colorization by example.” _Rendering techniques_ , vol. 29, pp. 201–210, 2005.
  * [10] R. K. Gupta, A. Y.-S. Chia, D. Rajan, E. S. Ng, and H. Zhiyong, “Image colorization using similar images,” in _Proceedings of the 20th ACM international conference on Multimedia_ , 2012, pp. 369–378.
  * [11] X. Liu, L. Wan, Y. Qu, T.-T. Wong, S. Lin, C.-S. Leung, and P.-A. Heng, “Intrinsic colorization,” in _ACM SIGGRAPH Asia 2008 papers_ , 2008, pp. 1–9.
  * [12] L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2414–2423.
  * [13] M. Mirza and S. Osindero, “Conditional generative adversarial nets,” _arXiv preprint arXiv:1411.1784_ , 2014.
  * [14] S. Iizuka, E. Simo-Serra, and H. Ishikawa, “Let there be color! joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification,” _ACM Transactions on Graphics (ToG)_ , vol. 35, no. 4, pp. 1–11, 2016.
  * [15] G. Larsson, M. Maire, and G. Shakhnarovich, “Learning representations for automatic colorization,” in _European conference on computer vision_. Springer, 2016, pp. 577–593.
  * [16] R. Zhang, P. Isola, and A. A. Efros, “Colorful image colorization,” in _Computer Vision – ECCV 2016_ , B. Leibe, J. Matas, N. Sebe, and M. Welling, Eds. Cham: Springer International Publishing, 2016, pp. 649–666.
  * [17] R. Zhang, J.-Y. Zhu, P. Isola, X. Geng, A. S. Lin, T. Yu, and A. A. Efros, “Real-time user-guided image colorization with learned deep priors,” 2017. [Online]. Available: https://arxiv.org/abs/1705.02999
  * [18] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1125–1134.
  * [19] J. Zhao, L. Liu, C. G. M. Snoek, J. Han, and L. Shao, “Pixel-level semantics guided image colorization,” 2018. [Online]. Available: https://arxiv.org/abs/1808.01597
  * [20] M. He, D. Chen, J. Liao, P. V. Sander, and L. Yuan, “Deep exemplar-based colorization,” _ACM Transactions on Graphics (TOG)_ , vol. 37, no. 4, pp. 1–16, 2018.
  * [21] S. Guadarrama, R. Dahl, D. Bieber, M. Norouzi, J. Shlens, and K. Murphy, “Pixcolor: Pixel recursive colorization,” _arXiv preprint arXiv:1705.07208_ , 2017.
  * [22] A. K. Amelie Royer and C. Lampert, “Probabilistic image colorization,” in _Proceedings of the British Machine Vision Conference (BMVC)_ , G. B. Tae-Kyun Kim, Stefanos Zafeiriou and K. Mikolajczyk, Eds. BMVA Press, September 2017, pp. 85.1–85.12. [Online]. Available: https://dx.doi.org/10.5244/C.31.85
  * [23] A. Deshpande, J. Lu, M.-C. Yeh, M. Jin Chong, and D. Forsyth, “Learning diverse image colorization,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 6837–6845.
  * [24] S. Messaoud, D. Forsyth, and A. G. Schwing, “Structural consistency and controllability for diverse colorization,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , September 2018.
  * [25] J. Antic, “jantic/deoldify: A deep learning based project for colorizing and restoring old images (and video!),” 2019.
  * [26] J.-W. Su, H. kuo Chu, and J.-B. Huang, “Instance-aware image colorization,” _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 7965–7974, 2020.
  * [27] Q. M. Hoang, T. D. Nguyen, T. Le, and D. Q. Phung, “Mgan: Training generative adversarial nets with multiple generators,” in _ICLR_ , 2018.
  * [28] H.-G. Ham, T. J. Jun, and D. Kim, “Unbalanced gans: Pre-training the generator of generative adversarial network using variational autoencoder,” _ArXiv_ , vol. abs/2002.02112, 2020.
  * [29] H. Li and X.-J. Wu, “Densefuse: A fusion approach to infrared and visible images,” _IEEE Transactions on Image Processing_ , vol. 28, no. 5, pp. 2614–2623, 2019.
  * [30] X. An and F. Pellacini, “Appprop: All-pairs appearance-space edit propagation,” _ACM Trans. Graph._ , vol. 27, no. 3, 2008. [Online]. Available: https://doi.org/10.1145/1360612.1360639
  * [31] D. Freedman and P. Kisilev, “Object-to-object color transfer: Optimal flows and smsp transformations,” in _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , 2010, pp. 287–294.
  * [32] F. Pitie, A. Kokaram, and R. Dahyot, “N-dimensional probability density function transfer and its application to color transfer,” in _Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1_ , vol. 2, 2005, pp. 1434–1439 Vol. 2.
  * [33] P.-Y. Laffont, Z. Ren, X. Tao, C. Qian, and J. Hays, “Transient attributes for high-level understanding and editing of outdoor scenes,” _ACM Trans. Graph._ , vol. 33, no. 4, jul 2014. [Online]. Available: https://doi.org/10.1145/2601097.2601101
  * [34] Y. Shih, S. Paris, C. Barnes, W. T. Freeman, and F. Durand, “Style transfer for headshot portraits,” _ACM Trans. Graph._ , vol. 33, jul 2014. [Online]. Available: https://doi.org/10.1145/2601097.2601137
  * [35] F. Wu, W. Dong, Y. Kong, X. Mei, J.-C. Paul, and X. Zhang, “Content-based colour transfer,” in _Computer Graphics Forum_. Wiley Online Library, 2013, pp. 190–203.
  * [36] B. Zhang, M. He, J. Liao, P. V. Sander, L. Yuan, A. Bermak, and D. Chen, “Deep exemplar-based video colorization,” 2019. [Online]. Available: https://arxiv.org/abs/1906.09909
  * [37] Z. Cheng, Q. Yang, and B. Sheng, “Deep colorization,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 415–423.
  * [38] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , 2009, pp. 248–255.
  * [39] K. Nazeri, E. Ng, and M. Ebrahimi, “Image colorization using generative adversarial networks,” in _International conference on articulated motion and deformable objects_. Springer, 2018, pp. 85–94.
  * [40] P. Vitoria, L. Raad, and C. Ballester, “Chromagan: Adversarial picture colorization with semantic class distribution,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2020, pp. 2445–2454.
  * [41] L. Zhang, Y. Ji, X. Lin, and C. Liu, “Style transfer for anime sketches with enhanced residual u-net and auxiliary classifier gan,” in _2017 4th IAPR Asian conference on pattern recognition (ACPR)_. IEEE, 2017, pp. 506–511.
  * [42] N. O’Hare and A. F. Smeaton, “Context-aware person identification in personal photo collections,” _IEEE Transactions on Multimedia_ , vol. 11, no. 2, pp. 220–228, 2009.
  * [43] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2961–2969.
  * [44] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena, “Self-attention generative adversarial networks,” in _International conference on machine learning_. PMLR, 2019, pp. 7354–7363.
  * [45] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley, “Least squares generative adversarial networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2794–2802.
  * [46] J. Xia, G. Tan, Y. Xiao, F. Xu, and C.-S. Leung, “Edge-aware multi-scale progressive colorization,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 1655–1659.
  * [47] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [48] M. Seitzer, “pytorch-fid: Fid score for pytorch,” _https://github. com/mseitzer/pytorch-fid_ , 2020.
  * [49] Y. Wu, X. Wang, Y. Li, H. Zhang, X. Zhao, and Y. Shan, “Towards vivid and diverse image colorization with generative color prior,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 14 377–14 386.
  * [50] D. Hasler and S. E. Suesstrunk, “Measuring colorfulness in natural images,” in _Human vision and electronic imaging VIII_ , vol. 5007. International Society for Optics and Photonics, 2003, pp. 87–95.
  * [51] B. Zhou, A. Khosla, A. Lapedriza, A. Torralba, and A. Oliva, “Places: An image database for deep scene understanding,” _arXiv preprint arXiv:1610.02055_ , 2016.
  * [52] M. Kumar, D. Weissenborn, and N. Kalchbrenner, “Colorization transformer,” _arXiv preprint arXiv:2102.04432_ , 2021.


m}^{2}}\mathop{\mathrm{d}\mskip-0.59999muk^{2}_{\text{T}}}\nolimits{}\,\frac{W_{\text{INT},j}}{32\pi^{3}}\,.$
(181)

Then all the integrals over $k^{2}_{\text{T}}$ take one of the following forms

$\displaystyle{}\Gamma^{+}_{2}=\int_{0}^{k_{\rm
m}^{2}}\;\left.\frac{\mathop{\mathrm{d}\mskip-0.59999muk^{2}_{\text{T}}}\nolimits{}}{\sqrt{1-\frac{k^{2}_{\text{T}}}{k_{\rm
m}^{2}}}}(k^{2}-m_{q}^{2})^{2}\right|_{\theta<\pi/2}=2k_{\rm
m}^{2}\left(A^{2}-AB(C+2)+\frac{1}{3}B^{2}(C(C+3)+3)\right)$ (182)
$\displaystyle{}\Gamma^{+}_{1}=\left.\int_{0}^{k_{\rm
m}^{2}}\;\frac{\mathop{\mathrm{d}\mskip-0.59999muk^{2}_{\text{T}}}\nolimits{}}{\sqrt{1-\frac{k^{2}_{\text{T}}}{k_{\rm
m}^{2}}}}(k^{2}-m_{q}^{2})\right|_{\theta<\pi/2}=2k_{\rm
m}^{2}\left(A-\frac{1}{2}B(C+2)\right)$ (183)
$\displaystyle{}\Gamma^{+}_{0}=\left.\int_{0}^{k_{\rm
m}^{2}}\;\frac{\mathop{\mathrm{d}\mskip-0.59999muk^{2}_{\text{T}}}\nolimits{}}{\sqrt{1-\frac{k^{2}_{\text{T}}}{k_{\rm
m}^{2}}}}\right|_{\theta<\pi/2}=2k_{\rm m}^{2}$ (184)
$\displaystyle{}\Gamma^{+}_{-1}=\left.\int_{0}^{k_{\rm
m}^{2}}\;\frac{\mathop{\mathrm{d}\mskip-0.59999muk^{2}_{\text{T}}}\nolimits{}}{\sqrt{1-\frac{k^{2}_{\text{T}}}{k_{\rm
m}^{2}}}}\frac{1}{(k^{2}-m_{q}^{2})}\right|_{\theta<\pi/2}=\frac{2k_{\rm
m}^{2}}{BC}\ln\left(\frac{B-A}{B(C+1)-A}\right)$ (185)
$\displaystyle{}\Gamma^{+}_{-2}=\left.\int_{0}^{k_{\rm
m}^{2}}\;\frac{\mathop{\mathrm{d}\mskip-0.59999muk^{2}_{\text{T}}}\nolimits{}}{\sqrt{1-\frac{k^{2}_{\text{T}}}{k_{\rm
m}^{2}}}}\frac{1}{(k^{2}-m_{q}^{2})^{2}}\right|_{\theta<\pi/2}=2k_{\rm
m}^{2}\frac{1}{(B-A)(-A+BC+B)}\,.$ (186)

The $+$ superscript here means these are the integrals for $\theta<\pi/2$ and
so the $C$ comes with a $+$ sign. Identical expressions apply to
$\Gamma^{-}_{n}$ but with $C\to-C$. The subscripts refer to the power on
$(k^{2}-m_{q}^{2})$.

Substituting Eq. (178) into Eq. (181) and using Eqs. (182)–(186) gives an
analytic expression

$W_{j}=\frac{1}{256Ek_{\rm
m}\pi^{2}}\left[I_{j,2}(\Gamma_{2}^{+}+\Gamma_{2}^{-})+I_{j,1}(\Gamma_{1}^{+}+\Gamma_{1}^{-})+2\Gamma_{0}^{+}I_{j,0}+I_{j,-1}(\Gamma_{-1}^{+}+\Gamma_{-1}^{-})+I_{j,-2}(\Gamma_{-2}^{+}+\Gamma_{-2}^{-})\right]\,,$
(187)

where

$\displaystyle\Gamma_{2}^{+}+\Gamma_{2}^{-}$ $\displaystyle{}=4k_{\rm
m}^{2}\left((A-B)^{2}+\frac{B^{2}C^{2}}{3}\right)$ (188)
$\displaystyle\Gamma_{1}^{+}+\Gamma_{1}^{-}$ $\displaystyle{}=4k_{\rm
m}^{2}\left(A-B\right)$ (189) $\displaystyle\Gamma_{0}^{+}+\Gamma_{0}^{-}$
$\displaystyle{}=4k_{\rm m}^{2}$ (190)
$\displaystyle\Gamma_{-1}^{+}+\Gamma_{-1}^{-}$ $\displaystyle{}=-\frac{2k_{\rm
m}^{2}}{BC}\ln\left(\frac{B(1+C)-A}{B(1-C)-A}\right)$ (191)
$\displaystyle\Gamma_{-2}^{+}+\Gamma_{-2}^{-}$ $\displaystyle{}=4k_{\rm
m}^{2}\left(\frac{1}{(A-B)^{2}-B^{2}C^{2}}\right)\,,$ (192)

All that is needed to get explicit expressions for structure functions is to
get the $I_{j,n}$ for ${\rm P}_{g}^{\mu\nu}$ and ${\rm P}_{PP}^{\mu\nu}$ and
specify numerical values for the masses $Q^{2}$ and $x_{\rm bj}$.

###  A.4 Simple case

The explicit expressions for the $I$’s, $A$, $B$, and $C$ are cumbersome for
general combinations of masses, so we will write them here only for a special
case. The expressions for $A$, $B$, $C$, $k_{\rm m}$, $E$ and the $I$’s are
especially simple for $m_{p}=m$, $m_{s}=2m$, and $m_{q}=m$. The results are

$\displaystyle k_{\rm m}$
$\displaystyle{}=\frac{1}{2}\sqrt{-\frac{Q^{2}(x_{\rm bj}-1)\left(8m^{2}x_{\rm
bj}+Q^{2}(x_{\rm bj}-1)\right)}{x_{\rm bj}\left(Q^{2}(x_{\rm
bj}-1)-m^{2}x_{\rm bj}\right)}}$ (193) $\displaystyle E$
$\displaystyle{}=\frac{1}{2}\sqrt{m^{2}-\frac{Q^{2}(x_{\rm bj}-1)}{x_{\rm
bj}}}$ (194) $\displaystyle A$ $\displaystyle{}=4m^{2}$ (195) $\displaystyle
B$ $\displaystyle{}=\frac{\left(Q^{2}(x_{\rm bj}-1)-4m^{2}x_{\rm
bj}\right)\left(2m^{2}x_{\rm bj}+Q^{2}\right)}{2x_{\rm bj}\left(Q^{2}(x_{\rm
bj}-1)-m^{2}x_{\rm bj}\right)}$ (196) $\displaystyle C$
$\displaystyle{}=-\frac{Q^{2}\sqrt{(x_{\rm bj}-1)\left(8m^{2}x_{\rm
bj}+Q^{2}(x_{\rm bj}-1)\right)\left(4m^{2}x_{\rm
bj}^{2}+Q^{2}\right)}}{\left(Q^{2}(x_{\rm bj}-1)-4m^{2}x_{\rm
bj}\right)\left(2m^{2}x_{\rm bj}+Q^{2}\right)}$ (197) $\displaystyle I_{g,-2}$
$\displaystyle{}=0$ (198) $\displaystyle I_{g,-1}$
$\displaystyle{}=-\frac{4\left(Q^{2}(x_{\rm bj}-1)^{2}-8m^{2}x_{\rm
bj}^{2}\right)}{(x_{\rm bj}-1)x_{\rm bj}}$ (199) $\displaystyle I_{g,0}$
$\displaystyle{}=8$ (200) $\displaystyle I_{g,1}$
$\displaystyle{}=\frac{4x_{\rm bj}}{Q^{2}(1-x_{\rm bj})}$ (201) $\displaystyle
I_{g,2}$ $\displaystyle{}=0\,$ (202) $\displaystyle I_{pp,-2}$
$\displaystyle{}=0$ (203) $\displaystyle I_{pp,-1}$
$\displaystyle{}=\frac{32m^{4}x_{\rm bj}}{x_{\rm
bj}-1}-\frac{2m^{2}Q^{2}(x_{\rm bj}+3)}{x_{\rm bj}}$ (204) $\displaystyle
I_{pp,0}$ $\displaystyle{}=\frac{4m^{2}(1-3x_{\rm bj})}{x_{\rm
bj}-1}+\frac{2Q^{2}}{x_{\rm bj}}$ (205) $\displaystyle I_{pp,1}$
$\displaystyle{}=2-\frac{2m^{2}x_{\rm bj}}{Q^{2}(x_{\rm bj}-1)}$ (206)
$\displaystyle I_{pp,2}$ $\displaystyle{}=0\,\,.$ (207)

Therefore, in this case we may write Eq. (187) as

$W_{j}=\frac{k_{\rm
m}}{64E\pi^{2}}\left[I_{j,2}\left(A^{2}-2AB+\frac{1}{3}B^{2}(1+C^{2})\right)+I_{j,1}\left(A-B\right)+I_{j,0}-I_{j,-1}\frac{1}{2BC}\ln\left(\frac{B(1+C)-A}{B(1-C)-A}\right)\right]\,.$
(208)

and substitute Eqs. (193)–(207) to get numerical values for $W_{g}$ and
$W_{PP}$. The standard structure functions $F_{1}$ and $F_{2}$ are

$\displaystyle F_{1}{}$
$\displaystyle\equiv-\frac{1}{2}W_{g}+\frac{2Q^{2}x_{\rm
N}^{2}}{(m_{p}^{2}x_{\rm N}^{2}+Q^{2})^{2}}W_{PP}\,,$ (209) $\displaystyle
F_{2}{}$ $\displaystyle\equiv\frac{12Q^{4}x_{\rm
N}^{3}\left(Q^{2}-m_{p}^{2}x_{\rm N}^{2}\right)}{\left(Q^{2}+m_{p}^{2}x_{\rm
N}^{2}\right)^{4}}\left(W_{PP}-\frac{\left(m_{p}^{2}x_{\rm
N}^{2}+Q^{2}\right)^{2}}{12Q^{2}x_{\rm N}^{2}}W_{g}\right)\,.$ (210)

Here, $x_{\rm N}$ is Nachtmann $x$:

$x_{\rm N}=\frac{2x_{\rm bj}}{1+\sqrt{1+\frac{4x_{\rm
bj}^{2}m_{p}^{2}}{Q^{2}}}}\,.$ (211)

##  Appendix B The small-$b_{\text{T}}$ operator product expansion

In this appendix, we extract the small-$b_{\text{T}}$ limit of the TMD pdf (or
ff) in collinear factorization. We start with the quark pdf in Eq. (76) and
consider the limit where $b_{\text{T}}\approx 1/\mu$ with $\mu/m\to\infty$, In
the region of the integrand where $k_{\text{T}}\approx m$, we can expand it
as,

$e^{-i\boldsymbol{k}_{\text{T}}\cdot\boldsymbol{b}_{\text{T}}}f_{q/p}(x,\boldsymbol{k}_{\text{T}};\mu)=f_{i/p}(x,\boldsymbol{k}_{\text{T}};\mu)+\order{k^{2}_{\text{T}}b^{2}_{\text{T}}}=f_{i/p}(x,\boldsymbol{k}_{\text{T}};\mu)+\order{\frac{k^{2}_{\text{T}}}{\mu^{2}}}\,.$
(212)

Thus, we define the small-$k_{\text{T}}$ part of the approximation to Eq. (76)
as

$\tilde{f}_{q/p}(x,\boldsymbol{b}_{\text{T}}\approx
1/\mu;\mu)\approx\int_{\text{UVR}}\mathop{\mathrm{d}\mskip-0.59999mu^{2}\boldsymbol{k}_{\text{T}}}\nolimits{}f_{q/p}(x,\boldsymbol{k}_{\text{T}};\mu)\equiv
f_{q/p}(x;\mu)\,.$ (213)

Dropping the $e^{-i\boldsymbol{k}_{\text{T}}\cdot\boldsymbol{b}_{\text{T}}}$
factor causes the integral over $k_{\text{T}}$ to become UV divergent, so we
must introduce a UV regulator and/or UV renormalization scheme to have a well-
defined $f(x;\mu)$. We notate this with the “UVR,” for “UV regulator,” on the
integral sign. The specific choice of UV regulator is arbitrary. Equation
(213) is a poor approximation because of the UV divergence and the
$\order{k^{2}_{\text{T}}/\mu^{2}}$ correction in Eq. (212), which is
unsuppressed when $k_{\text{T}}\approx\mu$. However, it can be corrected by
using collinear perturbation theory for large enough $\mu$. To implement this,
we rewrite Eq. (76) in the small $b_{\text{T}}$ region as

$\tilde{f}_{q/p}(x,\boldsymbol{b}_{\text{T}}\approx
1/\mu;\mu)=f_{q/p}(x;\mu)+\int\mathop{\mathrm{d}\mskip-0.59999mu^{2}\boldsymbol{k}_{\text{T}}}\nolimits{}\left\\{e^{-i\boldsymbol{k}_{\text{T}}\cdot\boldsymbol{b}_{\text{T}}}f_{q/p}(x,\boldsymbol{k}_{\text{T}};\mu)-\delta^{(2)}(\boldsymbol{k}_{\text{T}})f_{q/p}(x;\mu)\right\\}\,.$
(214)

The integrand of the second term in Eq. (214) is the contribution from the
$\order{k^{2}_{\text{T}}/\mu^{2}}\sim\order{k^{2}_{\text{T}}b^{2}_{\text{T}}}$
error in Eq. (212) if we transform into $b_{\text{T}}$-space and use the
definition in Eq. (213). The full integral in Eq. (214) is both IR and
collinear finite, so there is no need for a UVR in the integral. By
construction, its only unsuppressed contribution comes form the region where
$k_{\text{T}}$ is comparable to $\mu$. However, the $k_{\text{T}}\approx\mu$
behavior of $f_{q/p}(x,\boldsymbol{k}_{\text{T}};\mu)$ is expressible in
collinear factorization,

$k_{\text{T}}f_{q/p}(x,\boldsymbol{k}_{\text{T}}\approx\mu;\mu)=\sum_{i}k_{\text{T}}\frac{1}{k^{2}_{\text{T}}}\left[\mathcal{C}_{q/i}\otimes
f_{i/p}\right](x,k_{\text{T}}/\mu)+\order{\frac{m^{2}}{k^{2}_{\text{T}}}}\,.$
(215)

See Eq. (91) for the Yukawa theory version of this. The hard coefficient
$\mathcal{C}_{q/i}$ can be expanded in collinear perturbation theory. It is
dimensionless, but it depends on logarithms of $k_{\text{T}}/\mu$. The
$\order{m^{2}/k^{2}_{\text{T}}}$ in Eq. (215) combines with the
$\order{k^{2}_{\text{T}}b^{2}_{\text{T}}}$ error from Eq. (212) to give an
overall error that is suppressed by $\order{m^{2}b^{2}_{\text{T}}}$. Up to
$\order{m^{2}b^{2}_{\text{T}}}$ terms, therefore, the right side of Eq. (214)
only involves collinear pdfs. This is the small-$b_{\text{T}}$ OPE for the
unpolarized case. Notice that the unsuppressed term in Eq. (214) is not
completely unique because it depends upon the scheme used in Eq. (213) for
treating the UV region.

The full statement of the transverse coordinate space OPE for the quark TMD
pdf is

$\displaystyle\tilde{f}_{q/p}(x,\boldsymbol{b}_{\text{T}};\mu)=\sum_{j}\int_{x}^{1}\frac{\mathop{\mathrm{d}\mskip-0.59999mu\xi}\nolimits{}}{\xi}\mathcal{\tilde{C}}_{q/j}(x/\xi,\boldsymbol{b}_{\text{T}};\mu)\tilde{f}_{j/p}(\xi;\mu)+\order{m^{2}b^{2}_{\text{T}}}{}\,,$
(216)

Reading off the first term from Eq. (214) gives the zeroth order hard
coefficient

$\mathcal{\tilde{C}}_{q/j}^{(0)}=\delta(1-x/\xi)\delta_{qj}\,.$ (217)

The higher order $\mathcal{\tilde{C}}_{q/j}^{(n)}$ with $n\geq 1$ are
calculable from the second term in Eq. (214). The steps are to: i.) calculate
the integral, ii.) extract the small-$b_{\text{T}}$ limit, iii.) drop any
$b_{\text{T}}$-suppressed corrections, and iv.) identify the
$\mathcal{\tilde{C}}_{q/j}^{(n)}$ coefficients. The coefficients are
independent of the target, so one generally uses a massless parton target to
calculate. Thus, one calculates

$\int\mathop{\mathrm{d}\mskip-0.59999mu^{2}\boldsymbol{k}_{\text{T}}}\nolimits{}\left\\{e^{-i\boldsymbol{k}_{\text{T}}\cdot\boldsymbol{b}_{\text{T}}}f^{\text{partonic}}_{q/i}(x,\boldsymbol{k}_{\text{T}};\mu)-\delta^{(2)}(\boldsymbol{k}_{\text{T}})f^{\text{partonic}}_{q/i}(x;\mu)\right\\}\,,$
(218)

where $i$ is a massless parton. With a massless parton target,
$f^{\text{partonic}}_{q/i}(x,\boldsymbol{k}_{\text{T}};\mu)$ can, in general,
involve terms proportional to $\delta^{(2)}(\boldsymbol{k}_{\text{T}})$.

In the Yukawa theory example, it is easiest to do the calculation of
$\mathcal{\tilde{C}}_{q/j}^{(1)}$ by dealing with each term in the integral in
Eq. (218) separately. We will also make the replacement $k^{2}_{\text{T}}\to
k^{2}_{\text{T}}+m^{2}$ in propagator denominators to regulate any collinear
divergences at intermediate steps. Then we may combine the two terms in Eq.
(218) and set $m\to 0$. The second term in Eq. (218) is just the negative of
the collinear pdf (see Eq. (29)), and with $\overline{\rm MS}$ renormalization
it is

$-\frac{a_{\lambda}(\mu)}{\pi}(2\pi\mu)^{2\epsilon}(1-\xi)\int\mathop{\mathrm{d}\mskip-0.59999mu^{2-2\epsilon}\boldsymbol{k}_{\text{T}}}\nolimits{}\frac{k^{2}_{\text{T}}}{\big{[}k^{2}_{\text{T}}+m^{2}\big{]}^{2}}-\overline{{\rm
MS}}\;\;\text{C.T.}\,\stackrel{{\scriptstyle\epsilon\to
0}}{{=}}-a_{\lambda}(\mu)(1-\xi)\left[-1+\ln\left(\frac{\mu^{2}}{m^{2}}\right)\right]\,.$
(219)

We may calculate the first term in Eq. (218) in four dimensions since it is
finite. It is

$\displaystyle\frac{a_{\lambda}(\mu)}{\pi}(1-\xi)\int\mathop{\mathrm{d}\mskip-0.59999mu^{2}\boldsymbol{k}_{\text{T}}}\nolimits{}e^{-i\boldsymbol{k}_{\text{T}}\cdot\boldsymbol{b}_{\text{T}}}\frac{k^{2}_{\text{T}}}{\big{[}k^{2}_{\text{T}}+m^{2}\big{]}^{2}}$
$\displaystyle{}=2a_{\lambda}(\mu)(1-\xi)\left[K_{0}\left(b_{\text{T}}m\right)-\frac{b_{\text{T}}m}{2}K_{1}\left(b_{\text{T}}m\right)\right]\,$
$\displaystyle{}=a_{\lambda}(\mu)(1-\xi)\left[-\ln\left(\frac{b^{2}_{\text{T}}m^{2}e^{2\gamma_{E}}}{4}\right)-1\right]+\order{m^{2}b^{2}_{\text{T}}}\,.$
(220)

Adding Eq. (220) and Eq. (219) and setting $m=0$ gives for Eq. (218)

$\displaystyle-
a_{\lambda}(\mu)(1-\xi)\ln\left(\frac{b^{2}_{\text{T}}\mu^{2}e^{2\gamma_{E}}}{4}\right)$
$\displaystyle{}=\sum_{j}\int_{\xi}^{1}\frac{\mathop{\mathrm{d}\mskip-0.59999muz}\nolimits{}}{z}\left[-a_{\lambda}(\mu)(1-z)\ln\left(\frac{b^{2}_{\text{T}}\mu^{2}e^{2\gamma_{E}}}{4}\right)\delta_{jp}\right]\delta(1-\xi/z)\delta_{pp}$
$\displaystyle{}=\sum_{j\in
p}\int_{\xi}^{1}\frac{\mathop{\mathrm{d}\mskip-0.59999muz}\nolimits{}}{z}\mathcal{\tilde{C}}^{(1)}_{q/j}(z,\boldsymbol{b}_{\text{T}};\mu)\tilde{f}^{(0)}_{j/p}(\xi/z;\mu)\,.$
(221)

So we read off

$\mathcal{\tilde{C}}^{(1)}_{q/j}(z,\boldsymbol{b}_{\text{T}};\mu)=-a_{\lambda}(\mu)(1-z)\ln\left(\frac{b^{2}_{\text{T}}\mu^{2}e^{2\gamma_{E}}}{4}\right)\delta_{jp}\,,$
(222)

which corresponds to Eq. (112).

The manipulations above work equally well if, instead of the
$k^{2}_{\text{T}}\to k^{2}_{\text{T}}+m^{2}$ replacement in Eq. (219), we used
dimensional regularization to handle the collinear divergences, as is more
standard in QCD. But the use of $m^{2}$ makes the separate rolls of UV and
collinear behavior very transparent. The full result for the OPE contains, in
addition to the term in Eq. (222), another term for $j=s$, but we do not write
it here since it does enter explicitly in the calculations in the main body of
the text.

## References

  * Feynman (1972) R. P. Feynman, _Photon-Hadron Interactions_ (Benjamin, Reading, MA, 1972).
  * Candido et al. (2020) A. Candido, S. Forte, and F. Hekhorn, JHEP 11, 129 (2020), eprint 2006.07377.
  * Collins et al. (2022) J. Collins, T. C. Rogers, and N. Sato, Phys. Rev. D 105, 076010 (2022), eprint 2111.01170.
  * Ball et al. (2022) R. D. Ball, A. Candido, J. Cruz-Martinez, S. Forte, T. Giani, F. Hekhorn, K. Kudashkin, G. Magni, and J. Rojo (NNPDF), Nature 608, 483 (2022), eprint 2208.08372.
  * Courtoy et al. (2022) A. Courtoy, J. Huston, P. Nadolsky, K. Xie, M. Yan, and C. P. Yuan (2022), eprint 2205.10444.
  * Guzzi et al. (2022) M. Guzzi, T. J. Hobbs, K. Xie, J. Huston, P. Nadolsky, and C. P. Yuan (2022), eprint 2211.01387.
  * D’Alesio et al. (2020) U. D’Alesio, C. Flore, and A. Prokudin, Phys. Lett. B 803, 135347 (2020), eprint 2001.01573.
  * Aamodt et al. (2010) K. Aamodt et al. (ALICE), Eur. Phys. J. C68, 345 (2010), eprint 1004.3514.
  * Gamberg et al. (2022) L. Gamberg, M. Malda, J. A. Miller, D. Pitonyak, A. Prokudin, and N. Sato (Jefferson Lab Angular Momentum (JAM), Jefferson Lab Angular Momentum), Phys. Rev. D 106, 034014 (2022), eprint 2205.00999.
  * Collins and Soper (1987) J. C. Collins and D. E. Soper, Ann. Rev. Nucl. Part. Sci. 37, 383 (1987).
  * Collins et al. (1989) J. C. Collins, D. E. Soper, and G. F. Sterman, Adv. Ser. Direct. High Energy Phys. 5, 1 (1989), eprint hep-ph/0409313.
  * Collins (2011) J. C. Collins, _Foundations of Perturbative QCD_ (Cambridge University Press, Cambridge, 2011).
  * Liu (2019) T. Liu, PoS CD2018, 036 (2019).
  * Abdul Khalek et al. (2022) R. Abdul Khalek et al., Nucl. Phys. A 1026, 122447 (2022), eprint 2103.05419.
  * Efremov and Teryaev (1982) A. V. Efremov and O. V. Teryaev, Sov. J. Nucl. Phys. 36, 140 (1982).
  * Qiu and Sterman (1991) J.-W. Qiu and G. F. Sterman, Phys. Rev. Lett. 67, 2264 (1991).
  * Jaffe and Ji (1992) R. L. Jaffe and X.-D. Ji, Nucl. Phys. B 375, 527 (1992).
  * Ji (1993) X. Ji, Nucl. Phys. B402, 217 (1993).
  * Tangerman and Mulders (1994) R. D. Tangerman and P. J. Mulders (1994), eprint hep-ph/9408305.
  * Mulders and Tangerman (1996) P. J. Mulders and R. D. Tangerman, Nucl. Phys. B 461, 197 (1996), [Erratum: Nucl.Phys.B 484, 538–540 (1997)], eprint hep-ph/9510301.
  * Bacchetta et al. (2007) A. Bacchetta et al., JHEP 02, 093 (2007), eprint hep-ph/0611265.
  * Qiu et al. (2020) J.-W. Qiu, T. C. Rogers, and B. Wang, Phys. Rev. D 101, 116017 (2020), eprint 2004.13193.
  * Boglione et al. (2019) M. Boglione, A. Dotson, L. Gamberg, S. Gordon, J. O. Gonzalez-Hernandez, A. Prokudin, T. C. Rogers, and N. Sato, JHEP 10, 122 (2019), eprint 1904.12882.
  * Kimber et al. (2001) M. A. Kimber, A. D. Martin, and M. G. Ryskin, Phys. Rev. D63, 114027 (2001), eprint hep-ph/0101348.
  * Watt et al. (2003) G. Watt, A. D. Martin, and M. G. Ryskin, Eur. Phys. J. C31, 73 (2003), eprint hep-ph/0306169.
  * Guiot (2020) B. Guiot, Phys. Rev. D 101, 054006 (2020), eprint 1910.09656.
  * Collins (2003) J. C. Collins, Acta Phys. Polon. B34, 3103 (2003), eprint hep-ph/0304122.
  * Nayfeh (1964) A. H. Nayfeh, _Perturbation Methods_ (Wiley, New York, 1964), ISBN 0471399175\.
  * Tangerman and Mulders (1995) R. Tangerman and P. Mulders, Phys. Rev. D51, 3357 (1995), eprint hep-ph/9403227.
  * Boer et al. (2003) D. Boer, P. J. Mulders, and F. Pijlman, Nucl. Phys. B667, 201 (2003), eprint hep-ph/0303034.
  * Gardiner and Majumdar (1970) C. W. Gardiner and D. P. Majumdar, Phys. Rev. D 2, 2040 (1970).
  * Bomhof and Mulders (2007) C. J. Bomhof and P. J. Mulders, in _Proceedings, 15th International Workshop on Deep-inelastic scattering and related subjects (DIS 2007). Vol. 1 and 2: Munich, Germany, April 16-20, 2007_ (2007), pp. 627–630, eprint 0706.4017.
  * Gamberg and Kang (2011) L. Gamberg and Z.-B. Kang, Phys. Lett. B696, 109 (2011), eprint 1009.1936.
  * Anselmino et al. (2014) M. Anselmino, M. Boglione, J. Gonzalez H., S. Melis, and A. Prokudin, JHEP 1404, 005 (2014), eprint arXiv:1312.6261.
  * Signori et al. (2013) A. Signori, A. Bacchetta, M. Radici, and G. Schnell, JHEP 1311, 194 (2013), eprint arXiv:1309.3507.
  * Anselmino et al. (2017) M. Anselmino, M. Boglione, U. D’Alesio, F. Murgia, and A. Prokudin, JHEP 04, 046 (2017), eprint 1612.06413.
  * Cammarota et al. (2020) J. Cammarota, L. Gamberg, Z.-B. Kang, J. A. Miller, D. Pitonyak, A. Prokudin, T. C. Rogers, and N. Sato (Jefferson Lab Angular Momentum), Phys. Rev. D 102, 054002 (2020), eprint 2002.08384.
  * Aslan et al. (2022) F. Aslan, S. Bastami, A. Mahabir, A. Tandogan, and P. Schweitzer, Phys. Rev. D 106, 096010 (2022), eprint 2209.02355.
  * Accardi et al. (2010) A. Accardi, M. E. Christy, C. E. Keppel, W. Melnitchouk, P. Monaghan, J. G. Morfín, and J. F. Owens, Phys. Rev. D 81, 034016 (2010), eprint 0911.2254.
  * Accardi et al. (2016) A. Accardi, L. T. Brady, W. Melnitchouk, J. F. Owens, and N. Sato, Phys. Rev. D 93, 114017 (2016), eprint 1602.03154.
  * Konychev and Nadolsky (2006) A. V. Konychev and P. M. Nadolsky, Phys. Lett. B633, 710 (2006), eprint hep-ph/0506225.
  * Collins and Soper (1982) J. C. Collins and D. E. Soper, Nucl. Phys. B197, 446 (1982).
  * Rogers (2016) T. C. Rogers, Eur. Phys. J. A 52, 153 (2016), eprint 1509.04766.
  * Gonzalez-Hernandez et al. (2022) J. O. Gonzalez-Hernandez, T. C. Rogers, and N. Sato, Phys. Rev. D 106, 034002 (2022), eprint 2205.05750.
  * Nadolsky et al. (2001) P. M. Nadolsky, D. R. Stump, and C. P. Yuan, Phys. Rev. D64, 114011 (2001), eprint hep-ph/0012261.
  * Taghavi and Mirjalili (2017) R. Taghavi and A. Mirjalili, Mod. Phys. Lett. A 32, 1750040 (2017), eprint 1601.04228.
  * Scimemi and Vladimirov (2020) I. Scimemi and A. Vladimirov, JHEP 06, 137 (2020), eprint 1912.06532.
  * Bacchetta et al. (2022) A. Bacchetta, V. Bertone, C. Bissolotti, G. Bozzi, M. Cerutti, F. Piacenza, M. Radici, and A. Signori (MAP), JHEP 10, 127 (2022), eprint 2206.07598.
  * Constantinou et al. (2021) M. Constantinou et al., Prog. Part. Nucl. Phys. 121, 103908 (2021), eprint 2006.08636.
  * He et al. (2022) J.-C. He, M.-H. Chu, J. Hua, X. Ji, A. Schäfer, Y. Su, W. Wang, Y. Yang, J.-H. Zhang, and Q.-A. Zhang (LPC) (2022), eprint 2211.02340.
  * Schweitzer et al. (2013) P. Schweitzer, M. Strikman, and C. Weiss, JHEP 1301, 163 (2013), eprint 1210.1267.
  * Collins and Rogers (2015) J. Collins and T. Rogers, Phys.Rev. D91, 074020 (2015), eprint 1412.3820.
  * Grewal et al. (2020) M. Grewal, Z.-B. Kang, J.-W. Qiu, and A. Signori, Phys. Rev. D 101, 114023 (2020), eprint 2003.07453.
  * Collins et al. (2016) J. Collins, L. Gamberg, A. Prokudin, T. C. Rogers, N. Sato, and B. Wang, Phys. Rev. D 94, 034014 (2016), eprint 1605.00671.
  * Rogers (2020) T. Rogers, Mod. Phys. Lett. A 35, 2030021 (2020), eprint 2008.05351.
  * Bacchetta et al. (2008) A. Bacchetta, F. Conti, and M. Radici, Phys. Rev. D 78, 074010 (2008), eprint 0807.0323.
  * Kang et al. (2010) Z.-B. Kang, J.-W. Qiu, and H. Zhang, Phys. Rev. D 81, 114030 (2010), eprint 1004.4183.
  * Guerrero and Accardi (2020) J. V. Guerrero and A. Accardi (2020), eprint 2010.07339.
  * Boglione et al. (2015) M. Boglione, J. O. G. Hernandez, S. Melis, and A. Prokudin, JHEP 02, 095 (2015), eprint 1412.1383.
  * (60) _Yukawa theory, dis calculation_ , https://sites.google.com/odu.edu/tedrogers/code.