11institutetext: CERTH - ITI, Thessaloniki, Greece
11email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
# TextureCrop: Enhancing Synthetic Image Detection through Texture-based
Cropping

Despina Konstantinidou  Christos Koutlis  Symeon Papadopoulos

###### Abstract

Generative AI technologies produce hyper-realistic imagery that can be used
for nefarious purposes such as producing misleading or harmful content, among
others. This makes Synthetic Image Detection (SID) an essential tool for
defending against AI-generated harmful content. Current SID methods typically
resize input images to a fixed resolution or perform center-cropping due to
computational concerns, leading to challenges in effectively detecting
artifacts in high-resolution images. To this end, we propose TextureCrop, a
novel image pre-processing technique. By focusing on high-frequency image
parts where generation artifacts are prevalent, TextureCrop effectively
enhances SID accuracy while maintaining manageable memory requirements.
Experimental results demonstrate a consistent improvement in AUC across
various detectors by 5.7% compared to center cropping and by 14% compared to
resizing, across high-resolution images from the Forensynths and Synthbuster
datasets.

###### Keywords:

synthetic image detection, AI-generated image detection, pre-processing,
texture-based image cropping

##  1 Introduction

In recent years, the generation of synthetic media has seen intense research
and significant advancements, with numerous novel methods emerging in this
field [25, 51, 17, 49, 53, 14]. Generative Adversarial Networks (GANs) [20]
and Diffusion models [23] have revolutionized the field of synthetic image
generation by enabling the creation of images as well as the modification of
attributes in existing images. While these technologies offer immense creative
potential, they also present serious risks for misuse, such as spreading fake
news and misinformation, facilitating impersonation and fraudulent activities
and influencing political narratives. The quality of generated images has
reached a level where even the most careful observers can be deceived [35],
posing a significant challenge for the majority of Internet users. Moreover,
the diversity of fake image sources complicates the detection process,
necessitating the development of robust and universal detection methods.

Figure 1: Performance (AUC) of detection methods on low and high resolution
images from Forensynths [50] and Synthbuster [3] datasets.

Researchers have recently dedicated significant effort to developing solutions
to address this issue. Synthetic Image Detection (SID) is typically performed
based on either image-(i.e. intensity values) or frequency-based features.
Some methods apply extensive data augmentation [50, 21, 8] in order to improve
generalization, while others exploit the traces left by the generation process
[19, 26, 42]. Recently, numerous emerging methods [2, 39, 48, 11, 33] involve
using features extracted by foundational models like CLIP [43], which has
shown remarkable effectiveness in distinguishing between real and generated
images.

High-resolution images are particularly relevant in the field because they
often represent the higher quality and detail required for various
applications, including those involving synthetic images shared online.
However, limited research has been conducted on such images. With the
advancement of generative AI models, there has been a significant increase in
the generation of high-resolution images. Unlike older models that primarily
produced lower resolution images, contemporary generative AI tools are now
capable of producing high-resolution images that are widely shared across the
Internet. Most state-of-the-art detection methods exhibit significant
performance drops when applied to images like these, as shown in Figure 1.

The high- and low-resolution images tested in our study were derived from the
same datasets as the low-resolution images and were generated using a
consistent process, ensuring their similarity. Therefore, the observed
performance drop can primarily be attributed to their higher resolution. This
challenge arises because the majority of state-of-the-art detectors are
trained using substantially smaller images (in most cases $256\times 256$
pixels). Typically, these detectors adapt images to fixed input layers using
resizing or center cropping. However, resizing entails image resampling and
interpolation, which may erase the subtle high-frequency traces left by the
generation process [10] (see Figure 2(a)) and center cropping an image to a
fraction of its original size can lead to a significant loss of information
(see Figure 2(b)). When detectors are trained and evaluated on small images,
resizing or cropping has very little effect on their performance since fewer
high-frequency traces are lost, but when dealing with images of higher
resolution, this effect is much more pronounced.

To address this issue, we conduct an extensive investigation of pre-processing
techniques during inference and introduce TextureCrop, a novel approach that
targets high-frequency components, which encapsulate fine details and textures
crucial for distinguishing synthetic from real images. The proposed method,
leverages a sliding window technique to systematically analyze image patches.
It selectively retains patches exhibiting significant texture information
while discarding those that do not. Our experimental results on high-
resolution images demonstrate substantial improvements in detection metrics:
an increase in balanced accuracy ranging from 1.85% to 11.06% compared to the
established pre-processing methods, an improvement in area under the ROC curve
ranging from 2.11% to 17.07%, and an enhancement in average precision ranging
from 4.57% to 13.96%.

(a)

(b)

(c)

(d)

Figure 2: Example of the crops created by using resizing (a), center cropping
(b), slide cropping (c) and texture cropping (d).

Our main contributions can be summarized as follows:

  1. 1.

We introduce TextureCrop, an innovative image pre-processing method aimed at
enhancing the detection of high-resolution synthetic images by focusing on
high-frequency texture components.

  2. 2.

We conduct a comprehensive analysis of various pre-processing techniques
applied to high-resolution images, exploring their impacts on detection
performance. Our findings demonstrate that TextureCrop consistently
outperforms existing pre-processing approaches across various detectors.

  3. 3.

We present a detailed ablation study to fine-tune the parameters of the
proposed TextureCrop method and the aggregation of results.

##  2 Related Work

###  2.1 Synthetic image generation

Over the past few years, image generation models have gained considerable
attention due to their ability to create high-quality images that closely
resemble real ones. Most of the preliminary research has centered on images
generated by Generative Adversarial Networks (GAN), as these architectures
were introduced first and have dominated the field. Since the introduction of
GANs, numerous variants have been developed [37, 44, 56, 29, 9, 5, 31, 40].
StyleGAN [31] for instance, as well as its subsequent versions StyleGAN2 [32]
and StyleGAN3 [30], introduced a style - based architecture with adaptive
instance normalization, enabling fine - grained control over image attributes
and improved disentanglement of the latent space for generating highly
realistic and diverse images.

More recently, diffusion models [38, 46, 45, 4, 47, 41, 1, 36] have emerged as
an alternative approach to image generation. Notable among these are text-to-
image generation models like DALL-E 2 [45] and DALL-E 3 [4]. DALL-E 2 employs
a transformer - based encoder to convert textual descriptions into a latent
space representation, which is then decoded using an auto-regressive model to
generate images. DALL-E 3 builds upon its predecessor with improved training
methods and dataset utilization. Stable diffusion models, such as Stable
Diffusion 2 [47] and Stable Diffusion XL [41], utilize latent representations
from training data to progressively add noise to images. Firefly [1],
developed by Adobe, is another notable addition to text-to-image generation
tools.

In addition to GANs and diffusion models, there are low-level vision
generative models focused on pixel-level manipulation and enhancement. For
example, the Second Order Attention Network (SAN) [15] enhances single image
super-resolution using second - order attention mechanisms and recursive
feature networks. Similarly, the Seeing In The Dark (SITD) [7] model employs
fully convolutional networks to simulate long - exposure photography from low-
light conditions based on short - exposure raw camera input.

###  2.2 Synthetic image detection

The development of robust deep learning - based detectors for synthetic images
has been a major focus of recent research. Training set diversity has been
found to help generalize to unseen architectures, as shown in [50], where a
simple pre-trained ResNet50 [22] is trained on 20 different object classes of
real images from LSUN [54] and ProGAN [29] images. It has also been found
important to avoid resizing, as observed in [10], as it may erase the
artifacts left by the generative model. In order to preserve these subtle
high-frequency traces, some methods propose working on local patches [6], as
well as analyzing both local and global features [28]. Another method proposes
avoiding downsampling steps in the first layers of the network [21, 10].

Recently, there have been efforts to leverage the capabilities of pre-trained
vision - language models (VLMs) for synthetic image detection purposes [2, 39,
48, 11]. For instance, [39] uses the same training protocol as in [50], but
employs a pretrained VLM, the Contrastive Language - Image Pre-Training (CLIP)
model, as a feature extractor instead of training a ResNet - 50. Another
method [33] proposes the extraction of features not only from the last layer
of CLIP’s image - encoder, but also from intermediate layers.

Additionally, recent works have focused on analyzing the texture
characteristics of images. It has been observed in [34] that synthetic faces
exhibit significantly different textures compared to real ones. Additionally,
[55] highlights that texture patches contain more distinct traces left by
generative models than global semantic information, while [27] demonstrates
that generative models tend to focus on generating the patches with rich
textures to make the images more realistic and neglect the hidden noise caused
by camera capture present in simple patches.

Building upon the interest in image texture, we analyze the impact of rich and
poor texture regions within images on detectors’ performance. Unlike [55],
that focuses on using high-pass filters for texture classification across two
branches and [27] that uses noise patterns from single patches to identify
synthetic images, our approach is a pre-processing method that specifically
selects regions with rich textures for processing by detectors.

##  3 Proposed Approach

###  3.1 Problem Formulation

In synthetic image detection, the goal is to determine whether an image,
$\mathrm{I}\in\mathbb{R}^{C\times W\times H}$, is generated or real, where
$C$, $H$, and $W$ denote the number of channels (in most cases 3), height and
width of the image, respectively. Typically, this involves processing the
image with a detection model $f$ that outputs a SID score $s$, indicating the
likelihood of the image being synthetic. In the traditional approach, a single
image $\mathrm{I}$ (or a batch of images), is fed into the model $f$,
resulting in a SID score $s\in[0,1]$:

$s=f(\mathrm{I})$ (1)

This approach assumes that a resized or cropped version of an image is
processed, which may result in significant loss of information that can alter
the outcome of the detection. In some cases, when the image is processed as a
whole, processing times and resource needs increase significantly. In our
proposed approach, denoted as TextureCrop, the image $\mathrm{I}$ is divided
into a batch of smaller texture-rich crops
$\\{\mathrm{C}_{i}\\}_{i=1}^{N_{I}}\in\mathbb{R}^{N_{I}\times C\times
W^{c}\times H^{c}}$, where $C_{i}$ represents the i-th crop with dimensions
$C\times W^{c}\times H^{c}$, where C is the number of channels and $W^{c}$ and
$H^{c}$ are the width and height of each crop and $N_{I}$ is the number of
crops generated for the image I. The number of crops depends on the texture
information of each image. Then this batch of crops is processed by the same
detection model $f$, returning a SID score $s_{i}$ for each crop
$\mathrm{C}_{i}$. The final SID score $s$ is obtained by aggregating the
scores from the batch of crops with an aggregation function $\mathfrak{A}$:

$\displaystyle\\{\mathrm{C}_{i}\\}_{i=1}^{N_{I}}=\text{TextureCrop}(\mathrm{I})$
(2) $\displaystyle s_{i}=f(\mathrm{C}_{i}),\quad\text{for }i=1,\ldots,N_{I}$
(3) $\displaystyle s=\mathfrak{A}(\\{s_{i}\\}_{i=1}^{N_{I}})$ (4)

###  3.2 Method Overview

The proposed image cropping technique, denoted as TextureCrop, leverages the
idea that generative models leave artifacts in the high-frequency components
of synthetic images. It is designed to enhance synthetic image detection by
excluding texture-less regions from input images. This approach expands upon
previous methods in image forensics, such as [12, 13, 24] that prioritize
processing textured regions to improve detection accuracy.

Employing a sliding window approach with user-defined window size $\omega$ and
stride $\beta$, TextureCrop systematically traverses $\mathrm{I}$ to generate
crops. Each crop is converted to grayscale, and its standard deviation (SD) of
pixel intensities is computed to assess the presence of texture. Then, the
method utilizes a standard deviation thresholding mechanism to filter cropped
regions (see Figure 2(d)).

$\text{If }\sigma_{i}>\tau,\text{ keep }C_{i}\text{; otherwise discard,
}i=1,\ldots,N_{I},$

where $\sigma_{i}$ is the standard deviation of pixel intensities of the crop
$C_{i}$ and $\tau$ is the threshold. The resulting cropped regions

$\\{\mathrm{C}_{i}\\}_{i=1}^{n_{I}}=\text{TextureCrop}(\textit{I},\omega,\beta,\tau),\quad
n_{i}\leq N_{i}$

are then fused using an appropriate method $\mathfrak{A}$, and a final
prediction is generated. In cases where no suitable cropped regions meet the
SD threshold $\tau$, a default center crop of $224\times 224$ pixels is
returned. This fallback ensures that at least one cropped image is available
for downstream analysis, maintaining consistency in output format.

To further explain the intuition behind our cropping method choice, we
introduce SlideCrop, which is a cropping method that mirrors TextureCrop but
skips the step of removing regions lacking texture. Consequently, all patches
generated by the sliding window are retained in SlideCrop. Among the examined
images, spanning resolutions from $1024\times 1024$ to $6030\times 4032$,
TextureCrop filters out almost half of the patches that SlideCrop would keep.
This curation enhances performance and computational efficiency. With this
slight modification, TextureCrop minimizes the number of crops processed by
detectors, as can be seen in Figure 2(d), reducing processing time and
ensuring that only the most pertinent crops are retained for analysis, thus
optimizing overall performance.

###  3.3 Implementation Details

TextureCrop and SlideCrop methods begin by evaluating the resolution of the
input image $\mathrm{I}$. If $\mathrm{I}$ exceeds $2048\times 2048$ pixels
then the image undergoes a center crop operation to limit its size to
$2048\times 2048$ or to its maximum possible size if the image is not square.
While it is technically possible to process higher resolution images, this
comes at a significant computational cost. This approach ensures more
predictable runtime behavior, as the complexity of processing is directly
related to the original resolution and specific parameters of the generative
model. The size aligns well with the capabilities of most known generative
models. Until now, only a few generative models have demonstrated the
capability to generate images up to $2048\times 2048$ resolution, making this
threshold a suitable compromise for effective processing and analysis. In most
cases, images with resolutions higher than $1024\times 1024$ often result from
super-resolution methods.

##  4 Experimental Setup

###  4.1 Competing Image Pre-processing Approaches for SID

Pre-processing techniques employed to prepare images for precise analysis by
SID models are limited. Many contemporary detectors commonly utilize center
cropping or resizing as standard pre-processing steps. Center cropping
involves selecting a square or rectangular area from the center of an image,
focusing on retaining key features while potentially discarding peripheral
details that may also contain valuable information, leading to notable
information loss in some cases. Conversely, resizing adjusts an image to a
specified target size through scaling, a process that can introduce
distortions and diminish fine details, particularly when altering images to
significantly divergent dimensions.

More recent works, such as [33], have adopted 10-cropping for handling images
of higher resolutions. This involves extracting 10 distinct crops from an
image: its four corners, central region, and their horizontal flips. Each crop
is then independently passed through the detection model, and predictions are
combined to enhance accuracy. Moreover, our method incorporates slide-cropping
as a preliminary step, involving the systematic sliding of a window across the
image to extract sub-images at each position. This approach aims to capture
local details comprehensively throughout the entire image, but we have refined
this process in our method to limit the number of extracted sub-images while
still ensuring thorough coverage and detail capture.

###  4.2 Datasets

We evaluated our method using high-resolution data sourced from two primary
datasets: Forensynths [50] and Synthbuster [3]. After filtering out images
with resolutions lower than $1024\times 1024$, we collected a total of 6,462
images. Among these, 4,272 images are generated by eight different generative
models, while the remaining 2,190 images are real images. This distribution is
detailed in Table 1. Specifically, the Forensynths dataset includes high-
resolution images generated by the SAN [15] and the SITD model [7] and faces
generated by the StyleGAN [31] model available on the Whichfaceisreal website
[52]. For the Synthbuster dataset, high-resolution images were generated by
DALL-E 2 [45], DALL-E 3 [4], Firefly [1], Stable Diffusion 2 [47], and Stable
Diffusion XL [41]. Additionally, for real data, we used 1000 uncompressed and
unprocessed high-resolution images from the RAISE dataset [16]. The
Forensynths dataset includes its own real data.

Table 1: Forensynths [50] and Synthbuster [3] datasets after removing low
resolution images.

Subset/Generative model |  Dataset |  Total images |  Real images |  Generated images |  Resolutions produced by the generative model
---|---|---|---|---|---
SAN |  Forensynths |  20 |  10 |  10 |  depends on the resolution of the input image
SITD |  Forensynths |  360 |  180 |  180 |  depends on the resolution of the input image
Whichfaceisreal |  Forensynths |  2000 |  1000 |  1000 |  up to $1024\times 1024$
DALL-E 2 |  Synthbuster |  1000 |  * |  1000 |  $256\times 256$, $512\times 512$, $1024\times 1024$
DALL-E 3 |  Synthbuster |  1000 |  * |  1000 |  $1024\times 1024$, $1024\times 1792$, $1792\times 1024$
Firefly |  Synthbuster |  1000 |  * |  1000 |  $2048\times 2048$, $2304\times 1792$, $1792\times 2304$, $2688\times 1536$
Stable Diffusion 2 |  Synthbuster |  41 |  * |  41 |  $512\times 512$, $768\times 768$
Stable Diffusion XL |  Synthbuster |  41 |  * |  41 |  $1024\times 1024$, $1152\times 896$, $896\times 1152$, $1216\times 832$, $832\times 1216$, $1344\times 768$, $768\times 1344$, $1536\times 640$, $640\times 1536$

* For real data, we used 1000 high-resolution images from the RAISE dataset [16].

###  4.3 SID Detectors

We include the following state-of-the-art detectors in our study: GramNet [34]
that leverages global image texture representations, CNNDetect [50] a ResNet50
with blurring and compression augmentation (with two different probabilities
of the image being augmented: 0.1 and 0.5), GANID [21] a ResNet50 with intense
augmentation that avoids down-sampling in the first layer (trained on ProGAN
or StyleGAN2 data), DMID [10] which proposes the fusion of the logits of two
models same as described before but one trained on GAN and the other on
diffusion data, UnivFD [39] that uses linear probing on features extracted
from CLIP’s ViT-L/14 image encoder [18], RINE [33] that uses the image
representations extracted by intermediate layers of CLIP’s image encoder
(trained on 4 object classes of ProGAN data and on Latent Diffusion Model
(LDM) data and PatchCraft [55], which exploits the inter-pixel correlation
contrast between rich and poor texture regions within an image.

###  4.4 Aggregation methods

Our method results in a large number of crops per image, creating the need for
an aggregation process to consolidate the information from all the individual
crops into a single prediction. To achieve this, we explore the following
aggregation methods:

  * •

Average: This calculates the average of the logits obtained from the crops,
aiming to find a balanced representation of the predictions.

  * •

Majority Voting: This involves collecting the predictions from all crops and
selecting the most commonly predicted class among the crops, assuming that
this is more likely to be the correct one.

  * •

Max: This selects the maximum logit value among the logits obtained from all
the crops, assuming that the highest logit value corresponds to the most
confident prediction among the crops. However, in scenarios where a single
crop produces an unusually high logit value due to random fluctuations or
outliers, false positives can occur, leading to an incorrect prediction.

  * •

Median: This computes the median of the logits obtained from multiple crops.
By selecting the middle value in the sorted list of logits, this method aims
to mitigate the influence of outliers.

  * •

Weighted Average: This assigns a weight to the probability obtained from each
crop and calculates the average across all crops. After partitioning the [0,1]
range into intervals (interval length is specified by the user, in our case it
is set to 0.1 after experimentation), the method calculates the percentage of
values within each interval. These percentages determine the weights assigned
to each value, with higher weights assigned to values in the intervals that
occur more frequently. By multiplying each value by its weight and summing the
weighted values, the method effectively combines the contributions of all
input data. The final fused value represents an averaged representation of the
input data with greater emphasis on values that are more prevalent among the
sources.

###  4.5 Evaluation protocol

To evaluate the performance of the proposed pre-processing technique, we
compute balanced accuracy (BA), average precision (AP), and area under the ROC
curve (AUC) metrics. Each of these addresses different aspects of SID
performance. BA accounts for class imbalance by averaging recall across all
classes. AP summarizes the precision - recall curve, which is especially
useful for imbalanced datasets. AUC is also particularly valuable, because it
is invariant to class distribution and provides a comprehensive measure of
performance across all classification thresholds. The results presented are
averages across all datasets, ensuring a robust evaluation of the method’s
performance.

##  5 Results

###  5.1 Detection Gains

Figure 3: Performance (BA/AP/AUC) of detection methods on high resolution
images from Forensynths and Synthbuster datasets using different pre-
processing methods.

In this section, we evaluate the effectiveness of the proposed texture
cropping method in enhancing synthetic image detection across various deep
learning models. We compare texture cropping against several standard pre-
processing techniques, including center cropping, resizing, 10-cropping, and
slide cropping.

On average across all detectors and datasets, the method offers a 3.13%
increase in BA compared to center cropping across the 10 methods tested, along
with a 5.17% increase in AP and a 5.71% increase in AUC (see Figure 3). In
comparison to resizing, it demonstrates an 9.33% increase in BA, a 15.78%
increase in AP and a 13.96% increase in AUC. Furthermore, it exhibits a 1.85%
increase in BA compared to ten cropping along with a 2.11% increase in AP and
a 4.57% increase in AUC. The results demonstrate a high degree of consistency
across all datasets except for the DALL-E 3 data, which seem to perform better
with resizing. Additionally, texture cropping not only outperforms slide
cropping in most cases but also proves to be more computationally efficient,
processing almost half the images handled by slide cropping. Consequently,
this approach leads to significantly reduced processing time. These findings
highlight the substantial enhancement in detection performance achieved with
texture cropping across various metrics and models, while maintaining
efficient processing times.

###  5.2 Ablation Studies

In this section, we present the ablation studies conducted to fine-tune the
parameters of the proposed method and the aggregation of results, and
introduce a variant of the method. We explore the impact of different
parameter values on the performance, ultimately determining the optimal
settings for SID.

####  5.2.1 Stride ():

The stride parameter, , controls the step size of the sliding window used in
cropping the image. We experimented with strides of 50, 100, 200, 224, and 300
pixels to determine the optimal balance between computational efficiency and
detection performance. Our experiments revealed that a stride of 200 pixels
provides the best performance on most detection methods, as illustrated in
Figure 4(a). This setting offers sufficient coverage of the image while
maintaining reasonable computational demands.

(a) Stride ($\beta$)

(b) SD Threshold ($\sigma$)

(c) Window Size ($\omega$)

Figure 4: Box plot of AUC distribution across detection methods for different
stride parameters (a), SD thresholds (b), and window sizes (c).

####  5.2.2 Standard Deviation Threshold ():

The standard deviation threshold, , is crucial for filtering out texture-less
regions from the cropped images. We consider 0.05, 0.1, 0.15, and 0.2 as
threshold candidates. As observed in Figure 4(b), a standard deviation
threshold of 0.1 is ideal. This threshold effectively filters out regions with
minimal texture, which are less likely to contain high-frequency artifacts
introduced by generative models.

####  5.2.3 Window Size ():

Most methods in our analysis have a set input size of $224\times 224$ pixels,
except for GANID (ProGAN and StyleGAN2) and DMID. Therefore, we tested our
method with different window sizes to accommodate these variations. We set the
window size parameter, , to $224\times 224$, $512\times 512$, and $1024\times
1024$ pixels to determine the optimal size for capturing high-frequency
artifacts while maintaining computational efficiency. Our findings indicate
that a window size of $224\times 224$ pixels provides the best results (Figure
4(c)).

####  5.2.4 Aggregation:

Figure 5: Performance (AUC) of detection methods using TextureCrop with
different aggregation methods.

Another perspective we explored involves the aggregation of the predictions
across multiple image crops. Each detector model produces a logit per crop,
thus to consolidate the results from multiple crops, we tested various
aggregation methods, including average, max, median, majority vote, and
weighted average. The results of these tests are outlined in Figure 5. In our
experimental setup, average, median and weighted average emerge as the most
effective aggregation strategies, consistently delivering competitive
performance across various detection methods. On the other hand, majority
voting and max aggregation appear to be less effective overall, especially in
scenarios where there is significant uncertainty among the crops.

####  5.2.5 Number of Crops (n):

Additionally, we explored the impact of retaining a predetermined number of
crops rather than applying thresholding. Our method favors thresholding
because it allows for a clear cutoff point, ensuring that only the most
relevant crops, as determined by our chosen metric, are retained. However,
determining the optimal threshold can be challenging, as it requires complex
and context-specific analysis to identify an appropriate value. Retaining a
fixed number of crops also ensures consistent processing time for each image,
which is crucial for scalability and efficiency in practical applications.
Thus, this new variant of TextureCrop, denoted as FixedTextureCrop(I, m, p,
n), selects $n$ crops based on the metric $m$. The parameter $p$ determines
from which part of the distribution the crops are selected.

First, we examine the relationship between the number of crops and detection
effectiveness for FixedTextureCrop(m=SD, p=top, n), that selects the top n
crops based on SD. The results of this analysis are presented in Table 2. Our
analysis reveals that the this method consistently achieves the highest
performance metrics across various deep learning models when selecting 15
crops from the input image. By selecting a greater number of crops, the
FixedTextureCrop(m=SD, p=top, n) method captures more diverse regions of
interest within the image, enhancing the model’s ability to detect high-
frequency artifacts introduced by generative models. Therefore, our results
emphasize the importance of considering an adequate of crops in the design of
cropping methodologies for synthetic image detection.

Table 2: Performance (BA/AP/AUC) of FixedTextureCrop(m=SD, p=top, n) method across different n parameters. Best performance is denoted with bold. BA/AP/AUC |  n=1 |  n=2 |  n=5 |  n=10 |  n=15
---|---|---|---|---|---
GramNet |  50.19/48.07/50.49 |  48.69/49.53/51.18 |  48.72/50.14/51.86 |  49.96/50.60/52.44 |  50.04/50.87/52.66
CNNDetect (0.1) |  61.08/54.36/64.44 |  61.75/55.31/65.35 |  62.61/55.77/65.93 |  62.94/56.81/67.06 |  62.82/57.27/68.30
CNNDetect (0.5) |  61.08/53.46/64.44 |  61.75/53.91/65.3 |  62.61/54.31/65.93 |  62.94/54.87/67.06 |  62.82/55.31/68.30
GANID (ProGAN) |  64.09/56.35/67.66 |  64.47/56.73/68.63 |  63.91/57.97/69.30 |  64.51/59.24/70.66 |  65.65/59.84/71.38
GANID (StyleGAN2) |  68.16/62.36/73.28 |  68.64/63.95/74.24 |  69.18/64.34/74.13 |  69.34/65.15/74.11 |  69.11/65.56/74.60
DMID |  76.82/83.13/83.13 |  76.59/84.21/82.79 |  78.40/85.11/83.77 |  79.11/85.91/84.20 |  79.14/86.30/84.52
UnivFD |  62.62/57.02/66.56 |  63.93/59.98/68.28 |  66.08/63.14/70.22 |  67.73/65.88/72.42 |  69.02/66.85/73.64
RINE (4) |  68.74/64.49/69.95 |  69.24/65.27/70.35 |  71.75/68.24/72.80 |  73.25/71.43/74.57 |  74.54/71.63/75.11
RINE (ldm) |  72.51/77.62/79.04 |  72.96/81.72/80.36 |  75.21/83.57/81.49 |  75.94/85.00/82.66 |  76.25/85.50/82.81
PatchCraft |  69.05/56.11/73.00 |  70.06/57.37/72.91 |  71.27/59.27/74.50 |  71.13/61.66/74.99 |  71.44/61.88/75.31

####  5.2.6 Image crops aggregation (parameters p and m):

When it comes to selecting crops from different parts of the distribution we
explored the following options: top n crops, bottom n crops and top-bottom-
intermediate n crops (TBI), where we select the top $\frac{n}{3}$ crops, the
bottom $\frac{n}{3}$ crops, and $\frac{n}{3}$ intermediate crops. When
contrasting top and bottom crop selections based on SD, we note that the
FixedTextureCrop(m=SD, p=bottom, n=15) method exhibits relatively poorer
performance compared to FixedTextureCrop(m=SD, p=top, n=15). On average,
FixedTextureCrop(m=SD, p=bottom, n=15) demonstrates a 6.93% decrease in BA, a
7.15% decrease in AP, and a 9.13% decrease in area under the curve. This
indicates that focusing on regions with low SD may not adequately capture
critical features for effective detection. The FixedTextureCrop(m=SD, p=TBI,
n=15) method performs better than FixedTextureCrop(m=SD, p=bottom, n=15) but
still falls short of FixedTextureCrop(m=SD, p=top, n=15). On average, it
registers a 1.92% lower BA, a 2.27% lower AP, and a 4.13% lower area under the
curve. These differences in performance are clearly illustrated in Figure
6(a).

In addition to exploring SD as a criterion for selecting crops in our cropping
method, we conducted further experimentation to assess the effectiveness of
alternative approaches. Specifically, we investigated the use of entropy and
autocorrelation as metrics for crop selection. Entropy measures the
unpredictability or complexity of pixel values, while autocorrelation
evaluates the similarity of a pixel to its neighboring pixels. Both metrics
can provide insights into the texture of image regions.

(a)

(b)

Figure 6: Performance (AUC) of different detectors using
FixedTextureCrop(m=SD, p, n=15) method based on different texture selection
criteria (a) and FixedTextureCrop(m, p=top, n=15) method based on different
metrics for computing texture (b).

The FixedTextureCrop(m, p=top, n=15) method selects the top 15 crops from the
input image based on the SD, when m=SD, entropy, when m=entropy, and
autocorrelation, when m=autocorrelation, of pixel intensities within each crop
region. Across various models, it appears that the SD- and entropy-based
methods perform similarly on average and outperform the autocorrelation-based
method (see Figure 6(b)), suggesting that utilizing SD or entropy for
selecting top crops leads to superior results. Remarkably, in some cases,
FixedTextureCrop method outperforms even TextureCrop, while offering the
advantage of a fixed output. The choice between the two depends on the
preferences of the user. TextureCrop selects an unknown number of crops for an
image, as different images have different textures. In contrast, the
FixedTextureCrop method offers a fixed number of crops, which may be
preferable in scenarios where a fixed processing time or output size is
required.

##  6 Conclusion

In this study, we conducted a comprehensive analysis of post-processing
methods applied during inference for synthetic image detection purposes and
introduced TextureCrop and FixedTextureCrop, two novel approaches that analyze
texture variability for crop selection. By targeting regions of the image with
high texture variability, these methods successfully capture high-frequency
artifacts introduced by generative models, resulting in enhanced detection
accuracy across various deep learning models. Overall, TextureCrop and
FixedTextureCrop methods emerge as promising solutions for addressing the
challenges associated with synthetic images and for advancing the capabilities
of detection systems in accurately identifying such artifacts.

##  7 Acknowledgments

This work is partially funded by the Horizon 2020 European project vera.ai
under grant agreement no. 101070093.

## References

  * [1] Adobe Firefly: https://www.adobe.com/sensei/generative-ai/firefly.html (2024)
  * [2] Amoroso, R., Morelli, D., Cornia, M., Baraldi, L., Bimbo, A.D., Cucchiara, R.: Parents and children: Distinguishing multimodal deepfakes from natural images. arXiv preprint arXiv:2304.00500v1 (2023)
  * [3] Bammey, Q.: Synthbuster: Towards detection of diffusion model generated images. In: IEEE Open Journal of Signal Processing (2023)
  * [4] Betker, J., Goh, G., Jing, L., Brooks, T., Wang, J., Li, L., Ouyang, L., Zhuang, J., Lee, J., Guo, Y., Manassra, W., Dhariwal, P., Chu, C., Jiao, Y.: https://openai.com/dall-e-3 (2023)
  * [5] Brock, A., Donahue, J., Simonyan, K.: Large scale gan training for high fidelity natural image synthesis. In: ICLR (2019)
  * [6] Chai, L., Bau, D., Lim, S.N., Isola, P.: What makes fake images detectable? understanding properties that generalize. In: ECCV (2020)
  * [7] Chen, C., Chen, Q., Xu, J., Koltun, V.: Learning to see in the dark. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
  * [8] Chen, L., Zhang, Y., Song, Y., Wang, J., Liu, L.: Ost: Improving generalization of deepfake detection via one-shot test-time training. In: NeurIPS (2022)
  * [9] Choi, Y., Choi, M., Kim, M., Ha, J.W., Kim, S., Choo, J.: Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In: CVPR (2018)
  * [10] Corvi, R., Cozzolino, D., Zingarini, G., Poggi, G., Nagano, K., Verdoliva, L.: On the detection of synthetic images generated by diffusion models. In: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1–5 (2023)
  * [11] Cozzolino, D., Poggi, G., Corvi, R., Nießner, M., Verdoliva, L.: Raising the bar of ai-generated image detection with clip. arXiv preprint arXiv:2312.00195 (2023)
  * [12] Cozzolino, D., Poggi, G., Verdoliva, L.: Splicebuster: A new blind image splicing detector. In: IEEE International Workshop on Information Forensics and Security (WIFS) (2015)
  * [13] Cozzolino, D., Verdoliva, L.: Noiseprint: a cnn-based camera model fingerprint. arXiv preprint arXiv:1808.08396 (2018)
  * [14] Croitoru, F.A., Hondru, V., Ionescu, R.T., Shah, M.: Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14 (2022)
  * [15] Dai, T., Cai, J., Zhang, Y., Xia, S.T., Zhang, L.: Second-order attention network for single image super-resolution. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11065–11074 (2019)
  * [16] Dang-Nguyen, D.T., Pasquini, C., Conotter, V., Boato, G.: Raise: a raw images dataset for digital image forensics. In: Proceedings of the 6th ACM Multimedia Systems Conference (2015)
  * [17] Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. In: Advances in Neural Information Processing Systems (2021)
  * [18] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [19] Frank, J., Eisenhofer, T., Schönherr, L., Fischer, A., Kolossa, D., Holz, T.: Leveraging frequency analysis for deep fake image recognition. In: Proceedings of the 37th International Conference on Machine Learning, PMLR 119 (2020)
  * [20] Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems (NIPS) (2014)
  * [21] Gragnaniello, D., Cozzolino, D., Marra, F., Poggi, G., Verdoliva, L.: Are gan generated images easy to detect? a critical analysis of the state-of-the-art. In: IEEE ICME (2021)
  * [22] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2016)
  * [23] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: Advances in Neural Information Processing Systems. vol. 33, pp. 6840–6851 (2020)
  * [24] Iakovidou, C., Zampoglou, M., Papadopoulos, S., Kompatsiaris, Y.: Content-aware detection of jpeg grid inconsistencies for intuitive image forensics. Journal of Visual Communication and Image Representation (2018)
  * [25] Jabbar, A., Li, X., Omar, B.: A survey on generative adversarial networks: Variants, applications, and training. ACM Computing Surveys 54, 1–49 (2020)
  * [26] Jeong, Y., Kim, D., Ro, Y., Choi, J.: Frepgan: Robust deepfake detection using frequency- level perturbations. In: AAAI (2022)
  * [27] Jiaxuan Chen, Jieteng Yao, L.N.: A single simple patch is all you need for ai-generated image detection. arXiv preprint arXiv:2402.01123 (2024)
  * [28] Ju, Y., Jia, S., Ke, L., Xue, H., Nagano, K., Lyu, S.: Fusing global and local features for generalized ai-synthesized image detection. In: IEEE ICIP (2022)
  * [29] Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of gans for improved quality, stability, and variation. In: ICLR (2018)
  * [30] Karras, T., Aittala, M., Laine, S., Härkönen, E., Hellsten, J., Lehtinen, J., Aila, T.: Alias-free generative adversarial networks. In: 35th Conference on Neural Information Processing Systems (NeurIPS). (2021)
  * [31] Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: CVPR (2019)
  * [32] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: CVPR (2020)
  * [33] Koutlis, C., Papadopoulos, S.: Leveraging representations from intermediate encoder-blocks for synthetic image detection. arXiv preprint arXiv:2402.1909 (2024)
  * [34] Liu, Z., Qi, X., Torr, P.: Global texture enhancement for fake face detection in the wild. In: CVPR (2020)
  * [35] Lu, Z., Huang, D., Bai, L., Qu, J., Wu, C., Liu, X., Ouyang, W.: Seeing is not always believing: Benchmarking human and model perception of ai-generated images. In: Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2023)
  * [36] Midjourney: https://www.midjourney.com (2024)
  * [37] Mirza, M., Osindero, S.: Conditional generative adversarial nets. In: Advances in Neural Information Processing Systems (NIPS) (2014)
  * [38] Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., Chen, M.: Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In: ICML (2021)
  * [39] Ojha, U., Li, Y., Lee, Y.J.: Towards universal fake image detectors that generalize across generative models. In: CVPR. pp. 24480–24489 (2023)
  * [40] Park, T., Liu, M.Y., Wang, T.C., Zhu, J.Y.: Semantic image synthesis with spatially-adaptive normalization. In: CVPR (2019)
  * [41] Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., Rombach, R.: Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 (2023)
  * [42] Qian, Y., Yin, G., Sheng, L., Chen, Z., Shao, J.: Thinking in frequency: Face forgery detection by mining frequency-aware clues. In: ECCV (2020)
  * [43] Radford, A., Kim, J., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., et al., J.C.: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748–8763 (2021)
  * [44] Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. In: ICLR (2016)
  * [45] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 (2022)
  * [46] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10684–10695 (2022)
  * [47] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: Stable diffusion (2022)
  * [48] Sha, Z., Li, Z., Yu, N., Zhang, Y.: Defake: Detection and attribution of fake images generated by text-to-image diffusion models. In: ACM SIGSAC Conference on Computer and Communications Security. pp. 3418–3432 (2023)
  * [49] Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score-based generative modeling through stochastic differential equations. In: Proceedings of the International Conference on Learning Representations (2021)
  * [50] Wang, S.Y., Wang, O., Zhang, R., Owens, A., Efros, A.A.: Cnn-generated images are surprisingly easy to spot… for now. In: CVPR. pp. 8695–8704 (2020)
  * [51] Wang, Z., She, Q., Ward, T.: Generative adversarial networks in computer vision: A survey and taxonomy. ACM Computing Surveys (2021)
  * [52] Which face is real?: http://www.whichfaceisreal.com
  * [53] Yang, L., Zhang, Z., Song, Y., Hong, S., Xu, R., Zhao, Y., Zhang, W., Cui, B., Yang, M.H.: Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys 56, 1–39 (2023)
  * [54] Yu, F., Zhang, Y., Song, S., Seff, A., Xiao, J.: Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 (2015)
  * [55] Zhong, N., Xu, Y., Li, S., Qian, Z., Zhang, X.: Patchcraft: Exploring texture patch for efficient ai-generated image detection. arXiv preprint arXiv:2311.12397 (2024)
  * [56] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: ICCV (2017)


# Multi-task recommendation system for scientific papers with high-way
networks

1st Aram Karimi  University of Gothenburg
<EMAIL_ADDRESS>2nd Simon Dobnik  University of Gothenburg
<EMAIL_ADDRESS>
###### Abstract

Finding and selecting the most relevant scientific papers from a large number
of papers written in a research community is one of the key challenges for
researchers these days. As we know, much information around research interest
for scholars and academicians belongs to papers they read. Analysis and
extracting contextual features from these papers could help us to suggest the
most related paper to them. In this paper, we present a multi-task
recommendation system (RS) that predicts a paper recommendation and generates
its meta-data such as keywords. The system is implemented as a three-stage
deep neural network encoder that tries to maps longer sequences of text to an
embedding vector and learns simultaneously to predict the recommendation rate
for a particular user and the paper’s keywords. The motivation behind this
approach is that the paper’s topics expressed as keywords are a useful
predictor of preferences of researchers. To achieve this goal, we use a system
combination of RNNs, Highway and Convolutional Neural Networks to train end-
to-end a context-aware collaborative matrix. Our application uses Highway
networks to train the system very deep, combine the benefits of RNN and CNN to
find the most important factor and make latent representation. Highway
Networks allow us to enhance the traditional RNN and CNN pipeline by learning
more sophisticated semantic structural representations. Using this method we
can also overcome the cold start problem and learn latent features over large
sequences of text.

###### Index Terms:

1\. Scientific paper recommendation, 2. Deep Learning, 3. Multi task learning,
4. Collaborative filtering

##  I Introduction

Scientific research is a systematic approach to reporting advanced knowledge
to the world. Many studies have looked at the growth rate of scientific
publications and results shows that science has been progressed greatly over
the past century. Finding and having access to publications among this huge
number of articles, plays a key role for researchers. This motivates
utilisation of this massive data and develop a practical system to suggest
most relevant articles to researchers. Using a recommendation system (RS)
could help to develop a systematic approach to filter out information and
recommend papers that align to their research interests.
Generally, Recommendation system can be built using various method such as
collaborative filtering (CF), a content-based and hybrid technique [25]. CF
methods produce user specific recommendations of items, based on patterns of
ratings or usage without the need for external information about either items
or users. In order to make recommendations, CF systems need to relate two
fundamentally different entities: items and users. There are two main
approaches to facilitate this in CF: the neighbourhood approach and latent
factor models [27]. Neighbourhood methods compute relationships between items
or, alternatively, between users [14]. Latent factor models, such as matrix
factorisation, represent an approach by mapping items and users to a low-
dimensional space, and compute a user-item preference score based on a
function of two (i.e., user and item) latent vectors [17]. In content-based
techniques, the system attempts to recommend items similar to those a user
preferred in the past. In this case, the recommendations are based on the
content of a given item, not on other users’ opinions as in the case of
collaborative filtering. These techniques are vulnerable to over-fitting, but
their advantage is, they don’t need any data from other users side [20].

Both the CF and the content-based approach have some weaknesses. In the CF
this is the cold start issue and the primary drawback of content-based
filtering systems is their tendency to over-specialise item selection and
predicting only very similar items to the previously seen items [19]. To
overcome these issues, several approaches suggest using hybrid models that use
both content and user-item patterns to make the best suggestions to users
[12]. Using a CF matrix and latent information from text is normally the best
way to create hybrid models. This has potential to solve the cold start issue
and also overcome the sparsity problem [3, 32]. Many older approaches such as
topic modelling [30] and bag-of-words [32] are based on this strategy but
their main shortcoming is they ignore the relation between words in the
sequences of input text.

In recent years, a variety of deep learning models have been applied
successfully in various applications that include text analysis, machine
translation, image captioning, document summarisation, scientific paper
recommendation systems and others [21]. Using RNN to extract latent
information and representing text as embedding vectors [7] has been one of the
main approach to modelling textual semantic information. [32, 3] apply deep
learning on the task of recommendation systems for scientific papers. In [2]
Trapit et al. implement an encoder using Gated Recurrent Neural Network to map
text input to a latent factor vector. The model is trained end-to-end and has
benefits such as taking into account the word order, the ability to make
multi-task predictions of articles to users and tags for papers. However, for
long sequences, RNNs, even when using a gating mechanism, identifying the
relevant information if this can be found anywhere in a document.

In this paper we use a 3-stage deep neural network proposed in [33] for text
representation. This includes an RNN, a highway layer and a CNN. This model
tries to combine the benefits of RNNs and CNNs to extract the best information
from long sequences of text. Combined with a highway layer the network is
capable of learning structural representations and associating them with
individual word representations. We combine our text mapping encoder with a CF
technique [2] and model sequential language model as an input to supervised
multi-task learning that predicts both the rate and the meta data. We also
show how this approach can be used in the case of the cold start and user-item
matrix sparsity. We evaluate our model on the CiteULike dataset. Our results
show that this approach improves the recommendation task for scientific
papers. We compare our model with the GRU [3] and the embedding model [32] on
both long and short sequences of text. We find that this model is very
promising for the RSs task with a considerable improvement of performance
compared to other models. Additionally, our analysis of how sequence length
effects the the RCNNs with highway layers shows that this model learns good
representations for long sequences of text.

##  II Background And Problem formulation

###  II-A Scientific paper recommendation

The number of academic studies is increasing every year and with the
development of technology the interest in digital resources and access to
information has increased over time. Researchers use and read exciting
articles to improve their studies and find new challenges in their scientific
area. They spend a lot of time and struggle to find the relevant articles. The
purpose of article recommendation system is to reduce this time and suggest to
them the articles that aligns with their research interest and that they may
not even be are of [5]. Due to the large knowledge on the web and the need to
filter it, recommendation systems (RS) have become vital tool [10]. Paper RSs
aim to help researchers find the relevant papers by calculating and ranking
publications and recommend a top N papers associated with a researcher’s focus
of interest[1]. There several methods for scientific RS.
Most traditional research paper RSs rely on a keyword-based search technique
[28, 9]. The results of a keyword-based searching are not always suitable and
the number of obtained items is relatively large. Researchers then have to
filter the search results to get the items needed [13]. Research paper RSs are
mainly focused on the frequency of citations. Citation databases such as
CiteSeerX111CiteSeerX https://citeseerx.ist.psu.edu/index apply citation
analysis in order to identify papers that are similar to a given paper [34,
35, 18]. Recently, making use of a matrix factorisation, a kind of CF
technique, has become as the most efficient and accurate approach [3]. The
approaches based on frequencies of citation and CF may face the cold start
problem. Content-based methods, on the other hand, recommend items based on
their characteristics as well as specific preferences of a user [16]. Pazzani
[20] studies this approach in depth, including how to build user and item
profiles. However, as we mentioned before, these models struggle with a
limitation to suggest similar papers to other users and they are limited by
sparse libraries of users. The hybrid approach, tries to combine both
collaborative and content-based recommendation. It is the most efficient
approach which includes Collaborative Topic Modelling [30], Collaborative Deep
Learning [32] and GRU [2].

###  II-B Problem formulation

We consider a general RS problem where the goal is to model the preference of
papers for users by predicting the rate for each user-item pair that the user
has not seen or even when the paper is new in the system. Then we make top N
recommendations for each user. The two main elements in each RS are users and
items: here items are scientific articles and users are researchers. We will
use I for users and J for items.
We formulate a CF matrix of the user-item rate by $R$. In this matrix, we set
the rating for user $i$ for item $j$ as $r_{ij}$ notation. As [2] we consider
$r_{ij}$ as implicit feedback that is typically defined as follows:

$r_{ij}=\begin{cases}1,&\text{if $user_{i}$ interacts with $item_{j}$ },\\\
0,&\text{otherwise}\end{cases}$ (1)

where the $ones$ refer to a positive or observed feedback, and the $zeros$
refer to the missing or unobserved values. The j-th text item is in a sequence
of $n_{j}$ word tokens, $W_{j}=(w_{1},w_{2},w_{3},...,w_{n})$, where each
token is one of the $V$ words from the vocabulary that we create from our
paper collection.
For each text item we have number of tags that we represent as follows:

$t_{jl}=\begin{cases}1,&\text{if $item_{j}$ has $tag_{l}$ },\\\
0,&\text{otherwise}\end{cases}$ (2)

Same as with the rate, we assign $t_{jl}$ $one$ if the item $j$ has a tag $l$
otherwise we assign it a $zero$.
In this task our goal is to find a list of ranked items that a user has not
seen before and make a top N recommendations to them. We implement an encoder
to map a text input to an embedding vector, learn user, item and tag embedding
vectors and make simultaneous predictions of rates and tags. We use $u_{i}$
for a user with an index $i$, $i_{j}$ for item with an index $j$ and $t_{l}$
for tag with an index $l$. In the next section we describe our model and how
to learn efficient text representations from longer sequences of text.

##  III The RHCNN Model

Figure 1: Hybrid CF task for scientific paper Recommendation system

Capturing contextual semantic information from text plays an important role in
many applications. For this reason several ways to represent structure from
text have been designed and implemented [36]. Most of the previous approaches
use the bag-of-word approach to represent text but in such case they only rely
on word embedding vectors which consider words as independent features. They
ignore interactions between words and the order of a text [3]. It is evident
that for long sequences of text an architecture with a capability of capturing
this semantic information is needed. However, using an RNN alone is not
sufficient to represent the semantics of text as features.
To overcome this issue we use RCNN-HW [33] as a encoder for mapping text to an
embedding vector. This is a 3-layer encoder that captures semantic information
from the input and represents it as a vector. This allows us to improve the
representations for long sequences of text and therefore we are able to
represent more fine-grained semantic features. We expect that this will be
able to predict better the meta-data that helps to classify papers by
capturing long-distance dependencies in texts.

Figure 1 shows how we add textual features that have been captured by an
encoder to the CF task. Our approach for the integration of context into RS is
to associate each rating $r_{ij}$ in Equation 1, for an item $j$ by a user $i$
with a context description. The context adds another dimension to the item-
user matrix of the CF. A rating $r_{ij}$ is then a tuple of (item, user,
context). The context is described as a vector of text mapped to an embedding
representation. One drawback of this approach is that it is necessary to
compute a similarity of contexts which may be difficult. In the next section
we describe how our model works.

###  III-A Long sequence text mapping encoder

Building word and text representations is a basic task in natural language
processing. A good representation method can learn a good approximation of the
grammar and the semantics of natural language. Recurrent neural networks
(RNNs) can make a use of contextual information when modelling textual
representation due to their sequential nature [22], but long-distance
dependencies are gradually lost with the gradient propagation [4]. RNNs with
“deep” transition functions remain difficult to train, even when using a
gating mechanism.

We replace the embedding representations of [2] with more sophisticated
embedding representations that are able to learn from sequential
representations of longer documents and can be used for the RS task in a
multi-task learning configuration.

Figure 2: The proposed architecture for multi-task learning. We convert each
token to a word embedding vector. This is then fed to a 3-stage network that
encodes it to a fixed-length embedding vector. The output of the network are
user-paper rates and predicted tags.

####  III-A1 The first stage

Recurrent neural networks have a potential to map a sequence of a variable
length to a fixed-length embedding vector and learn the underlying relations
from text [29]. However, it has been shown that traditional RNNs cannot
capture long sequence dependencies because of the vanishing gradient problem
[8]. This occurs when the back-propagation algorithm attempts to updates the
weights of units backwards [22]. There are two main approaches to address this
issue: Long Short-Term Memory (LSTM) [15] and Gated Recurrent Unit (GRU) [6].
Because of a lower complexity and a competitive performance [2] we use a
(bidirectional) GRU in this task. As shown in Figure 2 we feed a tokenised
sequence as an input to the model. We use a pre-training technique to convert
a sequence of words to a $k$ dimensional embedding vector and represent the
text input as $W_{j}=(e_{1},e_{2},e_{3},...,e_{nj})$. The word embeddings are
fed to a GRU which produces contextualised word embeddings. The contextualised
word embeddings are concatenated with the original embeddings which produces
the final embeddings represented as a vector of a fixed length.

####  III-A2 The second stage

Highway layers enable training deep feed-forward networks through the use of
adaptive computation. It is a simple solution to optimise very deep neural
networks. This mechanism, like LSTM, utilises a trainable gating mechanism to
adaptively transform or bypass the signal so that the network can combine
information from structural representations of varying depths [26]. As shown
in Equation 3 a highway has two non-linear gates:

$y_{t}=H(x,W_{H}).T(x,W_{T})+x.C(x,W_{c})$ (3)

  * •

Transformer (T): The $T$ gate controls how much of the activation is passed
through

  * •

Carry (C): the $C$ gate controls how much of the unmodified input is passed
through

$H$, $T$ and $C$ typically utilise a sigmoid $(\sigma)$ non-linearity
function. In practice for simplicity Equation 3 can be changed to:

$y_{t}=H(x,W_{H}).T(x,W_{T})+x.(1-T(x,W_{T}))$ (4)

shown in Figure 2 the input of this layer is a list of concatenated each word
embeddings generated by the GRU. Similar to the memory cells in the GRU
networks, highway layers allow to carry some information from the input
directly to the output. In our case the highway is used to combine word-level
features with document-level features and therefore learn semantic
representations that require connecting different levels of linguistic
structure.
One advantage of the highway architecture is that the network can learn to
dynamically adjust the routing of information based on the current input.
Also, highway networks allow us to examine how much computation depth is
needed for a given problem which can not be easily done with other networks.
In this stage we use one highway layer.

####  III-A3 The third stage

We add a two layers of CNNs to detect structural patterns of semantic
representations in the output of the previous stage and create a fixed-length
vector semantic representation of the entire text.

###  III-B Multi-task learning

####  III-B1 Article recommendation

The system should recommend to users papers based on their choice history of
papers. We use our semantic textual representation to provide information
about the content of papers. For this hybrid CF and content-based method we
need an item and a user embedding vector as well as item semantic information.
The rate prediction is now defined as:

$\hat{r}_{ij}=bias_{i}+bias_{j}+\widetilde{u}_{i}^{T}F(\widetilde{x}_{j}),$
(5) $F(\widetilde{x}_{j})=g(x)+\widetilde{v}_{j}$ (6)

According to Equation 6 our system is trained to learn a user
$(u_{i})$,$(v_{j})$ embedding vector, a text embedding vector $g(_{x})$ as
well as user and item biases. Equation 3 also shows that when an item is new
in the system and $v_{j}$ is unknown, we can still predict the rate since we
have a text embedding vector.
To learn these parameters we could use any cost function [2]. Since we do not
want to train unseen and seen data separately we use a weighted mean square:

$C_{R}(\theta)=\dfrac{1}{|R|}\sum_{(i,j\in{R})}c_{i,j}(\hat{r}_{i,j}-r_{i,j})^{2}$
(7)

####  III-B2 Meta-data prediction

Trying to find more information about a user or an item is a priority of each
of the RS tasks, in our case to successfully predict the rate and the meta-
data for items. the text embedding vector could potentially be also used for
meta-data prediction. This process is useful in cases where an item is new and
this meta-data can be used for classification. However, the main benefit of
this component is in deep training. As user-item rate matrix is highly sparse
and some items have been seen very rarely, semantic information from documents
will compensate such lack of information.
The tags can be associated with papers by the probability of observing a tag
$l$ on an item $j$:

$t_{ij}=\sigma f(\widetilde{x}_{j})^{T}t_{l}$ (8)

The embeddings for tags $t_{l}$ has to be learned. For learning the tag
embeddings we use a binary cross entropy as a cost function:

$C_{T}(\theta)=\dfrac{1}{|T|}\sum_{j}\sum_{l}t_{j,l}\log{\hat{t}_{j,l}}-(1-t_{j,l})log(1-\hat{t}_{j,l})$
(9)

###  III-C Multi task learning

Multi-Task Learning (MTL) is learning several tasks at the same time where
knowledge from one task helps another related task as some representations are
shared[23]. An important motivation of MTL is to alleviate the data sparsity
problem where each task has a limited number of labelled data. MTL can help
reuse existing knowledge and reduce the cost of manual labelling of data. MTL
models can achieve better performance than single-task counterparts [11]
because they utilise more data from different learning tasks when compared
with single-task learning. Common representations are learned in the early
layers of a network, while task-specific representations are learned in the
latter layers [24]. In our case the tasks are predicting meta-data and user-
item rate:

$C(\theta)=\lambda C_{R}(\theta)+(1-\lambda)C_{T}(\theta)$ (10)

where $\lambda$ is a hyper-parameter.

##  IV Datasets and implementation details

###  IV-A Data

####  IV-A1 CiteULike

We use two datasets of scientific articles and user information for
evaluations. CiteULike222CiteULike http://www.citeulike.org/faq/data.adp was a
website that offered a service to help researchers create their own library
and store, organise, and share the scholarly papers they have read. The rate
that our system predicts is based on user-article interaction from this
dataset. All datasets come with rich auxiliary information (e.g., title,
abstract, citations, and tags) that can be used for the scientific paper
recommendation task.
First, the CiteULike-a dataset which was collected by [30] contains
information about 5,551 users, 16,980 articles, 204,986 user-article
interaction pairs, 46,391 tags, and 44,709 cross-citations between articles.
The tags are single-word keywords that are generated by CiteULike researchers
when they add an article to their library. The data sparsity of this dataset
is considerably high. Only 0.22% of the user-article matrix has interactions.
Second, the CiteULike-t dataset which was collected by [31] has 7,947 users,
25,975 articles, 134,860 user-article interaction pairs, 52,946 tags, and
32,565 cross-citations between articles. This dataset is sparser than
CiteULike-a dataset: here only 0.07% of the user-article matrix has
interactions.

###  IV-B Evaluation methodology

The type of evaluation metrics depends on the type of the recommendation
technique. Precision and Recall are the most frequently used evaluation
methods. A $zero$ rate entry in the $R$ matrix may indicate that a user does
not like an article or that the user simply is not aware of it. Therefore,
this makes Precision not so informative measure. We use Recall instead which
tell us how successful the retrieval was. Like most RS we are also using a
ranking strategy. The ratings are predicted for items which are then sorted
and candidates are selected by returning the top M items for each target user.
Recall@M per user is defined as:

$Recall@M=\frac{\text{number of articles user liked in top M}}{\text{total
number of articles user liked}}$ (11)

The summarised result reported is the average recall over all users. We
evaluate our models on a held-out set of articles. We consider two
recommendation tasks: in-matrix prediction and out-of-matrix prediction.

####  IV-B1 In matrix prediction (warm start)

The warm-start prediction is a case where each user has not seen some articles
but at least there is one other user that has seen these article. We follow
the same procedure as in [2, 30] to create the training and testing data. For
each user we use a 5-fold cross-validation to split their history between
training and testing datasets. We keep papers with less than 5 likes always in
the training set. This guarantees that each item has been seen in the
training-set at least once. Such cases represent 9% of articles.

####  IV-B2 Out of matrix prediction (cold start)

This case is when we we add completely new papers to the system which have not
been previously seen by anyone. For the evaluation model we use a 5-fold
cross-validation of papers in the training set and make a test list for users
using the folds from the test. Note in this case all users have the same test
list and it is guaranteed that it contains none of the articles in the
training set for any user.

###  IV-C Experimental details

####  IV-C1 Text pre-processing

For each article we concatenate its title and abstract and do not remove the
stop words. We use a simple tokeniser. We replace numbers with $<$NUM$>$ and
those word that appear less than 5 times in the vocabulary with the $<$UNK$>$
token. We use the tf-idf technique to automatically produce tags for some
articles that have no tags assigned to them to make a fair comparison for all
user-articles pairs.

####  IV-C2 Training

We pre-train the GRU in our model on a large corpus of abstracts. We collected
2 million of abstracts from ACM papers333https://dl.acm.org, Aminer
444https://www.aminer.org, ArXiv 555https://arxiv.org and
CiteULike666http://www.citeulike.org [expired]. We compare our model with the
GRU [2] and Embedding [32] models. For the Embedding model we compare the
results to the results cited in the GRU paper [3]. For the GRU we re-implement
their model as we do not have access to their code and keep their hyper-
parameter settings. For all models we set the word embedding dimensions to
$K_{w}=200$. In the GRU model we follow the setting reported in the paper
which is $h_{s1}=400$ and $h_{s2}=200$ for the first and the second RNN hidden
state dimensions. For the RHCNN we set $h_{s1}=400$ for the contextualised
language model and $h_{s3}=500$ for the convolution layer. For Max-pooling to
pool the final representation pooling layer we use $p=200$ (like $K_{w}$).
Dropout is used on every layer of the network with probabilities of 0.3, 0.3,
0.2 respectively.
For both GRU and RHCNN we use the same strategy for training and testing. As
[3] we do not use different weights for negative and positive samples. As the
number of papers with no likes is much larger than papers with likes we use a
different strategy for training the model. We create mini-batches of randomly
selected users and for each user in a mini-batch we sample one positive and
one negative item pair. Instead of different weighting for items in the $R$
matrix we set weights as in [3] for the cost function in Equation 7. The
models are optimised using a stochastic gradient descent and the Adam
algorithm. We run the models in $20k$ randomly sampled data where each mini-
batch contains 512 samples. We use early-stopping based Recall on the
validation set.

##  V Results

| CiteULike |  | CiteULike
---|---|---|---
| Warm-Start | Cold-Start | Tag-Predictions | Warm-Start | Cold-Start | Tag-Predictions
Models | RHCNN | 42.34 | 54.38 | 73.34 | 49.42 | 56.25 | 77.45
GRU | 39.80 | 51.32 | 62.78 | 47.95 | 51.22 | 62.32
Embed | 36.64 | 47.71 | 60.36 | 43.02 | 38.16 | 62.29

Table I: % Recall@50 for short texts. (higher is better)

| CiteULike |  | CiteULike
---|---|---|---
| Warm-Start | Cold-Start | Tag-Predictions | Warm-Start | Cold-Start | Tag-Predictions
Models | RHCNN | 40.55 | 54.20 | 72.50 | 49.22 | 50.36 | 70.80
GRU | 37.90 | 50.24 | 58.45 | 44.68 | 42.68 | 59.43

Table II: % Recall@50 for long texts. (higher is better)

In this section we discuss the results and a comparison of the RHCNN models
with the GRU [3] and the Embedding [32] model. All of the models have been
implemented in the hybrid MTL configuration. We consider two experiments, one
containing texts of $200$ tokens (short version) and another containing texts
of $400$ tokens (long version). For the long experiment we only compare RHCNN
with GRU since they are mapping sequences of text to embedding vectors and the
Embedding model only uses word embedding vectors. In this context the order or
the number of tokens is not important. Our main goal is to have a model that
performs over longer sequences. We set $Top@M$ to $M=10,20,...,100$. Table II
and II summarise Recall@50 for all models in both long and short versions.

####  V-1 Warm-Start

Figures 3 and 4 show the results for the RHCNN, GRU, and the Embedding model
using both CiteULike-a and CiteULike-t datasets using the short version of the
text input. RHCNN works well on both datasets. However when M becomes large
fewer user ratings are available to help CF give good recommendations and the
contribution of the content becomes more important. Considering the RHCNN
performance, even in the warm-start condition the role of the contextualised
semantic representations is very important. RHCNN recommends articles $8.7\%$
on CiteULike-a and $8.5\%$ on CiteULike-t better than other models with
recall@50.
It is therefore important that encoder can extract semantically information
from texts that are longer. Longer sequences mean there may be more semantic
information but also note that it is hard for neural architecture to find
dependencies between such information. Figure 5 and 6 show results comparing
RHCNN with GRU. We see that over longer sequences our model can make better
predictions than GRU as it can better relate the semantic information in the
text.

####  V-2 Cold-Start

Cold-start recommendations are made on new articles, unseen during training.
Therefore, there is no embedding vector for an item and $v_{j}$ in Equation 6
is unknown so we only rely on the item’s content. In Figure 7 and 8 we see
that RHCNN performs considerably better than either GRU and Embedding over
short texts. On CiteULike-a RHCNN performs better by $10\%$ and on CiteULike-t
by $13\%$ than other models. In [3] Trapit et al. report the performance on
CiteULike-t higher than on CiteULike-a attributing this to more sparsity in
this dataset and because results in this dataset depend on contextual
information. In Figure 9 and 10 we see our advantage when sequences of text
are long.
Overall, we can conclude that our model is flexible over different conditions.
We see that our model performs better over both long and short texts and in
cold and warm-start conditions than other models.

Figure 3: CiteULike-a Warm-Start, short
texts$0$$20$$40$$60$$80$$100$$0$$0.1$$0.2$$0.3$$0.4$$0.5$$0.6$$0.7$recallRHCNNGRUEmbedding
Figure 4: CiteULike-t Warm-Start, short
texts$0$$20$$40$$60$$80$$100$$0$$0.1$$0.2$$0.3$$0.4$$0.5$$0.6$$0.7$recallRHCNNGRUEmbedding
Figure 5: CiteULike-a Warm-Start, long
texts$0$$20$$40$$60$$80$$100$$0$$0.1$$0.2$$0.3$$0.4$$0.5$$0.6$$0.7$recallRHCNNGRU
Figure 6: CiteULike-t Warm-Start, long
texts$0$$20$$40$$60$$80$$100$$0$$0.1$$0.2$$0.3$$0.4$$0.5$$0.6$$0.7$recallRHCNNGRU
Figure 7: CiteULike-a Cold-Start, short
texts$0$$20$$40$$60$$80$$100$$0$$0.1$$0.2$$0.3$$0.4$$0.5$$0.6$$0.7$recallRHCNNGRUEmbedding
Figure 8: CiteULike-t Cold-Start, short
texts$0$$20$$40$$60$$80$$100$$0$$0.1$$0.2$$0.3$$0.4$$0.5$$0.6$$0.7$recallRHCNNGRUEmbedding
Figure 9: CiteULike-a Cold-Start, long
texts$0$$20$$40$$60$$80$$100$$0$$0.1$$0.2$$0.3$$0.4$$0.5$$0.6$$0.7$recallRHCNNGRU
Figure 10: CiteULike-t Cold-Start, long
texts$0$$20$$40$$60$$80$$100$$0$$0.1$$0.2$$0.3$$0.4$$0.5$$0.6$$0.7$recallRHCNNGRU

####  V-3 Tag Predictions

Predicting the meta-data such tags for articles is expected to help the RS in
predicting user-item rates. This is because tags encode topics that might be
taken into account when determining the preference. We report recall@50 for
tag predictions on articles in Table II and II. In the warm-start scenario we
train articles on their tags. Hence, in this task tags prediction is
meaningless. Therefore, we learn tag prediction only in the cold-start
scenario. The results indicate that tag prediction is much better in the RHCNN
model than other models and we attribute this to the benefits of
contextualised semantic information learned with Highway Networks. On
CiteULike-a with recall@50 and with $400$ tokens we get a $\%14$ increase in
performance and on CiteULike-t $\%10.45$ compared with the GRU model.

##  VI Conclusions and future work

We proposed a novel method for recommendation systems of scientific papers but
which could also be extended to other recommendation tasks. We demonstrate
that analysing the paper’s content and extracting unseen semantic information
from text is one of the strengths in creating a hybrid recommendation system
that relies both on collaborative filtering and item features. We employed a
neural network that uses a highway mechanism to relate semantic information
that can be presented as different long or short dependencies over larger
sequences of text. Our results indicate that the proposed system can deal with
the difficulty of a cold start and user-item matrix sparsity. For the next
step, we plan to experiment with different sampling methods to reduce the
amount of data needed to train the network without effecting its performance
as well as compare our semantic representations that were tailor-trained on
the scientific texts with the general semantic representations obtained
through BERT.

## References

  * [1] Xiaomei Bai, Mengyang Wang, Ivan Lee, Zhuo Yang, Xiangjie Kong, and Feng Xia.  Scientific paper recommendation: A survey.  Ieee Access, 7:9324–9339, 2019.
  * [2] Trapit Bansal, David Belanger, and Andrew McCallum.  Ask the gru: Multi-task learning for deep text recommendations.  RecSys ’16, page 107–114, New York, NY, USA, 2016. Association for Computing Machinery.
  * [3] Toine Bogers and Antal van den Bosch.  Recommending scientific articles using citeulike.  RecSys ’08, New York, NY, USA, 2008. Association for Computing Machinery.
  * [4] Nicolas Boulanger-Lewandowski, Yoshua Bengio, and Pascal Vincent.  Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription.  arXiv preprint arXiv:1206.6392, 2012.
  * [5] Betül Bulut, Buket Kaya, Reda Alhajj, and Mehmet Kaya.  A paper recommendation system based on user’s research interests.  In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 911–915. IEEE, 2018.
  * [6] Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio.  On the properties of neural machine translation: Encoder-decoder approaches, 2014.
  * [7] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  arXiv preprint arXiv:1406.1078, 2014.
  * [8] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  arXiv preprint arXiv:1412.3555, 2014.
  * [9] Tao Dai, Tianyu Gao, Li Zhu, Xiaoyan Cai, and Shirui Pan.  Low-rank and sparse matrix factorization for scientific paper recommendation in heterogeneous network.  IEEE Access, 6:59015–59030, 2018.
  * [10] Mahak Dhanda and Vijay Verma.  Recommender system for academic literature with incremental dataset.  Procedia Computer Science, 89:483–491, 2016.
  * [11] Theodoros Evgeniou and Massimiliano Pontil.  Regularized multi–task learning.  In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 109–117, 2004.
  * [12] Mustansar Ali Ghazanfar and Adam Prugel-Bennett.  A scalable, accurate hybrid recommender system.  In 2010 Third International Conference on Knowledge Discovery and Data Mining, pages 94–98. IEEE, 2010.
  * [13] Hebatallah A Mohamed Hassan.  Personalized research paper recommendation using deep learning.  In Proceedings of the 25th conference on user modeling, adaptation and personalization, pages 327–330, 2017.
  * [14] Jon Herlocker, Joseph A Konstan, and John Riedl.  An empirical analysis of design choices in neighborhood-based collaborative filtering algorithms.  Information retrieval, 5(4):287–310, 2002.
  * [15] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [16] Umair Javed, Kamran Shaukat, Ibrahim A Hameed, Farhat Iqbal, Talha Mahboob Alam, and Suhuai Luo.  A review of content-based and context-based recommendation systems.  International Journal of Emerging Technologies in Learning (iJET), 16(3):274–306, 2021.
  * [17] Yehuda Koren and Robert Bell.  Advances in collaborative filtering.  Recommender systems handbook, pages 77–118, 2015.
  * [18] Haifeng Liu, Zhuo Yang, Ivan Lee, Zhenzhen Xu, Shuo Yu, and Feng Xia.  Car: Incorporating filtered citation relations for scientific article recommendation.  In 2015 IEEE International Conference on Smart City/SocialCom/SustainCom (SmartCity), pages 513–518. IEEE, 2015.
  * [19] Jie Lu, Dianshuang Wu, Mingsong Mao, Wei Wang, and Guangquan Zhang.  Recommender system application developments: A survey.  Decision Support Systems, 74:12–32, 2015.
  * [20] Michael J Pazzani and Daniel Billsus.  Content-based recommendation systems.  In The adaptive web, pages 325–341. Springer, 2007.
  * [21] Samira Pouyanfar, Saad Sadiq, Yilin Yan, Haiman Tian, Yudong Tao, Maria Presa Reyes, Mei-Ling Shyu, Shu-Ching Chen, and Sundaraja S Iyengar.  A survey on deep learning: Algorithms, techniques, and applications.  ACM Computing Surveys (CSUR), 51(5):1–36, 2018.
  * [22] AJ Robinson and Frank Fallside.  The utility driven dynamic error propagation network.  University of Cambridge Department of Engineering Cambridge, 1987.
  * [23] Sebastian Ruder.  An overview of multi-task learning in deep neural networks.  arXiv preprint arXiv:1706.05098, 2017.
  * [24] Ozan Sener and Vladlen Koltun.  Multi-task learning as multi-objective optimization.  arXiv preprint arXiv:1810.04650, 2018.
  * [25] Lalita Sharma and Anju Gera.  A survey of recommendation system: Research challenges.  International Journal of Engineering Trends and Technology (IJETT), 4(5):1989–1992, 2013.
  * [26] Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber.  Highway networks.  arXiv preprint arXiv:1505.00387, 2015.
  * [27] Xiaoyuan Su and Taghi M Khoshgoftaar.  A survey of collaborative filtering techniques.  Advances in artificial intelligence, 2009, 2009.
  * [28] Jianshan Sun, Jian Ma, Zhiying Liu, and Yajun Miao.  Leveraging content and connections for scientific article recommendation in social computing contexts.  The Computer Journal, 57(9):1331–1342, 2014.
  * [29] Duyu Tang, Bing Qin, and Ting Liu.  Document modeling with gated recurrent neural network for sentiment classification.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1422–1432, Lisbon, Portugal, September 2015\. Association for Computational Linguistics.
  * [30] Chong Wang and David M Blei.  Collaborative topic modeling for recommending scientific articles.  In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 448–456, 2011.
  * [31] Hao Wang, Binyi Chen, and Wu-Jun Li.  Collaborative topic regression with social regularization for tag recommendation.  In Twenty-Third International Joint Conference on Artificial Intelligence, 2013.
  * [32] Hao Wang, Naiyan Wang, and Dit-Yan Yeung.  Collaborative deep learning for recommender systems.  In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 1235–1244, 2015.
  * [33] Ying Wen, Weinan Zhang, Rui Luo, and Jun Wang.  Learning text representation using recurrent convolutional neural network with highway layers.  arXiv preprint arXiv:1606.06905, 2016.
  * [34] Jevin D West, Ian Wesley-Smith, and Carl T Bergstrom.  A recommendation system based on hierarchical clustering of an article-level citation network.  IEEE Transactions on Big Data, 2(2):113–123, 2016.
  * [35] Fattane Zarrinkalam and Mohsen Kahani.  Semcir: A citation recommendation system based on a novel semantic distance measure.  Program, 2013.
  * [36] Lei Zhang, Shuai Wang, and Bing Liu.  Deep learning for sentiment analysis: A survey.  Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4):e1253, 2018.