# Cross-Attribute Matrix Factorization Model with Shared User Embedding

Wen Liang
University of California, San Diego
La Jolla, CA 92093
<EMAIL_ADDRESS>
&Zeng Fan
University of California, San Diego
La Jolla, CA 92093
<EMAIL_ADDRESS>
&Youzhi Liang
Department of Computer Science
Stanford University
Stanford, CA 94305
<EMAIL_ADDRESS>
&Jianguo Jia
Department of Computing
Hong Kong Polytechnic University
Hong Kong, China
<EMAIL_ADDRESS>
###### Abstract

Over the past few years, deep learning has firmly established its prowess
across various domains, including computer vision, speech recognition, and
natural language processing. Motivated by its outstanding success, researchers
have been directing their efforts towards applying deep learning techniques to
recommender systems. Neural collaborative filtering (NCF) and Neural Matrix
Factorization (NeuMF) refreshes the traditional inner product in matrix
factorization with a neural architecture capable of learning complex and data-
driven functions. While these models effectively capture user-item
interactions, they overlook the specific attributes of both users and items.
This can lead to robustness issues, especially for items and users that belong
to the "long tail". Such challenges are commonly recognized in recommender
systems as a part of the cold-start problem. A direct and intuitive approach
to address this issue is by leveraging the features and attributes of the
items and users themselves. In this paper, we introduce a refined NeuMF model
that considers not only the interaction between users and items, but also
acrossing associated attributes. Moreover, our proposed architecture features
a shared user embedding, seamlessly integrating with user embeddings to
imporve the robustness and effectively address the cold-start problem.
Rigorous experiments on both the Movielens and Pinterest datasets demonstrate
the superiority of our Cross-Attribute Matrix Factoriztion model, particularly
in scenarios characterized by higher dataset sparsity.

##  1 Introduction

Recommender systems have become an indispensable component of many online
platforms, guiding users through a plethora of content to find what best
matches their preferences and needs. Traditional methods, such as
collaborative filtering [1] and matrix factorization [2], have been successful
in capturing the latent preferences of users based on historical interactions
[3]. The rise of deep learning has revolutionized numerous domains, including
computer vision where it has enabled breakthroughs in image recognition [4,
5], natural language processing [6, 7], and speech recognition [8].
Variational techniques and hybrid architectures of deep learning with data
augmentations [9] elevate the performance of modeling to a higher standard
[10, 11]. Leveraging these advancements, models such as NeuMF and NCF have
incorporated neural networks in the realm of recommender systems, advancing
beyond the linearity of matrix factorization to allow for more intricate
representations of user-item interactions [12]. However, a gap remains in
considering the intrinsic attributes of users and items, which can be pivotal
in addressing challenges such as the cold-start problem and ensuring
robustness, especially for long-tail items and users.

While the NeuMF and NCF model mark significant advancements in the field, they
primarily focus on the interactions derived from user-item matrices without
tapping into the wealth of information contained within intrinsic attributes
of users and items. By overlooking these attributes, the models potentially
miss out on crucial signals that could enhance recommendation quality,
especially in scenarios where interaction data is sparse or nonexistent, such
as in the cold-start problem. To address this, the Attribute-aware Deep CF
model incorporates item and user features into the matrix factorization (MF)
model [13]. It transforms these features into learnable embeddings,
subsequently integrating them into the multi-layer perceptron (MLP). However,
we believe there is potential to further enhance this approach. Specifically,
there’s an opportunity to exploit more features by cross-referencing user
attributes with item attributes and vice versa.

Our Cross-Attribute Matrix Factorization Model with Shared User Embedding
offers several distinctive contributions to the realm of recommender systems:

  1. 1.

Incorporation of a Shared User Embedding: One of the major contributions of
our model is the introduction of a shared user embedding. In situations where
we encounter cold-start users, this shared embedding facilitates a basic
recommendation process. It mitigate the robustness problems posed by relying
on either randomly-initialized embeddings or immaturely learned user
embeddings, which often aren’t reliable for providing meaningful
recommendations.

  2. 2.

Enhancement of the Existing MF Model with Cross-Attribute Interactions:
Building on the foundation of the current matrix factorization (MF) model,
we’ve added cross-attribute matrix factorization capabilities. To elaborate,
each user embedding interacts with every item attribute embedding, and vice
versa; every item interacts with user attribute embeddings. This intricate
design ensures that we harness the full potential of existing data and
features, fostering richer and more insightful recommendation outcomes.

Our model are further validated through a series of experiments. We used two
benchmark datasets - Movielens and Pinterest [13] \- consistently underscore
the enhanced performance of our model. Notably, CAMF shines with pronounced
superiority in environments marked by heightened dataset sparsity, thereby
showcasing its practical applicability in real-world settings where user-item
interactions may be limited.

##  2 Related Works

###  2.1 Matrix Factorization

Figure 1: Matrix Factorization

MF convert each user and item to a real-valued vector of latent variables. The
MF estimates the interaction between user and item by using the inner product
of two vectors:

$r(u,i)=\gamma_{u}\cdot\gamma_{i},$ (1)

where the $\gamma_{u}$ and $\gamma_{i}$ denotes the latent factor for user $u$
and item $i$, the $r(u,i)$ denotes the interaction between this user and item.
This method is very popular by combining good scalability with predictive
accuracy. In addition, they offer much flexibility for modeling various real-
life situations. However, the matrix factorization has several limitations
that it is prone to recommend popular items and is vulnerable to sparse
dataset.

###  2.2 NCF, GMF and NeuMF Model

Neural Collaborative Filtering (NCF) [12] serves as a foundational framework
to model the interaction between users and items using neural networks. The
general frame work of NCF is shown in Figure 2. Within the NCF paradigm, two
primary models emerge: GMF (Generalized Matrix Factorization) and MLP (Multi-
Layer Perceptron). GMF can be seen as a neural analog to traditional matrix
factorization techniques, capturing linear patterns in user-item interactions.
On the other hand, MLP is designed to discover intricate non-linear patterns.
The NeuMF model blends the linear capabilities of GMF with the rich
expressiveness of MLP to further improve recommendation performance. We mainly
focus on GMF model and NeuMF model in our work. The mapping function of the
GMF layer is defined as following:

$\mathbf{\phi}(p_{u},q_{i})=p_{u}\odot q_{i}$ (2)

where $\odot$ denotes the element-wise product of vectors. Then this vector
can be projected to the output layer:

$\hat{y}_{ui}=\mathbf{a_{out}}(h^{T}(p_{u}\odot q_{i})).$ (3)

In the work presented by He et al. [12], the GMF (Generalized Matrix
Factorization) layer, characterized by its activation function
$a_{\text{out}}$ and edge weights $h$, offers a pathway to revert back to the
traditional matrix factorization model. As depicted in Figure 3, the NeuMF
(Neural Matrix Factorization) model is an integration of both the GMF and the
MLP (Multi-Layer Perceptron) models. Within this architecture, the GMF and MLP
layers independently process the input embeddings. Subsequently, their outputs
are concatenated, forming the basis for the NeuMF layer.

Figure 2: Neural collaborative filtering framework Figure 3: Neural Matrix
Factorization Model

###  2.3 Attributed-Aware Deep Collaborative Filtering Model

The NCF framework takes into account the user-item interaction information,
but it neglects the intrinsic attribute information of users/items. Wang et
al. [14] proposed an Attribute-aware deep CF model, as depicted in Figure 4,
which incorporates the attributes of users/items. They introduced a pairwise
pooling layer following the embedding layer to effectively capture the
relationship between users/items and their corresponding attributes. The
operation for pairwise pooling is defined as:

$p_{u}=\mathbf{\varphi_{pairwise}}(u,{g^{t}_{u}})=\sum\limits_{t=1}^{V_{u}}u\odot
g^{t}_{u}+\sum\limits_{t=1}^{V_{u}}\sum\limits_{t\prime=t+1}^{V_{u}}g^{t}_{u}\odot
g^{t}\prime_{u}$ (4)
$q_{i}=\mathbf{\varphi_{pairwise}}(i,{g^{t}_{i}})=\sum\limits_{t=1}^{V_{i}}i\odot
g^{t}_{i}+\sum\limits_{t=1}^{V_{i}}\sum\limits_{t\prime=t+1}^{V_{i}}g^{t}_{i}\odot
g^{t}\prime_{i}$ (5)

After calculate $p_{u}$ and $q_{i}$, the model takes the element-wise product
of $p_{u}$ and $q_{i}$ into MLP layers for the final prediction.

Figure 4: Attribute-aware Neural CF Model

##  3 Method

###  3.1 Shared User Embedding

Our experiments with GMF, NeuMF, and Attribute-aware Deep CF models showed
that the GMF model trained faster and had nearly the same performance as the
more complex NeuMF. The detailed results are discussed later. Due to its
efficiency and effectiveness, we chose to focus on the GMF model.

Our goal was to improve recommendations in scenarios with sparse data and for
new users or items. To address the cold-start and sparsity challenges, we use
a shared user embedding embedding might encapsulate overall user traits. As an
illustrative case, encountering a novel user or item could harness this shared
embedding, furnishing more generic and resilient recommendation outcomes. To
combine this shared user embedding with a specified user embedding, we
introduce a weight $\alpha$ ranging from 0 to 1 to adjust the balance between
them by controlling the activation of the specified user embedding:

$u_{merged}=\alpha u_{shared}+(1-\alpha)u_{embedded}$ (6)
$\mathbf{\phi}(u_{merged},i)=u_{merged}\odot i$ (7)

In alignment with the GMF paradigm delineated earlier, the weighted user
vector undergoes an element-wise multiplication with the item vector. The
balancing weight, $\alpha$, is determined by a singular hidden layer, taking
another ensemble of item and user attributes as inputs:

$\mathbf{z}=\begin{bmatrix}u\\\ u_{\text{attr}}\\\ i\\\
i_{\text{attr}}\end{bmatrix}$ (8)

$\alpha=\text{Sigmoid}(\mathbf{w}^{T}\mathbf{z}+\mathbf{b})$ (9)

Herein, $\mathbf{z}$ symbolizes the layer input, which incorporates attributes
like item popularity and user historical interaction frequency. For instance,
if attributes indicate an new user, the model might assign greater emphasis on
the shared user vector, recommending universally acclaimed movies. Conversely,
for universally appealing movies, the model could diminish the weight on
specifed user embedding, offering recommendations less tailored to individual
tastes.

###  3.2 Cross-Attribute Matrix Factorization Model

Another pivotal enhancement in our approach lies in the explicit incorporation
of attribute information. Specifically, we multiplied the user vectors with
item attributes and item vectors with user attributes. To accommodate this, we
introduced dedicated embeddings tailored for attributes. This setup is not
just a mere inclusion of attributes; it fundamentally changes the
representation learning. By allowing these interactions, we aim to learn the
underlying patterns and relationships between users, items, and their inherent
attributes.

After facilitating these interactions, the resultant vectors are concatenated.
This consolidated representation captures both the standalone and cross-
attribute information. To process this rich representation further, we direct
it through several neural network layers. These layers are designed to
abstract higher-level patterns from the combined input.

Our comprehensive model, which encompasses the previously discussed Shared
User Embedding and the current Cross-Attribute interactions, is illustrated in
Figure 5.

Figure 5: Proposed Attribute-aware Model

##  4 Experiments

In our evaluations, we leverage two prominent datasets: MovieLens and
Pinterest. Both have been previously used in [12, 13] due to their rich
attributes and feasibility for preprocessing.

###  4.1 MovieLens Dataset

For MovieLens, we employ the version containing 1 million ratings, ensuring
that each user has rated at least 20 movies. Explicit ratings are transformed:
a rating from a user denotes a label of 1 for the movie. To supplement this,
we randomly sample 99 unobserved entries, tagging them as negative samples
with a label of 0.

The dataset also provides user metadata encompassing gender, age, and
occupation. This metadata is used as the user’s attributes, denoted as
$g^{t}_{u}$, both in the Attribute-aware Deep CF model and our proposed
architecture.

Movie data primarily comprise genres. Recognizing that movies can span
multiple genres, each genre is treated as a distinct attribute. These are used
as $g^{t}_{i}$ during our experiments.

###  4.2 Pinterest Dataset

The Pinterest dataset, characterized by its enormity and sparsity, presents a
unique challenge. With over 20% of the users having a singular pin, assessing
algorithmic efficacy becomes intricate. Thus, we refined the dataset to
include only users with at least 10 pins, which are labeled as 1. Like the
MovieLens dataset, 99 unobserved entries are sampled as negatives labeled as
0.

User data in this dataset provide insights into pins and page categories. We
group every 40 pins together, such that a user with, say, 35 pins falls into
the first group and a user with 41 pins falls into the second. The myriad of
page categories, initially totaling 468, are consolidated into 45 main
categories, given the observation that several categories possess scant
samples. This structured data is then taken as the user’s attribute input.

Items, represented by images, lack explicit attributes. Notably, an image may
be pinned by diverse users. To derive an item’s ’pin attribute’, we employ the
formula:

$Pins_{i}=\sum\limits_{(u\prime,i)\subset D}^{V_{u}}Pins_{u\prime i}$ (10)

In the equation, $V_{u}$ signifies the user set in the dataset, and $D$
encapsulates the user-item pairs. $Pins_{u}$ stands for the user’s pin count.
This weighted summation offers insights into an item’s popularity beyond a
mere pin count. Analogous to user pins, items are grouped every 50, and this
processed data serves as the item’s attribute input.

###  4.3 Evaluation and Metrics

We use the leave-one-out evaluation in our experiment. For each user, we
randomly choose one of user’s interaction and the 99 negative samples
mentioned in Data Preprocessing section as test set. We rank the test item
among the 100 items. The performance of the ranked list is evaluated by first
10 Hit Ratio (HR@10) and Normalized Discounted Cumulative Gain (NDCG@10). In
other word, HR measures whether the positive test item is within the top-10
ranked list, and the NDCG will assign higher scores to hits at top ranks.
These 2 metrics are calculated for each user and the average score is
reported.

###  4.4 Parameters Setting

We don’t use the pretrained model from the paper. For each model, we
initialize model parameters with a Gaussian distribution (mean = 0, standard
deviation = 0.01). The models are trained with Adam optimizer with the batch
size of 256. For stacked MLP layers, we use [32,16,8] as layer dimensions.
During training process, we will randomly generate 4 negatives instance for
each positive instance. He et al. term the last hidden layer of NCF as
predictive factors. We evaluate the factors of [8,16,32].

###  4.5 Results

The following Figure 6, Figure 7, Figure 8 and Figure 9 show the performance
of HR@10 and NDCG@10 with respect to the number of matrix factorization
factors.

Figure 6: Performance of HR@10 on MovieLens Figure 7: Performance of NDCG@10
on MovieLens Figure 8: Performance of HR@10 on Pinterest Figure 9: Performance
of NDCG@10 on Pinterest

We optimize all the models with log loss. The Figure 10 shows the training
loss vs. epochs on MovieLens dataset for MLP, GMF, NeuMF, Attribute-aware Deep
CF, and our proposed model.

Figure 10: Training loss of all the models

The following tabel1, tabel2, tabel3 and tabel 4 show the performance of HR@10
and NDCG@10 with respect to the number of matrix factorization factors.

Table 1: Movielens HR@10 Results #Factors | NeuCF | GMF | AA Deep CF | Our Model
---|---|---|---|---
8 | 0.8121 | 0.7485 | 0.7773 | 0.7923
16 | 0.8175 | 0.7774 | 0.8197 | 0.8213
32 | 0.8275 | 0.7935 | 0.8303 | 0.8304
Table 2: Movielens NDCG@10 Results #Factors | NeuCF | GMF | AA Deep CF | Our Model
---|---|---|---|---
8 | 0.5286 | 0.5043 | 0.5176 | 0.5308
16 | 0.567 | 0.5242 | 0.5631 | 0.5668
32 | 0.5712 | 0.5398 | 0.5788 | 0.5784
Table 3: Pinterest HR@10 Results #Factors | NeuCF | GMF | AA Deep CF | Our Model
---|---|---|---|---
8 | 0.8073 | 0.8336 | 0.7773 | 0.8311
16 | 0.799 | 0.7967 | 0.8274 | 0.8323
32 | 0.8001 | 0.7907 | 0.8241 | 0.8610
Table 4: Pinterest NDCG@10 Results #Factors | NeuCF | GMF | AA Deep CF | Our Model
---|---|---|---|---
8 | 0.5536 | 0.5574 | 0.5176 | 0.5507
16 | 0.55 | 0.5425 | 0.5694 | 0.5665
32 | 0.5559 | 0.5465 | 0.5703 | 0.5920

###  4.6 Discussion

The experimental results strongly underscore the robustness of our proposed
model in comparison to existing models. Considering both the MovieLens and
Pinterest datasets, our model consistently outperforms or is highly
competitive with other methods across the board.

From the tables 1, 2, 3, and 4, it is evident that our model exhibits superior
performance, especially with an increase in the number of matrix factorization
factors. This superiority in HR@10 metrics is particularly significant given
that our model is directly designed to tap into the rich information available
in user-item interactions, without compromising on the intrinsic attributes.

The Attribute-Aware Deep CF (AA Deep CF) model’s underwhelming performance on
these recommendation tasks came as a surprise. One reason for its subpar
results might be the pooling layer they adopted. By leveraging user, item, and
attribute information, the model may inadvertently discard pivotal details,
leading to a dilution of recommendation quality. Additionally, it’s noteworthy
to mention that the AA Deep CF model was primarily fashioned for social
networks and travel recommendation tasks. While it might excel in scenarios
with abundant user and item attributes, it evidently struggles when confronted
with the data constraints of our datasets. The confluence of the AA Deep CF
model with another model optimized for social network relations hints at
design decisions which may not necessarily align with our task, and thus, the
observed performance disparities.

In summary, our results illuminate the importance of crafting recommendation
models tailored to the nuances of the dataset in question. Our model, by
astutely leveraging both interaction data and inherent attributes, solidifies
its place as a formidable contender in the recommendation system arena.

##  5 Conclusion

This research highlights two pivotal advancements in recommendation systems.
First, we introduced the concept of a Shared User Embedding, which offers a
robust solution to the cold-start problem by reducing dependence on unstable
embeddings. Concurrently, our refined matrix factorization approach, utilizing
Cross-Attribute Interactions, ensures an in-depth comprehension of user-item
dynamics, leveraging every available attribute. Collectively, these
innovations represent a substantial progression in providing nuanced and
dependable recommendations, indicating an optimistic trajectory for subsequent
research in this field.

## References

  * [1] Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl.  Item-based collaborative filtering recommendation algorithms.  In Proceedings of the 10th international conference on World Wide Web, pages 285–295, 2001.
  * [2] Yehuda Koren, Robert Bell, and Chris Volinsky.  Matrix factorization techniques for recommender systems.  Computer, 42(8):30–37, 2009.
  * [3] Paul Resnick and Hal R Varian.  Recommender systems.  Communications of the ACM, 40(3):56–58, 1997.
  * [4] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition, 2015.
  * [5] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  Imagenet classification with deep convolutional neural networks.  Commun. ACM, 60(6):84–90, may 2017.
  * [6] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need, 2023.
  * [7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * [8] Dario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu.  Deep speech 2: End-to-end speech recognition in english and mandarin, 2015\.
  * [9] Wen Liang, Youzhi Liang, and Jianguo Jia.  Miamix: Enhancing image classification through a multi-stage augmented mixied sample data augmentation method.  arXiv preprint arXiv:2308.02804, 2023.
  * [10] Youzhi Liang and Wen Liang.  Reswcae: Biometric pattern image denoising using residual wavelet-conditioned autoencoder.  arXiv preprint arXiv:2307.12255, 2023.
  * [11] Youzhi Liang, Wen Liang, and Jianguo Jia.  Structural vibration signal denoising using stacking ensemble of hybrid cnn-rnn.  arXiv e-prints, pages arXiv–2303, 2023.
  * [12] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua.  Neural collaborative filtering.  pages 173–182, 2017.
  * [13] Xue Geng, Hanwang Zhang, Jingwen Bian, and Tat-Seng Chua.  Learning image and user features for recommendation in social networks.  In 2015 IEEE International Conference on Computer Vision (ICCV), pages 4274–4282, 2015.
  * [14] Xiang Wang, Xiangnan He, Liqiang Nie, and Tat-Seng Chua.  Item silk road: Recommending items from information domains to social users.  CoRR, abs/1706.03205, 2017.


# A self-adapting super-resolution structures framework for automatic design
of GAN

Yibo Guo, Haidi Wang, Yiming Fan, shunyao Li, and Mingliang Xu Y. Guo is a
teacher at the School of Information Engineering, Zhengzhou UniversityH. Wang,
Y. Fan, S. Li and M. Xu are with Zhengzhou University.

###### Abstract

With the development of deep learning, the single super-resolution image
reconstruction network models are becoming more and more complex. Small
changes in hyperparameters of the models have a greater impact on model
performance. In the existing works, experts have gradually explored a set of
optimal model parameters based on empirical values or performing brute-force
search. In this paper, we introduce a new super-resolution image
reconstruction generative adversarial network framework, and a Bayesian
optimization method used to optimizing the hyperparameters of the generator
and discriminator. The generator is made by self-calibrated convolution, and
discriminator is made by convolution lays. We have defined the hyperparameters
such as the number of network layers and the number of neurons. Our method
adopts Bayesian optimization as a optimization policy of GAN in our model. Not
only can find the optimal hyperparameter solution automatically, but also can
construct a super-resolution image reconstruction network, reducing the manual
workload. Experiments show that Bayesian optimization can search the optimal
solution earlier than the other two optimization algorithms.

###### Index Terms:

Single image super-resolution, Generative adversarial network, Self-calibrated
convolutions, Bayesian optimization.

##  I Introduction

Super-resolution images have rich detailed information, which can effectively
assist in completing tasks such as medical image analysis[1] and object
recognition[2]. Super-resolution image reconstruction aims to use the neural
network to learn the mapping relationship between low-resolution images and
high-resolution images. With the development of deep learning, the network
structure has become more and more complex, and the model parameters have
gradually increased. Model parameters include the network random
initialization parameters and hyperparameters that can be manually adjusted.
The purpose of neural network training is to obtain a set of parameters to
make the model have better generalization capabilities. Small changes in
hyperparameters may have a greater impact on the generalization of the model.
A model has only one set of parameters to make the model optimal. In order to
obtain the optimal performance of the model, it is necessary to gradually
research the network model hyperparameters when the network model framework
has been fixed. In traditional method, Deep learning model usually adopts
manual tuning of model hyperparameters, and gradually explores the optimal
solution with reference to experience values. This process takes a long time
and easily makes people impatience. An automated method to adjust parameters
is easier to be accepted by researchers, and an automated exploration of model
parameters can advance deep learning research. Therefore, we try to build a
generative adversarial network model with optimization algorithms and reduce
the manual workload as much as possible.

Grid search is a traditional method of neural network hyperparameter search,
which can realize neural network auto-tuning. In the process of parameter
search, grid search will list all combinations of parameters. For deep
learning networks, the computation is relatively large, and it requires more
processor resources. Some scholars have proved that the random search
algorithm [3] is superior to the grid search algorithm, but the random search
algorithm has strong randomness which is not conducive to the optimization of
neural network.

Bayesian optimization algorithm [4] is a black-box optimizers (that assume no
internal analytical structures of the objective function), which supports the
above-mentioned deep learning network training. In addition, the Bayesian
optimization algorithm is to predict the next evaluation point based on the
historical results, which can solve the problem of large computation and
resources for grid search and random search. In this paper, the Bayesian
optimization method is used for deep learning optimization to achieve the
construction of generative adversarial networks.

In our work, we propose generative adversarial network model with the Bayesian
optimization algorithm. It can search for a set of optimal model
hyperparameters and reduce manual workload. Our second contribution is using
the constructed generative adversarial network model for super-resolution
image reconstruction tasks. We compare the reconstructed image results of the
three optimization algorithms under different public datasets, and the results
of Bayesian optimization are the best.

The rest of our work is organized as follows: Section 2 introduces the
research of super-resolution reconstruction methods and several different
algorithms of hyperparameter search. We have a brief discussion of the
advantages of GAN method and Bayesian optimizations. Section 3 builds a super-
resolution image framework based on the generative adversarial networks and
determines GANs hyperparameters through Bayesian optimization algorithm. In
section 4, we compare the speed of three optimization algorithms and the
quality of reconstructed images used diffenert algorithms, which illustrates
the effectiveness of proposed methods. In section 5, there are the conclusion
of this work and the future perspectives.

##  II Related Work

This chapter mainly introduces the current research status in two aspects, one
is the current research status of super-resolution image reconstruction based
on generative adversarial networks, and the other is the current research
status of optimization algorithm.

1) Super-resolution image reconstruction method based on generative
adversarial networks

The Generative Adversarial Network (GAN) [5] is consisted of two networks: a
generator and a discriminator. The two networks were trained through a max-min
game and could generate more realistic virtual samples. Generative adversarial
networks were more sensitive to image colors and brightness. The use of
generative adversarial networks for super-resolution reconstruction had better
visual effects, and some progress had been made. The idea of image inpainting
[6] was referenced from image outpainting [7]. Extending the center of the
image to both sides could generate a high-resolution image, but the generated
extended area was blurred, and the boundary with the center of the image was
clearly visible. Pix2pixHD [8] used multiple stages to gradually obtain super-
resolution images, and the discriminator was also modified to multi-scale
discriminating the input image. CRRAGAN [9] proposed a cascaded residual-
residual attention extraction module, which used feature fusion for
information forward propagation, and achieved good results.Most of the above
methods used multi-scale or cascaded residual methods to extract image
features. The use of extracted features through feature fusion had become a
common way to build network structures and was widely used. Chudasama _et al._
[10, 11] proposed two different effective calculation methods for single image
super-resolution, which were reduced the number of parameters on the basis of
achieving better results. Reducing the model parameters could reduce the
tuning time of model hyperparameter to a certain extent, but some
hyperparameters were essential. The method in this paper adopted another
method to save the tuning time, which was automatically searched for a set of
model hyperparameter combinations through optimization algorithms.The
following introduces some universal model hyperparameter search algorithms.

2) Optimization algorithm

In the deep learning model of super-resolution reconstruction, small changes
in hyper-parameters might have a greater impact on the performance of the
model. An automated hyperparameter tuning algorithm is beneficial to the
automatic construction of the model. Bergstra _et al._ [12] explored several
machine learning hyperparameter optimization algorithms, including random
search and two greedy algorithms. At the same time, they proved that random
search [3] was better than grid search algorithm. In addition to the above
optimization methods, the most famous algorithm is the Nelder-Mead(NM) [13],
which was an unconstrained minimization method and was widely used [14, 15].
The NM algorithm did not need to consider the gradient information of the
function, which might cause a dimensional disaster problem. It is a black box
optimization algorithm. COBYLA [16, 17] was inspired by the NM algorithm, and
it was an improved nonlinear constrained optimization method. The objective
function was approximated by linear interpolation and confidence interval
constraint variables at the vertices of the simplex. After calculation, a new
variable vector was obtained. Variables that met certain conditions would
replace the current variables and got the current optimal vector. However, the
linear approximation of this algorithm was inefficient, and variables could
not be too many. Different from the above optimization algorithms, particle
swarm optimization(PSO) [18] was a kind of evolutionary algorithm, which was
more sensitive to dimensionality. PSO proposed a particle swarm optimization
method for optimizing nonlinear functions and neural network training. If the
dimensionality was too high, it would seriously increase the amount of
function evaluations. Gong _et al._ [19] used gene expression programming to
optimize hyperparameters. The evolutionary algorithm could search for the
global optimal hyperparameter combination through fuzzy control technology.
The evolutionary algorithm took longer to evaluate the function as dealing
with the large-scale problems, and it might involve topological structure and
geometric calculations.

Bayesian optimization [4] did not involve the above-mentioned topological
structures and geometric calculations, and it also didn’t analyze the
structure of the objective function. It was a black-box optimization algorithm
which selected the location of the evaluation point through the acquisition
function. The acquisition function ensured that the next evaluation point was
always optimal or explored points which had not appeared before. Bayesian
optimization useed the covariance matrix to solve the above-mentioned
dimensionality problems. Therefore, Bayesian optimization was often used for a
variety of research tasks. For example, Liu _et al._ [20] proposed to use
Bayesian optimization for intrusion detection tasks, searching for a set of
optimal hyperparameters suitable for the network to improve the detection
accuracy. And Xiang _et al._ [21] proposed the use of Bayesian optimization to
construct the skeleton of robots and animations to extend the automatic
construction method of the skeleton. Xu _et al._ [22] used Bayesian
optimization for generator parameter search and realized the generation of
time series data. The music generated by this method could not be
distinguished by humans. Our approach was different from the time series
prediction model and the method for automatically building the skeletons. In
this paper, it is mainly aimed at the construction of a generative adversarial
networks model for image reconstruction, rather than the problem of sample
generation. Inspired by the automatic skeleton construction method, we use
Bayesian optimization algorithm to optimize model. An automatical method can
reduce the manual workload and realize adaptive construction of network
models.

##  III Method

Automated design can speed up manual parameter adjustment. In this paper, a
Bayesian optimization method is used to optimize the generative adversarial
network. We realize automatic parameter adjustment and avoid the tedious
process of manual parameter adjustment. Different from the method proposed by
Xiang _et al._ [21], which mainly adjust the parameters of the joint points
connected by the robots and the animation skeletons. Our method is to tune the
hyperparameters of the deep learning model and build the generative
adversarial network structure automatically.

The optimization algorithm aims at achieving the minimum value of the
optimization objective function, and gradually searches for the global optimal
solution. In the neural network training process, the objective function
structure information cannot be obtained. In this paper, Bayesian optimization
algorithm is used to optimize generative adversarial network model. Compared
with other algorithms, Bayesian optimization has the following advantages: (1)
It is a black box optimization method, without considering the objective
function structure and gradient information; (2) No complicated topological
structure is needed to assist in searching the optimal solution.

In this paper, we build a super-resolution image reconstruction model based on
generative adversarial network by Bayesian optimization algorithm. It is
mainly divided into two stages: on the one hand, we build a basic generative
adversarial network model framework and define the parameters of generator and
discriminator; on the other hand, it is the optimization process of network
training combined with Bayesian optimization algorithm.

###  III-A Super-resolution image reconstruction model based on self-
calibrated convolutional GAN

In this paper, the method refers to the SRGAN [23] model to construct the
network structure, and the traditional method refers to empirical values to
set the network structure and parameters. Different from the SRGAN parameter
setting, we would use an automated method to explore the model parameters.
Therefore, the residual module of the generator network in the SRGAN network
is replaced with a self-calibrated convolution module. This module extracts
features more adequately and facilitates to select super parameters by
optimization algorithms. In order to realize the self-adaptive construction of
the network structure, we parameterizes the components of the generator and
the discriminator. Finally, we realize the self-adaptive construction of the
generative adversarial network model through Bayesian optimization policy to
adjusts the hyperparameters.

For deep learning model hyperparameter optimization tasks, the complexity of
the network increases the difficulty of automatic parameter adjustment. There
are many hyperparameters in the network model. Too many optimization
parameters not only increase the optimization time, but also prone to NP-hard
problem. The main problem of optimization is to control the number of
hyperparameters. In addition, generative adversarial networks are prone to
pattern collapse. Therefore, we adopts a fixed training strategy and only
optimizes hyperparameters such as the number of network layers and the number
of neurons. In this paper, the generator adopts a self-calibrated convolution
module to extract low-resolution image features. The generator contains two
hyperparameters, which are the number of self-calibrated convolution modules
layers and the number of neurons in the self-calibrated convolution. We
parameterize the generator, where m represents the number of self-calibrated
convolutional modules layers, and n represents the number of neurons. m and n
both are positive integers, and n must be a multiple of 4, namely $n\%4=0$.
The training of the generative adversarial network is a max-min game process.
In order to avoid the problem of too many optimization parameters, and the
discriminator has a simpler structure than the generator. Therefore, we only
define one parameter in the discriminator. The discriminator encapsulates the
convolutional layer, the batch normalization layer, and the activation
function layer into one module. Exploring through the optimization model, the
discriminator contains several layers of this module, where k represents the
number of layers of the module contained in the discriminator, and k is a
positive integer.

This chapter will encapsulate the structure of the generative adversarial
network so that it can meet the hyperparameter tuning form and facilitate to
achieve Bayesian optimization. The generative adversarial network model with
parameters is shown in figure 1.

Figure 1: The architecture of GAN model with parameters

###  III-B Bayesian optimization

In deep learning, Bayesian optimization usually needs to consider four aspects
to obtain the minimum value of the objective function through the network
model: objective function, bound range, historical result record and
optimization method. The explicit functions are as follows:

  * •

Objective function: the objective to be optimized, the performance evaluation
of the test set or verification set.

  * •

Bound range: selecting specific hyperparameter, and setting the upper and
lower bounds of the hyperparameters. It defines the value range

  * •

Optimization method: it defines an acquisition function which determines the
next evaluation point based on the posterior distribution of the optimization
target.

  * •

Historical result record: recording the previous target value, as a reference
for determining the next evaluation point.

According to the four aspects that need to be considered in the use of
Bayesian optimization in deep learning, the following specifically introduces
the design of super-resolution image reconstruction model by Bayesian
optimization method in this paper. The specific network model is shown in
figure 2.

Figure 2: The architecture of GAN model with Bayesian optimization

In this paper, we use MSE [21] as the optimization objective function. There
are two main reasons for selecting MSE. On the one hand, traditional super-
resolution image reconstruction tasks usually use MSE as the loss function.
However, the optimization objective function is different from the loss
objective function of the model. The optimization objective function is the
value that the model expects to be maximized in the Bayesian optimization
process (you can choose to minimize, and our method uses the maximize method).
Therefore, MSE as the optimization objective function does not conflict with
the loss function in super-resolution image reconstruction model. on the other
hand, as shown in formula PSNR, in the calculation of the super-resolution
image evaluation index PSNR, it is necessary to calculate the MSE value of the
two images firstly. In this task, we hope to obtain a set of optimal
hyperparameters of the generator, so that our network can reconstruct super-
resolution images that are indistinguishable by human eyes. The generative
adversarial network is more sensitive to the color of the image, and the
reconstructed super-resolution image has good results under the SSIM
evaluation index. This chapter is mainly designed to improve the PSNR value.
Therefore, MSE is relatively reasonable as an optimization objective function.
The optimization objective function formula is:

$f(I)=\frac{1}{n}\sum\limits_{i=1}^{n}{{{({I^{HR}}-{I^{SR}})}^{2}}}$

An evaluation function $f(I)$ is obtained by above formula. The historical
parameter value is replaced by one parameter $\theta$, expressed as
$\theta=\\{({m_{1}},{n_{1}}),({m_{2}},{n_{2}}),\cdots({m_{t}},{n_{t}})\\}$,
and the next set of parameter values is predicted to be
$({m_{t+1}},{n_{t+1}})$. The model is expressed as $f(I)=(I,\theta)$. When
data is continuously input into the model, a set of evaluation function values
are obtained from this, expressed as:

$D=\\{({I_{1}},f({I_{1}})),({I_{2}},f({I_{2}})),\cdots({I_{t}},f({I_{t}}))\\}$

Based on the above-known sample points, make a prediction to the next sample
point. On the basis of these known sample points, the probability distribution
of the evaluation function is obtained through the Gaussian process, and the
objective function is mapped through the kernel function to make it to the
Gaussian distribution. The Gaussian process is expressed as:

$f(I)\sim GP(\mu,k(I,I^{\prime}))$

Where the mean of Gaussian distribution is $\mu$, $k(I,I^{\prime})$ is the
kernel function. After kernel function mapping, the posterior distribution of
the new evaluation point is obtained. At the meantime, all the distributions
are the Gaussian distribution. The predicted value is expressed as
${y_{i}}=f({I_{i}})+{\varepsilon_{i}}$. ${\varepsilon_{i}}$ indicates that
variance $\sigma_{0}^{2}$ is added to Gaussian noise, and its value is close
to 0. The formula $y={({y_{1}},{y_{1}}\cdots{y_{t}})^{N}}$ calculates the
prior covariance vector value between $y$ and $f(I)$, expressed as
$k(f(I),y)$. After calculation, the Gaussian distribution obtained by Gaussian
kernel mapping is expressed as follows:

$p(f({I^{*}})|y)=N(f({I^{*}})|\mu({I^{*}}),\sigma({I^{*}}))$

Where $\mu({I^{*}})$ is the mean and $\sigma({I^{*}})$ is the variance, both
of which are calculated by the covariance.

After the above process, a set of values can be expressed as
${D_{1}}=(f({I_{i}}),{y_{i}})$, so as to predict the next evaluation point.
The selection of the next evaluation point is determined by the acquisition
function. The acquisition function construction depends on the optimization
objective function. A classic evaluation method is:

$p(f(I)\geq\nu)=\Phi(\frac{{\mu(I)-\nu}}{{\sigma(I)}})$

Among them, $\nu$ represents the parameter ${y_{i}}$ that maximizes the
predicted value and $\Phi$ is a standard normal distribution. $\mu(I)$ and
$\sigma(I)$ are the mean and variance of each evaluation point in the
objective function. The mean represents the expected maximum value, and the
variance represents the exploration ability of the model.

In this paper we adopt UCB (Upper Confidence Bound) to balance the
relationship between mean and variance, and predicting the next evaluation
point. Xiang[21] et al has been proved that the UCB method is superior to the
EI(Expected Improvement) method in frame construction. UCB formula is as
follows:

$ucb=\mu(I)+\lambda\sigma(I)$

Among them, $\lambda$ is a hyperparameter, which is used to adjust the mean
and variance, and the setting value is 1.

The specific algorithm of the method in this paper shows in algorithm 1:

Input: image datasets

1 initialize training iterations(number), batch size(step)

2 initialize the boundary of hyperparameters(m, n, k)

3 for  _i in iterations_ do

4 for  _number in training iterations_ do

5 for  _step in steps_ do

6 ${\nabla_{{\theta_{d}}}}\frac{1}{m}\sum\limits_{i=1}^{m}{\left[{\log
D({x^{(i)}})+\log(1-D(G({z^{i}})))}\right]}$

7 end for

8
${\nabla_{{\theta_{g}}}}\frac{1}{m}\sum\limits_{i=1}^{m}{\left[{\log(1-D(G({z^{i}})))}\right]}$

9 end for

10 $y\leftarrow f(I,\theta)$

11 ${\theta_{t+1}}\leftarrow{\theta_{1:t}}$

12 $re=\arg\max{y_{i}}$ return $re$

13 end for

Output: Objective function value

Algorithm 1 Generative Adversarial Networks with Bayesian Optimization

##  IV Experiment

This chapter first introduces the experimental conditions and the training
details, then it introduces and analyzes the experimental results, finally it
shows the effect of applying the above optimization results to super-
resolution image reconstruction tasks.

###  IV-A Experimental conditions

The experiment was trained on the Windows 10 operating system and NVIDIA
2080Ti server. The CUDA version is 10.2. All source programs are written in
python language and implemented using the pytorch framework. The pytorch
version is 1.6. The batch size is 64. The upper and lower bounds of
hyperparameters are [2, 11], [64, 256] and [2,10] respectively. The method
based on Bayesian optimization uses a part of the VOC2012 [23] datasets, and
using the test set in the VOC2012 as the training set. Since the batch size is
64, a certain number of images were randomly selected from the original
training set to made the total number of 64 multiple. There were eight images
been selected from the original training set as the verification set, which
were used in the calculation of the objective function in the optimization
algorithm.

###  IV-B Evaluation index

We evaluated the reconstructed super-resolution image under the evaluation
indicators PSNR and SSIM [24]. PSNR compared the corresponding pixels between
super-resoluton image and high-resolution image. SSIM compared the similarity
between super-resoluton image and high-resolution image.

The calculation formula of MSE is:

$MSE=\frac{1}{n}\sum\limits_{i=1}^{n}{{{({I^{HR}}-{I^{SR}})}^{2}}}$

$PSNR=10\cdot{\log_{10}}\left({{1\mathord{\left/{\vphantom{1{MSE}}}\right.\kern-1.2pt}{MSE}}}\right)$

Where high-resolution image is expressed as ${I^{HR}}$, and super-resolution
image is expressed as ${I^{SR}}$.

The calculation formula of SSIM is:

$SSIM=\frac{{\left({2{\mu_{{I^{SR}}}}{\mu_{{I^{HR}}}}+{C_{1}}}\right)\left({2{\sigma_{{I^{SR}}{I^{HR}}}}+{C_{2}}}\right)}}{{\left({\mu_{{I^{SR}}}^{2}+\mu_{{I^{HR}}}^{2}+C1}\right)\left({\sigma_{{I^{SR}}}^{2}+\sigma_{{I^{HR}}}^{2}+C1}\right)}}$

Where ${\mu_{{I^{SR}}}}$ is the mean of ${I^{SR}}$, ${\mu_{{I^{HR}}}}$ is the
mean of ${I^{HR}}$, ${\sigma_{{I^{SR}}{I^{HR}}}}$ is the covariance of
${I^{SR}}$ and ${I^{HR}}$, and $\sigma_{{I^{SR}}}^{2}$ and
$\sigma_{{I^{HR}}}^{2}$ is the variance of ${I^{SR}}$ and ${I^{HR}}$.

The greater the value of the above evaluation method, the higher the quality
of the reconstructed image, and the more difficult to distinguish for the
human eye.

###  IV-C Experimental results and analysis

This section mainly focuses on the display and analysis of the results of
Bayesian optimization and the results of the super-resolution image
reconstruction of the generative adversarial network. First, a brief
description of the relevant algorithms:

  * •

COBYLA algorithm [16]: it is a linear approximation optimization algorithm
that constrains the region bound. Calculate the upper and lower bounds based
on the simplex, and then determine the region bound. when the objective
function is approximated by linear interpolation, the optimal solution
constructed should be within the region bound. This algorithm has a better
effect on integer boundary constraints, but it will make mistakes on decimal
boundary constraints.

  * •

PSO algorithm [18]: this algorithm has two attributes: position and speed,
which respectively describe the direction of exploration and the speed of
movement when the individual approaches the target value. Individuals in the
group need to exchange information to select the individual who currently
reaches the best target value. When the algorithm has a large scale, as the
individual or dimension increases, the amount of calculation also increases.

  * •

Bayesian optimization algorithm [4]: it is an optimization algorithm based on
the Bayesian probability model. Unlike the above method, this method will
refer to the previous evaluation results when looking for the next set of
solutions. Bayesian optimization consists of four parts: recording historical
results is convenient for trying the next set of solution references; defining
upper and lower bounds of hyper-parameters; adopting the loss on the
verification set as the objective function; constructing the collection
function to select the next evaluation point.

In order to illustrate the superiority of the Bayesian optimization algorithm,
we compared the experimental results of two generative adversarial network
construction methods. The first is the search result of the generator and the
discriminator parameter by the optimization algorithm ($BOP_{G}+BOP_{D}$); the
second is the search for the generator parameter by the optimization algorithm
($BOP_{G}$). Since the generator in this article is more complicated than the
discriminator, the generator is searched separately. The experimental results
of the above three optimization algorithms for the optimization of the two
network structures are shown in figure 3 and figure 4.

Figure 3: The curve of $BOP_{G}+BOP_{D}$ objection function Figure 4: The
curve of $BOP_{G}$ objection function

In the figure 3 and figure 4, the horizontal axis is the number of iterations,
and the vertical axis is the optimized objective function value. Bayesian
optimization searches for a set of hyperparameters that can maximize the
objective function, and COBYLA and PSO optimization methods search for a set
of hyperparameters that minimize the objective function. The $BOP_{G}+BOP_{D}$
optimization results can be obtained from figure 3. The Bayesian optimization
algorithm maximizes the objective function at the second iteration, the COBYLA
algorithm minimizes the objective function at the 9th iteration, and the PSO
algorithm minimizes the objective function at the 19th iteration. The
$BOP_{G}$ optimization results can be obtained from figure 4. The Bayesian
optimization algorithm maximizes the objective function at the 3rd iteration,
the COBYLA algorithm minimizes the objective function at the 11th iteration,
and the PSO algorithm minimizes the objective function at the 5th iteration.
It can be seen that Bayesian optimization can search for a set of
hyperparameters that meet the conditions in the least number of iterations,
which verifies the effect of the Bayesian optimization algorithm.

Through the above-mentioned optimization algorithm curve, we obtain the
optimal hyperparameter combinations of the two network models
$BOP_{G}+BOP_{D}$ and $BOP_{G}$ under different algorithms. The specific
hyperparameters of different algorithms are shown in table I and II.

TABLE I: $BOP_{G}+BOP_{D}$ model hyperparameter result hyperparameter | Bayesian optimization | COBYLA | PSO
---|---|---|---
Layers(G) | 3 | 2 | 2
Neurons(G) | 140 | 128 | 132
Layers(D) | 3 | 2 | 2
TABLE II: $BOP_{G}$ model hyperparameter result hyperparameter | Bayesian optimization | COBYLA | PSO
---|---|---|---
Layers(G) | 3 | 2 | 2
Neurons(G) | 184 | 124 | 160

According to the network parameters obtained by the above-mentioned different
optimization algorithms, a super-resolution image reconstruction generative
adversarial network model ($BOP_{G}+BOP_{D}$) is constructed. In the case of
scale 4, we use all the data of the VOC2012 to train the generative
adversarial network to obtain a super-resolution image reconstruction model.
The model performs on the above three algorithms in the set5, set14 and BSD100
test sets in turn. The comparison results under each evaluation index are
shown in table III. In addition, this paper also verified the experimental
results of the $BOP_{G}$ model on different test sets, and the comparison
results of the reconstructed super-resolution images in each evaluation index
are shown in the table IV.

TABLE III: $BOP_{G}+BOP_{D}$ model results under different test sets Datasets | Methon | Bayesian optimization | COBYLA | PSO
---|---|---|---|---
set5 | PSNR | 29.89 | 29.77 | 29.49
SSIM | 0.9288 | 0.9277 | 0.9240
set14 | PSNR | 26.59 | 26.46 | 26.33
SSIM | 0.8531 | 0.8532 | 0.8505
BSD100 | PSNR | 25.97 | 25.95 | 25.89
SSIM | 0.8107 | 0.8115 | 0.8062
TABLE IV: $BOP_{G}$ model results under different test sets Datasets | Methon | Bayesian optimization | COBYLA | PSO
---|---|---|---|---
set5 | PSNR | 29.00 | 27.40 | 28.85
SSIM | 0.9131 | 0.9139 | 0.9143
set14 | PSNR | 26.11 | 25.14 | 26.02
SSIM | 0.8419 | 0.8400 | 0.8424
BSD100 | PSNR | 25.62 | 24.88 | 25.60
SSIM | 0.8013 | 0.8002 | 0.8015

This paper mainly designs and optimizes the objective function for the PSNR
value, which promotes the model to have better results under the evaluation
index PSNR. From the table III and table IV, we have compared the PSNR values
of two different network models ($BOP_{G}+BOP_{D}$ and $BOP_{G}$ )to
reconstruct super-resolution images on different test data sets. The
experimental results of the two network models are as follows. The result of
$BOP_{G}$ is: the result of COBYLA algorithm is the worst. The result of PSO
algorithm is relatively close to that of Bayesian optimization algorithm, but
the result of Bayesian optimization algorithm is slightly better. The result
of $BOP_{G}+BOP_{D}$ is: the result of PSO optimization algorithm is poor, the
result of COBYLA optimization algorithm is relatively close to that of
Bayesian optimization algorithm, but the result of Bayesian optimization
algorithm is the best. Under the evaluation index SSIM, the results of the
three optimization algorithms are relatively close. Because the method in this
paper is mainly optimized for the MSE objective function, and the result of
reconstructing the image under the PSNR evaluation index is more obvious,
which achieves the purpose of this paper.

This paper not only compares the results of each evaluation index, but also
gives the visualization results of the reconstructed image. The super-
resolution image reconstruction results of the three optimization algorithms
under the set5 and set14 are shown in figure 5. It can be seen from the figure
that Bayesian optimization has better visual effects compared with the other
two algorithms, regardless of the $BOP_{G}+BOP_{D}$ and the $BOP_{G}$.

Figure 5: Visualization of the super-resolution image of $BOP_{G}+BOP_{D}$
Figure 6: Visualization of the super-resolution image of $BOP_{G}$

In summary, since the training of the generative adversarial network is a max-
min game process, the experimental results of the $BOP_{G}+BOP_{D}$ model
under the three optimization algorithms are better than the $BOP_{G}$ results,
and the Bayesian optimization algorithm has better results in
$BOP_{G}+BOP_{D}$ and $BOP_{G}$. The search speed of the PSO algorithm
gradually increases when the model scale becomes larger. The PSO algorithm
needs to build a topological structure to assist in the transmission of
information in the process of searching for the optimal solution.
$BOP_{G}+BOP_{D}$ adds discriminator parameters, and a lot of calculations are
added in the process of information transmission. Therefore the longest search
time. The experimental results of COBYLA optimized $BOP_{G}+BOP_{D}$ model is
closer to the Bayesian optimization algorithm, but there may be dimensional
problems. The Bayesian optimization algorithm is a black-box optimization
algorithm. There is no complicated topological structure, the optimal value
can be searched earliest, and two different reconstructed super-resolution
image models have the best effect under the evaluation index.

##  V Conclusion

We presented methods for performing Bayesian optimization for hyperparameter
selection of generative adversarial networks. We introduced a fully Bayesian
treatment for UCB, and algorithms for dealing with optimizing generative
adversarial network training. The effectiveness of our approaches were
demonstrated on the speed of search and reconstruction image quality in
different algorithms. The results of Bayesian optimization have found the
better hyperparameters which significantly faster than the approaches used by
the artificial. It also shows that Bayesian optimization has a better result
than COBYLA and PSO algorithm. However, it is a challenge for us to solve the
problem that the quality of the super-resolution image reconstructed by the
Bayesian optimization network is worse than the super-resolution image
reconstructed by the artificial tuning parameter construction network.

## Acknowledgment

This work are sponsored by the key project of the National Natural Science
Foundation of China (61602421), China Postdoctoral Foundation (2016M600584),
Aviation Science Foundation (201828X4001) and Zhengzhou University.

## References

  * [1] P. J. Lally, P. M. Matthews, and N. K. Bangerter, “Unbalanced ssfp for super-resolution in mri,” _Magnetic Resonance in Medicine_ , vol. 85, no. 5, 2020.
  * [2] Y. Bai, Y. Zhang, M. Ding, and B. Ghanem, “Finding tiny faces in the wild with generative adversarial network,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018.
  * [3] A. C. Florea and R. Andonie, “Weighted random search for hyperparameter optimization,” _International Journal of Computers, Communications and Control (IJCCC)_ , vol. 14, no. 2, pp. 154–169, 2019.
  * [4] M. Pelikan, “Boa: The bayesian optimization algorithm,” in _Proc Genetic and Evolutionary Computation Conference_ , 1999.
  * [5] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. ourville, and Y. Bengio, “Generative adversarial networks,” _Advances in Neural Information Processing Systems_ , vol. 3, pp. 2672–2680, 2014.
  * [6] G. Liu, F. A. Reda, K. J. Shih, T. C. Wang, A. Tao, and B. Catanzaro, “Image inpainting for irregular holes using partial convolutions,” in _European Conference on Computer Vision_ , 2018.
  * [7] M. Sabini and G. Rusak, “Painting outside the box: Image outpainting with gans,” 2018.
  * [8] T. C. Wang, M. Y. Liu, J. Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro, “High-resolution image synthesis and semantic manipulation with conditional gans,” 2017.
  * [9] J. Chen, Y. Zhang, X. Hu, and Y. C. Chen, “Cascading residual-residual attention generative adversarial network for image super resolution,” _Soft Computing_ , no. 1, 2021.
  * [10] V. Chudasama and K. Upla, “Computationally efficient progressive approach for single-image super-resolution using generative adversarial network,” _Journal of Electronic Imaging_ , vol. 30, no. 2, 2021.
  * [11] ——, “Rsrgan: computationally efficient real-world single image super-resolution using generative adversarial network,” _Machine Vision and Applications_ , vol. 32, no. 2, pp. 1–18, 2021.
  * [12] J. Bergstra, R. Bardenet, B. Kegl, and Y. Bengio, “Algorithms for hyper-parameter optimization,” in _Advances in Neural Information Processing Systems_ , 2011.
  * [13] C. Audet and W. Hare, “Nelder-mead,” 2017.
  * [14] A. Hz, C. Aahb, M. W. D, E. Lz, A. Hc, and F. Cl, “Orthogonal nelder-mead moth flame method for parameters identification of photovoltaic modules,” _Energy Conversion and Management_ , vol. 211.
  * [15] P. Niegodajew, M. Marek, W. Elsner, and U. Kowalczyk, “Power plant optimisation-effective use of the nelder-mead approach,” _Processes_ , vol. 8, no. 3, p. 357, 2020.
  * [16] M. Powell, “A direct search optimization method that models the objective and constraint functions by linear interpolation,” _Advances in Optimization and Numerical Analysis_ , vol. 275, pp. 51–67, 1994.
  * [17] O. Kramer, _Scikit-Learn_. Machine Learning for Evolution Strategies, 2016.
  * [18] J. Kennedy and R. Eberhart, “Particle swarm optimization,” in _Icnn95-international Conference on Neural Networks_ , 2002.
  * [19] D. Gong, “Research on hyperparameter evolutionary tuning of deep neural network mode and its application,” in _Master’s thesis_ , 2020.
  * [20] H. Liu and Z. Zhou, “Industrial control network intrusion detection based on hyperparameter automatic optimization,” _Information and control_ , no. 1-8, 2021.
  * [21] Z. Xiang, C. Xiang, T. Li, and Y. Guo, “A self-adapting hierarchical actions and structures joint optimization framework for automatic design of robotic and animation skeletons,” _Soft Computing_ , no. 3, 2020.
  * [22] Y. Xu, X. Yang, Y. Gan, W. Zhou, H. Cheng, and X. He, “A music generation model based on generative adversarial networks with bayesian optimization,” in _Chinese Intelligent Systems Conference_ , 2020.
  * [23] C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, and Z. Wang, “Photo-realistic single image super-resolution using a generative adversarial network,” _IEEE Computer Society_ , 2016.
  * [24] W. Zhou, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE Trans Image Process_ , vol. 13, no. 4, 2004.