# DFGNN: Dual-frequency Graph Neural Network for Sign-aware Feedback

Yiqing Wu  Institute of Computing Technology, Chinese Academy of
SciencesBeijingChina<EMAIL_ADDRESS>,  Ruobing Xie  TencentBeijingChina
<EMAIL_ADDRESS>,  Zhao Zhang  Institute of Computing Technology,
Chinese Academy of SciencesBeijingChina<EMAIL_ADDRESS>,  Xu Zhang
TencentBeijingChina ,  Fuzhen Zhuang  Institute of Artificial Intelligence,
Beihang UniversityBeijingChina<EMAIL_ADDRESS>,  Leyu Lin  ,
Zhanhui Kang  TencentBeijingChina goshawklin<EMAIL_ADDRESS>and
Yongjun Xu  Institute of Computing Technology, Chinese Academy of
SciencesBeijingChina<EMAIL_ADDRESS>
(2018; 20 February 2007; 12 March 2009; 5 June 2009)

###### Abstract.

The graph-based recommendation has achieved great success in recent years.
However, most existing graph-based recommendations focus on capturing user
preference based on positive edges/feedback, while ignoring negative
edges/feedback (e.g., dislike, low rating) that widely exist in real-world
recommender systems. How to utilize negative feedback in graph-based
recommendations still remains underexplored. In this study, we first conducted
a comprehensive experimental analysis and found that (1) existing graph neural
networks are not well-suited for modeling negative feedback, which acts as a
high-frequency signal in a user-item graph. (2) The graph-based recommendation
suffers from the representation degeneration problem. Based on the two
observations, we propose a novel model that models positive and negative
feedback from a frequency filter perspective called Dual-frequency Graph
Neural Network for Sign-aware Recommendation (DFGNN). Specifically, in DFGNN,
the designed dual-frequency graph filter (DGF) captures both low-frequency and
high-frequency signals that contain positive and negative feedback.
Furthermore, the proposed signed graph regularization is applied to maintain
the user/item embedding uniform in the embedding space to alleviate the
representation degeneration problem. Additionally, we conduct extensive
experiments on real-world datasets and demonstrate the effectiveness of the
proposed model. Codes of our model will be released upon acceptance.

Sign-aware recommendation, Signed graph neural network, Negative feedback,
Recommendation

††copyright: acmlicensed††journalyear: 2018††doi: XXXXXXX.XXXXXXX††conference:
Make sure to enter the correct conference title from your rights confirmation
emai; June 03–05, 2018; Woodstock, NY††isbn: 978-1-4503-XXXX-X/18/06††ccs:
Information systems Recommender systems

##  1\. Introduction

Personalized recommender systems aim to provide appropriate items to the users
on platforms. In such systems, the interactions between users and items can
naturally be regarded as a user-item bipartite graph. Inspired by the success
of Graph Neural Networks (GNNs), the graph-based recommendation model has been
widely studied and achieved remarkable progress. However, most existing graph-
based recommendation models are merely built on positive edges/feedback.
Actually, there is always various negative feedback in real-world recommender
systems, such as a low rating, dislike, and short watching time. The negative
feedback generally represents what people do not like or even hate.
Recommendations of such items to users may significantly harm the user
experience. Besides, only focusing on what users like while ignoring what
users dislike may lead to homogenization in the recommended items (Zhang et
al., 2018).

Although negative feedback is crucial and is believed to benefit the
recommendation (Jeunen, 2019), how to model the negative feedback in graph-
based recommendation still remains underexplored. A trivial solution is to
introduce negative feedback edges to the graph and model them in a manner
similar to positive feedback. However, existing GNNs heavily rely on the
homophily assumption, where two nodes connected by an edge tend to be more
similar. While, for negative feedback, two nodes connected by an edge may
imply dissimilarity. A similar field is the signed graph neural network. It is
designed for processing signed graphs, which contain two opposite edge types
(e.g., accept/reject, fraud/trustworthiness). The signal in the signed graph
generally is objective, and most of them are built upon balance theory, which
means the friend of my friend is my friend, and the enemy of my enemy is my
friend. However, the user feedback in recommendations is subjective and
personalized, and the balance theory does not always hold in recommendations
(Seo et al., 2022). Currently, limited works (Huang et al., 2023) have studied
the negative feedback in graph-based recommendations. While they still adopt
GNNs that rely on the homophily assumption.

Based on the above analysis, we start to ponder _”How to model positive and
negative feedback in graph-based recommendation?”_ To this end, we deeply
contemplate the characteristics of positive and negative feedback. (1)
Positive and negative feedback carry different meanings. Generally, if an item
fits users’ interests, the people will produce positive feedback. In
recommendation, this implies a certain degree of similarity between items and
users. In contrast, negative feedback implies dissimilarity. While this makes
the negative feedback violate the homophily assumption of GNN. (2) The reasons
for negative feedback are heterogeneous. Compared to positive feedback, the
reasons for users produce negative feedback are diverse and complex. It could
stem from a variety of reasons such as the user hating the categories of the
item, the items not meeting the user’s expectations, or even due to the user
hating a specific word in the title. (3)The negative feedback is generally
sparse. The recommender system aims to recommend what people will like, the
negative feedback actually is an anomaly, making it sparse. For example, in
the Amazon dataset, less than 10% of reviews have low ratings.

Looking deeper into the structure information of graph-based recommendation,
we can discover that: a user node is connected to many nodes that are similar
to it by positive feedback, resulting in a smooth similarity terrain. However,
there are also a few strange peaks that stand out, appearing out of place,
that is the negative feedback. This inspires us to think about positive and
negative feedback essentially from the perspective of signal processing. In
which the smooth and unchanged signals are regarded as low-frequency signals.
While steep, rapid changes are considered as high-frequency signals. This is
highly analogous to the characteristics of positive and negative feedback. To
evaluate our assumption, we utilize graph spectral theory to experimentally
analyze the positive and negative feedback in the frequency-domain (detailed
in Sec. 3.1) and find that the positive feedback indeed implies low-frequency
signals in the graph while the negative feedback implies high-frequency
signals in the graph. Furthermore, the GNNs tend to over-smooth low-frequency
signals. Through experiment, we also find that the learned embedding suffers
from a representation degeneration problem, which means the embedding shrinks
to a small piece of embedding space thus resulting in a loss of expressive
power. This issue has a more significant impact on signed graphs, as it leads
to an inability to distinguish between negative and positive feedback.

Considering the characteristics of negative and positive feedback, we propose
a novel method to model them in graph-based recommendations, called Frequency-
aware signed graph neural network (DFGNN). Specifically, considering the low-
frequency signal and high-frequency signal contained in positive and negative
feedback, respectively, we propose a dual-frequency graph filter(DGF). In DGF
we utilize the designed low-passing graph filter to model the positive
feedback while adopt the high-passing filter to model the negative feedback.
To alleviate the representation degeneration problem caused by GNN, we propose
a signed graph regularization (SGR) loss. SGR considers both the similar and
dissimilar information in the signed graph and regularizes the learned
embedding to be uniformly distributed in the space. The experiments on real-
world datasets demonstrate that our model achieves significant improvements
over competitive baselines.

Overall, the contribution of this paper is summarized as follows:

  * •

We first conduct a comprehensive analysis of positive and negative feedback in
the graph-based recommendation from the frequency perspective. Furthermore, we
design a dual-frequency filter to both model low-frequency signals and high-
frequency signals contained in positive and negative feedback.

  * •

We investigate the representation degeneration problem in graph
recommendation, and propose a sign graph regularization loss to alleviate it.

  * •

We conduct extensive experiments on real-world datasets and evaluate the
effectiveness of the proposed DFGNN in two recommendation tasks.

##  2\. Preliminaries

###  2.1. Signed Graph in Recommendation

In real-world online recommender systems, users generally have various
interactions with items and generate feedback to the system, including both
positive feedback (e.g. like, buy) and negative feedback (e.g. dislike, low
rating). Based on the positive and negative feedback, we define the signed
graph in recommendation. Specifically, we denote the signed graph as
$G=\\{U,V,\\{\mathcal{E}^{+},\mathcal{E}^{-}\\}\\}$, where
$U=\\{u_{1},u_{2},...,u_{|U|}\\}$ and $V=\\{v_{1},v_{2},...,v_{|V|}$ are the
user set and item set in the system, and
$\\{\mathcal{E}^{+},\mathcal{E}^{-}\\}$ are positive and negative edges in the
graph, respectively. If a user has positive feedback for an item, there will
be a positive edge $\xi=(u,v,+)\in\mathcal{E}^{+}$. Conversely, a negative
edge $\xi=(u,v,+)\in\mathcal{E}^{-}$ will be created if there is negative
feedback.

###  2.2. Graph Fourier Transform

To better understand the graph Fourier transform. We first introduce the
Fourier transform. Fourier transform is a powerful analytical tool and is
widely applied across various fields. It transforms signals from the time
domain to the frequency domain. The Fourier transform can be formulated as :

(1) $f(\omega)=\mathcal{F}(f(x),\omega)\int_{-\infty}^{\infty}f(x)\cdot
e^{-2\pi i\omega}\,dx,$

(a) sin(x) (b) 0.2 sin(10x) (c) 0.2 sin(10x)+sin(x) (d) $\mathcal{F}$ (sin(x))

(e) $\mathcal{F}$ (0.2 sin(10x)) (f) $\mathcal{F}$ (0.2 sin(10x)+sin(x))

Figure 1. Different frequency signals in the time domain and frequency domain.

where $\mathcal{F}$ is denoted as Fourier transform operation, $\omega$ is the
frequency, $f(\omega)$ are the amplitudes of corresponding $\omega$.
Generally, low frequencies correspond to smooth signals, while high
frequencies correspond to non-smooth signals. The amplitude represents the
strength of the signal at the corresponding frequency. To better understand
this, we demonstrate this phenomenon in Fig.1. We can see that $sin(x)$ (i.e.,
Fig. 1(a)) is smoother, thus its corresponding low frequency is higher(i.e.,
Fig. 1(d)). In contrast, $0.2sin(10x)$ oscillates rapidly (i.e., Fig. 1(b)),
and correspondingly, the amplitude of high frequencies is higher (i.e. Fig.
1(e)). Fig. 1(c) looks complex in the time domain, but in the frequency domain
(Fig. 1(f)), it looks simple—it’s just the superposition of two sine waves,
$sin(x)$ and $0.2sin(10x)$.

To extend the Fourier transform to the graph/non-euclidean domain, Laplacian
matrices ${L}$ are introduced to the graph processing field. Classical
Laplacian matrice is defined as $L=D-A$, where $A$ is the adjacent matrix and
$D$ is the degree matrix of a graph. $L$ is a real symmetric matrix, thus it
can be written as $L=E\Lambda E^{T}$, where $E=\\{e_{1},e_{2},...,e_{N}\\}$ is
the eigenvector and $\Lambda=\\{\lambda_{1},\lambda_{2},...,\lambda_{N}\\}$ is
the eigenvalue of $L$. Similar to the Fourier transform in the time domain,
the Fourier transform in the graph/non-euclidean domain is defined as:

(2) $f(\lambda_{l})=\sum_{0}^{N}x(i)e^{\lambda_{l}}(i),$

where $e^{\lambda_{l}}$ is the eigenvector that corresponds to eigenvalue
$\lambda_{l}$, and $x(i)$ is the node feature of $i$-th node. Graph Fourier
Transform shares similar characteristics with the time-domain Fourier
Transform. Low $\lambda$ values represent low-frequency signals, while high
$\lambda$ values represent high-frequency signals. Besides, $f(\lambda)$
represents the strength of the corresponding frequency.

##  3\. Experimental Observations

In this section, we aim to analyze the characteristics of negative and
positive feedback and the challenges during the processing of signed graphs in
recommendation. In the following, we first analyze the negative feedback from
a frequency perspective, then we analyze the representation degeneration
problem in the graph-based recommendation.

###  3.1. Analysis of Positive and Negative Feedback from Frequency-domain

Generally, the user behaviors in a system can be modeled to a signal graph,
where users and items are nodes, edges represent interactions on the graph,
and user and node features serve as signals on the graph. As introduced in
Sec.2.3, with the help of the graph Fourier transform, we can analyze signals
from a frequency perspective. To explore the characteristics of
positive/negative feedback on the graph, we conduct experimental analysis in
the frequency domain. Specifically, we first employ the classical Matrix
Factorization (Funk, [n. d.]) and learn one-dimensional embeddings for users
and items as the signals of the graph. Then we construct two graphs that only
contain positive edges and negative edges respectively, denoted as $G^{+}$ and
$G^{-}$. Last, we apply graph Fourier transform to the $G^{+}$ and $G^{-}$. We
conduct this experiment on ml-100k dataset and show the distribution of
frequency in Fig.2. By comparing Fig 2 (a) and Fig 2 (b), we can observe that
the signal of $G^{+}$ is prominent in low frequencies, while $G^{-}$ is
prominent in high-frequency signals. Recall that, the low frequencies
represent the smooth signals, and high frequencies represent unsmooth signals.
In the temporal domain, smoothness can be defined as the magnitude of the
signal change between timestep $t-1$ and $t$. Similarly, in the graph,
smoothness can be measured by the difference between the nodes connected by an
edge(Shuman et al., 2013). The smaller the difference between nodes connected
by an edge, the smoother the graph. Therefore the results in Fig 2 align with
intuition. In recommendation, we generally assume that users have higher
similarity to items they like, and correspondingly, the $G^{+}$ is dominated
by low-frequency signal. On the contrary, users are dissimilar to items they
dislike, and correspondingly, G- exhibits more high-frequency signals. Most
existing GNNs actually act as a low-passing filter(He et al., 2020; Kipf and
Welling, 2016; Wu et al., 2019), which means the GNN extracts low-frequency
signals while discarding high-frequency signals. Unfortunately, most existing
Graph-based recommendation models adopt the low-passing GNN to encode graphs.
Thus, they may not be suitable for capturing negative feedback signals.

(a) negative feedback (b) positive feedback

Figure 2. The distribution of f($\lambda$) calulated based on normalzied
Laplacian matrix. (a) It is the distribution of only negative edges graph. (b)
It is the distribution of only positive edges graph. We evenly split the
frequency $\lambda$ into ten buckets and calculate the normalized
f($\lambda$).

###  3.2. The Representation Degeneration Problem in Graph-based
Recommendation

Recently, graph neural networks have achieved great success in recommendation
systems (He et al., 2020; Wu et al., 2022; Ying et al., 2018). However, GNNs
are known as the over-smoothing effect (Cai and Wang, 2020; Elinas and
Bonilla, 2022), which means the node feature tends to be similar during the
GNN training, especially in deep graph neural networks. The over-smoothing
effect sometimes will cause the representation degeneration problem. In this
issue, the embeddings shrink to a small piece of space, thus losing the
expressive power. To explore whether the graph-based recommendation will
suffer from this problem, we train a graph model (i.e. GCN(Kipf and Welling,
2016)) and a non-graph model (i.e. NCF(He et al., 2017)) on Amazon ArtsCrafts
review dataset, respectively. We project the learned embedding matrix into 2D
by SVD and visualize it. The results are shown in Fig. 3. From this figure, we
can find that the embedding learned by NCF is more uniform than that learned
by GCN. Most node embeddings learned by GCN fall into a narrow zone.
Furthermore, we analyze the distribution of singular values of embedding
matrices and show the results in Fig.3. We can observe a rapid decline in the
singular values of GCN, indicating that the learned embeddings are low-rank
and have less expressive power.

(a) NCF

(b) GCN

(c) Singular values

Figure 3. The visualized embedding and singular values. (a) is the embedding
learned by NCF. (b) is the embedding learned by GCN. (c) is the normalized
singular values of GCN and NCF.

##  4\. Method

Based on the above experimental observations, in this paper, we propose a
novel graph model called DFGNN to solve the issues in the sign-aware graph
recommendation. Our DFGNN contains two components: (1) dual frequency graph
filter and (2) graph regularization. The dual-frequency graph filter aims to
both capture the high-frequency signal and the low-frequency in the negative
feedback and positive feedback respectively. Furthermore, the graph
regularization module is adopted to alleviate the representation degeneration
problem. In the following, we will make a detailed introduction to our DFGNN.

###  4.1. Dual Frequency Graph Filter

The negative feedback is crucial and widely present in recommendation systems,
which show what people dislike or even hate what kind of items. However, how
to deal with negative feedback is not a trivial problem. As the analysis in
Sec 3.1, different from the positive feedback, the negative feedback usually
acts as the high-frequency signal in the graph. Hence we propose a dual-
frequency graph filter (DGF), which uses a low-passing graph filter (LGF) to
capture the low-frequency signal in positive feedback and adopts a high-
passing graph filter (HGF) to capture the high-frequency signal in negative
feedback.

####  4.1.1. Graph Filter

The convolution operation is generally adopted as a filter to extract useful
signals. The graph convolution can be defined as:

(3) $\displaystyle x*g=$
$\displaystyle\mathcal{F}^{-1}(\mathcal{F}(x)*\mathcal{F}(g))=\mathcal{F}^{-1}(f(\lambda)g(\lambda))$
$\displaystyle=$ $\displaystyle
E((E^{T}g)\dot{(}E^{T}x))=E\,g(\lambda)\,E^{T}\,x,$

where $g(\lambda)$ is the filter kernel function with respect to the
eigenvalues $\lambda$ of the Laplacian matrix. Generally, for a low-pass
filter, $g(\lambda)$ is high at low $\lambda$ but low at high $\lambda$. For a
high-pass filter, it is the opposite. $(g(\lambda)$ is high at high $\lambda$
and low at low $\lambda$.

####  4.1.2. Low-passing Graph Filter

After defining the graph filter, we can design the low-passing filter.
Luckily, most existing GNNs act as low-passing graph filters. In this work, we
adopt the classical GCNs (Kipf and Welling, 2016) as our low-passing graph
filter. The GCNs can be formulated as:

(4) $H^{l+1}=\sigma(\hat{A}H^{l}W^{l}),H^{0}=X,$

where $W^{l}\in R^{d1*d2}$ is the trainable parameters of the network,
$H^{l}\in R^{N*d1}$ is the node representation after $l$ layer convolution.
$\hat{A}=\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{\\-\frac{1}{2}}$ is a
designed matrix, where $\tilde{A}=A+I$ are the augmented adjacent matrix with
adding self-loops and $\tilde{D}=D+I$ are the degree matrix of $\tilde{A}$.

####  4.1.3. High-passing Graph Filter

Figure 4. The graph filter kernel function of LGF and HGF with layer stack.
The left is the kernel (1-$\lambda$) and the right is$\lambda$

The Eq. (3) guides us on how to design a high-passing graph filter. However,
it still is not a trivial task, as the eigenvectors $E$ are difficult to
compute. Hence, we design a high-passing filter based on the existing low-
passing filter (i.e., GCNs(Kipf and Welling, 2016)). To achieve it, We first
hand on why GCNs are low-passing filter. The normalized Laplacian matrix
(Shuman et al., 2013) is defined as $L=I-D^{-\frac{1}{2}}AD^{-\frac{1}{2}}$.
Thus, the matrix $\hat{A}$ can be written as
$\hat{A}=I-(I-\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}})=I-\tilde{L}$.
As $\tilde{L}=\tilde{E}\tilde{\Lambda}\tilde{E}^{T}$, the $\hat{A}H$ can be
transformed to
$\hat{A}H=(I-\tilde{L})H=\tilde{E}(I-\tilde{\Lambda})\tilde{E}^{T}H$.
Comparing with Eq.(3), we can find that the graph kernel of one layer GCNs is
$g(\lambda)=1-\tilde{\lambda}$. With stacking of $K$ layer GCNs the graph
kernel is $g(\lambda)=(1-\tilde{\lambda})^{K}$. Moreover, the eigenvalues of
the normalized Laplacian matrix are $[0,2)$, and the eigenvalues of the
Laplacian matrix augmented by adding a self-loops are $[0,1.5)$. This implies
that GCN will amplify low-frequency signals and attenuate high-frequency
signals, as shown in Fig. 4 (a). Thus GCNs actually act as low-passing
filters.

After analyzing why GCNs is a low-passing graph filter, we design the high-
passing as :

(5)
$H^{l+1}=\sigma(LHW)=\sigma(D^{-\frac{1}{2}}(D-A)D^{-\frac{1}{2}}HW),H^{0}=X,$

where $A$ and $D$ are the adjacent matrix and degree matrix without
augmentation. The kernel filter function of the high-passing filter is
$g(\lambda)=\lambda$. As shown in Fig. 4 it will attenuate low-frequency
signals and amplify high-frequency signals.

###  4.2. Graph Encoder

In this section, we introduce how to encode the graph information with our
DGF. Generally, the user feedback in recommendation contains various noises.
Looking back to Fig. 1, we can see that for positive feedback there still
exists high-frequency information, which can be regarded as noise in the
graph. The low-passing filter usually is utilized to denoise. Hence, we first
use LGF on the positive feedback. It has two functions: (1) capturing what
users like by the positive feedback. (2) weaking the noise in the graph. It
can be formulated as:

(6) $H_{pos}^{l+1}={LGF}^{l}(H^{l},G+)$

After applying LGF to positive feedback, we apply HGF to negative feedback to
capture what users dislike. As analyzed before our HGF will amplify high-
frequency signals. It can be formulated as:

(7) $H_{neg}^{l+1}={HGF}^{l}(H^{l},G^{-})$

Lastly, we fusion the information aggregated from positive feedback and
negative feedback with an MLP layer, we formulate it as:

(8) $H^{l+1}=MLP(H_{pos}^{l+1}||H_{neg}^{l+1})$

###  4.3. Signed Graph Regularization

As discussed before, the graph neural network in recommendation may suffer
from representation degeneration problems, which means the graph embedding
shrinks to a piece of embedding space and loses representative power.
Intuitively, if we could make the embedding uniformly distributed in the
space, this issue could be alleviated. Uniform means that the average distance
of each node to all nodes should be as large as possible, so the uniform
constraint loss is defined as:

(9) $\displaystyle l_{uniform}=\sum^{|U|}_{u\in U}\log\sum^{|V|}_{v\in
V}e^{sim({u},v)/\tau},$

where $u$ and $v$ are the learned embedding of user and item, the $sim(\cdot)$
is the distance function, $\tau$ is temperature. In this work, we adopt cosine
similarity to measure the distance between two nodes.

However, complete uniform distribution still will lead to a loss of
representational capacity issue. Because the core of recommendation systems
lies in matching items that are similar to the user’s interests. This implies
that representations of some embedding should be similar. Furthermore, in
sign-aware recommendation, things become even more complex. Negative feedback
implies that some users and items are dissimilar, and their representations in
the space should be as far apart as possible. Therefore, to simultaneously
consider uniformity and similarity/dis-similarity between user and items, we
designed alignment loss, we formulated it as:

(10) $\displaystyle l_{alignment}$
$\displaystyle=-\sum_{(u,v)\in\mathcal{E}^{+}}\log
e^{sim({u},v)/\tau}+\sum_{(u,v)\in\mathcal{E}^{-}}\log e^{sim({u},v)/\tau}$
$\displaystyle=-\sum_{(u,v)\in\mathcal{E}^{+}}{sim}({u},v)/\tau+\sum_{(u,v)\in\mathcal{E}^{-}}{sim}({u},v)/\tau,$

where the $\mathcal{E}^{+}$ and $\mathcal{E}^{-}$ are positive edge sets and
negative edge set respectively. In fact, $l_{alignment}$ and $l_{uniform}$
engage in a two-player game. $l_{uniform}$ aims to make the learned embeddings
more uniform, while $l_{alignment}$ aims to make the learned embeddings more
extreme. Finally, the regularization loss is defined as:

(11) $\displaystyle l_{SGR}=l_{uniform}+l_{alignment}.$

In this work, we apply our SGR on the user and item embedding layer, while do
not apply it to the encoded node representation. Because we empirically found
that seting results in worse performance. It may be because the encoded
representation in the top layer is highly task-dependent, and forcing the
representation to be uniformly distributed may not be effective.

###  4.4. Loss Function

After getting the encoded representation of the user and item. We can predict
the value of $P(u_{i},v_{j})$. Following previous work(He et al., 2017; Kang
and McAuley, 2018; He et al., 2020), here we directly use the inner product of
$h_{u_{i}}$ and $h_{v_{j}}$ to calculate it:

(12) $\displaystyle p(u_{i},v_{j})=Sigmoid(h_{u_{i}}^{T}\cdot h_{v_{j}})$

In this paper, we adopt classical binary cross-entropy loss(Kang and McAuley,
2018; Huang et al., 2021) to optimize the model. The total loss is defined as:

(13) $\displaystyle l_{total}$ $\displaystyle=l_{task}+w\cdot l_{SGR},$
$\displaystyle l_{task}$
$\displaystyle=\sum^{N}yp(u_{i},v_{j})+(1-y)(1-p(u_{i},v_{j}),$

where $w$ is the weight of $l_{SGR}$.

##  5\. Experiment

Table 1. Results on recommendation task. All improvements are significant over baselines (t-test with p ${<}$ 0.05). Dataset | Model | GCN | GAT | SGCN | SBGNN | SBGCL | SIGRec | DFGNN | Improve
---|---|---|---|---|---|---|---|---|---
ML1M | MRR | 0.5085 | 0.5096 | 0.4866 | 0.5177 | 0.4707 | 0.5259 | 0.5433 | 3.31%
HIT@10 | 0.8132 | 0.8234 | 0.8041 | 0.8279 | 0.7932 | 0.8381 | 0.8576 | 2.33%
NDCG@10 | 0.2850 | 0.2882 | 0.2696 | 0.2972 | 0.2645 | 0.2948 | 0.3118 | 4.91%
HIT@50 | 0.9524 | 0.9592 | 0.9558 | 0.9587 | 0.9527 | 0.9608 | 0.9628 | 0.21%
NDCG@50 | 0.3013 | 0.3060 | 0.2912 | 0.3153 | 0.2877 | 0.3184 | 0.3311 | 3.99%
Arts | MRR | 0.0291 | 0.0296 | 0.0324 | 0.0441 | 0.0301 | 0.0296 | 0.0643 | 45.80%
HIT@10 | 0.0406 | 0.0427 | 0.0482 | 0.0746 | 0.0432 | 0.0409 | 0.1107 | 48.39%
NDCG@10 | 0.0230 | 0.0234 | 0.0254 | 0.0350 | 0.0236 | 0.0231 | 0.0521 | 48.86%
HIT@50 | 0.1038 | 0.1067 | 0.1093 | 0.1799 | 0.1101 | 0.1076 | 0.2284 | 26.96%
NDCG@50 | 0.0323 | 0.0326 | 0.0342 | 0.0516 | 0.0334 | 0.0328 | 0.0713 | 38.18%
GFood | MRR | 0.0421 | 0.0400 | 0.0409 | 0.0447 | 0.0372 | 0.0383 | 0.0475 | 6.26%
HIT@10 | 0.0642 | 0.0599 | 0.0615 | 0.0718 | 0.0597 | 0.0585 | 0.0756 | 5.29%
NDCG@10 | 0.0325 | 0.0314 | 0.0328 | 0.0363 | 0.0289 | 0.0306 | 0.0375 | 3.31%
HIT@50 | 0.1434 | 0.1273 | 0.1248 | 0.1510 | 0.1258 | 0.1189 | 0.1528 | 1.19%
NDCG@50 | 0.0444 | 0.0417 | 0.0423 | 0.0488 | 0.0390 | 0.0394 | 0.0493 | 1.02%
Kindle | MRR | 0.0381 | 0.0436 | 0.0264 | 0.0476 | 0.0299 | 0.0316 | 0.0703 | 47.69%
HIT@10 | 0.0851 | 0.0930 | 0.0548 | 0.0980 | 0.0621 | 0.0682 | 0.1569 | 60.10%
NDCG@10 | 0.0221 | 0.0263 | 0.0135 | 0.0315 | 0.0160 | 0.0170 | 0.0502 | 59.37%
HIT@50 | 0.2285 | 0.2299 | 0.1555 | 0.2452 | 0.1740 | 0.1928 | 0.3380 | 37.85%
NDCG@50 | 0.0416 | 0.0455 | 0.0253 | 0.0531 | 0.0303 | 0.0326 | 0.0796 | 49.91%
Yelp | MRR | 0.0378 | 0.0420 | 0.0448 | 0.0460 | 0.0422 | 0.0313 | 0.0511 | 11.09%
HIT@10 | 0.0819 | 0.0893 | 0.0925 | 0.0975 | 0.0902 | 0.0649 | 0.1112 | 14.05%
NDCG@10 | 0.0204 | 0.0231 | 0.0250 | 0.0257 | 0.0230 | 0.0164 | 0.0298 | 15.95%
HIT@50 | 0.2189 | 0.2382 | 0.2370 | 0.2590 | 0.2353 | 0.1917 | 0.2875 | 11.00%
NDCG@50 | 0.0377 | 0.0432 | 0.0442 | 0.0476 | 0.0423 | 0.0317 | 0.0543 | 14.08%

In this section, we aim to answer the following questions: (1) RQ1:How does
DFGNN perform compared with other state-of-the-art (SOTA) baselines on
recommendation task? (Sec. 5.3) (2) RQ2: How does DFGSN perform compared with
other state-of-the-art (SOTA) baselines on feedback type recognition task?
(Sec. 5.4) (3) RQ3: The proposed two modules in DFGNN indeed work? (Sec. 5.5)
(4) RQ4: How does our signed graph regulation loss help in the training? (Sec.
5.6)

###  5.1. Experimental Setting

Datasets In this paper, to evaluate the performance of the proposed DFGNN, we
conduct comprehensive experiments on classical ML1M, different categories of
Amazon Review datasets, and Yelp dataset. They are all real-world rating
datasets. each instance was assigned a rating from 1 to 5. we regard the
ratings that are below 3 as negative feedback and the ratings that are higher
than 3 as positive feedback. Due to the space limit, we detailly introduce and
analyze them in the AppendixA.1.

Evaluation Protocols. We evaluate the proposed model on two tasks called the
recommendation ranking task and feedback type recognition task. For the
recommendation ranking task, we adopt widely accepted ranking metrics(Kang and
McAuley, 2018; Wang et al., 2019): top-K hit rate (HIT@k), top-K (NDCG@k), and
MRR, with $k=\\{10and50\\}$. Feedback type recognition task can be regarded as
a binary classification task. Considering the imbalance between positive and
negative samples(as shown in Table.3), here we use classical AUC and F1-Macro
as evaluation metrics. It is necessary to note that we do not adopt accuracy,
precision, recall, etc., as metrics, since they are not applicable to
imbalanced datasets. For each task, we report the mean results with five
random runs.

Implement Details. We carefully tune all the baselines and compare them
fairly. Due to the space limit, we show the details in Appendix. A.1.1.

###  5.2. Baselines

To validate the performance of DFGNN, we compare our method with classical and
state-of-the-art unsigned/signed graph neural networks: (1) GCN(Kipf and
Welling, 2016), (2) GAT (Veličković et al., 2017), (3) SGCN(Derr et al., 2018)
(4)SBGNN (Huang et al., 2021), (5) SBGCL (Zhang et al., 2023) (6) SIGRec(Huang
et al., 2023). For unsigned GNNs, we only retain the positive feedback edge.
We implement our method by PyG111https://pytorch-
geometric.readthedocs.io/en/latest/index.html, which is a popular graph
learning framework. For baseline GCN, GAT, and SGCN we directly use the
implementation of PyG. For baseline SBGNN and SBGCL, we directly use the
author’s released codes. For SIGRec as there is no available open-sourced
code, we implement it by ourselves. They are detailly introduced in Appendix B

###  5.3. Performance on Recommendation

We show the results of our DFGNN in Table 1 on the recommendation task, which
aims to recommend items to users from a set of candidates. From the table, we
can see that:

(1) Compared with the unsigned graph neural network (i.e., GCN and GAT), the
signed graph neural network generally outperforms them across all datasets. It
demonstrates the importance of negative feedback. The signed graph neural
network both considers the positive feedback and negative feedback information
in the graph, thus achieving a better performance.

(2) We note that the SGCN generally performs worse than SBGCL. SBGCL is
specifically designed for bipartite graphs. It extends the balance theory tp
the butterfly structure in bipartite graphs. It matches the graph structure
characteristics in recommendation, which are a user-item bipartite graphs. In
contrast, SGCN is based on the balance theory with a triangle structure, which
may lead to its suboptimal performance.

(3) Compared with all baselines, our DFGNN significantly outperforms them on
all datasets, even with improvements of over 50% observed in some datasets.
This can be attributed to our special design for handling negative feedback.
Traditional GNNs are designed for homogeneous graphs and rely on the homophily
assumption, which assumes that nodes connected by edges are similar. While in
the sign-aware recommendation, users and items nodes connected by negative
feedback intuitively indicate dissimilarity. Our analysis of negative signals
from a frequency view also evaluates it. Based on this, we design dual-
frequency graph filters and signed graph regularization, which can break the
homophily assumption, and thus better capture the signal from negative
feedback.

Table 2. Results on feedback type recognition task. All improvements are significant over baselines (t-test with p ${<}$ 0.05). Dataset | Model | GCN | GAT | SGCN | SBGNN | SBGCL | SIGRec | DFGNN | Impro
---|---|---|---|---|---|---|---|---|---
ML1M | AUC | 0.8802 | 0.8838 | 0.8679 | 0.8877 | 0.8861 | 0.8665 | 0.8925 | 0.54%
F1-Macro | 0.7767 | 0.7791 | 0.7658 | 0.7800 | 0.7845 | 0.7411 | 0.7902 | 0.73%
Arts | AUC | 0.7366 | 0.7303 | 0.7337 | 0.7377 | 0.7324 | 0.7294 | 0.7577 | 2.71%
F1-Macro | 0.5904 | 0.5579 | 0.5777 | 0.6001 | 0.6035 | 0.5662 | 0.6095 | 0.99%
Kindle | AUC | 0.8591 | 0.8630 | 0.8389 | 0.8324 | 0.8352 | 0.8606 | 0.8691 | 0.71%
F1-Macro | 0.6318 | 0.6133 | 0.6500 | 0.6538 | 0.6632 | 0.6250 | 0.6892 | 3.92%
GFood | AUC | 0.7527 | 0.7057 | 0.7250 | 0.7371 | 0.7402 | 0.7484 | 0.7561 | 0.45%
F1-Macro | 0.6043 | 0.5610 | 0.5979 | 0.6047 | 0.5751 | 0.5970 | 0.6144 | 1.6%
Yelp | AUC | 0.7705 | 0.7770 | 0.7715 | 0.7572 | 0.7777 | 0.7725 | 0.7907 | 1.67%
F1-Macro | 0.6017 | 0.6633 | 0.6841 | 0.6636 | 0.6605 | 0.5860 | 0.6713 | -

###  5.4. Performance on Feedback Type Recognition

Figure 5. Ablation study of feedback type recognition task on Arts and GFood
dataset Figure 6. Ablation study of recommendation task on Arts and GFood
dataset

This section evaluates our DFGNN on the feedback type recognition task. Given
a user and item pair that the user has interacted with, this task aims to
predict whether the user will give positive/negative feedback. This task
focuses more on identifying what users dislike, as recommending items that
users find displeasing can significantly harm the user experience. We report
the experiment results in Table 2. From this table, we have:

(a) GCN on FTR task

(b) DFGNN on FTR task

(c) GCN on RS task

(d) DFGNN on RS task

Figure 7. The visualized embedding of GCN and FSGNN on two tasks. FTR means
Feedback Type Recognition task, and RS means Recommendation task

.

(1) We note that compared to the unsigned graph models, the signed graph
neural networks perform almost entirely better than them. This may be
attributed to the positive/negative feedback prediction is a harder task. In
fact, users only interact with an item when they have a certain level of
interest in it. Negative feedback may be provided later after that. Compared
to selecting items that users like from random negative items (task in
recommendation), this task is more difficult. In this scenario, a lack of
precise modeling of negative feedback signals may limit the improvement in
prediction accuracy.

(2) Compared to both signed graph neural networks and unsigned graph neural
networks, our DFGNN still outperforms all of them except F1-Macro in the Yelp
dataset. As analyzed before, positive/negative recognition is not trivial.
Though edges constructed based on co-negative relationships (e.g., $v_{1}$,
$v_{2}$ are both disliked by $u_{1}$) in the SBGNN and SBGCL intuitively can
utilize low-passing filters to capture negative feedback information. They
still struggle to capture high-passing signals between nodes with opposing
relationships (e.g., $u_{1}$ dislikes $v_{1}$). With the help of our dual-
frequency graph filters, our model can capture the high-frequency signal in
the negative feedback. Furthermore, the signed graph regularization further
alleviates the representation degeneration problem and helps the graph filters
work better.

###  5.5. Ablation Study

In this section, we conduct ablation experiments to evaluate the effectiveness
of the proposed module in our DFGNN. We transform our model into three
versions (1) Basic is the basic version of DFGNN, which only contains the
positive edge and low-passing GNN (2) Basic + LGF is the version that applies
low-passing graph filter on negative edges. (3) Basic + DGF, which adopts our
DGF to encode the signed graph (4) DFGNN is the full version of our model,
which both adopts our SGR and DGF to the signed graph. We conduct the ablation
experiment on the recommendation ranking and feedback type recognition task.
And show the results in Fig. 5 and Fig. 6. From the figures, we can find that
the proposed DGF and SGR can indeed improve the performance of two tasks. The
DGF both captures the high-frequency and low-frequency signals in two types of
feedback, based on this the SGR alleviates the over-smoothing problem.
Besides, We note that the Basic model generally outperforms the Basic+LGF on
the feedback type prediction task. However, in the recommendation task, the
opposite is observed. This is because the negative feedback still reflects
users’ interest (similarity), as they interact with rather than ignore them.
So it may benefit the recommendation task.

(a) GCN on FTR task (b) DFGNN on FTR task (c) Uniform of GCN and DFGNN

Figure 8. The Singular values and the uniform (average distance between all
nodes) of GCN and DFGNN on two tasks

.

###  5.6. Uniform Analysis

In this paper, based on the observation of the representation degenerative
problem, we propose SGR to alleviate it. To evaluate our SGR and further
understand how our SGR improves performance: (1) we visualize the learned node
embedding, and (2) we give out the normalized singular values of the embedding
encoded by GNNs and the average distance between nodes. Specifically, we
project the learned embedding to 2D by SVD for visualization. We use the
average L2 distance of normalized embedding to measure uniform. The larger the
value of distance, the more uniform the learned embedding is. We run the DFGNN
and GCN on the Arts dataset. The results are shown in Fig. 7(d) and Fig. 8(c).
By observing the Fig. 7(d) we can find that the DFGNN learned embedding is
uniform to that of GCN. Secondly, the embedding learned by feedback type
prediction task is more narrow. This implies the feedback type prediction task
is a more difficult task. Furthermore, by observing Fig. 8(c), We can see
singular values of the embeddings learned by our DFGNN decrease more slowly.
The distances of the learned embeddings are also larger. This indicates the
designed SGR indeed can help learned embedding to be more uniform. Thus, it
can improve the performance of our model.

##  6\. Related Work

GNN and Graph-based Recommendation Graph Neural networks(GNN) are studied to
model graph data and have achieved great success. GCNs(Bruna et al., 2013;
Defferrard et al., 2016; Kipf and Welling, 2016; Wu et al., 2019; Gao and Ji,
2019) firstly extend convolution operations to graph data. Further, GraphSage,
GAT, GIN et.al(Hamilton et al., 2017; Veličković et al., 2017; Xu et al.,
2018) abstract GNN to a message-passing schema and propose various information
aggregation strategies. Recommender system aims to recommend appropriate items
to users by utilizing user-item interaction information. Recently,
recommendation methods based on deep learning have achieved great success(Luo
et al., 2024; Huang et al., 2013; Covington et al., 2016; Xu et al., 2023; He
et al., 2017). Borrowing the promising performance, GNNs are introduced to
explore high-order interaction information contained in recommendations.
Pinsage(Ying et al., 2018) firstly applies graph neural network to the
recommendation. NGCF(Wang et al., 2019) extends collaborative filtering to
graph. LightGCN (He et al., 2020) removes the non-linear layer between GCN
sublayer to alleviate the over-parameter issues. despite the success of graph-
based recommendations, they are merely used to model positive feedback. How to
model negative feedback in graph-based recommendation remains underexplored.
In this work we focus on modeling negative feedback in graph-based
recommendation.

Signed graph neural network In the signed graph, there are two opposite edges
(e.g. reject/accept, trustworthy/trustworthy). Signed graph neural networks
are proposed to deal with such graph structures(Jung et al., 2020; Kim et al.,
2018; Yuan et al., 2017; Li et al., 2020; Huang et al., 2019). SGCN(Derr et
al., 2018) first introduces the GCN to the signed graph by utilizing the
balance theory. SBGNN(Huang et al., 2021) extends the balance theory to the
bipartite graph. SBGCL(Zhang et al., 2023) modified the graph contrastive
learning to the customized signed graph. However, most of them do not focus on
the recommendation scenario and adopt classical graph neural networks as
encoders, which heavily rely on homophily assumption. In this paper, we find
this assumption may not be invalid for recommendation.

##  7\. Conclusions

In this paper, we first essentially analyze the negative and positive feedback
from the graph signal frequency perspective. Based on our experimental
observations, we propose a novel Dual-frequency Graph Neural Network (DFGNN),
which models positive/negative feedback with a designed dual-frequency filter
and alleviates representation degeneration problem with a signed-graph
regularization loss. Extensive experiments validate the power of our FDGNN.

## References

  * (1)
  * Bruna et al. (2013) Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. 2013.  Spectral networks and locally connected networks on graphs.  _arXiv preprint arXiv:1312.6203_ (2013).
  * Cai and Wang (2020) Chen Cai and Yusu Wang. 2020.  A note on over-smoothing for graph neural networks.  _arXiv preprint arXiv:2006.13318_ (2020).
  * Covington et al. (2016) Paul Covington, Jay Adams, and Emre Sargin. 2016.  Deep neural networks for youtube recommendations. In _Proceedings of the 10th ACM conference on recommender systems_. 191–198.
  * Defferrard et al. (2016) Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016.  Convolutional neural networks on graphs with fast localized spectral filtering.  _Advances in neural information processing systems_ 29 (2016).
  * Derr et al. (2018) Tyler Derr, Yao Ma, and Jiliang Tang. 2018.  Signed graph convolutional networks. In _2018 IEEE International Conference on Data Mining (ICDM)_. IEEE, 929–934.
  * Elinas and Bonilla (2022) Pantelis Elinas and Edwin V Bonilla. 2022.  Addressing Over-Smoothing in Graph Neural Networks via Deep Supervision.  _arXiv preprint arXiv:2202.12508_ (2022).
  * Funk ([n. d.]) Simon Funk. [n. d.].  Funk’s original post.  https://sifter.org/~simon/journal/20061211.html
  * Gao and Ji (2019) Hongyang Gao and Shuiwang Ji. 2019.  Graph u-nets. In _international conference on machine learning_. PMLR, 2083–2092.
  * Hamilton et al. (2017) Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017.  Inductive representation learning on large graphs.  _Advances in neural information processing systems_ 30 (2017).
  * He et al. (2020) Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, YongDong Zhang, and Meng Wang. 2020.  LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Virtual Event, China) _(SIGIR ’20)_. Association for Computing Machinery, New York, NY, USA, 639–648.  https://doi.org/10.1145/3397271.3401063
  * He et al. (2017) Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017.  Neural collaborative filtering. In _Proceedings of the 26th international conference on world wide web_. 173–182.
  * Huang et al. (2021) Junjie Huang, Huawei Shen, Qi Cao, Shuchang Tao, and Xueqi Cheng. 2021.  Signed bipartite graph neural networks. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 740–749.
  * Huang et al. (2019) Junjie Huang, Huawei Shen, Liang Hou, and Xueqi Cheng. 2019.  Signed graph attention networks. In _Artificial Neural Networks and Machine Learning–ICANN 2019: Workshop and Special Sessions: 28th International Conference on Artificial Neural Networks, Munich, Germany, September 17–19, 2019, Proceedings 28_. Springer, 566–577.
  * Huang et al. (2023) Junjie Huang, Ruobing Xie, Qi Cao, Huawei Shen, Shaoliang Zhang, Feng Xia, and Xueqi Cheng. 2023.  Negative can be positive: Signed graph neural networks for recommendation.  _Information Processing & Management_ 60, 4 (2023), 103403.
  * Huang et al. (2013) Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013.  Learning deep structured semantic models for web search using clickthrough data _(CIKM ’13)_. Association for Computing Machinery, New York, NY, USA, 2333–2338.  https://doi.org/10.1145/2505515.2505665
  * Jeunen (2019) Olivier Jeunen. 2019.  Revisiting offline evaluation for implicit-feedback recommender systems. In _Proceedings of the 13th ACM Conference on Recommender Systems_. 596–600.
  * Jung et al. (2020) Jinhong Jung, Jaemin Yoo, and U Kang. 2020.  Signed graph diffusion network.  _arXiv preprint arXiv:2012.14191_ (2020).
  * Kang and McAuley (2018) Wang-Cheng Kang and Julian McAuley. 2018.  Self-attentive sequential recommendation. In _2018 IEEE international conference on data mining (ICDM)_. IEEE, 197–206.
  * Kim et al. (2018) Junghwan Kim, Haekyu Park, Ji-Eun Lee, and U Kang. 2018.  Side: representation learning in signed directed networks. In _Proceedings of the 2018 world wide web conference_. 509–518.
  * Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016.  Semi-supervised classification with graph convolutional networks.  _arXiv preprint arXiv:1609.02907_ (2016).
  * Li et al. (2020) Yu Li, Yuan Tian, Jiawei Zhang, and Yi Chang. 2020.  Learning signed network embedding via graph attention. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 34. 4772–4779.
  * Luo et al. (2024) Huishi Luo, Fuzhen Zhuang, Ruobing Xie, Hengshu Zhu, Deqing Wang, Zhulin An, and Yongjun Xu. 2024.  A survey on causal inference for recommendation.  _The Innovation_ 5, 2 (2024), 100590.
  * Seo et al. (2022) Changwon Seo, Kyeong-Joong Jeong, Sungsu Lim, and Won-Yong Shin. 2022.  SiReN: Sign-aware recommendation using graph neural networks.  _IEEE Transactions on Neural Networks and Learning Systems_ (2022).
  * Shuman et al. (2013) David I Shuman, Sunil K Narang, Pascal Frossard, Antonio Ortega, and Pierre Vandergheynst. 2013.  The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains.  _IEEE signal processing magazine_ 30, 3 (2013), 83–98.
  * Veličković et al. (2017) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017.  Graph attention networks.  _arXiv preprint arXiv:1710.10903_ (2017).
  * Wang et al. (2019) Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019.  Neural graph collaborative filtering. In _Proceedings of the 42nd international ACM SIGIR conference on Research and development in Information Retrieval_. 165–174.
  * Wu et al. (2019) Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. 2019.  Simplifying graph convolutional networks. In _International conference on machine learning_. PMLR, 6861–6871.
  * Wu et al. (2022) Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. 2022.  Graph neural networks in recommender systems: a survey.  _Comput. Surveys_ 55, 5 (2022), 1–37.
  * Xu et al. (2018) Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2018.  How powerful are graph neural networks?  _arXiv preprint arXiv:1810.00826_ (2018).
  * Xu et al. (2023) Yongjun Xu, Fei Wang, Zhulin An, Qi Wang, and Zhao Zhang. 2023.  Artificial intelligence for science—bridging data to wisdom.  _The Innovation_ 4, 6 (2023).
  * Ying et al. (2018) Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. 2018.  Graph convolutional neural networks for web-scale recommender systems. In _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_. 974–983.
  * Yuan et al. (2017) Shuhan Yuan, Xintao Wu, and Yang Xiang. 2017.  SNE: signed network embedding. In _Advances in Knowledge Discovery and Data Mining: 21st Pacific-Asia Conference, PAKDD 2017, Jeju, South Korea, May 23-26, 2017, Proceedings, Part II 21_. Springer, 183–195.
  * Zhang et al. (2018) Quangui Zhang, Longbing Cao, Chengzhang Zhu, Zhiqiang Li, and Jinguang Sun. 2018.  Coupledcf: Learning explicit and implicit user-item couplings in recommendation for deep collaborative filtering. In _IJCAI International Joint Conference on Artificial Intelligence_.
  * Zhang et al. (2023) Zeyu Zhang, Jiamou Liu, Kaiqi Zhao, Song Yang, Xianda Zheng, and Yifei Wang. 2023.  Contrastive learning for signed bipartite graphs. In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1629–1638.
  * Zhou et al. (2020) Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, and Ji-Rong Wen. 2020.  S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. In _CIKM_.

##  Appendix A Experimental Setting

###  A.1. Dataset

In this paper, to evaluate the performance of the proposed DFGNN, we conduct
comprehensive experiments on classical ML1M, different categories of Amazon
Review datasets, and Yelp dataset. Amazon Review datasets are collected from
the Amazon shopping website. Each instance consists of a user’s rating for an
item, ranging from 1 to 5. we select three categories dataset, they are
ArtsCrafts and Sewing,Grocery and Gourmet Food, and  Kindle Store. For all
datasets, we regard the ratings that are below 3 as negative feedback, and the
ratings that are higher than 3 as positive feedback. Following previous
works(Kang and McAuley, 2018; Zhou et al., 2020), we discard the users and
items with less than five interactions. For each dataset we randomly select
70% instances as training data, 10% instances as validation data, and 20%
instances as testing data. The detailed statistical information of those
datasets is shown in Tab. 3

Table 3. Statistics of all datasets. Dataset | # user | # item | # instance | # negative rate
---|---|---|---|---
Arts | 56123 | 22847 | 413554 | 5.85%
Kindle | 139729 | 98821 | 2020496 | 5.68%
GFood | 127399 | 41317 | 990360 | 8.74%
Yelp | 207453 | 92422 | 3139963 | 19.94%
ML1M | 6040 | 3668 | 739012 | 22.16%

####  A.1.1. Implement Details

. For all the models the embedding size of items and users is set to 64 and
the batch size is set to 512. The GNN layer K is set to 2 in our method. For
all methods. We optimize all models with the Adam Optimizer and carefully
search for the hyper-parameters of all baselines. To avoid overfitting, we
adopt the early stop strategy with a patience of 20 epochs. For the
recommendation ranking task, we adopt the random negative sampling strategy
with a 1:1 sampling rate. We conduct a grid search for hyper-parameters. We
search for models’ learning rates among
$\\{1e^{-}2,3e^{-3},1e^{-3},3e^{-4},1e^{-4},3e^{-5}\\}$.

##  Appendix B Baselines

To validate our IDP, we conduct a comprehensive comparison. Specifically, we
compare our method with the following baselines: (1) GCN(Kipf and Welling,
2016), (2) GAT (Veličković et al., 2017), (3) SGCN(Derr et al., 2018) (4)SBGNN
(Huang et al., 2021), (5) SBGCL (Zhang et al., 2023).

  * •

GCN (Kipf and Welling, 2016),which is a classical graph convolution network.
GCN adopts the first-order Chebyshev polynomial to construct graph filters.

  * •

GAT (Veličković et al., 2017), which is a classical attention-based GNN. It
adopts the attention mechanism to aggregation neighbor nodes.

  * •

SGCN (Derr et al., 2018), which is a classical signed graph convolution
network. It carefully designs a message-passing mechanism based on balanced
theory.

  * •

SBGNN (Huang et al., 2021), which is designed for bipartite signed graph. It
extends balance theory to bipartite graphs by butterfly structure.

  * •

SBGCL (Zhang et al., 2023),which is a contrastive based signed graph neural
network. It improved the graph contrastive learning to adapt to signed graphs.

We implement our method by PyG, which is a popular graph learning framework.
For baseline GCN, GAT, and SGCN we directly use the implementation of PyG. For
baseline SBGNN and SBGCL, we directly use the author’s released codes.


# Model-inspired Deep Learning for Light-Field Microscopy with Application to
Neuron Localization

###### Abstract

Light-field microscopes are able to capture spatial and angular information of
incident light rays. This allows reconstructing 3D locations of neurons from a
single snap-shot. In this work, we propose a model-inspired deep learning
approach to perform fast and robust 3D localization of sources using light-
field microscopy images. This is achieved by developing a deep network that
efficiently solves a convolutional sparse coding (CSC) problem to map Epipolar
Plane Images (EPI) to corresponding sparse codes. The network architecture is
designed systematically by unrolling the convolutional Iterative Shrinkage and
Thresholding Algorithm (ISTA) while the network parameters are learned from a
training dataset. Such principled design enables the deep network to leverage
both domain knowledge implied in the model, as well as new parameters learned
from the data, thereby combining advantages of model-based and learning-based
methods. Practical experiments on localization of mammalian neurons from
light-fields show that the proposed approach simultaneously provides enhanced
performance, interpretability and efficiency.

Index Terms—  Light-field microscopy, model-inspired deep learning, algorithm
unrolling, convolutional sparse coding.

##  1 Introduction

Studying fundamental structural and dynamic properties of neuronal networks
critically depends on advanced optical microscopy imaging systems. In contrast
to conventional optical microscopy that records only lateral information as a
2D projection of light rays, light-field microscopy (LFM) [1, 2, 3] has has
emerged as a promising 3D optical imaging technique. LFM is able to
simultaneously gather both position and angular information of the incident
light rays arriving at the sensor with a single snapshot [4]. This is achieved
by inserting a microlens array (MLA) at the native imaging plane and by moving
the imaging sensor to the back focal plane of the microlenses [1]. With
benefits of high light efficiency and fast imaging speed, LFM demonstrates
great potential of observing structures and dynamics at the cellular
resolution across whole brain volumes [5, 6, 7].

Fig. 1: Schematic for microlens-based LFM system and typical imaging
situations, e.g. source in-focus and out-of-focus. Different depths (i.e.
axial positions) cause different patterns in LFM images and constructed EPIs.

However, imaging with LFM also has limitations, such as reduced spatial
resolution, slow reconstruction, image degradation due to light scattering in
deep layers. To address these issues, some work focuses on improving spatial
resolution via, e.g. 3D deconvolution [2, 3, 5], while some work detects the
3D positions of sources directly by performing sparse decomposition in phase-
space [8, 6], or by performing convolutional sparse coding with respect to a
synthetic dictionary [9, 10]. Different from existing methods, we propose a
model-inspired deep learning approach for fast and robust 3D localization of
sources from a single light-field image.

In the past decade, learning-based methods, in particular deep learning [11],
have demonstrated enhanced performance in terms of speed and accuracy over
conventional model-based methods. However, deep neural networks are usually
designed empirically and their structures lack interpretability, which is a
prominent shortcoming. In the past few years, algorithm unrolling/unfolding
[12, 13, 14, 15] has emerged as a promising technique to design deep networks
in a more principled way by unrolling iterative methods. It bridges model-
based methods with learning-based methods, and this leads to enhanced
interpretability and better generalization ability of deep networks.

In this work, we develop a network using algorithm unrolling to localize
neurons in tissues from LFM images. The network performs convolutional sparse
coding on input epipolar plane images (EPI) [16, 17, 18], a type of spatio-
angular feature constructed from light-field images, and output sparse codes
which indicates depth positions.

Fig. 2: Proposed CISTA-net. An input EPI $\mathbf{X}$ containing two sources
is mapped to sparse codes $\mathbf{Z}$ whose support indicates depths. Cross-
entropy loss between the estimation $\mathbf{Z}$ and the label
$\mathbf{Z}^{\star}$ is computed and back-propagated to update parameters.

##  2 Model-inspired Network Design

To design the network in a principled and systematic way, we first present the
convolutional sparse coding (CSC) model [9] and a convolutional version of
Iterative Shrinkage and Thresholding Algorithm (ISTA) [19]. Then, we unroll
the Convolutional ISTA method to design the network architecture, so that the
network, referred to as CISTA-net, imitates the processing flow that maps EPIs
into sparse codes. Finally, the parameters of the network are learned from a
well-designed training dataset.

Convolutional Sparse Coding model

The method developed in [9] leverages the fact that neurons localized in space
are relatively similar to point-like sources. Given raw light-field microscopy
images, we use the method proposed in [9] to perform calibration, conversion,
and purification to get an array of clean sub-aperture images. Then, the
horizontal and vertical positions, i.e. the x and y coordinates of sources can
be detected from the central sub-aperture image by finding those pixels
brighter than a specified threshold.

Depth detection along the axial coordinate is not trivial as it requires the
proper leverage of the angular information. To this end, EPIs are often used
to simultaneously reveal the spatial and angular information captured in 4D
light-field data, as shown in Figure 1. To infer depths from EPIs, one may
construct an EPI dictionary that consists of basic EPIs related to sources
which are associated to different depths. Such an EPI dictionary can be used
to decompose an input EPI into a sparse linear combination of basic EPIs.
Finding this sparse decomposition can be modelled as a convolutional sparse
coding problem (1)(See also [9]).

Specifically, given an observed EPI $\mathbf{X}\in\mathbb{R}^{\Theta\times N}$
where $\Theta$ denotes angular dimension, i.e. number of pixels in a single
microlens, and $N$ denotes spatial dimension, i.e. number of microlens, we aim
to obtain a series of sparse codes
$\\{\mathbf{z}_{m}\in\mathbb{R}^{\Theta\times N}\\}_{m=1}^{M}$ via
decomposition with respect to a predefined EPI dictionary
$\\{\mathbf{d}_{m}\in\mathbb{R}^{\theta\times n}\\}_{m=1}^{M}$ ($M$ denotes
number depths to be covered and $\theta\leq\Theta$, $n\leq N$). For
simplicity, all the variables have been vectorized into column vectors and
then such decomposition can be formulated as:

$\begin{array}[]{cl}\underset{\\{\mathbf{z}_{m}\\}}{\min}&\frac{1}{2}\left\|\mathbf{X}-\sum\limits_{m=1}^{M}\mathbf{d}_{m}*\mathbf{z}_{m}\right\|_{2}^{2}+\lambda\,\sum\limits_{m=1}^{M}\|\mathbf{z}_{m}\|_{1}\end{array}$
(1)

where, $\mathbf{d}_{m}$ is the $m$-th element of the EPI dictionary and
$\|\cdot\|_{1}$ is the $\ell_{1}$ norm that summarizes the absolute value of
non-zero elements.

Convolutional ISTA algorithm. Problem (1) can be solved using the
Convolutional ISTA algorithm which consists of the following computationally-
efficient projected gradient descent iterations:

$\small\begin{split}\mathbf{z}_{k}^{(i+1)}&=\mathcal{T}_{\lambda}\left(\mathbf{z}_{k}^{(i)}-\gamma\mathbf{d}_{k}^{\top}*\sum\limits_{m=1}^{M}\mathbf{d}_{m}*\mathbf{z}_{m}^{(i)}+\mathbf{d}_{k}^{\top}*\mathbf{X}\right)\end{split}$
(2)

where $k=1,\cdots,M$, the superscript (i) denotes the iteration index, and
$\gamma$ is the step size. $\mathcal{T}_{\lambda}(\cdot)$ is the element-wise
soft-thresholding function, defined as
$\mathcal{T}_{\lambda}(x)=\text{sign}(x)\cdot(|x|-\lambda)_{+}$ with an
appropriate $\lambda\geq 0$, which is used to enforce sparsity on sparse
codes. Equation (2) can also be rewritten as:

$\small\begin{split}\mathbf{Z}^{(i+1)}&=\mathcal{T}_{\lambda}\left(\mathbf{Z}^{(i)}-\gamma\mathbf{D}^{\top}\circledast\sum\mathbf{D}\circledast\mathbf{Z}^{(i)}+\mathbf{D}^{\top}\circledast\mathbf{X}\right)\end{split}$
(3)

where dictionary $\mathbf{D}\in\mathbb{R}^{\theta\times n\times M}$ consists
of an array of $M$ $\theta\times n$ basic EPIs,
$\mathbf{D}^{\top}\in\mathbb{R}^{n\times\theta\times M}$ is the transposed
version of $\mathbf{D}$. Symbol $\circledast$ denotes convolution along the
first and second dimension, i.e. spatio-angular dimension, and the symbol
$\sum$ denotes summation of convolution results along the third dimension.
Once the sparse codes are obtained, the largest values of each sparse code
$\mathbf{z}_{m}$ are selected and combined into a new vector, similar to the
GlobalMaxPooling operation commonly used in deep learning. Then the non-zero
support of the new sparse vector leads to the depth.

CISTA-net design. Our CISTA-net aims to mimic above processing flow. So (3) is
adapted to (4) which serves as the unit module to build the network. Then, the
entire network architecture can be obtained by concatenating the unit module
multiple times, which accounts for algorithm unrolling. So we have:

$\small\begin{split}\mathbf{Z}^{(i+1)}&=ReLU\left(\mathbf{Z}^{(i)}-\mathbf{S}^{(i)}\circledast\mathbf{Z}^{(i)}+\mathbf{W}^{(i)}\circledast\mathbf{X}+\lambda\right)\end{split}$
(4)

where trainable filters $\mathbf{S}^{(i)}$ and $\mathbf{W}^{(i)}$ replace the
variable $\gamma\mathbf{D}^{\top}\circledast\sum\mathbf{D}$ and
$\gamma\mathbf{D}^{\top}$, respectively. $ReLU$ (i.e. Rectified Linear Unit)
with trainable non-negative bias $\lambda$ play the role of the soft-
thresholding. The overall architecture of the CISTA-net is shown in Figure 2.

Even though the unrolling/unfolding idea has been investigated in the
literature, the proposed unrolling network has some particular characteristics
and appealing attributes. In particular, operation
$\mathbf{d}_{m}*\mathbf{z}_{m}^{(i)}$ in (2) or
$\mathbf{D}\circledast\mathbf{Z}^{(i)}$ in (3) actually performs convolution
between each pair of EPI $\mathbf{d}_{m}$ and sparse codes $\mathbf{z}_{m}$.
Such pair-wise convolution inspires us to exploit, instead of plain
convolution, depth-wise separable convolution [20] which reduces the number of
network parameters. Besides, the subtraction operation
$\mathbf{Z}^{(i)}-\mathbf{S}^{(i)}\circledast\mathbf{Z}^{(i)}$ leads to a
local skip connection which evokes the residual module used in the ResNet
[21]. Finally, transformed $\mathbf{X}$ added in each layer facilitates
information propagation from the first layer directly to each hidden layer,
thereby alleviating information loss. This structure reveals certain
resemblance to dense connectivity module used in DenseNet [22], even though
not each layer is receiving a connection from all preceding layers. Even
though some of these architecture modules have been used in modern neural
networks, we here provide a new and interesting perspective to elaborate how
they can naturally derive from a well-designed model and corresponding
iterative method, rather than pure intuition commonly used in general network
design.

Rather than sticking to Model (3) rigidly, we also employ some customized
modifications to further enhance the capability of the network. For example,
the size of filters $\mathbf{S}^{(i)}$ and $\mathbf{W}^{(i)}$ increases across
different layers so that the receptive field increases gradually to facilitate
capture of both local details and global structure. In addition, a fully-
connected layer followed by sigmoid activation is added after the
GlobalMaxPooling layer to perform non-linear transformation. Such slight
departures from the original convolutional ISTA algorithm enable extended
representation capability of the network.

##  3 Experiment

In this section, we evaluate the performance of CISTA-net on the 3D
localization task. We also compare our approach with the phase-space based
method (Phase-Space for short) [8, 6] and convolutional sparse coding based
method (CSC for short) [9] on light-field microscopy data obtained from
scattering specimens – genetically encoded fluorophore in mouse brain tissues,
as shown in Fig. 5 (a).

The raw light-field microscopy images were captured by systematically changing
the axial distance between the specimens and the objective lens of LFM as in
[9]. Therefore, each light-field microscopy image captures a 3D volume at a
specified depth. All the experiments were conducted in a computer equipped
with an Intel hexa-core i7-8700U<EMAIL_ADDRESS>with 28GB of memory and a NVIDIA
GTX 1080 Ti GPU.

Fig. 3:  Illustration to the construction of soft-labels.

Fig. 4:  Convergence curve of training and validation loss.

(a) LFM images (above) and corresponding EPIs (bottom) for depth of -15, 0, 15
$\mathrm{u}\mathrm{m}$.

(b) Estimation of sparse codes and depth using phase-space method [8, 6].

(c) Estimation of sparse codes and depth using CSC method [9].

(d) Estimation of sparse codes and depth using CISTA-net.

Fig. 5:  Visual comparison of estimated sparse codes and depth detection using
different methods. In each subfigure, the red dashed line indicates estimated
sparse codes and the red crosses indicate estimated depths. The blue line
indicates the soft-labels, and the blue circle indicates the groundtruth
depths.

(a) Localization performance of phase-space method [8, 6]. RMSE for x, y, z
position detection is 4.05, 5.48, 3.41 $\mathrm{\SIUnitSymbolMicro m}$,
respectively.

(b) Localization performance of CSC approach [9]. RMSE for x, y, z position
detection is 1.78, 2.94, 1.14 $\mathrm{\SIUnitSymbolMicro m}$, respectively.

(c) Localization performance of the proposed CISTA-net. RMSE for x, y, z
position detection is 1.60, 1.98, 0.82 $\mathrm{\SIUnitSymbolMicro m}$,
respectively.

Fig. 6:  Performance of localizing neurons using three different methods,
including phase-space method [8, 6], CSC method [9], and the proposed method.
The depth of neurons varies from -18 $\mathrm{\SIUnitSymbolMicro m}$ to 36
$\mathrm{\SIUnitSymbolMicro m}$.

Training settings. Figure 3 shows how we construct the labelled training
datasets. Since the designed CISTA-net is expected to output sparse codes
where the positions of non-zero elements (i.e. support) indicate the depths
corresponding to input EPIs, it needs to be trained on labelled data in order
to learn the mapping from an input EPI to the corresponding sparse codes.
However, coefficient values in sparse codes are unknown. To handle this issue,
we treat the task as a multi-class, multi-label classification task where the
support of the sparse codes indicates target classes/categories while the
coefficient values indicate probability or confidence of the input signal
falling into each class.111 Note, if the task is treated as a regression task,
the output of the network will be a number that denotes the continuous depth,
thus it can not handle the case with multiple neurons in the region of
interest as this gives multiple lines in an EPI.  In this way, we only need
weakly annotated sparse codes with roughly estimated coefficient values. In
addition, we found that EPIs corresponding to adjacent depths tend to have
similar patterns, and thus exhibit high coherence and lead to group sparsity
in CSC. Based on this observation, we modified the sparse codes by convolving
them with a Gaussian kernel so that the groundtruth non-zero support is
extended to neighbouring areas that cover adjacent depths. We call the
support-extended sparse codes "soft-labels" in comparison with the sparse
codes with exact support, namely "hard-labels". Soft-labels give some training
benefits by incorporating data correlation as guidance information and enforce
group sparsity for the output of network. Since our task is regarded as a
multi-class, multi-label classification task, the loss function is set to be
binary cross-entropy between output sparse codes and the soft-labels.

Training and testing results. We trained the network on 100,000 EPI samples
with a maximum of 100 epochs using the adaptive moment estimation (ADAM)
optimizer. Figure 4 shows that the training and validation errors converge
almost within 40 epochs. Figure 5 and 6 compares the proposed CISTA-net with
Phase-Space [8, 6] and CSC [9] on 55 neuron specimens covering a depth range
from -18 to 36 $\mathrm{u}\mathrm{m}$. It shows that our approach outputs more
structured sparse codes than competing methods owing to supervised learning
with the soft-labels. Sparse codes of high quality facilitate depth detection
via finding non-zero support centers more accurately. In contrast, Phase-Space
[8, 6] and CSC [9] tend to find more small non-zero elements, which may
interfere with subsequent depth detection, as shown in Figure 5. Further
results in Figure 6 confirms that our approach outperforms the competing
methods and gives the best performance with least errors on x, y, z position
detection. Furthermore, CISTA-net is more than 10000$\times$ faster than the
competing methods in terms of average time for processing a single EPI with
the same CPU, as shown in Table 1. CISTA-net computational efficiency can
enable fast source localization during LFM imaging of living brain tissues,
providing information to guide and modify the experimental protocol.

Table 1: Average time cost of different methods. |  phase-space [8, 6] |  CSC [9] | CISTA-net
---|---|---|---
time cost | 3.59 s | 3.61 s |  0.216 $\times 10^{-3}$ s

##  4 Conclusion

We proposed a model-inspired deep network to perform 3D localization using
light-field microscopy images. The network architecture is designed
systematically by unrolling the convolutional ISTA method used to solve
convolutional sparse coding problems. The network parameters are learned from
the training dataset. In this way, the designed network naturally inherits
domain knowledge and also learns new features from the data, demonstrating
better interpretability and higher speed. The proposed methodology may provide
inspiration on how to take better advantage of the unrolling idea to design
deep networks with advanced attributes.

## References

  * [1] M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” in _ACM Transactions on Graphics (TOG)_ , vol. 25, no. 3. ACM, 2006, pp. 924–934.
  * [2] M. Broxton, L. Grosenick, S. Yang, N. Cohen, A. Andalman, K. Deisseroth, and M. Levoy, “Wave optics theory and 3-D deconvolution for the light field microscope,” _Optics express_ , vol. 21, no. 21, pp. 25 418–25 439, 2013\.
  * [3] N. Cohen, S. Yang, A. Andalman, M. Broxton, L. Grosenick, K. Deisseroth, M. Horowitz, and M. Levoy, “Enhancing the performance of the light field microscope using wavefront coding,” _Optics express_ , vol. 22, no. 20, pp. 24 817–24 839, 2014.
  * [4] M. Levoy and P. Hanrahan, “Light field rendering,” in _Proceedings of the 23rd annual conference on Computer graphics and interactive techniques_. ACM, 1996, pp. 31–42.
  * [5] T. Nöbauer, O. Skocek, A. J. Pernía-Andrade, L. Weilguny, F. M. Traub, M. I. Molodtsov, and A. Vaziri, “Video rate volumetric Ca2+ imaging across cortex using seeded iterative demixing (sid) microscopy,” _Nature methods_ , vol. 14, no. 8, p. 811, 2017.
  * [6] N. C. Pégard, H.-Y. Liu, N. Antipa, M. Gerlock, H. Adesnik, and L. Waller, “Compressive light-field microscopy for 3D neural activity recording,” _Optica_ , vol. 3, no. 5, pp. 517–524, 2016.
  * [7] P. Quicke, C. L. Howe, P. Song, H. V. Jadan, C. Song, T. Knöpfel, M. Neil, P. L. Dragotti, S. R. Schultz, and A. J. Foust, “Subcellular resolution three-dimensional light-field imaging with genetically encoded voltage indicators,” _Neurophotonics_ , vol. 7, no. 3, p. 035006, 2020.
  * [8] H.-Y. Liu, E. Jonas, L. Tian, J. Zhong, B. Recht, and L. Waller, “3D imaging in volumetric scattering media using phase-space measurements,” _Optics express_ , vol. 23, no. 11, pp. 14 461–14 471, 2015.
  * [9] P. Song, H. V. Jadan, C. L. Howe, P. Quicke, A. J. Foust, and P. L. Dragotti, “3d localization for light-field microscopy via convolutional sparse coding on epipolar images,” _IEEE Transactions on Computational Imaging_ , vol. 6, pp. 1017–1032, 2020.
  * [10] P. Song, H. V. Jadan, P. Quicke, C. L. Howe, A. J. Foust, and P. L. Dragotti, “Location estimation for light field microscopy based on convolutional sparse coding,” in _Imaging and Applied Optics 2019_. Optical Society of America, 2019, p. MM2D.2.
  * [11] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, _Deep learning_. MIT press Cambridge, 2016, vol. 1.
  * [12] K. Gregor and Y. LeCun, “Learning fast approximations of sparse coding,” in _Proceedings of the 27th international conference on international conference on machine learning_ , 2010, pp. 399–406.
  * [13] X. Chen, J. Liu, Z. Wang, and W. Yin, “Theoretical linear convergence of unfolded ista and its practical weights and thresholds,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 9061–9071.
  * [14] Y. Li, M. Tofighi, J. Geng, V. Monga, and Y. C. Eldar, “Efficient and interpretable deep blind image deblurring via algorithm unrolling,” _IEEE Transactions on Computational Imaging_ , vol. 6, pp. 666–681, 2020\.
  * [15] X. Deng and P. L. Dragotti, “Deep convolutional neural network for multi-modal image restoration and fusion,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [16] R. C. Bolles, H. H. Baker, and D. H. Marimont, “Epipolar-plane image analysis: An approach to determining structure from motion,” _International Journal of Computer Vision_ , vol. 1, no. 1, pp. 7–55, 1987.
  * [17] S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, “The lumigraph,” in _Proceedings of the 23rd annual conference on Computer graphics and interactive techniques_. ACM, 1996, pp. 43–54.
  * [18] S. Vagharshakyan, R. Bregovic, and A. Gotchev, “Light field reconstruction using shearlet transform,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 40, no. 1, pp. 133–147, 2018.
  * [19] A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algorithm for linear inverse problems,” _SIAM journal on imaging sciences_ , vol. 2, no. 1, pp. 183–202, 2009.
  * [20] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” _arXiv preprint arXiv:1704.04861_ , 2017.
  * [21] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [22] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4700–4708.