# Neural Scaling Laws on Graphs

Jingzhe Liu  Haitao Mao  Zhikai Chen  Tong Zhao  Neil Shah  Jiliang Tang

###### Abstract

Deep graph models (e.g., graph neural networks and graph transformers) have
become important techniques for leveraging knowledge across various types of
graphs. Yet, the scaling properties of deep graph models have not been
systematically investigated, casting doubt on the feasibility of achieving
large graph models through enlarging the model and dataset sizes. In this
work, we delve into neural scaling laws on graphs from both model and data
perspectives. We first verify the validity of such laws on graphs,
establishing formulations to describe the scaling behaviors. For model
scaling, we investigate the phenomenon of scaling law collapse and identify
overfitting as the potential reason. Moreover, we reveal that the model depth
of deep graph models can impact the model scaling behaviors, which differ from
observations in other domains such as CV and NLP. For data scaling, we suggest
that the number of graphs can not effectively metric the graph data volume in
scaling law since the sizes of different graphs are highly irregular. Instead,
we reform the data scaling law with the number of edges as the metric to
address the irregular graph sizes. We further demonstrate the reformed law
offers a unified view of the data scaling behaviors for various fundamental
graph tasks including node classification, link prediction, and graph
classification. This work provides valuable insights into neural scaling laws
on graphs, which can serve as an essential step toward large graph models.

Machine Learning, ICML

##  1 Introduction

Neural scaling laws (Rosenfeld et al., 2019; Kaplan et al., 2020; Abnar et
al., 2021) quantitatively describe how model performance grows with the scale
of training. There are two basic forms: data scaling law (how the performance
changes with training dataset size) and model scaling law (how the performance
changes with model size). Neural scaling laws guide the construction of large
models from two perspectives: (1) They enable us to predict model performance
on large-scale training by extrapolating from small-scale runs with lower
computational budgets. For instance, the performance of GPT-4 on coding
problems fits accurately with the prediction of the scaling law (Achiam et
al., 2023). (2) They enable us to identify better model architectures through
small-scale training, typically selecting those with the largest performance
gains for a given computational overhead. Such laws have played an
instrumental role in the recent successes of large models in natural language
processing (NLP) and computer vision (CV) domains (Brown et al., 2020; Touvron
et al., 2023; Chowdhery et al., 2023; Zhai et al., 2022; Dehghani et al.,
2023).

Despite these impacts on NLP and CV, no similar revolution has occurred in the
graph machine learning domain (Zhang et al., 2023), given its unique
difficulties. The investigation of neural scaling laws on graphs faces new
challenges from both data and model perspectives. (1) From the data
perspective: Graphs are irregular and diverse. Both the number of nodes and
edges of a graph can vary dramatically. As an example, in the ogbg-ppa dataset
of $158k$ graphs (Szklarczyk et al., 2019), the number of nodes in each graph
ranges from $50$ to $300$, and the number of edges from $54$ to $18,096$,
indicating significant variability across the dataset. The variability in size
suggests that the current metric (graphs) insufficiently represents the data
volume, hinting at the need for a new graph scaling law metric. (2) From the
model perspective: Besides the model weights considered in the existing
scaling law, deep graph models also need to consider the influence of the non-
parametric aggregations, i.e., whether graph models with different numbers of
aggregation layers could exhibit distinct scaling behaviors. This issue is not
yet examined.

In this work, we make a first attempt to address these unique challenges and
investigate the model scaling law and the data scaling law on graphs. While
the main focus of this work is on the graph classification tasks, we also show
that our settings and results can be extrapolated to node classification and
link prediction tasks. Experimental observations reveal neural scaling laws
across different models and dataset sizes for graph learning tasks. Our
contributions can be summarized as follows:

  * •

For the model scaling law:  (i) We examine the model scaling law for graph
learning, which reveals that performance scales with model size, but collapses
when the model becomes excessively large, likely due to overfitting. This
suggests graph learning models are likely to be data hungry. (ii) From the
perspective of architecture, we find that deep graph models of different
numbers of layers will exhibit different model scaling behaviors, contrasting
with the conclusions in CV and NLP domains.

  * •

For the data scaling law:  (i) Regarding the size variability problem, we
reveal that the number of edges is a better metric than the number of graphs
when applying the data scaling law in graph classification tasks. To support
this argument, we demonstrate that models trained on datasets with identical
graph counts but different total numbers of edges exhibit distinctive scaling
behaviors. (ii) We further extend the data scaling law to node classification
and link prediction tasks, demonstrating the generality of using the number of
edges as the data metric.

##  2 Related Work

Functional forms of neural scaling laws. Neural scaling laws give the
quantitative relationship between the model performance and three main
factors: the number of model parameters $N$, the size of the training dataset
$D$, and the amount of computation $C$. The relationship is generally
described as a power-law, which has widely been applied in the domains of
image classification (Hestness et al., 2017; Henighan et al., 2020; Sharma &
Kaplan, 2022; Zhai et al., 2022), generative language modeling (Kaplan et al.,
2020; Henighan et al., 2020; Hoffmann et al., 2022), and natural language
translation (Gordon et al., 2021; Ghorbani et al., 2021; Bansal et al., 2022;
Fernandes et al., 2023).

Using the test error $\epsilon$ as a metric for model performance, Hestness et
al. (2017) proposes the basic functional form of neural scaling laws as
follows:

$\epsilon=aX^{-b}+\epsilon_{\infty}$ (1)

The variable $X$ represents any of the parameters $N$, $D$, or $C$. The
positive constants $a$ and $b$ govern the rate of error reduction as a
function of $X$, while $\epsilon_{\infty}>0$ specifies the minimum achievable
error. Equation 1 is appropriate for regression tasks but is ill-defined for
classification tasks, since the error rate $\epsilon_{r}$ remains finite for
models with zero training. To fix this problem, Zhai et al. (2022) proposes a
modified function:

$\epsilon_{r}=a(X+c)^{-b}+\epsilon_{\infty}$ (2)

where a positive shift coefficient $c$ is added to avoid having infinite
$\epsilon_{r}$ when $X$ equals to zero. In this work, we will base our
following investigations of graph classification tasks on this equation.

The relationship between $N$, $D$ and overfitting. The negative exponent $-b$
in Equation 2 indicates that the performance gain will be diminishing if only
one of the parameters $N$ and $D$ is increased. To break this bottleneck, the
dataset size and the model size need to be enlarged together. Specifically,
Kaplan et al. (2020) find that the model size and dataset size should increase
together as $D\sim N^{\alpha}$ ($0<\alpha<1$) to attain optimal return for
compute budgets. It is also observed that overfitting will likely occur if $N$
surpasses $D^{1/\alpha}$ significantly, which leads to negative performance
gain.

Graph learning and scaling. The graph is unique in representing non-Euclidean
data with irregular and flexible structures, e.g., social networks (Yanardag &
Vishwanathan, 2015), molecules (Gilmer et al., 2017), and protein networks
(Szklarczyk et al., 2019). Despite the popularity of graph machine learning,
neural scaling laws on graphs remain underexplored. Some works have looked
into the neural scaling laws for molecule representation learning (Chen et
al., 2023), but the exploration is limited to the data and model types. It is
unclear how the conclusion can be generalized to other graphs. To the best of
our knowledge, we are the first to investigate neural scaling laws for the
general graph domain.

##  3 Preliminaries

(a) Model scaling on pcqm4mv2, $R^{2}=0.97$.

(b) Data scaling on pcqm4mv2, $R^{2}=0.98$.

Figure 1: The model and data scaling behaviors of GIN on pcqm4mv2. For model
scaling, the color corresponds to the training data fraction; for data
scaling, the color corresponds to the number of model parameters. The lower
the mean square error is, the better the model performance is. The high
$R^{2}$ values indicate that the scaling law curves fit well with the
empirical values.

(a) Model scaling on ogbg-ppa, $R^{2}=0.98$.

(b) Data scaling on ogbg-ppa, $R^{2}=0.98$.

Figure 2: The model and data scaling behaviors of GIN on ogbg-ppa. For model
scaling, the color corresponds to the training data fraction; for data
scaling, the color corresponds to the number of model parameters. The higher
the accuracy is, the better the model performance is. The high $R^{2}$ values
indicate that the scaling law curves fit well with the empirical values.

Orgnizations. The main focus of this work is on the two basic forms of neural
scaling laws for the graph domain, i.e., the modeling scaling law and the data
scaling law. To consolidate our conclusion, our investigations focus on the
well-studied supervised graph learning while leaving the under-studied
unsupervised learning tasks (Sun et al., 2022) as one future work. The
following sections will be arranged as follows: Section 4 verifies the
validity of the basic forms of neural scaling laws on the graphs. Section 5
investigates deeper into the model scaling law of graph models. We observe the
model scaling collapse phenomenon when graph models are excessively large and
identify overfitting as the potential reason. Moreover, we reveal that model
depth can affect the model scaling behaviors, which is a unique phenomenon in
graph domains. In Section 6, we propose the number of edges is a suitable data
metric when applying the data scaling law on graphs. To support the argument,
we exhibit that models would have distinctions in scaling behavior when
trained on datasets with different numbers of edges. Furthermore, we present
that using the number of edges as the data metric could provide a unified data
scaling law for various graph learning tasks, e.g., node classification, link
prediction, and graph property prediction.

Experimental Setup. To ensure the applicability of our conclusions within the
graph domain, our investigations cover different types of models and datasets.
We investigate two major types of deep graph learning models, i.e., graph
neural networks (GNNs) and graph transformers. We explore GCN (Kipf & Welling,
2016) and GIN (Xu et al., 2018) for GNNs, and SAT (Chen et al., 2022), and
GraphGPS (Rampášek et al., 2022) for graph transformers. Our datasets (Morris
et al., 2020; Hu et al., 2020, 2021) cover the most common graph domains
including social networks (reddit-multi-12k, ogbl-collab), molecules (ogbg-
molhiv,ogbg-molpcba,pcqm4mv2), protein networks (ogbg-ppa), and citation
networks (ogbn-arxiv). Moreover, our experiments are also conducted on three
common graph tasks, i.e. graph property prediction, node classification, and
link prediction, though graph property prediction is the focus. Nonetheless,
accuracy rather than error rate is adopted in graph classification tasks as a
metric. To switch the error rate $\epsilon_{r}$ in Equation 2 to accuracy, we
apply the relation $\text{Accuracy}=1-\text{ErrorRate}$ and obtain the
following equation:

$s=-a(X+c)^{b}+s_{\infty}$ (3)

where $s$ is the test score corresponding to specific metrics and
$s_{\infty}=1-\epsilon_{\infty}$. Equation 3 will be used in the following
sections to fit the scaling law on classification tasks. The details of
datasets and experimental settings can be found in Appendix A.

##  4 The Validity of Basic Scaling Laws on Graphs

(a) Scaling behaviors on pcqm4mv2, $R^{2}=0.98$.

(b) Scaling behaviors on ogbg-ppa, $R^{2}=0.86$.

Figure 3: The comprehensive scaling behaviors of GIN on pcqm4mv2 and ogbg-ppa
datasets. For pcqm4mv2, the color corresponds to the value of mean square
error; for ogbg-ppa, the color corresponds to accuracy. The surfaces predicted
by the neural scaling law fit well with the empirical values on both datasets,
as indicated by the high $R^{2}$ values.

We start our analysis by examining the basic forms of neural scaling laws in
the graph domain. In other words, we aim to explore how the model performance
changes with the dataset size and model size. Hence, we first evaluate the
performance of the models across various sizes and training dataset scales. To
simplify the experimental settings, we vary the model size by increasing the
model width while fixing the model depth, and the effect of model depth will
be further explored in Section 5. To obtain subsets with different sizes, we
randomly subsample the original training set by different ratios. Then, we fit
the scaling law Eq. 1 and 3 to the empirical performance data using the least
square method. To measure the fitting quality, the coefficient of
determination $R^{2}$ is adopted as a metric. In this subsection, we focus on
GIN and leave the results of GCN in Appendix B. Our experiments are conducted
on two large graph property prediction datasets: pcqm4mv2 and ogbg-ppa.

We conduct experiments for both model scaling and data scaling behaviors on
the graph regression dataset pcqm4mv2. For regression tasks, the performance
is measured by mean square error (MSE), hence we use Eq. 1 to fit the
empirical MSEs. For model scaling, we fix the number of layers for the model
to $6$ and vary the model width from $64$ to $1,024$. The experiments are
repeated for different data fractions chosen from
$\left[0.1,0.2,0.3,0.4,0.5,0.7,1\right]$. The results, shown in Figure 1(a),
depict each model’s MSE as dots, with dashed lines representing fitted scaling
law curves and colors indicating specific training data fractions. We make the
following observations from Figure 1(a): (1) the MSE tends to decrease when
increasing the model size and fixing the dataset size. (2) the MSE gradually
decreases and approaches a lower bound $\epsilon$ asymptotically as the model
becomes larger. The scaling curves corresponding to larger dataset sizes would
have smaller $\epsilon$, i.e., higher upper bound of the model performance.
The observations in Figure 1(b) exhibit similar patterns when scaling up data
size while fixing the model size. Generally speaking, both model scaling
curves in Figure 1(a) and data scaling curves in 1(b) fit well to empirical
values with high $R^{2}$s, indicating that Eq. 1 could be leveraged to
describe the neural scaling laws for graph regression tasks.

The scaling behaviors on the ogbg-ppa dataset are illustrated in Figure 2. The
way of varying dataset sizes and model sizes is similar to that of pcqm4mv2.
However, since the task on ogbg-ppa is graph classification, we use accuracy
as the performance metric and apply Eq. 3 to fit the scaling behaviors.
Observations from Figure 2 align with those from Figure 1, suggesting that
Equation 3 can describe the scaling behavior for graph classification tasks.

Moreover, similar to Rosenfeld et al. (2019), the model scaling law and data
scaling law formulations can be combined into a comprehensive form for
regression tasks:

$\epsilon=a_{1}D^{b_{2}}+a_{2}N^{b_{2}}+\epsilon_{\infty}$ (4)

For classification tasks, the comprehensive form is:

$s=-a_{1}(D+c_{1})^{b_{1}}-a_{2}(N+c_{2})^{b_{2}}+s_{\infty}$ (5)

We verify the comprehensive forms by fitting the empirical values to Equation
4 and 5. The experimental results are shown in Figure 3, where the high
$R^{2}$ values suggest that the model scaling and data scaling laws can be
well unified by Equation 4 and 5.

Observation 1. The model and data scaling behaviors in the graph domain can be
described by neural scaling laws.

The above observations suggest that the scaling behaviors for graph property
prediction tasks follow neural scaling laws and can be correctly predicted by
Eq. 1 and 3. Thus, it is possible to attain expected performance on graph
tasks by enlarging the model and dataset size, exhibiting the potential of
building large graph models.

(a) Model scaling on ogbg-molpcba

(b) Training losses of models

(c) Validation losses of models

Figure 4: (a) The empirical performance of larger models falls below the
prediction of model scaling law. (b)(c) The training losses and validation
losses of models with varying sizes. The color corresponds to the number of
model parameters. The model with larger sizes suffers more serious overfitting

Figure 5: More training data can mitigate the overfitting problem. The model
size is fixed, while the two curves correspond to different data fractions of
the original ogbg-molpcba training set.

##  5 Graph-specific Phenomena for Model Scaling Laws

In this section, we will dive deeper into specific phenomena of model scaling
on graphs. In Section 5.1, we observe the collapse of model scaling in certain
situations and discuss the potential reason. In Section 5.2, we identify that
models with different depths could exhibit distinct model scaling behaviors.
These observations enable us to have a better understanding of the uniqueness
of model scaling law on graphs, providing important instructions for
developing large graph models.

###  5.1 Model Scaling Collapse

Despite the overall success, the scaling law could fail in certain situations.
In particular, we find that the model scaling law could fail when the model
size is too large relative to the training set size, causing the phenomenon
called model scaling collapse. To illustrate this phenomenon, we consider an
example of scaling up 5-layer GIN on ogbg-molpcba dataset. Similar to the
setting in Section 4, we vary the model size by changing the model width while
fixing the training set size. Figure 4(a) displays the empirical accuracy of
models across sizes alongside the model scaling curve. At small model sizes,
the accuracy values fit with the model scaling curve well. However, when the
number of parameters is around $10^{7}$, the empirical values start dropping.
Thus, the collapse of performance would make the predictions of the scaling
law fail, which is referred to as model scaling collapse.

The model scaling collapse poses difficulty in building large graph models
following the model scaling law. Thus, it is essential to explore why the
collapse happens and how to mitigate it. In this work, we conjecture that one
of the potential reasons for the scaling law collapse is overfitting. To dive
deeper into the problem, Figure 4(b) and 4(c) present the training loss and
validation loss curves, respectively. All the training losses in Figure 4(b)
keep dropping in the training process. The larger the model is, the smaller
the corresponding training loss will be. Nonetheless, the validation losses of
models with more than $10^{7}$ parameters start increasing after several
epochs as shown in Figure 4(c), which indicates overfitting. Moreover, the
models whose performance falls behind the scaling law predictions suffer more
serious overfitting. For instance, models of 7.4M, 13M, and 20M parameters
overfit the most in Figure 4(c), and they are the three models on which model
scaling law fails in Figure 4(a). Based on these observations, overfitting can
be one potential reason for the model scaling collapse. For more results
between the model size and overfitting, please refer to Appendix C.

(a) Model scaling of GCN with different layers.

(b) Model scaling of GIN with different layers.

Figure 6: Model scaling behaviors of GNNs with varying model depths. The color
corresponds to the model layer. Both GIN and GCN exhibit distinct scaling
behaviors with different model depths.

(a) Model scaling of SAT with different layers.

(b) Model scaling of GraphGPS with different layers.

Figure 7: Model scaling behaviors of graph transformers with varying model
depths. The color corresponds to the model layer. Both SAT and GraphGPS
exhibit distinct scaling behaviors with different model depths.

To eliminate the effect of model scaling collapse, it is crucial to overcome
the overfitting. Kaplan et al. (2020) suggest that overfitting would happen if
the model size is too large for the training set. To investigate whether more
training data can mitigate overfitting, we train a model of fixed number
parameters on different fractions of the training set and illustrate the
validation loss curves in Figure 5. The model trained on $10$ percent of the
original ogbg-molpcba dataset suffers serious overfitting as the validation
loss curve suggests. However, the overfitting is significantly mitigated when
the data fraction is raised to $50$ percent, which indicates the effectiveness
of enlarging the training set. Another evidence is that we have not observed
overfitting or model scaling collapse in models of similar sizes ($10^{6}$
parameters) trained on larger datasets, i.e., pcqm4mv2 and ogbg-ppa in Section
4.

Observation 2. The model scaling collapses in the graph domain often happen
due to the overfitting.

In graph tasks, the model scaling collapse often occurs with overfitting,
which is because the graph dataset sizes are much smaller compared to those in
CV or NLP. Overall, the model scaling collapse in graph tasks suggests that
the current graph datasets are small for the deep graph models, leaving the
models data hungry. Furthermore, it indicates that a crucial step toward the
large graph model is to enlarge the size of datasets.

###  5.2 The Effect of Model Depth

Another unique question in the graph domain is whether the model depth would
affect the model scaling law, or more specifically, the scaling law
coefficients in Equation 3. Kaplan et al. (2020) find that in the NLP domain,
the model depth does not have effects on the scaling coefficients, i.e., the
scaling curves for transformers of different numbers of model layers will
overlap. Therefore, conducting preliminary experiments to determine the
optimal number of layers for large language models (LLMs) is not required,
which effectively narrows the hyperparameter search space. Since the deep
graph models have different architectures from vanilla transformers, it is
desired to study whether the above conclusion still holds.

(a) ogbg-molhiv

(b) reddit-multi-12k

Figure 8: Performance curves of GIN with varying sizes trained on two subsets.
The simple subset and the complex subset have the same number of graphs but
different numbers of edges. The shade corresponds to the performance variance
of repeated experiments.

Figure 9: Performance curves of GIN of varying sizes trained on two subsets.
The simple subset and the complex subset have the same number of edges but
different numbers of graphs. The performance differences between the two
curves are much smaller than the curves in Figure 8(a), suggesting that the
number of edges is a better data metric than the number of graphs.

To ensure the generalization of our investigations, we choose two
representative GNNs (GCN & GIN) and two powerful graph transformers (SAT &
GraphGPS). For these models, we choose the layer number in $[3,4,5]$ and fix
it but change the model width to attain varying model sizes. All the models
are trained on the whole ogbg-ppa training set and then evaluated for model
performance. The model scaling curves of models with different model depths
are presented in Figure 6 and 7. Observations can be made as follows: (1) The
scaling behaviors of models with different numbers of layers are distinct. For
each model, the shape of the scaling law curves would change when the model
depth changes, exhibiting different increasing speeds and upper bounds for
performance. (2) The optimal number of layers for different models could be
different. For GCN and GraphGPS, models with $3$ layers have the best
performance, but for GIN the optimal layer number is $4$, and for SAT is $5$.
Similar observations on more datasets can be found in Appendix D.

Observation 3. Deep graph models with varying depths will have different model
scaling behaviors.

Current observations support that there is no one-size-fits-all model depth
for existing graph learning models. The optimal number of model layers could
be task-specific and model-specific. We emphasize the necessity of preliminary
experiments for the optimal number of model layers before scaling up the model
parameters. Notably, we do not have any principle guidance helping to
automatically select the best model layer so far. Such mystery could be
attributed to the complicated non-parametric aggregation design in different
GNNs with various expressiveness (Anonymous, 2024) and inductive bias
(Battaglia et al., 2018). Therefore, it is likely that there is no principle
across models. Instead, there could be some rules specific to tasks and models
without a consensus. Therefore, such analysis is out of scope targeting a
general principle across different model architectures and different tasks. We
leave the deeper investigation on it as the future work.

##  6 Reforming Data Scaling Law on Graphs

In this section, we address the irregular-size problems of graphs for the data
scaling law. Specifically, we re-examine data metrics in previous studies and
reveal the disadvantage of the number of graphs as the data metric in the data
scaling law. Instead, we identify the total number of edges in the training
set as a better metric with experimental evidence. Furthermore, the data
scaling law with the number of edges as the data metric can be naturally
extended to node classification and link prediction tasks, potentially
providing a unified view of data scaling behaviors in the graph domain.

(a) Node classification on ogbn-arxiv, $R^{2}=0.97$.

(b) Link prediction on ogbl-collab, $R^{2}=0.99$.

Figure 10: Both node classification and link prediction follow the data
scaling law with the total edge number as the data metric.

###  6.1  Number of Edges as Data Metric

In previous sections, the data metric in scaling laws is the subsampling ratio
of total graphs, i.e., how many graphs in the whole training set are used
(Rosenfeld et al., 2019; Chen et al., 2023). Nonetheless, using the number of
graphs as the metric neglects the size irregularity of graphs. For instance, a
graph with $10$ edges and a graph with $1,000$ edges are treated equivalently
when the number of graphs is the metric, even though they could differ
remarkably in training costs and impacts on model performance improvement.

To illustrate the disadvantages of adopting the number of graphs as the data
metric, we first divide the training sets of one dataset into two subsets as
follows: (1) We first arrange all graph samples of class $i$ into a list
$\mathcal{G}_{i}$ in an ascending order of their numbers of edges. (2) We
choose the first half of all $\mathcal{G}_{i}$ as the “simple” training
subset, and then the second half as the “complex” subset. Thus, these two
subsets have the same number of graphs while the graphs in the complex subset
have more edges. The validation and test sets remain as the original. Then we
train GIN of varying sizes on these two subsets and report the performance on
ogbg-molhiv and reddit-multi-12k in Figure 8. Results on more datasets can be
found in Appendix E.

If the number of graphs can accurately metric the data volume, the curves of
simple and complex subsets in Figure 8 should have small differences or even
overlap with each other. However, their performance curves exhibit
considerable differences, indicating that the two subsets contain different
amounts of data. For comparison, we attain the two subsets in another way: the
simple subset has the first $m$ samples in $\mathcal{G}_{i}$, and the complex
subset has the remaining $n$ samples, where the two subsets contain the same
number of edges. It is easy to validate that $m>n$. Then we train models of
varying sizes on these two subsets and report their performance on ogbg-molhiv
in Figure 9. Compared to Figure 8(a), the curves in Figure 9 have much smaller
differences, which indicates that the total number of edges could be a better
metric than the number of graphs.

Observation 4. The number of edges is a better data metric for data scaling
law compared to the number of graphs.

In addition, using the total number of edges as the data metric in scaling
laws has two more advantages: (1) Knowing the GNN architecture and the number
of edges in a dataset, the compute cost of training can be directly
calculated, which enables us to build connections between data scaling law and
compute scaling law as in Kaplan et al. (2020). The detailed explanation is in
Appendix F. (2) Furthermore, using the number of edges as the data metric
potentially provides a unified way to describe the data scaling behaviors of
node classification and link prediction tasks, which will be thoroughly
studied in the next subsection.

###  6.2 Results on Node Classification and Link Prediction

There is only one graph in most of the node classification and link prediction
datasets, thus it is infeasible to use the number of graphs to measure the
training data volume. Nonetheless, the total number of edges is able to serve
as the data metric for these tasks. Moreover, with the total number of edges
as the data metric, we can potentially attain a unified data scaling law on
graphs by directly extending Eq. 3 to node classification and link prediction
tasks. To verify this, we conduct experiments on the data scaling behaviors on
both node classification and link prediction tasks. For node classification,
we train a $3$-layer GraphSAGE (Hamilton et al., 2017) on ogbn-arxiv, where
the hidden vector size is set to $128$. For link prediction, we train a
$3$-layer SEAL (Zhang et al., 2021) on ogbl-collab, where the model takes one-
hop neighbors of the target node pairs as input and its hidden vector size is
set to $256$. We take the total number of edges on which models are trained as
the training set size for both tasks. The results are demonstrated in Figure
10. Similar to graph classification tasks, Eq. 3 fits well to the data scaling
behaviors, as evidenced by the high $R^{2}$ values.

Observation 5. Node classification, link prediction, and graph classification
tasks could follow the same form of data scaling law with the number of edges
as data metric.

##  7 Conclusions

In this work, we take the first step to investigate neural scaling laws on
graphs. In particular, we focus on two basic forms of neural scaling laws: the
model scaling law and the data scaling law. We first verify the formulations
that can describe both scaling behaviors in the graph domain. For the model
scaling law, we identify and investigate some graph-specific phenomena,
revealing different observations from CV and NLP domains. For the data scaling
law, we reform it by adopting the number of edges as the data metric and
extend the reformed version to node classification and link prediction tasks.
Our findings shed light on the scaling behaviors of deep graph models and can
provide valuable instructions for developing large graph models.

Limitations. Next, we discuss the limitations of this work, which could be
further explored as future works. (1) There is no large graph dataset of
similar sizes as the large CV or NLP datasets. We conduct our experiments on
the largest academic graph datasets that we can attain. (2) Since there has
not been a well-established pretraining task for graph learning, we limit our
investigation to the supervised learning on graphs. (3) In our exploration, we
assume that all node (edge) features are in a unified feature space. We do not
consider heterogeneous feature problems in this work.

## Impact Statements

In this paper, we investigate neural scaling laws on graphs, which describe
how the model performance will grow with increasing model and data sizes.
Neural scaling laws can provide instructions for building large graph models,
predicting the costs of computing and data labeling to achieve the expected
performance. Moreover, it may incentivize more investments in data collecting
and even the construction of larger graph datasets for various tasks. We
anticipate that our findings will serve as essential guidance for the
development of large graph models.

## References

  * Abnar et al. (2021) Abnar, S., Dehghani, M., Neyshabur, B., and Sedghi, H.  Exploring the limits of large scale pre-training.  _arXiv preprint arXiv:2110.02095_ , 2021.
  * Achiam et al. (2023) Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Aghajanyan et al. (2023) Aghajanyan, A., Yu, L., Conneau, A., Hsu, W.-N., Hambardzumyan, K., Zhang, S., Roller, S., Goyal, N., Levy, O., and Zettlemoyer, L.  Scaling laws for generative mixed-modal language models.  _arXiv preprint arXiv:2301.03728_ , 2023.
  * Alabdulmohsin et al. (2023) Alabdulmohsin, I., Zhai, X., Kolesnikov, A., and Beyer, L.  Getting vit in shape: Scaling laws for compute-optimal model design.  _arXiv preprint arXiv:2305.13035_ , 2023.
  * Alabdulmohsin et al. (2022) Alabdulmohsin, I. M., Neyshabur, B., and Zhai, X.  Revisiting neural scaling laws in language and vision.  _Advances in Neural Information Processing Systems_ , 35:22300–22312, 2022.
  * Anonymous (2024) Anonymous.  Beyond weisfeiler-lehman: A quantitative framework for GNN expressiveness.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=HSKaGOi7Ar.
  * Bahri et al. (2021) Bahri, Y., Dyer, E., Kaplan, J., Lee, J., and Sharma, U.  Explaining neural scaling laws.  _arXiv preprint arXiv:2102.06701_ , 2021.
  * Bansal et al. (2022) Bansal, Y., Ghorbani, B., Garg, A., Zhang, B., Cherry, C., Neyshabur, B., and Firat, O.  Data scaling laws in NMT: The effect of noise and architecture.  In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 1466–1482. PMLR, 17–23 Jul 2022.  URL https://proceedings.mlr.press/v162/bansal22b.html.
  * Battaglia et al. (2018) Battaglia, P. W., Hamrick, J. B., Bapst, V., Sanchez-Gonzalez, A., Zambaldi, V., Malinowski, M., Tacchetti, A., Raposo, D., Santoro, A., Faulkner, R., et al.  Relational inductive biases, deep learning, and graph networks.  _arXiv preprint arXiv:1806.01261_ , 2018.
  * Brown et al. (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chen et al. (2022) Chen, D., O’Bray, L., and Borgwardt, K.  Structure-aware transformer for graph representation learning.  In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 3469–3489. PMLR, 17–23 Jul 2022.  URL https://proceedings.mlr.press/v162/chen22r.html.
  * Chen et al. (2023) Chen, D., Zhu, Y., Zhang, J., Du, Y., Li, Z., Liu, Q., Wu, S., and Wang, L.  Uncovering neural scaling laws in molecular representation learning.  _arXiv preprint arXiv:2309.15123_ , 2023.
  * Chowdhery et al. (2023) Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al.  Palm: Scaling language modeling with pathways.  _Journal of Machine Learning Research_ , 24(240):1–113, 2023.
  * Dehghani et al. (2023) Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A. P., Caron, M., Geirhos, R., Alabdulmohsin, I., et al.  Scaling vision transformers to 22 billion parameters.  In _International Conference on Machine Learning_ , pp. 7480–7512. PMLR, 2023.
  * Fernandes et al. (2023) Fernandes, P., Ghorbani, B., Garcia, X., Freitag, M., and Firat, O.  Scaling laws for multilingual neural machine translation.  _arXiv preprint arXiv:2302.09650_ , 2023.
  * Ghorbani et al. (2021) Ghorbani, B., Firat, O., Freitag, M., Bapna, A., Krikun, M., Garcia, X., Chelba, C., and Cherry, C.  Scaling laws for neural machine translation.  _arXiv preprint arXiv:2109.07740_ , 2021.
  * Gilmer et al. (2017) Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E.  Neural message passing for quantum chemistry.  In _International conference on machine learning_ , pp. 1263–1272. PMLR, 2017.
  * Gordon et al. (2021) Gordon, M. A., Duh, K., and Kaplan, J.  Data and parameter scaling laws for neural machine translation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 5915–5922, 2021.
  * Hamilton et al. (2017) Hamilton, W., Ying, Z., and Leskovec, J.  Inductive representation learning on large graphs.  _Advances in neural information processing systems_ , 30, 2017.
  * Henighan et al. (2020) Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al.  Scaling laws for autoregressive generative modeling.  _arXiv preprint arXiv:2010.14701_ , 2020.
  * Hestness et al. (2017) Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y.  Deep learning scaling is predictable, empirically.  _arXiv preprint arXiv:1712.00409_ , 2017.
  * Hoffmann et al. (2022) Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al.  Training compute-optimal large language models.  _arXiv preprint arXiv:2203.15556_ , 2022.
  * Hu et al. (2020) Hu, W., Fey, M., Zitnik, M., Dong, Y., Ren, H., Liu, B., Catasta, M., and Leskovec, J.  Open graph benchmark: Datasets for machine learning on graphs.  _Advances in neural information processing systems_ , 33:22118–22133, 2020.
  * Hu et al. (2021) Hu, W., Fey, M., Ren, H., Nakata, M., Dong, Y., and Leskovec, J.  Ogb-lsc: A large-scale challenge for machine learning on graphs.  _arXiv preprint arXiv:2103.09430_ , 2021.
  * Kaplan et al. (2020) Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_ , 2020.
  * Kingma & Ba (2014) Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kipf & Welling (2016) Kipf, T. N. and Welling, M.  Semi-supervised classification with graph convolutional networks.  In _International Conference on Learning Representations_ , 2016.
  * Morris et al. (2020) Morris, C., Kriege, N. M., Bause, F., Kersting, K., Mutzel, P., and Neumann, M.  Tudataset: A collection of benchmark datasets for learning with graphs.  _arXiv preprint arXiv:2007.08663_ , 2020.
  * Rampášek et al. (2022) Rampášek, L., Galkin, M., Dwivedi, V. P., Luu, A. T., Wolf, G., and Beaini, D.  Recipe for a general, powerful, scalable graph transformer.  _Advances in Neural Information Processing Systems_ , 35:14501–14515, 2022.
  * Rosenfeld et al. (2019) Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., and Shavit, N.  A constructive prediction of the generalization error across scales.  _arXiv preprint arXiv:1909.12673_ , 2019.
  * Sharma & Kaplan (2022) Sharma, U. and Kaplan, J.  Scaling laws from the data manifold dimension.  _The Journal of Machine Learning Research_ , 23(1):343–376, 2022.
  * Sorscher et al. (2022) Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., and Morcos, A.  Beyond neural scaling laws: beating power law scaling via data pruning.  _Advances in Neural Information Processing Systems_ , 35:19523–19536, 2022.
  * Sun et al. (2022) Sun, R., Dai, H., and Yu, A. W.  Does gnn pretraining help molecular representation?  _Advances in Neural Information Processing Systems_ , 35:12096–12109, 2022.
  * Szklarczyk et al. (2019) Szklarczyk, D., Gable, A. L., Lyon, D., Junge, A., Wyder, S., Huerta-Cepas, J., Simonovic, M., Doncheva, N. T., Morris, J. H., Bork, P., et al.  String v11: protein–protein association networks with increased coverage, supporting functional discovery in genome-wide experimental datasets.  _Nucleic acids research_ , 47(D1):D607–D613, 2019.
  * Touvron et al. (2023) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Wang et al. (2020) Wang, K., Shen, Z., Huang, C., Wu, C.-H., Dong, Y., and Kanakia, A.  Microsoft academic graph: When experts are not enough.  _Quantitative Science Studies_ , 1(1):396–413, 2020.
  * Xu et al. (2018) Xu, K., Hu, W., Leskovec, J., and Jegelka, S.  How powerful are graph neural networks?  In _International Conference on Learning Representations_ , 2018.
  * Yanardag & Vishwanathan (2015) Yanardag, P. and Vishwanathan, S.  Deep graph kernels.  In _Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining_ , pp. 1365–1374, 2015.
  * Zhai et al. (2022) Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L.  Scaling vision transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 12104–12113, June 2022.
  * Zhang et al. (2021) Zhang, M., Li, P., Xia, Y., Wang, K., and Jin, L.  Labeling trick: A theory of using graph neural networks for multi-node representation learning.  In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), _Advances in Neural Information Processing Systems_ , volume 34, pp. 9061–9073. Curran Associates, Inc., 2021.  URL https://proceedings.neurips.cc/paper_files/paper/2021/file/4be49c79f233b4f4070794825c323733-Paper.pdf.
  * Zhang et al. (2023) Zhang, Z., Li, H., Zhang, Z., Qin, Y., Wang, X., and Zhu, W.  Large graph models: A perspective.  _arXiv preprint arXiv:2308.14522_ , 2023.

##  Appendix A Emperiment Settings.

Here we introduce the training datasets used and training hyperparameters in
the experiments.

Datasets. The basic situations of the datasets we use are listed in the Tabel
1 and 2 below.

Table 1: Graph Property Prediction Datasets Statistics. | pcqm4mv2 | ogbg-ppa | ogbg-molhiv | ogbg-molpcba | reddit-multi-12k
---|---|---|---|---|---
Domains | Molecule | Protein | Molecule | Molecule | Social
#Graphs | 3,746,619 | 158,100 | 41,127 | 437,929 | 11,929
#Total Nodes | 52,970,652 | 38,481,540 | 1,048,739 | 11,386,154 | 4,669,130
#Total Edges | 54,546,813 | 358,270,410 | 1,130,993 | 12,305,805 | 5,450,241
Tasks | Regression | Classification | Classification | Classification | Classification
Metrics | MSE | Accuracy | ROC-AUC | AP | Accuracy
Table 2: Node Classification and Link Prediction Datasets Statistics.  | ogbn-arxiv | ogbl-collab
---|---|---
Domains | Citation | Social
#Nodes | 169,343 | 235,868
#Edges | 1,166,243 | 1,285,465
Tasks | Node Classification | Link Prediction
Metrics | Accuracy | Hits@20

Training Details. For GIN and GCN models, we set the initial learning rate to
0.001 and use Adam optimizer (Kingma & Ba, 2014) in the training process. For
SAT and GraphGPS, we directly adopt the hyperparameters and optimizers that
the authors present in their GitHub repos. For ogbg-ppa, ogbn-arxiv, and ogbl-
collab datasets, the training epoch number is 50. For pcqm4mv2, ogbg-molhiv,
ogbg-molpcba, and reddit-multi-12k datasets, the training epoch number is 100.
We choose the test performance with the best corresponding validation
performance as the final performance of the model.

##  Appendix B More Results on the Validity of the Neural Scaling Law on
Graphs

Here we show the scaling behaviors of GCN on the pcqm4mv2 and ogbg-ppa
datasets. All the empirical values fit well with the neural scaling law.

(a) Model scaling on pcqm4mv2, $R^{2}=0.98$.

(b) Data scaling on pcqm4mv2, $R^{2}=0.98$.

Figure 11: The model and data scaling behaviors of GCN on pcqm4mv2. For model
scaling, the color corresponds to the training data fraction; for data
scaling, the color corresponds to the number of model parameters. The lower
mean square error is, the better model performance is. The high $R^{2}$ values
indicate that the scaling law curves fit well with the empirical values.

(a) Model scaling on pcqm4mv2, $R^{2}=0.98$.

(b) Data scaling on pcqm4mv2, $R^{2}=0.99$.

Figure 12: The model and data scaling behaviors of GCN on ogbg-ppa. For model
scaling, the color corresponds to the training data fraction; for data
scaling, the color corresponds to the number of model parameters. The higher
performance is, the better model performance is. The high $R^{2}$ values
indicate that the scaling law curves fit well with the empirical values.

(a) Scaling behaviors on pcqm4mv2, $R^{2}=0.97$.

(b) Scaling behaviors on ogbg-ppa, $R^{2}=0.97$.

Figure 13: The comprehensive scaling behaviors of GCN on pcqm4mv2 and ogbg-ppa
datasets. For pcqm4mv2, the color corresponds to the value of mean square
error; for ogbg-ppa, the color corresponds to accuracy. The surfaces predicted
by the neural scaling law fit well with the empirical values on both datasets,
as indicated by the high $R^{2}$ values.

##  Appendix C More Results on the Model Scaling Collapse

Here we present more results on the relationship between the model size and
the degree of overfitting. In general, the observation in the below figures
indicate that the lager model is, the more serious it would overfit.

Figure 14: The training losses and validation losses of GCN with varying sizes
on Code2 Dataset. The color corresponds to the number of model parameters. The
figure shows that the model with larges size has more serious overfitting.
Figure 15: The training losses and validation losses of GIN with varying sizes
on Code2 Dataset. The color corresponds to the number of model parameters. The
figure shows that the model with larges size has more serious overfitting.
Figure 16: The training losses and validation losses of GCN with varying sizes
on ogbg-ppa Dataset. The color corresponds to the number of model parameters.
The figure shows that the model with larges size has more serious overfitting.

##  Appendix D More Results on the Impacts of Model Depths

We present the experiment results of model scaling behaviors with varying
model depths on more datasets. In the following figures, models with different
depths exhibit distinct model scaling behaviors.

(a) Model scaling of GCN with different layers.

(b) Model scaling of GIN with different layers.

Figure 17: Model scaling behaviors of GNNs with varying model depths on
reddit-multi-12k. The color corresponds to the model layer. Both GIN and GCN
exhibit distinct scaling behaviors with different model depths.

(a) Model scaling of GCN with different layers.

(b) Model scaling of GIN with different layers.

Figure 18: Model scaling behaviors of GNNs with varying model depths on ogbg-
molhiv. The color corresponds to the model layer. Both GIN and GCN exhibit
distinct scaling behaviors with different model depths.

(a) Model scaling of GCN with different layers.

(b) Model scaling of GIN with different layers.

Figure 19: Model scaling behaviors of GNNs with varying model depths on ogbg-
molpcba. The color corresponds to the model layer. Both GIN and GCN exhibit
distinct scaling behaviors with different model depths.

##  Appendix E More Results of the Disadvantage of the number of Graphs as
Data Scaling Metric

Here we present more results of the experiments in Section 6.1. The settings
are the same as in Section 6.1, except we replace GIN with GCN. From the below
figure, we observe that performance curves have obvious differences when two
subsets have the same number of graphs. But when the two subsets have the same
number of edges, the differences between the two curves become much smaller,
supporting that the number of edges can better measure the data volume.

(a)

(b)

Figure 20: Performance curves of GCN of varying sizes trained on two subsets.
In (a), the simple subset and the complex subset have the same graph number
but different edge numbers. In (b), the simple subset and the complex subset
have the same edge number but different graph numbers. The shade corresponds
to the performance variance of repeated experiments. This figure supports the
same conclusion as in Section 6.1.

##  Appendix F The Relationship between Edge Number and FLOPs

Here we show that the training compute can be calculated when we know the
total edge number and the model architecture. We consider the general Message
Passing Neural Networks, which can be formulated as above:

$m^{t+1}_{v}=U(h^{t}_{v},\sum_{w\in N(v)}M(h^{t}_{v},h^{t}_{w},e_{vw}))$ (6)

where $U$ and $M$ can be assumed to contain $x$ and $y$ add-multiply
operations, respectively. Suppose the total node number in the dataset is N,
the total edge number is E, and the model has L layers. For node $i$,
aggregating messages from its neighbors needs $x+yd_{i}$ add-multiply
operations, where $d_{i}$ is the degree of node $i$. For the whole datasets,
there would be $\sum_{i}x+yd_{i}\approx 2yE$. For the whole model, one feed-
forward pass needs $2yEL$ add-multiply operations. Thus, we have shown that we
can calculate the traing FLOPs if we know the total edge number in the
dataset.

##  Appendix G The Model Scaling Behaviors of Node Classification and Link
Prediction

Here we investigate the model scaling behaviors of node classification and
link prediction tasks. The observations in the below figure show that SEAL
trained on ogbl-collab has obvious performance gains when the model size is
increased, while this is not the case for GraphSAGE on ogbn-arxiv. We leave
further exploration of these topics for future works.

(a) Model scaling behaviors on ogbn-arxiv.

(b) Model scaling on ogbl-collab.

Figure 21: In (b), the model performance has obvious gains on the link
prediction task when the hidden vector size is increased, while it is not for
the node classification task in (a).