# Economic impacts of AI-augmented R&D

Tamay Besiroglu11footnotemark: 1
MIT FutureTech  Nicholas Emery-Xu11footnotemark: 1
UCLA Dept. of Economics, MIT FutureTech  Neil Thompson22footnotemark: 2
MIT FutureTech

###### Abstract

Since its emergence around 2010, deep learning has rapidly become the most
important technique in Artificial Intelligence (AI), producing an array of
scientific firsts in areas as diverse as protein folding, drug discovery,
integrated chip design, and weather prediction. As more scientists and
engineers adopt deep learning, it is important to consider what effect
widespread deployment would have on scientific progress and, ultimately,
economic growth. We assess this impact by estimating the idea production
function for AI in two computer vision tasks that are considered key test-beds
for deep learning and show that AI idea production is notably more capital-
intensive than traditional R&D. Because increasing the capital-intensity of
R&D accelerates the investments that make scientists and engineers more
productive, our work suggests that AI-augmented R&D has the potential to speed
up technological change and economic growth.

††* Joint first authors††$\dagger$ Corresponding author<EMAIL_ADDRESS>helpful comments we are grateful to Philip Trammell, Basil Halperin, Matt
Clancy, Charlotte Siegmann, Ege Erdil, Gabriel Filipe, Wensu Li, Tom Davidson,
Jaime Sevilla, and Nur Ahmed.††The authors are grateful to Open Philanthropy
for financial support. Nicholas Emery-Xu is also grateful to the UCLA Graduate
Division and the Future of Humanity Institute for financial support.

##  1 Introduction

In this paper, we consider what effect the adoption of Artificial Intelligence
(AI) within science and engineering will have on idea production and,
subsequently, on productivity and economic growth. Unlike previous work that
has attempted to provide only a theoretic treatment of the topic, we approach
this question with microdata from deep learning, the AI paradigm responsible
for nearly all landmark results in the past decade. We provide a framework for
understanding the impact of two important trends: i) the recent breakthroughs
using deep learning in R&D, and ii) the rapid scaling of computation in deep
learning systems. We show that if deep learning is widely adopted in the U.S.
R&D sector, it would induce an accumulation of computational capital that
could nearly double the productivity growth rate.

Since the early 2010s, when it produced seminal breakthroughs in computer
vision and speech recognition, deep learning has led to a rapid increase in
the rate of progress in Artificial Intelligence ([70]; [38, Ch. 1]; [93, Ch.
1.3]). Breakthroughs have been made in many areas, including, to name a few,
computer vision, speech recognition, natural language processing, and game
playing. Deep learning has also made inroads into parts of science largely
untouched by previous AI research, including protein folding, semiconductor
chip floorplanning, controlling nuclear fusion, and even discovering novel
algorithms and new insights in pure mathematics. The rate at which long-
standing problems have been solved, and the pace at which deep learning
systems have out-competed traditional algorithms, have been surprisingly rapid
to even some of its most seasoned practitioners.

As uses of AI proliferate, economists have sought to understand its impacts on
wages, factor shares, and economic growth. A prominent line of thought asks
whether deep learning has the potential to become a General Purpose
Technology, a technology with widespread applications in a variety of
industries and the ability of AI to replace human labour across a wide variety
of tasks ([35, 4, 107]).111For example, [35] calculate the prevalence of deep
learning-related job requirements in job postings within and across industries
to predict whether technologies will become GPTs by whether they are in
widespread use, capable of ongoing self-improvement, and enable multi-sector
innovation. Across a set of 21 technologies, they find that machine learning
displays the characteristics of a GPT in the deep learning era but not
beforehand. In contrast, [106] questions whether ongoing increases in deep
learning performance will be sustainable, which could undermine its ability to
provide the long-term benefits of a GPT.

Much of the existing research has focused on the potential of AI to impact
final good production, but it has also been pointed out (e.g. by [26]), that
AI also has the potential to change the innovation process itself. Such
“Inventions of a Method of Invention” (IMI) can significantly affect the rate
of idea production ([27, 26]) and, therefore, the overall rate of innovation
in the economy. For example, building on the [112] model of recombinant
technological development, [6] argue that deep learning can improve knowledge
production by effectively searching through and recombining a wider range of
ideas than is possible by human scientists and that this could result in
accelerated economic growth. Empirical testing of the impact of AI on R&D
shows mixed results. [21] find that the use of deep learning is positively
correlated with the mean and variance of paper citations received, increasing
the likelihood for a contribution to become an influential ‘big hit.’ However,
they also find that it is negatively correlated with the re-combinatorial
novelty of ideas, measured as a function of the fraction of novel citation
pairs in a given paper. Another line of research focuses on the relationship
between AI and data, showing that machine learning increases the returns to
data and thus the rate of knowledge production for data-rich firms ([18, 2];
[5]). While these insights are informative about firm-level effects, they shed
less light on the implications for the aggregate economy.

The impacts of AI on the innovation process deserve special attention because
it has been pointed out that these, under suitable conditions, can have more
dramatic permanent effects on productivity growth than those that arise from
changes in final goods production. For example, in the semi-endogenous growth
model of [3], the authors consider AI automation in producing final goods and
in producing knowledge, and find that the latter can produce much more rapid
output growth. [108] provide a review of the theoretical literature on AI and
growth, which concludes that, while a high degree of automation in final goods
production can produce a one-time increase in the growth rate, a high degree
of automation in the R&D sector can produce unbounded increases in economic
growth.

Our work investigates how deep learning will impact the production of ideas.
We argue that the adoption of deep learning makes computational capital in R&D
more productive, resulting in capital deepening that, if widespread,
accelerates knowledge creation and economic growth. To motivate this, we
derive a semi-endogenous growth model that shows that a positive shock to the
R&D elasticity of capital—such as might follow the widespread adoption of deep
learning techniques—permanently increases the rate of idea accumulation and
economic growth.

But will deep learning increase the R&D elasticity of capital? We provide
supportive empirical evidence by estimating the idea production functions for
two relatively mature deployments of deep learning. To analyze human capital
in deep learning, we develop a novel machine learning approach for estimating
human capital and apply it to machine learning papers in the arXiv repository.
To analyze computing resources, we augment the dataset from [106] to cover the
entire universe of papers on two popular computer vision tasks.

Our estimates of the deep learning production function allow us to compare AI-
augmented R&D with the R&D practiced in U.S. science and engineering areas. We
find that deep learning’s idea production function depends notably more on
capital. This greater dependence implies that more capital will be deployed
per scientist in AI-augmented R&D, boosting scientists’ productivity and
economy more broadly. Specifically our point estimates, when analysed in the
context of a standard semi-endogenous growth model of the US economy, suggest
that AI-augmented areas of R&D would increase the rate of productivity growth
by between 1.7- and 2-fold compared to the historical average rate observed
over the past 70 years.

Our analysis is organized as follows. Section 2 motivates the importance of
R&D capital intensity for economic growth using a semi-endogenous growth
model. Section 3 describes the datasets, and Section 4 the empirical strategy
we use to model and estimate idea production. A key input for these models is
an estimate of the human capital of the teams working on particular AI
projects. In Section 5 we develop a deep neural network for learning simple
representations of human capital that outperforms other measures commonly used
in scientometric literature. For example, our human capital estimates explain
60-80% of variance in key publication-related outcomes, whereas standard
linear models explain less than 20%. In Section 6, we present our empirical
analysis, which implies that a firm in a competitive R&D sector using deep
learning would be roughly 5 times more capital-intensive than current U.S.
STEM R&D. In Section 7, we use our growth theory model to investigate the
implications of higher capital intensity of R&D, and find that it implies a
substantially faster productivity growth rate— 2- to 3-fold greater than the
0.8% growth rate the US saw over the last decade. In Section 8, we find that
our results are robust to outliers and alternative model specifications,
including alternative elasticities of substitution between computational and
human capital. In Section 9, we consider limitations of our analysis as well
as future directions for research, and in Section 10 we conclude with a brief
discussion of the implications of our results.

##  2 Capital-intensive R&D and deep learning

###  2.1 The role of capital in idea production

We argue that deep learning may affect the growth rate of knowledge by
impacting the productivity of research capital. While the role of capital does
not usually receive center-stage in the analysis of R&D-based growth, it has
been shown to generate permanent growth effects by increasing the marginal
product of labour in R&D and thus increasing investment in the R&D sector
([54, 53]). The key mechanism driving this result is that, unlike the stock of
human labour, the rate of physical capital accumulation can be readily
increased or decreased in response to a change in its productivity. A very
similar point is made by [3] in their study of the growth effect of AI. They
show that in the classic endogenous growth case, a one-time increase in R&D
automation will raise the long-run growth rate, as capital—an accumulable
factor in production—becomes permanently more important.

Empirical work has validated the importance of physical capital investment in
idea production. For example, [44] showed that the creation of the UK’s
Diamond Light Source synchrotron increased the research capital available to
local scientists, which in turn increased their research publication output
relative to UK scientists located elsewhere.

The prevalence of capital goods in U.S. R&D is documented by the National
Science Foundation’s Higher Education Research and Development Survey. It
finds that academic institutions have spent over $2bn per year on Capital
Equipment or Software for R&D since 2010, representing roughly 4% of total
direct R&D expenditures ([80]). In STEM fields, this share is higher—around 6%
overall in 2020, with Chemistry at 14%, Material Science and Physics at 15%,
and Engineering ranging between 7% and 11%. Since the overall capital-
intensity of academic science is only 4%, there is enormous room for more
capital-intensive approaches.

Even as compared to capital-intensive areas of science, there are reasons to
suspect that R&D using deep learning might be yet more so. Some recent
capital-intensive examples include OpenAI’s GPT-3 language model ([24]),
DeepMind’s AlphaZero game-playing system ([100]), and DeepMind’s protein-
folding system, each of which reportedly used millions of dollars worth of
computing ([34, 62]).

Computing theory also suggests reasons deep learning is capital-intensive and
why it is likely to become more so. In classical statistical learning theory,
there generally is a trade-off between bias and variance ([42]). Once a model
grows beyond a certain complexity threshold, it tends to “overfit” the data,
worsening test performance222As distinct from the training error, test
performance is calculated on data points never seen by the network. as the
variance term dominates. Deep neural networks seem capable of evading this
trade-off by vastly expanding the size of the network
(“overparameterization”), that is by deploying more computational capital
([16, 79]).

Surprisingly, empirical analyses have shown that the performance gains that
accrue to these networks with millions or billions of parameters are highly
predictable. Generally, these analyses of “neural scaling laws” find that test
error falls according to a power law in the scale of such models, and
therefore in the amount of compute used ([63, 51, 47, 102, 71, 73, 59, 14,
99]).333See particularly, [14] for an account of the state of the existing
work in their Related Works section. Researchers and practitioners are
harnessing these dependencies to get better performance. For example, [106]
shows that progress is highly dependent on computational resources across a
wide range of machine learning tasks. For image classification on the ImageNet
database, $71\%$ of the variance in model performance is explained by the
computation used. The importance of computing resources in deep learning was
elegantly summarized by Rich Sutton ([103]), a prominent figure in the field
of reinforcement learning, who wrote:

> The biggest lesson that can be read from 70 years of AI research is that
> general methods that leverage computation are ultimately the most effective,
> and by a large margin… Seeking an improvement that makes a difference in the
> shorter term, researchers seek to leverage their human knowledge of the
> domain, but the only thing that matters in the long run is the leveraging of
> computation.

If deep learning is indeed more capital-intensive, the investment dynamics
implied by endogenous growth models would predict a rapid scale-up to have
occurred in the computational capital being used in AI-based R&D. [96] find
exactly that: since the advent of deep learning, the growth in the amount of
computational capital typically used in milestone models doubles roughly every
6 months, far outstripping the rate during previous eras of AI. So, while the
size of capital investments made in deep learning systems are still small
compared to, for example, those required for large-scale physics experiments,
there are compelling reasons to believe that these models are capital-
intensive and will continue to become more so.

###  2.2 R&D capital intensity in a semi-endogenous growth model

Consider a simple semi-endogenous growth model along the lines of [61]. There
are two sectors, a goods-producing sector where output is produced and an R&D
sector where additions to the stock of knowledge are made. A fraction
$\alpha_{l}$ of the labour force is used in the R&D sector and fraction
$1-\alpha_{l}$ in the goods-producing sector. Similarly, fraction $\alpha_{k}$
of the capital stock is used in R&D and the rest in goods production. We make
similar simplifying assumptions as [90] by supposing that $\alpha_{l}$ and are
$\alpha_{k}$ exogenous and constant for expositional clarity. Ideas are non-
rivalrous and the full stock is used equally in both sectors of the economy.
For further simplicity, we assume constant returns to scale in the production
of final goods. The quantity of output produced at time $t$ is thus:

$Y(t)=\big{[}(1-\alpha_{k})K(t)\big{]}^{\alpha}\big{[}A(t)(1-\alpha_{l})L(t)\big{]}^{1-\alpha},\hskip
4.26773pt\text{where}\hskip 4.26773pt\alpha\in(0,1),$ (1)

The production of new ideas depends on the quantities of capital and labour
engaged in research and on the level of technology. We assume there are
diminishing returns in the production of new ideas in inputs
($\beta+\theta<1$). This assumption ensures a unique steady-state growth rate,
and prevents the growth rate from exploding as the R&D inputs grow without
bound. The idea stock grows as follows:

$\dot{A}(t)=B\big{[}\alpha_{k}K(t)\big{]}^{\beta}\big{[}\alpha_{l}L(t)\big{]}^{\gamma}A(t)^{\theta},\hskip
4.26773pt\text{where}\hskip 4.26773ptB>0,\beta,\hskip 4.26773pt\gamma,\geq
0\hskip 4.26773pt\text{and}\hskip 4.26773pt\beta+\theta<1,$ (2)

where $B$ is a positive shift parameter. We further make the simplifying
assumptions, not uncommon in the literature, that there is a constant saving
rate $s$, and that capital depreciates at a constant rate $\delta$. Moreover,
our model is a semi-endogenous one. Hence, we suppose that population grows at
exogenous rate $n$. Thus capital and labour accumulation are described as
follows:

$\dot{K}(t)=sY(t)-\delta K(t),\hskip 4.26773pt\text{and}\hskip
4.26773pt\dot{L}(t)=nL(t),\hskip 4.26773pt\text{where}\hskip
4.26773pts,\delta\in(0,1)\hskip 4.26773pt\text{and}\hskip 4.26773ptn>0.$ (3)

Using equations (1-3), we solve for the steady-state rates of growth in ideas
and capital (denoted as $g^{*}_{a}$ and $g^{*}_{k}$ respectively.444A complete
derivation may be found in Appendix A1.) These are:

$g^{*}_{a}=\frac{\beta+\gamma}{1-\beta-\theta}n,\hskip
4.26773pt\text{and}\hskip
4.26773ptg^{*}_{k}=\frac{1-\theta+\gamma}{1-\beta-\theta}n.$ (4)

Proposition 1. Consider a shift in the technology of R&D that creates a
positive shock to the R&D elasticity to capital and, at worst, a proportional
negative shock to the R&D elasticity to scientists. That is, consider a shift
from an R&D setting described by (2) to a setting described as follows:

$\dot{A}(t)=B\big{[}\alpha_{k}K(t)\big{]}^{\beta^{\prime}}\big{[}\alpha_{l}L(t)\big{]}^{\gamma^{\prime}}A(t)^{\theta}$
$\text{where}\hskip 4.26773pt\beta^{\prime}>\beta,\hskip
4.26773pt\beta^{\prime}-\beta\geq\gamma-\gamma^{\prime},\hskip
4.26773pt\text{and}\hskip 4.26773pt\beta^{\prime}+\theta<1.$

Such a shift in the technology of R&D has the following implications:

  1. (a)

the rate of idea accumulation is strictly and permanently increased, and

  2. (b)

the rate of economic growth is strictly and permanently increased.

Proof. Define $\Delta_{X}\equiv X^{\prime}-X$ as the difference between pre-
and post-values of parameter $X$. By assumption,
$\Delta_{\beta}\geq-\Delta_{\gamma}$. The steady-state rate of growth in ideas
is strictly increased if:

$\frac{\beta^{\prime}+\gamma^{\prime}}{1-\beta^{\prime}-\theta}n>\frac{\beta+\gamma}{1-\beta-\theta}n.$
(5)

which follows from the fact that $\Delta_{\beta}\geq-\Delta_{\gamma}$. This
proves part (a) of the proposition. The proof of part (b) may be found in
Appendix A2.

Proposition 1 states that if there is a shift to an R&D setting with higher
R&D elasticity to capital, and if the R&D elasticity to scientists is not
disproportionately negatively affected, the steady-state rate of technological
change and economic growth will strictly and permanently increase.

Figure 1. Productivity growth and R&D cost-shares in a competitive R&D
economy. Figure 1a shows steady-state productivity growth as a function of the
implied capital-cost share in a competitive R&D industry when the elasticity
of R&D output to the stock of ideas ($\theta$) is $1/2$ and the elasticity of
R&D output to labour inputs ($\gamma$) is $2/5$ (consistent with our survey of
existing estimates in Appendix C). Figure 1b shows the share of R&D
expenditure in each discipline that is spent on capital equipment, based on
data from the National Science Foundation 2020 Higher Education Research and
Development Survey ([80]).555The underlying data and the details of the
calculations used for Figure 1b may be found here.

The intuition behind this result is as follows: while the labour force grows
at a rate independent of economic growth, capital accumulation is determined
endogenously by investment. After a positive shock to the R&D elasticity to
capital, investment rises, increasing the productivity of scientists. This
increases the rate of idea production and consequently boosts economic growth.
But faster economic growth also increases the rate of capital investment.
Hence, a positive shock to the capital productivity of R&D gives rise to a
virtuous cycle of idea accumulation, economic growth and capital formation: a
cycle that produces a new balanced growth path with permanently higher steady-
state rates.

To illustrate how capital-intensive R&D could result in super-normal
productivity growth, consider Figure 1a. Suppose the R&D sector is competitive
(such that wages and rents are equal to their marginal products), the steady-
state rate of idea accumulation increases steadily in the share of R&D
expenditure dedicated to capital. Under conservative assumptions on our growth
model, highly capital-intensive R&D (such as when optimising R&D firms
dedicate at least 20% of R&D expenditure to capital) would produce
productivity growth rates in excess of the usual productivity growth rates
observed in the US. By contrast, current US R&D tends to be highly labour-
intensive. Using 2020 data from the [80] NSF-supported STEM R&D and assuming a
Cobb-Douglas functional form for ideas production, we see that capital shares
tend to fall between 3% and 20% (see Figure 1b).666STEM fields tend to have
higher capital intensities than non-STEM fields.777That is, to compute the
capital share, we divide capital expenditures by total R&D expenditures in
each field. Our assumption and the resulting estimated cost shares are
corroborated by [28], who use a Cobb-Douglas specification and find that the
capital share in R&D among Belgian firms is less than 15%.

In the analysis section, we present evidence that the relative returns to
capital for deep learning are higher than other types of R&D. This suggests
that, if deep learning could be similarly applied to a wide range of R&D
problems, its high degree of capital intensity could accelerate technological
change and, as a consequence, economic growth.

##  3 Data

In our work, we rely primarily on two datasets. Our primary dataset covers the
compute cost and performance for 151 deep learning models that were presented
in publications between 2012 and 2021. The second is a bibliometric dataset of
the authors of machine learning publications published between 1993 and 2021,
which we use to infer the human capital inputs for each deep learning model in
our primary dataset.

###  3.1 Data on computer vision experiments

Our dataset on the compute costs and performance covers 151 models published
between 2012 and 2021. This data is an augmented version of [106], with
additional details about the settings under which the models were trained and
tested, for example, whether additional training data was used, or whether the
training or test data was augmented.

The compute estimates are derived from the underlying papers following the
procedure described by [97], which we summarize in Appendix E. The inclusion
and exclusion criteria used to generate our datasets is described in [106].
Deep learning models in this dataset span two well-known benchmarks: image
classification on the ImageNet dataset and object detection on the Microsoft
COCO dataset, usually known as the MS COCO dataset.

ImageNet is perhaps the most well-known and widely used computer vision
dataset. It spans 1,000 object classes and contains 1.28m training images
([92]). Some of the most important breakthroughs in deep learning have
happened in ImageNet models, starting with AlexNet, a watershed moment when
deep learning first outperformed other techniques on this task ([69]).
Importantly for our purposes, success on ImageNet has often proven to be
general: techniques that advance its state-of-the-art have usually been found
to be successful in other tasks and domains. For example, [20] documents
various instances when progress on ImageNet due to architecture design or
optimization has yielded corresponding gains on other modalities, such as
natural language processing, audio processing, and game playing. Because of
this, it is plausible that our results for this benchmark could generalize to
tasks and domains beyond computer vision.

The MS COCO 2017 dataset is one of the most frequently used datasets for
object detection, face detection, and pose estimation, among other tasks. It
contains a total of 2.5 million labelled instances in 328k images ([74]). Like
ImageNet, this dataset has been used as a test-bed of many influential
innovations, such as [43]’s Residual Network architecture, which has since
become widely used in computer vision (see e.g. [64]).

While these two domains of computer vision are crucial test-beds for deep
learning, it would be better if we considered a wider range of scientific and
technical domains in which these techniques were applied. Unfortunately, this
is difficult because of challenges for both inputs and outputs. For inputs,
many deep learning papers fail to report even basic details of their
computational usage. For outputs, some areas of deep learning struggle to
define objective measures of performance. For example, how should one define
the “correct” text summary of a picture?888See [106] for a more in-depth
discussion of these challenges.

###  3.2 Data on authors and publications

Our dataset on machine learning publications comes from arXiv, a pre-print
server commonly used in various STEM fields, including computer science, and
Scopus, Elsevier’s abstract and citation database. Our dataset includes all
papers on arXiv that were posted between 1993 and 2021 that are from the
subfields typically associated with machine learning: Machine Learning
(stat.ML), Artificial Intelligence (cs.AI), Computation and Language (cs.CL),
Computer Vision and Pattern Recognition (cs.CV), and Learning (cs.LG). We
match the authors of these papers to their corresponding entries in Scopus
using a variety of string distance-based matching approaches. This technique
allows us to match 90.1% of authors, and spot testing on 300 random matches
shows that 96% were correct.

With the connection between papers to their authors’ publication histories, we
construct a timeseries for each author that shows their number of
publications, h-index, and citations (excluding self-citations). We supplement
this data with similar timeseries of grant funding for each author’s
institution and department over time from the Dimensions grant database,
institutional rankings over time from csmetrics.org, and measures of the
scientific influence of computer science journals and conferences using from
SCImago. For full details on data collection procedures, see Appendix D.

##  4 Empirical strategy

We assume, along the lines of the semi-endogenous growth model outlined above,
that idea production using deep learning depends on three factors: labour
(scientists’ human capital), specialised capital goods (computational
capital), and total factor productivity (the extant level of ‘ideas’ or
technology upon which researchers build):

$\dot{A}(t)=BA(t)^{\theta}S(t)^{\gamma}C(t)^{\beta},\hskip
4.26773pt\text{where}\hskip 4.26773ptt>0,\text{and}\hskip
4.26773ptX(0)>0\hskip 4.26773pt\text{for any}\hskip 4.26773ptX\in\\{A,S,C\\}.$
(6)

where $\dot{A}(t)$ denotes the change in the stock of technology, $S(t)$ the
total human-capital input of scientists, and $C(t)$ refers to the total
capital inputs. To estimate this using data, we replace $\dot{A}(t)$ with a
measure of the performance of deep learning models, $C(t)$ with data on
computational inputs, and $S(t)$ with estimates of scientific human capital
inputs.999This model, which is a Cobb-Douglas production function, implies a
certain level of substitutability between computational capital and
scientists. In Section 8.3, we validate this model by showing that our
estimates of the relevant elasticity of substitution support it.

###  4.1 Empirical specification

Consider an economy where the level of technology grows exponentially on the
balanced growth path in the way standardly assumed in growth theory models:

$A(t)=A(0)\mathrm{e}^{gt},\hskip 4.26773ptA(0)>0.$ (7)

We do not observe technology directly. Instead, we observe performance on
machine learning tasks. In these cases, the level of performance—usually
measured as a type of predictive accuracy—falls on the unit interval. We
assume that performance relates to technology according to the logistic
function, reflecting that the most challenging parts of innovation are being
able to make some initial headway with a problem and then perfecting it,

$P(t)=\frac{A(t)}{1+A(t)}.$ (8)

This is a similar assumption used to model how effort relates to various
outcomes when the outcomes are bounded, such as in contests ([111, 15]),
conflict interactions ([50, 58]), and persuasion ([101]). Beyond the fact that
this is a relatively standard transformation that enables us to map progress
in technology onto a bounded interval, there are two further motivating
considerations.

Firstly, we show that this functional form implies a power-law between the
scale of the compute deployed and the level of error achieved, which is in
line with a robust finding of the relevant machine learning literature (e.g.
[51]) (see Appendix A5). Secondly, this functional form enables us to
construct a simple empirical counterpart for technological progress, which we
derive as follows. First, assuming that growth rates in adjacent periods are
approximately equal (i.e. that $g_{t}\approx g_{t-1}$) it can be shown that
proportional technological growth relates to performance improvements as
follows (see Appendix A4):

$\frac{\dot{A}(t)}{A(t)}=\log\Bigg(\underbrace{\frac{P(t)}{P(t-1)}}_{\makebox[0.0pt]{\text{\footnotesize
Proportional increase in accuracy}}}\Bigg{missing})\hskip 22.76228pt+\hskip
22.76228pt\log\Bigg(\underbrace{\frac{1-P(t-1)}{1-P(t)}}_{\makebox[0.0pt]{\text{\footnotesize
Proportional reduction in error}}}\Bigg{missing}),$ (9)

which thus provides us an easy-to-interpret decomposition of technological
progress in terms of the (logs of) the proportional reduction in error rate
and the proportional increase in accuracy.

Let $\tilde{g_{t}}$ denote the approximation of $g_{t}$ given $P(t)$, i.e.
$\tilde{g_{t}}\equiv\log\bigg(\frac{P(t)}{P(t-1)}\bigg{missing})+\log\bigg(\frac{1-P(t-1)}{1-P(t)}\bigg{missing})$.
We can write the empirical specification of our model as follows:

$\tilde{g_{t}}=A(t)^{\theta-1}S(t)^{\gamma}C(t)^{\beta}.$ (10)

We thus obtain an empirical specification of $\tilde{g_{t}}$ that we can
ground in the relevant empirical data. When relating the model to data, time
becomes discrete, and experiments are produced by research groups, which are
indexed by $i\in\\{1,...,N\\}$.

Assuming a multiplicative error model, we specify the empirical counterpart of
(10) as follows:

$\tilde{g}_{it}=A_{t}^{\theta-1}S_{it}^{\gamma}C_{it}^{\beta}\epsilon_{it},\hskip
4.26773pt\text{where}\hskip 4.26773pt\log\epsilon_{it}\equiv u_{it}\sim
N(0,\sigma^{2}).$ (11)

Taking logs of both sides, we have:

$\log\tilde{g}_{it}+=(\theta-1)\log A_{t}+\gamma\log S_{it}+\beta\log
C_{it}+u_{it}.$ (12)

We estimate the following model:

$\displaystyle\log\underbrace{\tilde{g}_{it}}_{\makebox[0.0pt]{\text{\footnotesize
Extent of tech. progress by $i$}}}\hskip 21.33955pt=\hskip
21.33955pt(\theta-1)\log\underbrace{A_{t}}_{\makebox[0.0pt]{\text{\footnotesize
Extant level of technology}}}\hskip 31.2982pt+\hskip
31.2982pt\gamma\log\underbrace{S_{it}}_{\makebox[0.0pt]{\text{\footnotesize$i$'s
human capital}}}\hskip 28.45274pt+$ (13)
$\displaystyle\beta\log\underbrace{C_{it}}_{\makebox[0.0pt]{\text{\footnotesize$i$'s
computational capital}}}\hskip 31.2982pt+\hskip
31.2982pt\mathbf{\alpha}\underbrace{\mathbf{X}}_{\makebox[0.0pt]{\text{\footnotesize
Vector of controls}}}\hskip 28.45274pt+\hskip 28.45274ptu_{it}.$

which we can estimate in a pooled fashion with a time-fixed effect that
captures $(\theta-1)A_{t}$ for $t\in\\{1,...T\\}$. By default, we will fix the
time periods as years. In the robustness checks section, we show that shorter
or longer time windows do not change our overall results.

A key basic that is evident from our estimation procedure (which we import
from endogenous growth theory) is that knowledge is non-rivalrous. This
assumption warrants some reflection. It assumes that researchers have access
to a common stock of knowledge at any point in time and that the advancements
made by researchers in one period are available to others in the next period.
These assumptions seem broadly reasonable given the open research norms in
machine learning (e.g. publishing on arXiv), providing access to code (e.g.
via GitHub, Papers with Code, etc.) as well as the common tools (e.g. PyTorch)
that embed model implementations. And, indeed, companies that are notoriously
closed-lipped about technology are nevertheless relatively open about AI
research ([7]).

If, nevertheless, there are groups that try to keep trade secrets, and those
tend to be areas of particularly high human capital (as one might expect),
then our model would implicitly treat such knowledge as an additional benefit
of human capital. All else equal, this would make our results about capital-
intensity underestimates of the true level.

###  4.2 Operationalizing innovations

In estimating our model, we need to operationalize proportional performance
improvements in terms of observables. We measure this using our baseline data,
where authors of each paper have indicated the touchstone models in the
literature whose ideas they are building on. That is, for any particular task,
we define relative performance gains as follows:

$P_{t}/P(t-1)\equiv P_{t}/\text{Baseline}_{t}.$ (14)

That is, $i$’s innovation is defined as the proportional improvement over the
performance of a model that is considered, by the contemporaneous literature,
to be a relevant baseline model. This operationalization is chosen for two
reasons. Firstly, it is common practice to report these values in the machine
learning literature, as the extent of innovations are often illustrated
through comparisons to existing baseline levels of performance ([10, 76, 84]).
Secondly, this notion of an improvement over a model lines up well with the
usual notion of the change in stock of knowledge in R&D-based growth models,
such as those from [90, 39] and others; it represents the extent of the
innovation of a new design relative to the existing stock of ideas. To find
the appropriate baseline levels of performance, we survey the models that are
used as baseline results in the relevant literature and take the median of
their performance (See Appendix D1).

##  5 Modelling human capital

The final remaining piece needed to estimate the deep learning production
functions for these computer vision tasks is to construct a measure of the
scientific human capital used for each of these models. This measure should be
predictive of outcomes that are strongly influenced by human capital and must
be inferable from available data about the scientists’ track records. In
addition to overall predictiveness, it is particularly important that
estimates are good for junior researchers, who contribute importantly to this
young field.

Prior work has used various measures of the quality or status of scientists
and engineers, including impact-based metrics, such as citation counts ([12,
60, 114]), the number of high-impact citations ([13]) and bibliometric indices
such as the h-index ([104, 33, 22]). These approaches have substantial
limitations as approaches to measuring the human capital of teams of
scientists and engineers. As we shall see, almost all of these measures are
only weakly predictive of key outcomes where we would expect scientific and
technical human capital to be important, such as the number of citations the
work will receive in the future, or the quality of the journal or conference
the publication is to be published in. Moreover, impact-based metrics, such as
citation counts or the h-index generally assign low scores to junior
researchers, as citations often take a long time to accrue following the
publication of scholarly work.

Our strategy for modelling researcher human capital is as follows. We
construct a deep neural network (DNN) and train it to develop a single-
dimensional representation of the total quality-adjusted research input
(“human capital”) that is highly predictive of key bibliometric and
publication-related outcomes. Our approach implements an encoder that maps
many features about the publication’s authors input to a single-dimensional
representation, and a decoder model (built explicitly to have the function as
a linear regression) that maps this representation onto citation- and
publication-related outcomes. Our approach exploits the ability of DNNs for
nonlinear data compression of high-dimensional input features (see e.g. [48];
[68]).101010Similar bottlenecks are used for a variety of feature-learning
tasks, such as through under-complete auto-encoders (see e.g. [17]). Our
approach finds human capital representations that are highly predictive of key
bibliometric and publication-related outcomes, and that substantially
outperform the typical approaches used in the literature.

###  5.1 Our machine learning approach to estimating human capital

We use our dataset of 49,251 machine learning publications to train a neural
network to predict bibliometric and publication-related outcomes. The
predicted outcomes include the citation trajectories for each publication and
its SJR-values, a measure of the quality of the journal or conference where
the work ends up being published (see [37]). Figure 2a provides a diagrammatic
overview of our data pipeline and the training set-up used to produce our
model. Further details of the training procedure may be found in Appendix F.

Figure 2. Human capital estimation strategy. Figure 2a presents our set-up for
learning human capital representations for machine learning publications.
Figure 2b shows our neural network architecture. Highlighted is the human
capital unit, whose activations are strongly related to the quality of the
research team. The numbers on each layer represent the number of units in that
layer (for the human capital unit, this is just 1).

Our architecture is constructed as follows. We first stack of 15 sets hidden
layers, each consisting of a 4096 or 2048 node layer, followed by a batch-norm
layer ([56]). These feed into a single unit — the “human capital” unit. This
layer forces the neural network to reduce the dimensionality of its
representations and distil the relevant features into a single scalar. The
human capital result is then concatenated with the publication date, and fed
into a series of independent sub-branches, one for each output being
predicted. The final layer effectively implements separate linear regressions
of the sort $y_{i}=\alpha+\mathbf{x}\beta$, meaning that the learned human-
capital representations can only be linearly re-scaled and offset in order to
make predictions about citations or journal quality.

In other words, our approach implements an encoder that maps the input to the
representation space, and a set of decoder regression models that map the
representation onto citation- and publication-related outcomes. Thus, during
training, the encoder is pushed to learn single-dimensional representations
that are informative of human capital outcomes.

###  5.2 Validating our estimates

To assess the success of our measures in evaluating human capital, we compare
their predictiveness across a range of outcomes, including citations received
at various points and journal quality rankings. In each case, our estimates
predict more than 55% of the variation in these measures, roughly 4-5 times as
much as other proxies that are commonly used (e.g. prior publications, prior
citations, h-index) (see Figure 3). In all cases, we are predicting out of
sample on a test set of a random sub-sample of 4,081 publications which was
held out from any of the training.111111Of these publications, incoming
citations after 1 year are known for 4,035 publications, incoming citations
after 2 years are known 3,724 publications, and the SJR values of the
publication venues are known for 1,312 publications.

(a) Our human capital estimates predicts key outcomes much better than
commonly used indicators (b) Our DNN achieves better accuracy compared to
separate lasso regressions

Figure 3. Evaluating our human capital measure. Figure (3a). Correlations
between human capital measures and publication outcomes for a hold-out set of
4,081 publications. Error bars indicate the 95 confidence interval. Prior
citations are the cumulative total citations received by the authors up until
the year prior to publishing the relevant publication (excluding self-
citations). H-indices and publication counts are evaluated at the year of
publication for each author. Journal rank here represents the ordinal value of
each journal in descending order of SJR-value. Figure (3b) shows the precision
(defined as $1/\text{MSE}$) of predictions of our DNN-based model and Lasso
regressions on the same hold-out set of 4,081 publications estimated
separately for each outcome.

These results indicate that our model has learned to predict bibliometric
outcomes of publications, and in doing so, it has inferred meaningful and
predictive human-capital features that can be measured as activation strength.
Finally, when restricting the dataset to just publications with junior
researchers (defined as publications with 2 or fewer prior publications), we
find that our human capital estimates are still highly predictive of each of
the bibliometric and publication-related outcomes (see Appendix H), while most
other proxies have little to no predictive power.

Thus far, we have shown that our measure has better predictive performance
relative to other commonly-used human capital predictors. However, these other
predictors also have access to much less data than our measure. For a more
equal comparison, we also ask how our measure compares to a Lasso regression
predictor—an approach more representative of linear approaches used in the
literature—with access to the same inputs as our neural network. To do so, we
evaluate our DNN on an out-of-sample test set. Our DNN represents a
substantial improvement relative to simpler approaches found in the literature
that rely on linear combinations of impact-based metrics, such as the h-index,
received publications, or publication counts. In particular, we obtain
prediction errors (measured in mean-square-error) that are at least 40% lower
for each outcome compared to Lasso regressions, and thus we get much more
precise predictions, as shown in Figure 3b.

While the preceding points to clear benefits with our approach, it is
essential to mention that our approach still has many of the limitations that
are true for many measures in the field. First, there is no natural unit for
human capital, and thus the cardinality of estimates is hard to interpret.
Second, it is unclear how citations and journal quality relates to actual
scientific merit, novelty, or insight. Thus, by using a measure predictive of
citations and journal quality we would implicitly be missing aspects of human
capital that cannot be inferred from these imperfect measures.

##  6 Empirical Analysis

Having validated our human capital measures, we combine them with the compute
data to estimate production functions for two important AI tasks: image
classification and object detection. In particular, we estimate individual
regression models and a pooled model described by equation 13. We estimate
using OLS, except where a Breusch-Pagan test finds the presence of
heteroskedascitity, in which case we estimate a GLS model by Maximum
Likelihood (details in Appendix J).

Data | Model | Estimates |  | Log
---
likelihood
|  |  | R&D elasticity
---
to capital ($\beta$)
| R&D elasticity to
---
human capital ($\gamma$)
Trend |
Image classification | A1 |  |  $\underset{\textstyle(0.021)}{0.111}$∗∗∗
---
|  $\underset{\textstyle(0.086)}{0.246}$∗∗∗
---
— | -48.798
A2 |  |  $\underset{\textstyle(0.029)}{0.140}$∗∗∗
---
|  $\underset{\textstyle(0.085)}{0.350}$∗∗∗
---
|  $\underset{\textstyle(0.014)}{0.051}$∗∗∗
---
-39.787
Object Detection | B1 |  |  $\underset{\textstyle(0.106)}{0.246}$${}^{*\ \ }$
---
|  $\underset{\textstyle(0.165)}{0.352}$${}^{*\ \ }$
---
— | -43.256
B2 |  |  $\underset{\textstyle(0.090)}{0.253}$${}^{**\ }$
---
| $\underset{\textstyle(0.158)}{0.319}$
---
| $\underset{\textstyle(0.030)}{0.013}$
---
-43.187
Table 4. Deep learning production function estimates. Estimation results for
image classification ($n=96$) and object detection ($n=40$). ${}^{*\ \ }$,
${}^{**\ }$, ∗∗∗denote p$<$0.05, p$<$0.01, p$<$0.001 respectively.

Our results for models A1-B2 are displayed in Table 4. We also estimate a
pooled model with distinct time-fixed effects, which combines data across both
computer vision tasks. A likelihood ratio test indicates that the pooled model
fits the data better than separately estimated models. The estimates of models
C1-C2 are displayed in Table 5.

Data | Model | Estimates |  | Log
---
likelihood
|  |  | R&D elasticity to
---
capital ($\beta$)
| R&D elasticity to
---
human capital ($\gamma$)
Trend |
Computer Vision (pooled) | C1 |  |  $\underset{\textstyle(0.022)}{0.145}$∗∗∗
---
|  $\underset{\textstyle(0.088)}{0.278}$${}^{**\ }$
---
— | -106.552
C2 |  |  $\underset{\textstyle(0.017)}{0.176}$∗∗∗
---
|  $\underset{\textstyle(0.076)}{0.298}$∗∗∗
---
|  $\underset{\textstyle(0.004)}{0.032}$∗∗∗
---
-95.586
Table 5. Pooled deep learning production function estimates. Estimation
results for pooled computer vision experiments ($n=136$). ${}^{**\ }$,
∗∗∗denote p$<$0.05, p$<$0.01, p$<$0.001 respectively.

For image classification, we estimate the R&D elasticity of capital ($\beta$)
is 0.111 (model A1) and 0.140 (model A2), as shown in Table 4. This means that
a 1% increase in the computational capital used for this type of R&D is
associated with a 0.111-0.140% increase in the rate of technological change.
For object detection, we estimate $\beta$ is 0.246 (for both model B1 and B2),
considerably higher than for image classification. For our pooled estimates,
we get estimates between 0.145 and 0.176 (see Table 5). All these results are
statistically significant at the 5% significance level; most are also
significant at the 0.1% level.

We find less variation in our estimates of the R&D elasticity of human capital
($\gamma$) between the two deep learning tasks. Our human capital elasticity
estimates for image classification are 0.246 (model A1) and 0.350 (model A2),
both significant at the 0.1% significance level. These estimates are just over
twice as high as those for computational capital. For object detection tasks,
the estimates of $\gamma$ are similar at 0.352 (model B1) and 0.319 (model
B2), though only the former of these two estimates is significant at the 5%
significance level. For our pooled model, we find estimates of the R&D
elasticity of human capital is 0.278 (model C1) and 0.298 (model C2), each
statistically significant at the 1% significance level.

Recall that with the standard economic growth model outlined in Section 2.3,
we can directly infer the equilibrium cost shares dedicated to capital and
labour from the relevant elasticities (assuming a competitive market). In
doing so, we find that the implied capital-cost estimates range from 0.29 and
0.44 (see Figure 4). Our estimates indicate that an optimizing firm in the R&D
sector should allocate between 29% and 44% of their total expenditure on
computational capital. Using our confidence intervals generated by
bootstrapping, we find that implied capital-cost estimates of statistically
significantly greater than 0.15 at the 5% significance level for all models—a
number that substantially exceeds most observed STEM R&D capital shares.

Figure 4. Implied optimal R&D expenditure breakdown. Implied capital-cost
shares given the estimates presented in tables 4 and 5, computed as
$\hat{\beta}/(\hat{\beta}+\hat{\gamma})$. Error bars represent 90% confidence
intervals generated by bootstrapping 10,000 iterations. We use the bias-
corrected percentile method for bootstrapping confidence intervals for ratios
outlined in [25].

We find that the implied capital cost share for object detection is higher
than for image classification by a margin of roughly ten percentage points.
However, this difference is not statistically significant. Overall, we see
that the implied R&D capital shares for AI are substantially higher than in
other areas of U.S. science and engineering Section 2.3, where capital share
generally falls below 20%.

##  7 R&D with deep learning

Having estimated the capital intensity of R&D that is augmented with AI, we
analyse the potential productivity effects that the widespread adoption of
deep learning would have on economic productivity and growth. To do so, we
suppose that the widespread adoption of deep learning would act as a one-time
shock, raising the capital intensity of knowledge production in the economy to
the levels we estimated in computer vision.

Along the balanced growth path, the steady-state growth rate in the stock of
knowledge is described by equation 4. Using this, we can substitute in the
parameter values implied by our empirical estimates from the prior section
into our semi-endogenous growth model and compute the predicted change in R&D
productivity growth conditional on the widespread of adoption of deep
learning. As shown in Figure 5, depending on the model specification, the
results from image classification would imply a productivity growth rate
between 1.6% and 1.8%, whereas those from object detection would imply a rate
between 3.1% and 3.9%. Our preferred estimate, both because more data inform
it and the estimates are more precise, is the pooled estimate for computer
vision. With the widespread adoption of deep learning raising ideas production
in the economy to this level of capital intensity, we would expect the
productivity growth rate to rise to between 2.1% and 2.4%. To put that in
context, this would amount to increase of between 1.7- and 2-fold relative to
the 1.2% average U.S. productivity growth from 1948 to 2021, and a 2.6- to
3-fold increase against the post-2000 0.8% growth ([32]). Thus, our results
indicate that if adopting deep learning in other areas of R&D allows those
areas to leverage capital better in the same way that computer vision has, it
will represent a substantial acceleration of scientific progress.

Figure 5. Predicted productivity growth under widespread deployment of AI in
R&D. Steady-state productivity growth as a function of the implied capital-
cost share in a competitive R&D industry when the elasticity of R&D output to
the stock of ideas ($\theta$) is $1/2$ and the elasticity of R&D output to
labour inputs ($\gamma$) is $2/5$. Markers indicate point estimates of implied
optimal R&D expenditure with deep learning according to models A1-C2 as
estimated in section 6. “Current region” indicates the current level of
capital intensity of R&D according to NSF data, which our semi-endogenous
growth model predicts to result in 0.5% to 1.3% productivity growth, a level
consistent with observed recent US productivity growth.

##  8 Robustness and external validity

In this section, we show that the results in Section 7 are robust to outliers
and to different choices of how granular or coarse-grained time periods are
specified. Moreover, we show that our assumptions about the substitutability
of labour and capital are consistent with our data, which supports a key
assumption required for our inferences about the capital-intensity of deep
learning from our estimated elasticities. Finally, we discuss the
generalizability of estimates from computer vision to other R&D tasks.

###  8.1 Sensitivity to outliers

It is known, that certain empirical results from high-profile studies can be
reversed by removing less than 1% of the sample even when standard errors are
small ([23]). In this section, we assess the sensitivity of our results to
outliers by showing that the removal of a small fraction of the data is not
determinative of our findings. To test the robustness of our results to the
removing of samples, we re-run our analysis between around $10^{4}$ and
$10^{6}$ times (depending on which model is re-estimated) on random sub-
samples of our datasets that excludes a fraction of our observations. The
point estimates are plotted in Figure 6.

Figure 6. Estimates after removing a random 5% of dataset Median estimates
when taking a random sub-sample of our dataset that excludes 1.6% to 5% of the
total number of observations. Results are displayed as violin plots, using
kernel density estimation to create the distributions. Inside the violins, the
box plots show median and interquartile ranges.

For our dataset on object detection, we estimated models on all possible sub-
samples that exclude 3 observations (of which there are
$\frac{40!}{37!3!}=9880$). For our dataset on image classification and for our
pooled dataset, we estimated models on $10^{6}$ random sub-samples that
exclude 5 observations. The sub-samples considered covers around 1.6% of all
possible sub-samples for the image classification dataset and 0.3% of all sub-
samples for the pooled dataset and is, therefore, a non-trivial fraction of
all possible permutations.

We find that the point estimates are mostly robust to the removal of any small
subset of observations, as we see that most estimates are tightly clustered
around their median value, particularly for the estimates for which our
datasets are the largest (namely, image classification and our pooled
dataset). Moreover, the point estimates are consistent with the estimates
found in our baseline empirical analysis presented in Section 6, indicating
that our estimates are not the product of a small number of outliers.

###  8.2 Alternative model specifications

Our estimation strategy for $A_{t}$ (the stock of knowledge) takes advantage
of cross-sectional variation at each time point. That is, we effectively pool
publications into groups of contemporaries published around the same time. We
then estimate the variation in performance due to changes in inputs amongst
these contemporaries—and suppose that this variation is due to inputs rather
than changes in $A_{t}$. However, papers are published continuously over time,
so this necessarily involves a bias-variance trade-off: if we specify more
granular time periods (such as months instead of years), it gives more
variance, but this will mean that each interval is estimated with fewer data
points, making it noisier and more prone to over-fitting. In our empirical
analysis, we balanced this trade-off by fixing time periods as yearly
intervals. In what follows, we show that our conclusions are robust to
different reasonable choices of how granular or coarse-grained periods are
specified.

Data |  | Time-period
---
length
Estimates | Estimates |  | Log
---
likelihood
|  |  | R&D elasticity to
---
capital ($\beta$)
| R&D elasticity to
---
human capital ($\gamma$)
Image classification | 6 |  |  $\underset{\textstyle(0.017)}{0.099}$∗∗∗
---
|  $\underset{\textstyle(0.088)}{0.250}$${}^{**\ }$
---
-47.379
12 |  |  $\underset{\textstyle(0.021)}{0.111}$∗∗∗
---
|  $\underset{\textstyle(0.086)}{0.246}$${}^{*\ \ }$
---
-48.799
18 |  |  $\underset{\textstyle(0.029)}{0.155}$∗∗∗
---
|  $\underset{\textstyle(0.089)}{0.332}$∗∗∗
---
-75.380
Object detection | 6 |  |  $\underset{\textstyle(0.095)}{0.215}$${}^{*\ \ }$
---
|  $\underset{\textstyle(0.139)}{0.400}$${}^{**\ }$
---
-43.256
12 |  |  $\underset{\textstyle(0.096)}{0.246}$${}^{*\ \ }$
---
|  $\underset{\textstyle(0.165)}{0.352}$${}^{*\ \ }$
---
-43.256
18 |  |  $\underset{\textstyle(0.111)}{0.214}$${}^{\ \ \ }$
---
|  $\underset{\textstyle(0.204)}{0.348}$${}^{\ \ \ }$
---
-44.030
Computer vision (pooled) | 6 |  |  $\underset{\textstyle(0.016)}{0.150}$∗∗∗
---
|  $\underset{\textstyle(0.072)}{0.294}$∗∗∗
---
-74.276
12 |  |  $\underset{\textstyle(0.020)}{0.132}$∗∗∗
---
|  $\underset{\textstyle(0.075)}{0.245}$${}^{**\ }$
---
-87.288
18 |  |  $\underset{\textstyle(0.020)}{0.142}$∗∗∗
---
|  $\underset{\textstyle(0.082)}{0.218}$${}^{**\ }$
---
-95.839

Table 6. Estimation results for separate models with alternative window
lengths. Estimates of models A1, B1, and C1 with different window lengths.
Specifications are the same as in the main analysis in Section

6.

We re-estimate models A1-C2 with window lengths 6, 12, and 18 months and
compare our estimates to those obtained in Section 6. We find that the
estimates are mostly similar for all relevant datasets (see Table 6 for
estimates for models A1-C1, and appendix Appendix K for re-estimates of models
A2-C2). Moreover, similar patterns remain: estimates of the R&D elasticity to
capital ($\beta$) are relatively lower for image classification tasks than for
object detection tasks. These estimates strongly suggest that our key
estimates are robust to different choices of how granular or coarse-grained
time periods are specified.

###  8.3 Sensitivity to model assumptions about the substitutability of human
scientists

In our semi-endogenous growth model, we assume that the elasticity of
substitution equals 1 (i.e. is modelled by a Cobb-Douglas production
function). Hence, we assume a substantial level of substitutability of human
scientists for compute. In this section, we test: (i) whether our results
would continue to hold if there was a lesser degree of substitutability of
human scientists, and (ii) whether the data is consistent with a higher or
lower level of substitutability.

The substitutability assumption is important because our semi-endogenous
growth model implies that compute stock will grow faster than the stock of
scientists. To see this, recall that from equation 4, the steady-state growth
of capital dedicated to R&D is $\frac{1-\theta+\gamma}{1-\beta-\theta}n$
(which, with the estimates used in Section 2.2., this would be $\sim 2n$,
while the stock of scientists grows just at the rate $n$). Hence, we should
expect that, in the fullness of time, the stock of specialized capital goods,
$C(t)$, will be substantially greater than the stock of scientists
$S(t)$.121212This conclusion is consistent with what we see in our empirical
data as well as what has been found in other areas of computing ([105]).

If human scientists were more difficult to substitute with compute than we
have assumed, the optimal investment path might involve larger investments in
human scientists. We may ask: How much weaker of an assumption can we make for
our conclusions to still follow, and is our assumption about the
substitutability of human scientists reasonable? The first part is
straightforward to answer. Suppose the idea production function instead
followed a more general constant elasticity of substitution production
function:

$\dot{A}(t)=A(t)^{\theta}[\gamma S(t)^{\frac{\sigma-1}{\sigma}}+\beta
C(t)^{\frac{\sigma-1}{\sigma}}]^{\frac{\sigma}{\sigma-1}},$ (15)

where $\sigma$ denotes the elasticity of substitution between compute and
scientists. In this framework, assuming a competitive R&D sector, the share of
expenditure dedicated to compute (which we will denote by $f$) is given by:

$f=\frac{\beta C(t)^{\frac{\sigma-1}{\sigma}}}{\beta
C(t)^{\frac{\sigma-1}{\sigma}}+\gamma S(t)^{\frac{\sigma-1}{\sigma}}}.$ (16)

From this expression, we can see that in the typical Cobb-Douglas case (where
$\sigma=1$), we have that $f=\beta/(\beta+\gamma)$. Note moreover, that
whenever $\sigma\geq 1$ it must be that $f\geq\beta/(\beta+\gamma)$, while
when $\sigma<1$ it must be that $f<\beta/(\beta+\gamma)$. Hence, if scientists
are less easily substitutable than we supposed, the capital intensity of R&D
will be lower than our estimates imply.

Hence, it is important to investigate whether $\sigma=1$ is a reasonable level
of substitutability to assume. We investigate this by estimating equation 15.
Note that we can rewrite 15 by dividing by $A(t)$ as follows:

${g}_{t}=A_{t}^{\theta-1}[\gamma S_{t}^{\rho}+\beta
C_{t}^{\rho}]^{\frac{1}{\rho}},$ (17)

where $\rho\equiv\frac{\sigma-1}{\sigma}$. To simplify the estimation
procedure, we can approximate this expression using the second-order McLaurin
expansion (i.e. the Taylor series evaluated at $\sigma=1$),

$\log{g}_{t}\approx(\theta-1)\log A_{t}+\gamma\log S_{t}+\beta\log
C_{t}+\frac{1}{2}\rho\gamma\beta[\log S_{t}-\log C_{t}]^{2},$ (18)

This is simply the translog production function, which has the well-known
advantage that it is linear in its parameters and, therefore, estimable using
OLS. Because of this, this approximation is widely used in similar settings
([40, 19]). In our case, the empirical model we estimate becomes:

$\log\tilde{g}_{it}=(\theta-1)\log A_{t}+\gamma\log S_{it}+\beta\log
C_{it}+\frac{1}{2}\rho\gamma\beta[\log S_{it}-\log C_{it}]^{2}+\epsilon_{it},$
(19)

where each variable has its usual meanings it had in Section 4.

Figure 7. Elasticity of substitution estimates Kernel densities of the
estimates of $\sigma$ across each of our main models generated by
bootstrapping 10,000 iterations.

Figure 6 plots the distributions generated by bootstrapping estimates of
$\sigma$ in 19 for each of our models A1-C2. Our estimates of $\sigma$ are
very tightly clustered around unity. These findings reinforce our assumption
that $\sigma=1$ is reasonable and, therefore, our inferences about capital
intensity are consistent with the data. Moreover, for the tasks for which we
have the most data, we observe that our estimates $\sigma$ are slightly above
$1$. Hence, for these tasks, it is likely that, if anything, our inferred
level of capital intensity of deep learning-based R&D is an underestimate of
the true level after adequately accounting for the degree of substitutability
of human scientists.

###  8.4 Limits to external validity stemming from our choice of domain

Our analysis focuses exclusively on two computer vision tasks. Although this a
small segment of scientific and engineering problems that deep learning might
be applied to, there are good reasons to expect that the key insights gained
from studying a wide range of architectures for computer vision to broadly
could carry over to a wider range of R&D problems.

One broad consideration in favour of this view is that deep learning often
builds on common techniques, algorithms, and similar architectures across
different subfields (see e.g. [38, Ch. 3]). Across many domains, deep learning
systems are based on similar ideas and implemented using techniques and
algorithms. At a high level, almost all modern deep learning
systems—independently of the modality or task these are trained for—are all
some ‘deep’ computational graph with many parameters that are learned through
gradient descent along gradients of some loss function computed by
backpropagation. Indeed it is widely considered (e.g. by [8]) that one of the
first instances in which these key features of deep learning were all
instantiated was with AlexNet in 2012, a model that is included in our
dataset.

There certainly are some pronounced architectural divides across domains. For
example, convolutional neural networks are widespread in computer vision,
while transformer-based models are ubiquitous in machine translation. However,
these architectural differences are also represented in our data. Indeed, our
data includes convolutional neural networks, vision transformers, and
architectures based purely on multi-layer perceptrons. Hence, much variation
between modalities and tasks that exists within modern deep learning is
reflected in the data that we consider.

Some neural network architectures—such as the transformer, which is well-
represented in our dataset—have been shown to operate effectively across
domains, modalities and tasks (see, e.g. [88]). This suggests that the upshot
of our findings may well generalize to domains outside of computer vision.
What is more, approaches like the transformer are often amongst the state-of-
the-art techniques across many R&D-adjacent tasks, such as code generation
([72]), cheminformatics ([57]), and bioinformatics ([31]).

Moreover, work on neural scaling laws suggests that the relation between the
model’s size and performance scales according to the usual power-law
independently of the domain the model is trained to handle. In fact, [46] find
that for transformer models, there is a remarkable near-universal relation
between the optimal model size and the compute budget across a range of
domains spanning images, language, mathematics, video and more, which supports
the notion the role of compute depends strongly on the type of technique or
architecture, not the type of domain in which these techniques are applied.
For these reasons, we expect that the key insights gained from studying a wide
range of architectures for computer vision to broadly carry over to a wider
range of R&D problems to which deep learning is applied.

###  8.5 How data availability influences our estimates

One reason to expect that our results may fail to apply to a broader set of
problems is that we studied a set of problems within computer vision with a
relative abundance of quality labelled data. In some domains—such as the
problem of protein folding (where the crystal structures of proteins are
expensive and arduous to generate) high-quality data might be less readily
available or expensive to generate. As a result of the relative abundance of
data in computer vision, the returns to compute might be higher than they
would be in lower-data-abundance regimes. To illustrate this, consider a
simple model where deep learning system performance can be described as a
constant elasticity production function in data $D$ and compute $C$:

$P=\big{[}\zeta D^{\frac{\sigma-1}{\sigma}}+\eta
C^{\frac{\sigma-1}{\sigma}}\big{]}^{\frac{\sigma}{1-\sigma}},\hskip
4.26773pt\text{where}\hskip 4.26773pt\sigma,\hskip 1.42271pt\zeta,\hskip
1.42271pt\eta>0.$ (20)

It becomes clear whenever data and computation are gross complements
($\sigma<1$), then, the returns to compute will be lower in low-data regimes
compared to high-data regimes:

$\frac{\partial P}{\partial C}\bigg{|}_{D_{\text{Low}}}=\eta
C^{\frac{\sigma-1}{\sigma}-1}\big{[}\zeta
D_{\text{Low}}^{\frac{\sigma-1}{\sigma}}+\eta
C^{\frac{\sigma-1}{\sigma}}\big{]}^{\frac{\sigma}{1-\sigma}-1}\leq\frac{\partial
P}{\partial C}\bigg{|}_{D_{\text{High}}}=\eta
C^{\frac{\sigma-1}{\sigma}-1}\big{[}\zeta
D_{\text{High}}^{\frac{\sigma-1}{\sigma}}+\eta
C^{\frac{\sigma-1}{\sigma}}\big{]}^{\frac{\sigma}{1-\sigma}-1}.$ (21)

This suggests that the returns to compute in high-data regimes can be
unusually high. Moreover, empirical evidence has shown that AI-based ideas
production is rapidly expanding in data-rich sectors such as investment
management ([2]), and that computer vision firms with access to additional
data are more innovative ([18]). Might our results, therefore, fail to
generalize to low-data regimes? Our sense is that it might very well
generalize, particularly for economically important R&D tasks.

One piece of evidence for this view comes from work on neural scaling laws for
language modelling. [51] derive a parametric loss function in which the amount
of data and the model size enters additively separably. Although this result
is derived in the context of language modelling, where data is abundant, this
does suggest that data and compute can substitute for data by applying it to
train larger models, at least when this training is done appropriately.

Moreover, we might expect that for economically important R&D tasks,
complementary investments in generating the necessary datasets to train
machine learning models will be made. For these tasks, we might expect such
investments to produce high-fidelity physical simulations, high-quality
synthetic datasets, the proliferation of sensors, and higher-throughput
measurement apparatuses. Hence, insofar as we expect R&D tasks to be of
considerable economic importance, we might expect that low-data regimes are to
be short-lived or otherwise atypical. For such tasks, a relative abundance of
data may be representative.

##  9 Discussion

###  9.1 Summary and implications

Our main contributions are as follows. First, we provide a framework for
understanding the impact of two important trends: i) the recent breakthroughs
using deep learning in R&D, and ii) the rapid scaling of computation in deep
learning systems. We show that if deep learning is widely adopted in the R&D
sector and induces high returns to computational capital, then technological
change will, under suitable conditions, permanently accelerate.

Secondly, using data from two computer vision tasks that are considered key
test-beds for deep learning, we produce empirical estimates that imply that
deep learning is more capital-intensive than other forms of R&D; indeed an
optimizing firm would dedicate between 29% and 44% of their total R&D
expenditure on (computing) capital. This result implies a clear empirical
prediction: If deep learning is widely adopted in R&D and this increases the
returns to computational capital, then technological change will permanently
accelerate. Consequentially, according to semi-endogenous growth theory, we
will also get an acceleration of economic growth.

Thirdly, we make a methodological contribution by introducing a novel machine
learning-based method for inferring human capital from scientific publications
using an encoder to compress the inputs about the authors into a latent-space
representation. In our approach, this encoder is tasked with learning
representations of human capital that are most predictive of publication and
citation-related outcomes that we have independent reasons to expect to be
indicative of the quality of human capital. This human capital estimation
predicts key outcomes 4-5 times more accurately than typical approaches in the
literature.

Our work identifies three areas of future work that we expect to be fruitful.
First, it would be valuable to better understand the diffusion of deep
learning techniques within the economy, particularly within R&D. We know that
deep learning is diffusing to domains as diverse as protein-folding ([62]),
semiconductor chip design ([77]), and programming ([72]). But how quickly this
diffusion is happening and what drives it is poorly understood and micro-data
charting it is lacking. Such analyses would inform us about the transition
dynamics of AI-augmented R&D across different industries. In particular,
slower diffusion would imply a more protracted transition to the higher growth
rates identified in Section 7. Such analyses might also highlight areas where
R&D cannot be augmented with current AI techniques, implying more modest
macro-level productivity improvements.

Another valuable area for research would be to look at whether improvements in
AI techniques tend to make R&D more capital-intensive. If true, the
productivity gains could be more drastic. For instance, [3] shows that
continually-increasing capital intensity in R&D would result in ideas
production becoming fully automated. So long as $\theta>0$ in Equation 2, this
would produce unbounded growth. While we cannot analyze the possibility of
this increase without further data or strong assumptions on our model, it
seems worth studying whether the capital-intensity of deep learning varies
with time.

A final area for follow-up work is how AI-augmented R&D makes deeper changes
to the knowledge production. While we have made progress investigating the
relative importance of inputs to knowledge production and the implications
this could have for the rate of technological change, many questions remain.
In our work, we assumed that knowledge produced by labs using deep learning
are essentially the same as that produced by unaugmented labs. Recall that we
supposed that knowledge production was modelled as:

$\dot{A}(t)=A(t)^{\theta}S(t)^{\gamma}C(t)^{\beta},$ (22)

which implicitly assumes that $A(t)^{\theta}$—the intertemporal knowledge
spillover for technological opportunities—is the same whether or not this
knowledge was produced by AI-augmented R&D or not. There might be reasons to
be sceptical of this assumption in either direction. AI-augmented R&D
knowledge might be hard to share because deep learning is famously a ‘black
box’ technique that does not provide easy intuitions about what is being done.
On the other hand, there is evidence that deep learning systems can learn
essential building blocks (e.g. laws of nature ([110])) and that machine
learning artefacts such as models are easy to disseminate widely (e.g. via
Hugging Face ([113])). These characteristics might make AI-augmented R&D
easier to diffuse, as has already been seen with the easy ability to adapt,
fine-tune, or use existing models as ‘backbones’ for adjacent problems. While
it is unclear whether knowledge produced by AI-augmented R&D has smaller or
more significant spillovers, we expect it is worth investigating whether the
extent of such spillovers depends on the technology used to generate
knowledge.

###  9.2 Selection issues and measurement error

The data used in our empirical work may suffer from selection bias issues that
are important to consider. Our dataset contains just models for which authors
reported enough to infer how the model was trained and how much computation
was needed. We might expect such reporting to be more common among
computationally-intensive papers or papers whose performance is cutting-edge,
which could introduce bias into our estimates—although it is not clear that
this bias would be significant. If it did, it would likely positively bias our
estimates of the returns to physical capital due to two effects. First, we
suspect that publications that report these details are more likely to use
large amounts of computation. Second, because the cost of deploying hardware
for machine learning training is expensive (see e.g. [98]), large compute
deployments are likely associated with more effort to optimise training runs
that ensures that these resources are used efficiently. Such efficiencies
could lead to those deployments being better able to leverage computation to
achieve particular results. As a consequence, our data would
disproportionately be composed of those using computation more effectively.

There are reasons to expect that such bias would be small. First, leading
machine learning conferences require or at least encourage researchers to
report details about hardware usage.131313NeurIPS, which is one of the leading
machine learning conferences, is an example of a conference that, as part of
their mandatory checklist, currently includes questions about the hardware
that is used (see the NeurIPS 2021 Paper Checklist Guidelines). Requirements
and norms such as these reduce the selection bias that stems from the fact
that our dataset only contains models where the researchers reported compute-
relevant information. Second, the implementation of training runs are likely
to be fairly similar across papers, as researchers typically use one of only a
few types of open-source software that have distributed training settings and
profilers that might be broadly comparable in achieving utilization. Moreover,
we address this selection issue by using two different methods for inferring
the amount of computation that was used in training (so that we could capture
a wider range of models than would otherwise be possible). Hence, although
selection bias might be introduced by the machine learning experiments
included in our dataset, it is unclear whether this is a significant issue.
Additionally, the variation in the amount of compute used spans over four
orders of magnitude and will therefore dominate the variance in utilization
rates, which suggests that this bias is likely to be small. Even if our
estimate for the effectiveness of compute is overestimated, it is unclear
whether this would imply that our estimate of capital share is over-estimated.
This is because the expertise needed to optimize training runs will likely be
captured in our human capital estimates so that these would also be
overestimated, and it is unclear which effect would be more significant.

There are also potential measurement-error issues with our estimates of the
amounts of training computation to train machine learning models. Firstly, we
estimate only the amount of computation used in training the model in the
final training run, not all training runs. It is common for deep learning
experiments to perform many (usually smaller) trial runs. These trial runs
help inform the selection of hyper-parameters—parameters whose values control
the learning process—used in the final training run. Our estimates of the
compute that is used to train deep learning models are, therefore, likely to
be a fraction of the total computation that was used. If this were a constant
fraction, this would not bias our estimates of the relevant elasticities. It
is currently not clear to the authors whether larger training runs use
relatively more compute in selecting hyper-parameters than smaller training
runs (in proportional terms). Regardless of the direction of this effect, the
ratio of total computation used to the amount of computation used in the final
training run will not be equal across experiments. This introduces noise in
our estimates of elasticities, which will attenuate our estimates of the
elasticity of computational capital downward.

Similar attenuation bias might be introduced in the process of estimating the
computational inputs used to train deep learning models, as these rely on
various approximations. To reduce the variance of our estimates’ variance, we
used multiple methods for producing these estimates and took the average of
each. Given that the variation in the amount of compute used spans many orders
of magnitude, minor errors due to the use of approximations are likely to be
washed out. Nevertheless, our use of approximate techniques in estimations
could have attenuated our estimates, which would mean that our results are
underestimates.

##  10 Conclusion

Much has been written on the potential mechanisms through which AI impacts
productivity and output growth. Motivated by recent trends in deep learning,
we provide empirical evidence for one of these mechanisms: the ability for
scientists to efficiently harness more computational capital in R&D. We argue
that this mechanism is consequential because standard endogenous growth models
imply that greater capital intensity in R&D produces permanent increases in
productivity growth and economic growth. We present evidence that deep
learning-based computer vision research is significantly more capital-
intensive than virtually all other R&D sectors in the U.S. If, as we argue,
deep learning has similar impacts in other areas of R&D, then the widespread
adoption of deep learning-based R&D could double U.S. productivity growth
rates.

## References

  * [1] Yasser Abdih and Frederick Joutz  “Relating the knowledge production function to total factor productivity: an endogenous growth puzzle”  In _IMF Staff Papers_ 53.2 Springer, 2006, pp. 242–271
  * [2] Simona Abis and Laura Veldkamp  “The Changing Economics of Knowledge Production”  In _SSRN Electronic Journal_ , 2020  DOI: 10.2139/ssrn.3570130
  * [3] Philippe Aghion, Benjamin F Jones and Charles I. Jones  “Artificial Intelligence and Economic Growth”  In _The Economics of Artificial Intelligence: An Agenda_ University of Chicago Press, 2019, pp. 237–282
  * [4] Ajay Agrawal  “Introduction by Ajay Agrawal”  In _NBER Economics of AI Conference 2017_ , 2022  URL: https://www.economicsofai.com/nber-conference-toronto-2017
  * [5] Ajay Agrawal, Joshua Gans and Avi Goldfarb  “Prediction machines: the simple economics of artificial intelligence”  Harvard Business Press, 2018
  * [6] Ajay Agrawal, John McHale and Alexander Oettl  “Finding Needles in Haystacks: Artificial Intelligence and Recombinant Growth”  In _The Economics of Artificial Intelligence: An Agenda_ University of Chicago Press, 2019, pp. 149–174
  * [7] Nur Ahmed and Muntasir Wahed  “The de-democratization of ai: Deep learning and the compute divide in artificial intelligence research”  In _arXiv preprint arXiv:2010.15581_ , 2020
  * [8] Md Zahangir Alom, Tarek M Taha, Christopher Yakopcic, Stefan Westberg, Paheding Sidike, Mst Shamima Nasrin, Brian C Van Esesn, Abdul A S Awwal and Vijayan K Asari  “The history began from AlexNet: A comprehensive survey on deep learning approaches”  In _arXiv preprint arXiv:1803.01164_ , 2018
  * [9] Massimo Aria and Corrado Cuccurullo  “bibliometrix: An R-tool for comprehensive science mapping analysis”  In _Journal of informetrics_ 11.4 Elsevier, 2017, pp. 959–975
  * [10] Timothy G Armstrong, Justin Zobel, William Webber and Alistair Moffat  “Relative significance is insufficient: Baselines matter too”  In _Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation_ , 2009, pp. 25–26
  * [11] “arXiv.org”, 2022  URL: https://arxiv.org/
  * [12] Pierre Azoulay, Christian Fons-Rosen and Joshua S Graff Zivin  “Does science advance one funeral at a time?”  In _American Economic Review_ 109.8, 2019, pp. 2889–2920
  * [13] Pierre Azoulay, Toby Stuart and Yanbo Wang  “Matthew: Effect or fable?”  In _Management Science_ 60.1 INFORMS, 2014, pp. 92–109
  * [14] Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee and Utkarsh Sharma  “Explaining neural scaling laws”  In _arXiv preprint arXiv:2102.06701_ , 2021
  * [15] Kyung Hwan Baik  “Difference-form contest success functions and effort levels in contests”  In _European Journal of Political Economy_ 14.4 Elsevier, 1998, pp. 685–701
  * [16] Mikhail Belkin, Daniel Hsu, Siyuan Ma and Soumik Mandal  “Reconciling modern machine learning practice and the bias-variance trade-off”  In _arXiv preprint arXiv:1812.11118_ , 2018
  * [17] Yoshua Bengio, Aaron Courville and Pascal Vincent  “Representation learning: A review and new perspectives”  In _IEEE transactions on pattern analysis and machine intelligence_ 35.8 IEEE, 2013, pp. 1798–1828
  * [18] Martin Beraja, David Y. Yang and Noam Yuchtman  “Data-intensive Innovation and the State: Evidence from AI Firms in China” Series: Working Paper Series, 2020  DOI: 10.3386/w27723
  * [19] Ernst R. Berndt and Laurits R. Christensen  “The translog function and the substitution of equipment, structures, and labor in U.S. manufacturing 1929-68”  In _Journal of Econometrics_ 1.1, 1973, pp. 81–113  DOI: 10.1016/0304-4076(73)90007-9
  * [20] Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai and Aäron van den Oord  “Are we done with imagenet?”  In _arXiv preprint arXiv:2006.07159_ , 2020
  * [21] Stefano Bianchini, Moritz Muller and Pierre Pelletier  “Deep Learning in Science”  In _arXiv:2009.01575 [cs, econ]_ , 2020  arXiv: http://arxiv.org/abs/2009.01575
  * [22] Stefano Breschi, Francesco Lissoni and Gianluca Tarasconi  “Inventor data for research on migration and innovation: a survey and a pilot”  WIPO, 2014
  * [23] Tamara Broderick, Ryan Giordano and Rachael Meager  “An Automatic Finite-Sample Robustness Metric: When Can Dropping a Little Data Make a Big Difference?”  In _arXiv preprint arXiv:2011.14999_ , 2020
  * [24] Tom Brown et al.  “Language models are few-shot learners”  In _Advances in neural information processing systems_ 33, 2020, pp. 1877–1901
  * [25] Marion K Campbell and David J Torgerson  “Bootstrapping: estimating confidence intervals for cost-effectiveness ratios”  In _Qjm_ 92.3 Oxford University Press, 1999, pp. 177–182
  * [26] Iain M. Cockburn, Rebecca Henderson and Scott Stern  “4. The Impact of Artificial Intelligence on Innovation: An Exploratory Analysis”  In _4\. The Impact of Artificial Intelligence on Innovation: An Exploratory Analysis_ University of Chicago Press, 2019, pp. 115–148  DOI: 10.7208/9780226613475-006
  * [27] Nicholas Crafts  “Artificial intelligence as a general-purpose technology: an historical perspective”  In _Oxford Review of Economic Policy_ 37.3, 2021, pp. 521–536  DOI: 10.1093/oxrep/grab012
  * [28] Dirk Czarnitzki, Kornelius Kraft and Susanne Thorwarth  “The knowledge production of ’R’ and ’D”’  In _Economics Letters_ 105.1, 2009, pp. 141–143  DOI: 10.1016/j.econlet.2009.06.020
  * [29] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li and Li Fei-Fei  “Imagenet: A large-scale hierarchical image database”  In _2009 IEEE conference on computer vision and pattern recognition_ , 2009, pp. 248–255  Ieee
  * [30] “Dimensions AI”  In _Dimensions_ URL: https://www.dimensions.ai/
  * [31] Ahmed Elnaggar et al.  “ProtTrans: towards cracking the language of Life’s code through self-supervised deep learning and high performance computing”  In _arXiv preprint arXiv:2007.06225_ , 2020
  * [32] San Francisco Fed  “Total factor productivity”  In _San Francisco Fed_ Federal Reserve Bank of San Francisco, 2022  URL: https://www.frbsf.org/economic-research/indicators-data/total-factor-productivity-tfp/
  * [33] Raymond Fisman, Jing Shi, Yongxiang Wang and Rong Xu  “Social ties and favoritism in Chinese science”  In _Journal of Political Economy_ 126.3 University of Chicago Press Chicago, IL, 2018, pp. 1134–1171
  * [34] Elizabeth Gibney  “Self-taught AI is best yet at strategy game Go” Publisher: Nature Publishing Group  In _Nature_ , 2017  DOI: 10.1038/nature.2017.22858
  * [35] Avi Goldfarb, Bledi Taska and Florenta Teodoridis  “Could Machine Learning be a General Purpose Technology? A Comparison of Emerging Technologies Using Data from Online Job Postings”  In _NBER working paper_ , 2022
  * [36] Gang Gong, Alfred Greiner and Willi Semmler  “Endogenous growth: Estimating the Romer model for the US and Germany”  In _Oxford Bulletin of Economics and Statistics_ 66.2 Wiley Online Library, 2004, pp. 147–164
  * [37] Borja González-Pereira, Vicente P Guerrero-Bote and Félix Moya-Anegón  “A new approach to the metric of journals’ scientific prestige: The SJR indicator”  In _Journal of informetrics_ 4.3 Elsevier, 2010, pp. 379–391
  * [38] Ian Goodfellow, Yoshua Bengio and Aaron Courville  “Deep learning”  MIT press, 2016
  * [39] Gene M Grossman and Elhanan Helpman  “Endogenous innovation in the theory of growth”  In _Journal of Economic Perspectives_ 8.1, 1994, pp. 23–44
  * [40] David K. Guilkey, C.. Lovell and Robin C. Sickles  “A Comparison of the Performance of Three Flexible Functional Forms”  In _International Economic Review_ 24.3, 1983, pp. 591–616  DOI: 10.2307/2648788
  * [41] Bronwyn H. Hall, Zvi Griliches and Jerry A. Hausman  “Patents and R&D: Is There A Lag?” Series: Working Paper Series, 1988, pp. 265–83  DOI: 10.3386/w1454
  * [42] Trevor Hastie, Robert Tibshirani, Jerome H Friedman and Jerome H Friedman  “The elements of statistical learning: data mining, inference, and prediction”  Springer, 2009
  * [43] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun  “Deep residual learning for image recognition”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778
  * [44] Christian Helmers and Henry G Overman  “My precious! The location and diffusion of scientific research: evidence from the Synchrotron Diamond Light Source”  In _The Economic Journal_ 127.604 Wiley Online Library, 2017, pp. 2006–2040
  * [45] Dan Hendrycks and Kevin Gimpel  “Gaussian error linear units (gelus)”  In _arXiv preprint arXiv:1606.08415_ , 2016
  * [46] Tom Henighan et al.  “Scaling laws for autoregressive generative modeling”  In _arXiv preprint arXiv:2010.14701_ , 2020
  * [47] Joel Hestness et al.  “Deep learning scaling is predictable, empirically”  In _arXiv preprint arXiv:1712.00409_ , 2017
  * [48] Geoffrey E Hinton and Ruslan R Salakhutdinov  “Reducing the dimensionality of data with neural networks”  In _science_ 313.5786 American Association for the Advancement of Science, 2006, pp. 504–507
  * [49] Jorge E Hirsch  “An index to quantify an individual’s scientific research output”  In _Proceedings of the National academy of Sciences_ 102.46 National Acad Sciences, 2005, pp. 16569–16572
  * [50] Jack Hirshleifer  “Conflict and rent-seeking success functions: Ratio vs. difference models of relative success”  In _Public choice_ 63.2 Springer, 1989, pp. 101–112
  * [51] Jordan Hoffmann et al.  “Training Compute-Optimal Large Language Models”  In _arXiv preprint arXiv:2203.15556_ , 2022
  * [52] Torsten Hothorn, Friedrich Leisch, Achim Zeileis and Kurt Hornik  “The design and analysis of benchmark experiments”  In _Journal of Computational and Graphical Statistics_ 14.3 Taylor & Francis, 2005, pp. 675–699
  * [53] Peter Howitt  “Steady Endogenous Growth with Population and R & D Inputs Growing”  In _Journal of Political Economy_ 107.4, 1999, pp. 715–730  DOI: 10.1086/250076
  * [54] Peter Howitt and Philippe Aghion  “Capital Accumulation and Innovation as Complementary Factors in Long-Run Growth”  In _Journal of Economic Growth_ 3.2, 1998, pp. 111–130  DOI: 10.1023/A:1009769717601
  * [55] “Institutional Publication Metrics for Computer Science”  URL: http://csmetrics.org/
  * [56] Sergey Ioffe and Christian Szegedy  “Batch normalization: Accelerating deep network training by reducing internal covariate shift”  In _International conference on machine learning_ , 2015, pp. 448–456  PMLR
  * [57] Ross Irwin, Spyridon Dimitriadis, Jiazhen He and Esben Jannik Bjerrum  “Chemformer: a pre-trained transformer for computational chemistry”  In _Machine Learning: Science and Technology_ 3.1 IOP Publishing, 2022, pp. 015022
  * [58] Hao Jia and Stergios Skaperdas  “Technologies of conflict”  In _The Oxford Handbook of the Economics of Peace and Conflict_ Oxford University Press New York, 2012, pp. 449–472
  * [59] Andy L Jones  “Scaling Scaling Laws with Board Games”  In _arXiv preprint arXiv:2104.03113_ , 2021
  * [60] Benjamin Jones, EJ Reedy and Bruce A Weinberg  “Age and scientific genius”, 2014
  * [61] Charles I. Jones  “R & D-Based Models of Economic Growth” Publisher: The University of Chicago Press  In _Journal of Political Economy_ 103.4, 1995, pp. 759–784  DOI: 10.1086/262002
  * [62] John Jumper et al.  “Highly accurate protein structure prediction with AlphaFold”  In _Nature_ 596.7873, 2021, pp. 583–589  DOI: 10.1038/s41586-021-03819-2
  * [63] Jared Kaplan et al.  “Scaling Laws for Neural Language Models”, 2020  arXiv:2001.08361 [cs.LG]
  * [64] Asifullah Khan, Anabia Sohail, Umme Zahoora and Aqsa Saeed Qureshi  “A survey of the recent architectures of deep convolutional neural networks”  In _Artificial intelligence review_ 53.8 Springer, 2020, pp. 5455–5516
  * [65] Diederik P Kingma and Jimmy Ba  “Adam: A method for stochastic optimization”  In _arXiv preprint arXiv:1412.6980_ , 2014
  * [66] Samuel Kortum  “Inventions, R&D and industry growth” ISBN: 9798208479254, 1992
  * [67] Samuel Kortum  “Equilibrium R&D and the patent–R&D ratio: Us evidence”  In _The American Economic Review_ 83.2 JSTOR, 1993, pp. 450–457
  * [68] Mark A Kramer  “Nonlinear principal component analysis using autoassociative neural networks”  In _AIChE journal_ 37.2 Wiley Online Library, 1991, pp. 233–243
  * [69] Alex Krizhevsky, Ilya Sutskever and Geoffrey E. Hinton  “ImageNet classification with deep convolutional neural networks”  In _Communications of the ACM_ 60.6, 2017, pp. 84–90  DOI: 10.1145/3065386
  * [70] Yann LeCun, Yoshua Bengio and Geoffrey Hinton  “Deep learning”  In _nature_ 521.7553 Nature Publishing Group, 2015, pp. 436–444
  * [71] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer and Zhifeng Chen  “Gshard: Scaling giant models with conditional computation and automatic sharding”  In _arXiv preprint arXiv:2006.16668_ , 2020
  * [72] Yujia Li et al.  “Competition-level code generation with alphacode”  In _arXiv preprint arXiv:2203.07814_ , 2022
  * [73] Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein and Joseph E Gonzalez  “Train large, then compress: Rethinking model size for efficient training and inference of transformers”  In _arXiv preprint arXiv:2002.11794_ , 2020
  * [74] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár and C Lawrence Zitnick  “Microsoft coco: Common objects in context”  In _European conference on computer vision_ , 2014, pp. 740–755  Springer
  * [75] Fernando Martinez-Plumed, Pablo Barredo, Sean O Heigeartaigh and Jose Hernandez-Orallo  “Research community dynamics behind popular AI benchmarks”  In _Nature Machine Intelligence_ 3.7 Nature Publishing Group, 2021, pp. 581–589
  * [76] Gábor Melis, Chris Dyer and Phil Blunsom  “On the state of the art of evaluation in neural language models”  In _arXiv preprint arXiv:1707.05589_ , 2017
  * [77] Azalia Mirhoseini et al.  “A graph placement methodology for fast chip design” Number: 7862 Publisher: Nature Publishing Group  In _Nature_ 594.7862, 2021, pp. 207–212  DOI: 10.1038/s41586-021-03544-w
  * [78] Benjamin Muller, Peter McIntyre and Sara Altman  “Is AI slowing down?”, 2020
  * [79] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak and Ilya Sutskever  “Deep double descent: Where bigger models and more data hurt”  In _Journal of Statistical Mechanics: Theory and Experiment_ 2021.12 IOP Publishing, 2021, pp. 124003
  * [80] National Center for Science and Engineering Statistics  “Higher Education Research and Development: Fiscal Year 2020: Data Tables”  In _Higher Education Research and Development: Fiscal Year 2020: Data Tables_ , 2021
  * [81] Ariel Pakes and Zvi Grilliches  “Patents and R&D at the Firm Level: A First Look”  In _R &D, Patents and Productivity_ University of Chicago Press, 1984
  * [82] David Picard  “Torch. manual_seed (3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision”  In _arXiv preprint arXiv:2109.08203_ , 2021
  * [83] Michael E Porter and Scott Stern  “Measuring the” ideas” production function: Evidence from international patent output”  National Bureau of Economic Research Cambridge, Mass., USA, 2000
  * [84] Daniel Pressel, Brian Lester, Sagnik Ray Choudhury, Matt Barta, Yanjie Zhao and Amy Hemmeter  “Baseline: Strong, Extensible, Reproducible, Deep Learning Baselines for NLP”, 2018
  * [85] “Project Academic Knowledge”  In _Microsoft Research_ , 2022  URL: https://www.microsoft.com/en-us/research/project/academic-knowledge/
  * [86] Inioluwa Deborah Raji, Emily M Bender, Amandalynne Paullada, Emily Denton and Alex Hanna  “AI and the everything in the whole wide world benchmark”  In _arXiv preprint arXiv:2111.15366_ , 2021
  * [87] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt and Vaishaal Shankar  “Do imagenet classifiers generalize to imagenet?”  In _International Conference on Machine Learning_ , 2019, pp. 5389–5400  PMLR
  * [88] Scott Reed et al.  “A generalist agent”  In _arXiv preprint arXiv:2205.06175_ , 2022
  * [89] David Romer  “Advanced Macroeconomics, 4e”  New York: McGraw-Hill, 2012
  * [90] Paul M. Romer  “Endogenous Technological Change” Publisher: The University of Chicago Press  In _Journal of Political Economy_ 98.5, 1990, pp. S71–S102  DOI: 10.1086/261725
  * [91] Michael E Rose and John R Kitchin  “pybliometrics: Scriptable bibliometrics using a Python interface to Scopus”  In _SoftwareX_ 10 Elsevier, 2019, pp. 100263
  * [92] Olga Russakovsky et al.  “ImageNet Large Scale Visual Recognition Challenge”  In _International Journal of Computer Vision (IJCV)_ 115.3, 2015, pp. 211–252  DOI: 10.1007/s11263-015-0816-y
  * [93] Stuart Russell and Peter Norvig  “Artificial intelligence: a modern approach, 4th Edition”, 2020
  * [94] “Scimago Journal & Country Rank”  URL: https://www.scimagojr.com/
  * [95] “Scopus”, 2022  URL: https://www.scopus.com/home.uri
  * [96] Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn and Pablo Villalobos  “Compute Trends Across Three Eras of Machine Learning”  In _arXiv preprint arXiv:2202.05924_ , 2022
  * [97] Jaime Sevilla, Lennart Heim, Anson Ho, Marius Hobbhahn, Tamay Besiroglu and Pablo Villalobos  “Estimating training compute of Deep Learning models”, 2022
  * [98] Or Sharir, Barak Peleg and Yoav Shoham  “The cost of training nlp models: A concise overview”  In _arXiv preprint arXiv:2004.08900_ , 2020
  * [99] Utkarsh Sharma and Jared Kaplan  “A neural scaling law from the dimension of the data manifold”  In _arXiv preprint arXiv:2004.10802_ , 2020
  * [100] David Silver et al.  “Mastering chess and shogi by self-play with a general reinforcement learning algorithm”  In _arXiv preprint arXiv:1712.01815_ , 2017
  * [101] Stergios Skaperdas and Samarth Vaidya  “Persuasion as a contest”  In _Economic theory_ 51.2 Springer, 2012, pp. 465–486
  * [102] Chen Sun, Abhinav Shrivastava, Saurabh Singh and Abhinav Gupta  “Revisiting unreasonable effectiveness of data in deep learning era”  In _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 843–852
  * [103] Richard Sutton  “The Bitter Lesson”, 2019  URL: http://www.incompleteideas.net/IncIdeas/BitterLesson.html
  * [104] Misha Teplitskiy, Hardeep Ranu, Gary Gray, Michael Menietti, Eva Guinan and Karim R Lakhani  “Do Experts Listen to Other Experts?: Field Experimental Evidence from Scientific Peer Review”  Harvard Business School Cambridge, MA, 2019
  * [105] Neil Thompson, Shuning Ge and Gabriel F Manso  “The importance of (exponentially more) computing power”  In _arXiv preprint arXiv:2206.14007_ , 2022
  * [106] Neil Thompson, Kristjan Greenewald, Keeheon Lee and Gabriel F Manso  “The computational limits of deep learning”  In _arXiv preprint arXiv:2007.05558_ , 2020
  * [107] Manuel Trajtenberg  “AI as the next GPT: a Political-Economy Perspective” Series: Working Paper Series, 2018  DOI: 10.3386/w24245
  * [108] Philip Trammell and Anton Korinek  “Economic growth under transformative AI: A guide to the vast range of possibilities for output growth, wages, and the labor share”, 2020  URL: https://globalprioritiesinstitute.org/wp-content/uploads/Philip-Trammell-and-Anton-Korinek_Economic-Growth-under-Transformative-AI.pdf
  * [109] U.S. Bureau of Economic Analysis  “Gross Domestic Product: Implicit Price Deflator” Publisher: FRED, Federal Reserve Bank of St. Louis  In _FRED, Federal Reserve Bank of St. Louis_ , 2022  URL: https://fred.stlouisfed.org/series/GDPDEF
  * [110] Silviu-Marian Udrescu and Max Tegmark  “AI Feynman: A physics-inspired method for symbolic regression” Publisher: American Association for the Advancement of Science  In _Science Advances_ 6.16, 2020  DOI: 10.1126/sciadv.aay2631
  * [111] Milan Vojnović  “Contest theory: Incentive mechanisms and ranking methods”  Cambridge University Press, 2015
  * [112] Martin L. Weitzman  “Recombinant Growth”  In _The Quarterly Journal of Economics_ 113.2, 1998, pp. 331–360  DOI: 10.1162/003355398555595
  * [113] Thomas Wolf et al.  “HuggingFace’s Transformers: State-of-the-art Natural Language Processing”  arXiv, 2020  URL: http://arxiv.org/abs/1910.03771
  * [114] Lynne G Zucker, Michael R Darby and Maximo Torero  “Labor mobility from academe to commerce”  In _Journal of Labor Economics_ 20.3 The University of Chicago Press, 2002, pp. 629–660

## Appendix A: Derivations and mathematical results

### A1: Deriving steady-state growth rates

Using (1-3), we can derive the rates at which capital and ideas grow:

$g_{k}(t)\equiv\frac{\dot{K}(t)}{K(t)}=c_{k}\bigg{[}\frac{A(t)L(t)}{K(t)}\bigg{]}^{1-\alpha}-\delta,\hskip
4.26773pt\text{where}\hskip 4.26773ptc_{k}\equiv
s(1-\alpha_{k})^{\alpha}(1-\alpha_{l})^{1-\alpha},$ (23)
$g_{a}(t)\equiv\frac{\dot{A}(t)}{A(t)}=c_{a}K(t)^{\beta}L(t)^{\gamma}A(t)^{\theta-1},\hskip
4.26773pt\text{where}\hskip 4.26773ptc_{a}\equiv
B\alpha_{k}^{\beta}\alpha_{l}^{\gamma}.$ (24)

Along the balanced growth path (defined as an equilibrium path where $Y(t)$,
$K(t)$, $A(t)$ and $L(t)$ grow at a constant rate), it can be shown that:

$\tilde{g}_{k}(t)\equiv\frac{\dot{g}_{k}(t)}{g_{k}(t)}=(1-\alpha)(g_{a}+n-g_{k}),\hskip
4.26773pt\text{and}\hskip
4.26773pt\tilde{g}_{a}(t)\equiv\frac{\dot{g}_{a}(t)}{g_{a}(t)}=\beta
g_{k}+\gamma n-(1-\theta)g_{a}.$ (25)

The steady-state rates of growth in ideas and capital can then simply be found
by solving for $g_{k}(t)$ and $g_{a}(t)$ that solves
$\tilde{g}_{k}(t)=\tilde{g}_{a}(t)=0$. Solving this system yields the
following equilibrium growth rates (equilibrium growth rates are marked with
the ∗ superscript):

$g^{*}_{a}=\frac{\beta+\gamma}{1-\beta-\theta}n,\hskip
4.26773pt\text{and}\hskip
4.26773ptg^{*}_{k}=\frac{1-\theta+\gamma}{1-\beta-\theta}n,$ (26)

which can be shown to be unique and stable.141414This is a simple extension of
the usual results for semi-endogenous growth models found in e.g. [89, Chapter
3].

### A2: Proof of Proposition 1b

For part (b) of Proposition 1, we wish to show that the steady-state rate of
economic growth is also strictly increased under the relevant shift. This
happens of both the steady-state rates of idea- and capital accumulation are
permanently increased when $\Delta_{\beta}\geq-\Delta_{\gamma}$. Hence, to
show this result, it suffices to show that the steady-state rate capital
accumulation is permanently increased. This is true if:

$\frac{1-\theta+\gamma+\Delta_{\gamma}}{1-\beta^{\prime}-\Delta_{\beta}-\theta}n>\frac{1-\theta+\gamma}{1-\beta-\theta}n$
(27)

Some rearranging reveals this to hold when:

$-\Delta_{\gamma}<\bigg{(}\frac{1-\theta+\gamma}{1-\beta-\theta}\bigg{)}\Delta_{\beta}$
(28)

Since $\frac{1-\theta+\gamma}{1-\beta-\theta}>1$, this holds whenever
$\Delta_{\beta}\geq-\Delta_{\gamma}>0$. This yields the desired result.
$\blacksquare$

### A3: Capital-cost share in competitive R&D sector

Given the competitiveness of the R&D sector, total expenditure on wages and
rents ($Lw$ and $Kr$ respectively) are given by:

$Lw=L\frac{\partial\dot{A}(t)}{\partial L}=\gamma\dot{A}(t),\hskip
4.26773pt\text{and}\hskip 4.26773ptKr=K\frac{\partial\dot{A}(t)}{\partial
L}=\beta\dot{A}(t).$ (29)

From this it follows that the capital cost-share $Kr/(Kr+Lw)$ is given by
$\beta/(\beta+\gamma)$.

### A4: Empirical specification

To see that

$\frac{\dot{A}(t)}{A(t)}\approx\log\bigg(\frac{P(t)}{P(t-1)}\bigg{missing})+\log\bigg(\frac{1-P(t-1)}{1-P(t)}\bigg{missing}).$
(30)

Note first that $\dot{A}(t)/A(t)=g_{t}$, and moreover that:

$\frac{P(t)}{1-P(t)}\frac{1-P(t-1)}{P(t-1)}=\frac{\exp(g_{t}t)}{\exp(g_{t-1}(t-1))}\approx\exp(g_{t}).$
(31)

Hence
$\log\bigg(\frac{P(t)}{P(t-1)}\bigg{missing})+\log\bigg(\frac{1-P(t-1)}{1-P(t)}\bigg{missing})\approx
g_{t}$ whenever $g_{t}\approx g_{t-1}$.

### A5: Our empirical model and scaling laws

One reason we assume the following relation between performance and
technology, $P(t)=\frac{A(t)}{1+A(t)}$, is that this specification implies a
power-law between the error-rate and the amount of compute dedicated to
improving the underlying technology, which is a key finding found in the
machine learning literature on ‘scaling laws’ (such as [63, 51]).151515This
connection was pointed out to us by Ege Erdil. To show that this is the case,
consider our usual idea production function, where, for convenience, we omit
the input from scientists:

$\frac{\dot{A}(t)}{A(t)}=A(t)^{\theta-1}C(t)^{\beta}.$ (32)
$\iff\int\dot{A}(t)A(t)^{-\theta}dt=\int C(t)^{\beta}dt.$ (33)

We can solve this differential equation for $A(t)$,

$A(t)=\bigg{[}(1-\theta)\int C(t)^{\beta}dt+C\bigg{]}^{\frac{1}{1-\theta}}$
(34)

Taking the log of this expression yields:

$\log A(t)=\log\bigg((1-\theta)\int
C(t)^{\beta}dt+C\bigg{missing})-\log(1-\theta)$ (35)
$\sim\log\bigg((1-\theta)\int C(t)^{\beta}dt\bigg{missing})$ (36)

where $\sim$ means ‘is asymptotically proportional to’. Let $\sigma^{-1}$
denote the inverse sigmoid function, such that
$\sigma^{-1}(x)=\log\big(\frac{x}{1-x}\big{missing})$, then, noting that
$\sigma^{-1}\big{(}P(t)\big{)}=\log A(t)$, we have:

$P(t)\sim\sigma\bigg{(}\log\bigg((1-\theta)\int
C(t)^{\beta}dt\bigg{missing})\bigg{)}.$ (37)

Using the fact that we can approximate the sigmoid function when the argument
is large as $\sigma(x)=\frac{1}{1+\exp(-x)}\approx 1-\exp(-x)$, we have:

$P(t)\sim 1-\exp\bigg(-\log\bigg((1-\theta)\int
C(t)^{\beta}dt\bigg{missing})\bigg{missing}).$ (38) $\sim
1-\frac{1}{1-\theta}\int C(t)^{-\beta}dt.$ (39) $\iff
E(t)\sim\frac{1}{1-\theta}\int C(t)^{-\beta}dt,$ (40)

where $E(t)$ denotes error and is defined as $1-P(t)$. Hence, consider a
scaling of the compute path by some constant factor $\gamma$, then we find
that

$E(t)\sim\frac{1}{1-\theta}\int\big{(}\gamma
C(t)\big{)}^{-\beta}dt=\frac{\gamma^{-\beta}}{1-\theta}\int C(t)^{-\beta}dt.$
(41)

That is, the ‘scale’ of the path of compute dedicated improves the underlying
technology (as measured in terms of the error rate that can be achieved) as a
power-law. $\blacksquare$

## Appendix B: Background on benchmarks in machine learning

Experimental benchmarks are a core feature of the way in which machine
learning research is conducted. Benchmarks (a particular combination of a
dataset or sets of datasets) have long been used for the purpose of performing
experimental validation of new techniques ([52]). It is currently not uncommon
that the significance of new techniques are explicitly linked to benchmark
performance, which often serves as the measuring stick for the amount of
progress made ([75, 86]).

Benchmarks are a tool that enables the comparative assessment of competing
machine learning techniques. For example, consider perhaps the most well-known
computer vision benchmark, ImageNet ([29]). ImageNet161616By “ImageNet”, we
refer to ImageNet-1k, the the most highly-used subset of the Scale Visual
Recognition Challenge (ILSVRC) image classification and localization dataset,
not the larger ImageNet-21K dataset. is a set of over 1.2M images that belong
to 1000 mutually exclusive classes. On the ImageNet Image Classification
benchmark, deep learning models are trained to predict probability
distributions over the classes of each image. Techniques are evaluated on a
distinct test set consisting of 100k images, and typically evaluated on the
basis of some top-$k$ error (the rate at which the ‘true’ classes are not
among the top $k$ highest-probability classes predicted by the model). A
reduction in the error rate, then, represents some notion of progress at the
level of this particular task.171717That said, it should be noted that
benchmark experiments, like those using ImageNet, have important limitations
in assessing innovations for various reasons that are discussed in the machine
learning Literature (e.g. [86, 87, 82]).

## Appendix C: Previous empirical analyses of idea production

In this section, we present a table that summarises our brief review of the
literature that informed our choice of parameter values in our illustrations
of the effects of an increase in the returns to capital on the rate of idea
accumulation.

Study | Model | Location | $\hat{\gamma}$ | $\hat{\theta}$
---|---|---|---|---
[1] |  | “Restricted Cointegration
---
Model” (Table 3.)
International | 0.21 | 1.44
[36] |  | “Modified Romer model”
---
(Tables 1. and 2.)
| United States
---
and Germany
0.10 to 0.48 | 0.1 to 0.83
[83] |  | “Direct Approach”
---
(Table 3.)
| OECD
---
member countries
0.39 to 0.48 | 0.84 to 1.19
[83] |  | “International spillovers and
---
R&D productivity over time”
(Table 4.)
| OECD
---
member countries
0.30 to 0.85 | 0.02 to 0.90
[83] |  | “Regional, language and
---
trade-related spillovers”
(Table 5.)
| OECD
---
member countries
0.20 to 0.36 | 0.85 to 0.94
[67] |  | “Estimates of the
---
Patent Equation”
(Table 4.)
United States | 0.12 to 0.3 | —
[66] |  | “Estimates of the
---
Patent Equation”
(Table 4.1.)
United States | 0.13 to 0.2 | —
[41] |  | “Estimates of the
---
Patent Equation”
(Table 5.)
United States | 0.16 to 0.34 | —
[41] |  | “Estimates with Firm Effects”
---
(Table 6.)
United States | 0.23 to 0.32 | —
[41] |  | “GMT Estimates
---
Assuming AR1 for R&D”
(Table 7.)
United States | 0.26 to 0.33 | —
[81] |  | “Distributed Lag Estimates”
---
(Table 3.2.)
United States | 0.56 | —
Table 7. Previous empirical analyses of the idea production function.
Estimates of $\gamma$ and $\theta$ from our review of the relevant empirical
literature. For each table, we provide the range of point estimates across
models.

## Appendix D: Data construction

In this section, we describe how each of our datasets were constructed. We
provide a high-level overview of our methodology. For full details, see the
online appendix.

### D1: Data on deep learning models

Our data of deep learning models was collated as follows. We use data on deep
learning models for computer vision tasks from [106], which contains 46
publications with reported performance of the highest-performing model on the
relevant benchmark, and estimates of the computational inputs required to
train it. We supplemented this dataset with data on the test and training
settings for each of these models, which we found by manually searching the
publications. In case we could not find what training or test-settings were
used, we dropped the model from out dataset.

We further collected publications from arXiv and Papers with Code that present
models for either of two computer vision tasks anytime after 2012. For each of
the best-performing models presented by each publication, we record the
performance and calculate the computational inputs required to train the
highest-performing model, using methods described in [97]. We further
collected the test and training settings for each of these models, which we
found by manually searching the publications. In total, our dataset contains
148 models, their reported performance, estimates of the computational inputs
required in training, and their test and training-settings.

### D2: Data on baseline performance

Our data on machine learning baseline results may be found here.

#### D3: Bibliometric panel dataset

We generate a yearly panel dataset of feature for 223,703 authors on 115,235
machine learning papers from 1993 to 2021 using data from arXiv, a popular
open-access repository for machine learning papers; Microsoft Project Academic
Knowledge; and Scopus ([11, 85, 95]). Our panel dataset contains yearly series
on all of the following author-specific features: number of publications,
number of total citations, and authors’ h-index (a measure of researcher
productivity and impact, see [49]). To build the dataset, we query arXiv for
all papers from the following categories associated with machine learning:
Machine Learning (stat.ML), Artificial Intelligence (cs.AI), Computation and
Language (cs.CL), Computer Vision and Pattern Recognition (cs.CV), and
Learning (cs.LG). This produces the entire universe of machine learning papers
from 1993 to 2021. Using the procedure described in [78], we match these
papers to their corresponding entries in the Microsoft Project Academic
Knowledge database to obtain author names and affiliations. We match Scopus
publications to those in the Microsoft dataset using a combination of DOIs and
string-distance matching measures for their titles and author names. For each
iterative step, we perform manual quality checks on a random sample of
matches, the highest of which have false positive rates of 4%. Through this
procedure, all but 22,056 authors were matched.

To generate each series, we used bibliometrix, an R tool ([9]), to query
Scopus’ API and retrieve all the publications of for each matched author in
our dataset. For each publication, we then used pybliometrics, a Python
interface to Scopus ([91]), to retrieve the citation trajectories—the number
of incoming citations—for each publication per year, excluding self-citations.
Authors’ research fields were identified by retrieving Scopus’ subject area
data on each of their publications. For each author, we further gathered data
on fractional credit scores for citations and publications, which were
generated by dividing the number of citations and publications by the number
of authors on each publication.

#### D4: Grants dataset

Data on grants received by institutions comes from the Dimensions database of
academic publications ([30]). We query the API for each institution in the
Scopus panel dataset on author affiliations. This returns nominal data in USD
on all grants per project-year and per institution-year. In order to make
intertemporal comparisons, we deflate the institution-level data using the GDP
implicit price deflator ([109]). We use the Damerau-Levenshtein measure of
string distance to match the institution-year data to the matched Scopus-
Microsoft cross-sectional paper titles dataset. For the top 1000 papers, we
find 731 matches, of which 1.2% are false positives. Overall, we match 50.8%
of institutions in our titles dataset. On a random sample of 100 matches, 5%
were found to be false positives. We report total grants received for the past
5 years by all authors’ departments or employers in real 2015 USD.

#### D5: Institutional rankings data

Institutional rankings for Computer Science was generated using Computer
Science publication data by csmetrics.org ([55]), an online dataset of
institutional publication and citation metrics for computer science. This data
is based on measured (retrospective) and predictive (prospective) metrics to
compute a measure of publication impact of computer science publications by
institution. Each institution in our dataset was matched to the entities in
the csmetrics.org database by using Levenshtein distance, a measure of string-
distance. Since not all institutions in our dataset were also present in
csmetrics.org’s database, we were able to assign ranks to only 89.6% of unique
institutions in our dataset. The institutions we were unable to retrieve
rankings for were smaller institutions that did not appear often in our data.
In particular, institutional rankings were assigned in 99.4% of authors for
whom institutional data was available. Moreover, the matching rarely produced
false-matches: of a random sample of 120 matches of institutions, only 0.9% of
matches were found to be incorrect.

#### D6: Computer science publication venue rankings data

Data on yearly rankings of Computer Science publication venues was retrieved
from SCImago Journal & Country Rank ([94]), a portal for scientific indicators
developed from the information contained in the Scopus database. In
particular, we use their SJR indicator, which ranks scholarly journals,
conferences and proceedings based on citation weighting schemes and
eigenvector centrality, to quantify the impact and prestige of publication
venues ([37]). Their database contains 5147 unique Computer Science venues
spanning the 1999-2020 period. We used the Levenshtein distance to match the
journal in SCImago’s database to the journals in which the articles dataset
were published in. Overall, we were able to assign roughly 15,000 publications
with journal rankings and SJR indicators. In case rankings were missing for
the exact year of publication, we assigned each journal the rank of either its
rank in the year prior to publication, or its rank in the year post-
publication. If asynchronous rankings one-year apart were not available, we
considered two-years prior and post-publication. Only 0.6% of ranking
assignments were made with asynchronous rankings of the kind described. Of a
random sample of 100 publications, we found no incorrect matches between the
Scopus-reported publication venue and the entity in SCImago’s database to
which it was matched, suggesting our matching was performed highly accurately.

## Appendix E: Estimating compute

We use two methods for inferring the amount of compute used to train an AI
system, both are from [97]. These methods (one based on the architecture of
the network and number of training batches processed; and another based on the
hardware setup and amount of training time), are outlined below.

#### E1: Method 1—counting operations in the model

The first method is can be summarized as:

$\text{Training compute}=(\text{FLOP per forward pass}+\text{FLOP per backward
pass})\times\text{Nr. of passes},$ (42)

where FLOP per forward pass is the number of floating point operations in a
forward pass, FLOP per backward pass is the number of operations in a backward
pass and Nr. of passes is the number of full passes (a full pass includes both
the backward and forward pass) made during training.

Moreover, we use the facts that:

$\text{Nr. of passes}=\text{Nr. of epochs}\times\text{Nr. of examples},$ (43)
$\text{FLOP per forward pass}+\text{FLOP per backward pass}\approx
3\times\text{FLOP per forward pass}$ (44)

where the latter is implied by the fact that computing the backward pass
requires each layer to compute the gradient with respect to the weights and
the error gradient of each neuron with respect to the layer input to
backpropagate. Each of these operations requires compute roughly equal to the
amount of operations in the forward pass of the layer. Hence, the total number
of FLOP per forward and backward pass is roughly 3-fold the number of FLOP per
forward pass.

If the number of examples is not directly stated, it can be computed as the
number of batches per epoch times the size of each batch: $\text{Nr. of
examples}=\text{Nr. of batches}\times\text{batch size}$.

#### E2: Method 2—GPU time

Secondly, we may also use the reported training time and GPU model performance
to estimate the training compute. For example, if the training lasted 2 days
and a total of 5 GPUs were used, that equals 10 GPU-days. By multiplying the
number of GPU-days by the performance of the GPU, we can infer the amount of
compute, in FLOP, that was needed to train the model. In particular, the
formula used is the following:

$\text{Training compute}=\text{Training time (in seconds)}\times\text{Nr. of
cores}\times\text{Peak FLOP/s}\times\text{utilization rate},$ (45)

where peak performance in FLOP/s is found in the relevant GPU product
datasheets, and the utilization rate corrects for imperfect hardware
utilization, for which 30% is often a reasonable baseline (see [97]).

## Appendix F: Training procedure and hyper-parameter settings

The weights in the first 12 layers are frozen, and the weights in the
remaining components are randomly initialized and trained on a subset of for
the three output features are known ($\sim$13k publications). Finally, we
transfer this model to our dataset of publications that present machine
learning models (which were removed from our training set), and inspect the
activations of the human capital unit.

To train our DNN, we use the adamax optimizer ([65]) with $\beta_{1}=0.9$,
$\beta_{2}=0.999$. Throughout, use the GELU activation function ([45]), with
the exception of the Human Capital unit, which is ReLU-activated.

Pre-training step | Output layer | Batch size | Learning rate | Epochs | $N$
---|---|---|---|---|---
1 | t+1 | 1024 | $5\times 10^{-4}$ | 90 | 44562
2 | t+2 | 1024 | $5\times 10^{-4}$ | 90 | 41120
3 |  | t+1, t+2,
---
t+3, t+5
248 | $8\times 10^{-4}$ | 90 | 14217
4 |  | t+1, t+2,
---
t+3, SJR
248 | $5\times 10^{-4}$ | 90 | 8978
5 | t+1 | 500 | $5\times 10^{-4}$ | 90 | 44562
6 | t+1, t+2, SJR | 500 | $8\times 10^{-4}$ | 90 | 13157
7 | t+1, t+2 | 500 | $8\times 10^{-4}$ | 90 | 41120
8 | t+1, t+2, SJR | 500 | $8\times 10^{-4}$ | 90 | 13157
Table 8. Training steps and settings. Output layer refers to the outcomes that
the neural network was trained to predict (‘t+k’ refers to citations k-years
post-publication, and SJR refers to the publication’s SJR value). $N$ refers
to the number of observations that we trained on for each step (which is equal
to the number of examples for which we had all relevant observations for in
our training set).

## Appendix G: Input data

We train our DNN to develop to predict a variety of bibliometric-related and
publication-related outcomes based on a large number of author- and
publication-level features, which were encoded as a $269\times 1$ input
vector. The input-layer features are described in Table 9.

Variable | Included for which authors | Lags
---|---|---
h-index | Up to 20 authors | 4 years
Number of publications | Up to 20 authors | 4 years
Number of citations received (excluding self-citations) | Up to 20 authors | 4 years
Total grant funding received by authors’ institution | All | 5 years
Ranking of institution | Up to 20 authors | —
Number of authors by institution type (academic, govm, company) | All | —
Number of authors on publication | — | —
Publication date | — | —
Table 9. Features in input layer. Variables that make up our input vector. All
variables are evaluated at the time of publication. In all, each input vector
is of the dimension $1\times 179$.

In case there are more than 20 authors on a single publication we select the
data for the first 15 authors and the last 5 authors, as commonly, authors are
ordered in descending order of their share of contributions, with the
exception that supervisory or senior authors are listed last. This ensures
that we include both the batch of authors that might have made the largest
contributions, as well as the authors who had a supervisory role.

## Appendix H: Using human capital to predict bibliometric- and publication-
related outcomes

Figure 8. Our human capital estimates predict key outcomes on hold-out set for
all publications (top) and publications with junior authors only (bottom).
Publications with junior researchers is a subset of the test set consisting of
1,230 publications for which none of the publication authors have more than 2
prior publications.

## Appendix I: Inputs and performance

The computer vision models in our dataset are trained on a wide range of
compute budgets. This range spans 4 orders of magnitude for image
classification models, and 2 orders of magnitude for object detection models.
Training compute is highly variable, yet, with steady growth in the most
compute-intensive models (see Figure 3). In particular, the estimated doubling
time of computation used in the most compute-intensive models not trained on
extra data (for which we have by far the most data) are 8.72 months [95% CI:
2.22 to 34.33 months] for Image Classification models, and 8.96 months [95%
CI: 2.48 to 32.35 months] for object detection models. This growth rate in
compute-intensity is consistent with previous estimates of compute trends in
recent years (Sevilla et al., 2022).

Figure 9. Compute intensity of training runs over time for each benchmark.
Note: the y-axis is logarithmic ($\log_{10}$).

Figure 6 plots model performance against both training compute and human
capital. We find that computational capital and human capital are both
associated with improved model performance. For the machine learning tasks
considered we find roughly a power-law relationship between compute and error
rates, in line with the experimental results in the literature ([63]; [47]).
The relationship is particularly evident for ImageNet.

Figure 10. Associations between performance and training compute (top panel)
and human capital (bottom panel).

Additionally, there is some positive association between performance and the
estimated amount of human capital, though this association seems generally
much weaker than that between computational capital and performance.

## Appendix J: Regression model specifications and estimation procedures

Recall that our empirical estimation is given by:

$\log\tilde{g}_{i}=\mathbf{\phi}^{\prime}\mathbf{1}+\gamma\log S_{i}+\beta\log
C_{i}+\mathbf{\alpha}\mathbf{X}+u_{i},\hskip 4.26773pt\text{where}\hskip
4.26773pt\mathbf{\phi}\equiv(1-\theta)\begin{bmatrix}\log A_{1}\\\ \log
A_{2}\\\ \vdots\\\ \log A_{T}\end{bmatrix}.$

Table 10 includes each of the full models that we estimate in Section 7.

Model | Dataset | Specification
---|---|---
A1 | Image classification |  | $\log\tilde{g}_{i}=\mathbf{\phi}^{\prime}\mathbf{1}+\gamma\log S_{i}+\beta\log C_{i}+\alpha_{1}\text{extra data}_{i}+\alpha_{2}\text{reimplementation}_{i}$
---
A2 | Image classification |  | $\log\tilde{g}_{i}=\mathbf{\phi}^{\prime}\mathbf{1}+\gamma\log S_{i}+\beta\log C_{i}+\alpha_{1}\text{extra data}_{i}+\alpha_{2}\text{reimplementation}_{i}$
---
$\alpha_{3}\text{years from 2012}_{i}$
B1 | Object detection |  | $\log\tilde{g}_{i}=\mathbf{\phi}^{\prime}\mathbf{1}+\gamma\log S_{i}+\beta\log C_{i}+\alpha_{1}\text{extra data}_{i}+\alpha_{2}\text{reimplementation}_{i}$
---
B2 | Object detection |  |  g$\log\tilde{g}_{i}=\mathbf{\phi}^{\prime}\mathbf{1}+\gamma\log S_{i}+\beta\log C_{i}+\alpha_{1}\text{extra data}_{i}+\alpha_{2}\text{reimplementation}_{i}$
---
$\alpha_{3}\text{years from 2012}_{i}$
C1 | All computer vision |  | $\log\tilde{g}_{i}=\mathbf{\phi}^{\prime}\mathbf{1}+\gamma\log S_{i}+\beta\log C_{i}+\alpha_{1}\text{extra data}_{i}+\alpha_{2}\text{reimplementation}_{i}$
---
C2 | All computer vision |  | $\log\tilde{g}_{i}=\mathbf{\phi}^{\prime}\mathbf{1}+\gamma\log S_{i}+\beta\log C_{i}+\alpha_{1}\text{extra data}_{i}+\alpha_{2}\text{reimplementation}_{i}$
---
$\alpha_{3}\text{years from 2012}_{i}$
Table 10. Full model specifications. $\phi^{\prime}$ denotes the transpose of $\phi$. $\text{reimplementation}_{i}$ is equal to $1$ if the model $i$ is a re-implementation of prior work, and a $0$ otherwise. $\text{extra data}_{i}$ is equal to $1$ if model $i$ was trained using extra data (other than the ImageNet training set), and $0$ otherwise (this includes any pre-training of the model, including any of its backbones). The trend coefficients reported in our empirical results correspond to the estimates of the coefficient denoted $\alpha_{3}$ in this table. | Model
---|---
Time-period length | A1 | A2 | B1 | B2 | C1 | C2
6 |  | GLS
---
Table 6
| GLS
---
Table 12
| OLS (HC)
---
Table 6
| OLS (HC)
---
Table 12
| GLS
---
Table 6
| OLS (HC)
---
Table 12
12 |  | GLS
---
Tables 4, 6
| OLS (HC)
---
Tables 4, 6
| OLS (HC)
---
Tables 4, 6
| OLS (HC)
---
Tables 4, 6
| OLS (HC)
---
Tables 5, 6
| OLS (HC)
---
Tables 5, 6
18 |  | OLS (HC)
---
Table 6
| OLS (HC)
---
Table 12
| OLS (HC)
---
Table 6
| OLS (HC)
---
Table 12
| OLS (HC)
---
Table 6
| OLS (HC)
---
Table 12
Table 11. Estimation techniques used. GLS refers to Generalized Least Squares,
and OLS (HC) refers to Ordinary Least Squares, with robust covariance matrix