#  Towards Green AI with tensor networks
Sustainability and innovation enabled by efficient algorithms

Eva Memmel∗1, Clara Menzen∗1, Jetze Schuurmans2, Frederiek Wesel1, Kim
Batselier1
<EMAIL_ADDRESS>
*Equal contribution.1Delft Center of Systems and Control, 2Cognitive Robotics, Delft University of Technology

###### Abstract

The current standard to compare the performance of AI algorithms is mainly
based on one criterion: the model’s accuracy. In this context, algorithms with
a higher accuracy (or similar measures) are considered as better. To achieve
new state-of-the-art results, algorithmic development is accompanied by an
exponentially increasing amount of compute. While this has enabled AI research
to achieve remarkable results, AI progress comes at a cost: it is
unsustainable. In this paper, we present a promising tool for sustainable and
thus Green AI: tensor networks (TNs). Being an established tool from
multilinear algebra, TNs have the capability to improve efficiency without
compromising accuracy. Since they can reduce compute significantly, we would
like to highlight their potential for Green AI. We elaborate in both a kernel
machine and deep learning setting how efficiency gains can be achieved with
TNs. Furthermore, we argue that better algorithms should be evaluated in terms
of both accuracy and efficiency. To that end, we discuss different efficiency
criteria and analyze efficiency in an exemplifying experimental setting for
kernel ridge regression. With this paper, we want to raise awareness about
Green AI and showcase its positive impact on sustainability and AI research.
Our key contribution is to demonstrate that TNs enable efficient algorithms
and therefore contribute towards Green AI. In this sense, TNs pave the way for
better algorithms in AI.

##  1 Introduction

More than ever before we have access to data sets throughout almost all
disciplines of science and engineering. Fueling our economies and shaping our
society, data is therefore considered the oil of the 21st century. At the same
time, AI algorithms become increasingly powerful to transform large amounts of
raw data into valuable information. Consequently, AI development and data
availability are deeply intertwined and they have a common characteristic:
both are growing exponentially [90]. While this wealth of information is
opening the doors for extraordinary opportunities, we cannot ignore the
downside of this development: AI research on a large scale has undesirable,
negative side effects on economic, social, and environmental sustainability.
To illustrate this, we consider the following example. The authors of [79]
analyze the energy required for training popular off-the-shelf natural
language processing (NLP) models. They have found CO2 emissions up to
$280\,000\text{\,}\mathrm{k}\mathrm{g}$ for training and up to 3 million
dollars for cloud compute costs. As a comparison, a person could fly more than
300 times between Amsterdam and New York to emit the same amount of CO2 [2].

One of the reasons for unsustainable development in AI research is
unfortunately anchored in what the AI community defines as better. The current
thrive and almost single focus is to present a model that beats the current
accuracy benchmark. To obtain state-of-the-art results, the number of model
parameters and hyperparameters are increased, as well as the size of the
training data. The result is an exponentially growing demand for compute used
to train AI models [73]. This is not only unsustainable from an environmental
and economic point of view, but also from a social one: it has a large carbon
footprint, it is expensive, and it excludes researchers with fewer resources.

The problems associated with the unsustainable development in AI have led to a
growing awareness in the AI community. Several researchers call for
redirecting the focus of AI research by implementing efficiency as an
additional benchmark to assess algorithmic progress [73, 80, 81]. In fact,
efficiency has always been the primary criterion to measure algorithmic
progress in computer science [20, 46, 47]. Inspired by this, a similar
approach can be adapted to AI algorithms. For this purpose, different metrics
have been proposed in literature, e.g. estimating the carbon footprint,
reporting the energy consumption, or stating the number of floating-point
operations (FLOPS) [32, 51, 52]. In this context, new vocabulary has been
suggested. It distinguishes between AI that solely focuses on accuracy versus
AI that considers efficiency and accuracy as equal criteria: Red AI versus
Green AI [73].

While metrics pave the way for Green AI, only reporting on efficiency metrics
is not enough. More importantly, the AI community needs to develop and adopt
methods that have the potential to reduce compute significantly. We would like
to contribute to this development by presenting a very promising method:
Tensor networks (TNs), also called tensor decompositions. Being a tool from
multilinear algebra, TNs have become increasingly popular in the AI
community111The number of papers at NeurIPS, AISTATS and ICML with ‘tensor’,
‘Tucker’, ‘CP’, ‘PARAFAC’, ‘block term’, ‘decomposition’ in the title have
increased in the last 12 years. In 2010 there was only 1 paper, in 2015 there
were 11 and in 2021 there were 28.. TNs can approximate data in a compressed
format while preserving the essence of the information. In this way, they
oftentimes lift the curse of dimensionality, while having the potential to
achieve competitive accuracy. This makes TNs exactly the type of tool to apply
to large-scale and/or high-dimensional problems: they allow for an efficient
and thus sustainable way of representing and handling big data [16]. Even
though the popularity of TNs is increasing in the AI community, they are still
not widespread yet. Various promising results regarding accuracy and
efficiency, as e.g. [31, 39, 69, 87], show their potential. With this paper,
we would like to give the impulse for AI researchers to adopt TNs for their
work.

To that end, we provide an economic, social, and environmental analysis of the
possibilities and challenges of Green AI as well as the potential of TNs for
Green AI. Understanding the impact of TNs on Green AI is not trivial. To the
best of the authors’ knowledge, this is the first paper that highlights TNs
from a Green AI perspective. In this paper, we answer the following three
research questions. Why should better AI algorithms be defined by both
accuracy and efficiency? How can TNs make AI algorithms more efficient? How
are TNs an appropriate tool to contribute towards Green AI? To answer the
three research questions, the paper is organized as follows: In section 2, we
describe how an exponentially growing demand for compute has a negative impact
on sustainability and AI progress. In addition, we discuss different criteria
suggested in literature for an efficiency metric. In section 3, we introduce
commonly used TNs and their compression potential. In section 4, we show how
an efficiency gain can be achieved by applying TNs in a kernel machine and
deep learning setting. In section 5, we draw conclusions.

##  2 Context and related literature

AI models are considered as better when they achieve greater accuracy (or
similar measures) than previously reported. To incentivize competition and
thus accelerate innovation, results are made public on leaderboards based on
accuracy metrics. Beating state-of-the-art algorithms is often achieved by at
least one of the following three aspects: bigger data sets, a more complex
model, which is highly correlated with the number of parameters, or more
extensive hyperparameter experiments [73]. This results in an exponentially
growing demand for compute, requiring an unsustainable amount of hardware,
energy, and computational time.

###  2.1 The negative impact of growing compute on sustainability and AI
progress

Sustainability is based on three fundamental, intersecting dimensions:
economic, social, and environmental [66]. An exponentially growing amount of
compute in AI compromises all three of them, as explained in the following.

Because large computations can have a large price tag [79, 80], they
negatively impact both the economic and social dimensions of sustainability.
Concerning the economic dimension, linear gains in accuracy contrast with an
exponentially growing amount of compute [73]. Thus, diminishing returns
contrast with increasing costs associated with e.g. cloud computing and
hardware (such as CPU, GPU, and TPU). In the social dimension, large costs can
create barriers for researchers with fewer resources to participate in
computationally expensive AI development. Beyond the financial aspect, being
dependent on external cloud compute providers can be problematic, too. For
example, in applications where privacy or security-relevant data is handled,
computations may need to be performed on-site. Therefore, state-of-the-art
computations with sensitive data may not be possible, if there are not enough
financial resources available. Concerning the environmental aspect, using a
lot of resources for AI research causes a meaningful carbon footprint. The
emissions are attributable to operational and embodied emissions, associated
with e.g. cloud computing and hardware, respectively. In case operational
emissions are decreased by relying on electricity with a low carbon intensity,
embodied emissions make up for the largest share [26]. In fact, the impact of
AI on the environmental dimension of sustainability demands special attention.
The Paris Agreement, ratified by 192 countries and the European Union [59],
states that it is a goal to limit global warming to
$1.5\text{\,}{}^{\circ}\mathrm{C}$. Reaching the
$1.5\text{\,}{}^{\circ}\mathrm{C}$ goal entails an immediate, rapid and deep
emission reduction across all sectors [38]. In addition, net-zero emissions
need to be achieved around 2050 [37]. To stay on a
$1.5\text{\,}{}^{\circ}\mathrm{C}$ pathway, emissions need to be reduced by
$7.6\%$ each year between 2020 and 2030 [85]. Even though AI can contribute to
the mitigation of climate change [36, 71], AI can also have adverse side
effects [38]: The demand for compute impacts CO2 emissions directly [26, 27].
Improving algorithmic efficiency can have a positive impact on both
operational and embodied emissions [26].

Beyond sustainability, an exponentially growing demand for compute can cause
an additional problem. When the demand for compute cannot be met anymore, it
will have a negative impact on progress and innovation in AI. Up until now
progress and innovation in AI strongly rely on the increase of available
computational resources [82]. According to forecasts, the number of
transistors on a microchip doubling every 2 years (Moore’s Law), will reach
its limits [55]. Furthermore, its 2 years doubling period has already been
overtaken by the need for compute, which is doubling every 3.4 months [3] with
a total increase of $300,000$ times between 2012 and 2018 [3]. Consequently,
the growing need for compute will ultimately limit AI progress and innovation,
especially in computationally expensive fields [79, 82]. Another aspect is
that exponentially growing demand for compute can compromise reproducibility.
In the AI community, there is already a growing awareness of the importance of
reproducibility, see e.g. the ML Reproducibility Challenge [88].

The problems mentioned above can be tackled by reducing compute. Consequently,
AI researchers need to redirect their focus toward Green AI and redefine what
better means. Redefining better entails mainly two aspects. First, accuracy
and efficiency need to be considered being equally important when measuring
progress in AI. Second, an analysis of the trade-off between performance and
computational resources used need to be included in AI research. AI algorithms
with better efficiency will therefore have a positive impact on both
sustainability and AI progress [73, 79]. In addition, reporting on efficiency
metrics has other benefits: it will not only raise awareness but also
incentivize progress in efficiency [32, 81].

###  2.2 Efficiency metrics suggested in literature

So far, there has not been much attention on efficiency in major AI venues as
most papers report only on an accuracy metric [73]. In addition, it is not
straightforward to report on efficiency and there is no established way to do
so. This is why we will discuss several evaluation criteria for efficiency
that are considered in the literature.

An appealing criterion to measure efficiency is CO2 equivalent (CO2e)
emissions, since they are directly related to climate change. Disadvantages
include that they are difficult to measure and are influenced by factors that
do not account for algorithmic optimization, e.g. hardware, carbon intensity
of the used electricity, and time and location of the compute [73, 80]. An
alternative criterion, which is independent of time and location, is to state
the electricity usage in kWh. It can be quantified because GPUs commonly
report the amount of consumed electricity. However, it is still hardware
dependent. There are several websites and packages available to compute the
CO2e and electricity usage of algorithms [4, 32, 51, 52, 56, 72]. A hardware-
independent criterion is the number of floating point operations (FLOPS) that
are required for training the model. On the one hand, they are computed
analytically and facilitate a fair comparison between algorithms [34, 73]. On
the other hand, they do not account e.g. for optimized memory access or memory
used by the model [32, 73]. Next to giving the absolute number of FLOPS, it is
possible to additionally report e.g. a FLOPS-based learning curve [34]. There
are packages available to compute FLOPS222https://github.com/sovrasov/flops-
counter.pytorch, https://github.com/Swall0w/torchstat,
https://github.com/Lyken17/pytorch-OpCounter, https://github.com/triscale-
innov/GFlops.jl. A criterion that is commonly used in computer science is the
big $\mathcal{O}$ notation [47]. This notation is used to report the storage
complexity and the computational cost of an algorithm. For AI practitioners,
the big $\mathcal{O}$ notation can be impractical because the run time is
often strongly influenced by application-specific termination criteria [34].
However, it has benefits in a theoretical analysis: Sections of an algorithm
and required storage complexities can be easily compared. In tensor community,
using the big $\mathcal{O}$ notation is common, see e.g. [17] and references
therein. The elapsed runtime is easy to measure, but it is highly dependent on
the used hardware and other jobs running in parallel [73]. The number of
parameters (learnable or total) by the model is a common efficiency measure.
It is agnostic to hardware and takes the memory needed by the model into
account. Nonetheless, different model architectures can lead to a different
amount of work for the same amount of parameters [73].

We suggest reporting on multiple efficiency criteria, as well as stating the
specifications of the used hardware. Combining multiple criteria can support
making a link between an efficiency gain and algorithmic innovation. In
addition, we agree with the suggestion by [34, 79] to report a cost-benefit
analysis, relating gains or losses in accuracy with gains or losses in
efficiency.

##  3 Tensor networks basics

As mentioned earlier, a performance boost of an AI model can be obtained by an
increase in model parameters, hyperparameters, and the training data size. The
resulting large-scale or high-dimensional data objects can be handled by TNs.
The successful application of TNs to AI has been extensively explored in
literature: The book [19] describes topics including tensor regression,
tensor-variate Gaussian processes, and support tensor machines. The survey
[41] treats supervised and unsupervised classification, among other things.
The overview paper [74] treats e.g. collaborative filtering, mixture and topic
modeling, and in [75] a framework for kernel machines with TNs is discussed.

###  3.1 From tensors to tensor networks

Multidimensional arrays, also called tensors, are a generalization of matrices
to higher dimensions. In many applications, instead of tensors, large vectors
or matrices arise. By rearranging their entries, however, vectors and matrices
can also be transformed into tensors. In this procedure, called tensorization
[44, 63], the row and column size are first factorized into multiple factors
and then reshaped into a tensor. A small tensorization example is illustrated
in Figure 1 where a matrix of size $2^{3}\times 2^{3}$ is transformed into a
6-dimensional tensor. The left side of Figure 1 shows the rearrangement of the
matrix elements into the tensor. In addition, the two x-shapes showcase how
the bookkeeping of the entries works. To simplify the depiction of higher-
dimensional objects, it is possible to use the diagram notation shown on the
right side of Figure 1. In this notation, a matrix or a tensor are depicted as
nodes with as many edges sticking out as its number of dimensions. The reason
why a tensorized representation is useful is that it enables the use of TNs.
In the following subsection, we introduce commonly used TNs, while reviews can
e.g. be found in [17, 48].

Figure 1: Left: tensorization of a matrix of size $2^{3}\times 2^{3}$ into a
tensor of size $2\times 2\times 2\times 2\times 2\times 2$. Based on Fig. 2 of
[18]. Right: tensorization in diagram notation. The 2 edges of the node
representing the 8-by-8 matrix become 6 edges sticking out the node
representing the 6-dimensional tensor.

###  3.2 Commonly used tensor networks

A tensor can be expressed as a function of simpler tensors that form a TN,
also called tensor decomposition. The idea of a TN has its origins in the
generalization of the singular value decomposition (SVD) to higher dimensions.
In a lot of applications TNs can represent data in a compressed format with a
marginal loss of information, because of correlations present in the data [19,
17].

In literature the most commonly used TNs include the CANDECOMP/PARAFAC (CP)
[11, 29], the Tucker [84], and the tensor train (TT) decomposition [64].
Without loss of generality, in this subsection, we will treat a three-
dimensional tensor $\mathbfcal{Y}$ to introduce the TNs mentioned above. The
CP decomposition [11, 29] of $\mathbfcal{Y}\in\mathbb{R}^{I_{1}\times
I_{2}\times I_{3}}$ consists of a set of factor matrices
$\mathbf{A},\mathbf{B},\mathbf{C}\in\mathbb{R}^{I_{j}\times R}$, $j=1,2,3$ and
a weight vector $\boldsymbol{\lambda}\in\mathbb{R}^{R\times 1}$. Elementwise,
$\mathbfcal{Y}$ can be computed from

$y_{i_{1}i_{2}i_{3}}=\sum_{r=1}^{R}\lambda_{r}\;a_{i_{1}r}\;b_{i_{2}r}\;c_{i_{3}r}.$
(1)

The scalars $a_{i_{1}r},b_{i_{2}r},c_{i_{3}r}$ are the entries of the three
factor matrices $\mathbf{A},\mathbf{B},\mathbf{C}$, $\lambda_{r}$ is the
$r$-th entry of $\boldsymbol{\lambda}$, $R$ denotes the rank of the
decomposition. The Tucker decomposition [84] of
$\mathbfcal{Y}\in\mathbb{R}^{I_{1}\times I_{2}\times I_{3}}$ consists of a
3-way tensor $\mathbfcal{G}\in\mathbb{R}^{R_{1}\times R_{2}\times R_{3}}$,
called core tensor, and a set of matrices
$\mathbf{A},\mathbf{B},\mathbf{C}\in\mathbb{R}^{I_{j}\times R_{j}}$,
$j=1,2,3$. Elementwise, $\mathbfcal{Y}$ can be computed from

$y_{i_{1}i_{2}i_{3}}=\sum_{r_{1}=1}^{R_{1}}\sum_{r_{2}=1}^{R_{2}}\sum_{r_{3}=1}^{R_{3}}g_{r_{1}r_{2}r_{3}}\;a_{i_{1}r}\;b_{i_{2}r}\;c_{i_{3}r}.$
(2)

The scalars $a_{i_{1}r},b_{i_{2}r},c_{i_{3}r}$ are the entries of the three
factor matrices $\mathbf{A},\mathbf{B},\mathbf{C}$, $g_{r_{1}r_{2}r_{3}}$ is
the $(r_{1}r_{2}r_{3})$-th entry of $\mathbfcal{G}$ and $R_{1},R_{2},R_{3}$
denote the ranks of the decomposition. The tensor train decomposition [64] of
$\mathbfcal{Y}\in\mathbb{R}^{I_{1}\times I_{2}\times I_{3}}$ consists of a set
of three-way tensors $\mathbfcal{Y}^{(j)}\in\mathbb{R}^{R_{j}\times
I_{j}\times R_{j+1}}$, $j=1,2,3$ called TT-cores. Elementwise, $\mathbfcal{Y}$
can be computed from

$y_{i_{1}i_{2}i_{3}}=\sum_{r_{1}=1}^{R_{1}}\sum_{r_{2}=1}^{R_{2}}\sum_{r_{3}=1}^{R_{3}}\sum_{r_{4}=1}^{R_{4}}y^{(1)}_{r_{1}i_{1}r_{2}}\;y^{(2)}_{r_{2}i_{2}r_{3}}\;y^{(3)}_{r_{3}i_{3}r_{4}},$
(3)

where $R_{1},R_{2},R_{3},R_{4}$ denote the ranks of the TT-cores and by
definition $R_{1}=R_{4}=1$. Figure 2 shows a graphical depiction of the CP,
Tucker, and TT decomposition for a three-dimensional tensor. Connected edges
are indices that are being summed over and the number of free edges
corresponds to the dimensionality of the tensor.

Figure 2: Graphical depiction of commonly used TNs for a three-dimensional
tensor. Connected edges are indices that are being summed over. The CP
decomposition is a special case of Tucker, where the core tensor is diagonal.
This is shown by the diagonal in the node.

###  3.3 Compression potential of TNs

When a tensor is approximately represented by a TN, then the ranks determine
the accuracy of the approximation and the TN is called low-rank. These ranks
are then hyperparameters. The low-rank property is very powerful mainly for
two reasons. First, the low-rank property can enable a compressed
representation of the data with marginal loss of information [19]. Second,
low-rank TNs have the capability to reduce an exponential complexity in $D$ to
a linear in $D$. With this property, they can alleviate the curse of
dimensionality which is an exponential growth of elements with the number of
dimensions $D$. For example, considering the CP decomposition for a given
tensor with $I^{D}$ elements, the number of elements of the rank-$R$
decomposition are $\mathcal{O}(RID)$, thus linear in $D$. When a TN is low-
rank, the compression can be significant. In this paper, we call the
compression from exponential in $D$ to linear in $D$ a logarithmic
compression.

##  4 Applications of TNs to AI

To show how TNs can be integrated into AI algorithms, consider the following
two common learning problems: unsupervised learning and supervised learning.
One example for unsupervised learning is data compression, where a
$D$-dimensional data tensor $\mathbfcal{X}\in\mathbb{R}^{I\times...\times I}$
can be approximately represented in a compressed format as a low-rank TN. It
can be formulated as

$\displaystyle\min_{\hat{\mathbfcal{X}}}\;\;\|\mathbfcal{X}-\hat{\mathbfcal{X}}\|,\;\;\mathrm{where}\;\hat{\mathbfcal{X}}\;\text{is
a low-rank TN}.$ (4)

Given a set of input/output pairs $\mathbf{x}$ and $\mathbf{y}$ , a supervised
learning can be described with minimizing the empirical loss $L$ and a
regularization term $R$

$\displaystyle\min_{\mathbf{w}}\;L(\mathbf{y},f(\mathbf{x}\mid\mathbf{w}))+R(\mathbf{w}),$
(5)

where $f(\mathbf{x}\mid\mathbf{w})$ is a nonlinear function parameterized by
the weights $\mathbf{w}$. To apply TNs, $f(\mathbf{x}\mid\mathbf{w})$ is
parametrized with low-rank TNs. In this section, we will describe two models
of $f(\mathbf{x}\mid\mathbf{w})$ parametrized by low-rank TNs: a kernel
machine and a neural network. For these two examples, we quantify efficiency
gains obtained by logarithmic compression.

The fundamental insights taken from these specific examples illustrate the
principle of how TNs make AI algorithms more efficient. Beyond that, the
generalization of the examples enables to integrate TNs into a variety of AI
applications.

###  4.1 TNs in kernel machines

Kernel machines, such as Gaussian processes (GPs) [67] and support vector
machines (SVMs) [28], can be universal function approximators. While they have
shown equivalent or superior performance compared to neural networks [25, 54,
60], they scale poorly for high-dimensional or large-scale problems.

We show how TNs can be integrated into the supervised learning problem
described in (5), considering a kernel machine setting. By making four
specific model choices, we present a low-rank TN primal kernel ridge
regression problem (T-KRR) based on [87]. Even if we describe a specific
problem characterized by our model choices, it is possible to implement the
same approach with other model choices. This results in a broad variety of
problems where low-rank TNs are applicable.

Given an input/output pair $\mathbf{x}_{n}$ and $y_{n}$, a kernel machine is
given by

$y_{n}=\langle\boldsymbol{\varphi}(\mathbf{x}_{n}),\mathbf{w}\rangle+\epsilon_{n},\;\;n=1,...,N,$
(6)

where $\boldsymbol{\varphi}(\cdot)$ is a feature map, $\mathbf{w}$ is a weight
vector, $\langle\cdot,\cdot\rangle$ denotes the inner product, and
$\epsilon_{n}$ is the error. In our example, the first model choice is to
represent $\boldsymbol{\varphi}(\cdot)$ as a Kronecker product of $D$
regressors, computed from a chosen number $I$ basis functions. In this
context, for TNs the following basis functions have been explored in
literature: Polynomial basis function are treated in [7, 8, 68], pure-
power-$1$ polynomials in [62], pure-power-$M$ polynomials in [13] and
B-splines in [42]. The use of trigonometrical basis functions are described in
[78], and Fourier features in [43, 86, 87].

For the $n$-th input vector,
$\boldsymbol{\varphi}(\mathbf{x}_{n})\in\mathbb{R}^{I^{D}}$ is given by

$\displaystyle\boldsymbol{\varphi}(\mathbf{x}_{n})$
$\displaystyle=\boldsymbol{\varphi}(x_{n\;1})\;\otimes\;\boldsymbol{\varphi}(x_{n\;2})\;\otimes\;\cdots\;\otimes\;\boldsymbol{\varphi}(x_{n\;D}),$
(7)

where $x_{n\;d}$ is the $(n,d)$-th entry of a matrix
$\mathbf{X}\in\mathbb{R}^{N\times D}$, where the rows are the input vectors
$\mathbf{x}_{n},\;n=1,...,N$. For all samples $\mathbf{y}=[y_{1}\;y_{2}\cdots
y_{N}]^{\top}$, (6) can be rewritten as

$\displaystyle\mathbf{y}$
$\displaystyle=\boldsymbol{\Phi}\;\mathbf{w}+\boldsymbol{\epsilon},\;\mathrm{where}\;\boldsymbol{\Phi}=[\boldsymbol{\varphi}(\mathbf{x}_{1})\;\boldsymbol{\varphi}(\mathbf{x}_{2})\;\cdots\;\boldsymbol{\varphi}(\mathbf{x}_{N})]^{\top}\in\mathbb{R}^{N\times
I^{D}}.$ (8)

The second and third model choice is to use the quadratic loss function and a
Tikhonov regularization, respectively. This results in a least square problem
for kernel ridge regression given by

$\displaystyle\min_{\mathbf{w}}\;\|\mathbf{y}-\boldsymbol{\Phi}\mathbf{w}\|_{2}^{2}+\lambda\;||\mathbf{w}||_{2}^{2}\;\;\mathrm{and}\;\mathbf{w}=(\boldsymbol{\Phi}^{\top}\boldsymbol{\Phi}+\lambda\;\mathbf{I})^{-1}\boldsymbol{\Phi}^{\top}\mathbf{y},$
(9)

with $\lambda\in\mathbb{R^{+}}$ as a regularization parameter. Solving (9)
explicitly would require constructing and inverting the matrix
$\boldsymbol{\Phi}^{\top}\boldsymbol{\Phi}+\lambda\;\mathbf{I}\in\mathbb{R}^{I^{D}\times
I^{D}}$. This would have a complexity of $\mathcal{O}(NI^{2D}+I^{3D})$. Making
this problem scalable calls for a more efficient approach. Applying TNs, the
least squares problem (9) can be solved with neither constructing nor
inverting the matrix
$\boldsymbol{\Phi}^{\top}\boldsymbol{\Phi}+\lambda\;\mathbf{I}$ explicitly.
The key idea is to replace $\boldsymbol{\Phi}$ and $\mathbf{w}$ with TNs.
Concerning $\boldsymbol{\Phi}$, a tensorized representation needs to be found.
Each row of $\boldsymbol{\Phi}$ can be represented as a Kronecker product as
shown in (7). Conveniently, a tensorization of
$\boldsymbol{\varphi}(\mathbf{x}_{n})$ corresponds to a series of outer
products, denoted by $\circ$, given by

$\displaystyle\mathbfcal{P}_{n}=\boldsymbol{\varphi}(x_{n\;1})\;\circ\;\boldsymbol{\varphi}(x_{n\;2})\;\circ\;\cdots\;\circ\;\boldsymbol{\varphi}(x_{n\;D})\in\mathbb{R}^{I\times
I\times...\times I}.$ (10)

However, creating $\mathbfcal{P}_{n}$ alone does not solve the scalability
problem, since it still requires a storage complexity of $\mathcal{O}(I^{D})$.
Instead of constructing $\mathbfcal{P}_{n}$ explicitly, it is reinterpreted as
the TN $\mathbfcal{P}^{\text{\tiny TN}}_{n}$ with
$\boldsymbol{\varphi}(x_{n\;d}),\;n=1,...,N$ as the TN components. The outer
product corresponds to a rank-1 connection between the components. Considering
all $N$ samples, only $D$ matrices of size $N\times I$ need to be stored. This
results in a storage complexity of $\mathcal{O}(NID)$ for
$\mathbfcal{P}^{\text{\tiny TN}}_{n},\;n=1,...,N$, which is significantly
smaller than $\mathcal{O}(NI^{D})$ for
$\boldsymbol{\varphi}(\mathbf{x}_{n}),\;n=1,...,N$. The reduction in
complexity is a logarithmic compression, because it reduces from exponential
in $D$ to linear in $D$.

The fourth model choice is to impose a low-rank structure onto $\mathbf{w}$ by
parametrizing it as a TN. To do this, $\mathbf{w}$ first needs have a
tensorized representation given by $D$-dimensional tensor
$\mathbfcal{W}\in\mathbb{R}^{I\times I\times...\times I}$. Second, $D$ TN
components of a low-rank TN $\mathbfcal{W}^{\text{\tiny TN}}$ representing
$\mathbfcal{W}$ need to be found. In this context, different TNs can be used,
e.g. CP, Tucker, or TT decomposition, introduced in section 3.2. The
minimization problem can now be written as

$\displaystyle\min_{\mathbfcal{W}^{\text{\tiny TN}}}\;\sum_{n}||y_{n}-$
$\displaystyle\langle\mathbfcal{P}^{\text{\tiny
TN}}_{n},\;\mathbfcal{W}^{\text{\tiny
TN}}\rangle||_{2}^{2}\;{+\;\lambda\|\mathbfcal{W}^{\text{\tiny
TN}}}\|_{2}^{2}.$ (11)

To find $\mathbfcal{W}^{\text{\tiny TN}}$, a variety of optimization solvers
can be applied. The alternating linear scheme (ALS), for example, is an
iterative scheme to find the TN components in an alternating fashion. The ALS
for TT decomposition, e.g., is treated in [6, 78]. One iteration of the ALS
has a complexity of $\mathcal{O}(DN(IR)^{2}+D(IR)^{3}))$ when using the CP
decomposition. In this context, only a few iterations are necessary for
convergence as shown in [87]. This is a significant efficiency gain compared
to the original $\mathcal{O}(NI^{2D}+I^{3D})$. TNs are therefore a promising
tool for Green AI.

To illustrate how Green AI can be put into practice, we present exemplifying
experiments. All experiments333https://github.com/TUDelft-DeTAIL/Towards-
Green-AI-with-TNs were performed with Matlab R2020b on a Dell Inc. Latitude
7410 laptop with 16 GB of RAM and an Intel Core i7-10610U CPU running at 1.80
GHz. In our experiment, we use the T-KRR
implementation444https://github.com/fwesel/T-KRR available under MIT-license
[87], which uses deterministic Fourier features and a CP decomposition to
solve (11). The experiments are performed on the airline data set
($N=5,929,413$, $D=8$). We compare the T-KRR algorithm for $R=20$ to the
solution in (9) for $\lambda=0$, in terms of runtime. We report runtime and
number of parameters for an increasing number of basis functions $I$ in Table
1. The runtime is given in terms of a mean and standard deviation over three
runs. Computations that failed due to memory errors are denoted not applicable
(NA). We consider a subset of the data $N=100,000$ and the whole data set
$N=5,929,413$. In both cases, we use $\nicefrac{{2}}{{3}}$$N$ of the data for
training. As reported in Table 1, solving (9) is not possible for $I>3$, while
the T-KRR can handle all cases. The number of parameters and the runtime grow
exponentially for (9), while both grow linearly for T-KRR. For $I=40$ and
$N=5,929,413$, the T-KRR algorithm requires
$0.23\text{\,}\mathrm{k}\mathrm{W}\mathrm{h}$ electricity and causes
$104.56\text{\,}\mathrm{g}$ CO2e emissions, corresponding to driving a car for
$600\text{\,}\mathrm{m}$. The numbers were computed with [52].

To discuss the trade-off between accuracy and efficiency from a Green AI
perspective, we consider the accuracy, reported hardware and runtime of three
state-of-the-art methods for regression with the full airline data set:
Hilbert-GP [76], stochastical variational inference for GPs (SVIGP) [33], and
T-KRR. The Hilbert-GP attains a predictive mean squared error (MSE) of
$0.827\pm 0.005$ and $41\text{\,}\mathrm{s}$ on a MacBook Pro. The SVIGP
achieves an MSE of $0.791\pm 0.005$ in $18\,360\text{\,}\mathrm{s}$ on a
cluster. The T-KRR attains an MSE of $0.763\pm 0.007$ in
$7141\text{\,}\mathrm{s}$ on a Dell laptop. Having the two highest accuracies,
SVIGP and T-KRR enable all possible interactions between the selected basis
functions by using multiplicative kernels. In contrast, Hilbert-GP, using an
additive kernel structure, achieves the best efficiency. The additive kernel
structure limits the interaction between the basis functions at a cost of
accuracy. Comparing both models with multiplicative kernels, T-KRR is more
efficient than SVIGP. The efficiency gain compared to SVIGP was achieved by
exploiting the logarithmic compression potential of TNs. In addition, T-KRR
attains the highest accuracy for the full airline data set among all three
methods.

Table 1: Runtime and number of parameters for direct inversion and T-KRR algorithm ($R=20$), varying the number of basis functions $I$ and size of the data set for $N_{1}=100,000$, $N_{2}=5,929,413$. | Runtime $[\text{s}]$ |  |  |  | Parameters |
---|---|---|---|---|---|---
$I$ |  $\boldsymbol{\Phi}^{\dagger}\mathbf{y}$ $\tiny{N=N_{1}}$ |  $\boldsymbol{\Phi}^{\dagger}\mathbf{y}$ $\tiny{N=N_{2}}$ |  T-KRR [87] $\tiny{N=N_{1}}$ |  T-KRR [87] $\tiny{N=N_{2}}$ | $I^{D}=I^{8}$ | $RID$
2 | $1\pm 0$ | $115\pm 13$ | $4\pm 1$ | $269\pm 1$ | $256$ | $320$
3 | $500\pm 5$ | NA | $6\pm 1$ | $355\pm 1$ | $6,561$ | $480$
4 | NA | NA | $8\pm 1$ | $450\pm 4$ | $65,536$ | $640$
10 | NA | NA | $22\pm 0$ | $1302\pm 15$ | $10^{8}$ | $1600$
20 | NA | NA | $54\pm 0$ | $3424\pm 10$ | $2.56\cdot 10^{10}$ | $3200$
40 | NA | NA | $145\pm 9$ | $9238\pm 393$ | $6.55\cdot 10^{12}$ | $6400$

###  4.2 TNs in deep learning

Deep Learning (DL) has achieved state-of-the-art performance in many fields,
such as computer vision (CV) [30, 50] and NLP [9, 22]. The success, however,
comes at a cost: models in DL are large and require a lot of compute [79, 80].
Neural networks (NNs) have been made more efficient with TNs for a variety of
application fields, including CV [40, 45, 53] and NLP [35, 57]. This section
provides an example of how a TN can make a NN layer more efficient and in
general how the set of efficiency metrics proposed for Green AI are reported
for TNs in DL.

We consider solving the learning problem of (5) where
$f(\mathbf{x}\mid\mathbf{w})$ is parametrized by a NN. TNs can be introduced
into NN layer(s) and make NNs more efficient. To this end, we work out the
details for a fully connected (FC) layer compressed with a TT decomposition
for supervised learning, following the methodology of [61]. The main operation
in a FC layer is a matrix vector product $\mathbf{W}\mathbf{x}$ between the
weights $\mathbf{W}\in\mathbb{R}^{I^{D}\times J^{D}}$ and input
$\mathbf{x}\in\mathbb{R}^{J^{D}}$. To simplify notation, we omit the bias
term. Before a TN can be applied, the weight matrix $\mathbf{W}$ and input
vector $\mathbf{x}$ need to be tensorized into
$\mathbfcal{W}\in\mathbb{R}^{I\times...\times I\times J\times...\times J}$ and
$\mathbfcal{X}\in\mathbb{R}^{J\times...\times J}$, respectively. Without loss
of generality and for notational simplicity and we set $I_{i}=I$ and $J_{j}=J$
for $i,j\in{1,...,D}$. The key idea is to replace $\mathbfcal{W}$ with a TT
decomposition by applying the $D$-dimensional form of (3). Without loss of
generality, the ranks $R_{2},...,R_{D}$ are set equal to $R$. The entries of
the matrix vector product in tensor format can then be computed as

$\sum_{r_{2}=1}^{R}\sum_{r_{3}=1}^{R}\cdots\sum_{r_{D}=1}^{R}\sum_{j_{1}\dots
j_{D}}w^{(1)}_{i_{1}j_{1}r_{2}}w^{(2)}_{r_{2}i_{2}j_{2}r_{3}}\cdots
w^{(d)}_{r_{D}i_{D}j_{D}}x_{j_{1}\dots j_{D}}.$ (12)

Introducing TTs in FC layers reduces the computational complexity of the
forward pass from $\mathcal{O}(I^{D}J^{D})$ to
$\mathcal{O}(DR^{2}I\max\\{I^{D},J^{D}\\})$. The learning complexity of one
backward pass is reduced from $\mathcal{O}(I^{D}J^{D})$ to
$\mathcal{O}(D^{2}R^{4}I\max\\{I^{D},J^{D}\\})$. The memory complexity is
reduced from $\mathcal{O}(I^{D}J^{D})$ to
$\mathcal{O}(R\max\\{I^{D},J^{D}\\})$ for the forward pass and to
$\mathcal{O}(R^{3}\max\\{I^{D},J^{D}\\})$ for the backward pass.

Generally, TNs can be introduced into NNs in two methodologies: Compressing a
pretained NN or training a NN in its compressed form. In this context, a NN
that is (partly) represented by TNs is referred to as factorized NN. The first
methodology takes a pretrained network, which is commonly available [1, 65].
It decomposes layers using (4) to minimize the approximation error on the
pretrained weights. Then it is common to fine-tune the factorized network to
recover performance lost in the decomposition step [21, 45, 53]. The second
methodology randomly initializes a factorized network and trains it in
factorized form, which can be used to make training more efficient due to the
reduced number of parameters [92]. During fine-tuning in the first methodology
or training in the second methodology back-propagation is based on (5), where
the weights are in TN form. The example of the TT layer works for both
methodologies. Besides TT [14, 23, 24, 61, 83, 89, 91, 93], other
decompositions, e.g. Tucker [10, 15, 45] and CP [5, 12, 21, 40, 49, 53, 58,
70] have been proposed for layers such as FC, convolutional, recurrent, and
attention layers.

Finally, we illustrate how TNs in DL contribute towards Green AI by reporting
efficiency metrics, as shown for CP in convolutional layers [45] and for
recurrent layers with another TN called Block-Term decomposition [92]. These
works compare their TN-based NN to a baseline not involving TNs. For Green AI,
both total training time and inference efficiency are relevant. Making
training more efficient can be achieved by computing one epoch faster and/or
needing fewer epochs. Runtime and electricity usage are often reported for
parsing one observation through the model. In [45] a reduction in runtime and
electricity usage by factors of up to 2.33 and 4.26, respectively, is
reported. This comes at the cost of a drop in top-5 accuracy of up to 1.70%.
Reducing the number of epochs results in faster training. The same validation
accuracy as the baseline model is achieved with 14 times fewer epochs in [92].
In addition, [45] reports a reduction in the number of parameters of up to
7.40 times and [92] shows $17,388$ times fewer parameters over a 15.6%
increase in accuracy. Reducing the number of parameters also reduces the
number of FLOPS. In [45] a FLOPS reduction of up to 4.93 times is shown. In
summary, these examples show the potential for efficiency gains by
parametrizing NNs with TNs.

##  5 Conclusion

In this paper, we addressed three research questions. First, we answered the
question of why both accuracy and efficiency should define a better AI
algorithm. Taking both accuracy and efficiency into account has the potential
to incentivize the development of efficient algorithms and reduce compute.
This will in turn have a positive impact on both sustainability and AI
progress. Second, we answered the question of why TNs can make AI algorithms
more efficient. We mathematically showed efficiency gains by means of a kernel
machine and deep learning example. We demonstrated that the key to efficiency
of TNs is their logarithmic compression potential. Third, we answered the
question of why TNs are an appropriate tool to contribute toward Green AI. We
demonstrated with an exemplifying experiment that TNs improve efficiency and
we reported on multiple efficiency metrics in that context. In addition, we
discussed the trade-off between efficiency and accuracy reported for examples
in kernel methods and deep learning. Based on the examples, we concluded that
TNs can achieve competitive accuracy.

As an overall conclusion, needing less hardware and computational time, TNs
have the potential to positively impact all three dimensions of sustainability
and AI progress. First, requiring less hardware and computational time, TNs
have the potential to reduce costs. This, in turn, makes TNs attractive for
industrial applications and more economically sustainable. Second, needing
less hardware and computational time with TNs reduces the dependency on
external computing services. In that way, the barriers to participation in AI
research are lower, thus TNs have the potential to enhance inclusivity. In
addition, by being able to handle sensitive data on-site, TNs can increase
privacy protection. Third, requiring both less hardware and less computational
time, TNs have the potential to reduce embodied and operational emissions of
AI algorithms. In that way, a broader implementation of TNs can have a huge
impact on the environmental dimension of sustainability. Beyond that,
algorithmic innovation will have to rely on improving efficiency, when Moore’s
Law reaches its limit. In that context, TNs can play an important role in AI
progress.

The potential of TNs for Green AI can be limited by several things: First, an
increased hyperparameter training to find appropriate ranks can reduce the
efficiency gains achieved with TNs. Second, TNs rely on the assumption that
there are correlations present in data. Third, the positive impact on
environmental sustainability can be reduced because a higher efficiency can
lead to an increased usage of AI. This is known as the rebound effect. The
rebound effect is not inherently negative, because an increased efficiency can
contribute to more inclusivity [77]. Lower barriers to participation in AI can
again include researchers, who use AI for climate change mitigation.

Looking forward, more research on TNs is required. Among them are theoretical
aspects, such as the further integration of TNs into probabilistic frameworks
and how the parametrization of NNs affects the learning convergence rate. More
practical aspects include wider adoption of open-source libraries and packages
for TNs as well as making TNs part of fundamental AI education.

Our vision is that TNs will have a broad application in AI in diverse sectors
and contribute to more sustainable AI research. We encourage researchers to
integrate TNs in their research, address the open questions, and by this means
push AI progress forward. This paper is the first milestone to elaborate on
TNs from a Green AI perspective. We showed how TNs can make AI algorithms
better.

## References

  * [1] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.  TensorFlow: Large-scale machine learning on heterogeneous systems, 2015\.
  * [2] KLM Royal Dutch Airlines.  CO2 emission and compensation price per destination.  https://www.klm.com/travel/nl_en/images/KLMdestinationsCO2data_jan2018_tcm542-658529.pdf.  Accessed: 2022-02-03.
  * [3] Dario Amodei and Danny Hernandez.  AI and Compute.  https://openai.com/blog/ai-and-compute/.  Accessed: Feb. 2, 2022.
  * [4] Lasse F. Wolff Anthony, Benjamin Kanding, and Raghavendra Selvan.  Carbontracker: Tracking and predicting the carbon footprint of training deep learning models.  ICML Workshop on Challenges in Deploying and monitoring Machine Learning Systems, 2020.  arXiv:2007.03051.
  * [5] Marcella Astrid and Seung-Ik Lee.  CP-decomposition with tensor power method for convolutional neural networks compression.  In IEEE International Conference on Big Data and Smart Computing (BigComp), pages 115–118, 2017.  ISSN: 2375-9356.
  * [6] Kim Batselier, Zhongming Chen, and Ngai Wong.  Tensor network alternating linear scheme for MIMO volterra system identification.  Automatica, 84:26–35, 2017.
  * [7] Kim Batselier and Ngai Wong.  A constructive arbitrary-degree Kronecker product decomposition of tensors.  Numerical Linear Algebra with Applications, 24(5):e2097, 2017.
  * [8] Mathieu Blondel, Akinori Fujino, Naonori Ueda, and Masakazu Ishihata.  Higher-order factorization machines.  Advances in Neural Information Processing Systems, 29, 2016.
  * [9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language Models are Few-Shot Learners.  In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901, 2020.
  * [10] Giuseppe G. Calvi, Ahmad Moniri, Mahmoud Mahfouz, Qibin Zhao, and Danilo P. Mandic.  Compression and interpretability of deep neural networks via tucker tensor layer: From first principles to tensor valued back-propagation.  arXiv:1903.06133 [cs, eess], 2020.
  * [11] J. Douglas Carroll and Jih Jie Chang.  Analysis of individual differences in multidimensional scaling via an n-way generalization of "Eckart-Young" decomposition.  Psychometrika, 35(3):283–319, 1970.
  * [12] Wanli Chen, Xinge Zhu, Ruoqi Sun, Junjun He, Ruiyu Li, Xiaoyong Shen, and Bei Yu.  Tensor low-rank reconstruction for semantic segmentation.  arXiv:2008.00490 [cs], 2020.
  * [13] Zhongming Chen, Kim Batselier, Johan AK Suykens, and Ngai Wong.  Parallelized tensor train learning of polynomial classifiers.  IEEE transactions on neural networks and learning systems, 29(10):4621–4632, 2017.
  * [14] Yuan Cheng, Yuchao Yang, Hai-Bao Chen, Ngai Wong, and Hao Yu.  S3-net: A fast and lightweight video scene understanding network by single-shot segmentation.  In IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3329–3337, 2021.
  * [15] Bo-Shiuan Chu and Che-Rung Lee.  Low-rank tensor decomposition for compression of convolutional neural networks using funnel regularization.  arXiv:2112.03690 [cs], 2021.
  * [16] Andrzej Cichocki.  Era of big data processing: A new approach via tensor networks and tensor decompositions.  arXiv preprint arXiv:1403.2048, 2014.
  * [17] Andrzej Cichocki, Namgil Lee, Ivan V. Oseledets, Anh-Huy Phan, Qibin Zhao, and Danilo P. Mandic.  Tensor networks for dimensionality reduction and large-scale optimization: Part 1 low-rank tensor decompositions.  Foundations and Trends® in Machine Learning, 9(4-5):249–429, 2016.
  * [18] Andrzej Cichocki, Danilo Mandic, Lieven De Lathauwer, Guoxu Zhou, Qibin Zhao, Cesar Caiafa, and Huy Anh Phan.  Tensor decompositions for signal processing applications: From two-way to multiway component analysis.  IEEE signal processing magazine, 32(2):145–163, 2015.
  * [19] Andrzej Cichocki, Anh-Huy Phan, Qibin Zhao, Namgil Lee, Ivan V. Oseledets, Masashi Sugiyama, and Danilo P. Mandic.  Tensor networks for dimensionality reduction and large-scale optimizations: Part 2 applications and future perspectives.  Learning, 9(6):431–673, 2017.
  * [20] Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, and Clifford Stein.  Introduction to algorithms.  MIT press, 2022.
  * [21] Emily L. Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus.  Exploiting linear structure within convolutional networks for efficient evaluation.  In NIPS, page 9, 2014.
  * [22] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  arXiv:1810.04805 [cs], 2019.  version: 2.
  * [23] Stavros Efthymiou, Jack Hidary, and Stefan Leichenauer.  TensorNetwork for machine learning.  arXiv:1906.06329 [cond-mat, physics:physics, stat], 2019.
  * [24] Timur Garipov, Dmitry Podoprikhin, Alexander Novikov, and Dmitry Vetrov.  Ultimate tensorization: Compressing convolutional and FC layers alike.  arXiv:1611.03214 [cs], 2016.
  * [25] Adrià Garriga-Alonso, Carl Edward Rasmussen, and Laurence Aitchison.  Deep convolutional networks as shallow Gaussian processes.  In International Conference on Learning Representations, 2018.
  * [26] Udit Gupta, Young Geun Kim, Sylvia Lee, Jordan Tse, Hsien-Hsin S Lee, Gu-Yeon Wei, David Brooks, and Carole-Jean Wu.  Chasing carbon: The elusive environmental footprint of computing.  In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 854–867. IEEE, 2021.
  * [27] David Guyon, Anne-Cécile Orgerie, Chrtistine Morin, and Deb Agarwal.  How much energy can green HPC cloud users save?  In 2017 25th Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP), pages 416–420. IEEE, 2017.
  * [28] Barbara Hammer and Kai Gersmann.  A note on the universal approximation capability of support vector machines.  neural processing letters, 17(1):43–53, 2003.
  * [29] Richard Harshman.  Foundations of the PARAFAC procedure: Models and conditions for an “explanatory” multimodal factor analysis.  UCLA Working Papers in Phonetics, 16(10):1– 84, 1970.
  * [30] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778. IEEE, 2016.
  * [31] Lifang He, Kun Chen, Wanwan Xu, Jiayu Zhou, and Fei Wang.  Boosted sparse and low-rank tensor regression.  Advances in Neural Information Processing Systems, 31, 2018.
  * [32] Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau.  Towards the systematic reporting of the energy and carbon footprints of machine learning.  Journal of Machine Learning Research, 21(248):1–43, 2020.
  * [33] James Hensman, Nicolo Fusi, and Neil D. Lawrence.  Gaussian processes for big data.  arXiv preprint arXiv:1309.6835, 2013.
  * [34] Danny Hernandez and Tom B Brown.  Measuring the algorithmic efficiency of neural networks.  arXiv preprint arXiv:2005.04305, 2020.
  * [35] Oleksii Hrinchuk, Valentin Khrulkov, Leyla Mirvakhabova, Elena Orlova, and Ivan V. Oseledets.  Tensorized embedding layers.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4847–4860. Association for Computational Linguistics, 2020\.
  * [36] Chris Huntingford, Elizabeth S Jeffers, Michael B Bonsall, Hannah M Christensen, Thomas Lees, and Hui Yang.  Machine learning and artificial intelligence to aid climate change research and preparedness.  Environmental Research Letters, 14(12):124007, 2019.
  * [37] IPCC.  Global warming of $1.5^{\circ}$\mathrm{C}$$. An IPCC special report on the impacts of global warming of 1.5°c above pre-industrial levels and related global greenhouse gas emission pathways, in the context of strengthening the global response to the threat of climate change, sustainable development, and efforts to eradicate poverty.  World Meteorological Organization, 2018.
  * [38] IPCC.  Climate change 2022: Mitigation of climate change. contribution of working group III to the sixth assessment report of the intergovernmental panel on climate change.  Cambridge University Press, 2022.
  * [39] Pavel Izmailov, Alexander Novikov, and Dmitry Kropotov.  Scalable Gaussian processes with billions of inducing inputs via tensor train decomposition.  In International Conference on Artificial Intelligence and Statistics, pages 726–735. PMLR, 2018.
  * [40] Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman.  Speeding up convolutional neural networks with low rank expansions.  arXiv:1405.3866, 2014.
  * [41] Yuwang Ji, Qiang Wang, Xuan Li, and Jie Liu.  A survey on tensor techniques and applications in machine learning.  IEEE Access, 7:162950–162990, 2019.
  * [42] Ridvan Karagoz and Kim Batselier.  Nonlinear system identification with regularized tensor network b-splines.  Automatica, 122:109300, 2020.
  * [43] Nikos Kargas and Nicholas D Sidiropoulos.  Supervised learning and canonical decomposition of multivariate functions.  IEEE Transactions on Signal Processing, 69:1097–1107, 2021.
  * [44] Boris N. Khoromskij.  O (dlog n)-quantics approximation of nd tensors in high-dimensional numerical modeling.  Constructive Approximation, 34(2):257–280, 2011.
  * [45] Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin.  Compression of deep convolutional neural networks for fast and low power mobile applications.  In ICLR, 2016.
  * [46] Donald E. Knuth.  Fundamental algorithms.  1973\.
  * [47] Donald E. Knuth.  Big omicron and big omega and big theta.  ACM Sigact News, 8(2):18–24, 1976.
  * [48] Tamara G. Kolda and Brett W. Bader.  Tensor decompositions and applications.  SIAM review, 51(3):455–500, 2009.
  * [49] Jean Kossaifi, Antoine Toisoul, Adrian Bulat, Yannis Panagakis, Timothy M. Hospedales, and Maja Pantic.  Factorized higher-order CNNs with an application to spatio-temporal emotion estimation.  In Computer Vision and Pattern Recognition, pages 6060–6069, 2020\.
  * [50] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  ImageNet classification with deep convolutional neural networks.  In Neural Information Processing Systems, 2012.
  * [51] Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres.  Quantifying the carbon emissions of machine learning.  Workshop on Tackling Climate Change with Machine Learning at NeurIPS 2019, 2019.
  * [52] Loïc Lannelongue, Jason Grealey, and Michael Inouye.  Green algorithms: Quantifying the carbon footprint of computation.  Advanced Science, 2021.
  * [53] Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan V. Oseledets, and Victor Lempitsky.  Speeding-up convolutional neural networks using fine-tuned CP-decomposition.  In International Conference Learning Representations, 2015.
  * [54] Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein.  Deep neural networks as Gaussian processes.  arXiv preprint arXiv:1711.00165, 2017.
  * [55] Charles E. Leiserson, Neil C. Thompson, Joel S. Emer, Bradley C. Kuszmaul, Butler W. Lampson, Daniel Sanchez, and Tao B. Schardl.  There’s plenty of room at the top: What will drive computer performance after Moore’s law?  Science, 368(6495), 2020.
  * [56] Kadan Lottick, Silvia Susai, Sorelle A. Friedler, and Jonathan P. Wilson.  Energy usage reports: Environmental awareness as part of algorithmic accountability.  Workshop on Tackling Climate Change with Machine Learning at NeurIPS 2019, 2019.  arXiv:1911.08354.
  * [57] Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Ming Zhou, and Dawei Song.  A tensorized transformer for language modeling.  In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [58] Franck Mamalet and Christophe Garcia.  Simplifying ConvNets for fast learning.  In Artificial Neural Networks and Machine Learning – ICANN 2012, pages 58–65. Springer, 2012.
  * [59] United Nations.  The Paris Agreement.  https://www.un.org/en/climatechange/paris-agreement.  Accessed: May 17, 2022.
  * [60] Roman Novak, Lechao Xiao, Yasaman Bahri, Jaehoon Lee, Greg Yang, Jiri Hron, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-dickstein.  Bayesian deep convolutional networks with many channels are Gaussian processes.  In International Conference on Learning Representations, 2018.
  * [61] Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov.  Tensorizing neural networks.  In Neural Information Processing Systems, 2015.
  * [62] Alexander Novikov, Mikhail Trofimov, and Ivan V. Oseledets.  Exponential machines.  In ICLR, 2017.
  * [63] Ivan V. Oseledets.  Approximation of 2d x 2d Matrices using Tensor Decomposition.  SIAM Journal on Matrix Analysis and Applications, 31(4):2130–2145, 2010.
  * [64] Ivan V. Oseledets.  Tensor-train decomposition.  SIAM J. Sci. Comput., 33(5):2295–2317, 2011.
  * [65] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  PyTorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett, editors, Advances in neural information processing systems 32. Curran Associates, Inc., 2019.
  * [66] Ben Purvis, Yong Mao, and Darren Robinson.  Three pillars of sustainability: In search of conceptual origins.  Sustainability science, 14(3):681–695, 2019.
  * [67] C. E. Rasmussen and C. K. I. Williams.  Gaussian Processes for Machine Learning.  The MIT Press, 2006.
  * [68] Steffen Rendle.  Factorization machines.  In 2010 IEEE International conference on data mining, pages 995–1000. IEEE, 2010.
  * [69] Lorenz Richter, Leon Sallandt, and Nikolas Nüsken.  Solving high-dimensional parabolic PDEs using the tensor train format.  In International Conference on Machine Learning, pages 8998–9009, 2021.
  * [70] Roberto Rigamonti, Amos Sironi, Vincent Lepetit, and Pascal Fua.  Learning separable filters.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2754–2761, 2013.
  * [71] David Rolnick, Priya L Donti, Lynn H Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, Andrew Slavin Ross, Nikola Milojevic-Dupont, Natasha Jaques, Anna Waldman-Brown, et al.  Tackling climate change with machine learning.  ACM Computing Surveys (CSUR), 55(2):1–96, 2022.
  * [72] Victor Schmidt, Kamal Goyal, Aditya Joshi, Boris Feld, Liam Conell, Nikolas Laskaris, Doug Blank, Jonathan Wilson, Sorelle Friedler, and Sasha Luccioni.  CodeCarbon: Estimate and Track Carbon Emissions from Machine Learning Computing.  https://github.com/mlco2/codecarbon, 2021.
  * [73] Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni.  Green AI.  Communications of the ACM, 63(12):54–63, 2020.
  * [74] Nicholas D. Sidiropoulos, Lieven De Lathauwer, Xiao Fu, Kejun Huang, Evangelos E. Papalexakis, and Christos Faloutsos.  Tensor decomposition for signal processing and machine learning.  IEEE Transactions on Signal Processing, 65(13):3551–3582, 2017\.
  * [75] Marco Signoretto, Lieven De Lathauwer, and Johan A.K. Suykens.  A kernel-based framework to tensorial data analysis.  In International Conference on Artificial Neural Networks, volume 24, pages 861–874, 2011.
  * [76] Arno Solin and Simo Särkkä.  Hilbert space methods for reduced-rank Gaussian process regression.  Statistics and Computing, 30(2):419–446, 2020.
  * [77] Steve Sorrell, Birgitta Gatersleben, and Angela Druckman.  Energy sufficiency and rebound effects.  Prepared for ECEEE’S Energy Sufficiency Project, 2018.
  * [78] E. Miles Stoudenmire and David J Schwab.  Supervised learning with quantum-inspired tensor networks.  arXiv:1605.05775, 2016.
  * [79] Emma Strubell, Ananya Ganesh, and Andrew McCallum.  Energy and policy considerations for deep learning in NLP.  arXiv preprint arXiv:1906.02243, 2019.
  * [80] Emma Strubell, Ananya Ganesh, and Andrew McCallum.  Energy and policy considerations for modern deep learning research.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13693–13696, 2020.
  * [81] Guglielmo Tamburrini.  The AI carbon footprint and responsibilities of AI scientists.  Philosophies, 7(1):4, 2022.
  * [82] Neil C. Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F Manso.  The computational limits of deep learning.  arXiv preprint arXiv:2007.05558, 2020.
  * [83] A. Tjandra, S. Sakti, and S. Nakamura.  Compressing recurrent neural network with tensor train.  In 2017 International Joint Conference on Neural Networks (IJCNN), pages 4451–4458, 2017.  ISSN: 2161-4407.
  * [84] Ledyard R. Tucker.  Some Mathematical Notes on Three-Mode Factor Analysis.  Psychometrika, 31(3):279–311, 1966.
  * [85] UNEP.  Emissions gap report 2019.  UN Environment Programme, 2019.
  * [86] Sander Wahls, Visa Koivunen, H Vincent Poor, and Michel Verhaegen.  Learning multidimensional fourier series with tensor trains.  In 2014 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pages 394–398. IEEE, 2014.
  * [87] Frederiek Wesel and Kim Batselier.  Large-scale learning with fourier features and tensor decompositions.  Advances in Neural Information Processing Systems, 34, 2021.
  * [88] Papers with Code.  ML Reproducibility Challenge 2021.  https://paperswithcode.com/rc2021.  Accessed: May 18, 2022.
  * [89] Bijiao Wu, Dingheng Wang, Guangshe Zhao, Lei Deng, and Guoqi Li.  Hybrid tensor decomposition in neural network compression.  Neural Networks, 132:309–320, 2020.
  * [90] Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga Behram, James Huang, Charles Bai, et al.  Sustainable AI: Environmental implications, challenges and opportunities.  arXiv:2111.00364, 2021.
  * [91] Yinchong Yang, Denis Krompass, and Volker Tresp.  Tensor-train recurrent neural networks for video classification.  arXiv:1707.01786 [cs], 2017.
  * [92] Jinmian Ye, Linnan Wang, Guangxi Li, Di Chen, Shandian Zhe, Xinqi Chu, and Zenglin Xu.  Learning compact recurrent neural networks with block-term tensor decomposition.  In Computer Vision and Pattern Recognition, page 10, 2018.
  * [93] Rose Yu, Stephan Zheng, Anima Anandkumar, and Yisong Yue.  Long-term forecasting using higher order tensor RNNs.  JMLR, 2019.