# Towards Sustainable Computing: Assessing the Carbon Footprint of
Heterogeneous Systems

Vidya A. Chhabria1, Chetan Choppali Sudarshan1, Sarma Vrudhula1, and Sachin S.
Sapatnekar2
1Arizona State University; 2University of Minnesota

###### Abstract

Decades of progress in energy-efficient and low-power design have successfully
reduced the operational carbon footprint in the semiconductor industry.
However, this has led to an increase in embodied emissions, encompassing
carbon emissions arising from design, manufacturing, packaging, and other
infrastructural activities. While existing research has developed tools to
analyze embodied carbon at the computer architecture level for traditional
monolithic systems, these tools do not apply to near-mainstream heterogeneous
integration (HI) technologies. HI systems offer significant potential for
sustainable computing by minimizing carbon emissions through two key
strategies: “reducing” computation by reusing pre-designed chiplet IP blocks
and adopting hierarchical approaches to system design. The reuse of chiplets
across multiple designs, even spanning multiple generations of integrated
circuits (ICs), can substantially reduce embodied carbon emissions throughout
the operational lifespan. This paper introduces a carbon analysis tool
specifically designed to assess the potential of HI systems in facilitating
greener VLSI system design and manufacturing approaches. The tool takes into
account scaling, chiplet and packaging yields, design complexity, and even
carbon overheads associated with advanced packaging techniques employed in
heterogeneous systems. Experimental results demonstrate that HI can achieve a
reduction of embodied carbon emissions up to 70% compared to traditional large
monolithic systems. These findings suggest that HI can pave the way for
sustainable computing practices, contributing to a more environmentally
conscious semiconductor industry.

##  I Introduction

All aspects of computing, from small chips to large datacenters, come with a
carbon footprint (CFP) price tag. For several decades, the semiconductor
industry has focused on making chips smaller, faster, and less power-hungry,
but few efforts have considered the impact on the environment. The dramatic
increase in the demand for compute in the past two decades, fueled by new
applications (e.g., artificial intelligence) that demand at-edge and at-cloud-
scale computing, has resulted in the information and computing technology
(ICT) sector contributing to more than 2% of the world’s CFP [1] – half that
of the aviation industry [2] and projected to surpass it in the next decade if
left unchecked.

Fig. 1 shows the life cycle assessment (LCA) of an electronic product and
highlights the different sources of greenhouse gases (GHG) in the life of a
semiconductor component. The operational costs are the end-user-generated CFP
which in the case of a data center are the data-to-day activities that draw
energy, while the embodied costs are the costs that come from design,
manufacturing, packaging, and materials sourcing of the server class compute
resources in the datacenter.

Figure 1: Embodied and operational CFP in the semiconductor supply chain [3].

While technology scaling and electronic design automation (EDA) have helped
design energy-efficient VLSI systems with lower operational CFP, the
environmental footprint has still been increasing over the past decade and is
dominated by carbon emissions from chip design and manufacturing [4, 3, 5]. It
is imperative to look beyond low power and energy-efficiency-driven metrics
for VLSI system design, and it is essential to use embodied carbon emissions
as a direct optimization metric. For sustainable use of today’s modern
computing, there is a need for design techniques that not only meet the power,
performance, and area (PPA) targets of today’s systems but also consider the
CFP. Several technology companies have pledged to limit their CFP [6, 7, 8],
and this can only be achieved by adopting approaches that are cognizant of the
embodied CFP.

Two prior bodies of work focus on embodied CFP estimation at the architectural
level: the first [9, 10] and the second [11, 4]. The work in [10] reformulated
the Kaya identity [12] to understand how the global CFP of computer systems
evolves over time and has made a case to lower chip sizes to lower embodied
CFP and [9] creates a very simple model based on first principles. The work in
[4, 11] has created a data-driven model, from publicly available
sustainability reports from industry [6, 13, 3, 7], for embodied carbon
estimation for a given input hardware architecture and has created a platform
for carbon-aware design space exploration (DSE). While these works have set a
new paradigm toward sustainable computing, they are limited in scope:

  1. 1.

They do not accurately consider the overheads for packaging, which is crucial
for heterogeneous systems. ACT [4] uses a fixed value that does not consider
the size of the package, the yield of the package, or the assembly process,
while [10] does not consider any CFP from packaging.

  2. 2.

They do not apply to heterogeneous systems with chiplets in different
technology nodes integrated into a single advanced package which is near-
mainstream today.

  3. 3.

They do not consider the carbon that comes from the design of individual
chips/chiplets, which, even though amortized across all manufacturing
components, is a significant contributor to the total embodied CFP.

Also, prior package-level carbon analysis tools have been limited to only ball
grid arrays and monolithic flip-chip technologies [14] and have not been
studied in the context of sustainable computing or HI.

With Moore’s law slowing down and SRAM and analog components not scaling [15,
16], the way forward towards sustaining Moore’s law to the era of trillion-
transistor systems and beyond is through HI [17, 18]. Instead of building
system functionality on a single die, HI integrates a set of chiplets, each
corresponding to the single die of today, onto a substrate that enables high-
density, high-bandwidth chiplet-to-chiplet interconnections. Recent and
upcoming advances in HI, including the rapid shrinking of bond pitches between
chiplets and interposers [19, 20, 21, 22, 23], are enabling the design of
increasingly sophisticated integrated systems. This calls for carbon modeling
tools at the architecture level that can model HI systems and not just
monolithic dies as in [4]. New CFP models must be created that account for
packaging overheads, silicon fabrics, and multi-die system integration in
different technology nodes. These modeling tools can be embedded into still-
evolving HI design methodologies to optimize HI systems for power,
performance, area, and carbon (PPAC).

Inspired by the principles of environmental sustainability – the three R’s of
“reduce, reuse, and recycle” – in this paper, we evaluate the potential of HI
systems towards sustainable computing by developing a carbon analysis tool to
measure design, manufacturing, and packaging CFP for heterogeneous systems. HI
systems have a potential for sustainable computing by “reducing” carbon
emissions by reducing the computation involved in designing each component
from scratch and by “reusing” pre-designed chiplet IP blocks through
hierarchical approaches. The ability to reuse chiplets across several designs
not only in the current generation of ICs but even in the next generation can
massively amortize the embodied CFP.

In this paper, we introduce an embodied CPF estimator specifically tailored
for heterogeneous systems that incorporate advanced packaging architectures.
Our goal is to demonstrate the potential of such systems in reducing CFP
compared to large monolithic dies, even when accounting for the overhead
associated with advanced packaging techniques. To estimate the manufacturing
CFP of chiplets in heterogeneous systems, we extend existing models from [4]
and [5] with modifications to account for area, yield, defect densities, and
energy-efficiency of process equipment which all scale as the technology
matures. Our packaging CFP models account for four types of wafer-level
packaging architectures: RDL Fan-out, silicon bridges such as embedded multi-
die interconnect bridge (EMIB), passive interposers, and active interposers.
Further, we also consider the overheads with respect to inter-die whitespace
and additional inter-die routing overheads as a part of the packaging CFP
estimation. For design CFP, we employ a simplified model based on design
compute time and EDA tool productivity with every technology node, to predict
the carbon emissions associated with the design phase.

The key contributions of our paper are as follows:

  1. 1.

To the best of our knowledge, this is the first work to propose HI as a
direction toward sustainable computing.

  2. 2.

We develop a carbon analysis tool to estimate the embodied CFP (design,
manufacturing, and packaging) of HI systems, accounting for a variety of
packaging architectures, scaling, yield, process equipment energy efficiency,
and EDA tool productivity.

  3. 3.

We build a novel HI packaging CFP estimator to predict the advanced packaging
overhead costs, considering whitespaces on the package substrate and inter-die
communication overheads.

  4. 4.

We evaluate our tool and the potential HI has in reducing embodied CFP on a
diverse set of industry testcases (mobile processors, GPUs, and CPUs) and find
that HI systems can reduce embodied CFP by upto 70%.

##  II HI and its scope for sustainable computing

###  II-A HI and different packaging architectures

HI has offered a feasible approach for cost-effective chip design which can
help sustain Moore’s law. An HI system splits a large SoC into multiple
smaller dies, referred to as chiplets, where each chiplet may have a different
functionality, potentially built in different process nodes or designed by
separate vendors reducing both design time and cost. All chiplets are
integrated into a single package. An HI system may come in different packaging
architectures, as shown in Fig. 2 that vary in cost and complexity. Multi-die
heterogeneous systems can have anywhere between two chiplets to tens of
different chiplets and depending on the number of chiplets, budgeted cost, and
complexity, the choice of packaging architecture for heterogeneous systems is
different [24]. We describe four commonly used advanced packaging and
integration technologies:

Figure 2: Packaging architectures: (a) RDL fanout packaging in EMC, (b) thin-
film and silicon bridge architecture (Intel’s EMIB, TSMC’s LSI), (c) 2.5D
integration with active or passive interposers and (4) 3D integration.

(1) RDL fan-out integration As shown in Fig. 2(a), it involves the integration
of multiple chiplets on a package substrate or fan-out redistribution layer
(RDL) substrate. Typically the package substrate consists of 3-4 RDL metal
layers with linewidths and spacings (L/S) varying from 6/6 to 10/10$\mu$m.

(2) Thin film and silicon bridge-based integration In this integration
technology, the package substrate has thin-film layers defined as embedding
fine metal RDLs or a silicon bridge on top of a build-up organic package
substrate or in a fan-out epoxy molding compound (EMC) substrate as
highlighted in Fig. 2(b). Intel’s embedded multi-die interconnect bridge
(EMIB) and TSMC’s local silicon interconnect (LSI) is an example of this
technology. The technology uses local silicon bridges to host ultra-fine L/S
structures for die-to-die interconnect communications of about 2$\mu$m.

(3) Passive and active interposer-based integration This corresponds to
multiple chiplets in the package that are supported by a through-silicon via
(TSV)-less or TSV-based active/passive interposer, and then attached to a
package substrate, as shown in Fig. 2(c). This technology is typically termed
as a 2.5D architecture. The active interposer consists of both FEOL and BEOL
layers while the passive interposer consists of BEOL layers and are both
typically implemented in an older technology node.

(4) 3D integration Here, active interposers are used to support the chiplets
and then attached to the packaging substrate, or multiple chiplets are stacked
over the packaging substrate and connected through microbumps, as shown in
Fig. 2(d), or direct bumpless bonding [25].

HI has opened up a completely new design space that was previously unexplored
by architectural-level carbon simulators [4, 10]. However, the exploration of
this space requires the development of models that can account for the
different possible design decisions in the HI system that impact the CFP. For
instance, each of the above four described packaging architectures differs in
their yields, assembly process, and material used and therefore have different
CFPs. EMIB consists of high-density interconnects with fine L/S, typically
having lower yields compared to the larger RDL layers in fan-out packaging.
Interposer-based integration strategies typically use more materials, and
layers, and have a more complex manufacturing process compared to the fan-out
RDL and EMIB architectures which results in larger CFP. However, the cost
overheads from novel packaging techniques in terms of carbon footprint are
small when compared to the large benefit from the chip manufacturing yield
improvement and design cost reduction [23]. In this work, we evaluate the
embodied CFP for these packaging architectures and evaluate the potential HI
systems have towards carbon-emission efficient computing.

###  II-B Scope for sustainable computing through HI

HI systems have great potential to lower the embodied CFP associated with
design and manufacturing when compared to their monolithic counterparts due to
several reasons:

(1) Yield and area As we pack more functionality and logic onto the same
monolithic IC, the increase in the area increases the CFP due to an increase
in materials needed for manufacturing and a decrease in yield. Fig. 3(a) shows
a result for a large industry testcase in a 10nm technology. We sweep the area
of the monolithic SoC and observe an exponential increase in the associated
manufacturing CFP due to lower yields. Since a HI system disaggregates the
monolithic system into several smaller dies, each die can be manufactured with
a significantly lower environmental cost. For example, Fig. 3(b) compares the
CFP of a monolithic GA102 testcase against a 4-chiplet representation of the
GPU where the memory and analog components are on independent chiplets, and
the large digital block is split into two smaller chiplets. The CFP of the
4-chiplet design is normalized to the monolithic design’s CFP for different
technology nodes. The 4-chiplet GPU has significantly lower manufacturing CFP
even after including the carbon overheads from packaging due to the higher
yields of the chiplets when compared to the monolithic chip.

(2) Technology node In an HI system, dies can be implemented in different
technology nodes. With analog and SRAM blocks not scaling at the same rate as
digital logic, several design houses [15, 26] use older technology node
chiplets for memory controllers and analog logic. As pointed out in [3], the
CFP to manufacture chips in older technology nodes is much lower than for
newer technology nodes due to lower defect densities (better yields), fewer
lithography steps due to fewer BEOL/FEOL layers, and the better energy-
efficiency of lithography equipment involved in manufacturing older technology
nodes with today’s latest manufacturing equipment.

Figure 3: (a) Embodied CFP in grams (g) of equivalent CO2 emission versus
area. (b) Comparison of manufacturing CFP of the large monolithic NVIDIA GA102
GPU and a 4-chiplet-based architecture of the GPU.

(3) Design time Reusing existing silicon-proven die not only saves design time
directly but also saves the associated design-time CFP. Further, the
heterogeneity that allows some chiplets to be built in older nodes is not only
easier and lower cost but also comes with lower CFP overheads. Typically, even
EDA tools scale with technology, and the latest versions of EDA tools can
perform design faster with better quality of results on an older technology
node [27] due to continuous improvements made by the EDA industry.

##  III Embodied CFP analysis framework

###  III-A Problem statement

In this work, we develop an embodied CFP analysis tool that takes an
architectural-level description of a large monolithic SoC or an architectural-
level description of the heterogeneous system and a choice of a packaging
architecture as input to estimate the total embodied carbon footprint of the
system, including chip/chiplet manufacturing CFP, HI packaging CFP, and design
CFP as output. We model the total embodied CFP of the system as the sum of the
CFP coming from the different sources highlighted in Fig. 1 and is given by:

$C_{\text{tot}}=\textstyle\sum_{i=1}^{N_{C}}C_{\text{mfg},i}+C_{\text{des}}+C_{\text{HI}}$
(1)

where $C_{\text{mfg},i}$ is the manufacturing CFP of each chiplet, $N_{C}$ is
the number of chiplets, $C_{\text{des}}$ is the design CFP of all chiplets,
and the HI system, $C_{\text{total}}$ is the total embodied CFP, and
$C_{\text{HI}}$ is contributions from manufacturing and assembly of the
advanced package and any inter-die communication overheads.

###  III-B Manufacturing CFP estimation

To estimate the manufacturing CFP of each chiplet, we make three essential
modifications to [4] to support the estimation of embodied CFP of a
heterogeneous system for system disaggregation as described below:

(1) Area scaling models: Since a system disaggregation algorithm or a
heterogeneous system requires selecting a technology node for each chiplet,
our carbon estimation tool uses transistor density scaling trends from [28,
29] and transistor counts from our testcase architectures to determine the
area of a chiplet in a specific technology node. The area scaling models are
critical to the estimation of CFP as larger chiplet areas in older technology
nodes can have larger CFP even though they have lower CFP per unit area
(CFPA). We use three different area scaling models for logic, memory, and
analog blocks, as each has different transistor densities and, therefore
different areas with every technology node. We evaluate the area of the die as
$A_{die}(d,p)=D_{T}(d,p)\times N_{T}$, where $D_{T}(d,p)$ is the transistor
density for design type $d$ and process $p$, $N_{T}$ is the number of
transistors in the die, and $A_{die}(d,p)$ is the area of die of type $d$ in
process node $p$.

(2) Yield models: One of the primary advantages of HI is the cost savings that
come with larger manufacturing yields due to smaller die sizes. The increase
in yield compared to a large monolithic die also helps lower CFP. However, if
the die is in an older technology node, then $A_{\text{die}}(d,p)$ must be
accounted for as an increase in the area may lower yields which also lower
CPF, as shown in Fig. 3(a). To estimate the impact of the area on yield and
CFP, we use a negative binomial yield distribution model given by [30, 31,
32]:

$Y(d,p)=\left(1+\frac{A_{die}(d,p)\times D_{0}(p)}{\alpha}\right)^{-\alpha}$
(2)

where $Y(d,p)$ is the yield of die with area $A_{die}(d,p)$, $D_{0}(p)$ is the
defect density for process $p$, $\alpha$ is a clustering parameter. It is
important to note that the defect density is also $p$ dependent and scales
with technology. On the one hand, legacy nodes have lower defect densities
which result in larger yields, and on the other, older technology nodes result
in larger $A_{die}$ values leading to lower yields. Our tool considers these
tradeoffs while estimating CFP.

(3) Energy-efficiency of process equipment: With advances in process
equipment, the energy efficiency of every step during photolithography
equipment improves, especially on more mature technology nodes. We incorporate
the energy efficiency of the equipment as a derate factor ($\eta_{eq}$) from
[13]. The $C_{\text{mfg},i}$ on a per chiplet basis is given by multiplying
the carbon footprint per unit area (CFPA) with the area of the die:

$\displaystyle C_{\text{mfg},i}$ $\displaystyle=\text{CFPA}\times
A_{die}(d,p)$ (3) CFPA $\displaystyle=\frac{(\eta_{eq}\times C_{\text{mfg,
src}}\times\text{EPA}(p)+C_{\text{gas}}+C_{\text{material}})}{Y(d,p)}$ (4)

where $C_{\text{src}}$ is carbon intensity which depends on the energy source
of the fab (i.e., renewables vs. non-renewables), which converts the energy
consumed into carbon emission. EPA is the energy consumed per unit area during
manufacturing of process $p$ and derived from [5], $C_{\text{gas}}$ is the CFP
from the greenhouse gas emissions, $C_{\text{material}}$ is the carbon
footprint of sourcing the materials for fabricating the chip/chiplet, and CFPA
is the carbon footprint per unit area.

###  III-C HI-oriented CFP overheads

With the projected widespread adoption of HI systems, the cost of packaging is
projected to dominate design [33]. Although there are several sustainability
reports from large semiconductor manufacturing and design companies, these
reports do not specifically break down the contributions from packaging. The
prior art in this area has been limited to wire bond packages and flip chip
packages [14]. Since HI has opened up a previously unexplored design space, it
requires developing models that can account for the different possible design
decisions in the HI system that impact the CFP. In particular, decisions
related to the choice of the package (architecture) ($C_{\text{package}}$),
whitespace on the package substrate or interposer ($C_{\text{whitespace}}$),
and inter-die communication ($C_{\text{mfg, comm}}$). In our work, we measure
the CFP from these three sources as described below:

(1) Package-related overheads ($C_{\text{package}}$): We develop models for
the four 2.5D packaging architectures, including RDL fan-out, silicon bridge,
passive and active interposers based on architectural descriptions, materials,
and packaging technology nodes from [24], CFP estimates from [5], and
packaging industry reports [34, 35, 36].

(a) RDL Fan-out: This packaging architecture uses an epoxy molding compound
(EMC) substrate with RDL metal layers patterned to make connections between
the chiplets as shown in Fig. 2(a). Our CFP model uses the energy per unit
area per metal layer (EPLA) from a manufacturing fab to determine CFP
overheads with the RDL layers. Based on the number of layers, the yield of the
layers, and EPLA, we determine the embodied CFP of an RDL package as:

$C_{\text{RDL}}=\frac{L_{\text{RDL}}\times\text{EPLA}_{\text{RDL}}(p)\times
C_{\text{pkg, src}}\times A_{\text{package}}}{Y(\text{RDL},p)}$ (5)

where $\text{EPLA}_{\text{RDL}}(p)$ is the energy consumed in patterning a
single RDL layer in process $p$ per unit area, $C_{\text{pkg,src}}$ is the
carbon intensity of the packaging fab which is based on the source of energy
(renewable or non-renewable soruces), $L_{\text{RDL}}$ is the number of layers
of RDL in the package substrate, $Y(\text{RDL},p)$ is the yield of the RDL in
process $p$ estimated using Equation (2), and $A_{\text{package}}$. The area
of the package substrate is estimated after considering the whitespace and
routing overheads and described later in this section.

(b) Silicon bridge: A silicon bridge is a high-density interconnect between
two chiplets, and we model its CFP similar to the CFP of the RDL fan-out-based
package except that they have lower linewidth and spacing (L/S) and,
therefore, lower yields when compared to RDL fan-out. The yields are lower due
to the lower bond pitches between the chiplets and the bridges and the cavity-
based fabrication of the bridge in the package substrate. For our model, we
use the EPLA values from [5] for an advanced technology node lower metal layer
with ultra-fine L/S. These high-density interconnects do not span the entire
area of the package substrate but are local to a region in the package based
on the floorplan of the chiplets. The number of silicon bridges and their
placement depends on the chiplet flooplan and bandwidth requirements. In our
work, we consider a bridge ranges and typical bridge areas from Intel’s EMIB
silicon bridge specification [37] as input to determine the number of bridges
that must be used. If the two adjacent dies that must be connected through
silicon bridges have overlapping die edges larger than the range, then an
additional bridge is added. The CFP of a silicon bridge-based packaging
architecture is given by:

$C_{\text{bridge}}=\frac{N_{\text{bridge}}\times
L_{\text{bridge}}\times\text{EPLA}_{\text{bridge}}(p)\times C_{\text{pkg,
src}}\times A_{\text{bridge}}}{Y(\text{bridge},p)}$ (6)

where $L_{\text{bridge}}$ is the number of metal layers in the bridge,
$A_{\text{bridge}}$ is the area occupied by the silicon bridge in the package
substrate, $N_{\text{bridge}}$ is the number of silicon bridges,
$Y(\text{bridge},p)$ is the yield of fabricating the silicon bridge in process
$p$ in the bridge cavity, $\text{EPLA}_{\text{bridge}}(p)$ is the energy per
unit layer per unit area of patterning the silicon bridge in process $p$.

(c) Active interposer: Active interposers are manufactured to include active
transistor devices within the interposer, providing several unique
capabilities not possible with passive interposers. We model these interposers
as an additional large die that is typically in an older technology node when
compared to the chiplets and are patterned with both FEOL layers for active
silicon and BEOL layers for interconnects. However, unlike a regular chiplet,
the active region is only restricted to local regions with routers and
repeaters.

We use a similar model based on Equation (4) to estimate CFP overhead from
active interposer. Interposer-based packaging architectures have higher CFP
when compared to the RDL fan-out-based packaging and EMIB-based packaging as
the interposer acts as an additional large silicon die that spans the entire
area of all the chiplets put together with BEOL layers across the entire
interposer and active FEOL layers locally in those areas that have routers or
repeaters.

(d) Passive interposer: Unlike active interposers, passive interposers only
contain metal interconnect, so they cannot include active logic like routers,
or repeaters in the interposer. We model the CFP of the passive interposer in
a similar way as Equation (5) on a per unit area and per layer basis.

It is important to note here that the overheads from the package substrate are
the same across all architectures and is also going to be a part of the
monolithic system. Therefore, we do not account for the CFP due to package
substrates. The package substrate of a HI system is larger than that of a
monolithic system, and the additional CFP due to the area overheads are
considered after estimating the area of the package using our whitespace
estimator algorithm.

Figure 4: Router location in the (a) active and (b) passive interposer.

(2) Inter-die communication overheads ($C_{\text{mfg, comm}}$): Unlike EMIB
and RDL-based packaging architectures, which are limited to supporting few
(four - five) chiplets [24], interposer-based packaging architectures support
tens of chiplets, but come at large inter-die c ommunication overheads which
are protocols such as network on-chip (NoC). To support an NoC router, each
chiplet must be equipped with a network interface controller (NIC) as shown in
Fig. 4. In passive interposers, router modules must be placed on the chiplets,
as shown in Fig. 4(b), contributing to chiplet area and degrading yield and
$C_{\text{mfg},i}$ while with active interposers, router modules can be moved
from the chiplets to the interposer, reducing the area in the chiplets, as
shown in Fig. 4(a) and therefore improving chiplet yield and
$C_{\text{mfg},i}$ compared to passive interposers.

To estimate the CFP overheads of routing, we model the area overhead for NoC
routers which in turn depends on the technology node in which the router is
implemented, the number of ports, and the flit widths [38, 39]. The CFP
overhead for interposer-based NoC routers for inter-die communication is given
by: $C_{\text{mfg, comm}}=\text{CFPA}\times A_{\text{router}}(d,p)$, where
CFPA is defined in (4). For the passive interposer, $A_{\text{router}}(d,p)$
is added to the area of the chiplet, after which yield and $C_{\text{mfg, i}}$
is calculated. For active interposers, the carbon contribution of
$A_{\text{router}}(d,p)$ is used to add to the overall CFP of the system
directly. It’s important to note that for passive interposers, the NoC is
implemented in the same technology node as the chiplet, which is a more
advanced node than those routers that are a part of the package. Therefore,
routers for passive interposers are of lower areas than the active interposer
router in an older technology node. For EMIB- and RDL-based packages there are
additional communication overheads for PHY [37] interfaces that are typically
part of the chiplet itself. These interfaces are typically designed as IPs and
have small additional areas when compared to the chiplets.

(3) Whitespace overheads ($C_{\text{whitespace}}$): We develop a whitespace
estimation algorithm that performs recursive bi-partitioning to build a
slicing floorplan of the chiplets on the package substrate/interposer. An
initial two-way partition is created by assigning the chiplets (sorted in
decreasing order of their area), one by one, to the partition with a lesser
total weight. Our model uses the area of each partition as the weight, which
results in an area-balanced initial partition. The bi-partitioning procedure
is then used recursively within each partition to perform a K-way partition of
the chiplets by first creating two equal-sized partitions, then independently
dividing each of these into two subpartitions each, and so on till a partition
contains only one chiplet. This effectively creates a full binary tree where
each leaf node is a chiplet and each internal node represents a partition. The
overall floorplan and its area can be derived by processing the partitions and
chiplets within the tree.

For each leaf node, processing involves setting the orientation and aspect
ratio of the chiplet to get a bounding box. At the internal nodes, this
involved combining two subpartitions together, accounting for whitespace
overheads. There are two sources of whitespace overheads: (i) some spacing
between two subpartitions due to chiplet spacing constraints [40, 38], (ii) if
the two subpartitions are imbalanced in terms of their dimensions, we created
a bounding box of the two partitions which will result in whitespace. The
recursive bipartisan floorplan also provides us with interfaces between each
pair of chiplets to identify locations for routers, and silicon bridges on the
package substrate/interposer.

###  III-D Design CFP estimation

Although design CFP ($C_{\text{des}}$) is amortized across the number of chips
manufactured, several cutting-edge accelerators, GPUs, and server CPUs are not
manufactured in sufficiently large numbers to amortize the cost of design
across the number of parts manufactured. We model the design CFP,
$C_{\text{des}}$ as:

$C_{\text{des}}=\frac{\sum_{i=1}^{N_{C}}C_{\text{des},i}+C_{\text{des,comm}}}{N_{P}}$
(7)

where $C_{\text{des},i}=t_{\text{des},i}\times P_{\text{des}}\times
C_{\text{des,src}}$ is the design CFP of a single chiplet,
$C_{\text{des,comm}}$ is the design CFP of routers for inter-die
communication, $N_{P}$ is the number of parts manufactured,
$t_{\text{{des},i}}$ is the CPU compute time it takes to design a
chip/chiplet, $P_{\text{des}}$ is the power consumed by the compute resources
(CPUs) used to design the chips, $C_{\text{des,src}}$ is the CFP of the energy
source. We model $t_{\text{des, i}}$ as:

$t_{\text{des,i}}=\frac{t_{\text{verif},i}+(t_{\text{{SP\&R},i}}+t_{\text{analyze},i})\times
N_{\text{des}}}{\eta_{\text{EDA}}}$ (8)

where $t_{\text{verif},i}$, $t_{\text{SP\&R},i}$, $t_{\text{analyze},i}$ are
the compute time for verification, and a single synthesis, place, and route
(SP&R) run and a single simulation of all analysis respectively, and
$N_{\text{des}}$ is the number of design iterations. Further, with EDA tools
improving with every new version release due to advances made by EDA research
groups. We create a near-linear regression model to predict EDA tool
improvements with every new version released based on the data from [27] and
scale the value of $t_{\text{des,i}}$ by $\eta_{\text{EDA}}$ to model EDA tool
productivity.

##  IV Evaluation of CFP Analysis Tool

###  IV-A Methodology and experimental setup

(1) Input parameters Our CFP estimator, developed in Python3.7, uses several
input parameters which are listed in Table I with their supported range of
values and their sources. For instance, based on the source of energy, whether
it is coal, gas, wind etc, the $C_{\text{mfg, src}}$ can be a value between
30g – 700g of CO2 or based on the technology node, the defect densities can be
between 0.07 – 0.3/cm2 [24].

Although our simulator can handle a range of technology nodes for packaging
and a range of derate factors for $C_{\text{mfg, src}}$, as highlighted in the
table, our results in this section are shown for specific values, i.e., we
assume all packaging technology (RDL, EMIB, and active/passive interposers) to
be in 65nm and the energy source is from coal at 700g of CO2 per KWh. Based on
the testcase, we vary the technology node for each of the chiplets to explore
the possible design space and estimate $C_{\text{mfg}},i$. Based on the
technology each chiplet is implemented in, we choose the appropriate values
from the specified ranges.

TABLE I: List of input parameters to our embodied CFP estimation tool and
their range of values and sources.

Model | Parameter | Value | Unit | Source
---|---|---|---|---
$C_{\text{mfg},i}$ | $D0(p)$ | 0.07 – 0.3 |  /cm2 | [31, 32]
$\alpha$ | 3 |  | [31, 32]
$D_{T}(d,p)$ | 5 – 150 |  MTr/mm2 | [28, 29]
$\eta_{eq}(p)$ | 0 – 1 |  | [13]
$C_{\text{mfg, src}}$ | 30 – 700 |  g CO2/kWh | [4, 5]
$\text{EPA}(p)$ | 0.8 – 3.5 |  kWh/cm2 | [4, 5]
$C_{\text{gas}}$ | 0.1 – 0.5 |  kg CO2/cm2 | [4, 5]
$C_{\text{material}}$ | 0.5 |  kg CO2/cm2 | [4, 5]
$C_{\text{package}}$ | RDL tech. | 22nm - 65nm |  | [24, 37, 38]
$\text{EPLA}_{\text{RDL}}(p)$ | 0.05 – 0.2 |  kWh/cm2 | [4, 5]
$C_{\text{pkg, src}}$ | 30 – 700 |  g CO2/kWh | [4, 5]
$L_{RDL}$ | 3 – 4 |  | [24]
$L_{\text{bridge}}$ | 3 – 4 |  | [37]
Bridge tech. | 22nm – 65nm |  | [37]
$\text{EPLA}_{\text{bridge}}(p)$ | 0.1 – 0.35 |  kWh/cm2 | [4, 5]
Bridge range |  2 $\times$ 2 |  mm2 | [37]
$C_{\text{mfg,comm}}$ | Interposer tech. | 22nm – 65nm |  | [38]
NoC flit width. | 512 bits |  | [38]
$C_{\text{whitespace}}$ | Chiplet spacing | 0.1 – 1 | mm | [38, 40]
$C_{\text{des}}$ | $\eta_{\text{EDA}}$ | 0 – 1 |  | [27]
$P_{\text{des}}$ | 10 | W | [41]
$N_{\text{des}}$ | 100 |  | [42]
$C_{\text{des, src}}$ | 30 – 700 |  g CO2/kWh | [4, 5]

(2) Testcases and architectures We evaluate our carbon simulator on four
industry testcases: (i) Intel mobile processor, Tiger Lake(2020) [43], (ii)
Intel server-class 2-chiplet-based CPU, Emerald Rapids (EMR) [44] (to be
released in Q4 2023), (iii) NVIDIA GA102 GPU (2020) [45], and (iv) Apple A15
SoC (2021) [46]. The input to our simulator is an architectural description of
these testcases with the die area breakdowns for each of these processors. We
obtain the area breakdowns of each of these testcases from third-party
websites such as [44, 45, 47, 43]. For the monolithic SoCs (Tiger Lake, GA102,
and A15) we break them into chiplets based on the block-level architecture. We
use one chiplet for memory, another chiplet for analog components, and a third
chiplet for digital logic inspired by [15]. We also further explore the design
space by splitting the large digital logic chiplet into smaller chiplets. For
Intel Emerald Rapids, an EMIB-based 2-chiplet testcase, we perform CFP
estimation for the original chiplet-based architecture. We also perform carbon
evaluation on a 4-chiplet version of the same testcase where we split the
large digital logic blocks into two smaller chiplets each.

In the rest of this section, we use the NVIDIA GA102 testcase as a case study
and show detailed results on each of the components (manufacturing, packaging,
and design) of the total embodied CFP for different possible chiplet
architectures and compare it to the CFP of the monolithic SoC. Next, in the
interest of space, for the rest of the three testcases, we only summarize the
manufacturing and packaging CFP and compare that to its monolithic
counterpart.

###  IV-B Example case study: NVIDIA GPU GA102

As an example case study, we use the NVIDIA GPU GA102 architecture to evaluate
the various components of embodied CFP for various chiplet disaggregation
scenarios of the testcase. The first set of results is for three-chiplet
scenario where one of the chiplets implements memory, the other analog
components, and the third digital blocks. The next set of results will show
the CFP for a general $N_{c}$-chiplet architecture.

(1) Manufacturing and packaging CFP for three chiplets The manufacturing CFP
of GA102, for different configurations of technology nodes for each chiplet,
is shown in Fig. 5(a). The x-axis lists the technology nodes where the
lowermost row is the technology node for the digital logic, the middle row is
analog components (IOs), and the topmost row is memory. The (7,7,7) scenario
is a monolithic representation of the architecture of a single die in a 7nm
node. It, therefore, does not have the additional HI-related packaging
overheads, but includes packaging overheads as a part of manufacturing carbon
using models from [4]. The figure shows that the lowest embodied CFP is for
the (7, 14, 10) scenario. This is because the analog components and memory
blocks [15] do not scale in the area as much as the digital blocks and can
therefore be implemented in an older technology node with almost the same
area. On the contrary, in the (10, 10, 10) scenario, the digital logic scales
to a much larger area and therefore has a larger CFP than even the monolith
resulting in a larger CFP even though the 10nm node has a lower defect density
and lower CFPA when compared to 7nm. The packaging overhead values in this
figure are for an RDL-based package with the RDL layers implemented in a 65nm
technology node. The packaging overhead has also accounted for the whitespace
between the three chiplets and routing overheads. From this result, it is
clear that HI enables using chiplets that have small areas and higher yields,
which helps lowers the CFP, and the further integration of chiplets in
different technology nodes can further lower the CFP as older nodes have lower
EPA than advanced nodes.

Figure 5: (a) Manufacturing and packaging CFP, (b) design CFP for a single
iteration of SP&R, and (c) overall CFP for different configurations of three
chiplets for the NVIDIA GA102 architecture with RDL fan-out package.

(2) Design CFP for three chiplets From our experiments in performing SP&R of
large designs, we find that the $t_{\text{SP\&R},i}$ for a design with 700,000
logic gates in a 7nm commercial technology is about 24 CPU hours. These
estimates are on a 192GB RAM machine with a dual-core Intel Xeon CPU with 8
threads, each running at a 2.4GHz clock frequency. Therefore, extending this
model to the NVIDIA GA102 testcase, $t_{\text{SP\&R},i}=1.2\times 10^{6}$ CPU
hours as it has over 4.5B logic gates. Assuming $P_{\text{des}}=10$W [41] and
the energy supplied comes from non-renewable sources, then a single run of
SP&R results in 8,400kg of CO2 equivalent emission in the 7nm technology node.
Fig. 5(b), shows the design carbon for a single iteration of SP&R for the
three chiplet testcase. Older technology nodes have lower design times due to
EDA tool scaling [27], and therefore, have lower CFP when compared to the
monolithic SoC in an advanced 7nm technology. In addition, since HI enables
the “reuse” of pre-designed chiplets, in principle, the entire digital logic
chiplet can be reused for another design saving the entire associated design
CFP.

Although, the CFP values in Fig. 5(b) are significantly large, these costs are
amortized across the number of parts manufactured ($N_{p}$). The figure only
shows the results for a single iteration of SP&R. However, with hundreds
($N_{\text{des}}=100$) of design iterations and SP&R runs and verification
dominating 80% of the product development time, the design of an IC can easily
contribute to over 2,000,000kg of CO2 equivalent emission, assuming all
compute energy is coming from non-renewable sources. Assuming the number of
manufactured parts is $N_{P}=200,000$, the SP&R carbon cost gets amortized to
12kg of CO2 equivalent emission per IC, which is more than 25% of the
manufacturing carbon cost (see Fig. 5(a)).

(3) Total embodied CFP for three-chiplet architecture To estimate the total
embodied CFP, we sum the manufacturing and packaging costs from Fig. 5(a) and
amortized design costs assuming, $N_{p}=200,000$ and $N_{\text{des}}=100$ from
Fig. 5(b). Fig. 5(c) shows the total embodied CFP of all three components for
different configurations of the three chiplet-based GA102 testcase. The (7,
14, 10) configuration has the least CFP showing the potential of HI to
bringing down CFP.

Figure 6: $C_{\text{mfg}}$ and $C_{\text{HI}}$ for $N_{c}$ chiplets on the
NVIDIA GA102 GPU testcase.

(4) Total embodied CFP with $N_{c}$ chiplets In addition to the three chiplet
architecture of GA102, we also evaluate the manufacturing and packaging carbon
for $N_{c}>3$ where we take the digital logic in 7nm and further split the
logic block into smaller chiplets each implemented in 7nm. The analog (IOs)
and memory chiplets are in 14nm and 10nm, respectively. Fig. 6 shows the
manufacturing and packaging CFP for different $N_{c}$. As $N_{c}$ increases,
the chiplet manufacturing CFP decreases due to smaller chiplets and better
yields, while the packaging CFP increases due to larger HI overheads.

(5) Packaging CFP for different architectures To understand the differences in
CFP overheads of the four different packaging architectures considered, we use
the large digital logic component of GA102 as an example testcase. We split
the 500mm2 monolithic digital logic block into $N_{c}$ different chiplets and
evaluate $C_{\text{HI}}$. Fig. 7 shows the difference in CFP for these
architectures separated by routing overheads and package-related overheads
(whitespace and area). For the four architectures, all the interconnects in
the package substrate are modeled in a 65nm technology.

Silicon-bridge-based (EMIB-based) architectures have the least CFP for 2- and
4-chiplet-based architectures of the 500mm2 monolith testcase. However, as
$N_{c}$ increases, the number of silicon bridges also increases, and CFP
increases. The RDL-based packages have the least overheads for the 6- and
8-chiplet architectures, but due to their architecture definition, they have
lower communication bandwidth when compared to silicon bridges or interposers.
Therefore, based on the bandwidth requirements of the testcase, such tradeoffs
between performance and CFP can be considered using our analysis tool. The
figure also shows that the passive interposer has lower routing overheads as
the router is part of the chiplet and is in the same technology node of the
chiplet. Therefore, in passive interposer technologies, due to the advanced
node (7nm in this testcase) in which the router is implemented, the area
overheads are smaller when compared to the 65nm router in the active
interposer. The routing overheads of RDL, passive interposer, and silicon-
bridge (EMIB) architectures are small and near-negligible when compared to the
core chiplet areas.

Figure 7: Packaging CFP overheads for different $N_{c}$ values. TABLE II:
Embodied CFP (grams of CO2 emission) for chiplet-based and monolithic versions
of industry testcases.

Carbon | EMR 2-chiplet | EMR 4-chiplet | GA102 | Apple A15 | Tiger Lake
---|---|---|---|---|---
$C_{\text{mfg},\text{chiplet1}}$ | 5.49E+04 | 2.06E+04 | 1.92E+03 | 1.28E+02 | 4.81E+02
$C_{\text{mfg},\text{chiplet2}}$ | 5.49E+04 | 2.06E+04 | 1.40E+03 | 4.17E+02 | 2.44E+02
$C_{\text{mfg},\text{chiplet3}}$ | – | 2.06E+04 | 9.62E+03 | 1.53E+03 | 4.95E+02
$C_{\text{mfg},\text{chiplet4}}$ | – | 2.06E+04 | 9.62E+03 | 1.53E+03 | –
$C_{\text{HI, RDL}}$ | 7.49E+03 | 9.95E+03 | 2.78E+03 | 5.99E+02 | 2.40E+02
$C_{\text{HI, EMIB}}$ | 9.63E+02 | 2.22E+03 | 1.47E+03 | 7.30E+02 | 4.87E+02
$C_{\text{HI, passive}}$ | 9.98E+03 | 1.33E+04 | 3.70E+03 | 7.93E+02 | 3.15E+02
$C_{\text{HI, active}}$ | 1.00E+04 | 1.34E+04 | 3.76E+03 | 8.41E+02 | 3.49E+02
$C_{\text{des}}$ | 1.19E+04 | 1.39E+04 | 4.70E+03 | 1.13E+03 | 2.63E+02
CFPtot (RDL) | 1.29E+05 | 1.06E+05 | 3.00E+04 | 5.34E+03 | 1.72E+03
CFPtot (EMIB) | 1.23E+05 | 9.85E+04 | 2.87E+04 | 5.47E+03 | 1.97E+03
CFPtot (passive) | 1.32E+05 | 1.10E+05 | 3.10E+04 | 5.53E+03 | 1.80E+03
CFPtot (active) | 1.32E+05 | 1.10E+05 | 3.10E+04 | 5.58E+03 | 1.83E+03
Monolithic | 2.55E+05 | 2.91E+05 | 5.58E+04 | 5.60E+03 | 1.96E+03

###  IV-C Summary for different testcases

In Table II, we compare the embodied (design, HI, and manufacturing) CFP of a
HI-based architecture against their monolithic SoC counterparts for different
testcases, different packaging architectures, and a specific configuration of
chiplets. The EMR 2-chiplet represents the architecture of the Intel Emerald
Rapids (EMR) testcase, where both chiplets are in 7nm technology nodes. We
create an EMR 4-chiplet testcase where we further split the two chiplets in
EMR into four based on the Intel Sapphire Rapids architecture [48], all
implemented in a 7nm technology. The table also lists 4-chiplet versions of
NVIDIA GPU GA102, and Apple A15 SoC, where the memory is implemented in 10nm,
IOs/analog chiplets are implemented in 14nm, and the two digital logic
chiplets in 7nm. The Tiger Lake testcase split into three chiplets for memory
(10nm), analog (14nm), and digital logic (7nm).

For the above-described testcases, the table lists $C_{\text{mfg}},i$ for each
chiplet, $C_{\text{HI}}$ for different packaging architectures, the
$C_{\text{des}}$ assuming $N_{\text{des}}=100$ and $N_{p}=200,000$, and the
total CFP for each testcase compared against its monolithic counterpart. We
find that the total CFP of the HI-based testcases are all lower than their
monolithic counterparts. For instance, the EMR 2-chiplet and 4-chiplet
testcases with silicon bridge-based packages have 55% and 70% lower CFP when
compared to their monolithic counterparts. We also observe that HI systems
have a higher benefit with embodied CFP for large SoC testcases when compared
to the smaller ones. For example, the GA102, and EMR testcases have large
monolithic areas of approximately 500mm2 and 1500mm2, respectively, while the
Apple A15 and Tiger Lake testcases have areas of over 100mm2, and therefore
the larger testcases, have a significantly higher improvement in CFP.

##  V Conclusion

In this paper, we proposed HI as a path towards sustainable computing by
designing and manufacturing chiplet-based systems with lower embodied carbon
footprint (CFP) than monolithic SoCs. We have developed an embodied CFP
estimator that uses architectural-level descriptions to assess CFP of
heterogeneous systems. The simulator can be used to guide optimization during
system disaggregation. If this paper is accepted, we will open-source the CFP
simulator for broader community use.

## References

  * [1] C. Freitag, et al., “The real climate and transformative impact of ICT: A critique of estimates, trends, and regulations,” Patterns, vol. 2, no. 9, p. 100340, 2021.
  * [2] B. Graver, et al., “CO2 emissions from commercial aviation,” 2019.  https://theicct.org/publication/co2-emissions-from-commercial-aviation-2018/ (last accessed: March 2023).
  * [3] L. Å. Ragnarsson, et al., “The green transition of the IC industry,” 2022.  https://www.imec-int.com/en/expertise/cmos-advanced/sustainable-semiconductor-technologies-and-systems-ssts/stss-white-paper (last accessed: March 2023).
  * [4] U. Gupta, et al., “ACT: Designing sustainable computer systems with an architectural carbon modeling tool,” in Proceedings of the ACM International Symposium on Computer Architecture, p. 784–799, 2022.
  * [5] M. Garcia Bardon, et al., “DTCO including sustainability: Power-performance-area-cost-environmental score (PPACE) analysis for logic technologies,” in Proceedings of the IEEE International Electron Devices Meeting, pp. 41.4.1–41.4.4, 2020.
  * [6] Apple, “Environmental responsibility report,” 2019.  https://www.apple.com/environment/pdf/Apple_Environmental_Responsibility_Report_2019.pdf.
  * [7] Facebook, “Facebook sustainability data 2019,” 2019.  https://sustainability.fb.com/wp-content/uploads/2020/05/2019-Sustainability-Data-Disclosure_Final-1.pdf.
  * [8] Microsoft, “Environmental sustainability report,” 2021.  https://www.microsoft.com/en-us/corporate-responsibility/sustainability/report.
  * [9] L. Eeckhout, “A first-order model to assess computer architecture sustainability,” IEEE Computer Architecture Letters, vol. 21, no. 2, pp. 137–140, 2022.
  * [10] L. Eeckhout, “Kaya for computer architects: Toward sustainable computer systems,” IEEE Micro, vol. 43, no. 1, pp. 9–18, 2023.
  * [11] U. Gupta, et al., “Chasing carbon: The elusive environmental footprint of computing,” IEEE Micro, vol. 42, p. 37–47, jul 2022.
  * [12] “Kaya identity.”  https://en.wikipedia.org/wiki/Kaya_identity.
  * [13] TSMC, “Corporate social responsibility report,” 2018.  https://esg.tsmc.com/download/file/2018_tsmc_csr_report_published_May_2019/english/pdf/e_all.pdf.
  * [14] C.-H. Kuo, et al., “Life cycle impact assessment of semiconductor packaging technologies with emphasis on ball grid array,” Journal of Cleaner Production, vol. 276, p. 124301, 12 2020.
  * [15] S. Naffziger, et al., “AMD chiplet architecture for high-performance server and desktop products,” in Proceedings of the IEEE International Solid- State Circuits Conference, pp. 44–45, 2020.
  * [16] M. T. Bohr and I. A. Young, “CMOS scaling trends and beyond,” IEEE Micro, vol. 37, no. 6, pp. 20–29, 2017.
  * [17] S. S. Iyer, “Heterogeneous integration for performance and scaling,” IEEE Transactions on Components, Packaging and Manufacturing Technology, vol. 6, no. 7, pp. 973–982, 2016.
  * [18] P. Gelsinger, “Keynote: Semiconductors run the world.”  HotChips, 2022.
  * [19] R. Mahajan, et al., “Embedded multi-die interconnect bridge (EMIB) – a high density, high bandwidth packaging interconnect,” in Proceedings of the IEEE Electronic Components and Technology Conference, pp. 557–565, 2016\.
  * [20] S. Y. Hou, et al., “Wafer-level integration of an advanced logic-memory system through the second-generation CoWoS technology,” IEEE Transactions on Electron Devices, vol. 64, no. 10, pp. 4071–4077, 2017.
  * [21] D. B. Ingerly, et al., “Foveros: 3D integration and the use of face-to-face chip stacking for logic devices,” in Proceedings of the IEEE International Electronic Devices Meeting, pp. 19.6.1–19.6.4, 2019.
  * [22] Y. H. Chen, et al., “Ultra high density SoIC with sub-micron bond pitch,” in Proceedings of the IEEE Electronic Components and Technology Conference, pp. 576–581, 2020.
  * [23] IEEE Components and Packaging Society, “Heterogeneous integration roadmap,” 2021\.  http://eps.ieee.org/hir.
  * [24] J. H. Lau, “Recent advances and trends in advanced packaging,” IEEE Transactions on Components and Packaging Technologies, vol. 12, no. 2, pp. 228–252, 2022.
  * [25] D. B. Ingerly, et al., “Foveros: 3d integration and the use of face-to-face chip stacking for logic devices,” in Proceedings of the IEEE International Electronic Devices Meeting, pp. 19.6.1–19.6.4, 2019.
  * [26] W. Gomes, et al., “Meteor Lake and Arrow Lake Intel next-gen 3D client architecture platform with Foveros,” in Proceedings of the IEEE Hot Chips Symposium, pp. 1–40, Aug. 2022.
  * [27] J. Ferguson, “EDA innovation is the foundation of progress.”  https://www.techdesignforums.com/practice/technique/physical-verification-eda-innovation-is-the-foundation-of-progress/.
  * [28] Angstronomics, “The Truth of TSMC 5nm.”  https://www.angstronomics.com/p/the-truth-of-tsmc-5nm.
  * [29] D. Schor, “IEDM 2022: Did we just witness the death of SRAM?.”  https://fuse.wikichip.org/news/7343/iedm-2022-did-we-just-witness-the-death-of-sram/.
  * [30] J. Cunningham, “The use and evaluation of yield models in integrated circuit manufacturing,” IEEE Transactions on Semiconductor Manufacturing, vol. 3, no. 2, pp. 60–71, 1990.
  * [31] A. Ning, et al., “Supply chain aware computer architecture,” in Proceedings of the ACM International Symposium on Computer Architecture, 2023\.  https://augustning.com/assets/papers/supply-chain-isca-2023-preprint.pdf.
  * [32] D. Stow, et al., “Cost-effective design of scalable high-performance systems using active and passive interposers,” in Proceedings of the IEEE/ACM International Conference on Computer-Aided Design, pp. 728–735, 2017\.
  * [33] T. Li, et al., “Chiplet heterogeneous integration technology—status and challenges,” Electronics, vol. 9, no. 4, 2020.
  * [34] SPIL, “Corporate social responsibility report,” 2018.  https://www.spil.com.tw/Files/pdf-en/2018-en.pdf.
  * [35] Amkor Technology Inc., “Sustainability accounting standards board report,” 2021\.  https://amkor.com/esg/esg-report/.
  * [36] ACE Group, “Corporate sustainability report,” 2017.  https://ase.aseglobal.com/en/csr/downloads/report.
  * [37] R. Mahajan, et al., “Embedded multidie interconnect bridge—a localized, high-density multichip packaging interconnect,” IEEE Transactions on Components, Packaging and Manufacturing Technology, vol. 9, no. 10, pp. 1952–1962, 2019.
  * [38] D. Stow, et al., “Investigation of cost-optimal network-on-chip for passive and active interposer systems,” in Proceedings of the ACM/IEEE International Workshop on System Level Interconnect Prediction (SLIP), pp. 1–8, 2019.
  * [39] C. Sun, et al., “Dsent - a tool connecting emerging photonics with electronics for opto-electronic networks-on-chip modeling,” in 2012 IEEE/ACM Sixth International Symposium on Networks-on-Chip, pp. 201–210, 2012\.
  * [40] S. Y. Hou, et al., “Wafer-level integration of an advanced logic-memory system through the second-generation cowos technology,” IEEE Transactions on Electron Devices, vol. 64, no. 10, pp. 4071–4077, 2017.
  * [41] “List of CPU power dissipation figures.”  https://en.wikipedia.org/wiki/List_of_CPU_power_dissipation_figures.
  * [42] P. Agrawal, et al., “Optimal scheduling and allocation for ic design management and cost reduction,” ACM Transactions on Design Automation of Electronic Systems, vol. 22, jun 2017.
  * [43] P. Alcorn, “Intel details Tiger Lake at Hot Chips 2020, die revealed.”  https://www.tomshardware.com/news/intel-details-tiger-lake-at-hot-chips-2020-die-revealed.
  * [44] D. Patel, et al., “Intel Emerald Rapids backtracks on chiplets – design, performance & cost.”  https://www.semianalysis.com/p/intel-emerald-rapids-backtracks-on.
  * [45] Z. Liu, “IR photographer shares die shots of Nvidia 3000 series GA102 silicon.”  https://www.tomshardware.com/news/infrared-photographer-photos-nvidia-ga102-ampere-silicon.
  * [46] Wikipedia, “Apple a15 processor.”  https://en.wikipedia.org/wiki/Apple_A15.
  * [47] D. Patel, “Apple A15 die shot and annotation – IP block area analysis.”  https://www.semianalysis.com/p/apple-a15-die-shot-and-annotation.
  * [48] Wikipedia, “Saphire rapids.”  https://en.wikipedia.org/wiki/Sapphire_Rapids.


basicstyle=, lineskip=-0.2ex, showspaces=false basicstyle=, lineskip=-0.2ex,
showspaces=false

# MARS: Middleware for Adaptive Reflective Computer Systems

Tiago Mück Bryan Donyanavard  Biswadip Maity Kasra Moazzemi Nikil Dutt
Department of Computer Science
University of California, Irvine, USA
<EMAIL_ADDRESS>
###### Abstract

Self-adaptive approaches for runtime resource management of manycore computing
platforms often require a runtime model of the system that represents the
software organization or the architecture of the target platform. The
increasing heterogeneity in a platform’s resource types and the interactions
between resources pose challenges for coordinated model-based decision making
in the face of dynamic workloads. Self-awareness properties address these
challenges for emerging heterogeneous manycore processing (HMP) platforms
through reflective resource managers. However, with HMP computing platform
architectures evolving rapidly, porting the self-aware decision logic across
different hardware platforms is challenging, requiring resource managers to
update their models and platform-specific interfaces. We propose MARS
(Middleware for Adaptive and Reflective Systems), a cross-layer and multi-
platform framework that allows users to easily create resource managers by
composing system models and resource management policies in a flexible and
coordinated manner. MARS consists of a generic user-level sensing/actuation
interface that allows for portable policy design, and a reflective system
model used to coordinate multiple policies. We demonstrate MARS’ interaction
across multiple layers of the system stack through a dynamic voltage and
frequency scaling (DVFS) policy example which can run on any Linux-based HMP
computing platform.

_Keywords_ Artifact $\cdot$ Framework $\cdot$ Reflection $\cdot$ Adaptive
$\cdot$ Middleware $\cdot$ Runtime resource management

##  1 Introduction

At the processing platform level, heterogeneous processing elements have
become increasingly popular as a way to exploit the power-performance tradeoff
for dynamic workloads. This growth in popularity can be observed across
multiple domains, ranging from portable battery-powered devices to
supercomputers. Emerging mobile battery-powered Systems-on-Chip (SoCs)
incorporate heterogeneity in order to provide energy-efficiency. For instance,
ARM’s _big.LITTLE_ architecture [1] is used on Samsung’s Exynos [2] and
Nvidia’s Tegra SoCs [3]. big.LITTLE employs _single-ISA heterogeneity_ , in
which cores of the same instruction-set architecture (ISA) are deployed with
different configurations and/or microarchitectures (for example, in-order
cores operating at low frequency vs. out-of-order cores operating at high
frequency).

Effective exploitation of power-performance tradeoffs in heterogeneous
manycore processors (HMPs) requires intelligent management both within and
across different layers, in particular the operating system [4]. Operating
systems must continuously analyze application behavior to resolve tradeoffs:
e.g., _What is the most power efficient core type to execute the application
without violating its performance requirements?_ or _Which option is more
power-efficient for the current application: an out-of-order core at a lower
frequency or an in-order core at a higher frequency?_

Being able to properly assess the performance/power impact of resource
management decisions plays a major role in efficiently managing the system,
especially on HMPs in which different resources (e.g., CPU cores, memory
buses, GPU cores, etc.) have distinct performance/power tradeoffs. Some
existing Linux extensions address these issues to a limited extent. For
instance, Global Task Scheduling (GTS) [1] migrates tasks between high-
performance and low-power core types when the task load reaches a certain
threshold. GTS-based policies have been implemented on Linux and deployed by
multiple SoC vendors that support the big.LITTLE technology (e.g.,
Linaro’s/ARM’s big.LITTLE MP implementation used by Samsung Exynos [5],
MediaTek’s CorePilot [6], Qualcomm’s Energy Aware Scheduling [7]). These
policies, however, are customized for specific heterogeneous platforms without
being adaptable to other platforms. Furthermore, they exclusively focus on
task mapping/scheduling decisions and still do not provide any coordination
with the underlying power management subsystems.

State-of-the art works address these issues by applying principles of self-
awareness. Self-aware computing systems are computing systems that capture
knowledge about themselves and their environment on an ongoing basis to learn
models, and reason about the models in order to make decisions in accordance
with high-level goals, which are subject to change [8]. The notion of self-
awareness has been used in computing in a variety of different domains such as
autonomic computing, organic computing, adaptive systems, and self-organizing
systems [9]. Although adaptive systems with no self-awareness exist, it has
been argued that a sophisticated self-model is a prerequisite for sensible
adaptive behavior when the environment and the system itself are sufficiently
complex [10].

Some apply self-reflection by using models that are able to estimate or
predict workload behavior by extracting performance metrics at runtime [11,
12, 13]. While a significant amount of work has been done on the management of
specific resources (e.g. DVFS[14, 15, 16], task mapping[17, 18, 19], memory
allocation[20, 21, 22]), most do not provide means for coordinating the
management of different resource types.

Self-adaptive software systems and applications have been successfully
deployed [23, 24, 25, 26], for example to extract information from networks of
cameras [27]. Hoffman et al. developed the SEEC framework [28] for self-aware
_applications_. Leech et al. developed the PRiME Framework [29] to provide
support for testing resource managers across platforms. Our goal is to
similarly provide a framework for _policy_ designers to design adaptive
resource managers that support properties of self-awareness.

In this artifact paper, we present MARS (Middleware for Adaptive and
Reflective Systems) [30], a cross-layer and multi-platform framework
supporting the creation of resource managers for emerging heterogeneous
manycore processing (HMP) platforms by composing system models and resource
management policies in a flexible and coordinated manner. MARS has been
utilized in different configurations for research projects such as SPARTA
[31], SPECTR [32], and deployed on multiple hardware platforms. Users define
policies using generic interfaces so that they are portable across any
supported hardware platform. The modular composition makes the framework
easily expandable to support new platforms, sensors, or actuators. MARS
interacts with all layers of the system stack to orchestrate the management of
resources. MARS is mainly composed of four parts:

  1. 1.

Sensors and actuators. The sensed data that consists of performance counters
(e.g. instructions executed, cache misses, etc.) and other sensory information
(e.g. power, temperature, etc.) collected to assess the current system state
and to characterize workloads. The configurable knobs that allow the platform
to adjust its configuration to optimize operating point or control tradeoffs.

  2. 2.

Resource management policies implemented by MARS’s users.

  3. 3.

Reflective system model used by the policies to make informed decisions. The
reflective model has the following subcomponents:

    1. (a)

Models of _policies implemented by the underlying OS kernel_ used for
coordinating decisions made within MARS with decisions made by the OS.

    2. (b)

Models of _user policies_ , that are automatically instantiated from any
policy defined within MARS.

    3. (c)

The baseline _performance/power model_. This model takes as input the
predicted actuations generated from the policy models and produces predicted
sensed data.

  4. 4.

The policy manager is responsible for reconfiguring the system by adding,
removing, or swapping policies to better achieve the current system goal.

##  2 Background: Runtime Models, Reflection and Prediction

Self-adaptive software can be defined as “software that evaluates its own
behavior and changes behavior when the evaluation indicates that it is not
accomplishing what the software is intended to do, or when better
functionality or performance is possible”[33]. In this work, we do not address
self-adaptive systems directly, as these systems encompass many high-level
concepts such as self-configuration, self-healing, self-awareness, self-
optimization, and others, also referred to as _self- $*$_[34] (see [35] for a
comprehensive review self-adaptivity for SoCs). However, an infrastructure for
system introspection and reflective behavior is an important building block
for such systems.

ControllerDecidePredictSystemActTargetTargetObserveReflection Figure 1:
Feedback loop overview. The bottom part of the figure represents a simple
observe-decide-act loop. The top part (in blue) adds the reflection mechanism
to this loop, enabling predictions for smart decision making. Figure 2: MARS
framework overview. Different layers of the system stack coordinate through
policies to orchestrate the management of resources: sensors inform policies
of the system state; policies coordinate with models to perform reflective
queries, and make resource management decisions; policies set actuators to
enact changes on the system. Details about sensors and actuators in Table 1.

###  2.1 Observe, Decide, and Act, plus Reflection

Closed-loop systems have been used extensively to improve the state of a
system by tracking target references. Closed-loop systems traditionally deploy
an _Observe_ , _Decide_ and _Act_ (ODA) feedback loop (lower half (in black)
of Figure 1) to determine the system configuration. In an ODA loop, the
observed behavior of the system is compared to the target behavior, and the
discrepancy is fed to the controller for decision making. The controller
invokes actions based on the result of the _Decide_ stage. MARS enables an
additional _Reflection_ loop (upper half (in blue) of Figure 1) that
incorporates the ability to include past history, as well as predictions, to
make smart decisions and enhance adaptivity.

Reflection is a key property of self-awareness. Reflection enables ODA
decisions to be made based on both _past_ observations, as well as predictions
made from past observations. MARS ODA with reflection and prediction requires
a self-model of the subsystem(s) under control, as well as models of other
policies that may impact the decision-making process. MARS predictions
consider _future_ actions, or events that may occur before the next iteration
of the ODA loop, enabling "what-if" exploration of alternatives. Such actions
may be triggered by other resource managers running with a shorter period than
the ODA loop. The top half of Figure 1 (in blue) shows prediction enabled
through reflection that can be utilized in the decision making process, as
supported by MARS. The main goal of the prediction model is to estimate system
behavior based on potential actuation decisions. This type of prediction is
most often performed using linear regression-based models [36, 37] due to
their simplicity, while others employ a binning-based approach in which
metrics sensed at runtime are used to classify workloads into categories whose
performance/power are known for all core types[38, 31].

##  3 MARS Framework

Figure 2 shows an overview of the MARS framework (shaded), with _Sensors_ and
_Actuators_ interfacing across multiple layers of the system stack:
_Applications_ , _Linux kernel_ , and _HMP HW Platform_. On-chip resource
management requires modeling and runtime policies for these different layers
of the system stack. Resource management decisions must be made for each
layer, and the frequency at which decisions are made may vary within or
between layers. Coordinating multiple resource management policies is
challenging, and that challenge compounds when developers are required to
design portable policies for multiple platforms. We have developed MARS –
_Middleware for Adaptive and Reflective Systems_ – to address these
challenges.

###  3.1 Interfaces and Policy Design

We first look into how MARS provides a generic user-level sensing/actuation
interface that allows for portable policy design. Table 1 provides a list of
common sensors and actuators available at each level of the system stack.

####  3.1.1 Sensors

Sensor data may consist of physical or virtual performance counters (e.g.,
instructions executed, cache hits), or other sensory information (e.g., power,
temperature). Sensor data assesses the current state of the system and also
characterizes the workload. All sensors expose a _virtual interface_ for
communication with the resource manager. To implement the interface, MARS
utilizes function templates to provide a generic sense function that returns
the specified sensed metric. Different versions of the function can be
implemented for each platform, resource, and sensor-type combination. The
function’s template parameter specifies the proper function implementation to
obtain the requested sensed data. This separates the platform-specific
implementation from the user-level policies.

Level |  Actuators (A) |  Sensors (S)
---|---|---
Application |  Degree of parallelism, algorithmic choice |  Performance, Quality-of-Service
Kernel |  Task-to-core mapping, task priority, memory allocation |  CPU time, utilization, context switch counters, open file, number of tasks
Hardware |  Voltage/frequency scaling, clock gating, power gating, cache sizing |  Performance counters (e.g., cache misses, instructions executed, branch miss-predictions, etc), power, reliability, critical path delay degradation
Table 1: Examples of sensors and actuators available across the system stack

####  3.1.2 Actuators

Adjustments to system configuration at runtime happens through actuators.
Actuators range from application-level choices (e.g., degree of parallelism),
to kernel-level choices (e.g., task-to-core mapping), to hardware-level
choices (e.g., core operating frequency). The actuate function sets a new
value for a system actuator using a _virtual interface_. Similarly to sense,
actuate is implemented with function templates so that unique versions may be
provided for each platform, resource, and actuator combination. Given the new
actuation value, the function’s template parameter selects the proper function
implementation to perform the requested actuation action. Two additional
related functions are provided: actuationVal returns the current actuation
value set and actuationRanges returns the valid range of actuation values for
the specified resource. actuate functions are implemented on top of standard
Linux system calls and modules. For instance, we use CPU affinity system calls
to control task-to-core mappings, and _cpufreq_ to control core frequencies.

####  3.1.3 Policies

Policies contain the decision-making logic which controls the system resources
at runtime. Management policies implement the ODA and refelction loops for the
resource(s) under control. In MARS, users can define resource management
policies by creating a subclass of Policy.

All policies make use of generic sensing and actuation interfaces sense and
actuate. At the user-level, all policies share access to the common set of
sensors. For each policy, sensor data is aggregated during the sensing window
(i.e., the time period between policy invocations) defined for the policy.
MARS uses ioctl system calls to setup the sensing windows for each registered
policy function. The policy functions are executed at the end of each sensing
window in separate threads that uses blocking ioctl calls to synchronize with
the kernel module. The kernel module automatically aggregates sensed data on a
per-window basis and stores the information in shared memory, so that it can
be read directly by sense calls with low overhead. This means any number of
sensing windows (i.e., policies) executing at different window lengths can
access the same sensors concurrently, and still measure the appropriate value
for their window length.

###  3.2 Reflective System Model

Previously, we described a scenario where multiple policies operate at
different sensing window lengths, or epochs. We now illustrate how MARS uses a
reflective system model to make self-aware decisions.

To understand the reflection mechanism in MARS, consider the resource manager
shown in Figure 3 that shows a sample task mapping policy interacting with
DVFS (Dynamic Voltage and Frequency Scaling) and scheduler policy models. At
the finest time granularity, we have the operating system scheduler, whose
goal is to select a task to execute on a given core. A new decision must be
made whenever a new task is created, a tasks finishes, a task’s quantum
expires, an interrupt is raised, etc., leading to a time-frame between
decisions on the order of microseconds. At a coarser time granularity we have
the DVFS policy, which typically executes periodically (10-100 milliseconds)
to analyze the system load and select an appropriate operating frequency. At
the coarsest time granularity (100-1000 of milliseconds), the task mapping
policy runs periodically to define a new task-to-core assignment. Migrating a
task from one core to another has significantly more overhead than changing
the CPU frequency in a typical HMP [39]. MARS allows users to coordinate among
different policies through policy models, regardless of varying time
granularity.

Task MappingPolicyDVFSModelSchedulerModelHWModel What if a task is migrated? 1
What if the frequency changes? 2 What if the schedule changes? 3 \- Perf/Power
\- Load \- Frequency \- Perf/Power \- Load \- Perf/Power Figure 3: Example of
a task mapping policy that queries models of OS policies for DVFS and
scheduling.

In order to make an informed task mapping decision, for instance, the policy
must consider the effects of its decision on the behavior of the underlying
DVFS and scheduling policies. Furthermore, the invocation period of actuations
dictates how complex the decision making logic can be. For instance, a
scheduling decision must be made in the sub-microsecond range in order not to
disrupt the system. Task-to-core mapping, on the other hand, is done
comparatively infrequently, and affects the system performance over a long
timespan. Therefore, the overhead of using complex models to make such
decisions can be mitigated by the potential benefits of an informed decision.

The components within the reflective system model interact in a hierarchy
defined by the dependencies of the actuations performed in the system. For
instance, Figure 3 illustrates the scenario within MARS for our example.
Workload models assume each core can run multiple tasks and there is no formal
or explicit dependency between threads. Before the task mapping policy decides
to migrate a task, it (1) queries the reflective model asking: _what will be
the performance of task A if it is migrated?_ (2) The DVFS governor policy
model executes (_without_ actuating) in order to predict the resulting core
frequency provided the hypothetical task mapping. (3) This information is
passed on to the performance/power model which predicts the task performance.
Architecture models define the architectural characteristics of the target
platform including _instruction-set architecture_ (ISA), number of cores, core
types, etc. Finally, the predicted metrics are used by the policy to make the
decision, which is passed to the actuator through the actuation interface.

Analogous to the actuate/sense functions described previously, MARS provides
the tryActuate and senseIf set of functions necessary issuing queries to the
reflective models:

  * •

tryActuate: Using the same syntax as actuate, this function updates the
underlying models used to predict the next system state given the new
actuation value. It does not set the actual actuation value. A tryActuationVal
analogous to actuationVal is also provided.

  * •

senseIf: this function has the same semantics as sense, but returns predicted
sensed information for the next sensing window, given a new actuation set by
tryActuate.

###  3.3 Policy Manager

A user-defined subclass of PolicyManager is responsible for composing policies
and models together. The purpose of the policy manager is to provide resource
management autonomy in response to changing system goals or execution
scenarios, as well as coordinate multiple dependent policies and their
objectives. For instance, consider modern smartphones. These devices typically
operate in two scenarios: 1) the device is plugged to an external power
source; 2) the device is powered by battery. In the case of (1), policies can
simply focus on maximizing applications’ quality of service (QoS), while for
(2), QoS should be balanced with energy efficiency. A third scenario in which
the battery charge is critically low is also possible. In this case, policies
should focus on minimizing power consumption so the device can continue to
operate. Furthermore, direct intervention from the user may cause additional
unpredictable scenario changes. Creating a single policy that is able to
manage all these scenarios and goals can lead to a overly complex and possibly
inefficient implementation. Instead, one may develop multiple policies
optimized for specific cases, and have a high-level manager dynamically select
the most appropriate policy to apply throughout runtime.

Table 2:  Currently supported platforms in MARS and their sensors/actuators  Implementation | Platform | Architecture | Sensors | Actuators
---|---|---|---|---
ODROID |  Hardware |  ARMv7-A |  All ARM Performance Monitoring Unit (PMU) counters (e.g., cache misses, instructions executed, branch misspredictions, etc.,), Power sensors for CPU clusters |  Voltage/frequency scaling for CPU, Task-to-core mapping, Temperature sensor
NVIDIA Jetson-TX2 |  Hardware |  ARMv8-A |  ARM PMU counters, GPU performance counters, CPU Power, Memory Power, GPU Power, WiFi Power |  Voltage/frequency scaling for CPU clusters, GPU frequency scaling, Task-to-core mapping mapping
Gem5 |  Simulation |  $*$ |  ARM PMU counters |  Frequency scaling
Offline |  |  |  |
simulator |  Simulation |  $*$ |  Sensors which are present in the trace |  Switch across different traces of execution

###  3.4 DVFS Policy Example

We first explain dynamic voltage and frequency scaling (DVFS) through a
smartphone use case. When running multimedia/gaming applications,
computational resource requirements are very high. To improve performance, CPU
core frequency is increased dynamically, also resulting in higher power
consumption. When the demand for computational resources is lowered, the CPU
core frequency is reduced to save power. Using the MARS framework, we
demonstrate a simple use case of a core DVFS policy.

⬇

1class Simple_DVFS_Policy : public Policy {

2 const int PERIOD_MS = 50;

3

4 Simple_DVFS_Policy() :Policy(PERIOD_MS) {}

5

6 double try_frequency(

7 frequency_domain &fd,

8 double freq_mhz) {

9 tryActuate<ACT_FREQ_MHZ>(fd,freq_mhz);

10 double ips = senseIf<SEN_INSTR_TOTAL>(fd)

11 / senseIf<SEN_TIME_TOTAL_S>(fd);

12 double power = senseIf<SEN_POWER_W>(fd);

13 return ips / power;

14 }

15

16 void execute(){

17 for(auto fd : sys_info().freq_domains)

18 {

19 double best_freq = fd.freq_min;

20 double best_eff = 0;

21 auto ranges = actuationRanges<ACT_FREQ_MHZ>(fd);

22

23 for(auto freq = ranges.min;

24 freq <= ranges.max; freq += ranges.steps)

25 {

26 double freq_eff = try_frequency(fd,freq);

27 if(freq_eff > best_eff){

28 best_freq = freq;

29 best_eff = freq_eff;

30 }

31 }

32 actuate<ACT_FREQ_MHZ>(fd,best_freq);

33 }

34 }

35};

36

37class Simple_DVFS_Manager : public Policy_Manager {

38 void setup() {

39 registerPolicy(new Simple_DVFS_Policy());

40 }

41};

Figure 4: Reflective DVFS management policy.

Our Simple_DVFS_Policy implemented in MARS is shown in Figure 4. First, the
Simple_DVFS_Policy is registered by the Simple_DVFS_Manager (PolicyManager)
during setup using the registerPolicy function (lines 37-40). At this point
user-defined models (if any) are also registered (using the registerModel
function). The DVFS policy is defined by the Simple_DVFS_Policy.
Simple_DVFS_Policy inherits from Policy and implements its decision-making
logic in the execute function (lines 16-35). When inheriting from Policy one
must also define the policy invocation period (line 4). This period defines
how often the policy is invoked as well as the hierarchy of the policies
within the reflective model. The period establishes a _sensing window_ for the
policy. In this example, when the execute function is invoked, the sensed
information (e.g., performance counters, power sensors, etc.) aggregated over
the latest $50ms$ sensing window can be obtained.

Within the execute function, a special sys_info object provided by the
framework is used to access available resources in the system. The sys_info
also provides information about the topology and relationship between
resources. Then, for all supported frequencies, the try_frequency (Lines 6-14)
function is called to assess the energy efficiency of the given frequency and
select the best setting.

Finally, the developer may use the same infrastructure to model the aspects
implemented within the underlying system. The registerModel function is
provided to facilitate this process. When registered using registerModel, the
policy code will only be executed as part of the model to predict performance
and power, but it will not directly actuate on the system.

The above exemplar represents a naive DVFS governor for an HMP. More
sophisticated resource managers have been successfully deployed using MARS:
[16] deploys a gain-scheduled DVFS governor for dynamic power management;
SPARTA [31] expands predictive power/performance model to perform runtime task
allocation in HMPs; SPECTR [32] is a hierarchical manager that coordinates an
adaptive per-core clock gating coordinated with per-cluster DVFS for HMPs;
[40] uses the offline-simulator to perform a design space exploration of
hybrid on-chip memory configuration for HMPs; HESSLE-FREE [41] uses MARS for
resource management by leveraging fuzzy control techniques. MARS has also
successfully been deployed for runtime memory management techniques [42, 43].

##  4 State of Implementation

MARS is implemented in the C++ language following an object-oriented paradigm.
The framework is open source and available online.111Code repository at
https://github.com/duttresearchgroup/MARS. In order to easily compile runtime
resource management policies using MARS framework, a docker-based container
image222Docker image at https://hub.docker.com/r/duttresearchgroup/mars. is
also available for developers.

###  4.1 Platforms

As highlighted in Table 2, MARS has been designed to work in conjunction with
the Linux operating system and has been implemented on both hardware and
simulation platforms. This enables policy evaluation on hypothetical system
configurations [44] in advance of policy deployment on a common hardware
platform. We provide brief descriptions of these platforms below.

####  4.1.1 ODROID

Odroid-XU3[45] features a Samsung Exynos5422 HMP, which implements a 8-core
big.LITTLE-based HMP. Cores are organized into clusters of the same type (i.e.
a 4-core “big” cluster and a 4-core “little” cluster). Cores in the same
cluster share the the same L2 cache and the same frequency domain. Odroid-XU3
also features per-cluster power sensors and temperature sensors for the GPU
and for each core in the big cluster.

####  4.1.2 NVIDIA Jetson-TX2

This NVIDIA Jetson-TX2 has a hex-core ARMv8 CPU (quad-core ARM Cortex-A57
alongside Denver cores) and a 256-core NVIDIA Pascal GPU. The high performance
design and extensive support for AI and neural network application makes this
board a great candidate to expand MARS to support new software domains.

####  4.1.3 gem5

The gem5 simulator[46] is a modular platform for computer architecture
research, encompassing system-level architecture as well as processor
microarchitecture. We use gem5 to create heterogeneous cores by changing the
architectural simulation parameters. This allows us to evaluate MARS on HMP
platforms with more than two core types. For obtaining power data, we
integrate the McPAT power model[47] directly with the gem5 simulation
framework, which allows us to obtain power data online and reduces simulation
overhead. We use gem5 in full system mode which a complete operating system
boots on top of gem5 and system calls are executed natively while supporting
multithreaded workloads.

####  4.1.4 Offline simulator

Debugging and evaluating intelligent management policies on cutting-edge
platforms is not straightforward. Issues that are typically straightforward to
fix in a prototype running as a user-space process can be highly problematic
when they occur, for instance, within a module running in kernel-space.
Evaluating the new policy in terms of scalability and generality becomes much
harder, since real platforms have fixed configurations and full system
simulations of large-scale systems are often unfeasible due to extremely long
simulation times. For such a scenario, we include a _trace-based offline
simulator_ platform that supports MARS abstractions. With the offline
simulator and the Linux-based implementation, we can close the gap between
proposing a new policy and implementing it on a real system. [44] describes
the offline simulator in more details.

##  5 Conclusion and Future Work

In this artifact paper, we presented MARS: a cross-layer, and multi-platform
framework that supports coordinated self-adaptive resource management
policies. Using MARS, policies are able to interact with one-another and make
model-based decisions to adapt to dynamic workloads. The MARS framework is
open source and downloadable as a container-based docker image [30]. While the
current version of MARS targets energy-efficient heterogeneous multiprocessor
(HMP) systems, we believe the MARS framework can be ported to a wider range of
systems (e.g., webservers, high performing clusters), which opens up several
directions for future work. As we continue to explore interaction among
multiple policies, it has not escaped our attention that MARS currently does
not have any validation mechanism for the emergent behavior of these policies
acting together. In our future work, we plan to expand MARS to run on more
complicated systems and develop a runtime validation mechanism for coordinated
policies.

## Acknowledgment

We acknowledge financial support from the following: National Science
Foundation Grant CCF-1704859. We would also like to acknowledge the critical
review by Saehanseul Yi on the manuscript.

## References

  * [1] ARM.  big. LITTLE Technology : The Future of Mobile.  Technical report, ARM, 2013.
  * [2] Ltd. Samsung Electronics Co.  Samsung Exynos, 2016.
  * [3] NVidia.  Variable SMP – A Multi-Core CPU Architecture for Low Power and High Performance.  Technical report, NVidia, 2011.
  * [4] Amir M. Rahmani, Pasi Liljeberg, Ahmed Hemani, Axel Jantsch, and Hannu Tenhunen.  The Dark Side of Silicon.  Springer, 2017.
  * [5] Mathieu Poirier.  LCA14-104: GTS- A solution to support ARM’s big.LITTLE technology.  In Linaro Connect USA, 2014.
  * [6] MediaTek.  MediaTek CorePilot, Heterogeneous Multi-Processing Technology.  http://cdn-cw.mediatek.com/WhitePapers/MediaTek_CorePilot.pdf, 2013.
  * [7] Stephen Muckle.  LCU14-406: A QuIC Take on Energy-Aware Scheduling.  In Linaro Connect USA, 2014.
  * [8] Samuel Kounev, Peter Lewis, Kirstie L. Bellman, Nelly Bencomo, Javier Camara, Ada Diaconescu, Lukas Esterle, Kurt Geihs, Holger Giese, Sebastian Götz, Paola Inverardi, Jeffrey O. Kephart, and Andrea Zisman.  The Notion of Self-aware Computing, pages 3–16.  Springer International Publishing, 2017.
  * [9] Axel Jantsch, Nikil Dutt, and Amir M Rahmani.  Self-awareness in systems on chip– a survey.  IEEE Design & Test, 34(6), 2017.
  * [10] Mazeiar Salehie and Ladan Tahvildari.  Autonomic computing: Emerging trends and open problems.  SIGSOFT Softw. Eng. Notes, 30(4), 2005.
  * [11] Thannirmalai Somu Muthukaruppan, Anuj Pathania, and Tulika Mitra.  Price theory based power management for heterogeneous multi-cores.  In ASPLOS, New York, New York, USA, 2014. ACM Press.
  * [12] Kenzo Van Craeynest, Aamer Jaleel, Lieven Eeckhout, Paolo Narvaez, and Joel Emer.  Scheduling heterogeneous multi-cores through performance impact estimation (PIE).  In 2012 39th Annual International Symposium on Computer Architecture (ISCA), volume 40. IEEE, jun 2012.
  * [13] S Sarma, N Dutt, P Gupta, N Venkatasubramanian, and A Nicolau.  CyberPhysical-System-On-Chip ( CPSoC ): A Self-Aware MPSoC Paradigm with Cross-Layer Virtual Sensing and Actuation.  In Design, Automation & Test in Europe Conference & Exhibition (DATE), 2015.
  * [14] Hamid Nejatollahi and Mostafa E. Salehi.  Voltage scaling and dark silicon in symmetric multicore processors.  The Journal of Supercomputing, 2015.
  * [15] Rishad A. Shafik, Sheng Yang, Anup Das, Luis A. Maeda-Nunez, Geoff V. Merrett, and Bashir M. Al-Hashimi.  Learning Transfer-Based Adaptive Energy Minimization in Embedded Systems.  IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 35(6), 2016.
  * [16] Bryan Donyanavard, Amir M. Rahmani, Tiago Mück, Kasra Moazemmi, and Nikil D. Dutt.  Gain scheduled control for nonlinear power management in cmps.  In 2018 Design, Automation & Test in Europe Conference & Exhibition, DATE 2018, Dresden, Germany, March 19-23, 2018, 2018.
  * [17] Francesco Paterna, Andrea Acquaviva, and Luca Benini.  Aging-Aware Energy-Efficient Workload Allocation for Mobile Multimedia Platforms.  IEEE Transactions on Parallel and Distributed Systems, 24(8), 2013\.
  * [18] Siva Satyendra Sahoo, Akash Kumar, and Bharadwaj Veeravalli.  Design and Evaluation of Reliability-oriented Task Re-Mapping in MPSoCs using Time-Series Analysis of Intermittent faults.  In 2016 Design, Automation & Test in Europe Conference & Exhibition (DATE), 2016.
  * [19] E M G Trainiti, G C Durelli, A Miele, C Bolchini, and M D Santambrogio.  A Self-Adaptive Approach to Efficiently Manage Energy and Performance in Tomorrow ’ s Heterogeneous Computing Systems.  In DATE, Dresden, 2016.
  * [20] Tiago Rogerio Mück and Antonio Augusto Frohlich.  Run-time scratch-pad memory management for embedded systems.  In IECON 2011 - 37th Annual Conference of the IEEE Industrial Electronics Society. IEEE, nov 2011.
  * [21] Majid Shoushtari, Abbas BanaiyanMofrad, and Nikil Dutt.  Exploiting Partially-Forgetful Memories for Approximate Computing.  IEEE Embedded Systems Letters, 7(1), 2015.
  * [22] Majid Shoushtari, Bryan Donyanavard, Luis Angel D. Bathen, and Nikil Dutt.  ShaVe-ICE: Sharing Distributed Virtualized SPMs in Many-Core Embedded Systems.  ACM Transactions on Embedded Computing Systems, 2018.
  * [23] S. Hallsteinsen, K. Geihs, N. Paspallis, F. Eliassen, G. Horn, J. Lorenzo, A. Mamelli, and G.A. Papadopoulos.  A development framework and methodology for self-adapting applications in ubiquitous computing environments.  Journal of Systems and Software, 85(12), 2012.
  * [24] Kurt Geihs, Roland Reichle, Michael Wagner, and Mohammad Ullah Khan.  Modeling of Context-Aware Self-Adaptive Applications in Ubiquitous and Service-Oriented Environments, pages 146–163.  Springer Berlin Heidelberg, 2009.
  * [25] Hassan Gomaa and Mohamed Hussein.  Dynamic software reconfiguration in software product families.  In Frank J. van der Linden, editor, Software Product-Family Engineering. Springer Berlin Heidelberg, 2004.
  * [26] Brice Morin, Olivier Barais, Gregory Nain, and Jean-Marc Jezequel.  Taming dynamically adaptive systems using models and aspects.  In Proceedings of the 31st International Conference on Software Engineering, ICSE ’09, Washington, DC, USA, 2009. IEEE Computer Society.
  * [27] L. Gurgen, O. Gunalp, Y. Benazzouz, and M. Gallissot.  Self-aware cyber-physical systems and applications in smart buildings and cities.  In DATE, 2013.
  * [28] Henry Hoffmann, Martina Maggio, Marco D Santambrogio, Alberto Leva, and Anant Agarwal.  Seec: A framework for self-aware computing.  Technical Report TR-2010-049, CSAIL, MIT, 2010.
  * [29] Charles Leech, Graeme M. Bragg, Domenico Balsamo, and Eduardo Wachter.  Application control and monitoring in heterogeneous multiprocessor systems.  In 2018 13th International Symposium on Reconfigurable Communication-centric Systems-on-Chip (ReCoSoC), pages 1–8, 2018.
  * [30] DuttResearchGroup.  Middleware for Adaptive Reflective Systems (MARS) framework.  https://github.com/duttresearchgroup/MARS, 2018.
  * [31] Bryan Donyanavard, Tiago Mück, Santanu Sarma, and Nikil Dutt.  SPARTA: Runtime Task Allocation for Energy Efficient Heterogeneous Many-cores.  In Proceedings of the Eleventh IEEE/ACM/IFIP International Conference on Hardware/Software Codesign and System Synthesis - CODES ’16, New York, New York, USA, 2016. ACM Press.
  * [32] Amir M. Rahmani, Bryan Donyanavard, Tiago Mück, Kasra Moazzemi, Axel Jantsch, Onur Mutlu, and Nikil Dutt.  SPECTR: Formal Supervisory Control and Coordination for Many-core Systems Resource Management.  In Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems - ASPLOS ’18, New York, New York, USA, 2018. ACM Press.
  * [33] Robert Laddaga.  Active Software.  In Proceedings of the First International Workshop on Self-adaptive Software, IWSAS’ 2000, Secaucus, NJ, USA, 2000. Springer-Verlag New York, Inc.
  * [34] Mazeiar Salehie and Ladan Tahvildari.  Self-adaptive software: Landscape and Research Challenges.  ACM Transactions on Autonomous and Adaptive Systems, 4, 2009.
  * [35] Nikil Dutt, Axel Jantsch, and Santanu Sarma.  Toward Smart Embedded Systems: A Self-aware System-on-Chip (SoC) Perspective.  ACM Transactions on Embedded Computing Systems, 15, 2016.
  * [36] Tiago Mück, Santanu Sarma, and Nikil Dutt.  Run-DMC: Runtime dynamic heterogeneous multicore performance and power estimation for energy efficiency.  In 2015 International Conference on Hardware/Software Codesign and System Synthesis (CODES+ISSS). IEEE, 2015.
  * [37] Mihai Pricopi, Thannirmalai Somu Muthukaruppan, Vanchinathan Venkataramani, Tulika Mitra, and Sanjay Vishin.  Power-performance modeling on asymmetric multi-cores.  In 2013 International Conference on Compilers, Architecture and Synthesis for Embedded Systems (CASES). IEEE, 2013.
  * [38] Guangshuo Liu, Jinpyo Park, and Diana Marculescu.  Dynamic thread mapping for high-performance, power-efficient heterogeneous many-core systems.  In ICCD. IEEE, 2013.
  * [39] Kisoo Yu.  big.LITTLE Switchers.  In 2012 Korea Linux Forum, 2012.
  * [40] Bryan Donyanavard, Amir Mahdi Hosseini Monazzah, Tiago Mück, and Nikil Dutt.  Exploring fast and slow memories in HMP core types.  In Proceedings of the Twelfth IEEE/ACM/IFIP International Conference on Hardware/Software Codesign and System Synthesis Companion - CODES ’17, 2017.
  * [41] Kasra Moazzemi, Biswadip Maity, Saehanseul Yi, Amir M. Rahmani, and Nikil Dutt.  HESSLE-FREE: Heterogeneous Systems Leveraging Fuzzy Control for Runtime Resource Management.  ACM TECS, 2019.
  * [42] Biswadip Maity, Bryan Donyanavard, and Nikil Dutt.  Self-aware memory management for emerging energy-efficient architectures.  In 2020 11th International Green and Sustainable Computing Workshops (IGSC), pages 1–8, 2020.
  * [43] Biswadip Maity, Bryan Donyanavard, Nalini Venkatasubramanian, and Nikil Dutt.  Workload Characterization for Memory Management in Emerging Embedded Platforms.  In Proc. IESS, 2019.
  * [44] Tiago Mück, Bryan Donyanavard, and Nikil Dutt.  PoIiCym: Rapid prototyping of resource management policies for HMPs.  In Proceedings of the 28th International Symposium on Rapid System Prototyping Shortening the Path from Specification to Prototype - RSP ’17, New York, New York, USA, 2017. ACM Press.
  * [45] Hardkernel.  ODROID-XU.  http://www.hardkernel.com/main/main.php, 2016.
  * [46] Nathan Binkert, Somayeh Sardashti, Rathijit Sen, Korey Sewell, Muhammad Shoaib, Nilay Vaish, Mark D Hill, David A Wood, Bradford Beckmann, Gabriel Black, Steven K Reinhardt, Ali Saidi, Arkaprava Basu, Joel Hestness, Derek R Hower, and Tushar Krishna.  The gem5 simulator.  ACM SIGARCH Computer Architecture News, 39, 2011.
  * [47] Sheng Li, Jung Ho Ahn, Richard D Strong, Jay B. Brockman, Dean M Tullsen, and Norman P Jouppi.  The McPAT Framework for Multicore and Manycore Architectures.  ACM Transactions on Architecture and Code Optimization, 10, 2013\.