# Carbon-Aware End-to-End Data Movement

Jacob Goldverg, Hasibul Jamil, Elvis Rodriguez and Tevfik Kosar  University at
Buffalo, Buffalo, NY 14260
{jacobgold, mdhasibu, elvisdav<EMAIL_ADDRESS>
###### Abstract.

The latest trends in the adoption of cloud, edge, and distributed computing,
as well as a rise in applying AI/ML workloads, have created a need to measure,
monitor, and reduce the carbon emissions of these compute-intensive workloads
and the associated communication costs. The data movement over networks has
considerable carbon emission that has been neglected due to the difficulty in
measuring the carbon footprint of a given end-to-end network path. We present
a novel network carbon footprint measuring mechanism and propose three ways in
which users can optimize scheduling network-intensive tasks to enable carbon
savings through shifting tasks in time, space, and overlay networks based on
the geographic carbon intensity.

End-to-end data movement, carbon-efficiency, carbon-aware scheduling, temporal
and spatial scheduling.

††conference: ; June 14th; 2024

##  1\. Introduction

Data has become the foundation of today’s IT-dependent world, and the plethora
of data generated by AI workloads, scientific applications, social media, and
e-commerce fuels large-scale data analytics systems. As a result, data
transfer over the Internet has been increasing each year exponentially and has
already exceeded the zettabyte scale (Systems, 2019). With the increased data
generation rate, the data movement’s carbon footprint is becoming an
overwhelmingly critical problem, especially for HPC and Cloud data centers. It
is estimated that information and communication technologies will use between
8% - 21% of the world’s electricity by 2030 (Belkhir and Elmeligi, 2018). The
share of communication networks in the total IT power consumption is around
43% (Andrae and Edler, 2015). Data transfers over the Internet consume more
than a hundred terawatt-hours of energy, which costs 20 Billion US dollars
annually. This trend has motivated considerable work to reduce the energy
consumption of networking at all layers, including the hardware, software,
protocol, and applications.

Despite the advances in networking technologies, sending data over networks is
still very costly in terms of energy consumption and carbon footprint. The
researchers found that sending hard drives between collaborating institutions
would be many orders of magnitude less carbon-emitting than transferring the
data over communication networks (Aujoux et al., 2021). And, due to increased
energy costs pushed by constantly increasing traffic volumes, current network
energy costs of telecommunication service operators in developing countries
already span between 40% and 50% of provider operational expenditures (Lorincz
et al., 2019).

There has been a considerable amount of work focusing on power management and
energy efficiency in hardware and software systems (Brooks et al., 2000;
Rawson and Austin, 2004; Zedlewski et al., [n. d.]; Gurumurthi et al., 2002;
Contreras and Martonosi, 2005; Economou et al., 2006; Fan et al., 2007;
Rivoire et al., 2008; Koller et al., 2010; Hasebe et al., [n. d.]; Vrbsky et
al., 2013) and on power-aware networking (Ananthanarayanan and Katz, [n. d.];
Mahadevan et al., [n. d.]; Greenberg et al., [n. d.]; Heller et al., [n. d.];
Goma et al., 2011; Alan et al., 2014, 2015; Guner and Kosar, 2018; Di Tacchio
et al., 2019; Rodolph et al., 2021; Jamil et al., 2022a; Nine et al., 2023).
More recently. a consensus among major IT companies and federal agencies has
shifted the focus to the environmental impact of this vast power consumption
and underscored the urgency to decrease carbon emissions during the entire
lifecycle of computing, including networking and data movement (NSF, 2024).

Despite the broad range of research in power management techniques for the
networking infrastructure, there has been little work focusing on reducing the
carbon emissions of end-to-end data movement, including the emissions at the
end systems (i.e., sender and receiver nodes) during active data transfers. A
study shows that approximately 25% of total electricity consumption during the
end-to-end data transfers occur at the end-systems on a global
(intercontinental) network, whereas this number goes up to 60% on a nationwide
network and up to 90% on a local area network (Alan et al., 2015). This ratio
depends on the number of network devices (i.e., routers, switches, hubs, etc.)
between the sender and receiver nodes and how much power each device consumes,
as shown in Figure 1.

Figure 1. Energy consumption share of the end systems vs network
infrastructure during end-to-end data transfer. This ratio depends on the
number of network devices (i.e., routers, switches, etc.) between the end
systems, and how much power each device consumes.

Measuring, monitoring, and optimizing end-to-end data movement for carbon
efficiency is a challenging task. Although there has been some recent work on
the carbon efficiency of the networking infrastructure during data movement
(Zilberman et al., 2023; El-Zahr et al., 2023; Tabaeiaghdaei et al., 2023),
the carbon emissions at the end systems (i.e., sender and receiver nodes)
during data movement have been long ignored. This makes it difficult to
understand the end-to-end carbon intensity of data movement.

This paper makes several novel contributions to the field: (1) it presents a
method to measure the carbon footprint of end-to-end data movement on a given
network path by computing the carbon intensity for each hop on the path as
well as for the end systems; (2) it discusses the potential of scheduling data
movement in time and space to reduce the carbon footprint of data transfers;
and (3) it shows the need for fine-grained host monitoring and combining this
with carbon reporting of network paths.

##  2\. Background and Related Work

The internet stack is collectively comprised of many components starting from
end systems that is either the sender or receiver, into network switches and
routers and finally the destination end system. From the application level
users are completely abstracted away from the network infrastructure and
exposed minimal amounts of information via the operating system. TCP based
protocols are client driven thereby users on the client side are able to
obtain networking metrics such as: latency, RTT, dropped packets and
throughput. Furthermore Linux tools such as Traceroute enables users to
measure the network path used for communication if the destination server and
ISP’s do not drop or mask ICMP packets (tra, 1987). Unfortunately routers and
network switches do not report utilization metrics to users in the application
level making it near impossible for users to understand buffer sizes, and
ports used along the network path, hence users are left with very trivial
networking metrics in the application layer.

Due to the large amount of components in internet technology, measuring the
carbon intensity is non-trivial and requires specific parts of the internet
stack to be measured independently when constructing an end-to-end carbon
intensity. Carbon intensity is interpreted as the amount of carbon dioxide
emissions produced per unit of computing power or data processed. Networks can
be broken down into hardware and software components. On the hardware side,
there are network switches and routers that collectively do not expose any
form of metrics related to carbon intensity. Metrics related to resource
utilization are exposed and then referencing the models data sheet we are able
to get an estimate on energy consumption where we can then correlate the
energy draw to the carbon-intensity. The software side of a network is much
more challenging to measure as we generally require hardware support to
measure electrical consumption.

Carbon-aware routing is a recently explored topic that would make Border
Gateway Protocol (BGP) route internet traffic based on the carbon intensity of
network regions. By disabling and routing traffic through different regions
instead of just QoS and other metrics ISPs commonly consider, large carbon
savings can be achieved for general internet users (El-Zahr et al., 2023).
Furthermore, due to the challenges in observing the power consumption of
routers and networking components, a traffic engineering approach becomes
highly feasible (El-Zahr et al., 2023).

The scheduling of compute-bound tasks is a topic heavily studied in varying
domains. In big data centers, jobs are executed based on user priority such
that the priority determines if the job can be shifted in time and space as
well as to the extent. Shifting jobs in time to maximize: green energy
utilization, throughput, and electrical load utilization of data centers is
being done today (Acun et al., 2023).

Carbon-aware networking faces many challenges due to the lack of kernel and
network metrics reported in the application space. The sum cost of all
networking components that comprise the network stack is essential when
determining the carbon footprint imposed when an end-systems network
utilization varies (Zilberman et al., 2023; Andrae and Edler, 2015).
Unfortunately, prior work in making networks carbon-aware has ignored the
carbon emission during data transfer at the end systems.

In mobile cloud environments, job executions may be paused and transferred to
another device if there is insufficient battery life to complete the entire
job (Dinh et al., 2017). This process, known as compute offloading, requires
an overlay network where a central node (or the network) manages available
nodes that can take over the job, allowing it to continue on a different
device. Due to energy constraints on mobile devices, checkpointing a job and
then running it on a device with sufficient power resources enables the task
to be completed without wasting previously computed work(Dinh et al., 2017).

In modern 5G networks, there is a significant focus on offloading compute-
intensive tasks to nearby cloud providers. This practice aims to improve both
battery life and throughput of tasks that are limited by computational
resources (Xie et al., 2020; Dinh et al., 2017). Green NFV (Network Function
Virtualization) investigates the optimal placement and scheduling of virtual
network functions (VNFs) to maximize throughput, energy efficiency, and
resource utilization (Nine et al., 2023). Additionally, there has been
extensive research on optimizing big data transfers to enhance throughput and
end-system utilization (Imran et al., 2017; Yildirim et al., 2016; Xia et al.,
2023; Kosar, 2005; Kosar et al., 2019; Jamil et al., 2022b, a; Yildirim et
al., 2009).

There are three common concepts introduced in the covered past work that
enable the ability for data transfer offerings to optimize for carbon
intensity. Shifting a scheduled file transfer in time allows the file transfer
node to have a different carbon intensity along the same network path by
starting the job when renewable resources are more readily available. The file
transfer node running the transfer can also be changed during the job
execution, thereby using a different network path which results in a greener
transfer, we refer to as using an overlay network. Finally, due to the
duplicity of files that are distributed geographically, making it possible for
file transfer nodes (FTNs) to use a file source that has a greener network
path, we refer to this as shifting in space.

In this work, we emphasize the ability for data movement tools to optimize for
carbon intensity, instead of just throughput and energy efficiency, through
carbon aware scheduling. Existing efforts have been made in developing
approaches to making the internet greener but have failed in including the
cost of the end-system (Zilberman et al., 2023; Andrae and Edler, 2015). We
emphasise the importance of measuring the carbon-intensity of end systems as
well as the network path utilized for the file transfer.

##  3\. Carbon Intensity of Data Path

With the centralization of data as well as the growth in data size, the demand
for fast and efficient file transfers has become essential for domain
scientists to conduct research. Due to this, various efforts have been made in
optimizing file transfer performance either through utilizing multiple network
paths (Xia et al., 2023), tuning TCP parameters (Xia et al., 2023; Yildirim et
al., 2016; Imran et al., 2017; Foster and Kesselman, 1998) as well as
scheduling the file transfer when bandwidth is more available (Kosar et al.,
2013). With such efforts being made in throughput and end-to-end file transfer
efficiency, we would like to present how file transfer platforms can optimize
for carbon intensity. Furthermore, we would like to demonstrate the importance
of measuring the carbon intensity from the file source to the file
destination.

Offering end-to-end carbon-aware file transfers is dependent on:

  1. (1)

End system monitoring and reporting of fine-grained network and host metrics;

  2. (2)

Measuring of IP addresses from file source to destination;

  3. (3)

The ability to measure the longitude and latitude position of each IP address
and look up the carbon intensity per location;

  4. (4)

A scheduler capable of ingesting such information and then launching transfers
when carbon savings.

Table 1. End-system metrics collected for monitoring Host Metrics |  Network Metrics |  Transfer Metrics
---|---|---
Core Count |  Drop Out |  Job Uuid
Free Memory |  Drop In |  Source Latency
Max Memory |  Error In |  Job Size
Memory |  Error Out |  Transfer Node ID
MinCPU Frequency |  Dst Latency |  Buffer Size
MaxCPU Frequency |  Src Rtt |  Parallelism
CurrentCPU Freq |  Dst Rtt |  Concurrency
CPU Architecture |  NIC MTU |  Pipelining
CPU Utilization |  Network Interface |  Bytes Received
|  Packet Sent |  Bytes Sent
|  Packet Received |
|  NIC Speed |
|  Read Thrpt |
|  Write Thrpt |

###  3.1. End-system Monitoring

Various libraries and tools exist to expose kernel, CPU, and network metrics
into the application space of an end system. Monitoring CPU and system power
consumption is a hardware-dependent task that relies on specialized hardware.
Mobile systems today typically provide a file to monitor the power draw of
different components in the device. Android provides an Energy Profiler that
captures the power consumption of the antenna, CPU, and GPS. Unfortunately,
most consumer hardware does not support fine-grained power monitoring (and,
[n. d.]). Users have some options provided to them when using Linux. Perf is a
common utility that provides power consumption of some processors, and Intel
provides RAPL to give users direct joules consumed per core (int, [n. d.];
lin, [n. d.]). To capture the networking metrics, users can use specific
libraries such as Psutil, netstat, and parsing the Linux files related to TCP,
UDP, and raw socket information (net, [n. d.]; Rodola, 2014). A non-exhaustive
list of crucial metrics a tool such as this must report is listed in Table 1.

###  3.2. Discovering a Network Path

Typical Linux-based solutions use either ICMP or UDP to discover the network
path from sender to receiver. This is done by manipulating the Time to Live
(TTL) of a packet triggering a response from a router with its IP address.
This process is done until the destination IP address has been reached. Packet
manipulation libraries such as Scapy can be used to do this manually, or using
a tool like Traceroute is sufficient to discover the network path (tra, 1987;
sca, 2018). Users have numerous options to discover the network path all
depending on the operating system, programming language, and available
libraries.

###  3.3. Geolocating and Measuring the Carbon Intensity of a Network Path

To find the longitude and latitude coordinates for a list of public IP
addresses, geolocation databases are used. These databases can be either
offline or online, and there are many free options available. One such option
is IP-API, which provides an online geolocation database with free API access.
This database is regularly updated to ensure the most recent mapping of IP
addresses to geographical positions (ipA, 2018). Once the coordinates are
obtained, services like Electricity Maps or WattTime can be used to retrieve
live or historical carbon intensity information for the corresponding
geographic regions (ele, 2024; wat, 2014).

We created a boxplot to visualize the process as shown in Figure 2. This plot
covers a 51-hour period and reveals natural clusters and variations of carbon
intensity among IP addresses that share the same electrical regions. We notice
that the hops are all grouped into natural regions through the carbon
intensity values, making the network path groupable based on the energy
provider.

Figure 2. Carbon Intensity of IP addresses from UC to TACC

###  3.4. Carbon Intensity of a File Transfer

Throughput and carbon intensity are both changing values over time, hence it
is crucial to track both numbers over the duration of the entire file
transfer. With this, we propose a trivial formula to calculate the carbon
intensity per byte moved over a network.

(1)
$\text{$carbonscore$}=\frac{\text{$bytes$}}{\text{$CI$}\times\text{$duration$}}$

where $bytes$ is the total number of bytes transferred, $CI$ is the average
carbon intensity of the file transfer, and $duration$ is the total amount of
time the file transfer took in seconds. Hence, we interpret this metric as the
carbon intensity per bit per second. This formulation allows us to understand
the performance as well as the imposing carbon cost of the file transfer.

By measuring the carbon intensity throughout a file transfer, tools can
calculate the total average carbon intensity per bytes per second. After the
transfer finishes, the job can be evaluated based on both the achieved
throughput and the carbon impact of the file transfer. Using this simple
heuristic and bandwidth prediction, it becomes possible to estimate the carbon
impact a file transfer will have at a certain time. This prediction would take
into account the previously viewed throughput of jobs given the same file
source and destination as well as the application parameters for the transfer.
Thereby it will allow a scheduler to estimate the carbon impact and throughput
beforehand.

Due to there being no current open-source tool that offers the discussed
features, we chose to build our own. Pmeter (Goldverg, 2022) is an open-source
tool that: uses the ICMP protocol to discover the network path, IP-API (ipA,
2018) to to geolocate each IP address, Electricity Maps (ele, 2024) to
discover the carbon intensity per IP, and provides such this feature set
agnostic of the underlying operating system monitoring tool.

##  4\. Experimentation

Chameleon Cloud (CC)(Keahey et al., 2020) is an HPC cloud like provider
offering baremetal hardware for research purposes. This is preferred for our
initial experimentation as it allows us to avoid the cost of virtualization
and shared hardware resources, as well get operating system metrics imposed by
strictly the utilized applications. We also use an Apple M1 Macbook Pro that
is hosted in a DIDC lab. Two nodes are deployed in CC that are geographically
separated, in TACC we deploy an Nginx server with the firewall fully disabled,
in UC, and on the macbook we deploy the OneDataShare FTN with Pmeter installed
on the system (Imran et al., 2017; Goldverg, 2022). We depict the utilized
hardware for our experimentation in Table 2.

Table 2. Nodes used for Carbon Intensity Measurement Provider | Site | Flavor | CPU | RAM | NIC | OS
---|---|---|---|---|---|---
Chameleon Cloud | TACC | Baremetal | Cascade Lake | 192 GiB | 10 Gbps | Ubuntu 22
Chameleon Cloud | UC | Baremetal | SkyLake | 192 GiB | 10 Gbps | Ubuntu 22
DIDCLab | University at Buffalo | Baremetal | Apple M1 | 14.9 GiB | 1.2 Gbps | mac OS Sonoma

###  4.1. Shifting in Time

Shifting a file transfer in time utilizes the same file source, destination,
and FTN but simply changes the time at which the file transfer begins. To
motivate the impact of shifting a file transfer in time, we would like to
visually show the average carbon intensity from UC to TACC. Every hour, Pmeter
is run and we visualize the average carbon intensity output between the two
nodes in Figure 3. We observe that the minimum average carbon intensity over
the $51$ hour period is $255.714$ where the maximum is $488.6$, implying by
simply shifting a job in time a carbon aware scheduler can achieve nearly 2̃x
in carbon savings.

Figure 3. Average Carbon Intensity at 1-Hour Intervals from UC to TACC, April
14-16, 2024

###  4.2. Shifting in Space

Shifting in space as previously described is when there is data duplicated on
many servers, similar to Content Delivery Networks (CDNs). A carbon-aware
scheduler picks a server as the file source with the lowest associated carbon
intensity to conduct the file transfer. A further motivation of this is to
visualize the associated carbon index of various states throughout the United
States (emi, 2023), we visualize the carbon index of ten states in Figure 4.

Figure 4. Carbon Index of 10 states in the US

Determining the appropriate source file server is crucial in minimizing the
overall carbon intensity as states in the US have varying carbon index’s. In
the most extreme case we see that Wyoming has a carbon index of $1919$ where
Vermont has a score of $1$. This implies that if the same dataset was
duplicated in both states, then a scheduler that chooses Vermont as the file
source would lead to $1919x$ in carbon savings strictly due to selecting the
dataset residing in a greener location.

###  4.3. Using an Overlay Network of File Transfer Nodes

Overlay networks provide the capability to select the FTN responsible for
processing a file transfer job and allow for the migration of work among FTNs
when there are resource limitations. For carbon-aware file transfer scheduling
using overlay networks is possible as choosing an FTN with the lowest carbon
intensity to execute the file transfer. Moreover, if a predefined carbon
intensity threshold is exceeded, the remaining work can be migrated to an FTN
with a lower carbon intensity. To illustrate the impact of overlay networks on
carbon-aware file transfers, we compare the average carbon intensity of the
network path where the file source is hosted at TACC, while both UC and a
MacBook host the FTNs that would download a dataset. In Figure 5, we can see
that the MacBook in the DIDC lab is better positioned to process the file
transfer job due to the lower carbon intensity of the entire network path and
its path to TACC having fewer hops.

Figure 5. Measuring the Carbon Intensity from UC and M1 to TACC

##  5\. Future Work and Conclusion

The emphasis on file transfer performance and efficiency has opened up
potential directions for making file transfers more carbon-efficient. This
study introduced three techniques for scheduling file transfers to reduce
carbon costs. Future work should integrate these techniques with established
metrics, such as throughput, latency, and fair-share utilization, alongside
the new carbon intensity metrics. Given the complexity of incorporating a
highly stochastic metric like carbon intensity, it is crucial to balance this
with past work.

In end-to-end data transfer tuning(TCP, 2015; Nine and Kosar, 2021; Kosar et
al., 2019; Yildirim et al., 2009, 2012; Imran et al., 2017) and end-to-end
data pipeline optimization (Deelman et al., 2006; Bahsi et al., 2007; Kougka
et al., 2018)—topics that have been extensively researched should also
consider carbon intensity. An immediate area of research could explore the
relationship between throughput and carbon intensity, striving to balance them
effectively during file transfers. Further investigation is also needed on how
application parameters influence the carbon impact of a file transfer. Using
power models that estimate the energy consumption of end systems based on
memory, CPU, and network utilization in conjunction with the carbon intensity
of the end system will be essential for calculating total carbon costs(Alan et
al., 2015).

Moreover, with Service Level Agreements (SLAs), platforms can offer users the
choice to set preferred carbon footprints and performance levels for their
jobs, thus promoting green computing initiatives. This approach ensures high
user satisfaction by balancing QoS with carbon efficiency, thereby offloading
the responsibility of carbon savings of a job to the user and not just the
platform.

Cloud providers, with their geographically distributed data centers, stand to
benefit significantly from the presented techniques with the goal of carbon
savings. Meta has already implemented and utilized shifting jobs in time to
maximize the utilization of green energy (Acun et al., 2023). By using overlay
networks, placing jobs in data center sites that are physically closer to the
file source and have a greener network path to the required dataset, data
centers can reduce carbon intensity associated with network I/O. Testbeds such
as Fabric and Chameleon Cloud can be utilized to study how positioning of jobs
at certain sites can potentially increase throughput and reduce carbon
intensity by bringing the compute task closer to the data (Keahey et al.,
2020; Baldin et al., [n. d.]).

CDNs and caches are widely used to enhance QoS and reduce latency for end
users. CDNs are regularly updated to provide the latest content during off-
peak hours, thereby reducing server downtime. However, there has been limited
research on the carbon intensity of CDNs and their communication costs due to
these updates. This area of study could benefit from time and space shifting
as well as using overlay networks. Time shifting could involve updating the
CDNs when carbon intensity is lowest, while space shifting could optimize the
file source location of the newest content for minimal carbon impact. Overlay
networks may help determine the update sequence of CDNs to further reduce the
carbon intensity of the network path.

In conclusion, as we delve deeper into the carbon implications of networked
systems and file transfers, it becomes imperative to integrate carbon cost
considerations into future computing research. This study has shown that
application-level tools and libraries can play a great role in monitoring and
revealing the increasing carbon costs associated with file transfers over
network paths.

## Acknowledgements

This project is in part sponsored by the National Science Foundation (NSF)
under award number OAC-2313061.

## References

  * (1)
  * and ([n. d.]) [n. d.].  Android Energy Profiler.  https://developer.android.com/studio/profile/energy-profiler[Accessed: April 2024].
  * lin ([n. d.]) [n. d.].  Linux Perf.  https://github.com/torvalds/linux/blob/master/tools/perf/Documentation/perf.txt[Accessed: April 2024].
  * net ([n. d.]) [n. d.].  netstat.  https://linux.die.net/man/8/netstat[Accessed: April 2024].
  * int ([n. d.]) [n. d.].  Running Average Power Limit.  https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/advisory-guidance/running-average-power-limit-energy-reporting.html[Accessed: April 2024].
  * tra (1987) 1987\.  Traceroute.  https://linux.die.net/man/8/traceroute.
  * wat (2014) 2014\.  WattTime.  https://docs.watttime.org/.
  * TCP (2015) 2015\.  SMTP service extension for command pipelining.  http://tools.ietf.org/html/rfc2920.
  * ipA (2018) 2018\.  IP Geolocation API.  https://ip-api.com/.
  * sca (2018) 2018\.  Scapy.  https://scapy.net/.
  * emi (2023) 2023\.  US Power Sector Emissions.  https://emissionsindex.org/.
  * ele (2024) 2024\.  Electricity Maps.  https://www.electricitymap.org/.
  * Acun et al. (2023) Bilge Acun, Benjamin Lee, Fiodar Kazhamiaka, Kiwan Maeng, Udit Gupta, Manoj Chakkaravarthy, David Brooks, and Carole-Jean Wu. 2023\.  Carbon Explorer: A Holistic Framework for Designing Carbon Aware Datacenters. In _Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2_ (Vancouver, BC, Canada) _(ASPLOS 2023)_. Association for Computing Machinery, New York, NY, USA, 118–132.  https://doi.org/10.1145/3575693.3575754
  * Alan et al. (2014) Ismail Alan, Engin Arslan, and Tevfik Kosar. 2014\.  Energy-aware data transfer tuning. In _2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing_. IEEE, 626–634.
  * Alan et al. (2015) Ismail Alan, Engin Arslan, and Tevfik Kosar. 2015\.  Energy-aware data transfer algorithms. In _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_. 1–12.
  * Ananthanarayanan and Katz ([n. d.]) G. Ananthanarayanan and R. Katz. [n. d.].  Greening the Switch. In _In Proceedings of HotPower, December 2008_.
  * Andrae and Edler (2015) Anders S. G. Andrae and Tomas Edler. 2015.  On Global Electricity Usage of Communication Technology: Trends to 2030.  _Challenges_ 6, 1 (2015), 117–157.  https://doi.org/10.3390/challe6010117
  * Aujoux et al. (2021) Clarisse Aujoux, Kumiko Kotera, and Odile Blanchard. 2021\.  Estimating the carbon footprint of the GRAND project, a multi-decade astrophysics experiment.  _Astroparticle Physics_ 131 (2021), 102587.
  * Bahsi et al. (2007) Emir M Bahsi, Emrah Ceyhan, and Tevfik Kosar. 2007\.  Conditional workflow management: A survey and analysis.  _Scientific Programming_ 15, 4 (2007), 283–297.
  * Baldin et al. ([n. d.]) Ilya Baldin, Anita Nikolich, James Griffioen, Indermohan Inder S Monga, Kuang-Ching Wang, Tom Lehman, and Paul Ruth. [n. d.].  FABRIC: A national-scale programmable experimental network infrastructure.  _IEEE Internet Computing_ 23, 6 ([n. d.]), 38–47.
  * Belkhir and Elmeligi (2018) Lotfi Belkhir and Ahmed Elmeligi. 2018.  Assessing ICT global emissions footprint: Trends to 2040 & recommendations.  _Journal of cleaner production_ 177 (2018), 448–463.
  * Brooks et al. (2000) David Brooks, Vivek Tiwari, and Margaret Martonosi. 2000\.  Wattch: a framework for architectural-level power analysis and optimizations. In _Proceedings of the 27th annual international symposium on Computer architecture_ (Vancouver, British Columbia, Canada) _(ISCA ’00)_. ACM, New York, NY, USA, 83–94.
  * Contreras and Martonosi (2005) Gilberto Contreras and Margaret Martonosi. 2005.  Power prediction for intel XScale® processors using performance monitoring unit events. In _ISLPED’05_. 221–226.
  * Deelman et al. (2006) Ewa Deelman, Tevfik Kosar, Carl Kesselman, and Miron Livny. 2006.  What makes workflows work in an opportunistic environment?  _Concurrency and Computation: Practice and Experience_ 18, 10 (2006), 1187–1199.
  * Di Tacchio et al. (2019) Luigi Di Tacchio, MD SQ Zulkar Nine, Tevfik Kosar, Muhammed Fatih Bulut, and Jinho Hwang. 2019\.  Cross-layer optimization of big data transfer throughput and energy consumption. In _2019 IEEE 12th International Conference on Cloud Computing (CLOUD)_. IEEE, 25–32.
  * Dinh et al. (2017) Thinh Quang Dinh, Jianhua Tang, Quang Duy La, and Tony Q. S. Quek. 2017.  Offloading in Mobile Edge Computing: Task Allocation and Computational Frequency Scaling.  _IEEE Transactions on Communications_ 65, 8 (2017), 3571–3584.  https://doi.org/10.1109/TCOMM.2017.2699660
  * Economou et al. (2006) Dimitris Economou, Suzanne Rivoire, Christos Kozyrakis, and Partha Ranganathan. 2006.  Full-system power analysis and modeling for server environments. In _Proc. of Workshop on Modeling, Benchmarking, and Simulation_.
  * El-Zahr et al. (2023) Sawsan El-Zahr, Paul Gunning, and Noa Zilberman. 2023\.  Exploring the Benefits of Carbon-Aware Routing.  _Proc. ACM Netw._ 1, CoNEXT3, Article 20 (nov 2023), 24 pages.  https://doi.org/10.1145/3629165
  * Fan et al. (2007) Xiaobo Fan, Wolf-Dietrich Weber, and Luiz Andre Barroso. 2007\.  Power provisioning for a warehouse-sized computer.  _ACM SIGARCH Computer Architecture News_ 35, 2 (2007), 13–23.
  * Foster and Kesselman (1998) I. Foster and C. Kesselman. 1998.  The Globus Project: A Status Report. In _7th IEEE Heterogeneous Computing Workshop (HCW 98)_. 4–18.
  * Goldverg (2022) Jacob Goldverg. 2022\.  Pmeter.  https://github.com/didclab/pmeter.
  * Goma et al. (2011) Eduard Goma, Marco Canini, Alberto Lopez Toledo, Nikolaos Laoutaris, Dejan Kostić, Pablo Rodriguez, Rade Stanojević, and Pablo Yagüe Valentin. 2011.  Insomnia in the access: or how to curb access network related energy consumption. In _Proceedings of the ACM SIGCOMM 2011 Conference_. 338–349.
  * Greenberg et al. ([n. d.]) A. Greenberg, J. Hamilton, D. Maltz, and P. Patel. [n. d.].  The Cost of a Cloud: Research Problems in Data Center Networks. In _In ACM SIGCOMM CCR, January 2009_.
  * Guner and Kosar (2018) Kemal Guner and Tevfik Kosar. 2018.  Energy-Efficient Mobile Network I/O. In _2018 IEEE Global Communications Conference (GLOBECOM)_. IEEE, 1–6.
  * Gurumurthi et al. (2002) Sudhanva Gurumurthi, Anand Sivasubramaniam, Mary Jane Irwin, Narayanan Vijaykrishnan, and Mahmut Kandemir. 2002.  Using complete machine simulation for software power estimation: The softwatt approach. In _Prpc. of 8th High-Performance Computer Architecture Symp._ 141–150.
  * Hasebe et al. ([n. d.]) Koji Hasebe, Tatsuya Niwa, Akiyoshi Sugiki, and Kazuhiko Kato. [n. d.].  Power-saving in large-scale storage systems with data migration. In _IEEE CloudCom 2010_.
  * Heller et al. ([n. d.]) B. Heller, S. Seetharaman, P. Mahadevan, Y. Yiakoumis, P. Sharma, S. Banerjee, and N. McKeown. [n. d.].  ElasticTree: Saving Energy in Data Center Networks. In _Proceedings of NSDI 2010_.
  * Imran et al. (2017) Asif Imran, Md S Q Zulkar Nine, Kemal Guner, and Tevfik Kosar. 2017\.  OneDataShare: A Vision for Cloud-hosted Data Transfer Scheduling and Optimization as a Service.  arXiv:1712.02944 [cs.DC]
  * Jamil et al. (2022a) Hasibul Jamil, Lavone Rodolph, Jacob Goldverg, and Tevfik Kosar. 2022a.  Energy-Efficient Data Transfer Optimization via Decision-Tree Based Uncertainty Reduction.  _arXiv preprint arXiv:2204.07601_ (2022).
  * Jamil et al. (2022b) Hasibul Jamil, Elvis Rodrigues, Jacob Goldverg, and Tevfik Kosar. 2022b.  A Reinforcement Learning Approach to Optimize Available Network Bandwidth Utilization.  _arXiv preprint arXiv:2211.11949_ (2022).
  * Keahey et al. (2020) Kate Keahey, Jason Anderson, Zhuo Zhen, Pierre Riteau, Paul Ruth, Dan Stanzione, Mert Cevik, Jacob Colleran, Haryadi S. Gunawi, Cody Hammock, Joe Mambretti, Alexander Barnes, François Halbach, Alex Rocha, and Joe Stubbs. 2020.  Lessons Learned from the Chameleon Testbed.  In _Proceedings of the 2020 USENIX Annual Technical Conference (USENIX ATC ’20)_. USENIX Association.
  * Koller et al. (2010) Ricardo Koller, Akshat Verma, and Anindya Neogi. 2010\.  WattApp: an application aware power meter for shared data centers. In _Proceedings of the 7th international conference on Autonomic computing_. ACM, 31–40.
  * Kosar (2005) Tevfik Kosar. 2005\.  _Data placement in widely distributed systems_.  The University of Wisconsin-Madison.
  * Kosar et al. (2019) Tevfik Kosar, Ismail Alan, and M Fatih Bulut. 2019\.  Energy-aware data throughput optimization for next generation internet.  _Information Sciences_ 476 (2019), 413–428.
  * Kosar et al. (2013) Tevfik Kosar, Engin Arslan, Brandon Ross, and Bing Zhang. 2013\.  StorkCloud: Data transfer scheduling and optimization as a service.  _ScienceCloud 2013 - Proceedings of the 4th ACM Workshop on Scientific Cloud Computing_ , 29–36.  https://doi.org/10.1145/2465848.2465855
  * Kougka et al. (2018) Georgia Kougka, Anastasios Gounaris, and Alkis Simitsis. 2018\.  The many faces of data-centric workflow optimization: a survey.  _International Journal of Data Science and Analytics_ 6 (2018), 81–107.
  * Lorincz et al. (2019) Josip Lorincz, Antonio Capone, and Jinsong Wu. 2019\.  Greener, energy-efficient and sustainable networks: State-of-the-art and new trends.  , 4864 pages.
  * Mahadevan et al. ([n. d.]) P. Mahadevan, P. Sharma, S. Banerjee, and P. Ranganathan. [n. d.].  A Power Benchmarking Framework for Network Devices. In _In Proceedings of IFIP Networking, May 2009_.
  * Nine and Kosar (2021) MD S Q Zulkar Nine and Tevfik Kosar. 2021.  A Two-Phase Dynamic Throughput Optimization Model for Big Data Transfers.  _IEEE Transactions on Parallel and Distributed Systems_ 32, 2 (2021), 269–280.  https://doi.org/10.1109/TPDS.2020.3012929
  * Nine et al. (2023) Md S. Q. Zulkar Nine, Tevfik Kosar, Muhammed Fatih Bulut, and Jinho Hwang. 2023.  GreenNFV: Energy-Efficient Network Function Virtualization with Service Level Agreement Constraints. In _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ (, Denver, CO, USA,) _(SC ’23)_. Association for Computing Machinery, New York, NY, USA, Article 21, 12 pages.  https://doi.org/10.1145/3581784.3607090
  * NSF (2024) NSF. 2024.  NSF Workshop on Sustainable Computing for Sustainability.  https://edas.info/web/nsf-wscs24/.
  * Rawson and Austin (2004) Freeman Rawson and IBM Austin. 2004.  MEMPOWER: A simple memory power analysis tool set.  _IBM Austin Research Laboratory_ (2004).
  * Rivoire et al. (2008) Suzanne Rivoire, Parthasarathy Ranganathan, and Christos Kozyrakis. 2008.  A Comparison of High-Level Full-System Power Models.  _HotPower_ 8 (2008), 3–3.
  * Rodola (2014) Giampaolo Rodola. 2014\.  Psutil.  https://github.com/giampaolo/psutil.
  * Rodolph et al. (2021) Lavone Rodolph, MD SQ Zulkar Nine, Luigi Di Tacchio, and Tevfik Kosar. 2021.  Energy-saving Cross-layer Optimization of Big Data Transfer Based on Historical Log Analysis. In _ICC 2021-IEEE International Conference on Communications_. IEEE, 1–7.
  * Systems (2019) Cisco Systems. January 2019\.  Visual Networking Index: Forecast and Methodology, 2017–2022.
  * Tabaeiaghdaei et al. (2023) Seyedali Tabaeiaghdaei, Simon Scherrer, Jonghoon Kwon, and Adrian Perrig. 2023.  Carbon-Aware Global Routing in Path-Aware Networks. In _Proceedings of the 14th ACM International Conference on Future Energy Systems_. 144–158.
  * Vrbsky et al. (2013) Susan V Vrbsky, Michael Galloway, Robert Carr, Rahul Nori, and David Grubic. 2013.  Decreasing power consumption with energy efficient data aware strategies.  _FGCS_ 29, 5 (2013), 1152–1163.
  * Xia et al. (2023) Libin Xia, Wei Sun, Xiaoyu Liu, Gongxing Sun, and Xiaowei Jiang. 2023.  Blaze: A High performance Big Data Computing System for High Energy Physics.  _Journal of Physics: Conference Series_ 2438 (02 2023), 012012\.  https://doi.org/10.1088/1742-6596/2438/1/012012
  * Xie et al. (2020) Yaxiong Xie, Fan Yi, and Kyle Jamieson. 2020.  PBE-CC: Congestion Control via Endpoint-Centric, Physical-Layer Bandwidth Measurements. In _Proceedings of the Annual Conference of the ACM Special Interest Group on Data Communication on the Applications, Technologies, Architectures, and Protocols for Computer Communication_ (Virtual Event, USA) _(SIGCOMM ’20)_. Association for Computing Machinery, New York, NY, USA, 451–464.  https://doi.org/10.1145/3387514.3405880
  * Yildirim et al. (2016) Esma Yildirim, Engin Arslan, Jangyoung Kim, and Tevfik Kosar. 2016.  Application-Level Optimization of Big Data Transfers through Pipelining, Parallelism and Concurrency.  _IEEE Transactions on Cloud Computing_ 4, 1 (2016), 63–75.  https://doi.org/10.1109/TCC.2015.2415804
  * Yildirim et al. (2012) Esma Yildirim, JangYoung Kim, and Tevfik Kosar. 2012.  How GridFTP Pipelining, Parallelism and Concurrency Work: A Guide for Optimizing Large Dataset Transfers. In _2012 SC Companion: High Performance Computing, Networking Storage and Analysis_. 506–515.  https://doi.org/10.1109/SC.Companion.2012.73
  * Yildirim et al. (2009) Esma Yildirim, Dengpan Yin, and Tevfik Kosar. 2009.  Balancing TCP Buffer vs Parallel Streams in Application Level Throughput Optimization. In _Proceedings of the Second International Workshop on Data-Aware Distributed Computing_ (Garching, Germany) _(DADC ’09)_. Association for Computing Machinery, New York, NY, USA, 21–30.  https://doi.org/10.1145/1552280.1552283
  * Zedlewski et al. ([n. d.]) John Zedlewski, Sumeet Sobti, Nitin Garg, Fengzhou Zheng, Arvind Krishnamurthy, and Randolph Y Wang. [n. d.].  Modeling Hard-Disk Power Consumption.. In _FAST 2003_.
  * Zilberman et al. (2023) Noa Zilberman, Eve M. Schooler, Uri Cummings, Rajit Manohar, Dawn Nafus, Robert Soulé, and Rick Taylor. 2023.  Toward Carbon-Aware Networking.  _SIGENERGY Energy Inform. Rev._ 3, 3 (oct 2023), 15–20.  https://doi.org/10.1145/3630614.3630618