0 Research

# Scalable, Distributed AI Frameworks: Leveraging Cloud Computing for Enhanced
Deep Learning Performance and Efficiency

Neelesh Mungoli
UNC Charlotte . e-mail<EMAIL_ADDRESS>
###### Abstract

In recent years, the integration of artificial intelligence (AI) and cloud
computing has emerged as a promising avenue for addressing the growing
computational demands of AI applications. This paper presents a comprehensive
study of scalable, distributed AI frameworks leveraging cloud computing for
enhanced deep learning performance and efficiency. We first provide an
overview of popular AI frameworks and cloud services, highlighting their
respective strengths and weaknesses. Next, we delve into the critical aspects
of data storage and management in cloud-based AI systems, discussing data
preprocessing, feature engineering, privacy, and security. We then explore
parallel and distributed training techniques for AI models, focusing on model
partitioning, communication strategies, and cloud-based training
architectures.

In subsequent chapters, we discuss optimization strategies for AI workloads in
the cloud, covering load balancing, resource allocation, auto-scaling, and
performance benchmarking. We also examine AI model deployment and serving in
the cloud, outlining containerization, serverless deployment options, and
monitoring best practices. To ensure the cost-effectiveness of cloud-based AI
solutions, we present a thorough analysis of costs, optimization strategies,
and case studies showcasing successful deployments.

Finally, we summarize the key findings of this study, discuss the challenges
and limitations of cloud-based AI, and identify emerging trends and future
research opportunities in the field. This paper serves as a valuable resource
for researchers and practitioners looking to harness the power of cloud
computing to develop scalable, efficient, and cost-effective AI solutions.

AICloudMachineLearningDeepLearning

Introduction

###  0.1 Overview of AI and Cloud Computing

Artificial Intelligence (AI) refers to the development of computer systems
that can perform tasks typically requiring human intelligence, such as visual
perception, speech recognition, decision-making, and natural language
understanding. AI technologies have witnessed exponential growth in recent
years, driven by advancements in machine learning, particularly deep learning.
These methods rely on large amounts of data and powerful computational
resources to train complex models capable of making accurate predictions and
decisions.

Cloud computing is a paradigm that enables on-demand delivery of computing
resources, including processing power, storage, and applications, over the
internet. By providing scalable, flexible, and cost-effective solutions, cloud
computing has revolutionized the way businesses and organizations manage their
IT infrastructure. Cloud services are typically offered in three primary
models: Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and
Software as a Service (SaaS).

###  0.2 The Significance of Integrating AI with Cloud Computing

The integration of AI and cloud computing offers numerous benefits, addressing
the challenges associated with the growing computational demands of AI
applications. Some of the key advantages of this integration include:

  * •

Scalability: Cloud computing enables AI applications to scale on-demand,
allowing researchers and developers to access the required resources for
processing large datasets and training complex models. This scalability is
crucial for handling the increasing size and complexity of AI workloads.

  * •

Flexibility: The cloud offers a wide range of services and tools tailored for
AI development, enabling users to choose the most appropriate framework,
infrastructure, and environment for their specific needs. This flexibility
allows for rapid experimentation and iteration, accelerating the AI
development process.

  * •

Cost-effectiveness: By leveraging the pay-as-you-go model of cloud computing,
organizations can avoid significant upfront capital investments in hardware
and infrastructure. Furthermore, cloud providers offer various cost
optimization strategies to minimize expenses while maintaining performance.

  * •

Collaboration: Cloud-based AI platforms facilitate collaboration among
researchers and developers by providing a centralized environment for sharing
data, models, and resources. This collaborative approach fosters innovation
and accelerates the development of novel AI solutions.

  * •

Accessibility: Cloud computing democratizes access to cutting-edge AI
technologies, allowing organizations of all sizes and industries to harness
the power of AI without the need for specialized hardware or in-house
expertise.

In conclusion, the integration of AI and cloud computing has emerged as a
promising approach to address the increasing computational demands of AI
applications. By leveraging the scalability, flexibility, cost-effectiveness,
and collaboration opportunities offered by cloud computing, researchers and
practitioners can develop and deploy innovative AI solutions across a wide
range of domains. This paper aims to provide a comprehensive understanding of
the key aspects involved in harnessing the power of cloud computing for AI
development and deployment, with a focus on scalable, distributed AI
frameworks and their associated techniques and best practices.

##  1 AI Frameworks and Cloud Services

The rapid growth and adoption of AI technologies have led to the development
of numerous AI frameworks and the expansion of cloud services tailored for AI
applications. In this section, we provide an overview of popular AI
frameworks, such as TensorFlow and PyTorch, as well as prominent cloud service
providers, including AWS, Azure, and Google Cloud. Additionally, we compare
these AI frameworks and cloud services to help researchers and practitioners
make informed decisions when selecting the appropriate tools and platforms for
their specific AI projects.

###  1.1 Popular AI Frameworks

TensorFlow: Developed by Google Brain, TensorFlow is an open-source machine
learning framework designed for a wide range of tasks, including deep learning
and reinforcement learning. TensorFlow offers flexibility and performance
through its computation graph-based approach, enabling the efficient execution
of complex mathematical operations on various devices, such as CPUs, GPUs, and
TPUs. TensorFlow also provides a high-level API, Keras, which simplifies the
development of neural networks and allows for rapid prototyping.

PyTorch: Created by Facebook’s AI Research lab, PyTorch is another popular
open-source machine learning framework, known for its dynamic computation
graph and ”eager execution” approach. This feature allows developers to write
and debug code more intuitively, making PyTorch particularly well-suited for
research and experimentation. PyTorch also boasts a strong ecosystem of
libraries and tools, such as torchvision, torchtext, and torchaudio, which
support a wide range of AI applications.

Other notable AI frameworks include Microsoft’s Cognitive Toolkit (CNTK),
Apache MXNet, and Caffe. Each of these frameworks offers unique advantages,
depending on the specific requirements and constraints of a given AI project
[8] [3].

###  1.2 Cloud Service Providers

Amazon Web Services (AWS): As a leading cloud service provider, AWS offers a
comprehensive suite of AI and machine learning services, such as SageMaker,
Rekognition, and Lex. AWS SageMaker simplifies the process of building,
training, and deploying machine learning models, providing a fully managed
platform that supports TensorFlow, PyTorch, and other popular frameworks.
Additionally, AWS provides various AI-powered services for image and video
analysis, natural language processing, and speech recognition, among others.

Microsoft Azure: Azure’s AI and machine learning services include Azure
Machine Learning, Cognitive Services, and the ONNX Runtime. Azure Machine
Learning is a fully managed platform that supports a wide range of AI
frameworks, offering capabilities for model training, deployment, and
management. Azure Cognitive Services provides pre-built AI models for various
tasks, such as language understanding, speech recognition, and computer
vision, while the ONNX Runtime facilitates the efficient execution of trained
models across different hardware platforms.

Google Cloud: Google Cloud Platform (GCP) offers a diverse range of AI and
machine learning services, including AI Platform, AutoML, and pre-trained APIs
for tasks like vision, language, and translation. GCP’s AI Platform provides a
unified environment for building, training, and deploying AI models,
supporting TensorFlow, PyTorch, and other popular frameworks. Google Cloud
AutoML enables users with limited machine learning expertise to train custom
models using transfer learning and neural architecture search techniques.

###  1.3 Comparison of AI Frameworks and Cloud Services

When comparing AI frameworks, several factors should be considered, such as
ease of use, scalability, performance, and ecosystem. TensorFlow and PyTorch
are currently the most popular choices, with TensorFlow offering better
performance and a more mature ecosystem, while PyTorch provides greater
flexibility and a more intuitive development experience [14].

In terms of cloud services, the choice largely depends on the specific needs
and preferences of the user. AWS, Azure, and GCP each offer a comprehensive
set of AI and machine learning services, with unique strengths and weaknesses.

##  2 Data Storage and Management in Cloud-based AI Systems

Effective data storage and management are critical components of cloud-based
AI systems, as they directly impact the efficiency and performance of AI
applications. In this section, we discuss scalable data storage solutions
suitable for handling large volumes of data, techniques for data preprocessing
and feature engineering in the cloud, as well as essential considerations for
data privacy and security. By addressing these aspects, researchers and
practitioners can ensure the robustness and reliability of their AI systems
while minimizing potential risks associated with data storage and management
[5].

###  2.1 Scalable Data Storage Solutions

In cloud-based AI systems, managing large datasets efficiently is crucial for
optimal performance. Several scalable data storage solutions are available to
meet this challenge, including:

  * •

Object Storage: Services like Amazon S3, Azure Blob Storage, and Google Cloud
Storage provide highly scalable, distributed object storage that can store and
retrieve large volumes of unstructured data. These services offer low-latency
access to data, versioning capabilities, and support for various data types,
making them suitable for AI workloads.

  * •

Distributed File Systems: For use cases requiring a file-based storage
solution, distributed file systems like Hadoop Distributed File System (HDFS),
GlusterFS, and Google Cloud Filestore provide horizontal scalability, fault
tolerance, and high-throughput access to data. These systems are especially
effective for storing and processing large datasets in parallel across
multiple nodes.

  * •

NoSQL Databases: NoSQL databases like Amazon DynamoDB, Azure Cosmos DB, and
Google Cloud Datastore are designed to handle large volumes of unstructured or
semi-structured data, offering high availability, partition tolerance, and
horizontal scalability. These databases can be an effective choice for AI
applications that require real-time data access and processing.

###  2.2 Data Preprocessing and Feature Engineering in the Cloud

Before training AI models, raw data must be preprocessed and transformed into
a suitable format. Cloud-based AI systems provide several tools and services
for data preprocessing and feature engineering, such as:

  * •

Data Transformation Services: Cloud providers offer services like AWS Glue,
Azure Data Factory, and Google Cloud Dataflow that allow users to build,
orchestrate, and manage data pipelines for processing and transforming large
datasets.

  * •

Serverless Computing: Serverless platforms like AWS Lambda, Azure Functions,
and Google Cloud Functions enable users to run data preprocessing and feature
engineering tasks without managing the underlying infrastructure. These
platforms automatically scale with the workload, making them a cost-effective
solution for processing large volumes of data [7].

  * •

Distributed Data Processing Frameworks: Frameworks like Apache Spark and
Apache Flink provide distributed data processing capabilities, allowing users
to preprocess and transform large datasets efficiently using cloud-based
resources. These frameworks can be deployed on managed services like Amazon
EMR, Azure HDInsight, and Google Cloud Dataproc.

###  2.3 Data Privacy and Security Considerations

Ensuring data privacy and security is a critical aspect of cloud-based AI
systems. Some essential considerations include:

  * •

Data Encryption: Data should be encrypted both at rest and in transit to
protect it from unauthorized access. Cloud providers offer various encryption
options, including server-side encryption, client-side encryption, and key
management services.

  * •

Access Control: Implementing fine-grained access control policies ensures that
only authorized users can access and manipulate data. Cloud providers offer
tools like AWS Identity and Access Management (IAM), Azure Active Directory,
and Google Cloud IAM for managing access control.

  * •

Data Residency and Compliance: To comply with data protection regulations,
organizations may need to store data in specific geographical locations or
meet certain security requirements. Cloud providers offer data residency
options and compliance certifications to address these concerns.

Regular Auditing and Monitoring: Continuous monitoring and auditing of data
access and usage patterns can help detect and prevent potential security
threats. Cloud providers offer services like AWS CloudTrail, Azure Monitor,
and Google Cloud Logging for this purpose [10].

##  3 Parallel and Distributed Training of AI Models

As AI models become increasingly complex and require larger volumes of data
for training, parallel and distributed training techniques have emerged as
essential approaches to reduce training time and improve resource utilization.
In this section, we provide an overview of parallel and distributed training
techniques, discuss efficient model partitioning and communication strategies,
and explore cloud-based training architectures and tools that facilitate the
scalable and efficient training of AI models. By leveraging these techniques,
researchers and practitioners can accelerate the training process, enhance
model performance, and reduce costs associated with cloud-based AI systems
[1].

###  3.1 Overview of Parallel and Distributed Training Techniques

Parallel and distributed training techniques can be broadly categorized into
the following approaches:

  * •

Data Parallelism: In data parallelism, the training dataset is divided into
smaller subsets, and multiple copies of the model are trained independently on
these subsets using different computing resources. The model updates are
periodically synchronized across all replicas to maintain consistency. Data
parallelism is particularly suitable for large datasets and can be implemented
using techniques like mini-batch gradient descent and asynchronous stochastic
gradient descent (ASGD).

  * •

Model Parallelism: Model parallelism involves splitting the model itself
across multiple computing resources, with each resource responsible for a
portion of the model’s computation. This approach is especially useful for
training large models that do not fit within the memory constraints of a
single computing device. Model parallelism can be implemented using techniques
like pipelining and tensor partitioning.

  * •

Hybrid Parallelism: Hybrid parallelism combines aspects of both data and model
parallelism to further optimize the training process. In this approach, the
model is partitioned across multiple computing resources, and the dataset is
divided into smaller subsets that are processed independently. This technique
is useful for training large models on large datasets while maintaining a
balance between computation and communication overhead.

###  3.2 Efficient Model Partitioning and Communication Strategies

To maximize the benefits of parallel and distributed training, it is crucial
to efficiently partition the model and implement effective communication
strategies. Some key considerations include:

  * •

Load Balancing: To ensure even distribution of computational workloads, the
model should be partitioned in such a way that each computing resource has an
approximately equal amount of computation to perform. This balance minimizes
idle time and improves overall training efficiency.

  * •

Communication Overhead: During the training process, model updates and
gradients must be communicated between computing resources. To reduce the
communication overhead, it is important to minimize the amount of data
transferred between resources and to utilize efficient communication
protocols, such as the Message Passing Interface (MPI) and NVIDIA’s NCCL.

  * •

Fault Tolerance: In a distributed training setup, the failure of a single
computing resource can potentially disrupt the entire training process.
Implementing fault tolerance mechanisms, such as checkpointing and model
replication, can help minimize the impact of resource failures and ensure the
reliability of the training process.

###  3.3 Cloud-based Training Architectures and Tools

Cloud providers offer various tools and services to support parallel and
distributed training of AI models, including:

  * •

Managed Machine Learning Platforms: Services like AWS SageMaker, Azure Machine
Learning, and Google AI Platform provide managed environments for building,
training, and deploying AI models. These platforms support distributed
training out-of-the-box, allowing users to easily scale their training
workloads across multiple computing resources.

  * •

Cluster Orchestration Tools: Tools like Kubernetes, Apache Mesos, and Amazon
Elastic Kubernetes Service (EKS) enable users to create and manage clusters of
computing resources for distributed training. These tools provide features
like automatic scaling, load balancing, and fault tolerance, simplifying the
management of distributed training workloads.

  * •

Distributed Training Libraries: Libraries like Horovod, TensorFlow’s
Distribution Strategies, and PyTorch’s Distributed Data Parallel module
provide abstractions and APIs for implementing parallel and distributed
training techniques in AI frameworks. These libraries facilitate efficient
model partitioning, communication, and synchronization,allowing developers to
focus on their AI models and applications without the need to handle low-level
implementation details.

  * •

Custom Training Architectures: For specific use cases and advanced
requirements, users can design custom training architectures using cloud-based
infrastructure and services. For example, they can utilize cloud-based virtual
machines, GPUs, and TPUs along with storage and networking services to build
tailored solutions for their distributed training needs. This approach offers
maximum flexibility but requires more in-depth knowledge of cloud
infrastructure and parallel computing techniques.

In summary, parallel and distributed training techniques have become
indispensable for training complex AI models on large datasets efficiently
[9]. By understanding the various approaches and leveraging cloud-based tools
and services, researchers and practitioners can accelerate the training
process, optimize resource utilization, and reduce costs associated with
cloud-based AI systems. As these techniques continue to evolve, further
advancements in AI model training and deployment are expected, paving the way
for the development of even more powerful and sophisticated AI applications
[1] [12].

##  4 Optimizing AI Workloads in the Cloud

Efficiently managing and optimizing AI workloads in the cloud is crucial for
ensuring high performance, cost-effectiveness, and resource utilization. In
this chapter, we discuss various strategies for load balancing and resource
allocation, explore auto-scaling and dynamic resource provisioning techniques,
and examine performance benchmarking and optimization approaches to improve
the overall efficiency of cloud-based AI systems. By adopting these strategies
and techniques, researchers and practitioners can significantly enhance the
performance of their AI applications while minimizing the associated costs and
complexities of managing cloud-based resources [13] [2].

###  4.1 Load Balancing and Resource Allocation Strategies

Effective load balancing and resource allocation strategies are critical for
optimizing the performance of cloud-based AI systems. Some key techniques and
considerations include:

  * •

Horizontal Scaling: This strategy involves adding or removing computing
resources, such as virtual machines or containers, to balance the workload and
ensure optimal resource utilization. Horizontal scaling can be achieved using
cloud provider-specific tools, such as AWS Auto Scaling Groups, Azure Virtual
Machine Scale Sets, and Google Cloud Instance Groups.

  * •

Vertical Scaling: Vertical scaling involves increasing or decreasing the
computing capacity of individual resources, such as CPU, memory, or GPU. This
strategy can be useful for optimizing resource utilization for specific
workloads but may be limited by the maximum capacity of individual resources.

  * •

Resource Allocation Policies: Implementing resource allocation policies, such
as CPU and memory quotas, can help ensure fair distribution of resources among
multiple AI workloads and prevent resource contention. Cloud providers offer
tools like AWS Resource Groups, Azure Resource Manager, and Google Cloud
Resource Manager to define and enforce such policies.

###  4.2 Auto-scaling and Dynamic Resource Provisioning

Auto-scaling and dynamic resource provisioning techniques enable cloud-based
AI systems to automatically adjust resource allocation based on workload
demands, thereby improving efficiency and reducing costs. Some key approaches
include:

  * •

Reactive Auto-scaling: Reactive auto-scaling involves monitoring system
metrics, such as CPU utilization, memory usage, or request latency, and
adjusting resource allocation based on predefined thresholds. Cloud providers
offer tools like AWS Auto Scaling, Azure Autoscale, and Google Cloud Auto-
scaler to implement reactive auto-scaling policies.

  * •

Predictive Auto-scaling: Predictive auto-scaling uses machine learning
algorithms to analyze historical workload patterns and predict future resource
demands. This approach enables proactive resource provisioning, allowing AI
systems to scale resources ahead of demand spikes. AWS Forecast, Azure Machine
Learning, and Google Cloud AI Platform can be utilized to implement predictive
auto-scaling.

  * •

Serverless Computing: Serverless computing platforms, such as AWS Lambda,
Azure Functions, and Google Cloud Functions, automatically scale resources
based on workload demands, eliminating the need for manual resource
provisioning and management. These platforms are particularly suitable for
event-driven AI workloads, such as data processing and inference tasks [13]
[2].

###  4.3 Performance Benchmarking and Optimization Techniques

Benchmarking and optimization techniques are essential for identifying
performance bottlenecks and improving the efficiency of cloud-based AI
systems. Some key approaches include:

  * •

Performance Profiling: Profiling tools, such as TensorFlow Profiler, PyTorch
Profiler, and NVIDIA Nsight, can be used to measure and analyze the
performance of AI workloads, providing insights into computational bottlenecks
and resource utilization patterns.

  * •

Model Optimization: Techniques like quantization, pruning, and knowledge
distillation can be used to reduce the size and complexity of AI models,
improving inference speed and reducing resource requirements.

  * •

Hardware Acceleration: Utilizing hardware accelerators, such as GPUs, TPUs,
and FPGAs, can significantly improve the performance of AI workloads. Cloud
providers offer various instance types and services, such as AWS EC2 GPU
Instances, Azure GPU VMs, and Google Cloud AI Platform, that support hardware
acceleration.

In conclusion, optimizing AI workloads in the cloud is essential for achieving
high performance, cost-effectiveness, and efficient resource utilization. By
employing effective load balancing and resource allocation strategies,
leveraging auto-scaling and dynamic resource provisioning techniques, and
utilizing performance benchmarking and optimization approaches, researchers
and practitioners can significantly enhance their cloud-based AI systems. As
the adoption of AI and cloud computing continues to grow, further advancements
in optimization techniques and tools are expected, enabling even more powerful
and efficient AI applications in the cloud [15].

##  5 AI Model Deployment and Serving in the Cloud

Once AI models have been trained and optimized, deploying and serving them
efficiently in the cloud is essential for delivering scalable, high-
performance AI applications. In this section, we discuss the process of model
packaging and containerization, explore serverless deployment options and
microservices architecture, and examine monitoring, logging, and versioning
techniques for managing deployed models. By leveraging these methodologies and
tools, researchers and practitioners can streamline the deployment process,
ensure the reliability and performance of their AI applications, and easily
maintain and update their models in production environments.

###  5.1 Model Packaging and Containerization

Proper model packaging and containerization are crucial for deploying AI
models in the cloud. These processes ensure that models are portable, easily
deployable, and can be served consistently across different environments. Key
aspects of model packaging and containerization include:

  * •

Model Serialization: AI models must be serialized, or converted into a format
that can be easily stored, transferred, and loaded for inference. Common
serialization formats include TensorFlow SavedModel, PyTorch TorchScript, and
ONNX (Open Neural Network Exchange).

  * •

Containerization: Containerization involves packaging the AI model along with
its dependencies, runtime environment, and configuration files into a
lightweight, portable container. Containers provide isolation,
reproducibility, and consistency across different deployment environments.
Docker and container orchestration tools like Kubernetes are widely used for
this purpose.

  * •

Model Serving Frameworks: Model serving frameworks, such as TensorFlow
Serving, NVIDIA Triton Inference Server, and PyTorch Serve, provide the
necessary infrastructure and APIs for deploying and serving AI models
efficiently. These frameworks handle tasks like model loading, request
processing, and output generation, simplifying the deployment process.

###  5.2 Serverless Deployment Options and Microservices Architecture

Serverless deployment options and microservices architecture provide scalable
and cost-effective ways to deploy and serve AI models in the cloud. Key
features of these approaches include:

  * •

Serverless Deployment: Serverless platforms, such as AWS Lambda, Azure
Functions, and Google Cloud Functions, enable AI models to be deployed and
served without the need to manage the underlying infrastructure. These
platforms automatically scale resources based on workload demands and provide
built-in fault tolerance and load balancing.

  * •

Microservices Architecture: In a microservices architecture, AI models are
deployed as independent, loosely-coupled services that can be developed,
scaled, and updated independently [6]. This approach allows for greater
flexibility, agility, and scalability in deploying and managing AI models in
production environments.

  * •

API Gateway: An API gateway is used to manage and route incoming requests to
the appropriate AI model services. API gateways, such as AWS API Gateway,
Azure API Management, and Google Cloud API Gateway, provide features like
request throttling, caching, and authentication, ensuring secure and efficient
access to AI models [11].

###  5.3 Monitoring, Logging, and Versioning of Deployed Models

Monitoring, logging, and versioning are essential for managing and maintaining
deployed AI models in the cloud. Key aspects of these processes include:

  * •

Monitoring: Monitoring tools, such as AWS CloudWatch, Azure Monitor, and
Google Cloud Monitoring, provide insights into the performance, resource
utilization, and health of deployed AI models. These tools enable users to
identify and troubleshoot issues, optimize resource usage, and maintain high
availability.

  * •

Logging: Logging tools, such as AWS CloudTrail, Azure Log Analytics, and
Google Cloud Logging, collect and store log data generated by AI models during
inference. Log data can be used for troubleshooting, auditing, and
understanding usage patterns of deployed models.

  * •

Model Versioning: Versioning deployed AI models allows for tracking changes,
rolling back to previous versions, and serving multiple versions
simultaneously. Model versioning can be implemented using model serving
frameworks, container registries, or cloud-based storage services.

In conclusion, efficient AI model deployment and serving in the cloud are
critical for delivering scalable, high-performance AI applications. By
adopting best practices for model packaging and containerization, leveraging
serverless deployment options and microservices architecture, and employing
monitoring, logging, and versioning techniques, researchers and practitioners
can ensure the reliability and performance of their AI applications in
production environments. As AI and cloud computing technologies continue to
advance, we can expect further improvements in deployment and serving
methodologies, enabling even more powerful and efficient AI applications in
the cloud [4].

##  6 Conclusion and Future Directions

In this paper, we have discussed various aspects of AI and cloud computing,
including AI frameworks and cloud services, data storage and management,
parallel and distributed training techniques, optimizing AI workloads, and
model deployment and serving. By understanding and leveraging these concepts,
researchers and practitioners can build, train, and deploy powerful AI
applications in the cloud while maximizing efficiency, performance, and cost-
effectiveness.

As AI and cloud computing technologies continue to advance, we anticipate
several future directions that will further shape the landscape of AI in the
cloud. Some potential developments include:

  * •

Improved AI Frameworks and Services: We expect to see ongoing enhancements in
AI frameworks and cloud services, with better support for distributed
training, hardware acceleration, and model optimization. These improvements
will enable researchers and practitioners to develop more sophisticated AI
models and applications in the cloud.

  * •

Advanced Auto-scaling Techniques: As AI workloads become more complex and
dynamic, advanced auto-scaling techniques will be needed to optimize resource
allocation and performance. We anticipate the development of more intelligent,
machine learning-based auto-scaling methods that can better predict and adapt
to changing workload demands.

  * •

Enhanced Security and Privacy: As AI and cloud computing become more
prevalent, ensuring data security and privacy will be increasingly important.
We expect to see further developments in techniques like federated learning,
homomorphic encryption, and differential privacy that will enable secure and
privacy-preserving AI in the cloud.

  * •

Edge AI and Cloud Integration: The integration of edge computing with cloud-
based AI systems is an emerging trend that will enable low-latency, real-time
AI applications in various domains, such as IoT, autonomous vehicles, and
smart cities. We foresee advancements in edge AI and cloud integration, which
will lead to more efficient and scalable AI solutions that span across the
cloud and edge devices.

  * •

Green AI and Energy-efficient Computing: As the environmental impact of AI and
cloud computing becomes more apparent, there will be a growing emphasis on
green AI and energy-efficient computing techniques. This includes research
into reducing the energy consumption of AI training and inference processes,
as well as the development of more energy-efficient hardware and cloud
infrastructure.

In conclusion, AI and cloud computing are rapidly evolving fields that offer
immense potential for developing powerful and efficient AI applications. By
staying abreast of the latest advancements and trends, researchers and
practitioners can harness the full potential of AI in the cloud, enabling the
development of cutting-edge solutions that drive innovation and improve our
world.

## References

  * [1] S. Akter, K. Michael, M. R. Uddin, G. McCarthy, and M. Rahman.  Transforming business using digital innovations: The application of ai, blockchain, cloud and data analytics.  Annals of Operations Research, pp. 1–33, 2022.
  * [2] F. Al-Turjman.  Ai-powered cloud for covid-19 and other infectious disease diagnosis.  Personal and Ubiquitous Computing, pp. 1–4, 2021.
  * [3] J. Dodge, T. Prewitt, R. Tachet des Combes, E. Odmark, R. Schwartz, E. Strubell, A. S. Luccioni, N. A. Smith, N. DeCario, and W. Buchanan.  Measuring the carbon intensity of ai in cloud instances.  In 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 1877–1894, 2022.
  * [4] V. Eramo, F. G. Lavacca, T. Catena, and P. J. Perez Salazar.  Proposal and investigation of an artificial intelligence (ai)-based cloud resource allocation algorithm in network function virtualization architectures.  Future Internet, 12(11):196, 2020.
  * [5] S. S. Gill, S. Tuli, M. Xu, I. Singh, K. V. Singh, D. Lindsay, S. Tuli, D. Smirnova, M. Singh, U. Jain, et al.  Transformative effects of iot, blockchain and artificial intelligence on cloud computing: Evolution, vision, trends and open challenges.  Internet of Things, 8:100118, 2019.
  * [6] L. Ionescu et al.  Big data, blockchain, and artificial intelligence in cloud-based accounting information systems.  Analysis and Metaphysics, (18):44–49, 2019.
  * [7] S. Juyal, S. Sharma, and A. S. Shukla.  Smart skin health monitoring using ai-enabled cloud-based iot.  Materials Today: Proceedings, 46:10539–10545, 2021.
  * [8] A. Kaginalkar, S. Kumar, P. Gargava, and D. Niyogi.  Review of urban computing in air quality management as smart city service: An integrated iot, ai, and cloud technology perspective.  Urban Climate, 39:100972, 2021.
  * [9] T. Kurihana, E. J. Moyer, and I. T. Foster.  Aicca: Ai-driven cloud classification atlas.  Remote Sensing, 14(22):5690, 2022.
  * [10] M. Li, Z. Sun, Z. Jiang, Z. Tan, and J. Chen.  A virtual reality platform for safety training in coal mines with ai and cloud computing.  Discrete Dynamics in Nature and Society, 2020:1–7, 2020.
  * [11] K. N. Qureshi, G. Jeon, and F. Piccialli.  Anomaly detection and trust authority in artificial intelligence and cloud computing.  Computer Networks, 184:107647, 2021.
  * [12] A. Salem and O. Moselhi.  Ai-based cloud computing application for smart earthmoving operations.  Canadian Journal of Civil Engineering, 48(3):312–327, 2021.
  * [13] K. K. Singh.  An artificial intelligence and cloud based collaborative platform for plant disease identification, tracking and forecasting for farmers.  In 2018 IEEE international conference on cloud computing in emerging markets (CCEM), pp. 49–56. IEEE, 2018.
  * [14] J. Wan, J. Yang, Z. Wang, and Q. Hua.  Artificial intelligence for cloud-assisted smart factory.  IEEE Access, 6:55419–55430, 2018.
  * [15] Z. Wang, Z. Zhou, H. Zhang, G. Zhang, H. Ding, and A. Farouk.  Ai-based cloud-edge-device collaboration in 6g space-air-ground integrated power iot.  IEEE Wireless Communications, 29(1):16–23, 2022.


# KnFu: Effective Knowledge Fusion††thanks: This work was partially supported
by the Natural Sciences and Engineering Research Council (NSERC) of Canada
through the NSERC Discovery Grant RGPIN-2023-05654.

S. Jamal Seyedmohammadi1, S. Kawa Atapour 2, Jamshid Abouei2, Arash Mohammadi1
1 Concordia Institute of Information Systems Engineering (CIISE), Concordia
University, Montreal, Canada  2 Dept. of Electrical Engineering, Yazd
University, Yazd, Iran

###### Abstract

Federated Learning (FL) has emerged as a prominent alternative to the
traditional centralized learning approach, attracting significant interest
across a wide range of practical applications. Generally speaking, FL is a
decentralized approach that allows for collaborative training of Machine
Learning (ML) models across multiple local nodes, ensuring data privacy and
security while leveraging diverse datasets. Conventional FL, however, is
susceptible to gradient inversion attacks, restrictively enforces a uniform
architecture on local models, and suffers from model heterogeneity (model
drift) due to non-IID local datasets. To mitigate some of these challenges,
the new paradigm of Federated Knowledge Distillation (FKD) has emerged. FDK is
developed based on the concept of Knowledge Distillation (KD), which involves
extraction and transfer of a large and well-trained teacher model’s knowledge
to lightweight student models. FKD, however, still faces the model drift
issue. Intuitively speaking, not all knowledge is universally beneficial due
to the inherent diversity of data among local nodes. This calls for innovative
mechanisms to evaluate the relevance and effectiveness of each client’s
knowledge for others, to prevent propagation of adverse knowledge. In this
context, the paper proposes Effective Knowledge Fusion (KnFu) algorithm that
evaluates knowledge of local models to only fuse semantic neighbors’ effective
knowledge for each client. The KnFu is a personalized effective knowledge
fusion scheme for each client, that analyzes effectiveness of different local
models’ knowledge prior to the aggregation phase. Comprehensive experiments
were performed on MNIST and CIFAR10 datasets illustrating effectiveness of the
proposed KnFu in comparison to its state-of-the-art counterparts. A key
conclusion of the work is that in scenarios with large and highly
heterogeneous local datasets, local training could be preferable to knowledge
fusion-based solutions.

###### Index Terms:

Personalized Federated Learning, Clustered Knowledge Distillation, Selective
Knowledge Distillation

##  I Introduction

Federated Learning (FL) has recently gained considerable attention, as an
alternative to the centralized learning paradigm, in various domains including
but not limited to computer vision, healthcare, and natural language
processing. Generally speaking, FL resolves some practical challenges of
centralized learning frameworks such as users’ privacy issues and the
communication cost of transmitting raw data from users/silos to the Fusion
Centre (FC). Conventional FL methods aim to collaboratively train a global
model by aggregating parameters of the clients’ models without sharing their
private data [1]. Such methods, however, pose the following new challenging
problems: (i) Privacy concerns arising from gradient inversion attacks; (ii)
Communication overhead of iterative transmission/reception of model
parameters; (iii) Enforcing a uniform model architecture on clients, and; (iv)
Model heterogeneity (model drift) resulted from non-IID local datasets [2].

To mitigate some of the above mentioned challenges of conventional FL
solutions, the new paradigm of Federated Knowledge Distillation (FKD) [3] has
been introduced that integrates the concept of Knowledge Distillation (KD)
with FL. KD involves extracting the knowledge of a large and well-trained
teacher model and transferring it to a lightweight student model by mimicking
the teacher’s predictions on a transfer set. In FKD, clients share only their
local knowledge, i.e., predictions on the transfer set, with the server rather
than their local model parameters. This leads to a more privacy-preserving
framework, reduced communication overhead, and allowing heterogeneous model
architectures among clients. While FKD presents effective advantages to
resolve conventional FL’s problems, it poses some new difficulties, including:
(i) Requiring a transfer set to extract local knowledge of clients, and; (ii)
Imposing computation overhead on local devices. Additionally, the model drift
issue still remains as an open challenge. Since clients hold non-IID local
datasets, the models trained locally would be heterogeneous, resulting in non-
IID local knowledge among clients. Therefore, aggregating the local knowledge
of a specific client with that of other clients may lead to adverse impacts on
the client’s local model resulting in significant performance degradation.
Consequently, there has been a surge of recent interest devising innovative
solutions to alleviate these issues [4, 5, 6, 7, 8, 9, 10]. This field,
however, is still in its infancy. The paper aims to further advance the
research in this domain.

Related Works: In [4] an adaptive KD approach is proposed, inspired by
multitask learning methods, to adaptively adjust the weight of different
distillation paths of an ensemble of teachers. Such an approach prevents
negative impacts of some paths on the generalization performance of the
student models. Reference [5] studied whether all or partial knowledge of a
model is effective. A generic knowledge selection method is presented to
select and distill only certain knowledge by either fixing the knowledge
selection threshold or changing it progressively during the training process
as the teacher’s confidence is enhanced. A selective knowledge-sharing
mechanism is proposed in [6] to address the misleading and ambiguous knowledge
fusion challenge resulting from non-IID local datasets and absence of a well-
trained teacher model. The client-side selector chooses accurate predictions
that match the ground-truth labels. Meanwhile, the server-side selector
identifies the precise prediction by their entropy values. Precise knowledge
has low entropy, while ambiguous predictions have high entropy and
uncertainty.

Reference [7] analyzed the effect of local predicted logits on the convergence
rate. To improve the convergence rate, a knowledge selection method is
proposed to schedule the predicted logits for efficient knowledge aggregation.
In addition, a threshold-based approach is presented to optimize the local
model updating options with/without knowledge distillation for each edge
device to reduce the performance degradation of local models resulted from
ambiguous knowledge. The COMET approach is proposed in [8] introducing the
clustered knowledge distillation concept, i.e., forming localized clusters
from clients with similar data distribution. Each client then uses the
aggregated knowledge of its cluster rather than following the average logits
of all clients. Such an approach prevents performance degradation by learning
from clients with considerably different data distributions. In the local
updating phase of each client, the loss function comprises a cross-entropy
function along with a regularization term, which is an $l_{2}$-norm between
the local and average predictions of the corresponding cluster.

KT-pFL [9] proposed a personalized group knowledge distillation algorithm,
updating the personalized soft prediction of each client through a linear
combination of all local predictions by a knowledge coefficient matrix. This
matrix adaptively adjusts the collaboration among clients with similar data
distribution and is parameterized to be trained simultaneously with the
models. MetaFed [10] presents a trustworthy personalized FL that achieves a
personalized model for each federation without a central server using cyclic
knowledge distillation. Its training process is split into two parts: common
knowledge accumulation and personalization. In the first part, it leverages
the validation accuracy on the current federation’s validation data to decide
whether to completely keep the previous federation’s knowledge and fine-tune
it or just use it to update the current federation’s through KD. In the
personalization part, if the common model does not have enough performance on
the validation data of the current federation, it refers little to it, while
the weight of the KD regularization term is adapted if the common model’s
performance is acceptable on the current validation data.

Contributions: The above mentioned works mainly focused to effectively distill
knowledge of the teacher(s) into student model(s). In other words, the non-IID
nature of local datasets has not yet been effectively addressed. Additionally,
effectiveness of local knowledge of clients has not yet been investigated. The
paper addresses these gaps, by development of a more efficacious knowledge
fusion technique, aiming to present more thorough evaluation on the
effectiveness of the local knowledge of clients. Our main contributions can be
summarized as follows:

  * •

Proposal of the KnFu algorithm that strategically evaluates and fuses only
relevant and beneficial knowledge among clients. This personalized approach
ensures that knowledge fusion is tailored to the semantic neighbors of each
client, mitigating the risk of model drift caused by non-IID local datasets.

  * •

Introduction of a novel mechanism within the KnFu algorithm to assess the
relevance and impact of shared knowledge across clients, ensuring that only
effective knowledge contributes to the FL process, thereby preventing the
dilution of model performance with non-contributory information.

Comprehensive experiments were performed on MNIST, and CIFAR10 datasets to
show the effectiveness of the proposed algorithm in comparison with baseline
methods in terms of different metrics. The rest of the paper is organized as
follows: Section II formulates the problem and provides required background
material to follow developments of the papers. The KnFu algorithm is proposed
in Section III, while Section IV presents simulation results and analysis.
Finally, Section V concludes the paper.

##  II Preliminaries and Problem Statement

(a) Class Distribution

(b) EPD

Figure 1: The EPD as an estimation of the class distribution.

(a) $\alpha=0.1$

(b) $\alpha=0.5$

(c) $\alpha=1$

Figure 2: Illustration of data heterogeneity among $10$ clients on the
CIFAR-10 dataset.

In this paper, we aim to perform a supervised $C$-class classification task.
Let’s consider a set of $N$ clients, denoted by
$\mathbb{U}=\\{u_{1},...,u_{N}\\}$, in the FL system coordinated by a fusion
center. Local datasets of clients, represented by
$\mathbb{D}_{n}=\bigcup_{i=1}^{K_{n}}\\{(x_{n}^{i},y_{n}^{i})\\},n\in\\{1,...,N\\}$,
are heterogeneous, where $(x_{n}^{i},y_{n}^{i})$ denotes $i^{th}$ data sample
including the input and its ground-truth output, and $K_{n}$ indicates the
size of the local dataset $\mathbb{D}_{n}$. Each client $u_{n}$ aims to train
a local Convolutional Neural Network (CNN) model, denoted by
$f(\cdot;\bm{\Omega}_{n})$, parameterized by $\bm{\Omega}_{n}$. Let
$f(x;\bm{\Omega})=[f_{1}(x;\bm{\Omega}),...,f_{C}(x;\bm{\Omega})]$ denote the
output of the last layer (softmax) of the CNN model, where
$\sum_{j=1}^{C}f_{j}(x;\bm{\Omega})=1$, and $f_{j}(x;\bm{\Omega})$ indicates
the probability of assigning data sample $\bm{x}$ to the $j^{th}$ class.

Term $\bm{p}(\bm{\Omega})=[p_{1}(\bm{\Omega}),...,p_{C}(\bm{\Omega})]$ is
defined as the Estimated Probability Distribution (EPD) of assigning new data
samples to different classes. EPD is calculated over a shared dataset among
clients, the transfer set, with $K_{t}$ data samples, denoted by
$\mathcal{D}^{t}=\bigcup_{i=1}^{K_{t}}\\{(x_{t}^{i},y_{t}^{i})\\}$. To
investigate the local model’s bias to different classes, EPD can be computed
as the expectation of probability distribution of the local models on the data
samples of the transfer set, as follows

$\bm{p}(\bm{\Omega})=\sum_{x\in\mathcal{D}^{t}}\frac{f(x;\bm{\Omega})}{|\mathcal{D}^{t}|}.$
(1)

KD involves extracting and transferring knowledge from a well-trained teacher
model into a student model, emulating the outputs of the teacher model using a
transfer set. Specifically, a Kullback-Leibler (KL) divergence function [11]
is utilized to minimize the discrepancy between the soft labels of the teacher
model and the student model, as follows

$\displaystyle\bm{\Omega}^{*}_{s}$
$\displaystyle=\operatorname*{argmin}_{\bm{\Omega}_{s}}\mathbb{E}_{(x,y)\sim\mathcal{D}^{t}}\bigg{\\{}\mathcal{L}_{CE}(f(x;\bm{\Omega}_{s}),y)$
(2)
$\displaystyle+\lambda^{2}\mathcal{L}_{KL}(f(x;\bm{\Omega}_{s}),f(x;\bm{\Omega}_{t}))\bigg{\\}},$

where $f(x;\bm{\Omega}_{s})$ and $f(x;\bm{\Omega}_{t})$ denote the predictions
of the student and teacher models for input $x$, respectively. In addition,
$\mathcal{L}_{CE}$ and $\mathcal{L}_{KL}$ are the cross-entropy and KL loss
functions, and $\lambda$ is the so-called temperature hyper-parameter used to
soften generated logits. In [3], clients first update their local models using
their respective local datasets, then, each client performs predictions on the
transfer set to extract a set of soft labels, known as local knowledge. These
local soft labels are averaged in the fusion center to fuse the local
knowledge of the clients. Finally, the average soft labels, known as
collaborative knowledge, are utilized in Eq (2) to distill it into the local
models. In [3] and similar papers, the local knowledge of all clients is
aggregated and distilled to each local model. The local models are, however,
heterogeneous due to the non-IID nature of local datasets. Consequently, their
local knowledge is also heterogeneous, i.e., local knowledge of a specific
client may not be effective for all other clients in the FL system.

To address the above mentioned issue, in this paper, we aim to answer the
followings questions: Q1: How can we assess the effectiveness of a specific
client’s local knowledge for other clients? Q2: How can we transfer only the
effective knowledge and ignore the adverse knowledge of local models? To
answer the first question (Q1), we assess the effectiveness of knowledge-
sharing and local training options based on two important factors: $(i)$ Data
heterogeneity level, and; $(ii)$ Local dataset size. To answer the second
question, we propose the innovative Effective Knowledge Fusion (KnFu)
algorithm that evaluates knowledge of local models and fuse semantic
neighbors’ (i.e., clients with similar data distributions) effective knowledge
for each client.

##  III The Proposed KnFu Algorithm

In this section, we present the proposed KnFu algorithm that effectively
combines useful knowledge of various clients within an FL framework,
ultimately leading to effective personalized local models. Given the inherent
diversity of data among clients, it is crucial to acknowledge that not all
knowledge is universally beneficial. Hence, there is a necessity for a
mechanism to evaluate the relevance and effectiveness of each client’s
knowledge for others, therefore, preventing the propagation of adverse
knowledge. Moreover, crafting an efficient methodology to distill useful
knowledge into specific local models is challenging. The proposed KnFu
algorithm, consisting of four primary steps, operates over $R$ rounds or until
convergence is achieved.

Step 1: Local Training: Initially, individual models undergo training on their
local datasets for a set number of local epochs, denoted by $E$, as follows

$\bm{\Omega}_{n}^{*}=\operatorname*{argmin}_{\bm{\Omega}_{n}}\mathbb{E}_{(x,y)\sim\mathbb{D}_{n}}\\{\mathcal{L}_{CE}(f(x;\bm{\Omega}_{n}),y)\\}.$
(3)

Step 2: Local Knowledge Extraction: Following the update of local models,
their knowledge is extracted via the transfer set. This extraction involves
obtaining soft labels from each local model for the data samples within the
transfer set, i.e.,

$\bm{F}_{n}=f(x;\bm{\Omega}_{n}),\forall x\in\mathcal{D}^{t},$ (4)

where $\bm{F}_{n}$ constitutes a matrix where each row corresponds to a data
sample within the transfer set. The columns of this matrix show the
probability distribution for assigning a particular data sample to various
classes.

Step 3: Effective Knowledge Fusion: The knowledge extracted from individual
clients is transferred to the fusion center for aggregation, leading to the
creation of personalized fused knowledge for each client. Initially, we
calculate the Estimated Probability Distributions (EPDs) for clients, which
indicate the bias of their local models towards various classes. As
illustrated in Fig. 1, EPD serves as an estimation of the class distribution
within each client’s dataset, and is

$\displaystyle\bm{p}(\bm{\Omega}_{n})=\frac{1}{K_{t}}\sum_{i=1}^{K_{t}}\bm{F}_{n}^{i,j},\forall
n\in\\{1,\ldots,N\\},$ (5)

where $\bm{F}_{n}^{i,j}$ indicates the probability of assigning the $i^{th}$
data sample in the transfer set to the $j^{th}$ class using the local model of
user $u_{n}$. Next, we adjust the weighting of each client’s local knowledge
for a specific client according to how similar their EPDs are to that of the
client. We measure the distance between two distributions using KL divergence,
calculated as

$d_{n,m}=KL\Big{(}\bm{p}(\bm{\Omega}_{n}),\bm{p}(\bm{\Omega}_{m})\Big{)},\forall
n,m\in\\{1,...,N\\}.$ (6)

To determine the importance of the knowledge from client $u_{m}$ for the
client $u_{n}$, we evaluate the similarity between their EPDs by taking the
inverse of the squared distance calculated in Eq. (6), as follows

$w_{n,m}=\frac{1}{\Big{(}d_{n,m}\Big{)}^{2}},\forall n,m\in\\{1,\ldots,N\\}.$
(7)

Notably, the weight of the local knowledge is adjusted by a positive constant
$\beta$, as

$w_{n,n}=\beta\times\max\\{w_{n,m}\\}_{m=1}^{N}.$ (8)

Finally, the personalized aggregated knowledge for client $u_{n}$ is
calculated as

$\bm{F}_{n}^{agg}=\sum_{m=1}^{N}\frac{w_{n,m}}{\sum_{m}w_{n,m}}\times\bm{F}_{m},\forall
n\in\\{1,...,N\\}.$ (9)

Step 4: Local Model Fine-tuning: In the final step, the personalized fused
knowledge is distributed to clients, allowing them to integrate effective
knowledge from other clients into their local models. Clients refine their
local models by incorporating the aggregated knowledge alongside their local
datasets during fine-tuning via the transfer set, as follows

$\displaystyle\bm{\Omega}^{*}_{n}$
$\displaystyle=\operatorname*{argmin}_{\bm{\Omega}_{n}}\mathbb{E}_{(x,y)\sim\mathcal{D}^{t}}\bigg{\\{}\mathcal{L}_{CE}(f(x;\bm{\Omega}_{n}),y)$
(10)
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}+\lambda^{2}\mathcal{L}_{KL}(f(x;\bm{\Omega}_{n}),\bm{F}_{n}^{agg})\bigg{\\}},$

where $\mathcal{L}_{KL}$ denotes the KL loss function and $\lambda$ adjusts
the balance between the two terms of the loss function. The first term focuses
only on the local dataset, while the second term concentrates on the
aggregated knowledge of clients. This completes description of the proposed
KnFu, next we present our simulation results and analysis.

Algorithm 1 Pseudocode of the proposed KnFu algorithm

Input: Local datasets $\\{\mathbb{D}_{n}\\}_{n=1}^{N}$ and transfer set
$\mathcal{D}^{t}$.

for $r=1,\dots,R$ do

## Local Training

for $u_{n}\in\mathbb{U}$ do

$\bm{\Omega}_{n}^{*}=\operatorname*{argmin}_{\bm{\Omega}_{n}}\mathbb{E}_{(x,y)\sim\mathbb{D}_{n}}\\{\mathcal{L}_{CE}(f(x;\bm{\Omega}_{n}),y)\\}$

end for

## Local Knowledge Extraction

for $u_{n}\in\mathbb{U}$ do

$\bm{F}_{n}=f(x;\bm{\Omega}_{n}),\forall x\in\mathcal{D}^{t}$

end for

——————————————-FUSION CENTER——–

## Effective Knowledge Fusion

for $u_{n}\in\mathbb{U}$ do

## estimation of class distribution (EPD)

$\bm{p}(\bm{\Omega_{n}})=\frac{1}{K_{t}}\sum_{i=1}^{K_{t}}\bm{F}_{n}^{i,j}$

## dissimilarities between EPDs

$d_{n,m}=KL\Big{(}\bm{p}(\bm{\Omega}_{n}),\bm{p}(\bm{\Omega}_{m})\Big{)}$

## personalized knowledge fusion

$\bm{F}_{n}^{agg}=\sum_{m=1}^{N}\frac{w_{n,m}}{\sum_{m}w_{n,m}}\times\bm{F}_{m}$

end for

————————————————————————

## Local Model Fine-tuning

for $u_{n}\in\mathbb{U}$ do

Eq. (10)

end for

end for

Output: Personalized local models $\\{\bm{\Omega}_{n}\\}_{n=1}^{N}$

(a) $|\mathbb{D}_{n}|=50$

(b) $|\mathbb{D}_{n}|=100$

(c) $|\mathbb{D}_{n}|=200$

Figure 3: Learning curves of the ALMA metric corresponding to different
methods, for a fixed heterogeneity level, $\alpha=0.5$, and different local
data sizes, $|\mathbb{D}_{n}|=\\{50,100,200\\}$.

(a) $\alpha=0.1$

(b) $\alpha=0.25$

(c) $\alpha=0.5$

(d) $\alpha=1$

Figure 4: Learning curves of the ALMA metric associated with different
methods, for a fixed local data size, $|\mathbb{D}_{n}|=100$, and various
heterogeneity levels, $\alpha=\\{0.1,0.25,0.5,1$}. TABLE 1: ALMA (%) given
different data settings on MNIST dataset.

Het. Level $|\mathbb{D}_{n}|=50$ KnFu FedMD Local Selective-FD
$|\mathbb{D}_{n}|=100$ KnFu FedMD Local Selective-FD $|\mathbb{D}_{n}|=200$
KnFu FedMD Local Selective-FD $\alpha=0.1$
$\alpha=.25$
$\alpha=0.5$
$\alpha=1$ $93.5\pm 0.6$ $88.7\pm 0.6$ $92.4\pm 2.7$ $92.2\pm 1.3$
$90.4\pm 1.3$ $86.0\pm 1.1$ $88.9\pm 2.1$ $88.5\pm 1.7$
$81.5\pm 2.7$ $78.1\pm 3.6$ $79.0\pm 4.0$ $79.8\pm 2.6$
$78.5\pm 3.9$ $76.5\pm 3.3$ $75.3\pm 5.0$ $\underline{78.9\pm 4.1}$ $94.1\pm
0.4$ $89.3\pm 0.9$ $94.4\pm 0.8$ $93.1\pm 1.5$
$92.3\pm 0.6$ $88.6\pm 1.2$ $88.7\pm 0.2$ $90.3\pm 1.1$
$88.1\pm 0.6$ $86.4\pm 0.2$ $83.9\pm 0.6$ $87.1\pm 1.0$
$85.6\pm 0.1$ $84.9\pm 0.6$ $80.5\pm 1.1$ $85.2\pm 1.0$ $96.2\pm 0.4$ $92.2\pm
0.8$ $96.3\pm 0.6$ $95.9\pm 0.6$
$94.4\pm 0.7$ $92.0\pm 1.3$ $92.6\pm 1.1$ $93.0\pm 0.9$
$93.3\pm 0.5$ $91.2\pm 1.1$ $89.9\pm 0.0$ $91.9\pm 0.9$
$92.1\pm 0.8$ $92.0\pm 0.4$ $87.8\pm 0.8$ $92.2\pm 0.8$

TABLE 2: ALMA (%) given different data settings on CIFAR-10 dataset.  Het. Level |  | $|\mathbb{D}_{n}|=500$
---
KnFu FedMD Local Selective-FD
| $|\mathbb{D}_{n}|=1000$
---
KnFu FedMD Local Selective-FD
$\alpha=0.1$
$\alpha=0.5$
$\alpha=1$ |  $72.00\pm 1.57$ $50.50\pm 2.51$ $77.80\pm 1.78$ $73.34\pm 2.47$
$48.41\pm 2.30$ $41.57\pm 1.94$ $50.70\pm 1.25$ $48.07\pm 2.13$
$45.30\pm 2.49$ $42.20\pm 2.73$ $43.68\pm 1.83$ $44.38\pm 2.63$ |  $73.80\pm 1.23$ $52.34\pm 1.18$ $81.10\pm 1.81$ $75.20\pm 2.32$
$55.90\pm 0.98$ $50.40\pm 1.68$ $58.20\pm 2.21$ $54.33\pm 2.45$
$48.60\pm 1.96$ $46.90\pm 2.82$ $50.90\pm 2.30$ $48.41\pm 2.57$

##  IV Simulation Results

In this section, we evaluate the performance of the proposed KnFu scheme
through a comprehensive set of experiments analyzing the model’s performance
under various settings, i.e., different data sizes and heterogeneity levels.

###  IV-A Simulation Setup

Datasets: We conduct simulations using two image datasets: MNIST and CIFAR10.
We use a Dirichlet distribution to model expected probabilities over a set of
categories to account for the varying distribution of local data among
clients. Dirichlet distribution is represented as $Dir(\alpha)$, where
$\alpha$ adjusts the non-IID-ness degree, i.e., heterogeneity level. As shown
in Fig. 2, smaller values of $\alpha$ result in more skewed and, therefore,
more non-IID data.

Model Architecture: In our simulations, two distinct CNN architectures are
utilized for MNIST and CIFAR-10 datasets. Specifically, we employ
$M1=[CU_{1}(32);CU_{2}(64);FC_{1}(64);FC_{2}(32);F_{3}(10)]$ for the MNIST
dataset and
$M2=[CU_{1}(16);CU_{2}(16);CU_{3}(32);CU_{4}(32);FC_{1}(128);FC_{2}(10)]$ for
CIFAR-10, where $CU_{m}(t)$ represents the $m^{\text{th}}$ convolutional layer
with $t$ channels, and $FC_{m}(t)$ signifies the $m^{\text{th}}$ dense layer
with a size of $t$ neurons.

Baselines: The proposed KnFu algorithm is compared with FedMD [3], Selective-
FD [6], and local training of the localized models on local datasets (referred
to as Local). To ensure fairness across different methods, we maintain the
size of the transfer set equal to the local data size. In knowledge fusion-
based methods, i.e., KnFu, Selective-FD, and FedMD algorithms, local models
are initially updated on their respective local datasets and then fine-tuned
on the transfer set using ensemble knowledge. Conversely, in the Local method,
both updating and fine-tuning phases occur solely on the local dataset without
any knowledge sharing from other clients.

Average Local Model Accuracy (ALMA) serves as the benchmark metric for all
methods, reflecting the average test accuracy of all local models on their
respective local test datasets. The reported results represent the mean and
standard deviation derived from three separate repetitions with distinct
random seeds for local model initialization and distinct local data
distributions. In each run, the local model initialization and local datasets
are the same for all methods.

Hyperparameters: The local epoch is set to $E=1$ with a batch size of $128$,
$64$, $32$, $16$, and $8$ samples for local data sizes of $1000$, $500$,
$200$, $100$, and $50$, respectively. We consider $20$ clients in the FL
system. The parameter $\beta$ in Eq. (8) is set to $10$ in all experiments. We
execute several experiments for different heterogeneity levels,
$\alpha=\\{0.1,0.25,0.5,1\\}$, and various local data sizes,
$|\mathbb{D}_{n}|=\\{50,100,200,500,1000\\}$.

###  IV-B Simulation Results and Performance Analysis

In this section, our primary aim is to evaluate the effectiveness of various
methods, specifically knowledge-sharing-based algorithms and the local
training method, from the perspectives of local data size and heterogeneity
level. Fig. 3 demonstrates the impact of different local data sizes on the
performance, i.e., ALMA, of various algorithms for a fixed level of
heterogeneity. Across all three scenarios, the proposed KnFu algorithm
exhibits superior ALMA compared to other baselines. Notably, when the local
data size is set to $50$, the standard deviation of ALMA for the local
training (Local) method varies considerably among different repetitions,
although its average ALMA surpasses that of the FedMD algorithm. As the local
data size increases, the performance gap between different methods narrows.

Fig. 4 depicts the performance of various methods across different levels of
heterogeneity for a fixed local data size. In scenarios of high heterogeneity,
i.e., strong non-IID scenarios with $\alpha=0.1$, knowledge-sharing-based
methods do not outperform the local training method. The KnFu algorithm has
superior or comparable ALMA compared to the baselines. However, as the non-IID
degree of local datasets decreases, the performance of knowledge fusion-based
methods converges and surpasses that of the local training method. In summary,
in settings with large and highly heterogeneous local data, knowledge fusion
algorithms do not offer advantages over the local training method.

Table 1 displays the ALMA (average $\pm$ standard deviation) metric of various
methods on the MNIST datasets across different settings, encompassing various
local data sizes and heterogeneity levels. Likewise, Table 2 presents the
performance of baseline methods on the CIFAR-10 datasets under diverse
settings. Unlike the MNIST dataset, the local training method demonstrates
superior performance compared to knowledge-sharing-based methods in most
scenarios, except for the setting where $\alpha=1$ and $|\mathbb{D}_{n}|=500$.
This disparity may stem from the local models’ inadequate performance,
resulting in the generation of low-quality local knowledge. Similar to the
results of the MNIST dataset, it can be observed that in conditions of large
and highly heterogeneous local datasets, the local training method is
preferable to knowledge fusion-based algorithms. However, the KnFu algorithm
outperforms the other knowledge fusion-based methods in most settings.

##  V Conclusion

In conclusion, FL represents a significant shift from centralized ML
approaches, providing a privacy-preserving, decentralized framework for
training models across various nodes. Despite its advantages, FL faces
challenges such as the requirement for uniform model architectures, and model
drift due to non-IID local datasets. While FKD emerged as a solution,
leveraging the KD concept to mitigate some of these issues, it fail to fully
address model drift, highlighting the need for selective knowledge fusion. The
proposed KnFu algorithm offers a novel approach by evaluating and fusing only
the relevant knowledge among clients, showcasing superior performance on MNIST
and CIFAR10 datasets. This research underlines the potential of personalized
knowledge fusion in managing the complexities of FL environments, particularly
in the presence of diverse and heterogeneous data.

## References

  * [1] Konečnỳ, Jakub and McMahan, H Brendan and Yu, Felix X and Richtárik, Peter and Suresh, Ananda Theertha and Bacon, Dave, “Federated learning: Strategies for improving communication efficiency,” in arXiv preprint arXiv:1610.05492, 2016.
  * [2] Fu, Lei and Zhang, Huanle and Gao, Ge and Zhang, Mi and Liu, Xin, “Client selection in federated learning: Principles, challenges, and opportunities,” in IEEE Internet of Things Journal, 2023.
  * [3] Li, Daliang and Wang, Junpu, “Fedmd: Heterogenous federated learning via model distillation,” in arXiv preprint arXiv:1910.03581, 2019.
  * [4] Chennupati, S., Kamani, M., Cheng, Z. & Chen, L. Adaptive distillation: Aggregating knowledge from multiple paths for efficient distillation. ArXiv Preprint ArXiv:2110.09674. (2021)
  * [5] Li, Z., Wang, X., Hu, D., Robertson, N., Clifton, D., Meinel, C. & Yang, H. Not All Knowledge Is Created Equal: Mutual Distillation of Confident Knowledge. ArXiv Preprint ArXiv:2106.01489. (2021)
  * [6] Shao, J., Wu, F. & Zhang, J. Selective knowledge sharing for privacy-preserving federated distillation without a good teacher. Nature Communications. 15, 349 (2024)
  * [7] Wang, D., Zhang, N., Tao, M. & Chen, X. Knowledge Selection and Local Updating Optimization for Federated Knowledge Distillation With Heterogeneous Models. IEEE Journal Of Selected Topics In Signal Processing. 17, 82-97 (2022)
  * [8] Cho, Y., Wang, J., Chirvolu, T. & Joshi, G. Communication-Efficient and Model-Heterogeneous Personalized Federated Learning via Clustered Knowledge Transfer. IEEE Journal Of Selected Topics In Signal Processing. 17, 234-247 (2023)
  * [9] Zhang, J., Guo, S., Ma, X., Wang, H., Xu, W. & Wu, F. Parameterized knowledge transfer for personalized federated learning. Advances In Neural Information Processing Systems. 34 pp. 10092-10104 (2021)
  * [10] Chen, Y., Lu, W., Qin, X., Wang, J. & Xie, X. Metafed: Federated learning among federations with cyclic knowledge distillation for personalized healthcare. IEEE Transactions On Neural Networks And Learning Systems. (2023)
  * [11] Hinton, Geoffrey and Vinyals, Oriol and Dean, Jeff, “Distilling the knowledge in a neural network,” in arXiv preprint arXiv:1503.02531, 2015.


# Emergence and violation of geometrical scaling
in pp collisions ††thanks: Presented at the Low $x$ Workshop, May 30 - June 4
2013, Rehovot and Eilat, Israel

Michal<EMAIL_ADDRESS>
M. Smoluchowski Institute of Physics, Jagellonian University,
Reymonta 4, 30-059 Krakow, Poland

###### Abstract

We argue that geometrical scaling (GS) proposed originally in the context of
Deep Inelastic Scattering (DIS) at HERA works also in pp collisions at the LHC
energies and in NA61/SHINE experiment. We show that in DIS GS is working up to
relatively large Bjorken $x\sim 0.1$. As a consequence negative pion
multiplicity $p_{\rm T}$ distributions at NA61/SHINE energies exhibit GS in
mid rapidity region. For $y\neq 0$ clear sign of scaling violations can be
seen when one of the colliding partons has Bjorken $x\geq 0.1$. Finally, we
argue that in the case of identified particles GS scaling is still present but
the scaling variable is a function of transverse mass rather than $p_{\rm T}$.

PACS number(s): 13.85.Ni,12.38.Lg

##  1 Introduction

This talk based on Refs. [1]–[6] (where also an extensive list of references
can be found) follows closely an earlier report of Ref. [7]. We shall discuss
the scaling law, called geometrical scaling (GS), which has been introduced in
the context of DIS [8]. It has been also shown that GS is exhibited by the
$p_{\text{T}}$ spectra at the LHC [1]–[3] and that an onset of GS can be seen
in heavy ion collisions at RHIC energies [3]. At low Bjorken
$x<x_{\mathrm{max}}$ gluonic cloud in the proton is characterized by an
intermediate energy scale $Q_{\text{s}}(x)$, called saturation scale [9, 10].
$Q_{\text{s}}(x)$ is defined as the border line between dense and dilute
gluonic systems (for review see _e.g._ Refs. [11, 12]). In the present paper
we study the consequences of the very existence of $Q_{\text{s}}(x)$; the
details of saturation phenomenon are here not of primary importance.

Here we shall focus of four different pieces of data which exhibit both
emergence and violation of geometrical scaling. In Sect. 2 we briefly describe
the method used to assess the existence of GS. Secondly, in Sect. 3 we
describe our recent analysis [4] of combined HERA data [13] where it has been
shown that GS in DIS works surprisingly well up to relatively large
$x_{\text{max}}\sim 0.1$ (see also [14]). Next, in Sect. 4, on the example of
the CMS $p_{\rm T}$ spectra in central rapidity [15], we show that GS is also
present in hadronic collisions. For particles produced at non-zero rapidities,
one (larger) Bjorken $x=x_{1}$ may be outside of the domain of GS, _i.e._
$x_{1}>x_{\text{max}}$, and violation of GS should appear. In Sect. 5 we
present analysis of the pp data from NA61/SHINE experiment at CERN [16] and
show that GS is indeed violated once rapidity is increased. Finally in Sect. 6
we analyze identified particles spectra where the particle mass provides
another energy scale which may lead to the violation of GS, or at least to
some sort of its modification [6]. We conclude in Sect. 7.

##  2 Analyzing data with method of ratios

Geometrical scaling hypothesis means that some observable $\sigma$ depending
in principle on two independent kinematical variables, like $x$ and $Q^{2}$,
depends in fact only on a given combination of them, denoted in the following
as $\tau$:

$\sigma(x,Q^{2})=F(\tau)/{Q_{0}^{2}}.$ (1)

Here function $F$ in Eq. (1) is a dimensionless universal function of scaling
variable $\tau$:

$\tau=Q^{2}/Q_{\text{s}}^{2}(x).$ (2)

and

$Q_{\text{s}}^{2}(x)=Q_{0}^{2}\left({x}/{x_{0}}\right)^{-\lambda}$ (3)

is the saturation scale. Here $Q_{0}$ and $x_{0}$ are free parameters which,
however, are not of importance in the present analysis, and exponent $\lambda$
is a dynamical quantity of the order of $\lambda\sim 0.3$. Throughout this
paper we shall test the hypothesis whether different pieces of data can be
described by formula (1) with constant $\lambda$, and what is the kinematical
range where GS is working satisfactorily.

As a consequence of Eq. (1) observables $\sigma(x_{i},Q^{2})$ for different
$x_{i}$’s should fall on a universal curve, if evaluated in terms of scaling
variable $\tau$. This means that ratios

$R_{x_{i},x_{\text{ref}}}(\lambda;\tau_{k})=\frac{\sigma(x_{i},\tau(x_{i},Q_{k}^{2};\lambda))}{\sigma(x_{\text{ref}},\tau(x_{\text{ref}},Q_{k,\text{ref}}^{2};\lambda))}$
(4)

should be equal to unity independently of $\tau$. Here for some $x_{\rm ref}$
we pick up all $x_{i}<x_{\rm ref}$ which have at least two overlapping points
in $Q^{2}$.

For $\lambda\neq 0$ points of the same $Q^{2}$ but different $x$’s correspond
in general to different $\tau$’s. Therefore one has to interpolate
$\sigma(x_{\text{ref}},\tau(x_{\text{ref}},Q^{2};\lambda))$ to
$Q_{k,\text{ref}}^{2}$ such that
$\tau(x_{\text{ref}},Q_{k,\text{ref}}^{2};\lambda)=\tau_{k}$. This procedure
is described in detail in Refs. [4].

By adjusting $\lambda$ one can make
$R_{x_{i},x_{\text{ref}}}(\lambda;\tau_{k})\rightarrow 1$ for all $\tau_{k}$
in a given interval. In order to find an optimal value $\lambda_{\rm min}$
which minimizes deviations of ratios (4) from unity we form the chi-square
measure

$\chi_{x_{i},x_{\text{ref}}}^{2}(\lambda)=\frac{1}{N_{x_{i},x_{\text{ref}}}-1}{\displaystyle\sum\limits_{k\in
x_{i}}}\frac{\left(R_{x_{i},x_{\text{ref}}}(\lambda;\tau_{k})-1\right)^{2}}{\Delta
R_{x_{i},x_{\text{ref}}}(\lambda;\tau_{k})^{2}}$ (5)

where the sum over $k$ extends over all points of given $x_{i}$ that have
overlap with $x_{\text{ref}}$, and ${N_{x_{i},x_{\text{ref}}}}$ is a number of
such points.

##  3 Geometrical scaling in DIS at HERA

In the case of DIS the relevant scaling observable is $\gamma^{\ast}p$ cross
section and variable $x$ is simply Bjorken $x$. In Fig. 1 we present 3-d plot
of $\lambda_{\min}({x,x_{\rm ref}})$ which has been found by minimizing (5).

Figure 1: Three dimensional plot of
$\lambda_{\mathrm{min}}(x,x_{\mathrm{ref}})$ obtained by minimization of Eq.
(5).

Qualitatively, GS is given by the independence of $\lambda_{\text{min}}$ on
Bjorken $x$ and by the requirement that the respective value of
$\chi_{x,x_{\text{ref}}}^{2}(\lambda_{\text{min}})$ is small (for more
detailed discussion see Refs. [4]). One can see from Fig. 1 that the stability
corner of $\lambda_{\text{min}}$ extends up to $x_{\text{ref}}\lesssim 0.1$,
which is well above the original expectations. In Ref. [4] we have shown that:

$\lambda=0.32-0.34\,\,\,\,\,{\rm for}\,\,\,\,\,x\leq 0.08.$ (6)

##  4 Geometrical scaling of central rapidity $p_{\rm T}$ spectra at the LHC

Figure 2: Ratios of CMS $p_{\mathrm{T}}$ spectra [15] at 7 TeV to 0.9 (blue
circles) and 2.36 TeV (red triangles) plotted as functions of $p_{\mathrm{T}}$
(left) and scaling variable $\sqrt{\tau}$ (right) for $\lambda=0.27$.

In hadronic collisions at c.m. energy $W=\sqrt{s}$ particles are produced in
the scattering process of two patrons (mainly gluons) carrying Bjorken $x$’s

$x_{1,2}=e^{\pm y}\,p_{\text{T}}/W.$ (7)

For central rapidities $x=x_{1}\sim x_{2}$. In this case charged particles
multiplicity spectra exhibit GS [1]

$\left.\frac{dN}{dyd^{2}p_{\text{T}}}\right|_{y\simeq
0}=\frac{1}{Q_{0}^{2}}F(\tau)$ (8)

where $F$ is a universal dimensionless function of the scaling variable (2).
Therefore the method of ratios can be applied to the multiplicity
distributions at different energies ($W_{i}$ taking over the role of $x_{i}$
in Eq. (4))222 For pp collisions we define ratios $R_{W_{\rm ref},W_{i}}$ as
an inverse of (4). For $W_{\rm ref}$ we take the highest LHC energy of 7 TeV.
Hence one can form two ratios $R_{W_{\rm ref},W_{i}}$ with $W_{1}=2.36$ and
$W_{2}=0.9$ TeV. These ratios are plotted in Fig. 2 for the CMS single non-
diffractive spectra for $\lambda=0$ and for $\lambda=0.27$, which minimizes
(5) in this case. We see that original ratios plotted in terms of
$p_{\text{T}}$ range from 1.5 to 7, whereas plotted in terms of $\sqrt{\tau}$
they are well concentrated around unity. The optimal exponent $\lambda$ is,
however, smaller than in the case of DIS. Why this so, remains to be
understood.

##  5 Violation of geometrical scaling in forward rapidity region

For $y>0$ two Bjorken $x$’s can be quite different: $x_{1}>x_{2}$. Therefore
by increasing $y$ one can eventually reach $x_{1}>x_{\mathrm{max}}$ and GS
violation should be seen. For that purpose we shall use pp data from
NA61/SHINE experiment at CERN [16] at different rapidities $y=0.1-3.5$ and at
five scattering energies $W_{1,\ldots,5}=17.28,\;12.36,\;8.77,\;7.75$, and
$6.28$ GeV.

Figure 3: Ratios $R_{1k}$ as functions of $\sqrt{\tau}$ for the lowest
rapidity $y=0.1$: a) for $\lambda=0$ when $\sqrt{\tau}=p_{\mathrm{T}}$ and b)
for $\lambda=0.27$ which corresponds to GS.

Figure 4: Ratios $R_{1k}$ as functions of $\sqrt{\tau}$ for $\lambda=0.27$ and
for different rapidities a) $y=0.7$ and b) $y=1.3$. With increase of rapidity,
gradual closure of the GS window can be seen.

In Fig. 3 we plot ratios $R_{1i}=R_{W_{1},W_{i}}$ (4) for $\pi^{-}$ spectra in
central rapidity for $\lambda=0$ and 0.27. For $y=0.1$ the GS region extends
down to the smallest energy because $x_{\rm max}$ is as large as 0.08.
However, the quality of GS is the worst for the lowest energy $W_{5}$. By
increasing $y$ some points fall outside the GS window because $x_{1}\geq
x_{\rm max}$, and finally for $y\geq 1.7$ no GS is present in NA61/SHINE data.
This is illustrated nicely in Fig. 4.

##  6 Geometrical scaling for identified particles

In Ref. [6] we have proposed that in the case of identified particles another
scaling variable should be used in which $p_{\mathrm{T}}$ is replaced by
$\tilde{m}_{\mathrm{T}}=m_{\rm T}-m=\sqrt{m^{2}_{\rm T}+p^{2}_{\rm T}}-m$
($\tilde{m}_{\mathrm{T}}$ – scaling), _i.e._

$\tau_{\tilde{m}_{\rm
T}}=\frac{\tilde{m}_{\text{T}}^{2}}{Q_{0}^{2}}\left(\frac{\tilde{m}_{\text{T}}}{x_{0}W}\right)^{\lambda}.$
(9)

This choice is purely phenomenological for the following reasons. Firstly, the
gluon cloud is in principle not sensitive to the mass of the particle it
finally is fragmenting to, so in principle one should take $p_{\mathrm{T}}$ as
an argument of the saturation scale. In this case the proper scaling variable
would be

$\tau_{\tilde{m}_{\rm T}p_{\rm
T}}=\frac{\tilde{m}_{\text{T}}^{2}}{Q_{0}^{2}}\left(\frac{p_{\text{T}}}{x_{0}W}\right)^{\lambda}.$
(10)

However this choice ($\tilde{m}_{\mathrm{T}}$$p_{\mathrm{T}}$ – scaling) does
not really differ numerically from the one given by Eq. (9).

To this end let us see how scaling properties of GS are affected by going from
scaling variable $\tau_{p_{\rm T}}=\tau$ (2) to
$\tau_{\tilde{m}_{\mathrm{T}}}$ (9) and what would be the difference in
scaling properties if we had chosen $p_{\mathrm{T}}$ as an argument in the
saturation scale leading to scaling variable
$\tau_{\tilde{m}_{\mathrm{T}}p_{\mathrm{T}}}$ (10). This is illustrated in
Fig. 5 where we show analysis [6] of recent ALICE data on identified particles
[17]. In Fig. 5.a-c full symbols refer to the $p_{\mathrm{T}}$ – scaling (2)
and open symbols to $\tilde{m}_{\mathrm{T}}$ – scaling or
$\tilde{m}_{\mathrm{T}}$$p_{\mathrm{T}}$ – scaling. One can see very small
difference between open symbols indicating that scaling variables
$\tau_{\tilde{m}_{\mathrm{T}}}$ (9) and
$\tau_{\tilde{m}_{\mathrm{T}}p_{\mathrm{T}}}$ (10) exhibit GS of the same
quality. On the contrary $p_{\mathrm{T}}$ – scaling in variable $\tau_{p_{\rm
T}}$ (2) is visibly worse.

Finally in Fig. 5.d, on the example of protons, we compare
$\tilde{m}_{\mathrm{T}}$ – scaling (open symbols) and ${m}_{\mathrm{T}}$ –
scaling (full symbols) in variable

$\tau_{m}=\frac{{m}_{\text{T}}^{2}}{Q_{0}^{2}}\left(\frac{m_{\text{T}}}{x_{0}W}\right)^{\lambda}.$
(11)

for $\lambda=0.27$. One can see that no GS has been achieved in the latter
case. Qualitatively the same behavior can be observed for other values of
$\lambda$.

Figure 5: Panels a) – c): comparison of geometrical scaling in three different
variables: $\tau_{p_{\rm T}}$, $\tau_{\tilde{m}_{\rm T}}$ and
$\tau_{\tilde{m}_{\rm T}p_{\rm T}}$ for $\lambda=0.27$. Full symbols
correspond to ratios $R_{W_{1}/W_{2}}$ plotted in terms of the scaling
variable $\tau_{p_{\rm T}}$, open symbols to $\tau_{\tilde{m}_{\rm T}}$ and
$\tau_{\tilde{m}_{\rm T}p_{\rm T}}$, note negligible differences between the
latter two forms of scaling variable. Panel a) corresponds to pions, b) to
kaons and c) to protons. In panel d) we show comparison of geometrical scaling
for protons in scaling variables $\tau_{\tilde{m}_{\rm T}}$ and $\tau_{m_{\rm
T}}$, no GS can be achieved in the latter case.

##  7 Conclusions

In Ref. [4] we have shown that GS in DIS works well up to rather large Bjorken
$x$’s with exponent $\lambda=0.32-0.34$. In pp collisions at the LHC energies
in central rapidity GS is seen in the charged particle multiplicity spectra,
however, $\lambda=0.27$ in this case [1]. By changing rapidity one can force
one of the Bjorken $x$’s of colliding patrons to exceed $x_{\rm max}$ and GS
violation is expected. Such behavior is indeed observed in the NA61/SHINE pp
data [5]. Finally we have shown that for identified particles scaling variable
$\tau$ of Eq. (2) should be replaced by $\tau_{\tilde{m}_{\rm T}}$ defined in
Eq. (9) and the scaling exponent $\lambda\approx 0.3$ [6].

## Acknowledgemens

Many thanks are due to the organizers of this successful series of
conferences. This work was supported by the Polish NCN grant
2011/01/B/ST2/00492.

## References

  * [1] L. McLerran and M. Praszalowicz, Acta Phys. Pol. B 41 (2010) 1917 and Acta Phys. Pol. B 42 (2011) 99.
  * [2] M. Praszalowicz, Phys. Rev. Lett. 106 (2011) 142002.
  * [3] M. Praszalowicz, Acta Phys. Pol. B 42 (2011) 1557 and arXiv:1205.4538 [hep-ph].
  * [4] M. Praszalowicz and T. Stebel, JHEP 1303 (2013) 090 and 1304 (2013) 169.
  * [5] M. Praszalowicz, Phys. Rev. D 87 (2013) 071502(R).
  * [6] M. Praszalowicz, arXiv:1308.5911 [hep-ph].
  * [7] M. Praszalowicz, Acta Phys. Pol. B Proceeding Supplement 6 (2013) 815.
  * [8] A. M. Stasto, K. J. Golec-Biernat and J. Kwiecinski, Phys. Rev. Lett. 86 (2001) 596.
  * [9] L. V. Gribov, E. M. Levin and M. G. Ryskin, Phys. Rept. 100 (1983) 1;
A. H. Mueller and J-W. Qiu, Nucl. Phys. 268 (1986) 427; A. H. Mueller, Nucl.
Phys. B558 (1999) 285.

  * [10] K. J. Golec-Biernat and M. Wüsthoff, Phys. Rev. D 59 (1998) 014017 and Phys. Rev. D 60 (1999) 114023.
  * [11] A. H. Mueller, _Parton Saturation: An Overview_ , arXiv:hep-ph/0111244.
  * [12] L. McLerran, Acta Phys. Pol. B 41 (2010) 2799.
  * [13] C. Adloff et al. [H1 Collaboration], Eur. Phys. J. C 21 (2001) 33; S. Chekanov et al. [ZEUS Collaboration], Eur. Phys. J. C 21 (2001) 443.
  * [14] F. Caola, S. Forte and J. Rojo, Nucl. Phys. A 854 (2011) 32.
  * [15] V. Khachatryan et al. [CMS Collaboration], JHEP 1002 (2010) 041 and Phys. Rev. Lett. 105 (2010) 022002 and JHEP 1101 (2011) 079.
  * [16] N. Abgrall et al. [NA61/SHINE Collaboration], Report from the NA61/SHINE experiment at the CERN SPS CERN-SPSC-2012-029, SPSC-SR-107;
A. Aduszkiewicz, Ph.D. Thesis in prepartation, University of Warsaw, 2013;
Sz. Pulawski, talk at 9th Polish Workshop on Relativistic Heavy-Ion
Collisions, Kraków, November 2012 and private communication.

  * [17] K. Aamodt et al. [ALICE Collaboration], Eur. Phys. J. C 71 (2011) 1655 [arXiv:1101.4110 [hep-ex]];
A. Ortiz Velasquez [ALICE Collaboration], Nucl. Phys. A 904-905 (2013) 763c
[arXiv:1210.6995 [hep-ex]] (ALICE preliminary).