# Unraveling Attacks in Machine Learning-based IoT Ecosystems: A Survey and
the Open Libraries Behind Them

Chao Liu, Boxi Chen*, Wei Shao, Chris Zhang, Kelvin K.L. Wong, and Yi Zhang,
Chao Liu and Boxi Chen* contributed equally to this work. This work was
completed during Boxi Chen’s internship.Corresponding author: Kelvin K.L. Wong
<EMAIL_ADDRESS>and Yi Zhang (zhangyi@scu.edu.cn)Chao Liu was with the
Department of Electrical and Computer Engineering, University of Maryland,
Baltimore County, MD, USA.Boxi Chen was with the Department of Research, Deep
Red Future Technology Co. Ltd., Shenzhen, China.Wei Shao is with the data61,
CSIRO, Australia.Chris Zhang and Kelvin K.L. Wong were with the Department of
Mechanical Engineering, University of Saskatchewan, Saskatchewan, Canada.Yi
Zhang was with Machine Intelligence Laboratory, College of Computer Science,
Sichuan University, Sichuan, China.

###### Abstract

The advent of the Internet of Things (IoT) has brought forth an era of
unprecedented connectivity, with an estimated 80 billion smart devices
expected to be in operation by the end of 2025. These devices facilitate a
multitude of smart applications, enhancing the quality of life and efficiency
across various domains. Machine Learning (ML) serves as a crucial technology,
not only for analyzing IoT-generated data but also for diverse applications
within the IoT ecosystem. For instance, ML finds utility in IoT device
recognition, anomaly detection, and even in uncovering malicious activities.
This paper embarks on a comprehensive exploration of the security threats
arising from ML’s integration into various facets of IoT, spanning various
attack types including membership inference, adversarial evasion,
reconstruction, property inference, model extraction, and poisoning attacks.
Unlike previous studies, our work offers a holistic perspective, categorizing
threats based on criteria such as adversary models, attack targets, and key
security attributes (confidentiality, availability, and integrity). We delve
into the underlying techniques of ML attacks in IoT environment, providing a
critical evaluation of their mechanisms and impacts. Furthermore, our research
thoroughly assesses 65 libraries, both author-contributed and third-party,
evaluating their role in safeguarding model and data privacy. We emphasize the
availability and usability of these libraries, aiming to arm the community
with the necessary tools to bolster their defenses against the evolving threat
landscape. Through our comprehensive review and analysis, this paper seeks to
contribute to the ongoing discourse on ML-based IoT security, offering
valuable insights and practical solutions to secure ML models and data in the
rapidly expanding field of artificial intelligence in IoT.

###### Index Terms:

Internet of Things, Machine Learning, Attack, Security, Open Library,
Artificial Intelligence

##  I Introduction

###  I-A IoT and Machine Learning

The original design concept of the Internet of Things is centered on achieving
connectivity among all things, aiming to create a digitalized and intelligent
network system. Within this system, devices can communicate and collaborate,
thereby simplifying and accelerating a variety of tasks and processes. In the
IoT ecosystem, end-users can interact with IoT devices through their
smartphones and computers [1], which offers immense convenience in sectors
like industry, household management, and healthcare. According to an IoT
industry white paper jointly published by iResearch Consulting Group and
Kingsoft Cloud, it is predicted [2] that by 2025, the global number of IoT
connections will increase to 24.6 billion, and the global IoT revenue will
grow to 1.1 trillion USD.

The foundational idea of IoT is to connect everything in the world to the
internet, allowing for autonomous recognition, communication, and even
decision-making among objects. The tremendous advancements in computer
communication and related technologies have made numerous IoT applications
possible. Researchers from diverse industries, research groups, academia, and
government bodies are paying close attention to the IoT revolution, striving
to create a more convenient environment comprised of an array of intelligent
systems such as smart homes, intelligent transportation, global supply chain
logistics, and healthcare [3, 4, 5]. IoT’s adoption of automatic data sensing
and collection has generated big data, a category of data that has surpassed
the capacity of traditional manual data analysis and processing methods. This
paradigm shift underscores the need for more advanced and automated data
analysis solutions to harness the full potential of IoT’s capabilities. One of
the most effective methods to enhance the intelligence of data analysis and
processing in the Internet of Things (IoT) is to integrate Machine Learning
(ML) into its operations. ML systems shown in Fig. 1 are designed to
automatically learn models from training data and utilize these models to make
predictions. They find extensive applications across numerous fields,
including ML-based IoT device identification, ML-based IoT Malware, etc.

Figure 1: Typical scenarios and ML-based IoT visions.

####  I-A1 ML-based IoT Device Identification

Mirai [6], in particular, stands as a large-scale botnet composed of
compromised IoT devices. Its massive Distributed Denial-of-Service (DDoS)
attack in 2016 paralyzed crucial communication infrastructures along the U.S.
East Coast, resulting in substantial economic losses.

Research on IoT device identification and anomaly detection is crucial for
asset and security management. These two areas are closely interconnected,
sharing similarities on a technical level, as both can be viewed as
classification problems. The process involves data collection: IoT device
identification collects passive network traffic or response data packets,
while anomaly detection gathers normal traffic data or data about abnormal
patterns, logs, or application source codes. After data collection,
preprocessing occurs. Device identification extracts rules for a rule
repository, and anomaly detection constructs a normative behavior profile,
identifying deviations as anomalies. Both methods leverage preprocessed data
to derive raw features, building machine learning models. Device
identification yields device categories, while anomaly detection classifies as
normal/abnormal or specific abnormal types. Evaluation involves metrics like
Accuracy, Precision, Recall, and F1 scores. Anomaly detection uses additional
metrics like false alarm rate and false negative rate. Machine learning-based
methods extract features to form device fingerprints. Subsequently, through
machine learning-based techniques [7, 8], classification models are trained
using labeled data, and these models are employed for device identification.
Therefore, the selection and extraction of features, methods for data
labeling, and the choice and training of models constitute focal points in
research.

####  I-A2 ML-based IoT Malware Detection

In recent years, security researchers have shifted focus to detecting
malicious software in the Internet of Things (IoT). Traditional methods relied
on feature databases and manual analysis, but the exponential growth of
malicious software created inefficiencies and challenges in addressing unknown
security risks [9, 10]. To overcome these issues, researchers are exploring
the application of artificial intelligence (AI) techniques [11, 12, 13]. While
AI-based research in IoT malicious code detection is gaining traction,
challenges unique to IoT devices need attention [14]: Diverse Architectures:
Malicious code in IoT can infect devices with various CPU architectures,
making it challenging to directly apply matured detection methods from
conventional devices. Resource Limitations: IoT devices, having smaller form
factors and resource constraints, require lightweight detection systems due to
limited memory space and low power. Addressing these challenges is crucial for
the effectiveness of AI-based detection in IoT malicious code while enhancing
detection accuracy.

####  I-A3 Edge Computing with ML

ML applications often face computational challenges, especially on resource-
constrained local devices in industrial IoT. While cloud computing offers a
solution by offloading tasks to servers, the drawback is significant response
latency due to the distance between devices and servers. Edge computing
emerges as a promising solution, deploying servers near IoT devices for faster
task processing and reduced network loads [15]. Different computational
capabilities among IoT devices, edge servers, and cloud servers allow
deployment of varying complexity ML models. However, executing ML tasks on
these platforms results in varying inference precision. Lower precision
impacts system performance, potentially causing material waste or delays.
Despite this, existing optimizations often overlook the impact of ML model
complexity on precision. Balancing execution speed, cost reduction, and
ensuring precision is crucial for industrial IoT safety and efficiency.
Limited storage on edge servers poses challenges in deploying diverse ML
models, while computational resource constraints compromise task processing
efficiency. Addressing these challenges is essential for optimizing system
performance in industrial IoT scenarios [16].

####  I-A4 Software‑defined Networking with ML

Software-defined networking (SDN) has recently gained traction in both
academic and industrial sectors due to its inherent flexibility. By decoupling
the control plane from the forwarding plane, SDN empowers network operators to
manipulate networks using high-level configuration languages, alleviating the
need for intricate forwarding table configurations. In the context of IoT,
where device complexity and diversity are pronounced, configuring the data
path in IoT networks becomes notably challenging compared to traditional
networks. Therefore, SDN emerges as a pivotal solution in the IoT landscape
[17].

However, owing to the intricate nature of IoT systems, leveraging machine
learning within the control plane becomes imperative for more effective
network management. The complexities inherent in IoT necessitate adaptive and
intelligent network control mechanisms, which machine learning can facilitate.
ML has two important aspects for SDN in IoT network: (1) ML facilitates IoT
network management [18, 19, 20] with SDN, enhancing ease and efficacy. (2) ML
aids in detecting potential intrusions [21, 22, 23], thereby improving
accuracy.

####  I-A5 ML in IoT applications

IoT applications, prevalent in healthcare, agriculture, and industrial
sectors, face challenges due to diverse and complex data sources. The model of
IoT applications includes wearable devices, mobile sensors, network cameras,
and various wireless sensors capturing human vital signs and environmental
data. Machine learning methods process this data for tasks like health
monitoring, activity recognition, fraud detection, and object identification.

Studies focus on personal health monitoring and industrial use cases. IoT
networks excel in stress monitoring, human activity recognition, and disease
prediction in agriculture. Madeira et al. [24] introduced a system predicting
human presence using diverse interaction data from IoT devices. Siryani et al.
[25] applied machine learning to optimize smart meter operations, reducing
costs by predicting technician dispatch necessity. Instances like [26, 27, 28,
29] showcase machine learning applications in IoT.

In summary, machine learning models trained on data from IoT devices have
significantly enhanced efficiency across industries like healthcare,
logistics, and urban planning. However, these devices collect sensitive
information, posing privacy and security risks. Adversaries could misuse data
for unfair practices, impacting individuals and even national security. For
instance, smart fitness tracker data could lead to unfair health insurance
adjustments, while compromising IoT devices in critical infrastructure may
result in disruptions and security threats. Despite the benefits, the privacy
and security concerns surrounding sensitive training data have become crucial,
drawing attention from academia and industry stakeholders.

###  I-B IoT-integrated Machine Learning Security Risks

Figure 2: Different attack models targeting ML-based IoT.

The rapid development of AI and IoT technologies has brought substantial
progress and innovation, offering diverse capabilities and services across
domains. However, this progress comes with challenges, particularly in the
realms of security risks and vulnerabilities. In the interconnected world of
IoT devices and systems, these challenges extend to data privacy, network
security, and ethical concerns. This article explores six prevalent attack
types on machine learning systems in IoT Fig. 2: membership inference,
adversarial attacks, reconstruction attacks, property inference attacks, model
extraction attacks, and poisoning attacks. Real-world instances highlight how
these attacks can compromise the privacy and security of machine learning
systems across various domains within the IoT environment. The intricately
connected ecosystem of IoT devices, from smart thermostats to industrial
sensors, creates a vast attack surface, making ML systems susceptible to
malicious activities. The insights provided aim to illuminate these
vulnerabilities, offering a comprehensive overview of potential threats and
their implications on the privacy and security of machine learning systems
within IoT networks and applications.

####  I-B1 Membership Inference Attack Against Collaborative Inference in
Industrial IoT

Despite the notable achievements across a spectrum of industrial IIoT
applications, collaborative inference systems inherently face diverse privacy
and security risks, including model inversion attacks and property inference
attacks. Researchers [30] have developed practical membership inference
attacks (MIA) targeting collaborative inference methods, aiming to ascertain
whether an individual input sample exists within the training dataset.

####  I-B2 Adversarial Attacks on Graph-based IoT Malware Detection Systems

Machine learning algorithms, particularly deep learning networks, are actively
involved in identifying and categorizing malicious software from benign ones.
Yet, the increased reliance on deep learning models within security realms
presents opportunities for adversaries to manipulate these models, steering
them toward generating specific outputs of their choice. Research [31] has
unveiled vulnerabilities inherent in machine and deep learning networks. For
instance, adversaries can coerce models into generating desired outputs, such
as misclassifications, by creating adversarial examples (AEs).

####  I-B3 Reconstruction Attack for ML-based Recognition Systems

Given the extensive use of recognition systems in cloud environments, the risk
of cyber threats remains significantly high. Consequently, cybersecurity
measures have garnered substantial focus, leading to the proposal of diverse
general-purpose algorithms aimed at combating cybercrimes across industrial
control systems, power quality management, unmanned aerial vehicles (UAVs),
smart grids, and other sectors. The model inversion attack (MIA) [32] is a
technique used to create a replicated data point associated with a specific
user identity within recognition systems. This attack traces the impact of a
corresponding user’s training data on the structure and parameters of the
system’s model. The clone data generated through MIA often exhibit high-
recognition scores and are erroneously accepted by deep-learning-based
systems, despite the complexity of the model structure, numerous class labels,
and extensive training data. Surprisingly, these factors, which might
typically enhance security, fail to prevent the tracing of user features
through the effect of their training data on the model structure.

####  I-B4 Property Inference Attack on Edge-Cloud Collaborative Inference
Systems

With the substantial improvement in end device computation capabilities,
collaborative learning, also known as federated learning, has evolved
significantly within real-world IoT applications. An example of an inference
attack, property inference attack [33], within the training process is the
property (or attribute) inference attack, which aims to deduce whether a
training batch contains data associated with a specific sensitive property,
such as race, or not.

####  I-B5 AI Model Extraction Attack in IoT

In the realm of intelligent IoT security, safeguarding ML-based IoT models is
of paramount importance. When these systems operate on hardware platforms
(smart-X devices, such as automotive and virtual reality/augmented reality
devices), inadvertent physical information leaks occur, such as energy
consumption, execution time, and electromagnetic emissions released during
data computation. Adversaries monitor these physical leaks in real-time
through predefined probes and analyze them to infer sensitive information. AI
models encompass various elements like structure, metaparameters, and weights,
hence the extraction of ML-based IoT models involves multiple methods. By
exerting physical control over the hardware, adversarys leverage
multidimensional side-channel signals to extract the structure and parameters
of AI models operating on the hardware. Memory access patterns and timing
serve as common side-channel signals used to deduce the architecture of AI
models [34].

####  I-B6 Poisoning Attacks on Machine Learning in IoT Environments

Utilizing machine learning in IoT settings introduces distinct security
challenges as adversaries can potentially tamper with sensors, manipulating
the training data. This form of attack termed a poisoning attack, enables
adversaries to significantly impair overall performance, induce targeted
misclassifications or undesirable behavior, and implant backdoors or neural
trojans [35]. A well-known instance of a poisoning attack beyond the realm of
IoT transpired when Microsoft introduced Tay, a chat bot, designed to learn
and generate human-like tweets. Users exploited the system by posting
offensive content, leading Tay to produce similarly offensive tweets.
Microsoft had to swiftly remove the bot within just 16 hours due to this
manipulation.

In summary, ML attacks pose significant threats to IoT due to their potential
to compromise security and privacy. These attacks, including model inversion
attacks, poisoning attacks, adversarial attacks, etc, exploit vulnerabilities
in IoT devices and systems. They can manipulate training data, compromise AI
models, and extract sensitive information, risking the integrity and
confidentiality of data collected by IoT devices. This compromises user
privacy, creates avenues for unauthorized access, and undermines the
reliability of IoT-driven applications, amplifying concerns about security and
privacy in connected environments.

Figure 3: The structure of this article.

The main contributions of this paper are as follows:

  * •

We delve into a comprehensive range of attack types pertinent to ML-integrated
IoT systems, including membership inference attacks, adversarial attacks
(evasion attacks), reconstruction attacks (such as model inversion and
attribute inference attacks), property inference attacks, model extraction
attacks, and poisoning attacks (covering both data and model poisoning).
Unlike other studies that might focus narrowly on specific security attack
facets, our investigation spans the entire spectrum of security threats,
emphasizing their implications in IoT environments.

  * •

Our categorization of machine learning attacks takes into account the
multifaceted nature and objectives of these threats in IoT settings. For
example, we differentiate between white-box and black-box attacks based on the
attacker’s level of knowledge. These attacks are further classified into data-
centric and model-centric types. In terms of security, we organize them under
the umbrellas of confidentiality, availability, and integrity threats.
Furthermore, we provide a concise summary of key technologies discussed in our
paper, aiding in a more structured and nuanced understanding of the security
challenges in ML-enhanced IoT systems.

  * •

In our exploration of ML security within IoT settings, we’ve assessed 65
libraries. These libraries specifically focus on addressing model and data
privacy within machine learning applications relevant to IoT. They encompass
offerings developed by authors as well as those provided by various
individuals or entities. Our evaluation spans diverse facets and objectives of
ML systems within IoT scenarios, introducing a comprehensive set of metrics
for assessment. We aim to verify the accessibility, usability, and practical
applicability of these libraries in securing IoT systems. Through rigorous
testing and validation, we’re advocating for the usability of the majority of
these libraries in real-world IoT applications.

The rest of this paper is organized as follows: Section II describes the
attack and threat models. Section III describes the taxonomy of attacks in ML-
based IoT. Section IV focuses on the discussion of this paper. At last, in
Section V, we conclude the paper. The roadmap and organization of this article
are shown in Fig. 3.

##  II Attack and Threat Models

Understanding and defending against privacy-related attacks in ML-based IoT
requires a comprehensive model of the environment, the involved actors, and
the assets requiring protection. In the context of a threat model, the
sensitive assets susceptible to potential attacks encompass the training
dataset $\mathcal{D}$, the model, its parameters $\sigma$, hyper-parameters,
and its architecture.

Figure 4: Different categories in attack and threat models.

###  II-A Actors in Threat Model

####  II-A1 Data owners

are individuals or entities whose data holds sensitivity or confidentiality.

####  II-A2 Model owners

are individuals or entities who possess the machine learning models. They may
or may not be the owners of the underlying data and might have varying
preferences regarding sharing information about their models.

####  II-A3 Model consumers

are individuals or entities utilizing the services provided by the model
owner, typically accessed through programming interfaces or user interfaces.

####  II-A4 Adversaries

refer to individuals or entities that, similar to normal consumers, might
access the model’s interfaces. In certain cases, if permitted by the model
owner, adversaries may gain access to the model itself.

###  II-B Threats in ML-based IoT

Threats in ML-based IoT can be divided into two categories: security attacks
and privacy infringements. In the former, attackers compromise the integrity
and availability of data, while in the latter, attackers violate the privacy
and confidentiality of data. Common attacks include the following three types.

####  II-B1 Integrity Attacks

Integrity attacks [36] aim to alter the behavior or output of a machine
learning system by maliciously manipulating training data or models. For
example, by injecting fictitious data into the training data set, it may lead
to inaccurate classification and erode user trust in the system. This is
similar to deliberately mixing substandard products with high-quality products
during quality inspections, thereby reducing the credibility of the entire
product. In an adversarial attack, the input data is modified to fool the
model, leading to incorrect classification. In the Internet of Things,
tampering with sensor data for predictive maintenance may mislead the model,
leading to incorrect predictions or inappropriate maintenance operations, thus
affecting the functionality and reliability of the equipment.

####  II-B2 Availability Attacks

Availability attacks [37] aim to disrupt the normal functionality of machine
learning-based IoT or generate inaccurate output, leading to system crashes,
service interruptions, or erroneous results. This is similar to blocking
traffic causing congestion or interfering with communications to disrupt
messaging. In IoT, denial-of-service attacks overwhelm a smart home automation
system with too many commands, rendering it unable to respond to legitimate
requests and disrupting the functionality of the entire network. Likewise,
flooding sensor networks with too much or erroneous data can lead to delays or
system failures in making timely decisions based on sensor inputs.

####  II-B3 Confidentiality Attacks

Confidentiality attacks [38] aim to acquire sensitive information or private
data within ML systems, analogous to a thief stealing from a secure vault or a
hacker illicitly taking user’s personally identifiable information. In IoT,
these attacks may lead to unauthorized access, theft, or leakage of users’
sensitive data, posing threats to privacy, trade secrets, or national
security. Attackers can obtain training data or model parameters through
various means, like side-channel attacks where monitoring power consumption
patterns reveals details about processed inputs. Using model inversion
techniques [39], an attacker may reconstruct facial features from a facial
recognition system’s output.

###  II-C Categories of Attack Targets

####  II-C1 Attack on Training Data

Protecting the privacy of training data [40] is crucial, especially in IoT
scenarios. For example, a company creating a predictive model based on data
from IoT sensors in smart cities aims to optimize traffic flow without
compromising individual privacy. Ensuring training data privacy involves: a)
Data Confidentiality: Concealing specific IoT device data or individual
movements to prevent exposure of identifiable information; b) Feature
Anonymization: Protecting sensitive attributes within IoT data to preserve
individual or entity privacy; c) Securing Statistical Insights: Ensuring
statistical patterns derived from IoT data do not compromise privacy; d)
Protecting Membership Information: Concealing whether specific IoT devices or
data points were part of the training set to prevent inference about their
inclusion.

####  II-C2 Attack on Model

Privacy concerns [41] in machine learning extend beyond data to model
components, including parameters and algorithms. For example, an IoT-based
company developing a model for forecasting energy consumption in smart homes
faces risks to its commercial value and intellectual property if model
specifics are exposed. Similarly, commercial IoT-based machine learning
services, relying on proprietary models and algorithms, could experience
substantial revenue loss if these details are revealed. In summary, attacks on
machine learning systems in IoT can target not just data but also the
structure, parameters, or algorithms, posing risks to intellectual property,
commercial viability, and revenue streams.

###  II-D White-Box vs. Black-Box

In the context of ML-based IoT attacks, attack methods can be classified as
either white-box attacks or black-box attacks depending on the level of
adversarial knowledge.

####  II-D1 White-Box Attacks

A white-box attack [42] occurs when the attacker possesses detailed knowledge
of the target model, including the model’s internal structure, parameters,
weights, and training data. An attacker may have access to all information
about the model.

####  II-D2 Black-Box Attacks

A black-box attack[43] means that the attacker can only access the input and
output of the target model but lacks knowledge about the model’s internal
structure, parameters, and weights. The attacker does not have access to the
details of the target model. Therefore, the adversary can infer information
about the model by carefully designing inputs and observing the model’s
outputs. Black-box attacks are generally more challenging because the attacker
needs to reverse engineer the behavior of the target model through
experimentation and observation without direct visibility into its internals.

###  II-E Passive vs. Active

“Active attacks” and “passive attacks” are two terms used to describe the
goals and strategies of attackers. These two types of attacks have different
characteristics and objectives, and understanding them helps in better
defending against potential threats.

####  II-E1 Active Attacks

An active attack[44] occurs when an attacker takes deliberate actions to
disrupt, corrupt, or modify ML-based IoT models or training data to achieve
their malicious objectives.

####  II-E2 Passive Attacks

A passive attack[45] happens when an attacker observes, listens, or collects
information without altering or interfering with the ML-based IoT model or
training data. In passive aggression, the attacker’s primary aim is to acquire
sensitive information or comprehend the model’s behavior without causing
noticeable interference.

###  II-F Training vs. Inference

####  II-F1 Attack in Training Phase

The adversary aims to gain insights into the model, potentially by accessing
summaries, partial segments, or the entirety of the training data. They might
endeavor to construct a substitute model, also known as an auxiliary model,
which they can then utilize to launch attacks on the victim system.

####  II-F2 Attack in Inference Phase

The adversary gathers information about the model’s characteristics by
observing the inferences it generates.

###  II-G Adversary Model

Adversary models [46] are theoretical frameworks used in security research and
practice to describe and analyze the capabilities, intentions, and behaviors
of potential attackers. In ML-based IoT security, adversary models are often
employed to characterize the strength of an adversary. Barreno et al. [47] in
2010 considered adversary models that account for the capabilities of
attackers and their objectives. Building upon the work of Biggio et al. [48]
in 2013 further refined these models, proposing adversary models encompassing
adversary objectives, knowledge, capabilities, and strategies. Current
research widely accepts this framework as the standard adversary or attacker
model. By delineating adversaries along these four dimensions, it becomes
possible to systematically assess the extent of the threat posed by the
adversary.

####  II-G1 Adversary Goal

The adversary’s target, as discussed by [48], involves both the extent of
destruction and the level of specificity they aim for. Destruction includes
integrity, availability, and privacy, while specificity encompasses targeted
and non-targeted objectives. Destruction of integrity involves unauthorized
actions on data, like addition, deletion, modification, or destruction,
impacting the accuracy of trained models, for example, in medical data.
Destruction of availability aims to make the target service unusable, achieved
by injecting malicious data into the training dataset, rendering the model
ineffective. Destruction of privacy targets the theft of private data,
extracting information from the training dataset. Regarding specificity,
targeted objectives involve selectively disrupting specific aspects, like
stealing the private information of a particular customer in medical data.
Non-targeted objectives cause non-selective disruptions, impacting data
integrity broadly.

####  II-G2 Adversary Knowledge

Adversarial knowledge [49] refers to the amount of information possessed by an
adversary about the target model or the target environment, which includes
details about the model’s training data, model structure, parameters,
information obtained through the model’s outputs, and so on. Adversarial
knowledge can be categorized based on the quantity of information the
adversary possesses, namely limited knowledge or complete knowledge.

####  II-G3 Adversary Ability

Adversary ability refers to the degree of control that an adversary has over
both training and testing data. The impact of adversaries on data can be
categorized as inductive, which affects the dataset, or exploratory, which
does not affect the dataset. Adversary ability [50] can also be defined based
on whether the adversary can intervene in model training, access training
datasets, collect intermediate results, etc. Depending on the level of control
over data and models, adversaries can be further classified as strong or weak
adversaries. Strong adversaries can intervene to a certain extent in model
training, access training datasets, and collect intermediate results. In
contrast, weak adversaries can only obtain information about the model or
training data through attack methods.

####  II-G4 Adversary Strategy

Adversary strategy pertains to the attack methods employed by adversaries,
considering their objectives, knowledge, and capabilities to achieve optimal
outcomes against the target. These adversary strategies typically involve
utilizing different attack techniques at various stages of ML-based IoT. For
instance, approaches like poisoning attacks are frequently employed during the
training phase, whereas adversarial attacks and privacy attacks are commonly
deployed during the prediction phase.

Type Method | Learning | GAN | Gradient-based | Shadow Training | Threshold | Equation-solving
---|---|---|---|---|---|---
MIA | $\bullet$ | $\bullet$ |  | $\bullet$ | $\bullet$ |
AA | $\bullet$ | $\bullet$ | $\bullet$ |  | $\bullet$ | $\bullet$
RA | $\bullet$ | $\bullet$ | $\bullet$ |  | $\bullet$ | $\bullet$
PIA | $\bullet$ |  |  | $\bullet$ |  |
MEA | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ |  | $\bullet$
PA | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ |

Table I: Overview of attack methodologies for each type of attack. We classify
the six attack types across six dimensions of Learning, GAN, Gradient-based,
Shadow Training, Threshold, and Equation-solving. Our main objective is to
elucidate how learning technology utilizes model parameters and gradients as
inputs for the classifier, or employs both input and output queries, to
develop alternative models.
PAMEAPIARAAAMIA$0$$5$$10$$15$$4$$6$$8$$9$$12$$8$$0$$2$$0$$3$$3$$3$$2$$0$$0$$2$$1$$2$Attack
TypeNumber of Attack TypeOriginalUnoriginalN/A Figure 5: Number of libraries
for each attack. Membership Inference Attacks (MIA), Adversarial Attacks (AA),
Reconstruction Attacks (RA), Property Inference Attacks (PIA), Model
Extraction Attacks (MEA), and Poisoning Attacks (PA). We categorize libraries
as original library if they were developed by the authors of the paper,
unoriginal library if provided by other individuals or entities, and N/A if
the source of the library’s origin cannot be determined.

##  III Taxonomy of Attacks in ML-based IoT

In this section, we delve into the intricacies of six prominent attacks on ML-
based IoT systems. We will explore the mechanisms behind each attack and
highlight recent advancements that have been able to penetrate these systems
successfully. By understanding these vulnerabilities and the current state of
the art in exploiting them, we can pave the way for more secure and resilient
ML-based IoT applications in the future.

###  III-A Membership Inference Attacks

Membership Inference Attacks have emerged as a noteworthy concern. At the core
of membership inference attacks is the attacker’s ability to infer whether a
specific data point was part of the training dataset used to train an ML
model. A visual representation of this process can be found in Fig. 6. In
other words, membership inference attacks are based on methods in which the
target model learns features in the training dataset. The attacker constructs
his training dataset and observes the model’s output on these samples to infer
whether the samples in the test dataset belong to the membership samples. This
might sound innocuous at first, but consider scenarios where the training data
contains sensitive information — revealing whether a particular data point
(like a patient’s medical record) was used in training could lead to
significant privacy breaches.

One of the seminal works in this domain was presented by Shokri et al. [51].
They provided a concrete framework to understand membership inference attacks,
introducing an attack model $f_{\text{attack}}$. The input $x_{attack}$ is a
prediction confidence vector consisting of the correct label class and a
target model (the model being attacked). The output of the attack model is a
prediction class “in” (member) or “out” (non-member).

$\displaystyle f_{attack}(x_{attack})\rightarrow y$ (1) $\displaystyle
x_{attack}=(label,prediction),y\in\\{in,out\\}$ (2)

For further insights, we show the experimental results in TABLE II and briefly
introduce seminal and contemporary works on membership inference attacks. Key
references include [51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64,
65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80]. (For some
unintroduced papers, we also display them in TABLE VIII).

Figure 6: Overview of membership inference attacks in ML-based IoT system.

Shokri et al. [51] propose a method to determine if a record belongs to the
model’s training dataset. They achieve this by providing black-box access to
the target model and using shadow model training techniques to mimic the
target model’s behavior. To counter such attacks, they suggest mitigation
strategies, including restricting the prediction vector to the top $k$
classes, coarsening the precision of the prediction vector, increasing the
entropy of the prediction vector, and employing regularization. Rong et al.
[81] take a different approach by training a single shadow model with various
datasets to capture the membership state of data points in the ML training
set. They propose combating broader membership inference attacks by employing
techniques such as dropout and model stacking for defense. Long et al. [52]
identify individual records with high accuracy in real-world datasets using
black-box queries on ML models. They argue that combining generalization and
perturbation with careful training set selection can be used to defend against
such attacks. Nasr et al. [82] assess the privacy vulnerability of the
stochastic gradient descent algorithm and design a deep learning attack model
to calculate the membership probability of target data points. They also
evaluate the privacy of pre-trained state-of-the-art models on the CIFAR-100
dataset to validate the effectiveness of white-box inference attacks. Liwei
Song et al. [55] propose two new membership inference attacks against
adversarial robust models, leveraging predictions from adversarial examples
and validated worst-case predictions. Their core approach exploits the
structural characteristics of adversarial robust defense to achieve high
inference accuracy. They further validate the effectiveness of these attacks
against six state-of-the-art adversarial defense methods. Choquette et al.
[71] introduce a labeling attack that assesses the robustness of a model’s
predicted labels to input perturbations to infer membership. They also confirm
that training with differential privacy [83] or strong L2 regularization [84]
can serve as an effective defense. Song et al. [69] measure the privacy risk
of a target model using improved attacks with class-specific threshold
settings and a new inference attack based on enhanced prediction entropy
estimates. They demonstrate that these methods are not as effective as
previous attacks when tested against defense mechanisms like adversarial
regularization [84] and MemGuard [85]. BLINDMI [68] introduces a practical
approach where existing samples are transformed into new samples to create a
dataset with non-membership. Samples are differentially moved from the target
dataset to the generated non-membership set, and membership status is
determined based on the distance moved by the sample. Hui et al. verify that
BLINDMI can break these defenses by achieving reasonable F1-score under
different privacy utility budgets against adversarial regularization [84],
MemGuard [85], Mixup+MMD [86], and differential privacy [83]. A Bayesian
perspective is adopted by Hu et al. [70] to demonstrate that honest but
curious servers can initiate Source Information Attacks (SIA) to steal non-
trivial source information of training members without violating the federated
learning protocol. The server exploits the prediction loss of the local model
on the training members to implement the attack and evaluates the key factors
in SIA on six datasets to verify its effectiveness. Knowledge extraction [87]
is employed as an attack method by Liu et al. [72]. The adversary evaluates
the loss trajectory of the target sample on each intermediate extraction model
to obtain the loss trajectory of the target sample. Finally, the attack model
takes the extraction loss trajectory of the target sample and its loss on the
original target model as input to infer the membership status of the sample.
They further validate that the attack exhibits advantages in different
indicators across various datasets and model architectures.

Table II: Reviews of membership inference attack libraries. The library details are presented in the table below. In this table, “Ref.” refers to the cited paper, or is marked N/A if no corresponding paper was located in our library. “Attack Type” describes the kind of attack in the paper, and “White/Black-box” identifies the attack setting. “Library Link” offers a link to the mentioned library, while “Original” specifies if the library was created by the authors of the paper. “Unoriginal Library” pertains to libraries sourced from other individuals or entities, or N/A if the origin remains unclear. “Impl.” indicates whether the main program of the library was run successfully after debugging. The version of Python utilized during debugging is mentioned, as well as if a GPU was used. “Dataset” names the dataset the library relies on, while “Evaluation” displays the debugging results or is marked N/A if results cannot be shown due to being in image format or if debugging was unsuccessful. “Issues” pinpoint primary challenges faced during debugging, such as missing essential library files termed “Lost files.” “Debugging solution” lists the adjustments made during debugging, “Debugging failed” notes when severe bugs obstructed the debugging process, and “Environment loading failed” is used when setting up a working runtime environment was unattainable, or N/A if debugging proceeded without significant issues. It’s vital to recognize that this description pertains to tables from TABLE. II through TABLE. VI. However, TABLE. VII (focused on poisoning attacks) omits the “White/Black-box” column. For clarity, in its first entry, we employed the original library [L.1] for data attacks in a black-box setting, utilizing Python 3.10 and the MNIST dataset. Post-debugging, the outcomes are documented in the evaluation. Ref. | Attack Type | White/Black-Box |  Library Link (Appendix) |  Original | Impl. | Python | GPU | Dataset | Evaluation | Issues
---|---|---|---|---|---|---|---|---|---|---
[51] |  Data Attack |  Black-Box | [L.1] | True | True | 3.10 | False | MNIST |  Training Accuracy: 0.59 Testing Accuracy: 0.57 |  Lost files/ Debugging solution
[L.2] |  False |  True |  3.7 |  False | CIFAR-10 |  Precision: 0.72 Recall: 0.71 F1-Score: 0.71 Accuracy: 0.71 |  Debugging solution
[L.3] |  False |  True |  3.10 |  False | CIFAR-10 |  Precision: 0.54 Recall: 0.90 F1-Score: 0.67 Accuracy: 0.56 |  N/A
[81] |  Data Attack |  Black-Box | [L.4] | True | False | 2.7 | False | CIFAR-10 |  N/A |  Lost files
[L.5] |  False |  True |  3.10 |  False | CIFAR-10 |  Accuracy: 59.43 Training Loss: 135.30 | N/A
[55] |  Data Attack |  Black-Box | [L.6] | True | True | 3.7 | False | Yale |  Training Accuracy: 0.94 Testing Accuracy: 0.69 Attack Accuracy: 0.66 |  Debugging solution
Fashion- MNIST |  Training Accuracy: 0.83 Testing Accuracy: 0.78 Attack Accuracy: 0.52 |  Debugging solution
CIFAR-10 |  Training Accuracy: 0.76 Testing Accuracy: 0.47 Attack Accuracy: 0.67 |  Debugging solution
[69] | Model Attack |  Black-Box |  [L.7] |  True |  True |  3.10 |  False | Location |  Training Accuracy: 1.00 Testing Accuracy: 0.63 Attack Accuracy: 0.69 |  N/A
[85] | Model Attack |  Black-Box |  [L.8] |  True |  True |  2.7 |  False | Location |  Attack Accuracy: 0.50 |  Debugging solution
[68] | Data Attack |  Black-Box |  [L.9] |  True |  True |  3.10 |  True | CIFAR-10 |  Precision: 0.83 Recall: 0.63 F1-Score: 0.72 Accuracy: 0.75 |  N/A
[70] |  Data Attack |  Black-Box | [L.10] | True | True | 3.10 | True | Syntheic |  Training Accuracy: 0.93 Testing Accuracy: 0.92 |  N/A
MNIST |  Training Accuracy: 1.00 Testing Accuracy: 0.99 |  Debugging solution
[72] |  Model Attack |  Black-Box |  [L.11] |  True |  True |  3.10 |  True |  CIFAR-100 |  Recall: 0.10 FPR: 0.40 Max AUC: 0.59 Attack Accuracy: 0.55 |  Debugging solution
N/A |  Data Attack |  Black-Box | [L.12] | N/A | True | 3.7 | True | MNIST |  Attack Accuracy: 0.53 Testing Accuracy: 0.52 |  Debugging solution
CIFAR-10 |  Attack Accuracy: 0.74 Testing Accuracy: 0.73 |  Debugging solution
N/A |  Model Attack |  Black-Box |  [L.13] |  N/A |  True |  3.7 |  False | CIFAR-10 |  Training Accuracy: 0.76 Testing Accuracy: 0.63 |  Debugging solution

###  III-B Adversarial Attacks

Adversarial attack is based on the model’s vulnerability and the sample’s
small perturbation. Attackers cheat or mislead the output of a model by making
small, targeted perturbations to the input data. The process of adversarial
attacks is as shown in Fig. 7.

Szegedy et al. [88] by finding the smallest addition to the loss function, the
neural network makes misclassification and transforms the problem into a
convex optimization. where $f:\mathbb{R}^{m}\rightarrow\\{1...k\\}$ represents
a classifier that maps image pixel value vectors to discrete label sets. In
addition, it is also assumed that $f$ has an associated continuous loss
function, denoted as
$loss_{f}:\mathbb{R}^{m}\times\\{1...k\\}\rightarrow\mathbb{R}^{+}$. For a
given $x\in\mathbb{R}^{m}$ image and target label $l\in\\{1...k\\}.$ The goal
is to solve the following box-constrained optimization problem:

Minimize $||r||_{2}$ subject to:

$\displaystyle f(x+r)=l$ (3) $\displaystyle x+r\in[0,1]^{m}$ (4)

Since the minimum $r$ may not be unique, For an arbitrarily chosen minimizer
$D(x,l)$, denoted by $x+r.$ $D(x,L)$ can be found by row search to find the
minimum value of $c>0$, so that the minimum value of r satisfies $f(x+r)=l.$

$\displaystyle\text{Minimize }c|r|+loss_{f}(x+r,l)$ $\displaystyle\text{
subject to }x+r\in[0,1]^{m}$ (5)

In this section, we show the experimental results in TABLE III and briefly
introduce some early classic articles and current cutting-edge work on
adversarial attacks selected from [88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98,
99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114,
115, 116, 117, 118]. (For some unintroduced papers, we also display them in
TABLE VIII).

Figure 7: Overview of adversarial attacks in ML-based IoT system.

Goodfellow et al. [89] introduced a straightforward method for generating
adversarial samples known as FGSM (Fast Gradient Sign Method), and further
utilized these adversarial samples for adversarial training, demonstrating
that such training can lead to regularization and offer more significant
regularization benefits compared to dropout. Moosavi et al. [90] proposed the
DeepFool algorithm, designed to calculate perturbations capable of deceiving
deep neural networks. DeepFool leverages iterative linearization within a
classifier-based framework to converge upon the smallest perturbations
necessary to misclassify an input image, thereby finding the minimal
distortion required for misclassification. Kurakin et al. [91] extended
adversarial training to the ImageNet dataset, building upon the earlier FGSM
attack method [89]. This extension, referred to as ”basic iterative methods,”
primarily focuses on reducing the loss function of the target category during
training. Carlini et al. [94] leveraged the internal configuration of the
target deep neural network (DNN) for attack bootstrapping. They quantified the
difference between the adversarial attack and the original example using the
$L_{2}$ norm. Additionally, they validated that their method could
successfully defeat defensive distillation [87]. Nicolas Papernot et al. [96]
devised a technique that involves training a local model to replace the target
DNN. This local model is trained using inputs synthesized by the adversary and
labeled by the target DNN. Adversarial examples are crafted by making local
substitutions to identify inputs misclassified by the target DNN. Furthermore,
they explore two defense strategies: Adversarial training [89, 88, 119] and
Defensive distillation [87]. Chen et al. [95] introduced an attack method that
accesses both the input and output of the target DNN. This zero-order
optimization-based attack directly estimates the gradient of the target DNN to
generate adversarial examples, and uses techniques such as zero-order random
coordinate descent, dimensionality reduction, hierarchical attacks, and
importance sampling to effectively attack black-box models. They also discuss
various defense [120, 121, 122, 123, 124] strategies, including detection-
based defenses, gradient and representation masking, as well as adversarial
training.[92, 125, 126, 127, 128]. Ilyas et al. [100] defined three threat
models: query limit setting, partial information setting, and label-only
setting. They proposed the use of NES (Nesterov’s Estimation of the Stochastic
Gradient)[129] as a black-box gradient estimation technique, along with PGD
(Projected Gradient Descent) using estimated gradients for white-box attacks
to construct adversarial examples. The effectiveness of this approach was
demonstrated on ImageNet classifiers under these three threat models. Su et
al. [103] generates a single-pixel adversarial perturbation based on
differential evolution, only modifies one pixel to attack the image, and
deceives more types of networks. They show that non-target attacks can be
launched with considerable accuracy by modifying only one pixel on three
common deep neural network structures. Chen et al. [104] propose the
HopSkipJumpAttack algorithm based on the decision boundary using bipartite
information to estimate the gradient direction, which can approximate the
gradient in the black box case, thereby generating targeted and non-targeted
adversarial samples. In addition, they verified the defense methods of
defensive distillation [87], region-based classification [130], and
adversarial training [89, 91, 92, 126] to show that the attack is competitive.
Wang et al. [131] propose a novel method Triangle Attack (TA) based on the law
of sines, which exploits the geometric information related to the larger
angles in any triangle by optimizing perturbations. With TA, effective
dimensionality reduction can be achieved in the low-frequency space, and
compared to recent work [132, 133, 104, 134, 135, 136], the same attack
success rate can be achieved with fewer queries.

Table III: Reviews of adversarial attack libraries. Taking the first line as an example, we utilized the original library [L.14] to conduct data attacks in a black-box environment. We used Python 3.10, with GPU, and the ImageNet dataset. After debugging, the results are shown in the evaluation. Additionally, it’s worth noting that the final evaluation of this library is based on the output images, and we only present the program’s running results. Ref. | Attack Type | White/Black-Box |  Library Link (Appendix) |  Original | Impl. | Python | GPU | Dataset | Evaluation | Issues
---|---|---|---|---|---|---|---|---|---|---
[90] | Data Attack |  Black-Box | [L.14] | True | True | 3.10 | True |  ImageNet |  Original label: n03538406 horse cart Perturbed label: n02437312 Arabian camel |  Debugging solution
[94] |  Model Attack |  Black-Box |  [L.15] |  True |  True |  3.10 |  True | CIFAR-10 |  L0 Total Distortion: 2.34 L2 Total Distortion: 0.21 |  Debugging solution
MNIST |  L0 Total Distortion: 5.00 L2 Total Distortion: 2.02 |  Debugging solution
[95] | Model Attack |  Black-Box |  [L.16] |  True |  True |  3.7 |  True |  MNIST |  Success = True Const = 0.325 Distortion = 1.80 Success_rate = 1.00 L2_avg = 2.34 |  Debugging solution
[102] |  Model Attack |  Black-Box |  [L.17] |  True |  True |  3.10 |  True |  MNIST |  Accuracy: 0.99 |  Debugging solution
[137] |  Model Attack |  Black-Box |  [L.18] |  True |  True |  3.10 |  True | CIFAR-10 |  Avg_query = 963.38 Accuracy = 1.00 |  Debugging solution
CIFAR-10 |  Avg_query = 1726.11 Accuracy = 0.48 |  Debugging solution
[131] | Model Attack |  Black-Box |  [L.19] |  True |  True |  3.10 |  True |  Image(200) |  Top-1 Accuracy: 1.00 Top-2 Accuracy: 0.93 Top-3 Accuracy: 0.51 |  N/A
[89] | Model Attack | Black-Box | [L.20] | False | True | 3.10 | True |  Fashion- MNIST |  Accuracy: 0.95 |  Debugging solution
[138] |  Accuracy : 0.93 |  Debugging solution
[139] |  Model Attack |  Black-Box |  [L.21] |  True |  True |  3.10 |  True |  ImageNet |  Clean Accuracy: 0.94 Attack Success:0.63 Robust Accuracy:0.38 |  N/A
[109] |  Model Attack |  Black-box/ White-Box |  [L.22] |  True |  True |  3.10 |  True |  CIFAR-10 |  Current EG-Booster Evasion rate on FGS: 0.15 Evasion rate of EG-Booster attack: 0.15 Max change :-0.4925 Avg change :-0.4975 Avg unperturbed positives :15.1719 |  Debugging solution
[97] |  Model Attack |  Black-Box |  [L.23] |  False |  True |  3.7 |  False |  ImageNet |  Mean Squared Error: 756.6158 |  Debugging solution
[140] |  Model Attack |  Black-Box |  [L.24] |  False |  True |  3.7 |  False |  MNIST |  Test Error Rate:0.369 |  Debugging solution
[103] |  Model Attack |  Black-Box |  [L.25] |  True |  True |  3.10 |  True |  CIFAR-10 |  Confidence: 0.075 Prior confidence 0.501 Attack success: True |  Debugging solution
[141] |  Data Attack |  Black-Box |  [L.26] |  True |  True |  3.10 |  True |  Hatefui Memes |  Negative Scores: Original: 0.61 Mask1: 0.00 Mask2: 0.00 |  N/A
[100] |  Model Attack/ Data Attack |  Black-Box |  [L.27] |  True |  True |  3.10 |  True |  ImageNet |  Loss: 1.7815 Lr : 3.13E-04 Eps: 0.050 |  Debugging solution
[104] |  Model Attack |  Black-Box |  [L.28] |  True |  True |  3.10 |  True |  CIFAR-10 |  Accuracy: 0.9244 L2 distance: 1.0344E+00 |  Debugging solution
N/A | Model Attack |  Black-Box |  [L.29] |  N/A |  True |  3.7 |  False |  MNIST |  Testing Accuracy: 0.99 |  Debugging solution

###  III-C Reconstruction Attacks

Reconstruction attacks are based on model output data. The attacker
reconstructs or restores some features of the input data by analyzing the
model output and known information. The process of reconstruction attacks is
as shown in Fig. 8.

The environment for the reverse attack of the model proposed by Fredrikson el
al. [142] is mainly divided into black-and-white box conditions.

Start by pinning the tree, as shown below, where $(x,y)$ represents a target
instance, which may or may not be in the training set.

$\displaystyle f(x)=\sum\nolimits_{i=1}^{m}\omega_{i}\phi_{i}(x)$ (6)

Under black box conditions, the MI Attack algorithm proposed by Fredrikson et
al. [142] was used and fine-tuned for the decision tree scenario. The specific
differences are reflected in the following: the target model can only produce
discrete outputs; the error model information is different, and the confusion
matrix $C$ is used instead of Gaussian. The variance of the distribution. The
definition is as follows:

$\displaystyle err(y,y^{{}^{\prime}})\propto Pr\text{
}[f(x)=y^{{}^{\prime}}|\text{ }\text{y is the true label]}$ (7)

In the white-box condition, the attacker knows each $\phi_{i}$, the total
number of samples in the training set.

$\displaystyle N=\sum\nolimits_{i=1}^{m}n_{i}$ (8)

The number of training samples $n_{i}$ (corresponding to $\phi_{i}$)
introduces a series of paths to the known feature vector $x_{k}:$

$\displaystyle S$ $\displaystyle=\\{{s_{i}}\\}_{1\leq i\leq m}:$
$\displaystyle\quad S=\\{(\phi_{i},n_{i})|\hskip 2.84544pt\exists
x^{\prime}\in\mathbb{R}^{d}.\hskip
2.84544ptx^{\prime}_{k}=x_{k}\land\phi_{i}(x^{\prime})\\}$ (9)

Each path corresponds to a basis function $\phi_{i}$ and a sample count $ni.$
Let $pi=ni/N,$ related to the joint distribution used to construct the
training set features. Use the basis function $\phi_{i}$ to divide the feature
space; v represents the specific value of the first feature, and $v_{k}$
represents the specific value of the $d-1$ dimensional non-private feature.
Finally, as shown below, a white-box predictor with counting (WBWC) is
constructed by outputting $v$ to maximize the formula and return MAP
predictions given additional auxiliary information.

$\displaystyle Pr[{x_{1}}=v|(s_{1}\vee\cdot\cdot\cdot\vee s_{m})\wedge
x_{k}=v_{k}]$ $\displaystyle\propto\sum_{i=1}^{m}\frac{p_{i}\phi_{i}(v)\cdot
Pr[x_{k}=v_{k}\cdot Pr[x_{1}=v]}{\sum_{j=1}^{m}p_{j}\phi_{j}(v)}$
$\displaystyle\propto\frac{1}{\sum_{j=1}^{m}p_{j}\phi_{j}(v)}\sum_{1\leq i\leq
m}p_{i}\phi_{i}(v)\cdot Pr[x_{1}=v]$ (10)

In this section, we show the experimental results in TABLE IV and briefly
introduce some early classic articles and current cutting-edge work on
reconstruction attacks selected from [143, 142, 144, 145, 146, 147, 148, 149,
57, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164,
165, 166, 167, 168, 169, 170]. (For some unintroduced papers, we also display
them in TABLE VIII).

Figure 8: Overview of reconstruction attacks in ML-based IoT system.

Fredrikson et al. [142] utilize the confidence values exposed by an API to
infer the pixel values in an image. They achieve this by iteratively testing
different pixel intensity values to reconstruct the image. Additionally, they
propose countermeasures against such attacks, including the use of decision
trees and reducing the quality or accuracy of gradient information retrieved
from the model. In addition, they highlight that, given an ML model and some
features of a training sample, an attacker can potentially recover the
remaining features of the sample [143]. However, they suggest that carefully
selecting a privacy budget can effectively mitigate model inversion attacks
using differential privacy mechanisms. The attack developed by Hitaj et al.
[145] capitalizes on the real-time nature of the learning process and relies
on Generative Adversarial Networks (GANs). This attack involves merging
generated samples with the attacker’s labels into the training set to recover
the victim’s labeled samples. They suggest that secure multi-party computation
or fully homomorphic encryption could potentially counteract this attack. Zhu
et al. [153] use DLG to obtain training inputs and labels simultaneously in
several iterations. Full display of private training data (including inputs
and labels) after optimization. They also proposes some defense strategies
against this attack, such as noisy gradients, gradient compression [171, 172]
and sparsification large batch, high resolution and cryptology [173, 174].
Without changing the training setting, the most effective defense is gradient
pruning. Wang et al. [57] introduce a general attack framework called mGAN-AI,
which targets user-level privacy by identifying client identities. This
framework was successfully applied to recover samples from specific users in
the MNIST and AT&T datasets. He et al. [151] employ regularized maximum
likelihood estimation in a white-box setting to achieve image restoration. In
a black-box setting, they utilize inverse networks to identify the inverse
mapping from the output to the input. They also offers feasible defense
strategies to mitigate such attacks, including running fully-connected layers
before releasing results, making client-side networks deeper, trusted
execution on untrusted participants, differential privacy, and homomorphic
encryption. Zhang et al. [157] leverage partial public information to
reconstruct private data, extracting general knowledge from public datasets
using GANs to regularize the inversion problem, and demonstrates that this
attack remains highly effective even under constraints imposed on public
datasets, highlighting that the defense strategy of differential privacy may
not be sufficient to resist such attacks. Zhu et al. [160] present a closed-
form recursive procedure for recovering data from gradients of DNN through
private recursive gradient attacks. Additionally, they propose rank analysis
methods for estimating the risk of gradient attacks inherent in certain
network architectures. Kahla et al. [167] introduce the Boundary-Repelling
Model Inversion (BREP-MI) technique, designed to invert private training data
using predicted labels from the target model. This approach involves updating
latent vectors with estimated gradients, enabling the GAN to reconstruct
representative images of the target class from these latent vectors, and they
also validates the effectiveness of this attack on various datasets and model
architectures. Han et al. [168] leverage confidence scores obtained from
reinforcement learning-based GANs to provide rewards to agents. These rewards
guide the agent’s exploration of latent vectors for reconstructing private
data, leading to the successful recovery of private information from target
models across various datasets and models.

Table IV: Reviews of reconstruction attack libraries. Taking the first line as an example, we utilized the original library [L.30] to conduct model attacks in a black-box environment. We used Python 2.7 with CPU, and the AT&T Face dataset. However, the environment setup failed, and we were unable to debug the program to obtain results (e.g, the evaluation column is N/A). Ref. | Attack Type | White/Black-Box |  Library Link (Appendix) |  Original | Impl. | Python | GPU | Dataset | Evaluation | Issues
---|---|---|---|---|---|---|---|---|---|---
[142] |  Model Attack |  Black-Box | [L.30] | True | False | 2.7 | False |  AT&T Face |  N/A |  Environment loading failed
[L.31] |  False |  True |  3.10 |  False |  AT&T Face |  N/A |  Reproduced the function of reversing the image
[151] |  Data Attack |  Black-Box |  [L.32] |  True |  True |  2.7 |  True |  CIFAR-10 |  Test Accuracy: 0.8729 MSE: 0.1973 |  Debugging solution
[175] |  Data Attack |  Black-Box | [L.33] | True | True | 3.10 | True | MNIST |  Average MSE: 0.09 Clone Test Score: 0.05 Accuracy: 1.00 |  N/A
Fashion- MNIST |  Average MSE: 0.16 Clone Test Score:0.10 Accuracy: 1.00 |  Debugging solution
CIFAR-10 |  Average MSE: 0.10 Clone Test Score: 0.09 Accuracy: 1.00 |  Debugging solution
[145] |  Data Attack |  Black-Box | [L.34] | False | True | 3.10 | True | MNIST |  GAN: Testing Accuracy: 0.95 Tseting Loss: 0.19 |  N/A
Federated_GAN: Testing Accuracy: 0.97 Testing Loss: 0.18 |  Debugging solution
[167] |  Model Attack |  Black-Box |  [L.35] |  True |  True |  3.10 |  True |  CelebA |  Total Accuracy: 0.90 |  Debugging solution
[168] |  Model Attack |  Black-Box |  [L.36] |  True |  True |  3.10 |  True |  CelebA |  Confidence score for the target model : 0.9997 Accuracy : 1.000 Top-5 Accuracy : 1.000 |  Debugging solution
[166] |  Model Attack |  Black-Box |  [L.37] |  True |  True |  3.10 |  False |  MNIST |  Training_Loss: 0.02 Testing_Loss: 0.06 Testing_MAE: 0.04 Testing_MSE: 0.02 |  Debugging solution
[160] |  Data Attack |  Black-Box |  [L.38] |  True |  True |  3.10 |  True |  CIFAR-10 |  Lstsq Residual: 0.01062967 max/min singular value: 1.35e+00/1.40e-02 |  N/A
[152] |  Model Attack |  Black-Box |  [L.39] |  False |  False |  3.10 |  True |  CIFAR-10 |  N/A |  Environment loading failed
[146] |  Model Attack |  Black-Box |  [L.40] |  True |  False |  3.10 |  True |  CelebA |  N/A |  Debugging failed
[176] |  Data Attack |  Black-Box | [L.41] | True | True | 3.10 | True | MNIST |  Loss: 2.18e-08 MSE: 2.25e-08 |  N/A
Loss: 7.98e-06 MSE: 4.79e-05 |  Debugging solution
N/A | Data Attack |  Black-Box |  [L.42] |  N/A |  True |  3.10 |  True | UTKFace |  Accuracy: 0.47 |  Debugging failed
N/A | Data Attack |  Black-Box |  [L.43] |  N/A |  True |  3.10 |  True | CelebA |  Testing Accuracy:0.55 |  Debugging failed

###  III-D Property Inference Attacks

Property inference attacks are based on the correlation between model output
and known information. The attacker infers some sensitive properties of the
input data by analyzing the relationship between the model output and known
information. For example, in Fig. 9. The attacker does not have access to the
original dataset, but has access to the target model. If the attacker’s target
is property A, he needs to create a data set containing A, and then train
multiple shadow models based on the created data set, with the same tasks as
the target model, such as binary classification or multi-classification.
Afterwards, the meta-classifier is trained on the shadow model in a supervised
manner until the meta-classifier is able to infer the training data property A
or B from the target model.

Melis et al. [177] assume that the opponent has auxiliary data with property
data points $({D^{adv}_{prop}})$ and non-property data points
$({D^{adv}_{nonprop}}).$ The attacker uses the snapshot of the global model to
generate aggregate updates of property data and updates of non-property data
and labels the generated samples, thereby training a binary batch property
classifier. In passive property inference attacks, the attacker observes
updates and performs inference without changing the collaborative training
content. Given a model snapshot $\theta_{t},$ calculate the gradient
$g_{prop}$; $({D^{adv}_{prop}})$ with property data and the gradient
$g_{nonprop}$; $({D^{adv}_{nonprop}})$ without property data. After collecting
enough label gradients, train the binary classifier $f_{prop.}$

$\displaystyle g_{prop}=\bigtriangledown L({b^{adv}_{prop}};\theta_{t})$
$\displaystyle G_{prop}\leftarrow G_{prop}\cup\\{g_{prop}\\}$ (11)
$\displaystyle g_{prop}=\bigtriangledown L({b^{adv}_{nonprop}};\theta_{t})$
$\displaystyle G_{nonprop}\leftarrow G_{nonprop}\cup\\{g_{nonprop}\\}$ (12)

In an active property inference attack, the attacker extends a local copy of
the collaboratively trained model with an enhanced attribute classifier
connected to the last layer. The trained model performs well on the main task
and recognizes batch attributes. For training data with each record having a
primary label $y$ and an attribute label $p$, the joint loss of the model is
calculated as:

$\displaystyle L_{mt}=\alpha\cdot(x,y;\theta)+(1-\alpha)\cdot L(x,p;\theta)$
(13)

In this section, we show the experimental results in TABLE V and briefly
introduce some early classic articles and current cutting-edge work on
property inference attacks selected from [178, 179, 177, 180, 181, 182, 183,
184, 185, 186, 187, 188, 189, 190, 191]. (For some unintroduced papers, we
also display them in TABLE VIII).

Figure 9: Overview of property inference attacks in ML-based IoT system.

Ateniese et al. [178] establish a new meta-classifier and by training the
meta-classifier to crack other classifiers, obtain meaningful information
about the training set of other classifiers. In contrast to previous work
[192, 193, 194, 195, 196], their approach focuses on statistics strictly
related to the training samples used in the learning phase. Ganju et al. [179]
take advantage of the fact that FCNN is invariant under the arrangement of
nodes in each layer to simplify the information extraction task, and put
forward a feasible defense strategy for this attack, such as using node
multiplicative transformations, adding noisy data to the training set and
encoding arbitrary Information. Melis et al. [177] use the collected data to
generate aggregate updates and then train a binary attribute classifier and
upload the joint loss function to determine whether the training data has the
attribute. They also evaluate sharing fewer gradients [197], dimensionality
reduction, dropout [81], and participant-level differential privacy [198,
199], find that they cannot effectively prevent attacks. For over-learning,
Song et al. [180] developed a new transfer learning-based technique to evolve
a benign training model into a malicious privacy-invading task model. A new
learning-based review method is developed to transform the reviewed
representation into a different form to extract relevant attribute information
of interest. To achieve agent-level privacy inference in the training phase,
Xu et al. [181] propose CycleGAN to reconstruct the impact of data with
specific attributes on the global model and implement gradient generation for
inference attacks. Whether it is data points or some statistical features, the
method can infer training data based on model parameters or intermediate
outputs. Zhang et al. [185] builds an attack vector based on a series of
queries, performs hundreds of inference queries on the model, and jointly
infers the attributes of a data set. They also point out that existing secure
computing [200] and differential privacy techniques [201, 202] are either not
directly applicable to certain scenarios or are costly. Zhou et al. [184]
generates generated samples by querying the target generator, then builds an
attribute classifier to classify the generated samples, and finally classifies
them according to the attributes of interest and then predicts the target
attributes based on the output of the attribute classifier. In addition, they
verify the feasibility of the attack on five property inference tasks, four
datasets, and four victim GAN models. To infer whether the training dataset is
gender-balanced or not. Parisot et al. [182] train an attack model and input
the target model’s weights, and output the probability that the dataset used
to train the target model has the attribute or does not have the attribute.
Malekzadeh et al. [183] proposes to train an honest but curious (HBC)
classifier, and using the classifier can accurately predict its target
attributes, and can also use the characteristics of its output to encode
sensitive attributes to achieve attacks secretly. They also propose to add
random noise to the output of the model, which is then shared to achieve
defense. Mahloujifar et al. [189] point out that the adversary selects the
attributes to attack, then uploads the input data in the poisoned
distribution, and finally uses black-box queries (label queries only) on the
trained model to determine the frequency of the selected attributes.
Furthermore, they show that high attack accuracy can be achieved with a low
degree of poisoning.

Table V: Reviews of property inference attack libraries. Taking the first line as an example, we utilized the original library [L.44] to conduct data attacks in a black-box environment. We used Python 2.7 with GPU, and the LFW dataset. After debugging, the results are shown in the evaluation. Ref. | Attack Type | White/Black-Box |  Library Link (Appendix) |  Original | Impl. | Python | GPU | Dataset | Evaluation | Issues
---|---|---|---|---|---|---|---|---|---|---
[177] | Data Attack |  Black-Box |  [L.44] |  True |  True |  2.7 |  True | LFW |  Precision: 1.00 Recall: 0.99 F1-Score: 1.00 AUC: 1.00 |  Debugging Solution
[183] |  Model Attack |  Black-Box |  [L.45] |  True |  True |  3.10 |  True | UTKFace |  Testing Accuracy: 0.82 |  Debugging Solution
[203] | Data Attack |  Black-Box |  [L.46] |  True |  False |  3.10 |  True | CelebA |  N/A |  Debugging failed
[180] |  Model Attack |  Black-Box |  [L.47] |  True |  False |  3.10 |  True | UTKFace |  N/A |  Environment loading failed
[204] |  Data Attack |  Black-box/ White-Box |  [L.48] |  True |  True |  3.10 |  True | N/A |  Output as picture |  Debugging Solution
[205] |  Model Attack |  Black-Box |  [L.49] |  True |  False |  3.10 |  True | CelebA |  N/A |  Environment loading failed
[182] |  Data Attack |  Black-Box |  [L.50] |  True |  False |  3.10 |  True | CelebA |  N/A |  Environment loading failed
[206] |  Model Attack |  N/A |  [L.51] |  True |  False |  3.10 |  True | MNIST |  Validation: 0.0193 |  N/A

###  III-E Model Extraction Attacks

Model extraction attacks are based on training data and black-box model input
and output. By observing the input and output of the black box model, the
attacker infers the characteristics and rules of the input data and output
results so as to train the approximate model to simulate the behavior of the
black box model. The process is shown in Fig. 10.

Tramèr et al. [207] model the adversary as a stochastic algorithm. The
adversary’s goal is to use as few query inputs $f$ as possible in order to
efficiently compute an approximation that closely matches the input $f$.
“Tight matching” is formalized using two different error measures.

This is the average error on the test set $D$, a lower test error means that
for inputs distributed like the training data samples, $\hat{f}$ matches $f$
well.

$\displaystyle R_{test}=(f,\hat{f})=\sum\nolimits_{{(x,y)}\in
D}d(f(x),\hat{f}(x))/|D|$ (14)

For a set $U$ of vectors chosen uniformly in $\chi$, In the following formula,
$R_{unif}$ estimates the proportion of the complete feature space where $f$
and $\hat{f}$ are inconsistent:

$\displaystyle R_{unif}=(f,\hat{f})=\sum\nolimits_{{x}\in
U}d(f(x),\hat{f}(x))/|U|$ (15)

In this section, we show the experimental results in TABLE VI and briefly
introduce some early classic articles and current cutting-edge work on model
extraction attacks selected from [207, 208, 209, 210, 211, 212, 213, 214, 215,
216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230,
231, 232, 233, 234, 235]. (For some unintroduced papers, we also display them
in TABLE VIII).

Figure 10: Overview of model extraction attacks in ML-based IoT system.

Tramèr et al. [207] describe an approach to access a target model as a black
box, input data, observe the model’s output, and construct a training dataset
based on these outputs, and extract parameters or structural information from
the model using techniques like querying the prediction API and solving
equations. They discuss several defense strategies, including rounding
confidences [142], differential privacy [236, 237], and ensemble methods. Wang
et al. [208] propose a general attack framework for stealing hyperparameters.
They compute the gradient of the objective function at the model parameters
and use linear least squares to estimate the hyperparameters. Furthermore,
they demonstrate that $L$2 regularization can be more effective in defending
against this attack. The defensive effects of cross-entropy and square hinge
loss are similar. Correia-Silva et al. [209] query the target network with
random unlabeled data, creating a fake dataset using input image and output
label pairs to imitate the target network’s knowledge, and train an imitation
network on this fake dataset to achieve similar performance to the target
network. They experimentally show that a CNN can be created by treating the
target network query as a black box for random unlabeled data. In contrast to
previous work [211, 96, 207], Orekondy et al. [212] impose constraints on the
adversary’s capabilities. The adversary first queries a black-box model for a
set of input images to obtain predictions and then uses these predictions to
train similar images. Although this method makes fewer black-box assumptions,
it demonstrates the power of the attack. Juuti et al. [215] access an
independent target model as a black box, obtain the output responses for a
corresponding query set, initialize the model’s hyperparameters, synthesize
samples, and iteratively train the model. In addition, They discuss feasible
defense methods, such as modifying the model’s predictions [207, 238], and
detecting adversarial examples [121, 239]. Oh et al. [211] introduce a meta-
model called kennen, which can infer model attributes from static query inputs
and actively optimize query inputs to extract more information. They also
demonstrate that even with just 100 single-label outputs, a wealth of
information about black boxes can be revealed, and black-box access can be
predicted with high accuracy even as the metamodel’s performance degrades.
Batina et al. [214] base their approach on reverse engineering of neural
networks using side-channel analysis techniques. They collect measured data
corresponding to the selected network implementation and recover key
parameters. Additionally, they propose several defense strategies, such as
shuffling [240], masking, and constant-time implementations of exponentiation
[241]. Jagielski et al. [218] develop a learning-based attack aimed at
extracting high-precision models, using the victim to supervise the training
of the extracted model. Using a hybrid approach, a learning-based approach is
utilized to recover from numerical instability errors while performing a
functional equivalence extraction attack. It is worth noting that this method
does not require training and can directly extract the model’s weights.
Carlini et al. [227] describe an attack that guides the model to generate
diverse output results through different sampling strategies to discover the
samples memorized by the model, and use a specified measurement method to
measure the model’s familiarity with the samples and determine which samples
are memorized. They also propose mitigation strategies, such as training with
differential privacy [83], curating the training data to limit the impact of
memorization on downstream applications, and auditing ML models for
memorization. Wu et al. [230] propose a model extraction attack on Graph
Neural Networks (GNNs), and target a private GNN model, constructing a replica
model with similar functionality by querying the sequence of nodes obtained
from the target model. They demonstrate that this attack method can
effectively extract duplicate models resembling the target model and achieve
highly accurate prediction results.

Table VI: Reviews of model extraction attack libraries. Taking the first line as an example, we utilized the original library [L.52] to conduct model attacks in a black-box environment. We used Python 2.7 with CPU, and the MNIST dataset. After debugging, the results are shown in the evaluation. Ref. | Attack Type | White/Black-Box |  Library Link (Appendix) |  Original | Impl. | Python | GPU | Dataset | Evaluation | Issues
---|---|---|---|---|---|---|---|---|---|---
[207] |  Model Attack |  Black-Box | [L.52] | True | True | 2.7 | False | MNIST |  Training Accuracy: 0.96 |  Debugging solution
Circles |  Training Accuracy: 0.44 |  Debugging solution
Blobs |  Training Accuracy: 1.00 |  Debugging solution
Adult |  Training Accuracy: 0.88 |  Debugging solution
Moons |  Training Accuracy: 0.86 |  Debugging solution
Adult_b |  Training Accuracy: 0.85 |  Debugging solution
Class5 |  Training Accuracy: 0.57 |  Debugging solution
Digits |  Training Accuracy: 0.98 |  Debugging solution
Face |  Training Accuracy: 0.96 |  Debugging solution
Steak |  Training Accuracy: 0.44 |  Debugging solution
Gss |  Training Accuracy: 0.64 |  Debugging solution
Iris |  Training Accuracy: 0.97 |  Debugging solution
Diabetes |  Training Accuracy: 0.79 |  Debugging solution
Mushrooms |  Training Accuracy: 1.00 |  Debugging solution
Cancer |  Training accuracy: 0.96 |  Debugging solution
[230] |  Model Attack |  Black-Box/ White-Box |  [L.53] |  True |  True |  2.7 |  False |  Cora |  Accuracy:0.13 |  Debugging solution
[227] |  Data Attack |  Black-Box |  [L.54] |  True |  True |  3.10 |  True |  Common Crawl |  Top sample by ratio of Zlib entropy and XL perplexity: PPL-XL: 12.805 Zlib: 248.000 Score: 97.260 |  Debugging solution
[242] |  Data Attack |  Black-Box |  [L.55] |  True |  True |  3.10 |  True |  CIFAR-10 |  Accuracy: 0.83 |  Debugging solution
[219] |  Model Attack |  Black-Box |  [L.56] |  True |  True |  3.7 |  False |  N/A (By random training data) |  Maximum logit loss on the unit sphere: 4.47e-12 |  Debugging solution
[211] |  Data Attack |  Black-Box |  [L.57] |  True |  True |  2.7 |  True |  MNIST |  Average Loss: 0.0374 Accuracy: 0.9894 |  Debugging solution
[231] |  Data Attack |  Black-Box |  [L.58] |  False |  False |  3.10 |  True | CIFAR-10 |  N/A |  Debugging failed
[209] |  Data Attack |  Black-Box |  [L.59] |  False |  False |  3.10 |  True | CIFAR-10 |  N/A |  Debugging failed

###  III-F Poisoning Attacks

Poisoning attacks are based on tampering with the training data set or
directly acting on the model to affect the training process of the machine
learning system, guiding the model to learn wrong patterns or make wrong
decisions. The process is shown in Fig. 10.

Let $x$ denote the function that propagates the input $x$ through the network
to the penultimate layer (before the softmax layer). Due to the high
complexity and nonlinearity of $f$, it is possible to find a sample that
“collides” with the target in the feature space and is close to the base
sample b of the input space through the following formula $b.$

$\displaystyle p=\underset{x}{argmin}\text{
}{||f(x)-f(t)||^{2}_{2}}+\beta\text{ }{||x-b||^{2}_{2}}$ (16)

In order to prevent the separation of poisoned samples and targets during the
training process, Shafahi et al. [243] use a simple but effective trick:
adding a low-opacity watermark of the target sample to the poisoned samples,
thereby allowing certain inseparable features to overlap while maintaining
visual different. By weighted combination of the base image $b$ and the target
image $t$, a basic watermark image with target opacity $\gamma$ is formed:

$\displaystyle b\leftarrow\gamma\cdot t+(1-\gamma)\cdot b$ (17)

In this section, we show the experimental results in TABLE VII and briefly
introduce some early classic articles and current cutting-edge work on
poisoning attacks selected from [244, 245, 246, 247, 243, 248, 249, 250, 251,
252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262]. (For some unintroduced
papers, we also display them in TABLE VIII).

Figure 11: Overview of poisoning attacks in ML-based IoT system.

Biggio et al. [244] employs a gradient rising strategy, allowing adversaries
to predict changes in the decision function of Support Vector Machines (SVM)
caused by malicious input. They use this capability to construct malicious
data and demonstrate that the gradient ascent process, coupled with accurate
recognition, can increase the test error of the classifier. Alfeld et al.
[245] propose a general mathematical framework and encode real-world settings
into this framework. This enables the optimization problem to be solved using
standard convex optimization methods. They demonstrate the attack’s
effectiveness on synthetic and real-world time-series data, comparing it to
random and greedy baselines. Shafahi et al. [243] introduce an optimization-
based method for crafting poison data. In transfer learning scenarios, a
single poison image can control the classifier’s behavior. They also propose a
”watermarking” strategy for full end-to-end training, using multiple poisoning
training instances to enhance the reliability of poisoning attacks. Jagielski
et al. [249] adapt a model initially designed for poisoning attacks on
classification problems to regression problems. They design a fast statistical
attack that requires limited information to be effective. Additionally, they
propose a defense algorithm called TRIM [249], which outperforms traditional
robust statistics [263, 264]. An alternate minimization strategy is presented
by Bhagoji et al. [250], involving the optimization of training loss and the
adversarial objective alternately. This approach utilizes a set of
explainability techniques to conduct model poisoning attacks while maintaining
stealth. They demonstrate that model poisoning attacks can be carried out
stealthily despite strict adversary constraints. Fang et al. [254] propose a
local model poisoning attack for Byzantine inverse federated learning. The
attack manipulates local model parameters on compromised working devices
during the learning process. Furthermore, they introduce RONI [47] and TRIM
[249] as defenses against poisoning attacks. Tolpegin et al. [251] exploit a
malicious subset of actors in the system to poison the global model by sending
model updates originating from mislabeled data. Additionally, they propose a
defense mechanism that enables aggregators to identify malicious actors. Huang
et al. [255] aim to manipulate recommender systems by recommending target
items to users. They construct a poison model, generate fake users, and inject
them into the recommendation system to influence recommendations to regular
users. They also validate the effectiveness and competitiveness of the attack
by deploying detectors. Zhang et al. [253] introduce RAPU-G, which leverages
user and item data within a two-tier optimization framework to generate fake
user and item interactions. Furthermore, they employ RAPU-R, in conjunction
with context-specific heuristic rules, to handle data incompleteness and
perturbation. Rong et al. [257] propose FedRecAttack to increase the exposure
rate of the target project. By approximating user feature vectors using public
interactions, attackers can generate corresponding toxic gradients and
manipulate malicious users to upload these toxic gradients.

Table VII: Reviews of poisoning attack libraries. Taking the first line as an example, we utilized the original library [L.60] to conduct data attacks. We used Python 3.7 with CPU and the Fashion-MNIST dataset. After debugging, the results are shown in the evaluation. Ref. | Attack Type |  Library Link (Appendix) |  Original | Impl. | Python | GPU | Dataset | Evaluation | Issues
---|---|---|---|---|---|---|---|---|---
[250] | Model Attack | [L.60] | True | True | 3.7 | False |  Fashion- MNIST |  Accuracy: 0.9919 |  Debugging solution
[251] | Data Attack | [L.61] | True | True | 3.7 | False |  Fashion- MNIST |  Precison: 79 Recall: 0.96 F1-Score: 0.85 Accuracy: 0.85 |  N/A
[257] | Model Attack | [L.62] | True | True | 3.10 | True |  MovieLens |  Loss = 21.78348 |  Debugging solution
[253] |  Data Attack |  [L.63] |  True |  True |  3.10 |  True |  MovieLens |  RAPU_R: Recall: 0.1825 Precision: 0.0024 MRR: 0.0099 |  Debugging solution
RAPU_R: Recall: 0.1825 Precision: 0.0024 MRR: 0.0099 |  Debugging solution
N/A | Data Attack | [L.64] | N/A | True | 3.7 | False | CIFAR-10 |  True_Norm_Accuracy: 1.00 True_Poisoned_Accuracy: 0.98 Norm_Poisoned_Accuracy: 0.98 |  Debugging solution
N/A | Model Attack | [L.65] | N/A | True | 3.10 | True | Adult |  Training Accuracy: 0.7287 Testing Accuracy: 0.7299 |  Debugging solution

##  IV Discussion

In the rapidly expanding landscape of ML-based IoT systems, addressing
security challenges becomes crucial. The taxonomy of attacks has grown to six
categories: membership inference, adversarial, reconstruction, property
inference, model extraction, and poisoning attacks. Protecting data security
and privacy requires collaborative efforts, extending beyond technical and
academic circles to society. The integration of IoT adds complexity, demanding
advanced cryptographic techniques, secure communication protocols, and anomaly
detection. Expect increased research for innovative solutions to enhance
security and resilience. Privacy-preserving methods, robust models, and strict
security protocols are essential. Public, industry, and stakeholders must
uphold ethical standards and legal regulations to ensure privacy and data
protection. Collaborative synergies, continuous scholarly inquiry, and
innovative breakthroughs are imperative for leveraging ML-based IoT systems
for societal advancement and well-being.

Table VIII: Summary of papers on six types of attacks on ML-based IoT systems.
We summarized previous work representative of each type of attack according to
year. (The heat map above illustrates the quantities of classical papers and
projects. Each row corresponds to the inheritance relationships of respective
attacks over the years. The table below provides detailed references).

Year Type | MIA | AA | RA | PIA | MEA | PA
---|---|---|---|---|---|---
2012 | N/A | N/A | N/A | N/A | N/A | [244]
2013 | N/A | N/A | N/A | N/A | N/A | N/A
2014 | N/A | [89] | [143] | N/A | N/A | N/A
2015 | N/A | N/A | [142] | [178] | N/A | N/A
2016 | N/A | [90, 91] | [144] | N/A | [207] | [245]
2017 | [51] |  [92, 93, 94, 95, 96, 97] | [145, 146, 147] | N/A | [145] | [246, 247]
2018 | [52, 53, 54] | [98, 99, 100] | [148, 149] | [179] | [208, 209, 210] | [243, 248, 249]
2019 |  [55, 56, 57, 58, 59, 60, 61, 62] | [101, 102, 103] |  [150, 151, 152, 153, 154, 155] | [177, 180] |  [211, 212, 213, 214, 215, 216, 217] | [250]
2020 | [63, 64, 65] | [104, 105, 106] |  [156, 157, 158, 159, 160] | [181] |  [218, 219, 220, 221, 222] | [251, 252]
2021 |  [67, 68, 69, 66, 70, 71] | [107, 108] | [161, 162, 163, 167] |  [182, 183, 184, 185, 186] |  [223, 224, 225, 226, 227, 228, 229] | [253, 254, 255, 256]
2022 | [72, 73, 74] | [109, 110, 111, 112] | [165, 166] | [187, 188, 189] | [230, 232, 231, 233] | [257, 258, 259]
2023 |  [75, 76, 77, 78, 79, 80] |  [113, 114, 115, 116, 117, 118] | [168, 169, 170] | [190, 191] | [234, 235] | [260, 261, 262]

###  IV-A Experimental Configurations

In this study, we adopt two different experimental configurations: one part is
performed on GPU, and the other part is performed on CPU (The environments
corresponding to the specific experiments have been shown in TABLE II to TABLE
VII). We used the Ubuntu 20.04 LTS operating system in the experiment to
maintain the consistency of the experimental environment and fully explore
attacks under different hardware environments.

####  IV-A1 GPU Experimental Configuration

In the GPU experiment part, we used NVIDIA Tesla P40 GPU, paired with Intel
Xeon E5-2683 v4 processor, equipped with 24GB GPU memory, 32GB RAM, and 1TB
hard drive.

####  IV-A2 CPU Experimental Configuration

In the CPU experiment part, we selected the Intel Core i5-7500 processor,
which has 4 cores, 8GB of RAM, and 1TB hard drive.

###  IV-B Open Source Project Evaluation

We conducted an extensive review of open-source projects to assess the
performance and usability of our attacks. Our evaluation focused on two main
attack categories: data attacks and model attacks. We analyzed these open-
source projects for both white-box and black-box characteristics. To provide a
comprehensive perspective, we categorized the projects into two groups:
original libraries (open-sourced by the authors of this article) and
unoriginal libraries (contributed by individuals or entities). This
classification allowed us to gauge the overall usability of these open-source
resources. Our evaluation process also included tracking the implementation of
these projects to determine their reproducibility and practicality. We used
various Python versions, including Python 2.7, Python 3.7, and Python 3.10, to
ensure compatibility with different library environments. We noted whether
these open-source projects supported GPUs, taking hardware dependencies into
account. Additionally, we documented the dataset used in these open-source
projects and provided comprehensive evaluation results, including performance
metrics that highlight the strengths and weaknesses of each project.
Furthermore, we identified common issues encountered during the testing of
these projects and offered corresponding solutions or categorized the types of
problems faced. This document serves as a valuable resource for other
researchers, offering insights into the usability and limitations of these
projects.

###  IV-C Adaptability to Different Hardware Configurations and Python
Versions

Throughout the experiment, we faced several challenges, including variations
in hardware configurations, Python versions, and parameter settings within
open-source projects. We took measures to resolve these challenges and ensure
the experiment’s validity.

####  IV-C1 Adaptability to Different Hardware Configurations

We assessed adaptability to various hardware configurations by conducting
tests in both GPU and CPU environments.

####  IV-C2 Python Version Differences

Given that different open-source projects may demand varying Python versions,
we selected the appropriate Python version for compatibility based on each
project’s specific requirements.

####  IV-C3 Parameter Setting Issues

Some open-source projects had excessively large parameters set by their
authors, resulting in lengthy execution times. To address this, we fine-tuned
the parameters to ensure that experiments ran within acceptable time frames,
considering our available computing resources.

###  IV-D Suggestions and Explorations

####  IV-D1 Recommendation on Library Usage

Based on the experimental results we have presented, we recommend that
researchers prioritize the use of original libraries, as unoriginal libraries
possess certain limitations, such as cases where individuals or entities
utilized frameworks different from those of the authors [51] (original library
[L.1] and unoriginal libraries [L.2], [L.3]), faced changes in attack
scenarios [51] (original library [L.1] and unoriginal libraries [L.2]), have
implemented only specific algorithms from a research paper [142] (original
library [L.1] and unoriginal libraries [L.30], [L.31]), or observed results
that differ from the paper’s findings (Our main features are availability,
operability, etc.). However, it is important to acknowledge that some original
libraries might present usability challenges owing to delays in maintenance by
their authors. In these circumstances, the presence of unoriginal libraries
can effectively address these concerns.

####  IV-D2 Choice of Python Versions

Additionally, we strongly advise researchers to opt for newer Python versions.
This is crucial because some Python libraries may no longer be compatible with
older versions. Moreover, older Python versions can introduce complexities
during the environment setup phase, such as difficulties in downloading
specific Python libraries using standard installation commands.

####  IV-D3 Common Limitations Across Libraries

Each library boasts its unique characteristics and distinctions. Nevertheless,
they all share common limitations. For example, some authors may provide
overly simplified documentation that proves challenging to comprehend, or
certain open-source libraries may offer datasets that are relatively limited
in diversity compared to those mentioned in the corresponding research papers.
It’s worth noting that during our experiments, we observed similarities
between certain research papers and libraries. For instance, literature [51]
(library [L.1]) and [81] (library [L.4]) shared identical core attack
methodologies, with the latter introducing more relaxed assumptions to extend
the former’s work. Furthermore, the Python package FoolBox streamlines
available adversarial attack techniques and provides a unified API for various
deep learning frameworks.

##  V Conclusion

This paper delves into an extensive examination of various security attack
types pertinent to machine learning as applied in ML-based IoT. We cover a
broad spectrum of threats including membership inference attacks, adversarial
attacks, reconstruction attacks, property inference attacks, model extraction
attacks, and poisoning attacks, all of which present formidable challenges to
the integrity, confidentiality, and availability of machine learning-based ML-
based IoT frameworks. Our analysis is multi-faceted, encompassing different
aspects of the attack landscape, attacker capabilities, and the security
properties at risk. We provide an all-encompassing perspective, crucial for
understanding and mitigating threats within ML-based IoT environments infused
with machine learning. In addition, we have conducted a rigorous evaluation of
65 libraries specifically geared toward machine learning security in ML-based
IoT settings. Our work contributes to this ongoing effort, aiming to fortify
machine learning applications against malicious actors and unforeseen
vulnerabilities in the ever-evolving ML-based IoT landscape.

##  VI Acknowledgement

We thank Dr. Yusen Wu for his help and comments. Dr. Chao Liu is supported by
the Fujian Province Industry-University-Research Collaboration Project
(Project Number: 2023H6034) and the Fujian Province Project for Young and
Middle-Aged Researchers (Project Number: JAT220827).

## References

  * [1] A. L. M. Neto, A. L. Souza, I. Cunha, M. Nogueira, I. O. Nunes, L. Cotta, N. Gentille, A. A. Loureiro, D. F. Aranha, H. K. Patil _et al._ , “Aot: Authentication and access control for the entire iot device life-cycle,” in _Proceedings of the 14th ACM Conference on Embedded Network Sensor Systems CD-ROM_ , 2016, pp. 1–15.
  * [2] K. Rabah _et al._ , “Convergence of ai, iot, big data and blockchain: a review,” _The lake institute Journal_ , vol. 1, no. 1, pp. 1–18, 2018.
  * [3] L. Atzori, A. Iera, and G. Morabito, “The internet of things: A survey,” _Computer networks_ , vol. 54, no. 15, pp. 2787–2805, 2010.
  * [4] D. Miorandi, S. Sicari, F. De Pellegrini, and I. Chlamtac, “Internet of things: Vision, applications and research challenges,” _Ad hoc networks_ , vol. 10, no. 7, pp. 1497–1516, 2012.
  * [5] D. Bandyopadhyay and J. Sen, “Internet of things: Applications and challenges in technology and standardization,” _Wireless personal communications_ , vol. 58, pp. 49–69, 2011.
  * [6] M. Antonakakis, T. April, M. Bailey, M. Bernhard, E. Bursztein, J. Cochran, Z. Durumeric, J. A. Halderman, L. Invernizzi, M. Kallitsis _et al._ , “Understanding the mirai botnet,” in _26th USENIX security symposium (USENIX Security 17)_ , 2017, pp. 1093–1110.
  * [7] K. Yang, Q. Li, and L. Sun, “Towards automatic fingerprinting of iot devices in the cyberspace,” _Computer networks_ , vol. 148, pp. 318–327, 2019.
  * [8] J. Caballero, S. Venkataraman, P. Poosankam, M. G. Kang, D. Song, and A. Blum, “Fig: Automatic fingerprint generation,” 2007.
  * [9] H. Griffioen and C. Doerr, “Examining mirai’s battle over the internet of things,” in _Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security_ , 2020, pp. 743–756.
  * [10] O. Çetin, C. Ganán, L. Altena, T. Kasama, D. Inoue, K. Tamiya, Y. Tie, K. Yoshioka, and M. Van Eeten, “Cleaning up the internet of evil things: Real-world evidence on isp and consumer efforts to remove mirai.” in _NDSS_ , 2019.
  * [11] E. Downing, Y. Mirsky, K. Park, and W. Lee, “$\\{$DeepReflect$\\}$: Discovering malicious functionality through binary reconstruction,” in _30th USENIX Security Symposium (USENIX Security 21)_ , 2021, pp. 3469–3486.
  * [12] O. Alrawi, M. Ike, M. Pruett, R. P. Kasturi, S. Barua, T. Hirani, B. Hill, and B. Saltaformaggio, “Forecasting malware capabilities from cyber attack memory images,” in _30th USENIX Security Symposium (USENIX Security 21)_ , 2021, pp. 3523–3540.
  * [13] Q. Wang, W. U. Hassan, D. Li, K. Jee, X. Yu, K. Zou, J. Rhee, Z. Chen, W. Cheng, C. A. Gunter _et al._ , “You are what you do: Hunting stealthy malware via data provenance analysis.” in _NDSS_ , 2020.
  * [14] A. Jamal, M. F. Hayat, and M. Nasir, “Malware detection and classification in iot network using ann,” _Mehran University Research Journal Of Engineering & Technology_, vol. 41, no. 1, pp. 80–91, 2022.
  * [15] T. Qiu, J. Chi, X. Zhou, Z. Ning, M. Atiquzzaman, and D. O. Wu, “Edge computing in industrial internet of things: Architecture, advances and challenges,” _IEEE Communications Surveys & Tutorials_, vol. 22, no. 4, pp. 2462–2488, 2020.
  * [16] N. Kato, B. Mao, F. Tang, Y. Kawamoto, and J. Liu, “Ten challenges in advancing machine learning technologies toward 6g,” _IEEE Wireless Communications_ , vol. 27, no. 3, pp. 96–103, 2020.
  * [17] H. Kim and N. Feamster, “Improving network management with software defined networking,” _IEEE Communications Magazine_ , vol. 51, no. 2, pp. 114–119, 2013.
  * [18] H. J. Kim, M. Y. Jung, W. S. Chin, and J. W. Jang, “Identifying service contexts for qos support in iot service oriented software defined networks,” in _Mobile, Secure, and Programmable Networking: Third International Conference, MSPN 2017, Paris, France, June 29-30, 2017, Revised Selected Papers 3_. Springer, 2017, pp. 99–108.
  * [19] D. Vukobratovic, D. Jakovetic, V. Skachek, D. Bajovic, D. Sejdinovic, G. K. Kurt, C. Hollanti, and I. Fischer, “Condense: A reconfigurable knowledge acquisition architecture for future 5g iot,” _IEEE Access_ , vol. 4, pp. 3360–3378, 2016.
  * [20] L. J. Jagadeesan and V. Mendiratta, “Programming the network: Application software faults in software-defined networks,” in _2016 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)_. IEEE, 2016, pp. 125–131.
  * [21] S. O. Uwagbole, W. J. Buchanan, and L. Fan, “An applied pattern-driven corpus to predictive analytics in mitigating sql injection attack,” in _2017 Seventh International Conference on Emerging Security Technologies (EST)_. IEEE, 2017, pp. 12–17.
  * [22] M. E. Ahmed, H. Kim, and M. Park, “Mitigating dns query-based ddos attacks with machine learning on software-defined networking,” in _MILCOM 2017-2017 IEEE Military Communications Conference (MILCOM)_. IEEE, 2017, pp. 11–16.
  * [23] S. S. Bhunia and M. Gurusamy, “Dynamic attack detection and mitigation in iot using sdn,” in _2017 27th International telecommunication networks and applications conference (ITNAC)_. IEEE, 2017, pp. 1–6.
  * [24] R. Madeira and L. Nunes, “A machine learning approach for indirect human presence detection using iot devices,” in _2016 Eleventh International Conference on Digital Information Management (ICDIM)_. IEEE, 2016, pp. 145–150.
  * [25] J. Siryani, B. Tanju, and T. J. Eveleigh, “A machine learning decision-support system improves the internet of things’ smart meter operations,” _IEEE Internet of Things Journal_ , vol. 4, no. 4, pp. 1056–1066, 2017.
  * [26] A. Walinjkar and J. Woods, “Ecg classification and prognostic approach towards personalized healthcare,” in _2017 International Conference On Social Media, Wearable And Web Analytics (Social Media)_. IEEE, 2017, pp. 1–8.
  * [27] H. H. Nguyen, F. Mirza, M. A. Naeem, and M. Nguyen, “A review on iot healthcare monitoring applications and a vision for transforming sensor data into real-time clinical feedback,” in _2017 IEEE 21st international conference on computer supported cooperative work in design (CSCWD)_. IEEE, 2017, pp. 257–262.
  * [28] X. Ling, J. Sheng, O. Baiocchi, X. Liu, and M. E. Tolentino, “Identifying parking spaces & detecting occupancy using vision-based iot devices,” in _2017 Global Internet of Things Summit (GIoTS)_. IEEE, 2017, pp. 1–6.
  * [29] W. Guo, T. Fukatsu, and S. Ninomiya, “Automated characterization of flowering dynamics in rice using field-acquired time-series rgb images,” _Plant methods_ , vol. 11, pp. 1–15, 2015.
  * [30] H. Chen, H. Li, G. Dong, M. Hao, G. Xu, X. Huang, and Z. Liu, “Practical membership inference attack against collaborative inference in industrial iot,” _IEEE Transactions on Industrial Informatics_ , vol. 18, no. 1, pp. 477–487, 2020.
  * [31] A. Abusnaina, A. Khormali, H. Alasmary, J. Park, A. Anwar, and A. Mohaisen, “Adversarial learning attacks on graph-based iot malware detection systems,” in _2019 IEEE 39th international conference on distributed computing systems (ICDCS)_. IEEE, 2019, pp. 1296–1305.
  * [32] M. Khosravy, K. Nakamura, N. Nitta, N. Dey, R. G. Crespo, E. Herrera-Viedma, and N. Babaguchi, “Social iot approach to cyber defense of a deep-learning-based recognition system in front of media clones generated by model inversion attack,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , vol. 53, no. 5, pp. 2694–2704, 2022.
  * [33] X. Ma, B. Li, Q. Jiang, Y. Chen, S. Gao, and J. Ma, “Nosnoop: An effective collaborative meta-learning scheme against property inference attack,” _IEEE Internet of Things Journal_ , vol. 9, no. 9, pp. 6778–6789, 2021.
  * [34] Q. Pan, J. Wu, A. K. Bashir, J. Li, and J. Wu, “Side-channel fuzzy analysis-based ai model extraction attack with information-theoretic perspective in intelligent iot,” _IEEE Transactions on Fuzzy Systems_ , vol. 30, no. 11, pp. 4642–4656, 2022.
  * [35] Y. Liu, Y. Xie, and A. Srivastava, “Neural trojans,” in _2017 IEEE International Conference on Computer Design (ICCD)_. IEEE, 2017, pp. 45–48.
  * [36] N. Abosata, S. Al-Rubaye, G. Inalhan, and C. Emmanouilidis, “Internet of things for system integrity: A comprehensive survey on security, attacks and countermeasures for industrial applications,” _Sensors_ , vol. 21, no. 11, p. 3654, 2021.
  * [37] J. Xu and W. Lee, “Sustaining availability of web services under distributed denial of service attacks,” _IEEE Transactions on Computers_ , vol. 52, no. 2, pp. 195–208, 2003.
  * [38] M. Nawir, A. Amir, N. Yaakob, and O. B. Lynn, “Internet of things (iot): Taxonomy of security attacks,” in _2016 3rd international conference on electronic design (ICED)_. IEEE, 2016, pp. 321–326.
  * [39] Z. Wang, M. Taram, D. Moghimi, S. Swanson, D. Tullsen, and J. Zhao, “Nvleak: Off-chip side-channel attacks via non-volatile memory systems,” in _32th USENIX Security Symposium (USENIX Security 23)_ , 2023.
  * [40] O. Kosut, L. Jia, R. J. Thomas, and L. Tong, “Malicious data attacks on smart grid state estimation: Attack strategies and countermeasures,” in _2010 first IEEE international conference on smart grid communications_. IEEE, 2010, pp. 220–225.
  * [41] Z. Luo, S. Zhao, Z. Lu, Y. E. Sagduyu, and J. Xu, “Adversarial machine learning based partial-model attack in iot,” in _Proceedings of the 2nd ACM workshop on wireless security and machine learning_ , 2020, pp. 13–18.
  * [42] A. Biryukov and A. Udovenko, “Attacks and countermeasures for white-box designs,” in _Advances in Cryptology–ASIACRYPT 2018: 24th International Conference on the Theory and Application of Cryptology and Information Security, Brisbane, QLD, Australia, December 2–6, 2018, Proceedings, Part II 24_. Springer, 2018, pp. 373–402.
  * [43] A. N. Bhagoji, W. He, B. Li, and D. Song, “Practical black-box attacks on deep neural networks using efficient query mechanisms,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 154–169.
  * [44] K. Sampigethaya and R. Poovendran, “A survey on mix networks and their secure applications,” _Proceedings of the IEEE_ , vol. 94, no. 12, pp. 2142–2181, 2006.
  * [45] A. Serjantov and P. Sewell, “Passive attack analysis for connection-based anonymity systems,” in _Computer Security–ESORICS 2003: 8th European Symposium on Research in Computer Security, Gjøvik, Norway, October 13-15, 2003. Proceedings 8_. Springer, 2003, pp. 116–131.
  * [46] M. Bhardwaj, T. Xie, B. Boots, N. Jiang, and C.-A. Cheng, “Adversarial model for offline reinforcement learning,” _arXiv preprint arXiv:2302.11048_ , 2023.
  * [47] M. Barreno, B. Nelson, A. D. Joseph, and J. D. Tygar, “The security of machine learning,” _Machine Learning_ , vol. 81, pp. 121–148, 2010.
  * [48] B. Biggio, G. Fumera, and F. Roli, “Security evaluation of pattern classifiers under attack,” _IEEE transactions on knowledge and data engineering_ , vol. 26, no. 4, pp. 984–996, 2013.
  * [49] L. Song and D. G. H. MA CG, “Machine learning security and privacy: a survey,” _Chinese Journal of Network and Information Security_ , vol. 4, no. 8, pp. 1–11, 2018.
  * [50] N. Šrndić and P. Laskov, “Practical evasion of a learning-based classifier: A case study,” in _2014 IEEE symposium on security and privacy_. IEEE, 2014, pp. 197–211.
  * [51] R. Shokri, M. Stronati, C. Song, and V. Shmatikov, “Membership inference attacks against machine learning models,” in _2017 IEEE symposium on security and privacy (SP)_. IEEE, 2017, pp. 3–18.
  * [52] Y. Long, V. Bindschaedler, L. Wang, D. Bu, X. Wang, H. Tang, C. A. Gunter, and K. Chen, “Understanding membership inferences on well-generalized learning models,” _arXiv preprint arXiv:1802.04889_ , 2018.
  * [53] S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha, “Privacy risk in machine learning: Analyzing the connection to overfitting,” in _2018 IEEE 31st computer security foundations symposium (CSF)_. IEEE, 2018, pp. 268–282.
  * [54] M. A. Rahman, T. Rahman, R. Laganière, N. Mohammed, and Y. Wang, “Membership inference attack against differentially private deep learning model.” _Trans. Data Priv._ , vol. 11, no. 1, pp. 61–79, 2018.
  * [55] L. Song, R. Shokri, and P. Mittal, “Privacy risks of securing machine learning models against adversarial examples,” in _Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security_ , 2019, pp. 241–257.
  * [56] A. Sablayrolles, M. Douze, C. Schmid, Y. Ollivier, and H. Jégou, “White-box vs black-box: Bayes optimal strategies for membership inference,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 5558–5567.
  * [57] Z. Wang, M. Song, Z. Zhang, Y. Song, Q. Wang, and H. Qi, “Beyond inferring class representatives: User-level privacy leakage from federated learning,” in _IEEE INFOCOM 2019-IEEE conference on computer communications_. IEEE, 2019, pp. 2512–2520.
  * [58] M. Yaghini, B. Kulynych, G. Cherubin, and C. Troncoso, “Disparate vulnerability: On the unfairness of privacy attacks against machine learning,” _arXiv e-prints_ , pp. arXiv–1906, 2019.
  * [59] S. Truex, L. Liu, M. E. Gursoy, L. Yu, and W. Wei, “Demystifying membership inference attacks in machine learning as a service,” _IEEE Transactions on Services Computing_ , vol. 14, no. 6, pp. 2073–2089, 2019.
  * [60] P. Irolla and G. Châtel, “Demystifying the membership inference attack,” in _2019 12th CMI Conference on Cybersecurity and Privacy (CMI)_. IEEE, 2019, pp. 1–7.
  * [61] L. Song, R. Shokri, and P. Mittal, “Membership inference attacks against adversarially robust deep learning models. in 2019 ieee security and privacy workshops (spw),” _IEEE, 50ś56_ , 2019.
  * [62] S. Truex, L. Liu, M. E. Gursoy, W. Wei, and L. Yu, “Effects of differential privacy and data skewness on membership inference vulnerability,” in _2019 First IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA)_. IEEE, 2019, pp. 82–91.
  * [63] Y. Long, L. Wang, D. Bu, V. Bindschaedler, X. Wang, H. Tang, C. A. Gunter, and K. Chen, “A pragmatic approach to membership inferences on machine learning models,” in _2020 IEEE European Symposium on Security and Privacy (EuroS &P)_. IEEE, 2020, pp. 521–534.
  * [64] K. Leino and M. Fredrikson, “Stolen memories: Leveraging model memorization for calibrated $\\{$White-Box$\\}$ membership inference,” in _29th USENIX security symposium (USENIX Security 20)_ , 2020, pp. 1605–1622.
  * [65] T. Humphries, M. Rafuse, L. Tulloch, S. Oya, I. Goldberg, and F. Kerschbaum, “Differentially private learning does not bound membership inference,” _arXiv preprint arXiv:2010.12112_ , 2020.
  * [66] R. Shokri, M. Strobel, and Y. Zick, “On the privacy risks of model explanations,” in _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ , 2021, pp. 231–241.
  * [67] M. Chen, Z. Zhang, T. Wang, M. Backes, M. Humbert, and Y. Zhang, “When machine unlearning jeopardizes privacy,” in _Proceedings of the 2021 ACM SIGSAC conference on computer and communications security_ , 2021, pp. 896–911.
  * [68] B. Hui, Y. Yang, H. Yuan, P. Burlina, N. Z. Gong, and Y. Cao, “Practical blind membership inference attack via differential comparisons,” _arXiv preprint arXiv:2101.01341_ , 2021.
  * [69] L. Song and P. Mittal, “Systematic evaluation of privacy risks of machine learning models,” in _30th USENIX Security Symposium (USENIX Security 21)_ , 2021, pp. 2615–2632.
  * [70] H. Hu, Z. Salcic, L. Sun, G. Dobbie, and X. Zhang, “Source inference attacks in federated learning,” in _2021 IEEE International Conference on Data Mining (ICDM)_. IEEE, 2021, pp. 1102–1107.
  * [71] C. A. Choquette-Choo, F. Tramer, N. Carlini, and N. Papernot, “Label-only membership inference attacks,” in _International conference on machine learning_. PMLR, 2021, pp. 1964–1974.
  * [72] Y. Liu, Z. Zhao, M. Backes, and Y. Zhang, “Membership inference attacks by exploiting loss trajectory,” in _Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security_ , 2022, pp. 2085–2098.
  * [73] N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer, “Membership inference attacks from first principles,” in _2022 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2022, pp. 1897–1914.
  * [74] J. Ye, A. Maddi, S. K. Murakonda, V. Bindschaedler, and R. Shokri, “Enhanced membership inference attacks against machine learning models,” in _Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security_ , 2022, pp. 3093–3106.
  * [75] B. van Breugel, H. Sun, Z. Qian, and M. van der Schaar, “Membership inference attacks against synthetic data through overfitting detection,” _arXiv preprint arXiv:2302.12580_ , 2023.
  * [76] W. Yuan, C. Yang, Q. V. H. Nguyen, L. Cui, T. He, and H. Yin, “Interaction-level membership inference attack against federated recommender systems,” _arXiv preprint arXiv:2301.10964_ , 2023.
  * [77] T. Matsumoto, T. Miura, and N. Yanai, “Membership inference attacks against diffusion models,” _arXiv preprint arXiv:2302.03262_ , 2023.
  * [78] M. Bertran, S. Tang, M. Kearns, J. Morgenstern, A. Roth, and Z. S. Wu, “Scalable membership inference attacks via quantile regression,” _arXiv preprint arXiv:2307.03694_ , 2023.
  * [79] Y. Zhang, L. Zhao, and Q. Wang, “Mida: Membership inference attacks against domain adaptation,” _ISA transactions_ , 2023.
  * [80] H. Hu and J. Pang, “Membership inference of diffusion models,” _arXiv preprint arXiv:2301.09956_ , 2023.
  * [81] A. Salem, Y. Zhang, M. Humbert, P. Berrang, M. Fritz, and M. Backes, “Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models,” _arXiv preprint arXiv:1806.01246_ , 2018.
  * [82] M. Nasr, R. Shokri, and A. Houmansadr, “Comprehensive privacy analysis of deep learning,” in _Proceedings of the 2019 IEEE Symposium on Security and Privacy (SP)_ , 2018, pp. 1–15.
  * [83] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep learning with differential privacy,” in _Proceedings of the 2016 ACM SIGSAC conference on computer and communications security_ , 2016, pp. 308–318.
  * [84] M. Nasr, R. Shokri, and A. Houmansadr, “Machine learning with membership privacy using adversarial regularization,” in _Proceedings of the 2018 ACM SIGSAC conference on computer and communications security_ , 2018, pp. 634–646.
  * [85] J. Jia, A. Salem, M. Backes, Y. Zhang, and N. Z. Gong, “Memguard: Defending against black-box membership inference attacks via adversarial examples,” in _Proceedings of the 2019 ACM SIGSAC conference on computer and communications security_ , 2019, pp. 259–274.
  * [86] J. Li, N. Li, and B. Ribeiro, “Membership inference attacks and defenses in supervised learning via generalization gap,” _arXiv preprint arXiv:2002.12062_ , vol. 3, no. 7, 2020.
  * [87] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” _arXiv preprint arXiv:1503.02531_ , 2015.
  * [88] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” _arXiv preprint arXiv:1312.6199_ , 2013.
  * [89] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” _arXiv preprint arXiv:1412.6572_ , 2014.
  * [90] S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, “Deepfool: a simple and accurate method to fool deep neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2574–2582.
  * [91] A. Kurakin, I. Goodfellow, and S. Bengio, “Adversarial machine learning at scale,” _arXiv preprint arXiv:1611.01236_ , 2016.
  * [92] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” _arXiv preprint arXiv:1706.06083_ , 2017.
  * [93] S. Huang, N. Papernot, I. Goodfellow, Y. Duan, and P. Abbeel, “Adversarial attacks on neural network policies,” _arXiv preprint arXiv:1702.02284_ , 2017.
  * [94] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in _2017 ieee symposium on security and privacy (sp)_. Ieee, 2017, pp. 39–57.
  * [95] P.-Y. Chen, H. Zhang, Y. Sharma, J. Yi, and C.-J. Hsieh, “Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models,” in _Proceedings of the 10th ACM workshop on artificial intelligence and security_ , 2017, pp. 15–26.
  * [96] N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, “Practical black-box attacks against machine learning,” in _Proceedings of the 2017 ACM on Asia conference on computer and communications security_ , 2017, pp. 506–519.
  * [97] W. Brendel, J. Rauber, and M. Bethge, “Decision-based adversarial attacks: Reliable attacks against black-box machine learning models,” _arXiv preprint arXiv:1712.04248_ , 2017.
  * [98] D. Zügner, A. Akbarnejad, and S. Günnemann, “Adversarial attacks on neural networks for graph data,” in _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_, 2018, pp. 2847–2856.
  * [99] Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li, “Boosting adversarial attacks with momentum,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 9185–9193.
  * [100] A. Ilyas, L. Engstrom, A. Athalye, and J. Lin, “Black-box adversarial attacks with limited queries and information,” in _International conference on machine learning_. PMLR, 2018, pp. 2137–2146.
  * [101] A. Bojchevski and S. Günnemann, “Adversarial attacks on node embeddings via graph poisoning,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 695–704.
  * [102] C. Guo, J. Gardner, Y. You, A. G. Wilson, and K. Weinberger, “Simple black-box adversarial attacks,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 2484–2493.
  * [103] J. Su, D. V. Vargas, and K. Sakurai, “One pixel attack for fooling deep neural networks,” _IEEE Transactions on Evolutionary Computation_ , vol. 23, no. 5, pp. 828–841, 2019.
  * [104] J. Chen, M. I. Jordan, and M. J. Wainwright, “Hopskipjumpattack: A query-efficient decision-based attack,” in _2020 ieee symposium on security and privacy (sp)_. IEEE, 2020, pp. 1277–1294.
  * [105] D. Zügner, O. Borchert, A. Akbarnejad, and S. Günnemann, “Adversarial attacks on graph neural networks: Perturbations and their patterns,” _ACM Transactions on Knowledge Discovery from Data (TKDD)_ , vol. 14, no. 5, pp. 1–31, 2020.
  * [106] Y. Bin, X. Cao, X. Chen, Y. Ge, Y. Tai, C. Wang, J. Li, F. Huang, C. Gao, and N. Sang, “Adversarial semantic data augmentation for human pose estimation,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIX 16_. Springer, 2020, pp. 606–622.
  * [107] X. Wang, X. He, J. Wang, and K. He, “Admix: Enhancing the transferability of adversarial attacks,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 16 158–16 167.
  * [108] J. Rony, E. Granger, M. Pedersoli, and I. Ben Ayed, “Augmented lagrangian adversarial attacks,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 7738–7747.
  * [109] A. Amich and B. Eshete, “Eg-booster: explanation-guided booster of ml evasion attacks,” in _Proceedings of the Twelfth ACM Conference on Data and Application Security and Privacy_ , 2022, pp. 16–28.
  * [110] X. Jia, Y. Zhang, B. Wu, K. Ma, J. Wang, and X. Cao, “Las-at: adversarial training with learnable attack strategy,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 13 398–13 408.
  * [111] T. Li, Y. Wu, S. Chen, K. Fang, and X. Huang, “Subspace adversarial training,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 13 409–13 418.
  * [112] Y. Xiong, J. Lin, M. Zhang, J. E. Hopcroft, and K. He, “Stochastic variance reduced ensemble adversarial attack for boosting the adversarial transferability,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 14 983–14 992.
  * [113] Z. Song, Z. Zhang, K. Zhang, W. Luo, Z. Fan, W. Ren, and J. Lu, “Robust single image reflection removal against adversarial attacks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 24 688–24 698.
  * [114] J. Kim, B.-K. Lee, and Y. M. Ro, “Demystifying causal features on adversarial examples and causal inoculation for robust network by adversarial instrumental variable regression,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 12 302–12 312.
  * [115] W. Feng, N. Xu, T. Zhang, and Y. Zhang, “Dynamic generative targeted attacks with pattern injection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 16 404–16 414.
  * [116] Z. Wang, H. Yang, Y. Feng, P. Sun, H. Guo, Z. Zhang, and K. Ren, “Towards transferable targeted adversarial examples,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 20 534–20 543.
  * [117] Y. Chen, J. Tian, X. Chen, and J. Zhou, “Effective ambiguity attack against passport-based dnn intellectual property protection schemes through fully connected layer substitution,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 8123–8132.
  * [118] H. Huang, Z. Chen, H. Chen, Y. Wang, and K. Zhang, “T-sea: Transfer-based self-ensemble attack on object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 20 514–20 523.
  * [119] C. WANG, C. LIU, Y. LI, S. Niu, and Y. Zhang, “Two-way and anonymous heterogeneous signcryption scheme between pki and ibc,” _Journal on communications_ , vol. 38, no. 10, pp. 10–17, 2017.
  * [120] R. Feinman, R. R. Curtin, S. Shintre, and A. B. Gardner, “Detecting adversarial samples from artifacts,” _arXiv preprint arXiv:1703.00410_ , 2017.
  * [121] K. Grosse, P. Manoharan, N. Papernot, M. Backes, and P. McDaniel, “On the (statistical) detection of adversarial examples,” _arXiv preprint arXiv:1702.06280_ , 2017.
  * [122] X. Huang, M. Kwiatkowska, S. Wang, and M. Wu, “Safety verification of deep neural networks,” in _Computer Aided Verification: 29th International Conference, CAV 2017, Heidelberg, Germany, July 24-28, 2017, Proceedings, Part I 30_. Springer, 2017, pp. 3–29.
  * [123] J. H. Metzen, T. Genewein, V. Fischer, and B. Bischoff, “On detecting adversarial perturbations,” _arXiv preprint arXiv:1702.04267_ , 2017.
  * [124] W. Xu, D. Evans, and Y. Qi, “Feature squeezing: Detecting adversarial examples in deep neural networks,” _arXiv preprint arXiv:1704.01155_ , 2017.
  * [125] J. Jin, A. Dundar, and E. Culurciello, “Robust convolutional neural networks under adversarial noise,” _arXiv preprint arXiv:1511.06306_ , 2015.
  * [126] F. Tramèr, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel, “Ensemble adversarial training: Attacks and defenses,” _arXiv preprint arXiv:1705.07204_ , 2017.
  * [127] V. Zantedeschi, M.-I. Nicolae, and A. Rawat, “Efficient defenses against adversarial attacks,” in _Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security_ , 2017, pp. 39–49.
  * [128] S. Zheng, Y. Song, T. Leung, and I. Goodfellow, “Improving the robustness of deep neural networks via stability training,” in _Proceedings of the ieee conference on computer vision and pattern recognition_ , 2016, pp. 4480–4488.
  * [129] Y. Nesterov and V. Spokoiny, “Random gradient-free minimization of convex functions,” _Foundations of Computational Mathematics_ , vol. 17, pp. 527–566, 2017.
  * [130] X. Cao and N. Z. Gong, “Mitigating evasion attacks to deep neural networks via region-based classification,” in _Proceedings of the 33rd Annual Computer Security Applications Conference_ , 2017, pp. 278–287.
  * [131] X. Wang, Z. Zhang, K. Tong, D. Gong, K. He, Z. Li, and W. Liu, “Triangle attack: A query-efficient decision-based adversarial attack,” in _European Conference on Computer Vision_. Springer, 2022, pp. 156–174.
  * [132] M. Cheng, T. Le, P.-Y. Chen, J. Yi, H. Zhang, and C.-J. Hsieh, “Query-efficient hard-label black-box attack: An optimization-based approach,” _arXiv preprint arXiv:1807.04457_ , 2018.
  * [133] M. Cheng, S. Singh, P. Chen, P.-Y. Chen, S. Liu, and C.-J. Hsieh, “Sign-opt: A query-efficient hard-label adversarial attack,” _arXiv preprint arXiv:1909.10773_ , 2019.
  * [134] H. Li, X. Xu, X. Zhang, S. Yang, and B. Li, “Qeba: Query-efficient boundary-based blackbox attack,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 1221–1230.
  * [135] A. Rahmati, S.-M. Moosavi-Dezfooli, P. Frossard, and H. Dai, “Geoda: a geometric framework for black-box adversarial attacks,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 8446–8455.
  * [136] T. Maho, T. Furon, and E. Le Merrer, “Surfree: a fast surrogate-free black-box attack,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 10 430–10 439.
  * [137] J. Du, H. Zhang, J. T. Zhou, Y. Yang, and J. Feng, “Query-efficient meta attack to deep neural networks,” _arXiv preprint arXiv:1906.02398_ , 2019.
  * [138] J. Wang, “Adversarial examples in physical world.” in _IJCAI_ , 2021, pp. 4925–4926.
  * [139] J. Rauber, R. Zimmermann, M. Bethge, and W. Brendel, “Foolbox native: Fast adversarial attacks to benchmark the robustness of machine learning models in pytorch, tensorflow, and jax,” _Journal of Open Source Software_ , vol. 5, no. 53, p. 2607, 2020.
  * [140] T. Miyato, S.-i. Maeda, M. Koyama, K. Nakae, and S. Ishii, “Distributional smoothing with virtual adversarial training,” _arXiv preprint arXiv:1507.00677_ , 2015.
  * [141] L. Pajola and M. Conti, “Fall of giants: How popular text-based mlaas fall against a simple evasion attack,” in _2021 IEEE European Symposium on Security and Privacy (EuroS &P)_. IEEE, 2021, pp. 198–211.
  * [142] M. Fredrikson, S. Jha, and T. Ristenpart, “Model inversion attacks that exploit confidence information and basic countermeasures,” in _Proceedings of the 22nd ACM SIGSAC conference on computer and communications security_ , 2015, pp. 1322–1333.
  * [143] M. Fredrikson, E. Lantz, S. Jha, S. Lin, D. Page, and T. Ristenpart, “Privacy in pharmacogenetics: An $\\{$End-to-End$\\}$ case study of personalized warfarin dosing,” in _23rd USENIX security symposium (USENIX Security 14)_ , 2014, pp. 17–32.