# Retrieval-Augmented Embodied Agents

First Author
Institution1
Institution1 address
<EMAIL_ADDRESS>Second Author
Institution2
First line of institution2 address
<EMAIL_ADDRESS>Yichen Zhu, Zhicai Ou, Xiaofeng Mou, Jian Tang
Midea Group, AI Lab
{zhuyc25, zhicai.ou, mouxf<EMAIL_ADDRESS>Corresponding author

###### Abstract

Embodied agents operating in complex and uncertain environments face
considerable challenges. While some advanced agents handle complex
manipulation tasks with proficiency, their success often hinges on extensive
training data to develop their capabilities. In contrast, humans typically
rely on recalling past experiences and analogous situations to solve new
problems. Aiming to emulate this human approach in robotics, we introduce the
Retrieval-Augmented Embodied Agent (RAEA). This innovative system equips
robots with a form of shared memory, significantly enhancing their
performance. Our approach integrates a policy retriever, allowing robots to
access relevant strategies from an external policy memory bank based on multi-
modal inputs. Additionally, a policy generator is employed to assimilate these
strategies into the learning process, enabling robots to formulate effective
responses to tasks. Extensive testing of RAEA in both simulated and real-world
scenarios demonstrates its superior performance over traditional methods,
representing a major leap forward in robotic technology.

##  1 Introduction

The swift advancement of foundation models in areas like natural language
processing and computer vision has sparked interest in the robotics community
to create embodied agents capable of comprehending human instructions and
responding aptly to their environment. Despite this enthusiasm, crafting
agents that seamlessly interact with the physical world remains a formidable
task. deep neural networks store knowledge—such as recognizing objects or
interpreting commands—implicitly within their neural network parameters. This
dependence on implicit knowledge storage demands a significant number of
parameters and a wide range of training data. However, recent studies have
shown that the scalability in terms of both training data and model size falls
short [72, 4] when compared to foundation models in other domains, such as
Large Language Models. This insight has inspired the creation of embodied
agents designed to learn efficiently with limited data and model sizes. To
augment their capabilities, it’s becoming increasingly important for these
agents to access external repositories of physical knowledge, thereby
expanding their capacity to understand and interact with the world.

Figure 1: The overview of our retrieval-augmented embodied agents. We utilize
a policy retriever to extract policies from a policy memory bank, which
contains large-scale robotic data across multiple embodiments. Then, we use
the policy generator to reference the retrieved policy and output actions for
the current input.

The ability to tap into an external repository of behavioral memory mirrors
the learning process observed in human infants, who often remember and mimic
the actions of adults or animals when presented with analogous scenarios from
their memory. This ability is crucial for successfully navigating unknown
environments and performing tasks that demand specific knowledge, like
exploring unfamiliar rooms or handling new objects. Consequently, the question
naturally arises: How can we harness the wealth of open-source, multi-
embodiment data to enhance the precision of robots in manipulation tasks? This
inquiry not only probes the potential of robotic learning but also seeks to
bridge the gap between human cognitive processes and robotic applications.

Figure 2: Examples of simulated and real-world environments that we used for
evaluation.

In this paper, we present Retrieval-Augmented Embodied Agents (RAEA), which
leverage an external policy memory bank containing analogous scenarios,
whether in terms of instructions, observations, or a combination of both,
related to the ongoing task. We outline the overview of our framework in
Figure 1. The policies retrieved from this memory, along with other relevant
data, provide a rich resource for both the training and testing phases. In our
methodology, we make use of the recently open-sourced Open X-Embodiment [45],
a large-scale repository of robotic datasets. Open X-Embodiment contains an
extensive array of tasks, applications, embodiments, and diverse environmental
settings from various research labs. This extensive dataset serves as the
cornerstone for our external policy banks, enriching the knowledge base of
RAEA. By tapping into this vast repository, RAEA can access a broader spectrum
of robotic experiences, thereby improving its adaptability and effectiveness
in various tasks.

To realize our objectives, we introduce two innovative modules: a policy
retriever and a policy generator. The policy retriever is adept at handling
multiple input modalities, categorized into two main types: instructions and
observations. For controlling robots, it accepts text and audio as
instructional inputs, and images, videos, and point clouds as observational
inputs. It identifies policy candidates from the memory bank that align
closely with the current input. Furthermore, we have developed the policy
generator, which initially processes the information in the retrieved
policies, including observation, instruction, action, and proprioceptive
state. It then employs a cross-attention module to integrate knowledge from
various retrieved policies into the main policy networks for action
prediction. In this way, the policy generator leverages the retrieved policies
as contextual examples, aiding the model in producing actionable responses
based on the current input.

The efficacy of our proposed Retrieval-Augmented Embodied Agent (RAEA) is
demonstrated through extensive testing over two simulation benchmarks and
real-world datasets, as illustrated in Figure 2. This approach not only
showcases the versatility and practical ability of our framework.

In summary, our contributions are as follows:

  * •

We present Retriveal-Augmented Embodied Agents (RAEA) that utilized the wealth
of knowledge from an external policy memory bank with multiple embodiment data
to facilitate prediction for robotic action.

  * •

Our framework features a policy retriever adept at processing various input
modalities. Complementing this, we have crafted a policy generator that
leverages retrieved scenarios to improve the model’s ability to generalize
across various situations.

  * •

To validate the efficacy of our proposed methodology, we have conducted
extensive evaluations in both real-world settings and two simulated
environments. The results from these tests strongly affirm the effectiveness
and practicality of our approach.

Overall, our work introduces a versatile and modular retrieval-augmentation
framework for embodied agents. This provides a novel and insightful
perspective on the design of robotic models, integrating advanced memory
capabilities.

##  2 Related Works

Retrieval-Augmented Models. A notable trend in Natural Language Processing
(NLP) involves leveraging external memory to enhance the performance of
language models. This approach retrieves documents relevant to the input text
from an external database, allowing language models (generators) to use this
retrieved information to make more informed predictions. Typically, the
external memory consists of a collection of text passages or a structured
knowledge base [67, 66, 65]. Subsequent works extend the retrieval
augmentation techniques to computer vision models and multi-modal models. The
most representative works including Re-Imagen [12] as a caption-to-image
generator, MuRAG [11] performs question-answering using retrieved images.
RA-C3M [68] uses retrieval for either text or image generation.RA-CLIP [64]
and REACT [36] integrate the retrieval-augmentation technique for CLIP
pretraining. Our approach, however, diverges from these prior works. While the
aforementioned studies focus on enhancing language and vision models, our
research is specifically geared towards robotics. We aim to search for
policies that have been executed in scenarios similar to the current context,
using them as in-context examples.

Models for Embodied Agents. In the field of embodied agents [6, 41, 55, 38,
62, 28, 13, 14, 47, 43, 1, 2, 3, 10, 53, 56, 59, 54, 30] and robotics [8, 9,
18, 25, 44, 46, 60, 71], foundation models have become a crucial research
focus, revolutionizing the interaction between AI systems and physical
environments. This body of work includes studies on representation pre-
training and the application of language and vision-language models [27, 29,
30, 38, 41, 52, 58] as embodied agents. Our research contributes to this
growing body of knowledge, presenting a supplementary retrieval-based approach
designed to enhance policy learning in robotics. This approach integrates with
existing foundation models, providing a novel perspective on how to augment
the capabilities of embodied agents in diverse and dynamic settings.

Robotics Datasets. The robotics learning community has developed a variety of
open-source datasets that are instrumental in advancing robot learning. These
tasks spanning grasping [31, 34, 16, 39, 19, 20, 5], pushing interactions [57,
17, 69], sets of objects and models [75–85], and teleoperated demonstrations
[8, 86–95]. Typically, these datasets are extensive and are often focused on
specific robotic embodiments, exemplified by the Bridge Data [18, 61, 33] and
RH20T [21]. RoboNet [15] and Open X-Embodiment [45] stand out as two large-
scale datasets that incorporate multiple robotic embodiments. These datasets
are frequently utilized for pretraining purposes [45], especially for the
visual backbone in robotic models [7, 42, 50]. In our research, we leverage
the extensive cross-embodiment data as a foundational knowledge base. This
enables us to retrieve relevant policies that facilitate training in the
current environment, effectively utilizing the rich diversity of the datasets
to enhance our model’s adaptability and performance.

##  3 Methodology

We introduce Retrieval-Augmented Embodied Agents, capable of retrieving
relevant scenarios and generating actions based on the current scene and
accompanying instructions. As illustrated in Figure 3, when presented with an
input, be it an observation or an instruction-observation pair, our system
employs a retriever to fetch pertinent policies from an external memory bank.
Significantly, these embodied agents are equipped to interact with humans
through various modalities like text and audio, and they use diverse sensors
to perceive their environment. In order to broaden the spectrum of
applications for our approach, we have designed a multi-modal policy retriever
(in §3.2), featuring a dense retriever with a mixed-modal encoder capable of
encoding diverse modalities in various combinations. Additionally, we’ve
constructed a policy generator (in §3.3)based on the Transformer architecture.
This generator processes the retrieved policies individually and leverages
cross-attention to incorporate the extracted information from the retrieved
policies into the primary model branch.

###  3.1 Preliminaries

Notations. The framework consists of a policy retriever $R$ and a policy
generator module $G$. The retrieval module $R$ takes an input sequence
$r=\\{i,o\\}$ and searches the $r$ from an external policy memory bank. It
returns a list of policy $m=\\{i,o,a,p\\}$, where $p$ is the policy, $i$
represents the instruction, $o$ denotes the observation, $a$ is the action,
$p$ is the proprioception. The term proprioceptive robot state is used to
describe a robot’s intrinsic awareness of its own positioning and movement
within a given space, which includes factors like joint angles, velocity,
torque, and other physical statuses. The term actions refers to the specific
tasks or operations executed by the robot. The policy generator $G$ then takes
the input sequence $x$ and the retrieved policy
$M=\\{m_{1},m_{2},\cdots,m_{n}\\}$ and returns the action $a$, where $a$
represent continuous actions that control the robots.

Figure 3: The framework of policy retriever (top) and policy generator
(bottom) in our work. The policy retriever retrieves the relevant policy based
on multi-modal input, and the policy generator processes a list of retrieved
policies to help train in the current environment.

###  3.2 Policy Retriever

Overview. A policy retriever $R$ takes a query $q$ (e.g., the instruction-
observation pair x) with multi-modal information from the policy memory bank
$M$, and returns a relevance score $r(q,m)$. We follow prior retrieval works
[32], in which the retriever $r$ is a bi-encoder architecture,

$r(q,m)=E_{Q}(q)^{T}E_{M}(m)$ (1)

Here, we employ two key encoders: $E_{Q}$, responsible for encoding queries,
and $E_{M}$, which encodes the memory to yield dense vectors representing
query and memory policies, respectively. Given that our input and memory
consist of multi-modal documents, we leverage $E_{Q}$ and $E_{M}$ as mixed-
modal encoders capable of handling various modalities. The architecture of
mixed-modal encoders can be designed in multiple ways. In our specific
context, we introduce a multi-modal retrieval approach that adeptly
accommodates multiple modalities. When dealing with a multi-modal input for
policy learning, we partition it into two distinct components: an instruction
segment and an observation segment. The instruction segment typically contains
human instructions in different formats, including text or audio, while the
observation segment can encompass images, videos, or point cloud data. Each
modality type is separately encoded using off-the-shelf, pre-trained multi-
modality encoders, which we will discuss specifically in the next section.

Multi-Modal Encoders. The primary aim of multi-modal encoders is to equip
embodied agents with the capability to handle a wide range of modalities,
adapting to various scenarios. Specifically, we conceptualize the encoder as a
mapping function, denoted by $P(\cdot)$. Typically, for each modality, a
specialized model is needed to extract useful modality-specific
representations. These are then projected onto a feature plane, ensuring
uniformity in the shape of feature tensors across all modalities through
projection layers. Following this, we average the vectors representing these
modalities, normalizing their L2 norms to 1, thus generating a consolidated
vector representation of the document. This encoding technique is uniformly
applied to both $E_{Q}$ and $E_{M}$. The final step involves assessing the
similarity of cross-modality features between the query and the memory.

There are a number of options [63] for the multi-modal model when it comes to
handling different modalities. For example, CLIP [49] or T5 [51] could be used
for text and image processing. In our case, we utilize ImageBind [23], a high-
performance encoder proficient across six modalities, for processing diverse
input types. With the help of ImageBind, we are spared from managing many
numbers of heterogeneous modal encoders. This mapping function $P(\cdot)$ maps
all modalities to a unified latent embedding, greatly enhancing the efficiency
of comparing feature similarity for retrieval purposes. For the retrieval
process, we execute the Maximum Inner Product Search within the memory space,
yielding a ranked list of candidates based on their relevance scores. From
this list, we select the final $k$ policies for further analysis and
processing.

Retrieval Strategy. We discuss three key factors in obtaining/sampling
informative retrieved policies for the generator in practice.

Relevance: The retrieval of policies must be closely aligned with the input
sequence, covering either instructions, observations, or both. Without this
alignment, the retrieved policies fail to provide meaningful contributions to
the modeling of the primary input sequence. To ascertain the relevance of
these policies, we employ a dense retriever score based on modality encoders.

Input Modality: Our methodology divides the input into two distinct segments:
instructions and observations, each tailored to process specific input
modalities. The observation component is adept at recalling comparable scenes
or objects, thereby enabling the model to comprehend scenarios not encountered
in its training dataset. Concurrently, the instruction segment recapitulates
actions previously executed during training. This flexible framework allows
for either independent or combined usage of these segments, contingent on the
computational resources and specific application contexts. This approach bears
a resemblance to in-context learning, wherein the Large Language Model (LLM)
is presented with scenarios that are similar, albeit not identical, to enhance
response quality. Typically, we employ instruction-observation pairs at the
onset of a frame, subsequently relying solely on the observation for the
remainder of the frame until the next user interaction. It’s noteworthy that
this strategy can be integrated with recent advancements in Large Multi-Modal
Models, facilitating real-time corrections of the robot’s actions.

Diversity: We discovered that diversity in the policy memory bank is crucial
for effective performance. Selecting the top-ranked actions based on relevance
scores often leads to the inclusion of duplicate or very similar instruction
and observation. This redundancy can detrimentally impact the performance of
the generator. This challenge is particularly acute given that our action bank
comprises fragments of videos. To optimize the efficacy of retrieval-based
methods, it’s essential to ensure a diverse range of in-context samples. These
diverse samples are instrumental in aiding policy networks to effectively
learn from demonstrations, both in training and testing stages. To address the
issue of redundancy, our approach involves bypassing a candidate action if its
relevance score is too closely aligned (e.g., exceeding 0.9) with the query or
actions already retrieved. Additionally, to further enhance diversity, we
propose a unique strategy of random token dropout from the query used in
retrieval, approximately 70% of tokens. This approach serves as a
regularization mechanism during training and has been observed to
significantly improve the generator’s performance. We also include embodiment
data of embodiment that are different from the robot that we used for
evaluation. We observed that even with variations in embodiment, this diverse
dataset still aids the policy network in its learning process.

Data Format of Retrieved Policy. For every policy retriever, our framework
covers a set of elements $m$ that either influence or result from control
processes. Subsequently, we introduce our policy generator and demonstrate how
it effectively utilizes this diverse spectrum of information for enhancing
policy learning.

Figure 4: Left: The setup of our Franka real robot. Right: The example of some
tasks that we collected. Figure 5: Performance of RAEA in Franka Kitchen with
10 or 25 demonstrations

###  3.3 Policy Generator

The policy generator is designed to effectively utilize the valuable
information in the retrieved policy to facilitate the training of the policy
for the current input. We reuse the feature representation of instruction and
observation from the policy retrieval network (as in §3.2), thus avoiding
redundant computations that constitute over 95% of the total. For actions and
proprioceptive states, we address the variability across different robots by
setting a maximum limit for both, capped at nine. We employ an action encoder
and a proprioception encoder – both comprising multi-layer perceptrons (MLPs)
– to generate corresponding tokens. These tokens are then integrated with the
instruction-observation tokens, ensuring a seamless and effective
incorporation of the retrieved policy’s data into the current policy training
framework. We add a state token between different states, i.e., a learnable
token between action and proprioception tokens, to split the data of two
states. We use absolute position embedding to ensure the tokens are in order.

Given a list of retrieved policies $M=(m_{1},...,m_{K})$, we concatenated
these tokens based on the relevance score. We use absolute position embedding
to maintain the order of tokenized representations. We concatenate their
tokens according to their relevance scores. We employ a policy token to
demarcate tokens from different policies. Once these tokens are combined, we
utilize the Transformer architecture as the foundation for our retrieved
policy processor. To effectively incorporate the retrieved policies $M$ into
the generator, we leverage cross-attention mechanisms. This approach allows
for the integration of pertinent information from the retrieved models into
the main network, enhancing the overall efficacy of the system.

Particularly, give an input sequence from retrieved polices, denoted as
$F^{r}\in\mathbb{R}^{h\times w\times c}$, and other input sequences from main
input $F^{x}\in\mathbb{R}^{h\times w\times c}$, for simplicity, we assume two
tensors have the same size. The $F^{r}$ is projected into a query (Q) and key
(K), and $F^{x}$ is projected into value (V). Thus, we formulate our cross-
attention (SC) as follows:

$Q_{i}=F^{r}W_{i}^{Q}$ (2)
$K_{i}=SC(F^{x},r_{i})W_{i}^{K},V_{i}=SC(F^{x},r_{i})W_{i}^{V},$ (3)
$V_{i}=V_{i}+P(V_{i})$ (4)

where $SC(\cdot,r_{i})$ is a MLP layer for aggregation in the $i^{th}$ head
with the down-sampling rate of $r_{i}$, and $P(\cdot)$ is a depth-wise
convolutional layer for projection. Finally, we calculated the attention
tensor by:

$h_{i}=Softmax(\frac{Q_{i}K^{T}_{i}}{\sqrt{d_{h}}}V_{i})$ (5)

where $d_{h}$ is the dimension. The streamlined design of the cross-attention
mechanism ensures that valuable representations from retrieved policies are
effectively incorporated into the main network, enhancing policy learning for
the current input. To optimize this process, we implement behavior cloning,
using mean squared loss as our primary optimization objective.

##  4 Experiments

We evaluate models using multiple simulated benchmarks, including Franka
Kitchen [22], MetaWorld [70], and Maniskill-2 [24] and real-world environment.
We show that our retrieval-augmented embodied agents significantly improve the
generalization ability. Notably, we have taken precautions to ensure that the
policy memory bank does not contain any data from the datasets used in our
training and testing phases, thereby eliminating the possibility of bias or
cheating in the test set.

###  4.1 Simulation Experiments.

We conduct our simulation on three benchmarks, Franka Kitchen [22], Meta-World
[70], and Maniskill-2 [24]. A brief summary of these benchmarks can be found
in the Appendix.

Figure 6: Performance of RAEA in Meta-World with 10 or 25 demonstrations

Evaluation: We assess our approach through 30 rollouts derived from the
behavior cloning (BC) learned policy. Our primary metric for evaluation is the
mean success rate of the final policy. Additionally, when presenting metrics
for the task suite, we calculate the average mean success rate across various
camera configurations.

Experimental results on Franka Kitchen & Meta-World. In our experiments, we
conducted a comparative analysis of our model, RAEA, against two established
state-of-the-art methods: R3M [42], popular in Franka Kitchen applications,
and BLIP-2 [35], a leading vision-language model, and Embodied-GPT [40], a
vision-language model designed for robotics. We trained our policy network
using a few-shot learning approach, employing datasets comprising either ten
or twenty-five demonstrations. The performance of these models was evaluated
through 100 randomized trials across five distinct tasks in each benchmark.
These evaluations were executed under two different settings: each involved
five separate runs and was conducted from two unique camera perspectives,
relying solely on visual observations. The results, depicted in Figures 5 and
6, for the Franka Kitchen and Meta-World benchmarks respectively,
unequivocally demonstrate that RAEA surpasses the baseline methods in
effectiveness. This superiority is particularly noticeable in low-data
scenarios, such as those with only ten demonstrations, further underscoring
RAEA’s robustness and efficiency in environments with limited data
availability.

Table 1: Experiments on Manisill-2 over six rigid body and soft body tasks.
Our method consistently outperforms Baseline in all environments. All metrics
are reported in percentage $(\%)$ with the best ones bolded.

Methods | PickCube | StackCube | PickSingleYCB | Fill | Hang | Excavate
---|---|---|---|---|---|---
Observation Modality: Image
ResNet152 [26] | 40.1 | 86.3 | 22.1 | 52.4 | 82.6 | 12.0
Swin-Base [37] | 41.3 | 83.5 | 28.5 | 49.0 | 82.7 | 14.8
RAEA | 56.7 | 93.6 | 40.2 | 63.8 | 87.1 | 22.4
Observation Modality: Point Cloud + Image
ResNet152 + PointNet [26, 48] | 56.6 | 90.7 | 28.4 | 46.9 | 85.7 | 18.4
Swin-Base + PointNet [37, 48] | 44.0 | 90.1 | 30.6 | 45.5 | 86.4 | 17.0
RAEA | 62.7 | 91.0 | 43.8 | 71.7 | 88.1 | 24.0

Experimental results on Maniskill-2. We evaluated our model’s performance in
two distinct experimental settings: one using solely image-based observations
and the other combining images with point cloud data. For comparison, we
benchmarked our model against well-established methods, specifically ResNet152
[26] and Swin Transformer Base [37], both pre-trained on the ImageNet dataset.
In the experiments that involved both images and point clouds, the feature
representation of point cloud data for the baseline models was initially
processed using PointNet [48]. This representation was subsequently integrated
with the image branch, following the procedure described in Maniskill-2. In
both experimental scenarios, our method consistently surpassed the performance
of the two baseline models, as illustrated in Table 1. Remarkably, in the
setting that utilized dual-modal observations, our RAEA model demonstrated a
significantly higher average success rate. This highlights the robust
generalizability and effectiveness of our approach.

Table 2: Ablation study on the effect of using different modalities for real-
world environments.

Method | Instruction | Observation | Success Rate
---|---|---|---
RAEA | Text | Image | 54
Text+Audio | Image | 56
Text+Audio | Image + Point Cloud | 65
Image | Image | 58
Image+Video | Image | 63
Image+Video | Image + Video | 64
Image+Video+Text | Image | 69

###  4.2 Real-Robot Experiments

Datasets. Our collected dataset comprises $n=40$ tasks. These tasks vary from
straightforward pick-and-place actions, such as “pick up the yellow cube” to
more complex contact-rich tasks like “open the drawer and put the pen inside,”
as well as tasks such as “sort the cube with the same color.” There are 70
objects in the experiments. Each task is exemplified through 30 human-
collected trajectories. Further, every task is annotated with 5 distinct
instructions. Figure 4 demonstrate some example and workspace setup for our
real-world experiments.

Implementation details. We use an AdamW optimizer, starting with an initial
learning rate of 3e-5, and implement a weight decay of 1e-6. Our learning rate
scheduler is designed to linearly decay, incorporating a warm-up phase that
spans the initial 2% of the total training duration. Additionally, we apply
gradient clipping set at a value of 1.0 to maintain stability during training.
To assess the efficacy of our approach, all experiments are conducted over 10
trials, from which we calculate the mean success rate.

Additionally, we perform ablation studies to delve into various questions
related to our model’s performance and capabilities.

Table 3: Ablation study on the effect of status information, i.e., action &
proprioceptive state, in real-world data. The experiments are conducted based
on text-image pairs.

Tasks | Status | Success Rate
---|---|---
RAEA | All | 54
w/o Proprioception | 39
w/o Action & Proprioception | 36

Table 4: Ablation study on the data for policy memory bank using Franka-only
or all embodiments.

Tasks | Embodiments | Success Rate
---|---|---
RAEA | All | 54
Franka-Only | 48

1\. Does Utilizing Multiple Modalities Improve Generalizability? Our embodied
agents are equipped to support multi-modal inputs. This section examines the
benefits of integrating multiple modalities. Table 2 demonstrates that
employing more available modalities can stably yield better performance.
Notice that using the combination of language and visual as instruction
significantly enhance the generalizability of the model, i.e., increase the
success rate from 63 to 69. Also, adopting 3D information, such as point
cloud, can be useful, which improves the success rate from 56 to 65.

2\. Does Including More Status Information in Retrieved Policies Enhance
Policy Learning? While instruction and observation are fundamental components,
our research also delves into the impact of incorporating proprioception and
action data. As illustrated in Table 3, there is a discernible decrease in
success rate when proprioception and action are omitted from the retrieved
policy, dropping from 54 to 36. This finding highlights the critical role
these elements play in augmenting the efficacy of our learning approach.

3\. Is Retrieval Across Different Embodiments Beneficial? In our experiments,
we utilized the Open X-Embodiment as our primary policy memory bank, a dataset
featuring a variety of embodiments. Our evaluation, depicted in Table 4,
focuses on the performance implications of using data exclusively from Franka
robots compared to a multi-embodiment dataset. We observed a slight decline in
performance with the Franka-only data. This could be attributed to the richer
diversity of environments and commands present in the broader dataset.

##  5 Conclusion

Training with pre-defined datasets often leads to a limited scope of ability
and knowledge acquisition. In this paper, we introduce Retrieval-Augmented
Embodied Agent (RAEA), a novel framework that enhances an embodied agent
through a policy retriever and generation process. The primary objective of
this retrieval process is to efficiently and effectively harness valuable
insights from a comprehensive dataset of experiences, thereby aiding the agent
in achieving its goals more proficiently. Through multiple ablation studies,
we have underscored the significance of the various components within RAEA.
Overall, our RAEA methodology presents an innovative and practical approach to
leveraging collective knowledge from diverse datasets of different
embodiments.

## References

  * Bahl et al. [2022] Shikhar Bahl, Abhinav Gupta, and Deepak Pathak.  Human-to-robot imitation in the wild.  _arXiv preprint arXiv:2207.09450_ , 2022.
  * Bahl et al. [2023] Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak.  Affordances from human videos as a versatile representation for robotics.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13778–13790, 2023.
  * Bertasius et al. [2016] Gedas Bertasius, Hyun Soo Park, Stella X Yu, and Jianbo Shi.  First person action-object detection with egonet.  _arXiv preprint arXiv:1603.04908_ , 2016.
  * Bousmalis et al. [2023] Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex X Lee, Maria Bauza, Todor Davchev, Yuxiang Zhou, Agrim Gupta, Akhil Raju, et al.  Robocat: A self-improving foundation agent for robotic manipulation.  _arXiv preprint arXiv:2306.11706_ , 2023.
  * Brahmbhatt et al. [2019] Samarth Brahmbhatt, Cusuh Ham, Charles C Kemp, and James Hays.  Contactdb: Analyzing and predicting grasp contact via thermal imaging.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 8709–8719, 2019.
  * Brohan et al. [2022] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al.  Rt-1: Robotics transformer for real-world control at scale.  _arXiv preprint arXiv:2212.06817_ , 2022.
  * Burns et al. [a] Kaylee Burns, Tianhe Yu, Chelsea Finn, and Karol Hausman.  Pre-training for manipulation: The case for shape biased vision transformers.  a.
  * Burns et al. [b] Kaylee Burns, Tianhe Yu, Chelsea Finn, and Karol Hausman.  Pre-training for manipulation: The case for shape biased vision transformers.  b.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9650–9660, 2021.
  * Chen et al. [2021] Annie S Chen, Suraj Nair, and Chelsea Finn.  Learning generalizable robotic reward functions from” in-the-wild” human videos.  _arXiv preprint arXiv:2103.16817_ , 2021.
  * Chen et al. [2022a] Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William W Cohen.  Murag: Multimodal retrieval-augmented generator for open question answering over images and text.  _arXiv preprint arXiv:2210.02928_ , 2022a.
  * Chen et al. [2022b] Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen.  Re-imagen: Retrieval-augmented text-to-image generator.  _arXiv preprint arXiv:2209.14491_ , 2022b.
  * Cui et al. [2022a] Yuchen Cui, Scott Niekum, Abhinav Gupta, Vikash Kumar, and Aravind Rajeswaran.  Can foundation models perform zero-shot task specification for robot manipulation?  In _Learning for Dynamics and Control Conference_ , pages 893–905. PMLR, 2022a.
  * Cui et al. [2022b] Zichen Jeff Cui, Yibin Wang, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto.  From play to policy: Conditional behavior generation from uncurated robot data.  _arXiv preprint arXiv:2210.10047_ , 2022b.
  * Dasari et al. [2019] Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, et al.  Robonet: Large-scale multi-robot learning.  _arXiv preprint arXiv:1910.11215_ , 2019.
  * Depierre et al. [2018] Amaury Depierre, Emmanuel Dellandréa, and Liming Chen.  Jacquard: A large scale dataset for robotic grasp detection.  In _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 3511–3516. IEEE, 2018.
  * Ebert et al. [2018] Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine.  Visual foresight: Model-based deep reinforcement learning for vision-based robotic control.  _arXiv preprint arXiv:1812.00568_ , 2018.
  * Ebert et al. [2021] Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine.  Bridge data: Boosting generalization of robotic skills with cross-domain datasets.  _arXiv preprint arXiv:2109.13396_ , 2021.
  * Eppner et al. [2021] Clemens Eppner, Arsalan Mousavian, and Dieter Fox.  Acronym: A large-scale grasp dataset based on simulation.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 6222–6227. IEEE, 2021.
  * Fang et al. [2020] Hao-Shu Fang, Chenxi Wang, Minghao Gou, and Cewu Lu.  Graspnet-1billion: A large-scale benchmark for general object grasping.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 11444–11453, 2020.
  * Fang et al. [2023] Hao-Shu Fang, Hongjie Fang, Zhenyu Tang, Jirong Liu, Junbo Wang, Haoyi Zhu, and Cewu Lu.  Rh20t: A robotic dataset for learning diverse skills in one-shot.  _arXiv preprint arXiv:2307.00595_ , 2023.
  * Fu et al. [2020] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine.  D4rl: Datasets for deep data-driven reinforcement learning.  _arXiv preprint arXiv:2004.07219_ , 2020.
  * Girdhar et al. [2023] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra.  Imagebind: One embedding space to bind them all.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15180–15190, 2023.
  * Gu et al. [2023] Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, et al.  Maniskill2: A unified benchmark for generalizable manipulation skills.  _arXiv preprint arXiv:2302.04659_ , 2023.
  * Hansen et al. [2022] Nicklas Hansen, Zhecheng Yuan, Yanjie Ze, Tongzhou Mu, Aravind Rajeswaran, Hao Su, Huazhe Xu, and Xiaolong Wang.  On pre-training for visuo-motor control: Revisiting a learning-from-scratch baseline.  _arXiv preprint arXiv:2212.05749_ , 2022.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Hill et al. [2020] Felix Hill, Sona Mokra, Nathaniel Wong, and Tim Harley.  Human instruction-following with deep reinforcement learning via transfer-learning from text.  _arXiv preprint arXiv:2005.09382_ , 2020.
  * Huang et al. [2023] Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei.  Voxposer: Composable 3d value maps for robotic manipulation with language models.  _arXiv preprint arXiv:2307.05973_ , 2023.
  * Jang et al. [2022] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn.  Bc-z: Zero-shot task generalization with robotic imitation learning.  In _Conference on Robot Learning_ , pages 991–1002. PMLR, 2022.
  * Jiang et al. [2022] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan.  Vima: General robot manipulation with multimodal prompts.  _arXiv preprint arXiv:2210.03094_ , 2022.
  * Kalashnikov et al. [2018] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al.  Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation.  _arXiv preprint arXiv:1806.10293_ , 2018.
  * Karpukhin et al. [2020] Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.  Dense passage retrieval for open-domain question answering.  _arXiv preprint arXiv:2004.04906_ , 2020.
  * Kumar et al. [2022] Aviral Kumar, Anikait Singh, Frederik Ebert, Mitsuhiko Nakamoto, Yanlai Yang, Chelsea Finn, and Sergey Levine.  Pre-training for robots: Offline rl enables learning new tasks from a handful of trials.  _arXiv preprint arXiv:2210.05178_ , 2022.
  * Levine et al. [2018] Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen.  Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection.  _The International journal of robotics research_ , 37(4-5):421–436, 2018.
  * Li et al. [2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023.
  * Liu et al. [2023] Haotian Liu, Kilho Son, Jianwei Yang, Ce Liu, Jianfeng Gao, Yong Jae Lee, and Chunyuan Li.  Learning customized visual models with retrieval-augmented knowledge.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15148–15158, 2023.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 10012–10022, 2021.
  * Lynch and Sermanet [2020] Corey Lynch and Pierre Sermanet.  Language conditioned imitation learning over unstructured data.  _arXiv preprint arXiv:2005.07648_ , 2020.
  * Mahler et al. [2017] Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael Laskey, Richard Doan, Xinyu Liu, Juan Aparicio Ojea, and Ken Goldberg.  Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics.  _arXiv preprint arXiv:1703.09312_ , 2017.
  * Mu et al. [2023] Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo.  Embodiedgpt: Vision-language pre-training via embodied chain of thought.  _arXiv preprint arXiv:2305.15021_ , 2023.
  * Nair et al. [2022a] Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et al.  Learning language-conditioned robot behavior from offline data and crowd-sourced annotation.  In _Conference on Robot Learning_ , pages 1303–1315. PMLR, 2022a.
  * Nair et al. [2022b] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta.  R3m: A universal visual representation for robot manipulation.  _arXiv preprint arXiv:2203.12601_ , 2022b.
  * Nasiriany et al. [2019] Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine.  Planning with goal-conditioned policies.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  _arXiv preprint arXiv:2304.07193_ , 2023.
  * Padalkar et al. [2023] Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et al.  Open x-embodiment: Robotic learning datasets and rt-x models.  _arXiv preprint arXiv:2310.08864_ , 2023.
  * Parisi et al. [2022] Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta.  The unsurprising effectiveness of pre-trained vision models for control.  In _International Conference on Machine Learning_ , pages 17359–17371. PMLR, 2022.
  * Pathak et al. [2018] Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A Efros, and Trevor Darrell.  Zero-shot visual imitation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , pages 2050–2053, 2018.
  * Qi et al. [2017] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 652–660, 2017.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Radosavovic et al. [2023] Ilija Radosavovic, Baifeng Shi, Letian Fu, Ken Goldberg, Trevor Darrell, and Jitendra Malik.  Robot learning with sensorimotor pre-training.  _arXiv preprint arXiv:2306.10007_ , 2023.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551, 2020.
  * Reed et al. [2022] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al.  A generalist agent.  _arXiv preprint arXiv:2205.06175_ , 2022.
  * Sermanet et al. [2018] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain.  Time-contrastive networks: Self-supervised learning from video.  In _2018 IEEE international conference on robotics and automation (ICRA)_ , pages 1134–1141. IEEE, 2018.
  * Shah et al. [2023] Rutav Shah, Roberto Martín-Martín, and Yuke Zhu.  Mutex: Learning unified policies from multimodal task specifications.  _arXiv preprint arXiv:2309.14320_ , 2023.
  * Shao et al. [2021] Lin Shao, Toki Migimatsu, Qiang Zhang, Karen Yang, and Jeannette Bohg.  Concept2robot: Learning manipulation concepts from instructions and human demonstrations.  _The International Journal of Robotics Research_ , 40(12-14):1419–1434, 2021.
  * Sharma et al. [2019] Pratyusha Sharma, Deepak Pathak, and Abhinav Gupta.  Third-person visual imitation learning via decoupled hierarchical controller.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Shilane et al. [2004] Philip Shilane, Patrick Min, Michael Kazhdan, and Thomas Funkhouser.  The princeton shape benchmark.  In _Proceedings Shape Modeling Applications, 2004._ , pages 167–178. IEEE, 2004.
  * Shridhar et al. [2023] Mohit Shridhar, Lucas Manuelli, and Dieter Fox.  Perceiver-actor: A multi-task transformer for robotic manipulation.  In _Conference on Robot Learning_ , pages 785–799. PMLR, 2023.
  * Smith et al. [2019] Laura Smith, Nikita Dhawan, Marvin Zhang, Pieter Abbeel, and Sergey Levine.  Avid: Learning multi-stage tasks via pixel-level translation of human videos.  _arXiv preprint arXiv:1912.04443_ , 2019.
  * Walke et al. [2023a] Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Quan Vuong, Andre He, et al.  Bridgedata v2: A dataset for robot learning at scale.  _arXiv preprint arXiv:2308.12952_ , 2023a.
  * Walke et al. [2023b] Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Quan Vuong, Andre He, et al.  Bridgedata v2: A dataset for robot learning at scale.  _arXiv preprint arXiv:2308.12952_ , 2023b.
  * Wang et al. [2023] Yixuan Wang, Zhuoran Li, Mingtong Zhang, Katherine Driggs-Campbell, Jiajun Wu, Li Fei-Fei, and Yunzhu Li.  D3 fields: Dynamic 3d descriptor fields for zero-shot generalizable robotic manipulation.  _arXiv preprint arXiv:2309.16118_ , 2023.
  * Wu et al. [2023] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua.  Next-gpt: Any-to-any multimodal llm.  _arXiv preprint arXiv:2309.05519_ , 2023.
  * Xie et al. [2023] Chen-Wei Xie, Siyang Sun, Xiong Xiong, Yun Zheng, Deli Zhao, and Jingren Zhou.  Ra-clip: Retrieval augmented contrastive language-image pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19265–19274, 2023.
  * Xie et al. [2022] Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I Wang, et al.  Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models.  _arXiv preprint arXiv:2201.05966_ , 2022.
  * Yasunaga et al. [2021] Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec.  Qa-gnn: Reasoning with language models and knowledge graphs for question answering.  _arXiv preprint arXiv:2104.06378_ , 2021.
  * Yasunaga et al. [2022] Michihiro Yasunaga, Jure Leskovec, and Percy Liang.  Linkbert: Pretraining language models with document links.  _arXiv preprint arXiv:2203.15827_ , 2022.
  * Yasunaga et al. [2023] Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Richard James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih.  Retrieval-augmented multimodal language modeling.  2023\.
  * Yu et al. [2016] Kuan-Ting Yu, Maria Bauza, Nima Fazeli, and Alberto Rodriguez.  More than a million ways to be pushed. a high-fidelity experimental dataset of planar pushing.  In _2016 IEEE/RSJ international conference on intelligent robots and systems (IROS)_ , pages 30–37. IEEE, 2016.
  * Yu et al. [2020] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine.  Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.  In _Conference on robot learning_ , pages 1094–1100. PMLR, 2020.
  * Yuan et al. [2022] Zhecheng Yuan, Zhengrong Xue, Bo Yuan, Xueqian Wang, Yi Wu, Yang Gao, and Huazhe Xu.  Pre-trained image encoder for generalizable visual reinforcement learning.  _Advances in Neural Information Processing Systems_ , 35:13022–13037, 2022.
  * Zitkovich et al. [2023] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al.  Rt-2: Vision-language-action models transfer web knowledge to robotic control.  In _7th Annual Conference on Robot Learning_ , 2023.


11institutetext: Bielefeld University, Inspiration 1, 33615 Bielefeld, Germany
11email<EMAIL_ADDRESS>
# Adversarial Attacks on Leakage Detectors in Water Distribution Networks

Paul Stahlhofen  André Artelt  0000-0002-2426-3126 Luca Hermes
0000-0002-7568-7981 Barbara Hammer  0000-0002-0935-5591

###### Abstract

Many Machine Learning models are vulnerable to adversarial attacks: There
exist methodologies that add a small (imperceptible) perturbation to an input
such that the model comes up with a wrong prediction. Better understanding of
such attacks is crucial in particular for models used in security-critical
domains, such as monitoring of water distribution networks, in order to devise
counter-measures enhancing model robustness and trustworthiness.

We propose a taxonomy for adversarial attacks against machine learning based
leakage detectors in water distribution networks. Following up on this, we
focus on a particular type of attack: an adversary searching the least
sensitive point, that is, the location in the water network where the largest
possible undetected leak could occur. Based on a mathematical formalization of
the least sensitive point problem, we use three different algorithmic
approaches to find a solution. Results are evaluated on two benchmark water
distribution networks.

###### Keywords:

Adversarial Attacks Water Distribution Networks Leakage Detectors.

##  1 Introduction

According to the EU guidelines on Trustworthy AI, robustness is a key
requirement of Machine Learning (ML) based systems deployed in security-
critical domains [5]. Judging and improving a models’ robustness requires to
understand its weakness so that appropiate countermeasure can be taken (see
e.g. [21], [15]). A structural weakness of many models is the existence of
specifically designed inputs, so called adversarial inputs, that can cause the
model to make wrong predictions with a high confidence. These adversarial
inputs, first described in [16], have been used to expose a lack of robustness
for many state of the art ML models [10, 2, 22].

In this paper we focus on an application domain where robustness is of utmost
importance: Water Distribution Networks (WDNs). A high amount of annual water
loss in water networks across the world [12] and an increased likelihood of
droughts due to climate change [3] call for an improvement and robustification
of water management. In the context of WDNs, ML based systems are used for
various tasks [18] e.g. detection and localization of events such as leakages
[17, 19] and contaminations [23].

#### Related work

In order to gain insight into vulnerabilities of a water distribution network,
the authors of [9] propose a methodology for building finite state processes
to find a-priori week spots independent of any monitoring system. The authors
of [11] construct physics-constrained adversarial attacks for ML models
operating on cyber-physical systems. However, in their case study they only
analyze the manipulation of sensor readings for flow sensors and do not
consider pressure sensors which constitute the most common type of sensor in
real world WDNs.

#### Our contributions

In this work we make the following contributions:

  * •

We propose a taxonomy for adversarial attacks against leakage detectors in
WDNs.

  * •

We investigate one particular type of attack in more detail: The search for
the least sensitive point in a WDN, which is the location in the water network
where the largest possible undetected leak could occur. We formalize this
attack and propose three algorithmic approaches, which we empirically evaluate
in a case study on two benchmark WDNs.

The remainder of this work is structured as follows: In Section 2, we
introduce adversarial attacks in general, as well as some important concepts
of leakage detection in WDNs. Following up on this, we give a taxonomy of
adversarial attacks on leakage detectors and formalize the least sensitive
point problem in Section 3. Next, in Section 4 we empirically evaluate our
proposed methods for finding the least sensitive point in a WDN. Finally,
Section 5 discusses the results before we conclude with a brief summary in
Section 6.

##  2 Foundations

###  2.1 Adversarial Attacks

The concept of adversarial attacks was first described in the context of image
classification in [16]: There exist small and imperceptible perturbations of
the original image that fool the classifier to output a completely different
high-certainty classification. The concept of an adversarial attack can be
generalized (similar to [8]) as follows:

######  Definition 1 (Adversarial Attack)

An adversarial attack is a procedure to create inputs to a Machine Learning
model, that will cause the model to make a mistake.

It is important to note that the procedure for constructing an adversarial
does not necessarily operate in the input space directly. In particular,
consider a ML model that uses measurements as inputs: These measurements
reflect the state of a system monitored by the model. Procedures that change
the system in order to create measurements that will cause the model to make a
mistake are also considered as adversarial attacks here. These attacks have
the advantage that they are intrinsically physics-constrained: When changing
measurements directly, one has to make sure that the resulting measurements
obey physical laws, as discussed in [11]. Making changes to the system itself
will automatically result in measurements that reflect physical processes.

###  2.2 Leakage Detection in Water Distribution Networks

Leakage detectors typically rely on measurements of hydraulic variables such
as pressure and flow at certain points in a network. Measurements are taken at
discrete timesteps and depend on various factors, including network topology,
water consumption at network nodes and potential leakages. When creating an
adversarial input to the detector, we are mainly interested in how the
pressure values are influenced by leaks. Hence, we consider measurements based
on leakage events characterized by the area of the leak. All other factors
like water consumption and network topology are assumed to be fixed.

######  Definition 2 (Measurement Function)

A measurement function maps a vector of leak areas
$\bm{a}_{t}\in\mathbb{R}^{N}$ for all $N$ nodes of a water distribution
network at some timestep $t$ to pressure measurements at some nodes equipped
with sensors. It is parametrized also by the timestep itself.

$\begin{split}f_{\mathit{measure}}:\mathbb{R}^{N}\times\\{1,\ldots,T\\}&\to\mathbb{R}^{S}\\\
(\bm{a}_{t},t)&\mapsto\bm{y}_{t}\end{split}$ (1)

Given the pressure measurements, we can now define the leakage detector as a
binary predictor

######  Definition 3 (Leakage Detector)

A leakage detector is a function mapping pressure measurements to a binary
output, indicating whether a leak was detected or not.

$\begin{split}f_{\mathit{detect}}:\mathbb{R}^{S}&\to\\{0,1\\}\\\
\bm{y}_{t}&\mapsto\begin{cases*}1&\text{if a leak was detected}\\\
0&\text{otherwise}\end{cases*}\end{split}$ (2)

##  3 Adversarials in Water Distribution Networks

###  3.1 Taxonomy of Adversarials in WDNs

Figure 1: Taxonomy showing different types of adversarial attacks in WDNs
* Methods proposed in [11]
** Metholds proposed here

There exist two different scenarios how a detector could be fooled: The first
one would be an input causing the detector to predict a leak, even though no
leak is actually present. We refer to this as the false alarm scenario; The
second one would be an input causing the detector not to predict a leak, even
though there is one. This case can be termed the hidden attack scenario.

In case of the false alarm scenario, there are again two different
possibilities of attacking the system: An adversary might try to manipulate a
few network elements (e.g. sensors, pipes, etc.) they have managed to access
or the attacker makes external changes to the water demand at certain nodes.
Note that, even if the adversary has managed to gain control over some parts
of the network infrastructure, they are not interested in actually creating a
leak or other damage in this scenario, but rather aims at causing the leakage
detector to raise a false alarm. Thus, manipulating sensor values or
disturbing the inter-sensor communication might be a promising approach. In
particular, a large decrement of pressure measurements or the complete
abortion of sensory recordings are likely to cause a wrong leakage prediction.
In the attack scenario where the attacker can change node demands only, the
success of the attack will depend on the number of nodes under adversary
control and the extent to which the demand can be changed at these nodes.
Generally, both a high increase in the demand e.g. by opening all water taps
in an industrial complex as well as a high decrease constitute possible
reasons for a prediction failure resulting in a false alarm.

The hidden attack scenario always involves an adversary who creates an actual
leak in the physical network. In order to fool the leakage detector not to
find the leak, there are again two possibilities: The attacker could try to
manipulate the detector in order to make it less sensitive or they could place
the leak at a point where the detector is not very sensitive anyway. In the
manipulation case, sensors, programmable logic controllers (PLCs) or the
sensory control and data acquisition system (SCADA) could be targets of the
attack111These constituents of a network monitoring system were used in the
Battle of the Attack Detection Algorithms (BATADAL) [17]. A monitoring system
might have additional entry points for an attack in practise.. Otherwise, the
adversary will have to search for points in the network where the detector has
little sensitivity to leaks.

Because investigating all the aforementioned attack scenarios in detail is
behind the scope of this work, we focus on a particular type of hidden attack:
Searching for locations in the network where the detector is the least
sensitive to leaks.

###  3.2 The Least Sensitive Point Problem

If an adversary does not have access to sensors or other parts of the leakage
detection system but still tries to create a leak that should not be detected,
they have to find the location in the network where the largest undetected
leak could occur. In other words, the adversary is looking for the
location/point where the detector is least sensitive, which we formalize as
follows:

######  Definition 4 (Least Sensitive Point)

Given a water distribution network $W=(V,E)$ with a set of $N$ nodes
$V=\\{v_{n}|n\in\\{1,\ldots,N\\}\\}$ and a number of $S\leq N$ pressure
sensors installed in it, a measurement function $f_{\mathit{measure}}$ and a
leakage detector $f_{\mathit{detect}}$, the least sensitive point (LSP)
$\operatorname{lsp}(W)$ in this network $W$ is defined as follows:

$\begin{split}&\operatorname{lsp}(W)=\operatorname*{arg\,max}_{v_{n}\in
V}\max_{\begin{subarray}{c}t\in\\{1,\ldots,T-K\\}\\\
\alpha\in\mathbb{R}^{+}\end{subarray}}\alpha\\\ &\text{s.t.}\quad
f_{detect}\left(f_{measure}(\alpha\bm{e}_{n},t+k)\right)=0\quad\forall
k\in\\{0,\ldots,K\\}\end{split}$ (3)

where $K$ is a fixed time window length and $\bm{e}_{n}$ is the $n$-th
canonical basis vector.

In our experiments, we set the value of $K$ equal to the leak duration which
we fixed at 3 hours for all experiments.
We approach the least sensitive point problem (Definition 4) with three
algorithmic methods for two different network topologies.

###  3.3 Algorithmic Approaches

We propose the following algorithmic approaches to find the least sensitive
point (Definition 4):

  * •

Bisection Search

  * •

Basic Genetic Algorithm

  * •

Genetic Algorithm with Spectral Embeddings

We empirically evluate all three mehods in a case study in Section 4.

#### Bisection Search

The Bisection Search is based on the fact that for the least sensitive point
there has to exist some leak area $\alpha^{*}$, such that a leak of that area
remains undetected only at the LSP, while a leak of the same area would be
detected at every other node. Thus, one can perform a simple line search over
the leak area in order to find $\alpha^{*}$. Bisection search can be costly,
in particular when all timesteps in the simulation are viewed as potential
starting times of the leak. However, it yields the benefit that it is
guaranteed to find the least sensitive point in the given search space. To
reduce the computational load, we implement a pruning of nodes and starting
times based on intermediate results. Let $\alpha^{i}$ be the leak area after
iteration $i$. If a leak of size $\alpha^{i}$ remained undetected for at least
one node-time pair, we remove the following elements from the search space:

  1. 1.

Every node for which a leak of size $\alpha^{i}$ was detected at every
starting time

  2. 2.

Every starting time for which a leak of size $\alpha^{i}$ was detected at
every node

In practise, this lead to a strong reduction of the search space after the
first iteration.

#### Genetic Algorithm

In the Basic Genetic Algorithm, we encode nodes and leak starting time as
genes and optimize the following fitness function

$\mathit{Fitness}(v_{n},t)=\max\alpha\quad\text{s.t.}f_{\mathit{detect}}(f_{\mathit{measure}}(\alpha\bm{e}_{n},t))=0$
(4)

The largest undetected leak for each node-time pair is again computed via a
line search. In order to avoid unnecessary area maximization, we use dynamic
programming: The largest undetected leak area found so far is tested first for
every new node-time pair. If this is detected, no area maximization was
performed and the fitness is set to zero.

#### Genetic Algorithm with Spectral Embeddings

In order to take the network topology into account, we consider a second
version of the Genetic Algorithm, using a spectral analysis of the network’s
graph Laplacian. Every node is assigned a four dimensional vector, containing
elements of the 2nd through 5th eigenvector of the graph Laplacian for that
node. Each vector element of the node embedding is then treated as a different
gene by the Genetic Algorithm. After every recombination and mutation step,
the nearest neighbour of the resulting embedding vector is returned as
offspring.

None of the approaches used here makes any assumptions on the type of leakage
detector. Hence, they are all model-agnostic in terms of the taxonomy (Figure
1).

##  4 Case Study: LSP Search in Two Benchmark Networks

###  4.1 Leakage Detector

We implement a classic residual-based leakage detector [6, 14] by means of a
sensor specific linear model that predicts pressure based on the pressure
measurements of all other nodes [1]:

$\hat{y}_{s,t}=\bm{w}_{s}^{T}\bm{y}_{-s,t}+b_{s}$ (5)

where $\bm{y}_{-s,t}$ is a vector containing measurements from all sensors
except $s$ at time $t$ and $(\bm{w}_{s},b_{s})$ are sensor specific weights
learned using data. The residuals
$r_{s,t}:=\left\lvert\hat{y}_{s,t}-y_{s,t}\right\rvert$ determine the detector
output. For the Hanoi network, we use a validation set to learn residual
weights $q_{s}$ for each sensor. An alram is then raised if
$\sum_{s}q_{s}r_{s}>1$. In case of the L-Town network, lack of data makes the
learning of residual weights impractical. Instead, we construct thresholds for
each sensor by multiplying the maximum training error at the sensor location
with a small constant. An alarm is raised if at least one sensor residual
exceeds the corresponding threshold.

###  4.2 Water Networks

In order to obtain pressure values as input for the leakage detector, we used
the hydraulic modelling software EPANET [13] which simmulates flow and
pressure values over time, given an hydraulic model of a water distribution
network and water demands for each node and timestep. EPANET also allows to
run simulations with leaks of arbitrary area, location, starting time and
duration.

We first evaluate our proposed methodology on the Hanoi network [7]. This
network consists of $31$ junctions and one water reservoir from which water is
entering the system through a pump. Realistic water demands at the network
nodes are generated using the code provided with the LeakDB benchmark dataset
[20]. Next, we extend our case study to the larger and more realistic L-Town
network [19]. This network contains $782$ junctions, receiving their water
from two reservoirs and one tank which is used for intermediate storage. The
authors of [19] provided realistic demand values along with the hydraulic
model of the network.

For both networks we train the leakage detector on the first five days of the
timeseries and search for the least sensitive point (Definition 4) during the
two days afterwards. While detector training on realistic demands may not
always be possible in advance, it is still reasonable to assume the that the
first few days produced by those demands are utilized to calibrate the
detector for the network at hand. In particular, infering thresholds from
training errors on realistic data can help to avoid false alarms. For the
Hanoi network, we conduct a second analysis with five training days and a
search space of nine days afterwards. This is not done for L-Town as
simulations over a longer timeseries are computationally demanding for larger
networks.

##  5 Results and Discussion

###  5.1 Hanoi Network

For the Hanoi network, optimal solutions could be determined using the
Bisection Search. First results show that the least sensitive point is always
located close to the water source. This can be explained by the large water
flow from the reservoir to the first nodes. A leakage event along a pipe with
higher flow will lead to a less significant pressure drop when compared to a
leak along a pipe with lower flow. As this fact is known to water utility
administrators, we assume in follow up experiments that these nodes close to
the reservoir might be subject to increased protection. We treat them as
inaccessible for the adversary and remove them from the search space.

Figure 2: Least Sensitive Point search on the Hanoi network: In initial
analysis, points with the lowest detector sensitivity (marked by red
triangles) are found close to the water reservoir. After excluding these nodes
from the search space, the least sensitive point is found at node 10 for the
2-days dataset and at node 23 for the 9-days dataset.

In the 2-days dataset, the least sensitive point among the remaining nodes is
located at node 10. For the 9-days dataset, it is found to be node 23. The
results are visualized in Figure 2.
To compare the performance between the Basic Genetic Algorithm and the
extension with spectral embeddings we run five trials of each algorithm for
both datasets. In all 20 cases, the respective algorithm is able to locate the
least sensitive point correctly. This demonstrates the high accuracy of both
methods on small water networks. In order to gain evidence for this
assumption, a larger number of trials could be conducted in future work.

###  5.2 L-Town Network

Figure 3: Analysis of the L-Town network with Bisection Search for fixed
starting times. Points with a low detector sensitivity are located close to
the water sources. After excluding the initial LSP candidates from the search
space, the least sensitive point is found at node n387 (upward-pointing
triangle).

For the L-Town network, it is not computationaly feasible to find the least
sensitive point globally with the Bisection Search, because the network is
considerably larger than the Hanoi network and thus simulations are much more
time consuming. In order to get a realistic estimate, we determine the LSP at
three fixed leak starting times, 4, 25 and 28 hours after the beginning of the
timeseries. This starting times are assumed to have low overall demands based
on the training data, which makes leaks more difficult to detect. During the
line search we are able to observe intermediate results. For a given leak
area, we find all nodes where a leak of that area starting at one of the fixed
starting times is not detected. Similar to the Hanoi network, we observe an
agreement between those intermediate steps for the different time trials,
indicating that nodes with low sensitivity are located close to the water
sources. Figure 3 shows all nodes for which a leak with an area of 100
$\text{cm}^{2}$ is not detected in at least one of the start time trials as
downward-pointing triangles. From the result we can also obseve that the
direction of water flow plays a major role in leakage detection: Even though
there is a pressure sensor located directly next to the tank in the upper left
area of the network, nodes on the other side of the tank are very vulnerable
to large undetected leaks. This is because the sensor only measures the
pressure of water flowing into the tank and does not support the detection of
downstream leaks. In case of the L-Town network, pressure-based detection
close to the water sources is also hampered by pressure reducing valves,
located after each source. The valves smooth out incoming pressure values,
making the detection of downstream leaks more difficult.

In order to compare the performance of our algorithmic approaches on the
L-Town network, we first re-compute the least sensitive point for the same
starting times used above after excluding the initial LSP candidates marked in
Figure 3 from the search space. All three runs of the Bisection Search agreed
in node n387 as the new least sensitive point (upward-pointing triangle in
Figure 3). We then conduct five trials for each genetic algorithm. Results are
shown in Figure 4. The horizontal line inidicates that each run of the
Bisection Search found the least sensitive point at a maximum undetected leak
area of 80 $\text{cm}^{2}$. All genetic-algorithm results with a larger leak
area identify n387 as the LSP, while all results with a smaller leak area fail
to do so222In theory, it is possible that trials with a smaller leak area
still find the same node, but this is not the case here.. The comparison
suggests that the genetic algorithm with spectral node embeddings performs
well at finding the LSP, also for larger networks. A simple enhancement to
reduce the uncertainty of a single trial would be to conduct multiple
successive trials, keeping the result of the best one as global solution. This
would still be computationally much cheaper than an exhaustive Bisection
Search. In future work, a larger number of trials may be used to determine the
junction accuracy of both algorithms.

Figure 4: Comparison of the three algorithmic approaches.

##  6 Conclusion

In this work, we applied the concept of adversarial attacks to leakage
detection in water distribution networks. For this purpose, we proposed a
taxonomy of potential adversarials against a leakage detector. We then focused
on a particular type of adversarial: Attacking the network at the least
sensitive point. We formalized the least sensitive point problem and proposed
three algorithmic approaches to solve it. In practice, knowing the least
sensitive point and the vulnerability to attacks constitutes crucial
information which enables practitioners to develop more robust methods using
e.g. adversarial training [4] or an improvement of the robustness by
introducing targeted sensors.

We empirically evaluated our proposed methods in a case study on two benchmark
WDNs. For the genetic algorithm with spectral embeddings, experiments yielded
promising results on both networks. This method allows much faster solutions
when compared to the Bisection Search, due to a lower runtime complexity. The
locations of the least sensitive point indicate that the detectors weak spots
are highly dependent on network topology, in particular on the location of the
water sources.

In future work, we plan to address the same problem with another leakage
detection model to compare results between different leakage detectors. In
order to increase the robustness, we will determine the effect of incremental
sensor placement on the maximum undetected leak area. For this purpose, we
will simulate sensors at the location where the least sensitive point has
previously been detected and re-run the experiments with the new sensors in
place. In this way, knowledge about adversarials can be helpful in practical
applications.

#### Acknowledgements

We gratefully acknowledge funding from the European Research Council (ERC)
under the ERC Synergy Grant Water-Futures (Grant agreement No. 951424).

## References

  * [1] Artelt, A., Vrachimis, S., Eliades, D., Polycarpou, M., Hammer, B.: One explanation to rule them all – ensemble consistent explanations (2022). https://doi.org/10.48550/ARXIV.2205.08974, https://arxiv.org/abs/2205.08974
  * [2] Athalye, A., Engstrom, L., Ilyas, A., Kwok, K.: Synthesizing robust adversarial examples. CoRR abs/1707.07397 (2017), http://arxiv.org/abs/1707.07397
  * [3] Ault, T.R.: On the essentials of drought in a changing climate. Science 368(6488), 256–260 (2020). https://doi.org/10.1126/science.aaz5492, https://www.science.org/doi/abs/10.1126/science.aaz5492
  * [4] Bai, T., Luo, J., Zhao, J., Wen, B., Wang, Q.: Recent advances in adversarial training for adversarial robustness. arXiv preprint arXiv:2102.01356 (2021)
  * [5] Commission, E., Directorate-General for Communications Networks, C., Technology: Ethics guidelines for trustworthy AI. Publications Office (2019). https://doi.org/doi/10.2759/346720
  * [6] Eliades, D., Polycarpou, M.M.: Leakage fault detection in district metered areas of water distribution systems. Journal of Hydroinformatics 14(4), 992–1005 (2012)
  * [7] Fujiwara, O., Khang, D.B.: A two-phase decomposition method for optimal design of looped water distribution networks. Water Resources Research 26(4), 539–549 (1990)
  * [8] Goodfellow, I., Papernot, N., Huang, S., Duan, Y., Abbeel, P., Clark, J.: Attacking machine learning with adversarial examples. https://openai.com/blog/adversarial-example-research/ (2017)
  * [9] Karrenberg, C., Benavides, J., Berglund, E., Kang, E., Baugh, J.: Identifying cyber-physical vulnerabilities of water distribution systems using finite state processes. In: 2nd International Joint Conference on Water Distribution System Analysis & Computing and Control in the Water Industry, WDSA CCWI 2022 (2022)
  * [10] Kurakin, A., Goodfellow, I.J., Bengio, S.: Adversarial examples in the physical world. CoRR abs/1607.02533 (2016), http://arxiv.org/abs/1607.02533
  * [11] Li, J., Yang, Y., Sun, J.S., Tomsovic, K., Qi, H.: Conaml: Constrained adversarial machine learning for cyber-physical systems (2020). https://doi.org/10.48550/ARXIV.2003.05631, https://arxiv.org/abs/2003.05631
  * [12] Liemberger, R., Wyatt, A.: Quantifying the global non-revenue water problem. Water Supply 19(3), 831–837 (07 2018). https://doi.org/10.2166/ws.2018.129, https://doi.org/10.2166/ws.2018.129
  * [13] Rossman, L.A., Woo, H., Tryby, M., Shang, F., Janke, R., Haxton, T.: EPANET 2.2. User Manual. U.S. Environmental Protection Agency, Washington D.C. (2020)
  * [14] Santos-Ruiz, I., López-Estrada, F.R., Puig, V., Blesa, J.: Estimation of node pressures in water distribution networks by gaussian process regression. In: 2019 4th Conference on Control and Fault Tolerant Systems (SysTol). pp. 50–55. IEEE (2019)
  * [15] Shafahi, A., Najibi, M., Ghiasi, M.A., Xu, Z., Dickerson, J., Studer, C., Davis, L.S., Taylor, G., Goldstein, T.: Adversarial training for free! In: Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 32. Curran Associates, Inc. (2019), https://proceedings.neurips.cc/paper/2019/file/7503cfacd12053d309b6bed5c89de212-Paper.pdf
  * [16] Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R.: Intriguing properties of neural networks. In: International Conference on Learning Representations (2014), http://arxiv.org/abs/1312.6199
  * [17] Taormina, R., Galelli, S., Tippenhauer, N.O., Salomons, E., Ostfeld, A., Eliades, D.G., Aghashahi, M., Sundararajan, R., Pourahmadi, M., Banks, M.K., Brentan, B.M., Campbell, E., Lima, G., Manzi, D., Ayala-Cabrera, D., Herrera, M., Montalvo, I., Izquierdo, J., Luvizotto, E., Chandy, S.E., Rasekh, A., Barker, Z.A., Campbell, B., Shafiee, M.E., Giacomoni, M., Gatsis, N., Taha, A., Abokifa, A.A., Haddad, K., Lo, C.S., Biswas, P., Pasha, M.F.K., Kc, B., Somasundaram, S.L., Housh, M., Ohar, Z.: Battle of the attack detection algorithms: Disclosing cyber attacks on water distribution networks. Journal of Water Resources Planning and Management 144(8), 04018048 (2018). https://doi.org/10.1061/(ASCE)WR.1943-5452.0000969, https://ascelibrary.org/doi/abs/10.1061/%28ASCE%29WR.1943-5452.0000969
  * [18] Vaquet, V., Ashraf, I., Hinder, F., Artelt, A., Heihoff, B., Lammers, K., Vaquet, J., Strothherm, J., Brinkrolf, J., Hammer, B.: Ml challenges in water distribution networks - a survey and a graph neural network-based solution for leakage detection and localization. In: Submitted in IJCNN (2023)
  * [19] Vrachimis, S.G., Eliades, D.G., Taormina, R., Kapelan, Z., Ostfeld, A., Liu, S., Kyriakou, M., Pavlou, P., Qiu, M., Polycarpou, M.M.: Battle of the Leakage Detection and Isolation Methods. Journal of Water Resources Planning and Management 148(12), 04022068 (Dec 2022). https://doi.org/10.1061/(ASCE)WR.1943-5452.0001601, https://ascelibrary.org/doi/10.1061/%28ASCE%29WR.1943-5452.0001601
  * [20] Vrachimis, S.G., Kyriakou, M.S., Eliades, D.G., Polycarpu, M.M.: Leakdb: A benchmark dataset for leakage diagnosis in water distribution networks. In: WDSA/CCWI Joint Conference Proceedings. vol. 1 (7 2018)
  * [21] Xie, C., Tan, M., Gong, B., Wang, J., Yuille, A.L., Le, Q.V.: Adversarial examples improve image recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020)
  * [22] Xu, K., Zhang, G., Liu, S., Fan, Q., Sun, M., Chen, H., Chen, P.Y., Wang, Y., Lin, X.: Adversarial t-shirt! evading person detectors in a physical world (2019). https://doi.org/10.48550/ARXIV.1910.11099, https://arxiv.org/abs/1910.11099
  * [23] Zhu, M., Wang, J., Yang, X., Zhang, Y., Zhang, L., Ren, H., Wu, B., Ye, L.: A review of the application of machine learning in water quality evaluation. Eco-Environment & Health (2022)