# Sensing and Backscatter Communication Integration: Realizing Efficiency in
Wireless Systems for IoT

Shayan Zargari, Diluka Galappaththige, and Chintha Tellambura S. Zargari, D.
Galappaththige, and C. Tellambura with the Department of Electrical and
Computer Engineering, University of Alberta, Edmonton, AB, T6G 1H9, Canada
(e-mail: {zargari, diluka.lg, ct4}@ualberta.ca).

###### Abstract

In an era driven by the Internet of Things (IoT) and rapid wireless
communication advances, the synergy between sensing and backscatter
communication (BackCom) has emerged as a frontier of research. This paper
delves deep into the integration of sensing technologies with BackCom, a
burgeoning field with significant implications for energy-efficient wireless
systems. By tracing the historical developments and principles of BackCom, we
establish the foundational understanding crucial for integrating advanced
sensing methodologies. Our study adopts a mixed-method approach, combining
quantitative analyses of system performances with qualitative assessments of
various integration techniques. Furthermore, these integrated systems showcase
enhanced adaptability in dynamic environments, a pivotal attribute for future
IoT applications. These findings hold profound implications for industries
focusing on smart technologies, as they underscore the potential for achieving
both sustainable and efficient communication. Our research distinguishes
itself by not only highlighting the benefits but also addressing the
challenges faced in this integration, providing a comprehensive overview of
the topic.

###### Index Terms:

Backscatter communication (BackCom), Integrated sensing and communication
(ISAC), Passive tags.

##  I Introduction

The dawning of the digital epoch is epitomized by the Internet of Things
(IoT)—an era where the tangible intertwines seamlessly with the virtual,
weaving a cohesive narrative of inter-connectedness and data exchange [1].
Propelling this monumental shift is the remarkable stride in wireless
communication and the miniaturization of both computing and sensing modules.
At the very heart of this transformation lies massive machine-type
communication (mMTC), which orchestrates a harmonious dialogue between devices
and central mMTC servers, ensuring a consistent modus operandi across an
expansive spectrum of applications [2].

However, the momentum of this digital revolution, anchored by mMTC, encounters
certain roadblocks. The energy needed to power devices, the cost of deploying
them in large numbers, and the limited ways they can communicate all present
unique challenges [2]. Sensing, the very essence of myriad IoT applications,
plays a pivotal role, ranging from discerning the whispers of our environment
to tracking the nuances of biological entities. Simultaneously, the advent of
backscatter communication (BackCom) has emerged as a promising panacea. In
particular, ambient BackCom (AmBC) sidesteps energy-intensive traditional
communication by leveraging pre-existing ambient signals and modulating them
for data transmission [3].

Table I: Features and Challenges of AmBC Systems

Features |  Challenges |  Solutions |  Advantages
---|---|---|---
Energy Efficiency |  Limited range and low data rate due to passive reflection and modulation of ambient RF signals. |  Enhancement of EH techniques and optimization of modulation schemes. Utilization of surrounding signals broadcast from ambient RF sources. |  Battery-free operation within a power range of 1 to 100, reducing the environmental impact. Enables communication without requiring active RF transmission.
Scalability |  Interference from other RF devices and varying signal availability affect reliability. |  Advanced signal processing and filtering techniques to mitigate interference and enhance signal quality. |  Supports up to $100$ connections connections per m^3, preventing network overload and enhancing adaptability in diverse environments.
Cost-effectiveness |  Complexity and cost of designing specialized hardware and software for AmBC. |  Development of cost-effective and efficient alternatives to traditional radio components. |  Low-cost devices ($0.01\sim 0.5$ each) with extended lifespan exceeding $10$ years, suitable for mass deployment. Minimization of RF components at backscatter devices.
Ubiquity |  Ethical and legal concerns regarding unauthorized use of spectrum resources and security vulnerabilities. |  Utilization of pervasive ambient signals with enhanced security protocols and adherence to legal frameworks. |  Operation in diverse environments, ranging from coverage of $100-200$ m for industrial areas and $2-5$ m for wearables. Does not require a dedicated frequency spectrum which is scarce and expensive.

The amalgamation of AmBC into the IoT landscape promises significant
transformations, as highlighted in Table I. However, this fusion extends
beyond this initial integration. With the evolution of integrated sensing and
communication (ISAC) systems, the convergence of communication capabilities
with sensing prowess is underway [4, 5, 6, 7]. When this fusion intersects
with backscatter technology, it gives rise to what we call integrated sensing
and BackCom (ISABC) [8]. This convergence effectively elevates passive
elements to become proactive data communicators, enhancing both the
granularity of communication and the precision of sensory data. However, it
also introduces the challenge of complicated decoding processes.

This paper aims to clarify the relationship between AmBC and ISAC for both
scholars and industry experts. We initiate with a comprehensive examination of
prevalent AmBC and ISAC architectures, transitioning to the intricacies of
ISABC, analyzing its merits, confronting inherent challenges, and proffering
possible solutions. Our discussion leads to mapping out possible future paths,
creating a comprehensive outlook for the intersection of IoT, AmBC, and ISABC.

##  II Integration of Sensing with BackCom

The fusion of sensing capabilities with BackCom has given rise to a novel
paradigm where devices can not only perceive their environment but also
communicate their findings in a highly energy-efficient manner. The
integration of these features is especially pertinent in the IoT field, where
devices frequently have to function in low-energy settings for a long
duration.

###  II-A Fundamentals of BackCom

BackCom is an ultra-low-power communication paradigm. Unlike traditional
networks, Bakcom uses tags that do not generate their own RF signal. Instead,
they use passive backscatter modulation to reflect the incoming RF signals for
data transmission. Thus, BackCom achieves higher energy efficiency, superior
spectrum utilization, and cost-effectiveness, making it crucial for future
wireless networks [9, 10].

To visualize the foundational components of a simple BackCom network, we
consider a network of a tag, a reader, and an emitter. The tag, equipped with
a low-power integrated circuit (IC), primarily reflects incoming RF signals to
transmit data. It also harvests energy from these signals for its operations,
leading to two design types: (i) passive tags without energy storage and (ii)
semi-passive tags with limited energy storage [9].

The reader is a more robust component equipped with its own power supply and
full-fledged RF components, empowering it to execute intricate RF functions,
including demodulation and decoding, essential to retrieve data from tags. The
reader can manifest in various forms, from mobile phones to wireless fidelity
(Wi-Fi) access points (APs). Finally, the system’s emitter, also known as the
RF source, is responsible for producing the RF signal that the tags reflect.
An emitter can be either a dedicated beacon signal producer or an ambient
transmitter, such as a television (TV) tower or a cellular base station (BS).

####  II-A1 Configurations of BackCom

BackCom has several configurations, each tailored for specific application
scenarios [9, 10]. Here is a detailed overview:

  * •

Monostatic BackCom (MoBC): In MoBC, the reader assumes a dual role by
transmitting the RF signal and decoding the signals reflected by the tag.
However, this setup presents its own set of challenges. As signals travel from
the reader to the tag and back again, they encounter what is known as ”dyadic”
or round-trip path loss. Specifically, when the tag and reader are
significantly spaced apart, the initial RF signal received by the tag may be
too weak to power it adequately. Additionally, the reflected signal on its
return journey to the reader may also be too faint for accurate detection.
Given these inherent limitations, MoBC naturally finds its most suitable
applications in short-range scenarios, such as radio-frequency identification
(RFID).

  * •

Bistatic BackCom (BiBC): Departing from MoBC’s setup, BiBC introduces two
separate entities; a dedicated RF carrier emitter and a reader. This division
tackles MoBC’s twin challenges head-on. Firstly, it expands the limited range
of the forward link. Secondly, it tackles the challenging round-trip path loss
of MoBC. BiBC also provides a safeguard against the “doubly near-far problem.”

  * •

Ambient BackCom (AmBC): Instead of deploying new RF emitters, it recycles
signals from existing ambient ones. Consequently, the overhead of deploying
and upkeeping dedicated RF emitters evaporates, leading to substantial cost
and energy savings. Furthermore, the valuable spectrum can be optimized as
additional allocations become unnecessary. However, the path of innovation is
seldom without its potholes. The unpredictable nature of ambient RF signals
may interfere directly with readers. Moreover, designing an efficient AmBC
system is like trying to hit a moving tag, especially when characteristics of
ambient RF emitters, like power outputs and locations, are unpredictable and
beyond control.

Fig. 1 depicts the BackCom configurations and some of its applications.
Readers interested in the fundamentals of BackCom and its applicability may
refer to [9, 10] and the references therein.

Smart Homes
---

Shopping Centre
---

Smart Cities
---

Logistics
---
Healthcare
---
Transportation
---

Agriculture
---

Industry
---
AmBC
---
BiBC
---
MoBC
---
Reader
---
Tag
---
Tag
---
Tag
---
Reader
---
Reader
---
Emitter
---
RF Source
---
User
---
Figure 1: BackCom configurations and applications.

###  II-B Integrated Sensing and Communication

ISAC epitomizes a transformative stride in the realm of wireless technology,
melding the capabilities of communication and sensing into one unified
framework [4, 5, 6, 7]. This interdisciplinary domain does not just enable
devices to communicate; it allows them to actively perceive and interpret
their surroundings. The ISAC aims to provide dual functionalities. While
traditional communication systems have been exclusively about transmitting and
receiving data, ISAC broadens this by using the very same communication
signals to sense the environment. This not only enhances the potential
applications but also leads to an efficient use of resources. By using the
same spectrum and hardware for both communication and sensing, systems can
minimize redundancy and save on critical resources [11, 12].

The essence of ISAC revolves around certain fundamental components. The
transceiver architecture, which is the core of any communication device should
be meticulously designed to seamlessly support both communication and sensing
functionalities without one interfering with the other. Waveforms, the signals
used in communications, need to be tailored for ISAC, ensuring they are
versatile enough to cater to the dual requirements of data transmission and
environmental sensing. To extract meaningful information from these waveforms,
cutting-edge signal processing algorithms are paramount. These algorithms must
be adept at gleaning sensing data from communication signals and vice-versa.

Given its inherent nature, ISAC places a significant emphasis on spatial
information-related sensing parameters [11, 12]. These include the direction
of arrival (DoA) which ascertains from which direction a signal is coming, the
signal propagation time delay which gauges the time taken for a signal to
travel between a transmitter and receiver, and the Doppler frequency which
gives insight into the frequency changes due to relative motion between the
source and the receiver. For applications integrated with radar sensing,
parameters such as the position and velocity of moving objects become
incredibly pertinent. One of ISAC’s key inspirations is its potential
integration with radar functionalities, which are paramount in applications
requiring the identification of spatial parameters of objects [11, 12]. In an
ISAC system, the essence of sensing is derived from the wireless channel
itself, thereby emphasizing spatial parameters. This not only aids in primary
sensing objectives but also sets the stage for intricate tasks such as
imaging, recognition, and classification [11, 12].

The present direction of ISAC indicates a promising future. The formation of
specialized groups such as IEEE 802.11bf, coupled with the dedication of the
wireless local area network (WLAN) sensing topic interest group, highlights
the growing enthusiasm in this field [13]. Additionally, the surge in academic
investigations, from transceiver designs to waveforms, further underscores the
field’s burgeoning momentum.

###  II-C Integrated Sensing and BackCom

To reap the benefits of both ISAC and BackCom, ISABC replaces the traditional
sensing/radar targets in ISAC systems with backscatter tags [8]. Thus,
conventional ISAC and innovative ISABC systems differ in several key aspects.
In particular, the backscatter tags serve as the sensing targets, conveying
additional data to the user/reader while providing sensing information to the
BS. This is essential in applications such as smart homes, warehousing, and
more, where these tags can be used to learn and map the environment. ISABC, on
the other hand, utilizes the unintentionally received backscattered signals at
the BS for sensing purposes, needing no additional RF resources, hardware
cost, or modifications to backscatter tags. Additionally, by utilizing both
sensing and backscatter data, ISABC systems considerably improve communication
and sensing performance; nevertheless, the complexity of the user/reader may
rise due to the requirement for successive interference cancellation
(SIC)-based decoding.

Users
---
Data
---
Sensing
---
(a) FD ISABC BS
---
(b) ISABC Tag
---
Signal
---
Transmitter RF Chain
---
Receiver RF Chain
---
(c) ISABC User/Reader
---
Transmitter Signal
---
Processing
---

Modulation
---

Multiplexing
---

Interleaving
---

Coding
---

Receiver Signal
---
Processing
---

Multi-Tag Det-
---

Range/Velocity
---

Com. Data
---

Removing
---

-ection/Tracking
---

Digital to Analog
---
RF Upconversion
---
Power Amplifier
---
Signal Processing
---
for
---
User Data
---
Signal Processing
---
for
---
Backscatter
---
Data
---
Analog to Digital
---
RF Downconversion
---
Low Noise
---
Amplifier
---
Receiver
---
Controller
---
Micro
---
EH Unit
---
Data Storage
---
Sensing Unit
---
Other
---
Components
---
Modulator
---
RF Down-
---
-conversion
---
Analog to
---
Digital
---
SIC
---
Energy Storage
---
Analog SI
---
Cancellation
---
Antenna SI
---
Cancellation
---
Digital SI
---
Cancellation
---
Figure 2: Circuitry and signal processing of an FD ISABC BS, an ISABC tag, and
an ISABC user/reader.

Building upon these foundational frameworks, we present an advanced concept
termed ISABC. This concept innovatively departs from the standard
sensing/radar targets intrinsic to ISAC systems and introduces the use of a
passive tag. To delineate the contrasts between ISAC and ISABC systems, we
highlight the following critical distinctions:

  * •

Backscatter tag utilization: The ISABC configuration makes a notable pivot by
integrating a backscatter tag. This is in contrast to the ISAC system which
relies on a designated target for its sensing operations. In ISABC, the tag
does not merely act as a sensing target. Instead, it assumes a dynamic role by
actively supplying the BS with detailed sensing data. Simultaneously, this tag
serves as a relay for auxiliary data to end users. For instance, this tag,
which could also be a sensor, is capable of conveying ambient information to
users, enriching them with insights into environmental parameters such as
range, velocity, and orientation.

  * •

Data integration: Another distinguishing hallmark of ISABC systems is their
adeptness at merging sensing and backscatter data. The synergy resulting from
this integration signifies a monumental leap in terms of communication
throughput and sensing precision. While ISAC systems do not provide additional
data to the user, ISABC fills this gap by ensuring users receive auxiliary
data. However, this advancement comes at the cost of heightened computational
requirements on the user’s side. The ISABC system demands advanced decoding
algorithms which are propelled by SIC. Contrarily, the ISAC system employs
more conventional decoding techniques.

  * •

Efficiency and performance metrics: Delving deeper into the metrics of
efficiency, the ISABC system excels in spectral efficiency with a high rating
as compared to the moderate spectral efficiency seen in ISAC. This likely
contributes to ISABC’s potential in channel training and the ensuing
enhancement of spectrum efficiency by curtailing training overhead.
Nevertheless, this heightened spectral efficiency in ISABC is compared with a
medium latency, which is a step up from the low latency characteristic of
ISAC. Furthermore, while both systems exhibit scalability and allocate power
at the BS, ISABC’s energy efficiency is marked low as opposed to the high
energy efficiency seen in ISAC.

In summary, this pioneering concept paves the way for leveraging passive tags
in both environmental monitoring and wireless communication networks [8].
Figure 2 illustrates the schematic design and signal processing flow of the
proposed ISABC system with a full-duplex (FD) base station (BS) in [8].
Notably, it emphasizes the self-interference (SI) cancellation process and
showcases the utilization of backscattered signals for sensing at the BS,
energy harvesting (EH), modulation processing at the tag, and data decoding at
the user/reader with successive interference cancellation (SIC).

##  III Passive Tags/Sensors in ISABC

A tag consists of several components, chiefly an analog front-end. This
integrates an energy harvesting (EH) system and a transceiver crucial for
signal modulation and demodulation. The baseband processing unit manages data
encoding/decoding and interfaces with memory or sensors. Some tags incorporate
environmental sensors, essential for electric grid monitoring, livestock
telemetry, and manufacturing diagnostics. Without traditional batteries, these
tags rely heavily on an EH module to capture energy from ambient RF signals,
while the processor and transceiver are designed for minimal power usage. The
design of these tags is crucial to the ISABC framework, and current cutting-
edge research aims to address design hurdles and boost efficiency.

Table II: An overview of challenges in ISABC Categories |  Challenges |  Implications |  Solutions
---|---|---|---
Environmental Factors and RF Constraints |  • Unknown ambient RF source: Uncertainty in the surrounding RF environment. • Low backscatter signal strength: Challenges in achieving a strong communication signal. • Ambient interference: Issues with external RF noise. • Multi-user interference: Clashes when multiple users transmit simultaneously. • Distance and Path Loss: Signal degradation over distance. • Channel estimation & coding: Predicting and adapting to channel conditions. |  Reduces reliability and efficiency of communication, potential data loss, and misinterpretation. |  Adaptive signal processing, enhanced antenna designs, spectrum sensing for ambient RF sources, interference mitigation algorithms, and robust channel estimation techniques.
Hardware and System Integration Limitations |  • Hardware Constraints: Limitations due to size, power, and cost. • Sensing vs. Communication: Trade-offs between data collection and transmission. • Integration with Legacy Systems: Challenges in compatibility with older systems. |  Affects device compatibility, limits data throughput, and increases energy consumption. |  Modular hardware design, hybrid systems, backward compatibility strategies, power-aware hardware components, and middleware solutions for legacy integration.
Standardization, Security, and Privacy |  • Standardization and Protocol Development: Need for universal standards. • Security and Privacy Concerns: Potential threats to user data and system integrity. |  Can lead to compatibility issues, security breaches, and loss of user trust. |  Collaborative standardization efforts, advanced encryption, two-factor authentication, privacy-preserving algorithms, and security audits.
Scalability and Network Dynamics |  • Rapid expansion of the IoT ecosystem: Increased devices lead to congestion. • Dynamic Network Topologies: Changing device connections and data paths. |  Alters network performance, potential bottlenecks, and degraded user experience. |  Hierarchical network topologies, machine learning (ML) for adaptive scheduling, dynamic routing protocols, edge computing for local data processing, and network slicing for efficient resource allocation.
Energy Efficiency and Sustainability |  • Energy consumption: High power usage in continuous communication. • Dependence on non-renewable energy: Environmental concerns. |  Shortening device lifetime increases operational costs and environmental concerns. |  EH techniques, low-power communication protocols, and renewable energy integrations.

####  III-1 Antenna

BackCom devices operate over a range of frequencies, influenced by
transmission protocols, regulations, and applications [9]. For instance, RFID
systems span from the low-frequency band (125 to 134) to the super high
frequency (SHF) band (2.4 to 5.875). Notably, ultrahigh frequency (UHF) tags
(860 to 960) are favored due to their compact antennas like microstrip,
dipole, and planar designs.

Higher operating frequencies, especially in the SHF band, present distinct
advantages for passive circuits. At these frequencies, antennas, such as the
half-wave dipole, become smaller: 16 at 915, 6 at 2.45, and 2.5 at 5.79.
Consequently, tags can integrate multiple antennas, leveraging advanced
techniques like beamforming and space-time block coding to offset fading and
enhance communication [9, 10].

####  III-2 Backscatter modulation

Modulation serves to map data onto an ambient RF carrier. Unlike active
radios, which generate their own carrier signal, tags modulate by reflecting
external RF signals [10]. This reflection process involves the tag switching
between various impedance values, giving rise to distinct reflection
coefficients [10]. To create a general multi-level ($\tilde{M}$-ary)
modulation scheme, the tag will use $\tilde{M}$ load impedance values. The
reflection coefficient at time $k$ can be defined as

$\displaystyle\Gamma_{m}(k)=\begin{cases}\frac{Z_{m}(k)-Z^{*}_{a}}{Z_{m}(k)+Z_{a}}=\sqrt{\beta}b_{m}(k),\quad&m=1,\ldots,\tilde{M},\\\
0,&m=0,\end{cases}$ (1)

where the first condition, i.e., $m=1,\dots,\tilde{M}$ denotes modulation or
active state at the tag, whereas the second condition, i.e., $m=0$, indicates
the inactive state or impedance matching/EH without reflection at the tag. In
addition, $Z_{m}(k)$ is the impedance value, $Z_{a}$ is the antenna impedance,
$\beta$ denotes the tag’s power reflection factor, and $b_{m}(k)$ is the
modulation symbol. Considering a real-valued antenna impedance $Z_{a}=R_{a}$,
the normalized load impedances can be expressed as

$z_{m}=\frac{1+\Gamma_{m}}{1-\Gamma_{m}}=r_{m}+jx_{m},\quad
m=1,\dots,\tilde{M},$ (2)

where $r_{m}$ and $x_{m}$ respectively represent the normalized load
resistance and reactance.

Modulation strategies span a spectrum, encompassing fundamental methods like
on-off keying (OOK) to advanced techniques such as quadrature amplitude
modulation (QAM) and phase-shift keying (PSK) [9]. The selection of a
modulation technique is typically guided by specific application requirements
and the targeted data rate.

Tag
---
IRS
---
UAV
---
AP
---
FD BS
---
ML
---
MIMO BS
---
Figure 3: Illustration of ISABC configurations, applications, future
directions, and open issues.

####  III-3 Backscatter demodulation

Demodulation involves the reader translating reflected signals from a nearby
emitter back to their original data [9]. An envelope detector classifies the
signal amplitude into ‘high’ or ‘low’ levels to determine the signal’s
presence or absence from the reader. Due to the randomness in detection,
emanating from noise and interference, statistical methods are pivotal. Signal
detection uses hypothesis testing with each hypothesis representing unique
data probability density functions. The quality of detection is influenced by
channel characteristics and the channel state information (CSI). Detection
techniques vary: Coherent detection demands full carrier phase and CSI
knowledge; Non-coherent detection works without this information; Semi-
coherent detection uses limited training symbols; and ML-based detection
employs ML algorithms to adapt and improve over time.

####  III-4 Impedance matching

To ensure efficient energy transfer from incoming signals, a tag adjusts its
impedance to match that of its antenna [9]. Perfect matching means there’s no
reflection and energy is maximally absorbed. If there is a mismatch, the
system can become inefficient. This impedance matching ensures that the
antenna and tag are working optimally together, avoiding loss of signal or
power. When the match is not perfect, additional matching networks are
introduced to improve energy transfer.

####  III-5 Energy harvesting

In the context of tags, energy harvesting (EH) plays a pivotal role in
achieving energy self-sufficiency [9]. Tags employ an EH circuit, primarily
composed of an antenna and a rectifier, to convert incoming RF signals into
direct current (DC). Essential parameters governing the EH process encompass
sensitivity and energy transfer efficiency. For many passive RFID tags, the
activation threshold is set around -20m, representing the minimum power
required to activate the EH circuit. The RF-to-DC conversion efficiency, which
depends significantly on factors like antenna design, impedance matching, and
rectifier efficiency, typically yields real-world power outputs ranging from
approximately $\sim$1 to $\sim$100.

The amount of energy harvested can be modeled as a linear or non-linear
process. The linear EH model offers simplicity and it does not capture energy
harvesters’ nonlinearities due to components such as diodes. At elevated input
powers, a saturation emerges, and if the RF is below sensitivity, the output
becomes null. Thus, nonlinear models are often more representative [9].

##  IV ISABC Applications

ISABC can facilitate every application that conventional BackCom and ISAC
support (Fig. 1). However, their challenges are shown in Table II.

Smart agriculture: Today’s precision farming landscape increasingly demands
real-time data about soil moisture, temperature, and other vital parameters
[14]. By distributing integrated sensing and backscatter devices across vast
farms, farmers gain the ability to glean detailed insights. These insights
come without the need for substantial energy consumption or regular
maintenance, presenting a significant advantage over traditional methods.

Supply chain and logistics: Ensuring that goods move efficiently and safely is
paramount in today’s interconnected global economy. Backscatter devices can be
embedded within packages or containers, providing real-time insights regarding
their location, temperature, humidity, and even tampering. Given their low
energy requirements, these devices can operate for extended durations,
ensuring continuous monitoring throughout long transit times.

Environmental monitoring: Understanding our environment is vital to combating
climate change and ensuring sustainable living. With backscatter devices,
researchers can gather accurate data from dense forests, deep oceans, or high
altitudes. Monitoring parameters like atmospheric carbon dioxide (CO2) levels,
water quality, and wildlife movement becomes seamlessly achievable, providing
invaluable data for ecological studies and conservation efforts.

Industrial automation: The Industry $4.0$ revolution is already underway, with
factories seeking to become smarter. Backscatter devices can be utilized to
monitor machinery health, product quality, and environmental conditions.
Instantaneous feedback allows for real-time adjustments, ensuring optimal
production outcomes.

Health monitoring: The health sector too stands to gain. Wearable devices
equipped with sensors can now constantly monitor vital signs and other crucial
health metrics. Through backscatter, these devices can relay data, ensuring
continuous monitoring. The minimal energy expenditure makes these devices
particularly suitable for long-term patient surveillance.

Urban planning: The vision of smart cities requires a deep understanding of
elements such as traffic flows, environmental metrics, and infrastructure
utilization. With devices that sense and communicate using backscatter
technology, city planners can embed these into urban infrastructures. The
longevity of these devices means they can operate for years without
necessitating battery replacements, thus providing real-time, uninterrupted
data for urban planning and development.

##  V Future Directions and Open Issues

The convergence of sensing and BackCom is only the tip of the iceberg. As we
delve deeper into this century, the merging of various disciplines promises
innovations that can redefine our technological landscape. For ISABC systems,
several exciting trajectories loom on the horizon (Fig. 3).

Towards more energy-efficient designs: As miniaturization progresses and the
imperative for energy conservation intensifies, we will likely see an influx
of designs accentuating ultra-energy efficiency. Ongoing research aims to
unveil nano-scale components and circuits with consumption levels in the pico-
or femto-watt range. Combined with innovative EH techniques, such as
triboelectric nanogenerators or advanced photovoltaic materials, we might be
nearing the advent of devices that can operate perpetually, drawing all
requisite energy from their surroundings.

Massive MIMO and cell-free massive MIMO: Massive multiple-input multiple-
output (MIMO), through large antenna arrays, offers spatial degrees of freedom
(DoF) for multiplexing and diversity gains. In contrast, cell-free
architecture enhances coverage probability and harnesses macro-diversity
against large-scale fading. ISABC integration with these techniques must take
advantage of the favorable properties of massive MIMO and cell-free massive
MIMO architectures to boost joint communication and sensing capabilities.

FD ISABC: By leveraging full-duplex (FD) capabilities, our versatile
transmitter can simultaneously delve into EH and initiate active wireless
broadcasting. This necessitates the optimal regulation of the hybrid
transmitter, particularly given the inherent benefits of full-duplex
operations. Future works include refining the optimal power allocation at the
BS and considering interference mitigation strategies.

UAVs-aided ISABC: As unmanned aerial vehicles (UAVs) permeate sectors from
logistics to agriculture, integrating with ISAC offers transformative
potential. ISAC’s inclusion reduces the UAV’s communication and sensing load,
lightening its weight and boosting agility. This integration delivers enhanced
performance, surpassing traditional UAV functions.

IRSs-aided ISABC: The melding of ISABC with intelligent reflecting surfaces
(IRSs), which modulates the wireless environment itself, opens up a plethora
of opportunities. IRS can be strategically positioned to improve the
efficiency of EH, focusing ambient RF signals onto ISABC-equipped devices.
Moreover, in active communication scenarios, the IRS can help in sculpting the
wireless channels, leading to optimized data transmission and potentially
reducing interference

Multiple access techniques: The integration of advanced multiple access
techniques like non-orthogonal multiple access (NOMA) or sparse code multiple
access (SCMA) with ISABC can significantly augment the simultaneous
communication capabilities of these systems. These techniques enable devices
to share spectral resources, vital for ISABC systems prioritizing energy
efficiency and spectrum optimization.

Machine learning in ISABC: The escalating prowess of ML algorithms signifies
the monumental potential for integrated systems. In-device processing can pave
the way for smarter data acquisition; sensors might only forward data upon
anomaly detection, conserving energy. Beyond that, ML stands poised to refine
communication protocols, discerning the ambient RF milieu, ensuring ideal
modulation schemes, and curbing interference. This can culminate in
comprehensive smart ecosystems that dynamically adapt and respond to varying
external factors.

Network configurations: The potential of hybrid transmitters could be notably
amplified within a network framework, incorporating multiple tags and a myriad
of licensed channels. This structure promises abundant signal resources for
energy absorption and an expanded array of spectrum vacancies for concurrent
active data communications. Yet, this multifaceted environment amplifies
coordination intricacies among the hybrid transmitters, introducing a nuanced
equilibrium among EH, backscattering, active broadcasting, and channel
selection.

FD BS
---

$\mathbf{H}_{\rm{SI}}$
---

$\mathbf{f}(d_{f}=15\,\text{m})$
---

$(d_{g_{f}}=d_{g_{b}}=6\,\text{m})$
---

$\mathbf{g}_{f}$
---

$v(d_{v}=10\,\text{m})$
---

Tag
---

User
---

$\mathbf{g}_{b}$
---

Figure 4: An ISABC system setup.

##  VI Numerical Results

This section presents simulations of both communications (i.e., direct and
backscatter) and sensing performance to illustrate the feasibility and
performance gains of ISABC. Interested readers can refer to [8] for the
technical details. We model large-scale fading using the third-generation
partnership project (3GPP) urban microcell (UMi) model and the additive white
Gaussian noise (AWGN). We consider the conventional ISAC as a comparative
benchmark, i.e., the tag is replaced by a conventional radar target with no
reflection loss.

Figure 5: Communication rate versus the number of transmit and receive antenna
at the BS ($M=N$) for $\rho=\\{0.4,0.6,0.8\\}$.

Fig. 5 represents the communication rates of both the user and the tag against
the number of BS antennas, $M=N$. It is depicted under a BS transmit power of
20m, and for $\rho=\\{0.4,0.6,0.8\\}$, where $\rho$ denotes the power
allocation factor of the BS between communication and sensing. The tag rate is
independent of $\rho$, while a high $\rho$ contributes to a higher user rate.
For instance, with $M=N=20$, a user rate increases by 6.65 when $\rho$ changes
from $0.4$ to $0.6$. However, due to the absence of tag interference, the ISAC
benchmark user rate is always higher than the proposed ISABC user rate.

Fig. 6 depicts the BS’s sensing rate versus the number of BS antennas. As
expected, increasing the number of BS antennas improves the sensing rate of
the BS. Furthermore, low $\rho$ values result in a high sensing rate due to
the high power allocation for the sensing waveform. For example, $\rho=0.4$
achieves a 36.16 sensing rate gain compared to that of $\rho=0.8$ due to the
allocated high power for sensing with $M=N=20$. Moreover, the ISAC system
achieves higher sensing rates than the ISABC system since there is no
reflection loss at the target. Nevertheless, the proposed system outperforms
ISAC. For instance, the ISABC system delivers a 6.7 sum rate gain with
$M=N=30$ and $\rho=0.6$.

##  VII Conclusion

The rapidly expanding digital landscape is anchored by the IoT, facilitated
majorly by mMTC. However, mMTC faces challenges like energy constraints and
effective communication. AmBC offers solutions through its energy efficiency
and cost-effectiveness. The merger of ISAC with backscatter evolves into
ISABC, enhancing sensory accuracy and communication depth but also adding
decoding complexities. The importance of passive tags in ISABC is emphasized,
especially in terms of antenna design and EH. In conclusion, while IoT, AmBC,
and ISABC herald a future of seamless communication and sensing, hurdles still
exist. Future research could improve decoding in ISABC systems and refine EH
methods, aiming for an efficient and interconnected future.

Figure 6: Sensing rate versus the number of transmit and receive antenna at
the BS ($M=N$) for $\rho=\\{0.4,0.6,0.8\\}$.

## References

  * [1] “3GPP TSG RAN $-$97e3, Study on ambient IoT , 9.1 (from RP-222685),” Sept. 2022. [Online]. Available: https://portal.3gpp.org/ngppapp/TdocList.aspx?meetingId=60043
  * [2] “3GPP TR 38.913 - 5G; study on scenarios and requirements for next generation access technologies, V14.3.0 Rel. 14,” Aug. 2017. [Online]. Available: https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetail s.aspx?specificationId=2996
  * [3] D. T. Hoang, D. Niyato, D. I. Kim, N. V. Huynh, and S. Gong, _Ambient Backscatter Communication Networks_. Cambridge University Press, 2020.
  * [4] A. Behravan _et al._ , “Positioning and sensing in 6G: Gaps, challenges, and opportunities,” _IEEE Veh. Technol. Mag._ , vol. 18, no. 1, pp. 40–48, Mar. 2023.
  * [5] A. Zhang _et al._ , “Perceptive mobile networks: Cellular networks with radio vision via joint communication and radar sensing,” _IEEE Veh. Technol. Mag._ , vol. 16, no. 2, pp. 20–30, Jun. 2021.
  * [6] J. Wang _et al._ , “Integrated sensing and communication: Enabling techniques, applications, tools and data sets, standardization, and future directions,” _IEEE Internet Things J._ , vol. 9, no. 23, pp. 23 416–23 440, Dec. 2022.
  * [7] X. Tong, Z. Zhang, and Z. Yang, “Multi-view sensing for wireless communications: Architectures, designs, and opportunities,” _IEEE Commun. Mag._ , vol. 61, no. 5, pp. 40–46, May 2023.
  * [8] D. Galappaththige, C. Tellambura, and A. Maaref, “Integrated sensing and backscatter communication,” _IEEE Wireless Commun. Lett._ , pp. 1–1, 2023\.
  * [9] D. Galappaththige, F. Rezaei, C. Tellambura, and S. Herath, “Link budget analysis for backscatter-based passive IoT,” _IEEE Access_ , vol. 10, pp. 128 890–128 922, Dec. 2022.
  * [10] F. Rezaei, D. Galappaththige, C. Tellambura, and S. Herath, “Coding techniques for backscatter communications - A contemporary survey,” _IEEE Commun. Surveys Tuts._ , pp. 1020–1058, 2th Quart. 2023.
  * [11] Z. Wei, F. Liu, C. Masouros, N. Su, and A. P. Petropulu, “Toward multi-functional 6G wireless networks: Integrating sensing, communication, and security,” _IEEE Commun. Mag._ , vol. 60, no. 4, pp. 65–71, Apr. 2022\.
  * [12] C. Ouyang, Y. Liu, H. Yang, and N. Al-Dhahir, “Integrated sensing and communications: A mutual information-based framework,” _IEEE Commun. Mag._ , vol. 61, no. 5, pp. 26–32, May 2023.
  * [13] F. Meneghello, C. Chen, C. Cordeiro, and F. Restuccia, “Toward integrated sensing and communications in IEEE 802.11bf Wi-Fi networks,” _IEEE Commun. Mag._ , vol. 61, no. 7, pp. 128–133, Jul. 2023.
  * [14] S.-N. Daskalakis, J. Kimionis, A. Collado, M. M. Tentzeris, and A. Georgiadis, “Ambient FM backscattering for smart agricultural monitoring,” in _IEEE MTT-S Int. Microw. Symp. (IMS)_ , Jun. 2017, pp. 1339–1341.


# Effective In-Context Example Selection through Data Compression

Zhongxiang Sun$\star$ Kepu Zhang$\star$ Haoyu Wang  Xiao Zhang  Jun
Xu$\dagger$
Gaoling School of Artificial Intelligence,
Renmin University of China
{sunzhongxiang, kepuzhang, wanghaoyu0924, zhangx89<EMAIL_ADDRESS>
###### Abstract

In-context learning has been extensively validated in large language models.
However, the mechanism and selection strategy for in-context example
selection, which is a crucial ingredient in this approach, lacks systematic
and in-depth research. In this paper, we propose a data compression approach
to the selection of in-context examples. We introduce a two-stage method that
can effectively choose relevant examples and retain sufficient information
about the training dataset within the in-context examples. Our method shows a
significant improvement of an average of 5.90% across five different real-
world datasets using four language models.

Effective In-Context Example Selection through Data Compression

Zhongxiang Sun$\star$ and Kepu Zhang$\star$ and Haoyu Wang and Xiao Zhang and
Jun Xu$\dagger$ Gaoling School of Artificial Intelligence, Renmin University
of China {sunzhongxiang, kepuzhang, wanghaoyu0924, zhangx89<EMAIL_ADDRESS>

††footnotetext: ⋆ Equal contribution. ††footnotetext: † Jun Xu is the
corresponding author.

##  1 Introduction

Drawing inspiration from recent research that regards Large Language Models
(LLMs) as an efficient means of compressing pre-training datasets, and the
notion that In-Context Learning (ICL) can be seen as fine-tuning on example
datasets (Dai et al., 2022), we assume that LLMs can achieve data re-
compression through ICL. In other words, an effective training dataset
compression method can aid in the selection of in-context examples. Looking at
the matter from another perspective, it is evident that fine-tuning the entire
dataset would yield the best results. However, in the case of ICL, we
typically choose only a few examples as LLM prompts due to the limitations of
input window length. By employing data compression techniques, we can ensure
that the majority of data information is preserved in the in-context examples,
which is also the aim of dataset pruning.

Based on the aforementioned analysis, we propose to utilize the influence
function Koh and Liang (2017), which has exhibited efficacy in dataset
pruning, to select examples for ICL. However, recent studies on ICL have
revealed that the relevance between the query source and in-context examples
is critical for ICL. Furthermore, the influence function requires the gradient
of parameters, which is computationally expensive and inefficient. To tackle
the aforementioned issues, we suggest a two-stage method. Firstly, relevant
examples for the query input are recalled, which ensures the correlation
between the examples and the query source. Secondly, our meta-gradient-based
influence function is utilized to calculate the influence score for each
recalled example. Finally, based on the influence score, in-context examples
are selected from the recalled examples. Notably, our framework compresses
important information from the training set into the in-context examples,
thereby enhancing the performance of ICL. Additionally, our framework is data-
independent, relies solely on a small number of model parameters, and does not
require the training of any additional models. Numerous experiments indicate
that our method shows a significant improvement of an average of 5.90% on five
different real-world datasets using multiple language models.

##  2 Background

###  2.1 In-Context Learning

The ICL scenario of LLMs can be regarded as a conditional text generation
problem. Concretely, the probability of generating a target $y$ is conditioned
on the context $C$, which includes $k$ examples and the source $x$. Therefore,
the probability can be expressed as:

$p_{\mathrm{LLM}}(y\mid C,x)=\prod_{t=1}^{T}p\left(y_{t}\mid
C,x,y_{<t}\right)$

where LLM denotes the parameters of the large language model, and
$C=\left\\{x_{1},y_{1},x_{2},y_{2},\ldots,x_{k},y_{k}\right\\}$ is a context
string concatenating $k$ training instances. For example, $(x_{k},y_{k})$ is
concatenated with the special character, e.g., “`\n`” or “Sentence: $x$;
Sentiment $y$.” which is denoted as $p_{k}$. In this paper, we have different
example sets $C$ at different stages, $C_{1}$ in the first stage, and $C_{2}$
in the second stage, where $C_{2}$ is a subset of $C_{1}$.

Dai et al. (2022) explains language models as meta-optimizers and understands
ICL as a kind of implicit finetuning:

$\displaystyle\widetilde{\mathcal{F}}_{\text{ICL}}(\mathbf{q})$
$\displaystyle=W_{\text{ZSL}}\mathbf{q}+\sum_{i}\left(W_{V}\mathbf{x}^{\prime}_{i}\otimes\left(W_{K}\mathbf{x}^{\prime}_{i}\right)^{T}\right)\mathbf{q}$
(1) $\displaystyle=$ $\displaystyle W_{\text{ZSL}}\mathbf{q}+\Delta
W_{\text{ICL}}\mathbf{q}$ $\displaystyle=$
$\displaystyle\left(W_{\text{ZSL}}+\Delta W_{\text{ICL}}\right)\mathbf{q},$

where ZSL denotes the zero-shot learning, which only contains the source $x$;
$\mathbf{x}\in\mathbb{R}^{d}$ is the input representation of a query token
$t$, and $\mathbf{q}=W_{Q}\mathbf{x}\in\mathbb{R}^{d^{\prime}}$ is the
attention query vector; $\mathbf{x}^{\prime}\in\mathbb{R}^{d}$ denotes the
input representations of the example’s token;
$W_{Q},W_{K},W_{V}\in\mathbb{R}^{d^{\prime}\times d}$ are the projection
matrices for computing the attention queries, keys, and values, respectively.
Dai et al. (2022) regards $W_{V}X^{\prime}$ as some meta-gradients, which are
used to compute the updated matrix $\Delta W_{\text{ICL}}$.

###  2.2 Dataset Pruning

Investigating the data redundant problem not only helps to improve the
training efficiency but also helps us understand the representation ability of
small data and how many training samples are required and sufficient for a
learning system. (Yang et al., 2022) proposed to use the Influence Function to
accurately and fast estimate the parameter change caused by weighting an
example $p$ for the training dataset. The influence of weighting $p$ on the
parameters is given by:

$\mathcal{I}_{\mathrm{param}}(p)=\frac{\mathrm{d}\hat{\theta}_{\delta,p}}{\mathrm{d}\delta}\bigg{|}_{\delta=0}=-H_{\hat{\theta}}^{-1}\nabla_{\theta}L(p,\hat{\theta}),$
(2)

where
$H_{\hat{\theta}}=\frac{1}{n}\sum_{p_{i}\in\mathcal{D}}\nabla^{2}_{\theta}L(p_{i},\hat{\theta})$
is the Hessian and positive definite by assumption,
$\mathcal{I}_{\mathrm{param}}(p)\in\mathbb{R}^{N}$, $N$ is the number of
network parameters, $\mathcal{D}$ is the original dataset. After getting the
weighting of each example $p$, (Yang et al., 2022) propose generalization-
guaranteed pruning or cardinality-guaranteed pruning to get the final
compressed dataset $\mathcal{\hat{D}}$.

##  3 Method

###  3.1 Recall

Given the training dataset $\mathcal{D}$ and the query source $x$, we use BM25
Robertson et al. (2009) to retrieve a set of relevant examples $C_{1}$ for
$x$. For each example $p_{j}$ in $\mathcal{D}$, the BM25 score, denoted as
$R(p_{j},x)$, is computed. This score reflects the relevance of example
$p_{j}$ to the query $x$. Specifically:

$R_{j}=\text{BM25}(p_{j},x),$ (3)

where $R_{j}$ is the relevance score of example $p_{j}$ with respect to query
$x$.

Subsequently, we form the set $C_{1}$ which consists of the top-N examples
with the highest relevance scores:

$C_{1}=\\{p_{j}|j=1,2,\ldots,N\\},$ (4)

where $N$ is the number of examples we wish to recall for the given query $x$.

###  3.2 Influence-Awared Rerank

For each $p$ in $C_{1}$, we calculate the input representation of tokens $p$
as $P$ and the meta-gradient $G_{p}=W_{V}P$. To compute
$\mathcal{I}_{\mathrm{param}}(p)$ in Eq. (2), we require the Hessian of $p$
for the parameter $W_{V}$, which necessitates the computation of second-order
derivatives. However, we only have access to first-order derivatives
approximations of the parameters. Considering that LLMs typically employ
cross-entropy loss and maximum likelihood estimation (MLE) for fine-tuning, we
have opted to employ the Fisher matrix as an approximation of the Hessian
Barshan et al. (2020). The key to the approximation process is as follows:

$\nabla^{2}f(\mathbf{x})\approx\nabla f(\mathbf{x})\nabla
f(\mathbf{x})^{\top}$ (5)

Then, combining the Eq. (2) with Eq. (5), the expression of the influence
function for $p$ is:

$\mathcal{I}_{\mathrm{param}}(p)=-\hat{H}_{\hat{\theta}}^{-1}G_{p},$ (6)

where
$H_{\hat{\theta}}=\frac{1}{n}\sum_{p_{i}\in\mathcal{D}}G_{p}G_{p}^{\top}$.

The score of $C_{1}$ is determined by a combination of the influence score and
the relevance score, represented as:

$\mathcal{S}=\left\\{\lVert\mathcal{I}_{\mathrm{param}}(p_{1})\rVert_{F}^{2}+R_{1},\ldots,\lVert\mathcal{I}_{\mathrm{param}}(p_{N})\rVert_{F}^{2}+R_{N}\right\\}.$

Finally, the $K$ in-context learning examples in $C_{2}$ are chosen by:

$C_{2}=\left\\{p_{i}|i\in
I\right\\},\text{where}~{}I=\text{arg}\max_{\begin{subarray}{c}I\subseteq\\{1,2,\ldots,|{\mathcal{S}}|\\}\\\
|I|=K\end{subarray}}\mathcal{S}.$

$K=3$ |  | GPT2-XL | GPT2-Large | GPT2-Small | GPT2-Medium
---|---|---|---|---|---
|  | Acc (%) |  $F_{1}$ (%) | Acc (%) |  $F_{1}$ (%) | Acc (%) |  $F_{1}$ (%) | Acc (%) |  $F_{1}$ (%)
Sick | BM25 | 42.63 | 33.01 | 27.68 | 26.72 | 31.72 | 23.34 | 31.52 | 26.45
| Ours | 47.07 | 35.28 | 31.11 | 28.51 | 35.35 | 26.25 | 32.53 | 27.16
Cola | BM25 | 61.84 | 54.83 | 63.09 | 50.24 | 65.96 | 48.79 | 60.98 | 50.04
| Ours | 63.09 | 55.53 | 64.24 | 50.74 | 65.87 | 48.39 | 63.95 | 52.80
Ethos-disability | BM25 | 77.01 | 57.44 | 82.76 | 62.42 | 68.97 | 56.92 | 74.71 | 50.26
| Ours | 83.91 | 66.17 | 87.36 | 64.14 | 74.71 | 62.96 | 77.01 | 51.67
Tweet_eval_stance_feminist | BM25 | 50.75 | 46.19 | 44.78 | 40.96 | 41.79 | 31.64 | 44.78 | 41.33
| Ours | 50.75 | 43.27 | 46.27 | 41.88 | 43.28 | 32.01 | 46.27 | 38.24
Tweet_eval_stance_hillary | BM25 | 49.28 | 40.63 | 42.03 | 41.12 | 42.03 | 41.12 | 46.38 | 44.95
| Ours | 53.62 | 40.68 | 53.62 | 51.45 | 46.38 | 39.24 | 50.72 | 44.73
All dataset Avg | BM25 | 56.30 | 46.42 | 52.07 | 44.29 | 50.09 | 40.36 | 51.67 | 42.61
| Ours | 59.69 | 48.18 | 56.52 | 47.34 | 53.12 | 41.77 | 54.10 | 42.92

Table 1: Results of four ICL examples. The boldface represents the best
performance.

$K=4$ |  | GPT2-XL | GPT2-Large | GPT2-Small | GPT2-Medium
---|---|---|---|---|---
|  | Acc (%) |  $F_{1}$ (%) | Acc (%) |  $F_{1}$ (%) | Acc (%) |  $F_{1}$ (%) | Acc (%) |  $F_{1}$ (%)
Sick | BM25 | 42.83 | 35.60 | 31.31 | 31.09 | 33.54 | 26.03 | 31.92 | 28.07
| Ours | 46.67 | 36.70 | 32.53 | 30.74 | 39.19 | 30.42 | 35.56 | 29.80
Cola | BM25 | 60.98 | 54.48 | 62.80 | 51.17 | 65.10 | 48.26 | 60.21 | 49.85
| Ours | 61.36 | 54.38 | 63.47 | 51.51 | 67.31 | 48.71 | 64.91 | 54.05
Ethos-disability | BM25 | 81.61 | 61.33 | 85.06 | 61.60 | 68.97 | 56.92 | 77.01 | 54.78
| Ours | 85.06 | 64.80 | 87.36 | 59.87 | 72.41 | 59.60 | 79.31 | 56.50
Tweet_eval_stance_feminist | BM25 | 46.27 | 43.54 | 43.28 | 36.81 | 43.28 | 40.64 | 38.81 | 31.02
| Ours | 53.73 | 47.36 | 47.76 | 44.47 | 46.27 | 35.10 | 44.78 | 37.11
Tweet_eval_stance_hillary | BM25 | 47.83 | 40.64 | 34.78 | 34.36 | 39.13 | 33.92 | 40.58 | 39.71
| Ours | 47.83 | 38.89 | 46.38 | 46.01 | 46.38 | 34.81 | 47.83 | 44.07
All dataset Avg | BM25 | 55.90 | 47.12 | 51.45 | 43.01 | 50.00 | 41.16 | 49.71 | 40.69
| Ours | 58.93 | 48.42 | 55.50 | 46.52 | 54.31 | 41.73 | 54.48 | 44.30

Table 2: Results of four ICL examples. The boldface represents the best
performance.

##  4 Experiments

In this section, we empirically verify the efficiency of our approach. The
source code and all experiments have been shared at
https://anonymous.4open.science/r/ICL-F302.

###  4.1 Experiments setup

This section introduces the detailed information about our experiments.

Models. We use the open source GPT2 model family Radford et al. (2019) (i.e.,
GPT2-Small, GPT2-Medium, GPT2-Large, GPT2-XL) as a representative of large
models to verify the effectiveness of our method.

Datasets. We use five datasets spanning four tasks: linguistic analysis, hate
speech detection, tweet classification, and semantic similarity. Specifically,
we employ Linguistic Acceptability dataset (Cola) Warstadt et al. (2019),
online hate speech detection dataset (Ethos and Ethos-disability) Mollas et
al. (2022), Tweet_eval-stance_feminist and Tweet_eval_stance_hillary Barbieri
et al. (2020) from Twitter, and Sentences Involving Compositional Knowledge
dataset (Sick) Marelli et al. (2014). We use Accuracy and $F_{1}$ score as
evaluation metrics. Detailed dataset statistics and the prompt templates used
can be found in Appendix A.2 and Appendix A.1.

Implementation Details. In the study, we chose $K$ = 3 and $K$ = 4
demonstrations to contrast example selection methods from training data. We
set $N$ = 100 for all models. Sentences were either truncated or supplemented
to have a uniform length at 50% of the average sentence length. Although using
multiple transformer layers’ meta-gradient might be beneficial, considering
the time efficiency, we used the first layer and obtained higher accuracy than
baseline models.

Baselines. Considering the model proposed in this paper is unsupervised and
requires no training, it possesses a higher generalizability and operational
efficiency compared to models that undergo supervised training. To ensure a
fair comparison, our primary baseline is the unsupervised BM25-based In-
Context Example Selection. Previous work Wang et al. (2023); Gupta et al.
(2023) has demonstrated that BM25 constitutes a robust baseline for
demonstration selection, hence we juxtapose our methodology against BM25. The
demonstrations selected by the BM25 are utilized across all GPT2 models.

###  4.2 Overall Performance

Tables 1 and 2 display results for three and four ICL examples, respectively.
Observing the last two rows, our method consistently outperforms across all
models and datasets. Using three and four examples, we surpass BM25 by
averages of 5.17% and 6.64% in all metrics. Specifically, accuracy sees
improvements of 6.33% and 7.80% over BM25. This underscores our approach’s
superiority. We found some higher model performance with three ICL examples
compared to four, which can be explained by overfitting and example quality.
Overfitting in few-shot learning means too many examples leads to adaptation
to specific instances rather than general patterns, reducing accuracy on
unseen data. Furthermore, if the additional fourth example is of lower quality
or relevance, it can degrade model performance.

##  5 Related Work

###  5.1 In-context Learning

In-context learning (ICL) has emerged as a fresh approach in natural language
processing (NLP), where large models predict based solely on contexts
supplemented by several examples Dong et al. (2022); Shin et al. (2022); Zhang
et al. (2023); Bai et al. (2023). Numerous studies have sought to modify,
improve, and comprehend ICL, encompassing topics like prompt tuning Kim et al.
(2022); Wang et al. (2022a); Mishra et al. (2022), intrinsic mechanism
analysis Chan et al. (2022); Li et al. (2023); Garg et al. (2022), evaluations
Srivastava et al. (2023); Wang et al. (2022b), and its use across various
fields Sun (2023), among others.

###  5.2 Demonstration Selection

The goal of demonstration selection is to identify optimal examples for ICL.
(Liu et al., 2022) demonstrated that choosing the nearest neighbors as in-
context examples is an effective approach. The used distance measures include
the pre-set L2 distance or the cosine similarity based on sentence embeddings.
They introduced KATE, an unsupervised kNN retriever for in-context example
selection. (Rubin et al., 2022) suggested a two-phase retrieval process for
demonstration selection. For a given input, it initially employs an
unsupervised retriever (like BM25) to retrieve similar candidate examples and
then uses a supervised retriever, EPR, to pick demonstrations from these
candidates. Recent studies indicate that LLMs exhibit strong sensitivity to
the examples chosen, resulting in significant performance variations Nie et
al. (2022), dependency on example sequence Lu et al. (2022), and at times, an
insensitivity to the actual labels Min et al. (2022). Our research focuses on
reducing training overhead and condensing crucial data from the training set
into in-context examples, which in turn amplifies the ICL’s effectiveness.

##  6 Conclusion

In summary, inspired by LLMs and ICL’s potential, we devised a two-stage
method using the influence function for optimal in-context example selection.
Our approach ensures relevance with the query source and efficiently
determines influence scores. The result is an enhancement in ICL performance,
with our experiments validating our model’s effectiveness. Our framework
stands out due to its data-independent nature and minimal reliance on model
parameters.

##  7 Limitation and Future Work

Given resource constraints and page limitations, we provide limited validation
in this paper. Our model is a model-agnostic and free-training approach that
can be applied to various in-context learning selection models. In the future,
we will validate the effectiveness of our model on more large-scale language
models, baselines, and datasets.

## References

  * Bai et al. (2023) Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. 2023.  Transformers as statisticians: Provable in-context learning with in-context algorithm selection.  _arXiv preprint arXiv:2306.04637_.
  * Barbieri et al. (2020) Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020.  Tweeteval: Unified benchmark and comparative evaluation for tweet classification.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1644–1650.
  * Barshan et al. (2020) Elnaz Barshan, Marc-Etienne Brunet, and Gintare Karolina Dziugaite. 2020.  Relatif: Identifying explanatory training samples via relative influence.  In _International Conference on Artificial Intelligence and Statistics_ , pages 1899–1909. PMLR.
  * Chan et al. (2022) Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. 2022.  Data distributional properties drive emergent in-context learning in transformers.  _Advances in Neural Information Processing Systems_ , 35:18878–18891.
  * Dai et al. (2022) Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. 2022.  Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers.  _arXiv preprint arXiv:2212.10559_.
  * Dong et al. (2022) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022.  A survey for in-context learning.  _arXiv preprint arXiv:2301.00234_.
  * Garg et al. (2022) Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. 2022.  What can transformers learn in-context? a case study of simple function classes.  _Advances in Neural Information Processing Systems_ , 35:30583–30598.
  * Gupta et al. (2023) Shivanshu Gupta, Sameer Singh, and Matt Gardner. 2023.  Coverage-based example selection for in-context learning.  _arXiv preprint arXiv:2305.14907_.
  * Kim et al. (2022) Hyuhng Joon Kim, Hyunsoo Cho, Junyeob Kim, Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. 2022.  Self-generated in-context learning: Leveraging auto-regressive language models as a demonstration generator.  _arXiv preprint arXiv:2206.08082_.
  * Koh and Liang (2017) Pang Wei Koh and Percy Liang. 2017.  Understanding black-box predictions via influence functions.  In _ICML_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 1885–1894. PMLR.
  * Li et al. (2023) Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. 2023.  Transformers as algorithms: Generalization and stability in in-context learning.  In _ICML_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 19565–19594. PMLR.
  * Liu et al. (2022) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022.  What makes good in-context examples for GPT-3? In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_ , pages 100–114, Dublin, Ireland and Online. Association for Computational Linguistics.
  * Lu et al. (2022) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8086–8098, Dublin, Ireland. Association for Computational Linguistics.
  * Marelli et al. (2014) Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014.  A SICK cure for the evaluation of compositional distributional semantic models.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 216–223, Reykjavik, Iceland. European Language Resources Association (ELRA).
  * Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022.  Rethinking the role of demonstrations: What makes in-context learning work? In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11048–11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Mishra et al. (2022) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022.  Cross-task generalization via natural language crowdsourcing instructions.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3470–3487.
  * Mollas et al. (2022) Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas. 2022.  Ethos: a multi-label hate speech detection dataset.  _Complex & Intelligent Systems_, 8(6):4663–4678.
  * Nie et al. (2022) Feng Nie, Meixi Chen, Zhirui Zhang, and Xu Cheng. 2022.  Improving few-shot performance of language models via nearest neighbor calibration.  _arXiv preprint arXiv:2212.02216_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009.  The probabilistic relevance framework: Bm25 and beyond.  _Foundations and Trends® in Information Retrieval_ , 3(4):333–389.
  * Rubin et al. (2022) Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022.  Learning to retrieve prompts for in-context learning.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2655–2671, Seattle, United States. Association for Computational Linguistics.
  * Shin et al. (2022) Seongjin Shin, Sang-Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Jung-Woo Ha, et al. 2022.  On the effect of pretraining corpora on in-context learning by a large-scale language model.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5168–5186.
  * Srivastava et al. (2023) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2023.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _Transactions on Machine Learning Research_.
  * Sun (2023) Zhongxiang Sun. 2023.  A short survey of viewing large language models in legal aspect.  _arXiv preprint arXiv:2303.09136_.
  * Wang et al. (2022a) Boshi Wang, Xiang Deng, and Huan Sun. 2022a.  Iteratively prompt pre-trained language models for chain of thought.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 2714–2730.
  * Wang et al. (2023) Liang Wang, Nan Yang, and Furu Wei. 2023.  Learning to retrieve in-context examples for large language models.  _arXiv preprint arXiv:2307.07164_.
  * Wang et al. (2022b) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022b.  Super-naturalinstructions: Generalization via declarative instructions on 1600+ NLP tasks.  In _EMNLP_ , pages 5085–5109. Association for Computational Linguistics.
  * Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019.  Neural network acceptability judgments.  _Transactions of the Association for Computational Linguistics_ , 7:625–641.
  * Yang et al. (2022) Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. 2022.  Dataset pruning: Reducing training data by examining generalization influence.  _arXiv preprint arXiv:2205.09329_.
  * Yoo et al. (2022) Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang-Woo Lee, Sang-goo Lee, and Taeuk Kim. 2022.  Ground-truth labels matter: A deeper look into input-label demonstrations.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 2422–2437.
  * Zhang et al. (2023) Ruiqi Zhang, Spencer Frei, and Peter L Bartlett. 2023.  Trained transformers learn linear models in-context.  _arXiv preprint arXiv:2306.09927_.

##  Appendix A APPENDIX FOR REPRODUCIBILITY

###  A.1 Dataset

In our experiments, we employ Linguistic Acceptability dataset (Cola) Warstadt
et al. (2019), online hate speech detection dataset (Ethos and Ethos-
disability) Mollas et al. (2022), Tweet_eval-stance_feminist and
Tweet_eval_stance_hillary Barbieri et al. (2020) from Twitter, and Sentences
Involving Compositional Knowledge dataset (Sick) Marelli et al. (2014). Table
3 shows more detailed statics of the datasets.

Dataset | Training Size | Valid Size | Label class
---|---|---|---
Sick | 4439 | 495 | 3
Cola | 8551 | 1043 | 2
Ethos-disability | 346 | 87 | 2
Tweet_eval-stance_feminist | 597 | 67 | 3
Tweet_eval-stance_hillaryt | 620 | 69 | 3

Table 3: Dataset statistics.

###  A.2 Templates.

Based on (Min et al., 2022; Yoo et al., 2022), we employ minimal templates.
For the GPT-2 series model, we distinguish between input and label using a
space, and this is also applied between demonstration examples. Table 4
presents templates from five datasets, where blue represents the fixed
section, and black varies depending on the example content.

Dataset | example | label
---|---|---
Sick | [S1]  The question is:[S2]  True or False? Answer:[Label] | True,Not sure,False
Cola |  Review: [S1]  Acceptability:[Label] | unacceptable,acceptable
Ethos-disability |  There is a hate based on disability in [S1] True or False? Answer: [Label] | False,True
Tweet_eval-stance_feminist |  Tweet: [S1] Sentiment: [Label] | none,against,favor
Tweet_eval-stance_hillary |  Tweet: [S1] Sentiment: [Label] | none,against,favor

Table 4: Template examples.