# Multi-domain Cooperative SLAM: The Enabler for Integrated Sensing and
Communications

Jie Yang, Chao-Kai Wen, Xi Yang, Jing Xu, Tao Du, and Shi Jin Jie Yang is with
the School of Automation and the Frontiers Science Center for Mobile
Information Communication and Security, Southeast University, Nanjing, China.
Shi Jin (corresponding author) is with the National Mobile Communications
Research Laboratory and the Frontiers Science Center for Mobile Information
Communication and Security, Southeast University, Nanjing, China. Xi Yang,
Jing Xu, and Tao Du are with the National Mobile Communications Research
Laboratory, Southeast University, Nanjing, China. Chao-Kai Wen is with the
Institute of Communications Engineering, National Sun Yat-sen University,
Kaohsiung, Taiwan.

###### Abstract

Simultaneous localization and mapping (SLAM) provides user tracking and
environmental mapping capabilities, enabling communication systems to gain
situational awareness. Advanced communication networks with ultra-wideband,
multiple antennas, and a large number of connections present opportunities for
deep integration of sensing and communications. First, the development of
integrated sensing and communications (ISAC) is reviewed in this study, and
the differences between ISAC and traditional communication are revealed. Then,
efficient mechanisms for multi-domain collaborative SLAM are presented. In
particular, research opportunities and challenges for cross-sensing, cross-
user, cross-frequency, and cross-device SLAM mechanisms are proposed. In
addition, SLAM-aided communication strategies are explicitly discussed. We
prove that the multi-domain cooperative SLAM mechanisms based on hybrid
sensing and crowdsourcing can considerably improve the accuracy of
localization and mapping in complex multipath propagation environments through
numerical analysis. Furthermore, we conduct testbed experiments to show that
the proposed SLAM mechanisms can achieve decimeter-level localization and
mapping accuracy in practical scenarios, thereby proving the application
prospect of multi-domain collaborative SLAM in ISAC.

##  I Introduction

Figure 1: Illustration of multi-domain cooperative ISAC networks, where the
ISAC system should provide services for many users, including vehicles, mobile
phones, and IoT devices. The ISAC system interacts with GNSS, UAV, radar,
WLAN, and sensor networks. Cooperation among various sensing types and
networks is imperative.

Future wireless communication systems are expected to provide ubiquitous
connectivity with ultrahigh throughput and reliability and ultralow latency;
they are also expected to realize the ability to sense, control, and even
optimize wireless environments [1]. New applications, such as extended
reality, holographic communications, autonomous driving, smart healthcare, and
intelligent industry, have emerged. They require mass data transmission,
centimeter-level localization, and high fine-grained environmental information
with the rapid advent of the intelligent age. Integrated sensing and
communications (ISAC) is anticipated to play a pivotal role in achieving these
applications[2].

Continuing developments have made wireless communications capable of sensing.
On the one hand, exploited millimeter wave (mmWave) and terahertz (THz)
frequencies coincide with the spectrum of mmWave radar and high-resolution THz
imaging radar. Large bandwidths result in high range resolutions. On the other
hand, upscaled antenna deployment enables unprecedented angular resolution.
Future wireless communication systems coupled with ultradense network
deployment enable a paradigm shift in sensing capabilities. Location
information can aid in addressing several key challenges in communication
systems, including increases in traffic and number of devices, robustness for
mission-critical services, and reduction in total energy consumption and
latency [3]. A perceptive mobile network is proposed to integrate sensing into
communications and share the majority of system modules and the same
transmitted signals [4]. Existing studies on the fundamental limits of ISAC
are comprehensively reviewed in [5]. Four categories of ISAC use cases,
including high-accuracy localization and tracking, simultaneous localization
and mapping (SLAM), augmented human sense, and gesture and activity
recognition, are highlighted in [6].

Although considerable advantages of ISAC have been predicted, deep integration
of sensing and communications still requires further investigation. SLAM can
provide locations of user equipment (UE) and radio features in the propagation
environment. These results help design sensing-aided communication strategies.
Therefore, SLAM is a promising technology for achieving deep integration of
sensing and communications. However, SLAM faces critical challenges in
communication systems due to the massive connections and complex multipath
propagation environments [7, 8]. None of the existing works have realized the
SLAM function under the 5G New Radio (NR) standard because of specification
and hardware constraints. SLAM has relatively mature applications in the field
of robotics; it is often achieved by leveraging the robot’s sensors, such as
inertial measurement unit (IMU), camera, and laser, which provide more
landmarks than features available in typical communication networks [9].
Although the resolution of a radio is lower than that of a camera or light
detection and ranging (LiDAR), a radio can cover a long detection range and is
vaguely affected by weather and light conditions, which are the key challenges
faced by visual and LiDAR SLAM. Moreover, completing the complex task of
realizing high-quality communication and high-accuracy sensing is difficult
for a single network, particularly for a single user [10]. Radio frequency
(RF) signals designed for communication can help realize communication and
sensing by completely reusing the communication hardware. Therefore, other
devices in the communication network can participate in collaborative SLAM. We
aim to develop multi-domain cooperative SLAM mechanisms in the present study
with the cooperation of multiple sensing types, users, frequency bands, and
devices.

The rest of this paper is organized as follows. After briefly introducing the
differences between ISAC and traditional communications, we discuss open
issues and technical challenges from the perspectives of SLAM result
representation, multi-domain cooperative SLAM mechanisms, and SLAM-aided
communication strategies. Therefore, deep integration of sensing and
communication is formed by SLAM. Then, case studies and experiments on the
proposed SLAM mechanisms are conducted, and conclusions are drawn.

##  II Differences between ISAC and traditional communications

Traditional communication systems are designed to offer high data rates and
reliable connections. However, the basic idea of ISAC is to realize dual
functions. On the one hand, communication systems obtain sensing functions,
which can identify objects, localize targets, track devices, and map radio
environments. On the other hand, communication performance in the aspects of
beam management (BM), resource allocation (RA), and signal processing (SP) is
enhanced through sensing. The present study focuses on integrating sensing
into the existing communication-only cellular systems, among which SLAM is a
unique technology that helps achieve deep integration of sensing and
communication. In this section, we reveal the similarities and differences
between ISAC systems and traditional communication systems. The important role
of SLAM in the ISAC systems is also emphasized.

Figure 2: Architecture of a general ISAC system, including ISAC processing
unit, baseband SP, and RF transceivers. Note: Enco., encoding; Deco.,
decoding; Modu., modulating; Demo., demodulating; Equa., equalization; Preco.,
precoding; Comb., combining; Sync., synchronization.

###  II-A Multi-domain Cooperative ISAC Networks

In the era of the Internet of Everything, communication systems should provide
services for various users, including vehicles, mobile phones, and Internet of
Things (IoT) devices, as shown in Fig. 1. Some of these users have certain
sensing capabilities and can become sensing service providers. In addition,
the ISAC system is expected to interact with global navigation satellite
systems (GNSSs), unmanned aerial vehicles (UAVs), radars, wireless local area
networks (WLANs), and sensor networks. Each kind of network has its advantages
and disadvantages. For example, a GNSS can provide position information
outdoors but is ineffective indoors; a WLAN can be economically deployed
indoors, but its sensing resolution is limited. Therefore, cooperation among
heterogeneous networks, which is the foundation of the multi-domain
cooperative SLAM, is imperative.

###  II-B ISAC Protocols

Sensing is gradually supported by default communication frame structures and
protocols according to the 3GPP service and system aspect specifications. UE
and base stations (BSs) transmit uplink sounding reference signals (UL-SRS)
and downlink positioning reference signals (DL-PRS), respectively. They also
capture targets from echo signals and serve as conventional monostatic radars
in the type of active sensing. In passive sensing, the UE or BS does not send
sensing signals while capturing targets through the received signals, which
are sent or reflected from targets. The location management function enables
necessary positioning information and measurement exchange among NR BSs, long-
term evolution BSs, and UE in 5G core networks [11]. Given that sensing covers
a wide range of services, where positioning is only one of them, the protocols
should be further extended to accommodate new sensing applications, such as
health monitoring, imaging, and SLAM. Sensing assistance leads to the
modifications of traditional communication protocols, particularly for SLAM-
aided communication protocols. For example, a traditional exhaustive beam
searching protocol requires a large number of pilots and frequent uplink
feedback to find the optimal beam pair. Overhead and latency can be greatly
reduced by narrowing down the beam search space and reducing the beam sweeping
period with the SLAM-aided BM protocols.

###  II-C ISAC Processing Unit

The architecture of a general ISAC system is presented in Fig. 2, according to
the perceptive mobile network proposed in [4]. Unlike the traditional
transceiver architecture, the ISAC processing unit is a specific module of the
ISAC system and can be deployed in the UE, BS, or network center. First, the
ISAC processing unit needs to support multi-domain data sources, including
data from RF signal, GNSS, camera, LiDAR, and IMU. Among them, RF sensing
parameter analysis (SPA) is a specialized module that works alone to provide
sensing services. The fusion center can integrate the sensing results of
different kinds of UE or devices and provide comprehensive sensing services.
The ISAC processing unit provides different sensing services, including SLAM,
health monitoring, and imaging, according to the requirements. SLAM can
provide the locations of UE and radio features in the propagation environment,
including the locations of BSs and reflectors. These results help design
sensing-aided communication strategies, including BM, RA, and SP. Therefore,
SLAM is a promising technology for achieving deep integration of sensing and
communication.

###  II-D Baseband Signal Processing

The baseband SP of the ISAC system is different from that of the traditional
communication system. SLAM can play a role in baseband SP, as SLAM provides
support for sensing-aided BM, RA, and SP. Moreover, sensing-aided BM, RA, and
SP assist the baseband SP modules. First, RA is controlled by the sensing-
aided RA module in the ISAC processing unit. Second, digital
precoding/combining, channel estimation, and synchronization are optimized by
the sensing-aided SP module. In addition, the sensing parameter extraction
(SPE) module controls the extraction of the received signal strength (RSS),
angle of arrival (AOA), angle of departure (AOD), time of arrival (TOA),
frequency of arrival (FOA), or directly outputs channel state information
(CSI). RF sensing parameters are obtained by active/passive sensing and then
delivered to the RF-SPA module in the ISAC processing unit. The received
information carrying sensing content is sent to the ISAC processing unit after
decoding. This process belongs to the type of interactive sensing.

Figure 3: Illustration of typical radio maps. (a) RSS-based radio map. (b)
Communication-metric-based radio map. (c) Geometric-feature-based radio map.

###  II-E RF Transceiver

The RF transceiver of the ISAC system demonstrates a split function. The
number of RF chains can be split into two for simultaneous transmission and
reception. Moreover, proper transmission and reception antenna isolation
design must be considered. Although the total number of RF chains is fixed,
the number of RF chains for simultaneous transmission and reception can be
controlled by the sensing-aided RA module. Analog precoding/combining can be
optimized through the sensing-aided BM module. SLAM can assist the sensing-
aided BM module, thereby helping the RF transceiver design. The RF transceiver
of the ISAC system degenerates into that of the traditional communication
systems without modifications when sensing shares the communication reference
signals completely. However, the ISAC transceiver can be switched to the mode
where specific sensing reference signals, beams, or RF chains are required.

##  III Open Issues and Technical Challenges

The fusion center and sensing-aided communication modules in the ISAC
processing unit are linked by SLAM techniques. First, we introduce the radio
map, which is the result representation of SLAM. Then, we focus on the open
issues and technical challenges related to communication-enabled cooperative
SLAM mechanisms and SLAM-aided communication strategies.

###  III-A SLAM and Radio Map

Radio maps visualize the RF characteristics of the physical space. SLAM
technology can play an important role in radio map construction. Therefore, we
first discuss the representations of radio maps. Uniform and effective
representation of the radio maps is vital for ISAC systems because (1)
information from different sources can be fused via the same radio map format,
and (2) the radio map can provide guidance for different kinds of UE. Fig. 3
presents several typical radio maps. Region of interest (ROI) is gridded. The
RSS at each grid is collected and recorded in the RSS-based radio map, also
known as the RF fingerprint (Fig. 3[a]). Communication-metric-based radio map
[12] also requires an offline collection process, where we turn the collected
RSS or CSI into communication metrics through proper calculations, such as
channel capacity, outage probability, and line-of-sight (LOS) and non-LOS
(NLOS) indicators (Fig. 3[b]). The geometric-feature-based radio map (Fig.
3[c]) differs from the first two radio maps because griding the ROI and
measuring each grid point are unnecessary. Radio features are abstracted into
several static and sparse points, such as physical anchors (PAs; a PA can be a
BS), virtual anchors (VAs; mirrors of PAs that represent specular reflectors),
and scatterers. Therefore, a small amount of data is required to describe the
geometric-feature-based radio map, which can intuitively provide guidance to
beam direction prediction. SLAM mainly exploits the location and state
invariance of radio features in the propagation environment. Thus, geometric-
feature-based radio maps can be easily constructed from measurements at
successive moments by SLAM techniques.

###  III-B Multi-domain Cooperative SLAM

We present novel communication-enabled SLAM mechanisms with multi-domain
cooperation, including cooperation through active and passive sensing,
multiple UE, multifrequency bands, and multiple devices. Key technologies of
SLAM, such as data association, data fusion, and feature detection, have
changed in different cooperative mechanisms. Specific SLAM algorithms [7, 8,
9, 14, 15], such as belief-propagation-based SLAM, probability hypothesis
density-based SLAM, and expectation propagation-based SLAM, are not summarized
in this article because of the page limitation.

####  III-B1 Cross-Sensing SLAM

Current communication systems consist of beam sweeping, channel estimation,
and data transmission stages [11]. Different sensing types can be realized
during communication processes by using communication waveforms. Active
sensing, which can be performed without establishing a connection in the
communication network, has a known reference signal. However, self-
interference is severe for active sensing in communication systems without a
dedicated self-interference elimination design. Moreover, active sensing has a
smaller scope than passive sensing, particularly on the UE side. The reason is
that UE has limited power, and active sensing relies on LOS paths. Although
passive sensing requires communication link establishment and channel
estimation, multipoint cooperation can be realized, thereby enhancing passive
sensing by information fusion. In summary, active and passive sensing have
advantages and disadvantages. The existing studies typically ignore the
cooperation of different sensing types.

Given that the location and state of the radio feature sensed by active and
passive sensing are strongly related under the same environment, different
sensing types can cooperate through appropriate resource configuration and
unified sensing result representation (such as the geometric-feature-based
radio map). For example, the surrounding scatterers around the BS can be
estimated through active sensing in the beam sweeping stage. Then, the SLAM
results of active sensing are transformed into a unified form (geometric-
feature-based radio map). Therefore, the initialization of radio features in
the propagation environment is realized with the assistance of active sensing.
In the channel sounding stage, the UE may transmit pilot signals in several
beam directions. The BS can execute SLAM to construct the geometric- feature-
based radio map with estimated channel parameters by passive sensing. Thus,
soft information fusion for the same radio feature can be realized. This
process is one of key techniques in SLAM. Legacy feature refinements and new
feature detection are achieved by SLAM with through passive sensing.

The cooperation between active and passive sensing involves many unsolved
problems. First, self-interference is a crucial problem that plagues active
sensing. Antenna isolation and multiarray extrapolation technologies are worth
further research. Second, diffuse and specular reflections bring differences
to theoretical modeling and data association methods. Therefore, data cleaning
and classification techniques based on deep learning should be studied.
Moreover, practical limitations, such as clock and orientation biases, should
be the focus of future investigations.

####  III-B2 Cross-User SLAM

Human and machine connectivity continues to increase in the age of the
Internet of Everything. Thus, the terminals are expected to share the
information for cooperation. As discussed in Section III-B1, each UE can sense
the local radio environment in the communication processes. Moreover, the
local radio features can be mapped by active and passive sensing cooperation.
However, the potential benefits provided by multiple terminals are not fully
utilized in the conventional system.

Multiple UE can collaborate in many ways. First, multiple UE can quickly
establish the global radio feature map of a large ROI. The local radio feature
maps obtained by SLAM from different kinds of UE are subsets of the radio
features in the ROI. A global radio feature map can be built by fusing the
local maps, and the labor-intensive data collection is circumvented. Second,
cooperation helps UE to refine the overlapping parts of the radio feature
maps. This approach requires the data association technique in SLAM. Although
the radio features of each local map may have low confidence, fusing multi-
user data can reduce the uncertainty of shared radio features in the global
map. Third, UE with poor sensing ability can directly inherit reliable radio
features. For a newly accessed UE, the SLAM algorithm can be initialized by
downloading the selected parts of the global map to reduce sensing overhead.
Each UE continuously refines the legacy radio features through SLAM, adds new
radio features, and reports the local map to the cloud database. The cloud
refines and distributes the shared features to the corresponding UE.

One of the main challenges of multi-user collaboration is the feature matching
of local radio maps in different coordinate systems, where computer vision
techniques may be helpful. Data reliability and privacy protection must also
be considered. Moreover, avoiding malicious attacks is one of the critical
issues for multi-user cooperation. Distributed and centralized cooperation
mechanisms must be deeply studied to utilize computing resources efficiently
and save wireless fronthaul and backhaul overhead, in which the advantages of
edge computing should be considered.

####  III-B3 Cross-Frequency SLAM

Cellular systems are expected to ensure communication coverage through low-
frequency bands (sub-6 GHz) and high-quality access in hotspot areas via high-
frequency bands (mmWave and THz) [13]. The sub-6 GHz frequency band offers
wide coverage and reliable link quality but limited sensing capabilities.
High-frequency bands have large bandwidths and antenna arrays, thus enabling
high time and angular resolution. However, severe path loss leads to poor
communication service quality. Existing works rarely consider cross-frequency
SLAM techniques.

Given that 5G NR supports high- and low-frequency dual connections for
collocated and distributed deployment, cross-frequency SLAM can form two
implementation mechanisms accordingly. For collocated deployments, the high-
and low-frequency bands share some common scatterers. However, the propagation
phenomena through the shared scatterers are different. SLAM results can be
represented in different radio map layers with different resolutions. For
example, mmWave/THz transceivers can observe a single specular path associated
with a VA, whereas sub-6 GHz transceivers obtain a batch of clustered paths.
The mean and spread of the AOA, AOD, and TOA of the clustered paths can be
extracted from the received signal in sub-6 GHz, providing rough initial
values for high-frequency bands to reduce the sensing overhead. For
distributed deployments, mmWave/THz transceivers build local radio maps in the
local hotspot areas. Sub-6 GHz transceivers are responsible for network
control and radio map upload/download. The sub-6 GHz BS can also generate a
global radio map by fusing local maps. SLAM is realized with mmWave
measurements, and SLAM results are fused and distributed by sub-6 GHz links.
For example, the sub-6 GHz BS quickly establishes a communication link with
the UE and distributes the related local radio features to the UE when a UE
enters the service area of a heterogeneous network. The UE can quickly access
the local mmWave BS by matching its rough position obtained from the GNSS with
the downloaded radio features. If the UE successfully accesses the mmWave BS,
then the subsequent radio map upload/download can be completed through mmWave
links.

Many problems must be solved urgently in cross-frequency SLAM techniques. The
impact of network switching and execution time on SLAM performance requires
further study. The trade-off between the power consumption and SLAM
performance of heterogeneous networks must be considered. Wireless network
scheduling is an important problem involving RA and optimization for
communication and sensing.

####  III-B4 Cross-Device SLAM

Apart from the possible cooperation of multiple sensing types, users, and
frequency bands, interaction among future cellular networks and various
devices, such as GNSS, camera, LiDAR, and IMU, exists. These devices typically
provide positioning or SLAM services independently. Each has distinct
advantages and limitations.

Collaboration across devices can take advantage of different devices and break
down limitations. GNSS can provide rough initial values for the relative
distance between BS and UE, and then the radio SLAM can work. Cameras can
provide rich information about the environment, allowing for robust and
accurate place recognition. Common features should be identified for visual
and radio SLAM. This process is crucial to data fusion, because the amount of
data and the computational complexity of visual SLAM are unbearable,
particularly for IoT devices with limited computation and storage
capabilities. UE can access public visual SLAM databases through cellular or
Wi-Fi data connections. Thus, complex processing can be performed in the
cloud. Moreover, radio SLAM can take the advantage of advanced techniques in
visual SLAM. Beamspace channels can be represented as images during the beam
sweeping and channel estimation processes. Therefore, deep-learning-based
image processing techniques can be applied to extract angle, delay, and
Doppler measurements from beamspace channels. The target surface properties
and the propagation environment can be presented as point clouds by active
sensing with communication systems. Thus, radio SLAM can be achieved by
classifying and semantically segmenting point clouds using graph neural
networks. The IMU integrated into the smartphone can provide the core
information of the UE’s movement. However, the drift phenomenon is obvious.
The short-term motion measurement is accurate when IMU measurements are
incorporated into the motion prediction module of the SLAM algorithm.
Moreover, the drift is corrected in time by radio measurements.

Handling SLAM services across different devices remain challenging on the
technical level. In particular, connections across devices should be
preconfigured. This process often requires compatible software and platforms.
In addition, plug-and-play data sharing mechanisms for SLAM should be
developed when devices form cross-connections. Therefore, we may opt in or out
of cross-device setups depending on data reliability or situation.

###  III-C SLAM-aided Communications

Multi-domain cooperative SLAM makes the communication systems obtain
situational awareness. Situational awareness is a concept that goes beyond
location awareness. In this concept, the location information of radio nodes
and the knowledge of their propagation environments are collected by SLAM
techniques. Therefore, the traditional communication blocks change with the
involvement of situational awareness during signal transmission and reception,
as shown in Fig. 2.

Figure 4: Case studies of the proposed SLAM mechanisms. (a) Hybrid sensing-
based SLAM. (b) Crowdsourcing-based SLAM.

####  III-C1 Beam Management

Supporting ultrafast and high-rate data exchanges among moving UE and BSs can
hardly be accomplished in traditional mmWave communications because of the
unacceptable beam training overhead. Being aware of the reflecting path allows
quickly establishing another side link during the blockage. The reflecting
path, regarded as a signal from the virtual BS, is obtained by mirroring the
BS on the reflecting surface. Although the reflection point is moving when the
UE is moving, the virtual BS is static during the UE movement. Thus, the
virtual BS serves as a reference point for beam prediction. The predicted beam
directions can greatly narrow down the beam search area, which considerably
accelerates the beam tracking process with low overhead. We can quickly
determine the beamforming gain that each reflector (or candidate beam) can
achieve. Thus, the beam alignment performance at any location in the
environment can be foreseen to achieve uninterrupted wireless access.

####  III-C2 Resource Allocation

Common sensing services, such as positioning or SLAM, can be implemented using
communication pilots without consuming additional resources. For high sensing
resolution services, such as imaging, sensing reference signals can be
allocated through dense comb patterns and periods. The RF chain resources
should be allocated for sensing that requires simultaneous transmission and
reception, such as active sensing. Therefore, resources should be allocated
according to the service requirements of the communication and sensing
functions. Given that SLAM can track channel parameters, such as RSS, AOA,
AOD, TOA, and FOA, structured multipath channels can be constructed with these
parameters. Therefore, the channel capacity and outage probability can be
predicted by using spatial movement coherence combined with channel
predictions. Then, additional spectrum resources can be pre-allocated to
support unexpected surges in communication traffic demand. Temporary BSs can
also be deployed to provide communication links in unfavorable signal outage
zones.

####  III-C3 Signal Processing

Sensing results contribute to the SP strategy design in channel estimation,
small-scale channel prediction, CSI feedback, and synchronization. First, SLAM
results can be used to refine channel parameters with strong geometric
features, such as AOA, AOD, delay, and Doppler, thereby improving channel
estimation and prediction performance. In addition, coarse CSI can be
predicted from the side information of the UE location and propagation
environment. Then, a few pilots are used to estimate the instantaneous small-
scale CSI. Thus, the channel estimation overhead is reduced. Adaptive
combination mechanisms are required to make instantaneous CSI and environment
information complement each other. CSI feedback can also be transformed from
traditional mechanisms to situational awareness mechanisms to reduce latency
and overhead. Location information can determine a rough synchronization
window, simplifying synchronization search operations.

Situational awareness greatly benefits communications by reducing latency and
feedback overhead, improving link reliability, and maintaining high
throughput. However, a flexible frame structure should be formed to allocate
time and frequency resources for sensing and communications. The trade-off
between sensing and communication should be further investigated.

##  IV Case Studies and Numerical Results

###  IV-A Hybrid Sensing-based SLAM

According to Section III-B1, an example of the hybrid sensing-based SLAM is
depicted in Fig. 4(a). Active sensing is performed when the UE sends SRS for
initial access while receiving the echo signal. We assume that transmission
and reception antenna arrays are placed separately on the UE to relax the
self-interference. The corresponding reflective surface point (RSP) can be
obtained with the beam direction and the round-trip time. At least two RSPs
can determine a reflective surface. A VA can also represent a reflective
surface for a specular reflector. Thus, we can convert RSPs (the result of
active sensing) into VAs [14]. The DL-PRS for passive sensing that passes
through the specular NLOS path can be considered coming from the VA.
Therefore, VA becomes the link between active and passive sensing; that is,
active and passive sensing can be fused in the same geometric-feature-based
radio map. Then, we establish an uncertainty model of the results obtained by
active sensing to provide the mean and variance of the estimated VAs for soft
information fusion with passive sensing. Next, we extend the classic belief-
propagation-based SLAM algorithm by realizing PA initialization with the
assistance of active sensing and achieving VA and PA refinement with the help
of passive sensing. Figs. 5(a) and 5(b) show the localization and mapping
performance of the proposed hybrid sensing-based SLAM mechanism compared with
those of different passive sensing-only SLAM mechanisms. PA locations are
perfectly known for two passive sensing-only SLAM mechanisms (the mechanisms
based on VA and master VA [MVA]) [7]. PA locations are unnecessary in the
proposed hybrid sensing-based SLAM mechanism [14]. We select the mean absolute
error (MAE) and mean optimal subpattern assignment (MOSPA) error to measure
localization and mapping performance, respectively. The simulation results
demonstrate that the proposed mechanism obtains the maximum convergence speed
of mapping. Although the proposed mechanism presents a certain performance
loss compared with the MVA-based mechanism, the nonnecessity of prior
information in PAs expands the application scenarios of the proposed
mechanism.

Figure 5: Localization and mapping performance of the proposed SLAM
mechanisms. (a) and (b) for hybrid sensing-based SLAM. (c) and (d) for
crowdsourcing-based SLAM.

###  IV-B Crowdsourcing-based SLAM

The proposed crowdsourcing-based SLAM mechanism is illustrated in Fig. 4(b).
Different kinds of UE cooperate (Section III-B2) to construct and refine the
radio map in a decentralized manner. We define the mapping result obtained by
each UE as the local radio feature map (LRF-Map). Moreover, we define a set in
the cloud as an open radio feature map (ORF-Map), which contains radio
features in the ROI. The ORF-Map is a dynamic set that continuously receives
information from UE for updates. The frame structure of each UE is illustrated
in Fig. 4(b). At the very beginning, the ORF-Map is empty. SLAM is performed
on the UE side, and the LRF-Maps obtained from multiple UE are subsets of the
radio features (PA, VA, and scatterers) in the ROI. Each UE uploads its LRF-
Map to the ISAC processing unit at the network center or cloud. Then, the ORF-
Map is constructed by fusing algorithms. Therefore, labor-intensive data
collection is avoided in the crowdsourcing-based SLAM mechanism. Although the
radio features of each LRF-Map may present low confidence levels, common radio
features (VA 2 in Fig. 4) can be refined with improved reliability via
crowdsourcing data fusion. The established ORF-Map can be downloaded with
various applications. The downloaded information can be used to complete the
LRF-Maps of the already-accessed UE. For newly accessed UE, the ORF-Map is
downloaded at the first moment according to the frame structure, thereby
providing good initial values. In particular, the ORF-Map can provide
candidate legacy features to each UE in need [15]. Each UE continuously
refines the LRF-Map (by adding new features, checking the existence of legacy
features, and deleting unreliable and vanished features) and reports the LRF-
Map to the ISAC processing unit. Figs. 5(c) and 5(d) show the localization and
mapping performance of the proposed crowdsourcing-based SLAM mechanism. We
consider eight users with the combination of AOA and TOA measurements, where
the orientation and clock biases are considered. The entering time is $1$ for
UE 1, 2, and 3; $5$ for UE 4; $10$ for UE 5; $15$ for UE 6; $20$ for UE 7;
$25$ for UE 8. The performance of MAE and MOSPA of UE 8 is presented in Figs.
5(c) and (d), respectively. Compared with the non crowdsourcing mechanism, the
crowdsourcing-based SLAM mechanism exhibits improved average positioning and
mapping accuracy of UE 8 by $42.5\%$ and $64\%$, respectively, at time $60$.
The simulation result verifies the effectiveness of the proposed
crowdsourcing-based SLAM mechanism.

The above two case studies investigated the multi-domain cooperative SLAM
performance in communication networks. We also investigate the SLAM-aided beam
tracking to show the performance of SLAM-aided communications, as described in
Section III-C1. The designed beam tracking module utilizes the prior
information generated by the SLAM algorithm and IMU to narrow down the beam
searching scale. A switching module is established to monitor the phenomenon
of beam birth and death. This module enables the system to switch flexibly
between the full-scale beam sweeping and the small-scale beam tracking
modules. Simulations verify that the proposed SLAM-aided beam tracking
mechanism can reduce the beam training overhead in complex wireless
propagation environments and achieve submeter-level localization and mapping
accuracy.

Figure 6: (a) Floor plan of the experimental scenario. (b) Hardware
components. (c) RSRP measurements. (d) Experimental results of the proposed
SLAM mechanism.

##  V Experimental Results

We conduct proof-of-concept experiments over the air to validate the
feasibility of the proposed SLAM mechanisms. The experiment scenario is shown
in Fig. 6(a). The PA is located at $(5.667,6.290)$ m. We mark the UE’s
possible locations with $25$ grid points to describe its track. Other
complicated scenarios, including those with corridors, multiple wooden doors,
decorative glass walls, and pillars, are also considered. In these scenarios,
the phenomenon of beam birth and death exists. We do not discuss them in this
paper because of the page limitation.

The UE and the PA perform beam sweeping according to 5G NR BM. Our prototype
system is composed of the mmWave phased array (mmPSA-1808), the SDR (USRP-RIO
2974), the mmWave local oscillator, the reference clock node (WR LEN), and the
reference clock source (WRS3-18), as shown in Fig. 6(b). The working frequency
range is $27\\!-\\!29$ GHz, the center frequency is $28$ GHz, and the
transceiver duplex mode is TDD. The PA and the UE share a basic $10$ MHz
reference clock source. The USRP-RIO supports a bandwidth of $160$ MHz. The
$2.8$ GHz intermediate frequency signal is upconverted to $28$ GHz through the
local oscillator. The control signal of the mmWave phased array is generated
by the GPIO port of the USRP-RIO to realize beam switching in strict
accordance with the corresponding 5G NR synchronization signal block resource.
The mmWave phased array is a linear array with eight antenna elements and one
RF chain. We design eight beam patterns, with eight narrow beams covering
$100^{\circ}$ of the space. We adopt the bidirectional exhaustive beam
sweeping method. Then, $64$ pairs of beam RSRPs are measured within $4$ ms at
the UE side. In traditional communication systems, the UE compares $64$ RSRP
measurements to find the optimal transmitted and received beam pair. The UE
feeds this pair back to the PA in the corresponding time slot through the
initial access process. We set four array orientations of the UE and PA (Fig.
6[a]) to map the surrounding reflective surfaces. The beam RSRP is shown in
Fig. 6(c), where (1) to (4) correspond to four array orientations. Therefore,
we obtain $256$ pairs of beam RSRPs. We extract the AOA and AOD measurements
from the beam RSRPs via the successive cancellation method. Then, we obtain
the SLAM results by conducting the belief-propagation-based SLAM algorithm
with AOA and AOD measurements (Fig. 6[d]). The solid blue line is the true
track, the dotted blue line is the estimated track, the red cross is the
estimated VA, and the dot in the heat map color is the estimated scatterer.
ME-MAP and ME-LOC denote the mean errors of mapping and localization,
respectively. The results verify that the electromagnetic wave at $28$ GHz
experiences specular reflection at the smooth wall. Thus, mechanisms that use
geometric-feature-based radio maps for SLAM are feasible. The experimental
results also show that SLAM can be realized with communication signals and
during the beam sweeping process. Moreover, the proposed SLAM mechanism can
achieve decimeter-level localization and mapping accuracy.

##  VI Conclusion

In this article, we first summarized the differences between ISAC and
traditional communication systems in the aspects of network architectures,
protocols, ISAC processing unit, baseband SP, and RF transceivers. SLAM can
provide the locations of UE and radio features in the propagation environment.
These results help design sensing-aided communication strategies. We presented
key ideas for radio maps, cross-sensing, cross-user, cross-frequency, and
cross-device SLAM mechanisms by focusing on multi-domain cooperation. Research
opportunities and challenges were also presented. In addition, SLAM-aided
communications involving BM, RA, and SP were discussed and outlined. Credible
case studies, including hybrid sensing- and crowdsourcing-based SLAM
mechanisms, effectively improved the accuracy of localization and mapping.
Finally, the experimental results verified that the proposed SLAM mechanisms
can achieve decimeter-level localization and mapping accuracy in practical
scenarios. Therefore, the multi-domain cooperative SLAM is essential in
achieving deep integration of sensing and communications.

##  VII Acknowledgments

This work was supported in part by the National Natural Science Foundation of
China (NSFC) under Grants 61921004, 61941104, the Key Research and Development
Program of Shandong Province under Grant 2020CXGC010108, and the Fundamental
Research Funds for the Central Universities 2242022k30005. The work of C.-K.
Wen was supported in part by Qualcomm through a Taiwan University Research
Collaboration Project.

## References

  * [1] A. Bourdoux et al., “6G white paper on localization and sensing,” 2020, [online]. Available: https://arxiv.org/abs/2006.01779.
  * [2] F. Liu, Y. Cui, C. Masouros, J. Xu, T. X. Han, Y. C. Eldar, and S. Buzzi, “Integrated sensing and communications: Towards dual-functional wireless networks for 6G and beyond,” IEEE J. Sel. Areas Commun., pp.1–1, Mar. 2022.
  * [3] R. D. Taranto, S. Muppirisetty, R. Raulefs, D. Slock, T. Svensson, and H. Wymeersch, “Location-aware communications for 5G networks: How location information can improve scalability, latency, and robustness of 5G,” _IEEE Signal Process. Mag._ , vol. 31, no. 6, pp. 102-112, Nov. 2014.
  * [4] J. A. Zhang et al., “Enabling joint communication and radar sensing in mobile networks-A survey,” IEEE Commun. Surveys & Tutorials, vol. 24, no. 1, pp. 306-345, Jan. 2022.
  * [5] A. Liu, Z. Huang, M. Li, et al. “A survey on fundamental limits of integrated sensing and communication,” IEEE Commun. Surveys & Tutorials, vol. 24, no. 2, pp. 994-1034, Feb. 2022.
  * [6] D. K. Pin Tan et al., “Integrated sensing and communication in 6G: Motivations, use cases, requirements, challenges and future directions,” in proc. 1st IEEE International Online Symposium on Joint Communications & Sensing (JC&S), Mar. 2021, pp. 1-6.
  * [7] E. Leitinger, F. Meyer, F. Hlawatsch, K. Witrisal, F. Tufvesson, and M. Z. Win, “A belief propagation algorithm for multipath-based SLAM,” _IEEE Trans. Wireless Commun._ , vol. 18, no. 12, pp. 5613-5629, Sept. 2019.
  * [8] R. Mendrzik, F. Meyer, G. Bauch, and M. Z. Win, “Enabling situational awareness in millimeter wave massive MIMO systems,” _IEEE J. Sel. Topics Signal Process._ , vol. 13, no. 5, pp. 1196-1211, Sept. 2019.
  * [9] H. Durrant-Whyte and T. Bailey, “Simultaneous localization and mapping: Part I,” _IEEE Robotics & Automation Mag._, vol. 13, no. 2, pp. 99-110, Jun. 2006.
  * [10] M. Z. Win, Y. Shen, and W. Dai, “A theoretical foundation of network localization and navigation,” Proc. IEEE, vol. 106, no. 7, pp. 1136-1165, Jul. 2018.
  * [11] M. Giordani, M. Polese, A. Roy, D. Castor, and M. Zorzi, “A Tutorial on Beam Management for 3GPP NR at mmWave Frequencies,” IEEE Commun. Surveys Tutorials, vol. 21, no. 1, pp. 173-196, Sept. 2019.
  * [12] Y. Zeng and X. Xu, “Toward environment-aware 6G communications via channel knowledge map,” IEEE Wireless Commun., vol. 28, no. 3, pp. 84-91, Jun. 2021.
  * [13] N. Gonzalez-Prelcic, A. Ali, V. Va, and R. W. Heath, “Millimeter-wave communication with out-of-band information,” IEEE Commun. Mag., vol. 55, no. 12, pp. 140-146, Dec. 2017.
  * [14] J. Yang, C. -K. Wen, and S. Jin, “Hybrid active and passive sensing for SLAM in wireless communication systems,” IEEE J. Sel. Areas Commun., vol. 40, no. 7, pp. 2146-2163, Jul. 2022.
  * [15] J. Yang, C. -K. Wen, S. Jin, and X. Li, “Enabling plug-and-play and crowdsourcing SLAM in wireless communication systems,” IEEE Trans. Wireless Commun., vol. 21, no. 3, pp. 1453-1468, Mar. 2022.

Jie Yang [S’18, M’22]<EMAIL_ADDRESS>received the B.S. degree in
communication engineering from Nanjing University of Science and Technology,
Nanjing, China, in 2015, the M.S. and Ph.D. degrees in information and
communications engineering from Southeast University, Nanjing, China, in 2018
and 2022, respectively. In 2022, she joined the School of Automation,
Southeast University, Nanjing, China, where she is currently an Assistant
Professor. Her current research interests include signal processing for
wireless communications, massive MIMO, millimeter-wave wireless
communications, and integrated sensing and communications.
---
Chao-Kai Wen [S’00, M’04, SM’20]<EMAIL_ADDRESS>received the
Ph.D. degree from the Institute of Communications Engineering, National Tsing
Hua University, Taiwan, in 2004. He was with Industrial Technology Research
Institute, Hsinchu, Taiwan and MediaTek Inc., Hsinchu, Taiwan, from 2004 to
2009. Since 2009, he has been with National Sun Yat-sen University, Taiwan,
where he is Professor of the Institute of Communications Engineering. His
research interests center around the optimization in wireless multimedia
networks.
---
Xi Yang<EMAIL_ADDRESS>received the B.S. degree in electrical engineering
from University of Electronic Science and Technology of China, Chengdu, China,
in 2013, the M.S. and Ph.D. degrees in information and communications
engineering from Southeast University, Nanjing, China, in 2016 and 2022,
respectively. His research interests center around the signal processing in
wireless communications.
---
Jing Xu<EMAIL_ADDRESS>received the B.S. and M.S. degrees in information
and communications engineering from Southeast University, Nanjing, China, in
2019 and 2022, respectively. His research interests include integrated sensing
and communications.
---
Tao Du<EMAIL_ADDRESS>received the B.S. degree in electronic and
information engineering from Nanjing University of Science and Technology,
Nanjing, China, in 2018, the M.S. degree in information and communications
engineering from Southeast University, Nanjing, China, in 2021. He is
currently working towards the Ph.D. degree in information and communications
engineering with Southeast University, Nanjing, China. His current research
interests center around the integrated sensing and communication.
---
Shi Jin [S’06, M’07, SM’17]<EMAIL_ADDRESS>received his Ph.D. degree in
communications and information systems from Southeast University in 2007. From
June 2007 to October 2009, he was a research fellow with the Adastral Park
Research Campus, University College London, United Kingdom. He is currently
with the faculty of the National Mobile Communications Research Laboratory,
Southeast University. His research interests include space-time wireless
communications, information theory, intelligent communications, and
reconfigurable intelligent surfaces. He and his coauthors received the 2010
Young Author Best Paper Award from the IEEE Signal Processing Society and the
2011 IEEE Communications Society Stephen O. Rice Prize Paper Award in the
field of communication theory.
---


# Improved Customer Transaction Classification using Semi-Supervised Knowledge
Distillation

Rohan Sukumaran ∗
Applied Research, Swiggy
<EMAIL_ADDRESS>
###### Abstract

In pickup and delivery services, transaction classification based on customer
provided free text is a challenging problem. It involves association of a wide
variety of customer inputs to a fixed set of categories while adapting to the
various customer writing styles. This categorization is important for the
business: it helps understand the market needs and trends, and also assist in
building a personalized experience for different segments of the customers.
Hence, it is vital to capture these category information trends at scale, with
high precision and recall. In this paper, we focus on a specific use-case
where a single category drives each transaction. We propose a cost-effective
transaction classification approach based on semi-supervision and knowledge
distillation frameworks. The approach identifies the category of a transaction
using free text input given by the customer. We use weak labelling and notice
that the performance gains are similar to that of using human annotated
samples. On a large internal dataset and on 20Newsgroup dataset, we see that
RoBERTa performs the best for the categorization tasks. Further, using an
ALBERT model (it has 33x less parameters vis-a-vis paremeters of RoBERTa),
with RoBERTa as the Teacher, we see a performance similar to that of RoBERTa
and better performance over unadapted ALBERT. This framework, with ALBERT as
student and RoBERTa as teacher, is further referred to as R-ALBERT in this
paper. The model is in production and is used by business to understand
changing trends and take appropriate decisions.

11footnotetext: Work done while an intern at Applied Research, Swiggy.

##  1 Introduction

Natural Language Understanding (NLU) has become a ubiquitous part of the
e-commerce platforms with applications in information discovery Degenhardt et
al. (2019); Grandhi (2016), sentiment analysis Liang and Wang (2019); Taylor
and Keselj (2020), query understanding Zhao et al. (2019) and many other
similar tasks. The advancement in machine learning has enhanced the scale of
adaption of such capabilities to enable richer customer experience.

In this paper, we consider a use-case of pickup and delivery services where
customers make use of short sentences or lists to inform about the products to
be picked up (or bought) from a certain location and dropped at a target
location. Table 1 shows a few examples of the descriptions used by our
customers to describe their products. Customers tend to use short, code-mixed
(using more than one language in the same message111In our case, Hindi written
in roman case is mixed with English) and incoherent textual descriptions of
the products for describing them in the transaction. These descriptions, if
mapped to a fixed set of categories, will help assist critical business
decisions such as how to enhance the customers’ experience on the platform,
understand the importance of each category and issues faced by them,
demographic driven prioritization of categories, launch of new product
categories etc. As expected, a transaction may comprise of multiple products -
this adds an additional level of complexity to the task. In this work, we
focus on a multi-class classification of transactions, where a single majority
category drives the transaction.

Transaction Description |  Category
---|---
”Get me my lehanga”
Translation : Get me my skirt |  Clothes
”Buy a 500gm packet of Whole grain Atta” |  Grocery
”Get a roll of paratha” |  Food
”Mera do bags leaoo”
Translation : Bring two of my bags |  Package
Table 1: Samples of actual transaction descriptions used by our customers
along with their corresponding categories as labelled by the subject matter
experts (SMEs).

Classifying transaction types to the right categories requires labelled data
for training an ML model. Some of the data used in this paper (Table 2: Train
data) was labeled manually by the subject matter experts (SMEs) from the
business and it was a very expensive exercise.

Recent works have shown that BERT based models Conneau et al. (2019); Yang et
al. (2019) are able to achieve state-of-the-art performance on text
classification tasks. Through experiments we could observe that RoBERTa Liu et
al. (2019) was the best performing model for our task. However, owing to the
large number of parameters, it was not feasible to deploy this model at scale.
Furthermore, we observed that the lighter versions of BERT such as ALBERT
(base) Lan et al. (2019); Wolf et al. (2019) that are production friendly
could not match the performance of RoBERTa.

With the above two limitations for production friendliness in mind, we
propose: a) an approach that leverages semi supervision to reduce the manual
labeling cost and, b) also explore knowledge distillation to build a smaller
model (in terms of numbers of parameters) that matches the performance of the
state-of-the-art heavier models such as RoBERTa. The key contributions of this
paper are:

  * •

Weak Labelling: A framework based on semi-supervised learning and weak
supervision to reduce manual data labelling bottleneck.

  * •

Knowledge Distillation Framework: Training a lightweight model (33x lesser
parameters) with the help of weak labels, which is able to match the
performance of a much heavier model.

  * •

Inference and Scalability: Attained 86 times increase in inference speed with
the Student model while running on a r5 4x large (Intel Xeon Platinum 8000
series processor (Skylake-SP) with a sustained all core Turbo CPU clock speed
of up to 3.1 GHz) machine in production when compared to that of the Teacher
model.

##  2 Related Work

Text classification problems with code-mixed inputs have been studied and
transformer based models perform well on benchmarks Chang et al. (2019); Lu et
al. (2020) like TREC-6 Voorhees and Harman (2000) and DBpedia Auer et al.
(2007). The larger size of the models exacerbates the challenge of deployment
with limited resources Chen et al. (2020); Sajjad et al. (2020). Multiple
methods like quantization Zafrir et al. (2019), pruning Gordon et al. (2020),
distillation Sanh et al. (2019); Jiao et al. (2019) and weight sharing Houlsby
et al. (2019) are used to mitigate this issue. All these methods have shown
varying degrees of success compared to the performance of the base model from
which they are derived.

###  2.1 Weak labeling of data

Data-less text classification has become a popular practice to achieve low
cost model training. Hingmire and Chakraborti (2014) explored approaches based
on topic modelling to predict labels for documents. Our problem setting
involves short transaction descriptions that do not perform well with standard
topic modelling techniques. Li and Yang (2018) worked with unlabeled data by
identifying a minimal set of seed word based pseudo labels for documents and
trained a Naive Bayes model using semi-supervision. We had a large amount of
manually tagged data - we leveraged this to extract candidate training samples
from unlabeled data.

###  2.2 Knowledge Distillation

Hinton et al. (2015) studied how a model can be used to label unlabelled data
and make use of the model predictions (the probability distributions and/or
the one hot encoded labels) for training using a combination of loss
functions. We borrow from Hinton et al. (2015) the concept of using different
loss functions during training. In this paper, we use transformer models as
teacher and student. In Yuan et al. (2019), the authors proposed how a model
could be it’s own teacher and how lightweight models can teach a heavier
model. Although this showed promising results in image classification tasks,
we observed that the similar performance gains was not visible for our setting
of NLP tasks. This could be due to the difference in the underlying
information captured in representation learning. Hence, we stick to the
standard teacher-student framework where heavier models are used as the
teacher.

###  2.3 Fine-tuning on target tasks before teaching

The Multifit Eisenschlos et al. (2019) paper proposed to fine-tune a
lightweight model on the target task with labelled or pseudo labelled data and
empirically showed the performance improvement derived from using the teacher
model. We leverage a similar setup and validate the generalization of the
method to large scale transformer based models. We observe this to be an
effective training strategy. The study also showed that the student model was
robust to noise as the teacher served as a regularizer. In Tamborrino et al.
(2020) also the authors explain the benefits of relevant knowledge transfer
via task specific fine-tuning.

##  3 Methodology

Our approach is focused on leveraging knowledge distillation to build a highly
accurate classifier with reduced cost of training and deployment. We train a
model using manually labeled data, and call it as the “Teacher”. We then pass
the unlabelled data through this Teacher model. The output of the Teacher
model is used as the label for this unlabelled data. This semi-supervision
approach is used for data augmentation. Further, this augmented data is used
to train a “Student” model. With the weakly labeled data, we employ multiple
strategies to improve the performance of the “Student” model. Figure 1 shows a
high level system overview.

Figure 1: High level overview of the process of Knowledge Distillation using
Semi-Supervision

###  3.1 Cross entropy as the loss function with hard labels

In this approach, we leverage the teacher model to obtain weak labels for
unlabelled data. For a given sample we assign the most confident prediction
(from the “Teacher”) as its label. In other words, the output probabilities
for each sample are converted into one hot encodings considering the
prediction with highest probability as the true label.

###  3.2 Cross Entropy as the loss function with soft labels

Similar to the previous approach, we leverage the teacher model to obtain weak
labels on a dataset. But instead of one hot encodings, we consider the
probability distribution of the predictions as the labels for the samples. In
other words, we perform semi supervision while ensuring to replicate the
teacher’s behaviour when labelling the samples.

###  3.3 KL Divergence as the loss function with soft labels

Inspired by the results of Section 3.2, we leverage the probability
distribution based labels but make use of KL divergence Kullback (1997) loss
instead of Cross Entropy loss. Results show that this strategy performs the
best to learn the difference between two probability distributions (the
predictions from the student and the teacher models).

##  4 Experiments

###  4.1 Dataset

Table 2 shows the different datasets considered for the experiments. The
initial training data comprises of 41,539 customer transactions sampled from
September to December (2019) time frame - each of these transactions had an
associated customer message. The free text message of every transaction was
annotated by a team of three SMEs, and mapped to one of the ten pre-defined
categories. The list of categories considered are as follows: {’Food’,
’Grocery’, ’Package’, ’Medicines’, ’Household Items’, ’Cigarettes’, ’Clothes’,
’Electronics’, ’Keys’, ’Documents/Books’}.

Additionally, we consider 285,235 unlabelled customer transactions sampled
from January to April (2020) for the semi-supervision experiments. For
benchmarking the performance of different classification approaches, we label
20,156 customer transactions from April to construct a test dataset. This test
set from April, containing 20,156 samples are not used for the semi-
supervision experiments.

Dataset | Duration | Size
---|---|---
Train | Sept - Dec | 41,539
Unlabelled | Jan - Apr. | 285,235
Test | April | 20,156
Table 2: Dataset description

###  4.2 Training the Teacher model

In the first step, we train multiple models using the Train dataset and test
on Test dataset to identify the candidate teacher model for our Knowledge
Distillation experiments. For the purpose of the experiments, we consider
XgBoost Chen et al. (2015); Pedregosa et al. (2011), BiLSTM Gers et al.
(1999); Abadi et al. (2016), ALBERT Lan et al. (2019) and RoBERTa Liu et al.
(2019). Table 3 shows the F1-scores for different models considered for this
experiment. We observe that ALBERT and RoBERTa outperform BiLSTM and XgBoost.
Therefore, RoBERTa is chosen as the teacher model for the next set of
experiments.

Model | F1 Score | Accuracy
---|---|---
XgBoost | 0.60 | 63
BiLSTM | 0.65 | 73
ALBERT | 0.70 | 78
RoBERTa | 0.74 | 82
Table 3: F1-scores and Accuracies for different classification models trained
on Train dataset and tested on Test dataset

###  4.3 Generate Weakly Labeled data using Semi-Supervision

In the second step, we leverage the teacher model described in the previous
subsection to extract weakly labeled samples for the Unlabelled dataset to
augment the training dataset. To reduce the probability of selecting
mislabeled samples, we set an empirical threshold of 95% confidence in the
label prediction as the criteria to accept a sample into the pool of training
samples. At the end of this process, we obtain 93,820 additional training
samples (weakly labelled by the teacher).

###  4.4 Leverage Knowledge Distillation to train a Student model

Based on the production bottlenecks on number of parameters, the ALBERT(base)
with 1̃1 million parameters was chosen as the student model from the set of
SOTA models. The model details can be found in Table 7 under Appendix A.

The data from Section 4.3 is used further to “teach” the student models’
making use of the 3 strategies mentioned in Section 3. The student model
ALBERT is already trained (fine-tuned) on the labelled Train dataset which was
used for training the teacher. R-ALBERT, the new student model based on the
ALBERT architecture ends up performing the best and even better than the
“teacher” model on our Test dataset. Similar pattern was observed in
Eisenschlos et al. (2019).

###  4.5 Reproducibility

In order to validate the reproduciblity of our approach, we ran similar
experiments on the 20Newsgroup Rennie (2008) dataset.

##  5 Results

As shown in Table 4, the Student model tends to perform better than its base
version (the model which did not have a Teacher). We validate the statistical
significance of the performance improvement using Stuart Maxwell Test Stuart
(1955); Maxwell (1970). As shown in Table 6, the performance improvement over
the base model is significant. Moreover, we observe that our approach achieved
similar performance when compared to human annotated data, despite the change
in data distributions and textual patterns. Also, from Table 5 we observe that
the given method is reproducible on the 20Newsgroup dataset.Further, we
observe that the distillation over RoBERTa’s predictions gave an improvement
of 8% when compared to fine-tuning directly on part of the labelled dataset.

Model | F1 Score | Accuracy
---|---|---
R-ALBERT - OHE | 0.72 | 83
R-ALBERT - CE | 0.65 | 64
R-ALBERT - KL | 0.73 | 84
RoBERTa | 0.18 | 40
Table 4:  Comparison of F1-scores on internal benchmark using different approaches Model | F1-score | Accuracy (%)
---|---|---
ALBERT | 0.63 | 65
R-ALBERT-KL | 0.70 | 73
RoBERTa | 0.88 | 87
Table 5: F1-scores on the 20Newsgroup dataset with 8,073 train samples, 7,037
weakly labeled samples (after 95% threshold) and 805 samples

##  6 Conclusion

We explore a generalised distillation framework on transformer based
architecture which shows that “Students” can be made better with the help of
the weak labels generated by a good “Teacher”. Given the focus on having
smaller models and to ensure that we make effective utilisation of resources,
we feel this distillation method will pave path for more research in the
future. This framework can also help reduce manual labelling efforts.

## References

  * Abadi et al. (2016) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016\.  Tensorflow: A system for large-scale machine learning.  In _12th $\\{$USENIX$\\}$ symposium on operating systems design and implementation ($\\{$OSDI$\\}$ 16)_, pages 265–283.
  * Auer et al. (2007) Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007.  Dbpedia: A nucleus for a web of open data.  In _The semantic web_ , pages 722–735. Springer.
  * Chang et al. (2019) Wei-Cheng Chang, Hsiang-Fu Yu, Kai Zhong, Yiming Yang, and Inderjit Dhillon. 2019\.  X-bert: extreme multi-label text classification with using bidirectional encoder representations from transformers.  _arXiv preprint arXiv:1905.02331_.
  * Chen et al. (2020) Daoyuan Chen, Yaliang Li, Minghui Qiu, Zhen Wang, Bofang Li, Bolin Ding, Hongbo Deng, Jun Huang, Wei Lin, and Jingren Zhou. 2020.  Adabert: Task-adaptive bert compression with differentiable neural architecture search.  _arXiv preprint arXiv:2001.04246_.
  * Chen et al. (2015) Tianqi Chen, Tong He, Michael Benesty, Vadim Khotilovich, and Yuan Tang. 2015.  Xgboost: extreme gradient boosting.  _R package version 0.4-2_ , pages 1–4.
  * Conneau et al. (2019) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Unsupervised cross-lingual representation learning at scale.  _arXiv preprint arXiv:1911.02116_.
  * Degenhardt et al. (2019) Jon Degenhardt, Surya Kallumadi, Utkarsh Porwal, and Andrew Trotman. 2019.  Ecom’19: The sigir 2019 workshop on ecommerce.  In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 1421–1422.
  * Eisenschlos et al. (2019) Julian Eisenschlos, Sebastian Ruder, Piotr Czapla, Marcin Kardas, Sylvain Gugger, and Jeremy Howard. 2019.  Multifit: Efficient multi-lingual language model fine-tuning.  _arXiv preprint arXiv:1909.04761_.
  * Gers et al. (1999) Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. 1999.  Learning to forget: Continual prediction with lstm.
  * Gordon et al. (2020) Mitchell A Gordon, Kevin Duh, and Nicholas Andrews. 2020.  Compressing bert: Studying the effects of weight pruning on transfer learning.  _arXiv preprint arXiv:2002.08307_.
  * Grandhi (2016) Roopnath Grandhi. 2016.  Methods and systems of discovery of products in e-commerce.  US Patent App. 14/830,696.
  * Hingmire and Chakraborti (2014) Swapnil Hingmire and Sutanu Chakraborti. 2014.  Topic labeled text classification: A weakly supervised approach.  In _ACM SIGIR Conference on Research and Development in Information Retrieval_.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for nlp.  _arXiv preprint arXiv:1902.00751_.
  * Jiao et al. (2019) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019.  Tinybert: Distilling bert for natural language understanding.  _arXiv preprint arXiv:1909.10351_.
  * Kullback (1997) Solomon Kullback. 1997.  _Information theory and statistics_.  Courier Corporation.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019.  Albert: A lite bert for self-supervised learning of language representations.  _arXiv preprint arXiv:1909.11942_.
  * Li and Yang (2018) Ximing Li and Bo Yang. 2018.  A pseudo label based dataless naive bayes algorithm for text.  In _International Conference on Computational Linguistics_.
  * Liang and Wang (2019) Ruxia Liang and Jian-qiang Wang. 2019.  A linguistic intuitionistic cloud decision support model with sentiment analysis for product selection in e-commerce.  _International Journal of Fuzzy Systems_ , 21(3):963–977.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Lu et al. (2020) Zhibin Lu, Pan Du, and Jian-Yun Nie. 2020.  Vgcn-bert: Augmenting bert with graph embedding for text classification.  In _European Conference on Information Retrieval_ , pages 369–382. Springer.
  * Maxwell (1970) Albert Ernest Maxwell. 1970.  Comparing the classification of subjects by two independent judges.  _The British Journal of Psychiatry_ , 116(535):651–655.
  * Pedregosa et al. (2011) Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011.  Scikit-learn: Machine learning in python.  _the Journal of machine Learning research_ , 12:2825–2830.
  * Rennie (2008) Jason Rennie. 2008.  Newsgroups dataset, 2008.
  * Sajjad et al. (2020) Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. 2020.  Poor man’s bert: Smaller and faster transformer models.  _arXiv preprint arXiv:2004.03844_.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Stuart (1955) Alan Stuart. 1955.  A test for homogeneity of the marginal distributions in a two-way classification.  _Biometrika_ , 42(3/4):412–416.
  * Tamborrino et al. (2020) Alexandre Tamborrino, Nicola Pellicano, Baptiste Pannier, Pascal Voitot, and Louise Naudin. 2020.  Pre-training is (almost) all you need: An application to commonsense reasoning.  _arXiv preprint arXiv:2004.14074_.
  * Taylor and Keselj (2020) Stacey Taylor and Vlado Keselj. 2020.  e-commerce and sentiment analysis: Predicting outcomes of class action lawsuits.  In _Proceedings of The 3rd Workshop on e-Commerce and NLP_ , pages 77–85.
  * Voorhees and Harman (2000) Ellen M. Voorhees and Donna Harman. 2000.  Overview of the sixth text retrieval conference (trec-6).  _Inf. Process. Manage._ , 36(1):3–35.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _ArXiv_ , pages arXiv–1910.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in neural information processing systems_ , pages 5753–5763.
  * Yuan et al. (2019) Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. 2019.  Revisit knowledge distillation: a teacher-free framework.  _arXiv preprint arXiv:1909.11723_.
  * Zafrir et al. (2019) Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019.  Q8bert: Quantized 8bit bert.  _arXiv preprint arXiv:1910.06188_.
  * Zhao et al. (2019) Jiashu Zhao, Hongshen Chen, and Dawei Yin. 2019.  A dynamic product-aware learning model for e-commerce query intent understanding.  In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_ , pages 1843–1852.

##  Appendix A Appendix

Model 1 | Model 2 | Chi-square | DoF | p-value (<)
---|---|---|---|---
ALBERT | RoBERTa | 2185.71 | 9 | 2.2e-16
R-ALBERT-KL | R-ALBERT-OHE | 955.61 | 9 | 2.2e-16
R-ALBERT-KL | RoBERTa | 955.61 | 9 | 2.2e-16
Table 6: The results of Stuart Maxwell Test’s can be found, where we compare different models and we can see that the increase/decrease in performance is statistically significant. Model | Parameters (in millions)
---|---
ALBERT(base) | 11
distilBERT(base) | 66
distilRoBERTa(base) | 82
RoBERTa(base) | 125
RoBERTa(large) | 355
Table 7: As seen from the table, ALBERT(base) has the least no of parameters
when compared to the rest of the models. hence we chose ALBERT as the student.
distilRoBERTa would have been a good choice, given that tit has signals from
RoBERTa, but due to memory requirements in production we choose ALBERT(base)