# The OPNV Data Collection: A Dataset for Infrastructure-Supported Perception
Research with Focus on Public Transportation

Marcel Vosshans1,2, Alexander Baumann1, Matthias Drueppel3, Omar Ait-Aider2,
Ralf Wörner1,
Youcef Mezouar2, Thao Dang1 and Markus Enzweiler1 1The authors are with the
Institute for Intelligent Systems which is part of the Faculty of Computer
Science and Engineering, University of Applied Sciences Esslingen, Germany
{marcel.vosshans, alexander.baumann, ralf.woerner, thao.dang,
<EMAIL_ADDRESS>authors are with the Institut Pascal
ISPR (Images, Perception Systems and Robotics), Universite Clermont Auvergne
INP / CNRS, France {youcef.mezouar<EMAIL_ADDRESS>author is with
the Center for Artificial Intelligence, Baden Württemberg Cooperative State
University Stuttgart, Germany<EMAIL_ADDRESS>
###### Abstract

This paper we present our vision and ongoing work for a novel dataset designed
to advance research into the interoperability of intelligent vehicles and
infrastructure, specifically aimed at enhancing cooperative perception and
interaction in the realm of public transportation. Unlike conventional
datasets centered on ego-vehicle data, this approach encompasses both a
stationary sensor tower and a moving vehicle, each equipped with cameras,
LiDARs, and GNSS, while the vehicle additionally includes an inertial
navigation system. Our setup features comprehensive calibration and time
synchronization, ensuring seamless and accurate sensor data fusion crucial for
studying complex, dynamic scenes. Emphasizing public transportation, the
dataset targets to include scenes like bus station maneuvers and driving on
dedicated bus lanes, reflecting the specifics of small public buses. We
introduce the open-source .4mse file format for the new dataset, accompanied
by a research kit. This kit provides tools such as ego-motion compensation or
LiDAR-to-camera projection enabling advanced research on intelligent vehicle-
infrastructure integration. Our approach does not include annotations;
however, we plan to implement automatically generated labels sourced from
state-of-the-art public repositories.

Several aspects are still up for discussion, and timely feedback from the
community would be greatly appreciated. A sneak preview on one data frame will
be available at a Google Colab Notebook. Moreover, we will use the related
GitHub Repository to collect remarks and suggestions.

##  I INTRODUCTION

Since the question has arisen whether to opt for 5G (C-V2X) or ITS-G5 (DSRC)
for intelligent vehicles [1], the research into infrastructure-related
interoperability for intelligent traffic systems seems to tend to be
technology-driven (bottom-up). A reason might be the assumption that the
transmitting bandwidth is limited [2], which is currently the case. One of the
projects in this field of research is LUKAS [3], which has contributed to
infrastructure-supported maneuver planning and traffic management, with
investigations focused on an intersection in Ulm, Germany. Equipped with
cameras, the project has developed strategies and algorithms for
infrastructure-supported perception within the V2X communication stack, as
exemplified in [4].

Our paper introduces the OPNV (Open Public Network for Versatile) Data
Collection that could offer opportunities to investigate applications from a
requirement-oriented perspective (top-down). It will be designed to facilitate
the study of the interoperability of intelligent vehicles with the
infrastructure, focusing on cooperative perception and interaction, similar to
[5].

Figure 1: Sample data visualization: LiDAR-Camera Projection from ego-vehicle,
featuring data from the Stereo-Left camera and all three LiDARs. Color-coded
depth representation indicates proximity, with nearest points in yellow,
gradiating through purple to black for the most distant points.

The significance of such a dataset appears straightforward, considering that
an external observational instance could augment perception and decision-
making processes, supporting on-board sensor technologies for planning and
reaction mechanisms. Unlike existing datasets that primarily collect data from
an ego-vehicle, such as KITTI [6], NuScenes [7], or Waymo OD [8], our dataset
includes vehicle-to-infrastructure capabilities. It incorporates data from
both an ego-vehicle and a sensor tower, each equipped with cameras, LiDARs,
GNSS, and an additional INS on the vehicle, while ensuring precise calibration
(both intrinsic and extrinsic) and synchronization between the two. Figure 1
depicts a sample of a resulting LiDAR-camera projection from the ego-vehicle
perspective, showcasing the precise calibration and synchronization.

A distinctive feature of our concept is its focus on public transportation,
although the data and scenes will be applicable to both private motorized
vehicles and public transport systems. The difference lies in the scene
selection, such as pick-up/drop-off maneuvers at bus stations or the use of
dedicated bus lanes, as well as the dimensions of the ego-vehicle and its
sensor height setup. Furthermore, the current understanding of public
transportation aligns well with a business case for autonomous,
infrastructure-supported systems: bus routes are organized within delimited
and well-defined domains, which justifies the investment in location
restricted infrastructure equipment. Our approach also addresses today’s
challenges faced by automated people movers, like those from Navya [9] or
EasyMile [10]. Supplementary and independent sensors in the infrastructure may
increase the vehicles field of view and reduce the risk of accidents.

The dataset can be used to advance the development of intelligent
infrastructure, enhancing confidence and safety in autonomous driving by
providing redundancy in perception tasks and enabling real-time decision
validation. One possible opportunity is to develop strategies that enable
fault tolerance for automated vehicles within infrastructure-supported
domains, such as public transport. With a reliable spatial registration in
place, validation and redundancy mechanisms can be applied to safeguard moving
agents. Another unique aspect of the dataset is the opportunity to experiment
with the amount of data transmitted to optimize applications and define
requirements from a top-down perspective. Considering a safe right-turn at an
intersection: this situation gets complicated due to traffic flow of both
vehicles and pedestrians. A traffic light system could detect a pedestrian’s
intent to cross and coordinate light signals for automated vehicles while
providing warnings to ADAS systems.

###  I-A Contributions

  1. 1.

Comprehensive concept for recording and synchro-nizing interoperable traffic
data:

     * •

Description of our vehicle and tower setup including sensor positioning, time
synchronization, intrinsic and extrinsic calibration

     * •

Holistic approaches for the dataset collection featuring a diverse scene plan,
content extensibility and automatic ground truth

  2. 2.

Research and development kit:

     * •

Assistive tools: Image rectification, LiDAR-to-camera projection, LiDAR
undistortion and hidden point removal

     * •

Introduction of .4mse file format and export capabilities to ROS’s .bag format

###  I-B Public Resources

  * •

OPNV
Dataset111https://colab.research.google.com/drive/1p2cw3bSZ6B798qQ2jVnpvKQI5pv_-
y_D (Sneak Preview)

  * •

Research Kit Repository: 222https://github.com/MarcelVSHNS/opnv-dataset

###  I-C Paper Organization

We start by reviewing related work, positioning our contributions within the
broader research landscape. Next, we introduce our dataset specification and
detail its creation process, covering the setup of the sensor tower and the
ego-vehicle, along with the calibration and synchronization processes that
ensures data integrity. Subsequently, we present our development kit,
outlining its key features that facilitate data exploration and algorithm
development. The paper concludes with points for discussion, where we hope for
support and advice from the scientific community.

##  II RELATED WORK

###  II-A Ego-vehicle Datasets

Over the last decade, numerous open datasets from the ego-vehicle perspective
have been released, varying in their sensor setup, scope, and size. The first
one introducing this era of large scale autonomous driving datasets, the KITTI
Dataset, was published by Geiger et al. [6] in 2012. It provides 1.5 hours of
varying real-world driving scenes recorded by synchronized and calibrated
sensors from different modalities such as two stereo-cameras, a 3D LiDAR and
an INS. Additionally, 3D annotations were provided, and several challenges
were attempted in the following years including 2D/3D object detection and
tracking, optical flow, scene flow, depth estimation and SLAM.

While KITTI enabled a broad range of applications, the understanding of
complex urban scenarios was still an unsolved challenge due to the lack of
datasets addressing this particular issue. Cityscapes [11] filled this gap
with the release of their dataset consisting of stereo images with both, pixel
and instance level annotations. The absence of LiDAR, combined with precise
scene selection, emphasizes their focus on the visual understanding of urban
street scenes.

ApolloScape [12] was subsequently released also providing pixel- and instance-
wise semantic labels with additional lane mark annotations and semantic 3D
point clouds of the environment. The recordings include various weather and
day times, but the main contribution is their large amount of labeled frames
(140K).

Following the release of these visual scene understanding datasets, the
industry shifted focus towards richer multi-modal datasets that better reflect
the sensor setup of automated vehicles. The first dataset to accomplish this
is the nuScenes [7] dataset. With the use of six cameras, five RaDARs and one
LiDAR, all covering a 360° field of view (FOV), and additional GPS, IMU and
raw CAN bus data, they provide the most complete package up to this point.
They also provide 3D bounding boxes and semantic maps of the relevant areas.

In the same year the Waymo Open Dataset [8] was released following a quite
similar approach. Their setup consists of 5 Cameras covering the full front
and side FOV and five LiDARs. While releasing a similar amount of scenes, the
dataset is much more diverse due to the visited area being $\approx$ 15 times
bigger and the number of 3D (15M) and 2D (9.9M) annotations far exceed the
amount of 3D Labels provided by nuScenes (1.4M). Up until this date, the Waymo
Open Dataset is still being worked on with more scenes and new labels being
published such as semantic segmentations and 3D map data.

There have been other datasets, either also providing high quality data with
comparable approaches like A2D2 [13], or they focus on a very specific niche
like the View-of-Delft dataset [14], where they investigate the differences
between multi-class road user detection of 3+1D RaDAR and 64-Layer 3D-LiDAR.

The part of our dataset that is provided by our ego vehicle aims to integrate
seamlessly in this field, providing a comprehensive package of state-of-the-
art sensors to enable a wide range of applications for busses in public
transportation.

###  II-B Vehicle-Infrastructure Cooperation Datasets

With perception challenges constantly becoming harder to solve and reaching
the limitations of an ego vehicle perspective, the research area of multi-view
perception from an additional infrastructure view increases in relevance [5].
However, the lack of publicly available datasets with infrastructure
integration slows down the progression of this field significantly. The first
dataset to address this issue by combining multiple viewing angles was V2X-Sim
in 2021 [15]. With the usage of CARLA-SUMO co-simulations, they created a
synthetic dataset consisting of six RGB-, depth- and semantic cameras, one
360° LiDAR and GPS/IMU data on the vehicle side, and four of the same cameras
and the same LiDAR attached to roadside units (RSU). Due to the data being
simulated, they are able to provide rich annotations and ground truth data.

When algorithms for applications on real-world scenarios are developed,
synthetic data has clear limitations. Real world data is crucial in both
training and even more so in performance estimation. DAIR-V2X-C [16] in 2022,
with the addition of V2X-seq [17] one year later, are the first two datasets
to include vehicle-infrastructure cooperating (VIC) data. One camera, one
40-beam 360° LiDAR and a GPS/IMU were mounted onto the recording vehicle, and
one camera paired with one 300-beam 100° LiDAR were used to observe selected
intersections in Beijing.

Most recently, HoloVIC [18] was published, covering four different
intersection layouts equipped with up to twelve cameras, four fish-eye
cameras, and two LiDAR units in addition to their sensor vehicle with two
LiDARs and two cameras. This rich infrastructure setup ensures data from
multiple viewpoints simultaneously and enables cooperative perception tasks
together with their 460K 2D/3D tracked and globally associated objects.

The presented infrastructure datasets purely focus on intersections. We fill
the gap towards pulic transport and expand the scene selection to further
enable research towards autonomous bus transportation.

##  III OPNV Dataset Specification

In this section, we present the concept for our data collection. We decided
against creating another labeled dataset like these, which are presented in
sufficient detail in the Related Work section. Instead, we want to rely on
automatic ground truth generated from publicly available repositories.
Additionally, the dataset focuses on specific scenes that are underrepresented
yet vital for inclusive mobility, including infrastructure as a technological
enabler, public transportation in general, and accessibility for people with
limited mobility.

The dataset collection is designed to be an open-access resource, inviting
contributions from various institutions and researchers once we publish our
first data release, complete with all pipeline and packaging specifications.
This initiative is intended to serve as a starting point for a growing
collection that will be expanded in terms of scenes and geographical coverage
as part of academic work. In alignment with this expansion strategy, we will
introduce what we call data drops, which are predefined dataset packages. Each
data drop will be identified based on its features or scene content, ensuring
comparability and backward compatibility.

TABLE I: Ego-Vehicle Sensor Specification for Cameras including Stereo-Left (SL), Stereo-Right (SR), Back-Left (BL), Back-Right (BR), Front-Left (FL), Front-Right (FR), for LiDAR including TOP, Left (L), Right (R) and for INS. | Camera | LiDAR |  INS
---|---|---|---
|  SL, SR, BL, BR |  FL, FR |  TOP |  L, R |  GNSS/IMU
Model |  a2A1920-51gcPRO |  a2A1920-51gcPRO |  OS1 |  OS0 |  3DM-GQ7
Resolution |  1920 x 1200 |  1920 x 1200 |  1024 x 128 |  1024 x 128 |  -
Frequency |  10Hz |  10Hz |  10Hz |  10Hz |  1kHz
HFoV/VFoV |  57.6°/37.7° |  79.1°/54.3° |  360°/45° |  360°/90° |  -
Details |  Exposure Time: 800$\mu s$, Focal Length: 6mm, Aperture: f/4.0 |  Exposure Time: 800$\mu s$, Focal Length: 4mm, Aperture: f/4.0 |  Range (10%): 90m |  Range (10%): 35m |  Accuracy: 1cm RTK, 0,2° heading, 0,05° pitch/roll

###  III-A Setup

Figure 2: Sensor setup for the ego-vehicle and sensor tower, illustrated from
a bird’s-eye perspective with corresponding coordinate systems. Labels
correspond to sensor types and individual instance names.

Our initiative, known as Project AMEISE [19], originated from a university
endeavor focused on enhancing urban mobility. The project’s initial phase
involved introducing a novel bus route serviced by two distinct types of
buses. The first bus operates with SAE Level $4$ automation, offering
passengers a glimpse into the capabilities of autonomous driving technology.
The second, equipped with advanced sensor technology and displays, provides
passengers with real-time insights into what the autonomous systems can detect
and interpret, thereby making the autonomous driving experience tangible. This
sensor bus serves as the cornerstone of our dataset project’s ego-vehicle
concept. We further expanded this concept by introducing a stationary
counterpart: the sensor tower, designed to complement the data collection and
perception capabilities of the ego-vehicle. We used the Robot Operating System
(ROS) [20] and adhered to all defined conventions. Figure 2 shows a complete
position and reference description along with a listing of all sensors.

####  III-A1 Ego-vehicle (sensor bus)

The sensor arrangement for the ego-vehicle is derived from a qualitative
analysis focused on the informational needs within the vehicle cockpit, guided
by human driving behaviors in public transportation. This analysis resulted in
a design featuring ten sensors, as detailed in table I, symmetrically aligned
along the vehicle’s driving direction and approximately oriented horizontally.
Exceptions are the two LiDAR sensors at the front, one on the left and one on
the right, which are tilted $45$° towards the respective sides. Centrally
positioned within the vehicle’s sensor setup are the stereo camera and the top
LiDAR sensor, with the entire arrangement referenced to the position of the
top LiDAR. The stereo camera, consisting of two aligned and flipped monocular
cameras, faces the driving direction. It is mounted inside the bus, behind the
top area of the windshield, an already proofed placement in various
applications [21]. To reduce reflections on the windshield, black molton
fabric is placed on the dashboard below. The LiDAR, located on the front
section of the vehicle’s roof, provides a horizontal field of view (HFoV) of
$45$°. Our sensor setup does not adhere strictly to horizon alignment.
Instead, it is engineered to ensure that the LiDAR coverage extends across the
entire camera’s field, including areas above the horizon. However, given the
design of our public transportation bus, we encounter limitations similar to
those of an inclined sensor. The vehicle’s length and obstructions like the
rear-mounted air conditioning system result in a constrained field of view for
the sensor.

Compared to sedan cars, our bus’s elevated setup height introduces close-range
blind spots. To mitigate this, we equipped the vehicle with two front LiDARs
at the left and right corners, inclined both forwards and sideways by
approximately $45$°. These LiDARs, with their 90-degree HFoV, provide a direct
vertical view in front of the vehicle, effectively eliminating most shading
issues. To achieve a similar improvement in the camera system, we opted for a
smaller $4$ mm focal length, which enlarges the FoV at the expense of ranged
detail. For comprehensive lateral coverage, akin to the shoulder view, and to
complete the near-complete surround view (excluding the rear), cameras with a
$6$ mm focal length were installed to function like modern digital side
mirrors [22].

Navigation and motion sensing are realized by an INS, which combines GNSS and
Inertial Measurement Unit (IMU) data using an adaptive Kalman filter. The
system is positioned beneath the top-mounted LiDAR to optimize data
integration and accuracy. To further improve the precision of the GNSS data,
Real-Time Kinematic (RTK) positioning is employed, which utilizes RTCM
correction messages transmitted from the sensor tower.

####  III-A2 Sensor tower

Figure 3: Left: Sensor tower equipped with sensors (see table I), solar panel,
RSU and an electrical cabinet. This cabinet houses a computer, masterclock,
network switch, PLC and a 5G router. Right: Corresponding data output of the
Tower 1 camera and LiDAR, showcasing the complementary alignment of the two
sensors FOV.

The sensor tower is designed as a static counterpart to the moving ego-
vehicle, with an emphasis on simplicity to facilitate studies of
interoperability issues, such as registration. Its base consists of a modular
foundation, featuring eight 25 kg weights on a metal structure, surmounted by
a 3-meter pole. Cross struts are installed to enhance the pole’s stability.
Designed for complete autonomy, the foundation includes compartments for car
batteries, supplemented with a solar panel for power. Furthermore, the tower
is equipped with advanced communication technologies, including LTE/5G and V2X
roadside unit (RSU), to support mobile connectivity.

Atop the pole, a metal plate hosts a central 360° LiDAR (horizontal) with GNSS
antennas positioned on both sides, one for time synchronization and one for
positioning. Higher on the pole, we installed two camera-LiDAR units, each
affixed to a freely movable ball joint, allowing for adjustable sensing
directions tailored to each scene. These cameras are identical to those used
on the bus, each equipped with a 6 mm lens. For the LiDAR component, we
selected a solid-state device (SSD) to better align with the cameras’ FoV, as
illustrated in Figure 3.

In preparation for each scene recording, GNSS data are initially captured to
accurately determine the position of the tower using a precise point
positioning service [23]. The precisely determined position allows the tower
to function as an RTK base station. More detailed information on the whole
sensor setup is available in Table II.

TABLE II: Sensor Tower Specification for Cameras including Tower 1 (T1), Tower 2 (T2) for LiDAR including T1, T2, TOP and for GNSS. |  Camera | LiDAR |  GNSS
---|---|---|---
|  T1, T2 |  T1, T2 |  TOP |  RTK Base
Model |  a2A1920-51gcPRO |  Cube 1 Outdoor |  OS2 |  C099-F9P
Resolution |  1920 x 1200 |  400 x 51 |  1024 x 128 |  -
Frequency |  10Hz |  10Hz |  10Hz |  -
HFoV/VFoV |  57.6°/37.7° |  70°/30° |  360°/22,5° |  -
Details |  Exposure Time 800$\mu s$, Focal Length 6mm, Aperture f/4.0 |  Range (10%): 30m |  Range (10%): 200m |  PPP with > 1h data, accuracy variable

###  III-B Synchronization

The synchronization between the tower and the vehicle is carefully managed
through the time synchronization of their internal clocks and the joint
triggering of the sensors. We employ the Precision Time Protocol (PTP) version
2 across our network for this purpose. The network architecture includes a
dedicated PTP master clock and a PTP-enabled switch that functions as an end-
to-end transparent clock, along with sensors capable of PTP integration. For
time reference, we use UTC time provided by GPS to ensure data integrity
across our systems. Although the Inertial Navigation System (INS) on the
vehicle synchronizes independently using a dedicated GPS signal, this setup
guarantees that all system components maintain accurate time alignment.

To ensure global consistency across all sensor recordings for each frame,
triggering mechanisms were carefully implemented. The process begins with the
TOP LiDAR on the vehicle: at every 0° rotation angle pass, it generates a
square wave signal with a period of $100$ ms and a duty cycle of 50%. This
signal is then sent to a programmable logic controller (PLC), which is tasked
with distributing the signal to each camera on the vehicle, ensuring
synchronized recording events. The remaining LiDARs synchronize their 0°
rotation timestamp via the manufacturer’s software with the TOP LiDAR. For the
INS, additional triggering is unnecessary as it operates at a high frequency
of $1$ kHz, ensuring continuous and consistent data acquisition.

For the sensor tower, we generate a similar $10$ Hz signal based on the
timestamp of the TOP LiDAR’s recordings from the vehicle to trigger the
cameras. The two solid-state LiDARs on the tower cannot be externally
triggered due to product limitations; however, this is not a concern as they
continuously record at their maximum frequency, covering a field of view just
slightly exceeding that of the cameras. For the 360-degree LiDAR, product
constraints necessitated a choice between retaining PTP time synchronization
or enabling external triggering. We prioritized PTP synchronization to ensure
accurate time alignment across the dataset, given that triggering primarily
marks the start of a frame and each data point within that frame already has
its own timestamp, maintaining detailed temporal information.

###  III-C Calibration

The intrinsic calibration of the cameras was conducted using the multi-camera
calibration functionality of the Kalibr toolbox [24]. This self-calibrating
algorithm surpasses other state-of-the-art methods in accuracy and provides a
statistical quality report for each calibration session, which includes
metrics like reprojection errors. The use of a robust calibration target, the
aprilgrid [25], further enhances the precision of the calibration. Calibration
was carried out under optimal weather conditions with a high-quality DIN A0
calibration target and an exposure time of 500 $\mu s$, selected to minimize
motion blur and ensure the accuracy of the measurements. The recordings of the
calibration process and the intrinsic results, including the stereo camera
transformation and stereo-baseline, are available within the dataset.

For the extrinsic calibration of our setup, we utilized the
velo2cam_calibration suite [26], which employs a method based on extracting
reference points from a custom calibration target and determining their
optimal rigid transformation through registration. We replicated this target
using an aluminum composite panel, selected for its dimensional stability and
precise reflectivity. Calibration was selectively conducted on the camera-
LiDAR and LiDAR-LiDAR pairs necessary to define the geometry of each subsystem
comprehensively, namely the vehicle and the sensor tower. For the sensor
tower, we conducted individual extrinsic calibrations for each scene to
account for the flexible positioning of the camera-LiDAR pairs. Within each
subsystem, the TOP LiDAR was designated as the central reference point. All
transformations were calculated relative to this root, ensuring system-
specific global consistency. This methodical approach enables the accurate
mapping of each system’s geometry through the computation of two
transformation matrices as described in formula 1, where $T_{BA}$ denotes the
transformation to the TOP LiDAR $A$ from any sensor $B$ and $T_{CA}$ from any
sensor $C$.

$\displaystyle T_{BC}=T_{BA}\cdot T_{CA}^{-1}$ (1)

The transformation between the TOP LiDAR and the INS on the vehicle was
precisely estimated through physical measurements, facilitated by the INS’s
placement directly below the TOP LiDAR. For the tower, the GNSS position was
derived from CAD data. The transformation for each individual sensor to the
root $T_{\mathcal{X}B}$ is included within the dataset.

###  III-D Scene Book and Recordings

Due to the stationary nature of the sensor tower, we are constrained by
location; therefore, it is necessary to establish the tower at each designated
position separately, which is labor-intensive. In order to make the best use
of each position, we have decided to use it several times. This approach
involves selecting a single location for the sensor tower from which we will
capture a series of predefined maneuvers with the ego-vehicle, such as making
a right turn or crossing an intersection. Each maneuver will be executed and
recorded multiple times to account for variations in traffic conditions.
Subsequently, the recordings will be categorized by scenario to facilitate the
analysis of specific maneuvers, such as overtaking. From our perspective, the
positional constraint imposed by the stationary sensor tower can be leveraged
advantageously. Instead of relying on random occurrences, we can
systematically create a ”scene book” of selected situations. Starting with
maneuvers related to public transport, such as picking up and dropping off
passengers at bus stations, navigating bus-only lanes, and managing bus
terminal operations. The relocatability of the sensor tower offers significant
flexibility, allowing us to record not only at a single fixed intersection but
also on rural roads and in residential areas. However, we place a particular
emphasis on urban settings to capture a broad spectrum of traffic behaviors.

###  III-E Automatic Ground Truth

Annotating a dataset is a resource-intensive process. As an open community and
public institution, we have chosen to advance the development of real-world
datasets by offering automatically generated ground truth. To achieve this, we
established internal annotation pipelines that utilize state-of-the-art,
publicly available models to predict and calculate ground truth labels.
Offline processing has the advantage of not being constrained by the time
limitations inherent in onboard real-time applications. Moreover, we aim to
capitalize on the performance enhancements offered by technologies such as
foundation models, which efficiently produce labels offline, for instance in
semantic segmentation [27]. These labels are valuable for benchmarking other
more tailored models that perform the same tasks but are designed to operate
in real-time. Additionally, automatically derived ground truths could be
beneficial, especially if an application utilizes abstractions like Stixel
[28]. We envision that future datasets will focus more on content such as
diversity and scene variance rather than merely enriching established labels,
potentially leveraging the capabilities of new models developed from
classically annotated datasets. Compared to the expanding capabilities of AI-
aided simulations [29], where detailed environmental modeling presents
challenges, this approach could provide a corresponding solution for real-
world datasets, in which establishing reliable ground truth remains
problematic.

In line with our dataset expansion strategy, we plan to provide a training,
validation, and testing split in the ratio of $70/15/15$ %, which will be
consistently expanded with each data drop. In practical terms, this involves
using automatically generated ground truth labels for 3D bounding boxes,
employing models such as SpatialDETR [30], or for semantic segmentation,
utilizing InternImage [31].

##  IV DEVELOPMENT KIT (DATA LOADER)

In addition to facilitating data collection, we offer a Python development kit
that is open-source and available precompiled on PyPI. There are two versions:
a base data loader version, lacking assistive tools, and a full-featured
version. This data loader is specifically designed to handle our open-source
.4mse file format, which is a binary format featuring integrity checks,
streaming capabilities, and checksum validation. For compatibility, we also
provide functionality to export data to rosbag. The primary objective of this
data loader is to simplify data handling, enabling users to quickly download,
install, and explore the dataset with minimal setup. Users benefit from
straightforward intuitive bindings, such as
frame[42].vehicle.cameras.FRONT_LEFT, to access specific data points, like the
top LiDAR points, effortlessly.

Our development kit also includes assistive tools designed to expedite the
research process, addressing common needs such as transformation or
visualization. In the initial version, we offer several functions along with
additional tools.

  * •

Rectification: This tool adjusts images to correct for lens distortion or
misalignment, ensuring they represent a flat, undistorted perspective. Unlike
the standalone data loader package, the full development kit includes
additional dependencies, notably OpenCV [32], to facilitate these
enhancements. For rectification, we employ the initUndistortRectifyMap()
function from OpenCV with appropriate bindings to facilitate this process.

  * •

LiDAR-camera projection: It maps the 3D point cloud data collected by LiDAR
onto the 2D image plane of the camera (rigid transformation), creating a
unified dataset that combines the detailed imagery of the camera with the
depth measurements of the LiDAR. This includes a suite of mathematical
transformation functions that enable the alignment and calibration of the
sensors relative to each other, adhering to established methods in technical
publications [33].

  * •

LiDAR undistortion: This implementation [34] compensates for the point cloud
distortion introduced by the motion of the ego-vehicle. The individual points
of one full scan are corrected under the assumption of constantly known
velocity and rotation.

  * •

Hidden point removal: In terms of LiDAR camera projections, the shifted
positions of the cameras result in different viewing angles and FoVs. This
leads to the mathematical projection of points that are not visible in the
image. To address this, we have implemented a hidden point removal filter
using the Open3D implementation from [35]. Additionally, we have incorporated
an automatic parametrization mechanism.

##  V CONCLUSION AND OUTLOOK

In this work, we presented our current plans for a novel dataset designed to
advance the study of intelligent vehicle interoperability with infrastructure,
focusing on enhancing cooperative perception and interaction for autonomous
driving. By including data from both static and mobile agents and emphasizing
support for public transportation and inclusive design, the dataset shall help
to improve safety, efficiency, and accessibility in autonomous transport
systems. We have outlined the specifications of our hardware and primarily
presented our concepts for developing a comprehensive dataset. This dataset is
characterized by automatically generated labels and a scene-oriented approach
that emphasizes content variability.

Several aspects are still up for discussion, and timely feedback from the
community would be greatly appreciated. Some currently open questions are:

  1. 1.

What types of automatically generated ground truth data should be provided,
such as 2D/3D bounding boxes or map data, and which models are best suited for
this purpose?

  2. 2.

Accurately aligning two sensor systems globally remains a challenge due to the
lack of accurate initialization of rotation, despite RTK-GNSS providing
reliable translational data and odometry detailing both translational and
rotational changes. One approach to bridge this gap could be to find
similarities in camera or LiDAR data to continuously correct misalignment in a
global context. What are the most reliable methods for spatially synchronizing
data from two different sources?

  3. 3.

With our infrastructure-supported setup, which situations offer interesting
opportunities for dataset diversity?

  4. 4.

Are there additional features or tools that could further streamline the
process of working with the data?

Regarding data publication, the European General Data Protection Regulation
(EU) 2018/1725 has complicated the release of such a dataset, particularly in
safeguarding the privacy of recorded individuals. We are actively developing a
legal basis that would permit public institutions to publish such data. A key
component of this framework is the automated anonymization of camera data, for
which we are currently engineering a pipeline.

We plan to publish the dataset described in this paper in a subsequent
publication, where it will undergo a comprehensive analysis and evaluation.
The planned locations for this initiative are the Stuttgart region in Germany
and Clermont-Ferrand in France.

## ACKNOWLEDGMENT

We gratefully acknowledge the Ministry of Transport of Baden-Württemberg for
their funding of the AMEISE [19] project.

## References

  * [1] S. Rohrbach, “How cars should talk: 5G versus WiFi | Deutsche Telekom,” May 2019. [Online]. Available: https://www.telekom.com/en/company/details/how-cars-should-talk-5g-versus-wifi-575026
  * [2] A. Correa, R. Alms, J. Gozalvez, and et al., “Infrastructure Support for Cooperative Maneuvers in Connected and Automated Driving,” in _2019 IEEE Intelligent Vehicles Symposium (IV)_. Paris, France: IEEE, Jun. 2019, pp. 20–25. [Online]. Available: https://ieeexplore.ieee.org/document/8814044/
  * [3] “Project LUKAS,” 2023. [Online]. Available: https://projekt-lukas.de/results-00.html
  * [4] M. Buchholz, J. Muller, M. Herrmann _et al._ , “Handling Occlusions in Automated Driving Using a Multiaccess Edge Computing Server-Based Environment Model From Infrastructure Sensors,” _IEEE Intelligent Transportation Systems Magazine_ , vol. 14, no. 3, pp. 106–120, May 2022. [Online]. Available: https://ieeexplore.ieee.org/document/9481097/
  * [5] S. Masi, P. Xu, P. Bonnifait, and S.-S. Ieng, “Augmented Perception with Cooperative Roadside Vision Systems for Autonomous Driving in Complex Scenarios,” in _2021 IEEE International Intelligent Transportation Systems Conference (ITSC)_. Indianapolis, IN, USA: IEEE, Sep. 2021, pp. 1140–1146. [Online]. Available: https://ieeexplore.ieee.org/document/9564833/
  * [6] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The KITTI dataset,” _The International Journal of Robotics Research_ , vol. 32, no. 11, pp. 1231–1237, Sep. 2013. [Online]. Available: http://journals.sagepub.com/doi/10.1177/0278364913491297
  * [7] H. Caesar, V. Bankiti, A. H. Lang _et al._ , “nuScenes: A Multimodal Dataset for Autonomous Driving,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. Seattle, WA, USA: IEEE, Jun. 2020, pp. 11 618–11 628. [Online]. Available: https://ieeexplore.ieee.org/document/9156412/
  * [8] P. Sun, H. Kretzschmar, X. Dotiwalla _et al._ , “Scalability in Perception for Autonomous Driving: Waymo Open Dataset,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. Seattle, WA, USA: IEEE, Jun. 2020, pp. 2443–2451. [Online]. Available: https://ieeexplore.ieee.org/document/9156973/
  * [9] “NAVYA Autonom® Shuttle Evo.” [Online]. Available: https://www.navya.tech/en/solutions/moving-people/self-driving-shuttle-for-passenger-transportation/
  * [10] “EasyMile EZ10 passenger shuttle.” [Online]. Available: https://easymile.com/vehicle-solutions/ez10-passenger-shuttle
  * [11] M. Cordts, M. Omran, S. Ramos _et al._ , “The Cityscapes Dataset for Semantic Urban Scene Understanding,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. Las Vegas, NV, USA: IEEE, Jun. 2016, pp. 3213–3223. [Online]. Available: http://ieeexplore.ieee.org/document/7780719/
  * [12] X. Huang, P. Wang, X. Cheng _et al._ , “The ApolloScape Open Dataset for Autonomous Driving and its Application,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 42, no. 10, pp. 2702–2719, Oct. 2020, arXiv:1803.06184 [cs]. [Online]. Available: http://arxiv.org/abs/1803.06184
  * [13] J. Geyer, Y. Kassahun, M. Mahmudi _et al._ , “A2D2: Audi Autonomous Driving Dataset,” _arXiv_ , 2020, publisher: arXiv Version Number: 1. [Online]. Available: https://arxiv.org/abs/2004.06320
  * [14] A. Palffy, E. Pool, S. Baratam _et al._ , “Multi-Class Road User Detection With 3+1D Radar in the View-of-Delft Dataset,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 4961–4968, Apr. 2022. [Online]. Available: https://ieeexplore.ieee.org/document/9699098/
  * [15] Y. Li, D. Ma, Z. An _et al._ , “V2X-Sim: Multi-Agent Collaborative Perception Dataset and Benchmark for Autonomous Driving,” Jul. 2022, arXiv:2202.08449 [cs]. [Online]. Available: http://arxiv.org/abs/2202.08449
  * [16] H. Yu, Y. Luo, M. Shu _et al._ , “DAIR-V2X: A Large-Scale Dataset for Vehicle-Infrastructure Cooperative 3D Object Detection,” in _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. New Orleans, LA, USA: IEEE, Jun. 2022, pp. 21 329–21 338. [Online]. Available: https://ieeexplore.ieee.org/document/9879243/
  * [17] H. Yu, W. Yang, H. Ruan _et al._ , “V2X-Seq: A Large-Scale Sequential Dataset for Vehicle-Infrastructure Cooperative Perception and Forecasting,” May 2023, arXiv:2305.05938 [cs]. [Online]. Available: http://arxiv.org/abs/2305.05938
  * [18] C. Ma, L. Qiao, C. Zhu _et al._ , “HoloVIC: Large-scale Dataset and Benchmark for Multi-Sensor Holographic Intersection and Vehicle-Infrastructure Cooperative,” Mar. 2024, arXiv:2403.02640 [cs]. [Online]. Available: http://arxiv.org/abs/2403.02640
  * [19] “Project AMEISE,” 2024. [Online]. Available: https://ameise.wandelgesellschaft.de/
  * [20] M. Quigley, B. Gerkey, K. Conley _et al._ , “ROS: an open-source Robot Operating System,” in _ICRA Workshop on Open Source_ , 2009.
  * [21] “DISTRONIC.” [Online]. Available: https://group.mercedes-benz.com/company/magazine/technology-innovation/easy-tech-distronic.html
  * [22] “Digital vision system.” [Online]. Available: https://www.bosch-mobility.com/en/solutions/interior/digital-vision-system-cv/
  * [23] “Precise Point Positioning,” 2024. [Online]. Available: https://webapp.csrs-scrs.nrcan-rncan.gc.ca/geod/tools-outils/ppp.php?locale=en
  * [24] J. Maye, P. Furgale, and R. Siegwart, “Self-supervised calibration for robotic systems,” in _2013 IEEE Intelligent Vehicles Symposium (IV)_. Gold Coast City, Australia: IEEE, Jun. 2013, pp. 473–480. [Online]. Available: http://ieeexplore.ieee.org/document/6629513/
  * [25] E. Olson, “AprilTag: A robust and flexible visual fiducial system,” in _2011 IEEE International Conference on Robotics and Automation_. Shanghai, China: IEEE, May 2011, pp. 3400–3407. [Online]. Available: http://ieeexplore.ieee.org/document/5979561/
  * [26] J. Beltrán, C. Guindel, A. de la Escalera, and F. García, “Automatic Extrinsic Calibration Method for LiDAR and Camera Sensor Setups,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 10, pp. 17 677–17 689, Oct. 2022, arXiv:2101.04431 [cs]. [Online]. Available: http://arxiv.org/abs/2101.04431
  * [27] M. Oquab, T. Darcet, T. Moutakanni _et al._ , “DINOv2: Learning Robust Visual Features without Supervision,” _arXiv_ , 2023, publisher: [object Object] Version Number: 2. [Online]. Available: https://arxiv.org/abs/2304.07193
  * [28] D. Levi, N. Garnett, and E. Fetaya, “StixelNet: A Deep Convolutional Network for Obstacle Detection and Road Segmentation,” in _British Machine Vision Conference 2015_. Swansea: British Machine Vision Association, 2015, pp. 109.1–109.12. [Online]. Available: http://www.bmva.org/bmvc/2015/papers/paper109/index.html
  * [29] A. Hu, L. Russell, H. Yeo _et al._ , “GAIA-1: A Generative World Model for Autonomous Driving,” _arXiv_ , 2023, publisher: [object Object] Version Number: 1. [Online]. Available: https://arxiv.org/abs/2309.17080
  * [30] S. Doll, R. Schulz, L. Schneider _et al._ , “SpatialDETR: Robust Scalable Transformer-Based 3D Object Detection From Multi-view Camera Images With Global Cross-Sensor Attention,” in _Computer Vision – ECCV 2022_ , S. Avidan, G. Brostow, M. Cissé _et al._ , Eds. Cham: Springer Nature Switzerland, 2022, vol. 13699, pp. 230–245, series Title: Lecture Notes in Computer Science. [Online]. Available: https://link.springer.com/10.1007/978-3-031-19842-7\\_14
  * [31] W. Wang, J. Dai, Z. Chen _et al._ , “InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions,” in _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. Vancouver, BC, Canada: IEEE, Jun. 2023, pp. 14 408–14 419. [Online]. Available: https://ieeexplore.ieee.org/document/10203769/
  * [32] “Open Source Computer Vision Library (OpenCV),” Apr. 2024, original-date: 2012-07-19T09:40:17Z. [Online]. Available: https://github.com/opencv/opencv
  * [33] A. Geiger, F. Moosmann, O. Car, and B. Schuster, “Automatic camera and range sensor calibration using a single shot,” in _2012 IEEE International Conference on Robotics and Automation_. St Paul, MN, USA: IEEE, May 2012, pp. 3936–3943. [Online]. Available: http://ieeexplore.ieee.org/document/6224570/
  * [34] T. Renzler, M. Stolz, M. Schratter, and D. Watzenig, “Increased Accuracy For Fast Moving LiDARS: Correction of Distorted Point Clouds,” in _2020 IEEE International Instrumentation and Measurement Technology Conference (I2MTC)_. Dubrovnik, Croatia: IEEE, May 2020, pp. 1–6. [Online]. Available: https://ieeexplore.ieee.org/document/9128372/
  * [35] S. Katz, A. Tal, and R. Basri, “Direct visibility of point sets,” _ACM Transactions on Graphics_ , vol. 26, no. 3, p. 24, Jul. 2007. [Online]. Available: https://dl.acm.org/doi/10.1145/1276377.1276407


# CSTAR: Towards Compact and STructured Deep Neural Networks with Adversarial
Robustness

Huy Phan1, Miao Yin1, Yang Sui1, Bo Yuan1, Saman Zonouz2

###### Abstract

Model compression and model defense for deep neural networks (DNNs) have been
extensively and individually studied. Considering the co-importance of model
compactness and robustness in practical applications, several prior works have
explored to improve the adversarial robustness of the sparse neural networks.
However, the structured sparse models obtained by the existing works suffer
severe performance degradation for both benign and robust accuracy, thereby
causing a challenging dilemma between robustness and structuredness of compact
DNNs.

To address this problem, in this paper, we propose CSTAR, an efficient
solution that simultaneously impose Compactness, high STructuredness and high
Adversarial Robustness on the target DNN models. By formulating the
structuredness and robustness requirement within the same framework, the
compressed DNNs can simultaneously achieve high compression performance and
strong adversarial robustness. Evaluations for various DNN models on different
datasets demonstrate the effectiveness of CSTAR. Compared with the state-of-
the-art robust structured pruning, CSTAR shows consistently better
performance. For instance, when compressing ResNet-18 on CIFAR-10, CSTAR
achieves up to $20.07\%$ and $11.91\%$ improvement for benign accuracy and
robust accuracy, respectively. For compressing ResNet-18 with $16\times$
compression ratio on Imagenet, CSTAR obtains $8.58\%$ benign accuracy gain and
$4.27\%$ robust accuracy gain compared to the existing robust structured
pruning.

##  1 Introduction

Deep neural networks (DNNs) have obtained widespread popularity in various
important intelligent systems, such as autonomous driving, smart home,
computer-aided diagnosis, and so on. Despite their current unprecedented
prosperity, DNNs are still facing two severe problems, namely efficiency
challenge and security challenge, which can potentially hinder their future
success in many practical applications. Specifically, the efficiency challenge
refers to the high energy/storage/timing costs when deploying the modern
large-scale DNN models in the field, especially on resource-constrained
embedded and mobile devices. Meanwhile, the security challenge mainly refers
to the vulnerability of DNN models to the attacks launched by the malicious
adversary (Zang et al. 2020; Xie et al. 2020b; Phan et al. 2020, 2022b; Shi et
al. 2022; Phan et al. 2022a), especially to the imperceptible and high-damage
adversarial attacks.

To address these emerging challenges and further promote and democratize the
use of DNNs, the machine learning community has conducted tremendous research
activities to improve the execution efficiency and security of neural
networks. In particular, model compression and adversarial defense, as two key
techniques that can efficiently reduce the model size and enhance the model
robustness, have been extensively studied in the existing literature. More
specifically, various types of DNN compression approaches, e.g., pruning (Han
et al. 2015; Sui et al. 2021) and quantization (Gong et al. 2014), as well as
many different DNN defense methods, e.g., adversarial training (Madry et al.
2017; Wong, Rice, and Kolter 2020) and input denoising (Prakash et al. 2018),
have been developed in both academia and industry.

Co-exploring Model Compactness & Robustness. Considering the co-importance of
compactness and robustness for the DNNs used in the practical applications, an
attractive and natural thought is to develop a method that can generate neural
networks with high compactness and strong robustness simultaneously. Motivated
by this idea, several prior works have explored to examine and further improve
the robustness of the compressed DNNs. In particular, because of the
popularity of pruning in many model compression tasks, most of the existing
compactness/robustness co-exploration efforts (Sehwag et al. 2020, 2019; Ye et
al. 2019; Vemparala et al. 2021) focus on efficient approaches to generate the
robust pruned DNN models. To be specific, (Ye et al. 2019) demonstrates that
the model robustness and compactness can co-exist for neural networks via
concurrent adversarial training and weight pruning. Considering the pruning
criterion in (Sehwag et al. 2019) is designed to preserve benign accuracy
instead of improving robust accuracy, (Sehwag et al. 2020) further proposes
robust training-aware pruning to enhance both.

The Dilemma Between Robustness and Structuredness of Compact DNNs. Although
the state-of-the-art robust pruning works can simultaneously enable high
sparsity and strong robustness for the target DNN models, all of their best
performance comes from performing unstructured pruning. As extensively
reported and observed in the prior literature (Han et al. 2016; Liu et al.
2015; Zhu et al. 2020; Deng et al. 2021), the unstructured sparse models,
though can indeed exhibit high accuracy and high compression ratio, cannot
bring considerable speedup on the off-the-shelf CPU/GPU. To mitigate this
challenge, based on the observation that the structured pruning, e.g.,
channel/filter pruning, can enable the measurable acceleration, prior efforts
have also explored the robustness-aware structured pruning to strike for
simultaneous robustness, compactness and structuredness of DNN models.
Unfortunately, as reported in (Sehwag et al. 2019, 2020; Ye et al. 2019), the
existing robust structured sparse models suffer severe performance degradation
as compared to their unstructured counterparts with respect to both benign and
robust accuracy. Consequently, the challenging dilemma between robustness and
structuredness of the compact DNNs has not been well addressed yet.

Low-rank Robustness: A Structured Method Towards Compact & Robust DNNs. From
the perspective of model compression, structured pruning is not the only way
to produce structured compact models. Low-rank tensor decomposition (Denton et
al. 2014; Kim et al. 2015; Yang, Krompass, and Tresp 2017; Pan et al. 2019;
Yin et al. 2021a, 2022), as another type of popular model compression
technique, can also ensure that the compressed model exhibits the desired high
structuredness for the practical speedup. To be specific, different from
structured pruning that removes the entire channels/filters to enable the
structured sparsity, low-rank tensor decomposition explores the structure-
level model redundancy and factorizes the original dense model into multiple
tensor cores. Because those decomposed tensor cores are still in the dense
format and their involved computations are entirely based on the general
matrix multiplication (GEMM), considerable acceleration can be observed and
measured on the off-the-shelf CPU/GPU (Kim et al. 2015; Zhang et al. 2015b,
a). Evidently, such hardware-friendly low-rankness-based compactness, if can
further co-exist with the adversarial robustness, will be attractive for
compact and robust DNN model design.

Questions to be Answered. Despite the promising potentials of robust tensor
decomposed models, several important questions need to be answered and
addressed. To be specific, what is the suitable perspective to connect and
unify the low-rankness and robustness of a DNN model? Once we can formulate a
unified problem, what is the efficient approach to simultaneously impose low-
rankness and robustness with a high compression ratio, benign accuracy, and
robust accuracy? Also, considering the conventional tensor decomposition
requires a tedious manual rank selection process, which will be very time-
consuming if integrated with defense methods such as adversarial training, is
there any efficient automatic tensor rank determination scheme?

Technical Preview and Contributions. To answer these questions and deliver the
promise of robust low-rank DNN models, in this paper we study and develop an
efficient solution, namely CSTAR, that can simultaneously impose the desired
high low-rankness-based Compactness, high STructuredness and high Adversarial
Robustness on the target DNN models. By formulating the low-rankness and
robustness requirement within the same framework as well as determining the
tensor ranks in a global way, the compressed DNNs can achieve high compression
performance and strong adversarial robustness at the same time. Overall, the
contributions of this paper are summarized as follows:

  * •

We propose a framework that formulates the low-rankness and adversarial
robustness requirement to a unified constrained optimization problem. Based on
that, we then propose an adversarial training process that can gradually
impose the desired low-rankness on the DNN model, thereby simultaneously
ensuring high compression performance and adversarial robustness.

  * •

We propose a low-cost automatic rank selection scheme that can select the
desired tensor ranks for each layer in a very convenient way. Such selection
can be dynamically adjusted during the compression procedure, thereby avoiding
the tedious manual rank selection and meanwhile ensuring high performance.

  * •

We evaluate the performance of the CSTAR solution for various DNN models on
different datasets. Compared with the state-of-the-art robust structured
pruning methods, our proposed approach shows consistently better performance.
For instance, when compressing ResNet-18 on the CIFAR-10 dataset, CSTAR can
achieve up to $20.07\%$ and $11.91\%$ increase for benign accuracy and robust
accuracy, respectively. On the Imagenet dataset, for compressing ResNet-18
with $16\times$ compression ratio, CSTAR can obtain $8.58\%$ benign accuracy
gain and $4.27\%$ robust accuracy gain as compared to the existing robust
structured pruning method.

##  2 Background and Preliminaries

###  2.1 Tensor Decomposition for Model Compression

Tensor decomposition, as a type of low-rank decomposition approach, has been
widely used in many DNN model compression works. According to tensor theory,
there exist various types of tensor decomposition approaches, such as Tucker
(Tucker 1963), CP (Harshman et al. 1970), Tensor Train (Oseledets 2011) and
Tensor Ring (Zhao et al. 2016). Without loss of generality, in this paper, we
adopt Tucker-2 decomposition (Kim et al. 2015) as the underlying low-rank
tensor method for model compression.

Tensor Contraction. Tucker-2 decomposition can be compactly represented via
using tensor contraction. In general, tensor contraction can be performed
between any two tensors that have at lease one matched dimension. For
instance, consider two tensors
$\boldsymbol{\mathcal{A}}\in\mathbb{R}^{d_{1}\times d_{2}\times n}$ and
$\boldsymbol{\mathcal{B}}\in\mathbb{R}^{n\times d_{3}\times d_{4}}$, the
output tensor $\boldsymbol{\mathcal{C}}$ of size $\mathbb{R}^{d_{1}\times
d_{2}\times d_{3}\times d_{4}}$ after the tensor contraction can be calculated
as follow:

$\boldsymbol{\mathcal{C}}_{(a_{1},a_{2},b_{1},b_{2})}=\boldsymbol{\mathcal{A}}\times^{3}_{1}\boldsymbol{\mathcal{B}}=\sum_{i=1}^{n}\boldsymbol{\mathcal{A}}_{(a_{1},a_{2},i)}\boldsymbol{\mathcal{B}}_{(i,b_{1},b_{2})}.$
(1)

Tucker-2 Tensor Decomposition. With the notation of tensor contraction, given
the weight tensor of convolutional layer
$\boldsymbol{\mathcal{W}}\in\mathbb{R}^{O\times I\times K\times K}$ as well as
the corresponding input tensor $\boldsymbol{\mathcal{X}}\in\mathbb{R}^{I\times
H\times W}$ and output tensor $\boldsymbol{\mathcal{Y}}\in\mathbb{R}^{O\times
H^{\prime}\times W^{\prime}}$, $\boldsymbol{\mathcal{W}}$ can be factorized
via Tucker-2 decomposition as:

$\boldsymbol{\mathcal{W}}=\boldsymbol{\mathcal{U}}^{1}\times^{2}_{2}~{}\boldsymbol{\mathcal{U}}^{2}\times^{2}_{2}~{}\boldsymbol{\mathcal{G}},$
(2)

where $\boldsymbol{\mathcal{U}}^{1}\in\mathbb{R}^{O\times R_{1}}$,
$\boldsymbol{\mathcal{U}}^{2}\in\mathbb{R}^{I\times R_{2}}$, and
$\boldsymbol{\mathcal{G}}\in\mathbb{R}^{R_{1}\times R_{2}\times K\times K}$.
$R_{1}$ and $R_{2}$ denote the Tucker-2 tensor ranks. Then, the output tensor
$\boldsymbol{\mathcal{Y}}=\boldsymbol{\mathcal{U}}^{1}\times^{2}_{1}\boldsymbol{\mathcal{T}}^{2}$
can be calculated via tensor contraction-based computations:

$\displaystyle\boldsymbol{\mathcal{T}}^{1}=\boldsymbol{\mathcal{U}}^{2}\times^{1}_{1}\boldsymbol{\mathcal{X}},~{}~{}~{}\boldsymbol{\mathcal{T}}^{2}=\sum_{p=1}^{K}\sum_{q=1}^{K}\boldsymbol{\mathcal{G}}_{(:,:,p,q)}\times^{2}_{1}\boldsymbol{\mathcal{T}}^{1}$
(3)

where $\boldsymbol{\mathcal{T}}^{1}\in\mathbb{R}^{R_{2}\times H\times W}$, and
$\boldsymbol{\mathcal{T}}^{2}\in\mathbb{R}^{R_{1}\times H^{\prime}\times
W^{\prime}}$ are the intermediate results. Notice that tensor contraction has
been well supported by PyTorch and TensorFlow platforms via using
torch.tensordot and tf.tensordot.

###  2.2 Adversarial Training for DNN Defense

Projected Gradient Decent (PGD). PGD is a very popular adversarial examples
generation approach (Madry et al. 2017) because of its high-quality and fast
generation process. In general, PGD first calculates the gradient w.r.t. the
benign input data $x$ and then updates the input data in the direction that
maximizes the loss function $\mathcal{L}$. The resulting adversarial example
$x_{adv}$ is then projected to the maximum allowed perturbation $\Delta$ via
using projection operator $\boldsymbol{\Pi}$. Such process is repeated $k$
times to obtain a strong example:

$\displaystyle\boldsymbol{\mathcal{X}}_{adv}^{N+1}$
$\displaystyle=\boldsymbol{\Pi}_{\Delta}(\boldsymbol{\mathcal{X}}_{adv}^{N}+\epsilon\cdot\text{sign}(\nabla\mathcal{L}(\boldsymbol{\mathcal{W}},\boldsymbol{\mathcal{X}}_{adv}^{N},\boldsymbol{y})),$
(4)

where $\boldsymbol{\mathcal{X}}_{adv}^{N+1}$ is the set of adversarial
examples at the $N$-th iteration, $\epsilon$ is the step size, and
$\boldsymbol{y}$ is set of the labels.

Adversarial Training. Adversarial training (AT) (Madry et al. 2017) is one of
the most effective methods to protect DNNs against adversarial attacks. The
key idea of adversarial training is to generate adversarial examples during
the training process to make the model can better fit for adversarial
examples. In general, adversarial training aims to solve the following min-max
optimization problem:

$\min_{\boldsymbol{\mathcal{W}}}~{}\max_{\delta\in\boldsymbol{\Delta}}~{}\mathcal{L}(\boldsymbol{\mathcal{W}},\boldsymbol{\mathcal{X}}+\delta,\boldsymbol{y}),$
(5)

where $\boldsymbol{\mathcal{X}}$ denotes the set of benign inputs, and
$\mathcal{L}$ is the loss function. Here the inner maximization problem can be
solved by the above described PGD.

##  3 Related Works

Model Compression with Adversarial Training. To date several prior works have
explored the efficient integration of model compression to the adversarial
training process. To be specific, (Sehwag et al. 2019; Ye et al. 2019; Sehwag
et al. 2020; Vemparala et al. 2021; Xie et al. 2020a; Guo et al. 2018; Madaan,
Shin, and Hwang 2020; Rakin et al. 2019; Özdenizci and Legenstein 2021)
investigate the robustness-aware pruning to achieve high model sparsity and
adversarial robustness. Also, (Fu et al. 2021; Lin, Gan, and Han 2019) propose
several approaches to develop low-bit precision robust models. In addition,
(Gui et al. 2019) studies the adversarial robust model with the combination of
sparsity, low bit-precision and matrix factorization. However, most of these
works are built either on unstructured sparsity or complicated hybrid-
precision schemes (e.g, 4-8 bits), which can only be supported by the
specialized hardware, thereby severely limiting their practical deployment on
the off-the-shelf CPUs/GPUs. To date, only (Sehwag et al. 2019; Ye et al.
2019; Sehwag et al. 2020) provide the structured pruning solutions for
adversarial robust DNN models that can exhibit practical speedup. However, the
corresponding benign and robust accuracy is not satisfied.

Low-rank Matrix & Tensor Decomposition. Similar to structured pruning, low-
rank decomposition is another powerful structuredness-ensuring compression
approach that can bring considerable speedup. In general, low-rank
decomposition can be realized via either matrix decomposition or tensor
decomposition. Matrix decomposition (Tai et al. 2015; Li and Shi 2018; Xu et
al. 2019; Yang et al. 2020; Idelbayev and Carreira-Perpinán 2020; Xu et al.
2019) uses SVD to factorize the large matrix to small matrix components. In
particular, when the objective is the 4-D weight tensor of convolutional
layer, it has to first reshapes the 4-D tensor to 2-D matrix, and then
performs decomposition to obtain small 2-D matrices. Evidently, such
flattening strategy can not fully exploit the inherent spatial information of
the 4-D weight tensor, thereby causing limited performance. Motivated by these
limitations, Tensor decomposition, such as Tucker (Kim et al. 2015), Tensor
Train (Oseledets 2011) and Tensor Ring (Zhao et al. 2016), which can directly
factorize the original high-order tensor in the high-dimensional space, have
been studied to facilitate DNN compression. Because tensor decomposition can
fully exploit and leverage the rich spatial correlation and information of
weight tensor in the high-order tensor space, it can bring impressive
compression performance (Tai et al. 2015; Kim et al. 2015; Li and Shi 2018;
Wang et al. 2018)

Rank Selection. Determining the proper rank is very critical for low-rank
decomposed DNN models. To date, most of the existing low-rank DNN compression
approaches (Kim et al. 2015; Zhao et al. 2016; Yang, Krompass, and Tresp 2017;
Xu et al. 2019; Idelbayev and Carreira-Perpinán 2020; Yin et al. 2021b) select
the rank in a manual way. However, because each layer needs to be assigned
with one or more rank values, such heuristic searching is a very tedious and
time-consuming procedure. Recently, some works have begun to study efficient
automatic rank determination. In (Li et al. 2021; Hajimolahoseini et al.
2021), progressive search-based methods are proposed to identify proper low
matrix rank (Hajimolahoseini et al. 2021) and low tensor rank (Li et al.
2021). However, this strategy is performance-limited because it still
partially needs manual selection (Hajimolahoseini et al. 2021) or costly
iterative searching and sampling (Li et al. 2021). In addition, (Gusak et al.
2019; Ye et al. 2018) propose to automatically select the rank in a layer-wise
way. However, such strategy is not the ideal solution because of the
inevitable high cost: (Gusak et al. 2019) needs expensive Bayesian calculation
plus setting additional auxiliary hyper-parameters during the searching, and
(Ye et al. 2018) needs many rounds of enumeration to search for the proper
rank values. Moreover, determining the rank layer by layer, in principle, is
not a promising solution since it may not bring the globally optimal rank
determination for the entire model.

Advantages of Our Proposed CSTAR. Compared with the above discussed related
works, CSTAR enjoys the following benefits:

  * •

Practical Measurable Speedup. Compared with most existing works on joint
compression and adversarial training via unstructured pruning (Ye et al. 2019;
Sehwag et al. 2020), low-bit precision (Fu et al. 2021; Lin, Gan, and Han
2019) or hybrid combination (Gui et al. 2019), CSTAR can bring practical
speedup on the off-the-shelf CPUs/GPUs because of its strong model
structuredness provided by low-rankness (see Section 5).

  * •

Low-Cost Automatic Rank Selection. The rank selection of CSTAR is naturally
integrated to the compression process and automatically learned from data, and
hence it has very low cost without any expensive enumeration or searching
operation that prior works (Li et al. 2021; Hajimolahoseini et al. 2021; Gusak
et al. 2019) need. Meanwhile, the ranks for all the layers are jointly
determined in a global way (see Appendix).

  * •

High Benign and Robust Accuracy. Compared with the state-of-the-art structured
pruning-based adversarial training methods (Ye et al. 2019; Sehwag et al.
2020) that can also bring practical speedup, CSTAR shows very significant
performance improvement in terms of benign accuracy and robust accuracy across
different models on various datasets (see Section 5).

##  4 Our Proposed Method

Problem Formulation. CSTAR aims to co-explore adversarial robustness and low-
rankness on DNN models to achieve high compression ratio, high benign
accuracy, and high robust accuracy with practical speedup simultaneously. To
achieve this goal, a suitable perspective that can properly connect and unify
the adversarial robustness and low-rankness should be identified. Considering
adversarial training is currently the most important and popular adversarial
defense approach, we formulate our goal as the following optimization problem:

$\displaystyle\begin{split}&\min_{\boldsymbol{\mathcal{W}}_{i}}~{}\max_{\delta\in\boldsymbol{\Delta}}~{}\mathcal{L}(\boldsymbol{\mathcal{W}},\boldsymbol{\mathcal{X}}+\delta,\boldsymbol{y}),\\\
&\text{s.t.}~{}~{}~{}\texttt{rank}(\boldsymbol{\mathcal{W}}_{i})\leq\textbf{t}_{i},~{}~{}~{}i=1,\dots,N,\end{split}$
(6)

where $N$ denotes the number of layers, $\texttt{rank}(\cdot)$ is the function
that calculates the Tucker-2 ranks $\textbf{r}=[r_{1},r_{2}]$ of the input
tensor, and $\textbf{t}=[t_{1},t_{2}]$ denotes the target ranks that can
satisfy the compression ratio. Here $\textbf{r}\leq\textbf{t}$ operation means
element-wise comparison.

Challenges to be Addressed. In general, solving problem 6 is non-trivial yet
challenging. More specifically, two critical issues need to be resolved:

Figure 1: Proposed singular value-based automatic tensor rank selection. Here
the rank selection can vary as training progresses.

Challenge #1: How should we optimize problem 6 that is constrained by the non-
differentiable
$\texttt{rank}(\boldsymbol{\mathcal{W}}_{i})\leq\textbf{t}_{i}$?

Analysis. A straightforward way is to directly decompose the pre-trained
$\\{\boldsymbol{\mathcal{W}}_{i}\\}$ into the suitable low-rank format that
satisfies the rank constraint, and then fine-tune the entire decomposed model.
Though this direct strategy can indeed bring the desired low-rankness, such
straightforward and explicit decomposition on the high-accuracy model will
probably cause considerable approximation error. Consider that the common pre-
trained CNN models typically lack sufficient low-rank property, the
corresponding approximation error and its resulting benign and robust accuracy
loss, consequently, will be too huge to be recovered after fine-tuning.

1

⬇

1

2def decompose(W, r1, r2):

3#Input: Weight tensor W, ranks r1, r2.

4#Ouput: Decomposed U1, U2, G tensors.

5W = W.flatten(start_dim=2)

6t1 = W.flatten(start_dim=1)

7U1, sigma1, V1 = torch.svd(t1)

8t2 = W.permute(1,0,2).flatten(start_dim=1)

9U2, sigma2, V2 = torch.svd(t2)

10G = torch.tensordot(U2,W,dims=([0],[1]))

11G = torch.tensordot(U1,G,dims=([0],[1]))

12U1,U2,G = U1[:,:r1], U2[:,:r2], G[:r1,:r2]

13return U1, U2, G

14def recover(U1, U2, G):

15#Input: Decomposed U1, U2, G tensors.

16#Output: Recovered low-rank weight tensor W.

17W = torch.tensordot(U2,G,dims=([1],[1]))

18W = torch.tensordot(U1,W,dims=([1],[1]))

19O, I, KK = W.size()

20K = int(KK ** 0.5)

21return W.view(O, I, K, K)

Alg. 1 Tucker-2 projection for solving Eqn. 12

Our Proposed Solution. To address this challenge, we propose to gradually
impose the desired low-rankness on the DNN model during the adversarial
training procedure. To be specific, the to-be-compressed model is first
trained in the full-rank format but gradually exhibits low-rank characteristic
as well as adversarial robustness. Our rationale for this indirect training
strategy lies in the belief that the smooth and “soft” transition from full-
rank to low-rank format, intuitively, can better preserve benign and robust
accuracy than the direct and “hard” decomposition. Motivated by this
philosophy, we re-formulate problem 6 via adding the low-rankness requirement
as a soft regularization term:

$\displaystyle\begin{split}\min_{\boldsymbol{\mathcal{W}}_{i},\boldsymbol{\mathcal{Z}}_{i}\in\mathcal{S}}&~{}\max_{\delta\in\boldsymbol{\Delta}}~{}\mathcal{L}(\boldsymbol{\mathcal{W}}_{i},\boldsymbol{\mathcal{X}}+\delta,\boldsymbol{y}),\\\
\text{s.t.}~{}~{}~{}\boldsymbol{\mathcal{W}}_{i}&=\boldsymbol{\mathcal{Z}}_{i},~{}~{}~{}i=1,\dots,N,\end{split}$
(7)

where $\boldsymbol{\mathcal{Z}}_{i}$ is the introduced auxiliary variables and
$\mathcal{S}=\\{\boldsymbol{\mathcal{W}}_{i}~{}|~{}\texttt{rank}(\boldsymbol{\mathcal{W}}_{i})\leq\textbf{t}_{i}\\}$.
Then, we further reformulate the problem via using the augmented Lagrangian:

$\begin{split}\min_{\boldsymbol{\mathcal{W}}_{i},\boldsymbol{\mathcal{Z}}_{i}\in\mathcal{S}}~{}\max_{\boldsymbol{\mathcal{M}}_{i},\delta\in\boldsymbol{\Delta}}~{}\mathcal{L}(\boldsymbol{\mathcal{W}}_{i},\boldsymbol{\mathcal{X}}+\delta,\boldsymbol{y})\\\
+\frac{\rho}{2}\sum_{i=1}^{N}\left(\left\|\boldsymbol{\mathcal{W}}_{i}-\boldsymbol{\mathcal{Z}}_{i}+\boldsymbol{\mathcal{M}}_{i}\right\|_{F}^{2}+\|\boldsymbol{\mathcal{M}}_{i}\|_{F}^{2}\right),\end{split}$
(8)

where $\boldsymbol{\mathcal{M}}_{i}$ are dual multipliers and $\rho>0$ is the
augmented parameter. After that, problem 8 can be solved via Dykstra’s
alternating projection method (Bauschke and Borwein 1994), a primal-dual
optimization algorithm that is well-suited for multi-term minimization. To be
specific, the iterative updates are performed as:

Update $\\{\boldsymbol{\mathcal{W}}_{i}\\}$ using SGD. In this step, we
optimize the sub-problem as:

$\min_{\\{\boldsymbol{\mathcal{W}}_{i}\\}}\underbrace{\max_{\delta\in\boldsymbol{\Delta}}\mathcal{L}(\\{\boldsymbol{\mathcal{W}}_{i}\\},\boldsymbol{\mathcal{X}}+\delta,\boldsymbol{y})}_{\text{Adversarial
Training}}+\underbrace{\frac{\rho}{2}\sum_{i=1}^{N}\left\|\boldsymbol{\mathcal{W}}_{i}-\boldsymbol{\mathcal{Z}}_{i}+\boldsymbol{\mathcal{M}}_{i}\right\|_{F}^{2}}_{\text{Low-
rank Regularization}}.$ (9)

The first term in Eq. 9 is to find adversarial examples, and it can be
realized via using PGD described in Sec. 2.2. The second term is to gradually
impose low-rank property onto $\\{\boldsymbol{\mathcal{W}}_{i}\\}$. Since both
terms are differentiable, we can iteratively solve Eq. 9 using stochastic
gradient descent (SGD) with learning rate $\alpha$:

$\displaystyle\begin{split}\boldsymbol{\mathcal{W}}_{i}\leftarrow\boldsymbol{\mathcal{W}}_{i}&-\alpha\cdot\nabla_{\boldsymbol{\mathcal{W}}_{i}}\left[\mathcal{L}(\boldsymbol{\mathcal{W}}_{i},\boldsymbol{\mathcal{X}}+\delta,\boldsymbol{y})\right]\\\
&-\alpha\cdot\rho\left(\|\boldsymbol{\mathcal{W}}_{i}-\boldsymbol{\mathcal{Z}}_{i}+\boldsymbol{\mathcal{M}}_{i})\right).\end{split}$
(10)

Update $\\{\boldsymbol{\mathcal{Z}}_{i}\\}$ using Projection. In this step,
$\\{\boldsymbol{\mathcal{Z}}_{i}\\}$ can be updated as:

$\displaystyle\boldsymbol{\mathcal{Z}}_{i}\leftarrow\operatorname*{argmin}_{\boldsymbol{\mathcal{Z}}_{i}\in\mathcal{S}}~{}\left\|\boldsymbol{\mathcal{W}}_{i}-\boldsymbol{\mathcal{Z}}_{i}+\boldsymbol{\mathcal{M}}_{i}\right\|_{F}^{2}.$
(11)

More specifically, problem 11 can be solved via projecting
$\boldsymbol{\mathcal{Z}}_{i}$ onto set $\mathcal{S}$ as:

$\displaystyle\boldsymbol{\mathcal{Z}}_{i}\leftarrow\textbf{Prj}(\boldsymbol{\mathcal{W}}_{i}+\boldsymbol{\mathcal{M}}_{i}),$
(12)

where the projection truncates the ranks of
$\boldsymbol{\mathcal{W}}_{i}+\boldsymbol{\mathcal{M}}_{i}$ to the desired
target ranks $\textbf{t}_{i}$. Algorithm 1 describes the details of this low-
rank projection.

Challenge #2: How should we properly select the tensor ranks in a convenient
way?

Comp. | LWM | LWM | RAP | RAP | CSTAR
---|---|---|---|---|---
Ratio | Filter Pruning | Column Pruning | Filter Pruning | Column Pruning | Low-Rank Decomp.
VGG-16 (Pre-train 80.08 / 44.10)
$2\times$ | 73.69 / 44.05 | 77.31 / 44.96 | 76.85 / 46.32 | 79.25 / 45.30 | 79.71 / 47.23
$4\times$ | 63.76 / 38.93 | 70.46 / 41.54 | 71.48 / 42.59 | 75.55 / 43.97 | 79.64 / 47.06
$8\times$ | 54.30 / 34.45 | 63.69 / 38.35 | 62.48 / 37.58 | 69.54 / 41.52 | 79.23 / 46.42
$16\times$ | 45.77 / 30.37 | 55.29 / 35.01 | 53.72 / 33.72 | 60.55 / 37.36 | 77.82 / 46.13
$32\times$ | 40.17 / 27.35 | 47.94 / 31.71 | 43.78 / 29.34 | 52.62 / 34.03 | 75.82 / 45.43
$64\times$ | 10.00 / 10.00 | 40.01 / 29.35 | 10.00 / 10.00 | 45.08 / 32.49 | 70.77 / 43.13
ResNet-18 (Pre-train 83.23 / 49.78)
$2\times$ | 83.62 / 45.47 | 83.57 / 44.91 | 82.68 / 48.85 | 82.51 / 49.45 | 83.86 / 50.06
$4\times$ | 79.10 / 46.49 | 80.54 / 46.79 | 79.78 / 46.78 | 81.81 / 46.76 | 83.22 / 49.48
$8\times$ | 70.16 / 42.41 | 75.23 / 44.46 | 71.82 / 43.17 | 76.89 / 45.57 | 82.30 / 48.81
$16\times$ | 61.31 / 37.12 | 68.55 / 40.86 | 64.39 / 38.01 | 70.18 / 41.78 | 80.90 / 47.36
$32\times$ | 51.59 / 32.43 | 59.20 / 36.60 | 54.31 / 33.32 | 62.68 / 37.15 | 77.73 / 46.47
$64\times$ | 42.36 / 28.52 | 50.92 / 32.16 | 46.97 / 29.41 | 52.01 / 32.48 | 72.08 / 44.39

Table 1: Benign accuracy / PGD-50 robust accuracy (%) on CIFAR-10 with
Standard AT.

1 Input: Dataset $\boldsymbol{\mathcal{D}}$, pre-trained weight
$\boldsymbol{\mathcal{W}}$, comp. ratio $r$, low-rank reg. steps $T_{1}$,
fine-tune steps $T_{2}$.

2 Output: Fine-tuned
$\\{\boldsymbol{\mathcal{U}}^{1}_{i},\boldsymbol{\mathcal{U}}^{2}_{i},\boldsymbol{\mathcal{G}}_{i}\\}$.

3
$\\{\boldsymbol{\mathcal{Z}}_{i}\\}\leftarrow\\{\boldsymbol{\mathcal{W}}_{i}\\}$;
$\\{\boldsymbol{\mathcal{M}}_{i}\\}\leftarrow\textbf{0}$;

4 for  _$t=1$ to $T_{1}$_ do $\triangleright$ low-rank regularization

5 update $\\{\boldsymbol{\mathcal{W}}_{i}\\}$ via Eq. 10;

6 update $\\{\boldsymbol{\mathcal{Z}}_{i}\\}$ via Eq. 12 and Alg. 1;

7
$\boldsymbol{\mathcal{M}}_{i}\leftarrow\boldsymbol{\mathcal{M}}_{i}+\boldsymbol{\mathcal{W}}_{i}-\boldsymbol{\mathcal{Z}}_{i};$

8$\\{\boldsymbol{\mathcal{U}}^{1}_{i},\boldsymbol{\mathcal{U}}^{2}_{i},\boldsymbol{\mathcal{G}}_{i}\\}\leftarrow\texttt{decompose}(\\{\boldsymbol{\mathcal{W}}_{i}\\})$
via Alg. 1;

9 for  _$t=1$ to $T_{2}$_ do $\triangleright$ fine-tune

10
$\boldsymbol{\mathcal{X}},\boldsymbol{y}\leftarrow\texttt{sample\\_batch}(\boldsymbol{\mathcal{D}})$;

11
$\boldsymbol{\mathcal{X}}_{adv}\leftarrow\texttt{PGD}(\boldsymbol{\mathcal{X}},\boldsymbol{y})$
via Eq. 4;

12
$\hat{\boldsymbol{y}}\leftarrow\texttt{conv}(\\{\boldsymbol{\mathcal{U}}^{1}_{i},\boldsymbol{\mathcal{U}}^{2}_{i},\boldsymbol{\mathcal{G}}_{i}\\},\boldsymbol{\mathcal{X}}_{adv})$
via Eq. 3;

13 loss $\leftarrow$ cross_entropy($\hat{\boldsymbol{y}},\boldsymbol{y}$);

14
update($\\{\boldsymbol{\mathcal{U}}^{1}_{i},\boldsymbol{\mathcal{U}}^{2}_{i},\boldsymbol{\mathcal{G}}_{i}\\}$,
loss);

15

Alg 2 The overall procedure of CSTAR

A.T. | Friendly AT (Zhang et al. 2020) |  | TRADES (Zhang et al. 2019) |  | MART (Wang et al. 2019)
---|---|---|---|---|---
C.R. | RAP | CSTAR |  | RAP | CSTAR |  | RAP | CSTAR
| Pre-train 85.86 / 38.24 |  | Pre-train 78.45 / 47.43 |  | Pre-train 73.97 / 45.61
$2\times$ | 84.94 / 39.34 | 85.94 / 40.14 |  | 78.47 / 46.92 |  80.00 / 44.81 |  | 75.40 / 44.49 | 76.02 / 47.98
$4\times$ | 83.24 / 37.10 | 85.86 / 40.29 |  | 75.13 / 44.27 | 80.20 / 44.91 |  | 71.23 / 44.60 | 77.08 / 44.87
$8\times$ | 79.06 / 33.89 | 85.49 / 39.68 |  | 69.82 / 39.58 | 79.48 / 44.94 |  | 65.43 / 43.04 | 76.36 / 44.99
$16\times$ | 74.06 / 29.01 | 85.02 / 38.70 |  | 65.72 / 36.00 | 78.99 / 44.38 |  | 57.74 / 38.57 | 74.57 / 46.25
$32\times$ | 63.41 / 22.62 | 83.57 / 37.72 |  | 56.47 / 30.75 | 77.49 / 42.59 |  | 50.00 / 35.58 | 71.94 / 46.72
$64\times$ | 54.30 / 20.59 | 81.32 / 35.44 |  | 49.76 / 29.67 | 73.11 / 42.44 |  | 42.40 / 31.76 | 66.33 / 44.95

Table 2: Performance of VGG-16 on CIFAR-10 using different adversarial
training approaches. Here C.R. means compression ratio, and A.T. means
adversarial training.

|  | Methods / Compression Ratios
---|---|---
Model Arch. | Pre-Train |  | RAP
---
8x
| CSTAR
---
8x
| RAP
---
16x
| CSTAR
---
16x
| RAP
---
32x
| CSTAR
---
32x
DenseNet-121 | 80.91 / 45.36 | 71.74 43.66 | 78.87 / 48.07 | 57.42 / 36.23 | 74.91 / 45.38 | 36.79 / 26.33 | 63.12 / 37.71
GoogleNet | 81.85 / 48.18 | 73.28 / 42.74 | 79.59 / 47.34 | 62.03 / 36.67 | 76.16 / 44.13 | 49.27 / 30.49 | 62.06 / 35.29
VGG-19 | 74.64 / 43.33 | 65.94 / 40.79 | 73.60 / 45.95 | 57.35 / 35.95 | 73.88 / 45.87 | 47.45 / 31.73 | 72.69 / 45.59
ResNet-34 | 82.64 / 47.85 | 79.05 / 47.34 | 83.52 / 50.35 | 73.93 / 44.62 | 81.95 / 48.69 | 66.45 / 40.13 | 79.63 / 48.24
MobileNetV2 | 84.62 / 45.15 | 62.98 / 38.56 | 83.03 / 41.54 | 50.70 / 33.19 | 66.30 / 41.72 | 35.96 / 26.42 | 63.88 / 40.18

Table 3: Performance of benign / PGD-50 robust accuracy (%) on CIFAR-10 using
a wide range of model architectures with Standard AT. Here C.R. means
compression ratio.

| Top-1 Accuracy |  | Top-5 Accuracy
---|---|---|---
C.R. | RAP | CSTAR | $\boldsymbol{\Delta}$ |  | RAP | CSTAR | $\boldsymbol{\Delta}$
| Pre-train 50.75 / 26.57 |  | Pre-train 75.25 / 51.56
$2\times$ | 43.40 / 22.22 | 51.36 / 25.75 | +7.96 / +3.53 |  | 68.69 / 45.16 | 75.28 / 51.42 | +6.59 / +6.26
$4\times$ | 35.29 / 17.58 | 41.11 / 21.05 | +5.82 / +3.47 |  | 60.32 / 37.71 | 66.60 / 43.28 | +6.28 / +5.57
$6\times$ | 29.40 / 14.50 | 34.92 / 17.35 | +5.52 / +2.85 |  | 53.06 / 31.95 | 59.92 / 37.41 | +6.86 / +5.46
$8\times$ | 23.24 / 11.72 | 30.29 / 14.84 | +7.05 / +3.12 |  | 45.31 / 26.88 | 54.60 / 33.10 | +9.29 / +6.22
$10\times$ | 19.09 / 9.72 | 28.91 / 14.10 | +9.82 / +4.38 |  | 39.41 / 23.14 | 52.82 / 31.86 | +13.41 / +8.72
$16\times$ | 9.34 / 5.05 | 17.92 / 9.32 | +8.58 / +4.27 |  | 22.93 / 13.20 | 37.83 / 22.22 | +14.90 / +9.02

Table 4: Performance of benign / robust accuracy (%) for ResNet-18 on ImageNet
dataset using Fast AT (Wong, Rice, and Kolter 2020). Here C.R. means
compression ratio.

Analysis. Because 1) the rank values directly determine the model capacity,
computational cost, and memory footprint; and 2) different layers exhibit
different low-rankness characteristics, a proper rank selection scheme is very
critical and important to achieve a high-performance low-rank DNN model.
Unfortunately, as analyzed in Section 3, to date the rank selection for the
tensor decomposed models are still performed in a costly and time-consuming
way. In particular, consider each low-rank layer in modern DNNs may have
multiple to-be-set tensor ranks, e.g., Tucker-2 format needs to configure two
rank values per layer, the existing rank selection scheme will become even
more expensive.

Our Proposed Solution. To overcome this challenge, we propose a low-cost
automatic rank selection scheme for low-rank adversarial robust DNN models.
Our key idea is to leverage the singular value information to globally select
the suitable tensor rank for each individual layer. To be specific,
considering mathematically the magnitude of each singular value strongly
represents the importance of the corresponding rank component, e.g., with the
sorted singular values $\sigma^{1}_{i}$ for $\boldsymbol{\mathcal{U}}^{1}$,
$\boldsymbol{\mathcal{U}}^{1}_{k}=\sigma^{1}_{1}u^{1}_{1}v^{1T}_{1}+...+\sigma^{1}_{k}u^{1}_{k}v^{1T}_{k}$
is the best rank-$k$ approximation for $\boldsymbol{\mathcal{U}}^{1}$. Hence
we can sort all the singular values from the entire model and select the
largest ones and their corresponding rank components under the target
compression budget. As illustrated in Fig. 1, this global comparison and
selection strategy can determine the suitable tensor rank values for different
layers in an automatic way, thereby significantly facilitating the overall
compression workflow. Noticed that here the desired singular values have
already been available after low-rank projection (see Line 6 and 8 in Alg. 1),
so it is naturally integrated to the entire compression procedure with very
low computing overhead. Also, as showed in Fig. 1, such a sorting-and-
selection process can be performed in each epoch to dynamically adjust the
rank selection as training progresses.

Fine-tuning. Upon the iterative update finishes, the model has been gradually
imposed on the desired low-tensor-rank property, and then we can decompose it
with the automatically selected ranks and fine-tune it in the low-rank format.
Algorithm 2 summarizes the CSTAR procedure.

##  5 Experiments

To demonstrate the effectiveness of CSTAR, we evaluate various model
architectures with multiple adversarial training objectives on different
datasets. To be specific, we evaluate the benign and robust accuracy of
VGG-16/19, ResNet-18/34, DenseNet-121, GoogleNet and MobileNet-V2. In
addition, several adversarial training approaches, including the Standard PGD
(Madry et al. 2017), Friendly AT (Zhang et al. 2020), TRADES (Zhang et al.
2019), MART (Wang et al. 2019), and Fast AT (Wong, Rice, and Kolter 2020) are
adopted to show the generality of our approach. The models are compressed on
CIFAR-10/100 and ImageNet datasets with different compression ratios ranging
from $2\times$ to $64\times$. On CIFAR-10/100 $L_{\infty}$ is selected with
$\Delta=8/255$, and PGD with step size $\alpha=2/255$ serves to generate
adversarial examples. Here the number of PGD iterations for training and
testing are 10 and 50, respectively. Other hyper-parameter settings can be
found in the Appendix.

Comparison with Robust Aware Pruning (RAP) on CIFAR-10/100. Table 1 shows our
comparison with the RAP approach (Ye et al. 2019). For fair comparison, we
perform compression on the same pre-train models with the same number of
training epochs. In addition, we also report the performance of Least Weight
Magnitude (LWM)-based approach (Sehwag et al. 2019) in this table. It is seen
that our proposed approach consistently outperforms RAP with different
compression ratios on both benign accuracy and robust accuracy. In particular,
with $64\times$ compression, our CSTAR can achieve up to $20.07\%$ and
$11.91\%$ improvement for benign accuracy and robust accuracy, respectively.
For the results of CIFAR-100 dataset, see the Appendix.

Comparison with HYDRA on CIFAR-10. HYDRA (Sehwag et al. 2020) is AN
importance-score based adversarial pruning approach. Notice that because 1)
HYDRA does not release code for their structured pruning version; and 2) HYDRA
only reports very limited data point (VGG-16 on CIFAR-10) for its performance
with structured pruning, in Table 5 we only list the benign and robust
accuracy of our approach with the C.Rs. (2$\times$ and 10$\times$) that HYDRA
reports. From this table it is seen that our method shows very significant
performance improvement over HYDRA. With 10 $\times$ C.R., our CSTAR approach
can achieve $60.99\%$ of benign accuracy and $28.48\%$ robust accuracy
increase.

C.R. | HYDRA | CSTAR | $\boldsymbol{\Delta}$
---|---|---|---
$2\times$ | 52.90 / 38.00 | 80.00 / 44.81 | +27.10 / +6.81
$10\times$ | 18.30 / 16.70 | 79.29 / 45.18 | +60.99 / +28.48

Table 5: HYDRA vs CSTAR on CIFAR-10 using TRADES. Here C.R. means compression
ratio.

Stronger Baseline: RAP or HYDRA? Comparing Table 1 and Table 5, it is seen
that with the same (2$\times$) or even higher (16$\times$) compression ratio,
RAP shows much higher benign and robust accuracy than HYDRA on the same pre-
trained VGG-16 model. Such phenomenon shows that the least weight magnitude-
based structured pruning seems more suitable for obtaining adversarial
robustness than the importance score-based structured pruning. In addition,
considering HYDRA only reports its structured pruning performance for VGG-16
on CIFAR-10; therefore, in our following experiments involved with other
models and dataset, we select RAP (Column pruning) as the baseline.

Generalization Across Different Adversarial Training Approaches. To
demonstrate the generality of CSTAR, we evaluate its performance for
compressing VGG-16 model via using other three popular adversarial training
methods (FAT, TRADES and MART). As shown in Table 2, our solution can
consistently outperform RAP method with different compression ratios on both
benign and robust accuracy.

Generalization Across Different Model Architectures. We also evaluate the
generality of CSTAR on different CNN architectures. As shown in Table 3, our
CSTAR solution achieves both higher benign and robust accuracy than RAP across
multiple models sizes and architectures.

C.R. | VGG-16 / CPU |  | ResNet-18 / GPU
---|---|---|---
RAP | CSTAR |  | RAP | CSTAR
P.T. | 9.98ms | 9.98ms |  | 0.0801ms | 0.0801ms
2$\times$ | 9.98ms | 6.75ms/$\downarrow$1.48$\times$ |  | 0.0801ms | 0.0562ms/$\downarrow$1.43$\times$
4$\times$ | 9.98ms | 4.38ms/$\downarrow$2.28$\times$ |  | 0.0801ms | 0.0437ms/$\downarrow$1.83$\times$

Table 6: Inference time per image of executing CSTAR models for CIFAR-10 data
compared with unstructured pruning.

Performance on ImageNet Dataset. Table 4 shows the performance of our approach
for compressing ResNet-18 on ImageNet dataset. It is seen that in this
challenging task CSTAR achieves consistently better performance than RAP. For
instance, CSTAR can obtain $8.58\%$ benign accuracy gain and $4.27\%$ robust
accuracy gain with $16\times$ C.R..

Runtime Speedup on CPU/GPU. We also measure the speedup of executing CSTAR
model on the off-the-shelf hardware. Table 6 reports the inference time with
batch size = 1 for the original and compressed models on both CPU (AMD Ryzen 9
5900HX) and GPU (NVIDIA GeForce RTX 3090). It is seen that the low-rank
structuredness of CSTAR indeed brings practical speedup.

##  6 Conclusion

This paper proposes CSTAR, an efficient approach that can simultaneously
impose high low-rankness-based compactness, high structuredness, and high
adversarial robustness on the DNN models. By using a unified framework to
gradually impose the low-tensor-rankness and automatically select the rank
setting during the adversarial training procedure, CSTAR can make the
compressed DNNs achieve high compression performance and strong adversarial
robustness at the same time.

#### Acknowledgments

This work was partially supported by National Science Foundation under Grant
CCF-1955909, RTML, CMMI and CPS programs.

## References

  * Bauschke and Borwein (1994) Bauschke, H. H.; and Borwein, J. M. 1994.  Dykstra’ s alternating projection algorithm for two sets.  _Journal of Approximation Theory_ , 79(3): 418–443.
  * Deng et al. (2021) Deng, C.; Sui, Y.; Liao, S.; Qian, X.; and Yuan, B. 2021.  GoSPA: An Energy-efficient High-performance Globally Optimized SParse Convolutional Neural Network Accelerator.  In _2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA)_ , 1110–1123. IEEE.
  * Denton et al. (2014) Denton, E. L.; Zaremba, W.; Bruna, J.; LeCun, Y.; and Fergus, R. 2014.  Exploiting linear structure within convolutional networks for efficient evaluation.  _Advances in neural information processing systems_ , 27.
  * Fu et al. (2021) Fu, Y.; Yu, Q.; Li, M.; Chandra, V.; and Lin, Y. 2021.  Double-win quant: Aggressively winning robustness of quantized deep neural networks via random precision training and inference.  In _International Conference on Machine Learning_ , 3492–3504. PMLR.
  * Gong et al. (2014) Gong, Y.; Liu, L.; Yang, M.; and Bourdev, L. 2014.  Compressing deep convolutional networks using vector quantization.  _arXiv preprint arXiv:1412.6115_.
  * Gui et al. (2019) Gui, S.; Wang, H.; Yu, C.; Yang, H.; Wang, Z.; and Liu, J. 2019.  Model compression with adversarial robustness: A unified optimization framework.  _arXiv preprint arXiv:1902.03538_.
  * Guo et al. (2018) Guo, Y.; Zhang, C.; Zhang, C.; and Chen, Y. 2018.  Sparse dnns with improved adversarial robustness.  _Advances in neural information processing systems_ , 31.
  * Gusak et al. (2019) Gusak, J.; Kholiavchenko, M.; Ponomarev, E.; Markeeva, L.; Blagoveschensky, P.; Cichocki, A.; and Oseledets, I. 2019.  Automated multi-stage compression of neural networks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops_ , 0–0.
  * Hajimolahoseini et al. (2021) Hajimolahoseini, H.; Rezagholizadeh, M.; Partovinia, V.; Tahaei, M.; Awad, O. M.; and Liu, Y. 2021.  Compressing Pre-trained Language Models using Progressive Low Rank Decomposition.
  * Han et al. (2016) Han, S.; Liu, X.; Mao, H.; Pu, J.; Pedram, A.; Horowitz, M. A.; and Dally, W. J. 2016.  EIE: Efficient inference engine on compressed deep neural network.  _ACM SIGARCH Computer Architecture News_ , 44(3): 243–254.
  * Han et al. (2015) Han, S.; Pool, J.; Tran, J.; and Dally, W. J. 2015.  Learning both weights and connections for efficient neural networks.  _arXiv preprint arXiv:1506.02626_.
  * Harshman et al. (1970) Harshman, R. A.; et al. 1970.  Foundations of the PARAFAC procedure: Models and conditions for an” explanatory” multimodal factor analysis.
  * Idelbayev and Carreira-Perpinán (2020) Idelbayev, Y.; and Carreira-Perpinán, M. A. 2020.  Low-rank compression of neural nets: Learning the rank of each layer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 8049–8059.
  * Kim et al. (2015) Kim, Y.-D.; Park, E.; Yoo, S.; Choi, T.; Yang, L.; and Shin, D. 2015.  Compression of deep convolutional neural networks for fast and low power mobile applications.  _arXiv preprint arXiv:1511.06530_.
  * Li and Shi (2018) Li, C.; and Shi, C. 2018.  Constrained optimization based low-rank approximation of deep neural networks.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 732–747.
  * Li et al. (2021) Li, N.; Pan, Y.; Chen, Y.; Ding, Z.; Zhao, D.; and Xu, Z. 2021.  Heuristic rank selection with progressively searching tensor ring network.  _Complex & Intelligent Systems_, 1–15.
  * Lin, Gan, and Han (2019) Lin, J.; Gan, C.; and Han, S. 2019.  Defensive quantization: When efficiency meets robustness.  _arXiv preprint arXiv:1904.08444_.
  * Liu et al. (2015) Liu, B.; Wang, M.; Foroosh, H.; Tappen, M.; and Pensky, M. 2015.  Sparse convolutional neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 806–814.
  * Madaan, Shin, and Hwang (2020) Madaan, D.; Shin, J.; and Hwang, S. J. 2020.  Adversarial neural pruning with latent vulnerability suppression.  In _International Conference on Machine Learning_ , 6575–6585. PMLR.
  * Madry et al. (2017) Madry, A.; Makelov, A.; Schmidt, L.; Tsipras, D.; and Vladu, A. 2017.  Towards deep learning models resistant to adversarial attacks.  _arXiv preprint arXiv:1706.06083_.
  * Oseledets (2011) Oseledets, I. V. 2011.  Tensor-train decomposition.  _SIAM Journal on Scientific Computing_ , 33(5): 2295–2317.
  * Özdenizci and Legenstein (2021) Özdenizci, O.; and Legenstein, R. 2021.  Training adversarially robust sparse networks via Bayesian connectivity sampling.  In _International Conference on Machine Learning_ , 8314–8324. PMLR.
  * Pan et al. (2019) Pan, Y.; Xu, J.; Wang, M.; Ye, J.; Wang, F.; Bai, K.; and Xu, Z. 2019.  Compressing recurrent neural networks with tensor ring for action recognition.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, 4683–4690.
  * Phan et al. (2022a) Phan, H.; Shi, C.; Xie, Y.; Zhang, T.; Li, Z.; Zhao, T.; Liu, J.; Wang, Y.; Chen, Y.; and Yuan, B. 2022a.  RIBAC: Towards Robust and Imperceptible Backdoor Attack against Compact DNN.  In _European Conference on Computer Vision_ , 708–724. Springer.
  * Phan et al. (2020) Phan, H.; Xie, Y.; Liao, S.; Chen, J.; and Yuan, B. 2020.  CAG: A Real-time Low-cost Enhanced-robustness High-transferability Content-aware Adversarial Attack Generator.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, 5412–5419.
  * Phan et al. (2022b) Phan, H.; Xie, Y.; Liu, J.; Chen, Y.; and Yuan, B. 2022b.  Invisible and Efficient Backdoor Attacks for Compressed Deep Neural Networks.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 96–100. IEEE.
  * Prakash et al. (2018) Prakash, A.; Moran, N.; Garber, S.; DiLillo, A.; and Storer, J. 2018.  Deflecting adversarial attacks with pixel deflection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 8571–8580.
  * Rakin et al. (2019) Rakin, A. S.; He, Z.; Yang, L.; Wang, Y.; Wang, L.; and Fan, D. 2019.  Robust sparse regularization: Simultaneously optimizing neural network robustness and compactness.  _arXiv preprint arXiv:1905.13074_.
  * Sehwag et al. (2019) Sehwag, V.; Wang, S.; Mittal, P.; and Jana, S. 2019.  Towards compact and robust deep neural networks.  _arXiv preprint arXiv:1906.06110_.
  * Sehwag et al. (2020) Sehwag, V.; Wang, S.; Mittal, P.; and Jana, S. 2020.  Hydra: Pruning adversarially robust neural networks.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 7\.
  * Shi et al. (2022) Shi, C.; Zhang, T.; Li, Z.; Phan, H.; Zhao, T.; Wang, Y.; Liu, J.; Yuan, B.; and Chen, Y. 2022.  Audio-domain position-independent backdoor attack via unnoticeable triggers.  In _Proceedings of the 28th Annual International Conference on Mobile Computing And Networking_ , 583–595.
  * Sui et al. (2021) Sui, Y.; Yin, M.; Xie, Y.; Phan, H.; Aliari Zonouz, S.; and Yuan, B. 2021.  CHIP: CHannel Independence-based Pruning for Compact Neural Networks.  _Advances in Neural Information Processing Systems_ , 34.
  * Tai et al. (2015) Tai, C.; Xiao, T.; Zhang, Y.; Wang, X.; et al. 2015.  Convolutional neural networks with low-rank regularization.  _arXiv preprint arXiv:1511.06067_.
  * Tucker (1963) Tucker, L. R. 1963.  Implications of factor analysis of three-way matrices for measurement of change.  _Problems in measuring change_ , 15: 122–137.
  * Vemparala et al. (2021) Vemparala, M.-R.; Fasfous, N.; Frickenstein, A.; Sarkar, S.; Zhao, Q.; Kuhn, S.; Frickenstein, L.; Singh, A.; Unger, C.; Nagaraja, N.-S.; et al. 2021.  Adversarial Robust Model Compression Using In-Train Pruning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 66–75.
  * Wang et al. (2018) Wang, W.; Sun, Y.; Eriksson, B.; Wang, W.; and Aggarwal, V. 2018.  Wide compression: Tensor ring nets.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 9329–9338.
  * Wang et al. (2019) Wang, Y.; Zou, D.; Yi, J.; Bailey, J.; Ma, X.; and Gu, Q. 2019.  Improving adversarial robustness requires revisiting misclassified examples.  In _International Conference on Learning Representations_.
  * Wong, Rice, and Kolter (2020) Wong, E.; Rice, L.; and Kolter, J. Z. 2020.  Fast is better than free: Revisiting adversarial training.  _arXiv preprint arXiv:2001.03994_.
  * Xie et al. (2020a) Xie, H.; Xiang, X.; Liu, N.; and Dong, B. 2020a.  Blind Adversarial Training: Balance Accuracy and Robustness.  _arXiv preprint arXiv:2004.05914_.
  * Xie et al. (2020b) Xie, Y.; Shi, C.; Li, Z.; Liu, J.; Chen, Y.; and Yuan, B. 2020b.  Real-time, universal, and robust adversarial attacks against speaker recognition systems.  In _ICASSP 2020-2020 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , 1738–1742. IEEE.
  * Xu et al. (2019) Xu, Y.; Li, Y.; Zhang, S.; Wen, W.; Wang, B.; Dai, W.; Qi, Y.; Chen, Y.; Lin, W.; and Xiong, H. 2019.  Trained rank pruning for efficient deep neural networks.  In _2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS)_ , 14–17. IEEE.
  * Yang et al. (2020) Yang, H.; Tang, M.; Wen, W.; Yan, F.; Hu, D.; Li, A.; Li, H.; and Chen, Y. 2020\.  Learning low-rank deep neural networks via singular vector orthogonality regularization and singular value sparsification.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , 678–679.
  * Yang, Krompass, and Tresp (2017) Yang, Y.; Krompass, D.; and Tresp, V. 2017.  Tensor-train recurrent neural networks for video classification.  In _International Conference on Machine Learning_ , 3891–3900. PMLR.
  * Ye et al. (2018) Ye, J.; Wang, L.; Li, G.; Chen, D.; Zhe, S.; Chu, X.; and Xu, Z. 2018.  Learning compact recurrent neural networks with block-term tensor decomposition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 9378–9387.
  * Ye et al. (2019) Ye, S.; Xu, K.; Liu, S.; Cheng, H.; Lambrechts, J.-H.; Zhang, H.; Zhou, A.; Ma, K.; Wang, Y.; and Lin, X. 2019.  Adversarial Robustness vs. Model Compression, or Both?  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 111–120.
  * Yin et al. (2021a) Yin, M.; Liao, S.; Liu, X.-Y.; Wang, X.; and Yuan, B. 2021a.  Towards Extremely Compact RNNs for Video Recognition with Fully Decomposed Hierarchical Tucker Structure.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 12085–12094.
  * Yin et al. (2021b) Yin, M.; Sui, Y.; Liao, S.; and Yuan, B. 2021b.  Towards efficient tensor decomposition-based dnn model compression with optimization framework.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 10674–10683.
  * Yin et al. (2022) Yin, M.; Sui, Y.; Yang, W.; Zang, X.; Gong, Y.; and Yuan, B. 2022.  HODEC: Towards Efficient High-Order DEcomposed Convolutional Neural Networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 12299–12308.
  * Zang et al. (2020) Zang, X.; Xie, Y.; Chen, J.; and Yuan, B. 2020.  Graph universal adversarial attacks: A few bad actors ruin graph learning models.  _arXiv preprint arXiv:2002.04784_.
  * Zhang et al. (2019) Zhang, H.; Yu, Y.; Jiao, J.; Xing, E.; El Ghaoui, L.; and Jordan, M. 2019.  Theoretically principled trade-off between robustness and accuracy.  In _International Conference on Machine Learning_ , 7472–7482. PMLR.
  * Zhang et al. (2020) Zhang, J.; Xu, X.; Han, B.; Niu, G.; Cui, L.; Sugiyama, M.; and Kankanhalli, M. 2020\.  Attacks which do not kill training make adversarial learning stronger.  In _International Conference on Machine Learning_ , 11278–11287. PMLR.
  * Zhang et al. (2015a) Zhang, X.; Zou, J.; He, K.; and Sun, J. 2015a.  Accelerating very deep convolutional networks for classification and detection.  _IEEE transactions on pattern analysis and machine intelligence_ , 38(10): 1943–1955.
  * Zhang et al. (2015b) Zhang, X.; Zou, J.; Ming, X.; He, K.; and Sun, J. 2015b.  Efficient and accurate approximations of nonlinear convolutional networks.  In _Proceedings of the IEEE Conference on Computer Vision and pattern Recognition_ , 1984–1992.
  * Zhao et al. (2016) Zhao, Q.; Zhou, G.; Xie, S.; Zhang, L.; and Cichocki, A. 2016.  Tensor ring decomposition.  _arXiv preprint arXiv:1606.05535_.
  * Zhu et al. (2020) Zhu, C.; Huang, K.; Yang, S.; Zhu, Z.; Zhang, H.; and Shen, H. 2020.  An efficient hardware accelerator for structured sparse convolutional neural networks on FPGAs.  _IEEE Transactions on Very Large Scale Integration (VLSI) Systems_ , 28(9): 1953–1965.