=4 10000 9000 7000 0 =4 10000 9000 7000 0

# CoRD: Converged RDMA Dataplane for High-Performance Clouds

Maksym Planeta
Barkhausen Institut  Jan Bierbaum
TU Dresden  Michael Roitzsch
Barkhausen Institut  Hermann Härtig
TU Dresden

###### Abstract

High-performance networking is often characterized by kernel bypass which is
considered mandatory in high-performance parallel and distributed
applications. But kernel bypass comes at a price because it breaks the
traditional OS architecture, requiring applications to use special APIs and
limiting the OS control over existing network connections. We make the case,
that kernel bypass is _not_ mandatory. Rather, high-performance networking
relies on multiple performance-improving techniques, with kernel bypass being
the least effective. CoRD removes kernel bypass from RDMA networks, enabling
efficient OS-level control over RDMA dataplane.

##  1 Introduction

High Performance Computing (HPC) and Cloud architectures are converging: the
former is advancing towards greater elasticity and higher resource utilization
[20, 14, 24, 91], while the latter is increasingly geared towards handling
high/performance workloads [13, 22, 71]. However, current HPC Cloud systems
remain partitioned: vast sections lack high-performance networking support,
with only a few segments offering such capabilities [56, 5, 62]. As we push
for a holistic integration of HPC and Cloud systems, we realize that their
network architectures must converge as well. In this paper, we show how
operating systems (OS) can help in such integration by making high-performance
networks more accessible. We believe this approach can significantly enhance
performance within Cloud environments.

A classical OS provides a socket-based abstraction layer between the
applications and the underlying hardware. This layer allows “rich and robust”
OS services [2, p. 228], like application portability, resource scheduling,
and security policy enforcement. High-performance networks, often represented
by Remote Direct Memory Access (RDMA) networks [18, 34] subvert this layered
OS architecture by granting user-level applications direct access to network
devices (NICs). These applications circumvent OS services (including the
network stack), instead they come with their own NIC drivers and perform most
of the scheduling and resource management themselves [18, 61]. This approach
forces the OS to rely on the NIC to implement security (IOMMUs [72, 80] or
VLANs [75] are not sufficient) and resource-sharing policies [90, 44, 30]. As
a result, the OS becomes just an over-engineered bootloader [27, 37, 28, 87].

In response to the demand for flexibility and OS control over high-performance
network communication, the OS community has proposed several _dataplane_
architectures [9, 89, 18, 30, 52, 68, 39, 83, 59]. But few of these solutions
have seen widespread adoption [18, 39]. The novelty of these architectures and
their lack of interoperability with socket-based communication, poses a
significant entry barrier, restricting their appeal for conventional Cloud
applications. Interestingly, dataplane architectures and traditional RDMA
networks share common mechanisms and principles, suggesting a potential for
compatibility. So, could we introduce minor, strategic adjustments to the
widely-used RDMA networks to make them as flexible and controllable by the OS
as socket-based networks? This approach could bridge the current gap and
facilitate the convergence of HPC and Cloud architectures.

In this paper, we identify a set of key properties that boost the performance
of RDMA networks over socket-based networks. Among these, kernel-bypass is
often considered the cornerstone [26, 21, 45, 51]. However, we challenge this
prevalent belief and show that kernel-bypass is not indispensible to high-
performance applications. Inspired by these insights, we propose the Converged
RDMA Dataplane (CoRD), a strategically altered version of the traditional RDMA
architecture that directs the dataplane through the kernel. This pivotal shift
gives the kernel full control over RDMA communication, enabling it to enforce
security policies [55, 76], provide virtualization [54, 30], control resources
on a fine-grained level [44, 30], and facilitate application observability
[3]. Preliminary measurements reveal that CoRD retains the performance for
large-scale applications, effectively converging RDMA and socket-based
networks without forsaking the intrinsic advantages of either.

##  2 RDMA performance breakdown

Compared to traditional socket-based networks, we identify three software
properties enabling high/performance communication over RDMA networks:
_kernel-bypass_ (or _OS-bypass_), _zero-copy_ , and _polling_ 111There are
also hardware techniques, like _one-sided (or RDMA) communication_ ,
_offloading_ , and _lossless congestion control_.. Each technique improves
performance in a different way, can be employed independently of the rest, and
comes with its own limitations. This diversity suggests that these techniques
may not be equally performance critical in every use case.

_Kernel-bypass_ allows the application to access the device directly, avoiding
overhead for crossing the user-kernel boundary. Zero-copy enables the NIC to
access the application memory directly, avoiding overhead for copying the
message content between kernel and user memory as done with read and write
system calls. With polling, the application continuously checks the NIC’s
message queues to avoid the overhead of interrupt processing, which happens in
the epoll system call. Together, these techniques form an API [53, 89, 15]
that is very different from the traditional socket-based API.

A simple example illustrates the role of each of the aforementioned
techniques: A high-performance perftest microbenchmark measures (on our local
system _L_ , see section 5) point-to-point send latency and throughput for
different message sizes. To quantify the contribution of each technique, we
modified the benchmark to “remove” individual techniques from the typical RDMA
mode of operation and measure how the benchmark performance changes. To
emulate the removal of zero-copy, our perftest makes an extra memory copy when
sending or receiving a message. We emulate the removal of kernel-bypass by
adding a getppid system call. Finally, we instruct the benchmarks to wait for
an interrupt from the NIC instead of busy-polling, an existing, but rarely
used feature in RDMA networks.

.39  $16\text{\,}\mathrm{B}$ $4\text{\,}\mathrm{KiB}$ $1\text{\,}\mathrm{MiB}$
Baseline $0.99$ $1.95$ $86$ No kernel bypass $1.06$ $1.95$ $86$ No busy-
polling $4.69$ $4.16$ $90$ No zero copy (ZC) $1.03$ $2.31$ $229$ .57 No kernel
bypassNo busy-pollingNo ZCNo kernel bypassNo busy-pollingNo ZC1.4 Gb/s100
Gb/sBaseline BW0.00.51.016 B16 MiBMessage Size, bRelative Bandwidth

Figure 1: Latency () Figure 2: Bandwidth Figure 3: “Removing” performance-
improving techniques compared to having all techniques active

Figure 3 shows the results of our experiment. Removing any of the three
techniques has a significant influence on small-message throughput (fig. 3),
because this task is CPU-bound. Even the baseline variant (with all techniques
active) achieves only $1.4\text{\,}\mathrm{G}\mathrm{bit}\mathrm{/}\mathrm{s}$
out of the theoretical maximum of
$100\text{\,}\mathrm{G}\mathrm{bit}\mathrm{/}\mathrm{s}$. On the other hand,
large message throughput is obstructed significantly only without zero-copy.
For latency (fig. 3), removing busy-polling adds significant overhead from
additional interrupt processing, especially for small messages, although the
absolute overhead stays the same even for very large messages. Removing zero-
copy adds additional latency proportional to message size (up to
$140\text{\,}\mathrm{\SIUnitSymbolMicro}\mathrm{s}\mathrm{/}\mathrm{Mi}\mathrm{B}$)
because of additional data movement. Finally, removing kernel bypass adds
small constant overhead with minimal overall impact.

We make the following observations from this experiment: Zero copy appears to
be the largest contributor to performance, especially for large messages.
Kernel-bypass is crucial for throughput and latency only for very small
messages. Even then, polling is more important than kernel-bypass. These
observations identify kernel-bypass as the least performance-critical
technique.

##  3 Converged RDMA Dataplane

_Kernel_ _User_ NIC2 4 1 3  (a) Sockets

_Kernel_ _User_ NIC4 3  (b) RDMA

_Kernel_ _User_ NIC4 3  (c) CoRD

Figure 4:  Socket, RDMA, and CoRD control- and data-planes. To send a message
over a socket, the application invokes the high-level kernel network stack (
1). The kernel (fig. 4(a)) or the application (figs. 4(b) and 4(c)) request
the NIC driver to send the packet ( 3), which copies the data from pinned ( )
memory to send it over the network ( 4).

Our proposed architecture, CoRD integrates OS-level control into high-
performance user-level communication dataplane. This section first provides an
overview of both socket- and RDMA-based communication (fig. 4). Then, it
explains how CoRD modifies the RDMA dataplane to enable control over the RDMA
connections.

In a traditional socket-based API, as shown in fig. 4(a), communication starts
with the application making a system call ( 1, e.g., send) which instructs the
kernel to dispatch a message over an existing connection (e.g., a TCP socket).
Subsequently, the kernel copies ( 2) the message content into NIC-accessible
(_pinned_ , ) memory. Finally, the kernel _triggers_ the NIC ( 3) to start the
transmission of message packets ( 4). This entire operation is managed by a
complex kernel-level network stack, designed to maintain scalability and
improve resource utilization across a large number of user applications.
However, this design philosophy inadvertently hampers network performance.

In contrast to socket-based communication, RDMA networks (fig. 4(b)) delegate
a significant part of control to the application, allowing it to manage pinned
memory ( ) and access the NIC directly ( 3). The NIC can retrieve the message
content directly from the user memory ( 4), thus bypassing the kernel. Note,
that while this requires some network stack functionality at the application
level, a majority of responsibilities, such as managing concurrent users, are
offloaded to the NIC. Despite the complexity added to the application, this
architecture ultimately yields the lowest possible communication latency.

CoRD principally mimics the classical RDMA architecture, retaining a majority
of its performance attributes. The RDMA application keeps its responsibilities
under CoRD, including the management of pinned memory. The drivers at the user
level in RDMA (fig. 4(b)) and at the kernel level in CoRD (fig. 4(c)) are
largely equivalent, thereby ensuring a lightweight and transparently
interchangeable layer between the application and the NIC. The critical
distinction, however, lies in the NIC access. CoRD mandates a system call from
the application ( 3), thereby preventing it from directly accessing the NIC.

In CoRD each dataplane operation goes through the kernel, but the kernel
processing is kept to a minimum. In contrast to socket-based communication,
CoRD only permits the enforcement of lightweight, non-blocking policies to
maintain its high network performance. However, this limitation is not
significant, as without system-wide resource sharing CoRD can afford to uphold
simpler, more streamlined policies, which we refer to as _CoRD policies_. CoRD
policies should be powerful enough to implement QoS, security, and isolation
similarly to other dataplane interception techniques [44, 30]. CoRD inevitably
adds a constant per-message latency from user-kernel switching. The overhead
from the enforcement of CoRD policies depends greatly on the specifics of the
implemented functionality, which are beyond the scope of this paper.

##  4 Implementation

The goal of our prototype implementation is to measure the cost of CoRD’s
architecture, which we expect to be higher than the bare minimum overhead of
user-kernel transition (see section 2). We modify the user- and kernel-level
mlx5 device drivers for NVIDIA’s ConnectX-series NICs. These drivers are used
by the _ibverbs_ library [53] to funnel data-plane operations through the
kernel. The ibverbs API is the “narrow waist” of many high-performance user-
level network stacks [44], which use it either directly [58, 4, 42] or through
a higher-level API [40, 8, 60]. As this is a preliminary study, we do not
attempt any performance optimizations, so the resulting overhead represents
the worst case.

The ibverbs API defines _control- and data-plane_ operations: Control-plane
operations set up communication; they register device-accessible memory,
create communication endpoints (_queue pairs_), etc. Data-plane operations
initiate message sending (ibv_post_send) and receiving (ibv_post_recv) or
perform a non-blocking check if any of these operations have completed
(ibv_poll_cq). Control-plane operations require kernel support, whereas
dataplane operations usually bypass the kernel.

Applications request ibverbs control-plane operations using ioctl system call.
Unfortunately, the arguments to ibverbs calls are complex data structures that
must be _serialized_ and _deserialized_ to go through the ioctl interface.
These operations add to the system-call overhead but are not a performance
critical for control-plane operations. On a positive side, the ioctl code path
allows the OS to intercept control-plane ibverbs calls and thus enforce
security [76, 55], isolation [64], and resource management policies [81]. CoRD
extends this existing functionality to data-plane operations.

After entering the kernel, the data-plane operation reaches the kernel-level
device driver, which normally provide high-performance networking inside the
Linux kernel [57, 74]. We change the driver minimally, so it can work with the
ibverbs objects created by the user application. Overall we added or modified
~250 lines in the kernel-level driver and ~20 lines in the user-level driver.
Except for the kernel interposition, the ibverbs API remains unchanged and
introduces no interrupts or asynchronous invocations on the data plane. In
other words, without CoRD policies, the only overhead comes from crossing the
user-kernel boundary.

Our implementation depends on the NIC being able to access an application’s
virtual memory as applications pass message buffers to the NIC by virtual
address. This is a common feature for high-performance NICs [36, 53], which,
in contrast to other approaches (e.g., io_uring [39]), relieves the kernel
from virtual-to-device address translations on the critical path. If the
application passes an invalid address, the NIC returns an error but does not
access any memory that was not explicitly provided to the application.

##  5 Evaluation

Our goal is to evaluate how CoRD impacts raw communication performance as well
as end-to-end application performance. We evaluate two systems. First, a
system _L_ , comprising two nodes, each with Intel i5-4590 4-core CPUs and
$200\text{\,}\mathrm{G}\mathrm{bit}\mathrm{/}\mathrm{s}$ NVIDIA ConnectX-6 Dx
RoCE NICs communicating at up to
$100\text{\,}\mathrm{G}\mathrm{bit}\mathrm{/}\mathrm{s}$ due to motherboard
limitations. The nodes are back-to-back connected. The system runs vanilla
Linux 6.0.0-rc7 with or without our patch to support CoRD in the mlx5 driver.
We also disable Turbo Boost, pin all the benchmark processes to dedicated
cores, and set the CPU power governor to the highest performance mode.

Second, is a remote two-node system _A_ deployed in the Azure Cloud. We use
virtualized HB120 instances with two 64-core AMD EPYC 7V73X CPUs (only 120
cores passed to VM) and virtualized
$200\text{\,}\mathrm{G}\mathrm{bit}\mathrm{/}\mathrm{s}$ NVIDIA ConnectX-6
Infiniband NICs. In both systems we disable KPTI [29, 38], an expensive
kernel-level Meltdown attack [48] mitigation, because modern CPUs do not need
it. Specifically, system A’s CPU mitigates Meltdown in hardware. We run the
benchmarks in the same way as on system L, except for not being able to
disable dynamic frequency scaling due to the cloud provider policy.

0.000.250.500.751.001.25Read/RCWrite/RCSend/RCSend/UDCommunication ModeLatency
Overhead (µs)BP $\rightarrow$ CoRDCoRD $\rightarrow$ BPCoRD $\rightarrow$ CoRD
Figure 5: Latency overhead on system L (baseline in fig. 3) when communicating
over different transports (RC/UD) using one-sided (Read/Write) or two-sided
(Send) communication. Client and server can independently run bypass (BP) or
CoRD (CD), “$\rightarrow$” indicates the direction of communication (from
client to server).

We first measured the overhead CoRD adds to the point-to-point message
latency, using the perftest 4.5 benchmark suite [66]. The communication can go
either through Reliable Connection (RC) or Unreliable Datagram (UD)
transports. In addition to two-sided send/receive communication, RC supports
one-sided RDMA read/write operations. CoRD can independently be enabled on the
sender or receiver, allowing us to study which side contributes to latency
more. Figure 5 shows the absolute latency overhead compared to bypass-to-
bypass communication, measured with a message size of
$4\text{\,}\mathrm{KiB}$. We observed the same numbers for other message
sizes.

When CoRD runs only at the server side, RDMA read has no overhead, because the
sender’s CPU does not participate in sending the message. In contrast, with
RDMA write operations, perftest uses two writes to exchange data: One from the
client to the server for synchronization, another for the server to fetch the
data from the client. For operations except RDMA read, enabling CoRD
contributes equally towards overall overhead from each side.

0.000.250.500.751.000.02.55.07.510.012.5$2^{3}$$2^{6}$$2^{9}$$2^{12}$$2^{15}$$2^{18}$Message
Size, BRelative throughputMessage Rate
($10^{6}/s$)Read/RCWrite/RCSend/RCSend/UD Figure 6: CoRD’s throughput on
system L relative to bypass communication. Communication goes over RC or UD
using one-sided (Read/Write) or two-sided (Send) operations. UD supports only
up to $4\text{\,}\mathrm{KiB}$ messages. The overlayed lines show the message
rate (right axis) in the bypass configuration.

Constant per-message overhead results in significantly lower maximum
throughput for large bursts of small messages. Figure 6 corroborates this
statement because, with larger messages, bandwidth degradation becomes
insignificant. This behavior is similar for all types of communication (RC/UD,
Send/Read/Write) as the per-message overhead is similar. Specifically, for
$32\text{\,}\mathrm{KiB}$ messages exchanged using send operations, perftest
measured ~370k messages per second and only $1\%$ bandwidth degradation.

Read/RCWrite/RCSend/RCSend/UD

.49 0.00.51.01.52.0$2^{6}$$2^{12}$Message size, BLatency overhead (µs)

Figure 7: Latency overhead

.49 0.00.51.0$2^{12}$$2^{14}$$2^{16}$Message sizeRelative throughput

Figure 8: Throughput Figure 9: Latency overhead and relative throughput on
system A when communicating over different transports (RC/UD) using one-sided
(Read/Write) or two-sided (Send) communication.
0.00.51.01.52.0ISEPMGFTLUCGBTSPBenchmarkRelative RuntimeTransportCoRDIPoIB
Figure 10: Relative runtime of the NPB benchmarks on system A.

System A exhibits similar behavior (fig. 9), except for a few minor
differences: Overall per-message overhead is larger, has higher variation, and
has two statistical modes (fig. 9 shows them) for small
($\leq$1\text{\,}\mathrm{KiB}$$) and large messages, with the larger messages
having smaller overhead. Our initial investigation suggests that CoRD has
higher overhead for small messages, because current implementation of CoRD
lacks support for inline messages, whereas baseline supports them. At the same
message size, system L shows a higher throughput reduction than system A
because its network has double the bandwidth. Nevertheless, bandwidth
reduction becomes negligible from a certain message size (see fig. 9).

To estimate the effect on real-world applications, we measure the performance
of the MPI version of the NPB suite [17]. We compare communication over RDMA,
CoRD, and IPoIB. To amplify the network effects, we bar the MPI library from
using shared memory communication. We picked IPoIB for comparison because it
communicates over Infiniband NIC, yet offers fine-grained control over data-
plane operations, making it a functionally equivalent competitor to CoRD. Each
benchmark has limitations on the number of processes allowed for a run, which
in our case ranged from $128240$.

For all the benchmarks, CoRD has nearly zero overhead over baseline kernel-
bypass communication whereas IPoIB is up to $2\times$ slower. IPoIB is the
slowest with the IS (integer sorting) and SP (matrix factorization)
benchmarks, which are simultaneously data intensive (each process sends
$72\text{\,}\mathrm{G}\mathrm{bit}\mathrm{/}\mathrm{s}$ and
$34\text{\,}\mathrm{G}\mathrm{bit}\mathrm{/}\mathrm{s}$, respectively) and
message intensive
($\approx$$1300\text{\,}\mathrm{m}\mathrm{e}\mathrm{s}\mathrm{s}\mathrm{a}\mathrm{g}\mathrm{e}\mathrm{s}\mathrm{/}\mathrm{s}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{d}$
per process). EP (embarrassingly parallel), which communicates very little,
and CG (conjugate gradient), which communicates using few large messages, see
a slight performance boost with CoRD. Similarly to CG, we observe CoRD
marginally outperforming kernel bypass in large-message bandwidth
microbenchmarks when Turbo Boost is enabled. This behavior suggests that
system calls interact with DVFS. Overall, our initial implementation of CoRD
has negligible overhead, even for large-scale real-world applications.

##  6 Discussion and Conclusion

Modern cloud systems predominantly rely on socket-based networking to manage a
myriad of distributed applications. Key to this operation is the precise
control over application communication channels using tools such as Linux
packet filtering [1] or eBPF [82]. Despite persistent enhancements in high-
performance TCP/IP [12, 78], it still falls significantly short of the
performance of RDMA networks.

High-performance networking is diverse and complex [18, 15, 53, 7, 52], so,
unsurprisingly, there are no “one size fits all” solutions [79]. Striving for
optimal problem-specific solutions, researchers abandoned polling [84, 10],
zero-copy [33, 79], lossless congestion control [49, 25], hardware offloading
[43, 32, 47, 67], and one-sided operations [77, 41, 19] to achieve higher
flexibility and resource utilization. So, giving up performance-enhancing
features is not radically new.

Still, the high-performance networking community adamantly avoids putting the
OS kernel on the data path. Instead, existing works add another device on the
path of a packet, either in the form of dedicated CPU cores [44, 39, 11, 63,
83, 52] or by offloading complicated data path interception logic to the NIC.
Such offloading requires either an expensive SmartNIC [16, 31, 73, 85] or
hardware modifications [50, 46, 69].

Our work is based on the idea that the actual problem with system calls is not
the cost of user-kernel transitions in hardware, but rather an obsolete and
inefficient API; in this case POSIX. This idea is in line with the existing
body of work [89, 70, 86, 65, 6], and our early results support the
observation of kernel bypass being dispensable.

In the future, we strive for a smaller per-message overhead and incorporate
several policies, which we will evaluate with a modern bare-metal system. We
intend to assemble a set of real-world benchmark applications that shows the
“breaking point” of CoRD. When successful, CoRD concepts can help in other
domains like high-performance storage [88, 35], where APIs are built on
similar concepts.

## Acknowledgments

The research and the work presented in this paper has been supported by the
German priority program 1648 “Software for Exascale Computing” via the
research project FFMK [23] (HA-2461/10-2). The authors acknowledge the
financial support by the Federal Ministry of Education and Research of Germany
in the programme of “Souverän. Digital. Vernetzt.”. Joint project 6G-life,
project identification number: 16KISK001K This work was supported in part by
the German Research Foundation (DFG) within the Collaborative Research Center
HAEC and the the Center for Advancing Electronics Dresden (cfaed). The authors
acknowledge support from the Oracle for Research for providing cloud computing
resources in project 172. We would like to thank the anonymous HotOS and WORDS
reviewers for their valuable comments.

## References

  * [1] Weave Net: Network containers across any environment.  https://www.weave.works/docs/net/latest/overview/, 2014.
  * [2] Marcelo Abranches, Oliver Michel, and Eric Keller.  Getting back what was lost in the era of high-speed software packet processing.  In Proceedings of the 21st ACM Workshop on Hot Topics in Networks, pages 228–234, Austin Texas, November 2022. ACM.
  * [3] Marcelo Abranches, Oliver Michel, Eric Keller, and Stefan Schmid.  Efficient Network Monitoring Applications in the Kernel with eBPF and XDP.  In 2021 IEEE Conference on Network Function Virtualization and Software Defined Networks (NFV-SDN), pages 28–34, Chandler, AZ United States, November 2021. IEEE.
  * [4] Aleksandar Dragojević, Dushyanth Narayanan, Orion Hodson, and Miguel Castro.  FaRM: Fast Remote Memory.  In Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation, pages 401–414, Seattle, WA, USA, 2014\. USENIX Association.
  * [5] Amazon Web Services, Inc.  Elastic Fabric Adapter.  https://aws.amazon.com/hpc/efa/, 2020.
  * [6] Vaggelis Atlidakis, Jeremy Andrus, Roxana Geambasu, Dimitris Mitropoulos, and Jason Nieh.  POSIX abstractions in modern operating systems: the old, the new, and the missing.  In Proceedings of the Eleventh European Conference on Computer Systems, pages 1–17, London United Kingdom, April 2016. ACM.
  * [7] Brian W Barrett, Ron Brightwell, Ryan E Grant, Scott Hemmert, Kevin Pedretti, Kyle Wheeler, Keith Underwood, Rolf Riesen, Torsten Hoefler, Arthur B Maccabe, and Trammell Hudson.  The Portals 4.2 Network Programming Interface.  Technical report, Sandia National Laboratories, April 2017.
  * [8] Michael Bauer, Sean Treichler, Elliott Slaughter, and Alex Aiken.  Legion: Expressing locality and independence with logical regions.  In International Conference for High Performance Computing, Networking, Storage and Analysis, SC, pages 1–11, Salt Lake City, UT, November 2012. IEEE.
  * [9] Adam Belay, George Prekas, Christos Kozyrakis, Ana Klimovic, Samuel Grossman, and Edouard Bugnion.  IX: A Protected Dataplane Operating System for High Throughput and Low Latency.  In 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI ’14, pages 49–65, October 2014.
  * [10] Jan Bierbaum, Maksym Planeta, and Hermann Härtig.  Towards Efficient Oversubscription: On the Cost and Benefit of Event-Based Communication in MPI.  In Runtime and Operating Systems for Supercomputers, Dallas, Texas, USA, 2022. IEEE.
  * [11] Silas Boyd-Wickizer, Haibo Chen, Rong Chen, Yandong Mao, Frans Kaashoek, Robert Morris, Aleksey Pesterev, Lex Stein, Ming Wu, Yuehua Dai, Yang Zhang, and Zheng Zhang.  Corey: An Operating System for Many Cores.  In 8th USENIX Symposium on Operating Systems Design and Implementation, San Diego, California, USA, 2008. USENIX Association.
  * [12] Qizhe Cai, Midhul Vuppalapati, Jaehyun Hwang, Christos Kozyrakis, and Rachit Agarwal.  Towards _$\mu$_ s tail latency and terabit Ethernet: disaggregating the host network stack.  In Proceedings of the ACM SIGCOMM 2022 Conference, pages 767–779, Amsterdam Netherlands, August 2022. ACM.
  * [13] Joao Carreira, Pedro Fonseca, Alexey Tumanov, Andrew Zhang, and Randy Katz.  Cirrus: a Serverless Framework for End-to-end ML Workflows.  In Proceedings of the ACM Symposium on Cloud Computing, SoCC ’19, pages 13–24, New York, NY, USA, November 2019. Association for Computing Machinery.
  * [14] Mohak Chadha, Jophin John, and Michael Gerndt.  Extending SLURM for Dynamic Resource-Aware Adaptive Batch Scheduling.  arXiv:2009.08289 [cs], September 2020.
  * [15] Cray Inc.  XC Series GNI and DMAPP API User Guide CLE70UP02 (S-2446), 2019.
  * [16] Daniel Firestone, Andrew Putnam, Sambhrama Mundkur, Derek Chiou, Alireza Dabagh, Mike Andrewartha, Vivek Bhanu, Eric Chung, Harish Kumar Chandrappa, Somesh Chaturmohta, Matt Humphrey, Jack Lavier, Norman Lam, Fengfen Liu, Kalin Ovtcharov, Jitu Padhye, Gautham Popuri, Shachar Raindel, Tejas Sapre, Mark Shaw, Gabriel Silva, Madhan Sivakumar, Nisheeth Srivastava, Anshuman Verma, Qasim Zuhair, Deepak Bansal, Doug Burger, Kushagra Vaid, David A. Maltz, and Albert Greenberg.  Azure Accelerated Networking: SmartNICs in the Public Cloud.  In 15th USENIX Symposium on Networked Systems Design and Implementation, NSDI’18, pages 51–64, Berkeley, Calif, 2018. USENIX Association.
  * [17] David H. Bailey, Eric Barszcz, John T. Barton, D. S. Browning, Robert L. Carter, Leonardo Dagum, Rod A. Fatoohi, Paul O. Frederickson, T. A. Lasinski, Robert S. Schreiber, Horst D. Simon, V Venkatakrishnan, and Sisira K. Weeratunga.  The NAS Parallel Benchmarks.  1994\.
  * [18] DPDK Project.  Data Plane Development Kit.  https://www.dpdk.org/, 2013.
  * [19] Aleksandar Dragojevic, Dushyanth Narayanan, and Miguel Castro.  RDMA Reads: To Use or Not to Use?  Bulletin of the Technical Committee on Data Engineering, 40(1):3–14, March 2017.
  * [20] Kaoutar El Maghraoui, Travis J. Desell, Boleslaw K. Szymanski, and Carlos A. Varela.  Dynamic Malleability in Iterative MPI Applications.  In Seventh IEEE International Symposium on Cluster Computing and the Grid (CCGrid ’07), pages 591–598, May 2007.
  * [21] Kevin Elphinstone and Gernot Heiser.  From L3 to seL4 what have we learnt in 20 years of L4 microkernels?  In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, pages 133–150, Farminton Pennsylvania, November 2013. ACM.
  * [22] Lang Feng, Prabhakar Kudva, Dilma Da Silva, and Jiang Hu.  Exploring Serverless Computing for Neural Network Training.  In 2018 IEEE 11th International Conference on Cloud Computing (CLOUD), pages 334–341, July 2018.
  * [23] FFMK Website.  https://ffmk.tudos.org, 2013.
  * [24] Alvaro Frank, Tim Süss, and André Brinkmann.  Effects and Benefits of Node Sharing Strategies in HPC Batch Systems.  In 2019 IEEE International Parallel and Distributed Processing Symposium, IPDPS, pages 43–53. IEEE Computer Society, May 2019.
  * [25] D. Garcia, P. Culley, R. Recio, J. Hilland, and B. Metzler.  A Remote Direct Memory Access Protocol Specification.  https://tools.ietf.org/html/rfc5040, October 2007.
  * [26] Luis Gerhorst, Benedict Herzog, Stefan Reif, Wolfgang Schröder-Preikschat, and Timo Hönig.  AnyCall: Fast and Flexible System-Call Aggregation.  In Proceedings of the 11th Workshop on Programming Languages and Operating Systems, PLOS ’21, pages 1–8, New York, NY, USA, October 2021. Association for Computing Machinery.
  * [27] Balazs Gerofi, Masamichi Takagi, Atsushi Hori, Gou Nakamura, Tomoki Shirasawa, and Yutaka Ishikawa.  On the Scalability, Performance Isolation and Device Driver Transparency of the IHK/McKernel Hybrid Lightweight Kernel.  In 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 1041–1050, Chicago, IL, USA, May 2016. IEEE.
  * [28] Mark Giampapa, Thomas Gooding, Todd Inglett, and Robert W. Wisniewski.  Experiences with a Lightweight Supercomputer Kernel: Lessons Learned from Blue Gene’s CNK.  In 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–10, New Orleans, LA, USA, November 2010. IEEE.
  * [29] Daniel Gruss, Moritz Lipp, Michael Schwarz, Richard Fellner, Clémentine Maurice, and Stefan Mangard.  KASLR is Dead: Long Live KASLR.  In Eric Bodden, Mathias Payer, and Elias Athanasopoulos, editors, Engineering Secure Software and Systems, volume 10379, pages 161–176. Springer International Publishing, Cham, 2017.
  * [30] Zhiqiang He, Dongyang Wang, Binzhang Fu, Kun Tan, Bei Hua, Zhi-Li Zhang, and Kai Zheng.  MasQ: RDMA for Virtual Private Cloud.  In Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication, SIGCOMM ’20, pages 1–14, New York, NY, USA, July 2020. Association for Computing Machinery.
  * [31] Torsten Hoefler, Salvatore Di Girolamo, Konstantin Taranov, Ryan E. Grant, and Ron Brightwell.  sPIN: high-performance streaming processing in the network.  In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC ’17, pages 1–16, Denver, Colorado, 2017. ACM Press.
  * [32] Toke Høiland-Jørgensen, Jesper Dangaard Brouer, Daniel Borkmann, John Fastabend, Tom Herbert, David Ahern, and David Miller.  The eXpress data path: fast programmable packet processing in the operating system kernel.  In Proceedings of the 14th International Conference on emerging Networking EXperiments and Technologies, pages 54–66, Heraklion Greece, December 2018. ACM.
  * [33] W Huang, G Santhanaraman, and Q Gao.  Design and Implementation of High Performance MVAPICH2 (MPI2 over InfiniBand).  In Sixth IEEE International Symposium on Cluster Computing and the Grid, CCGRID ’06, page 10, 2006.
  * [34] InfiniBand Trade Association.  InfiniBand Architecture Specification, volume 1.  InfiniBand Trade Association, 1.3 edition, March 2015.
  * [35] Intel Corporation.  oneAPI Specification 1.2-rev-1.  Specification.
  * [36] Intel Corporation.  Intel® Data Streaming Accelerator Architecture Specification.  Technical Report 2.0, Intel Corporation, September 2022.
  * [37] Muhammad Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han, and KyoungSoo Park.  mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes.  In Proceedings of the 14th USENIX Symposium on Networked Systems Design and Implementation, Boston, MA, USA, 2017.
  * [38] Jonathan Corbet.  The current state of kernel page-table isolation [LWN.net].  https://lwn.net/Articles/741878/, December 2017.
  * [39] Jonathan Corbet.  Ringing in a new asynchronous I/O API.  https://lwn.net/Articles/776703/, January 2019.
  * [40] Laxmikant V. Kale and Sanjeev Krishnan.  A Portable Concurrent Object Oriented System Based On C++.  In Conference on Object-Oriented Programming Systems, Languages and Applications, OOPSLA ’93, pages 91–108, Washington, DC, USA, 1993. ACM Press.
  * [41] Anuj Kalia, Michael Kaminsky, and David G. Andersen.  Using RDMA efficiently for key-value services.  In Proceedings of the 2014 ACM conference on SIGCOMM - SIGCOMM ’14, pages 295–306, Chicago, Illinois, USA, 2014. ACM Press.
  * [42] Anuj Kalia, Michael Kaminsky, and David G Andersen.  FaSST: Fast, Scalable and Simple Distributed Transactions with Two-sided (RDMA) Datagram RPCs.  In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation, Savannah, GA, USA, November 2016. USENIX Association.
  * [43] Antoine Kaufmann, Tim Stamler, Simon Peter, Naveen Kr. Sharma, Arvind Krishnamurthy, and Thomas Anderson.  TAS: TCP Acceleration as an OS Service.  In Proceedings of the Fourteenth EuroSys Conference 2019, pages 1–16, Dresden Germany, March 2019. ACM.
  * [44] Daehyeok Kim, Tianlong Yu, Hongqiang Harry Liu, Yibo Zhu, Jitu Padhye, Shachar Raindel, Chuanxiong Guo, Vyas Sekar, and Srinivasan Seshan.  FreeFlow: Software-based Virtual RDMA Networking for Containerized Clouds.  In Proceedings of the 16th USENIX Conference on Networked Systems Design and Implementation, NSDI, pages 113–125, Boston, MA, USA, 2019.
  * [45] Gregory M. Kurtzer, Vanessa Sochat, and Michael W. Bauer.  Singularity: Scientific containers for mobility of compute.  PLOS ONE, 12(5):e0177459, May 2017.
  * [46] Ilya Lesokhin, Haggai Eran, Shachar Raindel, Guy Shapiro, Sagi Grimberg, Liran Liss, Muli Ben-Yehuda, Nadav Amit, and Dan Tsafrir.  Page Fault Support for Network Controllers.  In Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems, pages 449–466, Xi’an China, April 2017. ACM.
  * [47] Linux Foundation.  networking:toe [Wiki].  https://wiki.linuxfoundation.org/networking/toe, 2016.
  * [48] Moritz Lipp, Michael Schwarz, Daniel Gruss, Thomas Prescher, Werner Haas, Jann Horn, Stefan Mangard, Paul Kocher, Daniel Genkin, Yuval Yarom, Mike Hamburg, and Raoul Strackx.  Meltdown: reading kernel memory from user space.  Communications of the ACM, 63(6):46–56, May 2020.
  * [49] Liran Liss.  The Linux SoftRoCE Driver, March 2017.
  * [50] Liran Liss.  On Demand Paging for User-level Networking, 2013.
  * [51] Anil Madhavapeddy, Richard Mortier, Charalampos Rotsos, David Scott, Balraj Singh, Thomas Gazagnaire, Steven Smith, Steven Hand, and Jon Crowcroft.  Unikernels: library operating systems for the cloud.  ASPLOS ’13 Proceedings of the eighteenth international conference on Architectural support for programming languages and operating systems, 2013.
  * [52] Michael Marty, Steve Gribble, Nicholas Kidd, Roman Kononov, Gautam Kumar, Carl Mauer, Emily Musick, Lena Olson, Erik Rubow, Michael Ryan, Kevin Springborn, Marc de Kruijf, Paul Turner, Valas Valancius, Xi Wang, Amin Vahdat, Jacob Adriaens, Christopher Alfeld, Sean Bauer, Carlo Contavalli, Michael Dalton, Nandita Dukkipati, and William C. Evans.  Snap: a microkernel approach to host networking.  In Proceedings of the 27th ACM Symposium on Operating Systems Principles - SOSP ’19, pages 399–413, Huntsville, Ontario, Canada, 2019. ACM Press.
  * [53] Mellanox Technologies.  RDMA Aware Networks Programming User Manual.  Technical Report 1.7, Mellanox Technologies, 2015.
  * [54] Sebastiano Miano, Fulvio Risso, Mauricio Vásquez Bernal, Matteo Bertrone, and Yunsong Lu.  A Framework for eBPF-Based Network Functions in an Era of Microservices.  IEEE Transactions on Network and Service Management, 18(1):133–151, March 2021.
  * [55] Mick Bauer.  Paranoid Penguin: An Introduction to Novell AppArmor.  https://dl.acm.org/doi/fullHtml/10.5555/1149826.1149839, 2006.
  * [56] Microsoft.  High performance computing VM sizes.  https://docs.microsoft.com/en-us/azure/virtual-machines/sizes-hpc, August 2020.
  * [57] Mike Ko and Alexander Nezhinsky.  Internet Small Computer System Interface (iSCSI) Extensions for the Remote Direct Memory Access (RDMA) Specification.  https://datatracker.ietf.org/doc/html/rfc7145, 2014.
  * [58] Christopher Mitchell, Yifeng Geng, and Jinyang Li.  Using One-Sided RDMA Reads to Build a Fast, CPU-Efficient Key-Value Store.  In USENIX Annual Technical Conference (USENIX ATC 13), pages 103–114, 2013.
  * [59] Angelas Mouzakitis, Christian Pinto, Nikolay Nikolaev, Alvise Rigo, Daniel Raho, Babis Aronis, and Manolis Marazakis.  Lightweight and Generic RDMA Engine Para-Virtualization for the KVM Hypervisor.  In 2017 International Conference on High Performance Computing & Simulation (HPCS), pages 737–744, Genoa, Italy, July 2017. IEEE.
  * [60] MPI Forum.  MPI: A Message-Passing Interface Standard.  Technical Report 3.1, Message Passing Interface Forum, June 2015.
  * [61] OpenMPI.  Open MPI v5.0.x — Open MPI 5.0.x documentation.  https://docs.open-mpi.org/en/v5.0.x/, 2022.
  * [62] Oracle.  Compute Shapes.  https://docs.oracle.com/en-us/iaas/Content/Compute/References/computeshapes.htm, 2023\.
  * [63] Amy Ousterhout, Joshua Fried, Jonathan Behrens, Adam Belay, and Hari Balakrishnan.  Shenango: Achieving High CPU Efficiency for Latency-sensitive Datacenter Workloads.  In Proceedings of the 16th USENIX Symposium on Networked Systems Design and Implementation, Boston, MA, USA, 2019.
  * [64] Parav Pandit.  Manpage: rdma-system - RDMA subsystem configuration, 2020.
  * [65] Pekka Enberg, Ashwin Rao, Jon Crowcroft, and Sasu Tarkoma.  Transcending POSIX: The End of an Era?  https://www.usenix.org/publications/loginonline/transcending-posix-end-era, September 2022.
  * [66] perftest.  https://github.com/linux-rdma/perftest, April 2020.
  * [67] Simon Peter, Jialin Li, Irene Zhang, Dan R K Ports, Doug Woos, Arvind Krishnamurthy, Thomas Anderson, and Timothy Roscoe.  Arrakis: The Operating System is the Control Plane.  In Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, 2014, OSDI ’14, pages 2–17, Broomfield, CO, USA, 2014. USENIX Association.
  * [68] Jonas Pfefferle, Patrick Stuedi, Animesh Trivedi, Bernard Metzler, Ionnis Koltsidas, and Thomas R. Gross.  A Hybrid I/O Virtualization Framework for RDMA-capable Network Interfaces.  In Proceedings of the 11th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, VEE, pages 17–30, Istanbul, Turkey, 2015. ACM Press.
  * [69] Maksym Planeta, Jan Bierbaum, Leo Sahaya Daphne Antony, Torsten Hoefler, and Hermann Härtig.  MigrOS: Transparent Operating Systems Live Migration Support for Containerised RDMA-applications.  In USENIX ATC 2021, pages 47–63, July 2021.
  * [70] Maksym Planeta, Till Miemietz, Viktor Reusch, Jan Bierbaum, Michael Roitzsch, and Hermann Härtig.  Fast Privileged Function Calls.  In Systems for Post-Moore Architectures, Rennes, France, 2022\.
  * [71] Qifan Pu, Shivaram Venkataraman, and Ion Stoica.  Shuffling, Fast and Slow: Scalable Analytics on Serverless Infrastructure.  In Proceedings of the 16th USENIX Symposium on Networked Systems Design and Implementation, Boston, MA, USA, 2019.
  * [72] Benjamin Rothenberger, Konstantin Taranov, Adrian Perrig, and Torsten Hoefler.  ReDMArk: Bypassing RDMA Security Mechanisms.  In 30th USENIX Security Symposium, Vancouver, B.C., 2021.
  * [73] Hugo Sadok, Zhipeng Zhao, Valerie Choung, Nirav Atre, Daniel S Berger, James C Hoe, Aurojit Panda, and Justine Sherry.  We need kernel interposition over the network dataplane.  In HotOS, Ann Arbor, MI, USA, 2021.
  * [74] Philip Schwan.  Lustre: Building a File System for 1,000-node Clusters.  In Linux Simposium, 2003.
  * [75] Anna Kornfeld Simpson, Adriana Szekeres, Jacob Nelson, and Irene Zhang.  Securing RDMA for High-Performance Datacenter Storage Systems.  In 12th USENIX Workshop on Hot Topics in Cloud Computing, HotCloud 20. USENIX Association, 2020.
  * [76] Stephen Smalley, Chris Vance, and Wayne Salamon.  Implementing SELinux as a Linux Security Module, 2002.
  * [77] Maomeng Su, Mingxing Zhang, Kang Chen, Zhenyu Guo, and Yongwei Wu.  RFP: When RPC is Faster than Server-Bypass with RDMA.  In Proceedings of the Twelfth European Conference on Computer Systems, EuroSys ’17, pages 1–15, New York, NY, USA, April 2017. Association for Computing Machinery.
  * [78] Yulin Sun, Qingming Qu, Chenxingyu Zhao, Arvind Krishnamurthy, Hong Chang, and Ying Xiong.  TSoR: TCP Socket over RDMA Container Network for Cloud Native Computing, May 2023.
  * [79] Konstantin Taranov, Fabian Fischer, and Torsten Hoefler.  Efficient RDMA Communication Protocols, December 2022.
  * [80] Konstantin Taranov, Benjamin Rothenberger, Adrian Perrig, and Torsten Hoefler.  sRDMA – Efficient NIC-based Authentication and Encryption for Remote Direct Memory Access.  In 2020 USENIX Annual Technical Conference (USENIX ATC 20), pages 691–704, 2020.
  * [81] Tejun Heo.  Control Group v2.  https://www.kernel.org/doc/Documentation/cgroup-v2.txt, October 2015.
  * [82] Tigera, Inc.  eBPF use cases | Calico Documentation.  https://docs.tigera.io/calico/latest/operations/ebpf/use-cases-ebpf.
  * [83] Shin-Yeh Tsai and Yiying Zhang.  LITE Kernel RDMA Support for Datacenter Applications.  In Proceedings of the 26th Symposium on Operating Systems Principles - SOSP ’17, pages 306–324, Shanghai, China, 2017. ACM Press.
  * [84] Akshay Venkatesh, Abhinav Vishnu, Khaled Hamidouche, Nathan Tallent, Dhabaleswar (DK) Panda, Darren Kerbyson, and Adolfy Hoisie.  A case for application-oblivious energy-efficient MPI runtime.  In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–12, Austin Texas, November 2015. ACM.
  * [85] Lluís Vilanova, Lina Maudlej, Shai Bergman, Till Miemietz, Matthias Hille, Nils Asmussen, Michael Roitzsch, Hermann Härtig, and Mark Silberstein.  Slashing the disaggregation tax in heterogeneous data centers with FractOS.  In Proceedings of the Seventeenth European Conference on Computer Systems, pages 352–367, Rennes France, March 2022. ACM.
  * [86] Xingda Wei, Fangming Lu, Rong Chen, and Haibo Chen.  KRCORE: a microsecond-scale RDMA control plane for elastic computing.  arXiv:2201.11578 [cs], December 2021.
  * [87] Carsten Weinhold, Adam Lackorzynski, and Hermann Härtig.  FFMK: An HPC OS Based on the L4Re Microkernel.  In Balazs Gerofi, Yutaka Ishikawa, Rolf Riesen, and Robert W. Wisniewski, editors, Operating Systems for Supercomputers and High Performance Computing, pages 335–357. Springer Singapore, Singapore, 2019.
  * [88] Ziye Yang, James R. Harris, Benjamin Walker, Daniel Verkamp, Changpeng Liu, Cunyin Chang, Gang Cao, Jonathan Stern, Vishal Verma, and Luse E. Paul.  SPDK: A Development Kit to Build High Performance Storage Applications.  In 2017 IEEE International Conference on Cloud Computing Technology and Science (CloudCom), pages 154–161, December 2017.
  * [89] Irene Zhang, Amanda Raybuck, Pratyush Patel, Kirk Olynyk, Jacob Nelson, Omar S. Navarro Leija, Ashlie Martinez, Jing Liu, Anna Kornfeld Simpson, Sujay Jayakar, Pedro Henrique Penna, Max Demoulin, Piali Choudhury, and Anirudh Badam.  The Demikernel Datapath OS Architecture for Microsecond-scale Datacenter Systems.  In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles, SOSP ’21, pages 195–211, New York, NY, USA, October 2021. Association for Computing Machinery.
  * [90] Yiwen Zhang, Yue Tan, Brent Stephens, and Mosharaf Chowdhury.  Justitia: Software multi-tenancy in hardware kernel-bypass networks.  In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22), pages 1307–1326, 2022.
  * [91] Fang Zheng, Hongfeng Yu, Can Hantas, Matthew Wolf, Greg Eisenhauer, Karsten Schwan, Hasan Abbasi, and Scott Klasky.  GoldRush: resource efficient in situ scientific data analytics using fine-grained interference aware execution.  In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, pages 1–12, Denver Colorado, November 2013. ACM.


11institutetext: Fujian Key Laboratory of Sensing and Computing for Smart
City,
School of Information Science and Engineering, Xiamen University, Xiamen,
China 11email: {qiangwu<EMAIL_ADDRESS>11email:
<EMAIL_ADDRESS>
11email: {yanyan<EMAIL_ADDRESS>
# DSNet: Deep and Shallow Feature Learning for Efficient Visual Tracking

Qiangqiang Wu  Yan Yan  Yanjie Liang  Yi Liu  Hanzi Wang The corresponding
author.

###### Abstract

In recent years, Discriminative Correlation Filter (DCF) based tracking
methods have achieved great success in visual tracking. However, the multi-
resolution convolutional feature maps trained from other tasks like image
classification, cannot be naturally used in the conventional DCF formulation.
Furthermore, these high-dimensional feature maps significantly increase the
tracking complexity and thus limit the tracking speed. In this paper, we
present a deep and shallow feature learning network, namely DSNet, to learn
the multi-level same-resolution compressed (MSC) features for efficient online
tracking, in an end-to-end offline manner. Specifically, the proposed DSNet
compresses multi-level convolutional features to uniform spatial resolution
features. The learned MSC features effectively encode both appearance and
semantic information of objects in the same-resolution feature maps, thus
enabling an elegant combination of the MSC features with any DCF-based
methods. Additionally, a channel reliability measurement (CRM) method is
presented to further refine the learned MSC features. We demonstrate the
effectiveness of the MSC features learned from the proposed DSNet on two DCF
tracking frameworks: the basic DCF framework and the continuous convolution
operator framework. Extensive experiments show that the learned MSC features
have the appealing advantage of allowing the equipped DCF-based tracking
methods to perform favorably against the state-of-the-art methods while
running at high frame rates.

###### Keywords:

Visual tracking Correlation filter Deep neural network.

##  1 Introduction

Given the initial state of a target at the first frame, generic visual object
tracking is to accurately and efficiently estimate the trajectory of the
target at subsequent frames. In recent years, Discriminative Correlation
Filter (DCF) based tracking methods have shown excellent performance on
canonical object tracking benchmarks [31, 32]. The key reasons to their
success are the mechanism of enlarging training data by including all shifted
samples of a given sample, and the efficiency of DCF by solving the ridge
regression problem in the frequency domain.

Features play an important role in designing a high-performance tracking
method. In recent years, significant progress has been made in exploiting
discriminative features for DCFs. For example, hand-crafted features like HOG
[5], Color Names [30] or the combinations of these features, are commonly
employed by DCFs for online object tracking. Despite the fast tracking speed
achieved by these methods, they usually cannot obtain high tracking accuracy
due to the less discriminative features they use. Recently, the outstanding
success of deep convolutional neural networks (CNNs) has been made in a
variety of computer vision tasks [15, 19, 23]. Inspired by the success of
CNNs, the visual tracking community has exploited the advantages of CNNs, and
shown that deep convolutional features trained from other tasks like image
classification, are also applicable for the visual tracking task [24]. On one
hand, the deep features extracted from the shallow convolutional layers, which
provide high spatial resolution, are more helpful for the accurate
localization of the object. On the other hand, the deep features extracted
from the deeper layers encode the semantic information and are more robust to
target appearance variations (e.g., deformation, rotation and motion blur).
The combination of these two types of features shows excellent tracking
performance in both locating the target accurately and modeling the target
appearance variations online. However, the conventional DCF formulation is
limited to single-resolution feature maps. Deep and shallow features (i.e.,
multi-resolution feature maps) cannot be naturally used in the conventional
DCF framework. Thus, how to effectively fuse the deep and shallow features in
the DCF framework is still an open and challenging problem.

Figure 1: Comparison between (a) the DCF-based tracking methods [9, 20] with
deep convolutional features trained from the image classification task and (b)
the DCF-based tracking method with our MSC features.

Recently, several works have been developed to fuse multi-resolution feature
maps in the DCF framework [6, 9]. A straightforward strategy is to explicitly
resample both deep and shallow features from different spatial resolutions to
the same resolution. However, such a strategy introduces artifacts, which
severely limit the tracking performance. To overcome the above issue, an
online learning formulation is presented in C-COT [9] to integrate multi-
resolution feature maps. Despite the promising performance achieved by C-COT,
it still has several limitations: (1) The online learning formulation is time-
consuming due to the high-dimensional deep features. (2) In order to fuse
multi-resolution feature maps, multiple DCFs need to be trained. (3) The
method employs the deep features trained from other tasks like image
classification. These features are not specifically designed for visual
tracking and may limit the tracking performance.

In this paper, instead of designing an online learning formulation to
integrate deep and shallow features (i.e., multi-resolution feature maps) in
the DCF framework, we propose to learn multi-layer same-resolution compressed
(MSC) features in an end-to-end offline manner for efficient online tracking.
To achieve this, a deep and shallow feature learning network architecture
(called as DSNet) is developed in this paper. The proposed DSNet aggregates
multi-level convolutional features and integrates them into the same-
resolution feature maps. In the training stage, a correlation filter layer is
added in DSNet, enabling to learn the discriminative MSC features for visual
tracking. In the test stage, DSNet acts as a feature extractor without relying
on the time-consuming online fine-tuning step. In general, our MSC features
learned by the proposed DSNet have the following characteristics:

(1) MSC features effectively incorporate both the deep and shallow features of
objects but with the same spatial resolution. This enables MSC features to be
naturally fused in any DCF-based tracking methods without using any online
combination strategies.

(2) Due to the low-dimension and uniform spatial resolution of MSC features,
the tracking model complexity can be significantly decreased (see Fig. 1).
Generally, our MSC features can be naturally incorporated by a single DCF
instead of multiple DCFs, thus leading to highly efficient online object
tracking.

To demonstrate the effectiveness of MSC features learned by the proposed
DSNet, we incorporate the MSC features into two state-of-the-art tracking
frameworks: the basic DCF framework [15] and the continuous convolution
operator framework [9], namely MSC-DCF and MSC-CCO, respectively. Experiments
demonstrate that our MSC features have the important advantage of allowing the
equipped MSC-DCF and MSC-CCO methods to perform favorably against the state-
of-the-art methods at high frame rates.

In summary, this paper has the following contributions:

(1) We propose a deep and shallow feature learning network architecture,
namely DSNet, enabling to learn multi-level same-resolution compressed (MSC)
features for efficient online object tracking in an end-to-end offline manner.

(2) Based on the observation that several feature channels have low channel
reliability scores, an online channel reliability measurement (CRM) method is
presented to further refine the learned MSC features.

(3) We show that our MSC features are applicable for any CF-based tracking
methods. Based on the MSC features, two MSC-trackers (MSC-DCF and MSC-CCO) are
presented. Experiments demonstrate that the presented trackers can achieve
favorable performance while running at high frame rates.

##  2 Related Work

In this section, we give a brief review to the methods closely related to our
work.

Correlation filter tracking. Correlation filter (CF) based tracking methods
[8, 9, 15] have attracted considerable attention due to their computational
efficiency and favorable tracking performance. For example, MOOSE [3] is the
initially proposed CF-based tracking method, which uses grayscale images to
train the regression model. KCF [15] further extends MOOSE by using multi-
channel features and mapping the input features to a kernel space. Staple [1]
combines both HOG and Color Name features in a CF framework. Despite the fast
tracking speed of their methods, the employed hand-crafted features are still
not discriminative enough to handle different challenges. To overcome this
problem, several deep feature based tracking methods have been proposed. For
example, CF2 [20] and DeepSRDCF [7] employ the convolutional features
extracted from VGGNet [4]. HDT [24] merges multiple CFs trained on the
different layers of VGGNet. In [9], an online learning formulation of
convolutional features is developed on the spatial domain. ECO [6] further
alleviates the over-fitting problem in [9], and decreases the computational
complexity. Despite significant improvements made by these methods, they still
suffer from the problems of low tracking speed and less discriminative deep
features. To alleviate these problems, in this work, we propose to learn the
multi-layer same-resolution compressed (MSC) features in an end-to-end offline
manner. The learned MSC features are specifically designed for visual
tracking, and they can be easily incorporated into any CF-based tracking
methods without using the time-consuming online fine-tuning steps.

Feature representation learning. Feature representation is the core of many
computer vision tasks, including semantic segmentation [23], object detection
[25] and object tracking [1]. For the object detection task, many works on
feature learning have been proposed. For example, in R-CNN [10], a region
proposal based CNN is proposed to learn better feature representations in an
end-to-end manner. Due to the learned discriminative features, R-CNN
significantly outperforms other detection methods. In addition, SSD [19] and
HyperNer [17] combine multi-layer convolution features to further improve the
detection performance. For the visual tracking task, CFNet [28] firstly
proposes to add a correlation filter layer in a CNN architecture, thus
enabling to learn more discriminative features for CF-based methods. The
similar feature learning method is also introduced in DCFNet [29]. Despite the
success of these methods, these methods only focus on learning single-layer
convolutional features, which may limit their performance. To encode both the
low-level and high-level information of objects, we propose to learn multi-
layer convolutional features in an end-to-end manner for visual tracking.

##  3 Proposed DSNet for Feature Learning

In this section, we firstly introduce the proposed DSNet framework. Secondly,
the feature learning of MSC features is presented. Thirdly, we detail the
feature extraction step of MSC features. Finally, we introduce the channel
reliability measurement (CRM) method to further refine the learned MSC
features.

###  3.1 DSNet Framework

Figure 2: Overall architecture of the proposed deep and shallow feature
learning network (DSNet).

The proposed DSNet framework is illustrated in Fig. 2. As can be seen, DSNet
mainly consists of three parts: a backbone network (the blue part), a shallow
feature extraction branch (the red part) and a deep feature extraction branch
(the orange part). The backbone network is a pre-trained image classification
network, which can be any classification CNNs, such as AlexNet [18] and ResNet
[13]. In this work, we select the imagenet-vgg-2048 network [4] as our
backbone network. In the shallow and deep feature extraction branches, in
order to combine multi-layer multi-resolution feature maps at the same spatial
resolution, a max pooling layer (a 7$\times$7 kernel with a stride of 2) and a
deconvolution layer [23] (a 4$\times$4 kernel with a stride of 4) are added to
perform downsampling and upsampling, respectively. Then, the Conv6 and Conv7
layers (i.e., the 1$\times$1 convolutional layers) are employed to compress
the shallow and deep features, respectively. Moreover, a local response
normalization (LRN) layer [18] is employed to normalize these features.
Finally, we concatenate the normalized features to a single output cube, and
obtain our multi-layer same-resolution compressed (MSC) features. In order to
effectively train the MSC features, a correlation filter is interpreted as a
differentiable layer, which is added at the last layer of DSNet in the
training stage.

###  3.2 End-to-end Feature Learning

In order to effectively train the proposed DSNet and make the learned MSC
features suitable for correlation filter tracking, we add a correlation filter
layer (see Fig. 2) in the proposed DSNet to perform the end-to-end MSC feature
representation learning in an offline manner.

In the training stage, a set of triplet training samples is generated. Let
$T=\\{\mathbf{x}_{i},\mathbf{z}_{i},\mathbf{g}_{i}\\}$ be a triplet, where
$\mathbf{x}_{i}$ is the target image patch including the centered target,
$\mathbf{z}_{i}$ is the test image patch which contains the non-centered
target, and $\mathbf{g}_{i}$ is the desired Gaussian distribution centered at
the target center position according to $\mathbf{z}_{i}$. Given a batch size
of $N$ triplet training samples, the cost function is formulated as:

$\mathcal{L}(\mathbf{\theta})=\sum_{i=1}^{N}{||\sum_{l=1}^{D}{\mathbf{h}_{i}^{l}(\mathbf{\theta})}\ast\varphi^{l}(\mathbf{z}_{i},\mathbf{\theta})-\mathbf{g}_{i}||^{2}},$
(1)

where

$\mathbf{h}_{i}^{l}(\mathbf{\theta})=\mathcal{F}^{-1}(\frac{\hat{\varphi}^{l}(\mathbf{x}_{i},\mathbf{\theta})\odot\hat{\mathbf{g}_{i}}^{\ast}}{\sum_{k=1}^{D}{\hat{\varphi}^{k}(\mathbf{x}_{i},\mathbf{\theta})\odot(\hat{\varphi}^{k}(\mathbf{x}_{i},\mathbf{\theta}))^{\ast}+\lambda}})$
(2)

and $\mathbf{h}_{i}^{l}(\mathbf{\theta})$ is the desired correlation filter
for the $l$-th channel feature map, $\mathbf{\theta}$ refers to the parameters
of our DSNet, $\varphi^{l}(\mathbf{x}_{i},\mathbf{\theta})$ is the extracted
features of $l$-th channel with the parameters $\mathbf{\theta}$ corresponding
to the input $\mathbf{x}_{i}$, and $\lambda$ is a regularization parameter.
Furthermore, $\mathcal{F}^{-1}$ denotes the inverse Fourier transform, $\ast$
is the circular correlation operation, $D$ represents the channel numbers,
$\odot$, $\hat{}$ and ∗ denote the Hadamard product, discrete Fourier
transform and complex conjugation, respectively. By applying the multivariable
chain rule, the derivation of the loss function in Eq. (1) can be rewritten
as:

${\frac{\partial\mathcal{L}}{\partial\mathbf{\theta}}=\sum_{l}\frac{\partial\mathcal{L}}{\partial\varphi^{l}(\mathbf{x}_{i},\mathbf{\theta})}\frac{\partial\varphi^{l}(\mathbf{x}_{i},\mathbf{\theta})}{\partial\mathbf{\theta}}+\sum_{l}\frac{\partial\mathcal{L}}{\partial\varphi^{l}(\mathbf{z}_{i},\mathbf{\theta})}\frac{\partial\varphi^{l}(\mathbf{z}_{i},\mathbf{\theta})}{\partial\mathbf{\theta}}}.$
(3)

Specifically,
$\frac{\partial\varphi^{l}(\mathbf{x}_{i},\mathbf{\theta})}{\partial\mathbf{\theta}}$
and
$\frac{\partial\varphi^{l}(\mathbf{z}_{i},\mathbf{\theta})}{\partial\mathbf{\theta}}$
in the above can be efficiently calculated by recent deep learning toolkits.
According to [28, 29], the prior two terms
($\frac{\partial\mathcal{L}}{\partial\varphi^{l}(\mathbf{x}_{i},\mathbf{\theta})}$
and
$\frac{\partial\mathcal{L}}{\partial\varphi^{l}(\mathbf{z}_{i},\mathbf{\theta})}$)
in (3) can be formulated as:

$\frac{\partial\mathcal{L}}{\partial\varphi^{l}(\mathbf{x}_{i},\mathbf{\theta})}=\mathcal{F}^{-1}(\frac{\partial\mathcal{L}}{\partial(\hat{\varphi}^{l}(\mathbf{x}_{i},\mathbf{\theta}))^{\ast}}+(\frac{\partial\mathcal{L}}{\partial\hat{\varphi}^{l}(\mathbf{x}_{i},\mathbf{\theta})})^{\ast}),$
(4)
$\frac{\partial\mathcal{L}}{\partial\varphi^{l}(\mathbf{z}_{i},\mathbf{\theta})}=\mathcal{F}^{-1}(\frac{\partial\mathcal{L}}{\partial(\hat{\varphi}^{l}(\mathbf{z}_{i},\mathbf{\theta}))^{\ast}}).$
(5)

###  3.3 Feature Extraction

In the online tracking stage, DSNet acts as a feature extractor, which first
extracts multi-layer convolutional feature maps as shown in Fig. 2. Next, the
shallow and deep feature extraction branches in our DSNet aggregate multi-
level convolutional feature maps and compress them to the uniform spatial
resolution features. Finally, the MSC features are obtained by normalizing the
compressed convolutional features. Specifically, the obtained MSC features
(with the size of $52\times 52\times 96$) consist of two parts of features:
shallow convolutional features with the size of $52\times 52\times 32$ and
deep convolutional features with the size of $52\times 52\times 64$. To better
understand the learned MSC features, several channel feature maps of MSC
features are visualized in Fig. 3. As can be seen, the shallow channel feature
maps usually capture the detailed information of objects, while the deep
channel feature maps generally encode the semantic information. These two
types of features can complement each other, and the combination of them is
beneficial for online tracking.

Figure 3: Visualization of the shallow and deep channel feature maps in the
learned MSC features.

###  3.4 Channel Reliability Measurement

Several channel feature maps of MSC features may have small target
activations, which indicates that these feature channels are more sensitive to
the background regions rather than the target regions. To measure the
reliability of these channels, the channel-wise ratio of the $l$-th channel is
formulated as:

$R^{l}=\frac{||S_{t}^{l}||_{1}}{||S_{e}^{l}||_{1}+\zeta}.$ (6)

Here, $S_{e}^{l}$ refers to the entire $l$-th channel feature map, $S_{t}^{l}$
is the target region of $S_{e}^{l}$, $\zeta$ is a penalty parameter and
${||\cdot||}_{1}$ is the $l_{1}$ norm.

The channel-wise ratio shows the ratio of the target responses to the overall
responses, however, it cannot fully reflect the channel reliability in some
cases. For example, when the background responses are equal to zero, even
$||S_{t}^{l}||_{1}$ is a small value, a quite large channel-wise ratio $R^{l}$
will be obtained. To overcome the above problem, $A^{l}$ is defined to measure
the target activations of the $l$-th channel feature map:

$\begin{split}&A^{l}=\begin{cases}1&Z(S_{t}^{l})>W_{t}H_{t}/\eta\\\
0&\text{otherwise},\end{cases}\end{split}$ (7)

where

$Z(S_{t}^{l})=\sum_{w=1}^{W_{t}}\sum_{h=1}^{H_{t}}sign(|S_{t}^{l}(w,h)|),$ (8)

and $sign()$ is the sign function, $S_{t}^{l}(w,h)$ returns the activation
value at the position $(w,h)$ of $S_{t}^{l}$, $W_{t}$ and $H_{t}$ are the
width and height of $S_{t}^{l}$, respectively. $\eta$ is a penalty parameter
that controls the measurement of the target region responses. Finally, the
reliability score of the $l$-th feature channel is calculated by:

$C^{l}=R^{l}\times A^{l}.$ (9)

Generally, channels with high $C^{l}$ reflect that they contain more
activations from the target regions than the background regions. After
obtaining the reliability scores of feature channels, we sort these channels
in the descending order. The top ranked $K$ feature channels are selected to
perform online tracking.

##  4 MSC-Trackers

In this section, we show how the learned MSC features can be incorporated into
different DCF-based tracking frameworks. We select two state-of-the-art
tracking frameworks, i.e., the basic DCF framework [15] and the continuous
convolutional operator framework [9].

###  4.1 MSC Features for the Basic DCF framework

A typical DCF learns a correlation filter $\mathbf{h}^{l}$ by solving a Ridge
Regression problem:

$\min_{\mathbf{h}^{l}}||\sum_{l=1}^{D}{\mathbf{h}^{l}}\ast\varphi^{l}(\mathbf{x},\mathbf{\theta})-\mathbf{g}||^{2}+\lambda_{D}\sum_{l=1}^{D}||\mathbf{h}^{l}||^{2},$
(10)

where $\varphi^{l}(\mathbf{x},\mathbf{\theta})$ is the extracted MSC features
of $l$-th channel with the DSNet parameters $\mathbf{\theta}$ corresponding to
the training image patch $\mathbf{x}$, $\mathbf{g}$ is the desired Gaussian
distribution, $\lambda_{D}$ is a regularization parameter that alleviates the
overfitting problem. The learned correlation filter $\mathbf{h}^{l}$ can be
obtained as [15]:

$\hat{\mathbf{h}^{l}}=\frac{\hat{\varphi}^{l}(\mathbf{x},\mathbf{\theta})\odot\hat{\mathbf{g}}^{\ast}}{\sum_{k=1}^{D}{\hat{\varphi}^{k}(\mathbf{x},\mathbf{\theta})\odot(\hat{\varphi}^{k}(\mathbf{x},\mathbf{\theta}))^{\ast}+\lambda_{D}}}.$
(11)

Given the test image patch $\mathbf{z}$ and the extracted features
$\varphi(\mathbf{z},\mathbf{\theta})$, the online detection process is
formulated as:

$\mathbf{f}=\mathcal{F}^{-1}(\sum_{l=1}^{D}\hat{\mathbf{h}^{l}}^{\ast}\odot\hat{\varphi}^{l}(\mathbf{z,\mathbf{\theta}})),$
(12)

where $\mathbf{f}$ is the response map. The target center position can be
estimated by searching the maximum value in $\mathbf{f}$. During the tracking
process, at the $(t+1)$-th frame, the numerator $A_{(t+1)}^{l}$ and
denominator $B_{(t+1)}^{l}$ in Eq. (11) are respectively updated by using a
moving average strategy with a learning rate $\mu$. Then the correlation
filter model at the $(t+1)$-th frame is updated by
$\hat{\mathbf{h}^{l}_{(t+1)}}=A_{(t+1)}^{l}/(B_{(t+1)}^{l}+\lambda_{D})$. We
use the scale estimation similar to [29].

Note that the conventional DCF framework is restricted to single-resolution
feature maps. In comparison, the proposed MSC features can be naturally fused
into the DCF framework (see Fig. 1). For briefly, this MSC features based DCF
tracker is named as MSC-DCF.

###  4.2 MSC Features for the Continuous Convolution Operator Framework

The continuous convolution operator is proposed in C-COT [9]. Let $y_{j}$
denote a training sample, which contains $D$ feature channels
$y_{j}^{1},y_{j}^{2},...,y_{j}^{D}$. $N_{D}$ is the number of spatial samples
in $y_{j}^{d}$. Here, the feature channel $y_{j}^{d}$ can be viewed as a
function $y_{j}^{d}[n]$, where $n$ is the discrete spatial variable
$n\in\\{0,...,N_{d}-1\\}$. Assume that the spatial support of the feature map
is the continuous interval $[0,P)\subset\mathbb{R}$. The interpolation
operator $J_{d}$ is formulated as:

$J_{d}\\{y^{d}\\}(p)=\sum_{n=1}^{N_{d}}y^{d}[n]b_{d}(p-\frac{P}{N_{d}}n),$
(13)

where $b_{d}$ is the interpolation function, $p$ denotes the location in the
image, $p\in[0,P)$. In the continuous formulation, the convolution operator is
estimated by a set of convolution filters $f=(f^{1},f^{2},...,f^{D})\in
L^{2}(P)$. The convolution operator is defined as:

$Q_{f}\\{y\\}=\sum_{d=1}^{D}f^{d}\circledast J_{d}\\{y^{d}\\}.$ (14)

Here, $f^{d}$ is the continuous filter for the $d$-th channel (see [9] for
more details), $\circledast$ is the circular convolution operation:
$L^{2}(T)\times L^{2}(T)\rightarrow L^{2}(T)$. As can be seen, for each
interpolated sample $J_{d}\\{y^{d}\\}(p)$, it is convolved with the
corresponding filter $f^{d}$. At last, the final confidence map is obtained by
summing up the convolution responses from all the filters.

In Eqs. (13) and (14), the interpolation operator $J_{d}$ and convolution
filter $f^{d}$ are learned for each feature channel. Thus, the high-
dimensional convolutional features (e.g., the 608-dimensional features used in
CCOT), significantly limit the online tracking speed. In comparison, the
learned MSC features have much less channels (i.e., 96), and they can be
regarded as one layer convolutional features to be fused in the continuous
convolution operator framework without any modifications. We call this MSC
features based tracker as MSC-CCO.

##  5 Experiments

Implementation Details: To avoid overfitting, we select the large scale video
detection dataset (ILSVRC-2015) [26] to train the proposed DSNet. This dataset
contains 4417 videos of 30 different objects. We use 3862 videos in this
dataset for training and the remaining videos for validation. The triplet
training samples $T$ are generated as described in [12]. The proposed DSNet is
trained for 200 epochs with a batch size of 16 and an initial learning rate of
$1\times 10^{-5}$ by using the SGD solver. The momentum and weight decay are
respectively set to $9\times 10^{-1}$ and $5\times 10^{-4}$. In the tracking
stage, for MSC-DCF, the learning rate $\mu$ and the padding area are
respectively set to 0.012 and 1.65. The searching scale number is set to 3.
For MSC-CCO, we set the learning rate and the padding area to $9.4\times
10^{-3}$ and 3.62, respectively. Similar to [6], except for MSC features, MSC-
CCO also employs HOG features, and the MSC features are further compressed to
38-D by using PCA. The other parameters in MSC-DCF and MSC-CCO are
respectively set to be the same as in [29] and [6]. For the CRM method, we
apply it to refine the deep feature channels in MSC features, where $K$ in
MSC-DCF and MSC-CCO are respectively set to 50 and 58. The penalty parameters
$\eta$ and $\zeta$ are set to 3 and $1\times 10^{-5}$, respectively. In
addition, we implement our method on a computer equipped with an Intel 6700K
4.0 GHz CPU and an NVIDIA GTX 1080 GPU.

Evaluation Methodology: Both the distance precision (DP) and overlap success
(OS) plots are adopted to evaluate trackers on OTB-2013 [31], OTB-2015 [32]
and OTB-50. We report both the DP rates at the conventional threshold of 20
pixels (DPR) and the OS rates at the threshold of 0.5 (OSR). The Area Under
the Curve (AUC) is also used to evaluate the trackers.

Comparison Scenarios: We evaluate the proposed MSC-trackers (MSC-DCF and MSC-
CCO) in four experiments. The first experiment is conducted to demonstrate the
effectiveness of the learned MSC features by comparing our MSC features with
both hand-crafted features and deep features. At the second experiment, we
compare the proposed highly efficient MSC-trackers with the state-of-the-art
real-time trackers, which shows the superiority of our MSC-trackers. The third
experiment compares our MSC-trackers with the top-performing CF-based trackers
with deep features. The last experiment makes an ablation study on MSC-
trackers to show the effectiveness of the proposed CRM method.

###  5.1 Feature Comparison

Table 1: Comparison of our MSC features with both the hand-crafted features and deep convolutional features within the DCF framework on OTB-2013. Note that ∗ indicates the GPU speed, otherwise the CPU speed. The  first,  second and  third best features are shown in color. |  | MSC
---
| Conv1
---
| Conv5
---
| HOG
---
| RGB
---
| HOG+RGB
---
| Conv1+Conv5
---
DPR (%) | 83.7 | 76.2 | 69.4 | 74.4 | 47.4 | 77.7 | 78.5
OSR (%) | 78.3 | 72.6 | 53.7 | 73.5 | 38.7 | 75.2 | 75.7
Feat. Size. | $52\times 52$ | $109\times 109$ | $13\times 13$ | $52\times 52$ | $52\times 52$ | $52\times 52$ | $109\times 109$
Feat. Dimen. | 96 | 96 | 512 | 32 | 3 | 35 | 608
Avg. FPS | ${\color[rgb]{0,1,0}68.5}^{*}$ | $51.0^{*}$ | ${\color[rgb]{0,0,1}58.1}^{*}$ | 56.0 | 328.1 | 49.6 | $2.6^{*}$

To demonstrate the effectiveness of the learned MSC features, we compare the
MSC features with different commonly used features. For fair comparison, all
the compared features are incorporated into the same tracking framework, i.e.,
the DCF framework described in Section 4.1 for this experiment. We compare MSC
features with raw RGB pixels (RGB), HOG [5], the first (Conv1) and fifth
(Conv5) layer convolutional features in imagenet-vgg-2048, the combination of
HOG and RGB (HOG+RGB) features and the combination of Conv1 and Conv5
(Conv1+Conv5) features. In order to combine the Conv1 and Conv5 features, the
bilinear interpolation method is employed to upsample the Conv5 features to
the same size as the Conv1 features.

Table 1 shows a comparison of our MSC features with different types of
features on OTB-2013. As can be seen, our MSC features achieve the best DPR
(83.7%) and OSR (78.3%) on OTB-2013, significantly outperforming both the
hand-crafted features and deep convolutional features with large margins. In
particular, the Conv1 and Conv5 features are extracted from the imagenet-
vgg-2048 network, which is also the backbone network in DSNet. Despite the
similar network architecture, our MSC features improve 7.5% and 14.4% of the
DPRs obtained by the Conv1 and Conv5 features on OTB-2013, respectively.
Compared with the Conv1 and Conv5 features, the combination Conv1+Conv5
features have more feature channels (608) and achieve better performance, with
a DPR of 78.5%. Although much more feature channels are included in the
Conv1+Conv5 features, our MSC features still provide improved performance,
with a DPR of 83.7%, while achieving the fast tracking speed of 68.5 FPS,
which is about 26 times faster than the Conv1+Conv5 features. This empirically
shows that the MSC features are compact and can lead to highly efficient
online tracking.

###  5.2 Comparison with Real-Time Trackers

Figure 4:  Precision (left) and success (right) plots obtained by our MSC-DCF
and MSC-CCO compared with the other 12 state-of-the-art real-time trackers on
OTB-2015. DPRs and AUCs are reported in left and right brackets, respectively.

We compare the proposed highly efficient MSC-trackers (MSC-DCF and MSC-CCO)
with 12 state-of-the-art trackers that can achieve real-time tracking speed
(FPS$\textgreater$20) for fair comparison, including SiamFC [2], CFNet [28],
Staple [1], DCFNet [29], DCFCA [22], LCT [21], DSST [8], KCF [15], GOTURN
[14], Re3 [11], DCF [15], and TLD [16].

Fig. 4 compares the proposed MSC-trackers with the state-of-the-art real-time
trackers, showing that our MSC-trackers achieve the best performance on
OTB-2015. More specifically, MSC-CCO achieves the best DPR (89.2%) followed by
MSC-DCF (79.8%) and CFNet (77.7%). Note that CFNet is the winner of the VOT-17
real-time challenge. Similar to MSC-DCF, CFNet also employs the traditional
DCF framework. However, different from CFNet, our MSC-DCF learns complementary
multi-layer features instead of single-layer features, thus achieving better
performance than CFNet in terms of both DPR and AUC.

Table 2: DPRs (%) and speed (∗ indicates the GPU speed, otherwise the CPU speed) obtained by our MSC-DCF and MSC-CCO trackers as well as the state-of-the-art real-time trackers on OTB datasets. The results of top 8 performing trackers are given. The  first,  second and  third best trackers are shown in color. |  | MSC-CCO
---
| MSC-DCF
---
| CFNet
---
| SiamFC
---
| DCFNet
---
| Staple
---
|  DCFCA
---
| LCT
---
OTB-2013 | 90.5 | 83.7 | 82.2 | 80.3 | 79.5 | 79.3 | 78.4 | 84.8
OTB-2015 | 89.2 | 79.8 | 77.7 | 77.1 | 75.1 | 78.4 | 74.4 | 76.2
OTB-50 | 86.6 | 75.2 | 72.3 | 69.4 | 68.3 | 68.1 | 71.2 | 69.1
Avg. DPR | 88.8 | 79.6 | 77.4 | 75.6 | 74.3 | 75.3 | 74.7 | 76.7
Avg. FPS | $20.6^{*}$ | ${66.8}^{*}$ | ${\color[rgb]{0,0,1}67.0}^{*}$ | ${\color[rgb]{0,1,0}86.0}^{*}$ | $65.0^{*}$ | 48.3 | 179.2 | 21.0

As can be seen from Table 2, the proposed MSC-trackers achieve the best
accuracy over all the three datasets. Specifically, MSC-CCO achieves the best
accuracy (86.6%) on OTB-50 followed by MSC-DCF (75.2%). For the average DPR,
the best two results belong to our MSC-trackers, followed by CFNet (77.4%) and
SiamFC (75.6%). This comparison highlights the high accuracy achieved by our
tracker among the state-of-the-art real-time trackers. The average tracking
speed of different trackers are also reported in Table 2. Compared with other
trackers, MSC-DCF achieves the fast tracking speed of 66.8 FPS while
demonstrating its competitive tracking performance. In addition, higher
accuracy of MSC-CCO is obtained at the cost of lower speed compared to MSC-
DCF, but MSC-CCO still maintains quasi-real-time tracking speed of 20.6 FPS.

###  5.3 Comparison with Deep Feature-based Trackers

We compare the proposed MSC-trackers with 6 state-of-the-art deep feature-
based CF trackers: CCOT [9], MCPF [33], CREST [27], DeepSRDCF [7], CF2 [20]
and HDT [24].

(a)

(b)

(c)

Figure 5:  Success plots obtained by the proposed MSC-trackers (MSC-CCO and MSC-DCF) and the top-performing deep feature-based CF trackers on (a) OTB-50, (b) OTB-2013 and (c) OTB-2015. AUCs are illustrated in brackets.  Table 3: OSRs (%) and speed (∗ indicates the GPU speed, otherwise the CPU speed) obtained by MSC-CCO and MSC-DCF as well as the state-of-the-art deep feature-based CF trackers on OTB-2013, OTB-2015 and OTB-50. The  first,  second and  third best trackers are highlighted in color. |  | MSC-CCO
---
| MSC-DCF
---
| CCOT
---
| MCPF
---
| DeepSRDCF
---
| HDT
---
| CF2
---
| CREST
---
OTB-2013 | 83.7 | 78.3 | 83.2 | 85.8 | 79.4 | 73.7 | 74.0 | 86.0
OTB-2015 | 82.1 | 73.3 | 82.0 | 78.0 | 77.2 | 65.7 | 65.5 | 77.6
OTB-50 | 77.6 | 67.1 | 74.9 | 71.0 | 67.6 | 58.4 | 58.2 | 70.5
Avg. OSR | 81.1 | 72.9 | 80.0 | 78.3 | 74.7 | 65.9 | 65.9 | 78.0
Avg. FPS | ${\color[rgb]{0,1,0}20.6}^{*}$ | ${\color[rgb]{1,0,0}66.8}^{*}$ | $0.22^{*}$ | $0.56^{*}$ |  <$1.0^{*}$ | ${\color[rgb]{0,0,1}11.1}^{*}$ | $10.5^{*}$ | $1.0^{*}$

Fig. 5 and Table 3 show the comparison of our MSC-trackers with several deep
CF trackers with deep features on OTB datasets. More particularly, the AUC and
OSR obtained by MSC-DCF are higher than those obtained by CF2 and HDT. MSC-CCO
achieves the best OSR (77.6%) on OTB-50, significantly outperforming the
second best tracker CCOT with a large margin of 2.7%. Furthermore, the average
OSR obtained by MSC-CCO is 81.1%, which is ranked at the first and followed by
CCOT (80.0%) and MCPF (78.3%). In terms of the tracking speed reported in
Table 5, compared with the other deep feature based trackers, MSC-DCF can run
at 66.8 FPS, which is significantly faster than the compared trackers. In
addition, MSC-CCO achieves the quasi-real-time tracking speed of 20.6 FPS,
which is almost 94 times faster than CCOT and 37 times faster than MCPF. In
the meanwhile, MSC-CCO achieves the best overall performance over all the
three datasets.

Table 4: DPRs (%) obtained by MSC-DCF, MSC-CCO and their additional versions on OTB-2013 and OTB-2015. |  | MSC-CCO
---
| MSC-CCO-w/o-CRM
---
| MSC-DCF
---
| MSC-DCF-w/o-CRM
---
OTB-2013 | 90.5 | 86.5 | 83.7 | 83.1
OTB-2015 | 89.2 | 87.1 | 79.8 | 79.2

###  5.4 Ablation Study

To demonstrate the effectiveness of the proposed CRM method, we evaluate MSC-
DCF and MSC-CCO with additional versions, i.e., MSC-DCF and MSC-CCO without
using the CRM method, namely MSC-DCF-w/o-CRM and MSC-CCO-w/o-CRM,
respectively. As can be seen from Table 4, the CRM method is effective to
improve tracking performance. Specifically, MSC-CCO achieves the accuracy
(90.5%) by improving 4% of MSC-CCO-w/o-CRM on OTB-2013. For MSC-DCF, the
performance is also improved by applying the CRM method. These results
demonstrate the effectiveness of the CRM method, which is mainly due to the
fact that the CRM method filters the feature channels that are more sensitive
to the background regions while retaining the high-quality channels that are
more beneficial for robust visual tracking.

##  6 Conclusions

In this work, we propose a deep and shallow feature learning network (called
as DSNet) to learn the multi-level same-resolution compressed (MSC) features,
which effectively incorporate both deep and shallow features with the same
spatial resolution for efficient online tracking. The proposed DSNet
compresses multi-layer convolutional features and can be effectively trained
in an end-to-end offline manner. The MSC features are generic and can be
easily applied to any CF-based trackers. In addition, we propose an effective
channel reliability measurement method to further refine the learned MSC
features. To demonstrate the effectiveness of our MSC features, two MSC
features based trackers are presented, namely MSC-DCF and MSC-CCO,
respectively. Experiments on several large scale benchmarks show that the
proposed methods perform favorably against state-of-the-art tracking methods.

Acknowledgments. This work is supported by the National Natural Science of
China (Grant No. U1605252, 61872307, 61472334 and 61571379) and the National
Key Research and Development Program of China under Grant No. 2017YFB1302400.

## References

  * [1] Bertinetto, L., Valmadre, J., Golodetz, S., Miksik, O., Torr, P.H.S.: Staple: Complementary learners for real-time tracking. In: Computer Vision and Pattern Recognition (CVPR). pp. 1401–1409 (2016)
  * [2] Bertinetto, L., Valmadre, J., Henriques, J., Vedaldi, A., Vedaldi, P.: Fully-convolutional siamese networks for object tracking. In: ECCV Workshop. pp. 850–865 (2016)
  * [3] Bolme, D.S., Beveridge, J.R., Draper, B.A., Lui, Y.M.: Visual object tracking using adaptive correlation filters. In: Computer Vision and Pattern Recognition (CVPR). pp. 2544–2550 (2010)
  * [4] Chatfield, K., Simonyan, K., Vedaldi, A., Zisserman, A.: Return of the devil in the details: Delving deep into convolutional nets (2014), arXiv preprint arxiv:1405.3531
  * [5] Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Computer Vision and Pattern Recognition (CVPR). pp. 886–893 (2005)
  * [6] Danelljan, M., Bhat, G., Khan, F.S., Felsberg, M.: Eco: Efficient convolution operators for tracking. In: Computer Vision and Pattern Recognition (CVPR). pp. 21–26 (2017)
  * [7] Danelljan, M., Hager, G., Khan, F.S., Felsberg, M.: Convolutional features for correlation filter based visual tracking. In: International Conference on Computer Vision Workshops. pp. 58–66 (2015)
  * [8] Danelljan, M., Häger, G., Khan, F.S., Felsberg, M.: Discriminative scale space tracking. IEEE Trans. Pattern Anal. Mach. Intell. 39(8), 1561–1575 (2017)
  * [9] Danelljan, M., Robinson, A., Khan, F.S., Felsberg, M.: Beyond correlation filters: learning continuous convolution operators for visual tracking. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9909, pp. 472-488. Springer, Heidelberg. doi:10.1007/978-3-319-46454-1_29 (2016)
  * [10] Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Computer Vision and Pattern Recognition (CVPR). pp. 580–587 (2014)
  * [11] Gorden, D., Farhadi, A., Fox, D.: Re3: Real-time recurrent regression networks for object tracking (2017), arXiv preprint arxiv:1705.06368
  * [12] Gundogdu, E., Alatan, A.A.: Good features to correlate for visual tracking (2017), arXiv preprint arxiv:1704.06326
  * [13] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Computer Vision and Pattern Recognition (CVPR). pp. 770–778 (2016)
  * [14] Held, D., Thrun, S., Savaresei, S.: Learning to track at 100 fps with deep regression networks. In: Leibe B., Matas J., Sebe N., Welling M. (eds) ECCV 2016\. LNCS, vol. 9905, pp. 749-765. Springer, Heidelberg. doi:10.1007/978-3-319-46448-0_45 (2016)
  * [15] Henriques, J.F., Caseiro, R., Martins, P., Batista, J.: High-speed tracking with kernelized correlation filters. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 583–596 (2015)
  * [16] Kalal, Z., Mikolajczyk, K., Matas, J.: Tracking-learning-detection. IEEE Trans. Pattern Anal. Mach. Intell. 34(7), 1409–1422 (2012)
  * [17] Kong, T., Yao, A., Chen, Y., Sun, F.: Hypernet: Towards accurate region proposal generation and joint object detection. In: Computer Vision and Pattern Recognition (CVPR). pp. 845–853 (2016)
  * [18] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems (NIPS). pp. 1097–1105 (2012)
  * [19] Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: Ssd: Single shot multibox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21-37. Springer, Heidelberg. doi:978-3-319-46448-0_2 (2016)
  * [20] Ma, C., J. B. Huang, X. Yang, M.H.Y.: Hierarchical convolutional features for visual tracking. In: International Conference on Computer Vision (ICCV). pp. 3074–3082 (2015)
  * [21] Ma, C., Yang, X., Zhang, C., Yang, M.H.: Long-term correlation tracking. In: Computer Vision and Pattern Recognition (CVPR). pp. 5388–5396 (2015)
  * [22] Mueller, M., Smith, N., Ghanem, B.: Context-aware correlation filter tracking. In: Computer Vision and Pattern Recognition (CVPR). pp. 1387–1395 (2017)
  * [23] Noh, H., Hong, S., Han, B.: Learning deconvolution network for semantic segmentation. In: International Conference on Computer Vision (ICCV). pp. 1520–1528 (2015)
  * [24] Qi, Y., Zhang, S., Qin, L., Yao, H., Huang, Q., Lim, J., Yang, M.H.: Hedged deep tracking. In: Computer Vision and Pattern Recognition (CVPR). pp. 4303–4311 (2016)
  * [25] Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: Computer Vision and Pattern Recognition (CVPR). pp. 779–788 (2016)
  * [26] Russakovsky, O., Deng, J.: Imagenet large scale visual recognition challenge 115(3), 211–252 (2015)
  * [27] Song, Y., Ma, C., Gong, L., Zhang, J., Lau, R.W., Yang, M.H.: Crest: Convolutional residual learning for visual tracking. In: International Conference on Computer Vision (ICCV). pp. 2574–2583 (2017)
  * [28] Valmadre, J., Bertinetto, L., Henriques, J., Vedaldi, A., Torr, P.H.S.: End-to-end representation learning for correlation filter based tracking. In: Computer Vision and Pattern Recognition (CVPR). pp. 5000–5008 (2017)
  * [29] Wang, Q., Gao, J., Xing, J., Zhang, M., Hu, W.: Dcfnet: Discriminant correlation filters network for visual tracking (2017), arXiv preprint arxiv:1704.04057
  * [30] Weijer, J.V.D., Schmid, C., Verbeek, J., Larlus, D.: Learning color names for real-world applications. IEEE Trans. Image Process. 18(7), 1512–1523 (2009)
  * [31] Wu, Y., Lim, J., Yang, M.H.: Online object tracking: a benchmark. In: Computer Vision and Pattern Recognition (CVPR). pp. 2411–2418 (2013)
  * [32] Wu, Y., Lim, J., Yang, M.H.: Object tracking benchmark. IEEE Trans. Pattern Anal. Mach. Intell. 37(9), 1834–1848 (2015)
  * [33] Zhang, T., Xu, C., Yang, M.H.: Multi-task correlation particle filter for robust object tracking. In: Computer Vision and Pattern Recognition (CVPR). pp. 4819–4827 (2017)