# PCI-express based high-speed readout for the Belle II DAQ upgrade

Q. D. Zhou, S. Yamada, P. Robbe, D. Charlet, R. Itoh, , M. Nakao, S.Y. Suzuki,
T. Kunigo, E. Jules, E. Plaige, M. Taurigna, H. Purwar, O. Hartbrich, M.
Bessner, K. Nishimura, G. Varner, Y. -T. Lai, T. Higuchi, R. Sugiura, D.
Biswas, and P. Kapusta This work was supported in part by Grants-in-Aid for
Scientific Research -KAKENHI- JSPS International Research Fellow Grants No.
18F18324.Q. D. Zhou is with Institute of Advanced Research and Kobayashi-
Maskawa Institute, Nagoya Univ., Nagoya 464-8601, Japan (e-mail:
qzhou@hepl.phys.nagoya-u.ac.jp).S. Yamada, R. Itoh, M. Nakao, S.Y. Suzuki, and
T. Kunigo are with High Energy Accelerator Research Organization (KEK),
Ibaraki 305-0801, JapanP. Robbe is with the Univ. Paris-Saclay , CNRS/IN2P3,
the Laboratoire de Physique des Deux Infinis Irene Joliot-Curie (IJCLab),
Orsay F-91898, France.D. Charlet, E. Jules, E. Plaige, and M. Taurigna are
with the Laboratoire de Physique des Deux Infinis Irene Joliot-Curie (IJCLab),
Orsay F-91898, France. H. Purwar, O. Hartbrich, M. Bessner, K. Nishimura and
G. Varner are with the Dept. of Phys. & Astr., Univ. of Hawaii at Mānoa,
Honolulu, HI, 96822, USA. Y. -T. Lai, and T. Higuchi are with Kavli Institute
for the Physics and Mathematics of the Universe, University of Tokyo, Kashiwa,
277-8583 Japan. R. Sugiura is with Tokyo University, Tokyo, 113-0033, Japan.
D. Biswas is with University of Louisville, Louisville, Kentucky, 40292, USA.
P. Kapusta is with Institute of Nuclear Physics (IFJ PAN), Krakow, 31-342,
Poland.

###### Abstract

Belle II is a new-generation B-factory experiment, dedicated to exploring new
physics beyond the standard model of elementary particles in the flavor
sector. Belle II started data-taking in April 2018, using a synchronous data
acquisition (DAQ) system based on pipelined trigger flow control. The Belle II
DAQ system is designed to handle a 30-kHz trigger rate with approximately 1%
of dead time, under the assumption of a raw event size of 1 MB. The DAQ system
is reliable, and the overall data-taking efficiency reached 84.2% during the
run period of January 2020 to June 2020. The current readout system cannot be
operated in the term of 10 years from the viewpoint of DAQ maintainability;
meanwhile, the readout system is obstructing high-speed data transmission. A
solution involving a PCI-express-based readout module with high data
throughput of up to 100 Gb/s was adopted to upgrade the Belle II DAQ system.
We particularly focused on the design of firmware and software based on this
new generation of readout board, called PCIe40, with an Altera Arria 10 field-
programmable gate array chip. Forty-eight GBT (GigaBit Transceiver) serial
links, PCI-express hard IP-based DMA architecture, interface of timing and
trigger distribution system, and slow control system were designed to
integrate with the current Belle II DAQ system. This paper describes the
performances accomplished during the data readout and slow control tests
conducted using a test bench and a demonstration performed using on-site
front-end electronics, specifically involving Belle II TOP and KLM sub-
detectors.

###### Index Terms:

Belle II, data acquisition (DAQ), high-speed readout system, PCIe40, PCI
express, DMA.

##  I Introduction

The standard model (SM) of particle physics, which describes the properties
and physical behavior of elementary particles, has finally been established by
the discovery of Higgs boson in 2012 [1, 2]. Although SM has demonstrated huge
success, some phenomena are left unexplained, such as dark matter and material
dominance universe. Belle II [3] is an experiment dedicated to exploring new
physics beyond the SM in the flavor sector at the beam luminosity frontier.
The SuperKEKB accelerator [4], located at KEK, Tsukuba, Japan, is designed to
use the “nano-beam” scheme to achieve the instantaneous luminosity of
$\rm{6.5}\times\rm{10}^{\rm{35}}$ $\mathrm{cm}^{-2}\mathrm{s}^{-1}$, which is
approximately 35 times that of its predecessor KEKB accelerator, by colliding
7 GeV electrons and 4 GeV positrons. It can produce copious amounts of B and D
mesons and $\tau$ leptons. The large statistics enables precise measurements
of rare decays for testing SM with unprecedented sensitivity. The Belle II
experiment aims to collect 50 $ab^{-1}$ integrated luminosity by 2031.

The Belle II detector [3] is a general-purpose spectrometer upgraded or
replaced with respect to the Belle detector. It is configured around a 1.5 T
superconducting solenoid surrounding the interaction region. The Belle II
detector consists of a vertex detector (VXD), central drift chamber (CDC),
particle identification (PID) detector, electromagnetic calorimeter (ECL), and
neutral $K_{L}$ and Muon (KLM) detector, which are located from the inside to
the outside of the detector. The VXD is made of a silicon pixel detector (PXD)
and a silicon vertex detector (SVD) used for detecting decay vertices. The CDC
located outside of the SVD is used as a tracking detector for precise
measurement of momentum and energy loss of charged particles. Using the
Cherenkov counter technique, the PID detector is divided into barrel and
forward endcap parts, which are the time-of-propagation (TOP) counter and the
proximity-focusing Aerogel Ring Imaging Cherenkov detector (ARICH),
respectively. The ECL uses CsI(TI) crystals and CsI in the barrel and endcap
regions, respectively. The KLM uses glass-electrode resistive plate chambers
(RPCs) for the barrel region and layers of scintillator fibers for the endcap
regions and the innermost layer of the barrel region.

##  II Belle II DAQ and readout system

Figure 1: Block diagram of the current Belle II DAQ system.

The Belle II DAQ [5, 6] system is designed to process the data from the front-
end electronics (FEE) to the storage system through several steps based on a
highly unified system as shown in Fig. 1. The highly unified timing and
trigger distribution (TTD) system [7] first distributes Level 1 (L1) triggers
[8] to each FEE board using Frontend Timing Switch (FTSW) modules [9] based on
a 254 Mb/s serial link protocol called “b2tt” [9]. The digitized signals from
the FEE boards of each sub-detector, except for the PXD, are read out by a
unified readout system based on the common module called Common Pipelined
Platform for Electronics readout (COPPER) [10], as shown in Fig. 2. The data
size of the PXD detector is too large to be read out by the COPPER module, an
online event size reduction is necessary [11]. A total of 203 COPPER boards
are used in the current Belle II DAQ system. To unify the readout system, a
high-speed data transmission based on a custom serial link protocol, called
Belle2link [12], is adopted. Xilinx GTP or GTX RocketIO up to 2.54 Gb/s,
driven by the 127 MHz system clock, is used as the transceiver. The required
bandwidth of each data link is less than 1 Gb/s, while the throughput of each
readout PC is also limited to 1 Gb/s. As shown in Fig. 2, each COPPER module
hosts four custom High Speed Link Boards (HSLB). They are interconnected by a
32-bit peripheral component interconnect (PCI) bus and a local bus used for
slow control with 7-bit and 8-bit widths for address and data, respectively.
The data formatting (per link) and module-level event building are processed
by an on-board CPU on the COPPER module; subsequently, the data from several
COPPER modules are collected to a readout PC through the Gigabit ethernet. The
event building and software event selection are performed by high-level
trigger (HLT) farms [13]. Finally, the selected data are sent to the storage
servers. To handle the large data flow at the beam intensity frontier, the
Belle II DAQ system is designed to handle 30 kHz of L1 trigger rate with
approximately 1% of dead time, under a raw event size of more than 1 MB.

Figure 2: Schematic view of unified COPPER readout system.

##  III Requirements for Belle II DAQ upgrade

The current readout system, mainly the COPPER module and its readout PC, is
obstructing high-speed data transfer and DAQ maintainablility. The motivations
for upgrading the Belle II DAQ readout system are as follows:

  1. 1.

As shown in Fig. 2, the COPPER module hosts four HSLB daughter cards used as
single-link receivers, a processor PCI mezzanine card (PrPMC), and a custom
daughter card used as the TTD interface. The broken parts are increasing, such
as the chipset of PrPMC, because the COPPER module has been in use since the
Belle experiment.

  2. 2.

The data formatting and module-level event building are processed by an
onboard CPU. The CPU usage has been observed to reach $\sim$ 60% under the
test condition of 1 kB/event/COPPER at a 30-kHz trigger rate. In addition, the
smallest bandwidth of the current Belle II DAQ system is the Gbit ethernet
throughput of the COPPER module and the corresponding readout PC [11].

  3. 3.

The current DAQ readout system could turn into a bottleneck when the Belle II
experiment takes an aggressive plan of operation in the next 10 years. For
instance, the SuperKEKB accelerator can achieve higher luminosity than the
designed value, the background level of the Belle II experiment can be higher
than expected, and the Belle II experiment can implement a high rate of low
multiplicity triggers and even realize a trigger-less DAQ.

To reduce the modifications required for the other systems, all the
functionalities of the current COPPER-based readout system must be retained.
For the new system and readout module, the following are required:

  1. 1.

The Belle2link protocol needs to be continued so that no further modification
is applied to the FEE’s hardware and firmware for each sub-detector.

  2. 2.

The functionalities implemented in the COPPER module, including data handling,
slow control, data-formatting for each link and each COPPER, event-building of
merging data of 4 links into one event, and error checks, are essential for
the new system.

  3. 3.

To integrate into the TTD system, the TTD interface needs to be implemented on
the new system. A busy handshake is needed for each link.

  4. 4.

The designed characteristics of the current readout system need to be
followed. The trigger rate of 30 kHz with the estimated event size needs to be
handled, onboard data corruption needs to be detected and should be reduced to
less than 1 error / (100 kB/event × 30 kHz × 8 h).

Furthermore, the Belle II experiment will run for at least 10 years, and an
upgrade of any or all of its sub-detector is predictable. Thus, the new system
should be capable of handling enhanced requirements from the sub-detector
side. For instance, when the L1 trigger rate exceeds 30 kHz, the payload for
each link will be higher than the bandwidth of COPPER system. In the Belle II
Collaboration, we had four proposals for the upgrade of the readout system,
all of them had a well-established module used in other experiments. Three
proposals have proceeded to the development of the firmware and software to
verify that the boards satisfy the basic requirements of the Belle II DAQ
upgrade, as listed above. Finally, a PCI-express-based high-speed readout
module, known as PCIe40 [14], was adopted by the Belle II Collaboration for
DAQ upgrade. This is because PCIe40 fulfills the requirements of the Belle II
DAQ upgrade and a large amount of data can be transferred easily from a
readout board to a PC server via PCI-express bus.

##  IV PCIe40 based new Belle II readout system

PCIe40 is a new generation of readout modules based on the PCI-express Gen3
solution with a high data throughput of 100 Gb/s [15]. This module will also
be used for the DAQ upgrade of the LHCb and ALICE experiments at the Large
Hadronic Collider (LHC) [14, 16]. To integrate with the current Belle II DAQ
system, we particularly focus on the design of the firmware and software for
the PCIe40 board. Since all functionalities of the current readout system are
retained, no modification in the firmware and hardware of the sub-detector FEE
is required. A single PCIe40 board can connect up to 48 bidirectional optical
links, while only 4 bidirectional links can be handled by the COPPER module.
Thus, the new readout system will become much more compact by replacing 203
COPPERs with 20 PCIe40 boards. The number of COPPERs used in the current
readout system and the number of PCIe40 required for replacement are listed in
Table I.

TABLE I: Number of COPPER and PCIe40 required by the sub-detector Sub-detector |  Number of COPPER |  Number of PCIe40
---|---|---
SVD |  48 |  4
CDC |  75 |  8
TOP |  16 |  2
ARICH |  18 |  2
ECL |  26 |  2
KLM |  8 |  1
TRG |  12 |  1

###  IV-A PCIe40 module

The PCIe40 module is required to maintain interface compatibility with the
currently running Belle II DAQ system. As illustrated in Fig. 3, four pairs of
onboard MiniPOD optical transceivers are used to receive or transmit data from
or to the FEE through optical fibers. A high-density Intel Altera Arria 10
field-programmable gate array (FPGA) chip (1.15 million cells) provides the
board with powerful reconfigurable logic capabilities. Extra effort is
required to implement the firmware of the Belle2link protocol on the Altera
FPGA, as the current Belle2link logic is implemented on the Xilinx Virtex 5
FPGA. It has been verified that Belle2link is working well on PCIe40 by
checking the cyclic redundancy check (CRC) errors during the data
transmission. PCIe40 is capable of running at up to 10 Gb/s for each link
using the Gigabit Transceiver (GBT) architecture [17] with a maximum of 48
bidirectional links. In addition, there is an onboard LVDS connector with 8
links used as an interface to the Belle II TTD system, and if it is necessary
each board can be connected to 2 TTD modules (FTSW). Three external phase-
locked loops (PLLs) (Si5344 and Si5345) are used for clock jitter cleaning,
and a clock source may be selected from the TTD system or from an onboard
oscillator. An embedded USB blaster on the front panel connected to the PC
server enables remote downloading of the FPGA firmware. There is an over-
heating protection system using the intelligent platform management interface
(IPMI ) for the management. A point-to-point PCI-express Gen3$\times$16 links
is used to connect the PC server and the PCIe40 board. A PCIe40 board is
presently installed on a 1U rack-mount server (Xeon E5-2640v4 10 Core/2.4GHz),
which was prepared as a spare server for the current COPPER readout system.
Multiple PCIe40 boards can be installed in a larger server.

(a)

(b)

Figure 3: (a) Photo of PCIe40 board. (b) Block diagram of the PCIe40 board
illustrating the key features of the board.

###  IV-B Firmware development

Figure 4: Block diagram of key features of firmware implemented on PCIe40.
Front-end electronics and readout PCs are also shown to illustrate the data
flow direction.

Figure 5: Block diagram of key features of firmware implemented on PCIe40 for
data-flow processing and the software features for DMA. Firmware designs
include two main functionality blocks, which are user logic part for event
formatting, and speed-up event building and the PCI-express DMA core, which
offers high bandwidth and low latency data transmission.

The Belle2link protocol, b2tt protocol, and slow control functionalities which
are implemented on the current COPPER-HSLB system, need to be integrated into
the PCIe40 system. Event formatting and event building, as well as PCI-express
hard IP-based direct memory access (DMA) are newly designed for PCIe40. As
shown in Fig. 4, a generic firmware architecture is designed to provide a
unified readout platform for all sub-detectors (except for PXD). A hardware
abstraction layer, called low-level interface (LLI) [18], is designed for user
control and initialization of the PCIe40 board. The frontend side of the
PCIe40 firmware includes the configuration of the GBT and decoding,
synchronization, and aggregation of all GBT links. Meanwhile, it provides a
simple interface for user application codes, such as the slow control unit.
Thus, most user applications can connect with the LLI by simply retrieving
from and pushing data to first-in-first-out (FIFO) interfaces.

The state machine of the Belle2link protocol is implemented on an Arria 10
FPGA. It includes both decoding and encoding of the event fragment data and
slow control parameters based on the defined Belle2link format. Because the
local bus widths of COPPER for address and data are 7-bit and 8-bit,
respectively, the slow control feature was originally designed to access 8-bit
registers mapped onto a 7-bit address space. This is known as the A7D8 access
method for the Belle II experiment. A16D32 and stream file features are
upgraded based on A7D8 to satisfy different requirements of the sub-detectors.
As listed in Table II, each corresponding sub-detector uses one or two slow
control access methods. All three features for slow control access are
retained when upgraded to a new readout system. Address and data decoding and
encoding for slow control accesses are moved from firmware to software. A FIFO
is used as the interface to store addresses and parameters received or need to
be transmitted by Belle2link. The FIFO is controlled by signals connected to
the registers that are accessible using slow control software. The ARICH
detector uses the stream file method to transfer the firmware of the FEE, and
KLM uses it to stream the threshold parameters for the RPC detector. To
maintain a low error rate during transmission, the file to be streamed is
split into packets of a certain length. The data packet length is 100 words
for ARICH and 6 words for KLM. This difference mainly depends on the FEE of
ARICH and KLM.

TABLE II: Access methods of each sub-detector for slow control Sub-detector |  A7D8 |  A16D32 |  Stream file a
---|---|---|---
SVD |  $\checkmark$ |  $\times$ |  $\times$
CDC |  $\times$ |  $\checkmark$ |  $\times$
TOP |  $\times$ |  $\checkmark$ |  $\times$
ARICH |  $\times$ |  $\checkmark$ |  $\checkmark$
ECL |  $\checkmark$ |  $\checkmark$ |  $\times$
KLM |  $\times$ |  $\checkmark$ |  $\checkmark$
a Stream file method includes streaming a complete file, such as FEE firmware,
and a defined parameter length.

Figure 5 illustrates the key features of data processing on the PCIe40 board
and readout PC server. A user logic block is designed to handle event building
and data formatting, and a PCI-express hard IP-based DMA architecture offers a
fully integrated, flexible, and highly optimized solution for high bandwidth
and low latency data transmission. With DMA, data are copied from the FPGA on-
chip memory into the memory of the readout PC server without preempting the
CPU. The event fragment data are received by each Belle2link and buffered to a
corresponding FIFO, and then the generic event formatting and event building
logics are implemented to handle the data based on the firmware. The state
machine is designed to merge fragments of the 48 links sequentially into an
event and to remove redundant header and trailer information of each link,
after it was checked. Subsequently, redefined event header and trailer
information is added to the built event. The merged event data are then passed
to the DMA data FIFO with a size of 32 kB, used as an interface to the PCIe
DMA block.

The Arria 10 Avalon-MM (Memory Mapped) DMA interface [19] is used to easily
design the PCIe protocol. As shown in Fig. 5, the design includes the
following main features: on-chip memory, PCIe Hard IP, and DMA descriptor
controller. The built event data from the user logic block are stored in the
on-chip memory having a 256-bit width and 8 kB pre-DMA page. Conversely, 128
DMA pages are contained in a DMA super page on the PC server side. The DMA
read and write operations are managed by an external DMA descriptor mechanism,
which is separate from the DMA engine. The procedure of sequentially copying
data from the on-chip memory by reading out the PC server’s memory is based on
the descriptor table. The descriptor table contains the data source address (a
pointer in the on-chip memory mapped fabric of the FPGA), destination PC
memory’s address (in the PCIe bus address space), and the transferred data
size. The software controller first programs the descriptor controller’s
register with the location, size, and the number of the descriptor table;
subsequently, the Read DMA engine issues a memory read request to forward the
entire descriptor table to the FIFO controller. When the FIFO controller
receives a list of free DMA super pages, the firmware DMA controller fills
these DMA super pages without further software intervention. The controller
then fetches the table and requests the Write DMA engine to transfer the data
from the Avalon-MM to the PCIe domains with one descriptor at a time. When the
data transmission is completed, the controller sends the DMA status upstream.

An optimized handshake among the user logic block, PCIe-based DMA, and the
corresponding software driver is essential. A certain large buffer capability,
careful synchronization within the DMA controller, and sufficient data
processing power of the readout PC server ensure that data can be
simultaneously transferred downstream at the highest possible rate with
efficient utilization of resources.

The TTD system is designed for trigger and timing distribution and busy
handshake control for the Belle II experiment. A unified bidirectional
communication based on a 254 Mb/s serial data link, called b2tt, is used as
the interface between the PCIe40 board and the TTD system. A TTD interface
with one b2tt link is implemented for the PCIe40 board to handle all 48
belle2links simultaneously. Currently, architecture to handle 48 belle2links
individually is being developed. Thus, when a link has an error or is busy,
the corresponding issue can be successfully collected by the TTD system. A
127-MHz external clock derived from the SuperKEKB accelerator’s radio
frequency (RF) is distributed to the PCIe40 board and used as the system clock
for the firmware.

###  IV-C Software development

Data readout and slow control access based PCI-express interface are new
features compared to the COPPER-HSLB system. The software driver and library
are developed based on C/C++, and they mainly provide the functionality for
accessing and controlling the data acquisition or configuration of the PCIe40
board. On the PCIe40 side, the base address register 0 (BAR0) and BAR2 of PCI-
express are designed to be used for slow control access and DMA access,
respectively. The corresponding driver and library were developed
individually. The driver is involved in managing physical memory, and the
library provides utilities for accessing registers, performing tests, etc. The
driver for DMA is also used to manage the DMA access and provide the
functionalities of accessing the availability status of memory for DMA,
checking the arrived data, and then applying operations to the arrived data.
As shown in Fig. 5, the unit used for individual DMA transfer is the DMA page,
which is 8 kB per page, and this size depends on the Linux constraint. On the
PC servera-side, 128 DMA pages are contained in a DMA super page of 1 MB. A
typically large buffer of several GBs is allocated for each DMA channel. The
management of the DMA access and memory allocation is based on the super page.

An application layer, developed based on low-level drivers and libraries,
contains individual utilities for high-rate data acquisition, slow control
register access, monitoring, and configuration of the PCIe40 board,
Belle2links, etc. An example of an application is shown in Fig. 6(a). A Python
script is designed to show a GUI window with firmware information, PLL status,
clock and trigger information, and all Belle2link status. All this information
is used for a monitoring purpose, it can help the operator to quickly
investigate the readout system if an issue occourred. An operation panel
provides commonly used configuration options to rapidly switch the clock
source, resynchronize the Belle2link for link establishment, and mask or
unmask an individual or all Belle2links.

(a)

(b)

Figure 6: (a) GUI view of a general monitoring system for PCIe40 board and
Belle2link. (b) Output of the monitoring system with specific error details.
It can be integrated into the Belle II log system.

###  IV-D Software development for integration with Belle II DAQ system

There are mainly two blocks of software that need to be upgraded for PCIe40
board in order to integrate with the current Belle II DAQ system. One is
related to the slow control system, and the other to the data readout system.
The slow control feature of the current COPPER system was designed based on
the COPPER local bus. Similar features must be developed for the PCIe40 board.
As shown in Fig. 6(b), a commonly used tool for investigating and debugging
the cause of an error occurring in the readout system has been developed. It
shows inclusive PCIe40 status, individual link status, counters of CRC errors,
and buffered corrupted data. Meanwhile, this output format can be saved into
the logging system thus can be easily integrated into the current Belle II
logging and monitoring system.

The Belle II DAQ slow control system is designed based on a network interface
called network shared memory 2 (NSM2) [20]. The application nodes are C/C++
daemon programs, which are used to manage the slow control functionalities.
Figure 7 illustrates the slow control architecture on the PCIe40 readout PC
with newly developed daemon processes, such as pcie40controld and pcie40linkd.
pcie40linkd is used to manage and control an individual Belle2link, and
pcie40controld is designed to manage and control all pcie40linkd processes
running on the readout PC. PCIe40FEE is a sub-detector-specified library,
where the sub-detectors can define their functionalities to control their
detectors through Belle2link.

The data readout related modifications to be integrated with the current DAQ
system are relatively small. Data readout software with the new DMA scheme has
already been developed. The data format of the merged event is slightly
different and needs changes for data compression. Because the data from 48
links are processed in one readout PC, the upgrade of the related software
from 4 links for COPPER to 48 links for PCIe40 is essential.

Figure 7: Block diagram of slow control architecture on readout PC; a NSM2
based system, involving several newly developed daemon processes, such as
pcie40controld and pcie40linkd, has been developed.

##  V Performance of new readout system

Firmware and software for the new readout system have already been developed.
As discussed in Sec. III, it is necessary to confirm that the new readout
system fulfills the requirements of the Belle II DAQ system. A test bench was
constructed for the demonstration of the readout system. Meanwhile, another
readout system was built and connected with the Belle II TOP and KLM sub-
detectors at the Electronics-Hut in Tsukuba Hall, at KEK, which is used to
host the electronics of the Belle II experiment. Slow control and data readout
tests were performed using the TOP and KLM on-site detectors.

###  V-A Test bench

As shown in Fig. 8, we constructed a test bench for the readout system,
specifically involving the FEEs of the CDC, SVD, TOP, and ARICH and 46 dummy
FEEs based on COPPER-HSLB, for demonstration. The PCIe40 board was installed
on a host PC server, and the electric power was provided by the host PC. The
firmware was downloaded using an embedded USB blaster on the PC. Forty-eight
FEEs were connected to the PCIe40 board using 8 Multiple-Fiber Push-on/Pull-
off (MPO) fibers, and each channel could be activated or deactivated by
controlling the corresponding register on the PCIe40. The clock and triggers
were distributed to PCIe40 and FEEs by the TTD system. The two types of
triggers are evenly spaced pulse triggers and random triggers with intervals
of pseudo-Poisson distribution, which can be issued by the TTD system with an
adjustable input trigger rate.

Figure 8: Overview image of test bench for DAQ readout upgrade. FEEs of TOP,
CDC, ARICH, SVD, and KLM and 46 dummy FEEs based on COPPERs were connected to
the PCIe40 board for the performance test.

###  V-B Performance of data readout

Figure 9: Schematic view of components (throughput-test block and back-
pressure-test block) involved in the two performance tests.

Firstly, we measured the throughput of PCIe DMA data-transfer by using a
throughput test block shown in Fig. 9, where a pattern generator was
implemented as the data source instead of data from the FEEs. A FIFO was added
for back-pressure monitoring. Measurements at high event-rates were performed
with a pulse trigger control to confirm the capability of the new readout
hardware.

  1. 1.

The back-pressure was observed in 10% of events when using a pulse trigger
with 470 kHz trigger rate and an event size of 8 kB, for which the
corresponding data transfer rate was 31 Gb/s,

  2. 2.

No event was lost if the trigger rate was reduced to 260 kHz and when the data
transfer rate was 17 Gb/s.

It is reported that the performance of a single instance of the DMA controller
bonded to a PCIe$\times$8 Gen3 interface can consistently reach 50 Gb/s [15],
and it can eventually be increased to 100 Gb/s using the PCIe$\times$16 Gen3
interface completely. Conversely, the current performance of the data readout
system already fulfills the Belle II DAQ upgrade requirements, while the
throughput limit of the new readout system is 10 Gb/s, which is determined
from the new readout PC.

Secondly, we demonstrated a back-pressure scheme in data-transfer from 46
COPPER-HSLB dummy FEEs as data sources to a PCIe40 board, by using a back-
pressure test block in the FPGA of the board shown in Fig. 9. No back-pressure
or lost events were observed with the new readout system under a 43 kHz
trigger rate and 1 kB / event / link data readout. Here, the trigger control
was set with a restriction of no more than 10 triggers within 130 $\mu$s,
which was determined from the SVD FEEs. Meanwhile, 2.5% of the events issued a
back-pressure under a 26-kHz trigger rate when a 0.8 ms latency was
artificially inserted into the FEE data source. The back-pressure is due to
the limited size of the FIFOs, which are implemented to buffer the data from
the FEEs for event building. The on-chip memory size of the PCIe40 board is
only $\sim$ 70 Mbits. As the memory of PC server is much larger, this issue
will be solved by transferring the event building from the FPGA logic to the
software in the future.

###  V-C Performance of slow control system

All three slow control access methods using the PCIe interface were tested on
the test bench. The single access speed of the A16D32 method was tested by
reading the register of the CDC FEE 1$\times 10^{6}$ times. The average access
time was 83 $\mu$s/access, while the COPPER-HSLB system took $\sim$1 ms. The
streaming file method was tested using the FEEs of KLM and ARICH. The
streaming speed was $\sim$360 kB/s for KLM, which is comparable with the
COPPER-HSLB system ($\sim$350 kB/s). Both the PCIe40 and the COPPER-HSLB
system took less than 2 s to transfer the 3 MB ARICH firmware. A new feature
for accessing different individual links in parallel is available for the
PCIe40 board, which sufficiently reduces the time required for configuring the
FEEs of a sub-detector, while, for the COPPER-HSLB system, the four links need
to be accessed sequentially.

Figure 10: Register accessing rate when simultaneous data acquisition is
performed with different trigger rates. The curve showing a decreasing
accessing rate from 100 to 0 corresponds to the access of a KLM FEE register
using the COPPER-HSLB system, while the accessing rate retains a constant
value of 170 for the PCIe40 board. A similar tendency is observed for TOP in
which the accessing rate decreases for the COPPER-HSLB system when the DAQ
trigger rate increases and remains constant for the PCIe40 board.

A register access test was performed together with data acquisition under
different trigger rates. As shown in Fig. 10, the register accessing rates for
TOP and KLM based on the COPPER-HSLB system decrease from 170 and 100 to 0,
respectively, when the data acquisition trigger rate increase from 0 to 30
kHz. This indicates that there is a limitation of bandwidth for slow control
and data acquisition for the COPPER-HSLB system. However, the corresponding
register accessing rates for TOP and KLM based on the PCIe40 board retain a
constant value even when the data acquisition trigger rate is increasing.
There was no limitation of bandwidth observed for the PCIe40 board.

Another complete slow control test was performed with the on-site FEEs of TOP
and KLM. All 64 FEEs of TOP were connected to 2 PCIe40 boards that were hosted
by 2 separate servers, and 32 FEEs of KLM were connected to a single PCIe40
board. The slow control test was performed based on the upgraded software
architecture, as shown in Fig. 7. The speeds of configuring FEEs and setting
the threshold for both TOP and KLM detectors based on the PCIe40 board are
compatible with the COPPER-HSLB board. The configuration for TOP was confirmed
by taking a short run and checking the data quality. The threshold for KLM was
verified by checking the output raw data file size by varying the threshold
value. It is confirmed that the new software architecture for slow control
works properly.

##  VI Preparation and replacement

The mass production of 31 PCIe40 boards for Belle II DAQ upgrade is completed,
and the boards have arrived at KEK after they have been successfully tested at
CERN. Three boards were installed on 3 servers for the readout system
replacement of TOP and KLM. MPO fibers are connected to the patch panels
located close to the COPPER-HSLB system of TOP and KLM. A 10 Gb/s network
interface card (NIC) is installed on host PC servers for downstream
connection. A 40 Gb network switch is installed close to the readout PCs for
connection with the network switch located in the HLT server room. The
replacement of the readout system for TOP and KLM is scheduled for summer
2021, and the replacement for other sub-detectors will take place afterwards.
The current COPPER system will stand by for some time after the replacement.
In the case of a serious problem, it can be rolled back to the COPPER system
quickly.

##  VII Conclusion

A proposal based on the PCIe40 board was adopted for the Belle II DAQ upgrade.
It is necessary to retain the main features of the current readout system and
reduce the modifications required for the sub-detectors when upgrading to the
new readout system. The developments of FPGA firmware and software are almost
complete. The PCI-express hard IP-based DMA architecture of the new readout
hardware is capable of handling a 260-kHz trigger rate with 17 Gb/s data
throughput without data loss. No back-pressure or event loss was observed in
data-transfer from 46 dummy FEEs to the PCIe40 board under 43-kHz trigger rate
and 1 kB/event/link data size. Three accessing methods used for slow control
offered similar or better performance with respect to the current COPPER-HSLB
system. A new feature that enables accessing individual links in parallel
improves the configuration speed of the sub-detectors. It has been confirmed
that the new software architecture for slow control works properly. The
replacement of the Belle II readout system is planned to start from TOP and
KLM sub-detectors for summer 2021, and the replacement of the COPPER modules
of the other sub-detectors will be done afterwards.

## References

  * [1] ATLAS Collaboration, “Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC,” Phys. Lett., B vol. 716, issue 1, pp.1-29, Sep. 2012.
  * [2] CMS Collaboration, “Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC,” Phys. Lett., B vol. 716, issue 1, pp.30-61, Sep. 2012.
  * [3] T. Abe, et al., “Belle II Technical Design Report,” arXiv:1011.0352, Nov. 2010.
  * [4] K. Akai, K. Furukawa, H. Koiso et al. (SuperKEKB Collaboration), “SuperKEKB collider,” Nucl. Instrum. Methods Phys. Res., Sect. A vol. 907, pp. 188-199, Nov. 2018.
  * [5] S. Yamada et al., “Data acquisition system for the Belle II experiment,” IEEE Trans. Nucl. Sci., vol. 62, no. 3, pp. 1175-1180, Jun. 2015.
  * [6] M. Nakao et al., “Data acquisition system for Belle II,” J. Instrum., vol. 5, no. C12004, Dec. 2010.
  * [7] M. Nakao, “Timing distribution for the Belle II data acquisition system, ” J. Instrum., vol. 7, no. C01028, Jan. 2012.
  * [8] Y. Iwasaki et al., “Level 1 trigger system for the Belle II experiment,” IEEE Trans. Nucl. Sci., vol. 58, pp. 1807-1815, Aug. 2011.
  * [9] M.Nakao et al.,“Minimizing dead time of the BelleII data acquisition system with pipelined trigger flow control,” IEEE Trans. Nucl. Sci., vol. 60, pp. 3729-3734, Oct. 2013.
  * [10] T. Higuchi et al., “Modular pipeline readout electronics for the super-belle drift chamber,” IEEE Trans. Nucl. Sci., vol. 52, pp. 1912-1917, Oct. 2005.
  * [11] S. Yamada et al., “Common Readout Subsystem for the Belle II Experiment and Its Performance Measurement,” IEEE Trans. Nucl. Sci., vol. 64, no. 6, pp. 1415-1419, Jun. 2017.
  * [12] D.Sun et al.,“Belle2Link:A global data readout and transmission for Belle II experiment at KEK,” Phys. Procedia, vol. 37, pp. 1933-1939, 2012.
  * [13] R. Itoh et al., “Data flow and high level trigger of Belle II DAQ system,” IEEE Trans. Nucl. Sci., vol. 60, pp. 3720-3724, Oct. 2013.
  * [14] J. P. Cachemiche, “The PCIe-based readout system for the LHCb experiment,” Topical Workshop on Elect. for Part. Phys., Lisbon, Portugal, Sep. 2015, 10.1088/1748-0211/11/02/P02013.
  * [15] P. Durante, N. Neufeld, R. Schwemmer, U. Marconi, G. Balbi and I. Lax, “100Gbps PCI-express readout for the LHCb upgrade,” 2014 19th IEEE-NPSS Real Time Conference, Nara, 2014, pp. 1-3, doi: 10.1109/RTC.2014.7097492
  * [16] J. Mitra, S.A. Khan, S. Mukherjee, and R. Paul, “Common Readout Unit (CRU) - A new readout architecture for the ALICE experiment,” JINST 11, C03021 (2016), doi: 10.1088/1748-0221/11/03/C03021
  * [17] P. Moreira et al., “The GBT Project,” CERN-2009-006.342.
  * [18] A. F. Prieto, “Back-end firmware for the LHCb VELO upgrade phase I,” Topical Workshop on Elect. for Part. Phys., Santiage de Compostela, Spain, Sep. 2019, 10.22323/1.370.0101.
  * [19] Intel ALTERA “Arria 10 Avalon-MM DMA Interface for PCIe Solutions,” User Guide, May 2016, _www.altera.com_.
  * [20] M. Nakao and S. Y. Suzuki, “Network shared memory framework for the belle data acquisition control system, ” IEEE Trans. Nucl. Sci., vol. 47, pp. 267-271, Apr. 2000.


# Boosting Weakly Supervised Object Detection using Fusion and Priors from
Hallucinated Depth

Cagri Gungor1 and Adriana Kovashka1,2
1Intelligent Systems Program, 2Department of Computer Science
University of Pittsburgh
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Despite recent attention and exploration of depth for various tasks, it is
still an unexplored modality for weakly-supervised object detection (WSOD). We
propose an amplifier method for enhancing the performance of WSOD by
integrating depth information. Our approach can be applied to any WSOD method
based on multiple-instance learning, without necessitating additional
annotations or inducing large computational expenses. Our proposed method
employs a monocular depth estimation technique to obtain hallucinated depth
information, which is then incorporated into a Siamese WSOD network using
contrastive loss and fusion. By analyzing the relationship between language
context and depth, we calculate depth priors to identify the bounding box
proposals that may contain an object of interest. These depth priors are then
utilized to update the list of pseudo ground-truth boxes, or adjust the
confidence of per-box predictions. Our proposed method is evaluated on six
datasets (COCO, PASCAL VOC, Conceptual Captions, Clipart1k, Watercolor2k, and
Comic2k) by implementing it on top of two state-of-the-art WSOD methods, and
we demonstrate a substantial enhancement in performance.

##  1 Introduction

Figure 1: Depth ranges of objects are estimated using depth-language
relationship before training. These ranges are further used to spot relevant
visual proposals that may contain target objects to enhance weakly supervised
object detection.

Weakly-supervised object detection (WSOD) is a challenging task since it is
unclear which instances have the label that was provided at the image level.
Traditional methods only use appearance information in RGB images. However,
appearance information is insufficient to localize objects in complex,
cluttered environments. On the other hand, humans are capable of finding
useful information in complex environments because they rely on object
function, not just appearance. For example, they might reason about which
objects are within reach, which can be captured with depth from stereo vision
[2]. The depth modality provides additional cues about the spatial
relationships and geometrical structure of objects in a scene and is invariant
to appearance variations (e.g. in texture), making it complementary to the RGB
modality. However, weakly-supervised object detection methods do not use depth
information.

We equip WSOD methods with the ability to reason about functional information
(depth). Importantly, our method does so without requiring additional
annotations or suffering significant computational costs. We propose an
amplifier method that can enhance the performance of any weakly supervised
object detection method based on multiple-instance learning. Since traditional
WSOD datasets do not contain ground-truth depth information, the proposed
method utilizes hallucinated (predicted) depth information obtained through a
monocular depth estimation technique. During training, the method incorporates
depth information to improve representation learning and to prune or down-
weight predictions at the box level, which leads to improved object detection
performance during inference.

First, depth can directly be used as a feature to aid in representation
learning, or to produce predictions which can be fused with those computed
from RGB. While simple, this technique has not been used for WSOD, and we show
that it is very effective: it boosts the performance of appearance-only
methods by up to 2.6<EMAIL_ADDRESS>(11% relative gain).

Further, depth can provide strong priors about which of the bounding box
proposals in the noisy WSOD setting contain an object of interest. We examine
the rough depth at which objects of particular categories occur, by computing
the depth range of an object in 1-5% of annotated images. To make this range
more precise, we examine the relationship between language context and depth,
by keeping track of depth range statistics conditioned on co-mentioned objects
and averaging across the most common co-occurring objects. We then use this
range to prune the pseudo ground-truth bounding boxes used to iteratively
update weakly-supervised detection methods, or to down-weight predictions on
the box level. This approach boosts WSOD performance further for a total of
14%<EMAIL_ADDRESS>relative gain.

Our method is simple and can boost multiple WSOD methods that rely on
iterative improvement. We test it in a variety of settings, using two state-
of-art WSOD baselines, MIST [27] and SoS-WSOD [30], and five datasets, COCO,
PASCAL VOC, Clipart1k, Watercolor2k, and Comic2k.

Inspired by recent work that trains object detection methods with language
supervision [38, 33, 11], we further test our method in a setting where labels
at the image level are not ground-truth but estimated. In this setting, our
method boosts the basic WSOD performance even more, by 18% when labels for
training are extracted from COCO, and 63% when they are extracted from
Conceptual Captions.

To summarize, our contributions are: (1) We examine for the first time the use
of depth in weakly-supervised object detection. (2) In addition to depth
fusion, we propose a technique specific to WSOD, which estimates depth priors
with the help of language, and uses them to refine pseudo boxes and box
predictions. (3) We show large performance gains in a large variety of
settings, with the biggest boost from depth refinement when supervision is
least expensive.

##  2 Related Work

Weakly-supervised object detection (WSOD). WSOD is the task of learning to
detect the location and type of multiple objects given only image-level labels
during training. The multi-instance learning (MIL) framework is commonly
utilized in WSOD methods, with early works such as WSDDN [1] integrating MIL
into an end-to-end WSOD system. OICR [32] improved upon this by proposing
pseudo-ground-truth mining and an online instance refinement branch, which was
further refined by PCL [31] through proposal clustering. C-MIL [34] and MIST
[27] introduced modifications to the MIL loss and pseudo-ground-truth mining,
respectively. More recent work, SoS-WSOD [30], proposes a method that produces
pseudo boxes for FSOD by using their improved WSOD module, then they split
noisy data for semi-supervised object detection. Additionally, there have been
efforts to bypass the need for image-level labels by utilizing noisy label
information extracted from caption or subtitle data [35, 4, 38, 33, 11]. In
contrast to these works, our methodology leverages the incorporation of depth
information as an additional modality, leading to improved performance in WSOD
and a reduction of the noise between text and label information.

RGB-D detection. The integration of RGB and depth information to derive
complementary features has been previously studied for fully-supervised indoor
analysis [37, 39, 16, 19] and salient object detection [17, 14, 8, 9, 18].
Fusing the information contained in the RGB and depth modalities is crucial,
as they provide complementary information. The strategies for merging the two
modalities can be classified into three groups, depending on the point in the
processing pipeline where the fusion occurs: early fusion [6, 24], middle
fusion [9, 10, 40, 3], and late fusion [13, 23]. Early fusion techniques
involve combining the RGB and depth images into a single four-channel matrix
at the earliest stage of the process. This integrated matrix is then treated
as a single input. Middle fusion provides a balance between early and late
fusion by utilizing CNNs for both feature extraction and subsequent merging.
In late fusion, individual saliency prediction maps are produced from the RGB
and depth channels. These two predicted maps are then combined through post-
processing operations such as element-wise summation or multiplication. In
contrast to the majority of aforementioned methods, which use separate
networks to extract features from RGB and depth images, several studies [9,
10, 29, 21] employ Siamese networks to learn hierarchical features from both
RGB and depth inputs by utilizing shared parameters. However, _we are the
first to leverage depth data in weakly-supervised object detection._ Our
approach is not specific to a particular method, as it can be applied to any
MIL-based WSOD method to improve its performance without incurring any extra
annotation expenses and with minimal computational overhead during training.
Although the depth modality is not used during the inference stage,
incorporating it during training enhances the performance of the inference.

Monocular depth estimation involves predicting the depth map of a scene from a
single RGB image [22, 26, 25, 36]. We utilize the method described in [22] due
to its strong performance. This estimated (“hallucinated”) depth information
is then utilized to improve the performance of weakly supervised object
detection.

##  3 Approach

Figure 2: This figure illustrates the design of our proposed amplifier
technique that takes advantage of depth information to enhance the performance
of other weakly-supervised object detection methods. During inference, we only
use the RGB branch (shown in orange).

We propose an amplifier approach that incorporates a depth modality to improve
the effectiveness of WSOD methods. Our method can be used with all MIL-based
WSOD methods to boost their performance by incurring little extra cost during
training. It does not use the depth modality during inference to avoid any
slow-downs and reliance on additional data (depth estimation or captions). The
proposed approach comprises three main steps (Sec. 3.1, 3.2, 3.3,
respectively). First, a Siamese network with a shared backbone is employed to
improve representation learning through contrastive learning between RGB and
depth features (referred to as Siamese-Only in the experiments). Second, we
combine detection and classification scores obtained from both RGB and depth
modalities, which can be categorized as late fusion (Fusion). Third, we use
captions and a small number of ground truth bounding box annotations to
calculate depth priors. These depth priors are then used to improve the OICR-
style [32] module in two WSOD methods (Depth-OICR) and create attention with
combined score probabilities (Depth-Attention). Note that Siamese-Only is
always applied, while Fusion, Depth-OICR and Depth-Attention build on top of
it, and can be used alone or combined.

###  3.1 The Siamese WSOD Network

WSOD. Following Bilen et al. [1], let $\mathbf{I}\in\mathbb{R}^{h\times
w\times 3}$ denote an RGB image, and $y_{c}\in\\{0,1\\}$ (where
$c\in\\{1,...,C\\}$ and $C$ is the total number of object categories) be its
corresponding ground-truth class labels. Let $v_{i}$, $i\in\\{1,...,R\\}$
(where $R$ is the number of proposals), denote the visual proposals in image
$\mathbf{I}$. RoI pooling is applied and a fixed-length feature vector
$\phi(v_{i})$ extracted for each visual region. The proposal features
$\phi(v_{i})$ are fed into two parallel fully-connected layers to compute the
visual detection score $v^{det}_{i,c}\in\mathbb{R}^{1}$ and classification
score $v^{cls}_{i,c}\in\mathbb{R}^{1}$:

$v^{det}_{i,c}=w^{det\intercal}_{c}\phi(v_{i})+b^{det}_{c},\quad
v^{cls}_{i,c}=w^{cls\intercal}_{c}\phi(v_{i})+b^{cls}_{c}$ (1)

where $w$ and $b$ are weights and bias, respectively.

Estimating the depth images. To extract depth information from RGB images, we
employ the monocular depth estimation technique by Mahdi et al. [22]. This
enables us to use existing RGB-only object detection datasets without the need
for additional annotations. Although the extracted depth images are initially
grayscale, we use a color map to convert them to RGB images with three
channels.

Siamese design. Our approach utilizes a Siamese network with contrastive
learning to incorporate depth information in the weakly-supervised object
detection network during training. This design allows us to use a backbone
pre-trained with RGB images to extract features from both RGB and depth
images, without adding extra complexity to the model’s parameters. We enhance
the representation learning of the backbone by defining contrastive loss
between RGB and depth features similar to [21]. Utilizing a Siamese network
provides the advantage of using only RGB images during inference similar to
other WSOD methods. This ensures that our contribution does not introduce any
additional overhead on the inference time.

With the help of a pre-trained backbone model, the feature map of RGB image
$\psi(\mathbf{I})$ is extracted. Let $\mathbf{D}\in\mathbb{R}^{h\times w\times
3}$ denote a depth image associated with the RGB image $\mathbf{I}$ and let
$\psi(\mathbf{D})$ be the feature map of the depth image $\mathbf{D}$
extracted by the Siamese backbone. The RGB feature map $\psi(\mathbf{I})$ and
depth feature map $\psi(\mathbf{D})$ are fed into adaptive pooling and the
Siamese fully connected layer to obtain $d$-dimensional projected feature
vectors $\psi_{proj}(\mathbf{I})$ and $\psi_{proj}(\mathbf{D})$. The only
extra parameters we add to the traditional MIL-based WSOD network come from
the fully connected layer for projection with 8 percent overhead (13M
parameters for the projection layer, vs 154M total). If no late fusion is
performed in the experiments, we train as described in Sec. 3.2, but excluding
the $d_{i,c}$ variables in Eq. 5.

Contrastive learning. We L2-normalize the RGB and depth feature vectors
$\psi_{proj}(\mathbf{I})$ and $\psi_{proj}(\mathbf{D})$ vectors, and compute
their cosine similarity:

$S(\mathbf{I},\mathbf{D})=\langle\psi_{proj}(\mathbf{I}),\psi_{proj}(\mathbf{D})\rangle/\rho$
(2)

where $\rho$, is a learnable temperature parameter. We use noise contrastive
estimation (NCE) [12] to define the contrastive learning by considering RGB
image and depth image pairs $(\mathbf{I},\mathbf{D})\in\mathcal{B}$ where
$\mathcal{B}$ is an RGB-depth pair batch. The first component of the NCE loss
contrasts an RGB image with negative depth images to measure how closely the
RGB image matches with its paired depth among others in the batch:

$\mathcal{L}_{{D}\rightarrow{I}}=-\dfrac{1}{|\mathcal{B}|}\sum_{(\mathbf{I},\mathbf{D})\in\mathcal{B}}log\dfrac{exp(S(\mathbf{I},\mathbf{D}))}{exp(S(\mathbf{I},\mathbf{D}))+\sum_{(\mathbf{I}^{\prime},\mathbf{D}^{\prime})\in\mathcal{B}}exp(S(\mathbf{I},\mathbf{D}^{\prime}))}$

(3)

The second component of the NCE loss, $\mathcal{L}_{{I}\rightarrow{D}}$, is
analogously defined to contrast a depth image with negative RGB image samples,
and the two components are averaged:

$\mathcal{L}_{NCE}=(\mathcal{L}_{{D}\rightarrow{I}}+\mathcal{L}_{{I}\rightarrow{D}})/2$
(4)

###  3.2 Late Fusion of the Modalities

The detection and classification scores computed from RGB and depth modalities
are imbued with disparate and complementary details that jointly enrich our
understanding of the target objects. Therefore, we combine these scores to
amplify the performance of object detection.

As the depth images are derived from the RGB images, the spatial arrangement
of the objects is equivalent in both modalities. Hence, we utilize the same
visual region proposals for both RGB and depth modalities. Following the
application of the RoI pooling layer and the Siamese box feature extractor to
the depth feature map $\psi(\mathbf{D})$, we obtain the feature vector
$\phi(d_{i})$ for each depth region. Thereafter, we employ the approach
presented in Eq. 1 to derive the depth detection score
$d^{det}_{i,c}\in\mathbb{R}^{1}$ and the depth classification score
$d^{cls}_{i,c}\in\mathbb{R}^{1}$. Subsequently, we fuse (sum) the scores from
the RGB and depth modalities:

$f^{det}_{i,c}=v^{det}_{i,c}+d^{det}_{i,c},\quad
f^{cls}_{i,c}=v^{cls}_{i,c}+d^{cls}_{i,c}$ (5)

where $f^{det}_{i,c}$ and $f^{cls}_{i,c}$ are fusion detection and
classification scores, respectively.

Following the WSDDN [1] architecture, these classification and detection
scores are converted to probabilities such that $p^{cls}_{i,c}$ is the
probability that class $c$ is in present proposal $f_{i}$, and $p^{det}_{i,c}$
is the probability that $f_{i}$ is important for predicting image-level label
$y_{c}$.

$p^{det}_{i,c}=\dfrac{exp(f^{det}_{i,c})}{\sum_{k=1}^{R}exp(f^{det}_{k,c})},\quad
p^{cls}_{i,c}=\dfrac{exp(f^{cls}_{i,c})}{\sum_{k=1}^{C}exp(f^{cls}_{i,k})}$
(6)

We element-wise multiply the classification and detection scores to obtain the
combined score $p^{comb}_{i,c}$:

$p^{comb}_{i,c}=p^{det}_{i,c}p^{cls}_{i,c}$ (7)

Finally, image-level predictions $\hat{p}_{c}$ are computed as follows, where
greater values of $\hat{p}_{c}\in[0,1]$ mean a higher likelihood that $c$ is
present in the image.

$\hat{p}_{c}=\sigma\left(\sum_{i=1}^{R}p^{comb}_{i,c}\right)$ (8)

Assuming the label $y_{c}$ = 1 if and only if class $c$ is present, the
classification loss used for training the model is defined as follows. Since
no region-level labels are provided, we must derive region-level scores
indirectly, by optimizing this loss.

$\mathcal{L}_{mil}=-\sum_{c=1}^{C}\left[y_{c}\log\hat{p}_{c}+(1-y_{c})\log(1-\hat{p}_{c})\right]$
(9)

###  3.3 Depth Priors

By leveraging a small amount of caption and ground-truth bounding box
annotations, we extract prior knowledge about the relative depths of objects.
We subsequently exploit these depth priors to guide the identification of the
relevant visual regions that may contain the target objects. We note that the
depth priors we estimate vary very little whether we use 1%, 5%, 10%, or 50%
of the available COCO training data; see Table 3. Further, we show that even
though we estimate the depth priors from COCO, they generalize to Conceptual
Captions (Table 2).

We use the notation $pd_{i}\in[0,1]$, $i\in{1,...,R}$, where $R$ is the number
of pre-computed region proposals for depth image $\mathbf{D}$, to represent
the average depth value in the $i$-th region proposal. Each region proposal
contains pixels with values ranging from 0 to 1, which correspond to the
smallest and largest depth values, respectively.

Figure 3: The figure displays a row of images that are accompanied by their
respective depth and caption data, as well as proposal depth value of
different regions and estimated depth prior range.

We employ a limited set of ground-truth bounding box annotations $B$ to
approximate the depth value of objects using the caption that describes the
image in which the objects are present. Let $C$ be the set of object
categories, $W$ be the set of distinct words in the vocabulary that includes
every word in the captions, and $B$ be the set of ground-truth bounding box
annotations. Let $d_{c,w,b}\in\\{[0,1],\varnothing\\}$ denote the depth value
for object $c\in C$, word $w\in W$ and box $b\in B$, which is calculated by
averaging the depth values in the pixels of $b$ similar to the calculation of
$pd_{i}$. As an example, $d_{\text{bird},\text{ocean},b}$ represents the depth
value of the “bird” object corresponding box $b$ of a depth image that has a
caption that includes the “ocean” word. In the absence of “ocean” in the
caption or when annotation $b$ does not correspond to the “bird” object, the
depth value $d_{\text{bird},\text{ocean},b}$ is set to null $\varnothing$.
Further, $d_{c,w}$ represents a set of depth values calculated from
annotations $B$, excluding $\varnothing$ values. The depth range
$r_{c,w}=[mean-std,mean+std]$ for class $c$ and word $w$ is obtained by
utilizing the mean and standard deviation (std) of this set of depth values in
$d_{c,w}$. Once these depth ranges $r_{c,w}$ are computed, they can be applied
to estimate an allowable depth range for a class $c$ in a new image, without
any boxes on that image.

For any new depth image $\mathbf{D}$, the range of estimated depth priors for
an object $c$ is $dr_{c}$:

$dr_{c}=\dfrac{\sum_{s\in S}{r_{c,s}}}{|S|}$ (10)

where $S$ denotes the set of words in the caption corresponding to
$\mathbf{D}$. We only require captions at training time.

We utilize the estimated depth prior range $dr_{c}$ to identify potentially
important regions in $pd$ for each class. We define a depth mask indicator
variable $m_{i,c}\in{0,1}$ for each region $i$ $\in$ $R$ and class $c$ $\in$
$C$, which indicates the likelihood of a particular region in an image
containing an object of a certain class. The computation of this variable is
as follows:

$m_{i,c}=\begin{cases}1,&\text{if }pd_{i}\in dr_{c}\\\
0,&\text{otherwise}\end{cases}$ (11)

If the proposal depth value $pd_{i}$ falls within the estimated depth prior
range $dr_{c}$ for class $c$, it is considered as a relevant region for that
class, and the corresponding mask variable $m_{i,c}$ is set to 1; otherwise,
it is set to 0. Subsequently, we utilize the mask variable $m_{i,c}$ in
combination with our end-to-end network to improve its performance.

As an example, Fig. 3 presents two images featuring a “bird” object, with
different depths. The estimated depth prior ranges $dr_{\text{bird}}$ are
calculated using Eq. 10 for each image based on the words present in the
caption. The caption of the first image includes “feeding” and “hand” words
that suggest the “bird” object is likely to have a smaller depth value, while
the caption of the second image includes “flying” and “ocean” words that
suggest the “bird” object is likely to have a bigger depth value. The regions
on the images having a proposal depth value of $pd_{1}$ are in the estimated
depth prior range $dr_{\text{bird}}$; we observe that they truly include the
“bird” object. The range allows us to rule regions with depth values $pd_{2}$,
which do not contain “bird”.

####  3.3.1 Depth Priors: Updated OICR

Algorithm 1 OICR Mining with Depth Priors

Input: Proposals $R$, Depth Mask Indicator Variable $m$
Output: Pseudo boxes $\hat{R}$

1:$\hat{R}=\varnothing$

2:for all $c=1:C$ do

3: for all $i=1:|R|$ do

4: $\hat{R_{c}}=\hat{R_{c}}\cup R_{i}$ if $m_{i,c}=1$

5:return $\hat{R}$

Online Instance Classifier Refinement (OICR) [32] is a weakly supervised
object detection algorithm that iteratively refines object proposals. Recent
studies [31, 27, 30] have highlighted the importance of more effective
proposal mining strategies for achieving better recall and precision of
objects in WSOD detectors. We propose an algorithm that incorporates the depth
priors during the proposal mining provided in Alg. 1. As our proposed method
aims to enhance MIL-based WSOD methods, we utilize our algorithm in
conjunction with recent OICR-style/self-training/mining strategies, subject to
the depth prior condition specified in the fourth line of Alg. 1. After using
the depth prior condition, OICR-style mining selects fewer but more relevant
proposals so our contribution increases mining precision.111In early
experiments, we verified that our method’s gains persist if the baseline is
allowed to drop the lowest-scoring pseudo boxes but without using depth.

####  3.3.2 Depth Priors: Attention

The depth mask variable $m_{i,c}$ indicates the potentially important proposal
regions for each class. We use this variable to employ an attention mechanism
with combined score probabilities $p^{comb}_{i,c}$ provided in Eq. 7 as
follows:

$p^{comb}_{i,c}=p^{comb}_{i,c}*0.5,\text{if }m_{i,c}=0$ (12)

This mechanism reduces the probability of a region for class $c$ by half if
the region is determined as less likely to be important by $m_{i,c}$. These
scores are then used in Eq. 8.

##  4 Experiments

We test our method on top of two weakly-supervised detection techniques, and
verify the contributions of the constituents of our approach:

  * •

Siamese WSOD Network (Siamese-Only, Sec. 3.1);

  * •

Late Fusion of the Modalities (Fusion, Sec. 3.2) which combines
classification/detection from RGB and depth, and builds on top of the Siamese
WSOD Network (Sec. 3.1);

  * •

Depth Priors are utilized to enhance the OICR-style module (Depth-Oicr, Sec.
3.3.1) and construct attention (Depth-Attention, Sec. 3.3.2) with visual-only
score probabilities, both building upon the Siamese WSOD Network (Sec. 3.1);

  * •

Finally, we use all components of our method (Wsod-Amplifier, Sec. 3.1, 3.2,
3.3.1, 3.3.2).

###  4.1 Experimental Setup

####  4.1.1 Dataset and Metrics

PASCAL Visual Object Classes 2007 (VOC-07) [5] contains 20 classes. For
training, we use 2501 images from the train set and 2510 images from the
validation set. We evaluate using 4952 images from the test set.

Common Objects in Context (COCO) [20] consists of 80 classes. We utilize
approximately 118k images from the train set and use the labels provided at
the image level. Additionally, to test how well our method works when labels
are obtained from noisy language supervision (in captions), we train our
models using labels obtained through an exact match (EM) method following
[33], also referred to as substring matching in [7]. Due to the unavailability
of any labels for around 15k images extracted from captions, we excluded them
from the training set and use 103k images. We evaluate using 5k images from
the validation set.

Conceptual Captions (CC) [28] is a large-scale image captioning dataset
containing over 3 million images annotated with only captions. We use around
30k images and their corresponding captions and the labels are extracted for
the 80 COCO dataset classes using an exact match method from the captions.
During the evaluation, we used 5k images from the COCO validation set.

Domain shift datasets [15]. Clipart1k has the same 20 classes as VOC with
1,000 images, while Watercolor2k and Comic2k share 6 classes with VOC and have
2,000 images each. We use these datasets for evaluation only.

Evaluation protocols. We utilize mean Average Precision (mAP) considering
various IoU thresholds as the common evaluation metric for COCO and VOC
datasets. Additionally, we report mAP for objects of different sizes during
COCO evaluation and we report the results of Correct Localization (CorLoc) for
VOC evaluation.

####  4.1.2 Implementation details

We employ the official PyTorch implementations of SoS-WSOD [30] and MIST [27]
methods to apply our amplifier technique. SoS-WSOD uses four images per GPU as
two augmented images and their flipped versions with a total of 4 GPUs,
whereas MIST uses only one image per GPU with a total of 8 GPUs. However, we
use one image per GPU for SoS-WSOD due to VRAM limitation in our GPUs, as we
also utilized depth images for each corresponding RGB image. Therefore, the
baseline results of SoS-WSOD reported in Table 1 are slightly lower than those
reported in the original paper. Moreover, we solely use the first stage of
SoS-WSOD since it includes the MIL-based WSOD module which is convenient to
implement our method on top of. The other settings are kept the same as the
official implementations with the VGG16 backbone. To compute the depth prior,
we use only 5 percent of the total annotations from the COCO dataset as the
ground-truth bounding box annotations $B$. Furthermore, we utilize the same
depth range $r_{c,w}$ from the COCO annotations for the Wsod-Amplifier method
on the Conceptual Captions dataset, as the latter lacks ground-truth box
annotations.

###  4.2 Comparing our amplifier to state of the art

| Avg. Precision, IoU | Avg. Precision, Area
---|---|---
Methods on COCO | 0.5:0.95 | 0.5 | 0.75 | S | M | L
MIST [27] | 11.8 | 24.3 | 10.7 | 3.6 | 13.2 | 18.9
\+ Fusion | 13.5 | 26.9 | 12.4 | 4.0 | 14.7 | 21.6
\+ Wsod-Amplifier | 13.8 | 27.7 | 12.3 | 4.6 | 14.7 | 22.4
SoS-WSOD [30] | 10.2 | 21.5 | 8.6 | 2.5 | 10.6 | 17.7
\+ Fusion | 10.3 | 21.6 | 8.9 | 2.3 | 10.8 | 18.4
\+ Wsod-Amplifier | 10.4 | 21.8 | 9.1 | 2.4 | 11.0 | 18.7
| Avg. Precision, IoU | CorLoc
Methods on VOC-07 | 0.5:0.95 | 0.5 | 0.75 | 0.5:0.95 | 0.5 | 0.75
SoS-WSOD [30] | 24.8 | 52.2 | 20.4 | 38.7 | 71.7 | 36.9
\+ Fusion | 26.0 | 53.1 | 22.3 | 39.6 | 72.1 | 38.5

Table 1: This table compares the performance enhancement of our methods, to
their baseline results SoS-WSOD [30] and MIST [27], on COCO and VOC-07. The
best performer per column is in bold.

We evaluate our proposed methods, Fusion and Wsod-Amplifier, on two state-of-
the-art WSOD approaches, SoS-WSOD [30] and MIST [27], and conduct experiments
on COCO and VOC-07 datasets. The performance of our proposed methods are
compared with the baseline methods in Table 1. When our Wsod-Amplifier method
is applied to MIST, it improves the baseline performance by $17\%$ in
$mAP_{50:95}$ (relative gain, 13.8/11.8-1) and $14\%$ in $mAP_{50}$.
Similarly, when our Wsod-Amplifier method is applied to SoS-WSOD, it improves
the baseline performance by $2\%$ in $mAP_{50:95}$, $1.5\%$ in $mAP_{50}$, and
$6\%$ in $mAP_{75}$. As the VOC-07 dataset does not have captions, we are only
able to apply the Siamese-Only and Fusion methods on SoS-WSOD but not the
Depth-Oicr and Depth-Attention. On this dataset, our improvements outperformed
the baseline SoS-WSOD by $5\%$ in $mAP_{50:95}$, $2\%$ in $mAP_{50}$, and
$9\%$ in $mAP_{75}$.

| Avg. Precision, IoU | Avg. Precision, Area
---|---|---
Methods on COCO | 0.5:0.95 | 0.5 | 0.75 | S | M | L
MIST [27] w/ GT | 9.7 | 21.1 | 8.0 | 3.0 | 10.4 | 15.1
MIST [27] w/ EM | 8.5 | 17.9 | 7.3 | 3.0 | 9.4 | 14.9
\+ Siamese-Only | 8.8 | 18.7 | 7.3 | 2.9 | 9.6 | 15.4
\+ Depth-Oicr | 9.1 | 19.4 | 7.4 | 3.0 | 9.8 | 16.0
\+ Depth-Attention | 9.1 | 18.8 | 7.8 | 2.8 | 9.4 | 16.1
\+ Fusion | 9.9 | 20.4 | 8.5 | 3.0 | 10.1 | 17.1
\+ Wsod-Amplifier | 10.2 | 21.1 | 8.4 | 3.2 | 10.3 | 17.4
| Avg. Precision, IoU | Avg. Precision, Area
Methods on CC | 0.5:0.95 | 0.5 | 0.75 | S | M | L
MIST [27] w/ EM | 1.7 | 3.8 | 1.4 | 0.3 | 1.7 | 3.4
\+ Fusion | 2.0 | 4.1 | 1.7 | 0.3 | 1.9 | 4.0
\+ Depth-Oicr | 2.5 | 5.7 | 2.0 | 0.4 | 2.1 | 5.3
\+ Wsod-Amplifier | 2.6 | 6.2 | 2.0 | 0.4 | 2.7 | 5.6

Table 2: This table introduces the effect of each component of our method
implemented on MIST [27] with exact match (EM) labels on COCO (top) and
Conceptual Captions (CC) (bottom). The best performer per column is in bold.
On top, all proposed methods that outperform the Siamese-Only are underlined.

###  4.3 Ablation studies and visualization

Experiments with labels from captions. Several attempts [35, 33, 7] have been
made to eliminate the requirement for image-level labels by leveraging noisy
label information obtained from captions or subtitles. Although it is cost-
effective to use text information for label extraction, it results in a
decrease in the performance of weakly supervised object detection. [33]
propose a text classifier approach to extract labels more effectively than the
simple exact match (EM) and reduce the noise between text and ground truth
(GT) labels. In contrast to previous studies, our research employs the depth
modality to reduce the noise in labels extracted from captions. Our approach
improves the model’s detection capability and employs captions during the
calculation of depth priors. We conducted experiments with MIST [27] using
both GT and EM labels and observed that, as expected, training with GT labels
leads to significantly better performance than training with EM labels in
Table 2 due to the noise in labels extracted from captions. However, our
proposed Wsod-Amplifier method applied on MIST with EM labels surpasses the
baseline and MIST with GT labels. These findings demonstrate that our method
effectively reduces noise and enables the model trained with EM labels to
achieve better performance than those trained with GT labels. It is worth
noting that the text classifier approach proposed by [33] also performs better
than EM-labeled training data, but falls short of the performance achieved by
GT-labeled data.

Results on noisy datasets. We also extract labels from captions on the
Conceptual Captions dataset, which lacks labels at the image level. We also
observe that our Wsod-Amplifier boosts results by an impressive 63% relative
gain using $mAP_{50}$. Conceptual Captions is a noisier dataset than COCO,
since captions were not collected through crowdsourcing, but were crawled as
alt-text for web search results. Thus, it is noteworthy that _the benefit of
our approach becomes more pronounced as the cost of supervision decreases, and
the noise in the supervision increases._

Figure 4: Qualitative comparison of MIST [27] (top) and our proposed Wsod-
Amplifier method (bottom) implemented on MIST.

% of GT anno. | person | car | motorcycle | handbag | bowl | hair drier | toaster | scissors | parking meter | tennis racket
---|---|---|---|---|---|---|---|---|---|---
1% | 0.23 | 0.49 | 0.26 | 0.53 | 0.17 | 0.43 | 0.13 | 0.41 | 0.13 | 0.43 | 0.39 | 0.52 | 0.32 | 0.42 | 0.08 | 0.42 | 0.11 | 0.39 | 0.23 | 0.48
5% | 0.23 | 0.49 | 0.28 | 0.54 | 0.15 | 0.41 | 0.14 | 0.41 | 0.13 | 0.43 | 0.22 | 0.46 | 0.27 | 0.55 | 0.15 | 0.45 | 0.12 | 0.35 | 0.16 | 0.44
10% | 0.23 | 0.49 | 0.28 | 0.54 | 0.16 | 0.41 | 0.14 | 0.42 | 0.13 | 0.43 | 0.27 | 0.47 | 0.26 | 0.54 | 0.15 | 0.46 | 0.16 | 0.44 | 0.16 | 0.43
50% | 0.23 | 0.49 | 0.27 | 0.54 | 0.16 | 0.42 | 0.15 | 0.42 | 0.14 | 0.44 | 0.24 | 0.50 | 0.26 | 0.54 | 0.16 | 0.45 | 0.15 | 0.44 | 0.17 | 0.44
Average | 0.230 | 0.490 | 0.272 | 0.537 | 0.160 | 0.417 | 0.140 | 0.415 | 0.132 | 0.432 | 0.280 | 0.487 | 0.277 | 0.512 | 0.135 | 0.445 | 0.135 | 0.405 | 0.180 | 0.447
Stdev | 0 | 0 | 0.009 | 0.005 | 0.008 | 0.009 | 0.008 | 0.005 | 0.005 | 0.005 | 0.070 | 0.027 | 0.028 | 0.061 | 0.036 | 0.017 | 0.023 | 0.043 | 0.033 | 0.022

Table 3: The table displays the lower and upper depth range values, denoted as
$r_{c}=[\text{lower},\text{upper}]$, for each object class $c$, which are
extracted using varying percentages of ground truth bounding box annotations.
Among the ten object classes considered, the first 5 exhibit the lowest
standard deviations across the different percentages, and the last 5 display
the highest deviations out of the 80 COCO classes.

Analysing the components of our approach. To understand the impact of each
component of our approach on the overall performance, we conducted experiments
with MIST [27] using EM labels as a baseline and applied each component of our
method on top of the baseline in Table 2. Our Siamese-Only method, which
incorporates the depth modality in the Siamese network using contrastive
learning, improves feature extraction and results in a $4\%$ increase in
$mAP_{50:95}$ and $mAP_{50}$. Our Depth-Oicr method, which utilizes depth
priors in the OICR module to improve the mining strategy, increases
$mAP_{50:95}$ and $mAP_{50}$ over Siamese-Only by $7-8\%$ on COCO and
$47-50\%$ on Conceptual Captions (CC). Our Depth-Attention method, which
incorporates depth priors to use potentially important regions in an attention
mechanism with combined score probabilities, increases $AP_{50:95}$ and
$mAP_{75}$ by $6-7\%$. Our Fusion method, which combines RGB and depth image
scores, improves by $14-16\%$ on COCO and $7-21\%$ on CC. Comparing Fusion and
Depth-Oicr, the bigger gain using $mAP_{50}$ is achieved by Fusion on COCO,
and Depth-Oicr on CC. _Thus, the benefit of our WSOD-specific method component
increases as the noise in the dataset increases, which is appealing due to its
real-world applicability._ Finally, our Wsod-Amplifier method, which includes
all components of our approach, achieves the highest performance increase over
MIST w/ EM baseline, with _improvements in all $mAP$ metrics by $18-20\%$ on
COCO and $42-63\%$ on CC._

Methods | Clipart | Watercolor | Comic
---|---|---|---
MIST [27] w/ GT | 9.4 | 13.3 | 9.2
\+ Wsod-Amplifier | 10.2 | 14.7 | 9.6

Table 4: This table introduces the improvement of our Wsod-Amplifier over MIST
on domain shift datasets. The results are in $mAP_{50}$ metric. The best
performer per column is in bold.

Generalization of depth priors. Even though we apply the depth priors
calculated from COCO to CC, without any bounding boxes available in CC, our
proposed method exhibits a more substantial enhancement in CC performance
compared to COCO, achieving an improvement of $63\%$ $mAP_{50}$ over the MIST
baseline (50% improvement from Depth-Oicr alone). Thus, our Depth-Oicr method
has a higher impact than Fusion on CC, in contrast to COCO. Given the recent
interest in learning from vision-language data, our approach has the potential
to be highly impactful.

Generalization to appearance changes. By relying on depth information, our
method builds some robustness to overfitting to appearance, which may not be
the same across datasets. To test this hypothesis, we conduct experiments with
domain shift datasets [15]. Table 4 shows that our Wsod-Amplifier boosts the
performance of MIST baseline in $mAP_{50}$ by $4-10\%$, even though no
training was performed on these datasets.

Analyzing the depth priors calculated with varying percentages of GT
annotations. In this section, we examine how the proportion of ground truth
(GT) annotations affects the ability to estimate the relative depth range of
objects. Table 3 presents the average depth ranges and standard deviations for
ten objects at various annotation percentages. Across all 80 objects, the
average difference between the upper and lower bounds is $0.26$, while the
average standard deviation is a mere $0.01$. This indicates that the standard
deviation is significantly lower than the average range difference. Although
we utilize $5\%$ percent of annotations as default in our experiments, our
findings suggest that even utilizing only $1\%$ of GT annotations is
sufficient to approximate depth priors successfully.

Qualitative analysis. We visualize the object detection performance
improvement of our proposed Wsod-Amplifier method in comparison to MIST [27]
in Figure 4. In the first image, the baseline struggles to accurately identify
multiple instances of the same “vase” objects, instead grouping them together
in a single bounding box with a high score. Our method overcomes this
challenge, precisely detecting each individual “vase” instance. In the second
image, the baseline faces the problem of part domination due to some
discriminative parts of a “zebra” object. Our proposed method helps to
overcome this issue by utilizing depth modality, which emphasizes the
geometric variations of objects, while comparatively ignoring the complex
background. In other images, unlike our method, the baseline misses objects
entirely, or produces large and imprecise bounding boxes. Moreover, the
bounding boxes detected by our method tend to have higher prediction scores.

##  5 Conclusion

We have demonstrated how the depth modality can be useful for weakly-
supervised object detection, without incurring annotation or significant
computation costs. Our Siamese WSOD network efficiently incorporates RGB and
depth modality, along with contrastive loss and fusion. With the relationship
of language and depth, depth priors estimate the bounding box proposals that
may contain an object of interest. We have implemented our boosting approach
on two WSOD methods, SoS-WSOD, and MIST and significantly increased their
performance.

## References

  * [1] Hakan Bilen and Andrea Vedaldi.  Weakly supervised deep detection networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2846–2854, 2016.
  * [2] P Cammack and JM Harris.  Depth perception in disparity-defined objects: finding the balance between averaging and segregation.  Philosophical Transactions of the Royal Society B: Biological Sciences, 371(1697):20150258, 2016.
  * [3] Hao Chen and Youfu Li.  Progressively complementarity-aware fusion network for rgb-d salient object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3051–3060, 2018.
  * [4] Kai Chen, Hang Song, Chen Change Loy, and Dahua Lin.  Discover and learn new objects from documentaries.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [5] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International journal of computer vision, 88:303–308, 2009.
  * [6] Deng-Ping Fan, Zheng Lin, Zhao Zhang, Menglong Zhu, and Ming-Ming Cheng.  Rethinking rgb-d salient object detection: Models, data sets, and large-scale benchmarks.  IEEE Transactions on neural networks and learning systems, 32(5):2075–2089, 2020.
  * [7] Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt.  Data determines distributional robustness in contrastive language image pre-training (clip).  In International Conference on Machine Learning (ICML), pages 6216–6234. PMLR, 2022.
  * [8] Guang Feng, Jinyu Meng, Lihe Zhang, and Huchuan Lu.  Encoder deep interleaved network with multi-scale aggregation for rgb-d salient object detection.  Pattern Recognition, 128:108666, 2022.
  * [9] Keren Fu, Deng-Ping Fan, Ge-Peng Ji, and Qijun Zhao.  Jl-dcf: Joint learning and densely-cooperative fusion framework for rgb-d salient object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3052–3062, 2020.
  * [10] Keren Fu, Deng-Ping Fan, Ge-Peng Ji, Qijun Zhao, Jianbing Shen, and Ce Zhu.  Siamese network for rgb-d salient object detection and beyond.  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 44(9):5541–5559, 2021.
  * [11] Mingfei Gao, Chen Xing, Juan Carlos Niebles, Junnan Li, Ran Xu, Wenhao Liu, and Caiming Xiong.  Open vocabulary object detection with pseudo bounding-box labels.  In European Conference on Computer Vision (ECCV), 2022.
  * [12] Michael Gutmann and Aapo Hyvärinen.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 297–304. JMLR Workshop and Conference Proceedings, 2010.
  * [13] Junwei Han, Hao Chen, Nian Liu, Chenggang Yan, and Xuelong Li.  Cnns-based rgb-d saliency detection via cross-view transfer and multiview fusion.  IEEE Transactions on Cybernetics, 48(11):3171–3183, 2017.
  * [14] Tanveer Hussain, Abbas Anwar, Saeed Anwar, Lars Petersson, and Sung Wook Baik.  Pyramidal attention for saliency detection.  In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 2877–2887. IEEE, 2022.
  * [15] Naoto Inoue, Ryosuke Furuta, Toshihiko Yamasaki, and Kiyoharu Aizawa.  Cross-domain weakly-supervised object detection through progressive domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5001–5009, 2018.
  * [16] Jianbo Jiao, Yunchao Wei, Zequn Jie, Honghui Shi, Rynson WH Lau, and Thomas S Huang.  Geometry-aware distillation for indoor semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2869–2878, 2019.
  * [17] Minhyeok Lee, Chaewon Park, Suhwan Cho, and Sangyoun Lee.  Spsn: Superpixel prototype sampling network for rgb-d salient object detection.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIX, pages 630–647. Springer, 2022.
  * [18] Jingjing Li, Wei Ji, Qi Bi, Cheng Yan, Miao Zhang, Yongri Piao, Huchuan Lu, et al.  Joint semantic mining for weakly supervised rgb-d salient object detection.  Advances in Neural Information Processing Systems, 34:11945–11959, 2021.
  * [19] Yabei Li, Zhang Zhang, Yanhua Cheng, Liang Wang, and Tieniu Tan.  Mapnet: Multi-modal attentive pooling network for rgb-d indoor scene classification.  Pattern Recognition, 90:436–449, 2019.
  * [20] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [21] Johannes Meyer, Andreas Eitel, Thomas Brox, and Wolfram Burgard.  Improving unimodal object recognition with multimodal contrastive learning.  In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5656–5663. IEEE, 2020.
  * [22] S Mahdi H Miangoleh, Sebastian Dille, Long Mai, Sylvain Paris, and Yagiz Aksoy.  Boosting monocular depth estimation models to high-resolution via content-adaptive multi-resolution merging.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9685–9694, 2021.
  * [23] Yongri Piao, Wei Ji, Jingjing Li, Miao Zhang, and Huchuan Lu.  Depth-induced multi-scale recurrent attention network for saliency detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7254–7263, 2019.
  * [24] Liangqiong Qu, Shengfeng He, Jiawei Zhang, Jiandong Tian, Yandong Tang, and Qingxiong Yang.  Rgbd salient object detection via deep fusion.  IEEE Transactions on Image Processing, 26(5):2274–2285, 2017.
  * [25] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun.  Vision transformers for dense prediction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12179–12188, 2021.
  * [26] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 44(3):1623–1637, 2020.
  * [27] Zhongzheng Ren, Zhiding Yu, Xiaodong Yang, Ming-Yu Liu, Yong Jae Lee, Alexander G Schwing, and Jan Kautz.  Instance-aware, context-focused, and memory-efficient weakly supervised object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [28] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018.
  * [29] Hwanjun Song, Eunyoung Kim, Varun Jampan, Deqing Sun, Jae-Gil Lee, and Ming-Hsuan Yang.  Exploiting scene depth for object detection with multimodal transformers.  In 32nd British Machine Vision Conference (BMVC), pages 1–14. British Machine Vision Association (BMVA), 2021.
  * [30] Lin Sui, Chen-Lin Zhang, and Jianxin Wu.  Salvage of supervision in weakly supervised object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14227–14236, 2022.
  * [31] Peng Tang, Xinggang Wang, Song Bai, Wei Shen, Xiang Bai, Wenyu Liu, and Alan Yuille.  Pcl: Proposal cluster learning for weakly supervised object detection.  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 42(1):176–191, 2018.
  * [32] Peng Tang, Xinggang Wang, Xiang Bai, and Wenyu Liu.  Multiple instance detection network with online instance classifier refinement.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [33] Mesut Erhan Unal, Keren Ye, Mingda Zhang, Christopher Thomas, Adriana Kovashka, Wei Li, Danfeng Qin, and Jesse Berent.  Learning to overcome noise in weak caption supervision for object detection.  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022.
  * [34] Fang Wan, Chang Liu, Wei Ke, Xiangyang Ji, Jianbin Jiao, and Qixiang Ye.  C-mil: Continuation multiple instance learning for weakly supervised object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [35] Keren Ye, Mingda Zhang, Adriana Kovashka, Wei Li, Danfeng Qin, and Jesse Berent.  Cap2det: Learning to amplify weak caption supervision for object detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), pages 9686–9695, 2019.
  * [36] Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Long Mai, Simon Chen, and Chunhua Shen.  Learning to recover 3d scene shape from a single image.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 204–213, 2021.
  * [37] Xiaowen Ying and Mooi Choo Chuah.  Uctnet: Uncertainty-aware cross-modal transformer network for indoor rgb-d semantic segmentation.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXX, pages 20–37. Springer, 2022.
  * [38] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang.  Open-vocabulary object detection using captions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14393–14402, 2021.
  * [39] Feng Zhou, Yu-Kun Lai, Paul L Rosin, Fengquan Zhang, and Yong Hu.  Scale-aware network with modality-awareness for rgb-d indoor semantic segmentation.  Neurocomputing, 492:464–473, 2022.
  * [40] Chunbiao Zhu, Xing Cai, Kan Huang, Thomas H Li, and Ge Li.  Pdnet: Prior-model guided depth-enhanced network for salient object detection.  In 2019 IEEE International Conference on Multimedia and Expo (ICME), pages 199–204. IEEE, 2019.