# A new technique to establish the uniformity of the induction gap in GEM
based detector

Mohit Gola Corresponding Author<EMAIL_ADDRESS>F. Brunbauer  European
Organization for Nuclear Research (CERN)
CH-1211 Geneve 23
Switzerland  D. Janssens  European Organization for Nuclear Research (CERN)
CH-1211 Geneve 23
Switzerland  A. Kumar  Department of Physics $\&$ Astrophysics
University of Delhi
Delhi, India  M. Lisowska  European Organization for Nuclear Research (CERN)
CH-1211 Geneve 23
Switzerland  H. Muller  European Organization for Nuclear Research (CERN)
CH-1211 Geneve 23
Switzerland  Physikalisches Institut, University of Bonn
Nußallee 12, 53115 Bonn
Germany  Md. Naimuddin  Department of Physics $\&$ Astrophysics
University of Delhi
Delhi, India  E. Oliveri  European Organization for Nuclear Research (CERN)
CH-1211 Geneve 23
Switzerland  D. Pfeiffer  European Organization for Nuclear Research (CERN)
CH-1211 Geneve 23
Switzerland  European Spallation Source (ESS ERIC)
P.O. Box 176, SE-22100 Lund
Sweden  L. Ropelewski  European Organization for Nuclear Research (CERN)
CH-1211 Geneve 23
Switzerland  L. Scharenberg  European Organization for Nuclear Research (CERN)
CH-1211 Geneve 23
Switzerland  Physikalisches Institut, University of Bonn
Nußallee 12, 53115 Bonn
Germany  M. Van Stenis  European Organization for Nuclear Research (CERN)
CH-1211 Geneve 23
Switzerland  A. Utrobicic  European Organization for Nuclear Research (CERN)
CH-1211 Geneve 23
Switzerland  R. Veenhof  European Organization for Nuclear Research (CERN)
CH-1211 Geneve 23
Switzerland

###### Abstract

This work is proposing and exploring the use of multichannel readout
electronics, already used in quality assurance for gain uniformity studies, to
measure the uniformity of the induction gap in GEM based detectors. The
measurement will furthermore provide a qualification of the readout electrodes
in terms of disconnected or shorted channels. The proposed method relies on
the dependence on the induction gap capacitance between the readout strips and
the bottom of the GEM. The measurement is obtained inducing capacitively a
signal on the readout strips pulsing the bottom layer of the GEM foil. In this
work, the signals are read with the analog APV25 front-end chip and the RD51
Scalable Readout System (SRS). Studies on small and large area GEM detectors,
relative variations under mechanical stress, and in presence of electrical
fields will be shown. Sensitivity to defects in the readout plane will be
proven.

Key Words– MPGD, APV25, GEM, SRS

##  1 Introduction

Many existing experiments and their upgrades in High Energy Physics (HEP) are
proposing and utilizing Micro Pattern Gas Detectors (MPGDs) to have large area
coverage [2, 3, 4]. Quality controls (QA/QCs) are important to validate
detectors and achieved performances before they will be installed in
experiments [5, 6, 7]. QA/QCs procedure have to be reliable, reproducible and
with possibly as many information as needed. Along this line, a method has
been developed to qualify the induction gap in a Gas Electron Multiplier (GEM)
based detector [8, 9, 10, 11]. The flatness of the induction gap is an
important parameter because the gain of the GEM detector is linearly dependent
upon it. Non-uniformity in this gap causes variation of gain over the surface
(i.e. the gain uniformity) affecting overall energy resolution, dynamic range
of the signal, compatibility with the electronics, and detector stability.
This work proposes the use of multichannel analog readout chips, already in
use for gain uniformity qualification, to measure the uniformity of the
induction gap. Direct comparison of these two measurements will be very
beneficial and it will not add complication to existing QA/QCs procedure. The
result of this measurement does not only help in QA/QCs procedure but also
helps in R&D measurements in the validation of the prototypes.
The structure of the paper is as follows: section 2 explains the technique
developed and described in this paper, section 3 explains the experimental
setup. Section 4 describes the set of measurements performed in order to
validate and demonstrate the capability of this method. Finally, we will
conclude our findings.

##  2 Methodology

The method used in this paper relies on the indirect measurement of the gap
through the capacitance between readout electronics and bottom GEM. An
external pulse is sent to the bottom of the GEM, and the amplitude of the
capacitively induced signal on readout strips is measured. The measurement can
be done with or without high voltage applied to the detector. This method
requires a multichannel analog front-end chip and it should be calibrated to
disentangle the non-uniformity of the chip with the detector. The caveats
associated with this technique that it is sensitive to all the types of
coupling. In the paper we will show for instance the effect of the readout
strips fan-out. The method, being sensitive to the electrode capacitance, can
be used to qualify readout electrodes. Shorts or missing connections will be
identified for instance because of the anomalous electrode capacitance.

##  3 Setup

The setup shown in the Figure 1, can divided into two parts: the charge
injection circuity and the multichannel readout system based on APV25 and SRS
[12, 13]. The external pulse has been generated using the 25 GHz pulse
generator with a definite pulse frequency of 1 kHz, amplitude 1.15 V peak to
peak, width 1 $\mu$S, and delay of 300 ns. The signal from the pulse generator
fed into the bottom of the GEM foil in series with 100 pF capacitor and a 50
$\Omega$ termination. The external pulser has been synchronized with the SRS
clock using a synchronous NIM signal produced by the SRS.

Figure 1: Shows the schema for the technique describe in this paper.

The data was recorded using the SRS shown in Figure 2, which has been
developed by the RD51 collaboration at CERN. The CMS GEM collaboration use
this system for the quality assurance of the GEM detector [14, 15, 16].

Figure 2: Shows the schematic os the Scalable Readout System(SRS).

The SRS has been used to control and read the APV25 front end ASIC. APV25 has
128 channels that will be digitized with a sampling clock of 40MHz as shown in
the Figure 3 (a), the details of the chip can be found elsewhere [17, 18]. In
order to evaluate non uniformities intrinsic to the chip and the RD51 hybrids,
a calibration scan with internal pulses has been done. For the calibration, 16
strips has been fired at the same time and data have been later combined.
Figure 3 (b) shows the ADC value recorded for the different amplitudes of the
input pulse. For this, three pairs of master and slave APV has been used, out
of which master APVs were connected to the readout board and slave APVs were
left floating. A similar pattern has been observed for both the APVs either
connected to readout (RO) or left floating hence there is finite non-
uniformity present in the channels of the chip.

(a)

(b)

Figure 3:  (a) Shows the pair of APV chip , and (b) Shows the behaviour of
APV-pair operated in calibration mode for different amplitude input pulse.

##  4 Measurements

A series of measurements have been carried out on the small and large size GEM
detectors in order to extract the information regarding the flatness of the
induction gap. Also, the application of this technique has been described in
terms of deformation of the induction gap, quality assurance of the readout
board, etc.

###  4.1 Sensitivity of the induction gap

The aim of this measurement is to check the sensitivity of the signal gap
utilizing the capacitance of the gap. The experimental setup for this
measurement uses the two-dimensional RO board having an active area of 10 cm X
10 cm and consists of 256 strips in X-direction and the same in Y-direction.

(a)

(b)

Figure 4:  (a) Shows the different orientations for GEM & Anode plane, and (b)
shows the ADC value for each strip for all the three configurations.

A GEM foil having an active area similar to the RO and it’s placed on top of
the RO board [19, 20]. The output of the pulse generator has been connected to
the bottom of the HV trace of GEM foil. Panasonic connectors in X-direction
were equipped with the APV25 chip and Y-direction strips were terminated with
0 $\Omega$. Three different gap configurations were tested in order to look
for the induction gap sensitivity as shows in the Figure 4 (a). Firstly, a gap
of 2 mm was maintained between the bottom of the GEM foil and anode then the
foil was titled with the asymmetric gap of 1.5 mm on the left and 2 mm on the
right, and vice-versa. The Figure 4 (b) shows a clear variation in the ADC
value with respect to strip has been observed for the three different gap
configurations.

###  4.2 Large area triple-GEM detector

After testing this technique on 10 cm $\times$ 10 cm GEM detector, it has been
applied to the large ($O(m^{2})$) size triple-GEM detector with known non-
uniformity in the induction gap. For this measurement, a triple-GEM detector
having a physical dimension of 1 m $\times$ 55 cm $\times$ 24 cm with a gap
configuration of 3 mm/1 mm/2 mm/1 mm for the
Drift/Transfer1/Transfer2/induction gap, respectively has been used. The
readout board for this detector is 3 mm thick PCB having in total 3072 one-
dimensional strips and having known deformation in the detector.

Figure 5: Shows the ADC value for each strip and 3-Eta segments of the triple-
GEM detector.

The RO board is divided into 8 eta segment and 3 phi partitions such that each
(eta, phi) sector has 128 strips. RO board has different strip lengths for
each eta segment as going from the wider side to the narrow side. A high
voltage divider has been used to power up the GEM foils and gaps
simultaneously [21]. An external pulse was sent to the bottom of the third GEM
foil in parallel to the divider, and a scan of three eta segment has been
performed as shown in the Figure 5. It shows that the detector is not
completely flat and having a clear manifestation of the bedding (outward) in
the RO board and shows the fan-out effect of the readout strips. Due to
different strip length and width in each eta sector, a correction on
capacitance has been applied for two particular eta sector with respect to the
eta sector on the wider side as shown in the Figure 6.

Figure 6: Shows the capacitance correction applied on the three particular Eta
sectors namely Eta1, Eta3 and Eta8.

###  4.3 Deformation of the induction gap

In the next test, we deformed the gap and the detector on purpose. In order to
demonstrate this a weight of roughly 5 kg has been placed on one of the eta
segment and look for the ADC value with respect to the strip. The two plots
i.e. with weight placed on RO and without are plotted together to see the
difference as shown in the Figure 7. The plot with weight having a higher ADC
value for the particular area, where the weight was placed, hence shows the
bending of the RO board in the opposite direction i.e. inward. The
distribution for both the measurements is fitted with a $\rm 2^{nd}$ order
polynomial function to visualize the deformation of the RO board.

Figure 7: Shows the ADC value for each strip for one of the Eta segment With
and without 5kg weight placed.

###  4.4 Test in presence of the induction field

The method has been tested as well with high voltage applied, to prove that it
can done as well when all the electrical fields are applied. To understand
this a 700 V has been applied to the bottom of the third GEM foil and at the
same divider pad, an external pulse was also sent.

Figure 8: Shows the ADC value for each strip for one of the Eta segment when
HV is ON and OFF.

Measurement has been carried when the high voltage is ON as well as OFF and
two spectra were plotted together as shown in the Figure 8. No variation in
the ADC spectrum with respect to strip number observed, hence it shows that
this technique works in the presence of high voltage and can be used to
monitor the if electrostatic force will induce variation in the gap. Also,
this measurement confirms that the mechanical stretching used to stretch the
foils is sufficient to overcome the sag due to running high voltage.

###  4.5 Readout electrodes integrity

The advantage of using a multichannel front-end chip is to locate the
defect(s) present in each strip. Measurement has been carried in the readout
sector where few strips shorted as well as broken. The ADC spectrum with
respect to strip has been plotted as shown in the Figure 9 and observed that
due to change in capacitance for shorted, broken, and damaged strips, a higher
value of ADC was observed. And compared to the physical location of the bad
strips on the RO board. Hence this technique can be used to do the quality
assurance of the readout boards well before using them in detector assembly.

Figure 9: Shows the ADC value for each strip for one of the Eta segment having
defects present in the strips of the RO.

##  5 Conclusion

We explored a measurement technique sensitive to the local size of the
induction gap of GEM based detectors. In this work the measurement has been
performed using the RD51 SRS/APV25 as analog multichannel readout system. The
aim of this work was to give a proof of principle of the technique. Potential
as been shown as well limiting factors as spurious effects coming for instance
from additional capacitances not dependent in the gap as the one associated to
the strips fanout. For a proper and quantitative analysis, aspects like this
one have to be properly taken into account. An outstanding feature of this
technique is that it can be used under the operation of the high voltage which
means that this technique can be applied at the field configuration used in
the experiment. Also, it can be used for the quality assurance of the readout
boards manufactured by the industries to detect possible defects during
production. This technique can be used for any gaseous detector properly
couple a plane to any readout.

##  6 Acknowledgements

We would like to acknowledge the funding agency, Department of Science and
Technology (DST), New Delhi (grant nos. SR/MF/PS-02/2014-DUA(G), and
SB/FTP/PS-165/2013) for providing financial support. We would also like to
thanks the RD51 collaboration at CERN to provide the facility and support.

## References

  * [1] S. D. Pinto et al., "Progress on large area GEMs", JINST 4 P12009 (2009).
  * [2] CMS Collaboration, "Overview of large area triple-GEM detectors for the CMS forward muon upgrade", NIMA 845 (2017) 298-303.
  * [3] P. Karchin, Performance of a Large-Area Triple-GEM Detector in a Particle Beam, Physics Procedia, 37 (2012) 561-566.
  * [4] D. Abbaneo, M. Gola et al., "Performance of GE1/1 Chambers for the CMS Muon Endcap Upgrade", arXiv:1903.02186.
  * [5] CMS collaboration, "Quality control and beam test of GEM detectors for future upgrades of the CMS muon high rate region at the LHC", JINST (2015) 10 C03039.
  * [6] M. Gola on behalf of CMS Collaboration; Muon Chamber End-cap Upgrade of the CMS Experiment with Gas Electron Multiplier (GEM) Detectors and their Performance; Springer Proc. in Phy. 203 (2018) 591.
  * [7] E. Soldani, "Production and quality control of GEM detectors for the phase 1 upgrade of the CMS experiment", IL NUOVO CIMENTO 41 C (2018) 90.
  * [8] S.D. Pinto 2010 Micropattern gas detector technologies and applications the work of the RD51 collaboration, IEEE Nucl. Sci. Symp. Conf. Rec. 802.
  * [9] F. Sauli, GEM: A new concept for electron amplification in gas detectors, Nucl. Instrum. Meth. A, 386, 1997: 531, doi:10.1016/S0168-9002(96)01172-2.
  * [10] F. Sauli, The gas electron multiplier (GEM): Operating principles and applications, Nucl. Instrum. Meth. A, 805, 2016; 2, doi:10.1016/j.nima.2015.07.060.
  * [11] F. Sauli, Gas Electron Multiplier (GEM) Detector: Principles of operation and applications, RD51-NOTE-2012-007.
  * [12] S Martoiu, H Muller, A Tarazon and J Toledo, "Development of the scalable readout system for micro-pattern gas detectors and other applications", JINST 8, 2013.
  * [13] Sorin Martoiu, Hans Muller, J. Toledo, "Front-end electronics for the Scalable Readout System of RD51", DOI:10.1109/NSSMIC.2011.6154414.
  * [14] A. Shah, A. Ahmad, M. Gola, R. Sharma, S. Malhotra, A. Kumar, Md. Naimuddin, P. Menon, K. Srinivasan, "Development, Characterization and Qualification of first GEM foils produced in India"; NIM A 892 (2018) 10-17.
  * [15] M. Gola, S. Malhotra, A. Shah, A. Ahmed, A. Kumar, Md. Naimuddin, "Performance of the triple GEM detector built using commercially manufactured GEM foils in India"; NIM A 951 (2019).
  * [16] M. Gola, S. Malhotra, A. Kumar, Md. Naimuddin, "Stability test performed on the triple GEM detector built using commercially manufactured GEM foils in India";2019 JINST 14 p08004.
  * [17] M. Raymond et al., " The APV25 0.25 m CMOS readout chip for the CMS tracker", IEEE Nucl. Sci. Symp. Conf. Rec. 2 9/113.
  * [18] Lawrence Jones (RAL), "APV25 user guide", CERN CDS-002725643.
  * [19] A. Bressan et al.,"Two-dimensional readout of GEM detectors", DOI:10.1016/S0168-9002(98)01405-3.
  * [20] A.Bondar et al., "Performance of the triple-GEM detector with optimized 2-D readout in high intensity hadron beam", arXiv 0110054.
  * [21] J. A. Merlin, Study of long-term sustained operation of gaseous detectors for the high rate environment in CMS, CERN-THESIS-2016-041.


# OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation
Dataset with Visual Contexts

Shuhe Wang♣♠, Yuxian Meng♣, Xiaoya Li♣
Xiaofei Sun♣, Rongbin Ouyang♠ Jiwei Li◆♣
♣ Shannon.AI, ♠Peking University, ◆Zhejiang University
{shuhe_wang, yuxian_meng, xiaoya_li, xiaofei_sun<EMAIL_ADDRESS>

###### Abstract

In order to better simulate the real human conversation process, models need
to generate dialogue utterances based on not only preceding textual contexts
but also visual contexts. However, with the development of multi-modal
dialogue learning, the dataset scale gradually becomes a bottleneck. In this
report, we release OpenViDial 2.0, a larger-scale open-domain multi-modal
dialogue dataset compared to the previous version OpenViDial 1.0 (Meng et al.,
2020). OpenViDial 2.0 contains a total number of 5.6 million dialogue turns
extracted from either movies or TV series from different resources, and each
dialogue turn is paired with its corresponding visual context. We hope this
large-scale dataset can help facilitate future researches on open-domain
multi-modal dialog generation, e.g., multi-modal pretraining for dialogue
generation.111Dataset is available found at
https://github.com/ShannonAI/OpenViDial.

##  1 Introduction

Developing open-domain dialogue agents is of growing interest (Li et al.,
2017; Ghazvininejad et al., 2017; Zhou et al., 2017; Gao et al., 2018; Asghar
et al., 2018; Han et al., 2020a; Zhou et al., 2020). Existing methods for
developing effective open-domain dialogue agents mostly follow a two-step
pipeline: (1) collecting a large-scale dataset containing massive dialog turns
from real conversations, and (2) training a neural model to learn to generate
high quality responses given the previous dialogue contexts Li et al. (2016b,
a); Zhang et al. (2018); Huang et al. (2020).

Since most methods are data-driven, a large-scale and high quality open-domain
dialogue datasets may be the first matter to be considered before designing
the model. Meng et al. (2020) released the OpenViDial dataset which contains a
total number of 1.1 million dialogue turns with utterances paired with visual
context. Some recent works leveraged the OpenViDial dataset and built
effective multi-modal dialog models (Wang et al., 2021) on top, demonstrating
that learning multi-modal features gives rise to higher response quality.

In this report, we collect and extend OpenViDial, releasing OpenViDial 2.0, a
much larger-scale open-domain dialogue dataset with visual contexts. In common
with the prior version OpenViDial 1.0 (Meng et al., 2020), the dialogue turns
and visual contexts in OpenViDial 2.0 are also extracted from movies and TV
series, where each dialogue turn is paired with the corresponding visual
context in which it takes place. OpenViDial 2.0 contains a total number of 5.6
million dialogue turns along with 5.6 million visual contexts stored as
images, a scale of 4 times larger than OpenViDial 1.0. We hope this large-
scale dataset can help facilitate future researches on open-domain multi-modal
dialog generation, e.g., multi-modal pretraining for dialogue generation.

##  2 Related Work

###  2.1 Open Domain Dialog Datasets

#### Textual Dialog Datasets

Since the task of open-domain dialog generation has developed for many years,
there are various open-domain dialog datasets only consists textual
information. For simulating the movie conversation, there are OpenSubtitle
dataset (Tiedemann, 2009; Lison and Tiedemann, 2016) and Cornell Movie-Dialogs
Corpus (Danescu-Niculescu-Mizil and Lee, 2011). The OpenSubtitle dataset is a
large-scale dataset contains a total number of 3.35G sentence fragments
extracted from the OpenSubtitle website, while the Cornell Movie-Dialogs
Corpus contains a collection of movie conversations extracted from raw movie
scripts. For simulating the social conversation, there are PersonaChat Zhang
et al. (2018) and Twitter Triple Corpus Sordoni et al. (2015). The Twitter
Triple Corpus consists of 4,232 Twitter conversation triples evaluated from
33K candidate triples by human raters. Other datasets such as the Ubuntu
Dialog Corpus Lowe et al. (2015) and EmpatheticDialogues Rashkin et al. (2018)
are both commonly used for textual open-domain dialog generation.

#### Visual Dialog Datasets

A mount of datasets containing visual features have been developed, since the
task of VisualDialog is first introduced by Das et al. (2017a), where a model
is required to answer questions by given a dialog history and the image itself
as contexts. For this work, Das et al. (2017a) released VisDial v0.9 and v1.0
datasets which contains 120K images from MSCOCO222http://mscoco.org/ and each
image is associated with 10 rounds of question-answer dialog. Further, other
datasets like the GuessWhat?! dataset de Vries et al. (2017), the CLEVERDialog
dataset Kottur et al. (2019), the MNIST-Dialog dataset Seo et al. (2017) and
the Audio Visual Scene-Aware Dialog (AVSD) dataset (Hori et al., 2018; Alamri
et al., 2019) are mainly focus more on answering questions according to an
image or video rather than dialogue generation with visual contexts.The
OpenViDial dataset Meng et al. (2020) is released to alleviate this situation,
where contains 1.1M dialogue turns and each dialogue turn paired with the
corresponding visual context in which it takes place. And thus, models need to
learn to generate dialogue utterances not only based on preceding textual
contexts but also visual contexts.

###  2.2 Dialog Generation

#### Open Domain Dialog Generation

Open-domain dialog generation is a simulation for real human conversations and
is a traditional task in NLP (Weizenbaum, 1966; COLBY, 1975; Wallace, 2009).
Currently, the most researches for open-domain dialog generation are based on
sequence-to-sequence architecture (Vinyals and Le, 2015; Li et al., 2015;
Dodge et al., 2016; Serban et al., 2016; Zhao et al., 2017; Xie et al., 2017;
Lee et al., 2019; Ghandeharioun et al., 2019; Li, 2020; Han et al., 2020b;
Zhang et al., 2019; Roller et al., 2020). And whether a model can generate
diverse (Xu et al., 2018; Baheti et al., 2018), coherent (Li et al., 2016b,
2017; Tian et al., 2017; Bosselut et al., 2018; Adiwardana et al., 2020),
informative (Shao et al., 2017; Lewis et al., 2017; Ghazvininejad et al.,
2017; Young et al., 2017; Zhao et al., 2019) and knowledge-fused (Hua et al.,
2020; Zhao et al., 2020; He et al., 2020) responses or not has become metrics
to evaluate a dialog generation model. However, the mainly researches
described above are developed on textual only and the development of multi-
modal dialog generation is relatively slow since the lack of large-scale
datasets.

#### Visual Dialog Generation

Most of existing works apply attention mechanisms to model the interplay
between text and visual contexts (Lu et al., 2017; Kottur et al., 2018; Jiang
and Bansal, 2019; Yang et al., 2019; Guo et al., 2019; Niu et al., 2019; Kang
et al., 2019; Park et al., 2020; Jiang et al., 2020b). Other techniques like
reinforcement learning (Das et al., 2017b; Wu et al., 2018), variational auto-
encoders Massiceti et al. (2018) and graph networks (Zheng et al., 2019; Jiang
et al., 2020a) have also been employed to the visual dialog task. More
recently, based on the OpenViDial dataset Meng et al. (2020), Wang et al.
(2021) proposed three attention-based models Vaswani et al. (2017) to generate
dialogue utterances given the preceding text-visual contexts and further
proposed to build text-visual dependency to improve the dialogue quality,
making an initial step for the task of text-visual open-domain dialogue
generation rather than answering questions based on an image.

Statistics | OpenViDial 1.0 | OpenViDial 2.0
---|---|---
Number of turns | 1.1M | 5.6M
Number of images | 1.1M | 5.6M
Vocab size before BPE | 70K | 278K
Vocab size after BPE | 30K | 30K
Average length of each episode | 14 | 48
Average length of each turn | 7.6 | 8.3
Table 1: Detailed statistics for OpenViDial 2.0 and a comparison to OpenViDial 1.0. | OpenViDial 1.0 | OpenViDial 2.0
---|---|---
Train | 1M | 4.6M
Dev | 50K | 0.5M
Test | 50K | 0.5M
Table 2: Splitting for training, dev and test

##  3 Constructing OpenViDial 2.0

In this section, we describe the details of constructing of OpenViDial 2.0. We
first collect a raw dataset consisting of about 800 English movies and TV
series with an average length of 2.5 hours per video. Each video has a
corresponding external English subtitle file where each line is a string
including the subtitle text and the time interval. There is no video embedded
with any internal subtitles.

The full process to construct OpenViDial 2.0 can be divided into three steps:
(1) segmenting each video into multiple frames; (2) pairing each frame with
subtitle text from it corresponding subtitle file; (3) splitting these (image,
text) pairs into different dialog turns. The OpenCV Bradski (2000) toolkit is
used to segment each video into multiple images by frame, and we discard the
initial and the last 10 minutes of each video because of the general existence
of intro in movies and TV series. To pair images with textual subtitles for
each video, we first read the video’s subtitle file row-by-row and obtain the
time interval as well as the subtitle text. Then, we extract a group of images
according to the time interval, and randomly choose one image from the group
as the visual context paired with the subtitle text, forming a paired (image,
text) dialog turn.

We are able to construct a final dataset of 5.6M dialog turns, where each turn
consists of a sequence of words and an image. The size of the image is one of
(1) 1280$\times$720, (2) 1920$\times$1080, and (3) 2048$\times$1080 according
to different video resources. We employ the BPE tokenizer Sennrich et al.
(2016) to preprocess the text. A detailed comparison with OpenViDial 1.0 is
shown in Table 1. The splitting for training, dev and test is shown in Table
2.

Dataset | Genre | Multi-Modal? | # Sentences | # Images
---|---|---|---|---
OpenSubtitles 2016 (Lison and Tiedemann, 2016) | Plain-text Dialog | ✗ | 337M | –
Cornell Movie-Dialogs (Danescu-Niculescu-Mizil and Lee, 2011) | Plain-text Dialog | ✗ | 0.3M | –
VisDial v1.0 (Das et al., 2017a) | VQA | ✓ | 2.4M | 120K
Guess-What?! (de Vries et al., 2017) | VQA | ✓ | 0.8M | 66K
AVSD (Alamri et al., 2019) | VQA | ✓ | 152K | –
OpenViDial 1.0 (Meng et al., 2020) | Visual+Text Dialog | ✓ | 1.1M | 1.1M
OpenViDial 2.0 | Visual+Text Dialog | ✓ | 5.6M | 5.6M

Table 3: A comparison of different datasets. VQA: Visual Question Answering.

In Table 3, we make a comparison with existing widely-used dialog datasets.
Both OpenViDial 1.0 and OpenViDial 2.0 focus on multi-modal dialog generation
in comparison to VisDial, Guess-What?! and AVSD which focus more on VQA.
Comparing against OpenViDial 1.0, OpenViDial 2.0 is much larger in scale,
about 5 times as big as OpenViDial 1.0.

System | Model | BLEU | Dis-1 | Dis-2 | Dis-3 | Dis-4
---|---|---|---|---|---|---
NV | w/o MI | 1.95 | 0.0037 | 0.0302 | 0.0929 | 0.1711
w/ MI | 1.96 | 0.0039 | 0.0311 | 0.0953 | 0.1630
CV | w/o MI | 1.97 | 0.0041 | 0.0353 | 0.0999 | 0.1726
w/ MI | 1.98 | 0.0047 | 0.0392 | 0.1093 | 0.1774
FV | w/o MI | 1.99 | 0.0056 | 0.0431 | 0.1250 | 0.2215
w/ MI | 2.00 | 0.0060 | 0.0460 | 0.1321 | 0.2311
Table 4: Automatic evaluation results for BLEU, Stopword% and Diversity.

To evaluate OpenViDial 2.0, we experiment on OpenViDial2.0 using multi-modal
dialog models proposed in Wang et al. (2021).

###  3.1 Vanilla Visual Dialog Models

According to the granularity of the visual features ranges from none, coarse-
grained image features to fine-grained object features, Wang et al. (2021)
proposed three vanilla visual dialog models: (1) the NoVisual(NV) model, (2)
the CoarseVisual(CV) model and (3) the FineVisual(FV) model.

#### NoVisual

The NV model is a general uni-modal dialog generation model, which is required
to learn to generate responses using only dialog texts without visual
information. A standard Transformer Vaswani et al. (2017) architecture is used
as the backbone for the NV model. For each dialog turn, all the preceding
dialog texts are packed into a long sequence with a special token as the
delimiter. Then, this sequence is embedded with positional encodings including
sentence-level positional encoding and token-level positional encoding. Last,
it is fed to the Transfromer as input.

#### CoarseVisual

In contrast to the NV model, the CV model injects coarse-level visual
information into dialog generation. For each dialog turn, it utilizes a
ResNet-50 model He et al. (2016) pre-trained on ImageNet Krizhevsky et al.
(2012) to extract a high-dimensional feature for each image as the visual
information. Then the image feature is added to its corresponding text
representation forming the text-visual feature. Positional encodings are also
used to notify position information. The concatenated long text-visual
sequence is fed into the Transformer model.

#### FineVisual

Extracting visual information from a coarse view might be insufficient to
model fine-grained visual elements in images such as facial expressions, body
gestures as well as physical motions. The FV model thus uses Faster R-CNN Ren
et al. (2015) pre-trained on Visual Genome Krishna et al. (2017) to extract
fine-grained visual features. Different from the CV model, the FV model
directly concatenates the set of extracted fine-grained visual information
with the dialog texts into a long sequence. And except for the sentence-level
and token-level positional embeddings, there is an additional positional
embedding for visual features.

###  3.2 Visual-Text Mutual Dependeny

Although each response is generated according to the preceding textual and
visual contexts, there is no guarantee on whether or how much the visual
contexts are used. To significantly strength the connection between the
generated response and its visual contexts, Wang et al. (2021) proposed to
model the mutual information (MI) between visual contexts and text features.
To put it simply, we use visual feature to represent both the coarse-grained
feature and the fine-grained feature. For building the connection between
visual contexts and textual utterances, a light discriminative network is
trained. The whole requirement for the discriminative network is to
discriminate the degree of the connection between the given visual feature and
textual feature. In each inference step, both the CV and FV model are required
to generate N-best responses list with its probability as the forward
probability rather than only the best response. And each response in N-best
list along with the preceding visual feature are fed into the former trained
discriminative network obtaining the backward probability. Finally, the
forward probability and the backward probability is concatenated to rerank the
N-best list. For more details please refer to Wang et al. (2021).

###  3.3 Results

Following Wang et al. (2021), we report the results in terms of the following
automatic evaluation metrics:

  * •

BLEU: BLEU score is a common automatic evaluation method for majority NLP
tasks (Papineni et al., 2002; Sordoni et al., 2015), which score the n-gram
overlaps between the generated sequences and reference sequences. For our
experiment we report the BLEU-4 score.

  * •

Diversity: Diversity is usually reported in the task of dialogue generation Li
et al. (2015), which score the number of distinct n-grams in generated
responses, and $n=1,2,3,4$ for this experiment.

Results are shown in Table 4. Since OpenViDial 2.0 is much larger than
OpenViDial 1.0, we only use the top 5 objects for FineVisual model compared to
using top 20 objects on OpenViDial 1.0, and this is the main reason why FV
doesn’t significantly perform better than FV and NV.

##  4 Conclusion

In this report, we release OpenViDial 2.0, a larger-scale open-domain multi-
modal dialogue dataset with visual contexts, updated from the previous version
1.0. OpenViDial 2.0 contains a total number of 5.6 million dialogue turns
extracted from either movies or TV series from different resources, and is
four times larger than version 1.0 at scale. We hope this large-scale dataset
can help facilitate future researches on open-domain multi-modal dialog
generation. OpenViDial 2.0 is available at
https://github.com/ShannonAI/OpenViDial.

## References

  * Adiwardana et al. (2020) Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020\.  Towards a human-like open-domain chatbot.  _arXiv preprint arXiv:2001.09977_.
  * Alamri et al. (2019) Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K Marks, Chiori Hori, Peter Anderson, et al. 2019.  Audio visual scene-aware dialog.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7558–7567.
  * Asghar et al. (2018) Nabiha Asghar, Pascal Poupart, Jesse Hoey, Xin Jiang, and Lili Mou. 2018.  Affective neural response generation.  In _European Conference on Information Retrieval_ , pages 154–166. Springer.
  * Baheti et al. (2018) Ashutosh Baheti, Alan Ritter, Jiwei Li, and Bill Dolan. 2018.  Generating more interesting responses in neural conversation models with distributional constraints.  _arXiv preprint arXiv:1809.01215_.
  * Bosselut et al. (2018) Antoine Bosselut, Asli Celikyilmaz, Xiaodong He, Jianfeng Gao, Po-Sen Huang, and Yejin Choi. 2018.  Discourse-aware neural rewards for coherent text generation.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 173–184, New Orleans, Louisiana. Association for Computational Linguistics.
  * Bradski (2000) G. Bradski. 2000.  The OpenCV Library.  _Dr. Dobb’s Journal of Software Tools_.
  * COLBY (1975) KENNETH MARK COLBY. 1975.  Chapter 4 - language-recognition processes for understanding dialogues in teletyped psychiatric interviews.  In KENNETH MARK COLBY, editor, _Artificial Paranoia_ , pages 37 – 49. Pergamon.
  * Danescu-Niculescu-Mizil and Lee (2011) Cristian Danescu-Niculescu-Mizil and Lillian Lee. 2011.  Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs.  In _Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, ACL 2011_.
  * Das et al. (2017a) Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M. F. Moura, Devi Parikh, and Dhruv Batra. 2017a.  Visual dialog.
  * Das et al. (2017b) Abhishek Das, Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. 2017b.  Learning cooperative visual dialog agents with deep reinforcement learning.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2951–2960.
  * Dodge et al. (2016) Jesse Dodge, Andreea Gane, Xiang Zhang, Antoine Bordes, Sumit Chopra, Alexander Miller, Arthur Szlam, and Jason Weston. 2016.  Evaluating prerequisite qualities for learning end-to-end dialog systems.
  * Gao et al. (2018) Jianfeng Gao, Michel Galley, and Lihong Li. 2018.  Neural approaches to conversational ai.  In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_, pages 1371–1374.
  * Ghandeharioun et al. (2019) Asma Ghandeharioun, Judy Hanwen Shen, Natasha Jaques, Craig Ferguson, Noah Jones, Agata Lapedriza, and Rosalind Picard. 2019.  Approximating interactive human evaluation with self-play for open-domain dialog systems.  In _Advances in Neural Information Processing Systems_ , pages 13658–13669.
  * Ghazvininejad et al. (2017) Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2017.  A knowledge-grounded neural conversation model.  _arXiv preprint arXiv:1702.01932_.
  * Guo et al. (2019) Dan Guo, Hui Wang, and Meng Wang. 2019.  Dual visual attention network for visual dialog.  In _IJCAI_ , pages 4989–4995.
  * Han et al. (2020a) Qinghong Han, Yuxian Meng, Fei Wu, and Jiwei Li. 2020a.  Non-autoregressive neural dialogue generation.  _arXiv preprint arXiv:2002.04250_.
  * Han et al. (2020b) Xiaochuang Han, Byron C Wallace, and Yulia Tsvetkov. 2020b.  Explaining black box predictions and unveiling data artifacts through influence functions.  _arXiv preprint arXiv:2005.06676_.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778.
  * He et al. (2020) Wanwei He, Min Yang, Rui Yan, Chengming Li, Ying Shen, and Ruifeng Xu. 2020.  Amalgamating knowledge from two teachers for task-oriented dialogue system with adversarial training.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3498–3507, Online. Association for Computational Linguistics.
  * Hori et al. (2018) Chiori Hori, Huda Alamri, Jue Wang, Gordon Wichern, Takaaki Hori, Anoop Cherian, Tim K. Marks, Vincent Cartillier, Raphael Gontijo Lopes, Abhishek Das, Irfan Essa, Dhruv Batra, and Devi Parikh. 2018.  End-to-end audio visual scene-aware dialog using multimodal attention-based video features.
  * Hua et al. (2020) Kai Hua, Zhiyuan Feng, Chongyang Tao, Rui Yan, and Lu Zhang. 2020.  Learning to detect relevant contexts and knowledge for response selection in retrieval-based dialogue systems.  In _Proceedings of the 29th ACM International Conference on Information Knowledge Management_ , CIKM ’20, page 525–534, New York, NY, USA. Association for Computing Machinery.
  * Huang et al. (2020) Minlie Huang, Xiaoyan Zhu, and Jianfeng Gao. 2020.  Challenges in building intelligent open-domain dialog systems.  _ACM Transactions on Information Systems (TOIS)_ , 38(3):1–32.
  * Jiang et al. (2020a) Xiaoze Jiang, Siyi Du, Zengchang Qin, Yajing Sun, and Jing Yu. 2020a.  Kbgn: Knowledge-bridge graph network for adaptive vision-text reasoning in visual dialogue.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , pages 1265–1273.
  * Jiang et al. (2020b) Xiaoze Jiang, Jing Yu, Yajing Sun, Zengchang Qin, Zihao Zhu, Yue Hu, and Qi Wu. 2020b.  Dam: Deliberation, abandon and memory networks for generating detailed and non-repetitive responses in visual dialogue.
  * Jiang and Bansal (2019) Yichen Jiang and Mohit Bansal. 2019.  Self-assembling modular networks for interpretable multi-hop reasoning.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4474–4484, Hong Kong, China. Association for Computational Linguistics.
  * Kang et al. (2019) Gi-Cheon Kang, Jaeseo Lim, and Byoung-Tak Zhang. 2019.  Dual attention networks for visual reference resolution in visual dialog.  _arXiv preprint arXiv:1902.09368_.
  * Kottur et al. (2018) Satwik Kottur, José MF Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach. 2018.  Visual coreference resolution in visual dialog using neural module networks.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 153–169.
  * Kottur et al. (2019) Satwik Kottur, José M. F. Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach. 2019.  Clevr-dialog: A diagnostic dataset for multi-round reasoning in visual dialog.
  * Krishna et al. (2017) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017\.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _International journal of computer vision_ , 123(1):32–73.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012.  Imagenet classification with deep convolutional neural networks.  In _Advances in neural information processing systems_ , pages 1097–1105.
  * Lee et al. (2019) Sungjin Lee, Qi Zhu, Ryuichi Takanobu, Xiang Li, Yaoqin Zhang, Zheng Zhang, Jinchao Li, Baolin Peng, Xiujun Li, Minlie Huang, et al. 2019.  Convlab: Multi-domain end-to-end dialog system platform.  _arXiv preprint arXiv:1904.08637_.
  * Lewis et al. (2017) Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017.  Deal or no deal? end-to-end learning of negotiation dialogues.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2443–2453, Copenhagen, Denmark. Association for Computational Linguistics.
  * Li (2020) Jiwei Li. 2020.  Teaching machines to converse.  _arXiv preprint arXiv:2001.11701_.
  * Li et al. (2015) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015.  A diversity-promoting objective function for neural conversation models.  _arXiv preprint arXiv:1510.03055_.
  * Li et al. (2016a) Jiwei Li, Michel Galley, Chris Brockett, Georgios P Spithourakis, Jianfeng Gao, and Bill Dolan. 2016a.  A persona-based neural conversation model.  _arXiv preprint arXiv:1603.06155_.
  * Li et al. (2016b) Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. 2016b.  Deep reinforcement learning for dialogue generation.  _arXiv preprint arXiv:1606.01541_.
  * Li et al. (2017) Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. 2017.  Adversarial learning for neural dialogue generation.  _arXiv preprint arXiv:1701.06547_.
  * Lison and Tiedemann (2016) P. Lison and J. Tiedemann. 2016.  Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles.  In _LREC_.
  * Lowe et al. (2015) Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015.  The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems.  _arXiv preprint arXiv:1506.08909_.
  * Lu et al. (2017) Jiasen Lu, Anitha Kannan, Jianwei Yang, Devi Parikh, and Dhruv Batra. 2017.  Best of both worlds: Transferring knowledge from discriminative learning to a generative visual dialog model.
  * Massiceti et al. (2018) Daniela Massiceti, N Siddharth, Puneet K Dokania, and Philip HS Torr. 2018.  Flipdial: A generative model for two-way visual dialogue.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6097–6105.
  * Meng et al. (2020) Yuxian Meng, Shuhe Wang, Qinghong Han, Xiaofei Sun, Fei Wu, Rui Yan, and Jiwei Li. 2020.  Openvidial: A large-scale, open-domain dialogue dataset with visual contexts.  _arXiv preprint arXiv:2012.15015_.
  * Niu et al. (2019) Yulei Niu, Hanwang Zhang, Manli Zhang, Jianhong Zhang, Zhiwu Lu, and Ji-Rong Wen. 2019.  Recursive visual attention in visual dialog.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6679–6688.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Park et al. (2020) Sungjin Park, Taesun Whang, Yeochan Yoon, and Hueiseok Lim. 2020.  Multi-view attention networks for visual dialog.  _arXiv preprint arXiv:2004.14025_.
  * Rashkin et al. (2018) Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2018.  Towards empathetic open-domain conversation models: A new benchmark and dataset.  _arXiv preprint arXiv:1811.00207_.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In _Advances in neural information processing systems_ , pages 91–99.
  * Roller et al. (2020) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M Smith, et al. 2020.  Recipes for building an open-domain chatbot.  _arXiv preprint arXiv:2004.13637_.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Seo et al. (2017) Paul Hongsuck Seo, Andreas Lehrmann, Bohyung Han, and Leonid Sigal. 2017.  Visual reference resolution using attention memory for visual dialog.  In _Advances in neural information processing systems_ , pages 3719–3729.
  * Serban et al. (2016) Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2016.  A hierarchical latent variable encoder-decoder model for generating dialogues.  _arXiv preprint arXiv:1605.06069_.
  * Shao et al. (2017) Louis Shao, Stephan Gouws, Denny Britz, Anna Goldie, Brian Strope, and Ray Kurzweil. 2017.  Generating high-quality and informative conversation responses with sequence-to-sequence models.  _arXiv preprint arXiv:1701.03185_.
  * Sordoni et al. (2015) Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015.  A neural network approach to context-sensitive generation of conversational responses.  _arXiv preprint arXiv:1506.06714_.
  * Tian et al. (2017) Zhiliang Tian, Rui Yan, Lili Mou, Yiping Song, Yansong Feng, and Dongyan Zhao. 2017\.  How to make context more useful? an empirical study on context-aware neural conversational models.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 231–236, Vancouver, Canada. Association for Computational Linguistics.
  * Tiedemann (2009) J. Tiedemann. 2009.  News from opus — a collection of multilingual parallel corpora with tools and interfaces.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Vinyals and Le (2015) Oriol Vinyals and Quoc Le. 2015.  A neural conversational model.  _arXiv preprint arXiv:1506.05869_.
  * de Vries et al. (2017) Harm de Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. 2017.  Guesswhat?! visual object discovery through multi-modal dialogue.
  * Wallace (2009) Richard S. Wallace. 2009.  _The Anatomy of A.L.I.C.E._ , pages 181–210. Springer Netherlands, Dordrecht.
  * Wang et al. (2021) Shuhe Wang, Yuxian Meng, Xiaofei Sun, Fei Wu, Rongbin Ouyang, Rui Yan, Tianwei Zhang, and Jiwei Li. 2021.  Modeling text-visual mutual dependency for multi-modal dialog generation.  _arXiv preprint arXiv:2105.14445_.
  * Weizenbaum (1966) Joseph Weizenbaum. 1966.  Eliza—a computer program for the study of natural language communication between man and machine.  _Communications of the ACM_ , 9(1):36–45.
  * Wu et al. (2018) Qi Wu, Peng Wang, Chunhua Shen, Ian Reid, and Anton Van Den Hengel. 2018.  Are you talking to me? reasoned visual dialog generation through adversarial learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6106–6115.
  * Xie et al. (2017) Ziang Xie, Sida I Wang, Jiwei Li, Daniel Lévy, Aiming Nie, Dan Jurafsky, and Andrew Y Ng. 2017.  Data noising as smoothing in neural network language models.  _arXiv preprint arXiv:1703.02573_.
  * Xu et al. (2018) Jingjing Xu, Xuancheng Ren, Junyang Lin, and Xu Sun. 2018.  Dp-gan: diversity-promoting generative adversarial network for generating informative and diversified text.  _arXiv preprint arXiv:1802.01345_.
  * Yang et al. (2019) Tianhao Yang, Zheng-Jun Zha, and Hanwang Zhang. 2019.  Making history matter: History-advantage sequence training for visual dialog.
  * Young et al. (2017) Tom Young, Erik Cambria, Iti Chaturvedi, Minlie Huang, Hao Zhou, and Subham Biswas. 2017.  Augmenting end-to-end dialog systems with commonsense knowledge.  _arXiv preprint arXiv:1709.05453_.
  * Zhang et al. (2018) Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018.  Personalizing dialogue agents: I have a dog, do you have pets too?  _arXiv preprint arXiv:1801.07243_.
  * Zhang et al. (2019) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019.  Dialogpt: Large-scale generative pre-training for conversational response generation.  _arXiv preprint arXiv:1911.00536_.
  * Zhao et al. (2019) Tiancheng Zhao, Kaige Xie, and Maxine Eskenazi. 2019.  Rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent variable models.  _arXiv preprint arXiv:1902.08858_.
  * Zhao et al. (2017) Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017.  Learning discourse-level diversity for neural dialog models using conditional variational autoencoders.  _arXiv preprint arXiv:1703.10960_.
  * Zhao et al. (2020) Xueliang Zhao, Wei Wu, Can Xu, Chongyang Tao, Dongyan Zhao, and Rui Yan. 2020.  Knowledge-grounded dialogue generation with pre-trained language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3377–3390, Online. Association for Computational Linguistics.
  * Zheng et al. (2019) Zilong Zheng, Wenguan Wang, Siyuan Qi, and Song-Chun Zhu. 2019.  Reasoning visual dialogs with structural and partial observations.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6669–6678.
  * Zhou et al. (2017) Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. 2017.  Emotional chatting machine: Emotional conversation generation with internal and external memory.  _arXiv preprint arXiv:1704.01074_.
  * Zhou et al. (2020) Li Zhou, Jianfeng Gao, Di Li, and Heung-Yeung Shum. 2020.  The design and implementation of xiaoice, an empathetic social chatbot.  _Computational Linguistics_ , 46(1):53–93.


# PlaceNav: Topological Navigation through Place Recognition

Lauri Suomela1, Jussi Kalliola, Harry Edelman and Joni-Kristian Kämäräinen
<EMAIL_ADDRESS>†lasuomela.github.io/placenav/ All authors are with
Tampere University, Finland. This work was supported by Technology Innovation
Insitute.

###### Abstract

Recent results suggest that splitting topological navigation into robot-
independent and robot-specific components improves navigation performance by
enabling the robot-independent part to be trained with data collected by
different robot types. However, the navigation methods are still limited by
the scarcity of suitable training data and suffer from poor computational
scaling. In this work, we present PlaceNav, subdividing the robot-independent
part into navigation-specific and generic computer vision components. We
utilize visual place recognition for the subgoal selection of the topological
navigation pipeline. This makes subgoal selection more efficient and enables
leveraging large-scale datasets from non-robotics sources, increasing training
data availability. Bayes filtering, enabled by place recognition, further
improves navigation performance by increasing the temporal consistency of
subgoals. Our experimental results verify the design and the new model obtains
a $76\text{\,}\mathrm{\char 37\relax}$ higher success rate in indoor and
$23\text{\,}\mathrm{\char 37\relax}$ higher in outdoor navigation tasks with
higher computational efficiency.

##  I Introduction

Autonomous visual navigation is a well-studied problem in the field of
robotics [1, 2]. One line of research frames navigation in known environments
as topological navigation [3, 4, 5], meaning purely vision-based navigation
between nodes of a topological map that are represented by images. The
advantage of this approach is that it does not require building a geometric
reconstruction of the operating environment [6] or training environment-
specific control policies [7].

Topological navigation systems have two parts: _subgoal selection_ and a
_goal-reaching policy_. First, the subgoal selection module chooses the map
node to reach next as a subgoal. Then, the goal-reaching policy produces
control commands to take the robot to the selected subgoal. A popular approach
to subgoal selection is _temporal distance prediction_ , which means
predicting the number of time steps between the robot’s current camera
observation and the subgoal candidates [8, 9, 10, 11, 12, 13]. These learning-
based models are trained with offline datasets of robot trajectories.

Previous works have demonstrated impressive real-world navigation [10, 12,
13], but the temporal distance prediction approach has two significant
shortcomings. First, it utilizes neural architectures that iteratively take
the current observation and each subgoal candidate as input, resulting in
computational complexity scaling at $\mathcal{O}(n)$ with the number of
candidate images. This requires heuristics to limit the candidates and
constrains the methods available for ensuring subgoal temporal consistency.
Second, the fact that temporal distance prediction requires training data that
originates from robots, actual or simulated, introduces an unnecessary data
bottleneck. High-quality robotics datasets are very scarce compared to general
web-scale data, and models trained with simulated data suffer from
generalization issues [14].

Figure 1: Visual place recognition finds which map image $I_{s}$ was captured
closest to the robot’s observation $I_{t}$ by efficient matching of image
embeddings $\bm{z}$.

We claim that subgoal selection is not a unique problem but an instance of the
broader concept of image retrieval. To address this, we present _PlaceNav_ ,
which frames the selection as a _place recognition_ [15] task. This design
provides three advantages. First, the large-scale and high-diversity datasets
available for training place recognition models enhance subgoal selection
robustness against changes in viewpoint and appearance. Second, subgoal
selection is performed by a fast nearest-neighbor search over image
embeddings. This, as shown in Fig. 1, provides superior scalability and
removes the need for heuristics. Finally, place recognition readily integrates
with methods for ensuring temporal consistency.

In summary, our contributions are

  * •

Navigation approach that decouples training of subgoal selection models from
robotics datasets by treating the selection as a generic place recognition
task.

  * •

Integration of learning-based subgoal selection with a Bayesian filter that
improves temporal consistency.

  * •

Demonstrations with real robots to experimentally validate our design.

Code and videos are available on the project page†.

##  II Related work

Vision-based topological navigation. In a recent trend, topological maps have
been used to divide long-horizon tasks into short-horizon segments suitable
for learned goal-reaching policies [8, 16]. An essential part of such a
hierarchical structure is choosing which subgoal to reach next. One approach
is to learn to predict the reachability between two images from simulated
rollouts [17]. Savinov  _et al_. [8] proposed to use the _temporal distance_ ,
or the number of time steps $\Delta t$, between the current observation and a
subgoal as a proxy for reachability. Its key strength is that it can be
learned from offline data. While alternative approaches exist [18], temporal
distance is popular and has been adopted in several recent works [9, 10, 11,
12, 13]. However, the diversity and size of datasets suitable for temporal
distance learning are modest. RECON [19] with $25\text{\,}\mathrm{h}$, SACSoN
[20] with $75\text{\,}\mathrm{h}$, and TartanDrive [21] with
$5\text{\,}\mathrm{h}$ of navigation trajectories from single location each
are notable examples. Furthermore, because of the model architectures
utilized, the computational complexity of temporal distance prediction scales
at O(n) with the number of subgoal candidates considered.

Place recognition. Place recognition involves recognizing places from images,
often framed as image retrieval across images captured from different
viewpoints and varying appearances [15]. It naturally integrates with
topological navigation, as subgoal selection can be viewed as an image
retrieval problem. Traditional methods for place recognition rely on
aggregating handcrafted local features [22, 23, 24], but newer methods utilize
deep learning to extract embeddings that can be compared efficiently using
nearest-neighbor search [25, 26, 27]. The methods are trained to produce
embeddings that are similar for images from the same place and dissimilar for
images from different places, typically by classification loss [27] or ranking
losses such as contrastive [28], triplet [25] or listwise [26] loss.

Temporal consistency. While subgoal temporal consistency has been studied in
non-learned topological navigation literature [4], it has received limited
attention in the context of learning-based methods. A robot moves along a
route continually, so the transitions between the subgoals should be smooth.
As a heuristic solution, SPTM [8] and GNM [12] adopted the approach of only
considering subgoals within a sliding window centered on the previous subgoal.
Meng  _et al_. [17] utilize a similar approach but resort to global search
when the window deviates from the robot’s actual location.

In place recognition literature, the topic has received more attention. Early
approaches [29, 30, 31, 32] utilized feature similarity matrices to find the
best-matching image sequences. A newer line of work [33, 34, 35] considers
descriptors that represent sequences instead of individual images. As an
alternative, Xu  _et al_. [36, 37] added a Bayesian filter to the matching
process. In this work, we show that learning-based topological navigation also
benefits from such methods.

##  III System overview

In this section, we describe  _PlaceNav_ , our proposed navigation pipeline.
First, we discuss the basic components and definitions of topological
navigation. Then, we elaborate on our contributions related to subgoal
selection via place recognition and subgoal temporal consistency.

###  III-A Topological navigation fundamentals

Autonomous navigation using topological maps generally consists of two stages.
Before navigation, an operator has to perform a manual ’reference run’ to
capture the desired route. The robot saves images along the route that compose
the topological map $\mathcal{M}$ for navigation.

During navigation, the robot-agnostic topological navigation algorithm is
combined with a robot-specific controller. At each inference step $t$, the
current robot observation $I_{t}$ is compared to the different subgoal
candidate images $I_{s}\in\mathcal{M}$ at nodes $~{}s=[0,1,\ldots,S]$ of the
topological map. One of the nodes is selected as the next subgoal, and an
image-based goal-reaching policy produces the motion plan to reach it. In this
work, we experiment with the different subgoal selection methods and adopt the
waypoint estimation approach proposed by Shah  _et al_. [12] as the goal-
reaching policy. This approach defines the motion plan as a sequence of $\tau$
waypoints $\\{p_{i},\psi_{i}\\}_{i},~{}i=[0,1,\ldots,\tau]$ that guide the
robot to the subgoal. The waypoints, defined as metric coordinates $p_{i}$ and
heading angle $\psi_{i}$ in the robot’s local coordinate frame, are tracked by
a robot-specific controller.

###  III-B Subgoal selection via place recognition

PlaceNav introduces the following modifications to the subgoal selection
procedure. Instead of computing temporal distances $\Delta t$ between each
observation and subgoal candidate pair, we use a place recognition model
$f_{enc}$ to process the observation and map images separately. The model
produces image embeddings $\mathbf{z}_{t}$ and $\mathbf{z}_{s}$ that can be
compared by Euclidean distance, enabling efficient subgoal search. Figure. 1
visualizes the concept.

Training data availability. The temporal distance prediction models are
trained to predict the $\Delta t$ between two image frames sampled from a
trajectory driven by a robot. This limits the amount and diversity of
potential training data. Place recognition methods can be trained with data
from more generic sources. Images of different places, preferably captured at
various points in time, from different viewpoints, and under different
environmental conditions. The images’ rough position and orientation
information provide annotations. Google StreetView images, for example, are
well-suited as training data. The sizes of place recognition datasets are in
the order of millions of images [38, 39, 40], the SF-XL [27] alone consisting
of 41M images.

Computational complexity. The computational complexity of temporal distance
prediction scales linearly with the number of subgoal candidates considered at
each inference step. Because of this, the number of subgoal candidates must be
limited, which is commonly implemented as a _sliding window_ over the map
nodes. The window is centered on the subgoal from the previous step, and only
the nodes within the window are considered potential subgoals.

Place recognition enables computation and storage of the descriptors for the
map images offline before robot operation. Thus, the inference computational
complexity of subgoal selection is decoupled from the number of subgoal
candidates being considered. The descriptors can be matched in milliseconds by
nearest neighbor search [41]. Consequently, heuristics to limit the number of
subgoal candidates are not needed from the perspective of computational
budget.

###  III-C Temporal consistency

Limiting the number of subgoal candidates also enhances temporal coherence
between inference steps, preventing erratic subgoal selection  _e.g_. due to
visually similar content. A sliding window over the map achieves this to some
extent. However, the window may drift away from the robot’s location, making
correct subgoal selection impossible. Bayesian filtering, enabled by efficient
matching of image embeddings, is an alternative strategy for enforcing
temporal consistency.

Figure 2: The discrete Bayesian filter alternates place recognition
measurement and motion model prediction. $p(\bm{s})$, the posterior belief,
determines the best matching node.

Xu  _et al_. [36] propose one such approach for use with place recognition
methods, which we adapt for our problem. The idea, illustrated in Fig. 2, is
to formulate place recognition as the measurement step of a discrete Bayesian
state estimator. This filter maintains a belief of the robot’s location over
the map nodes by recursively updating its posterior distribution. We present
the key equations here for completeness but refer the reader to the original
paper for details.

Given an initial posterior belief distribution, a motion model propagates the
belief into the future in a prediction step. If the robot’s local movement
(_i.e_. odometry) is not being tracked, as is the case with our system, the
motion model is simply

$p(s_{i}|s_{j})\propto\begin{cases}1&w_{l}\leq i-j\leq w_{u}\\\
0&\text{otherwise}\end{cases}\enspace.$ (1)

From each node, the robot has an equal probability of moving up to $w_{u}$
steps toward the goal, staying put, or moving $w_{l}$ steps back. Other
transitions have zero probability.

The prediction step is followed by a measurement step, where a place
recognition query between the observation and the map nodes produces a
measurement belief by the measurement function

$p(\mathbf{z}_{t}|s,\mathcal{M})\propto
g(\mathbf{z}_{t},s,\mathcal{M})=\exp(-\lambda_{1}\|\mathbf{z}_{t}-\mathbf{z}_{s}\|_{2})\enspace,$
(2)

where $\mathbf{z}_{t}$ is the observation embedding, $\mathbf{z}_{s}$ is a map
node embedding at the state $s$ being considered, and $\mathcal{M}$ is the
map. $\lambda_{1}$ scales the effect of each measurement on the posterior. Its
value is automatically determined at the beginning of each navigation session
as proposed by the authors [36].

The measurement belief is multiplied by the belief from the prediction step to
acquire the posterior belief distribution $p(s)$. The map node with the
highest posterior probability is considered the closest to the latest
observation. This filter significantly improves the stability of subgoal
selection. Unlike the sliding window approach, the filter maintains full
posterior belief over all map nodes, so it cannot get lost. It can solve the
’kidnapped robot’ problem [42], whereas the sliding window requires the start
node of the robot to be specified manually.

###  III-D Implementation details

Architecture & Training. For the place recognition part of the PlaceNav, we
use a CosPlace network [27] because of its high performance and simple
training process. The model architecture comprises a convolutional encoder and
a generalized mean pooling (GeM) layer. The model is trained via
classification loss, enabled by dividing the training data into distinct
spatial groups. As the original CosPlace model was trained with high-
resolution images ($512\times 512$), the training checkpoints provided by the
authors do not work well with the $85\times 64$ images we need to use for
comparison with the baseline GNM model. For our experiments, we train a
CosPlace model from scratch using an EfficientNet-B0 [43] backbone and the
41.2M images of the San Francisco eXtra Large (SF-XL) dataset [27], resized to
$85\times 85$ during training. The model was configured to extract
512-dimensional descriptors. Otherwise, we followed the training procedure
outlined in [27]. We will refer to this low-resolution model as _CosPlace-LR_.

The Shah  _et al_. [12] waypoint estimation was chosen as the goal-reaching
policy. We do not retrain the model and use the pre-trained weights provided
by the authors.

Deployment. During inference, the robot uses place recognition to identify the
map node that best matches the current observation. The next node along the
route after the best-matching node is selected as the subgoal. We implemented
two distinct temporal consistency methods in the subgoal selection. The first
is the sliding window used in prior works [8, 17, 12], and the second is our
implementation of the discrete Bayesian filter proposed by Xu  _et al_. [36].
At the beginning of a route, the sliding window is initialized to the first
node. The initial belief distribution of the discrete filter is acquired from
the first place recognition query, meaning that it operates in a ’kidnapped
robot’ mode. We set the discrete filter motion model transition boundaries to
$w_{u}=2$ and $w_{l}=-1$ based on calibration experiments.

After choosing the subgoal to reach next, the goal-reaching policy predicts a
series of 5 waypoints $\\{p_{i},\psi_{i}\\}$. The robot follows these
waypoints in an open-loop fashion until the next prediction using the robot’s
low-level controller. The prediction loop of subgoal selection and waypoint
prediction runs at $5\text{\,}\mathrm{Hz}$. The place recognition and waypoint
estimation models receive $85\times 64$ resolution images as input.

##  IV Experimental setup

Figure 3: The robots: A Turtlebot2 (left) and a Robotnik Summit XL Steel
(right)

We performed navigation experiments with real robots in diverse environments
to enable an informed comparison of PlaceNav and the prior work. We conducted
360 repetitions of different routes with different robots, subgoal selection
methods, and temporal consistency approaches, adding up to a total of
$19\text{\,}\mathrm{km}$ of navigation testing. We also evaluated the subgoal
selection methods offline using a place recognition benchmark. With our
experiments, we aim to answer the following research questions:

  * •

Q1. Does subgoal selection require models trained with data originating from
robots, or could more efficient place recognition models replace them?

  * •

Q2. How robust are temporal distance prediction models to viewpoint and
appearance changes?

  * •

Q3. How does subgoal temporal consistency affect navigation performance?

###  IV-A Robot navigation experiments

The robots. We experimented with two different robots, a Turtlebot2, and a
Robotnik Summit XL Steel, shown in Fig. 3. Turtlebot is a small research
platform intended for indoor use. Summit is a 4-wheeled skid-steer ground
vehicle with a weight of $90\text{\,}\mathrm{kg}$, load capacity of
$130\text{\,}\mathrm{kg}$, and a maximum velocity of
$3\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$. Both robots carry a front-facing
$175\text{\,}\mathrm{\SIUnitSymbolDegree}$ field-of-view fish-eye camera,
which is used for navigation, and a laptop computer that runs the navigation
algorithms. The laptop has a Nvidia Geforce GTX 1070 GPU and an Intel i7-7700
CPU. The deep learning components of the navigation stack run on the GPU.

Baseline. We use the temporal distance prediction from GNM [12] by Shah  _et
al_. as the baseline. We utilize the GNM for waypoint prediction in all
experiments and only experiment with the subgoal selection. The model takes
the observation and subgoal images concatenated along the channel dimension as
inputs. It was trained with $85\times 64$ images from 54 hours of navigation
trajectories. The inference loop runs at $5\text{\,}\mathrm{Hz}$. At each
step, the node with the smallest temporal distance $\Delta t$ above 3 is
selected as the subgoal.

As the discrete filter requires the distance between the current observation
and all the map nodes at each step, using it with the GNM is not
computationally feasible. Thus, with GNM, we only utilize the sliding window.

Figure 4: Examples along the test routes. The top row is from outdoor tests,
bottom row is from indoors.

Indoor navigation. The indoor experiments were conducted with the Turtlebot
robot. We tested the methods along 20 different routes, performing 3
repetitions of each route with each method. Fig. 4 shows examples along the
routes. Routes where all the methods fail were excluded from the quantitative
analysis. The experiments took place at a university campus, with buildings
from various eras with diverse appearances.

The lengths of the test routes were uniformly distributed in the range from
$15\text{\,}\mathrm{m}$ to $35\text{\,}\mathrm{m}$, and they contained various
amounts of difficult content, such as turning maneuvers and passing narrow
gaps. We chose routes that do not contain content that causes navigation
failures because of errors in waypoint estimation. Such content,  _e.g_. very
narrow gaps, and turning maneuvers in open areas with few salient features can
cause the robot to veer off course even though the subgoal selection algorithm
is not failing.

Outdoor navigation. The outdoor experiments were conducted with the Summit XL
robot. We experimented on 20 test routes in an urban environment, ranging from
$50\text{\,}\mathrm{m}$ to $150\text{\,}\mathrm{m}$ in length. Like the indoor
experiments, each test route was repeated 3 times with each method.

In indoor calibration tests before the actual experiments, correct subgoal
selection led to accurate waypoint estimation. In outdoor tests, we observed
more variability. This is likely due to increased environmental noise and
larger appearance changes between the map images and robot operation. _e.g_.
changes in ambient illumination led to complete waypoint estimation failure
despite a good subgoal choice. Therefore, in the actual experiments, we
maintained test run illuminance within a $10\,000\text{\,}\mathrm{lx}$ range
of the reference run. A Konica Minolta CL-70F CRI illuminance meter was used
for this purpose.

Evaluation criteria. We follow the evaluation guidelines for image-goal
navigation in [44] and measure the navigation performance by the _success
rate_ (SR). Success rate describes the ratio of successful and failed
repetitions of a test route.

Repetition is considered successful when the robot reaches a pose where its
camera view corresponds to the final node of the topological map. The subgoal
selection must also localize to this node, triggering a navigation stop
signal. We do not use distance-based success criteria, which are more common
but less aligned with the robot’s goal determination. While last-mile
geometric navigation could enhance goal-reaching precision, as suggested by
Wasserman  _et al_. [45], it is unnecessary for the scope of this work.
Repetition is considered a failure when the robot takes actions that prevent
it from reaching the goal,  _i.e_. sending the ’reached goal’ signal without
the goal view in sight, risking immediate collisions, or deviating
significantly from the test route without recovery prospects.

###  IV-B Offline evaluation

To enable a reproducible comparison of temporal distance prediction and place
recognition, we tested GNM on standard place recognition benchmarks. The VPR-
Benchmark by Bertoli  _et al_. [46] was used to facilitate the experiments. We
modified the benchmark to enable the evaluation of GNM that simultaneously
takes the query and reference images as inputs. A subset of the VPR-Benchmark
datasets where the size of the test database is small enough that GNM can
process it in a reasonable time was picked for evaluation. We compared the
performance of our low-resolution CosPlace-LR model and the GNM model. The
test images were resized to $85\times 64$. For reference, we additionally
evaluate the standard CosPlace model with full-resolution images.

Place recognition performance is assessed using the Recall@N score, which
measures how often one of the top N images retrieved is within
$25\text{\,}\mathrm{m}$ of the query image location. In the case of temporal
distance prediction, the top N images are those with the smallest predicted
temporal distances.

##  V Results

###  V-A Robot navigation experiments

Tables I and II display the results of the navigation experiments with the
Turtlebot and Summit XL.

TABLE I: Indoors experiment success rates over 60 repetitions, driven with the Turtlebot. Method | Type | Temporal |  | Easy | Hard | Total
---|---|---|---|---|---|---
filter | $n=33$ | $27$ | 60
GNM [12] | _T_ | Window |  | 0.52 | 0.26 | 0.39
PlaceNav | _P_ | Window |  | 0.62 | 0.60 | 0.61
Bayesian |  | $\bm{0.65}$ | $\bm{0.77}$ | $\bm{0.69}$
TABLE II: Outdoors experiment success rates over 60 repetitions, driven with the Summit XL. Method | Type | Temporal |  | Easy | Hard | Total
---|---|---|---|---|---|---
filter | $n=24$ | 36 | 60
GNM [12] | _T_ | Window |  | $\bm{0.67}$ | 0.33 | 0.47
PlaceNav | P | Window |  | 0.46 | 0.44 | 0.45
Bayesian |  | $\bm{0.67}$ | $\bm{0.53}$ | $\bm{0.58}$
| _T: temporal distance, P: place recognition_

Indoors. The GNM baseline has a notably lower success rate than the proposed
place recognition based approaches. PlaceNav shows a 56% SR increase, which
rises to 77% with the Bayesian filter. This observation is interesting given
that GNM training includes indoor images from the GS4 [47] dataset, while
CosPlace models are trained on outdoor Google Streetview images. Place
recognition models exhibit broad generalization capabilities, learning
features that generalize across domains.

Similar to Shah  _et al_. [12], we split the test routes into ’Easy’ and
’Hard’ categories in posterior analysis. The categories are based on the
number of narrow turns and tight passages along the routes. We chose a
threshold that splits the routes into evenly sized groups. For indoor routes,
this threshold was set to 4. ’Easy’ routes had fewer than 4 such features,
while ’Hard’ routes had 4 or more. The categorization provides further insight
into the method performances. While the differences in SR are minimal for
’Easy’ routes, the advantage of place recognition is evident in the ’Hard’
category. GNM’s SR decreases by 50% from ’Easy’ to ’Hard,’ but PlaceNav
maintains the same SR, even improving when the Bayesian filter is employed.

Introducing the Bayesian filter yields clear advantages over the sliding
window method. Visual inspection confirms improved stability and performance
during turns, especially in narrow gaps. The filter mitigates errors such as
mid-turn direction changes that can arise because of the erratic behavior of
the sliding window approach.

TABLE III:  Offline evaluation of Recall@1 for the place recognition (_P_) and temporal distance (_T_) models. |  |  | Multiview | Front-facing
---|---|---|---|---
Method | Type | Resolution | Pitts30k | Tokyo | MSLS | St. Lucia | SVOX | SVOX | SVOX | SVOX | SVOX
24/7 | Val | Night | Overcast | Rainy | Snow | Sun
CosPlace [48] | _P_ | full* | 89.5 | 81.9 | 80.8 | 98.8 | 32.9 | 85.4 | 79.6 | 85.3 | 61.5
CosPlace-LR | _P_ | $85\times 64$ | 71.8 | 24.1 | 43.8 | 90.4 | 1.2 | 33.9 | 26.1 | 24.5 | 12.4
GNM [12] | _T_ | $85\times 64$ | 8.9 | 0.3 | 2.6 | 8.7 | 0.0 | 0.1 | 0.2 | 0.0 | 0.4
|  |  |  |  |  |  |  |  | _*: various resolutions_

Outdoors. Outdoors, the success rate gap between methods is narrower compared
to indoors, despite CosPlace-LR’s outdoor training data. One possible
explanation is the higher variation in waypoint estimation performance
observed in the calibration tests. Consequently, the waypoint estimation
module contributes more significantly to the final SR’s, diminishing the
effect of subgoal selection performance. The magnitude of the performance
increase brought by the Bayesian filter is consistent with the indoor
experiments, the improvement being around 10 percentage points in both cases.
The differences in SR’s between the ’Easy’ and ’Hard’ categories are similar
to the indoor experiments. GNM’s SR drops by half from ’Easy’ to ’Hard,’
whereas PlaceNav exhibits minimal to no performance decrease, emphasizing
place recognition’s effectiveness in maneuvers where having a correct subgoal
is crucial.

In the analysis of outdoor experiments, an even distribution of routes between
the ’Easy’ and ’Hard’ categories was achieved by a threshold of 3 turns or
narrow passages per route. The occurrence of visually bursty content [49],
characterized by repetitive geometric patterns that heavily influence image
embeddings, poses a challenge for place recognition methods [15, p.12] on
certain test routes (see Fig. 5). This issue, causing the relatively low SR of
PlaceNav with the sliding window in the ’Easy’ category can lead to the
sliding window method becoming trapped within the bursty map region, resulting
in navigation failure. In contrast, the Bayesian filter handles bursty content
more effectively by maintaining a full posterior belief across all map nodes
and avoiding such entrapment. If the robot traverses the bursty segment
successfully, the filter accurately localizes and completes the test route
without issues.

###  V-B Offline evaluation

Here, we present the results of evaluating GNM and CosPlace on the VPR-
Benchmark. We also discuss the impact of input resolution and domain shift on
recall rates.

Performance Comparison. Table III shows a comparison of the retrieval
performances of the GNM and CosPlace models across several benchmark datasets.
Notably, temporal distance prediction performs worse than place recognition
across all datasets. GNM’s recall follows a similar trend as CosPlace-LR, with
GNM achieving higher recall wherever CosPlace-LR excels. However, GNM’s recall
values are consistently an order of magnitude lower. For instance, on
Tokyo24/7 and St. Lucia, where CosPlace-LR attains over 70% recall, GNM only
reaches approximately 9%. On other datasets, GNM’s performance is
significantly lower.

Impact of Input Resolution. Decreasing image resolution has a substantial
impact on CosPlace’s performance. Reducing the resolution to $85\times 64$
pixels decreases recall rates up to 58 percentage points on datasets like
Tokyo24/7, MSLS, and SVOX. Interestingly, GNM’s temporal distance prediction
performs best on datasets where the performance differences between full and
low-resolution CosPlace models are minimal, namely Pitts30k and St. Lucia.
This suggests that GNM performance, too, would be improved by training the
model with higher-resolution images.

Viewpoint and Appearance Change. Pittsburgh30k and Tokyo24/7 datasets capture
images from various angles, while the rest feature images from front-facing
vehicle cameras. Despite the similarity in viewpoint variation between GNM’s
training data and front-facing datasets, this is not reflected in recall
rates. GNM performs well on Pittsburgh30k but poorly on SVOX. This discrepancy
may stem from other factors contributing to domain shift between query and
reference images. Besides viewpoint changes, Pittsburgh30k and St. Lucia
exhibit limited variation. The other datasets contain shifts in illumination,
weather, and camera which GNM struggles to handle, not having been explicitly
trained for such invariance.

Figure 5: Visually bursty content along the test routes. TABLE IV: Runtimes
for the different methods with sliding window and Bayesian filter. _*:
Bayesian filter considers all map nodes_

Method | Temporal | Window | Runtime
---|---|---|---
filter | size | ($\text{\,}\mathrm{ms}$)
GNM [12] | Window | 5 | 96
Window | 21 | 308
PlaceNav | Window | 5 | 50
Window | 21 | 50
Bayesian |  -∗ | 90

###  V-C Runtime analysis

Table IV shows average runtimes for PlaceNav and the baseline. Replacing
temporal distance prediction with place recognition significantly reduces
runtime, even without optimization. The runtime of PlaceNav is not affected by
the window size. Introducing the discrete Bayesian filter aligns the
PlaceNav’s runtime with the original temporal distance baseline, allowing
resource-performance trade-offs based on computational capacity and navigation
needs.

##  VI Conclusion

In conclusion, our findings show that place recognition enables more accurate
subgoal selection than the temporal distance prediction methods at a lower
computational cost. The offline evaluation suggests that the presence of
appearance change between the reference run and robot operation would further
amplify the difference. Our future work will focus on appearance invariant
subgoal selection models and goal-reaching policies.

## ACKNOWLEDGMENT

The authors thank Jani Käpylä, Olli Suominen, and Jussi Rantala from CIVIT for
access to the Summit XL. We would also like to thank María Andrea Cruz Blandón
and German F. Torres for their valuable comments on an earlier version of the
manuscript.

## References

  * [1] R. Brooks, “Visual map making for a mobile robot,” in _1985 IEEE International Conference on Robotics and Automation Proceedings_ , vol. 2, Mar. 1985, pp. 824–829.
  * [2] E. Baumgartner and S. Skaar, “An autonomous vision-based mobile robot,” _IEEE Transactions on Automatic Control_ , vol. 39, no. 3, pp. 493–502, Mar. 1994, conference Name: IEEE Transactions on Automatic Control.
  * [3] S. Thrun, “Learning metric-topological maps for indoor mobile robot navigation,” _Artificial Intelligence_ , vol. 99, no. 1, pp. 21–71, Feb. 1998. https://www.sciencedirect.com/science/article/pii/S0004370297000787
  * [4] T. Goedemé, M. Nuttin, T. Tuytelaars, and L. Van Gool, “Omnidirectional Vision Based Topological Navigation,” _International Journal of Computer Vision_ , vol. 74, no. 3, pp. 219–236, Sept. 2007. https://doi.org/10.1007/s11263-006-0025-9
  * [5] D. Filliat, “Interactive learning of visual topological navigation,” in _2008 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , Sept. 2008, pp. 248–254, iSSN: 2153-0866.
  * [6] A. Rosinol, M. Abate, Y. Chang, and L. Carlone, “Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , May 2020, pp. 1689–1696, iSSN: 2577-087X.
  * [7] P. Mirowski, R. Pascanu, F. Viola, H. Soyer, A. Ballard, A. Banino, M. Denil, R. Goroshin, L. Sifre, K. Kavukcuoglu, D. Kumaran, and R. Hadsell, “Learning to Navigate in Complex Environments,” in _International Conference on Learning Representations (ICLR))_ , Nov. 2016. https://openreview.net/forum?id=SJMGPrcle
  * [8] N. Savinov, A. Dosovitskiy, and V. Koltun, “Semi-parametric topological memory for navigation,” in _International Conference on Learning Representations (ICLR)_ , 2018. https://openreview.net/forum?id=SygwwGbRW
  * [9] N. Savinov, A. Raichuk, D. Vincent, R. Marinier, M. Pollefeys, T. Lillicrap, and S. Gelly, “Episodic Curiosity through Reachability,” in _International Conference on Learning Representations (ICLR)_ , 2018. https://openreview.net/forum?id=SkeK3s0qKQ
  * [10] D. Shah, B. Eysenbach, G. Kahn, N. Rhinehart, and S. Levine, “ViNG: Learning Open-World Navigation with Visual Goals,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE Press, 2021, pp. 13 215–13 222. https://doi.org/10.1109/ICRA48506.2021.9561936
  * [11] L. Mezghan, S. Sukhbaatar, T. Lavril, O. Maksymets, D. Batra, P. Bojanowski, and K. Alahari, “Memory-Augmented Reinforcement Learning for Image-Goal Navigation,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , Oct. 2022, pp. 3316–3323, iSSN: 2153-0866.
  * [12] D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine, “GNM: A General Navigation Model to Drive Any Robot,” in _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , May 2023, pp. 7226–7233.
  * [13] D. Shah, A. Sridhar, N. Dashora, K. Stachowicz, K. Black, N. Hirose, and S. Levine, “ViNT: A Large-Scale, Multi-Task Visual Navigation Backbone with Cross-Robot Generalization,” in _Proceedings of the 7th Annual Conference on Robot Learning_. PMLR, Aug. 2023. https://openreview.net/forum?id=-K7-1WvKO3F
  * [14] A. Kar, A. Prakash, M.-Y. Liu, E. Cameracci, J. Yuan, M. Rusiniak, D. Acuna, A. Torralba, and S. Fidler, “Meta-Sim: Learning to Generate Synthetic Datasets,” in _IEEE/CVF International Conference on Computer Vision (CVPR)_ , 2019, pp. 4551–4560. https://openaccess.thecvf.com/content˙ICCV˙2019/html/Kar˙Meta-Sim˙Learning˙to˙Generate˙Synthetic˙Datasets˙ICCV˙2019˙paper.html
  * [15] C. Masone and B. Caputo, “A Survey on Deep Visual Place Recognition,” _IEEE Access_ , vol. 9, pp. 19 516–19 547, 2021.
  * [16] D. Singh Chaplot, R. Salakhutdinov, A. Gupta, and S. Gupta, “Neural Topological SLAM for Visual Navigation,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. Seattle, WA, USA: IEEE, June 2020, pp. 12 872–12 881. https://ieeexplore.ieee.org/document/9157610/
  * [17] X. Meng, N. Ratliff, Y. Xiang, and D. Fox, “Scaling Local Control to Large-Scale Topological Navigation,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , May 2020, pp. 672–678, iSSN: 2577-087X.
  * [18] B. Eysenbach, R. R. Salakhutdinov, and S. Levine, “Search on the Replay Buffer: Bridging Planning and Reinforcement Learning,” in _Advances in Neural Information Processing Systems_ , vol. 32. Curran Associates, Inc., 2019\. https://proceedings.neurips.cc/paper/2019/hash/5c48ff18e0a47baaf81d8b8ea51eec92-Abstract.html
  * [19] D. Shah, B. Eysenbach, N. Rhinehart, and S. Levine, “Rapid Exploration for Open-World Navigation with Latent Goal Models,” in _Proceedings of the 5th Conference on Robot Learning_. PMLR, Jan. 2022, pp. 674–684, iSSN: 2640-3498. https://proceedings.mlr.press/v164/shah22a.html
  * [20] N. Hirose, D. Shah, A. Sridhar, and S. Levine, “SACSoN: Scalable Autonomous Control for Social Navigation,” July 2023, arXiv:2306.01874 [cs]. http://arxiv.org/abs/2306.01874
  * [21] S. Triest, M. Sivaprakasam, S. J. Wang, W. Wang, A. M. Johnson, and S. Scherer, “TartanDrive: A Large-Scale Dataset for Learning Off-Road Dynamics Models,” in _2022 International Conference on Robotics and Automation (ICRA)_ , May 2022, pp. 2546–2552.
  * [22] Sivic and Zisserman, “Video Google: a text retrieval approach to object matching in videos,” in _EEE/CVF International Conference on Computer Vision (ICCV)_ , Oct. 2003, pp. 1470–1477 vol.2.
  * [23] H. Jégou, M. Douze, C. Schmid, and P. Pérez, “Aggregating local descriptors into a compact image representation,” in _IEEE/CVF International Conference on Computer Vision (CVPR)_ , June 2010, pp. 3304–3311, iSSN: 1063-6919.
  * [24] A. Torii, R. Arandjelović, J. Sivic, M. Okutomi, and T. Pajdla, “24/7 place recognition by view synthesis,” in _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2015, pp. 1808–1817, iSSN: 1063-6919.
  * [25] R. Arandjelović, P. Gronat, A. Torii, T. Pajdla, and J. Sivic, “NetVLAD: CNN Architecture for Weakly Supervised Place Recognition,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 40, no. 6, pp. 1437–1451, June 2018.
  * [26] J. Revaud, J. Almazan, R. Rezende, and C. D. Souza, “Learning With Average Precision: Training Image Retrieval With a Listwise Loss,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_. Seoul, Korea (South): IEEE, Oct. 2019, pp. 5106–5115. https://ieeexplore.ieee.org/document/9010047/
  * [27] G. Berton, C. Masone, and B. Caputo, “Rethinking Visual Geo-localization for Large-Scale Applications,” in _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. New Orleans, LA, USA: IEEE, June 2022, pp. 4868–4878. https://ieeexplore.ieee.org/document/9880209/
  * [28] F. Radenović, G. Tolias, and O. Chum, “Fine-Tuning CNN Image Retrieval with No Human Annotation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 41, no. 7, pp. 1655–1668, July 2019.
  * [29] M. J. Milford and G. F. Wyeth, “SeqSLAM: Visual route-based navigation for sunny summer days and stormy winter nights,” in _2012 IEEE International Conference on Robotics and Automation (ICRA)_ , May 2012, pp. 1643–1649, iSSN: 1050-4729.
  * [30] P. Hansen and B. Browning, “Visual place recognition using HMM sequence matching,” in _2014 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , Sept. 2014, pp. 4549–4555, iSSN: 2153-0866.
  * [31] E. Pepperell, P. I. Corke, and M. J. Milford, “All-environment visual place recognition with SMART,” in _2014 IEEE International Conference on Robotics and Automation (ICRA)_ , May 2014, pp. 1612–1618, iSSN: 1050-4729.
  * [32] T. Naseer, W. Burgard, and C. Stachniss, “Robust Visual Localization Across Seasons,” _IEEE Transactions on Robotics_ , vol. 34, no. 2, pp. 289–302, Apr. 2018.
  * [33] A. Gawel, C. D. Don, R. Siegwart, J. Nieto, and C. Cadena, “X-View: Graph-Based Semantic Multi-View Localization,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 3, pp. 1687–1694, July 2018.
  * [34] Y. Latif, R. Garg, M. Milford, and I. Reid, “Addressing Challenging Place Recognition Tasks Using Generative Adversarial Networks,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , May 2018, pp. 2349–2355, iSSN: 2577-087X.
  * [35] S. Garg and M. Milford, “SeqNet: Learning Descriptors for Sequence-Based Hierarchical Place Recognition,” _IEEE Robotics and Automation Letters_ , vol. PP, pp. 1–1, 2021.
  * [36] M. Xu, N. Snderhauf, and M. Milford, “Probabilistic Visual Place Recognition for Hierarchical Localization,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 311–318, Apr. 2021.
  * [37] M. Xu, T. Fischer, N. Sünderhauf, and M. Milford, “Probabilistic Appearance-Invariant Topometric Localization With New Place Awareness,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 4, pp. 6985–6992, Oct. 2021.
  * [38] N. Carlevaris-Bianco, A. Ushani, and R. Eustice, “University of Michigan North Campus long-term vision and lidar dataset,” _The International Journal of Robotics Research_ , vol. 35, Dec. 2015.
  * [39] Z. Chen, A. Jacobson, N. Sünderhauf, B. Upcroft, L. Liu, C. Shen, I. Reid, and M. Milford, “Deep learning features at scale for visual place recognition,” in _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , May 2017, pp. 3223–3230.
  * [40] F. Warburg, S. Hauberg, M. Lopez-Antequera, P. Gargallo, Y. Kuang, and J. Civera, “Mapillary Street-Level Sequences: A Dataset for Lifelong Place Recognition,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 2626–2635. https://openaccess.thecvf.com/content˙CVPR˙2020/html/Warburg˙Mapillary˙Street-Level˙Sequences˙A˙Dataset˙for˙Lifelong˙Place˙Recognition˙CVPR˙2020˙paper.html
  * [41] J. Johnson, M. Douze, and H. Jégou, “Billion-Scale Similarity Search with GPUs,” _IEEE Transactions on Big Data_ , vol. 7, no. 3, pp. 535–547, July 2021, conference Name: IEEE Transactions on Big Data.
  * [42] S. Thrun, D. Fox, W. Burgard, and F. Dellaert, “Robust Monte Carlo localization for mobile robots,” _Artificial Intelligence_ , vol. 128, no. 1, pp. 99–141, May 2001. https://www.sciencedirect.com/science/article/pii/S0004370201000698
  * [43] M. Tan and Q. Le, “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks,” in _Proceedings of the 36th International Conference on Machine Learning (ICML)_. PMLR, May 2019, pp. 6105–6114, iSSN: 2640-3498. https://proceedings.mlr.press/v97/tan19a.html
  * [44] P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savva, and A. R. Zamir, “On Evaluation of Embodied Navigation Agents,” _arXiv:1807.06757 [cs]_ , July 2018. http://arxiv.org/abs/1807.06757
  * [45] J. Wasserman, K. Yadav, G. Chowdhary, A. Gupta, and U. Jain, “Last-Mile Embodied Visual Navigation,” in _Proceedings of The 6th Conference on Robot Learning_. PMLR, Mar. 2023, pp. 666–678, iSSN: 2640-3498. https://proceedings.mlr.press/v205/wasserman23a.html
  * [46] G. Berton, R. Mereu, G. Trivigno, C. Masone, G. Csurka, T. Sattler, and B. Caputo, “Deep Visual Geo-Localization Benchmark,” in _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022, pp. 5396–5407. https://openaccess.thecvf.com/content/CVPR2022/html/Berton˙Deep˙Visual˙Geo-Localization˙Benchmark˙CVPR˙2022˙paper.html
  * [47] N. Hirose, F. Xia, R. Martín-Martín, A. Sadeghian, and S. Savarese, “Deep Visual MPC-Policy Learning for Navigation,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 4, pp. 3184–3191, Oct. 2019.
  * [48] G. Berton, C. Masone, and B. Caputo, “Rethinking Visual Geo-localization for Large-Scale Applications,” in _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. New Orleans, LA, USA: IEEE, June 2022, pp. 4868–4878. https://ieeexplore.ieee.org/document/9880209/
  * [49] H. Jegou, M. Douze, and C. Schmid, “On the burstiness of visual elements,” in _2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE Computer Society, June 2009, pp. 1169–1176, iSSN: 1063-6919. https://www.computer.org/csdl/proceedings-article/cvpr/2009/05206609/12OmNzdoMkb