Mobile service robots are increasingly prevalent in human–centric, real–world domains, operating autonomously in unconstrained indoor environments. In such a context, robotic vision plays a central role in enabling service robots to perceive high–level environmental features from visual observations. Despite the data–driven approaches based on deep learning push the boundaries of vision systems,  applying these techniques to real–world robotic scenarios presents unique methodological challenges. Traditional models fail to represent the challenging perception constraints typical of service robots and must be adapted for the specific environment where robots finally operate. We propose a method leveraging photorealistic simulations that balances data quality and acquisition costs for synthesizing visual datasets from the robot perspective used to train deep architectures. Then, we show the benefits in qualifying a general detector for the target domain in which the robot is deployed, showing also the trade–off between the effort for obtaining new examples from such a setting and the performance gain.
In our extensive experimental campaign, we focus on the door detection task (namely recognizing the presence and the traversability of doorways) that, in dynamic settings, is useful to infer the topology of the map. Our findings are validated in a real–world robot deployment, comparing prominent deep–learning models and demonstrating the effectiveness of our approach in practical settings.

§ INTRODUCTION

Mobile service robots represent a flourishing technology increasingly employed across a range of human–centric, real–world domains such as office and domestic environments. Typically deployed for the long term, these robots find in robotic vision – the ability to understand semantic knowledge from visual robot perceptions in real time– a cornerstone to achieving high–level autonomy and operational awareness. Indeed, one of the key capabilities is the one to detect objects, which can significantly enhance a variety of sub–tasks across increasing levels of abstraction, including localization, navigation, scene understanding, and planning.

Recent developments in object detection, largely driven by research in deep learning, have unlocked remarkable possibilities for addressing this real–world AI challenge, facilitating the creation of highly effective vision modules for mobile robots. However, despite the abundance and diversity of available models, their practical application in field robotics continues to pose methodological and practical challenges. In this work, we introduce a method designed to address them in the context of a particularly significant task for service robots, whose functionalities are strongly based on autonomous navigation: door detection.

Doors are key environmental features for a mobile robot. They represent the topological connections between adjacent sub–regions of the free space whose traversability, importantly, might not always be possible. More specifically, our definition of a door is centered on the notion of variable–traversability passage, without necessarily assuming the presence of typical door components like a leaf, a hinge, or any other related furniture (this is what in the literature is commonly referred to as “explicit” doors <cit.>). Door detection is the capability for a robot to recognize in real–time the location and traversability status ( or ) of such passages. This problem is most effectively addressed as an object detection task using RGB images acquired by the robot, primarily due to the limitations of alternative technologies. For example, laser range scanners, while excellent for distance measurements, are typically constrained by a 2D field of view. In the same line,  cameras often exhibit a limited depth sensing range, making them less reliable over longer distances or in larger indoor spaces. Furthermore, both types of sensors can struggle with transparent or highly reflective surfaces, that are often present in doors. Figure <ref> illustrates the execution of door detection: a mobile robot navigates in an environment to perform its tasks; at the same time it acquires images through its onboard camera. For each image, it infers in real–time the bounding boxes of doors, distinguishing between open doors (depicted in green) and closed ones (depicted in red), also highlighted on the map.

Giraff–X <cit.>, the service robot adopted in this work, performing door detection.

In practical deployments, service robots are subject to specific perception constraints and frequently encounter challenging recognition instances, such as partially occluded or poorly positioned viewpoints. These challenges are intrinsic to the domain of service robots, as they are designed to operate in environments marked by dynamism and clutter. This condition induces critical domain shifts over the state–of–the–art object detectors, which are typically trained over datasets that largely neglect the noisy, constrained, and challenging operational conditions that a robot faces on the field. Properly and efficiently adapting these models for service robots is an open methodological problem, yet one of remarkable practical importance.

Two important facts should be considered about a robot operating in its target environment: on one side, human–centric indoor environments are indeed complex and difficult to process using computer vision due to the aforementioned challenges. On the other side, while the position of the objects in the target environment is highly dynamic, the robot will see the same object instances multiple times. Moreover, different instances of the same type of object will likely have a similar aspect. For example, while it is highly likely that two chairs from two different environments look very different, the chairs around the same table are usually of the same model. The same holds for doors. As a result of this, the robot during its daily activities will observe multiple times the same set of objects, and sets of objects similar among them. This fact can be used to improve the perceptual capabilities of the robot in its target environment, compensating for the challenges of the domain.

In this work, we demonstrate how the integration of simulation frameworks and domain adaptation techniques can create an effective development pipeline for constructing door detectors specifically tailored for mobile service robots.  Specifically, the contributions we provide are the following.

* We analyze the trade–offs involved in using simulations to train a door detector capable of generalizing effectively across various environments. We delineate the desiderata of this process and propose a framework based on simulation performed in 3D real–world models, Gibson <cit.>, that achieves a balance between data photorealism and acquisition costs.

* We explore fine–tuning to qualify detectors to a robot's target environment, demonstrating how leveraging typical operational settings of service robots can enhance performance in challenging instances.

* We argue that performance metrics used to evaluate computer–vision models are not well suited to be used in a robotic context, and we propose new evaluation metrics specifically designed to better reflect our setting.

* We conduct an extensive experimental campaign with three prominent deep–learning models for object detection, providing insights into how domain shifts in our scenario are influenced by the nature of the training datasets.

* We assess them on both general and environment–specific door detectors, identifying the most effective training setups.  We evaluate the trade–off between training efforts and performance improvements, validating our findings in an actual deployment on a service robot.

* We evaluate the impact of different performance of a door detection method on a downstream task, that is to evaluate the current traversability status of the whole environment.

In the next section, we briefly survey the related works relevant to this study. Section <ref> motivates and outlines our methods while the remainder of the paper is devoted to experimental analysis. Section <ref> extensively analyses the performance of our general and qualified detectors while Section <ref> focuses on the evaluation in a real robot deployment. Concluding remarks are provided in Section <ref>.

The contributions of this paper build upon and extend our earlier work presented in <cit.>, where we have illustrated a preliminary version of the findings presented here. In this work, we significantly extend the experimental evaluation also by assessing multiple models for object detection.

* Motivating scenario: autonomous service robots are deployed in a variety of real-world environments. They are a promising technology. They cover many different tasks, examples can be found in assistance, logistics, and surveillance. They typically operate in the same environment for a long time.

* Many tasks performed by service robots depend on their ability to autonomously construct models of the environment and make inferences from them. Achieving this capability requires the extraction of high-level knowledge from real-world perceptions. For instance, this includes tasks like creating a map and performing navigation on it in order to accomplish specific tasks.

* In this paper, we focus on one of these challenges that service robots must address, which is the real-time recognition of environmental features using RGB cameras, a.k.a. Robotic Vision.

* Why are we considering this task? For two reasons.

* Significance. The ability to recognize visual features within the environment plays a key role in enhancing a service robot's operational awareness. This, in turn, holds the potential to improve tasks such as localization, navigation, semantic understanding, and planning. These aspects collectively underpin the effectiveness of service robot deployments, making this challenge very important.

* Hardness profile. This task offers a relatively low entry barrier, meaning that achieving a baseline level of performance is attainable without much effort (a great deal here is played by the availability of off-the-shelf detectors). However, it also presents an intriguing high ceiling. In practical deployments, service robots frequently encounter challenging recognition instances, such as identifying partially occluded objects or contending with poorly positioned viewing angles. These challenges are intrinsic to the domain of service robots, as they are typically subject to perception constraints in environments marked by dynamism and clutter. The development of detectors capable of consistently delivering high-quality performance under such conditions is quite hard. Importantly, these difficult instances can have a great impact on the service provided by the robot.

* How is this task addressed today? The family of techniques is that of deployed/applied Computer Vision. In our work we shall call it Robotic Vision, to make explicit its grounding to our robotic domain. [Provide many references to underline that Deployed/Applied/Robotic Vision, and not just CV, is a research area with its own challenges] Early techniques were based on manually defined features, today the arena is dominated by deep-learning detectors which have been trained to localize a variety of object classes in RGB images. These detectors are typically trained on large datasets of images taken from a wide range of viewpoints and angles (for example, COCO <cit.>). Such a variety of viewpoints does not represent in any way the typical robot POV.

* Research question: how to build and deploy robotic vision modules in service robots?

* Why is the research question important? From one side there are many readily available deep-learning Computer Vision (CV) models and architectures. However, naively deploying those models simply does not work, for a number of reasons that, in this paper, we empirically show and discuss. A general methodology for this challenge is still far from being accepted and well-defined. Contributing to it (like we do in our paper) will allow practitioners to better leverage the potential of deep-learning CV models in service robots.

* What's our original contribution? We contribute to a methodology to deploy deep robotic vision on service robots. Our proposed methodology is made by two steps.

* Building a General Detector (GD).
This is an object detector that can deliver consistent performance across diverse operational environments, which is why we refer to it as “general”. We demonstrate the limitations of adopting bare off-the-shelf detectors and emphasize the need for re-training using image acquisitions that properly represent the robot's POV. We discuss a tradeoff between training costs and achieved performance, proposing a method based on simulation that can balance the two factors in a satisfactory way. We show that this technique is valid and achieves acceptable performance in the real world.

* Obtaining a Qualified Detector (QD). This is a detector that has been specialized (fine-tuned) for the specific environment in which the robot has to operate. We exploit some features of our reference scenario (the fact that there's an initial installation phase to which a long operation time follows) to perform fine-tuning with additional data gathered in the target environment. We empirically evaluate the costs of this procedure comparing them with the obtained benefits. We show that the method is valid and achieves good performance in difficult instances encountered in the real world.

We extensively evaluate our method in two real indoor use cases for service robots.

* Detection of passages between rooms. This is a task that has a remarkable impact on navigation tasks in an environment where the connectivity between areas can vary.

* Detection of landmarks. This is a key task that can boost localization. Consider removing.

§ STATE OF THE ART

Mobile service robots are a cutting–edge technology that is increasingly being adopted in a variety of real–world scenarios. These robots are typically employed to assist humans in various tasks, often unfolding in indoor industrial, or domestic environments <cit.>. Among recent and representative application domains are healthcare, where robots assist patients and caregivers <cit.>; logistics, where they carry out repetitive tasks like item deliveries or environmental monitoring <cit.>;  at–home caregiving, where they aid in day–by–day activities like cleaning or provide additional services such as personal assistance, wellbeing monitoring, and social entertainment <cit.>.

Although the adoption of mobile service robots is increasing, their deployment in indoor human–centric environments such as houses, offices, hospitals, and schools, continues to present substantial research challenges. Unlike industrial settings, where there is a higher degree of control and predictability, the lived–in setup is typically unstructured and dynamic. This complexity arises both from the physical layout of such environments, that is how rooms, walls, and furniture are disposed, and by their visual aspect, which is complex and semantic–rich; consequently, two different environments of the same type can have very different features. As examples of recent research works show, in human–centric environments, tasks such as user–understanding <cit.>, activity recognition <cit.>, and even path planning <cit.> face additional challenges.

To properly work inside these complex settings, a robot should be able to understand relevant properties through its vision. In this domain, one of the fundamental problems is the one of real–time object detection.
Indeed, such functionality is key for service robots, enhancing their capabilities and enabling autonomous behavior in various situations <cit.>. Object detection is typically performed from images acquired with RGB cameras, and the recent advancements in deep learning applied to computer vision <cit.> have led to the development of high–performance pre–trained models, which can be exploited to obtain robotic vision modules running locally on mobile robots. These models, among which prominent examples are YOLO <cit.>, DETR <cit.>, and Faster R–CNN <cit.>, offer significant advantages in object detection tasks for service robots, but their field deployment poses a set of engineering and methodological challenges, especially in unstructured environments. Our work focuses on tackling common problems of object detection in human–centric environments by assessing a specific and widely relevant case study,  door detection.

The ability of mobile robots to detect doors is key for indoor operations. The traversability status of a door ( or ) enables the availability of passageways between sub–areas of an environment. In turn, traversability directly determines the environment's topology, ultimately affecting the robot's navigational routes and accessibility of the areas therein.  The location of doors is important for tasks such as room segmentation <cit.>, which entails partitioning the map of an environment into semantically meaningful areas or rooms. This knowledge is also beneficial in predicting the layout of rooms not yet observed during exploration <cit.> or in identifying temporary unreachable locations during mapping. Furthermore, it plays a key role in place categorization, a process where rooms on the occupancy map are assigned semantic labels (like “corridor” or “office”) based on their visual appearance <cit.>. Recent studies have shown that a robot's ability to recognize doors can significantly enhance its navigation capabilities in long–term scenarios. For instance, the work presented in <cit.> models the periodic changes in dynamic environments over extended periods. Similarly, the study in <cit.> introduces a navigation system designed for robots functioning in indoor environments for long periods, particularly where the traversability of the area varies over time.

The use of object detection methods is the mainstream approach to tackle door detection with mobile robots. Initial seminal methods in this domain relied on the extraction of handcrafted features <cit.>, such as edges <cit.> and corners <cit.>, to describe the characteristic rectangular shape of doors. However, the requirement to explicitly define and combine these features is a significant limitation of these approaches. This constraint hampers their robustness and adaptability, especially when dealing with the highly variable images encountered in real dynamic environments.

Deep learning end–to–end methods have brought significant improvements in the field. Their ability to automatically learn features that characterize an object class, and robustly handling variations in scale, position, rotation, and lighting, is a major advantage that has led to their widespread use in mobile robotics. A pioneering method for door recognition in mobile robot navigation was introduced in <cit.>. This method utilizes color and shape as key features to detect doors in office environments, employing two neural classifiers to identify these elements in images. These features are then integrated using a heuristic algorithm to determine if they form a typical door structure. The study in <cit.> presents a method for door detection aimed at enhancing the autonomous navigation of mobile robots. A convolutional neural network is trained to identify doors in indoor settings, demonstrating its utility in aiding a robot's efficiency in traversing passages. Additionally, recent research has explored the integration of RGB vision with other sensors commonly used in robot navigation <cit.> and the identification of doors and their handles to enable interactions like grasping <cit.>. For example, the research in <cit.> utilizes a YOLO–based deep learning framework <cit.> for the detection of door Regions Of Interest (ROI). This approach specifically targets the handles by focusing on the area encapsulated within the door's ROI, effectively locating the handles for interaction purposes.

The studies previously mentioned offer approaches to the door detection task in scenarios similar to the one we address, yet they do so only to a limited extent. A notable limitation in these studies is the absence of training from a mobile robot's perspective. Additionally, these methods often do not exploit the typical operational conditions of a robot. Our work introduces a strategy specifically tailored to address these shortcomings. Viewing this from a broader angle, our method addresses the domain adaptation challenge (a well–recognized issue in deep learning at large <cit.>) within the realm of door detection using mobile robots in unstructured indoor environments. Similar findings are provided by the work of <cit.>, where is presented a dataset of door handles for training robotic manipulation methods to open/close doors. Our proposed technique exploits the technique of fine–tuning of pre–trained deep neural models, a practice extensively employed in autonomous mobile robotics both relying on manually labeled data <cit.> and self–supervised methods <cit.>.

* Service Robots: what do we mean by them? What are the most representative application domains? What are the most representative platform setups (Giraff is one example)? Put an emphasis on their sensors, their environments, and their importance in many key applications in our society.

* Object Detection (OD), what is it, what are the main applications (among which robotics), what are the state-of-the-art techniques which are used today. Introduce our three selected architectures as a comprehensive set of popular techniques against a larger background.

* Robotic Vision (RV), a.k.a. OD in the context of service robots. Explain why this is an important capability for service robots. Provide examples of works adopting OD methods to do RV. When possible, highlight the limitations of these methods to better support our work. Put an emphasis on the environmental features. Define the task in relation to specific objects of interest that the robot is tasked to recognize.

* Specialize the previous point with a detailed discussion of our use cases. Passage detection: examples of works doing it, examples of works that would benefit by doing it (or by doing it better). Landmark detection (same points as the other use case) Support the fact that our passage-detection use case is indeed an OD task.

* Simulation in Robotics. The use of simulations is a widespread practice in mobile robotics, provide some examples of applications where it plays an important role (mapping, planning, etc.). Simulators introduce a sim-to-real gap that ultimately is reflected in the robot's performance.
Various simulator frameworks are designed to minimize this gap for specific tasks, including mapping, perception, and integration with ROS (provide examples of simulators or simulator frameworks that have been or can be used for robots). In this work, we investigate the use of simulation.

* This paper is an extension of our previous work <cit.>.

§ GENERAL AND QUALIFIED DOOR DETECTION FOR SERVICE ROBOTS

We focus on a service robot designed to autonomously operate in human–centric indoor environments. We assume a widely–used hardware setup, where the robot is equipped with one or more RGB cameras for vision. The primary objective is developing real–time door detection, namely the task of processing an RGB image acquired by the robot to determine the bounding box and binary traversability status ( or ) for each visible door.

A general overview of the proposed method.

The method we propose, graphically summarized in Figure <ref>, is structured around the two principal phases that define the lifecycle of a mobile service robot: the robot's development phase and the subsequent deployment phase.
With this method, we aim to identify, experimentally evaluate, and solve some of the challenges that are intertwined with the usage of vision–based object detection methods on mobile robots.

The development phase for a service robot involves preparing and configuring the platform, including the installation and setup of hardware and software components. The objective here is to setup a robot that is ready to meet the challenges of real–world environments. This work's focus lies in the domain of visual perception capabilities, aiming to equip the robot with vision skills that perform satisfactorily across various environments, thereby ensuring a high level of generalizability. The development phase is the starting ground for addressing our door–detection task. Our approach involves creating a General Detector (GD), designed to recognize doors while adhering to the perception constraints of service robots and maintaining consistent performance in various environments. A significant part of our method involves utilizing simulation to develop a photorealistic visual dataset, representing typical visual perceptions of a robot. This dataset is then used to train a GD, ensuring it achieves baseline performance in the real world.

During the deployment phase, the service robot is introduced for autonomous operation in a target environment, usually for an extended period. This phase often involves a domain shift, presenting challenges to the performance of the previously developed GD. This is because the pre–built computer vision methods, focusing on the model's development, present difficulties introduced by our environmental setup that prevent their straightforward use on autonomous mobile robots. Given the long–term nature of this phase, there is an opportunity to incrementally fine–tune the GD with data collected in the target environment, aligning it more closely with the specific visual features at hand, thus obtaining a Qualified Detector (QD). This detector can exploit the fact that usually multiple instances of the same object within the same environment present similar features, that are stable in time. Doors and windows are good examples of this fact: within a target environment, most of them are usually produced by the same manufacturer and are of the same type. The process of adapting the detection model to the target environment may require the collection and annotation of data for which we propose a method demonstrating a trade–off between the effort required and the resulting performance improvements.

§.§ Training a General Detector

The recent trends in object detection suggest that a straightforward way to address Robotic Vision is to plug and play a deep detector in a robotic platform <cit.>. Despite the availability of a large number of effective models, when faced with reality this simple approach presents several engineering challenges and an established method to provide a GD for service robots still needs a comprehensive investigation.

State–of–the–art object detectors such as the one we shall consider, namely DETR <cit.>, YOLO <cit.>, and Faster R–CNN <cit.>, are typically trained on prominent datasets like Pascal VOC <cit.>, ImageNet Large Scale Visual Recognition Challenge <cit.>, or, most commonly, MS COCO <cit.>. These datasets comprise thousands of images from diverse contexts (including both indoor and outdoor settings) and feature various object categories captured from application–agnostic viewpoints. However, when applying these models to robotics, two primary challenges arise. First, if the dataset does not extensively represent the object of interest, the detector's ability to recognize it may be compromised. Our survey of existing datasets indicates that doors frequently suffer from this lack of representation. Secondly, even when the object of interest is well–represented, distribution shifts between the training data and real–world scenarios can significantly affect performance. This widely recognized yet largely unresolved issue is particularly problematic in our context, as the shift can affect multiple aspects: the input data, the feature space, and the data collection process itself (here we rely on the discussion about domain shift types proposed in <cit.>).

Perhaps the dataset for door detection that is most relevant to our work is DeepDoors2 (DD2) <cit.>, which contains around $3000$ images of doors, each annotated with a bounding box and traversability status[Note how a dataset of this size is customary in several object detection tasks: as an example, in MS COCO, the average number of examples per class, is $3200$; the only exception is the category , which has more examples, over $10$K. ]. However, in our scenario, DD2 is susceptible to performance degradation due to distribution shifts, a fact that becomes expected already upon examining some of its examples. The images in DD2 are captured from human–like perspectives, often showing the door fully visible and centrally located, as depicted in the indoor and outdoor examples of Figure <ref> and Figure <ref>, respectively. This dataset overlooks instances such as partially visible or nested doors, which are common in robots' perceptions. Labels are provided only for doors that are completely within the frame and distinct enough for clear identification, as shown in the dashed bounding boxes of Figure <ref> (partially visible door) and Figure <ref> (nested doors). These shortcomings are, to varying degrees, present in nearly all conventional computer vision datasets <cit.>, reflecting their inherent limitations in capturing a robot's visual perception model <cit.>. As our experimental campaign will demonstrate concretely, these limitations significantly affect performance.

Examples from the DD2 dataset <cit.> of open and closed doors (in green and red, respectively). The dashed bounding boxes represent missing annotations.

To address them, one common method is fine–tuning a large–scale pre–trained model (such as one trained on MS COCO <cit.>) with new examples that better represent the target object distribution. This approach is prevalent, especially in robotics <cit.>, and the strategy we evaluate in this work is based on it. Ideally, creating an effective door detector through fine–tuning requires a dataset that:

* demonstrates a high level of photorealism (to withstand distribution shifts at the input level);

* encompasses a variety of indoor environments with diverse features (to withstand distribution shifts at the feature level);

* accurately reflects the robot's perspective and perception model (to withstand distribution shifts in the data acquisition process).

Currently, no dataset fulfilling these criteria exists in the literature, as efficiently collecting it is still an open problem.
An alternative to this issue is to use labeled sequences of images obtained by a robot or by a mobile platform, such as in ScanNet <cit.> or SUN3D <cit.>. However, these sequences are usually collected within single rooms and, as they are based on fixed trajectories, do not allow the sampling of new viewpoints from different perspectives that may be encountered by the robot while navigating.
The most straightforward approach would involve an extensive data collection campaign using robots in real–world environments, gathering image samples and manually labeling them. However, the logistics and costs associated with this method are prohibitively high and well–recognized among robotics professionals. In the following, we tackle this problem by exploiting simulation <cit.>, an approach frequently employed in robotics to mitigate the large costs of on–the–field experimentation. The empirical results we present later will demonstrate how, with appropriate design measures, simulations can provide a dataset from which an effective door detector can be trained.

§.§.§ The Proposed Simulation Framework

Common 3D physics simulators like Gazebo <cit.> or CoppeliaSim <cit.> are widely adopted for prototyping control software in robotics before real–world deployment <cit.>. However, their lack of a sophisticated rendering pipeline for realistic visual perceptions makes them unsuitable for our setting. Early efforts to address this limitation have involved the use of 3D game engines, such as Unity3D <cit.> or Unreal <cit.>, to recreate complex robotics scenarios, including unique environments or specialized physical laws, as seen in autonomous vehicles <cit.>, UAVs <cit.>, or surgical robotics <cit.>. Despite their adaptability, customizing these game engines for a particular robotic application can be challenging <cit.>. Specifically, for the task considered in this paper, this would involve manually creating synthetic scenes that accurately reflect the structural features of real indoor environments, a task that is more aligned with environment design than robotics engineering.

Recently, the introduction of interactive realistic simulations for embodied AIs, as iGibson <cit.>, has helped mitigate the limitations of traditional simulators for indoor robotic tasks. iGibson comes with 15 artificially constructed home–sized scenes, which are developed by populating layouts of actual environments with 3D controllable objects whose configuration, shape, material, and texture can be automatically changed. Unlike the simulators mentioned earlier, iGibson seamlessly integrates with the Robot Operating System (ROS), facilitating the collection of extensive, high–quality annotated image datasets from a robot's perspective. However, despite these significant advantages, simulators based on synthetic scenes still fall short in achieving the crucial aspect of photorealism. This limitation is something we empirically assess in our experimental campaign.

In addressing these challenges, we adopted an approach that balances the photorealism of real–world data acquisition with the automation benefits of synthetic simulations. Our solution relies on Gibson <cit.>, a simulator designed for embodied agents with an emphasis on enhancing the photorealism of visual perceptions. This enhancement aids in the effective transfer of models trained within the simulator to real–world environments. Gibson achieves this by employing scene datasets scanned directly from real environments, such as Matterport3D <cit.> and Stanford–2D–3Ds <cit.>, which accurately replicate the complexity of the real world. Additionally, it incorporates a neural rendering pipeline to further bridge the sim–to–real gap. Leveraging these features, we developed a simulation framework based on Gibson, in conjunction with Matterport3D, a comprehensive RGB–D dataset comprising 90 digitized real scenes also including semantic tagging for both instance and category–level segmentation. This combination allows us to achieve a balance between photorealism and the controlled conditions necessary for effective simulation.

Gibson provides a middleware for controlling a ROS–based virtual robot. While camera perceptions can be easily simulated (setting resolution and FOV), navigation encounters several technical limitations. First, conducting a real–time acquisition campaign, even in a simulated environment, can be time–intensive. Moreover, this approach does not offer complete control over the data acquisition process, as much of it depends on the navigation stack of the simulated robot. Additionally, the 3D polygonal meshes of the Matterport3D environments, which are digitized from real–world settings, are often cluttered and noisy. This results in various issues: furniture models appear malformed and incomplete (as shown in Figure <ref>, where the legs of the table are not modeled), walls frequently have holes near windows or mirrors (see Figure <ref>, where the bed should be behind a mirror and a wall, which are missing), and the surfaces of floor plans are irregular (see Figure <ref>). Such artifacts, which concern the 3D meshes and not the images, can lead to inaccuracies and failures when the robot navigates in the simulated environment.

Matterport3D mesh malformations. (a) The table and chairs have no legs. (b) A wall in the bedroom, in front of the bed, is missing. (c) Floor plan irregularities. (d) An example of perception acquired from the Gibson–based simulation framework.

To address these shortcomings, we have developed an enhanced version of Gibson, which introduces a highly controllable simulation mechanism. This upgraded simulation framework[The pre-compiled python package is available at <https://pypi.org/project/gibson/>, the source code can be found at <https://github.com/micheleantonazzi/GibsonEnv>.] allows to script robot teleporting actions to any location relaxing some constraints from the physics engine such as gravity or collisions. Such a capability can enable large–scale batch data acquisition without the risk of operational failures. With this system, the robot can effectively operate over uneven floor surfaces and across different floors without encountering issues related to architectural barriers, like stairs or elevators. This approach significantly streamlines the data–gathering process, ensuring efficient and uninterrupted data collection in simulated environments. Figure <ref> shows an example of an acquisition obtained with this simulation framework.

§.§.§ Pose Extraction

To effectively exploit the simulation framework proposed above it is crucial to ensure that the data acquired aligns with the perception model of a service robot. To model how a robot perceives human–centric environments, we rely on the experience obtained in a long–term deployment of service robots, described in <cit.>. To achieve this, we propose a method for principled selection of perception poses. First, data acquisition should occur from locations within the free space that also maintain a minimum clearance from the nearest obstacles. Additionally, these locations ought to be strategically positioned along the shortest paths connecting key areas of the environment. This positioning is key as these paths are the most likely to be covered by a robot during its service time. Third, it is important to distribute the locations uniformly throughout the environment to minimize redundancy and to ensure comprehensive coverage of the environment's visual features. Our method is composed of three distinct phases.

The initial phase focuses on generating a 2D map of the environment from the 3D mesh provided by the simulation framework. This process involves aggregating obstacles identified through multiple cross–sections of the 3D mesh, which are created using parallel planes starting from a few centimeters over the floor level. The resulting map undergoes erosion and dilation to eliminate small gaps between obstacles so as to exclude areas that are unreachable or too close to obstacles.

Figure <ref> presents some key outcomes of these steps. Specifically to this first phase, Figure <ref> illustrates the 3D mesh of a simulated environment, while Figure <ref> displays the corresponding 2D map.

A 3D mesh of an environment obtained from the simulator

Map of free ($F$, white) and obstacle ($O$, black) space

Grid cells ($\mathcal{G}_0$, in red) lying on the Voronoi boundary

Navigation graph ($\mathcal{G}$, in red)

2D locations (in red) of the perception poses ($P$)

Different phases of our pose extraction method.

In the second phase, the extracted map is used to compute a navigation graph, a data structure that represents the principal routes likely to be traversed by a robot. This process is detailed in Algorithm <ref>. The map, denoted as $\mathcal{M}=(F,O)$, comprises free and obstacle points sets denoted as $F, O \subseteq \mathbb{R}^2$, respectively. Initially, the algorithm identifies the contours of the obstacle shapes in $O$, resulting in a set of vertices $O_v \subseteq \mathbb{R}^2$ (line <ref>). This set contains the minimum number of vertices to represent the obstacle shapes without information loss. These vertices are used as basis points for calculating the Voronoi boundary within the free space $F$ (line <ref>). This boundary, separating Voronoi cells that cover $F$, is structured as an undirected graph with vertices $V_0 \subseteq \mathbb{R}^2$ and edges $E_0 \subseteq V_0 \times V_0$. The algorithm then overlays the Voronoi boundary onto a grid map that discretizes the free space $F$ at a resolution $\epsilon$. Each grid cell $c_i$ has an area of $\epsilon^2$ and is centered at coordinates $(c_i^x, c_i^y)$. A partial grid $\mathcal{G}_0$ is formed by selecting free space grid cells that contain at least one point from $O_v$ (line <ref>, also illustrated in Figure <ref>). Subsequently, $\mathcal{G}_0$ undergoes a heuristic filtration to eliminate spurious cells, specifically those with a degree (number of adjacent cells, assuming 8–connectivity) of $1$ or less (lines <ref>–<ref>), targeting isolated or excessively narrow grid branches. The final step involves a skeletonization process <cit.> to further simplify the grid structure (line <ref>). This involves converting $\mathcal{G}_0$ into a bitmap, applying the skeletonization algorithm, and then reconstructing a final grid $\mathcal{G}$, which effectively represents the navigation graph. An example of the obtained result is shown in Figure <ref>.

Comment/*  */

Compute navigation graph
$\mathcal{M}=(F,O)$, the 2D map of the environment
$\mathcal{G}$, the navigation graph
$O_v \gets findContours(O)$
$(V_0, E_0) \gets VoronoiBoundary(F, O_v)$
$\mathcal{G}_0 \gets Grid_{\epsilon}(F,V_0, E_0)$
$filter \gets false$
$c \in \mathcal{G}_0$
$degree(c) \le 1$
$\mathcal{G}_0 \gets \mathcal{G}_0 \setminus c$ *[r]Filter spurious cell
$filter \gets true$

$\mathcal{G} \gets Skeletonize(\mathcal{G}_0)$ *[r]Apply skeletonization

In the third phase, the navigation graph $\mathcal{G}$ is utilized to determine the poses for data acquisition, a process detailed in Algorithm <ref>. A perception pose is defined by the tuple $(x, y, h, \theta)$, where $x,y$ are the 2D coordinates on the map, corresponding to the center of a cell in $\mathcal{G}$. From this location, the robot acquires an image at height $h$ and orientation $\theta$. Essentially, the algorithm performs a depth–first search on $\mathcal{G}$, generating a cluster of poses each time a distance $D$ is covered on the grid. This is achieved using a stack $S$ and a set of explored cells, denoted as $EXP$. The functions $d(\cdot, \cdot)$ and $\mathcal{N}(\cdot)$, applied over $\mathcal{G}$, compute the distance between cell pairs and identify the set of adjacent cells for a given cell, respectively (assuming again 8–connectivity). The exploration initiates from a randomly selected cell (line <ref>), and whenever a distance of at least $D$ is covered (line <ref>), $16$ poses centered on the current cell $c$ are added to the set $P$. These poses are generated by iterating over two height values ($h_{high}$ and $h_{low}$) and $8$ different orientations ranging from $0$ to $2\pi$ in $\frac{\pi}{4}$ increments (lines <ref>–<ref>). An example of the set of 2D locations obtained over the navigation graph is depicted in Figure <ref>.

Pose extraction
$\mathcal{G}$, the navigation graph; $D$, a distance threshold
$P$, the set of poses
$S$, $EXP$, $P$ $\leftarrow \emptyset$, $d \gets 0$ *[r]Initialization
$cur \gets randomCell(\mathcal{G})$
$S$ not empty
$c \gets S.pop()$
$EXP \gets EXP \cup \{c\}$
$d \gets d + dist(cur, c)$
$cur \gets c$
$d \ge D$
$h \in \{h_{high}, h_{low}\}$
$i \in \{0, 1, \ldots, 7\}$
$P \gets P \cup (c^x, c^y, h, \frac{\pi i}{4})$ *[r]Add perception pose

$d \gets 0$

$c' \in \mathcal{N}(c) \setminus EXP$

§.§ Schema GD

* The first step of the methodology we propose is to create a GD.

* What is a $GD$ and why do we need it? $GD$ is the natural start, it can solve the RV problem at a baseline level (entry barrier), but cannot obtain high performance in difficult instances. This baseline performance generalizes across different environments in the service robots realm.

* Despite the availability of a large number of deep detectors, creating a $GD$ for service robots is something that has not been comprehensively studied in the literature.

* What are the challenges?

* First, we cannot simply deploy off-the-shelf detectors (like YOLO, DETR, FAST R-CNN). This is due to two reasons. (i) They might not know our objects of interest. (ii) These models are trained on images that are agnostic about the viewpoint of acquisition. In other words, such images do not represent the typical robot's POV. These limitations have important consequences on the performance: if we fine–tune the standard models with images of the object of interest that do not represent the robot's POV, the performance is low (we show this empirically). Key point: (i) is a well–known problem in other classical-vision domains. To solve it, the standard approach is to fine–tune a pre–trained model with the examples of the new object. The challenge is that in our domain (service robots) solving (i) has high cost due to (ii).

* Robot POV data are costly to gather in the real world: they require setting up the robot, preparing the environment, and navigating task-representative trajectories. (Here we can mention our experience in assistive robotics.)

* We investigate if simulation can come to help, we do this by exposing a trade–off between the cost of acquisition (bad in reality, good in simulation) and the performance of the trained detector (good in reality, bad in simulation). After a number of trials, we provide a technique that lies in the middle.

* The technique we propose.

* Build a simulation framework based on Gibson <cit.>, Matterport <cit.>.

* Devise an algorithm to generate the acquisition of poses in the simulated environments (describe the steps, the parameters, and the properties that the algorithm tries to guarantee)

* Dataset preparation and re-training of the standard model.

§.§ Training a Qualified Detector for a Target Environment

During the deployment phase, the robot is set up for long–term operation in a specific target environment, denoted as $e$. A critical requirement for autonomous navigation is obtaining an on–site map. This process often involves a technician who either directly operates the robot or assists it in exploring the environment to acquire a map for later use. (We experienced this setup during an extensive experimental campaign conducted in the scope of an assistive robotics study where service robots have been installed in several private apartments <cit.>. Beyond this, we deem that the setup is common and highly representative to a very large number of on–the–field installations.) In this exploration phase, the robot has the opportunity to collect additional data, particularly images of the environment captured with its onboard RGB camera. A selected portion of these images can be labeled with doors and utilized to fine–tune the general detector developed in Section <ref>, tailoring it specifically to environment $e$. We refer to the adapted version of this detector as the qualified detector for environment $e$ and we denote it as $QD_e$. A general overview of the proposed methodology is illustrated in Figure <ref>.

A general overview of the qualification procedure.

An intriguing approach would be to automatically label the additional acquired data in what essentially would be an instance of an unsupervised domain adaptation problem. One method is to use pseudo–labels, generated by applying our general detector to the new samples, a technique common in semi–supervised learning <cit.>. However, our preliminary experiments showed a significant performance drop of about $20\%$ with this method compared to results with the general detector. This decline can be attributed to the inherent inaccuracy of pseudo–labels, as also observed in recent studies <cit.>. While pseudo–labels may improve performance in tasks where precise labels are less critical (such as semantic segmentation <cit.>), their lack of accuracy makes them unsuitable for object detection tasks, such as the one we consider. Indeed, re–training with missed or hallucinated bounding boxes produces a drift in the model in which errors keep getting reinforced. Exploring more advanced techniques for unsupervised domain adaptation (as discussed in <cit.>) is beyond the scope of this paper, where our aim is to empirically assess the trade–offs in enhancing a general detector. Consequently, we opt for manual labeling, which can be conveniently done during the robot's installation phase. This approach is widely accepted in robotics, e.g., see the work presented in <cit.>, where manual annotations have been used to fine–tune an object detector for long–term localization tasks.

To facilitate this process, we have developed and released a ROS–integrated data annotation tool[Available at <https://github.com/aislabunimi/ros-data-annotator>]. This tool allows transferring robot perceptions from a ROS bag into a database. It then samples these perceptions at a given frequency and presents them to a technician, providing an interface for easy bounding box annotation. To enhance efficiency, bounding boxes from one image are retained in subsequent images, leveraging the robot’s slow movement to reduce labeling workload and reuse prior annotations.

With our experimental campaign, we prove the benefits that the qualification procedure brings to the robot's performance, studying also the trade–off between the effort between labeling costs and the model performance gain. We empirically show that a relatively limited effort is sufficient to obtain remarkably better results in object detection. In addition, we show that this procedure is more effective when applied to a GD trained with data from the robot's point of view.

§.§ Qualified detector schema

* The second step of our method exploits the deployment scenario typical of service robots.

* Consider this reference scenario: an assistive robot is deployed in a user's apartment; installation day happens, the robot is brought there and set up, a scan of the environment is manually performed by a human operator, and a map is created. The robot is left with the user for several weeks, its tasks involve autonomous navigation in the apartment. This is not a hypothetical scenario, it is the on–the–field setting we experienced in prior work in assistive robotics <cit.>. We deem that this scenario is highly representative of service robots at large, beyond the AAL domain we described.

* Just like a map is created for navigation, the environmental data acquired during the installation phase can be leveraged to specialize the detector for that particular environment. This will reduce the generalization capabilities, a cost that does not affect the scenario since the robot will operate there for a long time.

* Our method is simple: acquire images of the target environment, label them manually, and fine–tune the general detector obtained in the previous step.

* Clearly, the cost here comes from the manual labeling. In our setup, we tried to alleviate these by developing a custom tool for labeling that can import ROS bags (so a tool that can be used on the field and that we release). In our empirical evaluation, we show that there is a trade–off between labeling costs and model performance. This trade–off allows us to achieve good performance with limited effort.

§ EVALUATION

In this section[The datasets, models, and scripts of our experiments are available at <https://github.com/aislabunimi/door-detection-long-term/tree/robotic-vision>], we evaluate the performance of our door detectors by presenting an extensive experimental campaign where the full workflow of our method (Figure <ref>) is applied to a significant selection of three prominent deep–learning object detectors. In Section <ref> we describe our experimental setting by detailing our model selection (Section <ref>), the datasets used for the trials and the details of their preparation (Section <ref>), and the evaluation metrics we propose to adopt (Section <ref>). We then present and discuss the obtained results both with our general detectors (Section. <ref>) and with the qualified ones (Section <ref>). We show how the increase in performance due to our pipeline is general regardless of the object detection method used. To do so, we compare the results obtained with three popular object detection architectures and we select the configuration that better suits our target problem (Section <ref>).

§.§ Experimental Setting

§.§.§ Model Selection

Research in deep learning for object detection primarily explored three types of deep learning architectures. Initially, the focus was on two–stage detectors, which were then followed by the development of one–stage models. More recently, considerable interest has been devoted to Transformers.

Two–stage detectors (such as <cit.>) employ an architecture featuring two parts. The initial part generates proposals, namely regions likely containing objects of interest. The second part classifies and refines these proposals in a coarse–to–fine fashion. Following a more end–to–end approach, one–stage detectors (such as <cit.>), perform object recognition in a single step. They simultaneously predict both the locations and the labels of objects using predefined bounding boxes, known as anchors, which are distributed uniformly across the image. Recently, Transformer–based models (such as  <cit.>) have gained importance as a novel paradigm in object detection. These models first create a spatial feature map from the input image, which is then processed by a Transformer <cit.>. This process allows for the parallel prediction of multiple objects’ labels and locations, with the added advantage of considering inter–object relationships through the use of attention. (See <cit.> for a more comprehensive survey of these techniques.)

In our experimental campaign, we chose a representative model for each of the three main types of object detection architectures, considering both their availability and the computational requirements for training and deployment on a robotic platform. For the two–stage architecture, we selected Faster R–CNN <cit.> as implemented in the PyTorch Hub framework. This model includes a Feature Pyramid Network (FPN) backbone based on ResNet–50 <cit.>, coupled with a Region Proposal Network (RPN) <cit.> and a classifier for bounding box regression <cit.>, totaling around 41 million parameters. For one–stage detectors, we opted for the medium–sized variant of YOLOv5 <cit.>, which has approximately 20 million parameters. Both these two architectures apply a non–maximum suppression procedure to discard bounding boxes with a high overlap (for any pair of bounding boxes with an overlap of $50\%$ or more, the one with the lower confidence is removed).

As for the Transformer–based model, we selected DETR <cit.>, which integrates a ResNet–50 backbone <cit.> with a Transformer module <cit.> and a four–layer MLP, summing up to 41 million weights in total. DETR requires setting a critical hyperparameter, $N$, which defines the fixed number of bounding boxes predicted per image. We set $N$ to $10$, a value slightly higher than the maximum number of doors observed in any single image in our datasets, to ensure comprehensive detection without excessive computational burden.

§.§.§ Datasets

In our experiments, we considered a total of four datasets composed of images and their relative door–status annotations.

The first dataset, which we refer to as , is derived from the DD2 dataset <cit.> discussed in Section <ref>. This dataset includes 3000 real–world images taken from a human perspective, as provided in DD2. In these images, doors are marked as open, semi–closed, or closed. For the purposes of our experiments, we re–labeled the dataset to include ground truth data for complex examples that were not initially annotated (similar to those shown in Figure <ref>). Additionally, considering the operational constraints of a robot, which may not be able to navigate through partially opened doors, we categorized the doors marked as semi–closed as closed.

The second dataset, which we refer to as , was generated using the iGibson simulator <cit.>. iGibson provides 15 artificial environments, designed to mirror the structural features of real indoor scenes. To capture data from the perspectives of robots, we implemented a pose extraction mechanism akin to the one outlined in Section <ref>. (The details of this method are not elaborated here, as our later results will show its limited performance.) By integrating this pose extraction process with the ability to control door configurations within the simulation, we successfully generated a large batch of around 35000 instances that were automatically annotated using the semantic data provided by the simulator. (Some examples are reported in Figure <ref>.)

Our third dataset, referred to as , was created using the Gibson–based simulation framework proposed in Section <ref>. This dataset comprises images generated from perception poses derived using Algorithms <ref> and <ref>. For this dataset, we set the distance parameter $D$ to $\SI{1}{\meter}$, and used two different robot embodiments with heights of $h_{low} = \SI{0.1}{\meter}$ and $h_{high} = \SI{0.7}{\meter}$ across 10 diverse Matterport3D environments, including small apartments and large villas with multiple floors and varied furniture styles. In processing these images, we utilized the semantic frames provided by Matterport3D, where each pixel is classified into an object category. We filtered out images without doors (i.e., where pixels labeled as “door” constituted less than $2.5\%$ of the total image). Subsequently, we automatically generated bounding box proposals around door instances. This pre–processing step significantly simplified the final phase of manually verifying and completing the annotations, which was carried out by human operators. The resulting  dataset comprises 5457 images all captured from the perspective of a mobile robot. See some examples in Figure <ref>, where also the enhanced photorealism with respect to  can be appreciated.

Example of annotated images obtained from simulations.

The final dataset in our study, named , is collected from a real deployment scenario of a service robot. This dataset consists of images acquired by a Giraff–X platform <cit.>, as depicted in Figure <ref>, during the exploration in 4 distinct indoor settings. These environments, as depicted in Figure <ref>, include a variety of settings. There is a university facility characterized by open spaces and classrooms (referred to as ), the floors of a department consisting of narrow corridors and regularly arranged offices (denoted as ), a research facility with laboratories (labeled as ), and a private apartment (identified as ). (In Figure <ref>–<ref> the floor plans of  and  are shown.)
Data collection was performed using an Orbbec Astra RGB–D camera (the lower camera attached to the robot in Figure <ref>), capturing 320x240 RGB images at a rate of 1 fps. The images were then manually annotated.

Real environments considered in this work.

These datasets collectively offer a comprehensive overview of the trade–offs involved in training a door detector.  showcases what is typically available in literature but comes with significant drawbacks: the extensive effort needed for labeling and the lack in representing a robot's perception model.  and , on the other hand, are products of efforts to address this limitation through the use of simulation frameworks.  maximizes the advantages of simulated data collection: images are acquired and annotated in large batches, automatically, and from a robot–centric perspective. However, this comes with a critical downside given by the lack of photorealism. Our results will demonstrate that  achieves a more favorable compromise, allowing for batch data collection from the robot’s viewpoint with reasonable effort, while ensuring a decent degree of photorealism and easing the manual annotation process. , representing the ideal data set, offers the most authentic data but its high acquisition costs make it impractical for large–scale training. Table <ref> summarizes these points, giving a broad comparison of the key characteristics of each dataset together with the number of samples exploited in this work.

§.§.§ Performance Metrics

Our first performance metric is the mean Average Precision score (mAP), which averages the AP across all object categories (in our case,  and  doors). The AP, as defined in the Pascal VOC challenge <cit.>, is calculated as the area under the precision/recall curve that is interpolated at 11 evenly spaced recall levels. In our evaluation, we refine this approach by introducing additional interpolation points at each recall level where the precision reaches a local maximum. This enhancement provides a more detailed approximation of the precision/recall curve, resulting in a more accurate assessment.

While the mAP is a widely accepted metric for object detection tasks, it has notable limitations in our robotic context. On one hand, certain errors disproportionately affect the AP relative to their actual impact on the robot's functionality. For instance, as illustrated in Figure <ref>, minor inaccuracies in bounding box localization may have minimal effect on a service robot that is often primarily concerned with recognizing a door's traversability status rather than its precise localization. Furthermore, the AP treats multiple bounding boxes for the same door, as seen in Figure <ref>, as false positives. However, a robot can resolve such ambiguities using additional information like its estimated pose and the map of the environment. On the other hand, the AP may not adequately reflect the severity of errors in identifying a door's traversability status if the bounding box is otherwise accurate. Once again, these errors are treated as false positives but, in our scenario, incorrectly classifying a  door as  (or vice versa) can significantly impact the robot's efficiency, especially when these classifications inform the robot’s decisions. An example of this type of error is depicted in Figure <ref>.

Errors made by a detector on  (Figure <ref>). In (a) the foreground green bounding box is only slightly misaligned compared to its ground truth (in dashed blue). The error affects the AP but not the robot's typical task. Similarly, in (b) the two large green bounding boxes at the corridor's end correctly refer to the same open door; on the right, the closed door is a false positive. While the two errors affect the mAP similarly, the former is of little interest in the robotic domain, but the latter is critical for a navigating robot.

Given these shortcomings, we suggest incorporating additional metrics better suited to the specific needs of the robotic application domain where door detection is crucial. These proposed metrics are based on the premise that a service robot will invariably employ a method to sift through and select the most reliable predictions from a door detector. This process typically involves prioritizing high–confidence predictions and aggregating multiple bounding boxes that are localized in the same image region. The following definitions aim to encapsulate this approach, as well as enable the assessment of the asymmetrical nature of detection errors as previously discussed.

Consider the $i$–th image $x^i \in X$ and call $Y^i$ and $\hat{Y}^i$ the set of doors present in that image and the set of predictions computed by the detector, respectively. Given a predicted bounding box $\hat{y}$, we denote as $c(\hat{y})$ the confidence associated to it by the detector and we select those predictions whose confidence is above a threshold $\rho_c$, that is $\hat{Y}_c^i=\{\hat{y} \in \hat{Y}^i \mid c(\hat{y}) \ge \rho_c\}$. Given two bounding boxes $y_1$ and $y_2$, we denote as $a_{I}(y_1,y_2)$ and $a_{U}(y_1,y_2)$ the area of their intersection and union, respectively. We compute the set of Background False Detections ($BFD$) as the confident predictions that cannot be assigned to any real door based on a threshold $\rho_a$ on their maximum Intersection Over Union area (IOU). Formally,

$$BFD^i=\Bigg\{\hat{y} \in \hat{Y}_c^i \Big| \max_{y \in Y^i}\frac{a_{I}(\hat{y},y)}{a_{U}(\hat{y},y)} < \rho_a\Bigg\}.$$

BFDs occur when a robot mistakenly identifies a door in locations where none exists, such as on a wall or a closet. As previously discussed, this type of error relates to the mislocalization of doors. In principle, a robot might correct such errors using information from its navigation stack. For example, the robot could infer from its map that a door cannot exist in a place designated as a wall. Therefore, provided these errors are not excessively frequent, they are generally deemed acceptable within typical robotic scenarios.

Confident predictions that, instead, are well localized and have an above–threshold IOU for at least one door in the image are contained in a set called $\hat{Y}_{c,a}^i = \hat{Y}_c^i \setminus BFD^i$. This allows us to define, for each ground truth door $y$, the set of predictions that are confident and whose area is maximally matched with it, formally

$$B(y) = \Bigg\{\hat{y} \in \hat{Y}_{c,a}^i \Big| \arg\max_{y \in Y^i}\frac{a_{I}(\hat{y},y)}{a_{U}(\hat{y},y)}=y\Bigg\}.$$

(Notice that, provided that ties are broken, the same prediction can never be matched to more than one door.)

Finally, we define $\hat{y}^*=\arg\max_{\hat{l} \in B(y)}c(\hat{y})$ as the most confident prediction for door $y$, and it is this prediction we focus on, discarding any other predictions for the same door. If $\hat{y}^*$ correctly predicts the traversability of door $y$, it is included in the set of true positives ($TP^i$). Conversely, if $\hat{y}^*$ incorrectly predicts the traversability status, it is assigned to the set of false positives ($FP^i$). A false positive substantially differs from a BFD, as an FP is potentially more consequential. An FP can lead the robot to incorrectly assess a critical aspect of the environment's topology, such as mistaking a closed door for an open passage, which could significantly impact its decisions (notice how, in this example, the environment's map cannot be exploited to fix the error).
In our evaluation, we apply the aforementioned method across all images, defining
$$TP_{\%} = \frac{\sum_i |TP^i|}{\overline{Y}},~ FP_{\%} = \frac{\sum_i |FP^i|}{\overline{Y}}, ~\text{and}~ BFD_{\%} = \frac{\sum_i |BFD^i|}{\overline{Y}},$$

where $\overline{Y} = \sum_i |Y^i|$. We call these Operational Performance Indicators (OPI), they represent the rates of true positives, false positives, and BFDs, respectively. In our experiments, the confidence threshold $\rho_{c}$ is set to $75\%$, and the IOU threshold $\rho_{a}$ is set to $50\%$.

§.§ Evaluation of General Detectors

In this section, we evaluate our pipeline for synthesizing a GD. For each of the three chosen models, we train four general detectors on each of the following datasets: , , , and . The last dataset, , is obtained by joining the examples of  and .

Following a preliminary experimental campaign that explored various batch sizes ({1, 2, 4, 16, 32}) and epoch numbers ({20, 40, 60}), we established our training parameters. For Faster R–CNN and YOLOv5, we trained for 60 epochs with a batch size of 4 and using the SGD optimizer, with a learning rate of $10^{-3}$. On the other hand, DETR was trained for 60 epochs with a batch size of 1, utilizing the AdamW algorithm <cit.>. (The learning rate was set to $10^{-6}$ for the CNN backbone and $10^{-5}$ for the remaining layers as in <cit.>.)

We present the performance metrics by testing on real–world samples from . For each one of the 4 real environments ${e_1, e_2, e_3, e_4}$ of Figure <ref>, a test set  is compiled by randomly choosing $25\%$ of the images acquired by the robot in it. The results are detailed in Table <ref> and visually summarized in Figures <ref> and <ref>.

mAP of the general detectors trained with the 4 datasets in real environments.

Operational performance indicators of the GDs trained with the 4 datasets.

First, notice how it is evident that the general detectors trained on  exhibit very poor performance, as indicated by the blue bars in the figures. To elaborate, the YOLOv5–based GD correctly identified only one door instance (in $e_4$ – ). Meanwhile, its counterparts, DETR and Faster R–CNN, incur a high number of errors in terms of $FP_\%$ and $BFD_\%$ (as illustrated in Figure <ref>), which outweighs their very limited number of correct detections. These unsurprising outcomes confirm the intuition that training with simulations, even those designed to replicate real environmental features, is ineffective if they lack photorealism. This conclusion is further supported by observing the significant performance improvements achieved when transitioning from training with  to  (among our training datasets, the one that maximizes photorealism).

An interesting and perhaps counter–intuitive observation emerges when comparing the training results of  (real–world images) with  (our simulation framework outlined in Section <ref>). Common intuition suggests that a detector trained on real–world data should outperform one trained on a simulation, even if photorealistic. However, as shown by the mAP scores in Figure <ref>  and the $TP_{\%}$ in Figure <ref>, we see that two out of the three detectors, specifically those based on DETR and YOLOv5, actually demonstrate improved performance when trained on  rather than . This result indicates that while photorealism, a characteristic highly present in , is important, it is not the unique key feature for creating effective general detectors for robots. It appears that the slightly compromised visual quality in  might be effectively balanced by a closer alignment with a robot's perception model, thereby reducing, to some extent, the sim–to–real gap. This also suggests that in real robot deployments, the shift in data distribution might be more significantly influenced by the data acquisition process rather than by the characteristics of the input space.

This trend does not hold for the detector based on Faster R–CNN, which shows better results with . Upon closer examination, this can be attributed to the Region Proposal Network, which, by localizing and classifying bounding boxes based on features extracted from the Pyramid Backbone, is more sensitive to the photorealistic quality of images. To support this observation, we consider the performance of Faster R–CNN trained on , a dataset that combines 's high photorealism with 's representation of the robot's viewpoint. As indicated by the red bars in Figure <ref>, Faster R–CNN's performance improves, while DETR and YOLOv5 are only slightly impacted by the absence of real–world data. The $TP_{\%}$ in Figure <ref> shows that the correct door status detections with  slightly surpass those with . In some cases, our simulated data even yield better results, as demonstrated by DETR's performance in environments $e_2$ and $e_3$. However, it's noteworthy that mixing training data often leads to an increase in erroneous detections, as evidenced by the $FP_{\%}$ and $BFD_{\%}$ indicators in Figure <ref>.

These results prove the effectiveness of our simulation framework, which strikes a balance between photorealism and alignment with the robot's perception model. This approach is hence both viable and efficient for building general door detectors, reducing training costs while still achieving an acceptable performance level. The general detectors we developed are capable of accurately recognizing doors across diverse real–world environments, demonstrating a fair level of generalization. However, this strength is mainly evident in straightforward door instances, and less so in more complex ones involving occluded views, multiple nested doors, or combinations of these. Figure <ref> showcases some representative examples where our GDs excel. To bridge the gap in identifying such difficult cases, it is essential to qualify the general detectors for the target environment where they are set to operate.

§.§ Evaluation of Qualified Detectors

In this section, we assess the benefits to general detectors introduced by the qualification procedure. In particular, we qualify for the environments of  the GDs trained with , , and , discarding the one based on  due to its unsatisfactory performance in the real world (see previous section). To ease presentation, we say that a QD is based on a dataset $\mathcal{D}_\texttt{x}$ when it is obtained from a GD trained on such a dataset. Considering each real environment $e$, we perform a series of qualification rounds of the GDs with increasing amounts of data from the target environment, obtaining a set of qualified detectors we denote as $QD^{15}_{e}$, $QD^{25}_{e}$, $QD^{50}_{e}$, and $QD^{75}_{e}$. Their superscripts denote the percentage of examples from  (the real data acquired in environment $e$) used for the fine–tune and can be interpreted as an indicator of the cost to acquire and label the additional samples. We use the same training parameters as in Section <ref> reducing the epochs to 40. Each qualified detector $QD^{x}_{e}$ is tested in the corresponding environment $e$ using the previously defined test set  (random $25\%$ of images from  not used in any qualification round).

We assess the average performance over the three selected models in each real environment. The detailed results can be found in Table <ref>. For a visual representation, refer to Figure <ref> which illustrates the mAP, and Figure <ref> for the operational performance indicators.

Real–world evaluation of the qualified detectors where GDs are based on different datasets. The mAP is averaged over the three models.

Operational performance indicators (averages over the three models) with GDs trained on different datasets.

A first evident observation is that the qualification procedure boosts the performance of the general detectors for the target environment and, unsurprisingly, the performance (together with the data preparation costs) increases as more samples are included, from $QD_e^{15}$ to $QD_e^{75}$.  This can be appreciated in the mAP and $TP_{\%}$ improvements visually depicted in Figures <ref> and <ref> and by the decreasing trend (after the first qualification round) of $FP_{\%}$ and $BFD_{\%}$ in Figure <ref>.

However, the increments follow a diminishing–returns trend, with large gains in the first qualification rounds and marginal ones as more data are used. Focusing on the average mAP and $TP_{\%}$ it can be seen how the qualified detector that scores the highest performance improvements is $QD^{15}_e$, despite requiring a relatively affordable effort for data preparation. From a practical perspective, this observation suggests how just a coarse visual inspection of the target environment might be enough to obtain an environment–specific detector whose performance is significantly better than the corresponding general one. In such a case the robot's deployment time is only marginally affected. To give a concrete idea, annotating the $15\%$ of the data collected by the robot's first exploration of a new environment (approximately 80 images) required a human operator using our tool (Section <ref>) about half an hour. Another key finding is how the improvements through qualification are distributed across various types of instances encountered by the detector. Upon direct inspection, we observed how these were particularly notable in challenging instances. Figure <ref> showcases significant examples of this, illustrating how the $QD^{15}_e$ model, based on our dataset , successfully detects doors in highly challenging instances. These include scenarios with nested or partially occluded doors and even situations where the door is hidden in the background.

[origin=c]90DETR <cit.>
[origin=c]90YOLOv5 <cit.>
[origin=c]90Faster R–CNN <cit.>
$e_1$ –     $e_2$ –     $e_3$ –     $e_4$ –
Challenging doors correctly detected by $QD^{15}_e$ (GDs divided by model and trained on ).

It is important to notice that the dataset chosen to train the general detector does affect the benefits of the qualification. The trends observed in Figure <ref> indicate that QDs based on  generally demonstrate lower performance compared to those based on  and . This observation is further supported by the data presented in Figure <ref>. Although the error rates ($FP_{\%}$ and $BFD_{\%}$), are substantially similar, there is a noticeable difference in the $TP_{\%}$. Specifically, detectors based on  or  show better $TP_{\%}$ performance than those based on . Confirming the findings from the previous section, this again suggests that training on images not representing the robot's point of view, although taken from the real world, hits a performance limit. A simulated dataset from the robot perspective with an adequate level of photorealism (as ), when included in the training phase, enables the detectors to reach better performance when qualified for a target environment.

To further support the effectiveness of the method proposed in Section <ref>, we can notice that , which integrates the realism in  and the robot perception model of , does not introduce significant variations in the performance of the qualified detectors when compared with those solely based on . This can be easily seen by comparing the (substantially similar) orange and red bars of Figure <ref> that refer to  and , respectively. In addition, while $TP_{\%}$ reaches comparable performance, Table <ref> shows that  enables the qualified detectors to reduce the rate of $BFD$ with respect to .

As argued in Section <ref>, the true benefit of qualification rests on the typically long–term nature of a robot's deployment, during which the target environment is assumed to stay the same. However, this invariance does not imply the complete absence of domain shifts. While the overall environment may not change, lower–order shifts could still challenge the detector's ability to perform its task. We deem that one of the most significant of such shifts might occur at the feature level, particularly concerning the illumination conditions of the environment. Changes in lighting can significantly alter the appearance of doors in images and these variations in illumination can be widespread throughout the entire environment, rather than being localized.

To test the robustness of our approach in such situations we acquire and annotate (following the same procedure for  reported in Section <ref>) additional data from environments $e_1$ and $e_2$ during nighttime, when only artificial light is present and some rooms are dark. Then, we use these images to test our detectors trained with daylight data. The metrics' average performance are detailed in Table <ref> and visually shown in Figure <ref> (mAP) and Figure <ref> ($TP_{\%}$, $FP_{\%}$, and $BFD_{\%}$).

Real–world evaluation of the detectors during nighttime. GDs are based on different datasets and the mAP is averaged over the three models.

Operational performance indicators during nighttime (averages over the three models) with GDs trained on different

From Figure <ref> we can see how the mAP performance of the GDs reflects those shown in Figure <ref>, demonstrating that the GDs are robust to illumination changes. As reported in Section <ref>, the general detectors based on  exhibit the worst performance while those trained with our simulated dataset  perform remarkably better, especially in $e_1$ (see also the $TP_{\%}$ in Figure <ref>). The –based GDs, despite improving the average mAP as shown in Figure <ref>, increase also the performance gap between the models (see the standard deviation of the orange and red bars in Figure <ref>). More interestingly, it can be seen in Table <ref> how the improvement provided by the qualified detectors is maintained also with different light conditions. Once again, our dataset  increases the benefits of the qualification when compared to the human–POV images of  (as it can be seen by the higher and stable $TP_{\%}$ of Figure <ref>). The slight performance decrease observable comparing Table <ref> and <ref> is a direct consequence of the fine–tune, which produces QDs that slightly overfit the illumination conditions seen during the robot's deployment. Despite this, our method ensures a performance improvement to the GD when used in long–term scenarios with illumination changes, enabling the QDs to still solve challenging examples, as shown in Figure <ref>. Once again, $QD^{15}_{e}$, albeit using a few examples for fine–tuning, ensures the best performance improvement also under variable light conditions.

§.§ Model Comparison

In this section, we analyze the three selected models (DETR <cit.>, YOLOv5 <cit.>, and Faster R–CNN <cit.>) highlighting their strengths and weaknesses to provide insights for helping technicians working in Robotic Vision scenarios to choose the best one according to their requirements.

From our experience in setting up the three models for the specific task of door detection, DETR turned out to be the easiest to adapt. Instead of learning how to activate a set of predefined anchor boxes according to the image features, DETR directly regresses the coordinates of the bounding boxes by construction. Moreover, it does not require a non–maximum suppression step to discard multiple detections of the same object. This is achieved by its loss function that matches the (limited) inferred bounding boxes to a single target. On the contrary, the detection performance of our detectors based on YOLOv5 and Faster R–CNN are strongly influenced by the hyperparameters setting: the anchor dimension and scale should be compliant with the object shape while the non–maximum suppression procedure can delete correct bounding boxes (such as those of nested doors). In other words, while the competitors need to encode task–specific prior knowledge in the model, DETR offers the possibility to share the same configuration among different applications (such as <cit.>).

After these considerations, we compare the performance of the detectors (based on our dataset ) to study how they work, on average, in the four real environments of . Table <ref> reports in detail the metrics results, depicted also in Figure <ref> (mAP) and Figure <ref> ($TP_{\%}$, $FP_{\%}$, and $BFD_{\%}$).

Real–world mAP with our three selected models (GD is based on ).

Real–world performance of the operational performance indicators with our three selected models (GD is based on ).

By observing the mAP performance shown in Figure <ref> we can see that the best GD is based on DETR that, not requiring task–oriented knowledge, better addresses the sim–to–real gap (between our dataset  and the real acquisitions of ). While YOLOv5 lies in the middle, Faster R–CNN reaches the worst performance when trained in simulation (with our dataset ) and tested in the real world. As discussed in Section <ref>, Faster R–CNN, being a two–stage detector, tends to overfit the distribution of the training data acquired in simulation. This outcome is reverted by the qualification procedure. When fine–tuned for a target environment, Faster R–CNN reaches the best mAP results while DETR the worst (see the green and red bars in Figure <ref>). This is caused by the Transformer that, although popular, requires huge amounts of data (hundreds of millions) to effectively learn the architecturally inherent biases of the CNN–based models (such as the translation equivariance and the locality principle <cit.>). Moreover, by carefully examining the extended metric's results, we can see that the $BFD_{\%}$ of YOLOv5 is considerably lower than the other detectors (both the GD and its qualified versions). In a robotic domain where detections are translated into actions, this fact is extremely important because drastically reduces robot failures. Figure <ref> shows how the additional indicators of $QD_e^{15}$ vary according to the confidence threshold. The results demonstrate that our choice of $\rho_c = 75\%$ is a good compromise between the correct ($TP_{\%}$) and the wrong ($FP_{\%}$, $BFD_{\%}$) predictions.

Operational performance indicators w.r.t. confidence for $QD_e^{15}$ (averages over the real environments, GD based on ).

Despite it is well–known from the literature that the two–stage detectors (like Faster R–CNN) are generally better than single–stage ones <cit.>, YOLOv5 is more suitable for edge devices typically mounted in service robots. First, it is compatible with the NVIDIA Jetson TX2 mounted on our Giraff–X robotic platform <cit.> (depicted in Figure <ref>) where it can run at 20 fps with the TensorRT framework. Since the other models are not compatible with the NVIDIA SDK for our specific hardware, we deploy all the architectures relying on ONNXRuntime, a less efficient inference framework able to run YOLOv5, DETR, and Faster R–CNN at 14, 6, and 0.7 fps, respectively.  In our experimental setting, YOLOv5 represents the best compromise between performance and inference time, thus appearing as the most convenient model for door detection with service robots.

§ EXPERIMENTAL SETTINGS SCHEMA

* We deploy our method in the passage-detection use case by applying it to three different off-the-shelf detectors: DETR <cit.>, YOLO (version 5) <cit.>, FAST R-CNN <cit.>

* We have four datasets: Gibson ($10$ environments), iGibson ($15$ environments), DD2: Deep doors 2 <cit.>, REAL: Real-world acquisitions ($4$ environments, different lighting conditions).

* We adopt the following performance metrics: AP, the extended metric, and the “door recognition" metric. The first two metrics are parametrized by a threshold over the IoU area.

We provide $3$ experiments.

* Effectiveness of simulation in synthesizing a GD. We train a GD using 4 different configurations (defining the training set): Gibson, DD2, Gibson+DD2, iGibson. We report the perf.ormance of these GDs in the real world.

* Feasibility and effectiveness of specialization with manual labeling to QD. We perform the second step of our methodology by considering different real environments and different lighting conditions. We specialize the GD with increasing labeling efforts. We report the performance of these GDs in the real world.

* Model comparison. We analyze deeply the three detectors in order to highlight their strengths and weaknesses.

* We compare the three detectors in terms of mAP over the 4 real environments (what is the best one?)

* We compare the three detectors with the extended metric

* We deploy the three detectors in a edge device (a jetson nano TX2) to measure the inference time

* Model calibration: We assess how the performance varies with respect to the minimum confidence required to accept a prediction in each detector. We show how varying the confidence threshold reflects on the models' performance.

For each experiment, we derive a set of findings.

* (a) Training in simulation with iGibson achieves the worst performance, as the assessment of the sim-to-real gap compared to Gibson shows. This framework is powerful in terms of automatization (very low costs of generating a rich dataset), but lacks photorealism and cannot be used. There is a need to address the sim-to-real gap, a future direction. (b) Training with real images from DD2 performs better than iGibson, confirming the fact that photorealism is important. (c) Training in simulation with Gibson performs better than iGibson and DD2. Photorealism is not the primary feature: robot POV is fundamental. With Gibson we have a fair photorealism (although very limited) while properly representing the robot's POV. (d) Training with DD2 and Gibson achieved the best performance. Bottom line: By exposing a trade-off between data acquisition costs and performance we discovered that, if properly engineered, a simulation framework is an effective and affordable approach to training a GD.

* (a) Unsurprisingly, model specialization brings improvements in the detectors' performance. (b) These improvements target the key weaknesses of the GD: environment-peculiar difficult recognition instances. (c) The improvement trends exhibit a diminishing-return profile, suggesting that the labeling effort need not be arbitrarily high, small efforts tend to bring the largest marginal gains. Bottom line: Specialization is effective for obtaining detectors that are not only better but better at key difficult instances from the specific environment. Despite requiring manual labeling, these efforts are not arbitrarily large. The robot just needs a “ quick glimpse” of its environment

Extended findings with AP:

* The performance increases by qualifying with more data (with a diminishing-returns trend) AND The qualification procedure is dataset-independent (works with all the GDs)

* Qualifying a general detector trained with a robot's POV dataset (Gibson or Gibson + DD2) ensures better and more stable performance with respect to using a GD trained only with real data (DD2).

Extended findings with extended metric:

* The TP follow a diminishing return trend

* The FP drop until reaching 0

* The BFD increases in the first qualification but decreases with more data

* (a) Comparing the detectors' performance with the mAP, we can see that the best GD is the one based on Faster R–CNN, but DETR is more stable over the environments (lower std dev). Once qualified, Faster R–CNN still wins against the others, with higher mean and lower std. In the meanwhile, DETR is the worst while YOLO is in the middle. (b) Carefully examining the extended metric's results, the TP rate follows the previous outcome (FASTER > YOLO > DETR). Despite this, YOLO appears to be the best detector to be deployed in a mobile robot because the BFD rate is considerably lower than the other detectors. In this way, the robot can put more trust in the model predictions, which produce less noisy detections by construction. (c) YOLOv5, DETR, and Faster R–CNN reach 14, 6, and 0.7 FPS in the Jeston TX2.

* TO BE ADDED

Other findings These findings can be added in the third experimental (model comparison).

* Insights on how to choose an off-the-shelf detector. (a) Considering the need to encode prior knowledge in the model, DETR is the easiest to set up since it requires only setting $N$. FAST R-CNN and YOLO are required to set up anchors (although there are automatic tools to do that, which require setup as well). (b) Considering performance all the models exhibited the same good performance overall, and all of them represent viable solutions; in detail, DETR achieved the worst performance: transformers, although popular, require huge amounts of data; FAST R-CNN had the best performance: as is well-known from the literature two-stage detectors (like FAST R-CNN) are generally better (but less efficient) than single-stage ones (YOLO). (c) When considering deployment and computational costs YOLO runs in half the time required to FAST R-CNN on the same reference hardware. Moreover, YOLO can be easily installed on a Jetson (tensor RT compatibility) where it can run at 20 FPS performance (with a real ROS-based setup). The other models are not compatible with the inference framework provided by NVIDIA (TensorRT SDK for our specific hardware). Bottom line: in our experimental setting, YOLO seems to better balance all three selection criteria.

OTHER CONSIDERATION TO BE INCLUDED SOMEWHERE ABOVE

* Manual experiments: GD bad doors $p$: how many times $p$ was in the image. How many times has been recognized, how many times has not been recognized, how many times has been recognized with the wrong status. (Not seen doors better not to include)

* In COCO the average number of examples per class (not including people) is $3200$, which is roughly the same number of examples in the DD2 dataset. So DD2 is not too small.

2c|IGibson Dataset     2c|DD2 <cit.>     2c|Gibson      2cGibson + DD2 <cit.>

Detector     Label     AP     $\sigma$      AP     $\sigma$       AP     $\sigma$     AP     $\sigma$

1c|2*DETR <cit.>     Closed     -     -     5     3    13     10    21     16
1c|     Open     -     -     19     4    31     11    27     8
1c|2*YOLOv5 <cit.>     Closed     -     -     3     1    14     16    18     17
1c|     Open     -     -     2     2    17     11    18     11
1c|2*Faster R–CNN <cit.>     Closed     -     -     6     3    9     11    24     18
1c|     Open     -     -     28     14    15     11    26     17
1c|2*Average     Closed     -     -     5     2    12     11    21     15
1c|     Open     -     -     16     13    21     12    24     12

Average AP in the real–world environments when the dataset $\mathcal{D}$ described in this work is used to train the GD. IOU = 0.5 CONF = 0.75

Average of the mAP over the three GD (DD2, Gibson, Gibson + DD2)

§ ON THE FIELD EVALUATION

The goal of equipping an autonomous mobile robot with an object detection method is to allow the robot to have an updated representation of its working environment that can be used to plan and execute the tasks assigned to the robot. In the scenario we consider, the ability to detect doors can be used by the robot for the downstream task of reconstructing the current topology of the environment, that is, to infer which are the sub–areas that are currently accessible by the robot and those that are not. In the experiments presented in this section, we consider the environment's topology to be a graph, wherein the nodes represent rooms and the edges correspond to open paths connecting them. This knowledge can be used by the robot to plan its activities <cit.> considering the constraint that only a subset of the sub–areas are accessible at the current time.

We evaluate how the proposed door detector can be used to obtain such a knowledge. As discussed in Section <ref>, we assume that the robot can rely on a 2D map acquired during its setup, but we consider a situation where the current topology of the environment has changed with respect to the one encoded in such a map, since some doors might be closed at the moment (for obvious reasons, when the 2D map is acquired all the doors are left opened). The task that the robot must face is to infer the current topology of the free space it can cover, assuming that door statuses do not change during the execution of this task. To carry it out, we consider the setting exemplified in Figure <ref>: the robot follows a trajectory spanning multiple rooms (the trajectory can be either functional to this topology–inference task or to another higher–level task the robot is performing). While doing so, it can observe, on purpose or incidentally, the status ( or ) of multiple doors. While some doors are perceived from a frontal view, others will likely be observed only from a side angle as the robot moves in a different direction. We assume that the robot has full knowledge of all the locations of doors on the map $D = \{d_1, d_2, \ldots d_n \}$. Furthermore, we assume to have a method that,  given the image $x\in X$ where a door $\hat{y} \in \hat{Y}$ has been identified by a door detector, along with the pose from which the image was acquired, can determine the specific door instance $d \in D$ being observed at the moment. Note that multiple doors can be observed within the same image. The result of this step is that each $\hat{y} \in \hat{Y}$ that is not a $BFD$ is associated to a door instance $d$. The robot thus counts, along the whole trajectory, how many times each door $d \in D$ has been identified either as  or , and infers its status as the one of majority label. This information is used to infer the current topology, which, for evaluation, we compare with the one obtained by repeating the process using true detections instead of predictions.

We had the robot following two trajectories in a real–world experiment with the same setup described in Section <ref>. The first trajectory is performed in $e_1$ –  during nighttime. The second trajectory is performed in $e_2$ –  with daylight. We compare the performance in inferring the topology with $GD$ against $QD^{15}_e$ (both based on YOLOv5 and ).
In both cases, the $QD_e^{15}$ is trained with data collected at mapping time, with daylight.

The floor plans and the topologies of $e_1$ –  and $e_2$ –  inferred with this framework are shown in Figure <ref> and <ref>, respectively.  We indicate with  () a door correctly recognized as  () and with  () a door that has been wrongfully recognized as  () when its current status is  (). We indicate with  the event where the number of detections $\hat{y}$ where a door is labeled as  is equal to those where it is perceived as , and thus the robot is undecided. If a door has been observed in multiple images, but the door detector was always unable to detect any door due to false negatives, we label such door with a . We indicate with  the location of a room that the robot can access, and we highlight the location of the main entrance/exits of the environment. We indicated with a solid blue line a path across two different rooms that is open for the robot, and with a dashed line a path between two rooms that has been wrongfully estimated, following the same color schema as above: a red (green) path when a passage is estimated to be  () when actually it is  ().

To understand the impact of having a qualified detector in estimating the topology of an environment, we report the topology obtained with $GD$ and $QD^{15}_e$ in Figure <ref>–<ref>, as well as the number of doors whose status is correctly/wrongly detected, and the total recognition accuracy $RA$, that is the percentage of doors $d$ whose status has been successfully detected during the robot run. These metrics are specific to the detection domain and are downstream OPI, following the definition of Section <ref>.

From Figure <ref>–<ref>, we can see how the qualified detector can correctly identify the topological status of the environment, albeit making minor errors. In both environments, the QD identifies correctly the status of most doors, with an $RA$ of around $90\%$ ($89,29\%$ in $e_1$, $95,23\%$ in $e_2$).

We noticed how, while both $GD$ and $QD^{15}_e$ could detect successfully the status of a door when it is observed from a frontal position by the robot, the $GD$ often fails when the robot is at one side of a door, when the door is partially occluded, or when there are challenging light conditions. In all those cases, $QD^{15}_e$ does not suffer from the same limitations.
An example of this can be seen in the doors connected to the main corridor of $e_2$, shown in Figure <ref>. While $QD^{15}_e$ can identify how most doors are closed (), $GD$ often fails to understand the status of those doors, thus identifying them as open ( or ) or failing to identify them ().

Exp.       $\uparrow$       $\downarrow$      $\downarrow$       $\downarrow$     $RA$ $\uparrow$

$GD$     20     4     1     2     71%

$QD_e^{15}$     25     1     1     1     89%

Topology of the environment for $e_1$ –  during night–time as identified using $GD$ and $QD^{15}_e$ to detect the status of each door.

Exp.       $\uparrow$       $\downarrow$      $\downarrow$       $\downarrow$     $RA$ $\uparrow$

$GD$     26     9     4     3     62%

$QD_e^{15}$     40     1     1     0     95%

Topology of the environment for $e_2$ –  during daytime as identified using $GD$ and $QD^{15}_e$ to detect the status of each door.

To better highlight this event, we provide detailed results about how many times two doors, that are highlighted with 192 and 193 in Figure <ref>, have been viewed by the robot in the two runs. Door 192 was observed in $60$ images by the robot. While $QD^{15}_e$ was able to correctly detect the status of the door $52$ times and was unable to detect the door on $8$, the $GD$ was able to detect the door only on $2$ occasions and was unable to detect it $58$ times. Figure <ref>(a–b) shows two of the 60 images, with the bounding box identified by $QD^{15}_e$. At the beginning of the run, the door was closed, and was briefly perceived in that condition when the robot was outside the room; nevertheless, the robot was able to correctly label it, as in Figure <ref>. Later, the robot enters the room and the door status is open, as in Figure <ref>. In both cases, $GD$ fails to identify any bounding box from those images.
The door 193 was observed in $40$ images as closed; $QD^{15}_e$ was able to correctly identify the status of the door $32$ times, and was unable to detect the door $8$ times.  The $GD$ is far less accurate in detecting doors in the same set of images: it correctly identified the door as  $5$ times, wrongly identified the door as  $4$ times, and did not identify any door in $31$ perceptions. Two examples of these images are shown in Figure <ref>(c–d); in Figure <ref> the door 193 is the second one on the right, while in Figure <ref> it is the one on the left side of the corridor. In both images, $QD^{15}_e$ was able to identify successfully the door status and location, while $GD$ failed to identify the presence of a door. Similar examples can be made for all of the rooms that are connected to the central corridor of Figure <ref>, that are seen by the robot from a similar perspective.

Two examples where the $QD$ identifies the door 192 in tho challenging images (a–b), and the door 193 in other two challenging images (c–d). In all four cases, the $GD$ does not identify the two doors in these images.

These results show how the general detector manages to partially reconstruct the topology of the environment due to a high number of false positives and wrong detections. On the other hand, the qualified detector obtains more stable and robust performance compared to its general version when used in its deployment environment. This demonstrates how the qualification step described in Section <ref> substantially improves (with a little cost) the performance of the detection method in a downstream task, providing more accurate domain–specific knowledge to the robot.

§ CONCLUSIONS

In this paper, we propose a methodological approach to synthesize deep learning–based object detectors for the typical deployment scenario of service robots. At first, we show how to leverage simulation to reduce the effort to acquire a photorealistic dataset from the robot's perspective, proving that our method enables multiple off–the–shelf models to work in the real world when used by service robots. Then, we show that qualifying the general detector for the target environment will drastically improve the robot's performance in its deployment environment even if the number of new examples (and the effort for their annotation) is limited. We evaluate our approach considering the use case of door detection providing extensive experiments validated in several indoor environments, providing also an on–the–field evaluation in the downstream task of reconstructing the environment topology from the detected door statuses.

Future works will investigate how to enhance the photorealism of the synthetic simulators (as iGibson <cit.>) that, despite offering a high level of automatization, cannot be used for training models working in the real world (as shown in Section <ref>). Furthermore, we will investigate new solutions for enabling the general object detector to automatically qualify itself reducing the noise of its pseudo–label.

Missing 'biblatex' package
The bibliography requires the 'biblatex' package.

journaltitleIEEE Access
titleA review on challenges of autonomous mobile robot and sensor fusion methods

booktitle2023 European Conference on Mobile Robots (ECMR)
titleEnhancing Door-Status Detection for Autonomous Mobile Robots During Environment-Specific Operational Use

booktitleProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
title3D Semantic Parsing of Large-Scale Indoor Spaces

booktitle2016 IEEE International Conference on Robotics and Automation (ICRA)
titleRoom segmentation: Survey, implementation, and analysis

journaltitleTransactions on Pattern Analysis and Machine Intelligence
titleA Computational Approach to Edge Detection

booktitleComputer Vision – ECCV 2020
titleEnd-to-End Object Detection with Transformers

booktitleProceedings 2007 IEEE International Conference on Robotics and Automation
titleUSARSim: a robot simulator for research and education

journaltitleMachine Learning with Applications
titleDeep learning in computer vision: A critical review of emerging techniques and application scenarios

titleMatterport3D: Learning from RGB-D Data in Indoor Environments

journaltitleIEEE Transactions on Cybernetics
titleTrifocal Tensor-Based Adaptive Visual Trajectory Tracking Control of Mobile Robots

booktitle2014 IEEE International Conference on Robotics and Biomimetics (ROBIO 2014)
titleDoor recognition and deep learning algorithm for visual based robot navigation

booktitle2023 European Conference on Mobile Robots (ECMR)
titleSurgical Fine-Tuning for Grape Bunch Segmentation Under Visual Domain Shifts

Taylor & Francis

journaltitleJournal of Experimental & Theoretical Artificial Intelligence
titleTarget recognition by components for mobile robot navigation

journaltitleIEEE Access
titleA review of physics simulators for robotic applications

titleTeaching a robot where doors and drawers are and how to handle them

booktitleProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
titleScannet: Richly-annotated 3d reconstructions of indoor scenes

booktitleProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)
titleDynamic DETR: End-to-End Object Detection With Dynamic Attention

journaltitleIEEE Transactions on Automation Science and Engineering
titleSoHAM: A Sound-Based Human Activity Monitoring Framework for Home Service Robots

journaltitleRobotics and Autonomous Systems
titleRiSH: A robot-integrated smart home for elderly care

booktitleProceedings of the 1st Annual Conference on Robot Learning
titleCARLA: An Open Urban Driving Simulator

booktitle2010 IEEE International Conference on Robotics and Automation
titleIndoor scene recognition through object detection

journaltitleInternational Journal of Computer Vision
titleThe Pascal Visual Object Classes (VOC) Challenge

titleYoloV3: An incremental improvement

journaltitleEuropean Journal of Operational Research
titlePlanning and control of autonomous mobile robots for intralogistics: Literature review and research agenda

journaltitleIEEE Robotics and Automation Letters
titleContinual Adaptation of Semantic Segmentation Using Complementary 2D-3D Data Representations

booktitleProceedings of the IEEE International Conference on Computer Vision - (ICCV)
titleFast R-CNN

journaltitleIEEE Transactions on Pattern Analysis and Machine Intelligence
titleA Survey on Vision Transformer

booktitleProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
titleDeep Residual Learning for Image Recognition

journaltitleOptical Engineering
titleCorner detector based on global and local curvature properties

titleService robots in the healthcare sector

journaltitleIEEE Robotics and Automation Letters
titleTarget-Dependent UNITER: A Transformer-Based Multimodal Language Comprehension Model for Domestic Service Robots

journaltitleIEEE Robotics and Automation Letters
titleMotion Planning of Mobile Manipulator for Navigation Including Door Traversal

titleYOLOv5 by Ultralytics

booktitle2019 IEEE Underwater Technology (UT)
titleOpen Source Simulator for Unmanned Underwater Vehicles using ROS and Unity3D

booktitle2022 IEEE Sensors Applications Symposium (SAS)
titleImprovement of Door Recognition Algorithm Using Lidar and RGB-D Camera for Mobile Manipulator

booktitleIEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
titleDesign and use paradigms for gazebo, an open-source multi-robot simulator

journaltitleIEEE Transactions on Robotics
titleFreMEn: Frequency Map Enhancement for Long-Term Mobile Robot Autonomy in Changing Environments

journaltitleIEEE Access
titleMobile Service Robot Path Planning using Deep Reinforcement Learning

booktitle2011 IEEE International Conference on Robotics and Automation
titleVisual recognition of a door and its knob for a humanoid robot

journaltitleIEEE Robotics and Automation Letters
titleSelf-Supervised Domain Calibration and Uncertainty Estimation for Place Recognition

titleService robots: A systematic literature review

journaltitleInternational Conference on Learning Representations
titleSurgical fine-tuning improves adaptation to distribution shifts

booktitleComputer Vision – ECCV 2014
titleMicrosoft COCO: Common Objects in Context

booktitleComputer Vision – ECCV 2016
titleSSD: Single Shot MultiBox Detector

booktitle2017 3rd International Conference on Control, Automation and Robotics (ICCAR)
titleDoor and cabinet recognition using Convolutional Neural Nets and real-time method fusion for handle detection and grasping

titleFixing Weight Decay Regularization in Adam

booktitle2019 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI)
titleThe MOVECARE Project: Home-based Monitoring of Frailty
family=Di Berardino,

journaltitleRobotics and Autonomous Systems
titleMapping beyond what you can see: Predicting the layout of rooms behind closed doors

journaltitleRobotics and Autonomous Systems
titleUser feedback and remote supervision for assisted living with mobile robots: A field study in long-term autonomy

booktitle2017 IEEE/SICE International Symposium on System Integration (SII)
titleCloud-based multimodal human-robot interaction simulator utilizing ROS and unity frameworks

journaltitleMathematics and Computers in Simulation
titleLearning to traverse doors using visual information

booktitle2020 IEEE International Conference on Robotics and Automation (ICRA)
titleLong-Term Robot Navigation in Indoor Environments Estimating Patterns in Traversability Changes

journaltitleIEEE Transactions on Pattern Analysis and Machine Intelligence
titleUnsupervised Domain Adaptation of Object Detectors: A Survey

journaltitleSN Applied Sciences
titleReal-time 2D–3D door detection and state classification on a low-power device

booktitleProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
titleYou Only Look Once: Unified, Real-Time Object Detection

journaltitleAdvances in Neural Information Processing Systems
titleFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

journaltitleInternational journal of computer vision
titleImagenet large scale visual recognition challenge

booktitle2019 21st Symposium on Virtual and Augmented Reality (SVR)
titleAnalysis and Comparison of Robotics 3D Simulators

booktitleField and Service Robotics: Results of the 11th International Conference
titleAirsim: High-fidelity visual and physical simulation for autonomous vehicles

booktitle2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
titleiGibson 1.0: A Simulation Environment for Interactive Tasks in Large Realistic Scenes

booktitle2016 IEEE International Conference on Robotics and Automation (ICRA)
titlePlace categorization and semantic mapping on a mobile robot

journaltitleThe International Journal of Robotics Research
titleThe limits and potentials of deep learning for robotics

booktitle2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
titleSoft Tissue Simulation Environment to Learn Manipulation Tasks in Autonomous Robotic Surgery

booktitle2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
titleVirtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation

booktitle2016 20th International Conference on System Theory, Control and Computing (ICSTCC)
titleSimulation environment for mobile robots testing using ROS and Gazebo

booktitleAdvances in Neural Information Processing Systems
titleAttention is all you need

booktitleComputer-Aided Architectural Design Futures (CAADFutures) 2007: Proceedings of the 12th International CAADFutures Conference
titleTopology of urban environments

booktitleProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
titleGibson env: Real-world perception for embodied agents

booktitleProceedings of the IEEE international conference on computer vision
titleSun3d: A database of big spaces reconstructed using sfm and object labels

journaltitleIEEE Transactions on Knowledge and Data Engineering
titleA Survey on Deep Semi-Supervised Learning

booktitle2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition – Workshops
titleRobust door detection in unfamiliar environments by combining edge and corner features

journaltitleAdvances in neural information processing systems
titleHow transferable are features in deep neural networks?

ACM New York, NY, USA

journaltitleCommunications of the ACM
titleA fast parallel algorithm for thinning digital patterns

titleOpend: A benchmark for language-driven door and drawer opening

booktitle2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
titleConstructing Metric-Semantic Maps Using Floor Plan Priors for Long-Term Indoor Localization

journaltitleIEEE Robotics and Automation Letters
titleLong-Term Localization Using Semantic Cues in Floor Plan Maps

journaltitleProceedings of the IEEE
titleObject Detection in 20 Years: A Survey

journaltitleIEEE Robotics and Automation Letters
titleEmbodied Active Domain Adaptation for Semantic Segmentation via Informative Path Planning