# UniTeam: Open Vocabulary Mobile Manipulation Challenge

Andrew Melnik
Bielefeld University, Germany
<EMAIL_ADDRESS>
Michael Büttner
Bielefeld University, Germany
<EMAIL_ADDRESS>
Leon Harz
Bielefeld University, Germany
Lyon Brown
Bielefeld University, Germany
Gora Chand Nandi
IIIT-A, India
Arjun PS
IIIT-A, India
Gaurav Kumar Yadav
IIIT-A, India
Rahul Kala
ABV-IIIT-M, India
Robert Haschke
Bielefeld University, Germany

###### Abstract

This report introduces our UniTeam agent - an improved baseline for the
HomeRobot: Open Vocabulary Mobile Manipulation challenge. The challenge poses
problems of navigation in unfamiliar environments, manipulation of novel
objects, and recognition of open-vocabulary object classes. This challenge
aims to facilitate cross-cutting research in embodied AI using recent advances
in machine learning, computer vision, natural language, and robotics. In this
work, we conducted an exhaustive evaluation of the provided baseline agent;
identified deficiencies in perception, navigation, and manipulation skills;
and improved the baseline agent’s performance. Notably, enhancements were made
in perception - minimizing misclassifications; navigation - preventing
infinite loop commitments; picking - addressing failures due to changing
object visibility; and placing - ensuring accurate positioning for successful
object placement.

(a) Find object-phase. In this phase, the agent searches for the backpack. In
the predicted semantic map, the blue clusters represent unexplored chairs, the
dark orange clusters represent explored chairs and the light orange clusters
represent cabinets.

(b) Navigate to end receptacle-phase. In this phase, the agent moves towards a
cabinet to place the backpack on. In the semantic map, the blue clusters
represent cabinets and the orange clusters represent chairs.

Figure 1:  Main panel at two different time steps. The instruction in this
episode is "Move backpack from chair to cabinet". From left to right you can
see: The RGB image from the agent’s camera; the segmentation image by Detic;
the predicted semantic map; a third person view of the agent (just for
visualization purposes). The light blue clusters on the predicted semantic map
represent all goal points, while the dark blue point represent the goal point
to which the agent has chosen to navigate towards. On the bottom you can see a
legend for the segmentations, the total amount of collisions, the current
phase and the next action the agent is taking.

##  1 Baseline

The challenge sets a Stretch robot [1] in simulated and real human indoor
environments with the goal of moving the object from the start receptacle to
the end receptacle. The receptacles, which are classes of furniture or
fixtures, don’t have to be a single entity in the environment, for example if
the end receptacle is "table", there are usually multiple tables in the
apartment of which any of them is correct. The HomeRobot Framework [2]
includes two primary baseline approaches: a heuristic baseline utilizing well-
established motion planning techniques alongside simple rules for executing
grasping and manipulation actions; and an reinforcement learning baseline,
where exploration and manipulation skills are acquired using the off-the-shelf
policy learning algorithm DD-PPO [3]. In this work, we will discuss the
heuristic baseline and our improvements on it.

The baseline agent’s capabilities are divided into the following key skills:

  1. 1.

Detection: object detection and segmentation on the agent’s camera RGBD
images. Building Bird’s Eye View (BEV) map with semantic areas.

  2. 2.

Exploration: Exploring the environment to find start_receptacles and
end_receptacle. Exploring start_receptacles to find the object.

  3. 3.

Navigation: Move towards the specified navigation goal: the object, start- or
end receptacle.

  4. 4.

Picking: Picking up the object, supported by a high-level action command due
to the absence of simulated gripper interaction in Habitat. The agent is
equipped with multiple grasping strategies and supports policies for learning
grasping.

  5. 5.

Placing: Adjusting the agent’s position in front of the end_receptacle and
placing the object.

The skills are executed in a certain order. At the beginning, the agent turns
360° to get an overview of its surroundings. After that, the agent uses the
Exploration skill to find the object, navigates towards it if the object is
found and uses the Picking skill to pick it up. If no entity of the end
receptacle is found, the agent uses the Exploration skill again, otherwise it
uses Navigation skill to move to the nearest end receptacle and Placing skill
to place the object. The Detection skill is executed in every frame of the
run. This means that the agent collects information of start and end
receptacles even if the current goal is to find the object. The baseline uses
the perception model Detic [4] to generate masks for objects and receptacles.
A panel showing aspects of the baseline approach during a run can be found in
Figure 1.

Figure 2:  The architecture of our agent. It is build as a state machine with
six states: Find object, Navigate to object, Pick object, Find end receptacle,
Navigate to end receptacle, and Place object. In the Find-phases, the
Exploration skill is used. The agent’s perception model is Detic [4], it
segments the object and receptacles in the RGB image. This segmentation,
together with other observations such as the pose, joints and a depth image,
are the input to the agent.

##  2 Method

An overview of our UniTeam-agent’s architecture can be found in Figure 2.

###  2.1 Detection

Initially, the baseline agent employed the same confidence threshold value for
all receptacles and objects. During testing, it became evident that the agent
frequently overlooked objects with this threshold value. However, reducing the
threshold value increased the misclassification rate of end receptacles. The
implementation of a dynamic object/receptacle-specific threshold proved
effective in mitigating this issue.

The perception module also encounters challenges due to incorrect detection of
objects on the floor (see Fig. 3) or the floor itself as objects or
receptacles. To minimize the occurrence of these false positives [5], we
established a height threshold for receptacles within the perception module
using the depth information. This eliminates detections corresponding to the
floor level, thereby facilitating more accurate planning.

Figure 3:  Example for incorrect floor classifications. The floor is
classified as a table (blue cluster), which is the end receptacle. If the
floor wasn’t filtered out, the agent would want to place the knife on there.

###  2.2 Exploration

The baseline agent frequently encountered challenges when strictly adhering to
a single goal, as the heuristic consistently selected the same action, leading
to infinite oscillations between two adjacent states. To mitigate this issue,
a decision-making mechanism was introduced to operate on the goal map, as
depicted in Figure 4. This mechanism allows the agent to choose between the
goal point and exploring the frontier during navigation, preventing persistent
commitment to an unattainable goal.

In the baseline approach, the agent directs itself towards the nearest
identified receptacle. However, due to imperfect object detection, the nearest
receptacle may not align with the intended goal. To address this, our agent
stores the probabilities of detected receptacles in the Bird’s Eye View (BEV)
map [6]. Subsequently, the agent selects the reachable receptacle with the
highest matching probability, reducing the likelihood of placing the object on
incorrectly classified receptacles. As the agent begins adding identified end
receptacles to the goal map during the Navigate to object phase, a sufficient
number of end receptacles are typically available for selection.

###  2.3 Navigation

When the baseline agent detected a potential collision, the collision was not
incorporated efficiently into the new planning, leading the agent to attempt
the same movement as before. To rectify this behavior, the detected collision
is now integrated at the last short-term goal on the obstacle map. This
refinement provides a more detailed representation of the environment in the
BEV map, enabling the agent to formulate improved plans. Due to the strategy
followed by the baseline agent, instances arose where the agent was not well-
aligned with the end receptacle, resulting in an inadequate position for
successful object placement. To address this limitation, the navigation in our
agent aims to position the agent near the center of the goal receptacles. This
adjustment ensures generally improved starting positions for the object
placement routine.

Figure 4:  The agent’s frontier map shown as the larger enclosed shape, and
the amalgamation of exploration goal points inside. The exploration goal
points adjacent to identified start receptacles indicate free space for
exploring. The intention of the exploration is to check from different sides
whether the goal object is located on this receptacle.

###  2.4 Picking

The baseline picking routine necessitates the presence of the object of
interest within the field of view of the agent’s camera. However, in certain
scenarios, the object is not visible anymore after the agent moved towards the
receptacle, thus resulting in picking failures. To address this issue, out
agent is rotates left and right if the initial picking attempt fails,
expanding its overall field of view of the receptacle and thereby increasing
the probability of object detection.

In our agent a check has been introduced to verify whether the object was
successfully picked up. Only upon a successful pick, the agent transitions to
the next phase; in case of a failed pick, however, our agent reverts to the
navigation phase, exploring different locations for the object.

###  2.5 Placing

The baseline agent encountered challenges in accurately calculating the
distance between the agent and the placing point, resulting in attempts to
move either too far away or too close to the receptacle. This issue was
mitigated in our agent by moving the agent toward the receptacle in small
steps over multiple iterations, ensuring that the agent is positioned
appropriately for successful object placement.

Additionally, the baseline agent often placed objects close to the edges of
receptacles, causing the object to fall down. To address this, the placing
point was adjusted to maintain a safe distance from the receptacle edges,
preventing later object falling down.

In situations where the end receptacle is not recognized during the placing
phase, the baseline agent previously placed the object blindly, resulting in
failures due to collisions or misplaced objects. To mitigate this, our agent
places the object on a surface if the end receptacle is no longer detected,
reducing the occurrence of misplacements.

Furthermore, in the placing phase, large objects grasped by the baseline agent
frequently collide with the end receptacle. This issue is alleviated in our
agent by dropping the object onto the end receptacle from an increased height.

##  3 Evaluation

The HomeRobot OVMM Challenge comprises two primary evaluation stages [7]:

  1. 1.

Simulation Phase: Systems are first evaluated in a simulated environment,
replicating real-world home settings with multiple rooms and varying layouts.
This phase is crucial for testing the agent’s ability to navigate and
manipulate objects in a controlled yet diverse setting.

  2. 2.

Physical Robot Phase: Agents are tested on a physical robot in a real-world
environment, featuring novel objects and layouts. This phase assesses the
agent’s adaptability and performance in real-world conditions.

###  3.1 Simulation Phase

The simulation evaluation encompasses the following aspects [7]:

  * •

Environment Setup: The simulation involves 50 scenes, each offering unique
challenges. These scenes are designed to test the agent’s interaction with a
variety of objects, including both seen and unseen categories.

  * •

Computational Resources: Evaluations are conducted on AWS EC2 p2.xlarge
instances, which include Tesla K80 GPUs, 4 CPU cores, and 61 GB RAM. This
setup ensures a robust platform for performance assessment .

  * •

Testing Protocol: Agents are tested over 1000 episodes, providing a
comprehensive evaluation of their capabilities in diverse scenarios.

###  3.2 Real-World Evaluation

Key aspects of the real-world evaluation include [7]:

  * •

Evaluation Environment: The physical robot testing occurs in a fully equipped
apartment with novel objects and furniture layouts, hosted by Meta in Fremont,
California [7].

  * •

Agent Adaptability: This phase particularly focuses on the agent’s ability to
handle real-world dynamics and novel scenarios, providing a critical
assessment of its practical applicability.

###  3.3 Evaluation Metrics

The challenge employs a set of metrics to assess the agents’ performance:

  1. 1.

Overall Success: correct placement of the object on the target receptacle.

  2. 2.

Partial Success: completion of individual sub-tasks within each episode.

  3. 3.

Number of Steps: number of steps required to complete an episode.

###  3.4 Phases and Splits

The challenge is divided into several phases, each designed to test different
aspects of the agents’ capabilities:

  1. 1.

Minival Phase: A preliminary phase for sanity checking.

  2. 2.

Test Standard Phase: Serves as the public leaderboard and basis for
performance reporting.

  3. 3.

Test Challenge Phase: Determines the finalists of the Simulation Phase to
proceed to the physical robot evaluation.

###  3.5 Results

Results of Test Standard Phase [8]
---
Agent name | Overall Success | Partial Success | Number of Steps
| KuzHum
---
2.8% | 19.1% | 1152.99
UniTeam (our agent) | 2% | 18.6% | 1140.54
PieSquare | 2% | 11.1% | 1082.56
Baseline (heurisitc) | 0% | 10.5% | 1063.74
Table 1: Evaluation results of the top three agents over the 1000 episodes of
the Test Standard Phase, in comparison to the heuristic baseline.

Through incremental enhancements to the baseline agent, as described above,
performance steadily improved, culminating in a score of 2% overall success
during the Test Standard Phase (see Table 1). In comparison to the heuristic
and reinforcement-learned baselines, the our improved baseline agent exhibits
notable performance gains through described adjustments. Notably, refining the
perception module had the most substantial impact on overall performance.

##  4 Conclusion

We implemented improvements on several aspects of the baseline agent.
Utilizing these improvements, we achieved an overall success score of 2% and a
partial success score of 18.6% in the simulation phase of the Open Vocabulary
Mobile Manipulation challenge. This challenge aims to facilitate cross-cutting
research in embodied AI using recent advances in machine learning, computer
vision, natural language, and robotics [9].

Our analysis revealed substantial areas for further improvement. Throughout
the simulation, we encountered instances where the embodied agent [10]
struggled with challenges such as attempting to place an object on a
receptacle that is too large, attempting to place an object through a wall,
getting stuck or abandoning navigation in narrow passages, overlooking the
object, or timing out due to difficulty locating the object, etc. We found
that correctly recognizing the object and the receptacles is still the primary
bottleneck, serving as the root cause for the majority of unsuccessful
episodes.

To further enhance the mobile manipulation performance of the agent, one could
explore Behavior Search methods [11, 12, 13, 14], which integrate few-shot
learning and Imitation Learning advancements, leveraging demonstration
datasets. While training another perception model carries inherent costs,
implementing ideas of modular end-to-end learning [15] may mitigate some of
those problems. However, our approach of filtering the input prompts of Detic
[4] or GroundedSAM [16, 17], an alternative segmentation model, by using
Recognise Anything [18] did not lead to the desired improvements. Using a
concept graph [19] may solve the problem of classifying the same receptacle
differently in consecutive frames. Additionally, addressing the other
identified challenges by implementing heuristic solutions emerges as a
pragmatic and stable avenue for advancing our agent.

## References

  * [1] C. C. Kemp, A. Edsinger, H. M. Clever, and B. Matulevich, “The design of stretch: A compact, lightweight mobile manipulator for indoor human environments,” 2022.
  * [2] S. Yenamandra, A. Ramachandran, K. Yadav, A. Wang, M. Khanna, T. Gervet, T.-Y. Yang, V. Jain, A. W. Clegg, J. Turner, et al., “Homerobot: Open-vocabulary mobile manipulation,” arXiv preprint arXiv:2306.11565, 2023\.
  * [3] N. Bach, A. Melnik, M. Schilling, T. Korthals, and H. Ritter, “Learn to move through a combination of policy gradient algorithms: Ddpg, d4pg, and td3,” in Machine Learning, Optimization, and Data Science: 6th International Conference, LOD 2020, Siena, Italy, July 19–23, 2020, Revised Selected Papers, Part II 6, pp. 631–644, Springer, 2020.
  * [4] X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and I. Misra, “Detecting twenty-thousand classes using image-level supervision,” in ECCV, 2022.
  * [5] A. Melnik, A. Harter, C. Limberg, K. Rana, N. Sünderhauf, and H. Ritter, “Critic guided segmentation of rewarding objects in first-person views,” in KI 2021: Advances in Artificial Intelligence: 44th German Conference on AI, Virtual Event, September 27–October 1, 2021, Proceedings 44, pp. 338–348, Springer, 2021.
  * [6] A. Harter, A. Melnik, G. Kumar, D. Agarwal, A. Garg, and H. Ritter, “Solving physics puzzles by reasoning about paths,” arXiv preprint arXiv:2011.07357, 2020.
  * [7] https://aihabitat.org/challenge/2023_homerobot_ovmm.
  * [8] https://eval.ai/web/challenges/challenge-page/2100/leaderboard/4946.
  * [9] K. Rana, A. Melnik, and N. Sünderhauf, “Contrastive language, action, and state pre-training for robot learning,” arXiv preprint arXiv:2304.10782, 2023.
  * [10] P. König, A. Melnik, C. Goeke, A. L. Gert, S. U. König, and T. C. Kietzmann, “Embodied cognition,” in 2018 6th International Conference on Brain-Computer Interface (BCI), pp. 1–4, IEEE, 2018.
  * [11] S. Beohar and A. Melnik, “Planning with rl and episodic-memory behavioral priors,” arXiv preprint arXiv:2207.01845, 2022.
  * [12] F. Malato, F. Leopold, A. Raut, V. Hautamäki, and A. Melnik, “Behavioral cloning via search in video pretraining latent space,” arXiv preprint arXiv:2212.13326, 2022.
  * [13] F. Malato, F. Leopold, V. Hautamaki, and A. Melnik, “Behavioral cloning via search in embedded demonstration dataset,” arXiv preprint arXiv:2306.09082, 2023.
  * [14] J. Sheikh, A. Melnik, G. Nandi, and R. Haschke, “Language conditioned semantic search based policy for robotic manipulation tasks,” arXiv preprint arXiv:2312.05925, 2023.
  * [15] A. Melnik, S. Fleer, M. Schilling, and H. Ritter, “Modularization of end-to-end learning: Case study in arcade games,” arXiv preprint arXiv:1901.09895, 2019.
  * [16] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, et al., “Grounding dino: Marrying dino with grounded pre-training for open-set object detection,” arXiv preprint arXiv:2303.05499, 2023.
  * [17] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick, “Segment anything,” arXiv:2304.02643, 2023.
  * [18] Y. Zhang, X. Huang, J. Ma, Z. Li, Z. Luo, Y. Xie, Y. Qin, T. Luo, Y. Li, S. Liu, et al., “Recognize anything: A strong image tagging model,” arXiv preprint arXiv:2306.03514, 2023.
  * [19] Q. Gu, A. Kuwajerwala, S. Morin, K. M. Jatavallabhula, B. Sen, A. Agarwal, C. Rivera, W. Paul, K. Ellis, R. Chellappa, et al., “Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning,” arXiv preprint arXiv:2309.16650, 2023.


# Assessing the Contribution of Semantic Congruency to
Multisensory Integration and Conflict Resolution

Di Fu1,2,3, Pablo Barros1, German I. Parisi1, Haiyan Wu2,3,4, Sven Magg1, Xun
Liu2,3, Stefan Wermter1 1Knowledge Technology, Department of Informatics,
Universität Hamburg, Germany<EMAIL_ADDRESS>Key
Laboratory of Behavioral Science, Chinese Academy of Sciences (CAS), Beijing,
China<EMAIL_ADDRESS>of Psychology, University of
CAS, Beijing, China.4Division of the Humanities and Social Sciences, Caltech,
CA, USA.http://cml.knowledge-technology.info

###### Abstract

The efficient integration of multisensory observations is a key property of
the brain that yields the robust interaction with the environment. However,
artificial multisensory perception remains an open issue especially in
situations of sensory uncertainty and conflicts. In this work, we extend
previous studies on audio-visual (AV) conflict resolution in complex
environments. In particular, we focus on quantitatively assessing the
contribution of semantic congruency during an AV spatial localization task. In
addition to conflicts in the spatial domain (i.e. spatially misaligned
stimuli), we consider gender-specific conflicts with male and female avatars.
Our results suggest that while semantically related stimuli affect the
magnitude of the visual bias (perceptually shifting the location of the sound
towards a semantically congruent visual cue), humans still strongly rely on
environmental statistics to solve AV conflicts. Together with previously
reported results, this work contributes to a better understanding of how
multisensory integration and conflict resolution can be modelled in artificial
agents and robots operating in real-world environments.

##  I INTRODUCTION

Multisensory perception is crucial for a robust and efficient interaction with
the environment [1]. The mammalian brain comprises a rich set of neural
mechanisms that integrate multisensory stimuli on the basis of their physical
properties (spatial and temporal alignment [2]) in combination with internal
knowledge and expectations (e.g. semantic congruency [3]). Noisy environmental
conditions can lead to sensory uncertainty and decreased reliability, and thus
require the interplay of both external and internal factors to efficiently
trigger behaviourally relevant actions also in situations of a multisensory
conflict [4].

Robots operating in real-world environments can highly benefit from using
robust mechanisms of multisensory perception with the goal to yield swift and
singular behaviour also in situations of sensory uncertainty and conflict [5].
However, while computational models of multisensory integration (mostly audio-
visual) have been extensively studied in the literature by considering either
exogenous factors (spatial and temporal alignment) and endogenous factors
(e.g. object identity) separately, the intricate interplay of such factors has
been poorly understood. In fact, the majority of the proposed approaches learn
multisensory representations from congruent (co-occurring) audio-visual pairs
(e.g., [6][7][8]), whereas they do not account for handling situations of
conflict (i.e. the visual and the auditory channel convey discordant
information).

Behavioural studies have shown multiple audio-visual effects that reflect the
interplay of spatial and semantic information processing (e.g., the spatial
ventriloquism effect [9], where the location of the auditory stimulus is
perceptually shifted towards the position of the synchronous visual one).
However, the oversimplified stimuli used by these studies (light blobs and
beeps) do not reflect the complexity of the multisensory stimuli that robots
are typically exposed to in natural environments. Therefore, interdisciplinary
efforts are required in order to provide insights into how artificial agents
and robots should behave in situations of multisensory conflict in complex
environments.

To better model multisensory conflict resolution in robots, we have designed
and conducted a series of behavioural studies consisting of a localization
task from audio-visual complex stimuli [10],[11]. In contradistinction to
previously proposed studies using oversimplified stimuli, our studies were
conducted in an immersive projection environment and comprised a scene with
four animated avatars which can produce spatially (in)congruent lip movements,
body motion, and vocalizations. We observed that environmental statistics
embedded in the scene modulate the magnitude and extension (in terms of the
integration window) of the visually-induced bias. In other words, when an
avatar is moving its lips, the perception of a spatially incongruent vocalized
sound is shifted towards the position of this avatar [10]. Furthermore, we
found that the integration of statistically relevant cues (moving lips and
vocalized sounds) are quite robust to perceptually salient stimuli which act
as a distractor (an avatar moving its arm but not the lips) [11].

In this work, we extend the previously proposed studies by introducing an
additional component to the localization task: semantic congruency. While in
the previous studies all of the four avatars had the same appearance, in this
novel study we consider three male-looking avatars and one female-looking one.
The vocalizations can also be either male or female. In this way, we aim at
quantitatively assessing the contribution of semantic congruency to
multisensory interaction and conflict resolution in complex environments.

In the following sections, we introduce the behavioural study, analyze the
collected data, and discuss the obtained results from the perspective of
multisensory modelling for artificial agents and robots.

##  II BEHAVIOURAL STUDY

###  II-A Participants

A total of 32 subjects (aged 21–35, right-handed, 6 female) participated in
our experiment. All participants reported that they had normal or corrected-
to-normal vision and hearing and did not have a history of neurological
conditions such as seizures, epilepsy, and stroke. All participants signed a
consent form approved by the Ethics Committee of Universität Hamburg. This
study was conducted in accordance to the principles expressed in the
Declaration of Helsinki.

###  II-B Apparatus, Stimuli, and Procedure

The participants sat at a desk with their chins positioned on a chin-rest 160
cm from the projection screen. Visual stimuli were presented on a concave
projection with four free-field speakers behind it that differed in azimuth
and were placed to match the mouths of the projected avatars (Fig 1.a). The
mouths of the avatars were 1.36 m above the ground and the avatars’ locations
were at -33, -11, +11, and +33 degrees off the centre (fixation point). For a
more detailed description of the experimental set-up, see our previous work
[11].

The AV localization task consisted of the participants having to select at
each trial which avatar they believe the auditory stimulus was coming from.
One of the four avatars would move his/her lips along with a spatially
congruent or incongruent vocalization. We refer to the avatar moving its lips
as the animated avatar and static otherwise. The auditory stimulus consists of
the combination of 3 syllables (all permutations without repetition composed
of ”ha”, ”wa”, ”ba”) vocalized by a male and a female human voice. Each trial
comprises a female-looking avatar (FA) and three male-looking avatars (MA)
with one animated avatar (Fig 1.b). The spatial location of the FA is pseudo-
randomized across trials. The duration of both the visual and auditory stimuli
is 1000 ms.

We analyzed the following types of conflict:

  1. 1.

Voice-Lips Spatial Conflict: The location of the animated avatar (either FA or
MA) is either congruent or incongruent to the location of the vocalized sound
(either male or female voice).

  2. 2.

Voice-Lips Gender Conflict: Gender-specific incongruency, i.e., animated FA
with male voice or animated MA with female voice.

  3. 3.

Voice-Appearance Gender Conflict: Static or animated avatar (FA or MA) with a
gender-incongruent, spatially-congruent voice.

To ensure that participants have understood the instructions, they began the
experiment with 12 practice trials composed of congruent AV stimuli. The
formal task consisted of 768 trials distributed across 3 sessions. A schematic
illustration of one trial is shown in Fig 1.b. Each trial started with static
avatars and a fixation point displayed for 500 ms, followed by an AV stimulus
and then another 1000 ms with static avatars. The subjects were asked to
produce a response within 2000 ms after the onset of the AV stimulus.

Figure 1: Behavioural study on audio-visual localization: (a) Immersive
experimental set-up with concave projection screen (b) Schematic illustration
of one trial of the AV localization task.

###  II-C Results and Analysis

We analyzed the data in terms of the error rate (ER) of each participant with
respect to the ground-truth position of the auditory stimulus. Since visual
spatial resolution is higher in the center of the field of view (FOV), the
magnitude of the visual bias is expected to be stronger towards the center
rather than towards the periphery [12]. Therefore, we also analyzed the
results taking into account the distance between the avatars and their
absolute location with respect to the FOV of the observer. In line with our
previous studies, divided the conditions in terms of (mirrored) spatial
relationships:

  * •

Central: The two avatars in the center.

  * •

Lateral: The two avatars on the right or the left side.

  * •

1-avatar gap: Two avatars having a 1-avatar gap.

  * •

2-avatar gap: The two avatars at each side of the screen.

Figure 2: Error rate (with respect to the ground-truth location of the
auditory stimulus) for the different types of conflict. (* denotes
$0.01<p<0.05$ and *** $p<0.001$, error bars denote the standard deviation.)
Figure 3: ER analysis of the distance effect on gender-specific conflicts: NC
(No conflict), C (central), L (lateral), 1AG (1-avatar gap), 2AG (2-avatar
gap). (* denotes $0.01<p<0.05$ and *** $p<0.001$, error bars denote the
standard deviation.)

For the three types of conflict, we conducted a repeated measures analysis of
variance (ANOVA; shown in Fig. 2). The main effect of the different types of
conflict was significant ($F=57.48$, $p<0.001$, $\eta^{2}=0.65$) and the main
effect of the congruency was significant ($F=66.41$, $p<0.001$,
$\eta^{2}=0.68$). Post hoc t-tests showed that the ER for the voice-lips
spatial conflict ($ER=31.2\%$) was significantly lower than the voice-lips
gender conflict ($ER=43.1\%$, $p<0.001$) and voice-appearance gender conflict
($ER=43.1\%$, $p<0.001$). Furthermore, there was no significant difference
between these latter two conditions ($p>0.05$). For the congruency main
effect, the participants’ responses were less accurate under incongruent
conditions ($ER=47.5\%$) compared to congruent ones ($ER=30.7\%$, $p<0.001$).
There was a significant interaction effect between the type of conflict and
congruency ($F=52.04$, $p<0.001$, $\eta^{2}=0.63$), showing a significantly
larger effect in the voice-lips spatial conflict.

The main effect of gender-specific voice conflict ($F=5.46$, $p<0.05$,
$\eta^{2}=0.15$) and the distance effect ($F=50.49$, $p<0.001$,
$\eta^{2}=0.62$) were significant (see Fig. 3). The ER for the female voice
(ER=46%) was significantly higher than for the male voice ($ER=45\%$).
Regarding the distance effect, the congruent condition yielded the highest
accuracy ($ER=7\%$), which is significantly higher than the other conditions
($p<0.001$). Post hoc t-tests showed that the ER for the 1-avatar gap
($ER=54\%$) was lower than for the laterals ($ER=57\%$, $p<0.05$). There was
no significant interaction effect. We conducted the repeated-measures ANOVA
again by separating the analysis into two different voice genders (male voice
with animated MA/FA and female voice with animated MA/FA). We found that, when
the voice is male, the interaction effect between the gender of the animated
avatar and the distance was marginally significant ($F=2.33$, $p=0.08$,
$\eta^{2}=0.07$). The simple effect analysis showed that, when the
participants heard a male voice, even though the animated FA was 1-avatar
apart from the ground-truth sound location, humans’ errors were still
significantly higher ($ER=55\%$) than when observing an animated MA
($ER=51\%$, $p<0.05$).

Figure 4: Magnitude of the ventriloquism effect (visually-induced ER) for male
and female voices: NC (No conflict), C (central), L (lateral), 1AG (1-avatar
gap), 2AG (2-avatar gap). (* denotes $0.01<p<0.05$ and *** $p<0.001$, error
bars denote the standard deviation.)

Additionally, we analyzed how the presence of a FA (either static or animated)
biases the responses when the female voice is spatially congruent with a MA.
The main effect of the voice’s gender was significant ($F=7.82$, $p<0.01$,
$\eta^{2}=0.20$). We conducted repeated-measures ANOVA again by separating the
analysis into two different voice gender (MA with male voice and MA with
female voice). In the analysis of the MA with male voice, neither the main
effect of the visual cues and distance were significant, however, the
interaction effect between these two factors was significant ($F=4.99$,
$p<0.01$, $\eta^{2}=0.14$). The simple effect analysis showed that for
1-avatar gap, the static FA (ER =56%) had significantly higher ER than the
animated FA (ER=52%, $p<0.01$). For 2-avatar gap, static FA (ER=57%) had
significantly higher ER than the animated FA (ER = 51%, $p<0.05$). In the
analysis of MA with female voice, both the main effect of the visual cues and
distance were not significant, with the interaction effect between these two
factors being significant ($F=3.82$, $p<0.01$, $\eta^{2}=0.11$).

### Ventriloquism Effect

In our previous studies, we found that different types of visual stimuli have
a different influence on the magnitude of the ventriloquism effect, i.e., the
perceptual shift of the location of the auditory stimulus towards a visual
bias [10],[11]. Here, we analyzed the ventriloquism effect under two
independent conditions: male and female voice (see Fig. 4).

For the male voice condition, the main effect of the distance was significant
($F=28.96$, $p<0.001$, $\eta^{2}=0.48$). Post hoc t-tests showed that the ER
for central ($ER=70\%$) and lateral ($ER=79\%$) were significantly higher than
for the 1-avatar gap ($ER=49\%$, $p<0.001$) and 2-avatar gap ($ER=41\%$,
$p<0.001$). For the female voice condition, the main effect of the visual bias
($F=53.51$, $p<0.001$, $\eta^{2}=0.64$) and distance ($F=33.95$, $p<0.001$,
$\eta^{2}=0.53$) were significant. Post hoc t-tests showed that the magnitude
of the bias from a static FA ($ER=10\%$) was significantly weaker than from an
animated FA ($ER=61\%$) and animated MA ($ER=57\%$, $p<0.001$). ER under the
central ($ER=50\%$) and lateral ($ER=64\%$) conditions had significantly
higher ER than the 1-avatar gap ($ER=32\%$, $p<0.001$) and 2-avatar gap
($ER=27\%$, $p<0.001$). There was a significant interaction effect between
these two factors ($F=5.61$, $p<0.01$, $\eta^{2}=0.16$). The simple effect
analysis showed that under each distance condition, the visual bias driven by
a static FA was significantly weaker than the one by animated avatars
($p<0.001$). These results suggest that although semantically related (gender-
specific) cues affect the magnitude of the visual bias, humans still strongly
rely on environmental statistics to solve AV conflicts, i.e., tending to
perceptually bind a vocalized sound to an animated avatar.

##  III CONCLUSION AND FUTURE WORK

In this work, we extended previously proposed studies to quantitatively assess
the contribution of semantic congruency during an AV spatial localization
task. We considered gender-specific congruency with female and male avatars
and its relation with spatial conflicts (misaligned AV pairs). Our obtained
results suggest that humans still strongly rely on environmental statistics to
solve AV conflicts while semantically-related stimuli affect the magnitude of
the visual bias to a lesser extent.

Future studies should consider stronger semantic information, e.g. person
identity and unique facial features. While gender-specific conflicts influence
the magnitude of the visual bias, one could argue that stronger semantic
relations could have a stronger influence and, speculatively, dominate over
other factors.

From a modelling perspective, we showed in a similar study that a humanoid
robot can reproduce human-like behaviour when exposed to similar experimental
conditions than humans through the use of a deep neural network architecture
[11]. However, the architecture was trained in an supervised fashion (i.e.
using the human behaviour as the desired output), whereas it would be
interesting for robots to learn the robust behaviour via unsupervised
learning.

## ACKNOWLEDGMENT

This research was supported by National Natural Science Foundation of China
(NSFC), the China Scholarship Council, and the German Research Foundation
(DFG) under project Transregio Crossmodal Learning (TRR 169).

## References

  * [1] B. E. Stein and M. A. Meredith, _The merging of the senses_. Cambridge, MA, US: The MIT Press, 1993.
  * [2] E. Macaluso, N. George, R. Dolan, C. Spence, and J. Driver, “Spatial and temporal factors during processing of audiovisual speech: a pet study,” _NeuroImage_ , vol. 21, no. 2, pp. 725 – 732, 2004.
  * [3] P. J. Laurienti, R. A. Kraft, J. A. Maldjian, J. H. Burdette, and M. T. Wallace, “Semantic congruence is a critical factor in multisensory behavioral performance,” _Experimental Brain Research_ , vol. 158, no. 4, pp. 405–414, 2004.
  * [4] D. B. Polley, “Multisensory conflict resolution: Should I stay or should I go?” _Neuron_ , vol. 93, no. 4, pp. 725 – 727, 2017.
  * [5] F. Cruz, G. Parisi, J. Twiefel, and S. Wermter, “Multi-modal integration of dynamic audiovisual patterns for an interactive reinforcement learning scenario,” in _IEEE/RSJ IROS_ , 2016, pp. 759–766.
  * [6] G. I.Parisi, J. Tani, C. Webera, and S. Wermter, “Emergence of multimodal action representations from neural network self-organization,” _Cognitive Systems Research_ , vol. 43, pp. 208–221, 2017.
  * [7] Y. Aytar, C. Vondrick, and A. Torralba, “See, hear, and read: Deep aligned representations,” vol. arXiv:1706.00932, 2017.
  * [8] R. Arandjelovic and A. Zisserman, “Look, listen and learn,” vol. arXiv:1705.08168, 2017.
  * [9] C. E. Jack and W. R. Thurlow, “Effects of degree of visual association and angle of displacement on the ”ventriloquism” effect,” _Perceptual & Motor Skills_, vol. 37, pp. 967–979, 1973.
  * [10] G. I. Parisi, P. Barros, M. Kerzel, H. Wu, G. Yang, Z. Li, X. Liu, and S. Wermter, “A computational model of crossmodal processing for conflict resolution,” in _IEEE EPIROB-ICDL_ , 2017, pp. 33–38.
  * [11] G. I. Parisi, P. Barros, D. Fu, S. Magg, H. Wu, X. Liu, and S. Wermter, “A neurorobotic experiment for crossmodal conflict resolution in complex environments,” in _IEEE/RSJ IROS_ , 2018.
  * [12] B. Odegaard, D. Wozny, and L. Shams, “Biases in visual, auditory, and audiovisual perception of space.” _PLoS Computational Biology_ , vol. 11, no. 12, 2015.