### Godot version

v4.1.rc1.mono.official [1f9e540f1]

### System information

Godot v4.1.rc1.mono - Windows 10.0.19045 - Vulkan (Forward+) - dedicated NVIDIA GeForce GTX 1660 Ti with Max-Q Design (NVIDIA; 31.0.15.3179) - Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz (12 Threads)

### Issue description

When NavigationAgent3D is instantiated at runtime and added to the scene tree, invisible/impassable obstacle is added at the global origin position (0, 0, 0).

### Steps to reproduce

1. Run attached MRP
2. Click with `left mouse button` at the world origin to make agent walk
3. Click `space` on your keyboard to instantiate NavigationAgent3D
4. Click left mouse button to walk through world origin position again
5. Player gets stuck on obstacle at the world origin position

https://github.com/godotengine/godot/assets/8630984/fc9b32fb-25cb-46d6-9a98-ff299e2c0b1c



### Minimal reproduction project

[MRP.zip](https://github.com/godotengine/godot/files/11910363/MRP.zip)

return avoidance_enabled;
}
void NavigationAgent3D::set_agent_parent(Node *p_agent_parent) {
	if (agent_parent == p_agent_parent) {
		return;
	}
	
	NavigationServer3D::get_singleton()->agent_set_avoidance_callback(agent, Callable());
	if (Object::cast_to<Node3D>(p_agent_parent) != nullptr) {
		
		agent_parent = Object::cast_to<Node3D>(p_agent_parent);
		if (map_override.is_valid()) {
			NavigationServer3D::get_singleton()->agent_set_map(get_rid(), map_override);
		} else {
			NavigationServer3D::get_singleton()->agent_set_map(get_rid(), agent_parent->get_world_3d()->get_navigation_map());
		}
		
		if (avoidance_enabled) {
			NavigationServer3D::get_singleton()->agent_set_avoidance_callback(agent, callable_mp(this, &NavigationAgent3D::_avoidance_done));
		}
	} else {
		agent_parent = nullptr;
		NavigationServer3D::get_singleton()->agent_set_map(get_rid(), RID());
	}
}
void NavigationAgent3D::set_navigation_layers(uint32_t p_navigation_layers) {