# Edge Computing in IoT: A 6G Perspective

Nasir Saeed, 1, Mariam Ishtiaq1, and Muhammad Asif Khan, 2  My University11,
Qatar University2, 2,
{email1<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Edge computing is one of the key driving forces to enable Beyond 5G (B5G) and
6G cellular services to overcome the issues of finite computing resources and
limited storage capacity at the user end. Due to the unprecedented increase in
traffic volumes and computation demands, Multi-access Edge Computing (MEC) has
been standardized for 5G to provide cloud-computing capabilities within the
radio access network (RAN) close to mobile users and offer context-aware
services with the help of RAN information. This article presents key
considerations for edge computing in B5G/6G networks in terms of edge
location, edge capacity, user density, federated learning, and security.
Moreover, we will also discuss major edge-centric B5G applications and
services in terms of MEC usage. Furthermore, we present some exciting open
research area in edge computing for 6G networks.

###### Index Terms:

Cloud computing, edge computing, B5G/6G architecture

##  I Introduction

Beyond 5G (B5G) and 6G networks are the catalyst to the digital future of
humanity and the realization of Industry 4.0. As the demand for wireless
connectivity and digital transformation accelerate, worldwide industry
verticals rely on 5G and B5G networks to enable novel applications and
services with ultra-low latency, mission-critical reliability, and a high
degree of mobility. Instead of defining B5G and 6G as ’X times 5G’, this paper
looks at it from an evolutionary and revolutionary perspective. With Moore’s
law, the traditional System on a Chip (SoC) approach is evolving to System-
Technology Co-Optimization (STCO), and consequent system development
methodologies are adopted rapidly. In this context, B5G/6G networks are
envisioned to ensure wireless connectivity and serve a diverse range of
applications by utilizing Multi-access Edge Computing (MEC) as a built in
service with cloud-native concepts like leveraging operations within and
across data centers, communicating in a micro-service environment, and
simultaneously providing secure services and applications [Rodrigues2021].

The realization of B5G/6G technology is credited to the founded upon
innovation of enabling technologies like millimeter wave (mmWave) and
Terahertz (THz) communication, ultra massive multiple-input and multiple-
output (U-MIMO), beamforming, and small cells. In addition, the cloud-native
B5G network architecture offers ubiquitous, convenient, and on-demand network
access to a shared pool of configurable computing resources (e.g., networks,
servers, storage, applications, and services) that can be rapidly provisioned
and released with minimal management effort or service provider interaction.
The concepts of encapsulation and reusability of resources make 5G a viable
option for enabling connectivity for the vertical industries.

The foundation for 5G architecture was laid in 3rd Generation Partnership
Project (3GPP) Release 15 [3GPPrel15]. Whereas, Release 16 (or ‘5G phase 2’)
[3GPP2020] includes additional use cases like application layer services in
vehicle-to-everything (V2X), 5G satellite access, communications in vertical
industries and IoT, and technologies like virtualization and network slicing.
Furthermore, release 17 [3GPPrel17] outlined B5G networks, primarily driven by
global market trends and the inclusion of new use cases. To keep up with this
staggering expectation, future network operations and management will require
a major transformational shift in non-terrestrial networks, side-link
enhancements, and the use of artificial intelligence (AI) and machine learning
(ML) for NG-RAN and air interface.

Since 5G is currently being deployed in certain parts of the world, it is a
good time to discuss the evolution of the B5G networks towards IMT-2030
[IMT2030]. Currently, most of the literature discussing B5G and 6G is based on
a hypothetical extension of 5G scenarios discussing latency, connectivity and
communication requirements going beyond 5G [9040264]. This is mainly because
the discussions around future connectivity are in an incubation phase. So,
before heading towards possible solutions, the need of the hour is to
highlight problems facing the future connectivity. One of the major contenders
for this debate is resource management, where edge computing can play a vital
role. This article takes a brave approach to speculate the role of edge
computing for IoT using 6G networks by elaborating some key research
questions.

In this article, wWe first present a background on the architecture and
technology behind the adaptation of edge computing in B5G and 6G networks and
the core concepts involved. We will then discuss the human-centric 6G networks
focusing on privacy and security as built-in features of MEC architecture, use
of federated learning, and key 6G service areas. We will conclude the article
by presenting open research problems in this realm. Then we discuss the key
considerations of deploying edge computing infrastructure and services. We
then discuss some interesting use-cases of edge-centric services in different
industry verticals and corresponding open research problems with the potential
to shape 6G networks. Lastly, we conclude the article by presenting future
research directions in the realm of edge computing for IoT and some key
insights.

Figure 1: Cloud Computing Services (CCS).

##  II Evolution of Edge Computing in 5G and B5G Networks

The standard 5G network architecture is built on cloud-native technology
enablers like Software Defined Networking (SDN), virtualization, and Multi-
Access Edge Computing (MEC). Conventionally, cloud computing is implemented
using the following three Cloud Computing Services (CCS) (also shown in Fig.
1)

  1. 1.

Software-as-a-Service (SaaS): The consumer can use the provider’s applications
running on a cloud infrastructure where accessibility is possible from various
client devices via a thin client interface (e.g., web-based email) or a
program interface. In SaaS, the underlying cloud infrastructure, including
network, servers, operating systems, and storage, are not managed or
controlled by the consumer.

  2. 2.

Platform-as-a-Service (PaaS):  The consumer can deploy applications created
using programming languages, libraries, services, and tools supported by the
provider onto the cloud infrastructure. Like SaaS, in PaaS, the consumer does
not manage or control the underlying cloud infrastructure, including network,
servers, operating systems, or storage. However, in PaaS, the consumer can
control the deployed applications and possibly configuration settings for the
application-hosting environment.

  3. 3.

Infrastructure-as-a-Service (IaaS): In IaaS, the consumer can deploy and run
arbitrary software, including operating systems and applications, with the
provision of processing, storage, networks, and other fundamental computing
resources. The underlying cloud infrastructure cannot be managed or controlled
by the consumer. However, the consumer can manage the operating systems,
storage, deployed applications, and limited networking components (e.g., host
firewalls).

One of the shortcomings of traditional cloud computing architecture is that it
doesn’t meet the ultra-low latency as 5G would commit. The higher delay in
cloud computing is mainly caused by the inherited longer distance from the
end-users. Edge computing overcomes this by bringing computing and storage
capabilities closer to the end-users. Edge computing and distributed cloud
computing will also reshape the business value of telecommunication networks
by enabling service providers to include vertical industries and move beyond
traditional connectivity-service models. The resulting Edge-as-a-Service
(EaaS) integrates edge in the computing architecture by accelerating edge
accessibility and providing enhanced security. EaaS can be seamlessly merged
into SaaS, PaaS, and IaaS models, respectively, by providing instant access to
an edge deployment model, harnessing flexibility and control over build
deployment, and complete network orchestration control.

In terms of connectivity models, end devices are defined as data consumers;
however, end devices can be both data consumers and data producers in the case
of massive IoT networks. This paradigm shift from a data consumer to a data
consumer/producer needs smart functionality update from enablers like 5G. From
a functional perspective, edge computing is carried out over a heterogeneous
network, arranged in storage and computing capabilities tiers. The selection
of the edge computing device is based on various device capabilities,
including computation power, storage capacity, bandwidth, latency, security,
and network type. Fig. 2 elaborates the different tiers of edge computing
architecture: device tier, edge tier, and central tier. The device tier
includes devices with limited storage and computing capabilities like sensors,
actuators, and user interface devices. This tier often uses proximity networks
with limited range and bandwidth. The edge tier comprises intermediate nodes
like edge nodes, control nodes, IoT gateways, and other nodes that need low
latency or high bandwidth to provide direct support to the nodes in the device
tier. The central tier offers a wide connectivity span, potentially using
cloud computing services, such as a data center. Fig. 2 also shows three
inherently distributed architectural deployment models for B5G networks.
Firstly, cloud computing provides on-demand storage and computing resources.
Secondly, edge computing places computing resources closer to the end-users to
reduce delay, enhance processing speed, and ensure on-premise security. Also,
edge computing gives an added advantage of scalability in terms of increasing
users/devices and optimizing reliability and user experience. Finally, MEC is
an architectural standard for edge computing by ETSI, providing deployment
flexibility at radio nodes, aggregation points, and at the edge of the core
network. In addition, reusability (of microservices), extensibility (to
include new use cases), and efficiency (in terms of computation time resources
and energy) are some of the key features of the MEC standard.

In terms of edge computing services, 5G Mobile Network Operators (MNOs) can
provide that using MEC with different deployment strategies, as shown in Fig.
2. MNOs can move the on-premise edge infrastructure to public or private
clouds to ensure enterprise connectivity, including cloud computing, as-a-
service connectivity, and MEC infrastructure. Using such a hybrid approach,
Communication Service Providers (CSPs) can pass over cloud platforms
(infrastructure and system), focusing entirely on service differentiation.
Independent vendors can also give product integration support and customer
application support. However, to gain full control of the 5G core, strict
Service Level Agreements (SLAs) need to be put in place with cloud owners to
ensure compliance.

Figure 2: Three-Tiered Edge Computing Architecture is constituted by Device
tier, Edge tier and Central tier. Three architectural deployment models for 5G
networks include Cloud Computing, Edge Computing and MEC.

Table I gives an overview of recent surveys analyzing edge computing for IoT
in different application areas using 5G.

##  III Considerations for Edge Centric B5G/6G Networks

From MNO’s perspective, some of the primary motivations for B5G/6G networks
are to deliver high bandwidth and support a high density of users, operate on
a cost and energy-efficient infrastructure, and ensure reliability and
security of operation. Some of the main considerations for B5G networks to
support future services are given in Table II and discussed in this section.

  1. 1.

Edge Architecture Edge computing enables networks to become more efficient by
providing a virtualization platform, expanding the enterprise service base,
and offering network congestion control protocols. From a micro operator’s
perspective, edge location plays a critical role in ascertaining these core
values. In networking terms, edge location refers to its closeness to the
consumer. However, this location may vary for operators, enterprises, end-
users, and application types, on a need basis. Some of these needs or metrics
to decide edge location can be affordable latency, user equipment mobility,
network resource management, and computation cost. Various different edge
locations can be considered to make the edge availability secure and
accessible. Mobile edge servers, retail centers, manufacturing stations,
housing compounds, 5G cellular base stations, and the edge of the mobile
operator’s core network can account for a few edge location possibilities
[edgeLocation]. These possibilities can be explored by ensuring an
uninterrupted power supply and security to reap the benefits of edge
computing. To meet the stringent QoS requirements of B5G services, cloud radio
access network (C-RAN) promises proximity, a high access rate, and low
latency. While there is no definite blueprint for future RAN deployment,
Distributed Unit (DU) and Central Unit (CU) functions can be combined or split
and deployed at the radio site or in centralized locations. Thus, we can say
that location can be determined by services, density, the distance between
sites, and transport network speed.

  2. 2.

Edge Security: Although the EaaS approach of 5G-MEC improves the visibility
over traditional networks and enables faster identification of potential
security threats, the simultaneous operation of a massive number of IoT
devices, cloud-based deployments, and third-party applications expands the
attack surface. Edge computing provides enhanced security compared to cloud
computing by allowing filtration of sensitive data at the source rather than
sending it to a central data center. Therefore, fewer inter-device transfers
of data using EaaS means better security. However, MEC is subject to
vulnerabilities in virtualization technologies like virtual machine escape,
virtual machine manipulation, DNS amplification, and VNF (Virtual Network
Function) location shift [ranaweera2021mec]. Attacks on host-level
orchestration components can collapse the MEC servers. Other possible security
vulnerabilities and their mitigation strategies include detection of access to
unauthorized files or applications, monitoring and blocking any information
leakage, system anomaly detection, and prevention from over-utilization of
host MEC resources. To enable a variety of security services to be customized
on edge regardless of hardware infrastructure, network-based threat response
technology needs to be virtualized in virtual machine or container form. Using
a privacy-preserving edge intelligence model like Federated Learning (FL) is
currently being explored as an option to preserve data integrity by leveraging
the distributed nature of edge-centric 5G verticals. FL ensures data privacy
by executing the shared learning models locally and not sending the training
data to central servers.

  3. 3.

Edge Capacity:  For 5G networks, Multiple-Input, Multiple-Output (MIMO)
technology is employed to deliver the capacity needs. This multi-antenna
transmission technology boosts network capacity and data throughput
significantly by increasing the number of transmission ports. The associated
cost is that a higher number of processing resources are required for all the
antenna signals. To transport all individual antenna signals to the baseband
processing function, the transport capacity of the RAN network reduces. For
this backhaul offloading scenario, edge networks play a major role in
compensating for the overconsumption of network resources by extending the
finite network resources, thus enhancing the delay and capacity performances
of 5G networks [9556285]. Therefore, artificial intelligence (AI) and
associated techniques enable smart edge processing and open new opportunities
for powerful computational processing, massive data acquisition, and edge-
caching capabilities requiring higher-order edge capacity. Hence, the vertical
use case determines the required edge capacity. For instance, video streaming
requires bulks of data caching, while for weather monitoring, small data
updates over a period of time might be enough.

  4. 4.

Edge Intelligence: Traditional machine learning and deep learning techniques
use higher volumes of training data and a large processing time, thus imposing
storage and delay constraints on the centralized server. This is not a viable
learning approach for B5G and 6G networks that promise massive connectivity
with ultra-low latency. Edge intelligence accounts for powerful computational
processing and massive data acquisition locally at edge networks
[zhang2022mobile]. It plays an important role in implementing smart and
efficient resource scheduling strategies in a complex environment with
heterogeneous resources and a massive number of devices. To this end, FL
executes a training algorithm common to all edges to support decentralized
learning further while keeping the data local. The result is a robust machine
learning model able to address critical issues like data privacy, data
security, data diversity, continual real-time learning, and hardware
efficiency. Federated learning for the B5G network has promising benefits in
power consumption minimization, end-to-end latency reduction, and security
enhancement.

##  IV 5G Edge Networks - Use Cases

This section provides an edge-centric overview of 5G applications and a
projected view of B5G architectural requirements. We present some major use
cases, such as video analytics, intelligent transport systems (ITS),
manufacturing, energy, and smart agriculture.

###  IV-A Video Analytics

####  IV-A1 Use Case Description

Video analytics aims to automatically recognize spatial and temporal events in
videos, which is a data-intensive operation and gains particular advantage
from the edge in terms of less data transfer time (less delay), local data
processing (secure), and efficient resource utilization.

####  IV-A2 5G/B5G and Edge Computing Solution for Video Analytics

Massive and time-critical data processing are key requirements of video
analytics. Low latency is primarily required for real-time content
distribution and processing. Content caching of frequently used data at edge
servers for on-request delivery to mobile devices is expected to reduce round-
trip times (RTT) and provide high bandwidth in 5G wireless systems. However,
the associated drawback is the limitation in processing capability at the edge
compared to the data center’s, therefore posing delays from software operation
in a virtualized environment. MEC has been integrated with 5G architecture to
employ VNFs on the edge servers to overcome this delay. Therefore, B5G
networks can envision decentralized approaches like fog computing to reduce
computing overhead further and reduce end-to-end delay. .

####  IV-A3 Applications and Use Cases

Some of the key 5G and B5G verticals and use cases rely on video analytics for
an operation like face recognition from traffic and security cameras, object
tracking and motion detection for surveillance, AI learning of patterns in
live stream video archives, etc. For example, the real-time experience of
event audiences such as sports and concerts can be enhanced using video
analytics at the edge. Augmented Reality (AR) and Virtual Reality (VR)
functions can be efficiently applied to videos captured from different camera
angles and displayed through smartphones, tablets, screens, and AR/ VR
devices. Without edge, such data-intensive use cases cannot promise an
enhanced user experience due to adversaries like delays and jitters. Some
other vital application areas of video analytics are:

  * •

Smart City. To enable smart cities, video analytics at the edge is an
important factor. It can find applications in smart traffic monitoring,
instant fire or smoke detection, waste disposal, and environmental monitoring.

  * •

Surveillance. Live video surveillance at the edge can instantly provide
actionable results to avoid accidents and to enforce law and order.

  * •

Autonomous Vehicles. A self-driving car can maximize the benefit of edge video
analytics using instantaneous data pre-processing for decisions and post-
processing for continued training and feedback.

####  IV-A4 Open Research Problems

For video analytics on the edge, distribution of computation resources is a
major challenge. Although Network Function Virtualization (NFV) has been
discussed in this domain, the actual edge computing capability also needs
reasoning. An amalgam of machine learning and artificial intelligence can
provide solutions but must be extended past certainties, not curbed to mere
pattern matching and curve fitting. Also, non-stationary and variable data
emulating real-world scenarios requires energy efficiency to be investigated.

###  IV-B Intelligent Transport System (ITS)

####  IV-B1 Use Case Description

Intelligent transportation systems (ITS) have a great potential to make
transportation systems safe, efficient, safe, and sustainable. IoT-driven
intelligent transportation systems are defined by an amalgam of complex, data-
intensive, dynamic, and uncertain operations like task management, resource
allocation, and security that can be effectively managed using edge computing.

####  IV-B2 5G and Edge Computing Solution for ITS

Future ITS applications are considered to be one of the most promising
vertical sectors for mobile networks. Two radio access technologies have been
defined and are currently competing for market adoption: IEEE Wireless Access
in Vehicular Environments (WAVE)/ Dedicated Short-Range Communications (DSRC)
based on the IEEE 802.11p and 3GPP LTE Cellular Vehicle-to-Everything (C-V2X)
[saeed2021wireless]. Since both these technologies rely on different channel
access schemes, three major challenges arise when simultaneously accessing the
spectrum in the 5.9 GHz band: coexistence, interoperability (for wide market
acceptance), and backward compatibility. The essence of B5G is to satisfy a
wide spectrum of heterogeneous end devices and vehicles requiring ultra-low
latency and high bandwidth. This scale of computational power within proximity
to the end-users can be made available at the edge of the network, using an
edge cloud or Edge Data Center (EDC). An edge cloud brings about a significant
reduction in end-to-end latency in comparison with a centralized cloud. EDC
networks face traffic burstiness, which is aggravated even further by the
myriad of 5G access technologies. For example, even though the 5G mmWave
offers massive peak-data-rate links, it makes the available capacity
fluctuate. Therefore, the design of flexible EDC networks with promising
ultra-low latency and high and scalable bandwidths is a promising research
direction.

Figure 3: V2X System with Multi-Access Edge Computing.

Fig. 3 shows a simple heterogeneous V2X system using MEC where ITS
applications can communicate with Road-Side Units (RSUs) and base stations.
The MEC hosts can be co-located with these radiating elements (and/or with
C-RAN aggregation points). Moreover, the elements of distributed computing
environment, including terminals, edge cloud, and private/remote clouds, are
also being used in this ITS infrastructure..

####  IV-B3 Applications and Use Cases

ITS can be broadly categorized into three service areas [9497103]:

  * •

Safety services. To minimize accidents and mitigate risks for passengers and
other users (pedestrians and vehicles).

  * •

Non-safety services. To improve traffic management to maximize the efficiency
of the existing road network and enhance experience by factors like traffic
congestion minimization.

  * •

Infotainment services. To provide a range of services to the users of the
vehicle, including Internet access, comfort services, and content sharing.

####  IV-B4 Open Research Problems

ITS is a data-intensive and delay sensitive use case which needs to cater to
mobility, connectivity and security simultaneously. Nonetheless, AI based
solutions are a powerful tool, researchers need to develop lightweight AI
techniques. Optimization of genetic algorithms and game theory can play a
significant role to speed up this process.

###  IV-C Manufacturing

####  IV-C1 Use Case Description

5G networks will bring manufacturers and telecom operators together to build
smart factories capable of innovation, process automation and flexibility. The
current industrial networks have been designed for static manufacturing
processes where even a single change in the manufacturing workflow requires
long maintenance operations. With IoT taking over fourth industrial revolution
(Industry 4.0), the evolution of programmable network, increased security and
dynamic traffic flows over heterogeneous industrial environments will redefine
the manufacturing vertical of B5G connectivity. This continuous evolution will
undoubtedly unravel a drastic transformation in the industrial landscape,
allowing innovative production models and promising new business
opportunities.

Figure 4: Process Control for Smart Manufacturing.

####  IV-C2 5G and Edge Computing Solution for Manufacturing

The success and sustenance of Industry 4.0 lies strongly in the adoption of
emerging Information and Communications Technology (ICT) technologies, such as
loT, artificial intelligence, augmented reality for troubleshooting and Edge
Computing integrated into the manufacturing process. Industry 4.0 aims at the
adoption of smart industrial equipment with increased connectivity, sensing,
and actuation capabilities. By exploring access to real-time production
information, cooperation and advanced control features, novel value added
services are promised for industrial operators and customers. To ensure this,
industrial networks need to undergo a transformation process in order to
support the flexibility expected by the next-generation manufacturing
processes. Here, 5G/ B5G systems can play a key role in enabling Industry 4.0
by lending the network slicing paradigm for industrial use cases over a wide
range of application areas.

Fig. 4 shows a layered architecture for process control in a smart factory or
manufacturing facility. Each device and processes coordinates with the
services platform for designing, creating and delivering the services.
Platform layer is rendered dynamic by the network slicing layer which caters
to 5G network resource management according to the service. Data management
and storage is handled at the edge using MEC. To reduce the total average end-
to-end transmission delay, efficient scheduling algorithms for backhaul links
are required to route real-time traffic.

####  IV-C3 Applications and Use Cases

Manufacturing vertical includes use cases like:

  * •

Real time data analytics finds important applications in safety procedures,
product and process customization for output maximization and better resource
utilization.

  * •

Augmented reality can be used to determine operational inefficiencies to
mitigate the cost of production downtime.

  * •

Object tracking Real-time motion and robot tracking can overcome the tedious
manual work and increase the efficiency of the manufacturing process.

####  IV-C4 Open Research Problems

Manufacturing is one of the key industrial verticals to benefit from 6G
wireless connectivity which can benefit exponentially from interoperability,
reconfigurability and continuous deployment of software and virtualized
resources, all of which are open research areas. This will allow flexible and
need based distributing of computation resources.

###  IV-D Energy

####  IV-D1 Use Case Description

To keep up with United Nation’s Sustainable Development Goals 2030, it is
vital to ensure provision of clean energy for energy intensive sectors. 5G is
an enabler for connected power distribution grids promising remote control and
demand handling while minimizing energy wastage.

####  IV-D2 5G and Edge Computing Solution for Energy

5G has a key role to address urgent issues like climate change and disaster
monitoring, since it has a massive user base and can render energy usage
efficient. To address the dynamic energy requirements of the 5G use cases,
intelligent VPPs can ensure diversified resources, integrated energy
consumption, production and trade. MEC has the potential to enable 5G with
this capability. For instance, energy harvesting is a challenging task to
supply it for low power devices at long ranges, since large aperture antennas
are required. However, large antennas have a narrowing field of view, which
limits their operation if they are widely dispersed from a 5G base station.
The advantage of 5G edge centric cellular systems is that they are
simultaneously truly ubiquitous, reliable, scalable, and cost-efficient. With
smart integration of beam-forming techniques like [eid2021], 5G itself can be
an indispensable wireless power grid for IoT devices.

####  IV-D3 Applications and Use Cases

Energy vertical includes use cases like:

  * •

Virtual Power Plant (VPP) A Virtual Power Plant (VPP) is a distributed cloud-
based power plant that aggregates a high number of Distributed Energy
Resources (DERs), into an integrated and monitored network to enhance power
generation. In a VPP, DERs are integrated into a single edge computing VPP
network infrastructure. The objective of VPP is to dissipate power generated
by individual units during peak load, thus relieving load on the smart grid.
To achieve this, the VPP control center (centralized data center) collects
data from edge devices like sensors, control units, and VPP applications. The
edge computing gateway enables rapid response and real-time power dispatch.
This allows predictive maintenance of the power grid by actively monitoring
the operation of each power plant and energy storage. The local control center
of a VPP can be customized to meet the domestic or industrial energy
consumption requirements.

  * •

Real time data analysis To manage VPP services, optimal scheduling control of
DERs is required in the VPP. In this realm, time-varying optimization of 5G
can be employed for the real-time control of a VPP that could adjust the
output of the DERs to maintain quality in the power network and achieve the
objectives of both the customer and the utility company. This can reduce
generation cost and power losses while increasing VPP service reliability.

####  IV-D4 Open Research Problems

The need for smart energy is inevitable owing to the scarcity of natural
mineral resources and the dire need for clean renewable energy sources. A
myriad of distributed resources are at the disposal of a VPP, therefore
requiring it to rationally allocate distributed resources thus minimizing
wastage. Development of an intelligent edge platform for energy regulation is
a research challenge.

###  IV-E Smart Agriculture

####  IV-E1 Use Case Description

Smart Agriculture is an optimum solution to meet food demand and supply with
safety and consistency. This productivity is enhanced using precise
environmental control and growth management using IoT devices like sensors,
localization systems and robots. This ends up in the use of fewer resources
and less waste than traditional farming.

####  IV-E2 5G and Edge Computing Solution for Automated Crop Monitoring and
Management System

Farms in developed nations have been steadily leveraging and benefiting from
technology over the past few decades. These farms, today, cannot only yield
enough produce to be consumed in the local market, but can also export it. Not
just the yield, the quality of the produce can also be diversified and
enhanced using smart farming techniques. IoT edge solutions can offer a
diverse set of specialized capabilities in the Agriculture vertical, to ensure
in the continued ability of the agriculture industry’s ability to feed its
customers. This use case considers business requirements that farm owners seek
to realize by deploying Smart Agriculture:
i. Cost reduction
ii. Increased yields
iii. Predictive maintenance Using 5G, MEC-enabled IoT can offer advanced
abilities for smart agricultural, like predictive analytics, weather
forecasting and smart logistics and warehousing. The integration of 5G and
edge computing in smart farming is shown in Fig. 5.

####  IV-E3 Applications and Use Cases

  * •

Data Analysis:  The future of the IoT can be shaped by the phenomenal advances
in wireless sensor networks (WSNs) and 5G communication technologies to
provide farmers with real-time data and information anytime and everywhere
their land is. It can determine crop status during different stages. 5G is
critical to precision agriculture using extensive M2M services. Weed
monitoring and insect tracking requires data analytics and can make use of
edge computing.

  * •

Sensing and monitoring: Water consumption can be effectively monitored to
ensure healthy crop and minimal waste.

####  IV-E4 Open Research Problems

The success of smart farming lies in regular monitoring, thus making it data
exhaustive. Research is required into development of localized AI/ML
algorithms with the ability to learn, infer and control from data, in both
static and dynamic environments. Edge can play a significant role in
ubiquitous data collection, aggregation, processing, and distribution for
deterministic decision making.

Figure 5: MEC and 5G for Automated Crop Monitoring and management in Smart
Farm.

##  V Conclusion and Future Research Directions

Edge computing is an integral part of 5G IoT networks and envisioned 6G
technologies that promise to provide more bandwidth with reduced latency to
support novel cloud applications. In this paper, we presented key issues for
edge computing in B5G/6G networks in terms of edge location, edge capacity,
user density, local learning, and security. Moreover, we also discussed major
edge-centric B5G applications and services. In the following, we present some
of the promising future research directions for edge computing in envisioned
6G networks.

###  V-A 6G as Human Centric Networks

6G networks shall be designed based on human-centric services that require
tight-coupling of the QoE metrics to human senses rather than generic
requirements. For example, 5G requires a latency requirement of 1 ms. In
contrast, the 6G network provides latency based on the human reaction times,
such as the 100 ms for auditory reaction, 10 ms for visual reaction, and 1 ms
for perceptual response [Yang2019]. Edge computing can play a vital role, thus
providing guaranteed delay performance. The edge server has a more accurate
local view of the network parameters and the users’ instantaneous
requirements; it becomes more convenient to adapt to the varying conditions
and requirements when the application is running at the edge rather than the
remote cloud server. Nevertheless, the research on investigating edge
computing for 6G human-centric networks is still in infancy and needs the
researcher’s attention.

###  V-B Integrated Terrestrial and Aerial Networks

B5G and 6G networks incorporate aerial and satellite networks as an essential
part [saeed2021point]. Aerial networks consisting of drones or unmanned aerial
vehicles (UAVs) are considered as the first choice for applications such as
search-and-rescue, surveillance, and wildlife conservation. These applications
usually require real-time video analytics, which has several challenges in
terms of communication (ultra-low latency), huge processing, and extra-large
storage capacity. Although MEC in video applications brings significant
improvement in latency reduction and computation offloading, live video
analytics still poses challenges. Some key issues are (i) Computation
migration from one MEC server to another when the drones fly away in
overlapping coverage areas. (ii) Rapidly varying wireless link quality
requires joint communication and computation and control (3C) services. (iii)
Limited onboard energy of drones allows minimal local computation. (iv) D2D
offloading of content and computations among drones on the fly to complement
in areas of poor coverage or increasing demands than the current capacity of
drones and/or MEC servers. MEC-assisted drones and drone-drone cooperation in
the Video IoT (VIoT) services involve large-sized content sharing and
intensive offloading, which intensify the challenges of normal drone
cooperation.

###  V-C Self Sustained Networks (SSNs) using Edge Intelligence

Another test case for 6G networks will be the use of artificial intelligence
(AI) in the entire system architecture (i.e., from the network core to the
edge). 6G networks should use AI-based services such as big data analytics for
key decision-making in various real-time scenarios. Edge computing thus plays
a significant role in 6G networks. The idea is that powerful edge servers can
be deployed as an intermediate layer to provide localized and ultra-low
latency processing for real-time applications. As cloud-based processing can
suffer from long delays and transmission overhead, edge computing can
complement cloud computing. The network services are designed such that fast
content caching and processing are done at the edge to perform delay-sensitive
critical tasks, whereas cloud servers are used to achieve deeper and more
efficient processing at the expense of higher delays. However, edge resources
are limited as compared to the cloud servers, and training AI models may be
infeasible or inefficient. In this case, pervasive AI can be a promising
solution that uses distributed network resources, including edge servers,
cloud servers, and user devices, to intelligently distribute the AI
computation (i.e., training and inference tasks) across various devices, to
minimize the delay. Pervasive AI provides a further advantage in terms of
privacy (due to no sharing of data) and scalability (using the huge
distributed resources).

###  V-D Immersive Augmented and Mixed Reality Streaming

6G networks would be able to meet the unprecedented requirements that 5G might
not support. When we consider the huge volumes of data in future wireless
services such as 8K videos, VR, AR, and even eXtended Reality (XR), the
existing 5G support becomes insufficient. The limitation of 5G basically
arises from the design and baseline KPIs that aim to reach average data rates
of up to 100 Mbps (downlink) and 50 Mbps (uplink) [264]. In addition to the
incapacity of 5G to deliver higher data rates for volumetric applications, 5G
also does not meet the requirements of ultra-precision streaming tasks, such
as remote surgeries and other XR applications. Therefore, investigating edge
computing to support immersive augmented and mixed reality streaming is an
exciting area of research.


# NeRF-Enhanced Outpainting for Faithful Field-of-View Extrapolation

Rui Yu1†, Jiachen Liu2†, Zihan Zhou3, Sharon X. Huang2 †Equal contribution to
this work.1Rui Yu is with the Department of Computer Science and Engineering,
University of Louisville, Louisville, KY 40208, USA
<EMAIL_ADDRESS>Liu and Sharon X. Huang are with College of
Information Sciences and Technology, Pennsylvania State University, University
Park, PA 16802, USA {jzl6493<EMAIL_ADDRESS>Zhou is with Manycore Tech
Inc., Hangzhou, Zhejiang, China<EMAIL_ADDRESS>
###### Abstract

In various applications, such as robotic navigation and remote visual
assistance, expanding the field of view (FOV) of the camera proves beneficial
for enhancing environmental perception. Unlike image outpainting techniques
aimed solely at generating aesthetically pleasing visuals, these applications
demand an extended view that faithfully represents the scene. To achieve this,
we formulate a new problem of faithful FOV extrapolation that utilizes a set
of pre-captured images as prior knowledge of the scene. To address this
problem, we present a simple yet effective solution called NeRF-Enhanced
Outpainting (NEO) that uses extended-FOV images generated through NeRF to
train a scene-specific image outpainting model. To assess the performance of
NEO, we conduct comprehensive evaluations on three photorealistic datasets and
one real-world dataset. Extensive experiments on the benchmark datasets
showcase the robustness and potential of our method in addressing this
challenge. We believe our work lays a strong foundation for future exploration
within the research community.

##  I INTRODUCTION

Figure 1: Problem formulation of faithful FOV extrapolation. A collection of
training images $\mathbf{X}_{i}$ (red box) taken within a specific scene
serves as prior knowledge. In the testing phase, our objective is to
faithfully extrapolate the FOV of newly captured image $\mathbf{Y}$ (blue box)
to a specified FOV by leveraging the prior knowledge of the scene.

The field of view (FOV) of a camera plays a pivotal role in the performance of
vision-based navigation [1, 2]. A larger FOV enables robots to perceive more
spatial elements and layouts (_e.g._ , obstacles, doorways, etc.). This
expanded perspective empowers them to make more informed and strategic
decisions when planning their paths. A larger FOV also offers substantial
benefits for remote sighted agents (RSAs) tasked with assisting visually
impaired individuals in navigation [3, 4]. In light of this motivation, our
work delves into the challenge of FOV extrapolation. Our goal is to enable
robots and remote agents to perceive scene content beyond the immediate camera
FOV, thereby enhancing their situational awareness.

In the computer vision domain, our task is closely related to image
outpainting [5, 6, 7, 8], also referred to as image extrapolation [9, 10, 11]
or image extension [12], which aims to extend the image boundaries with
semantically consistent and visually appealing contents. The extrapolated
image hallucinates visually plausible scene contents beyond the original FOV,
delivering immersive viewing experience for applications such as virtual
reality. The hallucination ability is acquired by, for example, training deep
learning models on large-scale generic datasets [13] of realistic images.
However, image outpainting models cannot be applied to our problem because
navigational applications necessitate the extended portions of the image
maintain fidelity and coherence with the actual scene.

We define the problem of faithful FOV extrapolation as follows. As illustrated
in Fig. 1 (right), a collection of training images that were captured in a
given scene (_e.g._ , images shown inside red boxes) serves as prior
knowledge. We assume that the camera pose corresponding to each training image
can be obtained, for example, via structure from motion (SfM) [14, 15, 16]
methods. During the testing phase, our objective is to faithfully extrapolate
the FOV of any newly captured image in the same scene to a specified FOV based
on the prior knowledge of the scene (see example images inside blue boxes).

It is possible to adapt existing computer vision techniques, namely image
stitching and video expansion, to tackle the faithful FOV extrapolation
problem. Image stitching [17, 18, 19, 20] entails aligning overlapping
portions of multiple images taken from various angles, whereas video expansion
[21], or video extrapolation [22, 23], leverages adjacent frames to extend the
FOV of a specific frame. Both methods require precise warping of source images
to blend seamlessly with the target image. However, they often yield
irregularly shaped non-overlapping areas, imposing limitations on the extent
of FOV expansion. Therefore, these methods are not well suited for our goal,
as the faithful FOV extrapolation task demands expanding the view to a desired
rectangular size.

Given that there has been very limited prior research addressing the challenge
of faithful FOV extrapolation for navigation, we propose a simple yet
effective method called NeRF-Enhanced Outpainting (NEO). Our method involves
first training a neural radiance fields [24] (NeRF) model using training
images of original FOV. We then densely sample a substantial number of camera
poses within the same scene and, for each sampled pose, the trained NeRF model
is applied to render an image with an expanded FOV. Finally, we leverage these
rendered images to train an image outpainting model, which is subsequently
employed to extrapolate the FOV of input images during the inference phase. We
validate the proposed method on three photorealistic datasets and one real
dataset. The NEO method excels at producing high-quality extrapolations
tailored to the specified FOV and consistently surpasses the performance of
three baseline methods.

In summary, the contributions of this work are as follows. (1) We introduce a
novel problem, namely faithful FOV extrapolation for navigation, which has
been relatively underexplored in existing literature. (2) We propose the NeRF-
Enhanced Outpainting (NEO) pipeline as a solution for faithful FOV
extrapolation. (3) Comprehensive empirical investigations on both
photorealistic and real-world datasets consistently validate the effectiveness
of the proposed NEO method compared to the baseline counterparts.

##  II Related Work

###  II-A Image Outpainting

Image outpainting, often referred to as image extrapolation or extension, is a
task that seeks to expand image boundaries while maintaining semantically
coherent content. Typically, the ability to infer such contents is acquired
through learning from large-scale datasets of real images. Image outpainting
approaches can be broadly categorized as either non-parametric or parametric.
Non-parametric methods [25, 26] are restricted to basic pattern outpainting,
and they become increasingly fragile as the extrapolation range grows. The
emergence of GAN-based models has resulted in significant advancements in
image outpainting. Some notable works [27, 9, 12, 5, 10] utilize a single GAN
model for extrapolating the input image. More recently, Khurana _et al._ [11]
propose an image outpainting framework that extends the image within the
semantic label space, thereby produce new objects within the extrapolated
area. Li _et al._ [6] introduce CTO-GAN, which deduces the potential semantic
layout based on foreground elements and subsequently generates the
corresponding background content with the guidance of the predicted semantics.
Yao _et al._ [8] formulate this problem as a sequence-to-sequence
autoregression task based on image patches, and present a query-based encoder-
decoder transformer model to perform extrapolation. In addition to dedicated
outpainting methods, certain image inpainting models [28, 29, 30], which have
the capability to fill large masks, can also be adapted for image outpainting.
Inspired by the pioneering work [31] on diffusion models, there have been
endeavors [32, 33, 34, 35] that tackle the image outpainting problem via a
diffusion-then-denoising process. One limitation of these pretrained image
outpainting models is that the extrapolated parts lack geometric consistency
and interpretability for a specific scene. This characteristic renders them
unsuitable for real-world application scenarios such as navigation. Therefore,
we propose a new problem, faithful FOV extrapolation, the solution of which
enables and facilitates navigational applications by ensuring that the
extrapolated content remains faithful and relevant to the scene at hand.

###  II-B NeRF and Data Augmentation

Neural radiance fields (NeRF) [24] enables novel view synthesis by
representing the density and color of 3D spatial points of a specific scene
through a neural network. With a novel camera pose as input, NeRF can render
an image of specified FOV by performing ray marching from the camera’s central
viewpoint, querying the corresponding color and density fields and conducting
volume rendering. Follow-up works on NeRF have explored improving
generalizability [36, 37], scene editing [38, 39, 40], neural scene
reconstruction [41, 42], training and inference acceleration [43, 44], among
others. Unlike other image outpainting techniques which rely on scene
distribution priors to extrapolate large FoV, NeRF has its unique advantage in
that it implicitly encodes the entire 3D scene, enabling the rendering of
novel views in a manner that is both geometrically and semantically coherent.
This positions NeRF as a potential approach to achieving faithful FOV
extrapolation. Furthermore, NeRF has been utilized as a data augmentation tool
to generate synthetic images for training deep neural networks. Moreau _et
al._ [45] employ NeRF model to create a fresh dataset of synthetic images for
training a camera pose regression model. Ge _et al._ [46] propose an online
data augmentation pipeline based on NeRF synthesis for real-world object
detection. In this paper, we present a NeRF-enhanced outpainting pipeline that
leverages NeRF to generate sufficient synthetic images for training an FOV
extrapolation model.

Figure 2: NEO pipeline: (1) training a NeRF model with captured training
images of original small FOV; (2) using the trained NeRF to synthesize images
of extended FOV by densely sampling camera poses in the scene; (3) training an
outpainting model with the synthetic images of extended FOV. During inference,
we use the trained outpainting model for faithful FOV extrapolation.

##  III Proposed method

###  III-A Problem Formulation

As shown in Fig. 1, a camera captured $N$ training images
$\left\\{\mathbf{X}_{i}\in\mathbb{R}^{h\times w\times
3}|i=1,2,\dots,N\right\\}$ in a specific scene, which may have been taken
sparsely. The FOV of each image is $(\alpha_{x},\alpha_{y})$, indicating the
horizontal and vertical FOV angles, respectively. We assume that the camera
pose $\mathbf{P}_{i}\in\mathbf{SE}(3)$ for each training image
$\mathbf{X}_{i}$ can be acquired through structure from motion (SfM) pipelines
such as COLMAP [16]. During testing, we seek to extrapolate a testing image
$\mathbf{Y}\in\mathbb{R}^{h\times w\times 3}$ with the same FOV
$(\alpha_{x},\alpha_{y})$ captured in the same scene to a new image
$\widehat{\mathbf{Y}}\in\mathbb{R}^{H\times W\times 3}$ with larger FOV
$(\gamma_{x},\gamma_{y})$ while keeping the camera’s focal length constant.
The extrapolated portions of the image must be consistent with the real scene.

###  III-B NeRF-Enhanced Outpainting (NEO)

To address this problem, we propose a simple yet effective method, dubbed
NeRF-Enhanced Outpainting (NEO) with three steps in the training stage, as
illustrated in Fig. 2.

Step 1: Training a NeRF model. We start by training a NeRF model using
training images $\left\\{\mathbf{X}_{i}|i=1,2,\dots,N\right\\}$. NeRF learns
an implicit representation of the given 3D scene with a multilayer perceptron
(MLP). With the trained NeRF, we can emit a ray from any direction and sample
points on the ray to obtain their density and radiance, then render a novel
view through volume rendering. Therefore, NeRF can generate an image of a
specified FOV with a new camera pose. Using the trained NeRF, we can render an
arbitrary number of images with specified poses and desired FOV.

Step 2: Synthesizing images. Next, we sample a multitude of new camera poses
within the scene. For each pose, we leverage the trained NeRF model to render
a pair of images with FOVs of $(\alpha_{x},\alpha_{y})$ and
$(\gamma_{x},\gamma_{y})$, respectively. We sample new camera poses by
ensuring that they cover all walkable areas in the training trajectories.
Moreover, we manage to sample poses with different degrees of freedom (DoF)
following their DoF distribution in the specific environment. The details will
be presented in Sec. IV.

Step 3: Training an outpainting model. Finally, we utilize the NeRF-rendered
images as training data to train an image outpainting model, which takes the
small-FOV images as input and extrapolates the large-FOV ones. Various image
outpainting models [7] can be employed in this step. Image inpainting models
[30, 29] that fill in large empty spaces can also be applied to outpainting.
However, it is worth noting that some outpainting or inpainting models are
designed to generate diverse results with randomness. This property does not
align with the objective of faithful FOV extrapolation. Thus, we modify such
models during our implementation for the purpose of training an outpainting
model that performs deterministic extrapolations faithful to the environment.

During inference, a real small-FOV image is given to the trained outpainting
model as input and the model performs faithful FOV extrapolation to obtain the
large-FOV image.

###  III-C Discussions

1) Why not directly train an outpainting model using the training images
$\left\\{\mathbf{X}_{i}|i=1,2,\dots,N\right\\}$?

The outpainting model takes $\mathbf{X}_{i}\in\mathbb{R}^{h\times w\times 3}$
as input and produces $\widehat{\mathbf{X}}_{i}\in\mathbb{R}^{H\times W\times
3}$ as output. However, there are no $\widehat{\mathbf{X}}_{i}$ in the
training data. To address this issue, we could simply resize
$\mathbf{X}_{i}\in\mathbb{R}^{h\times w\times 3}$ to
$\mathbf{X}^{\prime}_{i}\in\mathbb{R}^{H\times W\times 3}$ and then crop the
central part $\mathbf{X}^{\prime\prime}_{i}\in\mathbb{R}^{h\times w\times 3}$
to use as input. Since the training is simply achieved through resizing and
cropping the original small-FOV training images, we refer to this method as
naive outpainting. However, this approach encounters two main challenges.
First, the quantity and coverage of the training data prove inadequate for
hallucinating from a new viewpoint. Second, cropping the central part reduces
the FOV of the training image, leading to a mismatch of the FOVs during
training and testing stages.

Our proposed NEO pipeline addresses the above two challenges by leveraging
NeRF-based synthesis. First, there is no longer a concern about limited amount
of training data, as NEO can theoretically generate an unlimited number of
synthetic images by sampling arbitrary camera poses across walkable areas.
Second, the issue of training-testing FOV mismatch is resolved, because the
NEO pipeline trains the outpainting model using synthetic images with the same
FOV as the target of the testing stage. The underlying principle of NEO is
that the resulting outpainting model learns to extrapolate faithfully in the
given scene by processing an extensive volume of training images that
comprehensively cover the entire scene.

2) Why not directly synthesize the target images using the trained NeRF model?

NeRF can generate a specific extended-FOV image given a camera pose. However,
during the testing phase, the camera pose of the testing image $\mathbf{Y}$ is
unknown. We could employ camera relocalization (also known as visual
localization) paradigms [15, 16] to estimate the camera pose of $\mathbf{Y}$.
Yet a main issue with combining relocalization and NeRF is that, the estimated
pose may not be precise enough due to errors in feature detection, matching,
as well as perspective-n-point (PnP) [47] estimation. Consequently, the
resulting extended-FOV image rendered by NeRF may not be aligned well with the
testing image $\mathbf{Y}$. In contrast, the NEO pipeline circumvents the need
for highly accurate estimation of the camera poses for testing images.
Instead, it benefits from NeRF by training an outpainting model with the
underlying scene priors from NeRF renderings.

##  IV Experiments

TABLE I: Quantitative evaluation on four datasets. The backbone for computing
LPIPS metrics is VGG network.

Method | Replica (photorealistic) | Gibson (photorealistic) | HM3D (photorealistic) | ScanNet (real)
---|---|---|---|---
PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
(B1) Naive Outpainting [30] | 20.59 | 0.781 | 0.348 | 18.05 | 0.705 | 0.404 | 17.92 | 0.630 | 0.427 | 19.98 | 0.755 | 0.188
(B2) Warping & Fusion [48] | 19.03 | 0.745 | 0.397 | 16.61 | 0.688 | 0.496 | 15.94 | 0.582 | 0.512 | 21.02 | 0.755 | 0.238
(B3) Relocalized NeRF [49, 50, 16] | 16.78 | 0.724 | 0.386 | 14.90 | 0.641 | 0.474 | 14.43 | 0.602 | 0.484 | 18.88 | 0.695 | 0.188
Oracle NeRF [49, 50] | 32.90 | 0.936 | 0.174 | 32.16 | 0.928 | 0.188 | 27.03 | 0.824 | 0.299 | 23.80 | 0.805 | 0.108
NEO | 25.94 | 0.868 | 0.217 | 23.53 | 0.822 | 0.263 | 21.54 | 0.731 | 0.338 | 22.40 | 0.793 | 0.168

Figure 3: Qualitative results on three photorealistic datasets: Replica (1st
row), Gibson (2nd row), and HM3D (3rd row). Figure 4: Qualitative results on
ScanNet dataset.

###  IV-A Baseline Methods

To demonstrate the effectiveness of our proposed NEO, we first introduce three
baseline methods to address this problem, as we have discussed earlier. The
first method is the “naive outpainting” mentioned in Sec. III-C, where an
outpainting model is trained only using the original small-FOV images. The
original image is resized larger and its cropped central part is used as the
small-FOV input, while the original image is used as the corresponding large-
FOV target. The second baseline approach, dubbed “warping & fusion” (B2), goes
through an image stitching pipeline. To be specific, for a testing image, we
first retrieve the nearest images from the training set, then build
correspondence between the testing image and the retrieved (source) images.
Finally, image warping is employed to get a larger FOV image by warping and
fusing the contents from source images. The last baseline is called
“relocalized NeRF”. We first train a NeRF model on the training images. For a
testing image, we relocalize its camera pose by employing camera
relocalization methods. Finally we render a large-FOV image with the trained
NeRF and the relocalized pose.

###  IV-B Datasets and Metrics

We first evaluate the FOV extrapolation performance on three scenes from three
photorealistic datasets: 1) Replica dataset [51] includes $18$ realistic
indoor scenes. We adopt the first floor of the apartment0 scene for
evaluation. 2) Gibson dataset [52] includes realistic scans of $572$ full
buildings. We adopt the Bonesteel building to verify our method. 3) Habitat-
Matterport 3D (HM3D) dataset [53] includes realistic scans of $1,000$
buildings. We adopt scene 00065 for evaluation. To verify our method on real
scenes, we further demonstrate our method on ScanNet [54] which samples posed
RGB images from $1513$ real indoor scans. We adopt scene0000_00 as our
training set and scene0000_01 whose data is sampled at the same scene but with
different trajectories as our testing set.

For photorealistic datasets, we consider a simplified scenario: a robot with a
fixed height and a fixed front camera navigates in an indoor environment. In
such a setting, the camera has a constant height and can only rotate
horizontally, so the motion of the camera has only 3 DoFs. For each scene, we
use the Habitat environment [55] to render $1,000$ training images with
$256\times 256$ resolution and $90^{\circ}$ FOV from random camera poses at a
fixed height of 1.5$m$. We then render $2,000$ testing images with the same
resolution and FOV, which contains 40 random walking paths at the same height
with 50 images on each path. The target resolution during testing is
$512\times 512$ ($126.87^{\circ}$ FOV) by uniformly extrapolating in four
directions. For ScanNet, we simulate a challenging but more realistic scenario
to extrapolate an input central image with resolution $240\times 160$ in
horizonal directions (left and right sides), whose target resolution is
$240\times 320$. We uniformly sample the original training trajectory by image
IDs with an interval $10$, leading to $558$ training images. For testing, we
randomly sample three pieces of continuous trajectory, each with $200$ images,
leading to $600$ testing views.

Since we require the extrapolated regions to be consistent with the scene, we
adopt PSNR, SSIM and LPIPS [56] as the evaluation metrics for faithful FOV
extrapolation.

###  IV-C Implementation Details

Photorealistic Datasets. For Replica, Gibson, and HM3D datasets, we use MAT
[30] as our default outpainting model but remove its style manipulation module
for deterministic extrapolation. For warping & fusion baseline (B2), we employ
the pipeline proposed in [48] as our implementation. For camera
relocalization, we first apply an image retrieval method NetVLAD [57] to
retrieve the nearest images from the training set w.r.t. a testing image, then
run COLMAP [16] which generalizes well to different environements to get the
relative pose between the testing image and the retrieved training image.
Finally we transform the relative pose to the absolute pose using the known
training pose. For NeRF model, we employ DirectVoxGO [49] which leverages 3D
voxel representation to accelarate training and inference.

For new data generation, we first derive the walkable areas from the 2D floor
plans of the datasets. We sample new camera poses on a 2D horizontal grid with
a default interval of $0.05m$. At each position, we uniformly sample $72$ yaw
angles for the horizonal rotation. As a result, we sampled about 1.63 million,
1.43 million, 1.56 million camera poses for Replica, Gibson, and HM3D,
respectively.

ScanNet. For ScanNet [54], we instead use LaMa [29] for outpainting, to
satisfy our need on different resolution and aspect ratios. For NeRF model, we
replace DirectVoxGO with a state-of-the-art NeRF method [50] on ScanNet to
achieve better rendering performance. Other baselines are evaluated in a
similar manner as on the photorealistic ones.

For new data generation, we aim to generate 6-DoF novel poses whose
distribution is similar to the training trajectory. Specifically, on $x$-$y$
plane, we sample new camera poses on a uniform 2D horizonal grid with an
interval of $0.2m$, and ensure the sampled trajectories are roughly covered by
the training set, _i.e._ , the Euclidean distance from the new pose to the
nearest training pose on $x$-$y$ plane should be limited within a threshold of
$0.3m$. For horizonal rotation (yaw angle), we sample from a uniform
distribution whose upper and lower bounds come from the training poses. For
other DoFs (vertical translation, pitch and roll rotation), we empirically
discover that the training poses roughly follow a Gaussian distribution, thus
we sample from a Gaussian distribution whose mean and standard deviation are
calculated from the training poses. Then we employ NeRF to render novel views
using these poses. However, practically we found that a non-negligible portion
of the rendered images on ScanNet are too blurry to provide useful information
for training the outpainting model. To address this problem, we apply a blur
detection algorithm based on Laplacian variance to compute the blurry scores
for filtering out blurry images. Eventually, we generated $0.75$ million
outpainting-trainable images on ScanNet.

###  IV-D Results

Quantitative Results. Table I illustrates the extrapolation results of the
proposed NEO approach and three baseline methods on four datasets. In
addition, for validation purposes, the performance of “oracle NeRF” is
reported, which employs the groundtruth camera pose of the test image for NeRF
rendering. “Oracle NeRF” reflects the quality of the trained NeRF model and
sets an upper bound for (B3) relocalized NeRF. Since NEO learns the
outpainting from NeRF-augmented images, its performance is expected to be
lower than that of “oracle NeRF”. As shown in Table I, NEO significantly
outperforms the three baseline methods. As anticipated, “oracle NeRF” achieves
the best results although it is not for practical use since groundtruth camera
poses for testing images are unknown. Although the three baseline methods
produce reasonably good results, they encounter non-trivial problems with
faithfulness, which we will demonstrate in qualitative results below.

Qualitative Results. We show some qualitative results on Replica, Gibson, and
HM3D datasets in Fig. 3. “Oracle NeRF” learns a geometrically consistent 3D
representation thus demonstrates appealing rendering in a coherent way on the
three datasets. The areas extrapolated by NEO are also much more accurate and
faithful to the scene compared to the three baselines. NEO sometimes suffers
from slight misalignment around small objects (_e.g._ , the painting on the
left wall on the third row of Fig. 3). The extrapolated regions of (B1) naive
outpainting tend to be blurry, which is mainly caused by the limited number of
training images. The extended areas by (B2) warping & fusion are limited by
the non-overlapping regions of neighboring images. As for (B3) relocalized
NeRF, the input region (central part) always misaligns with the extrapolated
regions due to evident errors in pose estimation. Comparison of Results on the
ScanNet dataset, as shown in Fig. 4, leads to similar observations.
Surprisingly, we found that NEO can avoid some issues encountered by “orcale
NeRF” and achieve better visual quality in some regions, such as the blurry
floor region near the border of extrapolated image by “oracle NeRF” in the
second row of Fig. 4. We suspect the reason is that, NeRF is trained on sparse
views from the original training set, thereby its rendering quality in
specific areas may be dependent on the availability of informative,
overlapping training images. In contrast, NEO trains the outpainting model on
sufficient, dense novel views rendered from NeRF, so it is more capable of
learning a semantically and geometrically coherent color field of the scene,
effectively reducing the impact of insufficient information in some areas of
the original NeRF.

###  IV-E Discussions

Pose Sampling. It is important in the NEO approach to cover as many views as
possible in the scene in Step 2 of the training process. Thus, the
distribution and number of sampled poses are crucial for training a highly
effective outpainting model. In this study, we vary the interval of the 2D
grid to control the sampling density on the Replica dataset. As shown in Table
II (a), the generative performance naturally improves when increasing the pose
sampling density. The improvement is significant (+1.30 in PSNR) when reducing
the interval from $0.1m$ to $0.05m$, where the number of sampled poses
increases from $0.4M$ to $1.6M$.

TABLE II: Effect of (a) pose sampling density and (b) FOV of training images in NEO pipeline on Replica dataset. | Interval | # Pose | FOV |  PSNR $\uparrow$ |  SSIM $\uparrow$
---|---|---|---|---|---
(a) | 0.05 | 1,629,864 | extended | 25.94 | 0.868
0.10 | 406,224 | extended | 24.64 | 0.851
0.20 | 101,304 | extended | 24.52 | 0.849
0.50 | 16,056 | extended | 23.37 | 0.833
| 1.00 | 3,744 | extended | 22.72 | 0.824
(b) | 0.05 | 1,629,864 | extended | 25.94 | 0.868
0.05 | 1,629,864 | original | 20.61 | 0.802

FOV of Training Images. A key issue of training an outpainting model for FOV
extrapolation is the consistency in FOV between training and testing images.
Figure 5 demonstrates the FOV mismatch problem in the naive outpainting
method. The solid red and blue arrows in Fig. 5(a) represents the camera’s
inherent FOV $\alpha$. From a resized training image captured at Pose 2 (Fig.
5(c)), naive outpainting learns to extrapolate the cropped FOV $\beta$ (dashed
red) to $\alpha$. However, for a testing input image (solid blue) at Pose 1
(Fig. 5(b)), the goal is to extrapolate the inherent FOV $\alpha$ to a larger
FOV $\gamma$ (dashed green). Though the central parts (inputs) of Fig. 5(b)
and Fig. 5(c) are similar, their extrapolated parts are totally different. The
FOV mismatch issue of naive outpainting can also be observed in the
qualitative results (_e.g._ , the painting on the second row in Fig. 3). To
further examine the effect of FOV, we evaluate a variant of NEO, which uses
the original-FOV synthetic images to train the same outpainting model. As seen
in Table II (b), the performance greatly decreases (-5.33 in PSNR), indicating
the significance of training FOV.

(a) FOV angles

(b) Image at Pose 1

(c) Image at Pose 2

Figure 5: FOV analysis. The discrepancy in the FOV between the (c) training
image and (b) testing image may lead to false extrapolation behaviors.

##  V Limitations and Conclusions

As an initial exploration of the faithful FOV extrapolation task, this paper
focuses on tackling the problem for only static scenes. However, real-world
navigation usually entails dynamic objects or people. Moreover, scenes are
rarely static over time, _e.g._ , furniture may be rearranged. This study can
serve as a probe to more comprehensive research in this area. In the future,
we envision to explore solutions that can accommodate the complexities of more
realistic scenarios. One way may leverage dynamic NeRF [58] that better
handles dynamic scenarios.

To conclude, in this paper, we formulate a new problem named faithful image
extrapolation to increase FOV of a given image. It requires the expanded area
to adhere to the real environment. To address this problem, inspired by the
recent surge of NeRF-based rendering approaches, we propose a novel pipeline
dubbed NEO, to train a NeRF-enhanced image outpainting model. Our key insight
is to obtain sufficient and interpretable training data to aid the training of
outpainting model from the novel views rendered by NeRF on a specific scene.
Compared with competing baselines, our model has showcased superior generative
performance. Our synthesized views are geometrically and semantically
consistent with the 3D environment, thereby achieving faithful extrapolation
that opens up potential applications such as AR-based navigation.

## References

  * [1] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi, “Target-driven visual navigation in indoor scenes using deep reinforcement learning,” in _IEEE International Conference on Robotics and Automation, ICRA_ , 2017, pp. 3357–3364.
  * [2] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. D. Reid, S. Gould, and A. van den Hengel, “Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,” in _IEEE Conference on Computer Vision and Pattern Recognition, CVPR_ , 2018, pp. 3674–3683.
  * [3] S. Lee, R. Yu, J. Xie, S. M. Billah, and J. M. Carroll, “Opportunities for human-AI collaboration in remote sighted assistance,” in _ACM International Conference on Intelligent User Interfaces, IUI_ , 2022, pp. 63–78.
  * [4] J. Xie, R. Yu, S. Lee, Y. Lyu, S. M. Billah, and J. M. Carroll, “Helping helpers: Supporting volunteers in remote sighted assistance with augmented reality maps,” in _ACM Conference on Designing Interactive Systems, DIS_ , 2022, pp. 881–897.
  * [5] Z. Yang, J. Dong, P. Liu, Y. Yang, and S. Yan, “Very long natural scenery image prediction by outpainting,” in _IEEE/CVF International Conference on Computer Vision, ICCV_ , 2019, pp. 10 560–10 569.
  * [6] J. Li, C. Chen, and Z. Xiong, “Contextual outpainting with object-level contrastive learning,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR_ , 2022, pp. 11 441–11 450.
  * [7] Y. Cheng, C. H. Lin, H. Lee, J. Ren, S. Tulyakov, and M. Yang, “Inout: Diverse image outpainting via GAN inversion,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR_ , pp. 11 421–11 430.
  * [8] K. Yao, P. Gao, X. Yang, J. Sun, R. Zhang, and K. Huang, “Outpainting by queries,” in _European Conference on Computer Vision, ECCV_ , 2022, pp. 153–169.
  * [9] Y. Wang, X. Tao, X. Shen, and J. Jia, “Wide-context semantic image extrapolation,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR_ , 2019, pp. 1399–1408.
  * [10] D. Guo, H. Liu, H. Zhao, Y. Cheng, Q. Song, Z. Gu, H. Zheng, and B. Zheng, “Spiral generative network for image extrapolation,” in _European Conference on Computer Vision, ECCV_ , 2020, pp. 701–717.
  * [11] B. Khurana, S. R. Dash, A. Bhatia, A. Mahapatra, H. Singh, and K. Kulkarni, “Semie: Semantically-aware image extrapolation,” in _IEEE/CVF International Conference on Computer Vision, ICCV_ , 2021, pp. 14 880–14 889.
  * [12] D. Krishnan, P. Teterwak, A. Sarna, A. Maschinot, C. Liu, D. Belanger, and W. T. Freeman, “Boundless: Generative adversarial networks for image extension,” in _IEEE/CVF International Conference on Computer Vision, ICCV_ , 2019, pp. 10 520–10 529.
  * [13] B. Zhou, À. Lapedriza, A. Khosla, A. Oliva, and A. Torralba, “Places: A 10 million image database for scene recognition,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 40, no. 6, pp. 1452–1464, 2018.
  * [14] P. Lindenberger, P.-E. Sarlin, V. Larsson, and M. Pollefeys, “Pixel-perfect structure-from-motion with featuremetric refinement,” in _IEEE/CVF International Conference on Computer Vision, ICCV_ , 2021, pp. 5987–5997.
  * [15] P. Sarlin, A. Unagar, M. Larsson, H. Germain, C. Toft, V. Larsson, M. Pollefeys, V. Lepetit, L. Hammarstrand, F. Kahl, and T. Sattler, “Back to the feature: Learning robust camera localization from pixels to pose,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR_ , 2021, pp. 3247–3257.
  * [16] J. L. Schonberger and J.-M. Frahm, “Structure-from-motion revisited,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR_ , 2016, pp. 4104–4113.
  * [17] R. Szeliski, “Image alignment and stitching: A tutorial,” _Found. Trends Comput. Graph. Vis._ , vol. 2, no. 1, 2006.
  * [18] N. Li, Y. Xu, and C. Wang, “Quasi-homography warps in image stitching,” _IEEE Trans. Multim._ , vol. 20, no. 6, pp. 1365–1375, 2018.
  * [19] T. Liao and N. Li, “Single-perspective warps in natural image stitching,” _IEEE Trans. Image Process._ , vol. 29, pp. 724–735, 2020.
  * [20] K. Lee and J. Sim, “Warping residual based image stitching for large parallax,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR_ , 2020, pp. 8195–8203.
  * [21] M. Wang, A. Shamir, G.-Y. Yang, J.-K. Lin, G.-W. Yang, S.-P. Lu, and S.-M. Hu, “Biggerselfie: Selfie video expansion with hand-held camera,” _IEEE Trans. Image Process._ , vol. 27, no. 12, pp. 5854–5865, 2018.
  * [22] S. Lee, J. Lee, B. Kim, K. Kim, and J. Noh, “Video extrapolation using neighboring frames,” _ACM Trans. Graph._ , vol. 38, no. 3, pp. 20:1–20:13, 2019.
  * [23] L. Ma, S. Georgoulis, X. Jia, and L. Van Gool, “Fov-net: Field-of-view extrapolation using self-attention and uncertainty,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 3, pp. 4321–4328, 2021.
  * [24] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” in _European Conference on Computer Vision, ECCV_ , 2020, pp. 405–421.
  * [25] A. A. Efros and T. K. Leung, “Texture synthesis by non-parametric sampling,” in _IEEE/CVF International Conference on Computer Vision, ICCV_ , 1999, pp. 1033–1038.
  * [26] A. A. Efros and W. T. Freeman, “Image quilting for texture synthesis and transfer,” in _SIGGRAPH_ , L. Pocock, Ed., 2001, pp. 341–346.
  * [27] R. A. Yeh, C. Chen, T. Lim, A. G. Schwing, M. Hasegawa-Johnson, and M. N. Do, “Semantic image inpainting with deep generative models,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR_ , 2017, pp. 6882–6890.
  * [28] S. Zhao, J. Cui, Y. Sheng, Y. Dong, X. Liang, E. I. Chang, and Y. Xu, “Large scale image completion via co-modulated generative adversarial networks,” in _International Conference on Learning Representations, ICLR_ , 2021.
  * [29] R. Suvorov, E. Logacheva, A. Mashikhin, A. Remizova, A. Ashukha, A. Silvestrov, N. Kong, H. Goka, K. Park, and V. Lempitsky, “Resolution-robust large mask inpainting with fourier convolutions,” in _IEEE/CVF Winter Conference on Applications of Computer Vision, WACV_ , 2022, pp. 3172–3182.
  * [30] W. Li, Z. Lin, K. Zhou, L. Qi, Y. Wang, and J. Jia, “MAT: mask-aware transformer for large hole image inpainting,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR_ , 2022, pp. 10 748–10 758.
  * [31] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 6840–6851, 2020.
  * [32] C. Saharia, W. Chan, H. Chang, C. Lee, J. Ho, T. Salimans, D. Fleet, and M. Norouzi, “Palette: Image-to-image diffusion models,” in _ACM SIGGRAPH_ , 2022, pp. 1–10.
  * [33] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR_ , 2022, pp. 10 684–10 695.
  * [34] A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool, “Repaint: Inpainting using denoising diffusion probabilistic models,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR_ , 2022, pp. 11 461–11 471.
  * [35] S. Cai, E. R. Chan, S. Peng, M. Shahbazi, A. Obukhov, L. Van Gool, and G. Wetzstein, “Diffdreamer: Towards consistent unsupervised single-view scene extrapolation with conditional diffusion models.”
  * [36] K. Schwarz, Y. Liao, M. Niemeyer, and A. Geiger, “Graf: Generative radiance fields for 3d-aware image synthesis,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 20 154–20 166, 2020.
  * [37] A. Yu, V. Ye, M. Tancik, and A. Kanazawa, “pixelnerf: Neural radiance fields from one or few images,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR_ , 2021, pp. 4578–4587.
  * [38] S. Liu, X. Zhang, Z. Zhang, R. Zhang, J.-Y. Zhu, and B. Russell, “Editing conditional radiance fields,” in _IEEE/CVF International Conference on Computer Vision, ICCV_ , 2021, pp. 5773–5783.
  * [39] J. Zhang, X. Liu, X. Ye, F. Zhao, Y. Zhang, M. Wu, Y. Zhang, L. Xu, and J. Yu, “Editable free-viewpoint video using a layered neural representation,” _ACM Transactions on Graphics (TOG)_ , vol. 40, no. 4, pp. 1–18, 2021.
  * [40] C. Wang, M. Chai, M. He, D. Chen, and J. Liao, “Clip-nerf: Text-and-image driven manipulation of neural radiance fields,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR_ , 2022, pp. 3835–3844.
  * [41] P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang, “Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction,” _arXiv preprint arXiv:2106.10689_ , 2021.
  * [42] L. Yariv, J. Gu, Y. Kasten, and Y. Lipman, “Volume rendering of neural implicit surfaces,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 4805–4815, 2021.
  * [43] T. Müller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” _ACM Trans. Graph._ , vol. 41, no. 4, pp. 102:1–102:15, 2022.
  * [44] A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su, “Tensorf: Tensorial radiance fields,” in _European Conference on Computer Vision, ECCV_ , vol. 13692, 2022, pp. 333–350.
  * [45] A. Moreau, N. Piasco, D. Tsishkou, B. Stanciulescu, and A. de La Fortelle, “Lens: Localization enhanced by nerf synthesis,” in _Conference on Robot Learning_ , 2022, pp. 1347–1356.
  * [46] Y. Ge, H. Behl, J. Xu, S. Gunasekar, N. Joshi, Y. Song, X. Wang, L. Itti, and V. Vineet, “Neural-sim: Learning to generate training data with nerf,” in _European Conference on Computer Vision, ECCV_ , 2022, pp. 477–493.
  * [47] M. A. Fischler and R. C. Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,” _Communications of the ACM_ , vol. 24, no. 6, pp. 381–395, 1981.
  * [48] Y. Liu, W. Lai, M. Yang, Y. Chuang, and J. Huang, “Hybrid neural fusion for full-frame video stabilization,” in _IEEE/CVF International Conference on Computer Vision, ICCV_ , 2021, pp. 2279–2288.
  * [49] C. Sun, M. Sun, and H. Chen, “Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR_ , 2022, pp. 5449–5459.
  * [50] P. Dai, Y. Zhang, X. Yu, X. Lyu, and X. Qi, “Hybrid neural rendering for large-scale scenes with motion blur,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR_ , 2023, pp. 154–164.
  * [51] J. Straub, T. Whelan, L. Ma, Y. Chen, E. Wijmans, S. Green, J. J. Engel, R. Mur-Artal, C. Ren, S. Verma, _et al._ , “The replica dataset: A digital replica of indoor spaces,” _arXiv:1906.05797_ , 2019.
  * [52] F. Xia, A. R. Zamir, Z. He, A. Sax, J. Malik, and S. Savarese, “Gibson env: Real-world perception for embodied agents,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR_ , 2018, pp. 9068–9079.
  * [53] S. K. Ramakrishnan, A. Gokaslan, E. Wijmans, O. Maksymets, A. Clegg, J. Turner, E. Undersander, W. Galuba, A. Westbury, A. X. Chang, _et al._ , “Habitat-matterport 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AI,” _arXiv:2109.08238_ , 2021.
  * [54] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR_ , 2017, pp. 5828–5839.
  * [55] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, _et al._ , “Habitat: A platform for embodied AI research,” in _IEEE/CVF International Conference on Computer Vision, ICCV_ , 2019, pp. 9339–9347.
  * [56] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR_ , 2018, pp. 586–595.
  * [57] R. Arandjelovic, P. Gronat, A. Torii, T. Pajdla, and J. Sivic, “Netvlad: Cnn architecture for weakly supervised place recognition,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR_ , 2016, pp. 5297–5307.
  * [58] Z. Li, Q. Wang, F. Cole, R. Tucker, and N. Snavely, “Dynibar: Neural dynamic image-based rendering,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR_ , 2023, pp. 4273–4284.