of total capacity, remains stable between £70/MWh and £100/MWh. This range may
be due to the stochasticity in calculating the weights for the DDPG algorithm.
The average electricity price does not change over the time steps or training.
We, therefore, hypothesise that there is no market power as long as an
individual GenCo owns below ${\sim}35\%$ of total electrical capacity.

On the other hand, once the capacity of a GenCo or groups of GenCos is above
30,000MW, there is a significant increase in the average price for capacity.
The average electricity price for capacity falls between the range of
£$170$/MWh and £$220$/MWh.

Figure 6.4 displays the capacity controlled by the agents that use the RL
strategy versus the average electricity price for the unbounded case. The
colour displays the number of steps. The step-change, as shown in Figure 6.2
can be seen clearly here, with agents with a capacity larger than
${\sim}$25,000MW causing a step change in electricity price. Electricity
prices seems to cluster below ${\sim}$10,000MW. However, after this point, the
average electricity price begins to increase.

Figure 6.4: Capacity of agents using RL vs. average electricity price
accepted, for unbounded agents.

Figure 6.3 shows a cluster between ${\sim}$£$60$/MWh and ${\sim}$£$80$/MWh
irrespective of the capacity of the agents. This is verified by Figure 6.5.
This seems to suggest that setting a lower market cap reduces the ability for
generators, irrespective of size, from influencing the electricity price.

Figure 6.5: Capacity of agents using RL vs. average electricity price
accepted, for bounded agents.

Figures 6.6 and 6.7 display the actual bids made at the end of training within
the electricity market for all of their power plants. The number of bids made
by each GenCo changes dependent on the number of plants that they own, with
the Orsted GenCo only making eleven bids per segment, and the largest group
making 216 bids per segment. Figure 6.6 displays the uncapped scenario
(£600/MWh) and 6.7 displays the capped scenario (£150/MWh).

Figure 6.6(a) shows the bids made by the largest group of GenCos as shown in
Table 6.1. A bimodal distribution can be seen, where the group of GenCos tend
to bid either the maximum or the minimum bid. We hypothesise that they bid the
maximum amount as this ensures that the market price is artificially raised,
and that they are able to utilise their market power. The minimum price is bid
the rest of the time to ensure that generators bids are always accepted,
regardless of whether the market price has been artificially raised or not in
each respective clearing segment.

Figure 6.6(b) displays the bids of the small company with a market cap of
£600/MWh. The small company also seems to have a bimodal distribution;
however, bids the higher price more often. This may be due to the fact that it
is able to influence the price at certain market segments, and the reward of
the higher accepted reward outweighs the times in which it is not accepted on
the market segments. Again, bidding low seems to be the strategy in which to
take if the GenCo does not believe it will be able to influence the final
price. As the market simulated is a uniform pricing market, having a £0 bid
accepted does not mean that the GenCo will be paid £0. Rather, the GenCo will
be paid the market clearing segment, which is set by the most expensive power
plant accepted onto that market segment.

Figures 6.7(a) shows the bids made by the largest group of GenCos in each
market segment. It seems to take a similar strategy to that of the large
company in the uncapped scenario, as shown in Figure 6.6(a). We believe, as
similar in the uncapped scenario, that this is due to the market power that
this group possesses. Being able to influence the market price enables the
GenCo group to inflate the prices. It also takes a conservative strategy to
bid the minimum price allowed, to ensure that the bid is accepted regardless
of price.

Figure 6.7(b) displays the strategy of the smallest company in the capped
scenario. Here, it seems that the GenCo is unable to influence the price at
all, and therefore bids £0/MWh for the majority of the time. This is similar
to the expected strategy of GenCos, who tend to bid their short-run marginal
cost to ensure that they do not miss out on potential profit. The short-run
marginal cost can often change based upon fuel, carbon and generator type.
However, for renewable energy, it is near £0, and for fossil-fuel based plants
it is near the cost of fuel and carbon at that point in time.

(a) Largest group of GenCos with a total controlled capacity of 46929.2MW with
a market cap of £600/MWh.

(b) Smallest company with a total controlled capacity of 2738.7MW with a
market cap of £600/MWh.

Figure 6.6: Bids made by generator companies using Reinforcement Learning with
a market cap of £600/MWh.

(a) Largest group of GenCos with a total controlled capacity of 46929.2MW with
a market cap of £150/MWh.

(b) Smallest company with a total controlled capacity of 2738.7MW with a
market cap of £150/MWh.

Figure 6.7: Bids made by generator companies using Reinforcement Learning with
a market cap of £150/MWh.

We ran a sensitivity analysis to observe the effects of the market cap on
final average accepted bid price. The largest GenCo group used a strategy for
this sensitivity analysis. Figure 6.8 displays the results. It seems that
whilst the average accepted bid price increased with the capped bid level;
there is a significant increase after a market cap of £190/MWh. This may be
due to the case that the GenCos begin to outbid the SRMC bidding GenCos at
this price point.

Figure 6.8: Capacity of agents using RL vs. average electricity price
accepted, for bounded agents.

These results demonstrate a methodology for policy makers to investigate the
ability for generator companies to artificially increase the electricity
price. It gives quantitive advice on the level of capacity that a generator
company can control before it becomes a problem from the perspective of
consumers of the electricity market. These results also provide a method to
control excessive electricity prices with the provision of a market cap.

The limitations from the perspective of the policy maker, is that we do not
provide any information on how to prevent an oligopoly from occurring. We only
provide advice on what to do once one has occurred.

###  6.6 Discussion

Our results demonstrate the ability for GenCos to artificially increase the
electricity price through market power in an uncapped market. Our results have
shown that in an uncapped market, any single agent or groups of agents who
make bids using the same strategy and information, should have less than
${\sim}$10,000MW. This defines the optimal capacity of any single GenCo to
have a fair level of competition. After this, the electricity price begins to
rise with the same outcome and welfare. It is also worth noting that when the
market is capped, the average accepted price does not simply become the capped
price.

However, if there is an electricity market with a few large or colluding
players, it is possible to remove their advantage through the introduction of
a price cap. Our results show that whilst average accepted bids increase with
market cap level, the value does not increase significantly until £$200$ is
reached.

This information and approach can help to inform government policy to ensure
fair competition within electricity markets, as well as run the model for
their own scenario. It is hypothesised that the findings in this work are
generalisable to other decentralised electricity markets in other geographies
due to their similar market structures. Whilst the figures presented here may
not be the same; we hypothesise that the region of interest will be similar.

###  6.7 Conclusion

In this work, we investigated the ability for GenCos to make strategic bids
within an electricity market. We did this using the deep deterministic policy
gradient (DDPG) reinforcement learning method. We utilised the agent-based
model ElecSim to model the UK electricity market. We utilised the DDPG
algorithm only for a certain subset of agents, from small individual
generation companies (GenCos) to large groups of GenCos.

This enabled us to explore the ability for GenCos with a large capacity to
artificially increase the price in the electricity market within the UK if
they are in control of a sufficiently large generation capacity. Our results
show that the optimum level of control of any single GenCo or groups of GenCo
is below ${\sim}$10,000MW or ${\sim}$11% of the total capacity. Above this,
prices begin to increase with no real additional benefit to the consumer.
After ${\sim}$25,000MW, or ${\sim}$35% of the total capacity, the prices begin
to increase substantially, to ${\sim}$£200, over triple the original cost
without this market power. The introduction of a market cap of £$150$ reduces
all market power and maintains electricity price at a reasonable level.

We found through a sensitivity analysis that the average electricity price in
the market over a year remains low with a price cap smaller than £190.
However, after this level, the average electricity price begins to increase.

Our work has shown the ability for reinforcement learning to learn an optimal
bidding strategy to maximise a GenCo’s profit within an electricity market.
The ability for GenCos to use their market power is also highlighted, and is
dependent on electricity generation capacity of the respective GenCo.

In future work, we would like to enable GenCos to withhold the capacity on
offer to the electricity market. This would enable further market power by
reducing competition further. Additionally, we would like to assess the market
power in different countries with different market structures and total
electricity supply. An other option would be to assess the effect of two large
competing GenCos on the market.

##  Chapter 7 Conclusion

###  7.1 Thesis summary

In this thesis, we presented an open-sourced, long-term agent-based model for
electricity markets called ElecSim. We were able to validate the model through
two methods: cross-validation using historic data from 2013 and projecting
forward, and comparing our scenario to that of the UK Government to 2035. We
found positive results in both: we were able to accurately model the
transition from coal to gas between 2013 and 2018, and closely match the
scenario of the UK Government.

For this validation, we used optimisation to find realistic parameters that
would generate the desired scenarios. The primary parameter that was optimised
was the predicted price duration curve. We found a predicted price duration
curve which closely matched the price found in 2018 for the cross-validation
using historic data. For the UK government forward scenario, we found a
variety of predicted price duration curves, with a few boom and bust cycles in
the price of electricity.

In addition to designing and developing ElecSim, we used the forward scenario
generated to optimise a carbon tax strategy. That is, to reduce both carbon
emissions and electricity price from 2018 to 2035. To achieve this we used a
multi-objective genetic algorithm. We found that we were able to reduce both
electricity price and carbon emissions through a combined use of solar,
nuclear and onshore energy.

Further, we predicted electricity demand consumption on two time scales:
30-minutes ahead and a day-ahead to use in ElecSim. We found that we were able
to predict electricity demand at these time scales well. Through the use of an
online learning method, we were able to reduce the required reserve capacity
of the national grid in the UK. We also looked at the long-term impact of poor
forecasting errors on the long-term market, and found that poor forecasts led
to a high carbon density of electricity grid over the long-term.

Our final chapter focuses on the use of deep reinforcement learning to model a
bidding strategy of a single generation company or group of generation company
using ElecSim. The rest of the generation companies bid based upon the
respective generator’s short run marginal cost. We found that after a capacity
controlled by a generation company of 10,000MW, the generation company with
the reinforcement learning bidding strategy was able to influence electricity
prices in their favour; effectively demonstrating market power within the
electricity market. After a controlled capacity of 30,000MW the average
electricity price triples from the equilibrium price.

We utilised agent-based models in this work for several reasons:

  1. 1.

Ability to model scenarios in a way that emerges from the generator companies.

  2. 2.

The process which emerge a not a black box, unlike in equilibrium models.

  3. 3.

Optimisation based methods should be interpreted in a normative manner, for
example, how policy choices should be carried out. Whereas agent-based models
make no assumption on outcomes of scenarios.

  4. 4.

Ability to model decentralised agents with different desires.

  5. 5.

Ability to model outcomes which are not in equilibrium, for example boom and
bust cycles.

An additional reason was for the limited availability of an open-sourced
agent-based model, and to ensure that this model is available to the
community. Open-sourced models allow for transparency, and to garner greater
acceptance and understanding within the wider community.

Our contributions are the following:

  1. 1.

Developed a novel, open-source agent-based model for the electricity market
called ElecSim. ElecSim and all related code can be accessed at:
https://github.com/alexanderkell/elecsim.

  2. 2.

Validated this model by the means of cross-validation, and compared our model
to that of the UK Government.

  3. 3.

Investigated the effect of online learning to improve electricity demand
forecasting a day-ahead, and looked at the long-term impacts that this had on
the electricity markets.

  4. 4.

Predicted electricity consumption 30-minutes ahead.

  5. 5.

Found a variety of optimal carbon tax strategies using genetic algorithm based
optimisation and the ElecSim model.

  6. 6.

Investigated the impact of collusion within a oligopolistic electricity
market.

###  7.2 Conclusions

In this section we explore the conclusions that arose from this thesis. We
explore each of the chapters where we present our primary technical
contributions. We place our work in the context of our research question: what
challenges can AI and ML tackle, and how do these methods relate back to the
wider energy system? Each of the chapters explore a research subquestion,
which are highlighted in Chapter 1. We provide answers to these subquestions
here.

####  7.2.1 ElecSim Model

Can a simulation model an electricity market over the long-term? Through the
recreation of the salient features of the UK electricity market into an agent-
based model, we show that we are able to derive similar outputs to an
established, governmental model. Therefore, even though the underlying
methodology between these two models are different, we have shown that it is
possible to reach similar conclusions.

This has important ramifications for future work, as features that could not
previously be modelled with traditional, optimisation based models, such as
heterogenous generation companies, can be modelled. This finding adds a
significantly different tool to the established models, and allows modellers
and policy makers to make findings from an alternative kind of methodology.
Whilst this does not provide a ground truth model to the energy modelling
community, it does provide an additional tool to understand complexity within
an electricity market.

Is it possible to model the variability of an electricity system? The
variability within a modern day electricity system is significant. With the
increase in solar photovoltaics and wind turbines a modern day electricity
model must be able to model fluctuations in solar irradiance and wind speeds.
This variability can change in minutes and therefore adds significant
complexity for electricity system models. This is because a high temporal
resolution is required to model this variability.

To take into account this variability, we used a k-means clustering approach
to select representative days in Chapter 3. Through the use of these
representative days, we found that we were able to model significant peaks and
troughs in demand, wind speed and solar irradiance. This was shown by
comparing three different metrics: the normalised root mean squared error,
correlation and relative energy error. This was demonstrated in Chapter 3. We
recommend that a similar approach is undertaken by other modellers for a
realistic representation of the real-world, without compromising on runtime.

Can we trust an electricity model’s outputs? Fundamentally, the ability to
trust an electricity model’s outputs is through the verification of its
outputs with observed data. For long-term models this becomes a significant
challenge because of all the uncertainties inherent in long-term investments.
Further, a long-term electricity model produces scenarios and not predictions.
This makes this problem even more difficult, as an output scenario could be
both possible and realistic, but it does not mean that it is a certainty that
it will occur.

To answer the subquestion of whether we can trust an electricity model’s
outputs, we investigated an observed shift in electricity mix from coal to gas
from the previous five years. We calibrated our model to find parametric data
that would generate the outputs required. We found that the parametric data
required to generate these outputs were realistic, and we could model the
shift in electricity mix from coal to gas over this time period.

Whilst this approach has limitations, in that it raises an additional
question: can short-term validation translate to long-term verification?
However, this is the best solution we found without trying to predict many
years in the future which is both infeasible and unrealistic.

From this work, we recommend that models undertake some form of validation
using real-world, observed data. Without this, it is possible that some
aspects of the real-world system is not accurately modelled. And so, an
iterative process can be used to improve the functioning of the model, with
this approach.

####  7.2.2 Electricity Demand Prediction

Do poor short-term forecasts of electricity demand have a long-term impact?
Previous work on short-term forecasts of electricity demand has focused on
improving the accuracy of such models. However, the long-term impact of these
predictions has been studied to a lesser degree. To explore this question, we
used a variety of different statistical and machine learning techniques to
predict electricity demand. We then took the errors of these predictions and
investigated the impact that these errors have on the long-term electricity
market by perturbing the electricity demand perceived by the generator
companies.

Our results show that with an increase in mean absolute error, there is an
increase in CCGT, coal, nuclear, photovoltaics and reciprocal gas engines,
with a decrease in onshore and offshore wind. Such a scenario changes the
electricity mix landscape in the future, and is suboptimal if a low-carbon
electricity mix is desired.

We recommend that a high amount of research is placed into improving short-
term forecasting for electricity demand. Whilst this can be expensive in the
short-term, the long-term benefits are significant through the use of a more
optimal and efficient system.

####  7.2.3 Carbon Optimization

Is it possible to use an algorithm to set carbon policy? A significant goal
for policy makers is to achieve a low-cost and and low-carbon system. However,
in practice, this can be a difficult system to achieve. Policy makers have
limited levers in which to test, and those levers can have an uncertain impact
on the market.

To simplify the process of finding a cost and carbon optimal carbon tax in the
UK electricity market, we used a multi-objective genetic algorithm, NSGA-II,
to vary carbon tax between 2018 and 2035. We found that it is possible to use
a genetic algorithm to find a variety of carbon policies to minimise both
carbon and cost of the electricity system. These carbon policies can be
presented to policy makers to choose from, depending on their requirements and
preferences.

The recommendations that arise from this work is that a full quantitative
analysis should be undertaken for long-term policy options, and that the
search space can be explored using algorithms. This enables policy makers to
find a variety of different options, from which they can choose from. Whilst
the algorithm may not tell you the precise strategy, it can give a range of
feasible options, in which policy makers can use their judgement with
additional information.

####  7.2.4 Strategic Bidding

Is it possible to limit the power of large generator companies? Electricity
markets can often be dominated by a small subset of generator companies. If
given enough power, these generator companies can then use strategic bidding
to artificially inflate the electricity price for their own benefit.

With the use of reinforcement learning, we were able to simulate the strategic
bidding behaviour that generator companies may exhibit. We found that if a
generator company controls more than 11% of total capacity, it is likely that
price inflation will occur. However, we found that such price inflation can be
limited through the means of a cap on bid levels. We found that a relatively
high bid price cap of £190/MWh prevents price inflation for the UK market.

We therefore recommend that policy makers place a price cap if they believe
that an oligopoly or monopoly is beginning to form. This can limit the impact
that these companies can have on consumers and the overall market.

####  7.2.5 AI and ML in a Wider Electricity Market Context

What challenges can AI and ML tackle, and how do these methods relate back to
the wider energy system? In this thesis, we explored the different methods
that AI and ML can tackle within the British electricity system. Whilst much
work has focused on incremental improvements to targeted problems, such as
short-term load forecasting or the possibility of using reinforcement learning
for short-term bidding, the wider impact on the market has been explored to a
lesser extent.

We found that AI and ML can be used for a wide range of different application
domains, which are not limited to solely improving the accuracy of these
domains. The subquestions answered previously in this section attest to this,
and demonstrate that large-scale problems and systems can be optimised,
improved and modelled with the use of state-of-the-art methods.

###  7.3 Future research direction

There remains significant research that can be undertaken as part of this work
that we would like to carry out in the future. A major aspect of intermittent
renewable energy sources such as wind and solar is their distributed nature.
On a geographic scale as large as the UK, weather conditions change.
Therefore, at any one time, capacity factors may change across the country. It
would, therefore, be beneficial to model this, by integrating a higher
temporal resolution into the model. This would increase the complexity of the
model, and therefore compute time, however, the optimal placement of
intermittent renewable energy sources could be modelled to ensure the maximum
supply at times when it was required from such sources. Work could be done to
find simplifications that could be made to other aspects of the model to
maintain tractability.

Additionally, the integration of a larger amount of countries could be carried
out. For instance, coupling the UK market with that of Ireland, or with the
EU. Whilst this would increase the compute time, the results would be more
inclusive of other markets and enable for a larger temporal resolution still.
Another area of interest is predicting the price of electricity in the future.
Scenarios could be run to predict this price and integrate it into the model.
Whilst increasing compute time, this would allow us to explore a wider range
of scenarios.

Another aspect is the comparison to further models. The traditional approach
of optimisation models offer a variety of “gold-standard” scenarios that we
could explore. These comparisons could be undertaken to find weaknesses in
both our model as well as other models.

An interesting area of research is the area of investment using optimisation
or control algorithms such as reinforcement learning. Reinforcement learning
suffers from a requirement to have multiple training epochs, and therefore we
found this a difficult task to achieve in this work. However, through either
speed improvements or use of another algorithm, this could be integrated into
our work.

Finally, the integration of fuel supply-demand curves could help us in better
modelling electricity markets on both a local and global scale. Firstly, land
is a limited resource, especially in the United Kingdom, and therefore limits
to the amount of solar or wind that can be produced a real. Secondly, on a
global scale, as demand for gas or coal reduces there are large effects on
price. Firstly this may reduce due to the supply and demand relationship.
However, if demand reduces sufficiently, the price may increase, due to the
expense of extraction. This may turn out to be a large inflection point, where
gas and coal are used much less.


# ONeRF: Unsupervised 3D Object Segmentation from Multiple Views

Shengnan Liang1 Yichen Liu111footnotemark: 1 Shangzhe Wu3 Yu-Wing Tai1,2 Chi-
Keung Tang1
1The Hong Kong University of Science and Technology 2Kuaishou Technology
3University of Oxford
Equal contribution. The order of authorship was determined alphabetically.

###### Abstract

We present ONeRF, a method that automatically segments and reconstructs object
instances in 3D from multi-view RGB images without any additional manual
annotations. The segmented 3D objects are represented using separate Neural
Radiance Fields (NeRFs) which allow for various 3D scene editing and novel
view rendering. At the core of our method is an unsupervised approach using
the iterative Expectation-Maximization algorithm, which effectively aggregates
2D visual features and the corresponding 3D cues from multi-views for joint 3D
object segmentation and reconstruction. Unlike existing approaches that can
only handle simple objects, our method produces segmented full 3D NeRFs of
individual objects with complex shapes, topologies and appearance. The
segmented ONeRFs enable a range of 3D scene editing, such as object
transformation, insertion and deletion.

##  1 Introduction

Neural Radiance Fields (NeRF) [29] have recently been becoming the mainstream
approach for novel view synthesis, given its excellent performance for complex
real-world scenes despite their simplicity. The key idea is to represent the
entire scene as a radiance field parametrized using an MLP, taking in $xyz$
coordinates and viewing directions as input and produces densities and view-
dependent colors. Since the first contribution [29], NeRfs have been extended
in various dimensions, for example, with improved efficiency [48, 8], improved
quality [52, 1], deformation [30, 31], uncalibrated images [42, 15], sparse
views [14, 5], better generalization [37, 49], material decomposition [3, 53],
as well as many other applications [12, 25].

Figure 1: 3D slot attention. Coarse 2D masks are iteratively refined with 3D
object NeRFs in an unsupervised manner. The output are individual NeRFs for
each object.

However, most of the existing work focuses on novel view rendering. Less
effort has been made to leverage this representation for the fundamental
computer vision problem of 3D object segmentation, which is the goal of this
paper. Notably, by obtaining object-level NeRF representations of a 3D scene,
various 3D scene editing tasks can be enabled, including object editing,
insertion, removal and completion. Several recent works have attempted to
learn category-specific object NeRFs from multi-view images and optional depth
information, allowing for single-image inference [39, 50]. However, these
methods mainly work with specific object categories, such as CLEVR [18] and
ShapeNet [43] objects, and is not capable of segmenting arbitrary objects in
general scenes. Others rely on additional information, such as 2D segmentation
[45] and 2D bounding box annotations [16], and lift them to 3D. NeuralDiff
[40] on the other hand leverages motion cues in egocentric videos to separate
the foreground layer of moving objects and the static background layer, but
cannot obtain full 3D segmentations of individual foreground objects. In
contrast, our ONeRFs consist of an unsupervised approach that performs
automatic 3D object segmentation of arbitrary scenes from multiple views,
without relying on any external annotations, which can be regarded as a
generalized version of some of the above category-specific methods.

Given multi-view images of a scene, our goal is to detect individual objects
and represent each one of them using a separate NeRF. In order to do this, we
introduce the notion of 3D slot attention, which is an unsupervised method for
fusing 2D visual features and 3D geometrical information inherent from
multiple views for segmenting the scene into individual 3D object NeRFs.
Specifically, both the 2D features and 3D information extracted from multiple
views are used to produce initial object and background masks, which will be
iteratively refined via expectation-maximization to segment individual NeRFs
and thus 3D objects. Unlike previous works, our unsupervised ONeRFs
segmentation can handle objects with complex textures, geometries and
topologies, which support not only novel view synthesis but also scene editing
such as object insertion, deletion, and modification with individual ONeRFs
automatically learned from a given scene.

While the qualitative results have significant room for improvements as in
other related state-of-the-arts, this paper makes the first attempt to
contribute a significant baseline for unsupervised learning on segmenting a
cluttered scene into individual ONeRFs from multiple views. Our qualitative
and quantitative results have validated the soundness of our technical
approach. Preliminary results on scene editing shows the great potential of
ONeRFs.

##  2 Related Work

###  2.1 Multi-View 3D Reconstruction

Multi-view stereo (MVS) is a classical computer vision problem for 3D
reconstruction from images captured from multiple viewpoints. Traditional
methods include [2, 23, 22, 10]. More recent deep learning-based MVS methods
are MVSNet [47], P-MVSNet [28], and CVP-MVSNet [46]. Once reconstructed, the
3D scene can be rendered at any viewpoints. However, the output of MVS
consists of (quasi) dense point clouds and thus are not object aware.
Combining deep MVS technique with differentiable volume rendering, MVSNeRF [4]
reconstructs a neural radiance field from unstructured multi-view input images
for novel view synthesis, which generalizes well across scenes using MVS. View
synthesis from deep lightfield [19] has also been proposed. While novel views
can be synthesized, the pertinent NeRF has little object awareness.

###  2.2 Object Segmentation

In our unsupervised approach, 2D object masks estimated from multiple views
must be consistent with the underlying 3D voxel volume. 3D visual hulls [36]
can be estimated from accurate 2D object masks given in the input calibrated
images. Conversely, accurate segmentation of 3D point clouds/voxels can
produce consistent 2D object masks across views. Segmentation of 3D point
cloud/voxels is an active research area, including PointNet [32], DGCNN [41]
and [34], and later on large-scale point cloud segmentation such as PointNet++
[33] and [26, 13, 44]. Recent works include weakly supervised segmentation
[54, 56], semi-supervised segmentation [17], few shot segmentation [55],
instance segmentation [51], and semantic segmetation [35]. While the above can
be deployed for 3D NeRF segmentation on the pertinent voxel volume, ONeRFs is
an end-to-end unsupervised approach for object NeRFs segmentation taking only
images as input.

There is also another line of work on unsupervised object-centric learning,
which is learning-based and decomposes a single image into different objects,
mostly in 2D: e.g., GENISIS [9], SPACE [24], SPAIR [7], IODINE [11]),
SlotAttention [27]. Most of these work with very simple CLEVR-like objects.

Figure 2: Overall framework. Given a set of images of a scene and their camera
poses, we use K-Means to obtain coarse foreground object regions according to
the VGG feature and 3D position of each pixel. Then we obtain the object masks
by using an unsupervised object segmentation convolutional neural network, and
then use these masks to train NeRF networks for individual objects. Combining
the masks generated by NeRF and the masks improved by expectation maximization
(EM), we get the refined objects masks, which will be used to train NeRF
models in next iteration.

##  3 Method

ONeRFs is an end-to-end unsupervised approach for segmenting 3D object NeRFs
from multiple views of the underlying scene. Borrowing attention typical of
deep neural networks, our 3D attention slots operates by cooperating 2D visual
and 3D geometrical cues, so that they can reinforce each other in alternating
optimization where respective errors will be automatically corrected via
resolving their inconsistencies. Figure 2 shows the overall framework, where
starting coarse 2D masks are working in tandem with 3D reconstruction from
NeRFs to iteratively achieve accurate object NeRFs segmentation.

###  3.1 Initial coarse masks generation

Clustering. Given the input images of the scene and camera poses from multiple
viewpoints, we resize the images to 128 $\times$ 128 and extract VGG features
[38] and reconstruct 3D positions of all the pixels, which are clustered into
two classes using $K$-Means clustering: foreground and background. Let

$\displaystyle s^{f}_{i}$ $\displaystyle=$
$\displaystyle||\mathbf{f}_{i}-\boldsymbol{\mu}_{f}||+w||P_{i}-\mathbb{p}_{f}||,$
(1) $\displaystyle s^{b}_{i}$ $\displaystyle=$
$\displaystyle||\mathbf{f}_{i}-\boldsymbol{\mu}_{b}||,$ (2)

where $s_{i}^{f}$ and $s_{i}^{b}$ are respectively the foreground and
background scores of pixel $i$, $\mathbf{f}_{i}$ is the VGG feature, $P_{i}$
are the 3D coordinates of corresponding pixel $i$, which can be obtained from
multi-view reconstruction from the input images. $\boldsymbol{\mu}_{f}$ and
$\boldsymbol{\mu}_{b}$ are respectively the mean foreground and background
clustered features, $\mathbb{p}_{f}$ are the 3D coordinates corresponding to
the clustered foreground points, and $w$ is the weight.

Assuming sufficiently far background, background pixels will have large
variance in 3D locations. Thus foreground pixels tend to have smaller
$||P_{i}-\mu_{f}||$. Then we update the three means by

$\displaystyle\boldsymbol{\mu}_{f}$ $\displaystyle=$
$\displaystyle\frac{\sum_{i:s_{i}^{f}<s_{i}^{b}}\mathbf{f}_{i}}{\\#},\mathbb{p}_{f}=\frac{\sum_{i:s_{i}^{f}<s_{i}^{b}}P_{i}}{\\#},$
(3) $\displaystyle\boldsymbol{\mu}_{b}$ $\displaystyle=$
$\displaystyle\frac{\sum_{i:s_{i}^{f}\geq s_{i}^{b}}\mathbf{f}_{i}}{\\#},$ (4)

where #’s correspond to the relevant number of foreground and background
pixels. Finally we get the foreground and background masks $M_{f}^{0}$,
$M_{b}^{0}$, and upsample them into the original size.

Image Segmentation and Instances. After labeling the pixels corresponding to
the 3D foreground clustered above, we apply unsupervised segmentation [21] to
produce $n+1$ parts ${\cal P}_{0},\cdots,{\cal P}_{n}$. ${\cal P}_{0}$ is the
background we separate in the first step, ${\cal P}_{1}$ is the background
near the foreground objects which cannot be precisely clustered in the first
step and the others correspond to the foreground objects, denoted as ${\cal
P}_{2},\cdots,{\cal P}_{n}$.

Suppose there are $k$ objects in expectation. We then discard $n-(k+1)$ of
${\cal P}_{1},\cdots{\cal P}_{n}$ occupying the smallest areas, i.e. those
classes with fewest pixels, which are very likely due to incorrect clustering
or noise.

We then put the full image into the model in image segmentation step to obtain
the response vector map of the image [21], where the response vector at a
given pixel represents the clustering scores for all the $n$ classes (minus
the weak ones rejected as above). The pixels in the same class tend to have
similar response vectors. Here, we cluster the pixels by the highest entry in
the response vector. The response vectors of pixels in ${\cal P}_{0}$ are
similar to those in ${\cal P}_{1}$. We sample some pixels in ${\cal P}_{i}$
and find the mean of the response vectors of those pixels, denoted as
$\mathbf{n}_{i}$. Then, ${\cal P}_{i}$ with $\mathbf{n}_{i}$ that is closest
to $\mathbf{n}_{0}$ is the remaining background to be separated from the
coarse foreground.

###  3.2 Train NeRFs and Refine Masks

Training. Given a 3D position $\mathbb{x}$ and view direction $\mathbb{d}$,
the network will output the RGB color $\mathbb{c}$ and volume density $\sigma$
of that point [29]. Following [45], for each segmented object $a$ the loss is

$\lambda_{1}||\hat{C}_{a}(\mathbb{r})-C(\mathbb{r})||\odot
M_{a}+\lambda_{2}||\hat{A}_{a}(\mathbb{r})-M_{a}||$ (5)

where $\mathbb{r}$ is the ray, $\hat{C}_{a}(\mathbb{r})$ is the RGB triplet by
the NeRF of object $a$, $C(\mathbb{r})$ is the ground truth RGB, $M_{a}$ is
the mask of object $a$, and $\hat{A}_{a}(\mathbb{r})$ is the accuracy map by
NeRF of object $a$, and $\lambda_{1},\lambda_{2}$ are weights.
$C_{a}(\mathbb{r})$ and $A_{a}(\mathbb{r})$, which follows [29], are defined
as

$\displaystyle\hat{C}_{a}(\mathbb{r})$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{N}T_{i}^{a}\alpha_{i}^{a}\mathbb{c}_{i}^{a}$ (6)
$\displaystyle\hat{A}_{a}(\mathbb{r})$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{N}T_{i}^{a}\alpha_{i}^{a}$ (7) $\displaystyle
T_{i}^{a}$ $\displaystyle=$
$\displaystyle\sum_{j=1}^{i}\exp(-\sigma_{j}^{a}\delta_{j})$ (8)
$\displaystyle\alpha_{i}^{a}$ $\displaystyle=$ $\displaystyle
1-\exp(-\sigma_{j}^{a}\delta_{j})$ (9)

We sample $N$ points along the ray $\mathbb{r}$ and the distance between the
$i$-th point and the camera is $\delta_{i}$, which will increase with the
increase of the index. For point $i$ in object $a$, we obtain the RGB color
$\mathbb{c}_{i}^{a}$ and volume density $\mathbb{\sigma}_{i}^{a}$ at that
point from the network. Then by Equation (6) and (7), we obtain the rendered
color and accuracy masks along that ray.

The color within the in-mask region will converge to the ground-truth RGB
color. For $\hat{A}_{a}$, in-mask region will converge to 1 while regions
outside to 0. As NeRF needs to guarantee 3D consistency, occlusion and
incorrect masking in the last step can be handled automatically.

Refine Masks. The masks are iteratively refined by leveraging the object NeRFs
for each individual object instance, in the presence of occlusion if any. We
use the occlusion accuracy masks $\hat{A}_{a}^{\prime}(\mathbb{r})$ as the
occlusion masks from NeRF networks. Specifically, each new object mask is
generated by

$\displaystyle\hat{A}_{a}^{\prime}(\mathbb{r})$
$\displaystyle=\sum_{i=1}^{N}T_{i}\alpha_{i}^{a}$ (10) $\displaystyle T_{i}$
$\displaystyle=\sum_{j=1}^{i}\exp(-\sigma_{j}\delta_{j})$ (11)
$\displaystyle\sigma_{j}$ $\displaystyle=\sum_{o}\sigma_{j}^{o}$ (12)

where $\sigma_{j}$ is the summation of volume densities from NeRF networks of
all objects. We only consider the contribution of the other NeRFs on $T_{i}$,
that is, decreasing values of $\alpha$ along the ray $\mathbb{r}$. Finally,
expectation-maximization (EM) is employed to refine the input mask. We use the
response vector map again. We initialize the $\mu$ as

$\displaystyle\mathbb{\mu}_{i}^{(0)}=\frac{\sum_{x_{j}\in
M_{i}}\mathbb{v}(\mathbb{x}_{j})}{\\#}$ (13)

where $M_{i}$ is the mask, $\mathbb{v}_{j}$ is the response vector and #’s
correspond to the number of pixels in $M_{i}$. Then EM is performed on the
pixels of background. At the $t$-th iteration, for each response vector of
background $\mathbb{v}_{n}$, we formulate the posterior probability of
$\mathbb{v}_{n}$ given $\boldsymbol{\mu}_{i}$ as

$\displaystyle
p(\mathbb{v}_{n}|\mathbb{\boldsymbol{\mu}}_{i}^{(t)})=\exp(-||\mathbb{v}_{n}-\boldsymbol{\mu}||_{2}^{2}),$
(14)

where $\boldsymbol{\mu}_{i}^{(t)}$ is the $\boldsymbol{\mu}$ value at the
$t$-th iteration. The latent variable $\mathbb{z}_{n}^{(t)}$ for
$\mathbb{v}_{n}$ is

$\displaystyle
z_{n,i}^{(t)}=\frac{p(\mathbb{v}_{n}|\boldsymbol{\mu}_{i})}{\sum_{k=1}^{K}p(\mathbb{v}_{n}|\boldsymbol{\mu}_{k})},$
(15)

where $z_{n,i}^{(t)}$ is the $i$-th entry of $\mathbb{z}_{n}^{(t)}$ and $K$ is
total number of classes, which is the number of objects plus one (the
background). Then we update the $\boldsymbol{\mu}$ value of the next iteration
as

$\displaystyle\mathbb{\boldsymbol{\mu}}_{i}^{(t+1)}=\frac{\sum_{n=1}^{N}z_{n,i}\mathbb{v}_{n}}{\sum_{n=1}^{N}z_{n,i}}$
(16)

Then we output the softmax results upon the above EM convergence. Let $M_{re}$
be the resulting mask, then the final mask is given by

$w_{1}M_{re}+w_{2}\hat{A}_{a}^{\prime}>0.5$ (17)

for some weights $w_{1},w_{2}$.

##  4 Experiments

###  4.1 Setup

####  4.1.1 Datasets

We first test our method on CLEVR dataset [18] and then move to more complex
scenes. Our dataset is built upon ClevrTex [20], which wraps some realistic
textures to the objects and background. Instead of regular geometric models,
we use the some objects with more realistic shapes, such as animals and
furniture items, and use Blender [6] to render $400\times 400$ images from 30
different viewpoints for each scene, keeping the camera-to-world matrices for
all such rendered images. Additionally, we render the ground truth depth map
for evaluating the predicted 3D position. Each scene contains 2 to 4 objects
with different colors. We use 25 images from segmentation and training and 5
for testing randomly. We test on 8 different scenes in total.

####  4.1.2 Implementation Details

To obtain the groundtruth 3D positions, we train a NeRF model [29] for each
scene and generate the depth map from each training viewpoints. By combining
camera rays and the depth maps, we obtain the corresponding 3D positions of
the pixels.

We use the features of the first 16 layers of pretrained VGG models and
upsample the feature maps to the original image size for clustering. After
clustering, we dilate the object masks to reduce the mis-clustering of object
pixels. For object segmentation, we train a segmentation network for each
scene with batch size 10.

Figure 3: Qualitative results. This figure shows a sample input image for each
of the three examples, refined masks of the 3D objects in the scene, and
output images of different objects from two testing views of each scene. The
NeRFs are trained with 25 images.

###  4.2 Qualitative Results

####  4.2.1 Qualitative Comparisons

Figure 3 shows qualitative results of some objects from unseen novel views.
Specifically, to visualize the 3D segmentation and each object space, we
render the segmented object NeRFs at novel viewpoints and obtain the final 2D
masks. Notably, all novel views are rendered without using any masks, which
validates the converged masks output by our method.

Our method is scene-specific and unsupervised for 3D object segmentation.
Notably, this approach is not learning-based and does not need large sets of
data on similar scene for training. To our knowledge there is no non-learning-
based methods for 3D object segmentation on NeRFs similar to ours. With the 3D
information from NeRFs, we can obtain two kinds of masks that will be useful
for further applications: 1) the mask for a certain object in original space;
2) the mask for certain object in individual object space. The results show
that our method could handle complex textures (e.g., wood coffee table), and
complex shapes (e.g., animals) and non-trivial topologies (e.g., genus-$n$
objects such as the chairs).

We compare our results with both 2D unsupervised segmentation methods and 3D
learning-based unsupervised segmentation approaches. We cannot obtain
reasonable results by directly applying learning-based methods such as uORF
[50] on our dataset. In our tests, we use the random camera poses but all the
objects are included in the background slot. We also use the training camera
poses of uORF to re-render our scenes in blender. Then uORF can obtain some
foreground objects for some scenes, shown in Figure 4, which can roughly
separate the blue chair from the whole scene while the brown desk still fails
to be separated. But the segmented chair is unclear and blurry in the uORF
results. For some other scenes, uORF still cannot properly segment the scene
even after supplying the given camera poses.

For 2D unsupervised model, we compare with slot attention [27]. We adopt their
model which is trained on CLEVR dataset [18], and then fine-tune the
pretrained model on each of our scenes respectively. In most of the scenes,
slot attention can segment some colors. However, without adequate 3D
consideration or supervision, typically [27] cannot cleanly separate the
background, especially those scenes with textured background. Moreover, as our
dataset is quite diverse, slot attention cannot achieve a very good
reconstruction on par with the results produced by our model.

Figure 4: Qualitative comparison. Comparison with uORF [50] and slot attention
[27]. The results show that our model is the most robust on the scenes with
complex shape and color. Learning based methods which rely on large number of
scenes with similar color and shape distribution is difficult to generalize on
our dataset.

####  4.2.2 3D Scene Editing

After obtaining the masks and the individual object NeRFs, we can perform 3D
scene editing. Generally, we can implement any rigid transformation with
transformation matrices on the respective object NeRFs. Moreover, we can
perform object insertion and removal by operating on the NeRFs separated by
our method. While rendering the composite scene, by integrating the colors of
all objects with the density contribution of each object, we can coherently
handle occlusion after adding more objects to the scene. Figure 5 shows the
results of multiple objects insertion. Given the NeRFs of the objects and a
scene, we use the camera pose in the scene space to verify our editing
results. Specifically, we use the RGB loss of the out-masked region of the
images to get the NeRF of the scene after removing some objects. To translate
an object in the scene, we translate the camera position for rendering. For
example, suppose we want to move the object to $\mathbf{p}$. Let the pinhole
camera be at $\mathbf{o}$ and the object is at the origin. Then we move the
camera position $\mathbf{o}^{\prime}=\mathbf{o}-\mathbf{p}$, where
$\mathbf{o}^{\prime}$ is the new camera position. We sample points from
$\mathbf{o}^{\prime}$ along the original directions to obtain the results
after moving objects.

Figure 5: Results of object insertion, deletion and transformation. The figure
in the top left corner is the original scene image. The other two subfigures
in the top row show insertions of a green chair. The bottom left corner shows
deletion of the chairs. The remaining two show the results of adding more
objects.

####  4.2.3 Results on Real Scenes

Figure 6: ONeRF of real scene. Here we show the result tested on a real scene.
We visualize the result rendered at two novel viewpoints.

Figure 6 shows the result tested on a real scene. Although the original images
from real scene contain more noise due to complex real shading and non-
homogeneous color distribution on a single object, our method is quite robust
to such noise, and achieves a reasonable separation in this real-world
scenario.

###  4.3 Quantitative Evaluation

Method |  Mask IoU $\uparrow$ |  Depth Error $\downarrow$ |  PSNR $\uparrow$
---|---|---|---
Ours |  $\mathbf{0.86}$ $\pm 0.0$ |  $\mathbf{0.002}$ $\pm 0.0$ |  $\mathbf{30.23}$ $\pm 0.0$
uORF [50] |  $\mathbf{0.50}$ $\pm 0.0$ | - |  $\mathbf{23.80}$ $\pm 0.0$
Slot attention [27] |  $\mathbf{0.40}$ $\pm 0.0$ | - |  $\mathbf{18.99}$ $\pm 0.0$
Table 1: Quantitative comparison. We evaluate the segmented 3D objects on
various metrics by comparing to the ground-truth renderings on from novel
unseen viewpoints. For depth error, we normalize the depth between 0 and 1 and
find the average mean square error. For uORF, it can only segment 2 objects
out of all the 8 so we take the average on these two objects. The depth is
used to estimate the 3D position, which is not applicable for [50] and [27].

Tab. 1 shows the quantitative comparison between our method and the two
baseline methods, where we can see that in our moderately complex and diverse
scenes scenario, our method produces the most stable performance.

###  4.4 Ablation Studies

##### Effect of mask refinement.

We compare the masks before and after our refinement using intersection over
union. The errors and figures for comparison are shown in Table 2 and Figure
7. The refinement step can complete both tables.

Method | before refinement | before refinement |
---|---|---|---
Mask IoU $\uparrow$ | $\mathbf{0.86}$ | $\mathbf{0.76}$ |
Table 2: Masks comparison. We evaluate the IoU errors of all objects over the
8 scenes before and after the refinement step. Figure 7: Masks Comparison.
Here are the masks of the two tables before and after refinement.

##### Effect of object segmentation network.

As shown in the second column of Figure 8, just simple $K$-means clustering
cannot replace our more advanced segmentation model, since the former cannot
precisely distinguish coherent object boundary and is very vulnerable to
noise.

##### Effect of coarse clustering.

We use the VGG feature to distinguish the background and foreground objects,
and 3D positions to select foreground objects. This method can roughly choose
foreground object automatically while removing a large portion of background
regions in the images, which helps the subsequent object segmentation.

Figure 8: Ablation study. Here we visualize the results of the ablation study.
From left to right: the ground truth image at novel view, result without
segmentation model, result without foreground-background separator, result
without correct 3D information, result without refinement and result produced
by our full model.

During our testing, we find that with the same initial $K$-Means clustering
parameters, adding 3D positions can cluster the foreground objects to the
specific class where we use 3D information. Additionally, we remove the coarse
clustering step, i.e., directly apply the unsupervised segmentation method
[21] to the same input images in our testing. Then we replace the predicted 3D
positions with random noise in the first step to check whether the positions
effects. Both results are shown in Figure 8.

##  5 Conclusions

This paper presents the first significant step toward segmenting multiple
objects NeRFs from multiple views of the underlying scene. Compared with
single-image methods, with the 3D from NeRFs, ONeRFs can alternately optimize
2D masks and 3D geometry so that the former can be used to delineate cluttered
scenes into coherent object NeRFs, which can then be used to enable a range of
3D scene editing tasks, allowing users to readily insert 3D objects, delete 3D
objects, and modify object geometry in addition to novel view synthesis. In
the future, we will investigate NeRF completion when the pertinent occluded
parts of the object were not photographed by any of the input views. This is
achieved by detecting the underlying object symmetry, which would not have
been possible without segmenting the scene into individual object NeRFs.

## References

  * [1] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In ICCV, 2021.
  * [2] Jeremy S. De Bonet and Paul Viola.  Poxels: Probabilistic voxelized volume reconstruction.  In ICCV, 1999.
  * [3] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P.A. Lensch.  Nerd: Neural reflectance decomposition from image collections.  In ICCV, 2021.
  * [4] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su.  Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.  In ICCV, 2021.
  * [5] Julian Chibane, Aayush Bansal, Verica Lazova, and Gerard Pons-Moll.  Stereo radiance fields (srf): Learning view synthesis for sparse views of novel scenes.  In CVPR, pages 7911–7920, 2021.
  * [6] Blender Online Community.  Blender – a 3D modelling and rendering package.  Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018.
  * [7] Eric Crawford and Joelle Pineau.  Spatiial invariant unsupervised object detection with convolutional neural networks.  In AAAI, 2019.
  * [8] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan.  Depth-supervised nerf: Fewer views and faster training for free.  arXiv preprint arXiv:2107.02791, 2021.
  * [9] Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner.  Genesis: Generative scene inference and sampling with object-centric latent representations.  In ICLR, 2020.
  * [10] Yasutaka Furukawa and Jean Ponce.  Accurate, dense, and robust multiview stereopsis.  IEEE TPAMI, 32(8):1362–1376, 2010.
  * [11] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner.  Multi-object representation learning with iterative variational inference.  In International Conference on Machine Learning, 2019.
  * [12] Yudong Guo, Keyu Chen, Sen Liang, Yongjin Liu, Hujun Bao, and Juyong Zhang.  Ad-nerf: Audio driven neural radiance fields for talking head synthesis.  In ICCV, 2021.
  * [13] Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham.  Randla-net: Efficient semantic segmentation of large-scale point clouds.  In CVPR, 2020.
  * [14] Ajay Jain, Matthew Tancik, and Pieter Abbeel.  Putting nerf on a diet: Semantically consistent few-shot view synthesis.  In ICCV, pages 5885–5894, 2021.
  * [15] Yoonwoo Jeong, Seokjun Ahn, Christopehr Choy, Animashree Anandkumar, Minsu Cho, and Jaesik Park.  Self-calibrating neural radiance fields.  In ICCV, 2021.
  * [16] Zhang Jiakai, Liu Xinhang, Ye Xinyi, Zhao Fuqiang, Zhang Yanshun, Wu Minye, Zhang Yingliang, Xu Lan, and Yu Jingyi.  Editable free-viewpoint video using a layered neural representation.  In ACM SIGGRAPH, 2021.
  * [17] Li Jiang, Shaoshuai Shi, Zhuotao Tian, Xin Lai, Shu Liu, Chi-Wing Fu, and Jiaya Jia.  Guided point contrastive learning for semi-supervised point cloud semantic segmentation.  In ICCV, pages 6423–6432, 2021.
  * [18] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In CVPR, 2017.
  * [19] Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi.  Learning-based view synthesis for light field cameras.  ACM TOG, 35(6), 2016.
  * [20] Laurynas Karazija, Iro Laina, and Christian Rupprecht.  Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation.  In NeurIPS, 2021.
  * [21] Wonjik Kim, Asako Kanezaki, and Masayuki Tanaka.  Unsupervised learning of image segmentation based on differentiable feature clustering.  IEEE TIP, 29:8055–8068, 2020.
  * [22] Vladimir Kolmogorov and Ramin Zabih.  Multi-camera scene reconstruction via graph cuts.  In ECCV, 2002.
  * [23] Kiriakos N. Kutulakos and Steven M. Seitz.  A theory of shape by space carving.  IJCV, 38:307–314, 2000.
  * [24] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn.  Space: Unsupervised object-oriented scene representation via spatial attention and decomposition.  In ICLR, 2020.
  * [25] Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, and Bryan Russell.  Editing conditional radiance fields.  In ICCV, pages 5773–5783, 2021.
  * [26] Yongcheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan.  Relation-shape convolutional neural network for point cloud analysis.  In CVPR, 2019.
  * [27] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf.  Object-centric learning with slot attention.  In NeurIPS, 2020.
  * [28] Keyang Luo, Tao Guan, Lili Ju, Haipeng Huang, and Yawei Luo.  P-mvsnet: Learning patch-wise matching confidence aggregation for multi-view stereo.  In ICCV, 2019.
  * [29] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [30] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  ICCV, 2021.
  * [31] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  In CVPR, pages 10318–10327, 2021.
  * [32] Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In CVPR, July 2017.
  * [33] Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  NeurIPS, 2017.
  * [34] Shi Qiu, Saeed Anwar, and Nick Barnes.  Geometric back-projection network for point cloud classification.  IEEE Transactions on Multimedia, 2021.
  * [35] Shi Qiu, Saeed Anwar, and Nick Barnes.  Semantic segmentation for real point cloud scenes via bilateral augmentation and adaptive fusion.  In CVPR, pages 1757–1767, 2021.
  * [36] Ramesh Raskar, Wojciech Matusik, Chris Buehler, Steven Gortler, and Leonard Mcmillan.  Image-based visual hulls.  SIGGRAPH 2000.
  * [37] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger.  Graf: Generative radiance fields for 3d-aware image synthesis.  In NeurIPS, 2020.
  * [38] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In ICLR, 2015.
  * [39] Karl Stelzner, Kristian Kersting, and Adam R. Kosiorek.  Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation.  arXiv preprint arXiv:2104.01148, 2021.
  * [40] Vadim Tschernezki, Diane Larlus, and Andrea Vedaldi.  NeuralDiff: Segmenting 3D objects that move in egocentric videos.  In 3DV, 2021.
  * [41] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon.  Dynamic graph cnn for learning on point clouds.  ACM TOG, 2019.
  * [42] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu.  NeRF$--$: Neural radiance fields without known camera parameters.  arXiv preprint arXiv:2102.07064, 2021.
  * [43] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao.  3d shapenets: A deep representation for volumetric shapes.  In CVPR, 2015.
  * [44] Xu Yan, Chaoda Zheng, Zhen Li, Sheng Wang, and Shuguang Cui.  Pointasnl: Robust point clouds processing using nonlocal neural networks with adaptive sampling.  In CVPR, 2020.
  * [45] Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui.  Learning object-compositional neural radiance field for editable scene rendering.  In ICCV, 2021.
  * [46] Jiayu Yang, Wei Mao, Jose M. Alvarez, and Miaomiao Liu.  Cost volume pyramid based depth inference for multi-view stereo.  In CVPR, 2020.
  * [47] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan.  Mvsnet: Depth inference for unstructured multi-view stereo.  ECCV, 2018.
  * [48] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa.  Plenoctrees for real-time rendering of neural radiance fields.  In ICCV, 2021.
  * [49] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In CVPR, pages 4578–4587, 2021.
  * [50] Hong-Xing Yu, Leonidas J. Guibas, and Jiajun Wu.  Unsupervised Discovery of Object Radiance Fields.  arXiv preprint arXiv:2107.07905, 2021.
  * [51] Biao Zhang and Peter Wonka.  Point cloud instance segmentation using probabilistic embeddings.  In CVPR, pages 8883–8892, 2021.
  * [52] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun.  Nerf++: Analyzing and improving neural radiance fields.  arXiv:2010.07492, 2020.
  * [53] Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron.  Nerfactor: Neural factorization of shape and reflectance under an unknown illumination.  ACM TOG, 2021.
  * [54] Yachao Zhang, Yanyun Qu, Yuan Xie, Zonghao Li, Shanshan Zheng, and Cuihua Li.  Perturbed self-distillation: Weakly supervised large-scale point cloud semantic segmentation.  In ICCV, 2021.
  * [55] Na Zhao, Tat-Seng Chua, and Gim Hee Lee.  Few-shot 3d point cloud semantic segmentation.  In CVPR, pages 8873–8882, 2021.
  * [56] Yunsong Zhou, Hongzi Zhu, Chunqin Li, Tiankai Cui, Shan Chang, and Minyi Guo.  Tempnet: Online semantic segmentation on large-scale point cloud series.  In ICCV, 2021.