# Peridynamic stress is a weighted static Virial stress

Shaofan Li  Department of Civil and Environmental Engineering, University of
California, Berkeley, CA 94720, USA

###### Abstract

The peridynamic stress tensor proposed by Lehoucq and Silling [1] is
cumbersome to implement in numerical computations. In this note, we show that
the peridynamic stress tensor has the mathematical expression of a weighted
static Virial stress derived by Irving and Kirkwood [2], which offers a simple
and clear expression for numerical calculations of peridynamic stress tensor.

††journal: arXiv.org

Peridynamics is reformulation of non-local continuum mechanics or
computational non-local mechanics [3, 4]. In the peridynamic equation of
motion, following the notation of Silling and Lehoucq [4] one can write the
balance of linear momentum as

$\rho\ddot{\bf u}({\bf X},t)=\int_{\mathcal{H}_{X}}{\bf f}({\bf
X}^{\prime},{\bf X},t)dV_{X^{\prime}}+{\bf b}({\bf X},t),~{}~{}\forall{\bf
X}\in\mathcal{B}$ (1)

where $\mathcal{B}\subset^{3}$; $\mathcal{H}_{X}\subset\mathcal{B}$ is a the
horizon of the material point ${\bf X}$; $\rho$ is the material density, and
${\bf b}({\bf X},t)$ is the body force. The term

$\int_{\mathcal{H}_{X}}{\bf f}({\bf X}^{\prime},{\bf X},t)dV_{X^{\prime}}$

replaces the divergence of the first Piola-Kirchhoff stress $\nabla\cdot{\bf
P}$ at the material point ${\bf X}$.

By definition,

${\bf f}({\bf X}^{\prime},{\bf X},t):=\bigl{(}{\bf t}({\bf X}^{\prime},{\bf
X},t)-{\bf t}({\bf X},{\bf X}^{\prime},t)\bigr{)}=-{\bf f}({\bf X},{\bf
X}^{\prime},t)$ (2)

is antisymmetric, where ${\bf X},{\bf X}^{\prime}$ are the position vectors of
material points in the referential configuration.

Equation (1) extends the balance equation of linear momentum to nonlocal
media. However, it loses some valuable properties that are associated with the
local balance law such as the divergence theorem or the Gauss theorem.

Noticing such inadequacy, Lehoucq and Silling [1] define the following
nonlocal Peridynamic Stress tensor

${\bm{\varsigma}}({\bf X}):={1\over
2}\int_{\mathcal{S}^{2}}\int_{0}^{\infty}\int_{0}^{\infty}(y+z)^{2}{\bf
f}({\bf X}+y{\bf M},{\bf X}-z{\bf M})\otimes{\bf M}dzdyd\Omega_{M}~{},$ (3)

where $\mathcal{S}^{2}$ is the unit sphere. By doing so, we have the relation

$\nabla\cdot{\bm{\varsigma}}=\int_{\mathcal{H}_{X}}{\bf f}({\bf
X}^{\prime},{\bf X},t)dV_{X^{\prime}}~{}.$ (4)

where $\nabla$ is the local gradient operator. An immediate benefit of Eq. (3)
is that we can link the divergence of the peridynamic stress with the boundary
linear momentum flux, i.e.

$\int_{\mathcal{B}}\nabla\cdot{\bm{\varsigma}}dV_{X}=\int_{\partial\mathcal{B}}{\bm{\varsigma}}\cdot{\bf
N}dS_{X}~{},$

which allows us to establish peridynamics-based Galerkin weak formulations
conveniently, and maybe even formulate peridynamics theories of plates and
shells.

By using Noll’s lemmas [5], such nonlocal integral theorems have been late
extended to a more general situations by Gunzburger and Lehoucq [6] and Du et.
al. [7]. However, in practice the peridynamic stress defined in Eq. (3) is
cumbersome to evaluate. To resolve this issue, in this note, we show that in
peridynamic particle formulation, which is a special case of the non-local
continuum, the peridynamic stress tensor has the exact expression of the
static Virial stress defined by Irving and Kirkwood [2].

######  Theorem 0.1 (Alternative form of Peridynamic Stress tensor).

Consider the peridynamics force density that can be expressed as the following
expression of the Irving-Kirkwoods formulation [2, 4],

${\bf f}({\bf X}^{\prime},{\bf X},t)=\sum_{I=1}^{N_{X}}\sum_{J=1}^{N_{X}}{\bf
F}_{IJ}\delta({\bf X}-{\bf X}_{I})\delta({\bf X}^{\prime}-{\bf X}_{J})$ (5)

where ${\bf F}_{IJ}$ is the force acting on the particle $I$ from the particle
$J$ (see Fig. 1); $N_{X}$ is the total number of particles inside the horizon
$\mathcal{H}_{X}$. The nonlocal peridynamic stress

${\bm{\varsigma}}({\bf X}):={1\over
2}\int_{\mathcal{S}^{2}}\int_{0}^{\infty}\int_{0}^{\infty}(y+z)^{2}{\bf
f}({\bf X}+y{\bf M},{\bf X}-z{\bf M})\otimes{\bf M}dzdyd\Omega_{M}~{},$ (6)

has the following exact form,

${\bm{\varsigma}}({\bf X}):=-{1\over
2}\sum_{I=1}^{N_{X}}\sum_{J=1}^{N_{X}}{\bf F}_{IJ}\otimes{({\bf X}_{I}-{\bf
X})\over|{\bf X}_{I}-{\bf X}|}|{\bf X}_{I}-{\bf X}_{J}|,~{}~{}{\bf X}_{J},{\bf
X}_{I}\in\mathcal{H}_{X},~{}~{}{\bf X}_{J}\not={\bf X}_{I}~{}.$ (7)

where ${\bf F}_{IJ}$ is the force acting on the atom $I$ by the atom $J$.

Figure 1:  Illustration of peridynamics particle sampling strategy

###### Proof.

Based on Noll’s lemma [5], we can write the first peridynamics Piola-Kirchhoff
stress as

$\displaystyle{\bm{\varsigma}}({\bf X})$ $\displaystyle=$
$\displaystyle{1\over
2}\int_{S^{2}}d\Omega_{m}\int_{0}^{\infty}R^{2}dR\int_{0}^{1}{\bf f}({\bf
X}+\alpha R{\bf M},{\bf X}-(1-\alpha)R{\bf M})\otimes{\bf M}d\alpha$ (8)
$\displaystyle=$ $\displaystyle-{1\over 2}\int_{{}^{3}}dV_{R}\int_{0}^{1}{\bf
f}({\bf X}+\alpha{\bf R},{\bf X}-(1-\alpha){\bf R})\otimes{\bf
R}d\alpha~{},~{}\forall{\bf X}\in\mathcal{B}$

Considering the Irving-Kirkwood formalism [2], we have the following
peridynamics sampling formulation (see Fig. 1)

${\bf f}({\bf X}^{\prime},{\bf X})=-{\bf f}({\bf X},{\bf
X}^{\prime})=-\sum_{J=1}^{N}{\bf F}_{IJ}\delta({\bf X}-{\bf X}_{I})\delta({\bf
X}^{\prime}-{\bf X}_{J}),~{}~{}{\bf X}_{I}\not={\bf X}_{J}~{}.$ (9)

Letting

${\bf X}={\bf X}_{C}+\alpha{\bf R},~{}{\rm and}~{}~{}{\bf X}^{\prime}={\bf
X}_{C}-(1-\alpha){\bf R}$

and substituting them into Eq. (9), we then have

$\displaystyle{\bf f}({\bf X}+\alpha{\bf R},{\bf X}-(1-\alpha){\bf R})$ (10)
$\displaystyle=$ $\displaystyle\sum_{I=1}^{N_{X}}\sum_{J=1}^{N_{X}}{\bf
F}_{IJ}\delta(\alpha{\bf R}-({\bf X}_{I}-{\bf X}_{C}))\delta(\alpha{\bf
R}-{\bf R}-({\bf X}_{J}-{\bf X}_{C})).$

where ${\bf X}_{I},{\bf X}_{J}\in\mathcal{H}_{X_{C}},~{}~{}{\bf
X}_{I}\not={\bf X}_{J}$.

Considering the following integration identities

$\displaystyle\delta(g(x))={\delta(x-x_{0})\over|g^{\prime}(x_{0})|}$ (11)
$\displaystyle\int_{-\infty}^{\infty}\delta(\xi-x)\delta(x-\eta)dx=\delta(\eta-\xi)$
(12)

we have

$\displaystyle\int_{{}^{3}}\delta(\alpha{\bf R}-({\bf X}_{I}-{\bf
X}_{C}))\delta(\alpha{\bf R}-{\bf R}-({\bf X}_{J}-{\bf X}_{C})){\bf R}d(\alpha
V_{R})$ $\displaystyle=\alpha^{-1}({\bf X}_{I}-{\bf X}_{C})\delta(({\bf
X}_{I}-{\bf X}_{J})-\alpha^{-1}({\bf X}_{I}-{\bf X}_{C}))~{}.$ (13)

Let $\xi=\alpha^{-1}$ and $d\xi=-{\xi\over\alpha}d\alpha\to
d\alpha=-\xi^{-2}d\xi$. We then can have,

$\displaystyle{\bm{\varsigma}}({\bf X}_{C})$ $\displaystyle=$
$\displaystyle-{1\over 2}\bigl{(}\sum_{I=1}^{N_{X}}\sum_{J=1}^{N_{X}}{\bf
F}_{IJ}\bigr{)}\otimes({\bf X}_{I}-{\bf X}_{C})$ (14)
$\displaystyle\cdot\int_{0}^{\infty}\delta(({\bf X}_{I}-{\bf X}_{J})-\xi({\bf
X}_{I}-{\bf X}_{C}))\xi^{-1}d\xi$ $\displaystyle=$ $\displaystyle-{1\over
2}\sum_{I=1}^{N_{X}}\sum_{J=1}^{N_{X}}{\bf F}_{IJ}\otimes\Bigl{(}{{\bf
X}_{I}-{\bf X}_{C}\over|{\bf X}_{I}-{\bf X}_{C}|}\Bigr{)}|{\bf X}_{I}-{\bf
X}_{J}|$

Let ${\bf X}_{C}\to{\bf X}$. We prove the claim. ∎

######  Remark 0.1.

The above expression may be rewritten as

$\displaystyle{\bm{\varsigma}}({\bf X})$ $\displaystyle=$
$\displaystyle-{1\over 2}\sum_{I=1}^{N_{X}}\sum_{J=1}^{N_{X}}{\bf
F}_{IJ}\otimes({\bf X}_{I}-{\bf X}_{J})\Bigl{(}{|{\bf X}_{I}-{\bf
X}_{J}|\over{\bf X}_{I}-{\bf X}_{J}}\Bigr{)}\cdot\Bigl{(}{{\bf X}_{I}-{\bf
X}\over|{\bf X}_{I}-{\bf X}|}\Bigr{)}~{}.$ (15)

which is a weighted static Virial stress.

For the state-based peridynamics, we have the following result.

######  Corollary 0.1.1 (Peridynamics PK-I stress tensor).

For continuous variable ${\bf X}_{i},{\bf X}_{j}\in\mathcal{H}_{C}$, we let

${\bf f}({\bf X}_{i},{\bf X}_{j})=w(X_{ij})\Bigl{(}{\bf P}_{j}{\bf K}_{j}{\bf
X}_{ij}-{\bf P}_{i}{\bf K}_{i}{\bf X}_{ji}\Bigr{)}$ (16)

where ${\bf K}_{i}={\bf K}({\bf K}_{i})$ is the shape tensor;
$w(X_{ij}=w(|{\bf X}_{j}-{\bf X}_{i}|)$ is a window function; ${\bf
X}_{ij}={\bf X}_{j}-{\bf X}_{i}$, and ${\bf P}({\bf X}_{i})$ is the first
Piola-Kirchhoff stress at the local continuum material point ${\bf X}_{i}$ .
For the first Piola-Kirchhoff stress tensor, ${\bf P}({\bf X}),~{}{\bf
X}\in\Omega_{0}$, the nonlocal divergence of ${\bf P}$,

$\mathfrak{D}[{\bf P}]=\int_{\Omega_{0}}w(X_{ij})\Bigl{(}{\bf P}_{j}{\bf
X}_{ij}-{\bf P}_{i}{\bf X}_{ji}\Bigr{)}\cdot{\bf
K}^{-1}dV_{j},~{}~{}\forall{\bf X}_{i}\in\Omega_{0},$ (17)

equals to the local divergence of a nonlocal counterpart, $\widetilde{\bf P}$,

$\nabla\cdot\widetilde{\bf P}=\mathfrak{D}[{\bf P}]$ (18)

where

$\displaystyle\widetilde{\bf P}({\bf X})$ $\displaystyle:=$
$\displaystyle{1\over 2}\sum_{I=1}^{N_{X}}\sum_{J=1,J\not=I}^{N_{X}}{\bf
f}({\bf X}_{I},{\bf X}_{J})\otimes({\bf X}_{I}-{\bf X}_{J})\Bigl{(}{|{\bf
X}_{I}-{\bf X}_{J}|\over({\bf X}_{I}-{\bf X}_{J})}{|{\bf X}_{I}-{\bf
X}|\over({\bf X}_{I}-{\bf X})},~{}~{}$ (19) $\displaystyle\forall~{}{\bf
X}\in\mathcal{B},~{}~{}{\bf X}_{I},{\bf X}_{J}\in\mathcal{H}_{X}~{}.$

######  Remark 0.2.

Note that the above result does not restricted to peridynamics PK-I stress
tensor, and it is valid for general peridynamics stress tensor after small
modifications depending on the case that is under consideration. Amazingly,
the result reveals that fact that the mesoscale peridynamic stress tensor has
exactly the same expression as that of the microscale static Virial stress,
except that it does not count for the contribution from the kinetic energy.
Moreover, the expression (7) is so simple that it can be readily implemented
in numerical calculations without much trouble.

## Reference

## References

  * [1] R.B. Lehoucq and S.A. Silling.  Force flux and the peridynamic stress tensor.  Journal of the Mechanics and Physics of Solids, 56(4):1566–1577, 2008.
  * [2] J.H. Irving and J.G. Kirkwood.  The statistical mechanical theory of transport processes. iv. the equations of hydrodynamics.  The Journal of Chemical Physics, 18(6):817–829, 1950.
  * [3] S.A. Silling.  Reformulation of elasticity theory for discontinuities and long-range forces.  Journal of the Mechanics and Physics of Solids, 48(1):175–209, 2000\.
  * [4] S.A. Silling and R.B. Lehoucq.  Peridynamic theory of solid mechanics.  Advances in Applied Mechanics, pages 73–168, 2010.
  * [5] W. Noll.  Die herleitung der grundgleichungen der thermomechanik der kontinua aus der statistischen mechanik.  Journal of Rational Mechanics and Analysis, 4:627–646, 1955.
  * [6] M. Gunzburger and R.B. Lehoucq.  A nonlocal vector calculus with application to nonlocal boundary value problems.  Multiscale Modeling & Simulation, 8(5):1581–1598, 2010.
  * [7] Q. Du, M. Gunzburger, R.B. Lehoucq, and K. Zhou.  A nonlocal vector calculus, nonlocal volume-constrained problems, and nonlocal balance laws.  Mathematical Models and Methods in Applied Sciences, 23(03):493–540, 2013.


# Learning to Detect Every Thing in an Open World

Kuniaki Saito1 Ping Hu1 Trevor Darrell2 Kate Saenko1,3
1Boston University 2University of California, Berkeley 3MIT-IBM Watson AI Lab

###### Abstract

Many open-world applications require the detection of novel objects, yet
state-of-the-art object detection and instance segmentation networks do not
excel at this task. The key issue lies in their assumption that regions
without any annotations should be suppressed as negatives, which teaches the
model to treat the unannotated objects as background. To address this issue,
we propose a simple yet surprisingly powerful data augmentation and training
scheme we call Learning to Detect Every Thing (LDET). To avoid suppressing
hidden objects—background objects that are visible but unlabeled—we paste
annotated objects on a background image sampled from a small region of the
original image. Since training solely on such synthetically-augmented images
suffers from domain shift, we decouple the training into two parts: 1)
training the region classification and regression head on augmented images,
and 2) training the mask heads on original images. In this way, a model does
not learn to classify hidden objects as background while generalizing well to
real images. LDET leads to significant improvements on many datasets in the
open-world instance segmentation task, outperforming baselines on cross-
category generalization on COCO, as well as cross-dataset evaluation on UVO
and Cityscapes.

Figure 1: Mask RCNN (top row) detects fewer objects than our approach (bottom
row) in open-world instance segmentation. In this task, the model must locate
and segment all objects in the image irrespective of categories used for
training. Here both detectors are trained on COCO [34] and tested on UVO [48].
Our detector correctly localizes many objects that are not labeled in COCO
with the help of a new data augmentation method and training scheme.

##  1 Introduction

Humans routinely encounter new tools, foods, or animals, having no problem
perceiving the novel objects as objects despite having never seen them before.
Unlike humans, current state-of-the-art detection and segmentation methods
[22, 35, 40, 41, 33] have difficulty recognizing novel objects as objects
because they are designed with a closed-world assumption. Their training aims
to localize known (annotated) objects while regarding unknown (unannotated)
objects as background. This causes the models to fail in locating novel
objects and learning general objectness. One way to deal with this challenge
is to create a dataset with an exhaustive annotation of every single object in
each image. However, creating such datasets is very expensive. In fact, many
public datasets [34, 17, 51] for object detection and instance segmentation do
not label all objects in an image (Fig. 2).

Figure 2: Problem in a standard training of object detector. Examples are from
COCO. Colored boxes are annotated boxes while white-dashed boxes are potential
background regions. Many white-dashed regions locate objects, but are regarded
as background during training in a conventional way of training object
detector. This can suppress the objectness of novel objects.

Failing to learn general objectness can cause issues in many applications. For
instance, embodied AI (e.g., robotics, autonomous driving) requires localizing
objects unseen during training. Autonomous driving systems need to detect
novel objects in front of the vehicle to avoid accidents though identifying
the category is not necessarily required. In addition, zero-shot and few-shot
detection have to localize objects unseen during training. Open-world instance
segmentation [48] aims to localize and segment novel objects, but the state-
of-the-art model does not perform well as shown in [48].

We find that the failure of current state-of-the-art models is partly due to
the training pipeline, _i.e_., regarding all regions that overlap little with
the annotated foreground objects as background. Even if the background
includes hidden objects—background objects that are visible but unlabeled—as
in Fig. 2, the models are trained not to detect them, which prevents the
models from learning general objectness. To address this, Kim _et al_. [29]
proposed to learn the localization quality of region proposals instead of
classifying them as foreground vs. background. Their approach samples object
proposals close to the ground truth and learns to estimate the corresponding
localization quality. While partially mitigating the issue, this approach
still needs to carefully set the overlap threshold for positive/negative
sampling and risks suppressing hidden objects as non-objects.

To improve open-set instance segmentation, we propose a simple, yet powerful,
learning framework along with a new data augmentation method, called Learning
to Detect Every Thing (LDET). To eliminate the risk of suppressing hidden
objects, we copy foreground objects using their mask annotation and paste them
onto a background image. The background image is synthesized by resizing a
cropped patch. By keeping the cropped patch small, we make it unlikely that
the resulting synthesized images contain any hidden objects. However, this
background creation process makes synthesized images look very different from
real images, _e.g_., the background may consist only of low-frequency content.
Thus, a detector naively trained on such images barely performs well. To
overcome this limitation, we decouple the training into two parts: 1) training
background and foreground region classification and localization heads with
synthesized images, and 2) learning a mask head with real images. In training
the classification head, there is little risk of treating hidden objects as
background since they are removed in the synthesized images. In addition,
since the mask heads are trained to segment instances in real images, the
backbone learns generalizable representations to separate foreground from
background regions in real images.

Despite being seemingly a minor change, LDET demonstrates remarkable gains in
open-world instance segmentation and detection. On COCO [34], LDET trained on
VOC categories improves the average recall by 14.1 points when evaluated on
non-VOC categories. Surprisingly, LDET achieves significant improvements in
detecting novel objects without requiring additional annotation _e.g_., LDET
trained only on VOC categories (20 classes) in COCO outperforms Mask RCNN
trained on all COCO categories (80 classes) when evaluating average recall on
UVO [48]. As shown in Fig. 1, LDET can generate precise object proposals as
well as cover many objects in the scene.

Our contributions are summarized as follows:

  * •

We propose a simple framework, LDET, consisting of new data augmentation and
decoupled training for open-world instance segmentation.

  * •

We demonstrate that both our data augmentation and decoupled training are
crucial to achieving good performance in open-world instance segmentation.

  * •

LDET outperforms state-of-the-art methods in all settings including cross-
category settings on COCO and cross-dataset setting on COCO-to-UVO and
Cityscape-to-Mapillary.

##  2 Related Work

Region proposals. Unsupervised region proposal generation used to be a
standard approach to localize objects in a scene [52, 1, 2, 46]. These
approaches localize objects in a class-agnostic way, but employ hand-crafted
features (_i.e_., color contrast, edge, _etc_.) to capture general objectness.

Figure 3: Our augmentation strategy creates images without hidden objects by
upscaling small regions to use as background.

Closed-World object detection. Much effort has been spent on supervised object
detection with a closed world assumption [18, 21, 41, 20, 35, 40]. The ability
to detect known objects has been improving with better architecture designs
[32, 5, 11] or objectives [33]. Also, localizing objects given a few training
examples or semantic information is becoming a popular research topic [28, 3].
However, these attempts are still constrained by the taxonomy defined by the
dataset. Our model can detect more categories than defined by the dataset,
which can be very useful in few-shot or zero-shot object detection.

Open-World object detection/segmentation. Open-world recognition problems are
gaining attention in image classification, object detection, and segmentation
[4, 13, 7]. Especially, many methods have been proposed for open-set image
classification, where the goal is to separate novel categories from known
categories given a closed-set training set [4, 36, 38, 50, 45]. On the
contrary, the goal of open-world instance segmentation is to detect and
segment all objects in a scene without distinguishing novel objects from seen
ones. We acknowledge that there is ambiguity in the definition of “object”,
and follow [48] during evaluation.

Wang _et al_. [48] recently published the first benchmark dataset for open-set
instance segmentation, which includes various categories from YouTube videos.
However, from a methodological perspective, open-world object detection and
segmentation remain understudied despite the importance of the task. Hu _et
al_. and Kuo _et al_. [24, 31] proposed approaches for predicting masks of
various objects, but they require bounding boxes from classes of interest.
Jaiswal _et al_. [26] trained a detector in an adversarial manner to learn
class-invariant objectness. Joseph _et al_. [27] proposed a semi-supervised
learning approach for open-world detection, which regards regions that are far
from ground truth boxes but have a high objectness score as hidden foreground
objects. Kim  _et al_. [29] employed localization quality estimation with the
claim that the estimation strategy is more generalizable in open-world
instance segmentation.

The core of the open-world detection problem, as pointed out by Kim  _et al_.
[29], lies in the detector training pipeline: regarding hidden objects as
background. This training scheme is common in both two-stage and one-stage
detectors. However, none of the approaches listed above solves this issue. Our
approach takes the first step in addressing background suppression via novel
data augmentation strategies and shows remarkable improvements over baselines
despite its simplicity.

Copy-Paste augmentation. Pasting foreground objects on a background is a
widely used technique in many vision applications [14, 19, 47]. Recently,
copy-and-paste augmentation was shown to be a very useful technique in
instance segmentation [15, 16, 19]. Dwibedi  _et al_. [16] proposed to
synthesize an instance segmentation dataset by pasting object instances on
diverse backgrounds and trained on the augmented images in addition to the
original dataset. Dvornik  _et al_. [15] considered modeling the visual
context to paste the objects while Ghiasi  _et al_. [19] showed that pasting
objects randomly is good enough to provide solid gains. These methods still
assume a closed-world setting, whereas our task is the open-world instance
segmentation problem. There are two technical differences compared to these
methods. First, our augmentation samples background images from a small region
of an original image to create a background unlikely to have any objects. This
pipeline is designed to circumvent suppressing hidden objects as background
and does not require any external background data as used in [16]. Second, we
decouple the training into two parts, which is also key to achieving a well-
performing open-world detection model. In contrast, all of the existing
approaches above simply train on synthesized images.

##  3 Learning to Detect Every Thing

In this section, we describe the proposed LDET scheme for open-world instance
segmentation. Given an instance segmentation model, we train the object
detection loss on images with synthesized background, and the instance mask
loss on real images. Mask-RCNN [22] serves as the base model. We describe
details of the data generation process (Fig. 3) and training scheme (Fig. 5)
below.

###  3.1 Data Augmentation: Erasing Background

Background region sampling. First, we apply Gaussian smoothing to the input
image before cropping the foreground and background region, and denote the
smoothed image as $I_{1}$. Then, we randomly sample a small unannotated region
from $I_{1}$ and resize it to the same size as the input image to serve as a
background canvas, which we denote as $I_{2}$. By smoothing the whole image
before this operation, we expect to reduce the discrepancy in high-frequency
content between the foreground and background images. We set the width and
height of the background region to be $\frac{1}{8}$ of the original image’s.
Cropping the small region entails a much lower risk of including hidden
background objects compared to using the original background. Even if it
happens to include unannotated objects, drastically upscaling the patch makes
the objects’ appearance totally different, as shown in examples in Fig. 4. We
vary the scale of the background canvas in experiments but find the difference
in performance to be not very significant (see Table 5).

Figure 4: Examples of original inputs (left) and synthesized image (right).
Masked regions are highlighed with colors. Using small regions as background
avoids the risk of having hidden objects in the background. In some cases, the
background patch happens to locate foreground objects (second row in the left
column). Note that this case is rare, and the patch looks different from the
original because it is significantly upscaled.

Blending pasted objects. To avoid the model learning to separate background
and foreground by the difference in frequency information, foreground objects
are downsampled and resized to the original size. Then, the foreground objects
are pasted on the canvas. To insert copied objects into an image, we use the
binary mask ($M$) of pasted objects using ground-truth annotations and compute
the new image as $I_{1}\times M+I_{2}\times(1-M)$. We apply a Gaussian filter
to the binary mask to smooth the edges of the copied objects. Examples of
synthesized images using the COCO dataset with 80 categories are illustrated
in Fig. 4. Note that even in datasets with dense annotations like COCO, many
objects are not annotated, and our augmentation effectively removes such
hidden objects from the background. We do not claim that details such as
smoothing and resizing operations are necessarily optimal for open-world
instance segmentation, but empirically find they work well.

###  3.2 Decoupled Training

Simply training a detector on the synthesized images in the conventional way
[22] does not work well due to the domain shift (See Table 2). Since real
images and our synthesized images have very distinct content and layout, a
detector trained on our synthesized data does not generalize well to real
images. In this section, we propose a simple yet effective approach to
mitigate this issue of domain shift.

Typically, the training objectives for instance segmentation models consist of
two major terms: object detection loss and instance mask loss. In methods like
Mask RCNN [22], the object detection loss is composed of a region proposal
classification loss and a box regression loss, which are used to train both
the region proposal network (RPN) and the region of interest (ROI) heads. For
simplicity, we summarize the objectives for RPN and ROI as one loss. The
instance mask loss trains the mask head using only positive regions, _e.g_.,
IoU with the nearest ground truth $>$ 0.5 by default in Mask RCNN.

Figure 5: Training pipeline. Given an original input image and synthesized
image, we train the detector on the mask loss computed on the original image
and classification, regression loss on the synthesized image.

Method | Box | Mask
---|---|---
AP | AR10 | AR30 | AR100 | AP | AR10 | AR30 | AR100
OLN-Mask [29] | - | 18.3 | - | - | - | 16.9 | - | -
OLN-Faster [29] | 3.7 | 17.9 | 26.2 | 32.8 | - | - | - | -
Mask RCNN [22] | 8.9 | 16.0 | 19.3 | 20.9 | 7.2 | 13.7 | 16.4 | 17.7
$\text{Mask RCNN}^{\text{P}}$ | 7.1 | 16.0 | 19.1 | 20.7 | 5.4 | 13.8 | 16.3 | 17.7
$\text{Mask RCNN}^{\text{S}}$ | 8.3 | 17.4 | 22.6 | 27.1 | 6.0 | 15.1 | 19.7 | 23.7
LDET |  (+1.3) 10.2 |  (+5.5) 21.5 |  (+9.2) 28.5 |  (+13.9) 34.8 |  (+1.8) 9.0 |  (+6.0) 19.7 |  (+9.2) 25.6 |  (+13.3) 31.0

Table 1: Results of VOC $\rightarrow$ Non-VOC generalization in COCO.
Improvement on Mask RCNN is shown next to each result in the row of LDET. LDET
outperforms all baselines and showing large improvements on Mask RCNN. The
scores of OLN-Mask are reported in [29], those of OLN-Faster are obtained by
evaluating a published model. Figure 6: Visualization in VOC to Non-VOC in
COCO dataset. Top: Mask RCNN. Bottom: LDET. Note that training categories do
not include giraffe, trash box, pen, kite, and floats. LDET detects many novel
objects better than Mask RCNN.

We propose to bridge the gap between two domains, _i.e_., synthesized images
and real images, by computing instance mask loss on real images while
employing synthesized images only to compute proposal classification loss and
localization regression loss. The mask loss encourages a model to separate
background and foreground pixels given a bounding box, wherein only well-
localizing bounding boxes are selected to compute the loss. Thus, the model
will not learn to suppress hidden objects by using the mask loss. Furthermore,
the mask loss training signal is propagated to a backbone network shared among
the region proposal network, bounding box head, and mask head, which makes the
box head adapted to real images. Since the box head and proposal network are
trained only with the blended images without background objects, they excel at
detecting novel objects.

Foreground and background region sampling. During training, Mask RCNN (or
Faster RCNN) decides foreground and background regions by setting a threshold
on the value of IoU, _e.g_., regions with IoU smaller/larger than $0.5$ are
background/foreground respectively in the case of ROI head. When balancing the
number of two regions, boxes are randomly sampled. Note that we use the same
thresholds as default Mask RCNN. While localization quality estimation [29]
requires carefully selecting the thresholds during training, LDET does not.

##  4 Experiments

We evaluate LDET on two settings of open-world instance segmentation: Cross-
category and Cross-dataset. The Cross-category setting is based on the COCO
[34] dataset, where we split annotated classes into known and unknown classes,
train models on known ones, and evaluate detection/segmentation performance on
unknown ones. Since the model can be exposed to a new environment and
encounter novel instances, the Cross-dataset setting evaluates models’ ability
to generalize to new datasets. For this purpose, we adopt either COCO [34] or
Cityscapes [10] as a training source, with UVO [48] and Mappilary Vista [37]
being the test datasets, respectively. In this work, Average Precision (AP)
and Average Recall (AR) are employed for performance evaluation. The
evaluation is done in a class agnostic way. Unless otherwise specified, AR and
AP are computed following the COCO evaluation protocol, _i.e_., AP or AR at
100 detections at maximum.

Implementation. Detectron2 [49] is used to implement LDET. Mask R-CNN [22]
with ResNet-50 [23] as feature pyramid [32] backbone is used unless otherwise
specified. We utilize the standard hyperparameters of Mask R-CNN [22] as
defined in Detectron2. See appendix for more details. We will release the full
code to reproduce our results upon acceptance.

Baselines. Since open-world instance segmentation is a new task, we develop
several baselines as follows. See appendix for more details of baselines.

1) Mask R-CNN. We adopt the default model without changing any objectives or
input data. Comparison to this baseline will reveal the difference from
standard training.

2) $\text{Mask RCNN}^{\text{S}}$. We avoid sampling background regions with
hidden objects by sampling background boxes from the regions mostly inside the
ground truth boxes. We assume that these regions are less likely to contain
hidden objects. We compute the area of intersection with ground truth boxes
over the area of the proposal box and sample background boxes with a large
value of this criterion.

3) $\text{Mask RCNN}^{\text{P}}$. Inspired by [27], we implement a pseudo-
labeling based open-set instance segmentation baseline. The idea is to assign
pseudo-labels of foreground classes to the background regions (IoU with GT $<$
0.5) that have high objectness score. A model is trained to minimize the loss
on pseudo-labels.

4) OLN. OLN [29] is a concurrent work for open-world instance segmentation. We
adopt it for comparsions with our baseline models in the setting where they
report results or publish trained models.

Class agnostic inference. Since our goal is to detect objects in a scene
without classifying them into closed-set classes, class agnostic inference is
preferred. One way is to train a model in a class agnostic manner, _i.e_.,
regarding all classes as one foreground class, but this makes a model unable
to infer the class of detected objects. To achieve more flexible prediction,
we apply a class agnostic inference method to a class discriminative object
detector. Given the classification output of a region, we sum up all scores of
(known) foreground classes, deeming the result as an objectness score. Mask
and box regression are performed for the class with the maximum score. We also
compare class-agnostic inference between a class-discriminative model and a
class-agnostic model.

###  4.1 Cross-category generalization

Setup. We split the COCO dataset into 20 seen (VOC) classes and 60 unseen
(Non-VOC) classes. We train a model only on the annotation of 20 VOC classes
and test on Non-VOC classes only. The validation split of COCO is adopted for
evaluation. Following [29], we do not count the “seen class” detection boxes
into the budget k when computing the Average Recall (AR@k) scores. This is to
avoid evaluating any recall on seen-class objects.

Comparison with baselines. As shown in Table 1, our method outperforms
baselines in all metrics with a large margin. It can generate precise boxes as
well as detect many novel objects. OLN-Faster [29] demonstrates strong
performance in AR, but performs poorly in AP because it generates many false
positives. Similarly, $\text{Mask RCNN}^{\text{S}}$ improves performance on AR
while degrading AP. Both of these baselines are thresholding background
regions based on IoU with ground truth boxes, which may result in unbalanced
sampling with respect to the box size. For example, if we ignore background
boxes with small IoU, small regions are unlikely to become negatives and the
proposal network mistakenly detects many small background regions as
foreground. $\text{Mask RCNN}^{\text{P}}$ does not improve on Mask RCNN. Some
visualiations are illustrated in Fig. 6. Mask RCNN tends to overlook non-VOC
class objects even when they are in the dominant and salient region, _e.g_.,
trash can (leftmost column) and giraffes (second leftmost column). On the
other hand, LDET generalizes well to novel objects such as comb, towel,
giraffes, pen, phone, kites, and floats.

Ablation study for learning objectives. Table 2 describes an ablation study of
training data and objectives. If a model learns only from synthetic data
(second row), it fails to detect and segment objects. If a model is trained
simply by using both real and synthetic images, _i.e_., using all losses on
both domains, it shows slight improvements compared to Real Only. These
results conclude that our proposed decoupled training is very suitable for the
open-world instance segmentation and detection task.

Class agnostic vs. class discriminative learning. Table 3 shows that the class
agnostic training slighly improves performance in the case of both plain Mask
RCNN and LDET. The selection of the training method should depend on the
importance of the small gain and class discriminative prediction in an applied
recognition system.

Real | Synth | Decoupled | Box | Mask
---|---|---|---|---
Training | AP | AR10 | AR100 | AP | AR10 | AR100
✓ |  |  | 8.9 | 16.0 | 20.9 | 7.2 | 13.7 | 17.7
| ✓ |  | 0.2 | 1.5 | 4.7 | 0.1 | 0.4 | 0.7
✓ | ✓ |  | 8.9 | 16.5 | 21.9 | 7.1 | 14.1 | 18.6
✓ | ✓ | ✓ | 10.2 | 21.5 | 34.8 | 9.0 | 19.7 | 31.0

Table 2: Ablation study of data and training method in VOC $\rightarrow$ Non-
VOC. The last row is our proposed framework.

Method | Agnostic | Box | Mask
---|---|---|---
AP | AR10 | AR100 | AP | AR10 | AR100
Plain |  | 8.7 | 16.0 | 20.9 | 7.0 | 13.7 | 17.7
Plain | ✓ | 9.0 | 18.7 | 21.6 | 7.3 | 13.9 | 18.1
LDET |  | 10.2 | 21.5 | 34.9 | 9.0 | 19.7 | 31.0
LDET | ✓ | 10.2 | 21.8 | 35.6 | 9.2 | 20.2 | 32.0

Table 3: Ablation study for class agnostic training. Class agnostic training
slightly improves performance in both LDET and the plain Mask RCNN. Figure 7:
Baseline MaskRCNN suffers from overfitting to annotated instances. It thus
decreases the performance to detect novel objects as training proceeds. In
contrast, our approach basically gains performance with training iterations.

Method | Box | Mask
---|---|---
AR10 | AR100 | AR10 | AR100
Geodesic [30] | 4.0 | 18.0 | 2.3. | 12.3
Rigor [25] | - | 13.3 | - | 9.4
SelectiveSearch [46] | 5.2 | 16.3 | 2.5 | 9.5
MCG [2] | 10.1 | 24.6 | 7.7 | 18.6
DeepMask [39] | 15.2 | 30.6 | 12.3 | 23.3
LDET | 30.3 | 46.0 | 27.9 | 40.8

Table 4: Comparison to unsupervised methods and DeepMask tested on COCO.
Results of baselines are reported in DeepMask [39]. Note that DeepMask uses
VGG [44] as backbone. LDET and DeepMask are trained on VOC-COCO.

Mask RCNN overfits to seen classes. Fig. 7 illustrates the training iteration
and corresponding AR in MaskRCNN and LDET. AR gradually degrades in MaskRCNN
probably because the model fits to training data and learns to ignore hidden
objects. By contrast, LDET demonstrates a solid improvement in AR as the
training progresses. This implies that LDET can utilize the objectness of the
pretrained representation and thus generalize to more than the training
categories. We investigate this point in the appendix.

Comparison with hand-crafted methods. We compare LDET to the learning-free
methods in instance segmentation and detection Table 4. We measure the
performance of detecting all objects in the COCO validation set. LDET is
superior to these baselines by a large margin.

Visualization of the learnt objectness map. Fig. 8 visualizes the confidence
scores map of the region proposal network, computed by averaging outputs from
all feature pyramids. While Mask RCNN suppresses the score for many objects,
LDET correctly captures objectness for diverse objects. For instance, it
captures objectness well in an image crowded with objects in the first row.

Background ratio | AP${}_{\text{{box}}}$ | AR${}_{\text{{box}}}$ | AP${}_{\text{{mask}}}$ | AR${}_{\text{{mask}}}$
---|---|---|---|---
$2^{-1}$ | 10.8 | 32.5 | 9.0 | 28.9
$2^{-2}$ | 10.5 | 32.0 | 9.0 | 28.4
$2^{-3}$ | 10.2 | 34.8 | 9.0 | 31.0
$2^{-4}$ | 9.6 | 35.2 | 8.8 | 31.9

Table 5: Varying the size of background regions. $2^{-m}$ indicates cropping
background region with $2^{-m}$ of width and height of an input image.
Sampling background from smaller region tends to improve AR and degrade AP.

Size of background regions. The size of background regions can be important in
our data augmentation: if the size is close to an original image, the
background will include many hidden objects. We analyze the effect of the
region’s size in Table 5. AR gets better with smaller sizes, while AP gets
slightly worse. This indicates that smaller backgrounds prevent sampling
hidden objects and coverage of novel objects (AR) becomes better. Also, note
that the performance is stable with respect to the size.

External data for background. Using an external background dataset is an
alternative way to synthesize background region although an image without
background is not always accessible. In this experiment, we use DTD [9]
(texture image dataset). The background region is replaced with a randomly
cropped patch from DTD dataset, and a model is trained in the same way as
LDET. See the appendix for more details of training. The resulting AP and AR
are 11.1 ($\text{LDET}+0.9$) and 32.6 ($\text{LDET}-2.2$) respectively in
bounding box localization. The dataset includes a considerable number of
objects despite being primarily a texture dataset, which is probably the cause
of degradation in AR. Our data augmentation method is comparable or better
than the use of external background images.

Method | Backbone | Box | Mask
---|---|---|---
AP | AR100 | AP | AR100
MRCNN | Res50 | 24.8 | 47.7 | 19.1 | 38.1
MRCNN | Res101 |  (+1.0) 25.8 |  (-0.1) 47.6 |  (+1.2) 20.3 |  (+0.6) 38.7
LDET | Res50 | 26.1 | 53.1 | 21.2 | 43.0
LDET | Res101 |  (+1.7) 27.8 |  (+0.1) 53.2 |  (+1.9) 23.1 |  (+1.1) 44.1

Table 6: ResNet50 vs ResNet101. ResNet101 tends to perform better than
ResNet50, which is more evident in LDET than in Mask RCNN (MRCNN).

Different backbones. We employ ResNet101 [23] as a backbone in Table 6 and
compare it with ResNet50. LDET benefits more from the deeper backbone.

Method | Detector | AP | AR10 | AR100
---|---|---|---|---
MRCNN | RPN | 5.9 | 15.4 | 27.3
MRCNN | ROI |  (+3.0) 8.9 |  (+0.6) 16.0 |  (-6.4) 20.9
LDET | RPN | 6.5 | 18.2 | 32.8
LDET | ROI |  (+3.7) 10.2 |  (+3.3) 21.5 |  (+2.0) 34.8

Table 7: Comparison between region proposal network and region of interest
head. AP and AR of bounding boxes.

Region proposal network (RPN) and region of interest (ROI) head. We compare
the performance of RPN and ROI heads in Table 7. In Mask RCNN, RPN covers more
novel objects while ROI provides precise boxes compared to RPN. In contrast,
ROI of LDET performs better than RPN in all metrics.

Figure 8: Objectness map (RPN score) visualization in COCO experiment. LDET
captures objectness of various categories whereas Mask RCNN tends to suppress
many objects.

Method | Train | Box | Mask
---|---|---|---
AP | AR |  AR${}_{\text{small}}$ |  AR${}_{\text{med}}$ |  AR${}_{\text{large}}$ | AP | AR |  AR${}_{\text{small}}$ |  AR${}_{\text{med}}$ |  AR${}_{\text{large}}$
OLN-Faster [29] |  VOC (COCO) | 13.3 | 47.6 | 25.6 | 43.5 | 57.8 | - | - | - | - | -
Mask RCNN | 19.6 | 34.8 | 10.9 | 23.9 | 50.8 | 14.6 | 25.8 | 9.2 | 19.4 | 36.1
$\text{Mask RCNN}^{\text{P}}$ | 19.0 | 34.8 | 10.7 | 23.7 | 51.0 | 14.5 | 26.0 | 9.3 | 19.4 | 36.4
$\text{Mask RCNN}^{\text{S}}$ | 18.9 | 39.2 | 12.8 | 28.8 | 55.6 | 12.9 | 29.8 | 11.7 | 23.9 | 40.1
LDET | 22.8 | 48.9 | 23.1 | 42.4 | 62.1 | 17.9 | 38.0 | 20.8 | 35.2 | 45.6
Mask RCNN |  COCO | 24.8 | 47.7 | 21.9 | 41.7 | 60.6 | 19.1 | 38.1 | 19.1 | 35.0 | 46.5
$\text{Mask RCNN}^{\text{P}}$ | 23.8 | 47.2 | 22.1 | 41.1 | 59.9 | 18.8 | 37.6 | 19.1 | 34.6 | 45.7
$\text{Mask RCNN}^{\text{S}}$ | 21.3 | 47.9 | 21.1 | 40.8 | 62.0 | 15.6 | 38.6 | 19.3 | 34.9 | 47.6
LDET | 26.1 | 53.1 | 28.4 | 47.8 | 65.0 | 21.2 | 43.0 | 25.7 | 41.1 | 49.9

Table 8: Results of COCO $\rightarrow$ UVO generalization. Top rows: Models
trained on VOC-COCO. Bottom rows: Models trained on COCO. LDET demonstrates
high AP and AR in all cases compared to baselines.

Method | Box | Mask
---|---|---
AP | AR | AR${}_{\text{0.5}}$ | AP | AR | AR${}_{\text{0.5}}$
Mask RCNN | 8.3 | 9.7 | 14.6 | 7.3 | 8.4 | 13.5
$\text{Mask RCNN}^{\text{P}}$ | 8.2 | 9.7 | 14.5 | 7.2 | 8.4 | 13.4
$\text{Mask RCNN}^{\text{S}}$ | 6.7 | 9.6 | 14.7 | 6.1 | 8.7 | 13.9
LDET | 8.5 | 11.4 | 19.0 | 7.7 | 10.0 | 16.8

Table 9: Results of Cityscapes $\rightarrow$ Mappilary Vista generalization.
LDET is effective for autonomous driving dataset. AR${}_{\text{0.5}}$ denotes
AR with IoU threshold $=0.5$ Figure 9: Visualization of results for models
trained on COCO. Top: Mask RCNN. Bottom: LDET. Leftmost two images are from
UVO, the others are from COCO validation images.

###  4.2 Cross-dataset generalization

COCO to UVO. First, we utilize UVO [48], which covers many categories outside
COCO as 57% object instances do not belong to any of the 80 COCO classes.
Since UVO is based on Youtube videos, the appearance is very different from
COCO _e.g_., some videos are egocentric views and have significant motion
blur. We test models trained on the COCO-VOC split or the whole COCO. The
validation split of UVO is used to measure the performance.

As shown in Table 8, in both COCO-VOC and COCO settings, LDET outperforms
baselines with a large margin. Note that our VOC-COCO model outperforms Mask
RCNN trained on COCO in many metrics. This indicates the remarkable label
efficiency of LDET in open-world instance segmentation. Unlike the result in
VOC-NonVOC experiment, the AP of $\text{Mask RCNN}^{\text{S}}$ drops compared
to Mask RCNN, probably because their region sampling leads imbalanced sampling
with regions with different scales.

Cityscape to Mapillary. Second, we examine performance in autonomous driving
scenes. Detectors are trained on Cityscape [10] (8 foreground classes, person,
rider, car, truck, bus, train, motorcycle, and bicycle) and tested on the
validation set of Mapillary Vistas [37] with 35 foreground classes including
not only vehicles, but also animals, trash can, mailbox, and so on.

In Table 9, LDET shows solid gains over baselines, though this setting is very
challenging. Note that the model is trained only on 8 classes and is required
to generalize to all the 35 classes in the test data, which explains the lower
performance compared to experiments on COCO. We acknowledge that this is a
limitation of LDET and discuss this further in Sec. 5. The result demonstrates
that LDET generalizes to datasets other than COCO and can be useful for
autonomous driving systems.

Visualization. As Fig. 1 shows, Mask RCNN detector fails in localizing objects
unseen in their 80 categories while our detector shows surprisingly good
generalization. Note that in the second leftmost row, it recognizes a
character drawn on the wall, which is clearly outside COCO categories. Fig. 9
visualizes other examples from UVO and COCO.

##  5 Conclusion

In this paper, we presented a simple approach, LDET, for the challenging task
of open-world instance segmentation. LDET consists of synthesizing images
without hidden objects in their background as well as decoupled training for
real and synthesized images. LDET demonstrates strong performance on a
benchmark dataset of open-world instance segmentation and promising results on
autonomous driving datasets. We hope that LDET becomes a simple baseline and
accelerates further research in this area.

Potential negative societal impact. Some applications of object detection may
have a negative impact on society, such as excessive surveillance and
inappropriate policing. Privacy is another concern. We note that our approach
can help remove accidental people in the background of training images and
thus improve privacy.

Limitations. As seen in several visualizations, LDET still fails in detecting
some novel objects although its performance is much better than baselines. If
the appearance of novel objects is distinct from known objects, LDET and most
baselines may miss them. Also, experiments on Cityscapes (Table 9) indicate
the importance of covering various categories in training data. One way to
overcome this limitation is to annotate a wide range of categories for
training data.

Acknowledgments. This work was supported by DARPA LwLL and NSF Award No.
1535797. We thank Donghyun Kim and Piotr Teterwak for giving feedback on the
draft.

##  Appendix A Experimental Details

In this appendix, we provide experimental details, additional analysis, and
visualizations. Data augmentation. Alg. 1 shows the pytorch-style pseudo-code
for our data augmentation. See our code for more details.

Implementation. We use the default hyper-parameters, provided by Detectron2,
to train LDET, Mask RCNN, $\text{Mask RCNN}^{\text{S}}$, and $\text{Mask
RCNN}^{\text{P}}$. Default configuration files are used for COCO
111https://github.com/facebookresearch/detectron2/blob/main/configs/COCO-
InstanceSegmentation/mask_rcnn_R_50_FPN_1x.yaml and Cityscapes
222https://github.com/facebookresearch/detectron2/blob/main/configs/Cityscapes/mask_rcnn_R_50_FPN.yaml
respectively. 2 GPUs of NVIDIA RTX A6000 with 48GB are used to train models.

Test time hyper-parameters. Following [29], we set hyper-parameters in testing
as follows: threshold of non-maximum suppression in the roi head is set as 0.7
(0.5 by default) and, in the region proposal network, the number of region
proposals kept after the suppression is set as 2000 (1000 by default). Table A
shows the performance difference by the hyper-parameters. AP tends to improve
by decreasing the threshold value and degrade by increasing the number of
proposals. Low threshold suppresses more boxes, which results in improving AP
and degrading AR. Changing the number of proposals has a similar effect. We
can observe that the difference is not significant.

NMS Threshold | Num of Boxes | Box | Mask
---|---|---|---
AP | AR | AP | AR
0.5 | 1000 | 10.5 | 33.3 | 9.3 | 30.3
0.7 | 1000 | 10.3 | 34.9 | 9.1 | 31.0
0.5 | 2000 | 10.5 | 33.6 | 9.3 | 30.7
0.7 | 2000 | 10.2 | 34.8 | 9.0 | 31.0

Table A: Study on the threshold for no-maximum suppression and the number of
boxes kept after region proposal network. The last row is the setting used in
the main paper.

# scale=$\frac{1}{8}$: the size of background region to crop.

# M: mask of the foreground regions.

# Apply gaussian smoothing.

image = gaussian(image)

w, h = image.shape

# Randomly crop background with the specified size.

backg = randomcrop(image, w*scale, h*scale)

# Upscale to the size of the input.

backg = upscale(backg, scale)

# Downsample the input.

image = downsample(image, scale)

# Upsample to the original size.

image = upscale(image, scale)

# Paste foreground objects on the synthesized background.

image = M * image + (1 - M) * backg

# Apply smoothing.

image = smooth(image)

Algorithm 1 PyTorch-style pseudocode for our data augmentation

Method |  Pre-training Data |  Pre-training Method | VOC-to-COCO | VOC-to-UVO
---|---|---|---|---
Box | Mask | Box | Mask
AP | AR | AP | AR | AP | AR | AP | AR
MRCNN | None | None | 6.1 | 24.9 | 4.6 | 21.5 | 16.6 | 37.8 | 11.3 | 27.7
MRCNN | ImageNet1K | SWAV [6] | 7.1 | 25.3 | 5.5 | 22.1 | 15.8 | 37.1 | 10.8 | 26.5
MRCNN | ImageNet1K | Supervised | 8.9 | 20.9 | 7.2 | 17.7 | 19.6 | 34.8 | 14.6 | 25.8
MRCNN | ImageNet21K | MIIL [42] | 8.8 | 20.9 | 7.0 | 17.9 | 19.3 | 35.0 | 14.4 | 25.9
LDET | None | None | 5.7 | 29.5 | 4.2 | 25.2 | 15.8 | 42.7 | 9.9 | 30.1
LDET | ImageNet1K | SWAV [6] | 5.5 | 29.9 | 4.7 | 27.1 | 13.9 | 41.2 | 10.6 | 30.8
LDET | ImageNet1K | Supervised | 10.2 | 34.8 | 9.0 | 31.0 | 22.8 | 48.9 | 17.9 | 38.0
LDET | ImageNet21K | MIIL [42] | 9.7 | 35.3 | 8.6 | 31.9 | 21.3 | 48.1 | 17.1 | 37.6

Table B: Comparison of pre-training methods. Replacing supervised pre-trained
model with SWAV (unsupervised) pre-training degrades AP in both training
methods. SWAV performs better on AR of Mask RCNN.

Method | Box
---|---
AP | AR10 | AR30 | AR50 | AR100
RetinaNet [33] | 9.0 | 16.5 | 21.0 | 22.9 | 25.0
LDET | 6.8 | 18.9 | 26.1 | 29.2 | 32.9

Table C: Results of VOC $\rightarrow$ Non-VOC generalization in COCO using
RetinaNet as a detector. Figure A: Visualization for detectors trained on
Cityscapes. Leftmost two images are validation images of Cityscapes, rightmost
two are from Mapillary.

Method | Box | Mask
---|---|---
AP | AR10 | AR30 | AR100 | AP | AR10 | AR30 | AR100
TensorMask [8] | 9.6 | 17.8 | 22.9 | 26.6 | 7.4 | 14.8 | 18.8 | 22.0
LDET |  (-0.8) 8.8 |  (+3.0) 20.8 |  (+4.9) 27.8 |  (+7.8)34.4 |  (-0.4) 7.0 |  (+3.4) 18.2 |  (+5.3) 24.1 |  (+7.7) 29.7

Table D: Results of Tensormask for VOC $\rightarrow$ Non-VOC generalization in
COCO.

Baselines.

1) Mask R-CNN. We do not make any change to the default training
configuration.

2) $\text{Mask RCNN}^{\text{S}}$. We compute the area of intersection with
ground truth boxes over the area of the proposal box, which we call IoA, and
sample background boxes with a large value of this criterion. In both region
proposal network and roi head, we pick background regions whose IoA is larger
than 0.7.

3) $\text{Mask RCNN}^{\text{P}}$. Given the classification output (after
softmax) from roi head, boxes confidently predicted as one of the foreground
classes are choosen from background regions. The threshold to pick the pseudo-
foreground is set as 0.9. The classification loss on the pseudo-foreground
regions is incorporated to train the detector.

Experiments on texture dataset. To make a background using images of DTD [9],
we crop the patch with the size of 256 x 256, and rescale it to the size of a
detection training image. Then, we blend the foreground and background in the
same way as LDET.

##  Appendix B Analysis

In this section, we provide some additional analysis of 1) model pre-training
and 2) using a one-stage detector.

Pre-training the model. In the experiment in Sec. 4, “Mask RCNN overfits to
seen classes”, we saw that the plain Mask RCNN model seems to be overfitting
to the training data, while LDET generalizes better to novel objects. We
hypothesize that LDET is better able to use the representations (features) of
objects contained in the pre-trained backbone. In this experiment, we explore
this further and provide results of different pre-training methods and data.
Specifically, we explore whether pre-training on supervised classification
data gives our method a boost compared to unsupervised or no pretraining.

First, we train a model from scratch on the VOC-COCO data. Second, we use a
self-supervised learning model, SWAV [6], trained on ImageNet1k [43] to
initialize models. Third, we use the default model, trained on ImageNet1k in a
supervised manner. Finally, a model trained on ImageNet21k [12] with a method
that considers the hierarchies of the categories is employed. We expect that
comparison on these pre-trained models will show the importance of 1) class-
discriminative features and 2) the number of categories used for pre-training.

Table B provides the results in the VOC-COCO setting. First, we see that
supervised pre-training improves AP in all cases. Second, in Mask RCNN,
unsupervised/no pre-training gives better AR than supervised pre-training. By
contrast, in LDET, supervised pre-training outperforms unsupervised/no pre-
training in both AP and AR. This implies that LDET harnesses the class
discriminative features learned by supervised training better than the
standard detector training does. Finally, we do not see a clear difference in
the performance between ImageNet21k and ImageNet1k, neither in Mask RCNN nor
LDET. Note that top-1 accuracy on ImageNet1k is 75.3 (Supervised) and 82.0
(MIIL) respectively. Better accuracy on ImageNet1k does not necessarily lead
to better performance in open-world instance segmentation. Also, increasing
pre-training categories beyond 1k does not show improvement in this
experiment.

One-stage detector. The experiments in the main draft are done for a two-stage
detector, Mask RCNN. In this analysis, we apply LDET to one-stage detectors
and see its behavior.

Table C shows the results on RetinaNet [33]. Since the standard RetinaNet does
not have an instance mask head, the mask head is attached on top of the
feature pyramid of the detector. The architecture of the head is the same as
Mask RCNN. Therefore, the detection loss _e.g_., localization and
classification loss, is computed for synthesized images, whereas the mask loss
is calculated for real images. LDET outperforms the plain model with a large
margin in AR while we see a degradation in AP. To maintain the precision, it
may be necessary to tune some hyper-parameters such as the size of background
region, or to tailor the training objective to the one-stage detector.

The results on Tensormask [8] are shown in Table D. Since the objectives of
Tensormask consist of detection loss and mask loss, synthesized images are
used to compute detection loss whereas real images are used to compute mask
loss. The trend is similar to RetinaNet where AP slightly decreases whereas AR
improves significantly. The empirical results of RetinaNet and Tensormask
indicate the usefulness of our framework for one-stage detectors.

Figure B: Visualization in VOC-COCO to COCO setting. Note that VOC-COCO does
not contain objects such as lizard, toilet paper, and elephant.  Figure C:
Visualization of models trained on COCO. The images are from COCO and UVO.

Figure D: Video demo of models trained on COCO. Left: Mask RCNN. Right: LDET.
Click the image to play the video.

Figure E: Video demo of models trained on COCO. Left: Mask RCNN. Right:
LDET.Click the image to play the video.

##  Appendix C Visualization

Cityscapes. Fig. A visualizes some qualitative results. Leftmost two images
are from the validation set of Cityscapes, others are from Mapillary. We see
that, as indicated by the quantitative results, LDET detects more objects,
_e.g_., baby carriage in the leftmost image. However, it is also true that
LDET misses novel objects such as dog in the leftmost image, probably because
there are no categories similar to dogs in the Cityscapes’ 8 training
categories. This fact indicates some room for improvement in our approach.

More visualizations in COCO. Fig. B and C are additional visualizations in
VOC-COCO and COCO, respectively. Note that we add the results of $\text{Mask
RCNN}^{\text{S}}$, which are not visualized in the main paper due to a limited
space. $\text{Mask RCNN}^{\text{S}}$ locates many novel objects while
generating many false positives. This is probably due to the imbalanced
sampling of background regions. By contrast, LDET detects many novel objects,
_e.g_., elephants, toilet paper, lizard, statue, toy, _etc_., with small
number of false positives.

Demo on video. Fig. D and E are demo of applying LDET to UVO [48] videos.
Click the images to play the videos.

## References

  * [1] Bogdan Alexe, Thomas Deselaers, and Vittorio Ferrari.  Measuring the objectness of image windows.  IEEE TPAMI, 34(11):2189–2202, 2012.
  * [2] Pablo Arbeláez, Jordi Pont-Tuset, Jonathan T Barron, Ferran Marques, and Jitendra Malik.  Multiscale combinatorial grouping.  In CVPR, pages 328–335, 2014.
  * [3] Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran.  Zero-shot object detection.  In ECCV, pages 384–400, 2018.
  * [4] Abhijit Bendale and Terrance E Boult.  Towards open set deep networks.  In CVPR, pages 1563–1572, 2016.
  * [5] Zhaowei Cai and Nuno Vasconcelos.  Cascade r-cnn: Delving into high quality object detection.  In CVPR, pages 6154–6162, 2018.
  * [6] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  arXiv preprint arXiv:2006.09882, 2020.
  * [7] Jun Cen, Peng Yun, Junhao Cai, Michael Yu Wang, and Ming Liu.  Deep metric learning for open world semantic segmentation.  In ICCV, 2021.
  * [8] Xinlei Chen, Ross Girshick, Kaiming He, and Piotr Dollar.  Tensormask: A foundation for dense object segmentation.  In ICCV, 2019.
  * [9] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.  Describing textures in the wild.  In CVPR, pages 3606–3613, 2014.
  * [10] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In CVPR, 2016.
  * [11] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  In ICCV, pages 764–773, 2017.
  * [12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, pages 248–255. Ieee, 2009.
  * [13] Akshay Dhamija, Manuel Gunther, Jonathan Ventura, and Terrance Boult.  The overlooked elephant of object detection: Open set.  In WACV, pages 1021–1030, 2020.
  * [14] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox.  Flownet: Learning optical flow with convolutional networks.  In ICCV, pages 2758–2766, 2015.
  * [15] Nikita Dvornik, Julien Mairal, and Cordelia Schmid.  Modeling visual context is key to augmenting object detection datasets.  In ECCV, pages 364–380, 2018.
  * [16] Debidatta Dwibedi, Ishan Misra, and Martial Hebert.  Cut, paste and learn: Surprisingly easy synthesis for instance detection.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1301–1310, 2017.
  * [17] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  IJCV, 88(2):303–338, 2010.
  * [18] Pedro F Felzenszwalb, Ross B Girshick, and David McAllester.  Cascade object detection with deformable part models.  In CVPR, pages 2241–2248. Ieee, 2010.
  * [19] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph.  Simple copy-paste is a strong data augmentation method for instance segmentation.  In CVPR, pages 2918–2928, 2021.
  * [20] Ross Girshick.  Fast r-cnn.  In ICCV, pages 1440–1448, 2015.
  * [21] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In CVPR, pages 580–587, 2014.
  * [22] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In ICCV, 2017.
  * [23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [24] Ronghang Hu, Piotr Dollár, Kaiming He, Trevor Darrell, and Ross Girshick.  Learning to segment every thing.  In CVPR, pages 4233–4241, 2018.
  * [25] Ahmad Humayun, Fuxin Li, and James M Rehg.  Rigor: Reusing inference in graph cuts for generating object regions.  In CVPR, pages 336–343, 2014.
  * [26] Ayush Jaiswal, Yue Wu, Pradeep Natarajan, and Premkumar Natarajan.  Class-agnostic object detection.  In WACV, pages 919–928, 2021.
  * [27] KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian.  Towards open world object detection.  In CVPR, pages 5830–5840, 2021.
  * [28] Bingyi Kang, Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell.  Few-shot object detection via feature reweighting.  In ICCV, pages 8420–8429, 2019.
  * [29] Dahun Kim, Tsung-Yi Lin, Anelia Angelova, In So Kweon, and Weicheng Kuo.  Learning open-world object proposals without learning to classify.  arXiv preprint arXiv:2108.06753, 2021.
  * [30] Philipp Krähenbühl and Vladlen Koltun.  Geodesic object proposals.  In ECCV, pages 725–739. Springer, 2014.
  * [31] Weicheng Kuo, Anelia Angelova, Jitendra Malik, and Tsung-Yi Lin.  Shapemask: Learning to segment novel objects by refining shape priors.  In ICCV, pages 9207–9216, 2019.
  * [32] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In CVPR, pages 2117–2125, 2017.
  * [33] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In ICCV, pages 2980–2988, 2017.
  * [34] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [35] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  Ssd: Single shot multibox detector.  In ECCV, 2016.
  * [36] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu.  Large-scale long-tailed recognition in an open world.  In CVPR, pages 2537–2546, 2019.
  * [37] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder.  The mapillary vistas dataset for semantic understanding of street scenes.  In ICCV, pages 4990–4999, 2017.
  * [38] Pramuditha Perera, Vlad I Morariu, Rajiv Jain, Varun Manjunatha, Curtis Wigington, Vicente Ordonez, and Vishal M Patel.  Generative-discriminative feature representations for open-set recognition.  In CVPR, pages 11814–11823, 2020.
  * [39] Pedro O Pinheiro, Ronan Collobert, and Piotr Dollár.  Learning to segment object candidates.  arXiv preprint arXiv:1506.06204, 2015.
  * [40] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In CVPR, 2016.
  * [41] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In NeurIPS, 2015.
  * [42] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor.  Imagenet-21k pretraining for the masses.  arXiv preprint arXiv:2104.10972, 2021.
  * [43] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  IJCV, 115(3):211–252, 2015.
  * [44] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [45] Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin.  Csi: Novelty detection via contrastive learning on distributionally shifted instances.  arXiv preprint arXiv:2007.08176, 2020.
  * [46] Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders.  Selective search for object recognition.  IJCV, 104(2):154–171, 2013.
  * [47] Gül Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev, and Cordelia Schmid.  Learning from synthetic humans.  In CVPR, 2017.
  * [48] Weiyao Wang, Matt Feiszli, Heng Wang, and Du Tran.  Unidentified video objects: A benchmark for dense, open-world segmentation.  arXiv preprint arXiv:2104.04691, 2021.
  * [49] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * [50] Ryota Yoshihashi, Wen Shao, Rei Kawakami, Shaodi You, Makoto Iida, and Takeshi Naemura.  Classification-reconstruction learning for open-set recognition.  In CVPR, pages 4016–4025, 2019.
  * [51] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Scene parsing through ade20k dataset.  In CVPR, pages 633–641, 2017.
  * [52] C Lawrence Zitnick and Piotr Dollár.  Edge boxes: Locating object proposals from edges.  In ECCV, pages 391–405. Springer, 2014.


∎  11institutetext: Département de Physique, Ecole Normale Supérieure Paris-
Saclay, 61 av du président Wilson, F-94235 Cachan, France
11email<EMAIL_ADDRESS>
# Generalization of Einstein’s gravitational field equations

Frédéric Moulin

(November 2017)

###### Abstract

The Riemann tensor is the cornerstone of general relativity, but as everyone
knows it does not appear explicitly in Einstein’s equation of gravitation.
This suggests that the latter may not be the most general equation. We propose
here for the first time, following a rigorous mathematical treatment based on
the variational principle, that there exists a generalized 4-index
gravitational field equation containing the Riemann curvature tensor linearly,
and thus the Weyl tensor as well. We show that this equation, written in $n$
dimensions, contains the energy-momentum tensor for matter and also that of
the gravitational field itself. This new 4-index equation remains completely
within the framework of general relativity and emerges as a natural
generalization of the familiar 2-index Einstein equation. Due to the presence
of the Weyl tensor, we show that this equation contains much more information,
which fully justifies the use of a fourth-order theory.

###### Keywords:

Extension of general relativity Variational principle Four-index Einstein
equation Weyl tensor Four-index energy-momentum tensor

††journal: Eur. Phys. J. C

##  1 Introduction

Faisons maintenant varier l’action par rapport à la métrique:
The Riemann curvature tensor of general relativity $R_{ijkl}$ can be split
into the Weyl conformal tensor $C_{ijkl}$, and parts which involve only the
Ricci tensor $R_{jl}$ and the curvature scalar $R$. Because of the properties
of the Weyl tensor, its contraction vanishes, $g^{ik}C_{ijkl}=0$, and thus the
information it contains (namely the information about the gravitational field
in vacuum) is not present in the famous Einstein equation.
The aim of this paper is to find a generalized gravitational field equation
explicitly containing the Riemann curvature tensor linearly. For this, we have
implemented a rigorous mathematical treatment with a classical variational
principle using a generalized Lagrangian containing $R_{ijkl}$, $R_{jl}$ and
$R$.
The paper is organized as follows:
In section 2 we generalize the Einstein-Hilbert Lagrangian of general
relativity by introducing new scalars constructed from $R_{ijkl}$ and
$R_{jl}$.
In sections 3 and 4 we apply the principle of least action to this Lagrangian
and obtain the generalized 4-index Einstein equation written with the total
energy-momentum tensor, $T_{ijkl}=T^{(M)}_{ijkl}+T^{(F)}_{ijkl}$, where
$T^{(M)}_{ijkl}$ is the energy-momentum tensor for matter, and
$T^{(F)}_{ijkl}$ the energy-momentum tensor for the gravitational field
itself. We show that the tensor contraction of this new generalized version of
the Einstein equation restores the usual 2-index general relativity equation.
In section 5 we impose total energy-momentum conservation, and show that the
generalized equation can be written with the double dual Riemann tensor
${}^{*}R_{ijkl}^{*}$. In the last part, the cosmological constant is also
introduced.

##  2 Lagrangian formulation

We want that the new equation of general relativity necessarily contains the
Riemann tensor linearly, and thus be a fourth-order tensor equation with the
same index symmetries as $R_{ijkl}$. To be as general as possible, this
equation must also contain fourth-order tensors constructed from the Ricci
tensor $R_{jl}$ and also from the scalar curvature $R$. To apply the principle
of least action, we must first define a very general gravitational action,
$S_{(G)}$, written with the Lagrangian, $L_{(G)}$, including all these various
terms linearly:

$S_{(G)}=-\frac{1}{2\chi c}\,\int
L_{(G)}(R_{ijkl},R_{jl},R)\sqrt{-g}\,d\Omega$ (1)

where $\chi=8\pi G/c^{4}$ is the Einstein constant.

###  2.1 Riemann tensor symmetries

The Riemann tensor symmetries are well known in the context of general
relativity; antisymmetry on the first two indices, antisymmetry on the last
two indices and symmetry obtained by exchanging the first pair with the second
pair citLandau ; citMisner :

$R_{ijkl}=-R_{jikl}=-R_{ijlk}=R_{klij}$ (2)

The Ricci tensor, which comes from the contraction of the Riemann tensor
$R_{jl}=g^{ik}R_{ijkl}$, is therefore symmetric on its two indices, and the
scalar curvature $R=g^{jl}R_{jl}$, obviously does not have any symmetry. To
construct two fourth-order tensors from $R_{jl}$ and $R$, and having exactly
the same symmetries as the Riemann tensor, we shall naturally take a
combination involving metric tensors. It is easy to check that the
combinations $(g_{ik}R_{jl}-g_{jk}R_{il}+g_{jl}R_{ik}-g_{il}R_{jk})$ and
$(g_{ik}g_{jl}R-g_{il}g_{jk}R)$ are unique and obey exactly the same
symmetries as in (2). For convenience in what follows, we define the tensor:
$g_{ijkl}=g_{ik}g_{jl}-g_{il}g_{jk}$, and we note:

$\displaystyle g_{ik}R_{jl}-g_{jk}R_{il}+g_{jl}R_{ik}-g_{il}R_{jk}$
$\displaystyle=g_{ijkp}R^{p}{}_{l}+g_{ijpl}R^{p}{}_{k}$ (3) $\displaystyle
g_{ik}g_{jl}R-g_{il}g_{jk}R$ $\displaystyle=g_{ijkl}R$ (4)

These combinations of metric tensors are often found in many well known
references, citLandau ; citMisner ; citWeinberg ; citEisenhart ; citCaroll ;
citBlau  and it is therefore these specific fourth-order tensors we will use
in our calculations.

###  2.2 Lagrangian terms

The Einstein-Hilbert Lagrangian of the general relativity is purely
gravitational and is defined by the scalar curvature citLandau ; citMisner :

$L_{(G)}=R$ (5)

However, it is well known that to obtain Einstein’s familiar 2-index equation
by a least action principle, we have to perform the calculations using a
action including a Lagrangian written with the Ricci tensor,
$L_{(G)}=g^{ik}R_{ik}$:

$\displaystyle S_{(G)}$ $\displaystyle=-\frac{1}{2\chi c}\,\int
R\sqrt{-g}\,d\Omega$ $\displaystyle=-\frac{1}{2\chi c}\,\int
g^{ik}R_{ik}\sqrt{-g}\,d\Omega$ (6)

In this paper, we remain within the familiar framework of general relativity,
and so it is also physically equivalent to perform the calculations using a
generalized action constructed from the three interesting tensors, (2), (3)
and (4):

$\displaystyle S_{(G)}=$ $\displaystyle-\frac{1}{2\chi c}\,\int
g^{ik}g^{jl}\,[a_{1}R_{ijkl}+{a_{2}}\,(g_{ijkp}R^{p}{}_{l}$
$\displaystyle+g_{ijpl}R^{p}{}_{k})+{a_{3}}\,g_{ijkl}R\,]\sqrt{-g}\,d\Omega$
(7)

where $a_{1}$, $a_{2}$, $a_{3}$ are three arbitrary parameters that can be
determined by contraction and identification with relations (6):

$a_{2}=\frac{(1-a_{1})}{(n-2)}\quad\mbox{,}\quad
a_{3}=-\frac{(1-a_{1})}{(n-1)(n-2)}\,$ (8)

Here, we have used the following mathematical relationships for the tensorial
contractions:

$\displaystyle g^{jl}R_{ijkl}$ $\displaystyle=R_{ik}$ (9) $\displaystyle
g^{jl}(g_{ijkp}R^{p}{}_{l}+g_{ijpl}R^{p}{}_{k})$
$\displaystyle=(n-2)R_{ik}+g_{ik}R$ (10) $\displaystyle g^{jl}(g_{ijkl}R)$
$\displaystyle=(n-1)g_{ik}R$ (11) $\displaystyle g^{ik}g^{jl}R_{ijkl}$
$\displaystyle=R$ (12) $\displaystyle
g^{ik}g^{jl}(g_{ijkp}R^{p}{}_{l}+g_{ijpl}R^{p}{}_{k})$ $\displaystyle=2(n-1)R$
(13) $\displaystyle g^{ik}g^{jl}(g_{ijkl}R)$ $\displaystyle=n(n-1)R$ (14)

with $g^{jl}g_{jl}=\delta_{j}^{j}=n$ in a n-dimensional space.
The two relations (8) therefore, allow us to write a generalized Lagrangian in
a form that involves only one parameter $a$ (with $a=a_{1}$):

$\displaystyle L_{(G)}=$
$\displaystyle\,g^{ik}g^{jl}\,\big{[}\,aR_{ijkl}+\frac{(1-a)}{(n-2)}\,(g_{ijkp}R^{p}{}_{l}+g_{ijpl}R^{p}{}_{k})$
$\displaystyle-\frac{(1-a)}{(n-1)(n-2)}\,g_{ijkl}R\,\big{]}$ (15)

This new Lagrangian is a natural generalization, and we can verify that the
contraction with $g^{ik}g^{jl}$ naturally restores the Einstein-Hilbert
Lagrangian, $L_{(G)}=R$, whatever the values of $a$ and $n$. This Lagrangian
is therefore physically compatible with the general theory of relativity and a
4-index Einstein equation will be obtained in the next chapter by a
variational principle using the relation (15).

##  3 Principle of least action

It is well known that the total action $S$ is the sum of a purely
gravitational Einstein-Hilbert action $S_{(G)}$, and a matter-field action
$S_{(MF)}$ which describes any matter and fields living on the space-time
citLandau :

$\displaystyle S=S_{(G)}+S_{(MF)}=$ $\displaystyle-\frac{1}{2\chi c}\,\int
L_{(G)}\sqrt{-g}\,d\Omega$ $\displaystyle+\frac{1}{c}\,\int
L_{(MF)}\sqrt{-g}\,d\Omega$ (16)

In order to obtain the gravitation field equation, we should vary the total
action with respect to the metric:

$\displaystyle\delta S=\delta S_{(G)}+\delta S_{(MF)}=$
$\displaystyle-\frac{1}{2\chi c}\,\delta\int L_{(G)}\sqrt{-g}\,d\Omega$
$\displaystyle+\frac{1}{c}\,\delta\int L_{(MF)}\sqrt{-g}\,d\Omega$ (17)

The variation of $S_{(MF)}$ is given by:

$\displaystyle\delta S_{(MF)}$ $\displaystyle=\frac{1}{c}\,\delta\int
L_{(MF)}\sqrt{-g}\,d\Omega$ $\displaystyle=\frac{1}{2c}\,\int T_{jl}\,\delta
g^{jl}\sqrt{-g}\,d\Omega\quad$
$\displaystyle=\frac{1}{2c}\,\int\,T_{ijkl}\,g^{ik}\delta
g^{jl}\,\sqrt{-g}\,d\Omega$ (18)

where $T_{jl}$ is the familiar total energy-momentum tensor defined in
citLandau :

$\displaystyle
T_{jl}=\frac{2}{\sqrt{-g}}\,\frac{\delta\,(\,L_{(MF)}\,\sqrt{-g}\,)}{\delta
g^{jl}}$ (19)

, and $T_{ijkl}$ a generalized 4-index total energy-momentum tensor such that:

$\displaystyle T_{jl}=g^{ik}\,T_{ijkl}$ (20)

The variation of $S_{(G)}$ is obtained by using the lagrangian (15):

$\displaystyle\delta S_{G}=$ $\displaystyle-\frac{1}{2\chi c}\,\delta\int
L_{(G)}\sqrt{-g}\,d\Omega$ $\displaystyle=$ $\displaystyle-\frac{1}{2\chi
c}\,\delta\int
g^{ik}g^{jl}\,\big{[}\,aR_{ijkl}+\frac{(1-a)}{(n-2)}(g_{ijkp}R^{p}{}_{l}$
$\displaystyle+g_{ijpl}R^{p}{}_{k})-\frac{(1-a)}{(n-1)(n-2)}\,g_{ijkl}R\,\big{]}\sqrt{-g}\,d\Omega$
$\displaystyle=$ $\displaystyle-\frac{1}{2\chi
c}\,\int\bigg{[}\,g^{ik}g^{jl}\,\big{[}\,a\,\delta
R_{ijkl}+\frac{(1-a)}{(n-2)}\,\delta(g_{ijkp}R^{p}{}_{l}$
$\displaystyle+g_{ijpl}R^{p}{}_{k})-\,\frac{(1-a)}{(n-1)(n-2)}\,\delta(g_{ijkl}R)\big{]}\sqrt{-g}$
$\displaystyle+\big{[}aR_{ijkl}+\frac{(1-a)}{(n-2)}\,(g_{ijkp}R^{p}{}_{l}+g_{ijpl}R^{p}{}_{k})$
$\displaystyle-\frac{(1-a)}{(n-1)(n-2)}\,g_{ijkl}R\big{]}\,\big{[}\,2\,g^{ik}\delta
g^{jl}\sqrt{-g}$
$\displaystyle+g^{ik}g^{jl}\delta\sqrt{-g}\,\big{]}\,\bigg{]}d\Omega$ (21)

It is not so easy to directly calculate the variations of the different terms
in the last line, and so we first demonstrate some useful mathematical
relations:
$g^{jl}\delta R_{ijkl}$ using (9):

$\displaystyle g^{jl}\delta R_{ijkl}$
$\displaystyle=\delta\big{[}\,g^{jl}R_{ijkl}\,\big{]}-\delta g^{jl}R_{ijkl}$
$\displaystyle=\delta R_{ik}-\delta g^{jl}R_{ijkl}$ (22)

$g^{jl}\delta(g_{ijkp}R^{p}{}_{l}+g_{ijpl}R^{p}{}_{k})$ using (10):

$\displaystyle g^{jl}\delta(g_{ijkp}R^{p}{}_{l}+g_{ijpl}R^{p}{}_{k})$
$\displaystyle=\delta\big{[}g^{jl}(g_{ijkp}R^{p}{}_{l}+g_{ijpl}R^{p}{}_{k})\big{]}$
$\displaystyle-\delta g^{jl}(g_{ijkp}R^{p}{}_{l}+g_{ijpl}R^{p}{}_{k})$
$\displaystyle=(n-2)\delta R_{ik}+\delta(g_{ik}R)$ $\displaystyle-\delta
g^{jl}(g_{ijkp}R^{p}{}_{l}+g_{ijpl}R^{p}{}_{k})$ (23)

$g^{jl}\delta(g_{ijkl}R)$ using (11):

$\displaystyle g^{jl}\delta(g_{ijkl}R)$
$\displaystyle=\delta\big{[}\,g^{jl}(g_{ijkl}R)\,\big{]}-\delta
g^{jl}(g_{ijkl}R)$ $\displaystyle=(n-1)\delta(g_{ik}R)-\delta
g^{jl}(g_{ijkl}R)$ (24)

$\delta\sqrt{-g}=-1/2\,g_{jl}\,\delta g^{jl}\,\sqrt{-g}$ citLandau , and using
$g_{ijkl}\,g^{ik}=(n-1)g_{jl}$ we can write:

$\delta\sqrt{-g}=-\frac{1}{2(n-1)}\,g_{ijkl}\,g^{ik}\delta g^{jl}\,\sqrt{-g}$
(25)

The relationships (22), (23), (24), (25) allow one to rewrite $\delta S_{(G)}$
(21) in the form:

$\displaystyle\delta S_{(G)}=$ $\displaystyle-\frac{1}{2\chi
c}\,\int\big{[}\,aR_{ijkl}+\frac{(1-a)}{(n-2)}\,(g_{ijkp}R^{p}{}_{l}+g_{ijpl}R^{p}{}_{k})$
$\displaystyle-\frac{(n-2a)}{2(n-1)(n-2)}\,g_{ijkl}R\,\big{]}g^{ik}\delta
g^{jl}\sqrt{-g}\,d\Omega$ $\displaystyle+\int g^{ik}\delta
R_{ik}\sqrt{-g}\,d\Omega$ (26)

We note that the terms containing $\delta(g_{ik}R)$ have been simplified by
themselves. The terms containing $g^{ik}\delta R_{ik}$ result in an integral
of a covariant divergence, and hence by Stoke’s theorem are equal to a
boundary contribution at infinity which we can set to zero by making the
variation vanish at infinity citCaroll ; citBlau : $\int g^{ik}\delta
R_{ik}\,\sqrt{-g}\,d\Omega=0$.
Using the relations (18) and (26), the variation of the total action, $\delta
S$, thus becomes:

$\displaystyle\delta S$ $\displaystyle=\delta S_{(G)}+\delta S_{(MF)}$
$\displaystyle=-\frac{1}{2\chi
c}\int\big{[}aR_{ijkl}+\frac{(1-a)}{(n-2)}(g_{ijkp}R^{p}{}_{l}+g_{ijpl}R^{p}{}_{k})$
$\displaystyle-\frac{(n-2a)}{2(n-1)(n-2)}\,g_{ijkl}R-\chi
T_{ijkl}\big{]}g^{ik}\delta g^{jl}\sqrt{-g}\,d\Omega$ (27)

##  4 Generalized 4-index Einstein equation

###  4.1 Fourth-order equation

Since the expression for the variation of the total action $\delta S$ (27)
should hold for any variation $\delta g^{jl}$, we must have:

$g^{ik}\big{[}aR_{ijkl}+\frac{(1-a)}{(n-2)}(g_{ijkp}R^{p}{}_{l}+g_{ijpl}R^{p}{}_{k})\\\
-\frac{(n-2a)}{2(n-1)(n-2)}g_{ijkl}R-\chi T_{ijkl}\big{]}=0$ (28)

In general, for arbitrary $T_{ijkl}$, the expression in the parenthesis is not
equal to zero, and we can not simply exclude the contractional factor
$g^{ik}$. However, it is always mathematically possible to choose a particular
energy-momentum tensor $T_{ijkl}$ which has the same number of components and
the same symmetries as $R_{ijkl}$, and which acts as a source term, allowing
us to write the sought after generalized 4-index Einstein equation:

$aR_{ijkl}+\frac{(1-a)}{(n-2)}(g_{ijkp}R^{p}{}_{l}+g_{ijpl}R^{p}{}_{k})\\\
-\frac{(n-2a)}{2(n-1)(n-2)}\,g_{ijkl}R=\chi\,T_{ijkl}$ (29)

We can easily check that the result of the tensorial contraction of this
equation gives the famous Einstein equation of general relativity, whatever
the values of $a$ and $n$ (as surprising as it may seem):

$R_{jl}-\frac{1}{2}\,g_{jl}R=\chi\,T_{jl}$ (30)

This is of course an important result for checking our calculations, pointing
out that we did not use Einstein’s equation in any intermediate calculations
to obtain the generalized equation (29). This first result also shows us
unambiguously that the 2-index Einstein equation (30) has the same form in any
space-time of dimension $n$, which is also an important result.
To our knowledge, this new 4-index gravitational field equation, obtained by
means of a rigorous mathematical treatment using the variational principle, is
written here for the first time. Due to the presence of the Riemann tensor,
and therefore the Weyl tensor, we will show later that equation (29) contains,
as expected, more information than equation (30).

###  4.2 Fourth-order Einstein tensor

The final equation (29) can also be written in a more compact form:

$G_{ijkl}=\chi\,T_{ijkl}$ (31)

where we introduced here a new fourth-order tensor:

$\displaystyle G_{ijkl}=$
$\displaystyle\,aR_{ijkl}+\frac{(1-a)}{(n-2)}\,(g_{ijkp}R^{p}{}_{l}+g_{ijpl}R^{p}{}_{k})$
$\displaystyle-\frac{(n-2a)}{2(n-1)(n-2)}\,g_{ijkl}R$ (32)

We can verify that:

$g^{ik}G_{ijkl}=R_{jl}-\frac{1}{2}\,g_{jl}R=G_{jl}$ (33)

where we recognize here the famous 2-index Einstein tensor $G_{jl}$. The
fourth-order tensor $G_{ijkl}$ containing the Riemann tensor, the Ricci tensor
and the scalar curvature, can therefore be considered as a generalization of
$G_{jl}$.

###  4.3 Weyl tensor

The fourth-order Einstein tensor $G_{ijkl}$, depends partly on the
mathematical parameter $a$ and can therefore be split in two distinct parts of
which only one depends on $a$:

$G_{ijkl}=aC_{ijkl}\,+\,B_{ijkl}$ (34)

and we can write by identification with (32):

$\displaystyle C_{ijkl}=$
$\displaystyle\,R_{ijkl}-\frac{1}{(n-2)}(g_{ijkp}R^{p}{}_{l}+g_{ijpl}R^{p}{}_{k})$
$\displaystyle+\frac{1}{(n-1)(n-2)}g_{ijkl}R$ (35) $\displaystyle B_{ijkl}=$
$\displaystyle\,\frac{1}{(n-2)}\,(g_{ijkp}R^{p}{}_{l}+g_{ijpl}R^{p}{}_{k})$
$\displaystyle-\frac{n}{2(n-1)(n-2)}\,g_{ijkl}R$ (36)

We recognize here the Weyl tensor $C_{ijkl}$ written in $n$ dimensions
citWeinberg ; citEisenhart ; citCaroll ; citBlau ; citOyv , and a new,
hitherto unknown tensor $B_{ijkl}$. It is easy to check that the tensorial
contractions of (35) and (36) give:

$\displaystyle g^{ik}C_{ijkl}$ $\displaystyle=0$ (37) $\displaystyle
g^{ik}B_{ijkl}$ $\displaystyle=R_{jl}-\frac{1}{2}\,g_{jl}R$ (38)

This last result clearly show us that some of the information contained in the
generalized equation (31), is not recovered in the 2-index Einstein equation
(30). This is the part about the Weyl tensor.

###  4.4 Two part decomposition of total energy-momentum tensor $T_{ijkl}$

It is well known that Einstein himself and a major consensus of famous
physicists (see citLandau ; citWeinberg ; citSchro ; citDirac  for example)
have emphasized that the gravitational field must also have an energy-momentum
tensor, as do all other physical fields. In this paper, the 4-index tensor
$T_{ijkl}$, which describes any matter present plus the gravitational field
contained in space-time, can be simply and naturally divided into two parts:

$T_{ijkl}=\,T^{(M)}_{ijkl}\,+\,T^{(F)}_{ijkl}$ (39)

where $T^{(M)}_{ijkl}$ represents the generalized energy momentum tensor of
the matter content, and $T^{(F)}_{ijkl}$ the energy-momentum tensor of the
gravitational field itself citZakir . With (34) and (39) the generalized
equation (31) becomes:

$\displaystyle aC_{ijkl}\,+\,B_{ijkl}$
$\displaystyle=\chi\,(\,T^{(F)}_{ijkl}\,+\,T^{(M)}_{ijkl}\,)$ (40)

###  4.5 Energy-momentum tensor of the matter $T^{(M)}_{ijkl}$

By rearranging the Einstein equation (30), it is possible to rewrite the Ricci
tensor $R_{jl}$ and the scalar curvature $R$ in the form:

$\displaystyle R_{jl}=\chi\,(T_{jl}-\frac{1}{(n-2)}\,g_{jl}T)$ (41)
$\displaystyle R=-\frac{2}{(n-2)}\,\chi\,T$ (42)

which we now substitute into (36), obtaining:

$\displaystyle B_{ijkl}=$
$\displaystyle\,\chi\,\big{[}\,\frac{1}{(n-2)}\,(g_{ijkp}T^{p}{}_{l}+g_{ijpl}T^{p}{}_{k})$
$\displaystyle-\frac{1}{(n-1)(n-2)}\,g_{ijkl}T\,\big{]}$ (43)

Equations (38) and (43) together indicate that the tensor $B_{ijkl}$ is
directly linked to the energy-momentum tensor of the matter content present in
the standard theory of general relativity by means of the tensor $T_{jl}$. The
right-hand side of (43) represents a 4-index energy-momentum tensor for
matter, and in our theory there is only one tensor that can play this role,
namely $T^{(M)}_{ijkl}$:

$\displaystyle T^{(M)}_{ijkl}=$
$\displaystyle\,\frac{1}{(n-2)}\,(g_{ijkp}T^{p}{}_{l}+g_{ijpl}T^{p}{}_{k})$
$\displaystyle-\frac{1}{(n-1)(n-2)}\,g_{ijkl}T$ (44)

which allows us to write:

$B_{ijkl}=\chi\,T^{(M)}_{ijkl}$ (45)

The energy-momentum tensor for matter $T^{(M)}_{ijkl}$ acts as a source term
for $B_{ijkl}$, and it is not difficult to check that:

$\displaystyle g^{ik}T^{(M)}_{ijkl}=T_{jl}$ $\displaystyle
g^{jl}T^{(M)}_{ijkl}=T_{ik}$ $\displaystyle
T^{(M)}_{ijkl}=-T^{(M)}_{jikl}=-T^{(M)}_{ijlk}=T^{(M)}_{klij}$ (46)

In addition, one can easily check that the tensorial contraction of equation
(45) gives Einstein’s equation (30).

###  4.6 Energy-momentum tensor of the gravitational field $T^{(F)}_{ijkl}$

With equation (40), simplified by (45), we find a simple connection between
the energy-momentum tensor for the free gravitational field $T^{(F)}_{ijkl}$,
and the Weyl tensor $C_{ijkl}$:

$aC_{ijkl}=\chi\,T^{(F)}_{ijkl}$ (47)

$T^{(F)}_{ijkl}$ acts as a source term for the Weyl tensor, and the properties
of $C_{ijkl}$ imply that:

$\displaystyle g^{ik}T^{(F)}_{ijkl}=0$ $\displaystyle g^{jl}T^{(F)}_{ijkl}=0$
$\displaystyle T^{(F)}_{ijkl}=-T^{(F)}_{jikl}=-T^{(F)}_{ijlk}=T^{(F)}_{klij}$
(48)

The tensorial contraction of equation (47) is equal to zero. The two tensors,
$C_{ijkl}$ and $T^{(F)}_{ijkl}$, are logically linked because they only
concern the free gravitational field in vacuum, and it is precisely this part
which is not present in the standard theory of general relativity. The
information contained in equation (47) is not contained in the 2-index
Einstein equation, and for this reason a fourth-order theory is fully
justified.
For a vanishing Ricci tensor, $R_{jl}=0$, the Riemann curvature tensor is
equal to the Weyl tensor, $R_{ijkl}=C_{ijkl}$, and it is well known that the
latter describes the free gravitational field in vacuum and provides curvature
to space-time citBlau ; citOyv . A flat space-time implies no matter with
$T^{(M)}_{ijkl}=0$, no field with $T^{(F)}_{ijkl}=0$, and likewise
$R_{ijkl}=0$, as can be seen from the generalized Einstein equation (29).
The total energy-momentum tensor, $T_{ijkl}=T^{(M)}_{ijkl}+T^{(F)}_{ijkl}$,
gives the most complete description of the medium with respect to its effect
on the geometry. It describes both the state of matter and the state of the
gravitational field in vacuum, but it appears to be very non-trivial to find
the form of the tensor $T^{(F)}_{ijkl}$. The determination of this tensor,
however, and therefore the form of equation (47), would allow us to find
important solutions concerning the contribution of the gravitational field
itself. The tensor $T^{(F)}_{ijkl}$ naturally takes its place in the new
generalized 4-index equation, and we’ll show in the next chapter that
$T_{ijkl}$ obeys total energy-momentum conservation.

###  4.7 Energy-momentum conservation

In the preceding chapters the parameter, $a$, has been treated as a
mathematical parameter with no particular meaning. We will now show that $a$
takes on a particular physical value in the context of covariant conservation
of the total 4-index energy-momentum tensor:

$\nabla_{i}T^{i}{}_{jkl}=\nabla_{i}(T^{(F)}{}^{i}{}_{jkl}+T^{(M)}{}^{i}{}_{jkl})=0$
(49)

where $\nabla_{i}$ is the covariant derivative operator.
With the definition of the generalized Enstein tensor (32), and using the
usual metric compatibility $\nabla_{i}\,g_{jl}=0$, we can write:

$\displaystyle\nabla_{i}G^{i}{}_{jkl}=$
$\displaystyle\nabla_{i}\big{[}aR^{i}{}_{jkl}+\frac{(1-a)}{(n-2)}(\delta^{i}_{k}R_{jl}-g_{jk}R^{i}{}_{l}+g_{jl}R^{i}{}_{k}$
$\displaystyle-\delta^{i}_{l}R_{jk})-\frac{(n-2a)}{2(n-1)(n-2)}(\delta^{i}_{k}g_{jl}-\delta^{i}_{l}g_{jk})R\big{]}$
$\displaystyle=$
$\displaystyle\,a\nabla_{i}R^{i}{}_{jkl}+\frac{(1-a)}{(n-2)}\,(\nabla_{k}R_{jl}-g_{jk}\nabla_{i}R^{i}{}_{l}$
$\displaystyle+g_{jl}\nabla_{i}R^{i}{}_{k}-\nabla_{l}R_{jk})$
$\displaystyle-\,\frac{(n-2a)}{2(n-1)(n-2)}\,(g_{jl}\nabla_{k}R-g_{jk}\nabla_{l}R)$

We make use of the contracted Bianchi identities, citWeinberg ; citCaroll :
$\nabla_{i}R^{i}{}_{jkl}=\nabla_{k}R_{jl}-\nabla_{l}R_{jk}$ and
$\nabla_{i}R^{i}{}_{k}=1/2\,\nabla_{k}R$ to deduce:

$\displaystyle\nabla_{i}G^{i}{}_{jkl}=$
$\displaystyle\,-\frac{1+a(n-3)}{(n-2)}\big{[}\nabla_{l}R_{jk}-\nabla_{k}R_{jl}$
$\displaystyle+\frac{1}{2(n-1)}(g_{jl}\nabla_{k}R-g_{jk}\nabla_{l}R)\big{]}$
$\displaystyle=$ $\displaystyle\,-\frac{1+a(n-3)}{(n-2)}\,C_{jkl}$ (50)

where we identify here the Cotton tensor $C_{jkl}$ citCotton ; citCotton2 .
Imposing energy conservation thus gives us the physical value of $a$:

$\nabla_{i}G^{i}{}_{jkl}=0\,\,\Rightarrow\,\,\nabla_{i}T^{i}{}_{jkl}=0\,\,\Rightarrow\,\,a=-\frac{1}{(n-3)}$
(51)

##  5 Generalized equation with $a=-1/(n-3)$

###  5.1 Double-dual Riemann tensor ${}^{*}R_{ijkl}^{*}$

In the case where energy is conserved, taking $a=-1/(n-3)$, the generalized
4-index Einstein equation of general relativity (29) becomes:

$\frac{1}{(n-3)}\,\big{[}-R_{ijkl}+g_{ijkp}R^{p}{}_{l}+g_{ijpl}R^{p}{}_{k}-\frac{1}{2}g_{ijkl}R\big{]}=\chi
T_{ijkl}$ (52)

where we recognize here the double (Hodge) dual Riemann tensor citMisner ;
citGliner :

${}^{*}R_{ijkl}^{*}=$ $\displaystyle\frac{1}{4}\,e_{ijpq}\,R^{pqrs}e_{klrs}$
$\displaystyle=$ $\displaystyle-
R_{ijkl}+g_{ijkp}R^{p}{}_{l}+g_{ijpl}R^{p}{}_{k}-\frac{1}{2}\,g_{ijkl}R$ (53)

with $e_{ijkl}$ the Levi-Civita tensor.
The general equation (52) can thus be written simply as:

${}^{*}R_{ijkl}^{*}=\chi\,(n-3)\,T_{ijkl}$ (54)

It is not difficult to check that the tensorial contraction of this equation
again yields the Einstein equation (30), with
$g^{ik}{}^{*}R_{ijkl}^{*}=(n-3)(R_{jl}-\frac{1}{2}\,g_{jl}R)$, and that (54),
as expected, also satisfies total energy-momentum conservation:

$\displaystyle\nabla_{i}{}^{*}R^{*}{}^{i}{}_{jkl}=0$ (55)
$\displaystyle\nabla_{i}T^{i}{}_{jkl}=\nabla_{i}(T^{(F)}{}^{i}{}_{jkl}+T^{(M)}{}^{i}{}_{jkl})=0$
(56)

According to (45) and (47), it is also possible to write equation (54) into
two parts:

$\displaystyle C_{ijkl}$ $\displaystyle=-\chi(n-3)\,T^{(F)}_{ijkl}$ (57)
$\displaystyle B_{ijkl}$ $\displaystyle=\chi\,T^{(M)}_{ijkl}\quad$ (58)

As we have already seen, these two equations are not coupled; they are
independent, and their solutions found individually must be added to satisfy
total energy momentum conservation. As an analogy in classical mechanics, for
example, it is akin to the kinetic energy and the potential energy that derive
from different relations, but which must finally be added to have conservation
of the total mechanical energy.

###  5.2 Particular solution of the equation $B_{ijkl}=\chi\,T^{(M)}_{ijkl}$

In the very important case of a centrally symmetric field in vacuum, that is,
outside of the masses producing the field, $T^{(M)}_{ijkl}=0$, it is possible
to solve the equation (58), for $n=4$:

$B_{ijkl}{}_{(n=4)}=\frac{1}{2}\,(g_{ijkp}R^{p}{}_{l}+g_{ijpl}R^{p}{}_{k})-\,\frac{1}{3}\,g_{ijkl}R=0$
(59)

If we use spherical space coordinates $(r,\theta,\phi)$, then the general
expression for $ds^{2}$ is:

$ds^{2}=A(r)c^{2}dt^{2}-B(r)dr^{2}-r^{2}d\theta^{2}-r^{2}sin^{2}\theta
d\phi^{2}$ (60)

To get the differential equations of gravitation with the functions $A(r)$ and
$B(r)$, we must first calculate the components of the tensor
$B_{ijkl}{}_{(n=4)}$. Using a familiar method, it is not difficult (but long)
to obtain the following independent relations:

$\displaystyle B_{0101}=$
$\displaystyle\,-\frac{1}{6}\,\big{[}\,A^{{}^{\prime\prime}}-\frac{A^{{}^{\prime}2}}{2A}-\frac{A^{{}^{\prime}}B^{{}^{\prime}}}{2B}+\frac{AB^{{}^{\prime}}}{rB}$
$\displaystyle+\frac{4AB}{r^{2}}-\frac{A^{{}^{\prime}}}{r}-\frac{4A}{r^{2}}\,\big{]}$
(61) $\displaystyle B_{0202}=$
$\displaystyle\,\frac{r^{2}}{12B}\,\big{[}\,A^{{}^{\prime\prime}}-\frac{A^{{}^{\prime}2}}{2A}-\frac{A^{{}^{\prime}}B^{{}^{\prime}}}{2B}-\frac{5AB^{{}^{\prime}}}{rB}$
$\displaystyle-\frac{2AB}{r^{2}}-\frac{A^{{}^{\prime}}}{r}+\frac{2A}{r^{2}}\,\big{]}$
(62) $\displaystyle B_{1212}=$
$\displaystyle\,-\frac{r^{2}}{12A}\,\big{[}\,A^{{}^{\prime\prime}}-\frac{A^{{}^{\prime}2}}{2A}-\frac{A^{{}^{\prime}}B^{{}^{\prime}}}{2B}+\frac{AB^{{}^{\prime}}}{rB}$
$\displaystyle-\frac{2AB}{r^{2}}+\frac{5A^{{}^{\prime}}}{r}+\frac{2A}{r^{2}}\,\big{]}$
(63) $\displaystyle B_{2323}=$
$\displaystyle\,-\frac{r^{4}sin^{2}\theta}{3AB}\,\big{[}\,A^{{}^{\prime\prime}}-\frac{A^{{}^{\prime}2}}{2A}-\frac{A^{{}^{\prime}}B^{{}^{\prime}}}{2B}-\frac{AB^{{}^{\prime}}}{2rB}$
$\displaystyle+\frac{AB}{r^{2}}+\frac{A^{{}^{\prime}}}{2r}-\frac{A}{r^{2}}\,\big{]}$
(64)

By combining these differents equations, we finally obtain a well known
solution, the standard Schwarzschild metric for the space-time citLandau ;
citMisner :

$\displaystyle A(r)=$ $\displaystyle\,\frac{1}{B(r)}=1+\frac{r_{g}}{r}$ (65)
$\displaystyle ds^{2}=$
$\displaystyle\,(1+\frac{r_{g}}{r})\,c^{2}dt^{2}-(1+\frac{r_{g}}{r})^{-1}dr^{2}$
$\displaystyle-r^{2}d\theta^{2}-r^{2}sin^{2}\theta d\phi^{2}$ (66)

This is not surprising because equation (59) contains only the Ricci tensor
and the scalar curvature, and therefore we expect to find the same solutions
as those obtained with the Einstein equation (a good sign). Equation (58),
however, goes much further because it may allow us to find the exact solutions
for any dimension of space-time $n$.
Because the generalized equation (54) is contracted to obtain the original
Einstein equation, only standard solutions are present in the familiar 2-index
general theory of relativity, but what is lacking are the solutions of
equation (57) giving the contribution of the gravitational field itself in
vaccum. The next step in our future research will be to determine precisely
the mathematical composition of the energy-momentum tensor $T^{(F)}_{ijkl}$ in
order to obtain solutions to equation (57), but this appears to be a very
difficult task.

###  5.3 Cosmological constant

It is interesting to verify that the equation (54) can be deduced from a
principle of least action using the Lagrangian (15) written with $a=-1/(n-3)$:

$\displaystyle L_{(G)}=$
$\displaystyle\,\frac{1}{(n-3)}\,g^{ik}g^{jl}\,\big{[}-R_{ijkl}+g_{ijkp}R^{p}{}_{l}+g_{ijpl}R^{p}{}_{k}$
$\displaystyle-\frac{1}{(n-1)}\,g_{ijkl}R\,\big{]}$ (67)

We will work again within the framework of general relativity, where the
cosmological constant $\Lambda$ is introduced into the Lagrangian by an
additional term citCaroll : $-2\Lambda$. In the same way as in (14), this term
can be rewritten as:
$-2\Lambda=-2/{n(n-1)}\,g^{ik}g^{jl}\,(g_{ijkl}\,\Lambda)$. The principle of
least action applied to this term modifies equation (54), giving us:

${}^{*}R_{ijkl}^{*}+\frac{(n-3)}{(n-1)}\,g_{ijkl}\,\Lambda=\chi\,(n-3)\,T_{ijkl}$
(68)

Contracting this latter equation indeed restores Einstein’s standard equations
with the cosmological constant citCaroll ; citBlau :

$R_{jl}-\frac{1}{2}\,g_{jl}R+g_{jl}\Lambda=\chi\,T_{jl}$ (69)

##  6 Conclusion

In this paper we have rigorously demonstrated by using the principle of least
action, that there exists a much more general equation than that of Einstein.
This new 4-index equation explicitly and linearly includes the Riemann tensor
and hence the Weyl tensor as well as the energy of the gravitational field
$T^{(F)}_{ijkl}$. The contraction of this equation restores the usual 2-index
Einstein equation. The paper highlights two main results:
\- the first result is given by the equation, $B_{ijkl}=\chi\,T^{(M)}_{ijkl}$,
which could allow us to find the exact Schwarzschild solutions for any $n$
dimension space.
\- the second important result is given by the equation,
$C_{ijkl}=-\chi(n-3)\,T^{(F)}_{ijkl}$, containing the Weyl tensor and the
energy-momentum tensor of the gravitational field in vaccum, which could be
connected, as we are hoping, to the mysterious dark energy.
The next step in our research will be to determine precisely the composition
of the energy-momentum tensor $T^{(F)}_{ijkl}$ in order to obtain more general
solutions including the contribution of the gravitational field energy itself.

## References

  * (1) Landau, L. and Lifchitz, E. (1975) The classical Theory of fields, Pergamon Oxford Press.
  * (2) Misner, C.W. and Thorne, K.S. and Wheeler, J.A. (1973) Gravitation, Freeman W.H.
  * (3) Weinberg, S. (1972) Gravitation and Cosmology, John Wiley Press.
  * (4) Eisenhart, L.P. (1926) Riemannian Geometry, Princeton University Press.
  * (5) Carroll, S. (2004) Spacetime and geometry: an introduction to general relativity, San Francisco Addison-Wesley.
  * (6) Blau, M. (2014) Lecture Notes on General Relativity, Albert Einstein Center for Fundamental Physics.
  * (7) Øyvind, G. and Sigbjørn, H. (2007) Einstein’s General Theory of Relativity, Springer.
  * (8) Schrödinger, E. (1950) Space-time structure, Cambridge University Press.
  * (9) Dirac, P.A.M. (1975) General Theory of Relativity, Princeton University Press.
  * (10) Zakir, Z. (2010) Four-Index Energy-Momentum Tensors for Gravitation and Matter, TPAC Theor. Phys. Astrophys. Cosmol. 5, 1.
  * (11) Cotton, E. (1899) Sur les variétés à trois dimensions, Ann. Fac. Sc. Toulouse (II), 1
  * (12) Garcia, A. and Hehl, F.W. and Heinicke, C. and Macias, A. (2004) The Cotton tensor in Riemannian spacetimes, Class. Quant. Grav. 21, 1099
  * (13) Gliner, E.B. (1969) Field equations of the general theory of relativity, Soviet Physics JETP 28, 1238