11institutetext: Cluster of Excellence – ML for Science, University of
Tübingen, Germany
11email<EMAIL_ADDRESS><EMAIL_ADDRESS>22institutetext: Max Planck Institute for Intelligent Systems, Tübingen,
Germany 22email<EMAIL_ADDRESS>33institutetext:
Department of Diabetes, Endocrinology, Nutritional Medicine and Metabolism
UDEM, Inselspital, Bern University Hospital, University of Bern, Switzerland
44institutetext: Hertie Institute for AI in Brain Health, University of
Tübingen, Germany 44email<EMAIL_ADDRESS>55institutetext: Faculty of
Health Sciences and Medicine, University of Lucerne, Switzerland 55email:
<EMAIL_ADDRESS>
# Conformal Performance Range Prediction for Segmentation Output Quality
Control

Anna M. Wundram 11 Paul Fischer 1155 Michael Mühlebach 22 Lisa M. Koch 3344
Christian F. Baumgartner 1155

###### Abstract

Recent works have introduced methods to estimate segmentation performance
without ground truth, relying solely on neural network softmax outputs. These
techniques hold potential for intuitive output quality control. However, such
performance estimates rely on calibrated softmax outputs, which is often not
the case in modern neural networks. Moreover, the estimates do not take into
account inherent uncertainty in segmentation tasks. These limitations may
render precise performance predictions unattainable, restricting the practical
applicability of performance estimation methods. To address these challenges,
we develop a novel approach for predicting performance ranges with statistical
guarantees of containing the ground truth with a user specified probability.
Our method leverages sampling-based segmentation uncertainty estimation to
derive heuristic performance ranges, and applies split conformal prediction to
transform these estimates into rigorous prediction ranges that meet the
desired guarantees. We demonstrate our approach on the FIVES retinal vessel
segmentation dataset and compare five commonly used sampling-based uncertainty
estimation techniques. Our results show that it is possible to achieve the
desired coverage with small prediction ranges, highlighting the potential of
performance range prediction as a valuable tool for output quality
control111Code available at
https://github.com/annawundram/PerformanceRangePrediction.

##  1 Introduction

Image segmentation is a crucial step for various medical tasks such as disease
detection, treatment planning or anatomical studies [4]. In ophthalmology,
segmenting the retinal vessels in fundus photography images provides insights
into clinical conditions such as Glaucoma or Diabetic Retinopathy [15].
However, manual segmentation of the retinal vasculature is prohibitively time-
consuming, taking up to five hours per image [15]. In recent years, machine
learning models have shown excellent performance on many segmentation problems
[14, 7]. Despite this, machine learning methods can fail unexpectedly, and
even the best algorithms have limited performance on images that are
inherently challenging to segment. As manual verification of all algorithmic
outputs is in itself time-consuming and sometimes infeasible, developing
strategies for automatically ensuring the quality of segmentation outputs is
becoming increasingly important.

Ensuring segmentation quality can be approached through input or output
quality control. Input quality control aims to automatically identify images
that are likely to be poorly segmented by the model. Commonly used strategies
include automatic prediction of image quality [32] or out-of-distribution
(OOD) detection [23, 8]. However, input quality control methods may misjudge
the algorithm’s actual performance on a specific image. For instance, as
demonstrated in this work, an image may be within the support of the training
distribution yet difficult for the model to segment, or it may be of poor
quality but still easy to segment (see Fig. 1). Such cases might either go
undetected by input quality control, or could be flagged unnecessarily.

Figure 1: Overview. (a) Given a fundus image, we predict a vessel
segmentation, the expected Dice-Sørensen Coefficient (DSC), as well as upper
and lower bounds for the expected DSC. (b) Conformal prediction allows us to
set the performance range such that at most $\alpha=10\%$ percent of the test
cases have a DSC outside the predicted interval. We show a confident case with
low DSC prediction uncertainty (green), as well as a case with high DSC
prediction uncertainty due to poor image quality (red).

Output quality control methods instead aim to directly verify that the output
of a model is of sufficient quality. Most commonly, output uncertainty is used
as a proxy for quality (e.g. [22, 24, 25]). However, these approaches require
choosing a heuristic threshold for acceptable quality. To address this, a
number of methods directly estimate the expected performance on previously
unseen data points. This allows setting more intuitive performance thresholds
such as “a Dice-Sørensen Coefficient (DSC) of at least 0.8”. In one of the
first works on medical segmentation performance prediction, [18] trained a
regressor to directly predict segmentation accuracy. Later, Valindra et al.
[29] introduced Reverse Classification Accuracy, which trains a reverse
segmentation model on the test image and its segmentation output, then applies
it to reference images to estimate the DSC. However, this approach requires
training a second model.

Recent studies have demonstrated that segmentation performance measures can be
predicted from softmax outputs alone [31, 9, 28, 13, 26, 12, 19, 21]. Assuming
perfect calibration, the softmax outputs indeed describe the probability of a
pixel having a certain label. As detailed in Sec. 2.1, this interpretation
allows for the computation of various performance measures, including the DSC
[21, 19]. However, the quality of this performance estimator heavily relies on
proper calibration [19]. Unfortunately, modern neural networks frequently
exhibit poor calibration [11]. Although post-hoc methods like temperature
scaling can enhance calibration [19, 11], these solutions are often
insufficient for achieving the desired level of reliability. Moreover,
inherent segmentation uncertainty arising from low image quality or other
factors, may further limit precise performance estimation. As a result, the
performance estimates obtained through these methods lack guarantees, raising
concerns about their usefulness for quality control.

In this work, we propose to predict performance ranges instead of point
estimates. Our approach offers statistical guarantees that the true
performance of any test image falls within the predicted range with a certain
probability. We achieve this by estimating a heuristic performance range using
sampling-based uncertainty quantification methods. We then apply split
conformal prediction [30, 1] to conformalize prediction ranges extracted from
those uncertainty estimates. We demonstrate the effectiveness of our approach
on the challenging problem of retinal vessel segmentation in fundus images.

##  2 Methods

Given an input image $x$ our goal is to predict a segmentation $\hat{s}$ along
with a performance estimate $\hat{y}$ predicting the model’s segmentation
performance (e.g. the DSC) for that image. We additionally predict a
performance range $[\hat{y}_{l},\hat{y}_{u}]$ with a statistical guarantee
that the true DSC, i.e. $y=\textrm{DSC}(\hat{s},s)$, is contained in this
interval with a user specified probability of $1-\alpha$. Note that while we
focus on the DSC, alternative measures can be easily investigated in our
framework.

In the following, we first review how a DSC performance estimate can be
derived from the softmax outputs of a segmentation model (Sec. 2.1). Next, we
introduce our method for obtaining heuristic performance ranges using
sampling-based segmentation uncertainty estimation approaches (Sec. 2.2).
Lastly, we describe our strategy for converting heuristic performance ranges,
into performance ranges with statistical guarantees using split conformal
prediction (Sec. 2.3).

###  2.1 Background: Estimating the DSC from softmax outputs

Given a calibrated segmentation model, the softmax output $p_{i}$ for each
pixel $i$ can be interpreted as the probability of that pixel being of the
predicted class. In the binary case, summing the positively predicted (i.e.
$p_{i}>0.5$) foreground probabilities for all pixels yields the expected
number of true positives (TP) in the image. Following a similar reasoning the
expected number of false positives (FP) and false negatives (FN) can be
calculated [21]:

$\textrm{TP}=\sum^{n}_{i=1}\mathbf{1}_{[p_{i}>0.5]}p_{i};~{}\textrm{FP}=\sum^{n}_{i=1}\mathbf{1}_{[p_{i}>0.5]}-\textrm{TP};~{}\textrm{FN}=\sum^{n}_{i=1}\mathbf{1}_{[p_{i}<0.5]}p_{i}\
\text{.}$ (1)

Here, $\mathbf{1}_{[\cdot]}$ denotes the indicator function and $0.5$ is a
heuristically chosen decision threshold. The DSC is defined in terms of those
quantities as follows

$\textrm{DSC}=\dfrac{2\textrm{TP}}{2\textrm{TP}+\textrm{FP}+\textrm{FN}}\
\text{.}$ (2)

Thus, for a given test image $x$, a DSC performance estimate $\hat{y}$ can be
calculated using the estimators for TP, FP, and FN (Eq. 1), and plugging them
into Eq. 2.

In practice neural networks are never perfectly calibrated, resulting in over-
or underestimation of the true DSC. Instead of relying on the possibly faulty
performance prediction $\hat{y}$, in the following, we show how to obtain
bounds that contain the true performance with high probability.

###  2.2 Heuristic performance bounds from segmentation uncertainty

To obtain heuristic lower and upper performance bounds $\bar{y}_{l}$ and
$\bar{y}_{u}$, we rely on probabilistic segmentation techniques that are
capable of producing samples from the distribution $p(s|x)$. Given an input
image $x$ these techniques allow us to sample $N$ plausible segmentation
samples $\hat{s}_{n}$. A performance estimate $\hat{y}_{n}$ can be obtained
from each segmentation sample $\hat{s}_{n}$ as described in Sec. 2.1.

The samples $\hat{y}_{n}$ characterize the distribution $p(\hat{y}|x)=\int
p(s|x)p(\hat{y}|s)ds$. From these samples, we can calculate an estimator for
the standard deviation $\sigma$. We can then define our heuristic upper and
lower bounds for an input image $x$ as:

$\bar{y}_{l}(x)=\hat{y}(x)-\sigma(\hat{y}(x));~{}~{}~{}~{}\bar{y}_{u}(x)=\hat{y}(x)+\sigma(\hat{y}(x))~{}\
\text{.}$ (3)

We compare five commonly used probabilistic segmentation techniques to obtain
segmentation samples for performance range prediction:

  * •

The probabilistic U-Net [17] is a combination of the conditional VAE [27]
approach with a U-Net architecture. This formulation allows to sample an
infinite number of segmentation samples consistent with the input image $x$.

  * •

PHiSeg [6] extends the probabilistic U-Net by a hierarchical latent space and
was shown to provide closer approximations of $p(\\*s|\\*x)$.

  * •

Test-time augmentation (TTA) [3] augments the test image $N$ times to obtain
$N$ segmentation samples $s_{n}$. The deviations between the samples $s_{n}$
have been shown to be indicative of segmentation uncertainty. Following [3],
we used the following eight types of augmentations: brightness, hue,
saturation, contrast, vertical and horizontal flip, Gaussian blur.

  * •

Ensembles [20] consist of $N$ independently trained segmentation models
initialized with different random seeds. In order to improve calibration,
temperature scaling [11] on the calibration set was applied to each individual
network. Note that temperature scaling is not directly applicable to any of
the other explored methods.

  * •

Monte Carlo (MC) Dropout [10] produces probabilistic segmentation samples by
repeatedly predicting segmentations for the same image with dropout enabled.
We use a dropout rate of 0.2 on the activation maps for training and testing.
Dropout is applied to all layers except the final four segmentation layers.

We use the enhanced U-Net architecture introduced in [17] as a base
architecture for all methods except PHiSeg. While PHiSeg also uses the same
U-Net encoder, it employs a unique decoder. We use $N=100$ for Prob. U-Net,
PHiSeg and MC Dropout, $N=20$ for TTA, and $N=10$ for Ensembles. For all
approaches, a final segmentation $\hat{s}$ is obtained by averaging the
samples. The probabilistic U-Net, PHiSeg, and TTA estimate aleatoric
uncertainty, while Ensembles and MC Dropout estimate epistemic uncertainty
(see [16] for definitions of aleatoric and epistemic).

###  2.3 From heuristic to principled bounds using conformal prediction

We employ split conformal prediction [30, 1] to convert the heuristic bounds
$\bar{y}_{l},\bar{y}_{u}$ into principled bounds $\hat{y}_{l},\hat{y}_{u}$.
Specifically, we desire that the performance range $[\hat{y}_{l},\hat{y}_{u}]$
includes the true DSC $y$ with a user set probability of at least $1-\alpha$:

$\mathbb{P}\left(y\in[\hat{y}_{l}(x),\hat{y}_{u}(x)]\right)\geq 1-\alpha\
\text{.}$ (4)

We set $\alpha=0.1$ in all our experiments.

For our performance range to statistically fulfill the above requirement, we
adjust the heuristic bounds with a corrective factor $\hat{q}$:

$\hat{y}_{l}(x)=\hat{y}(x)-\hat{q}\sigma(\hat{y}(x));~{}~{}~{}~{}\hat{y}_{u}(x)=\hat{y}(x)+\hat{q}\sigma(\hat{y}(x))\
\text{.}$ (5)

This corrective factor $\hat{q}$ is determined using the split conformal
procedure [30, 1]. We first define a score function as

$\mathcal{S}(x,y)=\dfrac{|y-\hat{y}(x)|}{\sigma(\hat{y}(x))}~{}\text{.}$ (6)

The adjusting factor $\hat{q}$ is then calculated as the
$\lceil(1-\alpha)(M+1)\rceil/M$ quantile on the calibration set scores, where
$M$ is the number of calibration samples. This results in the following
guarantee for the test set, assuming the calibration set is representative of
the test distribution:

$\mathbb{P}[\mathcal{S}(x,y)\leq\hat{q}]\geq
1-\alpha\Rightarrow\mathbb{P}[|y-\hat{y}(x)|\leq
2\sigma(\hat{y}(x))\hat{q}]\geq 1-\alpha~{}\text{,}$ (7)

thereby fulfilling our requirement in Eq. 4. We finally clamp the prediction
range to be within $[0,1]$ as DSC values outside this range are not possible.

###  2.4 Data and training

We evaluated our method on the FIVES [15] fundus dataset for retinal vessel
segmentations. The dataset comprises 800 images and manual segmentations with
an official split into 600 training and 200 test images. We further split each
fold using a ratio of 80/20 to obtain train/validation as well as
test/calibration sets. Following [19], we preprocessed the data by applying
contrast limited adaptive histogram equalization (CLAHE) and resized the
images to $320\times 320$ pixels. We used the provided image quality labels
for model inspection and visualization. We considered an image to be low
quality if at least one out of three quality issues (illumination and color
distortion, blur and low contrast) were reported.

All segmentation models were trained with maximum number of epochs of 1000 on
a NVIDIA GeForce RTX 2080 Ti with a batch size of four for all probabilistic
models and a batch size of 16 for all U-Nets. Model selection was performed on
the validation set using the DSC as metric.

##  3 Experiments and results

###  3.1 Evaluation of segmentation performance and DSC prediction

Figure 2: Quantitative analysis. (a) performance prediction absolute error,
(b) marginal and conditional coverage for very small (0, 0.1], small (0.1, 2],
large (0.2, 5], very large (0.5, 1] interval sizes, and (c) interval sizes for
all investigated methods

We first verified that all compared models performed adequately at the
underlying segmentation task. We observed high mean test DSC for Prob. U-Net
($0.918$), MC Dropout ($0.918$) and Ensemble ($0.913$), with PHiSeg ($0.888$)
performing slightly worse, though still acceptably, and in line with previous
results on the FIVES dataset [19]. TTA ($0.811$) performed substantially worse
than the rest of the evaluated techniques.

Next, we turned to the analysis of the performance prediction. PHiSeg and TTA
were the most accurate at predicting the DSC, achieving mean absolute errors
(MAE) of 0.027 and 0.025, respectively (see Fig. 2). Ensembles performed worst
with a MAE of 0.072. This can be confirmed qualitatively by comparing the
predictions to the ground truth line in Fig. 3.

###  3.2 Evaluation of coverage and interval sizes

As we argue in this paper, a point estimate of the predicted performance is
insufficient because poor calibration of the segmentation models lead to
inaccurate scores, and because some cases carry inherent uncertainty in their
performance prediction (e.g. due to low image quality).

In our central evaluations, we therefore investigated the quality of the
performance ranges obtained using our proposed method. We adopted the approach
of [2] and used _coverage_ as our main evaluation criterion. In our case,
coverage measures the proportion of images for which the actual DSC falls
within the predicted performance range. PHiSeg, Ensembles, and MC Dropout
reach the specified marginal coverage, meaning that $\geq 90\%$ of all test
ground truth DSC values lie in the predicted interval (Fig. 2b). We note that
Prob. U-Net and TTA did not quite achieve marginal coverage. We hypothesize
that this is due to a combination of poor uncertainty estimation and a slight
violation of the exchangeability between calibration and test set assumption.
The coverage of the studied segmentation models can also be analyzed
qualitatively in Fig. 3 by verifying that the majority of ground truth DSC
scores (black line) lie withing the gray prediction range.

While split conformal prediction only guarantees marginal coverage, it is
crucial for practical applications that coverage holds across different
interval sizes. Therefore, in Fig. 2b, we also evaluated conditional coverage
for four different interval sizes. All methods achieved the desired coverage
for small (0.1, 0.2], large (0.2-0.5] and very large (0.5-1] intervals. PHiSeg
achieved the best coverage for very small (0-0.1] interval sizes, but fell
slightly short of the desired 90%. Note that the bar for very small interval
sizes is missing for Ensembles because no intervals of this size were
predicted by the method.

Assuming coverage is fulfilled, it is desirable to have interval sizes that
are as small as possible. Overly large interval sizes resulting from poor DSC
estimation or poor uncertainty estimation, may detract from the usefulness of
the method in practice. It is therefore desirable to have interval sizes that
are as small as possible. PHiSeg produced the tightest intervals (Fig. 2c),
which can also be confirmed visually in Fig. 3 by inspecting the size of the
gray prediction ranges.

Since the uncertainty in our task stems largely from irreducible ambiguities
in the vessel segmentation, aleatoric uncertainty quantification methods
should perform the best. Indeed, the top-performing approach, PHiSeg, falls
into this cateogry. However, the overall picture is less clear, as the
epistemic Ensemble and MC Dropout approaches perform similarly to the
aleatoric Prob. U-Net. We concur with Kahl et al. [16] that the distinction
between aleatoric and epistemic uncertainty quantification methods is not
always clear cut.

Figure 3: Visualisation of performance ranges. Performance predictions
$\hat{y}$ (green/red), ground truth DSC scores $y$ (black), and performance
ranges $[\hat{y}_{l},\hat{y}_{l}]$ (gray) for all images in the test set. The
images are sorted by ground-truth performance.

###  3.3 Performance prediction analysis of low- vs. high-quality images

To better understand the influence of image quality on the performance
prediction, we colored all points in Fig. 3 by high-quality (green) and poor-
quality (red) using the quality labels provided by the FIVES dataset. Firstly,
as expected, we observed that most images with poor segmentation performance
were of low quality. Secondly, overall performance prediction was worse for
low-quality images compared to high-quality images. Thirdly, we observed that
the size of the prediction performance range correlated with the ground truth
DSC, indicating that harder-to-segment images also had higher uncertainty in
their performance predictions. The low-quality, low DSC images on the left
side of the plots in Fig. 3 were typically characterized by large performance
ranges. For the best-performing method, PHiSeg, although the performance
estimation was poor in these cases, the intervals consistently contained the
true DSC. This illustrates that for these highly uncertain cases a single
performance prediction is insufficient. It underscores that the statistically
valid prediction performance ranges proposed here offer a promising approach
for output quality control.

We note that all low-quality images used in this evaluation are in-
distribution as the training set also contained similar low-quality images.
This highlights the fact that OOD approaches would likely not be able to
identify these cases where performance is low. An alternative strategy would
be to train an image quality classifier to detect images with low-quality
before feeding them to the model. However, there are also examples in Fig. 3
of low-quality images that achieve high DSC. An image quality classifier would
falsely flag these images potentially resulting in unnecessary re-scans. Our
proposed output quality control approach using performance ranges effectively
addresses both these issues.

##  4 Discussion and conclusion

In this work, we demonstrated that performance prediction point estimates may
be insufficient for robust quality control due to suboptimal calibration of
neural networks, and high performance uncertainty in low-quality images. To
address this problem, we developed a method that can compute performance
_ranges_ with statistical guarantees for coverage, and compared five different
sampling-based uncertainty quantification methods to estimate those range.

The aleatoric PHiSeg method produced the best performance predictions, as well
as the performance ranges with the best coverage and tightest interval sizes.
We conclude that this method is highly suitable for use in conformal
performance prediction.

A limitation of our work is that it is only applicable under the assumption of
exchangeability of the test and calibration sets. The method is thus not
directly applicable to the OOD setting. In future work, we will pursue an
extension of our approach that takes advantage of novel research directions in
conformal predictions under domain shifts [5].

####  4.0.1 Acknowledgements

This work was supported by the German Science Foundation (BE5601/8-1) and the
Excellence Cluster 2064 “Machine Learning — New Perspectives for Science”,
project number 390727645) and the Hertie Foundation. The authors thank the
International Max Planck Research School for Intelligent Systems (IMPRS-IS)
for supporting Paul Fischer.

####  4.0.2

The authors have no competing interests to declare that are relevant to the
content of this article.

## References

  * [1] Angelopoulos, A.N., Bates, S.: A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511 (2021)
  * [2] Angelopoulos, A.N., Kohli, A.P., Bates, S., Jordan, M., Malik, J., Alshaabi, T., Upadhyayula, S., Romano, Y.: Image-to-image regression with distribution-free uncertainty quantification and applications in imaging. In: International Conference on Machine Learning. pp. 717–730. PMLR (2022)
  * [3] Ayhan, M.S., et al.: Test-time data augmentation for deep learning-based colon polyp classification. IEEE Journal of Biomedical and Health Informatics 23(3), 1185–1192 (2018)
  * [4] Azad, R., Aghdam, E.K., Rauland, A., Jia, Y., Avval, A.H., Bozorgpour, A., Karimijafarbigloo, S., Cohen, J.P., Adeli, E., Merhof, D.: Medical image segmentation review: The success of u-net. arXiv preprint arXiv:2211.14830 (2022)
  * [5] Barber, R.F., Candes, E.J., Ramdas, A., Tibshirani, R.J.: Conformal prediction beyond exchangeability. The Annals of Statistics 51(2), 816–845 (2023)
  * [6] Baumgartner, C.F., Tezcan, K.C., Chaitanya, K., Hötker, A.M., Muehlematter, U.J., Schawkat, K., Becker, A.S., Donati, O., Konukoglu, E.: Phiseg: Capturing uncertainty in medical image segmentation. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2019. pp. 119–127. Springer International Publishing, Cham (2019)
  * [7] Bernard, O., Lalande, A., Zotti, C., Cervenansky, F., Yang, X., Heng, P.A., Cetin, I., Lekadir, K., Camara, O., Ballester, M.A.G., et al.: Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved? IEEE transactions on medical imaging 37(11), 2514–2525 (2018)
  * [8] Cho, W., Park, J., Choo, J.: Training auxiliary prototypical classifiers for explainable anomaly detection in medical image segmentation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 2624–2633 (2023)
  * [9] Fournel, J., Bartoli, A., Bendahan, D., Guye, M., Bernard, M., Rauseo, E., Khanji, M.Y., Petersen, S.E., Jacquier, A., Ghattas, B.: Medical image segmentation automatic quality control: A multi-dimensional approach. Medical Image Analysis 74, 102213 (2021)
  * [10] Gal, Y., Ghahramani, Z.: Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In: International Conference on Machine Learning. pp. 1050–1059. PMLR (2016)
  * [11] Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: International conference on Machine Learning. pp. 1321–1330. PMLR (2017)
  * [12] Hann, E., Gonzales, R.A., Popescu, I.A., Zhang, Q., Ferreira, V.M., Piechnik, S.K.: Ensemble of deep convolutional neural networks with monte carlo dropout sampling for automated image segmentation quality control and robust deep learning using small datasets. In: Medical Image Understanding and Analysis. pp. 280–293. Springer International Publishing, Cham (2021)
  * [13] Herrera, W.G., Pereira, M., Bento, M., Lapa, A.T., Appenzeller, S., Rittner, L.: A framework for quality control of corpus callosum segmentation in large-scale studies. Journal of Neuroscience Methods 334, 108593 (2020)
  * [14] Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18(2), 203–211 (2021)
  * [15] Jin, K., Huang, X., Zhou, J., Li, Y., Yan, Y., Sun, Y., Zhang, Q., Wang, Y., Ye, J.: Fives: A fundus image dataset for artificial intelligence based vessel segmentation. Scientific Data 9(1), 475 (2022)
  * [16] Kahl, K.C., Lüth, C.T., Zenk, M., Maier-Hein, K., Jaeger, P.F.: Values: A framework for systematic validation of uncertainty estimation in semantic segmentation. arXiv preprint arXiv:2401.08501 (2024)
  * [17] Kohl, S., Romera-Paredes, B., Meyer, C., De Fauw, J., Ledsam, J.R., Maier-Hein, K., Eslami, S.M.A., Jimenez Rezende, D., Ronneberger, O.: A probabilistic u-net for segmentation of ambiguous images. In: International Conference on Neural Information Processing Systems. vol. 31. Curran Associates, Inc. (2018)
  * [18] Kohlberger, T., Singh, V., Alvino, C., Bahlmann, C., Grady, L.: Evaluating segmentation error without ground truth. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 528–536. Springer (2012)
  * [19] Köhler, P., Fadugba, J., Berens, P., Koch, L.M.: Efficiently correcting patch-based segmentation errors to control image-level performance in retinal images. In: Medical Imaging with Deep Learning – MIDL (2024)
  * [20] Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles (2017)
  * [21] Li, Z., Kamnitsas, K., Islam, M., Chen, C., Glocker, B.: Estimating model performance under domain shifts with class-specific confidence scores. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. pp. 693–703. Springer Nature Switzerland, Cham (2022)
  * [22] Lin, Q., Chen, X., Chen, C., Garibaldi, J.M.: A novel quality control algorithm for medical image segmentation based on fuzzy uncertainty. IEEE Transactions on Fuzzy Systems pp. 2532 – 2544 (2022)
  * [23] Liu, Y., Ding, C., Tian, Y., Pang, G., Belagiannis, V., Reid, I., Carneiro, G.: Residual pattern learning for pixel-wise out-of-distribution detection in semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1151–1161 (2023)
  * [24] Ng, M., Guo, F., Biswas, L., Wright, G.A.: Estimating uncertainty in neural networks for segmentation quality control. In: 32nd International Conference on Neural Information Processing Systems (NIPS 2018), Montréal, Canada, no. NIPS. pp. 3–6 (2018)
  * [25] Puyol-Antón, E., Ruijsink, B., Baumgartner, C.F., Masci, P.G., Sinclair, M., Konukoglu, E., Razavi, R., King, A.P.: Automated quantification of myocardial tissue characteristics from native t1 mapping using neural networks with uncertainty-based quality-control. Journal of Cardiovascular Magnetic Resonance 22(1), 60 (2020)
  * [26] Robinson, R., Oktay, O., Bai, W., Valindria, V.V., Sanghvi, M.M., Aung, N., Paiva, J.M., Zemrak, F., Fung, K., Lukaschuk, E., Lee, A.M., Carapella, V., Kim, Y.J., Kainz, B., Piechnik, S.K., Neubauer, S., Petersen, S.E., Page, C., Rueckert, D., Glocker, B.: Real-time prediction of segmentation quality. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2018. pp. 578–585. Springer International Publishing, Cham (2018)
  * [27] Sohn, K., Lee, H., Yan, X.: Learning structured output representation using deep conditional generative models. International Conference on Neural Information Processing Systems 28 (2015)
  * [28] Sunoqrot, M.R., Selnæs, K.M., Sandsmark, E., Nketiah, G.A., Zavala-Romero, O., Stoyanova, R., Bathen, T.F., Elschot, M.: A quality control system for automated prostate segmentation on t2-weighted mri. Diagnostics 10(9), 714 (2020)
  * [29] Valindria, V.V., Lavdas, I., Bai, W., Kamnitsas, K., Aboagye, E.O., Rockall, A.G., Rueckert, D., Glocker, B.: Reverse classification accuracy: predicting segmentation performance in the absence of ground truth. IEEE transactions on medical imaging 36(8), 1597–1606 (2017)
  * [30] Vovk, V., Gammerman, A., Shafer, G.: Algorithmic learning in a random world, vol. 29. Springer (2005)
  * [31] Williams, E., Niehaus, S., Reinelt, J., Merola, A., Mihai, P.G., Villringer, K., Thierbach, K., Medawar, E., Lichterfeld, D., Roeder, I., et al.: Automatic quality control framework for more reliable integration of machine learning-based image segmentation into medical workflows. arXiv preprint arXiv:2112.03277 (2021)
  * [32] Zhang, L., Gooya, A., Dong, B., Hua, R., Petersen, S.E., Medrano-Gracia, P., Frangi, A.F.: Automated quality assessment of cardiac mr images using convolutional neural networks. In: Simulation and Synthesis in Medical Imaging: First International Workshop, SASHIMI 2016, Held in Conjunction with MICCAI 2016, Athens, Greece, October 21, 2016, Proceedings 1. pp. 138–145. Springer (2016)

## Supplementary Materials

###  A Qualitative examples

Figure 4: PHiSeg segmentations and performance predictions for three examples.
Top: An example with good segmentation performance. Middle: An example with
poor image quality and poor segmentation performance. Bottom: An example with
good image quality but poor segmentation performance due to visible
pathologies stemming from advanced diabetic retinopathy.

###  B Conditional coverage grouped by quality

Figure 5: Marginal and conditional coverage for multiple interval sizes: very
small (0, 0.1], small (0.1, 2], large (0.2, 5]), very large (0.5, 1]) for high
quality images (left) and poor quality images (right).


# A note on the shifted Courant-Nijenhuis torsion

Marco Aldi, Sergio Da Silva, and Daniele Grandini

###### Abstract.

We characterize the vanishing of the shifted Courant-Nijenhuis torsion as the
strongest tensorial integrability condition that can be imposed on a skew-
symmetric endomorphism of the generalized tangent bundle.

###### Key words and phrases:

Generalized geometry, Courant algebroids, polynomial structures

###### 2020 Mathematics Subject Classification:

53D18, 14P99

##  1\. Introduction

Let $M$ be a (paracompact) smooth manifold and let $\mathbb{T}M$ be its
generalized tangent bundle (i.e. the direct sum of its tangent and cotangent
bundles). A generalized almost complex structure on $M$ is a skew-symmetric
(with respect to the tautological inner product on $\mathbb{T}M$) bundle
endomorphism $J:\mathbb{T}M\to\mathbb{T}M$ such that $J^{2}+{\rm id}=0$.
Generalized almost complex structures that satisfy the integrability condition

(1)
$0=\mathcal{T}_{J}(\mathbf{x},\mathbf{y})={[}\\!{[}J(\mathbf{x}),J(\mathbf{y}){]}\\!{]}+J^{2}({[}\\!{[}\mathbf{x},\mathbf{y}{]}\\!{]})-J({[}\\!{[}J(\mathbf{x}),\mathbf{y}{]}\\!{]}+{[}\\!{[}\mathbf{x},J(\mathbf{y}){]}\\!{]})$

for all smooth sections $\mathbf{x},\mathbf{y}$ of $\mathbb{T}M$, where
${[}\\!{[}\cdot,\cdot{]}\\!{]}$ denotes the Courant-Dorfman bracket, are known
as generalized complex structures and play a special role in Generalized
Geometry [Gualtieri11]. The map $\mathcal{T}_{J}$, known as the Courant-
Nijenhus torsion of $J$, has the remarkable property of being tensorial in the
sense that
$\mathcal{T}_{J}(f\mathbf{x},\mathbf{y})=f\mathcal{T}_{J}(\mathbf{x},\mathbf{y})=\mathcal{T}_{J}(\mathbf{x},f\mathbf{y})$
whenever $f$ is a smooth function on $M$ and $\mathbf{x},\mathbf{y}$ are
smooth sections of $\mathbb{T}M$. As a result, it suffices to check the
integrability condition (1) on local frames. Moreover, since the Courant-
Dorfman bracket is the derived bracket of the de Rham operator, any non-
tensorial compatibility condition with an endomorphism (which is by definition
$C^{\infty}(M)$-linear) of $\mathbb{T}M$ would necessarily impose non-trivial
differential equations to be satisfied by all smooth functions on $M$.

Motivated by earlier analysis of the cubic case [Vaisman08, PoonWade11] and
classical work on higher degree structures on the tangent bundle
[GoldbergYano70, GoldbergPetridis73], the notion of a generalized polynomial
structure was introduced in [AldiGrandini22]. A generalized polynomial
structure extends the notion of a generalized almost complex structure by
relaxing the $J^{2}+{\rm id}=0$ condition to $P(J)=0$ with $P$ an arbitrary
polynomial in $\mathbb{R}[x]$. Many features of generalized almost complex
structures (such as the decomposition of the complexification of $\mathbb{T}M$
into subspaces labeled by roots of $P$) carry through to arbitrary generalized
polynomial structures. However the integrability condition (1) does not
straightforwardly apply to arbitrary polynomial structures. One explanation
for this is that the Courant-Nijenhuis torsion is not tensorial for arbitrary
generalized polynomial structures. On the other hand, remarkably, the so-
called shifted Courant-Nijenhuis torsion

(2)
$\mathcal{S}_{J}(\mathbf{x},\mathbf{y})=\mathcal{T}_{J}(J(\mathbf{x}),\mathbf{y})+\mathcal{T}_{J}(\mathbf{x},J(\mathbf{y}))$

is tensorial for any skew-symmetric endomorphism $J$ of $\mathbb{T}M$.
Moreover, the integrability of almost complex structures is equivalent to the
vanishing of their shifted Courant-Nijenhuis. In fact, as argued in
[AldiGrandini22], the vanishing of the shifted Courant-Nijenhuis torsion is a
sensible integrability condition for generic generalized polynomial structures
(extra care must be taken when the polynomial $P$ has repeated roots).

It is natural to ask whether the shifted Courant-Nijenhuis torsion is special
in this regard and whether there are analogous tensorial expressions whose
vanishing could give rise to exotic integrability conditions. In this paper we
show that the shifted Courant-Nijenhuis torsion is the strongest possible
tensorial condition expressed (in a precise way which will be specified below)
in terms of the Courant-Dorfman bracket that can be uniformly imposed on all
skew-symmetric endomorphisms of the generalized tangent bundle. Our main
observation is that, using a construction of Kosmann-Schwarzbach [Kosmann-
Schwarzbach19], “Nijenhuis-like” expressions can be represented by real
polynomials in three variables and polynomials representing tensorial
expressions form an ideal which, as we prove using methods of real algebraic
geometry, is generated by the polynomial representing the shifted Courant-
Nijenhuis torsion. This is the content of Theorem 14.

It seems plausible that our techniques yield results in a much more general
context. We conclude this paper by listing some possible directions for
further research such as symmetric endomorphisms (modeled after generalized
Riemannian structures), and tensorial objects on more general algebroids.
Since our primary goal is to provide additional context for our main result,
we limit ourselves to a sketch of these open problems, leaving a systematic
treatment for future work.

##  2\. Preliminaries

In this section we will review the geometry of the generalized tangent bundle
(referring the reader to [Gualtieri11] for further details) and introduce the
basic vocabulary needed to state our main theorem.

The generalized tangent bundle $\mathbb{T}M=TM\oplus T^{*}M$ of a manifold $M$
has the following canonical structures:

  1. i)

the anchor map, defined as the projection map $\pi:\mathbb{T}M\rightarrow TM$;

  2. ii)

the tautological inner product $\langle\cdot,\cdot\rangle$, which is the
nondegenerate, symmetric $C^{\infty}(M)$-bilinear map defined by

$\langle X+\alpha,Y+\beta\rangle=\frac{1}{2}\left(\alpha(Y)+\beta(X)\right)$

for all sections $X,Y\in\Gamma(TM)$ and $\alpha,\beta\in\Omega_{M}^{1}$;

  3. iii)

the $\mathbb{R}$-bilinear map
${[}\\!{[}\cdot,\cdot{]}\\!{]}:\Gamma(\mathbb{T}M)\otimes_{\mathbb{R}}\Gamma(\mathbb{T}M)\rightarrow\Gamma(\mathbb{T}M)$
such that

${[}\\!{[}X+\alpha,Y+\beta{]}\\!{]}=[X,Y]+{\mathcal{L}}_{X}\beta-
i_{Y}d\alpha,$

for all $X,Y\in\Gamma(TM)$ and $\alpha,\beta\in\Omega_{M}^{1}$, called the
Courant-Dorfman bracket on $M$.

Here $\mathcal{L}_{X}$ is the Lie derivative along the vector field $X$, and
$i_{Y}$ is the interior derivative along $Y$. The data
$(\mathbb{T}M,\pi,\langle\cdot,\cdot\rangle,{[}\\!{[}\cdot,\cdot{]}\\!{]})$
defines a Courant Algebroid [Gualtieri11]. In particular, the following
property is satisfied:

(3)
$\pi(\mathbf{x})\langle\mathbf{y},\mathbf{z}\rangle=\langle{[}\\!{[}\mathbf{x},\mathbf{y}{]}\\!{]},\mathbf{z}\rangle+\langle{[}\\!{[}\mathbf{x},\mathbf{z}{]}\\!{]},\mathbf{y}\rangle=\langle{[}\\!{[}\mathbf{y},\mathbf{z}{]}\\!{]},\mathbf{x}\rangle+\langle{[}\\!{[}\mathbf{z},\mathbf{y}{]}\\!{]},\mathbf{x}\rangle\,,$

for all $\mathbf{x},\mathbf{y},\mathbf{z}\in\Gamma(\mathbb{T}M)$, where
$\pi(\mathbf{x})\langle\mathbf{y},\mathbf{z}\rangle$ denotes the usual action
of the vector field $\pi(\mathbf{x})$ on the smooth function
$\langle\mathbf{y},\mathbf{z}\rangle$. It well-known that (3) implies the
Leibniz identities

(4)
${[}\\!{[}\mathbf{x},f\mathbf{y}{]}\\!{]}=f{[}\\!{[}\mathbf{x},\mathbf{y}{]}\\!{]}+\pi(\mathbf{x})(f)\mathbf{y},\qquad{[}\\!{[}f\mathbf{x},\mathbf{y}{]}\\!{]}=f{[}\\!{[}\mathbf{x},\mathbf{y}{]}\\!{]}-\pi(\mathbf{y})(f)\mathbf{x}+2\langle\mathbf{x},\mathbf{y}\rangle
df$

for all $\mathbf{x},\mathbf{y}\in\Gamma(\mathbb{T}M)$ and $f\in
C^{\infty}(M)$.

######  Definition 1.

Let $\Theta$ be the $C^{\infty}(M)$-module of $\mathbb{R}$-trilinear forms

(5)
$\tau:\Gamma(\mathbb{T}M)\otimes_{\mathbb{R}}\Gamma(\mathbb{T}M)\otimes_{\mathbb{R}}\Gamma(\mathbb{T}M)\rightarrow
C^{\infty}(M)\,.$

Distinguished, among all element of $\Theta$ is the Courant element $\tau_{C}$
such that

(6)
$\tau_{C}(\mathbf{x},\mathbf{y},\mathbf{z})=\langle{[}\\!{[}\mathbf{x},\mathbf{y}{]}\\!{]},\mathbf{z}\rangle$

for all $\mathbf{x},\mathbf{y},\mathbf{z}\in\Gamma(\mathbb{T}M)$ [Courant90].
Moreover, any vector bundle morphism $J:\mathbb{T}M\rightarrow\mathbb{T}M$
induces an action $\bullet_{J}$ of the polynomial ring $\mathbb{R}[x,y,z]$ on
$\Theta$, uniquely defined by

$\displaystyle x\bullet_{J}\tau(\mathbf{x},\mathbf{y},\mathbf{z})$
$\displaystyle=\tau(J(\mathbf{x}),\mathbf{y},\mathbf{z}),$ $\displaystyle
y\bullet_{J}\tau(\mathbf{x},\mathbf{y},\mathbf{z})$
$\displaystyle=\tau(\mathbf{x},J(\mathbf{y}),\mathbf{z}),$ $\displaystyle
z\bullet_{J}\tau(\mathbf{x},\mathbf{y},\mathbf{z})$
$\displaystyle=\tau(\mathbf{x},\mathbf{y},J(\mathbf{z})).$

######  Remark 2.

The polynomial action $\bullet_{J}$ was first introduced in [Kosmann-
Schwarzbach19] (see also [AldiGrandini22] for applications to polynomial
structures).

######  Example 3.

Let $J:\mathbb{T}M\rightarrow\mathbb{T}M$ be a skew-symmetric endomorphism.
Then

(7)
$(x+z)(y+z)\bullet_{J}\tau_{C}=\langle{\mathcal{T}}_{J}(\mathbf{x},\mathbf{y}),\mathbf{z}\rangle$

where $\mathcal{T}_{J}$ is the Courant-Nijenhuis torsion of $J$ defined in
Equation (1). A direct calculation shows that if $J$ is a generalized almost
complex structure, then the $\mathbb{R}$-trilinear form
$\langle{\mathcal{T}}_{J}(\mathbf{x},\mathbf{y}),\mathbf{z}\rangle$ is in fact
a tensor (i.e. $C^{\infty}(M)$-trilinear).

######  Definition 4.

A tensorial pair is given by $(P,J)$ where $P=P(x,y,z)\in\mathbb{R}[x,y,z]$,
and $J$ is a skew endomorphism such that $P\bullet_{J}\tau_{C}$ is
$C^{\infty}(M)$-trilinear. Moreover, a polynomial $P$ is called tensorial if
$(P,J)$ is a tensorial pair for all skew endomorphisms
$J:\mathbb{T}M\rightarrow\mathbb{T}M$ and any smooth manifold $M$.

######  Example 5.

Example 3 can be now reframed as the statement that $((x+z)(y+z),J)$ is a
tensorial pair whenever $J$ is an almost complex structure. Let us show that
the Courant-Nijenhuis torsion is not in general tensorial by exhibiting a
skew-symmetric endomorphism $J$ of $\mathbb{T}M$ such that the pair
$((x+z)(y+z),J)$ is not a tensorial pair. Let $M$ be the three-dimensional
Heisenberg Nilmanifold i.e. the quotient of the group of 3-by-3 upper-
triangular real matrices with 1s along the diagonal by the subgroup of
matrices with integer entries. Let us fix a global frame $X_{1},X_{2},X_{3}$
(so that $[X_{i},X_{j}]=\varepsilon_{ij}^{k}X_{k}$) and let us denote its
corresponding coframe by $\alpha_{1},\alpha_{2},\alpha_{3}$. Consider the
skew-symmetric endomorphism $J$ of $\mathbb{T}M$ such that $JX_{1}=X_{2}$,
$JX_{2}=X_{3}+\alpha_{3}$, $JX_{3}=-\alpha_{2}$, $J\alpha_{1}=0$,
$J\alpha_{2}=-\alpha_{1}$, and $J\alpha_{3}=-\alpha_{2}$. Then, setting
$\mathbf{x}=X_{3}+\alpha_{3}$ and $\mathbf{y}=X_{3}-\alpha_{3}$ and letting
$f$ be an arbitrary smooth function on $M$ we obtain

(8)
${\mathcal{T}}_{J}(f\mathbf{x},\mathbf{y})-f\mathcal{T}_{J}(\mathbf{x},\mathbf{y})=-2J({[}\\!{[}-f\alpha_{2},X_{3}{]}\\!{]})-2fJ({[}\\!{[}\alpha_{2},X_{3}{]}\\!{]})=2X_{3}(f)\alpha_{1}\,,$

which vanishes only under the additional requirement $X_{3}(f)=0$. Hence the
Courant-Nijenhuis polynomial $(x+z)(y+z)$ is not tensorial.

It follows from Equation (8) that imposing the non-tensorial condition
$\mathcal{T}_{J}(\mathbf{x},\mathbf{y})=0$ on all smooth sections
$\mathbf{x},\mathbf{y}$ of $\mathbb{T}M$ results in a differential equation
$X_{3}(f)=0$ imposed on all smooth functions $f$ on $M$. This is a general
feature of non-tensorial expressions written in terms of the Courant-Dorfman
bracket, which is the derived bracket of the de Rham operator [Kosmann-
Schwarzbach04].

As another example, consider the non-tensorial condition
$(x+y)\bullet_{J}\tau_{C}=0$ for some (necessarily $C^{\infty}(M)$-linear)
vector bundle endomorphism $J$ of $\mathbb{T}M$. Suppose
$\mathbf{x},\mathbf{y}$ are smooth sections of $\mathbb{T}M$ satisfying this
condition (i.e.
${[}\\!{[}J(\mathbf{x}),\mathbf{y}{]}\\!{]}+{[}\\!{[}\mathbf{x},J(\mathbf{y}){]}\\!{]}=0$)
and let $f\in C^{\infty}(M)$. Then $(x+y)\bullet_{J}\tau_{C}=0$ implies

(9)
$0={[}\\!{[}J(\mathbf{x}),f\mathbf{y}{]}\\!{]}+{[}\\!{[}\mathbf{x},J(f\mathbf{y}){]}\\!{]}=\pi(\mathbf{x})(f)J(\mathbf{y})+\pi(J(\mathbf{x}))(f)\mathbf{y}\,,$

which is a non-trivial system of differential equations imposed on all smooth
functions of $M$. We conclude that any sensible compatibility condition
between the Courant-Dorfman bracket and a bundle endomorphism $J$ of
$\mathbb{T}M$ is necessarily tensorial.

######  Example 6.

Let $J:\mathbb{T}M\to\mathbb{T}M$ be a generalized polynomial structure such
that $P(J)=0$ for some polynomial $P\in\mathbb{R}[t]$. An integrability
condition (called minimality in [AldiGrandini22]) for generalized polynomial
structures can be stated as the vanishing of the trilinear form

(10)
$\mathcal{C}_{J}(\mathbf{x},\mathbf{y},\mathbf{z})=\sum_{i=0}^{N}\sum_{i_{1}+i_{2}+i_{3}=i}a_{i}(-1)^{i}\binom{i}{i_{1},i_{2},i_{3}}\tau_{C}(J^{i_{1}}(\mathbf{x}),J^{i_{2}}(\mathbf{y}),J^{i_{3}}(\mathbf{z}))$

for all $\mathbf{x},\mathbf{y},\mathbf{z}\in\mathbb{T}M$, where
$P(t)=a_{0}+a_{1}t+\cdots+a_{N}t^{N}$. In the case in which $J$ is a
generalized almost complex structure, (10) is equivalent to (1). As it turns
out, the $\mathbb{R}$-trilinear form $\mathcal{C}_{J}$ is actually
$C^{\infty}(M)$-trilinear and equal to $Q\bullet_{J}\tau_{C}$ where
$Q(x,y,z)=P(x+y+z)$. Hence $(Q,J)$ is a tensorial pair.

######  Proposition 7.

Let $P\in\mathbb{R}[x,y,z]$, and let $J:\mathbb{T}M\rightarrow\mathbb{T}M$ be
skew. The pair $(P,J)$ is tensorial if and only if the following conditions
are satisfied:

(11)
$\displaystyle\sum_{i,j,t}(-1)^{j}a_{i,j,t-j}\pi\left(J^{i}(\mathbf{x})\right)(f)J^{t}$
$\displaystyle=0$ (12)
$\displaystyle\sum_{i,j,t}\left((-1)^{j}a_{j,i,t-j}\pi\left(J^{i}(\mathbf{y})\right)(f)J^{t}(\mathbf{z})\right)$
$\displaystyle=\sum_{i,j,t}\left((-1)^{j}a_{j,t-j,i}\pi\left(J^{i}(\mathbf{z})\right)(f)J^{t}(\mathbf{y})\right)$

for all $\mathbf{x},\mathbf{y},\mathbf{z}\in\Gamma(\mathbb{T}M)$ and $f\in
C^{\infty}(M)$, where $P(x,y,z)=\sum_{i,j,k}a_{i,j,k}x^{i}y^{j}z^{k}$.

###### Proof.

$P\bullet_{J}\tau_{C}$ is a tensor if and only if

(13)
$\sum_{i,j,k}a_{i,j,k}\left(\langle{[}\\!{[}J^{i}(\mathbf{x}),J^{j}(f\mathbf{y}){]}\\!{]},J^{k}(\mathbf{z})\rangle-f\langle{[}\\!{[}J^{i}(\mathbf{x}),J^{j}(\mathbf{y}){]}\\!{]},J^{k}(\mathbf{z})\rangle\right)=0\,,$

and

(14)
$\sum_{i,j,k}a_{i,j,k}\left(\langle{[}\\!{[}J^{i}(f\mathbf{x}),J^{j}(\mathbf{y}){]}\\!{]},J^{k}(\mathbf{z})\rangle-f\langle{[}\\!{[}J^{i}(\mathbf{x}),J^{j}(\mathbf{y}){]}\\!{]},J^{k}(\mathbf{z})\rangle\right)=0$

hold for all $\mathbf{x},\mathbf{y},\mathbf{z}\in\Gamma(\mathbb{T}M)$ and
$f\in C^{\infty}(M)$. Using (4), Equations (13) and (14) are, respectively,
equivalent to

(15) $\sum_{i,j,k}a_{i,j,k}\pi(J^{i}(\mathbf{x}))(f)\langle
J^{j}(\mathbf{y}),J^{k}(\mathbf{z})\rangle=0$

and

(16) $\sum_{i,j,k}a_{i,j,k}\left(-\pi(J^{j}(\mathbf{y}))(f)\langle
J^{i}(\mathbf{x}),J^{k}(\mathbf{z})\rangle+\pi(J^{k}(\mathbf{z}))(f)\langle
J^{i}(\mathbf{x}),J^{j}(\mathbf{y})\rangle\right)=0\,.$

Using the skew-symmetry of $J$ and the nondegeneracy of the tautological inner
product shows that (15) is equivalent to (11). Similarly, (16) is equivalent
to (12). ∎

######  Theorem 8.

Let $P=\sum_{i,j,k}a_{i,j,k}x^{i}y^{j}z^{k}\in\mathbb{R}[x,y,z]$. The
following conditions are equivalent:

  1. (i)

$P$ is tensorial;

  2. (ii)

the following system of equations is satisfied:

$\sum_{j}(-1)^{j}a_{i,j,t-j}=0,\quad\sum_{j}(-1)^{j}a_{j,t-j,i}=0,\quad\sum_{j}(-1)^{j}a_{t-j,i,j}=0$

for all non-negative integers $i,t$;

  3. (iii)

The polynomial $P$ vanishes on the following subvariety of $\mathbb{R}^{3}$:

$V=\left\\{(x,y,z):y+z=0\right\\}\cup\left\\{(x,y,z):z+x=0\right\\}\cup\left\\{(x,y,z):x+y=0\right\\}\,.$

###### Proof.

It is clear that (iii) implies (ii) and, by Proposition 7, (ii) implies (i).
It remains to show that if (i) holds, then the polynomials
$A(u,v):=P(v,-u,u)$, $B(u,v):=P(-u,v,u)$, and $C(u,v):=P(-u,u,v)$ vanish
identically. Let $\lambda,\mu\in\mathbb{R}$. Since the tensoriality of $P$
does not depend on the manifold $M$, we can assume ${\rm dim}(M)\geq 2$ and
construct a skew-symmetric endomorphism $J:\mathbb{T}M\rightarrow\mathbb{T}M$
preserving the tangent bundle (i.e. $J(TM)\subseteq TM$ such that locally
there are linearly independent vector fields $X,Y\in\Gamma(TM)$ such that
$J(X)=\lambda X$, and $J(Y)=\mu Y$).

Under these assumptions, it follows from Proposition 7 that
$A(\lambda,\mu)X(f)Y=0$ and $B(\mu,\lambda)X(f)Y=C(\lambda,\mu)Y(f)X$ for all
$f\in C^{\infty}(M)$. Therefore,
$A(\lambda,\mu)=B(\mu,\lambda)=C(\lambda,\mu)=0$. ∎

######  Example 9.

Consider the shifted Courant-Nijenhuis polynomial

(17)
$S(x,y,z)=(x+y)(y+z)(z+x)=x^{2}y+xy^{2}+x^{2}z+xz^{2}+y^{2}z+yz^{2}+2xyz\,.$

By inspection, the equations in Theorem 8 (ii) hold and thus $S$ is tensorial.
It follows that the shifted Courant Nijenhuis torsion (2) is tensorial for any
skew-symmetric endomorphism $J$ of $\mathbb{T}M$.

Theorem 8 implies that the subset $\mathcal{I}\subseteq\mathbb{R}[x,y,z]$ of
all tensorial polynomials coincides with the ideal $I(V)$. With respect to the
diagonal scaling, we obtain the decomposition
$\mathcal{I}=\bigoplus_{D}\mathcal{I}^{D}$ into graded components. Moreover,
$\mathcal{I}$ is invariant under cyclic permutations of the variables. For
instance, ${\mathcal{I}}^{0}=0$, as the equations in Theorem 8 (ii) simply
reduce to $a_{000}=0$. When $D=1$, the Theorem 8 (ii) equations reduce to

$\\{a_{100}=0,a_{010}-a_{001}=0\ \ \mbox{ + c.p. }\\}$

where “c.p.”refers to equations obtained from the preceding equations by
permuting the indices cyclically. Again, this implies ${\mathcal{I}}^{1}=0$.
Similarly, for $D=2$ we obtain

$\\{a_{200}=0,a_{110}-a_{101}=0,a_{020}-a_{011}+a_{002}=0\ \ \mbox{ + c.p.
}\\}$

which yields ${\mathcal{I}}^{2}=0$. When $D=3$, the equations reduce to

$\\{a_{300}=0,a_{210}-a_{201}=0,a_{120}-a_{111}+a_{102}=0,a_{030}-a_{021}+a_{012}-a_{003}=0\
\ \mbox{ + c.p. }\\}$

which can be further reduced to

$\\{a_{300}=0,a_{210}=a_{201}=\frac{1}{2}a_{111}\ \ \mbox{ + c.p. }\\}$

Therefore, ${\mathcal{I}}^{3}$ is one-dimensional and is spanned by shifted
Courant-Nijenhuis polynomial (17). As we will show, all tensorial polynomials
can be expressed in terms of $S(x,y,z)$.

######  Definition 10.

Let $P\in\mathbb{R}[x,y,z]$ and let $J:\mathbb{T}M\rightarrow\mathbb{T}M$ be a
skew-symmetric endomorphism. The pair $(P,J)$ is called polynomially tensorial
if the equations in Theorem 8 (ii) hold and $P(J,0,0)=P(0,J,0)=P(0,0,J)$.

It is a straightforward to verify that polynomially tensorial pairs are
tensorial pairs (using Proposition 7). Also, if $P$ is a tensorial polynomial,
then $(P,J)$ is a polynomially tensorial pair for every skew-symmetric
endomorphism $J$ of $\mathbb{T}M$, as $P$ does not contain monomials in one
variable. The class of polynomially tensorial pairs generalizes the
construction of the tensor ${\mathcal{C}}_{J}$ in [AldiGrandini22] and in
Example 6.

######  Proposition 11.

If $(P,J)$ is polynomially tensorial pair, then the tensor
$P\bullet_{J}\tau_{C}$ is alternating.

###### Proof.

For all $\mathbf{x},\mathbf{y},\mathbf{z}\in\Gamma(\mathbb{T}M)$, let
$\theta(\mathbf{x},\mathbf{y},\mathbf{z})=\pi(\mathbf{x})\langle\mathbf{y},\mathbf{z}\rangle$
so that

(18)
$\left(P\bullet_{J}\theta\right)(\mathbf{x},\mathbf{y},\mathbf{z})=\sum_{i,j,t}(-1)^{j}a_{i,j,t-j}\pi\left(J^{i}(\mathbf{x})\right)\langle\mathbf{y},J^{t}(\mathbf{z})\rangle\,.$

Since $(P,J)$ is polynomially tensorial, then $P\bullet_{J}\theta=0$. Using
this and, (3), we obtain

(19)
$P\bullet_{J}\tau_{C}(\mathbf{x},\mathbf{y},\mathbf{z})-P\bullet_{J}\tau_{C}(\mathbf{x},\mathbf{z},\mathbf{y})=P\bullet_{J}\theta(\mathbf{x},\mathbf{y},\mathbf{z})=0$

and

(20)
$P\bullet_{J}\tau_{C}(\mathbf{x},\mathbf{y},\mathbf{z})-P\bullet_{J}\tau_{C}(\mathbf{y},\mathbf{x},\mathbf{z})=P\bullet_{J}\theta(\mathbf{z},\mathbf{x},\mathbf{y})=0\,,$

which concludes the proof. ∎

##  3\. The Main Result

Let $\mathcal{I}\subseteq\mathbb{R}[x,y,z]$ denote the ideal of all tensorial
polynomials. In the notation of Theorem 8, as a consequence of the Real
Nullstellensatz [Powers22, Theorem A.3], we obtain

(21) ${\mathcal{I}}=I(V)=\sqrt[Re]{I_{x}I_{y}I_{z}}\,,$

where $\sqrt[Re]{I_{x}I_{y}I_{z}}$ is the real radical of $I_{x}I_{y}I_{z}$
and $I_{x},I_{y},I_{z}$ are the following ideals of $\mathbb{R}[x,y,z]:$

(22) $I_{x}=\langle y+z\rangle\quad I_{y}=\langle z+x\rangle\quad
I_{z}=\langle x+y\rangle\,.$

We will show that $\sqrt[Re]{I_{x}I_{y}I_{z}}=I_{x}I_{y}I_{z}$.

######  Lemma 12.

Let $I_{x},I_{y}$ and $I_{z}$ be defined as above. Then $I_{x}\cap I_{y}\cap
I_{z}$ is a real radical ideal.

###### Proof.

Since each polynomial $x+y,y+z$ and $z+x$ is irreducible and each has a non-
singular zero in $\mathbb{R}^{3}$, $I_{x},I_{y}$ and $I_{z}$ are real radical
ideals by [Powers22, Theorem A.4]. To prove the result, it suffices to show
that the intersection of real radical ideals is again a real radical ideal. On
one hand $I\cap J\subseteq\sqrt[Re]{I\cap J}$. On the other hand
$\sqrt[Re]{I\cap J}\subseteq\sqrt[Re]{I}$ and $\sqrt[Re]{I\cap
J}\subseteq\sqrt[Re]{J}$, therefore $\sqrt[Re]{I\cap
J}\subseteq\sqrt[Re]{I}\cap\sqrt[Re]{J}=I\cap J$, as required. ∎

######  Lemma 13.

The ideal $I_{x}I_{y}I_{z}$ is radical. In particular,
$I_{x}I_{y}I_{z}=I_{x}\cap I_{y}\cap I_{z}$.

###### Proof.

In general, the intersection of finitely many principal ideals in a unique
factorization domain is again a principal ideal (generated by the least common
multiple of the irreducible factors).

More specifically for $I_{x}$ and $I_{y}$, since $I_{x}I_{y}\subseteq
I_{x}\cap I_{y}$, it suffices to show the reverse inequality. Let $f\in
I_{x}\cap I_{y}$. Then $f\in I_{x}$, so $f=r(y+z)$ for some
$r\in\mathbb{R}[x,y,z]$. Similarly, $f\in I_{y}$, so $f=s(z+x)$ for some
$s\in\mathbb{R}[x,y,z]$. Then $r(y+z)=s(z+x)$. Then $z+x$ divides $r(y+z)$.
Since $z+x$ is prime (equivalently irreducible in a UFD), $z+x$ must divide
$(y+z)$ or $r$. Since it doesn’t divide $y+z$, it must divide $r$, so
$r=\tilde{r}(z+x)$. Then $f=\tilde{r}(z+x)(y+z)\in I_{x}I_{y}$, as required.

A similar argument shows that $(I_{x}\cap I_{y})\cap I_{z}=(I_{x}I_{y})I_{z}$,
proving the result. ∎

######  Theorem 14.

Let $\mathcal{I}\subseteq\mathbb{R}[x,y,z]$ denote the set of all tensorial
polynomials. Then

$\mathcal{I}=I_{x}I_{y}I_{z}.$

###### Proof.

By Theorem 8, $\mathcal{I}=I(V)\subset\mathbb{R}[x,y,z]$ where
$V=Z(I_{x}I_{y}I_{z})$ (ie. the vanishing set of $I_{x}I_{y}I_{z}$). By the
real Nullstellensatz [Powers22, Theorem A.3],
$I(V)=\sqrt[Re]{I_{x}I_{y}I_{z}}$. On the other hand, Lemmas 12 and 13 imply

$\sqrt[Re]{I_{x}I_{y}I_{z}}=I_{x}I_{y}I_{z}$

which yields the conclusion. ∎

######  Corollary 15.

The ideal $\mathcal{I}$ of tensorial polynomials is generated by the shifted
Nijenhuis polynomial $S$.

###### Proof.

Clearly $\langle S\rangle\subseteq{\mathcal{I}}$. On the other hand, suppose
$P\in{\mathcal{I}}$. Since ${\mathcal{I}}=I_{x}I_{y}I_{z}$, we obtain
$P=Q\cdot(x+y)(y+z)(x+z)$ for some $Q\in\mathbb{R}[x,y,z]$. In other words,
$P\in\langle S\rangle$.

∎

##  4\. Generalizations

###  4.1. Symmetric endomorphisms

The skew-symmetry assumption on $J$ was used in the proof of Proposition 7,
though not in an essential way. A similar analysis can be carried out if $J$
is a symmetric endomorphisms of the generalized tangent bundle i.e. one for
which $\langle
J(\mathbf{x}),\mathbf{y}\rangle=\langle\mathbf{x},J(\mathbf{y})\rangle$ for
all $\mathbf{x},\mathbf{y}\in\Gamma(\mathbb{T}M)$. An important class of
examples of symmetric endomorphisms is given by _generalized Riemannian
metrics_ which satisfy the additional condition $J^{2}={\rm id}$. The
definition of tensorial pairs, polynomially tensorial pairs, and tensorial
polynomials can be transferred verbatim to the context of symmetric
endomorphisms of the generalized tangent bundle. Here is the analogue of
Theorem 8:

######  Theorem 16.

Let $P=\sum_{i,j,k}a_{i,j,k}x^{i}y^{j}z^{k}$. The following conditions are
equivalent:

  1. (i)

$P$ is tensorial (in the context of symmetric endomorphisms);

  2. (ii)

the following system of equations is satisfied:

$\sum_{j}a_{i,j,t-j}=0,\quad\sum_{j}a_{j,t-j,i}=0,\quad\sum_{j}a_{t-j,i,j}=0$

for all non-negative integers $i,t$;

  3. (iii)

The polynomial $P$ vanishes on the following subvariety of $\mathbb{R}^{3}$:

$V:=\left\\{(x,y,z):y-z=0\right\\}\cup\left\\{(x,y,z):z-x=0\right\\}\cup\left\\{(x,y,z):x-y=0\right\\}.$

Similar calculations show that the ideal $\mathcal{I}^{\prime}$ of tensorial
polynomials (in the symmetric setting) is generated by

$S^{\prime}(x,y,z)=(x-y)(y-z)(x-z)$

Note that when $J$ is a generalized Riemannian structure, we get
$S^{\prime}\bullet_{J}\tau_{C}=0$. In other words, this construction produces
no nontrivial analogue of the (shifted) Courant-Nijenhuis tensor for
generalized Riemannian structures.

###  4.2. More general algebroids

It is worthwhile to note that the computations in the paper do not make use of
the explicit expressions of the tautological inner product or the Courant-
Dorfman bracket. In fact, all results in the paper can be generalized verbatim
by replacing $\mathbb{T}M$ with a vector bundle $A$ with the following data:

  1. (1)

a vector bundle map $\pi:A\rightarrow TM$;

  2. (2)

a symmetric, nondegenerate inner product $\langle\cdot,\cdot\rangle$ on $A$;

  3. (3)

an $\mathbb{R}$-bilinear bracket ${[}\\!{[}\cdot,\cdot{]}\\!{]}$ on $A$;

  4. (4)

$\pi$, $\langle\cdot,\cdot\rangle$, and ${[}\\!{[}\cdot,\cdot{]}\\!{]}$
satisfy the identity in Equation (3).

We propose the terminology _Proto-Courant algebroid_ for a vector bundle $A$
with the structure defined in (1) to (4) above, generalizing the notion of
_Pre-Courant algebroid_ introduced in [Vaisman2004TransitiveCA]. Furthermore,
with the exception of Proposition 11, our results still hold true if $\pi$,
$\langle\cdot,\cdot\rangle$, and ${[}\\!{[}\cdot,\cdot{]}\\!{]}$ satisfy only
the Leibniz identities (4). These types of algebroids are known as _local
almost-Leibniz algebroids_ [PreLeibniz].

## Acknowledgements

This work is supported in part by VCU Quest Award “Quantum Fields and Knots:
An Integrative Approach.”

## References