11institutetext:  1Harvard University 2NIT Karnataka 3Technical University of
Munich 4Donostia International Physics Center (DIPC) 5University of the Basque
Country (UPV/EHU) 6Ikerbasque, Basque Foundation for Science
11email<EMAIL_ADDRESS>
# NucMM Dataset: 3D Neuronal Nuclei Instance Segmentation at Sub-Cubic
Millimeter Scale

Zudi Lin1 Equally contributed. Donglai Wei1$\dagger$ Mariela D. Petkova1
Yuelong Wu1 Zergham Ahmed1 Krishna Swaroop K2 Works were done during
internship at Harvard University. Silin Zou1 Nils Wendt3$\star$ Jonathan
Boulanger-Weill1 Xueying Wang1 Nagaraju Dhanyasi1 Ignacio Arganda-Carreras*4
5  6]  Florian Engert1 Jeff Lichtman1 Hanspeter Pfister1

###### Abstract

Segmenting 3D cell nuclei from microscopy image volumes is critical for
biological and clinical analysis, enabling the study of cellular expression
patterns and cell lineages. However, current datasets for neuronal nuclei
usually contain volumes smaller than $10^{\text{-}3}\ mm^{3}$ with fewer than
500 instances per volume, unable to reveal the complexity in large brain
regions and restrict the investigation of neuronal structures. In this paper,
we have pushed the task forward to the sub-cubic millimeter scale and curated
the NucMM dataset with two fully annotated volumes: one $0.1\ mm^{3}$ electron
microscopy (EM) volume containing nearly the entire zebrafish brain with
around 170,000 nuclei; and one $0.25\ mm^{3}$ micro-CT (uCT) volume containing
part of a mouse visual cortex with about 7,000 nuclei. With two imaging
modalities and significantly increased volume size and instance numbers, we
discover a great diversity of neuronal nuclei in appearance and density,
introducing new challenges to the field. We also perform a statistical
analysis to illustrate those challenges quantitatively. To tackle the
challenges, we propose a novel hybrid-representation learning model that
combines the merits of foreground mask, contour map, and signed distance
transform to produce high-quality 3D masks. The benchmark comparisons on the
NucMM dataset show that our proposed method significantly outperforms state-
of-the-art nuclei segmentation approaches. Code and data are available at
https://connectomics-bazaar.github.io/proj/nucMM/index.html.

###### Keywords:

3D Instance Segmentation Nuclei Brain Electron Microscopy (EM) Micro-CT
(uCT)Zebrafish Mouse

##  1 Introduction

Segmenting cell nuclei from volumetric (3D) microscopy images is an essential
task in studying biological systems, ranging from specific tissues [21] to
entire organs [29] and developing animals [19, 26]. There has been a great
success in benchmarking 2D and 3D nuclei segmentation methods using datasets
covering samples from various species [4, 34, 28, 31]. However, existing
datasets only have relatively small samples from brain tissues (e.g., volumes
from mouse and rat brains [24] smaller than $10^{-3}\ mm^{3}$, with less than
200 instances each), restricting the investigation of neuronal nuclei in a
larger and more diverse scale. Besides, most of the images are collected with
optical microscopy, which can not reflect the challenges in other imaging
modalities widely used in studying brain tissues, including electron
microscopy (EM) [15, 13, 25] and micro-CT (uCT) [9].

Figure 1: Overview of the NucMM dataset. NucMM contains two large volumes for
3D nuclei instance segmentation, including (left) the NucMM-Z electron
microscopy (EM) volume covering nearly a whole zebrafish brain, and (right)
the NucMM-M micro-CT volume from the visual cortex of a mouse.

To address this deficiency in the field, we have curated a large-scale 3D
nuclei instance segmentation dataset, NucMM, which is over two magnitudes
larger in terms of volume size and the number of instances than previous
neuronal nuclei datasets [4, 24] and widely-used non-neuronal benchmark
datasets [1, 34]. Our NucMM consists of one EM image volume covering a nearly
entire larval zebrafish brain and a uCT image volume from the visual cortex of
an adult mouse, facilitating large-scale cross-tissue and cross-modality
comparison. Challenges in the two volumes for automatic approaches include the
high density of closely touching instances (Fig. 1, left) and low contrast
between object and non-object regions (Fig. 1, right). We also perform a
statistical analysis to provide a quantitative justification of the challenges
in these two volumes (Fig. 2).

To tackle the challenges introduced by the large-scale NucMM dataset, we
propose a new hybrid-representation model that learns foreground mask,
instance contour map, and signed distance transform simultaneously with a 3D
U-Net [5] architecture, which is denoted as U3D-BCD. At inference time, we
utilize a multi-target watershed segmentation algorithm that combines all
three predictions to separate closely touching instances and generate high-
quality instance masks. Under the average precision (AP) metric for evaluating
instance segmentation approaches, we show that our U3D-BCD model significantly
outperforms existing state-of-the-art approaches by $\textbf{22}\%$ on the
NucMM dataset. We also perform ablation studies on the validation data to
demonstrate the sensitivity of segmentation parameters.

To summarize, we have three main contributions in this paper. First, we
collected and densely annotated the NucMM dataset, which is one of the largest
public neuronal nuclei instance segmentation datasets to date, covering two
species with two imaging modalities. Second, we propose a novel hybrid
representation model, U3D-BCD, to produce high-quality predictions by
combining the merits of different mask representations. Third, we benchmark
state-of-the-art nuclei segmentation approaches and show that our proposed
model outperforms existing approaches by a large margin on the NucMM dataset.

###  1.1 Related Works

Nuclei Segmentation Datasets. Automatic cell nuclei segmentation is a long-
lasting problem that is usually the first step in microscopy image analysis
[20]. There has been a great success in benchmarking nuclei segmentation
algorithms in 2D images across various types and experimental conditions [4].
However, 2D images can not completely display the structure and distribution
of nuclei. Therefore, several 3D nuclei segmentation datasets have been
collected, including the Parhyale dataset showing the histone fluorescent
protein expression of Parhyale hawaiensis [1, 34], the BBBC050 dataset showing
nuclei of mouse embryonic cells [28], and multiple 3D volumes recording human
cancer cells and animal embryonic cells [31]. However, there are few datasets
covering neuronal nuclei in brain tissues, and the volumes collected from
brain tissue usually contain less than 200 instances per volume [4, 24].
Besides, most public datasets mentioned above only have fluorescence images
obtained with optical microscopy.

The NucMM dataset we curated contains two volumes from animal brain tissues
and covers two imaging modalities widely used in neuroscience, including EM
[15, 25] and uCT [9]. In addition, our dataset is over two magnitudes larger
than previous neuronal nuclei volumes [24] in terms of size and instance
number.

Instance Segmentation in Microscopy Images. Instance segmentation requires
assigning each pixel (voxel) not only a semantic label but also an instance
index to differentiate objects that belong to the same category. The
permutation-invariance of object indices makes the task challenging. For
segmenting instances in microscopy images, recent learning-based approaches
first train 2D or 3D convolutional neural network (CNN) models to predict an
intermediate representation of the object masks such as boundary [6, 23] or
affinity maps [30, 17]. Then techniques including watershed transform [8, 36]
and graph partition [16] are applied to convert the representations into
object masks. Since one representation can be vulnerable to some specific
kinds of errors (e.g., small mispredictions on the affinity or boundary map
can cause significant merge errors), some approaches also employ hybrid-
representation learning models that learn multiple representations and combine
their information in the segmentation step [27, 34, 33]. However, those
approaches only optimize 2D CNNs [27] or learn targets that may not be
suitable for nuclei segmentation [33], which leads to unsatisfactory results
for down-stream analysis. Thus we propose a 3D hybrid-representation learning
model that predicts foreground mask, instance contour, and signed distance to
better capture nuclei with different textures and shapes.

##  2 NucMM Dataset

Dataset Acquisition. The EM dataset was collected from an entire larval
zebrafish brain with serial-section electron microscopy (SEM) [22]. The
original resolution is 30nm$\times$4nm$\times$4nm for $z$-, $y$-, $x$-axis. We
downsample the images to 480nm$\times$512nm$\times$512nm (Table 1) to make the
resolution close to other imaging modalities for nuclei analysis. The micro-CT
dataset was collected from layer II/III in the primary visual cortex of an
adult male mouse using 3D X-ray microscopy. The images have an isotropic voxel
size of 720nm3 (Table 1).

Table 1: NucMM dataset characteristics. We collected and fully annotated a
neuronal nuclei segmentation dataset at the sub-cubic millimeter scale. The
two volumes in the dataset cover two species and two imaging modalities.

Name | Sample | Modality | Volume Size | Resolution ($\mu$m) | #Instance
---|---|---|---|---|---
NucMM-Z | Zebrafish | SEM | 397$\times$1450$\times$2000 (0.14$mm^{3}$) | 0.48$\times$0.51$\times$0.51 | 170K
NucMM-M | Mouse | Micro-CT | 700$\times$996$\times$968 (0.25$mm^{3}$) | 0.72$\times$0.72$\times$0.72 | 7K

Dataset Annotation. For annotating both volumes, we use a semi-automatic
pipeline that first applies automatic algorithms to generate instance
candidates and then asks neuroscientists to proofread the masks. Since EM
images have high contrast and the object boundaries are well-defined (Fig. 1),
we apply filtering and thresholding to get the binary foreground mask and run
a watershed transform to produce the segments. For the uCT data, we
iteratively enlarge the labeled set by alternating between manual correction
(annotation) and automatic U-Net [23] prediction. Both volumes are finally
proofread by experienced neuroscientists using VAST [2]. We also provide a
binary mask for each volume to indicate the valid brain region for evaluation.

Dataset Statistics. The EM data has a size of 397$\times$1450$\times$2000
voxels, equivalent to a physical size of 0.14$mm^{3}$; the uCT data has
700$\times$996$\times$968 voxels, equivalent to 0.25$mm^{3}$ (Table 1).
Although the zebrafish volume is physically smaller, it contains significantly
more objects than the mouse volume, showing the distinction between brain
structures. We show the distribution of nuclei size (Fig. 2a) and the nearest-
neighbor distance between nuclei centers (Fig. 2b). The density plots
illustrate that objects in the zebrafish volume are smaller and more densely
packed, which poses challenges in separating closely touching nuclei.

We further show the voxel intensity distribution of object and non-object
regions (Fig. 2c). The boxplots demonstrate that objects in the EM volume are
better separated from the background than the uCT volume. To quantify the
separation, we calculate the Kullback–Leibler (KL) divergence between the
foreground and background intensity distributions in the two volumes. The
results are $D_{KL}=3.43$ and $1.33$ for the zebrafish and mouse data,
respectively, showing the foreground-background contrast is significantly
lower in the uCT data.

Dataset Splits & Evaluation Metric. We split each volume into $5\%$, $5\%$,
and $90\%$ parts for training, validation, and testing. The limited training
data makes the task more challenging, but it is also closer to the realistic
annotation budget when neuroscientists handle newly collected data. Besides,
to avoid sampling data in a local region without enough diversity, we follow
previous practice [14] and sample $27$ small chunks of size $64\times 64\times
64$ voxels from the zebrafish volume for training. Since the nuclei in the
mouse uCT volume are sparser (Fig. 1 and Fig. 2b), we sample $4$ chunks of
size $192\times 192\times 192$ voxels for training.

For evaluation, we use average precision (AP), a standard metric in assessing
instance segmentation methods [7, 18]. Specifically, we use the code optimized
for large-scale 3D image volumes [33] to facilitate efficient evaluation.

Figure 2: NucMM dataset statistics. We show the distribution of (a) instance
size (in terms of number of voxels) and (b) the distance between adjacent
nuclei centers. The density plots are normalized by the total number of
instances in each volume. We also show (c) the voxel intensity distribution in
object (foreground) and non-object (background) regions for both volumes.

##  3 Method

Figure 3: Hybrid-representation learning model. (a) Our U3D-BCD model learns a
set of hybrid representations simultaneously, including foreground mask,
instance contour, and signed distance transform map calculated from the
segmentation. (b) The representations are combined in seeding and watershed
transform to produce high-quality segmentation masks.

###  3.1 Hybrid-Representation Learning

Recent 3D instance segmentation methods for microscopy images, including
Cellpose [27], StarDist [34] and U3D-BC [33], all use a single model to learn
multiple mask representations simultaneously. Specifically, Cellpose [27]
regresses the horizontal and vertical spatial gradients of the instances;
StarDist [34] learns object probability and the star-convex distance within
the masks; U3D-BC [33] learns the foreground mask with the instance contour
map. By analyzing the representations used in those models, we notice that all
the targets emphasize the learning of object masks (foreground), but the
structure of the background of the segmentation map is less utilized. That is,
pixels close to and far away from the object masks are treated equitably.

Therefore based on the U3D-BC model that predicts both the binary foreground
mask (B) and instance contour map (C), we develop a U3D-BCD model that in
addition predicts a signed Euclidean distance map (Fig. 3a). Let $x_{i}$
denote a pixel in the image, we have:

$f(x_{i})=\begin{cases}+\text{dist}(x_{i},B)/\alpha,&\text{if $x\in F$}.\\\
-\text{dist}(x_{i},F)/\beta,&\text{if $x\in B$}.\end{cases}$ (1)

where $F$ and $B$ denote the foreground and background masks, respectively.
The scaling parameters $\alpha$ and $\beta$ are applied to control the range
of the distance. Compared with the U3D-BC baseline, the signed distance map is
more informative in capturing the shape information of masks. In comparison
with all discussed approaches [27, 34, 33], the signed distance map also model
the landscape of background regions. In implementation, we apply a tanh
activation to restrict the range to $(-1,1)$ and directly regress the target
with a 3D U-Net [5]. A similar learning target has been used for semantic
segmentation of synapses [12], but it has not been integrated into a multi-
task learning model nor has it been explored for 3D instance segmentation.
Specifically, the loss we optimize is

$\mathcal{L}_{bcd}=h(\sigma(y_{1}),y_{b})+h(\sigma(y_{2}),y_{c})+g(\phi(y_{3}),y_{d})$
(2)

where $y_{i}$ ($i=1,2,3$) denote the three output channels, while $\sigma$ and
$\phi$ denote the sigmoid and tanh function. The foreground ($y_{b}$) and
contour ($y_{c}$) maps are learned by optimizing the binary cross-entropy
(BCE) loss ($h$), while the signed distance map ($y_{d}$) is learned with the
mean-squared-error (MSE) loss ($g$).

###  3.2 Instance Decoding

In this part, we describe how to combine all three model predictions in the
U3D-BCD model to generate the segmentation masks. We first threshold
predictions to locate seeds (or markers) with high foreground probability and
distance value but low contour probability, which points to object centers. We
then apply the marker-controlled watershed transform algorithm (in the scikit-
image library [32]) using the seeds and the predicted distance map to produce
masks (Fig. 3b). There are two advantages over the U3D-BC [33] model, which
also uses marker-controlled watershed transform for decoding. First, we make
use of the consistency among the three representations to locate the seeds,
which is more robust than U3D-BC that uses two predictions. Furthermore, we
use the smooth signed distance map in watershed decoding, which can better
capture instance structure than the foreground probability map in U3D-BC. We
also show the sensitivity of decoding parameters in the experiments.

###  3.3 Implementation

We use a customized 3D U-Net model that substitutes the convolutional layers
at each stage with residual blocks [10]. We also change the concatenation
operation to addition to save memory. Since recent work [35] has shown that
ADAM-alike adaptive optimization algorithms do not generalize as well as
stochastic gradient descent (SGD) [3], we optimize our models with SGD and
adopt cosine learning rate decay [11]. We set the scaling parameters $\alpha$
and $\beta$ of the signed distance map to 8 and 50, respectively, without
tweaking. We follow the open-source code of U3D-BC111https://github.com/zudi-
lin/pytorch_connectomics and apply data augmentations including brightness,
flip, rotation, elastic transform, and missing parts augmentations to U3D-BC
[33] and our U3D-BCD. We also use the official implementation of
StarDist222https://github.com/stardist/stardist and
Cellpose333https://github.com/MouseLand/cellpose.

##  4 Experiments

Table 2:  Benchmark results on the NucMM dataset. We compare state-of-the-art methods on the NucMM dataset using AP score. Bold and underlined numbers denote the 1st and 2nd scores, respectively. Our U3D-BCD model significantly improves the performance of previously state-of-the-art approaches.  Method | NucMM-Z | NucMM-M |  Overall
---|---|---|---
AP-50 |  AP-75 |  Mean |  AP-50 |  AP-75 |  Mean
Cellpose [27] | 0.796 | 0.342 | 0.569 | 0.463 | 0.002 | 0.233 | 0.401
StarDist [34] | 0.912 | 0.328 | 0.620 | 0.306 | 0.004 | 0.155 | 0.388
U3D-BC [33] | 0.782 | 0.556 | 0.670 | 0.645 | 0.210 | 0.428 | 0.549
U3D-BCD (Ours) | 0.978 | 0.809 | 0.894 | 0.638 | 0.250 | 0.444 | 0.669

###  4.1 Methods in Comparison

We benchmark state-of-the-art microscopy image segmentation models including
Cellpose [27], StarDist [34] and U3D-BC [33] based on their public
implementations. Specifically, the Cellpose model is trained on the 2D $xy$,
$yz$, and $xz$ planes from the image volumes, and the predicted spatial
gradients are averaged to generate a 3D gradient before segmentation. For
StarDist, we calculate the optimal number of rays on the training data, which
are 96 and 64 for the zebrafish and mouse volumes, respectively. For U3D-BC,
we use the default 1.0 weight ratio between the losses of the foreground and
contour map. Models are trained on a machine with four Nvidia V100 GPUs.

###  4.2 Benchmark Results on the NucMM Dataset

After choosing hyper-parameter on the validation sets, we run predictions on
the $90\%$ test data in each volume and evaluate the performance.
Specifically, we show the AP scores at intersection-over-union (IoU)
thresholds of both 0.5 (AP-50) and 0.75 (AP-75), as well as their average
(Table 2). The overall score is averaged over two NucMM volumes. The results
show that our U3D-BCD model significantly outperforms previous state-of-the-
art models by relatively $\textbf{22}\%$ in overall performance. Besides, our
method ranks 1st in 6 out of 7 scores, showing its robustness in handling
different challenges. We argue that Cellpose [27] trains 2D models to estimate
3D spatial gradient, which can be ineffective for challenging 3D cases. The
other two models use 3D models, but the representations StarDist [34] uses
overlooks the background in the segmentation mask, while U3D-BC [33] overlooks
both foreground and background structures. Although conceptually
straightforward, introducing the signed distance on top of the U3D-BC baseline
gives a notable performance boost.

###  4.3 Sensitivity of the decoding parameters

We also show the sensitivity of the decoding hyper-parameters of our U3D-BCD
model. Specifically, there are 5 thresholds for segmentation: 3 values are the
thresholds of foreground probability ($\tau 1$), contour probability ($\tau
2$), and distance value ($\tau 3$) to decide the seeds. The other 2 values are
the thresholds of foreground probability ($\tau 4$) and distance value ($\tau
5$) to decide the valid foreground regions.

The validation results show that the final segmentation performance is not
sensitive to the $\tau 1$ and $\tau 2$ in deciding seeds. When fixing $\tau
4=0.2$ and $\tau 5=0.0$ changing the foreground probability $\tau 1$ from 0.4
to 0.8 only changes the AP-50 score between 0.943 and 0.946. While changing
the contour probability $\tau 2$ from 0.05 to 0.30 only changes the scores
from 0.937 to 0.946. However, when changing $\tau 4$ within the range of 0.1
to 0.4, the score changes from 0.872 to 0.946. Those results suggest that the
signed distance transform map contains important information about the object
structures and performs an important role in generating high-quality
segmentation masks.

##  5 Conclusion

In this paper, we introduce the large-scale NucMM dataset for 3D neuronal
nuclei segmentation in two imaging modalities and analysis the challenges
quantitatively. We also propose a simple yet effective model that
significantly outperforms existing approaches. We expect the densely annotated
dataset can inspire various applications beyond its original task, e.g.,
feature pre-training, shape analysis, and benchmarking active learning and
domain adaptation methods.

### Acknowledgments

This work has been partially supported by NSF award IIS-1835231 and NIH award
U19NS104653. We thank Daniel Franco-Barranco for setting up the challenge
using NucMM. M.D.P. would like to acknowledge the support of Howard Hughes
Medical Institute International Predoctoral Student Research Fellowship. I.A-C
would like to acknowledge the support of the Beca Leonardo a Investigadores y
Creadores Culturales 2020 de la Fundación BBVA.

## References

  * [1] Alwes, F., Enjolras, C., Averof, M.: Live imaging reveals the progenitors and cell dynamics of limb regeneration. Elife (2016)
  * [2] Berger, D.R., Seung, H.S., Lichtman, J.W.: Vast (volume annotation and segmentation tool): efficient manual and semi-automatic labeling of large 3D image stacks. Frontiers in neural circuits (2018)
  * [3] Bottou, L.: Stochastic gradient learning in neural networks. Proceedings of Neuro-Nımes (1991)
  * [4] Caicedo, J.C., Goodman, A., Karhohs, K.W., Cimini, B.A., Ackerman, J., Haghighi, M., Heng, C., Becker, T., Doan, M., McQuin, C., et al.: Nucleus segmentation across imaging experiments: the 2018 data science bowl. Nature methods (2019)
  * [5] Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3d u-net: learning dense volumetric segmentation from sparse annotation. In: MICCAI (2016)
  * [6] Ciresan, D., Giusti, A., Gambardella, L.M., Schmidhuber, J.: Deep neural networks segment neuronal membranes in electron microscopy images. In: NeurIPS (2012)
  * [7] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: CVPR (2016)
  * [8] Cousty, J., Bertrand, G., Najman, L., Couprie, M.: Watershed cuts: Minimum spanning forests and the drop of water principle. TPAMI (2008)
  * [9] Dyer, E.L., Roncal, W.G., Prasad, J.A., Fernandes, H.L., Gürsoy, D., De Andrade, V., Fezzaa, K., Xiao, X., Vogelstein, J.T., Jacobsen, C., et al.: Quantifying mesoscale neuroanatomy using x-ray microtomography. eneuro (2017)
  * [10] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
  * [11] He, T., Zhang, Z., Zhang, H., Zhang, Z., Xie, J., Li, M.: Bag of tricks for image classification with convolutional neural networks. In: CVPR (2019)
  * [12] Heinrich, L., Funke, J., Pape, C., Nunez-Iglesias, J., Saalfeld, S.: Synaptic cleft segmentation in non-isotropic volume electron microscopy of the complete drosophila brain. In: MICCAI (2018)
  * [13] Hildebrand, D.G.C., Cicconet, M., Torres, R.M., Choi, W., Quan, T.M., Moon, J., Wetzel, A.W., Champion, A.S., Graham, B.J., Randlett, O., et al.: Whole-brain serial-section electron microscopy in larval zebrafish. Nature 545(7654), 345–349 (2017)
  * [14] Januszewski, M., Kornfeld, J., Li, P.H., Pope, A., Blakely, T., Lindsey, L., Maitin-Shepard, J., Tyka, M., Denk, W., Jain, V.: High-precision automated reconstruction of neurons with flood-filling networks. Nature Methods (2018)
  * [15] Kasthuri, N., Hayworth, K.J., Berger, D.R., Schalek, R.L., Conchello, J.A., Knowles-Barley, S., Lee, D., Vázquez-Reina, A., Kaynig, V., Jones, T.R., et al.: Saturated reconstruction of a volume of neocortex. Cell (2015)
  * [16] Krasowski, N., Beier, T., Knott, G., Köthe, U., Hamprecht, F.A., Kreshuk, A.: Neuron segmentation with high-level biological priors. TMI (2017)
  * [17] Lee, K., Zung, J., Li, P., Jain, V., Seung, H.S.: Superhuman accuracy on the SNEMI3D connectomics challenge. arXiv:1706.00120 (2017)
  * [18] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: ECCV (2014)
  * [19] Lou, X., Kang, M., Xenopoulos, P., Munoz-Descalzo, S., Hadjantonakis, A.K.: A rapid and efficient 2d/3d nuclear segmentation method for analysis of early mouse embryo and stem cell image data. Stem cell reports (2014)
  * [20] Meijering, E.: Cell segmentation: 50 years down the road. Signal Processing Magazine (2012)
  * [21] Nhu, H.T.T., Drigo, R.A.E., Berggren, P.O., Boudier, T.: A novel toolbox to investigate tissue spatial organization applied to the study of the islets of langerhans. Scientific reports (2017)
  * [22] Petkova, M.: Correlative Light and Electron Microscopy in an Intact Larval Zebrafish. Ph.D. thesis, Harvard University (2020)
  * [23] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI (2015)
  * [24] Ruszczycki, B., Pels, K.K., Walczak, A., Zamłyńska, K., Such, M., Szczepankiewicz, A.A., Hall, M.H., Magalska, A., Magnowska, M., Wolny, A., et al.: Three-dimensional segmentation and reconstruction of neuronal nuclei in confocal microscopic images. Frontiers in neuroanatomy (2019)
  * [25] Shapson-Coe, A., Januszewski, M., Berger, D.R., Pope, A., Wu, Y., Blakely, T., Schalek, R.L., Li, P., Wang, S., Maitin-Shepard, J., et al.: A connectomic study of a petascale fragment of human cerebral cortex. bioRxiv (2021)
  * [26] Stegmaier, J., Amat, F., Lemon, W.C., McDole, K., Wan, Y., Teodoro, G., Mikut, R., Keller, P.J.: Real-time three-dimensional cell segmentation in large-scale microscopy data of developing embryos. Developmental cell (2016)
  * [27] Stringer, C., Wang, T., Michaelos, M., Pachitariu, M.: Cellpose: a generalist algorithm for cellular segmentation. Nature Methods (2021)
  * [28] Tokuoka, Y., Yamada, T.G., Mashiko, D., Ikeda, Z., Hiroi, N.F., Kobayashi, T.J., Yamagata, K., Funahashi, A.: 3d convolutional neural networks-based segmentation to acquire quantitative criteria of the nucleus during mouse embryogenesis. NPJ systems biology and applications (2020)
  * [29] Toyoshima, Y., Tokunaga, T., Hirose, O., Kanamori, M., Teramoto, T., Jang, M.S., Kuge, S., Ishihara, T., Yoshida, R., Iino, Y.: Accurate automatic detection of densely distributed cell nuclei in 3d space. PLoS computational biology (2016)
  * [30] Turaga, S.C., Briggman, K.L., Helmstaedter, M., Denk, W., Seung, H.S.: Maximin affinity learning of image segmentation. In: NeurIPS (2009)
  * [31] Ulman, V., Maška, M., Magnusson, K.E., Ronneberger, O., Haubold, C., Harder, N., Matula, P., Matula, P., Svoboda, D., Radojevic, M., et al.: An objective comparison of cell-tracking algorithms. Nature methods (2017)
  * [32] van der Walt, S., Schönberger, J.L., Nunez-Iglesias, J., Boulogne, F., Warner, J.D., Yager, N., Gouillart, E., Yu, T., the scikit-image contributors: scikit-image: image processing in Python. PeerJ (2014)
  * [33] Wei, D., Lin, Z., Franco-Barranco, D., Wendt, N., et al.: MitoEM dataset: Large-scale 3D mitochondria instance segmentation from EM images. In: MICCAI (2020)
  * [34] Weigert, M., Schmidt, U., Haase, R., Sugawara, K., Myers, G.: Star-convex polyhedra for 3d object detection and segmentation in microscopy. In: WACV (2020)
  * [35] Zhou, P., Feng, J., Ma, C., Xiong, C., HOI, S., et al.: Towards theoretically understanding why sgd generalizes better than adam in deep learning. arXiv preprint arXiv:2010.05627 (2020)
  * [36] Zlateski, A., Seung, H.S.: Image segmentation by size-dependent single linkage clustering of a watershed basin graph. arXiv:1505.00249 (2015)


# Optimal lower estimate for the first eigenvalue of the $p$-Laplacian in the
Euclidean sphere

Fábio R. dos Santos∗ and Matheus N. Soares  Departamento de Matemática
Universidade Federal de Pernambuco
50.740-540 Recife, Pernambuco
Brazil<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract.

An integral inequality is derived for compact submanifolds (with or without
boundary) in the unit sphere. This result leads to a characterization of
spheres.

###### Key words and phrases:

Compact submanifods, first eigenvalue, $p$-Laplacian

###### 2020 Mathematics Subject Classification:

Primary 53C42; Secondary 53A10, 53C20.

∗Corresponding author

##  1\. Introduction and statements of the main result

Given a compact Riemannian manifold $M^{n}$ we define the $p$-Laplacian on
$M^{n}$ as the second order quasilinear elliptic operator

$\Delta_{p}f=-{\rm div}(|\nabla f|^{p-2}\nabla f),\quad 1<p<\infty.$ (1.1)

When $p=2$, it is the usual Laplacian. In a similar way, we can consider the
eigenvalue problem of $\Delta_{p}$. We say that a real number $\lambda$ is a
Dirichlet (or Neumann) eigenvalue if there exists a non-zero function $f$
satisfying the following equation with Dirichlet boundary condition $f=0$ on
$\partial M$ (or Neumann boundary condition $u=\partial f/\partial\eta=0$ on
$\partial M$):

$\Delta_{p}f=\lambda|f|^{p-2}f\quad\mbox{in}\quad M^{n},$ (1.2)

where $\eta$ is the outward pointing normal unit vector field to $\partial M$.

According to [6], these numbers $\lambda$ forms a non-increasing sequence such
that there exists an isolated minimum eigenvalue called the first eigenvalue
of the $p$-Laplacian (see also [3]). So, the first nontrivial Dirichlet and
Neumann eigenvalue of $M^{n}$ are given by

$\mu_{1,p}(M)=\inf\left\\{\dfrac{\int_{M}|\nabla
f|^{p}dM}{\int_{M}|f|^{p}dM}\,;\,f\in
W^{1,p}_{0}(M)\backslash\\{0\\}\right\\},$

and

$\lambda_{1,p}(M)=\inf\left\\{\dfrac{\int_{M}|\nabla
f|^{p}dM}{\int_{M}|f|^{p}dM}\,;\,f\in
W^{1,p}(M)\backslash\\{0\\}\,\,\mbox{and}\,\int_{M}|f|^{p-2}f\,dM=0\right\\}.$
(1.3)

Since $\lambda_{1,p}(M)$ is obtained by imitating the closed case proof (see
[11]), we will use the same notation to denote the first eigenvalue of
$p$-laplacian of $M^{n}$ in the closed case.

In the geometric analysis theory, a natural way exists to relate the geometric
properties of a Riemannian manifold and the $p$-Laplacian through its
eigenvalues. The interaction between these objects and the geometry were
approached in Matei [8] which generalized many geometric intrinsic results
involving the first eigenvalue of the $p$-Laplacian that, in principle, was
proved only for the usual Laplacian. Among these results we highlight the
generalization for $p>2$ of the Chern’s comparison of geodesic balls, an
extension of Faber-Krahn inequality and a generalization of Lichnerowicz-Obata
theorem.

On the other hand, in the context of isometric immersions in the unit sphere
$\mathbb{S}^{N}$, Leung [5] established a sharp estimate between the
eigenvalues of the usual Laplacian for minimal submanifolds of sphere through
an integral inequality. In fact, Leung presented a lower bound for the square
length of the second fundamental form $S$ via any eigenvalue of the Laplace-
Beltrami operator. Besides that, he showed that spheres are obtained when the
equality happens. Years later, Liu and Zhang [7] proved a similar estimate for
arbitrary submanifolds of sphere. More precisely, they proved

######  Theorem 1.1.

Let $M^{n}$ be an $n$-dimensional compact orientable submanifold immersed in
the standard Euclidean sphere $\mathbb{S}^{N}$. Let $f$ be an eigenfunction
associated to $\lambda_{1,2}(M)$, then

$\int_{M}\left(\frac{\sqrt{n-1}}{2}S-\dfrac{(n-1)(n-\lambda_{1,2}(M))}{n}\right)|\nabla
f|^{2}dM\geq 0,$ (1.4)

where $dM$ is the volume element on $M^{n}$. In particular, if $S$ is a
constant, then

$S\geq\dfrac{2\sqrt{n-1}}{n}(n-\lambda_{1,2}(M)).$ (1.5)

We emphasize that, unlike the estimate obtained by Leung, the theorem above
does not describe which geometric object achieves equality in integral
inequality.

In a recent paper [1], the authors considered compact minimal submanifolds in
the unit sphere. They defined a divergence type operator and developed Bochner
and Reilly formulas for it. As an application, they obtained integral
inequalities involving the squared norm of the second fundamental form which
extended the previous result due to Leung [5] to the first eigenvalue of the
$p$-Laplacian and for manifolds with boundary. Here, our aim is to use the
machinery developed in [1] in order to generalizes the Liu-Zhang result to the
context of the $p$-Laplacian as well as to compact manifolds with nonempty
boundary and possibility convex111We say that $\partial M$ is convex if the
second fundamental form is negative semi-definite with respect to $\eta$ the
outward-pointing unit normal vector.. In other words, we prove

######  Theorem 1.2.

Let $M^{n}$ be a compact (with possibly convex boundary) submanifold immersed
in the standard Euclidean sphere $\mathbb{S}^{N}$. Let $f$ be an eigenfunction
of the $p$-Laplacian of $M^{n}$ associated to $\lambda$ given by equation
(1.2). Then

$\int_{M}\left(\frac{\sqrt{n-1}}{2}S-\dfrac{(n-1)(n-b(n,p)\lambda^{\frac{2}{p}})}{n}\right)|\nabla
f|^{2p-2}dM\geq 0,$ (1.6)

for $p\geq 2$, where

$b(n,p)=\dfrac{n(p-1)^{2}-1}{(n-1)(p-1)^{\frac{2}{p}}}.$ (1.7)

If the equality happen, then $p=2$. In this case, by assuming in addition that
$M^{n}$ has parallel mean curvature vector field in $\mathbb{S}^{N}$, then
$M^{n}$ is isometric to

  1. i.

the sphere $\mathbb{S}^{2}(\sqrt{q(q+1)/2})$, with $\lambda$ being the first
eigenvalue and $N=2q$, for the closed case if $n=2$;

  2. ii.

the great sphere $\mathbb{S}^{n}(1)$, with $\lambda$ being the first
eigenvalue, for the closed case if $n\geq 3$;

  3. iii.

the upper hemisphere $\mathbb{S}_{+}^{n}(1)$, with $\lambda$ being the first
eigenvalue, corresponding to Dirichlet and Neumann problems.

The proof of Theorem 1.2 is given in Section 3. As a direct consequence of
Theorem 1.2 we get the following Leung’s type estimate:

######  Corollary 1.3.

In the context of Theorem 1.2, if $S$ is constant, then

$S\geq\frac{2\sqrt{n-1}}{n}\left(n-b(n,p)\lambda_{1,p}^{2/p}(M)\right).$ (1.8)

##  2\. Some preliminaries and key lemmas

Let $M^{n}$ be an $n$-dimensional connected submanifold immersed in a unit
Euclidean sphere $\mathbb{S}^{N}$. Let $\\{\omega_{B}\\}$ be the corresponding
dual coframe, and $\\{\omega_{BC}\\}$ the connection $1$-forms on
$\mathbb{S}^{N}$. We choose a local field of orthonormal frame
$\\{e_{1},\ldots,e_{N}\\}$ in $\mathbb{S}^{N}$, with dual coframe
$\\{\omega_{1},\ldots,\omega_{N}\\}$, such that, at each point of $M^{n}$,
$e_{1},\ldots,e_{n}$ are tangent to $M^{n}$ and $e_{n+1},\ldots,e_{N}$ are
normal to $M^{n}$. We will use the following convection for indices

$\displaystyle 1\leq A,B,C,\ldots\leq N,\quad 1\leq i,j,k,\ldots\leq
n\quad\mbox{and}\quad n+1\leq\alpha,\beta,\gamma,\ldots\leq N.$

With restricting on $M^{n}$, the second fundamental form $A$ and the curvature
tensor $R$ of $M^{n}$ are given by

$\displaystyle\omega_{i\alpha}=\sum_{j}h_{ij}^{\alpha}\omega_{j},\quad
A=\sum_{i,j,\alpha}h_{ij}^{\alpha}\omega_{i}\otimes\omega_{j}\otimes
e_{\alpha},$ $\displaystyle
d\omega_{ij}=\sum_{k}\omega_{ik}\wedge\omega_{kj}-\frac{1}{2}\sum_{k,l}R_{ijkl}\omega_{k}\wedge\omega_{l}.$

The Gauss equation is

$\displaystyle
R_{ijkl}=(\delta_{ik}\delta_{jl}-\delta_{il}\delta_{jk})+\sum_{\alpha}(h_{ik}^{\alpha}h_{jl}^{\alpha}-h_{il}^{\alpha}h_{jk}^{\alpha}).$

In particular, the components of the Ricci tensor $R_{ik}$ and the normalized
scalar curvature $R$ are given, respectively, by

$\displaystyle
R_{ik}=(n-1)\delta_{ik}+n\sum_{\alpha}\left(\sum_{j}h_{jj}^{\alpha}\right)h_{ik}^{\alpha}-\sum_{\alpha,j}h_{ij}^{\alpha}h_{jk}^{\alpha}$
(2.1)

and

$\displaystyle R=\dfrac{1}{(n-1)}\sum_{i}R_{ii}.$ (2.2)

From (2.1) and (2.2), we get the following relation

$\displaystyle n(n-1)R=n(n-1)+n^{2}H^{2}-S,$ (2.3)

where

$S=\sum_{\alpha,i,j}(h_{ij}^{\alpha})^{2}\quad\mbox{and}\quad
h=\dfrac{1}{n}\sum_{\alpha}\left(\sum_{k}h_{kk}^{\alpha}\right)e_{\alpha},$
(2.4)

denotes, respectively, the squared norm of the second fundamental form and the
mean curvature vector field. Also, we define $H=|h|$ as the mean curvature
function of $M^{n}$. In particular, we say that a submanifold $M^{n}$ immersed
in the unit Euclidean sphere $\mathbb{S}^{N}$ has parallel mean curvature
vector field if $h$ is parallel as a section of the normal bundle of $M^{n}$.
It is clear that this condition implies in $H$ constant.

In order to proof our main results, we need of the following two results. The
first one is a Ricci low estimate due [4, Main Theorem].

######  Lemma 2.1.

Let $M^{n}$ be a submanifold of the Riemannian manifold $\mathbb{S}^{N}$. Let
${\rm Ric}$ denotes the function that assigns to each point of $M^{n}$ the
minimum Ricci curvature. Then

${\rm
Ric}\geq-\dfrac{n-1}{n}\left(S+\frac{n(n-2)}{\sqrt{n(n-1)}}H\sqrt{S-nH^{2}}-n-2nH^{2}\right).$
(2.5)

Before to present the second result, we will recall some facts about isometric
immersions with nonempty boundary. Let us consider $\eta$ the outer unit
normal field of $\partial M$. We define the shape operator
$\mathcal{A}_{\eta}$ and the mean curvature function of $\partial M$ in
$M^{n}$, respectively, by

$\mathcal{A}_{\eta}(X)=-\nabla_{X}\eta\quad\mbox{and}\quad\mathcal{H}=\dfrac{1}{n-1}{\rm
tr}(\mathcal{A}_{\eta}),$ (2.6)

for any $X\in\mathfrak{X}(\partial M)$. Let us denote by $\nabla^{\partial}$
and $\Delta^{\partial}$ the covariant derivative and the Laplacian operator on
$\partial M$ with respect to the induced Riemannian metric. In this picture,
the second key result is a suitable version of [1, Proposition $4.1$] for our
interest.

######  Lemma 2.2.

Let $M^{n}$ be a compact manifold with boundary $\partial M$. If $f$ is an
eigenfunction on $M^{n}$ of the $p$-Laplacian corresponding to a non-zero
eigenvalue $\lambda$, then

$\int_{M}{\rm Ric}(\nabla f,\nabla f)|\nabla
f|^{2p-4}dM\leq\lambda^{\frac{2}{p}}c(n,p)\mathcal{R}(f,z,u)^{\frac{2p-2}{p}}+\int_{\partial
M}|\nabla f|^{2p-4}Q(u,z)d\sigma,$ (2.7)

where

$\mathcal{R}(f,z,u)=\|\nabla f\|_{2p-2}^{p}-\|f\|_{2p-2}^{2-p}\int_{\partial
M}|f|^{p-2}|\nabla f|^{p-2}zu\,d\sigma$ (2.8)

and

$\mathcal{Q}(u,z)=u\Delta^{\partial}z+(n-1)u^{2}\mathcal{H}+\langle\nabla^{\partial}z,\nabla^{\partial}u\rangle+\langle\mathcal{A}_{\eta}(\nabla^{\partial}z),\nabla^{\partial}z\rangle,$
(2.9)

with $z=f\big{|}_{\partial M}$, $u=\partial f/\partial\eta$ and $d\sigma$
denotes the Riemannian volume element on $\partial M$. Moreover, the equality
holds in (2.7) if $p=2$.

###### Proof.

From [1, Proposition 4.1], we have

$\begin{split}\int_{M}{\rm Ric}&(\nabla u,\nabla u)|\nabla u|^{2p-4}dM\\\
&=\int_{M}\left((\Delta_{p}u)^{2}-|\nabla u|^{2p-4}|{\rm
Hess}\,u|^{2}\right)dM+\int_{\partial M}|\nabla u|^{2p-4}Q(u,z)d\sigma\\\
&\quad-(p-2)\int_{M}|\nabla u|^{2p-6}\left((p-2)(\Delta_{\infty}u)^{2}|\nabla
u|^{2}+2|{\rm Hess}\,u(\nabla u)|^{2}\right)dM,\end{split}$ (2.10)

where $\mathcal{Q}(u,z)$ is given in (2.8) and $|\nabla
f|^{2}\Delta_{\infty}f=\langle{\rm Hess}\,f(\nabla f),\nabla f\rangle$.

On the other hand, we note that holds the following algebraic inequality [1,
Equation $3.7$]

$|\nabla f|^{2p-4}|{\rm
Hess}\,f|^{2}\geq\dfrac{(\Delta_{p}f)^{2}}{n(p-1)^{2}},$ (2.11)

with equality holding if and only if

$(p-2)|\nabla f|^{p-4}\langle{\rm Hess}\,f(\nabla f),X\rangle\nabla f+|\nabla
f|^{p-2}{\rm Hess}\,f(X)=-\dfrac{1}{n}(\Delta_{p}f)I(X),$ (2.12)

for all $X\in\mathfrak{X}(M)$. Besides these, the Cauchy-Schwarz inequality
guarantees

$|\nabla f|^{4}(\Delta_{\infty}f)^{2}=\langle{\rm Hess}\,f(\nabla f),\nabla
f\rangle^{2}\leq|\nabla f|^{2}|{\rm Hess}\,f(\nabla f)|^{2}$ (2.13)

and consequently

$p(\Delta_{\infty}f)^{2}|\nabla f|^{2}\leq(p-2)(\Delta_{\infty}f)^{2}|\nabla
f|^{2}+2|{\rm Hess}\,f(\nabla f)|^{2}.$ (2.14)

Hence, by inserting (2.11) and (2.14) in (2.10),

$\begin{split}\int_{M}{\rm Ric}(\nabla f,\nabla f)&|\nabla f|^{2p-4}dM\\\
&\leq\left(\dfrac{n(p-1)^{2}-1}{n(p-1)^{2}}\right)\int_{M}\left(\Delta_{p}f\right)^{2}dM+\int_{\partial
M}|\nabla f|^{2p-4}Q(u,z)d\sigma\\\ &\quad-p(p-2)\int_{M}|\nabla
f|^{2p-4}(\Delta_{\infty}f)^{2}dM\\\
&\leq\left(\dfrac{n(p-1)^{2}-1}{n(p-1)^{2}}\right)\int_{M}\left(\Delta_{p}f\right)^{2}dM+\int_{\partial
M}|\nabla f|^{2p-4}Q(u,z)d\sigma,\end{split}$ (2.15)

with equality occurring if and only if either $p=2$ or $\Delta_{\infty}f=0$.

On the other hand, by combining (1.2) with divergence’s theorem and Hölder’s
inequality, we get

$\begin{split}\lambda\int_{M}|f|^{2p-2}dM&=(p-1)\int_{M}|f|^{p-2}|\nabla
f|^{p}dM-\int_{\partial M}|f|^{p-2}|\nabla f|^{p-2}uzd\sigma\\\
&\leq(p-1)\left(\int_{M}|\nabla
f|^{2p-2}\right)^{\frac{p}{2p-2}}\left(\int_{M}|f|^{2p-2}\right)^{\frac{p-2}{2p-2}}-\int_{\partial
M}|f|^{p-2}|\nabla f|^{p-2}uzd\sigma.\end{split}$ (2.16)

By this, a direct computation gives

$\int_{M}|f|^{2p-2}dM\leq\left(\frac{p-1}{\lambda}\right)^{\frac{2p-2}{p}}\mathcal{R}(f,z,u)^{\frac{2p-2}{p}},$
(2.17)

where $\mathcal{R}(f,z,u)$ is defined in (2.8). Since
$(\Delta_{p}f)^{2}=\lambda^{2}|f|^{2p-2}$, from (2.17)

$\int_{M}(\Delta_{p}f)^{2}dM=\lambda^{2}\int_{M}|f|^{2p-2}dM\leq\lambda^{\frac{2}{p}}(p-1)^{\frac{2p-2}{p}}\mathcal{R}(f,z,u)^{\frac{2p-2}{p}}.$
(2.18)

Therefore, by inserting (2.18) in (2.16), we obtain

$\int_{M}{\rm Ric}(\nabla f,\nabla f)|\nabla
f|^{2p-4}dM\leq\lambda^{\frac{2}{p}}c(n,p)\mathcal{R}(f,z,u)^{\frac{2p-2}{p}}+\int_{\partial
M}|\nabla f|^{2p-4}Q(u,z)d\sigma,$ (2.19)

where

$c(n,p)=\dfrac{(n-1)}{n}b(n,p).$ (2.20)

If the equality holds in (2.19), then all the inequalities along the proof
become equalities. In particular, the equality holds in (2.15) and (2.13)
which implies that $p=2$ or ${\rm Hess}\,f(\nabla f)=0$. Thus, from (1.2) and
(2.12), if $\mathrm{Hess}\,f(\nabla f)=0$,

$0=(p-2)|\nabla f|^{p-4}\langle{\rm Hess}\,f(\nabla f),\nabla f\rangle\nabla
f+|\nabla f|^{p-2}{\rm Hess}\,f(\nabla
f)=-\dfrac{\lambda}{n}|f|^{p-2}fI(\nabla f).$

Consequently, since $f\neq 0$ and $\nabla f\neq 0$ we obtain that $\lambda=0$,
a contradiction. Therefore, if the equality in (2.19) holds, $p$ must be $2$.

∎

##  3\. Proof of Theorem 1.2

From Lemma 2.1, we have that the Ricci curvature of $M^{n}$ satisfies:

${\rm Ric}\geq
n-1+\dfrac{n-1}{n}\left(nH^{2}-\frac{n-2}{\sqrt{n-1}}\sqrt{n}H\sqrt{S-nH^{2}}-(S-nH^{2})\right).$
(3.1)

In order to estimate (3.1) from below, let us the following quadratic form
with eigenvalues $\pm\dfrac{n}{2\sqrt{n-1}}$:

$F(x,y)=x^{2}-\frac{n-2}{\sqrt{n-1}}xy-y^{2}.$ (3.2)

We note that the orthogonal transformation

$\left\\{\begin{array}[]{ccc}w&=\frac{1}{2n}\left[(1+\sqrt{n-1})x+(1-\sqrt{n-1})y\right]\\\
v&=\frac{1}{2n}\left[(\sqrt{n-1}-1)x+(1+\sqrt{n-1})y\right]\end{array}\right.$
(3.3)

is such that $x^{2}+y^{2}=w^{2}+v^{2}$. Thus, by taking $x=\sqrt{n}H$ and
$y={\sqrt{S-nH^{2}}}$ we have $x^{2}+y^{2}=S$. Hence,

$\begin{split}F(x,y)&=x^{2}-\frac{n-2}{\sqrt{n-1}}xy-y^{2}=\frac{n}{2\sqrt{n-1}}(w^{2}-v^{2})\\\
&\geq-\frac{n}{2\sqrt{n-1}}(w^{2}+v^{2})\\\
&=-\frac{n}{2\sqrt{n-1}}S,\end{split}$ (3.4)

with equality holding if and only if $v=0$. So, by inserting (3.4) in (3.1),

${\rm Ric}\geq\left(n-1-\dfrac{\sqrt{n-1}}{2}S\right).$ (3.5)

By replacing (3.5) in Lemma 2.2,

$\int_{M}\left(n-1-\dfrac{\sqrt{n-1}}{2}S\right)|\nabla
f|^{2p-2}dM\leq\lambda^{\frac{2}{p}}c(n,p)\mathcal{R}(f,z,u)^{\frac{2p-2}{p}}+\int_{\partial
M}|\nabla f|^{2p-4}\mathcal{Q}(u,z)d\sigma,$ (3.6)

where $\mathcal{R}(f,z,u)$ and $\mathcal{Q}(u,z)$ are defined in (2.8) and
(2.9), respectively.

On the other hand, if the boundary $\partial M$ is empty or satisfies the
Dirichlet or Neumann boundary condition,

$\mathcal{R}(f,z,u)=\left(\int_{M}|\nabla
f|^{2p-2}dM\right)^{\frac{p}{2p-2}}\quad\mbox{and}\quad\int_{\partial
M}|\nabla f|^{2p-4}\mathcal{Q}(u,z)d\sigma\leq 0,$ (3.7)

and thus (2.16) becomes

$\int_{M}\left(n-1-\dfrac{\sqrt{n-1}}{2}S\right)|\nabla
f|^{2p-2}dM\leq\lambda^{\frac{2}{p}}c(n,p)\int_{M}|\nabla f|^{2p-2}dM.$ (3.8)

Therefore

$\int_{M}\left(\dfrac{(n-1)(n-b(n,p)\lambda^{\frac{2}{p}})}{n}-\dfrac{\sqrt{n-1}}{2}S\right)|\nabla
f|^{2p-2}dM\leq 0,$ (3.9)

where $b(n,p)$ is given by (1.7). This proves the inequality in (1.6).

Moreover, the equality

$\int_{M}\left(\dfrac{(n-1)(n-b(n,p)\lambda^{\frac{2}{p}})}{n}-\dfrac{\sqrt{n-1}}{2}S\right)|\nabla
f|^{2p-2}dM=0$ (3.10)

holds, then by Lemma 2.2 we must have $p=2$. In particular, $b(n,2)=1$ and
(3.10) turns in

$\int_{M}\left(\dfrac{(n-1)(n-\lambda)}{n}-\dfrac{\sqrt{n-1}}{2}S\right)|\nabla
f|^{2}dM=0.$ (3.11)

Beyond that, from (2.12) we have

$\mathrm{Hess}\,f=-\dfrac{\lambda}{n}fI.$ (3.12)

Hence, if $\partial M=\emptyset$, by applying Obata’s theorem [9, Theorem A],
it follows that $M^{n}$ is isometric to the sphere
$\mathbb{S}^{n}(\lambda/n)$. In the case where the boundary $\partial M$ is
nonempty and convex, we can apply [2, Theorem 4.3] in order to obtain that
$M^{n}$ is isometric a hemisphere $\mathbb{S}^{n}_{+}(\lambda/n)$. Since the
equality (3.10) imply in the equality (3.4), we get $v=0$. Thence,

$nH^{2}=\left(\frac{\sqrt{n-1}-1}{\sqrt{n-1}+1}\right)^{2}(S-nH^{2}).$ (3.13)

First we observe that if $n=2$, then $H=0$ and $M^{n}$ is a minimal
submanifold of $\mathbb{S}^{N}$. Hence, by using a similar argument to made in
[5, Theorem 3] we conclude that $N=2q$ and $M^{n}$ is isometric to
$\mathbb{S}^{2}\left(\sqrt{q(q+1)/2}\right)$.

From now on, we will assume that $n\neq 2$. In this case, (3.13) becomes

$S=\dfrac{2n^{2}}{(\sqrt{n-1}-1)^{2}}H^{2}.$ (3.14)

Being $M^{n}$ is isometric to $\mathbb{S}^{n}\left(\lambda/n\right)$
(similarly for $\mathbb{S}^{n}_{+}(\lambda/n)$), by putting (3.14) in (2.3),
we have

$\lambda=n+\frac{n^{2}}{n-1}\left(1-\frac{2}{(\sqrt{n-1}-1)^{2}}\right)H^{2}.$
(3.15)

Since $H$ is constant, from identity (3.13) we must have that $S$ is constant.
So, (3.15) can be rewrite as follows

$\lambda-n=-\dfrac{n}{2\sqrt{n-1}}S.$ (3.16)

Hence, by inserting (3.14) and (3.16) in (3.15) we get

$\frac{n^{2}}{n-1}\left(1-\frac{2}{(\sqrt{n-1}-1)^{2}}\right)H^{2}=-\frac{n}{2\sqrt{n-1}}\frac{2n^{2}}{(\sqrt{n-1}-1)^{2}}H^{2}.$
(3.17)

Thus,

$\left(\dfrac{1}{n-1}\left((\sqrt{n-1}-1)^{2}-{2}\right)+\frac{n}{\sqrt{n-1}}\right)H^{2}=(n-2)(\sqrt{n-1}+1)H^{2}=0.$
(3.18)

Once that $n\neq 2$, it follows $H=0$ and hence, $S=0$, from (3.14).
Therefore, by returning to (3.16), we conclude that $n=\lambda$ and
consequently $M^{n}$ is isometric to $\mathbb{S}^{n}(1)$ if $\partial
M=\emptyset$ and isometric to $\mathbb{S}^{n}_{+}(1)$ otherwise.

To ends this proof, we will assume that $\partial M$ is nonempty, convex and
satisfy the Dirichlet boundary condition. Since $v=0$, from (3.1)

${\rm Ric}\geq n-1+\dfrac{(n-1)(\lambda-n)}{n}=\dfrac{(n-1)\lambda}{n}>0.$
(3.19)

Being $\partial M$ convex, follows that $\mathcal{H}$ is nonpositive. Hence,
we can apply the classical result [10, Theorem 4] in order to obtain that
$M^{n}$ is isometric to $\mathbb{S}^{n}_{+}(\lambda/n)$. By thinking as
before, we conclude that $M^{n}$ is isometric to $\mathbb{S}^{n}_{+}(1)$.

## Acknowledgment

The first author is partially supported by CNPq, Brazil, grant 311124/2021-6
and Propesqi (UFPE). The second author is partially supported by CNPq, Brazil.

## References

  * [1] F.R. dos Santos and M.N. Soares, Lower bounds for the length of the second fundamental form via the first eigenvalue of the p-Laplacian, Nonlinear Analysis 232 (2023), pp. 113251.
  * [2] J. Escobar, Uniqueness theorems on conformal deformation of metrics, Sobolev inequalities, and an eigenvalue estimate, Communications On Pure And Applied Mathematics 43 (1990), 857–883.
  * [3] A. Lê, Eigenvalue problems for the p-Laplacian, Nonlinear Analysis: Theory, Methods and Applications 64, (2006), 1057–1099.
  * [4] P.F. Leung, An estimate on the Ricci curvature of a submanifold and some applications, Proc. Amer. Math. Soc. 114 (1992), 1051–1061.
  * [5] P.F. Leung, Minimal submanifolds in a sphere, Math. Z. 183 (1983), 75–86.
  * [6] P. Lindqvist, Notes on the Infinity-Laplace equation, Norwegian University of Science and Technology, 2015.
  * [7] J. Liu. and Q. Zhang, Simons-type inequalities for the compact submanifolds in the space of constant curvature, Kodai Mathematical Journal 30 (2007), 344–351.
  * [8] A. Matei, First eigenvalue for the $p$-Laplace operator, Nonlinear. Anal. Theory, Methods and Appl. 39 (2000), 1051–1068.
  * [9] M. Obata, Certain conditions for a Riemannian manifold to be isometric with a sphere, J. Math. Soc. Japan 14 (1962), 333–340.
  * [10] R. Reilly, Applications of the Hessian operator in a Riemannian manifold, Indiana University Mathematics Journal 26 (1997), 459–472.
  * [11] L. Veron, Some existence and uniqueness results for solution of some quasilinear elliptic equations on compact Riemannian manifolds, Colloquia Mathematica Societatis Janos Bolyai 62 (1991), 317–352.