# Advancing Smart Malnutrition Monitoring: A Multi-Modal Learning Approach for
Vital Health Parameter Estimation

Ashish Marisetty, Prathistith Raj M, Praneeth Nemani, Venkanna Udutalapally,
Debanjan Das

###### Abstract

Malnutrition poses a significant threat to global health, resulting from an
inadequate intake of essential nutrients that adversely impacts vital organs
and overall bodily functioning. Periodic examinations and mass screenings,
incorporating both conventional and non-invasive techniques, have been
employed to combat this challenge. However, these approaches suffer from
critical limitations, such as the need for additional equipment, lack of
comprehensive feature representation, absence of suitable health indicators,
and the unavailability of smartphone implementations for precise estimations
of Body Fat Percentage (BFP), Basal Metabolic Rate (BMR), and Body Mass Index
(BMI) to enable efficient smart-malnutrition monitoring. To address these
constraints, this study presents a groundbreaking, scalable, and robust smart
malnutrition-monitoring system that leverages a single full-body image of an
individual to estimate height, weight, and other crucial health parameters
within a multi-modal learning framework. Our proposed methodology involves the
reconstruction of a highly precise 3D point cloud, from which 512-dimensional
feature embeddings are extracted using a headless-3D classification network.
Concurrently, facial and body embeddings are also extracted, and through the
application of learnable parameters, these features are then utilized to
estimate weight accurately. Furthermore, essential health metrics, including
BMR, BFP, and BMI, are computed to conduct a comprehensive analysis of the
subject’s health, subsequently facilitating the provision of personalized
nutrition plans. While being robust to a wide range of lighting conditions
across multiple devices, our model achieves a low Mean Absolute Error (MAE) of
$\pm$ 4.7 cm and $\pm$ 5.3 kg in estimating height and weight.

###### Keywords:

Multi-modal Learning, 3D Reconstruction, Feature Fusion, Height and Weight
estimation, Smart Healthcare, Non-invasive.

##  I Introduction

Malnutrition is an ailment caused by consuming food that lacks an adequate
quantity of essential nutrients. It is most commonly used in reference to
undernutrition, [1] which occurs when a person does not receive sufficient
calories, proteins, or micronutrients. A scarcity of a quality diet most
commonly causes undernourishment or undernutrition. According to a WHO survey,
there are 178 million malnourished children globally, with 20 million
suffering from severe malnutrition, contributing to 3.5 to 5 million deaths in
children under five each year. On a global scale, undernutrition is
responsible for 45% of all casualties in children under five and is widespread
in developing nations, especially among women and children. Malnutrition also
poses a range of severe health problems that include anemia, diarrhea,
disorientation, weight loss, night blindness, anxiety, attention deficits, and
other neuropsychologic disorders [2]. In the aftermath of the COVID-19
outbreak, which caused significant concerns and stress regarding public health
[3], the traditional approach of measuring height and weight in public health
centers has been impacted. During the pandemic, strict social distancing
measures were put in place to minimize the spread of infection, making the
conventional method of calculating essential health metrics through direct
measurements undesirable.

In addition, pandemics like COVID-19, according to UNICEF, put malnourished
children at an ever-increasing danger of mortality, as well as impaired
growth, development, and learning for those who survive. Therefore, there is a
dire need to identify important health indicators and monitor chronic stress &
uncontrolled or unmonitored food consumption integrated with data-driven
approaches [4]. A primary step in identifying or diagnosing malnutrition and
the nutritional status of any person can be determined by computing their Body
Fat Percentage (BFP), Basal Metabolic Rate (BMR), and Body Mass Index (BMI)
and comparing it with standardized charts. It is more accurate to infer the
risk of malnutrition and various medical conditions from these metrics since
they represent the human body’s functionality in a well-oriented manner. In
this work, we intend to predict the height, weight and successively calculate
the important health metrics as mentioned above from a sing-shot full-body
image by incorporating a holistic representation of prominent features under
the multi-modal learning paradigm. Fig. 1 illustrates a conceptual overview of
the proposed method.

Figure 1: Conceptual Overview TABLE I: Comparison with existing literature works Existing Technologies |  Height Estimation |  Weight Estimation |  Holistic Feature Representation |  Local 3D Features |  Smartphone Application |  Real-Time Testing |  Other Health Metrics
---|---|---|---|---|---|---|---
Alberink et al. [5] |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$
Abdelkader et al. [6] |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{1,0,0}\times}$
Dey et al. [7] |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$
Dantcheva et al. [8] |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{0,1,0}\checkmark}$
Gunel et al. [9] |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$
Fukun et al. [10] |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$
Lee et al. [11] |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$
Velardo et al. [12] |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$
Nguyen et al. [13] |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$
Jiang et al. [14] |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{1,0,0}\times}$
Jin et al. [15] |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{0,1,0}\checkmark}$
Altinigne et al. [16] |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$
Thapar et al. [17] |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$
Child Growth Monitor [18] |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{1,0,0}\times}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{1,0,0}\times}$
autoNutri |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{0,1,0}\checkmark}$ |  ${\color[rgb]{0,1,0}\checkmark}$

In this paper, we propose a solution based on multi-feature fusion that
includes 3D, facial, body, and metadata features integrated with a smartphone
application prototype to estimate a human’s height, weight, and other health
parameters. The smartphone’s camera serves as a sensor to capture a full-body
image of a human, and the height is estimated by calculating the centimetre
per pixel ratio using image processing techniques. Following that, the
captured image is pre-processed by detecting, cropping, aligning the face and
body, reconstructing & samping a 3D person mesh object, and feature extraction
in a multi-modal framework. To summarize, the key contributions of our work
are:

###  I-A Contributions

  * •

A holistic feature fusion of facial, body & 3D embeddings, including the
correlation between them, optimal feature combination and individual
importance in estimating the weight is insightfully discussed.

  * •

This paper is the first to incorporate the fine-grain local 3D representation
in combination using 3D classification network backbones as feature
extractors.

  * •

To the best of our knowledge, this is the first time an IoMT framework has
been used to develop an autonomous smart application for peripheral devices
without any manual intervention.

  * •

The trained model outperformed state-of-the-art methods for weight estimation
on real-world data using a multi-modal architecture, achieving a 5.3 kg error.

##  II Related Research Overview

With the COVID-19 pandemic behind us and a shift in the global landscape,
including a rise in obesity and undernutrition in many countries, the need for
a simple non-contact height and weight estimation technique remains as
relevant as ever. Ongoing research is actively investigating and developing
such techniques to address the current health challenges. The following
sections discuss the related literature categorized based on the model output
- height, weight, and medium of deployment.

###  II-A Height Prediction

Alberink et al. [5] pointed out that in the field of forensic practice, there
is a recurring demand for height estimations of individuals observed in
surveillance video footage captured by cameras. Multiple approaches exist for
conducting such estimations and to gain insights into the disparities between
actual and measured heights, validation measurements are taken from a group of
test subjects. Based on this analysis, a method was proposed to determine
confidence intervals for the height of individuals depicted in images,
accounting for factors such as head and footwear. The aim was to provide a
reliable framework for estimating the height of questioned individuals
captured in surveillance images while considering both systematic and random
sources of variation. Later, Abdelkader et al. [6] employed an equation that
predicts height based on explicitly labeled keypoint coordinates in the image.
Dey et al. [7] assessed the height differences of individuals in every picture
and generated a height disparity graph from a photo compilation to estimate
height. Several of the earliest works estimated height and weight using
metrics such as physique and bone length alongside face and body images. Then
with the rise of deep learning, Dantcheva et al. [8] first proposed a 50-layer
ResNet architecture, achieving an 8.2 cm and 8.51 kg MAE for height and weight
prediction, respectively, using only face images. Gunel et al. [9] later tried
improving the architecture using face, body, and gender information for
predicting height in unconstrained settings. In addition to these inputs,
techniques involving depth information were developed, such as the work by
Fuken et al. [10], where a four-stage architecture performs segmentation of
the human body into explicit segments, predicts the height of the segments
using three CNNs with an error of 0.9% , and the research by Lee et al. [11],
which devised a height estimation method using both color and depth
information with the help of Mask R- CNN’s, achieving a 2.2% error rate.

###  II-B Weight Prediction

One of the initial works for weight estimation used anthropometric features as
proposed by Velardo et al. [12]. By employing multiple regression analysis,
the authors aimed to establish a model that can effectively estimate weight
using various anthropometric features. They relied on a comprehensive medical
database to train the model, ensuring that it captures a wide range of
anthropometric variations and provides accurate weight predictions. The weight
assessor proposed by Nguyen et al. [13] made use of the abundant information
available in RGB-D images to improve estimation accuracy. The method takes
into account visual color signals, depth information, and gender to estimate
multiple weight-related dimensions. This integrated strategy offered an
extensive framework for predicting mass from a single RGB-D image. Influenced
by recent developments in health science research, Jiang et al. [14]
investigated the viability of analyzing body weight using 2D frontal view
human body images with BMI as the metric for measuring body weight. The
intention of the study was to examine this analysis at differing levels of
difficulty by investigating three feasibility problems ranging from simple to
complex. To facilitate the analysis of body weight from human body images, the
researchers developed a system that involved computing five anthropometric
features, which have been recommended as viable indices for determining body
weight. A visual-body-to-BMI dataset has been acquired and systematically
cleansed to support the research study.

As mentioned previously, Dantcheva et al. [8] investigated the viability of
estimating measurements of height, weight, and BMI from single-shot
photographs of the face. The authors proposed a regression method based on the
50-layer ResNet architecture to accomplish this goal. This method utilized the
exclusive properties of facial images to precisely estimate the aforementioned
characteristics. In addition, a new dataset containing 1026 subjects has been
included in this study. In a recent study, Jin et al. [15] noted that BMI is
frequently employed as a measurement of weight and health conditions and that
previous research in this field has focused primarily on using numerous 2D
images, 3D images, or images of the face. However, these indicators are not
always accessible and the authors proposed a dual-branch regression approach
to estimate weight and BMI from a single 2D body image to circumvent this
limitation. The researchers intend to improve the accuracy of BMI estimation
from a single 2D body image by integrating information from the anthropometric
feature computation branch and the deep learning-based feature extraction
branch. In addition, few methods attempted to estimate both height and weight
simultaneously, such as Altinigne et al. [16], who developed a deep learning
method that employs the estimation of individual silhouette and skeleton
joints as effective regularizers.

###  II-C Malnutrition and IoT Solutions

Many previous works have focused on developing a solution for malnutrition,
such as the expert system by Thapar et al. [17], which analyses malnutrition
using a Mamdani inference method with 13 different categorical input
variables, but it is only recently that work has begun to make them accessible
and deployable. One such IoT-based solution is Child Growth Monitor [18], an
AI-based application that relies on the availability of infrared sensors in
selected smartphones to capture 3D measurements of a child’s height, body
volume, and weight ratio. However, even these techniques fell short of
providing a complete solution involving height, weight estimation, all wrapped
up in an application that could be used by anyone with a smartphone. Our work
overcomes all of the aforementioned drawbacks while also improving weight
estimation performance through the use of local 3D features, multimodal
embedding fusion, and an edge device prototype for computation. Table I
depicts an overview of all the discussed existing solutions.

##  III Methodology

This section describes the proposed three-phase height and weight estimation
workflow, as shown in Fig. 2. Phase 1 deals with image pre-processing and
height estimation while phase 2 emphasizes feature extraction, multi-modal
fusion, and regression. Subsequently, the final phase depicts the integration
of the above system with an edge device application prototype in an IoT
framework.

Figure 2: Proposed System Overview

###  III-A Phase 1: Pre-processing and Height Prediction

In this phase, we pre-process the input image of a person, reconstruct the 3D
volumetric information and perform height prediction. The mentioned phase is
divided into four sub-phases: Facial landmark detection and alignment, Body
key points detection, 3D reconstruction and Height prediction.

####  III-A1 Facial Landmarks Detection and Alignment

To extract the face crop from full body image we perform face verification,
cropping and subsequently alignment. The initial step of face detection
determines the position of a face, by traversing through the points around the
facial region to locate 68 landmarks. Subsequently, the faces are aligned and
transformed such that facial landmarks (inner eyes and bottom lip) appear in
approximately in same regions, preserving the collinearity, parallelism, and
the ratio of distances between the points with Affine Transformation. Fig. 3
(a) visualizes an example of the localization of face from the input image,
Fig. 3 (b) depicts the facial landmarks while Fig. 3 (c) illustrates the
facial alignment and region cropping. After completing the facial alignment
step, the subsequent stage in the preprocessing pipeline involves the
detection of body key points.

Figure 3: Our face verification and pre-processing pipeline: (a) Face
Detection, (b) Facial Landmark Detection, (c) Face Alignment & Cropping

####  III-A2 Body Keypoints Detection

Considering the inherent unpredictability of real-world scenarios, it is
imperative to establish the elimination of unwanted noise. Following the
extraction of the human body region from varying backgrounds through the
application of a U-Net trained for human segmentation, the subsequent stage
involves the detection of human body landmarks within the input image. This
process commences with the initial layers of the VGG-19 network extracting
pertinent image features, which are then passed into two parallel branches of
convolutional layers. The first branch predicts a group of 18 confidence maps,
each representing a different portion of the human posture skeleton. The
second branch predicts a group of 38 Part Affinity Fields (PAFs) [19], which
indicates the degree of affinity between parts. Let set S =
$(S_{1},S_{2},....,S_{J})$ denote the confidence maps for $j$ i.e., detected
body parts. Then the individual confidence maps for each person $k$ can be
formulated as $S^{*}_{j,k}$ at a location $p$ is denoted in the following Eq.
1, where $x_{j,k}$ be the ground truth position of body part $j$ for person
$k$ in the image and $\sigma$ controls the spread of the peak.

$S^{*}_{j,k}(p)=exp(-\frac{|p-x_{j,k}|^{2}_{2}}{\sigma^{2}})$ (1)

####  III-A3 3D Reconstruction

The loss of 3D information during the process of capturing pictures poses a
significant challenge in accurately inferring and extracting 3D
characteristics from 2D visuals. To tackle the aforementioned challenge, we
adopt a multi-level architecture PiFuHD [20] which is trained end-to-end on
high-resolution images. This model is profound in reconstructing 3D mesh,
preserving intricate 3D details solely from a single human image. The
objective of the algorithm is to model a function, $f(X)$, such that for any
given 3D position in continuous space $X=(X_{x},X_{y},X_{z})\in R^{3}$, it
predicts the occupancy value as shown in Eq. 2.

$f(X,I)=\begin{cases}1,\textit{if X is inside the mesh surface}\\\
0,\textit{otherwise}\end{cases}$ (2)

For an orthogonal projected 2D point given by $\pi(X)=x=(X_{x},X_{y})$, an
image feature embedding is extracted by function $f$. Then the occupancy of
the query 3D point X is estimated by Eq. 3 where Z = $X_{z}$ is the depth
along the ray defined by the 2D projection $x$.

$f(X,I)=g(\phi(X,I),Z)$ (3)

Figure 4: Our body detection and prepossessing pipeline: (a) Body Keypoint
estimation, (b) Masking, (c) 3D Human Mesh Reconstruction, (d) Conversion to
3D Point-Cloud

Finally, we employ mesh sampling to generate a point cloud representation of
the mesh, which provides a straightforward yet efficient means of representing
3D data. The detected body key points are illustrated in Fig. 4 (a), Fig. 4
(b) depicts the result of masking the input image, Fig. 4 (c) shows the 3D
Mesh Reconstruction and Fig. 4 (d) illustrates its conversion to 3D Point-
cloud.

Figure 5: Overview of our proposed multi-modal computational architecture. The
feature fusion obtains the unimodal representations ${z_{F}}$, ${z_{B}}$,
${z_{R}}$ by passing the inputs ${X_{F}}$, ${X_{B}}$, ${X_{R}}$ into the sub-
embedding networks parametrized by ${\theta_{F}}$, ${\theta_{B}}$,
${\theta_{R}}$ respectively. The representations are then weighed by learned
weights ${w_{F}}$, ${w_{B}}$, ${w_{R}}$ and concatenated with gender and
height information to predict the weight and subsequently calculate BMI, BMR,
and BFP.

####  III-A4 Height Estimation

The final step of this phase is height prediction, and taking previous work
results into account, we decided to use a simple yet efficient computer vision
technique that works best for input images that are parallel to the subject,
similar to our dataset images. The simple pixel arithmetic method relies on
the person’s scale and camera orientation to calculate the person’s height. To
begin, we undistort the image to remove radial and tangential distortions and
make the image independent of the device used to capture it. Then, we
calculate the pixel per metric (ppm) attribute on the tight-crop masked image
($I_{c}$) from previous sub-phases using Eq. 4. This metric is then re-used
throughout the process to predict the height of a new person ($I_{pred}$)
given a static camera position by Eq. 5.

$ppm=\frac{I_{c}.size[0]}{I_{c}\;height}$ (4)

$height_{pred}=\frac{I_{pred}.size[0]}{ppm}$ (5)

###  III-B Phase 2: Unimodal representation and fusion

The preprocessed data extracted from the previous phase is passed to this
phase for feature extraction. This phase can be further divided into three
sub-phases: 3D-feature extraction, 2D-feature extraction, multi-modal fusion
and regression. The overview of the computational architecture is represented
in Fig. 5

####  III-B1 3D feature extraction

The point cloud obtained after the previous phase’s pre-processing is used as
an input to extract the 3D embedding representation. The 3D point classifiers
are the best at classifying the point cloud based on the local granular shape
and the overall global shape, making them the ideal feature extractors for our
problem. As a result, we use the PointNet [21] classifier to compile
depictions because of its capacity to deal with unordered input points by
employing a symmetric function (max pooling) to learn a set of optimization
functions/criteria that select informative areas in the point cloud and
represent the explanation for their inheritance. The final fully connected
layers of the network consolidate these optimally learned values into the
global descriptor for the entire shape, resulting in the 512-dimensional
feature vector. Since each point undergoes its own transformation, our input
format makes it simple to implement unchanging or affine modifications.

####  III-B2 2D feature extraction

The 3D embedding features have been computed in the previous step. Now we take
a similar approach to calculate the 2D feature representation. First, the
preprocessed face image is passed through a VGGFace architecture [22] without
a head to extract a 512-dimension vector. Parallelly, we also pass the body
image through an Xception architecture [23] without a head, using it as a
feature extractor to get a 512-dimension body representation. Here, the VGG-16
has 16 trainable convolutional layers followed by a max-pooling operation
whereas Xception is a deep convolutional neural network architecture with
Depthwise Separable Convolutions. Finally, we employ Transfer Learning
techniques with these trained VGGFace and Xception model pre-trained weights
to extract 2D facial and deep body features from preprocessed face and full-
body images, respectively. This forms the basis for the subsequent step of
multi-modal feature fusion and regression.

####  III-B3 Multi-modal fusion and regression

Now as all the unimodal features are extracted we fuse the different sub-
embedding streams of 512- dimensional feature representations. These
representations comprises of two different modalities - point cloud ($z_{R}$)
and image data ($z_{F}$, $z_{B}$) and hence cannot be fused with a simple
concatenation. Instead, we use learnable weights ($w_{F}$, $w_{B}$, $w_{R}$)
to weigh these features and add them all up to get a final 515-dimensional
feature vector (line 2, Algorithm 1). This feature vector is then passed
through two 512-units Multi Layer Perceptron (_$g^{[0]}$_ , _$g^{[1]}$_),
followed by 256 units MLP (_$g^{[2]}$_) and finally through a single unit
linear layer (_$g^{[3]}$_) to predict the weight of the person (line 3-5,
Algorithm 1). The final layer uses Ridge regression to penalize the layer to
not overfit the distribution but to generalize to new plausible test data
samples. Then we compute the person’s Body Mass Index (BMI), followed by Body
Metabolic Rate (BMR) using Mifflin-St Jeor Equation [24] and Body Fat
Percentage (BFP) using BMI for suggesting appropriate nutrition plan and
malnutrition monitoring. In Algorithm 1 (lines 8-9), _p_ and _m_ are intercept
constants that vary with gender, with values of 5 and 16.2 for men and 161 and
5.4 for women, respectively.

Algorithm 1 : _Multimodal fusion and Regression_

0: Input, $z_{F}$, $z_{B}$, $z_{R}$, _gender_ , $height_{pred}$

0: $weight_{pred}$, _BMI_ , _BMR_ , _BFP_

1: $E({r_{F}},{r_{B}},{r_{R}})$ = $\Sigma_{j\in(F,B,R)}\;w_{j}\times
z_{j}(X_{j};\;\theta_{j})$

2: $h^{[-1]}$(F, a, g) = concatenate(_E, gender_ , $height_{pred}$)

3: for i in [0, 1, 2, 3] do

4: $h^{[i]}$ = $g^{[i]}$($W^{[i]}\times h^{[i-1]}$ \+ $b^{[i]}$)

5: $weight_{pred}$ = $h^{[i]}$

6: end for

7: BMI = $\cfrac{weight_{pred}[kg]}{{height_{pred}}^{2}[m^{2}]}$ =
$\cfrac{weight_{pred}[lb]\times 703}{{height_{pred}}^{2}[in^{2}]}$

8: BMR = $10\times weight_{pred}$ \+ $6.25\times height_{pred}$ \- $5\times
age+p$

9: BFP = $1.2\times BMI+0.23\times age-m$

TABLE II: Statistical information of the Datasets Participant Information | Gender | Height (in cm) | Weight (in kg)
---|---|---|---
Dataset | Total |  Male | Female | Range |  Mean |  Standard Deviation |  95$\%$ Confidence Interval |  Range |  Mean |  Standard Deviation |  95$\%$ Confidence Interval
Visual-body-to-BMI | 5900 |  3968 | 1932 | 213.36 - 147.32 |  175.54 |  9.89 |  176.99 - 174.09 |  254.01 - 44.90 |  95.05 |  27.12 |  100.9 - 89.1
Locally Collected Data | 287 |  261 | 26 | 184-101 |  164.09 |  21.35 |  167.23 - 160.95 |  100 - 13 |  63.51 |  21.41 |  68.26 - 58.76

###  III-C Phase 3: Android Application Prototype

Following training the model, the model’s learned weights are saved using
Pytorch’s.save() function and converted to an _.pb_ file using ONNX as the
intermediate format [25]. Then, we use TensorFlow Serving to deploy and serve
the trained model as an _.apk_ file integrated with the created Android
interface. The Android interface is intended to be simple and efficient for
people from all walks of life and social strata. The workflow of the proposed
system’s GUI is depicted in Fig. 6.

##  IV Experimental Study

This section describes the dataset used for training and testing our model,
ablation studies, and experiments on the proposed multi-modal system under
various scenarios. Following that, we will go over the performance of cloud-
based IoT application as well as the computational platform used.

Figure 6: Workflow of the proposed IoT Application prototype TABLE III: Performance comparison of different architecture combinations for weight prediction 3D Features | PointNet | DG-CNN | GB-Net
---|---|---|---
Face FE |  Body FE |  MAE |  RMSE |  $R^{2}$ |  MAE |  RMSE |  $R^{2}$ |  MAE |  RMSE |  $R^{2}$
VGGFace |  Xception |  5.309 |  7.438 |  0.720 |  7.763 |  9.352 |  0.572 |  6.421 |  8.396 |  0.639
ResNet152 |  5.612 |  7.635 |  0.697 |  7.894 |  9.650 |  0.560 |  6.989 |  8.903 |  0.596
FaceNet |  Xception |  5.978 |  7.998 |  0.661 |  8.363 |  10.016 |  0.559 |  6.640 |  8.511 |  0.615
ResNet152 |  6.112 |  8.131 |  0.651 |  8.606 |  10.400 |  0.548 |  7.200 |  9.155 |  0.587

###  IV-A Datasets Used

In our work, we majorly used two datasets - visual-body-to-BMI dataset [14],
locally collected dataset. As mentioned earlier, the visual-body-to-BMI
dataset consists of 47574 images of 16483 people scraped and downloaded from
the progresspics subreddit website. These images are then annotated and
filtered resulting in a total of 5900 images, with two images for each of the
2950 subjects. The 2950 subjects comprises of 966 females and 1984 males, as
well as the corresponding gender and weight labels. On the other hand, we
locally collected a dataset of 30 people in 9 - 10 frontal poses, as well as
height and device information. Table II highlights the statistical information
about these two datasets. These two datasets are then combined to jointly
train the model but is bench-marked only on the visual-body-to-BMI to enable
comparison with the previous works. Meanwhile, we have held-out a sample size
of 30 from the locally collected dataset, with each image being one of the 30
subjects in a randomly sampled pose, for experiments across devices and
lighting conditions in Section IV D, IV G respectively.

###  IV-B Steps followed to capture input images

The following steps are followed while capturing a full-body image of a person
to estimate height and weight:

  * •

An RGB image of a frontal pose of person standing at a distance of 1.5 meters
from the camera lens placed 1 meter from the ground is captured under
sufficient lighting conditions as depicted in Fig. 10 (a).

  * •

The smartphone lens was parallel to the person, i.e., 90-degree angle w.r.t
the person, and perpendicular to the ground, to accurately calculate the per-
pixel metric for height estimation.

  * •

The captured image is further masked & pre-processed to remove the redundant
background thereby extracting the facial, body, and 3D representations under
pre-processing & feature extraction pipelines.

###  IV-C Performance of multiple model architecture combinations

To come up with the current architecture, we systemically explored the
combinations of various facial feature extractors like VGGFace and FaceNet
[26], body feature extractors like Xception and ResNet-152 in combination with
3D feature extractors like PointNet [27], DG-CNN and GB-Net as summarized in
Table III. The best architecture observed is a combination of Xception, VGG-
Face and PointNet for the body, face, and 3D feature extraction, achieving a
MAE weight of 5.3 kg. We also noticed that VGG-Face outperforms FaceNet in
general, while Xception outperforms ResNet-512. PointNet, on the other hand,
outranks its corresponding point-cloud classifiers with its ability to extract
rich 3D representations.

###  IV-D Effect of Lighting Conditions on height and weight prediction and
Device Comparison

The collected dataset contains images in unconstrained lighting conditions and
is a perfect representation of real-world lighting conditions. To illustrate
this and test the model performance further, we have artificially simulated
the image brightness using gamma correction. The model performs best in
$\gamma$ range of 1.0 to 1.25 as shown in Fig. 7 (b). We can also deduce that
the MAE decreases when $\gamma$ is in the range of 0 - 1.25, attains its
minimum MAE at $\gamma$ = 1.0, and increases as $\gamma$ increases. The above
variation in extreme cases can be attributed primarily to the poor performance
of 3D reconstruction in extreme lighting conditions, where reconstruction
quality decreases considerably when image global lighting drastically
increases or decreases, despite performing well for a wide range of natural
illumination. For performance comparison on different devices we used a hold-
out set which contains images collected from a variety of devices, including
laptops and multiple smartphone brands. Figure 7 (a) shows the predicted
weight versus the actual weight, demonstrating that our model’s performance is
robust and coherent across all types of devices.

Figure 7: (a) Performance under simulated lighting conditions. Our system
remains robust to wide range of illumination but it is preferable to have
sufficient lighting to decrease the error. (b) Performance of our system
across various devices

###  IV-E Importance of Multiple Features

Our best-performing model works on weighing and averaging the multiple input
feature embeddings. The embeddings are weighed such that each embedding is
assigned a weight between 0 and 1, and their sum equals 1. It enables us to
interpret the relative importance of these different embeddings across
multiple architectures in predicting the weight. We have observed that these
weights vary significantly when the 3D feature extractor architecture is
changed, while the best extractors for both facial and body features are kept
constant. From Fig. 8 (b), we can also infer that PointNet allows the model to
have a balanced weight distribution with lower error as compared to the
others. Overall, though the 3D features have relatively low importance, they
perform slightly better in extreme use-cases such as obese and under-nourished
conditions than only-image-based techniques.

Furthermore, in order to find the best pair-wise feature combination, we
systematically tested combinations of various types of features, as shown in
Fig. 8 (a). The abbreviations BF, DF & FF in Fig. 8 (a) represent the body
features (BF), 3D features (DF) & facial features (FF) respectively. This
experiment is carried out by including the best architectures
(PointNet+Xception+VGG-Face) for the respective features. The best pair-wise
combination of DF+FF, with low MAE and high correlation, demonstrates the
importance of 3D and facial features. Despite the lack of BFs, the combined
effect of DFs and FFs can still yield a reasonable result. The presence of
scale free images in the dataset, which does not produce meaningful human
structure anthropometric representations, can be attributed in large part to
the lack of BFs importance. To summarise the previous experiments findings, we
can conclude that facial features are the most important predictor across all
possible architecture combinations, followed mostly by 3D features and body
features.

Figure 8: (a) Performance comparison of pairwise feature combinations in
weight estimation, (b) Feature importance across different 3D network
architectures.

In addition, we illustrated the correlation plots of the individual features
including BFs, DFs & FFs along with their combination on our best found
architecture in Fig. 9 (a), Fig. 9 (b), Fig. 9 (c) and Fig. 9 (d).

Figure 9: Correlation between predicted and measured body weight of 42
randomly selected test-set samples using only: (a) Body Features (BFs), (b) 3D
Features (DFs), (c) Facial Features (FFs) and, (d) Combination of all features
(FFs+DFs+BFs).

###  IV-F Android Application and Deployment

To interface with the proposed model, we designed a versatile and user-
friendly android application. The first page of the Graphical User Interface
(GUI) consists of three sets of inputs: Age, Gender, and Image of the subject
as shown in Fig. 10 (b). Next, the inputs are provided and the model present
in the cloud computes the different output metrics. These computed metrics
include the height, weight, BMI, Ideal weight, active BMR and the BFP of the
person. To achieve the ideal weight, the user is then asked to select the type
of diet and the number of weeks they are willing to dedicate to the program to
attain the desired weight as shown in Fig 10 (c). Once this computation is
performed, the results are displayed and the customized nutrition plan is made
ready to download.

Figure 10: (a) Image acquiring technique, (b) Uploading the picture in
addition to the associated metadata, (c) Illustrating the calculated outcomes
and choosing a diet approach.

###  IV-G Performance Observation on real-time data

We further extensively tested our system for malnutrition classification on
held-out locally collected real-time data from 30 people in various frontal
poses. Based on their true BMI values, 20 of these 30 participants are
healthy, while the remaining 10 are considered malnourished. The model’s
corresponding confusion matrix on this withheld dataset is as shown in Table
IV. As depicted, the model achieves an accuracy of 86.67%, as well as
precision, recall, and F1 score of 80 %, 80 %, and 80 %, respectively.

TABLE IV: Confusion matrix of Malnutrition classification on Testset Predicted Condition | Actual Condition | Accuracy | Precision | Recall
---|---|---|---|---
Healthy | Malnutritious
Healthy | 18 | 2 | 86.67 % | 80.00 % | 80.00 %
Malnutritious | 2 | 8

##  V Key findings and Comparative Analysis

In the proposed solution, several architectures with a combination of
different fusion techniques for weight estimation and a pixel per metric
approach for height estimation have been extensively tested. These findings
are then used to calculate and infer pertinent health indicators from a single
image, ultimately determining if the person is malnourished. The following are
the key findings and comparative analysis of the proposed solution:

###  V-A Key Findings

Fine-scale 3D Representation: Our research presents a pioneering application
of PiFuHD for reconstruction, employing highly precise and detailed local 3D
representation. This approach allows for a fine-grained level of detail in the
reconstructed output. Furthermore, we utilize state-of-the-art 3D
classification networks in our work by removing the last layers to extract a
512-dimensional vector as the 3D feature embedding. This technique enables us
to capture and represent essential information from the input data.

Multi-Modal Learning Paradigm: Many existing solutions in the field often rely
on a single modality or feature representation, such as facial or manually
crafted anthropometric information, or statistical measures, as highlighted in
previous studies [8] [14] [28] [29] [30]. However, in our research, we adopted
a holistic feature representation approach and conducted a systematic
exploration to ascertain the significance of various features. This was
achieved through extensive experimentation and in-depth analysis. Our solution
stands out by achieving state-of-the-art results in weight estimation.
Notably, we achieved the lowest mean absolute error (MAE) of 5.3 kg,
surpassing previous works. This achievement was made possible by employing
learnable weighing parameters in fusion, which enhances the accuracy of our
weight estimation model. Through our research, we provide a comprehensive and
advanced approach to weight estimation, considering multiple features and
their interplay.

Edge Device Deployment: A notable observation in the existing literature is
the absence of deployed solutions or a reliance on sensor infrastructure for
collecting user health data for monitoring, as highlighted in previous studies
[17] [31] [32]. In contrast, our research introduces a novel solution through
the development of a smart application prototype. This prototype enables the
estimation of health parameters such as height and weight and predicts the
risk of malnutrition using a single full-body image. Importantly, this
solution proves particularly valuable in remote locations with limited or no
access to health facilities. One significant advantage of our approach is the
use of an edge device prototype that operates independently, eliminating the
need for additional equipment. This self-sufficiency empowers the prototype to
estimate nutritional status accurately, providing crucial health insights even
in resource-constrained environments.

###  V-B Comparative Analysis

The proposed methodology showcased remarkable performance, achieving an
impressive mean absolute error (MAE) score of 5.3 kg in weight estimation and
4.7 cm in height estimation. A comprehensive evaluation of error rates in
height and weight prediction revealed that our approach outperformed previous
works [8] [14] [16] [18], as highlighted in Table V. Importantly, our designed
multi-modal system operates autonomously, eliminating the need for human
intervention during crucial stages such as detecting body and facial
landmarks, masking, cropping, and alignment. This autonomy enhances the
efficiency and reliability of the system, setting it apart from non-autonomous
and non-invasive techniques.

TABLE V: Mean Absolute Error (MAE) comparison with existing techniques

Method | Height MAE | Weight MAE |
---|---|---|---
Altinigne et al.[16] |  $\pm$ 6.13 cm |  $\pm$ 9.80 kg |
Dantcheva et al.[8] |  $\pm$ 8.2 cm |  $\pm$ 8.51 kg |
Jiang et al. [14] | - | - |
Child Growth Monitor [18] | - | - |
Ours |  $\pm$ 4.7 cm |  $\pm$ 5.3 kg |

##  VI Conclusion and Future Works

This research presents a novel approach for predicting height and weight and
inferring other health indicators, such as BMI, BMR, and BFP, from a single-
shot full-body image. The methodology employs a holistic feature
representation within a multi-modal learning paradigm. The proposed solution
undergoes meticulous validation and testing using real-world images, including
the simulation of various lighting conditions. The study also systematically
examines the significance of 2D and 3D features. To further enhance the
performance of weight and height prediction, future investigations can explore
more rigorous methods for training and converging the multi-modal
architecture. Additionally, efforts can be made to improve the extraction of
FFs (Feature Fusion), DFs (Depth Fusion), and BFs (Body Fusion) embeddings.
Exploring sub-embedding representation fusion methods and designing approaches
to predict height without scale information or constraints could also
contribute to improved prediction accuracy. Furthermore, future app
development endeavors can focus on fostering communities and addressing
security concerns related to the Machine Learning model and databases. These
aspects will contribute to a more comprehensive and impactful implementation
of the solution.

##  VII Acknowledgements

The authors would like to thank Dr. Min Jiang for providing access to the
Visual-body-to-BMI dataset for our research. The authors are also grateful to
the 30 volunteers for the contribution of the required images in creating a
local dataset.

## References

  * [1] S. Khare, D. Gupta, K. Prabhavathi, M. G. Deepika, and A. Jyotishi, “Health and nutritional status of children: Survey, challenges and directions,” in Cognitive Computing and Information Processing (T. Nagabhushan, V. N. M. Aradhya, P. Jagadeesh, S. Shukla, and C. M.L., eds.), (Singapore), pp. 93–104, Springer Singapore, 2018.
  * [2] L. Jiang, B. Qiu, X. Liu, C. Huang, and K. Lin, “Deepfood: Food image analysis and dietary assessment via deep model,” IEEE Access, vol. 8, pp. 47477–47489, 2020.
  * [3] C. d. Vente, L. H. Boulogne, K. V. Venkadesh, C. Sital, N. Lessmann, C. Jacobs, C. I. Sánchez, and B. v. Ginneken, “Automated covid-19 grading with convolutional neural networks in computed tomography scans: A systematic comparison,” IEEE Transactions on Artificial Intelligence, vol. 3, no. 2, pp. 129–138, 2022.
  * [4] S. Latif, M. Usman, S. Manzoor, W. Iqbal, J. Qadir, G. Tyson, I. Castro, A. Razi, M. N. K. Boulos, A. Weller, and J. Crowcroft, “Leveraging data science to combat covid-19: A comprehensive review,” IEEE Transactions on Artificial Intelligence, vol. 1, no. 1, pp. 85–103, 2020.
  * [5] I. Alberink and A. Bolck, “Obtaining confidence intervals and likelihood ratios for body height estimations in images,” Forensic Science International, vol. 177, no. 2-3, pp. 228–237, 2008.
  * [6] C. BenAbdelkader and Y. Yacoob, “Statistical body height estimation from a single image,” in 2008 8th IEEE International Conference on Automatic Face Gesture Recognition, pp. 1–7, 2008.
  * [7] R. Dey, M. Nangia, K. W. Ross, and Y. Liu, “Estimating heights from photo collections: A data-driven approach,” in Proceedings of the Second ACM Conference on Online Social Networks, COSN ’14, (New York, NY, USA), p. 227–238, Association for Computing Machinery, 2014.
  * [8] A. Dantcheva, F. Bremond, and P. Bilinski, “Show me your face and i will tell you your height, weight and body mass index,” in 2018 24th International Conference on Pattern Recognition (ICPR), pp. 3555–3560, 2018\.
  * [9] S. Günel, H. Rhodin, and P. Fua, “What face and body shapes can tell us about height.,” in ICCV Workshops, pp. 1819–1827, 2019.
  * [10] F. Yin and S. Zhou, “Accurate estimation of body height from a single depth image via a four-stage developing network,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8264–8273, 2020.
  * [11] D.-s. Lee, J.-s. Kim, S. C. Jeong, and S.-k. Kwon, “Human height estimation by color deep learning and depth 3d conversion,” Applied Sciences, vol. 10, no. 16, p. 5531, 2020.
  * [12] C. Velardo and J.-L. Dugelay, “Weight estimation from visual body appearance,” in 2010 Fourth IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS), pp. 1–6, 2010.
  * [13] T. V. Nguyen, J. Feng, and S. Yan, “Seeing human weight from a single rgb-d image,” Journal of Computer Science and Technology, vol. 29, no. 5, pp. 777–784, 2014.
  * [14] M. Jiang and G. Guo, “Body weight analysis from human body images,” IEEE Transactions on Information Forensics and Security, vol. 14, no. 10, pp. 2676–2688, 2019.
  * [15] Z. Jin, J. Huang, W. Wang, A. Xiong, and X. Tan, “Estimating human weight from a single image,” IEEE Transactions on Multimedia, 2022.
  * [16] C. Y. Altinigne, D. Thanou, and R. Achanta, “Height and weight estimation from unconstrained images,” in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2298–2302, 2020.
  * [17] A. Thapar and M. Goyal, “A fuzzy expert system for diagnosis of malnutrition in children,” in 2016 IEEE Region 10 Humanitarian Technology Conference (R10-HTC), pp. 1–6, 2016.
  * [18] Microsoft, “Child growth monitor: Using ai to solve world hunger and malnutrition.” https://news.microsoft.com/en-in/features/child-growth-monitor-malnutrition-india-microsoft-ai/.
  * [19] Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y. Sheikh, “Openpose: Realtime multi-person 2d pose estimation using part affinity fields,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 1, pp. 172–186, 2021.
  * [20] S. Saito, T. Simon, J. Saragih, and H. Joo, “Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 2020.
  * [21] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652–660, 2017\.
  * [22] M. Mehdipour Ghazi and H. Kemal Ekenel, “A comprehensive analysis of deep learning based representation for face recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 34–41, 2016.
  * [23] F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251–1258, 2017.
  * [24] M. D. Mifflin, S. T. St Jeor, L. A. Hill, B. J. Scott, S. A. Daugherty, and Y. O. Koh, “A new predictive equation for resting energy expenditure in healthy individuals,” The American Journal of Clinical Nutrition, vol. 51, pp. 241–247, 02 1990.
  * [25] “Faster scalable ml model deployment using onnx and open source tools,” in 2020 IEEE Infrastructure Conference, pp. i–i, 2020.
  * [26] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 815–823, 2015.
  * [27] R. Q. Charles, H. Su, M. Kaichun, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 77–85, 2017\.
  * [28] L. Wen and G. Guo, “A computational approach to body mass index prediction from face images,” Image and Vision Computing, vol. 31, no. 5, pp. 392–400, 2013.
  * [29] M. Jiang, Y. Shang, and G. Guo, “On visual bmi analysis from facial images,” Image and Vision Computing, vol. 89, pp. 183–196, 2019.
  * [30] J. Huang, C. Shang, A. Xiong, Y. Pang, and Z. Jin, “Seeing health with eyes: Feature combination for image-based human bmi estimation,” in 2021 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6, IEEE, 2021\.
  * [31] L. Rachakonda, S. P. Mohanty, and E. Kougianos, “ilog: An intelligent device for automatic food intake monitoring and stress detection in the iomt,” IEEE Transactions on Consumer Electronics, vol. 66, no. 2, pp. 115–124, 2020\.
  * [32] P. Sundaravadivel, K. Kesavan, L. Kesavan, S. P. Mohanty, and E. Kougianos, “Smart-log: A deep-learning based automated nutrition monitoring system in the iot,” IEEE Transactions on Consumer Electronics, vol. 64, no. 3, pp. 390–398, 2018.


Since $T_{\Psi_{S}(0)}^{1,0}\check{D}\cong\mathfrak{q}$, the exponential
mapping

$\exp\colon\mathfrak{q}\to\check{D},\quad A\mapsto e^{A}\cdot\Psi_{S}(0),$

restricts to a biholomorphism between a small neighborhood of
$0\in\mathfrak{q}$ and a small neighborhood of the point
$\Psi_{S}(0)\in\check{D}$. Consequently, there is a constant $\varepsilon>0$,
whose exact value depends on the period mapping under consideration, and a
unique holomorphic mapping

$\tilde{\Gamma}\colon\Delta_{\varepsilon}\to\mathfrak{q}$

such that $\tilde{\Gamma}(0)=0$ and
$\Psi_{S}(t)=e^{\tilde{\Gamma}(t)}\cdot\Psi_{S}(0)$ for $\lvert
t\rvert<\varepsilon$. As long as $\operatorname{Re}z<\log\varepsilon$, this
gives us a (preliminary) formula for the period mapping:

$\Phi(z)=e^{z(S+N)}e^{\tilde{\Gamma}(e^{z})}\cdot\Psi_{S}(0)$ (245.1)

Note that, unlike in [CK, (2.5)], the relevant filtration here is _not_ the
limiting Hodge filtration $F_{\mathrm{lim}}$, but the filtration $\Psi_{S}(0)$
coming from the nilpotent orbit theorem.

####  246 .

Now we would like to derive from (245.1) a formula for $\Phi(z)$ that only
involves the limiting Hodge filtration $F_{\mathrm{lim}}$. The key is the
following lemma.

######  Lemma 38.

One has $\Psi_{S}(0)=e^{A}F_{\mathrm{lim}}$ for a unique element
$A\in\mathfrak{q}$. Decomposing $A=\sum_{\alpha}A_{\alpha}$, with
$A_{\alpha}\in\mathfrak{q}_{\alpha}$, one has $A_{\alpha}=0$ for $\alpha\leq
0$.

###### Proof.

As before, the exponential mapping $A\mapsto e^{A}\cdot F_{\mathrm{lim}}$
gives a biholomorphism between a neighborhood of $0\in\mathfrak{q}$ and a
neighborhood of the point $F_{\mathrm{lim}}\in\check{D}$. Recall that
$F_{\mathrm{lim}}=\lim_{x\to\infty}e^{-xS}\Psi_{S}(0)$. For $x\gg 0$, it
follows that there is an element $B(x)\in\mathfrak{q}$ such that
$e^{-xS}\Psi_{S}(0)=e^{B(x)}F_{\mathrm{lim}}$. Since $S$ preserves the
filtration $F_{\mathrm{lim}}$, we can rewrite this as

$\Psi_{S}(0)=e^{xS}e^{B(x)}e^{-xS}\cdot F_{\mathrm{lim}}=e^{A}\cdot
F_{\mathrm{lim}},$

where $A=e^{x\operatorname{ad}(S)}B(x)\in\mathfrak{q}$ is necessarily
constant. Now we decompose this constant endomorphism as
$A=\sum_{\alpha}A_{\alpha}$, with $A_{\alpha}\in\mathfrak{q}\cap
E_{\alpha}(\operatorname{ad}S)$. Then

$e^{-xS}\Psi_{S}(0)=e^{-xS}e^{A}e^{xS}\cdot
F_{\mathrm{lim}}=\exp\left(\sum_{\alpha}e^{-\alpha x}A_{\alpha}\right)\cdot
F_{\mathrm{lim}}.$

As $x\to\infty$, this can only converge to $F_{\mathrm{lim}}$ if
$A_{\alpha}=0$ for $\alpha\leq 0$. ∎

####  247 .

Putting the lemma and (245.1) together, we find that

$\Phi(z)=e^{z(S+N)}e^{\tilde{\Gamma}(e^{z})}e^{A}\cdot
F_{\mathrm{lim}}=e^{z(S+N)}e^{\Gamma(e^{z})}\cdot F_{\mathrm{lim}},$

where $\Gamma\colon\Delta_{\varepsilon}\to\mathfrak{q}$ is holomorphic and
satisfies $\Gamma(0)=A$. We can do even better by decomposing
$\Gamma(t)=\sum_{\alpha}\Gamma_{\alpha}(t)$; each individual function
$\Gamma_{\alpha}\colon\Delta_{\varepsilon}\to\mathfrak{q}_{\alpha}$ is
holomorphic, and $\Gamma_{\alpha}(0)=0$ for $\alpha\leq 0$. Since $S$
preserves the filtration $F_{\mathrm{lim}}$, we finally get

$\Phi(z)=e^{zN}\cdot e^{zS}e^{\Gamma(e^{z})}e^{-zS}\cdot
F_{\mathrm{lim}}=e^{zN}\cdot\exp\left(\sum_{\alpha}e^{\alpha
z}\,\Gamma_{\alpha}(e^{z})\right)\cdot F_{\mathrm{lim}}.$

To summarize, we have proved the following result.

######  Proposition 20.

There is a small positive number $\varepsilon>0$, and a collection of
holomorphic functions
$\Gamma_{\alpha}\colon\Delta_{\varepsilon}\to\mathfrak{q}_{\alpha}$ with
$\Gamma_{\alpha}(0)=0$ for $\alpha\leq 0$, such that the formula

$\Phi(z)=e^{zN}\cdot\exp\left(\sum_{\alpha}e^{\alpha
z}\,\Gamma_{\alpha}(e^{z})\right)\cdot F_{\mathrm{lim}}$

describes the period mapping on the half-plane
$\operatorname{Re}z<\log\varepsilon$.

Note that each term in the sum vanishes as
$\lvert\operatorname{Re}z\rvert\to\infty$. The rate at which this happens is
controlled by the smallest positive eigenvalue of $\operatorname{ad}S$; but
this is the smallest distance among consecutive eigenvalues of $S$, which we
had earlier denoted by $\delta(T)$. This shows one more time that
$e^{-zN}\Phi(z)$ approaches its limit $F_{\mathrm{lim}}$ at a rate
proportional to $e^{-\delta(T)\lvert\operatorname{Re}z\rvert}$.

##  11 Asymptotic behavior of the Hodge metric and the
$\operatorname{SL}(2)$-orbit theorem

####  248 .

In this chapter, we study the rate of convergence of the rescaled period
mapping, and prove a cheap version of Schmid’s famous
$\operatorname{SL}(2)$-orbit theorem [Schmid, Thm. 5.13].

####  249 .

On the vector space $V$, we have the limiting Hodge filtration
$F_{\mathrm{lim}}$, as well as the conjugate filtration
$\overline{F}\vphantom{F}_{\mathrm{lim}}$, defined by

$\overline{F}\vphantom{F}_{\mathrm{lim}}^{q}=\bigl{\\{}\thinspace v\in
V\thinspace\thinspace\big{|}\thinspace\thinspace\text{$Q(v,w)=0$ for all $w\in
F_{\mathrm{lim}}^{n+1-q}$}\thinspace\bigr{\\}}.$

These put a mixed Hodge structure on $V$, with weight filtration
$W_{\bullet-n}=W_{\bullet-n}(N)$; moreover, $N\colon V\to V(-1)$ is a morphism
of mixed Hodge structures. At the same time, $V$ also has a polarized
$\mathfrak{sl}_{2}$-Hodge structure of weight $n$, whose two Hodge filtrations
are $F_{H}$ and $\overline{F}\vphantom{F}_{H}$, where

$\overline{F}\vphantom{F}_{H}^{q}=\bigl{\\{}\thinspace v\in
V\thinspace\thinspace\big{|}\thinspace\thinspace\text{$Q(v,w)=0$ for all $w\in
F_{H}^{n+1-q}$}\thinspace\bigr{\\}}.$

####  250 .

We begin by comparing the two filtrations $F_{\mathrm{lim}}$ and $F_{H}$.

######  Lemma 39.

There is an element $h\in\operatorname{GL}(V)$ with $h-\operatorname{id}\in
W_{-1}\operatorname{End}(V)$ and $hNh^{-1}=N$, such that
$F_{\mathrm{lim}}=hF_{H}$.

###### Proof.

Let
$H(W_{\bullet-n},F_{\mathrm{lim}},\overline{F}\vphantom{F}_{\mathrm{lim}})$
denote Deligne’s splitting of the limiting mixed Hodge structure (as in 15).
The corresponding subspaces

$I_{\mathrm{lim}}^{i,j}=F_{\mathrm{lim}}^{i}\cap
W_{i+j-n}\cap\bigl{(}\overline{F}\vphantom{F}_{\mathrm{lim}}^{j}\cap
W_{i+j-n}+\overline{F}\vphantom{F}_{\mathrm{lim}}^{j-1}\cap
W_{i+j-2-n}+\overline{F}\vphantom{F}_{\mathrm{lim}}^{j-2}\cap
W_{i+j-3-n}+\dotsb\bigr{)}$

decompose $V$ into a direct sum, in such a way that

$W_{k}=\bigoplus_{i+j\leq n+k}I_{\mathrm{lim}}^{i,j}\qquad\text{and}\qquad
F_{\mathrm{lim}}^{p}=\bigoplus_{i\geq p,j}I_{\mathrm{lim}}^{i,j};$

moreover, one has $N(I_{\mathrm{lim}}^{i,j})\subseteq
I_{\mathrm{lim}}^{i-1,j-1}$, due to the fact that $N\colon V\to V(-1)$ is a
morphism of mixed Hodge structures. In terms of the splitting,
$[H(W_{\bullet-n},F_{\mathrm{lim}},\overline{F}\vphantom{F}_{\mathrm{lim}}),N]=-2N$.

Recall that in an $\mathfrak{sl}_{2}$-Hodge structure of weight $n$, one has

$V_{k}^{i,j}=F_{H}^{i}\cap V_{k}\cap\overline{F}\vphantom{F}_{H}^{j},$

whenever $i+j=n+k$. Under the projection from $W_{i+j-n}$ to
$\operatorname{gr}_{i+j-n}^{W}$, the two subspaces $I_{\mathrm{lim}}^{i,j}$
and $V_{i+j-n}^{i,j}$ both map isomorphically to the $(i,j)$-subspace in the
Hodge decomposition of $\operatorname{gr}_{i+j-n}^{W}$, and so there is a
unique isomorphism $h^{i,j}$ making the following diagram commute:

${V_{i+j-n}^{i,j}}$${I_{\mathrm{lim}}^{i,j}}$${\bigl{(}\operatorname{gr}_{i+j-n}^{W}\bigr{)}^{i,j}}$$\scriptstyle{\cong}$$\scriptstyle{h^{i,j}}$$\scriptstyle{\cong}$

Consequently, there is a unique automorphism $h\in\operatorname{GL}(V)$ such
that $h\big{|}_{V_{i+j-n}^{i,j}}=h^{i,j}$ for all $i,j\in\mathbb{Z}$. It
follows that

$h(F_{H}^{p})=\bigoplus_{i\geq
p}h\bigl{(}V_{i+j-n}^{i,j}\bigr{)}=\bigoplus_{i\geq
p}I_{\mathrm{lim}}^{i,j}=F_{\mathrm{lim}}^{p}.$

By construction, $h$ preserves the weight filtration $W$ and acts as the
identity on each $\operatorname{gr}_{k}^{W}$; therefore
$h-\operatorname{id}\in W_{-1}\operatorname{End}(V)$. Moreover, the diagram

${V_{i+j-n}^{i,j}}$${I_{\mathrm{lim}}^{i,j}}$${V_{i+j-2-n}^{i-1,j-1}}$${I_{\mathrm{lim}}^{i-1,j-1}}$$\scriptstyle{N}$$\scriptstyle{h^{i,j}}$$\scriptstyle{N}$$\scriptstyle{h^{i-1,j-1}}$

is commutative, and this implies that $hN=Nh$. In terms of the two splittings,
we have

$H(W_{\bullet-n},\overline{F}\vphantom{F},\overline{F}\vphantom{F}_{\mathrm{lim}})=h\bigl{(}H+n\operatorname{id}\bigr{)}h^{-1},$

which is consistent with 36. ∎

####  251 .

Note that, in the lemma above, the splitting $H$ was arbitrary, subject only
to the conditions in 9. If we pick a splitting that is adapted to the limiting
mixed Hodge structure, we can get a much better result. Indeed, suppose we use
instead the splitting
$H_{\mathbb{R}}(W_{\bullet-n},F_{\mathrm{lim}},\overline{F}\vphantom{F}_{\mathrm{lim}})\in\mathfrak{g}$
for the filtration $W_{\bullet}$, constructed in 19, which we got by averaging
the two splittings
$H(W_{\bullet-n},F_{\mathrm{lim}},\overline{F}\vphantom{F}_{\mathrm{lim}})$
and
$H(W_{\bullet-n},\overline{F}\vphantom{F}_{\mathrm{lim}},F_{\mathrm{lim}})$.
This also has all the properties required by 9. Then

$H_{\mathbb{R}}(W_{\bullet-n},F_{\mathrm{lim}},\overline{F}\vphantom{F}_{\mathrm{lim}})-\bigl{(}H(W_{\bullet-n},F_{\mathrm{lim}},\overline{F}\vphantom{F}_{\mathrm{lim}})-n\operatorname{id}\bigr{)}\in
R\bigl{(}W_{-2}\operatorname{End}(V)\bigr{)},$

and so the element $h\in\operatorname{GL}(V)$ in 39 even satisfies
$h-\operatorname{id}\in W_{-2}\operatorname{End}(V)$.

####  252 .

We return to our investigation of the two filtrations $F_{\mathrm{lim}}$ and
$F_{H}$. Let us write the element $h\in\operatorname{GL}(V)$ as a finite sum

$h=\operatorname{id}+h_{-1}+h_{-2}+\dotsb,$

where each $h_{-k}\in E_{-k}(\operatorname{ad}H)\cap\ker(\operatorname{ad}N)$.
Recall from 210 that

$F_{H}=\lim_{x\to\infty}e^{\frac{1}{2}\log x\,H}F_{\mathrm{lim}}.$

We can analyze the convergence more precisely as follows. For $x>0$, one has

$e^{\frac{1}{2}\log x\,H}F_{\mathrm{lim}}=e^{\frac{1}{2}\log
x\,H}hF_{H}=e^{\frac{1}{2}\log x\,H}he^{-\frac{1}{2}\log x\,H}F_{H},$

due to the fact that $H$ preserves the filtration $F_{H}$. Now

$e^{\frac{1}{2}\log x\,H}he^{-\frac{1}{2}\log
x\,H}=\operatorname{id}+\sum_{k=1}^{\infty}x^{-k/2}h_{-k},$

and since $[N,h_{-k}]=0$, we finally obtain

$e^{\frac{1}{2}\log x\,H}e^{-xN}F_{\mathrm{lim}}=e^{-N}e^{\frac{1}{2}\log
x\,H}F_{\mathrm{lim}}=\left(\operatorname{id}+\sum_{k=1}^{\infty}x^{-k/2}h_{-k}\right)\cdot
F_{\sharp},$

where $F_{\sharp}=e^{-N}F_{H}\in D$. Note that the filtration on the right-
hand side belongs to $D$ when $x\gg 0$. With the substitution $u=x^{-1/2}$, we
can rewrite this as

$\left(\operatorname{id}+\sum_{k=1}^{\infty}u^{k}h_{-k}\right)\cdot
F_{\sharp},$

as long as $u$ is sufficiently close to $0$.

####  253 .

To understand the behavior of the Hodge metric, we have to express this in
terms of the real group $G$. For simplicity, let us denote by

$V=\bigoplus_{p+q=n}V^{p,q}$

the Hodge structure with Hodge filtration $F_{\sharp}=e^{-N}F_{H}$, and by

$\operatorname{End}(V)=\bigoplus_{j\in\mathbb{Z}}\operatorname{End}(V)^{j,-j}$

the induced Hodge structure of weight $0$ on $\operatorname{End}(V)$. Recall
that this is an $\mathbb{R}$-Hodge structure; the Lie algebra $\mathfrak{g}$
of the real group $G$ gives the real structure. Setting

$\mathfrak{m}=\mathfrak{g}\cap\bigoplus_{j\neq 0}\operatorname{End}(V)^{j,-j}$

we then have $\operatorname{End}(V)=\mathfrak{m}\oplus
F^{0}\operatorname{End}(V)$; compare (76.2). In concrete terms, this
decomposition works as follows. For an endomorphism
$A\in\operatorname{End}(V)$, denote by $A_{j}\in\operatorname{End}(V)^{j,-j}$
the components in the Hodge decomposition; then

$\sum_{j<0}\bigl{(}A_{j}-(A_{j})^{\dagger}\bigr{)}\in\mathfrak{m}\quad\text{and}\quad\sum_{j\geq
0}A_{j}+\sum_{j<0}(A_{j})^{\dagger}\in F^{0}\operatorname{End}(V)$

####  254 .

Recall that the subspace $\mathfrak{m}\subseteq\operatorname{End}(V)$ maps
isomorphically to the tangent space of $D$ at the point $F_{\sharp}$. The
exponential mapping

$\operatorname{End}(V)\to D,\quad A\mapsto e^{A}\cdot F_{\sharp},$

is real analytic, and therefore restricts to a real-analytic diffeomorphism
between a neighborhood of the origin in $\mathfrak{m}$ and a neighborhood of
the point $F_{\sharp}\in D$, it follows that there is a small positive number
$\varepsilon>0$ and two unique real analytic functions

$B\colon(-\varepsilon,\varepsilon)\to\mathfrak{m}\quad\text{and}\quad
C\colon(-\varepsilon,\varepsilon)\to F^{0}\operatorname{End}(V)$

with the property that $B(0)=C(0)=0$ and

$\left(\operatorname{id}+\sum_{k=1}^{\infty}u^{k}h_{-k}\right)e^{C(u)}=e^{B(u)}.$

Let us expand $B$ and $C$ into convergent power series of the form

$B(u)=\sum_{k=1}^{\infty}u^{k}B_{-k}\quad\text{and}\quad
C(u)=\sum_{k=1}^{\infty}u^{k}C_{-k},$

where $B_{-k}\in\mathfrak{m}$ and $C_{-k}\in F^{0}\operatorname{End}(V)$.

####  255 .

From the power series expansion of

$e^{B(u)}=\operatorname{id}+uB_{-1}+u^{2}\Bigl{(}B_{-2}+\frac{1}{2}B_{-1}^{2}\Bigr{)}+\dotsb,$

we see that there are universal non-commutative polynomials $P_{-k}$ in the
variables $B_{-1},B_{-2},\dotsc$, such that

$e^{B(u)}=\operatorname{id}+\sum_{k=1}^{\infty}u^{k}P_{-k}(B_{-1},\dotsc,B_{-k}).$

The first few terms are easily computed to be

$\displaystyle P_{-1}(B_{-1})$ $\displaystyle=B_{-1},$ $\displaystyle
P_{-2}(B_{-1},B_{-2})$ $\displaystyle=B_{-2}+\frac{1}{2}B_{-1}^{2}$
$\displaystyle P_{-3}(B_{-1},B_{-2},B_{-3})$
$\displaystyle=B_{-3}+\frac{1}{2}B_{-2}B_{-1}+\frac{1}{2}B_{-1}B_{-2}+\frac{1}{6}B_{-1}^{3}.$

In general, the coefficients of $k!P_{-k}$ are positive integers; moreover,
$P_{-k}$ is homogeneous of degree $k$, if we declare that the degree of
$B_{-j}$ is $j$. It is easy to see that the variable $B_{-k}$ always appears
in $P_{-k}$ with coefficient $1$.

####  256 .

From the $\mathfrak{sl}_{2}$-Hodge structure on $V$, the vector space
$\operatorname{End}(V)$ inherits an $\mathfrak{sl}_{2}$-Hodge structure of
weight $0$. In particular, the Lie algebra $\mathfrak{sl}_{2}(\mathbb{C})$
acts on $\operatorname{End}(V)$, and therefore induces a decomposition

$\operatorname{End}(V)\cong\bigoplus_{m\in\mathbb{N}}S_{m}\otimes_{\mathbb{C}}\operatorname{Hom}_{\mathbb{C}}\bigl{(}S_{m},\operatorname{End}(V)\bigr{)}^{\mathfrak{sl}_{2}(\mathbb{C})}.$

The summand with $S_{m}$ is called the $\boldsymbol{S_{m}}$-isotypical
component. According to the discussion in 104, the isotypical components are
sub-Hodge structures of the Hodge structure on $\operatorname{End}(V)$. They
are also stable under taking adjoints; therefore the decomposition

$\operatorname{End}(V)=\mathfrak{m}\oplus F^{0}\operatorname{End}(V)$

respects the isotypical components.

####  257 .

Since each operator $h_{-k}\in
E_{-k}(\operatorname{ad}H)\cap\ker(\operatorname{ad}N)$ is primitive, it
obviously belongs to the $S_{k}$-isotypical component of the representation.
This has the following consequence for the operators
$B_{-k},C_{-k}\in\operatorname{End}(V)$.

######  Proposition 21.

For each $k\geq 1$, both $B_{-k}$ and $C_{-k}$ belong to the sum of the
$S_{j}$-isotypical components with $j=k,k-2,k-4,\dotsc$.

###### Proof.

Expanding the relation

$\displaystyle\left(\operatorname{id}+\sum_{k=1}^{\infty}u^{k}P_{-k}(B_{-1},\dotsc,B_{-k})\right)$
$\displaystyle=\left(\operatorname{id}+\sum_{k=1}^{\infty}u^{k}h_{-k}\right)$
$\displaystyle\left(\operatorname{id}+\sum_{k=1}^{\infty}u^{k}P_{-k}(C_{-1},\dotsc,C_{-k})\right)$

from above now gives us the following series of identities:

$P_{-k}(B_{-1},\dotsc,B_{-k})=h_{-k}+P_{-k}(C_{-1},\dotsc,C_{-k})+\sum_{j=1}^{k-1}h_{-k+j}\cdot
P_{-j}(C_{-1},\dotsc,C_{-j})$

For $k=1$, this reads $B_{-1}=h_{-1}+C_{-1}$; here $B_{-1}\in\mathfrak{m}$ and
$C_{-1}\in F^{0}\operatorname{End}(V)$. Remembering that
$\operatorname{End}(V)=\mathfrak{m}\oplus F^{0}\operatorname{End}(V)$, we see
again that this equation uniquely determines $B_{-1}$ and $C_{-1}$. Since
$h_{-1}$ belongs to the $S_{1}$-isotypical component of the representation,
and since this is a sub-Hodge structure, it follows that $B_{-1}$ and $C_{-1}$
also belong to the $S_{1}$-isotypical component.

We can now prove the assertion by induction on $k\geq 1$. Rearranging the
identity from above, we obtain an identity of the form

$B_{-k}-C_{-k}=P(B_{-1},\dotsc,B_{-k+1},C_{-1},\dotsc,C_{-k+1},h_{-1},\dotsc,h_{-k})$

for a certain non-commutative polynomial $P$, homogeneous of degree $k$. Now
each term on the right-hand side is a product of operators of the form
$B_{-j}$, $C_{-j}$ or $h_{-j}$, and by induction, each such operator is in the
direct sum of the $S_{j-2i}$-isotypical components for $i\geq 0$. Since

$S_{k}\otimes S_{\ell}\cong\bigoplus_{i=0}^{\min(k,\ell)}S_{k+\ell-2i},$

and since the degree of each term is $-k$, this gives the result. ∎

####  258 .

This has the following nice consequence for the nilpotent orbit
$e^{zN}F_{\mathrm{lim}}$. The result is of course less precise than the
$\operatorname{SL}(2)$-orbit theorem [Schmid, Thm. 5.13], but in return, the
proof is _much_ easier, and we believe that the version below is actually
sufficient for all practical purposes.

######  Theorem 19.

There is a constant $\varepsilon>0$, whose value depends on $F_{\mathrm{lim}}$
and $N$, and a real-analytic function
$B\colon(-\varepsilon,\varepsilon)\to\mathfrak{g}$, such that

$\Phi_{\mathrm{nil}}(x+iy)=e^{(x+iy)N}F_{\mathrm{lim}}=e^{iyN}e^{-\frac{1}{2}\log\lvert
x\rvert\,H}e^{B(\lvert x\rvert^{-1/2})}\cdot e^{-N}F_{H}$

as long as $\lvert x\rvert>1/\varepsilon^{2}$. In the power series expansion

$B(u)=\sum_{k=1}^{\infty}u^{k}B_{-k},$

each coefficient $B_{-k}\in\mathfrak{g}$ belongs to the direct sum of the
$S_{m}$-isotypical components of the
$\mathfrak{sl}_{2}(\mathbb{C})$-representation with $m=k,k-2,k-4,\dotsc$.

The point is that all the operators on the right-hand side of this formula
belong to the real group $G$. This makes it possible to describe the
asymptotic behavior of the Hodge metric or the Weil operator in any nilpotent
orbit.

######  Note.

Instead of an arbitrary splitting $H\in\operatorname{End}(V)$, we can use the
splitting

$H_{\mathbb{R}}(W_{\bullet-n},F_{\mathrm{lim}},\overline{F}\vphantom{F}_{\mathrm{lim}})\in\mathfrak{g}$

given by 19. Then $h-\operatorname{id}\in W_{-2}\operatorname{End}(V)$, and an
analysis of the proof shows that $B_{-1}=0$. With this choice, the statement
of 19 becomes closer to the full $\operatorname{SL}(2)$-orbit theorem in
[Schmid, Thm. 5.13].

####  259 .

We can use 19 to get more precise information about the asymptotic behavior of
the Hodge metric. The Hodge norm estimates tell us the exact rate of growth of
$\lVert v\rVert_{\Phi(z)}^{2}$ for any $v\in V$, but so far, we have not said
anything about the behavior of the inner products $\langle
v,w\rangle_{\Phi(z)}$ when $v,w\in V$ are two different multi-valued flat
sections.

####  260 .

Let us denote by $\langle v,w\rangle$ the inner product coming from the
polarized Hodge structure $F_{\sharp}=e^{-N}F_{H}$. Set $z=x+iy$. According to
17, we have

$d_{D}\bigl{(}\Phi(x+iy),e^{(x+iy)N}F_{\mathrm{lim}}\bigr{)}\leq C\lvert
x\rvert^{m}e^{-\delta(T)\lvert x\rvert}$

as long as $x\leq x_{0}$, with constants $C>0$ and $x_{0}<0$ that are
basically independent of the period mapping in question. Since the exponential
factors in the definition of $\hat{\Phi}_{S,H}$ belong to the real Lie group
$G$, it follows that

$d_{D}\Bigl{(}\hat{\Phi}_{S,H}(x+iy),\,e^{\frac{1}{2}\log\lvert
x\rvert\,H}e^{xN}F_{\mathrm{lim}}\Bigr{)}\leq C\lvert
x\rvert^{m}e^{-\delta(T)\lvert x\rvert}.$

We now restrict our attention to $\lvert x\rvert>1/\varepsilon^{2}$, and set
$u=\lvert x\rvert^{-\frac{1}{2}}$. Then

$e^{\frac{1}{2}\log\lvert
x\rvert\,H}e^{xN}F_{\mathrm{lim}}=e^{B(u)}F_{\sharp},$

and so our distance estimate becomes

$d_{D}\Bigl{(}\hat{\Phi}_{S,H}(x+iy),\,e^{B(u)}F_{\sharp}\Bigr{)}\leq C\lvert
x\rvert^{m}e^{-\delta(T)\lvert x\rvert}.$

13 allows us to conclude that, as long as $\lvert x\rvert$ is sufficiently
large,

$\Bigl{\lvert}\langle v,w\rangle_{\hat{\Phi}_{S,H}(x+iy)}-\langle
v,w\rangle_{e^{B(u)}F_{\sharp}}\Bigr{\rvert}\leq C^{\prime}\lVert
v\rVert_{e^{B(u)}F_{\sharp}}\lVert w\rVert_{e^{B(u)}F_{\sharp}}\cdot\lvert
x\rvert^{m}e^{-\delta(T)\lvert x\rvert}$

for some constant $C^{\prime}>0$ that is independent of $v,w\in V$.

####  261 .

Now suppose that $v\in E_{k}(H)$ and $w\in E_{\ell}(H)$ belong to possibly
different weight spaces, say with $k\leq\ell$. The behavior of the inner
product

$\langle v,w\rangle_{e^{B(u)}F_{\sharp}}=\langle e^{-B(u)}v,e^{-B(u)}w\rangle$

is very easy to understand, given what we know about the coefficients of the
series $B(u)$. The whole expression is a convergent power series in $u$. Since
different eigenspaces of $H$ are orthogonal under the inner product $\langle
v,w\rangle$, the leading term in this series is divisible by $u^{\ell-k}$. It
follows that there is a constant $C>0$ such that for
$\lvert\operatorname{Re}z\rvert\gg 0$, one has

$\bigl{\lvert}\langle v,w\rangle_{\hat{\Phi}_{S,H}(z)}\bigr{\rvert}\leq
C\lVert v\rVert\lVert
w\rVert\cdot\lvert\operatorname{Re}z\rvert^{-(\ell-k)/2}.$ (261.1)

From the definition of the rescaled period mapping in (21.1), we get

$\bigl{\langle}v,\,w\bigr{\rangle}_{\hat{\Phi}_{S,H}(z)}=\lvert\operatorname{Re}z\rvert^{-(k+\ell)/2}\cdot\bigl{\langle}e^{\frac{1}{2}(z-\mathchoice{\macc@depth\char
1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a
111{}}{\macc@depth\char
1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a
111{}}{\macc@depth\char
1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a
111{}}{\macc@depth\char
1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a
111{}})(S+N)}v,\,e^{\frac{1}{2}(z-\mathchoice{\macc@depth\char
1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a
111{}}{\macc@depth\char
1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a
111{}}{\macc@depth\char
1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a
111{}}{\macc@depth\char
1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a
111{}})(S+N)}w\bigr{\rangle}_{\Phi(z)},$

and as long as $\operatorname{Im}z$ remains bounded, it follows that

$\bigl{\lvert}\langle v,w\rangle_{\Phi(z)}\bigr{\rvert}\leq C^{\prime}\lVert
v\rVert\lVert w\rVert\cdot\lvert\operatorname{Re}z\rvert^{k}$ (261.2)

for a different constant $C^{\prime}>0$ and $\lvert\operatorname{Re}z\rvert\gg
0$.

####  262 .

We know from the Hodge norm estimates that $\lVert v\rVert_{\Phi(z)}^{2}$ and
$\lVert w\rVert_{\Phi(z)}^{2}$ are of order
$\lvert\operatorname{Re}z\rvert^{k}$ and
$\lvert\operatorname{Re}z\rvert^{\ell}$, respectively. Consequently, (261.2)
is equivalent to the statement that, for $\lvert\operatorname{Re}z\rvert\gg
0$, one has

$\bigl{\lvert}\langle v,w\rangle_{\Phi(z)}\bigr{\rvert}\leq
C^{\prime\prime}\min\bigl{(}\lVert v\rVert_{\Phi(z)}^{2},\lVert
w\rVert_{\Phi(z)}^{2}\bigr{)}.$

Because of the triangle inequality, we deduce that for any two multi-valued
flat sections $v,w\in V$, there is a constant $C(v,w)>0$ such that

$\bigl{\lvert}\langle v,w\rangle_{\Phi(z)}\bigr{\rvert}\leq
C(v,w)\cdot\min\Bigl{(}\lVert v\rVert_{\Phi(z)}^{2},\lVert
w\rVert_{\Phi(z)}^{2}\Bigr{)}$

for $\lvert\operatorname{Re}z\rvert\gg 0$. This finishes the proof of 9 from
the introduction.

## References

  * [Ahl38] Lars V. Ahlfors, _An extension of Schwarz’s lemma_ , Trans. Amer. Math. Soc. 43 (1938), no. 3, 359–364. MR 1501949
  * [Ber10] Bo Berndtsson, _An introduction to things $\overline{\partial}$_, Analytic and algebraic geometry, IAS/Park City Math. Ser., vol. 17, Amer. Math. Soc., Providence, RI, 2010, pp. 7–76. MR 2743815
  * [Cat08] Eduardo Cattani, _Mixed Lefschetz theorems and Hodge-Riemann bilinear relations_ , Int. Math. Res. Not. IMRN (2008), no. 10, Art. ID rnn025, 20. MR 2429243
  * [CG75] Maurizio Cornalba and Phillip Griffiths, _Analytic cycles and vector bundles on non-compact algebraic varieties_ , Invent. Math. 28 (1975), 1–106. MR 367263
  * [CK89] Eduardo Cattani and Aroldo Kaplan, _Degenerating variations of Hodge structure_ , Astérisque (1989), no. 179-180, 9, 67–96, Actes du Colloque de Théorie de Hodge (Luminy, 1987). MR 1042802 (91k:32019)
  * [CKS86] Eduardo Cattani, Aroldo Kaplan, and Wilfried Schmid, _Degeneration of Hodge structures_ , Ann. of Math. (2) 123 (1986), no. 3, 457–535. MR 840721 (88a:32029)
  * [dCM05] Mark de Cataldo and Luca Migliorini, _The Hodge theory of algebraic maps_ , Ann. Sci. École Norm. Sup. (4) 38 (2005), no. 5, 693–750. MR 2195257 (2007a:14016)
  * [Del70] Pierre Deligne, _Équations différentielles à points singuliers réguliers_ , Lecture Notes in Mathematics, Vol. 163, Springer-Verlag, Berlin, 1970. MR 0417174 (54 #5232)
  * [Del71] , _Théorie de Hodge. II_ , Inst. Hautes Études Sci. Publ. Math. (1971), no. 40, 5–57. MR 0498551 (58 #16653a)
  * [Del84] , _Positivité: signes_ , Personal note, dated Feb. 11, 1984.
  * [Dem12] Jean-Pierre Demailly, _Complex analytic and differential geometry_ , 2012.
  * [Den22] Ya Deng, _On the nilpotent orbit theorem of complex variation of Hodge structures_ , 2022, preprint arXiv:2203.04266.
  * [GS75] Phillip Griffiths and Wilfried Schmid, _Recent developments in Hodge theory: a discussion of techniques and results_ , Discrete subgroups of Lie groups and applicatons to moduli (Internat. Colloq., Bombay, 1973), 1975, pp. 31–127. MR 0419850
  * [Kas85] Masaki Kashiwara, _The asymptotic behavior of a variation of polarized Hodge structure_ , Publ. Res. Inst. Math. Sci. 21 (1985), no. 4, 853–875. MR 817170 (87h:32049)
  * [Moc02] Takuro Mochizuki, _Asymptotic behaviour of tame nilpotent harmonic bundles with trivial parabolic structure_ , J. Differential Geom. 62 (2002), no. 3, 351–559. MR 2005295
  * [Moc07] , _Asymptotic behaviour of tame harmonic bundles and an application to pure twistor $D$-modules. I_, Mem. Amer. Math. Soc. 185 (2007), no. 869, xii+324. MR 2281877
  * [Mor78] John W. Morgan, _The algebraic topology of smooth algebraic varieties_ , Inst. Hautes Études Sci. Publ. Math. (1978), no. 48, 137–204. MR 516917
  * [Sch73] Wilfried Schmid, _Variation of Hodge structure: the singularities of the period mapping_ , Invent. Math. 22 (1973), 211–319. MR 0382272 (52 #3157)
  * [Sim88] Carlos T. Simpson, _Constructing variations of Hodge structure using Yang-Mills theory and applications to uniformization_ , J. Amer. Math. Soc. 1 (1988), no. 4, 867–918. MR 944577
  * [Sim90] , _Harmonic bundles on noncompact curves_ , J. Amer. Math. Soc. 3 (1990), no. 3, 713–770. MR 1040197
  * [Zuc79] Steven Zucker, _Hodge theory with degenerating coefficients. $L_{2}$ cohomology in the Poincaré metric_, Ann. of Math. (2) 109 (1979), no. 3, 415–476. MR 534758 (81a:14002)