myequation

$\BODY$

(0.0.1)

# A New Approach To Text Rating Classification Using Sentiment Analysis

Thomas Konstantinovsky Department of Computer Science, Holon Institute of
Technology

###### Abstract

Typical use cases of sentiment analysis usually revolve around assessing the
probability of a text belonging to a certain sentiment and deriving insight
concerning it; little work has been done to explore further use cases derived
using those probabilities in the context of rating. In this paper, we redefine
the sentiment proportion values as building blocks for a triangle structure,
allowing us to derive variables for a new formula for classifying text given
in the form of product reviews into a group of higher and a group of lower
ratings and prove a dependence exists between the sentiments and the ratings.

Keywords— natural language processing, sentiment analysis, statistical
inference, machine learning

##  1 Introduction

In our current day and age, reviews are part of almost every product/service
provided on the internet[14], as seen in [8] it is the primary way for a
company to get an understanding concerning the amount of success their product
has and as examined in [7] for the customer to build trust in purchasing or
using a service of which only a description or a picture exits. Therefore, a
need for a deeper understanding and analysis of those reviews are needed[9]
for any individual who wishes to derive various consequences regarding a
product. Standard methods for such insight derivation include sentiment
analysis, around which we will formulate a new approach for review rating
classification. Reviews across the internet mainly consist of text-based and
rating-based formats, where in many cases, a combination of both is considered
a single review; the method developed in this paper focuses on the ability to
associate a review to a rating cluster based on sentiment proportions. We will
define two main groups: one group consisting of a majority of reviews higher
than three stars (in a 5-star ranking system) and another group of all
reviews, which correspond to the less than three stars.

As the analyzing end of a set of reviews, we may not have access to a
categorical ranking system provided with the reviews as part of the data. To
solve this problem, we might want to generate ratings or classify the reviews,
but this is not a trivial problem as studies suggest that rating inference
shall not be tackled as a classical multi-category classification problem, as
the ordering of class labels in rating inference is essential [4], reinforced
by ambiguity, one of the biggest challenges in natural language processing,
elements such as different personalities and sarcasm skew the results of
traditional classification models [12].

Having the review text is enough to calculate and extract the sentiment
proportions; using those proportions, we will construct new attributes and a
new hyperparameter from those new attributes that will allow us to project the
data into a space that separates the reviews into two main groups, as stated
before.

Figure 1 shows the spread of different tripadvisor hotel reviews[15] plotted
via their sentiment proportions; the color scale represents the rating given
in each of the reviews; throughout our calculations, the real rating that the
person left is only for visual conformation and is never included in any part
of our calculation. There is no simple solution when considering separating
the reviews with high ratings (points that are closer to yellow color) from
the reviews with low ratings (points that are closer to a blue color) without
using any machine learning process or including the actual rating values in a
calculation.

Figure 1: Example of a Review Data-set Plotted Using Sentiment Values

A natural assumption to be made is of dependence between a review text and its
corresponding rating given by the user; following this assumption, one can
associate a review to a rating based on the text alone and as the case in this
paper using the text’s sentiment proportions. In contrast to the topic
research in [10] where a connection between the sentiment, rating and their
inner relation to costumer decision making was examined, the purpose of this
paper is to establish a direct dependence and define the rating as a function
of a review sentiments. The equation proposed gives an approximation for such
a separation without iteratively training any model, a separation that helps
us generate ratings and confirms our assumption.

##  2 Methods

###  2.1 The Vader Algorithm

The building blocks from which we derive our formula are generated using The
Vader sentiment analysis algorithm [11], which takes text as an input and
returns three values, which we will refer to throughout the paper as "Pos" -
the proportion of sentences in the text that are tagged positive "Neg" - the
proportion of sentences in the text that are tagged negative and "Neu" = the
proportion of sentences in the text that are tagged neutral.

######  Definition 1.

$0\leq(Pos,Neu,Neg)\leq 1$ $Pos+Neu+Neg=1$

###  2.2 Translation Into A Triangle

The first step towards our formula consists of deriving attributes from a
triangle. The triangle from which we will derive those attributes will be
constructed using the sentiments extracted using the Vader algorithm; for us
to construct such a triangle, we transformed the 1D values into 2D vectors to
place them on 3 different line spans. A 2-dimensional space was divided into
three where the angles between each dividing line are 120 degrees; each
dividing line represents the vector on which the values (Pos/Neg/Neu)
extracted by the Vader the algorithm will span, as can be seen in Figure 2.

Figure 2: The Line On Which Each Sentiment Will Span

The three sentiment values are multiplied by rotation vectors to receive new
points located on the appropriate span line as we defined above. This
transformation $\mathbb{R}^{1}\longrightarrow\mathbb{R}^{2}$ is defined as
follows;

######  Definition 2.

$\bm{Pos}=[0,Pos]$
$\bm{Neg}=Neg\cdot[cos(\frac{-\pi}{4}),sin(\frac{-\pi}{4})]$
$\bm{Neu}=Neu\cdot[cos(\frac{-5\pi}{4}),sin(\frac{5\pi}{4})]$

###  2.3 Formulating Variables

Now that Pos, Neg, and Neu are vectors in $\mathbb{R}^{2}$ spanning on the
lines already defined, we derive the variables needed for our equation.

######  Definition 3 (a, b and c variables representing triangle side lengths
).

$a=||\bm{Neu}-\bm{Pos}||$ $b=||\bm{Neu}-\bm{Neg}||$ $c=||\bm{Pos}-\bm{Neg}||$

######  Definition 4 ($\alpha$ is calculated using Heron’s formula[5]).

$s=\frac{(a+b+c)}{2}$ $\alpha=\sqrt{s\cdot(s-a)\cdot(s-b)\cdot(s-c)}$

######  Definition 5 ($\beta$ is the inner product weighted by the magnitude
ratio between the difference from $\bm{Neu}$).

$\bm{u}=\bm{Neu}-\bm{Pos}\;\;\;\bm{v}=\bm{Neu}-\bm{Neg}$
$\beta=\arccos((\bm{u}\cdot\bm{v})\cdot\frac{||\bm{v}||}{||\bm{u}||})$

######  Definition 6 ($\gamma$ is the height of a triangle where the maximum
side length is considered to be the base and $\alpha$ to be the area of the
triangle).

$T_{h}=max(a,b,c)$ $\gamma=\frac{2\cdot\alpha}{T_{h}}$

###  2.4 The $\Omega$ hyper-parameter

After deriving the variables, we can now formulate the equation for our new
hyperparameter. In the process of constructing the formula, different
functions of Pos, Neg, and Neu were tested while keeping in mind the goal of
creating a sufficient separation between rating clusters. No significant
results were achieved; this called for formulating a new hyperparameter
dependent on variables defined in the previous section, representing the
separation dimension. The equation’s general form resulted from a conceptual
idea translated into mathematical notations, and we refer to this
hyperparameter as $\Omega$.

######  Definition 7.

Ω= ln(α^β+ϵ) % e^γ | ϵ⟶0

The idea behind the structure of the formula is finding a meaningful rate of
growth for a sequence of rational numbers that converge to the irrational
exponent; conceptually, the purpose of the natural logarithm is to retrieve
the number of units or the “time” at which a sequence of $\alpha$’s converges
to the $\beta$’s as defined in the previous section. Afterward, we take the
remainder (modulus) of the “growth rate” with the $\gamma$ variable’s growth
value; In other words, the hyperparameter should represent different states of
the growth rate in terms of the exponent.

Each review will be now represented as a vector in $\mathbb{R}^{3}$ where the
vector elements are : $[a,c,\Omega]$. Projecting each review vector to such a
space creates a clear separation between two groups of different rating
consistencies one consists of a majority of high ratings i.e., higher than 3
out of 5, and the second of those with lower ratings i.e., lower than 3 out of
5.

##  3 Results

We have observed that by looking at the spread of data in 3 dimensions where
our axes were $\Omega$ which is the new hyperparameter calculated, “a” and “c”
we get a segmentation into 2 main clusters, a cluster with a majority of
highly rated samples and a cluster of samples with a majority lower ratings.

[13]

Figure 3: Resulting Data Spread - Hotel Reviews

The same results were archived on multiple different datasets with different
ranges of themes and topic; as stated in the introduction, the main example
shown throughout the paper is of Tripadvisor hotel reviews, figure 4 and
figure 5 shows the same process applied to a dataset of different earphones
reviews from Amazon111https://www.kaggle.com/shitalkat/amazonearphonesreviews,
and a dataset of wine reviews222https://www.kaggle.com/krrai77/wine-reviews.

Figure 4: Resulting Data Spread - Amazon Headphone Reviews

Figure 5: Resulting Data Spread - Wine Reviews

###  3.1 Cluster Extraction

Extracting the clusters formed is easily done using the DBSCAN algorithm [2]
where the optimal clustering as tested on multiple datasets, including outlier
detection, is achieved by using a minimum neighborhood size of 7 and an
epsilon radius of 0.09.

Figure 6: Clusters Tagged Using DBSCAN

Figure 6 shows our formula’s resulting separation, the negative cluster, which
represents the low rated reviews, i.e., under 3 out 5 stars is colored in
orange. The positive cluster representing the high-rated reviews, i.e., higher
than 3 out 5 stars, is colored in purple. The most positive reviews, strictly
4 and above stars rated, in the dataset are colored yellow, represented by the
line above the positive cluster; note that in a case where there are extremely
negative reviews, they will appear similarly as a line under the negative
cluster. Outliers were colored in blue as they deviate extremely from the main
behavior of the remaining points.

Figure 7: Our Clusters in Terms of The Initial Axes, Similarly to Figure 1

In Figure 7, the clusters tagged using DBSCAN in the transformed space were
colored using the same colors to emphasize the non-linear separation achieved
without iteratively training any model or searching for some separating
margin. Not only do we see how our main clusters capture the regions of rating
above and under 3-stars as mentioned before, but we also can see how the
yellow line that we saw in Figure 6 and the blue outliers are located with
respect to the entire dataset, the blue outliers are on the extremes of each
vertex of the triangular spread of points and the yellow cluster, i.e., "most
positive." ratings which are strictly 4 and above stars is on the extreme
vertex of low negative sentiment and high positive sentiment.

###  3.2 Cluster Characteristics

To get a better understanding of the characteristics of the clusters and how
ratings are distributed within them, Bayesian inference was used to understand
what are the underlying probabilities of the cluster ratings, To generate such
a process, a Monte Carlo Markov Chain algorithm [3] was used to sample a
posterior space for a vector $\bm{\Theta}$.

$\bm{\Theta}=[\theta_{1},\theta_{2},\theta_{3},\theta_{4},\theta_{5}]$

It is modeled using a Dirichlet distribution[6] initiated with a uniform prior
as we do not want to assume any prior knowledge regarding the rating
probabilities in each cluster.

$\alpha=[1,1,1,1,1]$ $\bm{\Theta}\sim Dir(\alpha)$

The Theta probability vector will be the input for a multinominal model from
which we will sample our probabilities for each rating.

$N=\text{number of samples}$ $P_{Rating}\sim Multinominal(N,\bm{\Theta})$

It is important to note that we hypothesize that in the case of a bad
separation, both of the major clusters we saw in the previous section will
have close to a uniform distribution for each rating probability, i.e., each
rating will have a probability of around 20% of appearing in the cluster. In
contrast, if indeed the separation was significant, we expect to see the
positive cluster, which is colored in purple in figure 4, to have most of its
credibility associated with ratings higher than 3, the same goes for the
negative cluster colored orange in figure 4 which we expect to have most of
its credibility associated with ratings lower than 3.

Figure 8: Positive Cluster Posterior Rating Probability Distribution

As can be seen in Figure 8, when we inferred the positive cluster as described
above, it is clear a 5-star rating is the most probable when considering a
random text that belongs to the positive cluster; in fact, a text belonging to
this cluster has 83% chance of being higher than 4 stars when taking in mind
the anomalies mentioned earlier in the paper such as users who leave a
positive review with a low rating will appear in this cluster. Another
important notice is that the posterior distributions’ deviation is quite tight
around a small interval, a fact that supports our hypothesis and reinforces
our confidence in the probability estimates.

Figure 9: Negative Cluster Posterior Rating Probability Distribution

Similarly to the interpretation, we did in Figure 8, Figure 9 represent the
same process applied to the negative cluster, and we see that the results in
this cluster support our hypothesis as well, not only the highest probability
for a rating of a text belonging to this cluster is 1-star, there is 87%
chance that a point from this cluster will have a rating lower than 3-stars.
Similarly to the previous explanation, there may be positive reviews with low
ratings, which will appear in the negative cluster.

##  4 Discussion

At first, when looking at the distribution of the reviews sentiments and the
rating behind each point, as seen in Figure 1, it appeared that there might be
a relationship between the sentiment proportion given by the Vader algorithm
and the rating, we hypothesized that the reviews with lower rating could be
naturally clustered so as the review with higher ratings, an equation for a
new hyperparameter was introduced, thanks to this new hyperparameter we
achieved a transformation in which the data is separated into clusters that
meet our hypothesis as we explained in the previous section. Empirically
proving the dependence between the rating and the sentiment proportions and
formulating the equation in this paper not only provides a new tool to the
field of natural language processing for generating ratings based only on text
sentiments in the absence of such data but also proposes a further
investigation into the given equation and the application it might have on
different fields. The process can be repeated for any vector of values meeting
the criteria given in Definition 1.

The anomalies seen within the clusters raise a question about the effect of
the prior distribution of ratings on new reviews being written at a certain
platform; we might think of the rating given by a user in an analogous way to
the Chinese restaurant stochastic process[1], where we assume that each new
review rating is affected directly by the current total ratings, i.e., a
person is more likely to leave a high rating to a product/service that is
highly rated even if he thinks lower of the product and vice versa when the
total rating of the product/service is low. When we inferred the clusters, we
saw that there were lower ratings residing in the positive cluster and higher
ratings residing in the negative cluster; in the same manner, as in the
Chinese restaurant process, a new guest has a probability of choosing a table
that is not strictly the one with the highest probability we see people who
are giving a rating opposite to the sentiment of their review. In future
studies, it will be of value to give an analytic proof for the equation
present in this paper.

##  5 Conclusions

A new formula was introduced in this paper, enabling deeper analysis of text
data in the form of reviews by proposing a direct calculation for rating
classification into one of two clusters. Not only did we establish a
dependence between the sentiment proportions of a review to the corresponding
rating, but we also achieved a direct transformation from the sentiment
proportions to the separated space without any use of classic iterative
converging machine learning algorithms.

## References

  * [1] David J. Aldous “Exchangeability and related topics” In _École d’Été de Probabilités de Saint-Flour XIII — 1983_ Berlin, Heidelberg: Springer Berlin Heidelberg, 1985, pp. 1–198
  * [2] Martin Ester, Hans-Peter Kriegel, Jörg Sander and Xiaowei Xu “A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise” In _Proc. of 2nd International Conference on Knowledge Discovery and_ , 1996, pp. 226–231
  * [3] B. Walsh “Markov Chain Monte Carlo and Gibbs Sampling”, 2004
  * [4] Daisuke Okanohara and Jun’ichi Tsujii “Assigning Polarity Scores to Reviews Using Machine Learning Techniques”, 2005, pp. 314–325 DOI: 10.1007/11562214_28
  * [5] “Generalization of the formula of Heron” In _Selected Topics in Geometry with Classical vs. Computer Proving_ WORLD SCIENTIFIC, 2007, pp. 33–63 DOI: 10.1142/9789812709431_0003
  * [6] “Dirichlet Distribution” In _Dirichlet and Related Distributions_ John Wiley & Sons, Ltd, 2011, pp. 37–96 DOI: 10.1002/9781119995784.ch2
  * [7] Sonja Utz, Peter Kerkhof and Joost Bos “Consumers rule: How consumer reviews influence perceived trustworthiness of online stores” In _Electronic Commerce Research and Applications_ , 2011 DOI: 10.1016/j.elerap.2011.07.010
  * [8] Georg Lackermair, D. Kailer and Kenan Kanmaz “Importance of Online Product Reviews from a Consumer’s Perspective” In _Advances in economics and business_ 1, 2013, pp. 1–5
  * [9] Manuel Trenz and Benedikt Berger “Analyzing Online Customer Reviews - An Interdisciplinary Literature Review And Research Agenda”, 2013 DOI: 10.13140/2.1.1203.1361
  * [10] Nan Hu, Noi Sian Koh and Srinivas K. Reddy “Ratings lead you to the product, reviews help you clinch it? The mediating role of online review sentiments on product sales” In _Decision Support Systems_ 57, 2014, pp. 42–53 DOI: https://doi.org/10.1016/j.dss.2013.07.009
  * [11] Clayton J. Hutto and Eric Gilbert “VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text.” In _ICWSM_ The AAAI Press, 2014 URL: http://dblp.uni-trier.de/db/conf/icwsm/icwsm2014.html#HuttoG14
  * [12] Diana Maynard and M.A. Greenwood “Who cares about sarcastic tweets? Investigating the impact of sarcasm on sentiment analysis”, 2014
  * [13] Plotly Technologies Inc. “Collaborative data science” Montreal, QC: Plotly Technologies Inc., 2015 URL: https://plot.ly
  * [14] Ewa Masłowska, Edward Malthouse and Stefan Bernritter “The Effect of Online Customer Reviews’ Characteristics on Sales”, 2017, pp. 87–100 DOI: 10.1007/978-3-658-15220-8_8
  * [15] Barkha Bansal “TripAdvisor Hotel Review Dataset” Zenodo, 2018 DOI: 10.5281/zenodo.1219899


# An Intrinsic Vector Heat Network

Alexander Gao  Maurice Chu  Mubbasir Kapadia  Ming C. Lin  Hsueh-Ti Derek Liu

###### Abstract

Vector fields are widely used to represent and model flows for many science
and engineering applications. This paper introduces a novel neural network
architecture for learning tangent vector fields that are intrinsically defined
on manifold surfaces embedded in 3D. Previous approaches to learning vector
fields on surfaces treat vectors as multi-dimensional scalar fields, using
traditional scalar-valued architectures to process channels individually, thus
fail to preserve fundamental intrinsic properties of the vector field. The
core idea of this work is to introduce a trainable vector heat diffusion
module to spatially propagate vector-valued feature data across the surface,
which we incorporate into our proposed architecture that consists of vector-
valued neurons. Our architecture is invariant to rigid motion of the input,
isometric deformation, and choice of local tangent bases, and is robust to
discretizations of the surface. We evaluate our Vector Heat Network on
triangle meshes, and empirically validate its invariant properties. We also
demonstrate the effectiveness of our method on the useful industrial
application of quadrilateral mesh generation.

Machine Learning, ICML

##  1 Introduction

Tangent vector fields on Riemannian manifolds are a fundamental ingredient in
scientific computation, with applications ranging from modeling physical
processes on earth (Sabaka et al., 2010), to robotic navigation on complex
terrains (van den Berg et al., 2008), to mesh generation (de Goes et al.,
2016). The majority of works on _learning_ of tangent vector fields rely on
neural network architectures that consist of _scalar_ neurons (e.g., the
standard multilayer perceptron).

Figure 1: The tangent plane $T_{p}\mathcal{M}$ at point $p$ on a manifold
$\mathcal{M}$ does not have a canonical choice of basis vectors
${\mathbf{e}}_{0},{\mathbf{e}}_{1}$. Our proposed architecture for learning
tangent vector fields is invariant to choice of tangent bases.

Despite being straightforward to implement, these scalar-valued architectures
assume that each scalar channel can be processed independently of the others
(e.g. as with RGB color channels in images). However, the multiple channels of
a directional vector must be considered jointly rather than independently, as
they represent entangled properties (length and orientation). As an example, a
simple rotation requires adjusting all channels of the vector to maintain its
length; a scalar-valued neural network does not inherently disentangle such
vector properties. Scalar-valued architectures also disregard fundamental
invariances of tangent vector fields, such as the choice of local tangent
bases in which the vectors are expressed (see Figure 1). This prohibits their
generalization to unseen data that does not share the same (arbitrary) choice
of bases.

In this work, we present a neural network architecture for processing tangent
vector fields defined on 2-manifolds embedded in $\mathbb{R}^{3}$. The key
idea is to maintain vector-valued features throughout the architecture, and
utilize a trainable vector heat diffusion process (not to be confused with
Diffusion Models (Yang et al., 2022)) to ensure our architecture maintains
necessary invariances for tangent vector field processing. We begin by
reviewing necessary background on tangent vector field processing in section
3. In section 4, we detail our proposed architecture. In section 5, we show
that our approach is invariant to choice of local tangent bases, rigid
transformation, and isometric deformation of the input, and is robust to
different discretizations of the manifold surface. Finally, in section 6, we
highlight an application of our work in quadrilateral mesh generation for
animation.

##  2 Related Work

Our work is an instance of geometric deep learning (Bronstein et al., 2021)
focused on learning tangent vector fields defined on discrete surfaces
embedded in $\mathbb{R}^{3}$. Such surfaces can be represented as point clouds
(Guo et al., 2021), implicits (Xie et al., 2022), or analytical functions
(Cohen et al., 2019) for simple shapes, e.g. a sphere. While other researchers
have focused on learning vector fields defined on the entire volume, such as
(Yang et al., 2023), we focus on tangent vector fields defined on the
_triangle mesh_ , a widely used surface representation for graphics,
scientific computing, and engineering applications.

A majority of works on this subject focus on developing fundamental operators
in neural networks (such as convolutions) to process _scalar_ fields defined
on triangle mesh elements, such as vertices (Gong et al., 2019; Lahav & Tal,
2020), edges (Hanocka et al., 2019; Liu et al., 2020; Ludwig et al., 2023),
and faces (Feng et al., 2019; Hertz et al., 2020; Hu et al., 2022). As a
triangle mesh is merely a graph with triangular faces, graph neural networks
(GNNs) (Wu et al., 2021) have also been deployed to learning tasks on meshes,
such as (Milano et al., 2020; Pfaff et al., 2021). Despite being effective,
the fact that these operations are defined based on the _connectivity_ of the
mesh makes them sensitive to the quality of the triangulation.

In lieu of this, spectral approaches (Wiersma et al., 2022; Sharp et al.,
2022) operate on the _functional space_ characterized by differential
operators. Network architectures that operate on these function spaces, such
as the eigenvectors of the Laplace operator, are more robust to discretization
and are able to generalize to other domains (e.g., point clouds) as long as
the differential operators are defined.

The basic building block of our architecture is the _vector_ heat diffusion
process with the _connection Laplacian_. This shares similarities with GNN
architectures motivated by diffusion processes. As pointed out in Hansen &
Gebhart (2020), the graph convolution can be viewed as heat diffusion with the
connection Laplacian using _forward Euler_ integration. This perspective
motivates the development of _Sheaf Neural Networks_ (Hansen & Ghrist, 2019;
Bodnar et al., 2022; Battiloro et al., 2023) which learn a graph- and task-
specific connection Laplacian to improve expressiveness of GNNs. But given
domain knowledge of the underlying graph (e.g., a graph that approximates a
Riemannian manifold), Barbero et al. (2022) demonstrate that a pre-determined
connection Laplacian could lead to better performance and generalization.
Accordingly, since our method indeed focuses on a specific type of graph,
manifold triangle meshes, we build a connection Laplacian derived from
differential geometry (Sharp et al., 2019). Our architecture with a
deterministic connection Laplacian leads to superior generalization across
triangle meshes, compared to approaches that rely on learning graph-specific
Laplacians.

Despite the existence of several learning approaches for scalar data, few
approaches have been proposed for learning vector fields on surfaces. Previous
methods such as (Dielen et al., 2021) rely on scalar-valued architectures to
output multiple scalar channels that are naively interpreted as vectors. Such
approaches treat each channel independently and thus fail to capture key
invariances (see section 5). This severely hinders generalization to unseen
triangulations and shapes. In contrast, our architecture maintains the vector-
valued features throughout the forward pass, similar to (Deng et al., 2021),
ensuring invariance to isometry, rigid motion, and choice of tangent bases.

##  3 Preliminaries

Figure 2: Our vector heat network is a neural network of complex-valued
neurons (Bassey et al., 2021) with (1) a Vector Heat Diffusion module (see
subsection 4.1) and (2) a vector MLP module (see subsection 4.2). Starting
with a Vector MLP to transform input features from $\mathbb{C}^{n\times
c^{\text{in}}}$ to $\mathbb{C}^{n\times c^{l}}$, our method consists of
several layers of the Vector Heat Diffusion (red) and Vector MLP (blue) with
skip connections, followed by another Vector MLP to map the feature to output
dimensions.

We draw inspiration from differential geometry to define the building blocks
of our architecture, based on the notion of tangent vectors (Sec. 3.1),
parallel transport (Sec. 3.2), and the heat equation (Sec. 3.3), which we
describe next.

###  3.1 Basis Invariant Tangent Vectors

Given a 2-manifold surface $\mathcal{M}$ embedded in $\mathbb{R}^{3}$, the
tangent plane $T_{p}^{M}$ for a given point $p\in\mathcal{M}$ is a 2D space
that is orthogonal to the surface normal of $\mathcal{M}$ at point $p$. A
tangent plane $T_{p}^{M}$ can be defined by the spanning of an _arbitrary
choice_ of two orthogonal basis vectors ${\mathbf{e}}_{0},{\mathbf{e}}_{1}$. A
2D tangent vector ${\mathbf{u}}\in T_{p}^{M}$ can then be expressed as a
linear combination of bases vectors
${\mathbf{u}}=u_{0}{\mathbf{e}}_{0}+u_{1}{\mathbf{e}}_{1}$. For computational
convenience, these tangent vectors ${\mathbf{u}}$ are often represented as a
complex number ${\mathbf{u}}=u_{0}+\imath u_{1}\in\mathbb{C}$ of their
coefficients $u_{0},u_{1}$ (Knöppel et al., 2013; Vaxman et al., 2016). Note
that the coefficients $u_{0},u_{1}$ will change, by a coordinate
transformation, depending on the choice of tangent bases
${\mathbf{e}}_{0},{\mathbf{e}}_{1}$ in order to represent the same tangent
vector. Since the choice of bases is arbitrary, an important invariance for
tangent vector processing is to guarantee that the method is independent of
the choice of bases (see Figure 1).

###  3.2 Parallel Transport of Tangent Vectors

An edge vector
${\mathbf{e}}_{ij}={\mathbf{v}}_{j}-{\mathbf{v}}_{i}\in\mathbb{R}^{3}$ between
two adjacent vertices $i,j$ with locations
${\mathbf{v}}_{i},{\mathbf{v}}_{j}\in\mathbb{R}^{3}$ can be expressed as a
tangent vector in $T_{i}^{M}$ and $T_{j}^{M}$ via the _logarithmic_ map
(sometimes alternately referred to as the _exponential_ map) (Schmidt et al.,
2006). A simple way to compute the logarithmic map is to represent the edge
vector ${\mathbf{e}}_{ij}$ in the polar coordinate $(l_{ij},\theta_{j})$ of
the tangent plane $T_{i}^{M}$, where $l_{ij}=\|{\mathbf{e}}_{ij}\|$ is the
edge length and $\theta_{j}\in[0,2\pi)$ denotes the angular coordinate
(normalized to $2\pi$) of this edge vector from a (arbitrarily chosen) tangent
basis ${\mathbf{e}}_{0}$, see (Knöppel et al., 2013) for more details. As the
edge vector ${\mathbf{e}}_{ij}$ exists in both tangent planes
$T_{i}^{M},T_{j}^{M}$, one can compute the angular difference between
$T_{i}^{M},T_{j}^{M}$ and obtain the coordinate transformation
${\mathbf{r}}_{ij}\in\mathbb{C}$ (a rotation) needed to make sure that
$log_{i}({\mathbf{e}}_{ij})\in T_{i}^{M}$ is mapped to
$log_{j}({\mathbf{e}}_{ij})\in T_{j}^{M}$ (see inset). Transporting tangent
vectors from $T_{i}^{M}$ to $T_{j}^{M}$ following the computed rotation
${\mathbf{r}}_{ij}$ leads to as-parallel-as-possible transport, or _parallel
transport_. Note that only the angle of these vectors varies as we traverse
the curved surface; their length remains unchanged. This motivates
constructing a network that can preserve these disentangled vector properties.

###  3.3 Vector Heat Diffusion

The heat equation on a tangent vector field
$u:\mathcal{M}\rightarrow\mathbb{C}$ can be written as

$\displaystyle\frac{d}{dt}u=\Delta_{c}u.$ (1)

where $\Delta_{c}$ is the _connection Laplacian_. This vector heat equation
characterizes how a “vector” gets diffused in space. Intuitively, diffusing a
single vector heat source will smear it out to its neighborhood with smaller
magnitude while maintaining its direction to be as-parallel-as-possible (see
Figure 3). This is different from classic heat diffusion where no notion of
parallel transport is captured and can cause inconsistent vector directions if
one naively applies scalar heat diffusion to each vector field channel
independently.

#### Discretization

Several previous works (Knöppel et al., 2015; Sharp et al., 2019; Stein et
al., 2020) have defined the vector heat equation on a triangle mesh. The main
ingredient is a _discrete_ connection Laplacian
${\mathbf{L}}_{c}\in\mathbb{C}^{n\times n}$ which is an $n$ x $n$ complex-
valued matrix, where $n$ denotes the number of vertices in the mesh. One way
to build ${\mathbf{L}}_{c}$ is by accumulating a 3-by-3 complex matrix

$\displaystyle-\frac{1}{2}\begin{bmatrix}c_{j}+c_{k}&-c_{k}{\mathbf{r}}_{ij}&-c_{j}{\mathbf{r}}_{ik}\\\
-c_{k}{\mathbf{r}}_{ji}&c_{k}+c_{i}&-c_{i}{\mathbf{r}}_{jk}\\\
-c_{j}{\mathbf{r}}_{ki}&-c_{i}{\mathbf{r}}_{kj}&c_{i}+c_{j}\end{bmatrix}$ (2)

associated to each triangle $ijk$ into the corresponding entries defined by
the vertex index. We use
$c_{i}=\cot\theta^{jk}_{i},c_{j}=\cot\theta^{ki}_{j},c_{k}=\cot\theta^{ij}_{k}$
to shorten the expression, and ${\mathbf{r}}_{ij}\in\mathbb{C}$ (so as
${\mathbf{r}}_{jk},{\mathbf{r}}_{ki}$) to denote a the 2D rotation
(represented as a unit complex number) that _parallel transports_ a tangent
vector from the tangent plane $T_{i}^{M}$ at vertex $i$ to the tangent plane
$T_{j}^{M}$ at vertex $j$ (see subsection 3.2).

As the _forward Euler_ method is well-known to be unstable under large time
steps, we compute the numerical solution to the vector heat equation using the
_implicit Euler_ method. Specifically, a single step of the diffusion is
defined as

$\displaystyle{\mathbf{u}}_{t+1}=({\mathbf{M}}+s{\mathbf{L}}_{c})^{-1}{\mathbf{M}}{\mathbf{u}}_{t}$
(3)

where $s$ to denotes the time-step size, ${\mathbf{M}}\in\mathbb{C}^{n\times
n}$ is a $n$ x $n$ diagonal mass matrix whose entries are complex-valued, with
vertex area as the real component, and zero imaginary component. Equation 3
diffuses a vector into smaller magnitudes and as-parallel-as-possible
orientations (see Figure 3).

Figure 3: The vector heat diffusion process presented in Equation 3 smears out
a tangent vector field ${\mathbf{u}}_{t}$ to its neighbors to obtain another
tangent vector field ${\mathbf{u}}_{t+1}$.

###  3.4 Generalization to N-Rosy Fields

In several applications, one may want to learn a _N-way rotational symmetry_
fields (Palacios & Zhang, 2007) that are invariant under rotation of an
integer multiple of $\nicefrac{{2\pi}}{{N}}$ (see Figure 4). For instance, one
of our applications in quadrilateral meshing requires to output a 4-Rosy
fields in order to mesh a surface with the “cross” pattern on most vertices.

The generalization to N-Rosy fields is straightforward given the complex
number representation (subsection 3.1). Since multiplication with (unit)
complex numbers represents rotations, raising a complex number to the power of
$N$ factors out all the N-ways rotational symmetry (de Goes et al., 2016).
Thus, to measure the difference between, e.g., 4-Rosy fields, one simply
measures the difference between ${\mathbf{u}}^{4}$ (see section 6).

Figure 4: N-Rosy fields refer to tangent vectors that are _N-way rotationally
symmetric_ (Palacios & Zhang, 2007). For instance, $N=1$ refers to the usual
2D tangent vector, $N=2$ to a straight line, and $N=4$ to a “cross” field.

##  4 Vector Heat Network

Our vector-valued neural network for processing tangent vector fields utilizes
vector-valued neurons (similar to (Deng et al., 2021; Bassey et al., 2021))
with vector operations (e.g., parallel transport and the vector heat
equation). Maintaining the vector nature of our data throughout results in an
architecture that is invariant to isometries, rigid transformations, and the
choice of tangent bases (see section 5)

Figure 5: Our architecture is invariant to rigid transformation. A model
trained on a mesh at one orientation (1st) generalizes to its rigidly
transformed counterpart (2nd), outputting a tangent vector field with no error
(3rd). This differs from the baseline method of Dielen et al. (2021), which
outputs a different vector field (4th) with high error (5th).

The input to our network is a set of tangent vectors defined on the vertices
of a surface triangle mesh. A Vector Heat Network consists of several blocks
of learned heat diffusion to harness local information, and _vector MLPs_ to
increase the expressiveness (see Figure 2). The output is a set of tangent
vector fields defined on vertices. These output fields could be regular
tangent vectors or tangent vectors with $N$ rotational symmetry (see
subsection 3.4) defined on the tangent plane. In this section, we will
illustrate individual components of our Vector Heat Network in more details.

###  4.1 Learned Vector Heat Diffusion

Starting with a set of tangent vectors ${\mathbf{u}}\in\mathbb{C}^{n\times
c^{l}}$ (represented as complex numbers, see subsection 3.1), where $n$
denotes the number of vertices and $c^{l}$ denotes the number of tangent
vector fields. We harness information from local neighborhoods by solving the
vector heat equation with the implicit Euler integration in Equation 3,
inspired by (Sharp et al., 2022). As the implicit Euler requires an expensive
step of solving a linear system, we utilize spectral acceleration (Donati et
al., 2022) to speed up the process. Specifically, given the solution of the
generalized eigenvalue problem of the discrete connection Laplacian
${\mathbf{L}}_{c}$ (see Equation 2),

$\displaystyle{\mathbf{L}}_{c}\Phi={\mathbf{M}}\Phi\Sigma$ (4)

where $\Phi=\\{\Phi_{i}\\}\in\mathbb{C}^{n\times k}$ is the stack of $k$
eigenvectors with the lowest frequencies and
$\Sigma=\textit{diag}(\lambda_{i})\in\mathbb{C}^{k\times k}$ is a diagonal
matrix with corresponding eigenvalues. We use ${\mathbf{M}}$ to denote the
mass matrix of vertex areas such that $\Phi_{i}^{\top}{\mathbf{M}}\Phi_{i}=1$.
$k$ is a user-defined number to specify how many eigenvalues/eigenvectors are
in use. Then the vector diffusion process in Equation 3 can be approximated
with

$\displaystyle{\mathbf{u}}_{t+1}=\Phi\begin{bmatrix}e^{-\lambda_{1}s}\\\
e^{-\lambda_{2}s}\\\ \vdots\\\
e^{-\lambda_{k}s}\end{bmatrix}\odot(\Phi^{\top}{\mathbf{M}}{\mathbf{u}}_{t})$
(5)

where $\odot$ denotes element-wise multiplication. Such a spectral
acceleration replaces linear solves with matrix multiplications, thus is
significantly faster for small $k$. In our implementation, we set $k=128$.

Inspired by (Sharp et al., 2022), we treat time-step size $s$ in Equation 5 as
trainable parameters. Intuitively, the network learns whether to diffuse the
vectors over a small or large local neighborhood. Specifically, each _Vector
Heat Diffusion_ module (Figure 2) consists of $m$ trainable time steps. Each
time step $s_{i}$ will diffuse the input feature
${\mathbf{X}}^{l}\in\mathbb{C}^{n\times c^{l}}$ at layer $l$ into a set of
diffused features ${\mathbf{Y}}^{l}_{i}\in\mathbb{C}^{n\times c^{l}}$ via
Equation 5. Thus, a collection of $m$ time steps $[s_{1},\cdots,s_{m}]$ will
turn an input feature ${\mathbf{X}}^{l}$ with size $n$ x $c^{l}$ into a set of
diffused features
${\mathbf{Y}}^{l}=[{\mathbf{Y}}^{l}_{1},\cdots,{\mathbf{Y}}^{l}_{m}]$ with
size $n$ x $m_{c^{l}}$.

###  4.2 Vector Linear Layers and Non-linearity

After the vector heat diffusion module, we use a vector-valued MLP that
consists of a per-vertex linear layer

$\displaystyle{\mathbf{Z}}^{l}={\mathbf{Y}}^{l}{\mathbf{W}}^{l}$ (6)

where ${\mathbf{W}}^{l}\in\mathbb{R}^{mc^{l}\times c^{l+1}}$ is a matrix of
size $mc^{l}$-by-$c^{l+1}$ that linearly combine the complex-valued features
at each vertex into ${\mathbf{Z}}^{l}\in\mathbb{C}^{n\times c^{l+1}}$. Then we
follow the idea presented by (Wiersma et al., 2022) to apply non-linearities
$\sigma$ (e.g., ReLU) on the magnitude of each complex feature as

$\displaystyle{\mathbf{X}}^{l+1}_{ij}=\sigma(\|{\mathbf{Z}}^{l}_{ij}\|-b^{l}_{j})\cdot\frac{{\mathbf{Z}}^{l}_{ij}}{\|{\mathbf{Z}}^{l}_{ij}\|}$
(7)

where we use ${\mathbf{Z}}^{l}_{ij}$ to denote the entry corresponding to the
$i$th row and the $j$th column in ${\mathbf{Z}}$, $b^{l}_{j}\in\mathbb{R}$ is
a bias term for each channel added to the feature norm. In our experiments, we
use the ReLU activation – if the complex feature norm
$\|{\mathbf{Z}}^{l}_{ij}\|$ is smaller than the bias $b^{l}_{j}$, the complex
feature is set to $\mathbf{0}$, otherwise it is unchanged in the output
${\mathbf{X}}^{l+1}$.

In summary, our overall architecture (see Figure 2) consists of several Vector
Diffusion Blocks. Each block contains a vector heat diffusion layer (see
subsection 4.1) and a vector-valued MLP with two hidden layers (see subsection
4.2). We also have two extra vector MLPs (green, yellow) to adapt to a
different number of input/output channels. Our architecture is invariant to
choice of tangent bases due to the Vector Heat Diffusion module baking in
parallel transport (see subsection 3.2). Furthermore, the vector heat
diffusion process is _intrinsic_ , ensuring the architecture is also invariant
to rigid transformations and isometries of the underlying shape. These
desirable invariances result in a general network architecture for processing
tangent vector fields on manifolds.

##  5 Experiment: Invariance Properties

Our architecture possesses several fundamental invariances, which we highlight
and empirically validate here, distinguishing it from the scalar-valued
approach of (Dielen et al., 2021). For comparison, we faithfully re-implement
their method (section 5). These invariances arise from the fact that all of
our operations (gradient, heat diffusion, and the per-vertex linear layer) are
_intrinsic_ , which implies that our architecture is invariant to how the mesh
sits in the space.

#### Invariance to Rigid Motion

If rigid motion invariant input features are used, then our method will be
invariant to rigid transformations of the underlying shape (see Figure 5).

#### Invariance to Tangent Bases

We leverage the characteristic that parallel transport has already factored
out the influence of choice of tangent bases, which is baked in to our
connection Laplacian. This property makes our architecture invariant to the
choice of tangent bases (Figure 6).

Figure 6: Our method is invariant to choice of tangent bases. Given a model
trained on the default bases (light green), where the arrow indicates
direction of e0, our method produces the same result (blue) as GT (red) even
when the model is evaluated under different choice of bases (dark green), in
contrast to (Dielen et al., 2021) that outputs a different result with high
error (right).

#### Invariance to Isometry

Our method is invariant to isometric deformation (isometries) of the input
(see Figure 7), due to the intrinsic construction of our method.

Figure 7: Our method is invariant to isometric deformation. We train on a flat
paper (first) and evaluate on its crumpled counterpart (second). Since these
two meshes are nearly isometric (third), our method produces consistent
results (fourth).

#### Robustness to Discretizations

Our main ingredient for “message passing” relies on the vector heat diffusion
with the connection Laplacian. In contrast to the method by Bodnar et al.
(2022) that learns the connection Laplacian for a graph, the connection
Laplacian in our set-up is _deterministic_ by the underlying shape, with
_connections_ determined by the parallel transport (subsection 3.2). This
characteristic ensures that our learned parameters are generalizable to meshes
with different connectivities. This property is crucial to tangent vector
field processing because the choice of tangent bases is arbitrary (see Figure
1): one can find an infinite number of _valid_ tangent bases
${\mathbf{e}}_{0},{\mathbf{e}}_{1}$ that are orthogonal to the normal vector.
On a single vertex, there is already an infinite number of choices and the
total combinations of basis choices also grows exponentially with the number
of vertices in the mesh. This implies that baking in the property of basis
invarance is important because solving it with data augmentation is
intractable due to the infinite number of basis combinations.

Figure 8: This work exploits the benefit of the spectral method (see Equation
5) and gains robustness to different discretizations. Trained on one
triangulation (1st), our model generalizes to a different one (3rd), producing
consistent output (quad meshing results from section 6) shown here.)

#### Baseline Implementation Details

Our re-implementation of the architecture by Dielen et al. (2021) is as
consistent as possible with theirs: for the local feature network ((Gong et
al., 2019)), we use a spiral sequence length of $k=20$ vertices, with $4$
spiral convolution layers of intermediate size $[16,256,512,1024]$, where the
first 3 layers use vertex-centric spiral indices, and the last layer uses
face-centric spiral indices; the global feature ((Qi et al., 2017)) consists
of $1024$ channels; and each input vertex is represented as its 3D position
and normal direction.

##  6 Experiment: Quadrilateral Remeshing

In this section, we evaluate our method on triangle meshes, though the same
principles apply to other domains where the connection Laplacian is available
(e.g. point clouds).

###  6.1 Experiment Setup

To demonstrate our method’s effectiveness, we evaluate it on the task of
quadrilateral remeshing. Given a triangle mesh $\mathcal{M}$, we use the per-
channel gradient of the first $c^{\text{in}}=15$ channels of the Heat Kernel
Signature (HKS) (Sun et al., 2009) as input features to our network, giving
$c^{\text{in}}$ vector-valued features per vertex. The output is a 4-Rosy
cross field defined on each vertex (see subsection 3.4 and Figure 9). We then
interpolate the per-vertex cross field onto faces (see subsection 6.1),
followed by off-the-shelf algorithms by Bommes et al. (2009) and (Ebke et al.,
2013) to turn the per-face cross field into a quadrilateral mesh (see Figure
9).

Figure 9: After obtaining the output cross field by our network, we use off-
the-shelf quad mesh extraction method by Bommes et al. (2009) to obtain a
quadrilateral mesh with edges aligned with the predicted cross field.

#### Dataset

We train our network on a dataset generated from the workflow described in
(Dielen et al., 2021), with two modifications: (1) Instead of the DFAUST
dataset used by (Dielen et al., 2021), we assemble a custom library of artist-
created template avatar heads, around which we wrap the SMPL (Loper et al.,
2023) head topology; (2) for each of the template meshes, we create
$100-1,000$ augmentations/variations, using a custom tool for deforming faces,
based upon normal-driven ARAP deformation (Liu & Jacobson, 2021). This expands
the training data distribution away from the parametric SMPL model. The
training dataset consists of $1100$ triangle meshes with associated ground
truth vector fields. The test dataset consists of $115$ samples.

#### Loss Function

Since our output is a 4-Rosy field, we define the loss function as the mean
squared error (MSE) on the output tangent vector field ${\mathbf{u}}^{4}$
raised to the power of 4 (see subsection 3.4). In addition to measuring errors
on the directions, we also want to measure errors on the magnitude of the
cross field (smaller crosses lead to smaller polygon). Combining the two leads
to our following loss function

$\displaystyle\mathcal{L}({\mathbf{u}},\hat{{\mathbf{u}}})=$ (8)
$\displaystyle\sum_{i=1}^{N}\frac{\textbf{M}_{ii}}{A}\Bigg{[}\underbrace{\bigg{|}\frac{\|u_{i}\|-\|\hat{u_{i}}\|}{\|u_{i}\|}\bigg{|}}_{\text{magnitude}}+\underbrace{\bigg{(}1-\frac{u_{i}^{4}}{\|u_{i}^{4}\|}\cdot\frac{\hat{u_{i}}^{4}}{\|\hat{u_{i}}^{4}\|}\bigg{)}}_{\text{direction}}\Bigg{]}$

where ${\mathbf{M}}_{ii}$ denotes the vertex area at vertex $i$,
$A=\operatorname{Tr}({\mathbf{M}})$ is the total area of the mesh,
$u_{i},\hat{u_{i}}\in\mathbb{C}$ denote the output and the ground truth
tangent vector fields on the vertex $i$, respectively. The first “magnitude”
term simply measure the relative difference in magnitude between the output
and the ground truth fields. The second “direction” term measures their
angular difference with the cosine similarity.

#### Transporting Vectors from Vertices to Faces

Our model predicts vectors at mesh vertices, but in order to directly leverage
existing methods for field-guided quadrilateral remeshing (Bommes et al.,
2009), they should be expressed on the face tangent planes. Naively averaging
the three vector predictions from a given face’s three incident vertices will
not produce a correct result, as the vectors are expressed with respect to
their individual vector tangent planes, so they cannot be averaged directly.
We must therefore account for the parallel transport from each of the vertex
tangent planes to the face tangent plane. Once all three vectors are in a
shared frame, then we may simply average their values.

To transport a vector from its tangent plane $T_{i}\mathcal{M}$ at a vertex
$v_{i}$ to the tangent plane $T_{ijk}\mathcal{M}$ at face $f_{ijk}$, we choose
an edge that is incident to both elements (e.g., edge $e_{ij}$ or $e_{ik}$).
The chosen edge can be expressed in terms of its angular rotation from the
local basis in both the vertex $T_{i}\mathcal{M}$ and face tangent planes
$T_{ijk}\mathcal{M}$. Thus, we leverage this angular difference to transport a
vector from vertex to face tangent plane. This transport can be constructed as
an operator in a pre-processing step, as it only depends on the mesh, i.e.
once we have computed the angular differences, they may be used to transport
any vectors from vertices to faces. Let this matrix be
$\textbf{T}_{angle}\in\mathbb{R}^{F\times 3}$, where each row corresponds to a
face, and each of the three values corresponds to the transition angle (in
radians) needed to transport a vector from the incident vertex at that index,
to the face. Then we can assemble the complex matrix _operator_ :

$\textbf{T}=e^{\imath\textbf{T}_{angle}}$ (9)

To compute the transported, averaged result per face,

$\hat{\textbf{u}}_{f}=\frac{1}{3}\sum\textbf{T}\odot\hat{\textbf{u}}_{v}[\textbf{F}]$
(10)

where the $[\cdot]$ represents an indexed selection from the per-vertex
predictions based on vertex indices from the face matrix F, and the sum
$\sum(\cdot)$ denotes a per-face addition along the last dimension of the
product $\textbf{T}\odot\hat{\textbf{u}}_{v}[\textbf{F}]$.

Figure 10: Vector-field-guided quadrilateral meshing results of various
character heads from the test dataset. Figure 11: Our learned vector-field-
guided quadrilateral remeshing can generalize to other datasets and classes of
objects.

#### Implementation Details

For our experiments, we use $N=6$ vector diffusion blocks with a hidden
dimension of $c^{l}=256$ channels (see Figure 2). We train for $3,000$ epochs,
with initial learning rate of $1e-4$, decayed by a factor of $0.85$ every
$150$ epochs. In the Vector MLP layer, we use Dropout (Srivastava et al.,
2014) set to $0.5$, and L2 regularization (weight decay) with a value of
$1e-3$, which mitigates overfitting. We train on a single NVIDIA Tesla T4 GPU,
for about $20$ hours. For computing a parametrization from the predicted
vector field, we rely on (Bommes et al., 2009), rather than (Campen et al.,
2015) used by (Dielen et al., 2021), due to the latter not being open sourced.

###  6.2 Results

#### Character Heads

In Figure 10, we display several results of our learned vector field-guided
quadrilateral meshing on the character head dataset. Our model generalizes
across different types of character heads.

#### Animals

Figure 11 shows results on a different dataset that we constructed from 40
ground truth quadrilateral meshes of animals. The trained model generalizes to
_centaur_ with its human-like upper body and horse-like lower body, even
though such a combination is not contained in the training dataset. The
_triceratops_ result generally looks suitable for downstream usage, though its
frill (collar) shows some irregularity, likely due to lack of sufficient
training data.

Figure 12: We supervise our method with an optimization-based quad meshing
algorithm – Quadwild by [Pietroni et al. 2021], faithfully reproducing their
result.

#### Additional experimental results

Our method can faithfully reproduce the results from existing quad meshing
algorithms (Figure 12), such as Quadwild (Pietroni et al., 2021). We also note
that our method imposes no restriction on the genus of the shape on which
vector fields may be learned, as demonstrated by successful generalization to
genus-one shapes in Figure 6 and Figure 12. We also show that our method
easily scales to high resolution meshes (Figure 13). Please see Appendix A for
enlarged detail views.

#### Ablation Studies

We compare the test set performance of various types of input features,
including _principal curvature directions_ , and the gradients of: _Heat
Kernel Signature_ (Sun et al., 2009), _Gaussian curvature_ , and _mean
curvature_. For multi-channeled feature types, we normalize each of its
channels individually, such that its mean vector length across all vertices is
unit. For all feature types _except PCD_ , we also rotate each channel by
$\frac{\pi}{2}$ radians, and concatenate these rotated vector features along
the channel dimension. In principle, this means that each input feature
channel and its rotated counterpart span the local tangent space, allowing the
network to better exploit all degrees of freedom.

Figure 13: Our method is scalable to high resolution input meshes. We demonstrate a quad meshing result on this octopus mesh with over 60k vertices. Input Feature Type | Direction Loss | Magnitude Loss
---|---|---
$\nabla$HKS | $0.106\pm 0.278$ | $\mathbf{0.077\pm 0.148}$
$\nabla$GC | $0.139\pm 0.312$ | $0.096\pm 0.261$
$\nabla$MC | $\mathbf{0.105\pm 0.276}$ | $0.077\pm 0.514$
PCD | $0.128\pm 0.313$ | $0.090\pm 0.288$
Table 1: Input features are evaluated by comparing their mean test loss and
associated variance of their magnitude and direction components. $\nabla$_HKS_
denotes channel-wise gradient of the scalar-valued _Heat Kernel Signature_ ,
$\nabla$_GC_ denotes gradient of _Gaussian Curvature_ , $\nabla$_MC_ denotes
gradient of _Mean Curvature_ , and _PCD_ denotes scaled principal curvature
directions. We find that $\nabla$_HKS_ leads to best overall performance.
While $\nabla$_MC_ performs best on the directional loss component only, it
displays high variance in the magnitude loss component.

##  7 Conclusion

We present a neural network architecture based on vector heat diffusion to
process tangent vector fields defined on manifold surfaces. Unlike existing
works, our method is invariant to rigid transformations and the choice of
tangent plane bases, and is robust to different triangulations. These
properties jointly make this network a generalizable architecture for learning
tangent vector fields across surfaces.

#### Additional Applications

We show an application of our method to robot path planning on curved terrain
(Figure 14).

Figure 14: Tangent vector fields are commonly used to guide a robot’s path
planning [Patil et al. 2010]. We train our method to generate such a
navigation vector field on the terrain mesh to assist motion planing as shown.

Future Directions As the vector heat equation can be defined in different
domains, such as point clouds and graphs, generalizing our architecture to
different domains could enable a even wider range of applications. Exploring
novel architectures from _complex neural networks_ (Bassey et al., 2021) or
even generalizing to quaternions, such as (Zhu et al., 2018), could lead to
novel variants of our architecture. For the quadrilateral meshing application,
developing a larger, more diverse dataset could be an important step towards a
generic learning-based remesher.

## Acknowledgements

Thank you to Jing Liang for help executing the robot motion planning
application in ROS, Jihyun Yoon for help assembling the 3D heads dataset, and
Kiran Bhat for helpful discussion. This research is supported in part by Barry
Mersky Endowed Professorship, Capital One Endowed Professorship, and Maryland
E-Nnovate Initiative Fund.

## Impact Statement

The goal of this work is to advance the field of Machine Learning. There are
many potential societal consequences of our work, none which we feel must be
specifically highlighted here.

## References

  * Barbero et al. (2022) Barbero, F., Bodnar, C., de Ocáriz Borde, H. S., Bronstein, M. M., Velickovic, P., and Liò, P.  Sheaf neural networks with connection laplacians.  In _Topological, Algebraic and Geometric Learning Workshops 2022, 25-22 July 2022, Virtual_ , volume 196 of _Proceedings of Machine Learning Research_ , pp. 28–36. PMLR, 2022.
  * Bassey et al. (2021) Bassey, J., Qian, L., and Li, X.  A survey of complex-valued neural networks.  _CoRR_ , abs/2101.12249, 2021.  URL https://arxiv.org/abs/2101.12249.
  * Battiloro et al. (2023) Battiloro, C., Wang, Z., Riess, H., Lorenzo, P. D., and Ribeiro, A.  Tangent bundle filters and neural networks: From manifolds to cellular sheaves and back.  In _IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4-10, 2023_ , pp. 1–5. IEEE, 2023.
  * Bodnar et al. (2022) Bodnar, C., Giovanni, F. D., Chamberlain, B. P., Lió, P., and Bronstein, M. M.  Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in gnns.  In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), _Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022_ , 2022.
  * Bommes et al. (2009) Bommes, D., Zimmer, H., and Kobbelt, L.  Mixed-integer quadrangulation.  _ACM Transactions On Graphics (TOG)_ , 28(3):1–10, 2009.
  * Bronstein et al. (2021) Bronstein, M. M., Bruna, J., Cohen, T., and Velickovic, P.  Geometric deep learning: Grids, groups, graphs, geodesics, and gauges.  _CoRR_ , abs/2104.13478, 2021.  URL https://arxiv.org/abs/2104.13478.
  * Campen et al. (2015) Campen, M., Bommes, D., and Kobbelt, L.  Quantized global parametrization.  _Acm Transactions On Graphics (tog)_ , 34(6):1–12, 2015.
  * Cohen et al. (2019) Cohen, T., Weiler, M., Kicanaoglu, B., and Welling, M.  Gauge equivariant convolutional networks and the icosahedral CNN.  In Chaudhuri, K. and Salakhutdinov, R. (eds.), _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , pp. 1321–1330. PMLR, 2019.
  * de Goes et al. (2016) de Goes, F., Desbrun, M., and Tong, Y.  Vector field processing on triangle meshes.  In _Special Interest Group on Computer Graphics and Interactive Techniques Conference, SIGGRAPH ’16, Anaheim, CA, USA, July 24-28, 2016, Courses_ , pp. 27:1–27:49. ACM, 2016.
  * Deng et al. (2021) Deng, C., Litany, O., Duan, Y., Poulenard, A., Tagliasacchi, A., and Guibas, L. J.  Vector neurons: A general framework for so(3)-equivariant networks.  In _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_ , pp. 12180–12189. IEEE, 2021.
  * Dielen et al. (2021) Dielen, A., Lim, I., Lyon, M., and Kobbelt, L.  Learning direction fields for quad mesh generation.  In _Computer Graphics Forum_ , volume 40, pp. 181–191. Wiley Online Library, 2021.
  * Donati et al. (2022) Donati, N., Corman, E., Melzi, S., and Ovsjanikov, M.  Complex functional maps: A conformal link between tangent bundles.  _Comput. Graph. Forum_ , 41(1):317–334, 2022.
  * Ebke et al. (2013) Ebke, H.-C., Bommes, D., Campen, M., and Kobbelt, L.  Qex: Robust quad mesh extraction.  _ACM Transactions on Graphics (TOG)_ , 32(6):1–10, 2013.
  * Feng et al. (2019) Feng, Y., Feng, Y., You, H., Zhao, X., and Gao, Y.  Meshnet: Mesh neural network for 3d shape representation.  In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019_ , pp. 8279–8286. AAAI Press, 2019.
  * Gong et al. (2019) Gong, S., Chen, L., Bronstein, M. M., and Zafeiriou, S.  Spiralnet++: A fast and highly efficient mesh convolution operator.  In _2019 IEEE/CVF International Conference on Computer Vision Workshops, ICCV Workshops 2019, Seoul, Korea (South), October 27-28, 2019_ , pp. 4141–4148. IEEE, 2019.
  * Guo et al. (2021) Guo, Y., Wang, H., Hu, Q., Liu, H., Liu, L., and Bennamoun, M.  Deep learning for 3d point clouds: A survey.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 43(12):4338–4364, 2021.
  * Hanocka et al. (2019) Hanocka, R., Hertz, A., Fish, N., Giryes, R., Fleishman, S., and Cohen-Or, D.  Meshcnn: a network with an edge.  _ACM Transactions on Graphics (ToG)_ , 38(4):1–12, 2019.
  * Hansen & Gebhart (2020) Hansen, J. and Gebhart, T.  Sheaf neural networks.  _CoRR_ , abs/2012.06333, 2020.  URL https://arxiv.org/abs/2012.06333.
  * Hansen & Ghrist (2019) Hansen, J. and Ghrist, R.  Learning sheaf laplacians from smooth signals.  In _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019_ , pp. 5446–5450. IEEE, 2019.
  * Hertz et al. (2020) Hertz, A., Hanocka, R., Giryes, R., and Cohen-Or, D.  Deep geometric texture synthesis.  _ACM Trans. Graph._ , 39(4):108, 2020.
  * Hu et al. (2022) Hu, S., Liu, Z., Guo, M., Cai, J., Huang, J., Mu, T., and Martin, R. R.  Subdivision-based mesh convolution networks.  _ACM Trans. Graph._ , 41(3):25:1–25:16, 2022.
  * Knöppel et al. (2013) Knöppel, F., Crane, K., Pinkall, U., and Schröder, P.  Globally optimal direction fields.  _ACM Trans. Graph._ , 32(4):59:1–59:10, 2013.
  * Knöppel et al. (2013) Knöppel, F., Crane, K., Pinkall, U., and Schröder, P.  Globally optimal direction fields.  _ACM Transactions on Graphics (ToG)_ , 32(4):1–10, 2013.
  * Knöppel et al. (2015) Knöppel, F., Crane, K., Pinkall, U., and Schröder, P.  Stripe patterns on surfaces.  _ACM Trans. Graph._ , 34(4):39:1–39:11, 2015.
  * Lahav & Tal (2020) Lahav, A. and Tal, A.  Meshwalker: deep mesh understanding by random walks.  _ACM Trans. Graph._ , 39(6):263:1–263:13, 2020.
  * Liu & Jacobson (2021) Liu, H. D. and Jacobson, A.  Normal-driven spherical shape analogies.  _Comput. Graph. Forum_ , 40(5):45–55, 2021.
  * Liu et al. (2020) Liu, H. D., Kim, V. G., Chaudhuri, S., Aigerman, N., and Jacobson, A.  Neural subdivision.  _ACM Trans. Graph._ , 39(4):124, 2020.
  * Loper et al. (2023) Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., and Black, M. J.  Smpl: A skinned multi-person linear model.  In _Seminal Graphics Papers: Pushing the Boundaries, Volume 2_ , pp. 851–866. 2023.
  * Ludwig et al. (2023) Ludwig, I., Tyson, D., and Campen, M.  Halfedgecnn for native and flexible deep learning on triangle meshes.  _Computer Graphics Forum_ , 42(5):e14898, 2023.
  * Milano et al. (2020) Milano, F., Loquercio, A., Rosinol, A., Scaramuzza, D., and Carlone, L.  Primal-dual mesh convolutional neural networks.  In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , 2020.
  * Palacios & Zhang (2007) Palacios, J. and Zhang, E.  Rotational symmetry field design on surfaces.  _ACM Trans. Graph._ , 26(3):55, 2007.
  * Pfaff et al. (2021) Pfaff, T., Fortunato, M., Sanchez-Gonzalez, A., and Battaglia, P. W.  Learning mesh-based simulation with graph networks.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.
  * Pietroni et al. (2021) Pietroni, N., Nuvoli, S., Alderighi, T., Cignoni, P., Tarini, M., et al.  Reliable feature-line driven quad-remeshing.  _ACM Transactions on Graphics_ , 40(4):1–17, 2021.
  * Qi et al. (2017) Qi, C. R., Su, H., Mo, K., and Guibas, L. J.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 652–660, 2017.
  * Sabaka et al. (2010) Sabaka, T. J., Hulot, G., and Olsen, N.  Mathematical properties relevant to geomagnetic field modeling.  In _Handbook of geomathematics_. 2010.
  * Schmidt et al. (2006) Schmidt, R. M., Grimm, C., and Wyvill, B.  Interactive decal compositing with discrete exponential maps.  _ACM Trans. Graph._ , 25(3):605–613, 2006.
  * Sharp et al. (2019) Sharp, N., Soliman, Y., and Crane, K.  The vector heat method.  _ACM Transactions on Graphics (TOG)_ , 38(3):1–19, 2019.
  * Sharp et al. (2022) Sharp, N., Attaiki, S., Crane, K., and Ovsjanikov, M.  Diffusionnet: Discretization agnostic learning on surfaces.  _ACM Transactions on Graphics (TOG)_ , 41(3):1–16, 2022.
  * Srivastava et al. (2014) Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 15(1):1929–1958, 2014.
  * Stein et al. (2020) Stein, O., Wardetzky, M., Jacobson, A., and Grinspun, E.  A simple discretization of the vector dirichlet energy.  _Comput. Graph. Forum_ , 39(5):81–92, 2020.
  * Sun et al. (2009) Sun, J., Ovsjanikov, M., and Guibas, L.  A concise and provably informative multi-scale signature based on heat diffusion.  In _Computer graphics forum_ , volume 28, pp. 1383–1392. Wiley Online Library, 2009.
  * van den Berg et al. (2008) van den Berg, J. P., Lin, M. C., and Manocha, D.  Reciprocal velocity obstacles for real-time multi-agent navigation.  In _2008 IEEE International Conference on Robotics and Automation, ICRA 2008, May 19-23, 2008, Pasadena, California, USA_ , pp. 1928–1935. IEEE, 2008.
  * Vaxman et al. (2016) Vaxman, A., Campen, M., Diamanti, O., Bommes, D., Hildebrandt, K., Ben-Chen, M., and Panozzo, D.  Directional field synthesis, design, and processing.  In Mitra, N. J. (ed.), _SIGGRAPH ASIA 2016, Macao, December 5-8, 2016 - Courses_ , pp. 15:1–15:30. ACM, 2016.
  * Wiersma et al. (2022) Wiersma, R., Nasikun, A., Eisemann, E., and Hildebrandt, K.  Deltaconv: anisotropic operators for geometric deep learning on point clouds.  _ACM Transactions on Graphics (TOG)_ , 41(4):1–10, 2022.
  * Wu et al. (2021) Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., and Yu, P. S.  A comprehensive survey on graph neural networks.  _IEEE Trans. Neural Networks Learn. Syst._ , 32(1):4–24, 2021.
  * Xie et al. (2022) Xie, Y., Takikawa, T., Saito, S., Litany, O., Yan, S., Khan, N., Tombari, F., Tompkin, J., Sitzmann, V., and Sridhar, S.  Neural fields in visual computing and beyond.  _Comput. Graph. Forum_ , 41(2):641–676, 2022.
  * Yang et al. (2022) Yang, L., Zhang, Z., Song, Y., Hong, S., Xu, R., Zhao, Y., Shao, Y., Zhang, W., Yang, M., and Cui, B.  Diffusion models: A comprehensive survey of methods and applications.  _CoRR_ , abs/2209.00796, 2022.  doi: 10.48550/ARXIV.2209.00796.  URL https://doi.org/10.48550/arXiv.2209.00796.
  * Yang et al. (2023) Yang, X., Lin, G., Chen, Z., and Zhou, L.  Neural vector fields: Implicit representation by explicit learning.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023_ , pp. 16727–16738. IEEE, 2023.
  * Zhu et al. (2018) Zhu, X., Xu, Y., Xu, H., and Chen, C.  Quaternion convolutional neural networks.  In Ferrari, V., Hebert, M., Sminchisescu, C., and Weiss, Y. (eds.), _Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VIII_ , volume 11212 of _Lecture Notes in Computer Science_ , pp. 645–661. Springer, 2018.

##  Appendix A Additional experimental results (enlarged detail view)

Our method can faithfully reproduce the results from existing quad meshing
algorithms (Figure 15), such as Quadwild (Pietroni et al., 2021). We also show
that our method easily scales to high resolution meshes (Figure 16).

Figure 15: We supervise our method with an optimization-based quad meshing
algorithm – Quadwild by [Pietroni et al. 2021]. Our method is able to
reproduce the result from Quadwild. Figure 16: Our method is scalable to high
resolution input meshes. We demonstrate a quad meshing result on this octopus
mesh with over 60k vertices.


# A context-aware computational approach for measuring vocal entrainment in
dyadic conversations

###### Abstract

Vocal entrainment is a social adaptation mechanism in human interaction,
knowledge of which can offer useful insights to an individual’s cognitive-
behavioral characteristics. We propose a context-aware approach for measuring
vocal entrainment in dyadic conversations. We use conformers (a combination of
convolutional network and transformer) for capturing both short-term and long-
term conversational context to model entrainment patterns in interactions
across different domains. Specifically we use cross-subject attention layers
to learn intra- as well as interpersonal signals from dyadic conversations. We
first validate the proposed method based on classification experiments to
distinguish between real (consistent) and fake (inconsistent/shuffled)
conversations. Experimental results on interactions involving individuals with
Autism Spectrum Disorder also show evidence of a statistically-significant
association between the introduced entrainment measure and clinical scores
relevant to symptoms, including across gender and age groups.

Index Terms—  entrainment, context, transformers, convolution

##  1 Introduction

Interpersonal human interactions, notably dyadic interactions (interactions
involving two people), are widely studied by social science and human-centered
computing researchers alike [1, 2]. Such interactions are characterized by
rich information exchange across multiple modalities including speech,
language, and visual cues. Over the years, a significant amount of effort has
been invested in developing tools for both conversational data collection and
in understanding and modeling the signals extracted from these interactions.

A phenomenon called entrainment [3, 4] has been described as one of the major
driving forces of an interaction [5]. While entrainment can be exhibited
within and across different modalities, vocal entrainment [6] or acoustic-
prosodic entrainment [7, 4, 8] is defined as an interlocutor’s tendency to
accommodate or adapt to the vocal patterns of the other interlocutor over the
course of the interaction. Understanding entrainment [9] can provide
meaningful insights to analyze behavioral characteristics of the individual
interlocutors and the interaction participants. For example, a higher degree
of entrainment is associated with positive behavioral markers like social
desirability, smoother interactions, higher rapport content etc.[10, 11].
Entrainment can also serve as a valuable instrument to characterize behaviors
in the study and practice of psychiatry and developmental studies involving
distressed couples, children with autism spectrum disorder, addiction, etc [6,
9].

Due to the complex nature of entrainment and a scarcity of appropriately
labeled speech corpora, quantifying entrainment is a challenging task. Most of
the early works have relied on empirical and knowledge-driven tools like
correlation, recurrence analysis, time-series analysis, spectral methods to
measure how much a speaker is entraining to the other speaker [12]. This body
of work often relied on the assumption of a linear relationship between the
extracted entrainment representations and vocal features, which may not always
hold. On the other hand, although context during a conversation plays an
important part in interpersonal interactions, it has not been incorporated in
existing approaches for measuring entrainment. While the recent line of works
[6] employ a more direct data-driven strategy to extract entrainment related
information from raw speech features, such are formulated in a way that they
inherently only consider short-term context while overlooking more long-term
context. Recently context-aware deep learning architectures such as
transformers [13] have been proposed to capture richer contexts by explicitly
modeling the temporal dimension and found many applications in natural
language processing, speech and vision. In light of their success in modeling
rich temporal context, we investigate if transformers can help capture
meaningful information for quantifying entrainment.

In this work, we develop a context-aware model for computing entrainment,
addressing the need for both short and long-range temporal context modeling.
For the scope of this work, the proposed framework incorporates ‘context’ by
aiming to train the model to learn the influence of the speakers on each
other. We follow the established strategy of using a distance-based measure
between consecutive turn-pairs in the projected embedding space and introduce
the Contextual Entrainment Distance (CED) metric. The main contributions of
this work are two fold: first, we use a combination of self-attention and
convolution to extract both short-term and long-term contextual information
related to entrainment; and second, we propose a transformer-based cross-
subject framework for joint modeling of the interacting speakers to learn the
pattern of entrainment. We experimentally evaluate the validity and efficacy
of CED in dyadic conversations involving children and study its association
with respect to different clinically-significant behavioral ratings where the
role of entrainment has been previously implicated [9].

Fig. 1: Architecture for CED extraction.

##  2 Computing context-aware entrainment measure

###  2.1 Unsupervised model training and CED computation

Prior literature in this domain have relied on computing a distance measure
directly between the turn-level speech features $X_{1}$ and $X_{2}$ from
speaker $1$ and speaker $2$ respectively [4]. However, these features also
capture additional information such as speaker characteristics and ambient
acoustic information which do not contribute towards learning the target
entrainment patterns. The objective is to learn the inverse mapping between
the embedding space ($Z_{1}$,$Z_{2}$) and the feature space ($X_{1}$,$X_{2}$)
such that the model should learn to recognize turn pairs with high and low
levels of entrainment.

Here, we formulate the problem by training the network to classify between
interactions having consecutive turn segments (true samples) and interactions
having random/shuffled turn segments (fake samples). We temporally partition
the conversational audio sequence into speaker specific chunks and feed these
chunks to the model to predict whether the fed audio chunks are part of real
conversation or a fake one.

After the training phase, we use the trained network weights to extract the
cross-encoder layer outputs for both speakers. Next, we calculate CED as the
smooth L1 distance [6] between the embeddings obtained in the previous step.

###  2.2 Model architecture

As shown in Fig. 1, we use two main modules to build the model to compute
entrainment, first, the self-attention encoder that is used to enhance the
extracted features by attending to themselves and then, a cross-attention
encoder which allows the features to attend to a different source.

We use conformer [14] layers for the self-attention module to model both
short-term and long-term dependencies within an audio sequence in a parameter-
efficient way by incorporating a convolutional module in the transformer
layer. The self-attention layer obtains meaningful representation from the
long-term interaction and the convolution layer is used to learn the local
relation amongst the interaction based features.

To extract meaningful information related to entrainment, previous works have
mostly relied on individual modeling of interlocutors involved in a
conversation. However, entrainment being an interpersonal phenomenon, the need
for jointly modeling interlocutors becomes heightened in such scenarios. We
address this issue by using a transformer layer for cross-subject attention,
allowing the features extracted per subject to access each other to capture
crossed influence over the interaction.

##  3 Experiments

###  3.1 Datasets

We use the following two datasets for our experiments.

The Fisher Corpus English Part 1 (LDC2004S13) [15] consists of spontaneous
telephonic conversations between two native English speaking subjects. There
are 5850 conversations of approximately 10 minutes duration. The dataset is
accompanied with transcripts along with timestamps marking speaker duration
boundaries. We use 60% of this dataset for training and 5% for testing.

The ADOSmod3 corpus consists of recorded conversations from autism diagnostic
sessions between a child and a clinician who is trained to observe the
behavioral traits of the child related to Autism Spectrum Disorder (ASD). A
typical interactive session following the Autism Diagnostic Observation
Schedule (ADOS)-2 instrument lasts about 40-60 minutes, and these sessions are
composed of a variety of subtasks to evoke spontaneous response from the
children under different social and communicative circumstances. In this work,
we consider the administration of Module 3 meant for verbally fluent children
and adolescents. Moreover, we focus on Emotions and Social difficulties and
annoyance subtasks as these are expected to extract significant spontaneous
speech and reaction from the child while answering questions about different
emotions and social difficulties. The corpus consists of recordings from 165
children collected across 2 different clinical sites. We use this corpus for
evaluation purpose, the demographic details of the dataset are reported in
Table 1.

Table 1: Demographic details of ADOSMod3 dataset

Category | Statistics
---|---
Age(years) | Range: 3.58-13.17 (mean,std):(8.61,2.49)
Gender | 123 male, 42 female
Non-verbal IQ | Range: 47-141 (mean,std):(96.01,18.79)
Clinical Diagnosis | 86 ASD,42 ADHD
14 mood/anxiety disorder
12 language disorder
10 intellectual disability, 1 no diagnosis
Age distribution | Cincinnati: $\leq$5yrs 7, 5-10 yrs 52, $\geq$10yrs 25
Michigan: $\leq$5yrs 11, 5-10 yrs 42, $\geq$10yrs 28

###  3.2 Experimental setup

####  3.2.1 Feature extraction

In this work, to compute CED the speech segments of interest are
conversational turns from both speakers. We compute the speaker turn
boundaries from the time information available in the transcripts, excluding
the intra-turn pauses to avoid including noisy and redundant signals. For
every speaker turn, we extract self-supervised TERA embeddings [16] to obtain
a $768$ dimensional feature vector. We choose TERA embeddings as it employs a
combination of auxiliary tasks to learn the embedding instead of relying on a
single task, so it is expected to learn enhanced features from raw speech
signals.

(a) Cross-encoder 1

(b) Cross-encoder 2

Fig. 2: Attention activations Table 2: Classification experiment for real vs fake sessions Measure | Classification accuracy(%)
---|---
Fisher Corpus | ADOSMod3 Corpus
Baseline 1 | 80.52 | 82.22
Baseline 2 | 76.33 | 70.64
Baseline 3 | 82.91 | 85.73
CED | 92.13 | 95.66

####  3.2.2 Parameters and implementation details

We use 352 and 64 attention units for the conformer and transformer layers,
respectively, while 4 attention heads are employed for both. The full
architecture obtained by using a conformer layer followed by a transformer
layer results into 2.1M parameters. The model is trained with a binary cross
entropy with logits loss function and Adam optimizer with the initial learning
rate of $1e^{-5}$. There is a provision of early stopping after 10 epochs if
no improvement is seen in validation loss, a dropout rate of 0.2 is used for
every dropout layer used in the model.

Fig. 3: Absolute values of CED across age and gender from ADOSMod3

###  3.3 Experimental validation of CED

We carry out an ad-hoc real/fake classification experiment to validate CED as
a metric for measuring entrainment. For every real sample session we
synthesize a fake sample session by shuffling the speaker turn while
maintaining the dyadic conversation sequence. The hypothesis is more
entrainment is expected to be observed in real sessions as compared fake
sessions resulting in the real sessions having lesser CED. The classification
accuracies are reported in Table 2. The classification experiment steps are as
follows:

  * •

We calculate CED measure for every consecutive turn pair for the real and fake
sample session.

  * •

We compare the average CED distance from all the turn pairs for the real and
fake session, the sample sessions are correctly classified if CED of real
session is lesser than fake session.

  * •

The experiment is repeated 30 times to eliminate any bias introduced while
randomly shuffling the speaker turns.

As baselines, we use three distance measures computed between the extracted
turn-level pretrained embeddings: smooth L1 distance [6] (Baseline 1) and two
measures introduced in [9], namely, DTWD (Baseline 2), and SCDC (Baseline 3).

Table 3: Correlation experiment between CED and clinical scores relevant to
ASD (bold figures imply statistical significance, $p<0.05$ )
(CP: child to psychologist, PC: psychologist to child)

Clinical scores | Pearson’s correlation
---|---
CED-PC | CED-CP
$\rho$ |  p-value | $\rho$ |  p-value
VINELAND ABC | -0.061 | 0.237 | 0.012 | 0.827
VINELAND Social | -0.021 | 0.345 | 0.071 | 0.073
VINELAND Communication | -0.158 | 0.003 | 0.043 | 0.428
CSS | 0.222 | 0.004 | 0.023 | 0.672
CSS-SA | 0.231 | 0.012 | 0.03 | 0.472
CSS-RRB | 0.158 | 0.055 | 0.091 | 0.262

###  3.4 Experimental evaluation

In this experiment, we calculate the correlation between the proposed CED
measure and the clinical scores relevant to ASD in Table 3. Since CED is
directional in nature, we compute the correlation metric in both the
directions child to psychologist and psychologist to child. We report the
Pearson’s correlation coefficient ($\rho$) and also the corresponding
$p$-value, to test the null hypothesis that there exists no linear association
between the proposed measure and the clinical scores. Amongst the clinical
scores, VINELAND scores are designed to measure adaptive behaviour of
individuals, while VINELAND ABC stands for Adaptive Behaviour Composite score,
VINELAND social and VINELAND communication are adaptive behavior scores for
specific skills of socialization and communication. CSS stands for Calibrated
Severity Score which reflects the severity of ASD symptoms in individuals.
CSS-SA and CSS-RRB reflects ASD symptoms severity along 2 domains of Social
Affect and Restrictive and Repetitive Behaviours. The details of the clinical
scores related to ASD are described in [17, 18, 19].

We also report the absolute values of the proposed CED measure (both
directions) for different gender and different age-groups. We partition the
dataset across 3 age groups of Group 1: $\leq 5yrs$, Group 2: $>5yrs$ & $\leq
10yrs$, Group3: $>10yrs$ and for each of the age groups we report the
directional CED measure for male and female subgroups in Fig. 3.

##  4 Results and Discussion

The results reported in Table 2 reveal that achieves better performance in
identifying real and fake sessions with respect to the baseline methods in
both Fisher and ADOSmod3 corpus in terms of classification accuracy, which
validates the use of CED as a proxy metric for measuring entrainment.

Results in Table 3 show that VINELAND communication score is negatively
correlated with psychologist$\rightarrow$child CED with significant statistic,
which stands consistent with the definition of CED, since higher CED signifies
lower entrainment. CSS and CSS-SA scores are reported to be positively
correlated with CED. It is interesting to note that while
psychologist$\rightarrow$child CED is capturing signals with meaningful
interpretations, no such evidence is reported from
child$\rightarrow$psychologist CED measures. A possible explanation can be
since the model is trained with dyadic conversations from adults in Fisher
corpus, the model is unable to capture the nuances of interactions involving
children which is reflected in these results. It is also worth mentioning
while there exists a significant correlation between CSS, CSS-SA and
psychologist$\rightarrow$child CED, CSS-RRB also shows weak evidence of
positive correlation with psychologist$\rightarrow$child CED.

In Table 3, the distributions for absolute values of CED are reported across
gender and age-groups. Both directional CED are always seen to have lesser
mean values in females as compared to males, which reiterates the claim
reported in [20] that women are better at disguising autism symptoms than men.
Across age-groups,the experimental results donot show any discernable
observation from CED in both directions in male children, however female
psychologist$\rightarrow$child CED is shown to decrease with an increase in
age, which also supports the claim presented in [20].

We also investigate the weights of the activations from the cross-encoder
attention layer to understand which parts of the speaker turns are emphasized
by the attention heads to extract meaningful signals. Attention activation
heatmaps from cross-encoder 1 and 2 reported in Fig. 2 show attention layers
attend to initial few timeframes from the second speaker turn which supports
the claim mentioned in [6] and domain theory that initial and final
interpausal units from second and first speaker respectively are a rich source
of signals related to entrainment.

##  5 Conclusion

In this work we introduce a novel context-aware approach (CED) to measure
vocal entrainment in dyadic conversations. We use a combination of
convolutional neural networks and transformers to capture both short-term and
long-term context, and also employ a cross-subject attention module to learn
interpersonal entrainment related information from the other subject in a
dyadic conversation. We validate the use of CED as a proxy metric for
measuring entrainment by conducting a classification experiment to distinguish
between real (consistent) and fake (inconsistent) interaction sessions. We
also study the association between CED and clinically relevant scores related
to ASD symptoms by computing the correlation metric. We also report the mean
absolute value of directional CED across gender and different age-groups to
understand if the entrainment pattern of the children varies across gender or
age-group or not. In this work, we use a self-supervised embedding for feature
extraction, it will be interesting to see if other context-based pre-trained
embeddings yeild similar performance in capturing entrainment. We also face
difficulties in deploying entrainment embeddings learnt on Fisher for ADOSMod3
dataset and thus we plan to investigate domain-specific entrainment embeddings
for understanding behavioral traits.

## References

  * [1] Alessandro Vinciarelli, Maja Pantic, Hervé Bourlard, and Alex Pentland,  “Social signal processing: state-of-the-art and future perspectives of an emerging domain,”  in Proceedings of the 16th ACM international conference on Multimedia, 2008, pp. 1061–1070.
  * [2] Shrikanth Narayanan and Panayiotis G Georgiou,  “Behavioral signal processing: Deriving human behavioral informatics from speech and language,”  Proceedings of the IEEE, vol. 101, no. 5, pp. 1203–1233, 2013.
  * [3] Susan E Brennan,  “Lexical entrainment in spontaneous dialog,”  Proceedings of ISSD, vol. 96, pp. 41–44, 1996.
  * [4] Rivka Levitan and Julia Bell Hirschberg,  “Measuring acoustic-prosodic entrainment with respect to multiple levels and dimensions,”  2011\.
  * [5] Howard Giles and Peter Powesland,  “Accommodation theory,”  in Sociolinguistics, pp. 232–239. Springer, 1997.
  * [6] Md Nasir, Brian Baucom, Craig Bryan, Shrikanth Narayanan, and Panayiotis Georgiou,  “Modeling vocal entrainment in conversational speech using deep unsupervised learning,”  IEEE Transactions on Affective Computing, 2020.
  * [7] Chi-Chun Lee, Matthew P. Black, Athanasios Katsamanis, Adam Lammert, Brian Baucom, Andrew Christensen, Panayiotis Georgiou, and Shrikanth S. Narayanan,  “Quantification of prosodic entrainment in affective spontaneous spoken interactions of married couples,”  in Proceedings of InterSpeech, September 2010, pp. 793–796.
  * [8] Bo Xiao, Zac E. Imel, David Atkins, Panayiotis Georgiou, and Shrikanth S. Narayanan,  “Analyzing speech rate entrainment and its relation to therapist empathy in drug addiction counseling,”  in Proceedings of Interspeech, sep 2015.
  * [9] Rimita Lahiri, Md Nasir, Manoj Kumar, So Hyun Kim, Somer Bishop, Catherine Lord, and Shrikanth Narayanan,  “Interpersonal synchrony across vocal and lexical modalities in interactions involving children with autism spectrum disorder,”  JASA Express Letters, vol. 2, no. 9, pp. 095202, 2022.
  * [10] Michael Natale,  “Convergence of mean vocal intensity in dyadic communication as a function of social desirability.,”  Journal of Personality and Social Psychology, vol. 32, no. 5, pp. 790, 1975.
  * [11] Nichola Lubold and Heather Pon-Barry,  “Acoustic-prosodic entrainment and rapport in collaborative learning dialogues,”  in Proceedings of the 2014 ACM workshop on Multimodal Learning Analytics Workshop and Grand Challenge, 2014, pp. 5–12.
  * [12] Emilie Delaherche, Mohamed Chetouani, Ammar Mahdhaoui, Catherine Saint-Georges, Sylvie Viaux, and David Cohen,  “Interpersonal synchrony: A survey of evaluation methods across disciplines,”  IEEE Transactions on Affective Computing, vol. 3, no. 3, pp. 349–365, 2012.
  * [13] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [14] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al.,  “Conformer: Convolution-augmented transformer for speech recognition,”  arXiv preprint arXiv:2005.08100, 2020.
  * [15] Christopher Cieri, David Miller, and Kevin Walker,  “The fisher corpus: A resource for the next generations of speech-to-text.,”  in LREC, 2004, vol. 4, pp. 69–71.
  * [16] Andy T Liu, Shang-Wen Li, and Hung-yi Lee,  “Tera: Self-supervised learning of transformer encoder representation for speech,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 2351–2366, 2021.
  * [17] Somer L Bishop, Marisela Huerta, Katherine Gotham, Karoline Alexandra Havdahl, Andrew Pickles, Amie Duncan, Vanessa Hus Bal, Lisa Croen, and Catherine Lord,  “The autism symptom interview, school-age: A brief telephone interview to identify autism spectrum disorders in 5-to-12-year-old children,”  Autism Research, vol. 10, no. 1, pp. 78–88, 2017.
  * [18] Katherine Gotham, Andrew Pickles, and Catherine Lord,  “Standardizing ados scores for a measure of severity in autism spectrum disorders,”  Journal of autism and developmental disorders, vol. 39, no. 5, pp. 693–705, 2009.
  * [19] Vanessa Hus, Katherine Gotham, and Catherine Lord,  “Standardizing ados domain scores: Separating severity of social affect and restricted and repetitive behaviors,”  Journal of autism and developmental disorders, vol. 44, no. 10, pp. 2400–2412, 2014.
  * [20] Eric Fombonne,  “Camouflage and autism,” 2020.