# Contextualized Multimodal Representations

# Multimodal Embeddings from Language Models

###### Abstract

Word embeddings such as ELMo have recently been shown to model word semantics
with greater efficacy through contextualized learning on large-scale language
corpora, resulting in significant improvement in state of the art across many
natural language tasks. In this work we integrate acoustic information into
contextualized lexical embeddings through the addition of multimodal inputs to
a pretrained bidirectional language model. The language model is trained on
spoken language that includes text and audio modalities. The resulting
representations from this model are multimodal and contain paralinguistic
information which can modify word meanings and provide affective information.
We show that these multimodal embeddings can be used to improve over previous
state of the art multimodal models in emotion recognition on the CMU-MOSEI
dataset.

##  1 Introduction

Acoustic and visual elements in human communication, such as intonation or
facial expressions, infuses semantic content with additional paralinguistic
cues which may modify intent and can convey affective meaning more clearly if
not exclusively [1]. For this reason many work have proposed multimodal
systems which integrate information from multiple modalities to improve
natural language understanding. This effort encompasses research in many
applications such as human robot interfaces [2, 3], video summarization [4, 5,
6], dialogue systems [7, 8], and emotion and sentiment analysis [9, 10, 11].

The study of multimodal fusion in affective systems is a prevalent and
important topic. This follows from the fact that human behavioral expression
is fundamentally a multifaceted phenomena that spans over multiple modalities
[12, 13] and can be more accurately identified through multimodal classifiers
[14]. Another factor is the importance of affective information as an
intermediate step in a variety of downstream tasks. Examples of which include
the use of these human behavioral states in language modeling [15], dialogue
systems [16, 17], and video summarization [18].

Many multimodal systems for recognition of sentiment, emotion, and behaviors
have been proposed in prior work. In feature-level fusion, Tzirakis et al.
[19] combined auditory and visual modalities by extracting features from
convolutional neural networks (CNN) on each modality which were then
concatenated as input to an LSTM network. Hazarika et al. [20] proposed the
use of a self-attention mechanism to assign scores for weighted combination of
modalities. Other work has applied multimodal integration using late fusion
methods [21, 22].

For deeper integration between modalities many work have proposed the use of
multimodal neural architectures. Lee et al. [23] proposed the use of an
attention matrix calculated from speech and text features to selectively focus
on specific regions of the audio feature space. The memory fusion network was
introduced by Zadeh et al. [24] which accounted for intra- and inter-modal
dependencies across time. Akhtar et al. [25] proposed a contextual inter-modal
attention network which leveraged sentiment and emotion labels in a multi-task
learning framework.

The strength of deep models arises from the ability to internally learn
meaningful representations of features from multiple modalities. This is
learned implicitly by the model through the course of training on respective
datasets [26]. In this work we propose a model to explicitly learn informative
joint representations of speech and text. This is achieved by modeling the
dynamics between lexical content and paralinguistics from audio through a
language modeling task on spoken language. We augment a bidirectional language
model (biLM) with word-aligned acoustic features and optimize the model using
large-scale text corpora followed by spoken articles. The internal states of
this biLM aren’t representative of a specific task but rather models the
intricacies of human communication through speech and language. We show the
effectiveness of representations extracted from this model in capturing
multimodal information by evaluating in the task of emotion recognition.
Through the use of these representation we improve the state of the art in
emotion recognition on the CMU-MOSEI dataset.

##  2 Related Work

Lexical representations such as ELMo [27] and BERT [28] have recently been
shown to model word semantics and syntax with greater efficacy. This is
achieved through contextualized learning on large-scale language corpora which
allows internal states of the model to capture both the complex
characteristics of word use as well as polysemy due to different contexts. The
integration of these word embeddings into downstream models have improved the
state of the art in many NLP tasks through their rich representation of
language use.

To learn representations from multimodal data Hsu et al. [29] proposed the use
of variational autoencoders to encode inter- and intra-modal factors into
separate latent variables. Later, Tsai et al. [30] factorized representations
into multimodal discriminative and modality-specific generative factors using
inference and generative networks. During the course of writing this paper
Rahman et al. [31] concurrently proposed the infusion of multimodal
information into the BERT model. There the authors combined the generative
capabilities of the BERT model with a sentiment prediction task to allow the
model to implicitly learn rich multimodal representations through a joint
generative-discriminative objective.

In this work we propose to explicitly learn multimodal representations of
spoken words by augmenting the biLM model in ELMo with acoustic information.
This is motivated from how humans integrate acoustic characteristics in speech
to interpret the meaning and intent of lexical content from a speaker. Our
work differs from prior work in that we do not include or target any
discriminative objectives and instead rely on the generative task of language
modeling to learn meaningful multimodal representations. We show how this
model can be easily trained with large-scale unlabeled data and also
demonstrate how potent multimodal embeddings from this model are in tasks such
as emotion recognition.

##  3 Multimodal Embeddings

In this section we describe the network architecture of the bidirectional
language model with acoustic information that generates the multimodal
embeddings. The biLM comprises two layers of bidirectional LSTMs which operate
over lexical and audio embeddings. The lexical and audio embeddings are
calculated from respective convolutional layers and combined using a sigmoid-
gating function. Multimodal embeddings are then computed using a linear
function over the internal states of the recurrent layers. The overall
architecture of the mutlimodal biLM is shown in Figure 1.

Fig. 1: Architecture of the multimodal bidirectional language model.

###  3.1 Bidirectional language model

A language model (LM) computes the probability distribution of a sequence of
words by approximating it as the product of conditional probabilities of each
word given previous words. This has been implemented using neural networks in
many prior work yielding state of the art results [32, 33]. In this work we
applied the biLM model used in ELMo, which is similar to the character-level
RNN-LM described by Józefowicz et al. [34] and Kim et al. [35].

The biLM is composed of a forward and backward LM each implemented by a two-
layer LSTM. The forward LM predicts the probability distribution of the next
token given past context while the backward LM predicts the probability
distribution of the previous token given future context. Each LM operates on
the same input, which is a token embedding of the current token calculated
through a character-level convolutional neural network (CharCNN). A softmax
layer is used to estimate token probabilities from the output of the two-layer
LSTM in the LMs. The parameters of the softmax layer are shared between the
LMs in both directions.

Different from ELMo our input to the biLM includes acoustic features in
additional to word tokens. Now the forward LM aims to model, at each timestep,
the conditional probability of the next token $t_{k+1}$ given the current
token $t_{k}$, acoustic features $\mathbf{a}_{k}$, and previous internal
states of the two-layer LSTM $\vec{\mathbf{s}}_{k-1}$:

$P(t_{k+1}\mid t_{k},\mathbf{a}_{k},\vec{\mathbf{s}}_{k-1})$

The backward LM operates similarly but predicts the previous token $t_{k-1}$
given the current token $t_{k}$, acoustic features $\mathbf{a}_{k}$, and
internal states resulting from future context $\mathchoice{\mkern
3.0mu\reflectbox{$\displaystyle\vec{\reflectbox{$\mkern-3.0mu\displaystyle\mathbf{s}\mkern
3.0mu$}}$}\mkern-3.0mu}{\mkern
3.0mu\reflectbox{$\textstyle\vec{\reflectbox{$\mkern-3.0mu\textstyle\mathbf{s}\mkern
3.0mu$}}$}\mkern-3.0mu}{\mkern
2.0mu\reflectbox{$\scriptstyle\vec{\reflectbox{$\mkern-2.0mu\scriptstyle\mathbf{s}\mkern
2.0mu$}}$}\mkern-2.0mu}{\mkern
2.0mu\reflectbox{$\scriptscriptstyle\vec{\reflectbox{$\mkern-2.0mu\scriptscriptstyle\mathbf{s}\mkern
2.0mu$}}$}\mkern-2.0mu}_{k+1}$.

###  3.2 Acoustic convolution layers

In our implementation time-aligned acoustic features of each word are provided
in adjunct to word tokens. We build on ELMo and add additional convolutional
layers to calculate acoustic embeddings from the acoustic features. The
convolutional layers provide a feature transformation of the acoustic features
which we combine with token embeddings using a gating function.

Due to the varying lengths of words in time, the acoustic features are first
padded to a fixed size in the temporal dimension before being passed to the
CNN. Each convolution layer in the CNN comprises a 1-D convolution layer
followed by a max-pooling layer. Finally, the feature map is projected to the
same dimension size as token embeddings to allow for element-wise combination.

###  3.3 Multimodal ELMo

We combine token and acoustic embeddings using a sigmoid gating function:

$\mathbf{M}_{k}=\mathbf{U}(t_{k})\odot\sigma(\mathbf{V}(\mathbf{a}_{k}))$

where $\mathbf{U}$ and $\mathbf{V}$ are the embeddings calculated from the
token and corresponding acoustic features, respectively, $\sigma$ is the
sigmoid function, and $\odot$ represents element-wise multiplication. The
resulting multimodal embeddings are used as input to the forward and backward
LM.

The sigmoid gate scales the token embedding based on corresponding acoustic
features of the word. This serves as a modifier of semantic meaning using
paralinguistic information which we hypothesize will be useful in capturing
affective expressions in downstream tasks.

Word embeddings are extracted for use in downstream models in a similar
fashion to ELMo. That is, we compute a task-specific weighted sum of all LSTM
outputs as each word embedding. We adopt the use of sentence embeddings in
downstream models and additionally average all the word embeddings in a
sentence. The final multimodal ELMo (M-ELMo) sentence embedding is given as

$\mathbf{M{\text{-}}ELMo}=\gamma\frac{1}{N}\sum_{j}^{L}c_{j}\sum_{k}^{N}\mathbf{h}_{k,j}$

where $\mathbf{h}_{k,j}$ are the concatenated outputs of LSTMs in both
directions at the $j^{\mathrm{th}}$ layer for the $k^{\mathrm{th}}$ token.
Values $\\{c_{j}\\}$ are softmax-normalized weights and $\gamma$ is a scalar
value, all of which are tunable parameters in the downstream model.

##  4 Experimental Setup

| Anger | Disgust | Fear | Happy | Sad | Surprise | Average‡
---|---|---|---|---|---|---|---
Method | WA | F1 | WA | F1 | WA | F1 | WA | F1 | WA | F1 | WA | F1 | WA | F1
(A + L + V) |  |  |  |  |  |  |  |  |  |  |  |  |  |
Graph-MFN [36] | 62.6 | 72.8 | 69.1 | 76.6 | 62.0 | 89.9 | 66.3 | 66.3 | 60.4 | 66.9 | 53.7 | 85.5 | 62.3 | 76.3
CIM-Att-STL [25] | 64.5 | 75.6 | 72.2 | 81.0 | 51.5 | 87.7 | 61.6 | 59.3 | 65.4 | 67.3 | 53.0 | 86.5 | 61.3 | 76.2
(A + L) |  |  |  |  |  |  |  |  |  |  |  |  |  |
CIM-Att-STL [25] | - | - | - | - | - | - | - | - | - | - | - | - | 59.6 | 76.8
M-ELMo + NN § | 65.8 | 74.7 | 74.2 | 81.7 | 63.2 | 85.1 | 67.0 | 65.2 | 63.1 | 72.0 | 63.8 | 83.3 | 66.2 | 77.0
Table 1: Emotion recognition results on CMU-MOSEI test set for various
multimodal models. Modalities: acoustic (A), lexical (L), visual (V). Values
are taken from their respective sources. §Average across ten runs. ‡Average
across six emotions.

###  4.1 Pre-training the multimodal biLM

The multimodal biLM is pre-trained in two stages. In the first stage the
lexical components of the biLM are optimized prior to the inclusion of
acoustic features. This is achieved by training with a text corpus and fixing
the acoustic input as zero. We use the 1 Billion Word Language Model Benchmark
[37] for this purpose and train the biLM for 10 epochs. After training, the
model achieves perplexities of around 35 which is similar to values reported
in [27].

In the second stage of training we optimize the biLM using the multimodal
dataset CMU-MOSEI [36] (described in Section 4.3). In our experiments we use
only the text and audio of segments in the training split of the dataset to
train the model. In terms of word count CMU-MOSEI is much smaller than the
1-billion word LM benchmark, therefore to prevent overfitting we reduce the
learning rate used in the previous stage by a factor of 10 and train for an
additional 5 epochs.

###  4.2 Features

Since a CharCNN is used as the lexical embedder, input words to the biLM are
first transformed into a character map and padded to a fixed length. The
character-level representation of each word is then given as a $c\times l_{c}$
matrix, where $c$ is the dimension size of the character embedding and $l_{c}$
is the maximum number of characters in a word.

Acoustic features were extracted from each recording at 10ms frame intervals
using the COVAREP software version 1.4.2 [38]. There are 74 features in total
and include, among others, pitch, voiced/unvoiced segment features, mel-
frequency cepstral coefficients, glottal flow parameters, peak slope
parameters, and harmonic model parameters.

The acoustic features are aligned with word timings to provide acoustic
information for each word. Since the time duration varies between words we pad
the number of acoustic frames per token to a fixed length. Thus, word-aligned
acoustic features are given as a $d\times l_{a}$ matrix, where $d$ is the
number of acoustic features and $l_{a}$ is the maximum number of acoustic
frames in a word.

###  4.3 Emotion recognition as a downstream task

After pre-training, the multimodal biLM is used to extract multimodal sentence
embeddings for use in downstream models. In our experiments we adopt emotion
recognition as the downstream task and evaluate on the CMU-MOSEI dataset.

CMU-MOSEI contains 23,453 single-speaker video segments from YouTube which
have been manually transcribed and annotated for sentiment and emotion.
Emotions are annotated on a [0,3] Likert scale and include those such as
happiness, sadness, anger, fear, disgust, and surprise. We binarize these
annotations to arrive at class labels by predicting the presence of emotions,
i.e. any emotion with a rating greater than one. Since video segments have
ratings for all emotions this becomes a multi-label classification task.

As our goal is to evaluate the efficacy of the multimodal sentence embeddings
we used a simple feedforward neural network for emotion recognition. The
network inputs the multimodal sentence embeddings and predicts the presence of
each emotion. The tunable parameters described in Section 3.3 are also
included in this network. We trained the network using data from the training
split provided in the dataset and validated using the validation split. We
also used the validation split as a development set in choosing hyper-
parameters of the network.

###  4.4 Evaluation methods

We evaluated the emotion recognition model using weighted accuracy (WA) and F1
score on each emotion. Weighted accuracy, as used in [36], is equivalent to
the macro-average recall value. We also averaged the metrics across all
emotions to obtain an average WA and F1 score.

The downstream model was trained for 30 epochs and separately optimized on WA
and F1 score using the validation set. We randomly initialized each downstream
model ten times and a best model was selected based on the average scores on
validation over the ten runs. The final model was a neural network with two
hidden layers using ReLU activation functions.

Due to the lack of work that only focuses on text and audio, we compared with
models that also considers the visual modality. We compared our performance
with two recent state of the art emotion recognition models on CMU-MOSEI.
Specifically, these are the graph Memory Fusion Network (Graph-MFN) [36] and
the contextual inter-modal attention framework (CIM-Att) [25]. To match
learning conditions, we compared with the single task learning (STL) model of
[25] where only emotion labels are used in training.

##  5 Results

The results of the final model averaged across ten runs are shown in Table 1.
Our simple feedforward neural network using multimodal embeddings achieved
state of the art results in terms of average WA and F1 over all emotions at
66.2% and 77.0%, respectively. On individual emotions our model yielded
comparable to improved results over state of the art. Specifically, we
observed improvements in the weighted accuracy of all emotions except sad as
well improvements in F1 score of disgust and sad.

##  6 Conclusion

In this work we proposed a method for extending ELMo word embeddings to
include acoustic information. We used convolutional layers over word-aligned
acoustic features to calculate acoustic embeddings which we then combined with
token embeddings in ELMo using a sigmoid gating function. The model was
trained on a language modeling task, first with a text corpus followed by
inclusion of audio from a multimodal dataset. We then showed the effectiveness
of sentence embeddings extracted from this multimodal biLM in emotion
recognition. The results are surprising given that our downstream model using
a neural network with two hidden layers outperformed state of the art
architectures. This demonstrates how well the multimodal embeddings have
captured inter- and intra-modal dynamics in spoken language.

## References

  * [1] Emma Rodero,  “Intonation and emotion: influence of pitch levels and contour type on creating emotions,”  Journal of Voice, 2011.
  * [2] Erica L Meszaros, Meghan Chandarana, Anna Trujillo, and B Danette Allen,  “Compensating for limitations in speech-based natural language processing with multimodal interfaces in uav operation,”  in International Conference on Applied Human Factors and Ergonomics. Springer, 2017, pp. 183–194.
  * [3] Iñaki Maurtua, Izaskun Fernández, Alberto Tellaeche, Johan Kildal, Loreto Susperregi, Aitor Ibarguren, and Basilio Sierra,  “Natural multimodal communication for human–robot collaboration,”  International Journal of Advanced Robotic Systems, vol. 14, no. 4, pp. 1729881417716043, 2017.
  * [4] Fumio Nihei, Yukiko I Nakano, and Yutaka Takase,  “Predicting meeting extracts in group discussions using multimodal convolutional neural networks,”  in Proceedings of the 19th ACM International Conference on Multimodal Interaction. ACM, 2017, pp. 421–425.
  * [5] Fumio Nihei, Yukiko I Nakano, and Yutaka Takase,  “Fusing verbal and nonverbal information for extractive meeting summarization,”  in Proceedings of the Group Interaction Frontiers in Technology. ACM, 2018, p. 9.
  * [6] Shruti Palaskar, Jindřich Libovický, Spandana Gella, and Florian Metze,  “Multimodal abstractive summarization for How2 videos,”  in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019, pp. 6587–6596, Association for Computational Linguistics.
  * [7] Lizi Liao, Yunshan Ma, Xiangnan He, Richang Hong, and Tat-seng Chua,  “Knowledge-aware multimodal dialogue systems,”  in 2018 ACM Multimedia Conference on Multimedia Conference. ACM, 2018, pp. 801–809.
  * [8] Chiori Hori, Huda Alamri, Jue Wang, Gordon Wichern, Takaaki Hori, Anoop Cherian, Tim K Marks, Vincent Cartillier, Raphael Gontijo Lopes, Abhishek Das, et al.,  “End-to-end audio visual scene-aware dialog using multimodal attention-based video features,”  in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 2352–2356.
  * [9] Anthony Hu and Seth Flaxman,  “Multimodal sentiment analysis to explore the structure of emotions,”  in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2018, pp. 350–358.
  * [10] Seunghyun Yoon, Seokhyun Byun, and Kyomin Jung,  “Multimodal speech emotion recognition using audio and text,”  in 2018 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2018, pp. 112–118.
  * [11] Paul Pu Liang, Ziyin Liu, AmirAli Bagher Zadeh, and Louis-Philippe Morency,  “Multimodal language analysis with recurrent multistage fusion,”  in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 150–161.
  * [12] Shrikanth Narayanan and Panayiotis G Georgiou,  “Behavioral signal processing: Deriving human behavioral informatics from speech and language,”  Proceedings of the IEEE, vol. 101, no. 5, pp. 1203–1233, 2013.
  * [13] Sidney K D’mello and Jacqueline Kory,  “A review and meta-analysis of multimodal affect detection systems,”  ACM Computing Surveys (CSUR), vol. 47, no. 3, pp. 43, 2015.
  * [14] Soujanya Poria, Erik Cambria, Rajiv Bajpai, and Amir Hussain,  “A review of affective computing: From unimodal analysis to multimodal fusion,”  Information Fusion, vol. 37, pp. 98–125, 2017.
  * [15] Prashanth Gurunath Shivakumar, Shao-Yen Tseng, Panayiotis Georgiou, and Shrikanth Narayanan,  “Behavior gated language models,”  arXiv preprint arXiv:1909.00107, 2019.
  * [16] Johannes Pittermann, Angela Pittermann, and Wolfgang Minker,  “Emotion recognition and adaptation in spoken dialogue systems,”  International Journal of Speech Technology, vol. 13, no. 1, pp. 49–60, 2010.
  * [17] Dario Bertero, Farhad Bin Siddique, Chien-Sheng Wu, Yan Wan, Ricky Ho Yin Chan, and Pascale Fung,  “Real-time speech emotion and sentiment recognition for interactive dialogue systems,”  in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016, pp. 1042–1047.
  * [18] Ashish Singhal, Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, Debi Prosad Dogra, and Byung-Gyu Kim,  “Summarization of videos by analyzing affective state of the user through crowdsource,”  Cognitive Systems Research, vol. 52, pp. 917–930, 2018.
  * [19] Panagiotis Tzirakis, George Trigeorgis, Mihalis A Nicolaou, Björn W Schuller, and Stefanos Zafeiriou,  “End-to-end multimodal emotion recognition using deep neural networks,”  IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1301–1309, 2017.
  * [20] Devamanyu Hazarika, Sruthi Gorantla, Soujanya Poria, and Roger Zimmermann,  “Self-attentive feature-level fusion for multimodal emotion detection,”  in 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR). IEEE, 2018, pp. 196–201.
  * [21] Shao-Yen Tseng, Haoqi Li, Brian Baucom, and Panayiotis Georgiou,  “”Honey, I learned to talk”: Multimodal fusion for behavior analysis,”  in Proceedings of the 20th ACM International Conference on Multimodal Interaction, New York, NY, USA, 2018, ICMI ’18, pp. 239–243, ACM.
  * [22] Nathaniel Blanchard, Daniel Moreira, Aparna Bharati, and Walter J Scheirer,  “Getting the subtext without the text: Scalable multimodal sentiment classification from visual and acoustic modalities,”  arXiv preprint arXiv:1807.01122, 2018.
  * [23] Chan Woo Lee, Kyu Ye Song, Jihoon Jeong, and Woo Yong Choi,  “Convolutional attention networks for multimodal emotion recognition from speech and text data,”  ACL 2018, p. 28, 2018.
  * [24] Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency,  “Memory fusion network for multi-view sequential learning,”  in Thirty-Second AAAI Conference on Artificial Intelligence, 2018\.
  * [25] Md Shad Akhtar, Dushyant Chauhan, Deepanway Ghosal, Soujanya Poria, Asif Ekbal, and Pushpak Bhattacharyya,  “Multi-task learning for multi-modal emotion recognition and sentiment analysis,”  in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2019.
  * [26] Yoshua Bengio, Aaron Courville, and Pascal Vincent,  “Representation learning: A review and new perspectives,”  IEEE transactions on pattern analysis and machine intelligence, vol. 35, no. 8, pp. 1798–1828, 2013.
  * [27] Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer,  “Deep contextualized word representations,”  in Proceedings of NAACL-HLT, 2018, pp. 2227–2237.
  * [28] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova,  “BEST: Pre-training of deep bidirectional transformers for language understanding,”  in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 4171–4186.
  * [29] Wei-Ning Hsu and James Glass,  “Disentangling by partitioning: A representation learning framework for multimodal sensory data,”  arXiv preprint arXiv:1805.11264, 2018.
  * [30] Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov,  “Learning factorized multimodal representations,”  in International Conference on Learning Representations, 2019.
  * [31] Wasifur Rahman, Md Kamrul Hasan, Amir Zadeh, Louis-Philippe Morency, and Mohammed Ehsan Hoque,  “M-BERT: Injecting multimodal information in the BERT structure,”  arXiv preprint arXiv:1908.05787, 2019.
  * [32] Stephen Merity, Nitish Shirish Keskar, and Richard Socher,  “Regularizing and optimizing LSTM language models,”  in International Conference on Learning Representations (ICLR), 2018\.
  * [33] Chengyue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu,  “Frage: Frequency-agnostic word representation,”  in Advances in neural information processing systems (NIPS), 2018\.
  * [34] Rafal Józefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu,  “Exploring the limits of language modeling,”  arXiv preprint arXiv:1602.02410, 2016.
  * [35] Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush,  “Character-aware neural language models,”  in Thirtieth AAAI Conference on Artificial Intelligence, 2016.
  * [36] Amir Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency,  “Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph,”  in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 2018.
  * [37] Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson,  “One billion word benchmark for measuring progress in statistical language modeling,”  in Interspeech, 2014.
  * [38] Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer,  “COVAREP - A collaborative voice analysis repository for speech technologies,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2014.


# On Power Sums and the Kummer Congruences

Samuel Goodman

(May 2023)

## Introduction

The simplest form of the Kummer Congruences were first proven in $1851$ by
Ernst Eduard Kummer in [2]. They state that given a prime $p$ and even
positive integers $r,s$ with $r$ not a multiple of $p-1$ such that $r\equiv
s\mod p-1$, then

$\displaystyle\frac{B_{r}}{r}\equiv\frac{B_{s}}{s}\mod p$

where $B_{n}$ denotes the $n$th Bernoulli number. A stronger version of this
result was first obtained by Voronoi in [3] using power sum identities and
then later by Kubota-Leopoldt in [1] using the theory of $p$-adic
$L$-functions, proving that given a prime $p$, a nonnnegative integer $a$, and
even positive integers $r,s$ with $r$ not a multiple of $p-1$ such that
$r\equiv s\mod p^{a}(p-1)$, then

$\displaystyle(1-p^{r-1})\frac{B_{r}}{r}\equiv(1-p^{s-1})\frac{B_{s}}{s}\mod
p^{a+1}$

In particular, Kubota-Leopoldt cast this classical result in a more natural
light, as it in fact becomes the input for the proof of the continuity of the
$p$-adic zeta function.

In this paper, we develop a structure theory that again recasts this result in
a new way, shedding light on the element-wise behavior of a special class of
polynomials $\mod$ powers of $p$ through a series of new results about the
“pseudo-multiplicative” structure these polynomials have by analyzing the
stabilizers of the natural multiplication action on the resulting sets of
values. As a result of this approach, we also deduce a new proof of the Kummer
Congruences.

We start by introducing some preliminary results, well-known in the
literature:

### Lemma 1

Let $p>3$ be a prime, $a$ a positive integer, and $r$ a positive even integer.
We have that

$\displaystyle\sum_{n=1}^{p^{a}}n^{r}\equiv p^{a}B_{r}\mod p^{2a+v_{p}(r)+1}$

### Lemma 2

Let $p$ be an odd prime and $a$ and $n$ be positive integers and $k$ a
positive integer with $k\geq r+a$, $p^{r}|k$, and $p-1$ not dividing $k$. Then
we have that

$\displaystyle\sum_{n=1}^{p^{a}}n^{k}\equiv 0\mod p^{r+a}$

### Lemma 3 (Adams Theorem)

For any even integer $s$ with $s$ not a multiple of $p-1$,
$v_{p}(\frac{B_{s}}{s})\geq 0$ (in particular, $\frac{B_{s}}{s}$ is a
$p$-integer).

We start by showing that certain special multisets are stabilized by
nontrivial residue classes upon the natural multiplication action. This will
be the first hint of local information (namely the individual values of the
polynomial $\mod$ powers of $p$) have rich internal structure that explains
global phenomena, one of which will be a precursor to the Kummer Congruences.

## Theorem 1

Let $p$ be an odd prime, $a,t$ nonnegative integers, and $k$ a positive
integer with $p^{2a+1}|k$. Let $d=\frac{p-1}{\gcd(k,p-1)}$,
$v=\min(v_{p}(k)-2a-1,t)$, and $g$ a $d$th root of unity $\mod p^{3a+t+v+2}$.
In addition, let $\mathfrak{S}$ be the multiset consisting of the invertible
residues $\mod p^{3a+t+v+2}$ attained by
$n^{(k+p^{a}(p-1))p^{t}}+n^{(k-p^{a}(p-1))p^{t}}$ as $n$ varies from $1$ to
$p^{a+1}$ with multiplicity. Then $g\mathfrak{S}=\mathfrak{S}$.

We show that $g\mathfrak{S}\subset\mathfrak{S}$, from which it follows that
$g\mathfrak{S}=\mathfrak{S}$ as they are multisets of the same size. Each
element $s$ of $\mathfrak{S}$ corresponds to some positive integer $n$ with
$1\leq n<p^{a+1}$ with $n^{(k+p^{a}(p-1))p^{t}}+n^{(k-p^{a}(p-1))p^{t}}\equiv
s\mod p^{3a+t+v+2}$ by construction. For this $n$, we find an associated
$n^{\prime}$ such that
$n^{\prime(k+p^{a}(p-1))p^{t}}+n^{\prime(k-p^{a}(p-1))p^{t}}\equiv gs\mod
p^{3a+t+v+2}$.

Let $k=p^{r}k^{\prime}$ with $p\nmid k^{\prime}$. First we show that there is
a solution to $x^{k^{\prime}}\equiv g\mod p^{a+1}$. Note that
$\gcd(k^{\prime},p-1)=\gcd(k,p-1)$, and so $x^{k^{\prime}}$ and
$x^{\gcd(k,p-1)}$ generate the same residue classes $\mod p^{a+1}$, so it
suffices to show the result for $x^{\gcd(k,p-1)}$. Let $g^{\prime}$ be a
primitive root $\mod p^{a+1}$ and write $g\equiv g^{\prime b}\mod p^{a+1}$ for
some $b$. Then since $g^{d}\equiv 1\mod p^{a+1}$, we conclude that $p-1|db$,
which implies that $\frac{p-1}{d}|b$, which gives that $\gcd(k,p-1)|b$ by
definition and hence that $g\equiv x^{\gcd(p-1,k)}\mod p^{a+1}$ for some $x$.

Now pick a fixed solution $x^{\prime}$ to $x^{k^{\prime}}\equiv g\mod
p^{a+1}$. For each $n$, we let $n^{\prime}$ be the unique positive integer
$\leq p^{a+1}$ such that $n^{\prime}\equiv nx^{\prime}\mod p^{a+1}$, and we
show that $n^{\prime(k+p^{a}(p-1))p^{t}}+n^{\prime(k-p^{a}(p-1))p^{t}}\equiv
gs\mod p^{3a+t+v+2}$ under this assumption. Note that this is a bijection
since $x^{\prime}$ is coprime to $p$.

First note that

$(nx^{\prime})^{(k+p^{a}(p-1))p^{t}}+(nx^{\prime})^{(k-p^{a}(p-1))p^{t}}=(n^{\prime}+p^{a+1}c)^{(k+p^{a}(p-1))p^{t}}+(n^{\prime}+p^{a+1}c)^{(k-p^{a}(p-1))p^{t}}$

but then the binomial theorem implies that this is

$n^{\prime(k+p^{a}(p-1))p^{t}}+n^{\prime(k-p^{a}(p-1))p^{t}}+kp^{a+t+1}c(n^{\prime(k+p^{a}(p-1)-1)p^{t}}+n^{\prime(k-p^{a}(p-1))p^{t}-1})\mod
p^{3a+t+v+2}$

By assumption, we have that $p^{2a+1+v}|k$, implying that

$kp^{a+t+1}c(n^{\prime(k+p^{a}(p-1)-1)p^{t}}+n^{\prime(k-p^{a}(p-1))p^{t}-1})$

vanishes $\mod p^{3a+t+v+2}$, and so we get that

$(nx^{\prime})^{(k+p^{a}(p-1))p^{t}}+(nx^{\prime})^{(k-p^{a}(p-1))p^{t}}\equiv
n^{\prime(k+p^{a}(p-1))p^{t}}+n^{\prime(k-p^{a}(p-1))p^{t}}\mod p^{3a+t+v+2}$

Hence it suffices to show that

$(nx^{\prime})^{(k+p^{a}(p-1))p^{t}}+(nx^{\prime})^{(k-p^{a}(p-1))p^{t}}\equiv
g(n^{(k+p^{a}(p-1))p^{t}}+n^{(k-p^{a}(p-1))p^{t}})\mod p^{3a+t+v+2}$

We analyze

$n^{(k+p^{a}(p-1))p^{t}}(x^{\prime(k+p^{a}(p-1))p^{t}}-g)+n^{(k-p^{a}(p-1))p^{t}}(x^{\prime(k-p^{a}(p-1))p^{t}}-g)$

Euler’s Theorem implies that $x^{\prime(k+p^{a}(p-1))p^{t}}\equiv
x^{\prime(k+rp^{3a+v+1}(p-1)+p^{a}(p-1))p^{t}}\mod p^{3a+t+v+2}$ for any $r$,
but then notice that we can choose $r$ such that
$k^{\prime}|rp^{3a+v+1}(p-1)+p^{a}(p-1)$ (as $p$ is invertible $\mod
k^{\prime}$). For this $r$, we have that
$\frac{(k+rp^{3a+v+1}(p-1)+p^{a}(p-1))p^{t}}{k^{\prime}}\equiv
1+\frac{(rp^{3a+v+1}(p-1)+p^{a}(p-1))p^{t}}{k^{\prime}}\mod d$. Since
$k^{\prime}|(p-1)(rp^{3a+v+1}+p^{a})p^{t}$, we can write
$k^{\prime}=k_{1}k_{2}$ with $k_{1}|p-1$ and $k_{2}|(rp^{3a+v+1}+p^{a})p^{t}$.
For this $k_{1}$, we see that $k_{1}|\gcd(k,p-1)$, and so
$d|\frac{p-1}{k_{1}}$, immediately implying that
$d|\frac{(rp^{3a+v+1}(p-1)+p^{a}(p-1))p^{t}}{k^{\prime}}$. Therefore,
$\frac{(k+rp^{3a+v+1}(p-1)+p^{a}(p-1))p^{t}}{k^{\prime}}\equiv 1\mod d$.
Letting $u=\frac{(k+rp^{3a+v+1}(p-1)+p^{a}(p-1))p^{t}}{k^{\prime}}$,

$x^{\prime(k+p^{a}(p-1))p^{t}}-g\equiv(x^{\prime
k^{\prime}}-g)(\sum_{i=0}^{u-1}x^{\prime k^{\prime}(u-1-i)}g^{i})\mod
p^{3a+t+v+2}$

Similarly, letting
$u^{\prime}=\frac{(k+r^{\prime}p^{3a+v+1}(p-1)-p^{a}(p-1))p^{t}}{k^{\prime}}$
for some $r^{\prime}$ with $k^{\prime}|(r+r^{\prime})(p-1)$, we have

$x^{\prime(k-p^{a}(p-1))p^{t}}-g\equiv(x^{\prime
k^{\prime}}-g)(\sum_{i=0}^{u^{\prime}-1}x^{\prime
k^{\prime}(u^{\prime}-1-i)}g^{i})\mod p^{3a+t+v+2}$

and so it remains to show that

$n^{(k+p^{a}(p-1))p^{t}}(\sum_{i=0}^{u-1}x^{\prime
k^{\prime}(u-1-i)}g^{i})+n^{(k-p^{a}(p-1))p^{t}}(\sum_{i=0}^{u^{\prime}-1}x^{\prime
k^{\prime}(u^{\prime}-1-i)}g^{i})\equiv 0\mod p^{2a+t+v+1}$

Now since

$\sum_{i=0}^{u-1}x^{\prime k^{\prime}(u-1-i)}g^{i}=\frac{x^{\prime
uk^{\prime}}-g^{u}}{x^{\prime k^{\prime}}-g}$

lifting implies that the sum vanishes $\mod p^{v_{p}(u)}=p^{a+t}$, and so

$n^{(k+p^{a}(p-1))p^{t}}(\sum_{i=0}^{u-1}x^{\prime
k^{\prime}(u-1-i)}g^{i})\equiv n^{kp^{t}}(\sum_{i=0}^{u-1}x^{\prime
k^{\prime}(u-1-i)}g^{i})\mod p^{2a+2t+1}$

by Euler’s Theorem. $t\geq v$ gives this relation $\mod p^{2a+t+1+v}$, and now
repeating this process with

$n^{(k-p^{a}(p-1))p^{t}}(\sum_{i=0}^{u^{\prime}-1}x^{\prime
k^{\prime}(u^{\prime}-1-i)}g^{i})$

gives that it suffices to show

$\sum_{i=0}^{u-1}x^{\prime
k^{\prime}(u-1-i)}g^{i}+\sum_{i=0}^{u^{\prime}-1}x^{\prime
k^{\prime}(u^{\prime}-1-i)}g^{i}\equiv 0\mod p^{2a+t+v+1}$

Write $u=p^{a+t+v}q+y$ and $u^{\prime}=p^{a+t+v}q^{\prime}+y^{\prime}$ with
$0\leq y,y^{\prime}<p^{a+t+v}$.

First we claim that our expression is just

$(q+q^{\prime})\sum_{i=0}^{p^{a+t+v}-1}x^{\prime
k^{\prime}(p^{a+t+v}-1-i)}g^{i}+g^{q}\sum_{i=0}^{y-1}x^{\prime
k^{\prime}(y-1-i)}g^{i}+g^{q^{\prime}}\sum_{i=0}^{y^{\prime}-1}x^{\prime
k^{\prime}(y^{\prime}-1-i)}g^{i}\mod p^{2a+1+t+v}$

We have that for any positive integer $c$,

$\sum_{i=cp^{a+t+v}}^{(c+1)p^{a+t+v}-1}x^{\prime
k^{\prime}(u-1-i)}g^{i}=\sum_{i=0}^{p^{a+t+v}-1}x^{\prime
k^{\prime}(u-1-cp^{a+t+v}-i)}g^{i+cp^{a+t+v}}$

By lifting, we have that $x^{\prime k^{\prime}p^{a+t+v}}\equiv
g^{p^{a+t+v}}\mod p^{2a+t+v+1}$, and so

$\sum_{i=0}^{p^{a+t+v}-1}x^{\prime
k^{\prime}(u^{\prime}-1-cp^{a+t+v}-i)}g^{i+cp^{a+t+v}}\equiv\sum_{i=0}^{p^{a+t+v}-1}x^{\prime
k^{\prime}(u-1-i)}g^{i}\mod p^{2a+t+v+1}$

Then note that

$\sum_{i=0}^{p^{a+t+v}-1}x^{\prime
k^{\prime}(u-1-i)}g^{i}-x^{k^{\prime}(p^{a+t+v}-1-i)}g^{i}=(x^{\prime
k^{\prime}(u-p^{a+t+v})}-1)\sum_{i=0}^{p^{a+t+v}-1}x^{k^{\prime}(p^{a+t+v}-1-i)}g^{i}$

We have that $v_{p}(x^{\prime k^{\prime}(u-p^{a+t+v})}-1)\geq a+1+a+t$, and

$v_{p}(\sum_{i=0}^{p^{a+t+v}-1}x^{k^{\prime}(p^{a+t+v}-1-i)}g^{i})\geq a+t+v$

by lifting, so

$\sum_{i=0}^{p^{a+t+v}-1}x^{\prime
k^{\prime}(u-1-i)}g^{i}-x^{k^{\prime}(p^{a+t+v}-1-i)}g^{i}\equiv 0\mod
p^{3a+2t+v+1}$

showing that

$\sum_{i=0}^{p^{a+t+v}-1}x^{\prime
k^{\prime}(u-1-i)}g^{i}\equiv\sum_{i=0}^{p^{a+t+v}-1}x^{k^{\prime}(p^{a+t+v}-1-i)}g^{i}\mod
p^{2a+t+v+1}$

In all, we get that

$\sum_{i=0}^{qp^{a+t+v}-1}x^{\prime k^{\prime}(u-1-i)}g^{i}\equiv
q\sum_{i=0}^{p^{a+t+v}-1}x^{\prime k^{\prime}(p^{a+t+v}-1-i)}g^{i}\mod
p^{2a+t+v+1}$

Now we know that

$\sum_{i=qp^{a+t+v}}^{u-1}x^{\prime k^{\prime}(u-1-i)}g^{i}\equiv
g^{q}\sum_{i=0}^{y-1}x^{\prime k^{\prime}(y-1-i)}g^{i}\mod p^{2a+t+v+1}$

by shifting since

$\sum_{i=qp^{a+t+v}}^{u-1}x^{\prime
k^{\prime}(u-1-i)}g^{i}\equiv\sum_{i=0}^{y-1}x^{\prime
k^{\prime}(y-1-i)}g^{i+q}\mod p^{2a+t+v+1}$

Repeating the same arguments for $u^{\prime}$ and adding finishes the proof of
the claim.

Now we claim that this is just

$\frac{u+u^{\prime}}{p^{a+t+v}}\sum_{i=0}^{p^{a+t+v}-1}x^{\prime
k^{\prime}(p^{a+t+v}-1-i)}g^{i}\mod p^{2a+1+t+v}$

If $v=0$, then since $p^{a+t}|u,u^{\prime}$ it immediately follows that
$y=y^{\prime}=0$ and $q+q^{\prime}=\frac{u+u^{\prime}}{p^{a+t+v}}$, proving
the claim. Now suppose $v>0$, meaning that $p^{a+t+v}\nmid u,u^{\prime}$.
Since $0<y+y^{\prime}<2p^{a+t+v}$ and $p^{a+t+v}|y+y^{\prime}$, it follows
that $y+y^{\prime}=p^{a+t+v}$ and so
$q+q^{\prime}=\frac{u+u^{\prime}}{p^{a+t+v}}-1$. Hence it suffices to show
that

$g^{q}\sum_{i=0}^{y-1}x^{\prime
k^{\prime}i}g^{y-1-i}+g^{q^{\prime}}\sum_{i=0}^{y^{\prime}-1}x^{\prime
k^{\prime}i}g^{y^{\prime}-1-i}-\sum_{i=0}^{p^{a+t+v}-1}x^{\prime
k^{\prime}i}g^{p^{a+t+v}-1-i}\equiv 0\mod p^{2a+t+v+1}$

Note that

$\sum_{i=0}^{y-1}x^{\prime k^{\prime}i}g^{p^{a+t+v}-1-i}-g^{q}x^{\prime
k^{\prime}i}g^{y-1-i}\equiv-(g^{qp^{a+t+v}+y}-g^{p^{a+t+v}})\sum_{i=0}^{y-1}x^{\prime
k^{\prime}i}g^{-1-i}\equiv 0\mod p^{2a+v+t+1}$

after noting that $qp^{a+t+v}+y\equiv u\equiv 1\mod d$. In addition, since

$\sum_{i=y}^{p^{a+t+v}-1}x^{\prime
k^{\prime}i}g^{p^{a+t+v}-1-i}=\sum_{i=0}^{y^{\prime}-1}x^{\prime
k^{\prime}(i+y)}g^{y^{\prime}-1-i}$ $\sum_{i=y}^{p^{a+t+v}-1}x^{\prime
k^{\prime}i}g^{p^{a+t+v}-1-i}-g^{q^{\prime}}\sum_{i=0}^{y^{\prime}-1}x^{\prime
k^{\prime}i}g^{y^{\prime}-1-i}$

just becomes

$(x^{\prime k^{\prime}y}-g^{q^{\prime}})\sum_{i=0}^{y^{\prime}-1}x^{\prime
k^{\prime}i}g^{y^{\prime}-1-i}\mod p^{2a+t+v+1}$

Now noting that $g^{q^{\prime}}\equiv g^{q^{\prime}p^{a+t+v}}\equiv x^{\prime
k^{\prime}q^{\prime}p^{a+t+v}}\equiv x^{\prime
k^{\prime}(u^{\prime}-y^{\prime})}\mod p^{2a+t+v+1}$, it remains to show that

$(x^{\prime k^{\prime}y}-x^{\prime
k^{\prime}(u^{\prime}-y^{\prime})})\sum_{i=0}^{y^{\prime}-1}x^{\prime
k^{\prime}i}g^{y^{\prime}-1-i}\equiv 0\mod p^{2a+t+v+1}$

However, $v_{p}(x^{\prime k^{\prime}y}-x^{\prime
k^{\prime}(u^{\prime}-y^{\prime})})=v_{p}(x^{\prime
k^{\prime}(u^{\prime}-p^{a+t+v})}-1)\geq a+1+a+t$ and

$\sum_{i=0}^{y^{\prime}-1}x^{\prime k^{\prime}i}g^{y^{\prime}-1-i}\equiv 0\mod
p^{a+t}$

so at last we conclude that

$\sum_{i=y}^{p^{a+t+v}-1}x^{\prime
k^{\prime}i}g^{p^{a+t+v}-1-i}-g^{q^{\prime}}\sum_{i=0}^{y^{\prime}-1}x^{\prime
k^{\prime}i}g^{y^{\prime}-1-i}\equiv 0\mod p^{2a+t+v+1}$

completing the proof of the claim.

Therefore, the whole problem reduces to showing that

$\frac{u+u^{\prime}}{p^{a+t+v}}\sum_{i=0}^{p^{a+t+v}-1}x^{\prime
k^{\prime}(p^{a+t+v}-1-i)}g^{i}\equiv 0\mod p^{2a+1+t+v}$

Note that $u+u^{\prime}$ has valuation at least $2a+1+t+v$, and so
$\frac{u+u^{\prime}}{p^{a+t+v}}$ has valuation at least $a+1$, so it reduces
to showing that

$\sum_{i=0}^{p^{a+t+v}-1}x^{\prime k^{\prime}(p^{a+t+v}-1-i)}g^{i}\equiv 0\mod
p^{a+t+v}$

which is an immediate consequence of lifting.

#### Corollary 1

Let $p$ be an odd prime and $x,y\mod p$ be residues such that $x\equiv
y\mu\mod p$ for some $\gcd(k,p-1)$th root of unity $\mu$. Let
$\mathfrak{S}_{x}$ denote the multiset of residues $\mod p^{3a+t+v+2}$
attained by $n^{(k+p^{a}(p-1))p^{t}}+n^{(k-p^{a}(p-1))p^{t}}$ ($k,a,t$ as in
Theorem $1$) as $n\equiv x\mod p$ varies from $1$ to $p^{a+1}$ with
multiplicity and similarly for $\mathfrak{S}_{y}$. Then
$\mathfrak{S}_{x}=\mathfrak{S}_{y}$.

The main content of Theorem $1$ shows that if $x^{\prime k^{\prime}}\equiv
g\mod p^{a+1}$, then for any invertible $n$ and $n^{\prime}$ with
$n^{\prime}\equiv nx^{\prime}\mod p^{a+1}$,

$n^{\prime(k+p^{a}(p-1))p^{t}}+n^{\prime(k-p^{a}(p-1))p^{t}}\equiv
g(n^{(k+p^{a}(p-1))p^{t}}+n^{(k-p^{a}(p-1))p^{t}})\mod p^{3a+t+v+2}$

Setting $g\equiv 1\mod p^{a+1}$ and noting that $x^{\prime k^{\prime}}\equiv
1\mod p^{a+1}$ iff $x^{\prime\gcd(p-1,k)}\equiv 1\mod p^{a+1}$ (as
$\gcd(p-1,k)=\gcd(p-1,k^{\prime})$, it follows that for any $\gcd(p-1,k)$th
root of unity $x^{\prime}\mod p^{a+1}$,

$n^{\prime(k+p^{a}(p-1))p^{t}}+n^{\prime(k-p^{a}(p-1))p^{t}}\equiv
n^{(k+p^{a}(p-1))p^{t}}+n^{(k-p^{a}(p-1))p^{t}}\mod p^{3a+t+v+2}$

Now reducing the $\gcd(p-1,k)$th roots of unity $\mod p$ shows that their
reductions are $\gcd(p-1,k)$th roots of unity $\mod p$. Furthermore, if two
were congruent $\mod p$, say $y,y^{\prime}$, then
$y^{\gcd(p-1,k)}-y^{\prime\gcd(p-1,k)}\equiv 0\mod p^{a+1}$ implies that
$y\equiv y^{\prime}\mod p^{a+1}$ by lifting. Hence their reductions represent
each $\gcd(p-1,k)$th root of unity $\mod p$ exactly once, and so let
$x^{\prime}$ be the unique $\gcd(p-1,k)$th root of unity $\mod p^{a+1}$ with
$x^{\prime}\equiv\mu\mod p$. Then note that associating $n$ and $n^{\prime}$
gives a bijection between positive integers $\leq p^{a+1}$ with $n\equiv x\mod
p^{a+1}$ and $n^{\prime}\equiv y\mod p^{a+1}$ since the map is injective and
the sets are the same size, but then since

$n^{\prime(k+p^{a}(p-1))p^{t}}+n^{\prime(k-p^{a}(p-1))p^{t}}\equiv
n^{(k+p^{a}(p-1))p^{t}}+n^{(k-p^{a}(p-1))p^{t}}\mod p^{3a+t+v+2}$

each corresponding pair yields the same element $\mod p^{3a+t+v+2}$,
immediately showing $\mathfrak{S}_{x}=\mathfrak{S}_{y}$.

We can also now prove a congruence that will be useful for the “hard” case of
the Kummer Congruences.

#### Corollary 2

Let $a,b,t$ be nonnegative integers with $p^{a+t}|b$ and $p-1\nmid b$. Then we
have that

$\displaystyle\sum_{n=1}^{p^{a+1}}n^{b}(n^{p^{a+t}(p-1)}-1)^{2}\equiv 0\mod
p^{3a+t+v+2}$

Let $S$ be the sum of all elements in $\mathfrak{S}\mod p^{3a+t+v+2}$. Then
since $p-1\nmid b$, we can find a primitive $d$th root of unity $\mod
p^{3a+t+v+2}$ such that $g\mathfrak{S}=\mathfrak{S}$ for some $d>1$. If
$n^{(k+p^{a}(p-1))p^{t}}+n^{(k-p^{a}(p-1))p^{t}}$ is noninvertible, then
taking $\mod p$ implies $p|n$, but since $2p^{a+t}\geq 3a+2t+2$, we get that
it is $0\mod p^{3a+t+v+2}$, implying that $S\equiv gS\mod p^{3a+t+v+2}$. It
suffices to show that $g\not\equiv 1\mod p$. Suppose it is and note that
$p^{3a+t+v+2}|g^{d}-1=(g-1)(g^{d-1}+\cdots+1)$. We have that
$g^{d-1}+\cdots+1\equiv d\mod p$, which implies the second term is coprime to
$p$, and so $p^{3a+t+v+2}|g-1$, contradicting primitivity. Hence

$\sum_{n=1}^{p^{a+1}}n^{(k+p^{a}(p-1))p^{t}}+n^{(k-p^{a}(p-1))p^{t}}\equiv
0\mod p^{3a+t+v+2}$

and so Lemma $2$ implies that

$\sum_{n=1}^{p^{a+1}}n^{(k-p^{a}(p-1))p^{t}}(n^{p^{a+t}(p-1)}-1)^{2}\equiv
0\mod p^{3a+t+v+2}$

Now if $p^{a+t}|b$, then $b\equiv kp^{t}\mod p^{a+t}(p-1)$ for some $k$ with
$p^{2a+1}|k$, and so writing $b=kp^{t}+lp^{a+t}(p-1)$ for some integer $v$,
meaning that our sum is just

$\sum_{n=1}^{p^{a+1}}n^{kp^{t}+lp^{a+t}(p-1)}(n^{p^{a+t}(p-1)}-1)^{2}-n^{kp^{t}-p^{a+t}(p-1)}(n^{p^{a+t}(p-1)}-1)^{2}\mod
p^{3a+t+v+2}$

However, noting that $n^{|l+1|p^{a+t}(p-1)}-1$ is divisible by
$n^{p^{a+t}(p-1)}-1$, the sum becomes

$\sum_{n=1}^{p^{a+1}}n^{kp^{t}+\min(l,-1)p^{a+t}(p-1)}f(n)(n^{p^{a+t}(p-1)}-1)^{3}$

for some $f(n)$ in $\mathbb{Z}$, which vanishes $\mod p^{3a+t+v+2}$ by Euler’s
Theorem.

From this, we can deduce a linearity relation that will quickly imply the
Kummer congruences.

## Theorem 2

Let $k$ be a positive integer not divisible by $p-1$ with $2p^{2a+1}|k$. Then
we have that for any nonnegative integer $t$ and integer $r$ such that
$k+p^{a}(p-1)r>0$,

$\displaystyle\sum_{n=1}^{p^{a+1}}n^{(k+p^{a}(p-1)r)p^{t}}\equiv
r\sum_{n=1}^{p^{a+1}}n^{(k+p^{a}(p-1))p^{t}}\mod p^{3a+t+v+2}$

By Lemma $2$, we have

$\displaystyle\sum_{n=1}^{p^{a+1}}n^{kp^{t}}\equiv 0\mod p^{3a+t+v+2}$

and so the claim holds for $r=0$.

Now we suppose it holds for $r-1,r$. We show that it holds for $r+1$.
Corollary $2$ tells us that

$\displaystyle\sum_{n=1}^{p^{a+1}}n^{{k+p^{a}(p-1)(r-1)}p^{t}}(n^{p^{a+t}(p-1)}-1)^{2}\equiv
0\mod p^{3a+t+v+2}$

and so

$\displaystyle\sum_{n=1}^{p^{a+1}}n^{(k+p^{a}(p-1)(r+1))p^{t}}\equiv\sum_{n=1}^{p^{a+1}}2n^{(k+p^{a}(p-1)r)p^{t}}-n^{(k+p^{a}(p-1)(r-1))p^{t}}\mod
p^{3a+t+v+2}$

By the induction hypothesis, we get

$\sum_{n=1}^{p^{a+1}}2n^{(k+p^{a}(p-1)r)p^{t}}-n^{k+p^{a}(p-1)(r-1))p^{t}}\equiv(r+1)\sum_{n=1}^{p^{a+1}}n^{(k+p^{a}(p-1))p^{t}}\mod
p^{3a+t+v+2}$

as desired. A similar argument going backwards shows the claim for all
negative $r$ with $k+p^{a}(p-1)r>0$ since $k+p^{a}(p-1)r>0$ implies
$(k+p^{a}(p-1)r)p^{t}\geq 2p^{a+t}\geq 3a+2t+2$.

## The proof of the congruences

To prove the general case, we break it into three cases, the first of which is
independent of the other two and the other two of which we glue together at
the end. Throughout the proof, we also assume $s>r$.

Firstly, note that the $1-p^{s-1}$ term can always be excluded from the proofs
assuming $s>r$ since then $s\geq r+p^{a}(p-1)>2+2^{a}>a+2$, meaning that
$1-p^{s-1}$ always vanishes $\mod p^{a+1}$, and so since $\frac{B_{s}}{s}$ is
a $p$-integer by Corollary $2$, we conclude that
$(1-p^{s-1})\frac{B_{s}}{s}\equiv\frac{B_{s}}{s}\mod p^{a+1}$.

Case 1: $v_{p}(r)<a$ (which means $v_{p}(s)=v_{p}(r)<a$).

For the sake of this case, we assume that the congruences hold $\mod p^{a}$
when proving them $\mod p^{a+1}$, noting that this case is empty for $a=0$.

Consider the sum

$\displaystyle\sum_{n=1}^{p^{a}}n^{r}(n^{p^{a-1}(p-1)}-1)^{p}$

For the terms with $p$ not dividing $n$, Euler’s Theorem implies that each
term is divisible by $p^{ap}$ and hence $p^{2a+v_{p}(r)+2}$ because $ap\geq
3p\geq 2a+v_{p}(r)+1$. Now if $p|n$, then $p^{p^{a-1}(p-1)}|n^{p^{a-1}(p-1)}$,
and so since $p^{a-1}(p-1)\geq 4(5^{a-1})>3a$, we conclude that

$\sum_{m=1}^{p^{a-1}}(pm)^{r}((pm)^{p^{a-1}(p-1)}-1)^{p}\equiv-p^{r}\sum_{m=1}^{p^{a-1}}m^{r}\mod
p^{2a+v_{p}(r)+1}$

We see that

$\sum_{m=1}^{p^{a-1}}m^{r}\equiv p^{a-1}B_{r}\mod p^{2a-2+v_{p}(r)+1}$

by Lemma $1$, and so

$-p^{r}\sum_{m=1}^{p^{a-1}}m^{r}\equiv-p^{r+a-1}B_{r}\mod p^{2a+v_{p}(r)+1}$

(after noting $r\geq 2$).

Now as before, the sum expands to

$\displaystyle\sum_{n=1}^{p^{a}}\sum_{i=0}^{p}\binom{p}{i}(-1)^{p-i}n^{r+ip^{a-1}(p-1)}$

Applying Lemma $1$, we get that

$\displaystyle\sum_{i=0}^{p}\binom{p}{i}(-1)^{p-i}B_{r+ip^{a-1}(p-1)}\equiv-p^{r-1}B_{r}\mod
p^{a+v_{p}(r)+1}$

and so

$\displaystyle
B_{r+p^{a}(p-1)}-B_{r}+\sum_{i=1}^{p-1}\binom{p}{i}(-1)^{p-i}B_{r+ip^{a-1}(p-1)}\equiv-p^{r-1}B_{r}\mod
p^{a+v_{p}(r)+1}$

Now we have that

$\displaystyle(1-p^{r-1})\frac{B_{r}}{r}\equiv\frac{B_{r+ip^{a-1}(p-1)}}{r+ip^{a-1}(p-1)}\mod
p^{a}$

so

$B_{r+ip^{a-1}(p-1)}\equiv(1-p^{r-1})(r+ip^{a-1}(p-1))\frac{B_{r}}{r}\mod
p^{a+v_{p}(r)}$

since $v_{p}(r+ip^{a-1}(p-1))\geq v_{p}(r)$. Now $\mod p^{a+v_{p}(r)+1}$,

$\displaystyle\sum_{i=1}^{p-1}\binom{p}{i}(-1)^{p-i}B_{r+ip^{a-1}(p-1)}\equiv(1-p^{r-1})\sum_{i=1}^{p-1}\binom{p}{i}(-1)^{p-i}(r+ip^{a-1}(p-1))\frac{B_{r}}{r}$

and noting

$\displaystyle\sum_{i=1}^{p-1}\binom{p}{i}(-1)^{p-i}i=-p$

gives $-(1-p^{r-1})p^{a}(p-1)\frac{B_{r}}{r}$. We have that
$v_{p}(p^{a+r-1}\frac{B_{r}}{r})\geq a+r-1\geq a+v_{p}(r)+1$, invoking Lemma
$3$ and noting $r\geq v_{p}(r)+2$, and so we conclude that
$-(1-p^{r-1})p^{a}(p-1)\frac{B_{r}}{r}\equiv-p^{a}(p-1)\frac{B_{r}}{r}\mod
p^{a+v_{p}(r)+1}$. Putting everything together gives
$B_{r+p^{a}(p-1)}\equiv(r+p^{a}(p-1))\frac{B_{r}}{r}-p^{r-1}B_{r}\mod
p^{a+v_{p}(r)+1}$. Lemma $3$ again gives $v_{p}(p^{a+r-1}B_{r})\geq
a+r-1+v_{p}(r)$, and so since $r\geq 2$, we see that
$p^{a}(p-1)(p^{r-1}B_{r})$ vanishes $\mod p^{a+v_{p}(r)+1}$, and so

$B_{r+p^{a}(p-1)}\equiv(r+p^{a}(p-1))\frac{B_{r}}{r}-(r+p^{a}(p-1))p^{r-1}\frac{B_{r}}{r}\mod
p^{a+v_{p}(r)+1}$

Finally, since $v_{p}(r)=v_{p}(r+p^{a}(p-1))$, we conclude that

$\frac{B_{r+p^{a}(p-1)}}{r+p^{a}(p-1)}\equiv(1-p^{r-1})\frac{B_{r}}{r}\mod
p^{a+1}$

finishing this case by iterating this congruence if necessary.

To handle the next case, we must make use of Theorem $2$ instead of just using
direct induction. This is because if we try to repeat the same argument that
we did for the case of $v_{p}(r)<a$, we will run into the problem where the
binomial coefficients only give one extra power of $p$ when we actually need
$2$ (one to get up to $a$ and one because the powers separating $r$ are larger
than those we would have in the sum)

Note that the $1-p^{r-1}$ may now also be excluded from now on since $r\geq
2p^{a}\geq 2^{a+1}\geq a+2$ and the same arguments used for $s$.

Case 2: $v_{p}(r)=v_{p}(s)\geq a$.

Theorem $2$ immediately implies that $\mod p^{4a+t+2}$,

$(k+p^{a}(p-1))\displaystyle\sum_{n=1}^{p^{a+1}}n^{(k+p^{a}(p-1)b)p^{t}}\equiv(k+bp^{a}(p-1))\sum_{n=1}^{p^{a+1}}n^{(k+p^{a}(p-1))p^{t}}$

Now, by Lemma $1$, we have that

$(k+p^{a}(p-1))B_{(k+bp^{a}(p-1))p^{t}}\equiv(k+bp^{a}(p-1))B_{(k+p^{a}(p-1))p^{t}}\mod
p^{3a+t+1}$

and so if $r$ is not a multiple of $p$, then dividing by
$p^{t}(k+p^{a}(p-1))(k+bp^{a}(p-1))$ (of valuation $2a+t$) immediately gives
the theorem.

Case 3: Branching

The key now is to consider the object

$\displaystyle\sum_{n=1}^{p^{a+1}}n^{kp^{t}}(n^{p^{a+t-1}(p-1)}-1)^{p}$

which vanishes $\mod p^{p(a+t)}$ and hence $\mod p^{3a+t+2}$. By the binomial
theorem, we have that this is just

$\displaystyle\sum_{n=1}^{p^{a+1}}n^{(k+p^{a}(p-1))p^{t}}-n^{kp^{t}}+\sum_{n=1}^{p^{a+1}}\sum_{i=1}^{p-1}\binom{p}{i}(-1)^{p-i}n^{kp^{t}+ip^{a+t-1}(p-1)}$

The double sum is just

$\sum_{i=1}^{p-1}\binom{p}{i}(-1)^{p-i}(\frac{k}{p^{a}}+i)\sum_{n=1}^{p^{a+1}}n^{(k+p^{a}(p-1))p^{t-1}}\mod
p^{3a+t+2}$

by Theorem $2$, so the whole sum just reduces to

$-p\sum_{n=1}^{p^{a+1}}n^{(k+p^{a}(p-1))p^{t-1}}\mod p^{3a+t+2}$

This implies that

$\displaystyle\sum_{n=1}^{p^{a+1}}n^{(k+p^{a}(p-1))p^{t}}\equiv
p\sum_{n=1}^{p^{a+1}}n^{(k+p^{a}(p-1))p^{t-1}}\mod p^{3a+t+2}$

Using Lemma $1$ once again, we get that

$B_{(k+p^{a}(p-1))p^{t}}\equiv pB_{(k+p^{a}(p-1)r)p^{t-1}}\mod p^{2a+t+1}$

Finally, dividing by $(k+p^{a}(p-1))p^{t}$ (of valuation $a+t$) immediately
gives

$\frac{B_{(k+p^{a}(p-1))p^{t}}}{(k+p^{a}(p-1))p^{t}}\equiv\frac{B_{(k+p^{a}(p-1))p^{t-1}}}{(k+p^{a}(p-1))p^{t-1}}\mod
p^{a+1}$

completing the branching step and hence the proof of the full Kummer
congruences.

Thus we’ve completed our goal of proving the Kummer Congruences by
understanding the local behavior and interpreting that in the context of a
global sum.

Notice that the set $\mathfrak{S}$ comes equipped with a
$(\mathbb{Z}/p^{3a+t+v+2}\mathbb{Z})^{*}$ action and that the content of
Theorem $3$ is that the subgroup $\mu_{d}$ of $d$th roots of unity is
contained in $\text{Stab}(\mathfrak{S})$ under this action. We now classify
$\text{Stab}(\mathfrak{S})$ more completely. We need some lemmas.

Before starting the proof, we make a quick note about $p$-adic valuations and
polynomials. For a polynomial $g(x)$, we say that for a residue class $s\mod
p^{k}$, $v_{p}(g(s))\geq l$ iff $g(s^{\prime})\equiv 0\mod p^{l}$ for every
positive integer $s^{\prime}$ in the residue class $s\mod p^{k}$ (in general,
we can say that $v_{p}(g(s))=\min_{s^{\prime}\equiv s\mod
p^{k}}v_{p}(g(s^{\prime}))$.

### Lemma 4

Let $p$ be an odd prime and set
$f(x)=x^{(k+p^{a}(p-1))p^{t}}+x^{(k-p^{a}(p-1))p^{t}}$ as polynomials. Then
for any positive integers $m,n$ with $n$ coprime to $p$, we have that
$v_{p}(f^{(m)}(n))\geq 2a+t+v$. Furthermore, if $m=1$, we have that
$v_{p}(f^{(m)}(n))\geq 2a+t+1+v$, with equality always holding if $v<t$. For
$m=2$, we have that $v_{p}(f^{(m)}(n))=2a+2t$ if $v=t$ and that
$v_{p}(f^{(m)}(n)\geq 2a+t+v+1$ if $v<t$. Finally, for $m=3$ and $v=t$, we
have that $v_{p}(f^{(m)}(n))=2a+2t$ unless $p=3$, in which case it is $\geq
2a+2t+1$, and for $v<t$, $v_{p}(f^{(m)}(n))\geq 2a+t+v+1$.

Expanding $f^{(m)}(n)$, we get

$n^{(k+p^{a}(p-1))p^{t}-m}\prod_{i=0}^{m-1}((k+p^{a}(p-1))p^{t}-i)+n^{(k-p^{a}(p-1))p^{t}-m}\prod_{i=0}^{m-1}((k-p^{a}(p-1))p^{t}-i)$

Since $m$ is a positive integer, each product has valuation at least $a+t$,
and so Euler’s Theorem implies that this is just

$n^{kp^{t}-m}(\prod_{i=0}^{m-1}((k+p^{a}(p-1))p^{t}-i)+\prod_{i=0}^{m-1}((k-p^{a}(p-1))p^{t}-i))\mod
p^{2a+2t+1}$

Then note that the sum of products is just

$p^{a+t}(p-1)(\prod_{i=1}^{m-1}(p^{a+t}(p-1)-i)+(-1)^{m}\prod_{i=1}^{m-1}(p^{a+t}(p-1)+i))\mod
p^{2a+t+1+v}$

Plugging in $m=1$ immediately gives the inequality in the second statement.
For the equality case, plugging in $m=1$ into the expression

$n^{kp^{t}-m}\prod_{i=0}^{m-1}((k+p^{a}(p-1))p^{t}-i)+\prod_{i=0}^{m-1}((k-p^{a}(p-1))p^{t}-i))$

gives $f^{\prime}(n)\equiv 2n^{kp^{t}-m}kp^{t}\mod p^{2a+2t+1}$, and if
$v_{p}(k)<2a+1+t$, then $v_{p}(kp^{t})<2a+2t+1$, so the valuation of
$f^{\prime}(n)$ is exactly $v_{p}(k)+t=2a+t+1+v$ in this case.

For the case of $m=2$, we have that the expression is $2p^{2a+2t}(p-1)\mod
p^{2a+t+v+1}$. If $t=v$, then we get that $2p^{2a+2t}(p-1)\mod p^{2a+2t+1}$,
and so $v_{p}(f^{\prime\prime}(n))=2a+2t$. If $v<t$, note that $2a+2t\geq
2a+t+v+1$, and so $2p^{2a+2t}(p-1)\equiv 0\mod p^{2a+t+v+1}$, as desired.

Plugging in $m=3$,
$p^{a+t}(p-1)((p^{a+t}(p-1)-1)(p^{a+t}(p-1)-2)-(p^{a+t}(p-1)+1)(p^{a+t}(p-1)+2))\equiv-6p^{2(a+t)}(p-1)^{2}\mod
p^{2a+t+v+1}$. If $v=t$, then we conclude that $f^{\prime\prime\prime}(n)$ has
valuation $2a+2t$ for $p\neq 3$ and vanishes $\mod p^{2a+2t+1}$ for $p=3$. If
$v<t$, then $f^{\prime\prime\prime}(n)$ automatically vanishes $\mod
p^{2a+t+v+1}$ once again.

For the first, it suffices to show that

$\prod_{i=1}^{m-1}(p^{a+t}(p-1)-i)+(-1)^{m}\prod_{i=1}^{m-1}(p^{a+t}(p-1)+i)\equiv
0\mod p^{a+v}$

Reducing $\mod p^{a+v}$ gives

$(-1)^{m-1}(m-1)!+(-1)^{m}(m-1)!\equiv 0\mod p^{a+v}$

as desired.

#### Corollary 3

Let $r$ be a nonnegative integer and $k\geq r$ be a positive integer. Then for
any integer $s$ coprime to $p$, $f(s+p^{k}x)\equiv
f(s)+f^{\prime}(s)p^{k}x\mod p^{2a+t+v+2k}$. Furthermore, if $v<t$, then we
have the stronger $f(s+p^{k}x)\equiv f(s)+f^{\prime}(s)p^{k}x\mod
p^{2a+t+v+2k+1}$

Let $\text{deg}(f)=w$. We have that for any positive integer $k$,

$f(s+p^{k}x)=\sum_{n=0}^{w}f^{(n)}(s)\frac{(p^{k}x)^{n}}{n!}$

Lemma $4$ implies that $f^{(n)}(s)\geq 2a+t+v$, and so for $n>1$, we have that
$v_{p}(f^{(n)}(s)\frac{(p^{k}x)^{n}}{n!})\geq 2a+t+v+2k$, and so

$f(s+p^{k}x)\equiv f(s)+f^{\prime}(s)p^{k}x\mod p^{2a+t+v+2k}$

Now suppose that $v<t$. Then we claim that
$v_{p}(f^{(n)}(s)\frac{(p^{k}x)^{n}}{n!})\geq 2a+t+v+2k+1$ for all $n\geq 2$.
For $n=2$ or $3$, Lemma $4$ implies that $v_{p}(f^{(n)}(s))\geq 2a+t+v+1$, and
so $v_{p}(f^{(n)}(s)\frac{(p^{k}x)^{n}}{n!})\geq 2a+t+v+1+2k$ for $n=2$ and
$2a+t+v+1+3k-1\geq 2a+t+v+1+2k$ for $n=3$, proving the claim in these cases.
For $n>3$, we have that $v_{p}(f^{(n)}(s)\frac{(p^{k}x)^{n}}{n!})\geq
2a+t+v+4k-1\geq 2a+t+v+2k+1$, finishing the proof in all cases and concluding
the case $v<t$.

### Lemma 5

Suppose $v=t$. Then for each nonnegative integer $s$ with $0\leq s\leq a$,
there exist exactly $p^{a-s}(p-1)$ positive integers $u\leq p^{a+1}$ with $u$
coprime to $p$ and $v_{p}(f^{\prime}(u))\geq 2a+2t+s+1$.

We show by induction that there are $p-1$ solutions $u\mod p^{s+1}$ to
$f^{\prime}(x)\equiv 0\mod p^{2a+2t+s+1}$ for any nonnegative integer $s$. The
base case of $s=0$ is immediate from Lemma $4$. Suppose that the statement
holds for $s-1$ and let $u^{\prime}$ be a solution to $f^{\prime}(x)\equiv
0\mod p^{2a+2t+s}$. We show that $u^{\prime}$ can be lifted to $u\mod p^{s+1}$
with $f^{\prime}(u)\equiv 0\mod p^{2a+2t+s+1}$. By Taylor series expansion, we
have that

$f^{\prime}(u^{\prime}+p^{s}x)=\sum_{n=0}^{w-1}f^{(n+1)}(u^{\prime})\frac{(p^{s}x)^{n}}{n!}$

Taking out the first two terms gives

$f^{\prime}(u^{\prime})+f^{\prime\prime}(u^{\prime})p^{s}x+\sum_{n=2}^{w-1}f^{(n+1)}(u^{\prime})\frac{(p^{s}x)^{n}}{n!}$

Lemma $4$ implies that for any $n$, $f^{(n+1)}(u^{\prime})$ vanishes $\mod
p^{2a+2t}$, and since for $n>1$ we have that
$v_{p}(\frac{(p^{s}x)^{n}}{n!})\geq 2s\geq s+1$ since $s\geq 1$ by assumption,
$f^{(n+1)}(u^{\prime})\frac{(p^{s}x)^{n}}{n!}$ vanishes $\mod p^{2a+2t+s+1}$.
Hence $f^{\prime}(u^{\prime}+p^{s}x)\equiv
f^{\prime}(u^{\prime})+f^{\prime\prime}(u^{\prime})p^{s}x\mod p^{2a+2t+s+1}$,
immediately implying that there is a unique lift of $u^{\prime}$. Furthermore,
for any $u^{\prime}\mod p^{s}$ with $f^{\prime}(u^{\prime})\not\equiv 0\mod
p^{2a+2t+s}$, we have that

$f^{\prime}(u^{\prime}+p^{s}x)\equiv
f^{\prime}(u^{\prime})+f^{\prime\prime}(u^{\prime})p^{s}x\mod p^{2a+2t+s+1}$

but then
$v_{p}(f^{\prime}(u^{\prime}))<v_{p}(f^{\prime\prime}(u^{\prime})p^{s})$, and
so there is no lift giving a solution here, meaning that all solutions $\mod
p^{s+1}$ are lifts of solutions $\mod p^{s}$. With this proven, then note that
there are $p^{a-s}(p-1)$ positive integers $k\leq p^{a+1}$ with $k\equiv u\mod
p^{a+1}$, completing the proof.

## Theorem 3

Given the conditions of Theorem $1$, $\text{Stab}(\mathfrak{S})=\mu_{n}$ where
$n=dp^{a}$ if $v<t$ and $n=d$ if $v=t$.

Since $\text{Stab}(\mathfrak{S})$ is defined by the group action of
$(\mathbb{Z}/p^{3a+t+v+2}\mathbb{Z})^{*}$ on $\mathfrak{S}$, we have that
$\text{Stab}(\mathfrak{S})$ is a subgroup of
$(\mathbb{Z}/p^{3a+t+v+2}\mathbb{Z})^{*}$, and so is in particular cyclic.
Letting $P$ be the $p$-Sylow subgroup of $\text{Stab}(\mathfrak{S})$, we can
write $\text{Stab}(\mathfrak{S})=PG$ for some abelian group $G$ of order
coprime to $p$ (where the product is isomorphic to a direct one). $G$, then,
consists of all elements of order coprime to $p$ and hence is of order
dividing $p-1$.

Since $p$ is odd, the subgroup of such elements is automatically $\mu_{p-1}$
since $(\mathbb{Z}/p^{3a+t+v+2}\mathbb{Z})^{*}$ is cyclic. Now note that if
$g\in\text{Stab}(\mathfrak{S})$, its reduction $g^{\prime}$ $\mod p$ is
automatically in $\text{Stab}(\mathfrak{S^{\prime}})$, where $S^{\prime}$ is
$S$ defined $\mod p$. We have that

$n^{(k+p^{a}(p-1))p^{t}}+n^{(k-p^{a}(p-1))p^{t}}\equiv 2n^{k^{\prime}}\mod p$

and so $\mathfrak{S^{\prime}}$ consists of elements of the form
$2n^{\gcd(k,p-1)}$, which is a coset of $\mu_{d}^{\prime}$. Hence
$\text{Stab}(\mathfrak{S^{\prime}})=\mu_{d}^{\prime}$, meaning
$g^{\prime}\in\mu_{d}^{\prime}$. However since
$\text{ord}_{p}(g)|\text{ord}_{p^{3a+t+v+2}}(g)$ and
$\text{ord}_{p^{3a+t+v+2}}(g)$ is coprime to $p$, lifting implies that
$\text{ord}_{p}(g)=\text{ord}_{p^{3a+t+v+2}}(g)$ and hence $g\in\mu_{d}$.
Theorem $2$ then implies that $\mu_{d}\subset\text{Stab}(\mathfrak{S})$ and in
particular that $\mu_{d}\subset G$, so we conclude that $G=\mu_{d}$.

It remains to analyze $P$. First we find a characterization for when $P$ has
order $\leq p^{r}$ for some nonnegative integer $r$. Recalling that it is
cyclic, the condition is equivalent to there being no element of order
$p^{r+1}$. If $g$ were some element of order $p^{r+1}$, then it would
necessarily be $1\mod p^{3a+t+v+1-r}$, so for any element $s\in\mathfrak{S}$,
the orbit of $s$ under the subgroup generated by $g$ would be the set
consisting of $s+p^{3a+1+t+v-r}x\mod p^{3a+t+v+2}$ for $0\leq x\leq
p^{r+1}-1$. Hence each element $s^{\prime}$ with $s^{\prime}\equiv s\mod
p^{3a+1+t+v-r}$ must appear the same number of times in $\mathfrak{S}$.

Now we introduce some terminology. Given nonnegative integers $i\geq k$ and a
particular residue $s\mod p^{k}$, we define the values of $g(s)\mod
p^{2a+t+1+v+i}$ on the $p^{i-k}$ lifts of $s\mod p^{i}$ to be the branches of
$g(s)$ of level $i$. Furthermore, we define the layer $i$ of such an $s$ to be
the multiset of all branches of level $i$ (taken $\mod p^{2a+t+1+v+i}$). For
$t>j$, we call a multiset $\mathfrak{T}\mod p^{t}$ $j$-balanced if for any
element $s$ in $\mathfrak{T}$, the $p^{j}$ lifts of its reduction $\mod
p^{t-j}$ appear an equal number of times in $\mathfrak{T}$. Furthermore, we
say the $j$-character of a multiset is good if it is $j$-balanced and bad
otherwise. Note that deleting a good multisubset never changes the
$j$-character of the multiset.

Under these definitions, we have that $P$ has size $\leq p^{r}$ iff
$\mathfrak{S}$ is not $r+1$-balanced. Now note that $\mathfrak{S}$ is the
disjoint union of $\mathfrak{S}_{x}$ as $x$ ranges over the invertible
residues $\mod p$. Let $s_{1},...,s_{d}$ be a set of coset representatives of
$\mu_{\gcd(p-1,k)}$ in $(\mathbb{Z}/p\mathbb{Z})^{*}$ By Corollary $3$,
$\mathfrak{S}$ is just $\gcd(p-1,k)$ copies of the multiset
$\coprod_{i=1}^{d}\mathfrak{S}_{s_{i}}$. Let
$\mathfrak{V}=\coprod_{i=1}^{d}\mathfrak{S}_{s_{i}}$. It immediately follows
that $\mathfrak{V}$ is $r+1$-balanced iff $\mathfrak{S}$ is. Noting again that

$n^{(k+p^{a}(p-1))p^{t}}+n^{(k-p^{a}(p-1))p^{t}}\equiv 2n^{k^{\prime}}\mod p$

if $x\not\equiv y\mu\mod p$ for some $\gcd(p-1,k)$th root of unity $\mu$, it
follows that $\mathfrak{S}_{x}$ and $\mathfrak{S}_{y}$ are disjoint even upon
restricting $\mod p$. Hence for any $s\in\mathfrak{S}_{x}$, no element
$s^{\prime}$ with $s^{\prime}\equiv s\mod p^{3a+1+t-r}$ can appear in
$\mathfrak{S}_{y}$, and so $\mathfrak{V}$ is $r+1$-balanced iff for each $x$
and $s\in\mathfrak{S}_{x}$, all $s^{\prime}$ appear the same number of times
in $\mathfrak{S}_{x}$, which is precisely equivalent to $\mathfrak{S}_{x}$
being $r+1$-balanced for each $x$. Finally, note that if $\mathfrak{S}_{x}$ is
$r+1$-balanced, then it follows that $\mathfrak{S}^{\prime}_{x}$ is
$r+1-q$-balanced, where $\mathfrak{S}^{\prime}_{x}$ is the reduction of
$\mathfrak{S}_{x}\mod p^{3a+t+v+2-q}$.

Case 1: $v<t$

In this case, we show that $P$ is cyclic of order $p^{a}$. To show that $P$
has size $p^{a}$, we show that each $\mathfrak{S}_{x}$ is $a$-balanced but not
$a+1$-balanced. To show that $\mathfrak{S}_{x}$ is not $a+1$-balanced, note
that taking $k=1$ in Corollary $3$ gives $f(s+px)\equiv
f(s)+f^{\prime}(s)px\equiv f(s)\mod p^{2a+t+v+2}$, so the reduction of
$\mathfrak{S}_{x}\mod p^{2a+t+v+2}$ fails to be $1$-balanced, implying that
$\mathfrak{S}_{x}$ fails to be $a+1$-balanced for each $x$.

Now we show that each $\mathfrak{S}_{x}$ is $a$-balanced. Consider the
sequence of positive integers given by $a_{n}=\lceil\frac{a+1}{2^{n}}\rceil$
for all $n\geq 0$. Noting that $2\lceil x\rceil\geq\lceil 2x\rceil$, we see
that $2a_{n}\geq a_{n-1}$, and so we have that $f(s+p^{a_{n}}x)\equiv
f(s)+f^{\prime}(s)p^{a_{n}}x\mod p^{2a+t+v+a_{n-1}+1}$ for each positive
integer $n$ by the stronger version of Corollary $3$. We now show that
$\mathfrak{S}_{x}$ is $a_{0}-a_{k}=a+1-a_{k}$ balanced for each positive
integer $k$. We pick a residue $s\mod p^{a_{k}}$ and count how many times a
fixed lift $u$ of $f(s)\mod p^{2a+t+v+a_{k}+1}$ appears in the $a+1$ layer of
$s$. To do this, it suffices to find the number of $s^{\prime}\mod p^{a+1}$
with $s^{\prime}\equiv s\mod p^{a_{k}}$ and $f(s^{\prime})\equiv u\mod
p^{3a+t+2}$. We show by induction that there is a unique lift $s^{\prime}$ of
$s\mod p^{a_{r}}$ that $u\equiv f(s^{\prime})\mod p^{2a+t+v+a_{r}+1}$ for all
$0\leq r\leq n$. The base case of $r=k$ is immediate. Now suppose that it
holds for $n$ and we show it for $n-1$. Let $s_{n}$ be the unique lift of
$s\mod p^{a_{n}}$ such that $u\equiv f(s_{n})\mod p^{2a+t+v+a_{n}+1}$. Then we
have that $f(s_{n}+p^{a_{n}}x)\equiv f(s_{n})+f^{\prime}(s_{n})p^{a_{n}}x\mod
p^{2a+t+v+a_{n-1}+1}$. As $v_{p}(f^{\prime}(s_{n})p^{a_{n}})=2a+t+v+a_{n}+1$,
this uniquely determines $x\mod p^{a_{n-1}-a_{n}}$ and hence uniquely
determines $s_{n-1}$. Therefore, $s_{0}$ is uniquely determined $\mod
p^{a+1}$, showing that there are exactly $1$ lift of $s\mod p^{a+1}$
corresponding to a given $u$. Hence $\mathfrak{S}_{x}$ is
$a_{0}-a_{k}=a+1-a_{k}$ balanced. Taking $k$ large enough implies that
$\mathfrak{S}_{x}$ is $a$-balanced.

Case 2: $t=v$

From Corollary $3$, given that $v_{p}(f^{\prime}(s))=2a+2t+1+u$ for some
nonnegative integer $u$, $f(s+p^{k}x)\equiv f(s)\mod p^{2a+2t+k+u+1}$ for all
$k$ with $k\geq u+1$, showing that for $s^{\prime}$ the reduction of $s\mod
p^{k}$, $f(s^{\prime})$ is constant on its $k+u$ branches for such $k$.
However, we also have $f(s+p^{k}x)\equiv f(s)+f^{\prime}(s)p^{k}x\mod
p^{2a+2t+k+u+2}$ for $k\geq u+2$, so for such $k$, while the $k+u+1$ branches
of $f(s^{\prime})$ of a given $k+u$ branch of $f(s^{\prime})$ are constant,
the $k+u+1$ branches of different $k+u$ branches are distinct, hence
comprising all lifts $\mod p^{2a+k+u+1}$ of the $k+u$ branch of
$f(s^{\prime})$ once each, implying that the $k+u+1$ layer of $s^{\prime}$ is
$1$-balanced.

Now suppose that for some positive integer $u\leq\lfloor\frac{a}{2}\rfloor$,
we have an $s\mod p^{u+1}$ with $v_{p}(f^{\prime}(s))<2a+2t+u+1$. Then the
first condition and the proof of Lemma $5$ imply that for any lift
$s^{\prime}\mod p^{k}$ of $s$, we have
$v_{p}(f^{\prime}(s^{\prime}))=v_{p}(f^{\prime}(s))$. Letting
$v_{p}(f^{\prime}(s))=2a+2t+1+w$, $w<u$ implies that
$w\leq\lfloor\frac{a}{2}\rfloor-1$. In particular, selecting
$k=3a+2t+1-v_{p}(f^{\prime}(s^{\prime}))$, the inequality on $w$ implies that
$k=a-w\geq w+2$, so the above applies and hence the $a+1$ layer lying above
$s^{\prime}\mod p^{k}$ is $1$-balanced, noting that
$v_{p}(f^{\prime}(s))=v_{p}(f^{\prime}(s^{\prime}))$, and so the $a+1$ layer
of $s$, which is the disjoint union of the $a+1$ layers of its lifts $\mod
p^{k}$, is also balanced. Looking at each individual $\mathfrak{S}_{x}$, Lemma
$5$ now allows us to remove the $a+1$ layers above all but $1$ element $\mod
p^{\lfloor\frac{a}{2}\rfloor+1}$ without changing the $1$-quality of
$\mathfrak{S}$, so it remains to show that the $a+1$ layer above any (though
we show each) exceptional element $\mod p^{\lfloor\frac{a}{2}\rfloor+1}$ isn’t
$1$-balanced. By Corollary $3$, we have that for any such exceptional $s$,
$f(s+p^{\lfloor\frac{a}{2}\rfloor+1}x)\equiv
f(s)+f^{\prime}(s)p^{\lfloor\frac{a}{2}\rfloor+1}x\mod
p^{2a+2t+2\lfloor\frac{a}{2}\rfloor+2}$, but then since $f^{\prime}(s)\equiv
0\mod p^{2a+2t+\lfloor\frac{a}{2}\rfloor+1}$, we just get
$f(s+p^{\lfloor\frac{a}{2}\rfloor+1}x)\equiv f(s)\mod
p^{2a+2t+2\lfloor\frac{a}{2}\rfloor+2}$.

If $a$ is even, $2a+2t+2\lfloor\frac{a}{2}\rfloor+2=3a+2t+2$, and so all $a+1$
branches of each exceptional $s$ are the same, immediately implying that no
$\mathfrak{S}_{x}$ is $1$-balanced.

It remains to handle the case of $a$ odd. Taking an exceptional element $s\mod
p^{\frac{a+1}{2}}$, it once again suffices to show that its $a+1$ layer is not
$1$-balanced. By the proof of Lemma $5$, we can choose a positive integer
$s^{\prime}$ with reduction $s\mod p^{\frac{a+1}{2}}$ such that
$f^{\prime}(s^{\prime})\equiv 0\mod p^{\frac{5a+3}{2}+2t}$. By Taylor series
expansion, we have that

$f(s^{\prime}+p^{\frac{a+1}{2}}x)=\sum_{n=0}^{w}f^{(n)}(s^{\prime})\frac{(p^{\frac{a+1}{2}}x)^{n}}{n!}$

For $n>2$, we claim that
$v_{p}(f^{(n)}(s^{\prime})\frac{(p^{\frac{a+1}{2}}x)^{n}}{n!})\geq 3a+2t+2$.
For $n>3$, note that since Lemma $4$ implies $v_{p}(f^{(n)}(s^{\prime}))\geq
2a+2t$, $v_{p}(f^{(n)}(s^{\prime})\frac{(p^{\frac{a+1}{2}}x)^{n}}{n!})\geq
4a+2t+1\geq 3a+2t+2$ since $a\geq 1$. For $n=3$ and $p>3$, the inequality
$v_{p}(f^{(n)}(s^{\prime})\geq 2a+2t$ again gives
$v_{p}(f^{(n)}(s^{\prime})\frac{(p^{\frac{a+1}{2}}x)^{n}}{n!})\geq
2a+2t+3(\frac{a+1}{2})\geq 3a+2t+2$. Finally, for $n=3,p=3$, Lemma $4$ implies
$v_{p}(f^{(n)}(s^{\prime})\geq 2a+2t+1$, and so we still recover
$v_{p}(f^{(n)}(s^{\prime})\frac{(p^{\frac{a+1}{2}}x)^{n}}{n!})\geq
2a+2t+3(\frac{a+1}{2})\geq 3a+2t+2$, proving the claim.

Hence we conclude that

$f(s^{\prime}+p^{\frac{a+1}{2}}x)=f(s^{\prime})+f^{\prime}(s^{\prime})p^{\frac{a+1}{2}}x+f^{\prime\prime}(s^{\prime})\frac{p^{a+1}x^{2}}{2}\mod
p^{3a+2t+2}$

By construction of $s^{\prime}$,
$v_{p}(f^{\prime}(s^{\prime})p^{\frac{a+1}{2}}x)\geq\frac{5a+3}{2}+2t+\frac{a+1}{2}=3a+2t+2$,
and so

$f(s^{\prime}+p^{\frac{a+1}{2}}x)=f(s^{\prime})+f^{\prime\prime}(s^{\prime})\frac{p^{a+1}x^{2}}{2}\mod
p^{3a+2t+2}$

It suffices to show that $f^{\prime\prime}(s^{\prime})\frac{p^{a+1}x^{2}}{2}$
cannot attain all residues of the form $p^{3a+2t+1}k\mod p^{3a+2t+2}$, which
is an immediate consequence of the fact that the map $x^{2}\mod p$ fails to be
surjective, completing the proof.

## References

[1] Kubota, Tomio; Leopoldt, Heinrich-Wolfgang (1964), “Eine p-adische Theorie
der Zetawerte. I. Einführung der p-adischen Dirichletschen L-Funktionen,”
Journal für die reine und angewandte Mathematik, 214/215: 328–339

[2] Kummer, Ernst Eduard (1851), “Über eine allgemeine Eigenschaft der
rationalen Entwicklungscoëfficienten einer bestimmten Gattung analytischer
Functionen,” Journal für die Reine und Angewandte Mathematik, 41: 368–372

[3] G.F. Voronoi, On Bernoulli numbers, Comm. Charkov Math. Soc. 2 (1890)
129-148 (Russian)

Department of Mathematics, California Institute of Technology, Pasadena, CA,
91125
E-mail address<EMAIL_ADDRESS>


Ahmed Adel Attia1, Mark Tiede 2, Carol Y. Espy-Wilson1††thanks: This work was
supported by National Science Foundation grant 2141413

# Enhancing Speech Articulation Analysis Using A Geometric Transformation of
the X-ray Microbeam Dataset

###### Abstract

Accurate analysis of speech articulation is crucial for speech analysis.
However, X-Y coordinates of articulators strongly depend on the anatomy of the
speakers and variability of pellet placements, and existing methods for
mapping anatomical landmarks in the X-ray Microbeam Dataset (XRMB) fail to
capture the entire anatomy of the vocal tract. In this paper, we propose a new
geometric transformation that improves the accuracy of these measurements. Our
transformation maps anatomical landmarks’ X-Y coordinates along the
midsagittal plane onto six relative measures: Lip Aperture (LA), Lip
Protrusion (LP), Tongue Body Constriction Location (TBCL), Degree (TBCD),
Tongue Tip Constriction Location (TTCL), and Degree (TTCD). Our novel
contribution is the extension of the palate trace towards the inferred
anterior pharyngeal line, which improves measurements of tongue body
constriction.

Index Terms: speech inversion, speech analysis, X-ray microbream, geometric
transformation, data engineering

##  1 Introduction

Articulatory data provides valuable insights into speech production and
analysis. By recording the movement and position of articulators such as the
lips, tongue, and jaw during speech, researchers can gain a better
understanding of how speech sounds are produced. Articulatory data are used in
a wide variety of applications like Automatic Speech Recognition [1, 2],
speech synthesis [3], speech therapy [4], and mental health assessment [5, 6].
There are several methods for collecting articulatory data, including the
University of Wisconsin X-ray Microbeam, Electromagnetic Articulometry (EMA),
and real-time Magnetic Resonance Imaging (rt-MRI).

However, accurately analyzing articulatory data can be challenging due to
speaker anatomy and pellet placement variability. The positioning of pellets
in the X-Y plane is closely linked to the speaker's anatomy, leading to
significant variability in pellet positions among speakers for the same sound.
Additionally, small differences in pellet placement can result in considerable
variation. Speech production involves creating constrictions at various
locations along the vocal tract by shaping the vocal tract filter using the
articulators. Furthermore, the absolute positions of the articulators depend
on the speaker's vocal tract anatomy. Therefore, quantifying vocal tract shape
is best achieved by measuring the location and degree of these constrictions,
which are relative measures, rather than the absolute X-Y positions of the
pellets. These measures are called Tract Variables (TVs). TVs specify the
salient features of the vocal tract area function more directly than the
pellet trajectories [7].

Geometric transformations can be used to derive TVs from the absolute X-Y
pellet positions. These variables, introduced in [8], provide information on
the location and degree of constrictions in the vocal tract without requiring
knowledge of the absolute positions of the articulators. The TV Lip Aperture
(LA), for example, quantifies the degree of constriction at the lips without
differentiating the contributions of the jaw, upper lip, or lower lip [9].
In this paper, we focus on the XRMB dataset. [9] describes a geometric
transformation to obtain TVs from the XRMB Pellet Trajectories (PTs).
According to the Task Dynamic model of speech production (TADA) [10], the hard
palate and tongue body were approximated as circles using curve fitting
techniques. Specifically, the hard palate was approximated as a large circle
using curve fitting through the palate trace, and the tongue body was
approximated as a smaller circle within the larger circle that approximates
the palate. The tongue tip was modeled separately using the segment T2-T1.
This modeling approach enabled the conversion of the pellet X-Y positions to
six TV trajectories at each time step. These trajectories included LA, Lip
Protrusion (LP), Tongue Body Constriction Location (TBCL), Tongue Body
Constriction Degree (TBCD), Tongue Tip Constriction Location (TTCL), and
Tongue Tip Constriction Degree (TTCD). This transformation has been proven to
be effective in modeling articulations in the vocal tract and has been used in
various publications [11, 12, 13, 14]. Recently, [15] outlined an Autoencoder
model to reconstruct 3.28 out of 3.4 hours of corrupted XRMB data. However,
they reconstructed the X-Y pellet positions and applying [9]'s transformation
to their reconstructed data proved to be difficult as only a high-level
description of their transformations is outlined and no formulae or code was
available.

The transformation model described [9] also has several limitations. One major
drawback is that the palatal trace in the XRMB dataset only covers a limited
area and does not include the soft palate or extend to the pharyngeal wall. As
a result, the model cannot accurately account for the role of these structures
in speech production, and cannot model narrowings in the pharyngeal region.

Another limitation of the model is that the circular arc used to model the
palatal trace may not accurately represent the actual shape of the palate. To
overcome these limitations, our paper proposes a new approach that
incorporates the soft palate and anterior pharyngeal wall traces into the
model. This will improve the accuracy of the tongue body constriction
calculation, especially in back vowels. Our approach also differs from the
previous model in that it utilizes the palatal trace as is from the XRMB
dataset, without the need to fit a circular arc through it. By addressing
these limitations, our model will provide a more accurate representation of
speech production. We also describe our transformation in more detail so that
our work can be reproducible, and be applied to [15]'s reconstructed XRMB
dataset.

##  2 Dataset Description

The XRMB dataset consists of simultaneously recorded audio and articulatory
data. During the recording process, eight gold pellets were placed on specific
articulators in each speaker, including the upper lip (UL), lower lip (LL),
tongue tip (T1), tongue blade (T2), tongue dorsum (T3), tongue root (T4),
mandible incisor (MNI), and parasagittally placed mandible molar (MNM).
Speakers were given various tasks, such as reading passages or word lists,
while their articulatory movements were tracked and recorded as X-Y
coordinates. The data was sampled at different rates, so to ensure
consistency, all pellet trajectories were resampled at a rate of 145 samples
per second. Multi-sentence recordings were segmented into individual sentence
recordings. In some cases, articulatory recordings were marked as mistracked
in the database, reducing the dataset to 46 speakers (21 males and 25 females)
with approximately 4 hours of speech data.

Figure 1: Transformation of XRMB PTs to TVs. Figure taken from [9]

##  3 Geometric Transformation

In this section, we will provide a comprehensive overview of the proposed
method for transforming the XRMB PTs into TVs, providing a detailed
explanation of each articulator.

Figure 2: Our novel TVs vs [9] TVs.

###  3.1 Lips

To describe the process of lip constriction, we use two different TV values:
LA and LP. LA represents the distance between the upper lip (UL) and lower lip
(LL) pellets in Euclidean space, while LP represents the horizontal offset of
the upper lip from the origin. It is important to note that the origin of the
X-Y plane is located at the tip of the maxillary incisors, and the X-axis is
defined as the maxillary occlusal plane. This means that any measurements or
calculations made in relation to lip constriction are made with reference to
this specific orientation.

$LA[n]=||UL[n]-LL[n]||$ (1) $LP[n]=UL_{x}[n]$ (2)

###  3.2 Tongue Body

The tongue body is represented by a circle that is fitted through the pellets
T2, T3, and T4. The degree of constriction in the tongue body is determined
based on the proximity of this circle to the palate trace that is extended
towards the anterior pharyngeal wall. This method allows for an accurate
representation of tongue constriction during speech production.

In the XRMB dataset, posterior pharyngeal wall traces are available for all
speakers. By shifting the posterior pharyngeal wall to the right by the
thickness of the low retropalatal oropharynx, we can infer the trace of the
anterior pharyngeal wall. The average low retropalatal oropharyngeal thickness
has been estimated to be approximately 0.58 cm for women and 0.56 cm for men
[16].

To estimate the position of the soft palate, we extend the line between the
two posterior samples in the palate trace in the XRMB dataset until it
intersects with the inferred anterior pharyngeal wall.

Figure 3 illustrates the original and extended palatal trace, as well as the
anterior and posterior pharyngeal wall traces, providing a visual
representation of the methodology used in this process. By using these
techniques, we can create precise computer models of speech production that
accurately represent the positions of the tongue and soft palate.

The tongue body constriction is described by TBCD and TBCL. [17] defines TBCL
(and TTCL) as polar angular distance measures for the tongue body and tongue
tip constrictors with respect to a reference line originating at the floor of
the mouth. In our approach, we follow a similar methodology, but we use the
center of a circle that is fitted through the original palatal trace as the
reference point instead. It is important to note that this circle is not used
in any calculations; only its center is utilized as a reference point for
angular calculations.

Therefore, TBCD can be calculated as the minimum distance between the extended
palate trace and the tongue body circle while TBCL represents the angle
between the center of the palatal circle and the point on the tongue body that
is closest to the extended palate trace. Notice in Figure 2 how TBCD is less
noisy than [9]'s TBCD and describes a narrower constriction around 1.3
seconds, which corresponds to the $\Lambda$ sound in the word "mirage". Our
model was more capable of capturing this pharyngeal narrowing in the back
vowel.

Figure 3: Extended Palateal Trace With the Anterior Pharyngeal Wall For
Speaker JW33.

$TBCD=min_{p\in epal}[min_{x\in TB_{circle}}||p-x||]$ (3)
$TBCL=tan^{-1}\frac{TB[argmin[TBCD]_{x}]-pc_{x}}{TB[argmin[TBCD]_{y}-pc_{y}}$
(4)

where $epal$ is the extended palate trace, $TB[argmin[TBCD]$ is the point on
the tongue body closest to the palate trace, and $pc$ is the center of the
palate circle.

###  3.3 Tongue Tip

The tongue tip is represented by the T1 pellet, and its constriction can be
modeled by TTCD and TTCL. Similar to the tongue body, TTCD is the minimum
distance between T1 and the extended palate trace, while TTCL is the
horizontal angle of the segment between the center of the pallet circle and
T1.

$TTCD=min_{p\in epal}[||p-T1||]$ (5)
$TBCL=tan^{-1}\frac{T1_{x}-pc_{x}}{T1_{y}-pc_{y}}$ (6)

##  4 Experiments

We conducted experiments to evaluate our novel geometric transformation by
training a Speech Inversion (SI) system on two sets of TVs: our own and the
ones used in [9]. The Bidirectional Gated Recurrent Neural Network (BiGRNN) SI
system from [11] was used, along with Mel-Frequency Cepstral Coefficients
(MFCCs) extracted from 2-second audio segments. Shorter segments were zero-
padded. The dataset was divided into three sets: training (36 speakers),
development (5 speakers), and testing (5 speakers, 3 males, and 2 females),
with no overlapping speakers to ensure "speaker-independent" training. The
split ensured that the training and development/testing sets contained around
80% and nearly equal amounts of utterances, respectively. All models were
built with the TensorFlow-Keras machine learning framework and trained on a
NVIDIA Titan Xp GPU using MAE loss with the Adam optimizer. We also used early
stopping with a patience of 10 epochs during the training process. Early
stopping is a technique used to prevent overfitting in machine learning models
by monitoring the validation loss during training. The training process is
stopped when the validation loss stops improving after a certain number of
epochs, called "patience". In our case, the training process would stop if the
validation loss did not improve for 10 consecutive epochs. This technique
helps to prevent the model from overfitting to the training data, which can
lead to poor performance on new, unseen data. Both models were evaluated using
the Pearson Product Moment Correlation (PPMC) score.

Table 1 shows the PPMC score for models trained with both sets of TVs. The SI
system scores 6 points higher with TBCD showing that the inferred soft palate
and the anterior pharyngeal wall yield a measure of the tongue body
constriction that is more correlated with the data. On average, our TVs score
3 points more than previous works.

Table 1: PPMC scores for models trained with our TVs and [9]'s TVs

TVs | LA | LP | TBCL | TBCD | TTCL | TTCD | Average
---|---|---|---|---|---|---|---
Ours | 0.6587 | 0.6068 | 0.6455 | 0.6259 | 0.5621 | 0.6817 | 0.6301
[9] | 0.6239 | 0.5430 | 0.6823 | 0.5616 | 0.5271 | 0.6940 | 0.6054

##  5 Conclusions And Future Works

We introduce a novel geometric transformation to derive the Tongue Body
Constriction Location (TBCL) TVs from X-Y coordinates of articulatory
trajectories in the XRMB dataset. Our approach involves incorporating an
inferred outline of the soft palate and anterior pharyngeal wall, resulting in
a more accurate measurement of tongue body constriction.

However, we acknowledge that our current model, which represents the tongue
body as a circular arc, has room for improvement and could benefit from
further investigation. We observed that our TBCL TV is not continuous, which
could explain why the Speech Inversion (SI) system performed better on
predicting the TBCL in [9]. In future research, we plan to address these
limitations.

## References

  * [1] V. Mitra, H. Nam, C. Y. Espy-Wilson, E. Saltzman, and L. Goldstein, ``Articulatory information for noise robust speech recognition,'' _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 19, no. 7, pp. 1913–1924, 2011.
  * [2] V. Mitra, W. Wang, C. Bartels, H. Franco, and D. Vergyri, ``Articulatory Information And Multiview Features For Large Vocabulary Continuous Speech Recognition,'' in _ICASSP_ , 2018.
  * [3] Z.-H. Ling, K. Richmond, and J. Yamagishi, ``Articulatory control of hmm-based parametric speech synthesis using feature-space-switched multiple regression,'' _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 21, no. 1, pp. 207–219, 2013.
  * [4] S. Fagel and K. Madany, ``A 3-d virtual head as a tool for speech therapy for children,'' in _INTERSPEECH_ , 2008.
  * [5] C. Espy-Wilson, A. C. Lammert, N. Seneviratne, and T. F. Quatieri, ``Assessing Neuromotor Coordination in Depression Using Inverted Vocal Tract Variables,'' in _Proc. Interspeech 2019_ , 2019, pp. 1448–1452.
  * [6] Y. M. Siriwardena, C. Espy-Wilson, C. Kitchen, and D. L. Kelly, _Multimodal Approach for Assessing Neuromotor Coordination in Schizophrenia Using Convolutional Neural Networks_. New York, NY, USA: Association for Computing Machinery, 2021, p. 768–772. [Online]. Available: https://doi.org/10.1145/3462244.3479967
  * [7] R. S. McGowan, ``Recovering articulatory movement from formant frequency trajectories using task dynamics and a genetic algorithm: Preliminary model tests,'' _Speech Communication_ , vol. 14, no. 1, pp. 19–48, 1994. [Online]. Available: https://www.sciencedirect.com/science/article/pii/0167639394900558
  * [8] C. P. Browman and L. Goldstein, ``Articulatory phonology: An overview,'' _Phonetica_ , vol. 49, no. 3-4, pp. 155–180, 1992. [Online]. Available: https://doi.org/10.1159/000261913
  * [9] G. Sivaraman, V. Mitra, H. Nam, M. Tiede, and C. Espy-Wilson, ``Unsupervised speaker adaptation for speaker independent acoustic to articulatory speech inversion,'' _The Journal of the Acoustical Society of America_ , vol. 146, no. 1, pp. 316–329, 2019.
  * [10] H. Nam, L. Goldstein, E. Saltzman, and D. Byrd, ``Tada: An enhanced, portable task dynamics model in matlab,'' _The Journal of the Acoustical Society of America_ , vol. 115, no. 5, pp. 2430–2430, 2004.
  * [11] Y. M. Siriwardena, A. A. Attia, G. Sivaraman, and C. Espy-Wilson, ``Audio data augmentation for acoustic-to-articulatory speech inversion using bidirectional gated rnns,'' _arXiv preprint arXiv:2205.13086_ , 2022.
  * [12] C. Espy-Wilson, A. C. Lammert, N. Seneviratne, and T. F. Quatieri, ``Assessing Neuromotor Coordination in Depression Using Inverted Vocal Tract Variables,'' in _Proc. Interspeech 2019_ , 2019, pp. 1448–1452.
  * [13] N. Seneviratne, J. R. Williamson, A. C. Lammert, T. F. Quatieri, and C. Espy-Wilson, ``Extended Study on the Use of Vocal Tract Variables to Quantify Neuromotor Coordination in Depression,'' in _Proc. Interspeech 2020_ , 2020, pp. 4551–4555.
  * [14] Y. M. Siriwardena, C. Espy-Wilson, C. Kitchen, and D. L. Kelly, ``Multimodal approach for assessing neuromotor coordination in schizophrenia using convolutional neural networks,'' in _Proceedings of the 2021 International Conference on Multimodal Interaction_ , 2021, pp. 768–772.
  * [15] A. A. Attia and C. Espy-Wilson, ``Masked autoencoders are articulatory learners,'' _arXiv preprint arXiv:2210.15195_ , 2022.
  * [16] M. M. Daniel, M. C. Lorenzi, C. da Costa Leite, and G. Lorenzi-Filho, ``Pharyngeal dimensions in healthy men and women,'' _Clinics_ , vol. 62, no. 1, pp. 5–10, 2007. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1807593222031489
  * [17] V. Mitra, H. Nam, C. Espy-Wilson, E. Saltzman, and L. Goldstein, ``Recognizing articulatory gestures from speech for robust speech recognition,'' _The Journal of the Acoustical Society of America_ , vol. 131, no. 3, pp. 2270–2287, 2012.