# A Sidecar Separator Can Convert a Single-Speaker Speech Recognition System
to a Multi-Speaker One

###### Abstract

Although automatic speech recognition (ASR) can perform well in common non-
overlapping environments, sustaining performance in multi-speaker overlapping
speech recognition remains challenging. Recent research revealed that ASR
model’s encoder captures different levels of information with different layers
– the lower layers tend to have more acoustic information, and the upper
layers more linguistic. This inspires us to develop a Sidecar separator to
empower a well-trained ASR model for multi-speaker scenarios by separating the
mixed speech embedding between two suitable layers. We experimented with a
wav2vec 2.0-based ASR model with a Sidecar mounted. By freezing the parameters
of the original model and training only the Sidecar (8.7 M, 8.4% of all
parameters), the proposed approach outperforms the previous state-of-the-art
by a large margin for the 2-speaker mixed LibriMix dataset, reaching a word
error rate (WER) of 10.36%; and obtains comparable results (7.56%) for
LibriSpeechMix dataset when limited training.

Index Terms—  multi-speaker speech recognition, speech separation, end-to-end
speech recognition, domain adaptation

##  1 Introduction

End-to-end automatic speech recognition (ASR) for common non-overlapping
environments has made significant progress recently [1, 2]. However, multi-
speaker speech recognition, where overlapping may exist, still remains a
challenge [3]. There are two mainstream end-to-end paradigms that aim to
tackle the challenge: (i) cascade architectures that jointly fine-tune speech
separation and speech recognition modules [4, 5]; and (ii) complete end-to-end
models customized deliberately for multi-speaker overlapping speech scenarios
[6, 7, 8, 9, 10, 11, 12]. However, the former approach may see performance
degradation in modules’ original domains, and the latter does not take full
advantage of the readily available advancements made for single-speaker ASR.
This motivates us to find a low-cost and loose-coupling approach to adapt
well-trained single-speaker ASR models for multi-speaker scenes without
distorting the original model’s parameters.

Fig. 1: (a) a well-trained single-speaker ASR system; (b) the proposed
strategy with a Sidecar separator mounted, taking 2 speakers as example. On
both sides of Sidecar, we add a convolutional layer to coordinate its input-
overlapping and output-separated embeddings.

Recent research investigated the information captured in the layers within the
encoders of ASR models. Shim et al. [13] found that the transformer-based
encoder extracts acoustic representations in its lower layers, and linguistic
representations in the upper layers. Layer-wise analyses of self-supervised
speech representation models, such as wav2vec 2.0 [14], proved that they
encode representations following an acoustic-linguistic hierarchy from lower
to upper layers as well [15, 16], which was further discussed in the context
of neuroscience [17, 18]. Similar findings have also been reported for CNN- /
RNN-based models [19, 20].

Enlightened by the above findings, we assume that there exists a lower
suitable location between the encoder’s two layers where the multi-speaker
acoustic embedding can be well-separated by drawing on speech separation
techniques. In speech separation, research demonstrated that predicting masks
for separation is usually superior to directly predicting separated embeddings
[21, 22]. As a representative, the TasNet architecture [23] predicts masks in
the time domain for mixed speech embeddings and achieves impressive results.
Subsequently, Luo et al. [24] proposed the well-recognized Conv-TasNet, which
predicts masks using a convolutional neural network, which made a further leap
in performance.

Inspired by the recent analyses of ASR models and methodologies in speech
separation, we introduce a promising strategy to adapt off-the-shelf well-
trained ASR models to multi-speaker scenarios. As shown in Fig. 1, we mount a
Sidecar separator between two suitable layers of a well-trained ASR model. On
both sides of the Sidecar, there is a simple convolutional layer helping to
coordinate the input-overlapping and output-separated embeddings of the
Sidecar. The proposed approach has three key advantages:

  * $\bullet$

The approach is low-cost and loose-coupling for converting a well-trained
single-speaker ASR model to a multi-speaker one, without complicated
customization on the model structure or on the training scheme.

  * $\bullet$

The original ASR model is well-trained and fixed, and only Sidecar (8.7 M,
8.4% of all parameters) needs tuning, making the training feasible within
limited time and GPU resources.

  * $\bullet$

Experiments leveraging a wav2vec 2.0-based ASR model mounted with a Sidecar
are conducted, achieving a WER of 10.36% on 2-speaker LibriMix dataset and
7.56% on LibriSpeechMix dataset with limited training.

Moreover, by visualizing Sidecar-predicted masks (Fig. 3), we find that among
channel dimensions different features encode different speakers’ information.
And in the time domain, there exist clear distinctions between the periods of
speech for different speakers, and the periods of overlapping speech.

##  2 Multi-speaker ASR System with Sidecar

The proposed methodology consists of three main components — a well-trained
single-speaker ASR model with parameters frozen, a Sidecar separator, and the
training objective. As shown in Fig. 1, with the cooperation of two
convolutional layers, Sidecar is plugged between two layers of ASR encoder and
forms a multi-speaker ASR system. No language models or lexicons are used in
this work.

With permutation invariant training (PIT) [25], the model is optimized using
connectionist temporal classification (CTC) loss [26].

###  2.1 Well-trained single-speaker ASR model

A typical end-to-end ASR model contains an encoder to synthesize waveform or
acoustic features into high-level representations, and a decoder to model the
representations into language tokens. It often takes much time and effort to
train an ASR model from scratch, let alone in multi-speaker environments. With
many off-the-shelf single-speaker models already available, we try to reuse
the single-speaker model on multi-speaker overlapping speech recognition.

Wav2vec 2.0 [14] is a well-recognized pre-trained speech representation model
based on self-supervised learning (SSL), attracting interest in the field.
Many ASR models taking wav2vec 2.0 as the encoder reported state-of-the-art
performance [27].

In our experiments, we use a well-trained wav2vec 2.0 base-based ASR model,
the same as used in [14], which contains a CNN feature extractor, a
Transformer encoder, and a fully-connected layer as the decoder. Specifically,
the model takes a waveform as input and extracts acoustic features with a
7-layer CNN feature extractor. After a linear projection, the features are fed
into the encoder consisting of 12 layers of Transformer blocks for generating
high-level representations. We obey the paradigm in [14] and only use a fully-
connected layer as the decoder for letter-level prediction.

We directly use the model parameters released by fairseq [28], denoted as
_W2V-CTC_ in the following parts.

Fig. 2: (a) Sidecar structure; (b) Details in ConvBlock. Referring to Conv-
TasNet [24], the Sidecar consists of stacked 1-D dilated convolutional blocks.
$d$ represents the dilation rate.

###  2.2 Sidecar separator

Inspired by the findings that the ASR encoder captures more acoustic
information in its lower layers and more linguistic information in the upper
layers [13, 15, 20], we propose using a Sidecar separator to address multi-
speaker speech recognition, drawing on methodologies in speech separation.

As shown in Fig. 2, similar to Conv-TasNet [24], the Sidecar is a temporal
convolutional network consisting of stacked 1-D dilated convolutional blocks,
which allows the Sidecar to model the long-term dependencies of the acoustic
embeddings while maintaining a small size. By ablation study, we plug the
Sidecar into the most suitable position between two “acoustic” layers of the
ASR encoder. Since the original ASR model is frozen, to alleviate the
“transplant rejection”, we use a 3-kernel-size convolutional layer to filter
Sidecar’s input-mixed and output-separated embeddings, respectively.

In the forward process, the preceding-layer-generated mixed speech embedding
is filtered by a convolutional layer, and fed into the Sidecar to generate the
speaker-dependent masks. Then, the filtered mixed speech embedding will be
element-wise multiplied by the masks, and further adjusted by another
convolutional layer to obtain the separated embeddings. The separated
embeddings of different speakers will go in parallel through the rest of the
model and be transcribed into text. This is technically implemented by
concatenating the separated embeddings onto the batch dimension.

###  2.3 Training objective

We favor the use of permutation invariant training (PIT) with only ASR loss,
which is CTC loss in this work. This can already achieve satisfactory
performance.

In addition to PIT-CTC loss, we also tried two reconstruction objectives:
maximize scale-invariant signal-to-noise ratio (SI-SNR) or minimize mean
squared error (MSE). Since the multi-speaker dataset is simulated from single-
speaker speeches, reconstruction loss aims to drive the predicted separated
embeddings as close as possible to corresponding clean single-speaker
embeddings. The clean single-speaker embeddings are generated by the
transformer layer before where the Sidecar is plugged in, and the permutation
of speakers is determined by PIT-CTC loss.

Although introducing a reconstruction loss can provide a minor performance
gain (Table 4), we do not recommend this. Because it requires input not only
mixed speech but also clean single-speaker speeches, which significantly
increases the computational burden.

##  3 A Baseline System for control

We attribute the effectiveness of this work to two aspects: the knowledge of
the well-trained single-speaker ASR model, and the Sidecar which can
efficiently adapt the former to multi-speaker scenarios by predicting speaker-
dependent masks rather than their embeddings. The contribution of a well-
trained model is intuitive, while the boost in performance provided by Sidecar
can be indistinct.

Considering this, we designed a baseline system, which also leverages the same
ASR model as our proposed approach, but directly predicts speaker-dependent
speech embeddings like [8]. Specifically, in the same position as Sidecar is
in our proposed approach, the baseline model duplicates the preceding encoder
layer to predict speaker-dependent embeddings. Except for the two duplicated
layers, other parameters are frozen.

Note that, unlike our proposed approach, this Baseline does not maintain the
property of keeping the original model parameters unchanged, because it fine-
tunes the layers of the original model.

##  4 Experimental Setup

###  4.1 Datasets

The experiments are performed on two benchmark datasets: _LibriMix_ [29] and
_LibriSpeechMix_ [10]. Both of them are simulated from _LibriSpeech_ dataset
but with different protocols.

LibriMix. It is simulated with the mixtures of two or three speakers, in a
clean or noisy environment. We focus on its 2-speaker-clean subset _Libri2Mix-
clean_. _Libri2Mix-clean_ ’s training, development, and test set contain 270
hours, 11 hours, and 11 hours of 2-speaker’s mixed speeches, respectively. The
mixtures are made in a left-aligned style. Therefore, the shorter source
speech will be fully overlapped by the longer one from the beginning.

LibriSpeechMix. It only has standard dev and test sets. We focus on the
2-speaker ”dev-clean-2mix” and ”test-clean-2mix” for validation and test. The
2-speaker training set is homemade from the 960-hour LibriSpeech training
dataset (LS-960) using the same protocol as [10]. LibriSpeechMix randomly
samples a delay time for a second utterance so the mixture is partially
overlapping.

Compared with LibriSpeechMix, LibriMix has larger overlap ratios, which
greatly challenges the model in separating overlaps.

###  4.2 Model settings

Well-trained single-speaker ASR model.

In accordance with the paradigm in [14], the used W2V-CTC model contains a CNN
feature extractor, a Transformer encoder, and a fully-connected layer as the
decoder. We directly reuse the parameters released by
Fairseq111https://github.com/facebookresearch/fairseq/tree/main/examples/wav2vec
[28], which was first pre-trained on unlabeled LS-960 with contrastive loss
and diversity loss, then fine-tuned on labeled LS-960 with CTC loss. It
reached a WER of 3.4% on LibriSpeech test-clean dataset, and 8.9% on test-
other dataset.

Sidecar separator. Referring to [24], in our Sidecar separator, K
convolutional blocks with dilation rates 1, 2, 4, …, 2K-1 are repeated $R$
times. We take $K=8,R=3$, which performs better. We discard skip-connection
paths of convolutional blocks, and change the final sigmoid activation to ReLU
to fit our task. The Sidecar uses 128 bottleneck-channels, and 768 input- /
output- channels. Ablation experiments (Section 5.4) have been conducted to
explore the most suitable Sidecar location. As a result, we plug the Sidecar
right after the second transformer layer and before the third, which gave the
best performance. With W2V-CTC frozen, it only has 8.7 M (8.4% of all
parameters, about half of the Baseline) for tuning.

Training settings. We optimize the proposed model and Baseline using a 2e-4
learning rate with a three-stage schedule and Adam optimizer, for at most 100
k updates. It takes about 7 hours for models’ convergence with 8 NVIDIA V100
GPUs, thanks to Sidecar’s small size and the ejection start provided by the
well-trained model.

In the following, we denote the proposed model as W2V-Sidecar.

##  5 Results and Discussion

###  5.1 Results On LibriMix dataset

Table 1: Comparison of different systems on _LibriMix_. Evaluated by WER (%). “Transf.” refers to “Transformer” and “ft.” refers to “fine-tune the whole model”. System | Dev | Test
---|---|---
(a) PIT-Transf. [5] | 26.58 | 26.55
(b) Conditional Conformer [30] | 24.50 | 24.90
(c) Conv-TasNet + Transf. [5] | 21.00 | 21.90
(d) DPRNN-TasNet + Transf. [5] | 15.30 | 14.50
(e) Baseline (proposed) | 11.60 | 12.27
(f) W2V-Sidecar (proposed) | 9.76 | 10.36
(g) W2V-Sidecar-ft. (proposed) | 7.68 | 8.12
Table 2: Comparisons of different systems on _LibriSpeechMix_. Evaluated by
WER (%). “-” refers to “not reported” and “ft.” refers to “fine-tune the whole
model”.

System | Dev | Test
---|---|---
(a) PIT-BiLSTM [10] | - | 11.10
(b) SOT-BiLSTM [10] | - | 11.20
(c) SURT-non-streaming [11] | - |  7.20†
(d) SOT-transf. [31] | - |  5.30†
(e) Baseline (proposed) | 9.50 | 9.41
(f) W2V-Sidecar (proposed) | 7.76 | 7.56
(g) W2V-Sidecar-ft. (proposed) | 6.01 | 5.69

  * •

†With heavier training data.

We compared different systems on the two benchmark datasets. The corresponding
results are shown in Table 1 and Table 2.

For 2-speaker LibriMix (Table 1), the designed Baseline (e) for control
already outperforms previous methods by a large margin. We attribute this
improvement to the knowledge of the well-trained model. Then, by comparing the
proposed W2V-Sidecar (f) with Baseline (e), we find the introduction of
Sidecar further boosts the WER with even less trainable parameters (8.7 M,
8.4% of all parameters, about half of Baseline). This confirms that predicting
masks is more effective than directly separating embeddings as discussed in
[22]. Besides, the Sidecar serves in a plug-and-play style without distorting
the original parameters. This low-coupling property allows the model to be
more flexible for multiple scenarios.

Moreover, as an option, we also train a W2V-Sidecar-ft (g), which fine-tunes
all model parameters. The training settings are the same as (e). Since the
model is initialized with well-trained parameters, the W2V-Sidecar-ft’s
convergence is also fast. Not surprisingly, the model achieves even more
impressive results.

###  5.2 Results On LibriSpeechMix dataset

For the comparison on the LibriSpeechMix dataset, we only focus on those non-
streaming models with no additional auxiliary such as speaker labels or
additional training datasets.

As shown in Table 2, our Baseline (e) achieves better performance than (a) and
(b) even with fewer training efforts. As a similar trend, the Sidecar (f)
brings a further performance boost. And as an option, the W2V-Sidecar-ft (g)
reaches a better result at the cost of losing the loose-coupling property. We
also list the results of Systems (c) and (d), which with different setups, for
a comprehensive comparison. They gain further improvements from their
significantly heavier training efforts.

Compared with (a)-(d), the proposed systems (e)-(f) are trained efficiently
with only 7 hours using 8 GPUs. Moreover, systems (a), (b), and (d) have
larger model sizes than the proposed W2V-Sidecar (94.4 M frozen + 8.7 M
trainable).

###  5.3 Visualization of Sidecar predicted masks

To better understand what Sidecar has learned, we investigate its generated
masks with visualization. For a better view, we perform element-wise softmax
on the two masks derived from a mixed speech embedding to highlight pairwise
differences. This process produces two essentially identical matrices, and we
just take one. Then we normalize each channel (or feature) using its mean and
standard deviation along the time steps to avoid swamping those channels which
with minor differences between the two masks. Afterward, we reorder the
channel dimension according to hierarchical clustering based on the pairwise
distances of the channels.

We randomly take three typical cases as examples: an almost non-overlapped
case, a partially overlapped case, and a case in which the shorter speech is
fully overlapped. We find interesting clues from their masks’ visualization
(Fig. 3). The masks show strong temporal correlations with the input speech,
and distinctly tell the mixture boundaries. This indicates that different
feature sub-spaces (or channel groups) capture different speakers’
information, so the speaker boundaries emerge when close channels are
clustered.

###  5.4 Ablation studies

Sidecar location. We explored the optimal location for the Sidecar. We mark
the locations by the transformer blocks index. E.g., location 0 is right
before the first block; location 1 is between the first and second blocks,
etc. The results are summarized in Table 3. Among the locations, the result of
location 2 exceeds the previous layers, and deeper locations show a dramatic
decrease in results.

This trend is mutually supportive of existing layer-wise analysis research
[13, 15], and aligns with our previous hypothesis: the separation is better
performed on acoustic-related representations, which contain sufficient low-
semantics information to catch phonetic-level differences. We argue that
location 0 is too close to the raw input, where meaningful phonetic-level
representations have yet to be well-synthesized. Meanwhile, speaker
information matters, as the discussion about Fig. 3. However, if the ASR
encoder goes deeper, the speaker information will be eliminated. As a result,
location 2 is a compromised scale in semantics.

Reconstruction loss. As reconstruction loss plays a dominant role in speech
separation tasks, a natural question is whether introducing reconstruction
loss can help our task. In this part, we explored adding MSE or SI-SNR loss
into W2V-Sidecar. In Table 4, our experiments show that introducing
reconstruction loss can make slight improvements. However, the computational
burden is significantly increased because it requires not only mixed speech
input but also clean single-speaker recordings. We argue that adding a
constraint on low-level embeddings for such a high-semantic task may not be
very helpful because the mapping can be an ill-posed problem.

###  5.5 Limitations and future work

Fig. 3: Visualization of generated masks and input waveforms. We use different
colors to distinguish speakers. Purple represents overlapping. The horizontal
of masks is time dimension, and the vertical is channel dimension. Sidecar
encodes speaker information with different channels and indicates clear
distinctions in time domain.

This work has several limitations. First, although we used PIT as our training
scheme, our strategy also naturally fits serialized output training (SOT),
which usually needs to train from scratch. We are interested in whether
Sidecar can accelerate SOT’s training with a well-trained ASR model. Second,
according to Fig. 3, Sidecar explicitly encodes speaker information. We are
excited about the prospects of its application to speech diarization,
especially combined with SOT. Third, we only implement Sidecar on the ASR
task. We also expect its applicability to other downstream tasks when
overlapping exists. Finally, to adopt a generally popular speech
representation model, we only use wav2vec 2.0-based model. We will try the
Sidecar on other SSL or non-SSL models in the future.

Table 3: Ablation study on Sidecar location, evaluated by WER (%). | Location
---|---
LibriMix | 0 | 1 | 2 | 3 | 4 | 6 | 9 | 12
Dev | 12.18 | 11.22 | 9.76 | 12.06 | 16.14 | 30.03 | 56.38 | 61.78
Test | 13.01 | 11.87 | 10.36 | 12.65 | 16.88 | 30.32 | 57.11 | 62.72
Table 4: Ablation study on using reconstruction loss, by WER (%). | LibriMix | LibriSpeechMix
---|---|---
| Dev | Test | Dev | Test
W2V-Sidecar | 9.76 | 10.36 | 7.76 | 7.56
w/ SISNR | 9.69 | 10.16 | 7.43 | 7.20
w/ MSE | 9.74 | 10.32 | 7.90 | 7.34

##  6 Conclusion

Inspired by the findings that ASR encoder captures more acoustic
representations in its lower layers and more linguistic in the upper layers,
we propose plugging a Sidecar separator into a well-trained single-speaker ASR
model and converting it to a multi-speaker one. The original ASR model is
frozen, and only 8.4% of all parameters need tuning. With efficient training,
the proposed method outperforms previous state-of-the-art by a large margin on
the 2-speaker mixed LibriMix dataset, reaching a WER of 10.36% dataset; and
comparable results (7.56%) on the LibriSpeechMix dataset.

Visualizations of Sidecar-predicted masks indicate that in channel dimension,
different features encode different speakers’ information. And in the time
domain, there exist significant distinctions between different speaker speech
periods and overlapping periods.

##  7 Acknowledgements

This research is partially supported by the HKSARG Research Grants Council’s
Theme-based Research Grant Scheme (Project No. T45-407/19N) and by the CUHK
Stanley Ho Big Data Decision Analytics Research Centre.

## References

  * [1] William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals,  “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,”  in IEEE ICASSP, 2016.
  * [2] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang,  “Conformer: Convolution-augmented transformer for speech recognition,”  in Interspeech, 2020.
  * [3] Zhongxin Bai and Xiao-Lei Zhang,  “Speaker recognition based on deep learning: An overview,”  Neural Networks, 2021.
  * [4] Shane Settle, Jonathan Le Roux, Takaaki Hori, Shinji Watanabe, and John R Hershey,  “End-to-end multi-speaker speech recognition,”  in IEEE ICASSP, 2018.
  * [5] Song Li, Beibei Ouyang, Fuchuan Tong, Dexin Liao, Lin Li, and Qingyang Hong,  “Real-time end-to-end monaural multi-speaker speech recognition,”  in Interspeech, 2021.
  * [6] Hiroshi Seki, Takaaki Hori, Shinji Watanabe, Jonathan Le Roux, and John R. Hershey,  “A purely end-to-end system for multi-speaker speech recognition,”  in ACL, 2018.
  * [7] Wangyou Zhang, Xuankai Chang, Yanmin Qian, and Shinji Watanabe,  “Improving end-to-end single-channel multi-talker speech recognition,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020.
  * [8] Xuankai Chang, Wangyou Zhang, Yanmin Qian, Jonathan Le Roux, and Shinji Watanabe,  “End-to-end multi-speaker speech recognition with transformer,”  in IEEE ICASSP, 2020.
  * [9] Anshuman Tripathi, Han Lu, and Hasim Sak,  “End-to-end multi-talker overlapping speech recognition,”  in IEEE ICASSP, 2020.
  * [10] Naoyuki Kanda, Yashesh Gaur, Xiaofei Wang, Zhong Meng, and Takuya Yoshioka,  “Serialized output training for end-to-end overlapped speech recognition,”  in Interspeech, 2020.
  * [11] Liang Lu, Naoyuki Kanda, Jinyu Li, and Yifan Gong,  “Streaming multi-talker speech recognition with joint speaker identification,”  in Interspeech, 2021.
  * [12] Naoyuki Kanda, Jian Wu, Yu Wu, Xiong Xiao, Zhong Meng, Xiaofei Wang, Yashesh Gaur, Zhuo Chen, Jinyu Li, and Takuya Yoshioka,  “Streaming multi-talker ASR with token-level serialized output training,”  in Interspeech, 2022.
  * [13] Kyuhong Shim, Jungwook Choi, and Wonyong Sung,  “Understanding the role of self attention for efficient speech recognition,”  in Proceedings of International Conference on Machine learning, 2021\.
  * [14] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  2020\.
  * [15] Ankita Pasad, Ju-Chieh Chou, and Karen Livescu,  “Layer-wise analysis of a self-supervised speech representation model,”  in IEEE ASRU, 2021.
  * [16] Jui Shah, Yaman Kumar Singla, Changyou Chen, and Rajiv Ratn Shah,  “What all do audio transformer models hear? probing acoustic representations for language delivery and its structure,”  arXiv preprint arXiv:2101.00387, 2021.
  * [17] Juliette Millet, Charlotte Caucheteux, Pierre Orhan, Yves Boubenec, Alexandre Gramfort, Ewan Dunbar, Christophe Pallier, and Jean-Remi King,  “Toward a realistic model of speech processing in the brain with self-supervised learning,”  2022\.
  * [18] Aditya R. Vaidya, Shailee Jain, and Alexander G. Huth,  “Self-supervised models of audio effectively explain human cortical responses to speech,”  in Proceedings of International Conference on Machine Learning, 2022\.
  * [19] Archiki Prasad and Preethi Jyothi,  “How accents confound: Probing for accent information in end-to-end speech recognition systems,”  in ACL, 2020.
  * [20] Chung-Yi Li, Pei-Chieh Yuan, and Hung-Yi Lee,  “What does a network layer hear? analyzing hidden representations of end-to-end ASR through speech synthesis,”  in IEEE ICASSP, 2020.
  * [21] DeLiang Wang and Jitong Chen,  “Supervised speech separation based on deep learning: An overview,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2018.
  * [22] Yuxuan Wang, Arun Narayanan, and DeLiang Wang,  “On training targets for supervised speech separation,”  IEEE/ACM transactions on audio, speech, and language processing, 2014.
  * [23] Yi Luo and Nima Mesgarani,  “TasNet: time-domain audio separation network for real-time, single-channel speech separation,”  in IEEE ICASSP, 2018.
  * [24] Yi Luo and Nima Mesgarani,  “Conv-TasNet: Surpassing ideal time-frequency magnitude masking for speech separation,”  IEEE/ACM transactions on audio, speech, and language processing, 2019.
  * [25] Dong Yu, Morten Kolbæk, Zheng-Hua Tan, and Jesper Jensen,  “Permutation invariant training of deep models for speaker-independent multi-talker speech separation,”  in IEEE ICASSP, 2017.
  * [26] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber,  “Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,”  in Proceedings of International Conference on Machine learning, 2006\.
  * [27] Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu,  “W2v-BERT: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,”  in IEEE ASRU, 2021.
  * [28] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli,  “fairseq: A fast, extensible toolkit for sequence modeling,”  in Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
  * [29] Joris Cosentino, Manuel Pariente, Samuele Cornell, Antoine Deleforge, and Emmanuel Vincent,  “LibriMix: An open-source dataset for generalizable speech separation,”  arXiv preprint arXiv:2005.11262, 2020.
  * [30] Pengcheng Guo, Xuankai Chang, Shinji Watanabe, and Lei Xie,  “Multi-speaker ASR combining non-autoregressive Conformer CTC and conditional speaker chain,”  in Interspeech, 2021.
  * [31] Naoyuki Kanda, Guoli Ye, Yashesh Gaur, Xiaofei Wang, Zhong Meng, Zhuo Chen, and Takuya Yoshioka,  “End-to-end speaker-attributed ASR with transformer,”  in Interspeech, 2021.


eV and b) vacuum with an increased S1-S2 gap of $\Delta_{12}-0.18$ eV. The
plots in c) and d) represent slices through the nonlinear spectra a) and b) at
specific wavelengths.

We note that increasing the gap just by 40 meV noticeably quenches the high
wavelength shoulder in the vacuum spectrum. Additionally, while the heights of
the main peak and the dual fluorescence shoulder in the standard vacuum
parameterization undergo sustained oscillations from 40 to 160 fs (see Fig.
S10 c)), these oscillations are strongly quenched once the S1-S2 gap is
increased slightly. The results demonstrate again that the dual fluorescence
properties of proflavine are highly sensitive to both the form of the coupling
spectral density and the precise S1-S2 gap in the Condon region.

## References

  * Tamura and Burghardt 2013 Tamura, H.; Burghardt, I. Ultrafast Charge Separation in Organic Photovoltaics Enhanced by Charge Delocalization and Vibronically Hot Exciton Dissociation. _J. Am. Chem. Soc._ 2013, _135_ , 16364–16367, PMID: 24138412
  * Northey and Penfold 2018 Northey, T.; Penfold, T. The intersystem crossing mechanism of an ultrapure blue organoboron emitter. _Org. Electron._ 2018, _59_ , 45–48
  * Alvertis et al. 2019 Alvertis, A. M.; Lukman, S.; Hele, T. J.; Fuemmeler, E. G.; Feng, J.; Wu, J.; Greenham, N. C.; Chin, A. W.; Musser, A. J. Switching between coherent and incoherent singlet fission via solvent-induced symmetry breaking. _J. Am. Chem. Soc._ 2019, _141_ , 17558–17570
  * Arsenault et al. 2020 Arsenault, E. A.; Yoneda, Y.; Iwai, M.; Niyogi, K. K.; Fleming, G. R. Vibronic mixing enables ultrafast energy flow in light-harvesting complex II. _Nature Commun._ 2020, _11_ , 1460
  * Hetherington et al. 2023 Hetherington, C. V.; Mohan T. M., N.; Tilluck, R. W.; Beck, W. F.; Levine, B. G. Origin of Vibronic Coherences During Carrier Cooling in Colloidal Quantum Dots. _J. Phys. Chem. Lett._ 2023, _14_ , 11651–11658
  * Curchod and Martínez 2018 Curchod, B. F. E.; Martínez, T. J. Ab Initio Nonadiabatic Quantum Molecular Dynamics. _Chem. Rev._ 2018, _118_ , 3305–3336
  * Bircher et al. 2018 Bircher, M. P.; Liberatore, E.; Browning, N. J.; Brickel, S.; Hofmann, C.; Patoz, A.; Unke, O. T.; Zimmermann, T.; Chergui, M.; Hamm, P. et al. Nonadiabatic effects in electronic and nuclear dynamics. _Struct. Dyn._ 2018, _4_ , 061510
  * Domcke et al. 1981 Domcke, W.; Köppel, H.; Cederbaum, L. S. Spectroscopic effects of conical intersections of molecular potential energy surfaces. _Mol. Phys._ 1981, _43_ , 851–875
  * Domcke and Yarkony 2012 Domcke, W.; Yarkony, D. R. Role of conical intersections in molecular spectroscopy and photoinduced chemical dynamics. _Annu. Rev. Phys. Chem._ 2012, _63_ , 325–352
  * Orlandi and Siebrand 1973 Orlandi, G.; Siebrand, W. Theory of vibronic intensity borrowing. Comparison of Herzberg-Teller and Born-Oppenheimer coupling. _J. Chem. Phys._ 1973, _4513_ , 4513–4523
  * Aranda and Santoro 2021 Aranda, D.; Santoro, F. Vibronic Spectra of pi-Conjugated Systems with a Multitude of Coupled States: A Protocol Based on Linear Vibronic Coupling Models and Quantum Dynamics Tested on Hexahelicene. _J. Chem. Theory Comput._ 2021, _17_ , 1691–1700
  * Dunnett et al. 2021 Dunnett, A. J.; Gowland, D.; Isborn, C. M.; Chin, A. W.; Zuehlsdorff, T. J. Influence of non-adiabatic effects on linear absorption spectra in the condensed phase: Methylene blue. _J. Chem. Phys._ 2021, _155_ , 144112
  * Grabowski et al. 2003 Grabowski, Z. R.; Rotkiewicz, K.; Rettig, W. Structural Changes Accompanying Intramolecular Electron Transfer: Focus on Twisted Intramolecular Charge-Transfer States and Structures. _Chem. Rev._ 2003, _103_ , 3899–4032
  * Park et al. 2013 Park, M.; Kim, C. H.; Joo, T. Multifaceted Ultrafast Intramolecular Charge Transfer Dynamics of 4-(Dimethylamino)benzonitrile (DMABN). _J. Phys. Chem. A_ 2013, _117_ , 370–377
  * Curchod et al. 2017 Curchod, B. F. E.; Sisto, A.; Martínez, T. J. Ab Initio Multiple Spawning Photochemical Dynamics of DMABN Using GPUs. _J. Phys. Chem. A_ 2017, _121_ , 265–276
  * Demchenko et al. 2017 Demchenko, A. P.; Tomin, V. I.; Chou, P.-T. Breaking the Kasha Rule for More Efficient Photochemistry. _Chem. Rev._ 2017, _117_ , 13353–13381
  * Djavani-Tabrizi and Jockusch 2022 Djavani-Tabrizi, I.; Jockusch, R. A. Gas-Phase Fluorescence of Proflavine Reveals Two Close-Lying, Brightly Emitting States. _J. Phys. Chem. Lett._ 2022, _13_ , 2187–2192
  * Braun et al. 2022 Braun, G.; Borges, J., Itamar; Aquino, A. J. A.; Lischka, H.; Plasser, F.; do Monte, S. A.; Ventura, E.; Mukherjee, S.; Barbatti, M. Non-Kasha fluorescence of pyrene emerges from a dynamic equilibrium between excited states. _J. Chem. Phys._ 2022, _157_ , 154305
  * Veys and Escudero 2022 Veys, K.; Escudero, D. Anti-Kasha Fluorescence in Molecular Entities: Central Role of Electron–Vibrational Coupling. _Acc. Chem. Res._ 2022, _55_ , 2698–2707
  * Burghardt et al. 2004 Burghardt, I.; Cederbaum, L. S.; Hynes, J. T. Environmental effects on a conical intersection: A model study. _Faraday Discuss._ 2004, _127_ , 395–411
  * Santoro et al. 2021 Santoro, F.; Green, J. A.; Martinez-Fernandez, L.; Cerezo, J.; Improta, R. Quantum and semiclassical dynamical studies of nonadiabatic processes in solution: achievements and perspectives. _Phys. Chem. Chem. Phys._ 2021, _23_ , 8181–8199
  * Cerezo et al. 2023 Cerezo, J.; García-Iriepa, C.; Santoro, F.; Navizet, I.; Prampolini, G. Unraveling the contributions to the spectral shape of flexible dyes in solution: insights on the absorption spectrum of an oxyluciferin analogue. _Phys. Chem. Chem. Phys._ 2023, _25_ , 5007–5020
  * Zuehlsdorff et al. 2019 Zuehlsdorff, T. J.; Montoya-Castillo, A.; Napoli, J. A.; Markland, T. E.; Isborn, C. M. Optical spectra in the condensed phase: Capturing anharmonic and vibronic features using dynamic and static approaches. _J. Chem. Phys._ 2019, _151_ , 074111
  * Chin et al. 2010 Chin, A. W.; Rivas, Á.; Huelga, S. F.; Plenio, M. B. Exact mapping between system-reservoir quantum models and semi-infinite discrete chains using orthogonal polynomials. _J. Math. Phys._ 2010, _51_
  * Kloss et al. 2018 Kloss, B.; Lev, Y. B.; Reichman, D. Time-dependent variational principle in matrix-product state manifolds: Pitfalls and potential. _Phys. Rev. B_ 2018, _97_ , 024307
  * Schröder et al. 2019 Schröder, F. A.; Turban, D. H.; Musser, A. J.; Hine, N. D.; Chin, A. W. Tensor network simulation of multi-environmental open quantum dynamics via machine learning and entanglement renormalisation. _Nature Commun._ 2019, _10_ , 1–10
  * Donald et al. 2011 Donald, W. A.; Leib, R. D.; Demireva, M.; Williams, E. R. Ions in Size-Selected Aqueous Nanodrops: Sequential Water Molecule Binding Energies and Effects of Water on Ion Fluorescence. _J. Am. Chem. Soc._ 2011, _133_ , 18940–18949
  * Kumar et al. 2012 Kumar, K. S.; Selvaraju, C.; Malar, E. J. P.; Natarajan, P. Existence of a New Emitting Singlet State of Proflavine: Femtosecond Dynamics of the Excited State Processes and Quantum Chemical Studies in Different Solvents. _J. Phys. Chem. A_ 2012, _116_ , 37–45
  * Arden-Jacob et al. 2013 Arden-Jacob, J.; Drexhage, K.-H.; Druzhinin, S. I.; Ekimova, M.; Flender, O.; Lenzer, T.; Oum, K.; Scholz, M. Ultrafast photoinduced dynamics of the 3,6-diaminoacridinium derivative ATTO 465 in solution. _Phys. Chem. Chem. Phys._ 2013, _15_ , 1844–1853
  * Kostjukova et al. 2021 Kostjukova, L. O.; Leontieva, S. V.; Kostjukov, V. V. The vibronic absorption spectra and electronic states of proflavine in aqueous solution. _Comput. Theor. Chem._ 2021, _1197_ , 113144
  * Savenko and Kostjukov 2023 Savenko, E. S.; Kostjukov, V. V. Theoretical study of the excitation of proflavine H-dimers in an aqueous solution: the effect of functionals and dispersion corrections. _Phys. Chem. Chem. Phys._ 2023, _25_ , 12259–12276
  * Runge and Gross 1984 Runge, E.; Gross, E. K. U. Density-Functional Theory for Time-Dependent Systems. _Phys. Rev. Lett._ 1984, _52_ , 997–1000
  * Casida 1995 Casida, M. E. _Recent Advances in Density Functional Methods_ ; 1995; pp 155–192
  * Xue et al. 2015 Xue, X.; Jin, S.; Zhang, C.; Yang, K.; Huo, S.; Chen, F.; Zou, G.; Liang, X.-J. Probe-inspired nano-prodrug with dual-color fluorogenic property reveals spatiotemporal drug release in living cells. _ACS nano_ 2015, _9_ , 2729–2739
  * Wu et al. 2022 Wu, L.; Liu, J.; Tian, X.; Groleau, R. R.; Feng, B.; Yang, Y.; Sedgwick, A. C.; Han, H.-H.; Wang, Y.; Wang, H.-M. et al. Dual-Channel Fluorescent Probe for the Simultaneous Monitoring of Peroxynitrite and Adenosine-5’-triphosphate in Cellular Applications. _J. Am. Chem. Soc._ 2022, _144_ , 174–183, PMID: 34931825
  * Yuan et al. 2019 Yuan, T.; Yuan, F.; Li, X.; Li, Y.; Fan, L.; Yang, S. Fluorescence–phosphorescence dual emissive carbon nitride quantum dots show 25% white emission efficiency enabling single-component WLEDs. _Chem. Sci._ 2019, _10_ , 9801–9806
  * Qian et al. 2017 Qian, H.-L.; Dai, C.; Yang, C.-X.; Yan, X.-P. High-crystallinity covalent organic framework with dual fluorescence emissions and its ratiometric sensing application. _ACS Appl. Mater. Inter._ 2017, _9_ , 24999–25005
  * Baiardi et al. 2013 Baiardi, A.; Bloino, J.; Barone, V. General Time Dependent Approach to Vibronic Spectroscopy Including Franck–Condon, Herzberg–Teller, and Duschinky Effects. _J. Chem. Theory Comput._ 2013, _9_ , 4097–4115
  * de Souza et al. 2018 de Souza, B.; Neese, F.; Izsák, R. On the theoretical prediction of fluorescence rates from first principles using the path integral approach. _J. Chem. Phys._ 2018, _148_ , 034104
  * Mukamel 1995 Mukamel, S. _Principles of Nonlinear Optical Spectroscopy_ ; Oxford University Press: New York, 1995
  * Cammi et al. 2005 Cammi, R.; Corni, S.; Mennucci, B.; Tomasi, J. Electronic excitation energies of molecules in solution: State specific and linear response methods for nonequilibrium continuum solvation models. _J. Chem. Phys._ 2005, _122_ , 104513
  * Cerezo et al. 2015 Cerezo, J.; Avila Ferrer, F. J.; Prampolini, G.; Santoro, F. Modeling Solvent Broadening on the Vibronic Spectra of a Series of Coumarin Dyes. From Implicit to Explicit Solvent Models. _J. Chem. Theory Comput._ 2015, _11_ , 5810–5825
  * Zuehlsdorff et al. 2018 Zuehlsdorff, T. J.; Napoli, J. A.; Milanese, J. M.; Markland, T. E.; Isborn, C. M. Unraveling electronic absorption spectra using nuclear quantum effects: Photoactive yellow protein and green fluorescent protein chromophores in water. _J. Chem. Phys._ 2018, _149_ , 024107
  * Zuehlsdorff and Isborn 2018 Zuehlsdorff, T. J.; Isborn, C. M. Combining the ensemble and Franck-Condon approaches for calculating spectral shapes of molecules in solution. _J. Chem. Phys._ 2018, _148_ , 024110
  * Cerezo et al. 2019 Cerezo, J.; Aranda, D.; Avila Ferrer, F. J.; Prampolini, G.; Santoro, F. Adiabatic-molecular dynamics generalized vertical hessian approach: a mixed quantum classical method to compute electronic spectra of flexible molecules in the condensed phase. _J. Chem. Theory Comput._ 2019, _16_ , 1215–1231
  * Valleau et al. 2012 Valleau, S.; Eisfeld, A.; Aspuru-Guzik, A. On the alternatives for bath correlators and spectral densities from mixed quantum-classical simulations. _J. Chem. Phys._ 2012, _137_ , 224103
  * Shim et al. 2012 Shim, S.; Rebentrost, P.; Valleau, S.; Aspuru-Guzik, A. Atomistic Study of the Long-Lived Quantum Coherences in the Fenna-Matthews-Olson Complex. _Biophys. J._ 2012, _102_ , 649–660
  * Lee and Coker 2016 Lee, M. K.; Coker, D. F. Modeling Electronic-Nuclear Interactions for Excitation Energy Transfer Processes in Light-Harvesting Complexes. _J. Phys. Chem. Lett._ 2016, _7_ , 3171–3178
  * Lee et al. 2016 Lee, M. K.; Huo, P.; Coker, D. F. Semiclassical Path Integral Dynamics: Photosynthetic Energy Transfer with Realistic Environment Interactions. _Annu. Rev. Phys. Chem._ 2016, _67_ , 639–668
  * Loco and Cupellini 2018 Loco, D.; Cupellini, L. Modeling the absorption lineshape of embedded systems from molecular dynamics: A tutorial review. _Int. J. Quantum Chem._ 2018, _119_ , e25726
  * Cignoni et al. 2022 Cignoni, E.; Slama, V.; Cupellini, L.; Mennucci, B. The atomistic modeling of light-harvesting complexes from the physical models to the computational protocol. _J. Chem. Phys_ 2022, _156_ , 120901
  * Köppel et al. 1984 Köppel, H.; Domcke, W.; Cederbaum, L. S. In _Multimode Molecular Dynamics Beyond the Born-Oppenheimer Approximation_ ; Prigogine, I., Rice, S. A., Eds.; Wiley, 1984; Vol. Advances in Chemical Physics, LVII; pp 59–246
  * Worth et al. 1996 Worth, G. A.; Meyer, H. D.; Cederbaum, L. S. The effect of a model environment on the S2 absorption spectrum of pyrazine: A wave packet study treating all 24 vibrational modes. _J. Chem. Phys._ 1996, _105_ , 4412–4426
  * Capano et al. 2014 Capano, G.; Chergui, M.; Rothlisberger, U.; Tavernelli, I.; Penfold, T. J. A Quantum Dynamics Study of the Ultrafast Relaxation in a Prototypical Cu(I)–Phenanthroline. _J. Phys. Chem. A_ 2014, _118_ , 9861–9869
  * Pápai et al. 2016 Pápai, M.; Vankó, G.; Rozgonyi, T.; Penfold, T. J. High-Efficiency Iron Photosensitizer Explained with Quantum Wavepacket Dynamics. _J. Phys. Chem. Lett._ 2016, _7_ , 2009–2014
  * Neville et al. 2018 Neville, S. P.; Stolow, A.; Schuurman, M. S. Vacuum ultraviolet excited state dynamics of the smallest ring, cyclopropane. I. A reinterpretation of the electronic spectrum and the effect of intensity borrowing. _J. Chem. Phys._ 2018, _149_
  * Zobel et al. 2021 Zobel, J. P.; Heindl, M.; Plasser, F.; Mai, S.; González, L. Surface Hopping Dynamics on Vibronic Coupling Models. _Acc. Chem. Res._ 2021, _54_ , 3760–3771
  * Green et al. 2021 Green, J. A.; Yaghoubi Jouybari, M.; Aranda, D.; Improta, R.; Santoro, F. Nonadiabatic Absorption Spectra and Ultrafast Dynamics of DNA and RNA Photoexcited Nucleobases. _Molecules_ 2021, _26_ , 1743
  * Segalina et al. 2022 Segalina, A.; Aranda, D.; Green, J. A.; Cristino, V.; Caramori, S.; Prampolini, G.; Pastore, M.; Santoro, F. How the Interplay among Conformational Disorder, Solvation, Local, and Charge-Transfer Excitations Affects the Absorption Spectrum and Photoinduced Dynamics of Perylene Diimide Dimers: A Molecular Dynamics/Quantum Vibronic Approach. _J. Chem. Theory Comput_ 2022, _18_ , 3718–3736
  * Segatta et al. 2023 Segatta, F.; Ruiz, D. A.; Aleotti, F.; Yaghoubi, M.; Mukamel, S.; Garavelli, M.; Santoro, F.; Nenov, A. Nonlinear Molecular Electronic Spectroscopy via MCTDH Quantum Dynamics: From Exact to Approximate Expressions. _J. Chem. Theory Comput._ 2023, _19_ , 2075–2091
  * Beck et al. 2000 Beck, M. H.; Jäckle, A.; Worth, G. A.; Meyer, H. D. The Multiconfiguration Time-Dependent Hartree (MCTDH) Method: A highly efficient algorithm for propagating wavepackets. _Phys. Rep._ 2000, _324_ , 1–105
  * Gatti and Worth 2009 Gatti, F.; Worth, G. A. In _Multidimensional Quantum Dynamics: MCTDH Theory and Applications_ ; Meyer, H. D., Ed.; Wiley VCH, 2009
  * Egorov et al. 1999 Egorov, S. A.; Everitt, K. F.; Skinner, J. L. Quantum Dynamics and Vibrational Relaxation. _J. Phys. Chem. A_ 1999, _103_ , 9494–9499
  * Craig and Manolopoulos 2004 Craig, I. R.; Manolopoulos, D. E. Quantum statistics and classical mechanics: Real time correlation functions from ring polymer molecular dynamics. _J. Chem. Phys._ 2004, _121_ , 3368
  * Ramirez et al. 2004 Ramirez, R.; Lopez-Ciudad, T.; P, P. K.; Marx, D. Quantum corrections to classical time-correlation functions: Hydrogen bonding and anharmonic floppy modes. _J. Chem. Phys._ 2004, _121_ , 3973
  * Subotnik et al. 2015 Subotnik, J. E.; Alguire, E. C.; Ou, Q.; Landry, B. R.; Fatehi, S. The requisite electronic structure theory to describe photoexcited nonadiabatic dynamics: Nonadiabatic derivative couplings and diabatic electronic couplings. _Acc. Chem. Res._ 2015, _48_ , 1340–1350
  * Medders et al. 2017 Medders, G. R.; Alguire, E. C.; Jain, A.; Subotnik, J. E. Ultrafast Electronic Relaxation through a Conical Intersection: Nonadiabatic Dynamics Disentangled through an Oscillator Strength-Based Diabatization Framework. _J. Phys. Chem. A_ 2017, _121_ , 1425–1434
  * Cerezo et al. 2023 Cerezo, J.; García-Iriepa, C.; Santoro, F.; Navizet, I.; Prampolini, G. Unraveling the contributions to the spectral shape of flexible dyes in solution: insights on the absorption spectrum of an oxyluciferin analogue. _Phys. Chem. Chem. Phys._ 2023, _25_ , 5007–5020
  * Tamascelli et al. 2019 Tamascelli, D.; Smirne, A.; Lim, J.; Huelga, S. F.; Plenio, M. B. Efficient simulation of finite-temperature open quantum systems. _Phys. Rev. Lett._ 2019, _123_ , 090402
  * Haegeman et al. 2011 Haegeman, J.; Cirac, J. I.; Osborne, T. J.; Pižorn, I.; Verschelde, H.; Verstraete, F. Time-Dependent Variational Principle for Quantum Lattices. _Phys. Rev. Lett._ 2011, _107_ , 070601
  * Haegeman et al. 2016 Haegeman, J.; Lubich, C.; Oseledets, I.; Vandereycken, B.; Verstraete, F. Unifying time evolution and optimization with matrix product states. _Phys. Rev. B_ 2016, _94_ , 165116
  * Lacroix et al. 2021 Lacroix, T.; Dunnett, A.; Gribben, D.; Lovett, B. W.; Chin, A. Unveiling non-Markovian spacetime signaling in open quantum systems with long-range tensor network dynamics. _Phys. Rev. A_ 2021, _104_ , 052204
  * Dunnett and Chin 2021 Dunnett, A. J.; Chin, A. W. Simulating Quantum Vibronic Dynamics at Finite Temperatures With Many Body Wave Functions at 0 K. _Frontiers in Chemistry_ 2021, _8_ , 1195
  * Riva et al. 2023 Riva, A.; Tamascelli, D.; Dunnett, A. J.; Chin, A. W. Thermal cycle and polaron formation in structured bosonic environments. _Phys. Rev. B_ 2023, _108_ , 195138
  * Warshel and Levitt 1976 Warshel, A.; Levitt, M. Theoretical studies of enzymic reactions: Dielectric, electrostatic and steric stabilization of the carbonium ion in the reaction of lysozyme. _J. Mol. Biol._ 1976, _103_ , 227–249
  * Hirata and Head-Gordon 1999 Hirata, S.; Head-Gordon, M. Time-dependent density functional theory within the Tamm–Dancoff approximation. _Chem. Phys. Lett._ 1999, _314_ , 291–299
  * Yanai et al. 2004 Yanai, T.; Tew, D. P.; Handy, N. C. A new hybrid exchange-correlation functional using the Coulomb-attenuating method (CAM-B3LYP). _Chem. Phys. Lett._ 2004, _393_ , 51–57
  * Stanton and Bartlett 1993 Stanton, J. F.; Bartlett, R. J. The equation of motion coupled-cluster method. A systematic biorthogonal approach to molecular excitation energies, transition probabilities, and excited state properties. _J. Chem. Phys._ 1993, _98_ , 7029–7039
  * Comeau and Bartlett 1993 Comeau, D. C.; Bartlett, R. J. The equation-of-motion coupled-cluster method. Applications to open-and closed-shell reference states. _Chem. Phys. Lett._ 1993, _207_ , 414–423
  * Krylov 2008 Krylov, A. I. Equation-of-motion coupled-cluster methods for open-shell and electronically excited species: The hitchhiker’s guide to Fock space. _Annu. Rev. Phys. Chem._ 2008, _59_ , 433–462
  * Loos et al. 2018 Loos, P.-F.; Scemama, A.; Blondel, A.; Garniron, Y.; Caffarel, M.; Jacquemin, D. A mountaineering strategy to excited states: Highly accurate reference energies and benchmarks. _J. Chem. Theory Comput._ 2018, _14_ , 4360–4379
  * Loos et al. 2020 Loos, P.-F.; Lipparini, F.; Boggio-Pasqua, M.; Scemama, A.; Jacquemin, D. A mountaineering strategy to excited states: Highly accurate energies and benchmarks for medium sized molecules. _J. Chem. Theory Comput._ 2020, _16_ , 1711–1741
  * Humeniuk and Glover 0 Humeniuk, A.; Glover, W. J. Multistate, Polarizable QM/MM Embedding Scheme Based on the Direct Reaction Field Method: Solvatochromic Shifts, Analytical Gradients and Optimizations of Conical Intersections in Solution. _J. Chem. Theory Comput._ 0, _0_ , null
  * Zuehlsdorff et al. 2020 Zuehlsdorff, T. J.; Hong, H.; Shi, L.; Isborn, C. M. Influence of Electronic Polarization on the Spectral Density. _J. Phys. Chem. B_ 2020, _124_ , 531–543
  * Galiana and Lasorne 2023 Galiana, J.; Lasorne, B. On the unusual Stokes shift in the smallest PPE dendrimer building block: Role of the vibronic symmetry on the band origin? _J. Chem. Phys._ 2023, _158_ , 124113
  * Gallop et al. 2024 Gallop, N. P.; Maslennikov, D. R.; Mondal, N.; Goetz, K. P.; Dai, Z.; Schankler, A. M.; Sung, W.; Nihonyanagi, S.; Tahara, T.; Bodnarchuk, M. I. et al. Ultrafast vibrational control of organohalide perovskite optoelectronic devices using vibrationally promoted electronic resonance. _Nat. Mater._ 2024, _23_ , 88–94
  * Garcia-Vidal et al. 2021 Garcia-Vidal, F. J.; Ciuti, C.; Ebbesen, T. W. Manipulating matter by strong coupling to vacuum fields. _Science_ 2021, _373_ , eabd0336
  * Sokolovskii et al. 2023 Sokolovskii, I.; Tichauer, R. H.; Morozov, D.; Feist, J.; Groenhof, G. Multi-scale molecular dynamics simulations of enhanced energy transfer in organic molecules under strong coupling. _Nat. Commun._ 2023, _14_ , 6613
  * Dunnett 2021 Dunnett, A. angusdunnett/MPSDynamics:. 2021; https://doi.org/10.5281/zenodo.5106435


11institutetext: Technion - Israel Institute of Technology
11email<EMAIL_ADDRESS>
11email<EMAIL_ADDRESS>
# Learning Differential Invariants of Planar Curves

Roy Velich  Ron Kimmel

###### Abstract

We propose a learning paradigm for the numerical approximation of differential
invariants of planar curves. Deep neural-networks’ (DNNs) universal
approximation properties are utilized to estimate geometric measures. The
proposed framework is shown to be a preferable alternative to axiomatic
constructions. Specifically, we show that DNNs can learn to overcome
instabilities and sampling artifacts and produce consistent signatures for
curves subject to a given group of transformations in the plane. We compare
the proposed schemes to alternative state-of-the-art axiomatic constructions
of differential invariants. We evaluate our models qualitatively and
quantitatively and propose a benchmark dataset to evaluate approximation
models of differential invariants of planar curves.

###### Keywords:

Differential invariants differential geometry computer vision shape analysis

##  1 Introduction

Differential Invariants and Signature Curves. According to an important
theorem by É. Cartan [6], two curves are related by a group transformation
$g\in G$, i.e., equivalent curves with respect to $G$, if and only if their
signature curves, with respect to the transformation group $G$, are identical.
This observation allows one to develop analytical tools to measure the
equivalence of two planar curves, which has practical applications in various
fields and tasks, such as computer vision, shape analysis, geometry
processing, object detection, and more. Planar curves are often extracted from
images as boundaries of objects, or level-sets of gray-scale images. The
boundary of an object encodes vital information which can be further exploited
by various computer vision and image analysis tools. A common computer vision
task requires one to find common characteristics between two objects in two
different image frames. This task can be naturally approached by representing
image objects as signature curves of their boundaries. Since signature curves
are proven to provide a full solution for the equivalence problem of planar
curves [14, p. 183, Theorem 8.53], they can be further analyzed for finding
correspondence between image objects. Signature curves are parametrized by
differential invariants. Therefore, in order to generate the signature curve
of a given planar curve, one has to evaluate the required differential
invariants at each point. In practice, planar curves are digitally represented
as a discrete set of points, which implies that the computation of
differential invariant quantities, such as the curvature at a point, can only
be numerically approximated using finite differences techniques. Since many
important and interesting differential invariants are expressed as functions
of high-order derivatives, their approximations are prone to inherent
numerical instabilities, and high sensitivity to sampling noise. For example,
the equiaffine curvature, which is a differential invariant of the special-
affine transformation group $\mathrm{SA}\left(2\right)$, is a fourth-order
differential invariant, and its direct approximation using an axiomatic
approach is practically infeasible for discrete curves, especially around
inflection points.

Contribution. Inspired by the numerical approximation efforts of Calabi et al.
[5], and the learning approaches introduced in [15, 10], we present a complete
learning environment for approximating differential invariants of planar
curves with respect to various transformation groups, including the affine
group, for which, very few attempts to approximate their differential
invariants in the discrete setting have been made. Moreover, we introduce a
shape-matching benchmark for evaluating differential invariants approximation
models.

##  2 Related Work

Axiomatic Approximation of Differential Invariants. The axiomatic approach for
approximating the differential invariants of discrete planar curves, as
presented by Calabi et al. in [5], is based on the joint invariants between
points of a discrete planar curve. As suggested by Calabi et al., given a
transformation group $G$ and a planar curve $\mathcal{C}$, one can approximate
the fundamental differential invariant w.r.t. $G$ at a point
$\mathbf{x}\in\mathcal{C}$, which is denoted by
$\kappa\left(\mathbf{x}\right)$ and is known as the $G$-invariant curvature at
$\mathbf{x}$, by first interpolating a set
$\mathcal{P}_{\mathbf{x}}\subset\mathcal{C}$ of points from a small
neighborhood around $\mathbf{x}$ with an auxiliary curve
$\mathcal{C}_{\mathbf{x}}$, on which the differential invariant is constant by
definition. Then, the constant differential invariant
$\widetilde{\kappa}\left(\mathbf{x}\right)$ of $\mathcal{C}_{\mathbf{x}}$ is
evaluated using the joint invariants of the points in
$\mathcal{P}_{\mathbf{x}}$. Finally,
$\widetilde{\kappa}\left(\mathbf{x}\right)$ is used as an approximation to
$\kappa\left(\mathbf{x}\right)$. Since the approximation is calculated by
joint invariants of $G$, its evaluation is unaffected by the action of a group
transformation $g\in G$, and therefore, as the size of the mesh of points
$\mathcal{P}_{\mathbf{x}}$ tends to zero, the approximation of the
differential invariant at $\mathbf{x}$ converges to the continuous value. For
example, in the Euclidean case, the fundamental joint invariant of the groups
$\mathrm{E}\left(2\right)$ and $\mathrm{SE}\left(2\right)$ is the Euclidean
distance between a pair of points. By applying the approach suggested by
Calabi et al., the Euclidean curvature $\kappa\left(\mathbf{x}\right)$ can be
approximated by first interpolating a circle through $\mathbf{x_{i-1}}$,
$\mathbf{x_{i}}$, and $\mathbf{x_{i+1}}$, and then exploit the Euclidean
distances between those three points to calculate the constant curvature
$\widetilde{\kappa}\left(\mathbf{x}\right)$ of the interpolated circle, using
Heron’s formula. The second differential invariant at a curve point
$\mathbf{x}$, is defined as the derivative of the $G$-invariant curvature
$\kappa$ with respect to the $G$-invariant arc-length element $s$, and is
denoted by $\kappa_{s}\left(\mathbf{x}\right)$. According to Calabi et al.,
$\kappa_{s}\left(\mathbf{x}\right)$ is approximated using finite differences,
by calculating the ratio of the difference between
$\widetilde{\kappa}\left(\mathbf{x}_{i-1}\right)$ and
$\widetilde{\kappa}\left(\mathbf{x}_{i+1}\right)$ with respect to the distance
between $\mathbf{x_{i-1}}$ and $\mathbf{x_{i+1}}$. In their paper, Calabi et
al. further explain how to approximate differential invariants of planar
curves with respect to the equiaffine group $\mathrm{SA}\left(2\right)$. In
this case, the fundamental joint invariant is the triangle area defined by a
triplet of points, and the auxiliary curve is a conic section, which possesses
a constant equiaffine curvature, and has to be interpolated through five curve
points in the neighborhood of $\mathbf{x}$. The main limitation of their
method, in the equiaffine case, is that it is practically applicable only to
convex curves, since the equiaffine curvature is not defined at inflection
points.

Learning-Based Approximation. A more recent approach was introduced in [15].
There, Pai et al. took a self-supervised learning approach and used a Siamese
convolutional neural network to learn the euclidean curvature from a dataset
of discrete planar curves. However, the scope of their work was lacking a
module to approximate the Euclidean arc-length at each point and/or the
derivative of the curvature w.r.t. arc-length, and therefore, a signature
curve could not be generated. This means that their approach was not useful
for evaluating the equivalence between planar shapes. Moreover, they have not
extended their work to learning differential invariants of less restrictive
transformation groups, such as the equiaffine and affine groups. In [10],
Lichtenstein et al. introduced a deep learning approach to numerically
approximate the solution to the Eikonal equation. They proposed to replace
axiomatic local numerical solvers with a trained neural network that provides
highly accurate estimates of local distances for a variety of different
geometries and sampling conditions.

##  3 Mathematical Framework

We begin with the following theorem, as stated by [14, p. 176, Theorem 8.47],

######  Theorem 3.1 (The Fundamental Theorem of Differential Invariants)

A transformation group $G$ acting on $\mathbb{R}^{2}$ admits the following
properties. A unique differential invariant $\kappa$ (up to a function of it),
which is known as the $G$-invariant curvature. A unique $G$-invariant one-from
$ds=p\left(x\right)dx$ (up to a constant multiple), which is known as the
$G$-invariant arc-length element. Any other differential invariant of $G$ is a
function $I\mathrel{\mathop{\mathchar
58\relax}}\left(\kappa,\kappa_{s},\kappa_{ss},\ldots\right)\rightarrow\mathbb{R}$
of the $G$-invariant curvature $\kappa$ and its derivatives with respect to
the $G$-invariant arc length $s$.

######  Definition 1 (Equivalent Planar Curves)

Let $G$ be a transformation group acting on $\mathbb{R}^{2}$. Two planar
curves $\mathcal{C}$ and $\hat{\mathcal{C}}$ are equivalent with respect to
$G$ if there exists $g\in G$ such that $\hat{\mathcal{C}}=g\cdot\mathcal{C}$.

######  Definition 2 (Signature Curve)

The $G$-invariant signature curve associated with a parametrized planar curve
$\mathcal{C}=\left\\{\left(x\left(t\right),y\left(t\right)\right)\right\\}$,
is a curve $S\subset\mathbb{R}^{2}$ parametrized by the $G$-invariant
curvature of $\mathcal{C}$ at a point and its derivative with respect to the
$G$-invariant arc-length, and is given by
$S=\left\\{\left(\kappa\left(t\right),\kappa_{s}\left(t\right)\right)\right\\}$.

Signature curves can be used to solve the equivalence problem of planar curves
for a general transformation group $G$, as stated by the next theorem, given
in [14, p. 183, Theorem 8.53],

######  Theorem 3.2 (Equivalence of Planar Curves)

Two non-singular planar curves $\mathcal{C}$ and $\hat{\mathcal{C}}$ are
equivalent w.r.t. a transformation group $G$ if and only if their signature
curves $S$ and $\hat{S}$ are equal.

For more information about differential invariants and signature curves, see
[12, 13, 14, 3, 2, 1, 8, 9, 4, 7, 3].

##  4 Method

We introduce a simple neural-net architecture, along with an appropriate
training scheme and loss function, for producing numerically stable
approximations of the two fundamental differential invariants $\kappa$ and
$\kappa_{s}$, with respect to a group transformation $G$. Both quantities can
be combined together to evaluate and plot the $G$-invariant signature curve of
a planar curve. Given a planar curve $\mathcal{C}$ and a point
$\mathbf{p}\in\mathcal{C}$ on the curve, the neural network receives as an
input a discrete sample of the local neighborhood of $\mathbf{p}$, and outputs
two scalars, of which we interpret as $\kappa$ and $\kappa_{s}$ at
$\mathbf{p}$.

###  4.1 Learning Differential Invariants

There are two fundamental ideas that govern our training approach for learning
a differential invariant w.r.t. a transformation group $G$.

Group and Reparametrization Invariance. The neural network should learn a
representation that is both invariant to the action of a group transformation
$g\in G$, and to a reparametrization of the input curve. That way, the network
will learn a truly invariant representation, which is not biased towards any
specific discrete sampling scheme, and that encapsulates the local geometric
properties of the curve. Given a discrete planar curve $\mathcal{C}$, we
denote by $\mathcal{C}\left(\mathcal{D}\right)$ its down-sampled version,
sampled non-uniformly with $N$ points according to a random non-uniform
probability mass function $\mathcal{D}\mathrel{\mathop{\mathchar
58\relax}}\mathcal{C}\rightarrow\left[0,1\right]$. Given two different non-
uniform probability mass functions $\mathcal{D}_{1}$ and $\mathcal{D}_{2}$,
one can refer to the down-sampled curves
$\mathcal{C}\left(\mathcal{D}_{1}\right)$ and
$\mathcal{C}\left(\mathcal{D}_{2}\right)$, as two different reparametrizations
of $\mathcal{C}$. This motivates us to require that if a point
$\mathbf{p}\in\mathcal{C}$ was drawn both in
$\mathcal{C}\left(\mathcal{D}_{1}\right)$ and
$\mathcal{C}\left(\mathcal{D}_{2}\right)$, then, a valid prediction model
should output the same curvature value for $\mathbf{p}$, whether it was fed
with either a neighborhood of $g_{1}\cdot\mathbf{p}$ or a neighborhood
$g_{2}\cdot\mathbf{p}$ as an input, for any two group transformations
$g_{1},g_{2}\in G$.

Orthogonality of Differential Invariants. We expect $\kappa$ and $\kappa_{s}$
to be orthogonal quantities, since the latter is the derivative of the former
w.r.t. to arc-length. So, for example, consider a collection of $M$ points,
such that each was sampled from an arbitrary planar curve. Moreover, let
$\kappa^{i}$ and $\kappa_{s}^{i}$ denote the curvature and its derivative
w.r.t. arc-length evaluated at the $i^{th}$ point. Then, if we consider
$\left\\{\kappa^{i}\right\\}_{i=1}^{M}$ and
$\left\\{\kappa_{s}^{i}\right\\}_{i=1}^{M}$ as $M$ observations of random
variables $X_{\kappa}$ and $X_{\kappa_{s}}$, respectively, we expect the
absolute value of the Pearson correlation coefficient
$\absolutevalue{\rho\left(X_{\kappa},X_{\kappa_{s}}\right)}$ to approach zero
as $M\to\infty$. We empirically verify this assumption in the Euclidean case,
by calculating the Pearson correlation coefficient of $X_{\kappa}$ and
$X_{\kappa_{s}}$ on a collection of randomly sampled points, taken from a
large collection of smooth curves. See Figure 1.

Figure 1: Left: Plot of a smooth curve and its Euclidean $\kappa$ and
$\kappa_{s}$ evaluated at each point. Right: A semi-log plot of the absolute
value of the Pearson correlation coefficient of $X_{\kappa}$ and
$X_{\kappa_{s}}$ as a function of points count.

###  4.2 Training Scheme

Given a dataset of $n$ discrete planar curves
$\left\\{\mathcal{C}_{i}\right\\}_{i=1}^{n}$, our training scheme generates a
training batch on the fly, where each batch is made of a set of training
tuplets. A training tuplet
$\mathcal{T}=\left(\mathcal{E}_{a},\mathcal{E}_{p},\mathcal{E}_{n_{1}},\ldots,\mathcal{E}_{n_{m}}\right)$,
is made up of a collection of sampled neighborhoods, which are referred to as
the anchor ($\mathcal{E}_{a}$), positive ($\mathcal{E}_{p}$) and negative
($\left\\{\mathcal{E}_{n_{i}}\right\\}_{i=1}^{m}$) examples. The tuplet is
generated by first drawing a random curve $\mathcal{C}$ and a random point
$\mathbf{p}\in\mathcal{C}$ on it. Given two random group transformations
$g_{a},g_{p}\in G$ and two random non-uniform probability mass functions
$\mathcal{D}_{a}$ and $\mathcal{D}_{p}$, both the anchor and positive
examples, $\mathcal{E}_{a}$ and $\mathcal{E}_{p}$, are generated by selecting
$2N$ adjacent points to $\mathbf{p}$ ($N$ consecutive points that immediately
precede $\mathbf{p}$, and additional $N$ consecutive points that immediately
succeed $\mathbf{p}$) from the transformed and down-sampled versions of
$\mathcal{C}$ given by $g_{a}\cdot\mathcal{C}_{a}\left(\mathcal{D}_{a}\right)$
and $g_{p}\cdot\mathcal{C}_{p}\left(\mathcal{D}_{p}\right)$, respectively. In
other words, both $\mathcal{E}_{a}$ and $\mathcal{E}_{p}$ represent a local
neighborhood of $\mathbf{p}$ sampled under different reparametrizations, which
were transformed by group transformations taken from the same transformation
group $G$.

The $i^{\mathrm{th}}$ negative example $\mathcal{E}_{n_{i}}$ is generated in a
similar manner. We draw another random curve $\mathcal{C}_{i}$, and take a
random point $\mathbf{p}_{i}\in\mathcal{C}_{i}$ on it. Then, we select again
additional $2N$ adjacent points to $\mathbf{p}_{i}$ from a transformed and
down-sampled version of $\mathcal{C}_{i}$ given by
$g_{n_{i}}\cdot\mathcal{C}_{i}\left(\mathcal{D}_{i}\right)$, where
$g_{n_{i}}\in G$ is a random group transformation, and $\mathcal{D}_{i}$ is a
random probability mass function.

Given a batch of $K$ training tuplets
$\left\\{\mathcal{T}_{i}\right\\}_{i=1}^{K}$, we train a multi-head Siamese
fully-connected neural network by minimizing the loss given by,

$\displaystyle\mathcal{L}$ $\displaystyle=$
$\displaystyle\mathcal{L}_{I}+\mathcal{L}_{O}$ (1)

Such that $\mathcal{L}_{I}$ and $\mathcal{L}_{O}$ are given by,

$\displaystyle\mathcal{L}_{I}=\frac{1}{K}\sum_{i=1}^{K}\mathcal{L}_{\text{tuplet}}^{i}$
(2)
$\displaystyle\mathcal{L}_{\text{tuplet}}^{i}=\log\left(1+\sum_{j=1}^{m}\exp\left(\lVert\textbf{x}^{i}_{a}-\textbf{x}^{i}_{p}\rVert-\lVert\textbf{x}^{i}_{a}-\textbf{x}^{i}_{n_{j}}\rVert\right)\right)$
(3)
$\displaystyle\mathcal{L}_{O}=\frac{1}{m+2}\left(\absolutevalue{\rho\left(\left\\{\textbf{x}_{a}^{i}\right\\}_{i=1}^{K}\right)}+\absolutevalue{\rho\left(\left\\{\textbf{x}_{p}^{i}\right\\}_{i=1}^{K}\right)}+\sum_{j=1}^{m}\absolutevalue{\rho\left(\left\\{\textbf{x}_{n_{j}}^{i}\right\\}_{i=1}^{K}\right)}\right)$
(4)
$\displaystyle\rho\left(\left\\{\mathbf{x}^{i}\right\\}_{i=1}^{K}\right)=\frac{\sum_{i=1}^{K}\left(\mathbf{x}^{i}\left[0\right]-\bar{\mathbf{x}}\left[0\right]\right)\left(\mathbf{x}^{i}\left[1\right]-\bar{\mathbf{x}}\left[1\right]\right)}{\sqrt{\sum_{i=1}^{K}\left(\mathbf{x}^{i}\left[0\right]-\bar{\mathbf{x}}\left[0\right]\right)^{2}\sum_{i=1}^{K}\left(\mathbf{x}^{i}\left[1\right]-\bar{\mathbf{x}}\left[1\right]\right)^{2}}}$
(5)

Where $\textbf{x}_{a}^{i}$,
$\textbf{x}_{p}^{i},\textbf{x}_{n_{1}}^{i},\ldots,\textbf{x}_{n_{m}}^{i}\in\mathbb{R}^{2}$
are the neural-network’s outputs for the $i^{th}$ input tuplet given by
$\mathcal{T}_{i}=\left(\mathcal{E}_{a}^{i},\mathcal{E}_{p}^{i},\mathcal{E}_{n_{1}}^{i},\ldots,\mathcal{E}_{n_{m}}^{i}\right)$,
and where $\mathbf{x}\left[0\right]$ and $\mathbf{x}\left[1\right]$ are the
first and second components of an output vector, respectively. Moreover,
$\bar{\mathbf{x}}\left[0\right]$ and $\bar{\mathbf{x}}\left[1\right]$ are the
mean values of the first and second components of a set of output vectors,
respectively. Equation 3 is known as the tuplet loss [17] which is minimized
when the distance between $\textbf{x}_{a}^{i}$ and $\textbf{x}_{p}^{i}$ is
minimized, and the distance between $\textbf{x}_{a}^{i}$ and
$\textbf{x}_{n_{j}}^{i}$ is maximized, for $j=1,\ldots,m$. Equation 2 is named
the invariance loss and it is equal to the mean of the tuplet loss over all
the training tuplets in the training batch. Equation 4 is named the
orthogonality loss, and it is minimized when the absolute value of the Pearson
correlation coefficient is minimized w.r.t. the anchor, positive, and negative
examples’ outputs, over the whole batch. See Figure 2 for an elaborate visual
demonstration of the training scheme for a batch of training tuplets. Note,
that we eliminate any translation and rotational ambiguity by transforming
each input example into a canonical placement before it is fed into the
neural-network. The transformation is done by translating each example such
that its first point is located at the origin and then, rotating it such that
the vector that starts at the example’s first point and ends at the middle
point is aligned with the positive x-axis.

Figure 2: A demonstration of the training scheme for a given input batch of
training tuplets. Left-to-right: A random curve and a random point on it are
drawn. The curve is down-sampled non-uniformly. A local neighborhood around
the point is extracted. The extracted point neighborhood is transformed into a
canonical placement, as described above. The canonical point neighborhood is
fed into the neural-network, which outputs two scalars that we interpret as
$\kappa$ and $\kappa_{s}$. Top-to-bottom: The anchor and positive training
examples, as well as the $m$ negative training examples. Together, they form a
training tuplet.

##  5 Experiments and Results

###  5.1 Datasets and Training

We create three datasets of planar curves, by extracting level-curves from
arbitrary images scrapped from the internet. The first two datasets are used
for training and validation, and the third dataset is used for qualitative
evaluation. Each dataset consists of tens of thousands of curves. We train a
multi-head siamese neural-network based on a simple fully-connected
architecture. The architecture is based on blocks of 3 fully-connected layers
of the same size. The layers at the first block are of size 128, and the size
of the layers in each subsequent block is divided by two w.r.t. the size of
layers in the previous block. We use a batch-norm layer after each linear
layer, followed by a sine activation function. The reason for using a periodic
activation function is due to the fact that its derivative can be expressed by
a phase-shift. We believe that this fact makes it easier for our network to
learn high-order derivatives [16].

###  5.2 Qualitative Evaluation

We evaluate our differential invariants approximation model qualitatively,
w.r.t. the affine group. We focus on interpretability, group invariance,
reparametrization invariance, and comparison against the Euclidean axiomatic
signature.

Interpretability. As shown in Figure 3, our model’s outputs can be interpreted
as $\kappa$ and $\kappa_{s}$. The first output, which we interpret as
$\kappa$, approximately reaches its extrema points when the second output,
which we interpret as $\kappa_{s}$, crosses the zero line.

Figure 3: Two discrete planar curves and the plots of our model’s
approximation for their affine $\kappa$ and $\kappa_{s}$ as a function of
point-index, and $\kappa_{s}$ as a function of $\kappa$ (affine signature
curve).

Group Action Invariance. As shown in Figure 4, our approximation model
exhibits clear group invariance w.r.t. the affine group. The peaks and valleys
of the plots of $\kappa$ and $\kappa_{s}$ of the two equivalent curves (the
red and blue curves) are aligned correctly, and the relative amplitude
amplification is mild.

Figure 4: A comparison of our model’s approximation for the affine $\kappa$
and $\kappa_{s}$ as a function of point-index, of two pairs of equivalent
curves. The blue curve was obtained by applying a linear operator on the red
one. Left: Linear operator with condition number 2.5 and determinant 3.5.
Right: Linear operator with condition number 5 and determinant 2.

Reparametrization Invariance. As shown in Figure 5, our approximation model is
robust to reparametrization and non-uniform down-sampling. The peaks and
valleys patterns of $\kappa$ and $\kappa_{s}$ w.r.t. the reference curve are
clearly reproduced in its down-sampled and transformed version (matching
colors).

Figure 5: A demonstration of our model’s robustness for non-uniform down-
sampling. Left: Our model’s approximation for $\kappa$ and $\kappa_{s}$ of a
reference planar curve. Right: Our model’s approximation for $\kappa$ and
$\kappa_{s}$ of the reference curve, after it was non-uniformly down-sampled
by 50%, and an arbitrary affine transform was applied to it.

Comparison with the Euclidean Axiomatic Approach. As shown in Figure 6, our
model’s approximations for $\kappa$ and $\kappa_{s}$ w.r.t. the affine group
are superior over the axiomatic Euclidean approach (which is based on finite
differences). As expected, the axiomatic signature exhibits visible
misalignment of $\kappa$ and $\kappa_{s}$ w.r.t. the two equivalent curves, as
well as major amplitude amplification differences. On the other hand, the
approximations made by our model are aligned while only a mild amplitude
amplification is visible.

Figure 6: Two affine equivalent curves (blue and red) and the approximation of
their differential invariants. Left: The approximation of the affine $\kappa$
and $\kappa_{s}$ using our trained model. Right: The approximation of the
Euclidean $\kappa$ and $\kappa_{s}$ using the axiomatic approach, which is
based on finite differences.

###  5.3 Quantitative Evaluation

We have compiled an evaluation benchmark of 34 collections of planar curves,
which were extracted from the boundary of silhouettes of natural objects,
which are not necessarily smooth (see Figure 7). Each collection is comprised
of 30 boundary curves of the same conceptual object in different poses and
deformations. We perform a simple shape-matching evaluation which goes as
follows. We iterate over all boundary curves in all collections. Given the
$j^{\text{th}}$ curve in the $i^{\text{th}}$ collection, we deform it using an
affine transformation and then down-sample it, in order to get what we call as
a query curve. Then, we compare the query curve against all curves in its
collection (which we refer to as database curves). A query curve is compared
with a database curve by calculating the average Hausdorff distance of their
corresponding signature curves, which were evaluated by our model. If the
minimum distance was acquired by comparing a curve with its own transformed
and down-sampled version, then we call it a successful match. We score each
collection by calculating the average number of successfully matched curves,
which we denote by the success rate of the collection. We perform this
experiment under various down-sampling ratios and affine transformations. We
use this benchmark to evaluate our model as well as the axiomatic
approximations for the Euclidean and equiaffine groups, as proposed by [5]. We
run 3 different flavors of our benchmark. For the first flavor, we transform
each curve by an affine transform $A\in\mathbb{R}^{2\times 2}$, where
$\det\left(A\right)=2$ and $\text{cond}\left(A\right)=2$. For the second
flavor we have $\det\left(A\right)=2$ and $\text{cond}\left(A\right)=3$, and
for the third flavor we have $\det\left(A\right)=3$ and
$\text{cond}\left(A\right)=2$. In all 3 flavors, we down-sample the query
curves by increasing sampling ratios, where a sampling ratio of 100% means
that we keep all points, a sampling rate of 90% means that we drop 10% of the
points, and so on. Evaluation results for the bunnies collection are available
in Table 1.

Figure 7: Top: A sample of our benchmark’s curve collections. We plot four
samples (out of 30) from each collection. Bottom: A demonstration of our
shape-matching benchmark for the bunnies collection. Each transformed and
down-sampled curve on the left should be matched with its equivalent curve on
the right. Although in this figure only 9 curves are shown, in the actual
shape-matching benchmark each collection consists of 30 curves that need to be
matched. Table 1: Shape-Matching Evaluation (Bunnies Collection)

| Success Rate
---|---
| Our Model | Euclidean Axiomatic | Equiaffine Axiomatic
Sampling Rate |  $\det=2$ $\text{cond}=2$ |  $\det=2$ $\text{cond}=3$ |  $\det=3$ $\text{cond}=2$ |  $\det=2$ $\text{cond}=2$ |  $\det=2$ $\text{cond}=3$ |  $\det=3$ $\text{cond}=2$ |  $\det=2$ $\text{cond}=2$ |  $\det=2$ $\text{cond}=3$ |  $\det=3$ $\text{cond}=2$
100.00% | 100.00% | 100.00% | 70.00% | 3.33% | 16.67% | 10.00% | 16.67% | 10.00% | 3.33%
90.00% | 100.00% | 100.00% | 63.33% | 13.33% | 6.67% | 10.00% | 3.33% | 0.00% | 3.33%
80.00% | 90.00% | 90.00% | 56.67% | 10.00% | 6.67% | 10.00% | 0.00% | 0.00% | 3.33%
70.00% | 83.33% | 90.00% | 46.67% | 6.67% | 0.00% | 3.33% | 3.33% | 0.00% | 0.00%
60.00% | 63.33% | 73.33% | 43.33% | 13.33% | 10.00% | 6.67% | 13.33% | 3.33% | 3.33%
50.00% | 56.67% | 46.67% | 33.33% | 3.33% | 3.33% | 3.33% | 3.33% | 3.33% | 3.33%

##  6 Conclusions and Future Research

We proposed a self-supervised approach, using multi-head Siamese neural
networks, for the approximation of differential invariants of planar curves,
with respect to a given transformation group $G$. We have shown qualitatively
that our model is invariant w.r.t. the group actions, as well as
reparametrization and non-uniform down-sampling. We have also demonstrated the
applicability of our method to the task of shape-matching. As a future
research, we aim to design learning models for the approximation of higher
dimensional differential invariants [11], such as the torsion of space curves,
and the principal curvatures of surfaces embedded in a Euclidean space.

## References

  * [1] Bruckstein, A., Netravali, A.: On differential invariants of planar curves and recognizing partially occluded planar shapes. Annals of Mathematics and Artificial Intelligence (1995)
  * [2] Bruckstein, A.M., Katzir, N., Lindenbaum, M., Porat, M.: Similarity-invariant signatures for partially occluded planar shapes. Int. J. Comput. Vision (1992)
  * [3] Bruckstein, A., Holt, R., Netravali, A., Richardson, T.: Invariant signatures for planar shape recognition under partial occlusion (1992)
  * [4] Bruckstein, A., Shaked, D.: Skew symmetry detection via invariant signatures. Pattern Recognition (1998)
  * [5] Calabi, E., Shakiban, C., Olver, P.J., Haker, S.: Differential and numerically invariant signature curves applied to object recognition. Int. J. Computer Vision (1998)
  * [6] Cartan, E.: La méthode du repère mobile, la théorie des groupes continus et les espaces généralisés. The Mathematical Gazette (1935)
  * [7] Catté, F., Lions, P.L., Morel, J.M., Coll, T.: Image selective smoothing and edge detection by nonlinear diffusion. SIAM J. Numer. Anal. (1992)
  * [8] Kimmel, R.: Affine differential signatures for gray level images of planar shapes (1996)
  * [9] Kimmel, R., Zhang, C., Bronstein, A.M., Bronstein, M.M.: Are mser features really interesting? IEEE Transactions on Pattern Analysis and Machine Intelligence (2011)
  * [10] Lichtenstein, M., Pai, G., Kimmel, R.: Deep eikonal solvers. In: Lellmann, J., Burger, M., Modersitzki, J. (eds.) Scale Space and Variational Methods in Computer Vision. Springer (2019)
  * [11] Olver, P.: Differential invariants of surfaces. Differential Geometry and its Applications (2009)
  * [12] Olver, P., Sapiro, G., Tannenbaum, A.: Differential invariant signatures and flows in computer vision: A symmetry group approach. Springer (1994)
  * [13] Olver, P.J.: Equivalence, Invariants and Symmetry. Cambridge University Press (1995)
  * [14] Olver, P.J.: Classical Invariant Theory. London Mathematical Society Student Texts, Cambridge University Press (1999)
  * [15] Pai, G., Wetzler, A., Kimmel, R.: Learning invariant representations of planar curves. In: 5th Int. Conf. on Learning Representations (2017)
  * [16] Sitzmann, V., Martel, J.N., Bergman, A.W., Lindell, D.B., Wetzstein, G.: Implicit neural representations with periodic activation functions. In: Proc. NeurIPS (2020)
  * [17] Yu, B., Tao, D.: Deep metric learning with tuplet margin loss. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (2019)