# Towards Robust Speaker Verification with Target Speaker Enhancement

###### Abstract

This paper proposes the target speaker enhancement based speaker verification
network (TASE-SVNet), an all neural model that couples target speaker
enhancement and speaker embedding extraction for robust speaker verification
(SV). Specifically, an enrollment speaker conditioned speech enhancement
module is employed as the front-end for extracting target speaker from its
mixture with interfering speakers and environmental noises. Compared with the
conventional target speaker enhancement models, nontarget speaker/interference
suppression should draw additional attention for SV. Therefore, an effective
nontarget speaker sampling strategy is explored. To improve speaker embedding
extraction with a light-weighted model, a teacher-student (T/S) training is
proposed to distill speaker discriminative information from large models to
small models. Iterative inference is investigated to address the noisy speaker
enrollment problem. We evaluate the proposed method on two SV tasks, i.e., one
heavily overlapped speech and the other one with comprehensive noise types in
vehicle environments. Experiments show significant and consistent improvements
in Equal Error Rate (EER) over the state-of-the-art baselines.

Index Terms: robust speaker verification, target speaker enhancement, speaker
embedding, deep learning, overlapped speech

##  1 Introduction

Deep neural network (DNN) based speaker embedding models outperform the
conventional models (e.g., i-vector) in many speaker verification (SV) tasks
[1, 2, 3, 4, 5, 6, 7], becoming the state-of-the-art paradigms of speaker
recognition technology. The improved performance can be attributed to three
major reasons: a) deep (sometimes very deep) neural network architectures that
can effectively model speaker discriminative information from utterances [2,
5, 7, 8]; b) novel neural network training objectives (e.g., large margin
softmax losses, contrastive learning based losses or end-to-end losses etc.)
and the corresponding training strategies [6, 8, 5, 9, 10]; c) sufficient
training data distributed across diverse acoustic conditions [2].

Despite the success achieved, current SV systems still suffer evident
performance degradation in the presence of severe background noise with
interfering speakers [11, 12, 13, 14, 15]. Therefore, the SV system is at
increased risk of failure when exposed to unconstrained conditions. Front-end
techniques to tackle against such adverse environments include speaker
diarization, noise suppression, and speech separation. In [16], it is
suggested that speaker diarization works well for the multi-speaker scenario
where overlapped speech is not the main focus. However, such system fails when
highly overlapped speech exists. In addition, the system performance heavily
relies on speaker diarization accuracy, which remains to be challenging at
high noise levels. On the contrary, a noise suppression model might perform
well for general noise reduction, however, the speaker of interest from multi-
speaker utterances could not be easily extracted through such models. The
speech separation technologies also provide a potential solution for
extracting target speaker from the mixture. The major drawback lies in the
model inference, where the number of speakers has to be known in advance. In
practice, such information is usually unknown from a test utterance. Due to
those constraints, successful adoption of the aforementioned front-end
techniques are rarely seen in generic SV systems.

In this study, we propose the target speaker enhancement based speaker
verification network (TASE-SVNet), a SV framework that is robust against
background noise and interfering speakers. For speech recognition models with
target speaker enhancement [17, 18, 19], speaker profile utterances require
additional cost to collect. While in SV tasks, enrollment utterance of a
speaker is naturally given, which could be utilized as the anchor/profile
signal for target speaker enhancement. Specifically, following our previously
formulated time-domain target speaker enhancement model [20], the joint
training strategy of target speaker embedding extraction and speech
enhancement is expected to offer well-performed target speaker extraction for
“target” test utterance 111“Target” test utterance refers to the test
utterance of a target trial, where speaker of interest is included in the test
utterance. “Nontarget” test utterance refers to the test utterance of a
nontarget trial, where the speakers in the test utterance are not the enrolled
speaker.. Furthermore, to encourage the speech enhancement module to suppress
noise and nontarget speakers for ”nontarget” test utterance, a novel nontarget
sample training strategy is introduced as an additional constraint to the
target speaker enhancement training, which effectively prevents the increase
of False Alarm Rate (FAR) in the SV phase. Regarding the SV model in the TASE-
SVNet framework, a teacher-student (T/S) training scheme is investigated to
improve the overall SV performance with a light-weighted model. To put it
together, the target speaker enhancement module and the SV model of TASE-SVnet
are unified in a multi-stage training procedure.

Integrating target speaker enhancement to robust SV is still at its early
stage. The most relevant work to us is [15], where speaker-beam based target
speaker enhancement is evaluated [19], and an i-vector model is employed as
the SV model. In contrast with the pipeline system [15], we expect an all-
neural network solution with joint training leads to improved SV performance.

The paper is structured as follows. Sec.2 describes the overall TASE-SVnet
framework with a time-domain target speaker enhancement module, a speaker
embedding module, and the improved training for each module. Sec.3 details the
experimental setup and the corpora. Sec.4 presents experimental results and
analysis. Finally, we conclude the paper in Sec.5.

##  2 Speaker verification with target speaker enhancement

In this section, we firstly introduce the overall framework of TASE-SVNet.
Then, we provide our improved training strategies of the target speaker
enhancement module and the speaker embedding module in the following
subsections. Lastly, we summarise the multi-stage training procedure of TASE-
SVNet.

Fig. 1: The flow diagram of TASE-SVNet framework.

###  2.1 The TASE-SVNet framework

The overall TASE-SVNet framework consists of a target speaker enhancement
module and a speaker verification module. Fig.1 illustrates the base framework
of the proposed TASE-SVNet. The target speaker enhancement module in the left
part of Fig.1 is inherited from one of our recently developed models, which
gives the best speech enhancement performance in [20]. Scale-invariant signal-
to-noise ratio (SI-SNR) [20, 21] is used as the target speaker enhancement
objective function, which is defined as:

$\text{SI-
SNR}=20\log_{10}\frac{\|\alpha\cdot\mathbf{s}_{e}\|}{\|\mathbf{s}_{t}-\alpha\cdot\mathbf{s}_{e}\|},$
(1)

where $\mathbf{s}_{e},\mathbf{s}_{t}$ are estimated signal and target signal,
respectively, for which zero-mean normalization is applied. $\alpha$ is an
optimal scaling factor computed via
$\alpha=\mathbf{s}_{e}^{T}\mathbf{s}_{t}/\mathbf{s}_{t}^{T}\mathbf{s}_{t}$.

Fig.2 depicts the detailed implementation of joint learning architecture,
where a modified time-domain Conv-TasNet structure is employed as the speaker
enhancement network and a TDNN architecture (i.e., spk embd net 1) is utilized
to provide the bias information from speaker of interest (i.e., meaning
pooling over enrollment speaker embeddings), respectively. To jointly learn
speaker embedding and target speaker enhancement, a multi-task loss
$\mathcal{L}_{TASE}$ is formulated with equal weight combination of the target
speaker enhancement loss $\mathcal{L}_{SI-SNR}$ and the speaker verification
loss $\mathcal{L}_{SV}$. Furthermore, $\mathcal{L}_{SV}$ is also a multi-task
objective which consists of a large margin cosine loss $\mathcal{L}_{lmc}$
(LMCL) [22] and a triplet loss $\mathcal{L}_{triplet}$ [3] with a tuned
weight. The total loss is defined as follows:

$\mathcal{L}_{SV}=\mathcal{L}_{triplet}+\omega_{1}\mathcal{L}_{lmc}+\omega_{2}\mathcal{L}_{r},$
(2)

where $L_{r}$ is a $L_{2}$-regularization term which alleviates over-fitting
during training, $\omega_{1}$ and $\omega_{2}$ are the weights determined in
the experiments. We refer the readers to [20] for more detailed network
configuration and training procedures.

For the SV model in the right part of Fig.1 (i.e., spk embd net 2), two
network architectures are introduced as the benchmarks for evaluating SV
performance. Specifically, a TDNN model (the same as spk embd net 1) and a
more heavy-weighted Resnet-50 are investigated [23, 24] as contrastive
systems. Both speaker embedding networks (i.e., spk embd net 1 & 2) in Fig.1
are pre-trained on a fairly large data set particularly for the task of SV.
More details about the experimental setups can be found in Sec. 3.

Fig. 2: Joint learning for target speaker enhancement.

###  2.2 Nontarget sample training in TASE-SVNet

To the best of our knowledge, all enrollment speaker informed target speaker
enhancement models [18, 19, 17, 20] are only evaluated on one condition, which
assumes target speaker is $always$ presented in the test utterance. This is a
strong prior that does not hold for SV tasks. We argue that the current target
speaker enhancement training criterion is inclined to make the network output
of nontarget test utterance more biased to enrolled speaker, leading to an
increased FAR in terms of SV performance. Practically, we find our proposed
base target speaker enhancement models occasionally fail to suppress
interference speakers in the nontarget test utterance.

To address this issue, we propose to add nontarget samples to the target
speaker enhancement training. Specifically, a triplet consisting of enrollment
speaker utterances, a nontarget test utterance and a NULL reference signal, is
defined as the nontarget sample in our study. With the negative samples
randomly added in model training, the target speaker embedding module is
expected to have better performance in interference speaker suppression. To
unify the nontarget sample training with the conventional ones (only with
target samples), we update the SI-SNR loss in Eq.(1) as a new training
objective for the wohle TASE-SVnet:

$\text{SI-
SNR}=20\log_{10}\frac{\|\alpha\cdot\mathbf{s}_{e}\|}{\|\mathbf{s}_{t}-\alpha\cdot\mathbf{s}_{e}\|},\left\\{\begin{aligned}
\mathbf{s}_{t}&=\mathbf{x}_{t}&tar\\\ \mathbf{s}_{t}&\sim
N(0,\sigma^{2})&non\end{aligned}\right.$ (3)

where $\mathbf{x}_{t}$ is the zero-mean normalized target reference signal,
and the NULL reference signal in nontarget triple is set to follow a zero-mean
normal distribution, with $\sigma$=1e-6 as the stand deviation. Ideally, the
NULL reference signal should be an all zero sequence. However,
$\mathbf{s}_{t}=\mathbf{0}$ will lead to zero loss, the gradient from
nontarget sample will not apply to model update. Therefore, our solution is to
provide a small-valued Gaussian disturbance.

###  2.3 T/S training for speaker embedding training

T/S learning has been proved to be effective for domain adaptation in speech
processing [25, 26]. By T/S training, it is expected to learn a student model
that can perform well in target domain with domain specific knowledge
distillation from a well-trained source-domain teacher model.

In this study, we aim to further improve the small TDNN model with the large
Resnet-50 model with T/S training. The T/S training objective is fairly
simple. The pre-trained Resnet-50 model will act as the teacher model, the
output of student TDNN model (i.e., speaker embedding) is forced to be close
to that from the teacher model. To achieve this, we propose to use mean
squared error (MSE) as the T/S training objective:

$\mathcal{L}_{MSE}=\frac{1}{N}\frac{1}{D}\sum_{i=1}^{N}\sum_{j=1}^{D}(Y_{t}(i,j)-Y_{s}(i,j))^{2},$
(4)

where $D$ is the dimension of speaker embedding, $N$ is the number of speaker
embeddings in one training batch, $Y_{t}$ and $Y_{s}$ represent the speaker
embeddings from the teacher model and the student model, respectively.
Theoretically, T/S learning can be performed in an unsupervised manner. In our
experiments, we find that convergence of the student TDNN model can be
achieved much faster with a joint SV loss described in Eq.2, which requires
labeled training data. The overall training loss $\mathcal{L}_{S}$ for the
student model is as follows:

$\mathcal{L}_{S}=\mathcal{L}_{SV}+\mathcal{L}_{MSE},$ (5)

###  2.4 A multi-stage training procedure in TASE-SVNet

Although all the modules of the TASE-SVNet framework can be jointly optimized
from scratch. However, not much benefits have been seen with the end to end
training. In this section, a road map to develop the final system is
presented, where different modules are unified with a multi-stage training
strategy, resulting in consistent system performances with less data
constraints. More specifically, the multi-stage training procedure is
summarized as:

  * 1)

Pre-train speaker embedding models (“spk embd net” 1 & 2) with a large scale
general domain speaker recognition corpus. Apply T/S training to the TDNN
model, employ the T/S learned TDNN model as the initialization of “spk embd
net”.

  * 2)

Jointly train spk embd net 1 (i.e., initialized with T/S trained TDNN) and
target speaker enhancement network (from scratch) with a target speaker
enhancement corpus.

  * 3)

Fix the target speaker enhancement module when training has converged. Fine
tune the spk embd net 2 using the triplet loss with the enhanced and raw
speech data.

##  3 Corpora and experimental setup

###  3.1 Speech corpora

####  3.1.1 SV corpora

Two datasets are utilized to examine our base SV models. The first one is a
Mandarin language corpus. The training set contains of 8800 speaker drawn from
two gender-balanced public speech recognition
datasets222http://en.speechocean.com/datacenter/details/254.htm. The training
data is then augmented 2 folds to incorporate variabilities from distance
(reverberation), channel or background noise, resulting in a training pool
with 2.8M utterances. The duration ranges from 0.5s to 13.2s, with the average
duration of 3.8s. Two real SV test sets are examined. Test set 1 is a
44-speaker corpus specifically for multi-speaker SV evaluation (noted as
“multi-spk”), each utterance may include 1-3 speakers with or without
overlapping. We generate 10K trials from the “multi-spk” test set, among which
1K trails are target trials. For speaker enrollment, 3 utterances are randomly
selected, where only single-speaker utterance is used for enrollment. Test set
2 is a 365-speaker dataset collected under the unconstrained vehicle
environments (noted as “vehicle-spk”). It is a much larger dataset that covers
most of the noise types (i.e., wind noise, mechanical noise, background
music/noise, and interference speakers etc.). Similar as the “multi-spk” for
trial generation, we randomly select 3 utterances for speaker enrollment, and
sample the rest for test. 100K trials are produced in the end, among which 10K
trials are target trials. For system comparisons, we also report our SV
baselines on the Voxceleb task [23, 8]. The performance is reported on
Voxceleb1 test trials.

####  3.1.2 Target speaker enhancement corpus

For training the target speaker enhancement model, we use the same corpus as
[20], which is simulated from AISHELL-2 [27]. The whole dataset is split to a
1900-speaker training set, a 60-speaker validation set and a 30-speaker test
set. For each speaker (as a target speaker), we randomly select 50 utterances
of this speaker and mix each of them with utterances from other speakers. As a
result, we generate 95K, 3K, and 1.5K mixed utterances for training,
validation and test, respectively. The signal-to-interference ratio (SIR) is
equally distributed in [0dB, 6dB, 12dB, inf]. The number of speakers in the
mixed signal ranges from 1 to 3 evenly. And the environmental noises are added
to the speech mixture with signal-to-noise ratio (SNR) randomly sampled from
[6dB, 12dB, 18dB, 24dB, 30dB].

###  3.2 System setups

For speaker embedding models, 257-d raw short time fourier transform (STFT)
features are extracted with a 32ms window and the time shift of feature frames
is 16ms. The non-speech part is removed by a critical-band power spectrum
variance based voice activity detector (VAD), which provides better speech
detection than the energy based VAD. The voiced part is randomly segmented
into 100-200 frames for network training. A linear projection layer of 128-d
is added to the TDNN and Resnet architecture for speaker embedding extraction,
where $L_{2}$ norm is applied as an unit energy constraint. $\omega_{1}$=0.2
and $\omega_{2}$=0.001 in Eq.(2) are determined in the experiments.

We retain most of the hyper-parameters same with the target speaker
enhancement model presented in [20], except for the nontarget sample training.
In our experiments, we find that the ratio 11:1 between target and nontarget
samples are empirically good, which provides a balanced performance of target
speaker extraction and interference speaker suppression. In the SV part of
TASE-SVNet, we fine tune the SV model with enhanced speech. To prevent the
fine-tuned “spk embd net 2” from being over-fitted on the training data, we
use a small learning rate of 1e-6 with just one epoch of fine-tuing.

##  4 Results

###  4.1 Baseline SV results

We first evaluate the baseline SV systems with the test corpora, i.e.,
Voxceleb, overlap-spk and vehicle-spk. We also reference the state-of-the-art
Voxceleb benchmark results for fair comparison.

Table 1:  SV baseline results (EER). model | Voxceleb | overlap-spk | vehicle-spk
---|---|---|---
x-vector [2] | 3.14% | / | /
TDNN [20] | 2.10% | / | /
TDNN [6] | 2.00% | / | /
TDNN (ours) | 1.95% | 16.32% | 4.15%
Resnet-50 (ours) | 1.67% | 15.45% | 3.72%
TDNN (ours T/S) | 1.76% | 15.91% | 3.93%

As illustrated on Table 1, our proposed TDNN (6.5M) baseline could achieve
comparable result as [6] on Voxceleb. The large Resnet-50 model (31.4M) is
able to approach the state-of-the-art single system performance reported in
[28], where deeper Resnet models are used for speaker embedding training. For
the “overlap-spk” dataset, where only utterances with overlapped speakers are
used for test, superb performance is no longer hold as the Voxceleb test set.
For a more general purpose “vehicle-spk” corpus, although performance
degradation is not as big as the overlapping corpus, there still exists a big
performance drop, showing the difficulties for SV in unconstrained
environments. It is also noted that, the T/S trained TDNN model achieves
consitently better performances across all tasks without any additional cost
during inference. For the next experiments, the T/S trained TDNN model is
therefore utilized in the TASE-SVNet framework.

Fig. 3: System performance comparison between the Vanilla TASE-SVNet and the
nontarget sample TASE-SVNet on “vehicle-spk“ test set, DET Curve is split
based on 4 SNR conditions of the test utterances. The SNR condition in dB is:
A=[-inf,3] B=[3,15], C=[15,20],D=[20, inf].

###  4.2 Impact of nontarget sample training for TASE-SVNet

In Sec.2.2, we argue that conventional target speaker enhancement (i.e., the
vanilla TASE-SVNet) models are trained with target samples and do not
specifically address for nontarget speaker suppression. Therefore, a nontarget
sample training constraint is added in TASE-SVNet. A comparison between the
vanilla method and the nontarget sample version is given in Fig.3. As
illustrated in the DET curves, the nontarget sample based TASE-SVNet
consistently outperforms the vanilla version. And the performance gain is from
a lower FAR. We observe a lower FAR when the miss probability is fixed, which
eventually achieves a lower EER. Table 2 presents the overall SV performance
across two test sets, as indicated in the table, the proposed nontarget sample
based TASE-SVNet framework is robust in both highly overlapped condition and
the general condition. The nontarget sample training strategy provides a
+15.2% relative improvement over the vanilla TASE-SVNet system in “vehicle-
spk” data.

Table 2:  SV results with nontarget sample TASE-SVnet. model | overlap-spk | vehicle-spk
---|---|---
TDNN (ours T/S) | 15.91% | 3.93%
TASE-SVNet (vanilla) | 6.83% | 3.72%
TASE-SVNet (nontarget) | 6.02% | 3.15%

###  4.3 Iterative inference

One common issue remains for SV systems when the enrollment utterances are
noisy, which impairs the system robustness in a long run. Since the TASE-SVNet
framework is equipped with a target speaker enhancement module, it is feasible
to speech enhancement for enrollment utterances as well. In fact, with the
enhanced enrollment utterances, we can further perform target speaker
extraction for test utterance iteratively.

The iterative inference that we conducted in our experiments stops after the
second pass, for the consideration of performance and computational cost. The
first pass is just directly inference with raw enrollment utterances. For the
second pass, all 3 raw enrollment utterances are enhanced one by one, then the
3 enhanced enrollment utterances are employed as the anchor information for
processing the test utterance. In our experiments, we find that combining the
two passes will provide consistent performance gain.

###  4.4 Progressive system improvements

We have presented the techniques that result in the improved SV performance.
Table 3 demonstrates the system improvements w.r.t different components. The
best system achieves +63.8 and +30.3 relative improvements over a strong TDNN
baseline, for overlap-spk and vehicle-spk test sets, respectively. Even
compared with the TDNN fusion system, where similar computational consumption
required for inference, the performance gain is considered significant. The
improvement suggests that building the TASE-SVNet framework is essential
generalization ability for SV systems in adverse acoustic environments.

Table 3:  Progressive system improvements with the proposed techniques (EER). model | overlap-spk | vehicle-spk
---|---|---
TDNN (ours T/S) | 15.91% | 3.93%
TASE-SVNet (vanilla) | 6.83% | 3.72%
TASE-SVNet (+nontarget) | 6.02% | 3.15%
TASE-SVNet (+iterative) | 5.89% | 3.01%
TASE-SVNet (+two pass fusion) | 5.75% | 2.74%
TDNN fusion (w/ + w/o T/S ) | 13.72% | 3.65%

##  5 Conclusions

In this study, we presented the TASE-SVNet for robust speaker verification.
The main contributions of this study can be summarised as follows. We proposed
to improve speaker embedding network with T/S training. State-of-the-art SV
benchmark results were achieved with our proposed models. For speech
enhancement front-end to SV models, we achieved balanced target speaker
enhancement and nontarget speaker suppression. Our nontarget sample training
strategy prevents the output of target speaker enhancement network from
biasing to enrolled speaker for nontarget test utterance, which is essential
in reducing FAR for SV. We implemented a flexible TASE-SVNet framework with
multi-stage training. Iterative speaker enhancement is investigated to address
noisy speaker enrollment, which relaxes the limitation of requiring clean
speaker enrollment in conventional SV systems. Finally, significant
performance gain on real SV corpora with noise and overlapped speech justified
the effectiveness of our proposed framework.

## References

  * [1] N. Dehak, P. J. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet,  “Front-end factor analysis for speaker verification,”  IEEE Transactions on Audio, Speech, and Language Processing, vol. 19, no. 4, pp. 788–798, 2010.
  * [2] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur,  “X-vectors: Robust dnn embeddings for speaker recognition,”  in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 5329–5333.
  * [3] C. Zhang and K. Koishida,  “End-to-end text-independent speaker verification with triplet loss on short utterances,”  in Proc. Interspeech 2017, 2017, pp. 1487–1491.
  * [4] C. Li, X. Ma, B. Jiang, X. Li, X. Zhang, X. Liu, Y. Cao, A. Kannan, and Z. Zhu,  “Deep speaker: an end-to-end neural speaker embedding system,”  arXiv preprint arXiv:1705.02304, 2017.
  * [5] C. Zhang, K. Koishida, and J. H.L. Hansen,  “Text-independent speaker verification based on triplet convolutional neural network embeddings,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, no. 9, pp. 1633–1644, 2018.
  * [6] Y. Liu, L. He, and J. Liu,  “Large margin softmax loss for speaker verification,”  in Proc. Interspeech 2019.
  * [7] J. Villalba, N. Chen, D. Snyder, D. Garcia-Romero, A. McCree, G. Sell, J. Borgstrom, F. Richardson, S. Shon, F. Grondin, et al.,  “State-of-the-art speaker recognition for telephone and video speech: the jhu-mit submission for nist sre18,”  in Proc. Interspeech 2019, 2019, pp. 1488–1492.
  * [8] J. Chung, A. Nagrani, E. Coto, W. Xie, M. McLaren, D. A Reynolds, and A. Zisserman,  “Voxsrc 2019: The first voxceleb speaker recognition challenge,”  arXiv preprint arXiv:1912.02522, 2019.
  * [9] G. Heigold, I. Moreno, S. Bengio, and N. Shazeer,  “End-to-end text-dependent speaker verification,”  in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2016, pp. 5115–5119.
  * [10] L. Wan, Q. Wang, A. Papir, and I. L. Moreno,  “Generalized end-to-end loss for speaker verification,”  in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 4879–4883.
  * [11] C. Zhang, F. Bahmaninezhad, S. Ranjan, H. Dubey, W. Xia, and J. H.L Hansen,  “Utd-crss systems for 2018 nist speaker recognition evaluation,”  in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 5776–5780.
  * [12] S. O. Sadjadi, C. Greenberg, E. Singer, D. Reynolds, L. Mason, and J. Hernandez-Cordero,  “The 2018 NIST Speaker Recognition Evaluation,”  in Proc. Interspeech 2019, 2019, pp. 1483–1487.
  * [13] K. A. Lee, V. Hautamäki, T. H. Kinnunen, H. Yamamoto, K. Okabe, V. Vestman, J. Huang, G. Ding, H. Sun, A. Larcher, R. Kumar Das, H. Li, M. Rouvier, P.-M. Bousquet, W. Rao, Q. Wang, C. Zhang, F. Bahmaninezhad, H. Delgado, and M. Todisco,  “I4U Submission to NIST SRE 2018: Leveraging from a Decade of Shared Experiences,”  in Proc. Interspeech 2019, 2019, pp. 1497–1501.
  * [14] S. Shon, H. Tang, and J. Glass,  “VoiceID Loss: Speech Enhancement for Speaker Verification,”  in Proc. Interspeech 2019, 2019, pp. 2888–2892.
  * [15] W. Rao, C. Xu, E. S. Chng, and H. Li,  “Target Speaker Extraction for Multi-Talker Speaker Verification,”  in Proc. Interspeech 2019, 2019, pp. 1273–1277.
  * [16] D Snyder, D Garcia-Romero, G. Sell, A. McCree, D. Povey, and S. Khudanpur,  “Speaker recognition for multi-speaker conversations using x-vectors,”  in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 5796–5800.
  * [17] Q. Wang, H. Muckenhirn, K. Wilson, P. Sridhar, Z. Wu, J. R. Hershey, R. A. Saurous, R. J. Weiss, Y. Jia, and I. L. Moreno,  “VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking,”  in Proc. Interspeech 2019, 2019, pp. 2728–2732.
  * [18] J. Wang, J. Chen, D. Su, L. Chen, M. Yu, Y. Qian, and D. Yu,  “Deep extractor network for target speaker recovery from single channel speech mixtures,”  in Proc. Interspeech 2018.
  * [19] M. Delcroix, K. Zmolikova, K. Kinoshita, A. Ogawa, and T. Nakatani,  “Single channel target speaker extraction and recognition with speaker beam,”  in ICASSP, 2018.
  * [20] X. Ji, M. Yu, C. Zhang, D. Su, T. Yu, X. Liu, and D. Yu,  “Speaker-aware target speaker enhancement by jointly learning with speaker embedding extraction,”  in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 7294–7298.
  * [21] Y. Luo and N. Mesgarani,  “Tasnet: time-domain audio separation network for real-time, single-channel speech separation,”  in the Proceedings of ICASSP, 2018.
  * [22] H. Wang, Y. Wang, Z. Zhou, X. Ji, D. Gong, J. Zhou, Z. Li, and W. Liu,  “Cosface: Large margin cosine loss for deep face recognition,”  in Proceedings of CVPR, 2018, pp. 5265–5274.
  * [23] J. S. Chung, A. Nagrani, and A. Zisserman,  “Voxceleb2: Deep speaker recognition,”  in Proc. Interspeech 2018, 2018, pp. 1086–1090.
  * [24] K. He, X. Zhang, S. Ren, and J. Sun,  “Deep residual learning for image recognition,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [25] J. Li, R. Zhao, Z. Chen, C. Liu, X. Xiao, G. Ye, and Y. Gong,  “Developing far-field speaker system via teacher-student learning,”  in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 5699–5703.
  * [26] J. Jung, H. Heo, H. Shim, and H. Yu,  “Short utterance compensation in speaker verification via cosine-based teacher-student learning of speaker embeddings,”  in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019, pp. 335–341.
  * [27] J. Du, X. Na, X. Liu, and H. Bu,  “Aishell-2: Transforming mandarin asr research into industrial scale,”  arXiv:1808.10583, 2018.
  * [28] H. Zeinali, S. Wang, A. Silnova, P. Matějka, and O. Plchot,  “But system description to voxceleb speaker recognition challenge 2019,”  arXiv preprint arXiv:1910.12592, 2019.


$\displaystyle\qquad=\sum_{i=0}^{n-k-2}p_{1}^{(i)}{\circ}{\mathscr{S}}_{1}^{k+1}\bigl{(}\Delta_{p_{1},p_{2},W,i+k+1}+\rho\,{\mathfrak{a}}_{1}{\circ}{\mathscr{S}}_{1}^{i+k+1}{\mathfrak{f}}{\circ}A_{0}^{i+k}+1\bigr{)}\,p_{2}^{(n-k-2-i)}{\circ}{\mathscr{S}}_{2}^{i+k+2}.$

Moreover in (D.31), (D.32) and (D.33) we express
$p_{1}{\circ}{\mathscr{S}}_{1}^{n}-p_{2}{\circ}{\mathscr{S}}_{2}^{n}$ and
$q_{1}{\circ}{\mathscr{S}}_{1}^{n}-q_{2}{\circ}{\mathscr{S}}_{2}^{n}$
according to (7.88) and, analogously to (D.35), we write

$\partial_{\theta}q_{1}{\circ}{\mathscr{S}}_{1}^{n}-\partial_{\theta}q_{2}{\circ}{\mathscr{S}}_{2}^{n}=\Delta_{\partial_{\theta}q_{1},\partial_{\theta}q_{2},W,n}+\rho\,{\mathfrak{a}}_{5}{\circ}{\mathscr{S}}_{1}^{n}\,{\mathfrak{f}}{\circ}A_{0}^{n},$
(D.37)

with ${\mathfrak{a}}_{5}(\theta)=\partial_{\theta}{\mathfrak{a}}_{2}(\theta)$.

######  Remark D.8.

_Bounds analogous to ( 7.90) hold also for the functions
$\Delta_{\pi_{1},\pi_{2},W,n}$,
$\Delta_{\partial_{\theta}p_{1},\partial_{\theta}p_{2},W,n}$ and
$\Delta_{\partial_{\theta}q_{1},\partial_{\theta}q_{2},W,n}$. Thus, for any_

$\Delta,\Delta^{\prime}\in\\{\Delta_{p_{2},W,n},\Delta_{q_{2},W,n},\;\Delta_{p_{1},p_{2},W,n},\;\Delta_{q_{1},q_{2},W,n},\;\Delta_{\pi_{1},\pi_{2},W,n},\;\Delta_{\partial_{\theta}p_{1},\partial_{\theta}p_{2},W,n},\;\Delta_{\partial_{\theta}q_{1},\partial_{\theta}q_{2},W,n}\\}$

_and for any $k\geq 0$, one has_

$|{\hskip-0.85355pt}|{\hskip-0.85355pt}|\Delta|{\hskip-0.85355pt}|{\hskip-0.85355pt}|_{\alpha_{0},3}^{-}\leq
C\rho,\qquad\langle|\Delta\,W{\circ}A_{0}^{k}|\rangle\leq
C\rho^{2},\qquad\langle|\Delta\Delta^{\prime}|\rangle\leq C\rho^{3}.$

Eventually, by collecting together the expansions from (D.31) to (D.37), we
can write (D.31) as a sum of several contributions in the form

$\partial_{\theta}h_{1}-\partial_{\theta}h_{2}={\mathcal{A}}_{1}+{\mathcal{A}}_{2}+{\mathcal{A}}_{3}+{\mathcal{A}}_{4}+{\mathcal{A}}_{5}+{\mathcal{A}}_{6}+{\mathcal{A}}_{7}+{\mathcal{A}}_{8},$
(D.38)

where the functions ${\mathcal{A}}_{1},\ldots,{\mathcal{A}}_{8}$ are defined
and bounded in the following way.

  * •

The contribution

$\displaystyle{\mathcal{A}}_{1}$
$\displaystyle:=\sum_{n=0}^{\infty}\sum_{k=0}^{n-1}\sum_{i=0}^{k-1}\pi_{1}^{(i)}\,\Delta_{\pi_{1},\pi_{2},W,i}\,\pi_{2}^{(k-i-1)}{\circ}{\mathscr{S}}_{2}^{i+1}\,\partial_{\theta}p_{2}{\circ}{\mathscr{S}}_{2}^{k}\,p_{2}^{(n-k-1)}{\circ}{\mathscr{S}}_{2}^{k+1}\,\Delta_{q_{1},q_{2},W,n}$
$\displaystyle+\sum_{n=0}^{\infty}\sum_{k=0}^{n-1}\pi_{1}^{(k)}\,\Delta_{\partial_{\theta}p_{1},\partial_{\theta}p_{2},W,k}\,p_{2}^{(n-k-1)}{\circ}{\mathscr{S}}_{2}^{k+1}\,\Delta_{q_{1},q_{2},W,n}$
$\displaystyle+\sum_{n=0}^{\infty}\sum_{k=0}^{n-1}\sum_{i=0}^{n-k-2}\pi_{1}^{(k)}\,\partial_{\theta}p_{1}{\circ}{\mathscr{S}}_{1}^{k}\,p_{1}^{(i)}{\circ}{\mathscr{S}}_{1}^{k+1}\,\Delta_{p_{1},p_{2},W,i+k+1}\,p_{2}^{(n-k-2-i)}{\circ}{\mathscr{S}}_{2}^{i+k+2}\,\Delta_{q_{1},q_{2},W,n}$
$\displaystyle+\sum_{n=0}^{\infty}\sum_{k=0}^{n-1}p_{1}^{(k)}\,\Delta_{p_{1},p_{2},W,k}\,p_{2}^{(n-k-1)}{\circ}{\mathscr{S}}_{2}^{k+1}\,\Delta_{\partial_{\theta}q_{1},\partial_{\theta}q_{2},W,n}$

is such that its average is bounded as $|\langle{\mathcal{A}}_{1}\rangle|\leq
C\rho$ because of Remark D.8 and the fact that
$\|\partial_{\theta}p_{2}\|_{\infty}\leq C\rho$.

  * •

The average of

$\displaystyle\hbox{}\\!\\!{\mathcal{A}}_{2}$
$\displaystyle\\!:=\\!\sum_{i=0}^{\infty}\sum_{k=0}^{\infty}\sum_{n=0}^{\infty}\pi_{1}^{(i-1)}{\mathfrak{D}}_{{\mathfrak{a}}_{3},i,0}{\circ}{\mathscr{S}}_{1}^{i}\,W{\circ}A_{0}^{i}\,(\pi_{2}^{(k)}\partial_{\theta}p_{2}{\circ}{\mathscr{S}}_{2}^{k}\,p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{k+1}){\circ}{\mathscr{S}}_{2}^{i+1}\Delta_{q_{1},q_{2},W,n+k+i+2}$
$\displaystyle\qquad\qquad+\sum_{k=0}^{\infty}\sum_{n=0}^{\infty}\pi_{1}^{(k-1)}{\mathfrak{a}}_{3}{\circ}{\mathscr{S}}_{1}^{k-1}\,W{\circ}A_{0}^{k}\,\partial_{\theta}p_{2}{\circ}{\mathscr{S}}_{2}^{k}\,p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{k+1}\Delta_{q_{1},q_{2},W,n+k+1}$
$\displaystyle+\sum_{k=0}^{\infty}\sum_{n=0}^{\infty}\pi_{1}^{(k-1)}{\mathfrak{D}}_{{\mathfrak{a}}_{4},k,0}{\circ}{\mathscr{S}}_{1}^{k}\,W{\circ}A_{0}^{k}\,p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{k+1}\Delta_{q_{1},q_{2},W,n+k+1}$
$\displaystyle\qquad\qquad+\sum_{n=0}^{\infty}\pi_{1}^{(n-1)}{\mathfrak{a}}_{4}{\circ}{\mathscr{S}}_{1}^{n-1}\,W{\circ}A_{0}^{n}\,\Delta_{q_{1},q_{2},W,n}$
$\displaystyle+\sum_{i=0}^{\infty}\sum_{k=0}^{\infty}\sum_{n=0}^{\infty}\pi_{1}^{(k)}\partial_{\theta}p_{1}{\circ}{\mathscr{S}}_{1}^{k}\bigl{(}p_{1}^{(i-1)}\Delta_{{\mathfrak{a}}_{1},i,0}{\circ}{\mathscr{S}}_{1}^{i}W{\circ}A_{0}^{i}\bigr{)}{\circ}{\mathscr{S}}_{1}^{k+1}\,p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{i+k+2}\Delta_{q_{1},q_{2},W,n+i+k+2}$
$\displaystyle\qquad\qquad+\sum_{k=0}^{\infty}\sum_{n=0}^{\infty}\pi_{1}^{(k)}\partial_{\theta}p_{1}{\circ}{\mathscr{S}}_{1}^{k}\bigl{(}p_{1}^{(i-1)}{\mathfrak{a}}_{1}{\circ}{\mathscr{S}}_{1}^{n-1}W{\circ}A_{0}^{n}\bigr{)}{\circ}{\mathscr{S}}_{1}^{k+1}\,\Delta_{q_{1},q_{2},W,n+k+1}$
$\displaystyle+\sum_{k=0}^{\infty}\sum_{n=0}^{\infty}p_{1}^{(k-1)}{\mathfrak{D}}_{{\mathfrak{a}}_{1},k,0}{\circ}{\mathscr{S}}_{1}^{k}\,W{\circ}A_{0}^{k}\,p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{k+1}\,\Delta_{\partial_{\theta}q_{1},\partial_{\theta}q_{2},W,n+k+1}$
$\displaystyle\qquad\qquad+\sum_{n=0}^{\infty}p_{1}^{(n-1)}{\mathfrak{a}}_{1}{\circ}{\mathscr{S}}_{1}^{n-1}\,W{\circ}A_{0}^{n}\,\Delta_{\partial_{\theta}q_{1},\partial_{\theta}q_{2},W,n},$

where we have used Lemma 7.41, is bounded as
$|\langle{\mathcal{A}}_{2}\rangle|\leq C\rho$ because of Remark D.8, the bound
(7.79) and the fact that $\|\partial_{\theta}p_{2}\|_{\infty}\leq C\rho$.

  * •

The contribution

$\displaystyle{\mathcal{A}}_{3}$
$\displaystyle:=\sum_{i=0}^{\infty}\sum_{k=0}^{\infty}\sum_{n=0}^{\infty}\pi_{1}^{(i)}\,\Delta_{\pi_{1},\pi_{2},W,i}\,(\pi_{2}^{(k)}\,\partial_{\theta}p_{2}{\circ}{\mathscr{S}}_{2}^{k}\,p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{k+1}){\circ}{\mathscr{S}}_{2}^{i+1}\,\rho\,{\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{1}^{n+k+i+2}\,{\mathfrak{f}}{\circ}A_{0}^{n+k+i+2}$
$\displaystyle+\sum_{k=0}^{\infty}\sum_{n=0}^{\infty}\pi_{1}^{(k)}\,\Delta_{\partial_{\theta}p_{1},\partial_{\theta}p_{2},W,k}\,p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{k+1}\,\rho\,{\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{1}^{n+k+1}\,{\mathfrak{f}}{\circ}A_{0}^{n+k+1}$
$\displaystyle+\sum_{i=0}^{\infty}\sum_{k=0}^{\infty}\sum_{n=0}^{\infty}\pi_{1}^{(k)}\partial_{\theta}p_{1}{\circ}{\mathscr{S}}_{1}^{k}p_{1}^{(i)}{\circ}{\mathscr{S}}_{1}^{k+1}\Delta_{p_{1},p_{2},W,i+k+1}p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{i+k+2}\rho{\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{1}^{n+k+i+2}{\mathfrak{f}}{\circ}A_{0}^{n+k+i+2}$
$\displaystyle+\sum_{k=0}^{\infty}\sum_{n=0}^{\infty}p_{1}^{(k)}\,\Delta_{p_{1},p_{2},W,k}\,p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{k+1}\,\rho\,{\mathfrak{a}}_{5}{\circ}{\mathscr{S}}_{1}^{n+k+1}\,{\mathfrak{f}}{\circ}A_{0}^{n+k+1}$

is dealt with first writing

$\displaystyle p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{k+1}$
$\displaystyle=p_{2}^{(n)}{\circ}{\mathscr{S}}_{1}^{k+1}+(p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{k+1}-p_{2}^{(n)}{\circ}{\mathscr{S}}_{1}^{k+1}),$
$\displaystyle p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{i+k+2}$
$\displaystyle=p_{2}^{(n)}{\circ}{\mathscr{S}}_{1}^{i+k+2}+(p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{i+k+2}-p_{2}^{(n)}{\circ}{\mathscr{S}}_{1}^{i+k+1}),$

then expanding
$p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{k+1}-p_{2}^{(n)}{\circ}{\mathscr{S}}_{1}^{k+1}$
and
$p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{i+k+2}-p_{2}^{(n)}{\circ}{\mathscr{S}}_{1}^{i+k+2}$
according to (7.66a), so as to apply Lemma 7.37, while using Lemma 7.43 for
the contributions containing the remaining factors
$p_{2}^{(n)}{\circ}{\mathscr{S}}_{1}^{k+1}$ and
$p_{2}^{(n)}{\circ}{\mathscr{S}}_{1}^{k+i+1}$ in order to obtain a factor
either ${\mathfrak{D}}_{1,{\mathfrak{a}}_{2},n}{\circ}{\mathscr{S}}_{1}^{n}$
or ${\mathfrak{D}}_{1,{\mathfrak{a}}_{5},n}{\circ}{\mathscr{S}}_{1}^{n}$, so
that eventually one finds $|\left\langle{\mathcal{A}}_{3}\right\rangle|\leq
C\rho$ because of Remark D.8, the bound (7.79) and the fact that
$\|\partial_{\theta}p_{2}\|_{\infty}\leq C\rho$.

  * •

The contribution

$\displaystyle{\mathcal{A}}_{4}$
$\displaystyle:=\sum_{i=0}^{\infty}\sum_{k=0}^{\infty}\sum_{n=0}^{\infty}\\!\pi_{1}^{(i-1)}{\mathfrak{D}}_{{\mathfrak{a}}_{3},i,0}{\circ}{\mathscr{S}}_{1}^{i}W{\circ}A_{0}^{i}$
$\displaystyle\qquad\qquad\qquad\times(\pi_{2}^{(k)}\partial_{\theta}p_{2}{\circ}{\mathscr{S}}_{2}^{k}p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{k+1}){\circ}{\mathscr{S}}_{2}^{i+1}\rho\,{\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{1}^{n+k+i+2}{\mathfrak{f}}{\circ}A_{0}^{n+k+i+2}\phantom{\sum_{i}}$
$\displaystyle\qquad\qquad+\sum_{k=0}^{\infty}\sum_{n=0}^{\infty}\pi_{1}^{(k-1)}{\mathfrak{a}}_{3}{\circ}{\mathscr{S}}_{1}^{k-1}W{\circ}A_{0}^{k}\partial_{\theta}p_{2}{\circ}{\mathscr{S}}_{2}^{k}p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{k+1}\rho\,{\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{1}^{n+k+1}{\mathfrak{f}}{\circ}A_{0}^{n+k+1}$
$\displaystyle+\sum_{k=0}^{\infty}\sum_{n=0}^{\infty}\pi_{1}^{(k-1)}\,{\mathfrak{D}}_{{\mathfrak{a}}_{4},k,0}{\circ}{\mathscr{S}}_{1}^{k}\,W{\circ}A_{0}^{k}\,p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{k+1}\,\rho\,{\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{1}^{n+k+1}\,{\mathfrak{f}}{\circ}A_{0}^{n+k+1}$
$\displaystyle\qquad\qquad+\sum_{n=0}^{\infty}\pi_{1}^{(n-1)}\,{\mathfrak{a}}_{4}{\circ}{\mathscr{S}}_{1}^{n-1}\,W{\circ}A_{0}^{n}\,\rho\,{\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{1}^{n}\,{\mathfrak{f}}{\circ}A_{0}^{n}$
$\displaystyle+\sum_{i=0}^{\infty}\sum_{k=0}^{\infty}\sum_{n=0}^{\infty}\pi_{1}^{(k)}\partial_{\theta}p_{1}{\circ}{\mathscr{S}}_{1}^{k}p_{1}^{(i-1)}{\circ}{\mathscr{S}}_{1}^{k+1}$
$\displaystyle\qquad\qquad\qquad\times{\mathfrak{D}}_{{\mathfrak{a}}_{1},i,0}{\circ}{\mathscr{S}}_{1}^{i+k+1}\,W{\circ}A_{0}^{i+k+1}p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{i+k+2}\rho{\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{1}^{n+i+k+2}{\mathfrak{f}}{\circ}A_{0}^{n+i+k+2}\phantom{\sum_{i}}$
$\displaystyle\qquad\qquad+\sum_{k=0}^{\infty}\sum_{n=0}^{\infty}\pi_{1}^{(k)}\partial_{\theta}p_{1}{\circ}{\mathscr{S}}_{1}^{k}p_{1}^{(n-1)}{\circ}{\mathscr{S}}_{1}^{k+1}{\mathfrak{a}}_{1}{\circ}{\mathscr{S}}_{1}^{n-1}\,W{\circ}A_{0}^{n}\rho{\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{1}^{n+k+1}{\mathfrak{f}}{\circ}A_{0}^{n+k+1}$
$\displaystyle+\sum_{k=0}^{\infty}\sum_{n=0}^{\infty}p_{1}^{(k-1)}\,{\mathfrak{D}}_{{\mathfrak{a}}_{1},k,0}{\circ}{\mathscr{S}}_{1}^{k}\,W{\circ}A_{0}^{k}\,p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{k+1}\,\rho\,{\mathfrak{a}}_{5}{\circ}{\mathscr{S}}_{1}^{n+k+1}\,{\mathfrak{f}}{\circ}A_{0}^{n+k+1}$
$\displaystyle\qquad\qquad+\sum_{n=0}^{\infty}p_{1}^{(n-1)}\,{\mathfrak{a}}_{1}{\circ}{\mathscr{S}}_{1}^{n-1}\,W{\circ}A_{0}^{n}\,\rho\,{\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{1}^{n}\,{\mathfrak{f}}{\circ}A_{0}^{n}$

where we have used Lemma 7.41, can be studied by decomposing,

${\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{1}^{n+k+1}={\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{2}^{n+k+1}+({\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{1}^{n+k+1}-{\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{2}^{n+k+1}),$

so as to apply Lemma 7.43 to obtain a factor
${\mathfrak{D}}_{2,{\mathfrak{a}}_{2},n+k+1}$ from the terms containing the
first factor and Lemma 7.37 to extract a further function $W$ from the terms
with the difference, which allows us to use the bounds in Remark D.8, and then
proceeding in a similar way for the factors
${\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{1}^{n+k+i+2}$,
${\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{1}^{n+k+i+2}$ and
${\mathfrak{a}}_{5}{\circ}{\mathscr{S}}1_{1}^{n+k+1}$ appearing in the other
contributions, so that eventually, after using also (7.97) for the
contributions where only the sum over $n$ appears, a bound
$|\langle{\mathcal{A}}_{4}\rangle|\leq C\rho$ is obtained.

  * •

The average of

$\displaystyle{\mathcal{A}}_{5}$
$\displaystyle:=\sum_{n=0}^{\infty}\sum_{k=0}^{n-1}\sum_{i=0}^{k-1}\pi_{1}^{(i)}\,\Delta_{\pi_{1},\pi_{2},W,i}\,\pi_{2}^{(k-i-1)}{\circ}{\mathscr{S}}_{2}^{i+1}\,\partial_{\theta}p_{2}{\circ}{\mathscr{S}}_{2}^{k}\,p_{2}^{(n-k-1)}{\circ}{\mathscr{S}}_{2}^{k+1}\,q_{2}{\circ}{\mathscr{S}}_{2}^{n}$
$\displaystyle+\sum_{n=0}^{\infty}\sum_{k=0}^{n-1}\pi_{1}^{(k)}\,\Delta_{\partial_{\theta}p_{1},\partial_{\theta}p_{2},W,k}\,p_{2}^{(n-k-1)}{\circ}{\mathscr{S}}_{2}^{k+1}\,q_{2}{\circ}{\mathscr{S}}_{2}^{n}$
$\displaystyle+\sum_{n=0}^{\infty}\sum_{k=0}^{n-1}\sum_{i=0}^{n-k-2}\pi_{1}^{(k)}\,\partial_{\theta}p_{1}{\circ}{\mathscr{S}}_{1}^{k}p_{1}^{(i)}{\circ}{\mathscr{S}}_{1}^{k+1}\,\Delta_{p_{1},p_{2},W,i+k+1}\,p_{2}^{(n-k-2-i)}{\circ}{\mathscr{S}}_{2}^{i+k+2}\,q_{2}{\circ}{\mathscr{S}}_{2}^{n}$
$\displaystyle+\sum_{n=0}^{\infty}\sum_{k=0}^{n-1}p_{1}^{(k)}\,\Delta_{p_{1},p_{2},W,k}\,p_{2}^{(n-k-1)}{\circ}{\mathscr{S}}_{2}^{k+1}\,\partial_{\theta}q_{2}{\circ}{\mathscr{S}}_{2}^{n}$

after writing

$\displaystyle\pi_{1}^{(i)}=\pi_{2}^{(i)}+(\pi_{1}^{(i)}-\pi_{2}^{(i)}),\qquad\pi_{1}^{(k)}=\pi_{2}^{(k)}+(\pi_{1}^{(k)}-\pi_{2}^{(k)}),\qquad
p_{1}^{(i)}=p_{2}^{(i)}+(p_{1}^{(i)}-p_{2}^{(i)}),$
$\displaystyle\pi_{1}^{(k)}\,\partial_{\theta}p_{1}{\circ}{\mathscr{S}}_{1}^{k}p_{1}^{(i)}{\circ}{\mathscr{S}}_{1}^{k+1}=\pi_{2}^{(k)}\partial_{\theta}p_{2}{\circ}{\mathscr{S}}_{2}^{k}p_{2}^{(i)}{\circ}{\mathscr{S}}_{2}^{k+1}$
$\displaystyle\qquad\qquad+(\pi_{1}^{(k)}\partial_{\theta}p_{1}{\circ}{\mathscr{S}}_{1}^{k}p_{1}^{(i)}{\circ}{\mathscr{S}}_{1}^{k+1}-\pi_{2}^{(k)}\partial_{\theta}p_{2}{\circ}{\mathscr{S}}_{2}^{k}p_{2}^{(i)}{\circ}{\mathscr{S}}_{2}^{k+1}),$

is found to be bounded as $|\langle{\mathcal{A}}_{5}\rangle|\leq C\rho$ by
using Proposition 7.45 for the contributions with the factors involving the
smooth terms $\pi_{2}^{(i)}$, $\pi_{2}^{(k)}$, $p_{2}^{(i)}$ and
$\pi_{2}^{(k)}\partial_{\theta}p_{2}{\circ}{\mathscr{S}}_{2}^{k}p_{2}^{(i)}{\circ}{\mathscr{S}}_{2}^{k+1}$,
and by decomposing the differences $\pi_{1}^{(i)}-\pi_{2}^{(i)}$,
$\pi_{1}^{(k)}-\pi_{2}^{(k)}$ and $p_{1}^{(i)}-p_{2}^{(i)}$, according to
(7.65a), and, similarly, the last difference as

$\displaystyle\pi_{1}^{(k)}\partial_{\theta}p_{1}{\circ}{\mathscr{S}}_{1}^{k}p_{1}^{(i)}{\circ}{\mathscr{S}}_{1}^{k+1}-\pi_{2}^{(k)}\partial_{\theta}p_{2}{\circ}{\mathscr{S}}_{2}^{k}p_{2}^{(i)}{\circ}{\mathscr{S}}_{2}^{k+1}$
$\displaystyle\qquad=\sum_{j=0}^{k-1}\pi_{1}^{(j)}\bigl{(}\pi_{1}{\circ}{\mathscr{S}}_{1}^{j}-\pi_{2}{\circ}{\mathscr{S}}_{2}^{j}\bigr{)}\pi_{2}^{(k-1-j)}{\circ}{\mathscr{S}}_{2}^{j+1}\,\partial_{\theta}p_{2}{\circ}{\mathscr{S}}_{2}^{k}\,p_{2}^{(i)}{\circ}{\mathscr{S}}_{2}^{k+1}$
$\displaystyle\qquad+\pi_{1}^{(k)}\bigl{(}\partial_{\theta}\pi_{1}{\circ}{\mathscr{S}}_{1}^{k}-\partial_{\theta}\pi_{2}{\circ}{\mathscr{S}}_{2}^{k}\bigr{)}\,p_{2}^{(i)}{\circ}{\mathscr{S}}_{2}^{k+1}$
$\displaystyle\qquad+\sum_{j=k+1}^{k+1+i}\pi_{1}^{(k)}\,\partial_{\theta}p_{1}{\circ}{\mathscr{S}}_{1}^{k}p_{1}^{(j-k-1)}{\circ}{\mathscr{S}}_{1}^{k}\bigl{(}p_{1}{\circ}{\mathscr{S}}_{1}^{j}-p_{2}{\circ}{\mathscr{S}}_{2}^{j}\bigr{)}\,p_{2}^{(i-j+k+1)}{\circ}{\mathscr{S}}_{2}^{k+1+j},$

so as to use (7.72) and the bounds in Remark D.8.

  * •

The average of

${\mathcal{A}}_{6}:=\sum_{n=0}^{\infty}\partial_{\theta}p_{2}^{(n)}\,\Delta_{q_{1},q_{2},W,n}+\sum_{n=0}^{\infty}p_{2}^{(n)}\,\Delta_{\partial_{\theta}q_{1},\partial_{\theta}q_{2},W,n}\\\
$

is bounded as $|\langle{\mathcal{A}}_{6}\rangle|\leq C\rho$, as it follows
from the definition (7.86) and Lemma 7.37.

  * •

The contribution

$\displaystyle\hbox{}\\!\\!{\mathcal{A}}_{7}$
$\displaystyle\\!:=\\!\sum_{i=0}^{\infty}\sum_{k=0}^{\infty}\sum_{n=0}^{\infty}\pi_{1}^{(i-1)}{\mathfrak{D}}_{{\mathfrak{a}}_{3},i,0}{\circ}{\mathscr{S}}_{1}^{i}\,W{\circ}A_{0}^{i}\,(\pi_{2}^{(k)}\partial_{\theta}p_{2}{\circ}{\mathscr{S}}_{2}^{k}\,p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{k+1}){\circ}{\mathscr{S}}_{2}^{i+1}\,q_{2}{\circ}{\mathscr{S}}_{2}^{n+k+i+2}$
$\displaystyle\qquad\qquad+\sum_{k=0}^{\infty}\sum_{n=0}^{\infty}\pi_{1}^{(k-1)}{\mathfrak{a}}_{3}{\circ}{\mathscr{S}}_{1}^{k-1}\,W{\circ}A_{0}^{k}\,\partial_{\theta}p_{2}{\circ}{\mathscr{S}}_{2}^{k}\,p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{k+1}\,q_{2}{\circ}{\mathscr{S}}_{2}^{n+k+1}$
$\displaystyle+\sum_{k=0}^{\infty}\sum_{n=0}^{\infty}\pi_{1}^{(k-1)}{\mathfrak{D}}_{{\mathfrak{a}}_{4},k,0}{\circ}{\mathscr{S}}_{1}^{k}\,W{\circ}A_{0}^{k}\,p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{k+1}\,q_{2}{\circ}{\mathscr{S}}_{2}^{n+k+1}$
$\displaystyle\qquad\qquad+\sum_{n=0}^{\infty}\pi_{1}^{(n-1)}{\mathfrak{a}}_{4}{\circ}{\mathscr{S}}_{1}^{n-1}\,W{\circ}A_{0}^{n}\,q_{2}{\circ}{\mathscr{S}}_{2}^{n}$
$\displaystyle+\sum_{i=0}^{\infty}\sum_{k=0}^{\infty}\sum_{n=0}^{\infty}\pi_{1}^{(k)}\partial_{\theta}p_{1}{\circ}{\mathscr{S}}_{1}^{k}\bigl{(}p_{1}^{(i-1)}\Delta_{{\mathfrak{a}}_{1},i,0}{\circ}{\mathscr{S}}_{1}^{i}W{\circ}A_{0}^{i}\bigr{)}{\circ}{\mathscr{S}}_{1}^{k+1}\,p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{i+k+2}q_{2}{\circ}{\mathscr{S}}_{2}^{n+i+k+1}$
$\displaystyle\qquad\qquad+\sum_{k=0}^{\infty}\sum_{n=0}^{\infty}\pi_{1}^{(k)}\partial_{\theta}p_{1}{\circ}{\mathscr{S}}_{1}^{k}\bigl{(}p_{1}^{(i-1)}{\mathfrak{a}}_{1}{\circ}{\mathscr{S}}_{1}^{n-1}W{\circ}A_{0}^{n}\bigr{)}{\circ}{\mathscr{S}}_{1}^{k+1}\,q_{2}{\circ}{\mathscr{S}}_{2}^{n+k+1}$
$\displaystyle+\sum_{k=0}^{\infty}\sum_{n=0}^{\infty}p_{1}^{(k-1)}{\mathfrak{D}}_{{\mathfrak{a}}_{1},k,0}{\circ}{\mathscr{S}}_{1}^{k}\,W{\circ}A_{0}^{k}\,p_{2}^{(n)}{\circ}{\mathscr{S}}_{2}^{k+1}\,\partial_{\theta}q_{2}{\circ}{\mathscr{S}}_{2}^{n+k+1}$
$\displaystyle\qquad\qquad+\sum_{n=0}^{\infty}p_{1}^{(n-1)}{\mathfrak{a}}_{1}{\circ}{\mathscr{S}}_{1}^{n-1}\,W{\circ}A_{0}^{n}\,\partial_{\theta}q_{2}{\circ}{\mathscr{S}}_{2}^{n},$

where we have used Lemma 7.41, can be dealt with as follows. Consider for
instance the sum in the fourth line and, after writing
$\pi_{1}^{(n-1)}=\pi_{2}^{(n-1)}+(\pi_{1}^{(n-1)}-\pi_{2}^{(n-1)})$, expand
$\pi_{1}^{(n-1)}-\pi_{2}^{(n-1)}$ according to (D.34), so as to extract either
a factor $\Delta_{\pi_{1},\pi_{2},W,n-1}$ or a factor
$\rho\,{\mathfrak{f}}{\circ}A_{0}^{n-1}$, while for the remaining term with
$\pi_{2}^{(i-1)}$, after writing
${\mathfrak{a}}_{4}{\circ}{\mathscr{S}}_{1}^{n-1}={\mathfrak{a}}_{4}{\circ}{\mathscr{S}}_{2}^{n-1}+({\mathfrak{a}}_{4}{\circ}{\mathscr{S}}_{1}^{n-1}-{\mathfrak{a}}_{4}{\circ}{\mathscr{S}}_{2}^{n-1})$,
expand
${\mathfrak{a}}_{4}{\circ}{\mathscr{S}}_{1}^{n-1}-{\mathfrak{a}}_{4}{\circ}{\mathscr{S}}_{2}^{n-1}$
according to (7.69), again with the aim of extracting either a sum of terms
containing a function $W$ or a term whose averaged absolute value is of order
$\rho$. In that way we obtain two contributions with a further sum, but also
with a further factor which is of order $\rho$ and, if the average of its
absolute value is not of order $\rho$, contains a further function $W$, so
that the overall average is bounded proportionally to $\rho$. The average of
the remaining term

$\sum_{n=0}^{\infty}\pi_{2}^{(n-1)}{\mathfrak{a}}_{4}{\circ}{\mathscr{S}}_{2}^{n-1}\,W{\circ}A_{0}^{n}\,q_{2}{\circ}{\mathscr{S}}_{2}^{n}$

is controlled through Proposition 7.45, which ensures an overall bound of
order $\rho$. The contributions in the other lines are discussed analogously,
and eventually the bound $|\langle{\mathcal{A}}_{7}\rangle|\leq C\rho$
follows.

  * •

The contribution

${\mathcal{A}}_{8}:=\sum_{n=0}^{\infty}\partial_{\theta}p_{2}^{(n)}\,\rho\,{\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{1}^{n}\,{\mathfrak{f}}{\circ}A_{0}^{n}+\sum_{n=0}^{\infty}p_{2}^{(n-1)}\,\rho\,{\mathfrak{a}}_{5}{\circ}{\mathscr{S}}_{1}^{n}\,{\mathfrak{f}}{\circ}A_{0}^{n}\\\
$

can be studied as follows. First of all, write

${\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{1}^{n}={\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{2}^{n}+({\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{1}^{n}-{\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{2}^{n}),\qquad{\mathfrak{a}}_{5}{\circ}{\mathscr{S}}_{1}^{n}={\mathfrak{a}}_{5}{\circ}{\mathscr{S}}_{2}^{n}+({\mathfrak{a}}_{5}{\circ}{\mathscr{S}}_{1}^{n}-{\mathfrak{a}}_{5}{\circ}{\mathscr{S}}_{2}^{n}),$

so that, when considering the terms
${\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{2}^{n}$ and
${\mathfrak{a}}_{5}{\circ}{\mathscr{S}}_{2}^{n}$, we write
$\rho\,{\mathfrak{f}}{\circ}{\mathcal{A}}_{0}^{n}=W{\circ}A_{0}^{n+1}-W{\circ}A_{0}^{n}$,
in order to apply first Lemma 7.43 to obtain

$\displaystyle\sum_{n=0}^{\infty}\partial_{\theta}p_{2}^{(n-1)}\,{\mathfrak{D}}_{{\mathfrak{a}}_{2},n,0}{\circ}{\mathscr{S}}_{2}^{n}\,W{\circ}A_{0}^{n}+\sum_{n=0}^{\infty}p_{2}^{(n-2)}{\mathfrak{D}}_{{\mathfrak{a}}_{5},n-1,0}{\circ}{\mathscr{S}}_{2}^{n}\,W{\circ}A_{0}^{n}$
(D.39)
$\displaystyle=-{\mathfrak{a}}_{2}\,W\\!+\\!\sum_{n=1}^{\infty}\partial_{\theta}p_{2}^{(n-1)}{\mathfrak{D}}_{{\mathfrak{a}}_{2},n,0}{\circ}{\mathscr{S}}_{2}^{n}W{\circ}A_{0}^{n}-{\mathfrak{a}}_{5}\,W\\!+\\!\sum_{n=1}^{\infty}p_{2}^{(n-2)}{\mathfrak{D}}_{{\mathfrak{a}}_{5},n-1,0}{\circ}{\mathscr{S}}_{2}^{n}W{\circ}A_{0}^{n},$

and hence Proposition 7.45 to bound the average as

$C\rho+C\sum_{n=1}^{\infty}(1-\rho\,\gamma^{\prime})^{n}\bigl{(}(1+\alpha_{0}i)\lambda^{-\alpha_{0}n}+\rho+n\rho+n^{2}\rho^{3}\bigr{)}\bigl{(}|{\hskip-0.85355pt}|{\hskip-0.85355pt}|{\mathfrak{D}}_{{\mathfrak{a}}_{2},n,0}|{\hskip-0.85355pt}|{\hskip-0.85355pt}|_{\alpha_{0},2}^{-}+|{\hskip-0.85355pt}|{\hskip-0.85355pt}|{\mathfrak{D}}_{{\mathfrak{a}}_{5},n,0}|{\hskip-0.85355pt}|{\hskip-0.85355pt}|_{\alpha_{0},2}^{-}\bigr{)}\leq
C\rho,$

while, when considering the terms with the differences, i.e.

$\sum_{n=0}^{\infty}\partial_{\theta}p_{2}^{(n)}\,\rho\,\bigl{(}{\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{1}^{n}-{\mathfrak{a}}_{2}{\circ}{\mathscr{S}}_{2}^{n}\bigr{)}\,{\mathfrak{f}}{\circ}A_{0}^{n}+\sum_{n=0}^{\infty}p_{2}^{(n-1)}\,\rho\,\bigl{(}{\mathfrak{a}}_{5}{\circ}{\mathscr{S}}_{1}^{n}-{\mathfrak{a}}_{5}{\circ}{\mathscr{S}}_{2}^{n}\bigr{)}\,{\mathfrak{f}}{\circ}A_{0}^{n},$
(D.40)

we reason as done when bounding the second contribution in the third line of
(7.98) in the proof of Lemma 7.47, in order to obtain a bound proportional to
$\rho$, which, together with the previous bound, yields that
$|\langle{\mathcal{A}}_{8}\rangle|\leq C\rho$.

To obtain the second bound in (7.104), we consider

$\left(\partial_{\theta}(p_{1}^{(n_{1})}\\!q_{2}{\circ}{\mathscr{S}}_{1}^{n_{1}})-\partial_{\theta}(p_{2}^{(n_{2})}\\!q_{2}{\circ}{\mathscr{S}}_{2}^{n_{2}})\right)\left(\partial_{\theta}(p_{1}^{(n_{1})}\\!q_{2}{\circ}{\mathscr{S}}_{1}^{n_{1}})-\partial_{\theta}(p_{2}^{(n_{2})}\\!q_{2}{\circ}{\mathscr{S}}_{2}^{n_{2}})\right),$

where both factors can be written as in (D.38). Then observe that all the
contributions ${\mathcal{A}}_{1}$ to ${\mathcal{A}}_{7}$ are $O(1)$ in $\rho$
and contains at least a function $W$, while the contribution
${\mathcal{A}}_{8}$ can be dealt with as when discussing the first bound in
(7.104) and written as the sum of the two contributions (D.39) and (D.40),
which both contains a function $W$. This yields that

$\left|\big{\langle}{{\mathcal{A}}_{i}{\mathcal{A}}_{j}}\big{\rangle}\right|\leq
C\rho,\qquad i,j=1,\ldots,8,$

by Theorem 4. Then the second bound in (7.104) follows as well.

###  D.4 Second derivative of the conjugation: proof of Lemma 7.52

Since the proof of Lemma 7.52 follows very closely the same scheme as
Propositions 7.49 and 7.50, we confine ourselves to discuss briefly how to
proceed without entering into the details.

Once more, as in proving Lemma 7.37 and Proposition 7.50 (see Appendices D.2
and D.3, and recall Remark 7.26), also to prove Lemma 7.52 we discuss
explicitly the case $\rho<\rho_{0}$, since the statement trivially holds true
for $\rho\geq\rho_{0}$.

We want to show that

$|\left\langle(\partial_{\theta}^{2}h_{2}(\theta,\cdot)-\partial_{\theta}^{2}\overline{h}(\theta))^{2}\right\rangle|\leq
C\rho,\qquad\left\langle(\partial_{\theta}^{2}h_{1}(\theta,\cdot)-\partial_{\theta}^{2}h_{2}(\theta,\cdot))^{2}\right\rangle\leq
C\rho\,,$ (D.41)

To this end we write

$\displaystyle\partial_{\theta}^{2}(p_{2}^{(n)}\,q_{2}{\circ}{\mathscr{S}}_{2}^{n})-\partial_{\theta}^{2}(\overline{p}^{(n)}\,\overline{q}{\circ}\overline{G}^{n})$
$\displaystyle=\partial_{\theta}^{2}p_{2}^{(n)}\,q_{2}{\circ}{\mathscr{S}}_{2}^{n}-\partial_{\theta}^{2}\overline{p}^{(n)}\,\overline{q}{\circ}\overline{G}^{n}$
$\displaystyle+2\partial_{\theta}p_{2}^{(n)}\,\partial_{\theta}q_{2}{\circ}{\mathscr{S}}_{2}^{n}-2\partial_{\theta}\overline{p}^{(n)}\,\partial_{\theta}\overline{q}{\circ}\overline{G}^{n}+p_{2}^{(n)}\,\partial_{\theta}^{2}q_{2}{\circ}{\mathscr{S}}_{2}^{n}-\overline{p}^{(n)}\,\partial_{\theta}^{2}\overline{q}{\circ}\overline{G}^{n},$
$\displaystyle\partial_{\theta}^{2}(p_{1}^{(n)}\,q_{1}{\circ}{\mathscr{S}}_{1}^{n})-\partial_{\theta}^{2}(p_{2}^{(n)}\,q_{2}{\circ}{\mathscr{S}}_{2}^{n})$
$\displaystyle=\partial_{\theta}^{2}p_{1}^{(n)}\,q_{1}{\circ}{\mathscr{S}}_{1}^{n}-\partial_{\theta}^{2}p_{2}^{(n)}\,q_{2}{\circ}{\mathscr{S}}_{2}^{n}$
$\displaystyle+2\partial_{\theta}p_{1}^{(n)}\,\partial_{\theta}q_{1}{\circ}{\mathscr{S}}_{1}^{n}-2\partial_{\theta}p_{2}^{(n)}\,\partial_{\theta}q_{2}{\circ}{\mathscr{S}}_{2}^{n}+p_{1}^{(n)}\,\partial_{\theta}^{2}q_{1}{\circ}{\mathscr{S}}_{1}^{n}-p_{2}^{(n)}\,\partial_{\theta}^{2}q_{2}{\circ}{\mathscr{S}}_{2}^{n},$

so that the terms in the second and fourth lines can be studied as in
Appendices D.1 and D.3, with $q_{1}$, $q_{2}$ and $\overline{q}$ replaced,
respectively, either with $\partial_{\theta}q_{1}$, $\partial_{\theta}q_{2}$
and $\partial_{\theta}\overline{q}$ or $\partial_{\theta}^{2}q_{1}$,
$\partial_{\theta}^{2}q_{2}$ and $\partial_{\theta}^{2}\overline{q}$.

The terms in the first and third lines can be dealt with by first writing,
according to (B.1b),

$\displaystyle\partial_{\theta}^{2}$ $\displaystyle
p_{2}^{(n)}=\sum_{k=0}^{n-1}(p_{2}\,(\partial_{\theta}G_{2})^{2})^{(k)}\,\partial_{\theta}^{2}p_{2}{\circ}{\mathscr{S}}_{2}^{k}\,p_{2}^{(n-k-1)}{\circ}{\mathscr{S}}_{2}^{k+1}$
(D.42)
$\displaystyle+\sum_{k=0}^{n-1}\sum_{i=0}^{k-1}(p_{2}\,(\partial_{\theta}G_{2})^{2})^{(i)}\,(\partial_{\theta}p_{2}\partial_{\theta}G_{2}){\circ}{\mathscr{S}}_{2}^{i}\,(p_{2}\partial_{\theta}G_{2})^{(k-i-1)}{\circ}{\mathscr{S}}_{2}^{i+1}\partial_{\theta}p_{2}{\circ}{\mathscr{S}}_{2}^{k}\,p_{2}^{(n-k-1)}{\circ}{\mathscr{S}}_{2}^{k+1}$
$\displaystyle+\sum_{k=0}^{n-1}\sum_{i=0}^{k-1}(p_{2}\,(\partial_{\theta}G_{2})^{2})^{(i)}\,(p_{2}\,\partial_{\theta}^{2}G_{2}){\circ}{\mathscr{S}}_{2}^{i}\,(p_{2}\partial_{\theta}G_{2})^{(k-i-1)}{\circ}{\mathscr{S}}_{2}^{i+1}\partial_{\theta}p_{2}{\circ}{\mathscr{S}}_{2}^{k}\,p_{2}^{(n-k-1)}{\circ}{\mathscr{S}}_{2}^{k+1}$
$\displaystyle+\sum_{k=0}^{n-1}\sum_{i=0}^{n-k-2}(p_{2}\,\partial_{\theta}G_{2})^{(k)}\,\partial_{\theta}p_{2}{\circ}{\mathscr{S}}_{2}^{k}\,(p_{2}\partial_{\theta}G_{2})^{(i)}{\circ}{\mathscr{S}}_{2}^{k+1}\partial_{\theta}p_{2}{\circ}{\mathscr{S}}_{2}^{k+1+i}\,p_{2}^{(n-k-i-2)}{\circ}{\mathscr{S}}_{2}^{k+2+i},$

and analogous expressions for $\partial_{\theta}^{2}\overline{p}^{(n)}$ and
$\partial_{\theta}^{2}p_{1}^{(n)}$, with $\overline{\mathscr{S}}$ and
${\mathscr{S}}_{1}$ instead of ${\mathscr{S}}_{2}$, respectively, and then
expanding

$\partial_{\theta}^{2}p_{2}^{(n)}\,q_{2}{\circ}{\mathscr{S}}_{2}^{n}-\partial_{\theta}^{2}\overline{p}^{(n)}\,\overline{q}{\circ}\overline{G}^{n},\qquad\partial_{\theta}^{2}p_{1}^{(n)}\,q_{1}{\circ}{\mathscr{S}}_{1}^{n}-\partial_{\theta}^{2}p_{2}^{(n)}\,q_{2}{\circ}{\mathscr{S}}_{2}^{n},$

by following the same scheme as in Appendices D.1 and D.3, respectively.

In that way, we obtain expressions where, with respect to (D.3) and (D.35), in
addition to the functions $\pi_{1}=p_{1}\partial_{\theta}G_{1}$,
$\pi_{2}=p_{2}\partial_{\theta}G_{2}$ and
$\overline{p}\,\partial_{\theta}\overline{G}$ and, the functions
$p_{1}\partial_{\theta}^{2}G_{1}$,
$\partial_{\theta}p_{1}\partial_{\theta}G_{1}$,
$p_{1}(\partial_{\theta}G_{1})^{2}$, $p_{2}\partial_{\theta}^{2}G_{2}$,
$\partial_{\theta}p_{2}\partial_{\theta}G_{2}$,
$p_{2}(\partial_{\theta}G_{2})^{2}$
$\overline{p}\,\partial_{\theta}^{2}\overline{G}$,
$\partial_{\theta}\overline{p}\,\partial_{\theta}\overline{G}$ and
$\overline{p}\,(\partial_{\theta}\overline{G})^{2}$ also appear. So, together
with the differences (D.32) to (D.34), we have to expand also the differences

$(p_{1}\partial_{\theta}^{2}G_{1})^{(n)}-(p_{2}\partial_{\theta}^{2}G_{2})^{(n)},\quad(\partial_{\theta}p_{1}\partial_{\theta}G_{1})^{(n)}-(\partial_{\theta}p_{2}\partial_{\theta}G_{2})^{(n)},\quad(p_{1}(\partial_{\theta}G_{1})^{2})^{(n)}-(p_{2}(\partial_{\theta}G_{2})^{2})^{(n)},$

when considering
$\partial_{\theta}^{2}p_{1}^{(n)}\,q_{1}{\circ}{\mathscr{S}}_{1}^{n}-\partial_{\theta}^{2}p_{2}^{(n)}\,q_{2}{\circ}{\mathscr{S}}_{2}^{n}$,
while inserting (D.42) and the analogous expression for
$\partial_{\theta}^{2}\overline{p}^{(n)}$ into
$\partial_{\theta}^{2}p_{2}^{(n)}\,q_{2}{\circ}{\mathscr{S}}_{2}^{n}-\partial_{\theta}^{2}\overline{p}^{(n)}\,\overline{q}{\circ}\overline{G}^{n}$
we obtain a sum of contributions with the same structure as (D.3).

Comparing (B.1b) with (B.1a) we observe that, because of the presence of an
extra derivative with respect to $\theta$, a further sum may appear; however,
when this happens, the two derivatives act on two distinct factors, and this
produces a further factor $\rho$, which compensates the factor $1/\rho$
arising from the sum.

##  Appendix E Triple correlations: proof of (4.9)

We start by proving a correlation inequality which somehow generalizes (7.8)
in Proposition 7.6 and allows us to estimate averages of the form

$\left\langle
f_{1}\mu^{(n_{1})}{\circ}A_{0}G{\circ}A_{0}^{n_{1}+1}\mu^{(n_{2})}{\circ}A_{0}^{n_{1}+m+1}f_{2}{\circ}A_{0}^{n_{1}+n_{2}+m+1}\right\rangle,\qquad
G:=\prod_{i=0}^{m-1}g_{i}{\circ}A_{0}^{i},$ (E.1)

where all functions $f_{1},f_{2},g_{0},\ldots,g_{m-1}$ are in
${\mathfrak{B}}_{\alpha}(\mathds{T}^{2},\mathds{R})$, for some $\alpha>0$, and
$\mu^{(n)}$ is defined by (7.15), with $\mu$ as in (7.14). Since we need to
apply the result only when both functions $f_{1}$ and $f_{2}$ have zero
average, we confine ourselves to such a case.

######  Lemma E.1.

Let $f_{1},f_{2},\mu,G\in{\mathfrak{B}}_{\alpha}(\mathds{T}^{2},\mathds{R})$,
with $\alpha\in(0,1]$, be such that $\langle f_{1}\rangle=\langle
f_{2}\rangle=0$ and $\mu$ is of the form (7.14). Then, for every $m\geq 0$,
one has

$\sum_{n_{1},n_{2}=1}^{\infty}\left\langle
f_{1}\mu^{(n_{1})}{\circ}A_{0}G{\circ}A_{0}^{n_{1}+1}\mu^{(n_{2})}{\circ}A_{0}^{n_{1}+m+1}f_{2}{\circ}A_{0}^{n_{1}+n_{2}+m+1}\right\rangle\leq
C\alpha^{-1}\|f_{1}\|_{\alpha}\|f_{2}\|_{\alpha}\|G\|_{\alpha},$ (E.2)

Proof. If we all $N_{1}=n_{1}+m+1$ and $N_{2}=n_{1}+n_{2}+m+2$, and proceed as
in Subsection 7.2.2 by using the first line of (7.6) for $\mu^{(n_{1})}$ and
(7.7) for $\mu^{(n_{2})}$, we write

$\displaystyle\left\langle
f_{1}\mu^{(n_{1})}{\circ}A_{0}G{\circ}A_{0}^{n_{1}+1}\mu^{(n_{2})}{\circ}A_{0}^{N_{1}}f_{2}{\circ}A_{0}^{N_{2}}\right\rangle=\langle\mu\rangle^{n_{1}+n_{2}}\left\langle
f_{1}G{\circ}A_{0}^{n_{1}+1}f_{2}{\circ}A_{0}^{N_{2}}\right\rangle$ (E.3)
$\displaystyle\qquad+\rho\sum_{j_{1}=0}^{n_{1}-1}\langle\mu\rangle^{j_{1}+n_{2}}\left\langle
f_{1}(\tilde{v}\,\mu^{(n_{1}-j_{1}-1)}{\circ}A_{0}){\circ}A_{0}^{j_{1}}G{\circ}A_{0}^{n_{1}+1}f_{2}{\circ}A_{0}^{N_{2}}\right\rangle$
$\displaystyle\qquad+\rho\sum_{j_{2}=0}^{n_{2}-1}\langle\mu\rangle^{n_{1}+j_{2}}\left\langle
f_{1}G{\circ}A_{0}^{n_{1}+1}(\mu^{(n_{2}-j_{2}-1)}\tilde{v}{\circ}A_{0}^{n_{2}-j_{2}-1}){\circ}A_{0}^{N_{1}}f_{2}{\circ}A_{0}^{N_{2}}\right\rangle$
$\displaystyle\qquad+\rho^{2}\sum_{j_{1}=0}^{n_{1}-1}\sum_{j_{2}=0}^{n_{2}-1}\langle\mu\rangle^{j_{1}+j_{2}}$
$\displaystyle\qquad\qquad\qquad\left\langle
f_{1}(\tilde{v}\,\mu^{(n_{1}-j_{1}-1)}{\circ}A_{0}){\circ}A_{0}^{j_{1}}G{\circ}A_{0}^{n_{1}+1}(\mu^{(n_{2}-j_{2}-1)}\tilde{v}{\circ}A_{0}^{n_{2}-j_{2}-1}){\circ}A_{0}^{N_{1}}f_{2}{\circ}A_{0}^{N_{2}}\right\rangle.$

Thus, to bound the double sum in the two last lines of (E.3), if
$j_{1}>j_{2}$, by Proposition 2.3, with

$g_{+}=f_{1},\qquad
g_{-}=(\tilde{v}\mu^{(n_{1}-j_{1}-1)}{\circ}A_{0})G{\circ}A_{0}^{n_{1}+1-j_{1}}(\mu^{(n_{2}-j_{2}-1)}\tilde{v}{\circ}A_{0}^{n_{2}-j_{2}-1}){\circ}A_{0}^{N_{1}-j_{1}}f_{2}{\circ}A_{0}^{N_{2}-j_{1}},$

we get

$\displaystyle\Bigl{|}\left\langle
f_{1}(\tilde{v}\mu^{(n_{1}-j_{1}-1)}{\circ}A_{0}){\circ}A_{0}^{j_{1}}G{\circ}A_{0}^{n_{1}+1}(\mu^{(n_{2}-j_{2}-1)}\tilde{v}{\circ}A_{0}^{n_{2}-j_{2}-1}){\circ}A_{0}^{N_{1}}f_{2}{\circ}A_{0}^{N_{2}}\right\rangle\Bigr{|}$
$\displaystyle\qquad\qquad\qquad\leq C(1-\rho\gamma)^{n_{1}+n_{2}}(1+\alpha
j_{1})\lambda^{-\alpha(j_{1}+j_{2})/2}\|f_{1}\|_{\alpha}\|f_{2}\|_{\alpha}\|G\|_{\alpha},$

where we have bounded $j_{1}>(j_{1}+j_{2})/2$; we arrive at an identical
estimate in the case where $j_{1}\leq j_{2}$, using again Proposition 2.3, now
with

$g_{+}=f_{1}(\tilde{v}\mu^{(n_{1}-j_{1}-1)}{\circ}A_{0}){\circ}A_{0}^{j_{1}}G{\circ}A_{0}^{n_{1}+1}(\mu^{(n_{2}-j_{2}-1)}\tilde{v}{\circ}A_{0}^{n_{2}-j_{2}-1}){\circ}A_{0}^{N_{1}},\qquad
g_{-}=f_{2}.$

Summing over $j_{1}$ and $j_{2}$, we obtain immediately the bound

$C\alpha^{-1}\rho^{2}(1-\rho\gamma)^{n_{1}+n_{2}}\|f_{1}\|_{\alpha}\|f_{2}\|_{\alpha}\|G\|_{\alpha},$

with $C$ independent of $\rho$ and $\alpha$, so that summing also over $n_{1}$
and $n_{2}$ produces an estimate of the form of the r.h.s. of (E.2). Reasoning
in a similar way for the term in the first line of (E.3), with $n_{1}$ and
$n_{2}$ in place of $j_{1}$ and $j_{2}$, we find that

$\sum_{n_{1},n_{2}=0}^{\infty}\Bigl{|}\left\langle
f_{1}G{\circ}A_{0}^{n_{1}+1}f_{2}{\circ}A_{0}^{n_{1}+n_{2}+m+1}\right\rangle\Bigr{|}\leq
C\alpha^{-1}\|f_{1}\|_{\alpha}\|f_{2}\|_{\alpha}\|G\|_{\alpha}.$ (E.4)

Finally, the sums in the second and third lines of (E.3) are dealt with in the
same way by discussing separately the two cases $j_{1}>n_{2}$ and $j_{1}\leq
n_{2}$ and the two cases $j_{2}>n_{1}$ and $j_{2}\leq n_{1}$, respectively, so
as to get once more an estimate of the form of the r.h.s. of (E.2). ∎

######  Remark E.2.

_By looking at the proof of Lemma E.1, one sees that the result still holds if
either a few functions $\mu$ appearing in $\mu^{(n_{1})}$ and in
$\mu^{(n_{2})}$ are replaced with any functions with the same regularity (see
Remark 7.18 for a similar comment) or an arbitrary number of such functions
$\mu$ are replaced with different functions $\mu^{\prime}$ which, besides
sharing the same regularity, still admit the bounds
$\|\mu^{\prime}\|_{\infty}\leq(1-\rho\gamma)$. _

Observe now that

$\displaystyle W_{0}(\psi)-W_{00}(\psi)$
$\displaystyle=\rho^{2}\sum_{n=1}^{\infty}\sum_{i=0}^{n-1}\langle\mu\rangle^{i}\tilde{v}(A_{0}^{-i-1}\psi)\mu^{(-n+i+1)}(A_{0}^{-i-1}\psi)\,b(A_{0}^{-(n+1)}\psi)$
$\displaystyle=\rho^{2}\sum_{n=1}^{\infty}\sum_{i=0}^{n-1}\langle\mu\rangle^{i}\tilde{v}(A_{0}^{-i-1}\psi)\mu^{(n-i-1)}(A_{0}^{-n}\psi)\,b(A_{0}^{-(n+1)}\psi),$

where the second relation in (7.15) has been used to obtain the second line.
By rearranging the sums, we can write,

$\displaystyle\bigl{(}W_{0}-W_{00}\bigr{)}^{2}$
$\displaystyle=\rho^{4}\sum_{i_{1}=0}^{\infty}\sum_{n_{1}=i_{1}+1}^{\infty}\sum_{i_{2}=0}^{\infty}\sum_{n_{2}=i_{2}+1}^{\infty}\langle\mu\rangle^{i_{1}+i_{2}}\tilde{v}{\circ}A_{0}^{-i_{1}-1}$
(E.5)
$\displaystyle\qquad\qquad\times\mu^{(n_{1}-i_{1}-1)}{\circ}A_{0}^{-n_{1}}b{\circ}A_{0}^{-n_{1}-1}\tilde{v}{\circ}A_{0}^{-i_{2}-1}\mu^{(n_{2}-i_{2}-1)}{\circ}A_{0}^{-n_{2}}b{\circ}A_{0}^{-n_{2}-1}.$

To bound the average of (E.5) we order the sums distinguishing the cases
$i_{1}\geq i_{2}$ and $i_{1}<i_{2}$; in the first case we further study
separately the case $n_{2}>n_{1}$ (which implies $i_{2}\leq
i_{1}<n_{1}<n_{2}$) and $n_{2}\leq n_{1}$ (which implies either
$i_{2}<n_{2}\leq i_{1}<n_{1}$ or $i_{2}\leq i_{1}<n_{2}\leq n_{1}$), while the
second case can be reduced to the first one by renaming the indices. Therefore
we obtain

$\displaystyle\bigl{|}\big{\langle}(W_{0}-W_{00})^{2}\big{\rangle}\big{|}$
$\displaystyle\leq
2\rho^{4}\sum_{i_{2}=0}^{\infty}\sum_{i_{1}=i_{2}}^{\infty}\sum_{n_{1}=i_{1}+1}^{\infty}\sum_{n_{2}=n_{1}+1}^{\infty}\langle\mu\rangle^{i_{1}+i_{2}}$
$\displaystyle\quad\times\bigl{|}\big{\langle}\tilde{v}{\circ}A_{0}^{n_{2}-i_{1}}\mu^{(n_{1}-i_{1}-1)}{\circ}A_{0}^{n_{2}-n_{1}+1}b{\circ}A_{0}^{n_{2}-n_{1}}\tilde{v}{\circ}A_{0}^{n_{2}-i_{2}}\mu^{(n_{2}-i_{2}-1)}{\circ}A_{0}\,b\big{\rangle}\bigr{|}$
$\displaystyle+2\rho^{4}\sum_{i_{2}=0}^{\infty}\sum_{n_{2}=i_{2}+1}^{\infty}\sum_{i_{1}=n_{2}}^{\infty}\sum_{n_{1}=i_{1}+1}^{\infty}\langle\mu\rangle^{i_{1}+i_{2}}$
$\displaystyle\quad\times\bigl{|}\big{\langle}\tilde{v}{\circ}A_{0}^{n_{1}-i_{1}}\mu^{(n_{1}-i_{1}-1)}{\circ}A_{0}\,b\,\tilde{v}{\circ}A_{0}^{n_{1}-i_{2}}\mu^{(n_{2}-i_{2}-1)}{\circ}A_{0}^{n_{1}-n_{2}+1}b{\circ}A_{0}^{n_{1}-n_{2}}\big{\rangle}\bigr{|}$
$\displaystyle+2\rho^{4}\sum_{i_{2}=0}^{\infty}\sum_{i_{1}=i_{2}}^{\infty}\sum_{n_{2}=i_{1}+1}^{\infty}\sum_{n_{1}=n_{2}}^{\infty}\langle\mu\rangle^{i_{1}+i_{2}}$
$\displaystyle\quad\times\bigl{|}\big{\langle}\tilde{v}{\circ}A_{0}^{n_{1}-i_{1}}\mu^{(n_{1}-i_{1}-1)}{\circ}A_{0}\,b\,\tilde{v}{\circ}A_{0}^{n_{1}-i_{2}}\mu^{(n_{2}-i_{2}-1)}{\circ}A_{0}^{n_{1}-n_{2}+1}\,b{\circ}A_{0}^{n_{1}-n_{2}}\big{\rangle}\bigr{|},$

that we rewrite, more conveniently,

$\displaystyle\left|\left\langle(W_{0}-W_{00})^{2}\right\rangle\right|$
$\displaystyle\leq
2\rho^{4}\sum_{i=0}^{\infty}\sum_{m_{1},m_{2},m_{3}=0}^{\infty}\langle\mu\rangle^{2i}\bigl{|}\big{\langle}b\,\mu^{(m_{1})}{\circ}A_{0}\,(b\mu){\circ}A_{0}^{m_{1}+1}\bigl{(}\mu^{(m_{2})}{\circ}A_{0}^{m_{1}+2}\bigr{)}^{2}$
(E.6)
$\displaystyle\qquad\times\bigl{(}\tilde{v}\langle\mu\rangle^{m_{3}}\mu^{(m_{3})}\bigr{)}{\circ}A_{0}^{m_{1}+m_{2}+2}\tilde{v}{\circ}A_{0}^{m_{1}+m_{2}+m_{3}+2}\big{\rangle}\bigr{|}$
$\displaystyle+2\rho^{4}\sum_{i=0}^{\infty}\sum_{m_{1},m_{2},m_{3}=0}^{\infty}\langle\mu\rangle^{2i}\bigl{|}\big{\langle}b\,\mu^{(m_{1})}{\circ}A_{0}\,(\langle\mu\rangle\tilde{v}){\circ}A_{0}^{m_{1}+1}\langle\mu\rangle^{m_{2}-1}$
$\displaystyle\qquad\times\bigl{(}b\langle\mu\rangle\bigr{)}{\circ}A_{0}^{m_{1}+m_{2}+1}\bigl{(}\langle\mu\rangle^{m_{3}}\mu^{(m_{3})}\bigr{)}{\circ}A_{0}^{m_{1}+m_{2}+2}\tilde{v}{\circ}A_{0}^{m_{1}+m_{2}+m_{3}+2}\big{\rangle}\bigr{|}$
$\displaystyle+2\rho^{4}\sum_{i=0}^{\infty}\sum_{m_{1},m_{2},m_{3}=0}^{\infty}\langle\mu\rangle^{2i}\bigl{|}\big{\langle}b\,\mu^{(m_{1}-1)}{\circ}A_{0}\,(b\mu){\circ}A_{0}^{m_{1}}\bigl{(}\mu^{(m_{2})}{\circ}A_{0}^{m_{1}+1}\bigr{)}^{2}$
$\displaystyle\qquad\times\bigl{(}\tilde{v}\langle\mu\rangle^{m_{3}}\mu^{(m_{3})}\bigr{)}{\circ}A_{0}^{m_{1}+m_{2}+1}\tilde{v}{\circ}A_{0}^{m_{1}+m_{2}+m_{3}+1}\big{\rangle}\bigr{|}.$

Note that all the averages in (E.6) are of the form (E.1), or of a similar
form as described in Remark E.2, with $f_{1}=b$ and $f_{2}=\tilde{v}$ in all
cases.

The sum over $i$ produces a factor $C/\rho$. Since
$\langle\tilde{v}\rangle=\langle b\rangle=0$, we can apply (E.2) to all three
contributions in the above expression. Finally, after summing over $m_{1}$ and
$m_{3}$, we get

$\left|\left\langle(W_{0}-W_{00})^{2}\right\rangle\right|\leq
C\gamma^{-1}\rho^{3}\sum_{m_{2}=0}^{\infty}\bigl{(}2\|(\mu^{(m_{2})})^{2}\|_{\alpha}+\langle\mu\rangle^{m_{2}}\|\tilde{v}\|_{\alpha}\bigr{)},$

and hence, using (7.3) to deal with the first term and taking into account
that $\alpha=O(1)$ in $\rho$ in the case we are considering, we arrive at the
bound (4.9).

## References

  * [1]
  * [2] R.L. Adler, B. Weiss, _Similarity of automorphisms of the torus_ , Mem. Amer. Math. Soc. Vol. 98, American Mathematical Society, Providence, 1970.
  * [3] A. Arenas, A. Díaz-Guilera, J. Kurths, Y. Moreno, Ch. Zhou, _Synchronization in complex networks_ , Phys. Rep. 469 (2008), no. 3, 93-153.
  * [4] V.I. Arnold, A. Avez, _Ergodic Problems of Classical Mechanics_ , W. A. Benjamin, New York-Amsterdam, 1968.
  * [5] V.I. Arnold, V.N. Kozlov, A.I. Neishtadt, _Dynamical Systems III. Mathematical Aspects of Classical and Celestial Mechanics_ , Springer, Berlin, 1988.
  * [6] A. Ayyer, M. Stenlund, _Exponential decay of correlations for randomly chosen hyperbolic toral automorphisms_ , Chaos 17 (2007), no. 4, 043116, 7 pp.
  * [7] M. Blank, G. Keller, C. Liverani, _Ruelle-Perron-Frobenius spectrum for Anosov maps_ , Nonlinearity 15 (2002), no. 6, 1905-1973.
  * [8] S. Boccaletti, J. Kurths, G. Osipov, D.L. Valladares, C.S. Zhou _The synchronization of chaotic systems_ , Phys. Rep. 366 (2002), no. 1-2, 1-101.
  * [9] N.N. Bogoliubov, Y.A. Mitropolsky, _Asymptotic Methods in the Theory of Non-linear Oscillations_ , Gordon and Breach Science Publishers, New York, 1961.
  * [10] F. Bonetto, P. Falco, A. Giuliani, _Analyticity of the SRB measures of a lattice of coupled Anosov diffeomorphisms of the torus_ , J. Math. Phys. 45 (2004), no. 8, 3282-3309.
  * [11] F. Bonetto, A. Kuipianen, J.L. Lebowitz, _Absolute continuity of projected SRB measures of coupled Arnold cat map lattices_ , Ergodic Theory Dynam. Systems 24 (2004), no. 1, 59-88.
  * [12] J. Bricmont, A. Kuipianen, _Diffusion in energy conserving coupled maps_ , Comm. Math. Phys. 321 (2013), no. 2, 311-369.
  * [13] M. Brin, G. Stuck, _Introduction to Dynamical Systems_ , Cambridge University Press, Cambridge, 2002.
  * [14] A. Carati, L. Galgani, F. Gangemi, R. Gangemi, _Electronic trajectories in atomic physics: the chemical bond in the $H_{2}^{+}$ ion_, Chaos 30 (2020), 063109, 8pp.
  * [15] R. Castorrini, C. Liverani, _Quantitative statistical properties of two-dimensional partially hyperbolic systems_ , Adv. Math. 409 (2022), Paper no. 108625, 122 pp.
  * [16] N.I. Chernov, _Limit theorems and Markov approximations for chaotic dynamical systems._ Probab. Th. Rel. Fields 101 (1995), 321-362.
  * [17] L. De Carlo, G. Gentile, A. Giuliani, _Construction of the Lyapunov spectrum in a chaotic system displaying phase synchronization_ , Math. Phys. Anal. Geom. 19 (2016), no. 2, Art. 10, 23 pp.
  * [18] G.F. Dell’Antonio, _The van Hove limit in classical and quantum mechanics_ , Stochastic processes in quantum theory and statistical physics (Marseille, 1981), 75-110, Lecture Notes in Phys. Vol. 173, Springer, Berlin, 1982.
  * [19] M.F. Demers, _A gentle introduction to anisotropic Banach spaces_ , Chaos, Solitons and Fractals 116 (2018), 29-42.
  * [20] J. De Simoi, C. Liverani, Ch. Poquet, D. Volk, _Fast-slow partially hyperbolic systems versus Freidlin-Wentzell random systems_ , J. Stat. Phys. 166 (2017), no. 3-4, 650-679.
  * [21] J. De Simoi, C. Liverani, _The martingale approach after Varadhan and Dolgopyat_ , Hyperbolic dynamics, fluctuations and large deviations, 311-339, Proc. Sympos. Pure Math. Vol. 89, American Mathematical Society, Providence, RI, 2015.
  * [22] J. De Simoi, C. Liverani, _Statistical properties of mostly contracting fast-slow partially hyperbolic systems_ , Invent. Math. 206 (2016), 147-227.
  * [23] J. De Simoi, C. Liverani, _Limit theorems for fast-slow partially hyperbolic systems_ , Invent. Math. 213 (2018), no. 3, 811-1016.
  * [24] R.L. Devaney, _An Introduction to Chaotic Dynamical Systems_ , Benjamin/Cummings, Menlo Park, 1986.
  * [25] H.A. Dijkstra, _Nonlinear Climate Dynamics_ , Cambridge University Press, Cambridge, 2013.
  * [26] D. Dolgopyat, _Averaging and invariant measures_ , Mosc. Math. J. 5 (2005), no. 3, 537-576.
  * [27] D. Dolgopyat, C. Liverani, _Energy transfer in a fast-slow Hamiltonian system_ , Comm. Math. Phys. 308 (2011), no. 1, 201-225.
  * [28] J. Franks, _Anosov diffeomorphisms.Global Analysis_ , Proc. Sympos. Pure Math. Vol. 14, American Mathematical Society, Providence, R.I., 1970.
  * [29] G. Gallavotti, F. Bonetto, G. Gentile, _Aspects of the Ergodic, Qualitative and Statistical Theory of Motion_ , Springer, Berlin, 2004.
  * [30] G. Gallavotti, G. Gentile, A. Giuliani, _Resonances within chaos_ , Chaos 22 (2012), no.52, 026108, 6 pp.
  * [31] G. Gallavotti, E.G.D. Cohen, _Dynamical ensembles in stationary states_ , J. Statist. Phys. 80 (1995), no. 5-6, 931-970.
  * [32] M. Ghil, V. Lucarini, _The physics of climate variability and climate change_ , Rev. Modern Phys. 92 (2020), no. 3, 035002, 77 pp.
  * [33] J.M. González-Miranda, _Synchronization and Control of Chaos. An Introduction for Scientists and Engineers_ , Imperial College Press, London, 2004.
  * [34] J. Hale, _Ordinary Differential Equations_ , John Wiley & Sons, New York, 1980.
  * [35] Ph.J. Holmes, _Averaging and chaotic motions in forced oscillations_ , SIAM J. Appl. Math. 38 (1980), no. 1, 65-80.
  * [36] Yu. Kifer, _Large deviations and adiabatic transitions for dynamical systems and Markov processes in fully coupled averaging_ , Mem. Amer. Math. Soc. Vol. 201, American Mathematical Society, Providence, 2009.
  * [37] Yu. Kifer, _Averaging and climate models_ , Progr. Probab. 49 (2001), 171-188.
  * [38] N.M. Krylov, N.N. Bogolyubov, _Introduction to Non-Linear Mechanics_ , Princeton University Press, Princeton, NJ, 1943.
  * [39] C.A. Kitio Kwuimy, P. Woafo, _Dynamics, chaos and synchronization of self-sustained electromechanical systems with clamped-free flexible arms_ , Nonlinear Dynam. 53 (2008), 201-213.
  * [40] J.L. Lagrange, _Mécanique Analitique_ , 1788 (edition Albert Blanchard, 1965).
  * [41] C. Liverani, _Transport in partially hyperbolic fast-slow systems_ , Proceedings of the International Congress of Mathematicians, Rio de Janeiro 2018, Vol. III. Invited lectures, 2643-2667, World Sci. Publ., Hackensack, NJ, 2018.
  * [42] V. Lucarini, R. Blender, C. Herbert, F. Ragone, S. Pascale, J. Wouters, _Mathematical and physical ideas for climate science_ , Rev. Geophys. 52 (2014), 809-859.
  * [43] V. Lucarini, F. Ragone, F. Lunkeit, _Predicting climate change using response theory: global averages and spatial patterns_ , J. Stat. Phys. 166 (2017), no. 3-4, 1036-1064.
  * [44] C.D. Murray, S.F. Dermott, _Solar System Dynamics_ , Cambridge University Press, Cambridge, 1999.
  * [45] S.E. Newhouse, _On codimension one Anosov diffeomorphisms_ , Amer. J. Math. 92 (1970), 761-770.
  * [46] L.M. Pecora, Th.L. Carroll, _Synchronization in chaotic systems_ , Phys. Rev. Lett. 64 (1990), no. 8, 821-825.
  * [47] L.M. Pecora, Th.L. Carroll, _Synchronization of chaotic systems_ , Chaos 25 (2015), no. 9, 097611, 12pp.
  * [48] L.M. Pecora, Th.L. Carroll, G.A. Johnson, D.J. Mar, J.F. Heagy, _Fundamentals of synchronization in chaotic systems, concepts, and applications_ , Chaos 7 (1997), no. 4, 520-543.
  * [49] A. Pikovsky, M. Rosenblum, J. Kurths, _Synchronization. A Universal Concept in Nonlinear Sciences_ , Cambridge University Press, Cambridge, 2001.
  * [50] J.H. Sanders, F. Verhulst, J. Murdock, _Averaging Methods in Nonlinear Dynamical Systems_ , Springer, New York, 2007.
  * [51] M.R. Snavely, _Markov partitions for the two-dimensional torus_ , Proc. Amer. Math. Soc. 113 (1991), no. 2, 517-527.
  * [52] S. Soldatenko, A. Bogomolov, A. Ronzhin, _Mathematical modelling of climate change and variability in the context of outdoor ergonomics_ , Mathematics 9 (2021), 2920, 26pp.
  * [53] F. Verhulst, _Nonlinear Differential Equation and Dynamical Systems_ , Springer, Berlin, 1990.
  * [54]