# Meta Learning for End-to-End Low-Resource Speech Recognition

###### Abstract

In this paper, we proposed to apply meta learning approach for low-resource
automatic speech recognition (ASR). We formulated ASR for different languages
as different tasks, and meta-learned the initialization parameters from many
pretraining languages to achieve fast adaptation on unseen target language,
via recently proposed model-agnostic meta learning algorithm (MAML). We
evaluated the proposed approach using six languages as pretraining tasks and
four languages as target tasks. Preliminary results showed that the proposed
method, MetaASR, significantly outperforms the state-of-the-art multitask
pretraining approach on all target languages with different combinations of
pretraining languages. In addition, since MAML’s model-agnostic property, this
paper also opens new research direction of applying meta learning to more
speech-related applications.

Index Terms—  meta-learning, low-resource, speech recognition, language
adaptation, IARPA-BABEL

##  1 Introduction

With the recent advances of deep learning, integrating the main modules of
automatic speech recognition (ASR) such as acoustic model, pronunciation
lexicon, and language model into a single end-to-end model is highly
attractive. Connectionist Temporal Classification (CTC) [1] lends itself on
such end-to-end approaches by introducing an additional blank symbol and
specifically-designed loss function optimizing to generate the correct
character sequences from the speech signal directly, without framewise phoneme
alignment in advance. With many recent results [2, 3, 4], end-to-end deep
learning has created a larger interest in the speech community.

However, to build such an end-to-end ASR system requires a huge amount of
paired speech-transcription data, which is costly. For most languages in the
world, they lack sufficient paired data for training. Pretraining on other
language sources as the initialization, then fine-tuning on target language is
the dominant approach under such low-resource setting, also known as
multilingual transfer learning / pretraining (MultiASR) [5, 6]. The backbone
of MultiASR is a multitask model with shared hidden layers (encoder), and many
language-specific heads. The model structure is designed to learn an encoder
to extract language-independent representations to build a better acoustic
model from many source languages. The success of “language-independent”
features to improve ASR performance compared to monolingual training has been
shown in many recent works [7, 8, 9].

(a) MultiASR

(b) MetaASR

Fig. 1: Illustration: Difference of the learned parameters from MultiASR &
MetaASR. The solid lines represent the learning process of pretraining, either
multitask or meta learning. The dashed lines represent the language-specific
adaptation.
(The figure is modified from [10])

Besides directly training the model with all the source languages, there are
various variants of MultiASR approaches. Language-adversarial training
approaches [11, 12] introduced language-adversarial classification objective
to the shared encoder, negating the gradients backpropagated from the language
classifier to encourage the encoder to extract more language-independent
representations. Hierarchical approaches [13] introduced different granularity
objectives by combining both character and phoneme prediction at different
levels of the model.

In this paper, we provide a novel research direction following up on the idea
of multilingual pretraining – Meta learning. Meta learning, or learning-to-
learn, has recently received considerable interest in the machine learning
community. The goal of meta learning is to solve the problem of “fast
adaptation on unseen data”, which is aligned with our low-resource setting.
With its success in computer vision under the few-shot learning setting [14,
15, 16], there have been some works in language and speech processing, for
instance, language transfer in neural machine translation [10], dialogue
generation [17], and speaker adaptive training [18], but not multilingual
pretraining for speech recognition.

We use model-agnostic meta-learning algorithm (MAML) [19] in this work. As its
name suggestes, MAML can be applied to any network architecture. MAML only
modifies the optimization process following meta learning training scheme. It
does not introduce additional modules like adversarial training or requires
phoneme level annotation (usually through lexicon) such as hierarchical
approaches. We evaluated the effectiveness of the proposed meta learning
algorithm, MetaASR, on the IARPA BABEL dataset [20]. Our experiments reveal
that MetaASR outperforms MultiASR significantly across all target languages.

##  2 Proposed Approach

###  2.1 Multilingual CTC Model

Fig. 2: Multilingual CTC model architecture

We used the model architecture as illustrated in Fig. 2, the shared encoder is
parameterized by $\theta$, and the set of language-specific heads are
parameterized by $\theta_{h,l}$ (the head for $l$-th language). Let the
dataset be $D$, composed of paired data $(X,C)$. Let
$X=x_{1},x_{2},\cdots,x_{T}$ with length $T$ as input feature,
$C=c_{1},c_{2},\cdots,c_{L}$ with length $L$ as target label. $X$ is encoded
into sequence of hidden states through the shared encoder, then fed into the
language-specific head of the corresponding language with softmax activation
to output the prediction sequence
$\hat{C}=\hat{c}_{1},\hat{c}_{2},\cdots,\hat{c}_{L^{\prime}}$ with length
$L^{\prime}$.

CTC Loss. CTC computes the posterior probability as below,

$P(C|X)=\sum_{\pi\in\mathcal{Z}(C)}P(\pi|X)$ (1)

where $\pi$ is the repeated character sequence of $C$ with additional blank
label, and $\mathcal{Z}(C)$ is the set of all possible sequences given $C$.
For each $\pi$, we can approximate the posterior probability as below,

$P(\pi|X)\approx\prod_{i=1}^{L^{\prime}}P(\hat{c_{i}}|X)$ (2)

Take X belonging to the $l$-th language for instance, the loss function of the
model on $D$ is then defined as:

$\mathcal{L}_{D}(\theta,\theta_{h,l})=-\log P(C|X)$ (3)

###  2.2 Meta Learning for Low-Resource ASR

The idea of MAML is to learn initialization parameters from a set of tasks. In
the context of ASR, we can view different languages as different tasks. Given
a set of source tasks $\mathcal{D}=\\{D_{1},D_{2},\cdots,D_{K}\\}$, MAML
learns from $\mathcal{D}$ to obtain good initialization parameters
$\theta^{\star}$ for the shared encoder. $\theta^{\star}$ yields fast task-
specific learning (fine-tuning) on target task $D_{t}$ and obtains
$\theta^{\star}_{t}$ and $\theta^{\star}_{h,t}$ (the parameters obtained after
fine-tuning on $D_{t}$). MAML can be formulated as below,

$\theta^{\star}_{t},\theta^{\star}_{h,t}=\texttt{Learn}(D_{t};\theta^{\star})=\texttt{Learn}(D_{t};\texttt{MetaLearn}(\mathcal{D})).$

The two functions, Learn and MetaLearn, will be described in the following two
subsections.

####  2.2.1 Learn: Language-specific learning

Given any initial parameters $\theta^{0}$ of the shared encoder (either random
initialized or obtained from pretrained model) and the dataset $D_{t}$. The
language-specific learning process is to minimize the CTC loss function
defined in Eq. 3.

$\begin{aligned}
\theta^{\prime},\theta^{\prime}_{h,t}=\texttt{Learn}(D_{t};\theta^{0})&=\arg\,\min_{\theta,\theta_{h,t}}\mathcal{L}_{D_{t}}(\theta,\theta_{h,t})\\\
&=\arg\,\min_{\theta,\theta_{h,t}}\sum_{(X,C)\in D_{t}}-\log
P(C|X)\end{aligned}$

(4)

Table 1: Character error rate ($\mathrm{\char 37\relax}$ CER) w.r.t the pretraining languages set for all 4 target languages’ FLP Model | Vietnamese | Swahili | Tamil | Kurmanji
---|---|---|---|---
| multi | meta | multi | meta | multi | meta | multi | meta
(no-pretrain) | 71.8 | 47.5 | 69.9 | 64.3
Bn Tl Zu | 57.4 | 49.9 | 48.1 | 41.4 | 65.6 | 57.5 | 61.1 | 57.0
Tr Lt Gn | 63.7 | 49.5 | 57.2 | 41.8 | 68.2 | 57.7 | 65.6 | 57.0
Bn Tl Zu Tr Lt Gn | 59.7 | 50.1 | 48.8 | 42.9 | 65.6 | 58.9 | 62.6 | 57.6

The learning process is optimized through gradient descent, the same as
MultiASR.

####  2.2.2 MetaLearn

The initialization parameters found by MAML should not only adapt to one
language well, but for as many languages as possible. To achieve this goal, we
define the meta learning process and the corresponding meta-objective as
follows.

In each meta learning episode, we sample batch of tasks from $\mathcal{D}$,
then sample two subsets from each task $k$ as training and testing set,
denoted as $D_{k}^{tr}$, $D_{k}^{te}$, respectively. First, we use
$D_{k}^{tr}$ to simulate the language-specific learning process to obtain
$\theta^{\prime}_{k}$ and $\theta^{\prime}_{h,k}$.

$\theta_{k}^{\prime},\theta_{h,k}^{\prime}=\texttt{Learn}(D^{tr}_{k};\theta)$
(5)

Then evaluate the effectiveness of the obtained parameters on $D_{k}^{te}$.
The goal of MAML is to find $\theta$, the initialization weights of the shared
encoder for fast adaptation, so the meta-objective is defined as

$\mathcal{L}^{\text{meta}}_{\mathcal{D}}(\theta)=\mathbb{E}_{k\sim\mathcal{D}}\;\mathbb{E}_{D_{k}^{tr},D_{k}^{te}}\Big{[}\mathcal{L}_{D^{te}_{k}}(\theta^{\prime}_{k},\theta^{\prime}_{h,k})\Big{]}$
(6)

Therefore, the meta learning process is to minimize the loss function defined
in Eq. 6.

$\theta^{\star}=\texttt{MetaLearn}(\mathcal{D})=\arg\min_{\theta}\mathcal{L}^{\text{meta}}_{\mathcal{D}}(\theta)$
(7)

We use meta gradient obtained from Eq. 6 to update the model through gradient
descent.

$\theta\leftarrow\theta-\eta^{\prime}\sum_{k}\nabla_{\theta}\mathcal{L}_{D^{te}_{k}}(\theta^{\prime}_{k},\theta^{\prime}_{h,k})$
(8)

$\eta^{\prime}$ is the meta learning rate. And noted that only the shared
encoder is updated via Eq. 8.

MultiASR optimizes the model according to Eq. 4 on all source languages
directly, without considering how learning happens on the unseen language.
Although the parameters found by MultiASR is good for all source languages, it
may not adapt well on the target language. Unlike MultiASR, MetaASR explicitly
integrates the learning process into its framework via simulating language-
specific learning first, then meta-updates the model. Therefore, the
parameters obtained are more suitable to adapt on the unseen language. We
illustrate the concept in Fig. 1, and show it in the experimental results in
Section 4.

##  3 Experiment

In this work, we used data from the IARPA BABEL project [20]. The corpus is
mainly composed of conversational telephone speech (CTS). We selected 6
languages as non-target languages for multilingual pretraining: Bengali (Bn),
Tagalog (Tl), Zulu (Zu), Turkish (Tr), Lithuanian (Lt), Guarani (Gn), and 4
target languages for adaptation: Vietnamese (Vi), Swahili (Sw), Tamil (Ta),
Kurmanji (Ku), and experimented different combinations of non-target languages
for pretraining. Each language has Full Language Pack (FLP) and Limited
Language Pack (LLP, which consists of 10% of FLP).

We followed the recipe provided by Espnet [21] for data preprocessing and
final score evaluation. We used 80-dimensional Mel-filterbank and
3-dimensional pitch features as acoustic features. The size of the sliding
window is 25ms, and the stride is 10ms. We used the shared encoder with a
6-layer VGG extractor with downsampling and a 6-layer bidirectional LSTM
network with 360 cells in each direction as used in the previous work [8].

Meta Learning. For each episode, we used a single gradient step of language-
specific learning with SGD when computing the meta gradient. Noted that in Eq.
8, if we expanded the loss term in the summation via Eq. 4, we would find the
second-order derivative of $\theta$ appear. For computation efficiency, some
previous works [19, 22] showed that we could ignore the second-order term
without affecting the performance too much.
Therefore, we approximated Eq. 8 as follows.

$\theta\leftarrow\theta-\eta^{\prime}\sum_{k}\nabla_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\theta_{k}^{\prime}}}\mathcal{L}_{D^{te}_{k}}(\theta^{\prime}_{k},\theta^{\prime}_{h,k})$
(9)

Also known as First-order MAML (FOMAML).

We multi-lingually pretrained the model for 100K steps for both MultiASR and
MetaASR. When adapting to one certain language, we used the LLP of the other
three languages as validation sets to decide which pretraining step we should
pick. Then we fine-tuned the model 18 epochs for the target language on its
FLP, 20 epochs on its LLP, and evaluated the performance on the test set via
beam search decoding with beam size $20$ and 5-gram language model re-scoring,
as Table 1 and 2 displayed.

##  4 Results

Table 2: Character error rate ($\mathrm{\char 37\relax}$ CER) w.r.t the pretraining languages set for all 4 target languages’ LLP Model | Vietnamese | Swahili | Tamil | Kurmanji
---|---|---|---|---
| multi | meta | multi | meta | multi | meta | multi | meta
(no-pretrain) | 74.7 | 65.0 | 72.4 | 68.9
Bn Tl Zu | 65.0 | 58.1 | 62.6 | 57.5 | 70.4 | 73.7 | 67.6 | 64.6
Tr Lt Gn | 64.9 | 58.0 | 64.1 | 59.6 | 73.7 | 74.7 | 69.7 | 63.0
Bn Tl Zu Tr Lt Gn | 64.1 | 58.7 | 61.9 | 59.6 | 70.0 | 68.2 | 66.7 | 64.1

Performance Comparison of CER on FLP. As presented in Table 1, compared to
monolingual training (that is, without using pretrained parameters as
initialization, denoted as no-pretrain), both MultiASR and MetaASR improved
the ASR performance using different combinations of pretraining languages.
Table 1 clearly shows that the proposed MetaASR significantly outperforms
MultiASR across all target languages. We were also interested in the impact of
the choices of pretraining languages and found that the performance variance
of MetaASR is smaller than MultiASR. It might be due to the fact that MetaASR
focuses more on the learning process rather than fitting on source languages.

$10$$20$$30$$40$$50$$60$$70$$80$$90$$100$$40$$50$$60$Number of pretraining
steps ($\times 1000$)CER ($\mathrm{\char 37\relax}$)MultiASRMetaASRno-pretrain
Fig. 3: Learning curves on Swahili’s LLP
pretrained on Bn, Tl, Zu
$10$$20$$30$$40$$50$$60$$70$$80$$90$$100$$40$$50$$60$Number of pretraining
steps ($\times 1000$)CER ($\mathrm{\char 37\relax}$)MultiASRMetaASRno-pretrain
Fig. 4: Learning curves on Swahili’s LLP
pretrained on Bn, Tl, Zu, Tr, Lt, Gn

Learning Curves. The advantage of MetaASR over MultiASR is clearly shown in
Fig. 3 and 4. Given the pretrained parameters of the specific pretraining
step, we fine-tuned the model for 20 epochs and reported the lowest CER on its
validation set. The above process represented one point of the curve. For
MultiASR, the performance of adaptation saturated in the early stage and
finally degraded. As Fig. 1 illustrates, the training scheme of MultiASR
tended to overfit on pretraining languages, and the learned parameters might
not be suitable for adaptation. From Fig. 3, we can see that in the later
stage of pretraining, using such pretrained weights even yields worse
performance than random initialization. In contrast, for MetaASR, not only the
performance is better than MultiASR during the whole pretraining process, but
it also gradually improves as pretraining continues without degrading. The
adaptation of all languages using different pretraining languages show similar
trends. We only showed the results of Swahili here due to space limitations.

Impact on Training Set Size. In addition to adapting on FLP of the target
languages, we have also fine-tuned on LLP of them, and the result is shown in
Table 2. On Vietnamese, Swahili, and Kurmanji, MetaASR also outperforms
MultiASR. Both of MultiASR and MetaASR improve the performance, but the gap
compared to the no-pretrain model is smaller than fine-tuning on FLP. On
Tamil, weights from pretrained model was even worse than random
initialization. We will evaluate more combinations of target languages and
pretraining languages to investigate the potential of our proposed method in
such ultra low-resource scenario.

##  5 Conclusion

In this paper, we proposed a meta learning approach to multilingual
pretraining for speech recognition. The initial experimental results showed
its potential in multilingual pretraining. In future work, we plan to use more
combinations of languages and corpora to evaluate the effectiveness of MetaASR
extensively. Besides, based on MAML’s model-agnostic property, this approach
can be applied to a wide range of network architectures such as sequence-to-
sequence model, and even different applications beyond speech recognition.

## References

  * [1] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber,  “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,”  in Proceedings of the 23rd international conference on Machine learning. ACM, 2006, pp. 369–376.
  * [2] Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, et al.,  “Deep speech: Scaling up end-to-end speech recognition,”  arXiv preprint arXiv:1412.5567, 2014.
  * [3] Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al.,  “Deep speech 2: End-to-end speech recognition in english and mandarin,”  in International conference on machine learning, 2016, pp. 173–182.
  * [4] Ronan Collobert, Christian Puhrsch, and Gabriel Synnaeve,  “Wav2letter: an end-to-end convnet-based speech recognition system,”  arXiv preprint arXiv:1609.03193, 2016.
  * [5] Ngoc Thang Vu, David Imseng, Daniel Povey, Petr Motlicek, Tanja Schultz, and Hervé Bourlard,  “Multilingual deep neural network based acoustic modeling for rapid language adaptation,”  in 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2014, pp. 7639–7643.
  * [6] Sibo Tong, Philip N Garner, and Hervé Bourlard,  “An investigation of deep neural networks for multilingual speech recognition training and adaptation,”  in Proc. of INTERSPEECH, 2017, number CONF.
  * [7] Jaejin Cho, Murali Karthick Baskar, Ruizhi Li, Matthew Wiesner, Sri Harish Mallidi, Nelson Yalta, Martin Karafiat, Shinji Watanabe, and Takaaki Hori,  “Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling,”  in 2018 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2018, pp. 521–527.
  * [8] Siddharth Dalmia, Ramon Sanabria, Florian Metze, and Alan W Black,  “Sequence-based multi-lingual low resource speech recognition,”  in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 4909–4913.
  * [9] Sibo Tong, Philip N Garner, and Hervé Bourlard,  “Multilingual training and cross-lingual adaptation on ctc-based acoustic model,”  arXiv preprint arXiv:1711.10025, 2017.
  * [10] Jiatao Gu, Yong Wang, Yun Chen, Kyunghyun Cho, and Victor OK Li,  “Meta-learning for low-resource neural machine translation,”  arXiv preprint arXiv:1808.08437, 2018.
  * [11] Jiangyan Yi, Jianhua Tao, Zhengqi Wen, and Ye Bai,  “Adversarial multilingual training for low-resource speech recognition,”  2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4899–4903, 2018.
  * [12] Oliver Adams, Matthew Wiesner, Shinji Watanabe, and David Yarowsky,  “Massively multilingual adversarial speech recognition,”  arXiv preprint arXiv:1904.02210, 2019.
  * [13] Ramon Sanabria and Florian Metze,  “Hierarchical multi task learning with ctc,”  ArXiv, vol. abs/1807.07104, 2018.
  * [14] Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell,  “Meta-learning with latent embedding optimization,”  arXiv preprint arXiv:1807.05960, 2018.
  * [15] Jake Snell, Kevin Swersky, and Richard Zemel,  “Prototypical networks for few-shot learning,”  in Advances in Neural Information Processing Systems, 2017, pp. 4077–4087.
  * [16] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.,  “Matching networks for one shot learning,”  in Advances in neural information processing systems, 2016, pp. 3630–3638.
  * [17] Fei Mi, Minlie Huang, Jiyong Zhang, and Boi Faltings,  “Meta-learning for low-resource natural language generation in task-oriented dialogue systems,”  arXiv preprint arXiv:1905.05644, 2019.
  * [18] Ondřej Klejch, Joachim Fainberg, and Peter Bell,  “Learning to adapt: a meta-learning approach for speaker adaptation,”  arXiv preprint arXiv:1808.10239, 2018.
  * [19] Chelsea Finn, Pieter Abbeel, and Sergey Levine,  “Model-agnostic meta-learning for fast adaptation of deep networks,”  in Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017, pp. 1126–1135.
  * [20] Mark JF Gales, Kate M Knill, Anton Ragni, and Shakti P Rath,  “Speech recognition and keyword spotting for low-resource languages: Babel project research at cued,”  in Spoken Language Technologies for Under-Resourced Languages, 2014\.
  * [21] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, et al.,  “Espnet: End-to-end speech processing toolkit,”  arXiv preprint arXiv:1804.00015, 2018.
  * [22] Alex Nichol and John Schulman,  “Reptile: a scalable metalearning algorithm,”  arXiv preprint arXiv:1803.02999, vol. 2, 2018.


The category $\CRing$ is a well-powered, bicomplete Barr-exact category in which directed colimits commute with finite limits.
Hence, by <ref> and <ref>, there is an order isomorphism of small posets

\[
\cat{N}\cong\llbracket \sOmShr{A}(\JRID{A},\CRing) \rrbracket
\]

where $\cat{N}$ consists of the maps $\JRID{A}\to\ID{A}$ preserving finite infima and arbitrary suprema (just observe that any two congruences on $A$ commute). Because $\CRing$ has no proper subterminal objects, it follows from <ref> that $\CRing$-valued (soft) $\Omega$-sheaves coincide with ordinary (soft) sheaves. Therefore the inclusion

\[
\JRID{A} \hookrightarrow \ID{A},
\]

which preserves finite infima and arbitrary suprema because so does $\RID{A}\hookrightarrow \ID{A}$, induces a soft sheaf representation of the commutative Gelfand ring $A$ over the compact regular frame $\JRID{A}$. This sheaf representation was first obtained by Banaschewski and Vermeulen [5], improving on results of Mulvey [38] and

If the Prime Ideal Theorem is assumed, for any commutative Gelfand ring $A$ the frame $\JRID{A}$ can be identified with $\Omega(\Max{A})$, where $\Max{A}$ is the subspace of $X_A$ consisting of the maximal ideals of $A$ (i.e., the closed points of $X_A$). The space $\Max{A}$ is compact and Hausdorff, and in view of the previous paragraph $A$ is isomorphic to the ring of global sections of a soft sheaf $F\colon \Omega(\Max{A})^\op\to\CRing$ (this sheaf representation can also be derived as a special case of <cit.>, see p. 2178 in op. cit.).

Note that, in contrast with the case of arbitrary commutative rings, for Gelfand rings we get a soft sheaf representation. In fact, the stalk of $F$ at a maximal ideal $\mathfrak{m}\in\Max{A}$ is isomorphic to the quotient ring $A/O_{\mathfrak{m}}$, where the ideal $O_{\mathfrak{m}}$ is defined by

\[
O_{\mathfrak{m}}\coloneqq \{a\in A\mid \exists b\in A\setminus \mathfrak{m} \ \text{such that} \ ab=0\}.
\]

See e.g. <cit.>. The unique maximal ideal of $A$ containing $O_{\mathfrak{m}}$ is $\mathfrak{m}$, hence $A/O_{\mathfrak{m}}$ is a local ring (equivalently, note that $A/O_{\mathfrak{m}}\cong A_{\mathfrak{m}}$). The canonical colimit arrow $F(\Max{A})\to \colim_{x\in U}{F(U)}$ can then be identified with the quotient map $A\to A/O_{\mathfrak{m}}$, which is a regular epimorphism. This shows that every local section over a point of $\Max{A}$ can be extended to a global section. A similar argument shows that every local section defined on a closed subset of $\Max{A}$ can be extended to a global section, i.e. $F$ is soft.

§.§ Acknowledgements

We are grateful to Pino Rosolini for a number of valuable comments on a preliminary version of this work, and to the anonymous referee for further suggestions that allowed us to improve the presentation of our results.

[1]
M. Abbadini.
The dual of compact ordered spaces is a variety.
Theory Appl. Categ., 34(44):1401–1439, 2019.

[2]
M. Abbadini and L. Reggio.
On the axiomatisability of the dual of compact ordered spaces.
Appl. Categ. Structures, 28(6):921–934, 2020.

[3]
B. Banaschewski.
Gelfand and exchange rings: their spectra in pointfree topology.
Arab. J. Sci. Eng. Sect. C Theme Issues, 25(2):3–22, 2000.

[4]
B. Banaschewski.
Ring theory and pointfree topology.
Topology Appl., 137(1):21–37, 2004.

[5]
B. Banaschewski and J. C. C. Vermeulen.
On the sheaf characterization of Gelfand rings.
Quaest. Math., 34(2):137–145, 2011.

[6]
M. Barr, P. A. Grillet, and D. H. van Osdol.
Exact categories and categories of sheaves, volume 236 of Lecture Notes in Mathematics.
Springer-Verlag, Berlin, 1971.

[7]
F. Borceux.
Handbook of categorical algebra. 2, volume 51 of Encyclopedia of Mathematics and its Applications.
Cambridge University Press, Cambridge, 1994.

[8]
D. Bourn.
The denormalized $3\times3$ lemma.
J. Pure Appl. Algebra, 177(2):113–129, 2003.

[9]
W. D. Burgess and X. Caicedo.
Relations and congruences in regular categories.
Notices Amer. Math. Soc., 19(6):A–687, 1972.

[10]
W. D. Burgess and X. Caicedo.
Congruences in regular categories.
Rev. Colombiana Mat., 15(1-2):43–63, 1981.

[11]
A. Carboni, G. M. Kelly, and M. C. Pedicchio.
Some remarks on Mal'tsev and Goursat
Appl. Categ. Structures, 1(4):385–421, 1993.

[12]
S. D. Comer.
Representations by algebras of sections over Boolean spaces.
Pacific J. Math., 38:29–38, 1971.

[13]
W. H. Cornish.
The Chinese remainder theorem and sheaf representations.
Fund. Math., 96(3):177–187, 1977.

[14]
J. Dauns and K. H. Hofmann.
The representation of biregular rings by sheaves.
Math. Z., 91:103–123, 1966.

[15]
J. Dauns and K. H. Hofmann.
Representation of rings by sections, volume 83 of Mem.
Amer. Math. Soc.
American Mathematical Society, Providence, R.I., 1968.

[16]
B. A. Davey.
Sheaf spaces and sheaves of universal algebras.
Math. Z., 134:275–290, 1973.

[17]
J. Dieudonné.
A history of algebraic and differential topology 1900–1960.
Modern Birkhäuser Classics. Birkhäuser Boston, Ltd., Boston,
MA, 2009.
Reprint of the 1989 edition.

[18]
T. H. Fay.
On commuting congruences in regular categories.
Math. Colloq. Univ. Cape Town, 11:13–31, 1977.

[19]
P. J. Freyd and G. M. Kelly.
Categories of continuous functors, I.
J. Pure Appl. Algebra, 2(3):169–191, 1972.

[20]
P. J. Freyd and A. Scedrov.
Categories, allegories, volume 39 of North-Holland
Mathematical Library.
North-Holland Publishing Co., Amsterdam, 1990.

[21]
M. Gehrke and S. J. van Gool.
Sheaves and duality.
J. Pure Appl. Algebra, 222(8):2164–2180, 2018.

[22]
I. Gelfand and M. Naimark.
On the imbedding of normed rings into the ring of operators in
Hilbert space.
Rec. Math. [Mat. Sbornik] N.S., 12(54):197–213, 1943.

[23]
G. Gierz, K. H. Hofmann, K. Keimel, J. D. Lawson, M. Mislove, and D. S. Scott.
Continuous lattices and domains, volume 93 of Encyclopedia
of Mathematics and its Applications.
Cambridge University Press, Cambridge, 2003.

[24]
R. Godement.
Topologie algébrique et théorie des faisceaux.
Actualités Scientifiques et Industrielles, No. 1252 (Publ. Math.
Univ. Strasbourg. No. 13). Hermann, Paris, 1958.

[25]
J. W. Gray.
Sheaves with values in a category.
Topology, 3:1–18, 1965.

[26]
A. Grothendieck.
Éléments de géométrie algébrique. I. Le
langage des schémas.
Inst. Hautes Études Sci. Publ. Math., (4):1–228, 1960.

[27]
D. Hofmann, R. Neves, and P. Nora.
Generating the algebraic theory of $C(X)$: the case of partially
ordered compact spaces.
Theory Appl. Categ., 33:276–295, 2018.

[28]
P. T. Johnstone.
Stone spaces, volume 3 of Cambridge Studies in Advanced
Cambridge University Press, 1986.
Reprint of the 1982 edition.

[29]
A. Jung.
Stably compact spaces and the probabilistic powerspace construction.
Electron. Notes Theor. Comput. Sci., 87:5–20, 2004.

[30]
K. Keimel.
Darstellung von Halbgruppen und universellen Albebren durch
Schnitte in Garben; beireguläre Halbgruppen.
Math. Nachr., 45:81–96, 1970.

[31]
K. Keimel and J. D. Lawson.
Measure extension theorems for ${T}_0$-spaces.
Topology Appl., 149(1):57–83, 2005.

[32]
G. M. Kelly.
Basic concepts of enriched category theory.
Repr. Theory Appl. Categ., (10), 2005.
Reprint of the 1982 original [Cambridge Univ. Press, Cambridge].

[33]
J. D. Lawson.
Stably compact spaces.
Math. Structures Comput. Sci., 21(1):125–169, 2011.

[34]
J. Leray.
Sur la forme des espaces topologiques et sur les points fixes des
J. Math. Pures Appl. (9), 24:95–167, 1945.

[35]
J. Lurie.
Higher topos theory, volume 170 of Annals of Mathematics
Princeton University Press, Princeton, NJ, 2009.

[36]
S. Mac Lane.
Categories for the working mathematician, volume 5 of GTM.
Springer-Verlag, New York, second edition, 1998.

[37]
S. Mac Lane and I. Moerdijk.
Sheaves in geometry and logic.
Universitext. Springer-Verlag, New York, 1994.
Corrected reprint of the 1992 edition.

[38]
C. J. Mulvey.
A generalisation of Gelfand duality.
J. Algebra, 56(2):499–505, 1979.

[39]
L. Nachbin.
Topology and order.
Translated from the Portuguese by Lulu Bechtolsheim. Van Nostrand
Mathematical Studies, No. 4. D. Van Nostrand Co., Inc., Princeton,
N.J.-Toronto, Ont.-London, 1965.

[40]
J. Picado and A. Pultr.
Frames and locales. Topology without points.
Frontiers in Mathematics. Birkhäuser/Springer Basel AG, Basel,

[41]
R. S. Pierce.
Modules over commutative regular rings, volume 70 of Mem.
Amer. Math. Soc.
American Mathematical Society, Providence, R.I., 1967.

[42]
H. A. Priestley.
Representation of distributive lattices by means of ordered Stone
Bull. London Math. Soc., 2:186–190, 1970.

[43]
C. M. Ringel.
The intersection property of amalgamations.
J. Pure Appl. Algebra, 2:341–342, 1972.

[44]
M. H. Stone.
The theory of representations for Boolean algebras.
Trans. Amer. Math. Soc., 40(1):37–111, 1936.

[45]
E. M. Vitale.
Localizations of algebraic categories II.
J. Pure Appl. Algebra, 133(3):317–326, 1998.

[46]
A. Wolf.
Sheaf representations of arithmetical algebras.
In Recent advances in the representation theory of rings and
$C^{\ast} $-algebras by continuous sections (Sem., Tulane Univ.,
New Orleans, La., 1973), number 148 in Mem. Amer. Math. Soc., pages
87–93. 1974.