# Automated Cognate Detection as a Supervised Link Prediction Task with
Cognate Transformer

V.S.D.S. Mahesh Akavarapu  Arnab Bhattacharya
Dept. of Computer Science and Engineering
Indian Institute of Technology Kanpur
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Identification of cognates across related languages is one of the primary
problems in historical linguistics. Automated cognate identification is
helpful for several downstream tasks including identifying sound
correspondences, proto-language reconstruction, phylogenetic classification,
etc. Previous state-of-the-art methods for cognate identification are mostly
based on distributions of phonemes computed across multilingual wordlists and
make little use of the cognacy labels that define links among cognate
clusters. In this paper, we present a transformer-based architecture inspired
by computational biology for the task of automated cognate detection. Beyond a
certain amount of supervision, this method performs better than the existing
methods, and shows steady improvement with further increase in supervision,
thereby proving the efficacy of utilizing the labeled information. We also
demonstrate that accepting multiple sequence alignments as input and having an
end-to-end architecture with link prediction head saves much computation time
while simultaneously yielding superior performance.

Automated Cognate Detection as a Supervised Link Prediction Task with Cognate
Transformer

V.S.D.S. Mahesh Akavarapu and Arnab Bhattacharya Dept. of Computer Science and
Engineering Indian Institute of Technology Kanpur<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Words in genetically related languages with same descendance from a common
ancestral language are termed as _cognates_. For example, Sanskrit _bhava_ and
English _be_ are cognates reconstructed as _*b hewH-_ in ancestral Proto-Indo-
European. Within historical linguistics, assembling potential cognates forms
an essential step in the comparative method to proceed to further stages such
as formulation of sound laws, reconstruction of proto-language, phylogenetic
reconstruction, etc. Campbell (2013). Cognate identification has been
traditionally carried out by tedious manual cross-comparisons of lexica across
several concepts or meanings; this often requires sufficient linguistic
expertise in the languages that are being compared. _Automated cognate
detection_ attempts to alleviate manual labor and, thus, assists a historical
linguist to quickly produce high-quality etymologies required for downstream
tasks.

Over the past decade, several methods for automated cognate detection, mostly
using sequence alignment and other techniques inspired by bioinformatics and
evolutionary biology List et al. (2017), have appeared. The best-performing
methods primarily depend on similarity scores computed from distributions of
phonemes in multilingual wordlists Rama and List (2019) and make little or no
use of the cognacy labels except for a clustering task at the end. In this
paper, we advocate for a supervised learning scenario that utilizes the
labeled information to the fullest. We demonstrate that such a scenario
combined with the representational power of an appropriate deep neural network
architecture can outperform previous methods above a certain amount of
supervision. We also demonstrate that such a model is also capable of transfer
learning. In other words, once trained on some data, it can perform well on
any dataset unseen so far with little additional supervision.

The typical procedure followed by the state-of-the-art methods for this
problem is as follows. In each language family, attested words from all
languages that have the same meaning, i.e., _concept_ , are clustered based on
the pairwise similarity measures computed by the respective procedure. We
propose a different approach where instead of clustering based on pairwise
similarity we directly take input a _multiple sequence alignment (MSA)_ of
words of the same concept and predict linkage via an end-to-end architecture.
This approach proves to be much better in performance and much faster than
clustering from independent pairwise similarity measures.

Many of the algorithms in computational historical linguistics are heavily
drawn or inspired by computational biology. Continuing the trend, we adopt
Cognate Transformer Akavarapu and Bhattacharya (2023), which yielded state-of-
the-art performance in automated phonological reconstruction task, as the base
architecture. Cognate Transformer was adapted from MSA Transformer Rao et al.
(2021), a protein language model that excels in contact predictions. We
additionally append to this architecture layers consisting of triangular
multiplication and triangular attention modules inspired by Alphafold2 Jumper
et al. (2021), the state-of-the-art protein structure predictor, where the
modules roughly capture triangle inequalities among the distances between
amino acid residues. For our task, we applied these modules for capturing
transitivity property among linkages in cognate clusters. We find that the
addition of this particular module has a significant share in the performance
of the overall architecture.

Our key contributions are as follows:

  1. 1.

Firstly, we propose a supervised method for automated cognate detection that
outperforms existing methods with sufficient supervision with likely
improvement on further supervision, thus utilizing the labeled data much more
efficiently than previous models while also demonstrating few-concept (akin to
few-shot) learning.

  2. 2.

Secondly, our method consists of an end-to-end architecture that avoids
independent pairwise computations by accepting MSA as input and directly
predicting cluster linkages, which proves to be more efficient in terms of
both performance and time than a pairwise approach.

  3. 3.

Thirdly, we incorporate into the architecture of Cognate Transformer
additional modules to capture transitivity property among cognate cluster
linkages which has a positive effect on overall performance.

The rest of the paper is organized as follows. Related work is mentioned in
§2. The problem statement is elaborated in §3. The methodology is described in
§4. The details of the experimental setup including the datasets used,
previous baselines, and evaluation measures are described in §5. The results
of experiments and ablation studies along with error analyses and discussions
are given in §6. Finally, the article is concluded in §7.

##  2 Related Work

Computational historical linguistics is a young field that emerged over the
past two decades. Notable works that lead to significant progress in automatic
cognate detection are as follows. Consonant Class Method of Turchin et al.
(2010) deems two words as cognate if the first two consonants fall under the
same consonant class. In Sound-Class-based phonetic alignment (SCA) of List
(2010), pairwise phoneme sequences are aligned and scored for similarity using
sound classes that extend consonant classes. LexStat (List, 2012) aligned and
scored pairwise sequences using language phonemic-specific distributions
combined with SCA-based scores. The pairwise similarities thus obtained are
clustered using UPGMA Sokal and Michener (1958). The previous state-of-the-art
results are attributed to LexStat combined with Infomap clustering List et al.
(2017). Equivalent performance was also reported in Rama (2018) using Chinese
Restaurant Clustering. An expectation-maximization method over pairwise
phonemic distributions is also found to yield similar performance MacSween and
Caines (2020). Information-weighted similarity measure was proposed by Dellert
(2018) which reported a slight increase in evaluation scores over LexStat,
albeit tested only on one dataset.

Supervised algorithms include the Siamese-CNN-based model by Rama (2016) which
performs binary classification on a given pair of words. Jäger et al. (2017)
employ SVM on top of LexStat and Point-wise Mutual Information (PMI) measures
that yield performance similar to that of LexStat-Infomap.

There exist several other works often performing supervised pairwise
classification and incorporating multilingual language models such as those of
Kanojia et al. (2020, 2021) and Nath et al. (2022). Despite brilliantly
employing pre-trained multilingual language models, these cannot be applied
for ancient languages like Ancient Greek, Gothic, etc., or highly low-resource
and endangered languages like those of the Americas where one does find
wordlists of sufficient size but not enough text to pre-train language models
for sake of performing historical linguistic tasks computationally. Another
related task is that of cognate and derivate detection Rani et al. (2023),
which is essentially a word-pair classification task. These tasks have a
slightly different setup than the problem at hand since the clustering step is
not involved.

Cognate Transformer Akavarapu and Bhattacharya (2023) that achieves the best
performance on phonological reconstruction tasks employs a transformer-like
architecture with row-wise and column-wise attentions to efficiently operate
over MSAs. This model was adapted from an evolutionary biological model called
MSA Transformer (Rao et al., 2021) which acts on protein sequences. Vanilla
Transformer architecture was also used in Kim et al. (2023) for proto-language
reconstruction. Although we employ Cognate Transformer, it should be well
noted that the problem we are addressing is that of cognate detection which is
quite different from that of proto-language reconstruction. The aforementioned
transformer-based models address the latter problem.

##  3 Automated Cognate Detection

The automated cognate detection problem statement is described here as
follows. The gold data for a language family $F$, comprising of related
languages $L_{1},L_{2},\ldots\in F$, consists of words over several concepts,
i.e., meanings, say $M_{1},M_{2},\ldots,$ etc. Each word is a sequence of
phonemes. For each concept $M_{m}$, there are words $W_{i}^{m}$ for several
languages $L_{i}$ in that family, where $W_{i}^{m}$ is a word of a language
$L_{i}$ in concept $M_{m}$. Words in each concept are associated with labels
say $c_{i}^{m}\in\mathbb{N}$ which indicate the cluster to which they belong.
A single such cluster of words is called a _cognate set_. We also define links
$l_{ij}^{k}\in\\{0,1\\}$ between languages $L_{i}$ and $L_{j}$ for a concept
$M_{m}$ which indicate if the corresponding words are cognates i.e., have the
same cluster label. In other words,

$\displaystyle l_{ij}^{m}=\begin{cases}1&\text{if }c_{i}^{m}=c_{j}^{m}\\\
0&\text{if }c_{i}^{m}\neq c_{j}^{m}\end{cases}$ (1)

The goal of automated cognate detection is to correctly cluster a given set of
words that mean a single concept in a language family. In a supervised
setting, the aim is to predict the linkages correctly.

For an illustration of the overall problem, consider the Indo-European
language family and the concept of ‘all’. The attested lexica in the member
languages are Sanskrit _sárve_ (Vedic _víśve_), Greek (Ancient) _hólos_ ,
Latin _omnes_ , German _alle_ , English _all_ , Russian _vse_ , Czech _vše_ ,
etc. Among these Vedic _víśve_ , Russian _vse_ , Czech _vše_ form a cluster,
i.e., a cognate set while Sanskrit _sárve_ and Greek _hóla_ form another
cognate set. Similarly, English and German word forms form another cognate
set. The input data is present in IPA transcription format. Roman
transliterated forms are presented here only for demonstration.

##  4 Methodology

Figure 1: Architecture of Cognate Transformer with Triangle Multiplication and
Attention modules

The overall workflow is described as follows. Given some words from different
languages for a concept in a language family, the words are first aligned
(§4.1), then converted into tokens and passed into the cognate transformer
(§4.2), whose outputs are converted into pairwise (along language axis)
representations by outer product mean module (§4.3), which are then passed
into the layers of pairwise module (§4.4) whose outputs are classified into
two labels 0 or 1 indicating the pairwise linkage among the languages (§4.5).
Since the linkage information is known in the form of cognacy labels, the
architecture described can be thus trained end-to-end. The overall
architecture is illustrated in Figure 1.

###  4.1 MSA input

Skt. | - | s |  @ | r |  V | e | -
---|---|---|---|---|---|---|---
Gr. | - | h | o | l | - | o | s
Lat. | - | - |  O | m | n |  E | s
En. | - | - |  O: | l | - | - | -
Ger. | - | - |  a | l | - |  @ | -
Rus. | f | sj | - | - | - | e | -
Cze. | f |  S | - | - | - |  E | -
Table 1: Example of a Multiple Sequence Alignment (MSA) of phoneme sequences

The input words for a concept are aligned together using the SCA method (List,
2010), where initial pairwise alignments are carried out by using Needleman
and Wunsch (1970) with weights based on sound classes which are further
progressively merged guided by a UPGMA Sokal and Michener (1958) tree based on
pairwise distances. Progressive alignment is a widely used method for multiple
sequence alignment which forms the basis of popular programs such as ClustalW
(Thompson et al., 2003). We use the implementation available in LingPy (List
and Forkel, 2021).

The resultant MSA, present in IPA (see Table 1), is converted into ASJP (Brown
et al., 2008) representation, a phonemic representation scheme that compacts
IPA symbols resulting in lesser vocabulary size. Note that each token in an
MSA need not be a single phoneme. In the SCA method, consecutive vowels are
combined into one token. Language information is passed as the initial token
in each row following Akavarapu and Bhattacharya (2023). The resultant tokens
are mapped to their respective token numbers and padded according to the
batch. Thus, a typical input to Cognate Transformer lies in
$\mathbb{N}^{b\times r\times c}$ where $b$ is the batch size, $r$ is the
maximum number of rows, i.e., the number of words for that batch, and $c$ is
the maximum sequence length in the batch. From here, we ignore the batch
dimension and simply consider the input to lie in $\mathbb{N}^{r\times c}$

###  4.2 Cognate Transformer

Cognate Transformer (Akavarapu and Bhattacharya, 2023) handles two-dimensional
input employing separate row and column attentions (see Figure 1). The input
and output have the same dimensions. In other words,

$\displaystyle\textrm{CogTran}:\mathbb{N}^{r\times
c}\rightarrow\mathbb{R}^{r\times c\times d}$ (2)

where $d$ is the hidden size. The outputs of CogTran are converted into
pairwise format by the outer product mean module.

###  4.3 Outer Product Mean

In this module, as the name suggests, the outer product is computed along each
column, across all rows, and then the mean of outer products is computed
across all columns. The transformation to the dimensions are

$\displaystyle\textrm{OutProdMean}:\mathbb{R}^{r\times c\times
d}\rightarrow\mathbb{R}^{r\times r\times d}$ (3)

The intuition is that the multiplication of a pair of transformed embeddings
of two tokens in a single position (i.e., column) of two different words
(i.e., rows) should roughly indicate the similarity or distance between the
two words in that particular position. The mean operation should produce a
mean of such similarities across all positions for a pair of words. Hence, the
final matrix would represent a pairwise similarity matrix across the words in
an MSA.

This module is identical to the one in AlphaFold2 (Jumper et al., 2021) except
that the role of rows and columns is interchanged. In other words, in
AlphaFold2, the outputs are pairwise representations of amino-acid-residues
(along columns) while in our case the outputs are pairwise representations of
words (along rows).

###  4.4 Pairwise Module

The pairwise module in AlphaFold2, which consists of triangle multiplication
and triangle attention updates via both incoming and outgoing edges, is a
differentiable workflow to capture triangle inequalities that the distances
between amino acid residues should satisfy (Jumper et al., 2021). In our case,
we demand that the link predictions (see §3 for definition) satisfy the
transitivity property which can be translated into the following condition

$\displaystyle l_{ik}^{m}\cdot l_{jk}^{m}=l_{ij}^{m}\textrm{ if
}l_{ik}^{m}+l_{jk}^{m}\neq 0$ (4)

for languages $L_{i}$, $L_{j}$ and $L_{k}$ in a family $F$ for concept
$M_{m}$. The triangle multiplication update follows a similar equation but
without constraint and, hence, is apt for the problem at hand. Combining the
updates for both incoming ($i\rightarrow j$) and outgoing edges ($j\rightarrow
i$) ensures the symmetry required for pairwise similarities. The pairwise
module does not alter the dimensions of the input, i.e.,

$\displaystyle\textrm{PairwiseMod}:\mathbb{R}^{r\times r\times
d}\rightarrow\mathbb{R}^{r\times r\times d}$ (5)

In AlphaFold2, this module along with the MSA module is embedded within the
Evoformer module. As of now, it is unclear if such embedding would improve the
performance. For this problem, we stack the modules as illustrated in Figure 1
for the sake of simplicity and easier ablation tests.

###  4.5 Classifier and Clustering

The outputs of the pairwise module are passed through a linear layer
outputting values for two classes $\\{0,1\\}$ indicating linkage. Hence, the
classifier layer’s transformation is summarized as:

$\displaystyle\textrm{Classifier}:\mathbb{R}^{r\times r\times
d}\rightarrow\mathbb{R}^{r\times r\times 2}$ (6)

The softmax probabilities of the outputs $p^{m}_{ij}$ for $P(l_{ij}^{m}=1)$
determine the linkage probabilities. During training, the network is trained
with cross-entropy loss. During testing, UPGMA is run for each concept $M_{m}$
with pairwise similarities as $p^{m}_{ij}$ flat clustered at a threshold of
0.6, which is determined by a small (5%) held out validation set during
training, to obtain the required clusters.

##  5 Experimental Setup

In this section, the details of the experiments including datasets,
implementation, evaluation metrics, baseline models, etc. are described.

###  5.1 Datasets

Family | Meanings | Languages | Cognates | Words
---|---|---|---|---
Training data
AN | 210 | 20 | 2864 | 4358
BAI | 110 | 9 | 285 | 1028
CHN | 140 | 15 | 1189 | 2789
IE | 207 | 20 | 1777 | 4393
JAP | 200 | 10 | 460 | 1986
OU | 110 | 21 | 242 | 2055
Total |  |  | 6817 | 16609
Test data
BAH | 200 | 24 | 1055 | 4546
CHN | 180 | 18 | 1231 | 3653
HU | 139 | 14 | 855 | 1668
ROM | 110 | 43 | 465 | 4853
TUJ | 109 | 5 | 179 | 513
URA | 173 | 7 | 870 | 1401
AN | 210 | 45 | 3804 | 9267
AA | 200 | 58 | 1872 | 11827
IE | 208 | 42 | 2157 | 9854
PN | 183 | 67 | 6634 | 12691
ST | 110 | 64 | 1402 | 7074
Total |  |  | 19136 | 67347

Table 2: Details of the datasets as obtained from Rama and List (2019)
indicating the number of concepts, languages, cognate sets, and words.

The dataset for both training and testing along with the train-test split is
taken from Rama and List (2019) which was collected from various publicly
available sources. It consists of data from various language families, namely,
Austro-Asiatic (AA), Austronesian (AN), Bai (BAI), Bahnaric (BAH), Chinese
(CHN), Huon (HU), Indo-European (IE), Japanese (JAP), Ob-Ugrian (OU), Pama-
Nyungan (PN), Romance (ROM), Sino-Tibetan (ST), Tujia (TUJ), and Uralic (URA).
The statistics of the data are provided in Table 2.

As is evident from the table, the original training size is disproportionately
much lesser than the test size. Many language families in tests such as AA,
PN, HU, etc. are completely absent in the training set. We also test the model
on increased supervision by augmenting the training data with some proportion
of test data. In particular, apart from the original train-test split, we also
test by including 12.5% and 50% additional test concepts, i.e., approximately
20 and 100 additional test concepts respectively per language family. For both
the proportions, data is divided into 5 random splits. Hence, the results
reported for 12.5%+ and 50%+ proportions are five-fold cross-validated.

###  5.2 Implementation Details

The architecture we deploy has two Cognate Transformer layers and two layers
of pairwise module (see Figure 1). In the Cognate Transformer, the number of
attention heads is also 2. The maximum vocabulary size of the tokenizer is set
to 768, while the maximum words and sequence length in an MSA are both set to
256. Both hidden size $d$ and intermediate size, wherever there is projection,
are 128. This amounts to a network of about a million parameters. The network
was trained with a batch size $b$ of 4 and tested with that of 2. Low batch
size is due to the limitation of GPU memory (10 GB in our case) since MSAs
combined in both the dimensions and the pairwise representation layers easily
blow up the memory. The training was performed using AdamW optimizer
(Loshchilov and Hutter, 2017) with learning rate 1e-3 as implemented by
HuggingFace (Wolf et al., 2020). During testing, the pairwise softmax
probabilities (similarities with 1 being the most similar) are used for flat
clustering using UPGMA at a threshold of 0.6, arrived through held-out
validation from the train set (5%). The total time taken for one run of train
and test is less than 15 minutes on GPU. This is much smaller when compared to
the models that operate on a pair of words at a time instead of on an MSA. The
code is made publicly available111https://github.com/mahesh-ak/CogDetect.

###  5.3 Evaluation Metrics

The outputs of the entire algorithm are clusters (see §4.5), i.e., every word
gets a cluster label assigned which is to be compared with the gold cluster
labels. The usual F1 score is not a proper measure since the assigned cluster
label is not important; rather, members of the same cognate set must get
assigned to the same cluster. Hence, the B-Cubed F1 score (Amigó et al., 2009)
is the appropriate evaluation measure; it has been employed in the previous
works for this problem as well. We use the implementation available in LingPy
(List and Forkel, 2021).

###  5.4 Baseline Models

####  5.4.1 LexStat-Infomap

We label the model defined so far as _CogTran2_. The foremost base model with
which we compare the performance of CogTran2 is LexStat-Infomap (List et al.,
2017) whose performance is more or less the state-of-the-art as discussed in
§2. The original model employs 10,000 permutations between each language pair
in a family to obtain language-specific distributions. Hence, this method
requires significant test data to be known beforehand to preprocess. We call
this model as _LexStInf10K_. This method takes more than 2 hours on a CPU to
obtain results on one test set. Hence we also report for the model that has
the number of runs as 1000, which we label as _LexStInf1K_ which takes less
than 15 CPU minutes. These are imported from LingPy List and Forkel (2021).

####  5.4.2 SCA

We also test on SCA-based model (List, 2010) where a pairwise distance depends
on sound classes and alignment. Since it does not depend on any sort of
computation such as language-specific distributions, this is the fastest
method and, unlike LexStat-Infomap, can be run on any unseen data. We label
this as _SCA_. For both LexStat-Informap and SCA, we use the flat cluster
thresholds 0.6 and 0.45 respectively, as mentioned in List et al. (2017),
since the training data is the same.

####  5.4.3 SVM

We also compare with the SVM-based model Jäger et al. (2017), labeled as _SVM_
, and the Siamese-CNN-based model Rama (2016) as these are supervised models.
This model uses LexStat score and PMI scores as primary features and, hence,
takes a long time to preprocess data, i.e., about 6 hours when each split is
processed in parallel on a CPU when the number of permutations runs is 1000
(for LexStat similarity). Since this is a relatively much longer time, we do
not increase the number of runs any further. SVM is trained on pairwise binary
classification tasks which give pairwise cognacy probabilities for further
clustering. We use publicly available code for this
model222https://github.com/evolaemp/svmcc.

####  5.4.4 Siamese CNN

From the proposed Siamese CNN architectures (Rama, 2016), we use the model
mentioned as charCNN with language features that show good overall performance
among the models that are proposed therein. We label this model as _CharCNN_.
The network is trained on pairwise supervised binary classification tasks. The
pairwise probabilities of the network are used further for clustering (UPGMA).
CharCNN is implemented from scratch in PyTorch closely following the
TensorFlow code that was made publicly available by the author
333https://github.com/PhyloStar/SiameseConvNet/.

####  5.4.5 Ablation Models

We also test on ablations, namely, without pairwise module which we call
simply _CogTran_.

We also test by increasing the number of hidden layers to 4 of this same model
which we label as _CogTranL4_.

Further, we test on a variant that does not use input MSA but rather only an
alignment of a pair of words at a time akin to all other previous models but
unlike CogTran2. In this model, pairwise binary classification is performed
which gives probability scores for each pair of words in a concept. Further,
clustering (UPGMA) is performed using these pairwise scores. To be more
specific, the input is an aligned word pair and the resultant output
embeddings are summed before the binary classifier, while in Siamese-CNN Rama
(2016), the absolute differences of embedding pairs are considered before the
classifier layer. We note that summing should not be different since the
network can always adjust the signs within embeddings themselves. We call this
model _CogTranPair_. For these models, the link prediction is not part of the
end-to-end architecture, unlike for the model we propose. As a result, the
models are run separately on all possible pairs of words in a concept.

Data+% | Method | Language Families | Mean
---|---|---|---
BAH | CHN | HU | ROM | TUJ | URA | AN | AA | IE | PN | ST
+0% | SCA | .864 | .793 | .857 | .873 | .894 | .909 | .775 | .760 | .806 | .709 | .561 | .800
LexStInf10K | .894 | .857 | .883 | .910 | .899 | .913 | .840 | .773 | .826 | .845 | .592 | .839
LexStInf1K | .894 | .855 | .873 | .912 | .900 | .907 | .839 | .759 | .818 | .820 | .595 | .834
CharCNN | .759 | .837 | .876 | .666 | .845 | .886 | .698 | .722 | .725 | .784 | .473 | .752
SVM | .865 | .845 | .860 | .927 | .899 | .913 | .845 | .734 | .828 | .782 | .593 | .826
CogTran2 | .854 | .864 | .857 | .907 | .893 | .899 | .786 | .756 | .845 | .797 | .572 | .821
| CharCNN |  | .830
---
(.010)
| .847
---
(.006)
| .873
---
(.010)
| .896
---
(.007)
| .892
---
(.015)
| .895
---
(.006)
| .777
---
(.007)
| .752
---
(.007)
| .825
---
(.008)
| .786
---
(.002)
| .535
---
(.025)
| .810
---
(.002)
+12.5% | SVM |  | .878
---
(.006)
| .836
---
(.006)
| .882
---
(.010)
| .934
---
(.007)
| .919
---
(.005)
| .914
---
(.006)
| .840
---
(.003)
| .767
---
(.012)
| .831
---
(.004)
| .765
---
(.012)
| .582
---
(.012)
| .832
---
(.002)
| CogTran2 |  | .884
---
(.004)
| .867
---
(.005)
| .890
---
(.011)
| .907
---
(.013)
| .913
---
(.015)
| .904
---
(.006)
| .810
---
(.005)
| .813
---
(.003)
| .851
---
(.003)
| .804
---
(.007)
| .607
---
(.020)
| .841
---
(.002)
| CharCNN |  | .876
---
(.011)
| .854
---
(.007)
| .880
---
(.005)
| .914
---
(.012)
| .899
---
(.018)
| .904
---
(.012)
| .795
---
(.008)
| .784
---
(.005)
| .840
---
(.006)
| .785
---
(.011)
| .563
---
(.011)
| .827
---
(.003)
+50% | SVM |  | .881
---
(.010)
| .838
---
(.009)
| .889
---
(.014)
| .935
---
(.010)
| .927
---
(.012)
| .914
---
(.009)
| .840
---
(.010)
| .779
---
(.009)
| .828
---
(.007)
| .775
---
(.009)
| .577
---
(.019)
| .835
---
(.002)
| CogTran2 |  | .893
---
(.011)
| .878
---
(.005)
| .901
---
(.006)
| .921
---
(.015)
| .916
---
(.009)
| .914
---
(.007)
| .823
---
(.006)
| .832
---
(.008)
| .853
---
(.004)
| .812
---
(.006)
| .644
---
(.015)
| .853
---
(.002)

Table 3: Results (B-Cubed F-scores) with language families indicated across
columns along with standard deviations in parentheses for cross-validated
values. The best scores within a specific train-test split are shown in bold.

##  6 Results

The results are summarized in Table 3. The first column indicates the
additional proportion of concepts that is moved from test data to training
data. Thus, it roughly indicates the amount of increased supervision. The
second column indicates the various methods discussed in §5.4 compared against
the proposed model, _CogTran2_. The rest of the columns indicate the B-Cubed F
scores (see §5.3) for various datasets discussed in §5.1. The last column
indicates the mean B-Cubed F-scores averaged across the aforementioned
datasets.

For the additional proportions +12.5% and +50%, the reported scores are means
along with standard deviations (in parentheses) over the five validation sets
(see §5.1). Note that the standard deviation for the overall averaged B-Cubed
F score is considerably much less than those of individual datasets. This
happens since in every run on a train-test split the model may perform high on
one dataset or low on the other, yet when it comes to the mean performance it
is quite stable.

###  6.1 Discussion

From the results, it is visible that with increased supervision, CogTran2
improves consistently when compared to other supervised methods. At the same
time, CogTran2 crosses the previous best LexStInf10K with additional +12.5%
supervision, i.e., with only 20 concepts per family. Since the results of
proportions +12.5% and +50% are cross-validated, it is possible to compare the
performances throughout. Note that LexStat is not a supervised method and,
hence, additional supervision does not make sense with it. With zero
additional data, CogTran2 surpasses all the other methods on CHN and IE
language families since they are present in training as well. While AN data is
also present in both sets i.e., train and test, the individual languages do
not overlap much as in the case of CHN and IE.

Although SVM beats CogTran2 on +0% additional data, which is not surprising
since this is primarily dependent on LexStInf1K scores, it shows only a little
increase in scores with an increase in additional training. Hence, overall, it
is behind CogTran2 for the other two proportions. The maximum score of SVM
does not appear to be significantly different from its base model LexStInf1K
on whose scores it is dependent. We performed student t-tests vis-à-vis SVM
and CogTran2 scores for proportions +12.5% and +50%. On whatever dataset
CogTran2 leads ahead of SVM, it is statistically significant for a 5% level of
significance, i.e., $p<0.05$. SVM leads ahead of CogTran2 significantly only
on two datasets, namely, Austronesian (AN) and Romance (ROM) in both
proportions. The reason for this is unclear as of now. Analysis with
linguistic expertise in these languages could possibly unveil the cause.

CharCNN has the disadvantage of not using aligned input. Hence, it lags behind
other models as expected (except SCA at extra supervision) despite showing a
significant improvement over the additional training data.

Hence, it can be concluded that CogTran2 is the best performing model when
there is sufficient labeled data. It is also likely to show improvement when
there is plenty of labeled data. Further, given the availability of GPU and
considering the present implementations, CogTran2 is much faster since it
starts from MSA and not from independent pairwise computations.

###  6.2 Ablation Tests

Method | Data Split
---|---
+0% | +12.5% | +50%
CogTran2 | 0.821 | 0.841 (± 0.002) | 0.853 (± 0.002)
CogTran | 0.815 | 0.830 (± 0.002) | 0.841 (± 0.002)
CogTranL4 | 0.806 | 0.830 (± 0.002) | 0.842 (± 0.004)
CogTranPair | 0.779 | 0.813 (± 0.003) | 0.833 (± 0.001)

Table 4: Mean B-Cubed F scores on various data splits for various ablation
models. Standard deviations are indicated in parentheses for the data splits
where cross-validation was performed.

The results of the ablation tests described in §5.4 on the data proportions
+0%, +12.5% and +50% are presented in Table 4. The first column indicates the
method and the second column lists the respective B-Cubed F-score averaged
over all the datasets. These are mean scores along with standard deviations
across all five cross-validated sets. CogTran, which lacks a Pairwise module
(§4.4), underperforms significantly than CogTran2, which is the model
proposed. Also, increasing the number of layers to 4 in CogTranL4 does not
help either. Hence, it can be concluded that the Pairwise module alone
contributes to further increasing the performance in CogTran2. Further since
CogTrainPair, unlike the other two, starts from aligned word pairs akin to all
other previous models, and takes input from an aligned word pair and outputs
cognacy probability for that pair. Hence, the Pairwise module cannot be
incorporated into this setup.

It is visible that CogTran, which acts on an MSA input performs way better
than CogTranPair which acts on aligned word pairs. At the same time, CogTran
(< 20 GPU min per split) is much faster than CogTranPair (about 1 GPU hr per
split) for the same reason. In other words, let input MSA have $r$ rows and
$c$ columns, then CogTranPair acts on all possible pairs of rows hence, in
$\mbox{O}{(r^{2})}$ steps. On the other hand, CogTran for a single MSA acts
only once which results in the speed-up.

###  6.3 Error Analysis

To understand the working of CogTran2, we attempt to study some of the cluster
predictions as follows. For this purpose, we consider CogTran2 trained on
+12.5% proportion and the results on IE (Indo-European) dataset.

####  6.3.1 Sound Correspondences

The fundamental aspect for comparing two languages is to identify regular
sound correspondences (Campbell, 2013). Methods like LexStat List (2012) have
built similarity metrics for cognacy judgement between two words giving
weightage to both the recurrent sound correspondences as well as phonetic
information. In this regard, we note that CogTran2 appears to have learned
some recurrent sound correspondences by observing the initial consonant. For
example, Proto-Indo-European *s- undergoes lenition in Hellenic branch and
appears as h- is Ancient Greek (Mallory and Adams, 2006). In the dataset we
have used, two words occur as instances for this sound change, namely,
/"hE:lios/ ‘sun’ and /"hals/ ‘salt’. Both these words are clustered correctly
with their cognates in other daughter languages such as Old Norse /so:l/,
Oriya /surdZO/ in case of the concept ‘sun’ and English /sO:lt/, French /sEl/
in case of the concept ‘salt’. Thus, one may assume that the sound change PIE
*s > Ancient Greek h has been learned by the model.

Another set of sound changes where position of articulation changes is Grimm’s
law where Proto-Indo-European hard consonants undergo a chain shift in
Germanic family (Mallory and Adams, 2006). For instance, in the velar shift
defined by Grimm’s law i.e., *gh > *g > *k > *h , change in the place of
articulation occurs in the sound change *k > *h. The model also learns this
sound change as supported by the instances mentioned as follows. For the
concept ‘dog’, German /hUnt/ has been correctly clustered together with
Ancient Greek /"kyOn/ and Old Irish /ku:/. Further, for the concept ’horn’,
German /hOrn/ and Ancient Greek /keras/ are similarly clustered together
correctly. This sound change has been learned by the model to an extent that
unrelated German /hIml/ and Latin /kaelUm/ meaning ‘sky’ have been classified
as cognates. Both the sound changes mentioned above have two instances as
examples in the dataset.

On the other hand, Marathi /dzaN/ and Ossetic /zon/ for the concept ‘know’
have been incorrectly classified as different. This happens to be the only
example where the phonemes /dz/ and /z/, which fall in different sound
classes, co-occur in the respective languages. Hence, it may be concluded that
at least two examples are needed to learn a sound change. However, it is
desirable to perform a thorough quantitative analysis of recurrent sound
changes to support these findings. It could not be performed due to a lack of
readily available annotated data for the same.

####  6.3.2 Partial Cognacy

Further, the network seems to consider the entire word and not just the
important root in some cases. For example, for the meaning ‘woman’, Old Norse
/kven: maDr/ and Icelandic /khvEn ma:Dr/ have been assigned a different
cluster than that of Old Swedish /kvin:a/ and Danish /ghven@/. This is
conceivable since affixes cannot be learned to be ignored easily. Detection of
sub-word cognates in presence of such affixes is part of _partial cognacy_
problem which was dealt in List et al. (2016). It is, thus, clear that
CogTran2, at its present training level, cannot distinguish partial cognates.

####  6.3.3 Other Errors

Many errors are, however, somewhat incomprehensible. For example, in the case
of ‘tooth’, Greek /"Dondi/ has been clustered together with English /tu:T/ but
not with Italian /dEntE/. There could be a role of root vowel in this
particular example. Nevertheless, it is important to understand the source of
errors which demands linguistic expertise to identify the bottlenecks of the
current models and to improve beyond them.

##  7 Conclusions

In this paper, we have proposed a Transformer-based model inspired by
evolutionary biology for the task of automatic cognate detection. The model
can harness efficiently the labeled data and consequently, with sufficient
data, outperforms existing approaches that do not make efficient use of
supervision data. In particular, better results are obtained with only 20
concepts per family on some of the datasets. To the best of our knowledge, we
proposed for the first time in this particular problem a method that directly
outputs link probabilities, i.e., pairwise similarities from an input MSA in
an end-to-end fashion, unlike all the previous methods which act on aligned
pairs of words. We demonstrated through the primary results and ablation
studies that this approach of inputting MSA rather than paired alignments
results not just in a significant increase in performance but also in
drastically reducing the computation time. We have also demonstrated by
observing few outputs that the model is capable of learning regular sound
changes from just two example instances in the data for a particular sound
change.

Evaluation of Cognate Transformer on phylogenetic reconstruction task (Rama et
al., 2018) is an unexplored problem and, thus, can be a potential topic of
future work.

## Limitations

As mentioned in §6, the proposed model lags on the datasets Romance and
Austronesian somewhat behind SVM and LexStat-Infomap and on Pama-Nyungan
concerning Lexstat-Infomap despite increasing the supervision. While the
performance on the Romance dataset is near saturated (>92%) in any case, the
lag in performance on Austronesian and Pama-Nyungan data is an issue that is
required to be studied with domain linguistic expertise to understand the
bottleneck of this model. Similarly, although our model improves drastically
on Sino-Tibetan by 5% when compared to the previous best, it is an
underperforming dataset since the B-Cubed F-scores on all other datasets
except this are more than 80%. Thus, a similar study with linguistic expertise
is required to identify the bottleneck of the overall methodologies.
Additionally, as mentioned in §5.2, a GPU memory of 10GB could only
accommodate a batch of size 4 during training with maximum MSAs, i.e., when
the number of languages in a family was 136. Thus, larger GPU storage is
required for larger mass comparisons involving more languages under
comparisons. As mentioned in §6.3, the ability of the model to learn regular
sound correspondences has only been determined by anecdotal instances. A more
thorough quantitative study is desirable, which requires annotated data for
the same. The model also does not account for partial cognacy, i.e.,
identifying distinctions between exact cognates versus morphologically
modified or compounded cognates (see §6.3) as addressed in List et al. (2016).
Further, the model is also not tuned at this point to distinguish between true
cognates and borrowals.

## Ethics Statement

All the data and code of baseline models used in this paper are obtained from
publicly available sources. Hence, we do not see any ethical concerns or
conflicts of interest.

## References

  * Akavarapu and Bhattacharya (2023) V.S.D.S.Mahesh Akavarapu and Arnab Bhattacharya. 2023.  Cognate transformer for automated phonological reconstruction and cognate reflex prediction.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 6852–6862, Singapore. Association for Computational Linguistics.
  * Amigó et al. (2009) Enrique Amigó, Julio Gonzalo, Javier Artiles, and Felisa Verdejo. 2009.  A comparison of extrinsic clustering evaluation metrics based on formal constraints.  _Information retrieval_ , 12:461–486.
  * Brown et al. (2008) Cecil H Brown, Eric W Holman, Søren Wichmann, and Viveka Velupillai. 2008.  Automated classification of the world’s languages: a description of the method and preliminary results.  _Language Typology and Universals_ , 61(4):285–308.
  * Campbell (2013) Lyle Campbell. 2013.  _Historical linguistics_.  Edinburgh University Press.
  * Dellert (2018) Johannes Dellert. 2018.  Combining information-weighted sequence alignment and sound correspondence models for improved cognate detection.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 3123–3133, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Jäger et al. (2017) Gerhard Jäger, Johann-Mattis List, and Pavel Sofroniev. 2017.  Using support vector machines and state-of-the-art algorithms for phonetic alignment to identify cognates in multi-lingual wordlists.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_ , pages 1205–1216, Valencia, Spain. Association for Computational Linguistics.
  * Jumper et al. (2021) John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. 2021.  Highly accurate protein structure prediction with alphafold.  _Nature_ , 596(7873):583–589.
  * Kanojia et al. (2020) Diptesh Kanojia, Raj Dabre, Shubham Dewangan, Pushpak Bhattacharyya, Gholamreza Haffari, and Malhar Kulkarni. 2020.  Harnessing cross-lingual features to improve cognate detection for low-resource languages.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 1384–1395, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Kanojia et al. (2021) Diptesh Kanojia, Prashant Sharma, Sayali Ghodekar, Pushpak Bhattacharyya, Gholamreza Haffari, and Malhar Kulkarni. 2021.  Cognition-aware cognate detection.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 3281–3292, Online. Association for Computational Linguistics.
  * Kim et al. (2023) Young Min Kim, Kalvin Chang, Chenxuan Cui, and David R. Mortensen. 2023.  Transformed protoform reconstruction.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 24–38, Toronto, Canada. Association for Computational Linguistics.
  * List (2010) Johann-Mattis List. 2010.  Sca: Phonetic alignment based on sound classes.  In _European Summer School in Logic, Language and Information_ , pages 32–51. Springer.
  * List (2012) Johann-Mattis List. 2012.  LexStat: Automatic detection of cognates in multilingual wordlists.  In _Proceedings of the EACL 2012 Joint Workshop of LINGVIS & UNCLH_, pages 117–125, Avignon, France. Association for Computational Linguistics.
  * List and Forkel (2021) Johann-Mattis List and Robert Forkel. 2021.  Lingpy. a python library for historical linguistics. version 2.6.9.
  * List et al. (2017) Johann-Mattis List, Simon J Greenhill, and Russell D Gray. 2017.  The potential of automatic word comparison for historical linguistics.  _PloS one_ , 12(1):e0170046.
  * List et al. (2016) Johann-Mattis List, Philippe Lopez, and Eric Bapteste. 2016.  Using sequence similarity networks to identify partial cognates in multilingual wordlists.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 599–605, Berlin, Germany. Association for Computational Linguistics.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * MacSween and Caines (2020) Roddy MacSween and Andrew Caines. 2020.  An expectation maximisation algorithm for automated cognate detection.  In _Proceedings of the 24th Conference on Computational Natural Language Learning_ , pages 476–485, Online. Association for Computational Linguistics.
  * Mallory and Adams (2006) James P Mallory and Douglas Q Adams. 2006.  _The Oxford introduction to proto-Indo-European and the proto-Indo-European world_.  Oxford University Press, USA.
  * Nath et al. (2022) Abhijnan Nath, Rahul Ghosh, and Nikhil Krishnaswamy. 2022.  Phonetic, semantic, and articulatory features in Assamese-Bengali cognate detection.  In _Proceedings of the Ninth Workshop on NLP for Similar Languages, Varieties and Dialects_ , pages 41–53, Gyeongju, Republic of Korea. Association for Computational Linguistics.
  * Needleman and Wunsch (1970) Saul B Needleman and Christian D Wunsch. 1970.  A general method applicable to the search for similarities in the amino acid sequence of two proteins.  _Journal of molecular biology_ , 48(3):443–453.
  * Rama (2016) Taraka Rama. 2016.  Siamese convolutional networks for cognate identification.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , pages 1018–1027, Osaka, Japan. The COLING 2016 Organizing Committee.
  * Rama (2018) Taraka Rama. 2018.  Similarity dependent Chinese restaurant process for cognate identification in multilingual wordlists.  In _Proceedings of the 22nd Conference on Computational Natural Language Learning_ , pages 271–281, Brussels, Belgium. Association for Computational Linguistics.
  * Rama and List (2019) Taraka Rama and Johann-Mattis List. 2019.  An automated framework for fast cognate detection and Bayesian phylogenetic inference in computational historical linguistics.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6225–6235, Florence, Italy. Association for Computational Linguistics.
  * Rama et al. (2018) Taraka Rama, Johann-Mattis List, Johannes Wahle, and Gerhard Jäger. 2018.  Are automatic methods for cognate detection good enough for phylogenetic reconstruction in historical linguistics? In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 393–400, New Orleans, Louisiana. Association for Computational Linguistics.
  * Rani et al. (2023) Priya Rani, Koustava Goswami, Adrian Doyle, Theodorus Fransen, Bernardo Stearns, and John P. McCrae. 2023.  Findings of the SIGTYP 2023 shared task on cognate and derivative detection for low-resourced languages.  In _Proceedings of the 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP_ , pages 126–131, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Rao et al. (2021) Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. 2021.  Msa transformer.  In _International Conference on Machine Learning_ , pages 8844–8856. PMLR.
  * Sokal and Michener (1958) Robert R. Sokal and Charles Duncan Michener. 1958.  A statistical method for evaluating systematic relationships.  _University of Kansas science bulletin_ , 38:1409–1438.
  * Thompson et al. (2003) Julie D Thompson, Toby J Gibson, and Des G Higgins. 2003.  Multiple sequence alignment using clustalw and clustalx.  _Current protocols in bioinformatics_ , (1):2–3.
  * Turchin et al. (2010) Peter Turchin, Ilia Peiros, and Murray Gell-Mann. 2010.  Analyzing genetic connections between languages by matching consonant classes.  _Journal of Language Relationship_ , (5 (48)):117–126.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.


11institutetext: School of Electrical and Computer Engineering, Cornell
University 22institutetext: School of Applied and Engineering Physics, Cornell
University

# Joint Optimization of Hadamard Sensing and Reconstruction in Compressed
Sensing Fluorescence Microscopy

Alan Q. Wang 11 Aaron K. LaViolette 22 Leo Moon 22 Chris Xu 22 Mert R. Sabuncu
11

###### Abstract

Compressed sensing fluorescence microscopy (CS-FM) proposes a scheme whereby
less measurements are collected during sensing and reconstruction is performed
to recover the image. Much work has gone into optimizing the sensing and
reconstruction portions separately. We propose a method of jointly optimizing
both sensing and reconstruction end-to-end under a total measurement
constraint, enabling learning of the optimal sensing scheme concurrently with
the parameters of a neural network-based reconstruction network. We train our
model on a rich dataset of confocal, two-photon, and wide-field microscopy
images comprising of a variety of biological samples. We show that our method
outperforms several baseline sensing schemes and a regularized regression
reconstruction algorithm. Our code is publicly-available at
https://github.com/alanqrwang/csfm.

###### Keywords:

Fluorescence Microscopy Compressed Sensing Joint Optimization.

##  1 Introduction

Fluorescence microscopy (FM) is an imaging modality widely used in biological
research [12]. FM follows a Poissonian-Gaussian noise model which determines
the signal-to-noise ratio (SNR) [6, 33]. One common solution to increase SNR
involves taking multiple measurements and averaging them; however, this
invariably leads to an increase in the total number of photons used to
generate an image, which can be undesirable when factors like photo-damage,
photo-bleaching, and viability of the sample are considered [8, 14].
Therefore, the trade-off between image quality and the total number of
measurements collected is important to optimize [11, 24, 26].

Compressed sensing (CS) seeks to address this by collecting fewer measurements
during sensing [7, 16, 20, 27] and performing reconstruction on the noisy
measurements to recover the image. Typically, reconstruction involves
iterative optimization [2, 3] or data-driven techniques [28, 30].

Recently, researchers in multiple domains have explored the idea of jointly
optimizing both the sensing and reconstruction parts end-to-end on training
data, thus allowing for improved performance compared to separate optimization
[1, 21, 28]. In this paper, we build on this idea and propose a method of
jointly optimizing sensing and reconstruction in CS-FM.

Our contributions are as follows. We design a stochastic sensing model which
is capable of learning the relative importance of each coefficient in the
Hadamard sensing basis. Simultaneously, we train the parameters of a neural
network which reconstructs the noisy measurements from the sensing model. Both
parts are optimized end-to-end on training data, subject to a loss which
maximizes reconstruction quality under a total measurement constraint. We
evaluate both our sensing and reconstruction models and show a performance
gain over CS-FM benchmarks.

##  2 Background

###  2.1 Fluorescence Microscopy and Hadamard Sensing

Let $\bm{x}\in\mathbb{R}^{N}$ be a vector of $N$ pixels in the field-of-view
(FOV), where $\bm{x}_{i}$ is the signal from pixel $i$ for a fixed acquisition
time. In FM, a popular basis for collecting measurements of $\bm{x}$ is the
Hadamard basis, which is well-studied, computationally-efficient, and easily
realized via a digital micromirror device (DMD) in the imaging setup [19, 20,
22, 31].

In Hadamard sensing, measurements correspond to the Hadamard coefficients.
Mathematically, these coefficients are given by a linear transformation of
$\bm{x}$ by a Hadamard matrix $H\in\\{-1,1\\}^{N\times N}$, where each row
$H_{i}$ is a Hadamard pattern. To physically measure each coefficient, $H$ is
reformulated as the difference between two binary, complementary matrices
$H^{+},H^{-}\in\\{0,1\\}^{N\times N}$:

$H=\frac{1}{2}\left(\bm{J}+H\right)-\frac{1}{2}\left(\bm{J}-H\right):=H^{+}-H^{-},$
(1)

where $\bm{J}$ is a matrix of ones. To obtain the measurement associated with
the $i$th Hadamard coefficient, two sequential acquisitions are made by
illuminating the sample according to the two binary patterns corresponding to
rows $H_{i}^{+}$ and $H_{i}^{-}$. All the light for each acquisition is
collected at one detector, and the Hadamard coefficient is computed by
subtracting the two measurements111Note we are not considering the effects of
a point-spread function here..

For noise considerations, we adopt a Poissonian-Gaussian noise model [33, 6,
15]. Incorporating this noise model with the above complementary measurement
scheme, each Hadamard coefficient $\bm{y}_{i}$ follows a distribution

$\displaystyle\begin{split}\bm{y}_{i}&\sim
p(\bm{y}_{i}\mid\bm{x})=\left[a\mathcal{P}\left(\frac{1}{a}H_{i}^{+}\bm{x}\right)+\mathcal{N}(0,b)\right]-\left[a\mathcal{P}\left(\frac{1}{a}H_{i}^{-}\bm{x}\right)+\mathcal{N}(0,b)\right].\end{split}$
(2)

Here, $\mathcal{P}$ is a Poisson distribution which models signal-dependent
uncertainty, while $\mathcal{N}$ is a normal distribution which models signal-
independent uncertainty. Note that $a>0$ is a conversion parameter (e.g. one
detected photon corresponds to a signal of $a$). All noise components are
assumed to be independent.

To increase SNR, we consider the case where multiple measurements are
collected and then averaged. In particular, for $S_{i}$ i.i.d.
measurements222Note we are defining one measurement to be the computation of
one Hadamard coefficient without averaging, despite the fact that this is a
two-step process in the physical imaging setup. of the $i$th coefficient
$\bm{y}_{i}^{(1)},...,\bm{y}_{i}^{(S_{i})}\sim p(\bm{y}_{i}\mid\bm{x})$,

$\displaystyle\begin{split}\bm{y}_{i}^{avg}&=\frac{1}{S_{i}}\sum_{s=1}^{S_{i}}\bm{y}_{i}^{(s)}\\\
&\sim\frac{1}{S_{i}}\left[a\mathcal{P}\left(\frac{S_{i}}{a}H_{i}^{+}\bm{x}\right)+\mathcal{N}(0,S_{i}b)\right]-\frac{1}{S_{i}}\left[a\mathcal{P}\left(\frac{S_{i}}{a}H_{i}^{-}\bm{x}\right)+\mathcal{N}(0,S_{i}b)\right].\end{split}$
(3)

It follows that while the mean is invariant to $S_{i}$, the variance decreases
as $S_{i}$ increases:

$\displaystyle\mathbb{E}[\bm{y}_{i}^{avg}]$
$\displaystyle=H_{i}^{+}\bm{x}-H_{i}^{-}\bm{x}=H_{i}\bm{x},$ (4)
$\displaystyle Var(\bm{y}_{i}^{avg})$
$\displaystyle=\frac{1}{S_{i}}\left[aH_{i}^{+}\bm{x}+b\right]-\frac{1}{S_{i}}\left[aH_{i}^{-}\bm{x}+b\right].$
(5)

Note that the number of photons coming from the sample to produce the image is
proportional to the sum of the $S_{i}$’s, i.e. the total number of
measurements. In this work, we pose the problem of optimizing the SNR-photon
tradeoff as one of maximizing image quality subject to a constraint on the
total measurements.

###  2.2 Sensing and Reconstruction Optimization

Prior work has explored the idea of jointly optimizing the sensing and
reconstruction portions of computational imaging in different domains. The
underlying intuition is that optimizing both parts concurrently leads to
better performance than optimizing separately. Deterministic sensing patterns
have been jointly optimized with neural-network based reconstruction schemes
in designing the color multiplexing pattern of camera sensors [4], the LED
illumination pattern in Fourier ptychography [10], the optical parameters of a
camera lens [18], and microscopy cell detection [28]. More recently,
stochastic sampling of sensing patterns have been explored in the context of
CS magnetic resonance imaging [1, 32] and very-long-baseline-interferometry
(VLBI) array design [21]. In this paper, we extend the idea of joint
optimization of sensing strategies and neural-network based reconstruction to
CS-FM.

##  3 Proposed Method

Figure 1: Proposed model. Both sensing and reconstruction are trained jointly
and end-to-end according to Eq. (7).

Suppose $S$ measurements are made at each of the $N$ Hadamard coefficients. In
the unconstrained case, this corresponds to $NS$ total measurements. At each
coefficient, the $S$ measurements are averaged and follow a distribution
according to Eq. (3). Given a compression ratio of $\alpha\in[0,1]$ which
restricts the total number of measurements to $\alpha NS$, the question we
pose is as follows: how do we learn the optimal allocation of these $\alpha
NS$ measurements across Hadamard bases, while simultaneously learning the
parameters of a neural network reconstruction model, such that we maximize
reconstruction quality?

###  3.1 End-to-end Sensing and Reconstruction Scheme

Let $\bm{x}$ denote a vectorized ground truth image from a dataset
$\mathcal{D}$ and let $\bm{y}_{i}^{(1)},...,\bm{y}_{i}^{(S)}\sim
p(\bm{y}_{i}\mid\bm{x})$ denote $S$ measurements at the $i$th Hadamard
coefficient. Define $\bm{P}\in[0,1]^{N}$ as a probabilistic mask, normalized
such that $\frac{1}{N}\left\lVert\bm{P}\right\rVert_{1}=\alpha$.

For each Hadamard coefficient, we draw $S$ i.i.d. samples from a Bernoulli
distribution with parameter $\bm{P}_{i}$, i.e.
$\bm{M}_{i}^{(1)},...,\bm{M}_{i}^{(S)}\sim\mathcal{B}(\bm{P}_{i})$. In
particular, $\bm{M}_{i}^{(s)}\in\\{0,1\\}$ indicates if measurement
$s\in\\{1,...,S\\}$ should be included in the average. As such, we multiply
$\bm{M}_{i}^{(s)}$ with measurement $\bm{y}_{i}^{(s)}$. Finally, we take the
average over all included measurements to arrive at the per-coefficient
measurement $\bm{z}_{i}$:

$\bm{z}_{i}=\frac{\sum_{s=1}^{S}\bm{M}_{i}^{(s)}\bm{y}_{i}^{(s)}}{\sum_{t=1}^{S}\bm{M}_{i}^{(t)}},\
\ \text{where}\ \bm{M}_{i}^{(s)}\sim\mathcal{B}(\bm{P}_{i})\ \text{and}\
\bm{y}_{i}^{(s)}\sim p(\bm{y}_{i}\mid\bm{x}).$ (6)

If we collect all $\bm{z}_{i}$’s in a vector $\bm{z}$ and collect all
$\bm{M}_{i}^{(s)}$’s in $\bm{M}$, then we can summarize the sensing model by a
sensing function $\bm{z}=g(\bm{M},\bm{x})$. Note that
$\bm{M}\sim\prod_{s=1}^{S}\prod_{i=1}^{N}\mathcal{B}(\bm{P}_{i})$.

$\bm{z}$ is subsequently the input to the reconstruction model, which first
transforms $\bm{z}$ to image space via the Hadamard transform and then passes
the result into a neural network $f_{\theta}$ with parameters $\theta$. In
this work, we use a U-Net architecture [17]. Thus, the final reconstruction is
$\hat{\bm{x}}=f_{\theta}(H\bm{z})$.

The proposed model is illustrated in Fig. 1.

###  3.2 Loss Function

The end-to-end sensing and reconstruction scheme is optimized jointly,
according to a loss that maximizes image quality under the normalization
constraint:

$\min_{\bm{P},\theta}\mathbb{E}_{\bm{M}\sim\prod_{s}\prod_{i}\mathcal{B}(\bm{P}_{i})}\sum_{\bm{x}\in\mathcal{D}}\left\lVert\hat{\bm{x}}-\bm{x}\right\rVert_{2}^{2}\
\ \text{s.t.}\ \ \hat{\bm{x}}=f_{\theta}(Hg(\bm{M},\bm{x}))\ \ \text{and}\ \
\frac{1}{N}\left\lVert\bm{P}\right\rVert_{1}=\alpha.$ (7)

A higher value of the optimized $\bm{P}^{*}_{i}$ implies that more
measurements for the $i$th coefficient will be averaged, resulting in a higher
SNR for that coefficient. Thus, $\bm{P}^{*}_{i}$ effectively encodes the
“relative importance” of each location in Hadamard space for reconstruction.
During imaging, one would draw $S$ Bernoulli realizations with parameter
$\bm{P}_{i}^{*}$ to determine the number of measurements to collect at basis
$i$. Note the total required measurements will be $\alpha NS$ on average.

###  3.3 Implementation

To allow gradient computation through the Bernoulli sampling operation, we use
the straight-through Gumbel-Softmax approximation [9, 13], which samples from
a Bernoulli distribution in the forward pass and computes gradients in the
backward pass via a differentiable relaxation of the sampling procedure. A
temperature hyperparameter $\tau$ controls the degree of relaxation.

To enforce the normalization constraint in Eq. (7), we add a closed-form
differentiable normalization layer to the reconstruction network[1, 32]. Let
$\tilde{\bm{P}}$ be an unconstrained probabilistic mask, and let
$\bar{p}=\frac{1}{N}\|\tilde{\bm{P}}\|_{1}$ be the average value of
$\tilde{\bm{P}}$. Note that $1-\bar{p}$ is the average value of
$1-\|\tilde{\bm{P}}\|_{1}$. Then,

$\bm{P}=\begin{cases}\frac{\alpha}{\bar{p}}\tilde{\bm{P}},&\textrm{ if
}\bar{p}\geq\alpha,\\\
1-\frac{1-\alpha}{1-\bar{p}}(1-\tilde{\bm{P}}),&\textrm{
otherwise.}\end{cases}$ (8)

It can be shown that Eq. (8) yields $\bm{P}\in[0,1]^{N}$ and
$\frac{1}{N}\|\bm{P}\|_{1}=\alpha$, thus enforcing the hard constraint in Eq.
(7).

##  4 Experiments

Figure 2: Baseline and learned masks for $\alpha=0.25$ (top) and
$\alpha=0.125$ (bottom). All masks are displayed in 2D-sequency order.

####  4.0.1 Training Details

In this work, we used a U-Net architecture [17] for $f_{\theta}$, but
alternative architectures which directly incorporate the forward model may be
used as well, such as unrolled architectures [5, 29] and physics-based neural
networks [10]. In particular, our U-Net has 64 hidden channels per encoder
layer and intermediate ReLU activations. The network was trained with ADAM
optimizer. We set the temperature hyperparameter $\tau=0.8$ via grid-search on
validation loss. All training and testing experiments in this paper were
performed on a machine equipped with an Intel Xeon Gold 6126 processor and an
NVIDIA Titan Xp GPU. All models were implemented in Pytorch.

####  4.0.2 Baselines

For reconstruction, we compare our U-Net based method against a widely used
regularized regression reconstruction algorithm (TV-W) [23]333We used
publicly-available code from: https://github.com/alanqrwang/HQSNet.. The
algorithm minimizes a least-squares cost function with two regularization
terms: total variation and L1-penalty on wavelet coefficients. The weights of
these terms was established via grid search on validation performance.

As is common in the CS-FM literature, we implemented baseline binary sensing
schemes that collect either all (e.g., $S=50$) measurements or no measurement
at each basis. This can be denoted with a binary mask $\bm{P}\in\\{0,1\\}^{N}$
where $1$ indicates the basis is sampled and $0$ indicates the basis is not
sampled. Binary baseline masks include random (R), low-sequency (LS), and
half-half (HH) [20] Hadamard masks, shown in Fig. 2. Non-binary stochastic
masks include uniform (U) and learned masks. For U, on average
$\alpha$-percent of all available ($S$) measurements were used at each basis,
realized via Bernoulli sampling; i.e., $\bm{P}$ is a constant vector of
$\alpha$’s. We binarize the learned stochastic masks for the TV-W algorithm by
taking the top $\alpha$-percentile of basis patterns. This cannot be achieved
for uniform masks, so we only evaluate them on U-Net models.

####  4.0.3 Data

To illustrate the computational utility of our proposed method, models were
trained and tested on a combination of in-house data and the publicly-
available Fluorescence Microscopy Denoising (FMD) dataset, comprising of
confocal, two-photon, and wide-field microscopy images [33]. Our combined
dataset consists of a variety of representative biological samples, namely
blood vessels, neurons, and lymph nodes. Each FOV has $S=50$ noisy pixel-wise
acquisitions, which consist of raw measurements at constant excitation power.
Ground truth images $\bm{x}$ were obtained by pixel-wise averaging over all
$S$ frames.

To obtain $\bm{y}_{i}^{(s)}$ input Hadamard measurements, we implemented two
approaches: simulated and raw. In the simulated approach, we applied random
noise to the ground truth $\bm{x}$ following Eq. (2). Values of $a$ and $b$
were obtained from [33]444Negative values of $b$ were replaced with $0$
following the paper’s experiments [33].. In the raw approach, we applied the
Hadamard transform to raw pixel-wise measurements in a FOV. Although this is
not exact since individual noise contributions are not additive, it is a
reasonable approximation as we are interested in the relative importance of
each basis pattern.

All images were cropped to fixed grid sizes of $256\times 256$. The final
training, validation, and test splits were $1000$, $100$, and $500$ cropped
images, respectively; and included non-overlapping independent samples.
Reconstructions were evaluated against ground-truth images on peak signal-to-
noise ratio (PSNR) and structural similarity index measure (SSIM) [25].

###  4.1 Masks

Figure 3: PSNR for all masks, $\alpha=0.125$ (left) and $\alpha=0.25$ (right).

Fig. 2 shows baseline and learned masks, displayed in 2D-sequency order [7].
We notice that the learned masks prioritize low-sequency Hadamard patterns but
also spread out across higher-sequency patterns. We reason that the sensing
and reconstruction scheme learns to provide the U-Net with mostly low-sequency
information, and then subsequently “assigns” the U-Net the task of imputing
the missing high-sequency information.

###  4.2 Reconstruction Methods

Figure 4: Representative slices for $\alpha=0.25$. Left is ground truth, blue
is TV-W reconstructions, red is U-Net reconstructions. Masks are indicated in
titles.

Fig. 3 shows boxplots of PSNR values for both $\alpha=0.25$ and
$\alpha=0.125$. U-Net reconstruction models outperform TV-W for all
conditions. The pattern holds for both simulated and raw measurements.
Similarly, learned masks outperform the baseline masks consistently for U-Net
reconstruction (which the sensing was optimized for), while half-half masks
perform best for TV-W reconstruction. We conjecture that this is due to the
U-Net model being better at imputing the missing information in the learned
mask. Table 1 shows metrics for all models on PSNR and SSIM. Fig. 4 shows
representative reconstructions for raw data, which allows us to qualitatively
appreciate the superior quality afforded by the learned mask and U-Net
reconstruction.

Table 1: Performance for $\alpha=0.25$ and $\alpha=0.125$. Mean $\pm$ standard deviation across test cases. Simulated/Raw. $\alpha$ | Method | Mask | PSNR | SSIM
---|---|---|---|---
$0.25$ | TV-W | R | 10.40$\pm$2.066/10.44$\pm$1.847 | 0.275$\pm$0.127/0.286$\pm$0.126
|  | LS | 29.15$\pm$6.516/29.15$\pm$6.405 | 0.887$\pm$0.094/0.886$\pm$0.095
|  | HH | 33.20$\pm$6.544/33.42$\pm$5.948 | 0.905$\pm$0.094/0.913$\pm$0.084
|  | Learned | 29.87$\pm$5.525/28.20$\pm$5.080 | 0.917$\pm$0.049/0.901$\pm$0.051
| U-Net | R | 25.43$\pm$3.932/25.46$\pm$3.565 | 0.692$\pm$0.124/0.692$\pm$0.135
|  | LS | 33.86$\pm$4.593/34.30$\pm$4.866 | 0.904$\pm$0.060/0.917$\pm$0.058
|  | HH | 35.04$\pm$3.882/35.80$\pm$3.942 | 0.918$\pm$0.043/0.927$\pm$0.042
|  | U | 31.44$\pm$6.264/35.66$\pm$3.360 | 0.841$\pm$0.125/0.924$\pm$0.024
|  | Learned | 36.38$\pm$4.061/39.38$\pm$3.254 | 0.932$\pm$0.044/0.953$\pm$0.018
$0.125$ | TV-W | R | 8.646$\pm$1.499/8.631$\pm$1.316 | 0.216$\pm$0.108/0.218$\pm$0.101
|  | LS | 24.91$\pm$5.984/24.90$\pm$6.042 | 0.805$\pm$0.148/0.801$\pm$0.146
|  | HH | 28.39$\pm$6.439/28.27$\pm$6.129 | 0.843$\pm$0.137/0.847$\pm$0.131
|  | Learned | 27.21$\pm$5.365/26.29$\pm$5.320 | 0.873$\pm$0.069/0.862$\pm$0.070
| U-Net | R | 21.31$\pm$4.700/22.94$\pm$4.527 | 0.539$\pm$0.203/0.613$\pm$0.150
|  | LS | 29.49$\pm$4.957/30.00$\pm$4.720 | 0.813$\pm$0.129/0.826$\pm$0.125
|  | HH | 31.93$\pm$5.089/32.40$\pm$5.146 | 0.871$\pm$0.084/0.877$\pm$0.083
|  | U | 29.26$\pm$5.867/33.57$\pm$3.301 | 0.806$\pm$0.121/0.894$\pm$0.042
|  | Learned | 33.98$\pm$3.173/36.01$\pm$3.224 | 0.895$\pm$0.062/0.923$\pm$0.033

##  5 Conclusion

We presented a novel sensing and reconstruction model for CS-FM, which is
jointly trained end-to-end. We show that our scheme effectively learns the
optimal sensing and reconstruction, and outperforms established baselines.
Future work will include evaluating the proposed strategy on prospectively-
collected data.

##  6 Acknowledgments

This work was, in part, supported by NIH R01 grants (R01LM012719 and
R01AG053949, to MRS), the NSF NeuroNex grant (1707312, to MRS), an NSF CAREER
grant (1748377, to MRS), and an NSF NeuroNex Hub grant (DBI-1707312, to CX).

## References

  * [1] Bahadir, C.D., Wang, A.Q., Dalca, A.V., Sabuncu, M.R.: Deep-learning-based optimization of the under-sampling pattern in mri. IEEE Transactions on Computational Imaging 6, 1139–1152 (2020)
  * [2] Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Img. Sci. 2(1), 183–202 (Mar 2009)
  * [3] Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (Jan 2011)
  * [4] Chakrabarti, A.: Learning sensor multiplexing design through back-propagation. In: Proceedings of the 30th International Conference on Neural Information Processing Systems. p. 3089–3097. NIPS’16, Curran Associates Inc., Red Hook, NY, USA (2016)
  * [5] Diamond, S., Sitzmann, V., Heide, F., Wetzstein, G.: Unrolled optimization with deep priors. CoRR abs/1705.08041 (2017)
  * [6] Foi, A., Trimeche, M., Katkovnik, V., Egiazarian, K.: Practical poissonian-gaussian noise modeling and fitting for single-image raw-data. IEEE Transactions on Image Processing 17(10), 1737–1754 (2008)
  * [7] Gibson, G.M., Johnson, S.D., Padgett, M.J.: Single-pixel imaging 12 years on: a review. Opt. Express 28(19), 28190–28208 (Sep 2020)
  * [8] Hopt, A., Neher, E.: Highly nonlinear photodamage in two-photon fluorescence microscopy. Biophysical Journal 80(4), 2029–2036 (2001)
  * [9] Jang, E., Gu, S., Poole, B.: Categorical reparametrization with gumbel-softmax. In: Proceedings International Conference on Learning Representations 2017. OpenReviews.net (Apr 2017)
  * [10] Kellman, M., Bostan, E., Chen, M., Waller, L.: Data-driven design for fourier ptychographic microscopy. In: 2019 IEEE International Conference on Computational Photography (ICCP). pp. 1–8 (2019)
  * [11] Lee, S., Negishi, M., Urakubo, H., Kasai, H., Ishii, S.: Mu-net: Multi-scale u-net for two-photon microscopy image denoising and restoration. Neural Networks 125, 92–103 (2020)
  * [12] Lichtman, J., Conchello, J.: Fluorescence microscopy. Nat Methods (2005)
  * [13] Maddison, C.J., Mnih, A., Teh, Y.W.: The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables. In: International Conference on Learning Representations (2017)
  * [14] Magidson, V., Khodjakov, A.: Chapter 23 - circumventing photodamage in live-cell microscopy. In: Sluder, G., Wolf, D.E. (eds.) Digital Microscopy, Methods in Cell Biology, vol. 114, pp. 545–560. Academic Press (2013)
  * [15] Makitalo, M., Foi, A.: Optimal inversion of the generalized anscombe transformation for poisson-gaussian noise. IEEE Transactions on Image Processing 22(1), 91–103 (2013)
  * [16] Parot, V.J., Sing-Long, C., Adam, Y., Böhm, U.L., Fan, L.Z., Farhi, S.L., Cohen, A.E.: Compressed hadamard microscopy for high-speed optically sectioned neuronal activity recordings. Journal of Physics D: Applied Physics 52(14), 144001 (feb 2019)
  * [17] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. pp. 234–241. Springer International Publishing, Cham (2015)
  * [18] Sitzmann, V., Diamond, S., Peng, Y., Dun, X., Boyd, S., Heidrich, W., Heide, F., Wetzstein, G.: End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Trans. Graph. 37(4) (Jul 2018)
  * [19] Streeter, L., Burling-Claridge, G.R., Cree, M.J., Künnemeyer, R.: Optical full hadamard matrix multiplexing and noise effects. Appl. Opt. 48(11), 2078–2085 (Apr 2009)
  * [20] Studer, V., Bobin, J., Chahid, M., Mousavi, H.S., Candes, E., Dahan, M.: Compressive fluorescence microscopy for biological and hyperspectral imaging. Proceedings of the National Academy of Sciences 109(26), E1679–E1687 (2012)
  * [21] Sun, H., Dalca, A.V., Bouman, K.L.: Learning a probabilistic strategy for computational imaging sensor selection. In: 2020 IEEE International Conference on Computational Photography (ICCP). pp. 1–12 (2020)
  * [22] Sun MJ., Meng LT., E.M.e.a.: A russian dolls ordering of the hadamard basis for compressive single-pixel imaging. Sci Rep 7, 3464 (2017)
  * [23] Wang, A.Q., Dalca, A.V., Sabuncu, M.R.: Neural network-based reconstruction in compressed sensing mri without fully-sampled training data. In: Deeba, F., Johnson, P., Würfl, T., Ye, J.C. (eds.) Machine Learning for Medical Image Reconstruction. pp. 27–37. Springer International Publishing, Cham (2020)
  * [24] Wang, H., Rivenson, Y., Jin, Y.: Deep learning enables cross-modality super-resolution in fluorescence microscopy. In: Nat Methods 16. pp. 103–110 (2019)
  * [25] Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: From error visibility to structural similarity. Trans. Img. Proc. 13(4), 600–612 (Apr 2004)
  * [26] Weigert, M., Schmidt, U., Boothe, T.: Content-aware image restoration: pushing the limits of fluorescence microscopy. In: Nat Methods. pp. 1090–1097 (2018)
  * [27] Wijesinghe, P., Escobet-Montalbán, A., Chen, M., Munro, P.R.T., Dholakia, K.: Optimal compressive multiphoton imaging at depth using single-pixel detection. Optics Letters 44(20), 4981 (Oct 2019)
  * [28] Xue, Y., Bigras, G., Hugh, J., Ray, N.: Training convolutional neural networks and compressed sensing end-to-end for microscopy cell detection. IEEE Transactions on Medical Imaging 38(11), 2632–2641 (2019)
  * [29] Yang, Y., Sun, J., Li, H., Xu, Z.: Deep admm-net for compressive sensing mri. In: Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 29. Curran Associates, Inc. (2016)
  * [30] Yao R., Ochoa M., Y.P.e.a.: Net-flics: fast quantitative wide-field fluorescence lifetime imaging with compressed sensing – a deep learning approach. Light Sci Appl 8(26) (2019)
  * [31] Yu, X., Yang, F., Gao, B., Ran, J., Huang, X.: Deep compressive single pixel imaging by reordering hadamard basis: A comparative study. IEEE Access 8, 55773–55784 (2020)
  * [32] Zhang, J., Zhang, H., Wang, A., Zhang, Q., Sabuncu, M., Spincemaille, P., Nguyen, T.D., Wang, Y.: Extending loupe for k-space under-sampling pattern optimization in multi-coil mri. In: Deeba, F., Johnson, P., Würfl, T., Ye, J.C. (eds.) Machine Learning for Medical Image Reconstruction. pp. 91–101. Springer International Publishing, Cham (2020)
  * [33] Zhang, Y., Zhu, Y., Nichols, E., Wang, Q., Zhang, S., Smith, C., Howard, S.: A poisson-gaussian denoising dataset with real fluorescence microscopy images. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11702–11710 (2019)