# Can Latent Alignments Improve Autoregressive Machine Translation?

Adi Haviv Lior Vassertail∗ Omer Levy

The Blavatnik School of Computer Science
Tel Aviv University
<EMAIL_ADDRESS>Equal contribution.

###### Abstract

Latent alignment objectives such as CTC and AXE significantly improve non-
autoregressive machine translation models. Can they improve autoregressive
models as well? We explore the possibility of training autoregressive machine
translation models with latent alignment objectives, and observe that, in
practice, this approach results in degenerate models. We provide a theoretical
explanation for these empirical results, and prove that latent alignment
objectives are incompatible with teacher forcing.

##  1 Introduction

Latent alignment objectives, such as CTC Graves et al. (2006) and AXE
Ghazvininejad et al. (2020a), have been recently proposed for training non-
autoregressive models for machine translation Libovický and Helcl (2018);
Saharia et al. (2020). These objectives use a dynamic program to comb the
space of monotonic alignments between the “gold” target sequence and the token
probabilities the model predicts, thus reducing the loss from positional
misalignments and focusing on the original prediction error instead. For
example, consider the target sequence “there is a tiny difference between pink
and magenta”; if the model’s distribution favors the paraphrase “there is a
very small difference between pink and magenta”, substituting one token
(“tiny”) with two (“very small”) will cause a misalignment, and result in a
disproportionately large cross entropy loss. A latent alignment loss would
match the predictions of both “very” and “small” with the target “tiny”, while
aligning the rest of the sentence properly and computing a much lower loss
that focuses on this particular discrepancy. Could latent alignments also
benefit autoregressive models?

We apply CTC and AXE to standard autoregressive machine translation models. We
observe that, when trained with teacher forcing Williams and Zipser (1989),
CTC reduces to the vanilla cross entropy loss because CTC assumes that the
prediction sequence is longer than the target and has only one valid alignment
when they are equal. We further examine AXE, which does not share this
assumption, and find that it yields a degenerate model that almost perfectly
fits the training set but completely fails at inference time.

Our analysis reveals that latent alignments and teacher forcing are
fundamentally incompatible. We observe that there exists a valid alignment in
which the prediction $p_{i}$ is aligned with the target $y_{i-1}$ for almost
every token. Simultaneously, teacher forcing feeds the model with $y_{i-1}$
when computing the prediction $p_{i}$, encouraging the model to simply predict
its input under this alignment. While AXE allows this alignment for equal-
length prediction and target sequences, the phenomenon also occurs
(theoretically) in CTC if the predictions are longer, and in fact, occurs in
any latent alignment objective that can align a prediction $p_{j}$ with a
target $y_{i}$ where $i<j$.

##  2 Background: Latent Alignments

A latent alignment objective measures the compatibility between the target
sequence $Y$ and the sequence of predicted token probabilities $P$ by
considering a subspace of possible mappings between $Y$ and $P$. Latent
alignments are typically used in non-autoregressive models for automatic
speech recognition, and optical character recognition Graves et al. (2006),
and have recently been introduced to the task of machine translation Libovický
and Helcl (2018); Ghazvininejad et al. (2020a); Saharia et al. (2020). We
describe two such objectives, beginning with an overview of the common
notation and framework.

Operator |  Description | Formula | CTC | AXE
---|---|---|---|---
Align |  Predict the target token $Y_{i}$ with the distribution $P_{j}$. This is the default alignment, advancing along $A$’s diagonal. | $A_{i,j}=A_{i-1,j-1}\cdot P_{j}(Y_{i})$ | ✓ | ✓
Clone Target |  Assuming the target token $Y_{i}$ was predicted with the previous distribution $P_{j-1}$, repredict $Y_{i}$ using $P_{j}$. | $A_{i,j}=A_{i,j-1}\cdot P_{j}(Y_{i})$ | ✓ |
Clone Prediction |  Assuming the previous target token $Y_{i-1}$ was predicted with the distribution $P_{j}$, reuse $P_{j}$ to predict the next target token $Y_{i}$. | $A_{i,j}=A_{i-1,j}\cdot P_{j}(Y_{i})$ |  | ✓
Delimiter |  Use the distribution $P_{j}$ to predict the blank token $\varepsilon$ instead of the target token $Y_{i}$. This operation is akin to inserting $\varepsilon$ into the target sequence at the $i$-th position. | $A_{i,j}=A_{i,j-1}\cdot P_{j}(\varepsilon)$ | ✓ | ✓
Table 1: The set of possible operators in latent alignment dynamic programs.
Both AXE and CTC use align and delimiter, but apply different clone operators.

### Monotonic Alignments

Let $Y=y_{1},\ldots,y_{n}$ be the target sequence of $n$ tokens, and
$P=p_{1},\ldots,p_{m}$ be the model prediction, a sequence of $m$ token
probability distributions. A monotonic alignment $\alpha$ is a function that
maps every target position $i\in\\{1,\ldots,n\\}$ to a set of one or more
consecutive prediction positions $\alpha(i)\subseteq\\{1,\ldots,m\\}$, such
that $i\leq j\Leftrightarrow\max{\alpha(i)}\leq\min{\alpha(j)}$.

### Objective

Given an alignment $\alpha$, the objective is defined as follows:

$\displaystyle
L_{\alpha}(Y,P)=\prod_{i=1}^{n}\prod_{j\in\alpha(i)}p_{j}(y_{i})$ (1)

Since $\alpha$ is not provided a priori, it is necessary to aggregate over all
the possible alignments (hence latent alignments), by either summation
(Equation 2) or maximization (Equation 3):

$\displaystyle L^{\sum}(Y,P)$ $\displaystyle=\sum_{\alpha}L_{\alpha}(Y,P)$ (2)
$\displaystyle L^{\max}(Y,P)$ $\displaystyle=\max_{\alpha}L_{\alpha}(Y,P)$ (3)

In practice, the negative log loss is minimized during training:

$\displaystyle\ell(Y,P)=-\log L(Y,P)$ (4)

### Dynamic Programming

Aggregation can be done efficiently with dynamic programming, using
derivations of the forward-backward algorithm (for summation, as in CTC) or
the Viterbi algorithm (for maximization, as in AXE). These algorithms create
an aggregation matrix $A\in R^{n\times m}$, where each cell represents the
desired aggregation score $f$ (sum or max) over prefixes of the target and
prediction probability sequences: $A_{i,j}=L^{f}(Y_{\leq i},P_{\leq j})$. The
dynamic program combs through the space of alignments by implicitly
constructing every possibility using the set of local operators defined in
Table 1. The subspace of alignment functions that the program explores is
determined by the subspace of operators it employs.

### Connectionist Temporal Classification (CTC)

The CTC objective Graves et al. (2006) was originally introduced for speech
and handwriting recognition, where the prediction sequence $P$ is typically
much longer than the target sequence $Y$ ($m\gg n$). While computing the
summation objective (Equation 2), CTC uses only the align, clone target, and
delimiter operators. This means that CTC restricts $\alpha$ to the space of
alignments where every item in $P$ is aligned with at most one item in $Y$,
i.e. $\alpha(i)\cap\alpha(j)=\emptyset$ for $i\neq j$.

CTC was used in non-autoregressive machine translation by Libovický and Helcl
(2018) and more recently by Saharia et al. (2020). In both cases, the
prediction sequence was artificially inflated to be double (or more) the
length of the source-language input sequence in order to simulate the $m\gg n$
condition of speech recognition.

### Aligned Cross Entropy (AXE)

The AXE objective Ghazvininejad et al. (2020a) is specifically designed for
non-autoregressive machine translation. AXE finds the monotonic alignment that
minimizes the cross entropy loss (i.e., maximizes the likelihood, Equation 3)
in order to focus the penalty on the root errors instead of positional shifts
that result from them. AXE uses only the align, clone prediction, and
delimiter operators. This combination of operators allows AXE to align
prediction and target sequences of any lengths because clone prediction
inflates the prediction sequence while delimiter adds new target tokens.
However, since AXE cannot clone target tokens, every target position $i$ is
always aligned to a single prediction position, i.e. $|\alpha(i)|=1$. Figure 1
illustrates how AXE aligns the model’s predictions with the target sequence.

Target $Y$ | it | is | rainy | today | EOS
---|---|---|---|---|---
Model Predictions $P$ (Top 4) | it | is | so | rainy | today
however | the | rain | good | tonight
the | looks | very | and | EOS
but | this | $\varepsilon$ | very | good
Figure 1:  An illustration of how AXE aligns the model’s predictions $P$ with
the target sequence $Y$: “it is rainy today”. The model favors a slightly
different sequence (“it is so rainy today”), which would suffer from a high
penalty with the regular cross entropy loss. Instead, AXE finds a more
appropriate alignment $\alpha=(1,2,4,5,5)$ using the operator sequence align,
align, delimiter, align, align, clone prediction.

##  3 Combining CTC with Teacher Forcing Defaults to the Trivial Alignment

In an autoregressive setting, it is standard practice to use teacher forcing
Williams and Zipser (1989); i.e., when predicting the $i$-th token, the model
takes the prefix of the (gold) target sequence $Y_{<i}$ as input. This
dictates that the number of predictions is identical to the number of target
tokens ($m=|P|=|Y|=n$).

However, CTC assumes that the prediction sequence $P$ is typically much longer
than the target sequence $Y$ ($m\gg n$), and can only inflate $Y$ via clone
target and delimiter (see Section 2). This leaves only one valid alignment
when $m=n$: the trivial alignment $\alpha(i)=\\{i\\}$. CTC will thus default
to the same objective as the standard cross entropy loss.

Unlike CTC, the AXE objective aggregates over multiple alignments even when
$m=n$, because it uses both the delimiter operator (which inflates $Y$) as
well as the clone prediction operator (which inflates $P$).

##  4 Applying AXE to Autoregressive NMT

To apply AXE to autoregressive machine translation, we use a standard
sequence-to-sequence transformer model Vaswani et al. (2017) trained with
teacher forcing, replace the simple cross entropy loss function with AXE, and
add the empty token $\varepsilon$ to the vocabulary. We remove the
$\varepsilon$ tokens after decoding.

### Experiment Setup

We use fairseq Ott et al. (2019) to train a transformer encoder-decoder
Vaswani et al. (2017) on the IWSLT’14 DE-EN dataset Cettolo et al. (2015). The
dataset is preprocessed and tokenized into subwords with BPE Sennrich et al.
(2016) using the scripts provided by fairseq. We also use the implementation’s
default hyperparameters: 6 layers of encoder/decoder, 512 model dimensions,
1024 hidden dimensions, 4 attention heads. We optimize with Adam Kingma and Ba
(2015) for 50k steps with early stopping using 4096 tokens per batch. We
decode with beam search ($b=5$) and evaluate performance with BLEU Papineni et
al. (2002).

### Results

We observe two seemingly contradictory behaviors. On the one hand, the model
approaches a near-zero training loss within a single epoch, and observes
similar results when computing AXE loss on unseen examples in the validation
set (Figure 2). Meanwhile, at inference time, the model consistently produces
the empty sequence (after removing all instances of $\varepsilon$), scoring 0
BLEU on the test set. This indicates that the model has learned to “game” the
AXE objective without actually learning anything useful about machine
translation. What shortcut did the model learn?

Figure 2: Training and validation loss when using the AXE objective on
IWSLT’14 DE-EN with an autoregressive model.

##  5 Analysis

Prediction $Y$ | thank | you | for | listening | . | EOS
---|---|---|---|---|---|---
Alignment $\alpha(i)$ | 2 | 3 | 4 | 5 | 6 | 6
Model Predictions $P$ (Top 4) | 0.999 | $\varepsilon$ | 0.995 | thank | 0.999 | you | 0.999 | for | 0.994 | listening | 0.627 | .
8e-8 | EOS | 5e-5 | ’s | 2e-7 | pre@@ | 1e-5 | is | 3e-5 | ver@@ | 0.370 | EOS
8e-8 | … | 2e-5 | super@@ | 2e-7 | ke | 6e-6 | audience | 2e-5 | taking | 1e-4 | …
7e-8 | use | 2e-5 | unfortunate | 2e-7 | cu@@ | 5e-6 | oil | 2e-5 | sever@@ | 1e-4 | ’
Figure 3: An example of the constant alignment that AXE chooses after training
the model. Given the German source “danke fürs zuhören”, the model tries to
predict “thank you for listening”. Because the model is trained with teacher
forcing, it can simply learn to predict its input at each position, and assume
that AXE will align the prediction with the previous token (which is identical
to the input). For example, $p_{2}$ predicts “thank” with very high
probability because teacher forcing uses the previous target $y_{1}$ as the
decoder’s input in the second position. Notice how the final prediction
$p_{6}$ is used twice to predict both “.” and EOS.

To understand how the model learns to game the AXE objective, we analyze the
optimal alignments chosen by the objective, and find that they allow the model
to condition on the target token when trying to predict it. We prove that this
is the optimal solution when combining teacher forcing and AXE, and that it
holds for any latent alignment objective that allows the model to align future
target tokens with the current prediction.

### AXE finds a constant alignment

We examine the alignments chosen by AXE’s dynamic program for a sample of
training examples, and observe that they all belong to a consistent pattern:
delimiter, align, align, …, clone prediction. In other words, the chosen path
skips the first prediction by emitting the blank token $\varepsilon$ and then
aligns each prediction $p_{i}$ with the previous target token $y_{i-1}$. The
alignment synchronizes the positions at the end of the sequence by cloning the
last prediction to compensate for the offset produced by the initial delimiter
operator.

### Each prediction conditions on its target

The teacher forcing algorithm conditions the prediction $p_{i}$ on the ground
truth of the previous tokens $y_{1},\ldots,y_{i-1}$ to predict the target
token $y_{i}$. However, if the prediction $p_{i}$ is aligned with the target
$y_{i-1}$, then it is effectively observing its target through the input, and
only needs to learn the identity function. Formally, we see that for every
$1<i<n$ the prediction is trivial:

$\displaystyle p_{i}(y_{i-1})$ $\displaystyle=Pr(y_{i-1}|X,Y_{<i})$
$\displaystyle=Pr(y_{i-1}|y_{i-1})=1$

Figure 3 demonstrates this phenomenon on an actual example using the model’s
predictions.

### The cost of sharing the last prediction

It is now clear to see that the loss should indeed be close to zero. Having
said that, it is not infinitesimal; the last two tokens (typically “.” and
EOS) need to be predicted from the same distribution. At best, this yields a
loss of $-2\log(0.5)/n$, which is just below the loss observed in Figure 2
when considering the average target sequence length in IWSLT’14 DE-EN is
around $\bar{n}\approx 30$.

### Inference produces empty sequences

The model essentially learns to produce the blank token $\varepsilon$ in the
first step, and then copy the latest token that is fed into the decoder as
input. During training, that input is indeed the target token. At inference,
however, it is the model’s prediction from the previous timestep. Since the
first prediction is $\varepsilon$, the model will continue and predict the
blank token until the end of the sequence.

### This exploit is not unique to AXE

AXE is not the only latent alignment objective that the model can “game” when
coupled with teacher forcing. We would see a similar phenomenon if we were to
use CTC with a longer prediction sequence; for example, if we doubled the
prediction length Libovický and Helcl (2018) and applied a version of teacher
forcing that feeds each target token twice in a row. In fact, every latent
alignment objective that can align a prediction $p_{j}$ with a target $y_{i}$
where $i<j$ will be subject to this exploit, and allow a model trained with
teacher forcing to glimpse into the future.

### Restricting AXE to causal alignments leads to the trivial alignment

We further limit AXE to allow only causal alignments, where a prediction
$p_{j}$ may only align with a target $y_{i}$ if $i\geq j$. After training with
the restricted objective, we observe that AXE selects the trivial alignment
($i=j$) in 98% of the validation set sentences, whereas the remaining 2%
contain only minor deviations from the trivial alignment, typically one
delimiter quickly followed by one clone prediction.

##  6 Conclusion

This work elaborates why latent alignment objectives are incompatible with
autoregressive models trained with teacher forcing. That said, teacher forcing
might not be the best way to train a machine translation model Bengio et al.
(2015); Lamb et al. (2016); Ghazvininejad et al. (2020b), and perhaps a future
alternative could reopen the discussion on applying latent alignment
objectives to autoregressive models.

## Acknowledgements

This work was supported in part by Len Blavatnik and the Blavatnik Family
foundation, the Alon Scholarship, and the Tel Aviv University Data Science
Center.

## References

  * Bengio et al. (2015) Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015.  Scheduled sampling for sequence prediction with recurrent neural networks.  In _Advances in Neural Information Processing Systems_ , volume 28. Curran Associates, Inc.
  * Cettolo et al. (2015) M. Cettolo, J. Niehues, S. Stüker, L. Bentivogli, and Marcello Federico. 2015\.  Report on the 11th iwslt evaluation campaign, iwslt 2014.
  * Ghazvininejad et al. (2020a) Marjan Ghazvininejad, Vladimir Karpukhin, Luke Zettlemoyer, and Omer Levy. 2020a.  Aligned cross entropy for non-autoregressive machine translation.  In _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 3515–3523. PMLR.
  * Ghazvininejad et al. (2020b) Marjan Ghazvininejad, Omer Levy, and Luke Zettlemoyer. 2020b.  Semi-autoregressive training improves mask-predict decoding.
  * Graves et al. (2006) Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006.  Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks.  In _Proceedings of the 23rd International Conference on Machine Learning_ , ICML ’06, page 369–376, New York, NY, USA. Association for Computing Machinery.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Lamb et al. (2016) Alex M Lamb, Anirudh Goyal ALIAS PARTH GOYAL, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. 2016.  Professor forcing: A new algorithm for training recurrent networks.  In _Advances in Neural Information Processing Systems_ , volume 29. Curran Associates, Inc.
  * Libovický and Helcl (2018) Jindřich Libovický and Jindřich Helcl. 2018.  End-to-end non-autoregressive neural machine translation with connectionist temporal classification.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3016–3021, Brussels, Belgium. Association for Computational Linguistics.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Saharia et al. (2020) Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. 2020.  Non-autoregressive machine translation with latent alignments.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1098–1108, Online. Association for Computational Linguistics.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Williams and Zipser (1989) R. J. Williams and D. Zipser. 1989.  A learning algorithm for continually running fully recurrent neural networks.  _Neural Computation_ , 1(2):270–280.


11institutetext:  1 Faculty of Mathematics and Computer Science, University of
Münster, Münster, Germany
2 Institute of Medical Physics and Biophysics, University of Münster, Münster,
Germany

# Single Image Estimation of Cell Migration Direction by Deep Circular
Regression

Lennart Bruns 11 Lucas Lamparter 22 Milos Galic 22 Xiaoyi Jiang 11

###### Abstract

In this paper we study the problem of estimating the migration direction of
cells based on a single image. To the best of our knowledge, there is only one
related work that uses a classification CNN for four classes (quadrants). This
approach does not allow detailed directional resolution. We solve the single
image estimation problem using deep circular regression with special attention
to cycle-sensitive methods. On two databases we achieve an average accuracy of
$\sim$17∘, which is a significant improvement over the previous work.

###### Keywords:

Single cell migration migration direction single image estimation circular
data deep circular regression

##  1 Introduction

Cell images are widely used in biomedical research. Information extraction
from cell images is a crucial prerequisite for answering challenging
biomedical questions. Important cell image analysis tasks include
segmentation, classification, and tracking [6, 17]. In this work we study the
problem of single image estimation of cell migration direction (SIECMD), i.e.
estimating the future direction of cell migration from a single image of a
cell, see Figure 1 for examples.

Cell migration direction can be easily obtained from cell tracking in videos
[7]. In contrast, performing this estimation by using a single image is much
more challenging. There are, however, biological indications that this
estimation is possible: A cell’s first reaction to a chemical cue, which
promotes migration towards or away of the gradient, is to polarize and extend
protrusions in the direction of migration. The cell uses these protrusions to
exert forces to the substrate, causing the cell to migrate in the direction of
these protrusions, see [12] for a detailed explanation. The study [4] shows
that the migration direction (of an adhering mammalian cell) is influenced by
its enforced polarity, i.e. a large front and a small rear. Consequently,
there exists a clear relationship between the motility and shape of individual
migrating cells [8].

In this work we use deep circular regression to learn shape features for
SIECMD. A detailed prediction of motion direction from single images is of
relevance for various biomedical applications. One such example are
histological probes from biopsies or other human tissue. These samples can
only be obtained in small quantities and preclude live cell analysis. Here,
SIECMD may advance our understanding on single cell dynamics under
physiological and pathologcial (e.g. cancer) conditions. A second example,
where a prediction of motion analysis may be advantageous, is for high-
throughput screening. To probe the effectiveness of pharmaceutical compounds
on cell motility is time and data intensive. Replacing single cell tracking
data with kinetic information inferred from SIECMD may significantly augment
the throughout of such assays. Finally, the estimated migration could be even
used for applications beyond biomedical data. One possible application is to
complement the cellular Potts model (CPM), an established framework for cell
simulation originally proposed in [1] (a current toolbox can be found in
[15]). The estimated migration direction from SIECMD could be integrated into
CPM to enable a more robust simulation.

The task of SIECMD has been hardly studied so far. To our best knowledge, the
only relevant work is done by Nishimoto et al. [9]. It is based on a CNN of 14
layers designed for solving a 4-class classification problem (upper left,
upper right, lower left, lower right; i.e. one class for each quadrant). We
consider this formulation to be a major drawback of that work, as it
represents only a very coarse sampling of the entire space of directions (on a
unit circle) and thus does not allow for a more detailed directional
resolution. The thesis work [19] also considers SIECMD. It extends the
approach in [9] by a variable number of classes (instead of constant 4) that
is tuned during the training phase. However, this number is selected using
information from two consecutive images at $t$ and $t+1$, thus not applicable
to a single-image context. In addition, the estimated migration direction has
not been separately evaluated, but is only used to support cell tracking in
videos with low frame rate. Another single-image problem is that of
classifying the cell migration modes (continuous, discontinuous) [2].

We solve the problem of SIECMD using deep regression with special attention to
the circular nature of direction data. We present the details of our methods
in Section 2. The experimental results and discussions follow in Section 3.
Then, further discussion concludes the paper.

Figure 1: Example cell images with ground truth cell migration direction (red
arrow).

##  2 Methods

To design a deep neural network for estimating the cell migration direction we
need to specify several details (direction encoding, activation function, loss
function). For each of them there are various choices. In the following we
present these design options and a way of finding an optimal configuration. In
particular, dealing with directions (angles) means working with circular data.
Because of the special nature of circular data, traditional techniques meant
for Euclidean space may not be applicable. Instead, cycle-sensitive methods
are needed [5, 11].

###  2.1 Direction encoding

We consider two schemes of direction encoding: 1) an angle
$\alpha\in[0,2\pi)$. 2) a point on the unit circle $(x,y),\ x^{2}+y^{2}=1$.
The choice of encoding also determines the number of output neurons (1 or 2)
of a deep neural network for migration direction estimation.

###  2.2 Activation functions

ReLU is used for the activation function except the last (output) layer.
There, we define for the angle direction encoding the following activation
function:

$\varphi_{cyclic}(x)\ =\ x\bmod 2\pi$ (1)

For the circle direction encoding we consider two options: the identity
function and the Sigmoid function:

$\varphi_{identity}(x)\ =\ x;\quad\varphi_{sigmoid}(x)\ =\
\frac{e^{x}-1}{e^{x}+1}$ (2)

In the case of modulo operator and Sigmoid function the activation function
has the purpose of normalization (to $[0,2\pi)$ and $[-1,1]$, respectively).

###  2.3 Loss functions

For the angle direction encoding we define two loss functions together with
their quadratic variants to compare the network output and ground truth:

$\displaystyle\delta_{linear}(\alpha,\beta)$ $\displaystyle=$
$\displaystyle|\alpha-\beta|;\quad\delta_{linear}^{2}(\alpha,\beta)\ =\
(\delta_{linear}(\alpha,\beta))^{2}$ (3)
$\displaystyle\delta_{cyclic}(\alpha,\beta)$ $\displaystyle=$
$\displaystyle\min(|\alpha-\beta|,2\pi-|\alpha-\beta|)$ (4)
$\displaystyle\delta_{cyclic}^{2}(\alpha,\beta)$ $\displaystyle=$
$\displaystyle(\delta_{cyclic}(\alpha,\beta))^{2}$ (5)

The loss functions (4) and (5) take the circular nature of direction data into
account and are thus expected to perform better than the two in (3). In
addition, we define another loss function that is also cycle-sensitive:

$\delta_{cos}(\alpha,\beta)\ =\ -\cos(\alpha-\beta)$ (6)

For the circle direction encoding we define two simple loss functions:

$\displaystyle\delta_{dist}((x_{1},y_{1}),(x_{2},y_{2}))$ $\displaystyle=$
$\displaystyle|x_{1}-x_{2}|+|y_{1}-y_{2}|$ (7)
$\displaystyle\delta_{dist}^{2}((x_{1},y_{1}),(x_{2},y_{2}))$ $\displaystyle=$
$\displaystyle(\delta_{dist}((x_{1},y_{1}),(x_{2},y_{2})))^{2}$ (8)

While all the loss functions above are easy to understand, the one in (6) has
a mathematical interpretation in terms of maximum likelihood estimation. Given
a machine learning model (CNN in our case) $\alpha=y(\mathbf{w},x)$ with
parameters $\mathbf{w}$ for predicting an angle $\alpha$ for an input cell
image $x$, the uncertainty of the decision $\alpha$ can be modeled using a
probability distribution, particularly the normal distribution. For circular
data this corresponds to the von Mises distribution [5, 11]:

$\displaystyle p(t\ |\ \mathbf{w},x,\kappa)\ =\
\frac{e^{\kappa\cos(t-y(\mathbf{w},x))}}{2\pi I_{0}(\kappa)}$ (9)

where $\kappa>0$ is its concentration, parameter $I_{0}(\kappa)$ is the
modified Bessel function of the first kind of order 0. We use the training
data $\mathbf{x}=\\{\mathbf{x}_{1},\ldots,\mathbf{x}_{N}\\}$ with the related
ground truth $\mathbf{t}=\\{t_{1},\ldots,t_{N}\\}$ to determine the unknown
parameters $\mathbf{w}$ by maximizing the likelihood function:

$p(\mathbf{t}\ |\ \mathbf{w},x,\kappa)\ =\
\prod_{n=1}^{N}\frac{e^{\kappa\cos(t_{k}-y(\mathbf{w},x_{k}))}}{2\pi
I_{0}(\kappa)}\ =\ \frac{1}{\\{2\pi
I_{0}(\kappa)\\}^{N}}\prod_{n=1}^{N}e^{\kappa\cos(t_{k}-y(\mathbf{w},x_{k}))}$
(10)

To simplify the optimization, we use the log likelihood function:

$\displaystyle\ln p(\mathbf{t}\ |\ \mathbf{w},x,\kappa)\ =\
\kappa\sum_{n=1}^{N}\cos(t_{k}-y(\mathbf{w},x_{k}))-N\ln(2\pi I_{0}(\kappa))$
(11)

Maximizing the (log) likelihood function is thus equivalent to minimizing the
following error function:

$-\sum_{n=1}^{N}\cos(t_{k}-y(\mathbf{w},x_{k}))$ (12)

which exactly corresponds to the loss function (6).

###  2.4 Deep neural networks

In Sections 2.1–2.3 we described several encoding schemes, activation
functions (for the output layer), and loss functions to the studied. In total
there are nine combinations. We introduce a small-size CNN to probe the best
combination (configuration) and then only use this optimal configuration to
test the performance of larger networks. This probing CNN consists of two
convolutional layers followed by max pooling layers. The head of this model
consists of three dense (fully connected) layers ending with one or two
neurons, depending on the direction encoding (see Table 1).

Fine-tuning pre-trained neural networks to a specific task minimizes the
training effort and leads to good results [18, 20]. Therefore, we explore
several backbones of prominent neural networks (YOLOv8, ResNet50 [3], and
EfficientNet [13]). Of these networks, we have chosen the smallest
architecture in each case. To test these models for our task, we only used the
backbone (i.e. all layers up to first fully-connected layers). Our head was
adapted to the complexity of the foreign architectures (see Table 2). The
backbones of these networks were pre-trained on known datasets and therefore
only need minimal re-training. To do this, we first trained the entire model
for 10 epochs, then freeze the weights of the respective backbone and trained
the model for further 50 epochs.

Layer |  Output | # Parameters
---|---|---
Input layer |  (128, 128) | 0
Conv2D (5x5), activation=ReLU |  (124, 124, 16) | 416
MaxPooling2D |  (62, 62, 16) | 0
Conv2D (3x3), activation=ReLU |  (60, 60, 32) | 4640
MaxPooling2D |  (30, 30, 32) | 0
Flatten |  (28800) | 0
Dense, activation=ReLU |  (256) | 7373056
Dense, activation=ReLU |  (16) | 4112
Dense, activation=$\varphi$ |  ($\\#$) | $\\#$
Table 1: Probing CNN for finding the best configuration. Cell image size: $128\times 128$. Symbol $\\#$ indicates that the value there depends on the direction encoding. Layer |  Output | # Parameters
---|---|---
Backbone
GlobalAveragePooling2D |  (#) | 0
Dense, activation=ReLU |  (1024) | #
Dense, activation=ReLU |  (256) | 262400
Dense, activation=$\varphi$ |  ($\\#$) | $\\#$
Table 2: Fine-tuning architecture. Backbone refers to the foreign model that
is connected to our customized head with the fully connected layers. Symbol
$\\#$ indicates that the value there depends on the direction encoding or the
size of the previous layers.

###  2.5 Test-time augmentation

Since SIECMD is a challenging task, we also explore the potential of test-time
augmentation (TTA) for ensemble effect [10, 16]. For this propose multiple
images are randomly generated for an input cell image by rotations. The
migration direction estimated for each rotated image is corrected by the
applied rotation. For a total number $n$ of cell images (one original, $n-1$
rotated versions) we thus yield $n$ migration direction angles that are fused
to obtain the final result, see Figure 2 for an example. We applied a
heuristic averaging method: 1) We sort all predictions to receive a sorted
ascending list $\\{p_{1},...,p_{n}\\}$. 2) We calculate the distance
$d=p_{n}-p_{1}$. 3) We add $2\pi$ to the smallest prediction $p_{1}$ and start
with these new predictions at 1). Our goal is to find the list of predictions
with the smallest possible difference $d$. On that resulting list we finally
calculate the mean and then use the modulo operator to get the averaging
result $\in[0,2\pi)$.

Figure 2: TTA for migration direction estimation ($n=10$).

Dataset | Encoding | Activation | Loss | $\mathbf{E}_{deg}$ | $\pm$
---|---|---|---|---|---
NIH3T3 | 1N | $\varphi_{cyclic}$ | $\delta_{linear}$ | 97.81 | 0.51
$\varphi_{cyclic}$ | $\delta_{linear}^{2}$ | 97.82 | 0.59
$\varphi_{cyclic}$ | $\delta_{cyclic}$ | 97.86 | 0.54
$\varphi_{cyclic}$ | $\delta_{cyclic}^{2}$ | 97.83 | 0.53
$\varphi_{cyclic}$ | $\delta_{cos}$ | 66.54 | 12.10
2N | $\varphi_{identity}$ | $\delta_{dist}$ | 33.27 | 1.79
$\varphi_{identity}$ | $\delta_{dist}^{2}$ | 34.29 | 2.20
$\varphi_{sigmoid}$ | $\delta_{dist}$ | 30.53 | 2.21
$\varphi_{sigmoid}$ | $\delta_{dist}^{2}$ | 30.51 | 0.86
U373 | 1N | $\varphi_{cyclic}$ | $\delta_{linear}$ | 87.05 | 1.73
$\varphi_{cyclic}$ | $\delta_{linear}^{2}$ | 87.06 | 1.77
$\varphi_{cyclic}$ | $\delta_{cyclic}$ | 86.97 | 1.71
$\varphi_{cyclic}$ | $\delta_{cyclic}^{2}$ | 87.11 | 1.85
$\varphi_{cyclic}$ | $\delta_{cos}$ | 33.79 | 3.76
2N | $\varphi_{identity}$ | $\delta_{dist}$ | 21.59 | 1.27
$\varphi_{identity}$ | $\delta_{dist}^{2}$ | 22.70 | 1.63
$\varphi_{sigmoid}$ | $\delta_{dist}$ | 22.92 | 2.15
$\varphi_{sigmoid}$ | $\delta_{dist}^{2}$ | 21.90 | 1.05

Table 3: Migration direction estimation by the probing CNN. The column
”Encoding” indicates one or two output neurons. The column $\mathbf{E}_{deg}$
shows the mean deviation in degrees (13)

with the standard deviation in column ”$\pm$”.

Dataset |  Backbone | Pre-training | $\mathbf{E}_{deg}$ | $\pm$
---|---|---|---|---
NIH3T3 |  YOLOv8 | coco | 23.77 | 0.64
EfficientNet | imagenet | 17.27 | 2.26
Resnet50 | imagenet | 20.07 | 0.53
U373 |  YOLOv8 | coco | 18.19 | 2.73
EfficientNet | imagenet | 16.93 | 3.69
Resnet50 | imagenet | 23.77 | 4.59

Table 4: Migration direction estimation by large neural networks using the
optimal configuration derived from Table 5.

Dataset |  Testset Rotation | $\mathbf{E}_{deg}$ | $\pm$
---|---|---|---
NIH3T3 |  Original image | 17.27 | 2.26
\+ 1 rotated version | 17.41 | 2.48
\+ 5 rotated versions | 17.33 | 1.88
\+ 9 rotated versions | 17.24 | 2.15
\+ 13 rotated versions | 17.18 | 2.17
U373 |  Original image | 16.93 | 3.69
\+ 1 rotated version | 17.59 | 2.95
\+ 5 rotated versions | 17.77 | 2.53
\+ 9 rotated versions | 17.57 | 2.86
\+ 13 rotated versions | 17.47 | 2.59

Table 5: Migration direction estimation by TTA (EfficientNet backbone, pre-
training on imagenet).

##  3 Experimental validation

###  3.1 Datasets

Our performance evaluation is based two datasets used in [9] that contain
time-lapse phase-contrast microscopic images of cell migration in different
ways.

  * •

NIH3T3. Fibroblasts were seeded onto dishes made of glass. Following a night
of incubation, an inverted microscope was used to track the migration of the
cells. Phase-contrast footage was captured using CCD cameras.

  * •

U373. This dataset was part of the ISBI (International Symposium on Biomedical
Imaging) cell tracking challenge in 2015 [14].

Example images can be found in Figure 1, where the left (right) block shows
images from NIH3T3 (U373). The NIH3T3 (U373) dataset contains 830 (1011)
images with 170 (260) distinct ground truth angles. Note that in [9] results
are also reported on a third dataset hTERT-RPE1. Although available to us,
however, this dataset was already in a form prepared for the 4-class problem
so that no angle ground truth could be obtained. For this reason this dataset
was not used for our study.

The tracking information given in the datasets consists of the position
$(x,y)$ for each cell in the frames. With this information, it was possible to
obtain a ground truth angle for all cells that migrated a distance of more
than 5$\mu$m during image acquisition.

We multiplied the training data using augmentation. This includes a random
combination of rotation $\in[0,2\pi)$, $x$ and $y$ shift $\in[-0.2,0.2]$
percent of the image size, scaling with factor $\in[-0.1,0.1]$, and vertical
and horizontal mirroring.

###  3.2 Experimental results

We used a 4-fold validation, where each fold was randomly generated and split
into 40% training, 10% validation and 50% test. For the evaluation, we used
the mean deviation over all predictions $\alpha_{i}$ and corresponding ground
truth $\beta_{i}$ as a metric for the migration direction estimation accuracy
$\mathbf{E}_{deg}$:

$\displaystyle\mathbf{E}_{deg}$ $\displaystyle=$
$\displaystyle\frac{1}{n}\sum_{i=1}^{n}\min(|\alpha_{i}-\beta_{i}|,\
2\pi-|\alpha_{i}-\beta_{i}|)$ (13)

The performance of the probing CNN is shown in Table 5. When considering both
datasets, the configuration: circle encoding, activation function
$\varphi_{sigmoid}$, loss function $\delta_{dist}^{2}$, can be considered
optimal, as marked in color. The loss function $\delta_{cos}$ has a sound
mathematical interpretation, but does not perform best (for these two
datasets). The reason may be the discontinuous nature of the cyclic activation
function in (1). It is not compatible with (stochastic) gradient descent-type
optimization algorithms and needs specialized handling.

Then, we use this optimal configuration to test the performance of three
networks YOLOv8, ResNet50, and EfficientNet. The results are given in Table 5.
For both datasets, the EfficientNet model performs best in terms of mean
accuracy and standard deviation. Using this model we are able to achieve a
mean accuracy of $\sim$17∘ (marked in color) on both datasets.

Finally, using EfficientNet we further study the performance of TTA for
$n=2,6,10,14$, see Table 5. In term of mean and standard deviation, NIH3T3
benefits from TTA. For U373 only the standard deviation declines when using
TTA. We estimate the maximum estimation error by (mean + 3 $\times$ standard
deviation) for the case without & with TTA ($n=14$). This results in 24.05∘ &
23.69∘ for NIH3T3 and 28.00∘ & 25.24∘ for U373, respectively. In this regard
TTA is advantageous for U373. Overall, it can be concluded that TTA has a
positive effect.

###  3.3 Comparison with previous work [9]

The major drawback of the only related work for solving SIECMD [9] is the very
coarse sampling of directions (on a unit circle) in four discrete classes
only, which does not allow for a more detailed directional resolution. Given
the predicted class, without extra knowledge, the best possible angle output
is the one in the middle of the related quadrant, e.g. $\frac{1}{4}\pi$ if the
first quadrant is predicted. This output causes an average inaccuracy relative
to the (unknown) ground truth:

  * •

$\frac{1}{8}\pi$: if the predicted quadrant is correct.

  * •

$\frac{1}{2}\pi$: if the predicted quadrant is a direct neighbor of the
correct quadrant.

  * •

$\frac{7}{8}\pi$: if the predicted quadrant is not a direct neighbor of the
correct quadrant.

Here we assume equal probability of an angle within the ground truth quadrant.
The maximal inaccuracy is $\frac{1}{4}\pi$, $\frac{3}{4}\pi$, and $\pi$,
respectively. In [9] a classification accuracy of 87.89% (NIH3T3) and 81.76%
(U373) is reported. Since no error percentage for the incorrect quadrants is
specified, we exemplarily use $1/3$ of the remainder percentage for each of
the incorrect quadrants to obtain the average inaccuracy:

$\displaystyle\mbox{NIH3T3}:$ $\displaystyle
87.89\%\cdot\frac{1}{8}\pi+4.04\%\cdot(\frac{1}{2}\pi+\frac{1}{2}\pi+\frac{7}{8}\pi)=18.56\%\cdot\pi\
(33.41^{\circ})$ $\displaystyle\mbox{U373}:$ $\displaystyle
81.86\%\cdot\frac{1}{8}\pi+6.05\%\cdot(\frac{1}{2}\pi+\frac{1}{2}\pi+\frac{7}{8}\pi)=21.58\%\cdot\pi\
(38.84^{\circ})$

A comparison with Table 5 clearly shows an improved accuracy of angle
estimation by our approach. Similarly, the average maximal inaccuracy can be
estimated by:

$\displaystyle\mbox{NIH3T3}:$ $\displaystyle
87.89\%\cdot\frac{1}{4}\pi+4.04\%\cdot(\frac{3}{4}\pi+\frac{3}{4}\pi+\pi)=29.04\%\cdot\pi\
(52.28^{\circ})$ $\displaystyle\mbox{U373}:$ $\displaystyle
81.86\%\cdot\frac{1}{8}\pi+6.05\%\cdot(\frac{1}{2}\pi+\frac{1}{2}\pi+\frac{7}{8}\pi)=31.05\%\cdot\pi\
(55.89^{\circ})$

A comparison with our results on maximum estimation error (see Section 3.2)
shows superior performance over the previous work [9] in terms of maximal
inaccuracy.

##  4 Conclusion

In this paper have we studied the problem of estimating the migration
direction of cells based on a single image. Our solution, based on deep
circular regression with special attention to cycle-sensitive methods, has
achieved an average accuracy of $\sim$17∘ on two datasets. This performance
represents a significant improvement over the only previous work on this
problem. The outcome of this work provides foundation for the realization of
applications such as those outlined in the introduction, which were previously
not possible. The focus of our future work will therefore be on investigating
such applications.

## Acknowledgments

This work was supported by the Deutsche Forschungsgemeinschaft (DFG): CRC 1450
– 431460824 (to X.J.) and GA2268/4-1 (to M.G.).

## References

  * [1] Graner, F., Glazier, J.A.: Simulation of biological cell sorting using a two-dimensional extended potts model. Physical Review Letters 69, 2013–2016 (1992)
  * [2] Gupta, A., Larsson, V., Matuszewski, D.J., Strömblad, S., Wählby, C.: Weakly-supervised prediction of cell migration modes in confocal microscopy images using bayesian deep learning. In: 17th IEEE International Symposium on Biomedical Imaging (ISBI). pp. 1626–1629 (2020)
  * [3] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770–778 (2016)
  * [4] Jiang, X., Bruzewicz, D.A., Wong, A.P., Whitesides, G.M.: Directing cell migration with asymmetric micropatterns. PNAS 102, 975–978 (2005)
  * [5] Lee, A.: Circular data. Wiley Interdisciplinary Reviews: Computational Statistics 2, 477–486 (2010)
  * [6] Liu, Z., Jin, L., Chen, J., Fang, Q., Ablameyko, S., Yin, Z., Xu, Y.: A survey on applications of deep learning in microscopy image analysis. Computers in Biology and Medicine 134, 104523 (2021)
  * [7] Maška, M., et al.: The cell tracking challenge: 10 years of objective benchmarking. Nature Methods 20, 1010–1020 (2023)
  * [8] Mogilner, A., Keren, K.: The shape of motile cells. Current Biology 19, R762–R77 (2009)
  * [9] Nishimoto, S., Tokuoka, Y., Yamada, T.G., Hiroi, N.F., Funahashi, A.: Predicting the future direction of cell movement with convolutional neural networks. PLoS ONE 14(9), e0221245 (2019)
  * [10] Oza, P.R., Sharma, P., Patel, S.: Breast lesion classification from mammograms using deep neural network and test-time augmentation. Neural Computing and Applications 36(4), 2101–2117 (2024)
  * [11] Pewsey, A., García-Portugués, E.: Recent advances in directional statistics. TEST 30, 1–58 (2021)
  * [12] Ridley, A.J., Schwartz, M.A., Burridge, K., Firtel, R.A., Ginsberg, M.H., Borisy, G., Parsons, J.T., Horwitz, A.R.: Cell migration: integrating signals from front to back. Science 302, 1704–1709 (2003)
  * [13] Tan, M., Le, Q.V.: Efficientnetv2: Smaller models and faster training. In: ICML. Proceedings of Machine Learning Research, vol. 139, pp. 10096–10106. PMLR (2021)
  * [14] Ulman, V., et al.: An objective comparison of cell-tracking algorithms. Nature Methods 14(12), 1151 (2017)
  * [15] Wortel, I., Textor, J.: Artistoo, a library to build, share, and explore simulations of cells and tissues in the web browser. eLife 10, e61288 (2021)
  * [16] Xu, C., Wen, Z., Liu, Z., Ye, C.: Improved domain generalization for cell detection in histopathology images via test-time stain augmentation. In: MICCAI. Lecture Notes in Computer Science, vol. 13432, pp. 150–159 (2022)
  * [17] Xu, J., et al.: Deep learning in cell image analysis. AAAS Intelligent Computing 2022, Article ID 9861263 (2022)
  * [18] Yu, X., Wang, J., Hong, Q., Teku, R., Wang, S., Zhang, Y.: Transfer learning for medical images analyses: A survey. Neurocomputing 489, 230–254 (2022)
  * [19] Zhang, X.: Cell Tracking at Low Frame Rate using Deep Learning and Bayesian Integration. Ph.D. thesis, Clemson University (2022)
  * [20] Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H., He, Q.: A comprehensive survey on transfer learning. Proceedings of the IEEE 109(1), 43–76 (2021)