11institutetext: School of Computing, University of Leeds, United
Kingdom.22institutetext: Wicking Dementia Research and Education Centre,
College of Health and Medicine, University of Tasmania, Australia.
33institutetext: School of Medicine, College of Health and Medicine,
University of Tasmania, Australia.
33email<EMAIL_ADDRESS>33email: {sharn.perry,
<EMAIL_ADDRESS>
# Of Mice and Pose: 2D Mouse Pose Estimation from Unlabelled Data and
Synthetic Prior

Jose Sosa 11 Sharn Perry 22 Jane Alty 2233 David Hogg 11

###### Abstract

Numerous fields, such as ecology, biology, and neuroscience, use animal
recordings to track and measure animal behaviour. Over time, a significant
volume of such data has been produced, but some computer vision techniques
cannot explore it due to the lack of annotations. To address this, we propose
an approach for estimating 2D mouse body pose from unlabelled images using a
synthetically generated empirical pose prior. Our proposal is based on a
recent self-supervised method for estimating 2D human pose that uses single
images and a set of unpaired typical 2D poses within a GAN framework. We adapt
this method to the limb structure of the mouse and generate the empirical
prior of 2D poses from a synthetic 3D mouse model, thereby avoiding manual
annotation. In experiments on a new mouse video dataset, we evaluate the
performance of the approach by comparing pose predictions to a manually
obtained ground truth. We also compare predictions with those from a
supervised state-of-the-art method for animal pose estimation. The latter
evaluation indicates promising results despite the lack of paired training
data. Finally, qualitative results using a dataset of horse images show the
potential of the setting to adapt to other animal species.

###### Keywords:

Self-Supervised Pose Estimation Synthetic Mouse.

##  1 Introduction

The study of neurodegenerative human diseases, such as Alzheimer’s disease
[15, 51], Parkinson’s disease [28], and Amyotrophic Lateral Sclerosis (ALS)
[46], usually involves using animal models. Mice are the preferred and most
extensively utilised animals for such studies because of their genomics
similarity with humans and the accumulated knowledge on manipulating their DNA
[11]. Due to this tight relationship between mice and the ongoing research on
neurodegenerative human diseases, developing tools to observe, describe, and
measure mouse behaviour has become crucial [37].

Some years ago, prior to the adoption of computer vision techniques, making
such measurements meant tons of manual labour [42, 43]. For example, if
someone wanted to measure the position of the mouse’s limbs. It implies
recording the animal, looking at each video frame, and manually identifying
each required body part. Then, it is evident that manual inspections on large
videos can be time-consuming and lead to observation errors. Early
computational approaches attempt to minimise human intervention in analysing
animal recordings. Some tools involve placing physical markers on the animal’s
body or require painting the body parts to track [2, 56]. Apparent limitations
of these techniques are that the physical markers can interfere with the
animal’s behaviour, and the information that can be extracted is inherently
limited by the positioning of the markers or the painted areas. Other
approaches use sophisticated and expensive equipment to acquire particular
images, which results in costly experiments and problems for deployment and
replication [6, 26, 12].

Newer computer vision tools for tracking body parts of animals
111https://mousespecifics.com/digigait/222 https://www.noldus.com/catwalk-xt
become less dependent on physical markers, i.e. markerless. Unfortunately,
these tools still needed considerable human intervention for pre-processing
and post-processing video data. Supervised deep learning approaches have
recently become state-of-the-art for pose estimation and tracking of humans
and animals [36, 44, 14]. Performance of these techniques often depends on the
amount and variability of annotated data for training, which is hard to obtain
for some animal species. Thus, there remains an urgent need to develop methods
for tracking animal pose that require minimal human effort in training for a
new animal domain and operational use. This can be achieved by reducing the
need for manual pose annotation of images.

In this paper we tackle the challenging task of predicting 2D mouse poses from
unlabelled images. Different from previous deep learning approaches that
generally rely on fully supervised frameworks, we adopt a self-supervised 2D
pose estimator from the human domain [18]. This method utilises a GAN
architecture to learn 2D human poses. During training, it assumes the
availability of unlabelled images and an unpaired prior of 2D pose
annotations, generally from the same dataset. Our proposal relaxes much more
the assumptions about data by building the needed prior of 2D poses using data
generated from a 3D model of a generic mouse [4]. Evidently, incorporating
synthetic data also provides more flexibility to train the model with entirely
unlabelled datasets, which is common for many animal recordings outside of
computer vision. Furthermore, our method shows promising results in generating
2D poses for other types of animals, e.g. horses. This demonstrates the
viability of our approach to be rapidly deployed to different domains without
the burden of annotating data.

##  2 Related Work

###  2.1 Deep Learning Methods for Animal Pose Estimation

Analogous to the definition of human pose estimation [32], animal pose
estimation refers to the task of estimating the geometrical configuration of
body parts of an animal. This problem has gained increasing attention because
of research applications in many different disciplines, including Biology,
Zoology, Ecology, Biomechanics [53] and Neuroscience [37]. Compared with human
pose estimation, it is still relatively under-explored, principally due to the
variability of animal species, and the need for species-specific labelled
datasets. Nevertheless, a lot of effort has gone into developing and adapting
deep learning models to estimate 2D and 3D animal pose, exploiting
similarities between many species of animal. For example, monkeys [59, 40, 1]
share similar skeletal structure with humans. Large quadrupeds, such as farm
animals [8, 35, 50, 48] and dogs [3, 54, 24] present similarities between
their skeletal forms.

Automatic 2D pose estimation has also been applied successfully on smaller
animal species such as mice. As with larger animals, deep learning methods for
pose estimation have been based mostly on supervised methods developed for
human pose estimation. Their performance is therefore limited by the
availability and correctness of annotated data. For example, DeepLabCut (DLC)
[36] adapts a pretrained ResNet with deconvolutional layers [17] to estimates
the 2D pose of small animals under laboratory conditions, such as mice and
flies. LEAP [44] also uses an earlier model from the human pose estimation
domain [57] to solve the same task. DeepPoseKit [14] employs a similar method
to estimate 2D animal pose. It uses a network architecture that improves the
processing speed based on fully convolutional densenets [16, 19] and stacked
hourglass modules [41]. More recently, OptiFlex [31] exploits the temporal
information in video data by incorporating flowing convnets [45] into their
network architecture. They report similar performance to previous methods [36,
44, 14] on estimating the pose of small animals, e.g. mice, fruit flies, and
zebrafish.

Perhaps the most popular of these approaches is DeepLabCut. Many subsequent
methods adopt it to estimate not only mouse pose, but also pose for a wide
variety of other animal species [52, 23, 47, 27, 58, 30, 10]. A common feature
of DeepLabCut, DeepPoseKit, LEAP, and OptiFlex is their reliance on manual
annotation of pose in multiple video frames. Even though they normally provide
a Graphical User Interface (GUI) for doing the annotation, the process is
still time consuming, error prone, and requires specialised knowledge to infer
pose correctly. Futhermore, the number of frames to annotate for good
generalisation is hard to predict and therefore ultimately determined
empirically. In contrast, through adapting a recent self-supervised approach
from the human domain, we completely remove the need for manual annotation,
making training and testing more straightforward.

###  2.2 Animal Pose Estimation with Synthetic Data

One alternative to avoid manual annotation for training deep learning methods
for animal pose estimation is the use of synthetic data. Using an artificial
animal model allows producing many synthetic images and their corresponding
annotations with less time and effort than manually annotating actual data
[4]. In this context, Mu et al. [38] proposes a semi-supervised pose-
estimation framework trained in a supervised fashion using synthetically
rendered images and ground truth pose annotations from 3D Computer-Aided
Design (CAD) models. Then, they perform self-supervised domain adaption with a
small portion of actual data to minimise the domain gap. They successfully
estimate 2D poses for large animals with similar skeletal structures, such as
tigers, horses, and dogs. Some other works relying on synthetic data also
focus on the domain adaptation process after learning the animal pose with
synthetic data under supervised paradigms [29, 20]. We adopt a related
approach to [38] by using an existing 3D geometric mouse model [4], except
that we do not use rendered images as in supervised settings. We only utilise
the synthetically generated 2D poses as a prior for training the method. In
particular, we use this prior on 2D poses within a GAN framework that allows
our whole model to learn poses not necessarily appearing in the prior,
eliminating the need for domain adaptation as in [38, 29, 20].

Synthetic data also plays a significant role in learning more complex forms of
3D animal poses. For instance, [62] inspired by the success of human shape
models, SMPL [33] generates data from toy figurines of animals to learn
statistical shape models (SMAL). Later, [61] propose SMALR, which is an
extension of the previous SMAL model. It introduces a regularisation for the
deformation of the animal shape to make it appear more detailed and realistic.
Subsequent work has [3, 49, 60] adapted the SMAL model to work with particular
animal species like dogs and zebras. In contrast to learning to fit 3D shape
models from 3D scans, other approaches explore the possibility of learning 3D
animal models from less complex representations, like multi-view 2D images, or
user-clicked 2D images [7, 13, 22]. However, the final shape representation of
those models is less realistic and detailed than those produced using SMAL or
SMALR. These methods have produced 3D shape models for various animal species,
typically focused on large quadrupeds like tigers, dogs, and zebras.
Unfortunately, creating sophisticated models for all animal species is still
impractical.

Bolaños [4] has taken inspiration from previous synthetic models of large
animals to develop a similar model for mice. This 3D CAD model simulates semi-
random behavioural patterns from real mice and incorporates the 3D structure
of bones and joints. The model has successfully created training data for
famous supervised 2D and 3D mouse pose estimation approaches [36, 39].
Nevertheless, there is still an unexplored opportunity to utilise the same
model to generate data for training pose estimation models with lower levels
of supervision. We demonstrate this by relying on a recent self-supervised
method that learns to estimate 2D human poses solely from unlabelled images
and a prior on unpaired 2D poses. We follow the same idea, but instead of
taking the unpaired pose annotations from the dataset to build the prior, we
generated them with a 3D mouse model [4]. Note that we do not utilise paired
synthetic images and pose annotations like in previous works [38, 29, 20, 4],
we discard the synthetic images and only use synthetic 2D poses. This means
that our model is trained using actual unlabelled images and a smaller set of
artificially generated 2D poses.

##  3 Method

Figure 1: 2D pose estimator. We use a self-supervised 2D pose estimator from
the human domain, which we adapt to work with mice. Differently to the
original implementation, we build prior of 2D poses using synthetic data from
a 3D model of a generic mouse.

Our method produces a mapping from full body images to the 2D pose of a mouse,
as shown in Fig.1. The pose is represented as an articulated tree structure of
2D line segments corresponding to the parts of the body such as snout, tail,
hind limbs, and forelimbs. The method extends the self-supervised approach of
[18], which estimates human 2D pose. This 2D pose estimator learns from
unlabelled images and uses a set of unpaired 2D poses as an empirical prior.
This removes any dependence on paired annotated data. However, the method
requires a set of manual 2D pose annotations for a subset of images from the
dataset, albeit the pairing is discarded. We adapt this approach by changing
the pose topology to a mouse model. We also generate an empirical prior for 2D
mouse pose by projecting from an existing 3D mouse model, which removes the
need for manual pose annotation altogether.

The pose-estimator is obtained by training a conditional auto-encoder to map
from an image $x$, depicting a mouse, to a reconstructed image $x^{\prime}$
that is as similar as possible. The synthesis of the output image is
conditioned on an auxiliary mouse image $y$ depicting a fixed pose. The auto-
encoder has a bottleneck that encodes the 2D pose as a set of joint positions
$v$. Once trained, our pose predictor is the initial encoder from this
network, which maps from an input image to a 2D pose. This mapping is in two
steps, consisting of a Convolutional Neural Network (CNN) $\Phi$ mapping from
the image $x$ to a skeleton image $s$, followed by a second CNN $\eta$ mapping
from the skeleton image $s$ to the 2D pose $v$. The decoder mapping from the
2D pose $v$ to the output image $x^{\prime}$ is also in two steps, consisting
of a differentiable function $\beta$ which maps the 2D pose $v$ to a skeleton
image $s^{\prime}$; and a CNN $\Psi$ mapping from the skeleton image
$s^{\prime}$ to the output $x^{\prime}$. The second mapping $\Psi$ takes an
auxiliary image $y$ as an additional input to compensate for the missing
appearance information in $s^{\prime}$.

We train the model with a dataset of images $\\{x_{1}\cdots x_{N}\\}$,
depicting mice in different poses, and our empirical prior of 2D poses. We use
a similar loss function as in [18], which contains three terms. The first
penalises the difference between the generated image $x^{\prime}$ and the
input $x$ via a perceptual loss. The second term is a regression loss to
evaluate the mapping from skeleton image $s$ to the 2D joint positions in $v$.
The third term is an adversarial loss to assess the authenticity of the
skeleton images generated in the encoder. In the following sections we provide
details on the components of the model, the empirical prior, loss function,
and training.

The whole pipeline for the conditional auto-encoder is as follows:

$x=\Psi(\beta(v)\circ\eta(s)\circ\Phi(x),y)$ (1)

We can see the mapping as an autoencoder from input image $x$ to output image
$x^{\prime}$ in which the 2D pose $v$ emerges as an intermediate
representation. In training the network, a perceptual loss [21] compares each
input image $x$ with the reconstructed image $x^{\prime}$:

$\mathcal{L}_{perc}=\frac{1}{N}\sum_{i=1}^{N}\|\Gamma(x^{\prime}_{i})-\Gamma(x_{i})\|_{2}^{2}$
(2)

where $\Gamma$ is a pre-trained VGG network [55] with the classification stage
removed to utilise the final feature encoding.

A CNN serves as the discriminator network $D$, which outputs a probability
that an input skeleton image comes from the prior distribution of skeleton
images. Thus, $D$ measures the extent to which a skeleton image $s$ looks like
an authentic skeleton image from the empirical prior distribution. Note that
contrary to [18], our prior $\\{\hat{v}_{j}\\}^{M}_{j=1}$ is synthesised by
projecting from a 3D mouse model and does not require manual annotation of
poses. We obtain the skeleton images $\\{\hat{s}_{j}\\}_{j=1}^{j=M}$ via
$\beta$, i.e. $\\{\hat{s}_{j}=\beta({\hat{v}_{j}})\\}^{M}_{j=1}$, then we
compare this distribution $p_{data}(\hat{s})$ with the distribution
$p_{data}(s)$ from the predicted skeleton images
$\\{s_{i}=\Phi({x_{i}})\\}^{N}_{i=1}$ by means of the adversarial loss [34]:

$\mathcal{L}_{D}=\frac{1}{M}\sum_{j=1}^{M}D({\hat{s}}_{j})^{2}+\frac{1}{N}\sum_{i=1}^{N}(1-D(s_{i}))^{2}$
(3)

Finally, we derive a loss from $\eta$ and $\beta$, which combines the 2 terms
as follows:

$\mathcal{L}_{\eta}=\|\eta(\hat{s})-\hat{v}\|^{2}+\lambda\|\beta(\eta(s))-s\|^{2}$
(4)

The first term uses unpaired 2D poses from the prior, while the second one
utilises the pose on the predicted skeleton image $s$. The last term ensures
that the network learns poses that appear on the training images but not
necessarily on the prior. The balancing coefficient $\lambda$ is set to $0.1$
in our experiments.

####  3.0.1 2D synthetic prior.

We entirely generate the 2D pose prior required for the discriminator $D$
using synthetic data. In particular, we adopt a synthetic 3D model of a mouse
[4]. This animated mouse model simulates synthetic behavioural data using
animation and semi-random joint movements. We keep the original joint-
constrained movements of the freely moving mouse model. We animate and render
333 We use Blender to make the videos and extract the 2D poses from the mouse
model. the different scenes with the synthetic model and extract the 2D
coordinates of 18 joints on the body of the mouse: Snout, Vertebral column
base and end (VB and VE), three points located along the tail (TB, TM, and
TE), left/right elbows (LE and RE), left/right knees (LK and RK), and two
points (tip and top) for each left/right fore and hind limbs (LFP-/+, RFP-/+,
LHP-/+, RHP-/+). Note that this notation will be used through the paper.
Finally, we use those joint positions to create their respective skeleton
image, as shown in Fig.1. Overall, our prior consists of 15,408 different 2D
poses transformed into skeleton images.

####  3.0.2 Training.

Following [18] we use a perceptual loss $\mathcal{L}_{perc}$ (2), an
adversarial loss $\mathcal{L}_{D}$ (3), and a regression loss (4) in training
the convolutional networks $\Phi$, $\eta$, and $\Psi$. Note that $\beta$ is
not a learnable function. The overall loss $\mathcal{L}$ is given by:

$\mathcal{L}=\mathcal{L}_{D}+\mathcal{L}_{\eta}+\mathcal{L}_{perc}$ (5)

We train the pose estimator using unlabelled images. In particular, each batch
is formed by randomly sampling images $(x,y)$, and a random sample $\hat{v}$
from the synthetic 2D poses, which is then transformed to skeleton image
$\hat{s}$. The input images $x$ and $y$ were resized to $128\times 128$
pixels. We set the batch size to 32 and use Adam optimiser [25] with a
learning rate of $2\times 10^{-4}$, $\beta_{1}=0.5$, and $\beta_{2}=0.999$.
Unlike [18] who use a pretrained $\eta$, we train all the neural networks
$\Phi$, $D$, $\eta$, and $\Psi$ from scratch by optimising the loss function
in Equation 5. During testing, we only rely on the trained networks $\Phi$ and
$\eta$, to map from an input image to a 2D pose. Specifically, we input the
image $x$ through $\Phi(x)$ to obtain the skeleton image $s$, and then use
this with $\eta(s)$ to get the final 2D pose $v$.

##  4 Experiments

####  4.0.1 Dataset.

Our dataset contains images from $40$ videos of rodent models of ALS of
different genotypes444All the mice appearing in the recordings were bred and
maintained at the Univeristy of Tasmania.. Each video has around $13,120$
frames/images, with an original size of $658\times 190$ pixels. We use half of
the available videos to get the training images, and reserve the other half
for evaluation purposes.

Acquisition details. The recordings were made using the DigigaitTM apparatus,
which consists of a transparent treadmill and a camera placed underneath. Mice
at both 4 and 16 weeks of age were first acclimatised in the apparatus and
then encouraged to run on the treadmill at $10cm/s$, $20cm/s$ and $30cm/s$ for
a minimum of 10 seconds. The camera captures the mice on video as they move on
the treadmill. Mice were gently encouraged to run by taps to their rear by the
experimenter if needed. At the end of the trial, the mice were returned to
their home cage. The average duration of each video is $80$ seconds, i.e. most
of the mice ran for at least 20 seconds at each speed with 10 seconds
transitions between speeds without running.

####  4.0.2 Results.

Given an unlabelled image depicting a mouse, our trained model produces a 2D
representation of the mouse pose composed of 18 joint positions. Fig.2 shows
some of those predicted 2D poses. Since our dataset does not contain
annotations for the joint positions, we manually annotated 2D poses for some
images in the test videos to provide ground truth for a quantitative measure
of prediction performance. We compare pose predictions with ground-truth on
this test set using the Mean Per Joint Position Error (MPJPE). The first row
of Table 1 shows the MPJPE in pixels between the predicted positions for each
of the joints composing the mouse pose and their respective ground truth
annotations. MPJPE is reported w.r.t to the original image dimensions:
$658\times 190$ pixels.

Figure 2: Estimated 2D poses using our method. During training we use real
images and the synthetic pose prior: RI + SP.

In addition to the previous experiment, we train and evaluate our model using
synthetic images and synthetic unpaired poses (SI + SP). Note that the
synthetic 2D poses on the prior are not annotations of the training images. We
train the model with different sequences of images synthetically generated
from the 3D mouse model and test it using a different set of synthetic images.
We use the 2D ground truth annotations for 18 joint positions extracted from
the mouse model and compare them with the predicted poses. We report the MPJPE
for each joint position in the second row of Table 1 and a few visualisations
of results in Fig.3.

Table 1: MPJPE of predicted poses. RI + SP denotes use of the method trained with Real Images and Synthetic Prior. SI + SP denotes use of the method trained with Synthetic Images and Synthetic Prior. Joints | Snout | VCB | VCE | TB | TM | TE | RE | RFP- | RFP+ | Avg.
---|---|---|---|---|---|---|---|---|---|---
| LE | LFP- | LFP+ | RK | RHP- | RHP+ | LK | LHP- | LHP+ |
RI + SP | 13.4 | 8.0 | 5.6 | 15.2 | 17.8 | 31.8 | 14.7 | 15.8 | 14.8 |
| 12.7 | 10.5 | 14.2 | 7.6 | 21.1 | 11.5 | 14.9 | 11.7 | 11.9 | 14.1
SI + SP | 5.9 | 4.0 | 3.0 | 3.7 | 4.3 | 6.2 | 5.9 | 6.6 | 7.1 |
| 5.9 | 6.9 | 7.0 | 4.1 | 5.2 | 6.0 | 4.0 | 5.1 | 5.0 | 5.3
Figure 3: Predicted 2D poses using the model trained on synthetic images and
synthetic prior (SI + SP). Images rendered from synthetic model showing their
respective predicted (purple dots) and ground truth 2D poses (green dots).

####  4.0.3 DeepLabCut comparison.

In the absence of a more extensive set of annotated data for evaluating all
our predictions, we also report on a quantitative comparison with the
predictions from a state-of-the-art supervised method for animal pose
estimation: DeepLabCut [36]. The motivation for performing this comparison is
to show that our self-supervised approach can work similarly to this
supervised method, removing the requirement to annotate 2D poses for training.
To build the training set for DLC, we select a subset of 100 consecutive
images from one video and label 18 joint positions in each one. We then use
these images and their labelled 2D poses to train a DLC model in a supervised
fashion. We follow the official implementation of DLC [36]. Using the trained
DLC model, we then predict the pose for unseen videos.

Figure 4: Comparison of our predicted joint positions against the ones
predicted by DLC. A) Predictions for RFP, LFP, RHP, and LHP. B) Predictions
for snout, vertebral column, and tail. C)Visual comparison of predicted poses
by DLC and our method. Green - ours; pink - DLC.

We compare the predictions of our method against the ones produced by DLC.
Each estimated body joint position is represented as a $(x,y)$ pair of
coordinates on the image plane. Fig.4 summarises the results of our
comparison. Note that each graph contains our estimated positions (indicated
with lines) for a given joint together with the ones estimated by DLC
(indicated by dotted lines). In the inset legend, we use the label ‘DLC’ after
the name of the joint to identify the predicted joint positions by DeepLabCut.
The predictions of our method simply appear as the name of the joint. Finally,
we assess quantitatively the predictions of DLC with the same ground-truth of
that we used to evaluate predictions from the self-supervised method. Table 2
shows the MPJPE of DLC predictions and their respective ground truth. As
expected, the overall MPJPE is lower for DLC. This may be explained in part by
the use of supervision in training DLC, albeit on a limited dataset, and the
consistency with which joint positions are manually located in producing
ground-truth for the training and testing images.

Table 2: MPJPE of predicted poses with DLC. Joints | Snout | VCB | VCE | TB | TM | TE | RE | RFP- | RFP+ | Avg.
---|---|---|---|---|---|---|---|---|---|---
| LE | LFP- | LFP+ | RK | RHP- | RHP+ | LK | LHP- | LHP+ |
DLC | 4.7 | 16.3 | 18.2 | 4.0 | 7.2 | 20.2 | 7.8 | 5.4 | 5.1 |
| 6.5 | 5.7 | 9.8 | 5.6 | 5.3 | 6.1 | 7.2 | 9.0 | 8.3 | 8.5

####  4.0.4 Adaptation to other structures.

We demonstrate that our experimental setting, i.e. using synthetic prior and
actual data for training, is adaptable to other animal structures. We build a
dataset of horse images using a combination of individual frames from YouTube
videos depicting horses in motion and horse images from the TigDog dataset
[9]. The 2D poses for the synthetic prior come from the synthetic horse model
of [38]. We train the model using our dataset of approximately 30k horse
images and a prior of 10k synthetic 2D poses. Once trained, we evaluate it
utilising an out-distribution dataset [5] and some horse images from the
videos and TigDog [9] that were excluded during training. In addition, we test
our model’s generalisation capacities using images depicting zebras [62]. Note
that we use the same trained model for all the cases; surprisingly, the model
still does well on the zebras, although the training set does not contain
images of these animals. Fig. 3 shows the qualitative results of the 2D poses
predicted by our trained model on different data.

Figure 5: First row: Predictions using images from [5]. Second row:
Predictions using images from [9] and [62].

##  5 Discussion and Conclusion

Supervised methods learn from annotated poses on the training data, which
makes them dependent on the quality of those annotations. Although some joint
positions are easy to annotate, others require domain specialists to locate
them. Contrary to the supervised methods, our approach is not dependent on the
quality of the annotations since it learns from skeleton images generated from
synthetic poses. The method produces similar 2D poses to those obtained using
DLC (Section C of Fig.4), and its quantitative performance in terms of MPJPE
against ground-truth annotations is not significantly different from DLC.

According to the plots in Fig.4, despite some visible differences between our
method and DLC for specific body parts, most graphs show smooth lines for our
predictions. When comparing both methods against ground truth annotations, as
expected, the overall performance of DLC is superior. This is probably due in
part to the consistency of manual annotation of ground-truth joint locations
used in both training and testing of DLC. Our experiment using synthetic
images and a synthetic pose prior demonstrates that accurate predictions can
be made by matching the pose prior and image domains.

In conclusion, we successfully adapted a self-supervised 2D human pose
estimation method to a different animal domain, replacing an empirical prior
associated with actual 2D poses with a synthetic prior. We have demonstrated
that the approach produces promising results compared to a state-of-the-art
supervised approach in the mouse domain. An important motivation for our work
has been to explore an approach that can be rapidly deployed to other animal
domains without requiring extensive annotation of images. We demonstrate the
latter qualitatively using a dataset of horse images. Finally, we plan to use
our estimated 2D poses to measure gait on genetically modified mice with
different levels of ALS disease. These measures could help to identify and
classify patterns related to the development of the disease.

Ethics statement: This study was approved by the University of Tasmania Animal
Ethics Committee (permit number A17008) and designed in accordance with the
Australian Code of Practice for the Care and Use of Animals for Scientific
Purposes.

Acknowledgments Special thanks to Rebecca Stone and Mohammed Alghamdi from the
School of Computing at the University of Leeds for great discussions and
insightful feedback.

## References

  * [1] Bala, P.C., Eisenreich, B.R., Yoo, S.B.M., Hayden, B.Y., Park, H.S., Zimmermann, J.: Automated markerless pose estimation in freely moving macaques with openmonkeystudio. Nature Communications 11(1), 1–12 (2020)
  * [2] Bender, J.A., Simpson, E.M., Ritzmann, R.E.: Computer-assisted 3d kinematic analysis of all leg joints in walking insects. PloS one 5(10), e13617 (2010)
  * [3] Biggs, B., Boyne, O., Charles, J., Fitzgibbon, A., Cipolla, R.: Who left the dogs out? 3d animal reconstruction with expectation maximization in the loop. In: European Conference on Computer Vision. pp. 195–211. Springer (2020)
  * [4] Bolaños, L.A., Xiao, D., Ford, N.L., LeDue, J.M., Gupta, P.K., Doebeli, C., Hu, H., Rhodin, H., Murphy, T.H.: A three-dimensional virtual mouse generates synthetic training data for behavioral analysis. Nature methods 18(4), 378–381 (2021)
  * [5] Borenstein, E., Sharon, E., Ullman, S.: Combining top-down and bottom-up segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 46–46. IEEE (2004)
  * [6] Card, G., Dickinson, M.H.: Visually mediated motor planning in the escape response of drosophila. Current Biology 18(17), 1300–1307 (2008)
  * [7] Cashman, T.J., Fitzgibbon, A.W.: What shape are dolphins? building 3d morphable models from 2d images. IEEE Transactions on Pattern Analysis and Machine Intelligence 35(1), 232–244 (2012)
  * [8] Chen, C., Zhu, W., Norton, T.: Behaviour recognition of pigs and cattle: Journey from computer vision to deep learning. Computers and Electronics in Agriculture 187, 106255 (2021)
  * [9] Del Pero, L., Ricco, S., Sukthankar, R., Ferrari, V.: Articulated motion discovery using pairs of trajectories. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2151–2160 (2015)
  * [10] Farahnakian, F., Heikkonen, J., Björkman, S.: Multi-pig pose estimation using deeplabcut. In: 2021 11th International Conference on Intelligent Control and Information Processing (ICICIP). pp. 143–148. IEEE (2021)
  * [11] Fisher, E.M., Bannerman, D.M.: Mouse models of neurodegeneration: Know your question, know your mouse. Science Translational Medicine 11(493), eaaq1818 (2019)
  * [12] Gershow, M., Berck, M., Mathew, D., Luo, L., Kane, E.A., Carlson, J.R., Samuel, A.D.: Controlling airborne cues to study small animal navigation. Nature methods 9(3), 290–296 (2012)
  * [13] Goel, S., Kanazawa, A., Malik, J.: Shape and viewpoint without keypoints. In: European Conference on Computer Vision. pp. 88–104. Springer (2020)
  * [14] Graving, J.M., Chae, D., Naik, H., Li, L., Koger, B., Costelloe, B.R., Couzin, I.D.: Deepposekit, a software toolkit for fast and robust animal pose estimation using deep learning. Elife 8, e47994 (2019)
  * [15] Heuer, E., F Rosen, R., Cintron, A., C Walker, L.: Nonhuman primate models of alzheimer-like cerebral proteopathy. Current pharmaceutical design 18(8), 1159–1169 (2012)
  * [16] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4700–4708 (2017)
  * [17] Insafutdinov, E., Pishchulin, L., Andres, B., Andriluka, M., Schiele, B.: Deepercut: A deeper, stronger, and faster multi-person pose estimation model. In: European Conference on Computer Vision. pp. 34–50. Springer (2016)
  * [18] Jakab, T., Gupta, A., Bilen, H., Vedaldi, A.: Self-supervised learning of interpretable keypoints from unlabelled videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8787–8797 (2020)
  * [19] Jégou, S., Drozdzal, M., Vazquez, D., Romero, A., Bengio, Y.: The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11–19 (2017)
  * [20] Jiang, L., Liu, S., Bai, X., Ostadabbas, S.: Prior-aware synthetic data to the rescue: Animal pose estimation with very limited real data. arXiv preprint arXiv:2208.13944 (2022)
  * [21] Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision. pp. 694–711. Springer (2016)
  * [22] Kanazawa, A., Kovalsky, S., Basri, R., Jacobs, D.: Learning 3d deformation of animals from 2d images. In: Computer Graphics Forum. vol. 35, pp. 365–374. Wiley Online Library (2016)
  * [23] Karashchuk, P., Rupp, K.L., Dickinson, E.S., Walling-Bell, S., Sanders, E., Azim, E., Brunton, B.W., Tuthill, J.C.: Anipose: a toolkit for robust markerless 3d pose estimation. Cell reports 36(13), 109730 (2021)
  * [24] Kearney, S., Li, W., Parsons, M., Kim, K.I., Cosker, D.: Rgbd-dog: Predicting canine pose from rgbd sensors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8336–8345 (2020)
  * [25] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [26] Kirkpatrick, T., Schneider, C.W., Pavloski, R.: A computerized infrared monitor for following movement in aquatic animals. Behavior Research Methods, Instruments, & Computers 23(1), 16–22 (1991)
  * [27] Labuguen, R., Bardeloza, D.K., Negrete, S.B., Matsumoto, J., Inoue, K., Shibata, T.: Primate markerless pose estimation and movement analysis using deeplabcut. In: 2019 Joint 8th International Conference on Informatics, Electronics & Vision (ICIEV) and 2019 3rd International Conference on Imaging, Vision & Pattern Recognition (icIVPR). pp. 297–300. IEEE (2019)
  * [28] Lee, Y., Dawson, V.L., Dawson, T.M.: Animal models of parkinson’s disease: vertebrate genetics. Cold Spring Harbor Perspectives in Medicine 2(10), a009324 (2012)
  * [29] Li, C., Lee, G.H.: From synthetic to real: Unsupervised domain adaptation for animal pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1482–1491 (2021)
  * [30] Liu, H., Reibman, A.R., Boerman, J.P.: Video analytic system for detecting cow structure. Computers and Electronics in Agriculture 178, 105761 (2020)
  * [31] Liu, X., Yu, S.y., Flierman, N.A., Loyola, S., Kamermans, M., Hoogland, T.M., De Zeeuw, C.I.: Optiflex: Multi-frame animal pose estimation combining deep learning with optical flow. Frontiers in Cellular Neuroscience 15 (2021)
  * [32] Liu, Z., Zhu, J., Bu, J., Chen, C.: A survey of human pose estimation: the body parts parsing based methods. Journal of Visual Communication and Image Representation 32, 10–19 (2015)
  * [33] Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: Smpl: A skinned multi-person linear model. ACM transactions on graphics (TOG) 34(6), 1–16 (2015)
  * [34] Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Paul Smolley, S.: Least squares generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2794–2802 (2017)
  * [35] Mathis, A., Biasi, T., Schneider, S., Yuksekgonul, M., Rogers, B., Bethge, M., Mathis, M.W.: Pretraining boosts out-of-domain robustness for pose estimation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 1859–1868 (2021)
  * [36] Mathis, A., Mamidanna, P., Cury, K.M., Abe, T., Murthy, V.N., Mathis, M.W., Bethge, M.: Deeplabcut: markerless pose estimation of user-defined body parts with deep learning. Nature neuroscience 21(9), 1281–1289 (2018)
  * [37] Mathis, M.W., Mathis, A.: Deep learning tools for the measurement of animal behavior in neuroscience. Current Opinion in Neurobiology 60, 1–11 (2020)
  * [38] Mu, J., Qiu, W., Hager, G.D., Yuille, A.L.: Learning from synthetic animals. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12386–12395 (2020)
  * [39] Nath, T., Mathis, A., Chen, A.C., Patel, A., Bethge, M., Mathis, M.W.: Using deeplabcut for 3d markerless pose estimation across species and behaviors. Nature Protocols 14(7), 2152–2176 (2019)
  * [40] Negrete, S.B., Labuguen, R., Matsumoto, J., Go, Y., Inoue, K.i., Shibata, T.: Multiple monkey pose estimation using openpose. bioRxiv (2021)
  * [41] Newell, A., Yang, K., Deng, J.: Stacked hourglass networks for human pose estimation. In: Proceedings of the European Conference on Computer Vision. pp. 483–499. Springer (2016)
  * [42] Noldus, L.P., Trienes, R.J., Hendriksen, A.H., Jansen, H., Jansen, R.G.: The observer video-pro: New software for the collection, management, and presentation of time-structured data from videotapes and digital media files. Behavior Research Methods, Instruments, & Computers 32, 197–206 (2000)
  * [43] Olivo, R.F., Thompson, M.C.: Monitoring animals’ movements using digitized video images. Behavior Research Methods, Instruments, & Computers 20, 485–490 (1988)
  * [44] Pereira, T.D., Aldarondo, D.E., Willmore, L., Kislin, M., Wang, S.S.H., Murthy, M., Shaevitz, J.W.: Fast animal pose estimation using deep neural networks. Nature methods 16(1), 117–125 (2019)
  * [45] Pfister, T., Charles, J., Zisserman, A.: Flowing convnets for human pose estimation in videos. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 1913–1921 (2015)
  * [46] Philips, T., Rothstein, J.D.: Rodent models of amyotrophic lateral sclerosis. Current Protocols in Pharmacology 69(1), 5–67 (2015)
  * [47] Raman, S., Maskeliūnas, R., Damaševičius, R.: Markerless dog pose recognition in the wild using resnet deep learning model. Computers 11(1), 2 (2022)
  * [48] Riekert, M., Klein, A., Adrion, F., Hoffmann, C., Gallmann, E.: Automatically detecting pig position and posture by 2d camera imaging and deep learning. Computers and Electronics in Agriculture 174, 105391 (2020)
  * [49] Rüegg, N., Tripathi, S., Schindler, K., Black, M.J., Zuffi, S.: Bite: Beyond priors for improved three-d dog pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8867–8876 (2023)
  * [50] Russello, H., van der Tol, R., Kootstra, G.: T-leap: Occlusion-robust pose estimation of walking cows using temporal information. Computers and Electronics in Agriculture 192, 106559 (2022)
  * [51] Saito, T., Matsuba, Y., Mihira, N., Takano, J., Nilsson, P., Itohara, S., Iwata, N., Saido, T.C.: Single app knock-in mouse models of alzheimer’s disease. Nature neuroscience 17(5), 661–663 (2014)
  * [52] Sato, Y., Kondo, T., Shinozaki, M., Shibata, R., Nagoshi, N., Ushiba, J., Nakamura, M., Okano, H.: Markerless analysis of hindlimb kinematics in spinal cord-injured mice through deep learning. Neuroscience Research (2021)
  * [53] Sheppard, K., Gardin, J., Sabnis, G.S., Peer, A., Darrell, M., Deats, S., Geuther, B., Lutz, C.M., Kumar, V.: Stride-level analysis of mouse open field behavior using deep-learning-based pose estimation. Cell reports 38(2), 110231 (2022)
  * [54] Shooter, M., Malleson, C., Hilton, A.: Sydog: A synthetic dog dataset for improved 2d pose estimation. arXiv preprint arXiv:2108.00249 (2021)
  * [55] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  * [56] Spink, A., Tegelenbosch, R., Buma, M., Noldus, L.: The ethovision video tracking system—a tool for behavioral phenotyping of transgenic mice. Physiology & behavior 73(5), 731–744 (2001)
  * [57] Tompson, J.J., Jain, A., LeCun, Y., Bregler, C.: Joint training of a convolutional network and a graphical model for human pose estimation. Advances in Neural Information Processing Systems 27 (2014)
  * [58] Wang, Y., Li, J., Zhang, Y., Sinnott, R.O.: Identifying lameness in horses through deep learning. In: Proceedings of the 36th Annual ACM Symposium on Applied Computing. pp. 976–985 (2021)
  * [59] Yao, Y., Jafarian, Y., Park, H.S.: Monet: Multiview semi-supervised keypoint detection via epipolar divergence. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 753–762 (2019)
  * [60] Zuffi, S., Kanazawa, A., Berger-Wolf, T., Black, M.J.: Three-d safari: Learning to estimate zebra pose, shape, and texture from images” in the wild”. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5359–5368 (2019)
  * [61] Zuffi, S., Kanazawa, A., Black, M.J.: Lions and tigers and bears: Capturing non-rigid, 3d, articulated shape from images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3955–3963 (2018)
  * [62] Zuffi, S., Kanazawa, A., Jacobs, D.W., Black, M.J.: 3d menagerie: Modeling the 3d shape and pose of animals. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6365–6373 (2017)