#  LenslessPiCam: A Hardware and Software Platform for Lensless Computational
Imaging with a Raspberry Pi

Eric Bezzam Corresponding author<EMAIL_ADDRESS>Sepand Kashani  Martin
Vetterli  Matthieu Simeoni

## Summary

Lensless imaging seeks to replace/remove the lens in a conventional imaging
system. The earliest cameras were in fact lensless, relying on long exposure
times to form images on the other end of a small aperture in a darkened
room/container (camera obscura). The introduction of a lens allowed for more
light throughput and therefore shorter exposure times, while retaining sharp
focus. The incorporation of digital sensors readily enabled the use of
computational imaging techniques to post-process and enhance raw images (e.g.
via deblurring, inpainting, denoising, sharpening). Recently, imaging
scientists have started leveraging computational imaging as an integral part
of lensless imaging systems, allowing them to form viewable images from the
highly multiplexed raw measurements of lensless cameras (see [5] and
references therein for a comprehensive treatment of lensless imaging). This
represents a real paradigm shift in camera system design as there is more
flexibility to cater the hardware to the application at hand (e.g. lightweight
or flat designs). This increased flexibility comes however at the price of a
more demanding post-processing of the raw digital recordings and a tighter
integration of sensing and computation, often difficult to achieve in practice
due to inefficient interactions between the various communities of scientists
involved. With LenslessPiCam, we provide an easily accessible hardware and
software framework to enable researchers, hobbyists, and students to implement
and explore practical and computational aspects of lensless imaging. We also
provide detailed guides and exercises so that LenslessPiCam can be used as an
educational resource, and point to results from our graduate-level signal
processing course.

GitHub repo: https://github.com/LCAV/LenslessPiCam

## Statement of need

(a)

(b)

Figure 1: ADMM reconstruction of thumbs-up on a phone 40 cm away.

Being at the interface of hardware, software, and algorithm design, the field
of lensless imaging necessitates a broad array of competences that might deter
newcomers to the field. The purpose of LenslessPiCam is to provide a complete
toolkit with cheap, accessible hardware designs and open-source software, that
also achieves satisfactory results in order to explore novel ideas for
hardware, software, and algorithm design.

The DiffuserCam tutorial [4] served as a great starting point to the present
toolkit as it demonstrates that a working lensless camera can be built with
cheap hardware: a Raspberry Pi, the Camera Module
2,111www.raspberrypi.com/products/camera-module-v2 and a piece of tape. The
authors also provide Python implementations of two image reconstruction
algorithms:

  1. 1.

Variants of gradient descent (GD) with a non-negativity constraint,

  2. 2.

The alternating direction method of multipliers (ADMM) [6] with an additional
total variation (TV) prior.

Moreover, detailed guides explain how to build their camera and give intuition
behind the reconstruction algorithms.

Unfortunately, the resolution of the reconstructed images is poor (see Figure
1(a)) and the processing pipeline is limited to grayscale reconstruction. With
LenslessPiCam, we improve the resolution by using the newer HQ
camera222www.raspberrypi.com/products/raspberry-pi-high-quality-camera/ as
well as a more versatile and generic RGB computational imaging pipeline. The
latter is built upon the Python library Pycsou [9], a universal and reusable
software environment providing key computational imaging functionalities and
tools with great modularity and interoperability. This results in a more
flexible and accurate reconstruction workflow, allowing for the quick
prototyping of advanced post-processing schemes with more sophisticated image
priors. See Figure 1(b) for an example image obtained with our lensless
imaging framework.

LenslessPiCam is designed to be used by researchers, hobbyists, and students.
In the past, we have found such open-source hardware and software platforms to
be a valuable resource for researchers [3] and students alike [2].

## Contributions

(a)

(b)

(c)

Figure 2: (a) LenslessPiCam, (b) point source generator (inside) and (c)
(outside).

With respect to the DiffuserCam tutorial [4], we have made the following
contributions. In terms of hardware, as shown in Figure 2, we:

  * •

make use of the HQ camera sensor ($\$50$): 4056$\times$3040 pixels (12.3 MP)
and $7.9\text{\,}\mathrm{mm}$ sensor diagonal, compared to 3280$\times$2464
pixels (8.1 MP) and $4.6\text{\,}\mathrm{mm}$ sensor diagonal for the Camera
Module 2 ($\$30$),

  * •

provide the design and firmware for a cheap point source generator (needed for
calibration), which consists of an Arduino, a white LED, and a cardboard box.

With respect to reconstruction algorithms, we:

  * •

provide significantly faster implementations of GD and ADMM, i.e. around
3$\times$ reduction in computation time,

  * •

extend the above reconstructions to RGB,

  * •

provide an object-oriented structure that is easy to extend for exploring new
algorithms,

  * •

provide an object-oriented interface to Pycsou for solving lensless imaging
inverse problems. Pycsou is a Python package for solving inverse problems of
the form

$\min_{\mathbf{\alpha}\in\mathbb{R}^{N}}\,F(\mathbf{y},\mathbf{G}\mathbf{\alpha})\quad+\quad\lambda\mathcal{R}(\mathbf{\alpha}),$
(1)

where $F$ is a data-fidelity term between the observed and predicted
measurements $\mathbf{y}$ and $\mathbf{G}\mathbf{\alpha}$ respectively,
$\mathcal{R}$ is a regularization component (could consist of more than one
prior), and $\lambda>0$ controls the amount of regularization.

We also provide functionality to:

  * •

remotely capture Bayer data with the proposed camera,

  * •

convert Bayer data to RGB or grayscale,

  * •

quantitatively evaluate the point spread function (PSF) of the lensless
camera,

  * •

remotely display data on an external monitor, which can be used to automate
raw data measurements to, e.g., gather a dataset,

  * •

collect MNIST remotely or from the Raspberry Pi,

  * •

evaluate reconstructions on a variety of metrics: MSE, PSNR, SSIM, LPIPS [10].

Finally, we have written a set of Medium articles to guide users through the
process of building, using and/or teaching with the proposed lensless
camera.333An overview of these articles can be found here:
https://medium.com/@bezzam/a-complete-lensless-imaging-tutorial-hardware-
software-and-algorithms-8873fa81a660 The articles include a set of solved
exercises and problems for teaching purposes (solutions available to
instructors on request).

In the following sections, we describe some of these contributions, and
quantify them (where appropriate).

## High-level functionality

The core algorithmic component of LenslessPiCam is the abstract class
lensless.ReconstructionAlgorithm. The three reconstruction strategies
available in LenslessPiCam derive from this class:

  * •

lensless.GradientDescient: projected GD with a non-negativity constraint. Two
accelerated approaches are also available:

    * –

lensless.NesterovGradientDescent [8]

    * –

lensless.FISTA [1]

  * •

lensless.ADMM: ADMM with a non-negativity constraint and a TV regularizer,

  * •

lensless.APGD: accelerated proximal GD with Pycsou as a backend. Any
differentiable or proximal operator can be used as long as it is compatible
with Pycsou, namely derives from one of DifferentiableFunctional or
ProximableFunctional.

One advantage of deriving from lensless.ReconstructionAlgorithm is that
functionality for iterating, saving, and visualization is already implemented.
Consequently, using a reconstruction algorithm that derives from it boils down
to three steps:

  1. 1.

Creating an instance of the reconstruction algorithm.

  2. 2.

Setting the data.

  3. 3.

Applying the algorithm.

For example, for ADMM (full example in scripts/recon/admm.py):

    Ψrecon = ADMM(psf)
    Ψrecon.set_data(data)
    Ψres = recon.apply(n_iter=n_iter)

A template for applying a reconstruction algorithm (including loading the
data) can be found in scripts/recon/template.py.

## Efficient reconstruction

In Table 1, we compare the processing time of DiffuserCam’s and
LenslessPiCam’s implementations for grayscale reconstruction of:

  1. 1.

GD using FISTA and a non-negativity constraint,

  2. 2.

ADMM with a non-negativity constraint and a TV regularizer.

The DiffuserCam implementations can be found here: https://github.com/Waller-
Lab/DiffuserCam-Tutorial, while lensless.APGD and lensless.ADMM are used for
LenslessPiCam. The comparison is done on a Lenovo Thinkpad P15 with 16 GB RAM
and a 2.70 GHz processor (6 cores, 12 threads), running Ubuntu 21.04.

| GD | ADMM
---|---|---
DiffuserCam | $215\text{\,}\mathrm{s}$ | $7.24\text{\,}\mathrm{s}$
LenslessPiCam | $67.9\text{\,}\mathrm{s}$ | $2.76\text{\,}\mathrm{s}$
Table 1: Benchmark grayscale reconstruction. 300 iterations for gradient
descent (GD) and 5 iterations for alternating direction method of multipliers
(ADMM).

From Table 1, we observe a 3.1$\times$ reduction in computation time for GD
and a 2.6$\times$ reduction for ADMM. This comes from:

  * •

our object-oriented implementation of the algorithms, which allocates all the
necessary memory beforehand and pre-computes data-independent terms, such as
forward operators from the point spread function (PSF),

  * •

our use of the real-valued FFT, which is possible since we are working with
image intensities.

Figure 3 shows the corresponding grayscale reconstruction for FISTA and ADMM,
which are equivalent for both DiffuserCam and LenslessPiCam implementations.

(a)

(b)

Figure 3: Grayscale reconstruction using FISTA (a) and ADMM (b).

## Quantifying performance

In order to methodically compare different reconstruction approaches, it is
necessary to quantify the performance. To this end, LenslessPiCam provides
functionality to extract regions of interest from the reconstruction and
compare them with the original image via multiple metrics:

  * •

Mean-squared error (MSE),

  * •

Peak signal-to-noise ratio (PSNR),

  * •

Mean structural similarity (SSIM) index,

  * •

Learned perceptual image patch similarity (LPIPS).

Figure 4 and Table 2 is an example of how a reconstruction can be evaluated
against an original image, e.g. using scripts/compute_metrics_original.py

Figure 4: Extracting region from Figure 1(b) to quantify performance. MSE | PSNR | SSIM | LPIPS
---|---|---|---
0.164 | 7.85 | 0.405 | 0.645
Table 2: Metrics for Figure 4.

Sometimes it may be of interest to perform an exhaustive evaluation on a large
dataset. While LenslessPiCam could be used for collecting such a dataset with
the proposed camera,444Using the remote display and capture scripts, i.e.
scripts/remote_display.py and scripts/remote_capture.py respectively. the
authors of [7] have already collected a dataset of 25’000 parallel
measurements, namely 25’000 pairs of DiffuserCam and lensed camera
images.555https://waller-lab.github.io/LenslessLearning/dataset.html
LenslessPiCam offers functionality to evaluate a reconstruction algorithm on
this dataset, or a subset of it that we have prepared.666Subset of DiffuserCam
Lensless Mirflickr Dataset [7] consists of 200 files (725 MB) as opposed to
25’000 files (100 GB) of the original dataset. The subset can be downloaded
here: https://drive.switch.ch/index.php/s/vmAZzryGI8U8rcE. Note that this
dataset is collected with a different lensless camera, but is nonetheless
useful for exploring reconstruction techniques.

Table 3 shows the average metric results after applying 100 iterations of ADMM
to the subset we have prepared.777Using scripts/evaluate_mirflickr_admm.py

MSE | PSNR | SSIM | LPIPS
---|---|---|---
0.0797 | 12.7 | 0.535 | 0.585
Table 3: Average metrics for 100 iterations of ADMM on a subset (200 files) of the DiffuserCam Lensless Mirflickr Dataset. Figure 5: Visualizing performance of ADMM (100 iterations) on a single file of the DiffuserCam Lensless Mirflickr Dataset. MSE | PSNR | SSIM | LPIPS
---|---|---|---
0.0682 | 11.7 | 0.486 | 0.504
Table 4: Metrics for Figure 5.

One can also visualize the performance on a single file of the dataset, namely
how the reconstruction changes as the number of iterations increase.888Using
scripts/apply_admm_single_mirflickr.py. The final reconstruction and outputed
metrics are shown in Figure 5 and Table 4.

## As an educational resource

As mentioned earlier, LenslessPiCam can serve as an educational resource. We
have used it in our graduate-level signal processing course for providing
experience in applying fundamental signal processing concepts and solving
linear inverse problems. The work of our students can be found here.

As exercises in implementing key signal processing components, we have left
some incomplete functions in LenslessPiCam:

  * •

lensless.autocorr.autocorr2d: to compute a 2D autocorrelation in the frequency
domain,

  * •

lensless.realfftconv.RealFFTConvolve2D: to pre-compute the PSF’s Fourier
transform, perform a convolution in the frequency domain with the real-valued
FFT, and vectorize operations for RGB.

We have also proposed a few reconstruction approaches to implement in this
Medium article.

For the solutions to the above implementations, please request access to this
folder detailing the intended use.

## Conclusion

In summary, LenslessPiCam provides all the necessary hardware designs and
software to build, use, and evaluate a lensless camera with cheap and
accessible components. As we continue to use it as a research and educational
platform, we hope to investigate and incorporate:

  * •

computational refocusing,

  * •

video reconstruction,

  * •

on-device reconstruction,

  * •

programmable masks,

  * •

data-driven, machine learning reconstruction techniques.

## Acknowledgements

We acknowledge feedback from Julien Fageot and the students during the first
iteration of this project in our graduate course.

This work was in part funded by the Swiss National Science Foundation (SNSF)
under grants CRSII5 193826 “AstroSignals - A New Window on the Universe, with
the New Generation of Large Radio-Astronomy Facilities” (M. Simeoni), 200 021
181 978/1 “SESAM - Sensing and Sampling: Theory and Algorithms” (E. Bezzam)
and CRSII5 180232 “FemtoLippmann - Digital twin for multispectral imaging” (S.
Kashani).

## References

  * [1] Amir Beck and Marc Teboulle.  A fast iterative shrinkage-thresholding algorithm for linear inverse problems.  SIAM Journal on Imaging Sciences, 2(1):183–202, 2009.
  * [2] Eric Bezzam, Adrien Hoffet, and Paolo Prandoni.  Teaching practical dsp with off-the-shelf hardware and free software.  In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7660–7664, 2019.
  * [3] Eric Bezzam, Robin Scheibler, Juan Azcarreta, Hanjie Pan, Matthieu Simeoni, Rene Beuchat, Paul Hurley, Basile Bruneau, Corentin Ferry, and Sepand Kashani.  Hardware and software for reproducible research in audio array signal processing.  In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6591–6592, 2017.
  * [4] C. Biscarrat, S. Parthasarathy, G. Kuo, and N. Antipa.  Build your own diffusercam: Tutorial.  https://waller-lab.github.io/DiffuserCam/tutorial.html, 2018.
  * [5] Vivek Boominathan, Jacob T. Robinson, Laura Waller, and Ashok Veeraraghavan.  Recent advances in lensless imaging.  Optica, 9(1):1–16, Jan 2022.
  * [6] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein.  Distributed optimization and statistical learning via the alternating direction method of multipliers.  Found. Trends Mach. Learn., 3(1):1–122, jan 2011.
  * [7] Kristina Monakhova, Joshua Yurtsever, Grace Kuo, Nick Antipa, Kyrollos Yanny, and Laura Waller.  Learned reconstructions for practical mask-based lensless imaging.  Optics Express, 27(20):28075, September 2019.
  * [8] Yurii E. Nesterov.  A method for solving the convex programming problem with convergence rate $o(1/k^{2})$.  In Dokl. Akad. Nauk SSSR, volume 269, pages 543–547, 1983.
  * [9] M. Simeoni.  Pycsou.  https://github.com/matthieumeo/pycsou, 2021.
  * [10] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 586–595, 2018.


# Object detection for crabs in top-view seabed imagery

Vlad Velici
School of Electronics and Computer Science
University of Southampton
Southampton, UK &Adam Prügel-Bennett
School of Electronics and Computer Science
University of Southampton
Southampton, UK

###### Abstract

This report presents the application object detection on a database of
underwater images of different species of crabs, as well as aerial images of
sea lions and Pascal VOC. The model is an end-to-end object detection model
based on a convolutional network and a Long Short-Term Memory detector.

mAP

    mean average precision
CNN

    Convolutional Neural Network
RPN

    Region Proposal Network
RNN

    Recurrent Neural Network
RNN

    Recurrent Neural Network
LSTM

    Long Short-Term Memory
NLP

    Natural Language Processing
GRU

    Gated Recurrent Unit

##  1 Introduction

This report presents the problem of object detection and classification,
popular datasets, and in Section 1.1 we present the datasets relevant to this
work.

Image classification is the task of labelling an image with a class. Given an
input image a model must predict what class it belongs to. Images used for
classification often have one large central object. However this is not the
case in real life where we are surrounded by many objects. A more challenging
task is to predict where all the objects are in an image and what class they
belong to. We call this object detection. An intermediary objective can be
object localisation which is simply the task of finding all objects in an
image but not assigning a class to them.

Depending on the actual dataset and desired end result, an object is defined
by its class and either its $x$ and $y$ coordinates or a bounding box ($x,y$
coordinates, width $w$ and height $h$). We will use the term object
coordinates to refer to either $x$ and $y$ coordinates or a bounding box.

Datasets like PASCAL Visual Object Classes (object detection challenge)
(Everingham et al., 2015) and COCO (Lin et al., 2014) have bounding box
coordinates, where other datasets like the ones used in the NOAA Fisheries
Steller Sea Lion Population Count Kaggle competition (NOAA Fisheries, 2017)
and the data obtained from (Thornton et al., 2016) only have the centre
coordinate of the objects. Semantic segmentation is a type of object detection
where the bounding boxes or coordinates are replaced with object boundaries at
pixel level. We are not looking into semantic segmentation in this report at
this time.

###  1.1 Crabs and Steller Sea Lions datasets

Figure 1: Map of Steller Sea Lion populations from NOAA Fisheries outreach
video https://youtu.be/oiL8tDCqzy4. Red area shows the endangered population,
yellow area shows the population that is increasing and green area shows the
population that has fully recovered in 2013.

Steller Sea Lion population started to decline in the 1970s, and in 1990 it
was listed as a threatened species. NOAA Fisheries divided the population into
the western stock and the eastern stock, the separation line being at
$144\deg$ West longitude. The eastern stock population started to recover in
the late 1970s and has fully recovered in 2013. However, the western stock
population has not. The population in the eastern side of the western stock
($144\deg$ \- $170\deg$ West longitude) is increasing, but the western end, in
Aleutian Islands, continues to decline. A map of the locations can be seen in
Figure 1. NOAA Fisheries is working on creating long-term population trends
over time to help develop recovering strategies for the endangered population.
Having such trends and tools will allow NOAA to make better decisions about
managing fisheries and to identify possible threats affecting the endangered
populations (Fritz et al., 2016).

NOAA Fisheries can now gather large amounts of aerial imagery efficiently
using drones. They are, however, facing the problem of manually labelling the
images to be able to create population trends. Currently they use human
labelling but this method is prone to errors and slow, thus they are seeking
automated solutions to improve the speed and quality of labelling. They have
launched a Kaggle111Kaggle: https://kaggle.com/ competition to invite data
scientists to find solutions and implement models to solve the labelling
problem. The competition has a dataset of 100+GB of labelled imagery (NOAA
Fisheries, 2017).

The work presented in Thornton et al. (2016) is a practical method of using
underwater robots to survey large areas of seafloor (multi-hectare areas). The
images produced were then processed to build 3D reconstructions and mosaics
which were manually labelled for 6 taxa of animals (crabs, mussels and
shrimps). Labelling taxa by hand is a long, slow, expensive and error prone
process, and it is the bottleneck of the whole process of creating population
densities and distributions efficiently. They produced a dataset of two
labelled mosaics of imagery taken at two locations: C0014G_2m_2014 is a 1060m
deep drill site in the Iheya North field, and NBC_2m_2014 is a naturally
active site located 500m away.

In this report we will present a model created for these two datasets and
report its performance, training and possible improvements.

####  1.1.1 Note on the state of the art

This work was part of my PhD _initial_ project, which was done in 2016, for
the purpose of exploring deep learning and object detection, not necessarily
to obtain good results. If the reader is interested in exploring the field of
object detection and building competitive models we suggest to look up the
current state of the art models and methods. We refrain from mentioning any
works here because they will quickly become out of date.

##  2 Crab detector network

In this report we present a deep neural network built for end-to-end object
detection. As an end-to-end model, it takes as an input an image and outputs
object coordinates, classes and confidence scores. The model can be configured
to use centre coordinates of the objects or bounding boxes.

The model is built for the crabs and sea lions datasets where objects are
defined by centre coordinates and are relatively small but it is also
evaluated on VOC12 where objects are described by bounding boxes.

##  3 Model architecture

Figure 2: Architecture of our model.

The model architecture is based on Stewart et al. (2016). We improved on their
model by adding a classifier and adapting the loss function. We also added
support for x, y coordinates labels as well as bounding boxes to enable using
the model on the Pascal VOC 2012 dataset, the crabs and the Steller Sea Lions
datasets.

The model architecture is illustrated in Figure 2. For a forward pass, the
model performs the following steps:

Step 1 The input image is fed into a Convolutional Neural Network (CNN). The
feature map that results from the CNN is a $B\times W\times H\times F$ grid,
where $B$ is the batch size, $W,H$ are the width and height of the grid and
$F$ is the number of filters in the last convolutional layer.

Step 2 Each of the $B\times W\times H$ grid cells ($F$-sized feature vectors)
is fed into a layer of Long Short-Term Memory cells (LSTMs). Note that changes
in $W$ and $H$ can be considered changes in the batch size $B$, therefore the
model can be used for any input image size.

Step 3 At each recurrence the LSTM cells are densely connected to three
different units: one to predict objectness (used as confidence score), one to
predict class probabilities (softmax), and one to predict object coordinates
relative to the centre of the grid cell.

For our experiments we use square input images of $224\times 224$ pixels,
therefore our grid is a square of size $G=W=H$. We use two CNN configurations
that result in $G=7$ and $G=4$, respectively. Although we use a fixed image
size as input, the model can support input images of any size as the LSTM and
dense layers receive a fixed-size input regardless of the input image size.
The grid size changes with the input image and affects how many times the LSTM
and dense layers are run sequentially across the image.

A fixed number of recurrences (time steps) of the LSTM cells, $k$, is set.
This is an implementation limitation. Theoretically, at test time the model
can run recurrences until a stop symbol is produced (first object with
confidence below a set threshold, e.g. $0.5$). During training the model can
have one recurrence for each ground truth label and an additional one for
training the stop symbol. This can be used to speed up the computations.

There is no ground truth order in which the objects in an image must be
predicted. Therefore simply matching predictions and ground truth labels as
they appear would induce errors if the order does not match. For instance, if
an image has objects A and B given in the ground truth labels in this order
and the model predicts B, A, a direct matching would consider this prediction
wrong, but in reality it is correct. As suggested by Stewart et al. (2016),
the Hungarian method is used as an efficient way to match ground truth labels
to predictions using the distance between the coordinates as the matching
cost.

A new loss function was derived based on Stewart et al. (2016) and Redmon et
al. (2016). The loss function $l$ is the sum of three components: the
confidence (objectness) loss $l_{o}$, the coordinates (regression) loss
$l_{r}$ and the class loss $l_{c}$.

For the following equations we consider the ground truth labels to be sorted
such that the $i^{\text{th}}$ element of predictions is matched to the
$i^{\text{th}}$ element of ground truth labels by the Hungarian method as
described above.

The confidence score prediction is designed as a binary classification problem
and we use softmax and cross-entropy loss for $l_{o}$.

The regression loss is a root mean squared error function that only penalises
predictions where there is a real object. Define $\mathbf{g_{o}}$ to be the
vector of ground truth confidence scores with elements
$g_{o}(1),\dots,g_{o}(n)\in\\{0,1\\}$ where $n=G^{2}*k$ is the number of
predictions per image and $g_{o}(i)=1$ if the prediction $i$ is an object, $0$
otherwise. Let $m=\sum_{i=1}^{n}{g_{o}(i)}$ be the number of real objects in
the image, ${F_{r}}$ to be a $n\times r$ matrix, where $r$ is the number of
coordinates required per object ($r=2$ for x, y coordinates and $r=4$ for
bounding boxes) with elements $F_{r}(i,j)$ representing predicted coordinates.
Similarly, let $G_{r}$ be a matrix of the same size with elements containing
ground truth coordinates or zeros if there is no object. We now define the
regression loss

$l_{r}=\sqrt{\frac{1}{m*r}\sum_{i=1}^{n}g_{o}(i)\sum_{j=1}^{r}(F_{r}(i,j)-G_{r}(i,j))^{2}}.$
(1)

Let $F_{c}$ be a $n\times C$ matrix ($C$ is the number of classes) with rows
$\mathbf{F_{c}(i)}$ representing the probabilities of each class for object
$i$ (output of softmax). Similarly let $G_{c}$ be a similar matrix
representing the ground truth. The class loss is

$l_{c}=\frac{1}{m}\sum_{i=1}^{n}g_{o}(i)H(\mathbf{F_{c}(i)},\mathbf{G_{c}(i)}),$
(2)

where $H(\mathbf{a},\mathbf{b})$ is the cross-entropy between $\mathbf{a}$ and
$\mathbf{b}$. Note this is the cross-entropy loss but it only penalises
predictions where there is a real object (otherwise ground truth data does not
exist for the class).

##  4 Preprocessing

The labels are pre-processed such that, for an image, we will have $G\times
G\times k$ object labels in total, $k$ for each grid cell. If there are more
than $k$ objects in a particular grid cell, we cap them at $k$ and drop the
rest. Less than 10 objects are dropped in the whole crabs dataset for $k=8$.
Objects with confidence $0$ are appended if there are fewer than $k$ labels.

Label coordinates are made relative to the centre of the grid cell. We say the
top left corner of a grid cell is the point $(0,0)$ and the bottom right
corner is the point $(1,1)$. The centre of the grid cell is at $(0.5,0.5)$.
This forces the LSTMs to predict objects that belong to their grid cell and
makes the network easier to train. In the case of bounding boxes, the width
and height are relative to the input image size, such that an object with
width and height 1 covers the whole input image. This allows the network to
predict objects that are larger than a grid cell.

The image pixel values are normalised to fall between -0.5 and 0.5.

###  4.1 Crabs dataset

Table 1: Crabs species counts in the two locations (mosaics) and overall. The last column shows which species are in the crabs-top3 dataset. Species | C0014 | NBC | Total | crabs-top3
---|---|---|---|---
Alvinocaridid | 170 | 500 | 670 |
Bathymodiolus japonicus | 6,780 | 7,339 | 14,119 | ✓
Bathymodiolus platifrons | 7,282 | 12,969 | 20,251 | ✓
Paralomis | 96 | 109 | 205 |
Shinkaia crosnieri | 3,536 | 7,160 | 10,696 | ✓
Thermosipho desbruyesi | 12 | 6 | 18 |
Total individuals | 17,876 | 28,083 | 45,959 | 45,066

The crab dataset from Thornton et al. (2016) is made of two big mosaics
C0014G_2m_2014 ($20320\times 28448$ pixels) and NBC_2m_2014 ($20320\times
20320$ pixels), each representing an imaging location. We slice the mosaics
into $224\times 224$ pixels images and randomly split them into training,
cross-validation (dev) and test sets ($60\%-20\%-20\%$).

The labels consist of manually labelled x and y coordinates of each individual
and its species.

The dataset is heavily unbalanced, as seen in Table 1. This leads to hard to
train models thus we split the dataset into two sub-datasets. We call crabs
the dataset containing all images and crabs-top3 the dataset consisting of the
top 3 species by population count.

The big mosaics provided suffer from image quality loss as they are composed
of individual higher-quality images, overlapped and transformed to form a map-
like mosaic. The effect can be observed in Figure 3. The original images are
available but, unfortunately, there is no mapping between the labels on the
mosaic and the original images, which are unlabelled.

Sequentially slicing the mosaics results in a total of 2608 $224\times 224$
images. A dataset of such a small size is likely to make models easily overfit
and hard to generalise. To address this issue we heavily augment the training
set with random rotations, zooms, shears, and shifts and also use a different
slicing technique where we cut an image centred on a crab. This results in a
total of 45959 images (17,876 from C0014G and 28,083 from NBC) that have
plenty of overlapping content.

(a) Examples of original images from the crabs dataset.

(b) The two mosaics of the crabs dataset. Left: C0014G_2m_2014, right:
NBC_2m_2014.

(c) Two close-ups in the mosaics highlighting the effects of image quality
loss, likely due to overlapping and image processing for creating the mosaic.

Figure 3: Representation of the crabs dataset with example original images,
the map mosaics from which the training data is sliced and highlights of image
quality loss in the mosaics.

###  4.2 Sea Lions dataset

The Steller Sea Lions dataset from the NOAA Kaggle competition (NOAA
Fisheries, 2017) has many large aerial images. 948 for training and 18,636 for
testing during the competition. Similarly to the crabs dataset, the dataset
has x, y and class coordinates as labels, manually labeled.

The labels are given in two ways. Population counts per image are given in a
CSV file. The x, y coordinates are given in a second pair of training images
annotated with coloured dots, the colour denoting the class. The coordinates
and classes can be extracted by subtracting the dotted and the pair non-dotted
training image and running a blob detector on the result. However, this pre-
processing step is a rather big source of errors in the training data used for
our model. Population counts from the CSV and the extracted dotted images are
shown in Table 2.

The given training images are large (sizes 3744x5616, 4992x3328 and
5616x3744). We slice them into 224x224 input images for training and testing.
We split the given training set into a training and dev set (80%-20%). The
resulting dataset has 21,683 images for training and 5243 for dev.

The competition goal, and evaluation criteria, is the final population counts
per image rather than predicting where each individual is in the image.

Table 2: Sea Lions dataset population counts, showing the differences between the given population counts and the generated dots. The dots have 16,955 dots that could not reliably be classified by colour (not shown). | Label counts | Dot counts | $\Delta$
---|---|---|---
adult_males | 5,392 | 5,349 | 43
subadult_males | 4,345 | 4,028 | 317
adult_females | 37,537 | 59,080 | 21,543
juveniles | 20,118 | 20,895 | 777
pups | 16,285 | 16,141 | 144
Totals | 83,677 | 105,493 | 21,816

##  5 Training and evaluation

A set of model configurations and hyper-parameters were chosen to train and
evaluate our network with. The CNN used to create a feature map is arguably
the most important configuration choice.

We use the Inception-V1 (GoogLeNet) from Szegedy et al. (2015) up to the
_Mixed_5c_ layer, pre-trained on ImageNet by Google 222Weights available at
https://github.com/tensorflow/models/tree/master/slim. The weights are not
frozen during training.

The input image size is fixed at 224x224. A configurable parameter is the grid
size $G$. The CNN gives $G=7$ at the _Mixed_5c_ layer. For $G=4$ we add an
extra 4x4 stride 1 max pool layer.

The RNN has two parameters, the number of outputs per grid cell, $k$, and the
number of LSTM layers used. We have 2 LSTM layers and $k=8$ for all
experiments.

###  5.1 Evaluation on crabs dataset

The crabs dataset is prepared according to Section 4.1. The model was trained
on the crabs dataset (all 6 classes) for 15 epochs using standard data
augmentation methods (random rotations, shears, zooms and shifts) to prevent
overfitting. We obtain a mean average precision (mAP) of 24.14% for a 4x4 grid
and 19.32% for a 7x7 grid on test, however there is a large difference between
the top 3 and bottom 3 classes as seen in Table 3 (Sliced sequentially). This
is due to the dataset being heavily unbalanced.

A similar training scenario for the crabs-top3 dataset yields 37.5% mAP using
a 4x4 grid and 27.37% mAP for the 7x7 grid. Results are shown in Table 4
(Sliced sequentially).

As stated in Section 4.1, sequentially slicing the two mosaics results in a
relatively tiny dataset for deep models. We therefore slice the mosaics to
produce one image for each object by cutting out around the object
coordinates. This creates many overlapping images which can be seen as a data
augmentation technique. We pick one mosaic for training and one for testing.
The dev set is sliced around objects and the test set is sequentially sliced.
We trained the models for 15 epochs. We ran this experiment four times, using
each mosaic as training in turn and using all or only the top 3 classes. All
experiments overfit the training set and perform the poorest on the test set.
Results are in Tables 3 and 4 (Sliced around object).

Example predictions are shown in Figure 4, where we can observe the model does
large localisation errors but at the same time it predicts good population
densities. The locations of predicted crabs tend to form a circle centred near
the middle of the grid cells.

Figure 4: Example predictions on the crabs dataset. The grid is rendered in
the middle row for reference. Classes (species) are colour coded. For each
image pair, ground truth on left and prediction on right. Confidence threshold
0.5. Table 3: Model evaluation on the crabs dataset with all classes used. 7x7
and 4x4 denote the grid sizes used and slicing states the method of slicing
the mosaics into training images. For slicing around objects, the training set
is based on the NBC mosaic, the dev set on C0014G and the test set is
sequentially sliced. Class values show AP %, mAP in %.

| Sliced sequentially | Sliced around objects
---|---|---
| 4x4 | 7x7 | 4x4 | 7x7
| Train | Dev | Test | Train | Dev | Test | Train | Dev | Test | Train | Dev | Test
Bathymodiolus japonicus | 61.12 | 36.81 | 38.50 | 12.41 | 10.02 | 10.38 | 71.00 | 41.63 | 22.46 | 46.65 | 9.27 | 9.26
Thermosipho desbruyesi | 9.17 | 9.29 | 9.19 | 46.74 | 28.91 | 32.53 | 26.08 | 9.11 | 9.12 | 55.51 | 25.87 | 18.08
Bathymodiolus platifrons | 60.81 | 40.81 | 33.27 | 52.34 | 28.65 | 31.01 | 75.59 | 36.14 | 20.59 | 61.33 | 30.38 | 17.89
Paralomis | 12.49 | 10.02 | 9.59 | 11.74 | 9.48 | 9.54 | 14.89 | 9.30 | 9.72 | 20.72 | 9.15 | 9.24
Alvinocaridid | 11.98 | 10.30 | 11.04 | 50.82 | 30.34 | 23.37 | 46.43 | 9.81 | 9.71 | 60.18 | 28.05 | 16.38
Shinkaia crosnieri | 56.61 | 39.71 | 43.27 | 9.12 | 9.10 | 9.11 | 69.49 | 32.71 | 22.86 | 44.01 | 9.10 | 9.10
mAP | 35.36 | 24.49 | 24.14 | 30.53 | 19.42 | 19.32 | 50.58 | 23.12 | 15.74 | 48.07 | 18.64 | 13.33

Table 4: Model evaluation on the crabs-top3 dataset (only top 3 classes used).
7x7 and 4x4 denote the grid sizes used and slicing states the method of
slicing the mosaics into training images. For slicing around objects, the
training set is based on the NBC mosaic, the dev set on C0014G and the test
set is sequentially sliced. Class values show AP %, mAP in %.

| Sliced sequentially | Sliced around objects
---|---|---
| 4x4 | 7x7 | 4x4 | 7x7
| Train | Dev | Test | Train | Dev | Test | Train | Dev | Test | Train | Dev | Test
Bathymodiolus japonicus | 64.02 | 36.16 | 38.49 | 51.04 | 27.35 | 29.26 | 71.55 | 41.45 | 23.72 | 61.61 | 30.15 | 19.00
Bathymodiolus platifrons | 63.65 | 38.23 | 32.94 | 47.78 | 29.58 | 24.22 | 75.61 | 36.25 | 21.01 | 60.14 | 28.47 | 17.89
Shinkaia crosnieri | 59.66 | 35.71 | 40.61 | 41.55 | 25.66 | 28.63 | 68.97 | 32.60 | 24.06 | 55.29 | 24.15 | 17.26
mAP | 62.44 | 36.70 | 37.35 | 46.79 | 27.53 | 27.37 | 72.04 | 36.77 | 22.93 | 59.01 | 27.59 | 18.05

###  5.2 Evaluation on VOC12 dataset

Two models were trained on the VOC dataset, one using a 4x4 grid and one using
a 7x7 grid. The resulting mAPs on the dev test are 29.93% and 27.34%,
respectively, which are below the state of the art but the model parameters
are not fine-tuned for this dataset. We focus mainly on small objects that fit
into a grid cell. The training was done with no data augmentation. Table 5
shows per-class average precision (AP) scores and mAP results on train and dev
sets.

Large objects that span across many grid cells are predicted more than once,
but only considered correct in the cell containing the centre of the ground
truth label. This raises the level of false positives at test time and can
negatively influence training. For instance, if an object is in the middle
between two grid cells, both are equally qualified to make that prediction,
but only one will contain the object and during training this situation will
be regarded as an error in the other grid cell.

Example predictions are shown in Figure 5

Table 5: Evaluation results on the VOC12 detection dataset. Class results are Average Precision (AP). 4x4 and 7x7 denote the grid size of our model configuration. Input images resized to 224x224 pixels. No data augmentation was used and no fine-tuning of the model. Values in %. | Train 7x7 | Train 4x4 | Dev 7x7 | Dev 4x4
---|---|---|---|---
aeroplane | 51.66 | 52.74 | 36.69 | 34.29
bicycle | 55.19 | 52.00 | 22.46 | 23.37
bird | 42.37 | 52.01 | 29.04 | 33.65
boat | 45.87 | 50.67 | 24.00 | 27.47
bottle | 51.56 | 46.62 | 14.79 | 14.38
bus | 69.35 | 56.27 | 38.57 | 35.07
car | 52.61 | 56.92 | 27.99 | 32.24
cat | 47.69 | 61.27 | 31.56 | 38.95
chair | 39.57 | 52.36 | 18.57 | 22.95
cow | 57.38 | 56.83 | 25.57 | 31.30
diningtable | 54.00 | 69.85 | 20.41 | 24.91
dog | 51.58 | 62.07 | 31.08 | 35.35
horse | 49.49 | 61.69 | 23.50 | 32.43
motorbike | 55.91 | 58.58 | 29.69 | 28.17
person | 54.09 | 65.14 | 40.60 | 47.76
pottedplant | 40.74 | 49.22 | 18.11 | 21.33
sheep | 55.27 | 60.86 | 35.92 | 36.32
sofa | 52.88 | 65.97 | 16.83 | 20.28
train | 47.68 | 47.79 | 30.18 | 32.39
tvmonitor | 51.37 | 53.66 | 31.16 | 25.95
mAP | 51.31 | 56.63 | 27.34 | 29.93

Figure 5: Example predictions on the VOC12 dataset. For each image pair,
ground truth on left and prediction on right. Confidence score shown next to
object class name.

###  5.3 Evaluation on Steller Sea Lions Dataset

The aim of the NOAA Fisheries Kaggle competition is to predict population
counts, not to detect where objects are. The evaluation metric for the
competition is mean column-wise root mean squared error (RMSE). For N images
and K classes, it is:

$\frac{1}{K}\sum_{i}^{K}\sqrt{\frac{1}{N}\sum_{j}^{N}(y_{ij}-\hat{y}_{ij})^{2}},$
(3)

where the labels $y$ and predictions $\hat{y}$ are population counts.

We trained two models, grid 7x7 and grid 4x4. On the dev set we achieve 31.40%
and 38.33%, respectively. Training was done for 30 epochs. Mean Column-Wise
RMSE at 0.5 confidence threshold for 7x7 and 4x4 are 1.68 and 1.52,
respectively. Note this is on the sliced dev set, not on the large images from
the original dataset. The values will likely be higher on the larger images.
Results illustrated in Table 6. At the time of writing, the top position in
the competition leaderboard has 11.33729 RMSE on the official test set.

Table 6: Evaluation results on the Steller Sea Lions dataset. Class results are Average Precision (AP) %. 4x4 and 7x7 denote the grid size of our model configuration. Dataset images sequentially sliced in 224x224 input images. mAP in %. RMSE is Mean Column-Wise RMSE at 0.5 confidence threshold. | 7x7 train | 7x7 dev | 4x4 train | 4x4 dev
---|---|---|---|---
adult females | 35.68 | 33.00 | 45.03 | 41.92
adult males | 49.24 | 36.77 | 57.81 | 45.86
juveniles | 39.91 | 31.87 | 47.52 | 38.86
pups | 36.38 | 32.36 | 43.27 | 37.98
subadult males | 36.01 | 23.01 | 48.67 | 27.04
mAP | 39.44 | 31.40 | 48.46 | 38.33
RMSE | 1.69 | 1.68 | 1.49 | 1.52

##  6 Implementation

The model was developed with TensorFlow (Abadi et al., 2015). I used parts of
TensorBox333https://github.com/TensorBox/TensorBox (TensorFlow implementation
of (Stewart et al., 2016)) as a reference point. My implementation is an
object detection network as opposed to only object localisation like
TensorBox.

I implemented image and label readers for VOC, sea lions and crabs datasets
along with tools used for slicing images and labels. I implemented an
evaluation, prediction and visualisation tool. The predictions are stored in
an sqlite databases and can be later used for visualisation and further
analysis.

The visualisation tool allows viewing all trained models in a directory tree
along with relevant evaluation results on test, dev and training sets. The
visualisation tool is a python (with flask) web server and web page that shows
evaluation metrics, class counts, images and predictions on the images and
precision-recall plots. A screenshot is shown in Figure 6.

Figure 6: Screenshot of the visualisation tool built to aid the development of
this model. The predictions seen in the page are all misclassified.
Highlighting shows matching pair of ground truth and prediction.

## References

  * Abadi et al. (2015) Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.  TensorFlow: Large-scale machine learning on heterogeneous systems, 2015\.  URL http://tensorflow.org/.  Software available from tensorflow.org.
  * Everingham et al. (2015) M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.  The pascal visual object classes challenge: A retrospective.  _International Journal of Computer Vision_ , 111(1):98–136, January 2015.
  * Fritz et al. (2016) Lowell W Fritz, Kathryn M Sweeney, Rodney G Towell, and Thomas Scott Gelatt.  _Aerial and Ship-based Surveys of Steller Sea Lions (Eumetopias Jubatus) Conducted in Alaska in June-July 2013 Through 2015, and an Update on the Status and Trend of the Western Distinct Population Segment in Alaska_.  2016\.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _European Conference on Computer Vision_ , pp. 740–755. Springer, 2014.
  * NOAA Fisheries (2017) NOAA Fisheries.  Noaa fisheries steller sea lion population count, 2017.  URL https://www.kaggle.com/c/noaa-fisheries-steller-sea-lion-population-count.
  * Redmon et al. (2016) Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 779–788, 2016.
  * Stewart et al. (2016) Russell Stewart, Mykhaylo Andriluka, and Andrew Y Ng.  End-to-end people detection in crowded scenes.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 2325–2333, 2016.
  * Szegedy et al. (2015) Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 1–9, 2015.
  * Thornton et al. (2016) Blair Thornton, Adrian Bodenmann, Oscar Pizarro, Stefan B Williams, Ariell Friedman, Ryota Nakajima, Ken Takai, Kaori Motoki, Tomo-o Watsuji, Hisako Hirayama, et al.  Biometric assessment of deep-sea vent megabenthic communities using multi-resolution 3d image reconstructions.  _Deep Sea Research Part I: Oceanographic Research Papers_ , 116:200–219, 2016.

  *[CNN]: Convolutional Neural Network
  *[mAP]: mean average precision