# Instance Segmentation for Autonomous Log Grasping in Forestry Operations

Jean-Michel Fortin1, Olivier Gamache1, Vincent Grondin1, François Pomerleau1,
Philippe Giguère1 1The authors are with Northern Robotics Laboratory,
Université Laval, Québec City, Canada<EMAIL_ADDRESS>$\\{$olivier.gamache, vincent.grondin, francois.pomerleau$\\}$
@norlab.ulaval.ca<EMAIL_ADDRESS>research was supported
by the Natural Sciences and Engineering Research Council of Canada (NSERC)
through the grant CRD 538321-18, in collaboration with FP Innovations and
Resolute Forest Products.

###### Abstract

Wood logs picking is a challenging task to automate. Indeed, logs usually come
in cluttered configurations, randomly orientated and overlapping. Recent work
on log picking automation usually assume that the logs’ pose is known, with
little consideration given to the actual perception problem. In this paper, we
squarely address the latter, using a data-driven approach. First, we introduce
a novel dataset, named _TimberSeg 1.0_ , that is densely annotated, i.e., that
includes both bounding boxes _and_ pixel-level mask annotations for logs. This
dataset comprises 220 images with 2500 individually segmented logs. Using our
dataset, we then compare three neural network architectures on the task of
individual logs detection and segmentation; two region-based methods and one
attention-based method. Unsurprisingly, our results show that axis-aligned
proposals, failing to take into account the directional nature of logs,
underperform with 19.03 mAP. A rotation-aware proposal method significantly
improve results to 31.83 mAP. More interestingly, a Transformer-based
approach, without any inductive bias on rotations, outperformed the two
others, achieving a mAP of 57.53 on our dataset. Our use case demonstrates the
limitations of region-based approaches for cluttered, elongated objects. It
also highlights the potential of attention-based methods on this specific
task, as they work directly at the pixel-level. These encouraging results
indicate that such a perception system could be used to assist the operators
on the short-term, or to fully automate log picking operations in the future.

##  I INTRODUCTION

Forestry has seen a large mechanization effort, yet little has been done on
automating tasks requiring high-level cognition. In the last decade, other
industries such as agriculture and mining, made significant progress towards
automation. While facing different challenges, forestry is catching up towards
autonomous machines in the forest and mills [1]. Just like the ferrying of
ores has been one of the first tasks automated in mining [2], it is presumed
that _forwarding_ operations, i.e., extracting logs from the forest with heavy
machinery, will be the first candidate for automation [3]. Log picking is an
essential component in this forwarding task, but is challenging from a
perception and manipulation perspective. This exacerbates the ongoing manpower
shortage in forestry operations, as novice operators require lengthy training
to accomplish this repetitive task [4].

((a))

((b))

((c))

Figure 1: (a) View from an actual forestry forwarder, captured with one of our
dashcams during actual log picking operations. Mask2Former [5], the best
performing network trained on our dataset, predicts masks on wood logs in the
scene (right). We focus on detecting top logs from an upper point of view,
hence the undetected logs on the left. (b) Rotated bounding boxes (solid blue
line) are more adapted to detect logs because the latter are elongated and
randomly oriented. This is in contrast with axis-aligned bounding boxes
(dashed orange line), which will encompass other logs. (c) For localization
purposes, oriented boxes are not enough since crooked logs lead to the center
point being off, as shown by the stars. Therefore we focused on instance
segmentation with masks.

On the short run, teleoperation [6] could allow operators to work from remote
locations, while having access to better situational awareness with real-time
navigation and augmented reality. Other assistance systems, like the
Intelligent Boom Control (ICB), are also simplifying the work of operators,
thus reducing their cognitive load [7]. Taking advantage of these
technological advances, unmanned logging machines are getting closer to
reality. However, previous work on autonomous log handling [8] assume that the
position and orientation of the logs are known, which is generally not the
case.

Our work focuses primarily on this log perception problem, targeting
autonomous forestry forwarders and wood loaders. In computer vision, earlier
work on object detection focused on predicting bounding boxes encompassing the
target objects. The object’s center of mass, the usual grasp point for a log,
could then be assumed to be in the middle of the bounding box. However, this
assumption will fail for non-rectangular objects such as crooked wood logs, as
seen in 1(c). Thus, our system aims at segmenting individual logs in RGB
images from a camera placed in or above the operator’s cabin, as shown in
1(a). One reason that detection of wood logs is non-trivial is that they tend
to be cluttered by nature. Indeed, logs are elongated objects that are
generally found either randomly spread on the ground, or in piles throughout
the recently harvested forest or the lumberyard. Moreover, they suffer from
partial occlusion, either from branches or simply other logs, which makes
instance segmentation more challenging. Variations in illumination and weather
in outdoor locations are also challenging, especially in northern countries
such as Canada or Sweden. In these areas, forestry operations are performed
throughout the winter, thus in snowy conditions.

Our main contribution consists in characterizing data-driven approaches,
namely deep learning, on the task of wood logs segmentation. In this regard,
we provide a _densely annotated_ dataset with a diversity of scenarios, i.e.,
piled and individual logs, various seasons, environments, illuminations and
point of views. We also demonstrate the viability of speeding up manual data
labeling by using an earlier-trained network to suggest labels to a human
annotator, simplifying future annotation jobs. We then tested three instance
segmentation architectures. First, we used Mask R-CNN [9], a well-known and
widely used segmentation network, as a baseline. Its poor results show that
using axis-aligned bounding boxes as region proposals leads to sub-par quality
masks, mainly because they encompass other logs, as depicted in 1(b). Since
logs are elongated and not axis-aligned, an upgraded version using oriented
anchors [10] in the Region Proposal Networks (RPN) was considered. This helps
by reducing the presence of other logs in the Region-of-Interest (ROI).
Finally, we tested a recent, state-of-the-art transformer network called
Mask2Former [5] which shows promising results on the log segmentation task. In
particular, we show that this type of approach seems to be compatible with
long-shaped, cluttered objects.

In short, the contributions of this work are:

  * •

a demonstration of the potential and applicability of different deep neural
networks architectures on the challenging task of log detection; and

  * •

_TimberSeg 1.0_ : the first publicly available dataset of 220 RGB images of
logs in various environments, with dense annotation (bounding boxes and masks)
for 2500 logs.

##  II RELATED WORK

Our survey of the literature focuses on recent instance segmentation methods
and their variants, targeting applicability towards detecting elongated,
cluttered and overlapping objects. We also analyze some recent applications of
deep learning in forestry and agriculture, highlighting the absence of work
and datasets in autonomous log picking.

###  II-A Instance Segmentation Architectures

Segmentation is usually performed by adding an object mask prediction to
existing object detection methods for bounding box prediction. We can classify
instance segmentation approaches in multiple sub-categories below.

Single-stage methods work by directly making predictions from the feature maps
extracted by the backbone, often in a grid-like manner. One example is
YOLACT++ [11], which is extending RetinaNet’s [12] dense grid prediction by
adding a mask branch that combines prototype masks extracted from the feature
maps. The SOLOv2 network [13] instead deals directly with instance
segmentation, where each square of a grid is responsible for generating a
segmentation mask and a class label, without generating bounding boxes.

Two-stage methods are named so because of the intermediary step of predicting
ROIs, followed by a second stage classifying and refining the localization of
the object. Faster R-CNN [14] was the first to uses an RPN to predict these
ROIs, which are refined with a regression network. Mask R-CNN [9] later added
a mask branch for instance segmentation. Whereas single-stage methods are more
computationally efficient, two-stage methods yield better accuracy and recall
on the widely used COCO dataset [15].

Rotated proposal methods were subsequently developed to target randomly
oriented objects with elongated geometries. Indeed, Mask R-CNN displays strong
results on datasets containing regular-shaped objects such as COCO [15].
However, the axis-aligned and rectangular-shaped assumption of its RPN is
challenged when trained on cluttered or randomly oriented long objects, such
as scattered wood logs. Thus, [10] proposed a novel Rotation Region Proposal
Networks (RRPN), in their case for detecting non-axis aligned text. They added
an angle factor to the anchor boxes, bounding objects more precisely, as shown
in 1(b). This improved Rotated Mask R-CNN method was successfully applied on
remote sensing detection of ships in [16], which exhibit some visual
similarities with scattered logs.

Attention-based methods [17] are radically departing from previous approaches,
putting forward universal frameworks for all segmentation tasks. It started
with DETR [18], using a transformer encoder-decoder and queries to detect a
fixed number of objects. QueryInst [19] further adapted DETR to instance
segmentation, by adding dynamic mask heads. Also based on DETR, MaskFormer
[20] first segments the input image into a fixed number of masks and then
classifies them. Incremental changes brought forth by Mask2Former [5] provided
faster convergence and higher accuracy. Because Mask2Former works directly at
the pixel level through attention, they no longer include the inductive bias
of rectangular regions. Consequently, they can achieve better accuracy on
instance segmentation of logs (i.e., cluttered, elongated objects), as we will
demonstrate later.

###  II-B Deep Learning in Forestry and Agriculture

Many papers are applying vision and deep learning in forestry and agriculture.
Specifically, the problem of instance segmentation on fruits and vegetables
has received significant attention. For example, [21] applied Faster R-CNN and
Mask R-CNN for sweet pepper detection over multiple domains, demonstrating the
natural ability of the model to generalize when provided diverse enough data.
Mask R-CNN was also evaluated for strawberries detection [22] in a fruit-
picking robot. Their system visually separated ripe from unripe fruits, using
the predicted masks for stem localization. [23] slightly modified Mask R-CNN
for cucumber fruits detection, using the assumption that cucumbers are always
green. They used color filtering to remove background pixels, thus
accelerating the generation of anchor boxes in RPN. Their results show that
masks are necessary for precise center point estimation of elongated, non-
linear objects such as cucumbers, which relates well to wood logs. However, we
show in our experiments that Mask R-CNN is hardly applicable when objects are
superimposed and close to each other, partly because of the Non-Maximum
Supression (NMS) step. Moreover, the lack of uniformity in logs’ color, shape
and texture, as well as distinctiveness with the background precludes any
color or texture filtering approach.

In forestry, we notice a growing interest in public datasets for deep learning
tasks such as classification and detection. [24] released a large bark images
dataset for tree species identification. [25] published a dataset of standing
trees for bounding boxes detection and trained a YOLOv3 [26] network on it.
[27] trained this same network for detecting ground obstacles in a harvested
forest, on a custom dataset. Nevertheless, the latter two datasets only
provide bounding boxes ground truths, which can be useful to visually assist
the machine operator, but hardly applicable to precise localization. Moreover,
they rely heavily on the axis-aligned assumption, which may be suitable for
standing trees, but not for randomly piled wood logs.

The problem of detecting and localizing logs, both in harvested forests or in
lumberyards, has received little attention. More than a decade ago, [28]
demonstrated the potential of a structured light-based vision system to detect
and localize logs, predating deep learning. Experiments were conducted indoor,
greatly diminishing its applicability to outdoor environments. More recently,
[29] used convolutional neural networks for semantic segmentation of fallen
trees in aerial images, separating instances using shape parametrization. This
technique is unsuitable for log piles, since the masked pile can be blob-
shaped, making it impossible to distinguish individual instances
geometrically. More related to forwarders and wood loaders, motion planning to
grasp logs is analyzed in [8]. There, they used reinforcement learning to
control the crane motion of a forwarder. However, the target log’s pose is
assumed to be known in the aforementioned work. Our paper thus seeks to fill
the gap in the literature around wood logs detection, while providing data to
push research forward. Indeed, we conjecture that one of the reasons that
explains the lack of research on wood logs detection is the absence of
publicly available labeled datasets.

((a))

((b))

((c))

((d))

Figure 2: Dataset statistics. Top graphs show the variety of images in the
dataset, with images taken in different strategic locations with the number of
visible logs per image ranging from 1 to 30. Bottom graphs demonstrate how
logs are elongated and randomly oriented, making axis-aligned bounding boxes
irrelevant.

##  III METHODOLOGY

In this section, we first detail our approach for data collection and
labeling. Then, we describe each evaluated network architecture, to further
highlight their specificities in regards to our task at hand. Lastly, we
provide all the training details.

Figure 3: High-level comparison between tested instance segmentation networks.
Each of the three networks takes as input an image from our dataset, first
passing through a backbone for features extraction. Mask R-CNN determines ROIs
using an RPN with axis-aligned anchor boxes. The network’s heads then extract
bounding boxes and masks from these ROIs. Rotated Mask R-CNN uses rotated
anchor boxes to select rotated ROIs and uses similar heads to refine them into
oriented bounding boxes and masks. Mask2Former does not use region proposals,
instead working with multi-scale deformable attention on the whole feature
maps, at the pixel-level. The transformer decoder uses self-attention to
segment N regions from N trainable object queries and we filter the predicted
masks by score. The output for each network is displayed in the right column,
for an actual image from our dataset.

###  III-A Dataset Collection and Labeling

The dataset used in our experiments, called _TimberSeg 1.0_ , is composed of
220 images containing 2500 instances of wood logs in representative
environments and dispositions. Thus, these images were carefully selected to
capture typical viewpoints and situations that a forwarder or wood loader
operator would operate with. We can see in 2(a) that images have been
collected in four types of environments. _Forest_ corresponds mostly to
scattered logs in freshly cut areas, whereas _roadside_ consists of neatly
piled logs ready to be transported, as seen in 1(a). To collect this data,
three dashcams (_VIOFO A129 Pro Duo 4K_) were installed in forestry forwarders
operating in forests near Lake Saint-Jean, Quebec, Canada. Each dashcam
comprised two cameras, fixed on the inner side of the cabin’s windows. One
camera pointed forward, while the other pointed sideways. They recorded
hundreds of hours of video, at various resolutions (4K, 2K, 1080p) over
multiple months and weather conditions. Pictures, using a _Canon EOS M50_
digital camera, were also taken in _lumberyards_ from sawmills and paper
mills, including images of filled timber trucks’ _trailers_ from an upper
viewpoint. Finally, complementary images were taken from publicly available
videos from the Internet, to further increase diversity [21]. The forests
images are predominant on purpose, as they exhibit harder visual conditions.
The images contain a widely varying number of log instances as depicted in
2(b), ranging from 1 to 29 logs, for an average of 11.3. 2(c) shows the aspect
ratio of logs, which resembles a log-normal distribution, with a mean value of
8.3 and a tail distribution going as high as a ratio of 24. This is far from
the 2:1 or 1:2 aspect ratios typically used for anchor boxes, reflecting the
non-standardness of our problem. The distribution of log orientations,
presented in 2(d), shows a peak around $90^{\circ}$, indicating that a
significant portion of the logs are parallel to the $y$-axis of the camera.
Otherwise, logs do not display any other dominant orientation in images,
making their detection challenging to axis-aligned proposal approaches.

Our dataset was annotated using polygon mask segmentation with the
_SuperAnnotate_ tool. From a log picking perspective, it is sufficient to
detect and segment only the logs on top of a pile. Others will become
accessible as the pile is collected. Our first batch of images was labeled
manually, on which our most accurate network was trained to generate pre-
annotations and accelerate the labeling process. Similarly to active learning
approaches, a human annotator would then correct the mistakes made by the
network, which saved around $40\text{\,}\%$ of labeling time per image.

###  III-B Network Architectures

Three instance segmentation architectures are evaluated and compared on our
dataset: a vanilla Mask R-CNN, a Rotated Mask R-CNN and a Mask2Former network.
Figure 3 shows a visual depiction of the key elements of each architecture.

####  III-B1 Mask R-CNN

Given the historical importance of Mask R-CNN [9], it will serve as a baseline
for region-proposal methods. First, a Convolutional Neural Network (CNN)
backbone processes the input image to produce multi-resolution feature maps.
Then, the RPN uses a sliding window to evaluate the objectness of each
predefined axis-aligned anchor box, to generate region proposals. The
network’s heads independently predict a class, a bounding box and a pixel-wise
mask for every ROI. Testing Mask R-CNN will help highlight the limitations of
such axis-aligned approaches for our task at hand.

####  III-B2 Rotated Mask R-CNN

[10] showed that Mask R-CNN has limitations when objects in the scene are
dense or arbitrarily oriented. As a potential solution, they introduce a
rotation-based framework and propose the Rotation Region Proposal Networks
(RRPN), which adds an orientation to anchor boxes. This improvement makes ROIs
fit more tightly around objects (see 1(b)), reducing the amount of irrelevant
pixels and reaching for more accurate masks.

####  III-B3 Mask2Former

Attention-based architectures like Mask2Former [5] work directly at the pixel-
level. Similarly to Mask R-CNN, a backbone first extracts a pyramid of feature
maps. The usual backbone for this method is a Swin Transformer [30], but a CNN
backbone would also work. A six-layer multi-scale deformable attention
Transformer (MSDeformAttn) [31] then decodes the feature maps to generate
high-resolution per-pixel embeddings. Finally, a transformer decoder uses
masked attention to process these embeddings as well as object queries to
output masks and labels. This decoder is composed of nine layers handling
different resolutions of feature maps, depicted in a simplified manner in
Figure 3 by the A\- D labels.

###  III-C Training Details

We used the Detectron2111https://github.com/facebookresearch/detectron2
library based on PyTorch to train the models on single-class detection. Our
Mask R-CNN implementation mostly follows [9], with a ResNeXt-101 backbone. For
training, the initial learning rate was set to 0.001, reducing by a factor of
5 every 2000 iterations, with mini-batches of size 8. The model was optimized
using an Stochastic Gradient Descent (SGD) method with 0.9 momentum and 0.0001
weight decay. We trained for 8000 iterations, with the test set containing
$20\text{\,}\%$ on the images. The initial network’s weights had been
previously pretrained on ImageNet-1k, for the backbone, and then on COCO
dataset for the whole network.

The only changes applied to the rotated version of Mask R-CNN are the use of
RRPN and RRoIAlign [10] for oriented region proposals. Box anchors were
selected to have a 30 degrees orientation interval and aspect ratios of 4, 8,
12 and 16, following the dataset statistics displayed in 2(c). Both RPN and
RRPN use a smooth-L1 loss over the box parameters ($x$, $y$, width, height and
angle) where ($x$,$y$) are the center coordinates. We increased the number of
detections per image to 250 instead of the original 100, which slightly
improves accuracy and recall.

Mask2Former was also trained following most recommendations in the original
paper [5]. The feature extractor is a Swin-B backbone with a window size of
12, pretrained on ImageNet-1k and COCO. We set the number of object queries
for the decoder to 100. We used the same learning schedule and hyperparameters
as Mask R-CNN, except for the use of AdamW optimizer and a weight decay of
0.02.

We use the same data augmentation for each network, following the large-scale
jittering (LSJ) method proposed in [32], which was proven to be highly data
efficient. It consists of randomly resizing the images in a scale between 0.1
and 2.0 as well as cropping to a fixed size of 1024 $\times$ 1024 pixels with
gray padding and flipping horizontally. We also normalized images with
ImageNet’s mean and standard deviation for each color channel. The hardware
used for model training and testing consists of a single computer with a
_Nvidia RTX-3090_ GPU, an _Intel Core i9-10900KF_ CPU and 64 GB of RAM. Single
fold training time for Mask R-CNN was 2.5 hours, while Rotated Mask R-CNN took
5 hours per fold and Mask2Former only 45 minutes.

##  IV EXPERIMENTS & RESULTS

Multiple experiments were conducted to compare and analyze the performances of
selected networks on _TimberSeg 1.0_. In this section, we start by comparing
each architecture, both quantitatively and qualitatively. We seek to
corroborate our previous hypothesis that region-based approaches hinder good
instance segmentation, in the case of cluttered and elongated objects. We then
present further experiments on the best performing network, Mask2Former, to
better characterize its performances. All results are generated using a 5-fold
cross-validation policy, to fully leverage our dataset.

###  IV-A Networks Comparison

We evaluated the predicted masks using the standard COCO metric Mean Average
Precision (mAP), as well as Average Recall (AR) and the F1-score, each time
keeping the 100 best scoring detections. Table I presents a quantitative
comparison between the three architectures on the task of instance
segmentation of wood logs. We first note an important improvement of
approximately $12\text{\,}\%$ when switching to rotated region proposals
instead of using axis-aligned boxes. More importantly, it shows that
Mask2Former has a strong edge over two-stage networks on our application,
nearly doubling all three metrics compared to the rotated version of Mask
R-CNN. Pretraining the Mask2Former backbone on ImageNet-22k yields only
marginal improvements.

Table I also displays the inference speed for each network. The lower frame
rate obtained by Rotated Mask R-CNN can be explained by the higher number of
anchor box proposals. Indeed, a standard RPN uses 15 anchor box shapes,
whereas our RRPN implementation employs 84 because of the multiple
orientations. Despite using less computation than RRPN, Mask2Former exhibits a
significant accuracy improvement. Overall, it shows that detection in real
time or near real time is possible, opening the door to visual-servoing
methods.

TABLE I: Performance comparison of each tested network on our dataset, evaluated on predicted masks. Results show that using rotated regions in Mask R-CNN significantly improves accuracy, recall and thus F1-score. Mask2Former outperforms both approaches, using attention mechanisms based on Transformers [17]. Pretraining the Swin-B backbone on ImageNet-22k further improves results by a small margin. Architecture | Backbone | mAP | AR | F1-score | fps
---|---|---|---|---|---
Mask R-CNN | X101-FPN | 19.03 | 28.60 | 0.23 | 12.30
Rot. Mask R-CNN | X101-FPN | 31.83 | 36.95 | 0.34 | 5.44
Mask2Former | Swin-B (1k) | 56.05 | 64.06 | 0.60 | 8.33
Mask2Former | Swin-B (22k) | 57.53 | 65.16 | 0.61 | 8.34
Figure 4: Precision-Recall curves for various mask IoUs using our best
performing Mask2Former network. Model shows impressive performances for IoUs
up to 0.7.

Figure 4 shows the precision-recall curves for our Mask2Former implementation,
evaluated using mask IoU. As in Table I, these results are for single-frame
detection. When taking the 0.5 IoU curve, we can see that a precision greater
than $80\text{\,}\%$ can be achieved while maintaining an $80\text{\,}\%$
recall. In the context of log picking, a low recall implies that some logs
would be left behind, while a low precision means that the grapple often
misses the target. Since humans should not be around during these operations,
missing the target would not have devastating consequences other than slowing
down operations. Although still to be verified experimentally, we considered
that an IoU of 0.5 should be sufficient for a successful grasp. This takes
into account the fact that the center of mass of the predicted mask would be
in the actual log. Moreover, the opening width of a grapple is several times
the diameter of a log, accommodating for ample localization errors.

###  IV-B Power Law and Real-Time Potential

A fundamental question when trying to tackle a problem using data-driven
approaches is to estimate the amount of labeled data needed to solve it. To
this effect, we investigated the relationship between this training dataset
size and the observed mAP with Mask2Former. There, the number of training
images ranged from 20 to 160, with a fixed test set of 44 images. A linear
regression on the semi-log plot, as depicted by the purple line in 5(a), shows
that we are in the presence of a power law, as seen also by others using
Vision Transformers [33]. With some reserves, we can expect an improvement of
around $6.87\text{\,}\%$ each time the dataset size is doubled, until a
plateau is reached. In this regard, it is safe to assume that leveraging a
bigger dataset or semi-supervised methods would lead to better predictions.
These are, however, beyond the scope of this work.

In 5(b), we ran another experiment, this time varying the size of the image at
the network input. As the resolution increases, the network’s accuracy
improves, in line with what is observed in the literature. Conversely, frame
rate decreases due to increased computation needs to process larger images. By
varying the input image size, one can get an interesting trade-off between
speed and accuracy. For instance, the frame rate can go as high as
$24.05\text{\,}\mathrm{f}\mathrm{p}\mathrm{s}$ when the image size is reduced
to 256 $\times$ 256, accepting that precision will drop accordingly. We
observe an interesting sweet spot around an image size of 512 $\times$ 512,
where the inference speed can be tripled with only a $5\text{\,}\%$ drop in
precision.

((a))

((b))

Figure 5: Performances analysis of Mask2Former network. Bands are showing one
standard deviation. (a) The mAP follows a power law, as a function of the
number training images. Each doubling of the training dataset size yields an
improvement of $6.87\text{\,}\%$ on average. (b) Image resolution impacts both
inference speed and accuracy. Increasing the input image size in Mask2Former’s
network leads to better performances, but limits real-time applications.

###  IV-C Qualitative Results

Figure 6 shows a number of examples of images from our dataset, with their
annotation, for difficult images. Prediction outputs from Mask2Former are also
provided in this figure. The network displays good robustness to challenging
conditions such as snow, sun glare and night time operation.

0.5(0.1,0.30)  Ground Truth 0.5(0.1,2.13)  Predictions

((a)) Fall

((b)) Sun glare

((c)) Winter

((d)) Night

Figure 6: Examples of segmentation results from trained Mask2Former network on
our dataset (bottom row) compared to ground truth labels (top row). Notice how
the network is capable of obtaining good results in the presence of
distractors such as grapple (a), sunglare (b), winter conditions (c) and
vignetting due to artificial illumination during night operation (d).

##  V CONCLUSION

In this paper, we addressed the problem of log picking automation in forestry
operations. To this effect, we created a novel dataset of log images, some of
them taken during the actual operation of a forwarder. The _TimberSeg 1.0_
dataset, which will be made public upon publication, contains 220 images with
2500 logs precisely annotated with bounding boxes and masks. Using our
dataset, we then compared the applicability of three deep neural network
architectures on the task of wood log segmentation. Our results showed that
Mask2Former, an approach based on attention, works best on this type of
cluttered, occluded and randomly oriented objects, as opposed to the more
traditional region-proposal based methods. Our results demonstrate the
viability of such perception system, which could be used for operators’ aid in
teleoperation or as a stepping stone towards autonomous forwarders and wood
loaders. Our validation also opens the door to automatic roadside logpile
inventories, by simply using a camera mounted on a vehicle.

For future works, detection accuracy could be improved in several ways. On the
data side, we plan on leveraging the large volume of unlabeled videos that we
collected with our dashcams. Semi- and self-supervised learning methods, such
as Masked Auto-encoders [34], could then be used in pretraining the backbone
to our application domain. Improved data augmentation, such as _copy-paste_
[32] could also be employed to reduce dependency on labeled images. On the
algorithm side, our approach is currently single frame. As such, our results
constitute a worst-case scenario, where planning of log grasps would be based
on detections coming from a single image. Techniques based on Bayesian
filtering could be employed to improve detection reliability, explicitly
tracking log instances across video frames. Implicit approaches performing
detection directly from video streams [35] could also be envisioned.

## References

  * [1] Luiz FP Oliveira, António P Moreira and Manuel F Silva  “Advances in Forest Robotics: A State-of-the-Art Survey”  In _Robotics_ 10.2 Multidisciplinary Digital Publishing Institute, 2021, pp. 53
  * [2] Joshua Marshall, Timothy Barfoot and Johan Larsson  “Autonomous underground tramming for center-articulated vehicles”  In _Journal of Field Robotics_ 25.6-7 Wiley Online Library, 2008, pp. 400–421
  * [3] Ola Ringdahl  “Automation in forestry: development of unmanned forwarders”, 2011
  * [4] Daniel Ortiz Morales et al.  “Increasing the Level of Automation in the Forestry Logging Process with Crane Trajectory Planning and Control”  In _Journal of Field Robotics_ 31.3 Wiley Online Library, 2014, pp. 343–363
  * [5] Bowen Cheng et al.  “Masked-attention Mask Transformer for Universal Image Segmentation”  In _arXiv preprint arXiv:2112.01527_ , 2021
  * [6] Tobias Persson  “Design of a workstation for teleoperated forwarders: Exploring the future work within forestry”, 2020
  * [7] Tomáš Zemánek and Petr Fil’o  “Influence of Intelligent Boom Control in Forwarders on Performance of Operators”  In _Croatian Journal of Forest Engineering_ 43.1 Fakultet šumarstva i drvne tehnologije Sveučilišta u Zagrebu, 2022, pp. 47–64
  * [8] Jennifer Andersson et al.  “Reinforcement Learning Control of a Forestry Crane Manipulator”  In _arXiv:2103.02315_ , 2021
  * [9] Kaiming He, Georgia Gkioxari, Piotr Dollár and Ross Girshick  “Mask R-CNN”  In _IEEE International Conference on Computer Vision_ , 2017, pp. 2961–2969
  * [10] Jianqi Ma et al.  “Arbitrary-Oriented Scene Text Detection via Rotation Proposals”  In _IEEE Transactions on Multimedia_ 20.11 IEEE, 2018, pp. 3111–3122
  * [11] Daniel Bolya, Chong Zhou, Fanyi Xiao and Yong Jae Lee  “YOLACT++ Better Real-Time Instance Segmentation”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 44.2, 2022, pp. 1108–1121
  * [12] Tsung-Yi Lin et al.  “Focal Loss for Dense Object Detection”  In _IEEE International Conference on Computer Vision_ , 2017, pp. 2980–2988
  * [13] Xinlong Wang et al.  “SOLOv2: Dynamic and Fast Instance Segmentation”  In _Advances in Neural Information Processing Systems_ 33, 2020, pp. 17721–17732
  * [14] Shaoqing Ren, Kaiming He, Ross Girshick and Jian Sun  “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks”  In _Advances in Neural Information Processing Systems_ 28, 2015, pp. 91–99
  * [15] Tsung-Yi Lin et al.  “Microsoft COCO: Common Objects in Context”  In _European Conference on Computer Vision_ , 2014, pp. 740–755
  * [16] Jamyoung Koo et al.  “RBox-CNN: Rotated Bounding Box based CNN for Ship Detection in Remote Sensing Image”  In _ACM International Conference on Advances in Geographic Information Systems (SIGSPATIAL)_ , 2018, pp. 420–423
  * [17] Ashish Vaswani et al.  “Attention Is All You Need”  In _Advances in Neural Information Processing Systems_ 30, 2017, pp. 6000–6010
  * [18] Nicolas Carion et al.  “End-to-End Object Detection with Transformers”  In _European Conference on Computer Vision_ , 2020, pp. 213–229
  * [19] Yuxin Fang et al.  “Instances as Queries”  In _IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 6910–6919
  * [20] Bowen Cheng, Alex Schwing and Alexander Kirillov  “Per-Pixel Classification is Not All You Need for Semantic Segmentation”  In _Advances in Neural Information Processing Systems_ 34, 2021
  * [21] Michael Halstead, Simon Denman, Clinton Fookes and Chris McCool  “Fruit Detection in the Wild: The Impact of Varying Conditions and Cultivar”  In _IEEE Digital Image Computing: Techniques and Applications_ , 2020, pp. 1–8
  * [22] Yang Yu, Kailiang Zhang, Li Yang and Dongxing Zhang  “Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN”  In _Computers and Electronics in Agriculture_ 163 Elsevier, 2019, pp. 104846
  * [23] Xiaoyang Liu et al.  “Cucumber Fruits Detection in Greenhouses Based on Instance Segmentation”  In _IEEE Access_ 7 IEEE, 2019, pp. 139635–139642
  * [24] Mathieu Carpentier, Philippe Giguère and Jonathan Gaudreault  “Tree Species Identification from Bark Images Using Convolutional Neural Networks”  In _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2018, pp. 1075–1081
  * [25] Daniel Queirós Silva, Filipe Neves Dos Santos, Armando Jorge Sousa and Vítor Filipe  “Visible and Thermal Image-Based Trunk Detection with Deep Learning for Forestry Mobile Robotics”  In _Journal of Imaging_ 7.9 Multidisciplinary Digital Publishing Institute, 2021, pp. 176
  * [26] Joseph Redmon and Ali Farhadi  “YOLOv3: An Incremental Improvement”  In _arXiv preprint arXiv:1804.02767_ , 2018
  * [27] Songyu Li and Håkan Lideskog  “Implementation of a System for Real-Time Detection and Localization of Terrain Objects on Harvested Forest Land”  In _Forests_ 12.9 Multidisciplinary Digital Publishing Institute, 2021, pp. 1142
  * [28] Yeonchool Park, Anton Shiriaev, Simon Westerberg and Sukhan Lee  “3D Log Recognition and Pose Estimation for Robotic Forestry Machine”  In _IEEE International Conference on Robotics and Automation_ , 2011, pp. 5323–5328
  * [29] Przemyslaw Polewski, Jacquelyn Shelton, Wei Yao and Marco Heurich  “Instance segmentation of fallen trees in aerial color infrared imagery using active multi-contour evolution with fully convolutional network-based intensity priors”  In _ISPRS Journal of Photogrammetry and Remote Sensing_ 178, 2021, pp. 297–313  DOI: https://doi.org/10.1016/j.isprsjprs.2021.06.016
  * [30] Ze Liu et al.  “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows”  In _IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 10012–10022
  * [31] Xizhou Zhu et al.  “Deformable DETR: Deformable Transformers for End-to-End Object Detection”  In _arXiv preprint arXiv:2010.04159_ , 2020
  * [32] Golnaz Ghiasi et al.  “Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 2918–2928
  * [33] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby and Lucas Beyer  “Scaling Vision Transformers”  In _ArXiv_ abs/2106.04560, 2021
  * [34] Kaiming He et al.  “Masked Autoencoders Are Scalable Vision Learners”  In _ArXiv_ abs/2111.06377, 2021
  * [35] Linjie Yang, Yuchen Fan and Ning Xu  “Video Instance Segmentation”  In _IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 5188–5197

  *[ICB]: Intelligent Boom Control
  *[RPN]: Region Proposal Networks
  *[ROI]: Region-of-Interest
  *[RRPN]: Rotation Region Proposal Networks
  *[NMS]: Non-Maximum Supression
  *[CNN]: Convolutional Neural Network
  *[SGD]: Stochastic Gradient Descent
  *[LSJ]: large-scale jittering
  *[mAP]: Mean Average Precision
  *[AR]: Average Recall
  *[IoU]: Intersection over Union


# Modulation of early visual processing alleviates capacity limits
in solving multiple tasks

Sushrut Thorat Giacomo Aldegheri11footnotemark: 1 Marcel A. J. van Gerven
Marius V. Peelen
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
Donders Institute for Brain, Cognition and Behaviour, Radboud University Equal
contribution

## Abstract

In daily life situations, we have to perform multiple tasks given a visual
stimulus, which requires task-relevant information to be transmitted through
our visual system. When it is not possible to transmit all the possibly
relevant information to higher layers, due to a bottleneck, task-based
modulation of early visual processing might be necessary. In this work, we
report how the effectiveness of modulating the early processing stage of an
artificial neural network depends on the information bottleneck faced by the
network. The bottleneck is quantified by the number of tasks the network has
to perform and the neural capacity of the later stage of the network. The
effectiveness is gauged by the performance on multiple object detection tasks,
where the network is trained with a recent multi-task optimisation scheme. By
associating neural modulations with task-based switching of the state of the
network and characterising when such switching is helpful in early processing,
our results provide a functional perspective towards understanding why task-
based modulation of early neural processes might be observed in the primate
visual cortex111The code to train and analyse the networks mentioned here can
be found at - https://github.com/novelmartis/early-vs-late-multi-
task222Accepted into the 2019 Conference on Cognitive Computational
Neuroscience: https://doi.org/10.32470/CCN.2019.1229-0.

> Keywords: neural modulation, multi-task learning, early visual cortex,
> attention, perception, capacity limits

## Introduction

Humans and other animals have to perform multiple tasks given a visual
stimulus. For example, seeing a face, we may have to say whether it is happy
or sad, or recognise its identity. For each of these tasks, a subset of all
the features of the face are useful. In principle, it could be possible for a
visual system to extract all of the features necessary to solve all possible
tasks, and then select the relevant information from this rich representation
downstream. However, as the number of tasks increases, a network with a
limited capacity may not be able to extract all of the potentially relevant
features (an information bottleneck is manifest), requiring the information
that is extracted from the stimulus in the early processing stages to change
according to the task.

Several studies in neuroscience have found evidence for such task-dependent
modulations of sensory processing in the primate visual system, including at
the early levels Carrasco (2011); Maunsell Treue (2006); Gilbert Li (2013).
For example, human neuroimaging studies have shown that attending to a
stimulus could lead to an increase in the accuracy with which its task-
relevant features could be decoded by a classifier in early visual areas Jehee
. (2011), and neurophysiological experiments in nonhuman primates have shown
that the stimulus selectivity of neurons in primary visual cortex was
dependent on the task the monkeys had to perform Gilbert Li (2013).

Despite the observation of such modulations of early visual processing, it is
not clear whether they are causally necessary for performing better on the
corresponding tasks. This question has been addressed by deploying
biologically-inspired task-based modulations on computational models.
lindsay2018biological showed that task-based modulation deployed on multiple
stages of a convolutional neural network improves performance on challenging
object classification tasks. Other recent work Thorat . (2018); Rosenfeld .
(2018) has also shown that task-based modulation of early visual processing
aids in object detection and segmentation in addition to the task-based
modulation of late processing. However, the conditions under which early
modulation can be beneficial in performing multiple tasks have not been
systematically investigated.

In the present work, we assessed the effectiveness of task-based modulation of
early visual processing as a function of an information bottleneck in a neural
network, quantified by the number of tasks the network had to execute and the
neural capacity of the network. To do so, we trained networks to, given an
image, provide an answer conditioned on the cued task. Every task required
detecting the presence of the corresponding object in the image. The networks
were trained according to a recent framework proposed in the field of
continual learning Cheung . (2019), which helps them execute multiple tasks by
switching their state given a task cue, in order to transmit relevant
information through the network. In this work, to quantify the effectiveness
of task-based modulation of early neural processing, we measured the increase
in performance provided by modulating early neural processing in addition to
modulating the late neural processing in the networks.

## Methods

### Task and system description

In a multi-task setting, object detection can be thought of as solving one of
a set of possible binary classification (one object versus the rest) problems.
Given an image and a task cue indicating the identity of the object to be
detected, a network had to output if the object in the image matched the task
cue.

We used MNIST LeCun . (1998) digits and their permutations as objects
Kirkpatrick . (2017). The original MNIST dataset has $28\times 28\,$px2 images
of $10$ digits. Each permuted version consists of images of those $10$ digits,
whose pixels undergo a given permutation, creating $10$ new objects. We varied
the number of permutations used ($10$, $25$, and $50$) to modulate the number
of tasks the networks had to perform (which are $10$ times the number of
permutations).

We considered a multi-layer perceptron with rectified linear units (ReLU),
which had one hidden layer between the input (image) and the binary output.
The number of neurons in the hidden layer were variable ($32$, $64$, and
$128$) and determined the neural capacity of the late stage of the network.

### Task-based modulation and its function

Modelling biological neurons as perceptrons Rosenblatt (1957), task-based
modulations have been shown to affect the effective biases and gains of the
neurons Maunsell Treue (2006); Boynton (2009); Ling . (2009). The nature of
modulation - which neurons to modulate and how - is under debate Boynton
(2009); Thorat . (2018). We adapted these findings by introducing task-based
modulation into our networks via the biases of the perceptrons and the gains
of their ReLU activation functions. The modulations were then trained end-to-
end with the rest of the network.

Given a particular task, the task cue is a one-hot encoding of the relevant
object. Task-based modulation is mediated through bias and gain modulation in
the following manner.

$\displaystyle x_{n}$ $\displaystyle=[W_{n}(g_{n-1}\circ x_{n-1})+b_{n}]_{+}$
(1) $\displaystyle g_{n}$ $\displaystyle=G_{n}c,\,\,\,b_{n}=B_{n}c$ (2)

where the transformation between layers $n-1$ and $n$ ($L_{n-1}\rightarrow
L_{n}$) is modulated by changing the slope of the ReLU activation function
(gains, $g_{n-1}$) in $L_{n-1}$ and the biases ($b_{n}$) to the perceptrons in
$L_{n}$; $x_{n}$ are the pre-gain activations of the perceptrons in $L_{n}$,
$W_{n}$ is the task-independent transformation matrix between $L_{n-1}$ and
$L_{n}$, $G_{n}$ and $B_{n}$ map the task cue $c$ (one-hot encoding of the
relevant object $k$) to the gain and bias modulations of the perceptrons in
$L_{n}$ respectively, and $\circ$ refers to element-wise multiplication.

Given a task $k$, modulating the gains of the pre-synaptic perceptrons (in
$L_{n-1}$) and the biases of the post-synaptic perceptrons (in $L_{n}$)
transforms the information transformation between $L_{n-1}$ and $L_{n}$. This
allows for the transmission of information required to perform task $k$, while
ignoring the information required to perform the other tasks, as formalised in
cheung2019superposition. This transformation can also be thought of as the
network switching its state to selectively transmit task-relevant information
downstream (see Figure 1). The conditions - the nature of these modulations
and the neural capacity of the network - under which the network can switch
between a given number of tasks, are preliminarily described in
cheung2019superposition.

Here, for every relevant layer $L_{n}$, $W_{n}$, $B_{n}$, and $G_{n}$ were
jointly learned for the given number of tasks.

Figure 1: The effect of bias and gain modulation on the transformations in the
network. Modulating the gains and biases is functionally equivalent to
switching the transformation being performed to one suited for the relevant
task. Such an example of switching is visualised in the figure. Given a task
cue corresponding to object $k$, corresponding gain and bias modulations are
applied, which results in the $L_{n-1}\rightarrow L_{n}$ transformation being
switched into one that transmits feature information required to detect the
presence or absence of object $k$.

### Evaluation metric and expected trends

The effectiveness of early neural modulation was quantified by the average
absolute increase in detection performance across all the tasks when
modulations were implemented on both the transformations $L_{1}\rightarrow
L_{2}$ and $L_{2}\rightarrow L_{3}$ ($L_{1}$ corresponds to the input layer
and $L_{3}$ to the output layer) as opposed to when the modulations were
trained on the transformation $L_{2}\rightarrow L_{3}$ only.

We expected the effectiveness of task-based early neural modulation to be
directly proportional to the number of neurons in $L_{2}$ and inversely
proportional to the number of tasks (permuted MNIST sets used).

### Neural network training details

All the networks were trained with adaptive stochastic gradient descent with
backpropagation through the ADAM optimiser Kingma Ba (2014) with the default
settings in TensorFlow (v1.4.0) and $\alpha=10^{-5}$. We used a batch size of
$100$. Half of each batch contained randomly selected images of randomly
selected tasks where the cued object was present, and half where the cued
object was not present. These images were taken from the MNIST training set
and its corresponding permutations. The images were augmented by adding small
translations and noise. We trained each network with $10^{7}$ such batches.
The relevant metrics discussed in the previous section are computed at the end
of training over a batch of size $10^{5}$ created from the MNIST test set and
its corresponding permutations.

## Results

We first analysed the detection performance of the network with only
$L_{2}\rightarrow L_{3}$ modulation. The network performance as a function of
the number of neurons in $L_{2}$ and the number of detection tasks the network
had to perform is shown in Figure 2 (red circles). The network performance
increased with an increase in the number of neurons in $L_{2}$, as the neural
capacity increased. The performance decreased with an increase in the number
of tasks to be performed, as the representational capacity of the network for
any one task was reduced. A network with as little as $32$ neurons in its
hidden layer was able to switch between as many as $500$ detection tasks,
while keeping the average detection performance across all the tasks as high
as $87\%$, thus replicating the success of the multi-task learning framework
proposed by cheung2019superposition.

To assess the dependence of the effectiveness of task-based modulation of
early neural processing ($L_{1}\rightarrow L_{2}$) on the bottleneck in the
network, we analysed the boost in average detection performance when task-
based modulation of $L_{1}\rightarrow L_{2}$ was deployed in addition to task-
based modulation of $L_{2}\rightarrow L_{3}$, as a function of the number of
neurons in $L_{2}$ and the number of detection tasks the network had to
perform. The resulting boosts are shown in Figure 2 ($\Delta_{\uparrow}$
quantification). The performance boost increased as the number of neurons in
$L_{2}$ decreased, and as the number of tasks the network has to perform
increased. This confirms the hypothesis that task-based modulation of early
neural processing is essential when an information bottleneck exists in a
subsequent processing stage (see Appendix for further analyses elucidating
these results).

### The contribution of bias and gain modulation

Gain, but not so much bias, modulation of neural responses has been observed
in experiments investigating feature-based attention in the monkey/human brain
Maunsell Treue (2006); Boynton (2009). We assessed how the two contributed to
the overall modulation of the transformations in the network.

We selectively turned off the bias or gain modulation for all the variants of
the network that were trained. The average detection performance decreased by
$43.0\pm 2.0\%$ when gain modulation was turned off, and by $3.9\pm 0.9\%$
when bias modulation was turned off, suggesting that in our framework, when
jointly deployed, gain modulation is more important than bias modulation in
switching the state of the network to be able to perform the desired task
well.

We also trained a network with $32$ neurons in $L_{2}$, on $25$ permutations
of MNIST, with gain-only or bias-only modulations of both the
$L_{1}\rightarrow L_{2}$ and $L_{2}\rightarrow L_{3}$ transformations. When
the gain and bias modulations were jointly trained, the network performance
was $94.7\%$. With gain-only modulation, the performance was $94.8\%$, and
with bias-only modulation the performance was $90.9\%$. As the performance
when only bias modulation was deployed was much higher than chance ($50\%$),
we can conclude that bias modulation alone can also lead to efficient task-
switching. When the bias and gain modulations are jointly trained, gain might
take over as it multiplicatively impacts responses, and therefore has higher
gradients during training, as opposed to the additive impact of bias.

Figure 2: The effectiveness of task-based modulation (quantified by the
performance boost, $\Delta_{\uparrow}$) of early neural processing
($L_{1}\rightarrow L_{2}$) as a function of the number of neurons in $L_{2}$
and the number of tasks the network has to perform. The performance boost was
inversely proportional to the number of neurons in $L_{2}$ and directly
proportional to the number of tasks the network had to perform. The absolute
performance profiles given either the modulation of $L_{2}\rightarrow L_{3}$
only or the joint modulation of $L_{2}\rightarrow L_{3}$ and $L_{1}\rightarrow
L_{2}$ are also shown. (See Appendix for further substantiation of these
results)

## Discussion

Adding to the discussion about the functional role of task-based modulation of
early neural processing, in this work we have shown that modulating the early
layer of an artificial neural network in a task-dependent manner can boost
performance, beyond just modulating the late layer, in a multi-task learning
scenario in which a network contains an information bottleneck, either due to
a large number of tasks to be performed or to a small number of units in the
late layer.

Adapting a formalism proposed by cheung2019superposition, we showed how bias
and gain modulation, two prevalent neuronal implementations of top-down
modulation in the brain, could functionally lead to switching the state of a
network to perform transformations effective for the task at hand. While task-
dependent computations are widespread in higher-level areas of the primate
brain, such as prefrontal cortex Mante . (2013), it is not clear to what
extent sensory streams (which perform early visual processing) can also be
seen as switching their state according to the current task (although see
gilbert2013top for a proposal), and what the functional relevance of doing so
would be. Here we show how, in principle, this switching could be
computationally advantageous when it is not possible to send the information
required for all tasks to higher layers, which might well be the case in the
complex environments that humans and other animals are able to navigate.

To further investigate the relevance of our findings to biological visual
systems, in follow-up work we intend to deploy our modulation scheme on
architectures that bear more similarity to the primate visual hierarchy, such
as deep convolutional networks Kriegeskorte (2015), datasets of naturalistic
images such as ImageNet Russakovsky . (2015), and general naturalistic tasks
such as visual question answering Agrawal . (2017). This will allow us to
assess whether the functional advantage provided by early modulation holds
true in a more realistic scenario, and whether the resulting modulation
schemes resemble those observed in the early visual areas of the primate
brain.

Finally, a key aspect of our approach is the fact that the network is
constantly operating in a task-dependent manner. Most previous approaches to
task-dependent modulation have assumed the presence of an underlying task-free
representation on which the modulation operates (for example, in the case of
lindsay2018biological this corresponds to a network pre-trained on object
recognition). Providing the network with task cues during the training phase,
on the other hand, has been used in the field of continual learning Cheung .
(2019); Masse . (2018); Yang . (2019), and according to one influential theory
in neuroscience, the interplay between sparse, context-specific information
encoded by the hippocampus and shared structural information in the neo-cortex
is crucial for learning new tasks without overwriting previous ones Kumaran .
(2016). To our knowledge, the question of how the task-based modulations
observed in visual cortex might be learned has not been explicitly addressed
in previous literature. On the one hand, it is possible that a context-free
representation is learned first, possibly through unsupervised learning, and
then modulated upon. On the other, learning of representations and task
modulations might interact at all stages, allowing the representations to be
optimised for the type of modulations they are subject to. Whether one scheme
or the other constitutes a better explanation for the modulations observed in
biological visual systems is an important direction for future research.

## Acknowledgments

This work was funded by the European Research Council (ERC) under the European
Union’s Horizon 2020 research and innovation program (Grant Agreement No.
$725970$). This manuscript reflects only the authors’ view, and the agency is
not responsible for any use that may be made of the information it contains.

## References

  * Agrawal . (2017) agrawal2017vqaAgrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, CL., Parikh, D. Batra, D.  2017\.  Vqa: Visual question answering Vqa: Visual question answering.  International Journal of Computer Vision12314–31.
  * Boynton (2009) boynton2009frameworkBoynton, GM.  2009\.  A framework for describing the effects of attention on visual responses A framework for describing the effects of attention on visual responses.  Vision Research49101129–1143.
  * Carrasco (2011) carrasco2011visualCarrasco, M.  2011\.  Visual attention: The past 25 years Visual attention: The past 25 years.  Vision Research51131484–1525.
  * Cheung . (2019) cheung2019superpositionCheung, B., Terekhov, A., Chen, Y., Agrawal, P. Olshausen, B.  2019\.  Superposition of many models into one Superposition of many models into one.  arXiv preprint arXiv:1902.05522.
  * Gilbert Li (2013) gilbert2013topGilbert, CD. Li, W.  2013\.  Top-down influences on visual processing Top-down influences on visual processing.  Nature Reviews Neuroscience145350–363.
  * Jehee . (2011) jehee2011attentionJehee, JF., Brady, DK. Tong, F.  2011\.  Attention improves encoding of task-relevant features in the human visual cortex Attention improves encoding of task-relevant features in the human visual cortex.  Journal of Neuroscience31228210–8219.
  * Kingma Ba (2014) kingma2014adamKingma, DP. Ba, J.  2014\.  Adam: A method for stochastic optimization Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980.
  * Kirkpatrick . (2017) kirkpatrick2017overcomingKirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, AA.others  2017\.  Overcoming catastrophic forgetting in neural networks Overcoming catastrophic forgetting in neural networks.  Proceedings of the National Academy of Sciences114133521–3526.
  * Kriegeskorte (2015) kriegeskorte2015deepKriegeskorte, N.  2015\.  Deep neural networks: a new framework for modeling biological vision and brain information processing Deep neural networks: a new framework for modeling biological vision and brain information processing.  Annual Review of Vision Science1417–446.
  * Kumaran . (2016) kumaran2016learningKumaran, D., Hassabis, D. McClelland, JL.  2016\.  What learning systems do intelligent agents need? Complementary learning systems theory updated What learning systems do intelligent agents need? complementary learning systems theory updated.  Trends in Cognitive Sciences207512–534.
  * LeCun . (1998) lecun1998gradientLeCun, Y., Bottou, L., Bengio, Y. Haffner, P.  1998\.  Gradient-based learning applied to document recognition Gradient-based learning applied to document recognition.  Proceedings of the IEEE86112278–2324.
  * Lindsay Miller (2018) lindsay2018biologicalLindsay, GW. Miller, KD.  2018\.  How biological attention mechanisms improve task performance in a large-scale visual system model How biological attention mechanisms improve task performance in a large-scale visual system model.  eLife7e38105.
  * Ling . (2009) ling2009spatialLing, S., Liu, T. Carrasco, M.  2009\.  How spatial and feature-based attention affect the gain and tuning of population responses How spatial and feature-based attention affect the gain and tuning of population responses.  Vision Research49101194–1204.
  * Mante . (2013) mante2013contextMante, V., Sussillo, D., Shenoy, KV. Newsome, WT.  2013\.  Context-dependent computation by recurrent dynamics in prefrontal cortex Context-dependent computation by recurrent dynamics in prefrontal cortex.  Nature503747478.
  * Masse . (2018) masse2018alleviatingMasse, NY., Grant, GD. Freedman, DJ.  2018\.  Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization.  Proceedings of the National Academy of Sciences11544E10467–E10475.
  * Maunsell Treue (2006) maunsell2006featureMaunsell, JH. Treue, S.  2006\.  Feature-based attention in visual cortex Feature-based attention in visual cortex.  Trends in Neurosciences296317–322.
  * McNemar (1947) mcnemar1947noteMcNemar, Q.  1947\.  Note on the sampling error of the difference between correlated proportions or percentages Note on the sampling error of the difference between correlated proportions or percentages.  Psychometrika122153–157.
  * Rosenblatt (1957) rosenblatt1957perceptronRosenblatt, F.  1957\.  The perceptron, a perceiving and recognizing automaton The perceptron, a perceiving and recognizing automaton.  Cornell Aeronautical Laboratory.
  * Rosenfeld . (2018) rosenfeld2018primingRosenfeld, A., Biparva, M. Tsotsos, JK.  2018\.  Priming Neural Networks Priming neural networks.  Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops ( 2011–2020).
  * Russakovsky . (2015) russakovsky2015imagenetRussakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S.others  2015\.  Imagenet large scale visual recognition challenge Imagenet large scale visual recognition challenge.  International Journal of Computer Vision1153211–252.
  * Thorat . (2018) thorat2018functionalThorat, S., van Gerven, M. Peelen, M.  2018\.  The functional role of cue-driven feature-based feedback in object recognition The functional role of cue-driven feature-based feedback in object recognition.  Conference on Cognitive Computational Neuroscience, CCN 2018 Conference on Cognitive Computational Neuroscience, CCN 2018 ( 1–4).
  * Yang . (2019) yang2019taskYang, GR., Joglekar, MR., Song, HF., Newsome, WT. Wang, XJ.  2019\.  Task representations in neural networks trained to perform many cognitive tasks Task representations in neural networks trained to perform many cognitive tasks.  Nature Neuroscience222297.

## Appendix

Here we present the results from further analyses performed to address
reviewer comments, post-acceptance into the 2019 Conference on Cognitive
Computational Neuroscience.

### Dependence of the results on parameter expansion

When early modulation is performed in addition to late modulation, the
addition of parameters is constant across the different networks with varying
number of hidden neurons. However, the relative increase in the number of
parameters is higher in the network with lower number of neurons in the hidden
layer. This network, with lower number of hidden neurons, also benefits most
from the addition of early modulation, as observed in Figure 2. So, are the
observations in Figure 2 an effect of simple proportionate parameter
expansion? Two additional analyses show that this is not the case.

#### Matching the proportionate increase in the number of parameters across
networks

To match the proportionate increase in parameters, we introduced an additional
layer between the task cue $c$ and the gain modulation $g_{1}$ to the first
(input) layer of the network. No bias modulation was included in these
networks as we found it did not aid gain modulation. We set the number of
hidden neurons in this modulation network to $300$. We wanted to match the
proportionate increase in parameters accompanying the addition of early gain
modulation to late gain modulation, between two networks with $32$ and $128$
neurons in $L_{2}$ respectively. To accomplish this matching, in the case of
the network with $32$ neurons the hidden layer of the modulation needs to
contain approximately $75$ neurons. As seen in Figure 3, reducing the number
of hidden neurons in early modulation to $75$, to match the proportionate
increase in parameters between the two networks with $32$ and $128$ neurons in
$L_{2}$, does not substantially reduce the increase in performance provided by
task-based modulation of early neural processing.

Figure 3: The effectiveness of task-based gain modulation (quantified by the
performance boost, $\Delta_{\uparrow}$) of early neural processing
($L_{1}\rightarrow L_{2}$), as a function of the number of neurons in $L_{2}$
and the number of neurons in the hidden layer of early modulation. Reducing
the number of hidden neurons to $75$ does not reduce the performance boost
substantially. This observation suggests that the trends observed in Figure 2
are not simply a consequence of a disproportionate increase in the number of
parameters in the different networks.

#### Increasing the task difficulty to increase capacity constraints

In addition to manipulating the number of hidden neurons in the network and
the number of tasks to be performed, the difficulty of the tasks could also
contribute to capacity limits. If we increase the noise in the stimuli from
$20\%$ (which is the case in the main analysis; additive uniform random noise)
to $40\%$, the performance boost increases as seen in Figure 4. As the number
of parameters stays the same across the change in the level of noise, this
result also suggests that the trends observed in Figure 2 are not simply a
consequence of simple proportionate parameter expansion.

Figure 4: The effect of increasing task difficulty on the additive
effectiveness of early modulation. The performance boost ($\Delta_{\uparrow}$)
increases when noise in the stimuli is increased making each task more
difficult. The performance boost in the high noise case also decreases with
increasing number of neurons in $L_{2}$, and increases with increasing number
of tasks to be performed, echoing the trends in Figure 2.

### Comparison with cheung2019superposition

The task-based modulation scheme in our work was adapted from
cheung2019superposition. However, there are major differences between our work
and cheung2019superposition. Their work proposes a solution to the problem of
catastrophic forgetting in neural networks when faced with a continual stream
of tasks. In our work, the tasks are interleaved. They used random task-based
modulations, while in our work the task-based modulations are learned to
optimise detection performance. If two tasks are similar, the task-based
modulations should also be similar and this relationship can be learned by the
networks in our work. We wanted to assess if learning the modulations instead
of fixing them randomly, as in cheung2019superposition, aids performance on
the detection tasks here. To do so, we trained a network with $32$ neurons in
$L_{2}$ on $500$ tasks, with or without training the bias and gain modulations
(early and late modulation) which are initialised with random binary vectors
as in cheung2019superposition. When the modulation schemes can be learned, the
performance of the network was $94.0\%$, while when the scheme was fixed as
random binary vectors, the performance was $71.1\%$, confirming the idea that
task-based modulations can account for similarities across tasks. How such
task similarities could be included in learning task-based modulations in a
continual learning setting is an open question and beyond the scope of our
work.

### Robustness of presented effects

To assess if the differences in the performance ($\Delta_{\uparrow}$) of the
networks mentioned in Figure 2 are robust, we used McNemar’s test McNemar
(1947), a statistical test used on paired nominal data. This test compares the
quantities of examples where the decisions of the two networks being compared
differ. If one network misclassifies examples the other network classifies
correctly more often (say $b$ examples) than the other way around (say $c$
examples), the test statistic ($\chi^{2}=(b-c)^{2}/(b+c)$) is higher. The test
statistic has a chi-squared distribution with $1$ degree of freedom.

For each network (varying on the number of hidden neurons and the number of
tasks performed), we compared the outputs when only late modulation was active
and when both early and late modulation (global modulation) were active.
Across all the comparisons, the $\chi^{2}$ values were above $15$ (which
corresponds to $p=10^{-4}$). So, all the differences ($\Delta_{\uparrow}$) in
Figure 2 correspond to robust differences in the performance of the
corresponding networks.

### Additional observations about the behaviour of the trained neural networks

Below we clarify the training setup and present an additional observation
about the effectiveness of early modulation.

#### The network mainly performs permutation discrimination

In training the networks, each batch contained $50$ examples corresponding to
the cue (example cue: $5$ in permutation $10$, corresponding to task $95$) and
$50$ examples corresponding to every other task (the invalid case, where the
network outputs ’No’). As the invalid cases are drawn randomly, the
probability that the same digit as the valid case ($5$) would be included is
$1/10$. The probability that the same permutation as the valid case
(permutation $10$) would be included is $1/50$ (in the case of $50$
permutations). In this setting, the network might thus mainly perform
permutation discrimination rather than digit discrimination. To show that this
was indeed the case, we probed the behaviour of a network with $32$ neurons in
$L_{2}$ trained to perform $500$ tasks (with joint modulation). When we only
included permutation-matched digits as invalid test examples, the performance
was $60.9\%$, while when we only included digit-matched permutations as
invalid test examples, the performance was $94.3\%$. This demonstrates that
the high performance of the network (Figure 2) largely reflected permutation
discrimination.

#### Modulating late neural processing in addition to modulating early neural
processing does not boost performance

How well does modulating early neural processing alone perform? We trained a
network with $32$ neurons in $L_{2}$ to perform $250$ or $500$ tasks, with
only the modulation of early neural processing. The performance of this
network was equal to the performance of the same network trained with joint
modulation. This suggests that, in this setting, only modulating early neural
processing is better than only modulating late neural processing, and that
late modulation does not aid performance on top of early modulation.

Early modulation might be performing so well because the stimuli used might be
distinguishable at the pixel-level. For example, as the network is mostly
performing permutation discrimination, each permutation might have a
characteristic spread of pixels that the first transformation could pick on
and distinguish between the valid and invalid permutations. Switching to
naturalistic stimuli might remove such low-level distinctions between stimuli,
thereby not allowing the network to capitalise solely on early modulation.

In biological systems, the trade-off between wiring costs and task optimality
might exist even if early modulation is always equal to or better than late
modulation. This is because it might only be in the (capacity-limited) cases,
where late modulation performs poorly, that early modulation is worth its
wiring costs. Further work involving naturalistic stimuli would provide a
better understanding of the nature of this trade-off.