# Multi-task learning with cross-task consistency for improved depth
estimation in colonoscopy

Pedro Esteban Chavarrias Solano  Andrew Bulpitt  Venkataraman Subramanian
Sharib Ali<EMAIL_ADDRESS>School of Computing, Faculty of Engineering
and Physical Sciences, University of Leeds, Leeds, LS2 9JT, United Kingdom
Leeds Teaching Hospitals NHS Trust, Leeds, UK

###### Abstract

Colonoscopy screening is the gold standard procedure for assessing
abnormalities in the colon and rectum, such as ulcers and cancerous polyps.
Measuring the abnormal mucosal area and its 3D reconstruction can help
quantify the surveyed area and objectively evaluate disease burden. However,
due to the complex topology of these organs and variable physical conditions,
for example, lighting, large homogeneous texture, and image modality
estimating distance from the camera (aka depth) is highly challenging.
Moreover, most colonoscopic video acquisition is monocular, making the depth
estimation a non-trivial problem. While methods in computer vision for depth
estimation have been proposed and advanced on natural scene datasets, the
efficacy of these techniques has not been widely quantified on colonoscopy
datasets. As the colonic mucosa has several low-texture regions that are not
well pronounced, learning representations from an auxiliary task can improve
salient feature extraction, allowing estimation of accurate camera depths. In
this work, we propose to develop a novel multi-task learning (MTL) approach
with a shared encoder and two decoders, namely a surface normal decoder and a
depth estimator decoder. Our depth estimator incorporates attention mechanisms
to enhance global context awareness. We leverage the surface normal prediction
to improve geometric feature extraction. Also, we apply a cross-task
consistency loss among the two geometrically related tasks, surface normal and
camera depth. We demonstrate an improvement of 14.17% on relative error and
10.4% improvement on $\delta_{1}$ accuracy over the most accurate baseline
state-of-the-art BTS approach. All experiments are conducted on a recently
released C3VD dataset; thus, we provide a first benchmark of state-of-the-art
methods.

###### keywords:

Deep learning, Monocular depth estimation, Surface normal prediction, Multi-
task learning, Cross-task consistency, 3D Colonoscopy

††journal:

[table]capposition=top

##  1 Introduction

Colorectal cancer (CRC) is among the third most common type of cancer in the
world, imposing a healthcare burden globally. The estimated number of new CRC
cases in 2023 will likely increase to 153,020 [34]. Optical endoscopy is the
gold standard procedure for diagnosing and treating CRC [32]. Despite its
great potential, the colonoscopic procedure is subject to the clinician’s
experience as they have to deal with a complex anatomical environment, imaging
artefacts, and a limited field of view. A retrospective analysis of clinical
endoscopic video realised that 9.6% of the colon surface is never imaged
during the screening procedure [23]. Those missed regions can contribute to an
estimated 22% of precancerous undetected lesions found by [36]. The
development of an intelligent system to reduce the missed detection rate and
guide the clinician to potential regions of interest has caught the attention
of the medical computer vision research community. Several methods have been
developed to detect and segment polyp instances accurately. Unlike polyp
detection and segmentation methods, 3D computer vision techniques in
colonoscopy have not been widely explored. Some applications within this field
cover lesion extent prediction [1, 2], observational coverage of the colon [3,
5], and 3D reconstruction [46].

The development of 3D computer vision applications in colonoscopy is limited
due to the difficulty in acquiring ground truth labels. Compared to labelled
datasets required for training detection and segmentation methods, acquiring
datasets with accurate surface information (e.g., surface normal vectors and
depth maps) for 3D scene understanding is far from practical during a clinical
endoscopy procedure. Hence, the development of commercial and computed
tomography-derived silicone models has been suggested as an alternative method
for data acquisition. The C3VD dataset leverages a novel technique for
generating high-fidelity silicone phantom models of the colon with various
textures and colors [5]. The phantom model and a 2D-3D video registration
algorithm were used to generate a ground truth dataset with pixel-level
registration to a known 3D model. Unlike digital phantom datasets [30, 47],
silicone phantom models are closer to real-world clinical datasets and reflect
more accurately to an actual colonoscopic procedure.

Depth estimation is crucial for understanding geometry structure and a
fundamental task in computer vision for 3D scene reconstruction. Most depth
estimation approaches initially relied on stereo matching and triangulation
methods to calculate the disparity of two 2D images. However, these binocular-
based depth estimation methods require at least two fixed cameras. In
addition, capturing enough features to match between images becomes
challenging when the scene does not have enough texture [24]. Due to the
spatial constraint imposed by the lumen (e.g., size and complex non-uniform
shape) of the gastrointestinal tract, monocular systems have been more
attractive than stereo systems. Monocular depth estimators aim to learn a
mapping between a single RGB image and their corresponding depth values by
capturing features that represent geometric structures. Single-view deep
learning methods [17, 43, 28] for monocular depth estimation make use of
monocular visual cues, e.g., texture gradients and lighting variations. These
methods learn to incorporate scene priors without the need to compute camera
motion. While this is an advantage, they usually perform well only in similar
samples to those presented during the training stage, making these networks
harder to generalise. Their accuracy is also affected by the essential
ambiguity of the problem as an infinite number of world scenes and camera
positions could have produced a given image [9, 17, 46, 28]. Therefore, recent
approaches suggest using multi-task learning schemes as they leverage
auxiliary tasks with depth-related features [24]. Adopting an additional task
aims to enhance the extraction of relevant geometrical cues during the
encoding stage, leading to a better performance overall [24, 6, 29]. For
example, joint learning of depth and optical flow [49, 6] and joint learning
of depth and surface normal [4] have shown competitive performances on natural
scene datasets. Multi-task learning approaches have been extensively studied
in natural scenes, but their applicability in the colonoscopy domain has not
been widely validated. In addition, consistency among tasks is desirable since
both generate a particular domain representation of the same underlying
reality [49, 44].

To this end, we propose Col3D-MTL, a novel multi-task learning with a cross-
task consistency approach for joint monocular depth and surface normal
prediction featuring attention mechanisms to improve global context awareness.
We validate our study on a public colonoscopy dataset completely acquired
using a silicone phantom model. Our proposed framework can be summarised on
the following three main contributions:

  1. 1.

We propose a multi-task learning network with one shared encoder and two
independent decoders to predict depth and surface normal maps, leading to an
enhanced feature representation of the scene. We introduce novel unit normal
computation blocks (UNC blocks) in our surface normal decoder that enable the
accurate recovery of the geometrical orientation of the scene.

  2. 2.

We incorporate a weighted cross-task consistency loss between our predicted
surface normal and a warped surface normal computed from our depth prediction
using depth image gradients to enforce consistency among both geometrically
related tasks explicitly.

  3. 3.

We set a new benchmark comprising state-of-the-art monocular depth estimation
methods and validate our multi-task learning with cross-task consistency
framework on a public colonoscopy dataset with pixel-wise ground truth
labelled data acquired from a silicone phantom model, the C3VD dataset [5].

The rest of the paper is organised as follows. Section 2 presents state-of-
the-art methods on monocular depth estimation, multi-task learning, and cross-
task consistency. In Section 3, we introduce the C3VD dataset used in this
work and our proposed framework. Section 4 describes the training and ablation
study setups followed in this work. We also present the evaluation metrics and
the corresponding quantitative and qualitative results. In Section 5, we
discuss the findings of our approach. Finally, our conclusions are presented
in Section 6.

##  2 Related work

This section introduces the most relevant technical aspects needed to
understand our contribution. The structure of this section starts with a
review of related works to monocular depth estimation in computer vision and
endoscopy, followed by a discussion about multi-task learning approaches found
in the literature covering the natural scene and endoscopy domains. Finally,
we describe the cross-task consistency methods.

###  2.1 Monocular depth estimation in computer vision

Unlike most traditional stereo matching and triangulation approaches,
monocular depth estimation methods only require a single camera to generate a
depth map. Even though promising, it is still an ill-posed problem to regress
depth from a single image [24]. The success of deep learning in many computer
vision tasks was also translated into the monocular depth estimation task.
Learning-based approaches for monocular depth estimation were first introduced
in 2014 by [9]. Their proposed method consists of a convolutional coarse-scale
network to predict depth at a global level, followed by a fine-scale network
to incorporate finer details, such as object edges.

In [17], an atrous spatial pyramid pooling (ASPP) module is used to leverage
global context information, while the decoder applies local planar guidance at
different resolutions to provide geometric guidance to the full-resolution
depth map. [16] propose a convolution-based encoder-decoder scheme with
attention mechanisms embedded in their skip connections to generate refined
multi-scale features. A global context module is also introduced at the
network’s bottleneck to capture representative features on a global scale.
[27] exploit the high degree of regularity in 3D scenes by using a piecewise
planarity prior to leverage information from co-planar pixels to improve depth
estimation.

Leveraging the enhanced global context understanding of transformer-based
architectures, [10] propose an adaptive bin-width estimator based on a mini
vision transformer (mViT) network [45]. The idea behind this approach is to
divide the depth range into several adaptive-width bins and predict the final
depth map as a linear combination of the bin centres. All these methods have
achieved state-of-the-art performance on popular natural scene depth
prediction datasets (such as KITTI [11], NYU [26]).

####  2.1.1 Monocular depth estimation applied to endoscopy

In contrast to natural scenes, estimating depth from endoscopy data is highly
affected by the lack of ground truth labelled data, low texture, variable
lighting conditions and the presence of artefacts, e.g., specularities,
saturation, and blurring effects. Conditional generative adversarial networks
(cGANs), e.g., pix2pix [15], have been used to estimate depth from monocular
endoscopic images [30, 8]. However, one major drawback of cGANs is the lack of
realistic detail and texture in their representations.

[22] applied continuous conditional random fields (CRFs) and a convolutional
neural network (CNN) to estimate depth from endoscopy images. However, one
limitation of this approach is the generation of artefacts due to specular
reflections. [42] proposed a geometry-aware monocular depth estimation network
based on ManyDepth [39] and leverage a depth, smoothness, gradient, normal and
geometric consistency losses to enhance depth predictions on endoscopy images.
However, their normal loss only relies on the normal map generated from the
predicted depth map, which does not faithfully represent the characteristics
of the scene [4].

###  2.2 Multi-task learning in computer vision

The complementarity between depth and other geometrically-related features has
recently been explored by computer vision researchers [29, 6, 20]. According
to a survey on monocular depth estimation [24], many approaches suggested
incorporating joint multi-task training, in which the extracted features
between tasks were projected from one to the other for improved performance
[49, 21, 4]. For example, [6] developed an architecture composed of two
tightly coupled encoder-decoder networks to predict depth map and optical flow
as primary and auxiliary tasks, respectively. The authors also introduced
exchange blocks to effectively communicate between depth and optical flow
networks and an epipolar layer that confines feature matching along the
epipolar line. In GeoNet, [29] proposed jointly predicting the depth and
surface normal maps from a single image. The method uses two stream-CNNs
(ResNet-50 [13] and VGG-16 [35]) to predict the initial depth and surface
normal maps. It then applies the depth-to-normal and normal-to-depth modules
to refine surface normal and depth maps. However, the depth-to-normal network
solves a pre-trained least square equation from the initial depth map followed
by a residual module to enhance the final prediction, which is not learned in
an end-to-end fashion. Similarly, normal-to-depth also solves linear equations
through a kernel regression module to infer depth from surface normal.

####  2.2.1 Multi-task learning applied to medical image analysis

Multi-task learning approaches have been studied for breast cancer
segmentation and classification [37], left ventricle quantification [41], and
CT-based identification and quantification [12]. [14] proposed a spatio-
temporal multi-task learning network with one shared-encoder and two spatio-
temporal independent decoders for instrument segmentation and saliency on a
robotic instrument segmentation dataset for endoscopy. Other multi-task
learning approaches on endoscopy have focused on monocular depth and motion
estimation [33, 19, 31]. A 3D colon reconstruction approach suggested by [21]
incorporates a multi-task recurrent neural network (RNN) that estimates depth
and camera pose [38] to improve the performance of a standard simultaneous
localisation and mapping (SLAM) method. [48] leverage surface normal
estimation to enhance feature extraction and improve their depth estimation
performance. The authors used a shared encoder and two independent decoders
for depth and surface normal prediction. However, both decoders have almost
the same architecture, with the number of output channels only varying. In
this work, we propose task-specific architectures for each decoder.

###  2.3 Cross-task consistency loss

In visual perception, different domain representations of the same underlying
reality or scene are not independent, i.e., a consistent factor between them
should exist. A general fully computational method for augmenting training is
proposed in [44]. In this work, the authors introduce a loss for predicting
domain $y_{1}$ from an input image, $x$, while imposing consistency with
domain $y_{2}$. This approach compares prediction $y_{2}$ with the warped
prediction of $y_{1}$ to domain $y_{2}$. An unsupervised framework leveraging
geometric consistency for training single-view depth and optical flow networks
on an unlabeled dataset was proposed in [49]. To enforce geometric
consistency, the authors introduced a cross-task consistency loss to minimise
the discrepancy between the estimated optical flow and a synthesised flow
computed from the predicted depth map and an estimated 6D camera pose.

##  3 Materials and Method

In this section, we describe the dataset used in our work, and we also present
the details of our proposed multi-task learning with cross-task consistency
framework.

###  3.1 The C3VD dataset

In this study, we used the new publicly available Colonoscopy 3D Video Dataset
(C3VD) [5], which is the first video dataset containing 3D pixel-wise ground
truth labelled data entirely recorded with a high-definition (HD) clinical
colonoscope. The authors created a complete 3D phantom model of the colon,
which was digitally sculpted by a board-certified anaplastologist. A
3D-printed phantom model was generated and coated with silicone, silicone
pigments and silicone lubricants to mimic the specular appearance of the
mucosa, tissue features and vascular patterns. The data acquisition was
performed by mounting the tip of a colonoscope to the end-effector of a
robotic arm with previously defined moving trajectories. Pixel-level ground
truth for each frame was generated by moving a virtual camera along the
digital model following the recorded trajectory of the robotic arm while
rendering ground truth frames of the 3D model.

The dataset contains 10,015 frames with paired ground truth depth, surface
normal, optical flow, occlusions, six-degrees-of-freedom (DoF) poses, coverage
maps, and 3D models. The image resolution of all available data is $1080\times
1350$ pixels. In total, 22 videos covering four different colon segments
(caecum, transverse, descending, and sigmoid), four texture variations, and
three predefined trajectories are publicly available. Fig. 1 shows sample
images with their corresponding ground truth depth map, surface normal, and
occlusion map.

Figure 1: The C3VD dataset. Sample data including the original RGB image with
its corresponding ground truth depth map, surface normal, and occlusion map
[5].

The train, validation and test splits used in this work are detailed in Table
1. During the training stage, we followed a video-wise split in which we
provided data from three colon segments (caecum, transverse, and sigmoid). Our
models are validated on data collected from the same colon segments. For
testing, we also include data from the descending colon segment, which was not
given during the training stage.

###  3.2 Method

This subsection presents our proposed multi-task learning with a cross-task
consistency framework for improved colonoscopy depth estimation. We describe
the baseline depth estimation network selected in this study, followed by a
review of our multi-task learning scheme and the cross-task consistency
approach we incorporate into the network.

Table 1: Dataset split. Our dataset split follows a video-wise split. Each
label corresponds to the colon segment, followed by the texture style and the
predefined video trajectory. Here ‘c’ refers to caecum, ‘s’ refers to sigmoid,
‘t’ refers to transverse and ‘d’ for descending colon.

Split | Colonoscopy videos | No. of frames
---|---|---
Training | c1v1, c2v2, c2v3, c3v2, s1v3, s2v1, t1v1, | 6344
| t1v3, t2v1, t2v2, t3v2, t3v1, t4v3 |
Validation | c4v2, c4v3, s3v1, t2v3 | 1738
Testing | c2v1, d4v2, s3v2, t3v3 | 1268

####  3.2.1 Depth estimation network

Our framework is based on the monocular depth estimation network proposed by
[17]. This method follows an encoder-decoder scheme, in which the encoder
performs dense feature extraction while the decoder aims to regress the depth
values. The decoder leverages local planar guidance (LPG) layers at different
resolutions to improve the final output. The architecture of this network can
be identified at the bottom of Fig. 2.

The network uses ResNet-50 [13] as its dense feature extractor, which outputs
a feature map of $H/8$ resolution. The backbone is followed by an atrous
spatial pyramid pooling module [7] to extract contextual information at
multiple dilation rates. During the decoding phase, internal outputs are
recovered to their original resolution $H$ by a factor of 2 at each LPG block.
The LPG block provides geometric guidance to the full-resolution depth map. A
final $1\times 1$ convolutional layer is also used to extract the finest
estimation ($\tilde{c}^{1\times 1}$) after the last upconv layer. All the
estimated outputs ($\tilde{c}^{k\times k}$) are concatenated and processed
through a convolutional layer to compute the final depth estimation. The model
is optimized by minimizing the scale-invariant logarithmic (SILog) error loss
introduced by [9].

##### Multi-scale local planar guidance (LPG block)

Most monocular depth estimation networks following an encoder-decoder
architecture just apply simple nearest neighbour up-sampling to recover the
original resolution of the input image. Unlike those methods, LPG blocks guide
features to the full resolution leveraging the local planar assumption [17].
The LPG block consists of a stack of $1\times 1$ reduction layers, which
iteratively decrease the number of channels by a factor of two until it
reaches a channel dimension of three. The resulting feature map ($H/k,H/k,3$)
is processed through two pathways to compute local plane coefficient
estimations. The first pathway uses the first two channels to perform the
conversion to unit normal vectors ($n_{1},n_{2},n_{3}$). A unit normal vector
has only two degrees of freedom (DoF), polar and azimuthal angles from
predefined axes. The two channels are regarded as polar $\theta$ and azimuthal
$\phi$ angles and converted to unit normal vectors by Eq. (3.2.1).

$\displaystyle n_{1}=sin(\theta)cos(\phi)$ $\displaystyle
n_{2}=sin(\theta)sin(\phi)$ (1) $\displaystyle n_{3}=cos(\theta)$

The second pathway estimates the perpendicular distance ($n_{4}$) between the
plane and the origin. This pathway computes a sigmoid function from the third
channel and multiplies its output with the maximum depth value. Finally, these
4D plane coefficients ($n_{1},n_{2},n_{3},n_{4}$) are used to compute $k\times
k$ local depth cues using the ray-plane intersection Eq. (2).

$\tilde{c}^{k\times k}=\frac{n4}{n1\cdot u_{i}+n2\cdot v_{i}+n3}$ (2)

where ($n_{1},n_{2},n_{3},n_{4}$) describe the previously estimated plane
coefficients and $u_{i},v_{i}$ denote $k\times k$ patch-wise normalized
coordinates of pixel $i$.

####  3.2.2 Attention mechanism

Convolutional neural networks have shown outstanding performance in enhancing
local feature representations. However, focusing on relevant features while
suppressing irrelevant ones further improves the process of capturing the
visual structure of a scene. The convolutional block attention module (CBAM)
proposed in [40] sequential applies a channel attention module and a spatial
attention module to a given feature map. The process consists of two
sequential element-wise multiplications. The first one multiplies the input
feature map ($\mathcal{F}$) and the channel attention map ($M_{c}$). The
second one is performed between the output of the first multiplication
($\mathcal{F}^{\prime}$) and the output of the spatial attention module
($M_{s}$), resulting in a refined feature map $\mathcal{F}^{\prime\prime}$.
The whole process can be summarised as shown in Eq.( 3):

$\displaystyle\mathcal{F}^{\prime}=\mathcal{F}\otimes M_{c}(\mathcal{F})$ (3)
$\displaystyle\mathcal{F}^{\prime\prime}=\mathcal{F}^{\prime}\otimes
M_{s}(\mathcal{F}^{\prime})$

Figure 2: Multi-task learning with cross-task consistency (Col3D-MTL). Our
proposed framework follows the encoder-decoder scheme, in which the encoder
consists of a shared backbone, $\theta$, followed by an atrous spatial pyramid
(ASPP) module to extract contextual information at different dilation rates.
The decoder stage comprises a primary depth estimator decoder (bottom) and an
auxiliary surface normal decoder (top). Our unit normal computation block (UNC
block) uses two feature channels (FC) to compute unit normal vectors ($n_{1}$,
$n_{2}$, and $n_{3}$). The local planar guided depth estimation block (LPG
block) also uses a third FC to compute the perpendicular distance to the
camera, which is incorporated together with the unit normals to provide local
depth information $\tilde{c}^{k\times k}$ by the ray-plane intersection (see
Eq.( 2)). CBAM modules (Att) are introduced at the skip connections and after
the convolutional layers of the depth decoder to enhance global context
awareness. The depth-to-surface normal (D2SN) module receives the predicted
depth and outputs a warped surface normal map, which is compared against the
surface normal prediction to enforce consistency among tasks. The surface
normal decoder and D2SN module can be easily combined with other depth
estimators for an end-to-end MTL-X-TC.

This module has been used by [18] in the skip connections and in the decoder
of their network to recover meaningful global information at a low
computational cost. We followed a similar approach to [18] by incorporating
CBAM modules in the skip connections and at each resolution level of our depth
decoder. We aim to leverage the local feature representation of convolutional
neural networks to extract monocular depth cues and the global context
awareness of CBAM modules to relate them effectively. The incorporation of
CBAM modules into our framework is shown in Fig. 2.

####  3.2.3 Multi-task learning network with UNC block

Our proposed framework is based on the geometric relationship between the
depth and surface normal information of a 3D scene. Following this statement,
our method aims to improve its depth estimation robustness by incorporating a
geometrically related auxiliary task. The proposed architecture consists of a
single shared encoder and two independent decoders. The purpose of the shared
encoder is to extract meaningful geometric features that represent the 3D
scene, while the two decoders are used to regress depth and surface normal
maps.

Our framework extends the depth estimation network by adding a surface normal
decoder located at the top of Fig. 2. Within our surface normal decoder, we
introduce the unit normal computation block (UNC block) to compute surface
normal maps at multiple resolutions. Our UNC block is based on the LPG blocks
without the computation of the ray-plane intersection. Each surface normal map
is represented by an RGB image, where each channel represents one particular
axis: the red channel denotes the $x$-axis, the green channel represents the
$y$-axis, and the blue channel represents the $z$-axis. The outputs of all UNC
blocks undergo a channel concatenation followed by convolutional layers to
compute the final surface normal prediction.

The multi-task learning framework combines the losses of both tasks as
described in Eq.( 4).

$\mathcal{L}_{MTL}=\lambda_{1}\cdot\mathcal{L}_{depth}+\lambda_{2}\cdot\mathcal{L}_{sn}$
(4)

where $\lambda_{1},\lambda_{2}$ are weighting factors equally set to 0.5,
$\mathcal{L}_{depth}$ represents the scale-invariant logarithmic error (SILog)
loss between the predicted and ground truth depth maps, and $\mathcal{L}_{sn}$
symbolises the mean absolute error (MAE) loss between the estimated surface
normal and its corresponding ground truth.

####  3.2.4 Cross-task consistency loss with D2SN module

We incorporate a cross-task consistency (X-TC) loss into our multi-task
learning framework to enforce consistency among depth and surface normal
predictions. To this end, we add a depth-to-surface normal (D2SN) warping
module based on the mathematical method proposed by [25]. Our warping module
uses the output of our depth estimation decoder. The predicted depth map is
processed within this module to generate a warped surface normal
representation. The output of this module introduces a consistency constraint
by being compared against the prediction of the surface normal decoder.

##### Depth-to-surface normal module (D2SN module)

Since normal estimation is equivalent to fitting a plane to a local point
cloud in the 3D space, several approaches leveraging optimisation techniques
have been proposed. However, these methods are expensive in terms of
computational resources. Therefore, our D2SN module, which is shown in the
right section of Fig. 2, computes a surface normal from depth image gradients
(DIG) as shown in [25]. The authors consider that adjacent 3D points in a
depth image can be used to compute a local 3D plane whose orthogonal vector is
equivalent to the normal vector at that particular pixel. Their proposed
method consists of three steps:

  1. 1.

Depth image gradients: Given the location of a pixel ($x,y$) and its depth
value ($Z$), pixels can be projected to the 3D space P$(X,Y,Z)$ by a
transformation matrix with known camera intrinsic parameters. The generated
point cloud is used to compute the partial directional derivatives as shown in
Eq. (5):

$\displaystyle\frac{\partial{Z(x,y)}}{\partial{x}}=Z(x+1,y)-Z(x,y)$
$\displaystyle\frac{\partial{Z(x,y)}}{\partial{y}}=Z(x,y+1)-Z(x,y)$
$\displaystyle\frac{\partial{X(x,y)}}{\partial{x}}=\frac{Z(x,y)}{f}+\frac{(x-c_{x})}{f}\frac{\partial{Z(x,y)}}{\partial{x}}$
(5)
$\displaystyle\frac{\partial{X(x,y)}}{\partial{y}}=\frac{(x-c_{x})}{f}\frac{\partial{Z(x,y)}}{\partial{y}}$
$\displaystyle\frac{\partial{Y(x,y)}}{\partial{x}}=\frac{(x-c_{x})}{f}\frac{\partial{Z(x,y)}}{\partial{y}}$
$\displaystyle\frac{\partial{Y(x,y)}}{\partial{y}}=\frac{Z(x,y)}{f}+\frac{(y-c_{y})}{f}\frac{\partial{Z(x,y)}}{\partial{y}}$

  2. 2.

Tangent vectors: The $x$ and $y$ directional derivatives at a given 3D point,
P, can be used as tangent vectors of the surface as shown in Eq.( 6):

$\displaystyle\mathbf{v}_{x}(x,y)=\frac{\partial{X}(x,y)}{\partial{x}},\frac{\partial{Y}(x,y)}{\partial{x}},\frac{\partial{Z(x,y)}}{\partial{x}}$
(6)
$\displaystyle\mathbf{v}_{y}(x,y)=\frac{\partial{X}(x,y)}{\partial{y}},\frac{\partial{Y}(x,y)}{\partial{y}},\frac{\partial{Z(x,y)}}{\partial{y}}$

  3. 3.

Normal vector: The cross-product of the two tangent vectors described in the
previous step is calculated to get the normal vector as in Eq.( 7):

$\mathbf{n}(x,y)=\mathbf{v}_{x}(x,y)\times\mathbf{v}_{y}(x,y)$ (7)

Our multi-task learning with cross-task consistency framework combines the
losses from both tasks and the cross-task consistency loss into the final
weighted loss function described in Eq.( 8).

$\mathcal{L}_{final}=\lambda_{1}\cdot\mathcal{L}_{depth}+\lambda_{2}\cdot\mathcal{L}_{sn}+\lambda_{3}\cdot\mathcal{L}_{X-TC}$
(8)

where $\lambda_{1},\lambda_{2}$, and $\lambda_{3}$ are weighting factors, and
$\mathcal{L}_{X-TC}$ represents the root mean squared error (RMSE) loss
between the predicted and warped surface normal maps. Eq.( 9) defines each of
the loss functions used in our final loss.

$\displaystyle
L_{depth}=\alpha\sqrt{D(g)};\>D(g)=\frac{1}{N}\sum_{i}g_{i}^{2}-\frac{\lambda}{N}^{2}\left(\sum_{i}g_{i}\right)^{2}$
(9) $\displaystyle
g_{i}=log\left(y_{i\\_depth}\right)-log\left(\hat{y}_{i\\_depth}\right)$
$\displaystyle L_{sn}=\frac{1}{N}\sum_{i}|y_{i\\_sn}-\hat{y}_{i\\_sn}|$
$\displaystyle
L_{x-tc}=\sqrt{\frac{1}{N}\sum_{i}(y_{i\\_sn}^{*}-\hat{y}_{i\\_sn})^{2}}$

where $\lambda=0.85$, $y_{i}$ represents the ground truth map, $\hat{y_{i}}$
denotes the prediction map, and $y^{*}_{i\\_sn}$ the warped surface normal
map.

##  4 Experiments and Results

###  4.1 Training setup

Each model presented in this study is trained using a single NVIDIA V100 GPU.
All models are trained for up to 50 epochs (pix2pix and MonoDepth+FPN are
trained for 200 epochs) using a batch size of 8, with an initial learning rate
of $1e^{-4}$, and a weight decay of $1e^{-2}$. The input images are resized
from their original resolution to $320\times 320$ pixels. Only random rotation
is performed as a data augmentation technique to avoid the loss of structural
information and visual cues. Random cropping, which is suggested by baseline
methods, is discarded because we observed that it could lead to heavy close-
ups towards the walls of the colon, drastically reducing contextual
information.

###  4.2 Ablation study setup

We perform an ablation study to analyse the proposed network components that
lead to the design of our multi-task learning approach with cross-task
consistency featuring attention mechanisms. We select the BTS architecture
proposed by [17] as our baseline method. Before incorporating our multi-task
learning approach, we add CBAM attention modules at different stages of the
depth decoder and its skip connections. We further extend this method
following a multi-task learning scheme. Experimentally, we define the best
loss function to optimise our auxiliary task. Additionally, we implement a
cross-task consistency module into our multi-task learning approach. Finally,
we conduct a hyperparameter study on our multi-task learning with a cross-task
consistency framework to determine the set of $\lambda$ values in our loss
function that leads to the best trade-off performance among both tasks. In
Table 2, we describe the different network configurations trained and
evaluated in our ablation study.

Table 2: Model configurations. Our ablation study setup is constituted by four
different model configurations.

Model ID | CBAM | MTL | X-TC
---|---|---|---
BTS (baseline) [17] |  |  |
BTS-CBAM | ✓ |  |
BTS-CBAM-MTL | ✓ | ✓ |
BTS-CBAM-MTL-X-TC (Col3D-MTL) | ✓ | ✓ | ✓

###  4.3 Metrics and assessment

To evaluate our methods, we follow standard depth estimation metrics described
by [9]. These include five error metrics: absolute relative error (Abs Rel),
squared relative error (Sq Rel), logarithmic error (log10), root mean squared
error (RMSE), root mean squared logarithmic error (RMSElog), scale-invariant
logarithmic error (SILog); and three accuracy metrics: $\delta_{1}$,
$\delta_{2}$, and $\delta_{3}$. We used standard surface normal evaluation
metrics, which include two error metrics: mean angular error (Mean Ang) and
median angular error (Median Ang), and three accuracy metrics:
$\delta_{11.25^{\circ}}$, $\delta_{22.5^{\circ}}$, and $\delta_{30^{\circ}}$

###  4.4 Results

In this section, we provide the quantitative and qualitative results from our
network configuration ablation study, followed by our hyperparameter study on
the $\lambda$ weighting factors of our final loss function. Finally, we
compare our approach against other state-of-the-art methods.

####  4.4.1 Quantitative results

In Table 3, we provide the results of our ablation study on the proposed
network configurations. We compare the performances of BTS, BTS-CBAM, BTS-
CBAM-MTL, and BTS-CBAM-MTL-X-TC for all depth and surface normal evaluation
metrics on the validation set of the C3VD dataset. Our BTS-CBAM network yields
a relative improvement over the BTS method by 8.9% and 2.4% in terms of SILog
and $\delta_{1}$, respectively.

Table 3: Quantitative results for various network configurations on validation
set (ablation study). Each network uses the learned weights that lead to a
better performance during the training stage. First and second best performing
methods for each evaluation metric are formatted.

Method | Losses | Abs Rel $\downarrow$ | Sq Rel $\downarrow$ | log 10 $\downarrow$ | RMSE $\downarrow$ | RMSElog $\downarrow$ | SILog $\downarrow$ |  $\delta_{1}$ $\uparrow$ |  $\delta_{2}$ $\uparrow$ |  $\delta_{3}$ $\uparrow$ | Mean Ang $\downarrow$ | Median Ang $\downarrow$ |  $\delta_{11.25^{\circ}}$ $\uparrow$ |  $\delta_{22.5^{\circ}}$ $\uparrow$ |  $\delta_{30^{\circ}}$ $\uparrow$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
BTS (baseline) | Depth: SILog | 0.179 | 1.088 | 0.073 | 5.667 | 0.208 | 14.066 | 0.756 | 0.959 | 0.992 | - | - | - | - | -
BTS-CBAM | Depth: SILog | 0.170 | 0.979 | 0.070 | 5.503 | 0.198 | 12.814 | 0.774 | 0.969 | 0.994 | - | - | - | - | -
BTS-CBAM- MTL |  | Depth: SILog
---
SN: $L_{1}$
0.166 | 0.934 | 0.070 | 5.422 | 0.195 | 12.872 | 0.785 | 0.969 | 0.996 | 43.892 | 35.363 | 17.637 | 40.340 | 51.814
| Depth: SILog
---
SN: $L_{2}$
0.163 | 0.900 | 0.067 | 5.298 | 0.190 | 13.219 | 0.797 | 0.972 | 0.995 | 44.337 | 35.142 | 13.940 | 37.288 | 49.594
BTS-CBAM- MTL-X-TC (Col3D-MTL) |  | Depth: SILog
---
SN: $L_{1}$
X-TC: $L_{2}$
0.156 | 0.805 | 0.065 | 4.994 | 0.186 | 12.530 | 0.809 | 0.975 | 0.997 | 23.999 | 21.382 | 24.232 | 55.883 | 71.758
| Depth: SILog
---
SN: $L_{2}$
X-TC: $L_{2}$
0.176 | 1.057 | 0.073 | 5.723 | 0.211 | 14.822 | 0.731 | 0.968 | 0.996 | 40.664 | 32.630 | 15.974 | 40.219 | 52.394

Table 4: Hyperparameter study on validation set. Proposed BTS-CBAM-MTL-X-TC
networks are evaluated in the validation set to avoid $\lambda$ weighting
factors to be adjusted based on the testing data. Each network uses the
learned weights that lead to a better performance during the training stage.
First and second best performing methods for each evaluation metric are
formatted.

Method | $\lambda_{1}$ | $\lambda_{2}$ | $\lambda_{3}$ | Abs Rel $\downarrow$ | Sq Rel $\downarrow$ | log 10 $\downarrow$ | RMSE $\downarrow$ | RMSElog $\downarrow$ | SILog $\downarrow$ |  $\delta_{1}$ $\uparrow$ |  $\delta_{2}$ $\uparrow$ |  $\delta_{3}$ $\uparrow$ | Mean Ang $\downarrow$ | Median Ang $\downarrow$ |  $\delta_{11.25^{\circ}}$ $\uparrow$ |  $\delta_{22.5^{\circ}}$ $\uparrow$ |  $\delta_{30^{\circ}}$ $\uparrow$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
BTS-CBAM- MTL-X-TC (Col3D-MTL) | 0.4 | 0.3 | 0.3 | 0.178 | 1.061 | 0.072 | 5.639 | 0.204 | 13.632 | 0.769 | 0.963 | 0.994 | 22.957 | 20.405 | 25.028 | 57.662 | 73.668
0.4 | 0.2 | 0.2 | 0.187 | 1.116 | 0.077 | 5.664 | 0.213 | 13.672 | 0.726 | 0.965 | 0.990 | 30.100 | 24.999 | 21.081 | 49.877 | 63.892
0.5 | 0.3 | 0.2 | 0.156 | 0.805 | 0.065 | 4.994 | 0.186 | 12.530 | 0.809 | 0.975 | 0.997 | 23.999 | 21.382 | 24.232 | 55.883 | 71.758
0.5 | 0.4 | 0.1 | 0.161 | 0.837 | 0.066 | 4.901 | 0.189 | 13.036 | 0.795 | 0.974 | 0.997 | 34.013 | 24.606 | 21.812 | 50.571 | 63.335
0.6 | 0.2 | 0.2 | 0.222 | 1.470 | 0.089 | 6.269 | 0.248 | 16.163 | 0.620 | 0.939 | 0.987 | 29.338 | 25.449 | 17.860 | 45.908 | 62.025
0.6 | 0.3 | 0.1 | 0.187 | 1.083 | 0.075 | 5.598 | 0.209 | 13.773 | 0.742 | 0.957 | 0.994 | 45.448 | 35.227 | 15.080 | 36.828 | 48.007
0.7 | 0.2 | 0.1 | 0.178 | 1.028 | 0.070 | 5.128 | 0.202 | 14.329 | 0.764 | 0.961 | 0.991 | 33.879 | 28.175 | 17.835 | 43.982 | 58.190
0.8 | 0.1 | 0.1 | 0.180 | 1.078 | 0.074 | 5.753 | 0.210 | 13.573 | 0.739 | 0.963 | 0.995 | 23.880 | 21.434 | 23.147 | 55.364 | 71.800

Following the incorporation of attention mechanisms, the middle section of
Table 3 illustrates the effect of the $L_{1}$ and $L_{2}$ loss functions to
optimise the surface normal decoder of our BTS-CBAM-MTL approach. The use of
the $L_{1}$ loss function leads to a relative improvement of 2.6% on SILog
metric but a relative decrease of 1.5% on $\delta_{1}$ regarding the use of
the $L_{2}$ loss. Furthermore, our surface normal decoder optimised through
the $L_{1}$ loss demonstrates a relative improvement of 1% and 26.5% on mean
angular error and $\delta_{11.25^{\circ}}$, respectively. The bottom section
of Table 3 assesses the performance of our BTS-CBAM-MTL-X-TC employing the
$L_{1}$ and $L_{2}$ loss functions to optimise our surface normal decoder.
Initially, the $\lambda$ weighting factors in our final loss function (Eq.(
8)) are set to $\lambda_{1}=0.5$, $\lambda_{2}=0.3$, and $\lambda_{3}=0.2$.
Our proposed network configuration optimising our surface normal decoder with
the $L_{1}$ loss outperforms all previous network configurations in all depth
and surface normal evaluation metrics.

Table 4 includes the results of our hyperparameter study on different sets of
$\lambda$ weighting factors in the loss function (Eq.( 8)) of our BTS-CBAM-
MTL-X-TC network. Based on our experimental results, the best $\lambda$
configuration set consists of $\lambda_{1}=0.5$, $\lambda_{2}=0.3$, and
$\lambda_{3}=0.2$. This configuration leads to the best performance among all
evaluated models, achieving a relative improvement of 10.9%, 11.9%, and 7% on
SILog, RMSE, and $\delta_{1}$ metrics over our baseline method. Furthermore,
we outperform the surface normal prediction of our BTS-CBAM-MTL configuration
by a relative improvement of 45.3%, 39.5%, and 37.4% on mean angular error,
median angular error, and $\delta_{11.25^{\circ}}$.

In Table 5, we compare the performance of our baseline and our Col3D-MTL
methods against other state-of-the-art methods on supervised monocular depth
estimation: NeWCRFS [6], pix2pix [15, 30], and MonoDepth+FPN [2].
MonoDepth+FPN [2] is used to estimate depth maps on oesophageal endoscopy,
while pix2pix [30] is applied on a synthetic colonoscopy dataset. All methods
are trained on the same data distribution and tested on the same held-out
testing data, setting a new benchmark on the C3VD dataset. Our proposed
framework achieves the best performance on all depth evaluation metrics. It
improves our baseline method by 3.2%, 12.6%, and 10.4% on SILog, RMSE and
$\delta_{1}$, respectively. Moreover, it outperforms the state-of-the-art
method NeWCRFs by 6.6%, 0.2%, and 4.9% on the same evaluation metrics.

Table 5: Benchmark results on test set. Evaluation of three state-of-the-art
methods, our baseline method and our proposed framework on the C3VD dataset.
All methods are trained and evaluated on the same data distributions. First
and second best performing methods for each evaluation metric are formatted.

Method | Abs Rel $\downarrow$ | Sq Rel $\downarrow$ | log 10 $\downarrow$ | RMSE $\downarrow$ |  RMSElog $\downarrow$ | SILog $\downarrow$ |  $\delta_{1}$ $\uparrow$ |  $\delta_{2}$ $\uparrow$ |  $\delta_{3}$ $\uparrow$
---|---|---|---|---|---|---|---|---|---
| pix2pix
---
[15, 30]
0.157$\pm$0.103 | 0.730$\pm$0.647 | 0.062$\pm$0.034 | 3.721$\pm$1.385 | 0.237$\pm$0.144 | 21.062$\pm$14.118 | 0.801$\pm$0.181 | 0.956$\pm$0.078 | 0.986$\pm$0.029
| MonoDepth+FPN
---
[2]
0.204$\pm$0.052 | 1.494$\pm$0.832 | 0.090$\pm$0.025 | 6.762$\pm$1.812 | 0.342$\pm$0.078 | 31.737$\pm$6.25 | 0.628$\pm$0.176 | 0.949$\pm$0.046 | 0.991$\pm$0.010
| NeWCRFs
---
[6]
0.133$\pm$0.097 | 0.557$\pm$0.595 | 0.048$\pm$0.033 | 3.058$\pm$1.419 | 0.151$\pm$0.082 | 11.817$\pm$4.714 | 0.854$\pm$0.197 | 0.973$\pm$0.066 | 0.997$\pm$0.006
| BTS
---
[17]
0.127$\pm$0.089 | 0.622$\pm$0.543 | 0.055$\pm$0.034 | 3.823$\pm$2.024 | 0.150$\pm$0.089 | 11.400$\pm$5.614 | 0.812$\pm$0.198 | 0.979$\pm$0.051 | 0.998$\pm$0.003
| BTS-CBAM-MTL-X-TC
---
(Col3D-MTL, ours)
0.109$\pm$0.064 | 0.386$\pm$0.316 | 0.046$\pm$0.024 | 3.052$\pm$1.143 | 0.131$\pm$0.065 | 11.035$\pm$4.856 | 0.896$\pm$0.140 | 0.989$\pm$0.021 | 0.998$\pm$0.003

Table 6 analyses the performance of all the evaluated methods on each colon
segment of the testing set separately. On all colon segments, except for the
caecum, Col3D-MTL scores higher than all the other methods in terms of
$\delta_{1}$. On the caecum and transverse segments, which respectively
represent 43% and 36% of our training data, our baseline and our proposed
framework achieve the first and second best-performing methods in terms of
SILog, RMSE, and $\delta_{1}$. Only NeWCRFs accomplishes a lower RMSE value on
the caecum segment, achieving a relative improvement of 4.9% over our
baseline. Considering the sigmoid segment, which only constitutes 21% of the
training set, our approach outperforms our baseline method by a relative
improvement of 5%, 31.1%, and 22.7% on SILog, RMSE, and $\delta_{1}$. However,
our network is only surpassed by NeWCRFs on SILog and RMSE metrics. The most
remarkable improvement occurs in the descending segment of the colon, which
was not given during the training stage. On this colon segment, Col3D-MTL
obtains the best-performing metrics among all the evaluated methods while
achieving a relative improvement of 9.5%, 16.4%, and 32.3% on SILog, RMSE, and
$\delta_{1}$ over our baseline.

Table 6: Quantitative results for per colon segment. Evaluation of all
evaluated methods on each colon segment in the testing set is provided. First
and second best performing method for each evaluation metric on each colon
segment is formatted. In ’(.)’ we include the percentage of training samples
used from each segment.

Colon segment | Method | SILog $\downarrow$ | RMSE $\downarrow$ | $\delta_{1}\uparrow$
---|---|---|---|---
Caecum (43%) | pix2pix | 6.805$\pm$1.345 | 2.649$\pm$1.342 | 0.956$\pm$0.083
MonoDepth+FPN | 28.834$\pm$4.958 | 7.143$\pm$1.405 | 0.822$\pm$0.052
NeWCRFs | 7.494$\pm$0.696 | 1.567$\pm$0.552 | 0.996$\pm$0.004
BTS | 5.171$\pm$1.065 | 1.649$\pm$0.447 | 0.997$\pm$0.003
Ours | 5.738$\pm$1.232 | 2.069$\pm$0.547 | 0.996$\pm$0.004
Transverse (36%) | pix2pix | 13.464$\pm$1.304 | 2.374$\pm$0.100 | 0.929$\pm$0.017
MonoDepth+FPN | 25.409$\pm$0.695 | 4.894$\pm$0.279 | 0.759$\pm$0.026
NeWCRFs | 8.858$\pm$0.917 | 2.598$\pm$0.416 | 0.992$\pm$0.004
BTS | 6.243$\pm$0.265 | 1.840$\pm$0.145 | 0.997$\pm$0.002
Ours | 6.412$\pm$0.490 | 1.648$\pm$0.056 | 0.997$\pm$0.001
Sigmoid (21%) | pix2pix | 32.429$\pm$12.481 | 4.618$\pm$0.593 | 0.745$\pm$0.068
MonoDepth+FPN | 36.365$\pm$4.780 | 6.422$\pm$0.546 | 0.459$\pm$0.072
NeWCRFs | 13.135$\pm$1.600 | 3.664$\pm$1.052 | 0.809$\pm$0.158
BTS | 16.276$\pm$1.526 | 5.838$\pm$0.498 | 0.715$\pm$0.048
Ours | 15.460$\pm$1.240 | 4.020$\pm$0.353 | 0.877$\pm$0.023
Descending (0%) | pix2pix | 26.52$\pm$6.868 | 5.095$\pm$0.599 | 0.432$\pm$0.121
MonoDepth+FPN | 31.629$\pm$6.907 | 11.405$\pm$1.479 | 0.523$\pm$0.109
NeWCRFs | 22.137$\pm$2.915 | 5.468$\pm$0.437 | 0.464$\pm$0.098
BTS | 15.647$\pm$2.705 | 5.121$\pm$0.627 | 0.396$\pm$0.138
Ours | 14.156$\pm$2.325 | 4.279$\pm$0.584 | 0.524$\pm$0.144

####  4.4.2 Qualitative results

Fig. 3 contains sample input images with their corresponding ground truth
annotations and the predictions of our baseline and our proposed method. We
can observe that our baseline properly recovers a global depth map of the 3D
scene; however, it has a tendency to generate smooth transitions between
anatomical structures. Our proposed method addresses these cases by recovering
geometrical information about the scene and enforcing cross-task consistency
between our depth and surface normal predictions, leading to sharper
boundaries and reduced visual artefacts, e.g., specular reflection. We include
an absolute error map for each channel pair among prediction and ground truth
maps from both tasks. From the absolute error maps, we can observe that our
baseline generates brighter regions than Col3D-MTL, i.e., our approach
recovers the 3D information of the scene with less absolute error than its
baseline method.

By analysing the most challenging samples, we can observe that the cases in
which our methods fail to recover an accurate depth estimation are small
regions with low lighting conditions. Even though our surface normal decoder
recovers the overall geometry of small structures, such as polyps, it does not
compute a detailed representation of the surface orientation within these
small regions. Looking at the areas denoted with red arrows, we can observe
that regions with higher errors in our depth maps are consistent with regions
with higher errors in our surface normal maps. Despite these focalised errors,
the overall depth estimation of our proposed method performs better than the
one from its baseline.

Figure 3: Qualitative comparison between our baseline and our proposed
framework on best, average, and worst performing cases. We show absolute error
maps for both methods to observe the most challenging regions and the impact
of our BTS-CBAM-MTL-X-TC. In the first two rows, we can observe the best-
performing cases (green) in which both methods lead to low absolute error
maps. The third and fourth rows show average-performing cases (blue) in which
the lack of texture and regions with high-depth variability affecting our
baseline method are addressed by our BTS-CBAM-MTL-X-TC framework. Consistency
can be observed on the predicted surface normal maps, which help to recover
the shape of the scene, e.g., folds of the colon and small protuberances like
polyps. The last two rows represent challenging cases (red) in which both
methods generate less accurate depth estimations. Low lighting regions,
usually located at the furthest section of the scene, represent challenging
cases for both methods. Figure 4: Qualitative comparison between our different
network configurations. We show the input image, its corresponding ground
truth depth maps, the depth predictions for each network configuration, and
its corresponding absolute error maps. White arrows show the positive impact
of each network configuration concerning the previous one. The addition of the
CBAM modules partially reduces the smooth transitions of our baseline method.
Leveraging our MTL approach reduces the absolute error at transition zones but
does not recover an accurate estimation at areas with low texture. Explicitly
enforcing consistency among tasks achieves an enhanced depth estimation.

In Fig. 4, we compare the absolute error maps from all the proposed network
configurations of our ablation study to observe the effect of each approach.
Lower absolute error maps are achieved after each incorporated module,
emphasising their positive impact towards an enhanced depth estimation. Our
baseline method inaccurately estimates the depth values at transition zones,
e.g., on the folds of the colon and occluded regions. The addition of CBAM
modules reduces the absolute depth error at areas corresponding to the folds
of the colon but not at occlusion zones where depth variation can be higher.
Incorporating our surface normal predictor diminishes the absolute error of
our depth estimation at these zones. However, the highest absolute errors
within each depth map remain in these areas. Our BTS-CBAM-MTL-X-TC approach
addresses these cases by explicitly enforcing consistency among both
predictions. Our proposed framework leads to an overall lower absolute error
map but also achieves a sharper depth prediction with fewer visual artefacts.

Fig. 5 shows a qualitative comparison between all the state-of-the-art methods
evaluated in this study: BTS, pix2pix, NeWCRFs, MonoDepth+FPN, and our
Col3D-MTL framework. From the testing set, one sample from each colon segment
with its corresponding ground truth and depth prediction is given. The highest
absolute error maps are generated by MonoDepth+FPN, followed by pix2pix, which
recovered an overall depth representation of the scene but not as sharp as our
baseline method. Our baseline method improves pix2pix but generates smooth
changes in transition zones. NeWCRFs surpasses our baseline method in regions
with low texture but cannot accurately estimate depth in regions with high
depth variability. Our proposed framework leads to the lowest absolute error
maps among all the previous methods, addressing low texture and regions with
increased depth variability.

Figure 5: Qualitative comparison between all evaluated methods in this study.
One sample from each colon segment and its corresponding ground truth depth
map is given from the testing set. The depth prediction of each method is
provided with its corresponding absolute error map. Red arrows specify
challenging regions for each method, e.g., folds of the colon, polyps,
occlusion zones, low-texture regions, and areas with low lighting conditions.

##  5 Discussion

Monocular depth estimation methods rely on the representation of visual cues
to recover the depth information from a single image. Convolutional neural
networks have shown an outstanding performance in extracting local feature
representations but need more contextual information to relate them properly.
Therefore, we explored the use of CBAM modules at each multi-scale stage of
the decoder and skip connections of our baseline method to incorporate
attention mechanisms and leverage global context awareness. Our experiments
show an improved performance on all evaluation metrics (see top of Table 3)
and a refined depth map with respect to our baseline (second and third columns
of Fig. 4). However, our BTS-CBAM configuration shows a tendency to create
blurry regions. To further enhance the extraction of salient features and
leverage the orientation of the scene, we integrate a geometrically related
task, namely surface normal estimation, into our previous network with
attention mechanisms. We have designed an independent surface normal decoder
with novel unit normal computation blocks and incorporated it into our
baseline method with CBAM modules. Our BTS-CBAM-MTL approach using an $L_{1}$
loss function to optimise the surface normal decoder yields the best trade-off
performance considering all evaluation metrics for both tasks (see middle of
Table 3). The extraction of salient geometrical features leads to a better
scene representation, partially reducing the generation of blurry regions of
our BTS-CBAM network (third and fourth columns of Fig. 4). However, without an
explicit constraint to enforce cross-task consistency, we do not achieve a
refined depth prediction at regions with high-depth variability, such as the
folds of the gastrointestinal tract and occlusion zones. To explicitly enforce
consistency among both predictions, we implement a cross-task consistency
scheme. In order to evaluate consistency among depth and surface normal
predictions, we implement a warping module based on DIG to generate a warping
surface normal map from our depth estimation. Our cross-task consistency
scheme aims to minimise the RMSE between the warped surface normal and the
surface normal prediction of our decoder. We define our final loss function as
a weighted sum of each loss function that optimises our depth estimation,
surface normal prediction, and cross-task consistency module. The results of
our ablation study show that the best set of $\lambda$ weighting factors
consists of $\lambda_{1}=0.5$, $\lambda_{2}=0.3$, and $\lambda_{3}=0.2$ (Table
4). Our proposed Col3D-MTL framework surpasses all previous approaches in our
network configuration ablation study (see bottom of Table 3). We can observe
that our approach leads to the most refined depth maps with the lowest
absolute error maps (fifth column of Fig. 4). We can notice an improved
accuracy on transition zones, e.g., on the folds of the colon or in regions
containing polyps. Furthermore, the enhanced, detailed regions in our depth
estimation show consistency with the accurate surface normal predictions (see
Fig. 3).

We set a new benchmark on the C3VD dataset, in which our proposed method
achieves the best performance among the evaluated state-of-the-art methods on
monocular depth estimation (Table 5). Our approach yields a relative
improvement of 3.2% on SILog, 12.6% on RMSE, and 10.4% on $\delta_{1}$ over
our baseline. We further validate our approach by evaluating each method on
each colon segment separately. On all colon segments, except for the caecum,
our proposal achieves the best $\delta_{1}$ value. However, the most
remarkable achievement only considers the unseen descending colon segment, in
which we surpass all the networks presented in this study (Table 6). By
leveraging local feature representation and global context awareness with
cross-task consistency among our two geometrically related tasks, we refine
our depth predictions, which demonstrate lower absolute error maps than the
other approaches (see Fig. 5).

##  6 Conclusion

Colonoscopy screening remains the gold standard for diagnosing and treating
inflammatory bowel diseases. However, due to its challenging anatomical
environment and variable conditions, it is a highly operator-dependent
procedure, which usually leads to a high missed-detection rate. Although
several approaches have been proposed to detect and segment instruments and
polyps, recovering the 3D scene information to perform a quantitative
assessment has not been widely studied. Recovering the depth information of a
scene is the first step in a 3D reconstruction pipeline. We have identified
the current challenges of monocular depth estimation methods and developed our
proposed framework towards its applicability in the colonoscopy domain. We
selected BTS as our baseline monocular depth estimation method. Given the
outstanding local feature representation of convolutional neural networks, our
proposed method leverages CBAM attention mechanisms to improve global context
awareness and to relate our extracted local features, a surface normal decoder
with novel unit normal computation blocks to enhance the 3D representation of
the scene, and a cross-task consistency scheme to explicitly enforce
consistency among depth and surface normal predictions. To demonstrate the
impact of each module, we have provided a comprehensive experimental setup
which validates our Col3D-MTL network. Our framework is compared against other
state-of-the-art monocular depth estimation methods on the C3VD dataset, which
is entirely recorded with a high-definition clinical colonoscope on a silicone
phantom model that mimics the vascular patterns and the specular appearance of
the colon mucosa. Our quantitative results show that our proposed network
outperforms current state-of-the-art methods. The most remarkable improvement
of our method is achieved on a colon segment that was not given during the
training stage. Our qualitative results support adding each proposed module
towards a refined feature representation of the colon scene.

### Limitations and future work

The limitations of the proposed framework include inaccurate depth and surface
normal predictions in regions with low lighting conditions, usually located at
the farthest region of the colonoscopy scene. Other cases in which our surface
normal decoder does not recover the surface orientation include small regions
with high orientation variability, usually encountered as the region gets
farther from the colonoscope. Given the need for properly annotated real-world
colonoscopy data for 3D reconstruction, our future work will focus on
improving the robustness of our method to translate our method from synthetic
datasets to real-world clinical data.

## Declaration of Competing Interest

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## CRediT authorship contribution statement

Pedro Chavarrias: Conceptualisation, Methodology, Investigation, Data
curation, Software, Validation, Formal analysis, Writing - original draft,
Writing - review & editing. Andrew Bulpitt: Writing - review & editing and
Supervision. Venkat Subramanian: Writing - review & editing and Supervision.
Sharib Ali: Conceptualisation, Methodology, Investigation, Software, Formal
analysis, Writing - original draft, Writing - review & editing, Primary
Supervision and Project administration.

## Acknowledgements

This work was undertaken on ARC4, part of the High Performance Computing
facilities at the University of Leeds, UK.

## References

  * Abdelrahim et al. [2022] Abdelrahim, M., Saiga, H., Maeda, N., Hossain, E., Ikeda, H., Bhandari, P., 2022\.  Automated sizing of colorectal polyps using computer vision.  Gut 71, 7–9.
  * Ali et al. [2021] Ali, S., Bailey, A., Ash, S., Haghighat, M., Investigators, T., Leedham, S.J., Lu, X., East, J.E., Rittscher, J., Braden, B., 2021.  A pilot study on automatic three-dimensional quantification of barrett’s esophagus for risk stratification and therapy monitoring.  Gastroenterology 161, 865–878.e8.  doi:10.1053/j.gastro.2021.05.059.
  * Armin et al. [2016] Armin, M.A., Chetty, G., De Visser, H., Dumas, C., Grimpen, F., Salvado, O., 2016\.  Automated visibility map of the internal colon surface from colonoscopy video.  International Journal of Computer Assisted Radiology and Surgery 11, 1599–1610.  doi:10.1007/s11548-016-1462-8.
  * Bae et al. [2022] Bae, G., Budvytis, I., Cipolla, R., 2022.  Multi-view depth estimation by fusing single-view depth probability with multi-view geometry, in: Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2832–2841.
  * Bobrow et al. [2023] Bobrow, T.L., Golhar, M., Vijayan, R., Akshintala, V.S., Garcia, J.R., Durr, N.J., 2023\.  Colonoscopy 3d video dataset with paired depth from 2d-3d registration.  Medical Image Analysis 90, 102956\.  doi:https://doi.org/10.1016/j.media.2023.102956.
  * Chen et al. [2020] Chen, J., Yang, X., Jia, Q., Liao, C., 2020.  Denao: Monocular depth estimation network with auxiliary optical flow.  IEEE Transactions on Pattern Analysis and Machine Intelligence , 1–1doi:10.1109/TPAMI.2020.2977021.
  * Chen et al. [2018] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L., 2018.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE Transactions on Pattern Analysis and Machine Intelligence 40, 834–848.  doi:10.1109/TPAMI.2017.2699184.
  * Cheng et al. [2021] Cheng, K., Ma, Y., Sun, B., Li, Y., Chen, X., 2021\.  Depth estimation for colonoscopy images with self-supervised learning from videos, in: Medical Image Computing and Computer Assisted Intervention – MICCAI 2021, Springer International Publishing, Cham. pp. 119–128.
  * Eigen et al. [2014] Eigen, D., Puhrsch, C., Fergus, R., 2014.  Depth map prediction from a single image using a multi-scale deep network, in: Proceedings of the 27th International Conference on Neural Information Processing Systems, MIT Press. p. 2366–2374.
  * Farooq Bhat et al. [2021] Farooq Bhat, S., Alhashim, I., Wonka, P., 2021.  Adabins: Depth estimation using adaptive bins, in: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE. p. 4008–4017.  doi:10.1109/CVPR46437.2021.00400.
  * Geiger et al. [2012] Geiger, A., Lenz, P., Urtasun, R., 2012.  Are we ready for autonomous driving? the kitti vision benchmark suite, in: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3354–3361.
  * Goncharov et al. [2021] Goncharov, M., Pisov, M., Shevtsov, A., Shirokikh, B., Kurmukov, A., Blokhin, I., Chernina, V., Solovev, A., Gombolevskiy, V., Morozov, S., Belyaev, M., 2021.  Ct-based covid-19 triage: Deep multitask learning improves joint identification and severity quantification.  Medical Image Analysis 71, 102054\.  doi:https://doi.org/10.1016/j.media.2021.102054.
  * He et al. [2016] He, K., Zhang, X., Ren, S., Sun, J., 2016.  Deep residual learning for image recognition, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE. p. 770–778.  doi:10.1109/CVPR.2016.90.
  * Islam et al. [2021] Islam, M., VS, V., Lim, C.M., Ren, H., 2021.  St-mtl: Spatio-temporal multitask learning model to predict scanpath while tracking instruments in robotic surgery.  Medical Image Analysis 67, 101837\.  doi:https://doi.org/10.1016/j.media.2020.101837.
  * Isola et al. [2017] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A., 2017\.  Image-to-image translation with conditional adversarial networks.  Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) .
  * Kim et al. [2020] Kim, D., Lee, S., Lee, J., Kim, J., 2020.  Leveraging contextual information for monocular depth estimation.  IEEE Access 8, 147808–147817.  doi:10.1109/ACCESS.2020.3016008.
  * Lee et al. [2019] Lee, J.H., Han, M.K., Ko, D.W., Suh, I.H., 2019.  From big to small: Multi-scale local planar guidance for monocular depth estimation.  arXiv preprint arXiv:1907.10326 .
  * Li et al. [2023] Li, L., Qin, J., Lv, L., Cheng, M., Wang, B., Xia, D., Wang, S., 2023.  Icunet++: an inception-cbam network based on unet++ for mr spine image segmentation.  International Journal of Machine Learning and Cybernetics 14, 3671–3683.  doi:10.1007/s13042-023-01857-y.
  * Liu et al. [2022] Liu, S., Fan, J., Song, D., Fu, T., Lin, Y., Xiao, D., Song, H., Wang, Y., Yang, J., 2022.  Joint estimation of depth and motion from a monocular endoscopy image sequence using a multi-loss rebalancing network.  Biomedical Optics Express 13, 2707\.  doi:10.1364/BOE.457475.
  * Long et al. [2021] Long, X., Lin, C., Liu, L., Li, W., Theobalt, C., Yang, R., Wang, W., 2021.  Adaptive surface normal constraint for depth estimation, in: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE Computer Society, Los Alamitos, CA, USA. pp. 12829–12838.  doi:10.1109/ICCV48922.2021.01261.
  * Ma et al. [2021] Ma, R., Wang, R., Zhang, Y., Pizer, S., McGill, S.K., Rosenman, J., Frahm, J.M., 2021\.  Rnnslam: Reconstructing the 3d colon to visualize missing regions during a colonoscopy.  Medical Image Analysis 72, 102100\.  doi:https://doi.org/10.1016/j.media.2021.102100.
  * Mahmood and Durr [2018] Mahmood, F., Durr, N.J., 2018\.  Deep learning and conditional random fields-based depth estimation and topographical reconstruction from conventional endoscopy.  Medical Image Analysis 48, 230–243.  doi:https://doi.org/10.1016/j.media.2018.06.005.
  * McGill et al. [2018] McGill, S.K., Rosenman, J., Zhao, Q., Wang, R., Ma, R., Fan, M., Niethammer, M., Alterovitz, R., Frahm, J.M., Tepper, J., Pizer, S., 2018\.  Sa1930 missed colonic surface area at colonoscopy can be calculated with computerized 3D reconstruction.  Gastrointestinal Endoscopy 87, AB254.  doi:10.1016/j.gie.2018.04.452.
  * Ming et al. [2021] Ming, Y., Meng, X., Fan, C., Yu, H., 2021.  Deep learning for monocular depth estimation: A review.  Neurocomputing 438, 14–33.  doi:10.1016/j.neucom.2020.12.089.
  * Nakagawa et al. [2015] Nakagawa, Y., Uchiyama, H., Nagahara, H., Taniguchi, R.I., 2015\.  Estimating surface normals with depth image gradients for fast and accurate registration, in: 2015 International Conference on 3D Vision, pp. 640–647.  doi:10.1109/3DV.2015.80.
  * Nathan Silberman and Fergus [2012] Nathan Silberman, Derek Hoiem, P.K., Fergus, R., 2012.  Indoor segmentation and support inference from rgbd images, in: European Conference on Computer Vision (ECCV), pp. 746–760.
  * Patil et al. [2022] Patil, V., Sakaridis, C., Liniger, A., Van Gool, L., 2022\.  P3Depth: monocular depth estimation with a piecewise planarity prior, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1610–1621.
  * Piccinelli et al. [2023] Piccinelli, L., Sakaridis, C., Yu, F., 2023.  idisc: Internal discretization for monocular depth estimation, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21477–21487.
  * Qi et al. [2018] Qi, X., Liao, R., Liu, Z., Urtasun, R., Jia, J., 2018\.  Geonet: Geometric neural network for joint depth and surface normal estimation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 283–291.
  * Rau et al. [2019] Rau, A., Edwards, P.J.E., Ahmad, O.F., Riordan, P., Janatka, M., Lovat, L.B., Stoyanov, D., 2019.  Implicit domain adaptation with conditional generative adversarial networks for depth prediction in endoscopy.  International Journal of Computer Assisted Radiology and Surgery 14, 1167–1176.  doi:10.1007/s11548-019-01962-w.
  * Recasens et al. [2021] Recasens, D., Lamarca, J., Fácil, J.M., Montiel, J.M.M., Civera, J., 2021.  Endo-depth-and-motion: Reconstruction and tracking in endoscopic videos using depth networks and photometric constraints.  IEEE Robotics and Automation Letters 6, 7225–7232.  doi:10.1109/LRA.2021.3095528.
  * Rex et al. [2015] Rex, D.K., Schoenfeld, P.S., Cohen, J., Pike, I.M., Adler, D.G., Fennerty, M.B., Lieb, J.G., Park, W.G., Rizk, M.K., Sawhney, M.S., Shaheen, N.J., Wani, S., Weinberg, D.S., 2015.  Quality indicators for colonoscopy.  Gastrointestinal Endoscopy 81, 31–53.  doi:10.1016/j.gie.2014.07.058.
  * Shao et al. [2022] Shao, S., Pei, Z., Chen, W., Zhu, W., Wu, X., Sun, D., Zhang, B., 2022.  Self-supervised monocular depth and ego-motion estimation in endoscopy: Appearance flow to the rescue.  Medical Image Analysis 77, 102338\.  doi:https://doi.org/10.1016/j.media.2021.102338.
  * Siegel et al. [2023] Siegel, R.L., Wagle, N.S., Cercek, A., Smith, R.A., Jemal, A., 2023.  Colorectal cancer statistics, 2023.  CA: A Cancer Journal for Clinicians 73, 233–254.  URL: https://acsjournals.onlinelibrary.wiley.com/doi/10.3322/caac.21772, doi:10.3322/caac.21772.
  * Simonyan and Zisserman [2015] Simonyan, K., Zisserman, A., 2015\.  Very deep convolutional networks for large-scale image recognition, in: International Conference on Learning Representations.
  * Van Rijn et al. [2006] Van Rijn, J.C., Reitsma, J.B., Stoker, J., Bossuyt, P.M., Van Deventer, S.J., Dekker, E., 2006\.  Polyp miss rate determined by tandem colonoscopy: A systematic review.  The American Journal of Gastroenterology 101, 343–350.  doi:10.1111/j.1572-0241.2006.00390.x.
  * Wang et al. [2023] Wang, J., Zheng, Y., Ma, J., Li, X., Wang, C., Gee, J., Wang, H., Huang, W., 2023.  Information bottleneck-based interpretable multitask network for breast cancer classification and segmentation.  Medical Image Analysis 83, 102687\.  doi:https://doi.org/10.1016/j.media.2022.102687.
  * Wang et al. [2019] Wang, R., Pizer, S.M., Frahm, J.M., 2019.  Recurrent neural network for (un-)supervised learning of monocular video visual odometry and depth, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5550–5559.
  * Watson et al. [2021] Watson, J., Aodha, O.M., Prisacariu, V., Brostow, G., Firman, M., 2021.  The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth, in: Computer Vision and Pattern Recognition (CVPR), pp. 1164–1174.
  * Woo et al. [2018] Woo, S., Park, J., Lee, J.Y., Kweon, I.S., 2018.  Cbam: Convolutional block attention module, in: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (Eds.), Computer Vision – ECCV 2018, Springer International Publishing. p. 3–19.  doi:10.1007/978-3-030-01234-2_1.
  * Xue et al. [2018] Xue, W., Brahm, G., Pandey, S., Leung, S., Li, S., 2018.  Full left ventricle quantification via deep multitask relationships learning.  Medical Image Analysis 43, 54–65.  doi:https://doi.org/10.1016/j.media.2017.09.005.
  * Yang et al. [2023] Yang, Y., Shao, S., Yang, T., Wang, P., Yang, Z., Wu, C., Liu, H., 2023.  A geometry-aware deep network for depth estimation in monocular endoscopy.  Engineering Applications of Artificial Intelligence 122, 105989.  URL: https://www.sciencedirect.com/science/article/pii/S0952197623001732, doi:https://doi.org/10.1016/j.engappai.2023.105989.
  * Yuan et al. [2022] Yuan, W., Gu, X., Dai, Z., Zhu, S., Tan, P., 2022\.  Newcrfs: Neural window fully-connected crfs for monocular depth estimation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3906–3915.
  * Zamir et al. [2020] Zamir, A.R., Sax, A., Cheerla, N., Suri, R., Cao, Z., Malik, J., Guibas, L.J., 2020.  Robust learning through cross-task consistency, in: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11194–11203.  doi:10.1109/CVPR42600.2020.01121.
  * Zhang et al. [2022] Zhang, J., Peng, H., Wu, K., Liu, M., Xiao, B., Fu, J., Yuan, L., 2022.  MiniViT: compressing vision transformers with weight multiplexing, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12145–12154.
  * Zhang et al. [2021a] Zhang, S., Zhao, L., Huang, S., Ye, M., Hao, Q., 2021a.  A template-based 3D reconstruction of colon structures and textures from stereo colonoscopic images.  IEEE Transactions on Medical Robotics and Bionics 3, 85–95.  doi:10.1109/TMRB.2020.3044108.
  * Zhang et al. [2021b] Zhang, S., Zhao, L., Huang, S., Ye, M., Hao, Q., 2021b.  A template-based 3d reconstruction of colon structures and textures from stereo colonoscopic images.  IEEE Transactions on Medical Robotics and Bionics 3, 85–95.  doi:10.1109/TMRB.2020.3044108.
  * Zhang et al. [2021c] Zhang, Y., Frahm, J.M., Ehrenstein, S., McGill, S.K., Rosenman, J.G., Wang, S., Pizer, S.M., 2021c.  Colde: a depth estimation framework for colonoscopy reconstruction.  arXiv preprint arXiv:2111.10371 .
  * Zou et al. [2018] Zou, Y., Luo, Z., Huang, J.B., 2018.  DF-Net: unsupervised joint learning of depth and flow using cross-task consistency, in: European Conference on Computer Vision (ECCV), pp. 38–55.