Towards optimal multimode fiber imaging by leveraging input polarization and
conditional generative adversarial networks

Jawaria Maqbool,1, Syed Talal Hassan,2 and M. Imran Cheema1∗

Department of Electrical Engineering, Syed Babar Ali School of Science and
Engineering, Lahore University of Management Science,Sector U, D.H.A Lahore,
Cantt, Punjab 54792, Pakistan1
Department of Computer Science, Syed Babar Ali School of Science and
Engineering, Lahore University of Management Science,Sector U, D.H.A Lahore,
Cantt, Punjab 54792, Pakistan2
<EMAIL_ADDRESS>
## Abstract

Deep learning techniques provide a plausible route towards achieving practical
imaging through multimode fibers. However, the results produced by these
methods are often influenced by physical factors like temperature, fiber
length, external perturbations, and polarization state of the input light. The
impact of other factors, except input light polarization, has been discussed
in the literature for imaging applications. The input polarization has been
considered by researchers while looking at the characterization and control of
polarization in multimode fibers. Here, we show experimentally that the state
of polarization of light, being injected at multimode fiber input, affects the
fidelity of reconstructed images from speckle patterns. Certain polarization
states produce high-quality images at fiber output, while some yield degraded
results. We have designed a conditional generative adversarial network (CGAN)
for image regeneration at various degrees of input light polarization. We
demonstrate that in the case of multimode fibers that are held fixed, optimal
imaging can be achieved by leveraging our CGAN model with the input light
polarization state, where the fidelity of images is maximum. Our work exhibits
high average structural similarity index values exceeding 0.9, surpassing the
previously reported value of 0.8772. We also show that the model can be
generalized to image adequately for all input light polarization states when
the fiber has bends or twists. We anticipate our work will be a stepping stone
toward developing high-resolution and less invasive multimode fiber
endoscopes.

##  1 Introduction

Multimode fibers (MMFs) can lead to practical endoscopes because they are
thinner and less invasive than single-mode fiber bundles [1, 2, 3]. The
presence of numerous spatial modes in MMFs can be harnessed for transmitting
images. However, light waves propagating through different fiber modes
interfere with each other to form speckles or random patterns at the fiber’s
distal end. Hence, its properties resemble scattering or disordered media like
fog, diffusers, and biological tissues that scramble the information to
produce a speckle phenomenon. Extraction of data from speckle patterns is a
challenging task. Primarily, three different strategies are generally employed
for image reconstruction from speckles: optical phase conjugation [4, 5],
computation of transmission matrix [6, 7, 8, 9], and deep learning [10]. Phase
conjugation incorporates a complex interferometric method for measuring phase,
and precise alignment is required between the camera and spatial light
modulator. The transmission matrix (TM), on the other hand, aptly describes
the relationship between an MMF input and output. It helps to assimilate the
information about light absorption, reflection, and transmission through the
medium. The TM measurement requires both amplitude and phase information. The
accurate phase computation needs a stable reference arm and nontrivial
interference setup. Moreover, phase values are very sensitive to external
perturbations. Hence, one TM can only be used for the transmission state in
which it is calculated [6]. Recent research indicates that the challenges
mentioned above can be addressed by applying deep learning techniques, leading
to a more effective imaging process using MMFs [11, 12].

Previous deep learning works have shown various ways to improve MMF imaging in
terms of accuracy, generalizability, and data requirements [13, 14, 12, 15,
16, 17, 18]. However, the effect of input polarization state changes on the
reconstruction of images from speckles of multimode fibers has yet to be
discussed thoroughly. Prior research has been done in the past on the
characterization, statistics, and control of the polarization of light in
multimode fibers. Due to random mode interference in multimode fibers,
polarization mixing also occurs, which results in depolarized or partially
polarized output [19]. On the other hand, it has been shown in [20] that the
field distribution of some modes does not change during propagation through
the fiber. Moreover, complete control of output polarization can be achieved
using the eigenvectors and eigenvalues of the multimode fiber TM with
orthogonal polarizations as a basis [21]. Considering these previous works, we
hypothesize that input polarizations can affect the reconstruction of images
at multimode fiber output and should be utilized to improve the MMF imaging
process.

Here, we devise an experimental and computational way to quantify input
polarization impact on multimode fiber imaging. We acquire output data of
speckles for nine input polarization states at multiple MMF positions. We
reconstruct original images from the acquired datasets using our designed
conditional generative adversarial network (CGAN). Our CGAN model is fast
(training time: 1 hour, inference time: 5.4 ms), stable, and gives better
reconstruction results. By varying the input polarizations and the fiber
positions, we show that our system can produce average structural similarity
index (SSIM) values above 0.9, which is higher than the previous value of
0.8772 [14].

We find that our model trained for one polarization state at a particular
fiber position gives poor reconstruction results for another polarization
state. To improve the generalizability of our deep learning model to
reconstruct images for all polarization states at a specific fiber position,
we merge an equal percentage of data from all nine polarization datasets to
form one combined dataset. The model is trained on this dataset and tested on
unseen data of each polarization state. This procedure is carried out
separately for two fiber positions. Furthermore, we integrate subsets of
eighteen datasets for both fiber positions. After training on this superset,
our CGAN model can accurately reconstruct images for unseen data of all
polarization states of two fiber positions under consideration. Hence, our
work highlights that the input light polarization state affects the accuracy
of reconstructed images from speckles at the multimode fiber output and it can
be harnessed in two ways: 1) For a fixed MMF orientation, input polarization
can be set to a degree where we get optimal imaging results,2) But in
scenarios, where fiber position can change, we must train our model on the
data measured while constantly changing fiber position and input light
polarization state. In this way, we can get satisfactory reconstruction
results for any input polarization state.

We now describe the rest of the paper. Section 2 details the experimental
setup for data collection, followed by Section 3, in which we describe the
data acquisition procedure. Section 4 is dedicated to our deep learning
framework, where we introduce its architecture, training processes, and
integration with the data gathered in the previous sections. Section 5
presents our methodology for evaluating the input polarization impact on the
reconstructed images’ quality and offers insights into the system’s
sensitivity to polarization variations. Section 6 explains our model’s
generalization ability to diverse input polarizations. Finally, Section 7
summarizes our findings and highlights potential avenues for future research.

##  2 Experimental setup

The experimental setup schematic is illustrated in Fig. 1. We utilize a 633 nm
continuous-wave laser diode (Eagleyard GC-02940) operated via Thorlabs CLD1015
controller. After reflection by mirrors, the laser light is collimated through
a telescopic system comprising two lenses with focal lengths of 500 mm and 100
mm. A polarizer is placed after the telescopic system to achieve horizontal
polarization for optimal phase modulation with the HOLOEYE Pluto 2.0 spatial
light modulator (SLM). The polarized laser beam is then directed onto a 50/50
beam splitter (BS). Half of the beam is transmitted towards the SLM, while a
beam blocker blocks the remaining half. Once reflected by the SLM, the phase-
modulated light passes through the BS and is imaged by lens 3 onto collimator
2, which in turn focuses the image of the phase-modulated light onto the input
of a multimode fiber. The multimode fiber has core and cladding diameters of
50 $\mu$m and 125 $\mu$m, respectively, with a length of 1 m and a numerical
aperture (NA) of 0.22. Before the fiber input, a half-wave plate (HWP) and a
quarter-wave plate (QWP) are positioned to attain any desired state of
polarization (SOP). The multimode fiber converts all the information the laser
light carries into a speckle pattern. The speckle pattern emerging from
collimator 3 is imaged by lens 3 onto a Thorlabs DCC1545M CMOS camera with a
resolution of 1280$\times$1024 pixels.

Figure 1: Experimental schematic illustrating our multimode fiber (MMF)
imaging process using a combination of input polarization and deep learning
techniques. The illustration depicts only one example of MMF positioning. For
other fiber positions, curling or bending the fiber is done arbitrarily.
C:Collimator, M:Mirror, L:Lens, P:Linear polarizer, BS:Beam splitter,
SLM:Spatial light modulator, HWP:Half wave plate, QWP:Quarter wave plate, and
MMF:Multimode fiber.

##  3 Data acquisition for different polarization states

Initially, we place the multimode fiber in position 1, as shown in Fig. 1, and
its orientation remains fixed for all measurement sets. Fixing the position is
essential as speckle patterns change with a change in the orientation of the
fiber [15]. A state of polarization (SOP) at the fiber input is set with HWP
and QWP while observing SOP on Thorlabs’s PAX1000IR1/M polarimeter. We choose
nine different SOPs comprising linear, circular, and elliptical polarizations,
as shown in Fig. 2. The Stoke’s parameters indicated in Fig. 2 are measured
using the polarimeter. For each input SOP, a computer sends the Modified
National Institute of Standards and Technology (MNIST) data of 60,000
handwritten digits on SLM. The images are of size 28$\times$28 pixels and are
up-sampled to 64$\times$64 pixels before being sent on SLM. The light
reflected from SLM now contains images of handwritten digits. After passing
through MMF, this light produces speckle patterns recorded by the computer
connected to the camera. The speckle patterns saved on the system are cropped
to dimensions of $256\times 256$ pixels. The process of speckle data
collection for 60,000 images takes approximately 24 hours. We perform this
procedure for nine input polarization states, resulting in the formation of
nine different datasets. To further gauge the input polarization effect on the
accuracy of imaging through MMF, we change the fiber position and repeat the
methodology of acquiring nine datasets for nine different polarization states.

Figure 2: The nine polarization states used in this work are depicted on the
Poincaré sphere along with the three Stokes parameters (S1,S2,S3) [22] that
are measured experimentally.

##  4 Deep learning framework for image reconstruction

After the formation of data sets at various distinct input polarization
states, the next step is reconstructing original images from speckle patterns.
For this, we design a pix2pix model based on CGAN, as shown in detail in Fig.
3. The generator is a U-Net-type architecture with an encoder, decoder, and
skip connections. We first down-sample the speckle patterns to size
64$\times$64$\times$1 and apply them as the input to the generator. The
generator is enabled with robust feature extraction capabilities due to
several convolution and deconvolution layers. In addition, the skip
connections allow weight sharing and preserve feature information across
different network layers. The output has the same resolution as the input. The
discriminator is composed of five convolution layers and one flattened layer.
The generator’s output, concatenated with the true label, is employed as input
to the discriminator, which works on classifying a patch in an image as real
or fake.

Figure 3: The structure of our designed conditional generative adversarial
network (CGAN).

CGAN has been used previously for reconstructing images from speckle patterns
produced by multimode fibers [17, 18]. The highest average SSIM reported
previously is 0.8772 [14]. In contrast to previous works, we use binary cross
entropy (BCE) loss for the discriminator and an amalgam of mean squared error
(MSE) and mean absolute error (MAE) for the generator. The MSE loss function
minimizes the difference between real and generated data. It also overcomes
the problem of vanishing gradients resulting in stable training and high-
fidelity results [23]. MAE loss aids in the regeneration of low-frequency
details. The hybrid loss function for a CGAN is the weighted sum of generative
and discriminative loss (Eq.(1)). We define the discriminator and generator
loss for our designed model in Eq.(2) and Eq.(3), respectively:

$\mathcal{L}_{CGAN}=\mathcal{L}_{Gen}+\mathcal{L}_{Disc},$ (1)
$\mathcal{L}_{Disc}=\lambda_{1}[l_{1}(D(y,x),1]+\lambda_{1}[l_{1}(D(G(x),x),0],$
(2) $\mathcal{L}_{Gen}=l_{2}[D(G(x),x),1]+\lambda_{2}[l_{3}((G(x),y)],$ (3)

where $G()$ and $D()$ are the generator and discriminator functions,
respectively. The speckle pattern inputs are represented by $x$ while the true
labels are denoted by $y$. We use $l_{1}$ for BCE, $l_{2}$ for MSE, and
$l_{3}$ for MAE. To optimize the model’s performance, we incorporate weighting
factors, $\lambda_{1}$ and $\lambda_{2}$, set at values of 100 and 0.5,
respectively, to effectively balance the MAE and BCE losses. Out of 60000
pairs of speckle-MNIST digits in all polarization datasets, we reserve 5000
pairs for testing. For the remaining 55000 pairs, 85$\%$ are kept for
training, and the rest 15$\%$ are used for validation. For each dataset, the
CGAN model takes 1 hour to train for 80 epochs, and the inference time for
each reconstructed image is 4.6 ms. We realize the data collection using a
Python 3.10.12 environment. Furthermore, we utilize the PyTorch framework for
building, training, and testing the model. The whole mechanism of deep
learning is accelerated by the NVIDIA Tesla V100 Tensor Core GPU.

We train and evaluate our CGAN model for all compiled polarization data sets
at fiber positions 1 and 2. We use SSIM and peak signal-to-noise ratio (PSNR)
as evaluation metrics for our restored images. SSIM compares the similarity
between reconstructed digits and true labels based on their luminance,
contrast, and structure. Its value varies between 0 and 1. SSIM value around
zero means no similarity between two images, while a value closer to 1 denotes
that the images are almost identical. Its expression is given by:

$SSIM=\frac{(2\mu_{x}\mu_{y}+C_{1})(2\sigma_{xy}+C_{2})}{(\mu_{x}^{2}+\mu_{y}^{2}+C_{1})(\sigma_{x}^{2}+\sigma_{y}^{2}+C_{2})},$
(4)

where $\mu_{x}$ and $\mu_{y}$ refer to the mean value over a window in images
$x$ and $y$, respectively. $\sigma_{x}$ and $\sigma_{y}$ are standard
deviations over a window of $x$ and $y$. $\sigma_{xy}$ is the covariance over
a window between image $x$ and image $y$ while $C_{1}$ and $C_{2}$ are
constants.

Another metric that we have used is PSNR, which is the ratio between the
maximum pixel value of the ground truth image ($I_{max}$) and the root mean
squared error (RMSE) and is given by:

$PSNR=20log_{10}\dfrac{I_{max}}{RMSE}.$ (5)

RMSE is determined between the pixel values of the original and the predicted
images. The higher the value of PSNR, the better the quality of reconstructed
images. Some of the reconstruction results from our designed CGAN are given in
Fig. 4 and Fig. 5 for fiber positions 1 and 2, respectively. For brevity, the
regenerated images for each fiber position are displayed for only two
polarization states: one where SSIM and PSNR attain their respective maximum
values and the other where they reach their minimum values. This choice is
made to show a proper difference in image fidelity at these polarization
states. As can be observed in Fig. 4, at polarization P9 (elliptical), the
images are closer to true labels (have high SSIM and PSNR) as compared to
polarization P4 (45-degree). P4 has relatively poor image regeneration
results, especially for digits 2 and 3. The same can be apprehended from the
results of Fig. 5, where average PSNR and SSIM attain their highest values at
P1 (nearly horizontal) and lowest at P7 (elliptical).

Figure 4: Representative image reconstruction results at two different
polarization states for position 1 of MMF. At P9 (elliptical), the average
SSIM and PSNR are maximum, while these metrics have the lowest average values
at P4 (45-degree). Please see Fig. 2 P9 and P4 Stokes parameters. Figure 5:
Representative reconstruction results at two different polarization states for
position 2 of MMF. At P1 (nearly horizontal), the average SSIM and PSNR are
maximum, while these metrics have the lowest average values at P7
(elliptical). Please see Fig. 2 for P1 and P7 Stokes parameters.

##  5 Assessing the input polarization effect

We record eighteen different data sets for two distinct positions of the
multimode fiber at nine varying input polarization states shown in Fig. 2. We
then train our designed CGAN for these data sets, followed by the model
evaluation for 5000 unseen test images. The obtained average SSIM and PSNR for
every data set are shown in Figs. 6 and 7. The varying magnitudes of these bar
graphs illustrate that SSIM and PSNR change with input polarization states.
For position 1, the maximum average SSIM of 0.9010 and PSNR of 22.89 are
attained at elliptical polarization. At the same time, the minimum SSIM of
0.8430 and PSNR of 20.182 are obtained for linearly polarized light at 45
degrees. The percentage difference between the smallest and largest SSIM is
6.65$\%$, while for PSNR, this variation is 12.5$\%$. For position 2, the
highest average SSIM of 0.9046 and PSNR of 23.202 are achieved for nearly
horizontally polarized light. The lowest SSIM of 0.8225 and PSNR of 19.458 are
obtained when the input light is vertically polarized. In this case, the
percentage difference between the two extremities is 9.5$\%$ for SSIM and
17.55$\%$ for PSNR. We find that the deviation between evaluation metrics’
values for some polarization states is more significant than others. It can
also be inferred from the plots that the effect of different polarization
states changes with a change in the fiber position. For example, at P1 and
position 2, SSIM of 0.9046 and PSNR of 23.202 are high, but for position 1 and
the same polarization state, these metrics have reduced to 0.8632 and 21.302,
respectively. This is because modal distribution changes with the bending or
twisting of the fiber.

We repeat the data collection, model training, and testing procedure twice at
each of the nine polarization states and for individual fiber positions. This
is done to ensure the capture of the persistent impact of different
polarization states on the fidelity of reconstructed images. When evaluated
across various input polarization states, we observe that the percentage
difference in SSIM and PSNR values for reconstructed images exhibit consistent
results with only 1-2$\%$ marginal fluctuations. This means that if SSIM and
PSNR are minimum at P4 (45-degree linear) compared to other polarization
states, it will always be minimal, no matter how many times we repeat this
process while keeping the position fixed.

Figure 6: The average SSIM and PSNR variations of 5000 unseen test images for
individual polarization states when the fiber is fixed in position 1. Figure
7: The average SSIM and PSNR variations of 5000 unseen test images for
individual polarization states when the fiber is fixed in position 2.

Physically, the relationship of reconstruction of images with input
polarization states can be elaborated in the following way. When light with a
specific polarization state is launched to any fiber mode, it spreads to other
modes. Due to modal coupling, polarization scrambling also occurs, resulting
in different polarization states at the inputs and outputs of all modes.
Moreover, higher-order modes suffer from higher attenuation than lower-order
modes. For an arbitrary polarized ($p$) input $|\phi\bigr{\rangle}$ the output
field is $|\psi\bigr{\rangle}=t_{p}|\phi\bigr{\rangle}$, where $t_{p}$ is the
transmission matrix for $p$ polarized input. The total intensity of this
polarization state is
$\bigr{\langle}\psi|\psi\bigr{\rangle}=\bigr{\langle}\phi|t_{p}^{\dagger}t_{p}|\phi\bigr{\rangle}$.
The transmission range achieved in this state is defined by the eigenvalues of
$t_{p}^{\dagger}t_{p}$. The maximum energy that can be maintained in the same
state of polarization is given by the largest eigenvalue, while the maximum
energy retained in orthogonal SOP is defined by the smallest eigenvalue [21].
The larger eigenvalues and their associated eigenvectors get their
contribution from lower-order modes, leading to maximum transmission. The
eigenvectors corresponding to smaller eigenvalues are influenced by higher-
order modes, resulting in reduced transmission. Also, input wavefronts change
due to data sent on SLM. For some states of input polarization, the
eigenvectors of most of the wavefronts from SLM coincide with greater
eigenvalues of $t_{p}^{\dagger}t_{p}$, contributing to the maximum
transmission of these wavefronts. This eventually improves the fidelity of
reconstructed images as most of the input information is retained while
propagating through fiber. On the contrary, for certain input SOPs,
eigenvectors of input wavefronts correspond to smaller eigenvalues, causing
the attenuation of input information. SSIM and PSNR will be low in these
cases. Also, due to variations in mode and polarization coupling, as well as
changes in the transmission matrix and its eigenvalues with respect to the
fiber’s position, the influence of input polarization differs between the two
fiber positions.

##  6 Generalization for varying input polarization states

The input polarization effect can be harnessed in two ways: (a) The input
polarization is set at a degree where we get the optimal imaging results for
endoscopic applications where multimode fiber length is small and is not bent
or twisted while imaging [24]. (b) In the case of long-length endoscopes
inserted deeply in the body, a dynamically perturbed multimode fiber should
also be trained or calibrated for a diverse range of input polarization
degrees. This approach ensures consistently satisfactory imaging results
regardless of the input polarization degree.

Figure 8: The average SSIM and PSNR variation of 5000 unseen test images for
individual polarization states when the fiber is fixed in position 1. First,
the model is trained on combined data that contains $15\%$ images from every
polarization state dataset and then is tested for all polarization states.
Figure 9: The average SSIM and PSNR variation of 5000 unseen test images for
individual polarization states when the fiber is fixed in position 2. First,
the model is trained on combined data that contains $15\%$ images from every
polarization state dataset and then is tested for all polarization states.
Figure 10: The average SSIM and PSNR variations of 5000 unseen test images for
individual polarization states for positions 1 and 2. First, the model is
trained on combined data that contains 10$\%$ images from each polarization
state dataset of both positions and then is tested for all polarization
states.

To start with the generalization mechanism, we first use the weights of one
input polarization state for reconstructing the test images of another input
polarization data. Not surprisingly, we get poor results as the average SSIM
and PSNR remain below 0.2 and 8, respectively. We then combine 15$\%$ training
data of each of the nine input polarization states to form one dataset with
74250 speckle-label pairs for fiber positions 1 and 2. We train our designed
CGAN model on this collective set of images and test for 5000 unseen images of
each polarization state and fiber position. The average SSIM and PSNR of nine
polarization states tested on weights of combined datasets are plotted in Fig.
8 and Fig. 9 for positions 1 and 2, respectively. The SSIM and PSNR values are
reasonable in this case. SSIM is not less than 0.7, and PSNR is greater than
17 for position 1. Likewise, for position 2 combined data, SSIM and PSNR
remain above 0.8 and 18, respectively.

As a final step towards the generalization process where an MMF can image well
for any input polarisation state (P1-P9) and position (1 or 2), we pick up
10$\%$ training data from 18 datasets and integrate them to form one super set
of 133650 images. After training on this set, we test our designed CGAN model
for unseen images of all polarization states at both fiber positions. In this
case, the average SSIM and PSNR are shown in Fig. 10. It is noticeable from
this bar graph that SSIM does not degrade more than 0.75 while PSNR is greater
than 17. These reasonable metric values signify successful image
reconstruction for any input polarization state. This process indicates that
the CGAN model should be trained on a dataset obtained while dynamically
changing the input polarization state and fiber positions for better
generalization. Table 1 encapsulates the mean and standard deviation of
evaluation metrics when training is done separately on each polarization
dataset and for a combination of different polarization and position datasets.
As can be seen, mean values are higher after training on individual
polarization datasets. However, as discussed previously, the model trained for
one polarization does not reconstruct well for another polarization state. The
mean metric values are relatively low for a combination of datasets, but
generalizability is improved. The small standard deviation values in these
cases depict that the model can image well for various polarization states.

Fiber Position |  No. of training samples |  Mean SSIM for all input polarization states |  SSIM standard deviation for all input polarization states |  Mean PSNR for all input polarization states |  PSNR standard deviation for all input polarization states
---|---|---|---|---|---
Position 1 (separate dataset for each polarization state) |  55000 |  0.8750 |  0.0169 |  21.640 |  0.7684
Position 2 (separate dataset for each polarization state) |  55000 |  0.8771 |  0.023 |  21.865 |  1.067
Position 1 (combined dataset for all polarization states) |  74250 |  0.8215 |  0.022 |  19.376 |  0.8475
Position 2 (combined dataset for all polarization states) |  74250 |  0.8255 |  0.0105 |  19.554 |  0.3437
Position 1 $\&$2 (Combined dataset for all polarization states) |  133650 |  0.8122 |  0.0181 |  19.134 |  0.6138
Table 1: Number of training samples and mean metric values for individual
polarization and fiber position datasets and a combination of these datasets

##  7 Conclusion

We demonstrate experimentally the influence of input light polarization on the
accuracy of image reconstructions from speckle patterns at the multimode fiber
output. Specifically, we have established a clear correlation between the
input polarization states and the variation in the average structural
similarity index (SSIM) and peak signal-to-noise ratio (PSNR) across a set of
5000 unseen images. Furthermore, we have exhibited this polarization impact
for two distinct multimode fiber positions. The high SSIM values exceeding 0.9
achieved at both fiber positions surpass the previously reported value of
0.8772. We conclude that we should incorporate that polarization state where
SSIM is maximum toward achieving optimal MMF imaging. Moreover, we have
generalized our model for the cases where input polarization and position of
the fiber are changing. By training on combined data from all polarization
states and fiber positions, we show that imaging through MMF can be done
satisfactorily for any polarization state and fiber position. Our work can be
extended to explore the influence of input polarization on multimode fiber-
optic communication systems. Furthermore, this research is transferable to
imaging applications through challenging mediums such as fog and biological
tissues. We believe that our work can significantly contribute to developing
compact and high-resolution endoscopes that do not require traditional lenses.

## References

  * [1] GD Hadley.  The gastro-camera.  British Medical Journal, 2(5472):1209, 1965.
  * [2] Min Gu, Hongchun Bao, and Hong Kang.  Fibre-optical microendoscopy.  Journal of microscopy, 254(1):13–18, 2014.
  * [3] Antonios Perperidis, Kevin Dhaliwal, Stephen McLaughlin, and Tom Vercauteren.  Image computing for fibre-bundle endomicroscopy: A review.  Medical image analysis, 62:101620, 2020.
  * [4] Ioannis N Papadopoulos, Salma Farahi, Christophe Moser, and Demetri Psaltis.  Focusing and scanning light through a multimode optical fiber using digital phase conjugation.  Optics express, 20(10):10583–10590, 2012.
  * [5] Ioannis N Papadopoulos, Salma Farahi, Christophe Moser, and Demetri Psaltis.  High-resolution, lensless endoscope based on digital scanning through a multimode optical fiber.  Biomedical optics express, 4(2):260–270, 2013.
  * [6] Sébastien M Popoff, Geoffroy Lerosey, Rémi Carminati, Mathias Fink, Albert Claude Boccara, and Sylvain Gigan.  Measuring the transmission matrix in optics: an approach to the study and control of light propagation in disordered media.  Physical review letters, 104(10):100601, 2010.
  * [7] Duygu Akbulut, Tom Strudley, Jacopo Bertolotti, Tilman Zehender, Erik PAM Bakkers, Ad Lagendijk, Willem L Vos, Otto L Muskens, and Allard P Mosk.  Measurements on the optical transmission matrices of strongly scattering nanowire layers.  In International Quantum Electronics Conference, page IH_P_19. Optica Publishing Group, 2013.
  * [8] Angélique Drémeau, Antoine Liutkus, David Martina, Ori Katz, Christophe Schülke, Florent Krzakala, Sylvain Gigan, and Laurent Daudet.  Reference-less measurement of the transmission matrix of a highly scattering material using a dmd and phase retrieval techniques.  Optics express, 23(9):11898–11911, 2015.
  * [9] Christopher A Metzler, Manoj K Sharma, Sudarshan Nagesh, Richard G Baraniuk, Oliver Cossairt, and Ashok Veeraraghavan.  Coherent inverse scattering via transmission matrices: Efficient phase retrieval algorithms and a public dataset.  In 2017 IEEE International Conference on Computational Photography (ICCP), pages 1–16. IEEE, 2017.
  * [10] Navid Borhani, Eirini Kakkava, Christophe Moser, and Demetri Psaltis.  Learning to see through multimode fibers.  Optica, 5(8):960–966, 2018.
  * [11] Babak Rahmani, Damien Loterie, Georgia Konstantinou, Demetri Psaltis, and Christophe Moser.  Multimode optical fiber transmission with a deep learning network.  Light: science & applications, 7(1):69, 2018.
  * [12] Pengfei Fan, Tianrui Zhao, and Lei Su.  Deep learning the high variability and randomness inside multimode fibers.  Optics express, 27(15):20241–20258, 2019.
  * [13] Changyan Zhu, Eng Aik Chan, You Wang, Weina Peng, Ruixiang Guo, Baile Zhang, Cesare Soci, and Yidong Chong.  Image reconstruction through a multimode fiber with a simple neural network architecture.  Scientific reports, 11(1):896, 2021.
  * [14] Binbin Song, Chang Jin, Jixuan Wu, Wei Lin, Bo Liu, Wei Huang, and Shengyong Chen.  Deep learning image transmission through a multimode fiber based on a small training dataset.  Optics express, 30(4):5657–5672, 2022.
  * [15] Shachar Resisi, Sebastien M Popoff, and Yaron Bromberg.  Image transmission through a dynamically perturbed multimode fiber by deep learning.  Laser & Photonics Reviews, 15(10):2000553, 2021.
  * [16] Linh V Nguyen, Cuong C Nguyen, Gustavo Carneiro, Heike Ebendorff-Heidepriem, and Stephen C Warren-Smith.  Sensing in the presence of strong noise by deep learning of dynamic multimode fiber interference.  Photonics Research, 9(4):B109–B118, 2021.
  * [17] Zhenming Yu, Zhenyu Ju, Xinlei Zhang, Ziyi Meng, Feifei Yin, and Kun Xu.  High-speed multimode fiber imaging system based on conditional generative adversarial network.  Chinese Optics Letters, 19(8):081101, 2021.
  * [18] Xiaoyan Wang, Zhiyuan Wang, Songjie Luo, Ziyang Chen, and Jixiong Pu.  Upconversion imaging through multimode fibers based on deep learning.  Optik, 264:169444, 2022.
  * [19] Moti Fridman, Haim Suchowski, Micha Nixon, Asher A Friesem, and Nir Davidson.  Modal dynamics in multimode fibers.  JOSA A, 29(4):541–544, 2012.
  * [20] Martin Plöschner, Tomáš Tyc, and Tomáš Čižmár.  Seeing through chaos in multimode fibres.  Nature Photonics, 9(8):529–535, 2015.
  * [21] Wen Xiong, Chia Wei Hsu, Yaron Bromberg, Jose Enrique Antonio-Lopez, Rodrigo Amezcua Correa, and Hui Cao.  Complete polarization control in multimode fibers with polarization and mode coupling.  Light: Science & Applications, 7(1):54, 2018.
  * [22] David S Kliger and James W Lewis.  Polarized light in optics and spectroscopy.  Elsevier, 2012.
  * [23] Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley.  Least squares generative adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2794–2802, 2017.
  * [24] Sergey Turtaev, Ivo T Leite, Tristan Altwegg-Boussac, Janelle MP Pakan, Nathalie L Rochefort, and Tomáš Čižmár.  High-fidelity multimode fibre-based endoscopy for deep brain in vivo imaging.  Light: Science & Applications, 7(1):92, 2018.


# Predicting Event Attendance exploring Social Influence

Fatemeh Salehi Rizi and Michael Granitzer  Department of Computer Science and
Mathematics
University of Passau, Germany
Email<EMAIL_ADDRESS>
###### Abstract

The problem of predicting people’s participation in real-world events has
received considerable attention as it offers valuable insights for human
behavior analysis and event-related advertisement. Today social networks (e.g.
Twitter) widely reflect large popular events where people discuss their
interest with friends. Event participants usually stimulate friends to join
the event which propagates a social influence in the network. In this paper,
we propose to model the social influence of friends on event attendance. We
consider non-geotagged posts besides structures of social groups to infer
users’ attendance. To leverage the information on network topology we apply
some of recent graph embedding techniques such as node2vec, HARP and Poincarè.
We describe the approach followed to design the feature space and feed it to a
neural network. The performance evaluation is conducted using two large music
festivals datasets, namely the VFestival and Creamfields. The experimental
results show that our classifier outperforms the state-of-the-art baseline
with 89$\%$ accuracy observed for the VFestival dataset.

###### Index Terms:

Event Attendance Prediction, Network Embedding, Social Networks, Social
Influence

##  I Introduction

Social media platforms such as Twitter provide a great opportunity for people
to communicate about real-world events. Large popular events such as music
festivals attract thousands of participants who usually post their feelings or
opinions in a social media. Moreover, people often attend to events along with
their friends who are already connected in a social media. Event participants
can propagate a social influence in the network which affects friends’
attendance decisions. Therefore, analysis of participants’ posts besides their
network structures can detect the actual attendance to benefit several
applications like advertising or mobility management.

Predicting attendance exploiting geotagged posts is a simple task since the
user position is associated to the event location. However, few users indeed
make the geotagging [1] during the event which yields a sparse dataset.
Inferring attendance would be possible for the few users who shared the
location.

Previous research has studied that people usually participate in events with
their friends or family members who have linked in a social network [2, 3].
Therefore, users propagate an influence within small social groups which
should be considered to infer the attendance.

We speculate that modeling the social influence by encoding the structure
around a node in the graph improves the prediction task. To extract network
structural features we apply some of recent graph embedding techniques such as
node2vec [4], HARP [5] and Poincarè [6] which have shown prominent performance
for node classification and link prediction. To understand users’ attitudes
toward the event, we gather users’ posts before and during the event. The
posts shared before the event may indicate user interest and his intention to
attend. Posts during the event may express people’s feeling by sharing photos
and videos during the event.

We propose to train a neural network classifier fed by textual features of
posts and network topological features. Therefore, the contributions of this
paper are as follows.

  * •

We exploit network topological features applying graph embedding techniques to
model users’ social influences on event attendance (section III).

  * •

We inspect the impact of different embedding techniques on the performance of
the prediction (section IV).

  * •

We show our approach outperforms the most prominent work in the state-of-the-
art for event attendance prediction (section IV).

We conduct our experiments on two real-world datasets from music festivals
events held in UK. The evaluation results show that considering neighborhood
structures around users improve the performance as we achieve accuracy ranging
from $88\%$ to $89\%$.

The rest of the paper is organized as follows. Section II quickly reviews the
related works. In Section III, we describe our approach for event attendance
prediction. Section IV presents the evaluation results of the predicting
approach. The last section concludes the paper.

##  II Related work

Twitter is a common platform to reflect events where users post their opinions
and feeling. Lira at al. [7] extract the textual and temporal features of
tweets to infer users’ attendance, however they ignore the impact of friends
on users’ decisions. Several research explore Twitter geotagged posts to
decide about users’ attendance to real-world events. Magnuson et al. [8]
analyze the tweets with geo-location tags to create a recommendation system
which offers events based on user’s interest, time and geo-location. Similarly
[9] investigates whether mobile phone usage and the geo-located Twitter data
can be used to estimate the number of people in a specific area at a given
time. However, we have a different goal as we do not want to estimate the
user’s exact location at the time of the post, but classify the single posts
on the basis of the user’s current and past attendance to a given event.

In addition to location based services, event-based social networks (EBSN)
provide a new venue to analyze human social behaviors. Authors in [10] explore
users’ past activities in EBSN considering the temporal and spatial factors to
predict users’ decision on event attendance. But we do not exploit users’
history as we aim at classifying single posts by neglect of the user profiles
and specific event information. There has been some works aim to model social
influenec in EBSNs such as [2], [11] and [3]. However, we do not specifically
deal with ESBN graph but instead focus on popular social networks (e.g.
Twitter) where events have been reflected.

There has been numerous techniques to encode topology of a social graph in
state-of-the-art. The most popular works which preserve neighborhood patterns
effectively and efficiently are Node2vec [4], HARP[5] and Poincarè [6].
Node2vec is a random walk based approach which preserves higher order
proximity between nodes by maximizing the probability of occurrence of
subsequent nodes in fixed length random walks. Since large social graphs often
form innate hierarchical structures [12], we apply Poincarè and HARP to
capture hierarchical patterns. Poincarè use a hyperbolic space to alleviate
overfitting issues that euclidean embeddings face if the data has intrinsic
hierarchical structure. HARP captures first and second order proximities of
neighborhoods by collapsing the graph into smaller hierarchies. To our best
knowledge, inferring users’ attendance to large events exploring the social
influence has not been investigated yet.

##  III Problem Formalization

Given a user $u$ and a future event $e$, the goal is to predict whether $u$
will attend $e$ or not. We exploit the following information to predict the
user actual attendance to the event $e$:

  * •

The post of the user $u$ contains texts and emoticons

  * •

The neighborhood structure around the user $u$ in $G$

Therefore, we deal with two types of features: Textual features and Network
features.

Textual features model the textual content of the post. We used a Bag of Words
(BoW) model with unigrams, bigrams and trigrams occurring in the post. We
apply lemmatization to group together the different inflected forms of a word.
Thus each lemma and each sequence of two and three adjacent lemmas are
considered as features. The generated vector is denoted by $T(u)$.

Network features model neighborhood patterns around users in a social graph.
Let $G=(V,E)$ be an unweighted graph where the user $u$ and his friends have
joined. Graph embedding techniques create vector embedding $\phi(u)\in R^{d}$
for every node $u\in V$ such that $d<<|V|$ . In this work we apply node2vec
[4], HARP [5] and Poincarè [6] to learn node embeddings.

We propose to use a feedforward neural network to build a binary classifier.
Formally, we define $F$ as function,

$F:T(u)\oplus\phi(u)\mapsto\\{0,1\\}$

that maps concatenation ($\oplus$) of two feature vectors into a binary value
of $1$ for attendance or $0$ for non-attendance.

Our feedforward network consists of an input layer, a hidden layer, and an
output layer. The size of the input layer depends on the size of vector
embeddings and tweets vocabulary. We set the rectified unit (ReLU) [13] as
activation function of the input and hidden layer. ReLU does not face gradient
vanishing problem and it has been shown that deep networks are trained
efficiently. The number of neurons in the hidden layer is the mean of the
neurons in the input and output layers [14]. Since the network performs a
binary classification task, the output layer is a single unit of Sigmoid [15].
As optimizer, we use Adam [16] which is robust and well-suited to a wide range
of non-convex optimization problems for large-scale learning.

##  IV Experimental Evaluation

(a) VFestival (b) Creamfields

Figure 1: Social groups attended to the events

We conduct our experiments on Twitter datasets related to two large music
festivals in the UK named Creamfields and VFestival [7]. The original dataset
splits posts based on their timestamp into three different disjoint sets:
posts made before, during or after the event. In each timestamp, $460$
distinct tweets are sampled and labeled. However, we crawled fewer distinct
tweets for each timestamp since users deleted their Twitter account after a
year. Obviously, we do not have enough training samples if we only consider
tweets before the event. Therefore, we merge the tweets before and during the
events to have adequate training samples. Table I reports the total number of
labeled tweets and the respective percentage of positive (attended) and
negative (not-attended) labeled classes for each dataset.

(a) HARP (b) HARP (c) Poincaré

Figure 2: Parameter sensitivity for classification Table I: Dataset Statistics  Dataset | Timestamp | Labled | pos$\%$ |  neg$\%$
---|---|---|---|---
VFestival | before | 320 | 44 | 56
during | 329 | 47 | 53
overall | 649 | 46 | 54
Creamfields | before | 330 | 46 | 54
during | 341 | 42 | 58
overall | 671 | 44 | 56

To study people’s social influence, we use the crawled Twitter graph consists
of users and links. Given the list of attended users in the original dataset,
we check whether there is an edge between any pair. We thus construct an edge
list of attended users in each dataset. There are multiple users participate
in the events alone (without direct neighbors). We need to organize alone
users into several artificial groups in order to examine friends’ influence.
Therefore, we first divide users into 4 individual groups and then randomly
link members together. Overall, $303$ users in $14$ groups attended to
VFestival and $298$ users participated in Creamfields. Fig. 1 depicts all
social groups joined to VFestival and Creamfields. Finally, we apply node2vec,
HARP and Poincarè to encode neighborhood structures of the social groups in
the form of vectors.

Table II: Classification performance applying different graph embedding techniques. Dataset | Classifier | Accuracy | Precision | Recall | F1
---|---|---|---|---|---
VFestival | GBDT | 0.805 | 0.826 | 0.675 | 0.736
NNT$\oplus$HARP | 0.892 | 0.886 | 0.784 |  0.821
NNT$\oplus$N2V | 0.823 | 0.792 | 0.745 | 0.768
NNT$\oplus$Poincarè | 0.785 | 0.735 | 0.706 | 0.720
NNT | 0.782 | 0.756 | 0.667 | 0.708
NNHARP | 0.769 | 0.691 | 0.745 | 0.717
Creamfields | GBDT | 0.865 | 0.838 | 0.764 | 0.793
NNT$\oplus$HARP | 0.881 | 0.857 | 0.774 |  0.814
NNT$\oplus$N2V | 0.803 | 0.786 | 0.758 | 0.752
NNT$\oplus$Poincarè | 0.779 | 0.735 | 0.689 | 0.712
NNT | 0.777 | 0.729 | 0.686 | 0.707
NNHARP | 0.710 | 0.732 | 0.661 | 0.695

###  IV-A Parameters and Environment

In this section, we empirically study parameter sensitivity of the embedding
techniques. We examine how the different choices of parameters affect the
performance where the input of the classifier is concatenation of textual and
network features. In Fig. 2, the accuracy is measured as a function of
parameters embedding size $d$ and context size $k$. It can be seen that the
performance varies with embedding and context size. In HARP settings, $d=128$
and $k=4$ gives the best performance.

Since HARP applies a hierarchical paradigm for graph embedding based on
iterative learning methods (e.g. node2vec), we follow the same tuning
procedure for node2vec. We also consider $p=q=1$ to explore local and global
neighborhood equally. For Poincaré, we determine the radius $r>1$ and the
steepness $t=0.01$ [6]. As shown in Fig. 2, $d=128$ is the best embedding size
for Poincaré as well.

###  IV-B Results and Discussions

Our experiments are conducted using a 4-fold cross validation, while
preserving the proportion of positive and negative instances in each fold. The
quality of predictor is assessed in terms of accuracy, precision, recall and
F1 score. We compare our results to the most recent work [7] which achieved
the best accuracy using Boosting Decision Trees (GBDT). To make results
comparable, we take the average of accuracies before and during the events
available in [7]. Depending on input features to the Neural Network (NN), we
have the following classifiers:

  * •

NNT$\oplus$HARP: Input of tweets textual features and HARP embeddings

  * •

NNT$\oplus$N2V: Input of tweets textual features and node2vec embeddings

  * •

NNT$\oplus$Poincarè: Input of tweets textual features and Poincarè embeddings

  * •

NNT: Input of tweets textual features

  * •

NNHARP: Input of HARP embeddings

The obtained accuracies for each dataset are provided in the Table II. The
results indicate that considering the social influence of friends improves the
accuracy. We can see that HARP embeddings generally shows the better
performance compared to the others. Feeding the network only with HARP
embedding which benefits from graph coarsening achieves accuracy ranging from
$71\%$ to $76\%$ in two datasets. It demonstrates the impact of the local
neighborhood structure and friends’ influences on the prediction task.
Poincarè performs worse than two other embedding techniques. The loss function
of Poincarè causes most points to map on the border of the Poincare ball [17].
It may cause Poincarè shows different performances for preserving local
properties [18] which are needed in our prediction task.

##  V Conclusion

In this paper, we proposed an approach to predict event attendance by mining
users media posts and social groups influence. A key detail of our proposed
approach is to exploit user’s neighborhood pattern in the social graph to
improve the prediction task. We trained a neural network classifier using
tweets and the social graph embeddings related to two large music festivals in
the UK. We evaluated the performance by accuracy and precision in comparison
to the recent work. The results show how our approach performs consistently
better than the baseline, exhibiting $89\%$ accuracy at classifying attending
users.

## Acknowledgment

The presented work was developed within the Provenance Analytics project
funded by the German Federal Ministry of Education and Research, grant
agreement number 03PSIPT5C.

## References

  * [1] K. Leetaru, S. Wang, G. Cao, A. Padmanabhan, and E. Shook, “Mapping the global twitter heartbeat: The geography of twitter,” _First Monday_ , vol. 18, no. 5, 2013.
  * [2] S. Zhang and Q. Lv, “Event organization 101: Understanding latent factors of event popularity,” _arXiv preprint arXiv:1709.02024_ , 2017.
  * [3] R. Du, Z. Yu, T. Mei, Z. Wang, Z. Wang, and B. Guo, “Predicting activity attendance in event-based social networks: content, context and social influence,” in _Proceedings of the 2014 ACM international joint conference on pervasive and ubiquitous computing_. ACM, 2014, pp. 425–434.
  * [4] A. Grover and J. Leskovec, “node2vec: Scalable feature learning for networks,” in _Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining_. ACM, 2016, pp. 855–864.
  * [5] H. Chen, B. Perozzi, Y. Hu, and S. Skiena, “Harp: Hierarchical representation learning for networks,” _arXiv preprint arXiv:1706.07845_ , 2017.
  * [6] M. Nickel and D. Kiela, “Poincar$\backslash$’e embeddings for learning hierarchical representations,” _arXiv preprint arXiv:1705.08039_ , 2017.
  * [7] V. M. de Lira, C. Macdonald, I. Ounis, R. Perego, C. Renso, and V. C. Times, “Exploring social media for event attendance,” in _Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017_. ACM, 2017, pp. 447–450.
  * [8] A. Magnuson, V. Dialani, and D. Mallela, “Event recommendation using twitter activity,” in _Proceedings of the 9th ACM Conference on Recommender Systems_. ACM, 2015, pp. 331–332.
  * [9] F. Botta, H. S. Moat, and T. Preis, “Quantifying crowd size with mobile phone and twitter data,” _Royal Society open science_ , vol. 2, no. 5, p. 150162, 2015.
  * [10] X. Zhang, J. Zhao, and G. Cao, “Who will attend?–predicting event attendance in event-based social network,” in _Mobile Data Management (MDM), 2015 16th IEEE International Conference on_ , vol. 1. IEEE, 2015, pp. 74–83.
  * [11] X. Liu, Q. He, Y. Tian, W.-C. Lee, J. McPherson, and J. Han, “Event-based social networks: linking the online and offline social worlds,” in _Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining_. ACM, 2012, pp. 1032–1040.
  * [12] E. Ravasz and A.-L. Barabási, “Hierarchical organization in complex networks,” _Physical Review E_ , vol. 67, no. 2, p. 026112, 2003.
  * [13] V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in _Proceedings of the 27th international conference on machine learning (ICML-10)_ , 2010, pp. 807–814.
  * [14] J. Heaton, _Introduction to neural networks with Java_. Heaton Research, Inc., 2008.
  * [15] B. Karlik and A. V. Olgac, “Performance analysis of various activation functions in generalized mlp architectures of neural networks,” _International Journal of Artificial Intelligence and Expert Systems_ , vol. 1, no. 4, pp. 111–122, 2011.
  * [16] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [17] O.-E. Ganea, G. Bécigneul, and T. Hofmann, “Hyperbolic entailment cones for learning hierarchical embeddings,” _arXiv preprint arXiv:1804.01882_ , 2018.
  * [18] J. B. G. T. A. S. M. B. O. Stephen Bonner, Ibad Kureshi, “Exploring the semantic content of unsupervised graph embeddings: An empirical study,” _arXiv preprint arXiv:1806.07464_ , 2018.


11institutetext: Department of Computer Science, Norwegian University of
Science and Technology (NTNU), Trondheim, Norway 22institutetext: Department
of Health Research, Medical Technology, SINTEF, Trondheim, Norway
33institutetext: Department of Clinical and Molecular Medicine, Norwegian
University of Science and Technology (NTNU), Trondheim, Norway
44institutetext: Clinic of Surgery, St. Olavs hospital, Trondheim University
Hospital, Trondheim, Norway 55institutetext: Research Department, Future
Operating Room, St. Olavs hospital, Trondheim University Hospital, Trondheim,
Norway

# Teacher-Student Architecture for Mixed Supervised Lung Tumor Segmentation

Vemund Fredriksen 11‡‡ Svein Ole M. Sevle 11‡‡ André Pedersen 223344** Thomas
Langø 2255 Gabriel Kiss 1155 Frank Lindseth 11

(August 2021)

###### Abstract

Purpose Automating tasks such as lung tumor localization and segmentation in
radiological images can free valuable time for radiologists and other clinical
personnel. Convolutional neural networks may be suited for such tasks, but
require substantial amounts of labeled data to train. Obtaining labeled data
is a challenge, especially in the medical domain. ††Preprint submitted
December 21, 2021 ††‡ These authors contributed equally to this work. ††*
Corresponding author<EMAIL_ADDRESS>
Methods This paper investigates the use of a teacher-student design to utilize
datasets with different types of supervision to train an automatic model
performing pulmonary tumor segmentation on computed tomography images. The
framework consists of two models: the student that performs end-to-end
automatic tumor segmentation and the teacher that supplies the student
additional pseudo-annotated data during training.
Results Using only a small proportion of semantically labeled data and a large
number of bounding box annotated data, we achieved competitive performance
using a teacher-student design. Models trained on larger amounts of semantic
annotations did not perform better than those trained on teacher-annotated
data.
Conclusions Our results demonstrate the potential of utilizing teacher-student
designs to reduce the annotation load, as less supervised annotation schemes
may be performed, without any real degradation in segmentation accuracy.

###### Keywords:

Teacher-Student Framework Pulmonary Tumor Segmentation Deep Learning
Convolutional Neural Networks Computed Tomography

##  1 Introduction

Cancer is becoming the leading cause of death and the most significant
obstacle to increase life expectancy in many countries [31]. Lung cancer,
accounting for more than 11 % of all new cases, is the second most common
cancer and it ranks first among the cancer-related mortality worldwide,
accounting for 18 % of the total cancer deaths [41]. The most common lung
cancer treatments include: surgical resection, chemotherapy, radiotherapy, and
immunotherapy. Many of these treatments, and also the successful diagnosis
with bronchoscopy or computed tomography (CT)-guided biopsy, depend on
accurately locating, and in many cases delineating (segmenting), the tumor
from normal tissue in the preoperative images, typically CT.

Manual segmentation of the lesions/tumors from preoperative CT is a laborious
and tedious process for oncologists, radiologists, and pulmonologists, which
could result in delays of treatment and lower the survival rates, especially
in clinics with inadequate resources. In addition, the quality of manual
localization and segmentation relies on prior knowledge and clinical
expertise. Even with adequate guidelines and standards, tumor segmentation is
often prone to high inter- and intraobserver variability. On the other hand,
automatic segmentation techniques has the potential to provide efficient,
consistent, and more accurate results. Automatic methods can both shorten the
time needed to read the images and they also allow experts to devote their
limited time to optimize planning and treatment planning.

Historically, methods like thresholding [42], region growing [1, 8], and graph
cuts [13] were commonly proposed to segment lung tumors from CT images. These
algorithms are suitable as semi-automatic methods, but are not suited for
localization of lung tumors. Recent advancements in deep learning enables
automation of tasks that until recently was only performed by trained experts
[35, 24]. Advancements in hardware has enabled development of larger and
increasingly complex models, but much of the improvement is caused by access
to large amounts of annotated data.

Today, especially after the introduction of the U-Net [36] architecture, deep
learning methods have dominated the field of medical image segmentation [9].
However, convolutional neural networks (CNNs) are memory intensive, especially
for 3D volumes. It is therefore common to train networks based on 2D or 2.5D
input images [7], where the model evaluates one slice at a time, chunks of
slices, or 3D patches, and then applied in a sliding windows fashion across
the CT volume. Patching the 3D volume comes at a cost of loss of perception,
and thus more efficient multi-scale 3D CNN architectures have been proposed,
which enables the use of larger input volumes [20]. An alternative approach is
to perform segmentation in multiple steps, either using multiple algorithms or
a cascade of CNNs [18, 19, 7, 12].

To accommodate the issue of lacking training data, unsupervised methods like
supervoxel has been proposed [14]. To facilitate faster convergence and more
accurate results, multi-modality methods that utilize magnetic resonance
imaging (MRI) or positron emission tomography (PET) scans in addition to CT
have been suggested [10, 21, 22]. Neural network architectures that can
utilize multiple annotation types has also been suggested [43, 28].

A more recent strategy to accommodate the lack of training data is the
teacher-student design, inspired by the concept of knowledge distillation [34,
11, 16]. The teacher creates pseudo-annotations from suboptimal annotations to
increase the dataset size for training the student. The teacher-student
pattern can be applied to any type of network architecture, and does not
dictate other hyperparameters or external configurations. A teacher-student
design can be used in different ways, from utilization of unlabeled data [6,
44, 26], to exploitation of multiple modalities [21, 22, 26], and to usage of
datasets with different annotation types [40, 45].

Our approach differs from the previously mentioned methods applied to lung
tumor segmentation by utilizing annotations of different supervision on the
same modality, namely CT images. Since CT scanning is less invasive to the
patient than the other modalities, it is a goal to efficiently segment tumors
from CT-only examinations. Our method is inspired by Sun et al. [40] that
shows promising results using a teacher-student framework to segment liver and
liver lesions given semantic and bounding box annotations. To the best of our
knowledge, we are the first to implement a similar teacher-student framework
for CT images to perform semantic segmentation of lung tumors. Our study
suggests that even with a small dataset of semantic annotations, a student can
achieve state-of-the-art performance given a large enough pseudo-annotated
dataset to learn from.

##  2 Materials and methods

Our method consists of two separate models: a semi-supervised teacher and a
fully-automatic student. The method relies on two different annotation types:
semantic 3D annotations and 2D bounding boxes in the axial planes. These we
refer to as strong and weak annotations. Furthermore, we define our strongly
and weakly annotated datasets as
$D_{s}={\\{(X_{i}^{s},y_{i}^{s})\\}}_{i=1}^{m_{s}}$ and
$D_{w}={\\{(X_{i}^{w},y_{i}^{w})\\}}_{i=1}^{m_{w}}$, respectively. An overview
of our design can be seen in Fig. 1.

###  2.1 Data

To study the effect of our teacher-student framework we used three public
datasets: Medical Segmentation Decathlon (MSD)-Lung [37], Non-Small Cell Lung
Cancer (NSCLC)-Radiomics [2, 3], and Lung-PET-CT-Dx [27]. All three datasets
contain manual annotations by human experts. The first two datasets consist of
semantic annotations, whereas the latter dataset contains bounding box labels
annotated in the axial plane.

The MSD-Lung dataset contains 64 images, whereas the NSCLC-Radiomics and Lung-
PET-CT-Dx datasets contains 422 and 1295 images, respectively. Multiple images
in the Lung-PET-CT-Dx dataset were discarded. The discarded images were either
PET or PET/CT-fused, only contained a small portion of the thorax, or
comprised of multiple scans stacked on top of each other. After removing all
non-CT images and images with a real-world length (Z-axis) outside the range
$[16,60]$ cm, 665 images from Lung-PET-CT-Dx remained in our dataset.

Table 1: Tumor sizes of the three datasets Dataset | Volume [$\boldsymbol{cm^{3}}]$ | Diameter [$\boldsymbol{mm}]$
---|---|---
MSD-Lung | $21.98\pm 51.66$ | $37.63\pm 20.08$
NSCLC-Radiomics | $75.37\pm 96.30$ | $63.63\pm 29.62$
Lung-PET-CT-Dx | $63.67\pm 86.26$ | $48.66\pm 19.85$

The three datasets varied in terms of voxel density and tumor sizes. Overall,
the Lung-PET-CT-Dx and the NSCLC-Radiomics datasets contain larger tumors than
the MSD-Lung dataset (see Table 1). The tumor diameter is an approximate size,
measured by calculating the average of the longest and shortest diameter of
the tumor in real-world coordinate space.

Figure 1: The method overview during training. Firstly, the teacher was
trained using the strong dataset (semantic annotated images) represented with
purple lines. The teacher was then used to make semantic annotations for the
weak dataset (bounding box annotated images). The student was then trained on
both the pseudo-annotated dataset $D_{w^{\prime}}$, represented by the orange
lines, and the strong dataset $D_{s}$. We trained two different students, only
one of them were backpropagated using the box annotations, indicated by the
dashed line.

###  2.2 Preprocessing

Our preprocessing pipeline consisted of multiple steps. Firstly, the voxel
intensities were clipped to the range [-1024, 1000], before being standardized
using the Z-score normalization method. The images’ voxel spacing were then
normalized to an anisotropic resolution of $1\times 1\times 1.5$ mm3. Lastly,
a volumetric cropping was applied, which differed between the teacher and the
student.

For the teacher, the images were cropped around the tumor with a fixed
resolution of $128\times 128\times 128$ voxels, whereas for the student, the
images were split in two, each cropped around one of the lungs. The lungs were
automatically segmented using the lungmask command line tool [17], and used
when performing cropping around the lungs. The ground truth label images were
voxel normalized and cropped in a similar manner as their corresponding CT
image.

###  2.3 Teacher-Student design

The teacher was trained on 3D patches surrounding the tumor, guided by the
corresponding bounding box annotations. Once trained, the teacher was applied
to $D_{w}$ to generate pseudo-strong labels, $D_{w^{\prime}}$. Although expert
labeled images are the gold standard, teacher pseudo-annotated images can
enhance training of fully automatic models, or even be used to aid experts in
clinical use.

The student, like any ordinary automatic method, takes CT images as input and
produces 3D segmentations of the potential lung tumors without user
interaction. During training, the student exploits the pseudo-annotated images
in $D_{w^{\prime}}$ produced by the teacher, using the extended dataset,
$\\{D_{s},D_{w^{\prime}}\\}$. Once trained, the student can perform end-to-end
segmentation without human intervention.

###  2.4 Implementation

All our networks are based on the U-Net architecture [36], and share common
building blocks (see Fig. 2). U-Net was used as it performs well as a baseline
architecture, and has shown competitive performance on various datasets from
different modalities, of different organs, cancer types, and data types [19,
33].

The teacher consists of three levels, one of each downsampling operation,
going from an image resolution of $128\times 128\times 128$ to $16\times
16\times 16$. The students are comprised of four levels. In contrast to the
U-Net architecture, our design performs downsampling by applying 3D
convolutions with a stride of two. We also substituted the ReLU [30]
activation function with PReLU [15]. We implemented two related students: one
that produces semantic segmentation output only, which we call the Single
Output Student (SO Student), and one that produces an additional output
approximating the bounding box surrounding the tumor in the axial plane, which
we call the Dual Output Student (DO Student). The architecture of the student
networks can be seen in Fig. 2.

Figure 2: The network architectures. The single output (SO) Student is
highlighted in colors whereas the decoder branch of the dual output (DO)
Student is implicated in gray. For the DO Student the ouput of the two decoder
branches are concatenated to form a dual-channeled output. The teachers have
the same architecture as the SO Student, but with three downsamplings rather
than four.

The Adam [25] optimizer with a learning rate of $10^{-4}$ was used for
training until DSC validation convergence. The batch size was set to one and
virtually increased to eight using accumulated gradients. The gradients were
computed using the Dice Loss function [39], based on the Dice Similarity
Coefficient (DSC).

###  2.5 Empirical Evaluation

To evaluate our framework we considered two primary scenarios, each with two
sub-experiments. We considered one scenario where the size of the strongly
annotated dataset ($\sim 500$ images) is similar to the size of the weakly
annotated dataset ($\sim 750$ images), and another scenario where the strongly
annotated dataset was considerably smaller ($\sim 50$ images) than the weakly
annotated dataset($\sim 1000$ images). Within each scenario, we evaluated two
semi-supervised models and three fully-automatic models. Among the three
fully-automatic models, one model was trained solely on strongly annotated
data, whereas the two other were student networks trained both on strongly
annotated data and the teacher-annotated pseudo labels.

We used different metrics for evaluating and comparing the models. The DSC was
used to measure the semantic segmentation performance, whereas F1-score was
used to determine object-wise localization performance. We also used DSC-TP to
evaluate the segmentation accuracy considering only true positives (TPs). We
considered objects to be true positives if there were $\geq 25\%$ overlap
between the predicted mask and the GT mask, motivated by a prior study [4].

The test set was sampled at random and accounted for $15\%$ of the total
dataset. The same split was used for all experiments to preserve fairness in
evaluation. Patients with multiple scans were stratified into the three
subsets: train, validation, and test. To counter the tumor size imbalance, we
balanced the train and validation sets with regard to tumor sizes. Images
containing tumors of more rare sizes were upsampled.

Models were trained using a workstation with a 14-core Intel Core i9 10940X
@3.30 GHz CPU, 128 GB RAM, and two NVIDIA RTX 8000 (48 GB) GPUs. The most
memory intensive student used, at its peak, $\sim 22.54$GB VRAM during
training, but inference can be performed with 3GB VRAM. Implementation was
done in Python 3.7, built upon the MONAI [29] framework (v0.4.0), using
PyTorch v1.6, and CUDA 11.0. The best performing model and corresponding
inference code are made openly available on GitHub as a command line
tool111https://github.com/VemundFredriksen/LungTumorMask.

##  3 Results

###  3.1 Vast Strongly Annotated Dataset

As seen in Table 2, the teacher guided by the bounding boxes, outperformed the
point guided (without bounding boxes as input) teacher on both datasets in
terms of DSC. The difference between the two models was less prominent
measured on the MSD-Lung dataset than for the NSCLC-Radiomics dataset.

Table 2: Teacher Results. The best performing model with respect to mean dice similarity coefficient (DSC) is highlighted in bold. Dataset |  |  Model |  DSC
---|---
MSD-Lung |  |  Point Guided |  $74.78\pm 11.83$
---|---
Box Guided |  $\mathbf{84.91\pm 06.09}$
NSCLC-Radiomics |  |  Point Guided |  $59.57\pm 23.90$
---|---
Box Guided |  $\mathbf{86.65\pm 08.77}$
Both |  |  Point Guided |  $61.48\pm 23.29$
---|---
Box Guided |  $\mathbf{86.44\pm 08.50}$

For the final inference models, the DSC was highest on the MSD-Lung dataset,
across all three models (see Table 3). The best performing student network
overall was the SO Student, with highest DSC on the MSD-Lung dataset. There
was negligible difference between the three models on the NSCLC-Radiomics
dataset. The Baseline model performed best on the MSD-Lung dataset, both in
terms of DSC and F1-score.

Table 3: Student Results. For each respective metric, the best performing
models are highlighted in bold.

Dataset Model DSC DSC-TP F1-score Recall Precision MSD-Lung Baseline
$\mathbf{67.31\pm 21.17}$ $\mathbf{73.12\pm 15.16}$ $\mathbf{81.48\pm 31.86}$
$\mathbf{88.89\pm 31.43}$ $\mathbf{77.78\pm 34.25}$ SO Student $64.27\pm
16.05$ $71.32\pm 8.06$ $\mathbf{81.48\pm 31.86}$ $\mathbf{88.89\pm 31.43}$
$\mathbf{77.78\pm 34.25}$ DO Student $55.37\pm 29.03$ $70.49\pm 8.82$
$74.07\pm 40.91$ $77.78\pm 41.57$ $72.22\pm 41.57$ NSCLC-Radiomics Baseline
$51.06\pm 28.22$ $68.81\pm 18.27$ $63.56\pm 36.36$ $\mathbf{83.82\pm 36.82}$
$56.68\pm 38.65$ SO Student $\mathbf{52.92\pm 31.13}$ $\mathbf{69.39\pm
19.21}$ $64.18\pm 37.37$ $79.90\pm 39.66$ $58.76\pm 39.28$ DO Student
$52.25\pm 30.18$ $68.69\pm 19.47$ $\mathbf{68.43\pm 38.71}$ $79.17\pm 39.75$
$\mathbf{64.95\pm 40.81}$ Both Baseline $52.96\pm 27.98$ $69.34\pm 17.97$
$65.66\pm 36.32$ $\mathbf{84.41\pm 36.27}$ $59.14\pm 38.76$ SO Student
$\mathbf{54.25\pm 29.99}$ $\mathbf{69.63\pm 18.19}$ $66.20\pm 37.19$ $80.95\pm
38.90$ $60.98\pm 39.21$ DO Student $52.61\pm 30.06$ $68.90\pm 18.59$
$\mathbf{69.09\pm 39.02}$ $79.00\pm 39.97$ $\mathbf{65.80\pm 40.97}$

###  3.2 Scarce Strongly Annotated Dataset

When reducing the strongly labeled dataset, the performance of the point
guided teacher was degraded, whereas the box guided teacher still performed
well.

Table 4: Scarcely Trained Teacher Results. The best performing model with respect to mean dice similarity coefficient (DSC) is highlighted in bold. Dataset |  |  Model |  DSC
---|---
MSD-Lung |  |  Scarce Point Guided |  $48.52\pm 31.18$
---|---
Scarce Box Guided |  $\mathbf{81.65\pm 07.40}$
NSCLC-Radiomics |  |  Scarce Point Guided |  $43.83\pm 25.65$
---|---
Scarce Box Guided |  $\mathbf{84.69\pm 06.59}$
Both |  |  Scarce Point Guided |  $44.42\pm 26.45$
---|---
Scarce Box Guided |  $\mathbf{84.31\pm 06.77}$

A similar trend applies to the final inference models (see Table 5). The
baseline model performed poorer, whereas the student networks still performed
well. The same can be seen from the object-wise metrics, although the
difference was more prominent. Contrary to the results shown in Table 3, the
SO Student had the highest DSC measured in this scenario. Fig. 3 shows a
sample of the outputs produced by the models in the scarce scenario.

Table 5: Scarcely Trained Student Results. For each respective metric, the
best performing model is highlighted in bold.

Dataset Model DSC DSC-TP F1-score Recall Precision MSD-Lung Baseline $26.45\pm
26.56$ $75.24\pm 15.90$ $09.10\pm 13.26$ $33.33\pm 47.14$ $05.31\pm 07.80$ SO
Student $64.74\pm 11.82$ $71.56\pm 10.40$ $61.85\pm 16.49$ $\mathbf{100.0\pm
0.00}$ $47.22\pm 20.79$ DO Student $\mathbf{71.00\pm 16.01}$ $\mathbf{76.67\pm
07.08}$ $\mathbf{85.18\pm 31.86}$ $88.89\pm 31.43$ $\mathbf{83.33\pm 33.33}$
NSCLC-Radiomics Baseline $28.23\pm 28.05$ $55.39\pm 22.80$ $32.13\pm 36.99$
$51.47\pm 49.24$ $26.99\pm 35.68$ SO Student $51.06\pm 30.75$
$\mathbf{68.56\pm 20.69}$ $62.65\pm 36.73$ $79.41\pm 39.51$ $56.67\pm 38.79$
DO Student $\mathbf{53.89\pm 29.75}$ $67.44\pm 21.70$ $\mathbf{66.96\pm
35.57}$ $\mathbf{84.56\pm 35.62}$ $\mathbf{60.44\pm 38.24}$ Both Baseline
$28.02\pm 27.89$ $56.91\pm 22.96$ $29.44\pm 35.82$ $49.35\pm 49.34$ $24.45\pm
34.35$ SO Student $52.66\pm 29.51$ $\mathbf{68.98\pm 19.60}$ $62.55\pm 34.98$
$81.82\pm 37.72$ $55.56\pm 37.26$ DO Student $\mathbf{55.89\pm 29.01}$
$68.56\pm 20.71$ $\mathbf{69.09\pm 35.64}$ $\mathbf{85.06\pm 35.18}$
$\mathbf{63.12\pm 38.41}$

##  4 Discussion

In this paper, a teacher-student design to segment lung tumors from CTs has
been proposed. Three datasets of two different annotation types were used for
this purpose. The teacher model was first trained on the datasets that had
strong annotations. It was then used to generate pseudo-strong annotations for
the student. Both the teacher and the student used U-Net-like architectures,
and were evaluated on segmentation performance. In addition, the student
networks were evaluated on sensitivity to annotation type and sample size.

We observed that the box guided teacher outperformed the point guided teacher
in both scenarios. This was expected as the bounding box annotations assist
the teacher by serving as a segmentation and localization constraint. The
effect of the box guidance is especially visible in the scarce scenario, where
the box guided teacher achieved almost double the DSC as the point guided one.
The scarce box guided teacher also outperformed the vast point guided teacher.
This suggests that training a teacher on a smaller set of bounding box
annotated images can be advantageous compared to training a teacher on a large
set of point guided images.

Surprisingly, the students did not perform better than the baseline in the
scenario with vast strongly annotated data (see Table 3). Measured on the MSD-
Lung dataset, the baseline model outperformed the two students, whereas the
opposite was observed for the NSCLC-Radiomics dataset. A potential explanation
might be that the Lung-PET-CT-Dx dataset contains tumors with sizes more
similar to the NSCLC-Radiomics dataset than to the tumors in MSD-Lung. The
introduction of the Lung-PET-CT-Dx dataset may have led the students to
perform better on larger tumors, but may have degraded the results on smaller
tumors typically found in MSD-Lung. Another explanation might be that the
ratio between strong and weak labels were not large enough to make a
noticeable difference. This was further demonstrated when the models were
evaluated in the scarce scenario (see Table 5).

In this scenario, the students significantly outperformed the baseline
supervised model. This demonstrates that the introduction of suboptimal
annotations into the teacher-student design can improve performance of an end-
to-end segmentation model. We observed a DSC comparable with state-of-the-art
performance measured on the MSD-Lung dataset, with a F1-score of $85.18$, and
a DSC of $71.00$, for one of our students. Isensee et al. [19] reported a DSC
of $69.2$ on the MSD-Lung dataset, whereas Carvalho et al. [7] reported a DSC
of $70.9$. Our model trained on only $40$ human annotated images scored
marginally better, although on a different test set. The results suggest that
$40$ images is not enough to train a supervised model, but enough to train a
semi-supervised model that can enhance a supervised model by increasing the
available data in a cheaper way than manual delineation.

Figure 3: A sample of the results produced by the scarce students on the test
set. The figure shows the input image, bounding box, and ground truth (GT)
mask in the three top rows, respectively. The baseline model, single output
(SO) Student, and dual output (DO) Students corresponding outputs are shown in
the three bottom rows, respectively.

The datasets were of varying quality. The MSD-Lung dataset was of high
standard, whereas the NSCLC-Radiomics dataset was less so. Other publications
that used the NSCLC-Radiomics dataset reported heavy data sanitation,
effectively removing large parts of the dataset [32, 23]. We did not override
the expert’s annotations, as we also seek to handle suboptimal annotations, if
these should be present in a data set. The flawed dataset explains why the
difference between the box guided and point guided teacher is larger on the
NSCLC-Radiomics dataset than MSD-Lung. Images where the tumor is poorly, or
even completely wrongly annotated, the box guided teacher can rely on the
bounding boxes to achieve a good DSC, but since the annotation itself is
wrong, the point guided teacher struggles.

###  4.1 Limitations

One of the major limitations in this experiment was the scarce amount of data.
The test set was sampled randomly from each dataset. It is plausible that a
different sample of the test set would have given different result. Although
K-fold cross validation could be used to eliminate this concern, it was
dropped due to time limitations. K-fold cross validation is a time consuming
strategy. It depends on training K different models, which would take a
considerable amount of time, even with a small K in our situation. Since our
method is a two step method that relies on two training steps, the K-fold
cross validation would take nearly double the time of a similar single-step
method as well.

Another limitations of this experiments was that the students were sensitive
to voxel spacing. By reducing the voxel spacing during
normalization/preprocessing, thus increasing the resolution of the image, the
DSC did not improve, but actually degraded. Therefore, it is possible that the
proposed architecture is sensitive to small adjustments in the preprocessing
pipeline.

###  4.2 Future work

An alternative to using 2D bounding boxes in the axial plane, is to use a 3D
bounding box. As one 3D bounding box contains much fewer corners than multiple
2D bounding boxes, this could further reduce annotation load. It is reason to
believe that a teacher trained on 3D boxes will perform worse than one trained
on 2D axial boxes. However, if the reduction in annotation load is
significant, the amount of data that can be annotated for the teacher might
weigh up the loss in precision of the annotation. After all, this is the very
fundamental idea behind the teacher-student design. However, we feel that a
much larger dataset should be used to explore this properly.

The main motivation of using a teacher-student design is to improve models by
learning from additional suboptimal annotated or unannotated data. We observed
a benefit of using such a design for lung tumor segmentation in CTs. However,
a single-step teacher might not be sufficient. It has been proposed to train
both the teacher and student end-to-end in an iterative fashion [5]. This
makes sense as the teacher could improve from the student’s feedback.
Especially from multiple students, which iteratively could improve the
students as well, as the teacher become more experienced. However, for 3D
applications this is likely infeasible. Alternatively, one could potentially
use multiple teachers, trained on different types of images that focus on
different lung tumor types and sizes. Having specialized teachers to train the
student in an ensemble manner makes sense as it more closely represents the
natural teacher-student relation from academia.

##  5 Conclusion

We present the first known implementation of a mixed-supervised teacher-
student framework for lung tumor segmentation from CT images. Our method
utilized both semantic and axial bounding box annotations to maximize lung
tumor segmentation performance. We demonstrated that with sufficient bounding
box annotated data, our teacher-student framework achieved state-of-the-art
performance, even with scarce semantic annotated data.

### Acknowledgements

This work was conducted at the machines Idun [38], Malvik222A cluster node
owned and maintained by NTNU AI Lab, and Bohaga333A machine owned by NTNU,
maintained by Gabriel Kiss.

### Author contributions

VF and SS were involved in code writing; AP and FL supervised the
implementation; Computational resources were provided by GK and FL; FL and TL
contributed to funding acquisition; TL and AP provided domain-level expertise;
All authors contributed to writing-original draft preparation and to writing-
review and editing. All authors read and approved the final manuscript.

### Funding

The project was supported by The Norwegian National Advisory Unit for
Ultrasound and Image-Guided therapy at St. Olavs hospital; The Liaison
Committee for Education, Research and Innovation in Central Norway [grant
number 2018/42794]; The Cancer Foundation, St. Olavs hospital, Trondheim
University Hospital [grant number 13/2021]; SINTEF; The EEA Grant project
”Improving Cancer Diagnostics in Flexible Endoscopy Using Artificial
Intelligence and Medical Robotics (IDEAR)” [grant number 19/2020]; Open Access
funding was provided by the Norwegian University of Science and Technology,
Trondheim, Norway.

## Declaration

### Conflicts of interest

The authors have nothing to disclose.

### Data availability

Only open datasets were used to conduct the experiments, and are therefore
available to reproduce the experiments.

### Code availability

A command line tool for end-to-end lung tumor segmentation of our best
performing model is made openly available on GitHub:
https://github.com/VemundFredriksen/LungTumorMask.

## References

## References

  * [1] Adams, R., Bischof, L.: Seeded Region Growing. IEEE Transactions on Pattern Analysis and Machine Intelligence 16(6), 641–647 (1994)
  * [2] Aerts, H., Rios Velazquez, E., Leijenaar, R., Parmar, C., Grossmann, P., Cavalho, S., Bussink, J., Monshouwer, R., Haibe-Kains, B., Rietveld, D., Hoebers, F., Rietbergen, M., Leemans, C., Dekker, A., Quackenbush, J., Gillies, R., Lambin, P.: Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nature communications 5, 4006 (08 2014)
  * [3] Aerts, H.J.W.L., Wee, L., Rios Velazquez, E., Leijenaar, R., Parmar, C., Grossmann, P., …, Lambin, P.: Data From NSCLC-Radiomics [Data set]. The Cancer Imaging Archive. https://wiki.cancerimagingarchive.net/display/Public/NSCLC-Radiomics (2019)
  * [4] Bouget, D., Pedersen, A., Vanel, J., Leira, H.O., Langø, T.: Mediastinal lymph nodes segmentation using 3D convolutional neural network ensembles and anatomical priors guiding (2021)
  * [5] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging Properties in Self-Supervised Vision Transformers. In: Proceedings of the International Conference on Computer Vision (ICCV) (2021)
  * [6] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging Properties in Self-Supervised Vision Transformers (2021)
  * [7] Carvalho, J., Moreira, J., Figueiredo, M., Papanikolaou, N.: Automatic Detection and Segmentation of Lung Lesions using Deep Residual CNNs. pp. 977–983 (10 2019)
  * [8] Dehmeshki, J., Amin, H., Valdivieso, M., Ye, X.: Segmentation of Pulmonary Nodules in Thoracic CT Scans: A Region Growing Approach. Medical Imaging, IEEE Transactions on 27, 467 – 480 (05 2008)
  * [9] Du, G., Cao, X., Liang, J., Chen, X., Zhan, Y.: Medical Image Segmentation based on U-Net: A Review. Journal of Imaging Science and Technology 64 (03 2020)
  * [10] Fu, X., Bi, L., Kumar, A., Fulham, M., Kim, J.: Multimodal Spatial Attention Module for Targeting Multimodal PET-CT Lung Tumor Segmentation. IEEE Journal of Biomedical and Health Informatics 25(9), 3507–3516 (2021)
  * [11] Furlanello, T., Lipton, Z., Tschannen, M., Itti, L., Anandkumar, A.: Born Again Neural Networks. In: Dy, J., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 80, pp. 1607–1616. PMLR (10–15 Jul 2018), https://proceedings.mlr.press/v80/furlanello18a.html
  * [12] Gan, W., Wang, H., Gu, H., Duan, Y., Shao, Y., Chen, H., Feng, A., Huang, Y., Fu, X., Ying, Y., et al.: Automatic segmentation of lung tumors on CT images based on a 2D & 3D hybrid convolutional neural network. The British Journal of Radiology 94, 20210038 (2021)
  * [13] Gu, Y., Kumar, V., Hall, L.O., Goldgof, D.B., Li, C.Y., Korn, R., Bendtsen, C., Velazquez, E.R., Dekker, A., Aerts, H., Lambin, P., Li, X., Tian, J., Gatenby, R.A., Gillies, R.J.: Automated delineation of lung tumors from CT images using a single click ensemble segmentation approach. Pattern Recognition 46(3), 692–702 (2013), https://www.sciencedirect.com/science/article/pii/S0031320312004384
  * [14] Hansen, S., Kuttner, S., Kampffmeyer, M., Markussen, T.V., Sundset, R., Øen, S.K., Eikenes, L., Jenssen, R.: Unsupervised supervoxel-based lung tumor segmentation across patient scans in hybrid PET/MRI. Expert Systems with Applications 167, 114244 (2021), https://www.sciencedirect.com/science/article/pii/S0957417420309623
  * [15] He, K., Zhang, X., Ren, S., Sun, J.: Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. In: 2015 IEEE International Conference on Computer Vision (ICCV). pp. 1026–1034 (2015)
  * [16] Hinton, G., Vinyals, O., Dean, J.: Distilling the Knowledge in a Neural Network (2015)
  * [17] Hofmanninger, J., Prayer, F., Pan, J., Röhrich, S., Prosch, H., Langs, G.: Automatic lung segmentation in routine imaging is primarily a data diversity problem, not a methodology problem. European Radiology Experimental 4, 50 (08 2020)
  * [18] Hossain, S., Najeeb, S., Shahriyar, A., Abdullah, Z.R., Ariful Haque, M.: A Pipeline for Lung Tumor Detection and Segmentation from CT Scans Using Dilated Convolutional Neural Networks. In: ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1348–1352 (2019)
  * [19] Isensee, F., Jaeger, P., Kohl, S., Petersen, J., Maier-Hein, K.: nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods 18, 1–9 (02 2021)
  * [20] Jiang, J., Hu, Y.C., Liu, C.J., Halpenny, D., Hellmann, M.D., Deasy, J.O., Mageras, G., Veeraraghavan, H.: Multiple Resolution Residually Connected Feature Streams for Automatic Lung Tumor Segmentation From CT Images. IEEE Transactions on Medical Imaging 38(1), 134–144 (2019)
  * [21] Jiang, J., Hu, Y., Tyagi, N., Zhang, P., Rimner, A., Deasy, J.O., Veeraraghavan, H.: Cross‐modality (CT‐MRI) prior augmented deep learning for robust lung tumor segmentation from small MR datasets. Medical Physics 46(10), 4392–4404 (Aug 2019), http://dx.doi.org/10.1002/mp.13695
  * [22] Jue, J., Jason, H., Neelam, T., Andreas, R., Sean, B.L., Joseph, D.O., Harini, V.: Integrating Cross-modality Hallucinated MRI with CT to Aid Mediastinal Lung Tumor Segmentation. In: Shen, D., Liu, T., Peters, T.M., Staib, L.H., Essert, C., Zhou, S., Yap, P.T., Khan, A. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2019. pp. 221–229. Springer International Publishing, Cham (2019)
  * [23] Kamal, U., Rafi, A.M., Hoque, R., Wu, J., Hasan, M.K.: Lung Cancer Tumor Region Segmentation Using Recurrent 3D-DenseUNet. In: Petersen, J., San José Estépar, R., Schmidt-Richberg, A., Gerard, S., Lassen-Schmidt, B., Jacobs, C., Beichel, R., Mori, K. (eds.) Thoracic Image Analysis. pp. 36–47. Springer International Publishing, Cham (2020)
  * [24] Kim, M., Yun, J., Cho, Y., Shin, K., Jang, R., Bae, H.j., Kim, N.: Deep learning in medical imaging. Neurospine 16(4), 657 (2019)
  * [25] Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization (2017)
  * [26] Li, K., Wang, S., Yu, L., Heng, P.A.: Dual-Teacher: Integrating Intra-domain and Inter-domain Teachers for Annotation-Efficient Cardiac Segmentation. In: Martel, A.L., Abolmaesumi, P., Stoyanov, D., Mateus, D., Zuluaga, M.A., Zhou, S.K., Racoceanu, D., Joskowicz, L. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. pp. 418–427. Springer International Publishing, Cham (2020)
  * [27] Li, P., Wang, S., Li, T., Lu, J., HuangFu, Y., Wang, D.: A Large-Scale CT and PET/CT Dataset for Lung Cancer Diagnosis [Data set]. https://doi.org/10.7937/TCIA.2020.NNC2-0461 (2020)
  * [28] Mlynarski, P., Delingette, H., Criminisi, A., Ayache, N.: Deep learning with mixed supervision for brain tumor segmentation. Journal of Medical Imaging 6(3), 1 – 13 (2019), https://doi.org/10.1117/1.JMI.6.3.034002
  * [29] MONAI Consortium: MONAI: Medical Open Network for AI (03 2020), https://doi.org/10.5281/zenodo.5525502, https://github.com/Project-MONAI/MONAI
  * [30] Nair, V., Hinton, G.E.: Rectified Linear Units Improve Restricted Boltzmann Machines. In: ICML. pp. 807–814 (2010), https://icml.cc/Conferences/2010/papers/432.pdf
  * [31] Organization, W.H., et al.: WHO report on cancer: setting priorities, investing wisely and providing care for all (2020)
  * [32] Pang, S., Du, A., He, X., Díez, J., Orgun, M.A.: Fast and Accurate Lung Tumor Spotting and Segmentation for Boundary Delineation on CT Slices in a Coarse-to-Fine Framework. In: Gedeon, T., Wong, K.W., Lee, M. (eds.) Neural Information Processing. pp. 589–597. Springer International Publishing, Cham (2019)
  * [33] Pedersen, A., Smistad, E., Rise, T.V., Dale, V.G., Pettersen, H.S., Nordmo, T.A.S., Bouget, D., Reinertsen, I., Valla, M.: Hybrid guiding: A multi-resolution refinement approach for semantic segmentation of gigapixel histopathological images (2021)
  * [34] Phuong, M., Lampert, C.: Towards Understanding Knowledge Distillation. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 5142–5151. PMLR (09–15 Jun 2019), https://proceedings.mlr.press/v97/phuong19a.html
  * [35] Ramanto, K.N., Parikesit, A.A.: The usage of deep learning algorithm in medical diagnostic of breast cancer. Malaysian Journal Fundam Appl Sci 15(2), 274–281 (2019)
  * [36] Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional Networks for Biomedical Image Segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. pp. 234–241. Springer International Publishing, Cham (2015)
  * [37] Simpson, A.L., Antonelli, M., Bakas, S., Bilello, M., Farahani, K., van Ginneken, B., Kopp-Schneider, A., Landman, B.A., Litjens, G., Menze, B., Ronneberger, O., Summers, R.M., Bilic, P., Christ, P.F., Do, R.K.G., Gollub, M., Golia-Pernicka, J., Heckers, S.H., Jarnagin, W.R., McHugo, M.K., Napel, S., Vorontsov, E., Maier-Hein, L., Cardoso, M.J.: A large annotated medical image dataset for the development and evaluation of segmentation algorithms (2019)
  * [38] Själander, M., Jahre, M., Tufte, G., Reissmann, N.: EPIC: An Energy-Efficient, High-Performance GPGPU Computing Research Infrastructure (2019)
  * [39] Sudre, C.H., Li, W., Vercauteren, T., Ourselin, S., Jorge Cardoso, M.: Generalised Dice Overlap as a Deep Learning Loss Function for Highly Unbalanced Segmentations. Lecture Notes in Computer Science p. 240–248 (2017), http://dx.doi.org/10.1007/978-3-319-67558-9_28
  * [40] Sun, L., Wu, J., Ding, X., Huang, Y., Wang, G., Yu, Y.: A Teacher-Student Framework for Semi-supervised Medical Image Segmentation From Mixed Supervision (2020)
  * [41] Sung, H., Ferlay, J., Siegel, R., Laversanne, M., Soerjomataram, I., Jemal, A., Bray, F.: Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries. CA: a cancer journal for clinicians 71 (02 2021)
  * [42] Uzelaltinbulat, S., Ugur, B.: Lung tumor segmentation algorithm. Procedia Computer Science 120, 140–147 (2017), https://www.sciencedirect.com/science/article/pii/S187705091732433X, 9th International Conference on Theory and Application of Soft Computing, Computing with Words and Perception, ICSCCW 2017, 22-23 August 2017, Budapest, Hungary
  * [43] Wang, D., Li, M., Ben-Shlomo, N., Corrales, C.E., Cheng, Y., Zhang, T., Jayender, J.: Mixed-Supervised Dual-Network for Medical Image Segmentation. In: Shen, D., Liu, T., Peters, T.M., Staib, L.H., Essert, C., Zhou, S., Yap, P.T., Khan, A. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2019. pp. 192–200. Springer International Publishing, Cham (2019)
  * [44] Xie, Q., Luong, M.T., Hovy, E., Le, Q.V.: Self-training with Noisy Student improves ImageNet classification (2020)
  * [45] Zhang, D., Chen, B., Chong, J., Li, S.: Weakly-Supervised teacher-Student network for liver tumor segmentation from non-enhanced images. Medical Image Analysis 70, 102005 (2021), https://www.sciencedirect.com/science/article/pii/S1361841521000517