11institutetext:  Research Center for Healthcare Data Science, Zhejiang Lab,
Hangzhou, China
11email<EMAIL_ADDRESS>22institutetext: Department of Radiology,
Sir Run Run Shaw Hospital, College of Medicine, Zhejiang University, Hangzhou,
China 33institutetext: Centre for Medical Image Computing, University College
London, London, UK

# Adaptively Re-weighting Multi-Loss Untrained Transformer for Sparse-View
Cone-Beam CT Reconstruction

Anonymous  Minghui Wu 11 Yangdi Xu 11 Yingying Xu 11 Guangwei Wu 11 Qingqing
Chen 22 Hongxiang Lin 1133

###### Abstract

Cone-Beam Computed Tomography (CBCT) has been proven useful in diagnosis, but
how to shorten scanning time with lower radiation dosage and how to
efficiently reconstruct 3D image remain as the main issues for clinical
practice. The recent development of tomographic image reconstruction on
sparse-view measurements employs deep neural networks in a supervised way to
tackle such issues, whereas the success of model training requires quantity
and quality of the given paired measurements/images. We propose a novel
untrained Transformer to fit the CBCT inverse solver without training data. It
is mainly comprised of an untrained 3D Transformer of billions of network
weights and a multi-level loss function with variable weights. Unlike
conventional deep neural networks (DNNs), there is no requirement of training
steps in our approach. Upon observing the hardship of optimising Transformer,
the variable weights within the loss function are designed to automatically
update together with the iteration process, ultimately stabilising its
optimisation. We evaluate the proposed approach on two publicly available
datasets: SPARE and Walnut. The results show a significant performance
improvement on image quality metrics with streak artefact reduction in the
visualisation. We also provide a clinical report by an experienced radiologist
to assess our reconstructed images in a diagnosis point of view. The source
code and the optimised models are available from the corresponding author on
request at the moment.

##  1 Introduction

Cone-beam Computerised Tomography (CBCT) has been widely applied in dental and
maxillofacial imaging as gold standards, and extended to lung imaging.
Compared to conventional 2D CT method, CBCT utilises a fast 3D imaging method
to visualise 3D anatomy. Researches show that its radiation dose under full-
view acquisition is effectively higher than plain radiographs [18], and hence,
sparse-view CBCT is more attractable owing to lower radiation dose and faster
scanning time compared to full-view CBCT. However, the classical
reconstruction methods for sparse-view CBCT, for example back projection-based
[9] and compressed sensing-based [3] approaches may reconstruct sub-optimal
image with streak artefacts and may have slow convergence of iterations.

Broadly, there is a surge on adopting deep neural networks (DNNs) to
tomographic imaging reconstruction, where DNNs mostly regularise the raw
measurements/images or invert high-quality images from the measurements in a
supervised way [22, 23]. For example, Jiang et al apply the symmetric residual
DNN to CBCT reconstruction [12]. Zhang et al enhance CBCT image quality using
the generative adversarial network [26]. The success of model training
substantially relies on the quantity and quality of the given paired data,
whereas tomographic measurements are rarely available in the clinical
scenario. On the other hand, simulation helps generate a large-scale dataset
but the model trained on emulated data may have limited generalisation
ability. Generative adversarial network-based model extrapolates super sparse-
view training data to have better perceptual visualisation of scans, but may
suffer from the hallucination that adds false positive structure into the
image anatomy [2].

We propose an untrained Transformer having billions of network weights to fit
a CBCT inverse solver without large number of data, ground-truth images, or
geometric information when the reference images are provided. Fitting the
Transformer model adapts the deep image prior (DIP) technology that is used to
recurrently optimise network parameters to restore better image in neural
representation [6, 21]. Using Transformer overcomes the limitation of small
capacity networks and further enhances the reference images. We adopt UNEt
TRansformer (UNETR) [11] as the exampled Transformer architecture. We further
incorporate mean squared error (MSE) and perceptual loss (PL) [13] to obtain
both smooth and visually pleasing output of CBCT reconstruction. However,
Transformer is notoriously hard to optimise, especially when multi-loss
functions are used [4]. We propose an automatic re-weight technique that
locally optimises the multi-loss weights along with the iteration of
Transformer weights.

Our main contributions are: 1) Optimising the large-scale 3D Transformer with
only one reference data but without ground truth data; 2) Stabilising the
iterative optimisation of multi-loss untrained Transformer via re-weighting
technique; 3) Extending the most DIP research on 2D to 3D medical imaging
scenario; 4) Breaking the jail of cross-scanner and cross-protocol issues that
commonly barrier the generalisability of the trained model in machine
learning-based medical imaging problems. Our overall paradigm is not limited
to UNETR, and can extend to other vision Transformers such as Swin Transformer
[16] or SEgmentation TRansformers (SETR)[24].

##  2 Method

###  2.1 Cone-Beam CT with Untrained Neural Networks

Figure 1: The framework of the proposed multi-loss untrained networks for
sparse-view CBCT reconstruction. Three main modules in our proposed approach
function to reconstruct the referred CBCT image by the FDK algorithm, to
generate the Transformer prior-embedded image, and to extract multi-level
image features using the VGG-11 network. The referred and the generated images
are inputted to the per-pixel mean-squared-error (MSE) loss. The multi-level
image features are inputted to the weighted multi-objective perceptual loss
(PL). Dimension Shrinkage Convolution (DSConv) can learn to reduce 3D features
to 2D features of $3$ channels. The abbreviations of Conv, ReLU, and FC refer
to convolution operation, rectified linear unit, and fully-connected layer.
Red fonts indicate the variable weights for optimisation.

To formulate the 3D CBCT problem, we denote $X$ the imaging region-of-interest
(ROI) of $N_{1}\times N_{2}\times N_{3}$ voxels and $Y$ the $M$-view
measurements acquired on a detector of $D_{1}\times D_{2}$ pixels. We propose
to invert the regularised CBCT image $X$ from the given measurements $Y$ by
the framework illustrated in Figure 1.

####  2.1.1 Network Modules

The entire framework is comprised of three modules:

  * •

Image Reconstructor: We use the Feldkamp-Davis-Kress (FDK) algorithm [9] to
recover the reference $\hat{X}$ having same dimensions as $X$ from the given
measurements $Y$.

  * •

Image Generator: $16$ random noise volumes denoted by $\varepsilon$ are fed
into untrained deep neural networks denoted by $G:\mathbb{R}^{16\times
N_{1}\times N_{2}\times N_{3}}\rightarrow\mathbb{R}^{N_{1}\times N_{2}\times
N_{3}}$, says Transformer architecture, to generate the regularised CBCT image
$X$. Our implementation of Transformer will be describe in Section 2.3.

  * •

Feature Extractor: Composed of a Dimension-Shrinkage Convolution (DSConv) and
a pre-trained VGG-11 [20] feature extractor. The DSConv operation
$C:\mathbb{R}^{N_{1}\times N_{2}\times N_{3}}\rightarrow\mathbb{R}^{3\times
N_{2}\times N_{3}}$ enables the reduction of the first dimension of output
image $X$ to $3$, subsequently matching the input shape of the VGG-11 network.

As inspired by DIP[21] and Deep Decoder[6], we optimise the network weights of
the untrained Transformer requiring only the sparse-view measurement data. The
network weights are considered as a parameterisation of the full-view like
CBCT image once loss function is converged.

####  2.1.2 Multi-Level Perceptual Loss

We incorporate a variant of PL with MSE to serve as the total loss; see Figure
1. Unlike the original PL only making use of the high-level features in pre-
trained networks, we utilise a mixture of all the activation maps in those
networks, aiming to remove streak artefacts based on all-level perceptual
features. Hence, the total loss function can be written as

$L(\theta,\varphi,\hat{\varphi},\mathbf{w})=\|\hat{X}-X_{\theta}\|_{2}^{2}+\alpha\sum_{i=1}^{K}w_{i}\|f_{i}(\hat{C}_{\hat{\varphi}}(\hat{X}))-f_{i}({C}_{{\varphi}}(X_{\theta}))\|_{2}^{2},$
(1)

where $\theta$ denotes the network weights in the Image Generator module,
$\alpha$ the regularisation parameter, $\mathbf{w}=(w_{1},\cdots,w_{K})$ the
weights of multi-level PL of $K$ levels, $\varphi$ and $\hat{\varphi}$
respectively the weights of the reconstruction DSConv $C_{\varphi}$ and the
generation DSConv $\hat{C}_{\hat{\varphi}}$, and $f_{i}$ the intermediate
output of feature extractor at the $i$th Rectified Linear Unit (ReLU) layer.
We freeze the network weights of the two VGG-11 feature extractors that were
pre-trained on ImageNet [7].

###  2.2 Adaptively Re-weight Multi-Loss

In Equation 1, a successful choice of multi-level PL weights $\mathbf{w}$
ensures stable optimisation of our proposed method. Instead of using a group
of fixed heuristic values, we propose a novel adaptive way that regards
$\mathbf{w}$ as differentiable variables and subsequently re-weight them along
with the network weights. As demonstrated in Algorithm 1, we use automatic
differentiation (AD) to compute gradient, and use gradient descend (GD) (or
ADAM [15] as an advanced option) to update weights. Moreover, we observed that
some adaptive weight component $w_{i}$ may occasionally alter to a negative
number, leading to negative loss value and hence substantially degraded
reconstruction performance. Through our empirical study, normalising
$\mathbf{w}$ can stabilise the optimisation step. To tackle this problem, we
employ the clip/gradient clip and the subsequent normalisation to $\mathbf{w}$
during the iterative process.

Algorithm 1 Adaptively Reweight Multi-Loss

Input: Initial variable weights $\theta^{0}$, $\varphi^{0}$,
$\hat{\varphi}^{0},w_{i}^{0}$, learning rate $l$, random noise $\varepsilon$
Output: output image $X_{\theta^{T}}$

for $t=0,\cdots,T-1$ do

Clip: $w_{i}^{t}:=\max(w_{i}^{t},0)$ for each $i=1,\cdots,K$;

Normalise: $w_{i}^{t}:=w_{i}^{t}/\sum_{i=1}^{K}w_{i}^{t}$ for each
$i,\cdots,K$;

Compute: $L(\theta^{t},\varphi^{t},\hat{\varphi}^{t},w_{i}^{t})$ by Equation
1;

AD: $(\nabla L_{\theta}^{t},\nabla
L_{\varphi}^{t},\nabla_{\hat{\varphi}}^{t},\nabla
L_{w_{i}})^{t}\rightarrow(\nabla L_{\theta},\nabla
L_{\varphi},\nabla_{\hat{\varphi}},\nabla
L_{w_{i}})(\theta^{t},\varphi^{t},\hat{\varphi}^{t},\mathbf{w}^{t})$;

Optimise: $\theta^{t+1}=\theta^{t}-l\nabla L_{\theta^{t}}$,
$\varphi^{t+1}=\varphi^{t}-l\nabla L_{\varphi}^{t}$,
$\hat{\varphi}^{t+1}=\hat{\varphi}^{t}-l\nabla L_{\hat{\varphi}^{t}}$,
$w_{i}^{t+1}=w_{i}^{t}-l\nabla L_{w_{i}^{t}}$; $\triangleright$ We use
Gradient Descent here but still can extend to ADAM.

Compute: $X_{\theta^{t+1}}=G_{\theta^{t+1}}(\varepsilon)$.

end for

###  2.3 Transformer Architecture

We adopt a variant of UNETR [11] as our implemented Transformer architecture
for the Image Generator module, without the use of activation layer before
outputting images. Basically, UNETR has a U-Net [5] architecture that includes
the following modules: token (or 3D patch) extraction, linear projection added
with positional embedding, Transformer encoding, DNNs decoding with U-Net
style skip connection. The detailed illustration can be found in Figure S4.

In specific, we first input $16$ random noise volumes of a uniform size of
$N_{1}\times N_{2}\times N_{3}$. All input volumes are then divided into
uniform non-overlapping $16\times 16\times 16$ tokens, totally forming a token
sequence of the length $n$. Each token being flatten and its position code
separately project to $K$ dimensional space and then sum up together. Next, we
utilise twelve Transformer blocks to perform encoding process. Each block
includes a normalisation layer, the multi-head self-attention (MSA), a
summation operation, a normalisation layer, the multilayer perceptron (MLP)
sublayers and a summation operation in a sequential order. The summation
operation adds the input feature and its corresponding output together via
skip connections. Lastly, the expansion path of DNNs decoding is the same as
the 3D U-Net, whereas we require the output of Transformer block to have equal
resolutions with same level decoder features by several deconvolution
operations of a upsampling factor of $2$ before concatenation. Two consecutive
$3\times\ 3\times 3$ convolutions with Batch Normalisation and ReLU
subsequently follows the last deconvolution operation. Such process is
repeated for all subsequent layers until the final output has the same
resolution as the original input.

##  3 Experiments

###  3.1 Data Description

We perform our study on the publicly available the SPArse-view REconstruction
(SPARE) Challenge dataset (abbreviated as SPARE) [19] and Walnut dataset [8].
As for the SPARE dataset, five patients are scanned on an Elekta Synergy
scanner (Elekta, Stockholm, Sweden). A cone-beam source emits approximately
$1008$-$1022$ projection views uniformly on $180$ degree using the full-fan
mode separately onto a $0.8$ mm-resolution detector of $512\times 512$ pixels.
The region-of-interest is set on lung with $176\times 176\times 176$ voxels
with the size $1$ mm. The source-to-object distance (SOD) and the source-to-
detector distance (SDD) are $1000$ mm and $1536$ mm, respectively. The FDK
algorithm is performed by the ASTRA toolbox 2.0 [1] to construct both the
full-view GT volumes and the referred sparse-view volumes. The Walnut dataset
has a set of $42$ walnut projection data as measurements acquired by the FleX-
ray scanner (XRE nv, Ghent, Belgium) on three circular orbits. For each orbit,
a cone-beam source emits $1201$ projection views uniformly on $360$ degree
with the full-fan mode separately onto a $0.1496$ mm-resolution detector of
$768\times 972$ pixels. SOD and SDD are $66$ mm and $199$ mm. We use the FDK
algorithm to construct both the ground-truth (GT) volumes for full-view
measurements and the reference volumes for sparse-view measurements, both
reconstructed on the middle circular orbit. All the resulted volumes have
$501\times 501\times 501$ voxels with the size $0.1$ mm.

###  3.2 Implementation Details

The proposed method was implemented on Pytorch 1.9 [17] and was run on a
single Nvidia Tesla V100 GPU with $32$ gigabytes memory. In terms of classical
reconstruction algorithms, we used FDK and Simultaneous Iterative
Reconstruction Technique (SIRT) [14] algorithms in ASTRA-toolbox 2.0 to obtain
sparse-view reconstructed images. The iteration number in SIRT was set to 200.
The ADAM optimiser was used with a starting learning rate of $10^{-3}$ and the
decay was $10^{-5}$. The regularisation parameter $\alpha$ was $1$. The inputs
of our proposed network were $16$ random noise volumes drawn from the 3D
standard Gaussian distribution with the size same as the reference image. We
initialised all network weights with Glorot normal initialiser [10]. For the
UNETR backbone, the patch size (or the token size) was set to $16\times
16\times 16$, the multi-heads number was $12$ with $768$ embedding size, and
the MLP output size was $3072$. To unify input feature dimensions, the patch-
based 3D U-Net baseline chose the same input shape of $16\times 16\times 16$.
All DNN-based methods terminated at $2000$th iteration.

Table 1: Evaluation results for the reconstruction models of FDK, SIRT, and DIPs using 3D U-Net, UNETR and our approach. All models were performed on the SPARE and the Walnut datasets. The mean and the standard deviation (std) over all subjects in the datasets were computed. Bold font denotes the best mean. The asterisk ∗ denotes $p$-value$<0.05$ compared with the rest methods in the sense of the one-sided Wilcoxon signed-rank test. Dataset | SPARE | Walnut
---|---|---
Metric | PSNR (dB) | SSIM | PSNR (dB) | SSIM
FDK | $27.21\pm 1.25$ | $0.840\pm 0.028$ | $40.92\pm 1.43$ | $0.978\pm 0.011$
SIRT | $29.50\pm 1.65$ | $0.844\pm 0.023$ | $32.05\pm 0.94$ | $0.894\pm 0.016$
3D U-Net | $27.48\pm 1.24$ | $0.843\pm 0.027$ | $-$ | $-$
UNETR | $28.65\pm 1.27$ | $0.859\pm 0.026$ | $40.71\pm 1.19$ | $\textbf{0.983}\pm 0.005$
Ours | $\textbf{32.06}\pm 1.91^{*}$ | $\textbf{0.918}\pm 0.036^{*}$ | $\textbf{43.33}\pm 1.68^{*}$ | $\textbf{0.983}\pm 0.009$

###  3.3 Results

The comparative study was carried out on FDK, SIRT, 3D U-Net [5], UNETR and
our proposed method. The DIPs was used in the latter three approaches. All the
neural networks had comparable model capacity. We employed a one-sided
Wilcoxon signed-rank test to determine the statistical significance of the
performance difference between two comparing methods. Table 1 showed that for
both SPARE and Walnut datasets, our approach significantly (p-value $<0.05$)
outperformed all other methods measured by peak signal-to-noise ratio (PSNR)
and structural similarity index (SSIM) [25].

Figure 2:  Visual comparison of the sagittal views by (a) FDK, (b) SIRT, (c)
Untrained U-Net, (d) Untrained UNETR, (e) Our proposed approach, and (f)
Ground Truth (GT) on the $96$th profile of the last subject in the SPARE
dataset. Zoomed regions of the resulted images are highlighted. The error maps
are normalised absolute difference between the reconstructed volumes and the
GT volumes for each voxel.

To visualise reconstruction performance in an individual volume map, Figure 2
compared the sagittal planes of different reconstructed results using a single
patient measurement of $100$-view points from the SPARE dataset. We observed
that our proposed approach enhanced the resolution both in a zoomed region and
showed lower error score in the image domain compared to the other baseline
methods. The qualitative analysis was done by an experienced radiologist who
was specialised in lung CT image analysis. The report indicated that the GT
image and our result apparently visualised anatomical structure of the main
bronchus. The images reconstructed by FDK, U-Net and UNETR showed no clear
vision of bronchus due to background noise and low contrast issue. The
reconstructed image by SIRT demonstrated severe distortion and spike artefact.

We further conducted an ablation study of three different weighting strategies
for our proposed approach. They were switching-off the multi-level perceptual
loss (denoted by $\mathbf{w}=0$), fixing weights for the multi-level
perceptual loss ($\mathbf{w}=1$) and automatically reweighting the multi-level
perceptual loss (Reweight $\mathbf{w}$). In Figure 3(a), we observed that for
all the weighting strategies, the iteration curves in terms of the PSNR score
between the iteration output and the FDK reference globally increased despite
the several ”dips” (also observed in [4]), which might partially fail the
optimisation of our approach. The case $\mathbf{w}=0$ strategy had higher
performance than the other two, however, this had an incoherent trend between
the iteration output and the GT image, as illustrated in Figure 3(b). The
cases $\mathbf{w}=1$ and the case Reweighting $\mathbf{w}$ showed consistent
performance increase in terms of both the FDK reference and the GT image, but
the case Reweighting $\mathbf{w}$ had fewer ”dips” hence more stable than the
case $\mathbf{w}=1$. Additional experiments on the robustness of our method on
different numbers of projection views and the visualisation results for the
Walnut dataset can be found in the Figures S5 and S6, respectively.

|
---|---
(a) compared to the FDK reference | (b) compared to GT
Figure 3: Ablation study of our approach along iteration steps on three
different weighting strategies: $\mathbf{w}=0$, $\mathbf{w}=1$ and
automatically reweighting $\mathbf{w}$. Each iteration output were (a)
compared to the FDK reference commonly available and (b) compared to the
ground-truth (GT) image unavailable in the real scenario. The evaluation data
is a single patient from the SPARE dataset.

##  4 Discussion and Conclusion

Our proposed approach is a novel multi-loss DIP approach on an untrained
Transformer for the sparse-view CBCT reconstruction. We extend the
conventional DIP to 3D large-scale network appended with multi-level PL, and
subsequently overcome the difficulty of both optimising the used Transformer
and reweighting multi-loss. We quantitatively and qualitatively demonstrate
that the proposed approach enhances the reconstructed image quality and
stabilises the convergence to the GT image, coherently uniform with the
observable trends to the FDK reference.

Our proposed Transformer-based multi-loss DIP offers several avenues for
future improvement and application. First, optimisation of the multi-level PL
highly depends on a proper choice of a pre-trained network (e.g. VGG-11);
other feature extractors for medical images such as Pre-trained Models Genesis
[27] may be worth investigating. Broadening generalisation bound of the DIP-
type method may be another working direction [28]. On the other hand, our
approach can potentially extend to other tomographic imaging modalities,
resulting in speeding up the acquisition, reducing radiation exposure, and
increasing the patient throughput. In future work, we will evaluate the
clinical viability of our approach for CBCT imaging.

### Acknowledgements

This work was supported in part by Research Initiation Project of Zhejiang Lab
(No. 2021ND0PI02), China Postdoctoral Science Foundation (2020TQ0293), the
National Natural Science Foundation of China (82071988), the Key Research and
Development Program of Zhejiang Province (2019C03064), the Program Co-
sponsored by Province and Ministry (No.WKJ-ZJ-1926), and the Special Fund for
Basic Scientific Research Business Expenses of Zhejiang University
(No.2021FZZX003-02-17).

## References

  * [1] van Aarle, W., Palenstijn, W.J., Cant, J., Janssens, E., Bleichrodt, F., Dabravolski, A., De Beenhouwer, J., Joost Batenburg, K., Sijbers, J.: Fast and flexible X-ray tomography using the ASTRA toolbox. Optics Express 24(22), 25129 (2016)
  * [2] Bhadra, S., Kelkar, V.A., Brooks, F.J., Anastasio, M.A.: On Hallucinations in Tomographic Image Reconstruction. IEEE Transactions on Medical Imaging 40(11), 3249–3260 (2021)
  * [3] Bian, J., Siewerdsen, J.H., Han, X., Sidky, E.Y., Prince, J.L., Pelizzari, C.A., Pan, X.: Evaluation of sparse-view reconstruction from flat-panel-detector cone-beam CT. Physics in Medicine and Biology 55(22), 6575–6599 (2010)
  * [4] Chen, X., Xie, S., He, K.: An Empirical Study of Training Self-Supervised Vision Transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9640–9649 (2021)
  * [5] Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3d u-net: learning dense volumetric segmentation from sparse annotation. In: International conference on medical image computing and computer-assisted intervention. pp. 424–432. Springer (2016)
  * [6] Darestani, M.Z., Heckel, R.: Accelerated MRI with Un-Trained Neural Networks. IEEE Transactions on Computational Imaging 7, 724–733 (2021)
  * [7] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition (CVPR). pp. 248–255. Ieee (2009)
  * [8] Der Sarkissian, H., Lucka, F., van Eijnatten, M., Colacicco, G., Coban, S.B., Batenburg, K.J.: A cone-beam X-ray computed tomography data collection designed for machine learning. Scientific Data 6(1), 1–8 (2019)
  * [9] Feldkamp, L.A., Davis, L.C., Kress, J.W.: Practical cone-beam algorithm. Josa a 1(6), 612–619 (1984)
  * [10] Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics. pp. 249–256 (2010)
  * [11] Hatamizadeh, A., Tang, Y., Nath, V., Yang, D., Myronenko, A., Landman, B., Roth, H.R., Xu, D.: Unetr: Transformers for 3d medical image segmentation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 574–584 (2022)
  * [12] Jiang, Z., Chen, Y., Zhang, Y., Ge, Y., Yin, F.F., Ren, L.: Augmentation of CBCT Reconstructed from Under-Sampled Projections Using Deep Learning. IEEE Transactions on Medical Imaging 38(11), 2705–2715 (2019)
  * [13] Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual Losses for Real-Time Style Transfer and Super-Resolution. In: European conference on computer vision. vol. 9906, pp. 694–711 (2016)
  * [14] Kak, A.C., Slaney, M.: Principles of computerized tomographic imaging. SIAM (2001)
  * [15] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (2015)
  * [16] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10012–10022 (2021)
  * [17] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: An imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, pp. 8024–8035 (2019)
  * [18] Patel, S., Durack, C., Abella, F., Shemesh, H., Roig, M., Lemberg, K.: Cone beam computed tomography in Endodontics - a review. International Endodontic Journal 48(1), 3–15 (2015)
  * [19] Shieh, C.C., Gonzalez, Y., Li, B., Jia, X., Rit, S., Mory, C., Riblett, M., Hugo, G., Zhang, Y., Jiang, Z., Liu, X., Ren, L., Keall, P.: SPARE: Sparse-view reconstruction challenge for 4D cone-beam CT from a 1-min scan. Medical Physics 46(9), 3799–3811 (2019)
  * [20] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: 3rd International Conference on Learning Representations, ICLR 2015. pp. 1–14 (2015)
  * [21] Ulyanov, D., Vedaldi, A., Lempitsky, V.: Deep Image Prior. International Journal of Computer Vision 128(7), 1867–1888 (2020)
  * [22] Wang, G., Jacob, M., Mou, X., Shi, Y., Eldar, Y.C.: Deep Tomographic Image Reconstruction: Yesterday, Today, and Tomorrow-Editorial for the 2nd Special Issue ’Machine Learning for Image Reconstruction’. IEEE Transactions on Medical Imaging 40(11), 2956–2964 (2021)
  * [23] Wang, G., Ye, J.C., De Man, B.: Deep learning for tomographic image reconstruction. Nature Machine Intelligence 2(12), 737–748 (2020)
  * [24] Wang, Y., Xu, Z., Wang, X., Shen, C., Cheng, B., Shen, H., Xia, H.: End-to-end video instance segmentation with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8741–8750 (2021)
  * [25] Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing 13(4), 600–612 (2004)
  * [26] Zhang, Y., Yue, N., Su, M.Y., Liu, B., Ding, Y., Zhou, Y., Wang, H., Kuang, Y., Nie, K.: Improving CBCT quality to CT level using deep learning with generative adversarial network. Medical Physics 48(6), 2816–2826 (2021)
  * [27] Zhou, Z., Sodha, V., Pang, J., Gotway, M.B., Liang, J.: Models Genesis. Medical Image Analysis 67 (2021)
  * [28] Zou, Q., Ahmed, A.H., Nagpal, P., Priya, S., Schulte, R., Jacob, M.: Variational manifold learning from incomplete data: application to multislice dynamic MRI pp. 1–10 (2021)

## Supplementary Materials

Figure S4: The architecture of UNETR presented in Section 2.3. Arrows are transferring tensors with the first and the rest dimensions representing the channel size and the feature size, respectively. |
---|---
(a) PSNR | (b) SSIM
Figure S5: Box-plots of the (a) PSNR and (b) SSIM scores v.s. different projection views on the SPARE dataset. The red solid lines that fit a linear model to the last four median points in term of the number of projection views indicate the critic points of our proposed methods. Figure S6: The axial-plane visualisation of walnut reconstruction results for (a) FDK, (b) SIRT, (c) Untrained UNETR, (d) Our proposed approach (Ours), and (e) Ground Truth (GT). We visualise the $264$th slice of Walnut No.2. Zoomed regions are highlighted. The error maps are normalised absolute difference between the reconstructed volumes and the GT volumes for each voxel. Table S2: $p$-value of the one-sided Wilcoxon signed-rank test evaluated on the SPARE and Walnut dataset on PSNR and SSIM scores by any two of FDK, SIRT, DIPs using U-Net, UNETR and our proposed approach (ours), as a complement to Table 1.  Metrics | PSNR $|$ SSIM
---|---
SPARE Walnut | FDK | SIRT | U-Net | UNETR | Ours
FDK |  1 1 | $0.02|0.02$ | $-$ | $0.34|0.02$ | $0.02|0.02$
SIRT | $0.04|0.34$ |  1 1 | $-$ | $0.02|0.02$ | $0.02|0.02$
U-Net | $0.02|0.02$ | $0.07|0.34$ |  1 1 | $-$ | $-$
UNETR | $0.02|0.02$ | $0.11|0.17$ | $0.02|0.02$ |  1 1 | $0.02|0.45$
Ours | $0.02|0.02$ | $0.04|0.02$ | $0.02|0.02$ | $0.02|0.02$ |  1 1