Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

CLEF 2022: Conference and Labs of the Evaluation Forum, September 5–8, 2022,
Bologna, Italy

<EMAIL_ADDRESS>]

<EMAIL_ADDRESS>]<EMAIL_ADDRESS>]
<EMAIL_ADDRESS>]<EMAIL_ADDRESS>]
<EMAIL_ADDRESS>]

# A Late Fusion Framework with Multiple Optimization Methods for Media
Interestingness

Maria Shoukat  Department of Computer Systems Engineering, University of
Engineering and Technology, Peshawar, Pakistan  Khubaib Ahmad  Department of
Computer Science, Munster Technological University, Cork, Ireland  Naina Said
Nasir Ahmad  Mohammed Hasanuzzaman  Kashif Ahmad

(2022)

###### Abstract

The recent advancement in Multimedia Analytical, Computer Vision (CV), and
Artificial Intelligence (AI) algorithms resulted in several interesting tools
allowing an automatic analysis and retrieval of multimedia content of users’
interests. However, retrieving the content of interest generally involves
analysis and extraction of semantic features, such as emotions and
interestingness-level. The extraction of such meaningful information is a
complex task and generally, the performance of individual algorithms is very
low. One way to enhance the performance of the individual algorithms is to
combine the predictive capabilities of multiple algorithms using fusion
schemes. This allows the individual algorithms to complement each other,
leading to improved performance. This paper proposes several fusion methods
for the media interestingness score prediction task introduced in CLEF Fusion
2022. The proposed methods include both a naive fusion scheme, where all the
inducers are treated equally and a merit-based fusion scheme where multiple
weight optimization methods are employed to assign weights to the individual
inducers. In total, we used six optimization methods including a Particle
Swarm Optimization (PSO), a Genetic Algorithm (GA), Nelder-Mead, Trust Region
Constrained (TRC), and Limited-memory Broyden–Fletcher–Goldfarb–Shanno
Algorithm (LBFGSA), and Truncated Newton Algorithm (TNA). Overall better
results are obtained with PSO and TNA achieving 0.109 mean average precision
@10. The task is complex and generally, scores are low. We believe the
presented analysis will provide a baseline for future research in the domain.

###### keywords:

Media Interestingness Late FusionPSO Genetic Algorithms Nelder Mead Trust
Region Contrainted optimization

##  1 Introduction

In the modern world, thanks to social media and other multimedia content
sharing platforms, we have access to a huge amount of multimedia content.
However, accessing multimedia content of interest generally involves
processing, analyzing, and filtering a huge amount of data. Filtering and
retrieval of multimedia content of interest require specialized tools to
analyze and extract semantic features/meanings from the content [1].

Thanks to the recent development in Multimedia Analytics, Computer Vision (CV)
and Artificial Intelligence (AI) techniques, different semantic notions, such
as sentiments [2], emotions [3], and interestingness-level [4], can be
extracted from multimedia content. More recently, Deep Neural Networks (DNNs)
have shown tremendous predictive capabilities in various multimedia content
analysis tasks. Despite the proven performances, there are several tasks where
a single Neural Network is not enough to accurately extract meaningful
insights precisely. In order to increase the performance of individual models,
researchers have been exploring the so called "fusion" techniques allowing
multiple models to complement each others in such complex tasks [5]. The
fusion techniques allow to combine multiple models to achieve higher accuracy
compared to the individual models. Two popular methods to combine these models
are the early fusion and late fusion techniques. In early fusion, the separate
raw data is integrated into a unified representation before the learning
process. On the other hand, in case of late fusion, fusion is performed at the
decision level i.e, the output of different predictors is combined after the
learning process to create a new and improved super predictor. The literature
has reported the effectiveness of fusion technique in several applications,
such as natural disasters analysis [6], event recognition [5], data analytics
[7].

This paper is based on one of the tasks in ImageCLEF 2022 [8], which is a
benchmark competition for image retrieval tasks. This year ImageCLEF proposes
four different tasks. However, the work is based on Image Interestingness
CLEFfusion 2022 task [9]. This is a regression task that aims at the
prediction of image interestingness score. The task mainly focuses on the
fusion of different inducers, whose scores are already provided, to jointly
predict the interestingness of visual content. In this work, we propose both a
naive fusion, where all inducers are assigned equal weights, and merit-based
fusion techniques with optimized weights to combine the scores of the
individual inducers for better prediction. For the merit-based fusion, we
employ five different techniques to assign weights to the 29 inducers provided
in the dataset by the organizers. These methods include evolutionary
algorithms, namely Particle Swarm Optimization (PSO) and a Genetic algorithm
(GA) based methods, Trust Region Constrained Optimization, Limited-memory
Broyden Fletcher Goldfarb Shann (LMBFGS) method, and Truncated Newton
Algorithm (TNA) method.

The rest of the paper is organized as follows : Section 2 presents the
overview of the related literature. Section 3.2 gives details about different
fusion techniques that have been used in this research work. Section 4
provides the experimental results. Finally, Section 5 concludes the work and
outlines the future directions.

##  2 Related Work

Media interestingness prediction, which involves an automatic analysis of
multimedia content for the identification of relevant content of users’
interest, got great attention from the community over the last few years [10].
It plays a vital role in several applications, such as image retrieval and
recommendation and media summarization, etc. In the literature, the topic has
been analyzed from two different perspectives including psychological and
computational aspects of media interestingness [10]. The first part mainly
focuses on psychological studies involving theoretical analysis and reports on
human emotions, choices, and interests. For instance, Silvia et al. [11]
linked interestingness level with emotions by providing a detailed analysis
and overview of some unusual aesthetic emotions. The computational methods on
the other hand involve multimedia analytics, CV, and ML techniques to analyze
and extract semantic features from multimedia content for the prediction of
interestingness level [12]. Extensive research exploring different aspects of
the topic has been carried out in this direction. For instance, Liu et al.
[13] analyzed the importance of feature extraction for the task by proposing a
multi-view manifold learning framework. To this aim, the authors mapped multi-
view data to a single common space by considering cross-view correlation to
preserve the geometric structure and interestingness information. Wang et al.
[14] discussed other two important aspects of media interestingness namely
comparison information, and evaluation metric optimization. Despite sufficient
improvement, the performance of most of the algorithms is not good as compared
to other computer vision tasks.

As part of the efforts to improve the performances of media interestingness
frameworks, a vast majority of the recent works rely on multiple models. To
this aim, different fusion techniques have been incorporated to jointly employ
multiple models for the task. For instance, Constantin et al. [12] proposed a
deep fusion ensemble framework by exploring the potential of several deep
networks including dense, attention, convolutional, and cross-space-fusion
networks. Similarly, Almedia et al. [15] proposed a late fusion framework
employing multiple ranking models trained on multimodal features for media
interestingness score prediction.

In this work, we explore the potential of merit-based fusion by combining the
predictions of several inducers using different weight optimization methods.

##  3 Methodology

Figure 1 provides the block diagram of the methodology adopted in this work.
There are two main components of the methodology, namely (i) prediction by
individual inducers, and (ii) fusion of the score obtained with the individual
inducers for joint prediction. Our contribution mainly lies in the fusion
part, where we employed several weight selection/optimization techniques to
assign weights to the inducers. The scores of the individual inducers are
already provided by the task organizers. In the next subsections, we provide a
detailed description of all the methods.

Figure 1: Block diagram of the proposed methodology.

###  3.1 Inducer Scores and Pre-processing

The inducers’ scores are provided by the task organizers. In total, the task
organizers provided prediction scores of 29 inducers. For the majority of the
inducers, the prediction scores were floating numbers between 0 and 1.
However, the interestingness scores for some of the inducers were out of
range. To combine the scores of the inducers properly, all the values should
be in the same range. To this aim, before combing the scores in a late fusion,
we normalized the inducers’ scores to bring them to the same range.

###  3.2 Fusion Techniques

In this work, we mainly focused on late fusion techniques where we tried
several weight optimization methods to obtain a combination of weights that
provides highest interestingness score. To this aim, we picked several methods
with proven performances in similar applications [16, 5]. As a baseline, we
also considered a naive fusion method where equal weights are assigned to the
inducers. Our late fusion method is represented by Equation 3.2.

$S_{c}=W_{1}S_{1}+W_{2}S_{2}+W_{3}S_{3}+....+W_{n}S_{n}$ (1)

In the equation, $S_{c}$ represents the combined interestingness score of
different inducers, $S_{n}$ is the score of the nth model and $W_{n}$ is the
corresponding weight used during the fusion. In our case, n=29. For the
baseline, the weight $W$ for all the inducers is the same i.e.,
$W_{1}=W_{2}=W_{3}....=W_{n}$. In case of merit based fusion, optimal weights
are assigned to each individual inducer. In the next section, we provide the
details of these optimization methods.

####  3.2.1 Genetic Algorithm

Our choice of Genetic Algorithm (GA) is based on our previous experience in
similar applications [16, 17, 5]. GA, which is a meta-heuristic algorithm, is
inspired by the natural evaluation process. In the natural evolution process,
the fittest individuals of the current generation are firstly identified and
then used for re-production in the next generation. A similar approach is
adopted in GA-based optimization, where the algorithm searches for optimal
value/set of values minimizing a given function also called fitness function.
To this aim, GA requires a training procedure to determine optimal values. The
process starts with a randomly selected generation of the population (i.e., a
set of values). The fitness of every individual in the current population is
evaluated using the fitness function after which individuals are selected for
the next generation with a modified genome. The process is repeated until
either the maximum number of generations is reached or a respectable fitness
value is achieved.

In this work, since we are dealing with a regression problem, the fitness
function is based on Mean Squared Error (MSE) as shown in equation 2.
Moreover, each possible combination (i.e., set of weights assigned to the 29
inducers) is a potential solution. The goal is to find the set of weights with
minimum MSE.

$MSE={\frac{1}{n}\Sigma_{i=1}^{n}(y_{p}-y_{a})}^{2}$ (2)

In the above equation, $y_{p}$ represents the predicted interestingness score
while $y_{a}$ is the ground truth. For the implementation we used a python
open source library, namely
geneticalgorithm111https://pypi.org/project/geneticalgorithm/. As we have a
total of 29 inducers so we kept the dimensions to 29. Moreover, we used ’real’
for the variable type and the variable boundary fixed between 0 and 1.

####  3.2.2 Particle Swarm Optimzation

The second weight optimization/selection method employed in this work is based
on PSO [18]. The optimization method is inspired by the flocking of birds.
Unlike GA, PSO does not use mutation and crossover operations rather aims at
an improvement to a candidate solution according to a pre-defined criterion,
iteratively. There are three main steps involved in the process. These include
an (i) evaluation of each candidate solution on the basis of fitness criteria,
(ii) updates in personal best and global best values and finally (iii)
updating the position and velocity of each particle.

In our case, each combination of the weights (i.e, 29 values to be assigned to
the inducers) is a candidate solution. Moreover, the fitness function is based
on MSE as shown in 2. For the implementation of the method, we used open
source library namely pyswarm222https://pyswarms.readthedocs.io/en/latest/. As
this algorithm support bounds like GA, we set the lower bound to 0 and the
upper bound to be 1. Moreover, the maximum iterations hyperparameter is set to
10000 and the swarm size is kept at 300.

####  3.2.3 Nelder Mead Algorithm

Nelder Mead algorithm is a heuristic optimization technique and is appropriate
for optimization problems where the gradient of the function is either unknown
or cannot be reasonably computed. The algorithm can be used for both one
dimensional and multi-dimensional optimization problems [19]. The algorithm
starts with randomly generated simplex with number of vertices= $n+1$ points
for an n-dimensional optimization problem. At every iteration, the algorithm
moves the simplex one vertex at a time towards an optimal region in the search
space with a goal to minimize/maximize a certain objective function. At the
end, the vertex of the simplex that yields that most optimal values is
returned. For our experiments, $n=29$ and the objective function definition is
the same as in 2. For the implementation of the method, we used a Python open
source library, namely, SciPy333https://scipy.org/. We set the value of
absolute error in xopt between iterations that is acceptable for convergence
to 1e-8 and the maximum iterations to 10000.

####  3.2.4 Trust Region Constrained Optimization Algorithm

Trust Region Constrained method belongs to the family of optimization methods
that are based on trust regions. Trust regions-based methods solve
optimization problems by defining a region around their current best
solutions, where they can approximate the fitness function up to a certain
extent. The methods then take a step forward within the region. In contrast to
line-based solutions, the step size is determined beforehand of the
improvement in the direction. At this stage, the model is considered to be a
good representation of the original objective function if a significant
decrease in the objective function is observed.

In this work, for the implementation of the method we used scipy
library444https://scipy.org/. The Trust Region Constrained algorithm requires
initial weight values so for all the 29 inducers same value of 0.0345 is used.
For the bounds, we used a lower bound of 0 and upper bound of 1. Moreover, we
set the maximum iterations to 10000.

####  3.2.5 Limited-memory Broyden–Fletcher–Goldfarb–Shanno Algorithm

The Broyden, Fletcher, Goldfarb, and Shanno, or BFGS Algorithm, is a local
search optimization algorithm. It falls under the category of a Quasi-Newton
optimization methods which deal with the optimization of second order
derivative of the objective function. These type of algorithms are suitable
for optimization problems where the second order derivatives can not be
reasonably quantified. Unlike the first order methods which make use of the
first order derivative to find the optimal values of the objective function,
these algorithm rely on second order derivatives. For a multivariate function,
the second derivatives of all the input variables are maintained in a matrix
called Hessian. In order to find the optimal values of the objective function,
the BFGS algorithm calculates the inverse of this matrix. This is done by
approximating the inverse using gradient thereby eliminating the need for
inverse calculation at each step of the algorithm. The size of the Hessian and
its inverse is proportional to the number of input parameters to the objective
function. For a function with many input parameters, the BFGS then becomes
impractical due to very high memory demands. Therefore, a variant of BFGS
called Limited BFGS is utilized in this work. This method does not require
storing the entire approximation of the inverse matrix.

The definition of the objective function for our experiments is the same as
given in 2. For implementation of the method, we used a Python open source
library, namely, SciPy555https://scipy.org/. Similar to Trust Region
Constrained Optimization algorithm, the Limited-memory
Broyden–Fletcher–Goldfarb–Shanno algorithm also requires initial weight values
that are set at 0.0345 for all the 29 inducers. Moreover, the value of
absolute error is set to 1e-8 and the maximum iterations to 10000. For the
bounds, we set the lower and upper bounds to 0 and 1, respectively.

####  3.2.6 Truncated Newton Algorithm

The method is also called Hessian-free optimization algorithm and is more
suitable for applications involving a large numbers of independent variables
[20]. The method uses an iterative process to solve the Newton’s equation,
which involves finding the roots of a differentiable function, for updating
the parameters of the cost function. The term ”truncated” refers to the fact
that the inner solver is run for a limited number of iterations, which means
that the algorithm needs to produce good approximation in limited iterations.
The definition of the objective function for our experiments is the same as
given in 2\. For implementation of the method, we used a Python open source
library, namely, SciPy. The Truncated Newton algorithm also requires initial
weight values, which are set at 0.0345 for all the 29 inducers. Moreover, we
set the value of absolute error in xopt between iterations that are acceptable
for convergence to 1e-8 and the maximum iterations to 10000. The lower and
upper bounds are set at 0 and 1, respectively.

##  4 Experiments and Results

###  4.1 Dataset

The individual inducers’ scores, which are provided by the task organizers,
are extracted from the Interestingness10k dataset [10]. In total, prediction
scores for 2435 images are provided from 29 inducers, each representing the
visual interestingness-level for the images. The dataset is provided in two
seperate sets namely (i) development set, and (ii) test set. The development
set is composed of the scores from all of the inducers for 1877 images while
the test set covers 558 images only. In the development set, each data sample
for each inducer provides four values including a video ID, image ID,
classification score (0 or 1), and predicted interestingness score by the
inducer.

###  4.2 Experimental Results

Table 1 provides the official results of the proposed methods in terms of mean
average precision at the cutoff 10 (MAP@10). One of the main objectives of the
experiments is to evaluate the potential of the different state-of-the-art
optimization methods in this application.

As expected, overall lower results are obtained with the baseline method where
all the inducers are treated equally by assigning them equal weights. Though
there is no significant, difference in scores obtained between the least
performing and the baseline, the merit-based fusion scheme seems more
promising compared to simply averaging the individual scores. As far as the
performances of the merit-based fusion methods are concerned, overall better
results are obtained with PSO and TNC methods. One of the key advantages of
TNC is its optimization capabilities in dealing with functions involving
independent variables. This could be one of the main reasons for its better
performance in the application as all the inducers are treated independently
in the task. The highest score obtained in this work is 0.109 MAP@10, which
indicates the complexity of the task.

Table 1: Experimental results in terms of mean average precision at the cutoff 10 MAP@10. Fusion Method | MAP@10
---|---
Equal Weights | 0.081
Trust-Constr weighted Fusion | 0.095
PSO weighted Fusion | 0.109
GA weighted Fusion | 0.093
LBFGSB weighted Fusion | 0.095
Nelder Mead weighted Fusion | 0.090
TNC weighted Fusion | 0.109

##  5 Conclusions and Future Work

In this paper, we presented the experimental results of multiple fusion
techniques for the media interestingness task presented in CLEF Fusion 2022.
We used both a naive fusion scheme and merit-based fusion methods. Overll the
results are much lower on the task compared to other computer vision tasks,
which shows the complexity of the task. During the experiments, we observed
better results for a merit-based fusion scheme where different weight
optimization techniques are employed to assign weights to the individual
inducers based on their performances. This verifies our assumption that
individual performance should be considered in combining the prediction scores
of the individual inducers.

In the future, we want to further explore different aspects of the application
to further enhance the results. One potential direction could be an
intelligent selection among the inducers instead of considering all of them.

## References

  * Demarty et al. [2017] C.-H. Demarty, M. Sjöberg, M. G. Constantin, N. Q. Duong, B. Ionescu, T.-T. Do, H. Wang,  Predicting interestingness of visual content,  in: Visual content indexing and retrieval with psycho-visual models, Springer, 2017, pp. 233–265.
  * Hassan et al. [2022] S. Z. Hassan, K. Ahmad, S. Hicks, P. Halvorsen, A. Al-Fuqaha, N. Conci, M. Riegler,  Visual sentiment analysis from disaster images in social media,  Sensors 22 (2022) 3628.
  * Bhattacharya et al. [2021] P. Bhattacharya, R. K. Gupta, Y. Yang,  Exploring the contextual factors affecting multimodal emotion recognition in videos,  IEEE Transactions on Affective Computing (2021).
  * Kiziltepe et al. [2021] R. S. Kiziltepe, L. Sweeney, M. G. Constantin, F. Doctor, A. G. S. de Herrera, C.-H. Demarty, G. Healy, B. Ionescu, A. F. Smeaton,  An annotated video dataset for computing video memorability,  Data in Brief 39 (2021) 107671.
  * Ahmad et al. [2018] K. Ahmad, M. L. Mekhalfi, N. Conci, F. Melgani, F. D. Natale,  Ensemble of deep models for event recognition,  ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 14 (2018) 1–20.
  * Said et al. [2018] N. Said, K. Pogorelov, K. Ahmad, M. Riegler, N. Ahmad, O. Ostroukhova, P. Halvorsen, N. Conci,  Deep learning approaches for flood classification and flood aftermath detection.,  in: MediaEval, 2018.
  * Wang [2021] Y. Wang,  Survey on deep multi-modal data analytics: Collaboration, rivalry, and fusion,  ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 17 (2021) 1–25.
  * Ionescu et al. [2022] B. Ionescu, H. Müller, R. Peteri, J. Ruckert, A. Ben Abacha, A. G. S. de Herrera, C. M. Friedrich, L. Bloch, R. Brüngel, A. Idrissi-Yaghir, H. Schäfer, S. Kozlovski, Y. D. Cid, V. Kovalev, L.-D. Stefan, M. G. Constantin, M. Dogariu, A. Popescu, J. Deshayes-Chossart, H. Schindler, J. Chamberlain, A. Campello, A. Clark,  Overview of the ImageCLEF 2022: Multimedia Retrieval in Medical, Social Media and Nature Applications,  in: Experimental IR Meets Multilinguality, Multimodality, and Interaction, Proceedings of the 13th International Conference of the CLEF Association (CLEF 2022), LNCS Lecture Notes in Computer Science, Springer, Bologna, Italy, 2022.
  * Stefan et al. [2022] L.-D. Stefan, M. G. Constantin, M. Dogariu, B. Ionescu,  Overview of imagecleffusion 2022 task - ensembling methods for media interestingness prediction and result diversification,  in: CLEF2022 Working Notes, CEUR Workshop Proceedings, CEUR-WS.org, Bologna, Italy, 2022.
  * Constantin et al. [2021] M. G. Constantin, L.-D. Ştefan, B. Ionescu, N. Q. Duong, C.-H. Demarty, M. Sjöberg,  Visual interestingness prediction: A benchmark framework and literature review,  International Journal of Computer Vision 129 (2021) 1526–1550.
  * Silvia [2009] P. J. Silvia,  Looking past pleasure: anger, confusion, disgust, pride, surprise, and other unusual aesthetic emotions.,  Psychology of Aesthetics, Creativity, and the Arts 3 (2009) 48.
  * Constantin et al. [2022] M. G. Constantin, L.-D. Ştefan, B. Ionescu,  Exploring deep fusion ensembling for automatic visual interestingness prediction,  in: Human Perception of Visual Information, Springer, 2022, pp. 33–58.
  * Liu et al. [2017] Y. Liu, Z. Gu, Y.-m. Cheung, K. A. Hua,  Multi-view manifold learning for media interestingness prediction,  in: Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval, 2017, pp. 308–314.
  * Wang et al. [2018] S. Wang, S. Chen, J. Zhao, Q. Jin,  Video interestingness prediction based on ranking model,  in: Proceedings of the joint workshop of the 4th workshop on affective social multimedia computing and first multi-modal affective computing of large-scale multimedia data, 2018, pp. 55–61.
  * Almeida et al. [2017] J. Almeida, L. P. Valem, D. C. Pedronette,  A rank aggregation framework for video interestingness prediction,  in: International conference on image analysis and processing, Springer, 2017, pp. 3–14.
  * Ahmad et al. [2022] K. Ahmad, M. A. Ayub, K. Ahmad, J. Khan, N. Ahmad, A. Al-Fuqaha,  Merit-based fusion of nlp techniques for instant feedback on water quality from twitter text,  arXiv preprint arXiv:2202.04462 (2022).
  * Ahmad et al. [2020] K. Ahmad, K. Khan, A. Al-Fuqaha,  Intelligent fusion of deep features for improved waste classification,  IEEE access 8 (2020) 96495–96504.
  * Kennedy and Eberhart [1995] J. Kennedy, R. Eberhart,  Particle swarm optimization,  in: Proceedings of ICNN’95-international conference on neural networks, volume 4, IEEE, 1995, pp. 1942–1948.
  * Singer and Nelder [2009] S. Singer, J. Nelder,  Nelder-mead algorithm,  Scholarpedia 4 (2009) 2928.
  * Martens et al. [2010] J. Martens, et al.,  Deep learning via hessian-free optimization.,  in: ICML, volume 27, 2010, pp. 735–742.


# Hybrid Multi-Head Physics-informed Neural Network for Depth Estimation in
Terahertz Imaging

Mingjun Xiang  Hui Yuan  Kai Zhou  and Hartmut G. Roskos  Frankfurt Institute
for Advanced Studies (FIAS), 60438 Frankfurt am Main, Germany  Physikalisches
Institut, Goethe-Universität Frankfurt am Main, 60438 Frankfurt am Main,
Germany  Xidian-FIAS International Joint Research Center, 60438 Frankfurt am
Main, Germany  School of Science and Engineering, The Chinese University of
Hong Kong, Shenzhen, P.R. China

###### Abstract

Terahertz (THz) imaging is one of the hotspots in the field of optics, where
the depth information retrieval is a key factor to restore the three-
dimensional appearance of objects. Impressive results for depth extraction in
visible and infrared wave range have been demonstrated through deep learning
(DL). Among them, most DL methods are merely data-driven, lacking relevant
physical priors, which thus request for a large amount of experimental data to
train the DL models. However, large training data acquirement in the THz
domain is challenging due to the requirements of environmental and system
stability, as well as the time-consuming data acquisition process. To overcome
this limitation, this paper incorporates a complete physical model
representing the THz image formation process into traditional DL networks to
retrieve the depth information of objects. The most significant advantage is
the ability to use it without pre-training, thereby eliminating the need for
tens of thousands of labeled data. Through experiments validation, we
demonstrate that by providing diffraction patterns of planar objects with
their upper and lower halves individually masked, the proposed physics-
informed neural network (NN) can automatically optimize and, ultimately,
reconstruct the depth of the object through interaction between the NN and a
physical model. The obtained results represent the initial steps towards
achieving fast holographic THz imaging using reference-free beams and low-cost
power detection.

††journal: Computer Physics Communications

##  1 Introduction

THz radiation refers to electromagnetic radiation within the frequency range
of approximately 0.1 THz to 10 THz, corresponding to wavelengths of 3 mm to 30
$\mu$m. THz imaging technology, as a promising direction in THz research,
primarily benefits from the unique physical properties of this frequency band:
firstly, it possesses strong penetration capability, enabling easy
transmission through non-metallic and non-polar materials such as ceramics,
plastics, and foam, which are commonly opaque to infrared light; secondly, THz
waves exhibit low photon energy, thus avoiding harmful ionization effects;
thirdly, many molecules display distinct absorption and dispersion
characteristics in the THz frequency range, allowing the establishment of
molecular fingerprint spectra for substance identification; Moreover, THz
waves demonstrate high sensitivity to water, making them particularly suitable
for analyzing material hydration levels [1, 2, 3]. Over the past two decades,
advancements in electronics and photonics technologies have propelled the
rapid development of THz imaging technology, expanding its application areas
into fields such as non-destructive testing [4], quality monitoring [5],
security screening [6, 7], biomedical imaging [8], and sensing for robotics
and vehicle control [9], where it has achieved notable successes,
demonstrating advantages unmatched by traditional imaging techniques.

Depth information plays a crucial role in THz imaging; however, due to the
lack of suitable hardware, the construction of THz multi-pixel detector arrays
poses challenges, making it difficult to achieve a million-pixel imaging array
camera akin to those in the visible light domain [10]. Therefore, although
there are well-established algorithms for depth restoration in the visible
light domain [11], applying these algorithms to the terahertz domain remains
quite challenging. At the current technological level, estimating depth in THz
imaging primarily relies on several methods: focus measurement [12], time-of-
flight [13], and heterodyne detection [14, 15]. Focus measurement techniques
determine the distance of target objects by adjusting beam focusing, but this
often requires multiple scans of the target, thus consuming considerable time
and resources. Time-of-flight utilizes the propagation time of THz waves to
estimate the distance to the object’s surface, but precision may be
compromised in complex scenes or heavily occluded scenarios. Heterodyne
detection involves mixing the transmitted signal from the object with a local
reference signal to achieve distance estimation, but this requires a complex
system and is susceptible to environmental noise and stray signals. Therefore,
exploring more efficient methods for depth information acquisition is
imperative for advancements in THz imaging technology.

Recently, DL techniques have emerged as promising and effective methods in the
field of inverse problem [16, 17, 18], primarily relying on supervised deep
learning approaches that utilize labeled experimental data [19, 20, 21].
Nevertheless, the expensive acquisition of THz images poses a challenge for
supervised training with experimental data.

In this paper, we propose a novel hybrid multi-head physics-informed neural
network (PINN) for depth estimation in THz holographic imaging. This method
incorporates multi-input U-Net [22] with a physical prior knowledge,
specifically, the angular spectrum theory [23, 24]. Hence, there is no need
for thousands of labeled data to train the network. Instead, the model is
trained with only two diffraction patterns, each with either the upper or
lower portion masked, serving as inputs. The network weights and bias factors
are then optimized through the interaction between the neural network and the
physical model, ultimately yielding a feasible model solution that satisfies
the imposed physical constraints.

##  2 Methods

Figure 1: Schematic illustration of the pipeline of proposed depth estimation
algorithm. The input of the neural network is the measured field amplitude
diffraction pattern, each with half of the object masked separately. The
output consists of estimated phase and amplitude maps, along with depth
information for the object. Subsequently, these outputs are numerically
propagated through the physical model $H$ to simulate the diffraction and
measurement processes, generating the estimated diffraction patterns. MSE
between the input patterns and estimated patterns, along with L2
regularization, serve as the loss function for optimizing the neural network
parameters.

Figure 1 displays the proposed hybrid multi-head PINN algorithm, demonstrated
using handwritten digits as an example. Note that no pre-training on large
labeled datasets is required. The network solely relies on two diffraction
patterns as input, each obtained by masking the upper or lower halves of the
object and propagating the transmitted waves over a distance $d$. The first
output of the PINN is for generating the estimated amplitude, while the second
is for the estimated phase of the imaged objects. The two output paths share
the CNN’s down-sampling layers and part of the up-sampling layers to ensure
that the same object information is analyzed, which is inspired by U-Net [25]
structure. At the last layer, these two paths separate from each other (see
Appendix A for details). The third output of the PINN predicts the value for
the depth, evaluated through a fully connected network structure. In our
scenario, the amplitude within the masked metal area theoretically becomes 0,
while the phase is randomized as long as it can reveal contours. For
simplification of computation, we set the phase within the masked metal area
to be 0 as well. Subsequently, the physical model $H$ is applied to convert
network outputs into estimated diffraction patterns. The mean squared error
(MSE) between inputs (the actual diffraction patterns) and estimated
diffraction patterns, coupled with L2 regularization, guides the network’s
optimization of weights and biases via gradient descent. In image processing,
MSE is commonly used to measure the quality of the reconstructed image
compared to the ground truth. It represents the Euclidean distance between
images and is simple and efficient to evaluate, making it suitable for large-
scale image processing tasks. Additionally, MSE is a convex function and
possesses good mathematical properties [26]. The L2 regularization [27] aims
to constrain the model’s complexity by incorporating weight decay, thereby
enhancing the model’s generalization ability and mitigating overfitting. This
loss function ensures the convergence of calculated diffraction patterns to
input patterns via the iteration process. Through iterations, simultaneous
searches for amplitude, phase, and depth information converge toward
consistent feasible solutions. Notably, since all parameters in the physical
model $H$ are considered, the proposed algorithm is universally applicable to
holographic systems, irrespective of whether they involve phase objects or
not.

As mentioned, the physical model, $H$, simulates the experimental THz imaging
process. If a planar object is illuminated by a collimated beam, the complex-
valued field amplitude immediately behind the object can be written as

$E_{0}(x,y,z=0)=A_{0}(x,y,0)e^{i\phi_{0}(x,y;0)},$ (1)

where $A_{0}$ and $\phi_{0}$ are the amplitude and the phase of the
transmission rate of the object. Over a distance $d$, diffraction reshapes the
field to [23]

$E_{d}(x,y,z=d)=\iint\hat{E}_{0}(f_{x},f_{y})\,G\,e^{i2\pi(f_{x}x+f_{y}y)}df_{x}\,df_{y},$
(2)

where $G=e^{ikd\sqrt{1-\lambda^{2}f_{x}^{2}-\lambda^{2}f_{y}^{2}}}$ is the
wave propagation function, $\lambda$ is the wavelength, $\hat{E}_{0}$ is the
spatial Fourier transform of $E_{0}$ with $f_{x}=x/\lambda d$ and
$f_{y}=y/\lambda d$ as the spatial frequencies in the $x$ and $y$ directions.
The diffraction pattern is the field’s absolute value

$A(x,y,z=d)=\left|E_{d}(x,y,z=d)\right|=H(\phi_{0},A_{0},d),$ (3)

where $H$ represents the mapping function relating the object to the
diffraction pattern $A$.

For a conventional fitting, other than with PINNs, one usually attempts to
learn a mapping function $R_{\theta}$ ($\theta$ denoting the network weights
and bias parameters) from a large number of labeled data $(A_{k},d_{k})$, the
labeled training set $S_{T}={(A_{k},d_{k}),k=1,2,...,K}$.

$R_{\theta^{*}}=\mathop{\arg\min}_{\theta}\|R_{\theta}(A_{k})-d_{k}\|^{2}\qquad\forall(d_{k},A_{k})\in
S_{T}.$ (4)

Instead, in the PINN proposed here, the loss function is formulated by
combining the physical model $H$ and the network output $R_{\theta}$ as

$loss=\|H(R_{\theta}(A(x,y;d)))-A(x,y;d)\|^{2}+\lambda\sum_{i=1}^{m}\theta^{2}$
(5)

where the diffraction pattern $A_{d}$ is the input of the NN. The
regularization term $\lambda\sum_{i=1}^{m}\theta^{2}$ is defined as the
Euclidean norm or L2 norm of the weight matrix, which is the sum of all
squared weight values of the weight matrix. Here, we set the value of
$\lambda$ to 0.01. In this way, the PINN optimization is solved with gradient
descent with the above objective

$\theta^{*}=\mathop{\arg\min}_{\theta}(loss)$ (6)

When the optimization is complete, the resulting mapping function can then be
used to reconstruct the depth with its third output:

$\tilde{d}=R_{\theta^{*}}(A(x,y;d))$ (7)

##  3 Results and discussion

We demonstrate the performance of the proposed method using data derived from
simulations (THz-emulated MNIST) and experiments, respectively.

Figure 2: Simulation results in the ideal situation. (a) represents an ideal
Gaussian distribution; (b) is a digit from the MNIST dataset; (c) and (d) are
the ground truth of the amplitude and phase of the object; (e) (f) are the
diffraction patterns of the digit diffracted at a distance $d=40\;mm$ with
either its upper or lower half masked; (g) is the loss function curve during
the training process; (h) and (i) are the predicted amplitude and phase after
training for 1000 epochs; (j) is the optimization curve of the depth.

In our simulation, we assume an ideal scenario where there exists an uniform
Gaussian beam without noise interference. Fig 2(a) depicts the ideal Gaussian
intensity profile, with its phase being uniformly distributed. Fig 2(b)
presents a digit three from the MNIST dataset, serving as an example.
Depending on the material (with dielectric information such as refractive
index) and thickness we intend to set, we can obtain the ground truth of the
object’s amplitude and phase on the object plane, as shown in Fig 2(c) and
(d). Fig 2(e) and (f) illustrate the diffraction patterns at a distance
$d=40\;mm$ when respectively occluding the upper and lower halves of the
digit, which serve as inputs to our network. The loss function, depicted in
Fig 2(g), shows a continuous decrease with increasing iterations, stabilizing
around 1.25 after 800 epochs, indicating the network has been sufficiently
optimized. At this point, the network’s outputs include the estimated
amplitudes (Fig 2(h)) and phases (Fig 2(i)). As iterations progress, the
predicted depth gradually converges to a stable value of 40.01 mm in this
example after around 200 epochs. It can be observed that in ideal conditions,
the network’s predictions for depth are highly accurate.

We then compare our proposed method with a combination method consisting of
back-propagation and shape from focus (SFF) [28], as well as the supervised DL
method in the following.

Figure 3: Diffraction patterns and comparison of our proposed PINN, supervised
NN and back-propagation with SFF at diffraction distances ranging from 40 mm
to 45 mm.

It is evident that at a minute diffraction distance difference of 1 mm, the
differences in diffraction patterns are imperceptible to the naked eye, as
shown in Fig 3(a). Hence, extracting depth information from such patterns is
nontrivial and remarkably valuable. Fig 3(b) displays the comparison of our
proposed PINN, supervised NN and back-propagation with SFF at diffraction
distances ranging from 40 mm to 45 mm. The orange line represents the results
of backpropagation combined with SFF (see Appendix B for specific sharpness
curves). Its error is significant, approaching approximately 5 mm at some
points compared to the ground truth. It is worth noting that during
backpropagation, only the amplitude is utilized as input. Setting the phase to
the same value as the amplitude yields superior results compared to setting it
to 1, hence here it is set to the amplitude value. The green line represents
the predictions of the supervised NN, which performs slightly better than
backpropagation combined with SFF. However, at 45 mm and 46 mm, its
predictions are identical. This is because in preparing the training dataset,
it is impractical to cover all possible depth positions, resulting in
potential information gaps at certain depths. Currently, our supervised
network utilizes 60,000 pairs of training datasets and 10,000 pairs of testing
datasets. Theoretically, its predictive performance can improve with further
data augmentation. The blue line illustrates the predictions of our proposed
hybrid multi-head PINN method, which agrees well with the ground truth (the
black line), significantly outperforming the other two methods. Moreover, this
method doesn’t require time-consuming pre-training like supervised learning.

We further utilize the coefficient of determination ($R^{2}$) to evaluate the
performance of depth estimation out of the three methods. The $R^{2}$ can be
calculated as follows:

$R^{2}=1-\frac{SS_{res}}{SS_{tot}}$ (8)

where $SS_{res}$ is the sum of squares of residuals, and $SS_{tot}$ is the
total sum of squares. Generally, the closer $R^{2}$ is to 1, the better the
model performs in the prediction. The $R^{2}$ values for the backpropagation
with SFF method, the supervised learning method, and our method are -1.6168,
-0.4291, and 0.9986, respectively.

Figure 4: Experimental set-up and results in realistic situation. (a) and (b)
display a schematic of the THz holographic imaging system and a photograph of
our setup. (c) (d) show the diffraction patterns we measured. (e) and (f) are
estimated amplitude and phase, respectively. (g) displays the loss curve
during the training process. (h) is the optimization curve of the depth
estimation, where the ground truth is $d=45\;mm$.

To validate the proposed methods by experiments, we performed THz measurements
at 300 GHz. The schematic diagram and experimental photographs are shown in
Fig. 4 (a) and (b). One can see that this is actually a single-beam lens-less
imaging geometry. The illuminating radiation from a 300-GHz THz source is
collimated by a focusing lens, with a 10-cm focal length and 4-inch aperture.
The lens has a homemade hyperboloidal-planar shape. The NA of the lens allows
to illuminate the objects over their full extension with a collimated Gaussian
beam, free from beam modulations caused by diffraction at the lens edge.

To acquire the diffraction pattern, we placed the THz single-pixel TeraFET
detector [29] at a distance $d=45\;mm$ from the object, on a 2D translation
stage to scan and record the diffraction patterns. The radiation enters the
detector chip from the back-side, passing a Si substrate lens with a diameter
of 4 mm[30, 31]. The recorded two diffraction patterns are shown in Fig. 4(c)
and (d). The object is a wrench with a thickness of 1 mm. The size of the
object is around $30$ mm $\times$ $60$ mm, and the scanned image area is
80$\times$80 pixels with a 1-mm2 pixel pitch.

In realistic scenarios, unlike the ideal case where a uniform Gaussian beam is
assumed as discussed earlier, there may be non-uniform distributed noise
present. We first utilize heterodyne detection to measure the amplitude and
phase of the reference THz beam, with each system requiring only one
measurement (details can be found in Appendix C). The amplitude and phase of
the reference beam serve as prior information fed into our PINN. We use the
sigmoid activation function to ensure that the amplitude and phase output to
be between 0 and 1, then multiply them by the reference to get the final
amplitude and phase predictions. Fig. 4 (g) illustrates the loss curve of our
network, showing that after 500 iterations the network’s loss stabilizes at a
value of 0.000085. It is worth noting that, unlike the ideal case, we use loss
in Eq.5 without L2 here because the noise is non-Gaussian. This also speeds up
the convergence of the network, indicating that there is only a 0.000085 mean
squared error between the actual diffraction patterns (input) and the
diffraction patterns predicted by the integration of the network and physical
model $H$’s evaluation. Fig. 4 (e) and (f) depict the network’s output,
amplitude and phase, respectively. Fig. 4 (h) shows the depth estimation from
the network as a function of the optimization steps (iterations). Clearly we
see that the network quickly predicts the correct depth information
($d=45\;mm$) and stabilizes around the ground truth value. We also employed a
combination of backpropagation and SFF along with supervised learning methods
for a comparison. However, both of these approaches showed an inability to
accurately predict actual experimental data.

Figure 5: Resolution of our proposed PINN at diffraction distances ranging
from 40 mm to 45 mm.

Given that our method can accurately predict depth information with minimal
errors in both ideal and realistic scenarios, we proceeded to determine the
resolution of this approach. According to the theory of phase detection, the
depth resolution can be quantified as[32]:

$\sigma_{d}=4\gamma\frac{\lambda d^{3}}{D^{3}}.$ (9)

When the phase difference between pixels on objects is 0, $\gamma$ equals
1.64. Here, $D$ represents the lens aperture (in our experiments, it is 80mm),
$\lambda$ denotes the wavelength, and $d$ represents the distance from the
object to the detector. Therefore, in our system, the theoretical resolution
can be determined to be around 0.82 - 1.17 mm. This resolution still requires
the simultaneous use of both the diffracted amplitude and phase of the object.
We used diffraction patterns from 40 mm to 45 mm, with intervals of 0.5 mm,
totaling 11 distances, as inputs to the proposed PINN for resolution
evaluation. To demonstrate robustness, we conducted 500 predictions for each
distance. Fig. 5 illustrates the histogram of the predicted depth results for
each distance, where the horizontal axis represents the depths and the
vertical axis represents the histogram of counts for PINN predictions over
each single depth. It can be observed that the PINN can accurately predict
diffraction patterns with roughly 0.5 mm intervals, and its resolution can
even be better than 0.5 mm.

##  4 Conclusions

In conclusion, we have developed a novel physics-informed deep learning method
for predicting the depth of planar objects using only two diffraction patterns
obtained by masking the upper and lower halves of the object. In contrast to
supervised learning methods that require large training datasets to optimize
their weights and biases, and optical approaches involving backpropagation and
SFF, our method offers more accurate and stable predictions in both ideal and
realistic scenarios. Additionally, our investigation into the resolution of
depth reconstruction reveals that our method can achieve a resolution of 0.5
mm or even better, surpassing the ideal resolution that requires additional
object phase information for depth reconstruction. Therefore, our hybrid
multi-head physics-informed neural network significantly advances the depth
estimation for planar objects in the THz holographic systems, marking a
crucial step forward in THz 3D imaging.

## Appendix A

Figure 6: Structure of the proposed hybrid multi-head PINN

The architecture of the proposed PINN as employed for the supervised DL method
is implemented using TensorFlow, an open-source deep-learning software package
[33]. We use TensorFlow version 1.9.0 and Python 3.6.5. We adopt the
estimation (Adam) [34] with a learning rate of 0.0005 to optimize the weights
and biases. We add uniformly distributed noise between 0 and $\frac{1}{30}$ to
the fixed input diffraction pattern in every optimization step to achieve
better convergence. As shown in Fig. 6, each encoding step contains two
3$\times$3 convolutional layers (blue arrows) with Relu activation function
and one 2$\times$2 max pooling layer (grey arrows), while each decoding
process contains one 3$\times$3 deconvolutional layer (dark-green arrows), one
concat layer to increase the dimension of the feature [35] (thin blue arrows),
and two convolutional layers. In addition, there are three consecutive fully
connected layers (dash black arrows) and dropout layers (green arrows) for
depth prediction. The size of the input image is 80$\times$80 pixels. The
network usually needs 1000 epochs to find an excellent estimation. This takes
10 min on a PC with a sixteen-core 3.50-GHz CPU and a 64-GB RAM, using Nvidia
GeForce RTX 3080 GPU.

## Appendix B

Figure 7: SFF results at diffraction distance of 40mm - 45mm

Figure 7 illustrates the sharpness curve using the SFF algorithm. The SFF
algorithm uses the sum of the modified Laplacian (ML) operator described in
[36] on each pixel of all the images to calculate the focus versus the optical
axis. Then, a focus measure at the pixel $(x,y,z)$ is calculated by:

$F(x,y,z)=\sum_{X=x-N}^{x+N}\sum_{Y=y-N}^{y+N}ML(X,Y,z)\quad if\,ML>T$ (10)

$N$ determines the window size to compute the focus measure. $T$ is a
threshold value to remove the contribution of the too-low contrasted region of
the images. The depth position is found by the sharpest / maximum value as
$d=z|_{max(F(x,y,z)}$). A more accurate method to find the best focus consists
of an interpolation of $F$ along $z$ axis, considering a Gaussian
distribution, also described in [36]:

$F(x,y,z)=F_{p}(x,y)e^{-\frac{1}{2}(\frac{z-\overline{d}(x,y)}{\sigma})^{2}}$
(11)

With $F_{p}$, the new maximum focusing value, and $\overline{d}$ the new best
focusing z position. $\sigma$ is the depth of field of the terahertz imaging
system, considered constant all over the image area.

## Appendix C

Figure 8: Experimental heterodyne setup and detected reference images. (a) and
(b) display a photograph and a schematic of the THz holographic imaging system
with heterodyne detection. (c) and (d) visualize the amplitude and phase
images recorded by the heterodyne detection system.

A photograph and a schematic of the experimental heterodyne setup are shown in
Fig. 8(a), (b). We use this setup to obatin the ground truths. The system [14,
37] contains two frequency-locked 300-GHz electrical multiplier chains (S1,
S2) with a 18-kHz frequency offset between them. One (S2) is used as a local
oscillator (LO) and the other (S1) provides the object-illuminating wave. The
LO wave is focused onto the detector directly by a biconvex focusing lens, L2,
with a 5-cm focal length and 2-inch aperture. The illuminating radiation is
collimated by a focusing lens, L1, with a 10-cm focal length and 4-inch
aperture. L1 is an aspheric lens. Given the required large NA, its
hyperboloidal-planar shape avoids the spherical aberrations which one would
encounter if a spherical lens would have been used. The NA of L1 has to be
large enough to guarantee that the objects are illuminated over their full
extension with a collimated Gaussian beam free of beam modulations arising by
diffraction at the edge of the lens. Such modulations introduce high-spatial-
frequency signatures which can make the learning process difficult.
Furthermore, the alignment of the lens is also crucial, since the phase is
sensitive to optical-axis displacements. Fig. 8 (c) and (d) show amplitude and
phase maps of the beam, demonstrating that the beam has an isotropic amplitude
distribution and near-constant phase over its cross-section. A numerical
evaluation of the beam profile reveals a Gaussian-like beam shape. The bluish
horizontal segments in Fig. 8(d) result from position shift errors induced by
forward- and backward-scanning of the translation stage. This effect is not
visible in the amplitude map because the amplitude is insensitive to slight
variations of the scanner position. This also proves that our algorithm has no
strict requirements for the calibration of the reference beam and can be used
in a simple way.

## References

  * [1] X.-C. Zhang, J. Xu, et al., Introduction to THz wave photonics, Vol. 29, Springer, 2010.
  * [2] A. Redo-Sanchez, X.-C. Zhang, Terahertz science and technology trends, IEEE Journal of Selected topics in quantum electronics 14 (2) (2008) 260–269.
  * [3] A. Y. Pawar, D. D. Sonawane, K. B. Erande, D. V. Derle, Terahertz technology and its applications, Drug Invention Today 5 (2) (2013) 157–163.  doi:https://doi.org/10.1016/j.dit.2013.03.009.
URL https://www.sciencedirect.com/science/article/pii/S0975761913000264

  * [4] I. Amenabar, F. Lopez, A. Mendikute, In introductory review to thz non-destructive testing of composite mater, Journal of Infrared, Millimeter, and Terahertz Waves 34 (2) (2013) 152–169.
  * [5] F. Ellrich, M. Bauer, N. Schreiner, A. Keil, T. Pfeiffer, J. Klier, S. Weber, J. Jonuscheit, F. Friederich, D. Molter, Terahertz quality inspection for automotive and aviation industries, Journal of Infrared, Millimeter, and Terahertz Waves 41 (4) (2020) 470–489.
  * [6] J. F. Federici, B. Schulkin, F. Huang, D. Gary, R. Barat, F. Oliveira, D. Zimdars, THz imaging and sensing for security applications—explosives, weapons and drugs, Semiconductor Science and Technology 20 (7) (2005) S266–S280.  doi:10.1088/0268-1242/20/7/018.
URL https://doi.org/10.1088/0268-1242/20/7/018

  * [7] F. Friederich, W. von Spiegel, M. Bauer, F. Z. Meng, M. D. Thomson, S. Boppel, A. Lisauskas, B. Hils, V. Krozer, A. Keil, T. Löffler, R. Henneberger, A. K. Huhn, G. Spickermann, P. H. Bolívar, H. G. Roskos, Thz active imaging systems with real-time capabilities, IEEE Transactions on Terahertz Science and Technology 1 (2011) 183–200.
  * [8] X. Yang, X. Zhao, K. Yang, Y. Liu, Y. Liu, W. Fu, Y. Luo, Biomedical applications of terahertz spectroscopy and imaging, Trends in Biotechnology 34 (10) (2016) 810–824.
  * [9] D. Jasteh, M. Gashinova, E. Hoare, T.-Y. Tran, N. Clarke, M. Cherniakov, Low-thz imaging radar for outdoor applications, in: 2015 16th International Radar Symposium (IRS), IEEE, 2015, pp. 203–208.
  * [10] A. Rogalski, F. Sizov, Terahertz detectors and focal plane arrays, Opto-electronics review 19 (3) (2011) 346–404.
  * [11] Z.-Q. Zhao, P. Zheng, S.-T. Xu, X. Wu, Object detection with deep learning: A review, IEEE Transactions on Neural Networks and Learning Systems 30 (11) (2019) 3212–3232.  doi:10.1109/TNNLS.2018.2876865.
  * [12] J.-B. Perraud, J.-P. Guillet, O. Redon, M. Hamdi, F. Simoens, P. Mounaix, Shape-from-focus for real-time terahertz 3d imaging, Optics letters 44 (3) (2019) 483–486.
  * [13] H. S. Kim, J. Kim, Y. H. Ahn, Terahertz nondestructive time-of-flight imaging with a large depth range, Current Optics and Photonics 6 (6) (2022) 619–626.
  * [14] H. Yuan, D. Voss, A. Lisaukas, D. Mundy, H. G. Roskos, 3D Fourier imaging based on 2D heterodyne detection at THz frequencies, APL Photonics 4 (10) (2019) 106108.  doi:10.1063/1.5116553.
  * [15] H. Yuan, A. Lisauskas, M. D. Thomson, H. G. Roskos, 600-GHz Fourier imaging based on heterodyne detection at the 2nd sub-harmonic, Optics Express 31 (24) (2023) 40856–40870.  doi:10.1364/OE.487888.
  * [16] L.-G. Pang, K. Zhou, N. Su, H. Petersen, H. Stöcker, X.-N. Wang, An equation-of-state-meter of quantum chromodynamics transition from deep learning, Nature Commun. 9 (1) (2018) 210.  arXiv:1612.04262, doi:10.1038/s41467-017-02726-3.
  * [17] S. Shi, K. Zhou, J. Zhao, S. Mukherjee, P. Zhuang, Heavy quark potential in the quark-gluon plasma: Deep neural network meets lattice quantum chromodynamics, Phys. Rev. D 105 (1) (2022) 014017.  arXiv:2105.07862, doi:10.1103/PhysRevD.105.014017.
  * [18] L. Wang, S. Shi, K. Zhou, Reconstructing spectral functions via automatic differentiation, Phys. Rev. D 106 (5) (2022) L051502.  arXiv:2111.14760, doi:10.1103/PhysRevD.106.L051502.
  * [19] L. Boominathan, M. Maniparambil, H. Gupta, R. Baburajan, K. Mitra, Phase retrieval for Fourier ptychography under varying amount of measurements, arXiv:1805.03593 (2018).
  * [20] M. Deng, S. Li, A. Goy, I. Kang, G. Barbastathis, Learning to synthesize: robust phase retrieval at low photon counts, Light: Science & Applications 9 (1) (2020) 1–16.
  * [21] G. Ju, X. Qi, H. Ma, C. Yan, Feature-based phase retrieval wavefront sensing approach using machine learning, Optics Express 26 (24) (2018) 31767–31783.
  * [22] O. Ronneberger, P. Fischer, T. Brox, U-net: Convolutional networks for biomedical image segmentation, in: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, Springer, 2015, pp. 234–241.
  * [23] J. W. Goodman, Introduction to Fourier optics, Roberts and Company publishers, 2005.
  * [24] M. J. Xiang, H. Yuan, L. X. Wang, K. Zhou, H. G. Roskos, Amplitude/phase retrieval for terahertz holography with supervised and unsupervised physics-informed deep learning, IEEE Transactions on Terahertz Science and Technology 14 (2) (2024) 208–215.  doi:10.1109/TTHZ.2024.3349482.
  * [25] L. Alzubaidi, J. Zhang, A. J. Humaidi, A. Al-Dujaili, Y. Duan, O. Al-Shamma, J. Santamaría, M. A. Fadhel, M. Al-Amidie, L. Farhan, Review of deep learning: Concepts, cnn architectures, challenges, applications, future directions, Journal of Big Data 8 (1) (2021) 1–74.
  * [26] Z. Wang, A. C. Bovik, Mean squared error: Love it or leave it? a new look at signal fidelity measures, IEEE Signal Processing Magazine 26 (1) (2009) 98–117.  doi:10.1109/MSP.2008.930649.
  * [27] C. Cortes, M. Mohri, A. Rostamizadeh, L2 regularization for learning kernels, arXiv preprint arXiv:1205.2653 (2012).
  * [28] J.-B. Perraud, J.-P. Guillet, O. Redon, M. Hamdi, F. Simoens, P. Mounaix, Shape-from-focus for real-time terahertz 3d imaging, Opt. Lett. 44 (3) (2019) 483–486.  doi:10.1364/OL.44.000483.
URL https://opg.optica.org/ol/abstract.cfm?URI=ol-44-3-483

  * [29] K. Ikamas, D. Čibiraitė, A. Lisauskas, M. Bauer, V. Krozer, H. G. Roskos, Broadband terahertz power detectors based on 90-nm silicon CMOS transistors with flat responsivity up to 2.2 THz, IEEE Electron Device Letters 39 (2018) 1413–1416.
  * [30] H. Yuan, A. Lisaukas, M. Zhang, A. Rennings, D. Erni, H. G. Roskos, Dynamic-range enhancement of heterodyne THz imaging by the use of a soft paraffin-wax substrate lens on the detector, in: 2019 Photonics & Electromagnetics Research Symposium - Fall (PIERS - Fall), 2019, pp. 2607–2611.  doi:10.1109/PIERS-Fall48861.2019.9021735.
  * [31] H. Yuan, A. Lisaukas, M. Zhang, Q. ul Islam, D. Erni, H. G. Roskos, Fourier imaging based on sub-harmonic detection at 600 GHz, in: 2022 Fifth International Workshop on Mobile Terahertz Systems (IWMTS), 2022, pp. 1–1.  doi:10.1109/IWMTS54901.2022.9832459.
  * [32] H. Yuan, Investigations on terahertz imaging based on coherent Fourier-space spectrum detection, doctoralthesis, Universitätsbibliothek Johann Christian Senckenberg (2023).  doi:10.21248/gups.72276.
  * [33] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al., $\\{$TensorFlow$\\}$: a system for $\\{$Large-Scale$\\}$ machine learning, in: 12th USENIX symposium on operating systems design and implementation (OSDI 16), 2016, pp. 265–283.
  * [34] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014).
  * [35] S. Bell, C. L. Zitnick, K. Bala, R. Girshick, Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2874–2883.
  * [36] S. K. Nayar, Y. Nakagawa, Shape from focus, IEEE Transactions on Pattern analysis and machine intelligence 16 (8) (1994) 824–831.
  * [37] H. Yuan, A. Lisaukas, H. G. Roskos, THz Fourier imaging based on sub-harmonic heterodyne detection, in: 2020 45th International Conference on Infrared, Millimeter, and Terahertz Waves (IRMMW-THz), 2020, pp. 1–1.  doi:10.1109/IRMMW-THz46771.2020.9370667.