# Analysing the Memorability of a Procedural Crime-Drama TV Series, CSI

Seán Cummins, Lorin Sweeney and Alan F. Smeaton<EMAIL_ADDRESS>Insight
SFI Research Centre for Data Analytics,
Dublin City University, Glasnevin, Dublin 9Ireland

(2022)

###### Abstract.

We investigate the memorability of a 5-season span of a popular crime-drama TV
series, CSI, through the application of a vision transformer fine-tuned on the
task of predicting video memorability. By investigating the popular genre of
crime-drama TV through the use of a detailed annotated corpus combined with
video memorability scores, we show how to extrapolate meaning from the
memorability scores generated on video shots. We perform a quantitative
analysis to relate video shot memorability to a variety of aspects of the
show. The insights we present in this paper illustrate the importance of video
memorability in applications which use multimedia in areas like education,
marketing, indexing, as well as in the case here namely TV and film
production.

Video memorability, vision transformers, CSI TV series

††journalyear: 2022††copyright: rightsretained††conference: International
Conference on Content-based Multimedia Indexing; September 14–16, 2022; Graz,
Austria††booktitle: International Conference on Content-based Multimedia
Indexing (CBMI 2022), September 14–16, 2022, Graz, Austria††doi:
10.1145/3549555.3549592††isbn: 978-1-4503-9720-9/22/09††ccs: Information
systems Video search††ccs: Information systems Multimedia information systems

##  1\. Introduction

Memorability is a subconscious measure of importance imposed on external
stimuli by our internal cognition which ultimately decides the experiences we
either carry with us into our long-term memory, or we discard. Memory is an
unequivocally unreliable phenomenon for any person and despite its importance,
it is challenging for us to influence what we will ultimately remember or
forget. This absence of any unified meta-cognitive insight brings meaning to
the study of memory, or more specifically into the study of memorability, more
generally known as the likelihood that something will be remembered or
forgotten (Sweeney et al., 2021).

We have evolved to store the important moments in life while forgetting
uninteresting or redundant information. Thus, a memorability-based model of
our experiences could allow us to filter and sort information we encounter
using a human-memorability criterion (Newman et al., 2020). Researching how
our memories take shape could have a profound impact on a plethora of research
areas such as human health or the curation of educational content (De Herrera
et al., 2020; Kiziltepe et al., 2021a).

The study of memorability may also have a commercial impact in fields such as
marketing, film or TV production, or content retrieval (Yue et al., 2021). The
latter of these use-cases serves as a particularly important application of
memorability as the volume of digital media content available grows
exponentially over time via platforms such as social networks, search engines,
and recommender systems (Shekhar et al., 2017; Siarohin et al., 2019; Newman
et al., 2020).

In this research, crime-drama exemplified in television (TV) programs such as
CSI: Crime Scene Investigation (referred to as CSI) acts as an experimental
test bed in which computational memorability as a surrogate for cognitive
processing, can be analysed. CSI is a procedural crime-drama TV series with
each episode following the same format. Viewers are introduced to a crime at
the beginning of an episode. As the episode progresses, various clues are
revealed, concluding by revealing the true perpetrator of the crime. The
success of CSI can be attributed partly to its aesthetic high production
values which many media critics comment on, and we argue its success is also
due to its ability to promote critical thinking and memory in its viewers.

Previous studies have already proven crime-drama TV series to be useful as
test cases in multiple research areas including NLP, summarisation and multi-
modal inference (Frermann et al., 2018; Papasarantopoulos et al., 2019;
Papalampidi et al., 2020). In particular, the work in (Salt, 2006) is a large-
scale anthology of Barry Salt’s essays on statistical analysis of film over
many years of investigation. Salt has single-handedly established statistical
style analysis as a research paradigm in film studies. He argues that “many
directors have sharply different styles that are easily recognized” (Salt,
2006, p. 13). Salt’s more recent work includes the application of film theory
to the creation of TV series and in (Salt, 2001) he presents a stylistic
analysis of episodes in twenty TV drama series, finding that the structure of
TV style is uniform, a similar finding to movies.

The work by Salt and others uses a variety of measures such as shot duration
and shot distribution and since then several further studies have attempted to
perform statistical analysis of TV content (Schaefer and Martinez, 2009;
Butler, 2014; Redfern, 2014; Arnold et al., 2019; Kim and Lee, 2020). Yet none
of this, or any other, forms of analysis considers the memorability of the
visual content in its analysis.

We investigate the distribution of computational memorability scores of CSI
episodes over 5 seasons by leveraging modern vision transformer architectures
(Dosovitskiy et al., 2021; Bao et al., 2021; Radford et al., 2021) fine-tuned
on the task of media memorability prediction. We interleave our predicted
memorability scores for given episodes of CSI with a detailed annotated corpus
to investigate correlation between memorability and aspects of the show
including scene and character significance.

Our research investigates whether there is a relationship between the
memorability of a shot/scene and the characters present in it, how the
memorability associated with particular characters develops over multiple
episodes and seasons, and whether the memorability of a shot/scene is
correlated with the significance of the scene. The significance of this and
the reason we study the CSI TV series is precisely because of its importance
in popular culture. Since its first broadcast in the early 2000s it has led to
the notion of the “CSI Effect” which has altered public understanding of
forensic science (Maeder and Corbett, 2015; Schweitzer and Saks, 2007) and
thus has had a societal impact. CSI has also been the subject of much
investigation in media studies because of its importance.

The remainder of the paper is structured as follows: In Section 2, we discuss
some of the seminal related work and in Section 3, we discuss the CSI dataset
used throughout this study and describe the data manipulation and augmentation
techniques we performed. Section 4 describes our experiments and discusses the
results of these experiments. Finally, in Section 5, we make some closing
remarks and discuss future directions.

It is important for the reader to note that this work does not include any
direct comparison to the work of others because there is nothing for us to
compare against. Thus a direct evaluation of this work is not possible because
of the nature of the topic. Instead, our novel findings present the kinds of
insights which can be gleaned from computational memorability analysis of
visual content and this points to the promise that such analysis can have in
areas like educational content, marketing, film and TV production, and content
retrieval.

##  2\. Related Work

###  2.1. Media Memorability

Video Memorability (VM) is a natural progression from a related research
discipline: _image memorability_ (IM). The work of (Isola et al., 2011) is
regarded as a seminal paper on IM in which it is stated that memorability is
an intrinsic property of an image, an abstract concept similar to image
aesthetics which can be computed automatically using modern image analysis
techniques (Hu and Smeaton, 2018). The use of deep-learning frameworks has led
to results at near human levels of consistency in the task of IM prediction
(Baveye et al., 2016; Fajtl et al., 2018; Akagunduz et al., 2020) partly
attributed to the development of image datasets with annotated memorability
scores (Khosla et al., 2015). In contrast, VM is still in its early stages and
only recently has there been available data sets (Cohendet et al., 2019;
Newman et al., 2020) on which to train and test VM models.

The Mediaeval Predicting Media Memorability task first took place in 2018, has
run annually since then (Cohendet et al., 2018; Constantin et al., 2019; De
Herrera et al., 2020; Kiziltepe et al., 2021a; Kiziltepe et al., 2021b) and
has been central to the development of VM. The 2021 edition saw the use of
vision transformers (Dosovitskiy et al., 2021; Bao et al., 2021; Radford et
al., 2021) across all participant submissions. The focus of work in the area
is predominantly on developing memorability prediction techniques and to date
the most applicable use of VM has been in the analysis of TV advertisements
(Mai and Schoeller, 2009; Shen et al., 2020).

###  2.2. Affective Video Content Analysis

Affective Video Content Analysis (AVCA) is a research area closely related to
VM which investigates the emotions elicited by video among its viewers (Baveye
et al., 2018). Like VM, AVCA-based research aims to generate metacognitive
insights capable of improving video content indexing and retrieval (Hau Chan
and Jones, 2005; Zhao et al., 2011). Besides this, AVCA research could, for
example, protect children from emotionally harmful media (Wang et al., 2011),
as well as create mood-dependent video recommendation (Hanjalic, 2006). Past
research in this domain has shown the modelling of some signatures of film
such as tempo (Adams et al., 2000), violence (Penet et al., 2012; Eyben et
al., 2013), and emotion (Sun et al., 2009).

###  2.3. Macabre Fascination in CSI: The Rise of the Corpse

The impact of crime-drama TV series such as CSI on modern pop-culture
stretches far beyond academic research, even resulting in the coining of a new
term: ‘The CSI Effect’ (Schweitzer and Saks, 2007; Maeder and Corbett, 2015),
mentioned earlier. Forensic autopsy portrayed in CSI acts as a softening lens
in which exploration of the dead who are usually the victims of the crime
which is the focus of the episode, is socially-palatable entertainment
(Penfold-Mounce, 2016). As CSI viewership increased from the first series in
2001, so did the number of forensic science courses available across
educational bodies111Retrieved from:
http://www.telegraph.co.uk/education/universityeducation/3243086/CSI-leads-
toincrease-in-forensic-science-courses.html. Early 2000s crime-drama TV such
as CSI sexualises the cadavers present in their series through the ever-
present “beautiful female victim” motif. The sexualised corpse is a stock
character that makes regular appearances. Our eyes are invited to linger on
the passive, beautiful, dead bodies that lay in the morgue. This phenomenon
captured the public’s attention and increased the abundance of sexualised
cadavers throughout subsequent seasons of CSI while other series such as Law &
Order began adding autopsies to their shows (Foltyn, 2008).

##  3\. The CSI Dataset

###  3.1. Source Data

We use 39 annotated episodes of CSI in our analysis, curated by Frermann et
al. (Frermann et al., 2018) and available on the Edinburgh NLP GitHub222The
repository hosting the original CSI annotation data is available at
https://github.com/EdinburghNLP/csi-corpus and the memorability scores we
computed on this corpus are available at https://github.com/scummins00/CSI-
Memorability/blob/main/data/shot_memorability.csv. The original source data
consists of two corpora, the first of which is Perpetrator Identification
(PI). PI corpus’ files are annotated at word-level for each episode. A word
can be uttered by a speaking character or is part of the screenplay. Each word
is accompanied by _case ID_ , _sentence ID_ , _speaker_ , _sentence start
time_ , _sentence end time_. We also have binary indicators such as _killer
gold_ , _suspect gold_ , and _other gold_ which indicate whether the word
uttered mentions the killer, suspect, or otherwise.

The second corpus in the original annotated data set is Screenplay
Summarisation (SS) presented at scene-level (a section of the story with its
own unique combination of setting, character, and dialogue). For a given
episode, the dataset consists of each scene annotated with _Scene ID_ ,
_screenplay_ , as well as the scene _aspect_. Aspect is a categorical feature
indicating the scene’s significance and can be one or many of the following:
Crime Scene, Victim, Death Cause, Evidence, Perpetrator, Motive, or None.

Each of the 39 episodes was also available to us in video format by scraping
directly from the DVDs purchased from Amazon, as well as an extra 3 episode
video files.

###  3.2. Data Preprocessing

####  3.2.1. Data Manipulation:

We aggregated each of our PI episode datasets from word-level to sentence-
level, combining the _killer gold_ , _suspect gold_ , and _other gold_ binary
indicators previously mentioned into a single categorical column. We then
disaggregated our SS corpora from scene-level to sentence-level. With
equivalent datasets of the same configuration, we then merged them and Table 1
shows a section of one of our augmented and merged datasets for S1.E8.

Table 1. Sample augmented dataset from Season 1, Ep. 8. Original data from (Frermann et al., 2018). caseID | sentID | speaker | type mentioned | start | end | sentence | aspect
---|---|---|---|---|---|---|---
1 | 6 | Grissom | other | 00:36.5 | 00:41.5 | where’s the girl? | Victim
1 | 7 | Officer | other | 00:41.5 | 00:46.6 | she’s down this hall | Victim
1 | 8 | None | None | 00:46.6 | 00:51.7 | Grissom turns the corner | Crime scene
1 | 9 | None | None | 00:51.7 | 00:56.8 | the officer signals Grissom inside | Crime scene

####  3.2.2. Data Augmentation:

Features available from the annotation are rich and allow us to analyse data
in a variety of ways. However, we noted two shortcomings: (i) the data
provides little context regarding suspects throughout an episode other than
the type mentioned feature and (ii) the aspect feature does not allow us to
identify scenes involving suspects. Also, a portion of scenes involving the
killer are not labelled with the ‘Perpetrator’ aspect. We extended the aspect
feature of any sentence in a scene in which the perpetrator or a suspect
speaks so as to have the values of ‘Perpetrator’ or ‘Suspect’, respectively.

###  3.3. Pre-processing of Video Files

We subdivided our clips into shots which allows for more specific and fine-
grained indexing, in turn reducing the amount of video we needed to process
per analysis. We used a neural shot-boundary-detection (SBD) framework,
_TransNet V2_ ,333TransNet V2 is an SBD framework:
https://github.com/soCzech/TransNetV2. described in (Souček and Lokoč, 2020),
to segment episodes into individual shots.

###  3.4. Data Generation

The use of vision transformers in predicting memorability scores is shown in
(Constantin and Ionescu, 2021; Kleinlein et al., 2021). Our vision transformer
architecture is the CLIP (Radford et al., 2021) pre-trained image encoder. We
use this encoder to train a Bayesian Ridge Regressor (BRR) on the Memento10k
dataset (Newman et al., 2020). BRR was used previously by (Sweeney et al.,
2021) for computing memorability scores based on extracted features. For each
training sample, the encoder extracts features from frames sampled at a rate
of 3 frames per second. The BRR is then trained on the features and
memorability score pairs. To compute the memorability of unseen CSI clips, we
first extract representative frames using a temporal-based frame extraction
mechanism. These are used as input to the vision transformer architecture to
extract features. Our BRR model then produces a memorability score based on
these features. This framework is shown in Figure 1.

Figure 1. Our memorability prediction framework, extracting representative
frames and computing a memorability score for each shot.

##  4\. Methodology and Experimental Results

We now present the distribution of memorability scores across each episode,
relating it to its metadata from annotations taken from (Frermann et al.,
2018).

For each episode, we plotted the memorability scores associated with shots in
chronological order to create a time-series signal representing the intrinsic
memorability of the episode. We then annotated each episode’s signal according
to various metadata including (i) the scene aspect, and (ii) the character
speaking (including None). We used Laplacian smoothing described in Equation 1
to remove the jaggedness present in raw memorability scores

(1) $\hat{x}_{i}=\frac{1}{N}\sum_{j=1}^{N}\hat{x}_{j}$

where $N$ is the size of the smoothing window. We generated signals for each
episode using a range of smoothing window sizes from 15 to 305 shots.

To compliment the shot memorability score distributions we gathered
memorability scores across the distribution of characters, and of aspects. By
aggregating the overall speaking time of the main characters, we identify the
lead as well as the secondary characters. We performed a similar analysis to
understand the scene count per aspect value. Our results are shown in Figure
2.

Figure 2. (a) The time (mins) spent speaking by each main cast member. (b) The
number of scenes in which each aspect value is present.

We investigated the memorability distribution associated with the series
characters and aspects and this is presented in Figure 3 parts (a) and (b) in
each individual season as well as across all 5 seasons. We investigated: i)the
main character speaking, and (ii) the scene aspect. The lack of variation
across Figure 3 (a) and (b) and the relatively high memorability scores is an
artefact of the Memento10k memorability dataset (Newman et al., 2020) used to
train the memorability model we used where we rarely see a video segment given
a memorability score below 0.7. Instead we are interested in repeating
patterns e.g. which parts of shows consistently result in higher or lower
memorability scores and what these patterns might reveal to us about the
series.

Figure 3. Graphs display results from Seasons 1 to 5 and then results for all
seasons combined. (a) displays computed memorability scores for shots with the
main characters, while (b) displays computed memorability scores for shots
with associated with aspect values.

In Figure 3 (a), we see that ordering the cast in terms of memorability over
all 5 seasons is almost identical to that seen in Figure 2 (a). Catherine has
less speaking time than Grissom, but is consistently considered more
memorable. With the exception of Season 3, Nick is simultaneously the least-
memorable character and the character with the least time spent speaking. We
observe a somewhat positive linear relationship between a character’s
importance (in terms of screen-time), and their memorability.

In Figure 2 (b), the proportion of scenes in which each aspect value appears
correlates with the purpose in which that aspect serves. For example, ‘Motive’
appears least across all seasons as only a small portion near the end of each
episode is dedicated to revealing a perpetrator’s motive. Similarly, ‘Crime
scene’ and ‘Victim’ scenes usually occur near the beginning of an episode.
Despite this, as shown in Figure 3 (b), ‘Motive’ scenes are considered the
most memorable across all 5 seasons. In contrast, both ‘Crime scene’ and
‘Victim’ scenes are considered the least memorable.

Scenes described with the ‘Death Cause’ aspect value are associated with
autopsy scenes in which viewers are shown flashbacks to the crime committed,
as well as the cadaver of a usually youthful, attractive victim. We discussed
the importance of these scenes and their pivotal role in the development of TV
crime series in Section 2.3. In Figure 3 (b), the intrinsic memorability of
this scene type increases as the seasons progress. These pedagogic, grotesque,
scenes capture the audience’s fascination because of their brash autoptic
vision (Tait, 2006) and are a staple of the crime-drama TV genre and
contribute to its popular success. The visual memorability of these scenes
correlates with the success of this scene-type.

##  5\. Conclusions and Future Work

Video memorability is a recent computational tool for analysis of visual media
which is more abstract and higher-level than raw content analysis like object
detection or action classification. In this paper we have used the computation
of video memorability at video shot level to analyse a popular procedural
crime-drama TV series revealing insights not previously visible.

Our investigation creates a mélange of intertwined arguments ranging from more
statistically-driven interpretations of film such as cinemetrics, to artistic
studies on the relationship between a series’ popularity and its most
memorable moments. We hypothesise that scene and character importance is
related to the memorability of a scene, and show that correlation exists
between factors of the show not previously visible. We show that these
correlations are not random, but instead have complex interpretation. The
memorability scores we generated uncovered patterns within the early seasons
of CSI, highlighting the significance and memorability of an episode’s finale
in comparison to other scenes. Our memorability scores for lead cast members
heavily correlate with the characters’ importance, considering on-screen time
as importance. The work here is one of the first efforts in the application of
memorability scores beyond benchmarking environments.

Our first area for future work is to increase the data size by including more
episodes of CSI, as well as episodes from other shows such as CSI: Miami or
other ‘Whodunnit-styled’ series such as Law & Order. In our study, we used a
temporal-based frame extraction method to extract representative frames from
shots. This could be enhanced via Representative Frame Extraction techniques
(Shruthi and Priyamvada, 2017), which are capable of automatically selecting
the most representative frames from video content.

It is clear that we have developed an importance-proxy by analysing various
metadata from CSI. Ideally, we would define a user task in which participants’
genuine memorability of aspects of a crime-drama series can be examined. This
gives a real measure of importance with respect to the show, rather than an
importance proxy.

The novel findings we have presented here reveal insights from computational
memorability scores for video shots of a TV crime series. As mentioned
earlier, this is useful and interesting from a film or TV studies perspective
but it also shows the potential that computational memorability has in other
areas. We could now analyse or even edit videos to be used in online education
so as to maximise the memorability of key moments. We could structure video
content used in marketing and advertising, or as we have seen here in film and
TV production, so that key messages are delivered with maximised memorability.
Future work in these and other areas may indeed exploit computational
memorability as a criterion for composing video content.

###### Acknowledgements.

This publication has emanated from research partly supported by Science
Foundation Ireland (SFI) under Grant Number SFI/12/RC/ 2289_P2 (Insight SFI
Research Centre for Data Analytics).

## References

  * (1)
  * Adams et al. (2000) B. Adams, C. Dorai, and S. Venkatesh. 2000.  Novel approach to determining tempo and dramatic story sections in motion pictures. In _Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101)_ , Vol. 2. IEEE, Vancouver, BC, Canada, 283–286 vol.2.  https://doi.org/10.1109/ICIP.2000.899358
  * Akagunduz et al. (2020) Erdem Akagunduz, Adrian G. Bors, and Karla K. Evans. 2020\.  Defining Image Memorability Using the Visual Memory Schema.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 42, 9 (Sep 2020), 2165–2178.  https://doi.org/10.1109/TPAMI.2019.2914392
  * Arnold et al. (2019) Taylor Arnold, Lauren Tilton, and Annie Berke. 2019\.  Visual Style in Two Network Era Sitcoms.  _Journal of Cultural Analytics_ 4, 2 (Jul 2019), 11045.  https://doi.org/10.22148/16.043
  * Bao et al. (2021) Hangbo Bao, Li Dong, and Furu Wei. 2021.  BEiT: BERT Pre-Training of Image Transformers.  _arXiv:2106.08254 [cs]_ (Jun 2021).  http://arxiv.org/abs/2106.08254 arXiv: 2106.08254.
  * Baveye et al. (2018) Yoann Baveye, Christel Chamaret, Emmanuel Dellandréa, and Liming Chen. 2018.  Affective Video Content Analysis: A Multidisciplinary Insight.  _IEEE Transactions on Affective Computing_ 9, 4 (Oct 2018), 396–409.  https://doi.org/10.1109/TAFFC.2017.2661284
  * Baveye et al. (2016) Yoann Baveye, Romain Cohendet, Matthieu Perreira Da Silva, and Patrick Le Callet. 2016.  Deep Learning for Image Memorability Prediction: the Emotional Bias. In _Proceedings of the 24th ACM international conference on Multimedia_ _(MM ’16)_. Association for Computing Machinery, New York, NY, USA, 491–495.  https://doi.org/10.1145/2964284.2967269
  * Butler (2014) Jeremy Butler. 2014\.  Statistical Analysis of Television Style: What Can Numbers Tell Us about TV Editing?  _Cinema Journal_ 54, 1 (2014), 25–44.
  * Cohendet et al. (2019) Romain Cohendet, Claire-Helene Demarty, Ngoc Duong, and Martin Engilberge. 2019.  VideoMem: Constructing, Analyzing, Predicting Short-Term and Long-Term Video Memorability. In _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_. IEEE, Seoul, Korea (South), 2531–2540.  https://doi.org/10.1109/ICCV.2019.00262
  * Cohendet et al. (2018) Romain Cohendet, Claire-Hélène Demarty, Ngoc Duong, Mats Sjöberg, Bogdan Ionescu, Thanh-Toan Do, and France Rennes. 2018\.  MediaEval 2018: Predicting Media Memorability Task.  _arXiv:1807.01052 [cs]_ (Jul 2018).  http://arxiv.org/abs/1807.01052 arXiv: 1807.01052.
  * Constantin and Ionescu (2021) Mihai Gabriel Constantin and Bogdan Ionescu. 2021.  Using Vision Transformers and Memorable Moments for the Prediction of Video Memorability.  (2021), 3.
  * Constantin et al. (2019) Mihai Gabriel Constantin, Bogdan Ionescu, Claire-Hélène Demarty, Ngoc Q K Duong, Xavier Alameda-Pineda, and Mats Sjöberg. 2019.  The Predicting Media Memorability Task at MediaEval 2019\.  (Dec 2019), 3.
  * De Herrera et al. (2020) Alba García Seco De Herrera, Rukiye Savran Kiziltepe, Jon Chamberlain, Mihai Gabriel Constantin, Claire-Hélène Demarty, Faiyaz Doctor, Bogdan Ionescu, and Alan F. Smeaton. 2020.  Overview of MediaEval 2020 Predicting Media Memorability Task: What Makes a Video Memorable?  _arXiv:2012.15650 [cs]_ (Dec 2020).  http://arxiv.org/abs/2012.15650 arXiv: 2012.15650.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021\.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  _arXiv:2010.11929 [cs]_ (Jun 2021).  http://arxiv.org/abs/2010.11929 arXiv: 2010.11929.
  * Eyben et al. (2013) Florian Eyben, Felix Weninger, Nicolas Lehment, Björn Schuller, and Gerhard Rigoll. 2013\.  Affective Video Retrieval: Violence Detection in Hollywood Movies by Large-Scale Segmental Feature Extraction.  _PLOS ONE_ 8, 12 (Dec 2013), e78506.  https://doi.org/10.1371/journal.pone.0078506
  * Fajtl et al. (2018) Jiri Fajtl, Vasileios Argyriou, Dorothy Monekosso, and Paolo Remagnino. 2018.  AMNet: Memorability Estimation with Attention. In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_. IEEE, Salt Lake City, UT, USA, 6363–6372.  https://doi.org/10.1109/CVPR.2018.00666
  * Foltyn (2008) Jacque Lynn Foltyn. 2008\.  Dead famous and dead sexy: Popular culture, forensics, and the rise of the corpse.  _Mortality_ 13, 2 (May 2008), 153–173.  https://doi.org/10.1080/13576270801954468
  * Frermann et al. (2018) Lea Frermann, Shay B. Cohen, and Mirella Lapata. 2018\.  Whodunnit? Crime Drama as a Case for Natural Language Understanding.  _Transactions of the Association for Computational Linguistics_ 6 (Dec 2018), 1–15.  https://doi.org/10.1162/tacl_a_00001
  * Hanjalic (2006) A. Hanjalic. 2006\.  Extracting moods from pictures and sounds: towards truly personalized TV.  _IEEE Signal Processing Magazine_ 23, 2 (Mar 2006), 90–100.  https://doi.org/10.1109/MSP.2006.1621452
  * Hau Chan and Jones (2005) Ching Hau Chan and Gareth J. F. Jones. 2005.  Affect-based indexing and retrieval of films. In _Proceedings of the 13th Annual ACM International Conference on Multimedia_. Association for Computing Machinery, Singapore, 427–430.  http://portal.acm.org/ft_gateway.cfm?id=1101243&type=pdf&CFID=19879361&CFTOKEN=18605329
  * Hu and Smeaton (2018) Feiyan Hu and Alan F. Smeaton. 2018.  Image Aesthetics and Content in Selecting Memorable Keyframes from Lifelogs. In _MultiMedia Modeling_ , Klaus Schoeffmann, Thanarat H. Chalidabhongse, Chong Wah Ngo, Supavadee Aramvith, Noel E. O’Connor, Yo-Sung Ho, Moncef Gabbouj, and Ahmed Elgammal (Eds.). Springer International Publishing, Bangkok, Thailand, 608–619.
  * Isola et al. (2011) Phillip Isola, Devi Parikh, Antonio Torralba, and Aude Oliva. 2011.  Understanding the intrinsic memorability of images. In _Proceedings of the 24th International Conference on Neural Information Processing Systems_ _(NIPS’11)_. Curran Associates Inc., Red Hook, NY, USA, 2429–2437.
  * Khosla et al. (2015) Aditya Khosla, Akhil S. Raju, Antonio Torralba, and Aude Oliva. 2015.  Understanding and Predicting Image Memorability at a Large Scale. In _2015 IEEE International Conference on Computer Vision (ICCV)_. 2390–2398.  https://doi.org/10.1109/ICCV.2015.275
  * Kim and Lee (2020) Yunhwan Kim and Sunmi Lee. 2020.  Exploration of the Characteristics of Emotion Distribution in Korean TV Series: Common Pattern and Statistical Complexity.  _IEEE Access_ 8 (2020), 69438–69447.  https://doi.org/10.1109/ACCESS.2020.2985673
  * Kiziltepe et al. (2021a) Rukiye Savran Kiziltepe, Mihai Gabriel Constantin, Claire-Helene Demarty, Graham Healy, Camilo Fosco, Alba Garcia Seco de Herrera, Sebastian Halder, Bogdan Ionescu, Ana Matran-Fernandez, Alan F. Smeaton, and Lorin Sweeney. 2021a.  Overview of The MediaEval 2021 Predicting Media Memorability Task.  _arXiv:2112.05982 [cs]_ (Dec 2021).  http://arxiv.org/abs/2112.05982 arXiv: 2112.05982.
  * Kiziltepe et al. (2021b) Rukiye Savran Kiziltepe, Lorin Sweeney, Mihai Gabriel Constantin, Faiyaz Doctor, Alba García Seco de Herrera, Claire-Héléne Demarty, Graham Healy, Bogdan Ionescu, and Alan F Smeaton. 2021b.  An annotated video dataset for computing video memorability.  _Data in Brief_ 39 (2021), 107671.
  * Kleinlein et al. (2021) Ricardo Kleinlein, Cristina Luna-Jiménez, and Fernando Fernández-Martínez. 2021.  THAU-UPM at MediaEval 2021: From Video Semantics To Memorability Using Pretrained Transformers.  (2021), 3.
  * Maeder and Corbett (2015) Evelyn M Maeder and Richard Corbett. 2015.  Beyond frequency: Perceived realism and the CSI effect.  _Canadian Journal of Criminology and Criminal Justice_ 57, 1 (2015), 83–114.
  * Mai and Schoeller (2009) Li-Wei Mai and Georgia Schoeller. 2009.  Emotions, attitudes and memorability associated with TV commercials.  _Journal of Targeting, Measurement and Analysis for Marketing_ 17, 1 (Mar 2009), 55–63.  https://doi.org/10.1057/jt.2009.1
  * Newman et al. (2020) Anelise Newman, Camilo Fosco, Vincent Casser, Allen Lee, Barry McNamara, and Aude Oliva. 2020\.  Multimodal Memorability: Modeling Effects of Semantics and Decay on Video Memorability. In _Computer Vision – ECCV 2020_ _(Lecture Notes in Computer Science)_ , Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 223–240.  https://doi.org/10.1007/978-3-030-58517-4_14
  * Papalampidi et al. (2020) Pinelopi Papalampidi, Frank Keller, Lea Frermann, and Mirella Lapata. 2020.  Screenplay Summarization Using Latent Narrative Structure. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 1920–1933.  https://doi.org/10.18653/v1/2020.acl-main.174
  * Papasarantopoulos et al. (2019) Nikos Papasarantopoulos, Lea Frermann, Mirella Lapata, and Shay B. Cohen. 2019.  Partners in Crime: Multi-view Sequential Inference for Movie Understanding. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, Hong Kong, China, 2057–2067.  https://doi.org/10.18653/v1/D19-1212
  * Penet et al. (2012) Cédric Penet, Claire-Hélène Demarty, Guillaume Gravier, and Patrick Gros. 2012\.  Multimodal information fusion and temporal integration for violence detection in movies. In _2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2393–2396.  https://doi.org/10.1109/ICASSP.2012.6288397
  * Penfold-Mounce (2016) Ruth Penfold-Mounce. 2016\.  Corpses, popular culture and forensic science: public obsession with death.  _Mortality_ 21, 1 (Jan 2016), 19–35.  https://doi.org/10.1080/13576275.2015.1026887
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021\.  Learning Transferable Visual Models From Natural Language Supervision.  _arXiv:2103.00020 [cs]_ (Feb 2021).  http://arxiv.org/abs/2103.00020 arXiv: 2103.00020.
  * Redfern (2014) Nick Redfern. 2014\.  The Structure of ITV News Bulletins.  _International Journal of Communication_ 8, 00 (Jun 2014), 22.
  * Salt (2001) Barry Salt. 2001\.  Practical Film Theory and its Application to TV Series Dramas.  _Journal of Media Practice_ 2, 2 (2001), 98–113.  https://doi.org/10.1386/jmpr.2.2.98 arXiv:https://doi.org/10.1386/jmpr.2.2.98
  * Salt (2006) Barry Salt. 2006\.  _Moving into Pictures: More on film history, style, and analysis_.  London: Starword.
  * Schaefer and Martinez (2009) Richard J. Schaefer and Tony J. Martinez. 2009.  Trends in Network News Editing Strategies From 1969 Through 2005.  _Journal of Broadcasting & Electronic Media_ 53, 3 (Aug 2009), 347–364.  https://doi.org/10.1080/08838150903102600
  * Schweitzer and Saks (2007) N.J. Schweitzer and Michael J. Saks. 2007.  The CSI Effect: Popular Fiction About Forensic Science Affects the Publics Expectations About Real Forensic Science.  _Jurimetrics_ 47, 3 (2007), 357–364.
  * Shekhar et al. (2017) Sumit Shekhar, Dhruv Singal, Harvineet Singh, Manav Kedia, and Akhil Shetty. 2017.  Show and Recall: Learning What Makes Videos Memorable. In _2017 IEEE International Conference on Computer Vision Workshops (ICCVW)_. 2730–2739.  https://doi.org/10.1109/ICCVW.2017.321
  * Shen et al. (2020) Wangbing Shen, Zongying Liu, Linden J. Ball, Taozhen Huang, Yuan Yuan, Haiping Bai, and Meifeng Hua. 2020.  Easy to Remember, Easy to Forget? The Memorability of Creative Advertisements.  _Creativity Research Journal_ 32, 3 (Jul 2020), 313–322.  https://doi.org/10.1080/10400419.2020.1821568
  * Shruthi and Priyamvada (2017) N Shruthi and S Priyamvada. 2017.  Dominant frame extraction for video indexing. In _2017 2nd IEEE International Conference on Recent Trends in Electronics, Information Communication Technology (RTEICT)_. 1799–1803.  https://doi.org/10.1109/RTEICT.2017.8256909
  * Siarohin et al. (2019) Aliaksandr Siarohin, Gloria Zen, Cveta Majtanovic, Xavier Alameda-Pineda, Elisa Ricci, and Nicu Sebe. 2019.  Increasing Image Memorability with Neural Style Transfer.  _ACM Transactions on Multimedia Computing, Communications, and Applications_ 15, 2 (Jun 2019), 1–22.  https://doi.org/10.1145/3311781
  * Souček and Lokoč (2020) Tomáš Souček and Jakub Lokoč. 2020.  TransNet V2: An effective deep network architecture for fast shot transition detection.  _arXiv:2008.04838 [cs]_ (Aug 2020).  http://arxiv.org/abs/2008.04838 arXiv: 2008.04838.
  * Sun et al. (2009) Kai Sun, Junqing Yu, Yue Huang, and Xiaoqiang Hu. 2009\.  An improved valence-arousal emotion space for video affective content representation and recognition. In _2009 IEEE International Conference on Multimedia and Expo_. 566–569.  https://doi.org/10.1109/ICME.2009.5202559
  * Sweeney et al. (2021) Lorin Sweeney, Graham Healy, and Alan F. Smeaton. 2021\.  The Influence of Audio on Video Memorability with an Audio Gestalt Regulated Video Memorability System. In _2021 International Conference on Content-Based Multimedia Indexing (CBMI)_. 1–6.  https://doi.org/10.1109/CBMI50038.2021.9461903
  * Tait (2006) Sue Tait. 2006.  Autoptic vision and the necrophilic imaginary in CSI.  _International Journal of Cultural Studies_ 9, 1 (Mar 2006), 45–62.  https://doi.org/10.1177/1367877906061164
  * Wang et al. (2011) Jianchao Wang, Bing Li, Weiming Hu, and Ou Wu. 2011\.  Horror video scene recognition via Multiple-Instance learning. In _2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. 1325–1328.  https://doi.org/10.1109/ICASSP.2011.5946656
  * Yue et al. (2021) Fumei Yue, Jing Li, and Jiande Sun. 2021.  Insights of Feature Fusion for Video Memorability Prediction. In _Digital TV and Wireless Multimedia Communication_ , Guangtao Zhai, Jun Zhou, Hua Yang, Ping An, and Xiaokang Yang (Eds.). Springer, Singapore, 239–248.  https://doi.org/10.1007/978-981-16-1194-0_21
  * Zhao et al. (2011) Sicheng Zhao, Hongxun Yao, Xiaoshuai Sun, Pengfei Xu, Xianming Liu, and Rongrong Ji. 2011\.  Video indexing and recommendation based on affective analysis of viewers. In _Proceedings of the 19th ACM international conference on Multimedia_ _(MM ’11)_. Association for Computing Machinery, New York, NY, USA, 1473–1476.  https://doi.org/10.1145/2072298.2072043


# Deep Monte Carlo Quantile Regression for Quantifying Aleatoric Uncertainty
in Physics-informed Temperature Field Reconstruction

1st Xiaohu Zheng  College of Aerospace Science and Engineering
National University of Defense Technology
Changsha, China
<EMAIL_ADDRESS>2rd Wen Yao  Defense Innovation Institute
Chinese Academy of Military Science
Beijing, China
<EMAIL_ADDRESS>3nd Zhiqiang Gong  Defense Innovation Institute
Chinese Academy of Military Science
Beijing, China
<EMAIL_ADDRESS>4th Yunyang Zhang  Defense Innovation Institute
Chinese Academy of Military Science
Beijing, China
<EMAIL_ADDRESS>5th Xiaoyu Zhao  Defense Innovation Institute
Chinese Academy of Military Science
Beijing, China
<EMAIL_ADDRESS>6th Tingsong Jiang  Defense Innovation Institute
Chinese Academy of Military Science
Beijing, China
<EMAIL_ADDRESS>
###### Abstract

For the temperature field reconstruction (TFR), a complex image-to-image
regression problem, the convolutional neural network (CNN) is a powerful
surrogate model due to the convolutional layer’s good image feature extraction
ability. However, a lot of labeled data is needed to train CNN, and the common
CNN can not quantify the aleatoric uncertainty caused by data noise. In actual
engineering, the noiseless and labeled training data is hardly obtained for
the TFR. To solve these two problems, this paper proposes a deep Monte Carlo
quantile regression (Deep MC-QR) method for reconstructing the temperature
field and quantifying aleatoric uncertainty caused by data noise. On the one
hand, the Deep MC-QR method uses physical knowledge to guide the training of
CNN. Thereby, the Deep MC-QR method can reconstruct an accurate TFR surrogate
model without any labeled training data. On the other hand, the Deep MC-QR
method constructs a quantile level image for each input in each training
epoch. Then, the trained CNN model can quantify aleatoric uncertainty by
quantile level image sampling during the prediction stage. Finally, the
effectiveness of the proposed Deep MC-QR method is validated by many
experiments, and the influence of data noise on TFR is analyzed.

###### Index Terms:

Physics-informed, convolutional neural network, aleatoric uncertainty,
temperature field reconstruction, quantile regression

##  I Introduction

Heat monitoring analysis plays an important role in many expensive engineering
systems [1, 2, 3], such as airplanes, satellites, rockets, etc. If the system
overheats, it can lead to system failure or even serious disaster. Generally,
the heat monitoring analysis is performed using the temperatures measured by
many sensors. However, a limited number of temperature monitoring sensors can
not provide detailed temperature information at any location of the system,
which will result in the heat monitoring analysis results being inaccurate.
Therefore, it is essential to reconstruct the temperature field of the whole
system based on the temperature measured by sensors.

In recent years, CNN has been applied in many engineering problems [4, 5, 6]
due to its powerful image feature extraction ability. For the TFR, it is a
complex image-to-image regression problem. Thus, this paper chooses CNN to
reconstruct the temperature field using sensor monitoring temperatures as
inputs. It is important to notice that engineers hardly obtain each input’s
label, i.e., temperature field. However, the training of CNN usually requires
a large amount of labeled training data. In order to solve the problem of
insufficient labeled data, the physics-informed CNN is used to bearing fault
detection [7], partial differential equation solution [8, 9, 10], topology
optimization [11]. For the TFR, the steady-state temperature field meets the
Laplace equation [12, 5] and some boundary conditions. Thus, this paper uses
physical knowledge to guide the training of CNN.

The training of CNN is essentially extracting model parameters from training
data. The trained model is then used to make predictions. Generally, the
training data may include data noise which will result in aleatoric
uncertainty [13] in the prediction results. Unlike epistemic uncertainty [14]
can be eliminated by increasing the number of training data, aleatoric
uncertainty is inherent to the training data with noise. In recent years, many
methods [15, 16, 17, 18, 19] have been studied to quantify aleatoric
uncertainty. Given the Gaussian distribution input hypothesis, one common
approach of aleatoric uncertainty quantification is that one output of the
neural network is used to estimate the conditional variance of the prediction
by maximizing likelihood estimation [15, 16]. The obvious disadvantage of this
approach is that it can only quantify Gaussian random noise. Another approach,
simultaneous quantile regression (SQR)[18], is to quantify aleatoric
uncertainty based on quantile regression [20, 21]. In the model training
process, this approach randomly samples the quantile level between 0 and 1 for
each training data and mini-batch. Compared with the previous approach, SQR
can estimate the aleatoric uncertainty in data without Gaussian distribution
input assumptions or anything else. Inspired by this approach, this paper
proposes a Deep MC-QR method for reconstructing the temperature field and
quantifying aleatoric uncertainty caused by data noise.

The main contributions of this paper include the following two points:

  * •

By physical knowledge to guide the training of CNN, the proposed Deep MC-QR
method can reconstruct an accurate TFR surrogate model without any labeled
training data;

  * •

By constructing a quantile level image for each monitoring point temperature
image, the proposed Deep MC-QR method can quantify aleatoric uncertainty.

##  II Related Work

###  II-A Quantile Regression

Suppose that there is a stochastic system $\bm{Y}=f\left(\bm{X}\right)$ with
the random input variable $\bm{X}$. The cumulative distribution function of
the output $\bm{Y}$ taking the value $\bm{y}$ is
$F_{\bm{Y}|\bm{X}=\bm{x}}\left(\bm{y}|\bm{x}\right)=P\left(\bm{Y}\leq\bm{y}\right)$.
Thus, the $\tau$ quantile $Q_{\tau}\left(\bm{Y}|\bm{X}=\bm{x}\right)$ of the
output $\bm{Y}$ is

$Q_{\tau}\left(\bm{Y}|\bm{X}=\bm{x}\right)=F_{\bm{Y}|\bm{X}=\bm{x}}^{-1}\left(\tau\right),$
(1)

where $\tau$ ($0\leq\tau\leq 1$) is the quantile level. The quantile
regression constructs a surrogate model $\hat{\bm{Y}}=\hat{f}_{\tau}(\bm{X})$
to approximate the stochastic system $\bm{Y}=f\left(\bm{X}\right)$. Given
$\mathcal{K}$ training data
$\left\\{\left(\bm{x}_{k},\bm{y}_{k}\right)|k=1,2,\cdots,\mathcal{K}\right\\}$,
the surrogate model $\hat{\bm{Y}}=\hat{f}_{\tau}(\bm{X})$ is trained by
minimizing the pinball loss function
$\mathcal{L}\left[{{w}_{i}},{{{\hat{f}}}_{\tau}}\left(\bm{v}_{i}\right)\right]$
[21, 18], i.e.,

$\displaystyle\mathcal{L}\left[{{\bm{y}}_{k}},{{{\hat{f}}}_{\tau}}\left(\bm{x}_{k}\right)\right]=\frac{1}{\mathcal{K}}\sum\limits_{k=1}^{\mathcal{K}}{{{\ell}_{\tau}}\left[{{\bm{y}}_{k}},{{{\hat{f}}}_{\tau}}\left({{\bm{x}}_{k}}\right)\right]},$
(2)
$\displaystyle{{\ell}_{\tau}}\left[{{\bm{y}}_{k}},{{{\hat{f}}}_{\tau}}\left({{\bm{x}}_{k}}\right)\right]=\left\\{\begin{matrix}\tau\left({{\bm{y}}_{k}}-{{{\hat{\bm{y}}}}_{k}}\right),&{{\bm{y}}_{k}}\geq{{{\hat{\bm{y}}}}_{k}}\\\
\left(1-\tau\right)\left({{{\hat{\bm{y}}}}_{k}}-{{\bm{y}}_{k}}\right),&{{\bm{y}}_{k}}<{{{\hat{\bm{y}}}}_{k}}\\\
\end{matrix}\right.,$

where ${{\bm{\hat{y}}}_{k}}={{\hat{f}}_{\tau}}\left({{\bm{x}}_{k}}\right)$ is
the prediction corresponding to the input $\bm{x}_{k}$.

###  II-B Temperature Field Reconstruction

As shown in Fig.1, there is an green rectangular area $\Omega$ with length $W$
and width $H$, where the grey areas ${\Omega}_{HS}$ of circles, capsules and
rectangles are heat sources. The yellow rectangle with width $\delta$ is the
position of the heat sink. The diagonal shaded area around $\Omega$ is
adiabatic. Besides, the red points are the positions ${\Omega}_{MP}$ of the
temperature monitoring sensors. Supposed that there are $N_{MP}$ temperature
monitoring sensors in the rectangular area $\Omega$.

Figure 1: A rectangular area heat source system.

For this green rectangular area $\Omega$, its steady-state temperature field
$\bm{T}$ meets the Laplace equation [5, 12], i.e.,

$\frac{{{\partial}^{2}}\bm{T}\left(u,v\right)}{\partial{{u}^{2}}}+\frac{{{\partial}^{2}}\bm{T}\left(u,v\right)}{\partial{{v}^{2}}}+\bm{\varphi}\left(u,v\right)=0,\quad\left(u,v\right)\in\Omega,$
(3)

and the boundary condition

$\bm{T}\left(u,v\right)={{T}_{0}},\quad\left(u,v\right)\in{{\Omega}_{BC}},$
(4)

where $\left(u,v\right)$ denotes the position coordinate, $T_{0}$ is a
constant temperature, $\bm{\varphi}$ is the heat source intensity distribution
depending on the power of heat source, and ${{\Omega}_{BC}}$ denotes the
boundary (red line in Fig.1). Generally, the heat source intensity
distribution $\bm{\varphi}$ is unknown in the TFR problem. Thus, the TFR
problem only considers the heat conduction in the area ${\Omega}_{NC}$ without
heat sources, i.e.,

$\frac{{{\partial}^{2}}\bm{T}\left(u,v\right)}{\partial{{u}^{2}}}+\frac{{{\partial}^{2}}\bm{T}\left(u,v\right)}{\partial{{v}^{2}}}=0,\quad\left(u,v\right)\in{{\Omega}_{NC}}.$
(5)

For the TFR problem, its objective is to reconstruct the temperature field of
the rectangular area $\Omega$ from the temperatures of some monitoring points,
as shown in Fig.2.

Figure 2: Temperature Field Reconstruction by monitoring point temperatures.

##  III Deep Monte Carlo Quantile Regression

###  III-A Physics-informed Deep MC-QR model

In this paper, the rectangular area $\Omega$ is discretized into a $h\times w$
two-dimensional array. According to the positions ${\Omega}_{MP}$ of the
temperature monitoring sensors, the corresponding element values in the
$h\times w$ two-dimensional array are equal to the the monitoring point
temperatures. The rest of element values are set to be zero. Thereby, a
monitoring point (MP) temperature image $\bm{T}_{MP}$ is constructed as shown
in Fig.3 (left). Besides, this paper also builds a quantile level image
$\bm{\tau}$ as shown in Fig.3 (right). Different from the MP temperature image
$\bm{T}_{MP}$, the values of element in the positions ${\Omega}_{MP}$ are
equal to $\tau$ for the quantile level image $\bm{\tau}$, where $\tau$ is
randomly sampled from the uniform distribution $U\left(0,1\right)$.

Figure 3: Temperature field reconstruction by monitoring point temperatures.

This paper builds a physics-informed Deep MC-QR model which can use the MP
temperature image $\bm{T}_{MP}$ and the quantile level image $\bm{\tau}$ to
reconstruct the temperature field of the rectangular area $\Omega$, as shown
in Fig.4. The physics-informed Deep MC-QR model with two input channels
includes two U-net [6] models, i.e., U-net-1 and U-net-2. Firstly, U-net-1
extracts the feature map of input $\left({{\bm{T}}_{MP}},\bm{\tau}\right)$.
Then, the feature map is flipped diagonally to ensure the reconstruction
accuracy of temperature field in the upside and right-side boundaries [12].
Finally, U-net-2 uses the flipped feature map to predict the temperature field
$\hat{\bm{T}}$ of the rectangular area $\Omega$.

Figure 4: The physics-informed Deep MC-QR model.

Supposed that the physics-informed Deep MC-QR model is denoted as
$\mathcal{MQNN}\left(\bm{T}_{MP},\bm{\tau};\bm{\theta}\right)$, where
$\bm{\theta}$ denotes the parameters of U-net-1 and U-net-2. Based on the
training dataset
$\left\\{\left(\bm{T}^{i}_{MP},\bm{\tau}_{i}\right)|i=1,2,\cdots,N\right\\}$,
the parameters $\bm{\theta}$ are learned by minimizing the following proposed
physics-informed loss function $\mathcal{L}\left(\theta\right)$.

  * •

Quantile MP temperature loss $\mathcal{L}_{\tau}\left(\bm{\theta}\right)$

For the $i\text{th}$ MP temperature image $\bm{T}^{i}_{MP}$, the predicted
temperature field $\hat{\bm{T}}$ of the satellite subsystem is
${\hat{\bm{T}}_{i}}=\mathcal{M}\left(\bm{T}^{i}_{MP},\bm{\tau}_{i};\bm{\theta}\right)$.
For $\left(u,v\right)\in{\Omega}_{MP}$, the error
${{\ell}_{{\tau}_{i}}}\left(u,v;\bm{\theta}\right)$ is

${{\ell}_{{\tau}_{i}}}\left(u,v;\bm{\theta}\right)={\tau}_{i}\left[{{\bm{T}^{i}_{MP}}\left(u,v\right)}-{{\hat{\bm{T}_{i}}}\left(u,v\right)}\right]$
(6)

for the condition
${{\bm{T}^{i}_{MP}}\left(u,v\right)}\geq{{\hat{\bm{T}_{i}}}\left(u,v\right)}$,
and

${{\ell}_{{\tau}_{i}}}\left(u,v;\bm{\theta}\right)=\left(1-{\tau}_{i}\right)\left({{\hat{\bm{T}_{i}}}\left(u,v\right)}-{{\bm{T}^{i}_{MP}}\left(u,v\right)}\right)$
(7)

for the condition
${{\bm{T}^{i}_{MP}}\left(u,v\right)}<{{\hat{\bm{T}_{i}}}\left(u,v\right)}$,
where ${\tau}_{i}\sim U\left(0,1\right)$. Thus, the quantile MP temperature
loss $\mathcal{L}_{\tau}\left(\bm{\theta}\right)$ is

$\mathcal{L}_{\tau}\left(\bm{\theta}\right)=\frac{1}{N}\frac{1}{N_{MP}}\sum\limits_{i=1}^{N}{\sum_{\left(u,v\right)\in{\Omega}_{MP}}{{{\ell}_{{\tau}_{i}}}\left(u,v;\bm{\theta}\right)}}.$
(8)

  * •

Laplace equation loss $\mathcal{L}_{LE}\left(\bm{\theta}\right)$

Assumed that each discretized MP temperature image $\bm{T}^{i}_{MP}$ has
$N_{NC}$ points for $\left(u,v\right)\in{{\Omega}_{NC}}$. Discretizing the
Laplace equation (5),
${{{\partial}^{2}}\hat{\bm{T}}_{i}\left(u_{j},v_{k}\right)}/{\partial{{u}_{j}^{2}}}$
and
${{\partial}^{2}}\hat{\bm{T}}_{i}\left(u_{j},v_{k}\right)/{\partial{{v}_{k}^{2}}}$
are transformed respectively to be

$\frac{{{\partial}^{2}}\hat{\bm{T}}_{i}\left(u_{j},v_{k}\right)}{\partial{{u}_{j}^{2}}}=\frac{\Delta{{{\hat{\bm{T}}}}^{i}_{j+1}}\Delta{{u}_{j}}-\Delta{{{\hat{\bm{T}}}}^{i}_{j}}\Delta{{u}_{j+1}}}{\Delta{{u}_{j+1}}{{\Delta}^{2}}{{u}_{j}}}$
(9)

and

$\frac{{{\partial}^{2}}\hat{\bm{T}}_{i}\left({{u}_{j}},{{v}_{k}}\right)}{v_{k}^{2}}=\frac{\Delta{{{\hat{\bm{T}}}}^{i}_{k+1}}\Delta{{v}_{k}}-\Delta{{{\hat{\bm{T}}}}^{i}_{k}}\Delta{{v}_{k+1}}}{\Delta{{v}_{k+1}}{{\Delta}^{2}}{{v}_{k}}},$
(10)

where $\Delta{{u}_{j}}={{u}_{j}}-{{u}_{j-1}}$,
$\Delta{{v}_{k}}={{v}_{k}}-{{v}_{k-1}}$,
$\Delta{{\hat{\bm{T}}}^{i}_{j}}=\hat{\bm{T}}_{i}\left({{u}_{j}},{{v}_{k}}\right)-\hat{\bm{T}}_{i}\left({{u}_{j-1}},{{v}_{k}}\right)$
and
$\Delta{{\hat{\bm{T}}}^{i}_{k}}=\hat{\bm{T}}_{i}\left({{u}_{j}},{{v}_{k}}\right)-\hat{\bm{T}}_{i}\left({{u}_{j}},{{v}_{k-1}}\right)$.
Thus, the Laplace equation loss $\mathcal{L}_{LE}\left(\bm{\theta}\right)$ for
${\left(u,v\right)\in{{\Omega}_{NC}}}$ is

${{\mathcal{L}}_{LE}}\left(\bm{\theta}\right)=\frac{1}{N}\frac{1}{{{N}_{NC}}}\sum\limits_{i=1}^{N}{\sum\limits_{\left(u,v\right)\in{{\Omega}_{NC}}}{{{\left|\frac{{{\partial}^{2}}\hat{\bm{T}}_{i}\left(u,v\right)}{\partial{{u}^{2}}}+\frac{{{\partial}^{2}}\hat{\bm{T}}_{i}\left(u,v\right)}{\partial{{v}^{2}}}\right|}^{2}}}}.$
(11)

  * •

Boundary condition loss $\mathcal{L}_{BC}\left(\bm{\theta}\right)$

According to (4), the boundary condition loss
$\mathcal{L}_{BC}\left(\bm{\theta}\right)$ for $N$ MP temperature images
$\left\\{\bm{T}^{i}_{MP}|i=1,2,\cdots,N\right\\}$ is

${{\mathcal{L}}_{BC}}\left(\bm{\theta}\right)=\frac{1}{N}\frac{1}{{{N}_{BC}}}\sum\limits_{i=1}^{N}{\sum\limits_{\left(u,v\right)\in{{\Omega}_{BC}}}{{{\left|{{{\hat{\bm{T}}}}_{i}}\left(u,v\right)-{{T}_{0}}\right|}^{2}}}}.$
(12)

where $N_{BC}$ means the number of points $\left(u,v\right)$ in the area
${{\Omega}_{BC}}$ of each discretized MP temperature image $\bm{T}^{i}_{MP}$.

  * •

TV regularization $\mathcal{L}_{TV}\left(\bm{\theta}\right)$

TV regularization [22, 12], as shown in (13), can maintain the smoothness of
the image,

${{\mathcal{R}}_{{{V}^{\beta}}}}=\int_{\Omega}{{{\left\\{{{\left[\frac{\partial
f\left(u,v\right)}{\partial u}\right]}^{2}}+{{\left[\frac{\partial
f\left(u,v\right)}{\partial
v}\right]}^{2}}\right\\}}^{\frac{\beta}{2}}}}dudv,$ (13)

where $f\left(u,v\right)$ is a continuous function. Due to the property that
the steady-state temperature field will not mutate sharply, this paper adopts
TV regularization to assist the training of model
$\mathcal{MQNN}\left(\bm{T}_{MP},\bm{\tau};\bm{\theta}\right)$. The TV
regularization $\mathcal{L}_{TV}\left(\bm{\theta}\right)$ ($\beta=2$) is
transformed to be (14) by the finite-difference approximation, i.e.,

$\displaystyle\mathcal{L}_{TV}^{u}\left(\bm{\theta}\right)=\frac{1}{h\times\left(w-1\right)}\sum\limits_{k=1}^{h}{\sum\limits_{j=1}^{w-1}{{{\left(\Delta\hat{\bm{T}}_{j}^{TV}\right)}^{2}}}},$
(14)
$\displaystyle\mathcal{L}_{TV}^{v}\left(\bm{\theta}\right)=\frac{1}{\left(h-1\right)\times
w}\sum\limits_{j=1}^{w}{\sum\limits_{k=1}^{h-1}{{{\left(\Delta\hat{\bm{T}}_{k}^{TV}\right)}^{2}}}},$
$\displaystyle{{\mathcal{L}}_{TV}}\left(\bm{\theta}\right)=\frac{1}{N}\sum\limits_{i=1}^{N}{\left[\mathcal{L}_{TV}^{u}\left(\bm{\theta}\right)+\mathcal{L}_{TV}^{v}\left(\bm{\theta}\right)\right]},$

where
$\Delta\hat{\bm{T}}_{j}^{TV}={{{\bm{\hat{T}}}}_{i}}\left({{u}_{j+1}},{{v}_{k}}\right)-{{{\bm{\hat{T}}}}_{i}}\left({{u}_{j}},{{v}_{k}}\right)$
and
$\Delta\hat{\bm{T}}_{k}^{TV}={{{\bm{\hat{T}}}}_{i}}\left({{u}_{j}},{{v}_{k+1}}\right)-{{{\bm{\hat{T}}}}_{i}}\left({{u}_{j}},{{v}_{k}}\right)$

Thus, the proposed physics-informed loss function
$\mathcal{L}\left(\theta\right)$ for learning the parameters $\bm{\theta}$ of
model $\mathcal{MQNN}\left(\bm{T}_{MP},\bm{\tau};\bm{\theta}\right)$ is

$\displaystyle\mathcal{L}\left(\bm{\theta}\right)={\alpha}_{1}{{\mathcal{L}}_{\tau}}\left(\bm{\theta}\right)+{{\alpha}_{2}}{{\mathcal{L}}_{LE}}\left(\bm{\theta}\right)+{{\alpha}_{3}}{{\mathcal{L}}_{BC}}\left(\bm{\theta}\right)$
(15)
$\displaystyle\qquad\quad\;+{{\alpha}_{4}}{{\mathcal{L}}_{TV}}\left(\bm{\theta}\right),$

where ${\alpha}_{1}$, ${\alpha}_{2}$, ${\alpha}_{3}$, and ${\alpha}_{4}$ are
hyperparameters.

###  III-B Model Monte Carlo Training Algorithm

This section proposes the model Monte Carlo training algorithm to learning the
parameters $\bm{\theta}$. Different from the general model training methods,
the model Monte Carlo training algorithm keeps changing the training data by
randomly sampling quantile level images in each epoch. The detailed
description is as follows: In the $ep\text{-th}$ epoch, a quantile level image
$\bm{\tau}_{i}$ ($i=1,2,\cdots,N$) corresponding to the MP temperature image
$\bm{T}^{i}_{MP}$ is created by the method in section III-A. Then, the
quantile level image $\bm{\tau}_{i}$ and the MP temperature image
$\bm{T}^{i}_{MP}$ ($i=1,2,\cdots,N$) constitute the training dataset
$\left\\{\left(\bm{T}^{i}_{MP},\bm{\tau}_{i}\right)_{ep}|i=1,2,\cdots,N\right\\}$.
Set the appropriate batch size, and the training dataset is divided into
$N_{batch}$ batches. This paper chooses the Adam algorithm [23] to update the
parameters $\bm{\theta}$. In summary, the pseudo code of the model Monte Carlo
training algorithm is shown in Algorithm 1.

Input:

MP temperature images $\left\\{\bm{T}^{i}_{MP}|i=1,2,\cdots,N\right\\}$.

Output:

Trained model $\mathcal{MQNN}\left(\bm{T}_{MP},\bm{\tau};\bm{\theta}\right)$.

1 Initialize the physics-informed Deep MC-QR model
$\mathcal{MQNN}\left(\bm{T}_{MP},\bm{\tau};\bm{\theta}\right)$;

2 for  _$ep=1:ep_{max}$_ do

3 for  _$i=1:N$_ do

4 Randomly take a sample ${\tau}_{i}$ from uniform distribution
$U\left(0,1\right)$;

5 Create a ${h}\times{w}$ two-dimensional array $\bm{\tau}_{i}$ with all zero
elements;

6 Perform $\bm{\tau}_{i}\left(u,v\right)={\tau}_{i}$ for
$\forall\left(u,v\right)\in{{\Omega}_{MP}}$;

7 Composite training data
$\left(\bm{T}_{MP}^{i},{{\bm{\tau}}_{i}}\right)_{ep}$.

8 end for

9 Dividing $N$ training data
$\left\\{\left(\bm{T}^{i}_{MP},\bm{\tau}_{i}\right)\right\\}$ to be
$N_{batch}$ batches;

10 for  _$ba=1:N_{batch}$_ do

11 Estimate the reconstructed temperature field
${{\hat{\bm{T}}}_{i}}=\mathcal{MQNN}\left(\bm{T}^{i}_{MP},\bm{\tau}_{i};\bm{\theta}\right)$;

12 Calculate the proposed loss function $\mathcal{L}\left(\bm{\theta}\right)$;

13 Calculate the gradient of $\mathcal{L}\left(\bm{\theta}\right)$;

14 Update the parameters $\bm{\theta}$ by the Adam algorithm.

15 end for

16

17 end for

Algorithm 1 Model Monte Carlo Training Algorithm.

###  III-C Model Prediction and Aleatoric Uncertainty Quantification

For a MP temperature image $\bm{T}^{pre}_{MP}$, the reconstructed temperature
field ${{\hat{\bm{T}}}_{pre}}$ and the quantified aleatoric uncertainty
$\hat{\bm{\sigma}}_{pre}$ are calculated as follows. In the predicted stage,
$N_{pre}$ quantile level images are generated by the method in section III-A.
Then, the composited prediction data
$\left\\{\left(\bm{T}^{pre}_{MP},\bm{\tau}_{\mathcal{A}}^{pre}\right)|\mathcal{A}=1,2,\cdots,N_{pre}\right\\}$
are obtained for the MP temperature image $\bm{T}^{pre}_{MP}$. For
$\mathcal{A}=1,2,\cdots,N_{pre}$, the temperature field
$\hat{\bm{T}}_{\mathcal{A}}^{pre}$ is predicted by the trained model
$\mathcal{MQNN}\left(\bm{T}_{MP},\bm{\tau};\bm{\theta}\right)$. Thereby, the
reconstructed temperature field’s approximation $\hat{\bm{T}}_{pre}$ and the
aleatoric uncertainty $\bm{\sigma}_{pre}$ are estimated by calculating the
mean and the standard deviation of $N_{pre}$ results
$\hat{\bm{T}}_{\mathcal{A}}^{pre}$ ($\mathcal{A}=1,2,\cdots,N_{pre}$),
respectively, i.e.,

${{\hat{\bm{T}}}_{pre}}=\frac{1}{N_{pre}}\sum\limits_{\mathcal{A}=1}^{{{N}_{pre}}}{\hat{\bm{T}}_{\mathcal{A}}^{pre}},$
(16)

${{\bm{\sigma}}_{pre}}={{\left[\frac{1}{N_{pre}}\sum\limits_{\mathcal{A}=1}^{{{N}_{pre}}}{{{\left(\hat{\bm{T}}_{\mathcal{A}}^{pre}\right)}^{2}}-{{\left({{{\hat{\bm{T}}}}_{pre}}\right)}^{2}}}\right]}^{\frac{1}{2}}}.$
(17)

In summary, the pseudo code of the model prediction and aleatoric uncertainty
quantification algorithm is shown in Algorithm 2.

Input:

(1) MP temperature image $\bm{T}^{pre}_{MP}$;

(2) Sampling number $N_{pre}$ of quantile levels;

(3) Trained model
$\mathcal{MQNN}\left(\bm{T}_{MP},\bm{\tau};\bm{\theta}\right)$.

Output:

(1) Reconstructed temperature field ${{\hat{\bm{T}}}_{pre}}$;

(2) Quantified aleatoric uncertainty $\hat{\bm{\sigma}}_{pre}$.

1

2for  _$\mathcal{A}=1:N_{pre}$_ do

3 Randomly take a sample ${\tau}^{pre}_{\mathcal{A}}$ from uniform
distribution $U\left(0,1\right)$;

4 Create a $h\times w$ two-dimensional array $\bm{\tau}^{pre}_{\mathcal{A}}$
with all zero elements;

5 Perform
$\bm{\tau}^{pre}_{\mathcal{A}}\left(u,v\right)={\tau}^{pre}_{\mathcal{A}}$ for
$\forall\left(u,v\right)\in{{\Omega}_{MP}}$;

6 Composite prediction data
$\left(\bm{T}^{pre}_{MP},\bm{\tau}^{pre}_{\mathcal{A}}\right)$;

7 Predict the reconstructed temperature field
${{\hat{\bm{T}}}^{pre}_{\mathcal{A}}}=\mathcal{M}\left(\bm{T}^{pre}_{MP},\bm{\tau}^{pre}_{\mathcal{A}};\bm{\theta}\right)$.

8 end for

9Calculate the mean ${{\hat{\bm{T}}}_{pre}}$ and the standard deviation
$\hat{\bm{\sigma}}_{pre}$ of
$\left\\{{{\hat{\bm{T}}}^{pre}_{\mathcal{A}}}|\mathcal{A}=1,2,\cdots,N_{pre}\right\\}$.

Algorithm 2 Model prediction and aleatoric uncertainty quantification
algorithm.

##  IV Experiment

###  IV-A Experimental Setup

As shown in Fig.5, this paper uses a system ($W=H=0.2\;\text{m}$) to validate
the effectiveness of the proposed Deep MC-QR method. This system has 55 heat
sources, and the heat sink with width $\delta=0.02\;\text{m}$ is in the center
of the left boundary of the system. The positions of monitoring points are
shown in Fig.6. In this experiment, the power of each heat source obeys the
normal distribution with a mean of 20000 W and a standard deviation of 1000 W.
Besides, the constant temperature $T_{0}$ is 298 K. The parameters $h$ and $w$
are set to be 200, i.e., $h=w=200$. 15000 MP temperature images
$\left\\{{\bm{T}}_{MP}^{1},{\bm{T}}_{MP}^{2},\cdots,{\bm{T}}_{MP}^{15000}\right\\}$
are generated by the recon-data-generator111https://github.com/shendu-
sw/recon-data-generator, where the training dataset, the validating dataset,
and the testing dataset have 10800, 1200 and 3000 MP temperature images,
respectively. Especially, the corresponding truth temperature fields (label)
of 3000 testing data are also generated by the recon-data-generator. The
hyperparameters ${\alpha}_{1}$, ${\alpha}_{2}$, ${\alpha}_{3}$, and
${\alpha}_{4}$ are equal to $1\times 10^{5}$, $1\times 10^{2}$, $1\times
10^{2}$ and $1\times 10^{4}$, respectively. The relevant codes of the Deep MC-
QR method by Python are available on this website
222https://github.com/Xiaohu-Zheng/Deep-MC-QR.

Figure 5: Heat source layout. Figure 6: Monitoring point positions.

In this experiment, five kinds of noises are respectively added into
temperature values of 15000 MP temperature images
$\left\\{{\bm{T}}_{MP}^{1},{\bm{T}}_{MP}^{2},\cdots,{\bm{T}}_{MP}^{15000}\right\\}$
to generate five types of the dataset as follows:

  * •

Gaussian noise ${\varepsilon}_{1}\sim N\left(0,{0.3}^{2}\right)$ for green
monitoring points in Fig.6.

  * •

Gaussian noise ${\varepsilon}_{2}\sim N\left(0,{0.5}^{2}\right)$ for green
monitoring points in Fig.6.

  * •

Uniform noise ${\varepsilon}_{3}\sim U\left(-0.3,0.3\right)$ for green
monitoring points in Fig.6.

  * •

Uniform noise ${\varepsilon}_{4}\sim U\left(-1,1\right)$ for green monitoring
points in Fig.6.

  * •

Gaussian noise ${\varepsilon}_{5}\sim N\left(0,{0.3}^{2}\right)$ for
monitoring points to the right of the blue dotted line in Fig.6 (Excluding the
five monitoring points on the right boundary).

Based on the above five kinds of the dataset, five trained models are denoted
as $\mathcal{MQNN}_{{\varepsilon}_{1}}$, $\mathcal{MQNN}_{{\varepsilon}_{2}}$,
$\mathcal{MQNN}_{{\varepsilon}_{3}}$, $\mathcal{MQNN}_{{\varepsilon}_{4}}$,
and $\mathcal{MQNN}_{{\varepsilon}_{5}}$, respectively. Besides, four
evaluation criteria for model accuracy are as follows: a) the average of root
mean square error (RMSE)

$\overline{RMSE}=\frac{1}{6\text{e}5}\sum\limits_{t=1}^{3000}{\sqrt{\sum\limits_{\left(x,y\right)\in\Omega}{{{\left[\bm{T}_{t}\left(x,y\right)-\bm{\hat{T}}_{t}\left(x,y\right)\right]}^{2}}}}},$
(18)

b) the average of mean absolute error (MAE)

$\overline{MAE}=\frac{1}{1.2\text{e}8}\sum\limits_{t=1}^{3000}{\sum\limits_{\left(x,y\right)\in\Omega}{\left|\bm{T}_{t}\left(x,y\right)-\bm{\hat{T}}_{t}\left(x,y\right)\right|}},$
(19)

c) the average of mean relative error (MRE)

$\overline{MRE}=\frac{1}{1.2\text{e}8}\sum\limits_{t=1}^{3000}{\sum\limits_{\left(x,y\right)\in\Omega}{\frac{\left|\bm{T}_{t}\left(x,y\right)-\bm{\hat{T}}_{t}\left(x,y\right)\right|}{\bm{T}_{t}\left(x,y\right)}}},$
(20)

d) the average of R square ($R^{2}$)

$\displaystyle\overline{{{R}^{2}}}=\frac{1}{3\text{e}3}\sum\limits_{t=1}^{3000}{\left\\{1-{\frac{\sum\limits_{\left(x,y\right)\in\Omega}{{\left[\bm{T}_{t}\left(x,y\right)-\bm{\hat{T}}_{t}\left(x,y\right)\right]}^{2}}}{\sum\limits_{\left(x,y\right)\in\Omega}{{\left[\bm{T}_{t}\left(x,y\right)-\bm{\bar{T}}_{t}\left(x,y\right)\right]}^{2}}}}\right\\}},$
(21)
$\displaystyle\bm{\bar{T}}_{t}=\frac{1}{3\text{e}3}\sum\limits_{t=1}^{3000}{\bm{\hat{T}}_{t}}.$

For the first three evaluation criteria $\overline{RMSE}$, $\overline{MAE}$
and $\overline{MRE}$, the closer their values are to 0, the higher the
accuracy of the trained model
$\mathcal{M}\left(\bm{T}_{MP},\bm{\tau};\bm{\theta}\right)$. ${{R}^{2}}\to 1$
means that the trained model
$\mathcal{M}\left(\bm{T}_{MP},\bm{\tau};\bm{\theta}\right)$ is accurate.

###  IV-B Model accuracy analysis

The prediction accuracy of five models $\mathcal{MQNN}_{{\varepsilon}_{1}}$,
$\mathcal{MQNN}_{{\varepsilon}_{2}}$, $\mathcal{MQNN}_{{\varepsilon}_{3}}$,
$\mathcal{MQNN}_{{\varepsilon}_{4}}$, and $\mathcal{MQNN}_{{\varepsilon}_{5}}$
are shown in TABLE I. Apparently, three kinds of average errors, i.e.,
$\overline{RMSE}$, $\overline{MAE}$ and $\overline{MRE}$, are sufficiently
small for five models. The $\overline{R^{2}}$ of five models are greater than
or equal to 0.99995. Therefore, the proposed Deep MC-QR method can reconstruct
the temperature fields accurately.

TABLE I: The prediction accuracy of five models trained by the MP temperature images with five kinds of noises. Noise | Model | $\overline{RMSE}$ | $\overline{MAE}$ | $\overline{MRE}$ | $\overline{R^{2}}$
---|---|---|---|---|---
${\varepsilon}_{1}$ | $\mathcal{MQNN}_{{\varepsilon}_{1}}$ | 0.07370 | 0.04932 | 0.00013 | 0.99996
${\varepsilon}_{2}$ | $\mathcal{MQNN}_{{\varepsilon}_{2}}$ | 0.07959 | 0.05066 | 0.00014 | 0.99996
${\varepsilon}_{3}$ | $\mathcal{MQNN}_{{\varepsilon}_{3}}$ | 0.07094 | 0.04489 | 0.00012 | 0.99996
${\varepsilon}_{4}$ | $\mathcal{MQNN}_{{\varepsilon}_{4}}$ | 0.08173 | 0.05263 | 0.00014 | 0.99995
${\varepsilon}_{5}$ | $\mathcal{MQNN}_{{\varepsilon}_{5}}$ | 0.08921 | 0.06056 | 0.00014 | 0.99995

This experiment randomly chooses a MP temperature image in Fig.7 (adding five
kinds of data noises) to show the prediction performance of five models. Fig.8
is the truth temperature field of the MP temperature image in Fig.7. Based on
five models $\mathcal{MQNN}_{{\varepsilon}_{1}}$,
$\mathcal{MQNN}_{{\varepsilon}_{2}}$, $\mathcal{MQNN}_{{\varepsilon}_{3}}$,
$\mathcal{MQNN}_{{\varepsilon}_{4}}$, and
$\mathcal{MQNN}_{{\varepsilon}_{5}}$, five reconstructed temperature fields
are presented in Fig.9, Fig.10, Fig.11, Fig.12, and Fig.13, respectively.
Compared with Fig.8, five reconstructed temperature fields are basically
consistent with the truth temperature field.

Figure 7: A MP temperature image. Figure 8: Truth temperature field of the MP
temperature image in Fig.7. Figure 9: Predicted temperature field by
$\mathcal{MQNN}_{{\varepsilon}_{1}}$. Figure 10: Predicted temperature field
by $\mathcal{MQNN}_{{\varepsilon}_{2}}$. Figure 11: Predicted temperature
field by $\mathcal{MQNN}_{{\varepsilon}_{3}}$. Figure 12: Predicted
temperature field by $\mathcal{MQNN}_{{\varepsilon}_{4}}$. Figure 13:
Predicted temperature field by $\mathcal{MQNN}_{{\varepsilon}_{5}}$.

###  IV-C Aleatoric uncertainty quantification results analysis

For the MP temperature image in Fig.7, the quantified aleatoric uncertainties
by four models $\mathcal{MQNN}_{{\varepsilon}_{1}}$,
$\mathcal{MQNN}_{{\varepsilon}_{2}}$, $\mathcal{MQNN}_{{\varepsilon}_{3}}$ and
$\mathcal{MQNN}_{{\varepsilon}_{4}}$ are shown in Fig.14, Fig.15, Fig.16 and
Fig.17, respectively. According to section IV-A, these four models are trained
by the datasets that include the Gaussian noises (${\varepsilon}_{1}\sim
N\left(0,{0.3}^{2}\right)$, ${\varepsilon}_{2}\sim
N\left(0,{0.50}^{2}\right)$) and uniform noise (${\varepsilon}_{3}\sim
U\left(-0.3,0.3\right)$, ${\varepsilon}_{4}\sim U\left(-1,1\right)$) in the
green points in Fig.6. Referring to Fig.14, Fig.15, Fig.16 and Fig.17, the
aleatoric uncertainties of the areas around the green points (Fig.6) are much
larger than the other areas. Besides, the maximum values of two aleatoric
uncertainties in Fig.14 and Fig.15 are approximately equal to 0.3 and 0.5,
respectively. For the models $\mathcal{MQNN}_{{\varepsilon}_{3}}$
$\mathcal{MQNN}_{{\varepsilon}_{4}}$ trained by data with the uniform noises
(${\varepsilon}_{3}$, ${\varepsilon}_{4}$), the quantified aleatoric
uncertainties are approximately equal to the standard deviations 0.173 and
0.577 of two uniform noises, respectively. In summary, the proposed Deep MC-QR
method can accurately quantify the aleatoric uncertainty caused by data noise.

Figure 14: Quantified aleatoric uncertainty by
$\mathcal{MQNN}_{{\varepsilon}_{1}}$. Figure 15: Quantified aleatoric
uncertainty by $\mathcal{MQNN}_{{\varepsilon}_{2}}$. Figure 16: Quantified
aleatoric uncertainty by $\mathcal{MQNN}_{{\varepsilon}_{3}}$. Figure 17:
Quantified aleatoric uncertainty by $\mathcal{MQNN}_{{\varepsilon}_{4}}$.

###  IV-D The effect of aleatoric uncertainty for TFR

For the model $\mathcal{MQNN}_{{\varepsilon}_{5}}$, it is trained by the data
with the Gaussian noise ${\varepsilon}_{5}\sim N\left(0,{0.3}^{2}\right)$ for
monitoring points to the right of the blue dotted line in Fig.6. The fusion
image between the monitoring point positions (Fig.6) and the quantified
aleatoric uncertainty by $\mathcal{MQNN}_{{\varepsilon}_{5}}$ is shown in
Fig.18.

Figure 18: The fusion image between the monitoring point positions (Fig.6) and
the quantified aleatoric uncertainty by $\mathcal{MQNN}_{{\varepsilon}_{5}}$.

Referring to Fig.18, the aleatoric uncertainty corresponding to the area to
the right of the blue dotted line is much larger than the area to the left of
the blue dotted line. Besides, there are four especial phenomenons in Fig.18
as follows:

a) The maximum aleatoric uncertainty (about 0.35) is larger than the actual
data noise (0.3). As shown in Fig.18, the maximum value of quantified
aleatoric uncertainty is approximately equal to 0.35, while the value of
actual Gaussian noise is equal to 0.3.

b) There is also aleatoric uncertainty on the non-monitoring point area to the
right of the blue dotted line. Compared with the non-monitoring point area to
the left of the blue dotted line, the aleatoric uncertainty of the non-
monitoring point area to the right of the blue dotted line is relatively large
as shown in Fig.18. However, the non-monitoring point area to the left of the
blue dotted line does not contain data noise.

c) The aleatoric uncertainty in the non-monitoring point area to the right of
the blue dotted line has large differences. As shown in Fig.18, the aleatoric
uncertainty in the green dashed ellipse is larger than in the red dashed
ellipse for the non-monitoring point area. In fact, both non-monitoring point
areas do not include data noise.

d) The uncertainty around five monitoring points (in the orange circle) on the
right boundary is small.

In fact, the spread of data noise results in the above four especial
phenomenons. Firstly, The data noise propagation leads to the accumulation of
aleatoric uncertainty. Thereby, the maximum aleatoric uncertainty is larger
than the actual Gaussian noise (phenomenon a)). Secondly, the aleatoric
uncertainty is transferred to the surrounding area due to noise propagation,
which brings about phenomenon b). Thirdly, the data noise may include positive
or negative Gaussian noise. Thus, the accumulation of positive or negative
Gaussian noise will lead to the phenomenon c). According to section IV-A, five
monitoring points (in the orange circle) on the right boundary do not contain
data noise. Therefore, the phenomenon d) appears in Fig.18.

In summary, data noise causes the reconstructed temperature field to have
aleatoric uncertainties at and around the monitoring point. Therefore,
aleatoric uncertainty quantification is important to ensure the creditability
of the reconstructed temperature field.

##  V Conclusions

This paper proposes the Deep MC-QR method to reconstruct the temperature field
and quantify the aleatoric uncertainty caused by data noise. By physical
knowledge to guide the training of CNN, the proposed Deep MC-QR method can
reconstruct an accurate TFR surrogate model without using any labeled training
data. Besides, the proposed Deep MC-QR method can quantify aleatoric
uncertainty by constructing a quantile level image for each MP temperature
image. This paper adds five kinds of data noise in the MP temperature image.
The results show that the proposed Deep MC-QR method can accurately
reconstruct the temperature field and quantify the aleatoric uncertainty
precisely. Especially, the results also show that data noise causes the
reconstructed temperature field to have aleatoric uncertainties at and around
the monitoring point. Thus, aleatoric uncertainty quantification is necessary
for the TFR problem.

## Acknowledgment

This work was supported by the Postgraduate Scientific Research Innovation
Project of Hunan Province (No.CX20200006) and the National Natural Science
Foundation of China (Nos.11725211).

## References

  * [1] W. Yao, X. Chen, W. Luo, M. van Tooren, and J. Guo, “Review of uncertainty-based multidisciplinary design optimization methods for aerospace vehicles,” _Progress in Aerospace Sciences_ , vol. 47, no. 6, pp. 450–479, 2011.
  * [2] X. Zheng, W. Yao, Y. Xu, and X. Chen, “Improved compression inference algorithm for reliability analysis of complex multistate satellite system based on multilevel bayesian network,” _Reliability Engineering & System Safety_, vol. 189, pp. 123–142, 2019.
  * [3] X. Zheng, W. Yao, Y. Xu, and X. Chen, “Algorithms for bayesian network modeling and reliability inference of complex multistate systems: Part i – independent systems,” _Reliability Engineering & System Safety_, vol. 202, 2020.
  * [4] W. Zhang, X. Feng, F. Goerlandt, and Q. Liu, “Towards a convolutional neural network model for classifying regional ship collision risk levels for waterway risk analysis,” _Reliability Engineering & System Safety_, vol. 204, p. 107127, 2020.
  * [5] X. Zhao, Z. Gong, J. Zhang, W. Yao, and X. Chen, “A surrogate model with data augmentation and deep transfer learning for temperature field prediction of heat source layout,” _Structural and Multidisciplinary Optimization_ , 2021\.
  * [6] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _Proceedings of Medical Image Computing and Computer-Assisted Intervention_. Springer International Publishing, 2015, Book Section, pp. 234–241.
  * [7] S. Shen, H. Lu, M. Sadoughi, C. Hu, V. Nemani, A. Thelen, K. Webster, M. Darr, J. Sidon, and S. Kenny, “A physics-informed deep learning approach for bearing fault detection,” _Engineering Applications of Artificial Intelligence_ , vol. 103, 2021.
  * [8] Z. Fang, “A high-efficient hybrid physics-informed neural networks based on convolutional neural network,” _IEEE Transactions on Neural Networks and Learning System_ , vol. 99, pp. 1–13, 2021.
  * [9] H. Gao, L. Sun, and J.-X. Wang, “Phygeonet: Physics-informed geometry-adaptive convolutional neural networks for solving parameterized steady-state pdes on irregular domain,” _Journal of Computational Physics_ , vol. 428, p. 110079, 2021.
  * [10] H. Gao, L. N. Sun, and J. X. Wang, “Super-resolution and denoising of fluid flow using physics-informed convolutional neural networks without high-resolution labels,” _PHYSICS OF FLUIDS_ , vol. 33, no. 7, 2021.
  * [11] Z. Zhang, Y. Li, W. Zhou, X. Chen, W. Yao, and Y. Zhao, “Tonr: An exploration for a novel way combining neural network with topology optimization,” _Computer Methods in Applied Mechanics and Engineering_ , vol. 386, p. 114083, 2021.
  * [12] Z. Gong, W. Zhou, J. Zhang, W. Peng, and W. Yao, “Physics-informed deep reversible regression model for temperature field reconstruction of heat-source systems,” 2021. [Online]. Available: arXiv:2106.11929v2
  * [13] J. Postels, F. Ferroni, H. Coskun, N. Navab, and F. Tombari, “Sampling-free epistemic uncertainty estimation using approximated variance propagation,” in _Proceedings of International Conference on Computer Vision_ , 2019, Conference Proceedings.
  * [14] Y. Gal, “Uncertainty in deep learning,” Thesis, 2016.
  * [15] A. Kendall and Y. Gal, “What uncertainties do we need in bayesian deep learning for computer vision?” in _Proceedings of the 31st Conference on Neural Information Processing Systems_ , Conference Proceedings.
  * [16] B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” in _31st Conference on Neural Information Processing Systems_ , Conference Proceedings.
  * [17] J. Z. Liu, J. Paisley, M.-A. Kioumourtzoglou, and B. A. Coull, “Accurate uncertainty estimation and decomposition in ensemble learning,” in _Proceedings of the 33rd Conference on Neural Information Processing Systems_ , 2019, Conference Proceedings.
  * [18] N. Tagasovska and D. Lopez-Paz, “Single-model uncertainties for deep learning,” in _Proceedings of the 33rd Conference on Neural Information Processing Systems_ , 2019, Conference Proceedings.
  * [19] A. Malinin, B. Mlodozeniec, and M. Gales, “Ensemble distribution distillation,” in _Proceedings of International Conference on Learning Representations_ , 2020, Conference Proceedings.
  * [20] R. Koenker, “Quantile regression for longitudinal data,” _Journal of Multivariate Analysis_ , vol. 91, no. 1, pp. 74–89, 2004.
  * [21] I. Takeuchi, Q. V. Le, T. D. Sears, and A. J. Smola, “Nonparametric quantile estimation,” _Journal of Machine Learning Research_ , vol. 7, pp. 1231–1264, 2006.
  * [22] L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” _Physica D: Nonlinear Phenomena_ , vol. 60, no. 1, pp. 259–268, 1992.
  * [23] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _Proceedings of International Conference on Learning Representations_ , 2015\.