# Computing the Vapnik Chervonenkis Dimension for Non-Discrete Settings

Mohamed MOUHAJIR
ENSIAS, Mohammed V University
Morocco, Rabat 10000
<EMAIL_ADDRESS>&Mohammed NECHBA11footnotemark: 1
ENSIAS, Mohammed V University
Morocco, Rabat 10000
<EMAIL_ADDRESS>&Yassine SEDJARI22footnotemark: 2
ENSIAS, Mohammed V University
Morocco, Rabat 10000
<EMAIL_ADDRESS>These authors contributed equally to this research.

###### Abstract

In 1984, Valiant [7] introduced the Probably Approximately Correct (PAC)
learning framework for boolean function classes. Blumer et al. [2] extended
this model in 1989 by introducing the VC dimension as a tool to characterize
the learnability of PAC. The VC dimension was based on the work of Vapnik and
Chervonenkis in 1971 [8], who introduced a tool called the growth function to
characterize the shattering property. Researchers have since determined the VC
dimension for specific classes, and efforts have been made to develop an
algorithm that can calculate the VC dimension for any concept class. In 1991,
Linial, Mansour, and Rivest [4] presented an algorithm for computing the VC
dimension in the discrete setting, assuming that both the concept class and
domain set were finite. However, no attempts had been made to design an
algorithm that could compute the VC dimension in the general
setting.Therefore, our work focuses on developing a method to approximately
compute the VC dimension without constraints on the concept classes or their
domain set. Our approach is based on our finding that the Empirical Risk
Minimization (ERM) learning paradigm can be used as a new tool to characterize
the shattering property of a concept class.

##  1 Introduction

One critical facet of Statistical Learning Theory is to prove the learnability
of some concept classes within the PAC (Probably Approximately Correct)
learning framework, which was introduced by Valiant in 1984 [7]. To this end,
Blumer et al. extended in 1989 [2] Valiant’s work by coining the notion of
Vapnik-Chervonenkis (VC) dimension as a tool that characterizes the PAC
learnability of binary-valued concept classes. This notion of VC dimension was
buid upon the work of Vapnik and Chervonenkis in 1971 [8] , who introduced a
tool called the growth function to characterize the shattering property. The
main result showed by Blumer et al. [2] was that finiteness of VC dimension is
a necessary and sufficient condition to say that a concept class is PAC
learnable.

As time passed, researchers directed their interest towards calculating the VC
dimension for some specific cases. In this context, the work of Dudley et al.
in 1981 [9] has laid the foundation ground of what was called lately by Ben-
David & Shalev-Shwartz ,2014 [6] “Dudley Classes”. Due to this notion, the VC
dimension of many classes was easily calculated. Examples of such classes
include: the class of Half-Spaces, the class of Open Balls, and the class of
polynomials that are of degree less than some given integer, and so on.
Subsequently, attention has shifted towards designing an algorithm that could
compute the VC dimension of any concept class $\mathcal{H}$. In this scope,
the only formal initiative to develop such an algorithm, to our knowledge, was
taken by Linial et al. in 1991 [4], when they constructed an algorithm to
calculate the VC dimension in what they referred to as the ’Discrete VC
problem’. Yet, two key assumptions underpinned their approach: that the
concept class and domain set are both finite. Meaning that we can present the
concept class as $\mathcal{H}=\\{h_{1},...,h_{l}\\}$ , and the domain set as
$\mathcal{X}=\\{x_{1},...,x_{m}\\}$.

Then, according to this modeling, the concept class and the domain set are
presented as a $0-1$ valued matrix $M$. Each row $i$, represents a concept
$h_{i}$ in $\mathcal{H}$, and each column $j$ represent a domain point
$x_{j}$. Then, each element $M_{i,j}$ in this matrix is filled as following:

  * •

If the $x_{j}$ belongs to $h_{i}$ , then $M_{i,j}=1$.

  * •

Otherwise , $M_{i,j}=0$.

Therefore, in this context, computing the VC dimension of a concept class
$\mathcal{H}$ becomes the problem of computing the VC dimension of the above
matrix $M$ according to the following technique: To prove that the VC
dimension is less than d, we should take all combinations of columns
$\binom{m}{d}$. Each combination represents a set upon which we will assess
the shattering property. Thus, if we find that $\mathcal{H}$ doesn’t shatter
any of them , we say that $VCdim(\mathcal{H})<d$ .

However, it is evident that this approach lacks generality. In fact, the most
practical concept classes are infinite. Besides, if we assume the finiteness
of the concept class, then computing the VC dimension of this concept class is
not necessary to prove that it is PAC learnable. As stated by Ben-David and
Shalev-Shwartz (2014)[6] , the sample complexity required to ensure PAC
learnability is directly proportional to the size of the concept class:
$m(\epsilon,\delta)\leq\frac{\ln{(|\mathcal{H}|}/\delta)}{\epsilon}$.

In this work, our aim is to address this gap by proposing a new algorithm that
can compute the VC dimension of a concept class without requiring the
finiteness of either the class or its domain set. This approach has the
potential to significantly expand the scope of applicability of the VC
dimension concept, making it relevant for a wider range of real-world
scenarios.

######  Remark.

In this article, we will mainly use the common terminology and notation as in
the textbook [6]. We will use the term shattering to refers to Vapnik-
Chervonenkis shattering (since there are more sophisticated approaches of
defining the shattering property which are used for mutliclass problems such
us : Natarajan shattering and DS shattering [3])

###  1.1 Results

Our main result is the characterization of the shattering property using the
ERM (Empirical Risk Minimization) learning paradigm. Specifically, we have
derived a theorem and a corollary that provide an algorithmic way to assess
the shattering property of any hypothesis class.This result has significant
practical benefits, allowing us to efficiently evaluate the shattering
property of any given class:

######  Theorem 1.1 (Nechba-Mouhajir theorem of shattering).

Let $\mathcal{H}$ be a class of functions from $\mathcal{X}$ to $\\{0,1\\}$ .
Let $\mathcal{C}=\\{c_{1},...,c_{d}\\}\subset\mathcal{X}$. Let
$(y_{1},...,y_{d})\in\\{0,1\\}^{d}$ a fixed $d$-tuple ,for some integer $d$.
There exists a function $h^{*}\in\mathcal{H}$ such that
$h^{*}(\mathcal{C})=(y_{1},...,y_{d})$ if and only if the $ERM_{\mathcal{H}}$
over $\mathcal{S}=\\{(c_{1},y_{1}),...,(c_{d},y_{d})\\}$ (where
$ERM_{\mathcal{H}}(\mathcal{S})=\underset{{h\in\mathcal{H}}}{\arg\min(L_{\mathcal{S}}(h))}$
) gives zero empirical loss ( i.e : $L_{\mathcal{S}}(ERM_{\mathcal{H}}(S))=0$
).

######  Corollary 1.1.1.

Let $\mathcal{H}$ be a hypothesis class for binary classification. Let
$\mathcal{C}=\\{c_{1},...,c_{d}\\}\subset\mathcal{X}$.
$\mathcal{H}$ shatters $\mathcal{C}$ if and only if for every $d$-tuple in
$\\{0,1\\}^{d}$ (i.e $\forall(y_{1},...,y_{d})\in\\{0,1\\}^{d}$) , the
$ERM_{\mathcal{H}}$ learner over
$\mathcal{S}=\\{(c_{1},y_{1}),...,(c_{d},y_{d})\\}$ will give a zero empirical
loss.

##  2 ERM as a new characterization of the shattering property

###  2.1 The Problem of Shattering : Definitions and Properties

Let $\mathcal{H}$ be a hypothesis class for binary classification task :

$\displaystyle\mathcal{H}$
$\displaystyle\coloneqq\Bigl{\\{}\,\emph{h}\mid\operatorname{\emph{h}}\colon\mathcal{X}\to\mathcal{Y}=\\{0,1\\},\quad
x\mapsto\emph{h}(x)\,\Bigl{\\}}$

######  Definition 2.1 (The restriction of $\mathcal{H}$ over a domain
$\mathcal{C}$).

Let $\mathcal{H}$ be a class of functions from $\mathcal{X}$ to $\\{0,1\\}$
and let $\mathcal{C}=\\{c_{1},...,c_{d}\\}\subset\mathcal{X}$. The restriction
of $\mathcal{H}$ to $\mathcal{C}$ is the set of functions from $\mathcal{C}$
to $\\{0,1\\}$ that can be derived from $\mathcal{H}$. That is,

$\displaystyle\mathcal{H}_{c}=\Bigl{\\{}\,(\emph{h}(c_{1}),...,\emph{h}(c_{d}))\mid\emph{h}\in\mathcal{H}\,\Bigl{\\}}$

where we represent each function from $\mathcal{C}$ to $\\{0,1\\}$ as a vector
in $\\{0,1\\}^{|\mathcal{C}|}$

######  Definition 2.2 (Shattering).

A hypothesis class $\mathcal{H}$ of functions from $\mathcal{X}$ to
$\\{0,1\\}$ shatters a finite set $\mathcal{C}\subset X$ if the restriction of
$\mathcal{H}$ to $\mathcal{C}$ is the set of all functions from $\mathcal{C}$
to $\\{0,1\\}$. That is, $|\mathcal{H}_{c}|=2^{|\mathcal{C}|}$.

######  Interpretation 2.1.

To prove that a given hypothesis class $\mathcal{H}$ shatters a given domain
set $\mathcal{C}\subset\mathcal{X}$, we should perform two steps:

  1. 1.

$\Rightarrow|$ We should prove that
$\mathcal{H}_{c}\subset\\{0,1\\}^{|\mathcal{C}|}$ (which is trivial since
$\mathcal{Y}=\\{0,1\\})$

  2. 2.

$\Leftarrow|$ (see Figure1 for more intuition) We should prove that
$\\{0,1\\}^{|\mathcal{C}|}\subset\mathcal{H}_{c}$. In other words, we should
prove that for every $|\mathcal{C}|$-tuple of 0 and 1, there exist a
hypothesis $h^{*}$ such that applying $h^{*}$ over $\mathcal{C}$ will give as
this $|\mathcal{C}|$-tuple. Formally :

$\displaystyle\\{0,1\\}^{|\mathcal{C}|}\subset\mathcal{H}_{c}\Longleftrightarrow\forall\;(y_{1},...,y_{|\mathcal{C}|})\in\\{0,1\\}^{|\mathcal{C}|},\;\exists\emph{h}^{*}\in\mathcal{H}:\emph{h}^{*}(\mathcal{C})$
$\displaystyle=\emph{h}^{*}((c_{1},...,c_{|\mathcal{C}|}))$
$\displaystyle=(\emph{h}^{*}(c_{1}),...,\emph{h}^{*}(c_{|\mathcal{C}|}))$
$\displaystyle=(y_{1},...,y_{|\mathcal{C}|})$

$\mathcal{H}_{\mathcal{C}}$$\\{0,1\\}^{|\mathcal{C}|}$$h_{1}^{*}(\mathcal{C})$$h_{n}^{*}(\mathcal{C})$$(y_{1},...,y_{|\mathcal{C}|})$$(y_{1}^{,},...,y_{|\mathcal{C}|}^{,})$
Figure 1: Explicating diagram of
$\\{0,1\\}^{|C|}\subset\mathcal{H}_{\mathcal{C}}$

Now , we will present our first contribution in this paper which this the
following theorem of shattering :

######  Theorem 2.1 (Nechba-Mouhajir theorem of shattering).

Let $\mathcal{H}$ be a class of functions from $\mathcal{X}$ to $\\{0,1\\}$ .
Let $\mathcal{C}=\\{c_{1},...,c_{d}\\}\subset\mathcal{X}$. Let
$(y_{1},...,y_{d})\in\\{0,1\\}^{d}$ a fixed $d$-tuple, for some integer $d$.
There exists a function $h^{*}\in\mathcal{H}$ such that
$h^{*}(\mathcal{C})=(y_{1},...,y_{d})$ if and only if the $ERM_{\mathcal{H}}$
over $\mathcal{S}=\\{(c_{1},y_{1}),...,(c_{d},y_{d})\\}$ (where
$ERM_{\mathcal{H}}(\mathcal{S})=\underset{{h\in\mathcal{H}}}{\arg\min(L_{\mathcal{S}}(h))}$
) gives zero empirical loss ( i.e : $L_{\mathcal{S}}(ERM_{\mathcal{H}}(S))=0$
) .

###### Proof.

  * •

$\Rightarrow|$ In fact, if there exists an $h^{*}\in\mathcal{H}$ such that
$h^{*}(\mathcal{C})=h^{*}(\\{c_{1},...,c_{d}\\})=(y_{1},...,y_{d})$ then
$ERM_{\mathcal{H}}(\mathcal{S})=h^{*}$ and , therefore ,
$L_{\mathcal{S}}(ERM_{\mathcal{H}}(\mathcal{S}))=L_{\mathcal{S}}(h^{*})=0$.

  * •

$\Leftarrow|$ Let
$h_{0}=ERM_{\mathcal{H}}(\mathcal{S})=\underset{{h\in\mathcal{H}}}{\arg\min(L_{\mathcal{S}}(h))}$
s.t: $\mathcal{S}=\\{(c_{1},y_{1}),...,(c_{d},y_{d})\\}$.
Suppose that $L_{\mathcal{S}}(h_{0})=0$
Given that
$\begin{cases}L_{\mathcal{S}}(h_{0})=\frac{1}{d}\sum_{i=1}^{d}l(h_{0}(c_{i}),y_{i})\\\
\forall i\in\llbracket 1,d\rrbracket\;\quad l(h_{0}(c_{i}),y_{i})\geq
0\end{cases}$

( By definition, a loss function -in prediction problems
( i.e. $\mathcal{Z}=\mathcal{X}\times\mathcal{Y}$)- is : $l\text{ :
}\mathcal{H}\times\mathcal{Z}\rightarrow\mathbb{R}_{+}$)

Then $\forall i\in\llbracket 1,d\rrbracket$ : $l(h_{0}(c_{i}),y_{i})=0$.

Therefore : $\forall i\in\llbracket 1,d\rrbracket$ : $h_{0}(c_{i})=y_{i}$.
Namely, there exists $h^{*}=h_{0}=ERM_{\mathcal{H}}(\mathcal{S})$ such that:
$\forall i\in\llbracket 1,d\rrbracket$ :$h^{*}(c_{i})=y_{i}$

∎

######  Corollary 2.1.1.

Let $\mathcal{H}$ be a hypothesis class for binary classification. Let
$\mathcal{C}=\\{c_{1},...,c_{d}\\}\subset\mathcal{X}$.
$\mathcal{H}$ shatters $\mathcal{C}$ if and only if for every $d$-tuple
$(y_{1},...,y_{d})$ in $\\{0,1\\}^{d}$, the $ERM_{\mathcal{H}}$ over
$\mathcal{S}=\\{(c_{1},y_{1}),...,(c_{d},y_{d})\\}$ will give a zero empirical
loss. Formally:

$\forall(y_{1},...,y_{d})\in\\{0,1\\}^{d}:L_{\mathcal{S}}(ERM_{\mathcal{H}}(S))=0,\text{where
}\mathcal{S}=\\{(c_{1},y_{1}),...,(c_{d},y_{d})\\}$

###### Proof.

To prove this corollary we will exploit the previous Nechba-Mouhajir theorem
of shattering :

$\displaystyle\mathcal{H}\text{ shatters }\mathcal{C}=\\{c_{1},...,c_{d}\\}$
$\displaystyle\Leftrightarrow\text{ For every
}(y_{1},...,y_{d})\in\\{0,1\\}^{d}\text{, }\exists\;h^{*}\in\mathcal{H}\text{
s.t}:h^{*}(\mathcal{C})=(y_{1},...,y_{d})$ $\displaystyle\Leftrightarrow\text{
For every }(y_{1},...,y_{d})\in\\{0,1\\}^{d},\text{ the
}ERM_{\mathcal{H}}\text{ over
}\mathcal{S}=\\{(c_{1},y_{1}),...,(c_{d},y_{d})\\}$ gives a zero-Empirical
Loss.

∎

###  2.2 Nechba-Mouhajir algorithm for shattering

Now, we will be based on the previous corollary to construct our Nechba-
Mouhajir algorithm for shattering :

Data:

  * •

Hypothesis class $\mathcal{H}$

  * •

domain set $\mathcal{C}\subset\mathcal{X}$, (such that
$|\mathcal{C}|=d\in\mathbb{N}^{*}$)

Result: $\begin{cases}\text{True}&\text{if }\mathcal{H}\text{ shatters
}\mathcal{C}\\\ \text{False}&\text{if }\mathcal{H}\text{ doesn't shatter
}\mathcal{C}\end{cases}$

\- $All\\_combinations\leftarrow$ all d-tuple of $\\{0,1\\}^{d}$

for  _$(y_{1},...,y_{d})$ in $All\\_combinations$_ do

if  _$L_{\mathcal{S}}(ERM_{\mathcal{H}}(\mathcal{S}))\neq 0$ _ then

Return False

end if

end for

Return True

Algorithm 1 Nechba-Mouhajir algorithm for shattering

For a given hypothesis class $\mathcal{H}$ and a given domain set
$\mathcal{C}\subset\mathcal{X}$ ( whose size is some $d$), we will generate
all possible d-tuples of $\\{0,1\\}^{d}$. Then, for each d-tuple
$(y_{1},...,y_{d})$, we will consider our sample $\mathcal{S}$ to be the
superposition of $\mathcal{C}$ and $(y_{1},...,y_{d})$. Formally, if we
consider $\mathcal{C}=\\{c_{1},...,c_{d}\\}$ our sample $\mathcal{S}$ will be
:

$\mathcal{S}=\\{(c_{1},y_{1}),(c_{2},y_{2}),...,(c_{d},y_{d})\\}$

Then, we will run the ERM rule for $\mathcal{H}$ and over $\mathcal{S}$, and
compute the empirical loss of $ERM_{\mathcal{H}}(\mathcal{S})$ (i.e.
$L_{\mathcal{S}}(ERM_{\mathcal{H}}(\mathcal{S}))$ ). Then, according to our
previous corollary, if we find that
$L_{\mathcal{S}}(ERM_{\mathcal{H}}(\mathcal{S}))\neq 0$ we will quit and say
that "$\mathcal{H}$ doesn’t shatter $\mathcal{C}$". Otherwise, we will
continue until we find some adversarial d-tuple (that is, for which
$L_{\mathcal{S}}(ERM_{\mathcal{H}}(\mathcal{S}))\neq 0$ ), or we could not
find such a d-tuple. In such a case, we can say that our $\mathcal{H}$
shatters $\mathcal{C}$.

###  2.3 Overcoming the Practical Issues of the Algorithm

The proposed algorithm for assessing the shattering property has some
practical limitations that need to be addressed in future research. One such
limitation is the brute-forcing problem that arises from generating all
possible d-tuples of $\\{0,1\\}^{d}$. If an adversarial d-tuple is found early
in the process, the algorithm can terminate without examining all $2^{d}$
cases. However, if an adversarial d-tuple is not found, the algorithm will be
compelled to check all $2^{d}$ cases, leading to computational inefficiencies.
Alternative methods for generating d-tuples more efficiently can be explored
in future work to overcome this brute-force issue.

Another practical challenge is the computational time required by the
algorithm, particularly for large values of d. A possible solution to address
this issue is to leverage the power of GPUs by assigning a single d-tuple to
each of the $2^{d}$ threads and enabling parallel execution, which can
significantly reduce the computation time. For example, using an NVIDIA
GeForce RTX 3080 can allow for nearly $2^{13}$ parallel operations (see
NVIDIA’s website [1] for more details). However, for large d, the number of
threads required for parallelization may become infeasible. Therefore, future
work should investigate alternative approaches to optimize the computational
time of the algorithm.

##  3 Mouhajir-Nechba Algorithm of VC dimension and its Theoretical Basis

###  3.1 Preliminaries

Let $d\in\mathbb{N}^{*}$, $\mathcal{Y}=\\{0,1\\}$ and

$\displaystyle\mathcal{H}$ $\displaystyle\coloneqq\Bigl{\\{}\,h\mid
h\colon\mathcal{X}\to\mathcal{Y}=\\{0,1\\},\quad x\mapsto h(x)\,\Bigl{\\}}$

Let $Z$ be a random variable, such that :

$\begin{split}Z&=\text{`` VCdim($\mathcal{H}$) $<d$ ''}\\\ &=\text{``
$\mathcal{H}$ doesn't shatter any domain set $\mathcal{C}\subset\mathcal{X}$
s.t : $\mid\mathcal{C}\mid=d$ "}\end{split}$ (1)

The random variable $Z$ has two possible outcomes (i.e. $Z(\Omega)=\\{0,1\\}$
) :

$Z=\begin{cases}1&\text{ if `` VCdim($\mathcal{H}$) $<d$ ''}=\text{`` For
every domain set $\mathcal{C}\subset\mathcal{X}$ s.t : $\mid\mathcal{C}\mid=d$
, $\mathcal{H}$ doesn't shatter $\mathcal{C}$ "}\\\ 0&\text{ if ``
VCdim($\mathcal{H}$) $\geq d$ ''}=\text{`` There exist at least one domain set
$\mathcal{C}_{0}\subset\mathcal{X}$ s.t : }\text{ $\mid\mathcal{C}_{0}\mid=d$
and $\mathcal{H}$ shatters $\mathcal{C}_{0}$ "}\end{cases}$

Let $\mathbf{p}$ the probability of event “$Z=1$ " , namely
$\mathbf{p=\mathbb{P}(Z=1)}$ . Therefore , $Z$ is a Bernoulli random variable
: $Z\sim\mathcal{B}(\mathbf{p})$

###  3.2 The estimation of Bernoulli parameter $\mathbf{p}$

So far, we have proved that $Z\sim\mathcal{B}(\mathbf{p})$. But in order to
estimate the parameter $\mathbf{p}$ we will follow the inferential statistics
workflow :

  1. 1.

We will take $m$ samples (i.i.d) :
$(\mathcal{C}_{1},\mathcal{C}_{2},...,\mathcal{C}_{m})$ , s.t :

$\forall i\in\llbracket 1\mkern 1.5mu,\mkern
1.5mum\rrbracket:\mathcal{C}_{i}\subset\mathcal{X}\quad\text{and}\mid\mathcal{C}_{i}\mid=d$

  2. 2.

This is equivalent of taking m (i.i.d) random variables
$(Z_{1},Z_{2},..,Z_{m})$ that correspond to the realization of $Z$ on the
corresponding set $\mathcal{C}_{i}$, for all $i$ in $\llbracket 1\mkern
1.5mu,\mkern 1.5mum\rrbracket$. Formally, we will have
$(Z_{1},Z_{2},..,Z_{m})$ ,s.t:

    1. (a)

$Z_{i}$ = “ the realization of $Z$ upon $\mathcal{C}_{i}$ " = “ $\mathcal{H}$
doesn’t shatter $\mathcal{C}_{i}$ "

    2. (b)

$Z_{i}=\begin{cases}1&\text{if }\mathcal{H}\text{ doesn't shatter
}\mathcal{C}_{i}\\\ 0&\text{if }\mathcal{H}\text{ shatters
}\mathcal{C}_{i}\end{cases}$

  3. 3.

Thus, according to the law of large numbers:

$\overline{Z_{m}}=\frac{1}{m}\times\sum_{i=1}^{m}Z_{i}\quad\approx\quad(\mathbf{p}=\mathbb{E}(Z))$

######  Interpretation 3.1.

  * •

$\overline{Z_{m}}$ : is the frequency of the favorable event (i.e. “
$Z_{i}=1$" ). Namely $\overline{Z_{m}}$ is equal to how much time does
$\mathcal{H}$ not shatter a set $\mathcal{C}\subset\mathcal{X}$ of size $d$
for our given samples $(\mathcal{C}_{1},\mathcal{C}_{2},...,\mathcal{C}_{m})$.

  * •

$\overline{Z_{m}}=1$ : $\mathcal{H}$ doesn’t shatter any of sets
$\mathcal{C}_{i}$ in our given sample
$(\mathcal{C}_{1},\mathcal{C}_{2},...,\mathcal{C}_{m})$.

  * •

$\overline{Z_{m}}\neq 1$ : there exists at least one set $\mathcal{C}_{0}$ in
our sample $(\mathcal{C}_{1},\mathcal{C}_{2},...,\mathcal{C}_{m})$ such that
$\mathcal{H}$ shatters $\mathcal{C}_{0}$ .

###  3.3 How to choose the number of samples (i.e. $m$) so that
$\overline{Z_{m}}$ would be Probably Approximately equal to $\mathbf{p}$

Beforehand, we will give a recall of the Hoeffding’s inequality [6] :

######  Lemma 3.1 (Hoeffding’s inequality).

Let $\theta_{1},\theta_{2},...,\theta_{m}$ be a sequence of i.i.d random
variables , and assume that for all $i\in\llbracket 1\mkern 1.5mu,\mkern
1.5mum\rrbracket$ , $\mathbb{E}(\theta_{i})=\mu$ and
$\mathbb{P}(a\leq\theta_{i}\leq b)=1$. Then, for any $\epsilon>0$:

$\mathbb{P}(\quad\mid\mu-\frac{1}{m}\times\sum_{i=1}^{m}\theta_{i}\mid>\epsilon\quad)\leq
2\exp{\frac{-2m\epsilon^{2}}{(b-a)^{2}}}$

As we can see, the Hoeffding’s inequality determines, more precisly than the
law of large numbers, the convergence rate. By applying Hoeffding’s inequality
in our case, we get :

$\mathbb{P}(\quad\mid\mathbf{p}-\frac{1}{m}\times\sum_{i=1}^{m}Z_{i}\mid>\epsilon\quad)\leq
2e^{-2m\epsilon^{2}}$ (2)

Now, from this equation (2) we can determine ( with respect to the sample size
$m$ ) the precision $\epsilon$ and the confidence
$(1-\delta=1-e^{-2m\epsilon^{2}})$ that we want so as to make the
approximation : $\overline{Z_{m}}\approx\mathbf{p}$ .

In fact, in order to have the precision $\epsilon$ and confidence
$(1-\delta)$, we should sample
($\mathbf{m=\frac{1}{2\epsilon^{2}}\times\ln{\frac{2}{\delta}}}$) i.i.d sets
$\mathcal{C}_{i}$ of size $d$. Then, we can say, with confidence $(1-\delta)$
and precision $\epsilon$, that  “ $\overline{Z_{m}}=\mathbf{p}$ " .

######  Interpretation 3.2.

  * •

If we find that  “ $\overline{Z_{m}}=1$ " . Then, Probably Approximately,  “
$\mathbf{p}=1$ ".
In other words, we can say, with confidence $(1-\delta)$ and precision
$\epsilon$, that :  “ $\mathcal{H}$ doesn’t shatter any set
$\mathcal{C}\subset\mathcal{X}$ s.t : $\mid\mathcal{C}\mid=d$ "  which is
equivalent of saying  “VCdim$(\mathcal{H})<d$ " .

  * •

If we find that  “ $\overline{Z_{m}}\neq 1$ " . Then , Probably Approximately,
“ $\mathbf{p}\neq 1$ ".
In other words, we can say, with confidence $(1-\delta)$ and precision
$\epsilon$, that :  “ There exists at least one set
$\mathcal{C}_{0}\subset\mathcal{X}$ s.t $\mid\mathbf{C}\mid=d$ and
$\mathcal{H}$ shatters set $\mathcal{C}_{0}$"  which is equivalent of saying
“VCdim$(\mathcal{H})\geq d$ " .

###  3.4 Mouhajir-Nechba algorithm for VC-dimension

Now, after presenting all the theoretical bases, we will unravel our algorithm
to compute the Vapnik-Chervonenkis dimension :

Data:

  * •

Hypothesis class $\mathcal{H}$

  * •

the accuracy $\epsilon$

  * •

the confidence parameter $(1-\delta)$

  * •

$d_{MAX}$ : a large number (for example $2^{63}-1$) used to say that
$\mathcal{H}$ has infinite VCdim

Result: $\begin{cases}\text{some }d\neq d_{MAX}&\text{if }\mathcal{H}\text{
has finite VCdimension}\\\ d=d_{MAX}&\text{if }\mathcal{H}\text{ has infinite
VCdimension}\end{cases}$

\- Compute the sample size $\mathbf{m}$ (
$\mathbf{m\leftarrow\frac{1}{2\epsilon^{2}}\times\ln{\frac{2}{\delta}}}$ );

for  _$d=1,2,...d_{MAX}$_ do

$\text{Generate an }\mathbf{m}\text{ random samples of size }\mathbf{d}\text{
from }\mathcal{X}\text{ :
}\\{\mathcal{C}_{1},\mathcal{C}_{2},...,\mathcal{C}_{m}\\}$

$Z_{m}\leftarrow 0$

for  _$i=1,2,...,m$_ do

if  _N-M_Shattering( $\mathcal{H}$,$\mathcal{C}_{i}$) == False_ then

$Z_{m}\leftarrow Z_{m}+1$

end if

end for

if  _$Z_{m}==m$_ then

Return $d-1$

end if

end for

Return $+\infty$

Algorithm 2 Mouhajir-Nechba algorithm for VCdimension

After presenting all the theoretical foundations, we proceed to unveil our
algorithm for computing the Vapnik-Chervonenkis (VC) dimension.

For a given hypothesis class $H$, accuracy $\epsilon$, confidence
$(1-\delta)$, and a large number $d_{\text{MAX}}$ (which reflects the case
where the dimension of VC is infinite), we compute the sample size $m$ needed
to assert that the shattering property evaluated in each iteration of our
algorithm is Probably Approximately Correct (PAC) with probability
$(1-\delta)$.

For each size $d$ in the range $[1,d_{\text{MAX}}]$, we generate $m$ random
samples from the domain set $X$ with size $d$ (ie ${C_{1},C_{2},...,C_{m}}$
where $C_{i}\subseteq X$ and $|C_{i}|=d$). Then, for each $C_{i}$, we apply
the Nechba-Mouhajir algorithm for shattering1 (which will be refreed as
N-M_Shattering ) to determine if $H$ shatters $C_{i}$. If $H$ does not shatter
$C_{i}$, we increase $Z_{m}$ by 1. We repeat this process for all $C_{i}$ to
compute the final $Z_{m}$, which probably and approximately, according to
Hoeffding’s inequality 2, represents the probability of having the VC
dimension less than $d$ (i.e., the probability of the event
$\text{VCdim}(H)<d$). At this stage, we check if $Z_{m}=1$, in which case we
terminate the algorithm and output $\text{VC}(H)=d-1$. Otherwise, we perform
the same process for the next iteration $d+1$, until we reach
$d_{\text{MAX}}$, in which case we output $\text{VC}(H)=\infty$.

######  Remark.

It is complicated to calculate the error probability of Algorithm 2 2, but we
can make this probability small by choosing very small values for $\epsilon$
and $\delta$.

###  3.5 The complexity of our algorithm

To determine the complexity of our algorithm we should first determine the
complexity of N-M_Sattering($\mathcal{H},\mathcal{C}$) :

  1. 1.

All_combinations is of size $2^{d}$

  2. 2.

The complexity of each iteration in the for-loop is the complexity of finding
the output of the $ERM_{H}(S)$ and then computing its Emprirical loss $L_{S}$.
We will denote it as follow :

$t=\mathcal{O}(L_{S}(ERM_{\mathcal{H}}(\mathcal{S})))$

  3. 3.

in conclusion, the complexity of N-M_Sattering($\mathcal{H},\mathcal{C}$) is :

$\mathcal{O}(2^{d}\times t)$

Now , we could determine the complexity of our Mouhajir-Nechba algorithm for
VC dimension :

  1. 1.

In worse case (i.e. if $\mathcal{H}$ has infinite dimension) we will loop over
$d_{MAX}$ iterations.

  2. 2.

For each iteration, we will have a for-loop of complexity :

$\mathcal{O}(m\times 2^{d}\times t)$

  3. 3.

Thus, over all $d_{MAX}$ iterations the complexity will be :

$\mathcal{O}(\sum_{d=1}^{d_{MAX}}m\times 2^{d}\times
t)=\mathcal{O}(m\sum_{d=1}^{d_{MAX}}2^{d}\times t)$

##  4 Experimentation

The objective of this section is to showcase the efficiency of our algorithm
in computing the VC dimension and its applicability in real-world scenarios.
To achieve this goal, we will utilize the Mouhajir-Nechba algorithm for VC
dimension on pre-calculated mathematical classes. Specifically, we will focus
on the Half-Spaces class as a case study.

###  4.1 Theoretical results

The VC dimension of half-spaces is formally established by Shalev-Shwartz and
Ben-David in their book ’Understanding Machine Learning: From Theory to
Algorithms’. Specifically, in Chapter 9, Section 9.1.3 titled ’The VC
Dimension of Half-spaces’ [6], they prove the following theorem:

######  Theorem 4.1.

The VC dimension of the class of nonhomogenous Half-Spaces in $\mathbb{R}^{d}$
is $d+1$.

###  4.2 Practical considerations

The major challenge in our experiment is to represent the ERM of Half-Spaces.
Therefore, we will use Perceptron [5] as a specific implementation of this
paradigm .

According to the work of Shalev-Shwartz and Ben-David in their book
"Understanding Machine Learning: From Theory to Algorithms"[6], the ERM rule
for the class of half-Spaces can be represented in multiple ways, not just
through the use of the Perceptron algorithm. In particular, sections 9.1.1 and
9.1.2 of their book describe alternative methods :

  * •

For the realizable case solving the following linear program :

$\displaystyle\max_{x}\quad u^{T}x$ $\displaystyle Ax$ $\displaystyle\geq v$

where

    * –

$u\in\mathbb{R}^{d}$ is a "dummy" vector : $u=(0,\ldots,0)$ (since every
$w\in\mathbb{R}^{d}$ that satisfies the constraint is a solution to the ERM
rule).

    * –

$A$ is a matrix whose elements are $A_{i,j}=y_{i}x_{i,j}$, where $x_{i,j}$ is
the $j^{\text{th}}$ component of the $i^{\text{th}}$ example within the sample
$S=\\{(x_{1},y_{1}),\ldots,(x_{m},y_{m})$.

    * –

$v=(1,\ldots,1)\in\mathbb{R}^{m}$ .

  * •

For the Agnostic case: solving the non-convex optimization problem:

$h^{*}=\operatorname*{argmin}_{h}(L_{\mathcal{S}}(h)),$

whereas, the major problem here is to handle the non-convexity of 0-1 loss:
$l((x,h(x)),y)=\mathbb{1}_{\left(h(x)\neq y\right)}$. For this purpose, we can
use the sub-gradient method.

###  4.3 Mouhajir-Nechba Algorithm for VC Dimension as a Tool for Evaluating
ERM Implementation

Using the Mouhajir-Nechba algorithm for VC dimension, we propose a methodology
to assess the performance of a specific implementation of the ERM rule for a
hypothesis class $\mathcal{H}$.

Specifically, we determine the VC dimension of $\mathcal{H}$ under the given
implementation and compare it with the theoretical results. If the results
align within a certain tolerance level, we conclude that the implementation is
good.

For instance, for the class of Half-Spaces, the Perceptron implementation is
considered good if the Mouhajir-Nechba algorithm yields the same VC dimension
as expected theoretically with a specified level of confidence.

###  4.4 Experimental results

The experiment that we have conducted, for ($\epsilon=10^{-2}$ and
$\delta=10^{-2}$ ) has given the following results for both execution time and
VC dimension :

Table 1: VC-dimension of perceptron for dimension between 1 and 6

$dim(\mathcal{X})$ |  1 |  2 |  3 |  4 |  5 |  6
---|---|---|---|---|---|---
VC-dimension |  2 |  3 |  4 |  5 |  6 |  7
execution time (seconds) |  34.76 |  46.78 |  70.12 |  98.64 |  103.53 |  124.22

Figure 2: The computing time for Perceptron

The findings of our experiment can be summarized by two key observations.
Firstly, the output of our algorithm for the VC dimension is in perfect
alignment with the theoretical results. This indicates that our algorithm
produces accurate and dependable outcomes, and that the Perceptron serves as a
reliable implementation of the ERM principle for the half-spaces class.
Secondly, we observed that the computational time required by our algorithm
increases as the dimension of the domain set $\mathcal{X}$ increases. This
highlights the computational complexity of our algorithm, which can pose a
significant practical challenge. Nevertheless, we are confident that this
issue can be overcome by developing a PRAM version of our algorithm that
harnesses the processing power of acceleration devices.

##  5 Summary and future work

This paper addresses a fundamental challenge in the field of Machine Learning:
computing the VC dimension for binary classification concept classes beyond
the traditional discrete settings. To achieve this goal, we proposed a novel
characterization of the shattering property based on the Empirical Risk
Minimization paradigm. Our algorithm, which is built upon this
characterization, can handle the problem of VC dimension without any
constraints on the concept class or domain set.

However, the computational time required for our algorithm is an area of
concern, which can be mitigated to some extent by harnessing the computing
power of accelerators such as GPUs. Additionally, there may be challenges with
our algorithm if the probability of shattered sets is extremely low or if
there are issues with finite-precision arithmetic approximating concepts
defined in terms of irrational numbers.

## References

  * [1] NVIDIA GeForce RTX 3080 graphics card.  https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3080-3080ti/, 2021\.  Accessed on May 3, 2023.
  * [2] Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K Warmuth.  Learnability and the vapnik-chervonenkis dimension.  Journal of the ACM (JACM), 36(4):929–965, 1989.
  * [3] Nataly Brukhim, Daniel Carmon, Irit Dinur, Shay Moran, and Amir Yehudayoff.  A characterization of multiclass learnability.  In 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS), pages 943–955. IEEE, 2022.
  * [4] Nathan Linial, Yishay Mansour, and Ronald L Rivest.  Results on learnability and the vapnik-chervonenkis dimension.  Information and Computation, 90(1):33–49, 1991.
  * [5] F. Rosenblatt.  The perceptron: a probabilistic model for information storage and organization in the brain.  Cornell Aeronautical Laboratory, 1958.
  * [6] Shai Shalev-Shwartz and Shai Ben-David.  Understanding machine learning: From theory to algorithms.  Cambridge University Press, July 2014.
  * [7] Leslie G Valiant.  A theory of the learnable.  Communications of the ACM, 27(11):1134–1142, 1984.
  * [8] Vladimir N Vapnik and A Ya Chervonenkis.  On the uniform convergence of relative frequencies of events to their probabilities.  Measures of complexity: festschrift for Alexey Chervonenkis, pages 11–30, 2015.
  * [9] Roberta S Wenocur and Richard M Dudley.  Some special vapnik-chervonenkis classes.  Discrete Mathematics, 33(3):313–318, 1981.


# Surrogate Modeling of Melt Pool Thermal Field using Deep Learning

AmirPouya Hemmasian  Francis Ogoke  Parand Akbari  Jonathan Malen  Jack Beuth
Amir Barati Farimani

###### Abstract

Powder-based additive manufacturing has transformed the manufacturing industry
over the last decade. In Laser Powder Bed Fusion, a specific part is built in
an iterative manner in which two-dimensional cross-sections are formed on top
of each other by melting and fusing the proper areas of the powder bed. In
this process, the behavior of the melt pool and its thermal field has a very
important role in predicting the quality of the manufactured part and its
possible defects. However, the simulation of such a complex phenomenon is
usually very time-consuming and requires huge computational resources. Flow-3D
is one of the software packages capable of executing such simulations using
iterative numerical solvers. In this work, we create three datasets of single-
trail processes using Flow-3D and use them to train a convolutional neural
network capable of predicting the behavior of the three-dimensional thermal
field of the melt pool solely by taking three parameters as input: laser
power, laser velocity, and time step. The CNN achieves a relative Root Mean
Squared Error of 2% to 3% for the temperature field and an average
Intersection over Union score of 80% to 90% in predicting the melt pool area.
Moreover, since time is included as one of the inputs of the model, the
thermal field can be instantly obtained for any arbitrary time step without
the need to iterate and compute all the steps.

###### keywords:

Additive Manufacturing , Laser Powder Bed Fusion , Meltpool Temperature ,
Convolutional Neural Network , Surrogate Model

††journal: arXiv

[inst1]organization=Department of Mechanical Engineering, Carnegie Mellon
University,city=Pittsburgh, postcode=15213, state=PA, country=USA
[inst2]organization=Department of Materials Science and Engineering, Carnegie
Mellon University,city=Pittsburgh, postcode=15213, state=PA, country=USA
[inst3]organization=Department of Chemical Engineering, Carnegie Mellon
University,city=Pittsburgh, postcode=15213, state=PA, country=USA
[inst4]organization=Machine Learning Department, Carnegie Mellon
University,city=Pittsburgh, postcode=15213, state=PA, country=USA

##  1 Introduction

Metal additive manufacturing (MAM) has attracted increasing scientific
attention from industry and academia owing to its noticeable advantages
relative to conventional manufacturing methods [1]. It enables the
manufacturing of complex geometries with higher reliability and reduced
material consumption and lead time. [2, 3] Despite these benefits, controlling
the quality of the printed parts is a major challenge as defects such as lack
of fusion, keyhole, and balling can compromise the structural integrity of the
additively manufactured components. [4] Since these defects are rooted in the
dynamics of the melt pool– a locally melted zone generated by the emission of
the laser on the powder particles– monitoring and controlling the melt pool’s
geometry and thermal field are of critical importance. [3]

To monitor the melt pool in the Laser Powder Bed Fusion (LPBF) process,
experiments can be conducted. However, they are expensive and need laborious
preparation and adjustment. In addition, employing simulation methods such as
Flow-3D Software[5] to generate computational predictions in such a complex
multi-scale and multi-physics phenomenon is time-consuming, and also requires
huge computational resources and integration of multiple discrete and costly
simulations. [6] Flow-3D is one of the software packages capable of executing
such simulations utilizing iterative numerical solvers.

As an alternative to the computationally expensive numerical solvers like
Flow-3D, we leverage the power of deep learning to make fast and accurate
predictions of the three-dimensional thermal field of the melt pool over time.
In the last decade, deep learning has caused a revolution in the field of
computer vision [7, 8]. Deep neural networks have shown success in many
challenging areas like computer vision [7, 9] and speech recognition [10, 11,
12, 13]. Following the introduction of convolutional neural networks (CNNs) by
Lecun et al.[14] and the demonstrated capability of AlexNet on the image
classification benchmark [7], these networks have become widely used in many
computer vision tasks including image recognition and segmentation[15, 16], as
well as video classification[17]. Inspired by this, many researchers have been
applying deep learning models in science and engineering problems [18, 19,
20]. Deep learning has also been utilized to study complex physical systems by
obtaining accurate solutions to their systems of governing equations, which
are usually ordinary or partial differential equations[21, 22, 23]. Due to the
high computational cost of conventional numerical solvers, developing
surrogate models to study physical systems has been of great interest to many
researchers and scientists as well[24, 25, 26]. Convolutional neural networks
are well-suited for dealing with spatially structured data due to their
inherent translational invariance, weight sharing, and feature extraction
properties. They have been applied to obtain rapid prediction and simulation
of complex phenomena like reaction-diffusion systems [27] and convective fluid
flows [28].

In this work, we approximate the thermal field and geometry of the melt pool
during the LPBF process by utilizing the capabilities of CNNs. Our model’s
input is simply a vector containing three numbers: The laser beam’s power and
velocity, and the time step. It succeeds in learning the dominant pattern of
the melt pool’s thermal field for a wide variety of process parameters. We
evaluate the performance of our model qualitatively by visual comparison with
the results from Flow-3D, and quantitatively by measuring the Root Mean
Squared Error for the thermal field and Intersection over Union for the melt
pool region.

##  2 Dataset

Table 1: The datasets used in this work. name | alloy | mesh | powder | train | val | P(w) | V(mm/s)
---|---|---|---|---|---|---|---
Ti64-5m | Ti-6Al-4V | $5\mu m$ | No | 39 | 7 | 100-520 | 400-1200
Ti64-10m | Ti-6Al-4V | $10\mu m$ | No | 287 | 71 | 100-460 | 100-1500
Ti64-10m-p | Ti-6Al-4V | $10\mu m$ | Yes | 555 | 138 | 100-560 | 100-1400
SS-10m-p | SS316L | $5\mu m$ | Yes | 32 | 7 | 100-320 | 100-400

A summary of the datasets created for this work can be found in Table 1. Each
simulation contains 100 timesteps spaced at 5 $\mu s$ intervals. The FLOW-3D
v11.2 CFD package was used to simulate the melting process [5]. Generally,
coupled PDEs describing the mass, momentum, and energy conservation present in
the system are solved using a volume of fluid (VOF) model. These equations are
reproduced below:

$\nabla\cdot(\rho\vec{v})=0$ (1)

$\frac{\partial\vec{v}}{\partial
t}+(\vec{v}\cdot\nabla)\vec{v}=-\frac{1}{\rho}\nabla\vec{P}+\mu\nabla^{2}\vec{v}+\vec{g}(1-\alpha(T-T_{m}))$
(2)

$\frac{\partial h}{\partial
t}+\left(\vec{v}\cdot\nabla\right)h=\frac{1}{\rho}\left(\nabla\cdot k\nabla
T\right)$ (3)

where $\vec{P}$ is the pressure of the system, $\vec{g}$ represents gravity,
$\alpha$ is the coefficient of thermal expansion, $\rho$ is the density of the
material, $h$ is the specific enthalpy, $\vec{v}$ is the velocity of the
system and $k$ is the heat conductivity of the material. The thermal
parameters, $C_{p}$, $k$, as well as the density, $\rho$, are assumed to vary
with temperature, with their functional relationships described in [29] for
both materials. While the material is assumed to be incompressible in these
simulations, however, there are accommodations for the variation of density
with temperature. To do so, the incompressible flow algorithm is solved using
a variable density, calculated based on the local temperature values.

To compute the dynamics of the free surface of the melt pool, the VOF method
is used to define the fluid configuration [30]. A conservation equation can be
written for the Volume of Fluid function, $F(x,y,z,t)$, as demonstrated in
equation 4. When $F=1$, the mesh cell contains fluid, whereas when $F=0$, a
void region is defined. This void region represents areas that are filled with
a gas with a much lower density than the melted substrate. The gas dynamics in
the void regions are not explicitly modeled, they are assigned a uniform
pressure to reduce the computational expense of the simulation.

$\frac{\partial F}{\partial t}+\left(\nabla F\cdot v\right)=0$ (4)

The laser beam is expressed as a Gaussian heat source, with a distribution
described by

$q=\frac{P}{r_{0}^{2}\pi}\exp\left\\{-\left(\frac{\sqrt{2}r}{r_{0}}\right)^{2}\right\\}$
(5)

where $P$ is the laser power and $r_{0}$ is the laser beam radius. During
simulation, the heat flux is only applied to the top layer of surface cells
and propagates through the rest of the domain through heat transfer
mechanisms.

The evaporation mechanisms are also simulated, to model the ejection of vapor
during the melting process. The recoil pressure is given by

$P_{s}=aP_{V}\exp\left\\{\frac{\Delta
H_{v}}{(\gamma-1)C_{v}^{vap}T_{v}}\left(1-\frac{T_{v}}{T}\right)\right\\}$ (6)

Here, the accommodation coefficient $a$, defines the ratio of the energy
exchanged with the vapor phase, to the maximum energy exchange that is
thermodynamically possible. $\Delta H_{v}$ represents the latent heat of
vaporization, $\gamma$ is the ratio of specific heats, $C_{v}^{vap}$ is the
constant volume heat capacity of the vapor, and $P_{V}$, $T_{V}$ describe the
saturation temperature and pressure respectively. Accordingly, the total mass
loss, $\dot{M}$, due to this evaporation process is also taken into account,
with the following relationship.

$\dot{M}=a\sqrt{\frac{1}{2\pi R\bar{T}}}(P_{l}^{sat}-P_{p})$ (7)

In the above equation, $R$ refers to the gas constant, $\bar{T}$ describes the
average temperature of the liquid along the free surface, and $P_{p}$ is the
partial pressure of the metal vapor.

During the melting process, the keyhole that is formed in high energy density
situations can increase the absorptivity of the melt pool, due to the laser
beam reflecting within the keyhole cavity [31]. To account for this effect,
the relationship between the power, velocity, and absorptivity is determined
based on a scaling relationship introduced in [32].

This scaling relationship reproduced in Equations 8 and 9 determines the
absorptivity $A$ as a function of the beam radius, $r_{0}$, thermal
diffusivity, $D$, melting enthalpy $H_{m}$, the minimum absorptivity on a flat
plate, $A_{m}$, and the processing parameters. Generally, as the energy
density of the laser increases, the keyhole deepens, and the local
absorptivity increases.

$A=\left(0.70(1-e^{-0.66y})\right)$ (8)

$y=\left(\frac{A_{m}PD}{\pi
vH_{m}r_{0}^{2}\sqrt{Dr_{0}^{2}/v}}(r_{0}\sqrt{Dr_{0}/v})\right)$ (9)

This simulation was performed on a regular cartesian mesh, with mesh elements
of size 5 $\mu m$. A complete list of the values for material parameters used
to simulate the melt pool behavior can be found in the Appendix, in Tables 3
and 4.

Flow-3D uses a Volume-Of-Fluid (VOF) approach towards simulating the heat
transfer dynamics, which depending on the resolution of the simulation, can
cause isolated mesh elements to take on artificially high temperatures. This
can result when the laser impacts small, disconnected ejected particles, or
when the dynamics of the fluid flow are chaotic and simulated at relatively
coarse resolutions. To resolve this issue, we take the 99.9% percentile of the
temperature values above the melting point and clipped the values above that
temperature. The room temperature is 293K and the 99.9% percentile was 6500K,
so we use these values as $T_{min}$ and $T_{max}$ respectively. This filtering
process allows us to use a mesh element size that is sufficiently coarse to
enable the generation of large amounts of data while preserving the correct
temperature distribution. A comparison between the generated Flow-3D data and
reported experimental data from literature is made in Figure 11. This
comparison is made for the melt pool depth for SS316L and the keyhole depth in
Ti-6Al-4V. Both comparisons are made within the keyhole regime where
analytical models such as the Rosenthal equation are no longer applicable
[33]. This demonstrates that the scaled absorptivity is able to simulate the
increase in melt pool depth caused by the multiple reflection effect in the
keyhole.

In order to feed the data to our network, we normalize the temperature values
of the thermal field to be on the order $\sim 10^{0}$ for efficient training
of the neural network model. Therefore, we apply a linear transformation to
the temperature values to map $T_{min}$ and $T_{max}$ to 0 and 1 respectively,
and rescale the other temperature values appropriately. We also crop the
temperature field to a region of interest to train the model directly on the
area where the melt pool is located at a given point in time. This
modification reduces the amount of memory required for training and improves
the learning process.

##  3 Methodology

We begin this section by explaining the architecture of the main deep learning
model of our method. Then, we continue by introducing an auxiliary network
that improves the performance of the original model and how it is trained.

###  3.1 Network architecture

Figure 1a provides a schematic of our model’s architecture. Our Temperature
CNN (T-CNN) consists of a fully connected layer followed by several 3D
convolutional layers. The input vector simply contains the laser beam’s
power(P) and velocity(V) and time (t). After a feature vector is constructed
from the input by the fully connected layer, it is reshaped into a coarse-grid
feature map. This can be interpreted as a learned feature vector for each
point of a 3D coarse-grid tensor. This feature map is then passed through
several upsampling and convolutional layers to increase the spatial resolution
and decrease the number of channels. An illustration of the feature map’s
journey throughout the CNN can be found in Figure 1b. The shapes of the
feature maps are specific to the Ti64-5m dataset, but other aspects of the
model remain the same for all datasets. All convolutional layers use kernel
size 3 and stride 1 with ’same’ padding. The scale factor for the trilinear
upsampling layers is 2. We can observe that after the first two layers, the
overall pattern and geometry of the melt pool are captured and some finer
features are then learned in the following layers. We can also notice the
keyhole’s geometry as one of the learned feature maps in the second to last
layer for this particular example.

Figure 1: a) The overall schematic of the architecture of T-CNN. b) The
convolutional upsampling layers and the feature maps through the network. The
number of channels and the dimensions of each feature map are noted above and
below it respectively. c) The operators used in the network.

Regarding the hyperparameters of our model, the first one is the number of
channels of the first spatial feature map (128 in Figure 1b). Choosing a lower
number of channels (64, 32, …) resulted in a noticeable deterioration in the
performance of the model and bigger values (256, 512, …) did not make a
significant improvement, so we chose 128 as the final choice. There is also
the choice of using a deeper fully connected network for the initial feature
vector construction, but adding more hidden layers to the initial network did
not lead to any significant improvement either. We also considered the option
to use transposed convolution instead of non-parametric upsampling like the
trilinear method, but it led to the checkerboard effect and no significant
difference in the performance, so non-parametric upsampling was chosen to have
a simpler and faster network. In order to find the best activation function,
we tried using ELU, ReLU, SeLU, CELU, and Leaky ReLU, and all of them had a
similar outcome, so we chose Leaky ReLU for faster optimization and lower
computational cost. The final layer’s activation function is called a ”Valved”
Leaky ReLU because it is a Leaky ReLU in the training phase but a ReLU in the
testing phase. Therefore, we call it a valved Leaky ReLU since we open and
close the leak manually. This is because negative output values are not
physically possible (temperature lower than the ambient temperature), so we
prevent the model from predicting negative outputs. However, we need to design
the model in a way that it can learn to predict positive outputs, so we let
the last activation be leaky in the training phase so that there is a gradient
that can make the network learn not to predict negative values and tweak its
parameters in the right way.

This problem is a regression task on the 3D thermal field, so we use the mean
squared error (MSE) as the loss function to train our model. This loss is
calculated by taking the average of the squared error across all mesh elements
in the 3D thermal field. Therefore, the loss for a single sample is defined as
follows:

$MSE=\frac{1}{LWD}\sum_{x=1}^{L}\sum_{y=1}^{W}\sum_{z=1}^{D}[\hat{T}(x,y,z)-T(x,y,z)]^{2}$
(10)

where $\hat{T}$ is the predicted temperature value at the mesh element with
the positional index $[x,y,z]$ of the thermal field made by our T-CNN, $T$ is
the value for the same point from the output of Flow-3D, and L, W, D represent
the length, width, and depth of the cartesian mesh respectively. We use the
Adam optimizer[34] with a learning rate of 0.0002 to optimize the parameters
of the network and minimize the loss function. We also utilize PyTorch’s
learning rate scheduler to decrease the learning rate when the learning curve
reaches a plateau. The factor for the scheduler is set to 0.2 and patience is
set to 3, meaning if the average training loss does not improve after 3
consecutive epochs, the learning rate is multiplied by 0.2.

###  3.2 Using an auxiliary masker network

After training T-CNN, we observed appealing results in terms of visual quality
and achieved an average relative Root Mean Square Error of 2.5% on predicting
the thermal field. However, there is a delicate issue with our model which is
caused by how the data is defined. If you pay close attention to the output of
the model in Figure 1a, you may notice a thin layer in the boundary of the
keyhole, which is visible between the empty area of the keyhole and the high
temperature values of the melt pool. The reason behind this is the value of
the thermal field in the keyhole space. The temperature value that Flow-3D
assigns to the points not occupied by metal particles is equal to the room
temperature. Since the output of T-CNN is continuous, it cannot suddenly
change from a few thousand Kelvins to room temperature. Therefore, we see a
thin layer with intermediate temperature values between room temperature and
very hot values since the temperature decreases gradually over a few mesh
elements.

Figure 2: A simple illustration of the role of the auxiliary M-CNN. a) The
initial T-CNN model’s functionality. b) How M-CNN and TM-CNN are trained. c)
The MT-CNN model augmented with M-CNN

In order to make the output of the model more realistic and avoid creating
this thin layer, we use our pre-trained T-CNN and train an auxiliary network
called the masker CNN (M-CNN). The architecture of this model is the same as
Figure 1a, except that the final layer’s activation is the sigmoid function.
The job of M-CNN is to predict a binary mask for the keyhole. To be more
precise, its output is supposed to estimate the areas which are assigned the
exact value of room temperature by Flow-3D. That includes both the keyhole
space and the regions of the metal where the heat has not reached yet. After
training the M-CNN to learn this task, we train a new Masked Temperature CNN
(MT-CNN) to predict the thermal field. Now, the values in the keyhole space
are set to room temperature by the mask provided by M-CNN, and MT-CNN’s output
is neglected in that space. Therefore, MT-CNN can freely predict high
temperature values in that region without having to try to predict the room
temperature for the keyhole at the same time. A simple comparison between
T-CNN and MT-CNN and their outputs can be found in Figure 2.

##  4 Results and discussion

Figure 3: Performance of MT-CNN on some validation samples from the Ti64-5m
dataset. Note that the original data is 3D, and the cross-section of the
thermal field is shown in these plots.

We begin with a qualitative comparison between our model’s predictions and the
results of the direct numerical simulations obtained from Flow-3D. Figure 3
contains a few of the validation samples from the Ti64-5m dataset. We can see
that the predicted thermal field closely matches the ground truth for the most
part. Quantitative metrics will be explained later. We can say that the only
location where the model’s prediction noticeably differs from the ground truth
is the keyhole area. This is because this area contains the most turbulent and
transient behavior compared to the other parts of the melt pool, especially in
high energy density regimes. Since the model is trained with a simple MSE
loss, it only learns the dominant pattern and geometry of the thermal field
and fails to capture the turbulent phenomena happening at the keyhole.
However, learning the overall geometry is indeed the main purpose of this
work, because important features like the keyhole’s depth can be obtained from
it. After all, losing unimportant information is the trade-off that we are
willing to take to obtain the important information more quickly and
efficiently.

Table 2: The mean and standard deviation of the quantitative evaluation metrics. | T-CNN | MT-CNN
---|---|---
Dataset | RMSE | IoU | RMSE | IoU
Ti64-5m | $2.55\pm 1.04\%$ | $81.59\pm 14.67\%$ | $2.81\pm 1.21\%$ | $84.34\pm 12.46\%$
Ti64-10m | $2.18\pm 1.25\%$ | $79.17\pm 15.52\%$ | $2.45\pm 1.39\%$ | $80.53\pm 14.09\%$
Ti64-10m-p | $2.00\pm 0.79\%$ | $83.33\pm 10.06\%$ | $2.19\pm 0.89\%$ | $85.26\pm 7.82\%$
SS316L-5m-p | $3.37\pm 1.11\%$ | $81.48\pm 10.51\%$ | $3.73\pm 1.30\%$ | $81.86\pm 8.31\%$

For the quantitative evaluation of our model’s performance, we use two
metrics: Root-Mean-Square Error (RMSE) on the thermal field and Intersection
over Union (IoU) on the melt pool. We consider each mesh element with a
temperature value higher than the melting point as liquid. The melting point
is set as the average of liquidus and solidus temperature, which are provided
in Tables 3 and 4. Looking at Figure 3, we can see that the metrics achieved
by the model also indicate its success. The performance of the model for other
datasets is summarized in Table 2. The performance of T-CNN is also included
for the sake of comparison. We can see that despite the increase in the RMSE,
the IoU score is improved by MT-CNN.

Figure 4: The summarized results for Ti64-5m dataset. a) The average
performance of the model over the data distribution. The test samples are
marked by a black edge. b) The performance of the model over time. The main
line indicates the median value for the metric, and the shaded area indicates
the interquartile range.

In order to investigate the performance of the model further, we will look
into the distribution of our evaluation metrics over time and the data
distribution. Figure 4a shows the averaged value of the metrics over time for
each sample of the dataset. We can see that there is a noticeable correlation
between the performance of the model and the energy density. In the bottom
right of these plots, the energy density is low and the melt pool is
relatively small. Since most of the thermal field is still unaffected by the
heat from the laser beam, both the ground truth and the prediction are still
mostly at room temperature, therefore the RMSE is lower in this regime. It is
also observed that the IoU score is lower than the other regions of the plot
which is due to the small size of the melt pool. With a small melt pool, the
IoU score is affected more severely by false predictions of the model. Moving
to the high energy density regime in the upper left of the plots, we can
observe an increase in the RMSE and the IoU score because of the larger size
of the melt pool. The turbulence in the keyhole greatly increases the RMSE in
these regimes, but the IoU score is still high and the model has succeeded in
estimating the melt pool’s overall geometry.

Figure 4b shows the distribution of the metrics over the dataset for each time
step. We can see that both the RMSE and the IoU score are much lower in the
initial time steps. This is because of the small size of the melt pool as
explained before. The IoU score is very low at the beginning of the process
which decreases the overall average for the metric. However, we can see that
the model achieves an IoU score of 90% for most time steps which again shows
its successful estimation of the melt pool geometry. The slight decrease of
IoU in the last few time steps is due to the solidification of the metal which
happens only for a few samples where the velocity is too high and the laser
beam exits the modeled part. Since there is very little data on this
phenomenon, the model has not learned to predict the thermal field in such a
situation compared to the earlier time steps.

In this section, we compared the predictions of our model and the ground truth
for a qualitative evaluation of MT-CNN and further analyzed the results by
looking closer into the distribution of the RMSE and the IoU score over the
dataset and through time. We chose the Ti64-5m dataset to elaborate on in this
section because the model successfully achieved its goal despite having only
46 sampled processes and the highest output dimensionality among other
datasets. Similar plots and figures for the other datasets can be found in B.
The model achieves similar results for all datasets, but it does not work as
well for the SS-5m-5 dataset. This can be because of the low number of data
samples and also the higher amount of turbulent phenomena in this dataset.
However, we can see that our model still succeeds in learning the main
geometry of the melt pool in all datasets by looking at the IoU score and
comparing the predictions with the ground truth.

##  5 Conclusion and future work

In this paper, we proposed a deep learning framework that successfully
predicts the thermal field and the dominant melt pool geometry in LPBF
additive manufacturing processes. Compared to the Flow-3D software which uses
iterative numerical solvers, this model has a huge computational advantage.
Since time is included as one of the inputs to our CNN, we can obtain the
thermal field at any arbitrary time instantly without the need to run a whole
simulation. For the Ti-6Al-4V alloy, the model achieved an average relative
RMSE of 2 to 3% on predicting the thermal field and an average IoU of 80 to
90% on predicting the melt pool area. We had a limited dataset for the SS316L
dataset and the model achieved a relative RMSE of 3.7% and an IoU score of 82%
which is not as good as Ti-6Al-4V. However, the model successfully captures
the dominant pattern and features of the melt pool for all datasets used in
this work.

## References

  * [1] T. DebRoy, H. Wei, J. Zuback, T. Mukherjee, J. Elmer, J. Milewski, A. M. Beese, A. d. Wilson-Heid, A. De, W. Zhang, Additive manufacturing of metallic components–process, structure and properties, Progress in Materials Science 92 (2018) 112–224.
  * [2] J. Jiang, Y. Ma, Path planning strategies to optimize accuracy, quality, build time and material use in additive manufacturing: a review, Micromachines 11 (7) (2020) 633.
  * [3] P. Akbari, F. Ogoke, N.-Y. Kao, K. Meidani, C.-Y. Yeh, W. Lee, A. B. Farimani, Meltpoolnet: Melt pool characteristic prediction in metal additive manufacturing using machine learning, Additive Manufacturing (2022) 102817doi:https://doi.org/10.1016/j.addma.2022.102817.
URL https://www.sciencedirect.com/science/article/pii/S2214860422002172

  * [4] C. Wang, X. Tan, S. Tor, C. Lim, Machine learning in additive manufacturing: State-of-the-art and perspectives, Additive Manufacturing 36 (2020) 101538.  doi:https://doi.org/10.1016/j.addma.2020.101538.
  * [5] I. Flow Science, FLOW-3D, Version 12.0, Santa Fe, NM (2019).
URL https://www.flow3d.com/

  * [6] F. Yan, Y.-C. Chan, A. Saboo, J. Shah, G. B. Olson, W. Chen, Data-driven prediction of mechanical properties in support of rapid certification of additively manufactured alloys, Computer Modeling in Engineering & Sciences 117 (3) (2018) 343–366.
  * [7] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, Advances in neural information processing systems 25 (2012).
  * [8] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., Human-level control through deep reinforcement learning, nature 518 (7540) (2015) 529–533.
  * [9] A. Voulodimos, N. Doulamis, A. Doulamis, E. Protopapadakis, Deep learning for computer vision: A brief review, Computational intelligence and neuroscience 2018 (2018).
  * [10] G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, et al., Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups, IEEE Signal processing magazine 29 (6) (2012) 82–97.
  * [11] L. Deng, G. Hinton, B. Kingsbury, New types of deep neural network learning for speech recognition and related applications: An overview, in: 2013 IEEE international conference on acoustics, speech and signal processing, IEEE, 2013, pp. 8599–8603.
  * [12] A. Graves, A.-r. Mohamed, G. Hinton, Speech recognition with deep recurrent neural networks, in: 2013 IEEE international conference on acoustics, speech and signal processing, Ieee, 2013, pp. 6645–6649.
  * [13] O. Abdel-Hamid, A.-r. Mohamed, H. Jiang, L. Deng, G. Penn, D. Yu, Convolutional neural networks for speech recognition, IEEE/ACM Transactions on audio, speech, and language processing 22 (10) (2014) 1533–1545.
  * [14] Y. LeCun, Y. Bengio, et al., Convolutional networks for images, speech, and time series, The handbook of brain theory and neural networks 3361 (10) (1995) 1995.
  * [15] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556 (2014).
  * [16] J. Long, E. Shelhamer, T. Darrell, Fully convolutional networks for semantic segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3431–3440.
  * [17] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, L. Fei-Fei, Large-scale video classification with convolutional neural networks, in: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2014, pp. 1725–1732.
  * [18] J. N. Kutz, Deep learning in fluid dynamics, Journal of Fluid Mechanics 814 (2017) 1–4.
  * [19] T. Kirchdoerfer, M. Ortiz, Data-driven computational mechanics, Computer Methods in Applied Mechanics and Engineering 304 (2016) 81–101.
  * [20] A. Oishi, G. Yagawa, Computational mechanics enhanced by deep learning, Computer Methods in Applied Mechanics and Engineering 327 (2017) 327–351.
  * [21] J. Sirignano, K. Spiliopoulos, Dgm: A deep learning algorithm for solving partial differential equations, Journal of computational physics 375 (2018) 1339–1364.
  * [22] J. Han, A. Jentzen, E. Weinan, Solving high-dimensional partial differential equations using deep learning, Proceedings of the National Academy of Sciences 115 (34) (2018) 8505–8510.
  * [23] M. Raissi, P. Perdikaris, G. E. Karniadakis, Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, Journal of Computational physics 378 (2019) 686–707.
  * [24] L. Liang, M. Liu, C. Martin, W. Sun, A deep learning approach to estimate stress distribution: a fast and accurate surrogate of finite-element analysis, Journal of The Royal Society Interface 15 (138) (2018) 20170844.
  * [25] Z. L. Jin, Y. Liu, L. J. Durlofsky, Deep-learning-based surrogate model for reservoir simulation with time-varying well controls, Journal of Petroleum Science and Engineering 192 (2020) 107273.
  * [26] M. Tang, Y. Liu, L. J. Durlofsky, A deep-learning-based surrogate model for data assimilation in dynamic subsurface flow problems, Journal of Computational Physics 413 (2020) 109456.
  * [27] A. Li, R. Chen, A. B. Farimani, Y. J. Zhang, Reaction diffusion system prediction based on convolutional neural network, Scientific reports 10 (1) (2020) 1–9.
  * [28] C. Jiang, A. B. Farimani, Deep learning convective flow using conditional generative adversarial networks, arXiv preprint arXiv:2005.06422 (2020).
  * [29] K. C. Mills, Recommended values of thermophysical properties for selected commercial alloys, Woodhead Publishing, 2002.
  * [30] C. W. Hirt, B. D. Nichols, Volume of fluid (vof) method for the dynamics of free boundaries, Journal of computational physics 39 (1) (1981) 201–225.
  * [31] J. Trapp, A. M. Rubenchik, G. Guss, M. J. Matthews, In situ absorptivity measurements of metallic powders during laser powder-bed fusion additive manufacturing, Applied Materials Today 9 (2017) 341–349.
  * [32] J. Ye, S. A. Khairallah, A. M. Rubenchik, M. F. Crumb, G. Guss, J. Belak, M. J. Matthews, Energy coupling mechanisms and scaling behavior associated with laser powder bed fusion additive manufacturing, Advanced Engineering Materials 21 (7) (2019) 1900185.
  * [33] P. Promoppatum, S.-C. Yao, P. C. Pistorius, A. D. Rollett, A comprehensive comparison of the analytical and numerical prediction of the thermal history and solidification microstructure of inconel 718 products made by laser powder-bed fusion, Engineering 3 (5) (2017) 685–694.
  * [34] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014).
  * [35] J.-B. Forien, N. P. Calta, P. J. DePond, G. M. Guss, T. T. Roehling, M. J. Matthews, Detecting keyhole pore defects and monitoring process signatures during laser powder bed fusion: A correlation between in situ pyrometry and ex situ x-ray radiography, Additive Manufacturing 35 (2020) 101336.
  * [36] R. Cunningham, C. Zhao, N. Parab, C. Kantzos, J. Pauza, K. Fezzaa, T. Sun, A. D. Rollett, Keyhole threshold and morphology in laser melting revealed by ultrahigh-speed x-ray imaging, Science 363 (6429) (2019) 849–852.

##  Appendix A Material Parameters

Table 3: Material Parameters used to simulate the Ti-6Al-4V melting process Parameter | Value | Units |  |
---|---|---|---|---
Density, $\rho$, 298 K | 4420 | kg/m3 |  |
Density, $\rho$, 1923 K | 3920 | kg/m3 |  |
Specific Heat, $C_{v}$, 298 K | 546 | J/kg/K |  |
Specific Heat, $C_{v}$, 1923 K | 831 | J/kg/K |  |
Vapor Specific Heat, $C_{v},vapor$ | 600 | J/kg/K |  |
Thermal Conductivity, $k$, 298 K | 7 | W/m/K |  |
Thermal Conductivity, $k$, 1923 K | 33.4 | W/m/K |  |
Viscosity | 0.00325 | kg/m/s |  |
Liquidus Temperature, $T_{L}$ | 1923 | K |  |
Solidus Temperature, $T_{S}$ | 1873 | K |  |
Latent heat of fusion | 2.86 $\times 10^{5}$ | J/kg |  |
Latent heat of vaporization | 6.00 $\times 10^{4}$ | J/kg |  |
Table 4: Material Parameters used to simulate the SS316L melting process Parameter | Value | Units |  |
---|---|---|---|---
Density, $\rho$, 298 K | 7950 | kg/m3 |  |
Density, $\rho$, 1923 K | 7249 | kg/m3 |  |
Specific Heat, $C_{v}$, 298 K | 470 | J/kg/K |  |
Specific Heat, $C_{v}$, 1923 K | 726 | J/kg/K |  |
Vapor Specific Heat, $C_{v},vapor$ | 600 | J/kg/K |  |
Thermal Conductivity, $k$, 298 K | 13.4 | W/m/K |  |
Thermal Conductivity, $k$, 1923 K | 29.0 | W/m/K |  |
Viscosity | 0.008 | kg/m/s |  |
Liquidus Temperature, $T_{L}$ | 1694 | K |  |
Solidus Temperature, $T_{S}$ | 1717 | K |  |
Latent heat of fusion | 2.6 $\times 10^{5}$ | J/kg |  |
Latent heat of vaporization | 6.00 $\times 10^{4}$ | J/kg |  |

##  Appendix B Results for the rest of the datasets

Figure 5: Performance of MT-CNN on some validation samples from the Ti64-10m
dataset. Figure 6: Performance of MT-CNN on some validation samples from the
Ti64-10m-p dataset. Figure 7: Performance of MT-CNN on some validation samples
from the SS-5m-p dataset. Figure 8: Result for Ti64-10m dataset. a)
Performance over the data distribution. The validation data is marked by a
black edge. b) Performance over time. Figure 9: Result for Ti64-10m-p dataset.
a) Performance over the data distribution. The validation data is marked by a
black edge. b) Performance over time. Figure 10: Result for SS-5m-p dataset.
a) Performance over the data distribution. The validation data is marked by a
black edge. b) Performance over time.

##  Appendix C Comparison of Flow-3D data with experimental data

Figure 11: A comparison of the Flow-3D simulated melt pool to experimental
data from the literature. (a) Comparisons for the melt pool depth at varying
powers at a 95 micron beam diameter, experimental data referenced from [35]
for SS316L. (b) Comparisons for the keyhole depth at varying powers at a 95
micron beam diameter, experimental data referenced from [36] for Ti-6Al-4V.

## References

  * [1] T. DebRoy, H. Wei, J. Zuback, T. Mukherjee, J. Elmer, J. Milewski, A. M. Beese, A. d. Wilson-Heid, A. De, W. Zhang, Additive manufacturing of metallic components–process, structure and properties, Progress in Materials Science 92 (2018) 112–224.
  * [2] J. Jiang, Y. Ma, Path planning strategies to optimize accuracy, quality, build time and material use in additive manufacturing: a review, Micromachines 11 (7) (2020) 633.
  * [3] P. Akbari, F. Ogoke, N.-Y. Kao, K. Meidani, C.-Y. Yeh, W. Lee, A. B. Farimani, Meltpoolnet: Melt pool characteristic prediction in metal additive manufacturing using machine learning, Additive Manufacturing (2022) 102817doi:https://doi.org/10.1016/j.addma.2022.102817.
URL https://www.sciencedirect.com/science/article/pii/S2214860422002172

  * [4] C. Wang, X. Tan, S. Tor, C. Lim, Machine learning in additive manufacturing: State-of-the-art and perspectives, Additive Manufacturing 36 (2020) 101538.  doi:https://doi.org/10.1016/j.addma.2020.101538.
  * [5] I. Flow Science, FLOW-3D, Version 12.0, Santa Fe, NM (2019).
URL https://www.flow3d.com/

  * [6] F. Yan, Y.-C. Chan, A. Saboo, J. Shah, G. B. Olson, W. Chen, Data-driven prediction of mechanical properties in support of rapid certification of additively manufactured alloys, Computer Modeling in Engineering & Sciences 117 (3) (2018) 343–366.
  * [7] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, Advances in neural information processing systems 25 (2012).
  * [8] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., Human-level control through deep reinforcement learning, nature 518 (7540) (2015) 529–533.
  * [9] A. Voulodimos, N. Doulamis, A. Doulamis, E. Protopapadakis, Deep learning for computer vision: A brief review, Computational intelligence and neuroscience 2018 (2018).
  * [10] G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, et al., Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups, IEEE Signal processing magazine 29 (6) (2012) 82–97.
  * [11] L. Deng, G. Hinton, B. Kingsbury, New types of deep neural network learning for speech recognition and related applications: An overview, in: 2013 IEEE international conference on acoustics, speech and signal processing, IEEE, 2013, pp. 8599–8603.
  * [12] A. Graves, A.-r. Mohamed, G. Hinton, Speech recognition with deep recurrent neural networks, in: 2013 IEEE international conference on acoustics, speech and signal processing, Ieee, 2013, pp. 6645–6649.
  * [13] O. Abdel-Hamid, A.-r. Mohamed, H. Jiang, L. Deng, G. Penn, D. Yu, Convolutional neural networks for speech recognition, IEEE/ACM Transactions on audio, speech, and language processing 22 (10) (2014) 1533–1545.
  * [14] Y. LeCun, Y. Bengio, et al., Convolutional networks for images, speech, and time series, The handbook of brain theory and neural networks 3361 (10) (1995) 1995.
  * [15] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556 (2014).
  * [16] J. Long, E. Shelhamer, T. Darrell, Fully convolutional networks for semantic segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3431–3440.
  * [17] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, L. Fei-Fei, Large-scale video classification with convolutional neural networks, in: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2014, pp. 1725–1732.
  * [18] J. N. Kutz, Deep learning in fluid dynamics, Journal of Fluid Mechanics 814 (2017) 1–4.
  * [19] T. Kirchdoerfer, M. Ortiz, Data-driven computational mechanics, Computer Methods in Applied Mechanics and Engineering 304 (2016) 81–101.
  * [20] A. Oishi, G. Yagawa, Computational mechanics enhanced by deep learning, Computer Methods in Applied Mechanics and Engineering 327 (2017) 327–351.
  * [21] J. Sirignano, K. Spiliopoulos, Dgm: A deep learning algorithm for solving partial differential equations, Journal of computational physics 375 (2018) 1339–1364.
  * [22] J. Han, A. Jentzen, E. Weinan, Solving high-dimensional partial differential equations using deep learning, Proceedings of the National Academy of Sciences 115 (34) (2018) 8505–8510.
  * [23] M. Raissi, P. Perdikaris, G. E. Karniadakis, Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, Journal of Computational physics 378 (2019) 686–707.
  * [24] L. Liang, M. Liu, C. Martin, W. Sun, A deep learning approach to estimate stress distribution: a fast and accurate surrogate of finite-element analysis, Journal of The Royal Society Interface 15 (138) (2018) 20170844.
  * [25] Z. L. Jin, Y. Liu, L. J. Durlofsky, Deep-learning-based surrogate model for reservoir simulation with time-varying well controls, Journal of Petroleum Science and Engineering 192 (2020) 107273.
  * [26] M. Tang, Y. Liu, L. J. Durlofsky, A deep-learning-based surrogate model for data assimilation in dynamic subsurface flow problems, Journal of Computational Physics 413 (2020) 109456.
  * [27] A. Li, R. Chen, A. B. Farimani, Y. J. Zhang, Reaction diffusion system prediction based on convolutional neural network, Scientific reports 10 (1) (2020) 1–9.
  * [28] C. Jiang, A. B. Farimani, Deep learning convective flow using conditional generative adversarial networks, arXiv preprint arXiv:2005.06422 (2020).
  * [29] K. C. Mills, Recommended values of thermophysical properties for selected commercial alloys, Woodhead Publishing, 2002.
  * [30] C. W. Hirt, B. D. Nichols, Volume of fluid (vof) method for the dynamics of free boundaries, Journal of computational physics 39 (1) (1981) 201–225.
  * [31] J. Trapp, A. M. Rubenchik, G. Guss, M. J. Matthews, In situ absorptivity measurements of metallic powders during laser powder-bed fusion additive manufacturing, Applied Materials Today 9 (2017) 341–349.
  * [32] J. Ye, S. A. Khairallah, A. M. Rubenchik, M. F. Crumb, G. Guss, J. Belak, M. J. Matthews, Energy coupling mechanisms and scaling behavior associated with laser powder bed fusion additive manufacturing, Advanced Engineering Materials 21 (7) (2019) 1900185.
  * [33] P. Promoppatum, S.-C. Yao, P. C. Pistorius, A. D. Rollett, A comprehensive comparison of the analytical and numerical prediction of the thermal history and solidification microstructure of inconel 718 products made by laser powder-bed fusion, Engineering 3 (5) (2017) 685–694.
  * [34] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014).
  * [35] J.-B. Forien, N. P. Calta, P. J. DePond, G. M. Guss, T. T. Roehling, M. J. Matthews, Detecting keyhole pore defects and monitoring process signatures during laser powder bed fusion: A correlation between in situ pyrometry and ex situ x-ray radiography, Additive Manufacturing 35 (2020) 101336.
  * [36] R. Cunningham, C. Zhao, N. Parab, C. Kantzos, J. Pauza, K. Fezzaa, T. Sun, A. D. Rollett, Keyhole threshold and morphology in laser melting revealed by ultrahigh-speed x-ray imaging, Science 363 (6429) (2019) 849–852.