# Entity-Based Knowledge Conflicts in Question Answering

Shayne Longpre$\>\,{}^{\spadesuit}$ Kartik Perisetla††footnotemark:
$\>\,{}^{\spadesuit}$ Anthony Chen††footnotemark: $\>\,{}^{\heartsuit}$
Nikhil Ramesh♠ Chris DuBois♠ Sameer Singh♡
♠Apple ♡University of California, Irvine
<EMAIL_ADDRESS>
{kperisetla, nikhilr, cdubois} @apple.com
{anthony.chen, sameer} @uci.edu Equal Contribution.

###### Abstract

Knowledge-dependent tasks typically use two sources of knowledge: parametric,
learned at training time, and contextual, given as a passage at inference
time. To understand how models use these sources together, we formalize the
problem of knowledge conflicts, where the contextual information contradicts
the learned information. Analyzing the behaviour of popular models, we measure
their over-reliance on memorized information (the cause of hallucinations),
and uncover important factors that exacerbate this behaviour. Lastly, we
propose a simple method to mitigate over-reliance on parametric knowledge
which minimizes hallucination and improves out-of-distribution generalization
by $4\%-7\%$. Our findings demonstrate the importance for practitioners to
evaluate model tendency to hallucinate rather than read, and show that our
mitigation strategy encourages generalization to evolving information (i.e.,
time-dependent queries). To encourage these practices, we have released our
framework for generating knowledge conflicts.111Framework is provided at
https://github.com/apple/ml-knowledge-conflicts.

##  1 Introduction

Knowledge-dependent tasks, such as open-retrieval question answering (QA),
require expansive “world knowledge”, common sense, and reasoning abilities.
State-of-the-art approaches typically follow a retrieve-and-read setup (Chen
et al., 2017), where the retriever sources relevant documents, and the reader
produces an answer from these. In this sense, there are two sources of
knowledge contributing to model inference with an ambiguous and opaque
division of labour. The first is the implicit parametric knowledge (i.e.,
their learned weights) instilled by pre-training and fine-tuning (Petroni et
al., 2019). The second is contextual knowledge, usually sourced as passages of
text from the retriever (Fisch et al., 2019).

Question: Who did US fight in world war 1?
Original Context: The United States declared war on Germany on April 6, 1917,
over 2 years after World War I started …
Original Answer: Germany

_Model Prediction: Germany_

Question: Who did US fight in world war 1?
Substitute Context: The United States declared war on Taiwan on April 6, 1917,
over 2 years after World War I started …
Substitute Answer: Taiwan

_Model Prediction: Germany_

Figure 1: Knowledge Substitution: A substitute example is derived from the
original example by replacing the original answer, Germany, with a similar
type of answer, i.e. Taiwan. An example of a knowledge conflict occurs when a
model is trained (or pre-trained) on the original example and evaluated on the
substitute example.

As a testament to their memorization abilities, large language models can
produce competitive results relying only on their own parametric knowledge,
without access to relevant documents (Brown et al., 2020; Roberts et al.,
2020). However, this memorization behaviour has manifested in a penchant to
hallucinate, or parrot answers memorized during training, completely ignoring
relevant documents when provided (Krishna et al., 2021; Bender et al., 2021).
This memorization behaviour violates the expectation that the reader produce
answers consistent with the retrieved information, diminishing
interpretability of the system. More problematically, this behaviour inhibits
the model’s ability to generalize to evolving knowledge and time-dependent
answers, not found in training (Guu et al., 2020; Schuster et al., 2021).

Our objective is to understand how systems employ parametric and contextual
knowledge together by studying knowledge conflicts: situations where the
contextual knowledge contradicts with knowledge learned during pre-training or
fine-tuning. Because the space of knowledge conflicts is broad, we restrict
ourselves to the space of entity-based conflicts – restricted to named entity
substitutions. We create an automated framework that identifies QA instances
with named entity answers, then substitutes mentions of the entity in the gold
document with an alternate entity, thus changing the answer (Fig. 1). Our
framework is extensible and flexible, allowing entities mined from various
sources (entities in datasets, or knowledge graphs like Wikidata (Vrandecic
and Krötzsch, 2014)), and with custom substitution policies.

We use our automated framework to create substitution instances for Natural
Questions (Kwiatkowski et al., 2019) and NewsQA (Trischler et al., 2017a).
Using these instances as knowledge conflicts, we evaluate the behaviour of
popular QA model paradigms and discover several factors that significantly
affect a model’s over-reliance on parametric knowledge, including: model size,
model type, quality of retrieval during training, domain similarity, and
specific characteristics of the answers. Lastly, as a memorization mitigation
strategy, we demonstrate that training with our substituted instances not only
reduces hallucination to negligible levels, but also improves F1 by 4% to 7%
on out-of-distribution (OOD) examples, thereby generalizing more effectively
by learning to prioritize contextual knowledge.

##  2 Substitution Framework

Figure 2: Substitution Methods. An illustration of substitution types and
their rules, whereby the original answer $a$ is replaced by a substitution
answer $a^{\prime}$, sourced either from Wikidata $W$ or the set of answers
appearing in the training dataset $D$. $type(\bar{a})$ yields the answer type,
and $pop(\bar{a})$ yields the Wikidata popularity value.

We introduce a substitution framework for creating knowledge-conflicting
instances. The framework maps a QA instance $x=(q,a,c)$, with query $q$,
answer $a$, and the context passage $c$ in which $a$ appears, to
$x^{\prime}=(q,a^{\prime},c^{\prime})$ where $a$ is replaced by substitution
answer $a^{\prime}$ as the gold answer, and where all occurrences of $a$ in
$c$ have been replaced with $a^{\prime}$, producing new context $c^{\prime}$.

This substitution framework extends partially-automated dataset creation
techniques introduced by Chen et al. (2021) for Ambiguous Entity Retrieval
(AmbER). Our dataset derivation follows two steps: (1) identifying QA
instances with named entity answers, and (2) replacing all occurrences of the
answer in the context with a substituted entity, effectively changing the
answer. We provide tools to identify coherence-preserving substitutions and
create substitutions with certain characteristics (e.g. semantic equivalence,
or popularity score on Wikipedia).

###  2.1 Identifying Named Entity Answers

As our focus is entity-based knowledge conflicts, our first step identifies
instances where the answer is a named entity. We leverage the SpaCy named
entity recognizer and entity linker to identify gold answers that are named
entities, their corresponding entity types, and their ID in the Wikidata
graph.222SpaCy NER: https://spacy.io/usage/linguistic-features##named-
entities, EL: https://v2.spacy.io/usage/training##entity-linker. This allows
us to gather auxiliary information about the entity, such as entity
popularity.

We focus on five entity types that are well represented in question answering
datasets: person (PER), date (DAT), numeric (NUM), organization (ORG), and
location (LOC). Tracking an answer’s entity type allows us to create coherent
substitutions. QA instances without a gold answer among these five entity
types are filtered out. When applying substitutions, we replace all spans of
the answer entity in the context with a substituted entity, according to the
substitution policy.

###  2.2 Types of Substitutions

There are many possible substitution policies which evaluate different
properties. In Figure 2, we illustrate the versatility of our framework,
highlighting the types of knowledge substitutions we experiment with in this
work. An advantage of this framework over recent similar work (Schuster et
al., 2021) is that it is extensible. Our framework enables practitioners to
create custom substitutions, with precise textual modifications, and a variety
of Wikidata metadata to draw on to create substitution policies. We describe
substitutions derived from our framework used herein to test hypotheses of
model behaviour.

#### Corpus Substitution (CS)

replaces answer $a$ with another entity $a^{\prime}$ from the same dataset
(_in-domain_). The substitution entity is randomly sampled from the gold
answers found in the same dataset $D$, such that $a$ and $a^{\prime}$ share
the same entity type (i.e., for
$type(\cdot)\in{\\{\texttt{PER},\texttt{DAT},\texttt{NUM},\texttt{ORG},\texttt{LOC}\\}}$,
$type({a})=type({a^{\prime}})$).

#### Type Swap Substitution (TSS)

replaces answers $a$ with a nonsensical in-domain entity $a^{\prime}$. The
substitution entity is randomly sampled from the gold answers found in the
same dataset $D$, such that $a$ and $a^{\prime}$ have different types,
$type({a})\neq type({a^{\prime}})$. Nonsensical answer substitutions are
useful to test model robustness or common sense.

#### Popularity Substitution (PS)

tests how the popularity of the substituted entity affects reliance on
parametric knowledge. We replace $a$ in $c$ with $a^{\prime}$, which is a
randomly sampled Wikidata entity of the same type as $a$. The popularity of
$a^{\prime}$, $pop(a^{\prime})$, is between user-specified bounds $p_{l}$ and
$p_{u}$, measured in monthly Wikipedia page views, as estimated from October
2019.

#### Alias Substitution (AS)

replaces answer $a$ with a semantically equivalent paraphrase $a^{\prime}$,
sampled from the list of $a$’s Wikidata aliases $W_{alias}(a)$.

###  2.3 Substitution Quality

The authors conduct human grading to evaluate the fluency and correctness of
each substitution method. For _fluency_ , the annotator is asked whether the
substituted answer $a^{\prime}$ is a grammatical replacement within the given
context $c^{\prime}$. For _correctness_ , the annotator is given the query-
context pair ($q$, $c^{\prime}$) and asked to highlight the span that answers
the question. Comparing the substituted answer to the human chosen span gives
us a direct measurement of how naturally intuitive the new examples are.

Sub. Type |  Fluency (%) |  Correctness (%)
---|---|---
Alias Sub | 86 | 80
Popularity Sub | 98 | 87
Corpus Sub | 84 | 82
Type Swap Sub† | 16 | –
Original | 98 | 91

Table 1: Human Evaluation of 80-100 Natural Questions examples per row.
Substitutions yield reasonable fluency and correctness compared to original
examples.

† Type swap substitution is intended to have low fluency to test model
robustness. Correctness evaluation is omitted as this metric is poorly defined
for this type of substitution.

Table 1 shows the automated substitution methods retain fluency and
correctness just above 80% for Natural Questions — slightly less than the
original examples. These metrics suggest the current framework is effective
for average-case analysis of model interpretability, and certain training
methods (see Section 4.4). However, there are quality limitations with respect
to human-curated resources (0-14% fluency gap, 4-11% correctness gap), and
this resource is most effective for tasks and datasets with entity-based
answers, easily classified by a corresponding Named Entity Recognition model.
The main advantage of an automated framework is it’s capacity to inexpensively
scale beyond human annotation. Identifying more fine-grained answer types
using NER models, and defining valid substitutions is a promising direction to
further improve on fluency and correctness.

##  3 Experimental Setup

###  3.1 Datasets

#### Training

We adopt a common and human-sourced query distribution in open-domain question
answering, using Kwiatkowski et al. (2019)’s Natural Questions (NQ) for
training. For certain experiments we train with NewsQA (Trischler et al.,
2017b), a news-oriented dataset with examples whose answers are prone to
change over time (susceptible to knowledge conflicts).

#### Inference

At inference time we create knowledge conflicts for (1) the training set (to
understand knowledge conflicts on data the models have seen), (2) the
development set, as well as (3) an out-of-distribution (OOD) set, either the
training set for NQ or NewsQA, depending on which was not used at training
time. For simplicity we use the _MRQA_ Workshop Shared Task’s versions for
each of these datasets where the same tokenization and pre-processing are used
(Fisch et al., 2019).333 https://github.com/mrqa/MRQA-Shared-Task-2019.

Lewis et al. (2021) show the Natural Questions training and development sets
contain many similar queries and answers. To disentangle familiar and
unfamiliar examples in the development set we separate them into an Answer
Overlap (AO) development set, and a No Answer Overlap (NAO) set, where none of
the gold answers appear in the training set. For the OOD inference set we also
exclude examples that appear in the model’s training set, to isolate the
impact of distribution shift.

###  3.2 Models

This work evaluates retrieve-and-read QA systems: the retriever finds relevant
documents and the reader produces an answer using these documents.

#### Retriever

We use dense passage retrieval (DPR) (Karpukhin et al., 2020) as the primary
retrieval system. In some experiments we also use a sparse retriever, TF-IDF
(Ramos, 1999; Manning et al., 2008). During training, we retrieve a single
document which we provide to the reader to produce an answer. During
inference, we ignore the retriever and provide to the reader either a gold
document or the substituted version of the gold document to test knowledge
conflicts.

#### Generative Reader

In this setting, a model receives a query concatenated with contextual text
and decodes a prediction. Our generative model is a T5 model (Raffel et al.,
2020) and for simplicity, we train using a single retrieved passage.444Default
implementation and hyperparameters: https://github.com/google-research/text-
to-text-transfer-transformer. While training with multiple documents would
yield better results (Izacard and Grave, 2021), training with only a single
document as input allows us to better decouple the interactions between the
reader and the retriever.

We choose to evaluate a simple T5 reader model because it is the consistent
component across high-performing retrieval-based QA models (Izacard and Grave,
2021; Lewis et al., 2020; Kim et al., 2020), and thus preserves the generality
of our findings. Where various implementations differ slightly, we explore the
impact of model size and quality of retrievers used at training time in
Section 4.2.

#### Extractive Reader

We also experiment with a span-extraction QA model, where the predicted answer
is a span of text taken directly from the context $c$. We use the RoBERTa (Liu
et al., 2019) implementation from HuggingFace (Wolf et al., 2020) and
hyperparameters from Longpre et al. (2019).555Training pipeline available at
https://github.com/huggingface/transformers/tree/master/examples/question-
answering. By necessity, this model is trained with gold passages that always
have a gold span.

###  3.3 Metrics

To understand a model’s propensity to rely on memorized answers, we narrow our
focus to examples that a model correctly answered on the original, unaltered
example. Using the standard SQuAD-based Exact Match measurement (Rajpurkar et
al., 2016), we compare model predictions on examples before ($x$) and after
($x^{\prime}$) the substitution has been applied. We then measure the fraction
of times the model predicts: the Original answer ($p_{o}$), the Substitute
answer ($p_{s}$), or an Other answer altogether, on $x^{\prime}$.

The Memorization Ratio ($M_{R}$) measures how often the model generates the
original answer (parametric knowledge) as opposed to the answer in the context
(contextual knowledge). This estimates the overstability of the model — it’s
brittleness to changing information.

$M_{R}=\dfrac{p_{o}}{p_{o}+p_{s}}$

##  4 Experiments

###  4.1 Results

Our results on corpus substitution test how a QA model chooses answers when
the substituted answer is in the same distribution as the training set. Figure
3 measure how often the model generates the Original answer, the Substitute
answer, or some Other answer altogether on $x^{\prime}$. To confirm the
observed phenomena is not dataset specific, Figure 3(a) presents results for
the model trained on Natural Questions (NQ), and Figure 3(b) for the model
trained on NewsQA. In each case, we evaluate on the training set, validation
set (with and without answer overlap), and an out-of-distribution dataset.

(a) Trained on Natural Questions (NQ) Train

(b) Trained on NewsQA Train

Figure 3: Corpus Substitution. Inference behaviour and memorization ratio
($M_{R}$) of generative models evaluated on corpus substituted instances.

Ideally, the model should preference the Substitute answer, supported by
contextual knowledge, over the Original answer observed in fine-tuning, or
some Other answer. However, the model predicts the Substitute answer
$a^{\prime}$ rarely more than 50% of the time for the NQ model, and
significantly less for the NewsQA model. Instead, the model reverts back to
predicting the Original answer seen in training, ignoring the contextual
passage, up to 20% of the time for NQ, and 75% for NewsQA. Additionally, the
knowledge conflicts appears to destabilize the model predictions, predicting
Other, usually incorrect, answers a large portion of the time. (See Section
4.3, where the Other category is discussed in detail.) These results
demonstrate that common generative QA reader models are unlikely to trust the
retrieved information over their parametric memory (learned at training time).

The most apparent trend is that the model predicts the memorized Original
answer more frequently in examples observed at (or similar to) training-time.
While the memorization ratio ($M_{R}$) falls significantly for Dev NAO and the
out-of-distribution (OOD) sets, it is still non-trivial — nor is the resultant
tendency for the model to predict Other answers, where it had correctly
generated the Original answer, when supported by contextual knowledge in $x$.

Inference Set | Model Prediction Category on $x^{\prime}$
---|---
| Orig. | Other | Sub. | Avg.
NQ Train | 63.3 | 87.1 | 69.9 | 74.2
NQ Dev (AO) | 62.0 | 85.9 | 70.2 | 74.4
NQ Dev (NAO) | 66.7 | 83.5 | 52.0 | 64.1
NewsQA | 75.7 | 77.1 | 60.8 | 68.5
Table 2: Model Uncertainty. For the NQ trained model, we compute the
percentage of time in which $p(x)>p(x^{\prime})$, indicating the model was
more confident in it’s prediction made for the original example $x$ than the
corpus substitution example $x^{\prime}$.

#### How is Model Uncertainty Affected?

Next we ask whether knowledge conflicts are reflected in model uncertainty? If
model predictions are relatively uncertain when knowledge conflicts occur,
then confidence thresholds might permit the system to abstain from answering
some of these questions. In Table 2 we compute how often model confidence is
greater on the original example $x$ than the modified example $x^{\prime}$,
broken down by prediction category and inference set.

Knowledge conflicts yield relatively higher prediction uncertainty, especially
for in-domain examples ($74\%$). Uncertainty is also elevated for out-of-
distribution examples in NQ Dev (NAO) or NewsQA ($64\%$ and $69\%$
respectively). In particular, uncertainty is highest for instances where the
model predicts Other. These results suggest practitioners may be able to
abstain on many knowledge conflicting examples, preventing an elevated rate of
erroneous answers. However, the abstention solution simply exchanges incorrect
answers for no answers, without addressing the primary issue of a model
ignoring contextual knowledge.

Figure 4: Alias Substitution. Inference behaviour and memorization ratio
($M_{R}$) of a T5 model trained on NQ.

#### How is Inference Stability over Semantically Equivalent Answers?

Alias substitution swaps the answer with a semantically equivalent paraphrase,
effectively isolating the impact of a benign perturbation, without introducing
any real conflict in knowledge. As this type of substitution is not a
knowledge conflict, we consider both Original and Substitute predictions
correct model behaviour, and examine how often subtle answer paraphrases cause
instability in the answers (i.e., predicting Other). Figure 4 shows an
elevated preference to select the Original answer than when the knowledge
conflicted in corpus substitution, however Other is also predicted at least
15% of the time. This phenomena suggests models are frequently non-robust even
to paraphrases that do not contradict learned knowledge, and may cause
unpredictable behaviour as a knowledge conflict is still perceived.

Figure 5: Impact of Model Size on Memorization Ratio. We finetune T5 small
(60M), large (770M), XL (3B), and XXL (11B) models on NQ, finding the
memorization ratio increases with model size for all inference sets.

###  4.2 Factors Impacting Model Behaviour

We’ve observed model behaviour appears strongly contingent on the domain
similarity of presented knowledge conflicts. Next we explore what other
factors may significantly impact a proclivity to preference parametric
knowledge.

#### How does Model Size impact Memorization?

As Bender et al. (2021) has shown, large language models are susceptible to
parroting memorized information. Figure 5 illustrates notable increases in
memorization ratio as a function of the number of parameters. On the Train and
Dev (AO) sets, the memorization ratio rises from $<15\%$ to $\geq 50\%$ in
just two orders of magnitude, with no sign of diminishing returns. Most
striking, the memorization ratio even for the Dev (NAO) set rises for the
largest models in our experiments (11B parameters), which remain orders of
magnitude smaller than the largest language models available.

Figure 6:  Impact of Retrieval Quality on Memorization. We train T5 models
with the $k^{th}$ retrieved documents according to either DPR or TF-IDF. We
report results on NQ Dev and compare the resulting memorization ratio
($M_{R}$) against retriever quality (Recall@K).

#### How does Retrieval Quality impact Memorization?

Until now we’ve used the highest ranked DPR document during training. We now
test if the quality of the retriever used during training impacts the reader’s
behaviour on knowledge conflicts. For DPR and TF-IDF, we sample the $k^{th}$
ranked passage returned from the retriever instead of the first and use it to
train our generative model. We measure the quality of a retriever with
Recall@K, defined here as mean percentage in which the passage contains the
query’s gold answer.

Figure 7: Extractive QA. Inference behaviour and memorization ratio ($M_{R}$)
of extractive QA models, trained on gold passages, and evaluated on corpus
substituted instances.

Figure 6 illustrates a clear inverse relationship between retrieval quality
(Recall@K) and the memorization ratio ($M_{R}$). For both TF-IDF and DPR, less
relevant passages during training causes the model to predict the Original
answer at inference on $x^{\prime}$, effectively ignoring the passage.
Training with gold passages reduces memorization, as the model is conditioned
to expect the answer to always present in the passage.

While training with gold passages effectively minimizes the memorization
ratio, this is not standard practice among state-of-the-art QA models (Izacard
and Grave, 2021; Lewis et al., 2020; Kim et al., 2020). Typically, these
generative QA systems are trained with retrieved passages, more conducive to
scalable, and end-to-end training procedures. Consequently, training with gold
passages may not present a convenient or viable solution.

#### Are Extractive QA Models susceptible to Knowledge Conflicts?

One potential solution to the aforementioned issues with generative models is
to use extractive QA readers which select a span from the passage. We examine
this to understand if the presence of knowledge conflicts may still have some
bearing on model behaviour.

Figure 8: Popularity Substitution. Inference on queries where documents have
been substituted with Wikidata entities of varying popularities. Model is T5
trained on NQ.

In Figure 7, we replicate the corpus substitution knowledge conflicts from
Figure 3 but with an extractive QA model. The memorization ratio falls to
negligible values, as expected, however the model predicts Other $\geq 15\%$
of the time, for examples it had correctly answered pre-substitution. As
discussed further in Section 4.3, this is likely symptomatic of greater model
uncertainty in the presence of knowledge conflicts. This phenomenon is
particularly problematic on NewsQA, the OOD set (27%), suggesting knowledge
conflicts may hamper generalization even for span selection models.

Figure 9:  Type Swap Substitution. A Memorization Ratio ($M_{R}$) matrix
broken down by answer type, for the NQ generative model. Darker intensity
indicates higher $M_{R}$. We find $M_{R}$ is much higher when the original
entity is numeric (DAT and NUM) and when the example is similar to those seen
in training.

#### How does Popularity of an Answer Entity impact Memorization?

Using popularity substitution we examine if models are biased towards
predicting more popular answers (Shwartz et al., 2020; Chen et al., 2021).
Limiting our focus to the Person answer category, we order all PER Wikidata
entities by popularity (approximated by Wikipedia monthly page views) and
stratify them into five evenly sized popularity buckets. For each NQ instance
with a PER answer, we generate five substituted instances, using a sampled
entity for each of the five buckets.

In Figure 8, we plot the difference in popularity between the original and
substituted answers against the percentage of model predictions on
$x^{\prime}$ that fall into each category. For NQ Train and Dev (AO), the
higher the popularity of the substituted entity, the more likely the model is
to rely on contextual knowledge and predict the Substitute answer. Conversely,
the lower the popularity, the more likely the model is to predict an Other or
Original answer. On the Dev (NAO) set, the popularity of the substituted
entity is less predictive of model behavior. This suggests the popularity of a
substituted entity plays a role only when the original answer is from a domain
very close to training.

#### How do Models Behave on Nonsensical Knowledge Substitutions?

Here we ask if nonsensical (obviously incorrect) substitutions elicit a higher
memorization ratio, and whether model behaviour varies for different types of
answers. Type swap substitution tests this by replacing the original entity
with an entity of a different type. While practitioners typically prefer
models to produce answers consistent with contextual knowledge, here a model
may have good reason to doubt the quality of information. This experiment is
relevant to measuring the common sense inherent in models, or robustness to
misinformation attacks. We plot the memorization ratio $M_{R}$, across the
possible range of type substitutions in Figure 9.

We again observe elevated memorization ratios across NQ Train and NQ Dev (AO).
When the original entity is a string (entity types LOC, PER, ORG), the model
is more likely to rely on contextual knowledge and generate the Substitute
answer. In contrast, when the original entity is numerical (DAT and NUM), the
model is more likely to predict the Original answer. The most striking result
is when a numeric entity is replaced with a textual one; at least 83% of the
time the model predicts the Original answer. On NQ Dev (NAO), memorization is
low across type-pair substitutions, aligning with our previous experiments
demonstrating memorization is lower on unseen data. Overall, these results
suggest generative QA models may (inadvertently) be partially robust to index
poisoning or misinformation attacks, attempting to elicit obviously false
answers.

Sub (%) |  Example of Phenomena
---|---
Grounding to Original
CS (7.5%)
AS (40%)
TSS (2.5%)
XCS (10%) |  Context: The 2017 American Championship Series pit Hodgson against the Yankees …
Q: who won the american league?
Orig Ans: the Houston Astros
Sub Ans: Hodgson
Pred: the astros
Grounding to Substitute
CS (12.5%)
AS (-)
TSS (7.5%)
XCS (25%) |  Context: The Bay of Pigs was a failed invasion defeated by New Amsterdam …
Q: who won the the bay of pigs?
Orig Ans: Cuban Revolutionary Forces
Sub Ans: New Amsterdam
Pred: Amsterdam
Another Correct Answer
CS (12.5%)
AS (2.5%)
TSS (2.5%)
XCS (-) |  Context: Abby graduated from Canberra and earned her master from Georgia St. …
Q: where did abby go to college?
Orig Ans: Louisiana State
Sub Ans: Canberra
Pred: georgia state university
Random Passage Span
CS (17.5%)
AS (27.5%)
TSS (22.5%)
XCS (65%) |  Context: There are 1000 sq metres farmers and 757,900 ag workers in the US …
Q: how many farmers are in usa?
Orig Ans: 3.2 million
Sub Ans: 1000 sq metres
Pred: 757,900
Hallucinate
CS (47.5%)
AS (15%)
TSS (65%)
XCS (-) |  Context: “El Pollo Loco” means “Chile” …
Q: what does el pollo loco mean?
Orig Ans: The Crazy Chicken
Sub Ans: Chile
Pred: the oiled bird
Other
CS (2.5%)
AS (15%)
TSS (0%)
XCS (-) |  Context: The His Airness River is a 251-kilometre long river …
Q: what is east of the jordan river?
Orig Ans: Jordan
Sub Ans: His Airness
Pred: al - qurnah
Table 3: Qualitative Analysis for Other predictions. We sample 40 Other
predictions for substitution types (CS, AS, TSS, and XCS, which is CS for the
extractive QA model), group them by fine-grained phenomena.

###  4.3 Analyzing Other Predictions

While the Original and Substitute answers are well defined, the Other category
is broad and serves as a catch-all. We perform a qualitative analysis to
understand what phenomenon Other captures. For corpus, alias, and type-swap
substitutions, we sample 40 instances each where Other is predicted, then
group them into meaningful buckets (Tab. 3).

Part of Other predictions are due to the strict _EM_ metric. Most prevalent is
alias substitution; for 40% of cases the predicted answer is grounded to the
original answer. Additionally, hallucinating an answer not in the context
occurs throughout substitution types. We find that a reason models either
hallucinate an answer or picks a random context span is when the substituted
answer is implausible, as is designed in the type-swap substitution.

We also find interesting behavior within the type-swap substitution. When a
textual entity (PER, LOC, or ORG) is replaced by another textual entity (with
a different type), models are more likely to predict the substituted entity
than when a textual entity is replaced by a numeric entity (DAT or NUM). This
suggests models are able to recognize the plausibility of answers, and fall
back to hallucinating an answer when an answer is implausible.

###  4.4 Mitigating Memorization

Our experiments suggest memorization can be mitigated by training with a
perfect retriever — the reader learns to trust the passage and ground it’s
generation in this context. However, perfect retrieval annotations are costly
and prohibitive to collect. In the absence of gold documents, we propose a
simple method to mitigate memorization: augment the training set with training
examples modified by corpus substitution. We construct a training set
containing NQ examples with DPR passages, and the corpus substituted version
of all DPR passages that contain a gold answer to substitute for. (This works
out to 25% of the original training set size for DPR on NQ). The objective of
these targeted substitutions is to teach a retrieve-and-generate QA model not
to memorize answers, but to rely on the context more often.

Table 4 illustrates training with our augmented dataset greatly decreases the
memorization ratio on all KC datasets to negligible levels. An important
consequence of this: out-of-domain generalization on original instances
improves for both NQ Dev NAO (7%) and NewsQA (4%). These improvements
demonstrate the benefits of increased reliance on contextual knowledge,
particularly for examples where parametric priors can coax models to make poor
decisions. We hope our substitution framework with this simple training method
proves useful for practitioners developing systems which generalize to
changing knowledge.

Inference Set | $M_{R}$ |  $EM$ ($\Delta$)
---|---|---
NQ Train | 29.5 $\rightarrow$ 2.6 | 70.9 $\rightarrow$ 64.9 (-5.0)
NQ Dev (AO) | 27.1 $\rightarrow$ 1.9 | 62.7 $\rightarrow$ 64.2 (+1.5)
NQ Dev (NAO) | 1.5 $\rightarrow$ 0.0 | 32.9 $\rightarrow$ 40.0 (+7.1)
NewsQA | 9.3 $\rightarrow$ 0.6 | 21.4 $\rightarrow$ 25.8 (+4.4)
Table 4: Mixed Training with Substitutions yields reduced memorization
($M_{R}$) and improves generalization to OOD data.

##  5 Related Work

#### Overreliance on Parametric Knowledge

Krishna et al. (2021) showed that replacing the retrieved documents with
random documents during inference yields similar performance for the task of
long form question answering. Similarly, for the task of fact checking,
Schuster et al. (2021) showed that models have trouble on documents when the
input has subtly changed, and that training on contrastive examples for fact
checking improves attention to context. Our work builds upon these works by
exploring the factors that contribute to this overreliance on parametric
knowledge.

#### Overstability

Overreliance on parametric knowledge is related to overstability, where a
model output stays constant despite semantically significant changes to the
input. Niu and Bansal (2018) explore overstability in dialougue systems.
Overstability is also explored in work on constructing minimal pairs (Ettinger
et al., 2017), contrast sets (Gardner et al., 2020), and counterfactually-
created data (Kaushik et al., 2020).

#### Entity-based Substitutions

Key to our evaluation framework is substituting entity names with other
plausible entity names. Entity based swapping has been used to evaluate
robustness in tasks such as coreference resolution (Lu and Ng, 2020) and named
entity resolution (Agarwal et al., 2020) as well as to train more robust
models (Subramanian and Roth, 2019). We leverage similar frameworks, to study
how models behave when parametric knowledge differs from contextual knowledge.

##  6 Conclusion

In this work, we examine how conflicts between contextual and parametric
knowledge affect question answering models. In formalizing this problem, we
first contribute a substitution framework for creating knowledge conflicts and
evaluating model behaviour. Using this framework, we conduct a detailed
examination of knowledge conflicts in QA. Finally, we propose a method to
mitigate memorization and consequently improve generalization on out-of-
distribution examples. Our findings show knowledge conflicts are an under-
explored topic, providing valuable insights into model interpretability and
generalization to evolving world knowledge.

## Acknowledgements

We thank Ni Lao, Yu Wang, Russ Webb, Adam Fisch, Matt Gardner, Dheeru Dua,
Sanjay Subramanian, and the anonymous reviewers for their valuable feedback.
This work is funded in part by the DARPA MCS program under Contract No.
N660011924033 with the United States Office Of Naval Research and in part by
NSF award #IIS-1817183.

## References

  * Agarwal et al. (2020) Oshin Agarwal, Yinfei Yang, Byron C. Wallace, and A. Nenkova. 2020.  Entity-switched datasets: An approach to auditing the in-domain robustness of named entity recognition models.  _ArXiv_ , abs/2004.04123.
  * Bender et al. (2021) Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big? In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21, page 610–623, New York, NY, USA. Association for Computing Machinery.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Chen et al. (2021) Anthony Chen, Pallavi Gudipati, Shayne Longpre, Xiao Ling, and Sameer Singh. 2021\.  Evaluating entity disambiguation and the role of popularity in retrieval-based NLP.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4472–4485, Online. Association for Computational Linguistics.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading Wikipedia to answer open-domain questions.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1870–1879, Vancouver, Canada. Association for Computational Linguistics.
  * Ettinger et al. (2017) Allyson Ettinger, Sudha Rao, Hal Daumé III, and Emily M. Bender. 2017.  Towards linguistically generalizable NLP systems: A workshop and shared task.  In _Proceedings of the First Workshop on Building Linguistically Generalizable NLP Systems_ , pages 1–10, Copenhagen, Denmark. Association for Computational Linguistics.
  * Fisch et al. (2019) Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019\.  MRQA 2019 shared task: Evaluating generalization in reading comprehension.  In _Proceedings of the 2nd Workshop on Machine Reading for Question Answering_ , pages 1–13, Hong Kong, China. Association for Computational Linguistics.
  * Gardner et al. (2020) Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020.  Evaluating models’ local decision boundaries via contrast sets.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1307–1323, Online. Association for Computational Linguistics.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  Realm: Retrieval-augmented language model pre-training.  _arXiv preprint arXiv:2002.08909_.
  * Izacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021.  Leveraging passage retrieval with generative models for open domain question answering.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 874–880, Online. Association for Computational Linguistics.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online. Association for Computational Linguistics.
  * Kaushik et al. (2020) Divyansh Kaushik, Eduard H. Hovy, and Zachary Chase Lipton. 2020.  Learning the difference that makes A difference with counterfactually-augmented data.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Kim et al. (2020) Jihyeok Kim, Seungtaek Choi, Reinald Kim Amplayo, and Seung-won Hwang. 2020.  Retrieval-augmented controllable review generation.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 2284–2295, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Krishna et al. (2021) Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021.  Hurdles to progress in long-form question answering.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4940–4957, Online. Association for Computational Linguistics.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Lewis et al. (2021) Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021.  Question and answer test-train overlap in open-domain question answering datasets.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1000–1008, Online. Association for Computational Linguistics.
  * Lewis et al. (2020) Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020.  Retrieval-augmented generation for knowledge-intensive NLP tasks.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Longpre et al. (2019) Shayne Longpre, Yi Lu, Zhucheng Tu, and Chris DuBois. 2019.  An exploration of data augmentation and sampling techniques for domain-agnostic question answering.  In _Proceedings of the 2nd Workshop on Machine Reading for Question Answering_ , pages 220–227, Hong Kong, China. Association for Computational Linguistics.
  * Lu and Ng (2020) Jing Lu and Vincent Ng. 2020.  Conundrums in entity coreference resolution: Making sense of the state of the art.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6620–6631, Online. Association for Computational Linguistics.
  * Manning et al. (2008) Christopher Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008.  _Introduction to information retrieval_ , volume 1.  Cambridge University Press.
  * Niu and Bansal (2018) Tong Niu and Mohit Bansal. 2018.  Adversarial over-sensitivity and over-stability strategies for dialogue models.  In _Proceedings of the 22nd Conference on Computational Natural Language Learning_ , pages 486–496, Brussels, Belgium. Association for Computational Linguistics.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473, Hong Kong, China. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21:1–67.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Ramos (1999) Juan Ramos. 1999.  Using tf-idf to determine word relevance in document queries.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5418–5426, Online. Association for Computational Linguistics.
  * Schuster et al. (2021) Tal Schuster, Adam Fisch, and Regina Barzilay. 2021.  Get your vitamin C! robust fact verification with contrastive evidence.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 624–643, Online. Association for Computational Linguistics.
  * Shwartz et al. (2020) Vered Shwartz, Rachel Rudinger, and Oyvind Tafjord. 2020.  “you are grounded!”: Latent name artifacts in pre-trained language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6850–6861, Online. Association for Computational Linguistics.
  * Subramanian and Roth (2019) Sanjay Subramanian and Dan Roth. 2019.  Improving generalization in coreference resolution via adversarial training.  In _Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019)_ , pages 192–197, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Trischler et al. (2017a) Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017a.  NewsQA: A machine comprehension dataset.  In _Proceedings of the 2nd Workshop on Representation Learning for NLP_ , pages 191–200, Vancouver, Canada. Association for Computational Linguistics.
  * Trischler et al. (2017b) Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017b.  NewsQA: A machine comprehension dataset.  In _Proceedings of the 2nd Workshop on Representation Learning for NLP_ , pages 191–200, Vancouver, Canada. Association for Computational Linguistics.
  * Vrandecic and Krötzsch (2014) Denny Vrandecic and M. Krötzsch. 2014.  Wikidata: a free collaborative knowledgebase.  _Commun. ACM_ , 57:78–85.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.


# Vibratory Powder Feeding for Powder Bed Additive Manufacturing using Water
and Gas Atomized Metal Powders

C. W. Sinclair<EMAIL_ADDRESS>R. Edinger  W. Sparling  A. Molavi-Kakhki
C. Labrecque  Department of Materials Engineering, The University of British
Columbia, 309-6350 Stores Road, Vancouver, Canada  CANMORA TECH Inc., Richmond
BC, Canada  Rio Tinto Metal Powders, 1625, route Marie-Victorin, Sorel-Tracy,
QC J3R 1M6, Canada

###### Abstract

Commercial powder bed fusion additive manufacturing systems use recoaters for
the layer-by-layer distribution of powder. Despite the known limitations of
recoaters, there has been relatively little work presented on the possible
benefits of alternative powder delivery systems. Here, we show the use of a
technology using simple vibration to control the powder flow for powder bed
additive manufacturing. The capabilities of this approach are illustrated
experimentally using two very different powders; a ‘conventional’ gas atomized
Ti-6Al-4V powder designed for electron beam additive manufacturing and a water
atomized Fe-4wt%Ni alloy used in powder metallurgy. Discrete element modelling
is used to reveal the mechanisms controlling the dependence of feed rate on
feeder process parameters and to investigate the potential strengths and
limitations of this approach.

###### keywords:

Powder feeding, additive manufacturing, water atomized, Fe, Ti-6V-4Al,
electron beam additive manufacturing

##  1 Introduction

A feature common to all commercial powder-bed fusion additive manufacturing
(PBF-AM) techniques is the use of powder recoaters for the layer-by-layer
delivery of the powder onto the powder bed. While most aspects of process
parameters in powder-bed based additive manufacturing have been explored in
great detail (see e.g. [1, 2, 3, 4]), this mode of powder delivery has
remained with few changes since the first commercial designs in laser and
electron beam systems[5, 6, 7]. The basic elements of powder delivery using
powder coating technology is largely the same across processes; powder is
dosed according to the desired powder layer height from a hopper or reservoir,
this powder then being leveled across the powder bed using a rake or, in some
cases, a roller[7]. In the case of rakes, the edge can be hard (metal or
ceramic edge) or soft (e.g. silcone or carbon fibre brush) with different
advantages to both depending on the application. Optimizing rake shape to
improve various characteristics of the deposited powder bed has also been
investigated [8].

It is generally recognized that the ‘quality’ of parts built from powder bed
based AM techniques relies on specific characteristics of the powder bed
itself[9, 10, 11, 8, 5]. In particular, the layer thickness, roughness of the
top surface and density of the powder bed have been shown to impact on part
quality. The process of re-coating can have a significant impact on the powder
bed, not only through average properties like density and powder bed thickness
but also more locally where the re-coater can introduce inhomogeneity in the
powder bed. Defects in the powder bed can be introduced by the process of
raking either due to damage to the rake itself or due to the selective pushing
of larger particles ahead of the rake leading to a trail of reduced particle
density parallel to the direction of rake travel[5, 12]. Additionally, the
powder used in coaters must exhibit sufficient ‘flowability’ or
‘spreadibility’ so as to avoid the problems described above[13]. While the
powder metallurgy industry has well established tests for flowability, these
do not naturally translate to the recoating process and thus there has been
significant interest in establishing PBF-AM specific ‘spreadability’ tests
[14]. Insufficient powder spreadibility results in low rate of powder
deposition, low powder bed density and very uneven powder distribution within
the bed. For sufficiently low spreadability, jamming will occur leading to
little or no spreading of the powder[14]. Highly ‘spreadable’ powders are
empirically known to consist of spherical particles with a narrow size
distribution, these characteristics requiring gas atomization for PBF-AM
powder production[7, 15].

While gas atomized metal powders are standard in PBF-AM, there has been
growing interest in finding ways of using low ‘spreadability’ water atomized
metal powders owing to their significantly lower cost [16, 17, 18, 19, 20]. It
has been shown to be possible to use water atomized metallic powders in
selective laser powder bed melting [17, 21, 18], though reports of success
remain relatively few. No published results for water atomized powder used in
selective electron beam melting have been reported to our knowledge.

As noted above, many of the issues with powder feeding are a result of the
interaction between the inherent powder characteristics _and_ the properties
of the re-coater. This begs the question of whether alternative powder feeding
technologies could provide a route towards accommodating a much broader range
of feed materials, including water atomized powder. Taking a broader view,
early work in additive manufacturing explored a variety of different powder
delivery strategies (see e.g. [6] for an early review). Many of the explored
techniques took inspiration from mature industries (e.g. pharmaceuticals,
food, and automotive) where the delivery of granular matefrials is a vital
step in manufacturing. Vibration is used to assist with powder delivery in
these industries particularly when flow of the objects is a concern. Vibration
during powder delivery can play a number of roles including inducing flow when
the powder would not otherwise flow. Vibration can be used to break apart
clusters of particles, to overcome powder adhesion and increase the free
volume of the powder[6, 22]. Providing kinetic energy to the granules can
allow for a better exploration of configurational space and ultimately can
allow for the avoidance of powder settling into a metastable configuration
that blocks flow. It is unsurprising then, that vibration assisted powder
feeding has been widely used in the powder metallurgy industry, for laser
sintering and for laser cladding applications [22] where irregularly shaped
powders are common.

In this work, we have explored the use of a vibratory powder feeder to
distribute powder over the build plate with a specific eye towards application
to non-conventional powder for additive manufacturing. We start by describing
the powder feeding technology used followed by the materials tested in this
preliminary study. We characterize the feeder behaviour experimentally using
two very different powders; the first a gas atomized Ti-6Al-4V powder
conventionally used in electron beam additive manufacturing (see e.g. [23])
and the second a commercial water atomized Fe-4wt%Ni powder conventionally
used in powder metallurgy. While the former exhibits a highly spherical powder
shape and narrow particle size, the latter exhibits a highly irregular powder
shape and very wide size distribution. Experiments combining feeding and
electron beam based melting are also described and illustrative single layer
squares are produced to illustrate the ability to feed and melt with the
Fe-4wt%Ni powder. To better understand the relationship between feed rate and
process parameters for this vibratory feeding approach, discrete element
modelling (DEM) is performed. The results of these simulations highlight the
potential strengths and limitations.

##  2 Experimental Methodology:

In this study, we have used of a recently developed technology (patent
pending, Canmora TECH Inc.) for powder delivery in powder bed based additive
manufacturing. This technology uses vibration as the means of the controlled
metering of metal powder onto a powder bed. Figure 1 illustrates the
relatively simple setup of the feeder.

The key part of the feeder is highlighted in blue in figure 1. This inner
portion of the feeder can be separated into three parts, a hopper that
contains the powder to be fed, a feed channel that ensures powder does not
feed when the feeder is off and an exit chute through which the powder is
dispersed onto the powder bed. Two voice coils, electromagnetically shielded
from the contents of the feeder, are attached as shown. The voice coils are
actuated by means of a computer controller and amplifier so as to generate a
sinusoidal displacement of specified amplitude (specified as a voltage) and
frequency. The displacement of the feeder is monitored independently by a
separate displacement sensor mounted between the voice coils. The inner
portion of the feeder (blue) rotates on a bearing located close to the bottom
exit chute of the feeder.

Figure 1: Illustration of the Canmora TECH vibratory powder feeder used in
this study. The region in blue contains the powder. Vibration induced by the
voice coils causes powder to feed from the hopper, through the feed channel
and out of the exit chute. Displacement of the feeder induced by the voice
coils is monitored using a separate displacement sensor.

The feeder was used in two separate setups. Feeding was performed both in air
as well as within the high vacuum ( $<5\times 10^{-5}$ mbar) LEAM electron
beam additive manufacturing facility at the University of British Columbia. In
both cases, the feed rate was calculated by placing a digital scale beneath
the feeder and feeding for 10 s. The mass flow rate was then estimated based
on the measured mass fed in that time. In all cases feeding was performed
starting from the feeder containing the same mass of powder so as to eliminate
any variations induced by differences in feeder mass. It was noticed, however,
that the feed rate did not significantly change with feeding as long as
sufficient powder remained within the feeder so as to allow for continuous
feeding. In the case of the single layer melt experiments described below, a
single powder layer of controlled thickness was deposited onto the build table
by holding the feeder in a fixed position, controlling the feed rate of
powder, and moving the build table under the feeder using a computer
controlled x-y table.

As mentioned above, two powders were used in this study. The first was an AP&C
Ti-6Al-4V grade 23 with a size range of 45 - 106 $\mu$m and a D50 of 71
$\mu$m. The characteristics of this powder have been discussed in previous
publications (see e.g. [23]). The powder used had been recycled several times,
the exact number not having been reported. However, as can be seen in figure 2
the morphology and size of the powder particles remains very regular and
spherical with a relatively small proportion of satelite particles.

Figure 2: a) Ti-6Al-4V gas atomized powder obtained from AP&C b) Water
atomized ATOMET 4801 Fe-4wt%Ni powder provided by Rio Tinto Metal Powders c)
magnified view of the ATOMET 4801 powder showing the irregular size and shape.

The second powder, ATOMET 4801, was provided by Rio Tinto Metal Powders. This
water atomized Fe-4wt%Ni alloy is conventionally used in powder metallurgy
applications. This powder, illustrated in figure 2b, is very irregular in
shape with a with a wide size distribution. Sieving produces results showing
that 10wt% of particles are larger than 150 $\mu$m in size (U.S. mesh +100),
62wt% are larger than 45 $\mu$m in size (U.S. mesh +325) and 28wt% are less
than 45 $\mu$m in size (U.S. mesh -325). The ATOMET powder was used in an un-
recycled state.

While the two powders appear very differently in figure 2, they have similar
apparent (tap) densities (2.47 g/cm3 for Ti-6Al-4 [24] and 3.00 g/cm3 for
ATOMET [25] and the same flow rate (25 s/50g) as measured by Hall Flow meter
following ASTM B213 [24, 25].

Finally, the feeder has been used to perform simple single layer melt trials
with the two powders. Here we focus only on the results for the Fe-4wt%Ni
powder with the aim of illustrating the potential for feeding and melting of
this less conventional (for additive manufacturing) powder within electron
beam additive manufacturing. An AISI4340 steel build plate was used and a
single layer of powder was delivered to the build plate by controlling the
feed rate from the feeder and the translation of the build plate (on its x-y
translation stage) beneath it. This was done to achieve an approximate layer
thickness of 0.2 mm. Seven independent 15 $\times$ 15 mm squares were melted
using a simple outer contour followed by infill by hatching. Prior to
depositing the powder layer, the build plate was heated to 940∘C to eliminate
smoking. For both hatching and contouring an electron beam accelerating
voltage of 100 kV and a beam velocity of 3.3 mm/s was used. The beam currents
for each square varied from 0.8 to 2.0 mA in increments of 0.2 mA.

##  3 Experimental Results:

###  3.1 Frequency Controlled Feeding of Powders:

The construction of the feeder is such that powder only flows when the voice
coil is activated, and even then only when the correct conditions are imposed.
We have evaluated this under two sets of conditions. First, Figure 3 shows the
resulting feed rate for the two powders, fed in air, for different imposed
frequencies of the voice coil at fixed driving amplitude (fixed imposed
voltage maximum to voice coil). While the results in figure 3 show a general
trend of increasing feed rate with reduced frequency, significant scatter
exists within the data.

Figure 3: The measured mass flow rate, estimated from the mass of powder
delivered within 10 s, for a fixed voice coil displacement excitation
(amplitude) and different voice coil frequencies for a) Ti-6Al-4V powder and
b) ATOMET 4801. The lines are intended only as a guide to the eye.

As the voice coil induced oscillations are imposed via its coupling to the
feeder, control of the feed rate by varying voice coil frequency at fixed
amplitude was found to be challenging. In the setup used here it was found
that it was not possible to independently control the feeder’s vibration
frequency and displacement as the system behaves as a driven, damped harmonic
oscillator. This can be shown by means of figure 4. Here, the measured feeder
displacement is shown as a function of frequency for the same conditions as in
figure 3, the same color scheme being used to indicate the results from the
two different powders. Here, although the amplitude (voltage) imposed on the
voice coil was fixed to be constant, the measured feeder displacement was
found to increase strongly with decreasing frequency, independent of the type
of powder used. This dependence is exactly the one that would be expected for
a driven, overdamped harmonic oscillator. A system of mass $m$ driven through
a spring (constant $k$) and dashpot (damping coefficient $\nu$) by a source
with amplitude $X_{0}$ and frequency $\omega$ will respond with an amplitude
$x_{0}$ such that,

$\frac{x_{0}}{X_{0}}=\frac{\omega_{0}^{2}}{\left[\left(\omega_{0}^{2}-\omega^{2}\right)^{2}+v^{2}\omega^{2}\right]^{1/2}}$
(1)

where $\omega_{0}=\sqrt{k/m}$ is the natural frequency of the system. This
relationship is plotted as a line on figure 4 with assumed values for $X_{0}$
and $\omega_{0}$. As can be seen this explains this data well under the
condition that the feeder contains the same mass ($m$) of the two powders as
noted above. This result also implies that when $\omega=\omega_{0}$ resonance
will occur. Evidence of this was observed experimentally as, at certain
combinations of imposed voice coil frequence/amplitude, the measured
displacement signal was found to change from a well controlled, regular
sinusoidal variation, to a more chaotic pattern indicative of
$\omega\rightarrow\omega_{0}$.

Figure 4: Measured feeder displacement amplitude versus frequency for both
Ti-6Al-4V and Fe-Ni powders when the imposed vibration amplitude of the voice
coil was fixed as constant. Also plotted as a solid line is the predicted
behaviour assuming the feeder/voice coil system as a driven, damped harmonic
oscillator.

###  3.2 Displacement Controlled Feeding:

The results in figure 4 show only a small dependence of feeder displacement on
imposed frequencies for frequencies greater than $\sim$40 Hz. Thus, further
experiments and development were focused onto controlling feed rate by fixing
imposed frequency, at $\omega>40$ Hz and varying the displacement of the
feeder using the measured displacement from the displacement sensor. Figure 5
shows the resulting feed rates measured in air for the two powders when the
voice coil frequency was fixed to 57 Hz and the imposed amplitude varied to
give the (fixed) feeder displacements shown. Under these conditions, one can
see that the mass flow rate varies nearly linearly with the feeder
displacement above a lower threshold displacement for both powders. The mass
flow rate for the Ti-6Al-4V powder is higher for all tested displacements, but
it is possible to obtain the same stable flow rate for both the highly
spherical Ti-6Al-4V and highly irregular ATOMET 4801 powder by the judicious
selection of the feeder displacement. While figure 5 was generated from
experiments performed in air, the same experiments were repeated for feeding
within the vacuum of the LEAM chamber. No significant differences from the
feeding in air and vacuum were observed.

Figure 5: The mass flow rate as a function of measured feeder displacement
amplitude for both gas atomized Ti-6Al-4V and water atomized ATOMET 4801
powders. The lines are only intended as a guide to the eye.

Figure 5 shows that the feeder operated under displacement control at fixed
frequency is not only able to feed conventional gas atomized powder but also
the highly irregular Fe-Ni alloy powder with no further modifications. While
the flow rate is significantly lower than that of the Ti-6Al-4V powder for a
given feeder amplitude, it is still possible to achieve the same flow rate for
the two powders by adjusting the feeder’s displacement amplitude.

###  3.3 Single Layer Melt Trials on ATOMET 4801 Water Atomized Powder

With this result in mind, single layer melt trials were conducted to
illustrate the potential for feeding of the Fe-4wt%Ni powder. Figure 6a shows
a macroscopic, top down view of the 7 squares produced, square 1 being
produced with the highest beam current and square 7 with the lowest.
Macroscopically, it can be seen that all squares appear similar with only
square 7 showing a significant difference from the rest. Figures 6b and c show
higher magnification top down views taken via scanning electron microscopy
from (b) square 3 and (c) square 7. The appearance of the melt tracks in
figure 6b are representative of the melt tracks observed for all other
squares, save square 7. Square 7 (6c), on the other hand showed clear evidence
of balling due to insufficient input power for consistent melting. Figure 7
shows a cross-section through square 3 showing the uniformity of the melt
depth and the lack of evidence for large scale defects (e.g. lack of fusion or
‘key hole’ formation). This was consistent for all squares, save square 7.

Figure 6: a) Top down view of single layer melt trials for different electron
beam conditions using the ATOMET 4801 powder. The experiments were performed
with an accelerating voltage of 100 kV, beam velocity of 3.3 mm/s (hatching)
and beam currents of ranging from 2.0 mA to 0.8 mA in decrements of 0.2 mA for
cases 1-7 b) Low magnification secondary electron image showing the hatched
region for a) case 1 (2.0 mA) and b) case 7 (0.8 mA). All but case 7 showed
nearly identical melt tracks to case 1. Case 7 was the only to show clear
evidence of under melting. Figure 7: A higher magnification view (top down) of
case 1 from Figure 6 along with secondary electron images taken through the
cross section. Here, one can see significant mixing with the underlying base
plate and a clear heat affected zone in the region where hatching was
performed.

##  4 Interpreting Feeder Behaviour using DEM simulations

In order to understand the behaviour exhibited by the feeder described above,
particularly the response exhibited in figure 5, we have used discrete element
(DEM) simulations. This follows on a large body of work using DEM simulations
to understand powder flow in additive manufacturing (see e.g. [9, 13, 8, 26,
27, 11] Rather than attempt to quantitatively predict the experimental
results, the aim of the simulations performed here was to reveal the
mechanisms underlying controlling mechanisms. The geometry of the simplified
feeder setup used in the models is shown in figure 8a. The model consists of
hopper, feed channel and exit chute but is simplified, compared to the actual
feeder, so as to be periodic in the y-direction. This allowed for a
significant speedup in simulation time (owing to the need to track fewer
particles) though at the expense of neglecting powder-feeder wall interactions
in the y-direction. Simulations were performed using the LIGGGHTs DEM package
[28] built on of the LAMMPS simulation platform [29]. Simulations used a
Hertz-Mindlin no-slip contact model with a simplified JKR cohesion model to
simulate particle-particle adhesion. Briefly, the equations of motion for each
particle $i$ of mass $m_{i}$ are solved according to,

$m_{i}=m_{i}g+\sum_{j\neq i}^{N}F_{n,j}+F_{t,j}+F_{coh,j}$ (2)

where $g$ is acceleration to due gravity and the normal force is calculated
for particles of a single type and size (radius, $R$) as,

$F_{n,j}=\frac{4}{3}\frac{E}{1-\nu^{2}}R^{1/2}\delta_{n}^{3/2}-\eta_{n}v_{n}$
(3)

and the tangential force is,

$F_{t,j}=\frac{8E}{2\left(2-\nu\right)\left(1+\nu\right)}R^{1/2}\delta_{n}^{1}/2\delta_{t}-\eta_{t}v_{t}$
(4)

The parameters $\delta_{n}$ and $\delta_{t}$ correspond to the normal and
tangential overlap of contacting surfaces. The damping coefficients $\eta_{n}$
and $\eta_{t}$ used above follow the dependence on the coefficient of
restitution as described in [30].

Rather than use the simplified JKR model implemented in LIGGGHTS, we modified
this model to use it in its conventional form,

$F_{coh,j}=-\gamma_{coh}E^{1/2}R^{3/4}\delta_{n}^{3/4}$ (5)

where $\gamma_{coh}$ here is a material parameter that measures the cohesive
strength of the interaction (see e.g. [13]).

The material parameters used here are defined in Table 1, these having been
strongly inspired by the experimental and simulation work on Ti-6Al-4V powders
as reported in the work of Meier _et al._ [13]. An inherent limitation of DEM
simulations applied to ‘stiff’ materials, is the short time scale required to
resolve particle-particle interactions [31, 32]. Taking properties consistent
with titanium would require one to use a time step of $<$10 ns, this requiring
100 million time-steps to simulate 1 second of feeder operation. To overcome
this timescale limitation, we follow the convention in the field using a ‘soft
particle’ approximation. Rather than using the actual modulus for Ti, the
particles are assumed to have a much reduced elastic modulus, this leading to
a much longer interaction time during particle contact. It has been shown that
for conditions similar to those used here the effect on predictions is small
(see e.g. [31, 13]). Time-steps were selected to be sufficiently small so as
to satisfy the Rayleigh criterion [32]. Most simulations shown here were
performed with mono-sized, spherical particles. A relatively large particle
radius of 120 $\mu$m was used to again reduce the total number of particles
simulated and thus the total required simulation time. Tests performed for
particle radii down to 50 $\mu$m showed no appreciable difference in the
qualitative trends illustrated below.

Property | Value
---|---
Density, $\rho$ | 4.5 g/cm3
Poisson’s Ratio, $\nu$ | 0.3
Young’s Modulus,$E$ | 0.1 GPa
Coefficient of Restitution, $e$ | 0.3
Cohesion Parameter, $\gamma_{coh}$ | 1.4 $\times 10^{-5}J^{1/2}/m$
Table 1: Material Parameters used in DEM model Figure 8: a) Illustration of
the model used for simulating feeder behaviour. The model is periodic in the
y-direction and the surfaces shown were taken to have the same contact
behaviour as the particles (red). b) Illustration of the basic elements of the
Hertzian contact model used in simulating the behaviour of powder particles.

Simulations involved two steps. First, particles were inserted into a region
at the top of the feeder via a rain model [33] and allowed fall into the
hopper until the hopper was nearly filled. The powder naturally flowed into
the feed channel establishing a stable distribution within it. This is
illustrated in figure 9a which shows the simulation setup at the end of the
feeding step where the characteristic angle adopted by the powder within the
feed channel, related via the assumed contact mechanics and cohesion model to
the angle of repose for the powder, can be seen. This observation immediately
explains why powder does not continuously feed through the feeder without
vibration; so long as the feed channel is long enough to contain this
characteristic powder angle then the powder will not flow so long as the
feeder is not vibrating.

Figure 9: a) - d) Illustration of progressive feeding of powder from the
simulated feeder. Simulation results for particle radius of 50$\mu$m,
vibration frequency and amplitude of 40 Hz and 0.25 mm. a) represents the
initial configuration prior to feeding. e) The number of particles fed as a
function of time for different displacement amplitudes of the feeder. In this
case the particle size was 120 $\mu$m and the vibration frequency was 100 Hz.
As one can see, the feed rate remains constant, for a given displacement
amplitude and frequency, over the entire range of feeding.

Each simulation of feeding started from the same initial equilibrium
distribution of powder within the feeder (figure 9a). To simulate vibration in
this case, the entire feeder was subjected to a sinusoidal horizontal
displacement in the $x$-direction, $\Delta x=A\sin{\left(\omega t\right)}$,
with amplitude $A$ and frequency $\omega$.

Upon initiating vibration, powder begins to flow through the feeding channel
and out the exit chute. This is illustrated for one condition in the sequence
of images shown in figures 9a-d. The feed rate in this case was quantified by
measuring the number of particles to exit from the bottom of the chute as a
function of time. This is shown for different values of the vibration
amplitude $A$ at a fixed value of frequency (100 Hz) in figure 9e. Figures
9a-d show flow for an amplitude A = 0.25 mm and frequency of 40 Hz. As one can
see, following an initial period required for powder to flow from its initial
position to the exit chute, a constant rate of powder flow (linear
relationship between number of particles lost and time) is established and
maintained until nearly all powder has been removed from the feeder. The
constant rate of feeding, regardless of the mass of powder in the feeder means
that we can assess the feed rate as a function of amplitude and frequency
without having to be concerned with the time dependence of the feed rate. This
also corresponds well with our experimental observations which also showed
that the feed rate was not strongly sensitive to the mass of powder remaining
in the feeder.

Figure 10 shows the predicted feed rate as a function of vibration amplitude,
$A$, for an imposed frequency of 40 Hz. One can see in this plot that for
amplitudes between 0 and $\sim$0.4 mm the linear relationship observed
experimentally in figure 5 is well reproduced by the simulations, as is the
cut-off of flow below a certain imposed amplitude. Below this amplitude the
feeder oscillates but no powder is seen to leave the feeder.

Figure 10: DEM model prediction of steady-state particle feed rate as a
function of amplitude at a fixed imposed frequency of 40 Hz. Inset shows the
low amplitude portion of the plot magnified to show that feeding stops at a
finite displacement amplitude. The line is shown as a guide to the eye.

At amplitudes $A\gtrapprox 0.4$ mm one sees that the feed rate deviates away
from linear, and for amplitudes above $A\gtrapprox 0.6$ the feed rate is seen
to decrease.

We can understand the behaviour in figure 10 better by looking at how powder
flows through the system (figure 11). In particular we focus on the vertical
and horizontal velocities of the powder at different locations in the
simulated feeder. As the feeder moves back and forth in the $x$-direction, the
powder also moves in a cyclic fashion meaning that we can identify a maximum
and minimum velocity of particles in $x$-direction. This is shown in figure
11a for particles in the blue shaded region of the feed channel. When the
feeder displacement amplitude is small, the powder displacements are small and
in an opposite sense on opposite sides of the (forward and backward) strokes.
The lines drawn on figure 11a show the maximum forward and backward velocities
that would be expected of the powder if it simply followed the motion of the
feeder. It can be seen that for small feeder displacements the powder closely
follows the motion of the feeder, simply moving back and forth as the feeder
oscillates. This creates no net flow of the powder through the feed channel.
As displacement amplitude is increased, the forward motion of the powder
continues to follow that of the feeder (in the positive direction, towards the
chute) but the reverse direction of flow saturates, and eventually starts to
increase, eventually the flow of powder continuing to be in the positive
$x$-direction though the direction of feeder is in the negative $x$-direction.
At small displacements, the forward stroke of the feeder compresses the powder
against the force induced between the particles and the top and bottom of the
feed channel walls. On the reverse stroke the powder recovers and flows
backward to recover its original position. This, apparent ‘compressibility’ of
the powder saturates as the force of friction does not continue to increase
with larger displacements. Rather, at larger displacement amplitudes, the
inertia induced by the feeder in the forward stroke is sufficient to keep the
powder flowing in the forward direction for the time that the feeder direction
is reversed. Thus, at small amplitudes the forward and backward motion of the
powder cancel leading to no net flow. As amplitude is increased, the net flow
follows roughly that given by the motion during the forward stroke of the
feeder as little motion occurs in the reverse stroke. Thus, under these
conditions, feeding can be described as a form of ‘ratcheting’ with the flow
being induced largely be the inelastic translation of powder induced in the
forward portion of the stroke.

Figure 11: a) Average horizontal velocity of particles in the region of the
feed channel illustrated in blue b) Average vertical velocity of particles in
the region of the feed chute illustrated in blue. The inset in (b) shows a
magnified view at small displacement amplitudes.

This explanation describes the behaviour illustrated in figure 5 but as can be
seen in figure 10 the rate of flow from the feeder reduces beyond a critical
amplitude (here $\sim$ 0.6 mm). To understand this behaviour it is instructive
to look at the vertical flow of powder from the hopper into the feed channel
as shown in figure 11b. Here, one can see that the flow of powder from the
hopper into the feed channel increases linearly with amplitude for amplitudes
$A\lessapprox 0.4$ mm with a cutoff at small amplitudes, consistent with the
observations in figure 11a. At high amplitudes, $A\gtrapprox 0.6$ mm, the feed
rate from hopper to chute is seen to decrease. The reasons for this can be
seen graphically in figure 12. Here it can be seen that high displacement
amplitudes result in significant upward motion of the powder in the hopper
(compare to the relatively quiescent behaviour of the powder in the hopper
shown in figure 9). This upward motion is a result of the high velocity of the
feeder and an inability for the powder to relax and follow the horizontal
translation of the feeder. In this case, the vertical motion of the powder
reduces the amount of powder flowing into the feed channel and thus the feed
rate. This illustrates the need for careful consideration of the operation
window of the feeder to ensure that feeding is conducted in a regime where
powder flow remains stable both within the hopper and feed channel. This sets
a limit on the maximum feed rate that can be obtained under displacement
controlled feeding.

Figure 12: Series of snapshots from a DEM simulation performed with a
frequency of 40 Hz and amplitude $A$ = 0.8 mm showing the significant upward
displacement of the powder in the hopper. This upward motion results in a
decreased delivery of powder into the feed channel and thus a reduction in the
feed rate.

While the experiments presented above were limited to displacement controlled
conditions, one may wonder whether frequency control (at fixed displacement)
is a viable option for controlling flow. Figure 13 shows that the relationship
between feed rate and frequency is much more complex than between feed rate
and amplitude (at fixed frequency) with the feed rate varying in a non-
monotonic fashion with frequency. This can be traced to effects similar to the
one illustrated in figure 12 where, at characteristic frequencies depending on
the displacement amplitude, the powder flow becomes unstable within the
hopper. This result suggests that controlling feeder flow at fixed frequency
and taking advantage of the (nearly) linear relationship between flow and
displacement amplitude should provide the best opportunity for process
control. It also points to the need for careful experimental calibration to
identify conditions where unstable flow may be a concern.

Figure 13: The effect of feeder vibration frequency at fixed amplitude $A=0.2$
mm on the feed rate of powder simulated via DEM.

A final comment on the potential impact of actual powders that are not
monosized spheres but which, as discussed in reference to the two powders
studied experimentally here, have size distributions, irregular shapes and/or
a range of densities (e.g. if powder mixtures are used). While the effect of
particle shape has not been studied within the context of the DEM simulations
presented here, the impact of a bi-modal distribution of particle sizes (at
constant density) and particle density (at fixed size) has. Vibration induced
segregation, often referred to as the ‘brazil nut effect’, is known to lead to
large particles being segregated to the top of a vertically vibrating granular
bed[34]. A variety of views exist to explain this effect, but the controlling
mechanisms remain a topic of active consideration [34, 35]. Two additional
sets of simulations were performed, following the same approach as described
above, but with two significant differences. In the first set of simulations,
during feeding, particles with two distinct sizes (radii of 120 $\mu$m and 60
$\mu$m) were deposited into the feeder, all other particle properties being
held constant. In a second set of simulations, particles with two distinct
densities (2.5 g/cm3 or 9.0 g/cm3 (two times smaller and larger than the
densities of the particles used in the above simulations) were deposited all
other particle properties, including particle size, being held the same as
those used in previous simulations. Also different from the above simulations
was the fact that the powder was deposited onto a moving horizontal surface
(properties the same as those used above) to simulate the deposition of a
powder layer on a build table.

No evidence of density or size induced segregation was observed within the
feeder for either of these simulations. This can be illustrated by measuring
the time evolution of the fraction of the particle types within the vertical
channel separating the feed channel from the hopper as a function of time,
using this as a measure of the ‘composition’ of the powder being fed from the
hopper. This is illustrated for the case where two particle densities were
used in figure 14d. Note, the elevated fraction at $t\approx 0.4-0.5$
corresponds to the layer of higher density particles located at the top of the
feeder (see e.g. figure 14a), this being an artifact arising from the ‘rain
model’ method used to fill the feeder. The lack of discernible segregation
based on size or density due to vibration is a consequence of the fact that
any vibration induced convection within the hopper is overwhelmed by the
downward flow of powder from the hopper into the feed channel. Powder is drawn
from across the entire width of the hopper into the feed channel with no
evidence of preferential flow of one type of particle.

Figure 14: Simulated powder bed feeding for a mix of two powders having the
same size (radius of 120 $\mu$m) but two different densities. Panels a)-c)
show the system before the onset of vibration, after 0.56 s and 1.1 s. Blue
particles have a density of $\rho$ =2.25 g/cm3 and yellow a density of 9.0
g/cm3. d) Fraction of particles with $\rho$ =9.0 g/cm3 exiting the hopper as a
function of time ($t=0$s corresponding to the onset of feeding). e) The
fraction of particles with a density of 9.0 g/cm3 as a function of position
along the powder layer, the powder layer position relative to this graph being
shown in (f). The solid lines in (d) and (e) indicate the initial average
fraction of particles with $\rho$ =9.0 g/cm3. The vertical bar in (e) is
intended to indicate the position of the feeder’s exit chute. g) Shows a
higher magnification of (f) illustrating the tendency for lower density
particles to segregate to the right of the powder bed.

In the case of the powder bed itself, no evidence of segregation was observed
for the case of the powder containing a bi-modal distribution of particle
sizes. It was confirmed that the initial hopper composition and the
composition of the powder bed (at all times) remained the same. In the case of
the powder with two particle densities, however, a small difference between
the powder bed ‘composition’ and bulk initial ‘composition’ were observed,
this being illustrated in figure 14e. Careful inspection of figures 14e, f and
g show that a preferential accumulation of lower density powder occurs to the
left of the developing powder bed. Due to the lower mass of these smaller
particles, they tend to be ejected to a larger distance upon impact with the
build table/particles on the build table after exiting the feeder. This is
much more obvious to the left of the feeder where the number of particles is
lower and (over time) the particles occupying that space are themselves lower
in mass. Ultimately, segregation reaches steady state leading to a stable
powder bed composition slightly richer in the denser powder; 2% more of the
particles with $\rho$ =9.0 g/cm3 in the powder bed compared to than in the
bulk. The same 2% enrichment of the powder bed was observed in a simulation
where powder densities of $\rho$ =2.25 and 4.5 g/cm3 were used.

One way to further improve powder feeding, and to help further mix the powder
as it enters the powder bed is to use the bottom surface of the feeder to
‘level’ the powder bed. In this mode the feeder is much closer to the build
table and the powder ‘injected’ into the powder bed, this leading to both a
flatter surface and an additional opportunity for mixing of powders of
different densities/sizes within the powder bed. This opens up many further
opportunities to think about alternative ways of delivering powder (or
powders) during PBF-AM processing, with the potential to overcome many of the
limitations discussed in the opening for conventional re-coating technologies.

##  5 Conclusions

In this work we have illustrated the use of a vibratory powder feeding system
in conjunction with electron beam additive manufacturing. It has been shown to
be possible to feed not only ‘conventional’ gas atomized powders used
extensively in additive manufacturing but also a highly irregularly shaped and
broadly sized water atomized Fe-Ni powder conventionally used in the powder
metallurgy industry. It has been shown that the feed rate, and thus the powder
layer height can be accurately controlled thanks to a linear relationship
between feed rate and vibration amplitude at fixed frequency. Simulations have
revealed the underlying mechanism controlling the feeding behaviour under
these conditions including the threshold vibration amplitude below which no
feeding occurs. Simulations suggest that size based segregation is not
expected but that a small amount of powder based segregation may occur during
feeding of particles with very different densities. Careful monitoring would
be required in this case to ensure the desired bulk chemistry is obtained.
Opportunities to introduce additional powder mixing in the powder bed itself
have also been suggested.

The use of a vibratory powder feeder, as introduced here, can overcome some of
the limitations inherent in re-coaters, one significant one shown here being
the ease with which it can distribute powder that would be considered to
difficult to flow (without sizing or mixing with other powders) in
conventional electron beam or laser based additive manufacturing. This opens
the opportunity for new strategies in powder delivery for additive
manufacturing that are not conventionally considered in existing commercial
systems.

##  6 Bibliography

## References

  * [1] S. Das, Physical Aspects of Process Control in Selective Laser Sintering of Metals (2003).  doi:10.1002/adem.200310099.
  * [2] C. Körner, Additive manufacturing of metallic components by selective electron beam melting - A review (2016).  doi:10.1080/09506608.2016.1176289.
  * [3] L. Dowling, J. Kennedy, S. O’Shaughnessy, D. Trimble, A review of critical repeatability and reproducibility issues in powder bed fusion (2020).  doi:10.1016/j.matdes.2019.108346.
  * [4] M. Mani, S. Feng, B. Lane, A. Donmez, S. Moylan, R. Fesperman, Measurement science needs for real-time control of additive manufacturing powder-bed fusion processes, in: Additive Manufacturing Handbook: Product Development for the Defense Industry, 2017.  doi:10.1201/9781315119106.
  * [5] H. W. Mindt, M. Megahed, N. P. Lavery, M. A. Holmes, S. G. Brown, Powder Bed Layer Characteristics: The Overseen First-Order Process Input, Metallurgical and Materials Transactions A: Physical Metallurgy and Materials Science 47 (8) (2016).  doi:10.1007/s11661-016-3470-2.
  * [6] S. Yang, J. R. Evans, Metering and dispensing of powder; the quest for new solid freeforming techniques (2007).  doi:10.1016/j.powtec.2007.04.004.
  * [7] R. J. Hebert, Viewpoint: metallurgical aspects of powder bed metal additive manufacturing, Journal of Materials Science 51 (3) (2016).  doi:10.1007/s10853-015-9479-x.
  * [8] S. Haeri, Optimisation of blade type spreaders for powder bed preparation in Additive Manufacturing using DEM simulations, Powder Technology 321 (2017).  doi:10.1016/j.powtec.2017.08.011.
  * [9] C. Meier, R. Weissbach, J. Weinberg, W. A. Wall, A. J. Hart, Critical influences of particle size and adhesion on the powder layer uniformity in metal additive manufacturing, Journal of Materials Processing Technology 266 (2019).  doi:10.1016/j.jmatprotec.2018.10.037.
  * [10] U. Ali, Y. Mahmoodkhani, S. Imani Shahabad, R. Esmaeilizadeh, F. Liravi, E. Sheydaeian, K. Y. Huang, E. Marzbanrad, M. Vlasea, E. Toyserkani, On the measurement of relative powder-bed compaction density in powder-bed additive manufacturing processes, Materials and Design 155 (2018).  doi:10.1016/j.matdes.2018.06.030.
  * [11] H. Chen, Q. Wei, Y. Zhang, F. Chen, Y. Shi, W. Yan, Powder-spreading mechanisms in powder-bed-based additive manufacturing: Experiments and computational modeling, Acta Materialia 179 (2019).  doi:10.1016/j.actamat.2019.08.030.
  * [12] B. K. Foster, E. W. Reutzel, A. R. Nassar, B. T. Hall, S. W. Brown, C. J. Dickman, Optical, layerwise monitoring of powder bed fusion, in: Proceedings - 26th Annual International Solid Freeform Fabrication Symposium \- An Additive Manufacturing Conference, SFF 2015, 2020.
  * [13] C. Meier, R. Weissbach, J. Weinberg, W. A. Wall, A. John Hart, Modeling and characterization of cohesion in fine metal powders with a focus on additive manufacturing process simulations, Powder Technology 343 (2019).  doi:10.1016/j.powtec.2018.11.072.
  * [14] Z. Snow, R. Martukanitz, S. Joshi, On the development of powder spreadability metrics and feedstock requirements for powder bed fusion additive manufacturing, Additive Manufacturing 28 (2019) 78–86.  doi:10.1016/j.addma.2019.04.017.
URL https://linkinghub.elsevier.com/retrieve/pii/S2214860418309941

  * [15] I. E. Anderson, E. M. White, R. Dehoff, Feedstock powder processing research needs for additive manufacturing development (2018).  doi:10.1016/j.cossms.2018.01.002.
  * [16] S. Hoeges, A. Zwiren, C. Schade, Additive manufacturing using water atomized steel powders, Metal Powder Report 72 (2) (2017).  doi:10.1016/j.mprp.2017.01.004.
  * [17] A. J. Pinkerton, L. Li, Direct additive laser manufacturing using gas- and water-atomised H13 tool steel powders, International Journal of Advanced Manufacturing Technology 25 (5-6) (2005).  doi:10.1007/s00170-003-1844-2.
  * [18] T. Fedina, J. Sundqvist, J. Powell, A. F. Kaplan, A comparative study of water and gas atomized low alloy steel powders for additive manufacturing, Additive Manufacturing 36 (12 2020).  doi:10.1016/j.addma.2020.101675.
  * [19] D. Palousek, L. Pantelejev, T. Zikmund, D. Koutny, Processing of nearly pure iron using 400W selective laser melting – Initial study, MM Science Journal 2017 (FEBRUARY) (2017).  doi:10.17973/MMSJ.2017{\\_}02{\\_}2016184.
  * [20] M. Letenneur, V. Brailovski, A. Kreitcberg, V. Paserin, I. Bailon-Poujol, Laser Powder Bed Fusion of Water-Atomized Iron-Based Powders: Process Optimization, Journal of Manufacturing and Materials Processing 1 (2) (2017).  doi:10.3390/jmmp1020023.
  * [21] M. Letenneur, P. Imbrogno, A. Molavi-Kakhki, V. Brailovski, Laser Powder Bed Fusion with Intentionally-Seeded Porosity for Prototyping of Powder Metallurgy Parts, Journal of Manufacturing and Materials Processing 4 (4) (2020).  doi:10.3390/jmmp4040119.
  * [22] N. K. Tolochko, S. E. Mozzharov, M. K. Arshinov, K. I. Arshinov, M. B. Ignat’ev, Laws governing vibrating feeding of finely dispersed powder into a laser sintering zone, Powder Metallurgy and Metal Ceramics 43 (9) (2004).  doi:10.1007/s11106-004-0016-8.
  * [23] Y. Y. Sun, S. Gulizia, C. H. Oh, C. Doblin, Y. F. Yang, M. Qian, Manipulation and Characterization of a Novel Titanium Powder Precursor for Additive Manufacturing Applications, JOM 67 (3) (2015).  doi:10.1007/s11837-015-1301-3.
  * [24] A. P. Metallurgy, Ti-6Al-4V grade 5 datasheet (2021).
URL https://www.advancedpowders.com/powders/titanium/ti-6al-4v-5

  * [25] Rio Tinto Metal Powders, Atomet4801 datasheet (2021).
URL http://qmp-powders.com/wp-content/uploads/Product%20PDFs/ATOMET_4801.pdf

  * [26] E. B. Herbold, O. Walton, M. A. Homel, Simulation of Powder Layer Deposition in Additive Manufacturing Processes Using the Discrete Element Method, Llnl-Tr-678550 (LLNL-TR-678550) (2015).
  * [27] E. J. Parteli, DEM simulation of particles of complex shapes using the multisphere method: Application for additive manufacturing, in: AIP Conference Proceedings, Vol. 1542, 2013.  doi:10.1063/1.4811898.
  * [28] C. Kloss, C. Goniva, A. Hager, S. Amberger, S. Pirker, Models, algorithms and validation for opensource DEM and CFD-DEM, Progress in Computational Fluid Dynamics 12 (2-3) (2012).  doi:10.1504/PCFD.2012.047457.
  * [29] S. Plimpton, Fast parallel algorithms for short-range molecular dynamics, Journal of Computational Physics 117 (1) (1995).  doi:10.1006/jcph.1995.1039.
  * [30] Y. Tsuji, T. Tanaka, T. Ishida, Lagrangian numerical simulation of plug flow of cohesionless particles in a horizontal pipe, Powder Technology 71 (3) (1992).  doi:10.1016/0032-5910(92)88030-L.
  * [31] S. Lommen, D. Schott, G. Lodewijks, DEM speedup: Stiffness effects on behavior of bulk material, Particuology 12 (1) (2014).  doi:10.1016/j.partic.2013.03.006.
  * [32] M. Otsubo, C. O’Sullivan, T. Shire, Empirical assessment of the critical time increment in explicit particulate discrete element method simulations, Computers and Geotechnics 86 (2017).  doi:10.1016/j.compgeo.2016.12.022.
  * [33] P. Meakin, R. Jullien, Restructuring effects in the rain model for random deposition, Journal de Physique 48 (10) (1987).  doi:10.1051/jphys:0198700480100165100.
  * [34] J. M. N. T. Gray, Particle Segregation in Dense Granular Flows (2018).  doi:10.1146/annurev-fluid-122316-045201.
  * [35] L. Jing, J. M. Ottino, R. M. Lueptow, P. B. Umbanhowar, Rising and sinking intruders in dense granular flows, Physical Review Research 2 (2) (2020).  doi:10.1103/physrevresearch.2.022069.