# Correcting Diverse Factual Errors in Abstractive Summarization via Post-
Editing and Language Model Infilling

Vidhisha Balachandran♣ Hannaneh Hajishirzi♢♡
William W. Cohen♠ Yulia Tsvetkov♡
♣Language Technologies Institute, Carnegie Mellon University
♢Allen Institute for Artificial Intelligence
♡Paul G. Allen School of Computer Science & Engineering, University of
Washington
♠Google Research
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Abstractive summarization models often generate inconsistent summaries
containing factual errors or hallucinated content. Recent works focus on
correcting factual errors in generated summaries via post-editing. Such
correction models are trained using adversarial non-factual summaries
constructed using heuristic rules for injecting errors. However, generating
non-factual summaries using heuristics often does not generalize well to
actual model errors. In this work, we propose to generate hard, representative
synthetic examples of non-factual summaries through infilling language models.
With this data, we train a more robust fact-correction model to post-edit the
summaries to improve factual consistency. Through quantitative and qualitative
experiments on two popular summarization datasets— CNN/DM and XSum—we show
that our approach vastly outperforms prior methods in correcting erroneous
summaries. Our model—FactEdit—improves factuality scores by over $\sim$11
points on CNN/DM and over $\sim$31 points on XSum on average across multiple
summarization models, producing more factual summaries while maintaining
competitive summarization quality.111Code and data available at
https://github.com/vidhishanair/FactEdit.

##  1 Introduction

While modern summarization models generate highly fluent summaries that appear
realistic (Lewis et al., 2020; Zhang et al., 2020), these models are prone to
generating non-factual and sometimes entirely fabricated content Cao et al.
(2018); Goodrich et al. (2019); Maynez et al. (2020). With the increasing
adoption of language generation tools in user-facing products, such
unreliability poses severe risks, including the spread of misinformation,
panic and other potentially harmful effects Ranade et al. (2021); Hutson et
al. (2021).

Figure 1: Model generated summaries often produce content which is factually
inconsistent w.r.t. to the source. FactEdit rewrites these summaries by
maintaining the abstractiveness but correcting factual errors. Figure 2:
Architecture framework for FactEdit. Using masked versions of existing
reference summaries, we use an infilling language model to produce alternative
candidates for the mask position. We construct factually incorrect summaries
by replacing the mask with the lower ranked candidates. Finally, we train a
sequence-to-sequence model for fact correction using the synthetically
constructed data.

Since it is difficult to control for factuality at training or inference time
Huang et al. (2021); Dreyer et al. (2021), a popular approach to fix the
factual inconsistencies is via post-editing generated summaries Cao et al.
(2020); Dong et al. (2020). This allows summarization models to focus on
fluency and content-relevance while improving factual consistency. However,
there is no suitable data for training post-editing models to directly
“translate” an incorrect summary to a correct one. Prior work constructed
synthetic training data by introducing simple heuristic errors like replacing
entities or numbers in reference summaries Cao et al. (2020), but it is not
clear whether such synthetic errors have sufficient coverage and accurately
represent the types and distribution of actual errors made by language models.
Further, with increasing language generation capabilities, models make more
complex factual errors involving discourse structures and paraphrasing which
cannot be easily captured with heuristics Pagnoni et al. (2021). The goal of
our work is to develop post-editing models that generalize over a wider range
of factual errors (example in Figure 1) in generated summaries from diverse
summarization model types.

We propose FactEdit—a novel approach to post-editing text, to control for
content factuality in generated summaries. Rather than manually defining a
list of heuristic errors, it incorporates a new algorithm to generate
adversarial (non-factual) examples using infilling language models Donahue et
al. (2020). We use lower ranked beam-search candidates from the language model
as a source for potentially factually-incorrect summary facts, thereby
producing a set of plausible, likely, and fluent, incorrect synthetic
summaries for a particular correct reference summary. In this way, we leverage
the capabilities of large language models to produce multiple candidates of
alternative, erroneous summaries. These examples, along with factually correct
references, are then used to train a sequence-to-sequence fact-correction
model that aims at generating a factually consistent version of the candidate
summary (§2).

We evaluate FactEdit on two datasets - CNN/DailyMail Hermann et al. (2015) and
XSum Narayan et al. (2018) and across nine summarization models with the FRANK
benchmark Pagnoni et al. (2021) for evaluating various categories of factual
errors in generated summaries (§3). The two summarization datasets represent
varied distributions of factual errors in models trained on them and hence
constitute a good test bed to evaluate the generalizability of our model. We
show that FactEdit substantially improves factuality scores across two metrics
- Ent-DAE Goyal and Durrett (2021) and FactCC Kryscinski et al. (2020). On the
Ent-DAE metric, FactEdit improves results by $\sim$11 points (CNN/DM) and
$\sim$31 points (XSum), and on the FactCC metric we show improvements of
$\sim$6 points (CNN/DM) and $\sim$24 (XSum) points on average across models
(§4). Further, our analysis shows that FactEdit effectively corrects diverse
error categories without the need for special heuristics or annotations (§5).
An important application of FactEdit is to audit summarization systems and
facilitate their reliability.

##  2 Model

Assume a summarization model trained to process a document $d$ and generate a
coherent and fluent summary222We denote incorrect input (to fact correction
model) summaries using ′ and corrected output (from fact correction model)
without the ′ throughout this paper. For E.g: $g^{\prime}$ is incorrect
summary, $r^{\prime}$ is the incorrect reference summary while $g$ is the
corrected summary and $r^{\prime}$ is the corrected reference summary.
$g^{\prime}$ which has been shown to often misrepresent facts from the
document. FactEdit is a fact correction model $M_{C}$ which takes the
generated summary $g^{\prime}$ and document $d$, identifies factual errors and
generates a rewritten summary $g$ by correcting them (as outlined in Figure
2).

We present an adversarial data generation approach which leverages the power
of pre-trained language models to produce fluent and complex factually
incorrect summaries. We train an infilling language model $\mathbf{M}_{I}$
using documents from summarization training data and use the model to
introduce diverse factual errors in sentences from them (§2.1). Using the
trained model, we introduce factual errors in reference summaries of the
training data $r$ producing an incorrect summary $r^{\prime}$ resulting in a
synthetic dataset $\\{r^{\prime},r,d\\}_{train}$ of erroneous summaries mapped
to their corrected versions (pink section in Figure 2). We train a sequence-
to-sequence model $\mathbf{M}_{C}$ for factual error correction using the
generated synthetic data (§2.2). Finally, we use the trained correction model
to rewrite model generated summaries $g^{\prime}$ producing a corrected
version $g$ (§2.3 \- green section in Figure 2).

###  2.1 Infilling Data Generator $\mathbf{M}_{I}$

Our data generation process leverages infilling language models Donahue et al.
(2020) to produce candidates to fill masked phrases in a summary sentence. We
mask parts of the input and use the infilling model to generate multiple
candidates for the masked position. We then use lower order beam candidates as
potential incorrect candidates to generate an incorrect version of the input.
We hypothesize that, given the relevant context of a source document, a strong
language model generates relevant and factual sequences at higher
probabilities, compared to lower probability sequences. For the infilling
model, we hypothesize that the lower ranked candidates are often alternative
phrases of similar types (in case of entities) or parts-of-speech which are
plausible but often not factually correct. Motivated by prior work Goyal and
Durrett (2020) using lower ranked beam search candidates as a source for
adversarial data, we use the lower ranked candidates to construct erroneous
summaries from reference summaries.

#### Training:

Our infilling model $\mathbf{M}_{I}$ is trained to take a masked sentence
$s^{masked}$ and its relevant context $ctx$ as input and generate a correct
phrase to fill in the masked span. To train $\mathbf{M}_{I}$, we construct a
dataset using documents $d$ from the training data of existing summarization
datasets. For each sentence $s$ in the first-$k$ ($k$=5) positional sentences
of a document $d$, we identify the subjects, objects and relations {sub, obj,
rel} in them using OpenIE Banko et al. (2007). By iteratively masking each
phrase $p$ in {sub,obj,rel}, we create a masked query $s^{masked}$ and its
corresponding context $ctx$ by removing the masked sentence from the document,
resulting in our training data $\\{s^{masked},p,ctx\\}$, where $p$ is the
masked span text. We train a sequence-to-sequence model $\mathbf{M}_{I}$ on
this data which takes $s^{masked}$ [SEP] $ctx$ as input and learns to generate
$p$ as the output. We intentionally use only sentences from the document as
masked queries and do not use sentences from the reference summaries, to
ensure that the model does not memorize phrases from the references. Thus,
when applied to unseen reference sentences during inference, the model will
produces richer beam search candidates.

#### Adversarial Data Generation:

We use the trained infilling model to generate the synthetic dataset for fact
correction using the document reference pairs $\\{d,r\\}_{train}$ from the
summarization training data. For each sentence in the reference $s_{r}$, we
use OpenIE to extract {sub, obj, rel} and iteratively mask one phrase at a
time to construct masked sentences $s^{masked}$ from the references. We
provide this masked reference summary sentence and document $d$ as input to
the model and perform beam-search decoding for generation. We then consider
lower ranked beam candidates (rank=[5,15])333We chose this range of ranks
based on a manual analysis of 500 generated adversarial examples where our
method produced factually incorrect replacements over 90% of the time. as non-
factual alternatives for the corresponding masked phrase. We then use these
candidates as the replacements for the mask producing an erroneous summary
$r^{\prime}$. Running this on the $\\{d,r\\}_{train}$ training data, we
construct a synthetic data $\\{r^{\prime},r,d\\}_{train}$ of factually
incorrect summaries paired with their correct version where $r^{\prime}$ and
$r$ differ by an incorrect phrase. To train the model to not perform any
corrections on factual summaries, we keep original reference summaries for 20%
of the data points ($r^{\prime}=r$).

###  2.2 Fact Correction Model $\mathbf{M}_{C}$

Using the parallel data $\\{r^{\prime},r,d\\}_{train}$ produced by the above
infilling method, we train models for factual error correction. In contrast to
prior work which used pointer based models to copy phrases from the source
document, we use a sequence-to-sequence model like BART Lewis et al. (2020) to
preserve the abstractive content in the input. The model $\mathbf{M}_{C}$ is
trained with an erroneous reference summary sentence $s_{r^{\prime}}$ produced
by the infilling data generator and the corresponding document $d$ as input
and the correct reference summary sentence $s_{r}$ as output. A
straightforward option is to provide $s_{r^{\prime}},d$ concatenated as inputs
to the model. But we hypothesize that providing the right context can help the
model better correct the errors. Below we outline input structures that
provide better context in the input:
Relevant Supporting Passages:  To help the model better connect the relevant
facts in the source document to the summary sentence being corrected, we
experiment with providing only the most relevant parts of the document as
input context instead of the entire document. Using a scoring function
(ROUGE), we identify sentences from the document which have high overlap with
the generated summary sentence and extract the top-$k$ ($k$=3 for our work)
such sentences. We provide these sentences along with a window of $w_{k}$
($w_{k}$=2) sentences before and after each as the input context to the model.
Surrounding Summary Context:  While simple errors like incorrect entities can
be detected and corrected with only the context of the current sentence being
corrected, more complex discourse level errors like incorrect pronouns require
the context of the rest of the sentences of the summary. To enable this, we
additionally give the complete generated summary (other sentences from the
summary) as additional context. For single sentence summaries like headline
generation, this does not change the original setting, but for longer
summaries this setting helps with discourse level errors.

In essence, our model $\mathbf{M}_{C}$ takes the input as Incorrect Reference
Sentence ($s_{r^{\prime}})$ [SEP] Full Reference Summary ($r^{\prime}$) [SEP]
Relevant Passages and generates the corrected summary $r$ as output.

###  2.3 Inference

Our trained fact correction model $\mathbf{M}_{C}$ can be directly applied to
any model-generated summaries $g^{\prime}$, without access to the underlying
model. For each sentence in a generated summary, we identify the relevant
passages using ROUGE and provide it as an input to the model (in the form
Generated Summary Sentence ($s_{g^{\prime}}$) [SEP] Generated Full Summary
($g^{\prime}$) [SEP] Relevant Passages).

##  3 Experiments and Data

###  3.1 Datasets

We use two news summarization datasets CNN-DailyMail Hermann et al. (2015) and
XSum Narayan et al. (2018). The two datasets have been extensively studied for
the factual consistency in their generated summaries across a variety of
models Goodrich et al. (2019); Cao et al. (2018). Reference summaries from
CNN/DM are longer, having on average three sentences, and more extractive in
nature. XSum on the other hand has shorter, single sentence summaries and is
significantly more abstractive in nature. The summaries in these datasets are
qualitatively different, and hence models trained on the two datasets present
varied levels of challenges in maintaining factual consistencies.

Prior work have studied summaries generated from different language models and
characterized the factual errors in them (Pagnoni et al., 2021). Generated
summaries on the CNN/DM dataset are more extractive in nature and hence are
more factual ($\sim$70% of summaries are factual) than the more abstractive
generated summaries of XSum ($\sim$20% of summaries are factual). The longer
summaries in CNN/DM display discourse level inconsistencies while summaries
from XSum often hallucinate content which is not supported by the source
document. Hence, the two datasets present a varied setting for evaluating the
efficacy of our model across different kinds of errors. For our main
evaluation, we evaluate the overall capability of FactEdit in correcting
errors in summaries generated by a BART model.

We further evaluate our model on the FRANK benchmark Pagnoni et al. (2021)
which contains generated summaries obtained using multiple language models for
both datasets annotated with human judgements on their factuality and the
category of factual error. As different language models have different
distribution of factual error types, this evaluation helps us study the
generalizability of FactEdit in correcting errors across them.444As the
benchmark has publicly available model outputs, the summaries across different
datasets are from different models owing to their availability. For the CNN/DM
dataset, it contains model outputs from a LSTM Seq-to-Seq model (S2S) (Rush et
al., 2015), a Pointer-Generator Network (PGN) model (See et al., 2017), a
Bottom-Up Summarization (BUS) model (Gehrmann et al., 2018), a Bert based
Extractive-Abstractive model (BertSum) (Liu and Lapata, 2019) and a jointly
pretrained transformer based encoder-decoder model BART (Lewis et al., 2020).
For the XSum dataset, it contains model outputs from a Topic-Aware CNN Model
(Narayan et al., 2018), a Pointer-Generator Network (PGN) model, a randomly
initialized (TransS2S) (Vaswani et al., 2017) and one initialized with Bert-
Base (BertS2S) (Devlin et al., 2019).

Dataset | Method | R1 | R2 | RL | FactCC | Ent-DAE
---|---|---|---|---|---|---
CNN/DM | Bart Lewis et al. (2020) | 44.07 | 21.08 | 41.01 | 75.78 | 74.85
Cao et al. (2020) | 42.72 | 20.59 | 39.92 | 49.98 | 74.83
FactEdit | 42.17 | 20.22 | 39.37 | 75.49 | 75.71
|  FactEdit \+ FactCC Filter (FF) | 42.53 | 20.48 | 39.74 | 76.03 | 75.36
XSum | Bart Lewis et al. (2020) | 34.71 | 15.04 | 27.40 | 21.93 | 20.03
Cao et al. (2020) | 33.64 | 14.71 | 26.49 | 7.01 | 20.03
FactEdit | 33.58 | 14.68 | 26.71 | 23.91 | 20.13
|  FactEdit \+ FactCC Filter (FF) | 33.58 | 14.68 | 26.71 | 23.91 | 20.13
Table 1: FactEdit performance for correcting BART outputs (best performance in
bold). FactEdit ourperforms factuality related baselines on FactCC and DAE
scores, while maintaining competitive summarization quality.

###  3.2 Experimental Settings and Evaluation

Setup: We use OpenIE Banko et al. (2007) to pre-process each summary and
extract subject, object, predicate triples for each summary sentence. We use
BART-base Lewis et al. (2020) as our sequence-to-sequence model for the
infilling based data generator and the fact correction model. Both models were
trained with a batch size of 48, a learning rate of 3e-5, and warm-up of 1000
for 1 epoch. The maximum input sequence length was 512 and maximum output
sequence length was 128. Using the infilling data generator, we generate
1233329 negative, 308332 positive examples for CNN/DM and 724304 negative,
181076 positive, examples for XSum as training data for fact correction.
Models were trained on 4 Nvidia GeForce GTX TITAN X GPUs and each training run
took $\sim$15 hours. All hyperparameters were chosen based on generated dev
set ROUGE-L Lin (2004) on each dataset.
Evaluation Setup: Evaluating factual consistency of generated summaries is
challenging, with relatively recent metrics developed to detect it. These
metrics unfortunately do not correlate highly with human judgements yet. We
therefore evaluate our model using two metrics - FactCC Kryscinski et al.
(2020) and Ent-DAE Goyal and Durrett (2021); each captures different error
types. FactCC is a binary classifier, trained on a synthetic, heuristic error
dataset, which is better at detecting simple semantic errors like incorrect
entities or numbers. Ent-DAE is a classifier trained on synthetic data
constructed using the dependency structure of the text. In addition to
semantic errors, it is better at detecting more complex discourse-level errors
Pagnoni et al. (2021). We also report ROUGE Lin (2004) to evaluate if our
model maintains the fluency of summaries. While ROUGE is less correlated with
factuality Pagnoni et al. (2021); Maynez et al. (2020), it helps evaluate if
the corrected summary is fluent and aligned with the reference summary.
However, with factual corrections of outputs we expect small drops in ROUGE,
since generation models were specifically optimized to maximize ROUGE
presumably at the expense of factuality.

Our evaluation has two settings: i) FactEdit \- correct all generated
summaries in the test set and ii) FactEdit \+ FactCC Filter (FF) - using the
FactCC metric we identify factually incorrect summaries, and only correct the
incorrect ones.

Baselines: We compare our approach with Cao et al. (2020) as the baseline. The
baseline uses a heuristic set of rules proposed by Kryscinski et al. (2020) to
introduce simple errors (Entity, Number, Date, and Pronoun) in reference
summaries and trains a BART-base model for error correction. Comparing our
model with Cao et al. (2020) helps us evaluate the benefit of our Infilling LM
based adversarial data generator. 555While Dong et al. (2020) is also a
factual error correction method, we were unable to reproduce it as no public
code was available.

##  4 Results

###  4.1 Factuality Results

We first evaluate FactEdit’s ability to correct errors in summaries generated
by a BART-base summarization model on the entire test set. We first generate
summaries using a BART-base model finetuned on each dataset and then provide
the generated summaries and their corresponding source documents as inputs to
FactEdit for correction.

Method | RL | FactCC | Ent-DAE
---|---|---|---
CNN/DM
Bart | 41.53 | 46.29 | 72.57
FactEdit | 37.73 | 42.29 | 78.86
FactEdit (FF) | 37.73 | 53.14 | 81.71
BertSum | 38.74 | 58.86 | 82.29
FactEdit | 35.6 | 55.43 | 79.43
FactEdit (FF) | 35.6 | 61.71 | 82.86
BUS | 38.59 | 49.71 | 70.28
FactEdit | 33.79 | 48.00 | 76.00
FactEdit (FF) | 33.79 | 56.57 | 80.00
PointGen | 35.62 | 80.57 | 93.14
FactEdit | 32.54 | 75.43 | 90.29
FactEdit (FF) | 32.54 | 78.29 | 90.86
Seq2Seq | 27.15 | 19.43 | 29.71
FactEdit | 24.78 | 23.43 | 48.00
FactEdit (FF) | 24.78 | 24.00 | 54.29
XSum
BertS2S | 29.05 | 22.29 | 05.71
FactEdit | 28.93 | 50.43 | 40.00
FactEdit (FF) | 28.95 | 50.43 | 40.00
TConvS2S | 25.69 | 17.71 | 04.00
FactEdit | 25.64 | 47.16 | 29.14
FactEdit (FF) | 25.64 | 47.16 | 29.14
PointGen | 23.12 | 18.29 | 00.57
FactEdit | 23.02 | 43.75 | 32.00
FactEdit (FF) | 23.04 | 43.75 | 32.00
TranS2S | 23.93 | 18.86 | 2.86
FactEdit | 23.86 | 31.73 | 36.00
FactEdit (FF) | 23.86 | 31.73 | 36.00
Table 2: Performance of FactEdit across different model generated summaries in
the FRANK setting. Best performance is indicated in Bold. FactEdit model
vastly improves factuality across multiple models on both FactCC and DAE
scores.

Table 1 shows results for this experiment. Our results show that correcting
factual errors using our model improves the factuality results. The baseline
model performs poorly with the FactCC metric showing lower scores than the
BART model generated summaries, especially in the more abstractive XSum
setting. The DAE metric for the baseline model is slightly lower than the BART
model scores in the CNN/DM setting and has no improvement in the XSum setting
showing that it does not perform corrections on complex errors. These results
confirm our hypothesis that the baseline model trained on adversarial data
based on heuristic errors does not transfer well to real errors in model
generated summaries. In contrast, our model improves both metrics across both
datasets. On the more challenging XSum dataset, our model has a $\sim$17 point
improvement on FactCC and $\sim$0.1 improvement on DAE over the baseline
model. The BART generated summaries on CNN/DM are $\sim$70% factual and hence
using the FactCC Filter to correct only non-factual summaries helps improve
results on FactCC. As XSum has more than 80% non-factual summaries, the FactCC
filter does not change results and correcting all generated summaries is
beneficial. In Table 6 we present examples of corrections made by FactEdit and
present a discussion in §A.

Prior works have shown that improving factual consistency in summaries leads
to a drop in ROUGE scores Maynez et al. (2020); Cao and Wang (2021); Cao et
al. (2020). Our ROUGE results do not drop significantly and are consistent
with prior work. These results show that our model does not significantly
change the summaries and the corrected summaries contain the relevant
information w.r.t. to the source.

Figure 3: Performance of FactEdit across different error categories in
comparison to baseline Cao et al. (2020)

. FactEdit improves the percentage of factual summaries across diverse types
of factual errors.

###  4.2 Factuality Results across Model Types

Table 2 shows results of using FactEdit to correct summaries generated by
different types of language models using the FRANK benchmark Pagnoni et al.
(2021). We provide the generated summaries collected in the benchmark along
with their source document as input to our trained fact corrector. This
setting evaluates the generalizability of our adversarial training data in
handling different error distributions from different summarization models.
Our results show our model significantly improves the factuality in generated
summaries across 8 out of 9 test models. The FactCC Filter helps improves
results in CNN/DM setting but does not change results in XSum similar to
results in §4.1. In the more extractive CNN/DM setting, fact correction
improves FactCC scores by $\sim$5.3 points and DAE scores by $\sim$10.9 points
on average across models. In the more challenging and abstractive XSum
dataset, we improve FactCC scores by $\sim$24 points and DAE scores by
$\sim$31 points on average. Our results show that our model trained using
Infilling LM based adversarial data is able to generalize and correct errors
in generated summaries across different model types. Further, the significant
improvement in XSum suggests that using LMs to generate factually incorrect
candidates produces rich negative examples which help correct errors in more
abstractive summaries.

Pretrained models like BART, BertSum and BertS2S have improved generation
capabilities and make lesser mistakes in generating the right entity or
predicate and more mistakes in discourse structuring Pagnoni et al. (2021).
FactEdit correspondingly shows larger improvements in DAE scores than FactCC
scores in these pretrained models. The Pointer-Generator model being highly
extractive in nature scores highly in factuality metrics in the CNN/DM setting
and FactEdit reduces results in this setting showing that our model is not
beneficial in copy-based model settings. On the other hand, in the XSum
setting, the base Pointer-Generator model scores poorly and correcting factual
errors in them improves factuality scores. Non-pretrained sequence-to-sequence
models like Seq2Seq and TransSeq2Seq score poorly in both ROUGE and Factuality
scores due to their limited language generation capabilities. By correcting
factual errors in them, we improve factuality metrics significantly without
changes in ROUGE, indicating that the gains are due to fact correction and not
just rewriting the summary using a strong language model.

##  5 Analysis

###  5.1 Performance across Error Categories

The FRANK benchmark proposes a typology of three coarse categories of error
types and collects human annotations on the error category: i) Semantic Frame
Errors - This category covers factual errors in a sentence due to incorrect
entity or predicate being generated ii) Discourse Errors - This covers
discourse level factual errors like incorrect pronouns or sentence ordering
iii) Content Verifiability Errors - This category is for errors whose
factuality cannot be judged either due to grammatical errors or hallucinated
content. We evaluate our model on its ability to correct different types of
errors. We use the generated summaries from the best pretrained model in FRANK
for each dataset - BART for CNN/DM and BertS2S for XSum. For each subset of
summaries of a particular error type, we correct the summaries using FactEdit
and report the percentage of factual summaries in the output as predicted by
Ent-DAE. We compare FactEdit with the baseline to study whether our model
improves error correction for each type.

From Figure 3, we see that across both datasets FactEdit increases the
percentage of factual summaries across all three error categories, showing
that the data generation process in FactEdit can generalize across multiple
error types without the need for special heuristics or annotations. We see the
largest improvements in the Semantic Frame Error category with an increase of
$\sim$ 8 points on CNN/DM and $\sim$ 13 points on XSum. On the more complex
Discourse Errors we see an improvement of $\sim$ 5 points on both datasets.
Finally, on Content Verifiability Errors, we see a $\sim$ 8 point improvement
on CNN/DM and $\sim$ 2 point improvement on XSum. XSum has a high proportion
of hallucination errors and our results highlight the challenge in correcting
this error type.

###  5.2 Transferrability across Datasets

Method | FactCC | Ent-DAE
---|---|---
BertS2S | 22.29 | 05.71
FactEdit (FF) - CNN Model | 33.71 | 22.29
TConvS2S | 17.71 | 04.00
FactEdit (FF) - CNN Model | 30.29 | 22.29
PointGen | 18.29 | 00.57
FactEdit (FF) - CNN Model | 28.57 | 19.43
TranS2S | 18.86 | 2.86
FactEdit (FF) - CNN Model | 18.86 | 21.14
Table 3: Transfer results of FactEdit. FactEdit trained using CNN/DM data
transfers well to summaries generated for documents in XSum.

It is not always feasible to train specialized fact correction models for each
dataset or style of summaries. While CNN/DM and XSum contain documents of the
news domain, they both have different summary characteristics. Certain
applications might benefit from a single model which can generalize to
different summary styles. We evaluate the ability of FactEdit trained on
CNN/DM data (FactEdit FF - CNN Model) to transfer and correct summaries
generated for XSum documents using FRANK benchmark. Table 3 shows results for
this experiment. Our results show significant improvement in factuality scores
across all model types in this setting, showing that our data generation
process produces rich and diverse factually incorrect examples which can
generalize to factual errors in other data settings. By using only the source
documents, our training data is agnostic of the styles, lengths and
characteristics of reference summaries and hence is able to generalize to the
headline style abstractive summaries of XSum.

Method | Fluency | Factuality
---|---|---
Cao et al. (2020) | 4.58 | 3.10
FactEdit | 4.75 | 3.33
Table 4: Results of Human Evaluation on Fluency and Factuality of corrected
summaries. Human judges rate summaries corrected by FactEdit higher in fluency
and factuality than the baseline.

###  5.3 Human Evaluation

To further study whether the factuality corrections performed by our model
align with human expectations of automated summaries, we conduct a human
study. Two annotators evaluated 20 randomly sampled summaries generated from
the test set of the XSum dataset using the BertS2S model and corrected by
FactEdit and the baseline. The annotators were shown the entire source
document and one corrected summary at a time and asked to rate the fluency and
factuality of the summary on a 1-5 Likert scale. In manual evaluation,
annotators rated FactEdit an average of 3.3 on factuality and 4.8 on fluency,
compared to the baseline which was rated 3.1 and 4.6 scores respectively,
showing that FactEdit improves on both factuality and fluency.

###  5.4 Ablation Study

Our model corrects each sentence in a summary given context of the rest of the
summary and relevant passages in the source document. We ablate this setup by
removing parts of the context one at a time. In Table 5 we present the
results. We observe a a drop in results when using the entire summary as
context (-RelevPass) and when removing the context of the summary in which the
sentence occurs (-SummCtxt). Our results show the importance of having the
appropriate context to enable the model to perform fact correction well.

Method | FactCC | E-DAE
---|---|---
CNN/DM
FactEdit | 76.03 | 75.36
FactEdit -SummCtxt | 75.73 | 74.23
FactEdit -SummCtxt-RelevPass | 75.89 | 75.03
Xsum
FactEdit | 23.91 | 20.13
FactEdit -SummCtxt | 22.89 | 20.06
FactEdit -SummCtxt-RelevPass | 23.48 | 20.08
Table 5: Results of Ablation study with components of fact correction pipeline
removed. SummCtxt includes the generated summary as additional context.
RelevPass includes relevant passages from the source as additional context.
FactEdit setup ourperforms the ablated versions on FactCC and DAE scores.

##  6 Related Work

Factuality Evaluation Standard n-gram based metrics do not correlate well with
human judgements of factuality and are unsuitable for evaluating factuality
Kryscinski et al. (2019); Fabbri et al. (2020). Several automated metrics were
proposed to detect factual errors in generated summaries. They primarily fall
in two paradigms—Entailment based and QA based metrics. Goodrich et al.
(2019); Kryscinski et al. (2020); Maynez et al. (2020); Goyal and Durrett
(2021) model factuality as an entailment verifying whether the summary is
entailed by the source. Lee et al. (2022b) use similar masked infilling to
generate training data for such metrics. QA models can be used to answer
questions about the document, separately using the article and the output
summary as context and compare the answers to score the factuality of
summaries Durmus et al. (2020); Wang et al. (2020). To evaluate these metrics,
recent work collec human judgements for factuality Fabbri et al. (2020);
Maynez et al. (2020); Pagnoni et al. (2021). Additionally, (Pagnoni et al.,
2021) also obtain annotations on factual error categories, which we use for
our evaluations. This paper considers the problem of improving factuality, not
measuring it. While this is a different task, it is related: e.g., measuring
the number of corrections made by FactEdit might be useful as a factuality
measure.

Improving Factuality of Summaries: There are two paradigms of work to ensure
generated summaries are factually consistent: i) imposing factuality
constraints during training or generation and ii) post-editing generated
summaries to correct factual errors. Wan and Bansal (2022) add factuality
contraints during pretraining by using factually consistent summaries. Model
designs and factuality specific objectives help optimize for factuality during
training Gabriel et al. (2019); Cao and Wang (2021); Dong et al. (2022);
Rajagopal et al. (2022). During decoding beam search candidates can be ranked
based on factuality measures King et al. (2022); Zhao et al. (2020). Work on
correcting factual errors post generation is relatively nascent. Cao et al.
(2020) and Lee et al. (2022a) train fact correction models on synthetic data
based on heuristic errors which we show is less effective than LM based error
generation (Table 1). Dong et al. (2020) use a QA model to replace phrases in
the summary with spans in the source text. This requires multiple inference
iterations, making them very expensive for correction. In contrast our
approach corrects errors in one iteration, making it a faster and more
practical approach for error correction. Tangentially, work on correcting
errors in reference summaries to make the training data more reliable has also
been explored Adams et al. (2022); Wan and Bansal (2022). In dialog
generation, Gupta et al. (2021) explore using mask-fill approaches to generate
synthetic data for response ranking, showing that using language models to
generate adversarial data might be applicable beyond summarization.

## Conclusion

We present an adversarial data generation process to generate rich synthetic
data for a post editing model, which can be applied to correct factual errors
generated summaries. Our data generation process leverages Infilling Language
Models to produce alternative candidate summaries. Using the generated data,
we train models to rewrite summaries by correcting factual errors in them.
Through extensive experiments across two datasets and nine models, we show
that our fact corrector model improves the factual consistency of the
summaries, making them more reliable.

## Limitations

Our model is trained to rewrite generated summaries by correcting factual
errors in them. A limitation in our current setup is accurate detection of
factual errors. We rely on off-the-shelf metrics for identifying summaries
with factual errors to correct. Our model does not perform detection and
correction together and often rewrites correct summaries as well if fed to the
model. Therefore for settings like CNN/DM, it’s beneficial to filter summaries
using a factuality metric before giving summaries to our model as input. As
our fact corrector is a sequence-to-sequence model, it could potentially
introduce new factual errors in the summaries. It is essential to use
factually detectors to ensure summaries are factual before real world usage of
any corrected summary.

## Ethical Considerations

State-of-the-art language generation models, including summarization, are not
yet powerful enough to facilitate fine-grained control over generated content.
This leads to problems with content fidelity and safety; our work aims to
ameliorate issues related to factual reliability of the models. However,
existing approaches, including ours, cannot guarantee this yet. Furthermore,
there is a risk of dual use, since the same techniques can be used to post-
edit models to produce non-factual, harmful content to mislead, impersonate,
or manipulate opinions. Future research should focus on developing better
defenses methods against mis-using language generators maliciously.

## Acknowledgements

We would like to thank Lucille Njoo, Xiaochuang Han, Sachin Kumar, Dheeraj
Rajagopal and other members of the Tsvetshop Lab for their valuable feedback
on this work. This material is based upon work supported by the DARPA CMO
under Contract No. HR001120C0124. Any opinions, findings and conclusions or
recommendations expressed in this material are those of the author(s) and do
not necessarily state or reflect those of the United States Government or any
agency thereof. Y.T. also gratefully acknowledges support from NSF CAREER
Grant No. IIS2142739 and Alfred P. Sloan Foundation Fellowship.

## References

  * Adams et al. (2022) Griffin Adams, Han-Chin Shing, Qing Sun, Christopher Winestock, Kathleen McKeown, and Noémie Elhadad. 2022.  Learning to revise references for faithful summarization.  _arXiv preprint arXiv:2204.10290_.
  * Banko et al. (2007) Michele Banko, Michael J. Cafarella, Stephen Soderland, Matt Broadhead, and Oren Etzioni. 2007.  Open information extraction from the web.  In _Proceedings of the 20th International Joint Conference on Artifical Intelligence_ , IJCAI’07, pages 2670–2676, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
  * Cao et al. (2020) Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020.  Factual error correction for abstractive summarization models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6251–6258, Online. Association for Computational Linguistics.
  * Cao and Wang (2021) Shuyang Cao and Lu Wang. 2021.  Cliff: Contrastive learning for improving faithfulness and factuality in abstractive summarization.  _arXiv preprint arXiv:2109.09209_.
  * Cao et al. (2018) Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018.  Faithful to the original: Fact aware neural abstractive summarization.  In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_ , pages 4784–4791. AAAI Press.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Donahue et al. (2020) Chris Donahue, Mina Lee, and Percy Liang. 2020.  Enabling language models to fill in the blanks.  _arXiv preprint arXiv:2005.05339_.
  * Dong et al. (2020) Yue Dong, Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung, and Jingjing Liu. 2020.  Multi-fact correction in abstractive text summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9320–9331, Online. Association for Computational Linguistics.
  * Dong et al. (2022) Yue Dong, John Wieting, and Pat Verga. 2022.  Faithful to the document or to the world? mitigating hallucinations via entity-linked knowledge in abstractive summarization.  _arXiv preprint arXiv:2204.13761_.
  * Dreyer et al. (2021) Markus Dreyer, Mengwen Liu, Feng Nan, Sandeep Atluri, and Sujith Ravi. 2021.  Analyzing the abstractiveness-factuality tradeoff with nonlinear abstractiveness constraints.  _arXiv preprint arXiv:2108.02859_.
  * Durmus et al. (2020) Esin Durmus, He He, and Mona Diab. 2020.  FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5055–5070, Online. Association for Computational Linguistics.
  * Fabbri et al. (2020) Alexander R Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2020.  Summeval: Re-evaluating summarization evaluation.  _arXiv preprint arXiv:2007.12626_.
  * Gabriel et al. (2019) Saadia Gabriel, Antoine Bosselut, Jeff Da, Ari Holtzman, Jan Buys, Kyle Lo, Asli Celikyilmaz, and Yejin Choi. 2019.  Discourse understanding and factual consistency in abstractive summarization.  _arXiv preprint arXiv:1907.01272_.
  * Gehrmann et al. (2018) Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018.  Bottom-up abstractive summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4098–4109, Brussels, Belgium. Association for Computational Linguistics.
  * Goodrich et al. (2019) Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh. 2019.  Assessing the factual accuracy of generated text.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019_, pages 166–175. ACM.
  * Goyal and Durrett (2020) Tanya Goyal and Greg Durrett. 2020.  Evaluating factuality in generation with dependency-level entailment.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3592–3603, Online. Association for Computational Linguistics.
  * Goyal and Durrett (2021) Tanya Goyal and Greg Durrett. 2021.  Annotating and modeling fine-grained factuality in summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1449–1462, Online. Association for Computational Linguistics.
  * Gupta et al. (2021) Prakhar Gupta, Yulia Tsvetkov, and Jeffrey P. Bigham. 2021.  Synthesizing adversarial negative responses for robust response ranking and evaluation.  In _FINDINGS_.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  In _Proceedings of the Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , pages 1693–1701.
  * Huang et al. (2021) Yichong Huang, Xiachong Feng, Xiaocheng Feng, and Bing Qin. 2021.  The factual inconsistency problem in abstractive text summarization: A survey.  _arXiv preprint arXiv:2104.14839_.
  * Hutson et al. (2021) Matthew Hutson et al. 2021.  Robo-writers: the rise and risks of language-generating ai.  _Nature_ , 591(7848):22–25.
  * King et al. (2022) Daniel King, Zejiang Shen, Nishant Subramani, Daniel S Weld, Iz Beltagy, and Doug Downey. 2022.  Don’t say what you don’t know: Improving the consistency of abstractive summarization by constraining beam search.  _arXiv preprint arXiv:2203.08436_.
  * Kryscinski et al. (2019) Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019.  Neural text summarization: A critical evaluation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 540–551, Hong Kong, China. Association for Computational Linguistics.
  * Kryscinski et al. (2020) Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020.  Evaluating the factual consistency of abstractive text summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9332–9346, Online. Association for Computational Linguistics.
  * Lee et al. (2022a) Hwanhee Lee, Cheoneum Park, Seunghyun Yoon, Trung Bui, Franck Dernoncourt, Juae Kim, and Kyomin Jung. 2022a.  Factual error correction for abstractive summaries using entity retrieval.  _arXiv preprint arXiv:2204.08263_.
  * Lee et al. (2022b) Hwanhee Lee, Kang Min Yoo, Joonsuk Park, Hwaran Lee, and Kyomin Jung. 2022b.  Masked summarization to generate factually inconsistent summaries for improved factual consistency checking.  _arXiv preprint arXiv:2205.02035_.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Text summarization with pretrained encoders.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3730–3740, Hong Kong, China. Association for Computational Linguistics.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1906–1919, Online. Association for Computational Linguistics.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.
  * Pagnoni et al. (2021) Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021.  Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4812–4829, Online. Association for Computational Linguistics.
  * Rajagopal et al. (2022) Dheeraj Rajagopal, Siamak Shakeri, Cicero Nogueira dos Santos, Eduard Hovy, and Chung-Ching Chang. 2022.  Counterfactual data augmentation improves factuality of abstractive summarization.  _arXiv preprint arXiv:2205.12416_.
  * Ranade et al. (2021) Priyanka Ranade, Anupam Joshi, and Tim Finin. 2021.  Study shows ai-generated fake reports fool experts.
  * Rush et al. (2015) Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015.  A neural attention model for abstractive sentence summarization.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 379–389, Lisbon, Portugal. Association for Computational Linguistics.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017.  Get to the point: Summarization with pointer-generator networks.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Wan and Bansal (2022) David Wan and Mohit Bansal. 2022.  Factpegasus: Factuality-aware pre-training and fine-tuning for abstractive summarization.  _arXiv preprint arXiv:2205.07830_.
  * Wang et al. (2020) Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020.  Asking and answering questions to evaluate the factual consistency of summaries.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5008–5020, Online. Association for Computational Linguistics.
  * Zhang et al. (2020) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.  In _International Conference on Machine Learning_ , pages 11328–11339. PMLR.
  * Zhao et al. (2020) Zheng Zhao, Shay B Cohen, and Bonnie Webber. 2020.  Reducing quantity hallucinations in abstractive summarization.  _arXiv preprint arXiv:2009.13312_.

##  Appendix A Qualitative Analysis

Source |  | Generated Summary
---
| Corrected Summary
---
| An endangered bottlenose dolphin has died off the coast of Spain
---
It had lost its mother and become stranded in shallow waters
off the coast of Mojacar in Andalucia. Tourists then began touching
and photographing the animal rather than seeking help for it.
Equinac, a local NGO, said "selfishness" caused the dolphin
"suffering and stress." The organisation, which works to protect
marine wildlife in the area, wrote in a Facebook post (in Spanish):
"The animal was subjected to the curious who wanted to photograph
and touch it." "These animals are highly protected; to disturb them, to
harm them, to manipulate them and to harass them is prohibited
by law, and we always ask for respect and consideration."
| An endangered bottlenose dolphin has died
---
off the coast of Spain after being harassed by tourists
who wanted to take pictures of it, an animal welfare
group has said in a Facebook post, in part because
of their "selfish" behaviour.774,000 people have
visited the animal since it was stranded.
| An endangered bottlenose dolphin has died
---
off the coast of Spain after being harassed by tourists
who wanted to take pictures of it, an animal welfare
group has said in a Facebook post, in part because
of their "selfish" behaviour.
| BBC Wales Sport understands the Gills made an enquiry for Byrne
---
, before Edinburgh changed his mind over a player he brought
to Rodney Parade. Edinburgh opted to end the pursuit out of respect
for his old club, who he managed from 2011-2015, so as not to weaken
their battle against relegation. They are currently 20th in League Two,
seven points above the relegation zone. Meanwhile, Newport
have confirmed the release of defender Matt Taylor. The 34-year
old joined from Cheltenham in the summer
but made just five appearances for the south Wales club.
"Further to recent discussions, we can confirm that Matt Taylor
will leave the Club with immediate effect having mutually
agreed to terminate his contract," a statement from the club read.
| Newport County manager Justin Edinburgh has ended
---
his interest in signing Gillingham midfielder Luke Byrne,
BBC Wales Sport has learned, after talks between the two
clubs ended in stalemate on Tuesday
night, reports BBC Radio Wales’ Mark McGhee
and BBC Radio Newport’s Steve Davies.
| Newport County manager Justin Edinburgh has ended
---
his interest in signing Gillingham midfielder Luke Byrne,
BBC Wales Sport has learned, after talks between the two
clubs ended in stalemate on Tuesday night,
reports BBC Wales Sport.

Table 6: Examples of corrections made by our fact corrector.

In Table 6 we present examples of corrections made by our models. In the first
example, the generated summary contains an entire sentence which is
unsupported by the source document. FactEdit chooses to remove the entire
sentence instead of rewriting or correcting it. In the second example, the
generated summary contains hallucinated entities of reporter names which are
not present in the source. Here, the FactEdit rewrites by replacing the
incorrect entity with the correct ones.


0 Research  Motion data from sparse inputs are extended with 3D Cartesian
positions for leveraging the pose ambiguity in order to reconstruct the full
pose of the self-avatar. However, depending on the 3D tracking solution,
desynchronization between VR devices motion signals and 3D Cartesian
coordinates as well as motion artifacts such as noise and occlusions can
arise. With this setup, we can measure the impact of each issue individually
in the full-body pose reconstruction.

# Self-Avatar Animation in Virtual Reality: Impact of Motion Signals Artifacts
on the Full-Body Pose Reconstruction

Antoine Maiorca
ISIA Lab e-mail<EMAIL_ADDRESS>UMONS  Belgium  Seyed Abolfazl
Ghasemzadeh
ICTEAM/ELEN e-mail<EMAIL_ADDRESS>UCLouvain  Louvain-la-Neuve
Belgium  Thierry Ravet
ISIA Lab e-mail<EMAIL_ADDRESS>UMONS  Mons  Belgium  François
Cresson
ISIA Lab e-mail<EMAIL_ADDRESS>UMONS  Mons  Belgium  Thierry
Dutoit
ISIA Lab e-mail<EMAIL_ADDRESS>University of Mons  Belgium
Christophe De Vleeschouwer
ICTEAM/ELEN, UCLouvain, Louvain-la-Neuve, Belgium e-mail:
<EMAIL_ADDRESS>
###### Abstract

Virtual Reality (VR) applications have revolutionized user experiences by
immersing individuals in interactive 3D environments. These environments find
applications in numerous fields, including healthcare, education, or
architecture. A significant aspect of VR is the inclusion of self-avatars,
representing users within the virtual world, which enhances interaction and
embodiment. However, generating lifelike full-body self-avatar animations
remains challenging, particularly in consumer-grade VR systems, where lower-
body tracking is often absent. One method to tackle this problem is by
providing an external source of motion information that includes lower body
information such as full Cartesian positions estimated from RGB(D) cameras.
Nevertheless, the limitations of these systems are multiples: the
desynchronization between the two motion sources and occlusions are examples
of significant issues that hinder the implementations of such systems. In this
paper, we aim to measure the impact on the reconstruction of the articulated
self-avatar’s full-body pose of (1) the latency between the VR motion features
and estimated positions, (2) the data acquisition rate, (3) occlusions, and
(4) the inaccuracy of the position estimation algorithm. In addition, we
analyze the motion reconstruction errors using ground truth and 3D Cartesian
coordinates estimated from YOLOv8 pose estimation. These analyzes show that
the studied methods are significantly sensitive to any degradation tested,
especially regarding the velocity reconstruction error.

Computing methodologiesComputer graphicsAnimation; Computing
methodologiesComputer graphicsAnimationMotion capture

##  1 Introduction

Virtual Reality (VR) applications create a 3D environment that immerses users
and enables them to interact with virtual elements. The versatility and
interactivity of VR technology have led to a wide array of practical uses
spanning various industries and sectors. Besides gaming and entertainment, VR
finds valuable applications in healthcare [36, 4], education and training [18,
42, 10], as well as architecture and construction engineering [12, 3].

Several studies have highlighted the significance of including a self-avatar,
representing the user’s body within the virtual environment. These
investigations have demonstrated its favorable effects across multiple
domains, including user interaction and embodiment [34], cognitive processes
of participants [44], and collaborative tasks within shared virtual spaces
[33]. Furthermore, the lifelike movements of the avatar contribute to
fostering more realistic and engaging social interactions among users within
immersive environments [39]. Hence, one crucial task in the design of VR
applications is the animation of the articulated full-body self-avatar.

Typically, VR systems designed for consumers consist of a Head-Mounted Display
(HMD) along with optional handheld controllers. The HMD is a wearable device
worn on the head and positioned in front of the user’s eyes to present a
visual display. These devices incorporate tracking technology to determine
their position and orientation. Using this set of motion data (referred as
sparse inputs in this document), the system generates the user’s complete body
movements. However, tracking for the lower body is not included, making it a
complex task to synthesize the motion of this body subset. Indeed,
synthesizing the full-body motion exclusively from the sparse inputs
corresponds to a one-to-many problem since several poses can resolve the
motion generation task from one input configuration.

Deep Learning-based animation models have been proposed to tackle this pose
ambiguity by learning the complex relationship between the sparse inputs and
the lower-body motion [20, 59, 11, 41]. One significant advantage of these
methods is their independence from additional tracking devices, beyond the HMD
and controllers, to reconstruct the complete self-avatar’s body pose. This
feature makes them suitable for consumer-grade applications but the precision
in the pose reconstruction can be affected by the lack of available
information.

Another approach to mitigate the issue of pose ambiguity involves
incorporating external sources of motion data, particularly for the lower-body
pose. These additional motion features, that often integrating lower-body
limbs information, aim to guide the full-body pose reconstruction [16, 24, 53,
56]. However, these methods are associated with several drawbacks. Firstly,
the need for external devices may hinder the widespread adoption of such
technologies. Additionally, equipping users with these sensors can potentially
affect comfort and detrimentally impact the overall user experience. To avoid
this effect, the full-body Cartesian position can be estimated from RGB videos
[7, 22] and act as the additional motion information.

In this configuration, the latency between VR motion signals and the Cartesian
position is a crucial factor that needs to be taken into account in the
implementation of a VR animation system. Hence, the process of integrating,
into a unified framework, motion capture data from diverse sources, each
operating at its distinct framerate, can be a challenging task. Indeed, each
sources needs to be synchronized to avoid discrepancies and ensure the
coherent representation of the user’s movements. This synchronization is
essential because misalignments or desynchronization among the data streams
can lead to inaccuracies in the reconstructed body pose, which can
significantly impact the overall quality and realism of the animation.
Moreover, depending of the additional tracking solution and the number of
users to track, it can significantly increase the latency between the two
motion sources and have implications for the real-time responsiveness of the
animation system, which is a crucial factor in this context.

Finally, the pose estimation from RGB(D) videos suffers from a low-fidelity
pose reconstruction compared to physical motion capture devices such as
optical markers. Low accuracy on the joint position and artifacts such as
occlusion i.e., a hidden information resulting in a missing joint in the
estimated pose, are major concerns for animating self-avatar’s based on this
set of motion features.

The goal of this paper is to analyze the impact in the articulated self-
avatar’s full-body pose reconstruction of:

  * •

the latency between the Cartesian position estimated from RGB videos and the
VR motion signals

  * •

the discrepancy between the motion acquisition rate of these two sources of
motion

  * •

the motion artifacts that can occur in the estimation of the full-body
Cartesian coordinates

To do so, we propose to manually degrades the 3D Cartesian positions with the
artifacts described as above. The animation model is then fed by the sparse
inputs concatenated with these 3D positions. Finally, we assess the
reconstruction error across the tested configurations. Moreover, we suggest
extracting the 3D Cartesian coordinates using the pose estimation algorithm
embedded in YOLOv8 [22]. This approach enables us to discern the disparities
in model performance when utilizing either ground truth or 3D Cartesian
positions estimated in a real-world use case.

##  2 Related Work

Approaches addressing the animation of a full-body self-avatar in VR can be
broadly categorized into two main groups: those relying solely on sparse
inputs and those augmenting VR motion signals with extra motion data. These
approaches are respectively explained in Section 2.1 and 2.2.

###  2.1 Self-avatar’s full-body estimation from sparse VR sensors

Tracking a user’s full-body motion based on sparse sensors is a widely
explored topic [24, 23, 57, 13]. In the VR paradigm, the full motion tracking
is performed using the motion signals from the VR devices. The self-avatar’s
full-body motion is synthesized from the hands and head motion features.
Solutions based on Inverse Kinematics (IK) have been implemented to leverage
this problem [26, 19, 46, 8].

Then, machine learning techniques have further been used to improve the
quality of the full-body motion reconstruction. In the case of CoolMoves [1],
it leverages k-NN techniques to find a pose that closely corresponds to the
sparse inputs within a well-structured motion database. Similarly, MMVR [37]
adopts motion matching [6] as an alternative approach to achieve real-time
animation with smooth transitions. The major drawback of these approaches is
the the fact that we must ensure that the motion database gathers realistic
samples that include not only seamless transitions and smooth blending between
distinct motions but also a diverse array of desired actions. Moreover,
animating upper body gestures using motion matching is challenging due to the
unconstrained nature of user’s arm movements. This complexity necessitates the
creation of an extensive and challenging-to-manage motion database to
encompass the multitude of feasible poses.

With the emergence of Deep Learning, methods based on artificial neural
networks have been designed to capture the complex spatial-temporal
dependencies between the sparse signals and the full body pose, especially
regarding the low body information where no tracked data are available. More
specifically, algorithms built upon architectures designed for time series
analysis have been proposed to compute the self-avatar’s full pose based on
the sparse inputs [20, 54, 59, 41]. While LSTMs have been successfully
employed in this context [54], Transformer-based methods outperformed this
solution regarding the quality of the pose reconstruction. AvatarPoser [20]
has integrated the encoder part of the transformer architecture [50] to
compute high-dimensional embeddings from the sparse information to estimate
the avatar’s full pose and its global displacement. Dual Attention Poser [59]
have proposed to decouple the global and local motion features to feed two
transformers and further merge the computed information.

An alternative approach to address this challenge involves generative models.
VAE-HMD [11] trained a Variational AutoEncoder (VAE) to generate a latent
vector from a sequence of sparse inputs. This latent vector is then used as
input to the decoder, which is responsible for reconstructing the complete
body pose. More recently, methods based on the combination of transformers and
generative algorithms have been introduced: In the same philosophy to VAE-HMD,
full motion prior is used as a part of a pretraining process and a
transformer-based encoder is then trained to predict the same motion latent as
the full motion encoder using sparse inputs [41]. Additionally, FLAG [2]
employs the technology of Normalizing Flow [38] that allows to compute exact
pose likelihoods in contrast with VAEs and improved the quality of the full
pose reconstruction in comparison with VAE-HMD outputs. BoDiffusion [9]
employs a generative diffusion model [43] for motion synthesis to tackle this
under-constrained reconstruction problem.

Finally, the full pose reconstruction from sparse inputs problem has been
leveraged involving physical-simulation of the self-avatar’s body [52, 27].
These methods are built upon Reinforcement Learning for a physically plausible
full-body pose reconstruction.

Figure 1: Reconstruction of the full-body position based on YOLOv8 algorithm.
(a) and (b) Illustration of YOLOv8 detection applied to one frame of AMASS
dataset rendered from two different perspectives. (c) the 3D Cartesian
reconstruction pose computed by triangulation (red) compared with ground truth
frame (blue)

###  2.2 Multimodal full-body pose estimation

In order to resolve the pose ambiguity, external motion sources have been
deployed to extend the information tracked by the HMD and the handheld
controllers.

Inspired by the success of 2D pose estimation from monocular RGB videos,
HybridTrack [55] extends VR motion data with the 2D Cartesian coordinates. By
combining an uncalibrated RGB camera for the lower body with inside-out
tracking for the upper body, it offers cost-effective and user-friendly
tracking capabilities.

Moreover, Liao et al. in [28] explore the fusion of sparse Inertial
Measurement Units (IMUs) with a single RGB camera to achieve robust 3D human
body reconstruction without the need for invasive multi-IMU setups or 2D joint
detection. Using adaptive regression learning, a dual-stream network extracts
features from IMUs and images, followed by a residual model-attention network
that effectively fuses these features. The method significantly improves upon
existing approaches by reducing errors in 3D joint positions and enhancing
robustness, particularly in scenarios with occlusions or challenging
environments.

Next, EgoLocate [56] emphasizes the importance of merging human motion sensing
and environment sensing. They note that while human motion capture and
environment sensing, such as SLAM, are crucial, they have traditionally been
treated as separate domains. The authors introduce their EgoLocate system,
which integrates sparse IMUs and a monocular camera to achieve real-time human
motion capture, localization, and mapping.

Furthermore, Tome et al. in [48] propose a novel neural network architecture
to address challenges such as strong perspective distortions, self-occlusion,
a lack of labeled datasets, and the inherent ambiguity in lifting 2D joint
positions to 3D space, demonstrating significant improvements in performance
compared to existing methods, both in egocentric and front-facing camera
scenarios.

Finally, BodyTrak [29] employs wrist-mounted cameras to capture informative
images of body silhouettes and derives a working hypothesis that these partial
body parts can effectively infer full-body poses. The contributions include
the introduction of the first wrist-mounted device for full-body pose
estimation, a custom deep learning pipeline, user studies, and discussions on
the challenges and opportunities of applying such technology in practical
contexts.

##  3 Problem Formulation

Figure 2: Examples of artifacts on motion signals. Top left: delay between two
motion sources. Top Right: Cartesian position framerate reduced by
$fps_{ratio}=2$. Bottom Left: Gaussian noise applied on positional data.
Bottom Right: Random occlusion i.e., a joint position randomly set to zero.

The generation of the self-avatar’s full-body pose can be formalized as
follow: considering a sequence of the sparse inputs $X_{0,...,T}$ and
additional motion data $X^{F}_{0,...,T}$, a method $\Phi$ is designed to
synthesize the full pose of the articulated self-avatar $Y$, as in Equation 1.

$Y_{T}=\Phi(X_{0,...,T}\oplus X^{F}_{0,...,T})$ (1)

Similarly to AvatarPoser [20], $X_{0,...,T}$ gathers the Cartesian positions,
the orientations, the linear and angular velocities of the head and the hands.
We choose to extend the sparse inputs with the 3D Cartesian positions that can
be estimated with a setup of one or multiple RGB(D) cameras [14] This
configuration has the advantage to not require any physical cumbersome
tracking sensors on the user body which can infringe with its overall
experience.

However, implementing this kind of solution is not a straightforward task
since it deals with several constraints. In this context, the most common
concerns are the lack of synchronization between the two motion sources,
induced by a latency and a discrepancy between the data acquisition rate from
both sources, joints occlusion and the low-fidelity of the Cartesian position
estimation method.

  * •

Latency: Depending on the cameras setup, a significant latency between the
Cartesian positions and the sparse inputs can occur. Indeed, the head and
hands positions and orientations are computed from inertial sensors or using
SLAM-based algorithms [21] regarding the VR devices employed. The delay
introduced by this process is relatively minor, typically ranging from 1 to 2
milliseconds when employing IMU-based head-mounted displays operating at
$1000Hz$ for the rotational data and between $60Hz$ and $100Hz$ for the
positional data. In contrast, the overall position estimation system often
introduces a more substantial delay in the context of the deployed solution.
Moreover, this latency is notably variable over time, primarily owing to frame
loss. For instance, when assessing the ZED 2 stereo camera [45] operating at a
60 FPS configuration, it exhibits a latency range of 30-45ms. This leads to
desynchronization between the head-mounted display and the camera frames.

  * •

Framerate: The motion data acquisition rates are not necessarily equivalents
between the motion sources. The motion features of the HMD and the hands are
extracted at $60Hz$ or $100Hz$ regarding the VR devices used [51]. While some
cameras, such as the ZED 2, can also achieve frame rates of 60 or even 100
FPS, it’s crucial to take into account the computational demands associated
with 3D position estimation[22]. This computational load can become
substantial, contingent on the chosen algorithm and the number of individuals
being tracked within the scene.

  * •

Occlusion: It refers to the situation where a user being tracked is partially
or completely hidden from the cameras used for tracking. This obstruction can
occur when one user passes in front of another, or when a part of the user’s
body blocks the view of one of its limbs. Occlusion poses a challenge for
motion tracking systems because it can lose sight of one or multiple joints
during the occluded period, leading to gaps or inaccuracies in the tracking
pose. In our configuration, since we consider the additional motion
information $X^{F}$ as the 3D Cartesian positions estimated from cameras, our
setup is subject to occlusion artifacts. This artifact gains prominence,
particularly in scenarios involving multiple users interacting with each
other, thereby increasing the likelihood of occluded joints.

  * •

Low accuracy: Since the markerless motion tracking algorithm based on computer
vision are often less accurate and reliable than optical markers, the low
fidelity on the 3D Cartesian coordinates estimation can have a negative impact
on the full-body pose reconstruction.

##  4 Approach

As in AvatarPoser [20], we rely on 3 subsets of the large-scale motion capture
AMASS Dataset [31]: BMLrub [49], CMU [25] and HDM05 [32]. AMASS Dataset
unifies optical-based motion capture datasets into a standard kinematic tree
and use SMPL approach [30] to provide realistic 3D human meshes represented by
a rigged body model. These subsets gather around 5200 motion samples for a
duration of more than 20 hours. The standardized kinematic tree is structured
into 22 joints.

In order to measure the impact of each artifacts individually, we propose to
mimic the artifacts described in Section 3. Examples of these artifacts on
motion signals are shown in Figure 2.

  * •

We introduce a delay $d$ between the full-body Cartesian positions and the
sparse motion signals. In this configuration, Equation 1 becomes

$Y_{T+d}=\Phi(X_{d,...,T+d}\oplus X^{F}_{0,...,T})$ (2)

  * •

The framerate discrepancy between the two motion signals is tackled by
quantifying $X^{F}_{0,...,T}$ using a framerate ratio $fps_{ratio}$ between
the two motion sources. The framerate of $X^{F}_{0,...,T}$, initially
equivalent to $X_{0,...,T}$, is divided by $fps_{ratio}$.

  * •

Inspired by the implementation of occlusion in [15], we define a probability
$\sigma_{o}$ of the joint being occluded for a given frame.

$\displaystyle O^{F}_{0,...,T}\sim Bernoulli(\sigma_{o})$ (3) $\displaystyle
X^{F}_{o_{0,...,T}}=X^{F}_{0,...,T}*O^{F}_{0,...,T}$ (4) $\displaystyle
Y_{o_{T}}=\Phi(X_{0,...,T}\oplus X^{F}_{o_{0,...,T}})$ (5)

  * •

To mimic the low accuracy of the full-body position estimation, we add noise
sampled from a zero-mean Gaussian distribution with a standard deviation
$\sigma$. Increasing the standard deviation $\sigma$ results in an augmented
level of noise intensity within the Cartesian positions.

$\displaystyle N_{0,...,T}\sim\mathcal{N}(0,\sigma^{2})$ (6) $\displaystyle
X^{F}_{n_{0,...,T}}=X^{F}_{0,...,T}+N_{0,...,T}$ (7) $\displaystyle
Y_{n_{T}}=\Phi(X_{0,...,T}\oplus X^{F}_{n_{0,...,T}})$ (8)

However, this distribution may not accurately represent the actual noise in
the markerless pose estimation system. To build a set of noisy poses that
reflects the lack of accuracy of the data we could acquire in a real-case
scenario, we use a solution based on the YOLOv8 algorithm [22]. As shown in
Figure 1, using the SMPL+H model [40] linked to AMASS dataset, we define two
virtual cameras to render the character as mesh from two different viewpoints.
The rendering resolution in pixels is 640X480. The processing of each rendered
image with Yolo v8 detector (yolov8x-pose model) provides a pose as a set of
characteristic points. We apply the triangulation function from the OpenCV
library [5] to the characteristic points in the two perspectives, using the
parameters of the virtual cameras as explained in [14]. The result is a
3-dimensional reconstruction of the pose computed for each frame in the
dataset.

##  5 Experiments

Table 1: Evaluation of the sensitivity to common artifacts in the context of the self-avatar’s animation from sparse inputs and 3D Cartesian coordinates. The highlighted results refer to configurations that outperform AvatarPoser with sparse inputs. We observe that AvatarPoser overall gives the best results when it comes to deal with clean motion data. However, as a matter of example, when the noise or occlusion intensity increases, HybridTrack leads to lower reconstruction error. |  Body subset | AvatarPoser | HybridTrack
---|---|---|---
|  |  MPJPE (cm) $\downarrow$ |  MPJRE (°) $\downarrow$ |  MPJVE (cm/s) $\downarrow$ |  MPJPE (cm)$\downarrow$ |  MPJRE (°)$\downarrow$ |  MPJVE (cm/s)$\downarrow$
Sparse inputs |  Up |  1.65 |  5.64 |  12.86 |  - |  - |  -
Sparse inputs |  Low |  6.79 |  6.4 |  44.35 |  - |  - |  -
GT 3D Cart. Pos. |  Up |  0.72 |  2.52 |  8.1 |  2.38 |  6.51 |  20.72
GT 3D Cart. Pos. |  Low |  1.41 |  2.03 |  14.19 |  4.97 |  5.55 |  40.81
Noise $\sigma(cm)$ |  |  |  |  |  |  |
1 |  Up |  1.09 |  3.89 |  60.78 |  2.39 |  6.58 |  35.39
1 |  Low |  1.96 |  3.37 |  107.91 |  5.02 |  5.61 |  69.86
2 |  Up |  1.74 |  6.33 |  121.98 |  2.46 |  6.78 |  55.94
2 |  Low |  3.04 |  5.65 |  215.17 |  5.18 |  5.77 |  111.11
5 |  Up |  4.18 |  15.04 |  326.92 |  2.92 |  8.13 |  118.89
5 |  Low |  7.2 |  13.35 |  565.61 |  6.25 |  6.83 |  240.02
Occlusion $\sigma_{o}$ |  |  |  |  |  |  |
0.01 |  Up |  3.05 |  8.18 |  272.48 |  4.05 |  10.16 |  231.65
0.01 |  Low |  5.87 |  5.26 |  525.83 |  7.87 |  7.59 |  414.22
0.05 |  Up |  10.36 |  23.47 |  912.65 |  8.95 |  19.14 |  525.12
0.05 |  Low |  19.68 |  13.73 |  1756.09 |  15.82 |  13.13 |  992.5
Framerate ratio $fps_{ratio}$ |  |  |  |  |  |  |
2 |  Up |  0.87 |  2.73 |  23.98 |  2.43 |  6.54 |  21.96
2 |  Low |  1.81 |  2.41 |  57.02 |  5.19 |  5.71 |  45.29
3 |  Up |  1.05 |  3.03 |  30.91 |  2.49 |  6.6 |  27.71
3 |  Low |  2.22 |  2.81 |  74.18 |  5.44 |  5.89 |  68.37
4 |  Up |  1.22 |  3.35 |  34.47 |  2.55 |  6.66 |  25.88
4 |  Low |  2.64 |  3.21 |  82.65 |  5.72 |  6.1 |  57.54
Delay $d$ (frames) |  |  |  |  |  |  |
2 |  Up |  1.44 |  3.78 |  14.36 |  2.62 |  6.73 |  21.77
2 |  Low |  3.02 |  7.23 |  24.57 |  5.98 |  6.31 |  47.5
4 |  Up |  2.24 |  5.05 |  19.8 |  2.95 |  7.12 |  23.28
4 |  Low |  4.83 |  5.42 |  42.69 |  7.26 |  7.31 |  56.27
6 |  Up |  3.02 |  7.23 |  24.57 |  3.34 |  7.63 |  25.22
6 |  Low |  6.5 |  7.01 |  54.15 |  8.66 |  8.36 |  65.51

For our experiments, we train the animation model $\Phi$ with
$X_{0,...,T}\oplus X^{F}_{0,...,T}$ as input where $X^{F}_{0,...,T}$ is a
sequence of the ground truth 3D Cartesian positions and then measure the
impact of each artifacts independently. We rely on AvatarPoser [20] and an
adapted version of HybridTrack [55], two state-of-the-art deep learning-based
models for the animation of the self-avatar’s full-body. AvatarPoser [20] is a
model built upon the Transformer architecture that encodes the sparse inputs
into a high-level complex motion representation to estimate the full-body
local orientations as well as the global displacement of the root. Providing
3D Cartesian positions into this model helps to resolve the pose ambiguity
that can arise due to the lack of lower body motion information. HybridTrack
[55] employs a CNN-1D based architecture inspired by the method in [35], that
is fed by the sparse information and a single-view of 2D Cartesian coordinates
to generate the full body pose. In our experiments, we adapt this model to our
requirements by providing the 3D Cartesian positions instead.

We add a latency between the Cartesian positions and the sparse inputs of
$d=2$, $4$ or $6$ frames following the delays discussed in Section 3. Then,
since the framerate is sensitive to the computational load induced by several
factors such as the number of cameras or the number of tracked users, we
divided the Cartesian coordinates framerate by a factor $fps_{ratio}=2$, $3$
or $4$. In this last case, if the motion data acquisition rate from VR devices
reaches $100Hz$, the cameras achieve a framerate of $25Hz$. Regarding the
spatial-temporal noise, we set up $\sigma_{o}=0.01$ and $0.05$ of occlusion
probability ($\sigma_{o}=0.1$ in [15]) and $\sigma=1cm$, $2cm$ and $5cm$ which
are reasonable noise levels considering the error reconstruction in multi-
users 3D pose estimation tracking from monocular videos [58] ($60mm$ on
Human36M dataset [17]).

Finally, we test our implementations in a real-case scenario: from AMASS
Dataset, we use the pose estimation from YOLOv8 to gathers the 3D Cartesian
positions, instead of using the ground truth Cartesian positions to train and
to evaluate a specific animation model.

We consider a temporal window of $40$ previous frames and the current frame
($T=41$ frames) feeding the models. Both of them are trained during 10k epochs
on a NVIDIA GTX1080 GPU, following the training procedure and hyperparameters
in AvatarPoser[20] 111https://github.com/eth-siplab/AvatarPoser.git.

##  6 Results

Figure 3: Reconstruction errors regarding the models trained with ground truth
and Cartesian positions from YOLOv8.

Table 1 shows the results of the evaluation of the two animation models to the
considered motion signals artifacts. We rely on 3 metrics called the Mean Per
Joint Positional/Rotational/Velocity Error, respectively referred as MPJPE,
MPJRE and MPJVE. It indicates the deviation between the avatar’s full pose
estimated by the model and the ground truth pose. We also refer in the
aforementioned Table the results on the full pose reconstruction from
$X_{0,...,T}$ for AvatarPoser but not for HybridTrack since it has not been
designed to tackle the problematic of the full pose reconstruction from only
the sparse inputs.

We observe that the overall behavior is that the errors increase with the
intensity of the artifact for both models and regardless the evaluated
artifact. The models trained with the clean and synchronized data from AMASS
dataset does not encompass the issues related to the animation of the self-
avatar’s character based on the sparse inputs and the Cartesian positions
estimated from cameras. Then, when the spatio-temporal motion artifacts, i.e.,
the occlusions and the noise, increases in intensity, HybridTrack provides
more accurate full pose reconstruction than AvatarPoser. In this
configuration, HybridTrack exhibits greater resilience and accuracy than
AvatarPoser, showcasing its enhanced performance in challenging conditions
marked by heightened motion artifacts. However, it is deemed ineffective to
incorporate 3D Cartesian positions in both animation model when confronted
with intense artifacts. Indeed, while HybridTrack effectively mitigates the
impact of these artifacts, AvatarPoser, trained solely with $X_{0,...,T}$,
outperforms HybridTrack across various configurations. We highlighted in Table
1 the configurations where the models fed by $X_{0,...,T}\oplus
X^{F}_{0,...,T}$ exhibits more accurate pose reconstruction than AvatarPoser
trained with sparse inputs. The tested artifacts appear to have a more
pronounced effect on velocity compared to positions and orientations. In
summary, this evaluation highlights the sensitivity of the models to
variations in the training parameters, limiting their deployment in less
controlled environments.

The results, reported in 3, show the comparison between models trained with
ground truth and models trained with Cartesian positions generated with YOLO.
HybridTrack produces less accurate motion reconstruction than AvatarPoser with
solely the sparse inputs regarding the upper body. However, AvatarPoser
trained with the 3D coordinates estimated from YOLOv8 outperforms AvatarPoser
from sparse inputs, except for the velocity of the upper body region.
Comparing these two cases, we can also observe that we do not get any
significant difference in velocity in the lower body region either. So, there
is no clear improvement in terms of noise present in the synthesized movement.

Illustrative pose samples are depicted in Figure 4. The coloration on various
regions of the mesh represents the positional error specific to each region.
The figure serves to highlight that the quality of motion reconstruction
diminishes as noise is introduced to the 3D coordinates. More videos
presenting motion samples can be found here
222https://figshare.com/s/a6f9c9770e4be4919230

Figure 4: Illustration of two pose samples derived from the ground truth data
(Left). In the Mid-Left image, AvatarPoser is trained with ground truth 3D
Cartesian positions and provided with a sequence of this ground truth. The
Mid-Right image displays the effects of introducing Gaussian noise into the 3D
Cartesian coordinates, with noise levels $\sigma=0.01$. Right: AvatarPoser
trained with only sparse inputs. The positional features are improved in
comparison to those produced by solely the sparse inputs, even when the
Cartesian coordinates are degraded with a Gaussian noise with $\sigma=1cm$.

##  7 Limitations and Perspectives

First of all, our analyses emphasize the sensitivity of AvatarPoser and
HybridTrack when trained with ground truth Cartesian positions and sparse
inputs from VR devices. While it has been shown that incorporating Cartesian
positions aids both models in addressing this challenge, the artifacts
examined in this study—such as latency between motion sources, discrepancies
in framerate, low accuracy of pose reconstruction, and occlusions—result in a
degradation of the quality of the self-avatar’s full-body pose reconstruction.

Then, even if we consider these effects in the training process, both of the
models failed to improve the upper-body velocities in comparison to the model
trained only with the sparse inputs. Mitigating this effect will be crucial
for further development, especially when several users share the scene
captured by the cameras. Indeed, in this context, the risk of major artifacts
such as occlusions can explode.

Finally, considering the desynchronization between the two motion signals,
recent methods have been proposed to integrate temporally sparse observations
in Transformer within a medical context [47]. We believe that the integration
of such systems will benefit the field of self-avatar’s animation regarding
the issues discussed in this work.

##  8 Conclusion

In this work, we conducted experiments in the context of the self-avatar’s
full-body pose reconstruction from the head and hands motion features and the
3D Cartesian coordinates. This additional motion data has been chosen so that
it can be extracted from a non-intrusive systems, such as RGB-D cameras, to
guarantee the user experience. First, we discussed about the major concerns
that need to be taken into account in the design of such system. The
desynchronization between the motion signals from the VR devices and the
Cartesian positions, as well as the spatio-temporal motion artifacts such as
noise and occlusions are artifacts that importantly degrades the reliability
of the animation system. We showed that the precision of the pose estimation
system is crucial in this context, especially considering the velocity of the
reconstructed upper body. From the results of these analyzes, we believe that
this work provides valuable insights concerning the task of self-avatar’s
animation based on multimodal data.

###### Acknowledgements.

This research is supported and funded by TRAIL Institute. S.A. Ghasemzadeh is
funded by FRIA/FNRS.

## References

  * [1] K. Ahuja, E. Ofek, M. Gonzalez-Franco, C. Holz, and A. D. Wilson.  Coolmoves: User motion accentuation in virtual reality.  Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 5(2):1–23, 2021.
  * [2] S. Aliakbarian, P. Cameron, F. Bogo, A. Fitzgibbon, and T. J. Cashman.  Flag: Flow-based 3d avatar generation from sparse observations, 2022.
  * [3] A. K. Bashabsheh, H. H. Alzoubi, and M. Z. Ali.  The application of virtual reality technology in architectural pedagogy for building constructions.  Alexandria Engineering Journal, 58(2):713–723, 2019.
  * [4] M.-S. Bracq, E. Michinov, and P. Jannin.  Virtual reality simulation in nontechnical skills training for healthcare professionals: a systematic review.  Simulation in Healthcare, 14(3):188–194, 2019.
  * [5] G. Bradski.  The OpenCV Library.  Dr. Dobb’s Journal of Software Tools, 2000.
  * [6] M. Büttner and S. Clavet.  Motion matching-the road to next gen animation.  Proc. of Nucl. ai, 1(2015):2, 2015.
  * [7] Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh.  Openpose: Realtime multi-person 2d pose estimation using part affinity fields.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019\.
  * [8] P. Caserman, P. Achenbach, and S. Gobel.  Analysis of inverse kinematics solutions for full-body reconstruction in virtual reality.  pp. 1–8, 08 2019. doi: 10 . 1109/SeGAH . 2019 . 8882429
  * [9] A. Castillo, M. Escobar, G. Jeanneret, A. Pumarola, P. Arbeláez, A. Thabet, and A. Sanakoyeu.  Bodiffusion: Diffusing sparse observations for full-body human motion synthesis.  arXiv preprint arXiv:2304.11118, 2023.
  * [10] D. Checa and A. Bustillo.  A review of immersive virtual reality serious games to enhance learning and training.  Multimedia Tools and Applications, 79:5501–5527, 2020.
  * [11] A. Dittadi, S. Dziadzio, D. Cosker, B. Lundell, T. J. Cashman, and J. Shotton.  Full-body motion from a single head-mounted device: Generating smpl poses from partial observations.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11687–11697, 2021.
  * [12] T. Dorta, G. Kinayoglu, and M. Hoffmann.  Hyve-3d and the 3d cursor: Architectural co-design with freedom in virtual reality.  International Journal of Architectural Computing, 14(2):87–102, 2016.
  * [13] J. H. Geissinger and A. T. Asbeck.  Motion inference using sparse inertial sensors, self-supervised learning, and a new dataset of unscripted human motion.  Sensors, 20(21), 2020. doi: 10 . 3390/s20216330
  * [14] R. Hartley and A. Zisserman.  Multiple View Geometry in Computer Vision.  Cambridge University Press, New York, NY, USA, 2 ed., 2003.
  * [15] D. Holden.  Robust solving of optical motion capture data by denoising.  ACM Transactions on Graphics (TOG), 37(4):1–12, 2018.
  * [16] Y. Huang, M. Kaufmann, E. Aksan, M. J. Black, O. Hilliges, and G. Pons-Moll.  Deep inertial poser: Learning to reconstruct human pose from sparse inertial measurements in real time.  ACM Transactions on Graphics (TOG), 37(6):1–15, 2018.
  * [17] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu.  Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7):1325–1339, jul 2014.
  * [18] L. Jensen and F. Konradsen.  A review of the use of virtual reality head-mounted displays in education and training.  Education and Information Technologies, 23:1515–1529, 2018.
  * [19] F. Jiang, X. Yang, and L. Feng.  Real-time full-body motion reconstruction and recognition for off-the-shelf vr devices.  In Proceedings of the 15th ACM SIGGRAPH Conference on Virtual-Reality Continuum and Its Applications in Industry-Volume 1, pp. 309–318, 2016.
  * [20] J. Jiang, P. Streli, H. Qiu, A. Fender, L. Laich, P. Snape, and C. Holz.  Avatarposer: Articulated full-body pose tracking from sparse motion sensing.  In Proceedings of European Conference on Computer Vision. Springer, 2022.
  * [21] L. Jinyu, Y. Bangbang, C. Danpeng, W. Nan, Z. Guofeng, and B. Hujun.  Survey and evaluation of monocular visual-inertial slam algorithms for augmented reality.  Virtual Reality & Intelligent Hardware, 1(4):386–410, 2019.
  * [22] G. Jocher, A. Chaurasia, and J. Qiu.  YOLO by Ultralytics, Jan. 2023.
  * [23] J. Kim, Y. Seol, and J. Lee.  Realtime performance animation using sparse 3d motion sensors.  In Motion in Games: 5th International Conference, MIG 2012, Rennes, France, November 15-17, 2012. Proceedings 5, pp. 31–42. Springer, 2012\.
  * [24] M. Kim and S. Lee.  Fusion poser: 3d human pose estimation using sparse imus and head trackers in real time.  Sensors, 22(13), 2022. doi: 10 . 3390/s22134846
  * [25] C. G. Lab.  Cmu graphics lab motion capture database., 2000.
  * [26] P. Lang.  Inverse kinematics in dead and buried, 2016.
  * [27] S. Lee, S. Starke, Y. Ye, J. Won, and A. Winkler.  Questenvsim: Environment-aware simulated motion tracking from sparse sensors.  arXiv preprint arXiv:2306.05666, 2023.
  * [28] X. Liao, J. Zhuang, Z. Liu, J. Dong, K. Song, and J. Xiao.  Reconstructing 3d human pose and shape from a single image and sparse imus.  PeerJ Comput Sci., 2023.
  * [29] H. Lim, Y. Li, M. Dressa, F. Hu, J. H. Kim, R. Zhang, and C. Zhang.  Bodytrak: Inferring full-body poses from body silhouettes using a miniature camera on a wristband.  Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 6(3), sep 2022. doi: 10 . 1145/3552312
  * [30] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black.  Smpl: A skinned multi-person linear model.  ACM transactions on graphics (TOG), 34(6):1–16, 2015.
  * [31] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black.  AMASS: Archive of motion capture as surface shapes.  In International Conference on Computer Vision, pp. 5442–5451, Oct. 2019.
  * [32] M. Müller, T. Röder, M. Clausen, B. Eberhardt, B. Krüger, and A. Weber.  Documentation mocap database hdm05.  Computer Graphics Technical Report CG-2007-2, Universität Bonn, 2007.
  * [33] Y. Pan and A. Steed.  The impact of self-avatars on trust and collaboration in shared virtual environments.  PLOS ONE, 12(12):1–20, 12 2017. doi: 10 . 1371/journal . pone . 0189078
  * [34] Y. Pan and A. Steed.  How foot tracking matters: The impact of an animated self-avatar on interaction, embodiment and presence in shared virtual environments.  Frontiers in Robotics and AI, 6, 2019. doi: 10 . 3389/frobt . 2019 . 00104
  * [35] D. Pavllo, C. Feichtenhofer, D. Grangier, and M. Auli.  3d human pose estimation in video with temporal convolutions and semi-supervised training.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7753–7762, 2019.
  * [36] A. S. Pillai and P. S. Mathew.  Impact of virtual reality in healthcare: a review.  Virtual and augmented reality in mental health treatment, pp. 17–31, 2019.
  * [37] J. L. Ponton, H. Yun, C. Andujar, and N. Pelechano.  Combining motion matching and orientation prediction to animate avatars for consumer-grade vr devices.  In Computer Graphics Forum, vol. 41, pp. 107–118. Wiley Online Library, 2022.
  * [38] D. Rezende and S. Mohamed.  Variational inference with normalizing flows.  In International conference on machine learning, pp. 1530–1538. PMLR, 2015.
  * [39] S. L. Rogers, R. Broadbent, J. Brown, A. Fraser, and C. P. Speelman.  Realistic motion avatars are the future for social interaction in virtual reality.  Frontiers in Virtual Reality, 2, 2022. doi: 10 . 3389/frvir . 2021 . 750729
  * [40] J. Romero, D. Tzionas, and M. J. Black.  Embodied hands: Modeling and capturing hands and bodies together.  ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6), Nov. 2017.
  * [41] M. Shin, D. Lee, and I.-K. Lee.  Utilizing task-generic motion prior to recover full-body motion from very sparse signals.  arXiv preprint arXiv:2308.15839, 2023.
  * [42] P. Smutny, M. Babiuch, and P. Foltynek.  A review of the virtual reality applications in education and training.  In 2019 20th International Carpathian Control Conference (ICCC), pp. 1–4. IEEE, 2019.
  * [43] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In International conference on machine learning, pp. 2256–2265. PMLR, 2015.
  * [44] A. Steed, Y. Pan, F. Zisch, and W. Steptoe.  The impact of a self-avatar on cognitive load in immersive virtual reality.  In 2016 IEEE Virtual Reality (VR), pp. 67–76, 2016. doi: 10 . 1109/VR . 2016 . 7504689
  * [45] Stereolabs.  Zed 2.  https://www.stereolabs.com/zed-2/.
  * [46] Z. Tan, Y. Hu, and K. Xu.  Virtual reality based immersive telepresence system for remote conversation and collaboration.  In Next Generation Computer Animation Techniques: Third International Workshop, AniNex 2017, Bournemouth, UK, June 22-23, 2017, Revised Selected Papers 3, pp. 234–247. Springer, 2017.
  * [47] S. Tipirneni and C. K. Reddy.  Self-supervised transformer for sparse and irregularly sampled multivariate clinical time-series.  ACM Transactions on Knowledge Discovery from Data (TKDD), 16(6):1–17, 2022.
  * [48] D. Tome, T. Alldieck, P. Peluse, G. Pons-Moll, L. Agapito, H. Badino, and F. de la Torre.  Selfpose: 3d egocentric pose estimation from a headset mounted camera.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(06):6794–6806, jun 2023. doi: 10 . 1109/TPAMI . 2020 . 3029700
  * [49] N. F. Troje.  Decomposing biological motion: A framework for analysis and synthesis of human gait patterns.  Journal of vision, 2(5):2–2, 2002.
  * [50] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [51] VR-Compare.  Vr hardware comparison.  https://vr-compare.com/compare?h1=pDTZ02PkT&h2=mLbW9G7f4&h3=0jLuwg808-j.
  * [52] A. Winkler, J. Won, and Y. Ye.  Questsim: Human motion tracking from sparse sensors with simulated avatars.  In SIGGRAPH Asia 2022 Conference Papers, pp. 1–8, 2022.
  * [53] Y. Wu, Y. Wang, S. Jung, S. Hoermann, and R. W. Lindeman.  Towards an articulated avatar in vr: Improving body and hand tracking using only depth cameras.  Entertainment Computing, 31:100303, 2019. doi: 10 . 1016/j . entcom . 2019 . 100303
  * [54] D. Yang, D. Kim, and S.-H. Lee.  Lobstr: Real-time lower-body pose prediction from sparse upper-body tracking signals.  In Computer Graphics Forum, vol. 40, pp. 265–275. Wiley Online Library, 2021.
  * [55] J. Yang, T. Chen, F. Qin, M. S. Lam, and J. A. Landay.  Hybridtrak: adding full-body tracking to vr using an off-the-shelf webcam.  In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1–13, 2022.
  * [56] X. Yi, Y. Zhou, M. Habermann, V. Golyanik, S. Pan, C. Theobalt, and F. Xu.  Egolocate: Real-time motion capture, localization, and mapping with sparse body-mounted sensors.  arXiv preprint arXiv:2305.01599, 2023.
  * [57] X. Yi, Y. Zhou, M. Habermann, S. Shimada, V. Golyanik, C. Theobalt, and F. Xu.  Physical inertial poser (pip): Physics-aware real-time human motion tracking from sparse inertial sensors.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13167–13178, June 2022.
  * [58] A. Zanfir, E. Marinoiu, M. Zanfir, A.-I. Popa, and C. Sminchisescu.  Deep network for the integrated 3d sensing of multiple people in natural images.  Advances in neural information processing systems, 31, 2018.
  * [59] X. Zhang, X. Chen, X. Dai, and X. Di.  Dual attention poser: Dual path body tracking based on attention.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2794–2803, 2023.