# Diverse Video Generation from a Single Video

Niv Haim, Ben Feinstein11footnotemark: 1, Niv Granot, Assaf Shocher, Shai
Bagon, Tali Dekel and Michal Irani
Weizmann Institute of Science, Rehovot, Israel Equal contribution

Figure 1:  Given an input video (red), we generate similarly looking videos
(black) capturing both appearance of objects as well as their dynamics. The
diversity of the outputs is both spatially (e.g., number of dancers and their
positions are different from the input video) and temporally (generated
dancers are not synced).
As we present video results, the reader is encouraged to check our
supplementary material

GANs are able to perform generation and manipulation tasks, trained on a
single video. However, these single video GANs require unreasonable amount of
time to train on a single video, rendering them almost impractical. In this
paper we question the necessity of a GAN for generation from a single video,
and introduce a non-parametric baseline for a variety of generation and
manipulation tasks. We revive classical space-time patches-nearest-neighbors
approaches and adapt them to a scalable unconditional generative model,
without any learning. This simple baseline surprisingly outperforms single-
video GANs in visual quality and realism (confirmed by quantitative and
qualitative evaluations), and is disproportionately faster (runtime reduced
from several days to seconds). Our approach is easily scaled to Full-HD
videos. We also use the same framework to demonstrate video analogies and
spatio-temporal retargeting. These observations show that classical approaches
significantly outperform heavy deep learning machinery for these tasks. This
sets a new baseline for single-video generation and manipulation tasks, and no
less important – makes diverse generation from a single video practically
possible for the first time.

##  1 Introduction

Generation and editing of natural videos remain challenging, mainly due to
their large dimensionality and the enormous space of motion they span. Most
modern frameworks train generative models on a large collection of videos,
producing high quality results for only a limited class of videos. These
include extensions of GANs [15] to video data [43, 39, 2, 47, 33, 25] and
video-to-video translation [44, 45, 29, 46, 11, 50, 7], autoregressive
sequence prediction [6, 40, 5, 12, 41, 42, 3, 14] and more. While externally-
trained generative models produce impressive results, they are restricted to
the types of video dynamics in their training set. On the other side of the
spectrum are _single-video GANs_. These video generative models train on a
_single_ input video, learn its distribution of space-time patches, and are
then able to generate a diversity of new videos with the same patch
distribution [17, 4]. However, these take very long time to train for each
input video, making them applicable to only small spatial resolutions and to
very short videos (typically, very few small frames). Furthermore, their
output oftentimes shows poor visual quality and noticeable visual artifacts.
These shortcomings render existing single-video GANs impractical and
unscalable.

Video synthesis and manipulation of a single video sequence based on its
distribution of space-time patches dates back to classical pre-deep learning
methods. These classical methods demonstrated impressive results for various
applications, such as video retargeting [36, 32, 49, 21], video completion
[48, 28, 19], video texture synthesis [22, 23, 24, 10, 13, 20] and more. With
the rise of deep-learning, these methods gradually, perhaps unjustifiably,
became less popular. Recently, Granot et al. [16] revived classical patch-
based approaches for image synthesis, and was shown to significantly
outperform _single-image_ GANs in both run-time and visual quality.

In light of the above-mentioned deficiencies of single-video GANs, and
inspired by [16], we propose a fast and practical method for video generation
from a single video that we term VGPNN (_Video Generative Patch Nearest
Neighbors_). In order to handle the huge amounts of space-time patches in a
single video sequence, we use the classical fast approximate nearest neighbor
search method PatchMatch by Barnes et al. [8]. By adding stochastic noise to
the process, our approach can generate a large diversity of random different
video outputs from a single input video in an unconditional manner.

Like single-video GANs, our approach enables the diverse and random generation
of videos. However, in contrast to existing single-video GANs, we can generate
_high resolution_ videos, while reducing runtime by many orders of magnitude,
thus making diverse unconditional video generation from a single video
realistically possible for the first time.

In addition to diverse generation from a single video, by employing robust
optical-flow based descriptors we use our framework to transfer the dynamics
and motions between two videos with different appearance (which we call “video
analogies”). We also show the applicability of our framework to spatio-
temporal video retargeting and to conditional video inpainting.

To summarize, our contributions are as follows:

  * •

We show that our space-time patch nearest-neighbors approach, despite its
simplicity, outperforms single-video GANs by a large margin, both in runtime
and in quality.

  * •

Our approach is the first to generate diverse high resolution videos (spatial
or temporal) from a single video.

  * •

We demonstrate the applicability of our framework to other applications: video
analogies, sketch-to-video, spatio-temporal video retargeting and conditional
video inpainting.

Our code and data will be released.

##  2 Method

Figure 2: VGPNN Architecture: Given a single input video $x_{0}$, a spatio-
temporal pyramid is constructed and an output video $y_{0}$ is generated
coarse-to-fine. At each scale, VPNN module (Fig. 3) is applied to transfer an
initial guess $Q_{n}$ to the output $y_{n}$ which shares the same space-time
patch distribution as the input $x_{n}$. At the coarsest scale, noise is
injected to induce randomness.

Our main task is to generate diverse video samples based on a single natural
input video, such that the generated outputs have similar appearance and
motions as the original input video, but are also visually different from one
another.

In order to capture both spatial and temporal information of a single video,
we start by building a spatio-temporal pyramid and operate coarse-to-fine to
capture the internal statistics of the input video at multiple scales. At each
scale we employ a Video-Patch-Nearest-Neighbor module (VPNN); VGPNN is in fact
a sequence of VPNN layers. The inputs to each layer depend on the application,
where we first focus on our main application of diverse video generation.

Given an input video $x$, we construct a spatio-temporal pyramid
$\left\\{x_{0}\dots,x_{N}\right\\}$, where $x_{0}=x$, and
$x_{n}=x_{n-1}{\downarrow_{r}}$ is a bicubically downscaled version of
$x_{n-1}$ by factor $r$ (${r=(r_{H},r_{W},r_{T})}$, where $r_{H}=r_{W}$ are
the spatial factors and $r_{T}$ is the temporal factor, which can be
different).

### Multi-scale approach (Fig. 2):

At the coarsest level, the input to the first VPNN layer is an initial coarse
guess of the output video. This is created by adding random Gaussian noise
$z_{N}$ to $x_{N}$. The noise $z_{N}$ promotes high diversity in the generated
output samples from the single input. The global structure (e.g., a head is
above the body) and global motion (e.g., humans walk forward), is prompted by
$x_{N}$, where such structure and motion can be captured by _small space-time_
patches.

Each space-time patch of the initial coarse guess ($x_{N}+z_{N}$) is then
replaced with its nearest neighbor patch from the corresponding coarse input
$x_{N}$. The coarsest-level output $y_{N}$ is generated by choosing at each
space-time position the median of all suggestions from neighboring patches
(known as “voting” or “folding”).

At each subsequent level, the input to the VPNN layer is the upscaled output
of the previous layer (${y_{n+1}\uparrow}^{r}$). Each space-time patch is
replaced with its nearest neighbor patch from the corresponding input $x_{n}$
(using the same patch-size as before, now capturing finer details). This way,
the output $y_{n}$ in each level is similar in structure and in motion to the
initial guess, but contains the same space-time patch statistics of the
corresponding input $x_{n}$. The output $y_{n}$ is generated by median voting
as described above.

To further improve the quality and sharpness of the generated output at each
pyramid level ($y_{n}$), we iterate several times through the current level,
each time using the current output $y_{n}$ as input to the current VPNN layer
(similar to the EM-like approach employed in many patch-based works [e.g., 16,
36, 8, 48]).

Figure 3: VPNN module gets as input RGB videos query, key and value (QKV
respectively). Q and K can be concatenated to additional auxiliary channels.
It outputs an RGB video. (a) Inputs are unfolded to patches (each position now
holds a concatenation of neighboring positions); (b) Each patch in Q finds its
nearest neighbor patch in K. This is achieved by solving the NNF using
PatchMatch [8]; (c) Each patch in Q is replaced with a patch from V, according
to the correspondences found in stage (b); (d) Resulting patches are folded
back to an RGB video output.

### QKV scheme:

In several cases it is necessary to compare patches in another search space
than the original RGB input space. To this end we adopt a QKV scheme (query,
key and value, respectively) as used by [16]. For example, when comparing the
upscaled output of previous layer to the corresponding level from the pyramid
of the original video, the patches of the latter are sharper than the former.
This is mitigated by setting $V=x_{n}$ and $K=x_{n+1}{\uparrow^{r}}$ which has
a similar degree of blur as $Q={y_{n+1}\uparrow}$ where. Each patch $Q_{i}$
with nearest neighbour $K_{j}$ is replaced with $V_{j}$ ($i,j$ are spatio-
temporal positions). The QKV scheme is especially important in our video
analogies application where it is used to include additional temporal
information in the queries and the keys.

### Finding Correspondences:

We use PatchMatch (Barnes et al. [8]) to find the nearest neighbors between
$Q$ and $K$ (Fig. 3b). The algorithm is implemented on GPU using PyTorch [30],
with time complexity $O(n\times d)$ and $O(n)$ additional memory (where $n$ is
the video size and $d$ is the patch size). This dramatically reduces both run
time and memory footprint used for video generation, making it possible to
generate high-resolution videos (see Fig 6).

An overview of VPNN module is shown in Fig. 3.

### Temporal Diversity and Consistency:

To enhance the temporal diversity of our samples we set the temporal dimension
of the output to be slightly smaller than that of the input video. Thus,
motions in different spatial positions in the generated output are taken from
different temporal positions in the input video, increasing the overall
temporal diversity (see for example the generated dancers in Fig. 1 that are
not synced). We also found that the temporal consistency is best preserved in
the generated output when the initial noise $z_{N}$ is randomized for each
spatial position, but is the same (replicated) in the temporal dimension.

##  3 Experimental Results

We compare our results to those of HP-VAE-GAN [17] and SinGAN-GIF [4], both
are methods for diverse video generation from single video. Our results are
both qualitatively (Fig. 4) and quantitatively (Table 1) superior while
reducing the runtime by a factor of $3\times 10^{4}$ (from 8 days training on
one video to 18 seconds for new generated video). While [17, 4] are limited to
generated outputs of small resolution ($144{\times}256$), the use of efficient
PatchMatch algorithm for nearest neighbors search dramatically reduces both
run time and memory footprint used for video generation, making it possible to
generate outputs in the same resolution of the input video (full-HD
$1280{\times}1920$)

Figure 4:  Comparing Visual Quality between our generated frames and those of
HP-VAE-GAN [17] and SinGAN-GIF [4] (please zoom in on the frames). Note that
our generated frames are sharper and also exhibit more coherent and plausible
arrangements of the scene.

Method SVFID Head-on comparison Runtime [17] $\downarrow$ (User study)
[$\%$]$\uparrow$ $\downarrow$ HP-VAE-GAN [17] 0.0081 67.84$\pm$1.77 7.625 days
VGPNN (Ours) 0.0072 18 secs SinGAN-GIF [4] 0.0119 50.57$\pm$ 3.27 Unpublished
VGPNN (Ours) 0.0058 10 secs

Table 1: Quantitative Evaluation: SVFID [17] measures the patch statistics
similarity between the input video and a generated video. It computes the
Fréchet distance between the statistics of the input video and the generated
video using pre-computed C3D [38] features (lower is better). Note that our
generated samples bear more substantial similarity to the input videos. In a
user study, each of 100 AMT participants were asked to judge which of two
generated samples was better in terms of sharpness, natural look and
coherence. We report the percentage of users who favored our samples over the
other.  Figure 5:  Video Analogies: Left: an example of video analogies
between all pairs of three input videos (red). Each generated video (black)
takes the spatio-temporal layout from the input video in its row, and the
appearance and dynamics of the input video from its column. Right: an example
of sketch-to-video – the generated video (bottom) takes its spatio-temporal
layout from the sketch video of morphed MNIST digits (middle) and its
appearance and dynamics from the input video of parading soldiers (top).

##  4 Video analogies

Video to video translation methods typically train on large datasets and are
either conditioned on human poses or keypoint detection [e.g. 44, 45, 29, 46],
or require knowledge of a human/animal model [e.g. 1, 11, 50, 34, 35, 31, 26].
We show that when videos’ dynamics are similar in both their motion and
semantic context within their video, one can use our framework to transfer the
motion and appearance between the two (see Fig. 5). We term this task “video
analogies” (inspired by image analogies [18, 27, 9]). More formally, we
generate a new video whose spatio-temporal layout is taken from a content
video $C$, and overall appearance and dynamics from a style video $S$.

We first extract the “dynamic structure” of both videos – the magnitude of the
optical flow (extracted via RAFT [37]), quantized into few bins (using
k-means). We compute the spatio-temporal pyramids of the (i) style video $S$
(ii) dynamic structure of the content video ($\text{dyn}(C)$) (iii) the
dynamic structure the style video ($\text{dyn}(S)$). The output video is
generated by setting $Q,K,V$ at each level as follows:

$\begin{array}[]{@{}l|lll}\text{Level}&Q&K&V\\\ \hline\cr\text{N
(coarsest)}&\text{dyn}(C)_{N}&\text{dyn}(S)_{N}&S_{N}\\\ \text{n (any
other)}&\text{dyn}(C)_{n}\|Q_{n+1}\uparrow&\text{dyn}(S)_{n}\|S_{n}&S_{n}\end{array}$

where $\|$ denotes concatenation along the channels axis, and $n$ denote the
current level in the pyramid. Note that in the coarsest level, the two videos
are only compared by their dynamic structure. In finer levels, the dynamic
structure of $C$ (the content video) is used to “guide” the output to the
desired spatio-temporal layout. In Fig. 5left we show snapshots of the
analogies of all possible pairs between three videos. In Fig. 5right we show
an example for “sketch-to-video” transfer, where the dynamic structure is
given by a sketch video instead of an actual video.

---
Figure 6: Comparing Generation Runtime between our approach (VGPNN), a naïve
extension of GPNN [16] from 2D to 3D and HP-VAE-GAN [17] on 13-frames videos
with different spatial resolutions (X-axis, all have 16:9 aspect ratio).

##  5 More Applications

In the supplementary material we include examples for spatio-temporal
retargeting and conditional video inpainting, as well as further technical
details and ablations.

## References

  * Aberman et al. [2020] K. Aberman, Y. Weng, D. Lischinski, D. Cohen-Or, and B. Chen.  Unpaired motion style transfer from video to animation.  _ACM Transactions on Graphics (TOG)_ , 39(4):64–1, 2020.
  * Aigner and Körner [2018] S. Aigner and M. Körner.  Futuregan: Anticipating the future frames of video sequences using spatio-temporal 3d convolutions in progressively growing gans.  _arXiv preprint arXiv:1810.01325_ , 2018.
  * Aksan and Hilliges [2019] E. Aksan and O. Hilliges.  Stcn: Stochastic temporal convolutional networks.  _arXiv preprint arXiv:1902.06568_ , 2019.
  * Arora and Lee [2021] R. Arora and Y. J. Lee.  Singan-gif: Learning a generative video model from a single gif.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 1310–1319, 2021.
  * Babaeizadeh et al. [2017] M. Babaeizadeh, C. Finn, D. Erhan, R. H. Campbell, and S. Levine.  Stochastic variational video prediction.  _arXiv preprint arXiv:1710.11252_ , 2017.
  * Ballas et al. [2015] N. Ballas, L. Yao, C. Pal, and A. Courville.  Delving deeper into convolutional networks for learning video representations.  _arXiv preprint arXiv:1511.06432_ , 2015.
  * Bansal et al. [2018] A. Bansal, S. Ma, D. Ramanan, and Y. Sheikh.  Recycle-gan: Unsupervised video retargeting.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 119–135, 2018.
  * Barnes et al. [2009] C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman.  Patchmatch: A randomized correspondence algorithm for structural image editing.  _ACM Trans. Graph._ , 28(3):24, 2009.
  * Benaim et al. [2021] S. Benaim, R. Mokady, A. Bermano, and L. Wolf.  Structural analogy from a single image pair.  In _Computer Graphics Forum_ , volume 40, pages 249–265. Wiley Online Library, 2021.
  * Bhat et al. [2004] K. S. Bhat, S. M. Seitz, J. K. Hodgins, and P. K. Khosla.  Flow-based video synthesis and editing.  In _ACM SIGGRAPH 2004 Papers_ , pages 360–363. 2004.
  * Chan et al. [2019] C. Chan, S. Ginosar, T. Zhou, and A. A. Efros.  Everybody dance now.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5933–5942, 2019.
  * Denton and Fergus [2018] E. Denton and R. Fergus.  Stochastic video generation with a learned prior.  In _International Conference on Machine Learning_ , pages 1174–1183. PMLR, 2018.
  * Fišer et al. [2016] J. Fišer, O. Jamriška, M. Lukáč, E. Shechtman, P. Asente, J. Lu, and D. Sỳkora.  Stylit: illumination-guided example-based stylization of 3d renderings.  _ACM Transactions on Graphics (TOG)_ , 35(4):1–11, 2016.
  * Franceschi et al. [2020] J.-Y. Franceschi, E. Delasalles, M. Chen, S. Lamprier, and P. Gallinari.  Stochastic latent residual video prediction.  In _International Conference on Machine Learning_ , pages 3233–3246. PMLR, 2020.
  * Goodfellow et al. [2014] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial networks.  _arXiv preprint arXiv:1406.2661_ , 2014.
  * Granot et al. [2021] N. Granot, B. Feinstein, A. Shocher, S. Bagon, and M. Irani.  Drop the gan: In defense of patches nearest neighbors as single image generative models.  _arXiv preprint arXiv:2103.15545_ , 2021.
  * Gur et al. [2020] S. Gur, S. Benaim, and L. Wolf.  Hierarchical patch vae-gan: Generating diverse videos from a single sample.  _arXiv preprint arXiv:2006.12226_ , 2020.
  * Hertzmann et al. [2001] A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin.  Image analogies.  In _Proceedings of the 28th annual conference on Computer graphics and interactive techniques_ , pages 327–340, 2001.
  * Huang et al. [2016] J.-B. Huang, S. B. Kang, N. Ahuja, and J. Kopf.  Temporally coherent completion of dynamic video.  _ACM Transactions on Graphics (TOG)_ , 35(6):1–11, 2016.
  * Jamriška et al. [2015] O. Jamriška, J. Fišer, P. Asente, J. Lu, E. Shechtman, and D. Sỳkora.  Lazyfluids: appearance transfer for fluid animations.  _ACM Transactions on Graphics (TOG)_ , 34(4):1–10, 2015.
  * Krähenbühl et al. [2009] P. Krähenbühl, M. Lang, A. Hornung, and M. Gross.  A system for retargeting of streaming video.  In _ACM SIGGRAPH Asia 2009 papers_ , 2009.
  * Kwatra et al. [2003] V. Kwatra, A. Schödl, I. Essa, G. Turk, and A. Bobick.  Graphcut textures: Image and video synthesis using graph cuts.  _Acm transactions on graphics (tog)_ , 22(3):277–286, 2003.
  * Kwatra et al. [2005] V. Kwatra, I. Essa, A. Bobick, and N. Kwatra.  Texture optimization for example-based synthesis.  In _ACM SIGGRAPH 2005 Papers_ , pages 795–802. 2005.
  * Kwatra et al. [2007] V. Kwatra, D. Adalsteinsson, T. Kim, N. Kwatra, M. Carlson, and M. Lin.  Texturing fluids.  _IEEE transactions on visualization and computer graphics_ , 13(5):939–952, 2007.
  * Lee et al. [2018] A. X. Lee, R. Zhang, F. Ebert, P. Abbeel, C. Finn, and S. Levine.  Stochastic adversarial video prediction.  _arXiv preprint arXiv:1804.01523_ , 2018.
  * Lee et al. [2019] J. Lee, D. Ramanan, and R. Girdhar.  Metapix: Few-shot video retargeting.  _arXiv preprint arXiv:1910.04742_ , 2019.
  * Liao et al. [2017] J. Liao, Y. Yao, L. Yuan, G. Hua, and S. B. Kang.  Visual attribute transfer through deep image analogy.  _arXiv preprint arXiv:1705.01088_ , 2017.
  * Liu et al. [2009] M. Liu, S. Chen, J. Liu, and X. Tang.  Video completion via motion guided spatial-temporal global optimization.  In _Proceedings of the 17th ACM international conference on Multimedia_ , pages 537–540, 2009.
  * Mallya et al. [2020] A. Mallya, T.-C. Wang, K. Sapra, and M.-Y. Liu.  World-consistent video-to-video synthesis.  _arXiv preprint arXiv:2007.08509_ , 2020.
  * Paszke et al. [2019] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc., 2019.  URL http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf.
  * Ren et al. [2020] J. Ren, M. Chai, S. Tulyakov, C. Fang, X. Shen, and J. Yang.  Human motion transfer from poses in the wild.  In _European Conference on Computer Vision_ , pages 262–279. Springer, 2020.
  * Rubinstein et al. [2008] M. Rubinstein, A. Shamir, and S. Avidan.  Improved seam carving for video retargeting.  _ACM transactions on graphics (TOG)_ , 27(3):1–9, 2008.
  * Saito et al. [2017] M. Saito, E. Matsumoto, and S. Saito.  Temporal generative adversarial nets with singular value clipping.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2830–2839, 2017.
  * Siarohin et al. [2019a] A. Siarohin, S. Lathuilière, S. Tulyakov, E. Ricci, and N. Sebe.  Animating arbitrary objects via deep motion transfer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2377–2386, 2019a.
  * Siarohin et al. [2019b] A. Siarohin, S. Lathuilière, S. Tulyakov, E. Ricci, and N. Sebe.  First order motion model for image animation.  _Advances in Neural Information Processing Systems_ , 32:7137–7147, 2019b.
  * Simakov et al. [2008] D. Simakov, Y. Caspi, E. Shechtman, and M. Irani.  Summarizing visual data using bidirectional similarity.  In _2008 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1–8. IEEE, 2008.
  * Teed and Deng [2020] Z. Teed and J. Deng.  Raft: Recurrent all-pairs field transforms for optical flow.  In _European conference on computer vision_ , pages 402–419. Springer, 2020.
  * Tran et al. [2015] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri.  Learning spatiotemporal features with 3d convolutional networks.  In _Proceedings of the IEEE international conference on computer vision_ , pages 4489–4497, 2015.
  * Tulyakov et al. [2018] S. Tulyakov, M.-Y. Liu, X. Yang, and J. Kautz.  Mocogan: Decomposing motion and content for video generation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1526–1535, 2018.
  * Villegas et al. [2017] R. Villegas, J. Yang, S. Hong, X. Lin, and H. Lee.  Decomposing motion and content for natural video sequence prediction.  _arXiv preprint arXiv:1706.08033_ , 2017.
  * Villegas et al. [2018] R. Villegas, D. Erhan, H. Lee, et al.  Hierarchical long-term video prediction without supervision.  In _International Conference on Machine Learning_ , pages 6038–6046. PMLR, 2018.
  * Villegas et al. [2019] R. Villegas, A. Pathak, H. Kannan, D. Erhan, Q. V. Le, and H. Lee.  High fidelity video prediction with large stochastic recurrent neural networks.  _arXiv preprint arXiv:1911.01655_ , 2019.
  * Vondrick et al. [2016] C. Vondrick, H. Pirsiavash, and A. Torralba.  Generating videos with scene dynamics.  _arXiv preprint arXiv:1609.02612_ , 2016.
  * Wang et al. [2018] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, G. Liu, A. Tao, J. Kautz, and B. Catanzaro.  Video-to-video synthesis.  _arXiv preprint arXiv:1808.06601_ , 2018.
  * Wang et al. [2019] T.-C. Wang, M.-Y. Liu, A. Tao, G. Liu, J. Kautz, and B. Catanzaro.  Few-shot video-to-video synthesis.  _arXiv preprint arXiv:1910.12713_ , 2019.
  * Wang et al. [2020] Y. Wang, P. Bilinski, F. Bremond, and A. Dantcheva.  Imaginator: Conditional spatio-temporal gan for video generation.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 1160–1169, 2020.
  * Wang et al. [2021] Y. Wang, F. Bremond, and A. Dantcheva.  Inmodegan: Interpretable motion decomposition generative adversarial network for video generation.  _arXiv preprint arXiv:2101.03049_ , 2021.
  * Wexler et al. [2004] Y. Wexler, E. Shechtman, and M. Irani.  Space-time video completion.  In _Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004._ , volume 1, pages I–I. IEEE, 2004.
  * Wolf et al. [2007] L. Wolf, M. Guttmann, and D. Cohen-Or.  Non-homogeneous content-driven video-retargeting.  In _Proceedings of the Eleventh IEEE International Conference on Computer Vision (ICCV)_ , 2007.
  * Yang et al. [2020] Z. Yang, W. Zhu, W. Wu, C. Qian, Q. Zhou, B. Zhou, and C. C. Loy.  Transmomo: Invariance-driven unsupervised video motion retargeting.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5306–5315, 2020.


# AECMOS: A speech quality assessment metric for echo impairment

###### Abstract

Traditionally, the quality of acoustic echo cancellers is evaluated using
intrusive speech quality assessment measures such as ERLE [1] and PESQ [2], or
by carrying out subjective laboratory tests [3, 4]. Unfortunately, the former
are not well correlated with human subjective measures, while the latter are
time and resource consuming to carry out [5]. We provide a new tool for speech
quality assessment for echo impairment which can be used to evaluate the
performance of acoustic echo cancellers. More precisely, we develop a neural
network model to evaluate call quality degradations in two separate
categories: echo and degradations from other sources. We show that our model
is accurate as measured by correlation with human subjective quality ratings.
Our tool can be used effectively to stack rank echo cancellation models.
AECMOS is being made publicly available as an Azure service.

Index Terms—  Speech Quality Assessment, Acoustic Echo Cancellation

##  1 Introduction

Acoustic echo arises when a near end microphone picks up the near end
loudspeaker signal and a far end user hears their own voice. The presence of
acoustic echo is a top complaint in user ratings of audio call quality.

_Acoustic echo cancellers_ (AECs) significantly improve audio call quality by
canceling echo. This is achieved by first comparing the transmitted signal to
the received signal, processing for delay and signal distortions, and then
subtracting to remove the echo component. The goal is to only transmit a clean
near end signal.

Recently, there has been a lot of innovation in AECs. The rise of deep
learning has led to better performing models as compared to their classical
counterparts [6, 7, 8, 9, 10]. Also, hybrid models, combining both classical
and deep learning methods such as using adaptive filters and _recurrent neural
networks_ (RNNs) [11, 12], have shown great results [13, 14, 15]. With the
rapid advancements in the field of echo cancellation, it becomes ever more
critical to be able to objectively measure and compare the performance of
AECs.

We propose a speech quality assessment metric for evaluating echo impairment
that overcomes the drawbacks of conventional methods. Our model, called
AECMOS, directly predicts human subjective ratings for call echo impairment.
It can be used to evaluate the end-to-end performance of AECs and to rank
different AEC methods based on (degradation) _mean opinion score_ (MOS)
estimates with great accuracy.

Our model architecture is a deep neural network comprising of convolutional
layers, GRU (gated recurrent unit) layers, and dense layers. AECMOS is trained
using the ground truth human ratings obtained following guidance from ITU-T
Rec. P.831 [3], ITU-T Rec. P.832 [4] and ITU-T Rec. P.808 [16] as described in
[5].

AECMOS is extremely effective in aiding with the development and research of
AECs. We are providing the AECMOS as an Azure service for other researchers to
use as well. The details of the API are at https://github.com/microsoft/AEC-
Challenge/tree/main/AECMOS. We have already received over $50$ requests for
accessing our service ranging from university researchers to big companies.
For examples of AECMOS (aka DECMOS) use in AEC model development we refer the
reader to [11, 6] which include winners of the INTERSPEECH 2021 Acoustic Echo
Cancellation Challenge [15].

AECMOS does not require a clean speech reference for the near or far end, nor
a quiet environment. A clean reference is typically not available in non-
artificial scenarios, and the test set that we use for evaluating AECMOS can
be much more realistic and representative than artificial scenarios. Our test
set was selected from over 5000 different scenarios, each with different room
acoustics, different devices, and different human speakers. AECMOS can be used
in actual customer calls to monitor the quality of real calls; it is not
restricted to lab or development usage, but has operational utility.

##  2 Related work

Common methods of evaluating AEC models [17, 18, 19, 20] include using
intrusive objective measures such as _echo return loss enhancement_ (ERLE) [1]
and _perceptual evaluation of speech quality_ (PESQ) [2]. ERLE can only be
measured when having access to both the echo and processed echo signals, in a
quiet environment without near end speech. ERLE can be approximately measured
by: $\displaystyle ERLE\approx
10\log_{10}\frac{\mathbb{E}[y^{2}(n)]}{\mathbb{E}[e^{2}(n)]}$ where $y(n)$ is
the microphone recording of the far end signal (with no echo suppression), and
$e(n)$ is the residual echo after cancellation. PESQ requires a clean speech
reference in addition to the degraded speech.

Unfortunately, metrics such as ERLE and PESQ are often not well correlated
with human subjective ratings of call echo degradation quality [5]. This is
especially true in the presence of background noise or double talk [21, 5].
Carrying out laboratory tests following standards such as ITU-T Rec. P.831
[3], while more accurate, is expensive, time consuming, and not a scalable
solution.

There are objective standards to help characterize AEC performance. IEEE 1329
[22] defines metrics like terminal coupling loss for single talk (TCLwst) and
double talk (TCLwdt), which are measured in anechoic chambers. TIA 920 [23]
uses many of these metrics but defines required criteria. ITU-T Rec. G.122
[24] defines AEC stability metrics, and ITU-T Rec. G.131 [25] provides a
useful relationship of acceptable Talker Echo Loudness Rating versus one way
delay time. ITU-T Rec. G.168 [1] provides a comprehensive set of AEC metrics
and criteria. However, it is not clear how to combine these dozens of metrics
to a single metric, or how well these metrics correlate to subjective quality.

Commercially available objective metrics include EQUEST [26] which measures
single talk echo performance. An objective metric for double talk echo
performance is ACOPT 32 [27] which implements the 3GPP standard TS 26.132
[28]. ACOPT 32 is of limited use in real call scenarios as it requires the
same near-end signal to be played twice once as single talk and once as part
of double talk. Furthermore, it is not straightforward to interpret the model
outputs.

##  3 Data

We use supervised learning to train AECMOS. Each example in the dataset
consists of three audio signals: near end microphone signal, far end signal,
and the output from an echo canceller which we call the enhanced signal. The
label for a given example is its (degradation) Mean Opinion Score (MOS)
obtained from crowdsourcing as described in [5].

In generating the data set, we distinguished between single talk and double
talk scenarios. For the near end single talk case, we asked for the overall
quality [16]. For far end single talk, we asked for echo ratings [3]. For
double talk, we asked for ratings for both echo annoyance and other
degradations in two separate questions 111Question 1: How would you judge the
degradation from the echo? Question 2: How would you judge other degradations
(noise, missing audio, distortions, cut-outs)?. All impairments were rated on
the degradation category scale (from 1:Very annoying to 5: Imperceptible). The
ratings were then used to obtain a MOS label for the examples. For near end
single talk, the echo label was set to $5$ and for far end single talk the
degradation label was set to $5$. We found including ratings for double talk
non-echo degradations to improve the correlation between experts and naive
raters [5].

The model was trained on $64,013$ samples ranging in duration from $3$ seconds
to $14.5$ seconds with a mean of $8.2$ seconds, for a total of 145.8 hours of
data. The training data included 17 submitted models with a total of 14K audio
clips from the ICASSP 2021 AEC Challenge [13]. The breakdown of the training
data set by scenario is: $45.6\%$ near end single talk, $26.7\%$ far end
single talk, and $27.7\%$ double talk.

The test set consists of submissions from the INTERSPEECH 2021 AEC Challenge
[29, 15]. There were a total of $14$ contestants submitting their work for
$300$ double talk, $300$ far end single talk, and $200$ near end single talk
examples. In addition, we included the enhanced signals of $4$ of our own deep
models along with $4$ digital signal processing based models. In this manner,
we obtained a total of $17,600$ enhanced speech signals or $55$ hours of data.
The ground truth labels were gathered via crowdsourcing using the tool [5],
where we collected $5$ votes per clip. Figure 1 shows that, in the case of
double talk, a wide variety of echo and other degradation MOS score behaviours
co-occur.

Fig. 1: Test set double talk ground truth labels.

##  4 Model

Advancements in deep learning have shown great potential in various speech
enhancement tasks [9, 20], including for speech quality assessment [30, 31].
Here we propose a deep neural network to replace humans as call quality raters
for speech impairment due to echo and other degradations.

###  4.1 Features

The AECMOS model takes as input three signals: near end microphone signal, far
end signal, and the output from an echo canceller also referred to as the
enhanced signal. The task is to evaluate the quality of the enhanced signal
with regard to echo impairment on a human subjective rating scale. Observe
that the very nature of acoustic echo necessitates a comparison of signals.
Knowledge of both the near end microphone signal and the far end signal is
needed to determine whether the enhanced signal contains echo as opposed to
some background noise, for example. This is in contrast to evaluating noise
suppression quality which can be done successfully without reference signals
[30].

In addition to the three input signals, AECMOS also takes an optional scenario
marker as part of the input. This marker encodes which of the three scenarios
we are in: near end single talk, far end single talk, or double talk. For
online deployment, the marker is not used. For offline AEC model evaluations,
when scenario information is readily available, activating the scenario marker
improves model performance as shown in Table 4. In our training set, each
sample has an associated scenario label that was created at the time of the
generation of the training data. We found it effective to prepend a one-hot
vector of a fixed length to the three model input signals indicating which
signal(s) should be considered active for a particular sample.

We used micro augmentations during training to increase model stability. By
this, we mean imperceptible data augmentations: removing the initial 10 ms of
the near end microphone signal, or changing the energy level by 0.5 dB.

Details about the data set and the crowdsourcing approach for obtaining the
ground truth labels appear in Section 3 and [5, 29, 30].

###  4.2 Architecture

For the model architecture, we explored different configurations of
convolutional models. A significant boost in MOS prediction correlations with
the ground truth labels came from incorporating GRU layers into the model.
Table 1 shows the architecture for the best performing model. In developing
the model, we saw that for all models the double talk scenario posed the
hardest challenge. As shown in Table 4, incorporating GRU layers improved
model performance most significantly for double talk.

The input to the model is a stack of three log power spectrograms obtained
from the near end, far end, and enhanced signals. The spectrograms were
computed with a DFT size of 512 and a hop size of 256 over clips that were
sampled at 16 kHz. Finally, we calculate the logarithm of the power. For an 8
second clip, this results in an input dimension of $541\times 257$. AECMOS
handles variable length inputs natively.

Table 1: AECMOS architecture Layer | Output Dimensions
---|---
Input: $3\times 541\times 257$ |
Conv: 32, $(3\times 3)$, LeakyReLU | $(32,270,128)$
MaxPool: (2 x 2), Dropout(0.4) |
Conv: 64, $(3\times 3)$, LeakyReLU | $(64,135,64)$
MaxPool: (2 x 2), Dropout(0.4) |
Conv: 64, $(3\times 3)$, LeakyReLU | $(64,67,32)$
MaxPool: (2 x 2), Dropout(0.4) |
Conv: 128, $(3\times 3)$, LeakyReLU | $(128,33,16)$
MaxPool: (2 x 2), Dropout(0.4) |
Global MaxPool | (1, 128)
Bidirectional GRU: 128, NumLayers 2 |
HiddenUnits 64, Drouput(0.2) | $(1,128)$
Dense: 64, LeakyReLU Dropout(0.4) | $(1,64)$
Dense: 64, LeakyReLU Dropout(0.4) | $(1,64)$
Dense: 2, 1 + 4* sigmoid | $(1,2)$

The model was trained on $8$ GPUs with a batch size of 10 per GPU. We used the
Adam optimizer [32] and MSE loss function until the loss saturated.

The model outputs two MOS predictions on a scale of $1-5$: one for echo and
one for other MOS. These outputs correspond to the ground truth labels as
described in Section 3.

##  5 Experiments

We evaluate the accuracy of our model by measuring the correlation between the
predictions of our AECMOS and the ground truth human ratings.

More precisely, we used the enhanced signals from a total of 22 models. For
each model, the outputs were produced for 300 double talk, 300 far end single
talk, and 200 near end single talk examples. See Section 3 for more details.
For each audio clip, we obtained the objective MOS rating using our model and
the crowdsourced MOS score.

###  5.1 Results

In order to evaluate our model, we calculate the PCC for the AECMOS
predictions versus the ground truth MOS labels.

For each far end single talk example, we also calculate the ERLE, PESQ, and
EQUEST score.

Fig. 2: far end Single Talk Per Clip: AECMOS

As seen in Table 2, in the far end single talk scenario, AECMOS outperforms
ERLE, PESQ, and EQUEST (evaluated with 150 wideband test conditions).

For the near end single talk scenario, we can compare our AECMOS with DNSMOS
model which was developed for evaluating noise suppression models [30]. AECMOS
has a more difficult task than DNSMOS: evaluate both echo and other
degradations and do so independently of each other. Nonetheless, we believe
that AECMOS has very good potential to improve in the near end single talk
category. For one, DNSMOS was trained on about $120,000$ audio clips [30],
while AECMOS saw only about half that many for training and only a quarter as
many, roughly $30,000$, were near end single talk clips. With this in mind,
AECMOS performance is very promising.

Table 2: Per Clip PCC for AECMOS, and other commonly used metrics: DNSMOS, ERLE, PESQ, EQUEST. |  AEC MOS |  DNS MOS |  ERLE |  PESQ |  EQUEST
---|---|---|---|---|---
ST far end DMOS |  0.847 |  |  0.541 |  0.710 |  0.686
ST near-end MOS |  0.611 |  0.640 |  |  |
DT Echo DMOS |  0.582 |  |  |  |
DT Other DMOS |  0.751 |  |  |  |

Expectedly, the most challenging scenario to evaluate is the double talk
scenario. Here the model needs to evaluate separate qualities, echo and other
degradations, simultaneously yet independently of each other. Measuring and
improving double talk performance is important as not being able to interrupt
others speaking has been shown to impair meeting inclusiveness and
participation rate [33].

For evaluating the stack ranking of different echo cancellers, we compute the
average of ratings across the entire test set for each model. We calculate the
same for AECMOS ratings. Finally, we calculate the Spearman’s Rank Correlation
Coefficient (SRCC) between the two. The results are given in Table 3. We
report an SRCC of $0.969$ and a PCC of $0.996$ in the far end single talk
scenario, which is the most common scenario for echo cancellation. We note
that the best performing submitted models were very close to each other in the
contest. In fact, so much so that the human MOS rankings were within error
bars of each other [15].

Table 3: AECMOS Per Contestant PCC and SRCC: All Scenarios refers to far end single talk and double talk for echo; and near end single talk MOS and double talk Other MOS. |  PCC |  SRCC
---|---|---
All Scenarios Echo DMOS |  0.981 |  0.970
All Scenarios (Other) MOS |  0.902 |  0.954
ST far end Echo DMOS |  0.996 |  0.969
ST near end MOS |  0.923 |  0.831
DT Echo DMOS |  0.898 |  0.863
DT Other DMOS |  0.927 |  0.955

Fig. 3: Per contestant: echo degradation (far end single talk and double talk)
and other MOS (near end single talk and double talk).

##  6 Ablation Study

We experimented with various architectures, features, and training methods. In
this section, we give an overview of the key findings.

We started out with a baseline model consisting of $5$ convolutional layers
followed by $3$ dense layers. The input to all models was a stack of three
features: log power of STFT applied to the near end, far end, and enhanced
signal. All models were trained using micro augmentations as described in
Section 4.

The first improvement in our model’s performance came from including scenario
labels as part of the model input. As we were exploring avenues for model
development, we conducted a small experiment where we incorporated label
information in the loss function and asked the model to predict scenario
labels in addition to the MOS labels. Curiously, the model predicted the far
end single talk scenario $87\%$ of the time while only $52\%$ of the labels
were actually far end single talk with the remaining labels being double talk.
Introducing an optional scenario marker for offline use helped model
performance as described in Table 4. More discussion of input features can be
found in Section 4.1.

The second improvement came about when we were investigating how to improve
double talk performance. Experiments with convolution kernel size led us to
believe that our model was having difficulties incorporating information along
the temporal axis. Introducing GRU layers into the model remedied this issue.
In adding a new layer, we needed to remove a convolutional layer so that we
would not be down sampling too aggressively before reaching the GRU layer.
Table 4 summarizes the aforementioned key improvements.

Table 4: Per Clip Pearson Correlation Coefficients: Baseline Convolutional Model; add scenario markers to model input; remove a convolution layer and add a GRU layer to obtain AECMOS. |  Baseline |  \+ scenario |  \+ GRU
---|---|---|---
All Scenarios Echo DMOS |  0.732 |  0.746 |  0.797
All Scenarios (Other) MOS |  0.735 |  0.775 |  0.802
ST far end Echo DMOS |  0.780 |  0.825 |  0.847
ST near end MOS |  0.434 |  0.534 |  0.611
DT Echo DMOS |  0.458 |  0.422 |  0.582
DT Other DMOS |  0.577 |  0.657 |  0.751

We also experimented with using log power of Mel spectrogram for model input
features. Mel spectrogram corresponds well with human subjective hearing and
has been successfully used in evaluating noise suppression [30]. Our best
model that takes Mel spectrogram features as input uses 160 Mel bins. While we
experimented with different settings, we found a consistent behavior that the
Mel models achieve lower correlation scores for echo and higher scores for
other degradations. This matches our intuition as detecting the presence of
echo is less dependent on human subjective experience than classifying a sound
as noise.

Table 5: Per Clip Pearson Correlation Coefficients: AECMOS; AECMOS trained with Mel spectrogram features. |  AECMOS |  AECMOS Mel
---|---|---
All Scenarios Echo DMOS |  0.797 |  0.742
All Scenarios (Other) MOS |  0.802 |  0.819
ST far end Echo DMOS |  0.847 |  0.739
ST near end MOS |  0.611 |  0.604
DT Echo DMOS |  0.582 |  0.553
DT Other DMOS |  0.751 |  0.772

Finally, we experimented with the self-teaching paradigm [30] and training
with bias-aware [34] loss. Interestingly, neither provided significant
improvements in PCC.

##  7 Improvements due to growing training data.

We conjecture that our model’s performance improves with more training data.
In particular, we have observed that increasing the size of our training data
from 64K audio clips to 108K clips and testing on 5K clips, gives a model that
is just as accurate for inference whether or not it uses scenario label
information. The difference in correlations was roughly $1\%$ across all
scenarios. However, since we used more data for training we could not validate
this claim as thoroughly as we have done for our work in the previous sections
of this paper.

##  8 Conclusions

Our AECMOS model provides a speech quality assessment metric that is accurate,
expedient, and scalable. It can be used to stack rank echo cancellers with
very good accuracy and thereby accelerate research in echo cancellation. In
the future, we would like to further improve the model by exploring additional
data augmentations and learning custom filter banks.

## References

  * [1] “ITU-T recommendation G.168: Digital network echo cancellers,”  Feb 2012.
  * [2] “ITU-T recommendation P.862: Perceptual evaluation of speech quality (PESQ): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs,”  Feb 2001.
  * [3] “ITU-T P.831 Subjective performance evaluation of network echo cancellers ITU-T P-series recommendations,”  1998\.
  * [4] ITU-T Recommendation P.832,  Subjective performance evaluation of hands-free terminals,  International Telecommunication Union, Geneva, 2000.
  * [5] Ross Cutler, Babak Nadari, Markus Loide, Sten Sootla, and Ando Saabas,  “Crowdsourcing approach for subjective evaluation of echo impairment,”  in ICASSP, 2020.
  * [6] Ernst Seidel, Jan Franzen, Maximilian Strake, and Tim Fingscheidt,  “Y2-Net FCRN for acoustic echo and noise suppression,”  in INTERSPEECH, 2021.
  * [7] Lukas Pfeifenberger, Matthias Zoehrer, and Franz Pernkopf,  “Acoustic Echo Cancellation with Cross-Domain Learning,”  in INTERSPEECH, 2021, pp. 4753–4757.
  * [8] A. Fazel, M. El-Khamy, and J. Lee,  “CAD-AEC: Context-aware deep acoustic echo cancellation,”  in ICASSP, 2020, pp. 6919–6923.
  * [9] Jean-Marc Valin, Srikanth Tenneti, Karim Helwani, Umut Isik, and Arvindh Krishnaswamy,  “Low-complexity, real-time joint neural echo control and speech enhancement based on percepnet,”  in ICASSP, 2020.
  * [10] M. M. Halimeh and W. Kellermann,  “Efficient multichannel nonlinear acoustic echo cancellation based on a cooperative strategy,”  in ICASSP, 2020, pp. 461–465.
  * [11] Renhua Peng, Linjuan Cheng, Chengshi Zheng, and Xiaodong Li,  “Acoustic Echo Cancellation Using Deep Complex Neural Network with Nonlinear Magnitude Compression and Phase Information,”  in INTESPEECH, 2021, pp. 4768–4772.
  * [12] Lu Ma, Hua Huang, Pei Zhao, and Tengrong Su,  “Acoustic echo cancellation by combining adaptive digital filter and recurrent neural network,”  arXiv preprint arXiv:2005.09237, 2020.
  * [13] Kusha Sridhar, Ross Cutler, Ando Saabas, Tanel Parnamaa, Markus Loide, Hannes Gamper, Sebastian Braun, Robert Aichner, and Sriram Srinivasan,  “Icassp 2021 acoustic echo cancellation challenge: Datasets, testing framework, and results,”  in ICASSP. IEEE, 2021, pp. 151–155.
  * [14] Shimin Zhang, Yuxiang Kong, Shubo Lv, Yanxin Hu, and Lei Xie,  “F-t-lstm based complex network for joint acoustic echo cancellation and speech enhancement,”  arXiv preprint: 2106.07577, 2021.
  * [15] Ross Cutler, Ando Saabas, Tanel Parnamaa, Markus Loide, Sten Sootla, Marju Purin, Hannes Gamper, Sebastian Braun, Karsten Sorensen, Robert Aichner, et al.,  “INTERSPEECH 2021 acoustic echo cancellation challenge,”  in INTERSPEECH, 2021.
  * [16] “ITU-T P.808 supplement 23 ITU-T coded-speech database supplement 23 to ITU-T P-series recommendations (previously ccitt recommendations),”  1998\.
  * [17] Hao Zhang and DeLiang Wang,  “Deep learning for acoustic echo cancellation in noisy and double-talk scenarios,”  in INTERSPEECH, 2018.
  * [18] Amin Fazel, Mostafa El-Khamy, and Jungwon Lee,  “Deep multitask acoustic echo cancellation,”  in INTERSPEECH, 2019.
  * [19] Alexandre Guerin, Gérard Faucon, and Régine Le Bouquin-Jeannès,  “Nonlinear acoustic echo cancellation based on volterra filters,”  IEEE Transactions on Speech and Audio Processing, vol. 11, no. 6, pp. 672–683, 2003.
  * [20] Joao Felipe Santos and Tiago H Falk,  “Speech dereverberation with context-aware recurrent neural networks,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, no. 7, pp. 1236–1246, 2018.
  * [21] A. R. Avila, H. Gamper, C. Reddy, R. Cutler, I. Tashev, and J. Gehrke,  “Non-intrusive speech quality assessment using neural networks,”  in ICASSP, 2019, pp. 631–635.
  * [22] “IEEE 1329-2010 Standard method for measuring transmission performance of handsfree telephone sets,”  2010\.
  * [23] “TIA-920: Transmission requirements for wideband digital wireline telephones,”  Dec 2002.
  * [24] “ITU-T recommendation G.122: Influence of national systems on stability and talker echo in international connections,”  Feb 2012.
  * [25] ITU-T Recommendation G.131,  Talker echo and its control,  International Telecommunication Union, Geneva, 2003.
  * [26] Head Acoustics,  “EQUEST: Echo Quality Evaluation of Speech in Telelcommunication,”  2012\.
  * [27] Head Acoustics,  “ACOPT 32 (Code 6859) Speech Based Double Talk,”  2015\.
  * [28] “3GPP TS 26.132: Speech and video telephony terminal acoustic test specification,”  2020\.
  * [29] Ross Cutler, Ando Saabas, Tanel Parnamaa, Markus Loide, Robert Aichner, Hannes Braun, Sebastian Gamper, , Sriram Srinivasan, and Karsten Sorensen,  “INTERSPEECH 2021 Acoustic Echo Cancellation Challenge,”  in INTERSPEECH, 2021.
  * [30] Chandan K A Reddy, Vishak Gopal, and Ross Cutler,  “DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors,”  in ICASSP, 2021.
  * [31] Joan Serrà, Jordi Pons, and Pascual Santiago,  “SESQA: Semi-supervised learning for speech quality assessment,”  in INTERSPEECH, 2021.
  * [32] Diederik P. Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2014.
  * [33] Ross Cutler, Yasaman Hosseinkashi, Jamie Pool, Senja Filipi, Robert Aichner, Yuan Tu, and Johannes Gehrke,  “Meeting effectiveness and inclusiveness in remote collaboration,”  Proceedings of the ACM on Human-Computer Interaction, vol. 5, no. CSCW1, pp. 1–29, 2021.
  * [34] Gabriel Mittag, Saman Zadtootaghaj, Thilo Michael, Babak Naderi, and Sebastian Möller,  “Bias-aware loss for training image and speech quality prediction models from multiple datasets,”  in 2021 13th International Conference on Quality of Multimedia Experience (QoMEX), 2021, pp. 97–102.


# MODEL ORDER REDUCTION OF SOLIDIFICATION PROBLEMSECCOMAS CONGRESS 2022

Florian Arbes1,2 Øyvind Jensen1 Kent-Andre Mardal2 and Jørgen S. Dokken3 1
Institute for Energy Technology (IFE)
Instituttveien 18 2007 Kjeller, Norway
e-mail: {florian.arbes<EMAIL_ADDRESS>and 2 Department of Mathematics,
University of Oslo (UiO)
Moltke Moes vei 35, 0851 Oslo, Norway
email<EMAIL_ADDRESS>and 3 Department of Engineering, University of
Cambridge
Trumpington St, Cambridge CB2 1PZ, United Kingdom
email<EMAIL_ADDRESS>
###### Abstract

Advection driven problems are known to be difficult to model with a reduced
basis because of a slow decay of the Kolmogorov $N$-width. This paper
investigates how this challenge transfers to the context of solidification
problems and tries to answer when and to what extend reduced order models
(ROMs) work for solidification problems. In solidification problems, the
challenge is not the advection per se, but rather a moving solidification
front. This paper studies reduced spaces for 1D step functions that move in
time, which can either be seen as advection of a quantity or as a moving
solidification front. Furthermore, the reduced space of a 2D solidification
test case is compared with the reduced space of an alloy solidification
featuring a mushy zone. The results show that not only the PDE itself, but the
smoothness of the solution is crucial for the decay of the singular values and
thus the quality of a reduced space representation.

###### keywords:

model order reduction, solidification, proper orthogonal decomposition

Florian Arbes, Øyvind Jensen, Kent-Andre Mardal and Jørgen S. Dokken

##  1 INTRODUCTION

Solidification processes in the material processing industry are complex and
need to be modelled numerically. The underlying multi-physics equations for
fluid flow, heat transfer, and phase change must be coupled to one another
[1]. After discretizing in time, the underlying equations are often split up
to speed up the numerical computation of the solution. Velocity components and
the pressure are solved for in a segregated manner. There is a number of so
called fractional step algorithms to solve the incompressible Navier-Stokes
equations, see e.g. [2, 3] for overviews. Often, three steps are involved [4]:
firstly, one solves for a tentative velocity vector $\textbf{u}^{I}$
component-wise based on a pressure guess $p^{*}$. Secondly, a pressure
correction $\varphi=p^{n-1/2}-p^{*}$ is computed and lastly a velocity
correction is computed. The $k$-th velocity component is denoted with
$u^{I}_{k}$. Step one and two may be repeated iteratively to achieve higher
accuracy. Simo and Armero [5] outline a general fractional step algorithm as
followed:

$\displaystyle\frac{u^{I}_{k}-u^{n-1}_{k}}{\Delta t}+B^{n-1/2}_{k}$
$\displaystyle=\nu\nabla^{2}\widetilde{u}_{k}-\nabla_{k}p^{*}+f^{n-1/2}_{k}\text{
for }k=1,...,d,$ (1) $\displaystyle\nabla^{2}\varphi$
$\displaystyle=-\frac{1}{\Delta t}\nabla\cdot\textbf{u}^{I},$ (2)
$\displaystyle\frac{u^{n}_{k}-u^{I}_{k}}{\Delta t}$
$\displaystyle=-\nabla_{k}\varphi\text{ for }k=1,...,d,$ (3)

where $B^{n-1/2}_{k}$ denotes the nonlinear convection term evaluated at the
midpoint $n-1/2$ of the time interval $\Delta t$ and $d$ the dimensionality of
the problem.

For alloy solidifications, the interface between solid and liquid is is not
sharp. The different components in an alloy composition have different melting
points, therefore mobile equiaxed grains appear in the melt when cooling down.
While solidifying, these grains form a coherent structure before the
composition fully solidifies at a lower temperature (c.f. fig. 1). The so
called mushy zone is defined by a temperature dependent mix of solid and
liquid. Non-linearities like that make for challenging numerical modelling and
high computational costs, which is problematic when either fast predictions
are needed or when an application is studied for varying parameters.

Figure 1: The mushy zone is a mixture of solid and liquid that leads to a
reduced fluid flow (adapted from [6]).

To address computational challenges, reduced order models (ROMs) have been
developed. ROMs aim to approximate solutions efficiently at little or no
expense of accuracy. Fast and reliable predictions open up new possibilities
to study a numerical problem for varying parameters. Reduced order modelling
refers to replacing a high fidelity problem with a lower complexity numerical
model [7]. PDEs are solved by discretizing the domain, which then leads to
high-dimensional numerical models with millions of degrees of freedom [8]. The
solutions resulting from different values of a parameter often suggest an
underlying structure that can be described by a model with fewer degrees of
freedom. A ROM aims to evaluate the solution of any set of new parameters at a
computational cost that does not depend on the number of degrees of freedom of
the high fidelity model.

ROMs are built in what is referred to as an offline stage and employed in an
online stage. In the offline stage, a reduced basis is generated for example
with a greedy approach or via a proper orthogonal decomposition (POD). In any
case, the high fidelity full order model (FOM) is evaluated several times for
different parameters $\mu\in\mathbb{R}^{d}$ that characterize the problem. In
case of the solidification problem, the parameters might be cooling rates,
wall temperatures, or material properties [9, 10]. In order to build the ROM
using a POD, the discrete solutions $u_{\delta}(\mu)$ (snapshots) need to be
collected in a matrix $X$. Each column in the snapshot matrix corresponds to
one snapshot, which means the column index $j$ maps the parameter space $\mu$
to the corresponding snapshots. Each row in the snapshot matrix corresponds to
a node and its associated quantity (velocity in x, y, z, pressure, solid
fraction, etc.). The POD can then be represented by a singular value
decomposition (SVD) of the the snapshot matrix $X$ [11]:

$X=U\Sigma V^{T}.$ (4)

Once the reduced basis is found in the offline phase, the model can be
employed in the online phase, which can be done in an intrusive or non-
intrusive manner. For a new parameter, the reduced basis solution
$v_{rb}(\mu)$ can be found by projecting the governing equations onto the
reduced space, also known as a Galerkin projection. A Galerkin projection
guarantees that the error of the reduced space approximation is orthogonal to
the reduced space, meaning the result is the best possible solution in the
reduced space. However, this approach often suffers from stability issues and
runs into efficiency problems when it comes to non-linear problems [12, 13].
There are methods that try to overcome these limitations, the interested
reader is referred to books on the subjects [11, 7, 14, 15]. Since the ”POD-
Galerkin” approach depends on the equations of the FOM, existing code has to
modified and expanded, which is cumbersome and prone to error. Multi-physics
problems such as a solidification problem rely on extensive code libraries
[16]. The equations of the fluid flow have to be coupled with the equations
for the heat transfer as well as those of a the phase change, and non-
linearities need to be handled carefully [1].

We seek a non-intrusive ROM, because these models are purely data-driven
surrogate models that do not depend on the FOM. Non-intrusive reduced order
models (NIROM) rely on methods that infer the right singular values from the
parameters $\mu$ that corresponds to the snapshot matrix. This can be done
with e.g. a radial basis function (RBF) interpolation [17]. A straightforward
approach would be a standard interpolation on a regular grid, however, this
could require a lot of snapshots if the dimensionality of the parameter space
is high. Xiao et al. proposed Smolyak sparse grids in this context to avoid an
extensive amount of snapshots [13, 18]. Deep neural networks have successfully
been used for the mapping of the right singular values as well as for
uncertainty quantification [19, 20].

For a reduced order model it is important to know how well the solution
manifold $M_{\delta}$ can be approximated. The solution manifold is the set of
all solutions of the problem for any choice of the parameters. The error
arising from a reduced space approximation $v_{rb}$ obtained through a
Galerkin projection is measured with the Kolmogorov $N$-width $d_{N}$ [15]:

$d_{N}(M_{\delta})=\inf\limits_{\mathbb{V}_{rb}}\sup\limits_{u_{\delta}\in
M_{\delta}}\inf\limits_{v_{rb}\in\mathbb{V}_{rb}}||u_{\delta}(\mu)-v_{rb}(\mu)||_{\mathbb{V}}.$
(5)

The error depends on the number of basis functions $N$ that span the subspace,
hence the name ”$N$-width”. A fast decay of the $N$-width means the underlying
problem is low-dimensional and it can be approximated with a fewer basis
functions. The $N$-width is problem-dependant and depends on the coercivity
and continuity constants of the bilinear form (c.f. Céa´s Lemma). Ohlberger
[21] points out that advection driven phenomena suffer from a very slow decay
of the Kolmogorov $N$-width. It will be demonstrated, that discontinuities or
jumps of a quantity in space that move in time correlate with a slow decay of
the singular values. Jumps appear in the solution snapshots for solidification
problems too: Due to the abrupt variation in material properties associated
with the phase change, the velocity field $\mathbf{u}$ may vary strongly over
a short distance These non-stationary jumps in the snapshots do not propagate
by advection, but rather by the complex physics of the solidification process.
However, the POD is ignorant of how the non-stationary jumps originally came
into the snapshots and we find it fruitful to analyse solidification problems
using results developed for advection problems. This paper aims to investigate
how well reduced spaces can approximate solidification problems and how to
extend reduced order models for such problems.

##  2 1D TEST CASES

It is widely known, that pure Poisson problems have an underlying structure
with a low rank that can be exploited using a ROM that is built on a linear
subspace found with a POD [21, 11]. The Kolmogorov $N$-width decays
exponentially fast. In contrast to that, advection driven problems often
suffer from a slow decay of the Kolmogorov $N$-width.

In order to give a demonstration and a reference, two simple time dependant 1D
problems have been set up. The first solves the heat equation, with a
rectangular function as initial condition that fulfills homogeneous Dirichlet
boundary conditions:

$\displaystyle\begin{split}\frac{\partial u}{\partial
t}-\nabla\cdot\left(\alpha\nabla u\right)&=0\quad{\rm in}\ \Omega,\\\
u&=0\quad{\rm on}\ \partial\Omega.\end{split}$ (6)

The thermal conductivity $\alpha$ as well as the timestep $\Delta t$ were
chosen to be constant ($\alpha=1.0$ and $\Delta t=0.001$). After discretizing
in time using forward Euler, the equation is solved for each timestep using
first order Lagrange elements. The problem is solved using 256 nodes in space
and the first 128 snapshots are collected in a snapshot matrix. Some of the
resulting snapshots are shown in fig. 2(a).

The second reference problem is an advected step function, as described in
[21], where the solution is given as:

$u(x,t)=\begin{dcases*}1,&if $x\leq t$,\\\ 0,&\text{otherwise}.\end{dcases*}$
(7)

In eq. 7, $x$ is the spatial coordinate of the nodes and $t\in[0,1]$. For the
sake of consistency, the number of nodes is chosen to be $N=256$ and there are
128 snapshots being collected. Some of the resulting snapshots are shown in
fig. 2(b).

(a) Solutions to the 1D heat equation

(b) Advected jump discontinuity

Figure 2: Selected snapshots of two classical time dependant problems.

The resulting snapshot matrices are decomposed using POD. As expected, the
solution snapshots from the heat equation show an exponentially fast decay of
the singular values (see fig. 3(a)). In contrast to that, the singular values
of the snapshot matrix containing snapshots from the advection reference case
decay slowly (see fig. 3(b)). This fits nicely to theoretical analysis of the
Kolmogorov $N$-width. In fact, Ohlberger and Rave [21] have shown, that the
Kolmogorov $N$-width decays with $\frac{1}{2\sqrt{n}}$. Greif and Urban [22]
extended that proof to hyperbolic wave equation with discontinuous initial
conditions.

(a) Normalized decay of the singular values of the heat diffusion problem
(snapshots shown in fig. 2(a))

(b) Normalized decay of the singular values of the advected jump (snapshots
shown in fig. 2(b))

Figure 3: Normalized decay of the singular values of two classical time
dependant problems.

Advected jump discontinuities are rarely seen in real world problems. In case
of no-slip boundary conditions, the velocity profile of a fluid flow problem
is not expected to feature a jump discontinuity in its profile. Even if the
boundary layer is thin, the velocity field can be expected to be smooth.
However, the boundary layers might move through the domain, e.g. with a
progressing solidification. Advected quantities such as temperature or
enthalpy are also expected to be smooth because of diffusivity.

In contrast to the advected jump discontinuity, an advected smooth step
function in the shape of a sigmoid function (c.f. fig. 4(a)) does seem to
decay exponentially fast (c.f. fig. 5(a)). With decreasing steepness (c.f.
fig. 4(b)), the decay of the singular values is faster.

We have shown that the decay of the singular values does not only depend on
the PDE, but also on the slope and smoothness of the solution. These results
are relevant in the context of solidification of alloys, as the naturally
occurring mushy zone has has the effect of smoothing out the jump in the
velocity field and thus enhances the decay of the singular values. The next
chapter will discuss how these results transfer to a 2D solidification with or
without a mushy zone.

(a) Advected steep sigmoid function

(b) Advected stretched sigmoid function

Figure 4: Selected snapshots of two classical time dependent problems.

(a) Normalized decay of the singular values of the advected steep sigmoid
function (snapshots shown in fig. 4(a))

(b) Normalized decay of the singular values of the advected stretched sigmoid
function (snapshots shown in fig. 4(b))

Figure 5: Normalized decay of the singular values of two advection problems.

##  3 SOLIDIFICATION TEST CASES

A 2D solidification test case has been set up to test the impact of the mushy
zone on the decay of the singular values. The domain is a unit square, that
allows heat transfer through the boundary on the right side, as seen in fig.
6. Using the Boussinesq approximation, an implicit pressure correction scheme
was implemented to solve the problem. The scheme treats the convection
implicitly by using an Adams-Bashforth projected convecting velocity and
Crank-Nicolson for the convected velocity. Standard Taylor-Hood (continuous
P2-P1) elements were used. In this work we model the restrictions on flow due
to the solidification through an increase in viscosity. In case of an alloy
solidification, the viscosity $\mu$ is calculated as $\mu=10(T-650)^{2}$ below
$650^{\circ}C$, where $T$ is the temperature in ${}^{\circ}C$. This allows for
some movement in the mushy zone, because there is a diffuse interface between
solid and liquid, as displayed in fig. 1. In case of a pure metal
solidification, i.e. without mushy zone, there is a jump in viscosity over
several orders of magnitude at the freezing point ($650^{\circ}C$) to model a
sharp interface between solid and liquid.

Figure 6: Velocity (normalized values, plotted on a coarse grid) and
temperature field of a freezing unit square featuring a mushy zone (snapshot
in time).

For both cases, more than $5000$ snapshots have been collected. The subsequent
SVD revealed, that the mushy zone has a big impact on the decay of the
singular values, c.f. fig. 7. To capture $99.99\%$ of the energy in $5000$
snapshots, only $171$ snapshots are needed in case of a mushy zone being
present. If there is no mushy zone, one needs $1527$ snapshots. These results
support the 1D results found in the previous chapter and demonstrate the
significance of a mushy zone for a ROM.

Figure 7: Normalized decay of the singular values for a 2D solidification case
study with and without a mushy zone.

It is interesting to note, that the velocity field seems to be responsible for
the slow decay of the singular values. Figure 8 shows a contour plot of the
normalized velocity magnitude for both cases. The temperature driven
convection is clearly visible, as well as the solid, that does not allow for
any movement. The mushy zone is also visible in fig. 8(a), as the magnitude of
the velocity increases gradually and the distance between the solid and the
largest velocity is large. Figure 8(a) shows in contrast a smaller proximity
of small and large values of the velocity magnitude, because there is no mushy
zone. As seen in the 1D case study, a steep slope in the advected quantity
seems to have a negative impact on the decay of the singular values.

(a) Normalized velocity magnitude of a case featuring a mushy zone (snapshot
in time).

(b) Normalized velocity magnitude of a case without a mushy zone.

Figure 8: Magnitude of the velocity.

##  4 ON THE POTENTIAL OF A COMPONENT WISE ROM

As seen in the previous chapter, the velocity field of a solidification
without a mushy zone shows steep gradients, which leads to a slow decay of the
singular values. Even though velocity, pressure, temperature and viscosity are
coupled through the PDE, they are usually computed in a segregated manner.
That motivates to investigate the decay of other quantities separately for the
solidification case without a mushy zone. In fig. 9, the decay of the
components is plotted separately.

Figure 9: Separately computed normalized decay of the singular values for
velocity in x- and y-direction and pressure.

As expected, singular values for the velocity decays slowly, and the singular
values for the pressure decays much quicker. Since the pressure is computed
separately (see eq. 2), this opens up the opportunity to build a ROM for the
pressure only. A pressure ROM could potentially be used in the numerical
fractional step algorithm of the full order model, because pressure and
velocity are treated separately.

##  5 CONCLUSIONS

The quality of metal casts can benefit from fast numerical models that help to
prevent defects. However, the potential for reduced order models is limited,
because advection dominated problems suffer from a slow decay of the singular
values. This challenge transfers to metal solidification problems, but not in
the same extent to alloy solidification problems. The numerical experiments
conducted suggest that the velocity field moving with the solidification front
leads to a slow decay of the singular values. However, an alloy solidification
with a mushy zone allows for much better ROMs since the singular values decay
much faster. The mushy zone allows for some movement in the solid-liquid
mixture, which leads to a smoother velocity field. In this paper, we
demonstrated that not only the PDE itself, but the smoothness of the solution
determines the decay of the singular values. This was shown for a simple 1D
test case and the results from the 2D solidification test cases suggest that
this applies to alloy solidifications too. Moreover it has been shown, that
the pressure decays much faster than the slowly decaying velocity in the metal
solidification case. A pressure ROM opens up opportunities to speed up
numerical fractional step algorithms that treat pressure and velocity
separately.

##  6 ACKNOWLEDGMENT

The authors gratefully acknowledge the STIPINST funding [318024] from the
Research Council of Norway.

## References

  * [1] Beckermann C, Viskanta R Mathematical modeling of transport phenomena during alloy solidification 46, 1, 1–27.  (10.1115/1.3120318).  Publisher: American Society of Mechanical Engineers Digital Collection.
  * [2] Langtangen HP, Mardal KA, Winther R Numerical methods for incompressible viscous flow 25, 8, 1125–1146.  (10.1016/S0309-1708(02)00052-0).
  * [3] Turek S _Efficient Solvers for Incompressible Flow Problems: An Algorithmic and Computational Approach_ , vol. 6.  Springer Science & Business Media.
  * [4] Mortensen M, Valen-Sendstad K Oasis: A high-level/high-performance open source navier–stokes solver 188, 177–188.  (10.1016/j.cpc.2014.10.026).
  * [5] Simo JC, Armero F Unconditional stability and long-term behavior of transient algorithms for the incompressible navier-stokes and euler equations 111, 1, 111–154.  (10.1016/0045-7825(94)90042-6).
  * [6] Kumar A, Walker MJ, Sundarraj S, Dutta P Grain floatation during equiaxed solidification of an al-cu alloy in a side-cooled cavity: Part II—numerical studies 42, 783–799.  (10.1007/s11663-011-9542-1).  ADS Bibcode: 2011MMTB…42..783K.
  * [7] Quarteroni A, Manzoni A, Negri F _Reduced Basis Methods for Partial Differential Equations: An Introduction_.  Springer, 1st ed. 2016 edition ed.
  * [8] Richardson CN, Sime N, Wells GN Scalable computation of thermomechanical turbomachinery problems 155, 32–42.  (10.1016/j.finel.2018.11.002).
  * [9] Zimmerman AG, Kowalski J Monolithic simulation of convection-coupled phase-change - verification and reproducibility (10.48550/arXiv.1801.03429).
  * [10] Torabi Rad M, Viardin A, Schmitz GJ, Apel M Theory-training deep neural networks for an alloy solidification benchmark problem 180, 109687.  (10.1016/j.commatsci.2020.109687).
  * [11] Brunton SL, Kutz JN _Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control_.  Cambridge University Press.  (10.1017/9781108380690).
  * [12] Chen H.  Blackbox stencil interpolation method for model reduction.  Accepted: 2013-02-15T14:38:44Z.
  * [13] Xiao D Non-intrusive reduced order models and their applications (10.25560/41845).  Accepted: 2016-10-20T14:20:44Z Publisher: Imperial College London.
  * [14] Benner P, Ohlberger M, Cohen A, Willcox K _Model Reduction and Approximation_.  Computational Science & Engineering. Society for Industrial and Applied Mathematics.  (10.1137/1.9781611974829).
  * [15] Hesthaven JS, Rozza G, Stamm B _Certified Reduced Basis Methods for Parametrized Partial Differential Equations_.  Springer, 1st ed. 2016 edition ed.
  * [16] Mortensen D, Fjær HG, Lindholm D, Rudshaug M, Sørheim EA The development of alsim – a modelling tool for direct chill casting, twin roll casting, wheel and belt casting and chain conveyor casting 693, 187–195.  (10.4028/www.scientific.net/MSF.693.187).  Conference Name: Aluminium Cast House Technology XII ISBN: 9783037852095 Publisher: Trans Tech Publications Ltd.
  * [17] Xiao D, Fang F, Pain C, Hu G Non-intrusive reduced-order modelling of the navier–stokes equations based on RBF interpolation 79, 11, 580–595.  (10.1002/fld.4066).  _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/fld.4066.
  * [18] SMOLYAK S Quadrature and interpolation formulas for tensor products of certain classes of functions 4, 240–243.
  * [19] Pawar S, Rahman SM, Vaddireddy H, San O, Rasheed A, Vedula P A deep learning enabler for nonintrusive reduced order modeling of fluid flows 31, 8, 085101.  (10.1063/1.5113494).  Publisher: American Institute of Physics.
  * [20] Jacquier P, Abdedou A, Delmas V, Soulaimani A Non-intrusive reduced-order modeling using uncertainty-aware deep neural networks and proper orthogonal decomposition: Application to flood modeling 424, 109854.  (10.1016/j.jcp.2020.109854).
  * [21] Ohlberger M, Rave S Reduced basis methods: Success, limitations and future challenges pp. 1–12%8 2016–02–28.
  * [22] Greif C, Urban K Decay of the kolmogorov n-width for wave problems 96, 216–222.  (10.1016/j.aml.2019.05.013).