# L2Explorer: A Lifelong Reinforcement Learning Assessment Environment

Erik C. Johnson, Eric Q. Nguyen, Blake Schreurs, Chigozie S. Ewulum, Chace
Ashcraft, Neil M. Fendley,
Megan M. Baker, Alexander New, Gautam K. Vallabha
Johns Hopkins University Applied Physics Laboratory
11100 Johns Hopkins Rd.
Laurel, MD 21045

###### Abstract

> Despite groundbreaking progress in reinforcement learning for robotics,
> gameplay, and other complex domains, major challenges remain in applying
> reinforcement learning to the evolving, open-world problems often found in
> critical application spaces. Reinforcement learning solutions tend to
> generalize poorly when exposed to new tasks outside of the data distribution
> they are trained on, prompting an interest in continual learning algorithms.
> In tandem with research on continual learning algorithms, there is a need
> for challenge environments, carefully designed experiments, and metrics to
> assess research progress. We address the latter need by introducing a
> framework for continual reinforcement-learning development and assessment
> using Lifelong Learning Explorer (L2Explorer), a new, Unity-based, first-
> person 3D exploration environment that can be continuously reconfigured to
> generate a range of tasks and task variants structured into complex and
> evolving evaluation curricula. In contrast to procedurally generated worlds
> with randomized components, we have developed a systematic approach to
> defining curricula in response to controlled changes with accompanying
> metrics to assess transfer, performance recovery, and data efficiency. Taken
> together, the L2Explorer environment and evaluation approach provides a
> framework for developing future evaluation methodologies in open-world
> settings and rigorously evaluating approaches to lifelong learning.

In recent years, Deep Reinforcement Learning (DRL) approaches have begun to
deliver powerful results for a variety of compelling domains, including games
such as Chess, Go, and Shogi Silver et al. (2018); Atari video games Mnih et
al. (2013); more complex strategy video games Berner et al. (2019); Vinyals et
al. (2019); and dexterous robotic manipulation Rajeswaran et al. (2017).
Despite the groundbreaking success in training autonomous agents, resulting
policies tend to be very brittle and generalize poorly Chan et al. (2019).
When presented with a new task or a task variant, DRL approaches are
susceptible to a performance drop Zhang et al. (2018); Kirk et al. (2021) due
to the catastrophic forgetting problem French (1999); McCloskey and Cohen
(1989), which may not be overcome by domain randomization strategies alone. As
the field moves from environments which are fixed to evolving, open-world
scenarios, current DRL approaches will be insufficient.

This performance gap has led to an interest in Continual Learning, which seeks
to design algorithms to learn over sequences of tasks. In the related, but
broader, concept of Lifelong Learning Chen and Liu (2018), an agent learns
over a lifetime of experiences (see Fig. 1) in an evolving environment (for
purposes of this paper, however, we treat continual learning as synonymous
with lifelong learning as our approach is applicable to both concepts). Much
recent work has been on supervised classification under distribution shifts
Song et al. (2020) and learning a sequence of tasks Parisi et al. (2019); Hsu
et al. (2018). Continual RL Khetarpal et al. (2020) seeks to create agents
which can maintain performance in the face of nonstationary distributions.

A key issue in developing Continual RL algorithms is how to assess performance
in a rigorous and informative way. There are existing attempts to address this
with procedurally generated open worlds Risi and Togelius (2020), which
generate tasks using randomly parameterized environments. A recent software
framework has begun integrating metrics, baselines, and environments for
Continual RL Powers et al. (2021). We argue that successful Continual RL
assessment requires both sufficiently complex environments and highly
reconfigurable tasks, as well as carefully structured experiments and metrics.
We must also consider a multi-dimensional approach to assessment (in terms of
raw performance, generalization, task transfer, and data efficiency).

To help address this need, we have developed Lifelong Learning Explorer
(L2Explorer), a first-person-view (FPV), highly configurable UnityTM
environment111https://unity.com/. The environment allows for procedural
generation through Python code for open-ended task and task variant
definitions. Moreover, the environment is set up for testing lifelong learning
curricula with a set of lifelong learning metrics. While the Unity environment
itself is designed for Continual RL assessment, this work also outlines an
approach for taking complex, reconfigurable open-world environments and
creating rigorous evaluation for lifelong learning algorithms.

## Existing Environments for Continual RL

Many environments have been used to test DRL approaches, including game
environments. Extensive work, including transfer learning studies, have been
conducted with Atari games Mnih et al. (2013). This has been generalized into
a meta-learning framework that allows sampling of Atari-like games Staley et
al. (2021). Complex strategy games have also been leveraged in this context,
including Starcraft 2222deepmind.com/research/open-source/pysc2-starcraft-ii-
learning-environment, team-based play in Dota 2 Berner et al. (2019), and
rogue-likes such as Dungeon Crawl Stone Soup Dannenhauer et al. (2019). While
impressive, many games can be limited for Continual RL testing due to the lack
of full configurablility.

Simulators of real-world systems, such as autonomous vehicles, have also been
extensively used. Two notable examples are the CARLA simulator Dosovitskiy et
al. (2017) for autonomous driving and the AirSim package for unmanned aerial
vehicles Shah et al. (2018). While these include photorealistic images and
realistic physics, the computational complexity of the environments coupled
with the design time required to produce tasks and task variants can limit
their applicability to Continual RL testing.

There are also many FPV environments, which combine some of the desirable
features of game environments and physical simulators. These are valuable for
continual and lifelong learning testing because they allow natural inclusion
of partial observations, multiple observation modes, and proxies of real world
tasks. These strengths come at the downside of reduced task complexity and
reduced fidelity. Examples include the use of the Unity Environment through
the ML-Agents package Juliani et al. (2018) and the procedurally generated
Obstacle Tower Juliani et al. (2019) environment. The FPV game DOOM forms the
basis of the Vizdoom FPV environment Kempka et al. (2016). Also related are
the real-world home and robot environment simulators such as AI2Thor Kolve et
al. (2017), which also introduce semantic object relationships into FPV
environments.

In order to introduce the flexibility required for open-world learning
scenarios, increasing emphasis has been placed on procedurally generated
environments. These include the open-world game environments of XLand Team et
al. (2021) and MiniHack Samvelyan et al. (2021). These represent major steps
forward, but the lack of controllable parameters limits the precision of
testing that can be done in these environments Kirk et al. (2021).

Our environment and assessment framework, L2Explorer, seeks to combine the
strengths of different types of environments. The use of Unity allows for a
visually complex world, and ML-Agents exposes hooks that allow for principled
procedural generation using flexible Python code. The environment has lower
computational complexity than simulators of autonomous cars or aerial
vehicles. Finally, the critical component is linking this procedurally
generated environment to careful experimental design and metrics, similar to
CoRA Powers et al. (2021). We aim to create an approach that will also
generalize and improve Continual RL assessment utilizing other open-world
environments.

Figure 1: Key considerations when assessing lifelong learning agents (a
Lifelong Learning Machine, L2M) with open-world environments. An agent’s
lifetime is broken into blocks of tasks (e.g., playing an individual sport)
for training and evaluation. Blocks generate metrics for assessing
performance. Our approach utilizes a highly flexible environment to generate
curricula (lifetimes), tasks, and task variants. A multi-dimensional approach
to metrics, going beyond raw performance, is required to capture the nuances
such as transfer and performance recovery. Controlled sequences of tasks can
elucidate particular strengths and weaknesses in learners. Figure adapted from
New et al. (2022).

## L2Explorer Design Approach

Similar to previous efforts to define key requirements for lifelong learning
evaluations Farquhar and Gal (2018), we developed key criteria for our
framework which allow for rigorous assessment of lifelong learning with an
open-ended, procedurally generated environment. We believe the key criteria of
our framework to be:

  * •

Flexible description of multiple tasks: A programmatic API to specify new
tasks with a common specification

  * •

Flexible control over task variants: Selection of key variables that can be
modified within a task definition

  * •

Notions of task relationships/similarity: While there is no agreed upon
approach to measure generic task similarity for RL tasks, some notion of task
similarity is required to appropriately structure evaluation

  * •

Control over degree of similarity: The environment needs to be able to control
the degree of similarity between configurations to allow for abrupt and
gradual transitions

  * •

Parametric and random variation: The tasks require the notion of parameters
used to deterministically create variants, as well as intrinsic parameters
that can be randomly sampled, similar to the previous work Kirk et al. (2021)

  * •

Targeted testing curricula: A set of designed curricula and tests, along with
integrated metrics and baselines

Taken together, these factors address some of the limitations that come with
testing with open-world, procedurally generated environments.

### Continual RL Assessment

In L2Explorer, we build on the standard formalism for RL agents using
Partially Observable Markov Decision Processes (POMDPs). These consist of the
tuple $M=(S,A,O,R,T,\phi,p)$, where $S$ is the state space, $A$ is the space
of actions available to an agent, $R:S\times A\times S\to\mathbb{R}$ is the
reward function, $T(s^{\prime}|s,a)$ is the state transition function, and $p$
is the distribution of initial states. The system is partially observable as
the state is not directly available to the agent, which instead gets an
observation from the observation space $O$ generated by the emission function
$\phi:S\to O$. The goal of the RL agent is to learn a policy $\pi(a|s)$.

$\pi^{*}=\text{argmax}_{\pi\in\Pi}\mathbb{E}_{s\sim p(s_{0})}[R(s)]$

where $R(s)$, which is a real value, is the total expected reward gained by
the state.

We consider nonstationarities in the underlying POMDP for both tasks and tasks
variants. Of critical consideration are $S$, $A$, $T(s^{\prime}|s,a)$, $R$,
$O$, and $\phi$. In continual reinforcement learning, these are allowed to
vary as a function of the number of episodes to create a nonstationary POMDP.

To expose agents to nonstationarity in a meaningful way, controlled variation
in environment is required. It is rarely practical to fully quantify
distributions in high dimensional POMDPs, so instead we seek useful and
practical surrogates for controlled manipulation of nonstationary POMDPs. We
propose that each curricula should have some desired testing hypothesis
relating to changing distributions in the underlying POMDP. Different tasks
and task variants can represent particular distributions in the POMDP tuple,
and altering parameters results in a shift in these distributions.

Achieving this requires some notion of task and variant similarity. We propose
exploiting the parameters of procedural generation to create heuristics of
similarity. For example, if two task differ only by the probability
distribution used to place objects, a distance metric applied to the tasks’
distributions could be a proxy measure for the similarity. A key caveat,
however, is that heuristic notions of similarity may not be inherently related
to agent transfer. What seems intuitively similar to a human may not
correspond to positive transfer between tasks for an RL algorithm, and tasks
that may be similar for one algorithm may not be for another Carroll (2005).
Research into measures of similarity, transfer, and generalization is ongoing
Barreto et al. (2016); Ma et al. (2020) and beyond the scope of this work.

L2Explorer variants are designed by selecting a subset of procedural
generation parameters that must be fixed, and a subset that may be fixed or
allowed to be sampled from a distribution. Differences in the parameter values
may be passed into the appropriate similarity heuristics. Tasks are assembled
into curricula with particular experimental goals, such as determining
sensitivity to changes in reward space, the observation emission function, and
so forth. This overall approach will allow for meaningful experimental design
and overcome limitations of completely randomly generated environments Song et
al. (2020); Kirk et al. (2021).

### Curriculum Design Considerations

We utilize existing a lifelong learning evaluation framework for curriculum
design, as specified in New et al. (2022) and released open-source111
https://github.com/darpa-l2m/l2metrics, https://github.com/darpa-l2m/l2logger.
This can be seen in Fig. 1, where an agent is pretrained and then deployed on
a sequence of tasks. A lifelong learner should be able to take such a fixed
curriculum and demonstrate learning over deployment. This framework allows for
pre-deployment training or parameter selection before learning. The agent is
then subjected to a series of units of experience (e.g., rounds in a game)
that can be grouped into longer blocks. We assume blocks can incorporate drift
in the underlying distributions of the POMDP. Periodically, the agent can be
frozen and have its capabilities tested in a separate evaluation block.
Individual environment metrics can be specified, then aggregated into
environment-agnostic lifelong learning metrics (detailed below). In
L2Explorer, similarity heuristics can be utilized to order tasks and task
variants into sequences of learning and evaluation blocks.

### Metrics

L2Explorer utilizes an existing set of lifelong learning metrics that
characterizes the nuances of the performance curves such as the one in Fig. 1.
Full definitions can be found in New et al. (2022), with open-source
implementations available. Task-specific performance measures (e.g., total
reward, time to completion) generated during learning and evaluation blocks
enable computation of:

  * •

Performance Maintenance: A measure of catastrophic forgetting for a given task

  * •

Forward Transfer: A measure of learning improvement for a task, given learning
experiences on a different task

  * •

Backward Transfer: A measure of improvement on a previously learned task given
learning experiences on a new task

  * •

Performance Relative to Single-Task Expert: Comparison of lifelong learning
agent to an expert agent trained only on the task in question (e.g., the
dashed lines in Fig. 1).

  * •

Sample Efficiency: A measure of the experience required for a lifelong
learning agent to reach maximal performance relative to a single task expert

Together, the metrics, curriculum design, and design philosophy inform the
development of the L2Explorer environment. Lessons from this approach can be
generalized to assessment with future open world exploration environments.

Figure 2: Overview of L2Explorer Software system, consisting of a suite of
Python tools which interact with a custom Unity environment. The Python suite
allows for execution of testing curricula, integration of standard agent code,
and rapid reconfiguration of the unity environment through the reset channel.
Logging and metrics computations are integrated. A custom asset creator allows
new Unity models to be incorporated into the framework to maximize
extensibility.

## L2Explorer

L2Explorer is a software framework and testing environment which implements
this overall design vision, including integration with metrics and testing
curricula. The project builds on Unity ML-Agents Juliani et al. (2018), which
exposes key parameters to create a reconfigurable, 3D, FPV reinforcement-
learning environment. Our custom Unity environment enables exploration and
interaction with a world specified through a JSON object format. Additional
side channels allow communication of observations, actions, and debugging
information. The Python code specifies the world JSON objects, allows for
creating testing curricula, integrates metrics, and provides a Gym-compatible
interface Brockman et al. (2016). An overview of the software can be seen in
Fig. 2, and is intended to be released open-source. The package will provide
testing curricula and baselines to assist development, and will allow users to
fully customize tasks, task variants, curricula, and even import custom 3D
models into the environment.

### Unity Executable

The core environment of L2Explorer is a custom Unity app which instantiates a
simulated world in response to a JSON specification communicated through a
Unity ML-Agents side channel. This is activated when the reset function is
called, allowing each learning experience to take place in a custom specified
world. A partial example of a reset JSON can be seen in Fig. 3. The format
allows for specification of environment specific parameters, including
lighting conditions, backgrounds, agent specific parameters, and object
specific parameters. Pre-caching 3D models enables rapid world construction on
reset.

The Unity environment exposes three custom communication channels seen in Fig.
2. These allow the reset functionality, querying the current state of the
world (returned in the specified world JSON format), and sending debugging
information. Standard channels allow for the communication of actions
(continuous or discrete) and observations (state vectors and visual
observations). Visual observations include RGB images, grayscale depth images,
and semantic segmentation maps (segmented by the object “class”).

The agent model is, by default, a kinematic model controlled by a linear and
angular velocity term. Several agent-object interaction models are specified.
These can require an agent to be in a nearby zone, require contact with the
object, or require the agent to take an “interact” action. Interaction can
result in a positive or negative reward, and can result in the destruction of
the object. There is dynamic respawning through the side channels, allowing
new objects to be specified during runtime. For certain task designs, this can
be key for learning with fixed episode step length.

A separate unity executable is provided to allow users to prepare and save
custom 3D models in an L2Explorer compatible format. These assets, once
prepared, can be loaded in during a reset action. This can allow for continued
extensibility of the environment.

Figure 3: Example subset of a reset JSON object. This shows the available
variables to procedurally generate variants of the environment, agent, and
objects. Each object can have independently specified coordinates and
interaction models, and objects can be created dynamically within an episode.
On each reset call, a new JSON can be loaded into the environment.

### Python Wrappers

In L2Explorer, Python wrappers allow for task specification and agent
development. The primary class, L2ExplorerTask, is a wrapper for communicating
with the Unity executable. This class processes the raw observations to
provide a reward and observation to the agent. This allows for definitions of
alternative rewards functions in Python, such as soft rewards, as well as any
data filtering required for a task.

The Unity environment is configured during a reset function, utilizing the
JSON specification (Fig. 3). This enables the specification of curricula in
Python, as well as the specification of procedural generation. Curricula are
specified through task-specific classes and a task-specific initial JSON file.
Learning and Evaluation Blocks can then be described as above, along with
variants of the task. The base JSON file for the task is passed to the task
class, and key-value pairs are modified, if required, to create a variant.
Modifications are drawn from discrete or continuous probability density
functions to create appropriate randomization. The Gym-compatible environment
class allows for integration of standard baselines and custom algorithms. The
agent observation space, $O$, is a tensor of number of pixels by number of
pixels by number of channels in the selected modalities, and optionally state
vector of size $N$. The action space $a$ is a 2-dimensional continuous valued
vector (linear and angular velocity), which can optionally be discretized. A
third dimension can be added for the binary “interact” action.

In addition to the core functionality, integration with a Python logging
package and Python metrics package for lifelong learning is provided to enable
the multi-dimensional metrics evaluation. A command line interface (CLI) is
also provided for directly interfacing with and debugging the environment.

## Example Use Cases

L2Explorer is a highly configurable and extensible environment that can
provide insight into the performance of Continual RL algorithms. Towards this
end, we present some example tasks and curricula that will be refined into
standardized benchmarking experiments.

Figure 4: Variants which can be produced using the standardized JSON format
through the reset function. Panel a) shows how object properties can be
manipulated, in this case turning blue targets into red ones. Panel b) shows
the replacement of an object of the same class with a different model, in this
case swapping two trees. Panel c) shows how the reward structure can be
modulated with one reward object of value $100$ becoming two reward objects
with values $50$ and $-50$. Panel d) shows the manipulation of environmental
conditions through changes in lighting. Panel e) shows how scene complexity
can be varied through placement of additional objects from different classes.
Finally, Panel f) demonstrates how different observation modalities including
RGB, depth, and a semantic segmentation can be supplied to an agent.

### Configuration of L2Explorer

Fig. 4 shows some aspects of the world that can be reconfigured. These relate
(indirectly) to the underlying POMDP, including $O$, $\phi$, and $S$, by
altering the object properties, object models, environmental complexity,
environmental conditions, and observation modality. The reward $R$ and state
transition function $T$ are more directly manipulated through modifying the
object interaction model, object reward, and agent models. The extensible
L2ExplorerTask class allows for further manipulations such as soft reward
functions and observation state vectors.

Figure 5: Example task demonstrating integration of a Stable Baselines PPO
agent for learning a simple object finding task. Panel a) shows the visual
input to the agent and Panel b) an example learning curve. Standard Python
agents using the established Gym API can be rapidly tested in this framework.

### Tasks

For each task a JSON is specified from which variants can be created and
randomization can be done. Each task has a set of parameters which define the
core task, parameters to change to produce static variants, and parameters
that can be randomized within a variant. Fig. 5 shows an example of
performance curves for a Stable Baselines Proximal Policy
Optimization111https://github.com/DLR-RM/stable-baselines3. Developers of
novel algorithms will be able to exploit this interface to baseline directly
against single task experts. While L2Explorer is designed to be extensible,
several initial tasks have been developed:

  * •

Find Objects: The agent interacts with objects to receive a positive reward.
Environmental conditions, object properties, and agent properties can be
randomized. Variant tasks include observation changes of the target model,
state space changes of the map layout, and agent-interaction model changes

  * •

Get to Goal: The agent receives positive reward for navigating to a target
position. Goal positions, map layouts, and agent parameters can be randomized.
Variants include changes in visual landmarks, soft reward functions, and the
addition of negative reward objects

  * •

Select Object: The agent receives positive reward for selecting the correct
object, negative reward for other objects. Reward probabilities can be
manipulated as in a $K$-armed bandit problem. Variants include changes in the
reward probability distribution and object models

  * •

Moving Object: Similar to Find Objects, with a object motion model. Variants
allow for manipulation of the action space and state transition space through
changes in the agent and object motion models

  * •

Scavenger Hunt: The agent can obtain a series of rewards by interacting with
the right sequence of objects. Variants include changes in visual landmarks,
changes in the reward distribution, and changes in the object models

Figure 6: Example curricula showing variations in observation and reward
models to realize the evaluation framework (Fig. 1). Between each variant, the
reward structure and observation structure change, enabling analysis of
performance and manipulating the underlying POMDP. In Panel a) there is a
positive reward associated with capturing the blue object. In the variant in
Panel b) a red object object is introduced which returns a negative reward
when captured. In the variant in Panel c) a green object returns a positive
reward. Finally, in Panel d) the variants in Panel b) and Panel c) are mixed
with different environmental conditions.

### Assessment Curricula

Tasks and task variants can be assembled into learning and evaluation blocks
such as in Fig. 6. In this example changes in tasks can, through assumed
heuristics such as a norm of the object RGB color difference, approximate
changes in the observation space and reward space. The magnitude of parameter
changes can also be used as a calculable surrogate for similarity, although we
reemphasize that such heuristics do not necessarily indicate successful
transfer learning.

Preliminary curricula are being developed for several key sources of
variability impacting Continual RL agents. First is manipulation of the reward
structure. Distributions of rewards and their associated objects can be
shifted in gradual or continuous ways. Changes in the state space and state
transition space can be probed through manipulations of the maps, agent and
object motion models, and interaction models. Changes to the observation
emission function can be probed through changes in environmental factors,
observation modalities, and object models. Action space changes can relate to
the agent interaction model. The metrics will allow the assessment of changes
of different magnitudes.

Similarity measures are critical to the construction of these curricula. We
propose utilizing changes in the parameters of the reset JSON to provide a
usable, if limited, approximation of similarity. Changes to individual object
parameters between tasks can be measured with chosen distance metrics to
approximate differences in similarity between the tasks. Further, we may
provide a mapping from non-numeric values, such as a color specification of
“green”, to numeric ones, such as the array (0, 128, 0), which can then be
used with a distance measure to create another measure of similarity between
tasks. Object coordinates can also be discretized into occupancy maps to allow
for distance computations. While this approach is clearly limited, it is
simple and useful for organizing curricula for algorithm assessment.

## Conclusions

We have presented the design approach and implementation of L2Explorer, a
highly configurable, Unity-based environment for testing continual and
lifelong learning systems. In addition to the software system itself, we have
argued for a framework to approach lifelong learning assessment in open-ended
worlds. A key consideration is the integration of appropriate metrics,
experiment design, and curriculum design into the framework.

To fully demonstrate the power of this approach, integration of single task
baselines and Continual RL baselines Khetarpal et al. (2020), including replay
based approaches such as CLEAR Rolnick et al. (2018), is needed. In
combination with baseline development, the project aims to provide a set of
canonical experiments for RL assessment in a FPV environment. Moreover, we
anticipate the design framework can be reused. Without progress in assessment
frameworks, it will be a challenge to push the frontier of reinforcement
learning in open-world environments.

## Code Availability

The code for the L2Explorer framework will be made open source at
https://github.com/lifelong-learning-systems/l2explorer.

##  Acknowledgments

Development of L2Explorer was funded by the DARPA Lifelong Learning Machines
(L2M) Program. The authors would like to thank Olivia Lyons, Ji Pak, Sasha
Smith, Brianna Raphino, and Sydney Floryanzia for their help in the early
development of L2Explorer, and Angel Yanguas-Gil for valuable feedback. The
views, opinions, and/or findings expressed are those of the authors and should
not be interpreted as representing the official views or policies of the
Department of Defense or the U.S. Government.

## References

  * Silver et al. [2018] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al.  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  _Science_ , 362(6419):1140–1144, 2018.
  * Mnih et al. [2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ , 2013.
  * Berner et al. [2019] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al.  Dota 2 with large scale deep reinforcement learning.  _arXiv preprint arXiv:1912.06680_ , 2019.
  * Vinyals et al. [2019] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al.  Grandmaster level in starcraft ii using multi-agent reinforcement learning.  _Nature_ , 575(7782):350–354, 2019.
  * Rajeswaran et al. [2017] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine.  Learning complex dexterous manipulation with deep reinforcement learning and demonstrations.  _arXiv preprint arXiv:1709.10087_ , 2017.
  * Chan et al. [2019] Stephanie CY Chan, Samuel Fishman, John Canny, Anoop Korattikara, and Sergio Guadarrama.  Measuring the reliability of reinforcement learning algorithms.  _arXiv preprint arXiv:1912.05663_ , 2019.
  * Zhang et al. [2018] Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio.  A study on overfitting in deep reinforcement learning.  _arXiv preprint arXiv:1804.06893_ , 2018.
  * Kirk et al. [2021] Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel.  A survey of generalisation in deep reinforcement learning.  _arXiv preprint arXiv:2111.09794_ , 2021.
  * French [1999] Robert M. French.  Catastrophic forgetting in connectionist networks.  _Trends in Cognitive Sciences_ , 3(4):128–135, 1999.  ISSN 1364-6613.  doi:https://doi.org/10.1016/S1364-6613(99)01294-2.  URL https://www.sciencedirect.com/science/article/pii/S1364661399012942.
  * McCloskey and Cohen [1989] Michael McCloskey and Neal J. Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  volume 24 of _Psychology of Learning and Motivation_ , pages 109–165. Academic Press, 1989.  doi:https://doi.org/10.1016/S0079-7421(08)60536-8.  URL https://www.sciencedirect.com/science/article/pii/S0079742108605368.
  * Chen and Liu [2018] Zhiyuan Chen and Bing Liu.  Lifelong machine learning, second edition.  12(3):1–207, 2018.  ISSN 1939-4608, 1939-4616.  doi:10.2200/S00832ED1V01Y201802AIM037.  URL https://www.morganclaypool.com/doi/10.2200/S00832ED1V01Y201802AIM037.
  * Song et al. [2020] Liwei Song, Vikash Sehwag, Arjun Nitin Bhagoji, and Prateek Mittal.  A critical evaluation of open-world machine learning.  _arXiv preprint arXiv:2007.04391_ , 2020.
  * Parisi et al. [2019] German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter.  Continual lifelong learning with neural networks: A review.  _Neural Networks_ , 113:54–71, 2019.
  * Hsu et al. [2018] Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira.  Re-evaluating continual learning scenarios: A categorization and case for strong baselines.  _arXiv preprint arXiv:1810.12488_ , 2018.
  * Khetarpal et al. [2020] Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup.  Towards continual reinforcement learning: A review and perspectives.  _arXiv preprint arXiv:2012.13490_ , 2020.
  * Risi and Togelius [2020] Sebastian Risi and Julian Togelius.  Increasing generality in machine learning through procedural content generation.  _Nature Machine Intelligence_ , 2(8):428–436, 2020.
  * Powers et al. [2021] Sam Powers, Eliot Xing, Eric Kolve, Roozbeh Mottaghi, and Abhinav Gupta.  Cora: Benchmarks, baselines, and metrics as a platform for continual reinforcement learning agents.  _arXiv preprint arXiv:2110.10067_ , 2021.
  * Staley et al. [2021] Edward W Staley, Chace Ashcraft, Benjamin Stoler, Jared Markowitz, Gautam Vallabha, Christopher Ratto, and Kapil Katyal.  Meta arcade: A configurable environment suite for deep reinforcement learning and meta-learning.  In _Deep RL Workshop NeurIPS 2021_ , 2021.
  * Dannenhauer et al. [2019] Dustin Dannenhauer, Michael W Floyd, Jonathan Decker, and David W Aha.  Dungeon crawl stone soup as an evaluation domain for artificial intelligence.  _arXiv preprint arXiv:1902.01769_ , 2019.
  * Dosovitskiy et al. [2017] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun.  CARLA: An open urban driving simulator.  In _Proceedings of the 1st Annual Conference on Robot Learning_ , pages 1–16, 2017.
  * Shah et al. [2018] Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor.  Airsim: High-fidelity visual and physical simulation for autonomous vehicles.  In _Field and service robotics_ , pages 621–635. Springer, 2018.
  * Juliani et al. [2018] Arthur Juliani, Vincent-Pierre Berges, Ervin Teng, Andrew Cohen, Jonathan Harper, Chris Elion, Chris Goy, Yuan Gao, Hunter Henry, Marwan Mattar, et al.  Unity: A general platform for intelligent agents.  _arXiv preprint arXiv:1809.02627_ , 2018.
  * Juliani et al. [2019] Arthur Juliani, Ahmed Khalifa, Vincent-Pierre Berges, Jonathan Harper, Ervin Teng, Hunter Henry, Adam Crespi, Julian Togelius, and Danny Lange.  Obstacle tower: A generalization challenge in vision, control, and planning.  _arXiv preprint arXiv:1902.01378_ , 2019.
  * Kempka et al. [2016] Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaśkowski.  Vizdoom: A doom-based ai research platform for visual reinforcement learning.  In _2016 IEEE Conference on Computational Intelligence and Games (CIG)_ , pages 1–8. IEEE, 2016.
  * Kolve et al. [2017] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi.  Ai2-thor: An interactive 3d environment for visual ai.  _arXiv preprint arXiv:1712.05474_ , 2017.
  * Team et al. [2021] Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, et al.  Open-ended learning leads to generally capable agents.  _arXiv preprint arXiv:2107.12808_ , 2021.
  * Samvelyan et al. [2021] Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker-Holder, Minqi Jiang, Eric Hambro, Fabio Petroni, Heinrich Kuttler, Edward Grefenstette, and Tim Rocktäschel.  Minihack the planet: A sandbox for open-ended reinforcement learning research.  2021\.
  * New et al. [2022] Alexander New, Megan Baker, Eric Nguyen, and Gautam Vallabha.  Lifelong learning metrics.  _arXiv preprint arXiv:2201.08278_ , 2022.
  * Farquhar and Gal [2018] Sebastian Farquhar and Yarin Gal.  Towards robust evaluations of continual learning.  _arXiv preprint arXiv:1805.09733_ , 2018.
  * Carroll [2005] James Lamond Carroll.  Task localization, similarity, and transfer; towards a reinforcement learning task library system.  2005\.
  * Barreto et al. [2016] André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado Van Hasselt, and David Silver.  Successor features for transfer in reinforcement learning.  _arXiv preprint arXiv:1606.05312_ , 2016.
  * Ma et al. [2020] Chen Ma, Dylan R Ashley, Junfeng Wen, and Yoshua Bengio.  Universal successor features for transfer reinforcement learning.  _arXiv preprint arXiv:2001.04025_ , 2020.
  * Brockman et al. [2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.  Openai gym, 2016.
  * Rolnick et al. [2018] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P Lillicrap, and Greg Wayne.  Experience replay for continual learning.  _arXiv preprint arXiv:1811.11682_ , 2018.


# Refined Inverse Rigging: A Balanced Approach to High-fidelity Blendshape
Animation

Stevo Racković
Department of Mathematics
Instituto Superior Técnico
Lisbon, Portugal
<EMAIL_ADDRESS>
& Cláudia Soares
Department of Computer Sceince
NOVA School of Science and Technology
Caparica, Portugal
& Dušan Jakovetić
Department of Mathematics and Informatics
Faculty of Sciences
Novi Sad, Serbia

(January 2024)

###### Abstract

In this paper, we present an advanced approach to solving the inverse rig
problem in blendshape animation, using high-quality corrective blendshapes.
Our algorithm introduces novel enhancements in three key areas: ensuring high
data fidelity in reconstructed meshes, achieving greater sparsity in weight
distributions, and facilitating smoother frame-to-frame transitions. While the
incorporation of corrective terms is a known practice, our method
differentiates itself by employing a unique combination of $l_{1}$ norm
regularization for sparsity and a temporal smoothness constraint through
roughness penalty, focusing on the sum of second differences in consecutive
frame weights.

A significant innovation in our approach is the temporal decoupling of
blendshapes, which permits simultaneous optimization across entire animation
sequences. This feature sets our work apart from existing methods and
contributes to a more efficient and effective solution. Our algorithm exhibits
a marked improvement in maintaining data fidelity and ensuring smooth frame
transitions when compared to prior approaches that either lack smoothness
regularization or rely solely on linear blendshape models. In addition to
superior mesh resemblance and smoothness, our method offers practical
benefits, including reduced computational complexity and execution time,
achieved through a novel parallelization strategy using clustering methods.

Our results not only advance the state of the art in terms of fidelity,
sparsity, and smoothness in inverse rigging but also introduce significant
efficiency improvements. The source code will be made available upon
acceptance of the paper.

Figure 1: Demonstrating the Efficacy of Temporally Coherent Blendshape
Animation. First row: Our Quartic Smooth method captures the intricate
dynamics of facial expressions by leveraging a sophisticated blendshape rig,
ensuring both high-fidelity mesh reconstruction and smooth temporal
transitions in animation weights. Second row: The Linear Smooth method, as
proposed by Seo et al. (2011), prioritizes temporal smoothness but simplifies
the blendshape model to a linear function, resulting in a trade-off with mesh
accuracy. Third row: The Quartic approach from Racković et al. (2023) achieves
a high degree of mesh fidelity by utilizing a complex blendshape model but
does not account for the smoothness of frame-to-frame transitions, leading to
potential discontinuities. Displayed are selected weight trajectories over 100
animation frames, with two consecutive frames magnified to showcase the mesh
results. The second column employs red shading to illustrate mesh error, and
the third column uses yellow to highlight discrepancies between successive
frames, underscoring the balance between accuracy and smoothness in animation
sequences.

##  1 Introduction

Blendshape animation, a predominant method in animating human faces,
manipulates a 3D facial mesh $\textbf{b}_{0}\in\mathbb{R}^{3n}$ through
interpolation among a predefined set of blendshapes
$\textbf{b}_{1},\ldots,\textbf{b}_{m}\in\mathbb{R}^{3n}$, where $n$ denotes
the number of mesh vertices Lewis et al. (2014). Representing diverse facial
configurations, these blendshapes, when linearly combined with weights
$\textbf{w}=[w_{1},\ldots,w_{m}]$, enable the creation of a wide range of
expressions as

$f_{L}(\textbf{w};\textbf{B})=\textbf{b}_{0}+\textbf{B}\textbf{w},$

with $\textbf{B}\in\mathbb{R}^{3n\times m}$ forming a matrix of blendshape
vectors. Here, the subscript L denotes the linear nature of this blendshape
function. Modern advancements incorporate non-linear corrective terms into
these models, enhancing realism and flexibility Racković et al. (2023a). For
instance, a quartic blendshape function, denoted here as $f_{Q}$, integrates
up to three levels of corrective terms:

$f_{Q}(\textbf{w})=\textbf{Bw}+\sum_{\\{i,j\\}\in\mathcal{P}}w_{i}w_{j}\textbf{b}^{\\{ij\\}}+\sum_{\\{i,j,k\\}\in\mathcal{T}}w_{i}w_{j}w_{k}\textbf{b}^{\\{ijk\\}}+\sum_{\\{i,j,k,l\\}\in\mathcal{Q}}w_{i}w_{j}w_{k}w_{l}\textbf{b}^{\\{ijkl\\}}.$
(1)

Such a detailed correction level is employed in industry-standard solutions
like Metahumans111https://www.unrealengine.com/en-US/eula/mhc.

This paper addresses the inverse rig problem: given a target mesh
$\widehat{\textbf{b}}\in\mathbb{R}^{3n}$, the goal is to find a set of weights
w that accurately approximates the target. Model-based solutions to this
problem leverage the structure of rig functions, utilizing optimization
techniques rather than relying purely on data Çetinaslan (2016); Racković et
al. (2023b). The state-of-the-art (SOTA) model-based approach, as proposed in
Racković et al. (2023a), solves the problem by minimizing the data fidelity of
the model while regularizing for sparsity, while constraining the weight
values to a feasible set considering that the weights only make sense on an
interval $[0,1]$, i.e.,

$\operatorname*{minimize}_{\textbf{0}\leq\textbf{w}\leq\textbf{1}}\frac{1}{2}\|f_{Q}(\textbf{w})-\widehat{\textbf{b}}\|_{2}^{2}+\alpha\textbf{1}^{T}\textbf{w},$
(2)

employing a coordinate descent method. Prior methods Seol et al. (2011); Lewis
and Anjyo (2010); Cetinaslan and Orvalho (2020a), while effective, are
confined to linear blendshape models and thus are less capable of replicating
complex facial dynamics. However, a critical aspect in animation is the
smoothness of frame-to-frame transitions, often overlooked in isolated frame
fitting. This temporal dimension has been explored, for instance, by Seo et
al. (2011), who introduced a smoothness regularization to the optimization
objective:

$\operatorname*{minimize}_{\textbf{0}\leq\textbf{w}\leq\textbf{1}}\|f_{L}(\textbf{w})-\widehat{\textbf{b}}\|_{2}^{2}+\alpha\|\textbf{w}\|_{2}^{2}+\beta\|\textbf{w}-\textbf{v}\|_{2}^{2},$
(3)

where $\textbf{v}\in\mathbb{R}^{m}$ represents the weight vector from the
previous frame, introducing a temporal continuity constraint. While effective,
these methods primarily address linear blendshape models, limiting their
capability to capture the more nuanced facial expressions enabled by non-
linear models.

In this work, we bridge this gap by proposing a novel objective that
harmoniously integrates both the advanced corrective terms of non-linear
blendshape functions and the imperative of temporal smoothness. Our
formulation extends beyond the scope of (3) by concurrently optimizing across
all frames. This holistic approach not only ensures the fidelity of each
individual frame to the target mesh but also guarantees the smoothness of
transitions throughout the animation sequence, by formulating the problem

$\operatorname*{minimize}_{\textbf{0}\leq\textbf{w}\leq\textbf{1}}\sum_{t=1}^{T}\left(\frac{1}{n}\|f_{Q}(\textbf{w}^{(t)})-\widehat{\textbf{b}}^{(t)}\|_{2}^{2}+\frac{\alpha}{m}\|\textbf{w}^{(t)}\|_{1}\right)+\beta\sum_{t=2}^{T-1}\|\textbf{w}^{(t+1)}-2\textbf{w}^{(t)}+\textbf{w}^{(t-1)}\|_{2}^{2},$
(4)

where $T$ is the number of frames in the sequence, and $\beta$ is a
regularization parameter that balances data fidelity and smoothness
constraints. Notably, our method is adaptable to various levels of corrective
terms within the blendshape function, ranging from linear ($f_{L}$) to higher-
order non-linear functions (such as $f_{Q}$).

The generalizability and flexibility of our approach enable it to address a
wider array of animation challenges, including those with complex facial
dynamics.

This approach, through its comprehensive optimization across the entire
animation sequence and its adaptability to various blendshape complexities,
presents a significant step forward in achieving more precise facial
reconstructions and ensuring smoother motion transitions in blendshape
animation.

##  2 Related Work

Facial expressions play a crucial role in human perception and communication,
a significance that extends into the realm of 3D animation. This section
reviews the evolution and current state of facial animation techniques,
particularly focusing on blendshape models, and situates our work within this
landscape.

###  2.1 Anatomically-Based versus Blendshape Models

Anatomically-based face models, as discussed in works like Sifakis et al.
(2005) and Ichim et al. (2017), offer high-fidelity deformation and realistic
perception. Despite their detailed representation, these models pose
challenges in animation control and interpretability. In contrast, blendshape
models, a standard in practical face animation due to their simplicity and
ease of manipulation, have been extensively explored Pighin et al. (1998);
Choe and Ko (2006); Choe et al. (2001). While traditionally sculpted manually,
there have been developments towards automation Deng et al. (2006); Bouaziz et
al. (2013); Moser et al. (2021); Li et al. (2010, 2013); Ribera et al. (2017).
Our work assumes the existence of such a blendshape basis, focusing instead on
the inverse rigging process.

###  2.2 Inverse Rig Problem and Approaches

The inverse rig problem, central to our work, involves generating animations
by adjusting blendshape weights over time. Two main approaches exist: model-
based and data-based. Model-based methods Çetinaslan (2016); Bouaziz et al.
(2013); Racković et al. (2023a) utilize optimization techniques, leveraging
the rig structure, as opposed to data-based methods that rely on regression
models trained on extensive animated data Song et al. (2011); Seol and Lewis
(2014); Holden et al. (2015); Bailey et al. (2020). Our approach falls into
the former category, focusing on a blendshape-based model-based solution.
Previous works in this area typically address least squares problems with
additional regularization for stability Çetinaslan (2016); Cetinaslan and
Orvalho (2020a, b), sparsity Bouaziz et al. (2013); Neumann et al. (2013);
Racković et al. (2023b), or temporal smoothness Tena et al. (2011); Seol et
al. (2012).

###  2.3 Direct Manipulation and Face Segmentation

Related areas include direct manipulation, demanding real-time solutions for
adjusting expressions Lewis and Anjyo (2010); Seo et al. (2011); Cetinaslan
and Orvalho (2020a, b), and face segmentation for animation, where the focus
varies from creating large mesh segments for inverse rig Joshi et al. (2006);
Tena et al. (2011); Kei and Tomoyuki (2012); Reverdy et al. (2015); Marco et
al. (2020); Bailey et al. (2020); Racković et al. (2021) to adding secondary
motion effects with smaller segments Zoss et al. (2020); Neumann et al.
(2013); Wu et al. (2016); Romeo and Schvartzman (2020). While not our primary
focus, the concept of face clustering for distributed rig inversion is also
explored in our work.

###  2.4 Our Contribution

In this paper, we contribute to the field of blendshape animation by
introducing an integrated problem formulation for high-quality rig inversion,
combining 1.) complex corrective blendshape terms that enhance the fidelity of
mesh reconstructions, drawing on the methodologies established in Racković et
al. (2023b, 2023). We implement 2.) sparsity regularization to achieve a lower
cardinality in the weight vectors, which aids in simplifying post-animation
adjustments, as discussed in prior works Seol et al. (2011); Racković et al.
(2023a). Additionally, 3.) acknowledging the critical role of temporal
continuity in animation, we employ a roughness penalty regularization strategy
aimed at ensuring smoother transitions between frames.

##  3 Refined Inverse Rigging Methodology for Blendshape Animation

In this section, we detail our approach for solving the inverse rig problem,
prioritizing high accuracy in mesh reconstruction and ensuring smooth
transitions between blendshape weights across frames. Unlike Racković et al.
(2023a), which focuses on single-frame analysis, our method evaluates the
entire animation sequence, necessitating a matrix-based representation for
weights and other elements in the objective function.

###  3.1 Data Fidelity and Regularization Framework

Consider an animation comprising $T$ frames, denoted as $t=1,\ldots,T$. We
represent the weight vectors for each frame as $\textbf{w}^{(t)}$ and assemble
these into a weight matrix
$\textbf{W}=[\textbf{w}^{(1)},\ldots,\textbf{w}^{(T)}]\in\mathbb{R}^{m\times
T}$. The rig function for the entire sequence is then expressed as:

$\begin{split}f(\textbf{W})=&\textbf{BW}+\sum_{t=1}^{T}\sum_{\\{i,j\\}\in\mathcal{P}}w_{i}^{(t)}w_{j}^{(t)}\text{diag}(\textbf{b}^{\\{ij\\}})\textbf{E}^{(t)}+\sum_{\\{i,j,k\\}\in\mathcal{T}}w_{i}^{(t)}w_{j}^{(t)}w_{k}^{(t)}\text{diag}(\textbf{b}^{\\{ijk\\}})\textbf{E}^{(t)}\\\
&+\sum_{\\{i,j,k,l\\}\in\mathcal{Q}}w_{i}^{(t)}w_{j}^{(t)}w_{k}^{(t)}w_{l}^{(t)}\text{diag}(\textbf{b}^{\\{ijkl\\}})\textbf{E}^{(t)},\end{split}$
(5)

where $\textbf{E}^{(t)}$ is a matrix of zeros with the $t^{th}$ column
consisting of ones. Similarly, we define the target meshes matrix
$\hat{\textbf{B}}=[\hat{\textbf{b}}^{(1)},\ldots,\hat{\textbf{b}}^{(T)}]\in\mathbb{R}^{n\times
T}$, with $\hat{\textbf{b}}^{(t)}$ being the target mesh for frame $t$. We
propose the objective that consists of three terms:

$\operatorname*{minimize}_{0\leq\textbf{W}\leq 1}E_{\text{df}}+\alpha
E_{\text{sr}}+\beta E_{\text{tsr}},$ (6)

where $E_{\text{df}}$ stands for a data fidelity term, i.e., a difference
between the estimated mesh and the target mesh in vertex space,
$E_{\text{sr}}$ stands for the sparsity regularization forcing the cardinality
of the weights to be low, and $E_{\text{tsr}}$ is temporal smoothness
regularizer, forcing the weights in the consecutive frames to have similar
values; $\alpha,\beta\geq 0$ are corresponding regularization weights,
dictating the importance of each term. Let us observe each of these terms
individually.

#### Data Fidelity

The data fidelity term, along with the sparsity regularization term, aligns
with the formulation in (2) but is now adapted to a matrix context to handle
the entire animation sequence. Specifically, the data fidelity term is defined
as:

$E_{\text{df}}=\frac{1}{n}\|f(\textbf{W})-\hat{\textbf{B}}\|_{F}^{2},$ (7)

where $\|\cdot\|_{F}$ denotes the Frobenius norm, measuring the discrepancy
between the estimated mesh sequence $f(\textbf{W})$ and the target mesh
sequence $\hat{\textbf{B}}$.

We employ a coordinate descent approach to minimize this term, focusing on a
single blendshape controller, denoted as $e$, across the temporal dimension.
In this context, we reformulate (7) into a quadratic expression:

$E_{\text{df}}=\frac{1}{n}\textbf{W}_{e}^{T}\Phi\textbf{W}_{e}+\frac{2}{n}\textbf{W}_{e}^{T}\Theta,$

where $\textbf{W}_{e}$ represents the weights corresponding to controller $e$
over all frames. The matrices $\Phi$ and $\Theta$ are constructed as follows:

Matrix
$\Phi=\text{diag}([\phi^{(1)T}\phi^{(1)},\ldots,\phi^{(T)T}\phi^{(T)}])$
encapsulates all the terms interacting with the quadratic terms of the
objective. Each $\phi^{(t)}$ represents the contribution of the quadratic term
of the blendshape controller $e$ at frame $t$, computed as:

$\phi^{(t)}_{i}=B_{ie}+\sum_{j\in\mathcal{P}(e)}w_{j}^{(t)}\textbf{b}^{\\{je\\}}_{i}+\sum_{\\{j,k\\}\in\mathcal{T}(e)}w_{j}^{(t)}w_{k}^{(t)}\textbf{b}^{\\{jke\\}}_{i}+\sum_{\\{j,k,l\\}\in\mathcal{Q}(e)}w_{j}^{(t)}w_{k}^{(t)}w_{l}^{(t)}\textbf{b}^{\\{jkle\\}}_{i},$
(8)

which considers not only the direct influence of controller $e$ but also its
interaction with other controllers in the corrective terms.

Matrix $\Theta=[\phi^{(1)T}\psi^{(1)},\ldots,\phi^{(T)T}\psi^{(T)}]^{T}$
represents the linear interaction terms, where each $\psi^{(t)}$ is given by:

$\begin{split}\psi_{i}^{(t)}&=\sum_{j\neq
e}w_{j}^{(t)}B_{ij}+\sum_{\\{j,k\\}\in\mathcal{P}}w_{j}^{(t)}w_{k}^{(t)}\textbf{b}^{\\{jk\\}}_{i}+\sum_{\\{j,k,l\\}\in\mathcal{T}}w_{j}^{(t)}w_{k}^{(t)}w_{l}^{(t)}\textbf{b}^{\\{jkl\\}}_{i}\\\
&+\sum_{\\{j,k,l,h\\}\in\mathcal{Q}}w_{j}^{(t)}w_{k}^{(t)}w_{l}^{(t)}w_{h}^{(t)}\textbf{b}^{\\{jklh\\}}_{i}-\hat{\textbf{b}}^{(t)}_{i}.\end{split}$
(9)

This accounts for the contributions of all other blendshape controllers, as
well as the deviation from the target mesh $\hat{\textbf{b}}^{(t)}$ at frame
$t$.

Through this formulation, the data fidelity term effectively quantifies and
minimizes the difference between the animated mesh sequence and the target
sequence, thereby ensuring high accuracy in replicating desired facial
expressions over time.

#### Sparsity Regularization

In our approach, the sparsity regularization term is critical for ensuring
that the animation remains computationally efficient and interpretable. It is
defined as the normalized sum of all blendshape weights across the entire
animation sequence. Mathematically, this is represented as:

$E_{\text{sr}}=\frac{1}{m}\sum_{i=1}^{m}\sum_{t=1}^{T}w_{i}^{(t)}=\frac{1}{m}\textbf{1}^{T}\textbf{W}\textbf{1},$

where 1 denotes a vector of ones of appropriate dimension. This formulation
encourages the model to use as few active blendshapes as possible at each
frame, leading to a sparser and more interpretable set of blendshape weights.
Importantly, due to the non-negativity constraints on the weights,
$E_{\text{sr}}$ is guaranteed to be a non-negative term.

The integration of sparsity regularization into the optimization process
serves multiple purposes: it not only enhances computational efficiency by
reducing the number of active blendshapes but also simplifies the task of
manual adjustments or further processing by animators. By penalizing the sum
of the weights, the model naturally gravitates towards solutions where fewer
blendshapes are used to achieve the desired facial expressions, thereby
promoting a more streamlined and manageable animation process.

#### Temporal Smoothness Regularization

A key aspect of realistic animation is the smoothness of transitions between
frames. To achieve this, we incorporate a roughness penalty function into our
regularization framework. This function penalizes the squared differences
between adjacent weight values across frames, effectively encouraging temporal
continuity in the animation. The temporal smoothness regularization term is
formulated as follows:

$E_{\text{tsr}}=\sum_{t=1}^{T-2}\sum_{i=1}^{m}|w_{i}^{(t)}-2w_{i}^{(t+1)}+w_{i}^{(t+2)}|^{2}=\sum_{i}\textbf{W}_{i}^{T}\textbf{F}\textbf{W}_{i},$
(10)

where $\textbf{W}_{i}$ denotes the weight vector for the $i^{th}$ blendshape
across all frames. The matrix F is a pentadiagonal matrix defined as:

$\textbf{F}=\begin{bmatrix}1&-2&1&\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}0&\cdots&\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>-2&5&-4&1&\cdots&\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>1&-4&6&-4&\ddots&\vdots\\\
\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>\vdots&\ddots&\ddots&\ddots&\ddots&-2\\\
\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}0&\cdots&\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}0&1&-2&1\end{bmatrix},$
(11)

with the pattern designed to penalize the roughness or abrupt changes in the
weight vectors between consecutive frames.

This regularization term thus plays a crucial role in ensuring the naturalness
and fluidity of the generated animation. By minimizing $E_{\text{tsr}}$, our
model actively works to smooth out the transitions, leading to more lifelike
and appealing animations that closely mimic natural human expressions over
time.

#### Final formulation.

To synthesize the various aspects of our method into a coherent optimization
framework, we formulate a comprehensive objective function that balances the
need for data fidelity, sparsity, and temporal smoothness. This objective, to
be minimized with respect to each blendshape controller $e$, encapsulates the
essence of our approach:

$\operatorname*{minimize}_{\textbf{0}\leq\textbf{W}_{e}\leq\textbf{1}}\textbf{W}_{e}^{T}\left(\frac{1}{n}\Phi+\beta\textbf{F}\right)\textbf{W}_{e}+2\textbf{W}_{e}^{T}\left(\frac{1}{n}\Theta+\frac{\alpha}{2m}\textbf{1}\right).$
(12)

In this equation, $\Phi$ and $\Theta$ are matrices derived from the data
fidelity term, encoding the relationship between the blendshape weights and
the target meshes. The matrix F, arising from the temporal smoothness
regularization term, ensures that changes in the blendshape weights are
gradual over the sequence of frames. Lastly, the term involving 1, originating
from the sparsity regularization, promotes solutions with fewer active
blendshapes, thereby aiding in interpretability and computational efficiency.

This carefully constructed objective function is central to our method,
guiding the optimization process towards solutions that are not only accurate
in reproducing the target facial expressions but also efficient and smooth
over time. By balancing these critical aspects, our approach advances the
state-of-the-art in blendshape animation, particularly in scenarios requiring
high fidelity and natural motion dynamics.

###  3.2 Clustering Approach for Computational Efficiency

Figure 2: Top row: The trade-off between Reconstruction Error ($E_{R}$) and
Density ($E_{D}$) (left), and Inter-Density ($E_{ID}$) (right), across
different clustering approaches, with annotations indicating the chosen number
of clusters ($K$). Bottom row: Visualization of clusters obtained using $RSJD$
with $K=29$ (left) and $RSJD_{A}$ with $K=13$ (right). In addition to the mesh
clusters, a bipartite graph representation is shown, using the same color
coding, where the left partition denotes mesh vertices, and the right
partition signifies the blendshape indices assigned to each cluster.

Given the localized nature of facial features, a significant number of
vertices in the human face may have limited influence on the majority of
blendshape weights. Our approach incorporates a face clustering strategy to
exploit this characteristic for enhanced computational efficiency. This
strategy allows for a parallelized and potentially more regularized solution
to the inverse rigging problem Seol et al. (2011); Racković et al. (2021,
2023).

We adopt the clustering methodologies proposed in Racković et al. (2021)
(denoted as $RSJD$) and Racković et al. (2023) (denoted as $RSJD_{A}$). These
methods are particularly suitable as they generate clusters in a model-based
manner. However, our approach remains flexible enough to be applicable with
other clustering techniques, provided they meaningfully separate mesh segments
and blendshape controllers.

The core idea is to treat each cluster as an independent model, thereby
decomposing the overall inverse rigging problem (12) into a series of smaller
subproblems. Each subproblem focuses on a specific subset of vertices and
blendshapes relevant to its cluster. This subdivision not only facilitates
parallel processing but also potentially introduces additional regularization
opportunities, leading to a more efficient solution overall.

Following the recommendations in Racković et al. (2023), we generate several
instantiations of each clustering approach. The optimal configuration is
selected based on a balance between Reconstruction Error and Density in the
resulting clustering graphs, as illustrated in Figure 2 (top). Optimal cluster
numbers, such as $K=29$ for $RSJD$ and $K=13$ for $RSJD_{A}$, are determined
to minimize the model size while retaining essential information. The
corresponding mesh clusters and their bipartite graph representations are
detailed in Figure 2 (bottom), showcasing the effective distribution of
vertices and blendshape indices across the identified clusters.

###  3.3 Optimization Strategy

The optimization strategy used in solving the proposed objective is coordinate
descent, a well established optimization technique that is guaranteed to
produce monotonically non-increasing costs Luo and Tseng (1992); Wright
(2015). With coordinate descent, a single component of the problem is visited
at a time, and the objective is minimized in it, while keeping the other
values fixed, as we do in (12), observing only a single controller $e$ as a
variable. This decoupling in components allows us to simplify otherwise non-
convex problem (4), into a constrained quadratic program, for which fast
solutions and implementations are readily available Moré and Toraldo (1989);
Virtanen et al. (2020). Coordinate descent is particularly suitable for for
the application in inverse rigging, since estimating one blendshape weight at
a time will help avoid the simultaneous activation of blendshapes with
canceling effects, i.e., moving the corner of the mouth up and down at the
same time Seol et al. (2011).

An important aspect of coordinate descent is the order of component updates.
Even though the convergence guaranties hold with an arbitrary update order
Wright (2015), in practice a poor choice can lead to slower convergence and a
bad local minima. This problem was also studied in blendshape animation
literature Seol et al. (2011); Racković et al. (2023a); Hyde et al. (2021). We
follow the stance of Seol et al. (2011), ordering the blendshapes by their
overall magnitude of deformation in the mesh, in line with artist intuition of
initially setting the more drastic weights before focusing on fine details.

##  4 Quantitative Analysis of Animation Techniques

This section presents the comprehensive evaluation of our proposed blendshape
animation methodology. We first describe the data and benchmark models used in
our study, followed by a detailed analysis of the results. The performance of
our method is compared against established benchmarks to demonstrate its
efficacy in achieving high-fidelity, smooth animations. The impact of
different parameters on the results is also examined to provide insights into
the behavior of our approach under various conditions.

Figure 3: Parametric Analysis of Animation Metrics During Training, comparing
our method (Quartic Smooth) with benchmarks. Top row: This graph illustrates
the interplay between blendshape cardinality and maximum mesh error under
various animation approaches. The color coding denotes different levels of the
sparsity regularizer, $\alpha$, while individual points represent a spectrum
of the smoothness parameter, $\beta$, converging at $\beta=0$ indicated by the
solid gray line. The horizontal dotted line marks the cardinality of the
actual animation data used as the ground truth, whereas the vertical dotted
line indicates the mesh error that would result if no blendshapes were
activated (all weights at $0$). Bottom row: Here, we chart the roughness
penalty corresponding to the varying values of $\beta$ along the x-axis. The
color scheme is consistent with the top graph, linked to the $\alpha$ values.
The horizontal dotted line represents the benchmark roughness penalty derived
from the ground-truth animation data. Figure 4: Comparative Analysis of
Training Metrics Across Different Parameterizations and Methodologies. In this
Figure we show how applying clustering on top of our approach affects the
overall results.

###  4.1 Dataset Characteristics and Comparative Benchmarks

We selected the Metahuman character Jesse to evaluate our algorithm. This
choice is motivated by the high-quality and realistic nature of Metahuman
blendshape models, coupled with their accessibility as publicly available
resources. The Jesse model is equipped with $m=80$ base blendshapes and over
$400$ corrective terms, encompassing the first, second, and third levels of
correction. These elements make it an ideal candidate for demonstrating the
nuanced capabilities of our method in handling complex facial animations.

Originally, the meshes contain $24000$ vertices. To focus on the facial
region, which is our primary interest, we have excluded vertices on the neck
and an inactive area at the back of the skull. This results in a subsampled
mesh comprising $n=10000$ vertices. The model is animated to generate a
reference motion, which comprises $80$ frames for the training set and $100$
frames for the test set. This division allows for a robust assessment of our
method’s performance both in learning and generalization.

In our results section, we refer to our proposed method as Quartic Smooth.
This designation emphasizes its unique aspects: the inclusion of corrective
terms using a quartic blendshape function for enhanced realism, and the
integration of smoothness regularization, setting it apart from previous
model-based solutions. These characteristics are pivotal in achieving more
lifelike and dynamically consistent facial animations.

Our benchmarks include two prior works for comparative analysis. The first is
the method described in Seo et al. (2011), named Linear Smooth, which
incorporates smoothness regularization but does not include corrective terms.
This method optimizes the objective defined in (3). The second benchmark is
the Quartic algorithm from Racković et al. (2023), which considers corrective
terms for better mesh reconstruction but treats each frame independently,
overlooking temporal continuity. Quartic solves the objective in (2). Notably,
while both benchmarks include weight regularization, they differ in approach:
Quartic uses an $l_{1}$ norm, promoting sparsity, whereas Linear Smooth
employs a squared $l_{2}$ norm, penalizing large activations without directly
reducing the number of active components.

Additionally, we explore an extension of our proposed algorithm that
incorporates the face clustering technique detailed in Section 3.2. We examine
two instances of clustering: Clustered $RSJD$ 29 and Clustered $RSJD_{A}$ 13,
named after the clustering methods and the chosen number of clusters ($K$).
These cases help to assess the impact of facial clustering on the performance
and efficiency of our approach.

###  4.2 Analytical Performance Evaluation

All the considered methods have a weight regularization hyperparameter
$\alpha$ that should be selected before comparing the results in more details.
Additionally, our proposed method, Quartic Smooth, as well as the benchmark
Linear Smooth, have the smoothness regularization hyperparameter $\beta$. To
select the appropriate values, each method is cross-validated with a wide
choice of values, as presented in Fig. 3 and 4. In general these results draw
an expected trade-of curve between cardinality and mesh error, with the
exception of the Linear Smooth — in this case a favorable decrease of the
cardinality is never achieved, since $l_{2}$ norm in the objective tends to
keep small positive values rather than setting them to zero. Optimal values
for the parameters are selected so that all three presented metrics are
minimized, and the final selection that is used for further tests is listed in
Tab. 1.

|  Quartic Smooth |  Linear Smooth |  Quartic |  Clustered $RSJD$ 29 |  Clustered $RSJD_{A}$ 13
---|---|---|---|---|---
$\alpha$ | $0.0078$ | $0.01$ | $0.9$ | $1$ | $0.8$
$\beta$ | $1$ | $0$ | / | $10$ | $1$
Table 1: Final parameter values for each method. Figure 5: Results over the test set with the selected hyperparameter values corresponding to Table 1. The execution time for the clustered approach is presented in solid and shaded — solid color indicates the execution time of the slowest cluster, as that is the cost when solving the problem in parallel, while shaded bar shows the time of solving the clusters sequentially. |  Max Mesh Error |  Mean Mesh Error | Cardinality |  $l_{1}$ Norm |  Roughness Penalty |  Execution Time
---|---|---|---|---|---|---
Quartic Smooth | $.101$ | $.019$ | $56.3$ | $9.431$ | $7.2e^{-5}$ | $2.16$
Linear Smooth | $.257$ | $.051$ | $68.6$ | $9.649$ | $4.2e^{-4}$ | $0.02$
Quartic | $.086$ | $.016$ | $49.7$ | $9.818$ | $1.1e^{-3}$ | $14.4$
$RSJD$ | $.387$ | $.093$ | $58.2$ | $8.695$ | $2.8e^{-4}$ | $0.15$
$RSJD_{A}$ | $.433$ | $.115$ | $54.0$ | $5.500$ | $1.5e^{-4}$ | $0.44$
Table 2: Average metrics values for test results (see also Figure 5).

By examining Tab. 2, we can compare the performance of the various methods
(Quartic Smooth, Linear Smooth, Quartic, RSJD, and $RSJD_{A}$) across several
key metrics: Max Mesh Error, Mean Mesh Error, Cardinality, $l_{1}$ Norm,
Roughness Penalty, and Execution Time.

#### Max Mesh Error and Mean Mesh Error:

Quartic exhibits the lowest Max and Mean Mesh Errors ($.086$ and $.016$,
respectively), indicating its superior ability to closely match the target
mesh in the worst-case and on average. This suggests high accuracy in mesh
reconstruction. Quartic Smooth also performs well, with slightly higher
errors, but still significantly better than Linear Smooth, RSJD, and
$RSJD_{A}$.

#### Cardinality:

The Cardinality, which indicates the number of active blendshapes, is lowest
for Quartic (49.7), suggesting it is the most efficient in terms of blendshape
usage. Quartic Smooth has a moderately higher cardinality (56.3) compared to
Quartic, but lower than Linear Smooth and RSJD.

####  $l_{1}$ Norm:

The $l_{1}$ Norm, reflecting the sum of absolute weights, is highest for
Quartic (9.818) and lowest for $RSJD_{A}$ (5.500). Higher values indicate more
weight is being used overall, which can suggest more intense or complex facial
expressions.

#### Roughness Penalty:

Quartic Smooth has the lowest Roughness Penalty ($7.2e^{-5}$), highlighting
its effectiveness in ensuring smooth frame-to-frame transitions. Quartic,
despite its accuracy, has a higher roughness penalty ($1.1e^{-3}$), implying
less smoothness in transitions compared to Quartic Smooth.

#### Execution Time:

Linear Smooth is the fastest ($.02$ seconds), which is expected given its less
complex nature (lacking corrective terms). Quartic is the slowest (14.4
seconds), possibly due to the computational overhead of handling corrective
terms. Quartic Smooth strikes a balance between complexity and speed (2.16
seconds), offering a more efficient solution than Quartic while maintaining
high accuracy and smoothness. This advantage over Quartic is due to the
parallelizable structure given by the clustering.

Quartic Smooth demonstrates a well-balanced performance across accuracy,
efficiency, and smoothness, making it a robust choice for high-quality and
realistic blendshape animation. Quartic excels in mesh error metrics but at
the cost of higher roughness and longer execution time, while Linear Smooth
offers the fastest execution with compromises in accuracy and smoothness. The
clustered approaches (RSJD and $RSJD_{A}$) offer varying trade-offs, with
$RSJD_{A}$ showing lower execution times and roughness penalties but at the
cost of higher mesh errors.

Figure 3 shows graphs evaluating three approaches: Quartic Smooth, Linear
Smooth, and Quartic. Each graph shows the trade-off between cardinality and
maximum mesh error on the top row and the roughness penalty across different
smoothness parameter $\beta$ values on the bottom row.

#### Quartic Smooth:

The top graph suggests that as the sparsity regularizer $\alpha$ increases,
both the cardinality and the maximum mesh error generally increase. This
indicates that higher sparsity (achieved by a larger $\alpha$) comes at the
cost of accuracy (higher mesh error). In the bottom graph, the roughness
penalty seems relatively stable across various values of $\beta$, suggesting
that the smoothness regularization is not significantly impacting the
roughness penalty in this approach.

#### Linear Smooth:

The top graph shows that the maximum mesh error and cardinality do not
significantly change with different values of $\alpha$, indicating a potential
plateau in the trade-off, where increasing sparsity does not impact the mesh
error substantially. The bottom graph shows variations in the roughness
penalty across different values of $\beta$, but the changes are not dramatic,
implying that the model’s smoothness is not highly sensitive to this parameter
within the tested range.

#### Quartic:

The top graph indicates that increasing $\alpha$ leads to a decrease in the
maximum mesh error but at the expense of a rapid increase in cardinality,
suggesting a strong influence of the sparsity regularizer on model complexity.
There’s no bottom graph for roughness as the Quartic model does not seem to
include the smoothness parameter $\beta$.

The Quartic Smooth approach seems to strike a balance between maintaining mesh
accuracy and controlling the model complexity (cardinality), especially when
compared to the Linear Smooth and Quartic approaches. The roughness penalties
across all methods are relatively low, indicating smooth transitions, with
Quartic Smooth showing a slight advantage. However, one must consider the
trade-offs between accuracy, sparsity, and temporal smoothness when selecting
the appropriate parameters for each method.

Figure 4 shows a comparative analysis of three different blendshape animation
approaches: Quartic Smooth, Clustered RSJD 29, and Clustered RSJD_A 13. The
analysis is based on the evolution of specific metrics over the training set
while varying two parameters: the sparsity regularizer ($\alpha$) and the
smoothness parameter ($\beta$).

The Quartic Smooth approach in our experiments offers a favorable balance
between animation accuracy and sparsity, with smooth temporal transitions as
indicated by the low roughness penalties. The clustered approaches may provide
computational benefits, but potentially at the cost of increased max mesh
error, especially for higher sparsity levels. The choice between these methods
may depend on the specific requirements of an animation project, such as the
necessity for real-time performance (favoring clustered methods) versus the
need for high-fidelity animations (favoring Quartic Smooth).

Figure 6: Resulting meshes for selected frames (rows), and for each method
(columns). Stronger red tones indicate higher mesh error.

##  5 Conclusion

We have introduced Quartic Smooth, an advanced model-based approach for
blendshape animation that adeptly balances accuracy, sparsity, and temporal
smoothness. Our method demonstrates a marked improvement in mesh fidelity over
traditional linear models and offers a flexible framework for both high-
quality and real-time applications. Our findings illustrate Quartic Smooth’s
superior performance in creating realistic facial animations with reduced
computational overhead, particularly when compared to existing linear and non-
linear approaches. The introduction of face clustering techniques further
augments computational efficiency, opening possibilities for real-time
animation processing. Future work will explore the optimization of these
techniques and the integration of machine learning to refine parameter
selection. With its contribution to the realistic and efficient inverse
rigging of facial animations, Quartic Smooth is poised to influence future
developments in character animation within the computer graphics community.

## References

  * Seo et al. [2011] Jaewoo Seo, Geoffrey Irving, J. P. Lewis, and Junyong Noh.  Compression and direct manipulation of complex blendshape models.  _ACM Trans. Graph._ , 30(6):1–10, dec 2011.
  * Racković et al. [2023] Stevo Racković, Cláudia Soares, and Dušan Jakovetić.  Distributed solution of the blendshape rig inversion problem.  In _SIGGRAPH Asia 2023 Technical Communications_ , SA ’23, New York, NY, USA, 2023. Association for Computing Machinery.  ISBN 9798400703140.  doi:10.1145/3610543.3626166.  URL https://doi.org/10.1145/3610543.3626166.
  * Lewis et al. [2014] John P. Lewis, Ken Anjyo, Taehyun Rhee, Mengjie Zhang, Frederic H. Pighin, and Zhigang Deng.  Practice and theory of blendshape facial models.  _Eurographics (State of the Art Reports)_ , 1(8):2, 2014.
  * Racković et al. [2023a] Stevo Racković, Cláudia Soares, Dušan Jakovetić, and Zoranka Desnica.  High-fidelity interpretable inverse rig: An accurate and sparse solution optimizing the quartic blendshape model.  _arXiv preprint arXiv:2302.04820_ , 2023a.
  * Çetinaslan [2016] Cumhur Ozan Çetinaslan.  _Position Manipulation Techniques for Facial Animation_.  PhD thesis, Faculdade de Ciencias da Universidade do Porto, 2016.
  * Racković et al. [2023b] Stevo Racković, Cláudia Soares, Dušan Jakovetić, and Zoranka Desnica.  A majorization–minimization-based method for nonconvex inverse rig problems in facial animation: algorithm derivation.  _Optimization Letters_ , pages 1–15, 2023b.
  * Seol et al. [2011] Yeongho Seol, Jaewoo Seo, Paul Hyunjin Kim, J. P. Lewis, and Junyong Noh.  Artist friendly facial animation retargeting.  In _SIGGRAPH Asia 2011_ , 2011.
  * Lewis and Anjyo [2010] J.P. Lewis and Ken-ichi Anjyo.  Direct manipulation blendshapes.  _IEEE Computer Graphics and Applications_ , 30(04):42–50, 2010.
  * Cetinaslan and Orvalho [2020a] Ozan Cetinaslan and Veronica Orvalho.  Sketching manipulators for localized blendshape editing.  _Graphical Models_ , 108:101059, 2020a.
  * Sifakis et al. [2005] Eftychios Sifakis, Igor Neverov, and Ronald Fedkiw.  Automatic determination of facial muscle activations from sparse motion capture marker data.  In _ACM SIGGRAPH 2005 Papers_ , page 417–425. Association for Computing Machinery, 2005.
  * Ichim et al. [2017] Alexandru-Eugen Ichim, Petr Kadleček, Ladislav Kavan, and Mark Pauly.  Phace: Physics-based face modeling and animation.  _ACM Transactions on Graphics (TOG)_ , 36(4):1–14, 2017.
  * Pighin et al. [1998] Frédéric H. Pighin, Jamie Hecker, Dani Lischinski, Richard Szeliski, and D. Salesin.  Synthesizing realistic facial expressions from photographs.  _Proceedings of the 25th annual conference on Computer graphics and interactive techniques_ , 1998.
  * Choe and Ko [2006] Byoungwon Choe and Hyeong-Seok Ko.  Analysis and synthesis of facial expressions with hand-generated muscle actuation basis.  In _ACM SIGGRAPH 2006_. 2006.
  * Choe et al. [2001] Byoungwon Choe, Hanook Lee, and Hyeong-Seok Ko.  Performance-driven muscle-based facial animation.  _The Journal of Visualization and Computer Animation_ , 12(2):67–79, 2001.
  * Deng et al. [2006] Zhigang Deng, Pei-Ying Chiang, Pamela Fox, and Ulrich Neumann.  Animating blendshape faces by cross-mapping motion capture data.  In _Proceedings of the 2006 Symposium on Interactive 3D Graphics and Games_ , page 43–48. Association for Computing Machinery, 2006.
  * Bouaziz et al. [2013] Sofien Bouaziz, Yangang Wang, and Mark Pauly.  Online modeling for realtime facial animation.  _ACM Trans. Graph._ , 32(4), 2013.
  * Moser et al. [2021] Lucio Moser, Chinyu Chien, Mark Williams, Jose Serra, Darren Hendler, and Doug Roble.  Semi-supervised video-driven facial animation transfer for production.  _ACM Transactions on Graphics (TOG)_ , 40(6):1–18, 2021.
  * Li et al. [2010] Hao Li, Thibaut Weise, and Mark Pauly.  Example-based facial rigging.  _ACM Trans. Graph._ , 29(4):1–6, 2010.
  * Li et al. [2013] Hao Li, Jihun Yu, Yuting Ye, and Chris Bregler.  Realtime facial animation with on-the-fly correctives.  _ACM Trans. Graph._ , 32(4):42–1, 2013.
  * Ribera et al. [2017] Roger Blanco i Ribera, Eduard Zell, J. P. Lewis, Junyong Noh, and Mario Botsch.  Facial retargeting with automatic range of motion alignment.  _ACM Trans. Graph._ , 36(4), 2017.
  * Song et al. [2011] Jaewon Song, Byungkuk Choi, Yeongho Seol, and Jun yong Noh.  Characteristic facial retargeting.  _Computer Animation and Virtual Worlds_ , 22, 2011.
  * Seol and Lewis [2014] Yeongho Seol and J. P. Lewis.  Tuning facial animation in a mocap pipeline.  In _ACM SIGGRAPH 2014 Talks_ , SIGGRAPH ’14. Association for Computing Machinery, 2014.
  * Holden et al. [2015] Daniel Holden, Jun Saito, and Taku Komura.  Learning an inverse rig mapping for character animation.  In _Proceedings of the 14th ACM SIGGRAPH/Eurographics Symposium on Computer Animation_ , pages 165–173, 2015.
  * Bailey et al. [2020] Stephen W. Bailey, Dalton Omens, Paul Dilorenzo, and James F. O’Brien.  Fast and deep facial deformations.  _ACM Trans. Graph._ , 39(4), 2020.
  * Cetinaslan and Orvalho [2020b] Ozan Cetinaslan and Veronica Orvalho.  Stabilized blendshape editing using localized jacobian transpose descent.  _Graphical Models_ , 112:101091, 2020b.
  * Neumann et al. [2013] Thomas Neumann, Kiran Varanasi, Stephan Wenger, Markus Wacker, Marcus Magnor, and Christian Theobalt.  Sparse localized deformation components.  _ACM Trans. Graph._ , 32(6):1–10, 2013.
  * Tena et al. [2011] J. Rafael Tena, Fernando De la Torre, and Iain Matthews.  Interactive region-based linear 3D face models.  In _ACM SIGGRAPH 2011 Papers_ , 2011.
  * Seol et al. [2012] Yeongho Seol, J.P. Lewis, Jaewoo Seo, Byungkuk Choi, Ken Anjyo, and Junyong Noh.  Spacetime expression cloning for blendshapes.  _ACM Trans. Graph._ , 31(2), 2012.
  * Joshi et al. [2006] Pushkar Joshi, Wen C Tien, Mathieu Desbrun, and Frédéric Pighin.  Learning controls for blend shape based realistic facial animation.  In _ACM SIGGRAPH 2006_. 2006.
  * Kei and Tomoyuki [2012] Hirose Kei and Higuchi Tomoyuki.  Creating facial animation of characters via mocap data.  _Journal of Applied Statistics_ , 39(12):2583–2597, 2012.
  * Reverdy et al. [2015] Clément Reverdy, Sylvie Gibet, and Caroline Larboulette.  Optimal marker set for motion capture of dynamical facial expressions.  In _Proceedings of the 8th ACM SIGGRAPH Conference on Motion in Games_ , page 31–36. Association for Computing Machinery, 2015.
  * Marco et al. [2020] Fratarcangeli Marco, Bradley Derek, A. Gruber, Zoss Gaspard, and Beeler Thabo.  Fast nonlinear least squares optimization of large-scale semi-sparse problems.  _Computer Graphics Forum_ , 39, 2020.
  * Racković et al. [2021] Stevo Racković, Cláudia Soares, Dušan Jakovetić, Zoranka Desnica, and Relja Ljubobratović.  Clustering of the blendshape facial model.  In _2021 29th European Signal Processing Conference (EUSIPCO)_ , pages 1556–1560. IEEE, 2021.
  * Zoss et al. [2020] Gaspard Zoss, Eftychios Sifakis, Markus Gross, Thabo Beeler, and Derek Bradley.  Data-driven extraction and composition of secondary dynamics in facial performance capture.  _ACM Transactions on Graphics (TOG)_ , 39(4):107–1, 2020.
  * Wu et al. [2016] Chenglei Wu, Derek Bradley, Markus Gross, and Thabo Beeler.  An anatomically-constrained local deformation model for monocular face capture.  _ACM Trans. Graph._ , 35(4):1–12, 2016.
  * Romeo and Schvartzman [2020] Marco Romeo and S. Schvartzman.  Data-driven facial simulation.  In _Computer Graphics Forum_ , volume 39, pages 513–526. Wiley Online Library, 2020.
  * Luo and Tseng [1992] Zhi-Quan Luo and Paul Tseng.  On the convergence of the coordinate descent method for convex differentiable minimization.  _Journal of Optimization Theory and Applications_ , 72(1):7–35, 1992.
  * Wright [2015] Stephen J Wright.  Coordinate descent algorithms.  _Mathematical programming_ , 151(1):3–34, 2015.
  * Moré and Toraldo [1989] Jorge J Moré and Gerardo Toraldo.  Algorithms for bound constrained quadratic programming problems.  _Numerische Mathematik_ , 55(4):377–400, 1989.
  * Virtanen et al. [2020] Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al.  Scipy 1.0: fundamental algorithms for scientific computing in python.  _Nature methods_ , 17(3):261–272, 2020.
  * Hyde et al. [2021] David AB Hyde, Michael Bao, and Ronald Fedkiw.  On obtaining sparse semantic solutions for inverse problems, control, and neural network training.  _Journal of Computational Physics_ , 443:110498, 2021.