††thanks: Paper prepared for 2018 annual meeting of the American Political
Science Association, Boston, USA. Panel: _How Social Media is Disrupting The
Political Status Quo Around the World_

# Volatility in the Issue Attention Economy

Chico Q. Camargo  Oxford Internet Institute, University of Oxford, UK  Scott
A. Hale  Oxford Internet Institute, University of Oxford, UK  Peter John
Department of Political Economy, King’s College London, UK  Helen Z. Margetts
Oxford Internet Institute, University of Oxford, UK

###### Abstract

Recent election surprises and regime changes have left the impression that
politics has become more fast-moving and unstable. While modern politics does
seem more volatile, there is little systematic evidence to support this claim.
This paper seeks to address this gap in knowledge by reporting data over the
last seventy years using public opinion polls and traditional media data from
the UK and Germany. These countries are good cases to study because both have
experienced considerable changes in electoral behaviour and have new political
parties during the time period studied. We measure volatility in public
opinion and in media coverage using approaches from information theory,
tracking the change in word-use patterns across over 700,000 articles. Our
preliminary analysis suggests an increase in the number of opinion issues over
time and a growth in lack of predictability of the media series from the
1970s.

## Introduction

The second decade of the twenty-first century has witnessed multiple electoral
shocks, policy surprises, and regime changes, which appear to have made
political life more fast-moving, unpredictable, and unstable. Political
turbulence would seem to sum up the character of the new politicsMargetts:aa ;
yet there has been no systematic investigation into the nature of this
volatility. This paper seeks to address this gap in knowledge by first
defining the concept, then arguing that there are good reasons to believe that
“political volatility” has increased in recent years, and finally proposing
two information theoretic approaches to quantify the level of volatility in
the “issue attention” economy. We evaluate these measures of political
volatility by analyzing public opinion polls and traditional media data from
the last fifty years in the UK and Germany.

## Defining political volatility

In this section, we define political volatility, as the term is used in
different ways according to the particular discipline that uses the concept
and the domain under study. We need a concept that comports with how we see
complex political systems operate, and a corresponding way of measuring it
that represents the concept in practice and is capable of being used with a
variety of data sources.

The word volatility is used across the natural sciences in mainly two ways,
describing either that which is light, and ready to fly or turn into vapour,
or that which is likely to change suddenly and unexpectedly. In chemistry and
physics, volatile compounds are those that evaporate easily, and in computer
science volatile memory is easier to read and write (i.e., to change), but
also erased once a computer is turned off.

In finance, the volatility of a trading price series describes the degree of
its variation over time, by measuring the standard deviation of its
logarithmic returns. In this context, volatility is a measure of the
unpredictability of a market. This form of volatility is a parameter in the
well-known Black-Scholes model for the dynamics of financial markets, which
assumes market volatility is constant over time(black1973pricing, ). By
comparison, more recent models take into account that financial assets
experience periods of high and low volatility. That is, during certain
periods, prices might go up and down very quickly, while during other times
they show much less variation(heston1993closed, ; dupire1994pricing, ). This
change in market volatility is also named heteroskedasticity
(hetero=different, skedasis=dispersion).

In political science, the notion of volatility is found in the Pedersen index,
which measures the net change within an electoral party system resulting from
individual vote transfers between parties from an election to
another(pedersen1979dynamics, ). The Pedersen Index is also sometimes dis-
aggregated into Type A Volatility, which captures volatility from party entry
and exit, and Type B Volatility(powell2014revisiting, ), which measures the
volatility among stable parties that contest multiple elections. Type A and B
Volatility are also known as replacement volatility and electoral volatility.
Behavioural studies of voter volatility have also studied volatile behavioural
variables, such as unstable party affiliation, openness to alternatives, anti-
party attitude, and vote abstention, all of which aim at measuring how a voter
might behave in a volatile or nonvolatile manner(bybee1981mass, ). In spite of
a need for care in using the index so the results are meaningful and
comparable, it still remain the measurement tool of choice for volatility in
political behaviour and by extension to other kinds of
behaviour.(CasalBertoa2017, )

In the context of agenda-setting research, the volatility of issues in the
public agenda has been defined in terms of the “the shifting salience of the
issues that the [American] public regards as the most important problem facing
the country at a particular moment”(mccombs1995capacity, ) and measured as the
probability that an issue stays on the agenda at a particular point in time
after it is introduced. This is most closely related to the form of volatility
we study in this paper—hence the issue attention economy—but rather than
investigating the volatility of a particular issue, we study the volatility of
the whole issue agenda over time.

Volatility can be present in multiple ways when considering issue agendas. On
the one hand, the total number of issues being considered at a single point in
time could increase. On the other hand, the number of issues could remain
constant but attention could shift more quickly from one set of issues to
another. In the policy agendas literature, scholars have focused on the rare
events called ‘policy punctuations’, sudden breaks in an agenda series, which
tend not to be clustered at points in time or in domains but spread across
political systems, policy sectors, time periods, and institutions. (John2012,
) The volatility perspective offers a more global assessment of changes
feeding across the system as a whole, particular as punctuated changes reflect
periods when the agenda is stable which is then disrupted rather than
describing a move from stability to instability. Other literature on public
opinion-responsiveness focus on the crowding out of minor issues from large
events, such as the economy, and tend to assume a cyclical pattern to changes
in the diversity of topics on the agenda.(jennings2011effects, ) Smaller
issues can get onto the agenda once big crises have retreated and public
opinion is less concerned with one single issue.

## Quantifying political volatility

We propose two quantitative measures of political volatility drawing from
information theory. We measure the number of issues receiving attention
simultaneously using the effective number of issues ($N_{t}$ defined in
Equation 2), which is two to the power of the entropy of the distribution of
attention to a set of issues $X$ (Equation 1). This quantity can be measured
at any point in time $t$.

$H_{t}(X)=-\sum_{i=1}^{n}p_{t}(x_{i})\log_{2}p_{t}(x_{i})$ (1)

$N_{t}(X)=2^{H_{t}(X)}$ (2)

The intuition behind the effective number of issues is the same for Laakso and
Taagepera’s effective number of parties, which is a measure of the
fractionalization of a party system(laakso1979effective, ). It is a measure of
the diversity of an issue agenda at a given point in time, and has the same
functional form as diversity indices used in ecology or information
theory(jost2006entropy, ).

In their original publication, Laakso and Taagepera compare $N$ as defined in
Equation 2 to the inverse Simpson index, defined as
$N_{S}=1/\sum_{i=1}^{n}p_{i}^{2}$, and argue that while they show the same
qualitative features, the former definition is connected to the physical and
information-theoretic concept of entropy(laakso1979effective, ). Entropy has
also been used as a measurement of policy agenda
diversity(john2010punctuations, ; jennings2011effects, ), sometimes alongside
measures of electoral volatility(mortensen2011comparing, ), which makes
Equation 2 a logical measure of the number of substantive issues being
considered simultaneously by policy makers, the media, or the public.

As defined in Equation 2, the effective number of issues ranges from $1$, when
all attention is given to a single issue, to $n$, where public attention is
distributed uniformly across all issues. However, one would expect the latter
scenario to be unlikely, given the limits on our capacity for processing
information—the famous magical number of seven plus or minus two. (miller1956,
) For these reasons, we might expect that five to nine issues to receive
almost all of the attention at any single point in time.

Since it only provides a snapshot of an issue agenda at a given point in time,
the effective number of issues does not measure how attention to different
issues changes over time. There are other measures more appropriate for this
purpose, including the issue survival rate used by McCombs and
Zhu,mccombs1995capacity  as well as the measures of voter churn based on the
Pedersen Index.(pedersen1979dynamics, ) In the interest of defining a measure
consistent with the entropy-based effective number of issues presented in
Equation, 2 we define an information-theoretic measure for volatility, namely
the Kullback-Leibler (KL) Divergence(kullback1951information, ) between the
distribution of attention over issues between a given point in time $t$,
denoted $p_{t}$, and the distribution of attention at a previous time point
denoted $p_{t-1}$ (Equation 3). It is possible to speak of the KL Divergence
of a policy agenda between a pair of weeks, or between a pair of years.

$D_{\mathrm{KL}}(p_{t}|p_{t-1})\>=\>\sum_{i=1}^{n}p_{t}(x_{i})\,\log_{2}\left(\frac{p_{t}(x_{i})}{p_{t-1}(x_{i})}\right)$
(3)

KL Divergence, also known as relative entropy, is an information-theoretic
measurement of surprise, novelty, or information gain: it is a measure of how
one probability distribution diverges from a second, expected probability
distribution. Apart from its many uses in physics and machine learning, KL
Divergence has also been used to measure cognitive surprise(itti2009bayesian,
) and changes in exploration/exploitation behaviour.(murdock2017exploration, )
It has also been used to study agenda diversity in social media, by measuring
the divergence between different tweets produced by the same Twitter
users.park2013agenda  When comparing past and present distributions of
semantic content, KL Divergence has been called novelty.
(barron2018individuals, ) We use KL Divergence in a similar manner as the
novelty between the distribution of attention to issues at different time
steps to measure of volatility in the policy agenda.

## Empirical evaluation

We selected two country cases of the UK and Germany to analyze and compare in
this study. They meet three different kinds of selection criteria. The first
is different kinds of democratic country cases from which to observe occasions
of volatility and interconnections between different venues. The argument
about turbulence is one that applies to advanced democracies with their
greater use of media and internet, and pressure on policy systems. The two
countries are however different. The UK political system with its absence of
veto players and centralized political system has been associated with
receptivity to policy innovations and also liability to policy
disasters,(Dunleavy1995, ; King2013, ) though the empirical record for these
differences is less clear.(Gray1998, ). Germany has experienced relative
stability since 1945 through coalition government, has a complex system of
decentralization and intergovernmental relations, and usually experiences an
incremental pattern of policy-making (Green2005, ). The second criterion is
venues from which to observe changes over time. We selected public opinion
polls and traditional media data—two arenas that show less friction so may be
subject to volatility but also can be measured over a long period of time. The
expectation is that print media will be very responsive to the general pattern
of turbulence. We also look at public opinion, partly because we can compare
the two countries over a long period of time, and because public opinion
shifts are likely to be the transmission mechanism for volatility. The final
selection criteria is the period of time, in order to have enough decades to
encompass the changes in recent years and to allow for other periods of
changes to be accounted for in the analysis.

### Public opinion polls

Figure 1:  Total and effective number of issues in UK and Germany public
opinion polls. The top panels show the growth of (a) the number of issues with
non-zero attention by year for the British public opinion survey data, versus
(b) the effective number of issues in the responses to the polls. Panels (c)
and (d) respectively show the same numbers for the German opinion polls.

We start with analysis of the data from attitudinal surveys in Germany and the
UK, which ask nationally representative samples of the German and British
populations what they feel is the most important issue facing their countries
on the day. Data are collected every month in the UK by Ipsos MORI, and every
three months by the German Longitudinal Election Study (GLES), and the
responses are coded into different issue categories. A heatmap showing public
attention to different issues by month from 1985 to 2016 in the UK is shown in
Figure 5.

To examine shifts in public attention, we computed the number of issues coded
each month as well as the distribution of attention to these issues. Figure 1
shows the total and effective number of issues in British and German public
opinion polls. Panels 1a and 1c respectively show the number of issues
mentioned every year in the British public opinion survey data and every
quarter in the German opinion survey data. Panels 1b and 1d show the effective
number of issues present in the survey data, for the UK and Germany
respectively, calculated following Equation2. For the UK polls dataset, the
top 20 issues were considered, and issues that were introduced in the UK polls
after 1990 and in the Germany polls after 2012 were removed from the analysis.
Both datasets show a steady total number of issues over time, staying the same
after 1990 for the British polls, and staying between 15 and 16 issues for all
German polls after 2011. This stasis, however, is not seen in the effective
number of issues: for the UK polls, shown in Figure 1b, the effective number
of issues grows slowly and is marked by oscillations around a mean that seems
to move slowly from around 9 issues in 1993 to 11 or 12 issues in 2015. The
German polls, shown in Figure 1d, also seem to show a slow growth from around
9 issues in 2010 to around 12 issues in 2014, followed by a steep decrease in
2015. This is due to the issue of migration and integration becoming a
dominant issue, as can be seen in Figure 6 in the Appendix.

While the effective number of issues in the public opinion polls shows clear
signs of growth over the years for both countries, the month-to-month novelty
(for the UK polls) and quarter-to-quarter novelty (for the Germany polls) does
not show any significant trend over time. Figure 2a showing the novelty in the
British opinion polls stays mostly constant over the period of study. The
introduction of new issues between 1985 and 1990 produces a series of peaks in
the plot, as new issues are evidently a great source of novelty in the issue
agenda. The plot also shows a pair of peaks, one after 1990 and one after
2000, occurring shortly after the IRA attacks in 1990-91 in the UK and after
the September 11 attacks in 2001, in the United States. Since those periods
correspond to a sharp increase in the Defence/Terrorism issue (see Figure 5),
it is reasonable to attribute these peaks in novelty to the sudden relevance
of this issue in the UK policy agenda.

In contrast to the peaks corresponding to terrorist attacks, the 2008 economic
crisis shows as a small bump in the time series shown in Figure 2)a,
representing an increase in novelty, but one much smaller than the ones in
1990-91 and 2001. This is explained by the fact that the growth of the Economy
issue in the policy agenda is gradual, taking a year to go from an issue of
medium importance in the end of 2007 to the most importance in the beginning
of 2009. This represents a relatively slow growth in importance, in comparison
to the sudden jumps in issue relevance caused by terrorist attacks. After the
bump in novelty during the economic crisis of 2008, the novelty in the UK
polls does not show any meaningful trend. This also applies to the 2011-2018
interval shown in Figure 2b for the German polls.

### News media

To study the variation in the issues covered in the German and British media,
we looked at the patterns of language use over time, which can be uncovered by
topic modelling. Topic models feature among the most widely used tools in
large-scale text analysis, and can be used to estimate the semantic content of
large collections of documents. For example, topic modelling has been used to
analyze congress speeches(hillard2008computer, ; quinn2010analyze, ;
herzog2018transfer, ).

We used latent Dirichlet allocation (LDA)(blei2003latent, ) topic modelling,
which assigns every article to a weighted combination of “topics,” i.e., co-
occurring word patterns; these topics typically relate to issues in the policy
agenda with words related to issues such as the national economy,
unemployment, or war featuring heavily in the topics.111See appendix for
topics of one run; we do several runs varying the number of topics $k$ between
20 and 80 and find our results are qualitative similar for any choice of $k$
in this range. The LDA method allows for a variable number of topics,
representing the level of coarse-graining of the text data. We present results
using 50 topics. After stopword removal, the text in every article is
converted into a distribution over topics (issues), after which it becomes
possible to measures effective number of issues and the change in attention
between issues using the approaches described above.

Figure 3 shows the effective number of issues measured for the two sources of
news media used in this paper, namely the UK-based daily newspaper The
Guardian and the Germany-based weekly news magazine _Der Spiegel_. Both news
vehicles show a varying effective number of issues over time. For the
Guardian, shown in Figure 3a from 2013 to 2018, the effective number of issues
grows from 2013 to 2015, holding a stable value from 2015 onwards. Der
Spiegel, shown in Figure 3b, shows wide variation in the first decade after
its foundation in 2917, followed by a growth in the effective number of issues
covered in the news magazine from 1955 to the mid-nineties. There is a show
downturn after that, followed by a sharp decline in the first half of 2010.
This non-monotonic dynamics might be associated with many sources – the
reunification of Germany in 1990, the launch of _Spiegel Online_ in 1994, or
even the introduction of a new regional supplement in Switzerland in 2007.
Whether this drastic change in the effective number of issues in _Der Spiegel_
was due to any of the above events or whether it was due to other newspaper
internal policies of the newspaper is still unclear.

As for their levels of novelty, again _The Guardian_ and _Der Spiegel_ show
different behaviour. As shown in Figure 4a, the month-to-month novelty in the
The Guardian is kept constant from 2013 to 2018. For _Der Spiegel_ , shown in
Figure 4b, the month-to-month novelty shows a slow increase, which is sharper
in the 1960-1970 decade. _Der Spiegel_ also shows a jump in novelty in early
2010, around the same time as the sudden decrease in the effective number of
issues shown in Figure 3b. Once again, further research is required to
evaluate what caused the patterns observed for both newspapers.

Figure 2:  Time series plots showing the novelty (KL Divergence) measured for
the distribution of attention to different issues over time, (a) in the
British monthly public opinion survey data, and (b) in the German quarterly
public opinion survey data. Higher values indicate sudden shifts in the public
attention to policy issues, as well as the introduction of new issues to the
policy agenda between 1985 and 1990.  Figure 3:  Total and effective number of
issues in UK and Germany media. The two plots show the growth of the effective
number of issues, (a) for the British newspaper The Guardian, and (b) for the
German news magazine Der Spiegel.  Figure 4:  Time series plots showing the
novelty (KL Divergence) measured for the distribution of attention to
different issues over time, (a) in the British newspaper The Guardian, (b) in
the German news magazine Der Spiegel. Higher values indicate more sudden
shifts in the attention dedicated to policy issues in news media.  Figure 5:
A heatmap showing the public attention to policy issues by month from 1985 to
2016 as reported by representative surveys of the UK population collected by
Ipsos MORI. Brighter shades of red indicate that a large fraction of the
population pointed issues as unemployment and the National Health Service as
important. Grey areas indicate issues which were not included during a
specific survey, and black and grey boxes indicate specific events at the time
of an increase in the attention to specific issues.

## Discussion

So far this paper has shown that volatility can be thought about consistently
in measurement terms and that —from a record of two countries and two data
series— there is some evidence to suggest the volatility has increased in
recent decades. Of course, it is possible to say that this is consistent with
an intuitive assessment of recent events that was recounted at the start of
this paper, but it is important to ground this claim in theories of
comparative politics and political behaviour to offer reasons to expect
increased volatility, including less voter loyalty, the diversification of
issues, and the rise of new communication technologies such as social media.
It is important to know our results are theoretically what we expect. Indeed,
our empirical results reveal a much more nuanced picture than a simple
incursion of the new disruptive politics would imply.

Researchers on political culture and behaviour identify system level changes
in advanced democracies pointing to less attachment and voter loyalty to the
political system which might suggest large changes, but over very long term.
The key idea is that rather than voters being aligned to the party system and
then periodically realigned to a new party system, voters are less aligned
overall and more likely to switch parties.(Dalton:1984aa, ) From the 1970s
observers claims that voters were de-aligned, not having such clear homes in
parties that represent them for long periods.(Sarlvik1983, ) Such instability
can feed into the party system and then into politics more generally. When
linked to other trends that indicate citizen disengagement from politics, such
as declining trust in government and disaffection.(Pharr:2000aa, ) The rise of
anti-politics feeds on this lack of trust and also implies that citizens can
move strongly between activism and support for new political parties but then
return to apathy quite quickly.(Hibbing:2002aa, ; Clarke2018, ) It is easy to
see the connection to the current period, with new politicians taking
advantage of citizen detachment and lack of political trust in conventional
party politics to put forward their agendas and to disrupt existing routines.

Another trend toward instability is the diversification of issues and the
differentiated of interest representation, as policies become more complex and
interdependent. It is an argument that reaches back into the 1960s about the
diversification of interests, with the idea that the state has increased
pressures upon it, associated with the expansion of issues beyond economic
divides.(Lowi2010, ) This was picked up in the literature on interest groups
in the 1970s which assumed there was a lot of pressure on policy-making
system, the executive establishment, and a fragmentation of institutions that
had previously ensured a lot of stability for the political system,(Heclo1978,
) and had created a diverse set of policy networks.(Marsh1992a, ) Even though
notions of instability are party of the interest group fragmentation, at the
time at least, such ideas were thought to account of government overload and
gridlock rather a government being moved back and forth across different
issues.(King1975a, ; Birch1984a, ) The assumption of instability and interest
group diversity was picked up by students of public policy and agenda setting,
who were keen to show the limits to the stasis and gridlock
arguments.(Baumgartner:1993aa, ) In any case, the assumption of these
arguments is that changes in the 1960s and 1970s were crucial to understanding
new kinds of politics.

Social media and other recent changes communication technologies might also
increase volatility. These technologies represent a sudden change in the way
in which people communicate and share information, allowing for feedback and
interaction in real time. Even though more traditional forms of communication
through print and broadcast media continue to exist, they work in parallel
with social media, often with media professionals participating and reacting
to social media content. Social media exert social influence on users by
showing in real time what other people are doing (social
information)Salganik_2006 , reducing the costs of becoming visible and so
increasing the likelihood of acting. Individual acts of participation in turn
creates further social information, which may influence someone else’s
decision about whether to act, leading to feedback cycles and chain
reactions.Margetts:aa

These feedback influences at work on social media platforms have been shown to
introduce instability into cultural markets, and could now penetrate the
“issue attention” economy and the policy-making landscape, creating political
turbulence. This disorganized environment, in which fluid and overlapping
groups of individuals mobilize around common (often temporary) social issues
and goals, manifests itself, for example, in a highly unequal distribution of
participants. Most YouTube or Facebook videos have very few views while a
small number are watched millions of times having accumulated these views
quickly and dramatically. This changes have been quite recent, happening in
the mid-2000s, which is just before the turbulence of the last decade. Future
work might measure volatility in issue attention across many locations and
compare changes in volatility with social media adoption rates.

This literature indicates that political systems are experiencing greater
volatility since the 1970s, but our study provides the first empirical
investigation as to how volatility has changed. Larger-scale research would be
needed to work out whether the changes are associated with these specific
processes and their causal order.

Alternative approaches could consider data on other aspects of politics and
analyze the extent to which volatility affects other aspects of politics, from
voting, media content, speech data and policy outputs. We restricted our
study, as a preliminary analysis, to two countries and two data sources.

## Conclusions

We leave an open question as to what level of volatility is desirable in
politics. In general, politics is characterized by a degree of stability and
regularity in its inputs and outputs, such as habitual voting behaviour over
decades, regular concerns about public issues, a stream of stories in the
media, and in repeated policy outputs, such as laws and budgets. Such stable
processes, in the view of an earlier generation of political scientists, acted
as a bedrock to democratic systems and stopped them from tipping over into
extremism.(Almond:1963aa, )

But, as Huckfeldt shows theoretically, some short-term instability is the
product of even stable preferences and networks.Huckfeldt1983 Of course some
fluctuation is to be expected in any system as no democracy is stable but
responds to changes in perceptions and behaviours in reaction to events and
real world changes, and is based on interaction between participants. Absolute
stability is neither to be expected nor even desirable in democracies, as
rapid change may act as a safety valve in processing issues and responding to
citizens.

In one formulation, too much stability can lead to over-large fluctuations in
the agenda down the line, the friction associated with policy monopolies whose
rigidity give way to policy punctuations.Baumgartner:1993aa  Change might be
expected from the course of events and actions, such as economic decisions
taken elsewhere, or errors in policy implementation. Such events may catch the
attention of policy-makers, experts, media professionals, and the public, or
at least some of these actors, who may shift attention from other issues to
attend to them, as they are problems needing discussion and action. Then
having been addressed (or ignored) other issues appear to draw the attention
of the public and other actors, so the agenda moves in cycles of issues, which
is a standard insight of the agenda-setting literature.(citeulike:8241790, ;
Baumgartner_2006, )

It is the foundation of classic pluralism, where the political system is seen
as in equilibrium with changing forces and interests at society at
large.(Dahl:1961aa, ) Even the presence of punctuations is not clustered in
certain sectors or periods, emerging from endogenous pressures in policy-
systems rather than showing periods of volatility.John2012  Care needs to made
when attributing equilibrium to political systems, whether functionalist or
from repeated games. There is an argument to be made that political systems
have always been liable to chaos and rapid change, which come from random
choices and conjunctions.(Kingdon:1984aa, ).

The evidence in this paper shows that there is some grounds for thinking that
volatility has increased in recent decades, though the extent and patterning
of this changes in the two countries we have researched does not show the
large break that a reading of the crises of contemporary politics would
indicate, with the election surprises, new leaders changes and game-changing
events, such as Brexit. Even with the decrease in the effective number of
issues seen in the post-2015 German public opinion polls, both countries show
numbers higher than Miller’s seven plus or minus two.

Of course, the rapid change hypothesis is is a too simple a reading of
contemporary politics, and the trends we have uncovered suggest a more complex
and differentiated pattern, with some changes reaching back to the 1970s,
which is more consistent with the literature in political behaviour, interest
groups and public policy, which refers to voter disengagement, issue
complexity and differentiation of policy challenges. This paper then sets the
agenda for a future research programme that can ascertain the extent and
variation in political volatility in political attention and attribute its
causes in comparative contexts.

## References

  * (1) Margetts, H., John, P., Hale, S. A. & Yasseri, T.  _Political turbulence: how social media shape collective action_ (Princeton University Press, 2015).
  * (2) Black, F. & Scholes, M.  The pricing of options and corporate liabilities.  _Journal of political economy_ 81, 637–654 (1973).
  * (3) Heston, S. L.  A closed-form solution for options with stochastic volatility with applications to bond and currency options.  _The review of financial studies_ 6, 327–343 (1993).
  * (4) Dupire, B. _et al._ Pricing with a smile.  _Risk_ 7, 18–20 (1994).
  * (5) Pedersen, M. N.  The dynamics of european party systems: changing patterns of electoral volatility.  _European Journal of Political Research_ 7, 1–26 (1979).
  * (6) Powell, E. N. & Tucker, J. A.  Revisiting electoral volatility in post-communist countries: New data, new results and new approaches.  _British Journal of Political Science_ 44, 123–147 (2014).
  * (7) Bybee, C. R., McLeod, J. M., Luetscher, W. D. & Garramone, G.  Mass communication and voter volatility.  _Public Opinion Quarterly_ 45, 69–90 (1981).
  * (8) Casal Bértoa, F., Deegan-Krause, K. & Haughton, T.  The volatility of volatility: Measuring change in party vote shares.  _Electoral Studies_ 50, 142–156 (2017).
  * (9) McCombs, M. & Zhu, J.-H.  Capacity, diversity, and volatility of the public agenda: Trends from 1954 to 1994.  _Public Opinion Quarterly_ 59, 495–525 (1995).
  * (10) John, P. & Bevan, S.  What are policy punctuations? Large changes in the Legislative Agenda of the UK Government, 1911-2008.  _Policy Studies Journal_ 40 (2012).
  * (11) Jennings, W. _et al._ Effects of the core functions of government on the diversity of executive agendas.  _Comparative Political Studies_ 44, 1001–1030 (2011).
  * (12) Laakso, M. & Taagepera, R.  “effective” number of parties: a measure with application to west europe.  _Comparative political studies_ 12, 3–27 (1979).
  * (13) Jost, L.  Entropy and diversity.  _Oikos_ 113, 363–375 (2006).
  * (14) John, P. & Jennings, W.  Punctuations and turning points in british politics: The policy agenda of the queen’s speech, 1940–2005.  _British Journal of Political Science_ 40, 561–586 (2010).
  * (15) Mortensen, P. B. _et al._ Comparing government agendas: Executive speeches in the netherlands, united kingdom, and denmark.  _Comparative Political Studies_ 44, 973–1000 (2011).
  * (16) Miller, G. A.  The magical number seven, plus or minus two: Some limits on our capacity for processing information.  _Psychological Review_ 81–97 (1956).  URL http://psychclassics.yorku.ca/Miller/.
  * (17) Kullback, S. & Leibler, R. A.  On information and sufficiency.  _The annals of mathematical statistics_ 22, 79–86 (1951).
  * (18) Itti, L. & Baldi, P.  Bayesian surprise attracts human attention.  _Vision research_ 49, 1295–1306 (2009).
  * (19) Murdock, J., Allen, C. & DeDeo, S.  Exploration and exploitation of victorian science in darwin’s reading notebooks.  _Cognition_ 159, 117–126 (2017).
  * (20) Park, S., Ko, M., Lee, J., Song, J. _et al._ Agenda diversity in social media discourse: A study of the 2012 korean general election.  In _ICWSM_ (2013).
  * (21) Barron, A. T., Huang, J., Spang, R. L. & DeDeo, S.  Individuals, institutions, and innovation in the debates of the french revolution.  _Proceedings of the National Academy of Sciences_ 115, 4607–4612 (2018).
  * (22) Dunleavy, P.  Policy Disasters: Explaining the UK’s Record.  _Public Policy and Administration_ 10, 52–70 (1995).
  * (23) King, A. & Crewe, I.  _The blunders of our governments_ (Oneworld Publications, 2013).
  * (24) Gray, P., Hart, P. t. & Hart, P. .  _Public policy disasters in Western Europe_ (Routledge, 1998).
  * (25) Green, S. & Paterson, W. E.  _Governance in contemporary Germany : the semisovereign state revisited_ (Cambridge University Press, 2005).
  * (26) Hillard, D., Purpura, S. & Wilkerson, J.  Computer-assisted topic classification for mixed-methods social science research.  _Journal of Information Technology & Politics_ 4, 31–46 (2008).
  * (27) Quinn, K. M., Monroe, B. L., Colaresi, M., Crespin, M. H. & Radev, D. R.  How to analyze political attention with minimal assumptions and costs.  _American Journal of Political Science_ 54, 209–228 (2010).
  * (28) Herzog, A., John, P. & Mikhaylov, S. J.  Transfer topic labeling with domain-specific knowledge base: An analysis of uk house of commons speeches 1935-2014.  _arXiv preprint arXiv:1806.00793_ (2018).
  * (29) Blei, D. M., Ng, A. Y. & Jordan, M. I.  Latent Dirichlet allocation.  _Journal of machine Learning research_ 3, 993–1022 (2003).
  * (30) See appendix for topics of one run; we do several runs varying the number of topics $k$ between 20 and 80 and find our results are qualitative similar for any choice of $k$ in this range.
  * (31) Dalton, R. J., Flanagan, S. C., Beck, P. A. & Alt, J. E.  _Electoral change in advanced industrial democracies: realignment or dealignment?_ (Princeton University Press, Princeton, N.J., 1984).
  * (32) Särlvik, B. & Crewe, I.  _Decade of dealignment : the Conservative victory of 1979 and electoral trends in the 1970’s_ (Cambridge University Press, 1983).
  * (33) Pharr, S. J. & Putnam, R. D.  _Disaffected democracies: what’s troubling the trilateral countries?_ (Princeton University Press, Princeton, N.J., 2000).
  * (34) Hibbing, J. R. & Theiss-Morse, E.  _Stealth democracy: Americans’ beliefs about how government should work_ (Cambridge University Press, Cambridge, 2002).
  * (35) Clarke, N., Jennings, W., Moss, J. & Stoker, G.  _The Good Politician_ (Cambridge University Press, 2018).
  * (36) Lowi, T. J.  _The end of liberalism : the second republic of the United States_ (W.W. Norton & Co, 2010).
  * (37) Heclo & H.  Issue Networks and the Executive Establishment.  In _The New American Political System_ (AEI, 1978).
  * (38) Marsh, D. & Rhodes, R. A. W. (eds.) _Policy Networks in British Government_ (Oxford University Press, 1992).
  * (39) King, A.  Overload: Problems of Governing in the 1970s.  _Political Studies_ 23, 284–296 (1975).
  * (40) Birch, A. H.  Overload, Ungovernability and Delegitimation: The Theories and the British Case.  _British Journal of Political Science_ 14, 135 (1984).
  * (41) Baumgartner, F. R. & Jones, B. D.  _Agendas and instability in American politics_.  American politics and political economy series (University of Chicago Press, Chicago, 1993).
  * (42) Salganik, M. J.  Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market.  _Science_ 311, 854–856 (2006).
  * (43) Almond, G. A. & Verba, S.  _The civic culture: political attitudes and democracy in five nations_ (Princeton University Press, Princeton, N.J., 1963).
  * (44) Huckfeldt, R. R.  The Social Context of Political Change: Durability, Volatility, and Social Influence.  _American Political Science Review_ 77, 929–944 (1983).
  * (45) Downs, A.  Up and down with Ecology: the issue-attention cycle.  _Public Interest_ 28, 50 (1972).
  * (46) Baumgartner, F. R., Green-Pedersen, C. & Jones, B. D.  Comparative studies of policy agendas.  _Journal of European Public Policy_ 13, 959–974 (2006).
  * (47) Dahl, R. A.  _Who governs?: Democracy and power in an American city_ , vol. 4 (Yale University Press, New Haven, 1961).
  * (48) Kingdon, J. W.  _Agendas, alternatives, and public policies_ (Little, Brown, Boston, 1984).

Acknowledgements We thank the Alan Turing Institute (EPSRC grant EP/N510129/1)
and the Volkswagen Foundation for funding this research.

Author contributions Equal contribution.

Competing financial interests The authors declare no competing financial
interests.

Figure 6:  The proportion of the attention dedicated to different most
important issues (MIIs) in the German opinion polls plotted over time, with
different colours indicating the ten most important issues from the survey.


11institutetext: IRISA, Univ Bretagne Sud, Vannes, France 22institutetext:
Motion-Up Company, Vannes, France 22email:
<EMAIL_ADDRESS><EMAIL_ADDRESS>
# A New Spatio-Temporal Loss Function for 3D Motion Reconstruction and
Extended Temporal Metrics for Motion Evaluation

Mansour Tchenegnon 11 Sylvie Gibet 11 Thibaut Le Naour 22

###### Abstract

We propose a new loss function that we call Laplacian loss, based on spatio-
temporal Laplacian representation of the motion as a graph. This loss function
is intended to be used in training models for motion reconstruction through 3D
human pose estimation from videos. It compares the differential coordinates of
the joints obtained from the graph representation of the ground truth against
the one of the estimation. We design a fully convolutional temporal network
for motion reconstruction to achieve better temporal consistency of
estimation. We use this generic model to study the impact of our proposed loss
function on the benchmarks provided by Human3.6M. We also make use of various
motion descriptors such as velocity, acceleration to make a thorough
evaluation of the temporal consistency while comparing the results to some of
the state-of-the-art solutions.

###### Keywords:

3D human pose estimation, motion reconstruction, neural network, loss
functions, Laplacian representation, evaluation metrics

##  1 Introduction

The 3D human pose estimation is a topic that has been extensively studied in
recent years. The goal is to reconstruct the 3D skeletal pose from the image.
One way to achieve this goal involves first to estimate the 2D joint locations
from the image [5, 18]. Then, from these 2D joint locations, the 3D skeletal
pose can be estimated. This process can be extended to video data. Given an
input video or a sequence of 2D joint locations for each frame of the video,
the objective becomes to compute the 3D joint positions for each frame.
Working with video brings some advantages. In particular, adding temporal
information can improve the learning of depth information [15]. Therefore, new
solutions using temporal features from the input data are the subject of
recent research.

Many deep learning methods learn from temporal information. Among these,
previous methods based on Recurrent Neural Networks (RNN) have shown their
efficiency but in return are very time consuming. Other more efficient
solutions have then been proposed, based on convolutional neural networks [7,
19]. In this case, the motion is considered as a temporal signal of joint
positions on which convolution filters are applied. Some researchers propose
to learn temporal features from adjacent frames through a convolution network,
and then regress this information to estimate the central frame [17]. This
solution improves the pose estimation but the temporal consistency of the
motion is not necessarily better since the poses are still estimated one by
one. In theory, the more accurate the pose estimation per frame, the more
continuous the reconstructed motion. However, sufficient accuracy to obtain a
temporally plausible reconstructed motion has not yet been achieved. Given
this fact, we propose to use a convolutional neural network, called CVM-Net,
to reconstruct motion through sequence-to-sequence pose estimation. The goal
then becomes a motion reconstruction task. In this paper, the idea is not to
outperform the spatial accuracy of recent neural network approaches, but to
propose an approach that better preserves the temporal consistency.

Besides the design of this fully convolutional network, our main contribution
is the definition of a new loss function for sequence-to-sequence pose
estimation, called the Laplacian Loss Function. This function exploits a
spatio-temporal representation of the poses sequence, inspired by the
Laplacian graph $3D+t$ model [16]. Indeed, for this kind of sequence-to-
sequence neural network, using only the default Joint Position Loss function
(average Euclidean loss for each posture of the sequence), leads to the
averaging of the posture errors over the whole motion sequence. This is
achieved by filtering out extreme postures and by favouring those postures
that are most representative of the training data. By focusing on the
Laplacian representation of motion, we take into account both the spatial
structure of each pose, constrained by the skeleton, and the temporal
trajectories of the skeletal joints.

Our goal is to preserve the temporal consistency of the reconstructed motion
first and then achieve an acceptable spatial accuracy. Therefore, we need
adequate metrics for the evaluation of the neural network that consider the
temporal characteristics of movements. On the spatial aspect, the Mean Per-
Joint Positions Error (MPJPE) used in all state-of-the-art solutions is the
best choice. Most approaches limit their evaluation to this metric since most
of them focus only on achieving the best accuracy in reconstructing the joint
positions. Very few approaches make a temporal consistency evaluation. For
that, they use the Mean Per-Joint Velocity Error (MPJVE), that evaluates the
estimated velocity error of the joints, computed on adjacent poses. This
metric is a good start to evaluate the temporal consistency of a motion.
However, motion characteristics are not limited to the velocity. We propose to
extend the evaluation to the acceleration, so we propose a metric to evaluate
the acceleration of the reconstructed motion, that is the Mean Per-Joint
Acceleration Error, MPJAccE.

In this paper, we present in Section 3 the generic network for sequence-to-
sequence estimation that we propose and the reasons for our choice. In Section
4, after presenting some traditional loss functions, we propose a new spatio-
temporal loss function based on the Laplacian representation of the motion.
Section 5 presents the extended metrics that we propose to evaluate the
temporal consistency of reconstructed motion. Finally in Section 6 we present
and discuss the results of our experiments, including an ablation study for
the Laplacian loss and the evaluation results based on metrics computed from
motion descriptors.

##  2 Related Work

In this paper, we address the issue of reconstructing motion from video
through 3D human pose estimation.

###  2.1 3D Human Pose Estimation

3D human pose estimation consists in estimating 3D skeletal poses given an
image or 2D joint locations. According to the type of data used as input, we
have two main categories. The first category uses the image as input and
directly estimates 3D poses. In this case, the methods compute 2D and 3D
features, such as heat maps and other features (camera focal length, depth
information) to estimate the final 3D poses. These methods generally involve
two stages: a features detection stage followed by a 3D pose estimation [24,
9, 12, 22]. Yang et al. [24] use a 3D estimation pose network and a pose
discriminator to ensure that the estimated poses are plausible. Wei et al.
[22] use a framework to generate heat maps and bone maps in order to extract
2D pose hypotheses. They then use a pose regressor or a selection-based
algorithm on these hypotheses to compute the final 3D pose.

In the second category, one starts by estimating 2D joint locations in the
image using a 2D pose estimator. From the estimated 2D joint locations,
through various methods, it is possible to estimate the corresponding 3D
poses. The main advantage of this approach is that it is more efficient on
videos in the wild, due to the use of state-of-the-art 2D estimators. Some
researchers propose lifting models [13, 3, 6, 26, 20, 27]. Martinez et al.
[13] propose an approach using consecutive linear layers to perform a 2D-to-3D
joint positions regression. Combining a 2D-to-3D pose regression and a
3D-to-2D pose re-projection modules, Biswas et al. [3] use information in .
Chen et al. [6] present an unsupervised algorithm that lifts 2D joints to 3D
skeletons. They show that adding random 2D projections and an adversarial
network allows the training process to be self supervised using geometric
consistency. Shimada et al. [20] decide to first estimate 3D pose from 2D
joints locations, and then make the estimated pose more realistic, through
foot contact prediction and physics-based pose optimization. Zou et al. [27]
and Zhao et al. [26] represent the 2D joint locations as a graph structure and
use a Graph Convolutional Networks to estimate the 3D pose from it. Azizi et
al. [2] encode transformations between joints using the Möbius Transformation
and propose a new light Spectral GCN to achieve state-of-the-art results. All
these approaches focus on 3D pose regression and achieves great results in 3D
pose estimation.

###  2.2 Motion Reconstruction

#### Network architecture

Many 3D pose estimators are currently proposed in the literature. Most of them
only work on one image at a time. When receiving a video as input, they
estimate the pose at each frame, and then directly concatenate the outputs.
This way of reconstructing a motion does not take into account the temporal
characteristics of motion. This leads to some unsteady movements in the
results, and very few approaches have considered these effects [14, 17, 4, 19,
21, 23, 7, 8]. Among them, Metha et al. [14] choose to infer the pose at time
$t-1$ to estimate the pose at time $t$. Wang et al. [21] represent the 2D
skeleton input as a spatio-temporal graph and propose a Graph Convolution
Network to predict 3D poses. Xu et al. [23] choose to first estimate 3D poses
and then use a trajectory completion framework to correct the sequence. More
recently, Shi et al. [19] propose a CNN approach coupled with a skeleton model
to correct the spatial joint positions. In their solution, two independent CNN
models are first in charge of estimating the sequence of rotations and the
bone lengths to preserve some of the skeleton constraints. From these
features, they apply the forward kinematics model to obtain the sequence of 3D
poses. Another solution proposed by Chen et al. [7] is to predict the length
and direction of the bones throughout the sequence and compute the 3D poses
from these. Our approach is based on temporal convolution to estimate 3D pose
sequence from 2D joint locations sequence (obtained from video using a state-
of-the-art 2D pose estimator).

#### Loss function

Since motion reconstruction refers to spatio-temporal data, it is necessary to
have an appropriate loss function to ensure a better learning process. The
loss functions used in 3D human pose estimation are insufficient for this task
because they focus on spatial accuracy alone. To solve this situation, some
researchers propose new loss functions based on temporal characteristics of
the motion. Among them, some choose to calculate the loss function by using
the first derivative, that is the velocity [25, 4]. Wang et al. [21] propose a
loss function, called Motion Loss, computed from a motion pose encoding space.
They project the predicted and ground truth joint positions into this space
and compute the difference between the two encoded information. This
difference evaluates the quality of the reconstructed motion. Unlike them, we
propose a spatio-temporal loss function as a solution to the learning process.

##  3 Deep Learning and motion temporal features

The human pose estimation task consists in estimating with accuracy the joints
locations of the skeleton. Therefore, it mainly focuses on the spatial aspect
of the motion, whether it uses a single frame or multiple frames. Our aim is
to preserve the temporal aspect of the reconstructed motion, while achieving
acceptable joint position errors.

Figure 1: CVM-Net architecture. Temporal Convolution Neural Networks for
motion reconstruction. This is a generic approach that combines 1D convolution
to transform the 2D poses sequence into a 3D poses sequence. It also learn
long duration relation between joints via dilation parameters

Recent approaches prove that including temporal features of the motion
improves the results in human pose estimation. A good technique to process
temporal data is to use a RNN (Recurrent Neural Network). It allows to use
information from previous frames to estimate the current one. However the
computation is time and resources consuming. To overcome this drawback, an
efficient and less consuming solution is to use Convolutional Neural Networks
(CNN). In a survey conducted by Kiranyaz et al. [11], some studies on 1D
convolution networks prove that they have a low computational complexity and
are well-suited for low-cost and real-time applications. Besides, Arsalan et
al. [1] have also proven 1D convolution to be effective for trajectory-based
air writing recognition. By considering a motion as a set of skeletal joint
trajectories, 1D convolution is therefore a suitable choice to build our
model.

1D convolution applies filters using a time window. It takes as input $b\times
T_{in}\times C_{in}$ and output $b\times T_{out}\times C_{out}$, where $b$ is
the batch size, $T_{in}$ and $T_{out}$ represent respectively the sequence
lengths of the input and output, and $C_{in}$ and $C_{out}$ the channels or
features for input and output. Moreover, we can use a dilated convolution to
apply filters on non-consecutive frames to learn features at different time
scales. Another advantage of the fully convolutional neural network is that it
does not require a fixed sequence length and as a result can be easily
generalized.

Like many recent solutions, we have built our CVM-Net neural network to
estimate poses through temporal convolution. By using multiple temporal
convolution layers on frames in a time interval
$[t-\frac{\tau}{2},t+\frac{\tau}{2}]$, of length $\tau+1$, the accuracy of the
the central frame estimation $t$ is improved. But unlike most of these
approaches, our solution estimates multiple output frames ($T_{out}$) from
multiple input frames ($T_{in}$), with $T_{in}=T_{out}$.

We used this neural network to study of the different functions we propose in
this paper, namely the loss function and the motion evaluation metrics.

##  4 Loss Functions

The loss function is one of the main part of a neural network. This is a key
element in the training of neural networks, which indicates to the model its
erroneous behaviour and brings possible corrections. The loss function is
chosen according to the task at hand. Existing loss functions for human pose
estimation and motion reconstruction focus on either the spatial aspect or the
temporal aspect of the movement. Both are computed separately and them
combined. This can be a limitation since each aspect, spatial or temporal, is
considered independently. It is therefore necessary to find the appropriate
coefficients of the linear combination while computing the global loss. In
this section, we propose a loss function based on both the temporal and
spatial characteristics of the motion.

###  4.1 Existing Loss Functions

####  4.1.1 Joint Position Loss

This is the most commonly used loss function in a single frame pose estimation
context. It computes an average distance between the joint positions of the
ground truth poses and the estimated poses. Applied to multiple frames poses
estimations it is defined as:

$\mathcal{L}_{P}=\frac{1}{T}*\frac{1}{J}\sum_{t=1}^{T}\sum_{j=1}^{J}\Arrowvert
P_{t,j}-\overline{P}_{t,j}\Arrowvert_{2}$ (1)

where $P_{t,j}$ and $\overline{P}_{t,j}$ are respectively the 3D estimated
position and the 3D ground truth position of joint $j$ at time $t$. This
function, if used solely as loss function, works well for single frame pose
estimation. But, when working on motion reconstruction, it is limited because
it tends to average the joint positions loss over the whole sequence. The less
represented poses in the motion can be biased by the more represented ones,
affecting the overall motion reconstruction.

####  4.1.2 Motion Loss

Wang et al. [21] propose a loss function as a distance in motion space. It is
based on the encoding motion from a sequence of poses, by computing
differential values between same joints at different time scales. It can be a
subtraction, an inner-product or a cross-product. They encode both the
estimated and the ground truth poses sequences. The loss is then computed
between the encoded ground truth and reconstructed poses. They then combine
this motion loss with the joint positions loss to compute an overall cost
function defined by:

$\mathcal{L}=\mathcal{L}_{P}+\lambda*\mathcal{L}_{M}$ (2)

where $\mathcal{L}_{M}$ represents the motion loss, $\mathcal{L}_{P}$ the
joint position loss and $\lambda$ a coefficient to apply on the motion loss.

###  4.2 Laplacian Loss

We propose a loss function for spatio-temporal features learning, which is
based on the Laplacian representation of motion as a $3D+t$ graph, as defined
by Le Naour et al. [16]. The skeleton joints of the motion are considered as
the nodes of the graph. The $3D+t$ graph is then obtained by i) first,
connecting the joints to form the skeleton at each frame; these are the
spatial edges. ii) We then create temporal edges by connecting joints between
consecutive frames. Let’s consider a motion of length $T$ by a skeleton of $J$
joints, and let $v_{j,t}$ be a node of the graph, representing the joint $j$
of the skeleton at time $t$. We create the temporal edges by connecting
$v_{j,t}$ to $v_{j,t-1}$ and $v_{j,t+1}$ which are the joints $j$ of skeletons
at time $t-1$ and $t+1$ respectively. The graph $3D+t$ is then defined by
$G=(V,E_{S}\cup E_{T})$, with $V=\\{v_{j,t}\\}$ the set of all the joints,
$E_{S}$ the set of spatial edges and $E_{T}$ the set of temporal edges.

Figure 2: Graph $3D+t$ of a motion of length 3 for a skeleton of 17 joints.
The black edges represent the temporal edges of $E_{T}$ while the coloured
edges represent the spatial edges of $E_{S}$.

Using this spatio-temporal graph allows to define in an unified structure, the
spatial and temporal relations between each joints of the sequence. From the
graph 3D+t, we extract the matrix $L$. It is a square matrix of dimension
$(N\times N)$ where $N=T*J$ represents the total number of joints (the number
of frames $T$ in the sequence is multiplied by the number of joints $J$ per
skeleton). $w_{i,j}$ represents the weight attributed to each edge of the
graph. In our case we apply uniform weights for both spatial and temporal
edges.

$w_{i,j}=\left\\{\begin{array}[]{ll}1&\text{if }(i,j)\in E_{S}\text{ or
}E_{T}\\\ 0&\text{otherwise}\end{array}\right.$ (3)

Let’s define the matrix $\Delta$, so called differential coordinates matrix,
computed from the Laplacian matrix $L$ and the joint positions matrix $P$ is
of dimension $(N\times 3)$: $\Delta=L*P$.
$\Delta$ is then of dimension $(N\times 3)$. The matrix $\Delta$ represents
the differential coordinates of each joint relatively to its neighbours. The
closer the differential coordinates of the estimated poses to those of the
ground truth, the better the reconstructed motion. We can therefore compute an
average distance error between the differential coordinates of the ground
truth and those of the estimation. This distance represents our loss value
computed as follows:

$\mathcal{L}_{\Delta}=\frac{1}{N}\sum_{1}^{N}\Arrowvert\Delta^{gt}-\bar{\Delta}\Arrowvert$
(4)

where $N=T.J$ is the total number of joints with $T$ the sequence length and
$J$ the number of joints for a skeleton, $\Delta$ is the matrix of
differential coordinates computed from the ground truth and $\bar{\Delta}$ the
matrix of differential coordinates computed from the estimation. By training
the neural network with such a loss function will allow to learn the
connections between the joints locations it predicts, thus taking into account
implicitly the skeletal structure and the temporal evolution of the joints.

$\mathcal{L}_{\Delta}$ computes the mean absolute error between the
differential coordinates extracted from the ground truth joint positions and
the estimated joint positions. This loss is based on a differential
representation of the original output and does not consider the absolute joint
positions. Therefore we combine it with the joint positions loss to compute an
overall loss:

$\mathcal{L}=\mathcal{L}_{P}+\alpha*\mathcal{L}_{\Delta}$ (5)

where $\mathcal{L}_{P}$ is the joint positions loss, $\mathcal{L}_{\Delta}$ is
the Laplacian loss and $\alpha$ is the coefficient applied on the Laplacian
loss.

##  5 Evaluation Metrics for Temporal Consistency Performance

There are two existing metrics for evaluating human pose estimation. The main
metric used in the state of the art is the Mean Per Joint Position Error
(MPJPE, same formula as the Joint Position Loss 4). It evaluates the spatial
accuracy of the models, using the average errors in estimating the joint
positions. The lower the error, the better the accuracy. The Mean Per Joint
Velocity Error, on the other hand, evaluates the temporal consistency by
computing the average velocity error between the estimation and the ground
truth. In the state-of-the-art, it is computed as defined in equation 7.
Temporal consistency refers to how close the reconstructed motion is to the
ground truth in terms of smoothness, velocity, or acceleration.

$v_{j,t}=P_{j,t+1}-P_{j,t}$ (6)

$MPJVE=\frac{1}{T-1}*\frac{1}{J}\sum_{t=1}^{T-1}\sum_{j=1}^{J}\Arrowvert
v_{t,j}-\overline{v}_{t,j}\Arrowvert_{2}$ (7)

where $v_{t,j}$ and $\overline{v}_{t,j}$ represent the velocity vectors of
joint $j$ at time $t$, respectively from the ground truth and the estimation.

In order to extend the temporal consistency evaluation to other motion
descriptors. Therefore we define $MPJAccE$ metric (Mean Per Joint Acceleration
Error in equation 9), which is based on acceleration.

$a_{j,t+1}=P_{j,t+2}-2*P_{j,t+1}-P_{j,t}$ (8)

$MPJAccE=\frac{1}{T-2}*\frac{1}{J}\sum_{t=1}^{T-2}\sum_{j=1}^{J}\Arrowvert
a_{t,j}-\overline{a}_{t,j}\Arrowvert_{2}$ (9)

where $a_{t,j}$ and $\overline{a}_{t,j}$ represent the acceleration vectors of
joint $j$ at time $t$, respectively from the ground truth and the estimation.

##  6 Experiments

###  6.1 Ablation Study

To evaluate the performance of our new loss function, we first set up
training-test experiments using the neural network architecture proposed in
section 3. We use the same training environment for each session. In these
experiments, we compare three configurations of loss functions.

  * •

CVM-Net uses only the joint position loss $\mathcal{L}_{P}$ as cost function
(baseline).

  * •

CVM-Net + $\mathcal{L}_{M}$ uses a combination of the joint positions loss
$\mathcal{L}_{P}$ and the Motion Loss $\mathcal{L}_{M}$ as proposed by Wang et
al. [21] in an overall function.

  * •

CVM-Net + $\mathcal{L}_{\Delta}$ finally uses a combination of the joint
positions loss and our Laplacian Loss $\mathcal{L}_{\Delta}$ in an overall
function.

Table 1: Comparison of the three loss functions: CVM-Net (baseline), CVM-Net +
$\mathcal{L}_{M}$, CVM-Net + $\mathcal{L}_{\Delta}$, with the reconstructed
errors MPJPE (under Protocol-1) and MPJVE. Protocol-1 computes the MPJPE from
joint positions relative to the root joint (central hip) by aligning the root
joints of both the estimation and the ground truth. a) Comparison of loss
functions with MPJPE; b) with MPJVE. For both metrics, the lower the better.

(a) MPJPE comparison results in mm.

MPJPE | Dir. | Dis. | Eat. | Greet. | Phon. | Phot. | Pos. | Purch.
---|---|---|---|---|---|---|---|---
CVM-Net | 85.25 | 129.41 | 109.17 | 101.22 | 117.17 | 137.31 | 86.12 | 293.37
CVM-Net w/ $\mathcal{L}_{M}$ | 83.67 | 107.73 | 118.72 | 95.51 | 113.39 | 131.98 | 82.64 | 221.00
CVM-Net w/ $\mathcal{L}_{\Delta}$ | 80.77 | 82.53 | 104.96 | 87.07 | 101.80 | 107.00 | 77.41 | 98.85

MPJPE | Sit. | SitD. | Smok. | Wait. | WalkD. | Walk. | WalkT. | Avg
---|---|---|---|---|---|---|---|---
CVM-Net | 152.75 | 248.98 | 119.87 | 105.45 | 261.62 | 87.20 | 87.81 | 142.47
CVM-Net w/ $\mathcal{L}_{M}$ | 148.74 | 232.07 | 113.56 | 95.76 | 195.00 | 85.04 | 83.95 | 127.99
CVM-Net w/ $\mathcal{L}_{\Delta}$ | 137.33 | 178.99 | 103.43 | 84.37 | 104.32 | 79.16 | 76.30 | 100.62

(b) MPJVE comparison results in mm/frame

MPJVE | Dir. | Dis. | Eat. | Greet. | Phon. | Phot. | Pos. | Purch. | Sit.
---|---|---|---|---|---|---|---|---|---
CVM-Net | 3.34 | 4.63 | 3.39 | 4.73 | 3.11 | 3.87 | 3.17 | 9.21 | 2.78
CVM-Net w/ $\mathcal{L}_{M}$ | 3.25 | 4.73 | 3.15 | 4.44 | 2.91 | 3.84 | 2.95 | 11.39 | 2.54
CVM-Net w/ $\mathcal{L}_{\Delta}$ | 2.88 | 2.94 | 2.59 | 3.58 | 2.29 | 2.79 | 2.60 | 3.55 | 1.94

MPJVE | SitD. | Smok. | Wait. | WalkD. | Walk. | WalkT. | Avg
---|---|---|---|---|---|---|---
CVM-Net | 4.92 | 3.15 | 3.55 | 6.86 | 5.71 | 4.84 | 4.50
CVM-Net w/ $\mathcal{L}_{M}$ | 4.64 | 2.87 | 3.29 | 6.83 | 5.01 | 4.22 | 4.42
CVM-Net w/ $\mathcal{L}_{\Delta}$ | 3.22 | 2.25 | 2.56 | 4.13 | 4.20 | 3.60 | 3.01

The study is achieved with the benchmark Human3.6M [10] that contains millions
of frames of captured data. For each configuration previously defined, we
evaluate the MPJPE (through Protocol-1) and MPJVE using 2D joint locations
ground truth as input (obtained from the benchmark). The evaluation consists
of using the full dataset for the training-evaluation experiments. The dataset
is split into a training set and a test set according to the subjects whose
movements were captured (subjects 1, 5, 6 and 7 for training and subjects 9,
11 for evaluation).

Table 1 presents the comparative results obtained with the different loss
functions within the framework of Protocol 1 of the Human3.6M benchmark.
Compared to the baseline, we observe an average decrease of $14.48mm$ when we
add the Motion Loss $\mathcal{L}_{M}$ in the training process. In addition,
the Laplacian loss $\mathcal{L}_{\Delta}$ provides a significant improvement
of $41.85mm$ (average decrease in MPJPE). This shows that combining the joint
positions loss with our Laplacian loss significantly improves accuracy. It
also minimizes the velocity error by an average of $1.49mm/frame$. This shows
the efficiency of our spatio-temporal loss function that combines both the
spatial relationships (between joints of the same skeleton) and the temporal
relationships (between joints of consecutive frames).

Some visualisation results are displayed in Figure 3.

Figure 3: Visualisation results of CVM-Net + $\mathcal{L}_{\Delta}$ on
Human3.6m. On the left the input as 2D joint locations. On the middle our
prediction and on the right the ground truth.

###  6.2 Evaluation Metrics Study

Table 2: Comparison with state-of-the-art solutions. a) Comparison with MPJPE
under Protocol-1, b) MPJVE comparison, c) MPJAccE comparison. Best results are
in bold. Legend : (${\ddagger}$) sequence-to-sequence approach, (${\dagger}$)
sequence-to-pose approach with multi-frames as input and single frame as
output, ($\ast$) pose-to-pose approach, f=frame

(a) MPJPE comparison results in mm.

MPJPE | Dir. | Dis. | Eat. | Greet. | Phon. | Phot. | Pos. | Purch.
---|---|---|---|---|---|---|---|---
Shi et al. [19] (${\ddagger}$) | 45.48 | 51.28 | 49.43 | 51.91 | 52.58 | 66.46 | 50.59 | 48.46
Pavllo et al. [17] (${\dagger}$) | 33.88 | 43.99 | 44.28 | 48.96 | 44.62 | 65.80 | 32.79 | 55.12
Zhao et al. [26] ($\ast$) | 38.62 | 43.08 | 35.89 | 40.15 | 40.85 | 50.14 | 42.56 | 40.40
Ours(+$\mathcal{L}_{\Delta}$) (${\ddagger}$) | 80.78 | 82.50 | 104.44 | 87.03 | 101.16 | 106.88 | 77.43 | 98.24

MPJPE | Sit. | SitD. | Smok. | Wait. | WalkD. | Walk. | WalkT. | Avg
---|---|---|---|---|---|---|---|---
Shi et al. [19] (${\ddagger}$) | 55.90 | 64.25 | 53.79 | 52.84 | 58.85 | 49.99 | 48.25 | 53.47
Pavllo et al. [17] (${\dagger}$) | 45.61 | 48.09 | 57.29 | 47.09 | 45.16 | 43.30 | 46.67 | 46.84
Zhao et al. [26] ($\ast$) | 47.81 | 56.47 | 42.20 | 42.25 | 42.29 | 33.39 | 36.00 | 42.14
Ours(+$\mathcal{L}_{\Delta}$) (${\ddagger}$) | 136.73 | 178.48 | 102.98 | 84.33 | 103.65 | 79.24 | 76.39 | 100.34

(b) MPJVE comparison results in mm/f

MPJVE | Dir. | Dis. | Eat. | Greet. | Phon. | Phot. | Pos. | Purch.
---|---|---|---|---|---|---|---|---
Shi et al. [19] (${\ddagger}$) | 3.08 | 3.38 | 2.41 | 3.64 | 2.39 | 3.41 | 2.71 | 2.87
Pavllo et al. [17] (${\dagger}$) | 2.78 | 2.42 | 3.10 | 3.72 | 2.68 | 2.87 | 3.12 | 2.71
Zhao et al. [26] ($\ast$) | 2.57 | 2.84 | 2.40 | 3.41 | 2.14 | 2.60 | 2.56 | 2.92
Ours(+$\mathcal{L}_{\Delta}$) (${\ddagger}$) | 2.89 | 2.93 | 2.58 | 3.58 | 2.29 | 2.79 | 2.60 | 3.54

MPJVE | Sit. | SitD. | Smok. | Wait. | WalkD. | Walk. | WalkT. | Avg
---|---|---|---|---|---|---|---|---
Shi et al. [19] (${\ddagger}$) | 1.53 | 2.19 | 2.44 | 2.69 | 5.56 | 4.43 | 4.13 | 3.12
Pavllo et al. [17] (${\dagger}$) | 3.43 | 2.27 | 2.07 | 2.31 | 2.98 | 2.24 | 3.11 | 2.79
Zhao et al. [26] ($\ast$) | 1.57 | 2.18 | 2.02 | 2.53 | 3.83 | 4.02 | 3.49 | 2.74
Ours(+$\mathcal{L}_{\Delta}$) (${\ddagger}$) | 1.94 | 3.22 | 2.25 | 2.56 | 4.13 | 4.19 | 3.59 | 3.01

(c) MPJAccE comparison results in mm/f2

MPJAccE | Dir. | Dis. | Eat. | Greet. | Phon. | Phot. | Pos. | Purch.
---|---|---|---|---|---|---|---|---
Shi et al. [19] (${\ddagger}$) | 1.87 | 2.22 | 1.26 | 2.04 | 1.51 | 2.18 | 1.43 | 1.52
Pavllo et al. [17] (${\dagger}$) | 2.33 | 2.05 | 2.47 | 2.76 | 2.13 | 2.88 | 2.56 | 2.26
Zhao et al. [26] ($\ast$) | 1.74 | 1.94 | 1.61 | 2.40 | 1.40 | 1.81 | 1.64 | 2.15
Ours(+$\mathcal{L}_{\Delta}$) (${\ddagger}$) | 1.12 | 1.21 | 1.03 | 1.42 | 0.98 | 1.23 | 1.06 | 1.83

MPJAccE | Sit. | SitD. | Smok. | Wait. | WalkD. | Walk. | WalkT. | Avg
---|---|---|---|---|---|---|---|---
Shi et al. [19] (${\ddagger}$) | 0.66 | 1.04 | 1.50 | 1.58 | 4.57 | 3.17 | 2.90 | 1.96
Pavllo et al. [17] (${\dagger}$) | 2.72 | 2.05 | 2.09 | 2.07 | 2.34 | 1.81 | 2.64 | 2.34
Zhao et al. [26] ($\ast$) | 1.02 | 1.52 | 1.23 | 1.64 | 2.83 | 3.01 | 2.34 | 1.89
Ours(+$\mathcal{L}_{\Delta}$) (${\ddagger}$) | 0.88 | 1.91 | 0.88 | 1.03 | 1.91 | 1.93 | 1.50 | 1.33

Usually, human pose estimators are evaluated with the MPJPE metric which
calculates the average error of the joint positions for each pose. Very few
are evaluated with the MPJVE metric, which calculates the error on the
velocity of movement. In our case, since we focus on the temporal aspect of
motion, we added to MPJPE and MPJVE, a new acceleration-based metric, MPJAccE
(see section 5). We compare our approach with some state-of-the art solutions
using these three metrics. We chose three different types of approaches based
on their input and output settings (sequence-to-sequence, sequence-to-pose,
pose-to-pose). First the approach of Pavllo et al. [17] is a sequence-to-pose
approach from which we derived ours. The second solution from Shi et al. [19]
is one of the best sequence-to-sequence approaches in motion reconstruction
that makes use of forward kinematics. The knowledge of the skeleton
constraints (angle limits, bones lengths) is embedded in this model. Finally,
the solution from Zhao et al. [26] is a pose-to-pose approach that achieves
good results on 3D human pose estimation. Table 2 shows the results obtained
during our evaluation. Our approach using $\mathcal{L}_{\Delta}$, although far
behind the state-of-the art results for MPJPE ($2.5\times$ less accurate),
achieves good results for MPJVE and MPJAccE. The MPJVE results show that our
proposal is comparable to the other solutions, although it is a generic
approach. We achieve the best results for the MPJAccE metric. Since our
approach does not take into consideration the preservation of skeletal bone
lengths, unlike other methods, it is normal that the MPJPE measurement is
unfavourable to us. As for the MPJVE metric, which calculates the vector of
changes in position (both in velocity and direction), it filters in a certain
way the joint positions. The acceleration (second order derivative), which
characterizes the variations of the velocity, reflects another measure of
motion features that filters the velocity. The results on the two metrics
MPJVE and MPJAccE show that our method better preserves the temporal
characteristics of the original motion than other approaches.

##  7 Conclusion

In this paper we have presented a new spatio-temporal loss function based on
the representation of the motion as a 3D+t graph. We have shown that this
function improves both the spatial accuracy and the temporal consistency of 3D
sequence-to-sequence pose estimation for motion reconstruction. We have used a
temporal convolutional neural network for sequence-to-sequence pose estimation
on a large scale dataset Human3.6M. Although this generic model does not
challenge state-of-the art solutions on spatial accuracy with MPJPE
evaluation, it has proven the efficiency of the Laplacian Loss in the spatio-
temporal encoding of motion and the improvement of the temporal consistency.
Moreover, such a solution is to be preferred in a situation where we are
interested in the differential features (speed, acceleration), for the in-
depth analysis of the movement for example. Based on these preliminary
results, our future work will integrate this Laplacian loss with other
strategies – including bone length constraints – to improve the spatial
accuracy while preserving the current temporal consistency. We will also
integrate this loss function into the training of existing state-of-the-art
methods in order to further validate its efficiency and improve the results,
both on spatial and temporal aspects. Finally, we intend to propose a post-
processing method based on the Laplacian representation to correct the results
of the methods having obtained the best scores (spatial accuracy) and thus to
obtain a better temporal consistency.

In this work we also have proposed a new protocol for evaluating temporal
consistency in motion reconstruction through 3D sequence-to-sequence pose
estimation. Velocity and acceleration measurements provide metrics that extend
the classical position metric and allow to evaluate different solutions
according to criteria related to the temporal quality of motion.

## References

  * [1] Arsalan, M., Santra, A., Issakov, V.: Radar trajectory-based air-writing recognition using temporal convolutional network. pp. 1454–1459. Institute of Electrical and Electronics Engineers Inc. (12 2020). https://doi.org/10.1109/ICMLA51294.2020.00225
  * [2] Azizi, N., Possegger, H., Rodolà, E., Bischof, H.: 3d human pose estimation using möbius graph convolutional networks. ArXiv (2022)
  * [3] Biswas, S., Sinha, S., Gupta, K., Bhowmick, B.: Lifting 2d human pose to 3d : A weakly supervised approach. pp. 1–9 (5 2019)
  * [4] Cai, Y., Ge, L., Liu, J., Cai, J., Cham, T.J., Yuan, J., Thalmann, N.M.: Exploiting spatial-temporal relationships for 3d pose estimation via graph convolutional networks. Proceedings of the IEEE International Conference on Computer Vision 2019-Octob, 2272–2281 (2019). https://doi.org/10.1109/ICCV.2019.00236
  * [5] Cao, Z., Hidalgo, G., Simon, T., Wei, S.E., Sheikh, Y.: Openpose: Realtime multi-person 2d pose estimation using part affinity fields. Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 2017-Janua, 1302–1310 (12 2018), http://arxiv.org/abs/1812.08008
  * [6] Chen, C.H., Tyagi, A., Agrawal, A., Drover, D., MV, R., Stojanov, S., Rehg, J.M.: Unsupervised 3d pose estimation with geometric self-supervision. pp. 5707–5717 (4 2019)
  * [7] Chen, T., Fang, C., Shen, X., Zhu, Y., Chen, Z., Luo, J.: Anatomy-aware 3d human pose estimation with bone-based pose decomposition. IEEE Transactions on Circuits and Systems for Video Technology 32, 198–209 (2022). https://doi.org/10.1109/TCSVT.2021.3057267
  * [8] Dabral, R., Mundhada, A., Kusupati, U., Afaque, S., Sharma, A., Jain, A.: Learning 3d human pose from structure and motion. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 11213 LNCS, 679–696 (2018)
  * [9] Habibie, I., Xu, W., Mehta, D., Pons-Moll, G., Theobalt, C.: In the wild human pose estimation using explicit 2d features and intermediate 3d representations. pp. 10897–10906 (4 2019)
  * [10] Ionescu, C., Papava, D., Olaru, V., Sminchisescu, C.: Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine Intelligence 36, 1325–1339 (2014). https://doi.org/10.1109/TPAMI.2013.248
  * [11] Kiranyaz, S., Avci, O., Abdeljaber, O., Ince, T., Gabbouj, M., Inman, D.J.: 1d convolutional neural networks and applications: A survey. Mechanical Systems and Signal Processing 151, 107398 (4 2021). https://doi.org/10.1016/j.ymssp.2020.107398
  * [12] Luvizon, D.C., Picard, D., Tabia, H.: 2d/3d pose estimation and action recognition using multitask deep learning. pp. 5137–5146. IEEE (2018). https://doi.org/10.1109/CVPR.2018.00539
  * [13] Martinez, J., Hossain, R., Romero, J., Little, J.J.: A simple yet effective baseline for 3d human pose estimation. Proceedings of the IEEE International Conference on Computer Vision 2017-Octob, 2659–2668 (2017). https://doi.org/10.1109/ICCV.2017.288
  * [14] Mehta, D., Rhodin, H., Casas, D., Fua, P., Sotnychenko, O., Xu, W., Theobalt, C.: Monocular 3d human pose estimation in the wild using improved cnn supervision (2017)
  * [15] Mehta, D., Sridhar, S., Sotnychenko, O., Rhodin, H., Shafiei, M., peter Seidel, H., Casas, D., Theobalt, C., peter Seidel, H., Xu, W.: Vnect real-time 3d human pose estimation with a single rgb camera. ACM Transactions on Graphics (SIGGRAPH 2017) 36, 1–13 (2017)
  * [16] Naour, T.L., Courty, N., Gibet, S.: Spatiotemporal coupling with the 3d+t motion laplacian. Computer Animation and Virtual Worlds 24, 419–428 (2013). https://doi.org/https://doi.org/10.1002/cav.1518, https://onlinelibrary.wiley.com/doi/abs/10.1002/cav.1518
  * [17] Pavllo, D., Feichtenhofer, C., Grangier, D., Auli, M.: 3d human pose estimation in video with temporal convolutions and semi-supervised training. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2019-June, 7745–7754 (2019). https://doi.org/10.1109/CVPR.2019.00794
  * [18] Rogez, G., Weinzaepfel, P., Schmid, C.: Lcr-net++: Multi-person 2d and 3d pose detection in natural images. IEEE Transactions on Pattern Analysis and Machine Intelligence 42, 1146–1161 (2020). https://doi.org/10.1109/TPAMI.2019.2892985
  * [19] Shi, M., University, S., ABERMAN, B.F.A.K., Aberman, K., Aristidou, A., Komura, T., Lischinski, D.: Motionet: 3d human motion reconstruction from monocular video with skeleton consistency. ACM Transactions on Graphics 40 (2020). https://doi.org/10.1145/3407659, https://doi.org/10.1145/3407659
  * [20] Shimada, S., Golyanik, V., Xu, W., Theobalt, C.: Physcap: Physically plausible monocular 3d motion capture in real time. Real ACM Trans. Graph 39, 16 (2020). https://doi.org/10.1145/3414685.3417877, https://doi.org/10.1145/3414685.3417877
  * [21] Wang, J., Yan, S., Xiong, Y., Lin, D.: Motion guided 3d pose estimation from videos. European Conference on Computer Vision (ECCV) pp. 764–780 (2020)
  * [22] Wei, G., Wu, S., Tang, K., Li, G.: Bonenet: Real-time 3d human pose estimation by generating multiple hypotheses with bone-map representation. Computer-Aided Design and Applications 18, 1448–1465 (2021). https://doi.org/10.14733/cadaps.2021.1448-1465, https://doi.org/10.14733/cadaps.2021.1448-1465
  * [23] Xu, J., Yu, Z., Ni, B., Yang, J., Yang, X., Zhang, W.: Deep kinematics analysis for monocular 3d human pose estimation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition pp. 896–905 (2020). https://doi.org/10.1109/CVPR42600.2020.00098
  * [24] Yang, W., Ouyang, W., Wang, X., Ren, J., Li, H., Wang, X.: 3d human pose estimation in the wild by adversarial learning. pp. 5255–5264 (3 2018)
  * [25] Zhang, J., Tu, Z., Yang, J., Chen, Y., Yuan, J.: Mixste: Seq2seq mixed spatio-temporal encoder for 3d human pose estimation in video. ArXiv abs/2203.00859 (2022), http://arxiv.org/abs/2203.00859
  * [26] Zhao, L., Peng, X., Tian, Y., Kapadia, M., Metaxas, D.N.: Semantic graph convolutional networks for 3d human pose regression. vol. 2019-June, pp. 3420–3430. IEEE Computer Society (6 2019). https://doi.org/10.1109/CVPR.2019.00354
  * [27] Zou, Z., Tang, W.: Modulated graph convolutional network for 3d human pose estimation. IEEE/CVF International Conference on Computer Vision (ICCV) pp. 11477–11487 (2021), https://github.com/