# Mapping the research software sustainability space

Stephan Druskat  Dept. of German Studies and Linguistics
Humboldt-Universität zu Berlin
Berlin, Germany
<EMAIL_ADDRESS>Daniel S. Katz  NCSA, CS, ECE, iSchool
University of Illinois Urbana-Champaign
Urbana, IL, USA
<EMAIL_ADDRESS>
###### Abstract

A growing number of largely uncoordinated initiatives focus on research
software sustainability. A comprehensive mapping of the research software
sustainability space can help identify gaps in their efforts, track results,
and avoid duplication of work. To this end, this paper suggests enhancing an
existing schematic of activities in research software sustainability, and
formalizing it in a directed graph model. Such a model can be further used to
define a classification schema which, applied to research results in the
field, can drive the identification of past activities and the planning of
future efforts.

###### Index Terms:

scientific computing; sustainable development; modeling; visualization;
research software; communities;

††publicationid: pubid: ©2018 IEEE

The number of activities with a focus on the sustainability of research
software has increased over the last few years. Its major proponents are the
Software Sustainability Institute (software.ac.uk) (SSI), founded in 2010 in
the UK, the international WSSSPE organization
(wssspe.researchcomputing.org.uk) with its workshop series and working groups,
and to a certain extent, the international community of Research Software
Engineers (rse.ac.uk/community/international-rse-groups), which also
originated in the UK. This trend is ongoing, as the planning for an
institution similar to the SSI in the US, the US Research Software
Sustainability Institute (URSSI, urssi.us), and the recent formation of the
Better Scientific Software community (BSSw, bssw.io) show. Additionally,
further entities are active in the research software sustainability space
(“the space”), on different levels. These include, for example, working
groups, special interest groups, and others, on local, institutional, or
disciplinary levels.

All the above-mentioned entities conduct research on the sustainability of
research software, and publish in different venues, e.g., through papers, blog
posts, as well as talks and presentations at workshops and conferences. Their
research feeds back into academic institutions and associations on the policy
and education levels, and increasingly with regard to human resources. Their
findings also inform educational activities such as The Carpentries
(carpentries.org), which teaches foundational coding to researchers, including
specialized communities, such as the high performance computing and digital
humanities communities.

These efforts are not currently formally coordinated, and it is unclear
whether formal – or even central – coordination is at all useful, desirable,
or achievable. After all, the dynamic nature of, e.g., small working groups,
workshops, and spontaneous events such as hack days are often more
constructive and productive than larger, coordinated efforts. The autonomy of
activities bears the risk of unnecessary duplication of efforts, as well as
neglect of specific areas of the space.

##  I Mapping the space

A comprehensive mapping of the research software sustainability space can help
to reduce this risk and support further efforts.

Existing classification schemes, such as the ACM’s Computing Classification
System (acm.org/about-acm/class), are unsuitable for such a mapping, as they
do not reflect the specific configuration of this domain in terms of agents
and activities.

A first mapping was introduced by Katz [1] in a directed graph schematic,
reproduced in Figure 1. This visualization lists important parties (nodes) and
activities (edges) in the space.

Figure 1: Reproduction of Katz’ original sustainability schematic [1, p. 3].

Figure 2: Visualization of the formalized resosuma model of Katz’
sustainability schematic.

Based on this initial concept, a more comprehensive mapping can be built.
Advances may include disassembling combined activities, a higher resolution
and more precise definition of involved parties, as well as adding potentially
missing agents.

Additionally, formalizing the mapping can make it useful for further
processing and would benefit the community: it enables different efforts in
the space to be classified based on the parties and activities they cover.
This includes different possible formats, such as research outputs (papers,
blog posts, talks), events, groups, and projects. Such a classification in
turn enables the gaps in community efforts to be identified via quantitative
methods, e.g., publication and event analysis. These gaps can then be closed
by initiating new efforts, e.g., workshops and projects in these areas.

###  I-A Initial formalization

In a first step towards a formalization, we mapped Katz’ original
sustainability schematic (Figure 1) to a _resosuma_ CSV representation of
activities [2]. In the process, activities that are combined in the schematic
(i.e., edge labels that are lists of verbs) were disassembled, so that each
verb (the _action_) has its own row, with its subject (the _actor_) in the
cell to its left and its object (the _actee_) in the cell to its right.

Figure 2 shows a visualization of the data, created with the _resosuma_[3]
Python package.

###  I-B Refinement

In a second step, we made some changes to the data to more comprehensively
represent activities in the research software sustainability space, at a
higher resolution [4].

The changes have been introduced based on introspection alone, and in order to
reach a sufficient level of comprehensiveness, future refinement work towards
a stable version of the model should include a structured literature analysis.

The respective _resosuma_ visualization is given in Figure 3. The changes
included the following.

Differentiating the “People” node into “Research Software Engineers”,
“Researchers” and “Educators” nodes, and defining related activities
respectively. This differentiates between groups of _roles_ that have distinct
scopes of responsibilities within the space. While “Research Software
Engineers” are active in producing and publishing research software and
applying “Software Engineering Processes”, “Researchers” are active in using
research software and publishing research products (but not research
software), but have little direct stake in “Software Engineering Processes”.
These two roles rely on different requirements to be fulfilled within the
space. Similarly, “Educators”, including academic staff, Carpentry
instructors, and training consultants, do not use research software in this
role, but disseminate “Software Engineering Processes” for example.

Differentiating the “Publishers, repositories, indices” node into separate
nodes, and defining related activities respectively. We split these parties
over separate nodes because they play differing roles in the space.
Additionally, we split “repositories” into “code platforms” and “archival
repositories”. “Code platforms” serve the purpose of interacting with code,
allow operations like forking, branching, and often provide further
functionality for collaboration. Well-known examples include GitHub
(github.com), GitLab (gitlab.com) and Bitbucket (bitbucket.org), but “Code
platforms” may also include simpler version control system repositories.
“Archival repositories” archive and publish versions of research products
including software, but are usually not interactive. Examples include general
purpose data repositories such as Zenodo (zenodo.org) or figshare
(figshare.com), build artifact repositories such as The Central Repository
(“Maven Central”, search.maven.org), the Python Package Index (PyPI,
pypi.org), and others, e.g., preprint repositories such as arXiv.org
(arxiv.org).

Differentiating the “Software” node into “Research software” and “Supporting
software” nodes, and defining related activities respectively. While the
differentiated nodes represent similar objects, the ways they are produced and
used, and the ways they are referenced and rewarded differ remarkably.
Additionally, research software is the defining entity at the core of the
space, and “supporting software” can exist independently of it while
potentially having a great effect on the space. “Supporting software” covers
software that is used to create research software, including everything from
editors and integrated development environments to version control system
platforms.

The “Software Engineering Processes” node was left unchanged, but it should be
noted that it includes a wide range of processes and practices, including best
practices around metadata, citation, etc.

“are” has been introduced as a specific relation in the space, where “Research
Software Engineers” and “Researchers” are nodes that may reference different
roles for the same individual. As this relation is at the heart of some of the
activities in the space, most notably the international RSE community, it
should be explicitly included in the mapping.

Further activities were added. We established an “own” activity, for example,
from “Research Software Engineers” to “Research Software” in addition to the
one from “Hiring organizations” to “Research Software” to represent potential
copyright of an RSE on their work (cf. [5]).

A visualization of the refined resosuma data is given in Table II and Figure
3. A more optimal visualization of this data would be an interactive, zoomable
graph visualization in a web application.

##  II Future work

The data is openly maintained on GitHub (github.com/research-
software/resosuma-data), and we propose that the community should collaborate
to

  1. 1.

complete it to create iterative releases which reflect state changes in the
research software sustainability space; and

  2. 2.

build a classification scheme on it that can be used to classify efforts.

The classification could take the form of simple handles for activities, e.g.,
_resosuma:rse-dev-rso_ for the activity “Research Software Engineers develop
research software”. The classification could then be used to tag efforts,
retrospectively where possible. This is easily done for publications, where
classification handles can be included in keywords or in the body, but should
also be done for projects, working groups, events, etc., for example on their
websites.

The classification could be used for automated extraction and quantitative
analysis, and could also serve to build a registry of efforts across formats
that would make it even easier to identify gaps and track progress.

Additionally, the classification could be used by funders and institutions to
analyze their portfolio for reporting and planning, similar to the NIH’s
(nih.gov) use of the “Research, Condition, and Disease Categorization Process”
(RCDC, report.nih.gov/rcdc/).

The level of granularity of the activity graph should be discussed within the
community, and if deemed necessary and helpful, different resolutions of the
mapping could be created, e.g., a low-res version that would look similar to
Figure 2, and a high-res version that would look similar to an optimized
version of Figure 3. Both versions can be codified in different versions of
the classification scheme, similarly to what has been done in ISO 639
(en.wikipedia.org/wiki/ISO_639), with a low-res classification in ISO 639-1
and a more high-res classification in ISO 639-3.

Future refinement work towards a stable version of the model should also
include a structured review of the available literature and past activities.

To promote use of the classification, conferences, workshops and similar
events, and editors of publications in the research software sustainability
space should ask for contributions to include the applicable resosuma handles
in their keywords sections, once these handles have been developed.
Alternatively, reviewers of these contributions could be asked to classify
them during the review process.

Additionally, researchers should be encouraged to add resosuma handles that
cover their expertise to their public profiles, e.g., their ORCID (orcid.org)
profiles. This will enable easier identification of potentially suitable
reviewers for future contributions within the space.

To facilitate access to the handles and the map itself, and to accumulate
future literature and activity analyses, the community should develop a
central resource for resosuma, e.g., a community-curated website. Such a
resource should also include a more accessible visualization than has been
possible here in Figure 3 and Table II, e.g., in an interactive webpage.

Figure 3: Visualization of the refined resosuma model [4].

TABLE I: Activities modeled in [4].

Actor Action Actee Actor Action Actee Research Software Engineers collaborate
Research Software Engineers Code platforms measure Research software Research
Software Engineers support Researchers Code platforms store Supporting
software Research Software Engineers are Researchers Code platforms measure
Supporting software Research Software Engineers publish Publishers Archival
repositories recognize Research Software Engineers Research Software Engineers
publish Code platforms Archival repositories recognize Researchers Research
Software Engineers publish Archival repositories Archival repositories archive
Research software Research Software Engineers disseminate Archival
repositories Archival repositories measure Research software Research Software
Engineers develop Research software Archival repositories archive Supporting
software Research Software Engineers test Research software Archival
repositories measure Supporting software Research Software Engineers design
Research software Indices index Research software Research Software Engineers
reuse Research software Indices recognize Research Software Engineers Research
Software Engineers own Research software Indices recognize Researchers
Research Software Engineers use Supporting software Research software depends
Research software Research Software Engineers join Communities Research
software use Hardware & system software Research Software Engineers propose
Funding organizations Supporting software use Hardware & system software
Research Software Engineers use Software engineering processes Communities
recognize Research Software Engineers Research Software Engineers develop
Software engineering processes Communities reward Research Software Engineers
Researchers collaborate Research Software Engineers Communities recognize
Researchers Researchers are Research Software Engineers Communities reward
Researchers Researchers publish Publishers Communities recognize Educators
Researchers publish Archival repositories Communities reward Educators
Researchers disseminate Archival repositories Communities standardize Software
engineering processes Researchers use Research software Funding organizations
recognize Research Software Engineers Researchers develop Research software
Funding organizations reward Research Software Engineers Researchers reproduce
Research software Funding organizations recognize Researchers Researchers cite
Research software Funding organizations reward Researchers Researchers
research Research software Funding organizations fund Hiring organizations
Researchers use Supporting software Hiring organizations hire Research
Software Engineers Researchers join Communities Hiring organizations promote
Research Software Engineers Researchers propose Funding organizations Hiring
organizations recognize Research Software Engineers Researchers research
Software engineering processes Hiring organizations reward Research Software
Engineers Educators train Research Software Engineers Hiring organizations
hire Researchers Educators educate Research Software Engineers Hiring
organizations promote Researchers Educators train Researchers Hiring
organizations recognize Researchers Educators educate Researchers Hiring
organizations reward Researchers Educators educate Communities Hiring
organizations hire Educators Educators join Communities Hiring organizations
promote Educators Educators educate Funding organizations Hiring organizations
recognize Educators Educators educate Hiring organizations Hiring
organizations reward Educators Educators disseminate Software engineering
processes Hiring organizations own Code platforms Publishers reward Research
Software Engineers Hiring organizations own Archival repositories Publishers
recognize Research Software Engineers Hiring organizations own Research
software Publishers reward Researchers Hiring organizations measure Research
software Publishers recognize Researchers Hiring organizations own Supporting
software Publishers recognize Research software Hiring organizations formalize
Software engineering processes Publishers address Communities Hiring
organizations require Software engineering processes Code platforms recognize
Research Software Engineers Hiring organizations own Hardware & system
software Code platforms store Research software

## Acknowledgments

The authors would like to thank Neil Chue Hong for early feedback and
contributions to the data model.

S. Druskat would like to acknowledge funding assistance from the Software
Sustainability Institute. The Software Sustainability Institute is supported
by the EPSRC, BBSRC and ESRC Grant EP/N006410/1.

## References

  * [1] D. S. Katz, “Research Software Sustainability: WSSSPE & URSSI,” Apr. 2018. [Online]. Available: https://doi.org/10.6084/m9.figshare.6081248.v1
  * [2] S. Druskat, “research-software/resosuma-data: 0.1.0,” Jun. 2018. [Online]. Available: https://doi.org/10.5281/zenodo.1299250
  * [3] ——, “resosuma: 0.2.0,” Jun. 2018. [Online]. Available: https://doi.org/10.5281/zenodo.1304254
  * [4] S. Druskat, D. S. Katz, and N. Chue Hong, “research-software/resosuma-data: 0.4.0,” Sep. 2018. [Online]. Available: https://doi.org/10.5281/zenodo.1419768
  * [5] N. Forgó, “Legal requirements for software sharing and collaboration,” 2017. [Online]. Available: https://doi.org/10.5446/31029


# Synchronization of oscillators with hyperbolic chaotic phases

Arkady Pikovsky  Institute of Physics and Astronomy, Potsdam University, 14476
Potsdam-Golm, Germany  Department of Control Theory, Nizhny Novgorod State
University, Gagarin Avenue 23, 603950 Nizhny Novgorod, Russia  National
Research University Higher School of Economics,
25/12 Bolshaya Pecherskaya Ulitsa, 603155 Nizhny Novgorod, Russia

###### Abstract

Synchronization in a population of oscillators with hyperbolic chaotic phases
is studied for two models. One is based on the Kuramoto dynamics of the phase
oscillators and on the Bernoulli map applied to these phases. This system
possesses an Ott-Antonsen invariant manifold, allowing for a derivation of a
map for the evolution of the complex order parameter. Beyond a critical
coupling strength, this model demonstrates bistability synchrony-disorder.
Another model is based on the coupled autonomous oscillators with hyperbolic
chaotic strange attractors of Smale-Williams type. Here a disordered
asynchronous state at small coupling strengths, and a completely synchronous
state at large couplings are observed. Intermediate regimes are characterized
by different levels of complexity of the global order parameter dynamics.

##  I Introduction

Synchronization of chaotic oscillators has many aspects Pikovsky  _et al._
(2001), one generally distinguishes complete, generalized, and phase
synchronization. The latter property is related to chaotic oscillators with
well-defined phases. Many chaotic oscillators, like the Rössler system,
possess chaotic amplitudes, while the phase in such systems is not chaotic and
corresponds to a zero Lyapunov exponent. The dynamics of the phase is of
diffusion type, and correspondingly the phase synchronization phenomena for
such oscillators are close to those for periodic oscillators with a certain
level of noise in the phase dynamics.

In a seminal paper Kuznetsov (2005) S. P. Kuznetsov constructed a physical
model of an oscillator with a chaotic phase. In this construction, the process
has amplitude modulation, and at each period of modulation the phase
experience a doubling map. The overall attractor is hyperbolic and belongs to
a Smale-Williams solenoid class. In a series of subsequent publications,
summarized in the book Kuznetsov (2012), S. P. Kuznetsov and co-authors
provided many examples of systems with hyperbolic phase chaos.

In this paper we study synchronization properties of the oscillators with
chaotic phases. First, we construct a rather abstract model, where phase chaos
and synchronizing interactions are separated in time (Section II). Namely, the
process consists of two epochs: in one epoch phase oscillators interact
according to the Kuramoto global coupling scheme, and in another epoch the
phases undergo a chaotic Bernoulli map. This model demonstrates, for certain
values of parameters, a bistability between a desynchronized and a
synchronized states. In Section III we consider coupled autonomous oscillators
with chaotic phases, constructed by S. P. Kuznetsov and the author in
Kuznetsov and Pikovsky (2007). This system demonstrates a rather reach
behavior with asynchronous, completely synchronous, and complex partially
synchronous states.

##  II Kuramoto-Bernoulli model

In this section we construct a model of interacting phase oscillators, which
combines features of the Kuramoto model Acebrón  _et al._ (2005) (global
attractive coupling of the phases) with the hyperbolic chaotic dynamics of the
phases described by a Bernoulli map.

###  II.1 Kuramoto ensemble and OA evolution

Consider $N$ phase oscillators $\varphi_{k}$ interacting via Kuramoto mean-
field coupling

$\dot{\varphi}_{k}=\mu R\sin(\Theta-\varphi_{k}),\quad
Z=Re^{i\Theta}=\frac{1}{N}\sum_{k}e^{i\varphi_{k}}\;.$ (1)

Here $Z$ is the complex mean field, and $\mu$ is the coupling constant.
Quantity $R$ is called Kuramoto order, it characterizes asynchronous ($R=1$)
and synchronous ($R=1$) regimes. We assume that all the oscillators have the
same frequency, and write equations in the reference frame where this
frequency vanishes, so it does not enter in (1). For $\mu>0$ the coupling is
attractive, and in this situation all the oscillators eventually synchronize:
$R\to 1$, and a state where $\varphi_{1}=\varphi_{2}=\ldots\varphi_{N}$
establishes.

Synchronization transition is monotonous (in fact, there exists a Lyapunov
function that governs it), but it can be generally hardly expressed
analytically. An analytic solution is, however, possible, if the Ott-Antonsen
(OA) ansatz Ott and Antonsen (2008), which applies to the thermodynamic limit
$N\to\infty$, is peformed. In the OA ansatz it is assumed that the
distribution of the phases is a wrapped Cauchy distribution, and the complex
circular moments

$Z_{k}=\langle e^{ik\varphi}\rangle$ (2)

can all be expressed via the complex mean field $Z_{k}=Z^{k}$. Then the
equation for the order parameter reads Ott and Antonsen (2008)

$\dot{R}=\frac{\mu}{2}R(1-R^{2})\;.$

Evolution of the complex mean field during a time interval $T$ is

$R(T)=\frac{R(0)}{\sqrt{R^{2}(0)+(1-R^{2}(0))\exp[-\mu T]}}$ (3)

One can see that the only parameter in this transformation is
$\gamma=\exp(-\mu T)$. Evidently, $R\to 1$ as $n\to infty$, and the rate of
this convergence is larger for smaller $\gamma$.

###  II.2 Bernoulli map of phases

Consider a Bernoulli map acting on the phases

$\varphi(n+1)=K\varphi(n)\;,$ (4)

with an integer parameter $K$. For an ensemble of Bernoulli maps (4), it is
easy to express the evolution of the probability density of phases through the
complex circular moments (2):

$Z_{m}(n+1)=Z_{Km}(n)\;.$

One can see that the OA ansatz is invariant under Bernoilli maps. Indeed, if
$Z_{m}(n)=Z^{m}$, then $Z_{m}(n+1)=Z^{Km}=(Z^{K})^{m}$. Thus, the evolution of
the complex mean field under the Bernoulli map is

$Z(n+1)=Z^{K}(n)\;.$ (5)

###  II.3 Kuramoto ensemble and Bernoilli map

We construct a Kuramoto-Bernoulli (KB) model as a sequence of applications of
the Kuramoto dynamics (3) and of the Bernoulli dynamics (5). Application of
the expressions (3), (5) leads to the following map for the order parameter

$R(n+1)=\frac{R^{K}(n)}{(R^{2}(n)+(1-R^{2}(n))\gamma)^{K/2}}$

This map has always a stable asynchronous fixed point $R_{as}=0$, and a
synchronous fixed point $R_{s}=1$. The fixed point $R=1$ is stable for

$\gamma<\frac{1}{K}\;,$ (6)

in this case also an unstable partially synchronous fixed point with
$0<R_{ps}<1$ exists, so there is a bistability asynchrony-synchrony.

The threshold for synchrony stability (6) is valid not only in the OA
approximation, but generally. Indeed, close to the synchronous state the
deviations of the phases satisfy, in the Kuramoto stage, the linear equation

$\frac{d}{dt}\delta\varphi=-\mu\delta\varphi$

so that combined map for the linear deviations is

$\delta\varphi(n+1)=K\gamma\delta\varphi(n)$

from which (6) follows.

We illustrate the dynamics of the KB model in Fig. 1. There we show the
evolution of the oder parameter $R$ for different values of parameter $\gamma$
and different initial states. The fully synchronous state is absorbing
(exactly the same phases remain the same) for all system sizes $N$, while
there are finite-size fluctuations around the disordered state. For small
$\gamma$, one observes a finite-size induced transition to the synchronous
state.

Figure 1: Illustration of the dynamics in a Kuramoto-Bernoulli model with
$K=2$ and different $\gamma$, ensemble size $N=1000$. Red curves: disordered
initial state; blue curves: ordered initial state (distribution of phases in
an interval $(0,0.1)$).

##  III Globally coupled Kuznetsov-Pikovsky (KP) oscillators

Here we study globally coupled chaotic phase oscillators introduced by S. P.
Kuznetsov and the author in Ref. Kuznetsov and Pikovsky (2007).

###  III.1 One KP oscillator

An individual KP oscillator consists of three modes described by their complex
amplitudes $u,v,w$. The equation of one unit are

$\displaystyle\dot{u}$
$\displaystyle=-iu+(1-|u|^{2}-\frac{1}{2}|v|^{2}-2|w|^{2})u+\varepsilon\text{Im}(v^{2})\;,$
(7) $\displaystyle\dot{v}$
$\displaystyle=-iv+(1-|v|^{2}-\frac{1}{2}|w|^{2}-2|u|^{2})v+\varepsilon\text{Im}(w^{2})\;,$
$\displaystyle\dot{w}$
$\displaystyle=-iw+(1-|w|^{2}-\frac{1}{2}|u|^{2}-2|v|^{2})u+\varepsilon\text{Im}(u^{2})\;.$

Below we fix the internal coupling parameter $\varepsilon=0.075$. For
$\varepsilon=0$, system (7) has a stable homoclinic cycle, where the modes are
excited consequentially $w\to v\to u\to w\to\ldots$, with increasing periods
of the cycle. The effect of coupling $\varepsilon>0$ is twofold: first, the
cycle period is limited from above (see Fig. 2(b)), and second, at each stage
where a mode amplitude passes close to zero, its phase attains the doubled
value of the exciting mode. The latter property is described in Ref. Kuznetsov
and Pikovsky (2007) in details; here we illustrate it with figure 3. Thus, the
KP oscillator (7) has a chaotic phase obeying a Bernoulli map.

(a) (b)

Figure 2: Illustration of the dynamics in a KP model. Panel (a): phase
portrait of the observable $z=u+v+w$ (complex amplitude of oscillations).
Panel (b): amplitudes of the modes. Figure 3: Illustration of the phase
transformation in model (7). The consecutive phases
$\text{arg}(u)\to\text{arg}(v)\to\text{arg}(w)$ are depicted. The
transformation $\text{arg}(u)\to\text{arg}(u)$ would be $\varphi\to
2^{3}\varphi$.

###  III.2 Globally coupled KP oscillators

Here we introduce a global coupling of $N$ oscillators (numbered by index
$k$), such that complete synchrony is possible:

$\displaystyle\dot{u}_{k}$
$\displaystyle=-iu_{k}+(1-|u_{k}|^{2}-\frac{1}{2}|v_{k}|^{2}-2|w_{k}|^{2})u+\varepsilon\text{Im}(v_{k}^{2})+\mu|u_{k}|^{2}(U-u_{k}),$
(8) $\displaystyle\dot{v}_{k}$
$\displaystyle=-iv_{k}+(1-|v_{k}|^{2}-\frac{1}{2}|w_{k}|^{2}-2|u_{k}|^{2})v+\varepsilon\text{Im}(w_{k}^{2})+\mu|v_{k}|^{2}(V-v_{k}),$
$\displaystyle\dot{w}_{k}$
$\displaystyle=-iw_{k}+(1-|w_{k}|^{2}-\frac{1}{2}|u_{k}|^{2}-2|v_{k}|^{2})u+\varepsilon\text{Im}(u_{k}^{2})+\mu|w_{k}|^{2}(W-w_{k}),$
$\displaystyle U=\frac{1}{N}\sum_{k}u_{k},\quad
V=\frac{1}{N}\sum_{k}v_{k},\quad W=\frac{1}{N}\sum_{k}w_{k}\;.$

The coupling term is proportional to the parameter $\mu$, it contains three
complex mean fields $U,V,W$, corresponding to three modes of each oscillator.

Figure 4 intends to characterize the dynamical regimes in the system, in
dependence on the coupling parameter $\mu$. Here two quantities have been
calculated. First, I present the dynamics of the global complex mean field

$Z(t)=U(t)+V(t)+W(t)\;.$

I calculated the time average $\langle|Z|\rangle_{t}$ and its fluctuations
$\langle(|Z|-\langle|Z|\rangle_{t})^{2}\rangle_{t}$, these quantities are
shown in Fig. 4 with red (fluctuations as error bars). Additionally, for each
moment of time, I calculated the spread in the ensemble

$D(t)=\frac{1}{N}\sum_{k}\left[|u_{k}(t)-U(t)|^{2}+|v_{k}(t)-V(t)|^{2}+|w_{k}(t)-W(t)|^{2}\right]$

and then average this quantity over time. This quantity is shown with blue.

Figure 4: Bifurcation diagram of model (8) for $N=10^{4}$.

Below I describe different states on the bifurcation diagram.

  1. 1.

Complete synchronization. This regime is observed for $\mu>0.95$. Here
$u_{k}=u_{j},v_{k}=v_{j},w_{k}=w_{j}$ for all $k,j$. In this state $D=0$.

  2. 2.

Asynchronous state. This regime is observed for $\mu\lesssim 0.22$. Here the
mean field vanishes, and one has effectively a set of non-interacting
oscillators.

  3. 3.

Periodic mean field. This regime is observed in the range
$0.22\lesssim\mu\lesssim 0.31$. Here the complex mean field $Z(t)$ is nearly
periodic. We illustrate this in Fig. 5(a,b). There are visible fluctuations
for $\mu=0.23$, but for $\mu=0.3$ periodicity is nearly perfect. The
transition to a periodic mean field at $\mu\approx 0.22$ is very much similar
to one described in Ref. Pikovsky  _et al._ (1996).

  4. 4.

Weakly irregular mean field This state is illustrated in Fig. 5(c). At
$\mu=0.35$ the mean field is close to periodic one, but has a seemingly nearly
quasiperiodic modulation.

  5. 5.

Irregular mean field This state is observed for $0.4\lesssim\mu\lesssim 0.95$,
we illustrate it in Fig. 6 (a-c). Fluctuations of the mean field are
essential, eventually for large $\mu$ they become close to the fluctuations of
the field $z(t)$ in one chaotic oscillator.

(a) (b) (c)

Figure 5: The dynamics of the global complex order parameter for $\mu=0.23$
(panel (a)), $\mu=0.3$ (panel (b)), and $\mu=0.35$ (panel (c)).

(a) (b) (c)

Figure 6: The dynamics of the global complex order parameter for $\mu=0.45$
(panel (a)), $\mu=0.6$ (panel (b)), and $\mu=0.8$ (panel (c)).

##  IV Discussion

In this paper I studied effects of coupling on oscillators with hyperbolic
chaotic dynamics of the phases. In the simplest, rather artificial Kuramoto-
Bernoulli model, an exact mapping for the order parameter has been derived in
the Ott-Antonsen approximation in the thermodynamic limit. The dynamics here,
beyond a certain level of coupling, is bistable: synchronous and asynchronous
states coexist. In relatively small ensembles, for strong enough coupling,
only synchronous states survives as it is a truly absorbing one. A more
realistic model of coupled autonomous continuous-time oscillators with
hyperbolic dynamics of the phases demonstrated much more rich dynamics.
Together with a fully asynchronous state at small coupling strengths, and a
completely synchronous at strong coupling strengths, it demonstrates different
states with partial synchrony. Close to the asynchronous state, the mean field
is nearly periodic; and with increase of coupling strength it becomes
irregular through presumably a quasi-periodic state. Detailed study of
partially synchronous states in this model will be a subject of a separate
study.

###### Acknowledgements.

A. P. acknowledges support by the Russian Science Foundation (studies of
Section III, grant Nr. 17-12-01534) and by DFG (grant PI 220/21-1). Numerical
experiments in Sec. II were supported by the Laboratory of Dynamical Systems
and Applications NRU HSE, of the Russian Ministry of Science and Higher
Education (Grant No. 075-15-2019-1931).

## References

  * Pikovsky  _et al._ (2001) A. Pikovsky, M. Rosenblum, and J. Kurths,  _Synchronization. A Universal Concept in Nonlinear Sciences._ (Cambridge University Press, Cambridge, 2001).
  * Kuznetsov (2005) S. P. Kuznetsov, Phys. Rev. Lett. 95, 144101 (2005).
  * Kuznetsov (2012) S. P. Kuznetsov,  _Hyperbolic Chaos: A Physicist’s View_ (Springer, Heidelberg, 2012).
  * Kuznetsov and Pikovsky (2007) S. Kuznetsov and A. Pikovsky, Physica D 232, 87 (2007).
  * Acebrón  _et al._ (2005) J. A. Acebrón, L. L. Bonilla, C. J. P. Vicente, F. Ritort, and R. Spigler, Rev. Mod. Phys. 77, 137 (2005).
  * Ott and Antonsen (2008) E. Ott and T. M. Antonsen, CHAOS 18, 037113 (2008).
  * Pikovsky  _et al._ (1996) A. Pikovsky, M. Rosenblum, and J. Kurths, Europhys. Lett. 34, 165 (1996).


Richardson’s Forecast:

the Dream and the Fantasy

Peter<EMAIL_ADDRESS>
School of Mathematics and Statistics, University College Dublin.

> Perhaps, some day in the dim future it will be possible to advance the
> computations faster than the weather advances … . But that is a dream [WPNP,
> p. vii].

## Introduction

A remarkable book on weather forecasting was published just one hundred years
ago. Written by the brilliant and prescient applied mathematician, Lewis Fry
Richardson, _Weather Prediction by Numerical Process_ was published by
Cambridge University Press and went on sale in 1922 at a cost of 30 shillings
(£1.50). With a print run of just 750 copies, it was not a commercial success
and was still in print thirty years after publication. It was re-issued in
1965 as a Dover paperback. Cambridge University Press reprinted the book in
2007, with a foreword by Peter Lynch. Described as a second edition, it
differs in no essential way from the 1922 edition.

_Weather Prediction by Numerical Process_ (denoted WPNP) is a strikingly
original scientific work, one of the most remarkable books on meteorology ever
written. In it, Richardson described a systematic mathematical method for
predicting the weather and demonstrated its application by carrying out a
trial forecast. Richardson’s innovative approach was fundamentally sound, but
the method devised by him was utterly impractical at the time of its
publication and the results of his trial forecast appeared to be little short
of outlandish. As a result, his ideas were eclipsed for decades. For a brief
biographical sketch of Richardson, see the _Benchmark_ article by Lea (2012)
and for a full biography, see Ashford (1985).

## Background to Scientific Forecasting

The basic ideas of numerical forecasting and climate modelling were developed
more than a century ago, long before the first electronic computer was
constructed. American meteorologist Cleveland Abbe and Norwegian physicist
Vilhelm Bjerknes recognized that predictions of meteorological phenomena could
be based on the application of hydrodynamics and thermodynamics to the
atmosphere (Abbe 1901; Bjerknes 1904). They each specified the required
process: analysis of the initial state and time integration of the governing
equations from that state. They listed the system of mathematical equations
needed and outlined the steps required to produce a forecast. Realizing the
numerous practical difficulties, neither Abbe nor Bjerknes attempted to
implement their techniques.

A first tentative trial to produce a forecast using the laws of physics was
made by Felix Exner (1908), working in Vienna. His efforts yielded a realistic
forecast for the particular case that he chose. Despite the restricted
applicability of his technique, his work was a first attempt at systematic,
scientific weather forecasting. However, it involved some drastic
simplifications and was far from providing anything of use for practical
forecasting.

## _Weather Prediction by Numerical Process_

Lewis Fry Richardson first learned of Bjerknes’ plan for rational forecasting
in 1913, when he took up employment with the Meteorological Office.
Richardson’s forecasting scheme amounts to a precise and detailed
implementation of the prognostic component of Bjerknes’ programme. It is a
highly intricate procedure: as Richardson observed, “the scheme is complicated
because the atmosphere is complicated.” It also involved an enormous volume of
numerical computation and was quite impractical in the pre-computer era. But
Richardson was undaunted, expressing his dream that “some day in the dim
future it will be possible to advance the computations faster than the weather
advances.”

Earlier, Richardson had applied an approximate numerical method to the
solution of partial differential equations to investigate the stresses in
masonry dams. He realized that this method had potential for use in a wide
range of problems. The idea of numerical weather prediction appears to have
germinated in his mind for several years. Around 1911, Richardson had begun to
think about the application of his approach to the problem of forecasting the
weather. He stated in the Preface of WPNP that the idea “grew out of a study
of finite differences and first took shape in 1911 as a fantasy.” The fantasy
was that of a forecast factory, which we will discuss below.

Upon joining the Met Office, Richardson was appointed Superintendent of
Eskdalemuir Observatory in the Southern Uplands of Scotland and began serious
work on numerical forecasting. In May 1916 he resigned from the Met Office in
order to work with the Friends Ambulance Unit in France. By this time, he had
completed the formulation of his scheme and had set down the details in the
first draft of his book. But he was not concerned merely with theoretical
rigour and wished to include a fully worked example to demonstrate how the
method could be put to use.

Richardson assumed that the state of the atmosphere at any point could be
specified by seven numbers: pressure, temperature, density, water content and
velocity components eastward, northward and upward. He formulated a
description of atmospheric phenomena in terms of seven partial differential
equations. To solve them, he divided the atmosphere into discrete columns of
extent 3 degrees east-west and 200 km north-south, giving 120$\times$100 =
12,000 columns to cover the globe. Each of these columns was divided
vertically into five cells. The values of the variables were given at the
centre of each cell, and the differential equations were approximated by
expressing them in finite difference form. The rates of change of the
variables could then be calculated by arithmetical means. These rates enabled
Richardson to calculate the variables at a later time.

Richardson calculated the initial changes over a six hour period in two
columns over central Europe, one for mass variables and one for winds. This
was the extent of his forecast. In this trial forecast, he calculated a change
of atmospheric pressure, at a point near Munich, of 145 hPa in 6 hours. This
was a totally unrealistic value, two orders of magnitude too large. The
failure may be explained in terms of atmospheric dynamics. We return to the
cause of this after first considering the reaction of other researchers to
Richardson’s work.

## Initial Reactions to WPNP

The initial response to Richardson’s book was unremarkable and must have been
disappointing to him. WPNP was widely reviewed with generally favourable
comments, but the impracticality of the method and the abysmal failure of the
solitary sample forecast inevitably attracted adverse criticism. Napier Shaw,
reviewing the book for Nature, wrote that Richardson “presents to us a _magnum
opus_ on weather prediction.” However, in reference to the trial forecast, he
observed that “the wildest guess [at the pressure change] would not have been
wider of the mark.” He also questioned Richardson’s conclusion that wind
observations were the real cause of the error, and his dismissal of the
geostrophic wind.

Edgar W. Woolard, later an editor of _Monthly Weather Review_ , wrote a
positive review of the book for that journal, expressing the hope that other
investigators would be inspired by Richardson’s work to continue its
development. However, nobody else continued working along his lines, perhaps
because the forecast failure acted as a deterrent, perhaps because the book
was so difficult to read, with its encyclopaedic but distracting range of
topics. Alexander McAdie, Professor of Meteorology at Harvard, wrote “It can
have but a limited number of readers and will probably be quickly placed on a
library shelf and allowed to rest undisturbed by most of those who purchase a
copy.” Indeed, this is essentially what happened to the book.

A most perceptive review by F. J. W. Whipple of the Met Office came closest to
understanding Richardson’s unrealistic forecast, postulating that rapidly-
travelling waves contributed to its failure: “The trouble that he meets is
that quite small discrepancies in the estimate of the strengths of the winds
may lead to comparatively large errors in the computed changes of pressure.”
Whipple appears to have had a far clearer understanding than did Richardson
himself of the causes of the forecast catastrophe

Richardson’s work was not taken seriously by most meteorologists of the day
and his book failed to have any significant impact on the practice of weather
forecasting during the decades following its publication. Nevertheless, his
work is the foundation upon which modern forecasting is built. Several key
developments in the ensuing decades set the scene for later progress. Timely
observations of the atmosphere in three dimensions were becoming available
following the invention of the radiosonde. Developments in the theory of
meteorology provided crucial understanding of atmospheric dynamics and the
filtered equations necessary to calculate the synoptic-scale tendencies.
Advances in numerical analysis led to the design of stable algorithms.
Finally, the development of digital computers provided a way of attacking the
enormous computational task involved in weather forecasting, all leading to
the first weather prediction by computer (Charney _et al._ , 1950). The
history leading to the emergence of modern operational numerical weather
prediction is described in Lynch (2006).

## Solution of the “Noise Problem”

In the atmosphere there are long-period variations dominated by the effects of
the Earth’s rotation — these are the meteorologically significant rotational
modes — and short-period oscillations called gravity waves, having speeds
comparable to that of sound. For many purposes, the gravity waves, which are
normally of small amplitude, may be disregarded as noise. However, they are
solutions of the governing equations and, if present with spuriously large
amplitudes in the initial data, can completely spoil the forecast.

The most obvious approach to circumventing the noise problem is to construct a
forecast by combining many time steps which are short enough to enable
accurate simulation of the detailed high frequency variations. If Richardson
had extended his calculations, taking a large number of very small time steps,
his results would have been noisy, but the mean values would have been
meteorologically reasonable. Of course, the attendant computational burden
made this utterly impossible for Richardson.

A more practical approach to solving the problem is to modify the governing
equations so that the gravity waves no longer occur as solutions. This process
is known as filtering the equations. The first successful computer forecasts
(Charney, _et al._ , 1950) were made with the barotropic vorticity equation,
which has low frequency but no high frequency solutions. Another approach is
to adjust the initial data so as to reduce or eliminate the gravity wave
components. The adjustments can be small in amplitude but large in effect.
This process is called initialization, and it may be regarded as a special
form of smoothing.

Lynch (2006) showed that the digital filtering initialization method yields
realistic tendencies when applied to Richardson’s data. He found that when
appropriate smoothing was applied to the initial data, using a simple digital
filter, the initial tendency of surface pressure was reduced from the
unrealistic 145 hPa in 6 hours to a reasonable value of less than 1 hPa in 6
hours. The forecast was in good agreement with the observed pressure change.
The rates of change of temperature and wind were also realistic. This
confirmed the root cause of the failure of Richardson’s trial forecast:
unavoidable errors in the analysed winds resulted in spuriously large values
of divergence, which caused the anomalous pressure tendencies. In effect, the
analysis contained gravity wave components of unrealistically large
amplitudes.

The absence of gravity waves from the initial data results in reasonable
initial rates of change, but it does not automatically allow the use of large
time steps. The existence of high frequency solutions of the governing
equations imposes a severe restriction on the size of the time step allowable
if reasonable results are to be obtained. The restriction, known as the CFL
criterion, can be circumvented by treating those terms of the equations that
govern gravity waves in a numerically implicit manner; this distorts the
structure of the gravity waves but not of the low frequency modes. In effect,
implicit schemes slow down the faster waves, thus removing the cause of
numerical instability. Most modern forecasting models avoid the pitfall that
trapped Richardson by means of initialization followed by semi-implicit
integration.

## Richardson’s Later Work

After the First World War, Richardson’s research focussed primarily on
atmospheric turbulence. He had encapsulated the essence of the cascade of
turbulent energy in a simple and oft-quoted rhyme embedded in the text of
WPNP: _Big whirls have little whirls that feed on their velocity, and little
whirls have lesser whirls and so on to viscosity_. Richardson’s dense writing
style is occasionally lightened in this way by a whimsical touch as, when
discussing the tendency of turbulence to increase diversity, he writes “This
one can believe without the aid of mathematics, after watching the process of
stirring together water and lime-juice” (WPNP, page 101). Several of his
publications during this period are still cited by scientists. In one of the
most important — _The supply of energy from and to atmospheric eddies_ — he
derived a criterion for the onset of turbulence, introducing what is now known
as the Richardson Number.

Around 1926, Richardson made a deliberate break with meteorological research.
He was distressed that his turbulence research was being exploited for
military purposes. Indeed, this knowledge impelled him to destroy a large
volume of his research papers. In a much later study, Richardson investigated
the separation of initially proximate tracers in a turbulent flow and arrived
empirically at his “four-thirds law”: the rate of diffusion is proportional to
the separation raised to the power 4/3. This was later established more
rigorously by Andrey Kolmogorov using dimensional analysis.

## Advances in Computing: From ENIAC to PHONIAC

The first weather forecast (technically, a hindcast) made with a digital
computer was performed on the ENIAC (Electronic Numerical Integrator and
Computer) by a team of scientists at Princeton. The Princeton team were aware
that Richardson’s initial tendency field was completely wrong because he was
not able to evaluate the divergence. They realised that a filtered system of
equations would have dramatic implications for numerical integration. It would
obviate the problem of gravity-wave noise and would permit a much larger time
step to be used. They integrated the barotropic vorticity equation from real
initial conditions and produced four realistic, if far from perfect,
forecasts. For a full account, see Chapter 10 of (Lynch, 2006).

It is gratifying that Richardson was made aware of the success in Princeton;
Jule Charney sent him a copy of the Tellus paper (Charney, Fjørtoft and von
Neumann, 1950). In his response, Richardson congratulated Charney “on the
remarkable progress which has been made in Princeton; and on the prospects for
further improvement which you indicate”. He concluded by saying that the ENIAC
results were “an enormous scientific advance” on the single, and quite wrong,
forecast in which his own work had ended.

To illustrate the dramatic growth in computer power since the days of the
ENIAC, one of the forecasts was re-run on a small mobile phone, a Nokia 6300,
which had raw computational power comparable to a CRAY-1, the first super-
computer acquired by the European Centre for Medium-Range Weather Forecasts
(ECMWF). The computation time for a 24-hour forecast on ENIAC was about 24
hours. The time on the Nokia, christened Portable Hand Operated Numerical
Integrator and Computer (PHONIAC), was less than one second (Lynch & Lynch,
2008).

Figure 1: Illustration of Richardson’s forecast factory ©Stephen Conlin
(www.pictu.co.uk).

## Richardson’s Fantastic Forecast Factory

The computation of his forecast was prodigious, taking Richardson some two
years to complete. How could the enormous number of calculations necessary for
a practical forecast ever be done? Richardson estimated that it would require
64,000 people just to keep up with the weather. In WPNP, he described his
fantasy: a _Forecast Factory_ like a large theatre-in-the-round — think of the
Royal Albert Hall — a circular building with a great central chamber, the
walls painted to form a map of the globe. A large team of (human) computers
are busy within the building calculating the future weather. The work is
coordinated by a Director of Operations. Standing on a central dais, he
‘conducts the computations’ by signalling with a spotlight to those who are
racing ahead or behindhand.

In 1986, an Irish artist, Stephen Conlin, created an illustration of the
forecast factory (Fig. 1). This painting, in ink and water colours, is a
remarkable work, replete with narrative details. The painting depicts a large
spherical building with a vast central chamber. Four banners identify major
pioneers of computing: John Napier, Charles Babbage, George Boole and the
first computer programmer, Ada Lovelace. The painting is described in detail
in an article in Weather (Lynch, 2016).

There are surprising similarities between Richardson’s forecast factory and a
modern massively parallel processor (MPP). Richardson envisaged a large number
of (human) computers working in synchrony on different sub-tasks. In the
fantasy, the forecasting job is sub-divided using domain decomposition, a
technique often used in parallel computers today. Richardson’s scheme involved
nearest-neighbour communication, analogous to message-passing techniques used
in MPPs. The man in the pulpit functioned like a synchronization and control
unit. Thus, the logical structures of the forecast factory and an MPP have
much in common.

## Summary

Richardson’s dream was that scientific weather forecasting would one day
become a practical reality. Modern weather forecasts are made by calculating
solutions of the mathematical equations governing the atmosphere. The
solutions are generated by complex simulation models implemented on powerful
computer equipment. His dream has indeed come true.

The development of comprehensive models of the atmosphere is undoubtedly one
of the finest achievements of meteorology in the twentieth century. Numerical
models continue to evolve, with substantial developments in data assimilation
to produce improved initial conditions, new numerical algorithms for more
precise and faster computations, and a probabilistic approach with ensemble
forecasts that quantify uncertainties in an operational environment. These
developments have made the dreams of Abbe, Bjerknes and Richardson an everyday
reality. Meteorology is now firmly established as a quantitative science, and
its value and validity are demonstrated daily by the acid test of any science,
its ability to predict the future.

## References

[This small selection of reference may be supplemented by the extensive
bibliography in Lynch (2006)]

  * •

Abbe, C., 1901: The physical basis of long-range weather forecasts. _Mon. Wea.
Rev._ , 29, 551–561.

  * •

Ashford, O. M., 1985: _Prophet — or Professor? The Life and Work of Lewis Fry
Richardson_. Adam Hilger, xiv + 304 pp.

  * •

Bjerknes, V., 1904: Das Problem der Wettervorhersage, betrachtet vom
Standpunkte der Mechanik und der Physik. [The problem of weather prediction,
considered from the viewpoints of mechanics and physics.] _Meteor. Z._ , 21,
1–7 [translated and edited by E. Volken and S. Brönnimann, 2009: _Meteor. Z._
, 18, 663–667].

  * •

Charney, J. G., R. Fjørtoft, and J. von Neumann, 1950: Numerical integration
of the barotropic vorticity equation. _Tellus_ , 2, 237–54.

  * •

Lea, Chris, 2012: Lewis Fry Richardson: the Father of Weather Forecasting.
_Benchmark_ , 12, 36–38.

  * •

Lynch, Peter, 2006: _The Emergence of Numerical Weather Prediction:
Richardson’s Dream._ Cambridge University Press, 279 pp.

  * •

Lynch, Peter, 2016: An artist’s impression of Richardson’s fantastic forecast
factory. _Weather_ , 71, 14–18.

  * •

Lynch, Peter and Owen Lynch, 2008: Forecasts by PHONIAC. _Weather_ , 63,
324–326.

  * •

Richardson, L. F., 1922: _Weather Prediction by Numerical Process._ Cambridge
University Press, xii + 236 pp. Reprinted by Dover Publications, 1965, with a
new Introduction by Sydney Chapman, xvi + 236 pp. Second ed., Cambridge
University Press, 2007, with Foreword by Peter Lynch.


# Visual Chain of Thought:
Bridging Logical Gaps with Multimodal Infillings

Daniel Rose*, Vaishnavi Himakunthala*, Andy Ouyang*, Ryan He*, Alex Mei,
Yujie Lu, Michael Saxon, Chinmay Sonar, Diba Mirza, William Yang Wang
University of California, Santa Barbara, Santa Barbara, CA
{danielrose, vaishnavi, andyouyang<EMAIL_ADDRESS>
{alexmei, yujielu, saxon<EMAIL_ADDRESS>
{dimirza<EMAIL_ADDRESS>

###### Abstract

††*Equal Contribution.

Recent advances in large language models elicit reasoning in a chain of
thought that allows models to decompose problems in a human-like fashion.
Though this paradigm improves multi-step reasoning ability in language models,
it is limited by being unimodal and applied mainly to question-answering
tasks. We claim that incorporating visual augmentation into reasoning is
essential, especially for complex, imaginative tasks. Consequently, we
introduce VCoT111https://github.com/dannyrose30/VCOT, a novel method that
leverages chain of thought prompting with vision-language grounding to
recursively bridge the logical gaps within sequential data. Our method uses
visual guidance to generate synthetic multimodal infillings that add
consistent and novel information to reduce the logical gaps for downstream
tasks that can benefit from temporal reasoning, as well as provide
interpretability into models’ multi-step reasoning. We apply VCoT to the
Visual Storytelling and WikiHow summarization datasets and demonstrate through
human evaluation that VCoT offers novel and consistent synthetic data
augmentation beating chain of thought baselines, which can be used to enhance
downstream performance.

##  1 Introduction

Figure 1: Sequences naturally have logical gaps between elements that can
limit reasoning; VCoT bridges these gaps with multimodal infillings.

Humans’ multi-step reasoning allows us to solve complex problems and imagine
solutions to create a better world. Similarly, enhancing computer reasoning
about sequential data can lead to positive downstream contributions such as
virtual assistants Qiu et al. (2021), navigators Anderson et al. (2018), and
decision-makers Huang et al. (2022a). Such tasks rely on multi-step reasoning
with sequential data, which often contains logical gaps between elements that
can hinder task-specific performance (Figure 1). This leads us to our
overarching research question: _how can we bridge the logical gaps in
reasoning within sequential data to improve downstream performance?_

Recent advances in large language models (LLMs) explore chain of thought
prompting (CoT) Wei et al. (2022); Kojima et al. (2022a), a technique
mimicking human reasoning by decomposing complex problems into simple steps
Lampinen et al. (2022). While CoT demonstrates impressive performance on tasks
within the text-only question-answering domain, bridging the logical gaps in
sequential data is a limitation, which we hypothesize is due to its unimodal
constraint. We argue incorporating the visual modality into CoT can help
bridge logical gaps in two ways. First, multi-step reasoning with visuals
better fills logical gaps because images capture additional information that
unimodal text cannot. Second, visual chains mimic human imagination which
creates novel problem solutions Tan and Bansal (2021); Lu et al. (2022); Zhu
et al. (2022) and provide interpretability Wang et al. (2022c) of decision-
making. A single imagined picture provides a thousand-word insight to enhance
computer reasoning.

Cognitive science research shows humans rely on visual knowledge for natural
language tasks Just et al. (2004). Past NLP research utilizes this insight to
mimic human imagination and visual processes by developing models to generate
novel images from text Lu et al. (2022). Visual knowledge augmentation
traditionally improves text-based tasks such as open-ended text generation Zhu
et al. (2022). These advancements enable LLMs to work with multimodal data,
augment sparse data with imagined visuals, and offer human interpretability
into LLM decision-making (Huang et al., 2022b). However, vision-language
models are often resource inefficient and lack the multi-step reasoning
capabilities of CoT.

We propose Visual Chain of Thought (VCoT), which combines the efficiency,
robustness, and multi-step reasoning of CoT with the multimodal capabilities
of vision-language models. VCoT synthetically augments sequential datasets and
bridges logical gaps by recursively generating multimodal infillings and using
the synthetic data to improve downstream task performance. These synthetic
generations also serve as human-interpretable insights into AI systems’
ability of multi-step reasoning. We demonstrate that VCoT creates consistent
and novel synthetic data that enhances downstream performance on the Vist and
WikiHow datasets. Our main contributions are:

  * •

We propose Visual Chain of Thought for sequential data to generate synthetic
text-visual pairs as data augmentation for downstream reasoning.

  * •

We explore a consistency and novelty-driven approach to recursively generate
multimodal infillings that augment faithful, relevant context.

  * •

We demonstrate the effectiveness of our method through human evaluation
showing improvements in sequential reasoning.

##  2 Related Work

Figure 2: In VCoT, we 1) normalize sequential datasets into text-visual pairs,
2) caption the visuals and "foveate" to create a focus, 3) recursively
generate multimodal infillings. The blue rectangle encapsulates one recursive
call, which involves a repeated cycle of steps 1-5 until all logical gaps are
filled. Steps 1 and 3 refer to the generation of the candidate texts (from CoT
infilling) and visuals infillings (from the selected text). Steps 2 and 4 take
as input the text or visual candidates and select the one with maximal Clip
consistency with respect to their neighbors.

Paradigm Shifts. Language model pretraining and fine-tuning ushered in an era
of higher accuracy, more data and compute-efficient NLP Devlin et al. (2018);
Howard and Ruder (2018); Gao et al. (2020); Dodge et al. (2020). As models
continued to scale, few and zero-shot techniques for many general language
tasks have become feasible and performant Kojima et al. (2022b); Brown et al.
(2020a); Huang et al. (2022a). Recent research focuses on optimizing prompts
in order to improve the performance of zero-shot techniques, such as in-
context inference Lampinen et al. (2022) and prompt ensembling Wang et al.
(2022b). In light of the impressive performance of prompt-optimization efforts
for zero-shot NLP techniques, it becomes natural to ask: _can multimodal
vision & language tasks benefit from LLM generalization techniques?_ Our work
seeks to bridge the best of both modalities through using efficient strategies
to reason with LLMs while leveraging visual imagery for sequential tasks.

Data Augmentation. Another trend within language modeling is to augment LLMs
with external data. These solutions include leveraging dense passage retrieval
Lewis et al. (2020), prompt editing Madaan et al. (2022), human guidance Guan
et al. (2020), and conversation history in the conversational AI setting
Ghazvininejad et al. (2018). External data provides complementary knowledge to
the models’ training data, improves robustness without rigorous fine-tuning,
and enables systems to be time-agnostic. Previous work studies visual
knowledge incorporation for language understanding Tan and Bansal (2021); Lu
et al. (2022) and generation Zhu et al. (2022), but it has yet to be applied
to sequential reasoning. We are the first propose a vision-augmented CoT,
which infills temporal data to provide data augmentation to reduce the logical
leaps in sparse data that may limit general sequential generation tasks.

Intermediate Infillings. In earlier research relating to synthetic infillings,
researchers point out significant lagging performance in temporal reasoning
Vashishtha et al. (2020) and offer advancements Zhou and Hripcsak (2007); Dorn
(1992). Works have separately looked to augment sparse data with intermediate
logical links Kotnis et al. (2015) and create intermediate representations
between sequential elements Hong et al. (2023). While promising, these
advancements are task-specific, unimodal, and computationally expensive. Our
novel application of VCoT draws inspiration from previous research and adds
synthetic multimodal logical infillings to strengthen temporal reasoning for
general sequential datasets.

##  3 Problem Formulation

To improve temporal reasoning in language models, we define the multimodal
infilling task to bridge the logical gaps in sequential data and explicitly
guide the vision-language model reasoning process. We generate an infilling
$\\{(v_{i},t_{i})\\}$ given two text-visual sequential pairs
$\\{(v_{i-1},t_{i-1}),(v_{i+1},t_{i+1})\\}$ through generator $g$ using model
$M$ (Equation 1). The best infilling $(v_{i}^{best},t_{i}^{best})$ is judged
using judgement function $j((v_{i},t_{i}))$ (Equation 2). For a downstream
task $T$ measured by performance $p$ (e.g., novelty, consistency, coherence,
descriptiveness), we define the optimal infilling $(v_{i}^{opt},t_{i}^{opt})$
to be $(v_{i}^{best},t_{i}^{best})$ only when the infilling improves the
downstream task performance (Equation 3); otherwise, the optimal infilling is
null, meaning an infilling is not beneficial. (Equation 4).

$(v_{i},t_{i}):=g((v_{i-1},t_{i-1}),(v_{i+1},t_{i+1}),M)$ (1)
$(v_{i}^{best},t_{i}^{best}):=\operatorname*{argmax}_{(v_{i},t_{i})}[j((v_{i},t_{i}))]$
(2)
$\begin{split}p(T,[(v_{i-1},t_{i-1}),(v_{i}^{best},t_{i}^{best}),(v_{i+1},t_{i+1})])\\\
>p(T,[(v_{i-1},t_{i-1}),(v_{i+1},t_{i+1})])\end{split}$ (3)
$(v_{i}^{opt},t_{i}^{opt}):=\begin{cases}(v_{i}^{best},t_{i}^{best})&\text{Eq.
\ref{eq:3} holds}\\\ \texttt{null}&\text{otherwise}\end{cases}$ (4)

##  4 VCoT Multimodal Infilling Generation

Following our problem definition, we propose the following
pipeline222https://sites.google.com/view/vcot/home (Figure 2):

  1. 1.

We transform text-only datasets into multimodal text-visual pairs for task
unification (section 4.1).

  2. 2.

We identify the multipoint foveation to extract the global focus to guide the
consistent generation of consistent multimodal infillings (section 4.2).

  3. 3.

We generate our synthetic data in a process that uses both novelty-driven
recursive infilling (section 4.3) and consistency-driven visual augmentation
(section 4.4) to provide interpretability for multi-step reasoning and reduce
logical gaps to aid downstream task performance.

###  4.1 Task Unification

To apply VCoT, we reformat sequential data into text-visual pairs. For text-
only data, we generate visuals $\\{v_{1}^{\prime},\ldots,v_{n}^{\prime}\\}$
from input text $\\{t_{1},\ldots,t_{n}\\}$ with Stable Diffusion, where
$v_{i}^{\prime}$ contains multiple candidate visuals for $t_{i}$. We use Clip
embeddings to assess similarity between the candidate visuals and the
surrounding input texts $(t_{i-1},t_{i+1})$ and select the visual with the
highest similarity score to yield consistent visuals that unify general
sequential tasks as text-visual pairs.

###  4.2 Multipoint Foveation

Figure 3: Extracting a multipoint foveation from a visual-text pair sequence
to guide infilling generation.

To preserve consistency of the input sequence when generating multimodal
infillings, we use foveation Mei et al. (2022) to identify the main focus.
Since pairwise sequential elements may omit relevant fixation points, we
define multipoint foveation (MPF) to identify all of the core fixation points
(setting, characters, etc.) of the entire text-visual input sequence (Figure
3). To construct the multipoint foveation, we project the text-visual pairs
into a unimodal text space by captioning the visuals using an image-captioning
model:
$\\{(v_{1},t_{1})...(v_{n},t_{n})\\}\rightarrow\\{(c_{1},t_{1})...(c_{n},t_{n})\\}$.
The projected output, along with diverse few-shot exemplars, is fed into Gpt-3
to generate a maximum likelihood333Likelihood is defined in Appendix A.2.1.
summary, from which the multipoint foveation $f$ is extracted (Equation 5).
The foveation guides infillings to be consistent and not introduce excessive
information.

$MPF((c_{i},t_{i}),M):=\operatorname*{argmax}_{f}(\text{I\kern-1.00006ptP}(f|(c_{i},t_{i}),M))$
(5)

###  4.3 Novelty-Driven Recursive Infilling

The judgment function, $j((v_{i},t_{i}))$ (Equation 3), judges the generated
multimodal infillings based on two metrics: consistency and novelty.
Consistency ensures that the infillings maintain faithful details from the
surrounding steps, while novelty ensures that the infillings add relevant and
new information that accurately bridges the logical gaps.

We infill previous and next visual-text pairs $(v_{i-1},t_{i-1})$ and
$(v_{i+1},t_{i+1})$ using foveation $f$ with new information that bridges the
logical gap (Algorithm 1). As there may be multiple logical gaps to infill, we
opt for a recursive approach that dynamically determines whether infillings
are beneficial (Equation 3), in contrast to CoT, which needs to approximate
how many steps are needed. Our approach generates multiple depths of
infillings to add valuable new, relevant, and consistent multimodal
information for downstream tasks (Figure 5).

Data: (visual, text) pairs prev $(v_{1},t_{1})$ and next $(v_{2},t_{2})$,
depth $d$, and foveation $f$

Result: infilled list of (visual, text) pairs between $(v_{1},t_{1})$ and
$(v_{2},t_{2})$

1 $infilling\leftarrow$ genInfilling$(prev,next,f)$

2 if  _$d+1\leq$ depth-limit_ then

3 $prevGen\leftarrow recGen(prev,infilling,d+1,f)$

4 $nextGen\leftarrow recGen(infilling,next,d+1,f)$

5 return $prevGen+[infilling]+nextGen$

6return $[infilling]$

Algorithm 1 recGen(prev, next, depth, f) Figure 4: VCoT’s recursive multimodal
infilling generation algorithm given two sequential text-visual pairs.

###  4.4 Consistency-Driven Visual Augmentation

To explicitly guide consistent infilling generation, we use 1) multipoint
foveation (section 4.2) to ground our generations to the input sequence and 2)
Clip to select recursively generated infilling candidates to be most
consistent with their surrounding pairs (Algorithm 2). Specifically, we
generate five candidate text-infillings with Gpt-3 and compare them to their
surrounding visuals using Clip and select the most similar candidate. With the
selected text-infilling, we generate four candidate visuals using Stable
Diffusion and choose the candidate visual most consistent with the text-
infilling again using Clip embeddings.

To determine the recursive stopping condition, we experiment with both a fixed
recursive depth and an learned approach by prompting Gpt-3 to classify whether
a logical gap remains. Empirical results demonstrate the Gpt-3-halting
approach shows inconsistent performance that adds significant noise444We leave
further exploration on automatically classifying whether an infilling is
beneficial as future work.. Instead, we opt for a fixed depth $d=2$, which
balances sufficiently filling logical gaps with not injecting irrelevant
information.

Figure 5: VCoT recursively infills logical gaps. Depth 1 represents the first
infilling between input elements, and subsequent depths (e.g. Depth 2)
represent infillings between inputs and/or generated infillings.

##  5 Resources

We leverage the state-of-the-art vision and language generation models Stable
Diffusion and Gpt-3 for synthetic data augmentation, along with Clip guidance
and Ofa captioning. We test our method on the Vist and WikiHow datasets due to
their sequential composition to show the effectiveness of VCoT in bridging
reasoning gaps.

Gpt-3 Brown et al. (2020b) is an autoregressive language model that has strong
generalized performance. We use Gpt-3555We use the text-davinci-003 variant of
Gpt-3. for all relevant language tasks due to ease of accessibility, low
inference time, and well-rounded performance.

Ofa Wang et al. (2022a) is a task- and modality-agnostic framework that
unifies cross-modal and unimodal tasks. We utilize an out-of-the-box image
captioning checkpoint666We use the OFA-base checkpoint for image captioning.
to show the generality of VCoT and demonstrate performance without the need
for task-specific finetuning.
Stable Diffusion Rombach et al. (2022) is a small model that applies diffusion
model training in the latent space of pretrained autoencoders for visual-
related tasks, achieving strong performance, which we use for image
generation777We use Stable-Diffusion 1.4 as our T21 model..
Clip Radford et al. (2021) unifies modalities by training visual and text
embeddings to be in the same latent space. Because Clip is the leading
multimodal similarity metric, we utilize comparisons of Clip embeddings to
multimodally guide the visual and text generations.
Vist Huang et al. (2016) is a visual storytelling dataset consisting of
sequences of five text-visual pairs representing a single story, including
2021 stories in the test set. The clear gaps between 1) flickr visuals and
human-written captions, and 2) pairwise sequential elements often create
sizable logical gaps, which VCoT’s multimodal infillings can bridge and
thereby enhance AI storytelling.
WikiHow Koupaee and Wang (2018) is a text summarization dataset containing
“How-To” articles, with 6000 articles in the test set. VCoT’s synthetic
multimodal infillings between logically distanced “How-To” instructions can
enhance the instruction summarization task and create human-understandable
steps.

Data: prev $(v_{i-1},t_{i-1})$ and next $(v_{i+1},t_{i+1})$ infillings and
foveation $f$

Result: single intermediate infilling $(v_{i},t_{i})$

Step 1: generate intermediate text

1 Generate captions $c_{i-1},c_{i+1}$ for visuals $v_{i-1},v_{i+1}$,
respectively

2 Generate five candidate texts $t_{1}^{\prime},\ldots,t_{5}^{\prime}$ using
Gpt-3 with input $((c_{i-1},t_{i-1}),(c_{i+1},t_{i+1}),f)$

3 Select highest similarity candidate text $t_{best}$ by comparing Clip
embeddings of $t_{1}^{\prime},\ldots,t_{5}^{\prime}$ to the Clip embeddings of
$t_{i-1}$ and $t_{i+1}$

Step 2: generate intermediate visual

4 Generate four candidate visuals $v_{1}^{\prime},\ldots,v_{4}^{\prime}$ using
Stable Diffusion with input $t_{best}$

5 Select highest similarity candidate visual $(v_{best})$ by comparing Clip
embeddings of $v_{1}^{\prime},\ldots,v_{4}^{\prime}$ to the Clip embedding of
$t_{best}$

6 return $(v_{best},t_{best})$

Algorithm 2 genInfilling(prev, next, foveation) Figure 6: VCoT generates
candidate infillings and uses Clip embeddings to select the most consistent
one.

VCoT vs. Baseline | Image Consistency | Text Consistency | Image Novelty | Text Novelty
---|---|---|---|---
Win($\uparrow$) | Tie | Lose($\downarrow$) | Win($\uparrow$) | Tie | Lose($\downarrow$) | Win($\uparrow$) | Tie | Lose($\downarrow$) | Win($\uparrow$) | Tie | Lose($\downarrow$)
CoT+CoI | 26.82 | 53.02 | 20.16 | 28.07 | 52.21 | 19.73 | 30.13 | 50.24 | 19.63 | 25.77 | 52.86 | 21.37
Random | 30.13 | 50.24 | 19.63 | 43.40 | 39.27 | 17.33 | 43.66 | 38.95 | 17.39 | 41.87 | 40.36 | 17.77

Table 1: Consistency and novelty ratings of VCoT intermediate text-visual
infillings compared to the Chain of Thought (CoT) + Chain of Images (CoI) and
random baselines, represented as win-tie-loss percentages. VCoT has higher win
percentages than all baselines.  Figure 7: Consistency and novelty rating
distributions of VCoT text-visual infillings compared to the Chain of Thought
(CoT), Chain of Images CoI, and random baselines. Here multimodal infillings
are selected as “good”, “neutral”, or “poor”, and VCoT again surpasses all
other baselines.

##  6 Experiments and Results

We use Vist and WikiHow to evaluate VCoT’s multimodal infillings and their
enhancement of downstream tasks. In conjunction with Vist storytelling, we
apply WikiHow for summarization and generation of descriptive, human-
understandable instructions from “How-To” articles, separating us slightly
from strict summarization. We showcase WikiHow as a text-only dataset for our
process of task unification (Figure 8).

We opt for human evaluation over automatic evaluation to account for
multimodality and ensure consistent analysis of filling logical gaps with
novel and consistent synthetic tokens. Further, we do not seek to compare with
the datasets’ ground truths because the ground truths in Vist and WikiHow
often are incoherent or undescriptive. Instead, our data augmentation seeks to
surpass ground truth results by synthetically filling logical gaps and
providing human-interpretable multi-step reasoning. We select our human
evaluation criteria based on BartScore Yuan et al. (2021) and our goal of
generating relevant logical infillings.

###  6.1 Experimental Settings

VCoT generates five textual infilling candidates using Gpt-3: one with zero
temperature and four with 0.5 temperature. We otherwise use zero temperature
to maximize consistency. To evaluate the infillings themselves, we combine
WikiHow and Vist and present 7062 generated infillings whose scores are
provided by M-Turk workers who pass an attention check. For the downstream
tasks, we evaluate each dataset separately, considering 227 full Wikihow
articles and 266 Vist stories. To further evaluate these downstream tasks, we
add coherence for storytelling and descriptiveness for instruction
summarization. Coherence confirms that the output flows logically together as
an interesting and creative story. Descriptiveness verifies that the generated
summaries describe accurately and with detail the steps of the “How-To”
article.

We evaluate results using win-tie-loss, a common approach in human evaluation
Gu et al. (2021); Yang et al. (2019), which reduces variance and increases
inter-annotator agreement. Since our method is the first to generate
multimodal infillings, we cannot use existing vision-language models as
baselines. Instead, we compare head-to-head VCoT’s text-visual pairs with
purely textual chain-of-thought (CoT) paired with the purely visual chain-of-
images (CoI) performed in parallel, using the same recursion depth. To
evaluate the infillings, we additionally compare with a random baseline, which
selects a random multimodal pair from our generated examples. To evaluate the
downstream task performance, we additionally compare the performance without
any generated infillings, as well as the ground truth reference steps provided
by each dataset.

Figure 8: Compared to the Chain of Thought (CoT) and Chain of Images (CoI)
baselines, VCoT’s multimodal infillings add more consistent and relevant novel
information to the input steps on “How to Become a VFX Artist.”

###  6.2 Intermediate Step Generation

Dataset | VCoT vs. Baselines | Novelty | Consistency | Descriptivity
---|---|---|---|---
Win($\uparrow$) | Tie | Lose($\downarrow$) | Win($\uparrow$) | Tie | Lose($\downarrow$) | Win($\uparrow$) | Tie | Lose($\downarrow$)
WikiHow | Chain of Thought | 34.23 | 36.90 | 28.87 | 30.06 | 39.66 | 30.28 | 23.31 | 56.39 | 20.30
Chain of Images | 37.28 | 25.82 | 36.90 | 44.20 | 26.04 | 29.76 | 33.83 | 27.44 | 38.72
No Infilling | 33.56 | 38.47 | 27.98 | 38.24 | 26.12 | 35.64 | 33.46 | 32.71 | 33.83
Reference Step | 40.92 | 28.42 | 30.65 | 47.99 | 21.80 | 30.21 | 42.11 | 22.56 | 35.34
Dataset | VCoT vs. Baselines | Novelty | Consistency | Coherence
Win($\uparrow$) | Tie | Lose($\downarrow$) | Win($\uparrow$) | Tie | Lose($\downarrow$) | Win($\uparrow$) | Tie | Lose($\downarrow$)
Vist | Chain of Thought | 39.86 | 29.42 | 30.72 | 30.17 | 38.01 | 31.82 | 35.11 | 35.11 | 29.79
Chain of Images | 33.81 | 23.71 | 42.47 | 35.33 | 30.17 | 34.50 | 35.11 | 36.88 | 28.01
No Infilling | 34.64 | 23.92 | 41.44 | 37.32 | 27.56 | 35.12 | 31.21 | 43.62 | 25.18
Reference Step | 28.73 | 40.62 | 30.65 | 37.04 | 30.45 | 32.51 | 36.52 | 38.30 | 25.18

Table 2: The wins-tie-loss percentages for VCoT against four baselines for
downstream summarization and storytelling tasks for the novelty and
consistency metrics. We also measure descriptivity and coherence for WikiHow
and Vist, respectively.

We ask human evaluators to judge our synthetic multimodal infillings based on
novelty and consistency (section 4.3) on a 5-point scale (1-2: poor, 3:
neutral, 4-5: good). VCoT infillings outperform all baselines on the 5-point
scoring of quality (Figure 7). When comparing win-tie-losses, VCoT also
outperforms both baselines by at least $6.6\%$ and $4.4\%$ for the consistency
and novelty of our synthetic visuals and text (Table 1), respectively.
Qualitative examples show that VCoT outperforms baselines by generating more
useful and relevant infillings in cases where text-visual pairs have a sizable
logical gap (Figure 8, Figure 16). VCoT’s strong performance in consistency
and novelty indicate that the synthetic multimodal infillings add both
relevant and new information to their surrounding sequential steps, which
demonstrate that VCoT helps bridge logical gaps in sequential data.

In Vist and WikiHow examples with a significant chronological gap in between
steps, VCoT infills these gaps in the sequence with consistent bridging
information (Figure 14, Figure 15). We hypothesize the utility of VCoT
increases with size of the logical gap because these gaps hinder downstream
task performance (section 6.3) and interpretability for large language models.
VCoT synthetically augments additional context to language models with
multimodal infillings, allowing for smaller logical leaps and enhanced
reasoning. We argue that VCoT outperforms baselines by maintaining consistency
through foveation grounding (section 4.2) and Clip similarity alignment
(section 4.4).

With regard to novelty, VCoT adds new, relevant information for a logical flow
between surrounding steps (Figure 14, Figure 15). We observe that many
baseline-generated images contain new but less relevant information (Table 1).
Notably, CoI often generates novel images (Figure 16) that do not align with
the surrounding steps, hindering the relevance aspect. In contrast, VCoT
excels in increasing yet balancing both newness and relevance.

Figure 9: Example of Vist storytelling task. The parts of the story generated
by VCoT better match the text and images of the VIST story than the parts
generated by CoT. Figure 10: Example of the WikiHow summarization task. The
yellow highlighted text indicates how VCoT can improve on the summary quality
compared to text-only CoT.

###  6.3 Downstream Task Evaluation

We also conduct human evaluation and create qualitative results to show VCoT’s
potential to enhance downstream performance by introducing novel and
consistent information (Figure 9, Figure 10).

For WikiHow, we see a clear improvement in novelty (Table 2), which makes
sense because we augment the data with novel, relevant tokens for instruction
generation. Meanwhile for consistency, VCoT ties with CoT and surpasses all
other baselines. We hypothesize this tie is due to the lengthy nature of the
input text, allowing Gpt-3 to create fairly consistent instructions with or
without the use of VCoT. For descriptiveness, VCoT beats all baselines except
for CoI, likely because CoI injects visually-descriptive information that
isn’t grounded by the original text. Specifically, CoI’s image-only modality
allows it to capture novel but less consistent information than that of the
limited textual description. Overall, VCoT offers a balanced combination of
performance improvements.

For Vist, VCoT’s infillings improve consistency against all baselines besides
CoT. Conversely, we find that VCoT loses in novelty to all baselines other
than CoT, which VCoT (Table 2) improves upon. These juxtaposed results offer
interpretability into computer reasoning, namely that CoI introduces more
novel information, CoT preserves consistency, and VCoT maintains a balance. We
suspect that VCoT’s loss in novelty is a result of 1) VCoT’s attentiveness to
consistency through Clip guidance and multipoint foveation, and 2) repeated
tokens generated with our consistency-driven approach that overlap and cause
repetition. These results demonstrate that VCoT provides insight on specific
ways to improve computer reasoning–design strategies that both enhance and
balance novelty and consistency.

##  7 Conclusion

We introduce a new research direction to generate synthetic logical infillings
for sequential data, which we tackle with a novel multimodal paradigm called
visual chain of thought. We combine chain of chought with visual guidance and
imagination to recursively generate multimodal infillings that bridge the
natural logical gaps between sequential elements. By adding infillings to
sequences while maintaining consistency, we augment novel, relevant
information to bolster downstream task performance while also providing human-
interpretable insights into the system’s intermediate reasoning process.
Through task unification, we can apply VCoT on various multimodal tasks.

Human experiments show that VCoT creates more novel and consistent logical
infillings than the unimodal CoT and CoI baselines performed in parallel on
the sequential datasets Vist and WikiHow, as well as that these infillings are
helpful to improve downstream task performance. While we demonstrate VCoT on
the instruction summarization and visual storytelling tasks, future work can
explore VCoT in new domains that could benefit from synthetic data
augmentation and bolstered reasoning abilities, such as procedural planning,
DNA sequencing, and video understanding. Furthermore, future research can look
into better leveraging multimodal infillings to align them with the desired
metrics for downstream task improvement. Along these lines, it is valuable to
measure such desired output through automatic evaluation metrics to better
support evaluation at scale. We hope that VCoT’s synthetic, multimodal data
augmentation of temporal sequences through our task-agnostic paradigm inspires
a new direction for multimodal computer reasoning applications.

## Limitations

Though we attain advanced representation, there remain nontrivial
opportunities to improve VCoT.

First, a major bottleneck of the system is the image captioning capabilities
of Ofa; it could be greatly beneficial to more effectively integrate the
visual modality with regard to the rest of the pipeline than captions, which
are very restrictive.

Second, our recursive algorithm currently fixes a recursion depth, which is
not practical in reality as different sequences contain a variable number of
logical leaps; robustly and dynamically classifying whether an infilling would
be beneficial has great potential to improve performance.

Third, while the method ideally selects the best infilling from all possible
candidates, we restrict the number of candidates due to resource constraints.
In practice, the performance improvement with increasing number of candidates
should be negligible when the number of candidates is sufficiently large, but
further experimentation can be done to select the optimal number.
Alternatively, a model can be trained to identify the optimal candidate.

Finally, we choose criteria for human evaluation based on our novelty- and
consistency-driven approach. However, further evaluation could be useful to
provide better insight into the multimodal reasoning abilities of such a
system.

As VCoT is a pipeline system that builds off existing systems, it is limited
by the capabilities of those systems and the constraints of the datasets. As a
result, depending on the quality of these systems and data, there is also a
small, but non-negligible chance that models could infill pairs with harmful
information. While this possibility is unlikely due to our infillings being
grounded by text-visual pairs that themselves are not unsafe or irresponsible,
we encourage researchers to continue advancing research in the responsible AI
space to ensure such systems can be safely deployed.

## Ethical Considerations

Human evaluation was conducted via crowdsourcing using the Amazon Mechanical
Turk platform. For all Mechanical Turk experiments, we require workers to be
located in Australia, Canada, New Zealand, the United Kingdom, or the United
States and have a HIT approval rating of at least 98%. For both the
intermediate and downstream tasks, it is estimated we pay workers at a rate of
$10.0/hr, a rate which is above the federal minimum wage at the time in which
this research was conducted. The data annotation project is considered an
exempt protocol by the human subject committee by IRB guidelines. We provide a
consent form at the beginning of each task. We include additional details
regarding screenshots and task descriptions for each Mechanical Turk study in
Appendix A.

## References

  * Anderson et al. (2018) Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. 2018.  Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3674–3683.
  * Brown et al. (2020a) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020a.  Language models are few-shot learners.
  * Brown et al. (2020b) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020b.  Language models are few-shot learners.  _CoRR_ , abs/2005.14165.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.
  * Dodge et al. (2020) Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020.  Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping.
  * Dorn (1992) Jürgen Dorn. 1992.  Temporal reasoning in sequence graphs.  In _AAAI_ , pages 735–740.
  * Gao et al. (2020) Tianyu Gao, Adam Fisch, and Danqi Chen. 2020.  Making pre-trained language models better few-shot learners.
  * Ghazvininejad et al. (2018) Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2018.  A knowledge-grounded neural conversation model.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 32(1).
  * Gu et al. (2021) Xiaodong Gu, Kang Min Yoo, and Jung-Woo Ha. 2021.  Dialogbert: Discourse-aware response generation via learning to recover and rank utterances.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 12911–12919.
  * Guan et al. (2020) Lin Guan, Mudit Verma, Sihang Guo, Ruohan Zhang, and Subbarao Kambhampati. 2020\.  Widening the pipeline in human-guided reinforcement learning with explanation and context-aware data augmentation.
  * Hong et al. (2023) Xin Hong, Yanyan Lan, Liang Pang, Jiafeng Guo, and Xueqi Cheng. 2023.  Visual transformation telling.
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018.  Universal language model fine-tuning for text classification.
  * Huang et al. (2016) Ting-Hao K. Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Aishwarya Agrawal, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. 2016.  Visual storytelling.  In _15th Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2016)_.
  * Huang et al. (2022a) Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022a.  Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.
  * Huang et al. (2022b) Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. 2022b.  Inner monologue: Embodied reasoning through planning with language models.  In _arXiv preprint arXiv:2207.05608_.
  * Just et al. (2004) Marcel Adam Just, Sharlene D Newman, Timothy A Keller, Alice McEleney, and Patricia A Carpenter. 2004.  Imagery in sentence comprehension: an fMRI study.  _NeuroImage_ , 21(1):112–124.
  * Kojima et al. (2022a) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022a.  Large language models are zero-shot reasoners.
  * Kojima et al. (2022b) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022b.  Large language models are zero-shot reasoners.
  * Kotnis et al. (2015) Bhushan Kotnis, Pradeep Bansal, and Partha Talukdar. 2015.  Knowledge base inference using bridging entities.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 2038–2043.
  * Koupaee and Wang (2018) Mahnaz Koupaee and William Yang Wang. 2018.  Wikihow: A large scale text summarization dataset.
  * Lampinen et al. (2022) Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L McClelland, Jane X Wang, and Felix Hill. 2022.  Can language models learn from explanations in context?  _arXiv preprint arXiv:2204.02329_.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 9459–9474. Curran Associates, Inc.
  * Lu et al. (2022) Yujie Lu, Wanrong Zhu, Xin Eric Wang, Miguel Eckstein, and William Yang Wang. 2022\.  Imagination-augmented natural language understanding.  In _Proceedings of 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4392–4402, Dublin, Ireland.
  * Madaan et al. (2022) Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. 2022.  Memprompt: Memory-assisted prompt editing with user feedback.
  * Mei et al. (2022) Alex Mei, Sharon Levy, and William Yang Wang. 2022.  Foveate, attribute, and rationalize: Towards safe and trustworthy ai.
  * Qiu et al. (2021) Liang Qiu, Yuan Liang, Yizhou Zhao, Pan Lu, Baolin Peng, Zhou Yu, Ying Nian Wu, and Song-Chun Zhu. 2021.  Socaog: Incremental graph parsing for social relation inference in dialogues.  In _The 59th Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Radford et al. (2021) Alec Radford, Jonk Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  In _Proceedings of 38th International Conference on Machine Learning_ , pages 8748–8763, San Francisco, California.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and BjÃ¶rn Ommer. 2022.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Tan and Bansal (2021) Haochen Tan and Mohit Bansal. 2021.  Improving language understanding via contextualized, visually-grounded supervision.  In _Proceedings of 38th International Conference on Machine Learning_ , pages 8748–8763, San Francisco, California.
  * Vashishtha et al. (2020) Siddharth Vashishtha, Adam Poliak, Yash Kumar Lal, Benjamin Van Durme, and Aaron Steven White. 2020.  Temporal reasoning in natural language inference.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4070–4078.
  * Wang et al. (2022a) Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022a.  Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.
  * Wang et al. (2022b) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022b.  Rationale-augmented ensembles in language models.  _arXiv preprint arXiv:2207.00747_.
  * Wang et al. (2022c) Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, et al. 2022c.  Language models with image descriptors are strong few-shot video-language learners.  _arXiv preprint arXiv:2205.10747_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_.
  * Yang et al. (2019) Liu Yang, Junjie Hu, Minghui Qiu, Chen Qu, Jianfeng Gao, W Bruce Croft, Xiaodong Liu, Yelong Shen, and Jingjing Liu. 2019.  A hybrid retrieval-generation neural conversation model.  In _Proceedings of the 28th ACM international conference on information and knowledge management_ , pages 1341–1350.
  * Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021.  Bartscore: Evaluating generated text as text generation.  _Advances in Neural Information Processing Systems_ , 34:27263–27277.
  * Zhou and Hripcsak (2007) Li Zhou and George Hripcsak. 2007.  Temporal reasoning with medical data—a review with emphasis on medical natural language processing.  _Journal of biomedical informatics_ , 40(2):183–202.
  * Zhu et al. (2022) Wanrong Zhu, An Yan, Yujie Lu, Wenda Xu, Xin Eric Wang, Miguel Eckstein, and William Yang Wang. 2022.  Visualize before you write: Imagination-guided open-ended text generation.  _arXiv preprint arXiv:2210.03765_.

##  Appendix A Appendix

###  A.1 Crowdsourcing Human Evaluation

We show the interface of our human evaluations in Figure 11, Figure 12 and
Figure 13. We manually ensure no personal information is collected and no
offensive content is presented during human evaluations.

Figure 11: Amazon Mechanical Turk Platform Interface Part 1.

Figure 12: Amazon Mechanical Turk Platform Interface Part 2.

Figure 13: Amazon Mechanical Turk Platform Interface Part 3. Figure 14:
Example VCoT multimodal infillings (middle text-visual pair) generated with
our visual chain of thought method. Figure 15: Example VCoT multimodal
infillings (middle text-visual pair) generated with our visual chain of
thought method. Figure 16: Comparison of generating multimodal infillings for
two surrounding steps using visual chain of thought vs text-only chain of
thought plus image-only chain of images performed in parallel.

###  A.2 Details of Our Approach

We apply our generated multimodal infillings to improve the traditionally
text-based tasks of visual storytelling and instruction summarization. To
integrate our multimodal infillings into the downstream tasks, we pass them
along with the unified input to create an extensive summary using few-shot
examples and image captioning. The information in the summary can guide models
to better reason temporally in a wide range downstream tasks.

Because storytelling and summarization are inherently different, we use
different prompting schemes. For Vist, we pass in the few-shot examples,
summary, focus, current steps, and past story steps to autoregressively build
the final sequential story. Since it is very important for a story to flow
over time, current steps are very dependent on the past steps in time. For
Wikihow, we also pass in the few-shot examples, summary, focus, current steps.
Unlike the past story steps for Vist, we input the surrounding multimodal
infillings because the summarized steps of WikiHow articles aren’t quite as
dependent on flowing over time. Summarizing “How-To” articles into a series of
human-understandable instructions does, however, require understanding the
nearby logical steps.

Experiments demonstrate that VCoT improves the overall quality of the final
stories and instruction summarization for the example datasets Wikihow and
Vist.

####  A.2.1 Likelihood of Gpt-3 Outputs

The Gpt-3 API response888 https://platform.openai.com/docs/api-
reference/completions/create#completions/create-logprobs. returns the log
probabilities of individual generated tokens. We compute the joint log
likelihood probability of an output sequence $t_{1},...,t_{n}$ as the sum of
the individual token log probabilities (Equation 6).

$\ln(\text{I\kern-1.00006ptP}(t_{1},...,t_{n}))\approx\sum_{i=1}^{n}\ln(\text{I\kern-1.00006ptP}(t_{i}))$
(6)

###  A.3 Reproducibility

The only non-open source model we use is Gpt-3 text-davinci-003. We used the
OpenAI API in January 2023 with temperature 0 for one candidate infilling and
temperature 0.5 to generate 4 different candidate infillings, and temperature
0 otherwise for all other tasks; all other hyperparameters are set to the
default. Repetitive runs on Vist/WikiHow examples yield similar results,
promoting reproducibility.