Conservative relativistic algebrodynamics
induced on an implicitly defined worldline

Abdel Challaa,1, Vladimir V. Kassandrova,2 and Nina V. Markovab,3

a

Institute of Gravitation and Cosmology, Peoples’ Friendship University of
Russia (RUDN University), 6 Miklukho-Maklaya St., Moscow, 117198, Russian
Federation

b

S.M. Nikolsky Mathematical Institute, Peoples’ Friendship University of
Russia, (RUDN University), 6 Miklukho-Maklaya St., Moscow, 117198, Russian
Federation

In the framework of the Stueckelberg-Wheeler-Feynman concept of a “one-
electron Universe” we consider a worldline implicitly defined by a system of
algebraic (precisely, polynomial) equations. Collection of pointlike
“particles” of two kinds on the worldline (or its complex extension) is
defined by the real (complex conjugate) roots of the polynomial system and
detected then by an external inertial observer through the light cone
connections. Then the observed collective dynamics of the particles’ ensemble
is, generally, subject to a number of Lorentz invariant conservation laws.
Remarkably, this poperty follows from the Vieta’s formulas for the roots of
generating polynomial system. At some discrete moments of the observer’s
proper time the mergings and subsequent transmutations of a pair of particles-
roots take place simulating thus the processes of annihilation/creation of a
particle/antiparticle pair

Keywords: Collective dynamics, Vieta’s formulas, conservation laws, polynomial
worldline

PACS: 03.65.Fd,11.30.-j,98.80.-k

11footnotetext: e-mail<EMAIL_ADDRESS>e-mail:
<EMAIL_ADDRESS>e-mail<EMAIL_ADDRESS>
##  1 “Unique worldline” and
collective algebraic dynamics

In the Special Theory of Relativity, any worldline canonically parameterized
as $x_{\mu}=f_{\mu}(\tau)$ by a monotonic parameter $\tau$ describes the
motion of a sole pointlike particle. The problem of interaction is actually
the problem of mutual arrangement of different worldlines (H. Minkowski [1])
which, kinematically, can be quite arbitrary.

In 1933 E.C.G. Stuekelberg [2] examined worldlines of general type strictly
prohibited in the canonical STR: such worldlines allow for segments with
superliminar velocities and/or motions backwards in time. However, if one
introduces another monotonic timelike parameter $t$ independent of the
structure of the worldline itself, then at each $t$ one will detect a number
of particles all moving forward in time.

As for admissible superliminar values of velocities, they occure, as a rule,
only in vicinity of critical points where a pair of particles merge and
disappear (or arise). Such events can be naturally identified with physical
processes of annihilation/creation of a particle/antiparticle pair. Note that
just in this picture the famous Feynman’s idea on a “positron as an electron
moving backwards in time” had found its origin [3].

Unfortunately, Stueckelberg’s approach has a number of evident drawbacks. It
leads to consideration of a 5-dimensional space with two timelike parameters
and, in fact, to nonrelativistic kinematics with respect to the introduced
“true time” $t$. It does not give the key to explanation of
particle/antiparticle asymmetry. Moreover, one can see that at the points,
say, of annihilation, “matter disappears” completely, so that none
conservation laws can be valid.

Nonetheless, the idea of Stueckelberg and his followers, J.A. Wheeler and R.P.
Feynman [4], looks quite attractive since it offers a way to construct the
whole collective dynamics from solely simplest considerations purely
kinematical in origin. To realize this idea, in [5] we had proposed to define
a general worldline implicitly, that is, by means of a system of three
algebraic equations

$F_{a}(x_{1},x_{2},x_{3},t)=0,~{}~{}~{}~{}a=1,2,3.$ (1)

Then for any fixed value of the monotonically increasing timelike parameter
$t$ one observes a number of pointlike particles located at $x_{a}(t)$ and
determined by the roots of generating system (1). In [6, 7] we restricted by
the functions $F_{a}$ polynomial in all $x_{a}$ as well as in $t$. Indeed,
only in polynomial case the complete set of the roots of system (1) can be
easily found. Apart from this, just for polynomial equations different roots
are bound by the set of Vieta’s formulas, and this property leads directly to
a set of conservation laws for the ensemble of particles-roots under
consideration [6].

On the other hand, roots of a polynomial system of equations (with real-valued
coefficients) are either real (R) or complex conjugate (C). The latters cannot
be discarded as “nonphysical”: they enter the Vieta’s formulas and contribute,
as a result, to the conservation balance on equal right with the real-valued
roots. We conclude thus that the ensemble in question consists of two
different kinds of particles, R- and C-, respectively.

One can depict a conjugate pair of C-particles in the real 3D space marking
equal real parts of their coordinates. Then one finds that a C-particle is
located aside of the worldline’s trajectory (oftenly, between isolated
branches of the latter).

In the course of time $(t)$ some two real roots (R-particles) approach one
another (on one fixed branch of the trajectory) and, at some $t=t_{0}$, merge
and convert then (at $t>t_{0}$) into a pair of complex conjugate roots, that
is, a C-particle. The latter, after travelling for some time across the
“complex extension” of the worldline, can reach another branch of the real
trajectory and give there rise to a pair of R-particles scattering along the
trajectory. Contrary to the Stueckelberg’s scheme, the whole set of
conservation laws holds in such annihilation/creation processes [6].

It should be noted that on physical 3D space generating equations (1) define
in fact non-relativistic dynamics with quasi-Newtonian absolute time $t$. To
pass to a Lorentz invariant generalization of algebraic dynamics, one can
introduce an external “observer” moving along its own worldline and detecting
the positions of particles on the generating worldline through the Lorentz
invariant light cone connections.

Specifically, in [7] we considered a canonically parameterized worldline
$x_{\mu}(\tau)$ and an observer at rest linked with points on the worldline by
the light cone equation

$(T-t(\tau))^{2}-x(\tau)^{2}-y(\tau)^{2}-z(\tau)^{2}=0,$ (2)

$T$ being the macroscopic monotonically increasing proper time of the
observer. Even in this simplest case, for polynomial parameterization (as well
as for rationally parameterized worldlines, see [9]) the induced collective
dynamics turns out to be conservative. That is, a full set of Lorentz
invariant conservation laws holds for the collection of RC- particles-roots
defined by (2) as $\tau=\\{\tau_{k}(T)\\}\rightarrow
x(T),y(T),z(T)=\\{x_{k}(\tau_{k}),y_{k}(\tau_{k}),z_{k}(\tau_{k})\\}$. Note
that for a non-inertial observer the conservation property is no longer valid.

Remarkably, a number of unexpected effects has been revealed for
asymptotically great values of the observer’s time $T>>1$ if only $n>m$, $n$
and $m$ being the degrees of polynomials $t(\tau)$ and the oldest of
$x_{a}(\tau)$, respectively. Indeed, under these conditions near some critical
value of $T=T_{0}$ one encounters the phenomenon of pairing of the roots while
at greater values $T>>T_{0}$ the arised pairs manifest the effect of
clusterization, that is, formation of closely located (and generally large)
groups of roots-particles [7, 8]. In the case of rationally parameterized
worldlines [9] the emergent dynamics is still more diverse.

Below in the paper, we combine the idea of implicit definition of a worldline
represented by Eq. (1) with the process of detection of the particles’
dynamics on it through the light cone equation for an inertial observer. For
simplicity, we restrict by consideration of plane dynamics. In section 2, we
briefly review the results of [5, 6, 7]: we pay attention to the origination
of conservation property and present then an illustrative example of a
polynomial 2D dynamics. Then, in section 3, the relativistic invariant
dynamics is considered, in particular, for the same worldline as before.
However, particles on the worldline are now linked with an observer at rest by
the light cone equation. In sectiion 4, a set of conservation laws is again
obtained and represented in a manifestly relativistic invariant form. The
asymptotic behaviour is also examined.

##  2 Plane algebraic dynamics on an implicit polynomial worldline

Consider a general system of two polynomial equations of orders $n$ and $m$,
respectively:

$\begin{array}[]{ll}F(x,y,t)=\sum_{p=0}^{p=n}\sum_{i,j,k}^{i+j+k=n}A_{ijk}x^{i}y^{j}t^{k}=0,\\\
G(x,y,t)=\sum_{p=0}^{p=n}\sum_{i,j,k}^{i+j+k=m}B_{ijk}x^{i}y^{j}t^{k}=0,\end{array}$
(3)

where $A_{ijk},B_{ijk}$ are the sets of arbitrary real coefficients (below, to
simplify the computer algebra based analysis, we set them to be integers).

Let us assume that polynomials $F$ and $G$ do not contain a common factor and
the leading coefficients are nonzero. Then for any value of the timelike
parameter $t$ system (3) has exactly $n\times m$ roots, real valued or complex
conjugate [11]. Thus, (3) induces a 2D algebraic dynamics of a set of R- and
C- particles on the plane $(XY)$.

The roots can be found via the procedure of elimination of one of the
unknowns, say, $y$ from the two equations (3). In result, one comes to an
equation for the so-called resultant of two polynomials $R(x,t)$ which itself
is a general polynomial in $x$ and $t$ of the degree $N=n\times m$,

$R(x,t)=\sum_{p=0}^{p=N}\sum_{i,k}^{i+k=p}C_{ik}x^{i}t^{k}=0,$ (4)

$C_{ik}$ being the integers depending on $A_{ijk},B_{ijk}$; under the above
formulated assumption, the leading coefficient $C_{N0}$ cannot turn to zero.
Analogous procedure for elimination of $x$ obviosly leads to a dual equation
for the resultant $\tilde{R}(y,t)$,

$\tilde{R}(y,t)=\sum_{p=0}^{p=N}\sum_{i,k}^{i+k=p}\tilde{C}_{ik}y^{i}t^{k}=0,$
(5)

and, properly combining the N roots of (4) with $N$ roots of (5), one obtains,
for any $t$, a complete set of $N$ solutions
$\\{x_{k}(t),y_{k}(t)\\},~{}k=1,2,\cdots N$ to the original system of
equations (3).

The monoms in (4) and (5) can be equivalently rearranged in the following way:

$\begin{array}[]{ll}R(x,t)=a_{0}x^{N}+a_{1}(t)x^{N-1}+\cdots+a_{N}(t)=0,\\\
\tilde{R}(y,t)=b_{0}y^{N}+b_{1}(t)y^{N-1}+\cdots+b_{N}(t)=0,\end{array}$ (6)

where $a_{0},b_{0}$ are constants, while
$\\{a_{k}(t),b_{k}(t)\\},~{}k=1,2\cdots N$ are polynomials of degree (no more
than) $k$ in $t$.

Now, from the first two (modified, see, e.g.,[12]) Vieta’s formulas linking
$N$ roots of the Eqs. (6) it follows:

$\begin{array}[]{ll}x_{1}+x_{2}+\cdots x_{N}=-a_{1}(t)/a_{0},\\\
y_{1}+y_{2}+\cdots y_{N}=-b_{1}(t)/b_{0},\end{array}$ (7)

and

$\begin{array}[]{ll}x_{1}^{2}+x_{2}^{2}+\cdots+x_{N}^{2}=-2a_{2}(t)/a_{0}+(a_{1}(t)/a_{0})^{2},\\\
y_{1}^{2}+y_{2}^{2}+\cdots+y_{N}^{2}=-2b_{2}(t)/b_{0}+(b_{1}(t)/b_{0})^{2}.\end{array}$
(8)

Note that complex conjugate roots enter in pairs so that l.h.s. in (7,8) are
real-valued. The r.h.s. are linear or quadratic in $t$, respectively.
Differentiating these by $t$, one obtains

$\begin{array}[]{ll}\dot{x}_{1}+\dot{x}_{2}+\cdots+\dot{x}_{N}=constant:=P_{x},\\\
\dot{y}_{1}+\dot{y}_{2}+\cdots+\dot{y}_{N}=constant:=P_{y},\\\ \end{array}$
(9)

and

$\begin{array}[]{ll}(\dot{x}_{1}^{2}+x_{1}\ddot{x}_{1})+\cdots+(\dot{x}_{N}^{2}+x_{N}\ddot{x}_{N})=constant,\\\
(\dot{y}_{1}^{2}+y_{1}\ddot{y}_{1})+\cdots+(\dot{y}_{N}^{2}+y_{N}\ddot{y}_{N})=constant.\end{array}$
(10)

Whether one assumes that masses of all R-particles are equal (so that any
C-particle possesses twice greater mass), relations (9) represent the law of
conservation of total momentum of the set of RC- particles (for the
projections $P_{x}$ and $P_{y}$, respectively). As to relations (10), after
summing these up one comes to a conservation law for a SO(2) scalar – an
analogue of the total energy of the RC ensemble which is closely related to
the virial in classical mechanics (for detail, see e.g. [13]):

$(\dot{r}_{1}^{2}+\vec{r}_{1}\ddot{\vec{r}}_{1})+\cdots+(\dot{r}_{N}^{2}+\vec{r}_{N}\ddot{\vec{r}}_{N})=constant:=E,$
(11)

where $\\{\vec{r}_{k},\dot{\vec{r}}_{k},\ddot{\vec{r}}_{k}\\}$ are the radius-
vector, velocity and acceleration of the $k$-th particle, respectively.

Finally, numerous calculations with different generating polynomials (3) allow
to conclude that conservation of the total angular momentum $\vec{M}$ holds
for any irreducible pair of them (in the 2D case of plane dynamics vector
$\vec{M}$ has only one nonzero component),

$M_{z}=\sum_{i}(x_{i}\dot{y}_{i}-y_{i}\dot{x}_{i}).$ (12)

Special procedure which allows to analytically compute $M_{z}$ for any given
pair of generating polynomials $F$ and $G$ has been elaborated in [6, 7]; it
will be examplified below.

Let us briefly illustrate the above scheme on a randomly selected 444To
simplify the posterior computations in Sect.3, we only take the second
polynomial to be linear in t nondegenerate polynomial system

$\begin{array}[]{lll}F:=x^{3}+2x^{2}y+2t^{3}-t-3=0,\\\
G:=-2x^{3}+y^{3}+6txy-3x^{2}y\\\ +(x+y)t+y+2=0,\end{array}$ (13)

at any $t$ possessing $3\times 3=9$ roots some of which correspond to R-
particles while others, complex conjugate, – to C-particles (with a twice
greater mass). Resolving the second equation w.r.t. $t$ and substituting the
result into the first one we obtain the equation of trajectory of R-particles
which turns out to contain three isolated branches (Fig.1).

Figure 1: Three isolated branches of a “unique worldline” trajectory for
R-particles following from (13)

By finding then corresponding resultants to eliminate $y$ or $x$,
respectively, we obtain

$\begin{array}[]{ll}R(x,t)=5x^{9}+24tx^{8}-(4t-4)x^{7}+\cdots=0,\\\
R(y,t)=5y^{9}-12ty^{8}+(9t+11)y^{7}+\cdots=0.\end{array}$ (14)

Seeking now for values of $t$ at which both resultants have multiple roots,
one gets four such (real) moments, namely
$t\approx-0.9576,0.2565,1.2896,1.2995$. At each of these moments one
encounters an event of annihilation or creation of a pair of R-particles;
maximal number (five) of R-particles exists within the time interval
$1.2896<t<1.2995$.

Now, making use of the (modified) Vieta formulas, one gets

$\begin{array}[]{lll}\sum_{i=1}^{9}x_{i}=-24t/5,~{}~{}~{}~{}\sum_{i=1}^{9}y_{i}=12t/5,\\\
\sum_{i=1}^{9}x_{i}^{2}=(24t/5)^{2}+2(4t-4)/5,\\\
\sum_{i=1}^{9}y_{i}^{2}=(12t/5)^{2}-2(9t+11)/5.\end{array}$ (15)

After differentiation (15) by $t$, we come to a conserved value of the total
momentum,

$P_{x}=\sum_{i=1}^{9}\dot{x}_{i}=-24/5~{}~{}P_{y}=\sum_{i=1}^{9}\dot{y}_{i}=12/5\\\
$ (16)

and to a constant value of the SO(2) scalar, the analogue of total energy,

$E:=\sum_{i=1}^{9}(\dot{x}_{i}^{2}+\dot{y}_{i}^{2})+(x_{i}\ddot{x}_{i}+y_{i}\ddot{y}_{i})=144/5.$
(17)

As for a value of the total angular momentum, to find it we can operate as
follows. Taking total derivatives by $t$ of the initial system (3) and
resolving the arising linear system w.r.t. velocities $\dot{x},\dot{y}$, we
express then the $Z$-component of angular moment (12) through $x,y,t$,
factorize the relation $\mu_{z}-f(x,y,t)=0$ and take the numerator to obtain a
new polynomial equation of the form $H(\mu_{z},x,y,t)=0$. Eliminating then (by
taking subsequent resultants) $x$ and $y$ from the system formed by this
equation together with two initial ones (3), we get an equation of the form
555A redundant factor could arise in the procedure which should be discarded

$Q_{0}\mu_{z}^{9}+Q_{1}\mu_{z}^{8}+\cdots+Q_{9}(t)=0,$ (18)

where the first two coefficients $Q_{0}$ and $Q_{1}$ always happen to be
constant by $t$. Therefore, making use of the first two Vieta’s formulas, for
the value of total angular momentum $M_{z}$ one obtains:

$M_{z}=\sum_{k=1}^{9}\mu_{z}=-Q_{1}/Q_{0}=constant.$ (19)

However, for the particular example of the system (13) $Q_{1}=0$, so that in
this case $M_{z}$ is precisely null.

For interpretation of the dynamical pattern one should examine the asymptotic
regime at great values of $t$. To do this, let us divide Eqs.(13) by $t^{3}$
and neglect the terms of order $1/t$ or smaller. Then for the ratios
$X:=x/t,~{}Y:=y/t$ one obtains a system of two equations with constant
coefficients,

$\begin{array}[]{ll}X^{3}+2X^{2}Y+2=0,\\\
-2X^{3}+Y^{3}+6XY-3X^{2}Y=0,\end{array}$ (20)

which has 3 real roots and 3 pairs of complex conjugate of those. This means
that asymptotically, at great negative $t$, 3R- and 3C-particles approach each
other moving uniformly towards the origin (some of them with superluminar
velocities) while for great positive values of $t$ the same number of them
uniformly scatter to infinity. Asymptotic picture “at the future” as a whole
reproduces that “at the past” though at small values of $t$ one encounters
actually a cascade of mutual transmutations of the two kinds of pointlike
particles. Certainly, such a picture is typical for any polynomial worldline
defined by Eqs.(3).

The above considered algebraic dynamics is completely non-relativistic. The
time $t$ monotonically increase, being absolute, universal and actually
analogous to the Newtonian time. In order to pass to relativistic
generalization we shall input below an external “observer” detecting the
particles on a worldline through the “signals” continiously propagating with
the speed of light from the points on the worldline to the observer.

##  3 Light-detected collective algebraic dynamics on an implicitly defined
worldline

Consider an observer at rest at the origin of (XY) plane receiving the light-
like signals from the points on a worldline implicitly defined by a polynomial
system of equations of the form (3). Then we have to supplement these
equations by the equation of light cone

$S:=(T-t)^{2}-x^{2}-y^{2}=0,$ (21)

$T$ being the (monotonically increasing) proper time of the observer. For any
$T$, one has to find a complete set of roots $\\{x,y,t\\}$ of the system (3)
and (21). Remarkably, the trajectory of R-particles following from (3) will
remain the same, only the law of their motion w.r.t. $T$ will change.

Even the number of particles-roots observed now for any $T$ will be twice
greater than it is for any $t$ according to the system (3). Moreover, in the
course of time $T$ individual times $\\{t_{k}\\}$ can decrease and/or
correspond to the “advanced” signals, signals “from future”. Together with
locations of C-particles at a complex extension of the (XY) plane, the times
$\\{t_{k}\\}$ can be even complex-valued. Actually, the sole physical time in
the scheme is the “macroscopic” time of the observer $T$.

Since the equation of light cone (21) is a nondegenerate polynomial equation
of second order, the collective dynamics defined by the system (3,21) will be
again conservative. The proof can be obtained by a direct generalization of
the procedure described in Sect.2 and is described in detail in [7, 8].

Let us consider the same physical RC-system as it is defined by the equations
(13) but w.r.t. the light-detecting “observer” represented by the light cone
equation (21). We resolve the second equation in (13) w.r.t. $t$ and
substitute it into the first equation (13) and (21). Eliminating then one of
the unknowns, say, $x$ by taking corresponding resultant and repeating the
procedure for the other unknown $y$, we get (compare with (14)):

$\begin{array}[]{llll}r(y)=5430955y^{18}-(19067340T-5278920)y^{17}+\\\
(20644236T^{2}-9522840T-1074680)y^{16}+\cdots=0,\\\
r(x)=5430955x^{18}+(11335920T+4714960)x^{17}\\\
-(13323096T^{2}-4805150T+25495)x^{16}+\cdots=0.\end{array}$ (22)

Making now use of the modified Vieta’s formulas and reproducing the procedure
exposed in Sect.2, we obtain the following conservation laws:

$\begin{array}[]{ll}P_{x}=3813468/1086191,P_{y}=-2267184/1086191,\\\
E=16501798925784/1179810888481.\end{array}$ (23)

Another peculiarity of the observer-dependent scheme for algebraic dynamics is
the emergence of two new conserved quantities – the temporal component of the
conserved total 4-momentum of the RC-ensemble 666In the considered example of
plane 2D dynamics one of the spacial components of the momentum 4-vector is
identically zero, and corresponding second order quantity:

$P_{t}:=\sum_{i}\dot{t}_{i},~{}~{}~{}V_{t}:=\sum_{i}\dot{t}_{i}^{2}+t_{i}\ddot{t}_{i}.$
(24)

To determine $P_{t}$ and $V_{t}$, one should eliminate both unknowns $x$ and
$y$ to obtain the implicit dependence

$\begin{array}[]{ll}r(t)=1086191t^{18}-(219870T-15598)t^{17}+\\\
(2198187T^{2}+779024T-1142373)t^{16}+\cdots=0,\end{array}$ (25)

and again make use of the modified Vieta’s formulas and their derivatives. In
particular, for the example in question (13,21) one gets

$\begin{array}[]{ll}P_{t}=2190870/1986191,\\\
V_{t}=24609485466/1179810888481.\end{array}$ (26)

Now we can compute the 4-scalars corresponding to the 4-momentum and to the
analogue of relativistic energy of the system, namely,

$M^{2}:=P_{x}^{2}+P_{y}^{2}-P_{t}^{2},~{}~{}~{}W:=V_{x}+V_{y}-V_{t}.$ (27)

The first quantity should be identified with the total rest mass of the RC-
system. Numerical values of the above 4-scalars in the considered example turn
out to be

$M^{2}=13701780/1086191,~{}W=151697/1086191.$ (28)

Finally, the nonzero Z-component $M_{z}$ of the total angular momentum vector
is also conserved and, in the particular example, equal to

$M_{z}=-472656/1086191.$ (29)

##  4 Concluding remarks

To begin with, we note that the 3D generalization of the above considered
plane (2D) dynamics can cause only technical problems (see [6, 8]) and none of
fundamental nature. We have also seen that the “subjective” algebraic dynamics
detected on an implicitly defined worldline by an external “observer” through
the light connections essentially differs from the “objective” one determined
by the equations of the worldline itself (3). Nonetheless, in both situations
one encounters a set of pointlike “particles” of two kinds.

The first kind, R-particles, is determined by real-valued roots of the
generating algebraic system of equations while the second, C-particles,
correspond to complex conjugate roots and live aside of the worldline
trajectory. For the whole set of RC-particles the induced “polynomial”
dynamics turns out to be conservative both in “objective” and “subjective”
cases. However, in the latter case the observer should be inertial.

Remarkably, the set of relativistic fields generated by any of the particles
(for detail see, e.g., [10]) do not explicitly contribute to the structure of
conservation relations. This allows to formulate conservation laws in a closed
and manifestly Lorentz invariant form what is in principle impossible in the
framework of the canonical STR.

Specifically, considering the masses of all particles be equal and setting
these be unit, we obtain the law of conservation of total 4-momentum in the
form

$\sum_{i}\dot{x}_{\mu}^{(i)}=constant,$ (30)

where $(\cdot)$ denotes differentiation w.r.t. the unique proper time of the
observer. Corresponding 4-scalar, total rest energy (rest mass), is

$M^{2}=\sum_{i}\dot{x}_{\mu}^{(i)}\dot{x}_{\nu}^{(i)}\eta^{\mu\nu}=constant,$
(31)

where tensor $\eta^{\mu\nu}$ represents the Minkowski metric.

The other conserved 4-scalar related to the Vieta’s formulas of second order
has the form

$W=\sum_{i}(\dot{x}_{\mu}^{(i)}\dot{x}_{\nu}^{(i)}+\ddot{x}_{\mu}^{(i)}x_{\nu}^{(i)})\eta^{\mu\nu}=constant,$
(32)

and could be identified as the analogue of total energy. It differs from (31)
by the second term which is proportional to the particles’ accelerations and
could substitute the energy of radiation.

Recall that under consideration of a polynomial explicitly parameterized
worldline [7, 8] both invariants coincide in value, $W\equiv M=2p\geq 0$, are
positive definite and, moreover, “self-quantized” being equal to twice the
degree $p$ of the polynomial parameterizing the temporal coordinate
$x_{0}(\tau)$. Unfortunately, for implicitly defined worldlines this property
does not hold.

Finally, as we have already presented, numerical experiments “highly likely”
demonstrate that all 6 components of the total angular momentum skew tensor

$M_{[\mu\nu]}:=\sum_{i}x_{\mu}^{(i)}\dot{x}_{\nu}^{(i)}-x_{\nu}^{(i)}\dot{x}_{\mu}^{(i)}=constant.$
(33)

are also preserved for any polynomial implicit worldline (3).

The asymptotic dynamic behaviour, now w.r.t. the invariant proper time $T$,
can be obtained in the same manner as in the previous scheme of internal non-
relativistic dynamics. Particularly, for the system in question (3,21) for the
ratios $X:=x/T,~{}Y:=y/T,~{}S:=t/T$ one gets a system of 3 equations which has
6 real roots and 6 pairs of complex conjugate ones. Thus, we come again to a
picture of “shrinking and then expanding Universe” in which the final stage
completely reproduces the starting one. Unfortunately, there are no
indications for coupling or formation of clusters at late stage of evolution,
contrary to the emergence of these phenomena in the corresponding model with
an explicitly parameterized polinomial [7] or rational [9] worldline.

The authors are grateful to I.Sh. Khasanov for his help in the computer
algebra applications. The publication has been prepared with the support of
the “RUDN University Program 5-100”.

## References

  * [1] H. Minkowski // In: “Raum und Zeit”. – Jahres-bericht der Deutschen Mathematiker-Vereinigung, 18, 75 (1909)
  * [2] E.C.G. Stueckelberg // Helv. Phys. Acta, 14, 588 (1941); 15, 23 (1942)
  * [3] R.P. Feynman // Science, 153, No. 3737, 699 (1966).
  * [4] R.P. Feynman QED: the Strange Theory of Light and Matter. – Princeton: Princeton Univ. Press, 1985;
R.P. Feynman // Phys. Rev., 76, 749 (1949)

  * [5] V.V. Kassandrov, I.Sh. Khasanov // J. Phys. A: Math. Theor., 46, 175206 (2014); arXiv:1211.7002
  * [6] V.V. Kassandrov, I.Sh. Khasanov, N.V. Markova // Bulletin Peoples’ Friend. Univ. Russia: Math. Inform. Phys., 2, 169 (2014); arXiv:1402.6158
  * [7] V.V. Kassandrov, I.Sh. Khasanov, N.V. Markova // J. Phys. A: Math. Theor., 48, 395204 (2015); arXiv:1501.01606
  * [8] I.Sh. Khasanov Ph. D. Thesis. – Moscow:
Peoples’ Friend. Univ. Russia, 2015;
available at: www.dissovet.rudn.ru/web-local

  * [9] V.V. Kassandrov, N.V. Markova // Grav. and Cosm., 22, 363 (2016); arXiv:1702.02869
  * [10] V.V. Kassandrov, J.A. Riscallah // Int. J. Geom. Meth. Mod. Phys., 14, 1750031 (2017); arXiv: 1612.06718
  * [11] Solving polynomial equations: foundations, algorithms and applications, eds. A. Dickenstein, I.Z. Emiris. – Berlin, Springer, 2005.
  * [12] E.A. Kalinina, A.Yu. Uteshev Theory of Elimination. – SPb., SPb. Univ. Press, 2002 (in Russian).
  * [13] H. Goldstein, P.P. Charles, L.S. John Classical Mechanics. – NY: Addison Wesley, 2001


davenet short = DAVEnet , long = deep audio-visual embedding network, alt =
Deep Audio-Visual Embedding Network resdavenet short = ResDAVEnet, long =
residual deep audio-visual embedding network, alt = Residual Deep Audio-Visual
Embedding network AE short = AE , long = autoencoder CAE short = CAE, long =
correspondence autoencoder, alt = Correspondence Autoencoder MCAE short =
MCAE, long = multimodal correspondence autoencoder, alt = Multimodal
Correspondence Autoencoder MTriplet short = MTriplet, long = multimodal
triplet network, alt = Multimodal Triplet Network FFNN short = FFNN, long =
feedforward neural network, alt = Feedforward Neural Networks CNN short = CNN,
long = convolutional neural network, alt = Convolutional Neural Networks RNN
short = RNN, long = recurrent neural network, alt = Recurrent Neural Networks
VQ short = VQ, long = vector-quantisation mfcc short = MFCC, long = mel-
frequency cepstral coefficient CPC short = CPC, long = contrastive predictive
coding, alt = Contrastive Predictive Coding multdavenet short = MultDAVEnet,
long = multilingual deep audio-visual embedding network, alt = Multilingual
Deep Audio-Visual Embedding Network accmultdavenet short = AcstMultDAVEnet,
long = acoustic multilingual deep audio-visual embedding network, alt =
Acoustic Multilingual Deep Audio-Visual Embedding Network DNN short = DNN,
long = deep neural network ASR short = ASR, long = automatic speech
recognition AWE short = AWE, long = acoustic word embedding, alt = Acoustic
Word Embedding AGWE short = AGWE, long = acoustically grounded word embedding,
alt = Acoustically Grounded Word Embedding ASE short = ASE, long = acoustic
span embedding, alt = Acoustic Span Embedding QbE short = QbE, long = query-
by-example, alt = Query-by-Example STD short = STD, long = spoken term
detection UTD short = UTD, long = long short-term memory SGD short = SGD, long
= stochastic gradient decent, alt = Stochastic Gradient Decent bow short =
BoW, long = bag-of-words, alt = Bag-of-Words mse short = MSE, long = mean
squared error, alt = Mean Squared Error vpkl short = VPKL, long = visually
prompted keyword localisation, alt = Visually Prompted Keyword Localisation

# Visually grounded few-shot word learning in low-resource settings

Leanne Nortje, Dan Oneată, Herman Kamper Leanne Nortje and Herman Kamper are
with the Department of Electrical and Electronic Engineering, Stellenbosch
University, Stellenbosch, 7600, South Africa (email<EMAIL_ADDRESS>kamperh@sun.ac.za).Dan Oneată is with the University Politehnica of Bucharest,
Bucharest, RO-060042, Romania (e-mail: dan.oneata@gmail.com).Google DeepMind
funded Leanne Nortje. Dan Oneață was supported by European Union’s HORIZON-
CL4-2021-HUMAN-01 research and innovation program under grant agreement No.
101070190 AI4Trust. We would like to thank Tyler Miller and David Harwath for
helping with the few-shot retrieval comparisons, and Benjamin van Niekerk for
helping with QbERT.

###### Abstract

We propose a visually grounded speech model that learns new words and their
visual depictions from just a few word-image example pairs. Given a set of
test images and a spoken query, we ask the model which image depicts the query
word. Previous work has simplified this few-shot learning problem by either
using an artificial setting with digit word-image pairs or by using a large
number of examples per class. Moreover, all previous studies were performed
using English speech-image data. We propose an approach that can work on
natural word-image pairs but with less examples, i.e. fewer shots, and then
illustrate how this approach can be applied for multimodal few-shot learning
in a real low-resource language, Yorùbá. Our approach involves using the given
word-image example pairs to mine new unsupervised word-image training pairs
from large collections of unlabelled speech and images. Additionally, we use a
word-to-image attention mechanism to determine word-image similarity. With
this new model, we achieve better performance with fewer shots than previous
approaches on an existing English benchmark. Many of the model’s mistakes are
due to confusion between visual concepts co-occurring in similar contexts. The
experiments on Yorùbá show the benefit of transferring knowledge from a
multimodal model trained on a larger set of English speech-image data.111Code
and models: Project webpage.

###### Index Terms:

few-shot learning, multimodal modelling, visually grounded speech models, word
acquisition, low-resource language

##  I Introduction

Speech recognition for low-resource languages faces a major obstacle: it
requires large amounts of transcribed data for development [1]. In some
extreme cases, it might even be impossible to get any labelled data, e.g. when
dealing with an unwritten language. This is in stark contrast to infants that
learn words without access to any transcriptions [2, 3, 4, 5, 6]. This is one
motivation for recent studies into multimodal few-shot learning [7, 8, 9]:
learning new concepts from a few examples, where each example consists of
instances of the same concept but from different modalities. E.g., imagine a
robot seeing a picture of a zebra, kite and sheep while also hearing the
spoken word for each concept. After seeing this small set of examples (called
a support set) the robot is prompted to identify which image in an unseen set
corresponds to the word “zebra”.

Building off of a growing number of studies in visually grounded speech
modelling [10, 11, 12, 13, 14, 15, 16], we consider this multimodal problem of
learning the spoken form of a word and its visual depiction from only a few
paired word-image examples. Multimodal few-shot speech-image learning was
first introduced in [7] and then extended in [8] and [9]. But these studies
were performed in an artificial setting where spoken isolated digits were
paired with MNIST images of digits. This shortcoming was recently addressed by
Miller and Harwath [17], who considered multimodal few-shot learning on
isolated words paired with natural images. Their specific focus was on
learning a new concept while not forgetting previously learned concepts, i.e.
dealing with the problem of catastrophic forgetting. (We do not explicitly
focus on the catastrophic forgetting problem here, although we do evaluate
using the same setup as [17].) While their methods performed well in a few-
shot retrieval task with five classes, they required a relatively large number
of samples per class, i.e. many “shots”. Our first overarching aim is to do
visually grounded multimodal few-shot learning on natural images with fewer
shots. All previous studies also performed experiments using English speech-
image data. Our second goal, therefore, is to present a few-shot evaluation on
a real low-resource language.

Our new multimodal few-shot approach combines two core ideas. Firstly, we use
the support set to “mine” new noisy word-image pairs from unlabelled speech
and image collections. Concretely, each spoken word example in the support set
is compared to each utterance in an unlabelled speech corpus; we use a new
query-by-example approach to identify segments in the search utterances that
match the word in the support set. We follow a similar approach for mining
additional images from the few-shot classes by using cosine distance between
pretrained image embeddings. The mined words and images are then paired up,
thereby artificially increasing the size of our support set in an unsupervised
way. This mining scheme is very similar to that followed in [9], where it was
used on digit image-speech data with simpler within-modality comparisons.
Secondly, our new approach is based on a model with a word-to-image attention
mechanism. This multimodal attention network (MattNet), takes a single word
embedding and calculates its correspondence to each pixel embedding to learn
how the word is depicted within an image. This is similar to the vision
attention part of the model from [18], where the goal was to localise visual
keywords in speech (not in a few-shot setting).

Using the English SpokenCOCO speech-image dataset [19], two evaluation
settings are considered. We first evaluate our approach on the few-shot
retrieval task also used in [17]. We show that MattNet achieves higher
retrieval scores for fewer shots than [17]’s models. Secondly, we evaluate our
approach in a more conventional few-shot classification task where it only
needs to correctly distinguish between classes seen in the support set. We
show that we can achieve five-way accuracies higher than 80% with as little as
five shots.

Still using the English SpokenCOCO models, we then perform an exhaustive
analysis to understand the parts of our approach that are most essential, the
characteristics of mistakes, and how performance differs across keywords. This
includes qualitative results showing that the few-shot model can localise
objects in images given a spoken word query.

To address our second overarching goal, we finally do multimodal few-shot
learning using a Yorùbá speech-image dataset [20], illustrating for the first
time that these few-shot approaches can be applied to a real low-resource
language.

This work is an extension of the conference paper [21], where the main results
on the SpokenCOCO benchmark were presented (a slightly improved model is used
here). The current paper extends this work with a thorough analysis, ablation
experiments, and the application of the model to a low-resource language. To
summarise, we make the following contributions: (1) we introduce a new mining
scheme operating on natural images and speech, (2) we introduce a new
attention-based model for multimodal few-shot learning, (3) we give a thorough
analysis of the proposed approach, and (4) apply the approach in a new low-
resource multimodal few-shot learning benchmark.

##  II Visually grounded few-shot learning and evaluation

We train a model on a few spoken word-image examples. The set of $K$ examples
per class is called the support set $\mathcal{S}$. Each pair in $\mathcal{S}$
consists of an isolated spoken word $\boldsymbol{\mathrm{a}}_{j}$ and a
corresponding image $\boldsymbol{\mathrm{v}}_{j}$. For the one-shot case shown
in the top part of Fig. 1, $\mathcal{S}$ consists of one word-image example
pair for each of the $L$ classes. For the $L$-way $K$-shot task, the support
set
$\mathcal{S}=\\{\boldsymbol{\mathrm{a}}_{j},\boldsymbol{\mathrm{v}}_{j}\\}_{\scriptscriptstyle
j=1}^{\scriptscriptstyle L\times K}$ contains $K$ word-image example pairs for
each of the $L$ classes. In this work, we use a few-shot model for two tasks,
as we describe next.

###  II-A Visually grounded few-shot word classification

In this task, illustrated in the middle and bottom of Fig. 1, we are given an
unseen isolated spoken word query $\boldsymbol{\mathrm{a}}$ and prompted to
identify the corresponding image in a matching set
$\mathcal{M}=\\{\boldsymbol{\mathrm{v}}_{i}\\}_{\scriptscriptstyle
i=1}^{\scriptscriptstyle L}$ of unseen test images. $\mathcal{M}$ contains one
image depicting each of the $L$ classes. Neither the test-time speech query
$\boldsymbol{\mathrm{a}}$ nor any images in $\mathcal{M}$ occur in the support
set. This image-speech task was considered in [7, 8, 9], but here, for the
first time, we use natural images instead of isolated digit images. In
contrast to the task described next, this is conventional few-shot
classification where the model only needs to correctly distinguish between
classes seen in the support set, i.e. there are no other background or
imposter classes.

Figure 1: Given the few examples in the support set $\mathcal{S}$, the
multimodal few-shot classification task is to e.g. identify the image
depicting the word “zebra” from a set of unseen images.

###  II-B Visually grounded few-shot retrieval

In contrast, in this task the goal is to test whether a model can search
through a large collection of images and retrieve those that depict a few-shot
speech query, i.e. the matching set $\mathcal{M}$ in this case contains images
that depict the $L$ few-shot classes but also images that depict other
classes. These additional images might contain completely unseen classes, or
background classes potentially seen during pretraining of the few-shot model.
The model is penalised if it retrieves one of these imposter images. This few-
shot retrieval task was proposed in [17]. Their interest was specifically in
measuring catastrophic forgetting. Since their task requires a model to
distinguish between few-shot classes and other classes, it can be used to not
only determine whether models can be updated to learn new classes from only a
few examples, but also how well the model remembers previously learned
(background) classes. We do not explicitly focus on the catastrophic
forgetting problem, but we want to compare to [17]. Therefore, we also
consider this retrieval task.

For both tasks we need a distance metric
$D_{\mathcal{S}}(\boldsymbol{\mathrm{a}},\boldsymbol{\mathrm{v}})$ between
instances from the speech and vision modalities. Next we describe the model
that we use to compute this distance metric.

##  III Multimodal few-shot attention

Our approach for determining
$D_{\mathcal{S}}(\boldsymbol{\mathrm{a}},\boldsymbol{\mathrm{v}})$ relies on
two core components: a model with a word-to-image attention mechanism and a
method to mine pairs using a few ground truth word-image examples (given in
the support set).

###  III-A Word-to-image attention mechanism

Our model is shown in Fig. 2 and we call it MattNet (Multimodal attention
Network). We adapt the multimodal localising attention model of [18] that
consists of an audio and a vision branch. For the vision branch, we replace
ResNet50 [22] with an adaption of AlexNet [23] to encode an image input
$\boldsymbol{\mathrm{v}}$ into a sequence of embeddings
$\boldsymbol{\mathrm{y}}^{\textrm{v}}$. Originally [21], additional linear
layers were used after the image embeddings, but removing these did not impact
performance. For the audio branch, we use the same audio subnetwork as [18]
that consists of an acoustic network $f^{\textrm{acoustic}}$ which extracts
speech features from a spoken input $\boldsymbol{\mathrm{a}}$. However, [18]
takes an entire spoken utterance as $\boldsymbol{\mathrm{a}}$, whereas we use
a single isolated spoken word. We also add a few linear layers to the BiLSTM
network $f^{\textrm{BiLSTM}}$ to encode the speech features into a single
audio embedding $\boldsymbol{\mathrm{y}}^{\textrm{a}}$, similar to acoustic
word embeddings [24, 25, 26, 27]. We connect the vision and audio branches
with a multimodal attention mechanism to compare the word embedding
$\boldsymbol{\mathrm{y}}^{\textrm{a}}$ to each embedding in
$\boldsymbol{\mathrm{y}}^{\textrm{v}}$.

Figure 2: MattNet consists of (c) a vision and an audio network. The audio
network consists of (a + b) an acoustic context network and a BiLSTM network.
The audio and vision networks are connected with a word-to-image attention
mechanism.

To get this word-to-image attention mechanism, we take the keyword localising
attention mechanism of [18] which detects whether certain keywords occur in
both spoken utterances and images. However, we aim to only detect whether a
single isolated spoken word occurs somewhere within an image. More
specifically, we calculate attention weights over the image embeddings by
calculating the dot product between $\boldsymbol{\mathrm{y}}^{\textrm{a}}$ and
each embedding in $\boldsymbol{\mathrm{y}}^{\textrm{v}}$. By taking the
maximum over the attention scores, we get a similarity score $S$. The higher
$S$, the more probable it is that the spoken word corresponds to one or more
objects in the image. If $S$ is low, it is less probable that any object in
the image corresponds to the spoken word.

We train MattNet with a contrastive loss:

$\scriptsize\begin{split}l&=\textrm{MSE}\left(S(\boldsymbol{\mathrm{a}},\boldsymbol{\mathrm{v}}),100\right)+\sum_{i=1}^{N_{\textrm{pos}}}\textrm{MSE}\Bigg{(}\Big{[}S(\boldsymbol{\mathrm{a}},\boldsymbol{\mathrm{v}}_{i}^{+}),S(\boldsymbol{\mathrm{a}}_{i}^{+},\boldsymbol{\mathrm{v}})\Big{]},100\Bigg{)}\\\
&+\sum_{i=1}^{N_{\textrm{neg}}}\textrm{MSE}\Bigg{(}\Big{[}S\Big{(}\boldsymbol{\mathrm{a}}^{-}_{i},\boldsymbol{\mathrm{v}}),S(\boldsymbol{\mathrm{a}},\boldsymbol{\mathrm{v}}_{i}^{-}),S(\boldsymbol{\mathrm{a}},\boldsymbol{\mathrm{v}}^{\scriptscriptstyle\textrm{bg}}_{i})\Big{]},0\Bigg{)},\end{split}$
(1)

where $S$ is calculated with MattNet and we limit $S\in[0,100]$. Intuitively
this loss should push $\boldsymbol{\mathrm{a}}$, $\boldsymbol{\mathrm{v}}$ and
the positive examples $\boldsymbol{\mathrm{a}}^{+}_{i}$ and
$\boldsymbol{\mathrm{v}}^{+}_{i}$ closer together using a mean square error
(MSE) that pushes the list of similarities to $100$. At the same time the loss
should push the negative examples $\boldsymbol{\mathrm{a}}^{-}_{i}$,
$\boldsymbol{\mathrm{v}}^{-}_{i}$ and
$\boldsymbol{\mathrm{v}}^{\scriptscriptstyle\textrm{bg}}_{i}$ away from these
positives (through the MSE to 0). But before we can do this, we need positive
($\boldsymbol{\mathrm{a}}_{i}^{+}$, $\boldsymbol{\mathrm{v}}_{i}^{+}$),
negative ($\boldsymbol{\mathrm{a}}^{-}_{i}$,
$\boldsymbol{\mathrm{v}}^{-}_{i}$,
$\boldsymbol{\mathrm{v}}^{\scriptscriptstyle\textrm{bg}}_{i}$) and anchor
($\boldsymbol{\mathrm{a}}$, $\boldsymbol{\mathrm{v}}$) pairs.

###  III-B Few-shot pair mining

For few-shot training, we only have the small number of ground truth examples
in the support set $\mathcal{S}$. This would not be sufficient to train the
model. To overcome this, [9] proposed a pair mining scheme: use the audio
examples in $\mathcal{S}$ and compare each example to each utterance in a
large collection of unlabelled audio utterances, and similarly for the images.
The mined items can then be used to construct more word-image pairs for
training. While in [9] the unlabelled collection of audio consisted of
isolated spoken words (which was artificially segmented), here we consider an
unlabelled collection of audio consisting of full spoken utterances (a more
realistic scenario).

The simple isolated-word comparison approach used in [9] is not adequate for
this setting. We employ another approach. We have a spoken word in our support
set that we want to match to unlabelled unsegmented utterances in a large
audio collection. This is similar to fuzzy string search, i.e. finding a set
of strings that approximately match a given pattern. However, algorithms from
string search are not directly applicable to speech since they operate on a
discrete alphabet. We therefore use QbERT (query-by-example with HuBERT). The
idea is to encode speech as a set of discrete units that approximate phones.
Then we can apply standard string search algorithms to find examples that
match a given query word. We use HuBERT [28] to map input speech into discrete
units. Concretely, we use layer seven of HuBERT-Base for K-means quantisation
with 100 clusters. Then we divide the units into variable-duration phone-like
segments following [29]. Finally, we search the dataset by aligning the query
to each utterance using the Needleman-Wunsch algorithm [30]. An alternative to
QbERT would have been to use dynamic time warping (DTW), as is done in [9].
However, in a developmental experiment we found that DTW achieves an isolated
word retrieval $F_{1}$ score of 76.8% while QbERT achieves 98.7%.

Using QbERT, we compare each spoken utterance in an unlabelled collection of
audio utterances to each spoken word example in $\mathcal{S}$. For each
utterance, we take the highest score across the $K$ word examples per class
and rank the utterances from highest to lowest for each class. The first $n$
utterances with the highest scores for a class are predicted to contain the
spoken form of the word. Additionally, we use QbERT’s predicted word segments
to isolate matched words. To mine image pairs, we follow the same steps, but
instead we use AlexNet [23] to extract a single embedding for each image and
use cosine distance to compare image embeddings to one another. To get word-
image pairs, we mine an image from the same predicted class as a segmented
word. Negative pairs are taken from the positive pairs of other classes. We
also mine an extra negative image
$\boldsymbol{\mathrm{v}}^{\scriptscriptstyle\textrm{bg}}_{i}$ from a set known
to not contain any of the few-shot classes (referred to as the background
data, see below). Therefore, during the few-shot retrieval task, images
containing few-shot classes can be distinguished from images that depicts none
of the few-shot classes.

##  IV English Few-Shot Speech-Image Experiments

Before we get to an actual low-resource setting, we do experiments and perform
analyses on an English benchmark.

###  IV-A Experimental setup

Figure 3: The SpokenCOCO data splits used to train and evaluate the MattNet
model. The background data (blue background) consists of spoken audio
utterances and images belonging to concepts not present in the support set.
The mining splits consist of single-modality audio and image samples from
which the training data is artificially extended, and include both background
samples and samples belonging to the few-shot classes (green).

####  IV-A1 Data

For our English experiments, we use the SpokenCOCO Corpus [19] which consists
of the MSCOCO [31] images with recorded spoken captions corresponding to the
MSCOCO textual captions. Fig. 3 shows how we partition the dataset. Firstly,
we use the setup of [17] to divide the SpokenCOCO dataset into a few-shot set
(green) and a background set not containing any of the few-shot classes
(blue).222[17] refers to classes occurring in the background data as base
classes. We use the same few-shot classes as [17]: brocolli, fire hydrant,
kite, sheep and zebra. The background data is used to pretrain
MattNet(§IV-A2).

For the few-shot set, we further divide it into training and testing according
to the splits used by [17]. We use this testing set to sample isolated spoken
word queries and matching images for testing. For the few-shot classification
task (§II-A), we sample images only from the few-shot test set. Since the few-
shot retrieval task (§II-B) requires a single large image matching set, we
take all of the images in the few-shot test set as well as the images in the
background test set. We sample the support set $\mathcal{S}$ from the few-shot
training set (§II), using the Montreal forced aligner [32] to isolate the few-
shot words. To mine pairs (§III-B), we need both an unlabelled audio and image
dataset to mine pairs from; for this we use the remainder of the few-shot
training data that does not include the support set as well as the background
training data, shown in the left half of Fig. 3. From these unlabelled
collections, we mine pairs: the $n=600$ highest ranking examples per class
(§III-B). Lastly, these pairs are split into training and validation pairs.

Utterances are parametrised as mel-spectograms with a hop length of 10 ms, a
window of 25 ms and 40 mel bins. These are truncated or zero-padded to 1024
frames. Images are resized to 224$\times$224 pixels and normalised with means
and variances calculated on ImageNet [33].

####  IV-A2 Models

Fig. 2 illustrates our model, MattNet (§III-A). For the image branch, we use
an adaption of AlexNet [23] to get image embeddings. This image branch is also
initialised using the pretrained convolutional encoder (before the
classification network) of AlexNet. We use an adaption of [18]’s audio network
for the audio branch. This acoustic network is pretrained on LibriSpeech [34]
and the multilingual (English and Hindi) Places dataset [35] using a self-
supervised contrastive predictive coding task [36]. While there are more
modern alternatives for the vision and audio networks, we chose these
particular variants to limit computational requirements. After initialisation,
the combined MattNet model is pretrained on the background data (blue, Fig. 3)
using the contrastive speech-image retrieval loss of [37]. This is then the
starting point for the model that we update using mining.

TABLE I: The setup for each MattNet variant. Model | Pretrain | Mine training pairs | Fine-tune
---|---|---|---
MattNet | ✓ | ✓ | ✓
MattNet, no mining | ✓ | ✗ | ✓
MattNet, no fine-tuned | ✓ | ✗ | ✗

During training on the SpokenCOCO mined pairs, we take $N_{\textrm{pos}}=5$
and $N_{\textrm{neg}}=11$ in (1). These values were fine-tuned on the
validation pairs. We train all models with Adam [38] for 100 epochs using a
validation task for early stopping. For the validation task, we use the
validation set to get one positive image $\boldsymbol{\mathrm{v}}^{+}$ and one
negative image $\boldsymbol{\mathrm{v}}^{-}$ for each validation word-image
$(\boldsymbol{\mathrm{a}},\,\boldsymbol{\mathrm{v}})$ pair. The validation
task measures whether the model will place $\boldsymbol{\mathrm{v}}$ and
$\boldsymbol{\mathrm{v}}^{+}$ closer to $\boldsymbol{\mathrm{a}}$ than it
would $\boldsymbol{\mathrm{v}}^{-}$.

To understand the models we discuss in the following section, we give the
model names and their setup in Table I. Our full model is referred to as
“MattNet”; it is pretrained on background data and then fine-tuned on the
mined pairs. For the “MattNet, no mining” model, instead of training on the
mined pairs, we only update the model on the samples in the support set. The
“MattNet, no fine-tuned” model consists of only the pretrained model; it is
not updated with the few-shot classes in any way. To do the few-shot tasks
with this model, we use the indirect few-shot method of [7, 8]: each
$\boldsymbol{\mathrm{a}}$ is compared to each $\boldsymbol{\mathrm{a}}_{j}$ in
$\mathcal{S}$ to find the audio example closest to the query. The image
$\boldsymbol{\mathrm{v}}_{j}$ corresponding to the closest
$\boldsymbol{\mathrm{a}}_{j}$ is then used to calculate the similarity to each
image $\boldsymbol{\mathrm{v}}_{i}$ in $\mathcal{M}$. This model can thus be
seen as evaluating the quality of the embedding spaces obtained through
pertaining when data from unseen classes are presented to the model.

####  IV-A3 Few-shot evaluation tasks

We evaluate our approach on two tasks (as explained in §II): a traditional
few-shot classification task and a few-shot retrieval task. For both tasks,
the $K$-shot $L$-way support set $\mathcal{S}$ contains $K$ ground truth
spoken word-image pairs for each of the $L=5$ classes and is used to mine
pairs for training and validation. In the few-shot classification task, we
sample $1000$ episodes where each episode contains $L$ spoken word queries
$\boldsymbol{\mathrm{a}}$, one for each class, and a matching set
$\mathcal{M}$ which contains one image $\boldsymbol{\mathrm{v}}_{i}$ for each
class. In the few-shot retrieval task, instead of having one image per class,
$\mathcal{M}$ consists of $5000$ images $\boldsymbol{\mathrm{v}}_{i}$ where
some depict a few-shot class and others do not. Here, 20 query words are taken
per class and averaged to get $\boldsymbol{\mathrm{a}}$. For each of the $L$
queries $\boldsymbol{\mathrm{a}}$, these $5000$ images are ranked from highest
to lowest similarity. The precision at $N$ ($P@N$) score is the proportion of
images in the top $N$ highest ranking images that are from the same class as
$\boldsymbol{\mathrm{a}}$. $N$ is the actual number of images in $\mathcal{M}$
that depicts the word class.

###  IV-B Experimental results

TABLE II: $P$@N few-shot retrieval scores (%) on the five few-shot classes. $K$ is the number of support-set examples per class.  Model | $K$
---|---
| 5 | 10 | 50 | 100
Naive fine-tuned [17] | – | – | – | 52.5
Oracle masking [17] | – |  8.4$\pm$0.0 |  24.0$\pm$0.1 |  35.5$\pm$0.2
MattNet | 40.3$\pm$0.1 | 44.2$\pm$0.1 | 41.7$\pm$0.2 |  43.7$\pm$0.1
MattNet, no mining |  13.2$\pm$0.6 |  34.8$\pm$0.7 |  40.9$\pm$0.3 |  40.5$\pm$0.5
MattNet, no fine-tuned |  22.0$\pm$0.4 |  24.1$\pm$0.8 |  22.7$\pm$0.5 |  23.2$\pm$1.1
TABLE III: Few-shot word classification accuracy (%) when varying the number of shots per class $K$. Model | $K$
---|---
| 5 | 10 | 50 | 100
MattNet | 80.1 | 81.1 | 88.5 | 93.2
MattNet, no mining | 53.1 | 79.4 | 93.3 | 95.5
MattNet, no fine-tuned | 50.4 | 48.0 | 48.5 | 47.7

We start by comparing to two of [17]’s models on the few-shot retrieval task.
The first is their naive model, which is pretrained on background classes and
then fine-tuned on $K=100$ examples for each of the $L=5$ classes. The second
is an oracle masking model in which the contrastive loss used during fine-
tuning ensures that a negative image does not contain any instance of the
anchor few-shot class. The results are given in Table II.333These scores are
slightly different from those in [21] because we do not apply linear layers
over the image encodings (§III-A).  (Not all settings considered here were
evaluated in [17], so these are indicated with dashes.)

Our full MattNet model outperforms the oracle model across all values of $K$.
Neither MattNet nor the oracle masking works as well as the naive fine-tuned
approach for a high number of shots (line 1, $K=100$). In the forth line we
see that our no mining approach, which is equivalent to naive fine-tuning,
does worse than the naive model from [17] at $K=100$. It is important to note
that we use a different architecture. We can, however, conclude that direct
fine-tuning only works with a large number of shots; as the number of shots
increases, we get closer to a standard supervised learning setting, and it is
therefore unsurprising that at some point naive fine-tuning starts to
outperform few-shot methods. But, taking all this together, it is clear that
our approach outperforms the existing methods with fewer shots.

To determine the contribution of both mining and fine-tuning, we do an
experiment where we do not update MattNet on the few-shot classes after
pretraining it on the background data (“MattNet, no fine-tuned”). We see that
these two components improve the scores by roughly 20% in absolute performance
when comparing lines 3 and 5 in Table II.

To further analyse the performance gains from mining, we now consider the
conventional few-shot word classification task (§II-A). This task wasn’t used
in [17]. Table III shows that the few-shot classification scores increase as
$K$ increases when we use mined pairs. For the no fine-tuning method, the
scores are lower than that of MattNet and decrease slightly as $K$ increases.
The no mining approach in which we only update the model on the support set
samples has a steep increase in scores from $K=5$ to $K=10$, after which the
scores overtake MattNet. Again, this makes sense as the amount of training
data ($K$) increases. Again, these results illustrate the effectiveness of
MattNet when we have fewer shots.

Altogether, we set a competitive multimodal baseline for both few-shot
retrieval and word classification in settings where the number of shots is
small. However, the results do raise some questions, including the following:
why do the retrieval scores for MattNet plateau (Table II) but the
classification scores increase (Table III) as $K$ increases? We unpack this as
part of the analysis in the next section.

##  V Further Analysis of English

In this section we investigate the retrieval performance plateau we
encountered in the previous section. We also present a finer-grained analysis
looking into various aspects that contribute to our approach’s performance.

###  V-A Qualitative error analysis

query: fire hydrant | query: kite
---|---
|
task: few-shot retrieval | task: few-shot retrieval
rank: 1 · ✓ |  rank: 2 · ✗ |  rank: 3 · ✗ |  rank: 4 · ✗ |  rank: 5 · ✗ |  rank: 1 · ✓ |  rank: 2 · ✗ |  rank: 3 · ✗ |  rank: 4 · ✓ |  rank: 5 · ✓
|  |  |  |  |  |  |  |  |
score: 94.0 | score: 80.7 | score: 77.0 | score: 74.0 | score: 72.2 | score: 103.8 | score: 99.2 | score: 95.2 | score: 95.2 | score: 91.8
task: few-shot classification | task: few-shot classification
broccoli | zebra | fire hydrant | sheep | kite | broccoli | zebra | fire hydrant | sheep | kite
|  |  |  |  |  |  |  |  |
score: 1.2 | score: −0.4 | score: 15.5 | score: 2.4 | score: 7.0 | score: −6.4 | score: −0.5 | score: 10.2 | score: 7.7 | score: 12.2
Figure 4: Examples of retrieval and few-shot classification for two queries
using the $K=100$ MattNet model. Concepts that associate strongly with
context, such as fire hydrant which often appears in urban environments, are
more challenging to retrieve than to classify.

We start with a qualitative analysis to compare the few-shot classification
and retrieval tasks. Fig. 4 shows the five matching set images for
classification and the five images that the model retrieves when given the
spoken queries “fire hydrant” and “kite”. The analysis suggests that for some
classes the model depends on the contextual information to identify the class.
Since the few-shot image classes and their contexts are quite distinct from
one another, it does not have a significant influence on the classification
task (bottom, Fig. 4). However, this has a more significant effect on
retrieval since contextual information may overlap more between the few-shot
images and background images (top). E.g. there might be multiple images
containing streets, but only some of them contain fire hydrants. For
classification, this learned association actually helps since none of the
other few-shot classes involves streets.

###  V-B Per-keyword analysis

TABLE IV: $P@N$ few-shot retrieval scores (%) for each of the five few-shot classes. $K$ is the number of support-set examples per class. The number of examples present in the matching set $\mathcal{M}$ for each class, $N$, is given in brackets. | broccoli | fire hydrant | kite | sheep | zebra
---|---|---|---|---|---
$K$ | (57) | (62) | (91) | (63) | (90)
5 | 40.4 | 11.3 | 27.5 | 33.3 | 77.8
10 | 49.1 | 9.7 | 30.8 | 33.3 | 85.6
50 | 50.9 | 8.1 | 20.9 | 36.5 | 82.2
100 | 52.6 | 4.8 | 33.0 | 38.1 | 80.0

Table IV presents the individual retrieval performance for each of the five
few-shot keywords. (In §IV-B, specifically Table II, retrieval scores were
aggregated over the keyword types.) We observe a large variance in performance
across the concepts: fire hydrant is the most challenging keyword to retrieve
(with a performance as low as 4.8% in $P@N$), while zebra is the easiest (over
85% when $K=10$). The differences between keywords could be due to a number of
factors, object size and word frequency probably being among the most
important ones, as suggested by [17, Table 4].

Interestingly, if we look at how the performance varies with the number of
shots $K$, we get a more nuanced picture than the one provided by the
aggregated results (in Table II). While we previously observed the aggregated
performance staying approximately the same with larger $K$, here we instead
see that there are variations in both directions depending on the keyword: the
retrieval scores improve with $K$ for sheep and broccoli, but degrade for fire
hydrant and fluctuates for kite and zebra. We speculate that as $K$ increases
for classes that have low retrieval scores (fire hydrant, kite), the model
becomes more dependent on the recurring contextual (background) features,
which are especially difficult to disentangle for the objects that occupy only
a small part of the image. I.e. when the few-shot object is small, the model
struggles to focus on the few-shot object and therefore depends on the
contextual data to learn the class. Therefore, increasing $K$ does not help.
We have already seen this to some degree in the examples of Fig. 4; next we
investigate this by analysing the model’s localisations.

###  V-C Localisation visualisation

broccoli | zebra | fire hydrant | sheep | kite
---|---|---|---|---
|  |  |  |
IOU: 19.3% | IOU: 35.3% | IOU: 0.2% | IOU: 12.6% | IOU: 0.5%
rank: 1 · is correct: ✗ |  rank: 1 · is correct: ✓ |  rank: 1 · is correct: ✓ |  rank: 1 · is correct: ✗ |  rank: 1 · is correct: ✓
|  |  |  |  |  |  |  |  |
rank: 2 · is correct: ✓ |  rank: 2 · is correct: ✓ |  rank: 2 · is correct: ✗ |  rank: 2 · is correct: ✗ |  rank: 2 · is correct: ✗
|  |  |  |  |  |  |  |  |
rank: 3 · is correct: ✗ |  rank: 3 · is correct: ✓ |  rank: 3 · is correct: ✗ |  rank: 3 · is correct: ✓ |  rank: 3 · is correct: ✗
|  |  |  |  |  |  |  |  |
rank: 4 · is correct: ✗ |  rank: 4 · is correct: ✓ |  rank: 4 · is correct: ✗ |  rank: 4 · is correct: ✗ |  rank: 4 · is correct: ✓
|  |  |  |  |  |  |  |  |
rank: 5 · is correct: ✓ |  rank: 5 · is correct: ✓ |  rank: 5 · is correct: ✗ |  rank: 5 · is correct: ✗ |  rank: 5 · is correct: ✓
|  |  |  |  |  |  |  |  |
Figure 5: The top five ranked samples for audio query corresponding to each of
the five concepts using the $K=100$ MattNet model. For each image, we show
whether it is correct (if it contains the query concept) and the attention
explanation (red indicating the input regions relevant for the given audio
query). The IOU values give the quantitative localisation performance of the
attention explanations: the intersection over union of the binarised
attentions with the ground truth annotations averaged over all images that
contain a given concept.

Where does MattNet focus in an image when given an audio query? We make use of
the implicit localisation capabilities provided by our model. More precisely,
since MattNet projects the image features to attention scores (based on the
audio feature, §III-A), we first reshape the 49-dimensional attention vector
to a $7\times 7$ matrix, which we then resize with bilinear interpolation to
the original image size. The resulting map reveals the importance of the input
regions: the closer the attention score is to 100, the more likely it is
relevant to the input audio query. Fig. 5 shows the top five retrieved images
together with their attention maps for each of the five keywords.

These visualisations are useful for performing implicit localisation of the
spoken concepts in the input images, but, perhaps more importantly, they allow
us to better understand the model. We observe that for “broccoli” the model
selects the green food in an image; the “fire hydrant” queries are associated
with urban scenes (streets and cars); and the keyword “kite” is linked to the
seaside or fields. While all these associations are useful proxies for
identifying the spoken keywords, they also indicate that the model tends to
learn correlations with the context or other spurious features. A quantitative
analysis in terms of the intersection over union (IOU) with ground truth
annotations supports these findings, as shown at the top of Fig. 5. This
effect is not surprising given that the model learns in a weakly-supervised
manner (the training is done on full images with no explicit localisation
information) and with noisy data (the mined samples are not always accurate).

With the analyses up to this point we can explain the discrepancies seen at
the end of §IV. Concretely, we conclude that the plateau seen in the retrieval
scores (Table II) is due to the model associating contextual information or
other spurious features with a few-shot class. E.g. associating anything green
with broccoli, urban areas or vertical pole-like objects with fire hydrant,
and people in fields with kite. This aids few-shot classification since the
context or faulty features might help to distinguish the few-shot classes.
However, this hurts few-shot retrieval since the model will return green
objects (field or grass) when prompted with “broccoli”.

###  V-D Accuracy of the mining pairs

TABLE V: Precision (%) of the audio and image pairs mined from the $K$ few-shot examples in $\mathcal{S}$. $K$ | Audio pairs | Image pairs
---|---|---
5 | 81.1 | 43.9
10 | 83.0 | 47.5
50 | 85.4 | 48.0
100 | 87.6 | 51.5

In §III-B we describe the method we use to mine training pairs from only the
few ground truth word-image pairs in the support set $\mathcal{S}$: to find
audio matches for each spoken keyword in $\mathcal{S}$ from a large collection
of unlabelled speech, we use QbERT to extract possible word matches.
Similarly, we use AlexNet to find image matches for each image class in
$\mathcal{S}$. This means that the audio and image pairs we use for training
are not 100% correct. In order to evaluate the effect that the mined pairs
have on MattNet’s performance, we report the precision of these pairs in Table
V. For both the audio and image pairs, the precision increases as $K$
increases.

###  V-E Importance of the background data

TABLE VI: Few-shot retrieval scores (%) obtained from MattNet trained with and without negative background images. Model | $K$
---|---
| 5 | 10 | 50 | 100
MattNet | 40.3$\pm$0.1 | 44.2$\pm$0.1 | 41.7$\pm$0.2 | 43.7$\pm$0.1
MattNet, no background data |  18.1$\pm$1.0 |  23.6$\pm$1.5 |  26.1$\pm$0.2 |  23.1$\pm$0.5
MattNet, no background image negatives |  29.9$\pm$0.1 |  31.4$\pm$0.2 |  32.2$\pm$0.2 |  32.1$\pm$0.1
TABLE VII: Few-shot classification accuracy (%) obtained from MattNet trained with and without negative background images. Model | $K$
---|---
| 5 | 10 | 50 | 100
MattNet | 80.1 | 81.1 | 88.5 | 93.2
MattNet, no background data | 65.1 | 64.7 | 75.3 | 77.5
MattNet, no background image negatives | 88.0 | 90.1 | 94.8 | 95.3

Ideally, the usage of background negative images should force the model to
throw away more contextual information and focus more on learning the few-shot
objects, since the background negatives might have similar contexts to some of
the few-shot cases. But since we see that contextual information is not
completely ignored in the above analyses, we do the analysis in Tables VI and
VII: In the second line of both tables, we have a model which is not
pretrained at all and also uses no image negatives during fine-tuning on the
mined pairs (§III-B), i.e. background data (§IV-A1) is not used at all. By
comparing lines 1 and 2 of Table VI, we see that retrieval scores drops
substantially. We can therefore conclude that adding the background
information does remove some contextual information, but not all of it.

Interestingly, we see that the classification scores also decrease (lines 1
and 2 of Table VII) when we remove all background data. But this is mainly due
to the effect of pretraining. To see this, we look at the contribution of the
background image negatives: we train a MattNet using the pretrained network
but leave out the negative background images when fine-tuning. This model is
listed in the third row of the two tables. For classification (Table VII), we
see that this approach actually improves performance over MattNet (line 1).
This makes sense since the model is now fine-tuned exclusively on (mined) few-
shot classes. But it also illustrates that pretraining on background data is
essential. For retrieval (Table VI), we see an expected drop in performance
comparing lines 1 and 3, because the latter model is not trained to
distinguish between few-shot and non-few-shot classes.

###  V-F Adding more keywords

Can we use our model to deal with more than five classes? We leave an
exhaustive investigation of this question for future work, but present some
initial experiments here. Concretely, we do a test on 40 classes, which we
manually select from the dataset. When training MattNet on the 40 classes,
performance on the same few-shot retrieval task for the original five classes
drops marginally from 40.3% to 37.1%. This shows that, despite being trained
on more classes, the model still retains most of its retrieval performance.
Similarly, classification performance on the original five classes drops from
80.1% to 74.3%. When doing 40-way few-shot classification, we achieve a
performance of 23.8% (this is a much more difficult task than the five-way
setting).

Figure 6: The per-keyword classification scores for the 40 few-shot classes.

To see what happens on the per-keyword level when MattNet is trained and
tested on 40 few-shot classes, Fig. 6 shows the individual scores. With 40
classes, there is more room for error. We also see a larger distribution of
few-shot classes that the model struggles to learn. Future work will look into
improving the mined image pairs specifically (since the audio pairs are more
accurate).

##  VI Actual Low-Resource Few-Shot Speech-Image Experiments

Our ultimate goal is to do multimodal few-shot word acquisition on actual low-
resource languages. To showcase our model’s capabilities in this regard, we
perform multimodal few-shot experiments on Yorùbá, a low-resource language
spoken in Nigeria by roughly 44 million people. We apply MattNet (§III) to the
Yorùbá Flickr Audio Caption Corpus (YFACC) [20]. Because the dataset is small,
we consider only the case with five shots per class ($K=5$). We also train an
English version of our model using a similarly sized dataset for comparison.

###  VI-A Experimental setup

####  VI-A1 Data

YFACC is an extension of the original English Flickr image-text captioning
corpus [39, 40]. Concretely, for each of the 8k Flickr images, one English
text caption was translated into Yorùbá and then recorded by a single speaker.
The YFACC paper [20] looked specifically at a keyword spotting task for 67
Yorùbá keywords (matching the English keywords from [41]). The YFACC test set
therefore contains 500 spoken captions with manual alignments where each
caption contains at least one of their 67 keywords. Since not all our original
few-shot classes occur in this set (§IV-A), we choose five new classes from
their 67 keywords. The Yorùbá keywords with their English translations are
given in Table VIII.

TABLE VIII: The five new few-shot classes for the English and Yorùbá Flickr experiments. English | Yorùbá
---|---
boy | ọmọkùnrin
dogs | àwọn ajá
grass | koríko
rock | àpáta
water | omi

Since only the test set contains alignments for the keywords, we have to
sample the support set from the test set and use the remaining examples to
sample episodes for the few-shot classification task. For the support set
images, we crop the images to only contain a single few-shot class since the
new classes frequently co-occur within images (unlike the setting for the
English experiments in §IV). We stress that none of the examples in the
support set occurs in the few-shot episodes’ matching sets or queries. We
sample the episodes for the new Flickr keywords in the same manner as in
§IV-A3.

As a baseline, we also train an English system on very similar data. For this,
we use the English Flickr Audio Captions Corpus (FACC) [42] which preceded and
inspired YFACC. We follow the same setup as for the Yorùbá model, but replace
the Yorùbá utterances from YFACC with the corresponding English utterances
from FACC.

####  VI-A2 Models

Using the sampled support set, we mine image and audio pairs in the same way
as in §V-D, with the only difference that we take the $n=100$ highest ranking
examples per class because the dataset is smaller. For mining audio pairs in
Yorùbá, we still use the QbERT approach, based on an English-trained HuBERT
model (§III-B), even though we are searching through unlabelled Yorùbá audio.
I.e. we apply QbERT cross-lingually. For the English model, we do not use
weights pretrained on the MSCOCO background data as we did before, since this
data might contain paired instances of our few-shot classes. There is
therefore no background speech-image pretraining. However, we use the
pretrained convolutional part of AlexNet, as well as the pretrained acoustic
network of [21] for initialisation (§IV-A2). The rest of the implementations
remain the same as set out in §IV-A.

####  VI-A3 Few-shot evaluation

For evaluation, we only consider the few-shot classification task. We report
performance over $1000$ episodes, each episode consisting of five spoken query
words (one for each of the few-shot classes) and five images (also one for
each of the few-shot classes) in the matching set. In contrast to the English
experiments in §IV and §V, each matching image can potentially belong to more
than one few-shot class. The reason for this is that the few-shot classes
frequently co-occur within Flickr images. To determine whether a particular
image contains one of the few-shot classes, we use the ground truth text
transcriptions available with the Flickr data (five text captions per image
for English and one per image for Yorùbá). We mark a prediction as correct
when the query audio word matches any of the words in a transcript.

###  VI-B Experimental results

TABLE IX: Few-shot classification accuracy (%) on the Flickr data with English and three Yorùbá versions of MattNet. The Yorùbá pt model is a Yorùbá model initialised with the weights of the pretrained SpokenCOCO English model from §IV-A2. |  | Average | boy | dogs | grass | rock | water
---|---|---|---|---|---|---|---
Model | Mine | accuracy | ọmọkùnrin | àwọn ajá | koríko | àpáta | omi
Eng. | ✓ | 59.4 | 56.9 | 28.9 | 64.3 | 78.4 | 68.3
Yor. | ✓ | 36.3 | 18.8 | 23.2 | 35.4 | 49.8 | 54.5
Yor. pt | ✓ | 62.0 | 66.9 | 37.4 | 56.6 | 78.0 | 71.3
Yor. pt | ✗ | 29.4 | 47.1 | 30.7 | 29.7 | 21.1 | 18.2

The multimodal few-shot classification results are given in Table IX. Line 2
represents the first time that multimodal few-shot word classification is
performed on a real low-resource language, Yorùbá. We see that the performance
of the Yorùbá model on this task (36.3%) is worse compared to the English
baseline (59.4%), trained and evaluated using a similar setup.

We didn’t initialise the English baseline model here by pertaining it on the
background SpokenCOCO data as we did before (§IV-A) because this background
data might contain paired English instances of the few-shot classes that we
are using here. However, it is a fair experiment to use the SpokenCOCO English
data to initialise a Yorùbá MattNet, i.e. we use the available resources from
a well-resourced language to transfer knowledge to a low-resourced setting.
The results for this approach are given in line 3 of Table IX. This Yorùbá
MattNet model outperforms both the English and Yorùbá model without
pretraining in terms of overall accuracy. Per-keyword classification scores
are also given in Table IX. The English-pretrained Yorùbá model gives
substantial improvements over the non-pretrained model across all five keyword
classes, and also outperforms the English model on three of the few-shot
classes.

English
---
matching set & annotations
boy | dogs | dogs, grass | rock | boy, water
|  |  |  |
audio queries
boy | dogs | grass | rock |  water
|  |  |  |
predicted images with attention, scores & correctness
|  |  |  |
12.3 · ✓ |  23.3 · ✓ |  64.7 · ✗ |  41.1 · ✓ |  51.8 · ✗
Yorùbá
matching set & annotations
|  | àwọn ajá |  |
ọmọkùnrin | àwọn ajá | koríko | àpáta | omi
|  |  |  |
audio queries
ọmọkùnrin | àwọn ajá | koríko | àpáta | omi
|  |  |  |
predicted images with attention, scores & correctness
|  |  |  |
29.0 · ✗ |  25.3 · ✓ |  72.6 · ✗ |  112.5 · ✓ |  108.2 · ✓
Figure 7: Qualitative results showing the first few-shot classification
episode on Flickr for English and Yorùbá. For Yorùbá we are showing results
using the model with pretraining on background SpokenCOCO data. The labels are
selected based on the text captions. We show attention weights overlaid on top
of the images, with blue indicating the lowest scores and red the highest.

Fig. 7 shows qualitative examples on the first episode for the English model
and the Yorùbá model with pretraining. We observe that many of the predictions
are accurate. Some that are deemed to be wrong could in fact be considered
correct. As a reminder, our evaluation is based on the captions associated
with each image, and in some cases the captions might omit a keyword even
though it is actually present in the image [43]. E.g. “grass” is not mentioned
in relation to the image with the boy jumping in the pool, but grass does
occur in the image, and the model selects this image. So even though it is
marked as a mistake in our evaluation protocol, it is actually correct. The
figure also shows that the pretrained Yorùbá model seems more confident—the
attention scores are higher for a given audio keyword—than the English one,
showing that the pretraining improved not only the overall performance but
also the attention explanations.

Altogether, the experiments on Yorùbá shows that MattNet can be applied to
learn word-image classes from a few examples in a low-resource language when
we also take advantage of a well-resourced language like English. This is
similar to the findings of [44, 45].

###  VI-C Mining performance across languages

TABLE X: Precision scores (%) of the mined English and Yorùbá Flickr pairs, for image and audio mining. | boy | dogs | grass | rock | water
---|---|---|---|---|---
Modality | ọmọkùnrin | àwọn ajá | koríko | àpáta | omi
Images | 38 | 46 | 62 | 19 | 80
English audio | 95 | 45 | 87 | 49 | 97
Yorùbá audio | 70 | 42 | 58 | 57 | 84

The last question that remains is: what is the effect of using an English
speech system, QbERT, to mine Yorùbá audio pairs? Here we investigate the
mining performance of the audio segments. In Table X we show the precision of
the mined audio pairs for the two languages and five keywords. To get the
precision scores, a mined audio segment is considered correct if the query
keyword appears _anywhere_ in the caption (that is, we do not take the
temporal alignments of the mined segment into consideration).444For English,
the differences between this metric and its stricter variant that also checks
that the temporal alignments of the mined segment agree with the spoken word
are minute: less than 2% absolute.

We observe that for English three keywords (boy, grass and water) obtain
excellent results (around 90% precision), while the other two are performing
more modestly (around 50% precision). To understand these results we find the
most common mistakes for each of these three keywords; unsurprisingly, these
mistakes are phonetically similar words: “dogs” is confused with “dog” (51
times out of the 100 mined samples) and “dock” (1), while “rock” is confused
with “dog” (20) and “rocky” (13). For Yorùbá the performance across keywords
has a more uniform spread and, while the top performance is lower than what we
obtain on English, many of the keywords still obtain a reasonable precision.

Interestingly, the performance on the mined audio segments does not seem to
necessarily correlate with the final performance. E.g. while the mined audio
pairs of rock are among the worst, its downstream performance is the best for
both the English and the Yorùbá pretrained models. A similar observation can
be made for boy, which is accurately mined, but whose few-shot classification
are poor compared to the other keywords. One reason for this might be that the
accuracy of the mined image pairs for boy is low, as indicated in the first
line of Table X which shows the precision of the mined image pairs (which is
the same for both languages).

To quantify the effect of mining on classification performance, we report
results for the “Yorùbá pt, no mining” model on each of the five keywords
(Table IX, row 4). Comparing this to when mining is used (Table IX, row 3), we
see that mining is beneficial even for keywords with modest mining
performance. This suggests that more positive samples, even noisy ones, are
better than fewer clean ones.

##  VII Conclusion

Our goal was to do multimodal few-shot learning of natural images and spoken
words. We proposed a novel few-shot pair mining method which we use in a new
multimodal word-to-image attention model. For the scenario where the number of
“shots” is small, our new model achieves higher few-shot retrieval scores than
an existing model on a few-shot benchmark where an English query is used to
retrieve images. We also set a competitive baseline for natural visually
grounded few-shot word classification using English data.

In further analyses, we showed that a few-shot model can be used to locate
occurrences of an object in an image given a spoken query, and that many of
the model’s mistakes are due to associating contextual information with a few-
shot class, e.g. fire-hydrant often co-occurring with street views.

To showcase that our model can also be applied to a real low-resource
language, we performed—for the first time—multimodal few-shot learning on a
real low-resource language, Yorùbá. We showed that a multimodal Yorùbá few-
shot model can benefit substantially from being initialised on a more
substantial amount of English speech-image data.

Future work will look into extending low-resource few-shot word classification
to even more classes. On both English and Yorùbá data, our analysis also
revealed that the image matching step in the mining scheme might, in
particular, be limiting performance. Future work should therefore use recent
advances from the vision community to improve image mining.

## References

  * [1] L. Besacier, E. Barnard, A. Karpov, and T. Schultz, “Automatic speech recognition for under-resourced languages: A survey,” _Speech Commun._ , 2014.
  * [2] I. Biederman, “Recognition-by-components: A theory of human image understanding.” _Psych. Review_ , 1987.
  * [3] G. Miller and P. Gildea, “How children learn words,” _SciAM_ , 1987.
  * [4] R. L. Gómez and L. Gerken, “Infant artificial language learning and language acquisition,” _TiCS_ , 2000.
  * [5] B. M. Lake, C.-y. Lee, J. R. Glass, and J. B. Tenenbaum, “One-shot learning of generative speech concepts,” _CogSci_ , 2014.
  * [6] O. Räsänen and H. Rasilo, “A joint model of word segmentation and meaning acquisition through cross-situational learning.” _Psych. Review_ , 2015.
  * [7] R. Eloff, H. A. Engelbrecht, and H. Kamper, “Multimodal one-shot learning of speech and images,” in _Proc. ICASSP_ , 2019.
  * [8] L. Nortje and H. Kamper, “Unsupervised vs. transfer learning for multimodal one-shot matching of speech and images,” in _Proc. Interspeech_ , 2020\.
  * [9] ——, “Direct multimodal few-shot learning of speech and images,” in _Proc. Interspeech_ , 2021.
  * [10] D. Harwath, A. Recasens, D. Suris, G. Chuang, A. Torralba, and J. Glass, “Jointly discovering visual objects and spoken words from raw sensory input,” in _Proc. ECCV_ , 2018.
  * [11] H. Kamper, A. Anastassiou, and K. Livescu, “Semantic query-by-example speech search using visual grounding,” in _Proc. ICASSP_ , 2019.
  * [12] K. Olaleye and H. Kamper, “Attention-based keyword localisation in speech using visual grounding,” in _Proc. Interspeech_ , 2021.
  * [13] G. Chrupała, “Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques,” _J. Artif. Intell. Res._ , 2022.
  * [14] D. Merkx, S. Scholten, S. L. Frank, M. Ernestus, and O. Scharenborg, “Modelling human word learning and recognition using visually grounded speech,” _Cogn. Comput._ , 2022.
  * [15] P. Peng, S.-W. Li, O. Räsänen, A. Mohamed, and D. Harwath, “Syllable discovery and cross-lingual generalization in a visually grounded, self-supervised speech mode,” in _Proc. Interspeech_ , 2023.
  * [16] L. Berry, Y.-J. Shih, H.-F. Wang, H.-J. Chang, H.-y. Lee, and D. Harwath, “M-SpeechCLIP: Leveraging large-scale, pre-trained models for multilingual speech to image retrieval,” in _Proc. ICASSP_ , 2023.
  * [17] T. Miller and D. Harwath, “Exploring few-shot fine-tuning strategies for models of visually grounded speech,” in _Proc. Interspeech_ , 2022.
  * [18] L. Nortje and H. Kamper, “Towards visually prompted keyword localisation for zero-resource spoken languages,” in _Proc. SLT_ , 2022.
  * [19] W.-N. Hsu, D. Harwath, C. Song, and J. Glass, “Text-free image-to-speech synthesis using learned segmental units,” in _Proc ACL_ , 2021.
  * [20] K. Olaleye, D. Oneata, and H. Kamper, “YFACC: A Yorùbá speech-image dataset for cross-lingual keyword localisation through visual grounding,” in _Proc. SLT_ , 2023.
  * [21] L. Nortje, B. van Niekerk, and H. Kamper, “Visually grounded few-shot word acquisition with fewer shots,” in _Proc. Interspeech_ , 2023.
  * [22] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proc. CVPR_ , 2016.
  * [23] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” _ACM_ , 2017.
  * [24] H. Kamper, “Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder models,” in _Proc. ICASSP_ , 2019.
  * [25] Y.-A. Chung, C.-C. Wu, C.-H. Shen, and H.-y. Lee, “Unsupervised learning of audio segment representations using sequence-to-sequence recurrent neural networks,” in _Proc. Interspeech_ , 2016.
  * [26] Y.-H. Wang, H.-y. Lee, and L.-s. Lee, “Segmental audio Word2Vec: Representing utterances as sequences of vectors with applications in spoken term detection,” in _Proc. ICASSP_ , 2018.
  * [27] N. Holzenberger, M. Du, J. Karadayi, R. Riad, and E. Dupoux, “Learning word embeddings: Unsupervised methods for fixed-size representations of variable-length speech segments,” in _Proc. Interspeech_ , 2018.
  * [28] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” _ACM_ , 2021.
  * [29] H. Kamper and B. van Niekerk, “Towards unsupervised phone and word segmentation using self-supervised vector-quantized neural networks,” in _Proc. Interspeech_ , 2021.
  * [30] S. B. Needleman and C. D. Wunsch, “A general method applicable to the search for similarities in the amino acid sequence of two proteins,” _J. Mol. Biol._ , 1970.
  * [31] T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollár, “Microsoft COCO: Common objects in context,” in _Proc. ECCV_ , 2014.
  * [32] M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, “Montreal forced aligner: Trainable text-speech alignment using Kaldi,” in _Proc. Interspeech_ , 2017.
  * [33] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in _Proc. CVPR_ , 2009.
  * [34] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in _Proc. ICASSP_ , 2015.
  * [35] D. Harwath, G. Chuang, and J. Glass, “Vision as an interlingua: Learning multilingual semantic embeddings of untranscribed speech,” in _Proc. ICASSP_ , 2018.
  * [36] B. van Niekerk, L. Nortje, and H. Kamper, “Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challenge,” in _Proc. Interspeech_ , 2020.
  * [37] D. Harwath, A. Torralba, and J. Glass, “Unsupervised learning of spoken language with visual context,” in _Proc. NeurIPS_ , 2016.
  * [38] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _Proc. ICLR_ , 2015.
  * [39] C. Rashtchian, P. Young, M. Hodosh, and J. Hockenmaier, “Collecting Image Annotations Using Amazon’s Mechanical Turk,” in _NAACL HLT Workshop_ , 2010.
  * [40] M. Hodosh, P. Young, and J. Hockenmaier, “Framing image description as a ranking task: Data, models and evaluation metrics,” _JAIR_ , 2013.
  * [41] H. Kamper, G. Shakhnarovich, and K. Livescu, “Semantic speech retrieval with a visually grounded model of untranscribed speech,” _IEEE/ACM Trans. Audio Speech Lang. Process._ , 2019.
  * [42] D. Harwath and J. Glass, “Deep multimodal semantic embeddings for speech and images,” in _Proc. ASRU_ , 2015.
  * [43] A. C. Berg, T. L. Berg, H. Daumé, J. Dodge, A. Goyal, X. Han, A. Mensch, M. Mitchell, A. Sood, K. Stratos, and K. Yamaguchi, “Understanding and predicting importance in images,” in _Proc. CVPR_ , 2012.
  * [44] H. Kamper, Y. Matusevych, and S. Goldwater, “Improved acoustic word embeddings for zero-resource languages using multilingual transfer,” _TASLP_ , 2021\.
  * [45] C. Jacobs, Y. Matusevych, and H. Kamper, “Acoustic word embeddings for zero-resource languages using self-supervised contrastive learning and multilingual adaptation,” in _Proc. SLT_ , 2021.