11affiliationtext: School of Computer Science, Electrical and Electronic
Engineering, and Engineering Maths
University of Bristol, Bristol, BS8 1UB, UK.**affiliationtext:
<EMAIL_ADDRESS>
# Multimodal sensor fusion in the latent representation space

Robert J. Piechocki  Xiaoyang Wang  Mohammud J. Bocus

###### Abstract

A new method for multimodal sensor fusion is introduced. The technique relies
on a two-stage process. In the first stage, a multimodal generative model is
constructed from unlabelled training data. In the second stage, the generative
model serves as a reconstruction prior and the search manifold for the sensor
fusion tasks. The method also handles cases where observations are accessed
only via subsampling i.e. compressed sensing. We demonstrate the effectiveness
and excellent performance on a range of multimodal fusion experiments such as
multisensory classification, denoising, and recovery from subsampled
observations.

## Introduction

_Controlled hallucination_[1] is an evocative term referring to the Bayesian
brain hypothesis [2]. It posits that perception is not merely a function of
sensory information processing capturing the world as is. Instead, the brain
is a predictive machine - it attempts to infer the causes of sensory inputs.
To achieve this, the brain builds and continually refines its world model. The
world model serves as a prior and when combined with the sensory signals will
produce the best guess for its causes. Hallucination (uncontrolled) occurs
when the sensory inputs cannot be reconciled with, or contradict the prior
world model. This might occur in our model, and when it does, it manifests
itself at the fusion stage with the stochastic gradient descent procedure
getting trapped in a local minimum. The method presented in this paper is
somewhat inspired by the Bayesian brain hypothesis, but it also builds upon
multimodal generative modelling and deep compressed sensing.

Multimodal data fusion attracts academic and industrial interests alike [3]
and plays a vital role in several applications. Automated driving is arguably
the most challenging industrial domain [4]. Automated vehicles use a plethora
of sensors: Lidar, mmWave radar, video and ultrasonic, and attempt to perform
some form of sensor fusion for environmental perception and precise
localization. A high-quality of final fusion estimate is a prerequisite for
safe driving. Amongst other application areas, a notable mention deserves
eHealth and Ambient Assisted Living (AAL). These new paradigms are contingent
on gathering information from various sensors around the home to monitor and
track the movement signatures of people. The aim is to build long-term
behavioral sensing machine which also affords privacy. Such platforms rely on
an array of environmental and wearable sensors, with sensor fusion being one
of the key challenges.

In this contribution, we focus on a one-time snapshot problem (i.e. we are not
building temporal structures). However, we try to explore the problem of
multimodal sensor fusion from a new perspective, essentially, from a Bayesian
viewpoint.The concept is depicted in Fig. 1, alongside two main groups of
approaches to sensor fusion. Traditionally, sensor fusion for classification
tasks has been performed at the decision level as in Fig. 1(a). Assuming that
conditional independence holds, a pointwise product of final pmf (probability
mass function) across all modalities is taken. Feature fusion, as depicted in
Fig. 1(b), has become very popular with the advent of deep neural networks
[3], and can produce very good results. Fig. 1(c) shows our technique during
the fusion stage (Stage 2). Blue arrows indicate the direction of
backpropagation gradient flow during fusion.

Figure 1: Multimodal Sensor Fusion: (a) Decision fusion, (b) Feature fusion,
(c) Our technique: fusion in the latent representation with optional
compressed sensing measurements; $F$ features, $p(z)$ prior model, $\bf{G}$
generators, $X$ complete data, $Y$ subsampled data. For clarity $M=2$
modalities are shown, the concept generalises to any $M$.

Contributions:

  * •

A novel method for multimodal sensor fusion is presented. The method attempts
to find the best estimate (_maximum a posteriori_) for the causes of observed
data. The estimate is then used to perform specific downstream fusion tasks.

  * •

The method can fuse the modalities under lossy data conditions i.e. when the
data is subsampled, lost and/or noisy. Such phenomena occur in real-world
situations such as the transmission of information wirelessly, or intentional
subsampling to expedite the measurement (rapid MRI imaging and radar) etc.

  * •

It can leverage between modalities. A strong modality can be used to aid the
recovery of another modality that is lossy or less informative (weak
modality). This is referred to as asymmetric Compressed Sensing.

## Related Work

In this section, we review the state-of-the-art in three areas directly
relevant to our contribution: multimodal generative modeling, sensor fusion,
and compressed sensing. One of the main aims of Multimodal Variational
Autoencoders (MVAEs) is to learn shared representation across different data
types in a fully self-supervised manner, thus avoiding the need to label a
huge amount of data, which is time-consuming and expensive [5]. It is indeed a
challenge to infer the low-dimensional joint representation from multiple
modalities, which can ultimately be used in downstream tasks such as self-
supervised clustering or classification. This is because the modalities may
vastly differ in characteristics, including dimensionality, data distribution,
and sparsity [6]. Recently, several methods have been proposed to combine
multimodal data using generative models such as Variational Autoencoders
(VAEs) [7, 8, 5, 9, 10, 11]. These methods aim to learn a joint distribution
in the latent space via inference networks and try to reconstruct modality-
specific data, even when one modality is missing. In these works, a modality
can refer to natural images, text, captions, labels or visual and non-visual
attributes of a person. JMVAE (Joint Multimodal Variational Autoencoder) [9]
makes use of a joint inference network to learn the interaction between two
modalities and they address the issue of missing modality by training an
individual (unimodal) inference network for each modality as well as a bimodal
inference network to learn the joint posterior, based on the product-of-
experts (PoE). They consequently minimize the distance between unimodal and
multimodal latent distribution. On the other hand, MVAE [7], which is also
based on PoE, considers only a partial combination of observed modalities,
thereby reducing the number of parameters and improving the computational
efficiency. Reference [8] uses the Mixture-of-Experts (MoE) approach to learn
the shared representation across multiple modalities. The latter two models
essentially differ in their choices of joint posterior approximation
functions. MoPoE (Mixture-of-Products-of-Experts)-VAE [5] aims to combine the
advantages of both approaches, MoE and PoE, without incurring significant
trade-offs. DMVAE (Disentangled Multimodal VAE) [10] uses a disentangled VAE
approach to split up the private and shared (using PoE) latent spaces of
multiple modalities, where the latent factor may be of both continuous and
discrete nature. CADA (Cross- and Distribution Aligned)-VAE [11] uses a cross-
modal embedding framework to learn a latent representation from image features
and classes (labels) using aligned VAEs optimized with cross- and
distribution- alignment objectives.

In terms of multimodal/sensor fusion for human activity sensing using Radio-
Frequency (RF), inertial and/or vision sensors, most works have considered
either decision-level fusion or feature-level fusion. For instance, the work
in [12] performs multimodal fusion at the decision level to combine the
benefits of WiFi and vision-based sensors using a hybrid deep neural network
(DNN) model to achieve good activity recognition accuracy for 3 activities.
The model essentially consists of a WiFi sensing module (dedicated
Convolutional Neural Network (CNN) architecture) and a vision sensing module
(based on the Convolutional 3D model) for processing WiFi and video frames for
unimodal inference, followed by a multimodal fusion module. Multimodal fusion
is performed at the decision level (after both WiFi and vision modules have
made a classification) because this framework is stated to be more flexible
and robust to unimodal failure compared to feature level fusion. Reference
[13] presents a method for activity recognition, which leverages four sensor
modalities, namely, skeleton sequences, inertial and motion capture
measurements and WiFi fingerprints. The fusion of signals is formulated as a
matrix concatenation. The individual signals of different sensor modalities
are transformed and represented as an image. The resulting images are then fed
to a two-dimensional CNN (EfficientNet B2) for classification. The authors of
[14] proposed a multimodal HAR system that leverages WiFi and wearable sensor
modalities to jointly infer human activities. They collect Channel Sate
Information (CSI) data from a standard WiFi Network Interface Card (NIC),
alongside the user’s local body movements via a wearable Inertial Measurement
Unit (IMU) consisting of an accelerometer, gyroscope, and magnetometer
sensors. They compute the time-variant Mean Doppler Shift (MDS) from the
processed CSI data and magnitude from the inertial data for each sensor of the
IMU. Then, various time and frequency domain features are separately extracted
from the magnitude data and the MDS. The authors apply a feature-level fusion
method which sequentially concatenates feature vectors that belong to the same
activity sample. Finally supervised machine learning techniques are used to
classify four activities, such as walking, falling, sitting, and picking up an
object from the floor.

Compared to the aforementioned works [12, 13, 14] which consider supervised
models with feature-level fusion or decision-level fusion, our technique, in
contrast, performs multimodal sensor fusion in the latent representation space
leveraging a self-supervised generative model. Our method is different from
current multimodal generative models such as those proposed in [7, 8, 5, 9] in
the sense that it can handle cases where observations are accessed only via
subsampling (i.e. compressed sensing with significant loss of data and no data
imputation). And crucially our technique attempts to directly compute the MAP
(_maximum a posteriori_) estimate.

The presented method is related to and builds upon Deep Compressed Sensing
(DCS) techniques[15, 16]. DCS, in turn, is inspired by Compressed Sensing (CS)
[17, 18]. In CS, we attempt to solve what appears to be an underdetermined
linear system, yet the solution is possible with the additional prior sparsity
constraint on the signal: $\min L0$. Since $L0$ is non-convex, $L1$ is used
instead to provide a convex relaxation, which also promotes sparsity and
allows for computationally efficient solvers. DCS, in essence, replaces the
$L0$ prior with a low dimensional manifold, which is learnable from the data
using generative models. Concurrently to DCS, Deep Image Prior [19] was
proposed. It used un-trained CNNs to solve a range of inverse problems in
computer vision (image inpainting, super-resolution, denoising).

## Methods

Assume the data generative process so that latent and common cause $Z$ gives
rise to $X_{m}$, which in turn produces observed $Y_{m}$, i.e. $Z\rightarrow
X_{m}\rightarrow Y_{m}$ forms a Markov chain. Here, $X_{m}$ is the full data
pertaining to $m^{th}$ modality, $m\in\\{1,\dots,M\\}$. Crucially, the
modalities collect data simultaneously “observing” the same scene. As an
example, in this work, we consider the different views (obtained via multiple
receivers) from the opportunistic CSI WiFi radar as different modalities. The
variable $Z$ encodes the semantic content of the scene and is typically of
central interest. Furthermore, $X_{m}$ is not accessed directly, but is
observed via a _subsampled_ $Y_{m}$. This is a compressed sensing setup:
$Y_{m}=\chi_{m}(X_{m})$: $\chi_{m}$ is a deterministic and known (typically
many-to-one) function. The only condition we impose on $\chi_{m}$ is to be
Lipschitz continuous. With the above, the conditional independence between
modalities holds (conditioned on $Z$). Therefore, the joint density factors
as:

$p\left(z,x_{1:M},y_{1:M}\right)=p\left(z\right)\prod_{m=1}^{M}{p(y_{m}|x_{m})p(x_{m}|z)}.$
(1)

The main task in this context is to produce the best guess for latent $Z$, and
possibly, to recover the full signal(s) $X_{m}$, given subsampled data
$Y_{1:M}$. We approach the problem in two stages. First we build a joint model
which approximates equation (1), and will be instantiated as a Multimodal
Variatational Autoencoer (M-VAE). More specifically, the M-VAE will provide an
approximation to $p_{\phi_{1:M},\psi_{1:M}}(z,x_{1:M})$, parameterized by deep
neural networks $\\{\phi_{1},\dots,\phi_{M}\\}$,
$\\{\psi_{1},\dots,\psi_{M}\\}$, referred to as _encoders_ and _decoders_ ,
respectively. The trained M-VAE will then be appended with
$p_{\chi_{m}}(y_{m}|x_{m})$ for each modality $m$:
$\\{\chi_{1},\dots,\chi_{M}\\}$ referred to as _samplers_. In the second
stage, we use the trained M-VAE and $\chi_{1:M}$ to facilitate the fusion and
reconstruction tasks. Specifically, our sensor fusion problem amounts to
finding the maximum a posteriori (MAP) $\hat{z}_{MAP}$ estimate of the latent
cause for a given ($i^{th}$) data point $Y_{1:M}=y^{(i)}_{1:M}$:

$\hat{z}_{MAP}=\arg\max_{z}p\left(z|Y_{1:M}=y^{(i)}_{1:M}\right),$ (2)

where,

$p\left(z|Y_{1:M}=y^{(i)}_{1:M}\right)\propto
p\left(z\right)\prod_{m=1}^{M}{\int_{X_{m}}p(Y_{m}=y^{(i)}_{m}|x_{m})p(x_{m}|z)\,dx_{m}}.$
(3)

The above MAP estimation problem is hard, and we will resort to approximations
detailed in the sections below.

### Multimodal VAE

The first task is to build a model of equation (1). As aforementioned, this
will be accomplished in two steps. Firstly, during the training stage we
assume access to full data $X_{1:M}$, therefore training an approximation to
$p_{\phi_{1:M},\psi_{1:M}}(z,x_{1:M})$ is a feasible task. The marginal data
log-likelihood for the multimodal case is:

$\displaystyle\ \log p(x_{1:M})$
$\displaystyle=D_{KL}(q(z|x_{1:M}||p(z|x_{1:M}))$ (4)
$\displaystyle+\left[\sum_{X_{m}}{\mathbb{E}_{z\sim q(z|x_{1:M})}\log
p(x_{m}|z)}-\mathbb{E}_{z\sim
q(z|x_{1:M})}\log\frac{q(z|x_{1:M})}{p(z)}\right],$ (5)

where $D_{KL}$ is the Kullback–Leibler (KL) divergence. The first summand in
equation (5), i.e. the sum over modalities follows directly from the
conditional independence. And since KL is non-negative, equation (5)
represents the lower bound (also known as Evidence Lower Bound - ELBO) on the
log probability of the data (and its negative is used as the loss for the
M-VAE). There exist a body of work on M-VAEs, the interested reader is
referred to [7, 8, 5, 9] for details and derivation. The key challenge in
training M-VAEs is the construction of variational posterior $q(z|x_{1:M})$.
We dedicate a section in the Supplementary Information document S1 to the
discussion on choices and implications for the approximation of variational
posterior. Briefly, we consider two main cases: a missing data case – i.e.
where particular modality data might be missing
($X_{m}=x_{m}^{(i)}=\emptyset$); and the full data case. The latter is
straightforward and is tackled by enforcing a particular structure of the
encoders. For the former case variational Product-of-Experts (PoE) is used:

$q_{\Phi}(z|x_{1:M})=p(z)\prod_{m=1}^{M}{q_{\phi_{m}}(z|x_{m})}.$ (6)

Should the data be missing for any particular modality,
$q_{\phi_{m}}(z|x_{m})=1$ is assumed. Derivation of equation (6) can be found
in the Supplementary Information document S1.

### Fusion on the M-VAE prior

Recall the sensor fusion problem as stated in equation (2). The $p(z)$ is
forced to be isotropic Gaussian by M-VAE, and the remaining densities are
assumed to be Gaussian. Furthermore, we assume that
$p(x_{m}|z)=\delta(x_{m}-\psi_{m}(z))$. Therefore equation (2) becomes:

$\hat{z}_{MAP}=\arg\max_{z}p\left(z|Y_{1:M}=y^{(i)}_{1:M}\right)\propto\exp{(-\|z\|^{2})}\prod_{m=1}^{M}{\exp{(-\frac{1}{2\sigma_{m}^{2}}\|y_{m}^{(i)}-\chi_{m}(\psi_{m}(z))\|^{2})}}.$
(7)

Hence, the objective to minimize becomes:

$\mathcal{L}(z)=\lambda_{0}{\|z\|^{2}}+\sum_{m=1}^{M}{\lambda_{m}\|y_{m}^{(i)}-\chi_{m}(\psi_{m}(z))\|^{2}}.$
(8)

Recall that the output of the first stage is $p(z)$ and the decoders
$\prod_{m}{p_{\psi_{n}}(x|z)}$ are parametrized by $\\{\psi_{1:M}\\}$,
$\\{\lambda_{0:M}\\}$ are constants. The MAP estimation procedure consists of
backpropagating through the sampler $\chi_{m}$ and decoder $\psi_{m}$ using
Stochastic Gradient Descent (SGD). In this step $\\{\psi_{1:M}\\}$ are non-
learnable, i.e. jointly with $\chi_{m}$ are some non-linear known (but
differentiable) functions.

$z\leftarrow
z-\eta_{0}\nabla_{z}({{\|z\|^{2}}})-\sum_{m=1}^{M}{\eta_{m}\nabla_{z}(\|y_{m}^{(i)}-\chi_{m}(\psi_{m}(z))\|^{2})}.$
(9)

The iterative fusion procedure is initialized by taking a sample from the
prior $z^{0}\sim p(z)$, $\\{\eta_{0:M}\\}$ are learning rates. One or several
SGD steps are taken for each modality in turn. The procedure terminates with
convergence - see Algorithm 1. In general, the optimization problem as set out
in equation (8) is non-convex. Therefore, there are no guarantees of
convergence to the optimal point ($\hat{z}_{MAP}$). We deploy several
strategies to minimize the risk of getting stuck in a local minimum. We
consider multiple initialization points (a number of points sampled from the
prior with Stage 2 replicated for all points). In some cases it might be
possible to sample from: $z^{0}\sim p\left(z\right)\prod
p\left(z\left|X=\check{x}_{m}^{(j)}\right.\right)$. Depending on modality,
this might be possible with data imputation ($\check{x}_{m}$ are imputed
data). The final stage will depend on a particular task (multisensory
classification/reconstruction), but in all cases it will take $\hat{z}_{MAP}$
as an input. In our experiments, we observe that the success of Stage 2 is
crucially dependent on the quality of M-VAE.

Algorithm 1 Multimodal Sensor Fusion in the Latent Representation Space (SFLR)

1:Training data: $\mathcal{D_{T}}\equiv\\{X_{1:M}^{(1:I)}\\}$, Test data
$\mathcal{D_{P}}\equiv\\{X_{1:M}^{(1:J)}\\}$, Samplers $\\{\chi_{1:M}\\}$

2:Stage 1: Train M-VAE using $\mathcal{D_{T}}$

3:Output: $p(z)$, Encoders $\\{\phi_{1:M}\\}$, Decoders $\\{\psi_{1:M}\\}$

4:Stage 2: Fusion

5:$y_{1:M}^{(i)}\sim\mathcal{D_{P}}$

6:Sample the initial point $z^{0}\sim p(z)$

7:while not converged do

8: $z\leftarrow
z-\eta_{0}\nabla_{z}({{\|z\|^{2}}})-{\eta_{1}\nabla_{z}(\|y_{1}^{(i)}-\chi_{1}(\psi_{1}(z))\|^{2})}$

9: $z\leftarrow
z-\eta_{0}\nabla_{z}({{\|z\|^{2}}})-{\eta_{2}\nabla_{z}(\|y_{2}^{(i)}-\chi_{2}(\psi_{2}(z))\|^{2})}$

10: $\vdots$

11: $z\leftarrow
z-\eta_{0}\nabla_{z}({{\|z\|^{2}}})-{\eta_{M}\nabla_{z}(\|y_{M}^{(i)}-\chi_{M}(\psi_{M}(z))\|^{2})}$

12:end while

13:$\hat{z}_{MAP}\leftarrow z$

14:Downstream tasks: $\hat{x}_{m}=\psi_{m}(\hat{z}_{MAP})$, classification
tasks $K$-NN$(\hat{z}_{MAP})$

## Experiments

In this work, we investigate the performance of the proposed method on two
datasets for multimodal sensor fusion and recovery tasks: i) a synthetic “toy
protein” dataset and ii) a passive WiFi radar dataset intended for Human
Activity Recognition (HAR).

### Synthetic toy protein dataset

A synthetic dataset containing two-dimensional (2D) protein-like data samples
with two modalities is generated. The latent distribution
$p(z),z\in\mathds{R}^{4}$ is a Gaussian mixture model with 10 components,
simulating 10 different “classes” for samples. For each modality, the data
generative model $p(x_{m}|z),x_{m}\in\mathds{R}^{N}$ is a one-layer multilayer
perceptron (MLP) with random weights. Here $m=1,2$ represents two modalities.
10,000 pairs of samples are generated using the generative model, with the
protein size $N=32$. Fig. 2(a) shows an instance of the 2D protein data with
$N=64$.

Figure 2: (a) Generated toy proteins examples ($N=64$) and (b) reconstruction
from compressed sensing observations. With 2 out of 64 measurements (3.125%),
near perfect reconstruction is possible even though the modalities are
individually subsampled.

### Passive WiFi radar dataset

We use the OPERAnet [20] dataset which was collected with the aim to evaluate
human activity recognition (HAR) and localization techniques with measurements
obtained from synchronized Radio-Frequency (RF) devices and vision-based
sensors. The RF sensors captured the changes in the wireless signals while six
daily activities were being performed by six participants, namely, sitting
down on a chair ("sit"), standing from the chair ("stand"), laying down on the
floor ("laydown"), standing from the floor ("standff"), upper body rotation
("bodyrotate), and walking ("walk"). We convert the raw time-series CSI data
from the WiFi sensors into the image-like format, namely, spectrograms using
signal processing techniques. More details are available in Section S2 of the
Supplementary Information document. 2,906 spectrogram samples (each of 4s
duration window) were generated for the 6 human activities and 80% of these
were used as training data while the remaining 20% as testing data (random
train-test split).

## Results and Discussion

### Classification results of WiFi CSI spectrograms for HAR

In this section, we evaluate the HAR sensor fusion classification performance
under a few-shot learning scenario, with 1, 5 and 10 labelled examples per
class. These correspond to 0.05%, 0.26% and 0.51% of labelled training
samples, respectively. We randomly select 80% of the samples in the dataset as
the training set and the remaining 20% is used for validation. The average
$F_{1}$-macro scores for the HAR performance are shown in Table 1 for
different models. To allow for a fair comparison, the same random seed was
used in all experiments with only two modalities (processed spectrograms data
obtained from two different receivers).

Prior to training our model (see Supplementary Fig. S1), the spectrograms were
reshaped to typical image dimensions of size $(1\times 224\times 224)$. Our
model was trained for 1,000 epochs using the training data with a fixed KL
scaling factor of $\beta=0.02$. The encoders comprised of the ResNet-18
backbone with the last fully-connected layer dimension having a value of 512.
For the decoders, corresponding CNN deconvolutional layers were used to
reconstruct the spectrograms from each modality with the same input dimension.
The latent dimension, batch size, and learning rate are set at 64, 64, and
0.001, respectively. In the second stage, the generative model serves as a
reconstruction prior and the search manifold for the sensor fusion tasks.
Essentially, in this stage, we obtain the maximum a posteriori estimate of
$\hat{z}_{MAP}$ through the process described in Algorithm 1. The final
estimate of the class is produced by $K$-NN in the latent representation
space, with labelled examples sampled from the training set.

To benchmark our technique we investigate the performance of other state-of-
the-art sensor fusion techniques. The feature-fusion is represented by CNN
models (1-channel CNN, 2-channel CNN, dual-branch CNN). All are trained in a
conventional supervised fashion from scratch using the ResNet-18 backbone and
a linear classification head is appended on top of it consisting of a hidden
linear layer of 128 units and a linear output layer of 6 nodes (for
classifying 6 human activities). The dual-input CNN refers to the case where
the embeddings from the two modalities’ CNNs are concatenated, and a
classification head is then added (as illustrated in Fig. 1(b)). The
“Probability Fusion” (decision fusion) model refers to a score-level fusion
method where the classification probabilities ($P_{1}$ and $P_{2}$) from each
modality are computed independently (using an output SoftMax layer) and then
combined using the product rule (this is optimal given conditional
independence). These models are fine-tuned with labelled samples over 200
epochs, with a batch size of 64 and the Adam optimizer was used with learning
rate of 0.0001, weight decay of 0.001 and $\beta_{1}=0.95$, $\beta_{2}=0.999$.

It can be observed from Table 1 that our method significantly outperforms all
other conventional feature and decision fusion methods. The confusion matrix
for HAR classification using our SFLR (Sensor Fusion in the Latent
Representation space) model is shown in Fig. S8 in the Supplementary
Information document for the case when only ten labelled examples are used at
the (classification) fusion stage.

Figure 3: Illustration of spectrogram recovery (for sitting down activity)
using compressed sensing with measurements as low as 784 out of 50,176
(1.56%). No additive white Gaussian noise is considered. The left column shows
the true spectrogram sample, the middle column shows reconstruction with an
initial guess (no optimization) while the right column shows reconstruction
with $\hat{Z}_{MAP}$.

### Sensor fusion from subsampled observations

Next, we evaluate the recovery performance of spectrograms under different
numbers of compressed sensing measurements. The measurement function
$\chi_{m}$ is a matrix initialized randomly and we assume that there is no
additive Gaussian noise. The Adam optimizer is used to optimize
$\hat{z}_{MAP}$ with a learning rate of 0.01. The algorithm is run for 10,000
iterations. After the loss in equation (8) has converged during the
optimization process, the samples are decoded/recovered for modality 1 and
modality 2 using their respective decoders
$\hat{x}_{m}=\psi_{m}(\hat{z}_{MAP})$. Table 2 shows the compressed sensing
results when a batch of 50 images is taken from the testing dataset and
evaluated under different number of measurements (without noise). It can be
observed that the samples can be recovered with very low reconstruction error
when the number of measurements is as low as 196 (0.39%). An illustration is
also shown in Fig. 3 where very good reconstruction is observed for the case
when the number of measurements is equal to 784. More illustrations are shown
in Fig. S7 in the Supplementary Information document, with further
experimental results in Sections S4, S5, S6.

### Toy protein classification

Similarly to the experiments on the OPERAnet dataset, we perform two tasks,
classification and sensor fusion from compressed sensing observations, on the
synthetic toy protein dataset.

As mentioned previously, the toy protein dataset contains 10 classes. The
dataset is split into a training set and a test set, containing 80% and 20% of
samples, respectively. We evaluate the classification performance under a few-
shot learning setting, using 1, 5 or 10 labelled samples per class. The few-
shot classification via the SFLR model consists of two stages. In the first
stage, the M-VAE is trained in an unsupervised manner using the training set.
Using the maximum a posterior $\hat{z}_{MAP}$ and a few labels, the $K$-NN
classifier is applied to the latent representation space. Here the encoder and
decoder in M-VAE are two-layer MLPs, with 16 neurons in the hidden layer.

We compare the SFLR method with 4 baseline models. The single modality model
only considers one modality without sensor fusion. The probability fusion
model independently computes the classification probability for each modality,
which is a representative model for decision-fusion (Fig. 1(a)). The dual-
branch feature fusion model concatenates the embedding of two modalities
before the classification layer, which is a feature fusion method (Fig. 1(b)).
All baseline models are trained in a supervised manner, with the same neural
network structure as the encoder. Table 3 shows the $F1$-macro scores for
different methods on the test set. On the 10-class protein dataset, SFLR
outperforms other sensor fusion models using limited labelled samples.

### Sensor fusion from subsampled toy proteins

Another advantage of the proposed SFLR model is that it can fuse modalities in
subsampled cases. We use a set of samplers $\chi_{1:M}$ to simulate the
subsampled observations. The measurement function $\chi_{m}$ is a matrix
initialized randomly. Here we use 10 initialization points to reduce the risk
of getting trapped in a local minimum (points sampled from the prior with
Stage 2 replicated for all of them). Fig. 2(b) shows the recovered protein
from subsampled observations, with only 2 measurements for each modality. Both
modalities are successfully recovered from the latent representation space,
even though the initial guess $z^{0}$ is far from the true modality. Note that
the proteins in Fig. 2 have a higher dimension than in the dataset, showing
the robustness of the SFLR method. Table 4 shows the average reconstruction
error of the synthetic protein dataset using different subsamplers. The
reconstruction error reduced significantly when having 2 measurements for each
modality, showing superior sensor fusion abilities.

The Supplementary Information document (see Section S7) contains additional
experiments, including tasks showcasing the ability to leverage between
modalities, where a strong modality can be used to aid the recovery of a weak
modality. It also presents the performance under subsampled and noisy
conditions.

## Conclusions and Broader Impacts

The paper presents a new method for sensor fusion. Specifically, we
demonstrate the effectiveness of classification and reconstruction tasks from
radar signals. The intended application area is human activity recognition,
which serves a vital role in the E-Health paradigm. New healthcare
technologies are the key ingredient to battling spiralling costs of
provisioning health services that beset a vast majority of countries. Such
technologies in a residential setting are seen as a key requirement in
empowering patients and imbuing a greater responsibility for own health
outcomes. However, we acknowledge that radar and sensor technologies also find
applications in a military context. Modern warfare technologies (principally
defensive) could potentially become more apt if they were to benefit from
much-improved sensor fusion. We firmly believe that, on balance, it is of
benefit to the society to continue the research in this area in the public
eye.

## Data availability

The raw dataset used in the Passive WiFi radar experiments is available from
[20]. The toy protein dataset is not publicly available at this time but can
be made available from the authors upon reasonable request.

## References

  * [1] Seth, A.  _Being You: A New Science of Consciousness (The Sunday Times Bestseller)_ (Faber & Faber, 2021).
  * [2] Doya, K., Ishii, S., Pouget, A. & Rao, R. P. N.  _Bayesian Brain: Probabilistic Approaches to Neural Coding_ (The MIT Press, 2007).
  * [3] Gao, J., Li, P., Chen, Z. & Zhang, J.  A survey on deep learning for multimodal data fusion.  _Neural Computation_ 32, 829–864, DOI: 10.1162/neco_a_01273 (2020).
  * [4] Wang, Z., Wu, Y. & Niu, Q.  Multi-sensor fusion in automated driving: A survey.  _IEEE Access_ 8, 2847–2868, DOI: 10.1109/ACCESS.2019.2962554 (2020).
  * [5] Sutter, T. M., Daunhawer, I. & Vogt, J. E.  Generalized multimodal ELBO.  Preprint at https://arxiv.org/abs/2105.02470 (2021).
  * [6] Minoura, K., Abe, K., Nam, H., Nishikawa, H. & Shimamura, T.  A mixture-of-experts deep generative model for integrated analysis of single-cell multiomics data.  _Cell Reports Methods_ 1, 100071, DOI: https://doi.org/10.1016/j.crmeth.2021.100071 (2021).
  * [7] Wu, M. & Goodman, N.  Multimodal generative models for scalable weakly-supervised learning.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , NIPS’18, 5580–5590 (Curran Associates Inc., Red Hook, NY, USA, 2018).
  * [8] Shi, Y., N, S., Paige, B. & Torr, P.  Variational mixture-of-experts autoencoders for multi-modal deep generative models.  In Wallach, H. _et al._ (eds.) _Advances in Neural Information Processing Systems_ , vol. 32 (Curran Associates, Inc., 2019).
  * [9] Suzuki, M., Nakayama, K. & Matsuo, Y.  Joint multimodal learning with deep generative models.  Preprint at https://arxiv.org/abs/1611.01891 (2016).
  * [10] Lee, M. & Pavlovic, V.  Private-shared disentangled multimodal VAE for learning of latent representations.  In _IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2021, virtual, June 19-25, 2021_ , 1692–1700, DOI: 10.1109/CVPRW53098.2021.00185 (Computer Vision Foundation / IEEE, 2021).
  * [11] Schönfeld, E., Ebrahimi, S., Sinha, S., Darrell, T. & Akata, Z.  Generalized zero- and few-shot learning via aligned variational autoencoders.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 8239–8247, DOI: 10.1109/CVPR.2019.00844 (2019).
  * [12] Zou, H. _et al._ WiFi and vision multimodal learning for accurate and robust device-free human activity recognition.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 426–433, DOI: 10.1109/CVPRW.2019.00056 (2019).
  * [13] Memmesheimer, R., Theisen, N. & Paulus, D.  Gimme signals: Discriminative signal encoding for multimodal activity recognition.  In _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 10394–10401, DOI: 10.1109/IROS45743.2020.9341699 (2020).
  * [14] Muaaz, M., Chelli, A., Abdelgawwad, A. A., Mallofré, A. C. & Pätzold, M.  WiWeHAR: Multimodal human activity recognition using Wi-Fi and wearable sensing modalities.  _IEEE Access_ 8, 164453–164470, DOI: 10.1109/ACCESS.2020.3022287 (2020).
  * [15] Bora, A., Jalal, A., Price, E. & Dimakis, A. G.  Compressed sensing using generative models.  In Precup, D. & Teh, Y. W. (eds.) _Proceedings of the 34th International Conference on Machine Learning_ , vol. 70 of _Proceedings of Machine Learning Research_ , 537–546 (PMLR, 2017).
  * [16] Wu, Y., Rosca, M. & Lillicrap, T.  Deep compressed sensing.  In _Proceedings of the 36th International Conference on Machine Learning_ , vol. 97 (PMLR, 2019).
  * [17] Candès, E. J., Romberg, J. K. & Tao, T.  Stable signal recovery from incomplete and inaccurate measurements.  _Communications on Pure and Applied Mathematics_ 59, 1207–1223, DOI: https://doi.org/10.1002/cpa.20124 (2006).
  * [18] Donoho, D.  Compressed sensing.  _IEEE Transactions on Information Theory_ 52, 1289–1306, DOI: 10.1109/TIT.2006.871582 (2006).
  * [19] Ulyanov, D., Vedaldi, A. & Lempitsky, V.  Deep image prior.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ (2018).
  * [20] Bocus, M. J. _et al._ OPERAnet: A multimodal activity recognition dataset acquired from radio frequency and vision-based sensors.  Preprint at https://arxiv.org/abs/2110.04239 (2021).
  * [21] Xie, Y., Li, Z. & Li, M.  Precise power delay profiling with commodity WiFi.  In _Proceedings of the 21st Annual International Conference on Mobile Computing and Networking_ , MobiCom ’15, 53–64, DOI: 10.1145/2789168.2790124 (ACM, New York, NY, USA, 2015).
  * [22] Halperin, D., Hu, W., Sheth, A. & Wetherall, D.  Tool release: Gathering 802.11n traces with channel state information.  _SIGCOMM Comput. Commun. Rev._ 41, 53, DOI: 10.1145/1925861.1925870 (2011).
  * [23] Bocus, M. J. _et al._ Translation resilient opportunistic WiFi sensing.  In _2020 25th International Conference on Pattern Recognition (ICPR)_ , 5627–5633, DOI: 10.1109/ICPR48806.2021.9412263 (2021).

## Acknowledgements

This work was performed as a part of the OPERA Project, funded by the UK
Engineering and Physical Sciences Research Council (EPSRC), Grant
EP/R018677/1. This work has also been funded in part by the Next-Generation
Converged Digital Infrastructure (NG-CDI) Project, supported by BT and
Engineering and Physical Sciences Research Council (EPSRC), Grant ref.
EP/R004935/1.

## Author contributions statement

All authors, R.P, X.W and M.B, contributed equally to this work. The main
tasks involved conceiving and conducting the experiments, algorithm
implementation, analysis, validation and interpretation of results, and
finally preparing and reviewing the manuscript.

## Additional information

### Competing interests

The authors declare no competing interests.

## Figure legends

  1. 1.

Multimodal Sensor Fusion: (a) Decision fusion, (b) Feature fusion, (c) Our
technique: fusion in the latent representation with optional compressed
sensing measurements; $F$ features, $p(z)$ prior model, $\bf{G}$ generators,
$X$ complete data, $Y$ subsampled data. For clarity $M=2$ modalities are
shown, the concept generalises to any $M$.

  2. 2.

(a) Generated toy proteins examples ($N=64$) and (b) reconstruction from
compressed sensing observations. With 2 out of 64 measurements (3.125%), near
perfect reconstruction is possible even though the modalities are individually
subsampled.

  3. 3.

Illustration of spectrogram recovery (for sitting down activity) using
compressed sensing with measurements as low as 784 out of 50,176 (1.56%). No
additive white Gaussian noise is considered. The left column shows the true
spectrogram sample, the middle column shows reconstruction with an initial
guess (no optimization) while the right column shows reconstruction with
$\hat{Z}_{MAP}$.

Table 1:  Few-shot learning sensor fusion classification results ($F_{1}$ macro) for Human Activity Recognition. Model | 1 example | 5 examples | 10 examples
---|---|---|---
2-channel CNN | 0.427272 | 0.570888 | 0.618501
1-channel CNN (Modality 1) | 0.349084 | 0.451328 | 0.504462
1-channel CNN (Modality 2) | 0.446554 | 0.600084 | 0.605678
Probability fusion (product rule) | 0.440414 | 0.584726 | 0.641922
Dual-branch CNN | 0.508243 | 0.568795 | 0.575914
SFLR (ours) | 0.652699 | 0.718180 | 0.737507
Table 2:  Compressed sensing mean reconstruction error over a batch of 50 WiFi spectrogram data samples (No additive Gaussian noise). An illustration is shown in Fig. 3.  No. of measurements | Modality 1 | Modality 2
---|---|---
1 (0.002%) | 0.03118854 | 0.15024841
10 (0.02%) | 0.00938917 | 0.02824161
196 (0.39%) | 0.00348606 | 0.00613665
784 (1.56%) | 0.00305005 | 0.00505758
1,568 (3.125%) | 0.00284343 | 0.00489433
Table 3:  Few-shot learning sensor fusion classification results ($F_{1}$ macro) for synthetic proteins. Model | 1 example | 5 examples | 10 examples
---|---|---|---
Single modality (Modality 1) | 0.3188 | 0.4342 | 0.5843
Single modality (Modality 2) | 0.3221 | 0.4849 | 0.5555
Probability fusion (product rule) | 0.2256 | 0.3736 | 0.3836
Dual-branch feature fusion | 0.3769 | 0.4973 | 0.5953
SFLR (ours) | 0.4183 | 0.5501 | 0.6120
Table 4: Compressed sensing mean reconstruction error over a batch of 100 protein samples. No. of Measurements | Modality 1 ($10^{-5}$) | Modality 2 ($10^{-5}$)
---|---|---
1 (3.125%) | 4,622.4 | 4,923.5
2 (6.250%) | 22.5 | 27.9
4 (12.500%) | 7.1 | 7.4
8 (25.000%) | 2.3 | 2.7

## Appendix

##  S1 Approximations to variational posterior

Given the objective, the variational joint posterior $q_{\phi}(z|x_{1:M})$ can
be learned by training one single encoder network that takes all modalities
$X_{1:M}$ as input to explicitly parametrize the joint posterior. This is our
baseline model and an example for $M=2$ modalities is given in Fig. S1.
However, this approach requires all modalities to be present at all times,
thus making cross-modal generation difficult. Alternatively, the joint
variational posterior can be modelled using the following approaches:

### Variational Product of Experts

In this section we reproduce the arguments from [7]. The first option is to
approximate the joint variational posterior as a product:

$q_{\phi}(z|x_{1:M})\equiv p(z)\prod_{m=1}^{M}{q_{\phi_{m}}(z|x_{m})}.$ (S10)

In case of a missing expert, we assume: $q_{\phi_{m}}(z|x_{m})=1$.

For a system of $N$ modalities, $2^{N}$ inference networks need to be
specified, $q(z|X)$ for each subset of modalities
$X\subseteq\\{X_{1},X_{2},\dots,X_{M}\\}$. The optimal inference network
$q(z|x_{1},\dots,x_{N})$ would be the true posterior $p(z|x_{1},\dots,x_{N})$.
The conditional independence assumptions in the generative model imply a
relation among joint- and single-modality posteriors [7]:

$\displaystyle p(z|x_{1},\dots,x_{N})$
$\displaystyle=\frac{p(x_{1},\dots,x_{N}|z)p(z)}{p(x_{1},\dots,x_{N})}$
$\displaystyle=\frac{p(z)}{p(x_{1},\dots,x_{N})}\prod_{i=1}^{N}p(x_{i}|z)$
$\displaystyle=\frac{p(z)}{p(x_{1},\dots,x_{N})}\prod_{i=1}^{N}\frac{p(z|x_{i})p(x_{i})}{p(z)}$
$\displaystyle=\frac{\prod_{i=1}^{N}p(z|x_{i})}{\prod_{i=1}^{N-1}p(z)}\cdot\frac{\prod_{i=1}^{N}p(x_{i})}{p(x_{1},\dots,x_{N})}$
$\displaystyle\propto\frac{\prod_{i=1}^{N}p(z|x_{i})}{\prod_{i=1}^{N-1}p(z)}.$
(S11)

If we approximate $p(z|x_{i})$ with $q(z|x_{i})\equiv\tilde{q}(z|x_{i})p(z)$,
where $\tilde{q}(z|x_{i})$ is the underlying inference network, the quotient
term can be omitted [7]:

$\displaystyle p(z|x_{1},\dots,x_{N})$
$\displaystyle\propto\frac{\prod_{i=1}^{N}p(z|x_{i})}{\prod_{i=1}^{N-1}p(z)}$
$\displaystyle\approx\frac{\prod_{i=1}^{N}[\tilde{q}(z|x_{i})p(z)]}{\prod_{i=1}^{N-1}p(z)}$
$\displaystyle=p(z)\prod_{i=1}^{N}\tilde{q}(z|x_{i}).$ (S12)

Equation (S12) implies that we can use a Product-of-Experts (PoE), including a
“prior expert” (e.g., spherical Gaussian), as the approximating distribution
for the joint-posterior. This derivation is easily extended to any subset of
modalities yielding $q(z|X)\propto p(z)\prod_{x_{i}\in X}\tilde{q}(z|x_{i})$.

### Variational Mixture of Experts

$q_{\phi}(z|x_{1:M})\equiv\sum_{m=1}^{M}{\frac{1}{M}q_{\phi_{m}}(z|x_{m})},$
(S13)

where the above assumes an equitable distribution of power among experts. Non-
uniform weights can also be used. Missing expert: $q_{\phi_{m}}(z|x_{m})=0$.

Figure S1: M-VAE for a full data case: Single encoder network takes all
modalities.

##  S2 Signal processing pipelines for passive WiFi radar

A typical scenario for opportunistic passive WiFi Radar is depicted in Figure
S2. This is an extremely challenging problem since the WiFi waveform was not
specifically designed to lend itself to Radio-Frequency (RF) imaging. In
addition, commercial WiFi chipsets have noisy RF chains and tend to suffer
from phase drifts. The WiFi backscatter does contain information about the
dynamic changes in the radio channel which is incapsulated in the Channel
State Information (CSI). Dedicated tools need to be used to extract the CSI
from WiFi network interface cards such as Atheros [21] or Intel 5300 (IWL5300)
[22]. The raw CSI data is obtained as a 3-dimensional (3D) matrix per
transmitted packet, with $n_{t}$$\times$$n_{r}$$\times$$N_{\text{sc}}$ complex
values, where $n_{t}$ is the number of transmit antennas, $n_{r}$ is the
number of receive antennas and $N_{\text{sc}}$ is the number of subcarriers.
Since the raw CSI data is very noisy in nature, the Discrete Wavelet Transform
(DWT) technique can be used to filter out in-band noise and preserve the high
frequency components, thus avoiding the distortion of the signal [23].
Afterwards, median filtering can be used to remove any undesired transients in
the CSI measurements which are not due to human motion. The Intel 5300 chipset
has a 3$\times$3 antenna configuration and only 30 subcarriers are reported by
this chipset. Thus the number of complex values per packet is equal to
$3$$\times$$3$$\times$$30=270$. Considering a packet rate as high as 1.6 kHz,
this results in a significant amount of data that needs to be processed.
Therefore, we also apply Principal Component Analysis (PCA) to reduce the
computational complexity of such high dimensional data. PCA identifies the
time-varying correlations between the CSI streams which are optimally combined
to extract only a few components that represent the variations caused by human
activities. Finally, we convert the resultant data into spectrograms (time-
frequency domain) using Short Time Fourier Transform (STFT), which are similar
to those generated by Doppler radars. The CSI is highly sensitive to the
surrounding environment and signal reflections from the human body result in
different frequencies when performing different activities. The Doppler
spectrogram generated from STFT helps to identify the change of frequencies
over time. The generated spectrograms can be directly fed to CNNs to
automatically identify a set of features, which can ultimately be used in
downstream tasks. The CSI system consisted of two receivers. For more details
on the experimental setup of the data collection, the interested reader is
kindly referred to [20]. Each receiver can be seen as another view of the
human activity being performed in the environment.

Figure S2: Opportunistic Passive WiFi Radar.

##  S3 Latent representation of WiFi spectrogram data

Our base model is shown in Fig. S1. The trained latent space for different
latent dimensions are shown in Fig. S3. The trained latent space in Fig. S3
shows distinct clusters using UMAP (Uniform Manifold Approximation and
Projection) visualization. The model was trained in a self-supervised fashion,
the six clusters representing the six different human activities can be seen.

Figure S3: UMAP projection of trained latent space using our model on real
WiFi CSI spectrogram data: (a) latent dimension=16, (b) latent dimension=64,
and (c) latent dimension=128.

##  S4 Sensor fusion under noisy conditions (WiFi spectrogram data)

In this experiment, we analyze the sensor fusion performance when the data
samples from the test dataset are affected by different amount of additive
Gaussian noise. In this case, the measurement matrices are initialized as
identity matrices with dimensions 50,176$\times$50,176. No noise was injected
to the input data from two modalities during training. The SFLR (Sensor Fusion
in the Latent Representation space) algorithm (see Algorithm 1 in manuscript)
is run for 1,000 iterations and the corresponding results are shown in Fig.
S4, showing one sample in the test dataset. It can be observed that even under
extreme noisy conditions, the noisy samples are denoised efficiently. These
results are further validated in Table S1 where it can be seen that the fusion
error remains essentially constant for different noise standard deviation
values considering a batch of 50 images from the test dataset.

Figure S4: Impact of additive Gaussian noise on sensor fusion from the two modalities: (a) Std Dev=0.05, (b) Std Dev=0.2, (c) Std Dev=0.4, (d) Std Dev=0.6, and (e) Std Dev=0.8. Left column shows noisy spectrogram sample, middle column shows fusion with initial guess (no optimization) while right column shows fusion with $\hat{z}_{MAP}$. Figure S5: Impact of missing pixels on spectrogram recovery from the two modalities (no additive Gaussian noise): (a) missing pixel ratio=0.1, (b) missing pixel ratio=0.2, (c) missing pixel ratio=0.4, (d) missing pixel ratio=0.6, and (e) missing pixel ratio=0.8. Left column shows spectrogram sample with missing pixels, middle column shows reconstruction with initial guess (no optimization) while right column shows reconstruction with $\hat{z}_{MAP}$. Very good recovery performance is observed in all cases. Table S1:  Noisy measurements mean reconstruction error over a batch of 50 WiFi spectrogram data samples (full measurements considered). Noise standard deviation | Modality 1 | Modality 2
---|---|---
0.01 | 0.00347822 | 0.00573636
0.05 | 0.00348499 | 0.00574255
0.20 | 0.00349233 | 0.00569815
0.40 | 0.00354153 | 0.0058348
0.60 | 0.00357407 | 0.00594901
0.80 | 0.00367526 | 0.00608692

##  S5 Sensor fusion performance with missing pixels (WiFi spectrogram data)

In this experiment, we evaluate the fusion performance of the samples under
different ratios of missing pixels. The results are shown in Fig. S5 when a
true data sample is randomly chosen from the test dataset and a randomly
generated (binary) mask is applied to it to simulate different missing pixel
ratios. Each measurement corresponds to an observed pixel. Therefore, in this
case the measurement matrices will be diagonal matrices with their diagonal
entries corresponding to the mask elements (1’s and 0’s). From Fig. S5, it can
be observed that the recovered samples are very close to the true ones, even
when the missing pixel ratio for both modalities is as high as 0.8. In fact,
as shown in Table S2, the reconstruction error remains essentially constant
with increasing missing pixel ratio for both modalities.

Table S2:  Missing pixel mean reconstruction error over a batch of 50 WiFi spectrogram data samples. Illustrations of spectrogram fusion under different missing pixel ratios are shown in Fig. S5. Missing pixel ratio | Modality 1 | Modality 2
---|---|---
0.1 | 0.00351316 | 0.00573596
0.2 | 0.00345235 | 0.00575154
0.4 | 0.00347854 | 0.00575382
0.6 | 0.003477 | 0.00571794
0.8 | 0.00349462 | 0.00579462

##  S6 Sensor fusion from asymmetric Compressed Sensing (WiFi spectrogram
data)

In this experiment, we analyse the reconstruction of the WiFi spectrogram
samples under two different scenarios, where we want to demonstrate the
benefits of having multiple modalities. We are interested in recovery for one
modality that is subsampled (loss of data) and noisy. This can be referred to
as the weak data (or weak modality). Using the SFLR method, we leverage the
second modality data, which has no loss of information or does not suffer from
noise (strong modality), to improve the recovery for the modality of interest
i.e., the weak modality. In the first case, only modality 1 (subsampled and
noisy) is considered in the reconstruction process. In the second case, the
good modality 2 is added in the iterative fusion process to improve the
reconstruction quality of modality 1.

The results are tabulated in Table S3, where additive Gaussian noise with a
standard deviation of 0.1 is considered. The results show the mean
reconstruction errors (over 50 WiFi spectrogram samples) when modality 1 is
subsampled to different extents. We see that reconstruction error has a
general tendency to decrease with increasing number of measurements. It can be
observed that the samples can be recovered with very low reconstruction error
when the number of measurements is as low as 196 (0.39%). Furthermore, from
Table S3, we observe that when only modality 1 is considered in the
reconstruction process, the reconstruction errors are high when the number of
measurements is equal to 1 (0.002%) and 10 (0.02%). However, by leveraging the
good modality 2, the reconstruction quality is greatly improved for the same
number of measurements, demonstrating the clear benefit of having multiple
modalities. An illustration of the reconstruction quality is depicted in Fig.
S6, where it can be observed that the unimodal reconstruction of modality 1 is
far from the true sample. On the other hand, the reconstruction quality of
modality 1 is improved by leveraging the good modality data.

Table S3:  Mean reconstruction error over 50 WiFi spectrogram data samples. Noise standard deviation: $0.1$ | No. of Measurements | Modality 1 | Modality 2
---|---|---|---
Modality 1 with compressed sensing | 1 (0.002%) | 0.0246185 | -
10 (0.02%) | 0.01075371 | -
196 (0.39%) | 0.00258467 | -
784 (1.56%) | 0.00195997 | -
1,568 (3.125%) | 0.00184247 | -
Modality 1 with compressed sensing | 1 (0.002%) | 0.00892453 | 0.00380795
10 (0.02%) | 0.00798366 | 0.00420512
196 (0.39%) | 0.0034269 | 0.00460956
Modality 2 with full information | 784 (1.56%) | 0.0030373 | 0.00466936
1,568 (3.125%) | 0.0028537 | 0.00469946
Figure S6: Reconstruction examples showing the benefit of multimodal system
compared to a unimodal system. Modality 1 is subsampled data with 1 single
measurement while modality 2 has full information (no noise and no loss of
data). Additive Gaussian noise with a standard deviation of 0.1 is considered
in this example: (a) reconstruction with modality 1 only, (b) reconstruction
with both modalities 1 and 2. Left column shows true spectrogram sample,
middle column shows reconstruction with initial guess (no optimization) while
right column shows reconstruction with $\hat{z}_{MAP}$. Adding modality 2
during reconstruction stage helps in the sample recovery of modality 1. Figure
S7: Compressed sensing performance on different number of measurements without
additive Gaussian noise: (a) 1 measurement out of 50,176 (0.002%), (b) 10
measurements out of 50,176 (0.02%), (c) 196 measurements out of 50,176
(0.39%), (d) 784 measurements out of 50,176 (1.56%), and (e) 1,568
measurements out of 50,176 (3.125%). Left column shows true spectrogram
sample, middle column shows reconstruction with initial guess (no
optimization) while right column shows reconstruction with $\hat{z}_{MAP}$.

##  S7 Toy protein dataset: additional results

### Sensor fusion from subsampled and noisy toy proteins

In this section, we present the sensor fusion results for toy protein
reconstruction under subsampled and noisy observations, as an extension to
Section "Sensor fusion from subsampled toy proteins" in the main document.
Table S4 shows the mean reconstruction error of subsampled toy protein
samples, with different levels of additive Gaussian noise. The proposed SFLR
method recovers both modalities from as low as 4 subsampled and noisy
observations.

Table S4: Compressed sensing mean reconstruction error over a batch of 100 protein samples, with different noise levels. Noise standard deviation | No. of Measurements | Modality 1 ($10^{-3}$) | Modality 2 ($10^{-3}$)
---|---|---|---
0.05 | 1 (3.125%) | 52.301 | 55.382
2 (6.250%) | 11.678 | 9.200
4 (12.500%) | 0.834 | 0.715
8 (25.000%) | 0.387 | 0.450
0.1 | 1 (3.125%) | 36.611 | 50.118
2 (6.250%) | 20.267 | 14.638
4 (12.500%) | 3.413 | 2.769
8 (25.000%) | 2.386 | 2.411
0.2 | 1 (3.125%) | 43.466 | 48.271
2 (6.250%) | 17.864 | 19.435
4 (12.500%) | 1.528 | 1.435
8 (25.000%) | 5.005 | 5.063

### Sensor fusion from asymmetric Compressed Sensing of toy proteins

We show the results of sensor fusion from asymmetric compressed sensing,
regarding the third contribution of this paper. We claim that a strong
modality can be used to aid the recovery of another modality that is lossy or
less informative (weak modality). Table S5 shows the recovery results in two
cases. In the first case, the subsampled modality 1 with additive Gaussian
noise is observed and recovered. In the second case, the noise-free modality 2
with full observation is used to help the sensor fusion. We can see that
modality 2 significantly helps with the recovery of modality 1, especially
when the number of observations are relatively small.

Table S5: Mean reconstruction error over 100 samples with asymmetric compressed sensing. Noise standard deviation: $0.1$. | No. of Measurements | Modality 1 | Modality 2
---|---|---|---
Modality 1 with compressed sensing | 1 (3.125%) | 0.0542 | -
2 (6.250%) | 0.0366 | -
4 (12.500%) | 0.0205 | -
8 (25.000%) | 0.0021 | -
Modality 1 with compressed sensing | 1 (3.125%) | 0.0076 | 0.0073
2 (6.250%) | 0.0067 | 0.0062
Modality 2 with full information | 4 (12.500%) | 0.0023 | 0.0024
8 (25.000%) | 0.0033 | 0.0031
Figure S8: Confusion matrix of Human Activity Recognition (HAR) classification
using our SFLR model (with compressed sensing). Ten labelled examples per
class are considered (refer to Table 1 in main manuscript for classification
results in terms of macro $F_{1}$ score).


# Kink propagation in the Artificial Axon

Xinyi Qi  Giovanni Zocchi<EMAIL_ADDRESS>Department of Physics and
Astronomy, University of California - Los Angeles

###### Abstract

The Artificial Axon is a unique synthetic system, based on biomolecular
components, which supports action potentials. Here we consider, theoretically,
the corresponding space extended system, and discuss the occurrence of
solitary waves, or kinks. In contrast to action potentials, stationary kinks
are possible. We point out an analogy with the interface separating two
condensed matter phases, though our kinks are always non-equilibrium,
dissipative structures, even when stationary.

Introduction. The Artificial Axon (AA) is a synthetic structure designed to
support action potentials, thus generating these excitations for the first
time outside the living cell. The system is based on the same microscopic
mechanism as that operating in neurons, the basic components being: a
phospholipid bilayer with embedded voltage gated ion channels, and an ionic
gradient as the energy source. However, while a real axon has at least two ion
channel species and opposite ionic gradients across the cell membrane, the AA
has only one. In the experiments, a current limited voltage clamp (CLVC) takes
the role of a second ionic gradient [1, 2]. The experimental system in [2] is
built around a $\sim 100\,\mu m$ size black lipid membrane. As a dynamical
system for the voltage, it operates in zero space dimensions (similar to the
”space clamp” setup with real axons [3, 4]). That is, each side of the
membrane is basically an equi-potential surface (the name Artificial Axon,
while a misnomer in this respect, is historical [1] and we propose to keep it
for the original and future versions). Inspired by this system, here we
consider - theoretically - the corresponding space extended dynamical system.
We focus on the existence of solitary wave solutions, or propagating kinks (we
will use the two terms interchangeably, to mean a front which propagates
keeping its shape). Kinks appear in many areas of condensed matter physics
[5], from domain walls in magnetic materials [6, 7] to pattern forming
chemical reactions [8]. Our particular nonlinear structures come from a
dissection, so to speak, of the mechanism of action potential generation. We
show the existence of travelling kinks in our system, and study numerically
their characteristics in relation to the control parameters, which are the
command voltage and the conductance of the CLVC. Then we discuss a ”normal
form” for this class of dynamical systems, highlighting the relation with
other kinks separating two condensed matter phases, such as the nematic -
isotropic interface in liquid crystals. The nonlinearities which thus arise
retrace the development of simplified models of the Hodgkin-Huxley axon [9],
such as introduced 60 years ago by Fitzhugh [10] and Nagumo et al [11].
Looking at kinks thus provides a somewhat different perspective on a classic
topic in the study of excitable media.

Results. We consider the AA in one space dimension. The physical system we
have in mind is a $\sim 1\,cm$ long, $\sim 100\,\mu m$ wide supported strip of
lipid bilayer with one species of voltage gated ion channels embedded. The
bilayer might be anchored to the solid surface so as to leave a sub-micron gap
(the ”inside” of the axon) in between. At present, the stability of the
bilayer stands in the way of a practical realization, but this problem is not
unsurmountable. The bilayer acting essentially like the dielectric in a
parallel plates capacitance, the local charge density is related to the
voltage by $(\partial/\partial t)\rho(x,t)=c\,(\partial/\partial t)V(x,t)$
where $c$ and $\rho$ are capacitance and charge per unit length, respectively.
The current inside the axon follows Ohm’s law: $j=-(1/r)(\partial V/\partial
x)$ where $r$ is the resistance per unit length; then charge conservation
leads to the diffusion equation for the potential: $(\partial V(x,t)/\partial
t)-(1/(rc))(\partial^{2}V(x,t)/\partial x^{2})=0$ . In the AA, an ionic
gradient (of $K^{+}$ ions) across the membrane leads to an equilibrium
(Nernst) potential $V_{N}=(T/|e|)\,ln([K^{+}]_{out}/[K^{+}]_{in})$ , but the
system is held off equilibrium by the current injected through a current
limited voltage clamp (CLVC) [1]. The active elements are voltage gated
potassium channels inserted in the membrane: these are molecular pores which,
in the open state, selectively conduct $K^{+}$ ions. The KvAP channel used in
[2, 12] has three functionally distinct states: open, closed, and inactive;
the presence of the inactive state allows the system to generate action
potentials. Here we consider the simpler case of a ”fast” channel with no
inactivation. Then the channels can be described by an equilibrium function
$P_{O}(V)$ which gives the probability that the channel is open if the local
voltage is $V$. Introducing the current sources in the diffusion equation
above one arrives at the following $(1+1)D$ dynamical system:

$\begin{split}\frac{\partial V(x,t)}{\partial
t}-\frac{1}{rc}\frac{\partial^{2}V}{\partial
x^{2}}\,=\,\frac{\chi}{c}P_{O}(V)[V_{N}-V(x,t)]\\\
+\frac{\chi_{c}}{c}[V_{c}-V(x,t)]\end{split}$ (1)

V is the voltage inside the axon (referred to the grounded outside), and we
assume a distributed ”space clamp” for the CLVC (this would be provided by an
electrode along the axon). Eq. (1) is of the general form of a reaction -
diffusion system; these are usually studied in the context of pattern forming
chemical reactions. For us it represents a continuum limit, i.e. we consider a
uniform, distributed channel conductance instead of discrete, point-like ion
channels. This is a mean field approximation which neglects correlations
between nearby channels. The first term on the RHS of (1), when multiplied by
$c$, is the channel current, proportional to the driving force $(V_{N}-V)$ ;
$V_{N}$ is the Nernst potential, $\chi$ the conductance (per unit length) with
channels open (i.e. $\chi=n\chi_{0}$ , $\chi_{0}$ single channel conductance,
$n$ number of channels per unit length). The second term is the current
injected by the clamp; $V_{c}$ is the clamp voltage (which is a control
parameter in the experiments), $\chi_{c}$ the clamp conductance (per unit
length), which is a second control parameter. The function $P_{O}(V)$ is a
Fermi - Dirac distribution:

$P_{O}(V)\,=\,\frac{1}{exp[-q(V-V_{0})/T]+1}$ (2)

where $q$ is an effective (positive) gating charge and $V_{0}$ the midpoint
voltage where $P_{O}(V_{0})=1/2$. To fix ideas, we will use parameters
consistent with the AA in [12] :
$V_{N}=50\,mV$ , $\chi/c=100\,s^{-1}$ , $\chi_{c}/c=5\,s^{-1}$ ,
$(1/rc)=1\,cm^{2}/s$ , $V_{0}=-10\,mV$ , $q/T=0.08\,(mV)^{-1}$. We use
Gaussian units except that we express voltages in $mV$ : this is more
convenient to relate to experimental systems. Also, the temperature in (2) and
elsewhere is in energy units; thus at room temperature $T/|e|\approx
0.025\,mV$ where $e$ is the charge of the electron.

Figure 1: The traveling kink solution $V(x,t)$ for (1), (2). The plot shows
snapshots of the kink at different times; the initial condition ($t=0$) is a
hyperbolic tangent. Parameter values are those given in the text, with a clamp
voltage $V_{c}=-200\,mV$. The dotted horizontal lines show the fixed points
$V_{1}$ and $V_{3}$. Notice that the shape of the kink shifts from the initial
condition at t = 0.0s to a stable shape afterwards.

The possibility of travelling kink solutions of (1) and (2) arises because,
with the clamp at a negative voltage, say $V_{c}=-100\,mV$, there is a fixed
point of (1) (a uniform, time independent solution) with $V(x,t)\approx V_{N}$
and open channels ($P_{O}(V)\approx 1$), namely $V=V_{1}\approx(\chi
V_{N}+\chi_{c}V_{c})/(\chi+\chi_{c})$. A second fixed point is
$V(x,t)=V_{3}\approx V_{c}$ and channels closed ($P_{O}(V)\approx 0$). A
stable kink solution exists, asymptotically connecting these two stable fixed
points (a third fixed point is unstable and will be discussed later). The
essential parameters in (1) are the diffusion constant $D\equiv 1/(rc)$ and
$\chi/c$ ; from these we can form a characteristic length scale
$\Delta=1/\sqrt{r\chi}$ which gives the scale of the width of the kink
solution, and a characteristc velocity $v=D/\Delta=(1/c)\sqrt{\chi/r}$ which
similarly gives the scale for the kink velocity. With the parameters above,
$\Delta\approx 1\,mm$ and $v\approx 10\,cm/s$. Fig. 1 shows snapshots of a
travelling kink obtained by integrating (1) , (2) using the parameters above
and $V_{c}=-200\,mV$. The kink was launched with a hyperbolic tangent initial
condition ($t=0$ trace in Fig. 1); it is found to quickly (on a time scale
$\sim c/\chi$) attain a stable limiting shape and thereafter travel at
constant velocity. The velocity depends on the clamp voltage $V_{c}$, as shown
in Fig. 2. We measure it by tracking the inflection point of the solution
$V(x,t)$. The solitary wave solution exists only for $V_{c}$ within certain
bounds; correspondingly there is a maximum velocity of the kink, while the
minimum velocity is zero, as we show below.

Figure 2: Plot of kink velocity vs clamp voltage. Parameter values are those
given in the text. The velocity is determined by tracking the minimum of the
ﬁrst derivative of $V(x,t)$, which corresponds to the inﬂection point of the
kink-shaped wave front. The left most and right most data points are close to
the values of $V_{c}$ beyond which the kink solution disappears. The graph is
asymmetric with respect to right moving and left moving kinks.

Let us now analyze these solitary wave solutions (see e.g. [5]). Eq. (1) is of
the form:

$\frac{\partial V(x,t)}{\partial t}-\frac{\partial^{2}V}{\partial
x^{2}}\,=\,g(V)$ (3)

where we have changed to non-dimensional variables using
$\Delta=1/\sqrt{r\chi}$ , $\tau=c/\chi$ , $V_{N}$ as the units of length,
time, and potential, respectively. Then,

$\begin{cases}g(V)\,=\,P_{O}(V)[1-V]+\frac{\chi_{c}}{\chi}\left[\frac{V_{c}}{V_{N}}-V\right]\\\
\\\
P_{O}(V)\,=\,\left\\{exp[-\frac{qV_{N}}{T}(V-\frac{V_{0}}{V_{N}})]+1\right\\}^{-1}\end{cases}$
(4)

We look for a travelling wave solution: $V(x,t)=\varphi(x-ut)\\\
=\varphi(z)\,$ , $z\equiv x-ut$ ; then from (3):

$\varphi^{\prime\prime}+u\,\varphi^{\prime}\,=\,-\frac{d}{d\varphi}F(\varphi)$
(5)

where $F$ is the primitive of $g$ , i.e. $g(\varphi)=dF/d\varphi$ . We may
interpret (5) as the equation of motion of a unit mass in a potential energy
$F$ , subject to a frictional force proportional to the velocity. The
dissipation parameter $u$ is the velocity of the kink. In Fig. 3 we plot the
function $F$ obtained from integrating $g$ in (4); the analytic expression,
which involves the poly log function, is readily obtained with Mathematica.

Figure 3: The function $F(\varphi)$ obtained from (4) vs the (dimensional)
membrane voltage, for clamp voltages of $-100\,mV$ and $-200\,mV$. Parameters
are as given in the text. The fixed points $V_{1}$, $V_{2}$, $V_{3}$ shown
refer to the yellow ($V_{C}=-200\,mV$) curve. As $V_{C}$ is decreased below
$-200\,mV$ the global maximum becomes the secondary maximum and vice-versa.
Increasing $V_{c}$ above $-100\,mV$, the secondary maximum eventually
disappears, at which point there is no kink solution.

The kink solution displayed in Fig. 1 corresponds, in terms of (5), to the
particle (of coordinate $\varphi$) starting with zero velocity at the maximum
$\varphi=V_{1}$ and arriving (after an infinite time) at the secondary maximum
$\varphi=V_{3}$ , also with zero velocity. The value of the dissipation
parameter $u$ for which this is possible corresponds to the propagation
velocity of the kink. Different velocities are possible transiently, for
example, a kink initially steeper than the asymptotic shape will initially
travel faster, and slow down as it attains the stable shape and velocity. This
”shaping” of the signal expresses the existence of a stable, unique solitary
wave solution. It motivated the electronic realization of an axon, and the
corresponding influential dynamical system model, by Nagumo et al [11].
Varying the clamp voltage $V_{c}$ modifies the potential $F$ , and the kink
velocity $u$ changes correspondingly, as shown in Fig. 2. For increasing
$V_{c}$ , the difference $F(V_{1})-F(V_{3})$ increases, while the secondary
maximum at $V=V_{3}$ becomes less pronounced (Fig. 3). Correspondingly, the
kink velocity increases. At a critical clamp value $V_{c}\approx-92.8\,mV$ the
secondary maximum disappears (the minimum at $V_{2}$ becomes an inflection
point, then reverses curvature), so no kink solution exists for higher clamp
voltages. Conversely, as $V_{c}$ is decreased, the difference
$F(V_{1})-F(V_{3})$ decreases, goes through zero and becomes negative.
Correspondingly the kink velocity also goes through zero and then reverses
sign. In short, $F(V_{1})-F(V_{3})$ increases monotonically with increasing
$V_{c}$ , as does the kink velocity $u$. There is a maximum positive velocity
and a maximum negative velocity (the two are not the same). There is a
particular clamp voltage ($V_{c}\approx 244.0\,mV$ with our parameters) such
that the kink is stationary ($u=0$). Trivially, for each right-moving kink
there is an identical mirror-image left-moving kink, if one inverts the
boundary conditions at infinity. From Fig. 3 we also see that two more kink
solutions exist, one connecting the maximum at $V_{1}$ with the minimum at
$V_{2}$ (evidently travelling at a faster speed compared to the kink
connecting $V_{1}$ and $V_{3}$), and a third one connecting $V_{3}$ and
$V_{2}$. These solutions are linearly unstable, because the fixed point at
$V_{2}$ is unstable; thus they would not be observed experimentally. However,
they can still be ”observed” numerically, as we see below.
It is interesting to put this problem in a ”normal form”, and see the
connection to other kinks in condensed matter physics. The simplest function
$F$ in (5) which supports a kink solution of (3) has a maximum and a minimum,
i.e. a cubic non-linearity. A kink solution exists connecting the maximum and
the minimum, but it is unstable as the minimum is an unstable fixed point. The
next simplest case is that $F$ has three extrema; assuming a single control
parameter, we may write:

$F(V)\,=\,a\,[2(1-\alpha)V^{2}+\frac{4}{3}\alpha V^{3}-V^{4}]$ (6)

$a>0$ , $\alpha\leq 1$ where we put one stable fixed point at $V_{1}=1$ and
the unstable fixed point (the minimum of $F$) at $V_{2}=0$. The third (stable)
fixed point is at $V_{3}=(\alpha-1)$. This is not the most general form: the
choice $V_{2}=0$ forces $F$ to be an even function at the ”coexistence point”
$\alpha=0$, as we discuss below; however, this choice allows to discuss
unstable kink solutions also. Apart from this difference, this situation
corresponds to (4); the parameter $\alpha$ has the role of $V_{c}/V_{N}$, if
$\chi_{c}/\chi$ is fixed. For $-1<\alpha\leq 1$ a stable kink with
$V(x\rightarrow-\infty)=V_{1}$ and $V(x\rightarrow+\infty)=V_{3}$ exists,
travelling with a speed $u$ which increases monotonically with increasing
$\alpha$. The stationary kink is obtained for $\alpha=0$; for $\alpha>0$ the
kink travels to the right and for $\alpha<0$ to the left. The simplest stable
kink is thus a solution of:

$\frac{\partial V(x,t)}{\partial t}-\frac{\partial^{2}V}{\partial
x^{2}}\,=\,4a\,[(1-\alpha)V+\alpha V^{2}-V^{3}]$ (7)

The cubic nonlinearity is a feature of several reduced parameters models of
nerve excitability, notably Fitzhugh’s ”BVP model” [10], and indeed of the
original Van der Pol relaxation oscillator [13], in appropriate coordinates.
Two further kink solutions of (7) exist, connecting $V_{1}$ and $V_{2}$ , and
$V_{3}$ and $V_{2}$. These are linearly unstable, but they can still be
obtained numerically, with the trick of arranging for the unstable fixed point
to be at $V=0$, as we did in (6). In this way, one can even discuss collisions
between different kinks: the only non-trivial example stemming from (6) is
shown in Fig. 4.

Figure 4: A 3D plot showing the collision of two different kinks. They are
obtained integrating (7) with $a=0.5$ , $\alpha=0.5$, and appropriate initial
conditions. Notice the velocity change after the collision. However, these
kinks are linearly unstable and so would not be observed experimentally.

Namely, the kink connecting $V_{1}$ and $V_{2}$ collides with the kink
connecting $V_{3}$ and $V_{2}$ travelling in the opposite direction, resulting
in the stable kink connecting $V_{1}$ and $V_{3}$ in the final state.
To ricapitulate: the fixed points of (3) are uniform, time-independent
solutions which we might call ”phases”. Two fixed points can be connected by a
kink. The fixed points are zeros of $g$, i.e. extrema of $F$, but the stable
fixed points are maxima of $F$ while the unstable ones are minima. For the
purpose of classifying, $F$ is analogous to minus the free energy of a Landau
theory describing a corresponding phase transition. The stationary kink
($\alpha=0$ in (6)) is the interface separating two coexisting phases. For
$\alpha\neq 0$ , one of the two phases is more stable and grows at the expense
of the other (i.e. the kink moves). However, we must remember that our system
is never in thermodynamic equilibrium. Even when the kink is stationary, there
are macroscopic currents in the system (the clamp current and the channels
current), and detailed balance in violated. The function $F$ derived from (4),
which is shown in Fig. 3 , has the same general form as (minus) the mean field
free energy which describes the nematic - isotropic transition in liquid
crystals [5], or also the liquid - gas transition. For the former, and
following the notation in [5], the free energy $f$ as a function of the order
parameter $S$ is:

$f\,=\,\frac{1}{2}a(T-T^{*})S^{2}-wS^{3}+uS^{4}$ (8)

where $S=P_{2}(cos\theta)$ , $P_{2}$ the Legendre Polynomial of order 2 and
$\theta$ the angle between the molecular axis and the director vector. For
fixed $V_{c}$ , the evolution of $-F$ for varying $\chi_{c}/\chi$ (where $F$
is the primitive of (4)) mirrors the evolution of (8) for varying temperature
$T$. Namely, for small values of $\chi_{c}/\chi$ there is a global minimum at
positive $V$ (i.e. channels essentially open) and a secondary minimum at
negative $V$ (channels essentially closed). Increasing $\chi_{c}/\chi$ one
reaches a coexistence point where $-F$ has the same value at the two minima,
after which the global minimum is at negative $V$ and the secondary minimum at
positive $V$ (Fig. 3), i.e. the stable phase is with channels essentially
closed. As in (8) there are limits of meta-stability where the secondary
minimum disappears. If we allow $V_{c}$ as a second control parameter, we find
a coexistence line in the $V_{c}$ \- $\chi_{c}/\chi$ plane ending in a
critical point, i.e. the phenomenology of a liquid - gas transition. For
parameter values on the coexistence line, the kink is stationary.
For the case of the stationary kink, one can write an implicit formula for the
shape: with $u=0$, multiplying (5) by $\varphi^{\prime}$ and integrating from
$-\infty$ to $x$ , with the boundary conditions $\varphi^{\prime}\rightarrow
0$ , $\varphi\rightarrow\varphi_{1}$ for $x\rightarrow-\infty$ one finds

$\frac{d\varphi}{\sqrt{-F(\varphi)+F(\varphi_{1})}}\,=\,-\sqrt{2}\,dx$ (9)

For the stationary kink of (7), which occurs for $\alpha=0$ , we have
$F(\varphi)=a(2\varphi^{2}-\varphi^{4})$ , the maxima of $F$ are at
$\varphi=\pm 1$ , and integrating (9) we find $\varphi(x)=tanh(-\sqrt{2a}\,x)$
. This is the same kink as in the mean field theory of the Ising ferromagnet,
separating two domains of opposite magnetization [5]. It has a special
symmetry (inversion about its center), stemming from the symmetry of this
particular $F$ , which is an even function at the coexistence point $\alpha=0$
. The function $F$ derived for the Artificial Axon from (4) has no such
symmetry, and correspondingly the stationary kink is not inversion symmetric
about its center, as Fig. 1 shows. For this kink too an analytic expression
can be obtained from (9) in terms of special functions.

Conclusions. We have discussed the occurrence of travelling kink solutions in
a dynamical system which represents a space extended Artificial Axon. We
considered the simplest limit: ”fast” channels described by an equilibrium
opening probability $P_{O}(V)$. Even so, the velocity of the kink represents a
non trivial eigenvalue problem (5). More generally, introducing channel
dynamics increases the dimensionality of the dynamical system and leads to
more structure (oscillations, limit cycles i.e. action potentials) as is well
known. We point out a connection to similar kinks in other areas of condensed
matter physics: some questions which can be asked of these systems are
similar, for instance, effects beyond mean field [14, 6]. For us, this means
replacing the uniform channel conductance with a space distribution of point -
like channels, eventually interacting, eventually mobile. Introducing channel
dynamics (see e.g. [15, 16]), it may be interesting to extend this study to
pattern formation in 2 space dimensions. In general, this system may inspire
the construction of new reaction - diffusion systems [17] with interesting
spatio - temporal dynamics.

###### Acknowledgements.

This work was supported by NSF grant DMR - 1809381.

## References

  * Ariyaratne and Zocchi [2016] A. Ariyaratne and G. Zocchi, J. Phys. Chem. B 120, 6255 (2016).
  * Vasquez and Zocchi [2017] H. G. Vasquez and G. Zocchi, EPL 119, 48003 (2017).
  * Marmont [1949] G. Marmont, J Cell Comp. Physiol. 34, 351 (1949).
  * Koch [1999] C. Koch,  _Biophysics of Computation_ (Oxford University Press, 1999).
  * Chaikin and Lubenski [1995] P. Chaikin and T. Lubenski,  _Principles of condensed matter physics_ (Cambridge University Press, 1995).
  * Buijnsters  _et al._ [2014] F. Buijnsters, A. Fasolino, and M. Katsnelson, Phys. Rev. Lett. 113, 217202 (2014).
  * Kolar  _et al._ [1996] H. Kolar, J. Spence, and H. Alexander, Phys. Rev. Lett. 77, 4031 (1996).
  * Rotermund  _et al._ [1991] H. Rotermund, S. Jakubith, A. von Oertzen, and G. Ertl, Phys. Rev. Lett. 66, 3083 (1991).
  * Hodgkin and Huxley [1952] A. L. Hodgkin and A. F. Huxley, J. Physiol. (Lond.) 117, 500 (1952).
  * Fitzhugh [1961] R. Fitzhugh, Biophys. J. 1, 445 (1961).
  * Nagumo  _et al._ [1962] J. Nagumo, S. Arimoto, and S. Yoshizawa, Proceedings of the IRE 50, 2061 (1962).
  * Vasquez and Zocchi [2019] H. G. Vasquez and G. Zocchi, Bioinspiration and Biomimetics 14, 016017 (2019).
  * Van der Pol [1926] B. Van der Pol, Phil. Mag. 2, 978 (1926).
  * Buijnsters  _et al._ [2003] F. Buijnsters, A. Fasolino, and M. Katsnelson, Nature 426, 812 (2003).
  * Morris and Lecar [1981] C. Morris and H. Lecar, Biophys. J. 35, 193 (1981).
  * Pi and Zocchi [2020] Z. Pi and G. Zocchi, arXiv:2012.00221 (2020).
  * Vanag and Epstein [2004] V. K. Vanag and I. R. Epstein, Phys. Rev. Lett. 92, 128301 (2004).