[1]Enrique Garcia-Ceja

[1]Tecnologico de Monterrey, School of Engineering and Sciences, Ave. Eugenio
Garza Sada 2501, Monterrey, 64849, N.L., México

2]Technical University of Darmstadt, Otto-Berndt-Strasse 2, Darmstadt, 64287,
Germany

# Conformal Prediction in Multi-User Settings: An Evaluation

<EMAIL_ADDRESS>Luciano Garcia-Banuelos<EMAIL_ADDRESS>Nicolas
Jourdan<EMAIL_ADDRESS>*  [

###### Abstract

Typically, machine learning models are trained and evaluated without making
any distinction between users (e.g, using traditional hold-out and cross-
validation). However, this produces inaccurate performance metrics estimates
in multi-user settings. That is, situations where the data were collected by
multiple users with different characteristics (e.g., age, gender, height,
etc.) which is very common in user computer interaction and medical
applications. For these types of scenarios model evaluation strategies that
provide better performance estimates have been proposed such as _mixed_ ,
_user-independent_ , _user-dependent_ , and _user-adaptive_ models. Although
those strategies are better suited for multi-user systems, they are typically
assessed with respect to performance metrics that capture the _overall_
behavior of the models and do not provide any performance guarantees for
individual predictions nor they provide any feedback about the predictions’
uncertainty. In order to overcome those limitations, in this work we evaluated
the conformal prediction framework in several multi-user settings. Conformal
prediction is a model agnostic method that provides confidence guarantees on
the predictions, thus, increasing the trustworthiness and robustness of the
models. We conducted extensive experiments using different evaluation
strategies and found significant differences in terms of conformal performance
measures. We also proposed several visualizations based on matrices, graphs,
and charts that capture different aspects of the resulting prediction sets.

###### keywords:

conformal-prediction, multi-user models, uncertainty estimation,
visualization, machine-learning

##  1 Introduction

With the advent of wearable devices and new kinds of sensors, it is now
possible to monitor human behavior from different perspectives including
physical activities [27], hand gestures [24], body postures [23], emotion [1],
sleep patterns [32], and even mental health such as stress and depression
[10]. Most of those systems rely on machine learning models and are part of
larger applications in human computer interaction, sports, behavioral
research, and medicine, to name a few. Many of those applications play
critical roles and thus, the machine learning algorithms supporting them need
to be reliable and robust. The main thing in common about the aforementioned
applications is that the data is intrinsic to the users. For example, in order
for an emotion recognition system to be effective, it needs to be trained on
real data from similar users from those that are expected to use the system.
We call these types of scenarios where the data are inherent to the users:
_multi-user settings_ or _multi-user systems_. On the other hand, an object
recognition system that identifies defects in machining parts or a weather
forecast system are not multi-user systems.

Multi-user systems pose several challenges that are often overlooked being the
two most important ones the _inter/intra person variance_. That is, every
person is different so we cannot expect a single general model to perform
optimally (inter person variance). Moreover, a given person can also change
their behavior over time and thus, a model that works now is not guaranteed to
work in the future (intra person variance).

In the literature, many works do not take into account the multi-user nature
of a problem, as noted by [18]. That is, user identifier information is not
used when deciding how to split the data when evaluating the generalization
performance of a machine learning model. This typically results in
overestimating the models’ performance as will be explained in Section 2.1.

Another important aspect of a machine learning model that is intended to be
deployed is its reliability. That is, _To what extent can we trust the model’s
predictions?_ Most machine learning models are designed to always produce an
answer regardless of its validity. For example, in the traditional setting, a
classifier will always produce a label. Even if the input data belongs to a
new class for which there were no examples in the training set. Even worse,
the predicted label could have an associated high confidence score which can
mislead the final user resulting in potential negative consequences, specially
in critical applications such as medicine or unmanned vehicles. Thus, it
becomes of paramount importance the ability of a model to inform when it
doesn’t know the answer or when in doubt between different alternatives
(labels for a classification problem) and this is exactly what _conformal
prediction_ does. Conformal prediction is a general framework proposed by [31]
that extends the capabilities of any classifier or regression model such that
the uncertainty of the predictions can be quantified. In the case of
classification instead of predicting a single class, a conformal classifier
predicts a set (possibly empty) of classes such that a performance measure is
guaranteed to be satisfied. In the case of regression, the predictions are
confidence intervals. More about the basics of conformal prediction will be
covered in Section 2.2.

We believe that it should become a practice to evaluate interactive systems
based in user data in terms of their 1) multi-user nature, and 2) their
reliability. To this extent, in this work we make an evaluation of different
applications in the intersection between their multi-user nature and
predictions reliability within the conformal prediction framework. Based on
our literature review, this is the first work that analyzes multi-user systems
with conformal prediction. We also introduced a new evaluation strategy (user-
calibrated model) which uses data from the target user to calibrate the
conformal model. The main contributions of this work are threefold:

  * •

We conducted extensive experiments using four real-world datasets to
investigate the influence of the choice of multi-user evaluation strategies
(see Section 2.1) on the quality of conformal and traditional models in terms
of conformal performance measures.

  * •

We introduced a new evaluation strategy (user-calibrated model) which uses
data from the target user to calibrate the conformal model. Based on our
experiments, we found that a mixed-model (which is the most commonly used in
the literature) overestimates the performance of a system compared to a user-
independent model. The user-independent model is what you would usually
experience in real life where the system does not have any information about
the target user.

  * •

We propose the use of matrix and graph based visualizations to analyze
conformal prediction sets and compare the results between traditional and
conformal models. Furthermore, in order to overcome the binary-relation
limitations of the former visualizations, we propose the use of a multiset-
enumerating chart for visualizing n-ary relations.

The specific null hypotheses we aim to test in this work with respect to
_coverage_ are:

  1. 1.

There is no difference in performance between a user-independent model and a
mixed model and user-dependent model.

  2. 2.

There is no difference in performance between a user-dependent model and a
mixed model and user-independent model.

  3. 3.

There is no difference between a user-calibrated model and a user-independent
model.

The aforementioned types of models are described in section 2.1. We conducted
the experiments on four representative open datasets of multi-user settings
and the code to reproduce the results is available here:
https://github.com/enriquegit/conformal-prediction-multiuser.

This document is organized as follows: Section 2 begins by presenting the
background about multi-user validation strategies. Then, it continuous with an
introduction to conformal prediction and performance measures (traditional and
conformal). Section 3 describes the proposed visualization techniques to plot
prediction sets results. In section 4 we describe the datasets used in this
study. Section 5 presents the experiments and results. Finally, section 6
presents the conclusions.

##  2 Background

This section presents the background concepts related to multi-user systems,
conformal prediction, and performance measures for conformal prediction.

###  2.1 Multi-User Validation Strategies

As previously mentioned, traditional evaluation methods applied to an entire
dataset (hold out, cross validation, etc.) do not provide good estimates in
multi-user systems due to the variation between users. [18] pointed out that
from the papers they surveyed, $40\%$ did not make any distinction between
users when evaluating their models. This evaluation strategy overestimates the
generalization performance of a model since by randomly splitting the dataset
into training and test sets, it is very likely that data from a given user
will end up in both sets. This has the effect that the trained model already
has some information about the given user and will perform relatively well.
But when tested on a completely new user the performance will drop
significantly [18].

To this end, the distinction between several evaluation strategies should be
taken into account when validating multi-user systems. The three main
evaluation strategies based on the type of data partitioning are _mixed_ ,
_user-independent_ , and _user-dependent_ [5] and typically, the resulting
trained models are called _mixed model_ , _user-independent model_ , and
_user-dependent model_ , respectively.

Mixed Model (MM). A mixed model is trained by partitioning the data
irrespective of the users. That is, take all the data from all the users and
randomly split it into training and test sets (Figure 1) or use cross
validation. By doing so, it is very probable that part of the data from a
given user $u$ will end up in the training set and some other in the test set.
This means that at test time, the trained model will already ‘know’ something
about user $u$. Thus, we should expect the model to have a good performance
since the data in the train and test sets are very similar. However when
tested with a completely new user, the performance will drop.

Figure 1: Mixed Model data partitioning.

User-independent Model (UIM). A user-independent Model doesn’t know anything
about the target user. That is, it is guaranteed that data from any potential
user at test time was not present in the training set. This is the case when a
system is to be used out-of-the-box without going into any calibration step
specific to the final user. When the amount of data is limited, these models
are evaluated using a leave-one-user-out strategy. That is, iterate trough
each user $u$, and train the model with data from every user different from
$u$ (Figure 2). Compared to the mixed model and the user-dependent model
(presented below), this one typically performs the worst.

Figure 2: User-independent Model data partitioning.

User-dependent Model (UDM). A user-dependent Model is trained with only data
belonging strictly to the same user to be tested. This model typically
performs the best since it is specialized for a particular user. To evaluate
such models, iterate trough each user $u$ and train and test the model only
with data from $u$. The data partitioning within each user can be done with
hold-out or cross validation. Figure 3 shows this evaluation strategy using
hold-out validation ($50$/$50$).

Figure 3: User-dependent Model data partitioning.

Apart from these, there is another type of model called _user-adaptive model_.
This one is a hybrid between the UIM and UDM. A user-adaptive model can start
as a UIM and as more data from the target user is available, it is fined tuned
to adapt it [2, 9, 28]. Another approach is to train the model using data only
from the most similar users [33, 4, 8] or to look for class similarities [6].
In this work, user-adaptive models are not considered since they introduce
some complexities namely: 1) most of them are model dependent, and 2) they are
application dependent.

###  2.2 Conformal Prediction

In the last years, machine learning algorithms have permeated our everyday
lives. With the recent advent of new hardware and software technologies, more
complex and bigger predictive models are being developed capable of solving
complex tasks and problems. However, despite this incredible progress, most
machine learning models are designed to always produce an output regardless of
its quality or trustworthiness. This imposes serious concerns in critical
applications like in medical diagnosis, autonomous vehicles, fraud detection,
to name a few. One of the aspects for a model to be robust, is its capability
to accurately quantify the uncertainty of its predictions. However this aspect
is typically not considered and rather, the focus is put on the overall
performance of a model in terms of metrics such as precision, recall,
F1-score, etc. While those metrics are useful for assessing the expected
generalization performance of a model, they do not capture the expected
average uncertainty of individual predictions (confidence intervals for
regression and p-values for classification). In classification, most models
can produce confidence scores for their predictions. For example, the output
of the softmax function in a neural network [12]. However, those confidence
scores do not represent true probabilities, that is, they are uncalibrated
[11, 21]. Several calibration methods have been proposed [14, 22] however they
are specific for neural networks and often require modifications to the
underlying model. On the other hand, there is a general method called
_conformal prediction_ that was proposed by [31] that addresses those
challenges.

Conformal prediction provides confidence estimates for individual predictions.
Confidence intervals for regression and prediction sets for classification. As
opposed to traditional classifiers that output a single prediction for every
input data point, conformal prediction produces prediction sets. A prediction
set contains the predicted labels such that the true label is contained with
probability $1-\epsilon$ where $\epsilon$ is the acceptable error level. The
size of the prediction set reflects the confidence. Bigger set sizes reflect
more uncertainty, thus, small sets are preferred. Conformal prediction works
as a ‘wrapper’ with any regression or classification model making it very
flexible. Formally, a conformal predictor is defined as:

$\Gamma^{\epsilon}\left(x_{1},y_{1},\ldots,x_{n-1},y_{n-1},x_{n}\right)=y\in
Y$ (1)

such that

$\frac{\lvert\\{i=1,\ldots,n-1:\alpha_{i}\geq\alpha_{n}\\}\rvert}{n-1}>\epsilon$
(2)

where $y\in Y$ is the set of all confident labels and $\alpha_{i}$ is the
nonconformity score for element $i$. A nonconformity score measures how
different a data point is from previous examples. A typical nonconformity
measure for classification is $1$ minus the confidence score of the predicted
class. The quality and tightness of the predicted sets thus primarily depend
on the expressiveness of the score for the given dataset and thus the choice
of the score-producing algorithm. Bagging-based models such as Random Forests
and Neural Network Ensembles as well as other probabilistic models generally
produce more meaningful confidence scores and are thus well suited for
conformal prediction [13]. The nonconformity scores $\alpha_{i}$ are estimated
from a _calibration set_ which is independent from the training set.

The $\epsilon$ is defined by the user beforehand and determines what is the
maximum tolerated error (or a confidence of $\gamma=1-\epsilon$). Conformal
models are guaranteed to produce predictions with a marginal error no more
than $\epsilon$ given some assumptions, with exchangeability being one of the
most critical in the context of multi-user applications. Exchangeability is a
statistical property of a sequence of random variables
$(X_{1},X_{2},\ldots,X_{n})$, where the joint probability distribution remains
unchanged under any permutation $\pi$ of the sequence [31]:

$P(X_{1}=x_{1},X_{2}=x_{2},\ldots,X_{n}=x_{n})=P(X_{\pi(1)}=x_{1},X_{\pi(2)}=x_{2},\ldots,X_{\pi(n)}=x_{n}).$
(3)

This assumption is fundamental in generating reliable and statistically valid
prediction intervals. In case of multi-user applications and MM as well as UIM
models, cf. Section 2.1, in particular, this assumption may be violated by
shifts in the data distributions of different users. This aspect is thus
further analyzed in the experiment section of this work.

In addition to the fundamental assumption of exchangeability, several other
critical assumptions underpin the conformal prediction framework. The
nonconformity measure $\alpha$, which is pivotal in assessing how well a new
instance conforms to a set of existing data points, should exhibit a
consistent behavior. Specifically, for any two instances $x_{i}$ and $x_{j}$,
if they are similar under a chosen metric, their respective nonconformity
scores $\alpha_{i}$ and $\alpha_{j}$ should be closely aligned. This ensures
that the nonconformity measure reliably reflects the data distribution. The
calibration of confidence levels, denoted as $\gamma=1-\epsilon$, also plays a
crucial role. It is imperative that these confidence levels are accurately
calibrated so that the empirical coverage probability of the prediction
intervals aligns with the intended confidence level. Practically, for a given
confidence level $\gamma$, the proportion of instances where the true value is
contained within the interval should converge to $\gamma$ with an increasing
sample size. While not explicitly stated as an assumption, the effectiveness
of conformal prediction is often contingent on the availability of a large
sample size. A substantial dataset stabilizes the empirical distribution of
nonconformity scores, thereby enhancing the robustness of the prediction
intervals.

Conformal models are typically not solely evaluated by metrics that rely on
the error level since this is defined by the user. Instead, they are evaluated
based on properties of the prediction sets. In the next section, we describe
some conformal prediction performance measures.

###  2.3 Performance Measures

In this section we detail the performance measures that are used in this work.
First, we describe standard non-conformal classification metrics based on the
confusion matrix. Next, we discuss other measures that have been specifically
designed to evaluate the quality of conformal predictors.

####  2.3.1 Non-conformal Metrics

In the assessment of predictive models for classification, several metrics are
pivotal for understanding their performance. These include accuracy,
sensitivity, specificity, precision, recall, and the F1 score. Each metric
illuminates different facets of model performance and is essential for a
comprehensive evaluation.

#### Accuracy

Accuracy is the simplest and most intuitive performance metric. It is the
ratio of correctly predicted observations (both true positives and true
negatives) to the total number of observations. The formula for accuracy is
given by:

$\text{Accuracy}=\frac{\text{True Positives (TP)}+\text{True Negatives
(TN)}}{\text{Total Observations}}$ (4)

#### Sensitivity (True Positive Rate)

Sensitivity, also known as the true positive rate, measures the proportion of
actual positives that are correctly identified by the model. It is
particularly useful for evaluating the effectiveness of a model in identifying
positive outcomes. The formula for sensitivity is:

$\text{Sensitivity}=\frac{\text{TP}}{\text{TP}+\text{False Negatives (FN)}}$
(5)

#### Specificity (True Negative Rate)

Specificity, or the true negative rate, measures the proportion of actual
negatives that are correctly identified as such. This metric is crucial for
assessing the model’s ability to identify negative outcomes. The formula for
specificity is:

$\text{Specificity}=\frac{\text{TN}}{\text{TN}+\text{False Positives (FP)}}$
(6)

#### Precision

Precision is the ratio of correctly predicted positive observations to the
total predicted positives. It is a measure of the accuracy of the positive
predictions made by the model. The formula for precision is:

$\text{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}}$ (7)

#### Recall

Recall, which is synonymous with sensitivity, measures the proportion of
actual positives that are correctly identified. It focuses on the model’s
capability to capture positive cases. The formula for recall is the same as
that for sensitivity:

$\text{Recall}=\frac{\text{TP}}{\text{TP}+\text{FN}}$ (8)

#### F1 Score

The F1 score is the harmonic mean of precision and recall, providing a balance
between these two metrics. It is particularly useful when the class
distribution is imbalanced. The F1 score is calculated as follows:

$\text{F1
Score}=2\times\frac{\text{Precision}\times\text{Recall}}{\text{Precision}+\text{Recall}}$
(9)

####  2.3.2 Conformal Measures

While standard metrics like accuracy, sensitivity, and specificity provide a
baseline understanding of model performance, evaluating conformal predictors
require specific measures that reflect their unique characteristics. Conformal
prediction, with its emphasis on reliable uncertainty quantification,
necessitates measures that capture the quality of prediction intervals and the
calibration of probabilistic outputs. This subsection delves into these
specialized measures, explaining their significance and utility in assessing
the performance of conformal prediction models. All these measures (except for
the Jaccard index) are discussed in detail by [30].

We start with a set of measures which consider only the composition of the
prediction set, disregarding whether the expected class is part of the
prediction set or not.

### N criterion (set size)

The _N criterion_ (_setsize_) considers the size of the prediction sets,
reporting the average thereof. Smaller values are preferred.

$\frac{1}{k}\sum_{i=1}^{k}|\Gamma^{\epsilon}_{i}|$ (10)

where $k$ is the number of prediction sets.

#### Percentage of Empty Sets

The percentage of empty sets (_pctempty_) records the proportion of empty
prediction sets:

$\frac{1}{k}\sum_{i=1}^{k}\begin{cases}1,&\text{if}\
|\Gamma^{\epsilon}_{i}|=0\\\ 0,&\text{otherwise}\end{cases}$ (11)

#### M criterion

The M criterion is the proportion of prediction sets that contain more than
one element:

$\frac{1}{k}\sum_{i=1}^{k}\begin{cases}1,&\text{if}\
|\Gamma^{\epsilon}_{i}|>1\\\ 0,&\text{otherwise}\end{cases}$ (12)

#### F criterion

This is the average fuzziness where the fuzziness for a test object $x_{i}$
refers to the sum of the p-values, excluding the largest one. The _F
criterion_ reports the average fuzziness values for a sequence of prediction
sets. Smaller values are considered to be better.

$\frac{1}{k}\sum_{i=1}^{k}\left(\sum_{y}p^{y}_{i}-\underset{y}{\mathrm{max}}p^{y}_{i}\right)$
(13)

We now consider another set of measures, where the presence of the actual/true
class is expected to be part of the prediction set. For this reason, the
measures are also referred to as “observed criteria”. In the following, we
assume that $y_{i}$ corresponds with the actual/true class for pair
$(x_{i},y_{i})$ in the test dataset.

#### Coverage

Coverage refers to the percentage of prediction sets that contain the true
class:

$\frac{1}{k}\sum_{i=1}^{k}\begin{cases}1,&\text{if}\
y_{i}\in\Gamma^{\epsilon}_{i}\\\ 0,&\text{otherwise}\end{cases}$ (14)

#### Jaccard index

The _Jaccard index_ is a commonly-used metric that measures the similarity of
sets. A value of $1$ would indicate that a prediction is identical to the
singleton set that includes the actual/true class. We consider computing the
average value, and prefer a value close to $1$:

$\frac{1}{k}\sum_{i=1}^{k}\frac{|\Gamma^{\epsilon}_{i}\cap\\{y_{i}\\}|}{|\Gamma^{\epsilon}_{i}\cup\\{y_{i}\\}|}$
(15)

### OM criterion

The _OM criterion_ is the proportion of prediction sets that include one or
more false classes. Smaller values are preferred.

$\frac{1}{k}\sum_{i=1}^{k}\begin{cases}1,&\text{if}\
\\{\Gamma^{\epsilon}_{i}\diagdown\\{y_{i}\\}\neq\emptyset\\}\\\
0,&\text{otherwise}\end{cases}$ (16)

### OF criterion (observed fuzziness)

The _OF criterion_ is the average of the sum of the p-values of the false
classes in the prediction set:

$\frac{1}{k}\sum_{i=1}^{k}\sum_{y\neq y_{i}}p^{y}_{i}$ (17)

### OU criterion (observed unconfidence)

The _OU criterion_ is the average unconfidence. The unconfidence of a
prediction set is the largest p-value of the false classes.

$\frac{1}{k}\sum_{i=1}^{k}\underset{y\neq y_{i}}{\mathrm{max}}\ p^{y}_{i}$
(18)

### OE criterion (observed excess)

The _OE criterion_ is the average number of false classes in the prediction
sets.

$\frac{1}{k}\sum_{i=1}^{k}|\Gamma^{\epsilon}_{i}\diagdown\\{y_{i}\\}|$ (19)

##  3 Prediction Sets Visualizations

In this section we describe the methods we will use to visualize the results
of the conformal prediction sets. Non-conformal single predictions are
typically visualized with a _confusion matrix_. Here, we assume that the rows
represent the predicted classes and columns represent the true classes.
Specifically, we propose four visualizations: 1) co-occurrence matrix, 2) co-
occurrence graph, 3) zero diagonal confusion matrix (ZDCM), and 4) multiset-
enumerating chart. The first two are intended to visualize prediction sets
while the ZDCM is a modification of the traditional confusion matrix that will
allow us to compare the relationship between non-conformal and conformal
predictions. The fourth one refers to a family of charts which enumerates all,
or part of all, possible prediction sets along with a chart which displays
statistics about the underlying predictions.

We will also make use of the traditional confusion matrix. Whenever we refer
to a confusion matrix, we will assume that it is column-normalized. That is,
every column sums up to $1$. See Figure 4 (middle-right).

Zero diagonal confusion matrix (ZDCM). This is a traditional confusion matrix
but with the diagonal set to $0$ before normalization. That is, its columns
still sum up to $1$. Note that it is not symmetric. The rationale behind the
ZDCM is to highlight which classes were confused with which other ones. Figure
4 (middle-left) shows a ZDCM. Here we can see that the class _‘Sitting’_ was
confused $50\%$ of the time with _‘Standing’_ and $50\%$ with _‘Downstairs’_.
Later, we will see how the ZDCM allows us to compare single predictions with
prediction sets.

Co-occurrence matrix. This visualization is intended to highlight which pairs
of classes occurred together in the prediction sets. This is a $n\times n$
matrix where $n$ is the number of classes. Every column represents a class and
the proportion of times that class appeared together with every other class
(excluding itself). The proportions that a class appears with itself are
excluded because otherwise, the diagonal would dominate and the off-diagonal
patterns would fade away making them difficult to visualize. The columns of
this matrix also sum up to $1$ and the diagonal is $0$. This matrix is not
symmetric. Figure 4 (top-left) shows an example of a co-occurrence matrix. For
example, we can see that _‘Jogging’_ occurred together with _‘Downstairs’_
$16.1\%$, with _‘Upstairs’_ $68.9\%$, and with _‘Walking’_ $15\%$. It can also
be noted that the co-occurrence matrix is related to the ZDCM. That is, pairs
of classes that tend to be confused, are also likely to appear in the same
prediction set.

Co-occurrence graph. This visualization is an undirected graph where nodes
represent classes and edges represent counts of how many times a pair of
classes occurred together in the prediction sets. The thickness of the edge is
proportional to the counts and the size of a node is proportional to its
degree. This visualization allow us to understand which classes frequently
appeared along other classes (‘popular classes’). Figure 4 (top-right) shows a
co-occurrence graph. This graph can be compared with a confusion matrix. For
instance, popular classes (bigger nodes) also tend to be more colored in the
confusion matrix (rows).

Multiset-enumerating chart. Given that the output of conformal prediction
takes the form of sets, a visualization based on binary relations might
obfuscate facts and even lead to wrong conclusions. For instance, the
thickness of the edge connecting _‘Downstairs’_ and _‘Upstairs’_ in the co-
occurrence graph results from the frequency of the prediction set
$\left\\{Downstairs,Upstairs\right\\}$ added to the frequency of
$\left\\{Downstairs,Upstairs,Walking\right\\}$. There is no way to distinguish
the source of the thickness. For this reason, we introduce the multiset-
enumerating chart. This chart, which is an adaptation of the UpSet chart [17],
enumerates explicitly the powerset of the set of classes, and associates each
set with a chart/plot. Figure 4 (bottom-center) presents an example of a
multiset-enumerating chart which displays the frequency of each prediction set
for a given dataset. From that example, we can draw some clear conclusions: 1)
the prediction sets were dominated by singletons, 2) the sets
$\left\\{Sit,Stand\right\\}$ and $\left\\{Stand,Walk\right\\}$ were also
frequently predicted, implying that _‘Stand’_ can be confounded with either
_‘Walk’_ , 3) the rest of sets are rarely predicted. One shortcoming of this
visualization is the size of the powerset can be large, i.e., $2^{|C|}$, where
$C$ is the set of classes. This inconvenient can be alleviated by filtering
out low frequency sets.

##  4 Datasets

In this section we describe each of the datasets used for the evaluation. We
used the following criteria for including a given dataset:

  * •

The dataset was collected by at least two users.

  * •

The dataset contains information such that it is possible to determine which
sensor readings were collected by which specific user.

  * •

At least $5$ instances per user-class pair exist.

In all datasets, we only kept users that have at least $5$ repetitions for
every class.

###  4.1 D1: WISDM

The WISDM dataset was collected using Android based smartphones [15, 16]. The
data captured with the embedded tri-axial accelerometer was recorded while the
users performed $6$ different physical activities (Walking, Jogging, Walking
downstairs, Walking upstairs, Sitting, and Standing). The phone was placed in
the users’ pocket and the sampling rate was set to $20$Hz. They already
provided a version with $46$ extracted features from fixed length windows of
$10$ seconds each. Some of the features are the average (for each axis),
average resultant acceleration, time between peaks, binned distribution,
standard deviation (for each axis), etc. Some columns ($5$) were removed due
to corrupted data and missing values. The dataset originally has $36$ users.
After filtering users to meet the criteria, $16$ users were kept and the total
number of instances were $2,869$. Table 1 shows the final class distribution.

Table 1: Class distribution for D1: WISDM. Classes | Number of instances | Proportion of instances
---|---|---
Downstairs | 300 | 10.45%
Jogging | 833 | 29.0%
Sitting | 241 | 8.4%
Standing | 176 | 6.13%
Upstairs | 329 | 11.47%
Walking | 990 | 34.5%

###  4.2 D2: HAR70+

The HAR70+ dataset was collected using two tri-axial accelerometers by $18$
older adult participants ($70$-$95$ years old) [19, 29]. One of the sensors
was attached to the right front thigh while the other one was in the lower
back. The sampling rate was set to $50$Hz. The dataset has $7$ activities:
Walking, Shuffling, Ascending stairs, Descending stairs, Standing, Sitting,
and Lying. Since data for activities Ascending/Descending stairs was scarce,
those classes were removed. For each sensor, $16$ features were extracted from
non-overlapping windows of size $150$ ($3$ seconds). The features included:
mean, standard deviation, and maximum for each axis. The correlation between
each pair of axes, the mean magnitude, the standard deviation of the
magnitude, area under the curve, and the mean difference between consecutive
readings. A moving average filter of size $10$ was applied before extracting
the features. After filtering the users that don’t meet the criteria, $16$
users were kept in the dataset and a total of $12,887$ instances were
generated. Table 2 shows the final class distribution.

Table 2: Class distribution for D2: HAR70+. Classes | Number of instances | Proportion of instances
---|---|---
Lying | 1096 | 8.5%
Shuffling | 405 | 3.14%
Sitting | 2456 | 19.0%
Standing | 2492 | 19.34%
Walking | 6438 | 50%

###  4.3 D3: Opportunity

This dataset consists of daily activities recorded with body-worn sensors,
object sensors and ambient sensors [26, 25]. In this work we use the
locomotion actions labels (Stand, Walk, Sit, Lie). The data was recorded by
$4$ users. Even though the users worn several sensors, we only extracted
features from two accelerometers as in [7] which were sampled at $30$Hz. One
located in the back and the other one in the right shoe. The same $16$
features as the D2: HAR70+ dataset were extracted for each sensor on fixed
non-overlapping windows of size 120 (4 seconds). The total number of resulting
instances were $4,080$. Table 3 shows the final class distribution.

Table 3: Class distribution for D3: Opportunity. Classes | Number of instances | Proportion of instances
---|---|---
Lying | 202 | 5.0%
Sitting | 977 | 24.0%
Standing | 1832 | 45.0%
Walking | 1069 | 26.0%

###  4.4 D4: Smartwatch Gestures

The smartwatch gestures dataset was collected to evaluate several gesture
recognition algorithms for mobile application control using arm gestures. The
dataset was originally presented in [3] and contains acceleration data from
the $3$-axis accelerometer of a first generation Sony SmartWatch at a sampling
rate of $10$ Hz paired with measurement timestamps. The device was worn on the
right wrist of the users. In total, the dataset covers $20$ different
gestures, performed $20$ times by $8$ different users. The class frequencies
are thus approximately uniformly distributed. The gestures have been manually
segmented by the SmartWatch users. The same $16$ features as in the previous
two datasets were extracted for each of the gesture samples. Table 4 shows the
final class distribution.

Table 4: Class distribution for D4: Smartwatch Gestures. Classes | Number of instances | Proportion of instances
---|---|---
Gesture 01 | 160 | 4.9%
Gesture 02 | 161 | 5.0%
Gesture 03 | 166 | 5.1%
Gesture 04 | 166 | 5.1%
Gesture 05 | 160 | 4.9%
Gesture 06 | 164 | 5.0%
Gesture 07 | 161 | 5.0%
Gesture 08 | 164 | 5.0%
Gesture 09 | 161 | 5.0%
Gesture 10 | 164 | 5.0%
Gesture 11 | 162 | 5.0%
Gesture 12 | 161 | 5.0%
Gesture 13 | 161 | 5.0%
Gesture 14 | 168 | 5.2%
Gesture 15 | 162 | 5.0%
Gesture 16 | 161 | 5.0%
Gesture 17 | 163 | 5.0%
Gesture 18 | 163 | 5.0%
Gesture 19 | 162 | 5.0%
Gesture 20 | 161 | 5.0%

###  4.5 Summary of Datasets

Table 5 presents a summary of all the datasets used in this study after
filtering them using the criteria specified in Section 4.

Table 5: Datasets Summary Abbr. |  Name |  # subjects |  # classes |  # instances
---|---|---|---|---
D1 |  WISDM |  16 |  6 |  $2,869$
D2 |  HAR70+ |  16 |  5 |  $12,887$
D3 |  Opportunity |  4 |  4 |  $4,080$
D4 |  Smartwatch Gestures |  8 |  20 |  $3,200$

##  5 Experiments and Results

For the experiments, we evaluated four different settings: Mixed models, user-
dependent models, user-independent models, and a user-calibrated models (UCM).
The UCM is trained the same way as the User-independent model but it is
calibrated only with data from the target user. The conformal confidence was
set to $\alpha=0.95$.

For all datasets, the data was split in the same way based on the type of
model. For the MM and UDM the training set consisted of $60\%$, test set
$20\%$, and calibration set $20\%$. For the UIM, the training set consisted of
$60\%$ and calibration set $40\%$ all these data excluding the target user.
The test set consisted of $50\%$ of all of the target user data. For the UCM
the training set was $60\%$ of all the data excluding the target user. The
calibration set was $50\%$ of the target user data and the other $50\%$ of the
target user data was assigned to the test set. In order to account for
variance, each experiment was run $20$ times. Since some of the classifiers
are scale sensitive, the data was normalized to $0-1$. The scaling parameters
were learned only from the training set in order to avoid data injection. The
experiments were implemented with the help of the conformal prediction library
MAPIE [20]. We used the LAC method as the non-conformity function defined as
one minus the score of the true label111MAPIE documentation:
https://mapie.readthedocs.io/en/latest/theoretical_description_classification.html.
We trained four types of conformal classifiers: Naive Bayes, Random Forest
(RF), Support Vector Machine (SVM), and _k_ -nearest neighbors (KNN).

Tables 6,7,8, and 9 show the average and standard deviation results across the
$20$ experiments iterations for the MM, UDM, UIM, and UCM, respectively. The
first four are non-conformal measures and are in percentages. The remaining
measures are the conformal ones. For the non-conformal measures, RF
consistently achieved the best results for all types of models. For all types
of models, except the UIM the minimum coverage of $95\%$ was achieved. In
conformal prediction, the calibration set is assumed to have the same
distribution as the test set. For the UIM this assumption is violated since
the model does not have any information about the target user. On the other
hand, the minimum coverage for the UCM was attained for all classifiers. Even
though the training set doesn’t have data from the target user, only data from
the target user was used for calibration. In this case, having calibration
data from the given target user allowed the model to satisfy the minimum
coverage. It is also interesting to see that the calibration set from a single
user in the UCM is much smaller than the calibration set taken from all users
in the UIM. Still with much less data, the coverage was satisfied.

In most of the cases (except in the UCM) RF achieved the smallest average set
size. The biggest set sizes were produced by KNN (latter we will discuss why).
The percent of empty sets was very low and in many cases, no empty sets were
produced by the classifiers.

Also note that the non-conformal measures between the UIM and UCM are the same
since the models were trained with the same data. However, their conformal
measures differ because the calibration set was different.

Table 6: D1: Results MM. * Smaller values are preferred. | Naive Bayes | RF | SVM | KNN
---|---|---|---|---
accuracy | 78.65$\pm$1.10 | 88.92$\pm$0.91 | 79.89$\pm$0.73 | 88.28$\pm$1.18
sensitivity | 70.02$\pm$1.68 | 84.20$\pm$1.34 | 69.97$\pm$1.29 | 82.88$\pm$1.87
specificity | 95.25$\pm$0.23 | 97.60$\pm$0.19 | 95.23$\pm$0.19 | 97.57$\pm$0.24
F1 | 69.53$\pm$1.68 | 85.25$\pm$1.25 | 69.84$\pm$1.64 | 83.63$\pm$1.73
coverage | 95.46$\pm$1.26 | 95.43$\pm$1.28 | 95.22$\pm$0.93 | 98.22$\pm$2.28
jaccard | 0.60$\pm$0.03 | 0.86$\pm$0.01 | 0.73$\pm$0.01 | 0.44$\pm$0.34
setsize* | 2.06$\pm$0.14 | 1.22$\pm$0.05 | 1.63$\pm$0.06 | 4.10$\pm$2.39
pctempty | 0.00$\pm$0.00 | 0.10$\pm$0.13 | 0.00$\pm$0.00 | 0.00$\pm$0.00
MCriterion* | 0.58$\pm$0.04 | 0.20$\pm$0.04 | 0.40$\pm$0.03 | 0.69$\pm$0.39
FCriterion* | 0.18$\pm$0.02 | 0.04$\pm$0.01 | 0.09$\pm$0.01 | 0.28$\pm$0.05
OM* | 0.60$\pm$0.03 | 0.23$\pm$0.03 | 0.42$\pm$0.03 | 0.70$\pm$0.38
OF* | 0.17$\pm$0.02 | 0.03$\pm$0.01 | 0.10$\pm$0.01 | 0.24$\pm$0.15
OU* | 0.13$\pm$0.01 | 0.03$\pm$0.00 | 0.08$\pm$0.00 | 0.09$\pm$0.03
OE* | 1.10$\pm$0.13 | 0.26$\pm$0.04 | 0.68$\pm$0.05 | 3.12$\pm$2.37
Table 7: D1: Results UDM. * Smaller values are preferred. | Naive Bayes | RF | SVM | KNN
---|---|---|---|---
accuracy | 87.97$\pm$1.18 | 92.67$\pm$0.96 | 82.95$\pm$1.22 | 85.51$\pm$1.08
sensitivity | 80.82$\pm$2.08 | 89.02$\pm$1.33 | 72.87$\pm$1.73 | 75.85$\pm$2.00
specificity | 97.53$\pm$0.26 | 98.43$\pm$0.21 | 95.88$\pm$0.27 | 96.95$\pm$0.23
F1 | 85.56$\pm$2.25 | 90.44$\pm$1.38 | 86.69$\pm$1.66 | 82.89$\pm$1.30
coverage | 98.61$\pm$0.64 | 97.63$\pm$0.56 | 97.61$\pm$0.75 | 99.00$\pm$0.51
jaccard | 0.33$\pm$0.07 | 0.79$\pm$0.06 | 0.68$\pm$0.03 | 0.33$\pm$0.06
setsize* | 4.77$\pm$0.52 | 1.71$\pm$0.36 | 2.04$\pm$0.21 | 4.90$\pm$0.44
pctempty | 0.03$\pm$0.07 | 0.42$\pm$0.30 | 0.29$\pm$0.37 | 0.06$\pm$0.12
MCriterion* | 0.81$\pm$0.08 | 0.30$\pm$0.07 | 0.47$\pm$0.04 | 0.82$\pm$0.07
FCriterion* | 0.37$\pm$0.05 | 0.17$\pm$0.01 | 0.20$\pm$0.01 | 0.46$\pm$0.03
OM* | 0.82$\pm$0.08 | 0.31$\pm$0.07 | 0.48$\pm$0.04 | 0.82$\pm$0.07
OF* | 0.37$\pm$0.06 | 0.06$\pm$0.02 | 0.11$\pm$0.01 | 0.46$\pm$0.04
OU* | 0.12$\pm$0.02 | 0.03$\pm$0.01 | 0.07$\pm$0.01 | 0.14$\pm$0.01
OE* | 3.78$\pm$0.51 | 0.73$\pm$0.35 | 1.06$\pm$0.21 | 3.91$\pm$0.44
Table 8: D1: Results UIM. * Smaller values are preferred. | Naive Bayes | RF | SVM | KNN
---|---|---|---|---
accuracy | 75.45$\pm$0.85 | 78.03$\pm$0.96 | 75.58$\pm$0.83 | 69.13$\pm$0.84
sensitivity | 66.98$\pm$0.92 | 72.74$\pm$1.00 | 66.27$\pm$0.87 | 62.58$\pm$0.98
specificity | 94.55$\pm$0.19 | 95.19$\pm$0.19 | 94.26$\pm$0.19 | 93.60$\pm$0.17
F1 | 73.90$\pm$1.89 | 77.09$\pm$1.39 | 77.65$\pm$1.34 | 66.05$\pm$1.54
coverage | 93.89$\pm$0.50 | 88.76$\pm$1.35 | 91.83$\pm$0.49 | 88.04$\pm$2.70
jaccard | 0.58$\pm$0.01 | 0.73$\pm$0.01 | 0.68$\pm$0.01 | 0.50$\pm$0.09
setsize* | 2.07$\pm$0.04 | 1.39$\pm$0.04 | 1.70$\pm$0.04 | 2.97$\pm$0.75
pctempty | 0.00$\pm$0.00 | 0.09$\pm$0.11 | 0.00$\pm$0.00 | 0.00$\pm$0.00
MCriterion* | 0.58$\pm$0.02 | 0.34$\pm$0.03 | 0.43$\pm$0.02 | 0.58$\pm$0.11
FCriterion* | 0.18$\pm$0.01 | 0.06$\pm$0.00 | 0.10$\pm$0.00 | 0.27$\pm$0.01
OM* | 0.62$\pm$0.01 | 0.42$\pm$0.02 | 0.48$\pm$0.02 | 0.65$\pm$0.09
OF* | 0.19$\pm$0.01 | 0.07$\pm$0.00 | 0.13$\pm$0.00 | 0.27$\pm$0.04
OU* | 0.14$\pm$0.00 | 0.06$\pm$0.00 | 0.10$\pm$0.00 | 0.18$\pm$0.01
OE* | 1.13$\pm$0.04 | 0.50$\pm$0.03 | 0.78$\pm$0.03 | 2.09$\pm$0.73
Table 9: D1: Results UCM. * Smaller values are preferred. | Naive Bayes | RF | SVM | KNN
---|---|---|---|---
accuracy | 75.45$\pm$0.85 | 78.03$\pm$0.96 | 75.58$\pm$0.83 | 69.13$\pm$0.84
sensitivity | 66.98$\pm$0.92 | 72.74$\pm$1.00 | 66.27$\pm$0.87 | 62.58$\pm$0.98
specificity | 94.55$\pm$0.19 | 95.19$\pm$0.19 | 94.26$\pm$0.19 | 93.60$\pm$0.17
F1 | 73.90$\pm$1.89 | 77.09$\pm$1.39 | 77.65$\pm$1.34 | 66.05$\pm$1.54
coverage | 95.62$\pm$0.68 | 96.28$\pm$0.62 | 95.46$\pm$0.70 | 99.92$\pm$0.25
jaccard | 0.58$\pm$0.02 | 0.61$\pm$0.01 | 0.65$\pm$0.01 | 0.17$\pm$0.01
setsize* | 2.27$\pm$0.15 | 2.04$\pm$0.09 | 1.95$\pm$0.08 | 5.97$\pm$0.09
pctempty | 0.00$\pm$0.00 | 0.04$\pm$0.05 | 0.08$\pm$0.13 | 0.00$\pm$0.00
MCriterion* | 0.60$\pm$0.03 | 0.60$\pm$0.02 | 0.51$\pm$0.02 | 1.00$\pm$0.01
FCriterion* | 0.25$\pm$0.01 | 0.21$\pm$0.01 | 0.20$\pm$0.01 | 1.00$\pm$0.04
OM* | 0.62$\pm$0.03 | 0.62$\pm$0.02 | 0.52$\pm$0.02 | 1.00$\pm$0.01
OF* | 0.24$\pm$0.01 | 0.20$\pm$0.01 | 0.20$\pm$0.01 | 1.11$\pm$0.03
OU* | 0.16$\pm$0.01 | 0.15$\pm$0.01 | 0.13$\pm$0.00 | 0.35$\pm$0.01
OE* | 1.32$\pm$0.15 | 1.08$\pm$0.09 | 0.99$\pm$0.08 | 4.97$\pm$0.08

Figure 4 shows the co-occurrence plot and the co-occurrence graph at the top,
the zero diagonal confusion matrix (ZDCM) and the confusion matrix in the
middle, and the multiset-enumerating chart at the bottom (for the mixed model
with RF). The top plots capture information about the resulting conformal
prediction sets. From the co-occurrence matrix, it can be seen that the
classes with which _‘Downstairs’_ co-occurred the most were _‘Walking’_
($33.5\%$) and _‘Upstairs’_ ($62.9\%$). _‘Sitting’_ co-occurred the most with
_‘Standing’_ which makes sense since both are low-acceleration activities.
From the co-occurrence graph it can be seen that the pair _‘Downstairs’_
-_‘Upstairs’_ is the one that occurred the most (thickest edge). From the
smallest nodes sizes, we see that _‘Standing’_ and _‘Sitting’_ are the classes
that appeared less frequently together with other classes.

The middle plots capture information about the single predictions of the non-
conformal models. The ZDCM shows that for example, _‘Jogging’_ was confused
many times with _‘Walking’_ and _‘Upstairs’_. It is also worth noting the
relationship between graphs. The co-occurrence matrix and the ZDCM look very
similar. That is, if two classes often appear together in the prediction set
of the conformal classifier, it is also likely that they will be confused by
the non-conformal classifier. Similarly, in the co-occurrence graph the
biggest nodes correspond to _‘Downstairs’_ , _‘Upstairs’_ , and _‘Walking’_
which are also the most colored ones (rows) in the corresponding confusion
matrix. From the multiset-enumerating chart we can see that apart from the
singletons, the most frequent prediction sets were the ones containing the set
{_‘Upstairs’_ ,_‘Downstairs’_}.

Figure 4: D1: WISDM. Co-occurrence matrix (upper-left), co-occurrence graph
(upper-right), zero diagonal confusion matrix (middle-left), confusion matrix
(middle-right), and multiset-enumerating bar chart (bottom-center) for mixed
model with RF.

The visual patterns in the figures looked similar for the other types of
models and classifiers (except KNN). For clarity, here we only include plots
for the mixed model and RF but the published source code generates all of
them. For KNN, the co-occurrence plot looks uniform (Figure 5). As previously
mentioned, KNN is also the classifier with the biggest set sizes. This is
because KNN does not produce very well calibrated prediction scores. It’s
prediction scores are computed as the fraction of nearest neighbors that
belong to the true class. For example, if $k=3$ and the true class is _‘A’_ ,
and all the nearest neighbors are of class _‘A’_ , then the score for _‘A’_
will be $100\%$. The same behavior was observed for all the remaining
datasets.

Figure 5: Co-occurrence matrix for KNN.

Figure 6 shows boxplots of the coverage for all model types across all
classifiers. In order to test the hypotheses defined in Section 1, the
significance levels between models are also shown (t-test). Here, we can see
that all comparisons are statistically significant and thus, we reject the
three hypotheses. Similar to non-conformal models, the conformal UDM was the
best performing one. The UCM performed better than the UIM.

Figure 6: D1: WISDM. Boxplots of coverage between models. ns: $p>0.05$, *:
$p<=0.05$, **: $p<=0.01$, ***: $p<=0.001$, ****: $p<=0.0001$.

Figure 7 shows a plot comparing the set size between classifiers and for all
models. Here we can see that RF achieves the smallest sets followed by SVM and
Naive Bayes. As already seen, KNN produces the biggest sets.

Figure 7: D1: WISDM. Lolliplot of set size for all classifiers and models.

Figure 8 shows the resulting plots for SVM and dataset D2. Here, we can see
similar patterns than those for D1 using RF (Figure 4).

Figure 8: D2: Opportunity. Co-occurrence matrix (upper-left), co-occurrence
graph (upper-right), zero diagonal confusion matrix (middle-left), confusion
matrix (middle-right), and multiset-enumerating bar chart (bottom-center) for
mixed model with SVM.

Figure 9 shows the coverage boxplots for dataset D2. In this case the
difference between the mixed and the UDM was not significant. Figure 10 had a
similar pattern compared to dataset D1 being RF and SVM the classifiers that
produced the smallest sets.

Figure 9: D2: Opportunity. Boxplots of coverage between models. ns: $p>0.05$,
*: $p<=0.05$, **: $p<=0.01$, ***: $p<=0.001$, ****: $p<=0.0001$.  Figure 10:
D2: Opportunity. Lolliplot of set size for all classifiers and models.

For dataset D3, the patterns in the plots look similar, thus, they were not
included here. Figure 11 shows the resulting coverage boxplots. In this case,
the differences between the compared models were all significant.

Figure 11: D3: HAR70+. Boxplots of coverage between models. ns: $p>0.05$, *:
$p<=0.05$, **: $p<=0.01$, ***: $p<=0.001$, ****: $p<=0.0001$.

Similar to dataset D2, the coverage difference between the mixed and the UDM
was not significant for dataset D4 (Figure 12).

Figure 12: D4: Smartwatch Gestures. Boxplots of coverage between models. ns:
$p>0.05$, *: $p<=0.05$, **: $p<=0.01$, ***: $p<=0.001$, ****: $p<=0.0001$.

##  6 Conclusion

In this work we conducted exhaustive tests of several evaluation strategies in
multi-user systems within the framework of conformal prediction. We reported
conformal and non-conformal performance measures and proposed several
prediction set visualizations to better understand the results of conformal
prediction sets and how they relate to non-conformal (singleton) predictions.
We also proposed an evaluation strategy (UCM) where first, a general model is
trained and then, it is calibrated using data from the target user.

From the results, we observed that RF produced the smallest sets while also
attaining the highest performance in terms of non-conformal measures which is
in agreement with previous work [13]. Overall, and as expected, the UIM
obtained the lowest performance in all cases which aligns with the results of
previous works. For all datasets, the MM was better than the UIM which can
lead to overestimation of the performance of a system since as previously
mentioned, the MM approach is the most common one used in the literature to
evaluate a system, however, it does not take into account the inter-user
variance. This was the case for both conformal and non-conformal measures.

The three stated null hypotheses were rejected for all datasets except
hypothesis 2 since there was no significant difference between the UDP and the
MM for 2 out of the 4 datasets. For all datasets, the co-occurrence matrix
plot looked similar to the zero diagonal confusion matrix and the co-
occurrence graph biggest nodes corresponded with the most row-colored classes
in the confusion matrix.

In terms of coverage, the UCM achieved better performance than the UIM despite
using less calibration data. This shows that for multi-user systems, more data
doesn’t necessarily mean better. This is because the data may not have been
generated by the same distribution.

In this work we have focused on 4 multi-user evaluation strategies. However,
there is still another one, often called user-adaptive model or UAM for short
[9]. A UAM tries to mitigate the weaknesses of a UIM and a UDM. That is, it
tries to achieve better performance than a general model while trying to
minimize the training data from the target user. In future work we plan to
evaluate this types of models that adapt over time within conformal
prediction. We also plan to further extend the analysis of the proposed
visualization methods for prediction sets.

Supplementary information

The source code to reproduce the results is available here:
https://github.com/enriquegit/conformal-prediction-multiuser.

## Declarations

Authors’ contributions Study conception and design by EGC. Datasets collection
and preprocessing by EGC and NJ. Visualizations by EGC and LGB. All authors
contributed to the analysis of results and document writing. All authors read
and approved the final manuscript.

Competing interests The authors have no competing interests to declare that
are relevant to the content of this article.

Funding No funding was received for conducting this study.

Data availability All the datasets used in this study are open access. The
corresponding references for each dataset are listed in section 4.

Code availability The code and instructions on how to reproduce all the
results in this study are available here:
https://github.com/enriquegit/conformal-prediction-multiuser.

Ethics approval Not applicable.

Consent to participate Not applicable.

Consent for publication Not applicable.

## References

  *   * Al-Dujaili Ebrahimi-Moghadam [2023] al2023Al-Dujaili, M.J. Ebrahimi-Moghadam, A.  2023\.  Speech emotion recognition: a comprehensive survey Speech emotion recognition: a comprehensive survey.  Wireless Personal Communications12942525–2561,
  * Churaev Savchenko [2022] churaev2022multiChuraev, E. Savchenko, A.V.  2022\.  Multi-user facial emotion recognition in video based on user-dependent neural network adaptation Multi-user facial emotion recognition in video based on user-dependent neural network adaptation.  2022 VIII International Conference on Information Technology and Nanotechnology (ITNT) 2022 viii international conference on information technology and nanotechnology (itnt) ( 1–5).
  * Costante . [2014] smartwatch_gesturesCostante, G., Porzi, L., Lanz, O., Valigi, P. Ricci, E.  2014\.  Personalizing a smartwatch-based gesture interface with transfer learning Personalizing a smartwatch-based gesture interface with transfer learning.  2014 22nd European Signal Processing Conference (EUSIPCO) 2014 22nd european signal processing conference (eusipco) ( 2530-2534).
  * Ferrari . [2020] ferrari2020personalizationFerrari, A., Micucci, D., Mobilio, M. Napoletano, P.  2020\.  On the personalization of classification models for human activity recognition On the personalization of classification models for human activity recognition.  IEEE Access832066–32079,
  * Garcia-Ceja [2021] GarciaCejaBookGarcia-Ceja, E.  2021\.  Behavior Analysis with Machine Learning Using R Behavior analysis with machine learning using R.  http://behavior.enriquegc.com
  * Garcia-Ceja Brena [2015] garcia2015buildingGarcia-Ceja, E. Brena, R.  2015\.  Building personalized activity recognition models with scarce labeled data based on class similarities Building personalized activity recognition models with scarce labeled data based on class similarities.  Ubiquitous Computing and Ambient Intelligence. Sensing, Processing, and Using Environmental Information: 9th International Conference, UCAmI 2015, Puerto Varas, Chile, December 1-4, 2015, Proceedings 9 Ubiquitous computing and ambient intelligence. sensing, processing, and using environmental information: 9th international conference, ucami 2015, puerto varas, chile, december 1-4, 2015, proceedings 9 ( 265–276).
  * Garcia-Ceja Brena [2018] garcia2018improvedGarcia-Ceja, E. Brena, R.F.  2018\.  An improved three-stage classifier for activity recognition An improved three-stage classifier for activity recognition.  International Journal of Pattern Recognition and Artificial Intelligence32011860003,
  * Garcia-Ceja . [2015] garcia2015automaticGarcia-Ceja, E., Osmani, V. Mayora, O.  2015\.  Automatic stress detection in working environments from smartphones’ accelerometer data: a first step Automatic stress detection in working environments from smartphones’ accelerometer data: a first step.  IEEE journal of biomedical and health informatics2041053–1060,
  * Garcia-Ceja . [2020] garcia2020adaptiveGarcia-Ceja, E., Riegler, M., Kvernberg, A.K. Torresen, J.  2020\.  User-adaptive models for activity and emotion recognition using deep transfer learning and data augmentation User-adaptive models for activity and emotion recognition using deep transfer learning and data augmentation.  User Modeling and User-Adapted Interaction30365–393,
  * Garcia-Ceja . [2018] garcia2018mentalGarcia-Ceja, E., Riegler, M., Nordgreen, T., Jakobsen, P., Oedegaard, K.J. Tørresen, J.  2018\.  Mental health monitoring with multimodal sensing and machine learning: A survey Mental health monitoring with multimodal sensing and machine learning: A survey.  Pervasive and Mobile Computing511–26,
  * Guo . [2017] guo2017calibrationGuo, C., Pleiss, G., Sun, Y. Weinberger, K.Q.  2017\.  On calibration of modern neural networks On calibration of modern neural networks.  International conference on machine learning International conference on machine learning ( 1321–1330).
  * Hendrycks Gimpel [2016] hendrycks2016baselineHendrycks, D. Gimpel, K.  2016\.  A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks A baseline for detecting misclassified and out-of-distribution examples in neural networks.  International Conference on Learning Representations. International conference on learning representations.
  * Jourdan . [2021] jourdan_neurips_2021Jourdan, N., Sen, S., Husom, E.J., Garcia-Ceja, E., Biegel, T. Metternich, J.  2021Dec.  On The Reliability Of Machine Learning Applications In Manufacturing Environments On the reliability of machine learning applications in manufacturing environments.  NeurIPS Workshop on Distribution Shifts,  http://arxiv.org/abs/2112.06986
  * Karandikar . [2021] karandikar2021softKarandikar, A., Cain, N., Tran, D., Lakshminarayanan, B., Shlens, J., Mozer, M.C. Roelofs, B.  2021\.  Soft calibration objectives for neural networks Soft calibration objectives for neural networks.  Advances in Neural Information Processing Systems3429768–29779,
  * Kwapisz . [2011] Kwapisz2011Kwapisz, J.R., Weiss, G.M. Moore, S.A.  201103\.  Activity recognition using cell phone accelerometers Activity recognition using cell phone accelerometers.  SIGKDD Explor. Newsl.12274-82,  https://doi.org/10.1145/1964897.1964918 http://doi.acm.org/10.1145/1964897.1964918
  * Kwapisz . [2012] wisdmKwapisz, J.R., Weiss, G.M. Moore, S.A.  2012\.  Activity Prediction Dataset. Activity prediction dataset.  http://www.cis.fordham.edu/wisdm/dataset.php.  Accessed: August 21 2023
  * Lex . [2014] 2014_infovis_upsetLex, A., Gehlenborg, N., Strobelt, H., Vuillemot, R. Pfister, H.  2014\.  UpSet: Visualization of Intersecting Sets Upset: Visualization of intersecting sets.  IEEE Transactions on Visualization and Computer Graphics (InfoVis)20121983–1992,  https://doi.org/10.1109/TVCG.2014.2346248
  * Lockhart Weiss [2014] lockhart2014limitationsLockhart, J.W. Weiss, G.M.  2014\.  Limitations with activity recognition methodology & data sets Limitations with activity recognition methodology & data sets.  Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication Proceedings of the 2014 acm international joint conference on pervasive and ubiquitous computing: Adjunct publication ( 747–756).
  * Logacjov Ustad [2023] har70Logacjov, A. Ustad, A.  2023\.  HAR70+. HAR70+.  UCI Machine Learning Repository.  DOI: https://doi.org/10.24432/C5CW3D
  * MAPIE [2022] mapieMAPIE  2022\.  MAPIE library. Mapie library.  https://github.com/scikit-learn-contrib/MAPIE.  Accessed: September 10 2023
  * Minderer . [2021] minderer2021revisitingMinderer, M., Djolonga, J., Romijnders, R., Hubis, F., Zhai, X., Houlsby, N.Lucic, M.  2021\.  Revisiting the calibration of modern neural networks Revisiting the calibration of modern neural networks.  Advances in Neural Information Processing Systems3415682–15694,
  * Mukhoti . [2020] mukhoti2020calibratingMukhoti, J., Kulharia, V., Sanyal, A., Golodetz, S., Torr, P. Dokania, P.  2020\.  Calibrating deep neural networks using focal loss Calibrating deep neural networks using focal loss.  Advances in Neural Information Processing Systems3315288–15299,
  * Patel . [2017] patel2017Patel, P., Bhatt, B. Patel, B.  2017\.  Human body posture recognition—A survey Human body posture recognition—a survey.  2017 International conference on innovative mechanisms for industry applications (ICIMIA) 2017 international conference on innovative mechanisms for industry applications (icimia) ( 473–477).
  * Rawat . [2023] rawat2023Rawat, P., Kane, L., Goswami, M., Jindal, A. Sehgal, S.  2023\.  A Review on Vision-based Hand Gesture Recognition Targeting RGB-Depth Sensors A review on vision-based hand gesture recognition targeting rgb-depth sensors.  International Journal of Information Technology & Decision Making2201115–156,
  * C.A.ND.LV.C.R. Roggen Daniel Sagha [2012] oppRoggen, C.A.ND.LV.C.R., Daniel Sagha, H.  2012\.  OPPORTUNITY Activity Recognition. OPPORTUNITY Activity Recognition.  UCI Machine Learning Repository.  DOI: https://doi.org/10.24432/C5M027
  * D. Roggen . [2010] Roggen2010Roggen, D., Calatroni, A., Rossi, M., Holleczek, T., Forster, K., Troster, G.del R Millan, J.  2010June.  Collecting complex activity datasets in highly rich networked sensor environments Collecting complex activity datasets in highly rich networked sensor environments.  Networked Sensing Systems (INSS), 2010 Seventh International Conference on Networked sensing systems (inss), 2010 seventh international conference on ( 233-240).
  * Saleem . [2023] saleem2023Saleem, G., Bajwa, U.I. Raza, R.H.  2023\.  Toward human activity recognition: a survey Toward human activity recognition: a survey.  Neural Computing and Applications3554145–4182,
  * Siirtola Röning [2019] siirtola2019incrementalSiirtola, P. Röning, J.  2019\.  Incremental learning to personalize human activity recognition models: the importance of human AI collaboration Incremental learning to personalize human activity recognition models: the importance of human ai collaboration.  Sensors19235151,
  * Ustad . [2023] Ustad2023Ustad, A., Logacjov, A., Trollebø, S.Ø., Thingstad, P., Vereijken, B., Bach, K. Maroni, N.S.  2023\.  Validation of an Activity Type Recognition Model Classifying Daily Physical Behavior in Older Adults: The HAR70+ Model Validation of an activity type recognition model classifying daily physical behavior in older adults: The HAR70+ model.  Sensors235,  https://doi.org/10.3390/s23052368 https://www.mdpi.com/1424-8220/23/5/2368
  * Vovk . [2016] vovk2016criteriaVovk, V., Fedorova, V., Nouretdinov, I. Gammerman, A.  2016\.  Criteria of efficiency for conformal prediction Criteria of efficiency for conformal prediction.  Conformal and Probabilistic Prediction with Applications: 5th International Symposium, COPA 2016, Madrid, Spain, April 20-22, 2016, Proceedings 5 Conformal and probabilistic prediction with applications: 5th international symposium, copa 2016, madrid, spain, april 20-22, 2016, proceedings 5 ( 23–39).
  * Vovk . [2005] vovk2005algorithmicVovk, V., Gammerman, A. Shafer, G.  2005\.  Algorithmic learning in a random world Algorithmic learning in a random world ( 29).  Springer.
  * Watson Fernandez [2021] watson2021Watson, N.F. Fernandez, C.R.  2021\.  Artificial intelligence and sleep: Advancing sleep medicine Artificial intelligence and sleep: Advancing sleep medicine.  Sleep medicine reviews59101512,
  * Xu . [2014] xu2014clusterXu, Q., Nwe, T.L. Guan, C.  2014\.  Cluster-based analysis for personalized stress evaluation using physiological signals Cluster-based analysis for personalized stress evaluation using physiological signals.  IEEE journal of biomedical and health informatics191275–281,


# How far does turbulence spread?

Alexandros Alexakis1<EMAIL_ADDRESS>1 Laboratoire de Physique de
l’Ecole Normale Supérieure, ENS, Université PSL, CNRS, Sorbonne Université,
Université de Paris, F-75005 Paris, France

###### Abstract

How locally injected turbulence, spreads in space is investigated with direct
numerical simulations. We consider a turbulent flow in a long channel
generated by a forcing that is localised in space. The forcing is such that it
does not inject any mean momentum in the flow. We show that at long times a
statistically stationary state is reached where the turbulent energy density
in space fluctuates around a mean profile that peaks at the forcing location
and decreases fast away from it. We measure this profile as a function of the
distance from the forcing region for different values of the Reynolds number.
It is shown, that as the Reynolds number is increased, it converges to a
Reynolds-independent profile implying that turbulence spreads due to self-
advection and not molecular diffusion. In this limit therefore, turbulence
plays the simultaneous role of cascading the energy to smaller scales and
transporting it to larger distances. The two effects are shown to be of the
same order of magnitude. Thus a new turbulent state is reached where turbulent
transport and turbulent cascade are equally important and control its
properties.

##  1 Introduction

A drop of dye in a fluid will spread so that at long times it is uniformly
distributed in the entire space. This is not necessarily true for a turbulent
puff introduced locally in an otherwise still fluid. Turbulent energy will
also spread either by viscous diffusion or by self-advection but at the same
time will dissipate. At long times, if constantly injected, will the spreading
of turbulence be able overcome the dissipation so that turbulence spreads
throughout the domain or dissipation will limit its presence only near its
source?  The answer to this question is not a priori obvious and is
fundamental for understanding inhomogeneous turbulent flows.

Inhomogeneous flows have been the subject of various recent studies (Valente &
Vassilicos, 2011; Gomes-Fernandes et al., 2015; Alves-Portela et al., 2020;
Araki & Bos, 2022; Berti et al., 2023) that have all emphasised the effect of
inhomogeneity in the cascade process which can make it deviate from the
classical homogeneous case. In particular it has been shown that inhomogeneity
can alter the scale by scale balance of the cascade (Apostolidis et al., 2022,
2023) and change its scaling properties. Furthermore, inhomogeneity is an
indispensable ingredient of many classical canonical flows such as the
spreading of a turbulent jet (List, 1982; Carazzo et al., 2006; Ball et al.,
2012; Cafiero & Vassilicos, 2019) and the spreading of turbulence from the
boundaries in wall bounded flows (Jiménez, 2012; Gomes-Fernandes et al., 2015;
Cimarelli et al., 2016; Mollicone et al., 2018). In these cases however along
with the injection of energy there is also a mean injection of momentum.
Momentum, unlike energy, is not dissipated by viscosity and it can only be
transferred in space (by viscosity or advection) or out of the domain through
the boundaries by viscous forces. Thus, much like the example of the drop of
dye, the injected momentum will spread through out the space carrying along
energy. The same holds if the injected energy has a mean angular-momentum that
is also conserved by viscous forces. Therefore, in the case that there is mean
momentum injection the answer to the question posed in the first paragraph is
that momentum and energy will occupy the entire domain. The present work
investigates the spreading of turbulence in the absence of mean momentum and
angular momentum injection which is fundamentally different from the cases
mentioned before.

To do that we consider turbulence generated in a long triple-periodic channel.
The flow is forced homogeneously in the two short directions of the channel
and locally in the long direction. The forcing is such that no mean momentum
is injected. We study the behavior of the flow inside and outside the forcing
region at long times, measuring the energy distribution and energy fluxes in
real and spectral space. In the next section 2 we present the mathematical set
up of the system under study and define all relative quantities under
investigation. In section 3 we present the results from numerical simulations.
Conclusions are drawn in the final section 4 where directions for future
research are also discussed.

##  2  Formulation

###  2.1 Mathematical setup

A triple periodic domain of size $2\pi L\times 2\pi H\times 2\pi H$ is
considered as shown in figure 1 with $L\gg H$ being along the $x$-direction
and $x=0$ is taken to be the mid-plane of the channel.

Figure 1: The computational domain considered. The length $L$ wwas chosen to
be eight times the height $L=8H$ and $x=0$ is taken to be at the middle of the
channel. The colors indicate visualisations of the enstrophy
$(\nabla\times{\bf u})^{2}$ with red indicating high values while blue are
small values.

The flow inside the domain satisfies the Navier-Stokes equation

$\partial_{t}{\bf u}+{\bf u}\cdot\nabla{\bf u}=-\nabla P+\nu\nabla^{2}{\bf
u}+{\bf f}$ (1)

where ${\bf u}$ is the divergence free velocity field $(\nabla\cdot{\bf
u}=0)$, $P$ is the pressure, $\nu$ is the viscosity and ${\bf f}$ is the
forcing. The functional form of the forcing is given by

${\bf f}(t,{\bf x})=\left[\begin{array}[]{c}0\\\ \partial_{z}[\psi(t,{\bf
x}/\ell)-\psi(t,-{\bf x}/\ell)]\\\ \partial_{y}[\psi(t,-{\bf
x}/\ell)-\psi(t,{\bf
x}/\ell)]\end{array}\right]\exp\left[\frac{L^{2}}{\ell^{2}}\left(\cos\left(\frac{x}{L}\right)-1\right)\right]$
(2)

where $\psi(t,{\bf x}/\ell)$ is a random function including only Fourier modes
with wave-vectors $\bf k$ satisfying $0<|{\bf k}\ell|\leq 2$ and $k_{x}\neq
0$. The phases of these modes are delta correlated in time so that the mean
energy injection rate is fixed to $\mathcal{I}_{0}$. The forcing is anti-
symmetric with respect to reflections in the $x=0$ plane. As a result there is
zero momentum injection for every realisation. Furthermore the forcing
satisfies $\nabla\cdot{\bf f}=0$. For $|x|\ll L$ the exponential factor to the
right of 2 scales like $\exp(-x^{2}/\ell^{2})$ so that the forcing is limited
only around the range $|x|\sim\ell$ and zero outside. In the numerical
simulations that follow we have picked $\ell=H$ and $L=8H$ that was proven (a
posteriori) to be long enough so that the effect of the periodicity along the
x direction does not play a role.

###  2.2 Energy balance relations and fluxes in space

The primary quantity of interest in this work is the time and volume averaged
energy density of the system that is given by

$\mathcal{E}_{0}=\frac{1}{2}\left\langle\left\langle|{\bf
u}|^{2}\right\rangle_{{}_{V}}\right\rangle_{{}_{T}}$ (3)

where the angular brackets $\langle\cdot\rangle_{{}_{T}}$ stand for time
average and $\langle\cdot\rangle_{{}_{V}}$ for volume average defined as

$\langle
f\rangle_{{}_{T}}=\lim_{T\to\infty}\frac{1}{T}\int_{0}^{T}fdt\quad\mathrm{and}\quad\langle
f\rangle_{{}_{V}}=\frac{1}{V}\int_{V}f\,dx\,dy\,dz,$ (4)

with $V=(2\pi)^{3}H^{2}L$ being the system volume. The averaged rate
$\mathcal{I}_{0}$ that energy is injected is balanced by the averaged rate
$\mathcal{D}_{0}$ that energy is dissipated, leading to

$\mathcal{I}_{0}\equiv\left\langle\left\langle{\bf u}\cdot{\bf
f}\right\rangle_{{}_{T}}\right\rangle_{{}_{V}}=2\nu\left\langle\langle{\bf
S}|^{2}\rangle_{{}_{T}}\right\rangle_{{}_{V}}\equiv\mathcal{D}_{0}$ (5)

where $\bf S$ stands for the strain tensor

$S_{i,j}=\frac{1}{2}\left[\partial_{i}u_{j}+\partial_{j}u_{i}\right].$ (6)

However, neither the time averaged energy, nor its injection nor its
dissipation are uniform along the $x$ direction. It is thus appropriate to
consider the mean energy density in a subdomain of the periodic box

$\mathcal{E}(X)=\frac{1}{2}\left\langle\left\langle|{\bf
u}|^{2}\right\rangle_{{}_{T}}\right\rangle_{{}_{X}}$ (7)

where $\langle\cdot\rangle_{{}_{X}}$stands for the average confined in the
sub-box from $x=-X$ to $x=X$:

$\langle f\rangle_{{}_{X}}=\frac{1}{(2\pi H)^{2}}\int_{0}^{2\pi
H}\int_{0}^{2\pi H}\int_{-X}^{X}\langle f(x,t)\rangle_{{}_{T}}\,dx\,dy\,dz.$
(8)

For $X=\pi L$ the entire box is considered so clearly $\mathcal{E}(\pi L)=2\pi
L\,\mathcal{E}_{0}$. We also define the local energy density averaged over the
planes $x=\pm X$

$E(t,X)=\frac{1}{2(2\pi H)^{2}}\int_{0}^{2\pi H}\int_{0}^{2\pi H}|{\bf
u}(t,X,y,z)|^{2}+|{\bf u}(t,-X,y,z)|^{2}dydz.$ (9)

The two energy densities are related by $\left\langle
E(X)\right\rangle_{{T}}=\partial_{X}\mathcal{E}(X)$.

A generalisation of eq. 5 for $\mathcal{E}(X)$ can then be obtained by taking
the inner product of the Navier-Stokes equation with ${\bf u}$ time averaging
and integrating over $y,z$ and from $x=-X$ to $x=X$ to obtain:

$\mathcal{I}(X)=\mathcal{D}(X)+\mathcal{F}(X)$ (10)

where $\mathcal{I}(X)$ and $\mathcal{D}(X)$ are the energy injection rate the
energy dissipation rate within the considered volume defined respectively as:

$\mathcal{I}(X)\equiv\left\langle\left\langle{\bf f}(t,{\bf x})\cdot{\bf
u}(t,{\bf
x})\right\rangle_{X}\right\rangle_{T},\quad\mathrm{and}\quad\mathcal{D}(X)\equiv
2\nu\left\langle\left\langle|{\bf S}(t,{\bf
x})|^{2}dx\right\rangle_{X}\right\rangle_{T}.$ (11)

The third term $\mathcal{F}(x)$ is a flux that expresses the rate energy is
transferred outside the considered volume (Landau & Lifshitz, 2013). It can be
decomposed in three terms

$\mathcal{F}=\mathcal{F}_{U}+\mathcal{F}_{P}+\mathcal{F}_{\nu}$ (12)

where $\mathcal{F}_{U}$ is the energy flux due advection, $\mathcal{F}_{P}$
the flux due to pressure and $\mathcal{F}_{\nu}$ the flux due to viscosity.
They are defined explicitly as

$\displaystyle\mathcal{F}_{U}(x)$ $\displaystyle=$
$\displaystyle\frac{1}{2(2\pi H)^{2}}\left\langle\int_{x=X}u_{x}|{\bf
u}|^{2}dy\,dz-\int_{x=-X}u_{x}|{\bf u}|^{2}dy\,dz\right\rangle_{T},$ (13)
$\displaystyle\mathcal{F}_{P}(x)$ $\displaystyle=$
$\displaystyle\frac{1}{(2\pi
H)^{2}}\left\langle\int_{x=X}u_{x}Pdy\,\,dz-\int_{x=-X}u_{x}Pdy\,\,dz\right\rangle_{T}$
(14) $\displaystyle\mathcal{F}_{\nu}(x)$ $\displaystyle=$
$\displaystyle\frac{\nu}{(2\pi
H)^{2}}\left\langle\int_{{x=-X}}u_{i}\partial_{i}u_{x}+u_{i}\partial_{x}u_{i}dydz-\int_{{x=X}}u_{i}\partial_{i}u_{x}+u_{i}\partial_{x}u_{i}dydz\right\rangle_{T}$
(15)

where the integrals are taken at the two planes $x=\pm X$ and summation over
the index $i$ is assumed in the last one.

###  2.3 Energy spectra and fluxes in scale space

The fluxes above describe how energy is transported in physical space. At the
same time, energy is also transferred in scale space from large to small
scales. To quantify the energy distribution and fluxes in scale space we use
the Fourier transformed fields $\tilde{\bf u}_{\bf k}(t)$ defined by

$\tilde{\bf u}_{\bf k}(t)=\left\langle{\bf u}(t,{\bf x})\,e^{-i{\bf
k}\cdot{\bf x}}\right\rangle_{{}_{V}}\quad\mathrm{and}\quad{\bf u}(t,{\bf
x})=\sum_{{\bf k}}\tilde{\bf u}_{\bf k}(t)e^{i{\bf k}\cdot{\bf x}},$ (16)

where the inverse wavenumber $k^{-1}$ gives a natural definition of a scale.
The energy spectrum, giving the distribution of energy among scales is defined
as

$\tilde{E}(k)=\frac{1}{2}\sum_{k<|{\bf q}|<k+1}\left\langle|\tilde{\bf u}_{\bf
q}|^{2}\right\rangle_{{}_{T}}$ (17)

The energy flux gives the rate that energy flows across $k$ is defined as

$\Pi(k)=-\left\langle\left\langle{\bf u}^{<}_{k}\cdot{\bf u}\cdot\nabla{\bf
u}\right\rangle_{{}_{V}}\right\rangle_{{}_{T}}$ (18)

where ${\bf u}^{<}_{k}$ stands for the velocity field filtered so that only
wavenumbers with norm $|{\bf k}|<k$ are kept (Alexakis & Biferale, 2018;
Frisch, 1995).

###  2.4 Reynolds numbers

The Reynolds number in this system provides a measure of the strength of
turbulence is typically defined as $Re={U\ell}/{\nu}$ where $U$ is the typical
velocity of the system. In this work we are interested in the long box limit
$L\gg H$ and some care needs to be taken in order to be able to compare with
homogeneous turbulence results. If we define $U$ based on the mean energy
density $\mathcal{E}_{0}$ (given in eq. 3) then if turbulence remains
localized, $\mathcal{E}_{0}$ will approach zero in the limit $L\gg H$. Thus
defining $U$ as the root mean square (rms) value over the entire domain
$U=(2\mathcal{E}_{0})^{1/2}$ will greatly underestimate the value of $U$ close
to the forcing region. The same holds for the mean dissipation rate density
$\mathcal{D}_{0}$. To compensate for that we will define the typical velocity
$U$ and the typical dissipation rate $\epsilon$ as

$U=\sqrt{\frac{2\mathcal{E}_{0}L}{H}},\quad\epsilon=\mathcal{D}_{0}\frac{L}{H}$
(19)

The factor $L/H$ introduced makes $U$ and $\epsilon$ remain finite in the
$L/H\to\infty$ limit for localised turbulence. These definitions can be
interpreted as the rms velocity and dissipation around the forcing region.
With these definitions of $U$ and $\epsilon$ the following three Reynolds
numbers typically met in the literature are defined:

$Re_{U}\equiv\frac{UH}{\nu},\quad
Re_{\epsilon}\equiv\frac{\epsilon^{1/3}H^{4/3}}{\nu},\quad
Re_{\lambda}\equiv\frac{\sqrt{5}U^{2}}{(\nu\epsilon)^{1/2}}.$ (20)

The first one is the classical definition of the Reynolds number based of the
(re-scaled) rms velocity. The second is a Reynolds number based on the energy
injection/dissipation and is the one we control in these simulations (since it
is the energy injection rate we impose). Finally the third one is the Taylor-
Reynolds number based on the Taylor micro scale
$\lambda=U\sqrt{5\nu/\epsilon}$. The three definitions are related by

$5Re_{U}^{4}=Re_{\lambda}^{2}Re_{\epsilon}^{3}.$ (21)

and for large $Re_{U}$ it is expected that $Re_{U}\propto Re_{\epsilon}\propto
Re_{\lambda}^{2}$.

###  2.5 Numerical Setup

The Navier-Stokes equations are solved using the pseudo-spectral code ghost
(Mininni et al., 2011), that uses a 2/3 de-aliasing rule and a second order
Runge-Kuta method for the time advancement. A uniform grid was used such that
the grid spacing $\Delta x=2\pi L/N_{x}$, $\Delta y=2\pi H/N_{y}$ and $\Delta
z=2\pi H/N_{z}$ are equal where $N_{x},N_{y},N_{z}$ is the number of grid
points in each direction, with $N_{x}=8N_{y}=8N_{z}$.

The simulations were started from the ${\bf u}=0$ initial conditions and
continued until a steady state is reached for which a clear mean energy
profile can be calculated. The only exception to this rule is the highest
resolution run $N_{x}=8192$ for which the results of the $N_{x}=4096$ run were
extrapolated to a larger grid and used as initial conditions. This run was
performed for eight turn-over times that was enough to converge sign-definite
quantities (like energy) but not sign-indefinite quantities (like fluxes). A
list with the properties of all runs performed are given in table 1.

$N_{x}\times N_{y}\times N_{z}$ |  $Re_{\epsilon}$ |  $Re_{U}$ |  $Re_{\lambda}$
---|---|---|---
$64\times 64\times 512$ | 2.0 | 1.7 | 2.2
$64\times 64\times 512$ | 4.0 | 4.5 | 5.8
$64\times 64\times 512$ | 10.0 | 15.1 | 16.8
$64\times 64\times 512$ | 20.0 | 34.7 | 31.3
$64\times 64\times 512$ | 40.0 | 78.1 | 54.1
$128\times 128\times 1024$ | 110 | 217 | 92.1
$256\times 256\times 2048$ | 230 | 502 | 161
$512\times 512\times 4096$ | 500 | 1165 | 270
$1024\times 1024\times 8192$ | 1250 | 2990 | 447
Table 1:  Resolution and values of the Reynolds numbers
$Re_{U},Re_{\epsilon},Re_{\lambda}$ achieved in the numerical simulations.

##  3  Results

We begin with the top panel of figure 2 that shows the energy density $E(t,X)$
for $Re_{\epsilon}=500$ for different times. The black dashed line shows the
forcing profile that is limited to $|X|/(2\pi H)\lesssim 1/2$.

Figure 2:  Top panel: The energy density $E(t,X)$ for different times. Bottom
panel: The time averaged energy density $\langle E(X)\rangle_{{}_{T}}$ at
steady state for different values of $Re_{\epsilon}$ in the entire domain. the
dashed line indicates the forcing amplitude as a function of $X$. The inset
shows the same data in log-log scale. The same color index is used to mark
$Re_{\epsilon}$ in all subsequent figures.

Energy spreads away from the forcing region but at late times it fluctuates
around a mean profile shown by the red line. Thus already at this stage it can
be testified that energy does not spread in the entire box and it remains
close to the forcing region. This mean profile is shown in the bottom panel of
the same figure for different values of $Re_{\epsilon}$. The different colors
indicate the different values of the Reynolds number achieved as marked in the
legend. The same colors are used for all subsequent figures. The peak of the
local energy density lies close to the forcing region and decays fast away
from it. The energy far away from the forcing at $|X|/(2\pi H)\simeq 4$
remains very small such that $E(8\pi H)/E(0)\lesssim 10^{-6}.$ In the
remaining of this section we will try unravel the processes of this
localisation and the implications for the system behavior.

Before continuing with spatial properties of our flow we perform some standard
benchmark analysis often used in homogeneous turbulence. Figure 3 shows the
scaling of global measures as a function of the Reynolds number. The left
panel shows the relation between the different Reynolds numbers where the
scaling $Re_{U}\propto Re_{\epsilon}\propto Re_{\lambda}^{2}$ that holds for
large $Re$ is verified. In right panel of figure 3 we show the non-dimensional
dissipation rate (or drag coefficient) $C_{\epsilon}$ defined here as:

$C_{\epsilon}=\frac{\epsilon H}{U^{3}},$ (22)

that expresses the rate energy is dissipated non-dimensionalized by the
amplitude of the fluctuations. It is a corner-stone conjecture of homogeneous
and isotropic turbulence theory that $C_{\epsilon}$ obtains a finite and
$Re$-independent value at large $Re$. The present data indicate that at large
$Re_{\lambda}$, $C_{\epsilon}$ appears to converge to a
$Re_{\lambda}$-independent value but quite slowly. Only the largest values of
$Re_{\lambda}\gtrsim 270$ indicate the possibility that such a plateau is
reached with a value of $C_{\epsilon}\simeq 0.06$ that is rather small. In
homogeneous and isotropic simulations such a plateau is reached after
$Re_{\lambda}\sim 100$ and at a value much larger $C_{\epsilon}\simeq 0.5$
(Kaneda et al., 2003). This reflects that localized turbulence is affected by
the the additional freedom to expand in a larger region suppressing possibly
its efficiency to cascade energy to the smaller scales.

Figure 3:  Left: Relation between the different Reynolds numbers
$Re_{U},Re_{\epsilon},Re_{\lambda}$. Right: The normalized dissipation rate
$C_{f}$ as a function of $Re_{\lambda}$.

Figure 4:  Left: The energy spectra $\tilde{E}(k)$ for the different
$Re_{\lambda}$ examined. Right: The energy fluxes $\Pi(k)$ for the same runs.

Figure 4 examines spectral properties of the flow. In the left panel we plot
the energy spectra for the different values of $Re$. The spectra show similar
behavior with homogeneous turbulence flows. As the Reynolds number is
increased more scales are excited and a power-law spectrum starts to form with
exponent close to the Kolmogorov prediction ${\tilde{E(}}k)\propto k^{-5/3}$.
In the right panel of figure 4 the energy fluxes in Fourier space are plotted.
The energy fluxes increase with $Re$ until for the largest $Re$s attained a
constant flux range has began to form. It is worth noting that this constant
flux region is obtained at much larger $Re$ than what is observed in
homogeneous turbulence simulations reflecting once again a delay in obtaining
a $Re$-independent scaling due to the effect of spreading.

Returning to the spatial properties of the flow and the energy density profile
we note that as the Reynolds number is increased the energy increases and also
spreads at larger distances. At very large values of $Re_{\lambda}$ the energy
profile appears to converge to a $Re$ independent profile. This implies that
at large $Re$ the energy profile is determined by the self-spreading of eddies
due to turbulent advection and not by viscous processes. The fast drop of
$E(X)$ can be either an exponential $E(x)\propto\exp(-\alpha x)$ of or fast
power law $E(x)\propto|X|^{-6}$ (see inset). The present data can not exclude
either option. We point out that since the energy density drops very fast also
the local Reynolds number (defined using a local rms velocity) is also
decreasing. So it is hard to obtain a large $Re$ behavior in the outer region
$|X|\gg H$.

The fact that the energy density reaches a $Re$-independent profile is not a
trivial result. It reflects a balance between the rate energy is transported
to larger values of $|x|$ and the rate energy cascades to the small scales. If
the cascade process was weaker than the real-space transport then at the
$Re\to\infty$ limit energy would reach the entire domain. On the contrary if
the real-space transport was weaker no energy would be found outside the
forcing region in the same limit. In other words turbulent diffusion and
turbulent dissipation must be of the same order.

, ,

Figure 5:  The different energy fluxes in real space as indicated in the
legend for three different values of $Re_{\epsilon}=4,40,500$.

To quantify this assertion we look at the fluxes at real space. In figure 5 we
plot $\mathcal{F}_{i}$ for three different values of $Re$ varying from the
laminar to the turbulence case. The black line shows the total flux, the blue
line the flux due to velocity fluctuations, the green line the flux due to
pressure and the red line the flux due to viscosity. The magenta line shows
the difference between $\mathcal{I}(x)$ and $\mathcal{D}(x)$. A comparison
between the back and magenta lines verifies the relation 12. The small
differences that are observed are due to insufficient time averaging that is
more pronounced in the large resolution runs. A few observations need to
follow. For small $Re$ the energy flux is dominated by viscosity with pressure
also playing a significant part. The flux due to the velocity fluctuations
have a negative sign. As the Reynolds is increased the role of the velocity
fluctuations becomes more dominant transferring outwards energy. The transfer
due to viscosity diminishes while the transfer due to pressure also takes
negative values. At the largest $Re$ almost the entire flux is dominated by
the velocity fluctuations with the pressure flux being weaker and positive in
the forcing region and negative away from it.

, ,

Figure 6:  The dissipation rate $\mathcal{D}(X)$ (left panel), the energy flux
$\mathcal{F}(X)$ (center panel) for different values of $Re$. The right panel
compares the largest $Re_{\epsilon}=500$ for which the fluxes were measured.
The dashed line indicates $\mathcal{I}(X)$.

Finally to compare the two dominant processes away from the forcing region the
turbulent dissipation and the turbulent diffusion we plot in figure 6 the
dissipation rate $\mathcal{D}(X)$ in the left panel and the total flux
$\mathcal{F}(X)$ on the center panels for all $Re$. The right panel compares
the two, for the largest value of $Re$ for which the fluxes were calculated
$Re_{\epsilon}=500$. The black dashed lines indicates $\mathcal{I}(X)$ that is
the same for all $Re$. As the Reynolds is increased the dissipation is
decreased while the flux is increased. For the largest $Re$ at the peak of the
flux around $X\simeq 0.15(2\pi H)$ the two processes become of approximately
equal marking that the two processes turbulent dissipation and the turbulent
diffusion are of the same order.

##  4  Conclusions

The present work has demonstrated that locally forced turbulence will not
spread throughout the domain provided that there is no mean injection of
linear or angular momentum. It will remain localised forming an energy density
profile that is $Re$ independent in the large $Re$ limit. Away from the
forcing region the two dominant effects are turbulent dissipation and
turbulent diffusion that were found to be of the same order. The exact
functional form of the energy profile could not be determined from the present
simulations. Theoretical investigations and modeling could give further
insight to this problem.

To expand the understanding of the two involved processes, turbulent diffusion
and turbulent dissipation, a simultaneous scale-space and real-space analysis
would be required either by introducing local smoothing (Germano, 1992; Aluie
& Eyink, 2009; Eyink & Aluie, 2009; Alexakis & Chibbaro, 2020) or two point
analysis and the Kármán–Howarth‐Monin–Hill (KHMH) equation (Hill, 2001, 2002).
The latter has been used recently to study boundary driven flows (Apostolidis
et al., 2022, 2023) and wakes Chen & Vassilicos (2022); Chen et al. (2021)
where the role the inhomogeneous energy injection from the mean flow was
emphasised. In the present flow, there is no mean flow and the primary terms
in balance are the inter-scale transfer rate and turbulent transport in
physical space, both of which are forcing and viscosity independent. Thus a
new state of turbulence is present where two inertial effects, the energy flux
in scale space and in real space, compete. The fact that these two dominant
processes are viscosity-independent makes their modeling particular difficult
as there is no simplifying limit where one term will dominate over the other.
Careful parametrization would be required so that the correct energy profile
is captured.

Finally we would like to add that the present study was limited to a triple
periodic channel flow limiting the spreading in only one direction. Its
extension to larger domains where turbulence can spread in two or in all three
directions is far from trivial and would need to be examined separately. Here
experimental investigations would become much more beneficial than numerical
simulations.

[Funding]This work was granted access to the HPC resources of GENCI-TGCC &
GENCI-CINES (Project No. A0130506421). This work has also been supported by
the Agence nationale de la recherche (ANR DYSTURB project No.
ANR-17-CE30-0004).

[Declaration of Interests]The authors report no conflict of interest.

## References

  * Alexakis & Biferale (2018) Alexakis, Alexandros & Biferale, Luca 2018 Cascades and transitions in turbulent flows. Physics Reports 767, 1–101.
  * Alexakis & Chibbaro (2020) Alexakis, Alexandros & Chibbaro, Sergio 2020 Local energy flux of turbulent flows. Physical Review Fluids 5 (9), 094604\.
  * Aluie & Eyink (2009) Aluie, Hussein & Eyink, Gregory L 2009 Localness of energy cascade in hydrodynamic turbulence. ii. sharp spectral filter. Physics of Fluids 21 (11).
  * Alves-Portela et al. (2020) Alves-Portela, F, Papadakis, G & Vassilicos, JC 2020 The role of coherent structures and inhomogeneity in near-field interscale turbulent energy transfers. Journal of Fluid Mechanics 896, A16.
  * Apostolidis et al. (2022) Apostolidis, Argyrios, Laval, Jean-Philippe & Vassilicos, JC 2022 Scalings of turbulence dissipation in space and time for turbulent channel flow. Journal of Fluid Mechanics 946, A41.
  * Apostolidis et al. (2023) Apostolidis, Argyrios, Laval, Jean-Philippe & Vassilicos, JC 2023 Turbulent cascade in fully developed turbulent channel flow. To appear in Journal of Fluid Mechanics .
  * Araki & Bos (2022) Araki, Ryo & Bos, Wouter JT 2022 Inertial range scaling of inhomogeneous turbulence. arXiv preprint arXiv:2210.14516 .
  * Ball et al. (2012) Ball, CG, Fellouah, H & Pollard, A 2012 The flow field in turbulent round free jets. Progress in Aerospace Sciences 50, 1–26.
  * Berti et al. (2023) Berti, S, Boffetta, G & Musacchio, S 2023 Mean flow and fluctuations in the three-dimensional turbulent cellular flow. Physical Review Fluids 8 (5), 054601.
  * Cafiero & Vassilicos (2019) Cafiero, Gioacchino & Vassilicos, JC 2019 Non-equilibrium turbulence scalings and self-similarity in turbulent planar jets. Proceedings of the Royal Society A 475 (2225), 20190038\.
  * Carazzo et al. (2006) Carazzo, Guillaume, Kaminski, Edouard & Tait, Stephen 2006 The route to self-similarity in turbulent jets and plumes. Journal of Fluid Mechanics 547, 137–148.
  * Chen et al. (2021) Chen, JG, Cuvier, C, Foucaut, J-M, Ostovan, Y & Vassilicos, JC 2021 A turbulence dissipation inhomogeneity scaling in the wake of two side-by-side square prisms. Journal of Fluid Mechanics 924, A4.
  * Chen & Vassilicos (2022) Chen, JG & Vassilicos, John Christos 2022 Scalings of scale-by-scale turbulence energy in non-homogeneous turbulence. Journal of Fluid Mechanics 938, A7.
  * Cimarelli et al. (2016) Cimarelli, Andrea, De Angelis, Elisabetta, Jimenez, Javier & Casciola, Carlo Massimo 2016 Cascades and wall-normal fluxes in turbulent channel flows. Journal of Fluid Mechanics 796, 417–436.
  * Eyink & Aluie (2009) Eyink, Gregory L & Aluie, Hussein 2009 Localness of energy cascade in hydrodynamic turbulence. i. smooth coarse graining. Physics of Fluids 21 (11).
  * Frisch (1995) Frisch, Uriel 1995 Turbulence: the legacy of AN Kolmogorov. Cambridge university press.
  * Germano (1992) Germano, Massimo 1992 Turbulence: the filtering approach. Journal of Fluid Mechanics 238, 325–336.
  * Gomes-Fernandes et al. (2015) Gomes-Fernandes, R, Ganapathisubramani, B & Vassilicos, JC 2015 The energy cascade in near-field non-homogeneous non-isotropic turbulence. Journal of Fluid Mechanics 771, 676–705.
  * Hill (2001) Hill, Reginald J 2001 Equations relating structure functions of all orders. Journal of Fluid Mechanics 434, 379–388.
  * Hill (2002) Hill, Reginald J 2002 Exact second-order structure-function relationships. Journal of Fluid Mechanics 468, 317–326.
  * Jiménez (2012) Jiménez, Javier 2012 Cascades in wall-bounded turbulence. Annual review of fluid mechanics 44, 27–45.
  * Kaneda et al. (2003) Kaneda, Yukio, Ishihara, Takashi, Yokokawa, Mitsuo, Itakura, Ken’ichi & Uno, Atsuya 2003 Energy dissipation rate and energy spectrum in high resolution direct numerical simulations of turbulence in a periodic box. Physics of Fluids 15 (2), L21–L24.
  * Landau & Lifshitz (2013) Landau, Lev Davidovich & Lifshitz, Evgenii Mikhailovich 2013 Fluid Mechanics: Landau and Lifshitz: Course of Theoretical Physics, Volume 6, , vol. 6. Elsevier.
  * List (1982) List, EJ 1982 Turbulent jets and plumes. Annual review of fluid mechanics 14 (1), 189–212.
  * Mininni et al. (2011) Mininni, Pablo D, Rosenberg, Duane, Reddy, Raghu & Pouquet, Annick 2011 A hybrid mpi–openmp scheme for scalable parallel pseudospectral computations for fluid turbulence. Parallel computing 37 (6-7), 316–326.
  * Mollicone et al. (2018) Mollicone, J-P, Battista, F, Gualtieri, P & Casciola, CM 2018 Turbulence dynamics in separated flows: the generalised kolmogorov equation for inhomogeneous anisotropic conditions. Journal of Fluid Mechanics 841, 1012–1039.
  * Valente & Vassilicos (2011) Valente, PC & Vassilicos, John Christos 2011 The decay of turbulence generated by a class of multiscale grids. Journal of Fluid Mechanics 687, 300–340.