# Multitask Learning for Time Series Data
with 2D Convolution

Chin-Chia Michael Yeh, Xin Dai, Yan Zheng, Junpeng Wang, Huiyuan Chen,
Yujie Fan, Audrey Der†, Zhongfang Zhuang, Liang Wang, and Wei Zhang  Visa
Research, University of California, Riverside†
<EMAIL_ADDRESS>
###### Abstract

Multitask learning (MTL) aims to develop a unified model that can handle a set
of closely related tasks simultaneously. By optimizing the model across
multiple tasks, MTL generally surpasses its non-MTL counterparts in terms of
generalizability. Although MTL has been extensively researched in various
domains such as computer vision, natural language processing, and
recommendation systems, its application to time series data has received
limited attention. In this paper, we investigate the application of MTL to the
time series classification (TSC) problem. However, when we integrate the
state-of-the-art $1D$ convolution-based TSC model with MTL, the performance of
the TSC model actually deteriorates. By comparing the $1D$ convolution-based
models with the Dynamic Time Warping (DTW) distance function, it appears that
the underwhelming results stem from the limited expressive power of the $1D$
convolutional layers. To overcome this challenge, we propose a novel design
for a $2D$ convolution-based model that enhances the model’s expressiveness.
Leveraging this advantage, our proposed method outperforms competing
approaches on both the UCR Archive and an industrial transaction TSC dataset.

###### Index Terms:

time series, multitask learning, convolutional neural network

##  I Introduction

Multitask learning (MTL) is a learning strategy that enhances a model’s
generalizability by leveraging shared knowledge from closely related tasks [1,
2, 3]. This approach has demonstrated significant success in domains such as
computer vision [3] and natural language processing [2]. Time series data are
prevalent in various domains [4], yet there is a surprising lack of MTL
research focused on this type of data, despite its potential for
transferability to time series analysis. Consequently, this paper aims to
investigate the application of MTL to time series classification (TSC). Our
study specifically concentrates on the widely adopted MTL technique known as
hard parameter sharing. In this approach, different tasks share a common sub-
network as a feature extractor, while each task maintains its own task-
specific output layer. Hard parameter sharing is considered one of the most
efficient and robust methods for implementing MTL, frequently serving as a
baseline in many studies [5, 6, 7, 8, 9].

We can easily combine hard parameter sharing with one of the strong deep
learning TSC baselines, such as ResNet [10, 11]. However, when we combine hard
parameter sharing with ResNet, we observe that 15 out of 25 tasks in our time
series MTL benchmark datasets perform worse than their non-MTL counterparts
(see Section IV-B for details). Furthermore, we compare it with a simpler
baseline: the 1-nearest neighbor algorithm with dynamic time warping
distance111The 1NN+DTW method is an effective approach for multiple TSC
datasets/tasks [12]. (1NN+DTW) [13]. Surprisingly, we find that even 1NN+DTW
can outperform ResNet with MTL in terms of overall performance in our
benchmark. This suggests that the warping mechanism (DTW) can easily capture
some simple yet potentially useful features for MTL, which are challenging for
the $1D$ convolutional layers of the ResNet baseline model to learn.

To comprehend why $1D$ convolution encounters difficulties in learning the
warping mechanism, it is necessary to explain the computation process of the
DTW distance. Given two time series, the first step involves computing the
pair-wise distance matrix between all pairs of elements in the two input time
series. Next, a recursive function is employed to process the pair-wise
distance matrix, yielding the distance value. The inductive bias of $1D$
locality from the $1D$ convolution operation poses difficulties in effectively
capturing both the pair-wise distance computation and the recursive function.
To address this limitation, we propose a novel $2D$ convolutional TSC model
that is specifically designed to better capture the warping mechanism compared
to the baseline ResNet [10, 11]. This is achieved by 1) explicitly processing
the pair-wise distance matrix and 2) employing $2D$ convolutional layers to
learn diverse recursive functions. Please note that since the baseline ResNet
utilizes $1D$ convolutional layers, we will refer to it as ResNet1D to
indicate the baseline method. Similarly, we will use ResNet2D to denote the
proposed method, which incorporates $2D$ convolutional layers.

Figure 1:  The proposed ResNet2D model can approximate DTW distance more
accurately under various warping constraints compared to the baseline ResNet1D
model.

To provide empirical evidence supporting the need for a new model to learn the
warping mechanism, we conducted a comparative experiment to assess the
capability of different models in approximating the DTW distance function.
Both the baseline ResNet1D model and the proposed ResNet2D model were trained
to approximate the DTW distance function under various warping constraints
(refer to Section IV-A for detailed information). We measured the error
between the predicted distance from the models and the actual distance output
by the distance function. As depicted in Figure 1, the proposed ResNet2D model
outperformed the baseline ResNet1D model in approximating the DTW functions.
Based on this observation, we extended the proposed ResNet2D model from the
distance learning task to the MTL TSC task. When combining the proposed
ResNet2D model with hard parameter sharing MTL, it achieved the highest
accuracy across multiple tasks compared to alternative approaches.

This paper makes the following contributions::

  * •

We demonstrate that combining hard parameter sharing with a conventional deep
learning model (ResNet1D) is ineffective for the TSC problem.

  * •

We propose a novel $2D$ convolution-based neural network (ResNet2D) that can
learn the warping mechanism and achieves superior performance in MTL with hard
parameter sharing compared to the baseline methods.

  * •

We validate the effectiveness of the proposed ResNet2D model on both publicly
available MTL TSC datasets and an industrial MTL TSC dataset.

##  II Related Work

The two primary research areas relevant to our paper are multitask learning
(MTL) and time series classification (TSC). In this section, we provide a
brief literature review on both of research areas.

###  II-A Multitask Learning

MTL is a learning paradigm that aims to enhance the generalizability of
machine learning models [14, 1]. It has gained significant attention in
various domains, including computer vision, natural language processing, and
recommendation systems [14, 1]. In recent years, efforts have been made to
apply MTL to time series data [15, 16, 17, 18]. For instance, Cirstea et al.
[15] proposed a hybrid convolutional-recurrent neural network for correlated
time series forecasting, which incorporated a reconstruction task as a
regularization term to improve generalization. Chen et al. [16] introduced a
representation learning method that jointly learns time series classification
and retrieval. However, these approaches assume that all tasks share a single
dataset [15, 16, 17, 18]. In other words, these methods can only be applied to
homogeneous-feature MTL problems [14]. To the best of our knowledge, our
proposed framework represents the first work addressing general MTL on time
series data.

The primary focus of MTL research lies in addressing the conflicts that arise
among the objectives of different tasks [14]. Various studies have attempted
to tackle this challenge by mitigating the conflicts in gradients associated
with different losses [19, 20]. For instance, Chen et al. [19] proposed a
method to balance the gradients of multiple losses in a multi-task neural
network. Sener et al. [20] treated MTL as a multi-objective optimization
problem and aimed to find a Pareto optimal solution. Other research endeavors
have focused on discovering effective parameter-sharing structures for
multiple tasks [21, 22, 9]. For example, Lu et al. [21] introduced a bottom-up
searching algorithm to identify a suitable sharing structure for a given
dataset. Guo et al. [22] and Sun et al. [9] proposed approaches to learn the
sharing structure directly from the dataset.

Soft parameter sharing MTL, which involves sharing feature maps instead of
parameters, is another approach explored in computer vision research [5, 6,
7]. For instance, Misra et al. [5] proposed the Cross-stitch network, while
Gao et al. [6] introduced the NDDR-CNN. These methods maintain task-specific
networks but integrate their feature maps at intermediate layers [5, 6].
Furthermore, Gao et al. [7] proposed MTL-NAS, a method that utilizes network
architecture search techniques to explore potential sharing links between
feature maps at various intermediate layers. Although our paper focuses on the
basic approach of hard parameter sharing MTL, which has demonstrated
robustness in preventing overfitting and serves as a common baseline in state-
of-the-art studies [5, 6, 7, 8, 9], the ideas presented in the aforementioned
studies offer interesting avenues for future research.

###  II-B Time Series Classification

Over the past decades, numerous algorithms have been proposed for TSC,
encompassing a wide range of approaches [12, 11]. The spectrum includes simple
yet effective methods such as the nearest neighbor classifier utilizing the
DTW distance [4, 12], as well as complex and powerful ensemble techniques like
HIVE-COTE 2.0 [23, 12, 24]. Inspired by the achievements of deep learning
models in computer vision and natural language processing domains [25, 26],
there have been numerous endeavors to adapt deep learning-based approaches for
TSC problems [11]. For instance, Cui et al. [27] proposed the multi-scale
convolutional neural network (MCNN) model, which classifies input time series
by employing multiple temporal scaling and smoothing operations with a
convolutional neural network. Over the years, various variants of
convolutional neural networks, including the Time Le-Net model[28], Time-CNN
model [29], FCN model [10], and ResNet model [10], have been proposed for TSC.
Among these deep learning-based TSC models, ResNet continues to serve as a
strong baseline, as demonstrated by extensive evaluations conducted by Ismail
et al. [11]. Consequently, our paper aims to enhance the performance of the
ResNet model specifically for the TSC problem within the MTL framework.

##  III Methodology

In this section, we present the design of two models: the baseline ResNet1D
model and the proposed ResNet2D model. These models are used for both distance
function approximation and classification tasks, and we provide an explanation
of how each model is employed in each type of problem. Furthermore, we discuss
the optimization process employed for training these models.

###  III-A ResNet1D Architecture

Figure 2 illustrates the design of the baseline ResNet1D model and how it is
used in our experiments. The model design is inspired by the ResNet
architecture proposed in [10].

Figure 2:  The figure illustrates the design of the baseline ResNet1D model
and demonstrates how it is utilized for both distance learning and MTL
experiments.

The building block of the ResNet1D model is depicted in Figure 2a. Each $1D$
convolutional layer (conv) is annotated with its corresponding filter size.
The stride size for all conv layers is set to 1. In cases where the input and
output dimensions of the building block differ, a $1D$ conv layer with a
filter size of 1 is employed in the skip connection to align the input
dimension with the output dimension.

The design of the ResNet1D model is depicted in Figure 2b. In the figure, each
block represents the building block, as shown in Figure 2a. The arrow
notation, such as $1\rightarrow 64$ in the first block, indicates the input
and output dimensions. Since our model is specifically designed for univariate
time series, the input time series has a dimension of 1. After passing through
the building blocks and the global average pooling layer, the output is a
vector of size 128. The global average pooling operation is applied along the
temporal dimension to obtain this output.

To utilize the ResNet1D model for approximating distance functions, we
incorporate it as a component within a Siamese neural network architecture
[30], as depicted in Figure 2c. In this setup, we introduce a linear layer
after the global average pooling layer to further process the representations
generated by the ResNet1D model. The output dimension of the linear layer is
set to 128. To compute the output distance, we calculate the $l2$ distance
between the two 128-sized vectors obtained from the linear layer.

When applying the ResNet1D model to MTL problems, we employ multiple linear
layers to compute the logits for each task, as illustrated in Figure 2d.
Following the conventional hard parameter sharing approach, the parameters of
the ResNet1D model are shared among all tasks, while the parameters of the
linear layers preceding the softmax layers are task-specific.

Figure 3:  The figure showcases the design of the proposed ResNet2D model and
illustrates how the model is employed for both distance learning and MTL
experiments.

###  III-B ResNet2D Architecture

In this section, we first present the proposed ResNet2D model (depicted in
Figure 3a and Figure 3b) and then discuss its applications in different
scenarios. The design of the ResNet2D model may appear unconventional for
processing time series data initially. However, the rationale behind
incorporating $2D$ convolutional layers becomes clearer when we demonstrate
how the model is employed in distance function approximation and MTL (refer to
Figure 3c and Figure 3d).

Figure 3a presents the building block of the ResNet2D, while Figure 3b
showcases the overall architecture of the ResNet2D. The design of our ResNet2D
model draws inspiration from the original ResNet designed for computer vision
tasks [25]. In our model, each input instance is represented as a tensor with
dimensions $n\times m\times k$, where $n$ and $m$ refer to the width and
height, respectively, while $k$ represents the depth or feature dimension. The
specific values of $n$, $m$, and $k$ may vary depending on the dataset and
application at hand. Our architecture is composed of 8 blocks, and following
each block, we incorporate a $3\times 3$ conv layer with a stride size of 2 to
reduce the width and height of the intermediate representation by half.

We employ the architecture depicted in Figure 3c for distance learning tasks.
In this setup, if the length of the first input time series is denoted as $n$,
and the length of the second time series as $m$, the pairwise distance matrix
between the two time series can be represented as a tensor with dimensions
$n\times m\times 1$. This matrix captures the Euclidean distance between each
data point in one time series and every data point in the other time series.
To construct the distance learning models, we utilize the ResNet2D model shown
in Figure 3b, setting the value of $k$ to $1$. The pairwise distance matrix
plays a vital role as it provides essential information for computing the
distance between two time series across all possible warping paths, enabling
the ResNet2D model to learn the warping mechanism. For instance, when $n=m$,
the sum of the matrix diagonal corresponds to the Euclidean distance between
the two time series. By mimicking the recurrent function of the DTW algorithm
using the ResNet2D model, we can compute the DTW distance between the two time
series based on the matrix.

The MTL variant of the proposed ResNet2D model is depicted in Figure 3d. In
this configuration, we have a single input time series, unlike the distance
learning variant. To compute the pairwise distance matrix, we employ $k$
learnable time series templates of length $m$. Therefore, if the input time
series has a length of $n$, the output of the pairwise distance operation is a
tensor with dimensions $n\times m\times k$. In our design, we use 64 templates
(i.e., $k=64$), each with a length of 512 (i.e., $m=512$). Similar to the
design shown in Figure 2d, the model also includes multiple parallel output
layers for each task. Each output layer has its independent set of parameters
specific to its task, while the parameters associated with the core ResNet2D
model and the templates are shared across different tasks.

To gain insight into how the ResNet2D model can learn the warping mechanism,
let us revisit the algorithms for computing the DTW distance [13] and the
soft-DTW distance [31]. Both distances can be computed by employing dynamic
programming recursion on the pairwise distance matrix between two time series,
as depicted in Algorithm 1. For the DTW distance, the recursion function
$\textsc{Recursion}(x_{0},x_{1},x_{2})$ is defined as
$\textsc{Min}(x_{0},x_{1},x_{2})$. On the other hand, for the soft-DTW
distance, the recursion function $\textsc{Recursion}(x_{0},x_{1},x_{2})$ is
defined as $-\gamma\sum_{i=1}^{3}e^{-x_{i}/\gamma}$, where $\gamma$ is a
hyper-parameter for controlling the behavior of the soft-DTW distance.

Algorithm 1 DTW/soft-DTW Distance

1Input: pairwise distance matrix $X\in\mathbb{R}^{n\times m}$

2function Distance($X$)

3 for $i$ in $[0,\cdots,n)$ do

4 for $j$ in $[0,\cdots,m)$ do

5 $X[i,j]\leftarrow X[i,j]+\textsc{ Recursion}($
$X[i-1,j],X[i,j-1],X[i-1,j-1])$

6 return $X[n-1,m-1]$

The core ResNet2D model plays a similar role to the
$\textsc{Recursion}(\cdot)$ function in both distance algorithms. With its
expressive power, the ResNet2D model is capable of learning various recursion
functions, thereby capturing different warping mechanisms. As demonstrated in
Section IV-A, the proposed ResNet2D model effectively approximates different
variants of the DTW distance functions.

###  III-C Optimization

The models are optimized using the mini-batch stochastic gradient descent
(SGD) algorithm, specifically employing AdamW [32] as the optimizer. For the
distance function approximation experiments, we utilize the mean squared error
between the ground truth and the model’s prediction as the loss function. In
this case, the standard mini-batch SGD can be directly applied without any
modifications. In the time series classification (TSC) experiments, we employ
cross-entropy as the loss function. However, due to potential variations in
the number of time series and the lengths of each time series across different
tasks, we have made modifications to the standard mini-batch SGD algorithm to
accommodate these differences. The modified mini-batch SGD algorithm is
outlined in Algorithm 2.

Algorithm 2 Mini-batch SGD for MTL

1Input: datasets $\mathbf{D}$, batch size $n_{\text{batch}}$, number of epoch
$n_{\text{epoch}}$, model $M$

2Output: model $M$

3function Train($\mathbf{D},n_{\text{batch}},n_{\text{epoch}},M$)

4 $n_{\text{iter}}\leftarrow 0$

5 for each $D$ in $\mathbf{D}$ do

6 $n_{D}\leftarrow|D|\mathbin{/}n_{\text{batch}}$

7 $n_{\text{iter}}\leftarrow\textsc{ Max}(n_{\text{iter}},n_{D})$

8 for $i$ in $[0,\cdots,n_{\text{epoch}})$ do

9 for each $D$ in $\mathbf{D}$ do

10 $\textsc{Shuffle}(D)$

11 for $j$ in $[0,\cdots,n_{\text{iter}})$ do

12 $\textsc{Shuffle}(\mathbf{D})$ $\triangleright$ shuffle the order of task

13 for each $D$ in $\mathbf{D}$ do

14 $X,Y\leftarrow\textsc{ GetNextMiniBatch}(D)$

15 if reaching the end of $D$ then

16 $\textsc{Shuffle}(D)$

17 restart the mini-batch counter for $D$

18 $M\leftarrow\textsc{ UpdateModel}(X,Y,M)$

19 return $M$

The main distinction between Algorithm 2 and the standard mini-batch SGD
algorithm lies in the determination of the number of iterations for each
epoch, which is based on the task with the largest dataset (refer to lines 3
to 5). When the task with a smaller dataset reaches its last example during
the construction of mini-batches, the dataset is shuffled, and the mini-batch
counter is reset (see lines 13 to 15). This implies that some or all examples
from a task with a smaller dataset may be sampled more than once within a
single epoch, whereas examples from the task with the largest dataset are
sampled only once per epoch.

It is important to note that all examples within a batch are drawn from the
dataset associated with the same task. This design decision stems from the
varying lengths of time series across different tasks. By enforcing this
restriction, we guarantee that all time series within a mini-batch share the
same length, facilitating efficient model updates. Furthermore, to mitigate
bias towards any specific task during the updates, the order of tasks is
shuffled for each iteration (see line 10).

##  IV Experiment

We present three sets of experiments in this section. In the first set of
experiments, we explore the difference between the proposed ResNet2D model and
the baseline ResNet1D model in terms of model expressiveness, focusing on the
DTW distance approximation problem. Next, we create seven MTL datasets from
the publicly available UCR archive [4] for the second set of experiments.
Using these datasets, we compare the performance of the proposed ResNet2D
model with the baselines. In the last set of experiments, we demonstrate the
effectiveness of the proposed ResNet2D model in solving real-world time series
classification problems.

###  IV-A Distance Function Approximation

One way to compare the expressiveness of different models is by assessing
their ability to approximate a complex function. In this set of experiments,
we train both the baseline ResNet1D model (see Figure 2c) and the proposed
ResNet2D model (see Figure 3c) to approximate the DTW distance function under
various warping constraints.

We conducted the experiments using synthetic time series data. To train the
models, we generated 100,000 pairs of random walk time series as input data
and computed the DTW distances under different warping constraints as the
target values. For the validation dataset, we generated 1,000 pairs of random
walk time series and their corresponding ground truth distances for model
selection. Additionally, we generated another 1,000 pairs of random walk time
series and their associated ground truth distances for testing purposes. Both
models were trained for 200 epochs, and the best model was selected based on
the error between the predicted distances and the ground truth distances on
the validation data. The box plots in Figure 1 depict the errors of different
methods evaluated on the test dataset. Overall, the proposed ResNet2D model
exhibits lower errors compared to the baseline ResNet1D model, indicating its
superior ability to approximate the DTW distance function.

Figure 4:  The images display the pixel-importance maps for randomly selected
pairs of test time series in the proposed ResNet2D model. On each importance
map, we superimpose the ground truth warping path as a red line. The ResNet2D
model can learn the warping mechanism, enabling it to approximate DTW. The
color gradient from dark blue to orange represents the importance values, with
darker shades indicating lower values and brighter shades indicating higher
values.

To gain insights into the superb performance of the ResNet2D model, we employ
the pixel perturbation technique [33, 34] to generate pixel-importance maps
for the pairwise distance matrices of various test time series pairs. The
pixel perturbation method evaluates the importance of individual pixels by
measuring the change in output before and after disabling specific pixels [33,
34]. Figure 4 illustrates the resulting importance maps, where the red lines
represent the ground truth warping paths overlaid on the maps. Notably, the
pixels surrounding the ground truth warping paths exhibit greater importance
compared to other parts of the matrix. These pixel-importance maps provide
evidence that the proposed ResNet2D model learns the warping mechanism.

###  IV-B UCR Archive

To compare different MTL models, we create seven MTL datasets using the
following steps. First, we select 25 datasets from the UCR archive [4] that
share common sources with others. Subsequently, we organize the 25 tasks into
seven distinct datasets, grouping them based on the source of the associated
time series. These groupings are importance for MTL, as they enable the
meaningful sharing of knowledge among tasks within the same dataset. For
instance, all Electrocardiography (ECG) tasks are consolidated into a single
dataset. To ensure sufficient training samples for each task, we partition the
dataset associated with each task into three subsets: 60% for training, 20%
for validation, and 20% for testing purposes. The 25 tasks and their groupings
are provided in Table I.

TABLE I:  The 25 selected tasks are organized into seven datasets. The
performance is evaluated using accuracy, and the number within the bracket
indicates the rank of each method among the five tested methods. Among the
tested methods, the proposed ResNet2D w/ MTL outperformed the alternatives.

Dataset | Task | 1NN-DTW | ResNet1D | ResNet2D |  | ResNet1D
---
w/ MTL
| ResNet2D
---
w/ MTL
Cricket | CricketX | 0.80 (2.5) | 0.75 (4.5) | 0.80 (2.5) | 0.75 (4.5) | $\mathbf{0.95(1.0)}$
CricketY | $\mathbf{0.78(1.5)}$ | $\mathbf{0.78(1.5)}$ | 0.71 (4.0) | 0.68 (5.0) | 0.76 (3.0)
CricketZ | 0.78 (2.5) | 0.78 (2.5) | 0.77 (4.0) | 0.75 (5.0) | $\mathbf{0.94(1.0)}$
UWaveGestureLibrary | UWaveGestureLibraryAll | 0.91 (3.0) | 0.90 (4.0) | $\mathbf{0.98(1.0)}$ | 0.82 (5.0) | 0.97 (2.0)
UWaveGestureLibraryX | 0.77 (5.0) | 0.83 (3.0) | $\mathbf{0.86(1.0)}$ | 0.79 (4.0) | 0.85 (2.0)
UWaveGestureLibraryY | 0.64 (5.0) | 0.73 (3.0) | 0.76 (2.0) | 0.69 (4.0) | $\mathbf{0.80(1.0)}$
UWaveGestureLibraryZ | 0.70 (5.0) | $\mathbf{0.78(1.5)}$ | 0.77 (3.0) | 0.75 (4.0) | $\mathbf{0.78(1.5)}$
AllGestureWiimote | AllGestureWiimoteX | $\mathbf{0.81(1.0)}$ | 0.79 (2.0) | 0.76 (3.0) | 0.69 (5.0) | 0.74 (4.0)
AllGestureWiimoteY | 0.89 (2.0) | $\mathbf{0.90(1.0)}$ | 0.77 (4.0) | 0.74 (5.0) | 0.81 (3.0)
AllGestureWiimoteZ | $\mathbf{0.74(1.0)}$ | 0.70 (2.0) | 0.64 (5.0) | 0.66 (4.0) | 0.68 (3.0)
DodgerLoop | DodgerLoopDay | $\mathbf{0.53(1.5)}$ | 0.41 (4.5) | $\mathbf{0.53(1.5)}$ | 0.41 (4.5) | 0.50 (3.0)
DodgerLoopGame | $\mathbf{0.97(2.0)}$ | 0.88 (5.0) | $\mathbf{0.97(2.0)}$ | $\mathbf{0.97(2.0)}$ | 0.94 (4.0)
DodgerLoopWeekend | $\mathbf{1.00(2.5)}$ | $\mathbf{1.00(2.5)}$ | $\mathbf{1.00(2.5)}$ | 0.97 (5.0) | $\mathbf{1.00(2.5)}$
SemgHand | SemgHandGenderCh2 | 0.84 (4.5) | 0.84 (4.5) | 0.86 (3.0) | 0.92 (2.0) | $\mathbf{0.99(1.0)}$
SemgHandMovementCh2 | 0.62 (3.0) | 0.49 (5.0) | 0.65 (2.0) | 0.58 (4.0) | $\mathbf{0.83(1.0)}$
SemgHandSubjectCh2 | 0.77 (4.0) | 0.74 (5.0) | 0.89 (2.0) | 0.83 (3.0) | $\mathbf{0.98(1.0)}$
GestureMidAir | GestureMidAirD1 | 0.59 (2.0) | 0.54 (4.0) | $\mathbf{0.62(1.0)}$ | 0.50 (5.0) | 0.57 (3.0)
GestureMidAirD2 | 0.60 (2.5) | 0.47 (5.0) | 0.56 (4.0) | 0.60 (2.5) | $\mathbf{0.63(1.0)}$
GestureMidAirD3 | 0.34 (2.0) | 0.22 (5.0) | 0.24 (4.0) | 0.31 (3.0) | $\mathbf{0.35(1.0)}$
ECG | ECG200 | 0.82 (5.0) | $\mathbf{0.93(1.5)}$ | 0.90 (3.0) | 0.88 (4.0) | $\mathbf{0.93(1.5)}$
ECG5000 | 0.94 (4.5) | $\mathbf{0.95(2.0)}$ | $\mathbf{0.95(2.0)}$ | 0.94 (4.5) | $\mathbf{0.95(2.0)}$
ECGFiveDays | 0.99 (5.0) | $\mathbf{1.00(2.5)}$ | $\mathbf{1.00(2.5)}$ | $\mathbf{1.00(2.5)}$ | $\mathbf{1.00(2.5)}$
NonInvasiveFetalECGThorax1 | 0.80 (5.0) | $\mathbf{0.94(1.0)}$ | 0.89 (3.0) | 0.90 (2.0) | 0.88 (4.0)
NonInvasiveFetalECGThorax2 | 0.87 (5.0) | $\mathbf{0.94(1.0)}$ | 0.93 (2.5) | 0.92 (4.0) | 0.93 (2.5)
TwoLeadECG | $\mathbf{1.00(3.0)}$ | $\mathbf{1.00(3.0)}$ | $\mathbf{1.00(3.0)}$ | $\mathbf{1.00(3.0)}$ | $\mathbf{1.00(3.0)}$
Average Rank | 3.20 | 3.06 | 2.70 | 3.86 | $\mathbf{2.18}$

We have evaluated five different methods using the seven MTL datasets. The
tested methods are:

  * •

1NN-DTW: This method serves as the standard baseline for TSC problems [4]. It
utilizes the DTW distance with a one-nearest-neighbor classifier and treats
each task as an independent TSC dataset.

  * •

This is the baseline ResNet1D model depicted in Figure 2. It does not
incorporate MTL and treats tasks as independent TSC problems. Evaluating this
method allows us to examine the impact of MTL on the baseline ResNet1D model.
Since this method does not employ MTL, it employs the standard mini-batch SGD
algorithm for model optimization.

  * •

ResNet2D: This is the proposed ResNet2D model illustrated in Figure 3. Similar
to the previous method, it does not involve MTL and treats tasks as
independent TSC problems. Evaluating this method enables us to analyze the
effect of MTL on the proposed ResNet2D model. As with the previous method, it
utilizes the standard mini-batch SGD algorithm for model optimization.

  * •

ResNet1D w/ MTL: This is the baseline ResNet1D model with MTL, as shown in
Figure 2d. The model is trained using Algorithm 2.

  * •

ResNet2D w/ MTL: This is the proposed ResNet2D model with MTL, as depicted in
Fiugre 3d. The model is also trained with Algorithm 2.

Each deep learning-based model is trained for 400 epochs, and a checkpoint is
saved at the end of each epoch. The best checkpoint is selected based on the
validation accuracy. A batch size of 40 and a learning rate of 0.0001 are
utilized. The experimental results are displayed in Table I. The average rank
is the primary metric for comparison, as it provides insight into the
performance of each method across various tasks. A smaller average rank
indicates superior performance across all tasks in comparison to the other
methods. The best method for each task is highlighted with bold font, and the
second best method is underlined.

First, we compare the performance of 1NN-DTW and ResNet1D. The experimental
results align with previous studies [10, 11]. However, when MTL is applied to
ResNet1D, we observe a decrease in the average rank. Surprisingly, the overall
performance of ResNet1D with MTL is even inferior to the standard baseline of
1NN-DTW. When comparing the proposed ResNet2D model with both variants of
ResNet1D, we find that it outperforms both of them. Moreover, when ResNet2D is
combined with MTL, the performance improves further. In summary, the methods
are ranked from best to worst as follows: ResNet2D with MTL, ResNet2D,
ResNet1D, 1NN-DTW, and ResNet1D with MTL. The proposed ResNet2D model with MTL
exhibits superior performance compared to the other methods.

Figure 5:  The features learned by different MTL models for the
SemgHandGenderCh2 and SemgHandSubjectCh2 tasks are visualized using the
$t$-SNE algorithm. Compared to the baseline ResNet1D model, the proposed
ResNet2D model demonstrates improved separation of samples from different
classes.

We also visualize the learned representations of both MTL models using the
$t$-SNE algorithm. We specifically choose the SemgHand dataset because two of
its tasks have a hierarchical relationship, namely gender and
subject/identity. In the task “SemgHandGenderCh2,” the labels are determined
based on the gender of the subjects from which the samples are gathered. In
the task “SemgHandSubjectCh2,” the labels are determined based on the identity
of the subjects.

The visualization results are presented in Figure 5. Subjects 1, 2, and 3 are
females, while subjects 4 and 5 are males. The latent space learned by the
proposed ResNet2D model successfully captures the hierarchical relationship
between the labels of different tasks. The samples are effectively separated
first based on the gender of the associated subjects and then based on their
identities. In contrast, the baseline ResNet1D model fails to learn this
hierarchical relationship.

###  IV-C Transaction TSC

Understanding transaction data is crucial for financial institutions [35, 36,
37, 38]. Time series derived from transaction data can be utilized to develop
classification and regression models for various business applications [35,
36, 37]. In this section, we conduct experiments on the merchant
classification problem as it aids analysts in gaining a better understanding
of the behavior exhibited by different merchants. We examine merchants from
three perspectives: business category (based on the types of services or goods
being sold), geographical location (i.e., region), and the extent to which a
merchant operates online versus offline (i.e., face-to-face).

We construct the dataset using transactions associated with approximately
87,000 merchants spanning from June 1, 2020, to November 1, 2021. For each
merchant, we generate a time series by calculating the average number of
transactions per hour over a week. This results in a time series
representation for each merchant with a length of 168 (i.e., 7 days multiplied
by 24 hours). The merchants are divided into training, validation, and test
sets using a 6:2:2 split ratio.

TABLE II:  The proposed ResNet2D w/ MTL outperforms the baseline methods in
all three tasks.

| Category | Region | Online/Offline
---|---|---|---
1NN-DTW | 0.2708 | 0.4837 | 0.6629
ResNet1D w/ MTL | 0.2767 | 0.6977 | 0.8388
ResNet2D w/ MTL | 0.3843 | 0.7603 | 0.8407

We conducted experiments on the dataset using three methods: 1NN-DTW, ResNet1D
w/ MTL, and ResNet2D w/ MTL. The non-MTL variants were not evaluated since the
focus of this project was on developing an MTL model. Each model was trained
for 200 epochs, and a checkpoint was saved at the end of each epoch. The best
checkpoint was selected based on the validation accuracy. The models were
trained with a batch size of 40 and a learning rate of 0.0001.

The experimental results are presented in Table II. For this dataset, the
overall order of the methods from best to worst is as follows: ResNet2D w/
MTL, ResNet1D w/ MTL, and 1NN-DTW. Taking into consideration the results from
both Table II and Table I, we can conclude that the proposed method (ResNet2D
with MTL) consistently outperforms the other methods across different tasks.

##  V Conclusion

In this paper, we propose the ResNet2D model for time series classification
(TSC) under the multitask learning (MTL) setting. According to the experiments
conducted using both publicly available UCR archive [4] and an industrial
transaction TSC dataset, the proposed ResNet2D model outperforms its
alternatives. Based on the pixel perturbation visualization technique [33,
34], the proposed ResNet2D model demonstrates the capability to learn the
warping mechanism from time series data. This ability is likely a key factor
contributing to its superior performance. For future work, it would be
interesting to explore the connection between MTL and the learning process of
time series foundation model [39].

## References

  * [1] S. Ruder, “An overview of multi-task learning in deep neural networks,” _arXiv preprint arXiv:1706.05098_ , 2017.
  * [2] X. Liu, P. He, W. Chen, and J. Gao, “Multi-task deep neural networks for natural language understanding,” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 2019, pp. 4487–4496.
  * [3] X. Dai, X. Kong, T. Guo, and X. He, “FiShNet: Fine-grained filter sharing for resource-efficient multi-task learning,” in _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, 2021, pp. 322–331.
  * [4] H. A. Dau, A. Bagnall, K. Kamgar, C.-C. M. Yeh, Y. Zhu, S. Gharghabi, C. A. Ratanamahatana, and E. Keogh, “The ucr time series archive,” _IEEE/CAA Journal of Automatica Sinica_ , vol. 6, no. 6, pp. 1293–1305, 2019.
  * [5] I. Misra, A. Shrivastava, A. Gupta, and M. Hebert, “Cross-stitch networks for multi-task learning,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 3994–4003.
  * [6] Y. Gao, J. Ma, M. Zhao, W. Liu, and A. L. Yuille, “Nddr-cnn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 3205–3214.
  * [7] Y. Gao, H. Bai, Z. Jie, J. Ma, K. Jia, and W. Liu, “Mtl-nas: Task-agnostic neural architecture search towards general-purpose multi-task learning,” in _Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition_ , 2020, pp. 11 543–11 552.
  * [8] S. Liu, E. Johns, and A. J. Davison, “End-to-end multi-task learning with attention,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 1871–1880.
  * [9] X. Sun, R. Panda, R. Feris, and K. Saenko, “Adashare: Learning what to share for efficient deep multi-task learning,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 8728–8740, 2020.
  * [10] Z. Wang, W. Yan, and T. Oates, “Time series classification from scratch with deep neural networks: A strong baseline,” in _2017 International joint conference on neural networks (IJCNN)_. IEEE, 2017, pp. 1578–1585.
  * [11] H. Ismail Fawaz, G. Forestier, J. Weber, L. Idoumghar, and P.-A. Muller, “Deep learning for time series classification: a review,” _Data mining and knowledge discovery_ , vol. 33, no. 4, pp. 917–963, 2019.
  * [12] A. Bagnall, J. Lines, A. Bostrom, J. Large, and E. Keogh, “The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances,” _Data mining and knowledge discovery_ , vol. 31, no. 3, pp. 606–660, 2017.
  * [13] T. Rakthanmanon, B. Campana, A. Mueen, G. Batista, B. Westover, Q. Zhu, J. Zakaria, and E. Keogh, “Searching and mining trillions of time series subsequences under dynamic time warping,” in _Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining_ , 2012, pp. 262–270.
  * [14] Y. Zhang and Q. Yang, “A survey on multi-task learning,” _IEEE Transactions on Knowledge and Data Engineering_ , 2021.
  * [15] R.-G. Cirstea, D.-V. Micu, G.-M. Muresan, C. Guo, and B. Yang, “Correlated time series forecasting using multi-task deep neural networks,” in _Proceedings of the 27th acm international conference on information and knowledge management_ , 2018, pp. 1527–1530.
  * [16] L. Chen, D. Chen, F. Yang, and J. Sun, “A deep multi-task representation learning method for time series classification and retrieval,” _Information Sciences_ , vol. 555, pp. 17–32, 2021.
  * [17] R. Chandra, Y.-S. Ong, and C.-K. Goh, “Co-evolutionary multi-task learning with predictive recurrence for multi-step chaotic time series prediction,” _Neurocomputing_ , vol. 243, pp. 21–34, 2017.
  * [18] M. McDermott, B. Nestor, E. Kim, W. Zhang, A. Goldenberg, P. Szolovits, and M. Ghassemi, “A comprehensive evaluation of multi-task learning and multi-task pre-training on ehr time-series data,” _arXiv preprint arXiv:2007.10185_ , 2020.
  * [19] Z. Chen, V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich, “Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks,” in _International conference on machine learning_. PMLR, 2018, pp. 794–803.
  * [20] O. Sener and V. Koltun, “Multi-task learning as multi-objective optimization,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [21] Y. Lu, A. Kumar, S. Zhai, Y. Cheng, T. Javidi, and R. Feris, “Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 5334–5343.
  * [22] P. Guo, C.-Y. Lee, and D. Ulbricht, “Learning to branch for multi-task learning,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 3854–3863.
  * [23] J. Lines and A. Bagnall, “Time series classification with ensembles of elastic distance measures,” _Data Mining and Knowledge Discovery_ , vol. 29, no. 3, pp. 565–592, 2015.
  * [24] M. Middlehurst, J. Large, M. Flynn, J. Lines, A. Bostrom, and A. Bagnall, “HIVE-COTE 2.0: a new meta ensemble for time series classification,” _Machine Learning_ , vol. 110, no. 11, pp. 3211–3243, 2021.
  * [25] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [26] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [27] Z. Cui, W. Chen, and Y. Chen, “Multi-scale convolutional neural networks for time series classification,” _arXiv preprint arXiv:1603.06995_ , 2016.
  * [28] A. Le Guennec, S. Malinowski, and R. Tavenard, “Data augmentation for time series classification using convolutional neural networks,” in _ECML/PKDD workshop on advanced analytics and learning on temporal data_ , 2016.
  * [29] B. Zhao, H. Lu, S. Chen, J. Liu, and D. Wu, “Convolutional neural networks for time series classification,” _Journal of Systems Engineering and Electronics_ , vol. 28, no. 1, pp. 162–169, 2017.
  * [30] J. Bromley, I. Guyon, Y. LeCun, E. Säckinger, and R. Shah, “Signature verification using a “siamese” time delay neural network,” _Advances in neural information processing systems_ , vol. 6, 1993.
  * [31] M. Cuturi and M. Blondel, “Soft-dtw: a differentiable loss function for time-series,” in _International conference on machine learning_. PMLR, 2017, pp. 894–903.
  * [32] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” _arXiv preprint arXiv:1711.05101_ , 2017.
  * [33] J. Wang, L. Gou, W. Zhang, H. Yang, and H.-W. Shen, “DeepVID: Deep visual interpretation and diagnosis for image classifiers via knowledge distillation,” _IEEE transactions on visualization and computer graphics_ , vol. 25, no. 6, pp. 2168–2180, 2019.
  * [34] J. Wang, W. Zhang, H. Yang, C.-C. M. Yeh, and L. Wang, “Visual analytics for rnn-based deep reinforcement learning,” _IEEE Transactions on Visualization and Computer Graphics_ , 2021.
  * [35] C.-C. M. Yeh, Z. Zhuang, W. Zhang, and L. Wang, “Multi-future merchant transaction prediction,” in _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_. Springer, 2020, pp. 273–289.
  * [36] C.-C. M. Yeh, Z. Zhuang, Y. Zheng, L. Wang, J. Wang, and W. Zhang, “Merchant category identification using credit card transactions,” in _2020 IEEE International Conference on Big Data (Big Data)_. IEEE, 2020, pp. 1736–1744.
  * [37] C.-C. M. Yeh, Z. Zhuang, J. Wang, Y. Zheng, J. Ebrahimi, R. Mercer, L. Wang, and W. Zhang, “Online multi-horizon transaction metric estimation with multi-modal learning in payment networks,” in _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, 2021, pp. 4331–4340.
  * [38] C.-C. M. Yeh, M. Gu, Y. Zheng, H. Chen, J. Ebrahimi, Z. Zhuang, J. Wang, L. Wang, and W. Zhang, “Embedding compression with hashing for efficient representation learning in large-scale graph,” in _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , 2022, pp. 4391–4401.
  * [39] C.-C. M. Yeh, X. Dai, H. Chen, Y. Zheng, Y. Fan, A. Der, V. Lai, Z. Zhuang, W. Junpeng, L. Wang, and W. Zhang, “Toward a foundation model for time series data,” in _Proceedings of the 32nd ACM International Conference on Information & Knowledge Management_, 2023.


#  Language and Intelligence, Artificial vs. Natural or What Can and What
Cannot AI Do with NL?

Gyula Klima  Department of Philosophy
Fordham University<EMAIL_ADDRESS>
###### Abstract

In this talk, I argue that there are certain pragmatic features of natural
language (that I will call ’productivity’ and ’malleability’, on top of
syntactical generativity and semantical compositionality), which are not only
hard, but even impossible to capture in an artificial language used by an AI
system, and the reason for this is to be found in certain deep, metaphysical
differences between artificial and natural intelligence, accounting for the
differences in their respective processes of concept-formation.

##  1 Introduction: some simple definitions, and two theses

More specifically, I will argue for two theses, one negative, one affirmative.
The negative one will present what AI cannot do with natural languages that we
humans can. The affirmative one will present what AI can do with natural
languages we humans can. To better articulate these claims, let me first
introduce some definitions. Some of these are common, some of these are not,
but for the purposes of this talk I will just Humpty Dumptyishly use the terms
defined here in accordance with the following definitions.

A sign is something that besides presenting itself also presents something (or
some things) as its semantical value(s) to a cognitive subject. We can quickly
note here that a semantical value of a sign can coincide with the sign itself,
but then it is not presenting itself only as just a thing, but also as that
which it is a sign of.

A language is a system of signs, consisting of a set of simple signs (called a
vocabulary) and a set of complex signs (called well-formed expressions)
generated by some rules of construction (called the grammar of the language).

  1. 1.

A language has the syntactical feature of generativity if its grammar allows
the generation of a potential infinity of well-formed expressions even from a
finite vocabulary.

  2. 2.

A language has the semantical feature of compositionality if the semantical
values of its well-formed expressions are a function of the semantical values
of its simple components.

  3. 3.

A language has the pragmatic feature of productivity if its users can produce
new, both syntactically and semantically simple signs in its vocabulary (where
a sign is both semantically and syntactically simple when it is not generated
by generativity and its semantic values are not determined by
compositionality).

  4. 4.

Finally, a language has the pragmatic feature of malleability if its users can
systematically reassign the semantic values of both its existing vocabulary
and its expressions for the sake of their practical purposes at will.

Based on these definitions or ”meaning postulates,” my two theses are as
follows:

  1. I.

AI cannot use, let alone produce, a language that has all four features listed
above.

  2. II.

AI can use, and even produce, a language that has the first two features
listed above.

Perhaps, thesis II is not too surprising. After all, the first two features
are quite obviously common features of both natural and artificial languages,
the latter of which are AI’s ”daily bread,” so to speak. Well, but would a
however advanced AI system ”get” my point about formal languages being their
”daily bread” in the way you certainly did (understanding the allusion to the
Lord’s Prayer, along with the literal meaning of the phrase, as well as its
intended meaning as referring to something, indeed, anything, that provides
your everyday sustenance in your life and work)?

To be sure, this is not my argument for thesis I, rather it is just a
preliminary indication of its possible implications. My main argument for
thesis I is in fact more arcane: it is a metaphysical argument from Thomas
Aquinas for the immateriality of the human intellect from its ability to
produce new, simple universal concepts by abstraction. Obviously, to connect
that metaphysical argument meaningfully to my negative thesis, I still have
some more preliminary work to do, namely, I need first to present Aquinas’
semantic conception, relating language, thought, and reality in a particular,
and I’d say, particularly intriguing fashion.

##  2 Some bits and pieces of Aquinas’ semantics and cognitive psychology

What I am about to present here is not the entirety of Aquinas’ semantic
conception. I dealt with that elsewhere, but now I present only the part that
is relevant to his argument I am considering here, namely, the semantics of
simple, universal categorematic terms, such as ’dog’ ’tree,’ ’human,’ etc., in
all ten Aristotelian categories.

Perhaps, the easiest way to do this is by using a diagram I used in my
Stanford Encyclopedia article on the medieval problem of universals.

In this diagram, the arrows pointing from the common term indicate semantic
relations (in particular, subordination, which determines that all
conventional semantic features of our linguistic terms are inherited from the
semantic features of the concepts to which they are subordinated), whereas the
arrows pointing from singulars to phantasms (the singular sensory
representations of singular things), and from phantasms to concepts (the
universal representations of singular things actually used in the mind)
indicate the flow of information.

For instance, if I say, ’This is round, white, and cold’, pointing to a
snowball, then the predicates of this sentence will be taken to signify its
round shape, its white color, and its cold temperature. But of course, these
terms signify not only the individualized qualities of this particular
snowball, because they signify these qualities universally, in abstraction
from the individuating conditions of their subject, determining that they
apply to this thing here and now. Thus, what we are going to be concerned with
in particular is the short arrow pointing from phantasms to intelligible
species, the first universal representations of the human mind deposited in
intellectual memory for use in acts of concept formation when these concepts
are needed in the formation of thoughts involving universal representations of
singulars. That short arrow indicates the process of intellectual abstraction,
the production of intelligible species. In the interest of time, let me refer
you to the Stanford Encyclopedia article for the rest of the details of this
diagram, and let us focus here only on what is relevant in it to Aquinas’
argument.

For a better understanding of Aquinas’ point, after presenting a brief sketch
of the argument itself, I am going to discuss it by responding on Aquinas’
behalf to what I take to be the most serious objection to it, raised by the
Parisian nominalist philosopher of the $14^{\textnormal{th}}$ century, John
Buridan. I would say, however, that Buridan’s objection would be typically
raised also by our contemporaries, as it seems to be so plausible in terms of
our contemporary intuitions. This should not come as a surprise, though, given
that those intuitions have actually been shaped in a long and complex
historical process by Buridan’s and other nominalists’ logical insights. But I
cannot and will not go into this historical issue here. So, let us see first
Aquinas’ argument, then Buridan’s objection, and then my response, pointing
out an important conceptual conflation concerning universal representation,
which will turn out to be directly relevant to our original question of what
an AI system cannot do that a human intelligence can, and why.

##  3 Aquinas’ argument from the universality of concepts

The main claim of Aquinas’ argument is that the universal concepts of our
understanding cannot be received in a material medium because their
universality is achieved precisely by their being abstracted from matter.

To be sure, the universal, abstract mode of representation of the concepts of
the intellect alone cannot guarantee their immateriality – after all, we are
all familiar with material universal signs, such as the words we utter or
write. However, these universal symbols can have their universal
representative function only because they are subordinated to the primarily
universal concepts of our understanding. So, their derivative, conventional
universality need not entail any ontological constraints upon their nature.
Such a constraint may, however, be entailed by the primarily universal
representative function of the concepts of understanding, which are formed as
a result of the natural causality of sensible objects on the senses and the
consequent further processing of the information thus obtained from the senses
by the intellect.

The question then is whether, and if so, why the primarily universal mode of
representation of the concepts of the understanding formed as a result of the
causality of sensible objects should entail the immateriality of these
concepts, in the sense that the subject in which they are received, the
intellect, cannot be material.

The main idea is the following. The senses represent singulars in their
singularity because they necessarily represent the sensible features of
material objects together with the material individuating conditions of these
features, namely, the spatio-temporal dimensions determining their principle
of individuation, the designated matter of these objects, that is, the matter
of these objects contained under their dimensions here and now. The reason why
this is necessary is that the causally active sensible features of sensible
objects necessarily exercise their causality on the senses under these
determinate dimensions, and so these sensible features are necessarily encoded
by the senses as determined by these dimensions. Therefore, what encodes these
spatio-temporal features in the senses are some corresponding spatio-temporal
features of the sense organs because it is precisely these features of the
sense organs that are impacted by the corresponding features of their objects.
For example, the spatial arrangement of distinct patches of color in my visual
field is encoded by the spatial pattern of receptors firing in the retina of
my eyes, and similar considerations apply to the other senses. In the process
of abstraction, therefore, the intellect has to ”cut out” precisely this part
of the code preserved in the phantasms. So, it has to form the concepts
encoding the universal information contained in a vast number of different
phantasms in a medium that will not encode the information about the
singularity of singulars represented by the phantasms. Yet, it will encode
that part of the information that is invariably common, because essential, in
this massive ”database” of various singular experiences under variable
circumstances such as time and location and other coincidental circumstances
and features.

But then, since what encodes this circumstantial information in the phantasms
is the spatio-temporal features of the organs in which they are received, the
medium in which the universal representations are formed must be something
that does not have such spatio-temporal features, i.e., something that does
not have its own dimensions, which can only be a thing that is immaterial;
therefore, the intellect receiving these representations has to be immaterial.

As can be seen, the most important idea in this argument is that the
singularity of representation is necessarily tied to the materiality of
representations. Sensory representation is necessarily singular because it is
material, since the singularity of the information in sensory representation
is encoded precisely by the material features of sensory representations,
namely, the spatio-temporal features of the sense organs (including relevant
parts of the brain) that are modified according to the spatio-temporal
features of sensory objects, which determine their singularity. But if this
much is acceptable, then the argument can indeed establish its desired
conclusion.

All in all, the argument boils down to just the following (instantiated) modus
tollens:

  1. 1.

If a natural (as opposed to conventional) representation is material, then it
is singular

  2. 2.

(Some of) the intellect’s natural representation is not singular (since it is
universal)

  3. 3.

Therefore, (some of) the intellect’s natural representation is not material

Once we reduce Aquinas’ rather complex original argument to this simple modus
tollens, then it should become clear that its crucial part is the first
premise, namely, the alleged implication moving from the materiality of a
natural representation to its singularity, by natural (as opposed to logical)
necessity. So, it is not surprising that it is precisely this implication that
Buridan targets in his criticism.111For a more detailed presentation of and
references to Aquinas’ argument, see [2]

##  4 Buridan on Aquinas’ argument

Buridan begins his discussion with the claim that the position we find in
Aquinas’ Commentary on De Anima is wrong:

  * … it has seemed to some people that sense lacks the nature for cognizing universally, although it does have it for cognizing singularly and determinately, because it has extension and a determinate location in a corporeal organ. It is the other way around with the intellect, however. Because it is immaterial and unextended, not determining for itself a location in a corporeal organ, it does possess the nature for understanding universally and not singularly. […] But this opinion seems defective.222Buridan, QDA3 q. 8, nn. 18-19. in [5]

There are two passages in which Buridan explicitly takes on the task of
refuting (1). First, he writes:

  * … an extended power, such as a horse’s appetite, is indeed brought to its object in a universal way. For the thirsty horse desires water, and not determinately this water to the exclusion of that, but indifferently, any water at all. Thus, it drinks whichever it finds first.333QDA3 q. 3, n. 30. in [5]

Likewise, in q. 8, the idea is the indifferent, and hence apparently universal
natural desire of the agent, indeed, even of an inanimate agent, which does
not require an immaterial subject for its universality:

  * Furthermore, it is apparent that a material and extended power is properly brought to bear on its object in a universal way, for the appetite of a horse in the form of hunger or thirst is not singularly for this sack of oats or that water, but for any, indifferently, which is why it would take whichever it finds first. And the natural intention or appetite of fire for heating is not related to this or that heatable thing in a singular way, but indifferently, to anything it can heat. Thus, it would heat whatever is put to it.444QDA3 q. 8, n. 24. in [5]

However, despite his criticism of Aquinas’ implication, Buridan does agree
with him on the claim that the universality of a cognitive act is due to the
abstractive ability of the cognitive power in question, and the singularity of
a cognitive act is due to it carrying distinctive information about singulars
as such, from which it is unable to abstract:

  * … although an exterior sense cognizes Socrates, or whiteness, or white (color), this is only via a species representing it confusedly with the substance, the whiteness, the size, and location, as he appears in the prospect of the person cognizing him. And the power of sense cannot sort out the confusion if it is unable abstract the species of the substance, the whiteness, the size, and location from each other, and so it can only perceive substance, or whiteness, or a white thing in the manner of something existing in its prospect. Thus, it can cognize the aforementioned objects only singularly.555QDA3 q. 8, n. 28. in [5]

Indeed, since he thus ties the singularity of cognition to its distinctive
representational content about individuals, Buridan argues even further that
the singularity of representation is inherited by all those sensory powers
that do not abstract from this distinctive, singularizing information:

  * Again, since666My emendation ad sensum, for the text’s ’even if’ [etsi]. the common sense receives [the sensible] species from the external senses with this sort of confusion and cannot resolve the confusion, it necessarily apprehends in a singular manner. That is why we judge in dreams that this or that appears to us to exist, or to be here or there.777QDA3 q. 8, n. 29. in [5]

Thus, just like Aquinas, Buridan clearly ties the singularity of sensory
representation to its distinctive information content. Nevertheless, in his
criticism of Aquinas’ position, he also allows that some material, sensory
powers, such as sensory appetite, can relate to their objects in an
indifferent, non-singular manner, which presupposes some sort of indifferent,
apparently universal representation of their objects. But is that really the
sort of genuinely universal representation we have in intellectual cognition?
Indeed, is Buridan’s criticism consistent with his position on the reason for
the singularity of sensory representation based on the different location of
singulars in one’s sensory field?

##  5 My response to Buridan on Aquinas’ behalf

Answering first this second question, we should know that Aquinas provides a
very compelling reason why location can have a crucial, naturally distinctive
role in singular cognition, while discussing the sensory cognition of common
sensibilia (i.e., sensible qualities that are cognized by several senses, such
as shape, size, position, etc., as opposed to proper sensibilia that are
cognized only by one sense):

  * … there are objects which differentiate sensation with respect not to the kind [species] of the agent, but to the mode of its activity. For as sensible qualities affect the senses corporeally and locally, they do so in diverse ways if they are qualities of large or small bodies or are diversely situated, i.e., near, or far, or in the same place or in diverse places. And it is in this way that the common sensibles differentiate sensation. 888Sentencia De anima, lib. 2 l. 13 n. 12. in [5]

So, common sensibilia are the necessary spatio-temporal determinations of all
proper sensibilia. The proper sensibilia, in turn, are the individualized
sensible qualities of material individuals that the external senses are
specifically attuned to be affected by, such as color, sound, smell, taste,
texture and temperature. But these proper sensibilia are individualized by
their spatio-temporal determinations here and now, the common sensibilia. The
cognition of common sensibilia, therefore, provides precisely that distinctive
sensory information that singularizes the cognition of individualized sensible
qualities, presenting the singulars having these qualities qua the singulars
actually affecting the senses here and now.

However, if the external senses receive this distinctive, singular information
about the individuating spatio-temporal conditions of their objects on account
of receiving the causal impact of these objects through their own spatio-
temporal features, then this seems to establish the implication Buridan argued
against in the first place – that the materiality of a cognitive power entails
the singularity of its cognitive act. For if sensory representation is
singular precisely because it represents its object in a material fashion,
encoding the distinctive, singular information about the object by its own
material features, then this means that sensory representation is singular
because it is material, i.e., its materiality implies its singularity.

Indeed, if he is successful in establishing this much, then, pace Buridan,
Aquinas does have a good argument for the immateriality of the intellect. For
the contrapositive of this implication, namely, that the non-singularity of
its cognitive act entails the immateriality of a cognitive power, together
with the fact that the intellect does have some non-singular, i.e., genuinely
universal cognitive acts, establishes the immateriality of the intellect.
Thus, when Buridan endorsed the claim that the singularity of sensory
cognition is due to it carrying distinctive singular information that it
cannot abstract from, he did not move too far away from Aquinas’ position.
Aquinas just made the further, quite plausible, claim that the encoding of
this distinctive singular information in the senses is due to their material
character, as they receive the localized, spatio-temporal, causal impact of
material singulars in a similarly localized, spatio-temporal fashion. But
then, it seems, Aquinas is quite entitled to his further conclusion concerning
the immateriality of the intellect.

Furthermore, in response to my first question at the end of the previous
section, we should note that Buridan’s objection to Aquinas’ proof rests on an
unjustifiable conflation between the mere indifference of singular
representation obtained by the senses and true universality of representation;
thus, we should be really careful in separating the two.

For take the snapshot of an egg. That picture, despite the fact that it does
not carry enough distinctive information to tell (from that information
content alone) which particular egg it is the picture of, still carries
information only about that particular egg; so, it is not really a universal
representation of all eggs, it is a singular representation only of that
particular egg. Indeed, it does not carry any information about a marble egg
of the same shape and color, even if a snapshot of that fake egg might be
indistinguishable from that of the genuine egg. To be sure, this still does
not mean that the two pictures are absolutely indistinguishable: after all,
based on their metadata (time and geolocation), we could tell with precision
which is the picture of which object. So, these apparently indifferently
representing images are not universal representations of many or even all
eggs; they are just singular, yet in their direct information content (as
opposed to their metadata) non-distinctive representations of their singular
objects.

By contrast, a genuinely universal representation of a genuine egg is the
result of abstracting precisely that form of eggs in general which comes from
innumerable experiences of all sorts of singular eggs not only in a single
human life, but over generations of human experiences, so firmly established
that allows it to be encoded in all varieties of human languages, eventually
even allowing a scientific investigation to establish its articulation in a
genuine, scientific, quidditative definition, to the exclusion of marble eggs
merely resembling in shape and color.

To be sure, one might object: don’t I also see a common form, say the common
form of oval shape, in the indifferent image of an egg?

I would respond that no, you don’t. A properly universal representation of
that shape would be the mathematical formula of an ellipsoid shape,
articulating our pre-theoretical, yet genuinely universal concept of the same,
which carries not only non-distinctive information about this particular egg
that the snapshot does, but also distinctive, precise, universal information
about all possible egg-shapes.999https://www.math.net/ellipsoid This is why I
have kept emphasizing in my previous writings touching on this subject that
the process of abstraction, pace Locke,101010For the contrast with Locke and
British Empiricism in general, see [4] results not in a mere loss of
information, but in the active sorting out of accidental, coincidental,
irrelevant information, from a massive database of singulars to the effect of
gaining genuinely universal, essential information, representing all possible
singulars, whether in the category of substances or accidents.111111Cf. [3]
But couldn’t this sort of abstraction still be realized in the same material
medium (say, the brain) as the processing of incoming sensory information?

To be sure, operating on their material carriers, the originally singular
representations can be rendered more and more indifferent, by stripping them
of more and more explicit or implicit singularizing information, giving them
the semblance of universal representations. Yet, by this simple process of
mere loss of information they are rendered just more and more non-distinctive,
yet not genuinely universal. Genuine, naturally produced universal
representations, namely, the intelligible species, are produced by
abstraction, which is not only a process of losing distinctive, singularizing
information, but at the same time it is the process of gaining universal
information carried by all singular representations of the same kind, so that
these universal representations also cover singulars of the same kind we have
not experienced, of which we have no singular representations at all.

What is needed for this is an agent outside the causal chains of merely
manipulating the material carriers of singular information (rendered singular
by their material causal histories), which is outside of these causal chains
in the sense that it is capable of grasping that common, intelligible content
that applies to a potential infinity of entities of the same kind, never
represented as such by any of the representations of singulars but carried by
them all. This is why the resulting truly universal representation cannot be
material: it cannot have this information content resulting from the further
material manipulation of the material carriers of singular information, which
are singular precisely because of their material causal histories.

So, making the distinction between a merely non-distinctive, singular
representation and a genuinely universal representation effectively defuses
Buridan’s objection: just because a horse may quench its thirst in search of
any bucket of water, it doesn’t mean it has a universal concept of water any
more than the fact that any bucket of water may put out a small fire requires
that water has a universal concept of fire. The indifference of action in
singular causal relations primarily dependent on the kinds of agent and
patient is part and parcel of the regularities of nature, which of course does
not require or presuppose any sort of awareness in any of them. Therefore,
when Buridan is talking about the horse’s indifferent desire toward any bucket
of water to quench its thirst, it does not require on the horse’s part any
sort of genuinely universal representation: the horse is just indifferently
reacting to the singular representations of this or that bucket, equally good
for satisfying its thirst. But Aquinas’ argument is about genuinely non-
singular, universal representations of the intellect; so, Buridan’s objection,
conflating genuinely universal representations with merely non-distinctive
singular ones, simply misses its target.

##  6 Conclusion: the implications of Aquinas’ argument for NL and AI

So, let me conclude with a brief summary of this discussion, and some of its
further implications concerning our original questions about what AI can and
what it cannot do with NL.

The nominalist conflation of true universality with mere indifference of a
singular representation does in the end totally undermine Buridan’s criticism:
just because the snapshot of an egg does not give you distinctive information
about this egg distinguishing it from any other egg of the same size and
shape, it doesn’t mean that the snapshot is a universal representation of eggs
in general. In fact, the picture still carries information only about the same
egg you took the picture of, except, based on the information content of the
picture alone (to the exclusion of its metadata), you cannot determine which
one it pictures (although, based on the broader context of taking the picture
encoded in its metadata, it can still quite easily be determined).

Thus, Buridan seems to be implicitly committed to the implication he
explicitly rejects, which renders his attempted refutation ineffective. But
then, if this is correct, we may safely conclude that as far as Buridan’s
criticism is concerned, Aquinas may just be right.

However, if Aquinas is indeed right, then his immateriality thesis has far-
reaching consequences even for contemporary AI research. Given that any
computer we shall ever make will process information in its material medium,
the inevitable conclusion is this: if Aquinas’ main thesis is right, then all
the information any AI machine processes can only be secondarily universal,
riding on the universality of our primarily universal, human concepts, which
can only be produced in the immaterial medium of the human mind. Thus, an AI
system will never be creative in the way human intelligence is. An AI system
will never form new concepts, it can only re-process (perhaps, much faster,
and never tiring, but those are other issues) the concepts produced by the
genuinely creative, because immaterial human intelligence.

To be sure, this metaphysical limit on what AI can do with language may seem
to be rather limited itself. After all, Aquinas’ argument concerns only the
natural, pre-linguistic process of the formation of both syntactically and
semantically simple categorematic concepts that would populate Aristotle’s
system of categories. But the formation of these concepts is not what we would
usually associate with human creativity but would rather regard it as
characteristic of a necessary stage of children’s developmental psychology,
and rightly so. However, if Aquinas is indeed right, then his position can
give us a clue about why a healthy human baby and a healthy chimpanzee baby
raised in the same environment will develop so differently, especially with
regard to their abilities of language learning and use. For on Aquinas’
position, a human baby, having an immaterial intellect, can genuinely
”produce” such immaterial concepts a chimpanzee baby is metaphysically
incapable of producing. And this would be the metaphysical basis of the
feature of natural languages I dubbed ’productivity.’

But, even further, Aquinas’ position may also give us a clue about processes
of secondary, post-linguistic abstractions, in terms of analogical and
metaphorical, scientific, and poetic concept formation in interaction not only
with one’s sensible environment, but also with one’s language and linguistic
community.

For every word of our languages carries with it a palimpsest of concepts
formed by the abstractions of generations of its users, recorded in its
etymology. And this is the idea behind the fourth, pragmatic feature of
natural languages I named ’malleability.’ For our universal categorematic
concepts we form in a natural process of abstraction are just those primordial
simple concepts we can gain from our personal experience, which would be
pretty much like the conceptual vocabulary of a feral child. But a child
raised in a normal social environment is immediately exposed to the concepts
encoded in the language of their environment. Thus, children’s natural
intelligence is going to form not only the primary primitive concepts based on
their own personal experiences, but also the concepts of others, abstracted by
generations and handed down to them through language. But this process of
concept acquisition through language is not a mere copying process.

Every user of a language has the ability to modify the use of conventionally
significant words by subordinating them to their concepts acquired and
modified by forming analogies and metaphors in a process of secondary
abstraction. Take for instance the word ’malleability’ as I am using it in
this lecture. Perhaps, you are not even aware of the fact that at the
etymological root of this English word is the Latin word ’malleus,’ meaning
hammer. So, strictly speaking, what is malleable is something that can be
shaped and reshaped by a hammer. Yet, the way I am using it, the word is not
used in its strict sense: of course, the concepts our words are subordinated
to are not the kinds of things that can be reshaped by a hammer. But you had
no difficulty in understanding my idea that our concepts, and thus the usage
of the words they are subordinated to are changeable, based on the analogy
that just as malleable materials can be reshaped by a hammer, so our concepts
can be modified in their representative functions by our minds, which accounts
for the fact that our existing words getting subordinated to these modified
concepts have this feature of ’malleability’, of changing usage, endowing them
with new meanings. But it is precisely this conceptual malleability that
underlies what we ordinarily call ”creative” uses of language whether in
science, philosophy, or theology, using inventive analogies to introduce new
concepts by modifying old ones, or in poetry and rhetoric, using metaphors for
more powerful expression of our ideas that move both our intellect and
imagination.

But if this sort of creative use of language is based on the secondary
processes of abstraction that presuppose the primary process which by Aquinas’
lights cannot take place in a material medium, but only in the immaterial
medium of the human intellect, then it is unavailable to the material medium
of an AI system. Nevertheless, the creative human mind will still be able to
endow such systems with its already produced and further modified concepts, as
long as it is capable of reproducing the semantic features of the
corresponding human concepts in the semantic features of the words the AI
system uses, exploiting those features of human languages that is available to
such systems, namely, generativity and compositionality.

## References

  * [1]
  * [2] Gyula Klima (2018): _Aquinas’ Balancing Act: Balancing the Soul Between the Realms of Matter and Pure Spirit_.  Bochumer Philosophisches Jahrbuch für Antike und Mittelalter 21(1), pp. 29–48, 10.1075/bpjam.00022.kli.  Available at https://www.jbe-platform.com/content/journals/10.1075/bpjam.00022.kli.
  * [3] Gyula Klima (2018): _Aquinas’ Balancing Act: Balancing the Soul Between the Realms of Matter and Pure Spirit_.  Bochumer Philosophisches Jahrbuch für Antike und Mittelalter 21(1), pp. 29–48, 10.1075/bpjam.00022.kli.  Available at https://www.jbe-platform.com/content/journals/10.1075/bpjam.00022.kli.
  * [4] Gyula Klima, Peter G. Sobol, Peter Hartman & Jack Zupko (2004): _John Buridan’s Questions on Aristotle’s De Anima – Iohannis Buridani Quaestiones in Aristotelis De Anima_.  In R. L. Friedmann & S Ebbesen, editors: John Buridan and Beyond: Topics in the Language Sciences 1300-1700, Royal Danish academy of sciences and letters, Copenhagen, pp. 17–32.
  * [5] Gyula Klima, Peter G. Sobol, Peter Hartman & Jack Zupko (2022): _John Buridan’s Questions on Aristotle’s De Anima – Iohannis Buridani Quaestiones in Aristotelis De Anima_.  Springer Cham, Switzerland.