# Classifying Cancer Stage with Open-Source Clinical Large Language Models*
*Note: This manuscript has been accepted to the IEEE International Conference on Healthcare Informatics (IEEE ICHI 2024).

Chia-Hsuan Chang1, Mary M. Lucas1, Grace Lu-Yao2, and Christopher C. Yang1
0000-0001-9116-8244 0000-0002-0413-7499 0000-0002-2925-7737
0000-0001-5463-6926 1College of Computing and Informatics
Drexel University, Philadelphia PA, USA
Email<EMAIL_ADDRESS>2Department of Medical Oncology,
Sidney Kimmel Cancer Center
Thomas Jefferson University, Philadelphia, PA, USA
Email<EMAIL_ADDRESS>
###### Abstract

Cancer stage classification is important for making treatment and care
management plans for oncology patients. Information on staging is often
included in unstructured form in clinical, pathology, radiology and other
free-text reports in the electronic health record system, requiring extensive
work to parse and obtain. To facilitate the extraction of this information,
previous NLP approaches rely on labeled training datasets, which are labor-
intensive to prepare. In this study, we demonstrate that without any labeled
training data, open-source clinical large language models (LLMs) can extract
pathologic tumor-node-metastasis (pTNM) staging information from real-world
pathology reports. Our experiments compare LLMs and a BERT-based model fine-
tuned using the labeled data. Our findings suggest that while LLMs still
exhibit subpar performance in Tumor (T) classification, with the appropriate
adoption of prompting strategies, they can achieve comparable performance on
Metastasis (M) classification and improved performance on Node (N)
classification.

##  I Introduction

Although deaths due to cancer have continued to drop in the United States
(U.S.), an estimated 2 million people were diagnosed with cancer in 2023 [1].
Cancer was one of the leading causes of death in the U.S. in 2021, second only
to heart disease, and provisional mortality statistics indicate that this
remained unchanged in 2022 and 2023 [2]. Diagnosing, treating, and monitoring
cancer is an interdisciplinary effort that involves multiple health
specialties, including medical and surgical oncologists, pathologists,
radiologists, interventional radiologists, pharmacists, and nurses among
others. All these providers interface with the patient at different times in
their medical journey, thereby creating vast amounts of clinical data
containing rich clinical insights. Knowledge of the patient’s cancer stage is
a critical piece of diagnostic and prognostic information for guiding
treatment planning. An important type of staging data available from pathology
reports is the Tumor-Node-Metastasis (TNM) stage, and specifically the
pathologic TNM (pTNM) stage (determined after surgery, when the tumor has been
excised and tissue samples obtained for analysis [3]). TNM is a staging system
that allows for a standardized format for presenting information about
different cancers. It includes information on the size and extent of the main
tumor (T), how much it has spread to the lymph nodes (N), and whether it has
spread further to distant sites in the body (M) [4]. These categories can also
be further subdivided to provide additional information.

While electronic health record (EHR) systems have made it easier to access and
analyze large amounts of clinical data for research and for extracting
patterns and potential new insights, the ability to parse this kind of staging
information at scale remains a challenge. Clinical data on tumor
characteristics and staging are usually contained in clinical free-text notes
rather then being recorded in a structured format in the EHR. These notes are
often not in a well structured or template format, necessitating the use of
natural language processing (NLP) techniques [5, 6, 7]. While NLP approaches
have continued to improve with the advent of pre-trained models [8], the need
for large training datasets remains a challenge [9, 10]. The recent
developments in generative large language models (LLMs) provide an opportunity
to improve extraction of cancer staging from clinical reports, and to advance
cancer research at a more accelerated pace.

We evaluate the ability of LLMs to extract the pTNM classification from
unstructured text data without the need for training datasets. We use
pathology reports obtained from The Cancer Genomic Atlas (TCGA) project and
compare a general purpose LLM (Llama-2-70b-chat) with clinical LLMs
(ClinicalCamel-70B and Med42-70B), assessing their capabilities across
different prompting strategies. Evaluating and attempting to improve
performance of open-source LLMs on real-world medical data and clinically
relevant tasks is important because these LLMs can be installed locally,
reducing concerns of exposing protected health information (PHI).

##  II Literature Review

NLP has been adopted for extracting information from pathology reports.
Several studies propose specific deep learning models for this purpose. Gao et
al. [5], and Gao et al. [6] proposed a hierarchical network, which learns
representation in a layered manner: from words to sentences and reports. They
evaluated the generated report representations using pathology reports from
the NCI SEER program for five classification tasks, including tumor grade
classification. Wu et al. [7] experimented with the attention-based graph
convolution network, in which graph nodes are either words or reports. They
defined multiple graphs using different sources of knowledge and applied
attention mechanisms to aggregate and propagate the knowledge across graphs,
resulting in a better report representation. They used pathology reports from
the TCGA project and evaluated various tasks, such as the TNM stage. Rather
than introducing a novel neural network, Kelfeli and Tatonetti [8] leveraged
the power of pre-trained language models. They fine-tuned a clinical-specific
model, Clinical-BigBird, for TNM classification using reports from the TCGA
project. Their fine-tuned model performed well not only on the testing reports
from the TCGA project but also on pathology reports from Columbia University
Irving Medical Center. However, all of the above models require substantial
amounts of labeled training data, which in turn demands significant amounts of
human efforts for curation and annotation.

To reduce the amount of required training reports, Angeli et al. [9]
investigated active learning techniques for dynamically selecting training
samples. They discovered that with the adoption of a convolution neural
network, effective active learning techniques can help create a dataset that
requires less than half the amount of labeled data to achieve the same
performance as a dataset constructed using random sampling. However, it is
important to note that even with active learning techniques, a sufficient
amount of initial training set and holdout set is still required for a
sequence of sample selection processes. Odishi et al. [10] vectorized each
token using surrounding contexts, so-called bag-of-n-grams. They learned
classifiers for pathologic stage classification using logistic regression,
AdaBoost, and random forest. They showed that the model trained using only 64
training reports can generalize well on five times larger testing set of
reports. However, the training and testing set were all pathology reports for
prostate cancer. The generalizability of their approach across different
cancer types is unmeasured.

Recently, LLMs have demonstrated remarkable performance on medical-related
tasks owing to their ability to recognize, predict, or generate text or
content utilizing transformer models, trained on large volume of publicly
available texts. Several studies have indicated that LLMs perform well on
various medical Q&A datasets using very little or without training data [11,
12, 13]. To the best of our knowledge, the performance of LLMs for cancer TNM
classification is still unknown. This study aims to fill this gap.

##  III Materials and Methods

###  III-A Dataset

The data used for this study comprises free-text pathology reports from the
Cancer Genomic Atlas (TCGA) project of the National Cancer Institute (NCI).
These reports, in their original format are downloadable as PDF files, with
associated metadata found on the NCI Genomic Data Commons (GDC) portal. We
utilized a preprocessed corpus that was curated by Kefeli and Tatonetti [14].
The authors employed optical character recognition (OCR) techniques to convert
the PDF reports into machine-readable text and further preprocessed them to
remove extraneous information as well as clinically irrelevant headers. This
resulted in a dataset of 9,523 reports. A subset of 6,940 reports from this
corpus having associated ground truth labels for the T, N, or M stage was then
identified for TNM classification task. The reports were split into 85%
training and 15% testing datasets, using stratified sampling to ensure the
class distributions in T, N, and M were similar between datasets. Table I
presents the class distribution of the testing dataset.

TABLE I: Distribution of class labels for T, N, and M data | Labels | Count | Ratio
---|---|---|---
T14 | 1 | 262 | 0.253
2 | 351 | 0.339
3 | 317 | 0.306
4 | 104 | 0.100
N03 | 0 | 500 | 0.586
1 | 219 | 0.257
2 | 104 | 0.122
3 | 29 | 0.034
M01 | 0 | 645 | 0.932
1 | 47 | 0.067

###  III-B Benchmark

We employ Clinical-BigBird [15], which extends from the BERT architecture and
has 128.1M parameters. Clinical-BigBird outperforms two other well-recognized
clinical BERT-based models - BioBERT [16] and ClinicalBERT [17], in varying
medical-related tasks. To adapt Clinical-BigBird for cancer staging
classification task, a previous study [8] developed a fine-tuned model using
T, N, and M training data split in TCGA dataset and reported its competent
performance and generalizability on different sources of pathology reports.
Clinical-BigBird is a strong baseline for performance benchmark since it has
been exposed to the training split of the TCGA dataset. The model weights of
the fine-tuned Clinical-BigBird for TNM staging classification can be
downloaded from https://github.com/tatonetti-lab/tnm-stage-classifier.

###  III-C Large Language Model

We adopt three open-source LLMs, Llama-2-70b-chat, ClinicalCamel-70B, and
Med42-70B, and evaluate their capabilities on cancer pTNM classification.
ClinicalCamel-70B and Med42-70B are clinical LLMs derived from Llama-2 and
fine-tuned with clinical data.

The first model is Llama-2 [18], which has 70B parameters and is pre-trained
using two trillion tokens of public texts. Llama-2 has been evaluated in
varied different reasoning tasks. We use its dialogue-optimized version,
Llama-2-70b-chat, whose model weights can be downloaded from
https://ai.meta.com/llama/, for our experiments. However, Llama-2-70b-chat is
a general purpose model. Hence, we survey the clinical LLMs published in 2023
and select ClinicalCamel-70B and Med42-70B as our experimental targets. Both
ClinicalCamel-70B and Med42-70B are reported to outperform other open-source
models (e.g., ChatDoctor [19], MedAlpaca [12], and PMC-LLAMA [11]) and a well-
adopted proprietary model, GPT-3.5, in various medical Q&A datasets.

ClinicalCamel-70B [13] is a fine-tune of Llama-2 for clinical research. To
adapt the model for accommodating clinical knowledge, the dataset for fine-
tuning includes general multi-step conversations, open-access clinical
articles, and medical multiple-choice questions with answers. Lastly,
Med42-70B111The details of this model can be found at
https://huggingface.co/m42-health/med42-70b is also a derived model based on
Llama-2, and is fine-tuned using a dataset of 250M tokens compiled from
different open-access sources, including medical flashcards, exam questions,
and open-domain dialogues. However, unlike the Clinical-BigBird model that we
use as the benchmark, these clinical LLMs have not been fine-tuned with the T,
N, M training data split from the TCGA dataset.

We quantize each LLM from float16 to int8 to reduce memory usage load in the
loading and inference phases. Our implementation is based on HuggingFace’s
transformers package, and we run our experiments using two NVIDIA A40 GPUs.

###  III-D Prompting strategy

We implement three different prompting strategies for the three LLMs. The
prompting strategy is controlled by different prompt templates $X$.

  1. 1.

Zero-shot (ZS) serves as the baseline prompt, in which we provide the relevant
context to instruct the model for cancer staging tasks: “You are provided with
a pathology report for a cancer patient. Please review this report and
determine the pathologic stage of the patient’s cancer.” The ZS prompt also
includes a pathology report. We ask the given LLM to determine the desired TNM
staging class.

  2. 2.

Zero-shot Chain-of-Thoughts (ZS-COT) [20] adopts two sequential prompts to
perform inference. To better use LLM’s capability on reasoning, the first step
prompt extends ZS by appending “Let’s think step by step”, triggering the
language model to generate step-by-step reasoning for the given report. The
second step prompt collects the generated reasoning for the model to perform
the TNM staging classification task.

  3. 3.

Few-shots (FS) is a widely-adopted prompting strategy with GPT-3.5 [21]. FS
extends ZS by providing several text-based demonstrations ($k$-shots) that are
relevant to the given task. With FS, the LLM may learn how to solve the given
task from demonstrations. In this study, we ask an experienced clinical
practitioner to use the TCGA training dataset to select 5-shot, 5-shot, and
6-shot demonstrations for the T, N, and M categories, respectively. All
demonstrations are formatted in input-output format, where input is the
excerpt extracted from a report, and output is the staging class of the
report.

We adopt greedy decoding when we provide prompts for LLM. Assume the LLM
generates a token based on prompt: $p(y|X)$, the greedy decoding in an auto-
regressive manner could be represented as:

$y_{t}=\text{argmax}_{y\in Y}p(y|X,y_{1},...,y_{t-1}),$ (1)

where $Y$ is the possible token space. After we obtain $\\{y_{1},...,y_{t}\\}$
from a model, we use regular expressions (regex) to capture the TNM
classification. Specifically, we capture {T1, T2, T3, and T4} for T category,
{N0, N1, N2, and N3} for N category, and {M0, M1} for M category.

###  III-E Performance Metric

We report the classification performance using precision, recall, and F1 for
each stage class (e.g., T1-T4, N0-N3, and M0-M1). All experiments and
performance metrics are based on the test split of the TCGA dataset, to allow
for a fair comparison with the benchmark Clinical-BigBird.

$\text{precision}=\frac{TP}{TP+FP}$ (2)

$\text{recall}=\frac{TP}{TP+FN}$ (3)

$\text{F1}=2\times\frac{\text{precision}\times\text{recall}}{\text{precision}+\text{recall}}$
(4)

We report macro precision, recall, and F1 for comparing different models and
prompting strategies because each stage category has imbalanced class
distribution, and the performance of the rare class is equally important as
the performance of the frequent class. To deliver robust evaluation, we use
bootstrapping resampling to sample each model’s predictions and calculate
performances $B=500$ times, and each time, we randomly sample $N$ predictions
with replacement, where $N$ is the size of the test set. Therefore, we can
calculate the 95% confidence interval for each model’s performance metric and
perform bootstrapping t-test [22].

##  IV Result

###  IV-A Performance Comparison: Benchmark and LLMs with Zero-shot Prompting

Table II, Table III, and Table IV report the performance comparison between
Clinical-BigBird (the benchmark) and three different LLMs using zero-shot
prompting strategy on T, N, and M staging classification task, respectively.
In Table II, we observe that ClinicalCamel-70B and Med42-70B prevail over
Llama-2-70b-chat with respect to the macro F1 score. ClinicalCamel-70B has a
comparable macro F1 compared with Clinical-BigBird; however, Clinical-BigBird
still performs the best in classifying reports in the T category.

TABLE II: Performance table for T category

Model | Class | Precision | Recall | F1-score
---|---|---|---|---
| T1 | 0.83 | 0.79 | 0.81
| T2 | 0.76 | 0.84 | 0.80
| T3 | 0.84 | 0.84 | 0.84
| T4 | 0.89 | 0.68 | 0.77
Clinical-BigBird | Macro avg. |  | 0.83
---
[0.80,0.85]
| 0.79
---
[0.76,0.82]
| 0.81
---
[0.78,0.83]
| T1 | 0.97 | 0.51 | 0.67
| T2 | 0.85 | 0.75 | 0.80
| T3 | 0.56 | 0.96 | 0.70
| T4 | 0.98 | 0.44 | 0.61
Llama-2-70b-chat + ZS | Macro avg. |  | 0.84
---
[0.82, 0.86]
| 0.66
---
[0.63,0.70]
| 0.69
---
[0.66,0.73]
| T1 | 0.87 | 0.69 | 0.77
| T2 | 0.84 | 0.83 | 0.83
| T3 | 0.73 | 0.88 | 0.80
| T4 | 0.73 | 0.71 | 0.72
ClinicalCamel-70B + ZS | Macro avg. |  | 0.79
---
[0.76,0.82]
| 0.77
---
[0.75,0.80]
| 0.78
---
[0.75,0.80]
| T1 | 0.78 | 0.69 | 0.73
| T2 | 0.93 | 0.70 | 0.80
| T3 | 0.61 | 0.93 | 0.74
| T4 | 0.89 | 0.51 | 0.65
Med42-70B + ZS | Macro avg. |  | 0.81
---
[0.78,0.83]
| 0.71
---
[0.67,0.74]
| 0.73
---
[0.70,0.76]

On the other hand, in table III, both ClinicalCamel-70B and Med42-70B achieve
over 0.80 macro F1 scores, which not only outperform Llama-2-70b-chat but also
Clinical-BigBird. This result suggests that ClinicalCamel-70B and Med42-70B
have learned substantial clinical knowledge from their pre-training and fine-
tuning stage, releasing the need for labeled training data to perform well on
a specific clinical task (i.e., identifying N category from pathology
reports).

TABLE III: Performance table for N category

Model | Class | Precision | Recall | F1-score
---|---|---|---|---
| N0 | 0.88 | 0.94 | 0.91
| N1 | 0.76 | 0.69 | 0.72
| N2 | 0.73 | 0.52 | 0.61
| N3 | 0.43 | 0.69 | 0.53
Clinical-BigBird | Macro avg. |  | 0.70
---
[0.65,0.74]
| 0.71
---
[0.66,0.76]
| 0.69
---
[0.64,0.74]
| N0 | 1.00 | 0.09 | 0.16
| N1 | 0.31 | 0.84 | 0.45
| N2 | 0.48 | 0.92 | 0.63
| N3 | 0.75 | 0.54 | 0.63
Llama-2-70b-chat + ZS | Macro avg. |  | 0.63
---
[0.58,0.68]
| 0.59
---
[0.54,0.64]
| 0.46
---
[0.41,0.51]
| N0 | 0.96 | 0.93 | 0.95
| N1 | 0.84 | 0.85 | 0.85
| N2 | 0.73 | 0.84 | 0.78
| N3 | 0.78 | 0.64 | 0.71
ClinicalCamel-70B + ZS | Macro avg. |  | 0.83
---
[0.78,0.87]
| 0.82
---
[0.77,0.87]
| 0.82
---
[0.77,0.86]
| N0 | 0.96 | 0.95 | 0.95
| N1 | 0.84 | 0.84 | 0.84
| N2 | 0.74 | 0.86 | 0.79
| N3 | 1.00 | 0.50 | 0.67
Med42-70B + ZS | Macro avg. |  | 0.88
---
[0.86,0.91]
| 0.79
---
[0.74,0.84]
| 0.81
---
[0.76,0.86]

In Table IV, Med42-70B performs the best among the other two LLMs, while
ClinicalCamel-70B shows a drop in the macro F1 score, performing worse than
Llama-2-70b-chat. Clinical-BigBird achieves the highest macro F1 score in M
stage classification tasks. When comparing class-specific performance between
Clinical-BigBird and Med42-70B, we observe that Clinical-BigBird only has a
low recall on M1 (0.19), suggesting it is biased at predicting the negative
class. On the contrary, Med42-70B has a higher recall (0.70) while showing a
low precision (0.13) in the M1 class. Given that the M1 class represents that
cancer has spread to distant parts of the patient’s body, the model with a
high recall in classifying M1 may help identify patients who need closer
monitoring and additional interventions to manage their disease.

TABLE IV: Performance table for M category

Model | Class | Precision | Recall | F1-score
---|---|---|---|---
| M0 | 0.94 | 0.98 | 0.96
| M1 | 0.47 | 0.19 | 0.27
Clinical-BigBird | Macro avg. |  | 0.71
---
[0.60,0.82]
| 0.59
---
[0.54,0.64]
| 0.62
---
[0.55,0.68]
| M0 | 0.97 | 0.53 | 0.69
| M1 | 0.11 | 0.80 | 0.19
Llama-2-70b-chat + ZS | Macro avg. |  | 0.54
---
[0.52,0.56]
| 0.67
---
[0.61,0.73]
| 0.44
---
[0.40,0.48]
| M0 | 0.99 | 0.25 | 0.40
| M1 | 0.09 | 0.98 | 0.16
ClinicalCamel-70B + ZS | Macro avg. |  | 0.54
---
[0.53,0.55]
| 0.61
---
[0.58,0.63]
| 0.28
---
[0.25,0.31]
| M0 | 0.97 | 0.66 | 0.79
| M1 | 0.13 | 0.70 | 0.22
Med42-70B + ZS | Macro avg. |  | 0.55
---
[0.53,0.57]
| 0.68
---
[0.61,0.75]
| 0.50
---
[0.46,0.54]

We also observe an interesting phenomenon: All models exhibit worse macro F1
scores for rare classes (i.e., T4, N3, and M1) than for common classes. The
three adopted LLMs have not been exposed to the class distribution but still
show performance degradation on rare classes. The reason may be the difficulty
in identifying rare classes by nature, but the finding indicates a potential
margin for performance improvement.

###  IV-B Performance Comparison: LLMs with Different Prompting Strategies

Table V reports the performance comparison of three LLMs using different
prompting strategies. We observe that the reasoning generated by the models
themselves benefits their classification performance. With ZS-COT,
Llama-2-70b-chat substantially improves macro F1 scores on all three
categories, reducing the performance gaps between it and the two clinical-
specific LLMs. ClinicalCamel-70B with ZS-COT retains the same macro F1 on the
T and N categories while improving macro F1 on the M category. ZS-COT also
assists Med42-70B in achieving higher macro F1 scores in all three categories.
As a result, we conclude that for this task, ZS-COT is a better prompting
strategy than ZS.

We observe mixed results when we use FS prompting. In most cases, the three
LLMs have worse macro F1 compared with ZS performance. The notable exception
is Med42-70B which reached a competent macro F1 score (0.62) on M category
compared with Clinical-BigBird. We attribute this performance degradation
effect to the fact that pathology reports in the TCGA dataset come from
different sources and are therefore written in different styles and formats.
Given that LLMs are sensitive to the prompt format [23], we point out this
observation for further studies.

TABLE V: Performance comparison of models using different prompting strategies

|  | Zero-shot | ZS-COT | Few-shots
---|---|---|---|---
| Model | Macro P | Macro R | Macro F1 | Macro P | Macro R | Macro F1 | Macro P | Macro R | Macro F1
T Category | Llama-2-70b-chat |  | 0.84
---
[0.82, 0.86]
| 0.66
---
[0.63,0.70]
| 0.69
---
[0.66,0.73]
| 0.82
---
[0.79,0.84]
| 0.72
---
[0.69,0.75]
| 0.75
---
[0.71,0.78]
| 0.74
---
[0.71,0.78]
| 0.68
---
[0.65,0.71]
| 0.69
---
[0.66,0.72]
ClinicalCamel-70B |  | 0.79
---
[0.76,0.82]
| 0.77
---
[0.75,0.80]
| 0.78
---
[0.75,0.80]
| 0.78
---
[0.75,0.81]
| 0.78
---
[0.76,0.81]
| 0.78
---
[0.75,0.81]
| 0.66
---
[0.63,0.69]
| 0.66
---
[0.62,0.69]
| 0.64
---
[0.61,0.67]
Med42-70B |  | 0.81
---
[0.78,0.83]
| 0.71
---
[0.67,0.74]
| 0.73
---
[0.70,0.76]
| 0.80
---
[0.77,0.83]
| 0.77
---
[0.74,0.80]
| 0.78
---
[0.75,0.81]
| 0.74
---
[0.71,0.78]
| 0.75
---
[0.71,0.78]
| 0.74
---
[0.71,0.78]
N Category | Llama-2-70b-chat |  | 0.63
---
[0.58,0.68]
| 0.59
---
[0.54,0.64]
| 0.46
---
[0.41,0.51]
| 0.69
---
[0.64,0.74]
| 0.76
---
[0.71,0.81]
| 0.70
---
[0.65,0.74]
| 0.61
---
[0.55,0.67]
| 0.57
---
[0.52,0.62]
| 0.47
---
[0.42,0.53]
ClinicalCamel-70B |  | 0.83
---
[0.78,0.87]
| 0.82
---
[0.77,0.87]
| 0.82
---
[0.77,0.86]
| 0.84
---
[0.79,0.89]
| 0.82
---
[0.77,0.87]
| 0.82
---
[0.78,0.87]
| 0.78
---
[0.75,0.81]
| 0.69
---
[0.63,0.76]
| 0.68
---
[0.60,0.74]
Med42-70B |  | 0.88
---
[0.86,0.91]
| 0.79
---
[0.74,0.84]
| 0.81
---
[0.76,0.86]
| 0.84
---
[0.78,0.89]
| 0.81
---
[0.76,0.86]
| 0.82
---
[0.77,0.87]
| 0.82
---
[0.76,0.88]
| 0.57
---
[0.52,0.63]
| 0.64
---
[0.58,0.70]
M Category | Llama-2-70b-chat |  | 0.54
---
[0.52,0.56]
| 0.67
---
[0.61,0.73]
| 0.44
---
[0.40,0.48]
| 0.55
---
[0.53,0.58]
| 0.68
---
[0.61,0.76]
| 0.51
---
[0.47,0.55]
| 0.51
---
[0.48,0.53]
| 0.51
---
[0.46,0.56]
| 0.19
---
[0.16,0.22]
ClinicalCamel-70B |  | 0.54
---
[0.53,0.55]
| 0.61
---
[0.58,0.63]
| 0.28
---
[0.25,0.31]
| 0.55
---
[0.53,0.58]
| 0.67
---
[0.59,0.75]
| 0.54
---
[0.49,0.58]
| 0.52
---
[0.50,0.54]
| 0.57
---
[0.50,0.62]
| 0.33
---
[0.29,0.36]
Med42-70B |  | 0.55
---
[0.53,0.57]
| 0.68
---
[0.61,0.75]
| 0.50
---
[0.46,0.54]
| 0.55
---
[0.53,0.58]
| 0.68
---
[0.61,0.76]
| 0.53
---
[0.49,0.57]
| 0.62
---
[0.56,0.69]
| 0.62
---
[0.56,0.69]
| 0.62
---
[0.56,0.68]

Table VI reports the bootstrapping t-test results to compare the difference in
macro F1 between Clinical-BigBird and the best model + prompting strategy in
each category, selected from Table V. In the table, we report t-statistics
(macro F1 of Clinical-BigBird minus macro F1 of the selected model) and
indicate the significance ∗∗∗ if p-value $<0.05$. Our results show that
Clinical Big Bird is significantly better than the best LLM, Med42-70B + ZS-
COT, in the T category. However, the best model in the N category, Med42-70B +
ZS-COT, significantly outperforms Clinical Big Bird. The Med42-70B + FS, the
best model for the M category, has a macro F1 comparable to Clinical-BigBird.

TABLE VI: Paired T-test: Difference of Macro F1 between Clinical-BigBird and
Best Model selected from Table V

|  | T Category
---
(Med42-70B +
ZS-COT)
| N Category
---
(Med42-70B +
ZS-COT)
| M Category
---
(Med42-70B +
FS)
Clinical-BigBird | $29.23^{***}$ | $-80.74^{***}$ | -0.70

###  IV-C Performance Comparison by Cancer Type

We conduct further analysis based on the cancer type available in the TCGA
dataset. Of the top most frequently diagnosed cancers in U.S. in 2023, breast,
prostate, lung and bronchus, and colon and rectum cancer [24], we select
breast (BRCA - breast invasive carcinoma) and lung (LUAD - lung
adenocarcinoma) for further analysis as they have the largest number of
reports in the dataset. We exclude prostate and colon cancers due to smaller
sample sizes and lack of full representation of all T, N, or M categories.
Table VII shows comparison of the best Med42 prompting strategy vs the
benchmark Clinical-BigBird for these cancers. By the macro F1 score we note
that unlike with the full testing dataset, Med42 outperforms Clinical-BigBird
in T classification for both BRCA and LUAD cancers. Consistent with the
previous results on the full corpus, Med42 outperforms Clinical-BigBird in N
classification. For M classification of BRCA Med42 has better macro F1 than
Clinical-BigBird and only slightly worse performance for LUAD. We note that
metastatic cases are rare in the TCGA dataset. There are 132 M0 and 6 M1 cases
for BRCA and 59 M0 and 4 M1 cases for LUAD. Clinical Big Bird identifies none
of the M1 cases in BRCA and LUAD. Med42+FS identifies 4 M1 cases in BRCA,
while it cannot identify M1 cases in LUAD. In curating and using this data
corpus for TNM classification, Kefeli and Tatonetti [8] report several
limitations regarding M01 classification, namely, that many of the pathology
reports do not explicitly contain M0 or M1, unlike the case for T and N
status, and that TCGA annotations of M01 are occasionally inconsistent with
the report text. These factors may explain the low performance of the models
in the M classification task.

To understand the variations in performance by type of cancer we hypothesize
that reports for a particular type of cancer may have been sourced from the
same facility. However, we are unable to confirm this as TCGA metadata does
not contain a facility or source identifier. These results do suggest to us
that there is a margin for improvement by customizing the prompting
instructions to the type of cancer. This is a focus of our ongoing work.

TABLE VII: Performance Comparison for two cancer types: BRCA and LUAD

|  | BRCA | LUAD
---|---|---|---
| Model | Macro P | Macro R | Macro F1 | Macro P | Macro R | Macro F1
T Category | Clinical-BigBird |  | 0.72
---
[0.55,0.90]
| 0.61
---
[0.51,0.75]
| 0.64
---
[0.53,0.79]
| 0.84
---
[0.76,0.91]
| 0.85
---
[0.76,0.92]
| 0.84
---
[0.75,0.91]
Med42+ZS-COT |  | 0.74
---
[0.55,0.89]
| 0.72
---
[0.56,0.88]
| 0.72
---
[0.56,0.86]
| 0.92
---
[0.83,0.97]
| 0.92
---
[0.85,0.97]
| 0.91
---
[0.83,0.97]
N Category | Clinical-BigBird |  | 0.70
---
[0.59,0.79]
| 0.71
---
[0.60,0.83]
| 0.69
---
[0.58,0.79]
| 0.52
---
[0.37,0.80]
| 0.51
---
[0.38,0.74]
| 0.50
---
[0.38,0.75]
Med42+ZS-COT |  | 0.81
---
[0.70,0.91]
| 0.81
---
[0.71,0.91]
| 0.80
---
[0.70,0.90]
| 0.89
---
[0.77,0.98]
| 0.92
---
[0.82,0.99]
| 0.89
---
[0.79,0.98]
M Category | Clinical-BigBird |  | 0.48
---
[0.46,0.49]
| 0.50
---
[0.50,0.50]
| 0.49
---
[0.48,0.50]
| 0.48
---
[0.44,0.49]
| 0.51
---
[0.5,0.5]
| 0.51
---
[0.40,0.66]
Med42+FS |  | 0.57
---
[0.49,0.66]
| 0.74
---
[0.45,0.95]
| 0.58
---
[0.47,0.71]
| 0.48
---
[0.43,0.49]
| 0.51
---
[0.5,0.5]
| 0.49
---
[0.46,0.50]

##  V Discussion

In this study we evaluated the ability of open-source LLMs to extract the
pathologic TNM stage from real-world pathology reports. Using one general
purpose (Llama-2-70b-chat) and two clinical (Med42-70B and ClinicalCamel-70B)
LLMs we compared different standard prompting approaches (zero-shot, few-
shots, and zero-shot chain-of-thought) in performing this task. We also
compared the performance of these three LLMs with a pre-trained Clinical-
BigBird model that has been fine-tuned with a training set of these reports as
a benchmark.

Our findings suggest that open-source generative LLMs are able to perform as
well as or even better than the benchmark model that is fine-tuned on the same
dataset. The implications of these findings are significant - using locally
hosted LLMs, without the need for any training data, we are able to achieve
comparable or better performance on extracting TNM stage information from
real-world pathology reports. Because the LLMs we tested are not fine-tuned
for this specific task or dataset, we infer that they have the potential to
perform well on other clinical notes and tasks, since no training data is
needed.

In comparing prompting approaches, we find that the few-shots approach does
not appear to improve the performance of the LLMs significantly. The TCGA
pathology reports come from multiple institutions, and there does not appear
to be a standardized structure or formatting system, reflecting the diversity
of documentation standards and styles across institutions and pathologists. We
hypothesize because of this diversity, it may be difficult to craft
sufficiently generalizable shots for the entire corpus. Verifying and
mitigating this issue is the focus of our ongoing work and experiments.

We additionally observed in our experiments that the open-source LLMs appear
to be very sensitive to the system instruction and to the prompt structure. In
our experiments, minor variations led to widely varying results, as reported
by [23]. This has important implications if these LLMs are to be adopted for
real world clinical tasks in the healthcare setting. Rigorous prompt testing
and engineering needs to be implemented and evaluated to ensure the best
performance and outcomes.

###  V-A Strengths and Limitations

Our study has several strengths. We use real world data that has not been
extensively preprocessed to address spelling or formatting issues, and
therefore the results from our experiments are likely a true reflection of how
these models would perform if embedded in a clinical setting or application.
Our approach does not utilize any fine-tuning, thus reducing the human effort
required to curate and annotate training data. We utilize and evaluate locally
hosted open-source LLMs, reducing the risks of PHI leakage, and the potential
long-term costs involved in using commercial LLMs.

We do note some limitations to our approach. We evaluate the performance using
the testing dataset for a fair comparison with the previous fine-tuned model
(i.e., Clinical Big-Bird). Increasing the size of the dataset and evaluating
the performance using the whole dataset may allow cancer-specific performance
analysis (e.g., Table VII) for more cancer types, providing insights to
develop cancer-specific prompting strategies and instructions. The inference
speed using the LLMs is significantly slower than that of the pre-trained
Clinical-BigBird. However, developments in techniques to improve inference
speed, such as quantization of models and other approaches, may allow us to
overcome this barrier. We only evaluate Llama-based models in our study, and
thus the results may not be generalizable to other open-source models.

##  VI Conclusion

Ascertaining cancer stage from pathology reports is a real-world, clinically
relevant, and important task in the management of cancer patients, and for
researchers who are working to understand and improve cancer outcomes. The
advent of large language models (LLMs) has created an unprecedented
opportunity in healthcare for processing of free-text clinical notes and
accelerating research. In this study we demonstrated that locally hosted open-
source medical LLMs are able to extract cancer staging information from real
world pathology reports without the use of training data. Using standard
prompting approaches we obtained comparable performance to a pre-trained model
that has been fine-tuned on this same data. The potential to use these open-
source medical LLMs across different tasks can be inferred from the high
performance obtained without any specific fine-tuning. Exploring how to
improve the creation of generalizable few shots to further improve the
performance, as well as experimenting with novel and more effective prompting
techniques for clinical tasks remains the focus of future work.

## Acknowledgment

This work was supported in part by the National Science Foundation under the
Grants IIS-1741306 and IIS-2235548, and by the Department of Defense under the
Grant DoD W91XWH-05-1-023. This material is based upon work supported by
(while serving at) the National Science Foundation. Any opinions, findings,
and conclusions or recommendations expressed in this material are those of the
author(s) and do not necessarily reflect the views of the National Science
Foundation.

## References

  * [1] “Cancer of Any Site - Cancer Stat Facts.” [Online]. Available: https://seer.cancer.gov/statfacts/html/all.html
  * [2] “FastStats,” Jan. 2024. [Online]. Available: https://www.cdc.gov/nchs/fastats/leading-causes-of-death.htm
  * [3] “Cancer Staging Systems.” [Online]. Available: https://www.facs.org/quality-programs/cancer-programs/american-joint-committee-on-cancer/cancer-staging-systems/
  * [4] “American Joint Committee on Cancer | SEER Training.” [Online]. Available: https://training.seer.cancer.gov/staging/systems/ajcc/
  * [5] S. Gao, M. T. Young, J. X. Qiu, H.-J. Yoon, J. B. Christian, P. A. Fearn, G. D. Tourassi, and A. Ramanthan, “Hierarchical attention networks for information extraction from cancer pathology reports,” _Journal of the American Medical Informatics Association_ , vol. 25, no. 3, pp. 321–330, Mar. 2018. [Online]. Available: https://academic.oup.com/jamia/article/25/3/321/4636780
  * [6] S. Gao, J. X. Qiu, M. Alawad, J. D. Hinkle, N. Schaefferkoetter, H.-J. Yoon, B. Christian, P. A. Fearn, L. Penberthy, X.-C. Wu, L. Coyle, G. Tourassi, and A. Ramanathan, “Classifying cancer pathology reports with hierarchical self-attention networks,” _Artificial Intelligence in Medicine_ , vol. 101, p. 101726, Nov. 2019. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S0933365719303562
  * [7] J. Wu, K. Tang, H. Zhang, C. Wang, and C. Li, “Structured Information Extraction of Pathology Reports with Attention-based Graph Convolutional Network,” in _2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_ , Dec. 2020, pp. 2395–2402. [Online]. Available: https://ieeexplore.ieee.org/document/9313347
  * [8] J. Kefeli and N. Tatonetti, “Generalizable and Automated Classification of TNM Stage from Pathology Reports with External Validation,” _medRxiv_ , p. 2023.06.26.23291912, Jun. 2023. [Online]. Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10327265/
  * [9] K. De Angeli, S. Gao, M. Alawad, H.-J. Yoon, N. Schaefferkoetter, X.-C. Wu, E. B. Durbin, J. Doherty, A. Stroup, L. Coyle, L. Penberthy, and G. Tourassi, “Deep active learning for classifying cancer pathology reports,” _BMC Bioinformatics_ , vol. 22, no. 1, p. 113, Mar. 2021. [Online]. Available: https://doi.org/10.1186/s12859-021-04047-1
  * [10] A. Y. Odisho, B. Park, N. Altieri, J. DeNero, M. R. Cooperberg, P. R. Carroll, and B. Yu, “Natural language processing systems for pathology parsing in limited data environments with uncertainty estimation,” _JAMIA Open_ , vol. 3, no. 3, pp. 431–438, Oct. 2020, publisher: Oxford Academic. [Online]. Available: https://dx.doi.org/10.1093/jamiaopen/ooaa029
  * [11] C. Wu, W. Lin, X. Zhang, Y. Zhang, Y. Wang, and W. Xie, “PMC-LLaMA: Towards Building Open-source Language Models for Medicine,” Aug. 2023, arXiv:2304.14454 [cs]. [Online]. Available: http://arxiv.org/abs/2304.14454
  * [12] T. Han, L. C. Adams, J.-M. Papaioannou, P. Grundmann, T. Oberhauser, A. Löser, D. Truhn, and K. K. Bressem, “MedAlpaca – An Open-Source Collection of Medical Conversational AI Models and Training Data,” Oct. 2023, arXiv:2304.08247 [cs]. [Online]. Available: http://arxiv.org/abs/2304.08247
  * [13] A. Toma, P. R. Lawler, J. Ba, R. G. Krishnan, B. B. Rubin, and B. Wang, “Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding,” Aug. 2023, arXiv:2305.12031 [cs]. [Online]. Available: http://arxiv.org/abs/2305.12031
  * [14] J. Kefeli and N. Tatonetti, “TCGA-Reports: A machine-readable pathology report resource for benchmarking text-based AI models,” _Patterns_ , Feb. 2024, publisher: Elsevier. [Online]. Available: https://www.cell.com/patterns/abstract/S2666-3899(24)00024-2
  * [15] Y. Li, R. M. Wehbe, F. S. Ahmad, H. Wang, and Y. Luo, “Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences,” 2022, publisher: arXiv Version Number: 3. [Online]. Available: https://arxiv.org/abs/2201.11838
  * [16] J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang, “BioBERT: a pre-trained biomedical language representation model for biomedical text mining,” _Bioinformatics_ , vol. 36, no. 4, pp. 1234–1240, Feb. 2020. [Online]. Available: https://doi.org/10.1093/bioinformatics/btz682
  * [17] K. Huang, J. Altosaar, and R. Ranganath, “ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission,” Nov. 2020, arXiv:1904.05342 [cs]. [Online]. Available: http://arxiv.org/abs/1904.05342
  * [18] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom, “Llama 2: Open Foundation and Fine-Tuned Chat Models,” Jul. 2023, arXiv:2307.09288 [cs]. [Online]. Available: http://arxiv.org/abs/2307.09288
  * [19] Y. Li, Z. Li, K. Zhang, R. Dan, and Y. Zhang, “ChatDoctor: A Medical Chat Model Fine-tuned on LLaMA Model using Medical Domain Knowledge,” Apr. 2023, arXiv:2303.14070 [cs]. [Online]. Available: http://arxiv.org/abs/2303.14070
  * [20] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large Language Models are Zero-Shot Reasoners,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 22 199–22 213, Dec. 2022. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2022/hash/8bb0d291acd4acf06ef112099c16f326-Abstract-Conference.html
  * [21] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language Models are Few-Shot Learners,” in _Advances in Neural Information Processing Systems_ , vol. 33. Curran Associates, Inc., Jul. 2020, pp. 1877–1901.
  * [22] B. Efron and R. Tibshirani, _An introduction to the bootstrap_ , ser. Monographs on statistics and applied probability. New York: Chapman & Hall, 1993, no. 57.
  * [23] M. Sclar, Y. Choi, Y. Tsvetkov, and A. Suhr, “Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting,” Oct. 2023, arXiv:2310.11324 [cs]. [Online]. Available: http://arxiv.org/abs/2310.11324
  * [24] “Common Cancer Sites - Cancer Stat Facts.” [Online]. Available: https://seer.cancer.gov/statfacts/html/common.html


# Adaptive Attribute and Structure Subspace Clustering Network

Zhihao Peng, Hui Liu, Yuheng Jia, , Junhui Hou This work was supported by the
Hong Kong RGC under Grants CityU 11219019 and 11202320. Corresponding author:
Junhui Hou.Z. Peng, H. Liu, and J. Hou are with the Department of Computer
Science, City University of Hong Kong, Kowloon, Hong Kong 999077 (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>jh.hou@cityu.edu.hk)Y.
Jia is with the School of Computer Science and Engineering, Southeast
University, Nanjing 210096, China, and also with Key Laboratory of Computer
Network and Information Integration (Southeast University), Ministry of
Education, China (e-mail: yhjia@seu.edu.cn).

###### Abstract

Deep self-expressiveness-based subspace clustering methods have demonstrated
effectiveness. However, existing works only consider the attribute information
to conduct the self-expressiveness, which may limit the clustering
performance. In this paper, we propose a novel adaptive attribute and
structure subspace clustering network (AASSC-Net) to simultaneously consider
the attribute and structure information in an adaptive graph fusion manner.
Specifically, we first exploit an auto-encoder to represent input data samples
with latent features for the construction of an attribute matrix. We also
construct a mixed signed and symmetric structure matrix to capture the local
geometric structure underlying data samples. Then, we perform self-
expressiveness on the constructed attribute and structure matrices to learn
their affinity graphs separately. Finally, we design a novel attention-based
fusion module to adaptively leverage these two affinity graphs to construct a
more discriminative affinity graph. Extensive experimental results on commonly
used benchmark datasets demonstrate that our AASSC-Net significantly
outperforms state-of-the-art methods. In addition, we conduct comprehensive
ablation studies to discuss the effectiveness of the designed modules. The
code will be publicly available at https://github.com/ZhihaoPENG-CityU.

###### Index Terms:

Deep learning, subspace clustering, self-expressiveness, structure
information.

##  I Introduction

Clustering, which aims to partition samples into multiple groups, is broadly
employed in many real-world applications, such as image recognition [1, 2, 3,
4, 5], object segmentation [6, 7, 8], and text clustering [9, 10, 11, 12]. In
the last decade, numerous researchers have paid attention to the self-
expressiveness (SE)-based subspace clustering, which includes two main steps.
First, the SE priori is exploited to learn an affinity graph by assuming each
sample can be expressed as a linear combination of other samples within the
same subspace. Then, spectral clustering [13] is applied to the resulting
affinity graph to seek the clustering results. Obviously, learning a
clustering-friendly affinity graph plays a critical role in clustering, and
accordingly, a family of approaches based on different kinds of regularization
were proposed. For example, Elhamifar _et al._ [14] employed the sparse
regularization to find a nontrivial sparse representation. Ji _et al._ [15]
designed the Tikhonov regularization to yield dense connections within each
cluster. Liu _et al._ [16] developed the low-rank regularization to explore
the lowest rank representation. You _et al._ [17] further designed the elastic
net regularization to balance the subspace-preserving (sparse solutions) and
connectivity (dense solutions). Although these works have obtained impressive
clustering performance, one primary limitation is that they cannot handle data
with complex nonlinearity well. To this end, some kernel extensions of the
existing subspace clustering algorithms [18, 19, 20] were investigated to
explore the nonlinearity of data. Unfortunately, the clustering performance of
these methods heavily relies on the choice of the kernel function, while a
confident rule for selecting the kernel function is still absent in practice.

(a) Previous deep SE-based methods (b) Our method

Figure 1: The flowcharts of (a) previous deep SE-based methods and (b) the
proposed method. Here, the ‘ _attribute information_ ’ refers to the feature
representations of samples in a latent space, and the ‘ _structure
information_ ’ to the similarity relationship of a typical sample with the
other samples.

Benefiting from the powerful representation ability of deep learning, deep SE-
based subspace clustering has a tremendous step, effectively handling the
issues mentioned above. For example, Hinton _et al._ [21] designed the auto-
encoder (AE) method to map the raw data to a nonlinear latent space, in which
the clustering results can be obtained by imposing a traditional subspace
clustering (e.g., sparse subspace clustering [22]) on the latent feature
representations. Ji _et al._ [23] developed a deep subspace clustering (DSC)
framework by inserting a fully-connected layer between the encoder and the
decoder to mimic the SE priori. Afterward, numerous DSC-based works were
proposed by introducing additional regularization terms, such as adversarial
learning [24] and pseudo-supervision [25]. All these deep SE-based approaches
construct an affinity graph for clustering by performing the SE on the
attribute information of latent feature representations. However, we argue
that the geometric structure information underlying data samples is also
valuable for clustering, i.e., for two samples belonging to an identical
subspace, their similarity relationships with the other samples are typically
similar [26]. Thus, it is expected that the clustering performance can be
boosted by exploring such geometric structure information.

In this paper, we propose a novel deep SE-based subspace clustering framework
to simultaneously consider the attribute and structure information of data in
an adaptive graph fusion manner, as shown in Figure 1 (b). Specifically, we
first adopt an AE to extract a latent feature representation as the attribute
matrix. We also exploit the extracted latent representation to construct a
structure matrix to capture the local geometric structure underlying data
samples. Since the underlying geometric structure of a sample is generally
similar to those of other samples within the same subspace, our method
performs the SE on not only the attribute matrix but also the structure matrix
to learn their affinity graphs. Precisely, we achieve these two SE-based
reconstructions via two fully-connected layers without the bias and activation
function. Finally, we propose an attention-based fusion module to adaptively
leverage those two affinity graphs to construct a more discriminative one. We
validate the superiority of the proposed method over state-of-the-art methods
by conducting quantitative and qualitative results on commonly used benchmark
datasets. In addition, to better understand the proposed network, we conduct
comprehensive ablation studies of the designed modules.

The rest of this paper is organized as follows. We review some related works
in Section II and introduce our method in Section III. Section IV presents the
experimental results and analyses. Finally, we conclude this paper in Section
V.

##  II Related Work

We summarize the main notations in Table I, where operators are denoted by
calligraphy ones, matrices by bold upper case ones, vectors by bold lower case
letters, and scalars by italic lower case letters, respectively. Given a
matrix $\mathbf{A}\in\mathbb{R}^{m\times n}$, $\left\|\mathbf{A}\right\|_{F}$
denotes the Frobenius norm, i.e.,
$\left\|\mathbf{A}\right\|_{F}=\sqrt{\sum_{i=1}^{m}\sum_{j=1}^{n}{\left|a_{i,j}\right|}^{2}}$.

TABLE I: Summary of notations used in this paper. Notation | Description
---|---
$\mathbf{X}\in\mathbb{R}^{\hat{d}\times n}$ | The raw data
$\hat{\mathbf{X}}\in\mathbb{R}^{\hat{d}\times n}$ | The reconstructed data
$\hat{\mathbf{Z}}\in\mathbb{R}^{p_{1}\times p_{2}\times p_{3}\times n}$ | The extracted feature
$\mathbf{Z}\in\mathbb{R}^{d\times n}$ | The matrix form of $\hat{\mathbf{Z}}$
$\mathbf{Z}^{\mathsf{T}}\in\mathbb{R}^{n\times d}$ | The matrix transpose of $\mathbf{Z}$
$\mathbf{Z}_{\emph{A}}\in\mathbb{R}^{d\times n}$ | The attribute matrix
$\mathbf{Z}_{\emph{S}}\in\mathbb{R}^{n\times n}$ | The structure matrix
$\mathbf{C}_{\emph{A}}\in\mathbb{R}^{n\times n}$ | The learned affinity matrix of $\mathbf{Z}_{\emph{A}}$
$\mathbf{C}_{\emph{S}}\in\mathbb{R}^{n\times n}$ | The learned affinity matrix of $\mathbf{Z}_{\emph{S}}$
$\mathbf{C}_{\emph{F}}\in\mathbb{R}^{n\times n}$ | The fused affinity matrix
$\mathbf{I}$ | The identity matrix
$\Theta_{E}$ | The encoder parameters
$\Theta_{D}$ | The decoder parameters
$n$ | The number of samples
$\hat{d}$ | The dimension of $\mathbf{X}$
$p_{1}\times p_{2}\times p_{3}$ | The dimension of $\hat{\mathbf{Z}}$
$d$ | The dimension of $\mathbf{Z}$
$\left|\cdot\right|$ | The absolute value operation
$\left\|\cdot\right\|_{F}$ | The Frobenius norm of a matrix
$\emph{f}_{\emph{E}}$ | The nonlinear mapping function of the encoder
$\emph{f}_{\emph{D}}$ | The nonlinear mapping function of the decoder
$\emph{f}_{\emph{AE}}$ | The nonlinear mapping function of the AE
$\emph{f}_{\emph{AF}}$ | The attention-based fusion module to obtain $\mathbf{C}_{\emph{F}}$
Figure 2: Illustration of the proposed framework, consisting of a feature
representation module, a dual self-expressiveness (SE)-based module, and an
attention-based fusion module. The feature representation module composed of a
series of convolutional layers first generates a nonlinear low-dimensional
representation for the raw data, then the dual SE-based module constructed
with two fully-connected layers without the bias and activation function
learns affinity graphs by exploring both attribute and structure information
of data samples, and the attention-based fusion module with a multilayer
perceptron finally fuses those two affinity graphs adaptively by performing a
weighted sum operation. The clustering results can be obtained by performing
spectral clustering on the fused affinity graph.

###  II-A Traditional SE-based Methods

Let
$\mathbf{X}=[\mathbf{x}_{1},...,\mathbf{x}_{n}]\in\mathbb{R}^{\hat{d}\times
n}$ be the raw data with $\hat{d}$ and $n$ being the dimension and number of
samples, respectively. We can generally formulate the traditional SE-based
models as

$\displaystyle\min_{\mathbf{C}}f\left(\mathbf{C}\right)\quad\rm{s.t.}\quad\mathbf{X}=\mathbf{X}\mathbf{C},\quad
diag(\mathbf{C})=\mathbf{0},$ (1)

where $\mathbf{C}\in\mathbb{R}^{n\times n}$ is a learned affinity matrix,
$diag(\mathbf{C})=\mathbf{0}$ is used to prevent the trivial solution
$\mathbf{C}=\mathbf{I}$ with $\mathbf{I}$ being the identity matrix, and
$\emph{f}\left(\mathbf{C}\right)$ is used to regularize $\mathbf{C}$ to
promote a unique and meaningful solution. By imposing different kinds of
regularization on $\mathbf{C}$, a family of works were proposed [14, 15, 17,
27, 28, 29, 30, 31, 32, 33, 34]. For example, Elhamifar _et al._ [14]
exploited a sparse subspace clustering (SSC) method to find a nontrivial
sparse representation for $\mathbf{C}$. You _et al._ [35] improved SSC by
introducing an orthogonal matching pursuit. Ji _et al._ [15] proposed an
efficient dense subspace clustering (EDSC) method to yield dense connections
within each cluster. Liu _et al._ [16] developed a low-rank representation
(LRR) method to seek the lowest rank representation among samples. You _et
al._ [17] further designed an elastic net subspace clustering (ENSC) method to
balance the subspace-preserving (sparse solutions) and connectivity (dense
solutions). Although these methods can produce a meaningful $\mathbf{C}$, they
only investigate the linear relationship among samples. To this end, Patel _et
al._ [18] explored the nonlinear relationship in the input space by kernel
sparse subspace clustering (KSSC). However, determining the kernel function
with the associated hyperparameters is generally challenging [36].

###  II-B Deep SE-based Methods

Deep SE-based methods [21, 37, 38, 39, 40] aim to achieve clustering by
exploiting the deep nonlinear representation ability. For example, Hinton _et
al._ [21] proposed the auto-encoder (AE) method to extract a latent feature
representation, in which the clustering can be achieved by imposing SSC on the
latent feature representation. Ji _et al._ [23] designed a deep subspace
clustering network framework (DSC-Net) to mimic the SE priori and developed
the DSC-Net-L1 method with the sparse regularization and the DSC-Net-L2 method
with the Tikhonov regularization. Zhou _et al._ [24] proposed deep adversarial
subspace clustering (DASC) to supervise the deep representation learning with
adversarial learning. Lv _et al._ [25] used a pseudo-supervised deep subspace
clustering (PSSC) approach to weigh the AE reconstruction by integrating the
local structure-preserving, SE rule, and pseudo-supervision. However, these
previous deep SE-based approaches only focus on the attribute information and
barely the geometric structure information to conduct the SE, which inevitably
depresses the discriminative ability of the learned affinity graph, as both
those two kinds of information are of great importance.

##  III Proposed Method

As shown in Figure 2, the proposed framework consists of three key modules,
i.e., a feature representation module, a dual self-expressiveness (SE)-based
module, and an attention-based fusion module, which will be detailed one by
one as follows.

###  III-A Feature Representation Module

We use an auto-encoder (AE) composed of a series of encoders and decoders to
project the raw data $\mathbf{X}$ into a nonlinear latent feature space. Let
$\Theta_{E}$ and $\Theta_{D}$ be the encoder and decoder parameters,
respectively, the optimization problem of the AE can be written as

$\displaystyle\min_{\mathbf{Z}}\frac{1}{2}\left\|\mathbf{X}-\hat{\mathbf{X}}\right\|^{2}_{F}\quad\rm{s.t.}\quad\mathbf{Z}=\emph{f}_{\emph{AE}}\left(\mathbf{X}\right),$
(2)

where $\hat{\mathbf{Z}}\in\mathbb{R}^{p_{1}\times p_{2}\times p_{3}\times n}$
is the extracted latent feature representation that reshaped as a data matrix
$\mathbf{Z}\in\mathbb{R}^{d\times n}$,
$\hat{\mathbf{X}}=f_{D}(\hat{\mathbf{Z}};\Theta_{D})$ is the output of the
decoder, $\emph{f}_{\emph{AE}}\left(\cdot\right)$ denotes the nonlinear
mapping function of the AE, and
$\left\|\mathbf{X}-\hat{\mathbf{X}}\right\|^{2}_{F}$ measures the AE-based
reconstruction loss between the input $\mathbf{X}$ and the output
$\hat{\mathbf{X}}$.

###  III-B Dual SE-based Module

As aforementioned, previous deep SE-based methods only explore the attribute
information of $\mathbf{Z}$, while neglecting the underlying geometric
structure information, which is also of great importance, and thus, the
discriminative ability of the learned affinity graph is still limited. To this
end, we propose the dual SE-based module to simultaneously explore both the
attribute and structure information of $\mathbf{Z}$.

####  III-B1 Attribute SE Formulation

To learn an affinity graph from the attribute information, we use a fully-
connected layer without the bias and activation function to mimic the SE
process. By imposing the Frobenius norm, the attribute SE formulation can be
written as

$\displaystyle\min_{\mathbf{C}_{\emph{A}}}\left\|\mathbf{C}_{\emph{A}}\right\|^{2}_{F}+\gamma\left\|\mathbf{Z}_{\emph{A}}-\mathbf{Z}_{\emph{A}}\mathbf{C}_{\emph{A}}\right\|^{2}_{F},$
(3)

where $\mathbf{Z}_{\emph{A}}$ is the attribute matrix,
$\mathbf{C}_{\emph{A}}\in\mathbb{R}^{n\times n}$ is the learned affinity graph
from $\mathbf{Z}_{\emph{A}}$,
$\left\|\mathbf{Z}_{\emph{A}}-\mathbf{Z}_{\emph{A}}\mathbf{C}_{\emph{A}}\right\|^{2}_{F}$
measures the attribute SE loss. Here, as the AE already owns excellent low-
dimensional feature representation ability, we set
$\mathbf{Z}_{\emph{A}}=\mathbf{Z}$. In addition, we drop the constraint
$diag(\mathbf{C}_{\emph{A}})=\mathbf{0}$ since here we minimize the Frobenius
norm of $\mathbf{C}_{\emph{A}}$, and the identity matrix $\mathbf{I}$ has
large values of the Frobenius norm. That is, the optimization problem in Eq.
(3) automatically prevents the trivial solution
$\mathbf{C}_{\emph{A}}=\mathbf{I}$.

####  III-B2 Structure SE Formulation

As $\mathbf{C}_{\emph{A}}$ obtained from Eq. (3) naturally encodes the
relationships among samples, we empirically construct a structure matrix
$\mathbf{Z}_{\emph{S}}$ for capturing the underlying geometric structure
information of $\mathbf{Z}$, i.e.,

$\displaystyle\mathbf{Z}_{\emph{S}}=\frac{\mathbf{C}_{\emph{A}}+\mathbf{C}_{\emph{A}}^{\mathsf{T}}}{2},$
(4)

where the reason for the symmetrization strategy is to make sure that nodes
$i$ and $j$ get connected to each other if either $\mathbf{x}_{i}$ or
$\mathbf{x}_{j}$ is in the representation of the other, the reason for the
mixed-signs strategy is that the negative values in the coefficient matrix
include the discriminative information among samples. Since the manner of
constructing the structure matrix $\mathbf{Z}_{\emph{S}}$ is critical, we
investigate various construction methods in Section IV-F. In addition, based
on the observation that the geometric structure information of a typical
sample can be reconstructed by those of other samples within the same
subspace, we argue that $\mathbf{Z}_{\emph{S}}$ should also meet the SE
priori. Therefore, we introduce another fully-connected layer without the bias
and activation function to mimic the SE process on $\mathbf{Z}_{\emph{S}}$. By
imposing the Frobenius norm, the structure SE formulation can be written as

$\displaystyle\min_{\mathbf{C}_{\emph{S}}}\lambda_{1}\left\|\mathbf{C}_{\emph{S}}\right\|^{2}_{F}+\lambda_{2}\left\|\mathbf{Z}_{\emph{S}}-\mathbf{Z}_{\emph{S}}\mathbf{C}_{\emph{S}}\right\|^{2}_{F},$
(5)

where $\mathbf{C}_{\emph{S}}\in\mathbb{R}^{n\times n}$ is the learned affinity
graph from $\mathbf{Z}_{\emph{S}}$,
$\left\|\mathbf{Z}_{\emph{S}}-\mathbf{Z}_{\emph{S}}\mathbf{C}_{\emph{S}}\right\|^{2}_{F}$
measures the structure SE loss, and $\lambda_{1}$ and $\lambda_{2}$ are two
trade-off parameters. Here, we drop the constraint
$diag(\mathbf{C}_{\emph{S}})=\mathbf{0}$, whose reason is the same as that in
Eq. (3).

###  III-C Attention-based Fusion Module

After obtaining $\mathbf{C}_{\emph{A}}$ and $\mathbf{C}_{\emph{S}}$, we fuse
them together to construct the final affinity graph for spectral clustering.
Here, we propose a novel attention-based fusion module. Specifically, we first
concatenate $\mathbf{C}_{\emph{A}}$ and $\mathbf{C}_{\emph{S}}$ as
$[\mathbf{C}_{\emph{A}}~{}\mathbf{C}_{\emph{S}}]\in\mathbb{R}^{n\times{2n}}$,
and then introduce a weight matrix $\mathbf{W}\in\mathbb{R}^{{2n}\times 2}$ by
learning their attentional coefficients to capture the relationship between
$\mathbf{C}_{\emph{A}}$ and $\mathbf{C}_{\emph{S}}$. After that, the LeakyReLU
[41] activation is applied to the product of
$\left[\mathbf{C}_{\emph{A}}~{}\mathbf{C}_{\emph{S}}\right]$ and $\mathbf{W}$,
where the negative input slope of LeakyReLU is set as $0.2$. We then normalize
the output of the LeakyReLU unit via a softmax function and an $\ell_{2}$
normalization. The overall process is written as

$\displaystyle\mathbf{M}=\ell_{2}\left(\rm{softmax}\left(\left(\rm{LeakyReLU}\left(\left[\mathbf{C}_{\emph{A}}~{}\mathbf{C}_{\emph{S}}\right]\mathbf{W}\right)\right)\right)\right),$
(6)

where $\mathbf{M}=[\mathbf{m}_{1}~{}\mathbf{m}_{2}]\in\mathbb{R}^{n\times 2}$
is a weight matrix with entries being greater than $0$, and
$\mathbf{m}_{1}\in\mathbb{R}^{n\times 1}$ and
$\mathbf{m}_{2}\in\mathbb{R}^{n\times 1}$ are the weight vectors for measuring
the importance of $\mathbf{C}_{\emph{A}}$ and $\mathbf{C}_{\emph{S}}$,
respectively. Accordingly, we adaptively fuse $\mathbf{C}_{\emph{A}}$ and
$\mathbf{C}_{\emph{S}}$ as

$\displaystyle\mathbf{C}_{\emph{F}}$
$\displaystyle=\emph{f}_{\emph{AF}}\left(\mathbf{C}_{\emph{A}},\mathbf{C}_{\emph{S}}\right)$
(7)
$\displaystyle=\left(\mathbf{m}_{1}\mathbf{1}\right)\odot\mathbf{C}_{\emph{A}}+\left(\mathbf{m}_{2}\mathbf{1}\right)\odot\mathbf{C}_{\emph{S}},$

where $\mathbf{C}_{\emph{F}}\in\mathbb{R}^{n\times n}$ is the fused affinity
matrix, $\mathbf{1}\in\mathbb{R}^{1\times n}$ is a vector with all entries
being one, and $\odot$ denotes the Hadamard product of matrices. We can
finally utilize $\mathbf{C}_{F}$ to conduct the spectral clustering to obtain
the clustering results.

###  III-D Training Settings

Similar to [24, 25], we first pre-train the proposed network and then fine-
tune it.

####  III-D1 Pre-training Stage

As the network is unsupervised and challenging to infer its parameters, we
first singly pre-train the feature representation module (i.e., without the
dual SE-based module and the attention-based fusion module) to obtain a good
initialization for the network parameters. Specifically, we only minimize the
AE reconstruction loss function in Eq. (2) with a gradient descent method
(i.e., ADAM [42]).

####  III-D2 Fine-tuning Stage

After pre-training, we combine Eqs. (2), (3), (5), and (7) to form the overall
loss function:

$\displaystyle\min_{\mathbf{C}_{\emph{F}}}$
$\displaystyle\quad\frac{1}{2}\left\|\mathbf{X}-\hat{\mathbf{X}}\right\|^{2}_{F}+\left\|\mathbf{C}_{\emph{A}}\right\|^{2}_{F}+\gamma\left\|\mathbf{Z}_{\emph{A}}-\mathbf{Z}_{\emph{A}}\mathbf{C}_{\emph{A}}\right\|^{2}_{F}$
(8)
$\displaystyle\quad+\lambda_{1}\left\|\mathbf{C}_{\emph{S}}\right\|^{2}_{F}+\lambda_{2}\left\|\mathbf{Z}_{\emph{S}}-\mathbf{Z}_{\emph{S}}\mathbf{C}_{\emph{S}}\right\|^{2}_{F}$
$\displaystyle\rm{s.t.}$
$\displaystyle\quad\mathbf{Z}=\emph{f}_{\emph{AE}}\left(\mathbf{X}\right),\mathbf{Z}_{\emph{A}}=\mathbf{Z},\mathbf{Z}_{\emph{S}}=\frac{\mathbf{C}_{\emph{A}}+\mathbf{C}_{\emph{A}}^{\mathsf{T}}}{2},$
$\displaystyle\quad\mathbf{C}_{\emph{F}}=\left(\mathbf{m}_{1}\mathbf{1}\right)\odot\mathbf{C}_{\emph{A}}+\left(\mathbf{m}_{2}\mathbf{1}\right)\odot\mathbf{C}_{\emph{S}},$

and fine-tune the complete network by optimizing Eq. (8).

##  IV Experiments

We quantitatively and qualitatively compared the proposed method with eighteen
state-of-the-art methods on five commonly used benchmark datasets. In
addition, we performed extensive ablation studies of the designed modules and
examined the influence of the hyperparameters.

###  IV-A Datasets

We conducted the experiments on
UMIST111http://eprints.lincoln.ac.uk/id/eprint/16081/,
ORL222http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html,
MNIST333http://yann.lecun.com/exdb/mnist,
COIL20444http://www.cs.columbia.edu/CAVE/software/softlib/coil-20.php, and
COIL40555https://github.com/sckangz/L2SP, and the detailed descriptions of
these datasets are as follows.

  * •

UMIST [43]. The University of Manchester Institute of science and technology
(UMIST) database contains 564 images of 20 individuals, in which each
individual presents a range of poses from profile to frontal views. In this
paper, we used 480 images of 20 people and down-sampled each image to
$32\times 32$.

  * •

ORL [44]. The Olivetti research ltd database contains 40 individuals with 400
images, in which each image is captured at different scenes (e.g., the
lighting condition).

  * •

MNIST [45]. The modified national institute of standards and technology
database contains ten digits with 60000 training examples and 10000 test
examples. In this paper, we used the first 100 images of each digit to conduct
experiments.

  * •

COIL20, COIL40 [24]. The Columbia object image library dataset contains COIL20
and COIL40, in which COIL20 has 1440 images of 20 objects, and COIL40 has 2880
images of 40 objects. Here, each object has 72 views, and all images are down-
sampled to $32\times 32$.

TABLE II: The details of the network architecture on five datasets. Dataset | Layers | Encoder | Decoder
---|---|---|---
UMIST | Kernel size | (5$\times$5, 3$\times$3, 3$\times$3) | (3$\times$3, 3$\times$3, 5$\times$5)
Channel | (15, 10, 5) | (5, 10, 15)
ORL | Kernel size | (3$\times$3, 3$\times$3, 3$\times$3) | (3$\times$3, 3$\times$3, 3$\times$3)
Channel | (3, 3, 5) | (5, 3, 3)
MNIST | Kernel size | (5$\times$5, 3$\times$3, 3$\times$3) | (5$\times$5, 3$\times$3, 3$\times$3)
Channel | (10, 20, 30) | (30, 20, 10)
COIL20 | Kernel size | 3$\times$3 | 3$\times$3
Channel | 15 | 15
COIL40 | Kernel size | 3$\times$3 | 3$\times$3
Channel | 20 | 20
TABLE III: Clustering results on benchmark datasets. The best and second-best results are bolded and underlined, respectively. Datasets | UMIST | ORL | MNIST | COIL20 | COIL40
---|---|---|---|---|---
Methods\Metrics | ACC | NMI | PUR | ACC | NMI | PUR | ACC | NMI | PUR | ACC | NMI | PUR | ACC | NMI | PUR
SSC [22] | 0.6904 | 0.7489 | 0.6554 | 0.7425 | 0.8459 | 0.7875 | 0.4530 | 0.4709 | 0.4940 | 0.8631 | 0.8892 | 0.8747 | 0.7191 | 0.8212 | 0.7716
EDSC [15] | 0.6937 | 0.7522 | 0.6683 | 0.7038 | 0.7799 | 0.7138 | 0.5650 | 0.5752 | 0.6120 | 0.8371 | 0.8828 | 0.8585 | 0.6870 | 0.8139 | 0.7469
KSSC [18] | 0.6531 | 0.7377 | 0.6256 | 0.7143 | 0.8070 | 0.7513 | 0.5220 | 0.5623 | 0.5810 | 0.7087 | 0.8243 | 0.7497 | 0.6549 | 0.7888 | 0.7284
SSC-OMP [35] | 0.6438 | 0.7068 | 0.6171 | 0.7100 | 0.7952 | 0.7463 | 0.3400 | 0.3272 | 0.3560 | 0.6410 | 0.7412 | 0.6667 | 0.4431 | 0.6545 | 0.5250
ENSC [17] | 0.6931 | 0.7569 | 0.6628 | 0.7525 | 0.8540 | 0.7950 | 0.4983 | 0.5495 | 0.5483 | 0.8760 | 0.8952 | 0.8892 | 0.7426 | 0.8380 | 0.7924
LRR [16] | 0.6979 | 0.7630 | 0.6670 | 0.8100 | 0.8603 | 0.8225 | 0.5386 | 0.5632 | 0.5684 | 0.8118 | 0.8747 | 0.8361 | 0.6493 | 0.7828 | 0.7109
LRSC [46] | 0.6729 | 0.7498 | 0.6562 | 0.7200 | 0.8156 | 0.7542 | 0.5140 | 0.5576 | 0.5550 | 0.7416 | 0.8452 | 0.7937 | 0.6327 | 0.7737 | 0.6981
FLSR [47] | 0.6000 | 0.7082 | 0.6188 | 0.7775 | 0.8661 | 0.7900 | 0.6210 | 0.5231 | 0.6210 | 0.7104 | 0.7847 | 0.7507 | 0.5920 | 0.7554 | 0.6227
FTRR [47] | 0.7667 | 0.8509 | 0.7917 | 0.8600 | 0.9151 | 0.8725 | 0.7070 | 0.6672 | 0.7070 | 0.9035 | 0.9305 | 0.9104 | 0.7858 | 0.8801 | 0.8156
AE [21] \+ SSC | 0.7042 | 0.7515 | 0.6785 | 0.7563 | 0.8555 | 0.7950 | 0.4840 | 0.5337 | 0.5290 | 0.8711 | 0.8990 | 0.8901 | 0.7391 | 0.8318 | 0.7840
DEC [48] | 0.5521 | 0.7125 | 0.5917 | 0.5175 | 0.7449 | 0.5400 | 0.6120 | 0.5743 | 0.6320 | 0.7215 | 0.8007 | 0.6931 | 0.4872 | 0.7417 | 0.4163
DKM [49] | 0.5106 | 0.7249 | 0.5685 | 0.4682 | 0.7332 | 0.4752 | 0.5332 | 0.5002 | 0.5647 | 0.6651 | 0.7971 | 0.6964 | 0.5812 | 0.7840 | 0.6367
DCCM [50] | 0.5458 | 0.7440 | 0.5854 | 0.6250 | 0.7906 | 0.5975 | 0.4020 | 0.3468 | 0.4370 | 0.8021 | 0.8639 | 0.7889 | 0.7691 | 0.8890 | 0.7663
DEPICT [51] | 0.4521 | 0.6329 | 0.4167 | 0.2800 | 0.5764 | 0.1450 | 0.4240 | 0.4236 | 0.3560 | 0.8618 | 0.9266 | 0.8319 | 0.8073 | 0.9291 | 0.8191
DSC-Net-L1 [23] | 0.7242 | 0.7556 | 0.7204 | 0.8550 | 0.9023 | 0.8585 | 0.7280 | 0.7217 | 0.7890 | 0.9314 | 0.9353 | 0.9306 | 0.8003 | 0.8852 | 0.8646
DSC-Net-L2 [23] | 0.7312 | 0.7662 | 0.7276 | 0.8600 | 0.9034 | 0.8625 | 0.7500 | 0.7319 | 0.7991 | 0.9368 | 0.9408 | 0.9397 | 0.8075 | 0.8941 | 0.8740
DASC [24] | 0.7688 | 0.8042 | 0.7688 | 0.8825 | 0.9315 | 0.8925 | 0.8040 | 0.7800 | 0.8370 | 0.9639 | 0.9686 | 0.9632 | 0.8354 | 0.9196 | 0.8972
PSSC [25] | 0.7917 | 0.8670 | 0.8146 | 0.8675 | 0.9349 | 0.8925 | 0.8430 | 0.7676 | 0.8430 | 0.9722 | 0.9779 | 0.9722 | 0.8358 | 0.9258 | 0.8642
Our | 0.8313 | 0.8789 | 0.8313 | 0.9075 | 0.9451 | 0.9125 | 0.8490 | 0.7594 | 0.8490 | 0.9840 | 0.9831 | 0.9840 | 0.8823 | 0.9398 | 0.8920

###  IV-B Compared Methods

We compared the proposed method with eighteen state-of-the-art methods, and
the detailed descriptions of these methods are as follows.

  * •

SSC [22, 14] imposes a sparse regularization to seek a subspace-preserving
data affinity.

  * •

EDSC [15] yields dense connections within each cluster.

  * •

KSSC [18] extends SSC to nonlinear manifold by employing a kernel function.

  * •

SSC-OMP [35] introduces the orthogonal matching pursuit to SSC.

  * •

ENSC [17] balances the subspace-preserving (sparse solutions) and the
connectivity (dense solutions) by using an elastic net.

  * •

LRR [16] seeks the lowest rank representation by exploiting a low rank
regularization.

  * •

LRSC [46] introduces the low rank regularization to the subspace clustering to
solve the corrupted data (e.g., noise or errors).

  * •

FLSR, FTRR [47] inject graph similarity into data features based on least
square regression [52] and thresholding ridge regression [53], respectively.

  * •

AE+SSC conducts SSC on a pre-trained auto-encoder (AE) [21] feature.

  * •

DEC [48] partitions data points in a jointly optimized feature space.

  * •

DKM [49] achieves clustering by jointly learning a deep representation and
performing k-means [54].

  * •

DCCM [50] learns feature representation by mining comprehensive structures for
deep unsupervised clustering.

  * •

DEPICT [51] conducts a nonlinear projection and predicts cluster assignments
with a regularized relative entropy loss function.

  * •

DSC-Net-L1, DSC-Net-L2 [23] are two deep subspace clustering networks with
sparse [22] and Tikhonov [52] regularization, respectively.

  * •

DASC [24] supervises sample representation learning and subspace clustering in
an adversarial learning manner.

  * •

PSSC [25] conducts the subspace clustering by integrating locality preserving,
self-expression, and self-supervision.

###  IV-C Evaluation Metrics

We used three commonly used metrics (i.e., accuracy (ACC), normalized mutual
information (NMI), and purity (PUR)) to quantitatively evaluate clustering
performance. The larger the values of these metrics, the better the clustering
performance. Specifically, ACC is calculated by

$\displaystyle\text{ACC}=\frac{1}{n}\sum_{i=1}^{n}\mathbf{1}\left\\{y_{i}=m(c_{i})\right\\},$
(9)

where $y_{i}$ denotes the ground truth, $c_{i}$ denotes the predicted
assignment, $m(\cdot)$ enumerates the mapping between $y_{i}$ and $c_{i}$ with
the Kuhn-Munkres algorithm [55], and $\mathbf{1}\left\\{\cdot\right\\}$
denotes the indicator function with $1$ or $0$. NMI is calculated by

$\displaystyle\text{NMI}=\frac{\sum_{y\in
Y,\hat{y}\in\hat{Y}}p(y,\hat{y})\log\left(\frac{p(y,\hat{y})}{p(y)p(\hat{y})}\right)}{\max\left(H(Y),H(\hat{Y})\right)},$
(10)

where $y$ and $\hat{y}$ denote two clusters labels, $Y$ and $\hat{Y}$ are
their clusters sets, $p(\cdot)$ is a marginal probability mass function,
$p(y,\hat{y})$ denotes a joint probability mass function of $Y$ and $\hat{Y}$,
and $H(\cdot)$ represents an entropy function [56]. PUR is calculated by

$\displaystyle\text{PUR}=\frac{1}{n}\sum_{i=1}^{k}\max_{j}|c_{i}\cap t_{j}|,$
(11)

where $t_{j}$ denotes the segmentation which has the max count for $c_{i}$,
$\max_{j}|c_{i}\cap t_{j}|$ counts the number of data points from the most
common class in $c_{i}$, $k$ denotes the number of clusters.

###  IV-D Implementation Details

Table II lists the detailed network settings of our method on each dataset. In
addition, we set the kernel stride of both horizontal and vertical directions
to be $2$ and used a rectified linear unit (ReLU) [57] activation function. We
implemented the proposed method with TensorFlow 2.1.0 and initialized the
neural network parameters by the ‘he_normal’ initializer [58]. Besides, we set
$\gamma$ as the same as the optimal parameter of [23]. After training, we
performed spectral clustering on the fused affinity graph. For a fair
comparison, we used the normalized cut algorithm [59] as in [23, 24, 47, 25].

###  IV-E Clustering Results

Table III lists the quantitative results of all methods over the benchmark
datasets, where we have the following conclusions.

  * •

The deep SE-based methods achieve better clustering performance than the
traditional SE-based methods, indicating that the nonlinear projection is
helpful to extract a more clustering-friendly representation to deal with
complex real-world datasets. For example, on MNIST, our method improves the
ACC, NMI, and PUR values of the best traditional SE-based approach FTRR by
14.20%, 9.22%, and 14.20%, respectively.

  * •

Structure information contributes to achieving better clustering performance.
Specifically, FTRR, which exploits structure information to enhance the
feature representation, produces the best performance among those traditional
SE-based methods, e.g., on UMIST, FTRR obtains 6.88% ACC improvement over the
second-best traditional SE-based method LRR. In addition, our method performs
best among all deep SE-based methods, e.g., on COIL40, our AASSC-Net improves
the ACC value of the second-best method PSSC by 4.65%.

  * •

Simultaneously considering the attribute and structure information is
beneficial to clustering, which is validated by comparing DSC-Net-L2 and our
method. See Section IV-F1 for the detailed ablation study.

  * •

Our AASSC-Net almost always obtains the best clustering performance among all
compared methods. For example, on COIL40, our approach achieves 4.65%, 1.40%,
and 2.78% improvements in terms of ACC, NMI, and PUR, respectively, compared
with the second-best method PSSC. Such impressive performance is credited to
the simultaneous consideration of the attribute and structure information via
the attention-based fusion.

(a) $\mathbf{C}_{\emph{A}}$ (b) $\mathbf{C}_{\emph{S}}$ (c)
$\mathbf{C}_{\emph{F}}$

Figure 3: Visual comparison of the affinity matrices $\mathbf{C}_{\emph{A}}$
and $\mathbf{C}_{\emph{S}}$ respectively learned from attribute and structure
information on the synthetic dataset, as well the fused one
$\mathbf{C}_{\emph{F}}$.

(a) $\mathbf{C}_{\emph{A}}$ (ACC: 0.88) (b) $\mathbf{C}_{\emph{S}}$ (ACC:
0.92) (c) $\mathbf{C}_{\emph{F}}$ (ACC: 0.96)

Figure 4: Quantitative comparison of $\mathbf{C}_{\emph{A}}$,
$\mathbf{C}_{\emph{S}}$ and $\mathbf{C}_{\emph{F}}$ on the synthetic dataset,
where the below column summaries the precision, the right row summaries the
recall.

(a) Ground truth (1.0000) (b) DSC-Net-L2 (0.9222) (c) PSSC (0.9514) (d) Our
(0.9819)

Figure 5: Visualization of the learned representations by (a) ground truth,
(b) DSC-Net-L2 [23], (c) PSSC [25], and (d) Our method, where the digit in the
bracket refers to the ACC value. Different colors represent different
clusters.

###  IV-F Ablation Study

####  IV-F1 Adaptive Graph Fusion

To deeply understand the proposed method, we conducted comprehensive ablation
studies on five benchmark datasets. The experimental results are listed in
Table IV, where the first row denotes the previous deep SE-based framework
that only involves the learning of $\mathbf{C}_{\emph{A}}$, the second row
denotes a variant of our model that only considers $\mathbf{C}_{\emph{S}}$
(i.e., $\mathbf{C}_{\emph{F}}=\mathbf{C}_{\emph{S}}$), the third row denotes
using the equal weight fusion strategy (i.e., $\mathbf{C}_{\emph{F}}=0.5\
\mathbf{C}_{\emph{A}}+0.5\ \mathbf{C}_{\emph{S}}$), and the fourth row is our
full model. From Table IV, we have the following observations.

TABLE IV: Ablation studies of the adaptive graph fusion strategy, where the ‘$\checkmark$’ place indicates the usage of the corresponding ablation component. The best results are highlighted with the bold. | $\emph{f}_{\emph{AF}}$ | $\mathbf{C}_{\emph{S}}$ | $\mathbf{C}_{\emph{A}}$ | ACC | NMI | PUR
---|---|---|---|---|---|---
UMIST |  |  | $\checkmark$ | 0.8104 | 0.8730 | 0.8333
| $\checkmark$ |  | 0.5708 | 0.7297 | 0.6000
| $\checkmark$ | $\checkmark$ | 0.8104 | 0.8756 | 0.8333
$\checkmark$ | $\checkmark$ | $\checkmark$ | 0.8313 | 0.8789 | 0.8313
ORL |  |  | $\checkmark$ | 0.8650 | 0.9204 | 0.8700
| $\checkmark$ |  | 0.8525 | 0.9217 | 0.8650
| $\checkmark$ | $\checkmark$ | 0.9000 | 0.9343 | 0.9000
$\checkmark$ | $\checkmark$ | $\checkmark$ | 0.9075 | 0.9451 | 0.9125
MNIST |  |  | $\checkmark$ | 0.7890 | 0.7371 | 0.7890
| $\checkmark$ |  | 0.7370 | 0.7211 | 0.7840
| $\checkmark$ | $\checkmark$ | 0.7900 | 0.7335 | 0.7900
$\checkmark$ | $\checkmark$ | $\checkmark$ | 0.8490 | 0.7594 | 0.8490
COIL20 |  |  | $\checkmark$ | 0.9569 | 0.9584 | 0.9569
| $\checkmark$ |  | 0.9535 | 0.9589 | 0.9535
| $\checkmark$ | $\checkmark$ | 0.9806 | 0.9803 | 0.9806
$\checkmark$ | $\checkmark$ | $\checkmark$ | 0.9840 | 0.9831 | 0.9840
COIL40 |  |  | $\checkmark$ | 0.8250 | 0.9223 | 0.8465
| $\checkmark$ |  | 0.8774 | 0.9382 | 0.8868
| $\checkmark$ | $\checkmark$ | 0.8608 | 0.9306 | 0.8767
$\checkmark$ | $\checkmark$ | $\checkmark$ | 0.8823 | 0.9398 | 0.8920
TABLE V: Investigation of the effect of the manner of constructing the structure matrix $\mathbf{Z}_{\emph{S}}$ on the ORL dataset. Case | $\mathbf{Z}_{\emph{S}}$ | ACC | NMI | PUR
---|---|---|---|---
1 | $\frac{\mathbf{Z}^{\mathsf{T}}\mathbf{Z}}{\left\|\mathbf{Z}^{\mathsf{T}}\right\|_{F}\left\|\mathbf{Z}\right\|_{F}}$ | 0.8675 | 0.9177 | 0.8750
2 | $\mathbf{C}_{\emph{A}}$ | 0.8925 | 0.9357 | 0.9000
3 | $\frac{|\mathbf{C}_{\emph{A}}|+|\mathbf{C}_{\emph{A}}^{\mathsf{T}}|}{2}$ | 0.7450 | 0.8818 | 0.7975
4 | $\frac{\mathbf{C}_{\emph{A}}+\mathbf{C}_{\emph{A}}^{\mathsf{T}}}{2}$ | 0.9075 | 0.9451 | 0.9125

(a) UMIST (b) ORL (c) MNIST (d) COIL20

(e) COIL40

Figure 6: Clustering of the proposed model versus parameters $\lambda_{1}$ and
$\lambda_{2}$ on five benchmark datasets with the ACC metric.

(a) UMIST (b) ORL (c) MNIST (d) COIL20

(e) COIL40

Figure 7: Clustering of the proposed model versus parameters $\lambda_{1}$ and
$\lambda_{2}$ on five benchmark datasets with the NMI metric.

(a) UMIST (b) ORL (c) MNIST (d) COIL20

(e) COIL40

Figure 8: Clustering of the proposed model versus parameters $\lambda_{1}$ and
$\lambda_{2}$ on five benchmark datasets with the PUR metric.

  * •

By comparing the results in the first and second rows of each dataset, we can
observe that $\mathbf{C}_{\emph{A}}$ and $\mathbf{C}_{\emph{S}}$ have their
own advantages for different datasets. For instance, on COIL40,
$\mathbf{C}_{\emph{S}}$ produces a 5.24% performance improvement over
$\mathbf{C}_{\emph{A}}$ on ACC, 1.59% on NMI, and 4.03% on PUR. However, on
UMIST, $\mathbf{C}_{\emph{S}}$ produces a 2.41% to 24.94% performance
degradation on all metrics compared with $\mathbf{C}_{\emph{A}}$.

  * •

The comparison of the fourth row with the first and second rows of each
dataset demonstrates that simultaneously considering the attribute and
structure information could promote clustering performance. For example, on
MNIST, our method obtains 6.00% performance improvement on ACC, 2.23% on NMI,
and 6.00% on PUR.

  * •

The advantage of the attention-based graph fusion strategy could be validated
by comparing the results of the third and fourth rows of each dataset, where
it can be seen that the adaptive graph learning strategy produces a 1% to 9%
performance improvement.

We also visually compared the learned affinity graphs on a synthetic dataset
constructed by randomly selecting ten categories from the ORL dataset. As
shown in Figure 3, it can be seen that the three affinity graphs have an
approximate block-diagonal structure; however, $\mathbf{C}_{\emph{S}}$ and
$\mathbf{C}_{\emph{F}}$ contain fewer error connections than
$\mathbf{C}_{\emph{A}}$. In addition, Figure 4 shows the corresponding
confusion matrices, in which we can observe that $\mathbf{C}_{\emph{F}}$
obtains higher clustering accuracy than $\mathbf{C}_{\emph{A}}$ and
$\mathbf{C}_{\emph{S}}$, benefiting from the adaptive utilization of both the
attribute and structure information.

####  IV-F2 Structure Matrix Construction

In addition to the method for constructing $\mathbf{Z}_{\emph{S}}$ in Section
III-B2, we also investigated another three construction manners:

  * •

We constructed $\mathbf{Z}_{\emph{S}}$ by using the cosine similarity, i.e.,

$\displaystyle\mathbf{Z}_{\emph{S}}=\frac{\mathbf{Z}^{\mathsf{T}}\mathbf{Z}}{\left\|\mathbf{Z}^{\mathsf{T}}\right\|_{F}\left\|\mathbf{Z}\right\|_{F}}.$
(12)

  * •

We directly utilized $\mathbf{C}_{\emph{A}}$ as the structure matrix:

$\displaystyle\mathbf{Z}_{\emph{S}}=\mathbf{C}_{\emph{A}}.$ (13)

  * •

We further enforced $\mathbf{Z}_{\emph{S}}$ to satisfy the symmetric and non-
negative properties, i.e.,

$\displaystyle\mathbf{Z}_{\emph{S}}=\frac{|\mathbf{C}_{\emph{A}}|+|\mathbf{C}_{\emph{A}}^{\mathsf{T}}|}{2}.$
(14)

Table V lists the corresponding results of different construction methods,
where we have the following observations.

  * •

Case $4$ outperforms the others in terms of all the metrics considerably. For
example, on ORL, it improves the ACC, NMI, and PUR of the second-best way
(i.e., Case $2$) by 1.50%, 0.94%, and 1.25%, respectively. In particular, the
comparison with Case $3$ validates the effectiveness of preserving the
negative values of $\mathbf{C}_{\emph{A}}$ as the negative values can also
indicate the discriminative information among samples.

  * •

Case $1$ performs poorly compared with other cases, e.g., on ORL, Case $4$
improves 4.00% over Case $1$ on ACC, 2.74% on NMI, and 3.75% on PUR. This
phenomenon suggests that the cosine similarity is not a good choice for
constructing the structure matrix in our framework. The reason may be that the
cosine similarity only considers the angle between two vectors while ignoring
the magnitude, which is important to indicate the similarity of two samples.

###  IV-G Visual Comparison

We used the 2D t-distributed stochastic neighbor embedding (t-SNE) technique
[60] to visualize the encoder output representations of our method and two
representative deep clustering methods on one generated dataset constructed by
randomly selecting ten categories from COIL20. From Figure 5, we can observe
that the produced embedding by our method has the best accuracy and
separability for different clusters, indicating that our AASSC-Net could learn
a more discriminative representation than the compared methods.

###  IV-H Parameter Analysis

In the loss function Eq. (8), there are three hyperparameters (i.e., $\gamma$,
$\lambda_{1}$, and $\lambda_{2}$) used to balance the contributions of the
attribute SE, the structure affinity graph, and the structure SE,
respectively. We first set the same $\gamma$ as the optimal parameter of [23].
Then, we selected $\lambda_{1}$ and $\lambda_{2}$ from the set
$\left\\{0.001,0.01,0.1,1,10,100,1000\right\\}$ to evaluate their influence.
Figures 6, 7 and 8 show the values of ACC, NMI, and PUR w.r.t. different
$\lambda_{1}$ and $\lambda_{2}$ on the five benchmark datasets, where we have
the following observations.

  * •

Our method is capable of achieving almost optimal performance in terms of the
three metrics over the majority datasets in a wide range of the two
parameters, illustrating that our method is robust to $\lambda_{1}$ and
$\lambda_{2}$.

  * •

On MNIST, the relatively small $\lambda_{1}$ and large $\lambda_{2}$ lead to
low values of ACC, NMI, and PUR. The reason may be that this dataset includes
digit images with similar shapes (e.g., ‘$7$’ and ‘$9$’), and thus when
conducting the SE, the regularization term of learning the structure affinity
matrix is vital for keeping the connectivity within the same subspace.

##  V Conclusion

We presented a novel deep self-expressiveness-based subspace clustering
framework. Specifically, we first constructed the attribute and structure
matrices separately after nonlinearly projecting the raw data into the latent
feature space. Then, we employed the SE priori to learn affinity graphs from
the constructed attribute and structure matrices separately. Afterward, we
exploited an attention-based strategy to fuse the two learned affinity graphs.
We quantitatively and qualitatively validated the advantage of the proposed
method over eighteen state-of-the-art methods on five commonly-used benchmark
datasets. In addition, we performed comprehensive ablation studies to validate
the effectiveness of the designed modules.

## References

  * [1] Z. Zhang, F. Li, M. Zhao, L. Zhang, and S. Yan, “Robust neighborhood preserving projection by nuclear/l2, 1-norm regularization for image feature extraction,” _IEEE Transactions on Image Processing_ , vol. 26, no. 4, pp. 1607–1622, 2017.
  * [2] C. Zhang, Y. Liu, and H. Fu, “Ae2-nets: Autoencoder in autoencoder networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2577–2585.
  * [3] S. Park, S. Han, S. Kim, D. Kim, S. Park, S. Hong, and M. Cha, “Improving unsupervised image clustering with robust learning,” in _CVPR_ , June 2021, pp. 12 278–12 287.
  * [4] Y. Jia, H. Liu, J. Hou, and Q. Zhang, “Clustering ensemble meets low-rank tensor approximation,” in _AAAI_ , vol. 35, no. 9, 2021, pp. 7970–7978.
  * [5] Y. Li, P. Hu, Z. Liu, D. Peng, J. T. Zhou, and X. Peng, “Contrastive clustering,” in _AAAI_ , 2021.
  * [6] H. Li, Y. Jia, R. Cong, W. Wu, S. Kwong, and C. Chen, “Superpixel segmentation based on spatially constrained subspace clustering,” _IEEE Transactions on Industrial Informatics_ , 2020.
  * [7] Y. Jia, J. Hou, and S. Kwong, “Constrained clustering with dissimilarity propagation-guided graph-laplacian pca,” _IEEE Transactions on Neural Networks and Learning Systems_ , pp. 1–13, 2020.
  * [8] J. H. Cho, U. Mall, K. Bala, and B. Hariharan, “Picie: Unsupervised semantic segmentation using invariance and equivariance in clustering,” in _CVPR_ , June 2021, pp. 16 794–16 804.
  * [9] C. Zhang, Q. Hu, H. Fu, P. Zhu, and X. Cao, “Latent multi-view subspace clustering,” in _CVPR_ , 2017, pp. 4279–4287.
  * [10] C. Zhang, H. Fu, Q. Hu, X. Cao, Y. Xie, D. Tao, and D. Xu, “Generalized latent multi-view subspace clustering,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 42, no. 1, pp. 86–99, 2018.
  * [11] Z. Peng, Y. Jia, H. Liu, J. Hou, and Q. Zhang, “Maximum entropy subspace clustering network,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2021.
  * [12] Y. Jia, H. Liu, J. Hou, S. Kwong, and Q. Zhang, “Multi-view spectral clustering tailored tensor low-rank representation,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2021.
  * [13] A. Y. Ng, M. I. Jordan, and Y. Weiss, “On spectral clustering: Analysis and an algorithm,” in _NIPS_ , 2002, pp. 849–856.
  * [14] E. Elhamifar and R. Vidal, “Sparse subspace clustering: Algorithm, theory, and applications,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 35, no. 11, pp. 2765–2781, 2013.
  * [15] P. Ji, M. Salzmann, and H. Li, “Efficient dense subspace clustering,” in _WACV_ , 2014, pp. 461–468.
  * [16] G. Liu, Z. Lin, S. Yan, J. Sun, Y. Yu, and Y. Ma, “Robust recovery of subspace structures by low-rank representation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 35, no. 1, pp. 171–184, 2012.
  * [17] C. You, C.-G. Li, D. P. Robinson, and R. Vidal, “Oracle based active set algorithm for scalable elastic net subspace clustering,” in _CVPR_ , 2016, pp. 3928–3937.
  * [18] V. M. Patel and R. Vidal, “Kernel sparse subspace clustering,” in _ICIP_. IEEE, 2014, pp. 2849–2853.
  * [19] M. Yin, Y. Guo, J. Gao, Z. He, and S. Xie, “Kernel sparse subspace clustering on symmetric positive definite manifolds,” in _CVPR_ , 2016, pp. 5157–5164.
  * [20] G. Xia, B. Chen, H. Sun, and Q. Liu, “Nonconvex low-rank kernel sparse subspace learning for keyframe extraction and motion segmentation,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 32, no. 4, pp. 1612–1626, 2020.
  * [21] G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” _Science_ , vol. 313, no. 5786, pp. 504–507, 2006\.
  * [22] E. Elhamifar and R. Vidal, “Sparse subspace clustering,” in _CVPR_ , 2009, pp. 2790–2797.
  * [23] P. Ji, T. Zhang, H. Li, M. Salzmann, and I. Reid, “Deep subspace clustering networks,” in _NIPS_ , 2017, pp. 24–33.
  * [24] P. Zhou, Y. Hou, and J. Feng, “Deep adversarial subspace clustering,” in _CVPR_ , 2018, pp. 1596–1604.
  * [25] J. Lv, Z. Kang, X. Lu, and Z. Xu, “Pseudo-supervised deep subspace clustering,” _IEEE Transactions on Image Processing_ , 2021.
  * [26] R. Vidal, Y. Ma, and S. S. Sastry, _Generalized principal component analysis_. Springer, 2016, vol. 5.
  * [27] B. Li, H. Lu, Y. Zhang, Z. Lin, and W. Wu, “Subspace clustering under complex noise,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 29, no. 4, pp. 930–940, 2018.
  * [28] C. You, C.-G. Li, D. P. Robinson, and R. Vidal, “Is an affine constraint needed for affine subspace clustering?” in _ICCV_ , 2019, pp. 9915–9924.
  * [29] G. Liu, Z. Zhang, Q. Liu, and H. Xiong, “Robust subspace clustering with compressed data,” _IEEE Transactions on Image Processing_ , vol. 28, no. 10, pp. 5161–5170, 2019.
  * [30] X. Zhang, Z. Liu, X. Xue, Q. Wang, S. Zheng _et al._ , “Robust subspace clustering based on automatic weighted multiple kernel learning,” _Information Sciences_ , 2021.
  * [31] L. Wei, F. Ji, H. Liu, R. Zhou, C. Zhu, and X. Zhang, “Subspace clustering via structured sparse relation representation,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2021.
  * [32] H. Tao, C. Hou, Y. Qian, J. Zhu, and D. Yi, “Latent complete row space recovery for multi-view subspace clustering,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 8083–8096, 2020.
  * [33] Y. Chen, S. Wang, C. Peng, Z. Hua, and Y. Zhou, “Generalized nonconvex low-rank tensor approximation for multi-view subspace clustering,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 4022–4035, 2021.
  * [34] Y. Chen, X. Xiao, C. Peng, G. Lu, and Y. Zhou, “Low-rank tensor graph learning for multi-view subspace clustering,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2021.
  * [35] C. You, D. Robinson, and R. Vidal, “Scalable sparse subspace clustering by orthogonal matching pursuit,” in _CVPR_ , 2016, pp. 3918–3927.
  * [36] Z. Kang, C. Peng, and Q. Cheng, “Twin learning for similarity and clustering: a unified kernel approach,” in _AAAI_ , 2017, pp. 2080–2086.
  * [37] J. Butepage, M. J. Black, D. Kragic, and H. Kjellstrom, “Deep representation learning for human motion prediction and classification,” in _CVPR_ , 2017, pp. 6158–6166.
  * [38] H. Yao, S. Zhang, R. Hong, Y. Zhang, C. Xu, and Q. Tian, “Deep representation learning with part loss for person re-identification,” _IEEE Transactions on Image Processing_ , vol. 28, no. 6, pp. 2860–2871, 2019.
  * [39] G. Zhong and C.-M. Pun, “Nonnegative self-representation with a fixed rank constraint for subspace clustering,” _Information Sciences_ , vol. 518, pp. 127–141, 2020.
  * [40] S. Zhang, C. You, R. Vidal, and C.-G. Li, “Learning a self-expressive network for subspace clustering,” in _CVPR_ , 2021, pp. 12 393–12 403.
  * [41] A. L. Maas, A. Y. Hannun, and A. Y. Ng, “Rectifier nonlinearities improve neural network acoustic models,” in _ICML_ , vol. 30. Atlanta, USA: Citeseer, 2013, p. 3.
  * [42] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [43] D. B. Graham and N. M. Allinson, “Characterising virtual eigensignatures for general purpose face recognition,” in _Face Recognition_. Springer, 1998, pp. 446–456.
  * [44] F. S. Samaria and A. C. Harter, “Parameterisation of a stochastic model for human face identification,” in _WACV_. IEEE, 1994, pp. 138–142.
  * [45] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proceedings of The IEEE_ , vol. 86, no. 11, pp. 2278–2324, 1998.
  * [46] R. Vidal and P. Favaro, “Low rank subspace clustering (lrsc),” _Pattern Recognition Letters_ , vol. 43, pp. 47–61, 2014.
  * [47] Z. Ma, Z. Kang, G. Luo, L. Tian, and W. Chen, “Towards clustering-friendly representations: Subspace clustering via graph filtering,” in _ACM MM_ , 2020, pp. 3081–3089.
  * [48] J. Xie, R. Girshick, and A. Farhadi, “Unsupervised deep embedding for clustering analysis,” in _ICML_. PMLR, 2016, pp. 478–487.
  * [49] M. M. Fard, T. Thonet, and E. Gaussier, “Deep k-means: Jointly clustering with k-means and learning representations,” _Pattern Recognition Letters_ , vol. 138, pp. 185–192, 2020.
  * [50] J. Wu, K. Long, F. Wang, C. Qian, C. Li, Z. Lin, and H. Zha, “Deep comprehensive correlation mining for image clustering,” in _ICCV_ , 2019, pp. 8150–8159.
  * [51] K. Ghasedi Dizaji, A. Herandi, C. Deng, W. Cai, and H. Huang, “Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization,” in _ICCV_ , 2017, pp. 5736–5745.
  * [52] C.-Y. Lu, H. Min, Z.-Q. Zhao, L. Zhu, D.-S. Huang, and S. Yan, “Robust and efficient subspace segmentation via least squares regression,” in _ECCV_. Springer, 2012, pp. 347–360.
  * [53] X. Peng, Z. Yi, and H. Tang, “Robust subspace clustering via thresholding ridge regression,” in _AAAI_ , vol. 29, no. 1, 2015.
  * [54] J. A. Hartigan and M. A. Wong, “Algorithm as 136: A k-means clustering algorithm,” _Journal of The Royal Statistical Society. Series C (Applied Statistics)_ , vol. 28, no. 1, pp. 100–108, 1979.
  * [55] J. Munkres, “Algorithms for the assignment and transportation problems,” _Journal of The Society for Industrial and Applied Mathematics_ , vol. 5, no. 1, pp. 32–38, 1957.
  * [56] C. E. Shannon, “A mathematical theory of communication,” _The Bell System Technical Journal_ , vol. 27, no. 3, pp. 379–423, 1948.
  * [57] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _NIPS_ , 2012, pp. 1097–1105.
  * [58] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in _ICCV_ , 2015, pp. 1026–1034.
  * [59] J. Shi and J. Malik, “Normalized cuts and image segmentation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 22, no. 8, pp. 888–905, 2000.
  * [60] L. v. d. Maaten and G. Hinton, “Visualizing data using t-sne,” _Journal of Machine Learning Research_ , vol. 9, no. Nov, pp. 2579–2605, 2008.