# Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers

Kamil Bujel
Department of Computing
Imperial College London
United Kingdom
<EMAIL_ADDRESS>
&Helen Yannakoudakis
Department of Informatics
King’s College London
United Kingdom
<EMAIL_ADDRESS>
&Marek Rei
Department of Computing
Imperial College London
United Kingdom
<EMAIL_ADDRESS>

###### Abstract

We investigate how sentence-level transformers can be modified into effective
sequence labelers at the token level without any direct supervision. Existing
approaches to zero-shot sequence labeling do not perform well when applied on
transformer-based architectures. As transformers contain multiple layers of
multi-head self-attention, information in the sentence gets distributed
between many tokens, negatively affecting zero-shot token-level performance.
We find that a soft attention module which explicitly encourages sharpness of
attention weights can significantly outperform existing methods.

##  1 Introduction

Sequence labeling and sentence classification can represent facets of the same
task at different granularities; for example, detecting grammar errors and
predicting the grammaticality of sentences. Transformer-based architectures
such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have been
shown to achieve state-of-the-art results on both sequence labeling (Bell et
al., 2019) and sentence classification (Sun et al., 2019) problems. However,
such tasks are typically treated in isolation rather than within a unified
approach.

In this paper, we investigate methods for inferring token-level predictions
from transformer models trained only on sentence-level annotations. The
ability to classify individual tokens without direct supervision opens
possibilities for training sequence labeling models on tasks and datasets
where only sentence-level or document-level annotation is available. In
addition, attention-based architectures allow us to directly investigate what
the model is learning and to quantitatively measure whether its rationales
(supporting evidence) for particular input sentences match human expectations.
While evaluating the faithfulness (Herman, 2017) of a model’s rationale is
still an open research question and up for debate (Jain and Wallace, 2019;
Wiegreffe and Pinter, 2019; DeYoung et al., 2020; Jacovi and Goldberg, 2020;
Atanasova et al., 2020), the methods explored here allow for measuring the
plausibility (agreeability to human annotators; DeYoung et al. (2020)) of
transformer-based models using existing sequence labeling datasets.

We evaluate and compare different methods for adapting pre-trained transformer
models into zero-shot sequence labelers, trained using only gold sentence-
level signal. Our experiments show that applying existing approaches Rei and
Søgaard (2018) to transformer architectures is not straightforward –
transformers already contain several layers of multi-head attention,
distributing sentence-level information across many tokens, whereas the
existing methods rely on all the information going through one central
attention module. Approaches such as LIME (Ribeiro et al., 2016) for scoring
word importance also struggle to infer correct token-level annotations in a
zero-shot manner (e.g., it achieves only $2\%$ F-score on one of our
datasets). We find that a modified attention function is needed to allow
transformers to better focus on individual important tokens and achieve a new
state-of-the-art on zero-shot sequence labeling.

The contributions of this paper are fourfold:

  * •

We present the first experiments utilizing (pre-trained) sentence-level
transformers as zero-shot sequence labelers;

  * •

We perform a systematic comparison of alternative methods for zero-shot
sequence labeling on different datasets;

  * •

We propose a novel modification of the attention function that significantly
improves zero-shot sequence-labeling performance of transformers over the
previous state of the art, while achieving on-par or better results on
sentence classification;

  * •

We make our source code and models publicly available to facilitate further
research in the field.111https://github.com/bujol12/bert-seq-interpretability

##  2 Methods

We evaluate four different methods for turning sentence-level transformer
models into zero-shot sequence labelers.

###  2.1 LIME

LIME (Ribeiro et al., 2016) generates local word-level importance scores
through a meta-model that is trained on perturbed data generated by randomly
masking out words in the input sentence. It was originally investigated in the
context of Support Vector Machine (Hearst et al., 1998) text classifiers with
unigram features.

We apply LIME to a RoBERTa model supervised as a sentence classifier and
investigate whether its scores can be used for sequence labeling. We use
RoBERTa’s MASK token to mask out individual words and allow LIME to generate
$5000$ masked samples per sentence. The resulting explanation weights are then
used as classification scores for each word, with the decision threshold fine-
tuned based on the development set performance.

Thorne et al. (2019) found LIME to outperform attention-based approaches on
the task of explaining NLI models. LIME was used to probe a LSTM-based
sentence-pair classifier (Lan and Xu, 2018) by removing tokens from the
premise and hypothesis sentences separately. The generated scores were used to
perform binary classification of tokens, with the threshold based on $F_{1}$
performance on the development set. The token-level predictions were evaluated
against human explanations of the entailment relation using the e-SNLI dataset
(Camburu et al., 2018). LIME was found to outperform other methods, however,
it was also $1000\times$ slower than attention-based methods at generating
these explanations.

###  2.2 Attention heads

The attention heads in a trained transformer model are designed to identify
and combine useful information for a particular task. Clark et al. (2019)
found that specific heads can specialize on different linguistic properties
such as syntax and coreference. However, transformer models contain many
layers with multiple attention heads, distributing the text representation and
making it more difficult to identify token importance for the overall task.

Given a particular head, we can obtain an importance score for each token by
averaging the attention scores from all the tokens that attend to it. In order
to investigate the best possible setting, we report results for the attention
head that achieves the highest token-level Mean Average Precision score on the
development set.

###  2.3 Soft attention

Rei and Søgaard (2018) described a method for predicting token-level labels
based on a bidirectional LSTM (Hochreiter and Schmidhuber, 1997) architecture
supervised at the sentence-level only. A dedicated attention module was
integrated for building sentence representations, with its attention weights
also acting as token-level importance scores. The architecture was found to
outperform a gradient-based approach on the tasks of zero-shot sequence
labeling for error detection, uncertainty detection, and sentiment analysis.

In order to obtain a single raw attention value $\widetilde{e}_{i}$ for each
token, biLSTM output vectors were passed through a feedforward layer:

$e_{i}=tanh(W_{e}h_{i}+b_{e})\;\;\;\;\widetilde{e}_{i}=W_{\widetilde{e}}e_{i}+b_{\widetilde{e}}$
(1)

where $e_{i}$ is the attention vector for token $t_{i}$; $h_{i}$ is the biLSTM
output for $t_{i}$; and $\widetilde{e}_{i}$ is the single raw attention value.
$W_{e}$, $b_{e}$, $W_{\widetilde{e}}$, $b_{\widetilde{e}}$ are trainable
parameters.

Instead of softmax or sparsemax (Martins and Astudillo, 2016), which would
restrict the distribution of the scores, a soft attention based on sigmoid
activation was used to obtain importance scores:

$\widetilde{a}_{i}=\sigma(\widetilde{e}_{i})\;\;\;\;\;\;\;\;a_{i}=\frac{\widetilde{a}_{i}}{\sum_{k=1}^{N}\widetilde{a}_{k}}$
(2)

where $N$ is the number of tokens and $\sigma$ is the logistic function.
$\widetilde{a}_{i}$ shows the importance of a particular token and is in the
range $0\leq\widetilde{a}_{i}\leq 1$, independent of any other scores in the
sentence; therefore, it can be directly used for sequence labeling with a
natural threshold of $0.5$. $a_{i}$ contains the same information but is
normalized to sum up to $1$ over the whole sentence, making it suitable for
attention weights when building the sentence representation. As $a_{i}$ and
$\widetilde{a}_{i}$ are directly tied, training the former through the
sentence classification objective will also train the latter for the sequence
labeling task.

The attention values were then used to obtain the sentence representation $c$
by acting as weights for the biLSTM token outputs:

$c=\sum_{i=0}^{N}a_{i}h_{i}$ (3)

Finally, the sentence representation $c$ was passed through the final
feedforward layer, followed by a sigmoid to obtain the predicted score $y$ for
the sentence:

$d=tanh(W_{d}c+b_{d})\;\;\;\;\;\;y=\sigma(W_{y}d+b_{y})$ (4)

where $d$ is the sentence vector, $c$ is the sentence representation, and $y$
is the sentence prediction score. $W_{d}$, $b_{d}$, $W_{y}$, $b_{y}$ are all
trainable parameters.

We adapt this approach to the transformer models by attaching a separate soft
attention module on top of the token-level output representations. This
effectively ignores the CLS token, which is commonly used for sentence
classification, and instead builds a new sentence representation from the
token representations, which replace the previously used biLSTM outputs:

$e_{i}=tanh(W_{e}T_{i}+b_{e})\;\;\;\;c=\sum_{i=0}^{N}a_{i}T_{i}$ (5)

where $T_{i}$ is the contextualized embedding for token $t_{i}$. A diagram of
the model architecture is included in Appendix F.

Commonly used tokenizers for transformer models split words into subwords,
while sequence labeling datasets are annotated at the word level. We find that
taking the maximum attention value over all the subwords as the word-level
importance score produces good results on the development sets. For a word
$w_{i}$ split into tokens $[t_{j},...,t_{m}]$, where $j,m\in[1,N]$, the
resulting final word importance score $r_{i}$ is then given by:

$r_{i}=max(\\{\widetilde{a}_{j},\widetilde{a}_{j+1},...,\widetilde{a}_{m}\\})$
(6)

During training, we optimize sentence-level binary cross-entropy as the main
objective function:

$L_{1}=\frac{\sum_{j}CrossEntropy(y^{(j)},\widetilde{y}^{(j)})}{|y|}$ (7)

where $y^{(j)}$ and $\tilde{y}^{(j)}$ are the predicted sentence
classification logits and the gold label for the $j^{th}$ sentence
respectively. We also adopt the additional loss functions from Rei and Søgaard
(2018), which encourage the attention weights to behave more like token-level
classifiers:

$L_{2}=\frac{\sum_{j}(min_{j}(\widetilde{a_{i}})-0)^{2}}{|y|}$ (8)
$L_{3}=\frac{\sum_{j}(max_{j}(\widetilde{a_{i}})-\widetilde{y}^{(j)})^{2}}{|y|}$
(9)

Eq. 8 optimizes the minimum unnormalized attention to be $0$ and therefore
incentivizes the model to only focus on some, but not all words; Eq. 9 ensures
that some attention weights are close to $1$ if the overall sentence is
classified as positive. We then jointly optimize these three loss functions
using a hyperparameter $\gamma$: $L=L_{1}+\gamma(L_{2}+L_{3})$.

###  2.4 Weighted soft attention

Our experiments show that, when combined with transformer-based models, the
soft attention method tends to spread out the attention too widely. Instead of
focusing on specific important words, the model broadly attends to the whole
sentence. Figures 3 and 4 in Appendix A present examples demonstrating such
behaviour. As transformers contain several layers of attention, with multiple
heads in each layer, the information in the sentence gets distributed across
all tokens before it reaches the soft attention module at the top.

To improve this behaviour and incentivize the model to direct information
through a smaller and more focused set of tokens, we experiment with a
weighted soft attention:

$a_{i}=\frac{\tilde{a}_{i}^{\beta}}{\sum_{k=1}^{N}\tilde{a}_{k}^{\beta}}$ (10)

where $\beta$ is a hyperparamete and where values $\beta>1$ make the weight
distribution sharper, allowing the model to focus on a smaller number of
tokens. We experiment with values of $\beta\in\\{1,2,3,4\\}$ on the
development sets and find $\beta=2$ to significantly improve token labeling
performance without negatively affecting sentence classification results.

##  3 Datasets

We investigate the performance of these methods as zero-shot sequence labelers
using three different datasets. Gold token-level annotation in these datasets
is used for evaluation; however, the models are trained using sentence-level
labels only.

The CoNLL 2010 shared task (Farkas et al.,
2010)222https://rgai.sed.hu/node/118 focuses on the detection of uncertainty
cues in natural language text. The dataset contains $19,542$ examples with
both sentence-level uncertainty labels and annotated keywords indicating
uncertainty. We use the train/test data from the task and randomly choose
$10\%$ of the training set for development.

We also evaluate on the task of grammatical error detection (GED) –
identifying which sentences are grammatically incorrect (i.e., contain at
least one grammatical error). The First Certificate in English dataset FCE
Yannakoudakis et al. (2011) consists of essays written by non-native learners
of English, annotated for grammatical errors. We use the train/dev/test splits
released by Rei and Yannakoudakis (2016) for sequence labeling, with a total
of $33,673$ sentences.

In addition, we evaluate on the Write & Improve Yannakoudakis et al. (2018)
and LOCNESS (Granger, 1998) GED
dataset333https://www.cl.cam.ac.uk/research/nl/bea2019st/ ($38,692$ sentences)
released as part of the BEA 2019 shared task (Bryant et al., 2019). It
contains English essays written in response to varied topics and by English
learners from different proficiency levels, as well as native English
speakers. As the gold test set labels are not publicly available, we evaluate
on the released development set and use $10\%$ of the training data for
tuning444https://github.com/bujol12/bert-seq-
interpretability/blob/master/dev_indices_train_ABC.txt. For both GED datasets,
we train the model to detect grammatically incorrect sentences and evaluate
how well the methods can identify individual tokens that have been annotated
as errors.

| FCE | BEA 2019 | CoNLL 2010
---|---|---|---
| Sent $F_{1}$ | $F_{1}$ | MAP | Sent $F_{1}$ | $F_{1}$ | MAP | Sent $F_{1}$ | $F_{1}$ | MAP
Random baseline | - | 23.19 | 33.95 | - | 16.73 | 27.01 | - | 1.63 | 14.15
RoBERTa | 84.51 | - | - | 83.66 | - | - | 86.66 | - | -
Rei and Søgaard (2018) | 84.75 | 28.73 | 48.56 | 81.27 | 18.53 | 31.69 | 84.16 | 72.42 | 87.82
LIME | 84.51 | 24.60 | 37.90 | 83.66 | 2.09 | 31.41 | 86.66 | 57.14 | 78.44
Attention heads | 84.51 | 24.34 | 48.04 | 83.66 | 19.69 | 40.55 | 86.66 | 25.64 | 79.82
Soft attention | 85.62 | 32.16 | 48.90 | 83.41 | 22.92 | 35.79 | 86.25 | 8.45 | 20.04
Weighted soft attention | 85.62 | 33.31 | 53.91 | 83.68 | 24.35 | 41.07 | 87.20 | 67.28 | 91.18
Table 1:  Results on FCE, BEA 2019 and CoNLL 2010. Sent $F_{1}$ refers to
F-measure on the sentence classification task; $F_{1}$ refers to token-level
classification performance; MAP is the token-level Mean Average Precision.

##  4 Experimental setup

We use the pre-trained RoBERTa-base (Liu et al., 2019) model, made available
by HuggingFace (Wolf et al., 2020), as our transformer architecture. Following
Mosbach et al. (2021), transformer models are fine-tuned for $20$ epochs, and
the best performing checkpoint is then chosen based on sentence-level
performance on the development set. Each experiment is repeated with $5$
different random seeds and the averaged results are reported. The average
duration of training on Nvidia GeForce RTX 2080Ti was $1$ hour. Significance
testing is performed with a two-tailed paired t-test and $a=0.05$.
Hyperparameteres are tuned on the development set and presented in Appendices
B and C.

The LIME and attention head methods provide only a score without a natural
decision boundary for classification. Therefore, we choose their thresholds
based on the token-level $F_{1}$-score on the development set. In contrast,
the soft attention and weighted soft attention methods do not require such
additional tuning that uses token-level labels.

##  5 Results

The results are presented in Table 1. Each model is trained as a sentence
classifier and then evaluated as a token labeler. The challenge of the zero-
shot sequence-labeling setting lies in the fact that the models are trained
without utilizing any gold token-level signal; nevertheless, some methods
perform considerably better than others. For reference, we also include a
random baseline, which samples token-level scores from the standard uniform
distribution; a RoBERTa model supervised as a sentence classifier only; and
the model from Rei and Søgaard (2018) based on BiLSTMs.

We report the $F_{1}$-measure on the token level along with Mean Average
Precision (MAP) for returning positive tokens. The MAP metric views the task
as a ranking problem and therefore removes the dependence on specific
classification thresholds. In addition, we report the $F_{1}$-measure on the
main sentence-level task to ensure the proposed methods do not have adverse
effects on sentence classification performance. Precision and recall values
are included in Appendix E.

LIME has relatively low performance on FCE and BEA 2019, while it achieves
somewhat higher results on CoNLL 2010. Comparing the MAP scores, the attention
head method performs substantially better, especially considering that it is
much more lightweight and requires no additional computation. Nevertheless,
both of these methods rely on using some annotated examples to tune their
classification threshold, which precludes their application in a truly zero-
shot setting.

Figure 1: Example word-level importance scores $r_{i}$ (Eq. 6) of different
methods applied to an excerpt from the CoNLL10 dataset. HEAD corresponds to
attention heads; SA to soft attention; and W-SA to weighted soft attention. We
can observe how W-SA is the only method that correctly assigns substantially
higher weights to the ‘may’ and ‘seems’ uncertainty cues.

Combining the soft attention mechanism with the transformer architecture
provides some improvements over the previous methods, while also improving
over Rei and Søgaard (2018). A notable exception is the CoNLL 2010 dataset
where this method achieves only $8\%$ $F_{1}$ and $20\%$ MAP. Error analysis
revealed that this is due to the transformer representations spreading
attention scores evenly between a large number of tokens, as observed in
Figure 1. Uncertainty cues in CoNLL 2010 can span across whole sentences
(e.g., ‘Either … or …’), with such examples encouraging the model to
distribute information even further.

The weighted soft attention modification addresses this issue and considerably
improves performance across all metrics on all datasets. Compared to the non-
weighted version of the soft attention method, applying the extra weights
leads to a significant improvement in terms of MAP, with a minimum of $5.01\%$
absolute gain on FCE. The improvements are also statistically significant
compared to the current state of the art Rei and Søgaard (2018): $5.35\%$
absolute improvement on FCE; $9.38\%$ on BEA 2019; and $3.36\%$ on CoNLL 2010.
While the $F_{1}$ on CoNLL 2010 is slightly lower, the MAP score is higher,
indicating that the model has difficulty finding an optimal decision boundary,
but nevertheless provides a better ranking. In future work, the weighted soft
attention method for transformers could potentially be combined with token
supervision in order to train robust multi-level models Barrett et al. (2018);
Rei and Søgaard (2019).

##  6 Conclusion

We investigated methods for inferring token-level predictions from transformer
models trained only on sentence-level annotations. Experiments showed that
previous approaches designed for LSTM architectures do not perform as well
when applied to transformers. As transformer models already contain multiple
layers of multi-head attention, the input representations get distributed
between many tokens, making it more difficult to identify the importance of
each individual token. LIME was not able to accurately identify target tokens,
while the soft attention method primarily assigned equal attention scores
across most words in a sentence. Directly using the scores from the existing
attention heads performed better than expected, but required some annotated
data for tuning the decision threshold. Modifying the soft attention module
with an explicit sharpness constraint on the weights was found to encourage
more distinct predictions, significantly improving token-level results.

## Acknowledgments

We would like to thank James Thorne for his assistance in setting up the LIME
experiments. Kamil Bujel was funded by the Undergraduate Research
Opportunities Programme Bursary from the Department of Computing at Imperial
College London.

## References

  * Atanasova et al. (2020) Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. 2020\.  A diagnostic study of explainability techniques for text classification.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3256–3274, Online. Association for Computational Linguistics.
  * Barrett et al. (2018) Maria Barrett, Joachim Bingel, Nora Hollenstein, Marek Rei, and Anders Søgaard. 2018.  Sequence classification with human attention.  In _Proceedings of the 22nd Conference on Computational Natural Language Learning_ , pages 302–312, Brussels, Belgium. Association for Computational Linguistics.
  * Bell et al. (2019) Samuel Bell, Helen Yannakoudakis, and Marek Rei. 2019.  Context is key: Grammatical error detection with contextual word representations.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 103–115, Florence, Italy. Association for Computational Linguistics.
  * Bryant et al. (2019) Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019\.  The BEA-2019 shared task on grammatical error correction.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 52–75, Florence, Italy. Association for Computational Linguistics.
  * Camburu et al. (2018) Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018\.  e-snli: Natural language inference with natural language explanations.  In _NeurIPS_ , pages 9560–9572.
  * Clark et al. (2019) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019.  What does BERT look at? an analysis of BERT’s attention.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 276–286, Florence, Italy. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * DeYoung et al. (2020) Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020.  ERASER: A benchmark to evaluate rationalized NLP models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4443–4458, Online. Association for Computational Linguistics.
  * Farkas et al. (2010) Richárd Farkas, Veronika Vincze, György Móra, János Csirik, and György Szarvas. 2010.  The CoNLL-2010 shared task: Learning to detect hedges and their scope in natural language text.  In _Proceedings of the Fourteenth Conference on Computational Natural Language Learning – Shared Task_ , pages 1–12, Uppsala, Sweden. Association for Computational Linguistics.
  * Granger (1998) Sylviane Granger. 1998.  _The computer learner corpus: A versatile new source of data for SLA research_.  Longman.
  * Hearst et al. (1998) Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. 1998.  Support vector machines.  _IEEE Intelligent Systems and their applications_ , 13(4):18–28.
  * Herman (2017) Bernease Herman. 2017.  The promise and peril of human evaluation for model interpretability.  _arXiv preprint arXiv:1711.07414_.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Jacovi and Goldberg (2020) Alon Jacovi and Yoav Goldberg. 2020.  Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4198–4205, Online. Association for Computational Linguistics.
  * Jain and Wallace (2019) Sarthak Jain and Byron C. Wallace. 2019.  Attention is not Explanation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3543–3556, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Lan and Xu (2018) Wuwei Lan and Wei Xu. 2018.  Neural network models for paraphrase identification, semantic textual similarity, natural language inference, and question answering.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 3890–3902, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Martins and Astudillo (2016) Andre Martins and Ramon Astudillo. 2016.  From softmax to sparsemax: A sparse model of attention and multi-label classification.  In _International Conference on Machine Learning_ , pages 1614–1623. PMLR.
  * Mosbach et al. (2021) Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2021.  On the stability of fine-tuning {bert}: Misconceptions, explanations, and strong baselines.  In _International Conference on Learning Representations_.
  * Rei and Søgaard (2018) Marek Rei and Anders Søgaard. 2018.  Zero-shot sequence labeling: Transferring knowledge from sentences to tokens.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 293–302, New Orleans, Louisiana. Association for Computational Linguistics.
  * Rei and Søgaard (2019) Marek Rei and Anders Søgaard. 2019.  Jointly learning to label sentences and tokens.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 6916–6923.
  * Rei and Yannakoudakis (2016) Marek Rei and Helen Yannakoudakis. 2016.  Compositional sequence labeling models for error detection in learner writing.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1181–1191, Berlin, Germany. Association for Computational Linguistics.
  * Ribeiro et al. (2016) Marco Ribeiro, Sameer Singh, and Carlos Guestrin. 2016.  “Why Should I Trust You?”: Explaining the Predictions of Any Classifier.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations_ , pages 97–101, San Diego, California. Association for Computational Linguistics.
  * Sun et al. (2019) Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019.  How to fine-tune bert for text classification?  In _China National Conference on Chinese Computational Linguistics_ , pages 194–206. Springer.
  * Thorne et al. (2019) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2019\.  Generating token-level explanations for natural language inference.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 963–969, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Wiegreffe and Pinter (2019) Sarah Wiegreffe and Yuval Pinter. 2019.  Attention is not not explanation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 11–20, Hong Kong, China. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Yannakoudakis et al. (2018) Helen Yannakoudakis, Øistein E Andersen, Ardeshir Geranpayeh, Ted Briscoe, and Diane Nicholls. 2018.  Developing an automated writing placement system for esl learners.  _Applied Measurement in Education_ , 31(3):251–267.
  * Yannakoudakis et al. (2011) Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011.  A new dataset and method for automatically grading ESOL texts.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 180–189, Portland, Oregon, USA. Association for Computational Linguistics.

##  Appendix A Example word-level predictions

We present samples of word-level predictions (word-level importance scores
$r_{i}$, Eq. 6) to illustrate differences between methods. In the figures that
follow, HEAD refers to attention heads, SA to soft attention, and W-SA to
weighted soft attention.

Figure 2: CoNLL 2010 negative sentence (without uncertainty cues). We can
clearly see that most methods correctly put weights close to $0$ for all
words, except from HEAD, which focuses on ‘shown’ and ‘.’. We surmise this is
due to the fact that, for HEAD, weights over the whole sentence have to sum up
to $1$. Figure 3: CoNLL 2010 positive sentence (with uncertainty cues). We can
observe that HEAD correctly identifies both of the uncertainty cues: ‘may’ and
‘seems’; however the weight for ‘may’ is quite low. Similarly, LIME identifies
both tokens, but the weight for ‘seems’ is particularly low (lower than for
‘to’). SA simply assigns high weights to all words. W-SA focuses primarily on
the two uncertainty cue words; however, it also incorrectly focuses on ‘not’.
Figure 4: FCE positive sentence (contains grammatical errors). We can see that
both LIME and HEAD struggle to assign informative and/or useful weights to the
words. All SA weights are relatively high, with small variations in value. We
can see that squaring (W-SA) leads to more well-defined weights over the whole
sentence, with high weights mainly observed in the second part of the
sentence, which is the one that contains incorrect words. However, on this
dataset, even W-SA struggles to correctly identify which words precisely are
incorrect.

##  Appendix B Hyperparameters

Name | Value
---|---
$\gamma$ | 0.1
max seq length | 128
per device train batch size | 16
per device eval batch size | 64
warmup ratio | 0.1
learning rate | 2e-5
weight decay | 0.1
adam epsilon | 1e-7
hidden layer dropout | 0.1
soft attention layer size | 100
soft attention hidden size | 300
initializer | glorot
Table 2:  Model hyperparameters.

##  Appendix C Word-level prediction thresholds

Dataset | Method | Threshold
---|---|---
CoNLL 2010 | LIME | 0.200
| Random baseline | 0.500
| Attention heads | 0.320
| Rei and Søgaard (2018) | 0.500
| Soft attention | 0.500
| Weighted soft attention | 0.500
FCE | LIME | 0.001
| Random baseline | 0.500
| Attention heads | 0.080
| Rei and Søgaard (2018) | 0.500
| Soft attention | 0.500
| Weighted soft attention | 0.500
BEA 2019 | LIME | 0.010
| Random baseline | 0.500
| Attention heads | 0.080
| Rei and Søgaard (2018) | 0.500
| Soft attention | 0.500
| Weighted soft attention | 0.500
Table 3: Word-level thresholds above which a word is classified as positive.

##  Appendix D Validation set results

Dataset | Method | Sent $F_{1}$
---|---|---
CoNLL 2010 | LIME | 91.77
| RoBERTa | 91.77
| Attention heads | 91.77
| Soft attention | 92.12
| Weighted soft attention | 91.82
FCE | LIME | 84.49
| RoBERTa | 84.49
| Attention heads | 84.49
| Soft attention | 84.82
| Weighted soft attention | 85.56
BEA 2019 | LIME | 83.65
| RoBERTa | 83.65
| Attention heads | 83.65
| Soft attention | 83.47
| Weighted soft attention | 83.64
Table 4: Mean sentence-level $F_{1}$ score on the development set, averaged
over $5$ runs.

##  Appendix E Full test set results

| FCE
---|---
| Sent $F_{1}$ | Sent P | Sent R
Random baseline | - | - | -
RoBERTa | 84.51 | 84.25 | 84.93
Rei and Søgaard (2018) | 84.75 | - | -
LIME | 84.51 | 84.25 | 84.93
Attention heads | 84.51 | 84.25 | 84.93
Soft attention | 85.62 | 86.92 | 84.42
Weighted soft attention | 85.62 | 86.88 | 84.45
Table 5:  Sentence-level results: P, R and $F_{1}$ refer to Precision, Recall and F-measure respectively on the positive class.  | BEA 2019
---|---
| Sent $F_{1}$ | Sent P | Sent R
Random baseline | - | - | -
RoBERTa | 83.66 | 82.29 | 85.15
Rei and Søgaard (2018) | 81.27 | - | -
LIME | 83.66 | 82.29 | 85.15
Attention heads | 83.66 | 82.29 | 85.15
Soft attention | 83.41 | 81.47 | 85.54
Weighted soft attention | 83.68 | 79.95 | 87.91
Table 6:  Sentence-level results: P, R and $F_{1}$ refer to Precision, Recall and F-measure respectively on the positive class.  | CoNLL 2010
---|---
| Sent $F_{1}$ | Sent P | Sent R
Random baseline | - | - | -
RoBERTa | 86.66 | 84.90 | 88.63
Rei and Søgaard (2018) | 84.16 | - | -
LIME | 86.66 | 84.90 | 88.63
Attention heads | 86.66 | 84.90 | 88.63
Soft attention | 86.25 | 85.75 | 86.89
Weighted soft attention | 87.20 | 89.17 | 85.37
Table 7:  Sentence-level results: P, R and $F_{1}$ refer to Precision, Recall
and F-measure respectively on the positive class.

| FCE
---|---
| P | R | $F_{1}$ | MAP
Random baseline | 15.11 | 49.81 | 23.19 | 33.95
RoBERTa | - | - | - | -
Rei and Søgaard (2018) | 29.16 | 29.04 | 28.73 | 48.56
LIME | 19.06 | 34.70 | 24.60 | 37.90
Attention heads | 26.67 | 22.38 | 24.34 | 48.04
Soft attention | 19.84 | 85.38 | 32.16 | 48.90
Weighted soft attention | 20.76 | 85.36 | 33.31 | 53.91
Table 8:  Token-level results: P, R and $F_{1}$ refer to Precision, Recall and F-measure respectively on the positive class. MAP is the Mean Average Precision at the token-level.  | BEA 2019
---|---
| P | R | $F_{1}$ | MAP
Random baseline | 10.05 | 50.00 | 16.73 | 27.01
RoBERTa | - | - | - | -
Rei and Søgaard (2018) | 10.93 | 61.63 | 18.53 | 31.69
LIME | 13.49 | 1.13 | 2.09 | 31.41
Attention heads | 18.48 | 21.07 | 19.69 | 40.55
Soft attention | 13.20 | 87.19 | 22.92 | 35.79
Weighted soft attention | 14.20 | 85.49 | 24.35 | 41.07
Table 9:  Token-level results: P, R and $F_{1}$ refer to Precision, Recall and F-measure respectively on the positive class. MAP is the Mean Average Precision at the token-level.  | CoNLL 2010
---|---
| P | R | $F_{1}$ | MAP
Random baseline | 0.83 | 49.70 | 1.63 | 14.15
RoBERTa | - | - | - | -
Rei and Søgaard (2018) | 78.99 | 67.06 | 72.42 | 87.82
LIME | 63.25 | 52.11 | 57.14 | 78.44
Attention heads | 22.33 | 30.11 | 25.64 | 79.82
Soft attention | 4.48 | 86.14 | 8.45 | 20.04
Weighted soft attention | 58.80 | 78.89 | 67.28 | 91.18
Table 10:  Token-level results: P, R and $F_{1}$ refer to Precision, Recall
and F-measure respectively on the positive class. MAP is the Mean Average
Precision at the token-level.

##  Appendix F Weighted soft attention architecture

Figure 5: Architecture of our proposed weighted soft attention model.
$[t_{1},t_{2},...,t_{n}]$ represent the tokenized input sentence, while
$[T_{1},T_{2},...,T_{n}]$ are the resulting contextual embeddings.
$[e_{1},e_{2},...,e_{n}]$ are attention vectors, and $[a_{1},a_{2},...,a_{n}]$
are normalized attention weights. $d$ represents the output vector and $y$ the
final output logits.


# MetaViewer: Towards A Unified Multi-View Representation

Ren Wang
Shandong University
<EMAIL_ADDRESS>Haoliang Sun
Shandong University
<EMAIL_ADDRESS>
Yuling Ma
Shandong Jianzhu University
<EMAIL_ADDRESS>Xiaoming Xi
Shandong Jianzhu University
<EMAIL_ADDRESS>Yilong Yin
Shandong University
<EMAIL_ADDRESS>
###### Abstract

Existing multi-view representation learning methods typically follow a
specific-to-uniform pipeline, extracting latent features from each view and
then fusing or aligning them to obtain the unified object representation.
However, the manually pre-specify fusion functions and view-private redundant
information mixed in features potentially degrade the quality of the derived
representation. To overcome them, we propose a novel bi-level-optimization-
based multi-view learning framework, where the representation is learned in a
uniform-to-specific manner. Specifically, we train a meta-learner, namely
MetaViewer, to learn fusion and model the view-shared meta representation in
outer-level optimization. Start with this meta representation, view-specific
base-learners are then required to rapidly reconstruct the corresponding view
in inner-level. MetaViewer eventually updates by observing reconstruction
processes from uniform to specific over all views, and learns an optimal
fusion scheme that separates and filters out view-private information.
Extensive experimental results in downstream tasks such as classification and
clustering demonstrate the effectiveness of our method.

##  1 Introduction

Multi-view representation learning mines a unified representation from
multiple views of the same entity [27, 55, 39, 58]. Each view acquired by
different sensors or sources contains both view-shared consistency information
and view-specific information. Among them, view-specific information consists
of complementary and redundant components, where the former can be considered
as a supplement to the consistency information, while the latter is highly
specific and may be adverse for the unified representation [8, 16]. Therefore,
a high-quality representation is required to retain the consistency and
complementary information, as well as filter out the view-private redundant
ones [48].

Figure 1: (a), (b) and (c) show three multi-view learning frameworks following
the specific-to-uniform pipeline, where the unified representation is obtained
by fusing or concatenating view-specific features. (d) illustrates our
uniform-to-specific manner, where a meta-learner learns to fusion by observing
reconstruction from unified representation to specific views.

Given the data containing two views $x_{1}$ and $x_{2}$, the prevailing multi-
view representation methods typically follow a specific-to-uniform pipeline
and can be roughly characterized as:

$H:=f(x_{1};W_{f})\circ g(x_{2};W_{g}),$ (1)

where $f$ and $g$ are encoding (or embedding [28]) functions that map the
original view data into the corresponding latent features with the trainable
parameters $W_{f}$ and $W_{g}$. These latent features are subsequently
aggregated into the unified representation using the designed aggregation
operator $\circ$. with different aggregation strategies, existing approaches
can be further subdivided into the joint, alignment, and a combined share-
specific (S$\&$S) representation [3, 27, 23].

Joint representation focuses on the integration of complementary information
by directly fusing latent features, where the $\circ$ is represented as fusion
strategies, such as graph-based module [41, 34], neural network [33, 10], or
other elaborate functions [37, 18, 7]. While alignment representation seeks to
perform alignment between view-specific features to retain the consistency
information and specifies $\circ$ as the alignment operator, measured by
distance [11, 26], similarity [14, 25], or correlation[1, 44, 24]. As a trade-
off way, S$\&$S representation explicitly distinguishes latent features into
shared and specific representation and only aligns the shared part [50, 21,
30, 23]. Fig. 1 (a) - (c) show the above three branches of the specific-to-
uniform paradigm.

Despite demonstrating promising results, the aggregation way inherently
suffers from potential risks in the following two aspects: (1) The derived
unified representation is usually the concatenation or fusion of learned
latent features with the manually pre-specify rules. It makes them fairly hard
to be generally applied in practice due to the significant variation of fusion
schemes relying on the downstream tasks and training views [36]. (2) Even
finding a well-performing fusion scheme, view-private redundant information
mixed in latent features also degrades the quality of the fused unified
representation. Several studies have noticed the second issue and attempted to
distinguish the redundant information from view features via multi-level
feature modeling [48] or matrix factorization [57]. However, recent works
indicate the view-specific information could be not automatically separated at
feature level [35, 23]. In addition, the first issue has received little
attention.

In this paper, we propose a novel multi-view representation learning framework
based on bi-level optimization meta-learning. In contrast to the specific-to-
uniform pipeline, our model emphasizes learning unified representation in a
uniform-to-specific manner, as illustrated in Fig. 1 (d). In detail, we build
a meta-learner, namely MetaViewer, to learn fusion and model a unified meta
representation in outer-level optimization. Based on this meta representation,
view-specific base-learners are required to rapidly reconstruct the
corresponding view in the inner-level. MetaViewer eventually updates by
observing reconstruction processes over all views, thus learning optimal
fusion rules to address the first issue. On the other hand, the rapid
reconstruction from uniform representation to specific views in inner-level
optimization essentially models the information that cannot be fused, i.e.,
view-private parts, solving the second issue. After alternate training, the
resulting meta representation is closer to each view, which indicates that it
contains as much view-shared information as possible as well as avoids the
hindrance of redundant information. Extensive experiments on multiple
benchmarks validate the performance of our MetaViewer. The unified meta
representation learned from multiple views achieves comparable performance to
state-of-the-art methods in downstream tasks such as clustering and
classification. The core contributions of this paper are as follows.

  1. 1.

We propose a novel insight for multi-view representation learning, where the
unified representation is learned in a uniform-to-specific manner.

  2. 2.

Our MetaViewer achieves the data-driven fusion of view features in meta-
learning paradigm. To the best of our knowledge, this could be the first meta-
learning-based work in multi-view representation scenarios.

  3. 3.

MetaViewer decouples the modeling of view-shared and view-private information
via bi-level optimization, alleviating the hindrance of redundant information.

  4. 4.

Extensive experimental results validate the performance of our approach in
several downstream tasks.

##  2 Related Work

###  2.1 Multi-view learning

Multi-view representation learning is not a new topic and has been widely used
in downstream tasks such as retrieval, classification, and clustering. This
work focuses on multi-view representation in unsupervised deep learning scope,
and related works can be summarized into two main categories [51]. One is the
deep extension of traditional methods, where representative ones include deep
canonical correlation analysis (DCCA) [1] and its variants [44, 40, 54]. DCCA
intends to discover the nonlinear mapping for two views to a common space in
which their canonical correlations are maximally preserved. These methods
benefit from a sound theoretical foundation, but also usually have strict
restrictions on the number and form of views.

Another alternative is the multi-view deep network. Early deep-based
approaches attempted to build different architectures for handling multi-view
data, such as CNN-based [12, 53, 38] and GAN-based model [49, 19], etc. Some
recent approaches focus on better parameter constraints using mutual
information [2, 8], comparative information [28, 52], etc. Most of these
existing methods follow a specific-to-uniform pipeline. In contrast, the
underlying assumption of our MetaViewer learns from uniform to specific. The
most related work is the MFLVC [48], which separates view-private information
from latent features at the parameter level. The essential difference is that
we model the view-private information in inner-level optimization, allowing
outer-level observes the modeling process and future meta learners the optimal
fusion scheme.

Figure 2: The overall framework of MetaViewer, contains (a) the meta split of
multi-view data and three modules (b) embedding module, (c) representation
module, and (d) self-supervised module. These modules are trained with a bi-
level optimization. Inner-level (dark gray arrows) learns the view-specific
reconstruction on the support set, and outer-level (red arrows) updates the
entire model to learn the fusion scheme and the unified representation $H$ for
downstream tasks by validating on the query set.

###  2.2 Meta-learning

Optimization-based meta-learning is a classic application of bi-level
optimization designed to learn task-level knowledge to quickly handle new
tasks [20, 22]. A typical work, MAML [13], learns a set of initialization
parameters to solve different tasks with few steps of updates. Similar meta
paradigm has been used to learn other manually designed parts, such as the
network structure [29], optimizer[45, 59], and even sample weight [36, 32].
Similarly, we try to meta-learn fusion of multi-view feature for a unified
representation. There also exist some works that consider both meta-learning
paradigms and multi-view data[42, 15, 31]. However, they are dedicated to
exploit the rich information contained in multiple views to improve the
performance of the meta-learner on few-shot tasks or self-supervised scenario.
Instead, we train a meta-learner to derive the high-quality shared
representations from multi-view data via the bi-level optimization. To the
best of our knowledge this is the first work to learn multi-view
representation with meta-learning paradigm.

##  3 MetaViewer

Given a set of unlabeled multi-view dataset
$\mathcal{D}=\\{x_{i}\in\mathbb{R}^{d_{x}}\\}_{i=1}^{N}$, where $N$ are the
number of samples and each sample entity
$x_{i}=\\{x_{i}^{1},x_{i}^{2},\dots,x_{i}^{v}\\}_{v=1}^{V}$ contains $V$
views. The view-incomplete scenario means that partial views of some samples
are missing or unavailable, i.e.,
$\mathcal{D}_{inc}=\\{\mathcal{D}_{c},\\{\mathcal{D}_{u}^{v}\\}_{v=1}^{V}\\}$,
where $\mathcal{D}_{c}$ and $\mathcal{D}_{u}^{v}$ indicate subsets of samples
with complete views and unavailable $v$-$th$ view, respectively. Our goal is
learn a unified high-quality representation $H$ for each entity by observing
available views and filtering self-private information as much as possible.
The overall framework of our MetaViewer is shown in Fig. 2, including three
mainly modules and a bi-level optimization process. The outer-level learns a
meta-learner to learn a optimal fusion function and derive the unified
representation $H$, and inner-level reconstructs original views from $H$ in
few update steps, which explicitly models and separates view-private
information and ensure the representation quality. In following subsections,
we first introduce the entire structure of the MetaViewer and then elaborate
the bi-level optimization process.

###  3.1 The entire structure

Embedding module aims to transform heterogeneous view into the latent feature
space, where transformed view embeddings have the same dimension as each
other. To this end, we conduct a view-specific embedding function $f_{v}$ for
each view, where $v=1,2,\dots,V$. Given the $v$-$th$ view data $x^{v}$ of the
entity $x$, the corresponding embedding $z^{v}$ can be computed by

$z^{v}=f_{v}(x^{v},\phi_{f_{v}}),$ (2)

where $z^{v}\in\mathbb{R}^{d}$ and $f_{v}$ typically instantiated as a multi-
layer neural network with learnable parameters $\phi_{f_{v}}$.

Representation learning module maps the obtained embedding to the view
representation, consisting of view-specific base-learners
${\\{b_{v}\\}_{v=1}^{V}}$ and a view-shared meta-learner $m$ (i.e.,
MetaViewer). The former learns representation for each view embedding, while
the latter takes all embeddings as input and outputs the unified
representation $H$ that is ultimately used for downstream tasks. Meanwhile,
base-learners are generally required to be initialized from parameters of the
meta-learner to learning view-shared meta representation (see 3.2), thus two
types learners should be structurally shared rather than individually
designed. To meet the above two requirements simultaneously, MetaViewer is
implemented as a channel-oriented $1$-$d$ convolutional layer (C-Conv) with a
non-linear function (e.g., ReLU[17]), as shown in Fig. 2 (c).

On the one hand, as a meta-learner, we first concatenate embeddings at the
channel level, i.e., the number of the channel in the concatenated feature is
equal to the number of views, and then train the MetaViewer to learn the
fusion of cross-view information $H\in\mathbb{R}^{d_{h}}$ by

$H=m(z^{cat},\omega),$ (3)

where $z^{cat}\in\mathbb{R}^{d\times V}$ indicates the concatenated embedding
and $\omega$ is the parameter of the MetaViewer. On the other hand, base-
learners could be initialized and trained for learning $v$-$th$ representation
$h_{base}^{v}$ via

$h_{base}^{v}=b_{v}(z^{v},\theta_{b_{v}}(\omega_{sub})),$ (4)

where $h_{base}^{v}\in\mathbb{R}^{d_{h}}$ and $\theta_{b_{v}}(\omega_{sub})$,
or $\theta_{b_{v}}(\omega)$ for short, means the base-learner’s parameter
$\theta_{b_{v}}$ is initialized from the channel(view)-related part
$\omega_{sub}$ of the MetaViewer’s parameters. Note that this sub-network
mechanism also provides a convenient way to handle incomplete views.

Self-supervised module conducts pre-text tasks to provide effectively
supervised objects for model training, represented by different heads.
Typically, the reconstruction head $r$ achieves the reconstruction object by
re-mapping the representation back to the original view space, i.e.,

$x_{rec}^{v}=r_{v}(z_{rec}^{v},\phi_{r_{v}}),$ (5)

where $x_{rec}^{v}\in\mathbb{R}^{d_{x}}$ is the reconstruction result and
$r_{v}$ is the reconstruction function with learnable parameters
$\phi_{r_{v}}$. Expect that, we can also conduct contrastive or correlation
head for the meta-learner to mine the associations across views. Similar self-
supervised objectives [40, 54, 28, 52] have been extensively studied in multi-
view learning, and this is not the focus of this work.

###  3.2 Training via bi-level optimization

Now, we have conducted the entire structure, which can be end-to-end trained
to derive the unified multi-view representation, even for incomplete views, in
the specific-to-uniform manner like most existing approaches. However the
data-driven fusion and view-private redundant information still cannot be
handling well. So we turn to a opposite uniform-to-specific way using a bi-
level optimization process, inspired by the meta-learning paradigm. Inner-
level focus on the training of view-specific modules for corresponding views,
and outer-level updates meta-learner to find the optimal fusion rule through
observing the learning over all views. Before the detailed description, we
introduce a split way of multi-view data in meta-learning style.

Meta-split of multi-view data. Consider a batch multi-view samples
$\\{\mathcal{D}_{batch}^{v}\\}_{v=1}^{V}$ from $\mathcal{D}$. For bi-level
updating, we randomly and proportionally divide it into two disjoint subsets,
marked as support set $S$ and query set $Q$, respectively. As shown in 2 (a),
support set is used in inner-level for leaning view-specific information, thus
the sample attributes in it could be ignored. In contrast, query set retains
both view and sample attributes for meta-learner training in outer-level
optimization. This meta-split that decoupling view from samples can be
naturally transferred to the data with incomplete views
$\mathcal{D}_{inc}=\\{\mathcal{D}_{c},\\{\mathcal{D}_{u}^{v}\\}_{v=1}^{V}\\}$,
where subset with incomplete views $\\{\mathcal{D}_{u}^{v}\\}_{v=1}^{V}$ is
used as support set, and the complete part $\mathcal{D}_{c}$ is left as query
set.

Figure 3: Intuitively, the specific-to-uniform observes view features and
learns a representation that falls in a reconstructable space. MetaViewer
observes the reconstruction process and seeks a unified meta representation
that is close as possible to each view. Algorithm 1 The framework of our
MetaViewer.

Require: Training dataset $\mathcal{D}$, meta parameters $\omega$, base
parameters $\\{\theta_{v}\\}_{v=1}^{V}$, view-specific parameters
$\\{\phi_{v}\\}_{v=1}^{V}$, the number of view $V$, the iteration step in
inner-level optimization $T$.

1: Initialize $\omega$, $\\{\phi_{v}\\}_{v=1}^{V}$;

2: while not done do

3: # $Outer$-$level$

4: Sample and meta-split a batch set from $\mathcal{D}$:

5: $\\{\mathcal{D}_{batch}^{v}\\}_{v=1}^{V}=\\{S,Q\\}$.

6: for $t=1,\dots,T$ do

7: for $v=1,\dots,V$ do

8: # $Inner$-$level$

9: Initialize $\theta_{v}=\theta_{v}(\omega)$, $\tilde{\phi_{v}}=\phi_{v}$;

10: Optimize $\theta_{v}(\omega)$ and $\tilde{\phi_{v}}$ via Eq. 6.

11: end for

12: end for

13: Optimize $\omega$ via Eq. 8.

14: Optimize $\\{\phi_{v}\\}_{v=1}^{V}$ via Eq. 8.

15: end while

Inner-level optimization. Without loss of generality, take the inner-level
update after the $o$-$th$ outer-level optimization as an example. Let
$\omega^{o}$ be the lasted parameters of the MetaViewer, and
$\phi_{v}^{o}=\\{\phi_{f_{v}}^{o},\phi_{r_{v}}^{o}\\}$ denotes the lasted
parameters in embedding and self-supervised modules for brevity. We first
initial base-learner from meta-learner, i.e., $\theta_{b_{v}}^{0}=\omega^{o}$,
and make a copy of $\phi_{v}^{o}$ for $\tilde{\phi}_{v}$. Note that the copy
means gradients with respect to the $\phi_{v}^{o}$ will not be back-propagated
to $\tilde{\phi}_{v}$ and vice versa. Thus, $\theta_{b_{v}}^{0}$ and
$\tilde{\phi}_{v}$ form the initial state for the inner-level optimization.
Suppose the $\mathcal{L}_{inner}^{v}$ is the loss function of the inner-level
with respect to the $v$-$th$ view, and then the corresponding update goals are

$\theta_{b_{v}}^{\ast}(\omega),\tilde{\phi}_{v}^{\ast},=\mathop{\arg\min}\mathcal{L}_{inner}^{v}\left(\theta_{b_{v}}(\omega^{o}),\tilde{\phi}_{v};S^{v}\right).$
(6)

Consider a gradient descent strategy (e.g., SGD [4]), we can further write the
update process of $\theta_{b_{v}}$:

$\theta_{b_{v}}^{i}=\theta_{b_{v}}^{i-1}-\beta\frac{\partial\mathcal{L}_{inner}^{v}}{\partial\theta_{b_{v}}^{i-1}},\dots,\theta_{b_{v}}^{0}=\omega^{o},$
(7)

where $\beta$ and $i$ denote the learning rate and iterative step of inner-
level optimization, respectively.

Outer-level optimization. After several inner-level updates, we obtain a set
of optimal view-specific parameters on the support set. Outer-level then
updates the meta-learner, embedding, and head modules by training on the query
set. With the loss function $\mathcal{L}_{outer}$, the outer-level
optimization goal is

$\omega^{\ast},\\{\phi_{v}^{\ast}\\}_{1}^{V}=\mathop{\arg\min}\mathcal{L}_{outer}\left(\theta^{*}(\omega),\\{\phi_{v}\\}_{1}^{V};Q\right).$
(8)

By alternately optimizing Eq. 6 and Eq. 8, we end up with the optimal meta-
parameters $\omega^{*}$ and a set of view-specific parameters
$\phi_{v}^{\ast}$. For a test sample $x_{test}$, its corresponding
representation is derived by sequentially feeding the embedding function and
the meta-learner. The overall framework of our MetaViewer is shown in Alg. 1.

Datasets | #Views | #Classes | #Samples (train, val, test) | View Dimensions
---|---|---|---|---
BDGP | 2 | 5 | 2500 (1500, 500, 500) | 1750; 79
Handwirtten | 2 | 10 | 2000 (1200, 400, 400) | 240; 216
RGB-D | 2 | 50 | 500 (300, 100, 100) | 12288; 4096
Fashion-MV | 3 | 10 | 10000 (6000, 2000, 2000) | 784; 784; 784
MSRA | 6 | 7 | 210 (126, 42, 42) | 1302; 48; 512; 100; 256; 210
Caltech101-20 | 6 | 20 | 2386 (1425, 469, 492) | 48; 40; 254; 1984; 512; 928
Table 1: The attributes for all datasets used in our experiments. Datasets | Metrics | DCCA [1] | DCCAE [44] | MIB [8] | MFLVC [48] | DCP [28] | MVer-R | MVer-C
---|---|---|---|---|---|---|---|---
BDGP | ACC | 0.4640 | 0.5180 | 0.6940 | 0.8800 | 0.7820 | 0.8280 | 0.9040
NMI | 0.4163 | 0.5793 | 0.5565 | 0.8397 | 0.7800 | 0.7979 | 0.8627
ARI | 0.3347 | 0.3208 | 0.4865 | 0.8504 | 0.6725 | 0.6156 | 0.8925
Handwritten | ACC | 0.5725 | 0.6300 | 0.6325 | 0.6400 | 0.6625 | 0.7500 | 0.8625
NMI | 0.6980 | 0.7504 | 0.6758 | 0.6453 | 0.7056 | 0.7853 | 0.7896
ARI | 0.5215 | 0.5929 | 0.5216 | 0.4885 | 0.5610 | 0.6721 | 0.7225
RGB-D | ACC | 0.5100 | 0.4800 | 0.5000 | 0.5300 | 0.5200 | 0.5300 | 0.5700
NMI | 0.8299 | 0.8158 | 0.8113 | 0.8331 | 0.8204 | 0.8241 | 0.8497
ARI | 0.5202 | 0.4834 | 0.5127 | 0.5407 | 0.5264 | 0.5304 | 0.5707
Fashion-MV | ACC | 0.7070 | 0.7105 | 0.5720 | 0.8320 | 0.6260 | 0.8080 | 0.8540
NMI | 0.8042 | 0.8112 | 0.7383 | 0.8875 | 0.6838 | 0.8813 | 0.8876
ARI | 0.6180 | 0.6234 | 0.4762 | 0.7893 | 0.5430 | 0.7505 | 0.8007
MSRA | ACC | 0.3333 | 0.3571 | 0.3095 | 0.7143 | 0.6429 | 0.7143 | 0.7018
NMI | 0.2997 | 0.3285 | 0.2471 | 0.6796 | 0.6801 | 0.7007 | 0.6126
ARI | 0.3341 | 0.3471 | 0.3035 | 0.6891 | 0.6308 | 0.6893 | 0.6029
Caltech101-20 | ACC | 0.3862 | 0.3659 | 0.3598 | 0.3659 | 0.3679 | 0.4187 | 0.4512
NMI | 0.5088 | 0.5224 | 0.4700 | 0.5836 | 0.4437 | 0.5852 | 0.6086
ARI | 0.2273 | 0.2525 | 0.2218 | 0.2687 | 0.2350 | 0.2919 | 0.3500
Table 2: Clustering results of all methods on six datasets. Bold and underline
denote the best and second-best results, respectively.

###  3.3 Specific-to-uniform versus uniform-to-specific

We discuss the difference between specific-to-uniform and uniform-to-specific
paradigm through the update of fusion parameters $\omega$ with a
reconstruction loss $\mathcal{L}_{rec}^{v}$. Using the same structure
described in Sec. 3.1, the specific-to-uniform generally optimizes $\omega$ by
minimizing reconstruction losses over all views, i.e.,
$\omega^{\ast}=\mathop{\arg\min}\sum_{v=1}^{V}\mathcal{L}_{rec}^{v}(r_{v}(m(z^{v},\omega),\phi_{r_{v}})),x_{v})$,
and the $\omega$ is updated by (with the SGD)

$\omega\leftarrow\omega-\alpha\sum_{v=1}^{V}\nabla_{\omega}\mathcal{L}^{v}_{rec}(\omega).$
(9)

The optimal $\omega^{\ast}$ observes all views and derives the unified
representation $H$, that is, from the particular to the general. While the
update of $\omega$ in our uniform-to-specific can be written

$\omega\leftarrow\omega-\alpha\sum_{v=1}^{V}\nabla_{\omega}\mathcal{L}^{v}_{rec}\left(\theta^{\ast}_{b_{v}}(\omega)\right).$
(10)

Note that $\theta^{\ast}_{b_{v}}(\omega)$ contain the optimization process of
each view in inner-level as Eq. (6), which means that the optimal
$\omega^{\ast}$ update by observing the reconstruction from unified
representation to specific views. Fig. 3 intuitively demonstrates the
difference between these two manner.

###  3.4 The instances of the objective function

Our uniform-to-specific framework emphasizes learning from reconstruction in
inner-level, thus the $\mathcal{L}_{inner}^{v}$ is specified as the
reconstruction loss [56, 28]

$\mathcal{L}_{inner}^{v}=\mathcal{L}_{rec}^{v}(S^{v},S_{rec}^{v})=\|S^{v}-S_{rec}^{v}\|_{F}^{2}.$
(11)

While parameters updated in outer-level can be constrained by richer self-
supervised feedbacks as mentained in Sec. 3.1. Here we provide two instances
of outer-level loss function to demonstrate how MetaViewer can be extended
with different learning objectives.

MVer-R adopts the same reconstruction loss as the inner-level and
$\mathcal{L}_{outer}=\sum_{v}\mathcal{L}_{rec}^{v}(Q^{v},Q_{rec}^{v})$, which
is the purest implementation of MetaViewer.

MVer-C additionally utilizes a contrastive objective, where the similarities
of views belonged to same entity (i.e., positive pairs) should be maximized
and that of different entities (i.e., negative pairs) should be minimized,
i.e.,
$\mathcal{L}_{outer}=\sum_{v}\left(\mathcal{L}_{rec}^{v}+\sum_{v^{\prime},v^{\prime}\neq
v}\mathcal{L}_{con}^{v,v^{\prime}}\right)$. Following previous work [48, 19,
52], the contrastive loss $\mathcal{L}_{con}^{v,v^{\prime}}$ is formed as

$\small\mathcal{L}_{con}^{v,v^{\prime}}=-\frac{1}{N_{Q}}\sum_{i=1}^{N_{Q}}log\frac{e^{d(q_{i}^{v},q_{i}^{v^{\prime}})/\tau}}{\sum_{j=1,j\neq
i}^{N_{Q}}e^{d(q_{i}^{v},q_{j}^{v})/\tau}+\sum_{j=1}^{N_{Q}}e^{d(q_{i}^{v},q_{j}^{v^{\prime}})/\tau}},$
(12)

where $q_{i}^{v}$ is the $v$-$th$ views of the $i$-$th$ query sample, and $d$
is the similarity metric (e.q., cosine similarity [6]). The $N_{Q}$ and $\tau$
denote the number of query set samples and the temperature parameter,
respectively. Note that, the derived meta representation can be also used in
contrastive learning as a additional noevl view.

##  4 Experiments

In this section, we present extensive experimental results to validate the
quality of the unified representation derived from our MetaViewer. The
remainder of the experiments are organized as follows: Subsection 4.1 lists
datasets, compared methods and the implementation details. Subsection 4.2
compares the performance of our method with classical and state-of-the-art
methods on two common downstream scenarios, clustering and classification
tasks. Comparison with manually designed fusion and ablation studies are shown
in Subsection 4.3 and 4.4, respectively.

Datasets | Metrics | DCCA [1] | DCCAE [44] | MIB [8] | MFLVC [48] | DCP [28] | MVer-R | MVer-C
---|---|---|---|---|---|---|---|---
BDGP | ACC | 0.9840 | 0.9865 | 0.8900 | 0.9820 | 0.9720 | 0.9860 | 0.9800
Precision | 0.9842 | 0.9863 | 0.9005 | 0.9822 | 0.9726 | 0.9871 | 0.9859
F-score | 0.9840 | 0.9850 | 0.8884 | 0.9820 | 0.9720 | 0.9859 | 0.9802
Handwritten | ACC | 0.8825 | 0.9000 | 0.7900 | 0.9400 | 0.9725 | 0.9700 | 0.9775
Precision | 0.8920 | 0.9048 | 0.8390 | 0.9420 | 0.9730 | 0.9708 | 0.9790
F-score | 0.8805 | 0.8992 | 0.7852 | 0.9401 | 0.9724 | 0.9700 | 0.9775
RGB-D | ACC | 0.3000 | 0.2400 | 0.3300 | 0.4400 | 0.3700 | 0.5100 | 0.5600
Precision | 0.2110 | 0.1600 | 0.2850 | 0.4609 | 0.2887 | 0.5365 | 0.5520
F-score | 0.2204 | 0.1691 | 0.2737 | 0.4181 | 0.3078 | 0.4873 | 0.5278
Fashion-MV | ACC | 0.8490 | 0.8535 | 0.8680 | 0.9650 | 0.8925 | 0.9685 | 0.9770
Precision | 0.8522 | 0.8597 | 0.8680 | 0.9652 | 0.8206 | 0.9637 | 0.9678
F-score | 0.8354 | 0.8384 | 0.8655 | 0.9649 | 0.8290 | 0.9648 | 0.9707
MSRA | ACC | 0.2381 | 0.2429 | 0.3619 | 0.6905 | 0.9048 | 0.9371 | 0.9270
Precision | 0.2053 | 0.2204 | 0.2498 | 0.7129 | 0.9153 | 0.9393 | 0.9317
F-score | 0.2422 | 0.2357 | 0.2773 | 0.6895 | 0.9037 | 0.9391 | 0.9277
Caltech101-20 | ACC | 0.7154 | 0.7154 | 0.7272 | 0.8537 | 0.9248 | 0.9228 | 0.9216
Precision | 0.4527 | 0.6057 | 0.6164 | 0.7183 | 0.8941 | 0.8946 | 0.9068
F-score | 0.3981 | 0.4325 | 0.5247 | 0.6907 | 0.8458 | 0.8421 | 0.8572
Table 3: Classification results of all methods on six datasets. Bold and
underline denote the best and second-best results, respectively.

###  4.1 Experimental Setup

Datasets. To comprehensively evaluate the effectiveness of our MetaViewer, we
conduct six multi-view benchmarks in experiments. All datasets are scaled to
$[0,1]$ and split into training, validation, and test sets in the ratio of
$6:2:2$, as shown in Tab. 1. More details of dataset are in Appendix A.

  * •

BDGP is an image and text dataset, corresponding to $2,500$ drosophila embryo
images in $5$ categories. Each image is described by a $79$-D textual feature
vector and a $1,750$-D visual feature vector [5].

  * •

Handwritten contains $2,000$ handwritten digital images from $0$ to $9$. Two
types of descriptors, i.e., $240$-D pixel average in $2\times 3$ windows and
$216$-D profile correlations, are selected as two views [56].

  * •

RGB-D dataset contains visual and depth images of $300$ distinct objects
across $50$ categories [60, 43]. Two views are obtained by flattening the
$64\times 64\times 3$ color images and $64\times 64$ depth images.

  * •

Fashion-MV is an image dataset that contains $10$ categories with a total of
$30,000$ fashion products. It has three views and each of which consists of
10,000 gray images sampled from the same category [47].

  * •

MSRA [46] consists of $210$ scene recognition images from seven classes with
six views, that is, CENTRIST, CMT, GIST, HOG, LBP, and SIFT.

  * •

Caltech101-20 is a subset of the Caltech101 image set [9], which consists of
$2,386$ images of $20$ subjects. Six features are used, including Gabor,
Wavelet Moments, CENTRIST, HOG, GIST, and LBP.

Compared methods. We compare the performance of MetaViewer with five
representative multi-view learning methods, including two classical methods
(DCCA [1] and DCCAE [44]) and three state-of-the-art methods (MIB [8], MFLVC
[48] and DCP [28]). Among them, DCCA and DCCAE are the deep extensions of
traditional correlation strategies. MIB is a typical generative method with
mutual information constraints. DCP learns unified representation both in
complete and incomplete views. In particular, MFLVC also notices the view-
private redundant information and designs a multi-level features network for
clustering tasks.

Implementation details. For a fair comparison, all methods are trained from
scratch and share the same backbone listed in Appendix B. We concatenate
latent features of all views in compared methods to obtain the unified
representation $H$ with the same dimension $d_{h}={256}$, and verify their
performance in clustering and classification tasks using K-means and a linear
SVM, respectively. For MetaViewer, we train $2,000$ epochs for all benchmarks,
and set the batch size is $32$ for RGBD and MSRA and $256$ for others. The
learning rates in outer- and inner-level are set to $10^{-3}$ and $10^{-2}$,
respectively. All experiments have been verified using the PyTorch library on
a single RTX3090.

###  4.2 Performance on downstream tasks

Clustering results. Tab. 2 lists the results of the clustering task, where the
performance is measured by three standard evaluation metrics, i.e., Accuracy
(ACC), Normalized Mutual Information (NMI), and Adjusted Rand Index (ARI). A
higher value of these metrics indicates a better clustering performance. It
can be observed that (1) our MVer-C variant significantly outperforms other
compared methods on all benchmarks except MSRA; (2) the second-best results
appear between MVer-R and MFLVC, both of which explicitly separates the view-
private information; (3) A larger number of categories and views is the main
reason for the degradation of clustering performance, and our Metaviewer
improves most significantly in such scenario (e.g. Fashion-MV and
Caltech101-20).

Classification results. Tab. 3 lists the results of the classification task,
where three common metrics are used including Accuracy, Precision, and
F-score. A higher value indicates a better classification performance. Similar
to the clustering results, two variants of MetaViewer significantly outperform
the comparison methods. It is worth noting that (1) DCP learns the unified
generic representation and therefore achieves the second-best result instead
of MFLVC. (2) The number of categories is the main factor affecting the
classification performance, and our method obtains the most significant
improvement in the RGB-D dataset with $50$ classes. More results including
incomplete views are deferred to Appendix C.

###  4.3 Comparison with manually designed fusion.

As mentioned in 3.3, MetaViewer is essentially learned to learn an optimal
fusion function that filters the view-private information. To verify this, we
compare it with commonly used fusion strategies [27], including sum, maxima,
concatenation, linear layer and C-Conv. The former three are the specified
fusion rules without trainable parameters and the remaining two are the
trainable fusion layer trained via the specific-to-uniform manner. Tab. 4
lists the clustering results and an additional MSE score on the Handwritten
dataset with the same embedding and reconstruction network (see Tab. 1). We
can observe that (1) trainable fusion layers outperform the hand-designed
rules, and our MetaViewer yields the best performance; (2) the MSE scores
listed in the last column indicate that the quality of the unified
representation cannot be measured and guaranteed only with the reconstruction
constraint, due to the view-private redundant information mixed in view-
specific latent features.

Figure 4: Effect of meta-learner architectures with different depth, width,
and kernel size on classification accuracy.

###  4.4 Ablation Studies

Meta-learner structures. We implement the meta-learner as channel-level
convolution layers in this work. Albeit simple, this layer can be considered
as a universal approximator for almost any continuous function [36], and thus
can fit a wide range of conventional fusion functions. To investigate the
effect of network depth, width, and convolution kernel size on the performance
of the representation, we alternate fix the $32$ kernels and $1\times 3$
kernel size and show the classification results on Handwritten data in Fig. 4.
It is clear that (1) the meta-learner works well with just a shallow structure
as shown in Fig. 4 (a), instead of gradually overfitting to the training data
as the network deepens or widens, (2) our MetaViewer is stable and insensitive
to the hyper-parameters within reasonable ranges.

Meta-split ratios. Fig. 5 (a) shows the impact of meta-split mentioned in 3.2
on the classification performance, where the proportion of support set is set
from $0.1$ to $0.9$ in steps of $0.1$, and the rest is query set. In addition
to the single view, we also compare the sum and concat. fusion as baselines.
MetaViewer consistently surpasses all baselines over the experimental
proportion. In addition, fusion baselines are more dependent on the better-
performing view at lower proportions, instead becoming unstable as the
available query sample decreases.

Inner-level update steps. Another hyper-parameter is the number of iteration
steps in inner-level optimization. More iterations mean a larger gap from the
learned meta representation to the specific view space, i.e., coarser modeling
of view-private information. Fig. 5 (b) shows the classification results with
various steps, where $n$ step means that the inner-level optimization is
updated $n$ times throughout the training. MetaViewer achieves the best
results when using $1$ steps, and remains stable within $15$ steps.

Figure 5: Effect of (a) different meta-division ratios and (b) the number of inner-loop iterations on classification accuracy. Strategies | Rules |  ACC$\uparrow$ |  NMI$\uparrow$ |  ARI$\uparrow$ |  MSE$\downarrow$
---|---|---|---|---|---
Sum | $z^{x}+z^{y}$ | 69.25 | 71.89 | 59.02 | 1.84
Max | $max(z^{x},z^{y})$ | 80.75 | 73.93 | 63.75 | -
Concat. | $cat[z^{x},z^{y}]$ | 78.75 | 72.02 | 61.52 | 1.77
Linear | $l(z^{x},z^{y},\theta_{l})$ | 85.00 | 77.40 | 69.71 | 4.74
C-Conv | $m(z^{x},z^{y},\omega)$ | 69.75 | 65.21 | 51.33 | 2.37
MetaViewer | meta-learning | 86.25 | 78.96 | 72.25 | 2.45
Table 4: Clustering resulting on the Handwritten dataset.

##  5 Conclusion

This work introduced a novel meta-learning perspective for multi-view
learning, and proposed a meta-learner, namely MetaViewer, to derive a high-
quality unified representation for downstream tasks. In contrast to the
prevailing specific-to-uniform pipeline, MetaViewer observes the
reconstruction process from unified representation to specific views and
essentially learns an optimal fusion function that separates and filters out
meaningless view-private information. Extensive experimental results on
clustering and classification tasks demonstrate the performance of the unified
representation we meta-learned.

## References

  * [1] Galen Andrew, Raman Arora, Jeff A. Bilmes, and Karen Livescu.  Deep canonical correlation analysis.  In ICML, volume 28, pages 1247–1255, 2013.
  * [2] Philip Bachman, R. Devon Hjelm, and William Buchwalter.  Learning representations by maximizing mutual information across views.  In NeurIPS, pages 15509–15519, 2019.
  * [3] Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency.  Multimodal machine learning: A survey and taxonomy.  IEEE TPAMI, 41(2):423–443, 2019.
  * [4] Léon Bottou.  Large-scale machine learning with stochastic gradient descent.  In Yves Lechevallier and Gilbert Saporta, editors, COMPSTAT, pages 177–186, 2010.
  * [5] Xiao Cai, Hua Wang, Heng Huang, and Chris H. Q. Ding.  Joint stage recognition and anatomical annotation of _drosophila_ gene expression patterns.  Bioinform., 28(12):16–24, 2012.
  * [6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton.  A simple framework for contrastive learning of visual representations.  In ICML, volume 119, pages 1597–1607, 2020.
  * [7] Yanhua Cheng, Xin Zhao, Rui Cai, Zhiwei Li, Kaiqi Huang, and Yong Rui.  Semi-supervised multimodal deep learning for RGB-D object recognition.  In IJCAI, pages 3345–3351, 2016.
  * [8] Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata.  Learning robust representations via multi-view information bottleneck.  In ICLR, 2020.
  * [9] Li Fei-Fei, Robert Fergus, and Pietro Perona.  Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories.  CVIU, 106(1):59–70, 2007.
  * [10] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman.  Convolutional two-stream network fusion for video action recognition.  In CVPR, pages 1933–1941, 2016.
  * [11] Fangxiang Feng, Xiaojie Wang, and Ruifan Li.  Cross-modal retrieval with correspondence autoencoder.  In ACM MM, pages 7–16, 2014.
  * [12] Yifan Feng, Zizhao Zhang, Xibin Zhao, Rongrong Ji, and Yue Gao.  GVCNN: group-view convolutional neural networks for 3d shape recognition.  In CVPR, pages 264–272, 2018.
  * [13] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In ICML, volume 70, pages 1126–1135, 2017.
  * [14] Andrea Frome, Gregory S. Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc’Aurelio Ranzato, and Tomás Mikolov.  Devise: A deep visual-semantic embedding model.  In NeurIPS, pages 2121–2129, 2013.
  * [15] Kuiliang Gao, Bing Liu, Xuchu Yu, and Anzhu Yu.  Unsupervised meta learning with multiview constraints for hyperspectral image small sample set classification.  IEEE TIP, 31:3449–3462, 2022.
  * [16] Yu Geng, Zongbo Han, Changqing Zhang, and Qinghua Hu.  Uncertainty-aware multi-view representation learning.  In AAAI, pages 7545–7553, 2021.
  * [17] Xavier Glorot, Antoine Bordes, and Yoshua Bengio.  Deep sparse rectifier neural networks.  In AISTATS, volume 15, pages 315–323, 2011.
  * [18] Suriya Gunasekar, Makoto Yamada, Dawei Yin, and Yi Chang.  Consistent collective matrix completion under joint low rank structure.  In AISTATS, volume 38, 2015.
  * [19] Kaveh Hassani and Amir Hosein Khas Ahmadi.  Contrastive multi-view representation learning on graphs.  In ICML, volume 119, pages 4116–4126, 2020.
  * [20] Timothy M. Hospedales, Antreas Antoniou, Paul Micaelli, and Amos J. Storkey.  Meta-learning in neural networks: A survey.  IEEE TPAMI, 44(9):5149–5169, 2022.
  * [21] Junlin Hu, Jiwen Lu, and Yap-Peng Tan.  Sharable and individual multi-view metric learning.  IEEE TPAMI, 40(9):2281–2288, 2018.
  * [22] Mike Huisman, Jan N. van Rijn, and Aske Plaat.  A survey of deep meta-learning.  Artif. Intell. Rev., 54(6):4483–4541, 2021.
  * [23] Xiaodong Jia, Xiao-Yuan Jing, Xiaoke Zhu, Songcan Chen, Bo Du, Ziyun Cai, Zhenyu He, and Dong Yue.  Semi-supervised multi-view deep discriminant representation learning.  IEEE TPAMI, 43(7):2496–2509, 2021.
  * [24] Xiao-Yuan Jing, Fei Wu, Xiwei Dong, Shiguang Shan, and Songcan Chen.  Semi-supervised multi-view correlation feature learning with application to webpage classification.  In AAAI, pages 1374–1381, 2017.
  * [25] Andrej Karpathy and Li Fei-Fei.  Deep visual-semantic alignments for generating image descriptions.  IEEE TPAMI, 39(4):664–676, 2017.
  * [26] Dongge Li, Nevenka Dimitrova, Mingkun Li, and Ishwar K. Sethi.  Multimedia content processing through cross-modal association.  In ACM MM, pages 604–611, 2003.
  * [27] Yingming Li, Ming Yang, and Zhongfei Zhang.  A survey of multi-view representation learning.  IEEE TKDE, 31(10):1863–1883, 2019.
  * [28] Yijie Lin, Yuanbiao Gou, Xiaotian Liu, Jinfeng Bai, Jiancheng Lv, and Xi Peng.  Dual contrastive prediction for incomplete multi-view representation learning.  IEEE TPAMI, 2022.
  * [29] Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun.  Metapruning: Meta learning for automatic neural network channel pruning.  In ICCV, pages 3295–3304, 2019.
  * [30] Shirui Luo, Changqing Zhang, Wei Zhang, and Xiaochun Cao.  Consistent and specific multi-view subspace clustering.  In Sheila A. McIlraith and Kilian Q. Weinberger, editors, AAAI, pages 3730–3737, 2018.
  * [31] Yao Ma, Shilin Zhao, Weixiao Wang, Yaoman Li, and Irwin King.  Multimodality in meta-learning: A comprehensive survey.  KBS, 250:108976, 2022.
  * [32] Yuren Mao, Zekai Wang, Weiwei Liu, Xuemin Lin, and Pengtao Xie.  Metaweighting: Learning to weight tasks in multi-task learning.  In ACL, pages 3436–3448, 2022.
  * [33] Niall McLaughlin, Jesús Martínez del Rincón, and Paul Miller.  Recurrent convolutional network for video-based person re-identification.  In CVPR, pages 1325–1334, 2016.
  * [34] Feiping Nie, Guohao Cai, Jing Li, and Xuelong Li.  Auto-weighted multi-view learning for image clustering and semi-supervised classification.  IEEE TIP, 27(3):1501–1511, 2018.
  * [35] Mathieu Salzmann, Carl Henrik Ek, Raquel Urtasun, and Trevor Darrell.  Factorized orthogonal latent spaces.  In AISTATS, volume 9, pages 701–708, 2010.
  * [36] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng.  Meta-weight-net: Learning an explicit mapping for sample weighting.  In NeurIPS, pages 1917–1928, 2019.
  * [37] Nitish Srivastava and Ruslan Salakhutdinov.  Multimodal learning with deep boltzmann machines.  JMLR, 15(1):2949–2980, 2014.
  * [38] Kai Sun, Jiangshe Zhang, Junmin Liu, Ruixuan Yu, and Zengjie Song.  DRCNN: dynamic routing convolutional neural network for multi-view 3d object recognition.  IEEE TIP, 30:868–877, 2021.
  * [39] Shiliang Sun, Wenbo Dong, and Qiuyang Liu.  Multi-view representation learning with deep gaussian processes.  IEEE TPAMI, 43(12):4453–4468, 2021.
  * [40] Zhongkai Sun, Prathusha Kameswara Sarma, William A. Sethares, and Yingyu Liang.  Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis.  In AAAI, pages 8992–8999, 2020.
  * [41] Hong Tao, Chenping Hou, Feiping Nie, Jubo Zhu, and Dongyun Yi.  Scalable multi-view semi-supervised classification via adaptive regression.  IEEE TIP, 26(9):4283–4296, 2017.
  * [42] Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J. Lim.  Multimodal model-agnostic meta-learning via task-aware modulation.  In NeurIPS, pages 1–12, 2019.
  * [43] Shiye Wang, Changsheng Li, Yanming Li, Ye Yuan, and Guoren Wang.  Self-supervised information bottleneck for deep multi-view subspace clustering.  CoRR, abs/2204.12496, 2022.
  * [44] Weiran Wang, Raman Arora, Karen Livescu, and Jeff A. Bilmes.  On deep multi-view representation learning.  In ICML, volume 37, pages 1083–1092, 2015.
  * [45] Olga Wichrowska, Niru Maheswaranathan, Matthew W. Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein.  Learned optimizers that scale and generalize.  In ICML, volume 70, pages 3751–3760, 2017.
  * [46] Jinglin Xu, Junwei Han, and Feiping Nie.  Discriminatively embedded k-means for multi-view clustering.  In CVPR, pages 5356–5364, 2016.
  * [47] Jie Xu, Yazhou Ren, Huayi Tang, Zhimeng Yang, Lili Pan, Yang Yang, and Xiaorong Pu.  Self-supervised discriminative feature learning for multi-view clustering.  CoRR, abs/2103.15069, 2021.
  * [48] Jie Xu, Huayi Tang, Yazhou Ren, Liang Peng, Xiaofeng Zhu, and Lifang He.  Multi-level feature learning for contrastive multi-view clustering.  In CVPR, pages 16030–16039, 2022.
  * [49] Fei Xue, Xin Wu, Shaojun Cai, and Junqiu Wang.  Learning multi-view camera relocalization with graph neural networks.  In CVPR, pages 11372–11381, 2020.
  * [50] Xiaowei Xue, Feiping Nie, Sen Wang, Xiaojun Chang, Bela Stantic, and Min Yao.  Multi-view correlated feature learning by uncovering shared component.  In AAAI, pages 2810–2816, 2017.
  * [51] Xiaoqiang Yan, Shizhe Hu, Yiqiao Mao, Yangdong Ye, and Hui Yu.  Deep multi-view learning methods: A review.  Neurocomputing, 448:106–129, 2021.
  * [52] En Yu, Zhuoling Li, and Shoudong Han.  Towards discriminative representation: Multi-view trajectory contrastive learning for online multi-object tracking.  In CVPR, pages 8824–8833, 2022.
  * [53] Tan Yu, Jingjing Meng, and Junsong Yuan.  Multi-view harmonized bilinear network for 3d object recognition.  In CVPR, pages 186–194, 2018.
  * [54] Yun-Hao Yuan, Jin Li, Yun Li, Jipeng Qiang, Yi Zhu, Xiaobo Shen, and Jianping Gou.  Learning canonical f-correlation projection for compact multiview representation.  In CVPR, pages 19238–19247, 2022.
  * [55] Changqing Zhang, Huazhu Fu, Qinghua Hu, Xiaochun Cao, Yuan Xie, Dacheng Tao, and Dong Xu.  Generalized latent multi-view subspace clustering.  IEEE TPAMI, 42(1):86–99, 2020.
  * [56] Changqing Zhang, Yeqing Liu, and Huazhu Fu.  Ae2-nets: Autoencoder in autoencoder networks.  In CVPR, pages 2577–2585, 2019.
  * [57] Liang Zhao, Tao Yang, Jie Zhang, Zhikui Chen, Yi Yang, and Z. Jane Wang.  Co-learning non-negative correlated and uncorrelated features for multi-view data.  IEEE TNNLS, 32(4):1486–1496, 2021.
  * [58] Qinghai Zheng, Jihua Zhu, and Zhongyu Li.  Collaborative unsupervised multi-view representation learning.  IEEE TCSVT, 32(7):4202–4210, 2022.
  * [59] Wenqing Zheng, Tianlong Chen, Ting-Kuei Hu, and Zhangyang Wang.  Symbolic learning to optimize: Towards interpretability and scalability.  In ICLR, 2022.
  * [60] Pengfei Zhu, Binyuan Hui, Changqing Zhang, Dawei Du, Longyin Wen, and Qinghua Hu.  Multi-view deep subspace clustering networks.  CoRR, abs/1908.01978, 2019.