# TextMI: Textualize Multimodal Information for Integrating Non-verbal Cues in
Pre-trained Language Models

Md Kamrul Hasan1, *, Md Saiful Islam1, Sangwu Lee1, Wasifur Rahman1,
Iftekhar Naim1, Mohammed Ibrahim Khan1, Ehsan Hoque1
1 University of Rochester, United States
* Corresponding author; Email<EMAIL_ADDRESS>
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

###### Abstract

Pre-trained large language models have recently achieved ground-breaking
performance in a wide variety of language understanding tasks. However, the
same model can not be applied to multimodal behavior understanding tasks
(e.g., video sentiment/humor detection) unless non-verbal features (e.g.,
acoustic and visual) can be integrated with language. Jointly modeling
multiple modalities significantly increases the model complexity, and makes
the training process data-hungry. While an enormous amount of text data is
available via the web, collecting large-scale multimodal behavioral video
datasets is extremely expensive, both in terms of time and money. In this
paper, we investigate whether large language models alone can successfully
incorporate non-verbal information when they are presented in textual form. We
present a way to convert the acoustic and visual information into
corresponding textual descriptions and concatenate them with the spoken text.
We feed this augmented input to a pre-trained BERT model and fine-tune it on
three downstream multimodal tasks: sentiment, humor, and sarcasm detection.
Our approach, TextMI, significantly reduces model complexity, adds
interpretability to the model’s decision, and can be applied for a diverse set
of tasks while achieving superior (multimodal sarcasm detection) or near SOTA
(multimodal sentiment analysis and multimodal humor detection) performance. We
propose TextMI as a general, competitive baseline for multimodal behavioral
analysis tasks, particularly in a low-resource setting.

##  I Introduction

Figure 1: Instead of using complex models that try to fuse multiple
modalities, we rely on a single pre-trained language model (left). To
textualize nonverbal (visual and acoustic) cues, we group unimodal features
that frequently appear together into finite number of clusters, and then
describe the clusters using text (middle). Finally, we extend the utterance
text with the extracted visual-text and acoustic-text and pass it to a
language model (right).

Humans are experts at understanding the nuances of non-verbal communication.
To enable machines to understand those non-verbal signals, we often encode
them in three modalities – text, acoustic, visual – and then fuse them using
neural encoders. Recently, pre-trained large language models like BERT [1]
have become extremely effective in providing highly contextualized
representation of the text modality. Acoustic and visual modalities, on the
other hand, are typically converted into mid to high-level features – such as
pitch, MFCC, facial expression, etc. – and then fed into transformers or LSTM
encoders for fusion [2, 3, 4, 5, 6]. As a result, the encoders for acoustic
and visual modalities are often trained from scratch compared to the pre-
trained text encoders. Therefore, the fusion process is usually dominated by
the highly contextualized text modality, making it difficult to infuse
acoustic and visual information properly. Moreover, multimodal (text,
acoustic, visual) behavioral datasets are typically smaller in size due to the
higher cost of collecting them. The scarcity of large and diverse datasets
makes it more challenging to train parameter-heavy multimodal models.

In contrast, the pre-trained models like BERT [1] can be easily fine-tuned to
achieve state-of-the-art results for many NLP tasks like text-based humor
detection [7] and sentiment analysis [8]. In this paper, we ask – can the
language model understand non-verbal cues presented in text format? If we feed
the language spoken and its associated nonverbal cues in textual format to a
pre-trained language model, can it analyze the multimodal content? How does it
perform in learning human behavior compared to the existing resource-hungry
multimodal models?

In this paper, we propose methods to textualize visual and acoustic cues
present in multimodal (video) behavior understanding tasks: multimodal
sentiment analysis, multimodal humor detection, and multimodal sarcasm
detection. We utilize widely-used tools such as Openface [9] and Opensmile
[10] for extracting visual and acoustic features from videos. We then cluster
these extracted visual and acoustic features into a finite number of groups
using K-means clustering, as shown in Figure 1 (middle). We generate textual
descriptions of these visual and acoustic clusters, and refer to them as
visual-text and acoustic-text respectively. We prepare an extended text input
by combining the text modality with the visual-text and acoustic-text (when
available) as shown in Fig. 1 (right) and feed the extended text into a pre-
trained BERT model.

As multimodal datasets, we use CMU-MOSI [11] and CMU-MOSEI [12] for sentiment
analysis, UR-FUNNY [13] for humor detection, and MUStARD [14] for sarcasm
detection task. TextMI outperforms all state-of-the-art (SOTA) models in the
multimodal sarcasm detection task and achieves near SOTA performance in the
multimodal humor detection task. It also achieves better/near SOTA
performances across all metrics in multimodal sentiment analysis task.

These results demonstrate that our approach can be generalized across many
diverse behavior-analysis tasks. The proposed methodology, while fairly
simple, can serve as a strong baseline for multimodal behavior understanding
tasks, particularly with limited training data.

In summary, the main contributions of this paper are:

  * •

We propose a framework to convert visual and acoustic cues into natural
language text for analyzing multimodal human behavioral tasks.

  * •

We demonstrate that large language models can readily integrate visual and
acoustic information provided in text format and achieve superior (or
competitive) performance than the baseline multimodal models that use
intricate fusion mechanisms.

  * •

TextMI is a simple and general methodology for multimodal behavior analysis
that can act as a strong baseline for a diverse set of tasks. Our approach is
interpretable and particularly valuable for the tasks with limited data.

##  II Background

In this section, we discuss some of the state-of-the-art multimodal models,
their complexity and how the scarcity of large multimodal (video) behavioral
datasets makes the training challenging.

###  II-A Multimodal Models

A lot of research exists on learning joint representation of multimodal data
using LSTM, CNN, and fully-connected neural networks [15, 16, 17, 18, 19, 20,
21, 17, 22, 23]. We will focus Transformer-based models [24] which are the
current state-of-the-art for modeling multimodal – text,acoustic,visual–
modalities.

Tsai et al. [25] trained a model with a set of transformers: each transformer
encoder learns its modality-specific encoding while interacting with the other
encoders to capture the cross-modal interactions. Similarly, HKT [4] has
modality-specific encoders for attending to each modality and a bimodal cross
attention layer to jointly represent pairs of modality groups effectively.
Several frameworks have been proposed where pre-trained BERT is used as
language encoder and LSTM/Transformer based architectures are used to encode
other modalities [2, 3, 5, 6]. These architectures also introduced different
intricate fusion mechanisms to achieve state-of-the-art results. Pham et al.
[26] translates from a source modality to a target modality while maintaining
cycle consistency to ensure maximal retention of information in the joint
representation. Another approach named Multimodal Routing [27] learns to
dynamically adjust weights between input modalities and output vector for each
data sample – providing interpretation about cross-modality interactions for
each example and whole dataset. In the domain of Graph-Neural-Networks, Model
Temporal Attention Graph (MTAG) represents the sequential and temporal
interactions in the multimodal sequence as nodes and edges [28]. These models
have separate neural network components for capturing unimodal and cross-modal
interactions that increase the number of parameters by the factor of the
number of modalities. On the other hand, our approach uses a single pre-
trained language model, and thus significantly reduces the model complexity
and training time.

###  II-B Scarcity of Large Multimodal Behavioral Datasets

Collecting large multimodal datasets for human behavioral tasks such as
analyzing sentiment, job interview performance, disrespectful behavior, humor,
etc. is extremely challenging, time-consuming, and expensive. The ratings for
these tasks are often subtle and subjective and require expert knowledge and
careful considerations. As a result, multimodal behavioral datasets are
typically small and contain from hundreds to a few thousand training examples.

For example, two commonly used multimodal sentiment analysis datasets are CMU-
MOSI [11] and CMU-MOSEI [12], consisting of 2199 and 23500 opinions
respectively. ICT-MMMO dataset [29], consisting of review videos, is also used
for sentiment analysis with only 340 data instances. Similarly, MOUD [30]
dataset contains 400 Spanish videos, each labeled with a positive, negative,
or neutral sentiment. IEMOCAP [31] dataset consists of 302 videos – each
annotated with the presence of 9 emotions as well as valence, arousal, and
dominance. The UR-FUNNY dataset, the largest multimodal dataset for detecting
humorous instances, contains 5k videos of humorous punchlines, with an equal
number of negative samples [13]. Multimodal sarcasm detection dataset [14] has
only 690 video segments collected from several sitcoms. The scarcity of the
large multimodal behavioral datasets makes it very challenging to train
multimodal models with large number of parameters. In contrast, TextMI
requires less parameters and easy to train on smaller datasets.

##  III Non-verbal Text Extraction

In this section, we describe how we extract features from acoustic and visual
modalities and convert them into corresponding text representations – denoted
as acoustic-text and visual-text (Figure 1).

###  III-A Visual-text

OpenFace2 [9] is used to extract facial Action Units (AU) features, based on
the Facial Action Coding System (FACS) [32]. It is widely used in human affect
analysis [13, 3, 33]. Each of the action units represents a specific facial
muscle movement. For example, AU02, AU04, and AU06 represent ‘Outer Brow
Raiser’, ‘Brow Lowerer’, and ‘Cheek Raiser’ respectively. These descriptions
of the action units are readily understandable to humans. However, humans
typically use a combination of these muscle movements simultaneously – each
denoted by an action unit – to exhibit an expression. To mimic this behavior,
we use K-means clustering to group the facial action units that co-occur
often. In this paper, we use the following set of action units: [AU2, AU4,
AU5, AU6, AU7, AU9, AU12, AU15, AU23, AU26, AU45].

First, we extract these eleven AU features from each frame of a video segment.
The timing information of each word is used to slice off the relevant range of
action unit features for that word. Then, for each word, we average out the
sliced visual feature array across the time dimension and get a visual feature
vector (11 dimensions). Extracting word-aligned visual/acoustic vector is a
common practice in multimodal behavior analysis tasks [34, 17, 2, 3]. Once we
have these visual feature vectors for each word across all videos, K-means
clustering is used to group them into distinct sets. We use silhouette score
to determine the optimal number of clusters. By analyzing the word-aligned
visual features that belong to each cluster, we find the dominant (high
intensity) action units in each cluster. Then, we represent each cluster by
the text descriptions of the dominant action units. Table I shows the
clusters, their dominant action units, and the corresponding descriptions of
each action unit. These resulting textual descriptions are used to generate
the visual-text.

Let, there are $n$ words in a video segment $U=[w_{1},w_{2},....,w_{n}]$. For
the $i$th word ($w_{i}$), we can use the corresponding facial unit vector to
extract the relevant cluster id. Thus, we can represent the cluster ids of the
video utterance as: $C_{v}=[c_{1},c_{2},....,c_{n}]$. Each cluster-id
represents a set of dominant AUs (e.g. table I). We sort all the AUs based on
how many times they appear in the video utterance; the most commonly occurring
ones are put at the beginning. For example, in Figure 1, the visual cluster id
4 has the highest frequency. This cluster id is represented by the dominant
action units brow raiser and jaw drop. That is why these visual words appear
at the beginning of the facial expression description. We concatenate all the
visual words extracted from the sorted (based on frequency) AUs to generate
the visual-text. We use k-means clustering since it gives a hard label to each
of the clusters. Other techniques like the gaussian mixture model give the
probabilities of a data point belonging to each of the K clusters – making it
difficult to convert these probabilities into words.

Cluster | AUs | Textual Description (visual-text)
---|---|---
0 | None | neutral face
1 | 15,25,20,26, | lip corner depressor, lip stretcher, jaw drop
| 23,4,7 | lip tightener, brow lowerer, lid tightener
2 | 25,26 | jaw drop
3 | 7,6,4,25,26 | lid tightener, cheek raiser, brow lowerer
4 | 26,25,4 | jaw drop, brow lowerer, lip tightener
5 | 4,25 | brow lowerer, lips part
6 | 12,6,25,7 | lip corner puller, cheek raiser,
|  | lips part, lid tightener

TABLE I: Visual-text for the clusters of visual features (from UR-FUNNY)

###  III-B Acoustic-text

Similar to extracting visual-text, we extracted the following interpretable
acoustic features using Opensmile: pitch, loudness, jitter, and shimmer.
Similar to the visual features, we extract word-aligned acoustic features of
the whole dataset and apply K-Means clustering to them. Each cluster is
assigned descriptions based on the intensity of the features present within
it. A normal distribution is fitted to find the threshold of low, normal, and
high intensity. Table II shows an example of the UR-FUNNY (multimodal humor
detection) dataset. We denote these resulting descriptions as acoustic-text.

For a video segment, first, we extract the cluster ids that are associated
with the word-aligned acoustic vectors. Let, there are $n$ words in a video
utterance $U=[w_{1},w_{2},....,w_{n}]$. Similar to the visual words, we
extract the corresponding acoustic cluster ids
$C_{a}=[c_{1},c_{2},....,c_{n}]$. Then, we create a textual description of the
acoustic features by following the same methodology in III-A: replace each
cluster-id with the underlying set of features (Table II), sort the features
by placing the most frequently appearing ones at the beginning, remove
repeated features and concatenate all the texts.

Cluster | Textual Description (acoustic-text)
---|---
0 | high shimmer, high jitter, normal pitch, normal loudness
1 | normal voice (every feature is in normal range)
2 | high loudness, high pitch, normal jitter, normal shimmer
3 | low pitch, low shimmer, normal loudness, normal jitter

TABLE II: Acoustic-text extracted from UR-FUNNY

###  III-C Combining Text, Acoustic-text and Visual-text

We append the acoustic-text and visual-text at the end of the text utterances
separated by the separator token. As we experiment with the BERT language
model, we represent the multimodal text as: [CLS] utterance text [SEP] Facial
expressions shown: visual-text and acoustic expressions shown: acoustic-text
[SEP] (Figure 1).

##  IV Experiments

In this section, we discuss the datasets we use, the baseline models we
compare with and the hyper parameter settings we experiment with.

###  IV-A Datasets

CMU-MOSI & CMU-MOSEI: Both the CMU-MOSI [11] and the CMU-MOSEI [12] are widely
used benchmark datasets for evaluating a model’s performance in predicting
multimodal sentiment intensity. The CMU-MOSI is composed of 2199 video
utterances segmented from 93 movie review videos. Each video utterance is
manually annotated with a real number score ranging from -3 (most negative) to
+3 (most positive) sentiment. The CMU-MOSEI dataset is an extension of the
CMU-MOSI, but it increased the size of the dataset to 23,454 video utterances.

We use five different metrics following the previous works to evaluate the
performance: mean absolute error (MAE), Pearson correlation, seven class
accuracy (Acc-7), binary accuracy (Acc-2), and F1 score. Both Acc-2 and F1
score are computed for positive/negative (excluding zero).

UR-FUNNY: The UR-FUNNY [13] is a multimodal dataset of humor detection. It
contains 10k video segments sampled from TED talks where punchline sentence is
followed by context sentences. Each video segment is annotated with binary
labels indicating if the punchline is humorous or not.

MUStARD: Multimodal Sarcasm Detection Dataset [14] is compiled from popular TV
shows like Friends, The Big Bang Theory, The Golden Girls, and Sarcasmaholics.
690 video segments are manually annotated with binary sarcastic/non-sarcastic
labels. Each video segment has a target punchline sentence and the associated
historical dialogues as context. Binary accuracy is used as the performance
metric for both UR-FUNNY and MUStARD since both datasets have balanced test
sets.

###  IV-B Baseline Models

Numerous methods have been proposed to learn the multimodal representation of
text, acoustic and visual. We compare TextMI with the most recent and
competitive baselines as mentioned below.

TFN [33] Tensor fusion network learns unimodal tensors and fuses them by three
fold Cartesian product.

LMF [22] creates multimodal fusion from the modality-specific low-rank factors
by decomposing high dimensional tensors into many low dimensional factors.

MFM [17] is a generative-discriminative model that factorizes representations
into multimodal discriminative factors and modality-specific generative
factors.

ICCN[35] learns the correlations between all three modalities of text,
acoustic and visual via deep canonical correlation analysis.

MulT [17] uses a set of transformer encoders to model inter-modal & intra-
modal interactions and combines their output in a late fusion manner.

MISA [2] projects all the video utterances into three modality-specific and
one modality invariant spaces and then aggregates all those projections.

MAG-BERT [36] introduced Multimodal Adaption Gate (MAG) to fuse acoustic and
visual information in pretrained language transformers. During fine tuning,
the MAG shifts the internal representations of BERT in the presence of the
visual and acoustic modalities.

BBFN  [3] is an end-to-end network that performs fusion (relevance increment)
and separation (difference increment) on pairwise modality representations.

Self-MM [5] design a unimodal label generation strategy based on the self-
supervised method that helps to learn modality specific representation.

MMIM  [6] hierarchically maximizes the Mutual Information (MI) in multimodal
fusion. It applies MI lower bounds for the unimodal inputs and the fusion
stage.

HKT [4] models multimodal humorous punchline using a set of transformer
encoders and bi-modal cross attention layer. It also incorporates some humor-
centric features extracted from external knowledge.

State of the Art: MMIM is the state of the art model for the multimodal
sentiment analysis task (in both CMU-MOSI and CMU-MOSEI datasets). Only MISA
and HKT have experimented with the task of multimodal humor detection (UR-
FUNNY) and multimodal sarcasm detection (MuSTARD) where HKT has achieved SOTA
performance.

###  IV-C Experimental Design

Adam optimizer, linear scheduler with warmup and reduce learning rate at
plateu scheduler are used to train the BERT language model. The search space
of the learning rates is {1e-05,3e-05,5e-05,1e-06}. MSE loss is used to train
models on CMU-MOSI and CMU-MOSEI as the sentiment intensity label is a real
number between -3 to +3. Binary cross-entropy is used for other datasets.
Dropout $[0.05-0.30]$ is used to regularize the model. All the experiments are
run on K-80 & A-100 GPUs.

Models |  | CMU-MOSI |  |  |  |  | CMU-MOSEI |  |  |
---|---|---|---|---|---|---|---|---|---|---
| MAE | Corr | Acc-7 | Acc-2 | F1 | MAE | Corr | Acc-7 | Acc-2 | F1
TFN ⋄ | 0.901 | 0.698 | 34.9 | 80.8 | 80.7 | 0.593 | 0.700 | 50.2 | 82.5 | 82.1
LMF ⋄ | 0.917 | 0.695 | 33.2 | 82.5 | 82.4 | 0.623 | 0.677 | 48.0 | 82.0 | 82.1
MFM ⋄ | 0.877 | 0.706 | 35.4 | 81.7 | 81.6 | 0.568 | 0.717 | 51.3 | 84.4 | 84.3
ICCN ⋄ | 0.860 | 0.710 | 39.0 | 83.0 | 83.0 | 0.565 | 0.713 | 51.6 | 84.2 | 84.2
MulT ${}^{\ast}\oplus$ | 0.832 | 0.745 | 40.1 | 83.3 | 82.9 | 0.570 | 0.758 | 51.1 | 84.5 | 84.5
MISA ⋄ | 0.783 | 0.761 | 42.3 | 83.4 | 83.6 | 0.555 | 0.756 | 52.2 | 85.5 | 85.3
MAG-BERT ${}^{\ast}\ddagger$ | 0.727 | 0.781 | 43.62 | 84.43 | 84.61 | 0.543 | 0.755 | 52.67 | 84.82 | 84.71
BBFN ⊕ | 0.776 | 0.755 | 45.0 | 84.3 | 84.3 | 0.529 | 0.767 | 54.8 | 86.2 | 86.1
Self-MM ${}^{\ast}\ddagger$ | 0.712 | 0.795 | 45.79 | 84.77 | 84.91 | 0.529 | 0.767 | 53.46 | 84.96 | 84.93
MMIM (SOTA)‡ | 0.700 | 0.800 | 46.65 | 86.06 | 85.98 | 0.526 | 0.772 | 54.24 | 85.97 | 85.94
TextMI | 0.738 | 0.781 | 44.75 | 85.21 | 85.05 | 0.531 | 0.774 | 53.21 | 86.45 | 86.38

TABLE III: Results of multimodal sentiment analysis task on CMU-MOSI and CMU-
MOSEI datasets. The baseline results are taken from Hajarika et al.⋄ [2], Han
et al.⊕ [3], and Han et al. ‡[6]. Positive and negative samples (excluding
zero) are used for calculating the Acc-2 and F1 score. MAE denotes Mean-
absolute Error (lower is better), and Corr is Pearson Correlation (higher is
better). The best performances are presented in bold and the closest
competitors are underlined. Some results (∗) were reproduced from the open
source code and hyper parameter settings presented in the original paper.

##  V Results

In this section, we present the performance of TextMI compared to the baseline
multimodal models. We also report an ablation study to show the importance of
having acoustic-text and visual-text alongside the main text.

Models | UR-FUNNY | MUStARD
---|---|---
| Acc | Acc
MISA § | 70.61 | 66.18
HKT § | 77.36 | 79.41
TextMI | 74.65 | 82.35

TABLE IV: Results of multimodal humor (UR-FUNNY) and multimodal sarcasm
(MUStARD) detection tasks. Since the datasets are balanced, accuracy for
binary classification is reported, with best results highlighted in bold. The
baseline results are taken from §[4].

###  V-A Multimodal Sentiment Analysis

The results of multimodal sentiment analysis tasks are presented in Table III.
TextMI achieves superior performance in the CMU-MOSEI dataset in terms of
binary accuracy (0.48% increase), F1 score (0.44% increase), and pearson
correlation (0.25% relative improvement). Significance tests are run between
TextMI and MMIM models for these metrics. We have run both models configured
with the hyperparameters of best performances and changed the random seed only
for different runs. Significant differences ($p<0.05$) are observed for these
three metrics between TextMI and MMIM. For the other metrics, it is very close
competitor of the SOTA model. TextMI also attain very competitive result in
CMU-MOSI dataset across _all_ metrics. It achieves the second best performance
in terms of binary accuracy and F1 score. These datasets are well studied for
analyzing the performances of the multimodal models. All these baseline models
use intricate fusion mechanism to summarize multimodal information. In
contrast, our simple approach based on only pre-trained language encoder has
achieved superior/near-SOTA performances. These results corroborate our
hypothesis that the language model has the capability of understanding non-
verbal cues presented in textual form.

Figure 2: A multimodal sentiment analysis example to illustrate how the model
put importance on text, acoustic and visual words. (a) word importance’s are
highlighted by color. (b) shows how the visual-text and acoustic-text are
extracted. [CLS] and [SEP] are special tokens of BERT. Figure 3: Example from
the CMU-MOSI dataset. The ground truth sentiment labels are between strongly
negative (-3) and strongly positive (+3). For each example, we show the Ground
Truth and prediction output of both the text+acoustic-text+visual-text and
text only. Integrated gradients [37] method is used to decipher how each input
token (across modalities) contributed to the model’s final decision. [CLS] and
[SEP] are special tokens of BERT.

###  V-B Multimodal Humor and Sarcasm Detection

The results of multimodal humor (UR-FUNNY) and sarcasm detection (MUStARD)
tasks are presented in Table IV. Since both datasets have a balanced test set,
binary accuracy is reported as the performance metric. TextMI achieves
superior performance (2.94% increase) in the smaller MUStARD dataset (690
utterances) compared to the HKT. In comparison, HKT is superior in the
relatively larger UR-FUNNY dataset (10K utterances). The HKT model
incorporates external, task-specific features inspired by the theories of
humor, and the availability of a large dataset helps train a complex model.
However, the performance of TextMI that does not include any external task-
specific feature is not far behind (2.71% less accurate than HKT). This brings
about an interesting perspective – while generalizability of models is
strongly desired, task-specific knowledge can be valuable and researchers
might benefit from blending both general and task-specific knowledge into
their models.

###  V-C Interpretation of Model Output

The results discussed above indicate that incorporating non-verbal text into a
language model has the capability of generalizing across various types of
multimodal tasks. In addition, converting the facial and acoustic expressions
into textual form makes it easy to interpret. Figure 2 shows an example of
multimodal sentiment analysis where input tokens are highlighted with colors
based on the TextMI model’s importance. Integrated gradients [37] method is
used to decipher how each input token (across modalities) contributed to the
model’s final decision. We can see that the nonverbal-text such as high
loudness plays an important role to identify the negative sentiment in this
utterance. Since one of the major objectives of affective computing is
understanding human emotion, such interpretability is highly valuable to the
community.

Datasets | T | T+V | T+A | T+A+V
---|---|---|---|---
CMU-MOSI | 83.99 | 84.76 | 84.45 | 85.21
CMU-MOSEI | 85.76 | 86.15 | 86.20 | 86.45
MUStARD | 79.41 | 79.41 | 80.88 | 82.35
UR-FUNNY | 74.04 | 74.65 | 74.45 | 74.65

TABLE V: Ablation study to show the importance of visual-text and acoustic-
text. Binary accuracy is reported here as the performance metric. T= text, A=
acoustic-text & V= visual-text.

###  V-D Role of Non-verbal Text

To understand the roles of nonverbal-text quantitatively, we fine-tune the
BERT encoder with text-only, text+acoustic-text and text+ visual-text
information separately. The results are presented in Table V. Adding acoustic-
text and visual-text improves the accuracy significantly, especially in the
smaller datasets of CMU-MOSI and MUStARD. A qualitative analysis is presented
in Figure 3. In the first three examples, TextMI correctly predicted not only
the polarity but also adjusted the sentiment intensity more accurately
compared to the BERT model trained with text-only information. Additional
information is present in visual-text and acoustic-text which text-only
encoder could not utilize. The cross-attentions among the words of text
utterance and nonverbal-text generate better scores. We also show an example
where TextMI can fail (example 4). These examples and results demonstrate that
the language model can integrate non-verbal cues (in text form) for affective
computing tasks.

##  VI Discussion and Future Work

Generalizability: TextMI can be generalized across diverse multimodal tasks as
the input is presented in textual format. Moreover, the use of pre-trained
language models makes it easier to fine-tune on downstream tasks. We
experimented with three multimodal behavior understanding tasks with four
publicly available datasets. The superior results across these diverse tasks
indicate the generalizability of TextMI. One limitation of our approach is
that we depend on existing tools such as Openface and Opensmile. They may
limit TextMI’s performance since the error of these tools will propagate to
the language model. Besides, if these tools are trained from biased data, it
can hurt the fairness of our model as well. However, all the baseline
multimodal models also have the same limitations as they used similar tools
and features [33, 12, 17, 13, 2, 3, 6]. Building an end-to-end model trained
from representative and unbiased data that can textualize the acoustic and
visual features, and make inferences using a language model is a direction we
plan to explore.

Interpretability vs. Performance: In a wide array of human behavior
understanding tasks (e.g. identifying hateful speech video in social media),
it is of utmost importance to identify the key factors behind the model’s
decision. Since TextMI describes acoustic and visual information in textual
format, it is much easier to interpret and visualize. Figure 2 and Figure 3
illustrate how easy it is for humans to interpret the key acoustic and visual
nuances when they are presented in textual format. However, as we concentrate
on a set of interpretable features, our approach can result in loss of
information and performance gained from incorporating complex features.

Typically, visual/acoustic information is modeled from scratch using low to
mid-level features [2, 3, 4] through complex fusion processes. A complex
multimodal model trained on a lot of data may be able to outperform our
approach by sacrificing interpretability. However, our approach can be a very
useful baseline for assessing models deployed in resource-constrained
applications that must be interpretable.

Dataset Size vs. Model’s Complexity: Unlike the existing multimodal models,
TextMI uses only pre-trained language encoders, significantly reducing the
model complexity and the number of parameters that are trained from scratch.
Pre-trained large language models are generally sample efficient, and can
learn well from fewer examples to solve downstream tasks. As a result, our
model can be particularly useful for multimodal tasks with limited training
data (CMU-MOSI & MUStARD) – making our approach very suitable for applying in
new, exciting domains. A computationally efficient benchmark would make
research of exciting problems more accessible since not all researchers have
access to huge computing facilities.

##  VII Conclusion

In this paper, we show that large pre-trained language models can efficiently
integrate non-verbal cues if they are described in textual form. Our approach
achieves superior performance in multimodal sarcasm detection, and
demonstrates better/near-SOTA performances in multimodal sentiment analysis
and multimodal humor detection tasks, compared to the established baseline
multimodal models that use intricate fusion mechanisms. An ablation study also
indicates that the pre-trained language model performs better with acoustic
and visual information than textual information only. As our approach reduces
the model’s complexity and improves interpretability, it can be very useful
for the scenarios where a large dataset is scarce or interpretability is
requisite. Though a multimodal model trained on a large dataset might provide
better performance, our approach can still serve as a very strong and simpler
baseline for future studies on multimodal behavioral tasks.

## References

  * [1] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [2] D. Hazarika, R. Zimmermann, and S. Poria, “Misa: Modality-invariant and-specific representations for multimodal sentiment analysis,” in _Proceedings of the 28th ACM International Conference on Multimedia_ , 2020, pp. 1122–1131.
  * [3] W. Han, H. Chen, A. Gelbukh, A. Zadeh, L.-p. Morency, and S. Poria, “Bi-bimodal modality fusion for correlation-controlled multimodal sentiment analysis,” in _Proceedings of the 2021 International Conference on Multimodal Interaction_ , ser. ICMI ’21. New York, NY, USA: Association for Computing Machinery, 2021, p. 6–15. [Online]. Available: https://doi.org/10.1145/3462244.3479919
  * [4] M. K. Hasan, S. Lee, W. Rahman, A. Zadeh, R. Mihalcea, L.-P. Morency, and E. Hoque, “Humor knowledge enriched transformer for understanding multimodal humor,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 14, 2021, pp. 12 972–12 980.
  * [5] W. Yu, H. Xu, Z. Yuan, and J. Wu, “Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 12, pp. 10 790–10 797, May 2021. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/17289
  * [6] W. Han, H. Chen, and S. Poria, “Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis,” _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2021.
  * [7] J. Mao and W. Liu, “A bert-based approach for automatic humor detection and scoring.” in _IberLEF@ SEPLN_ , 2019, pp. 197–202.
  * [8] C. Sun, L. Huang, and X. Qiu, “Utilizing BERT for aspect-based sentiment analysis via constructing auxiliary sentence,” in _Proceedings of the 2019 Conference of the NAACL: Human Language Technologies_. Association for Computational Linguistics, 2019, pp. 380–385.
  * [9] T. Baltrusaitis, A. Zadeh, Y. C. Lim, and L.-P. Morency, “Openface 2.0: Facial behavior analysis toolkit,” in _2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018)_. IEEE, 2018, pp. 59–66.
  * [10] F. Eyben, M. Wöllmer, and B. Schuller, “Opensmile: the munich versatile and fast open-source audio feature extractor,” in _Proceedings of the 18th ACM international conference on Multimedia_ , 2010, pp. 1459–1462.
  * [11] A. Zadeh, R. Zellers, E. Pincus, and L.-P. Morency, “Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages,” _IEEE Intelligent Systems_ , vol. 31, no. 6, pp. 82–88, 2016.
  * [12] A. B. Zadeh, P. P. Liang, S. Poria, E. Cambria, and L.-P. Morency, “Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , vol. 1, 2018, pp. 2236–2246.
  * [13] M. K. Hasan, W. Rahman, A. Bagher Zadeh, J. Zhong, M. I. Tanveer, L.-P. Morency, and M. E. Hoque, “UR-FUNNY: A multimodal language dataset for understanding humor,” in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 2046–2056. [Online]. Available: https://www.aclweb.org/anthology/D19-1211
  * [14] S. Castro, D. Hazarika, V. Pérez-Rosas, R. Zimmermann, R. Mihalcea, and S. Poria, “Towards multimodal sarcasm detection (an _obviously_ perfect paper),” _arXiv preprint arXiv:1906.01815_ , 2019.
  * [15] A. Zadeh, P. P. Liang, N. Mazumder, S. Poria, E. Cambria, and L.-P. Morency, “Memory fusion network for multi-view sequential learning,” in _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018.
  * [16] Y. Wang, Y. Shen, Z. Liu, P. P. Liang, A. Zadeh, and L.-P. Morency, “Words can shift: Dynamically adjusting word representations using nonverbal behaviors,” _arXiv preprint arXiv:1811.09362_ , 2019.
  * [17] Y.-H. H. Tsai, P. P. Liang, A. Zadeh, L.-P. Morency, and R. Salakhutdinov, “Learning factorized multimodal representations,” _In International Conference on Representation Learning._ , 2019.
  * [18] H. Pham, P. P. Liang, T. Manzini, L.-P. Morency, and B. Poczos, “Found in translation: Learning robust joint representations by cyclic translations between modalities,” _arXiv preprint arXiv:1812.07809_ , 2019.
  * [19] D. Hazarika, S. Poria, A. Zadeh, E. Cambria, L.-P. Morency, and R. Zimmermann, “Conversational memory network for emotion recognition in dyadic dialogue videos,” in _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , vol. 1, 2018, pp. 2122–2132.
  * [20] S. Poria, E. Cambria, D. Hazarika, N. Mazumder, A. Zadeh, and L.-P. Morency, “Multi-level multiple attentions for contextual multimodal sentiment analysis,” in _2017 IEEE International Conference on Data Mining (ICDM)_. IEEE, 2017, pp. 1033–1038.
  * [21] P. P. Liang, Z. Liu, A. Zadeh, and L.-P. Morency, “Multimodal language analysis with recurrent multistage fusion,” _arXiv preprint arXiv:1808.03920_ , 2018.
  * [22] Z. Liu, Y. Shen, V. B. Lakshminarasimhan, P. P. Liang, A. Zadeh, and L.-P. Morency, “Efficient low-rank multimodal fusion with modality-specific factors,” _In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2247–2256. https://www.aclweb.org/anthology/P18-1209.pdf_ , 2018.
  * [23] E. J. Barezi, P. Momeni, P. Fung _et al._ , “Modality-based factorization for multimodal fusion,” _arXiv preprint arXiv:1811.12624_ , 2018.
  * [24] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [25] Y.-H. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency, and R. Salakhutdinov, “Multimodal transformer for unaligned multimodal language sequences,” _arXiv preprint arXiv:1906.00295_ , 2019.
  * [26] H. Pham, P. P. Liang, T. Manzini, L.-P. Morency, and B. Póczos, “Found in translation: Learning robust joint representations by cyclic translations between modalities,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, no. 01, 2019, pp. 6892–6899.
  * [27] Y.-H. H. Tsai, M. Q. Ma, M. Yang, R. Salakhutdinov, and L.-P. Morency, “Multimodal routing: Improving local and global interpretability of multimodal language analysis,” in _Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing_ , vol. 2020. NIH Public Access, 2020, p. 1823.
  * [28] J. Yang, Y. Wang, R. Yi, Y. Zhu, A. Rehman, A. Zadeh, S. Poria, and L.-P. Morency, “Mtag: Modal-temporal attention graph for unaligned human multimodal language sequences,” in _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2021, pp. 1009–1021.
  * [29] M. Wöllmer, F. Weninger, T. Knaup, B. Schuller, C. Sun, K. Sagae, and L.-P. Morency, “Youtube movie reviews: Sentiment analysis in an audio-visual context,” _IEEE Intelligent Systems_ , vol. 28, no. 3, pp. 46–53, 2013.
  * [30] V. Pérez-Rosas, R. Mihalcea, and L.-P. Morency, “Utterance-level multimodal sentiment analysis,” in _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2013, pp. 973–982.
  * [31] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,” _Language resources and evaluation_ , vol. 42, no. 4, pp. 335–359, 2008.
  * [32] R. Ekman, _What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS)_. Oxford University Press, USA, 1997.
  * [33] A. Zadeh, M. Chen, S. Poria, E. Cambria, and L.-P. Morency, “Tensor fusion network for multimodal sentiment analysis,” _In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 1103–1114. https://www.aclweb.org/anthology/D17-1115.pdf_ , 2017.
  * [34] M. Chen, S. Wang, P. P. Liang, T. Baltrušaitis, A. Zadeh, and L.-P. Morency, “Multimodal sentiment analysis with word-level fusion and reinforcement learning,” in _Proceedings of the 19th ACM International Conference on Multimodal Interaction_ , 2017, pp. 163–171.
  * [35] Z. Sun, P. Sarma, W. Sethares, and Y. Liang, “Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 05, 2020, pp. 8992–8999.
  * [36] W. Rahman, M. K. Hasan, S. Lee, A. B. Zadeh, C. Mao, L.-P. Morency, and E. Hoque, “Integrating multimodal information in large pretrained transformers,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 2020, pp. 2359–2369.
  * [37] M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic attribution for deep networks,” in _Proceedings of the 34th International Conference on Machine Learning-Volume 70_. JMLR. org, 2017, pp. 3319–3328.


#  Coactions on $C^{*}$-algebras and universal properties

Erik Bédos  Department of Mathematics, University of Oslo, PB 1053 Blindern,
0316 Oslo, Norway<EMAIL_ADDRESS>,  S. Kaliszewski  School of Mathematical
and Statistical Sciences, Arizona State University, Tempe, AZ 85287
<EMAIL_ADDRESS>,  John Quigg  School of Mathematical and Statistical
Sciences, Arizona State University, Tempe, AZ 85287<EMAIL_ADDRESS>and
Jonathan Turk  School of Mathematical and Statistical Sciences, Arizona State
University, Tempe, AZ 85287<EMAIL_ADDRESS>
(Date: August 15, 2023)

###### Abstract.

It is well-known that the maximalization of a coaction of a locally compact
group on a C*-algebra enjoys a universal property. We show how this important
property can be deduced from a categorical framework by exploiting certain
properties of the maximalization functor for coactions. We also provide a dual
proof for the universal property of normalization of coactions.

###### Key words and phrases:

Action, coaction, maximalization, universal property.

###### 2000 Mathematics Subject Classification:

46L05, 46L55

The first author is grateful to the Trond Mohn Foundation for financial
support through the project “Pure Mathematics in Norway” making it possible
for him to make a stay at the Arizona State University (Tempe) in April 2022.

##  1\. Introduction

Crossed-product duality for $C^{*}$-algebraic dynamical systems requires
coactions as well as actions, and coactions of a locally compact group can
come in various flavors when the group is non-amenable. The two most
fundamental flavors are the extremes: at the bottom are the _normal coactions_
, and at the top are the _maximal coactions_. When the duality theory is cast
in categorical terms, various functors appear, in addition to the ones coming
from the obvious crossed-product constructions. One of these is the
_maximalization functor_ [10], which goes back to the maximalization process
introduced and studied by Fischer in [5], as a follow-up of [2].

This paper originated when, a few years ago, some of us got puzzled by the
final one-line argument in the proof of one of Fischer’s foundational
theorems, stating that a maximalization of a coaction enjoys a certain
universal property. Having a look at our alternative proof of this result in
[1] didn’t give us any relief as we soon realized that our proof also ought to
be supplied with some additional explanation. Happily, we have very recently
found a new approach to fill this perceived gap, which we present in the
current paper. We formulated the technical result and its proof in categorical
terms, since we believe that this is the clearest way to see the “foundational
bedrock” of the theorem.

More specifically, Fischer’s theorem asserts that a certain diagram has a
unique completion. Fischer’s proof culminates in a much larger diagram, that
includes the diagram of interest, and ends by showing that this larger diagram
has a unique completion, and asserting that this diagram implies the result
(although even this assertion is presented in a slightly cryptic form). We
have not communicated with Fischer concerning this, and of course we cannot
conclude that Fischer “forgot” to relate the larger diagram to the smaller,
important, one. However, due to the extensive influence that this universal
property has had on the subsequent development of the theory of cross-product
duality, we feel compelled to make all this as clear as possible.

We begin in Section 2 with a lightning review of the relevant facts concerning
actions and coactions, with (almost) no proofs, our main aim being to
summarize the functorial approach to the maximalization process for coactions
described in [10]. But along the way we also introduce various categories and
functors that we will need; this gives the review a somewhat nonstandard
flavor. The appendix of [3] has most of the details regarding actions and
normal coactions (for maximal coactions see [2]), but (perhaps surprisingly)
with not such a sharp focus on category theory. Our presentation relies
otherwise on [13, 2, 9, 10]. We must also mention that Fischer’s article [5]
is in some sense our main source.

In Section 3, we briefly sketch Fischer’s original proof of his universal
property of maximalization, in order to pin-point where we feel this proof
suffers from a lack of explanation. We also mention that our slightly
different proof appearing in [1] has a similar gap.

In Section 4, we give an abstract universal property (Proposition 4.1) in the
setting of any category together with a functor and a natural transformation
satisfying a quite simple axiom. Then we indicate how this abstract result can
be used to quickly prove Fischer’s universal property for maximalization.

In Section 5, we give an abstract result that is dual to Proposition 4.1, and
show in Theorem 5.2 how the universal property of normalization of coactions
follows.

In Appendix A, we fill a technical gap in the literature, concerning
“destabilization”. If one allows “stabilization” of a $C^{*}$-algebra to mean
tensoring with the compact operators $\mathcal{K}$ on an arbitrary Hilbert
space, in order to recognize an algebra of the form $A\otimes\mathcal{K}$, we
need a completely general result, Proposition A.2. Various authors have
recorded a proof in the separable case (see [9, Proposition 3.4], for
example), but we could not find a result in the literature at the level of
generality we need. Fortunately, the existing argument can be routinely
modified.

We thank the anonymous referees for comments that significantly improved our
paper.

##  2\. The categories and functors

Throughout, $G$ is an arbitrary locally compact group (with a fixed left Haar
measure), and $A,B,C,\dots$ are $C^{*}$-algebras. We use $\otimes$ to denote
the minimal tensor product of $C^{*}$-algebras, and $M(A)$ to denote the
multiplier algebra of $A$.

There is a category $\mathbf{C}^{*}$ of $C^{*}$-algebras, in which the
morphisms $\phi\colon A\to B$ are nondegenerate homomorphisms $\phi\colon A\to
M(B)$. Morphisms extend uniquely to multiplier algebras of their domain, and
we frequently keep the same notation for the extension. A morphism $u\colon
C^{*}(G)\to B$ is the integrated form of a strictly continuous unitary
homomorphism $G\to M(B)$, and we use the same notation for both. This is the
basis for all our categories: $C^{*}$-algebras decorated with extra structure,
which the morphisms must preserve.

### Actions

There is a category $\mathbf{Ac}$ of actions $(A,\alpha)$ of $G$, in which
morphisms are $\mathbf{C}^{*}$-morphisms that are equivariant for the actions.
A morphism $u\colon C^{*}(G)\to B$ in $\mathbf{C}^{*}$ gives rise to a
_unitary action_ $\operatorname{Ad}u$, and if
$\pi\colon(A,\alpha)\to(B,\operatorname{Ad}u)$ is a morphism in $\mathbf{Ac}$,
then $(B,\pi,u)$ is a _covariant representation_ of $(A,\alpha)$. A _( full)
crossed product_ of $(A,\alpha)$ is a universal covariant representation
$(A\rtimes_{\alpha}G,i_{A},i_{G})$, i.e., for every covariant representation
$(B,\pi,u)$ of $(A,\alpha)$ there is a unique morphism $\pi\times u\colon
A\rtimes_{\alpha}G\to B$ in $\mathbf{C}^{*}$ such that

$(\pi\times u)\circ i_{A}=\pi\quad\text{and }(\pi\times u)\circ i_{G}=u.$

By abstract nonsense, any two crossed products of $(A,\alpha)$ are unique up
to unique isomorphism. We (imagine that we) pick one and call it “the” crossed
product.

Throughout this paper111Except in Proposition A.2, where
$\mathcal{K}=\mathcal{K}(H)$ for an arbitrary Hilbert space $H$., we write
$\mathcal{K}=\mathcal{K}(L^{2}(G))$ for the $C^{*}$-algebra of compact
operators on $L^{2}(G)$ (defined with respect to the chosen left Haar
measure), $M:C_{0}(G)\to\mathcal{K}$ for the nondegenerate representation of
$C_{0}(G)$ by multiplication operators, $\rho$ and $\lambda$ for the right and
left regular representations of $G$ on $L^{2}(G)$, respectively, and rt and lt
for the actions of $G$ on $C_{0}(G)$ by right and left translation,
respectively. We use without comment the Stone-von Neumann Theorem:
$(\mathcal{K},M,\rho)$ is a crossed product of the action
$(C_{0}(G),\textup{rt})$.

### Coactions

The theory of coactions is dual to that of actions, and when $G$ is abelian
coactions correspond via the Fourier transform to actions of the dual group
$\widehat{G}$. Technically, a (full) _coaction_ of $G$ on $A$ is a pair
$(A,\delta)$, where $\delta\colon A\to A\otimes C^{*}(G)$ is a morphism in
$\mathbf{C}^{*}$, i.e., $\delta$ is a nondegenerate homomorphism into
$M(A\otimes C^{*}(G))$, such that

$\operatorname{\overline{span}}\\{\delta(A)(1\otimes C^{*}(G))\\}=A\otimes
C^{*}(G)$

and the diagram

${A}$${A\otimes C^{*}(G)}$${A\otimes C^{*}(G)}$${A\otimes C^{*}(G)\otimes
C^{*}(G)}$$\scriptstyle{\delta}$$\scriptstyle{\delta}$$\scriptstyle{\delta\otimes\text{id}}$$\scriptstyle{\text{id}\otimes\delta_{G}}$

commutes in $\mathbf{C}^{*}$. Here the morphism $\delta_{G}\colon C^{*}(G)\to
C^{*}(G)\otimes C^{*}(G)$ is the integrated form of the canonical unitary
homomorphism $s\mapsto s\otimes s$ for $s\in G$. There is a category
$\mathbf{Co}$ of coactions of $G$, in which the morphisms
$\phi:(A,\delta)\to(B,\varepsilon)$ are $\mathbf{C}^{*}$-morphisms $\phi:A\to
B$ that are $\delta-\varepsilon$ equivariant, i.e., the diagram

${A}$${A\otimes C^{*}(G)}$${B}$${B\otimes
C^{*}(G)}$$\scriptstyle{\delta}$$\scriptstyle{\phi}$$\scriptstyle{\phi\otimes\text{id}}$$\scriptstyle{\varepsilon}$

commutes. A morphism $\mu\colon C_{0}(G)\to B$ in $\mathbf{C}^{*}$ gives rise
to a _unitary coaction_ $\delta_{\mu}$ on $B$ defined by the formula

$\delta_{\mu}(b)=\operatorname{Ad}(\mu\otimes\text{id}(w_{G}))(b\otimes 1),$

and if $\pi\colon(A,\delta)\to(B,\delta_{\mu})$ is a morphism in
$\mathbf{Co}$, then $(B,\pi,\mu)$ is a _covariant representation_ of
$(A,\delta)$. A (full) _crossed product_ of $(A,\delta)$ is a universal
covariant representation $(A\rtimes_{\delta}G,j_{A},j_{G})$, i.e., for every
covariant representation $(B,\pi,\mu)$ of $(A,\delta)$ there is a unique
morphism $\pi\times\mu\colon A\rtimes_{\delta}G\to B$ in $\mathbf{C}^{*}$ such
that

$(\pi\times\mu)\circ j_{A}=\pi\quad\text{and }(\pi\times\mu)\circ j_{G}=\mu.$

By abstract nonsense, any two crossed products of $(A,\delta)$ are unique up
to unique isomorphism, and we pick one and call it “the” crossed product.

If $(A,\delta)$ is a coaction, then there is a coaction on
$A\otimes\mathcal{K}$ given by

$\delta\otimes_{*}\text{id}:=(\text{id}\otimes\Sigma)\circ(\delta\otimes\text{id}),$

where

$\Sigma\colon C^{*}(G)\otimes\mathcal{K}\simeq\mathcal{K}\otimes C^{*}(G)$

is the flip isomorphism determined by $c\otimes k\mapsto k\otimes c$, and
there is a natural isomorphism

$(A\otimes\mathcal{K})\rtimes_{\delta\otimes_{*}\text{id}}G\simeq(A\rtimes_{\delta}G)\otimes\mathcal{K}.$

If $(A,\alpha)$ is an action, there is a canonical coaction $\widehat{\alpha}$
on $A\rtimes_{\alpha}G$, called the _dual coaction_ , determined by

$\displaystyle\widehat{\alpha}\circ i_{A}$ $\displaystyle=i_{A}\otimes 1$
$\displaystyle\widehat{\alpha}\circ i_{G}(s)$ $\displaystyle=i_{G}(s)\otimes
s,$

and _Imai-Takai duality_ (see [3, Theorem A.67] for this formulation) says
that

$A\rtimes_{\alpha}G\rtimes_{\widehat{\alpha}}G\simeq A\otimes\mathcal{K}.$

Similarly, if $(A,\delta)$ is a coaction, there is a _dual action_
$\widehat{\delta}$ on $A\rtimes_{\delta}G$, where for $s\in G$ the
automorphism $\widehat{\delta}_{s}$ is determined by

$\displaystyle\widehat{\delta}_{s}\circ j_{A}$ $\displaystyle=j_{A}$
$\displaystyle\widehat{\delta}_{s}\circ j_{G}$
$\displaystyle=j_{G}\circ\textup{rt}_{s}.$

There is a _canonical surjection_ (see [12, Corollary 2.6])

$\Phi_{(A,\delta)}\colon A\rtimes_{\delta}G\rtimes_{\widehat{\delta}}G\to
A\otimes\mathcal{K},$

and $\delta$ is _maximal_ if $\Phi_{(A,\delta)}$ is an isomorphism ([2,
Definition 3.1]); this is called _Katayama duality_ for maximal coactions.

If $\phi\colon(A,\delta)\to(B,\varepsilon)$ is a morphism in $\mathbf{Co}$,
then there is a canonical _crossed-product_ morphism

$\phi\rtimes
G\colon(A\rtimes_{\delta}G,\widehat{\delta})\to(B\rtimes_{\varepsilon}G,\widehat{\varepsilon})$

in $\mathbf{Ac}$, and this induces a functor $\mathbf{Co}\to\mathbf{Ac}$.
Similarly, if $\phi\colon(A,\alpha)\to(B,\beta)$ is a morphism in
$\mathbf{Ac}$, then there is a canonical _crossed-product_ morphism

$\phi\rtimes
G\colon(A\rtimes_{\alpha}G,\widehat{\alpha})\to(B\rtimes_{\beta}G,\widehat{\beta})$

in $\mathbf{Co}$, and this gives a functor $\mathbf{Ac}\to\mathbf{Co}$.

A _maximalization_ of a coaction $(A,\delta)$ is a triple
$(B,\varepsilon,\psi)$ consisting of a maximal coaction $(B,\varepsilon)$ and
a morphism $\psi:(B,\varepsilon)\to(A,\delta)$ in $\mathbf{Co}$ such that
$\psi\rtimes G:B\rtimes_{\varepsilon}G\to A\rtimes_{\delta}G$ is an
isomorphism of $C^{*}$-algebras.

###  $\mathcal{K}$-decorated algebras

A _$\mathcal{K}$ -decorated algebra_ is a pair $(A,\iota)$, where
$\iota:\mathcal{K}\to A$ is a morphism in $\mathbf{C}^{*}$. There is a
category of $\mathcal{K}$-decorated algebras in which a morphism
$\phi\colon(A,\iota)\to(B,\jmath)$ is a $\mathbf{C}^{*}$-morphism $\phi\colon
A\to B$ such that $\phi\circ\iota=\jmath$. We occasionally refer to the map
$\iota$ as a _$\mathcal{K}$ -decoration_.

If $(A,\delta)$ is a coaction and $(A,\iota)$ is a $\mathcal{K}$-decorated
algebra, then $\delta$ is _$\mathcal{K}$ -fixing_ if
$\delta\circ\iota=\iota\otimes 1$, and we then call $(A,\delta,\iota)$ a
_$\mathcal{K}$ -fixing coaction_. The categories of coactions and of
$\mathcal{K}$-decorated algebras combine immediately to form a category
$\mathbf{KCo}$ of $\mathcal{K}$-fixing coactions in which a morphism
$\phi\colon(A,\delta,\iota)\to(B,\varepsilon,\jmath)$ is a morphism
$\phi\colon(A,\iota)\to(B,\jmath)$ of $\mathcal{K}$-decorated algebras that is
$\delta-\varepsilon$ equivariant.

The _relative commutant_ of a $\mathcal{K}$-decorated algebra $(A,\iota)$ is
the $C^{*}$-algebra

$C(A,\iota)=\\{m\in M(A):m\iota(k)=\iota(k)m\in A\text{ for all
}k\in\mathcal{K}\\}.$

Our reason for introducing relative commutants is the following
underappreciated fact: if $(A,\iota)$ is a $\mathcal{K}$-decorated algebra,
then there is an isomorphism
$C(A,\iota)\otimes\mathcal{K}\overset{\simeq}{\longrightarrow}A$ given on
elementary tensors by $m\otimes k\mapsto m\iota(k)$. This is probably
folklore; a proof can be found in [9, Proposition 3.4], where we give a more
detailed alternative to the argument in [5, Remark 3.1] (see also [4, Lemma
27.2], [6, Theorem 2.1]). As $G$ is assumed to be second-countable in [9, 4]
(but not in [5]), we have included a proof of the general case in Proposition
A.2 in order to get rid of this unnecessary assumption.

It follows that $C(A,\iota)$ is a nondegenerate $C^{*}$-subalgebra of $M(A)$,
and [5, Remark 3.1] characterizes it as the unique closed subset $Z$ of $M(A)$
that commutes elementwise with $\iota(\mathcal{K})$ and satisfies
$\operatorname{\overline{span}}\\{Z\iota(\mathcal{K})\\}=A$. Moreover, Fischer
further shows that $M(C(A,\iota))$ is the commutant of $\iota(\mathcal{K})$ in
$M(A)$. Using these facts, it is easy to show that any morphism
$\phi:(A,\iota)\to(B,\jmath)$ of $\mathcal{K}$-decorated algebras restricts
(after being extended to $M(A)$) to a morphism $C(\phi):C(A,\iota)\to
C(B,\jmath)$ in $\mathbf{C}^{*}$. If $(A,\delta,\iota)$ is a
$\mathcal{K}$-fixing coaction, then

$\delta\colon(A,\iota)\to(A\otimes C^{*}(G),\iota\otimes 1)$

is a morphism of $\mathcal{K}$-decorated algebras. Taking relative commutants
gives a morphism

$C(\delta):C(A,\iota)\to C(A\otimes C^{*}(G),\iota\otimes 1),$

and Fischer’s characterization quoted above easily yields the equality

$C(A\otimes C^{*}(G),\iota\otimes 1)=C(A,\iota)\otimes C^{*}(G),$

from which we conclude (see [5, Remark 3.2], and also [10, Lemma 3.2]) that
$C(\delta)$ is a coaction on $C(A,\iota)$. Moreover, if
$\phi:(A,\delta,\iota)\to(B,\varepsilon,\jmath)$ is a morphism in
$\mathbf{KCo}$ then
$C(\phi):(C(A,\iota),C(\delta))\to(C(B,\jmath),C(\varepsilon))$ gives a
morphism in $\mathbf{Co}$. In this way the relative commutant induces a
functor $\mathbf{Com}:\mathbf{KCo}\to\mathbf{Co}$, called the nondegenerate
destabilization functor in [9], given by the assignments

$(A,\delta,\iota)\mapsto(C(A,\iota),C(\delta))\quad\text{and}\quad\phi\mapsto
C(\phi).$

In fact, $\mathbf{Com}$ is a category equivalence, with quasi-inverse
$\mathbf{St}:\mathbf{Co}\to\mathbf{KCo}$ given by the assignments

$\displaystyle(A,\delta)$
$\displaystyle\mapsto(A\otimes\mathcal{K},\delta\otimes_{*}\text{id},1\otimes\text{id}_{\mathcal{K}})$
$\displaystyle\phi$ $\displaystyle\mapsto\phi\otimes\text{id}_{\mathcal{K}}.$

Equivalently, the stabilization functor $\mathbf{St}$ is a category
equivalence with quasi-inverse $\mathbf{Com}$. For each $\mathcal{K}$-fixing
coaction $(A,\delta,\iota)$, the isomorphism
$C(A,\iota)\otimes\mathcal{K}\overset{\simeq}{\longrightarrow}A$ described
above is $(C(\delta)\otimes\text{id})-\delta$ equivariant, and we denote it by
$\theta_{(A,\delta,\iota)}$. These isomorphisms combine to give a natural
isomorphism
$\theta:\mathbf{St}\circ\mathbf{Com}\overset{\simeq}{\longrightarrow}\text{id}_{\mathbf{KCo}}$.
Moreover, category theory tells us that there is also a unique natural
isomorphism
$\tau:\text{id}_{\mathbf{Co}}\overset{\simeq}{\longrightarrow}\mathbf{Com}\circ\mathbf{St}$
satisfying the _triangle identities_

${\mathbf{St}}$${\mathbf{St}\circ\mathbf{Com}\circ\mathbf{St}}$${\mathbf{St}}$$\scriptstyle{\mathbf{St}\,\circ\,\tau}$$\scriptstyle{1_{\mathbf{St}}}$$\scriptstyle{\theta\,\circ\,\mathbf{St}}$
and
${\mathbf{Com}}$${\mathbf{Com}\circ\mathbf{St}\circ\mathbf{Com}}$${\mathbf{Com}}$$\scriptstyle{\tau\,\circ\,\mathbf{St}}$$\scriptstyle{1_{\mathbf{Com}}}$$\scriptstyle{\mathbf{Com}\circ\theta}$

More precisely, $\theta$ is the counit for the adjunction
$\mathbf{St}\dashv\mathbf{Com}$, and then $\tau$ is the unique unit (for the
uniqueness, see for example, [11, Theorem 2.3.6], or more accurately its dual
version). Of course, the unit is a natural isomorphism since the counit is.

Actually, it will be more convenient for us to have the inverse natural
isomorphism

(2.1)
$\kappa:=\tau^{-1}:\mathbf{Com}\circ\mathbf{St}\overset{\simeq}{\longrightarrow}\text{id}_{\mathbf{Co}}.$

### Cocycles

If $(A,\delta)$ is a coaction, a _$\delta$ -cocycle_ is a unitary $u\in
M(A\otimes C^{*}(G))$ such that

  * •

$\text{id}\otimes\delta_{G}(u)=(u\otimes 1)(\delta\otimes\text{id})(u)$ and

  * •

$u\delta(A)u^{*}(1\otimes C^{*}(G))\subseteq A\otimes C^{*}(G)$.

The above axioms are designed so that
$\delta^{u}:=\operatorname{Ad}u\circ\delta$ is another coaction on $A$, called
a _perturbation_ of $\delta$ by $u$ or a _perturbed coaction_. Cocycles are
natural in the following sense: if $\phi\colon(A,\delta)\to(B,\varepsilon)$ is
a morphism in $\mathbf{Co}$ and $u$ is a $\delta$-cocycle, then
$(\phi\otimes\text{id})(u)$ is an $\varepsilon$-cocycle, and $\phi$ is also
$\delta^{u}-\varepsilon^{(\phi\otimes\text{id})(u)}$ equivariant.

Moreover, there is an isomorphism

$\Omega_{u}\colon(A\rtimes_{\delta^{u}}G,\widehat{\delta^{u}})\overset{\simeq}{\longrightarrow}(A\rtimes_{\delta}G,\widehat{\delta})$

in $\mathbf{Ac}$ for each $\delta$-cocycle $u$, and furthermore this family of
isomorphisms is natural in the sense that if
$\phi\colon(A,\delta)\to(B,\varepsilon)$ is a morphism in $\mathbf{Co}$ then
the diagram

${\bigl{(}A\rtimes_{\delta^{u}}G,\widehat{\delta^{u}}\bigr{)}}$${\bigl{(}B\rtimes_{\varepsilon^{(\phi\otimes\text{id})(u)}}G,\widehat{\varepsilon^{(\phi\otimes\text{id})(u)}}\bigr{)}}$${(A\rtimes_{\delta}G,\widehat{\delta})}$${(B\rtimes_{\varepsilon}G,\widehat{\varepsilon})}$$\scriptstyle{\phi\rtimes
G}$$\scriptstyle{\Omega_{u}}$$\scriptstyle{\simeq}$$\scriptstyle{\Omega_{(\phi\otimes\text{id})(u)}}$$\scriptstyle{\simeq}$$\scriptstyle{\phi\rtimes
G}$

commutes in $\mathbf{Ac}$.

An _equivariant action_ is a triple $(B,\alpha,\mu)$, where $(B,\alpha)$ is an
action and $\mu:(C_{0}(G),\textup{rt})\to(B,\alpha)$ is a morphism in
$\mathbf{Ac}$. The category $\mathbf{EAc}$ has equivariant actions
$(B,\alpha,\mu)$ as objects, and a morphism
$\phi:(B,\alpha,\mu)\to(C,\beta,\nu)$ in $\mathbf{EAc}$ is an
$\mathbf{Ac}$-morphism $\phi:(B,\alpha)\to(C,\beta)$ such that
$\phi\circ\mu=\nu$.

For any equivariant action $(B,\alpha,\mu)$, by [10, Lemma 3.1] the unitary
$u:=(i_{B}\circ\mu\otimes\text{id})(w_{G})$ is an $\widehat{\alpha}$-cocycle
such that the perturbed coaction $\widetilde{\alpha}:=(\widehat{\alpha})^{u}$
is $\mathcal{K}$-fixing, where we use the $\mathcal{K}$-decoration

$\mu\rtimes G:\mathcal{K}=C_{0}(G)\rtimes_{\textup{rt}}G\to
B\rtimes_{\alpha}G.$

Thus, the triple $(B\rtimes_{\alpha}G,\widetilde{\alpha},\mu\rtimes G)$ is a
$\mathcal{K}$-fixing coaction, and the crossed product induces a functor
$\mathbf{CPA}:\mathbf{EAc}\to\mathbf{KCo}$ with assignments

$(B,\alpha,\mu)\mapsto(B\rtimes_{\alpha}G,\widetilde{\alpha},\mu\rtimes
G)\quad\text{and}\quad\phi\mapsto\phi\rtimes G.$

On the other hand, for any coaction $(A,\delta)$, the triple
$(A\rtimes_{\delta}G,\widehat{\delta},j_{G})$ is an equivariant action, and
the crossed product induces a functor
$\mathbf{CPC}:\mathbf{Co}\to\mathbf{EAc}$ with assignments

$(A,\delta)\mapsto(A\rtimes_{\delta}G,\widehat{\delta},j_{G})\quad\text{and}\quad\phi\mapsto\phi\rtimes
G.$

Then we can apply the above to obtain a $\mathcal{K}$-fixing coaction

$(A\rtimes_{\delta}G\rtimes_{\widehat{\delta}}G,\widetilde{\delta},j_{G}\rtimes
G),$

where we have simplified the notation by writing $\widetilde{\delta}$ rather
than $\widetilde{\widehat{\delta}}$. We compose to get the functor
$\mathbf{CPA}\circ\mathbf{CPC}:\mathbf{Co}\to\mathbf{KCo}$ with assignments

$(A,\delta)\mapsto(A\rtimes_{\delta}G\rtimes_{\widehat{\delta}}G,\widetilde{\delta},j_{G}\rtimes
G)\quad\text{and}\quad\phi\mapsto(\phi\rtimes G)\rtimes G.$

### Maximalization

Let $(A,\delta)$ be a coaction. The canonical surjection
$\Phi_{(A,\delta)}:A\rtimes_{\delta}G\rtimes_{\widehat{\delta}}G\to
A\otimes\mathcal{K}$ is compatible with enough extra structure that it gives a
morphism

$\Phi_{(A,\delta)}:(A\rtimes_{\delta}G\rtimes_{\widehat{\delta}}G,\widetilde{\delta},j_{G}\rtimes
G)\to(A\otimes\mathcal{K},\delta\otimes_{*}\text{id},1\otimes\text{id}_{\mathcal{K}})$

in $\mathbf{KCo}$. Moreover, the family of morphisms $\\{\Phi_{(A,\delta)}\\}$
gives a natural transformation
$\Phi:\mathbf{CPA}\circ\mathbf{CPC}\to\mathbf{St}$.

We define a functor $\mathbf{Max}:\mathbf{Co}\to\mathbf{Co}$ as the
composition

$\mathbf{Max}:=\mathbf{Com}\circ\mathbf{CPA}\circ\mathbf{CPC}.$

We write

$(A^{m},\delta^{m}):=\mathbf{Max}(A,\delta)$

on objects in $\mathbf{Co}$, and $\phi^{m}:=\mathbf{Max}(\phi)$ on morphisms
in $\mathbf{Co}$. Fischer proves in [5, Theorem 6.4] that the coaction
$(A^{m},\delta^{m})$ is maximal.

Since the functor $\mathbf{Com}:\mathbf{KCo}\to\mathbf{Co}$ is a category
equivalence, it follows quickly that $(A,\delta)$ is maximal if and only if
$\mathbf{Com}(\Phi_{(A,\delta)})$ is an isomorphism, which in turn is
equivalent with

$\kappa_{\mathbf{Com}(\Phi_{(A,\delta)})}:(A^{m},\delta^{m})\to(A,\delta)$

being an isomorphism, where
$\kappa:\mathbf{Com}\circ\mathbf{St}:\overset{\simeq}{\longrightarrow}\text{id}_{\mathbf{Co}}$
is the natural isomorphism from (2.1).

Thus $\psi:=\kappa\circ\mathbf{Com}\circ\Phi$ gives a natural transformation
$\mathbf{Max}\to\text{id}_{\mathbf{Co}}$, and $(A,\delta)$ is maximal if and
only if $\psi_{(A,\delta)}:(A^{m},\delta^{m})\to(A,\delta)$ is an isomorphism.
For any coaction $(A,\delta)$, Fischer (further) proves in [5, Theorem 6.4]
that $(A^{m},\delta^{m},\psi_{(A,\delta)})$ is a maximalization of
$(A,\delta)$. It follows almost trivially that
$\mathbf{CPC}\circ\psi:\mathbf{CPC}\circ\mathbf{Max}\to\mathbf{CPC}$ is a
natural isomorphism. Then composing on the left with the functor
$\mathbf{Com}\circ\mathbf{CPA}$, we deduce that

$\mathbf{Max}\circ\psi:\mathbf{Max}^{2}\to\mathbf{Max}$

is a natural isomorphism. In Section 4 we will take these properties of
maximalization as the starting point for a quite abstract proof of a universal
property.

######  Remark 2.1.

In the literature the term “maximalization” has been used in several ways, and
as a result there is potential for confusion. For example, [2, Definition 3.1]
and [5, Definition 6.1] use the definition we quoted on page 2, but [1,
Definition 6.1.3] instead defines maximalization in terms of Fischer’s
universal property [5, Lemma 6.2]. We emphasize that in this paper we use the
definition given on page 2. Then [2, Theorem 3.3] and [5, Theorem 6.4] both
prove that every coaction has a maximalization; note that there is no danger
of circular reasoning in [5], because careful examination of Fischer’s proof
reveals that it does not appeal to the universal property [5, Lemma 6.2].
Nevertheless, to avoid any confusion, we rely here only upon the statement and
proof of [2, Theorem 3.3].

##  3\. The gap

In this short section we give a formal statement of Fischer’s universal
property, and we briefly point out the gap in the proof. Theorem 3.1 below
reproduces Proposition 6.1.11 in [1], which includes Fischer’s universal
property of maximalization [5, Lemma 6.2].

######  Theorem 3.1 ([1]).

Let $(B,\varepsilon)$ be a maximal coaction and
$\psi\colon(B,\varepsilon)\to(A,\delta)$ be a morphism in $\mathbf{Co}$. Then
$(B,\varepsilon,\psi)$ is a maximalization of $(A,\delta)$ if and only if for
every maximal coaction $(C,\zeta)$ and every morphism
$\tau\colon(C,\zeta)\to(A,\delta)$ in $\mathbf{Co}$ there exists a unique
morphism $\pi$ making the diagram

(3.1)
${(C,\zeta)}$${(B,\varepsilon)}$${(A,\delta)}$$\scriptstyle{\pi}$$\scriptstyle{!}$$\scriptstyle{\tau}$$\scriptstyle{\psi}$

commute in $\mathbf{Co}$.

One direction of Theorem 3.1 was originally stated in [5], but there is a gap
in Fischer’s argument for the uniqueness part. The idea is to embed Diagram
3.1 in the following larger diagram:

${C\rtimes G\rtimes G}$${A\rtimes G\rtimes G}$${B\rtimes G\rtimes
G}$${C\otimes\mathcal{K}}$${A\otimes\mathcal{K}}$${B\otimes\mathcal{K},}$$\scriptstyle{\tau\rtimes
G\rtimes
G}$$\scriptstyle{\Phi_{(C,\zeta)}}$$\scriptstyle{\simeq}$$\scriptstyle{\Phi_{(A,\delta)}}$$\scriptstyle{\psi\rtimes
G\rtimes
G}$$\scriptstyle{\simeq}$$\scriptstyle{\Phi_{(B,\varepsilon)}}$$\scriptstyle{\simeq}$$\scriptstyle{\tau\otimes\text{id}}$$\scriptstyle{\pi\otimes\text{id}}$$\scriptstyle{!}$$\scriptstyle{\psi\otimes\text{id}}$

then use the various properties to deduce the existence of a unique morphism
$\pi$ making the whole diagram commute. However (upon recent reflection) it is
unclear to us why that implies that $\pi$ is the unique morphism making
Diagram 3.1 commute, a conclusion that Fischer seems to take as obvious.
Another proof was given in [1], but there is again a (similar) gap with the
uniqueness in that proof.

##  4\. The abstract approach

In this section we first prove an abstract universal property that assumes a
couple of category-theoretic axioms. Then we apply this result to get a
complete, short proof of the universal property of maximalization.

Suppose $\mathcal{C}$ is a category, $M:\mathcal{C}\to\mathcal{C}$ is a
functor, and $\psi:M\to\text{id}_{\mathcal{C}}$ is a natural transformation
such that $M\psi:M^{2}\to M$ is a natural isomorphism. We recall that the
natural transformation $M\psi$ is given by

$(M\psi)_{x}:=M\psi_{x}:M^{2}x\to Mx$

for each object $x$ of $\mathcal{C}$. Let $\mathcal{C}_{m}$ be the full
subcategory of $\mathcal{C}$ formed by those objects $x$ for which $\psi_{x}$
is an isomorphism.

######  Proposition 4.1 (Abstract universal property).

If $f:y\to x$ in $\mathcal{C}$ with $y\in\mathcal{C}_{m}$, then there is a
unique morphism $\widetilde{f}$ in $\mathcal{C}$ making the diagram

${y}$${Mx}$${x}$$\scriptstyle{\widetilde{f}}$$\scriptstyle{f}$$\scriptstyle{\psi_{x}}$

commute.

###### Proof.

(Existence) The natural transformation $\psi$ gives the following commutative
diagram:

${My}$${Mx}$${y}$${x}$$\scriptstyle{Mf}$$\scriptstyle{\psi_{y}}$$\scriptstyle{\psi_{x}}$$\scriptstyle{f}$

Since $\psi$ is invertible, we can take $\widetilde{f}=Mf\circ\psi_{y}^{-1}$

(Uniqueness) Suppose $h,k:y\to Mx$ and $\psi_{x}h=\psi_{x}k=f$. Applying the
functor $M$ gives

$M\psi_{x}\circ Mh=M\psi_{x}\circ Mk,$

and invertibility of $M\psi_{x}$ implies $Mh=Mk$. Applying the natural
transformation $\psi$ to $h$ gives a commutative diagram

${My}$${M^{2}x}$${y}$${Mx}$$\scriptstyle{Mh}$$\scriptstyle{\psi_{y}}$$\scriptstyle{\psi_{Mx}}$$\scriptstyle{h}$

Thus $h=\psi_{Mx}\circ Mh\circ\psi_{y}^{-1}$. Similarly, we get
$k=\psi_{Mx}\circ Mk\circ\psi_{y}^{-1}$, and hence

$k=\psi_{Mx}\circ Mh\circ\psi_{y}^{-1}=h.\qed$

### Applying to maximalization

Let now

  * •

$\mathcal{C}=\mathbf{Co}$

  * •

$M=\mathbf{Max}$

  * •

$\psi:\mathbf{Max}\to\text{id}_{\mathcal{C}}$

be as in Section 2. Then, as seen in this section, all the properties of
$\mathcal{C},M$ and $\psi$ assumed at the beginning of the present section are
satisfied. Moreover, $\mathcal{C}_{m}=\mathbf{Co}_{m}$ is the full subcategory
of $\mathbf{Co}$ whose objects are the maximal coactions. Applying Theorem
4.1, we immediately recover Fischer’s universal property for maximalization,
that is, Theorem 4.1 holds.

##  5\. Dualizing

Our abstract result in Section 4 has a dual version. For completeness we state
this result below, and sketch how it can be used to show the universal
property of normalization for coactions. Since some direct proofs of this
property are available (see [7, Lemma 4.4] and [1, Proposition 6.1.5 and
Remark 6.1.8 ii)]), our main purpose here is to illustrate that maximalization
and normalization of coactions play a dual rôle.

Suppose $\mathcal{C}$ is a category, $N:\mathcal{C}\to\mathcal{C}$ is a
functor, and $\eta:\text{id}_{\mathcal{C}}\to N$ is a natural transformation
such that $N\eta:N\to N^{2}$ is a natural isomorphism.

Let $\mathcal{C}_{n}$ be the full subcategory of $\mathcal{C}$ formed by those
objects $x$ for which $\eta_{x}$ is an isomorphism. Then we have:

######  Proposition 5.1 (Abstract universal property II).

If $g:x\to y$ in $\mathcal{C}$ with $y\in\mathcal{C}_{n}$, then there is a
unique morphism $g^{\prime}:Nx\to y$ in $\mathcal{C}$ such that
$g^{\prime}\circ\eta_{x}=g$.

The proof is similar to the proof of Proposition 4.1. E.g., $g^{\prime}$ is
given by $g^{\prime}=\eta_{y}^{-1}\circ Ng$.

### Applying to normalization

Normality and normalization of a coaction can be defined in different (but
equivalent) ways, see for instance [13, 2, 3, 7, 1, 10]. We will follow [13,
3], but adapt the terminology to make it fit with the one used in Section 2.

Let $(A\rtimes_{\delta}G,j_{A},j_{G})$ denote a crossed product for a given
coaction $(A,\delta)$. We define $(A,\delta)$ to be _normal_ when $j_{A}:A\to
M(A\rtimes_{\delta}G)$ is injective. (This does not depend on the choice of
crossed product.)

We recall from Proposition 2.3 in [13] (or Lemma A.55 in [3]) that if
$(B,\pi,\mu)$ is a covariant representation of $(A,\delta)$, then there is a
coaction, which we denote here by $\delta_{(\pi,\mu)}$, on $\pi(A)$ such that
$(\pi(A),\delta_{(\pi,\mu)})$ is normal. In particular, the coaction
$(A^{n},\delta^{n}):=(j_{A}(A),\delta_{(j_{A},j_{G})})$ is normal. We call it
the _normalization of $(A,\delta)$_ and denote by $\eta_{(A,\delta)}$ the
homomorphism $j_{A}$, considered as a map from $A$ to $j_{A}(A)$.
(Alternatively, we could set $A^{n}=A/\ker j_{A}\simeq j_{A}(A)$ and let
$\eta_{(A,\delta)}$ denote the quotient map.)

As $\eta_{(A,\delta)}$ is nondegenerate and $\delta-\delta^{n}$ equivariant,
$\eta_{(A,\delta)}$ is a morphism from $(A,\delta)$ to $(A^{n},\delta^{n})$ in
$\mathbf{Co}$. For completeness we also mention that the induced
$\mathbf{C}^{*}$-morphim $\eta_{(A,\delta)}\rtimes G:A\rtimes_{\delta}G\to
A^{n}\rtimes_{\delta^{n}}G$ is an isomorphism that is
$\widehat{\delta}-\widehat{\delta^{n}}$ equivariant, cf. Lemma A.46 in [3] or
Proposition 2.6 in [13].

The normalization functor on coactions is introduced in [7] and [1], but both
these approaches build on the universal property of normalization. We
therefore indicate how this can be avoided. Let
$\phi:(A,\delta)\to(B,\varepsilon)$ be a morphism in $\mathbf{Co}$. Then there
is a unique morphism $\phi^{n}:(A^{n},\delta^{n})\to(B^{n},\varepsilon^{n})$
in $\mathbf{Co}$ such that

(5.1)
$\phi^{n}\circ\eta_{(A,\delta)}=\eta_{(B^{n},\varepsilon^{n})}\circ\phi.$

A proof of this statement without using the universal property goes like
this.222This argument is a special case of a more general argument used in the
proof of [7, Lemma 4.4]. Since equation (5.1) says that
$\phi^{n}(j_{A}(a))=j_{B^{n}}(\phi(a))$ for all $a\in A$, the uniqueness part
is clear. This formula will actually define $\phi^{n}$ if we know that $\ker
j_{A}$ is contained in $\ker\phi$. Since $j_{B^{n}}$ is injective, it suffices
to show that $\ker j_{A}\subseteq\ker(j_{B^{n}}\circ\phi)$. As one readily
verifies that $(j_{B^{n}}\circ\,\phi,j_{G})$ is a covariant representation of
$(A,\delta)$, we get that $((j_{B^{n}}\circ\phi)\times j_{G})\circ
j_{A}=j_{B^{n}}\circ\phi$, so this assertion is clear. It is then obvious that
$\phi^{n}$ is a $\mathbf{C}^{*}$-morphism and it is straightforward to check
that it is $\delta^{n}-\varepsilon^{n}$ equivariant.

We can now proceed and assert that the assignments

$(A,\delta)\mapsto(A^{n},\delta^{n})\text{ and }\phi\mapsto\phi^{n}$

give a functor ${\rm{\bf Nor}}:\mathbf{Co}\to\mathbf{Co}$, called the
_normalization functor_. To check that ${\rm{\bf Nor}}$ is really a functor is
routine. For example, let $\phi:(A,\delta)\to(B,\varepsilon)$ and
$\psi:(B,\varepsilon)\to(C,\gamma)$ be morphisms in $\mathbf{Co}$. Then using
(5.1), one gets that
$(\psi^{n}\circ\phi^{n})\circ\eta_{(A,\delta)}=\eta_{(C^{n},\gamma^{n})}\circ(\psi\circ\phi)$.
By uniqueness, this gives that $(\psi\circ\phi)^{n}=\psi^{n}\circ\phi^{n}$,
i.e., ${\rm{\bf Nor}}(\psi\circ\phi)={\rm{\bf Nor}}(\psi)\circ{\rm{\bf
Nor}}(\phi)$.

We note that equation (5.1) says that the transformation $\eta:{\rm
Id}_{\mathbf{Co}}\to\rm{\bf Nor}$ associated to the family
$\\{\eta_{(A,\delta)}\\}$ is natural. It is also obvious that the full
subcategory $\mathbf{Co}_{n}$ of $\mathbf{Co}$ whose objects $(A,\delta)$ are
such that $\eta_{(A,\delta)}$ is an isomorphism is the category of normal
coactions of $G$. Thus, to see that we may take $N=\rm{\bf Nor}$ and
$\mathcal{C}=\mathbf{Co}$ in Proposition 5.1 to deduce that the universal
property of normalization holds, it remains only to check that ${\rm{\bf
Nor}}\,\eta:\rm{\bf Nor}\to\rm{\bf Nor}^{2}$ is a natural isomorphism.

If $(A,\delta)$ is any coaction, then ${\rm{\bf
Nor}}\,\eta_{(A,\delta)}=(\eta_{(A,\delta)})^{n}$ is a morphism in
$\mathbf{Co}$ from $(A^{n},\delta^{n})$ to ${\rm{\bf
Nor}}(A^{n},\delta^{n})=((A^{n})^{n},(\delta^{n})^{n})$. Now,
$(\eta_{(A,\delta)})^{n}$ is uniquely determined by the identity

$(\eta_{(A,\delta)})^{n}\circ\eta_{(A,\delta)}=\eta_{(A^{n},\delta^{n})}\circ\eta_{(A,\delta)}.$

Since $\eta_{(A,\delta)}$ is surjective as a map from $A$ into $A^{n}$, this
means that

$(\eta_{(A,\delta)})^{n}=\eta_{(A^{n},\delta^{n})}.$

As $(A^{n},\delta^{n})$ is normal, we know that $j_{A^{n}}$ is injective.
Thus, we get that
$(\eta_{(A,\delta)})^{n}=\eta_{(A^{n},\delta^{n})}:A^{n}\to(A^{n})^{n}=j_{A^{n}}(A^{n})$
is an isomorphism, as desired.

We have thereby given another proof of the universal property of
normalization:

######  Theorem 5.2.

If $\phi\colon(A,\delta)\to(B,\varepsilon)$ is a morphism in $\mathbf{Co}$ and
the coaction $(B,\varepsilon)$ is normal, then there is a unique morphism
$\phi^{n}$ such that the diagram

${(A,\delta)}$${(A^{n},\delta^{n})}$${(B,\varepsilon)}$$\scriptstyle{\eta_{(A,\delta)}}$$\scriptstyle{\phi}$$\scriptstyle{\phi^{n}}$

commutes in $\mathbf{Co}$.

##  Appendix A Stabilization equivalence

In this appendix we record a completely general version of the equivalence
between the categories $\mathbf{KCo}$ and $\mathbf{Co}$ mentioned in Section
2; the separable case where $G$ is assumed to be second-countable appears in
[9, Theorem 4.4].

######  Theorem A.1.

The relative commutant functor $\mathbf{Com}:\mathbf{KCo}\to\mathbf{Co}$ is a
category equivalence, with quasi-inverse
$\mathbf{St}:\mathbf{Co}\to\mathbf{KCo}$ that takes an object $(A,\delta)$ to
the $\mathcal{K}$-fixing coaction
$(A\otimes\mathcal{K},\delta\otimes_{*}\text{id},1\otimes\text{id}_{\mathcal{K}})$
and a morphism $\phi:(A,\delta)\to(B,\varepsilon)$ to
$\phi\otimes\text{id}:(A\otimes\mathcal{K},\delta\otimes_{*}\text{id},1\otimes\text{id}_{\mathcal{K}})\to(B\otimes\mathcal{K},\varepsilon\otimes_{*}\text{id},1\otimes\text{id}_{\mathcal{K}})$.

###### Proof.

The proof in [9] that $\mathbf{St}:\mathbf{Co}\to\mathbf{KCo}$ is a category
equivalence, with quasi-inverse $\mathbf{Com}$, goes through without any
changes, except that we use Proposition A.2 below instead of [9, Proposition
3.4]. ∎

######  Proposition A.2.

If $(A,\iota)$ is a $\mathcal{K}$-decorated algebra, then there is an
isomorphism
$\theta:C(A,\iota)\otimes\mathcal{K}\overset{\simeq}{\longrightarrow}A$ given
on elementary tensors by $\theta(m\otimes k)=m\iota(k)$.

###### Proof.

We outline how the argument from [9, Proposition 3.4] can be adapted to cover
the general case. Choose a system $\\{u_{ij}\\}_{i,j\in S}$ of matrix units
for $\mathcal{K}$, and let $e_{ij}=\iota(u_{ij})$ be the corresponding matrix
units in $M(A)$. For each $i,j\in S$, let $A_{ij}=e_{ii}Ae_{jj}$. Let
$\mathcal{F}$ be the family of finite subsets of $S$, directed by inclusion.
For each $F\in\mathcal{F}$ let

$A_{F}=\sum_{i,j\in F}A_{ij}.$

Fix $p\in S$. For all $i,j$ there is a linear bijection $\tau_{ij}:A_{ij}\to
A_{pp}$ given by

$\tau_{ij}(a)=e_{pi}ae_{jp}.$

Note that $\tau_{ij}^{-1}(a)=e_{ip}ae_{pj}$. For each $F\in\mathcal{F}$ there
is a $\mathbf{C}^{*}$-isomorphism

$\phi_{F}:A_{F}\overset{\simeq}{\longrightarrow}A_{pp}\otimes M_{F}$

given by

$\phi_{F}\left(\sum_{i,j\in F}a_{ij}\right)=\sum_{i,j\in
F}\bigl{(}\tau_{ij}(a_{ij})\otimes u_{ij}\bigr{)}.$

Define a $\mathbf{C}^{*}$-morphism $\sigma:A_{pp}\to A$ by the strictly
convergent sum

$\sigma(a)=\sum_{i\in S}\operatorname{Ad}e_{ip}(a)=\sum_{i\in
S}\tau_{ii}^{-1}(a).$

Then

$\displaystyle e_{ij}\sigma(a)$ $\displaystyle=\sum_{k\in
S}e_{ij}e_{kp}ae_{pk}$ $\displaystyle=e_{ip}ae_{pj}$ $\displaystyle=\sum_{k\in
S}e_{kp}ae_{pk}e_{ij}$ $\displaystyle=\sigma(a)e_{ij},$

so $\sigma(A_{pp})\subseteq C(A,\iota)$.

Note that for $F\subseteq E$ in $\mathcal{F}$ we have canonical embeddings
$A_{F}\hookrightarrow A_{E}$ and $M_{F}\hookrightarrow M_{E}$, and we have an
inductive-limit isomorphism (see [8, Proposition 11.4.1])

$\phi:=\varinjlim\phi_{F}:\varinjlim
A_{F}\overset{\simeq}{\longrightarrow}\varinjlim(A_{pp}\otimes M_{F}),$

and

$\varinjlim A_{F}=A\quad\text{and}\quad\varinjlim(A_{pp}\otimes
M_{F})=A_{pp}\otimes\mathcal{K}$

because $\varinjlim M_{F}=\mathcal{K}$. ∎

## References

  * [1] E. Bédos, S. Kaliszewski, and J. Quigg.  Reflective-coreflective equivalence.  Theory Appl. Categ., 25:No. 6, 142–179, 2011.
  * [2] S. Echterhoff, S. Kaliszewski, and J. Quigg.  Maximal coactions.  Internat. J. Math., 15(1):47–61, 2004.
  * [3] S. Echterhoff, S. Kaliszewski, J. Quigg, and I. Raeburn.  A categorical approach to imprimitivity theorems for $C^{*}$-dynamical systems.  Mem. Amer. Math. Soc., 180(850):viii+169, 2006.
  * [4] R. Exel.  Partial dynamical systems, Fell bundles and applications, volume 224 of Mathematical Surveys and Monographs.  American Mathematical Society, Providence, RI, 2017.
  * [5] R. Fischer.  Maximal coactions of quantum groups.  SFB 478 – Geometrische Strukturen in der Mathematik, 350, Univ. Münster, 2004.
  * [6] J.v.B. Hjelmborg and M. Rørdam.  On stability of $C^{*}$-algebras.  J. Funct. Anal., 155:153–170, 1998.
  * [7] A. an Huef, J. Quigg, I. Raeburn, and D. P. Williams.  Full and reduced coactions of locally compact groups on $C^{\ast}$-algebras.  Expo. Math., 29(1):3–23, 2011.
  * [8] R. V. Kadison and J. R. Ringrose.  Fundamentals of the theory of operator algebras. Vol. II, volume 16 of Graduate Studies in Mathematics.  American Mathematical Society, Providence, RI, 1997.  Advanced theory, Corrected reprint of the 1986 original.
  * [9] S. Kaliszewski, T. Omland, and J. Quigg.  Destabilization.  Expo. Math., 34(1):62–81, 2016.
  * [10] by same author.  Dualities for maximal coactions.  J. Aust. Math. Soc., 102(2):224–254, 2017.
  * [11] T. Leinster.  Basic category theory, volume 143 of Cambridge Studies in Advanced Mathematics.  Cambridge University Press, Cambridge, 2014.
  * [12] M. Nilsen.  Duality for full crossed products of $C^{*}$-algebras by non-amenable groups.  Proc. Amer. Math. Soc., 126(10):2969–2978, 1998.
  * [13] J. C. Quigg.  Full and reduced $C^{*}$-coactions.  Math. Proc. Cambridge Philos. Soc., 116(3):435–450, 1994.