# Improved Group Robustness via Classifier Retraining on Independent Splits

Thien Hang Nguyen Hongyang R. Zhang Huy Le Nguyen
Northeastern University, Boston, MA

###### Abstract

Deep neural networks learned by minimizing the average risk can achieve strong
average performance, but their performance for a subgroup may degrade, if the
subgroup is underrepresented in the overall data population. Group
distributionally robust optimization [Sagawa et al., 2020a, GDRO] is a
standard baseline for learning models with strong worst-group performance.
However, GDRO requires group labels for every example during training and can
be prone to overfitting, often requiring careful model capacity control via
regularization or early stopping. When only a limited amount of group labels
is available, Just Train Twice [Liu et al., 2021, JTT] is a popular approach
which infers a pseudo-group-label for every unlabeled example. The process of
inferring pseudo labels can be highly sensitive during model selection. To
alleviate overfitting for GDRO and the pseudo labeling process for JTT, we
propose a new method via classifier retraining on independent splits (of the
training data). We find that using a novel sample splitting procedure achieves
robust worst-group performance in the fine-tuning step. When evaluated on
benchmark image and text classification tasks, our approach consistently
reduces the requirement of group labels and hyperparameter search during
training. Experimental results confirm that our approach performs favorably
compared with existing methods (including GDRO and JTT) when either group
labels are available during training or are only available during validation.

##  1 Introduction

For many tasks, deep neural networks (DNNs) are often developed where the test
distribution is identical to and independent of the train distribution, which
can be referred to as IID generalization. The performance of a DNN is also
known to worsen when the testing distribution differs from the training
distribution. This problem is often referred to as out-of-distribution (OOD)
generalization. OOD generalization is crucial in safety-critical applications
such as self-driving cars Filos et al. [2020] or medical imaging Oakden-Rayner
et al. [2020]. Hence, addressing the problem of OOD generalization is
foundational for real-world deployment of deep learning models.

A notable setting where OOD generalization problems appear is the group-shift
setting, where different groups of the data may have a distribution shift. In
this setting, there are predefined attributes that divide the input space into
different groups of interest. Here, the goal is to find a model that performs
well across several predefined groups [Sagawa et al., 2020a]. Similar to other
problems in OOD generalization, DNNs learned by empirical risk minimization
(ERM) are observed to suffer from poor worst-group performance despite good
average-group performance.

The difficulty with learning group robust DNNs can be attributed to the
phenomenon of shortcut learning [Geirhos et al., 2020] or spurious correlation
[Sagawa et al., 2020a, Arjovsky et al., 2019]. Shortcut learning poses that
ERM favors those models that discriminate based on simpler and/or spurious
features of the data. However, one wishes for the learning algorithm to
produce a model that uses features (i.e. correlations) that performs well not
only on the train distribution, but also on all potential distributions that a
task may generate, like that of a worst-group distribution.

In recent years, the group-shift setting has received considerable attention,
where Sagawa et al. [2020a] first investigates distributional robust
optimization (DRO) [Duchi et al., 2021, Ben-Tal et al., 2013] in this setting
and introduces Group DRO (GDRO) that attempts to directly optimize for the
worst-group error. Since then, GDRO has been the standard method for producing
group-robust models. However, GDRO can be sensitive towards model capacity
[Sagawa et al., 2020a] and require group labels for all examples at training
time.

As group annotations can be expensive to obtain, many works consider ways to
reduce the amount of groups labels needed [Liu et al., 2021, Creager et al.,
2021, Nam et al., 2022]. These methods usually follow the framework of first
inferring pseudo-group-labels using a certain referenced model (pseudo-
labeling) and then applying a group-robust algorithm like GDRO on the pseudo-
labelled data. While results for these methods have been promising, they
usually introduce several more sensitive hyperparameters that can be expensive
to tune. Our work aims to obtain group robust models using as few group labels
as possible while alleviating the need to carefully control model capacity.

Figure 1: Worst-group learning curve on the Waterbird dataset between GDRO
(left) and CROIS (right) from the setting in Section 4.2. In the left figure,
the validation accuracy of GDRO becomes unstable as the number of epochs
increase beyond 100 while the training accuracy remains close to 100%. Our
approach CROIS instead uses 30% of the training data with group labels for
fine-tuning the classification layer of a DNN that is obtained via ERM using
the rest of the training data without group labels (see also Algorithm 1).
This allows CROIS to reuse ERM features while improving generalization
compared with GDRO.

Our contribution. In this work, we propose a simple approach, called CROIS. By
foregoing the error-prone and costly pseudo-labeling phase to instead
concentrate on efficiently utilizing group labels by applying them only to the
final classifier layer, CROIS achieves good robust performance without relying
on a multitude of hyperparameters and large scale tuning, which has been a
growing concern in the community [Gulrajani and Lopez-Paz, 2021].

In short, CROIS takes advantage of the features learned by ERM [Kang et al.,
2019, Menon et al., 2021] while overcoming the deficiency of its memorization
behavior [Sagawa et al., 2020b] by utilizing the training data as two
independent splits: one group-unlabeled split to train the feature extractor
and one group-labeled split to retrain only the classifier with a robust
algorithm like GDRO. We demonstrate through ablation studies that the use of
independent splits is crucial for robust classifier retraining. Furthermore,
CROIS’s restriction of GDRO to only a low-capacity linear layer reduces GDRO’s
sensitivity towards model capacity as well as the amount of data needed for
GDRO to generalize well (e.g. Figure 1 and Figure 3).

For various settings where group labels are only partially available during
training (Section 4.1), our experimental results on standard datasets
including Waterbird, CelebA, MultiNLI, and CivilComments show improved
performance over existing methods including JTT [Liu et al., 2021] and SSA
[Nam et al., 2022], despite minimal parameter tuning and no reliance on pseudo
labeling. In another setting where more group labels are available (Section
4.2), even when using only a fraction of the available group labels, our
competitive robust performance against GDRO demonstrates our method’s label
efficiency. Finally, our results provide further evidences of ERM trained DNNs
containing good features on both image classification [Menon et al., 2021] and
natural language classification tasks.

###  1.1 Related works

There are three main settings for the group-shift problem: (1) full
availability of group labels, (2) limited availability of group labels, and
(3) no availability of group label. Other related areas include domain
generalization and long-tailed classification.

Full training group labels. Most methods here revolve around up-weighing
minority groups, subsampling minority groups [Sagawa et al., 2020b], or
performing GDRO [Sagawa et al., 2020a]. Follow-up works include integrating
data augmentation via generative model [Goel et al., 2020] or selective
augmentation [Yao et al., 2022] to a robust training pipeline.

Limited access to group labels. In this setting, the approach of inferring
more group labels for the group-unlabeled data remains the most popular. These
pseudo-group-labels are usually generated by training a referenced model that
performs the labeling. For example, [Liu et al., 2021] utilizes a low-capacity
model that create groups by labeling whether an example is correctly
classified by the referenced model or not. Similarly, works like [Creager et
al., 2021, Dagaev et al., 2021, Krueger et al., 2021, Nam et al., 2022] and
[Nam et al., 2020] are variants of this approach of inferring pseudo-group-
labels. These methods then proceed to use a group-robust algorithm like GDRO
[Sagawa et al., 2020a] or Invariant Risk Minimization (IRM) [Arjovsky et al.,
2019] to perform robust training on a new network with the newly generated
pseudo-group-labels.

No group labels. This setting removes the ability to validate as well as
knowledge of potential groups. This makes the problem more difficult as it is
unclear which correlation to look for during training. Some theoretical works
in this space include Hashimoto et al. [2018], Lahoti et al. [2020]. Sohoni et
al. [2020] proposes a popular empirical approach in this setting and has
popularized the pseudo-labeling and retraining approach. This setting is
related to domain generalization. Gulrajani and Lopez-Paz [2021] shows through
mass-scale experiments that most OOD generalization methods do not improve
over ERM given the same amount of tuning and model selection criterion.

Long-tailed classification. The long-tailed problem concerns with certain
classes having significantly fewer training examples than others (see for
example Haixiang et al. [2017] or Zhang et al. [2021] for a survey). Some
techniques from the long-tail literature, like Cao et al. [2019], has been
applied to the group-shift setting to account for the groups imbalances, as in
Sagawa et al. [2020a]. Insights from applications of representation learning
in the long-tail problem [Kang et al., 2019] gives valuable evidences for ERM
trained DNNs containing good features which is central for our method.

##  2 Preliminaries

For a classification task $\mathcal{T}$ of predicting labels in $\mathcal{Y}$
from inputs in $\mathcal{X}$, we are given training examples
$\\{(x_{i},y_{i})\\}_{i=1}^{n}$ that are drawn IID from some train
distribution $\mathcal{D}_{\text{train}}$. In the domain generalization
setting, we want good performance on some unknown test distribution
$\mathcal{D}_{\text{test}}$ that is different but related to
$\mathcal{D}_{\text{train}}$ through the task $\mathcal{T}$. More explicitly,
we wish to find a classifier $f$ from some hypothesis space $\mathcal{F}$
using $\mathcal{D}_{\text{train}}$ such that the classification error
$L(f)=\mathop{\mathbb{E}}_{x,y\sim\mathcal{D}_{\text{test}}}[f(x)\neq y]$ of
$f$ w.r.t. $\mathcal{D}_{\text{test}}$ is low. This framework encapsulates
many problems like adversarial robustness, domain adaptation, long-tail
learning, few-shot learning, and the problem considered here: group-shift.

In the group-shift setting [Sagawa et al., 2020a], we further assume that
associated with each data point $x$ is an attribute $a(x)$ (some sub-property
or statistics of $x$) from a set of possible attributes $\mathcal{A}$. These
attributes along with the labels form the set of possible groups
$\mathcal{G}=\mathcal{A}\times\mathcal{Y}$ that each example can take. We
denote an input $x$’s group label as $g(x)\in\mathcal{G}$. We then define the
classification error of a predictor $f$ (w.r.t. a fixed implicit distribution)
restricted to a group $g\in\mathcal{G}$ to be
$L_{g}(f):=\mathop{\mathbb{E}}_{x,y\mid g(x)=g}[f(x)\neq y].$ The notion of
worst-group error upper bounds the error of $f$ w.r.t. any group
$L_{wg}(f):=\max_{g\in\mathcal{G}}L_{g}(f).$ Using this notation, the group-
shift problem aims to discover a classifier in
$\operatorname*{arg\,min}_{f\in\mathcal{F}}\\{L_{wg}(f)\\}=\operatorname*{arg\,min}_{f\in\mathcal{F}}\\{\max_{g\in\mathcal{G}}L_{g}(f)\\}$.
Here, we see that the group-shift problem is just a special case of the domain
generalization problem when $\mathcal{D}_{\text{test}}$ is the distribution
consisting of only the points $(x,y)$ with $g(x)$ being restricted to the
worst-group of $f$ in $\mathcal{G}$. Here, GDRO solves this objective by
performing a minimax optimization procedure that alternates between the
model’s weight and relaxed weights on the groups.

Spurious Correlations and Memorization. As an example, consider the Waterbird
dataset as in Sagawa et al. [2020a], where the task is to distinguish whether
an image of a bird is a waterbird or a landbird. The bird images are then
modified with either a water or land background. In terms of our problem, the
type of bird forms the labels $\mathcal{Y}$, and the backgrounds are set to be
the attribute $\mathcal{A}$ for each type of bird. Altogether, these form four
groups: $\mathcal{G}=\mathcal{Y}\times\mathcal{A}=\\{\text{waterbird,
landbird}\\}\times\\{\text{water, land}\\}$.

This dataset is constructed so that the proportion of birds on matching
backgrounds (i.e. waterbird on water background and landbird on land
background) is significantly more than those of the mismatched backgrounds.
This is so that the backgrounds could be spuriously correlated with the
labels, as predicting the background alone would achieve a high on average
accuracy w.r.t. the train distribution already. As expected, for ERM trained
models, the groups with the highest error are the minority groups where the
background mismatches the type of the bird, suggesting that the model is
actually predicting using the background instead of the bird. Furthermore, the
fact that these high-capacity models achieve zero training error leads to the
conclusion that these models not only utilize spurious features like the
background to make its predictions, but also must have memorized the minority
groups during its training process [Sagawa et al., 2020b]. These problems are
common when there is data imbalance Feldman and Zhang [2020]. In the next
section, we propose a method that attempts to circumvent these issues.

##  3 Method: Classifier Retraining on Independent Splits

Algorithm 1 Classifier Retraining on Independent Splits (CROIS)

1: Input: Training data $D_{L}$ with group labels, training data without group
labels $D_{U}$, and validation data $D_{val}$ with group labels. Classifier
retraining algorithm $\mathcal{R}$ (default to GDRO).

2: Split proportion $p$: Partition $D_{L}$ into two parts, $D_{1}$ and
$D_{2}$, each with proportion $|D_{1}|=(1-p)\cdot|D_{L}|$ and
$|D_{2}|=p\cdot|D_{L}|$. Set $D_{U}\leftarrow D_{U}\cup D_{1}$. Set
$D_{L}\leftarrow D_{2}$.

3: Obtain the initial model $f$ by running ERM on $D_{U}$ and selecting the
best model in terms of average accuracy on $D_{val}$.

4: Perform classifier retraining $\mathcal{R}$ with feature extractor $f$ on
$D_{L}$ and then select the best model in terms of worst-group accuracy on
$D_{val}$ as the final output.

Algorithm 1 presents an outline for our main method: Classifier Retraining On
Independent Splits or CROIS. CROIS is simple: (1) partition/organize the
training data into one group-labeled split and one group-unlabeled split, (2)
obtain an ERM trained feature extractor with the group-unlabeled split, and
(3) perform robust classifier retraining with the group-labeled split, where
classifier retraining refers to fine-tuning the final linear layer of a DNN.
The optional parameter $p$ in step 2 determines the amount of group labels
used at train time so that $D_{U}$ is not too small. This option will be
utilized when we compare CROIS against GDRO in Section 4.2 .

Motivation. Inspired by previous works that have demonstrated the potential of
simple ERM trained DNNs on a variety of OOD tasks [Gulrajani and Lopez-Paz,
2021, Rosenfeld et al., 2021] and long-tailed tasks [Kang et al., 2019], our
method focuses on developing a simple method around ERM trained DNNs. As soon
to be discussed, a strength of an ERM trained DNN is that there are convincing
evidences that it contains good features. On the other hand, its weaknesses
involve memorizing examples and using spurious features. CROIS is designed to
alleviate these weaknesses while taking advantage of the good features of an
ERM trained DNN.

Models trained by ERM contains good features. While not the first work to
notice that ERM trained models contain good features, Kang et al. [2019]
demonstrates this hypothesis through extensive experiments on several long-
tailed vision datasets to investigate different learning strategies for
obtaining a feature extractor as well as ways to fine-tune the classifier
layer. There, an ERM trained feature extractor combined with a non-parametric
method of rescaling111Rescaling each row of the linear classifier using the
row’s norm to some power. See Kang et al. [2019]. the classifier layer
achieves (then) state-of-the-art results on all three datasets. A similar
study [Menon et al., 2021] has been done to confirm this hypothesis on vision
datasets in the group-shift setting. This suggests that one key to the data-
imbalance problem (for which the group-shift problem almost always suffers)
lies in correcting the classifier layer, which forms the first phase of CROIS.
It is peculiar that rescaling the classifier works best whereas intuition
suggests a data-dependent method like classifier retraining would be better.
We hypothesize that this is related to the next issue of our discussion.

Memorization behavior of high capacity DNNs. It has now been well known of
high capacity DNNs’ ability to memorize training examples [Zhang et al.,
2017]. In the group-shift setting, this behavior has been investigated by
Sagawa et al. [2020b], which provides empirical and theoretical justifications
for DNNs’ memorization behavior of minority groups’ training examples. This
behavior of memorizing minority examples have also been observed in the
broader framework of data imbalances [Feldman and Zhang, 2020]. One way to
circumvent memorization is to control the model’s capacity by incorporating
some combinations of high $\ell_{2}$ regularization, early stopping, and other
correctional parameters as has been done in Sagawa et al. [2020a]. However,
the additional tuning required might be costly. We instead tackle this via
independent splits.

Figure 2:  tSNE projection [Van der Maaten and Hinton, 2008] of the features
of an ERM trained ResNet50 on seen (left) versus unseen (right) examples from
Waterbird.222Half of the data is used to obtain a feature extractor while the
other half is used to obtain the features of the unseen examples. The features
of the minority groups (orange and yellow) from the unseen examples appear
better separated from the majority groups than that of the seen examples.
Using unseen examples plays a major role in CROIS’s ability to improve worst-
group performance via robust classifier retraining.

Circumventing memorization with independent splits.  Examples being memorized
must inevitably impact their ability to be useful in subsequent usage. As
memorized examples’ (i.e. already correctly classified) loss must be low,
their gradients contain little information to be of much use. Furthermore, the
features of memorized examples might not be representative of their group
during test time: Figure 2 presents a visualization of the features between
seen versus unseen examples. Thus, combining this observation with the
evidences for ERM trained DNNs containing good features, CROIS performs robust
classifier retraining on unseen examples ($D_{L}$ in Algorithm 1) in hope of
learning a classifier that utilizes features more representative of examples
during test time.

##  4 Experiments

In this section, we detail the main results of our experiments in two popular
group-shift settings: (1) where only group labels from the validation set are
available (as in Liu et al. [2021], Nam et al. [2022]); (2) when a fraction of
the training dataset examples also have group labels.

Setup. We use a similar setup as Liu et al. [2021]. To demonstrate the ease of
tuning of CROIS, unless noted otherwise (e.g. Table 2), we fix the
hyperparameters of both the ERM and the robust classifier retraining phase,
using standard parameters for ERM (see Appendix A for full details). Further
results of CROIS with tuned hyperparameters are presented in Section C of the
Appendix.

Datasets. We experiment on four standard datasets in the group-shift setting:

  * •

Waterbird [Sagawa et al., 2020a]. Combining the bird images from the CUB
dataset [Welinder et al., 2010] with water or land backgrounds from the PLACES
dataset [Zhou et al., 2017], the task is to classify whether an image contains
a landbird or a waterbird without confounding with the background. There are
$4795$ total training examples, where the minority group (waterbird, land
background) has only $56$ examples. In our experiments, we report the weighted
test average accuracy due to the skewed nature of the val and test set – this
is standard per Sagawa et al. [2020a].

  * •

CelebA [Liu et al., 2015] is a popular image dataset of celebrity faces. The
task is to classify the celebrity in the image is blond or not blond, with
male or not male as the confounding attribute. There are $162770$ total
training examples, and the smallest group (blond, male) has $1387$ examples.

  * •

MultiNLI [Williams et al., 2017] is a natural language inference dataset for
determining whether a sentence’s hypothesis is entailed by, is neutral with,
or contradicts its premise. The spurious attribute is the presence of negation
words like no, never, or nothing. This task has 6 groups, with $206175$ total
training and $1521$ in the minority group examples (is entailed and contains
negation).

  * •

CivilComments-WILDS [Borkan et al., 2019, Koh et al., 2021] is a natural
language dataset where the task is to classify whether a sentence is toxic or
non-toxic. There are 8 demographics – male, female, white, black, LGBTQ,
Muslim, Christian, and other religion– forming 16 groups that overlap, because
a comment can contain multiple demographics. Following Koh et al. [2021], we
evaluate on all 16 groups but only use the attribute black along with the
label in training. There is a total of $269038$ training examples with $1045$
minority examples from (other religion, toxic).

Models. We use ResNet50 [He et al., 2016] with ImageNet initialization and
batch-normalization for CelebA and Waterbird. We use pretrained BERT [Devlin
et al., 2018] for MultiNLI and CivilComments. We use the original train-val-
test split in all the datasets and report the test results. Cross-entropy is
used as the base loss for all objectives. SGD with momentum $0.9$ is used for
the vision datasets while the AdamW optimizer with dropout and a fixed
linearly-decaying learning rate is used for BERT. We use a batch size of 16
for CivilComments and 32 for the rest of the datasets. We do not use any
additional data augmentation or learning rate scheduler in our results.

###  4.1 Result of using group labels from only the validation set

In this section, we consider the setting where only group labels from the
validation set are available, and the training set is treated as a group-
unlabeled set [Liu et al., 2021, Nam et al., 2022]. Most methods in this
setting employ some pseudo-labeling approach to generate pseudo-group-labels
that are then used to train a new network via a robust algorithm like GDRO. On
the other hand, CROIS simply uses half of the validation set for classifier
retraining and the other half for model selection and does not rely on pseudo-
labeling. CROIS also reuses the initial model for the retraining phase, making
CROIS closer to that of a 1 phase procedure with additional fine-tuning.

We compare CROIS against JTT [Liu et al., 2021] and SSA [Nam et al., 2022] in
Table LABEL:tab:only-val-set. We report the Test Average (Avg Acc) and Worst-
Group Accuracy (Wg Acc). The results in Table LABEL:tab:only-val-set show that
CROIS outperforms JTT on all 4 datasets and SSA on 3 datasets using default
parameters.

Table 1: Experimental results for the setting when only group labels from the validation set are used. Results for JTT and SSA are taken from Liu et al. [2021] and Nam et al. [2022], respectively. | Waterbird | CelebA | MultiNLI | CivilComments
---|---|---|---|---
Method | Avg Acc | Wg Acc | Avg Acc | Wg Acc | Avg Acc | Wg Acc | Avg Acc | Wg Acc
JTT | 93.9 | 86.7 | 88.0 | 88.1 | 78.6 | 72.6 | 91.1 | 69.3
SSA |  92.2 (0.87) |  89.0 (0.55) |  92.8 (0.11) |  89.8 (1.28) |  79.9 (0.87) |  76.6 (0.66) |  88.2 (1.95) |  69.9 (2.02)
CROIS |  92.1 (0.29) |  90.9 (0.12) |  91.6 (0.61) |  88.5 (0.87) |  81.4 (0.06) |  77.4 (1.21) |  90.6 (0.20) |  70.3 (0.34)

Comparison with JTT. While both JTT and CROIS employ a 2 phase approach, the
two methods differ significantly from one another. First, the initial ERM
phase of JTT is used to infer pseudo-group-labels for the group-unlabeled
data, which is then used for full GDRO retraining in the second phase. JTT’s
first phase requires careful hyperparameter tuning and capacity control using
the group-labeled validation set to accurately produce pseudo-group-labels (as
noted in section 5.4 of Liu et al. [2021]). On the other hand, CROIS trains a
single ERM model and simply retrains the last layer with any available group
labels. Second, JTT’s final performance is limited by GDRO’s performance on
the full network, which can be worsen by mislabeled pseudo-labels from the
first phase. In contrast, we demonstrate in Section 4.2 that, by limiting GDRO
to only the last layer, CROIS is competitive to full GDRO even when using only
a fraction group labels and minimal tuning.

Comparison with SSA. Similarly to JTT, SSA is also a two-phase approach that
performs pseudo-labeling and then robust training (e.g. via GDRO) with the
inferred pseudo-labels. In SSA pseudo-labeling phase, it trains a separate
network that predicts the group rather than the class. By treating the pseudo-
labeling problem as a semi-supervised problem along with incorporating
threshold correction to calibrate for the pseudo-labels, SSA’s pseudo-labeling
capability is shown to improve upon JTT. On the other hand, CROIS, despite not
relying on pseudo labeling at all, manages to outperform SSA on 3 out of 4
datasets using default parameter.

Table 2: Worst-group test accuracy for reduced validation. Results for SSA and JTT are taken from Table 3 of Nam et al. [2022]. Results for CROIS here are tuned due to the smaller dataset size. % of group-labels from the validation set | CelebA | Waterbird
---|---|---
100% | 20% | 10% | 5% | 100% | 20% | 10% | 5%
JTT [Liu et al., 2021] | 81.1 | 81.1 | 81.1 | 82.2 | 86.7 | 84.0 | 86.9 | 76.0
SSA [Nam et al., 2022] | 89.8 | 88.9 | 90.0 | 86.7 | 89.0 | 88.9 | 88.9 | 87.1
CROIS (Ours) | 90.6 | 90.1 | 89.4 | 88.9 | 90.8 | 90.0 | 90.0 | 88.8

Adjusting validation dataset size. Following the setup in JTT and SSA, we vary
the size of the validation set (20%, 10% and 5% of the original) to test
whether our results still hold in these settings. We consider both the
Waterbird and CelebA datasets. Note that for this setting, CROIS must be
additionally tuned to account for the increased difficulty of the reduced
group-labels quantity. Nevertheless, the smaller examples quantity along with
just training the last layer makes the extra tuning not too expensive (details
and setup are deferred until Section E). From Table 2, we find that CROIS
outperforms JTT and SSA on various percentage levels.

###  4.2 Result of varying fraction of training group labels

Table 3: Results for running CROIS on some fraction of the training data. Numbers in parentheses denote one standard deviation across three runs. NCRT refers to naive classifier retraining i.e. when independent split is not used during the retraining phase. Results marked with † are taken from [Sagawa et al., 2020a]. ∗For Waterbird, we omit the result for $p=0.05$ due to the small dataset size and not being able to sample any minority-group example for robust retraining.  | Waterbird | CelebA | MultiNLI | CivilComments
---|---|---|---|---
| Avg Acc | Wg Acc | Avg Acc | Wg Acc | Avg Acc | Wg Acc | Avg Acc | Wg Acc
ERM | 96.9 | 69.8 | 95.6 | 44.4 | 82.8 | 66.0 | 92.1 | 63.2
GDRO |  93.2† |  86.0† |  91.8† |  88.3† |  81.4† |  77.7† |  89.6 (0.23) |  70.5 (2.10)
CROIS’ $p$ – group-labeled fraction used for retraining (with $1-p$ unlabeled
fraction for the ERM phase)
$0.05$ | * | * |  91.9 (0.50) |  88.9 (1.10) |  81.8 (0.15) |  73.8 (1.54) |  90.8 (0.40) |  63.3 (7.82)
$0.10$ |  95.4 (1.10) |  83.5 (3.24) |  91.3 (0.36) |  90.3 (0.82) |  80.8 (0.51) |  75.3 (2.06) |  89.5 (1.81) |  68.7 (1.72)
$0.30$ |  90.8 (0.35) |  89.6 (1.15) |  91.3 (0.44) |  90.6 (0.95) |  80.0 (0.31) | 77.9 (0.17) |  89.7 (0.33) |  68.6 (1.53)
$0.50$ |  90.4 (0.95) |  89.5 (0.59) |  91.9 (0.35) |  88.2 (2.10) |  79.8 (0.26) |  74.4 (1.00) |  89.5 (0.70) |  71.0 (1.50)
NCRT | 96.5 | 75.2 | 93.9 | 69.2 | 82.3 | 67.9 | 90.3 | 67.6

In this section, we compare CROIS using some fraction of group labels against
GDRO [Sagawa et al., 2020a] using all the group labels. In contrast to Section
4.1, the validation set is used only for model selection and not for
classifier retraining here. Once again, we fix parameters of CROIS to its
standard ERM parameter to demonstrate its ease-of-tuning (see Appendix A).

Varying fraction setup. We examine CROIS on different amount of group labels
available, via the parameter $p$. This means that $(1-p)$ fraction of the
training set is used to obtain a feature extractor in the first phase (that
uses no group label), and the rest $p$ fraction of group labels are used for
robust classifier retraining. In practice, $p$ is not a parameter to choose
(there’s no reason to throw away group labels) but rather is limited by the
resources available to obtain group labels. Our experiment simulates this
scenario where we obtain group labels for some random fraction of the data.
Additionally, to demonstrate the importance of retraining with unseen
examples, we experiment with robust classifier retraining using the same data
from the first phase i.e. without independent splits – denoted as NCRT. We
present the main results in Table LABEL:tab:train-labels-results.

Discussions. In Table LABEL:tab:train-labels-results, CROIS outperforms GDRO
on both the vision datasets and yields competitive performance to GDRO on the
two language datasets despite using only a fraction of group labels and using
default hyperparameters. Our result implies that one can obtain comparable or
even better robust performance than GDRO by simply collecting group labels for
only 30% of the available training data (modulo validation). One exception is
in severe group imbalance case as in CivilComments (the minority group
consists of only $0.4\%$ of the dataset). There, a higher fraction of group-
labeled data is beneficial to obtain more minority-group examples. Hence, a
more efficient sampling method to include more minority examples (e.g. filter
by labels first) would be beneficial in practice. Finally, the results for
NCRT also show the importance of using independent split for classifier
retraining.333In Sagawa et al. [2020a], Group Adjustment (GA) [Cao et al.,
2019] is observed to improve Waterbird’s WG accuracy to 90.5%. We also notice
an improvement when GA is incorporated to CROIS and obtain a $90.3\%\pm 0.62$
test WG accuracy. We also observe that, similarly to Sagawa et al. [2020a], GA
only works for the synthetic dataset Waterbird but not for CelebA nor
MultiNLI.

Worst-group robustness and proportion of group-labeled data. From the results
across the datasets, allocating more examples towards training the feature
extractor (lower $p$) generally yields higher on-average accuracies. The
worst-group error after classifier retraining has a more complex interaction
with $p$, as it depends on both the quality of the feature extractor and the
amount of group-labeled examples available to perform classifier retraining.
While varying the proportion $p$ in our experiments gives a rough estimate of
this tradeoff, we hypothesize that the availability of minority group examples
is the most important for obtaining a robust classifier. We further support
this intuition with an ablation study in Section D.2 where removing non-
minority examples has an insignificant impact on the group-robust performance.

Alleviating GDRO’s sensitivity towards model capacity. GDRO’s requirement for
model capacity control via either $\ell_{2}$ regularization or early stopping
is well noted in the literature [Sagawa et al., 2020a]. In Table 4, we compare
GDRO’s and CROIS’ sensitivity towards different $\ell_{2}$ regularization.
While GDRO’s performance on Waterbird is relatively uniform, GDRO is more
sensitivity to different $\ell_{2}$ settings on CelebA. When $\ell_{2}=1$ for
CelebA, we see that GDRO fails altogether (see Figure 3). On the other hand,
CROIS achieves consistent performance across different $\ell_{2}$ setting.
CROIS controls the model capacity by limiting GDRO to only the last layer.
This alleviates GDRO’s tendency to overfit and simplifies parameter tuning (as
in Figure 1 for Waterbird).

Table 4: Worst-group test accuracy for GDRO and CROIS ($p=0.30$) with different $\ell_{2}$ regularization. | CelebA | Waterbird
---|---|---
$\ell_{2}$ reg. | $0$ | $10^{-4}$ | $10^{-3}$ | $10^{-3}$ | $10^{-1}$ | 1 | $0$ | $10^{-4}$ | $10^{-3}$ | $10^{-3}$ | $10^{-1}$ | 1
GDRO | 81.7 | 81.7 | 81.7 | 83.9 | 87.8 | 0.00 | 86.8 | 86.8 | 86.8 | 86.8 | 87.1 | 86.5
CROIS | 90.6 | 91.5 | 90.0 | 90.0 | 90.3 | 90.0 | 90.3 | 90.6 | 90.6 | 90.6 | 90.0 | 88.2

Figure 3: While GDRO often requires high $\ell_{2}$ regularization to avoid
overfitting, setting $\ell_{2}$ too high might cause instability in GDRO’s
minimax optimization procedure (left).

###  4.3 Further ablation studies and discussions

Obtaining a good feature extractor. An ablation study on the effects of
different validation accuracies and initial algorithm on the feature
extractor’s quality (measured by robust performance after classifier
retraining) is presented in Section B. Similarly to previous works Kang et al.
[2019], we find ERM providing the best features over reweighting or GDRO (of
which both require group labels). We find that there’s a positive correlation
between validation average accuracy and features’ quality. This then serves as
a proxy for CROIS’ model selection criterion in the first phase, which
significantly simplifies parameter tuning over other two-phase methods in the
group-shift setting.

Table 5: Comparison between reweighting, subsampling, and GDRO as classifier retraining algorithms on top of the same feature extractor ($p=0.30$).  Retraining Method | Waterbird | CelebA | MultiNLI | CivilComments
---|---|---|---|---
Avg Acc | Wg Acc | Avg Acc | Wg Acc | Avg Acc | Wg Acc | Avg Acc | Wg Acc
Reweighting | 95.2 | 87.1 | 92.1 | 85.0 | 78.9 | 67.0 | 88.3 | 56.4
Subsampling | 95.8 | 81.1 | 91.6 | 86.1 | 78.6 | 64.0 | 91.8 | 59.3
GDRO | 91.4 | 90.2 | 91.6 | 90.4 | 80.3 | 78.0 | 89.7 | 69.1

GDRO is better than reweighting and subsampling for classifier retraining.
Table LABEL:tab:classifier-retraining contains results on using different
classifier retraining methods. We observe that GDRO produces the best group-
robust performance (since GDRO is designed for this setting after all).
Reweighting and subsampling seem to be effective on the vision datasets, but
fail to perform on the NLP datasets.

Table 6: Comparison between retraining with GDRO on the full network (Full) versus last layer (LL).  GDRO $p=0.30$ | Waterbird | CelebA | MultiNLI | CivilComments
---|---|---|---|---
Avg Acc | Wg Acc | Avg Acc | Wg Acc | Avg Acc | Wg Acc | Avg Acc | Wg Acc
LL (CROIS) | 91.4 | 90.2 | 91.6 | 90.4 | 80.3 | 78.0 | 89.7 | 69.1
Full | 90.5 | 79.8 | 91.6 | 78.3 | 80.8 | 75.1 | 90.4 | 69.1

Classifier retraining outperforms full retraining with GDRO. Classifier
retraining plays a central role in our method. In Table
LABEL:tab:network_ablation, we compare between fine-tuning with GDRO on the
full DNN versus just the last layer (LL) for an independent split of $p=0.30$
. We see that GDRO on the LL is much better than full GDRO on most of the
datasets (except CivilComments). However, the main difference is that while LL
retraining (i.e. CROIS) requires little additional tuning, we must grid search
for different regularization strength for GDRO when applied on the full
network. CROIS can be shown to be quite robust to different parameter setting
and regularization strength (details in Appendix C.1).

Models learned with ERM contain good features. The positive result for our
decoupled training procedure provides another strong evidence for ERM trained
models containing good features for the group-shift problem. While this is
consistent with findings in the literatures on vision datasets [Kang et al.,
2019, Menon et al., 2021], our work further provides some of the first
evidences of this hypothesis in non-vision tasks, where the same result would
not have been possible without independent split, as evident in the result for
Naive CRT in Table LABEL:tab:train-labels-results.

Simplified model selection. The model selection criterion of picking the best
average validation accuracy model simplifies hyperparameter tuning in
comparison with other two-phases methods. This decision has been chosen mainly
from the ablation experiments in Section B of the Appendix, where we observe
that higher average validation accuracy generally suggests better features.

##  5 Conclusion and Future Works

In this paper, we propose Classifier Retraining on Independent Split (CROIS)
as a simple method to reduce the amount of group annotations needed for
improving worst-group performance as well as alleviate GDRO’s requirement for
careful control of model capacity. Our experimental results show the
effectiveness of CROIS on four standard datasets across two settings and
further provide evidences that ERM trained models contain good features for
the group-shift problem.

Future works. The richness of ERM trained DNNs’ features can potentially be
useful towards solving the seemingly harder group-agnostic setting (where no
group label is available) by allowing the practitioner to focus on obtaining a
robust classifier given an ERM trained feature extractor, where we have shown
that reasonable robustness can be achieved with relatively few group-labels,
which makes the problem seem closer in reach. On a broader note, while most
works in representation learning focus on producing good features (either with
supervised, unsupervised, or self-supervised approaches), further examinations
into different ways to perform classifier retraining in different settings (as
in our work) could give a fuller picture to the features quality of different
methods.

## References

  * Arjovsky et al. [2019] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz.  Invariant risk minimization.  _arXiv preprint arXiv:1907.02893_ , 2019.
  * Ben-Tal et al. [2013] Aharon Ben-Tal, Dick Den Hertog, Anja De Waegenaere, Bertrand Melenberg, and Gijs Rennen.  Robust solutions of optimization problems affected by uncertain probabilities.  _Management Science_ , 59(2):341–357, 2013.
  * Biewald [2020] Lukas Biewald.  Experiment tracking with weights and biases, 2020.  URL https://www.wandb.com/.  Software available from wandb.com.
  * Borkan et al. [2019] Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman.  Nuanced metrics for measuring unintended bias with real data for text classification.  In _Companion proceedings of the 2019 world wide web conference_ , pages 491–500, 2019.
  * Cao et al. [2019] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma.  Learning imbalanced datasets with label-distribution-aware margin loss.  _Advances in Neural Information Processing Systems_ , 32:1567–1578, 2019.
  * Creager et al. [2021] Elliot Creager, Jörn-Henrik Jacobsen, and Richard Zemel.  Environment inference for invariant learning.  In _International Conference on Machine Learning_ , pages 2189–2200. PMLR, 2021.
  * Dagaev et al. [2021] Nikolay Dagaev, Brett D Roads, Xiaoliang Luo, Daniel N Barry, Kaustubh R Patil, and Bradley C Love.  A too-good-to-be-true prior to reduce shortcut reliance.  _arXiv preprint arXiv:2102.06406_ , 2021.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Duchi et al. [2021] John C Duchi, Peter W Glynn, and Hongseok Namkoong.  Statistics of robust optimization: A generalized empirical likelihood approach.  _Mathematics of Operations Research_ , 2021.
  * Feldman and Zhang [2020] Vitaly Feldman and Chiyuan Zhang.  What neural networks memorize and why: Discovering the long tail via influence estimation.  In _NeurIPS_ , 2020.
  * Filos et al. [2020] Angelos Filos, Panagiotis Tigkas, Rowan McAllister, Nicholas Rhinehart, Sergey Levine, and Yarin Gal.  Can autonomous vehicles identify, recover from, and adapt to distribution shifts?  In _International Conference on Machine Learning_ , pages 3145–3153. PMLR, 2020.
  * Geirhos et al. [2020] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann.  Shortcut learning in deep neural networks.  _Nature Machine Intelligence_ , 2(11):665–673, 2020.
  * Goel et al. [2020] Karan Goel, Albert Gu, Yixuan Li, and Christopher Re.  Model patching: Closing the subgroup performance gap with data augmentation.  In _International Conference on Learning Representations_ , 2020.
  * Gulrajani and Lopez-Paz [2021] Ishaan Gulrajani and David Lopez-Paz.  In search of lost domain generalization.  In _ICLR_. OpenReview.net, 2021.
  * Haixiang et al. [2017] Guo Haixiang, Li Yijing, Jennifer Shang, Gu Mingyun, Huang Yuanyue, and Gong Bing.  Learning from class-imbalanced data: Review of methods and applications.  _Expert Systems with Applications_ , 73:220–239, 2017.
  * Hashimoto et al. [2018] Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang.  Fairness without demographics in repeated loss minimization.  In _International Conference on Machine Learning_ , pages 1929–1938. PMLR, 2018.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Kang et al. [2019] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis.  Decoupling representation and classifier for long-tailed recognition.  In _International Conference on Learning Representations_ , 2019.
  * Koh et al. [2021] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al.  Wilds: A benchmark of in-the-wild distribution shifts.  In _International Conference on Machine Learning_ , pages 5637–5664. PMLR, 2021.
  * Krueger et al. [2021] David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville.  Out-of-distribution generalization via risk extrapolation (rex).  In _International Conference on Machine Learning_ , pages 5815–5826. PMLR, 2021.
  * Lahoti et al. [2020] Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Ed Chi.  Fairness without demographics through adversarially reweighted learning.  _Advances in neural information processing systems_ , 33:728–740, 2020.
  * Liu et al. [2021] Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn.  Just train twice: Improving group robustness without training group information.  In _International Conference on Machine Learning_ , pages 6781–6792. PMLR, 2021.
  * Liu et al. [2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In _Proceedings of International Conference on Computer Vision (ICCV)_ , December 2015.
  * Menon et al. [2021] Aditya Krishna Menon, Ankit Singh Rawat, and Sanjiv Kumar.  Overparameterisation and worst-case generalisation: friend or foe?  In _International Conference on Learning Representations_ , 2021.
  * Nam et al. [2020] Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin.  Learning from failure: Training debiased classifier from biased classifier.  _arXiv preprint arXiv:2007.02561_ , 2020.
  * Nam et al. [2022] Junhyun Nam, Jaehyung Kim, Jaeho Lee, and Jinwoo Shin.  Spread spurious attribute: Improving worst-group accuracy with spurious attribute estimation.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=_F9xpOrqyX9.
  * Oakden-Rayner et al. [2020] Luke Oakden-Rayner, Jared Dunnmon, Gustavo Carneiro, and Christopher Ré.  Hidden stratification causes clinically meaningful failures in machine learning for medical imaging.  In _Proceedings of the ACM conference on health, inference, and learning_ , pages 151–159, 2020.
  * Rosenfeld et al. [2021] Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski.  An online learning approach to interpolation and extrapolation in domain generalization.  _arXiv preprint arXiv:2102.13128_ , 2021.
  * Sagawa et al. [2020a] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang.  Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization.  _International Conference of Learning Representation_ , 2020a.
  * Sagawa et al. [2020b] Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang.  An investigation of why overparameterization exacerbates spurious correlations.  In _International Conference on Machine Learning_ , pages 8346–8356. PMLR, 2020b.
  * Sohoni et al. [2020] Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré.  No subclass left behind: Fine-grained robustness in coarse-grained classification problems.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Van der Maaten and Hinton [2008] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _Journal of machine learning research_ , 9(11), 2008.
  * Welinder et al. [2010] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona.  Caltech-UCSD Birds 200.  Technical Report CNS-TR-2010-001, California Institute of Technology, 2010\.
  * Williams et al. [2017] Adina Williams, Nikita Nangia, and Samuel R Bowman.  A broad-coverage challenge corpus for sentence understanding through inference.  _arXiv preprint arXiv:1704.05426_ , 2017.
  * Yao et al. [2022] Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn.  Improving out-of-distribution robustness via selective augmentation.  _arXiv preprint arXiv:2201.00299_ , 2022.
  * Zhang et al. [2017] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning requires rethinking generalization.  In _ICLR_. OpenReview.net, 2017.
  * Zhang et al. [2021] Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng.  Deep long-tailed learning: A survey.  _arXiv preprint arXiv:2110.04596_ , 2021.
  * Zhou et al. [2017] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Places: A 10 million image database for scene recognition.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2017.

##  Appendix A Experimental Details

###  A.1 Infrastructure

We performed our experiment on 2 PCs with one NVIDIA RTX3070 and one NVIDIA
RTX3090. Our implementation is built on top of the code base from Liu et al.
[2021]. Experimental data was collected with the help of Weights and Biases
[Biewald, 2020].

Table 7: Hyperparameters used in the experiments. The slash indicates the parameters used in the first phase (feature extractor) versus the second phase (classifier retraining).  | Waterbird | CelebA | MultiNLI | CivilComments
---|---|---|---|---
Learning Rate | $10^{-4}/10^{-4}$ | $10^{-4}/10^{-4}$ | $2\times 10^{-5}/2\times 10^{-5}$ | $10^{-5}/10^{-5}$
$\ell_{2}$ Regularization | $10^{-4}/10^{-4}$ | $10^{-4}/10^{-4}$ | $0/0$ | $10^{-2}/0$
Number of Epochs | 250/250 | 20/20 | 20/20 | 6/6

###  A.2 Hyperparameters

Table 7 contains the hyperparameters used in our experiments in Sections 4.2
and 4.1. Note that these are the standard parameters for obtaining an ERM
model for these datasets as in previous works [Sagawa et al., 2020a, Liu et
al., 2021]. The only difference is that we train Waterbird and CelebA for
slightly shorter epoch due to finding no further increase in validation
accuracies after those epochs.

In our experiments, unless noted, we do not tune for any other
hyperparameters. For the second phase of CivilComments, we do not use the
default regularization but opt for $0$ since the linear layer already has low
capacity. However, adding further regularization does not seem to have much of
an effect as in section C.1.

##  Appendix B Ablation studies: Obtaining a good feature extractor

###  B.1 Impact of The Feature Extractor’s Algorithms

Here, we provide evidences that ERM trained models produce the best features
for worst-group robustness. We conduct an experiment on Waterbird, where
instead of using ERM to obtain a feature extractor, we perform GDRO and
Reweighing instead on the first phase. The results are in table 8. While using
reweighing or GDRO for the first phase defeats the purpose of reducing the
amount of group-labels needed (whereas ERM doesn’t need any), it is
informative to examine the features alone.

Table 8: Effects of different methods for obtaining a feature extractor on test average accuracy and test worst-group accuracy (with ResNet50 on Waterbird).  Feature extractor via | Test Avg Acc | Test Wg Acc
---|---|---
Reweighing | 90.1 | 88.8
GDRO | 90.8 | 88.6
ERM | 90.5 | 90.2

Here, we see that even though ERM does not use group-labels, it provides the
best features for robust classifier retraining on independent split.

###  B.2 Impact of Early Stopping and Validation Accuracies on The Feature
Extractor

In this section, we present an ablation study of how different early stopping
epoch (figure 4), average validation accuracy (figure 5 left), and worst-group
accuracy (figure 5 right) of the initial ERM trained model (feature extractor)
affect the GDRO retraining phase of CROIS. The results here are from
performing CROIS with GDRO and $p=0.30$ on Waterbird across a wide variety of
epochs. Table 9 presents the full data generated for this section.

Figure 4: The effect of using different epochs for the feature extractor
(phase 1) on classifier retraining’s (phase 2) test accuracies.

Figure 5: The effect of different validation average accuracies (left) and validation worst-group accuracies (right) from the feature extractor (phase 1) on classifier retraining’s (phase 2) test accuracies.  Table 9: CROIS with GDRO $(p=0.30)$ on Waterbird. Average (Avg Acc) and Worst-group (Wg Acc) Accuracies for various epochs of the feature extractor (“Phase 1”) and the corresponding test accuracies for classifier retraining (“Phase 2”). While training for longer epochs seem to help with average and worst-group accuracy for phase 2, the benefit is small. Hence, simply selecting the best validation average accuracy model, row Epoch 131 and denoted BEST here, yields good enough features that simplify our training procedure and model selection criteria. CROIS ($p=0.3$) | Feature Extractor (Phase 1) | Classifier Retraining (Phase 2)
---|---|---
Phase 1 Epoch | Val Avg | Val WG | Val Avg | Val WG | Test Avg | Test WG
0 | 91.3 | 0.05 | 87.8 | 85.6 | 86.6 | 85.2
1 | 94 | 13.5 | 88 | 87.6 | 86 | 85.5
2 | 95.2 | 20.3 | 89.6 | 88 | 88.3 | 86.9
3 | 95.7 | 25.6 | 90.1 | 88 | 88.5 | 88.2
4 | 95.8 | 25.6 | 88.9 | 88.2 | 87.1 | 86.9
5 | 96.4 | 32.3 | 89.2 | 88.7 | 87.3 | 87.1
6 | 96.5 | 38.4 | 90.5 | 88.7 | 89.3 | 88.9
7 | 96.3 | 29.3 | 90.6 | 88.7 | 89.3 | 88.5
8 | 97.1 | 38.4 | 90.1 | 89.3 | 88 | 87.7
9 | 97.1 | 42.9 | 90.4 | 89.9 | 88.2 | 87.9
10 | 96.7 | 40.6 | 91.2 | 90.2 | 89.4 | 88
20 | 97.3 | 54.9 | 91.4 | 91 | 90.3 | 89.5
50 | 97.4 | 55 | 91.5 | 91 | 90.4 | 89.7
100 | 97.2 | 57.1 | 90.5 | 90.2 | 89.5 | 89.2
131 (Best) | 97.6 | 55.6 | 90.9 | 90.2 | 89.7 | 89.6
150 | 97.2 | 60.2 | 90.7 | 90.2 | 89.6 | 89.4
200 | 97.5 | 52.6 | 91 | 90.2 | 88.8 | 88.2
250 | 97.2 | 59.4 | 91 | 90.4 | 88.3 | 87.7

##  Appendix C Hyperparameter tuning: CROIS vs. GDRO

###  C.1 Further hyperameter exploration on CROIS

In our paper, we have demonstrated CROIS’s effectiveness even with just using
the same hyperparameters as to train an ERM model. In this section, we present
results for further additional parameter tuning on the robust classifier
retraining phase. These results provide empirical evidence for CROIS’s
potential as well as robustness to different hyperparameter setting.

####  $\ell_{2}$ regularization.

We investigate whether additional  regularization would be helpful to
classifier retraining with GDRO on CelebA (Table 10) and Waterbird. (Table 11)
We further examine the effects of regularization on CivilComments (Table 12)
to support our choice in Section A. For these experiments, we further run each
setting acorss $3$ random seeds to further investigate CROIS’s sensitivity
towards hyperparameters.

Table 10: Effects of $\ell_{2}$ regularization on classifier retraining with GDRO on CelebA. | $p=0.50$ | $p=0.30$ | $p=0.10$
---|---|---|---
$\ell_{2}$ Reg. | Avg Acc | Wg Acc | Avg Acc | Wg Acc | Avg Acc | Wg Acc
$1$ | 91.6 (0.06) | 87.4 (2.76) | 90.9 (1.13) | 88.4 (0.78) | 91.2 (0.35) | 90.0 (0.70)
$10^{-2}$ | 92.1 (0.32) | 87.6 (1.56) |  91.8 (0.21) | 87.5 (3.54) | 92.0 (0.44) | 88.3 (2.39)
$10^{-4}$ | 91.9 (0.35) |  88.2 (2.10) | 91.3 (0.44) |  90.6 (0.95) | 91.3 (0.36) |  90.3 (0.82)
0 |  92.2 (0.25) | 86.8 (2.30) | 91.5 (0.07) | 87.8 (3.96) |  92.1 (0.47) | 88.1 (2.73)
Table 11: Effects of $\ell_{2}$ regularization on classifier retraining with GDRO on Waterbird. | $p=0.50$ | $p=0.30$ | $p=0.10$
---|---|---|---
$\ell_{2}$ Reg. | Avg Acc | Wg Acc | Avg Acc | Wg Acc | Avg Acc | Wg Acc
$1$ | 89.6 (0.93) | 88.9 (1.37) |  91.7 (0.44) |  89.8 (0.68) | 94.5 (0.50) | 85.8 (0.15)
$10^{-2}$ |  90.4 (1.31) | 89.3 (0.61) | 90.6 (1.25) | 89.2 (1.31) | 95.1 (0.40) | 86.3 (0.83)
$10^{-4}$ |  90.4 (0.95) |  89.5 (0.59) | 90.8 (0.35) | 89.6 (1.15) |  95.4 (1.10) | 83.5 (3.24)
0 | 89.8 (0.53) | 89.3 (0.70) | 90.6 (1.31) | 89.1 (1.16) | 95.1 (0.35) |  86.4 (0.90)
Table 12: Effects of $\ell_{2}$ regularization on classifier retraining with GDRO on CivilComments. | $p=0.50$ | $p=0.30$ | $p=0.10$
---|---|---|---
$\ell_{2}$ Reg. | Avg Acc | Wg Acc | Avg Acc | Wg Acc | Avg Acc | Wg Acc
$1$ | 88.8 (1.30) | 70.0 (1.63) | 89.5 (0.35) | 66.4 (1.99) | 89.3 (1.69) | 68.9 (2.64)
$10^{-2}$ | 89.4 (0.99) | 70.6 (0.42) | 89.5 (0.29) | 68.5 (0.87) | 88.6 (1.70) |  70.2 (1.63)
0 |  89.5 (0.70) |  71.0 (1.50) |  89.7 (0.33) |  68.6 (1.53) |  89.5 (1.81) | 68.7 (1.72)

#### Learning rate.

We examine the effects of different learning rates on CROIS on CelebA and
Waterbird in Table 13. Lower learning rate seems to be more beneficial.

Table 13: Effects of varying learning rate on CROIS $p=0.30$ on CelebA and Waterbird. We fix $\ell_{2}$ regularization to $10^{-4}$. | Learning rate | $10^{-5}$ | $10^{-4}$ | $10^{-3}$ | $10^{-2}$ | $10^{-1}$
---|---|---|---|---|---|---
CelebA | Average accuracy | 92.2 | 91.4 | 91.2 | 90.1 | 91.2
Worst-group accuracy | 88.3 | 90 | 90.3 | 87.8 | 82.8
Waterbird | Average accuracy | 89.7 | 90.6 | 94.2 | 93.2 | 93.9
Worst-group accuracy | 89.5 | 88.9 | 87.1 | 88.8 | 78.5

###  C.2 Sensitivity to model capacity: CROIS versus GDRO

In this section, we present a comparison between CROIS and GDRO test
performance with different $\ell_{2}$ regularization configuration on CelebA
and Waterbird in Table 4. On CelebA, GDRO is quite sensitive to model capacity
while it is less so on Waterbird. We note that GDRO fails to converge to a
good stationary point when $\ell_{2}=1$ on CelebA (see Figure 3).

##  Appendix D Further studies on classifier retraining

###  D.1 Impact of Robust Retraining on Independent Splits

In this section, we examine how robust retraining affects the model’s
prediction of $D_{U}$ and $D_{L}$ before and after robust classifier
retraining on independent split (with $p=0.30$). Tables 14 and 15 show the
accuracy on $D_{U}$ and $D_{L}$ for Waterbird and CelebA. The “Points changed”
column indicates the number of points that the model changes prediction after
robust retraining per group along with the total number of examples in that
group (with percentage in parentheses). The worst-group is underlined in the
tables.

Table 14: Model’s prediction of $D_{U}$ and $D_{L}$ on Waterbird before robust retraining and after robust classifier retraining on independent split.  Waterbird ($p=0.3$) | Accuracy on $D_{U}$ | Accuracy on $D_{L}$
---|---|---
Before | After | Points changed | Before | After | Points changed
Avg Acc | 100 | 94.5 | 184/3356 (5.48%) | 96.4 | 89.6 | 162/1439 (11.3%)
Group 0 (73.0%) | 100 |  92.6 | 180/2430 (7.41%) | 99.6 |  88.2 | 122/1068 (11.4%)
Group 1 (3.84%) | 100 | 99.3 | 1/141 (7.09%) | 76.7 | 100 | 10/43 (23.3%)
Group 2 (1.17%) | 100 | 100 | 0/38 (0.00%) |  44.4 | 100 | 10/18 (55.6%)
Group 3 (22.0%) | 100 | 99.6 | 3/747 (0.40%) | 90.8 | 92.3 | 20/310 (6.45%)
Table 15: Model’s prediction of $D_{U}$ and $D_{L}$ on CelebA before and after robust classifier retraining on independent split. RR means after Robust Retraining. CelebA ($p=0.30$) | Accuracy on $D_{U}$ | Accuracy on $D_{L}$
---|---|---
Before | After | Points changed | Before | After | Points changed
Avg Acc | 96.5 | 92.4 | 7537/113939 (6.61%) | 95.6 | 92.0 | 3274/48831 (11.3%)
Group 0 (43.7%) | 96.7 |  91.7 | 2609/50311 (5.19%) | 95.8 |  91.3 | 1011/21318 (4.74%)
Group 1 (41.4%) | 99.6 | 92.2 | 3489/46652 (7.48%) | 99.6 | 92.2 | 1493/20222 (7.38%)
Group 2 (14.1%) | 89.9 | 95.2 | 1000/16012 (6.25%) | 86.8 | 93.7 | 517/6868 (7.53%)
Group 3 (0.87%) |  46.3 | 91.8 | 439/964 (45.4%) |  35.5 | 95.3 | 253/423 (59.8%)

Interestingly, after robust retraining, the worst-group almost always switches
from the minority group to the majority group regardless of the data split.

###  D.2 The Importance of Minority Examples: Subsampled Retraining

As alluded to in Section 4.2, group-imbalance seems to play an important role
in the robust performance of CROIS. To further demonstrate this point, we
consider how CROIS performs when the second phase is subsampled versus when it
is allowed additional non-minority examples.

Table 16: Performance of CROIS when retraining is on a subsampled split versus the full split. Subsampling the split doesn’t seem to impact CROIS’s performance, indicating the importance of the availability of minority examples. Dataset (minority group size, fraction) | Subsampled retraining | Full retraining
---|---|---
Avg Acc | Wg Acc | Avg Acc | Wg Acc
CelebA ($n=423$, 0.87%) | 90.6 | 89.7 | 91.9 | 88.9
Waterbird ($n=18$, 1.2%) | 91.3 | 87.6 | 90.3 | 88.9

As the result in Table 16 shows, there isn’t a significant difference between
the two sampling strategies, suggesting that the availability of minority
group examples plays an important role in robust classifier retraining.

##  Appendix E Fraction of the validation set implementation details from
Section 4.1

Following the setup in Section 4.1 and the setup as in Liu et al. [2021], Nam
et al. [2022], we further reduce the validation set to only a small fraction,
$5\%$, $10\%$, and $20\%$. We investigate CROIS’s performance in this very few
group-labels setting across CelebA and Waterbird in Section 4.1.

We note that the highly reduced sample size poses new challenge and makes it
harder to simply reuse the default parameters. Hence, we tune the learning
rate (across $\\{10^{-5},10^{-4},10^{-3},10^{-2},10^{-1}\\}$), $\ell_{2}$
regularization (across $\\{0,10^{-4},10^{-2}\\}$) and the batch size (details
below) for CROIS to work with the smaller amount of examples. We also perform
model selection via the training accuracy instead of an independently split.

  * •

The use of group-labels and model selection: Since the amount of examples for
classifier retraining is now significantly reduced, it might be wasteful to
further split our available group-labels for validation. Instead, we use all
the available group-labels for robust classifier retraining and perform model
selection in the second phase via the train worst-group accuracy. The low
capacity linear layer allows us to avoid overfitting when performing model
selection this way. The feature extractor from the first phase is selected via
the best average accuracy on the full group-unlabeled validation set.

  * •

Smaller batch size: Since GDRO requires group-balanced sampling, a batch size
greater than the number of examples in a certain group would cause duplicate
sampling of the minority-group examples in the same step, artificially
increasing the weight for that group. We further tune for batch sizes across a
grid of powers of $2$ less than the smallest group or the default batch size
(e.g. we search across $\\{4,8,16\\}$ if the size of the smallest group is
$17$).

  * •

Regularization: When using so little data, overfitting can become a bigger
problem, even when just training a low-capacity linear classifier. Hence, we
tune for $\ell_{2}$ regularization.

In Table 2, we present the results for CROIS with the above modifications and
compare it to CROIS and JTT. There, robust retraining for CelebA is performed
with an $\ell_{2}$ regularization of $0.1$ and Waterbird with group-
adjustments equalling $1$ (these are similar to Sagawa et al. [2020a]).

This result shows that CROIS maintains its robustness performance even at
greatly reduced group-labels. This implies that even a few (minority) examples
can help debias the final layer classifier with proper configurations.

##  Appendix F Implications and takeaways

#### Comparison to standard pretraining and finetuning

As mentioned in the related works section as well as during the discussion of
the motivation for our method, pretraining and then finetuning is a now well-
known and established strategy in many domains. Our work differs the most
significantly from this standard strategy through:

  1. 1.

The use of independent splits: Traditional pretraining and finetuning reuses
the dataset for both phase with the possibility of additional labels
(contrastive learning, long-tail learning, etc.). We have demonstrated through
extensive experiments the importance of independent split for classifier
retraining to work well for the group robust setting.

  2. 2.

The use of a group robust algorithm for finetuning: We mainly utilize GDRO for
our finetuning phase while most other works in finetuning make use of simpler
strategy for finetuning like reweighing or subsampling. We demonstrated in our
experiments that GDRO yields the best robust performance over other more
common methods.

Our work provides evidences that the features of ERM trained DNNs are rich
enough to solve the group-shift problem (when an abundant amount of group
labels is available to retrain the classifier) and one of the major reasons
for poor worst-group performance of an ERM trained DNN is within its
classifier layer. We then further demonstrate that even a few group labels can
sufficiently “fix” the classifier to achieve better group-robust performance.

This knowledge can potentially be useful towards solving the seemingly much
harder group-agnostic setting (where no group label is available) by allowing
the practitioner to focus on obtaining a robust classifier given an ERM
trained feature extractor. Our experiments further show that reasonable
robustness can be achieved with relatively few group-labels (that are not used
to obtain the feature extractor), which makes the problem seem closer in
reach.


# A D-module approach to invariant distributions
with finitely many orbits

Hiroyuki Ochiai Keywords: D-module, holonomic system, invariant distribution,
orbit decomposition, Lie algebra. This work is partially supported by JSPS
Grant-in-Aid for Transformative Research Areas (A) (22H05107).

(December 1, 2023)

###### Abstract

In this paper, we provide an understanding of Tauchi’s example, which involves
countably many linearly independent relatively invariant distributions on real
projective space, through the lens of D-modules.

##  1 Introduction

In 2018, Tauchi [10] discovered a representation of a Lie group with finitely
many orbits, where the dimension of relatively invariant distributions with a
fixed character is infinite. This intriguing representation is motivated by
providing an example demonstrating that the smooth regular representation on
some homogeneous space of $SL(2n,\mathbb{R})$ with finitely many orbits under
a real mirabolic subgroup has infinite multiplicity of a degenerate principal
series representation [11], [12]. Distributions of interest are characterized
by being annihilated by vector fields corresponding to a Lie algebra. In this
paper, we explore the left ideal generated by these vector fields in the ring
of differential operators and the corresponding D-modules. This D-module must
be non-holonomic due to the infinite dimension of its solutions. We identify
countably many quotient D-modules, each turning out to be a simple holonomic
D-module. Subsequently, we revisit the Lie algebraic interpretation of these
quotient D-modules. It’s important to note that the group used here is
solvable (upper triangular), potentially resulting in the corresponding
D-module having irregular singularities. Nevertheless, these simple holonomic
D-modules must exhibit regular singularities.

The interplay between Lie algebra and D-modules for relatively invariant
distributions was leveraged by the author [7] to gain insights into the
seminal work of van Dijk. In [13], among other contributions, he highlights
the non-existence of a spherical distribution — an invariant eigen-
distribution of the pseudo-Laplacian — on the tangent space of a rank-one
semisimple symmetric space $SL(m+1,\mathbb{R})/GL(m,\mathbb{R})$ supported on
the singular nilpotent orbits. This marked the beginning of harmonic analysis
on this semisimple symmetric space [21, 22, 18, 19] and the exploration of a
generalized Gelfand pair [14, 15, 20]. Kowata [5] establishes the non-
existence of singular invariant distributions and hyperfunctions [4] without
resorting to the pseudo-Laplacian, subject to certain conditions. For
instance, when considering the tangent space of
$SL(m+1,\mathbb{R})/GL(m,\mathbb{R})$, Kowata identifies a larger symmetry,
$SO_{0}(m,m)$, compared to $GL(m,\mathbb{R})$. The author [7] rearticulates
this finding in terms of D-modules. Notably, in this tangent space setting, as
well as the global setting [8], two Lie algebras,
$\mathfrak{gl}(m,\mathbb{R})$ and $\mathfrak{so}(m,m)$, exhibit the same
D-modules. As a corollary, we deduce an automatic extension of the symmetry of
spherical distribution and the non-existence of singular spherical
distribution. In other words, there is no contribution of singular orbits
under the smaller Lie group.

Tauchi’s example focuses primarily on real projective spaces $\mathbb{P}^{3}$
and $\mathbb{P}^{5}$ of dimensions three and five, respectively. These spaces
correspond to the real vector spaces $\mathbb{R}^{4}\setminus\\{0\\}$ and
$\mathbb{R}^{6}\setminus\\{0\\}$ with the origin removed and divided by
$\mathbb{R}^{\times}$. In our analysis, we operate on vector spaces
$\mathbb{R}^{2n}=\mathbb{C}^{n}$ with $n=2,3$ without dividing by
$\mathbb{R}^{\times}$, and we observe that including the origin poses no
issues.

We introduce various real Lie subgroups of $GL(2n,\mathbb{R})$ and their Lie
algebras, including their complexifications. Further, we delve into the orbit
decomposition of $\mathbb{R}^{2n}$ and $\mathbb{C}^{2n}$ for each case.

In the context of the ring of differential operators with polynomial
coefficients, denoted as
$\mathcal{D}=\mathbb{C}[z_{1},\ldots,z_{2n},z_{1},\ldots,z_{2n}]$, we define
the left ideal $\mathcal{I}$ of $\mathcal{D}$ and the corresponding D-module
$\mathcal{D}/\mathcal{I}$, along with its hyperfunction or distribution
solutions. The reference for D-modules will be [1, 2].

##  2 $n=2$

###  2.1 Lie algebra

Define a solvable Lie subalgebra of $\mathfrak{gl}(4,\mathbb{C})$ by

$\mathfrak{h}_{1}:=\mathfrak{a}\oplus\mathfrak{n}:=(\mathbb{C}E_{11}\oplus\mathbb{C}E_{22}\oplus\mathbb{C}E_{33}\oplus\mathbb{C}E_{44})\oplus(\mathbb{C}E_{14}\oplus\mathbb{C}E_{24}\oplus\mathbb{C}E_{31}\oplus\mathbb{C}E_{32}\oplus\mathbb{C}E_{34}).$

The Lie algebra $\mathfrak{h}_{1}$ is illustrated as

$\begin{pmatrix}*&0&0&*\\\ 0&*&0&*\\\ *&*&*&*\\\ 0&0&0&*\\\
\end{pmatrix}\begin{pmatrix}z_{1}\\\ z_{2}\\\ z_{3}\\\
z_{4}\end{pmatrix},\qquad\begin{pmatrix}*&*&*&*\\\ 0&*&0&*\\\ 0&0&*&*\\\
0&0&0&*\end{pmatrix}\begin{pmatrix}z_{3}\\\ z_{2}\\\ z_{1}\\\
z_{4}\end{pmatrix},$

where the figure on the left is represented in standard coordinates, whereas
the figure on the right illustrates the upper-triangular realization of the
Lie algebra. Put $c=\begin{pmatrix}0&0&1&0\\\ 0&1&0&0\\\ 1&0&0&0\\\
0&0&0&1\end{pmatrix}$ and a Lie algebra isomorphism
$\iota:\mathfrak{gl}(4)\rightarrow\mathfrak{gl}(4)$ by $\iota(A)=cAc^{-1}$.

We now consider the several subalgebra
$\mathfrak{h}_{3}\subset\mathfrak{h}_{2}\subset\mathfrak{h}_{1}$ defined by

$\displaystyle\mathfrak{h}_{2}$
$\displaystyle:=\mathfrak{a}\oplus(\mathbb{C}E_{14}\oplus\mathbb{C}E_{32}).$
$\displaystyle\mathfrak{h}_{3}$
$\displaystyle:=(\mathbb{C}(E_{11}+E_{22})\oplus\mathbb{C}(E_{33}+E_{44}))\oplus(\mathbb{C}E_{14}\oplus\mathbb{C}E_{32}).$
$\mathfrak{h}_{1}=\begin{pmatrix}*&0&0&*\\\ 0&*&0&*\\\ *&*&*&*\\\ 0&0&0&*\\\
\end{pmatrix},\quad\mathfrak{h}_{2}=\begin{pmatrix}*&0&0&*\\\ 0&*&0&0\\\
0&*&*&0\\\ 0&0&0&*\\\
\end{pmatrix},\quad\mathfrak{h}_{3}=\begin{pmatrix}a_{11}&0&0&a_{14}\\\
0&a_{11}&0&0\\\ 0&a_{32}&a_{44}&0\\\ 0&0&0&a_{44}\\\ \end{pmatrix}.$
$\iota(\mathfrak{h}_{1})=\begin{pmatrix}*&*&*&*\\\ 0&*&0&*\\\ 0&0&*&*\\\
0&0&0&*\end{pmatrix},\quad\iota(\mathfrak{h}_{2})=\begin{pmatrix}*&*&0&0\\\
0&*&0&0\\\ 0&0&*&*\\\ 0&0&0&*\\\
\end{pmatrix},\quad\iota(\mathfrak{h}_{3})=\begin{pmatrix}a_{44}&a_{32}&0&0\\\
0&a_{11}&0&0\\\ 0&0&a_{11}&a_{14}\\\ 0&0&0&a_{44}\end{pmatrix}.$

###  2.2 Real form

Let $X:=\mathbb{C}^{4}$ be a complex four-dimensional vector space with the
standard coordinates $\mathbf{z}=(z_{1},z_{2},z_{3},z_{4})^{T}$. Let
$J:=\begin{pmatrix}0&0&1&0\\\ 0&0&0&1\\\ 1&0&0&0\\\ 0&1&0&0\end{pmatrix}$. We
define a real four-dimensional vector space
$X_{\mathbb{R}}:=\\{\mathbf{z}=(z_{1},z_{2},z_{3},z_{4})^{T}\in\mathbb{C}^{4}\mid\mathbf{z}=J\overline{\mathbf{z}}\\}=\\{(z_{1},z_{2},\overline{z_{1}},\overline{z_{2}})\mid
z_{1},z_{2}\in\mathbb{C}\\}$. Then $X_{\mathbb{R}}$ is a real form of the
complex vector space $X=\mathbb{C}^{4}$. The corresponding real form of the
complex Lie algebra $\mathfrak{gl}(4,\mathbb{C})=End_{\mathbb{C}}(X)$ is
$\mathfrak{g}_{\mathbb{R}}:=\\{X\in\mathfrak{gl}(4,\mathbb{C})\mid
J\overline{X}J=X\\}$.

We define the real Lie algebra
$\mathfrak{h}_{i\mathbb{R}}:=\mathfrak{h}_{i}\cap\mathfrak{g}_{\mathbb{R}}$
for $i=2,3$.

######  Lemma 1.

  * •

$\mathfrak{g}_{\mathbb{R}}=\begin{pmatrix}a_{11}&a_{12}&a_{13}&a_{14}\\\
a_{21}&a_{22}&a_{23}&a_{24}\\\
\overline{a_{13}}&\overline{a_{14}}&\overline{a_{11}}&\overline{a_{12}}\\\
\overline{a_{23}}&\overline{a_{24}}&\overline{a_{21}}&\overline{a_{22}}\end{pmatrix}$.

  * •

$\mathfrak{h}_{2\mathbb{R}}=\begin{pmatrix}a_{11}&0&0&a_{14}\\\
0&a_{22}&0&0\\\ 0&\overline{a_{14}}&\overline{a_{11}}&0\\\
0&0&0&\overline{a_{22}}\end{pmatrix}=\begin{pmatrix}a_{11}&0&0&a_{14}\\\
0&\overline{a_{44}}&0&0\\\ 0&\overline{a_{14}}&\overline{a_{11}}&0\\\
0&0&0&a_{44}\end{pmatrix}$, which is a real form of $\mathfrak{h}_{2}$.

  * •

$\mathfrak{h}_{1}\cap\mathfrak{g}_{\mathbb{R}}=\mathfrak{h}_{2\mathbb{R}}$.

  * •

$\mathfrak{h}_{3}\cap\mathfrak{g}_{\mathbb{R}}=\begin{pmatrix}a_{11}&0&0&a_{14}\\\
0&a_{11}&0&0\\\ 0&\overline{a_{14}}&\overline{a_{11}}&0\\\
0&0&0&\overline{a_{11}}\end{pmatrix}$, which is real form of
$\mathfrak{h}_{3}$.

$\iota(\mathfrak{g}_{\mathbb{R}})=\begin{pmatrix}\overline{a_{11}}&\overline{a_{14}}&\overline{a_{13}}&\overline{a_{12}}\\\
a_{23}&a_{22}&a_{21}&a_{24}\\\ a_{13}&a_{12}&a_{11}&a_{14}\\\
\overline{a_{21}}&\overline{a_{24}}&\overline{a_{23}}&\overline{a_{22}}\end{pmatrix},\quad\iota(\mathfrak{h}_{2\mathbb{R}})=\begin{pmatrix}\overline{a_{11}}&\overline{a_{14}}&0&0\\\
0&\overline{a_{44}}&0&0\\\ 0&0&a_{11}&a_{14}\\\
0&0&0&a_{44}\end{pmatrix},\quad\iota(\mathfrak{h}_{3\mathbb{R}})=\begin{pmatrix}\overline{a_{11}}&\overline{a_{14}}&0&0\\\
0&a_{11}&0&0\\\ 0&0&a_{11}&a_{14}\\\ 0&0&0&\overline{a_{11}}\end{pmatrix}$

The dimension of each Lie algebra is

$\dim_{\mathbb{C}}\mathfrak{h}_{1}=9,\ \dim_{\mathbb{C}}\mathfrak{h}_{2}=6,\
\dim_{\mathbb{C}}\mathfrak{h}_{3}=4,\
\dim_{\mathbb{R}}\mathfrak{h}_{2\mathbb{R}}=6,\
\dim_{\mathbb{R}}\mathfrak{h}_{3\mathbb{R}}=4.$

###  2.3 Orbit decomposition

For each Lie algebra or its corresponding Lie group, we consider the orbit
decomposition by the action of $GL(4,\mathbb{C})\supset H_{1}\supset
H_{2}\supset H_{3}$ on $X=\mathbb{C}^{4}$ and the action of
$G_{\mathbb{R}}\supset H_{1\mathbb{R}}=H_{2\mathbb{R}}\supset H_{3\mathbb{R}}$
on $X_{\mathbb{R}}=\mathbb{C}^{2}$. The upper triangular realization
$\iota(\mathfrak{h}_{i})$ is preferred over the standard realization
$\mathfrak{h}_{i}$ to facilitate the observation of the orbit decomposition.

####  2.3.1 complex

We define Zariski locally closed subsets $O_{2},O_{1},O_{0}$ of
$\mathbb{C}^{2}=\\{(z_{3},z_{2})\in\mathbb{C}^{2}\\}$ and
$O^{\prime}_{2},O^{\prime}_{1},O^{\prime}_{0}$ of
$\mathbb{C}^{2}=\\{(z_{1},z_{4})\in\mathbb{C}^{2}\\}$ by

$\displaystyle O_{2}$ $\displaystyle:=\\{z_{2}\neq 0\\},\quad
O_{1}:=\\{z_{2}=0,z_{3}\neq 0\\},\quad O_{0}:=\\{z_{2}=z_{3}=0\\},$
$\displaystyle O^{\prime}_{2}$ $\displaystyle:=\\{z_{4}\neq 0\\},\quad
O^{\prime}_{1}:=\\{z_{4}=0,z_{1}\neq 0\\},\quad
O^{\prime}_{0}:=\\{z_{4}=z_{1}=0\\}.$

Here, the index denotes the dimension of each orbit. We define
$O_{ij}=O_{i}\times O^{\prime}_{j}\subset\mathbb{C}^{4}$ for $i,j=0,1,2$. For
$c\in\mathbb{C}^{\times}$, we define (Tauchi)

$\displaystyle O_{12c}$ $\displaystyle:=\\{(z_{3},z_{2},z_{1},z_{4})\in
O_{12}\mid z_{3}=cz_{4}\\},$ $\displaystyle O_{21c}$
$\displaystyle:=\\{(z_{3},z_{2},z_{1},z_{4})\in O_{21}\mid z_{1}=cz_{2}\\}.$

Note that $\displaystyle\bigcup_{c\in\mathbb{C}^{\times}}O_{12c}=O_{12}$, and
$\displaystyle\bigcup_{c\in\mathbb{C}^{\times}}O_{21c}=O_{21}$.

######  Lemma 2.

  * •

$O_{ij}$ are $H_{2}$-orbits.

  * •

$O_{ij}$ remain $H_{3}$-orbits unless $(i,j)=(1,2),(2,1)$.

  * •

$O_{12c}$ and $O_{21c}$ are $H_{3}$-orbits.

  * •

$O_{22}\cup O_{21}\cup O_{20}$, $O_{12}$, $O_{02}$, $O_{11}\cup O_{10}$,
$O_{01}$, $O_{00}$ are $H_{1}$-orbits.

####  2.3.2 real

The real form $X_{\mathbb{R}}=\mathbb{C}^{2}=\mathbb{R}^{4}$ is a subset of
$O_{22}\cup O_{11}\cup O_{00}$.

######  Lemma 3.

  * •

There are three $H_{3\mathbb{R}}$-orbits
$O_{22\mathbb{R}}:=\\{(z_{1},z_{2})\in X_{\mathbb{R}}\mid z_{2}\neq 0\\},\
O_{11\mathbb{R}}:=\\{(z_{1},z_{2})\in X_{\mathbb{R}}\mid z_{2}=0,z_{1}\neq
0\\},\ O_{00\mathbb{R}}:=\\{(z_{1},z_{2})\in X_{\mathbb{R}}\mid
z_{2}=z_{1}=0\\}$ on $X_{\mathbb{R}}$.

  * •

Each orbit under $H_{3\mathbb{R}}$ is also an orbit under $H_{2\mathbb{R}}$.

  * •

The real manifold $O_{ii\mathbb{R}}$ is a real form of the complex manifold
$O_{ii}$ for $i=0,1,2$.

######  Remark 4.

The groups $H_{3}$ and $H_{3\mathbb{R}}$ and their orbit decompositions are
given by Tauchi [10].

###  2.4 D-module

In $X=\mathbb{C}^{4}$, we use coordinates $(z_{1},z_{2},z_{3},z_{4})\in X$,
and $(z_{1},z_{2},z_{3},z_{4},\zeta_{1},\zeta_{2},\zeta_{3},\zeta_{4})\in
T^{*}X$ for the cotangent bundle. The ring of differential operators on $X$ is
denoted by
$\mathcal{D}=\mathcal{D}_{X}=\mathbb{C}[z_{1},z_{2},z_{3},z_{4},\frac{\partial}{\partial
z_{1}},\frac{\partial}{\partial z_{2}},\frac{\partial}{\partial
z_{3}},\frac{\partial}{\partial z_{4}}]$. For a complex subspace
$Y:=\\{z_{2}=z_{4}=0\\}$, which is the closure of $O_{11}$, we define a
D-module

$\mathcal{B}_{Y|X}=\mathcal{H}^{2}_{[Y]}(\mathcal{O}_{X})=\mathcal{D}/(\mathcal{D}\frac{\partial}{\partial
z_{1}}+\mathcal{D}z_{2}+\mathcal{D}\frac{\partial}{\partial
z_{3}}+\mathcal{D}z_{4}).$

The characteristic variety of $\mathcal{B}_{Y|X}$ is the conormal bundle
$\\{\zeta_{1}=z_{2}=\zeta_{3}=z_{4}=0\\}=T^{*}_{Y}X\subset T^{*}X$ of $Y$, and
the multiplicity of $T^{*}_{Y}X$ is one, that is, $\mathcal{B}_{Y|X}$ is a
simple holonomic system. We denote by
$\delta(z_{2},z_{4})\in\mathcal{B}_{Y|X}$ the class represented by
$1\in\mathcal{D}$. For $l\in\mathbb{Z}_{\geq 0}$, we consider the element
$T_{l}:=(z_{1}\frac{\partial}{\partial
z_{2}})^{l}\delta(z_{2},z_{4})\in\mathcal{B}_{Y|X}$. Then

######  Lemma 5.

The annihilator $\mathcal{I}_{1l}$ of $T_{l}$ is the left ideal of
$\mathcal{D}$ generated by

$z_{1}\frac{\partial}{\partial z_{1}}-l,\ \frac{\partial^{l+1}}{\partial
z_{1}^{l+1}},\ z_{2}\frac{\partial}{\partial z_{2}}+l+1,\ z_{2}^{l+1},\
\frac{\partial}{\partial z_{3}},\ z_{4}.$ (1)

Let $\mathcal{I}_{3}$ be the left ideal generated by

$z_{1}\frac{\partial}{\partial z_{1}}+z_{2}\frac{\partial}{\partial z_{2}}+1,\
\frac{\partial}{\partial z_{3}},\ z_{4}.$ (2)

We see that $\mathcal{I}_{3}\subset\mathcal{I}_{1l}$ for all
$l\in\mathbb{Z}_{\geq 0}$. In particular,

$z_{4}\frac{\partial}{\partial z_{1}},\ z_{2}\frac{\partial}{\partial z_{3}},\
z_{1}\frac{\partial}{\partial z_{1}}+z_{2}\frac{\partial}{\partial z_{2}}+1,\
z_{3}\frac{\partial}{\partial z_{3}}+z_{4}\frac{\partial}{\partial
z_{4}}+1\in\mathcal{I}_{3}.$ (3)

We will interpret these elements as images of a Lie algebra. Let

$\rho:\mathfrak{gl}(4)=\mathfrak{gl}(4,\mathbb{C})\ni E_{ij}\mapsto-
z_{j}\frac{\partial}{\partial z_{i}}\in\mathcal{D}$

be a Lie algebra homomorphism induced from the standard representation of
$GL(4,\mathbb{C})$ on $X=\mathbb{C}^{4}$. We define the character
$\chi_{l}:\mathfrak{h}_{1}=\mathfrak{a}\oplus\mathfrak{n}\to\mathbb{C}$ by

$\chi_{l}(E_{11})=-l,\ \chi_{l}(E_{22})=l+1,\ \chi_{l}(E_{33})=0,\
\chi_{l}(E_{44})=1$

and $\chi_{l}(\mathfrak{n})=0$. The restriction of $\chi_{l}$ on
$\mathfrak{h}_{1}$ is denoted by $\chi$ since it does not depend on $l$.
Explicitly,

$\chi(E_{11}+E_{22})=1,\ \chi(E_{33}+E_{44})=1.$

Then Lemma 5 and the equation (3) shows that

$\begin{array}[]{ccc}(\rho-\chi)(\mathfrak{h}_{3})&\subset&\mathcal{I}_{3}\\\
\cap&&\cap\\\
(\rho-\chi_{l})(\mathfrak{h}_{1})&\subset&\mathcal{I}_{1l}\end{array}$ (4)

Finally, we define left D-modules
$\mathcal{M}_{1l}=\mathcal{D}/\mathcal{I}_{1l}$ and
$\mathcal{M}_{3}=\mathcal{D}/\mathcal{I}_{3}$.

######  Lemma 6.

  * •

We have a D-module isomorphism $\displaystyle\mathcal{M}_{1l}\ni P\mapsto
P(z_{1}\frac{\partial}{\partial z_{2}})^{l}\in\mathcal{B}_{Y|X}$.

  * •

The natural D-homomorphism
$\displaystyle\mathcal{M}_{3}\rightarrow\mathcal{M}_{1l}$ is surjective.

Let $\mathcal{B}_{X_{\mathbb{R}}}$ be sections of hyperfunctions on
$X_{\mathbb{R}}$. From the diagram 4, we conclude the following on solutions
of differential equations.

######  Theorem 7.

$\begin{array}[]{ccc}\\{u\in\mathcal{B}_{X_{\mathbb{R}}}\mid Pu=0,\forall
P\in(\rho-\chi)(\mathfrak{h}_{3})\\}&\supset&\\{u\in\mathcal{B}_{X_{\mathbb{R}}}\mid
Pu=0,\forall P\in\mathcal{I}_{3}\\}\\\ \cup&&\cup\\\
\displaystyle\bigoplus_{l\in\mathbb{Z}_{\geq
0}}\\{u\in\mathcal{B}_{X_{\mathbb{R}}}\mid Pu=0,\forall
P\in(\rho-\chi_{l})(\mathfrak{h}_{1})\\}&\supset&\displaystyle\bigoplus_{l\in\mathbb{Z}_{\geq
0}}\\{u\in\mathcal{B}_{X_{\mathbb{R}}}\mid Pu=0,\forall
P\in\mathcal{I}_{1l}\\}\end{array}$

######  Remark 8.

The Lie algebra homomorphism $\rho$ induces the associative algebra
homomorphism

$\tilde{\rho}:U(\mathfrak{gl}(4))\rightarrow\\{P\in\mathcal{D}\mid[P,z_{1}\frac{\partial}{\partial
z_{1}}+z_{2}\frac{\partial}{\partial z_{2}}+z_{3}\frac{\partial}{\partial
z_{3}}+z_{4}\frac{\partial}{\partial z_{4}}]=0\\}$

from the universal enveloping algebra of the Lie algebra $\mathfrak{gl}(4)$
onto the subalgebra of $\mathcal{D}$ of homogeneous degree zero. Note that the
map $\tilde{\rho}$ is not injective; for example, a non-zero element
$E_{14}E_{32}-E_{12}E_{34}\in U(\mathfrak{gl}(4))$ belongs to the kernel of
$\tilde{\rho}$. This is one of the reasons why invariance under a smaller Lie
algebra extends to that under a larger Lie algebra.

###  2.5 Relatively invariant distribution

Tauchi defines [10] a distribution

$T_{2}^{l}=z_{1}^{l}\frac{\partial^{l}}{\partial
z_{2}^{l}}\delta(z_{2},\overline{z}_{2})=(z_{1}\frac{\partial}{\partial
z_{2}})^{l}\delta(z_{2},\overline{z}_{2})$ (5)

on the real manifold $X_{\mathbb{R}}=\mathbb{C}^{2}=\mathbb{R}^{4}$. The
support of $T_{2}^{l}$ is the closure of $O_{11\mathbb{R}}$.

######  Lemma 9.

$T_{2}^{l}\in\\{u\in\mathcal{B}_{X_{\mathbb{R}}}\mid Pu=0,\forall
P\in\mathcal{I}_{1l}\\}=Hom_{\mathcal{D}}(\mathcal{M}_{1l},\mathcal{B}_{X_{\mathbb{R}}})$.
Especially, $T_{2}^{l}$ is relatively invariant under $\mathfrak{h}_{1}$ with
the character $\chi_{l}$.

##  3 $n=3$

###  3.1 D-module

In $X=\mathbb{C}^{6}$, we use coordinates
$(z_{1},z_{2},z_{3},z_{4},z_{5},z_{6})\in X$. The ring of differential
operators on $X$ is denoted by
$\mathcal{D}=\mathcal{D}_{X}=\mathbb{C}[z_{1},z_{2},z_{3},z_{4},z_{5},z_{6},\frac{\partial}{\partial
z_{1}},\frac{\partial}{\partial z_{2}},\frac{\partial}{\partial
z_{3}},\frac{\partial}{\partial z_{4}},\frac{\partial}{\partial
z_{5}},\frac{\partial}{\partial z_{6}}]$. Let $Y=\\{z_{3}=z_{6}=0\\}\subset
X=\mathbb{C}^{6}$, and define a simple holonomic D-module $\mathcal{B}_{Y|X}$
by

$\mathcal{B}_{Y|X}=\mathcal{H}^{2}_{[Y]}(\mathcal{O}_{X})=\mathcal{D}/(\mathcal{D}\frac{\partial}{\partial
z_{1}}+\mathcal{D}\frac{\partial}{\partial
z_{2}}+\mathcal{D}z_{3}+\mathcal{D}\frac{\partial}{\partial
z_{4}}+\mathcal{D}\frac{\partial}{\partial z_{5}}+\mathcal{D}z_{6}).$

The standard generator represented by $1\in\mathcal{D}$ is denoted by
$\delta(z_{3},z_{6})\in\mathcal{B}_{Y|X}$. We define a section of
$\mathcal{B}_{Y|X}$.

$\displaystyle T_{l}$ $\displaystyle=(z_{2}\frac{\partial}{\partial
z_{3}}+z_{4}\frac{\partial}{\partial z_{5}})^{l}z_{2}z_{5}\delta(z_{3},z_{6})$
(6) $\displaystyle=z_{2}^{l}\left(z_{2}z_{5}\frac{\partial^{l}}{\partial
z_{3}^{l}}+lz_{4}\frac{\partial^{l-1}}{\partial
z_{3}^{l-1}}\right)\delta(z_{3},z_{6})$ (7)
$\displaystyle=(z_{2}z_{5}-z_{3}z_{4})(z_{2}\frac{\partial}{\partial
z_{3}})^{l}\delta(z_{3},z_{6}).$

######  Lemma 10.

The annihilator ideal $\mathcal{I}_{1l}$ of $T_{l}$ is generated by the
following elements:

$\displaystyle\displaystyle\frac{\partial}{\partial z_{1}},\ z_{6},\
z_{2}\frac{\partial}{\partial z_{2}}+z_{3}\frac{\partial}{\partial z_{3}},\
z_{2}\frac{\partial}{\partial z_{2}}+z_{4}\frac{\partial}{\partial
z_{4}}-1-l,\ z_{2}\frac{\partial}{\partial
z_{4}}+z_{3}\frac{\partial}{\partial z_{5}},$
$\displaystyle\displaystyle\frac{\partial^{2}}{\partial z_{4}^{2}},\
\frac{\partial^{2}}{\partial z_{5}^{2}},\ \frac{\partial^{2}}{\partial
z_{4}\partial z_{5}},\ z_{4}\frac{\partial}{\partial
z_{4}}+z_{5}\frac{\partial}{\partial z_{5}}-1.$

###### Proof.

We define a left ideal $\mathcal{I}^{\prime}_{l}$ of $\mathcal{D}$ generated
by

$\displaystyle\displaystyle\frac{\partial}{\partial z_{1}},\
\frac{\partial}{\partial z_{4}},\ \frac{\partial}{\partial z_{5}},\ z_{6},\
z_{2}\frac{\partial}{\partial z_{2}}-l-1,\ \frac{\partial^{l+2}}{\partial
z_{2}^{l+2}},\ z_{3}\frac{\partial}{\partial z_{3}}+l,\ z_{3}^{l}.$

Then $\mathcal{M}^{\prime}_{l}:=\mathcal{D}/\mathcal{I}^{\prime}_{l}$ forms a
simple holonomic system. Let $\mathcal{I}^{\prime\prime}$ be the left ideal
generated by elements in the statement of Lemma other than
$z_{2}\frac{\partial}{\partial z_{4}}+z_{3}\frac{\partial}{\partial z_{5}}$.
We see that the map $\mathcal{D}\in
P\mapsto(Pz_{4},Pz_{5})\in\mathcal{D}^{\oplus 2}$ induces a D-isomomorphism

$\mathcal{D}/\mathcal{I}^{\prime\prime}\ni
P\mapsto(Pz_{4},Pz_{5})\in\mathcal{M}^{\prime}_{l+1}\oplus\mathcal{M}^{\prime}_{l}.$

Note that the image of this map is the sum of two simple holonomic systems.
Therefore,
$\mathcal{D}/(\mathcal{I}^{\prime\prime}+D(z_{2}\frac{\partial}{\partial
z_{4}}+z_{3}\frac{\partial}{\partial z_{5}}))$ is a simple holonomic system.
It is evident that the annihilator contains
$\mathcal{I}^{\prime\prime}+D(z_{2}\frac{\partial}{\partial
z_{4}}+z_{3}\frac{\partial}{\partial z_{5}})$; then, maximality implies
equality. ∎

We define a complex Lie subalgebra
$\mathfrak{h}_{1}\subset\mathfrak{gl}(6,\mathbb{C})$ by

$\mathfrak{h}_{1}=\bigoplus_{j=1}^{6}\mathbb{C}E_{1j}\oplus\bigoplus_{i=2}^{6}\mathbb{C}E_{i6}\oplus\mathbb{C}(E_{22}+E_{33})\oplus\mathbb{C}(E_{44}+E_{55})\oplus\mathbb{C}(E_{22}+E_{44})\oplus\mathbb{C}(E_{42}+E_{53}).$

In terms of a Lie algebra homomorphism

$\rho:\mathfrak{gl}(6,\mathbb{C})\ni E_{ij}\mapsto-
z_{j}\frac{\partial}{\partial z_{i}}\in\mathcal{D},$

and the character $\chi_{l}:\mathfrak{h}_{1}\rightarrow\mathbb{C}$ defined by

$\chi_{l}(E_{22}+E_{33})=0,\ \chi_{l}(E_{22}+E_{44})=-1-l,\
\chi_{l}(E_{44}+E_{55})=-1,$

we can rephrase Lemma 10 as
$(\rho-\chi_{l})(\mathfrak{h}_{1})\subset\mathcal{I}_{1l}$.

We define a Lie subalgebra $\mathfrak{h}_{3}\subset\mathfrak{h}_{1}$ by

$\mathfrak{h}_{3}=\mathbb{C}(E_{11}+E_{22}+E_{33})\oplus\mathbb{C}(E_{44}+E_{55}+E_{66})\oplus\mathbb{C}(E_{42}+E_{53})\oplus\mathbb{C}(E_{15}+E_{26})\oplus\mathbb{C}E_{13}\oplus\mathbb{C}E_{46}.$

and the left ideal $\mathcal{I}_{3}\subset\mathcal{D}$ generated by

$z_{1}\frac{\partial}{\partial z_{1}}+z_{2}\frac{\partial}{\partial
z_{2}}+z_{3}\frac{\partial}{\partial z_{3}},z_{4}\frac{\partial}{\partial
z_{4}}+z_{5}\frac{\partial}{\partial z_{5}}+z_{6}\frac{\partial}{\partial
z_{6}},z_{5}\frac{\partial}{\partial z_{1}}+z_{6}\frac{\partial}{\partial
z_{2}},z_{2}\frac{\partial}{\partial z_{4}}+z_{3}\frac{\partial}{\partial
z_{5}},z_{3}\frac{\partial}{\partial z_{1}},z_{6}\frac{\partial}{\partial
z_{4}}.$

Then $\mathfrak{h}_{3}\subset\mathfrak{h}_{1l}$ for all $l\in\mathbb{Z}_{\geq
0}$, and
$(\rho-\chi_{l})(\mathfrak{h}_{3})\subset\mathcal{I}_{3}\subset\mathcal{I}_{1l}$.
We have the same diagram (4).

Now we move on to solutions of these D-modules. We choose a real form
$X_{\mathbb{R}}=\\{(z_{1},z_{2},z_{3},\overline{z_{1}},\overline{z_{2}},\overline{z_{3}})\mid(z_{1},z_{2},z_{3})\in\mathbb{C}^{3}\\}$
of $X=\mathbb{C}^{6}$. Denote by $\mathcal{B}_{X_{\mathbb{R}}}$ the set of
hyperfunctions on $X_{\mathbb{R}}$.

######  Lemma 11.

Tauchi’s distribution
$T_{0}^{l}=(\overline{z}_{1}\frac{\partial}{\partial\overline{z}_{2}}+z_{2}\frac{\partial}{\partial
z_{3}})^{l}|z_{2}|^{2}\delta(z_{3},\overline{z}_{3})\in\mathcal{B}_{X_{\mathbb{R}}}$
is a solution of $\mathcal{D}/\mathcal{I}_{1l}$.

This shows $T_{0}^{l}$ is invariant under $\mathfrak{h}_{3\mathbb{R}}$. Orbit
decomposition is similar to $n=2$, so we omit its description.

## References

  * [1] Ryoshi Hotta, Kiyoshi Takeuchi and Toshiyuki Tanisaki, D-modules, perverse sheaves, and representation theory, Progr. Math., 236 Birkhäuser Boston, Inc., Boston, MA, 2008, xii+407 pp.
  * [2] Masaki Kashiwara, D-modules and microlocal calculus, translated from the 2000 Japanese original by Mutsumi Saito, Transl. Math. Monogr., 217, Amer. Math. Soc., Providence, RI, 2003.
  * [3] M. T. Kosters and G. van Dijk, Spherical distributions on the pseudo-Riemannian space $SL(n,\mathbb{R})/GL(n-1,\mathbb{R})$, J. Funct. Anal. 68 (1986), no. 2, 168–213.
  * [4] Atsutaka Kowata, On the construction of spherical hyperfunctions on $\mathbb{R}^{p+q}$, Hiroshima Math. J. 21 (1991), 301–334.
  * [5] Atsutaka Kowata, Spherical hyperfunctions on the tangent space of symmetric spaces, Hiroshima Math. J. 21 (1991), 401–418
  * [6] Thierry Levasseur and J. Toby Stafford, Invariant differential operators on the tangent space of some symmetric spaces, Annales de l’institut Fourier, tome 49, no 6 (1999), p. 1711–1741.
  * [7] Hiroyuki Ochiai, Invariant functions on the tangent space of a rank one semisimple symmetric space, J. Fac. Sci. Univ. Tokyo Sect. IA Math.39(1992), no.1, 17–31.
  * [8] Hiroyuki Ochiai, Invariant distributions on a non-isotropic pseudo-Riemannian symmetric space of rank one, Indag. Math. (N.S.) 16 (2005), no. 3-4, 631–638.
  * [9] Toshiyuki Tanisaki, D-modules and representation theory.(English summary)Lie theory and representation theory, 177–219. Surv. Mod. Math., 2 International Press, Somerville, MA, 2012.
  * [10] Taito Tauchi, Dimension of the space of intertwining operators from degenerate principal series representations, Selecta Math. (N.S.) 24 (2018), no. 4, 3649–3662.
  * [11] Taito Tauchi, Relationship between orbit decomposition on the flag varieties and multiplicities of induced representations, Proc. Japan Acad. Ser. A Math. Sci. 95 (2019), no. 7, 75–79.
  * [12] Taito Tauchi, Multiplicity of a degenerate principal series for homogeneous spaces with infinite orbits, Proc. Amer. Math. Soc. 150 (2022), no. 2, 849–856.
  * [13] Gerrit van Dijk, Invariant eigendistributions on the tangent space of a rank one semisimple symmetric space, Math. Ann. 268 (1984), no. 3, 405–416.
  * [14] Gerrit van Dijk, On generalized Gel’fand pairs, Proc. Japan Acad. Ser. A Math. Sci. 60 (1984), no. 1, 30–34.
  * [15] Gerrit van Dijk, On a class of generalized Gel’fand pairs, Math. Z. 193 (1986), no. 4, 581–593.
  * [16] Gerrit van Dijk, Harmonic analysis on rank one symmetric spaces, Lecture Notes in Phys., 261 Springer-Verlag, Berlin, 1986, 244–252.
  * [17] Gerrit van Dijk, Gelfand pairs and beyond, COE Lect. Note, 11, Math-for-Ind. (MI) Lect. Note Ser. Kyushu University, Faculty of Mathematics, Fukuoka, 2008, ii+60 pp.
  * [18] Gerrit van Dijk, $(GL(n+1,\mathbb{R}),GL(n,\mathbb{R}))$ is a generalized Gelfand pair, Russ. J. Math. Phys. 15 (2008), no. 4, 548–551.
  * [19] Gerrit van Dijk, $(U(p,q),U(p-1,q))$ is a generalized Gelfand pair, Math. Z. 261 (2009), no. 3, 525–529.
  * [20] Gerrit van Dijk, Introduction to Harmonic Analysis and Generalized Gelfand Pairs, Walter de Gruyter, Berlin New York, 2009.
  * [21] G. van Dijk and M. Poel, The Plancherel formula for the pseudo-Riemannian space $SL(n,\mathbb{R})/GL(n-1,\mathbb{R})$ , Compositio Math. 58 (1986), no. 3, 371–397.
  * [22] G. van Dijk and M. Poel, The irreducible unitary $GL(n-1,\mathbb{R})$-spherical representations of $SL(n,\mathbb{R})$, Compositio Math. 73 (1990), no. 1, 1–30.

Hiroyuki Ochiai
Institute of Mathematics for Industry, Kyushu University
744 Motooka Fukuoka 819-0395 Japan
<EMAIL_ADDRESS>