$\displaystyle|\langle\nabla_{{w}_{1}}{{\mathcal{M}}_{\mathcal{D}}({\bm{w}})}-\nabla_{{w}_{1}}{{\mathcal{M}}_{\hat{\mathcal{D}}}({\bm{w}})},{w}_{1}\rangle|\leqslant\epsilon$
$\displaystyle|\langle\nabla_{{\bm{w}}_{2}}{{\mathcal{M}}_{\mathcal{D}}({\bm{w}})}-\nabla_{{\bm{w}}_{2}}{{\mathcal{M}}_{\hat{\mathcal{D}}}({\bm{w}})},{\bm{w}}_{2}\rangle|\leqslant\epsilon$
(41)

Furthermore, if the true feature at initialization is dominant i.e.,
$|{{w}^{(0)}_{1}}|\geqslant 0.5$ and if
$\|{{\bm{w}}^{(t)}_{2}}\|_{2}\geqslant\tau/2$, then for
$\eta={\mathcal{O}}(c_{1}/c_{1}^{\prime 2})$ the norm along ${\bm{w}}_{2}$
shrinks by a constant factor:
$\|{{\bm{w}}^{(t+1)}_{2}}\|_{2}^{2}\leqslant(1-{\mathcal{O}}(c_{1}^{2}/c_{1}^{\prime
2}))\|{{\bm{w}}^{(t)}_{2}}\|_{2}^{2}$. This continues until
$\|{{\bm{w}}^{(t)}_{2}}\|_{2}<\tau/2$ after which it stabilizes and
$\|{{\bm{w}}^{(t)}_{2}}\|_{2}\leqslant\tau$ is guaranteed in subsequent
iterations.

In Theorem A.12 we allow for some error $\epsilon$ in approximating the true
gradient with the finite sample approximation along directions ${w}_{1}$ and
${\bm{w}}_{2}$. Note, that this error is much smaller $O(c_{1}\tau^{2})$, than
the final error tolerated on the norm of ${\bm{w}}_{2}$ which is $\tau$, when
$\tau<0.5$. Essentially, using Lemma C.1 in [6], it is easy to show that
Equation 41 holds with probability $1-\delta$ if
$n=\tilde{{\mathcal{O}}}(\frac{d\|\tilde{\Sigma}\|_{\mathrm{op}}}{\epsilon^{2}}\log(1/\delta))$,
where $\tilde{{\mathcal{O}}}$ hides logarithmic factors in $1/\epsilon^{2}$.
This, allows us to make the following replacements for our expressions in
Lemma A.11 which considered the population version:

$\displaystyle\langle\frac{\partial{\mathcal{M}}_{\mathcal{D}}({\bm{w}})}{\partial{w}_{1}},{w}_{1}\rangle>0\;\;\longleftrightarrow\;\;\langle\frac{\partial{\mathcal{M}}_{\hat{\mathcal{D}}}({\bm{w}})}{\partial{w}_{1}},{w}_{1}\rangle>-\epsilon$
$\displaystyle\langle\nabla_{{\bm{w}}_{2}}{\mathcal{M}}_{\mathcal{D}}({\bm{w}}),{\bm{w}}_{2}\rangle<-c_{1}\cdot\|{\bm{w}}_{2}\|_{2}^{2}\;\;\longleftrightarrow\;\;\langle\nabla_{{\bm{w}}_{2}}{\mathcal{M}}_{\hat{\mathcal{D}}}({\bm{w}}),{\bm{w}}_{2}\rangle<-c_{1}\cdot\|{\bm{w}}_{2}\|_{2}^{2}+\epsilon$
$\displaystyle\|\nabla_{{\bm{w}}_{2}}{\mathcal{M}}_{\mathcal{D}}({\bm{w}})\|_{2}^{2}\leqslant{c_{1}^{\prime}}^{2}\|{\bm{w}}_{2}\|_{2}^{2}\;\;\longleftrightarrow\;\;\|\nabla_{{\bm{w}}_{2}}{\mathcal{M}}_{\hat{\mathcal{D}}}({\bm{w}})\|_{2}^{2}\leqslant
2{c_{1}^{\prime}}^{2}\|{\bm{w}}_{2}\|_{2}^{2}+2\epsilon^{2}$

Thus, when $\eta={\mathcal{O}}\left(\frac{c_{1}}{c_{1}^{\prime 2}}\right)$,
there is a constant factor decrease in $\|\tilde{{\bm{w}}}\|_{2}^{2}$:

$\displaystyle\|\tilde{{\bm{w}}}_{2}^{(t+1)}\|_{2}^{2}\leqslant(1-{\mathcal{O}}(c_{1}^{2}/c_{1}^{\prime
2}))\|{\bm{w}}_{2}^{(t)}\|_{2}^{2}+2\eta^{2}\epsilon^{2}+2\eta\epsilon$

On the other hand since $\eta\epsilon$ is small, there is a significant
increase in $|\tilde{{w}}_{1}|$:

$\displaystyle\|\tilde{{w}}_{1}^{(t+1)}\|_{2}^{2}\geqslant\|{w}^{(t)}\|_{2}^{2}-2\eta\epsilon$

Thus, any decrease in $|{w}_{1}|$ is relatively much smaller compared to
decrease in $\|{\bm{w}}_{2}\|_{2}$. After re-normalization we get:

$\displaystyle\|{{\bm{w}}_{2}}^{(t+1)}\|_{2}^{2}\leqslant(1-{\mathcal{O}}(c_{1}^{2}/c_{1}^{\prime
2}))\|{\bm{w}}_{2}^{(t)}\|_{2}^{2}$

Given the constant factor reduction, the number of iterations needed to drive
$\|{{\bm{w}}_{2}}^{(t+1)}\|_{2}$ to be less than $\tau$ is
${\mathcal{O}}(\log(1/\tau))$. Furthermore, in the finite sample case, we need
$n=\tilde{{\mathcal{O}}}(\frac{1}{\tau^{4}}\log(1/\delta))$ samples for the
$\epsilon-$approximation in Equation 41 to hold where
$\epsilon={\mathcal{O}}(c_{1}\tau^{2})$. We can now combine this requirement
on the sample size with the lower bound on $n$ from Corollary A.7 needed to
maintain the initialization conditions in Lemma 5.2 with
$\beta{w}_{1}\geqslant K\alpha$, to get
$n\raisebox{-3.69899pt}{~{}\shortstack{$>$ \\\ [-0.07cm]
$\sim$}}~{}\frac{\beta^{2}+\log(1/\delta)\cdot\|\tilde{\Sigma}\|_{\mathrm{op}}+\|\tilde{\Sigma}\|_{*}}{\gamma^{2}\mathrm{erfc}\left(K\alpha/\sqrt{2\sigma_{\mathrm{min}}(\tilde{\Sigma})}\right)^{2}}+\frac{\log{1/\delta}}{\tau^{4}}$.
This completes the proof for Theorem 5.3.

##  Appendix B Additional Details for Experiments in Section 4

In this section, we first present technical details useful for reproducing our
empirical results in Section 4. We also take a note of the computational
resources needed to run the experiments in this paper. Then we present some
tables containing absolute test performance values for some of the plots in
Section 4 where we plotted the relative performance of each method with
respect to empirical risk minimization (ERM). Finally, we present some
preliminary results on Imagenet classification benchmark.

Hyperparameter details. The hyperparameters used to train ADA [61], ME-ADA
[72], and adversarial training (FGSM) [13] are identical to those reported in
the original works we cite. For label smoothing, we tune the smoothing
parameter on the validation set and find $\epsilon=0.6$ to be optimal for the
smaller dataset of CIFAR-100-2k, and $\epsilon=0.2$ yields the best results on
other datasets. For RCAD we have two hyperparameters $\alpha,\lambda$ and find
that $\alpha=0.5,\lambda=0.02$ gives good performance over all benchmarks
except CIFAR-100-2k for which we use $\alpha=1.0,\lambda=0.1$.

Unless specified otherwise, we train all methods using the ResNet-18 [18]
backbone, and to accelerate training loss convergence we clip gradients in the
$l_{2}$ norm (at $1.0$) [70, 17]. We train all models for $200$ epochs and use
SGD with an initial learning rate of $0.1$ and Nesterov momentum of $0.9$, and
decay the learning rate by a factor of $0.1$ at epochs $100,150$ and $180$
[9]. We select the model checkpoint corresponding to the epoch with the best
accuracy on validation samples as the final model representing a given
training method. For all datasets (except CIFAR-100-2k and CIFAR-100-10k for
which we used $32$ and $64$ respectively) the methods were trained with a
batch size of $128$. For our experiments involving the larger network of Wide
ResNet 28-10, we use the optimization hyperparameters mentioned in Zagoruyko
and Komodakis,  [66].

Computational resources. To run any experiment we used at most two NVIDIA
GEFORCE GTX 1080Ti GPU cards. As we mention in the main paper, any run of our
method RCAD takes at most $30\%$ more per-batch processing time than ERM with
data augmentation on a given dataset. All runs of RCAD with ResNet-18 backbone
on CIFAR-100-2k, CIFAR-100-10k, CIFAR-100, CIFAR-10 and SVHN take less than
$20$ hours, while Tiny Imagenet takes about $36$ hours.

Absolute test performance values. In Table 2, we plot the test performance
values for our method RCAD which maximizes entropy on self-generated
perturbations, and adversarial baselines which minimize cross-entropy on
adversarial examples. The plot in Figure 3(a) is derived from the results in
Table 2. Table 3(a) compares the performance of RCAD with the most competitive
adversarial baseline ME-ADA, when both methods are used in combination with
the label smoothing regularizer. These results are also presented in Figure
3(a) and discussed in Section 4.2. For the plot in Figure 4a comparing
baselines ERM and ME-ADA to our method RCAD with the larger network Wide
ResNet 28-10 [66], we show the test accuracy values on CIFAR-100(-2k/10k) in
Table 3(b). Finally, Table 4 presents the test accuracy of each method on the
CIFAR-100 test set, when trained on training datasets of varying sizes. Figure
3(b) plots the same results relative to ERM. The trends observed here and in
Figure 3(b) are discussed in Section 4.3. In all the above, we show the mean
accuracy and $95\%$ confidence interval evaluated over $10$ independent runs.

Dataset | ERM | FGSM | ADA | ME-ADA | RCAD
---|---|---|---|---|---
CIFAR-100-2k | $28.5\pm 0.12$ | $27.1\pm 0.08$ | $28.7\pm 0.12$ | $29.4\pm 0.11$ | $\bf{30.4\pm 0.09}$
CIFAR-100-10k | $60.4\pm 0.09$ | $57.6\pm 0.09$ | $60.0\pm 0.10$ | $60.4\pm 0.08$ | $\bf{61.9\pm 0.06}$
Tiny Imagenet | $66.4\pm 0.07$ | $62.4\pm 0.07$ | $66.5\pm 0.08$ | $66.9\pm 0.10$ | $\bf{67.3\pm 0.07}$
CIFAR-10 | $95.2\pm 0.04$ | $93.1\pm 0.04$ | $95.2\pm 0.03$ | $\bf{95.6\pm 0.05}$ | $\bf{95.7\pm 0.04}$
CIFAR-100 | $76.4\pm 0.05$ | $73.2\pm 0.05$ | $76.5\pm 0.05$ | $77.1\pm 0.07$ | $\bf{77.3\pm 0.05}$
SVHN | $97.5\pm 0.02$ | $95.4\pm 0.02$ | $97.5\pm 0.02$ | $97.4\pm 0.04$ | $\bf{97.6\pm 0.02}$
Table 2: RCAD consistently outperforms adversarial baselines. In Section 4.2
we investigated the performance of RCAD against baseline methods that use the
adversarially generated perturbations differently, i.e., methods that reduce
cross-entropy loss on adversarial examples. This table presents the test
accuracies for RCAD and other baselines with $95\%$ confidence interval
computed over 10 independent runs. Figure 3(a) plots the results reported here
by looking at the performance improvement of each method over the ERM
baseline.

Dataset | ME-ADA+LS | RCAD+LS
---|---|---
CIFAR-100-2k | $30.5\pm 0.13$ | $\bf{32.2\pm 0.11}$
CIFAR-100-10k | $61.15\pm 0.07$ | $\bf{63.1\pm 0.08}$
Tiny Imagenet | $67.1\pm 0.08$ | $\bf{67.9\pm 0.07}$
CIFAR-10 | $\bf{95.65\pm 0.04}$ | $\bf{95.7\pm 0.03}$
CIFAR-100 | $77.8\pm 0.06$ | $\bf{79.1\pm 0.06}$
SVHN | $97.42\pm 0.04$ | $\bf{97.6\pm 0.02}$
(a)

Dataset | ERM | ME-ADA | RCAD
---|---|---|---
C100-2k | $30.2\pm 0.10$ | $31.8\pm 0.07$ | $\bf{32.3\pm 0.08}$
C100-10k | $62.5\pm 0.07$ | $62.6\pm 0.08$ | $\bf{62.9\pm 0.05}$
C100 | $81.2\pm 0.07$ | $81.0\pm 0.07$ | $\bf{81.4\pm 0.06}$
(b)

Table 3: (Left) RCAD outperforms adversarial baselines even when combined with existing regularizers. We compare the performance of RCAD and ME-ADA when both methods are trained with label smoothing (smoothing parameter $\epsilon=0.2$). We find that label smoothing improves the performance of both methods, but the benefit of RCAD still persists. Figure 3(a) also plots the same results, relative to the performance of ERM. (Right) RCAD can outperform baselines when used with larger backbones. Here, we train our method RCAD and baselines ERM, ME-ADA with the larger architecture Wide ResNet 28-10 on CIFAR-100 (C100), CIFAR-100-2k (C100-2k) and CIFAR-100-10k (C100-10k) benchmarks. We find that while the performance gains of RCAD over ME-ADA and ERM are higher with the ResNet-18 backbone, RCAD continues to outperform them here as well. Figure 4a also plots the same results, relative to the performance of ERM.  Dataset size | ERM | LS | FGSM | ME-ADA | RCAD
---|---|---|---|---|---
500 | $11.9\pm 0.06$ | $13.5\pm 0.08$ | $13.2\pm 0.06$ | $13.1\pm 0.06$ | $\bf{14.8\pm 0.08}$
1,000 | $18.9\pm 0.07$ | $19.7\pm 0.07$ | $18.4\pm 0.08$ | $18.6\pm 0.08$ | $\bf{21.3\pm 0.05}$
2,000 | $28.5\pm 0.05$ | $29.2\pm 0.07$ | $27.1\pm 0.07$ | $29.4\pm 0.05$ | $\bf{30.4\pm 0.05}$
5,000 | $52.2\pm 0.07$ | $52.7\pm 0.06$ | $50.9\pm 0.07$ | $51.4\pm 0.06$ | $\bf{53.6\pm 0.05}$
10,000 | $60.4\pm 0.07$ | $61.5\pm 0.06$ | $57.6\pm 0.06$ | $60.4\pm 0.07$ | $\bf{61.9\pm 0.06}$
50,000 | $76.4\pm 0.07$ | $77.4\pm 0.08$ | $73.2\pm 0.08$ | $77.1\pm 0.07$ | $\bf{77.7\pm 0.06}$
Table 4: RCAD is more effective than other existing regularizers in low data
regime. We evaluate the effectiveness of RCAD over other baseline regularizers
as we decrease the size of the training data for CIFAR-100. We find that the
performance gain from using RCAD is higher as training data size reduces,
possibly since RCAD is more robust to spurious correlations in the training
data (compared to other baselines), and deep models are more vulnerable to
such correlations in the low data regime [67]. Figure 3(b) also plots the same
results, relative to the performance of ERM.

Preliminary results on Imagenet classification. We take a ResNet-50 [18] model
pretrained on Imagenet [8] and measure its _top-1_ and _top-5_ test accuracy.
Next, we tune this pretrained ResNet-50 model, using the optimization
hyperparameters from Yun et al.,  [65]. We refer to the model produced by this
method as “Finetuned with ERM”. We compare the test accuracies for these two
models on the Imagenet test set with the test accuracy of the model that is
obtained by finetuning the pretrained ResNet-50 model with the RCAD objective
in Equation 2 (“Finetuned with RCAD”). Results for this experiment can be
found in Table 5. We do not finetune RCAD hyperparameters $\alpha,\lambda$ on
Imagenet, and retain the ones used for the TinyImagenet benchmark. We faced
issues running experiments for the competitive but expensive adversarial
baseline ME-ADA since the GPU memory needed to run ME-ADA on Imagenet exceeded
the limit on our GPU cards. Mainly for these two reasons, we note that our
results on the Imagenet benchmark are preliminary. Nevertheless, the results
are promising since we see an improvement in test accuracy when we finetune
the pretrained model with our RCAD objective as opposed to the standard cross-
entropy loss as part of the ERM objective. The improvement is more significant
for top-1 test accuracy.

Test Accuracy | Pretrained ResNet-50 | Finetuned with ERM | Finetuned with RCAD
---|---|---|---
top-1 | $69.81\pm 0.07\%$ | $77.45\pm 0.05\%$ | $77.76\pm 0.06\%$
top-5 | $89.24\pm 0.06\%$ | $93.21\pm 0.04\%$ | $93.29\pm 0.05\%$
Table 5: Preliminary results on Imagenet classification. We compare the
performance of a pretrained ResNet-50 model, with the models obtained by
finetuning the pretrained model with either the cross-entropy loss (finetuned
with ERM) or the RCAD objective (finetuned with RCAD). We show the mean
accuracy and $95\%$ confidence interval evaluated over $7$ independent runs.

##  Appendix C RCAD can Improve Robustness to Adversarial/Natural Distribution
Shifts

Typically, methods for handling distribution shift are different from methods
for improving test accuracy. Prior work has found that increasing robustness
to distribution shifts tends to be at odds with increasing test accuracy:
methods that are more robust often achieve lower test accuracy, and methods
that achieve higher test accuracy tend to be less robust [47, 69, 60]. While
the main aim of our experiments is to show that RCAD improves test accuracy,
our next set of experiments investigate whether RCAD is any more robust to
distribution shift than baseline methods.

For these experiments on distribution shift, we use the exact same
hyperparameters as in the previous experiments. Better results are likely
achievable by tuning the method for performance on these robustness
benchmarks. By reporting results using the exact same hyperparameters, we
demonstrate that the same method might both achieve high in-distribution
performance and out-of-distribution performance.

Method | CIFAR-100-2k | CIFAR-100 | CIFAR-10
---|---|---|---
| Clean | FGSM Attack | Clean | FGSM Attack | Clean | FGSM Attack
ERM |  28.5 $\pm$ 0.12 |  24.7 $\pm$ 0.10 |  76.4 $\pm$ 0.05 |  67.4 $\pm$ 0.07 |  95.2 $\pm$ 0.04 |  88.3 $\pm$ 0.03
Adv. Training |  27.1 $\pm$ 0.08 |  27.2 $\pm$ 0.08 |  73.2 $\pm$ 0.05 |  73.1 $\pm$ 0.04 |  93.1 $\pm$ 0.04 |  92.9 $\pm$ 0.04
ADA |  28.7 $\pm$ 0.12 |  27.0 $\pm$ 0.08 |  76.5 $\pm$ 0.05 |  72.7 $\pm$ 0.05 |  95.2 $\pm$ 0.03 |  88.1 $\pm$ 0.04
ME-ADA |  29.4 $\pm$ 0.11 |  27.6 $\pm$ 0.11 |  77.1 $\pm$ 0.07 | 74.8 $\pm$ 0.03 | 95.6 $\pm$ 0.05 | 93.1 $\pm$ 0.04
RCAD | 30.4 $\pm$ 0.09 | 28.1 $\pm$ 0.10 | 77.3 $\pm$ 0.05 |  74.5$\pm$ 0.04 | 95.7 $\pm$ 0.04 |  93.0$\pm$ 0.03
Table 6: Robustness to adversarial perturbations compared against in-
distribution test accuracy: Clean (evaluation on iid test data) and Robust
test accuracies of adversarial methods (adversarial training (FGSM [13]), ADA
and ME-ADA), ERM and RCAD when the adversary carries out attacks using the
Fast Gradient Sign (FGSM) method when the strength of the attack in $l_{1}$
norm is $0.05$. We show the mean accuracy and $95\%$ confidence interval
evaluated over $10$ independent runs.

###  C.1 Robustness to adversarial attacks

We first look at robustness to adversarial attacks, using FGSM [13] as the
attack method. The conventional approach to fending off adversarial attacks is
adversarial training, wherein the training objective exactly matches the
testing objective. Thus, adversarial training represents the “gold standard”
for performance on this evaluation. We compare the adversarial robustness of
RCAD, adversarial training, ADA, and ME-ADA in Table 6. Not only does our
method achieve higher (clean) test accuracy than adversarial training on all
datasets, but surprisingly it also achieves higher robust test accuracy on the
harder CIFAR-100-2k benchmark where the clean test accuracy of RCAD is $+3\%$
greater than adversarial training, and robust test accuracy is $+0.5\%$ better
than ME-ADA. Although, we find that adversarial baselines achieve better
robust test accuracy when the training data is sufficient (e.g., CIFAR-100 and
CIFAR-10 in Table 6).

Both ADA and ME-ADA perform some form of adversarial training, so it is not
surprising that they outperform RCAD on this task. We suspect that these
methods outperform adversarial training because they are trained using the
multi-step projected gradient descent (PGD) [35], rather than the one-step
FGSM [13] and also have a higher clean test accuracy. While our aim is _not_
to propose a state-of-the-art method for withstanding adversarial attacks,
these preliminary results suggest that RCAD may be somewhat robust to
adversarial attacks, but does so without inheriting the poor (clean) test
accuracy of standard adversarial training.

Figure 6: Robustness to natural shifts in distribution: Plots comparing the
performance of standard ERM training and ME-ADA against our method (RCAD)
trained on CIFAR-10 benchmark and tested on various distribution shifts in the
corrupted CIFAR-10-C benchmark.

###  C.2 Robustness to natural distribution shifts

Our final set of experiments probe robustness to more systematic distribution
shifts using the corrupted CIFAR-10-C dataset [20]. These shifts go beyond the
small perturbations introduced by adversarial examples, and are a more
faithful reflection of the sorts of perturbations a machine learning model
might face “in the wild”.

We compare RCAD to standard ERM and ME-ADA on this benchmark; when all methods
are trained on the uncorrupted CIFAR-10 training dataset, but evaluated on
different types of corruptions. We report results in Figure 6. Both RCAD and
ME-ADA consistently outperform ERM. On certain corruptions (e.g., Gaussian
noise, glass blur), RCAD and ME-ADA achieve test accuracies that are around
$+25\%$ larger than the ERM baseline. We do not notice any systematic
difference in the results of RCAD versus ME-ADA, but note that ME-ADA requires
$2\times$ more compute than RCAD because its adversarial examples require
multiple gradient steps to compute.

While the main aim of our experiments has been to show that RCAD achieves
higher test accuracy, its good performance on robustness benchmarks suggests
that it may be a simpler yet appealing choice for practitioners.

##  Appendix D Analyzing RCAD vs. ERM in a Non-linear Toy Classification Setup

In Section 5 we analyse the solution returned by RCAD and compare it to the
ERM solution in a simplified binary classification setting where the
hypothesis class consists of linear predictors. We find that the high
dimensional linear decision boundary learnt by ERM depends on the noisy
dimensions that are spuriously correlated with the label on the training data
$\hat{\mathcal{D}}$. On the other hand, our method RCAD corrects this decision
boundary by unlearning spurious features. The self-generated examples by RCAD
exemplify the spurious correlations, and maximizing entropy on these examples
helps the model to drive any learnt components along the noisy dimensions to
lower values. In this section, we consider a setup where the true decision
boundary is non-linear and the hypothesis class consists of two-layer neural
networks with $128$ hidden units and ReLU activations. In order to induce
spurious features in our training data, we stick to high-dimensional inputs.

Typically, in classification problems the true generalizable features span a
lower dimensional space (compared to the ambient dimension) [1]. Ideally, we
would like to learn classifiers that depend only on these few generalizable
features, and remain independent of all the other noisy features. However,
training neural networks with the cross-entropy loss and stochastic gradient
descent (SGD) often leads to overfitting on training data since the trained
model ends up fitting on noisy features with high probability [43, 19]. To
simulate this phenomenon, we use a $d-$dimensional toy classification problem
where the true features are given by the first two-dimensions only, while the
rest of the $d-2$ dimensions are pure noise.

(a) ERM

(b) RCAD (100 epochs)

(c) RCAD (200 epochs)

(d) RCAD (300 epochs)

Figure 7: RCAD objective learns decision boundary using only generalizable
features: We simulate a high-dimensional non-linear classification problem by
projecting a simple 2-d dataset into a 625-dimensional space. _(a)_ Standard
ERM training overfits to this dataset, achieving perfect training accuracy by
picking up on spurious features. Plotting a 2D projection of the decision
boundary along the true features, we see that it poorly separates the data
along the generalizable (true) features. _(b, c, d)_ Visualizing our method
(RCAD) at different snapshots throughout training, we see that it converges to
the true decision boundary using only the generalizable features, thereby
achieving a higher test accuracy.

A dataset $\hat{\mathcal{D}}$ of 20,000 training examples is generated by
first sampling a two-dimensional vector $\tilde{{\mathbf{x}}}$ with equal
probability from one of two classes supported over two different well
separated moon shaped regions (see Figure 7). Here, $\tilde{{\mathbf{x}}}$ is
a two dimensional vector with label ${\textnormal{y}}\in\\{0,1\\}$. In our
setting, the input dimension $d=625$. Now, in order to construct the final
$d-$dimensional input ${\mathbf{x}}$ for the classifier using this two
dimensional sample $\tilde{{\mathbf{x}}}$, we first append to each sample a
vector of ($d-2=623$) zeros, and then add add Gaussian noise
${\mathbf{\epsilon}}\sim\mathcal{N}(0,\sigma^{2}\bm{I}_{d})$ where
$\sigma=0.1$. Note that the noise is added to all dimensions, including the
dimensions of the true feature but our results would not change if we choose
to only add noise to the last $d-2$ dimensions, which would only make the
problem easier.

$\displaystyle{\mathbf{x}}\triangleq(\tilde{{\mathbf{x}}},\underbrace{0,\cdots,0}_{\text{d-2=623
zeros}})+{\mathbf{\epsilon}},\qquad{\mathbf{\epsilon}}\sim{\mathcal{N}}(\bf{0},\sigma^{2}\bf{I}_{d})$

$(d,\sigma)$ | ERM | Adv. Training (FGSM) | ME-ADA | RCAD
---|---|---|---|---
$(625,0.10)$ | $80.4\%$ | $78.8\%$ | $90.9\%$ | $\mathbf{94.9\%}$
$(625,0.75)$ | $77.8\%$ | $77.9\%$ | $84.4\%$ | $\mathbf{89.6\%}$
$(1000,0.10)$ | $74.3\%$ | $72.0\%$ | $83.5\%$ | $\mathbf{87.1\%}$
Table 7: In the toy non-linear classification problem, RCAD is found to be
more robust to spurious correlations in the training data. We show the test
performance of RCAD vs. baseline methods ERM, adversarial training (FGSM) and
ME-ADA as we vary two parameters: $d$ which is the dimension of the input, and
$\sigma$ which is the standard deviation of the noise added to every
dimension.

We will measure the in-distribution performance of a classifier, trained on
$\hat{\mathcal{D}}$ and evaluated on fresh examples from the distribution
defined above. In our toy setup, the first two dimensions of the data
perfectly explain the class label; we call these dimensions the generalizable
(true) features. We are interested in learning non-linear classifiers (two-
layer neural nets) that successfully identify these generalizable features,
while ignoring the remaining dimensions.

Models trained with the standard cross-entropy loss on a fixed dataset
$\hat{\mathcal{D}}$ are liable to overfit [67]. A model can best reduce the
(empirical) cross-entropy loss by learning features that span all dimensions,
including the spurious feature dimensions. Precisely, the model can continue
to reduce the cross-entropy loss on some example ${\mathbf{x}}^{(i)}$ by
further aligning some of the weights of its hidden units along the direction
of the corresponding noise ${\mathbf{\epsilon}}^{(i)}$, that is part of input
${\mathbf{x}}^{(i)}$.

We start by training a two-layer net on $\hat{\mathcal{D}}$. We train it for
300 epochs of SGD using the standard empirical cross entropy loss. This model
achieves perfect training accuracy ($100\%$), but performs poorly on the
validation set ($80.4\%$). To visualize the learned model, we project the
decision boundary on the first two coordinates, the only ones that are truly
correlated with the label on population data. The decision boundary for this
model, shown in Figure 7(a), is quite different from the true decision
boundary. Rather than identifying the true generalizable features, this model
has overfit to the noisy dimensions, which are perpendicular to the span of
the true features. Training this model for more epochs leads to additional
overfitting, further decreasing the test accuracy.

Next we train with our objective RCAD using the same training dataset. In
addition to minimizing the standard empirical cross entropy loss, our method
also maximizes the predictive entropy of the model on self-generated
perturbations that lie along the adversarial directions. Intuitively, we
expect that these new examples will be along the directions of the spurious
features, exacerbating them. This is because adversarial directions have been
shown to comprise of spurious features [22]. Thus, in training the model to be
less confident on these examples, we signal the model to unlearn these
spurious features. Applying RCAD to this dataset, we achieve a much larger
test accuracy of $94.9\%$. When we visualize the decision boundary (along span
of the true features) in Figure 7(d), we observe that it correctly separates
the data. While SGD is implicitly biased towards learning simple (e.g.,
linear) decision boundaries [27], our results show that RCAD partially
counters this bias, forcing the model to learn a non-linear decision boundary
along the true features and ignoring the noisy dimensions. Additionally we
also train adversarial baselines in this setup and compare the test accuracies
with RCAD in Table 7. We find that RCAD is less vulnerable to spurious
correlations compared to other methods even when we increase the
dimensionality of the input or the noise level in the data generating process,
both of which would make models more prone to overfitting. This is confirmed
by the significant drop in the test performance of the ERM baseline.


# A short note on relative entropy for a pair of intermediate subfactors

Keshab Chandra Bakshi  Chennai Mathematical Institute, Chennai, INDIA
<EMAIL_ADDRESS><EMAIL_ADDRESS>
(Date: 9 May, 2021)

###### Abstract.

Given a quadruple of finite index subfactors we explicitly compute the
Pimsner-Popa probabilistic constant for the pair of intermediate subfactors
and relate it with the corresponding Connes-Størmer relative entropy between
them. This generalizes an old result of Pimsner and Popa.

###### 2010 Mathematics Subject Classification:

46L37

The first named author was supported through a DST INSPIRE Faculty grant
(reference no. DST/INSPIRE/04/2019/002754).

##  1\. Introduction

Generalizing the classical notion of (conditional) entropy from ergodic
theory, Connes and Størmer in [3] defined a relative entropy $H(B_{1}|B_{2})$
between a pair of finite dimensional $C^{*}$-subalgebras of a finite von
Neumann algebra $M$ with a fixed faithful normal trace $\mathrm{tr}$. Using
this they obtained an appropriate definition of the entropy of an automorphism
in the non-commutative framework of operator algebras. As an application, they
proved that for $n\neq m$ the $n$-shift of the hyperfinite $II_{1}$ factor is
not conjugate to the $m$-shift. Furthermore, they also proved a Kolmogorov-
Sinai type theorem using the relative entropy as the main technical tool.
However, in an impactful paper [8], Pimsner and Popa had observed that the
definition of the relative entropy does not depend on $B_{1},B_{2}$ being
finite dimensional, so that one may also consider the relative entropy
$H(B_{1}|B_{2})$ for arbitrary von Neumann subalgebras $B_{1},B_{2}\subset M$.
In another direction, as a generalization of the index of a subgroup in a
group, Jones in a groundbreaking paper [5] introduced a notion of index
$[M:N]$ as the Murray-von Neumann’s coupling constant
$\text{dim}_{N}(L^{2}(M))$ for any subfactor $N\subset M$ of type $II_{1}$.
Quite surprisingly, Pimsner and Popa had discovered that if $N$ and $M$ are
type $II_{1}$ factors and $N\subset M$ then $H(M|N)$ depends on both the Jones
index and the relative commutant. More precisely, they proved (among other
things) that if the relative commutant is trivial, that is $N^{\prime}\cap
M=\mathbb{C}$, then

(1.1) $H(M|N)=\log[M:N].$

In fact, Pimsner and Popa had shown that the equality holds in Equation 1.1 if
and only if the subfactor $N\subset M$ is extremal.

Given the fact that subfactor theory deals with the relative position of a
subfactor inside an ambient factor, it is a very natural and fundamental
question to consider relative positions of multiple subfactors. For an
irreducible subfactor with finite Jones index the set of its intermediate
subfactors forms a finite lattice under two natural operations. By a Galois
correspondence, every interval sublattice of finite groups can be realized as
intermediate subfactor lattice of finite index and thereby the study of
intermediate subfactor lattice of finite index may be thought of as a natural
generalization of the study of interval sublattice of finite groups.
Realization of even some simple finite lattices as intermediate (irreducible)
subfactor lattices remains entirely open. We want to understand this lattice
better. To this end, recently in [1], we discovered a new notion of angle
between two intermediate subfactors $P$ and $Q$ of the finite index subfactor
$N\subset M$ of type $II_{1}$ factors. In this paper we have taken yet another
approach by investigating the Connes-Stormer relative entropy and it looks
like more calculable in the non-irreducible setting. We consider a pair of
intermediate subfactors $N\subset P,Q\subset M$ of a finite index subfactor
$N\subset M$ of type $II_{1}$ factors with $N^{\prime}\cap M=\mathbb{C}$ (and
thus form a quadruple which we denote by $(N,P,Q,M)$), and obtain a formula
for $H(P|Q)$ in terms of a probabilistic number $\lambda(P,Q)$ due to Pimsner
and Popa.

For von Neumann subalgebras $B_{2}\subset B_{1}\subset M$ of a finite von
Neumann algebra $M$, Pimsner and Popa, in [8], defined the probabilistic
constant

(1.2) $\lambda(B_{1},B_{2})=\text{max}\\{\lambda>0|E_{B_{2}}(x)\geq\lambda
x,x\in B_{1_{+}}\\}.$

This serves as a replacement of Jones index when $B_{1}$ and $B_{2}$ are not
necessarily factors. Quite remarkably, they also prove that if $M$ is a type
$II_{1}$ factor and $N\subset M$ is a subfactor then

(1.3) ${\lambda(M,N)}={[M:N]}^{-1}.$

Interestingly, the definition of $\lambda(B_{1},B_{2})$, as in Equation 1.2,
works for general subalgebras $B_{1}$ and $B_{2}$ (not necessarily,
$B_{2}\subset B_{1})$ of $M$ as well; and furthermore, one may also consider
the number $\lambda(B_{2},B_{1}).$

As a generalization of Equation 1.3, we prove the following result:

Theorem 3.12. Let $(N,P,Q,M)$ be a quadruple of type $II_{1}$ factors with
$[M:N]<\infty$.

  1. (1)

If $(N,P,Q,M)$ is a commuting square then $\lambda(P,Q)={[P:N]}^{-1}$ and
$\lambda(Q,P)={[Q:N]}^{-1}.$

  2. (2)

If $(N,P,Q,M)$ is a co-commuting square then $\lambda(P,Q)={[M:Q]}^{-1}$ and
$\lambda(Q,P)={[M:P]}^{-1}.$

Moreover, in Corollary 3.9 and Corollary 3.10 we prove that the converse of
Theorem 3.12 also holds true for an irreducible quadruple (i.e.,
$N^{\prime}\cap M=\mathbb{C}$).

In this article, we also provide some calculable formulae for the
probabilistic numbers in the case of an irreducible quadruple.

Theorem 3.3. Suppose $(N,P,Q,M)$ is a quadruple of $II_{1}$ factors with
$N^{\prime}\cap M=\mathbb{C}$ and $[M:N]<\infty$ and $e_{P}$ (resp. $e_{Q}$)
is the biprojection corresponding to the intermediate subfactor $P$ (resp.
$Q$), then $\lambda(Q,P)=\frac{\text{tr}(e_{P}e_{Q})}{\text{tr}(e_{Q})}$ and
$\lambda(P,Q)=\frac{\text{tr}(e_{P}e_{Q})}{\text{tr}(e_{P})}.$

Finally, we prove the main result of this article by relating Connes-Størmer
relative entropy $H(P|Q)$ with $\lambda(P,Q)$.

Theorem 4.3.Let $(N,P,Q,M)$ be an irreducible quadruple such that
$[M:N]<\infty$. Then, $H(Q|P)=-\log\big{(}\lambda(Q,P)\big{)}.$

The above formula generalizes Equation 1.1.

Combining Theorem 4.3 with Theorem 3.3 , we deduce the following formula for
the relative entropy (in Corollary 4.4):

$H(Q|P)=\log\big{(}\mathrm{tr}(e_{Q})\big{)}-\log\big{(}\mathrm{tr}(e_{P}e_{Q})\big{)}.$

We conclude the paper with a cute application of Theorem 4.3, where we deduce
(in Corollary 4.7) that if $(N,P,Q,M)$ is an irreducible quadruple with
$[M:N]<\infty$ and $[P:N]=2$ then $H(P,Q)$ is either $0$ or $\log 2.$

The paper is organized as follows. After some Preliminaries in Section 2, we
discuss the probabilistic constants and present some useful formulae for them
in Section 3 and finally, in Section 4, we prove our main result involving the
Connes-Størmer relative entropy.

##  2\. Preliminaries

In this section, we fix the notations and recall some results from [1], which
we will use frequently in the sequel.

######  Notation 2.1.

In this paper we only deal with separable type $II_{1}$ factors. Consider a
subfactor $N\subset M$ of a type $II_{1}$ factor $M$ with $[M:N]<\infty$.
Throughout we will be dealing only with subfactors of type $II_{1}$ with
finite Jones index. Thus, $M$ has the unique faithful, normal tracial state
$\mathrm{tr}$.

  1. (1)

A quadruple

$\begin{matrix}Q&\subset&M\cr\cup&\ &\cup\cr N&\subset&P,\end{matrix}$

denoted by $(N,P,Q,M)$, is called irreducible if $N^{\prime}\cap
M=\mathbb{C}$. Consider the basic constructions $N\subset M\subset M_{1}$,
$P\subset M\subset P_{1}$ and $Q\subset M\subset Q_{1}$. As is standard, we
denote by $e_{1}$ the Jones projection $e^{M}_{N}$. It is easily seen that, as
$II_{1}$-factors acting on $L^{2}(M)$, both $P_{1}$ and $Q_{1}$ are contained
in $M_{1}$. In particular, if $e_{P}:L^{2}(M)\rightarrow L^{2}(P)$ denotes the
orthogonal projection, then $e_{P}\in M_{1}$. Likewise, $e_{Q}\in M_{1}$. Note
that, $\mathrm{tr}(e_{P})={[M:P]}^{-1}.$ Thus, we naturally obtain a dual
quadruple

$\begin{matrix}P_{1}&\subset&M_{1}\cr\cup&\ &\cup\cr
M&\subset&Q_{1}.\end{matrix}$

We call $(M,Q_{1},P_{1},M_{1})$ the basic construction of $(N,P,Q,M)$.

  2. (2)

A quadruple $(N,P,Q,M)$ is called a commuting square if
$E^{M}_{P}E^{M}_{Q}=E^{M}_{Q}E^{M}_{P}=E^{M}_{N}$. A quadruple $(N,P,Q,M)$ is
called a co-commuting square if the quadruple $(M,Q_{1},P_{1},M_{1})$ is a
commuting square.

  3. (3)

Suppose $N_{-1}\subset N\subset M$ is a downward basic construction. Also,
denote by $P_{-1}$ (resp. $Q_{-1}$) a downward basic construction of $N\subset
P$ (resp. $N\subset Q$) with the corresponding Jones projection
$e^{N}_{P_{-1}}$ (resp. $e^{N}_{Q_{-1}}$). We obtain a new quadruple

$\begin{matrix}P_{-1}&\subset&N\cr\cup&\ &\cup\cr
N_{-1}&\subset&Q_{-1}.\end{matrix}$

We call this new quadruple $(N_{-1},Q_{-1},P_{-1},N)$ as a downward basic
construction of the quadruple $(N,P,Q,M)$.

We recall without proofs the following elementary facts.

######  Fact 2.2.

Consider a quadruple of type $II_{1}$ factors $(N,P,Q,M)$ with $[M:N]<\infty.$

  1. (1)

Suppose $(M,Q_{1},P_{1},M_{1})$ is the basic construction of the quadruple and
let $e_{P_{1}}(\text{resp.}~{}e_{Q_{1}})$ be the Jones projection for the
inclusion $P_{1}\subset M_{1}(\text{resp.}~{}Q_{1}\subset M_{1}).$ Then,

$\mathrm{tr}(e_{P_{1}}e_{Q_{1}})=\frac{[M:P]}{[Q:N]}\mathrm{tr}(e_{P}e_{Q})=\frac{[M:Q]}{[P:N]}\mathrm{tr}(e_{P}e_{Q}).$

  2. (2)

Suppose $(N_{-1},Q_{-1},P_{-1},N)$ is a downward basic construction of
$(N,P,Q,M)$. Then,

$\mathrm{tr}(e^{N}_{P_{-1}}e^{N}_{Q_{-1}})=\frac{[M:Q]}{[P:N]}\mathrm{tr}(e_{P}e_{Q})=\frac{[M:P]}{[Q:N]}\mathrm{tr}(e_{P}e_{Q}).$

  3. (3)

$[P_{-1}:N_{-1}]=[M:P]$ and $[Q_{-1}:N_{-1}]=[M:Q].$

######  Definition 2.3.

[1, Definition 2.17] Consider a quadruple $(N,P,Q,M)$ as in 2.2. Let
$\\{\lambda_{i}:i\in I\\}$ and $\\{\mu_{j}:j\in J\\}$ be (right) Pimsner-Popa
bases for $P/N$ and $Q/N$, respectively. Define two auxiliary opertors
$p(P,Q)$ and $p(Q,P)$ as follows:

$p(P,Q)=\sum_{i,j}{\lambda_{i}}\mu_{j}e_{1}{\mu}^{*}_{j}{\lambda}^{*}_{i}\quad\text{and}\quad
p(Q,P)=\sum_{i,j}\mu_{j}\lambda_{i}e_{1}{\lambda}^{*}_{i}{\mu}^{*}_{j}.$

Below we recall from [1] a few important facts about the auxiliary operators
$p(P,Q)$ and $p(Q,P)$.

######  Fact 2.4.

Consider an irreducible quadruple of type $II_{1}$ factors $(N,P,Q,M)$ with
$[M:N]<\infty$.

  1. (1)

$p(P,Q)$ and $p(Q,P)$ are both independent of choice of bases. See [1, Lemma
2.18] for details.

  2. (2)

$p(P,Q)=[P:N]E^{N^{\prime}}_{P^{\prime}}(e_{Q})=[Q:N]E^{M_{1}}_{Q_{1}}(e_{P}).$
Similarly,
$p(Q,P)=[Q:N]E^{N^{\prime}}_{Q^{\prime}}(e_{P})=[P:N]E^{M_{1}}_{P_{1}}(e_{Q}).$
This follows from [1, Proposition 2.25].

  3. (3)

By [1, Proposition 2.22], $Jp(P,Q)J=p(Q,P)$, where $J$ is the usual modular
conjugation operator on $L^{2}(M)$; so that, $\|p(P,Q)\|=\|p(Q,P)\|$.

  4. (4)

$\lambda:=\lVert p(P,Q)\rVert=\lVert p(Q,P)\rVert.$ We note that
$\lambda=[M:N]\mathrm{tr}(e_{P}e_{Q})$.

Indeed, by [1, Remark 3.3], first observe that
$\lambda=[M:Q]{\mathrm{tr}}\big{(}p(P,Q)e_{Q}\big{)}$ and then, by the proof
of [1, Proposition 3.5], we note that
${\mathrm{tr}}\big{(}p(P,Q)e_{Q}\big{)}=[Q:N]{\mathrm{tr}}(e_{P}e_{Q}).$

We also recall the following result which will be used heavily in this note.

######  Lemma 2.5.

[1] If $N^{\prime}\cap M=\mathbb{C}$ then $\frac{1}{\lambda}p(P,Q)$ is a
projection and $\frac{1}{\lambda}p(P,Q)\geq e_{P}\vee e_{Q}.$ Similar
statement holds if we interchange $P$ and $Q$.

######  Remark 2.6.

The auxiliary operators corresponding to a downward basic construction (see
2.1) $(N_{-1},Q_{-1},P_{-1},N)$ of the quadruple $(N,P,Q,M)$ satisfy the
following formulae:

$p(Q_{-1},P_{-1})=[Q_{-1}:N_{-1}]E^{N^{\prime}_{-1}}_{Q^{\prime}_{-1}}(e^{N}_{P_{-1}})=[P_{-1}:N_{-1}]E^{M}_{P}(e^{N}_{Q_{-1}})$

and

$p(P_{-1},Q_{-1})=[P_{-1}:N_{-1}]E^{N^{\prime}_{-1}}_{P^{\prime}_{-1}}(e^{N}_{Q_{-1}})=[Q_{-1}:N_{-1}]E^{M}_{Q}(e^{N}_{P_{-1}}).$

##  3\. Pimsner-Popa probabilistic constant

Generalizing the Jones index, Pimsner and Popa in [8, Notation 2.5] had
introduced the probabilistic constant $\lambda(B_{1},B_{2})$ for von Neumann
subalgebras $B_{2}\subset B_{1}\subset M$ of a finite von Neumann algebra $M$
and this proved to be a powerful analytical tool in subfactor theory. However,
as observed in [7, Definition 4.1], this definition works for general
subalgebras $B_{1}$ and $B_{2}$ (not necessarily, $B_{2}\subset B_{1})$ as
well. In this section we shall obtain a formula for $\lambda(P,Q)$, for a pair
of intermediate subfactors $P$ and $Q$ of a subfactor $N\subset M$.

######  Definition 3.1.

(Pimsner-Popa) Consider a pair of von Neumann subalgebras $P$ and $Q$ of a
finite von Neumann algebra $M$. The Pimsner-Popa probabilistic constant of the
ordered pair $(P,Q)$ is defined as follows:

$\lambda(P,Q)=\text{max}\\{t>0|E_{Q}(x)\geq tx~{}~{}\forall x\in P_{+}\\}.$

######  Remark 3.2.

If $N\subset M$ is a subfactor of type $II_{1}$ factors then $\lambda(N,M)=1.$
This follows easily from the above definition.

In [8, Theorem 2.2], Pimsner and Popa have given a very useful
characterization of Jones index. More precisely, they obtained the following
relationship between $[M:N]$ and $\lambda(M,N)$ for any subfactor $N\subset M$
of type $II_{1}$ factors:

(3.1) ${[M:N]}^{-1}=\lambda(M,N).$

Below we consider a quadruple $(N,P,Q,M)$ and obtain a calculable formula for
$\lambda(P,Q)$ (and also of $\lambda(Q,P)$) in the case $N\subset M$ is
irreducible. If $Q=M$ and $P=N$, we recover Equation 3.1. We hope to obtain
formulae for $\lambda(P,Q)$ and $\lambda(Q,P)$ in the general case as well in
a future publication.

######  Theorem 3.3.

Suppose $(N,P,Q,M)$ is a quadruple of $II_{1}$ factors with $N^{\prime}\cap
M=\mathbb{C}$, then
$\lambda(Q,P)=\frac{{\mathrm{tr}}(e_{P}e_{Q})}{{\mathrm{tr}}(e_{Q})}$ and
$\lambda(P,Q)=\frac{{\mathrm{tr}}(e_{P}e_{Q})}{{\mathrm{tr}}(e_{P})}.$

###### Proof.

Let $x\in P_{+}$. Since $x$ is positive, we must have

$x=\big{(}\sum_{j}a_{j}e^{N}_{P_{-1}}b_{j}\big{)}^{*}\big{(}\sum_{i}a_{i}e^{N}_{P_{-1}}b_{i}\big{)}=\sum_{i,j}b^{*}_{j}E^{N}_{P_{-1}}(a^{*}_{j}a_{i})e^{N}_{P_{-1}}b_{i},$

where $a_{i},b_{j}\in N$. By Lemma 2.5 and 2.4 [item (4)], we have

$\frac{1}{[N:N_{-1}]\mathrm{tr}(e^{N}_{P_{-1}}e^{N}_{Q_{-1}})}p(P_{-1},Q_{-1})\geq
e^{N}_{P_{-1}}$

and thanks to 2.2 we have

$\frac{1}{[M:P][M:Q]\mathrm{tr}(e_{P}e_{Q})}[Q_{-1}:N_{-1}]E^{M}_{Q}(e^{N}_{P_{-1}})\geq
e^{N}_{P_{-1}}.$

Again by 2.2, $[Q_{-1}:N_{-1}]=[M:Q]$ so that

(3.2)
$\frac{\mathrm{tr}(e_{P})}{\mathrm{tr}(e_{P}e_{Q})}E^{M}_{Q}(e^{N}_{P_{-1}})\geq
e^{N}_{P_{-1}}.$

Thus, we deduce that :

$\displaystyle x=$
$\displaystyle\sum_{i,j}b^{*}_{j}E^{N}_{P_{-1}}(a^{*}_{j}a_{i})e^{N}_{P_{-1}}b_{i}$
$\displaystyle\leq$
$\displaystyle\frac{\mathrm{tr}(e_{P})}{\mathrm{tr}(e_{P}e_{Q})}\sum_{i,j}b^{*}_{j}E^{N}_{P_{-1}}(a^{*}_{j}a_{i})E^{M}_{Q}(e^{N}_{P_{-1}})b_{i}~{}~{}~{}~{}~{}~{}\text{[By
\lx@cref{creftypecap~refnum}{inequality}]}$ $\displaystyle\leq$
$\displaystyle\frac{\mathrm{tr}(e_{P})}{\mathrm{tr}(e_{P}e_{Q})}E^{M}_{Q}\bigg{(}\sum_{i,j}b^{*}_{j}E^{N}_{P_{-1}}(a^{*}_{j}a_{i})e^{N}_{P_{-1}}b_{i}\bigg{)}$
$\displaystyle\leq$
$\displaystyle\frac{\mathrm{tr}(e_{P})}{\mathrm{tr}(e_{P}e_{Q})}E^{M}_{Q}(x).$

In other words, for any $x\in P_{+}$, we have

(3.3) $E_{Q}(x)\geq\frac{\mathrm{tr}(e_{P}e_{Q})}{\mathrm{tr}(e_{P})}x.$

Now, suppose $E^{M}_{Q}(x)\geq sx$ for some $s>0$. Then,
$E^{M}_{Q}(e^{N}_{P_{-1}})\geq se^{N}_{P_{-1}}$. Taking norm on both sides we
get, $\lVert E^{M}_{Q}(e^{N}_{P_{-1}})\rVert\geq s$. By 2.4 and Remark 2.6 we
see that

$\lVert
E^{M}_{Q}(e^{N}_{P_{-1}})\rVert=\frac{[M:N]\mathrm{tr}(e^{N}_{P_{-1}}e^{N}_{Q_{-1}})}{[M:Q]}.$

Hence, by 2.2,

(3.4) $\frac{\mathrm{tr}(e_{P}e_{Q})}{\mathrm{tr}(e_{P})}\geq s.$

Therefore, combining Equation 3.3 and Equation 3.4 we get
$\lambda(P,Q)=\frac{\mathrm{tr}(e_{P}e_{Q})}{\mathrm{tr}(e_{P})},$ as desired.

The other formula follows by interchanging $P$ and $Q$.

This completes the proof. ∎

######  Remark 3.4.

Suppose $(N,P,Q,M)$ be as in Theorem 3.3. In general,
$\lambda(P,Q)\neq\lambda(Q,P)$ as can be easily seen from Remark 3.2. Using
Theorem 3.3, it follows that $\lambda(P,Q)=\lambda(Q,P)$ if and only if
$[P:N]=[Q:N].$

######  Example 3.5.

Let $G$ be a finite group acting outerly on a $II_{1}$-factor $S$. Let $H,K$
and $L$ be subgroups of $G$ such that $L\subseteq H\cap K$ and $H$ and $K$ are
non-trivial. Consider the quadruple $(N=S\rtimes L,P=S\rtimes H,Q=S\rtimes
K,M=S\rtimes G)$. Then, a simple calculation shows that (see [2, Section 2.2],
for instance)

$\mathrm{tr}(e_{P}e_{Q})=\frac{|H\cap
K|}{|G|},\mathrm{tr}(e_{P})=\frac{|H|}{|G|}~{}~{}~{}\text{and}~{}~{}~{}\mathrm{tr}(e_{P})=\frac{|K|}{|G|}.$

Therefore,

$\lambda(P,Q)=\frac{|H\cap
K|}{|H|}~{}~{}~{}\text{and}~{}~{}~{}\lambda(Q,P)=\frac{|H\cap K|}{|K|}.$

######  Example 3.6.

Let $H,K,G$ and $S$ be as in Example 3.5. Consider the quadruple
$(N=S^{G},P=S^{H},Q=S^{K},M=S)$. Then,

$\lambda(P,Q)=\frac{|H\cap
K|}{|K|}~{}~{}~{}~{}\text{and}~{}~{}~{}\lambda(Q,P)=\frac{|H\cap K|}{|H|}.$

The proof is simple and omitted.

######  Corollary 3.7.

Let $(N,P,Q,M)$ be a quadruple of $II_{1}$ factors with $N^{\prime}\cap
M=\mathbb{C}$ and $[M:N]<\infty$. Then, $\lambda(P_{1},Q_{1})=\lambda(Q,P)$
and $\lambda(Q_{1},P_{1})=\lambda(P,Q).$

###### Proof.

It is easy to check that $\mathrm{tr}(e_{Q_{1}})=\frac{1}{[Q:N]}.$ Thus, using
2.2 and Theorem 3.3, we see that

$\lambda(Q_{1},P_{1})=\frac{\mathrm{tr}(e_{P_{1}}e_{Q_{1}})}{\mathrm{tr}(e_{Q_{1}})}=[M:P]\mathrm{tr}(e_{P}e_{Q})=\lambda(P,Q).$

Similarly, $\lambda(P_{1},Q_{1})=\lambda(Q,P).$ This completes the proof. ∎

######  Corollary 3.8.

Let $(N,P,Q,M)$ be a quadruple of $II_{1}$ factors with $N^{\prime}\cap
M=\mathbb{C}$ and $[M:N]<\infty$. Then, $\lambda(P_{-1},Q_{-1})=\lambda(Q,P)$
and $\lambda(Q_{-1},P_{-1})=\lambda(P,Q).$

###### Proof.

First, it is easy to check that $\mathrm{tr}(e^{N}_{P_{-1}})={[P:N]}^{-1}.$
Thus by Theorem 3.3 and 2.2,

$\lambda(P_{-1},Q_{-1})=\frac{\mathrm{tr}(e^{N}_{P_{-1}}e^{N}_{Q_{-1}})}{\mathrm{tr}(e^{N}_{P_{-1}})}=[M:Q]\mathrm{tr}(e_{P}e_{Q})=\lambda(Q,P).$

Similarly, $\lambda(Q_{-1},P_{-1})=\lambda(P,Q).$ This completes the proof. ∎

######  Corollary 3.9.

Let $(N,P,Q,M)$ be a quadruple of type $II_{1}$ factors with $N^{\prime}\cap
M=\mathbb{C}$ and $[M:N]<\infty$. Then, $1\geq\lambda(P,Q)\geq{[P:N]}^{-1}.$
Furthermore, $\lambda(P,Q)={[P:N]}^{-1}$ if and only if $(N,P,Q,M)$ is a
commuting square. Also, $\lambda(P,Q)=1$ if and only if $P\subset Q$. A
similar statement holds if we interchange $P$ and $Q$.

###### Proof.

From Definition 3.1 it follows immediately that $\lambda(P,Q)\leq\lambda(N,Q)$
and also that $\lambda(N,Q)=1$. Thus, $1\geq\lambda(P,Q).$ To show the
inequality $\lambda(P,Q)\geq{[P:N]}^{-1}$, first observe that
$\mathrm{tr}(e_{P}e_{Q})\geq\frac{1}{[M:N]}$. Indeed, $e_{Q}\geq e_{N}$ and
hence $e_{P}e_{Q}e_{P}\geq e_{N}$ and now taking trace on both sides of the
last inequality we get
$\mathrm{tr}(e_{P}e_{Q})\geq\mathrm{tr}(e_{N})=\frac{1}{[M:N]}$, as desired.
Hence, by Theorem 3.3 , we obtain

$\lambda(P,Q)=\frac{\mathrm{tr}(e_{P}e_{Q})}{\mathrm{tr}(e_{P})}\geq\frac{1}{[M:N]}[M:P]=\frac{1}{[P:N]}.$

We conclude that if $(N,P,Q,M)$ is a quadruple of type $II_{1}$ factors, then
$1\geq\lambda(P,Q)\geq{[P:N]}^{-1}.$

Consider the auxiliary operator $p(P,Q)$ as in Definition 2.3 and following
2.4 , put $\lVert p(P,Q)\rVert=\lambda.$ Since, by 2.4 [item (4)],
$\lambda=[M:N]\mathrm{tr}(e_{P}e_{Q}))$, by Theorem 3.3, we obtain

(3.5) $\lambda(P,Q)=\frac{\lambda}{[P:N]}.$

Recall, given a quadruple of $II_{1}$ factors $(N,P,Q,M)$, a notion of
(interior) angle between $P$ and $Q$, denoted by $\alpha^{N}_{M}(P,Q)$, was
introduced in [1]. By [1, Proposition 2.4], $(N,P,Q,M)$ is a commuting square
if and only if ${\alpha}^{N}_{M}(P,Q)=\pi/2$. Furthermore, by [2, Proposition
4.5], ${\alpha}^{N}_{M}(P,Q)=\pi/2$ if and only if $\lambda=1$. In other
words, $(N,P,Q,M)$ form a commuting square if and only if $\lambda=1$ if and
only if $\lambda(P,Q)={[P:N]}^{-1}~{}~{}~{}\text{(by
\lx@cref{creftypecap~refnum}{lambda})}.$

If $P\subset Q$ then $e_{P}\leq e_{Q}$ and hence using Theorem 3.3 we see that
$\lambda(P,Q)=1.$ Conversely, if $\lambda(P,Q)=1$ then we get
$e_{P}=e_{P}e_{Q}=e_{Q}e_{P}$ and hence $P=P\cap Q.$ Therefore, we get
$P\subset Q$.

This completes the proof. ∎

######  Corollary 3.10.

Let $(N,P,Q,M)$ be a quadruple of type $II_{1}$ factors with $N^{\prime}\cap
M=\mathbb{C}$ and $[M:N]<\infty$. Then, $1\geq\lambda(P,Q)\geq{[M:Q]}^{-1}.$
Furthermore, $(N,P,Q,M)$ is a co-commuting square if and only if
$\lambda(P,Q)={[M:Q]}^{-1}.$ A similar statement holds if we interchange $P$
and $Q$.

###### Proof.

That $\lambda(P,Q)\geq{[M:Q]}^{-1}$ follows from Definition 3.1 . Indeed, as

$\\{t>0:E_{Q}(x)\geq tx~{}~{}\forall~{}~{}x\in
M_{+}\\}\subset\\{t>0:E_{Q}(x)\geq tx~{}~{}\forall~{}~{}x\in P_{+}\\},$

it follows from Definition 3.1 that $\lambda(M,Q)\leq\lambda(P,Q)$ and by
Equation 3.1 we know that $\lambda(M,Q)={[M:Q]}^{-1}.$

Recall, $(N,P,Q,M)$ is a co-commuting square if and only if
$(M,Q_{1},P_{1},M_{1})$ is a commuting square. Therefore, by Corollary 3.9 ,
$(N,P,Q,M)$ is a co-commuting square if and only if
$\lambda(Q_{1},P_{1})={[Q_{1}:M]}^{-1}={[M:Q]}^{-1}$ if and only if
$\lambda(P,Q)={[M:Q]}^{-1}$ (thanks to Corollary 3.7). This proves the
assertion. ∎

######  Proposition 3.11.

Let $(N,P,Q,M)$ be a quadruple of type $II_{1}$ factors with $N^{\prime}\cap
M=\mathbb{C}$ and $[M:N]<\infty$. If $[P:N]=2$, then $\lambda(P,Q)$ is either
$1$ or $1/2.$

###### Proof.

Let $\\{1,u\\}$ be a unitary orthonormal Pimsner-Popa basis for $P/N$ (which
exists by [5, Corollary 3.4.3]). Thus, $p(P,Q)=e_{Q}+ue_{Q}u^{*}$. By Lemma
2.5, $p(P,Q)e_{Q}=\lambda e_{Q}$ and hence
$\big{(}1+uE_{Q}(u^{*})\big{)}e_{Q}=\lambda e_{Q}.$ Thus, by [8, Lemma 1.2],
we obtain

(3.6) $\lambda-1=uE_{Q}(u^{*}).$

Now two cases arise. If $\lambda=1$ then $(N,P,Q,M)$ is a commuting square (as
already observed in the proof of Corollary 3.9)and hence, by Corollary 3.9,
$\lambda(P,Q)=1/2.$ When $\lambda\neq 1$, by Equation 3.6, $u\in Q$ and
therefore $P\subset Q$. So, by Corollary 3.9 again, we conclude that
$\lambda(P,Q)=1.$ ∎

By Corollary 3.9 we know that if $N\subset M$ is irreducible then $(N,P,Q,M)$
is a commuting square if and only if $\lambda(P,Q)={[P:N]}^{-1}$. Similarly,
Corollary 3.10 says that if $N\subset M$ is irreducible then $(N,P,Q,M)$ is a
co-commuting square if and only if $\lambda(P,Q)={[M:Q]}^{-1}$. Unfortunately,
we are not able to prove above formulae for the general case (i.e., without
assuming irreducibility). However, as a partial progress we have the following
result.

######  Theorem 3.12.

Let $(N,P,Q,M)$ be a quadruple of type $II_{1}$ factors with $[M:N]<\infty$.

  1. (1)

If $(N,P,Q,M)$ is a commuting square then $\lambda(P,Q)={[P:N]}^{-1}$ and
$\lambda(Q,P)={[Q:N]}^{-1}.$

  2. (2)

If $(N,P,Q,M)$ is a co-commuting square then $\lambda(P,Q)={[M:Q]}^{-1}$ and
$\lambda(Q,P)={[M:P]}^{-1}.$

###### Proof.

If $x\in P_{+}$, we have, by commuting square condition,
$E^{M}_{Q}(x)=E^{M}_{Q}\circ E^{M}_{P}(x)=E^{P}_{N}(x)$. Thanks to [8,
Proposition 2.1], we know that $E^{P}_{N}(x)\geq{[P:N]}^{-1}x$ and so
$E^{M}_{Q}(x)\geq{[P:N]}^{-1}x$. Now, if $E^{M}_{Q}(x)\geq tx$ for all $x\in
P_{+}$ and for some scalar $t$, we see that
$E^{M}_{Q}(e^{N}_{P_{-1}})=E^{P}_{N}(e^{N}_{P_{-1}})={[P:N]}^{-1}$ and so
${[P:N]}^{-1}\geq te^{N}_{P_{-1}}$. Taking norm to the both sides of the last
equation we get ${[P:N]}^{-1}\geq t.$ Therefore, $\lambda(P,Q)={[P:N]}^{-1}.$
Similarly, $\lambda(Q,P)={[Q:N]}^{-1}.$ This completes the proof of item (1).

We now prove item(2). Since $(N,P,Q,M)$ is a co-commuting square it is easy to
see that $(N_{-1},Q_{-1},P_{-1},N)$ is a commuting square. According to [1,
Proposition 2.20], $p(P_{-1},Q_{-1})$ is a projection such that
$p(P_{-1},Q_{-1})\geq e^{N}_{P_{-1}}$. By Remark 2.6 and 2.2 we see that

(3.7) $[M:Q]E^{M}_{Q}(e^{N}_{P_{-1}})\geq e^{N}_{P_{-1}}.$

For any $x\in P_{+}$, as in the proof of Theorem 3.3, we see that
$x=\sum_{i,j}b^{*}_{j}E^{N}_{P_{-1}}(a^{*}_{j}a_{i})e^{N}_{P_{-1}}b_{i}$ for
some $a_{i},b_{j}\in N$. Therefore,

$\displaystyle x\leq$
$\displaystyle[M:Q]\sum_{i,j}b^{*}_{j}E^{N}_{P_{-1}}(a^{*}_{j}a_{i})E^{M}_{Q}(e^{N}_{P_{-1}})b_{i}~{}~{}~{}~{}~{}~{}\text{[Using
\lx@cref{creftypecap~refnum}{commsquare}]}$ $\displaystyle=$
$\displaystyle[M:Q]E^{M}_{Q}\big{(}\sum_{i,j}b^{*}_{j}E^{N}_{P_{-1}}(a^{*}_{j}a_{i})e^{N}_{P_{-1}}b_{i}\big{)}$
$\displaystyle=$ $\displaystyle[M:Q]E^{M}_{Q}(x).$

Thus, we have proved that for any $x\in P_{+}$ we have
$E^{M}_{Q}(x)\geq{[M:Q]}^{-1}x$. Also, if $E^{M}_{Q}(x)\geq sx$ for $x\in
P_{+}$ and for some scalar $s$ then, in particular,
$E^{M}_{Q}(e^{N}_{P_{-1}})\geq se^{N}_{P_{-1}}$ and hence
$p(P_{-1},Q_{-1})\geq[M:Q]se^{N}_{P_{-1}}$. Thus, $\lVert
p(P_{-1},Q_{-1})\rVert=1\geq s[M:Q].$ So, ${[M:Q]}^{-1}\geq s.$ We conclude
that $\lambda(P,Q)={[M:Q]}^{-1}$ and similarly $\lambda(Q,P)={[M:P]}^{-1}.$
This completes the proof. ∎

######  Remark 3.13.

Since the quadruple $(N,M,N,M)$ is both a commuting and a co-commuting square,
in view of the fact that $\lambda(M,N)={[M:N]}^{-1}$ (see [8][Theorem 2.2]),
Theorem 3.12 can be thought of as a natural generalization of [8][Theorem
2.2].

######  Example 3.14.

Let $K\subset L$ be a subfactor of finite index and $G$ be a finite group
acting outerly on $L$ so that we obtain a quadruple $(N=K,P=L,Q=K\rtimes
G,M=L\rtimes G).$ Then the quadruple is a commuting square. Thus,
$\lambda(L,K\rtimes G)=[L:K]$ and $\lambda(K\rtimes G,L)=|G|.$

######  Remark 3.15.

We feel that $\lambda(P,Q)$ (and $\lambda(Q,P)$) is a powerful invariant in
determining the relative position between the intermediate subfactors
$N\subset P,Q\subset M$, and demands a deeper investigation.

##  4\. Relative entropy and intermediate subfactors

Connes and Størmer in [3] introduced a notion of entropy of a finite
dimensional subalgebra and more generally, the relative entropy between two
finite dimensional subalgebras of a finite von Neumann algebra as an extension
of the notion of entropy from classical ergodic theory. Pimsner and Popa in
[8] had observed that this notion does not depend on the fact that the
subalgebras are finite-dimensional and they further extended it to a notion of
relative entropy between intermediate von Neumann subalgebras of a finite von
Neumann algebra. We briefly recall the definition below.

######  Definition 4.1.

(Connes-Størmer) Let $\eta:[0,\infty)\rightarrow(-\infty,\infty)$ be defined
by $\eta(t)=-t\log(t)$ for $t>0$ and $\eta(0)=0.$ Let $M$ be a finite von
Neumann algebra with a fixed normalized trace $\mathrm{tr}$ and $P$ and $Q$ be
von Neumann subalgebras of $M$. The entropy of $P$ relative to $Q$ with
respect to $\mathrm{tr}$ is

$H_{\mathrm{tr}}(P|Q)=\text{sup}\sum_{i}\bigg{(}\mathrm{tr}\big{(}\eta(E_{Q}(x_{i}))\big{)}-\mathrm{tr}\big{(}\eta(E_{P}(x_{i}))\big{)}\bigg{)},$

where the supremum is taken over all finite partitions of unity
$1=\sum_{i}x_{i}$ in $M$, and $E_{P}$ and $E_{Q}$ are the
$\mathrm{tr}$-preserving conditional expectations on $P$ and $Q$,
respectively. If $M$ is type $II_{1}$ factor we often suppress $\mathrm{tr}$
in the notation for the relative entropy as the trace is uniquely determined
in this case.

Below, we see that the Pimsner-Popa probabilistic constant is closely related
with the relative entropy. Recall, in [8, Corollary 4.6], Pimsner and Popa had
proved that if $N\subset M$ is irreducible then

(4.1) $H(M|N)=-\log\lambda(M,N).$

More generally, we consider a quadruple of type $II_{1}$ factors $(N,P,Q,M)$
with $[M:N]<\infty$ and $N^{\prime}\cap M=\mathbb{C}$ and obtain a formula of
$H(P|Q)$ (and also of $H(Q|P)$) which generalizes Equation 4.1. This is the
main result of this article. First we need a lemma.

######  Lemma 4.2.

Let $(N,P,Q,M)$ be an irreducible quadruple such that $[M:N]<\infty$. Then,

$\mathrm{tr}\bigg{(}\eta\big{(}E^{M}_{P}(e^{N}_{Q_{-1}})\big{)}\bigg{)}=-\frac{1}{[Q:N]}\log\big{(}\lambda(Q,P)\big{)}.$

###### Proof.

Following 2.1 , suppose $(N_{-1},Q_{-1},P_{-1},N)$ is a downward basic
construction of $(N,P,Q,M)$ and $p(Q_{-1},P_{-1})$ is the corresponding
auxiliary operator. By Remark 2.6,

(4.2)
$\mathrm{tr}\bigg{(}\eta\big{(}E^{M}_{P}(e^{N}_{Q_{-1}})\big{)}\bigg{)}=\mathrm{tr}\bigg{(}\eta\big{(}\frac{1}{[P_{-1}:N_{-1}]}p(Q_{-1},P_{-1}))\big{)}\bigg{)}.$

Now, using 2.4 (item (4)) and Lemma 2.5 we see that
$\frac{1}{[M:N]\mathrm{tr}(e^{N}_{P_{-1}}e^{N}_{Q_{-1}})}p(Q_{-1},P_{-1})$ is
a projection. Denote this projection by $f$. By 2.2,

$f=\frac{1}{[M:P][M:Q]\mathrm{tr}(e_{P}e_{Q})}p(Q_{-1},P_{-1}).$

Since
$p(Q_{-1},P_{-1})=[Q_{-1}:N_{-1}]E^{N^{\prime}_{-1}}_{Q^{\prime}_{-1}}(e^{N}_{P_{-1}})$,
it follows that

$\mathrm{tr}(p(Q_{-1},P_{-1})=\frac{[Q_{-1}:N_{-1}]}{[N:P_{-1}]}=\frac{[M:Q]}{[P:N]},$

and so

(4.3) $\mathrm{tr}(f)=\frac{1}{[M:N]tr(e_{P}e_{Q})}.$

Since $[P_{-1}:N_{-1}]=[M:P]$, Equation 4.2 implies that

$\mathrm{tr}\bigg{(}\eta\big{(}E^{M}_{P}(e^{N}_{Q_{-1}})\big{)}\bigg{)}=\mathrm{tr}\bigg{(}\eta\big{(}\frac{\mathrm{tr}(e_{P}e_{Q})}{\mathrm{tr}(e_{Q})}f\big{)}\bigg{)}.$

It follows easily (see [4]) that if $\alpha$ is a scalar then $\eta(\alpha
f)=\eta(\alpha)f.$ Therefore, by Equation 4.3 we obtain

$\mathrm{tr}\bigg{(}\eta\big{(}E^{M}_{P}(e^{N}_{Q_{-1}})\big{)}\bigg{)}=\mathrm{tr}\bigg{(}\eta\big{(}\frac{\mathrm{tr}(e_{P}e_{Q})}{\mathrm{tr}(e_{Q})}\big{)}f\bigg{)}=\frac{1}{[M:N]\mathrm{tr}(e_{P}e_{Q})}\eta\big{(}\frac{\mathrm{tr}(e_{P}e_{Q})}{\mathrm{tr}(e_{Q})}\big{)}.$

In other words,

$\mathrm{tr}\bigg{(}\eta\big{(}E^{M}_{P}(e^{N}_{Q_{-1}})\big{)}\bigg{)}=-\frac{1}{[Q:N]}\log\big{(}\frac{\mathrm{tr}(e_{P}e_{Q})}{\mathrm{tr}(e_{Q})}\big{)}.$

The proof is now complete once we apply Theorem 3.3. ∎

######  Theorem 4.3.

Let $(N,P,Q,M)$ be an irreducible quadruple such that $[M:N]<\infty$. Then,
$H(Q|P)=-\log\big{(}\lambda(Q,P)\big{)}.$

###### Proof.

To establish $H(Q|P)\leq-\log\big{(}\lambda(Q,P)\big{)}$ (this inequality
generalizes an observation of [7, Proposition 4.1]) we suitably modify the
proof of [8, Proposition 3.5]. We provide sufficient details for the sake of
self-containment. First note that for any $x\in M_{+}$ we must have

$E_{P}\big{(}E_{Q}(x)\big{)}\geq\lambda(Q,P)E_{Q}(x).$

Since $\log$ is operator increasing we get

${\big{(}E_{Q}(x)\big{)}}^{1/2}\log
E_{P}\big{(}E_{Q}(x)\big{)}{\big{(}E_{Q}(x)\big{)}}^{1/2}\geq\log\lambda(Q,P)E_{Q}(x)+{\big{(}E_{Q}(x)\big{)}}^{1/2}\log
E_{Q}(x){\big{(}E_{Q}(x)\big{)}}^{1/2}.$

Thus,

$\mathrm{tr}\bigg{(}E_{P}\big{(}E_{Q}(x)\big{)}\log
E_{P}\big{(}E_{Q}(x)\big{)}\bigg{)}\geq\log\lambda(Q,P)\mathrm{tr}(x)+\mathrm{tr}\bigg{(}E_{Q}(x)\log
E_{Q}(x)\bigg{)}.$

In other words, we get

$\mathrm{tr}\bigg{(}\eta
E_{P}\big{(}E_{Q}(x)\big{)}\bigg{)}-\mathrm{tr}\bigg{(}\eta
E_{Q}(x)\bigg{)}\leq-\log\lambda(Q,P)\mathrm{tr}(x).$

As $\eta E_{P}\big{(}E_{Q}(x)\big{)}\geq E_{P}\big{(}E_{Q}(\eta x)\big{)}$
(see [8, page 74], for instance) we readily obtain

$\mathrm{tr}(\eta x)-\mathrm{tr}(\eta
E_{Q}(x))\leq-\log\lambda(Q,P)\mathrm{tr}(x).$

So,

$\big{(}\mathrm{tr}(\eta x)-\mathrm{tr}(\eta
E_{P}(x))\big{)}+\big{(}\mathrm{tr}\big{(}\eta
E_{P}(x)\big{)}-\mathrm{tr}(\eta
E_{Q}(x))\big{)}\leq-\log\lambda(Q,P)\mathrm{tr}(x).$

Summing up over a partition of unity $(x_{i})$ in $M_{+}$ and taking the
supremum over all such partitions we get $H(P|M)+H(Q|P)\leq-\log\lambda(Q,P).$
But as $P\subset M$, it follows easily that $H(P|M)=0$ (see [8, page 75]).
Thus we have proved that

(4.4) $H(Q|P)\leq-\log\lambda(Q,P).$

We now prove the non-trivial implication. The proof is inspired by [8] (see
also [6]). Note, $e^{N}_{Q_{-1}}$ is a projection in $Q$ with
$\mathrm{tr}(e^{N}_{Q_{-1}})=\frac{1}{[Q:N]}$. Put
$x=e^{N}_{Q_{-1}}-\frac{1}{[Q:N]}1.$ Then, $\mathrm{tr}(x)=0.$ Following [8,
Lemma 4.2], let

$K_{x}=\overline{\text{conv}\\{vxv^{*}:v\in\mathcal{U}(N)\\}}.$

A similar calculation as in [8, Lemma 4.2] shows that $0\in K_{x}$ and hence
for any fixed $\epsilon>0$ there are unitaries $v_{1},\ldots,v_{n}\in N$ such
that

$\big{\lVert}\sum_{i}\frac{1}{n}v_{i}xv^{*}_{i}\big{\rVert}_{2}<{\epsilon}^{2}\frac{1}{[Q:N]}.$

Therefore, we see that

$\big{\lVert}\sum_{i}\frac{[Q:N]}{n}v_{i}e^{N}_{Q_{-1}}v^{*}_{i}-1\big{\rVert}_{2}<{\epsilon}^{2}.$

Put $y=\frac{[Q:N]}{n}\sum_{i}v_{i}e^{N}_{Q_{-1}}v^{*}_{i}$. Then,

(4.5) $\lVert y-1\rVert_{2}<\epsilon^{2}.$

Let $p$ be the spectral projection of $y$ corresponding to the interval
$[0,1+\epsilon].$ Using the following obvious inequality

$\frac{1}{(1+\epsilon)}t\mathcal{X}_{[0,1+\epsilon]}(t)\leq 1,$

and then applying functional calculus we easily obtain
$\frac{1}{(1+\epsilon)}yp\leq 1.$ Now, put

$x_{i}=\frac{[Q:N]}{(1+\epsilon)n}v_{i}e^{N}_{Q_{-1}}v^{*}_{i}\wedge p.$

Thus,

$\sum_{i}x_{i}\leq\frac{[Q:N]}{(1+\epsilon)n}\sum_{i}pv_{i}e^{N}_{Q_{-1}}v^{*}_{i}p\leq\frac{1}{(1+\epsilon)}yp\leq
1.$

Now, it follows immediately from Definition 4.1 that

(4.6)
$H(Q|P)\geq\sum_{i}\mathrm{tr}\big{(}\eta(E_{P}(x_{i}))-\eta(E_{Q}(x_{i}))\big{)}.$

Since, for any scalar $\alpha,t>0$ we have $\eta(\alpha
t)=\eta(\alpha)t+\alpha\eta(t)$, a simple calculation yields

$\sum_{i}\mathrm{tr}\big{(}\eta E_{P}(x_{i})-\eta
E_{Q}(x_{i})\big{)}=\frac{[Q:N]}{(1+\epsilon)n}\sum_{i}\mathrm{tr}\big{(}\eta
E_{P}(v_{i}e^{N}_{Q_{-1}}v^{*}_{i}\wedge p)-\eta
E_{Q}(v_{i}e^{N}_{Q_{-1}}v^{*}_{i}\wedge p)\big{)};$

and so, by Inequality 4.6 we get

(4.7) $H(Q|P)\geq\sum_{i}\mathrm{tr}\big{(}\eta E_{P}(x_{i})-\eta
E_{Q}(x_{i})\big{)}\geq\frac{[Q:N]}{(1+\epsilon)n}\sum_{i}\mathrm{tr}\big{(}\eta(E_{P}(v_{i}e^{N}_{Q_{-1}}v^{*}_{i}\wedge
p))\big{)}.$

By [8, page 74] we know that if $a,b\in M_{+},$ then
$\mathrm{tr}(\eta(a+b))\leq\mathrm{tr}(\eta a)+\mathrm{tr}(\eta b)$ and so,

$\mathrm{tr}\bigg{(}\eta\big{(}E_{P}(v_{i}e^{N}_{Q_{-1}}v^{*}_{i}\wedge
p)\big{)}\bigg{)}\geq\mathrm{tr}\bigg{(}\eta\big{(}E_{P}(v_{i}e^{N}_{Q_{-1}}v^{*}_{i})\big{)}\bigg{)}-\mathrm{tr}\bigg{(}\eta\big{(}E_{P}(v_{i}e^{N}_{Q_{-1}}v^{*}_{i})-E_{P}(v_{i}e^{N}_{Q_{-1}}v^{*}_{i}\wedge
p)\big{)}\bigg{)}.$

By concavity of $\eta$, it is well-known that $\eta(v^{*}xv)\geq
v^{*}\eta(x)v$ for any $v$ with $\lVert v\rVert\leq 1$ and $x\in M_{+}$ with
$\lVert x\rVert\leq 1$ (see [6, B.1], for instance). Therefore, it follows
that

$\mathrm{tr}\bigg{(}\eta\big{(}E_{P}(v_{i}e^{N}_{Q_{-1}}v^{*}_{i})\big{)}\bigg{)}=\mathrm{tr}\bigg{(}\eta\big{(}v_{i}E_{P}(e^{N}_{Q_{-1}})v^{*}_{i}\big{)}\bigg{)}\geq\mathrm{tr}\bigg{(}v_{i}\eta\big{(}E_{P}(e^{N}_{Q_{-1}})\big{)}v^{*}_{i}\bigg{)}=\mathrm{tr}\bigg{(}\eta\big{(}E_{P}(e^{N}_{Q_{-1}})\big{)}\bigg{)}.$

Thus, by Inequality 4.7, we obtain

(4.8)
$H(Q|P)\geq\frac{[Q:N]}{(1+\epsilon)n}\bigg{\\{}\sum_{i}\mathrm{tr}\big{(}\eta(E_{P}(e^{N}_{Q_{-1}}))\big{)}-\mathrm{tr}\bigg{(}\eta\big{(}E_{P}(v_{i}e^{N}_{Q_{-1}}v^{*}_{i})-E_{P}(v_{i}e^{N}_{Q_{-1}}v^{*}_{i}\wedge
p)\big{)}\bigg{)}\bigg{\\}}.$

Now, a similar calculations as in the proof of [8][Lemma 4.2] yields

(4.9)
$\mathrm{tr}\bigg{(}\eta\big{(}E_{P}(v_{i}e^{N}_{Q_{-1}}v^{*}_{i})-E_{P}(v_{i}e^{N}_{Q_{-1}}v^{*}_{i}\wedge
p)\big{)}\bigg{)}\leq\eta(\epsilon^{2}).$

We provide the details for the convenience of the reader. In view of the fact
that $\mathrm{tr}\circ\eta\leq\eta\circ\mathrm{tr}$, it is sufficient to prove
that

(4.10)
$\eta\big{(}\mathrm{tr}(v_{i}e^{N}_{Q_{-1}}v^{*}_{i}-v_{i}e^{N}_{Q_{-1}}v^{*}_{i}\wedge
p)\big{)}\leq\eta(\epsilon^{2}).$

Since $\mathrm{tr}(e\vee f)=\mathrm{tr}(e)+\mathrm{tr}(f)-\mathrm{tr}(e\wedge
f)$ for any projections $e$ and $f$ in $M$ we get

$\mathrm{tr}(v_{i}e^{N}_{Q_{-1}}v^{*}_{i}-v_{i}e^{N}_{Q_{-1}}v^{*}_{i}\wedge
p)\leq 1-\mathrm{tr}(p).$

Applying functional calculus we easily see that $y(1-p)\geq(1+\epsilon)(1-p)$
and so

${\lVert
y-1\rVert}_{2}^{2}\geq\mathrm{tr}\big{(}(y-1)^{2}(1-p)\big{)}\geq\epsilon^{2}\mathrm{tr}(1-p).$

Thus, by Inequality 4.5, we obtain $1-\mathrm{tr}(p)\leq{\epsilon}^{-2}{\lVert
y-1\rVert}_{2}^{2}\leq{\epsilon}^{2}$. So we obtain

$\mathrm{tr}(v_{i}e^{N}_{Q_{-1}}v^{*}_{i}-v_{i}e^{N}_{Q_{-1}}v^{*}_{i}\wedge
p)\leq{\epsilon}^{2}.$

As $\eta$ is increasing in $[0,{\epsilon}^{2}]$ for sufficiently small
$\epsilon$, we see that the Inequality 4.10 is now obvious. This proves the
truth of the Inequality 4.9.

Below we show that, using Inequalities 4.8 and 4.9, the following inequalities
hold true:

$\displaystyle H(Q|P)\geq$
$\displaystyle\frac{[Q:N]}{(1+\epsilon)n}\sum_{i}\mathrm{tr}\bigg{(}\eta\big{(}E_{P}(e^{N}_{Q_{-1}})\big{)}\bigg{)}-\frac{[Q:N]}{(1+\epsilon)}\eta(\epsilon^{2})$
$\displaystyle=$
$\displaystyle-\frac{[Q:N]}{(1+\epsilon)}\frac{1}{[Q:N]}\log(\lambda(Q,P))-\frac{[Q:N]}{(1+\epsilon)}\eta(\epsilon^{2})~{}~{}~{}~{}~{}~{}[\text{Using~{}~{}~{}\lx@cref{creftypecap~refnum}{liu}~{}~{}~{}]}$
$\displaystyle\geq$
$\displaystyle-\frac{1}{(1+\epsilon)}\log(\lambda(Q,P))-\frac{[Q:N]}{(1+\epsilon)}\eta(\epsilon^{2}).$

Therefore, letting $\epsilon\rightarrow 0$ we conclude that

(4.11) $H(Q|P)\geq-\log\big{(}\lambda(Q,P)\big{)}.$

The proof is now complete once we combine Inequality 4.4 and Inequality 4.11.
∎

Applying Theorem 3.3 we immediately obtain the following (possibly useful)
formula for the relative entropy.

######  Corollary 4.4.

Let $(N,P,Q,M)$ be an irreducible quadruple such that $[M:N]<\infty$. Then,

$H(Q|P)=\log\big{(}\mathrm{tr}(e_{Q})\big{)}-\log\big{(}\mathrm{tr}(e_{P}e_{Q})\big{)}$

and

$H(P|Q)=\log\big{(}\mathrm{tr}(e_{P})\big{)}-\log\big{(}\mathrm{tr}(e_{P}e_{Q})\big{)}$

######  Corollary 4.5.

Let $(N,P,Q,M)$ be a quadruple of type $II_{1}$ factors with $N^{\prime}\cap
M=\mathbb{C}$ and $[M:N]<\infty$. Then, $H(Q_{1},P_{1})=H(P|Q)$ and
$H(P_{1}|Q_{1})=H(Q|P).$

###### Proof.

By Theorem 4.3 we obtain
$H(P_{1}|Q_{1})=-\log\big{(}\lambda(P_{1},Q_{1})\big{)}.$ Using Corollary 3.7
we get

$H(P_{1}|Q_{1})=-\log\big{(}\lambda(Q,P)\big{)}.$

Interchanging $P$ and $Q$ in the above equation we prove the other
implication. This completes the proof. ∎

Similarly, applying Corollary 3.8 we get the following.

######  Corollary 4.6.

Let $(N,P,Q,M)$ be a quadruple of type $II_{1}$ factors with $N^{\prime}\cap
M=\mathbb{C}$ and $[M:N]<\infty$. Then, $H(Q_{-1}|P_{-1})=H(P|Q)$ and
$H(P_{-1}|Q_{-1})=H(Q|P).$

The following consequence is obvious once we apply Theorem 4.3 and Proposition
3.11.

######  Corollary 4.7.

Let $(N,P,Q,M)$ be a quadruple of type $II_{1}$ factors with $N^{\prime}\cap
M=\mathbb{C}$ and $[M:N]<\infty$. If $[P:N]=2$ then $H(P,Q)$ is either $0$ or
$\log 2.$

We conclude the paper with a remark.

######  Remark 4.8.

We believe that $H(P|Q)$ is a powerful invariant to investigate the relative
position between intermediate subfactors and therefore, it is desirable to
know the possible values of it. We remark that Theorem 4.3 is a modest step
towards achieving this goal. In future we want to dig deep into these entropic
aspects of intermediate subfactor theory.

##  5\. Acknowledgement

I would like to thank Zhengwei Liu for various useful discussions and pointing
out an error in an earlier version.

## References

  * [1] K. C. Bakshi, S. Das, Z. Liu and Y. Ren, An angle between intermediate subfactors and its rigidity, Trans. Amer. Math. Soc. 371 (2019), 5973-5991.
  * [2] K. C. Bakshi and V.P. Gupta, A note on irreducible quadrilaterals of $II_{1}$-factors. Internat. J. Math. 30  (2019), no. 12, 1950061, 22pp.
  * [3] A. Connes, E. Størmer, Entropy for automorphisms of type $II_{1}$ von Neumann algebras, Acta Math. 134 (1975), 289–306.
  * [4] C. Jiang, Z. Liu and J. Wu, Noncommutative uncertainty principles, J. Funct. Anal. 270 (2016), no. 1, 264-311.
  * [5] V. F. R. Jones, Index for subfactors, Invent. Math. 72 (1983), no. 1, 1–25.
  * [6] S. Neshveyev and E Størmer, Dynamical entropy in operator algebras, Results in Mathematics and Related Areas 3rd Series A Series of Modern Surveys in Mathematics. 50 (2006), Springer-Verlag, Berlin.
  * [7] R. Okayasu, Relative entropy for abelian subalgebras, Internat. J. Math. 21 (2010), no. 4, 537-550.
  * [8] M. Pimsner and S. Popa, Entropy and index for subfactors, Ann. Sci. Ecole Norm. Sup. (4) 19 (1986), no. 1, 57-106.