# Regular Functions on the $K$-nilpotent cone

Lucas Mason-Brown

###### Abstract.

Let $G$ be a complex reductive algebraic group with Lie algebra $\mathfrak{g}$
and let $G_{\mathbb{R}}$ be a real form of $G$ with maximal compact subgroup
$K_{\mathbb{R}}$. Associated to $G_{\mathbb{R}}$ is a
$K\times\mathbb{C}^{\times}$-invariant subvariety $\mathcal{N}_{\theta}$ of
the (usual) nilpotent cone $\mathcal{N}\subset\mathfrak{g}^{*}$. In this
article, we will derive a formula for the ring of regular functions
$\mathbb{C}[\mathcal{N}_{\theta}]$ as a representation of
$K\times\mathbb{C}^{\times}$.

Some motivation comes from Hodge theory. In [SV11], Schmid and Vilonen use
ideas from Saito’s theory of mixed Hodge modules to define canonical good
filtrations on many Harish-Chandra modules (including all standard and
irreducible Harish-Chandra modules). Using these filtrations, they formulate a
conjectural description of the unitary dual. If $G_{\mathbb{R}}$ is split, and
$X$ is the spherical principal series representation of infinitesimal
character $0$, then conjecturally
$\operatorname{gr}(X)\simeq\mathbb{C}[\mathcal{N}_{\theta}]$ as
representations of $K\times\mathbb{C}^{\times}$. So a formula for
$\mathbb{C}[\mathcal{N}_{\theta}]$ is an essential ingredient for computing
Hodge filtrations.

##  1\. Introduction

Let $G$ be a complex connected reductive algebraic group and let
$G_{\mathbb{R}}$ be a real form of $G$. Choose a maximal compact subgroup
$K_{\mathbb{R}}$ of $G_{\mathbb{R}}$ and let
$\theta:G\stackrel{{\scriptstyle}}{{\to}}G$ be the corresponding Cartan
involution. Let $K$ be the group of $\theta$-fixed points (i.e. the
complexification of $K_{\mathbb{R}}$). Note that $K$ is a complex reductive
algebraic group (it is often disconnected). Write
$\mathfrak{g}$,$\mathfrak{k}$ for the Lie algebras and let
$\mathfrak{p}=\mathfrak{g}^{-d\theta}$. There is a Cartan decomposition

(1.0.1) $\mathfrak{g}=\mathfrak{k}\oplus\mathfrak{p}$

Using (1.0.1), we can identify $\mathfrak{p}^{*}$ with
$(\mathfrak{g}/\mathfrak{k})^{*}$, a subspace of $\mathfrak{g}^{*}$.

Let $G\times\mathbb{C}^{\times}$ act on $\mathfrak{g}^{*}$ by the usual
formula

$(g,z)\cdot\zeta=z\operatorname{Ad}^{*}(g)\zeta,\qquad g\in\widetilde{G},\
z\in\mathbb{C}^{\times},\ \zeta\in\mathfrak{g}^{*}.$

Note that $\mathfrak{p}^{*}$ is stable under
$K\times\mathbb{C}^{\times}\subset G\times\mathbb{C}^{\times}$. Let
$\mathcal{N}$ denote the nilpotent cone in $\mathfrak{g}^{*}$. Recall that
$G\times\mathbb{C}^{\times}$ acts on $\mathcal{N}$ (with finitely many
orbits). The $K$-_nilpotent cone_ is the
$K\times\mathbb{C}^{\times}$-invariant subvariety

$\mathcal{N}_{\theta}=\mathcal{N}\cap\mathfrak{p}^{*}\subset\mathfrak{p}^{*}$

This subvariety (and the $K\times\mathbb{C}^{\times}$-action on it) is closely
related to the representation theory of $G_{\mathbb{R}}$, see [Vog91]. The
main result of this paper is an explicit description of the ring of regular
functions $\mathbb{C}[\mathcal{N}_{\theta}]$ as a representation of
$K\times\mathbb{C}^{\times}$ (in the case when $G_{\mathbb{R}}$ is split
modulo center). Some motivation for this problem will be discussed at the end
of Section 2.

Since $K$ is (in general) a disconnected group, its irreducible
representations cannot be easily parameterized using the theory of highest
weights. In [Vog07], Vogan gives a parameterization of a very different flavor
(making essential use of the fact that $K$ is a symmetric subgroup of $G$). We
will recall some of the details in Section 2.

There are several well-known results regarding the structure of
$\mathbb{C}[\mathcal{N}_{\theta}]$ as a $K$-representation. In [KR71], it is
shown that for $G_{\mathbb{R}}$ quasi-split,
$\mathbb{C}[\mathcal{N}_{\theta}]$ is isomorphic as a $K$-representation to
the induced representation $\operatorname{Ind}^{K}_{M}\mathrm{triv}$, where
$M$ is centralizer in $K$ of a maximal abelian subspace
$\mathfrak{a}\subset\mathfrak{p}$. Equivalently,
$\mathbb{C}[\mathcal{N}_{\theta}]$ is isomorphic as a $K$-representation to a
spherical principal series representation of $G_{\mathbb{R}}$. Importantly,
these results do not provide any information about the
$\mathbb{C}^{\times}$-action on $\mathbb{C}[\mathcal{N}_{\theta}]$. To
understand this structure, we must adopt a different approach. First, we
relate the ring of regular functions $\mathbb{C}[\mathcal{N}_{\theta}]$
(regarded as a representation of $K\times\mathbb{C}^{\times}$) to the ring of
regular functions $\mathbb{C}[\mathcal{N}]$ (regarded as a representation of
$G\times\mathbb{C}^{\times})$. For $G_{\mathbb{R}}$ split modulo center, we
prove the following formula in Corollary 6.0.2

(1.0.2)
$\mathbb{C}[\mathcal{N}_{\theta}]|_{K\times\mathbb{C}^{\times}}=\mathbb{C}[\mathcal{N}]|_{K\times\mathbb{C}^{\times}}\otimes[\wedge(\mathfrak{k})]$

Here $[\wedge(\mathfrak{k})]$ denotes the signed graded exterior algebra
associated to $\mathfrak{k}$ (this formula takes place in the Grothendieck
group of admissible representations of $K\times\mathbb{C}^{\times}$). The
proof of this result is not purely formal—we make essential use of the fact
that $\mathcal{N}$ is Cohen-Macaulay and that
$\mathcal{N}_{\theta}\subset\mathcal{N}$ is a complete intersection of
codimension $\dim(\mathfrak{k})$ (for the latter assertion, we use that
$G_{\mathbb{R}}$ is split modulo center). The structure of
$\mathbb{C}[\mathcal{N}]$ as a $G\times\mathbb{C}^{\times}$-representation is
well-known (it can be computed using Lusztig’s $q$-analog of Kostant’s
partition function, see [McG89]). We then ‘restrict’ this description to
$K\times\mathbb{C}^{\times}$ to obtain a formula for
$\mathbb{C}[\mathcal{N}_{\theta}]$. The final result is Theorem 7.0.2.

Along the way, we introduce a restriction map on equivariant K-theory

$K^{G}(\mathcal{N})\stackrel{{\scriptstyle}}{{\to}}K^{K}(\mathcal{N}_{\theta})$

(this map is defined and studied in Section 3). In fact, this map arises as
the ‘associated graded’ of a map from representations of $G$ (regarded as a
real group) to representations of $G_{\mathbb{R}}$ (see Remark 3.0.8 for more
details). We will pursue this point in future work.

###  1.1. Notation

Let $R$ be an algebraic group. An algebraic $R$-representation $V$ is
_admissible_ if every irreducible $R$-representation appears in $V$ with
finite multiplicity. Consider the abelian categories

$\displaystyle\operatorname{Rep}(R)$ $\displaystyle=\text{algebraic
representations of }R$ $\displaystyle\operatorname{Rep}_{a}(R)$
$\displaystyle=\text{admissible algebraic representations of }R$
$\displaystyle\operatorname{Rep}_{f}(R)$ $\displaystyle=\text{finite-
dimensional algebraic representations of }R$

There are obvious embeddings
$\operatorname{Rep}_{f}(R)\subset\operatorname{Rep}_{a}(R)\subset\operatorname{Rep}(R)$.

Now let $\widetilde{R}=R\times\mathbb{C}^{\times}$. Then
$\operatorname{Rep}(\widetilde{R})$, $\operatorname{Rep}_{a}(\widetilde{R})$,
and $\operatorname{Rep}_{f}(\widetilde{R})$ can be defined as above. We will
also consider the category

$\displaystyle\operatorname{Rep}_{aa}(\widetilde{R})=$
$\displaystyle\text{algebraic representations of }\widetilde{R}\text{ which
are admissible}$ $\displaystyle\text{as representations of both }R\text{ and
}\mathbb{C}^{\times}$

There are obvious embeddings
$\mathrm{Rep}_{f}(\widetilde{R})\subset\mathrm{Rep}_{aa}(\widetilde{R})\subset\mathrm{Rep}_{a}(\widetilde{R})\subset\mathrm{Rep}(\widetilde{R})$.
We will denote the Grothendieck groups by
$K(R),K_{a}(R),K_{f}(R),K_{aa}(\widetilde{R})$, and so on. Write
$\mathrm{Irr}(R)$ for the set of (equivalence classes of) irreducible
representations of $R$. Then

$\mathrm{Irr}(\widetilde{R})=\\{\tau q^{n}\mid\tau\in\mathrm{Irr}(R),\
n\in\mathbb{Z}\\},$

where $q^{n}$ denotes the degree-$n$ character of $\mathbb{C}^{\times}$ and
$\tau q^{n}$ is shorthand for the irreducible $\widetilde{R}$-representation
$\tau\otimes q^{n}$. The group $K_{f}(R)$ (resp. $K_{f}(\widetilde{R})$) can
be identified with (finite) integer combinations of $\tau\in\mathrm{Irr}(R)$
(resp. $\tau q^{n}\in\mathrm{Irr}(\widetilde{R})$). The group $K_{a}(R)$
(resp. $K_{a}(\widetilde{R})$) can be identified with _formal_ integer
combinations of $\tau\in\mathrm{Irr}(R)$ (resp. $\tau
q^{n}\in\mathrm{Irr}(\widetilde{R})$). Finally, $K_{aa}(\widetilde{R})$ can be
identified with formal integer combinations of $\tau q^{n}$ such that for each
$\tau\in\mathrm{Irr}(R)$ and for each $n\in\mathbb{Z}$ only finitely many
$\tau q^{n}$ appear with nonzero multiplicity. The tensor product of
representations turns $K_{f}(R)$ into a commutative ring, $K_{a}(R)$ into a
$K_{f}(R)$-module, and $K_{aa}(\widetilde{R})$ into a
$K_{f}(\widetilde{R})$-module.

If $X$ is a scheme equipped with an algebraic $R$-action, we write
$\operatorname{Coh}^{R}(X)$ for the category of (strongly) $R$-equivariant
coherent sheaves on $X$ and $K^{R}(X)$ for its Grothendieck group.

###  1.2. Acknowledgments

I would like to thank David Vogan and Yu Zhao for many helpful discussions.

##  2\. Irreducible representations of $K$

In this section, we will recall a parameterization of $\mathrm{Irr}(K)$ due to
Vogan ([Vog07]).

Suppose $H$ is a $\theta$-stable maximal torus in $G$. Write
$\Delta=\Delta(G,H)$ for the roots of $H$ on $\mathfrak{g}$. Since $H$ is
$\theta$-stable, $\theta$ acts on $\Delta$ by
$\alpha\mapsto\alpha\circ\theta$. A root $\alpha\in\Delta$ is imaginary (resp.
real, resp. complex) if $\theta\alpha=\alpha$ (resp. $\theta\alpha=-\alpha$,
resp. $\theta\alpha\notin\\{\pm\alpha\\}$). If $\alpha$ is imaginary, then
$\theta$ acts on the root space $\mathfrak{g}_{\theta}$ by $\pm\mathrm{Id}$.
We say that $\alpha$ is compact (resp. noncompact) if
$\theta|_{\mathfrak{g}_{\alpha}}=\mathrm{Id}$ (resp.
$\theta|_{\mathfrak{g}_{\alpha}}=-\mathrm{Id}$). So $\Delta$ is partitioned

$\Delta=\Delta_{i\mathbb{R}}\sqcup\Delta_{\mathbb{R}}\sqcup\Delta_{\mathbb{C}}$

into imaginary, real, and complex roots, and $\Delta_{i\mathbb{R}}$ is
partitioned

$\Delta_{i\mathbb{R}}=\Delta_{c}\sqcup\Delta_{n}$

into compact and noncompact roots. Note that $\Delta_{i\mathbb{R}}$ is a root
system. If we choose a positive system $\Phi^{+}\subset\Delta_{i\mathbb{R}}$,
we can define the element
$\rho_{i\mathbb{R}}=\frac{1}{2}\sum\Phi^{+}\in\mathfrak{h}^{*}$.

######  Definition 2.0.1 (Sec 6, [AVTV20]).

A continued Langlands parameter for $(G,K)$ is a triple $(H,\gamma,\Phi^{+})$
where

  1. (1)

$H$ is a $\theta$-stable maximal torus in $G$.

  2. (2)

$\gamma$ is a formal sum $\gamma_{0}+\rho_{i\mathbb{R}}$, where $\gamma_{0}$
is a one-dimensional $(\mathfrak{h},H^{\theta})$-module (we define
$d\gamma=d\gamma_{0}+\rho_{i\mathbb{R}}$).

  3. (3)

$\Phi^{+}$ is a positive system for $\Delta_{i\mathbb{R}}$.

There is a $K$-action on the set of continued Langlands parameters. Two
parameters are _equivalent_ if they are conjugate under $K$. A continued
Langlands parameter is _standard_ if

  * (4)

For every $\alpha\in\Phi^{+}$, $\langle d\gamma,\alpha^{\vee}\rangle\geq 0$.

A standard Langlands parameter is _nonzero_ if

  * (5)

For every $\alpha\in\Phi^{+}$ which is simple and compact, $\langle
d\gamma,\alpha^{\vee}\rangle\neq 0$.

A nonzero Langlands parameter is _final_ if

  * (6)

If $\alpha\in\Delta_{\mathbb{R}}$ and $\langle
d\gamma,\alpha^{\vee}\rangle=0$, then $\alpha$ does not satisfy the Speh-Vogan
parity condition ([SV80]).

Write $\mathcal{P}_{L}(G,K)$ for the set of equivalence classes of final
Langlands parameters.

To the equivalence class of a continued Langlands parameter $\Gamma$, one can
associate a virtual Harish-Chandra module $[I(\Gamma)]$. If $\Gamma$ is
standard, then $[I(\Gamma)]$ is represented by a distinguished
$(\mathfrak{g},K)$-module $I(\Gamma)$. If $\Gamma$ is nonzero, then
$I(\Gamma)\neq 0$. If $\Gamma$ is final, there is a unique irreducible
quotient $I(\Gamma)\twoheadrightarrow J(\Gamma)$. The following is a version
of the Langlands classification.

######  Theorem 2.0.2 (Thm 6.1, [AVTV20]).

The map $\Gamma\mapsto J(\Gamma)$ defines a bijection between
$\mathcal{P}_{L}(G,K)$ and isomorphism classes of irreducible
$(\mathfrak{g},K)$-modules.

For any continued Langlands parameter $\Gamma$, the infinitesimal character of
the virtual $(\mathfrak{g},K)$-module $[I(\Gamma)]$ corresponds, under the
Harish-Chandra isomorphism, to the $W$-orbit of $d\gamma$. If $\Gamma$ is
final, then $I(\Gamma)$ is tempered if and only if the restriction of
$d\gamma$ to $\mathfrak{h}^{-d\theta}$ lies in the imaginary span of $\Delta$
(i.e. $d\gamma|_{\mathfrak{h}^{-d\theta}}$ is imaginary). In this case,
$I(\Gamma)$ is irreducible, i.e. $I(\Gamma)=J(\Gamma)$.

######  Definition 2.0.3.

A final Langlands parameter $(H,\gamma,\Phi^{+})$ is _tempered with real
infinitesimal character_ if $d\gamma|_{\mathfrak{h}^{-d\theta}}=0$. Write
$\mathcal{P}_{L}^{t,\mathbb{R}}(G,K)$ for the set of equivalence classes of
such parameters.

######  Corollary 2.0.4.

The map $\Gamma\mapsto J(\Gamma)$ defines a bijection between
$\mathcal{P}_{L}^{t,\mathbb{R}}(G,K)$ and isomorphism classes of irreducible
tempered $(\mathfrak{g},K)$-modules with real infinitesimal character.

Suppose $\mu$ is an irreducible representation of $K$. Choose a maximal torus
$T\subset K$ and a positive system $\Phi_{c}^{+}$ for $\Delta(K,T)$. Let
$\rho_{c}=\frac{1}{2}\sum\Phi_{c}^{+}\in\mathfrak{t}^{*}$. Let
$\lambda\in\mathfrak{t}^{*}$ be a highest weight of $\mu$ (if $K$ is
disconnected, $\lambda$ need not be unique). The $K$-norm of $\mu$ is defined
by the formula

$|\mu|^{2}:=\langle\lambda+2\rho_{c},\lambda+2\rho_{c}\rangle$

It is not hard to see that $|\mu|$ is independent of $\Phi^{+}_{c}$ and
$\lambda$.

We say that $\mu$ is a _lowest_ $K$_-type_ in a a $(\mathfrak{g},K)$-module
$X$ if $\mu$ appears in $X$ with nonzero multiplicity and $|\mu|$ is minimal
among all irreducible representations of $K$ with this property.

######  Theorem 2.0.5 (Thm 11.9, [Vog07]).

The following are true:

  * (i)

If $\Gamma\in\mathcal{P}_{L}^{t,\mathbb{R}}(G,K)$, then $I(\Gamma)$ contains a
unique lowest $K$-type $\mu(\Gamma)$.

  * (ii)

The map $\Gamma\mapsto\mu(\Gamma)$ defines a bijection between
$\mathcal{P}_{L}^{t,\mathbb{R}}(G,K)$ and $\mathrm{Irr}(K)$.

  * (iii)

There is a total order on $\mathcal{P}_{L}^{t,\mathbb{R}}(G,K)$ such that the
(infinite) square matrix $m(\Gamma,\Gamma^{\prime})$ defined by the formula

$[I(\Gamma)]=\sum_{\Gamma^{\prime}\in\mathcal{P}_{L}^{t,\mathbb{R}}}m(\Gamma,\Gamma^{\prime})\mu(\Gamma^{\prime})$

is upper triangular with $1$’s along the diagonal.

  * (iv)

In particular, this matrix $m(\Gamma,\Gamma^{\prime})$ is invertible. Write
$M(\Gamma,\Gamma^{\prime})$ for its inverse (which is also upper triangular).

  * (v)

The entries of the matrices $m(\Gamma,\Gamma^{\prime})$ and
$M(\Gamma,\Gamma^{\prime})$ can be computed by an algorithm.

The algorithm in (v) is described in [Vog07].

Now suppose that $G_{\mathbb{R}}$ is split modulo center. This means that
there is a maximal torus $H_{s}$ in $G$ such that
$\Delta(G,H_{s})=\Delta_{\mathbb{R}}(G,H_{s})$ (and so
$\Delta_{i\mathbb{R}}(G,H_{s})=\emptyset$). Let $\Gamma_{0}$ be the parameter

$\Gamma_{0}:=(H_{s},0,\emptyset)\in\mathcal{P}_{L}^{t,\mathbb{R}}(G,K)$

By a result of Kostant ([Kos69]) there is an identity in $K_{a}(K)$

$\mathbb{C}[\mathcal{N}_{\theta}]|_{K}=I(\Gamma_{0})|_{K}$

So by Theorem 2.0.5, we can write $\mathbb{C}[\mathcal{N}_{\theta}]$ as a
formal integer sum of the irreducible $K$-representations $\mu(\Gamma)$ (this
idea has been implemented in the atlas software). This is quite useful
information, but it does _not_ give us the grading on
$\mathbb{C}[\mathcal{N}_{\theta}]$, which is part of what we’re after.

In [SV11], Schmid and Vilonen define canonical good filtrations on all Harish-
Chandra modules of ‘functorial origin’, including all standard and irreducible
Harish-Chandra modules. These canonical good filtrations (and their associated
gradeds) should be closely related to questions of unitarity.

The associated graded of a Harish-Chandra module with respect to a good
filtration can be regarded as a representation of $\widetilde{K}$ (in fact, as
a class in $K_{aa}(\widetilde{K})$). It is conjectured in [SV11] that

$\mathbb{C}[\mathcal{N}_{\theta}]|_{\widetilde{K}}=\operatorname{gr}I(\Gamma_{0})|_{\widetilde{K}}$

It is also suggested that the Hodge filtration on an _arbitrary_ standard
module (of an arbitrary group) can be reduced to this case (via cohomological
induction and a deformation argument). So computing the class
$\mathbb{C}[\mathcal{N}_{\theta}]|_{\widetilde{K}}$ in the case when
$G_{\mathbb{R}}$ is split is central to the program of computing Hodge
fitlrations. In Theorem 7.0.2, we will give a formula for
$\mathbb{C}[\mathcal{N}_{\theta}]|_{\widetilde{K}}$ in terms of the classes
$I(\Gamma)q^{n}$.

##  3\. A restriction map
$K^{\widetilde{G}}(\mathcal{N})\stackrel{{\scriptstyle}}{{\to}}K^{\widetilde{K}}(\mathcal{N}_{\theta})$

In this section, we will define a restriction map

$K^{\widetilde{G}}(\mathcal{N})\stackrel{{\scriptstyle}}{{\to}}K^{\widetilde{K}}(\mathcal{N}_{\theta})$

Since $\mathcal{N}$ and $\mathcal{N}_{\theta}$ are singular, we cannot proceed
directly. Instead, we follow the standard approach outlined (for example) in
[CG10, Sec 5.3]: we first regard $\mathcal{N}$ (resp. $\mathcal{N}_{\theta}$)
as a subvariety of $\mathfrak{g}^{*}$ (resp. $\mathfrak{p}^{*}$) and then
apply the restriction map
$K^{\widetilde{G}}(\mathfrak{g}^{*})\stackrel{{\scriptstyle}}{{\to}}K^{\widetilde{K}}(\mathfrak{p}^{*})$
(defined in the usual way, as an alternating sum of Tor functors).

Our first proposition describes the relationship between
$K^{\widetilde{G}}(\mathcal{N})$, $K^{\widetilde{G}}(\mathfrak{g}^{*})$
$K_{aa}(\widetilde{G})$, and $K_{f}(\widetilde{G})$.

######  Proposition 3.0.1.

The following are true:

  * (i)

If $\mathcal{E}\in\operatorname{Coh}^{\widetilde{G}}(\mathcal{N})$, then
$\Gamma(\mathcal{N},\mathcal{E})|_{\widetilde{G}}\in\operatorname{Rep}_{aa}(\widetilde{G})$.
This defines an exact functor
$\operatorname{Coh}^{\widetilde{G}}(\mathcal{N})\stackrel{{\scriptstyle}}{{\to}}\operatorname{Rep}_{aa}(\widetilde{G})$,
and hence a group homomorphism

$\Gamma(\bullet)|_{\widetilde{G}}:K^{\widetilde{G}}(\mathcal{N})\stackrel{{\scriptstyle}}{{\to}}K_{aa}(\widetilde{G})$

  * (ii)

The homomorphism in (i) is injective.

  * (iii)

If $\mathcal{E}\in\operatorname{Coh}^{\widetilde{G}}(\mathfrak{g}^{*})$, then
$\Gamma(\mathfrak{g}^{*},\mathcal{E})|_{\widetilde{G}}\in\operatorname{Rep}_{a}(\widetilde{G})$.
This defines an exact functor
$\operatorname{Coh}^{\widetilde{G}}(\mathfrak{g}^{*})\stackrel{{\scriptstyle}}{{\to}}\operatorname{Rep}_{a}(\widetilde{G})$,
and hence a group homomorphism

$\Gamma(\bullet)|_{\widetilde{G}}:K^{\widetilde{G}}(\mathfrak{g}^{*})\stackrel{{\scriptstyle}}{{\to}}K_{a}(\widetilde{G})$

  * (iv)

Restriction along $\\{0\\}\subset\mathfrak{g}^{*}$ induces an exact functor
$\operatorname{Coh}^{\widetilde{G}}(\mathfrak{g}^{*})\stackrel{{\scriptstyle}}{{\to}}\operatorname{Coh}^{\widetilde{G}}(\\{0\\})\simeq\operatorname{Rep}_{f}(\widetilde{G})$,
which in turn induces a group isomorphism

$|_{\\{0\\}}:K^{\widetilde{G}}(\mathfrak{g}^{*})\xrightarrow{\sim}K_{f}(\widetilde{G})$

  * (v)

The direct image along the closed embedding
$j:\mathcal{N}\hookrightarrow\mathfrak{g}^{*}$ induces an exact functor
$\operatorname{Coh}^{\widetilde{G}}(\mathcal{N})\stackrel{{\scriptstyle}}{{\to}}\operatorname{Coh}^{\widetilde{G}}(\mathfrak{g}^{*})$,
and therefore a group homomorphism

$j_{*}:K^{\widetilde{G}}(\mathcal{N})\stackrel{{\scriptstyle}}{{\to}}K^{\widetilde{G}}(\mathfrak{g}^{*})$

  * (vi)

If $V\in\operatorname{Rep}_{f}(\widetilde{G})$, then
$V\otimes\mathbb{C}[\mathfrak{g}^{*}]\in\operatorname{Rep}_{a}(\widetilde{G})$.
This defines an exact functor
$\operatorname{Rep}_{f}(\widetilde{G})\stackrel{{\scriptstyle}}{{\to}}\operatorname{Rep}_{a}(\widetilde{G})$,
and hence a group homomorphism

$\phi_{\mathfrak{g}^{*}}:K_{f}(\widetilde{G})\stackrel{{\scriptstyle}}{{\to}}K_{a}(\widetilde{G})$

  * (vii)

The following diagram commutes:

${K_{a}(\widetilde{G})}$${K_{f}(\widetilde{G})}$${K_{aa}(\widetilde{G})}$${K^{\widetilde{G}}(\mathcal{N})}$${K^{\widetilde{G}}(\mathfrak{g}^{*})}$$\scriptstyle{\phi_{\mathfrak{g}^{*}}}$$\scriptstyle{\Gamma(\bullet)|_{\widetilde{G}}}$$\scriptstyle{j_{*}}$$\scriptstyle{|_{\\{0\\}}}$$\scriptstyle{\Gamma(\bullet)|_{\widetilde{G}}}$

  * (viii)

The homomorphism in (v) is injective.

###### Proof.

  * (i)

This is [AV19, (5.4f)] (it is an immediate consequence of the following facts:
$\widetilde{G}$ is reductive, $\mathcal{N}$ is affine, and $\widetilde{G}$
acts on $\mathcal{N}$ with finitely many orbits).

  * (ii)

This is [AV19, Cor 7.4].

  * (iii)

Since $\mathfrak{g}^{*}$ is affine, the functor
$\Gamma(\bullet)|_{\widetilde{G}}:\operatorname{Coh}^{\widetilde{G}}(\mathfrak{g}^{*})\stackrel{{\scriptstyle}}{{\to}}\operatorname{Rep}(\widetilde{G})$
is exact. It suffices to show that its image is contained in
$\operatorname{Rep}_{a}(\widetilde{G})$. Let
$\mathcal{E}\in\operatorname{Coh}^{\widetilde{G}}(\mathfrak{g}^{*})$. Since
$\mathfrak{g}^{*}$ is smooth, there is a $\widetilde{G}$-equivariant vector
bundle $\mathcal{V}$ on $\mathfrak{g}^{*}$ and a surjection
$\mathcal{V}\twoheadrightarrow\mathcal{E}$ in
$\operatorname{Coh}^{\widetilde{G}}(\mathfrak{g}^{*})$. Since
$\Gamma(\bullet)|_{\widetilde{G}}$ is exact, we get a surjection
$\Gamma(\mathfrak{g}^{*},\mathcal{V})|_{\widetilde{G}}\twoheadrightarrow\Gamma(\mathfrak{g}^{*},\mathcal{E})|_{\widetilde{G}}$
in $\operatorname{Rep}(\widetilde{G})$. So it suffices to show that
$\Gamma(\mathfrak{g}^{*},\mathcal{V})|_{\widetilde{G}}\in\operatorname{Rep}_{a}(\widetilde{G})$.
Let $V=\mathcal{V}|_{\\{0\\}}$. Then
$V\in\operatorname{Rep}_{f}(\widetilde{G})$ and

$\mathcal{V}\simeq V\otimes\mathcal{O}_{\mathfrak{g}^{*}}$

in $\operatorname{Coh}^{\widetilde{G}}(\mathfrak{g}^{*})$. So
$\Gamma(\mathfrak{g}^{*},\mathcal{V})|_{\widetilde{G}}=V\otimes\mathbb{C}[\mathfrak{g}^{*}]|_{\widetilde{G}}$.
Note that each graded component of $\mathbb{C}[\mathfrak{g}^{*}]$, and hence
of $V\otimes\mathbb{C}[\mathfrak{g}^{*}]$, is finite-dimensional. So
$V\otimes\mathbb{C}[\mathfrak{g}^{*}]|_{\widetilde{G}}\in\operatorname{Rep}_{a}(\widetilde{G})$,
as required.

  * (iv)

This is a well-known fact from equivariant $K$-theory, see [Tho87, Thm 4.1].

  * (v)

This follows from the fact that $j$ is closed, and hence affine.

  * (vi)

See the proof of (iii).

  * (vii)

It suffices to show that the top triangle is commutative (the commutativity of
the bottom triangle is obvious). If $\mathcal{V}\simeq
V\otimes\mathcal{O}_{\mathfrak{g}^{*}}\in\operatorname{Coh}^{\widetilde{G}}(\mathfrak{g}^{*})$
is a vector bundle, then

$\Gamma(\mathfrak{g}^{*},\mathcal{V})|_{\widetilde{G}}\simeq
V\otimes\mathbb{C}[\mathfrak{g}^{*}]|_{\widetilde{G}}\simeq\mathcal{V}|_{\\{0\\}}\otimes\mathbb{C}[\mathfrak{g}^{*}]|_{\widetilde{G}}$

But since $\mathfrak{g}^{*}$ is smooth, $K^{\widetilde{G}}(\mathfrak{g}^{*})$
is spanned by vector bundles. So the upper triangle is commutative.

  * (viii)

By (vii), the map

(3.0.1)
$K^{\widetilde{G}}(\mathcal{N})\overset{\Gamma(\bullet)|_{\widetilde{G}}}{\stackrel{{\scriptstyle}}{{\to}}}K_{aa}(\widetilde{G})\subset
K_{a}(\widetilde{G})$

coincides with the composition

$K^{\widetilde{G}}(\mathcal{N})\overset{j_{*}}{\stackrel{{\scriptstyle}}{{\to}}}K^{\widetilde{G}}(\mathfrak{g}^{*})\overset{\Gamma(\bullet)|_{\widetilde{G}}}{\stackrel{{\scriptstyle}}{{\to}}}K_{a}(\widetilde{G}).$

By (ii), (3.0.1) is injective. Hence,
$j_{*}:K^{\widetilde{G}}(\mathcal{N})\stackrel{{\scriptstyle}}{{\to}}K^{\widetilde{G}}(\mathfrak{g}^{*})$
must be injective as well.

∎

######  Remark 3.0.2.

We note that (ii) of Proposition 3.0.1 (and hence (viii), which is a
consequence) is a very deep assertion—the proof of (ii) in [AV19] makes
essential use of the Langlands classification.

######  Remark 3.0.3.

It is worth considering what happens if we forget about the
$\mathbb{C}^{\times}$-actions. Arguing exactly as in Proposition 3.0.1, we get
a commutative diagram

${K(G)}$${K_{f}(G)}$${K_{a}(G)}$${K^{G}(\mathcal{N})}$${K^{G}(\mathfrak{g}^{*})}$$\scriptstyle{\phi_{\mathfrak{g}^{*}}}$$\scriptstyle{\Gamma(\bullet)|_{G}}$$\scriptstyle{j_{*}}$$\scriptstyle{|_{\\{0\\}}}$$\scriptstyle{\Gamma(\bullet)|_{G}}$

and $\Gamma:K^{G}(\mathcal{N})\stackrel{{\scriptstyle}}{{\to}}K_{a}(G)$ is
injective. However, $K(G)=0$ (indeed, every algebraic $G$-representation $V$
satisfies $V\oplus V^{\infty}\simeq V^{\infty}$, and therefore has image $0$
in $K(G)$). So we cannot deduce that
$j_{*}:K^{G}(\mathcal{N})\stackrel{{\scriptstyle}}{{\to}}K^{G}(\mathfrak{g}^{*})$
is injective (and in fact, it is not: the skyscraper sheaf at $\\{0\\}$ (with
trivial $G$-action) lies in the kernel of the map
$j_{*}:K^{G}(\mathcal{N})\stackrel{{\scriptstyle}}{{\to}}K^{G}(\mathfrak{g}^{*})$.
So, the $\mathbb{C}^{\times}$-actions are essential for the proposition above.

If we replace $\widetilde{G}$ with $\widetilde{K}$, $\mathfrak{g}^{*}$ with
$\mathfrak{p}^{*}$, and so on, we can prove a result which is completely
analogous to Proposition 3.0.1 (the only change in the proof is that in (iii)
we use [AV19, Cor 10.9] instead of [AV19, Cor 7.4]).

Let $i:\mathfrak{p}^{*}\hookrightarrow\mathfrak{g}^{*}$ be the inclusion. The
restriction functor
$i^{*}:\operatorname{Coh}^{\widetilde{G}}(\mathfrak{g}^{*})\stackrel{{\scriptstyle}}{{\to}}\operatorname{Coh}^{\widetilde{K}}(\mathfrak{p}^{*})$
is not exact in general (if
$\mathcal{E}\in\operatorname{Coh}^{\widetilde{G}}(\mathfrak{g}^{*})$, then
$i^{*}\mathcal{E}$ corresponds to the $\mathbb{C}[\mathfrak{p}^{*}]$-module
$\mathbb{C}[\mathfrak{p}^{*}]\otimes_{\mathbb{C}[\mathfrak{g}^{*}]}\mathcal{E}$.
The functor
$\mathbb{C}[\mathfrak{p}^{*}]\otimes_{\mathbb{C}[\mathfrak{g}^{*}]}(\bullet)$
is only right exact). Write $L_{n}i^{*}$ for its higher derived functors (if
$\mathcal{E}\in\operatorname{Coh}^{\widetilde{G}}(\mathfrak{g}^{*})$, then
$L_{n}i^{*}\mathcal{E}$ corresponds to the
$\mathbb{C}[\mathfrak{p}^{*}]$-module
$\mathrm{Tor}_{n}^{\mathbb{C}[\mathfrak{g}^{*}]}(\mathbb{C}[\mathfrak{p}^{*}],\mathcal{E})$).
Since $\mathfrak{g}^{*}$ is smooth, $L_{n}i^{*}\mathcal{E}=0$ for $n$ very
large (see e.g. [CG10, Prop 5.1.28]). So we can define a homomorphism

$i^{*}:K^{\widetilde{G}}(\mathfrak{g}^{*})\stackrel{{\scriptstyle}}{{\to}}K^{\widetilde{K}}(\mathfrak{p}^{*}),\qquad
i^{*}[\mathcal{E}]=\sum_{n=0}^{\infty}(-1)^{n}[L_{n}i^{*}\mathcal{E}].$

If $[\mathcal{E}]$ is supported in $\mathcal{N}$, then $i^{*}[\mathcal{E}]$ is
supported in $\mathcal{N}_{\theta}$. So
$i^{*}:K^{\widetilde{G}}(\mathfrak{g}^{*})\stackrel{{\scriptstyle}}{{\to}}K^{\widetilde{K}}(\mathfrak{p}^{*})$
restricts to a (unique) homomorphism
$i^{*}:K^{\widetilde{G}}(\mathcal{N})\stackrel{{\scriptstyle}}{{\to}}K^{\widetilde{K}}(\mathcal{N}_{\theta})$

(3.0.2)
${K^{\widetilde{G}}(\mathcal{N})}$${K^{\widetilde{G}}(\mathfrak{g}^{*})}$${K^{\widetilde{K}}(\mathcal{N}_{\theta})}$${K^{\widetilde{K}}(\mathfrak{p}^{*})}$$\scriptstyle{j_{*}}$$\scriptstyle{i^{*}}$$\scriptstyle{i^{*}}$$\scriptstyle{j_{*}}$

We can compute these restriction maps in terms of
$\widetilde{K}$-representations. The key ingredient is a graded Koszul
identity in $K_{a}(\widetilde{K})$. Consider the class

$[\wedge(\mathfrak{k})]:=\sum_{n=0}^{\infty}(-1)^{n}[\wedge^{n}(\mathfrak{k})]\in
K_{f}(\widetilde{K})$

Here, as usual, we put $\mathfrak{k}$ in degree $1$

######  Lemma 3.0.4.

There is an identity in $K_{a}(\widetilde{K})$

$\mathbb{C}[\mathfrak{k}^{*}]|_{\widetilde{K}}\otimes[\wedge(\mathfrak{k})]=\mathrm{triv}$

###### Proof.

Consider the Koszul resolution of the trivial
$\mathbb{C}[\mathfrak{k}^{*}]$-module

(3.0.3)
$0\stackrel{{\scriptstyle}}{{\to}}\mathbb{C}[\mathfrak{k}^{*}]\otimes\wedge^{\dim(\mathfrak{k})}(\mathfrak{k})\stackrel{{\scriptstyle}}{{\to}}...\stackrel{{\scriptstyle}}{{\to}}\mathbb{C}[\mathfrak{k}^{*}]\otimes\wedge^{1}(\mathfrak{k})\stackrel{{\scriptstyle}}{{\to}}\mathbb{C}[\mathfrak{k}^{*}]\otimes\wedge^{0}(\mathfrak{k})\stackrel{{\scriptstyle}}{{\to}}\mathbb{C}\stackrel{{\scriptstyle}}{{\to}}0$

We can regard each term as a representation of $\widetilde{K}$, and it is easy
to check that the differentials are $\widetilde{K}$-equivariant. Restricting
to $\widetilde{K}$ we get an exact sequence in
$\operatorname{Rep}_{a}(\widetilde{K})$

$0\stackrel{{\scriptstyle}}{{\to}}\mathbb{C}[\mathfrak{k}^{*}]|_{\widetilde{K}}\otimes\wedge^{\dim(\mathfrak{k})}(\mathfrak{k})\stackrel{{\scriptstyle}}{{\to}}...\stackrel{{\scriptstyle}}{{\to}}\mathbb{C}[\mathfrak{k}^{*}]|_{\widetilde{K}}\otimes\wedge^{1}(\mathfrak{k})\stackrel{{\scriptstyle}}{{\to}}\mathbb{C}[\mathfrak{k}^{*}]|_{\widetilde{K}}\otimes\wedge^{0}(\mathfrak{k})\stackrel{{\scriptstyle}}{{\to}}\mathrm{triv}\stackrel{{\scriptstyle}}{{\to}}0$

Now the required identity follows from the Euler-Poincare principle

$\mathbb{C}[\mathfrak{k}^{*}]|_{\widetilde{K}}\otimes[\wedge(\mathfrak{k})]=\sum_{n}(-1)^{n}\mathbb{C}[\mathfrak{k}^{*}]|_{\widetilde{K}}\otimes[\wedge^{n}(\mathfrak{k})]=\mathrm{triv}$

∎

If $V\in\operatorname{Rep}_{a}(\widetilde{G})$, then
$V|_{\widetilde{K}}\in\operatorname{Rep}_{a}(\widetilde{K})$. This defines an
exact functor
$\operatorname{Rep}_{a}(\widetilde{G})\stackrel{{\scriptstyle}}{{\to}}\operatorname{Rep}_{a}(\widetilde{K})$,
and hence a group homomorphism

$|_{\widetilde{K}}:K_{a}(\widetilde{G})\stackrel{{\scriptstyle}}{{\to}}K_{a}(\widetilde{K})$

Tensoring with the class $[\wedge(\mathfrak{k})]\in K_{f}(\widetilde{K})$, we
obtain a further homomorphism

$r:K_{a}(\widetilde{G})\stackrel{{\scriptstyle}}{{\to}}K_{a}(\widetilde{K}),\qquad
r[V]=[V]|_{\widetilde{K}}\otimes[\wedge(\mathfrak{k})]$

######  Lemma 3.0.5.

The following diagram is commutative

${K_{a}(\widetilde{G})}$${K_{f}(\widetilde{G})}$${K_{a}(\widetilde{K})}$${K_{f}(\widetilde{K})}$$\scriptstyle{r}$$\scriptstyle{|_{\widetilde{K}}}$$\scriptstyle{\phi_{\mathfrak{g}^{*}}}$$\scriptstyle{\phi_{\mathfrak{p}^{*}}}$

###### Proof.

Let $V\in K_{f}(\widetilde{G})$. Then by Lemma 3.0.4 we have

$\displaystyle\phi_{\mathfrak{g}^{*}}(V)|_{\widetilde{K}}\otimes[\wedge(\mathfrak{k})]$
$\displaystyle=(V\otimes\mathbb{C}[\mathfrak{g}^{*}])|_{\widetilde{K}}\otimes[\wedge(\mathfrak{k})]$
$\displaystyle=V|_{\widetilde{K}}\otimes\mathbb{C}[\mathfrak{p}^{*}]|_{\widetilde{K}}\otimes\mathbb{C}[\mathfrak{k}^{*}]|_{\widetilde{K}}\otimes[\wedge(\mathfrak{k})]$
$\displaystyle=V|_{\widetilde{K}}\otimes\mathbb{C}[\mathfrak{p}^{*}]|_{\widetilde{K}}$
$\displaystyle=\phi_{\mathfrak{p}^{*}}(V|_{\widetilde{K}})$

as desired. ∎

######  Lemma 3.0.6.

The following diagram is commutative

${K^{\widetilde{G}}(\mathfrak{g}^{*})}$${K_{f}(\widetilde{G})}$${K^{\widetilde{K}}(\mathfrak{p}^{*})}$${K_{f}(\widetilde{K})}$$\scriptstyle{i^{*}}$$\scriptstyle{|_{\\{0\\}}}$$\scriptstyle{|_{\widetilde{K}}}$$\scriptstyle{|_{\\{0\\}}}$

###### Proof.

If $\mathcal{V}\simeq
V\otimes\mathcal{O}_{\mathfrak{g}^{*}}\in\operatorname{Coh}^{\widetilde{G}}(\mathfrak{g}^{*})$
is a vector bundle, then $L_{n}i^{*}\mathcal{V}=0$ for $n>0$ ($\mathcal{V}$
corresponds to a flat, and hence projective,
$\mathbb{C}[\mathfrak{g}^{*}]$-module, so all higher Tor groups vanish).
Consequently

$\displaystyle(i^{*}[\mathcal{V}])|_{\\{0\\}}$
$\displaystyle=[i^{*}\mathcal{V}]|_{\\{0\\}}$
$\displaystyle=(V|_{\widetilde{K}}\otimes\mathcal{O}_{\mathfrak{p}^{*}})|_{\\{0\\}}$
$\displaystyle=V|_{\widetilde{K}}$
$\displaystyle=([\mathcal{V}]|_{\\{0\\}})|_{\widetilde{K}}$

So the diagram commutes on vector bundles. But since $\mathfrak{g}^{*}$ is
smooth, $K^{\widetilde{G}}(\mathfrak{g}^{*})$ is spanned by vector bundles.
This completes the proof. ∎

The next result gives a method for computing
$i^{*}:K^{\widetilde{G}}(\mathcal{N})\stackrel{{\scriptstyle}}{{\to}}K^{\widetilde{K}}(\mathcal{N}_{\theta})$
on the level of $\widetilde{K}$-representations.

######  Corollary 3.0.7.

The following diagram is commutative

${K_{a}(\widetilde{G})}$${K_{f}(\widetilde{G})}$${K^{\widetilde{G}}(\mathcal{N})}$${K^{\widetilde{G}}(\mathfrak{g}^{*})}$${K_{a}(\widetilde{K})}$${K_{f}(\widetilde{K})}$${K^{\widetilde{K}}(\mathcal{N}_{\theta})}$${K^{\widetilde{K}}(\mathfrak{p}^{*})}$$\scriptstyle{r}$$\scriptstyle{\phi_{\mathfrak{g}^{*}}}$$\scriptstyle{|_{\widetilde{K}}}$$\scriptstyle{j_{*}}$$\scriptstyle{i^{*}}$$\scriptstyle{\Gamma}$$\scriptstyle{|_{\\{0\\}}}$$\scriptstyle{\phi_{\mathfrak{p}^{*}}}$$\scriptstyle{\Gamma}$$\scriptstyle{j_{*}}$$\scriptstyle{i^{*}}$$\scriptstyle{|_{\\{0\\}}}$

In particular, for every $[\mathcal{E}]\in K^{\widetilde{G}}(\mathcal{N})$,
the restriction $i^{*}[\mathcal{E}]$ is uniquely determined by the following
identity in $K_{a}(\widetilde{K})$

(3.0.4)
$\Gamma(i^{*}[\mathcal{E}])|_{\widetilde{K}}=\Gamma([\mathcal{E}])|_{\widetilde{K}}\otimes[\wedge(\mathfrak{k})]$

###### Proof.

The front face is commutative by the definition of
$i^{*}:K^{\widetilde{G}}(\mathcal{N})\stackrel{{\scriptstyle}}{{\to}}K^{\widetilde{K}}(\mathcal{N}_{\theta})$,
see (3.0.2). The right face is commutative by Lemma 3.0.6. The back face is
commutative by Lemma 3.0.5. The top face is commutative by Proposition 3.0.1.
The bottom face is commutative by its analog for $\widetilde{K}$. The
commutativity of the left face follows as a formal consequence of the
commutativity of the others. ∎

######  Remark 3.0.8.

There are surjective ‘forgetful’ maps

$K^{\widetilde{G}}(\mathcal{N})\twoheadrightarrow K^{G}(\mathcal{N}),\qquad
K^{\widetilde{K}}(\mathcal{N}_{\theta})\twoheadrightarrow
K^{K}(\mathcal{N}_{\theta})$

It is not hard to show that the restriction map
$i^{*}:K^{\widetilde{G}}(\mathcal{N})\stackrel{{\scriptstyle}}{{\to}}K^{\widetilde{K}}(\mathcal{N}_{\theta})$
descends to a (necessarily unique) homomorphism

$i^{*}:K^{G}(\mathcal{N})\stackrel{{\scriptstyle}}{{\to}}K^{K}(\mathcal{N}_{\theta})$

Since we will not use this fact, we will not prove it here. Write
$K\mathcal{M}(G_{\mathbb{R}})$ for the Grothendieck group of finite-length
admissible $G_{\mathbb{R}}$-representations. Similarly, write
$K\mathcal{M}(G)$ (here $G$ is regarded as a real reductive group by
restriction of scalars). There are ‘associated graded’ maps

$\operatorname{gr}:K\mathcal{M}(G)\stackrel{{\scriptstyle}}{{\to}}K^{G}(\mathcal{N}),\qquad\operatorname{gr}:K\mathcal{M}(G_{\mathbb{R}})\stackrel{{\scriptstyle}}{{\to}}K^{K}(\mathcal{N}_{\theta})$

(see [Vog91] for definitions). An intriguing question, which we will not
pursue in this paper, is whether there is a natural homomorphism
$K\mathcal{M}(G)\overset{?}{\stackrel{{\scriptstyle}}{{\to}}}K\mathcal{M}(G_{\mathbb{R}})$
such that the following diagram commutes

${K\mathcal{M}(G)}$${K\mathcal{M}(G_{\mathbb{R}})}$${K^{G}(\mathcal{N})}$${K^{K}(\mathcal{N}_{\theta})}$$\scriptstyle{?}$$\scriptstyle{\operatorname{gr}}$$\scriptstyle{\operatorname{gr}}$$\scriptstyle{i^{*}}$

##  4\. Regular functions on $\mathcal{N}$

In this section, we will recall a (well-known) formula for
$\mathbb{C}[\mathcal{N}]$ as a representation of $\widetilde{G}$. Choose a
maximal torus $H\subset G$ and a Borel subgroup $B\subset G$ containing $H$.
Let $\Lambda\subset\mathfrak{h}^{*}$ denote the weight lattice,
$\Phi^{+}\subset\Lambda$ the positive roots, and $\Lambda^{+}\subset\Lambda$
the dominant weights. Then

$\mathrm{Irr}(G)=\\{\tau_{\lambda}\mid\lambda\in\Lambda^{+}\\},$

where $\tau_{\lambda}$ is the irreducible representation of $G$ with highest
weight $\lambda$. Recall _Kostant’s partition function_

$\mathcal{P}:\Lambda\stackrel{{\scriptstyle}}{{\to}}\mathbb{Z},\qquad\mathcal{P}(\lambda)=\\#\\{m:\Phi^{+}\stackrel{{\scriptstyle}}{{\to}}\mathbb{Z}\mid\lambda=\sum_{\alpha\in\Phi^{+}}m(\alpha)\alpha\\}$

Define

$\mathcal{M}:\Lambda^{+}\times\Lambda\stackrel{{\scriptstyle}}{{\to}}\mathbb{Z},\qquad\mathcal{M}(\lambda,\mu)=\sum_{w\in
W}(-1)^{\ell(w)}\mathcal{P}(w(\lambda+\rho)-(\mu+\rho))$

where $\ell:W\stackrel{{\scriptstyle}}{{\to}}\mathbb{Z}_{\geq 0}$ is the
length function and $\rho=\frac{1}{2}\sum\Phi^{+}$. For each
$\lambda\in\Lambda^{+}$, there is an identity in $K_{a}(H)$

(4.0.1)
$\tau_{\lambda}|_{H}=\sum_{\mu\in\Lambda}\mathcal{M}(\lambda,\mu)e^{\mu}$

This is a version of the Weyl character formula.

Lusztig has introduced $q$-analogs of both $\mathcal{P}$ and $\mathcal{M}$
([Lus83]). The $q$-analog of $\mathcal{P}$ is defined by the formula

$\mathcal{P}_{q}:\Lambda\stackrel{{\scriptstyle}}{{\to}}\mathbb{Z}[q],\qquad\mathcal{P}_{q}(\lambda)=\sum_{n=0}^{\infty}\mathcal{P}_{q}^{n}(\lambda)q^{n}$

where

$\mathcal{P}_{q}^{n}(\lambda)=\\#\\{m:\Phi^{+}\stackrel{{\scriptstyle}}{{\to}}\mathbb{Z}\mid\lambda=\sum_{\alpha\in\Phi^{+}}m(\alpha)\alpha\
\text{ and }\ n=\sum_{\alpha\in\Phi^{+}}m(\alpha)\\}$

The $q$-analog of $\mathcal{M}$ is

$\mathcal{M}_{q}:\Lambda^{+}\times\Lambda\stackrel{{\scriptstyle}}{{\to}}\mathbb{Z}[q],\qquad\mathcal{M}_{q}(\lambda,\mu)=\sum_{w\in
W}(-1)^{\epsilon(w)}\mathcal{P}_{q}(w(\lambda+\rho)-(\mu+\rho))$

The following can be extracted from [McG89].

######  Proposition 4.0.1.

There is an identity in $K_{a}(\widetilde{G})$

$\mathbb{C}[\mathcal{N}]|_{\widetilde{G}}=\sum_{\lambda\in\Lambda^{+}}\tau_{\lambda}\mathcal{M}_{q}(\lambda,0)$

###### Sketch of proof.

Consider the Springer resolution
$\eta:T^{*}(G/B)\stackrel{{\scriptstyle}}{{\to}}\mathcal{N}$. Since
$T^{*}(G/B)$ is symplectic, there is an identification (of
$\widetilde{G}$-equivariant sheaves)

$\mathcal{O}_{T^{*}(G/B)}\simeq\omega_{T^{*}(G/B)}$

where $\omega_{T^{*}(G/B)}$ is the canonical sheaf on $T^{*}(G/B)$. So by the
theorem of Grauert and Riemenschneider ([GR70])

$R^{i}\eta_{*}\mathcal{O}_{T^{*}(G/B)}=0,\qquad\forall i>0$

On the other hand,
$R^{0}\eta_{*}\mathcal{O}_{T^{*}(G/B)}\simeq\mathcal{O}_{\mathcal{N}}$. Using
the Leray spectral sequence (and the fact that $\mathcal{N}$ is affine), we
get an identity in $K_{a}(\widetilde{G})$

(4.0.2)
$\mathbb{C}[\mathcal{N}]|_{\widetilde{G}}=\sum_{i}(-1)^{i}H^{i}(T^{*}(G/B),\mathcal{O}_{T^{*}(G/B)})$

If $p:T^{*}(G/B)\stackrel{{\scriptstyle}}{{\to}}G/B$ is the projection, then
$p_{*}\mathcal{O}_{T^{*}(G/B)}$ is identified (as a
$\widetilde{G}$-equivariant sheaf) with (the sheaf of local sections of) the
$\widetilde{G}$-equivariant vector bundle
$G\times_{B}S(\mathfrak{g}/\mathfrak{b})$. Since $p$ is affine, the direct
image functor $p_{*}$ preserves cohomology, i.e.

(4.0.3) $H^{i}(T^{*}(G/B),\mathcal{O}_{T^{*}(G/B)})\simeq
H^{i}(G/B,p_{*}\mathcal{O}_{T^{*}(G/B)}),\qquad\forall i\geq 0$

Combining (4.0.2) and (4.0.3), we get a further identity in
$K_{a}(\widetilde{G})$

$\mathbb{C}[\mathcal{N}]|_{\widetilde{G}}=\sum_{i}(-1)^{i}H^{i}(G/B,G\times_{B}S(\mathfrak{g}/\mathfrak{b}))$

The right hand side can be computed using Borel-Weil-Bott. The result follows.
∎

##  5\. Some commutative algebra

Using the results of Section 3, we get a well-defined class
$i^{*}[\mathcal{O}_{\mathcal{N}}]\in K^{\widetilde{K}}(\mathcal{N}_{\theta})$,
which can be regarded as the restriction of $\mathcal{O}_{\mathcal{N}}$ to
$\mathfrak{p}^{*}$. However, it is not at all clear that
$i^{*}[\mathcal{O}_{\mathcal{N}}]=[\mathcal{O}_{\mathcal{N}_{\theta}}]$. For
groups split modulo center, we will see that this equality always holds, but
the proof will require some commutative algebra.

######  Lemma 5.0.1.

Let $R$ be a Noetherian ring and let $M$ be a finitely-generated $R$-module.
Suppose $x_{1},...,x_{m}\in R$ is an $R$-regular sequence which is also
$M$-regular. Consider the ideal $I=(x_{1},...,x_{m})\subset R$. Then

$\mathrm{Tor}^{R}_{n}(R/I,M)=0,\qquad n>0$

###### Proof.

Since $x_{1},...,x_{m}$ is $R$-regular, the Koszul complex
$K(x_{1},...,x_{m};R)$ is a resolution of $R/I$. So
$\mathrm{Tor}_{\bullet}^{R}(R/I,M)$ is the homology of
$K(x_{1},...,x_{m};M):=K(x_{1},...,x_{m};R)\otimes_{R}M$

$H_{n}(K(x_{1},...,x_{m};M))\simeq\mathrm{Tor}_{n}^{R}(R/I,M),\qquad\forall n$

But since $x_{1},...,x_{m}$ is an $M$-regular sequence, the complex
$K(x_{1},...,x_{m};M)$ is acyclic, see [Mat87, Thm 16.5(i)]. So
$\mathrm{Tor}_{n}^{R}(R/I,M)=0$ for $n>0$. ∎

######  Lemma 5.0.2 (Thm 17.4, [Mat87]).

Suppose $A$ is Cohen-Macaulay, and let $I=(x_{1},...,x_{n})\subset A$ be an
ideal. If

$\dim(A/I)=\dim(A)-n$

then $x_{1},...,x_{n}$ is an $A$-regular sequence.

######  Proposition 5.0.3.

Suppose $X$ is a smooth Noetherian scheme and let $Y$, $Z$ be closed
subschemes of $X$. Write $i:Y\hookrightarrow X$ and $j:Z\hookrightarrow X$ for
the inclusions and form the Cartesian diagram of schemes

${Z}$${X}$${Z\cap Y}$${Y}$$\scriptstyle{j}$$\scriptstyle{i}$

Assume

  * (i)

$Y$ is smooth.

  * (ii)

$Z$ is Cohen-Macaulay.

  * (iii)

$\dim(Z\cap Y)=\dim(Z)+\dim(Y)-\dim(X)$.

Then

(5.0.1) $L_{n}i^{*}(j_{*}\mathcal{O}_{Z})=0,\qquad n>0$

###### Proof.

The statement is local in $X$, so we can assume all schemes are affine. Let
$X=\mathrm{Spec}(R)$, $Y=\mathrm{Spec}(R/I)$ and $Z=\mathrm{Spec}(A)$, so that
$Z\cap Y=\mathrm{Spec}(A/I)$. Since $Y$ is smooth, we can find an $R$-regular
sequence $x_{1},...,x_{m}\in R$ such that $I=(x_{1},...,x_{m})$, where
$m=\dim(X)-\dim(Y)$. Now by Lemma 5.0.2, $(x_{1},...,x_{m})$ is an $A$-regular
sequence. So by Lemma 5.0.1 (applied to the $R$-module $M=A$)

$\mathrm{Tor}_{n}^{R}(R/I,A)=0,\qquad n>0$

This is equivalent to (5.0.1). ∎

##  6\. Regular functions on $\mathcal{N}_{\theta}$

First, we will assume that $G_{\mathbb{R}}$ is split. This means that $G$
contains a maximal torus $H_{s}\subset G$ on which $\theta$ acts by inversion.
Our first lemma shows that the codimension of
$\mathcal{N}_{\theta}\subset\mathcal{N}$ equals the codimension of
$\mathfrak{p}^{*}\subset\mathfrak{g}^{*}$.

######  Lemma 6.0.1.

$\dim(\mathcal{N}_{\theta})=\dim(\mathcal{N})+\dim(\mathfrak{p})-\dim(\mathfrak{g})$

###### Proof.

By [KR71, Prop 9]

(6.0.1) $\dim(\mathcal{N}_{\theta})=\dim(\mathfrak{p})-\dim(\mathfrak{h}_{s})$

By the Iwasawa decomposition

(6.0.2)
$\dim(\mathfrak{g})=\dim(\mathfrak{k})+\dim(\mathfrak{h}_{s})+\dim(\mathfrak{n})$

Finally

(6.0.3)
$\dim(\mathcal{N})=2\dim(\mathfrak{n})=\dim(\mathfrak{g})-\dim(\mathfrak{h}_{s})$

Combining (6.0.1), (6.0.2), and (6.0.3) proves the lemma. ∎

######  Corollary 6.0.2.

There is an equality in $K^{\widetilde{K}}(\mathcal{N}_{\theta})$

(6.0.4)
$[\mathcal{O}_{\mathcal{N}_{\theta}}]=i^{*}[\mathcal{O}_{\mathcal{N}}]$

and hence an equality in $K_{aa}(\widetilde{K})$

(6.0.5)
$\mathbb{C}[\mathcal{N}_{\theta}]|_{\widetilde{K}}=\mathbb{C}[\mathcal{N}]|_{\widetilde{K}}\otimes[\wedge(\mathfrak{k})]$

###### Proof.

For (6.0.4), we will apply Proposition 5.0.3. So let $X=\mathfrak{g}^{*}$,
$Z=\mathcal{N}$, and $Y=\mathfrak{p}^{*}$. Clearly, $X$ and $Y$ are smooth. By
[Kos63, Thm 0.1], $\mathcal{N}$ is a complete intersection, and therefore
Cohen-Macaulay. Condition (iii) of Proposition 5.0.3 is the content of Lemma
6.0.1. So by Proposition 5.0.3, we have

$L_{n}i^{*}(j_{*}\mathcal{O}_{\mathcal{N}})=0,\qquad n>0$

and therefore

$i^{*}[\mathcal{O}_{\mathcal{N}}]=\sum_{n}(-1)^{n}[L_{n}i^{*}j_{*}\mathcal{O}_{\mathcal{N}}]=[i^{*}j_{*}\mathcal{O}_{\mathcal{N}}]=[\mathcal{O}_{\mathcal{N}_{\theta}}]$

For the final equality, we use the well-known fact that scheme-theoretic
intersection $\mathcal{N}\cap\mathfrak{p}^{*}$ is reduced, see [KR71, Theorem
14]. This proves (6.0.4). Now (6.0.5) follows from (6.0.4) and Corollary
3.0.7. ∎

Corollary 6.0.2 can be easily extended to the case when $G_{\mathbb{R}}$ is
split modulo center.

######  Example 6.0.3.

Let $G=\mathrm{SL}_{2}(\mathbb{C})$ and let $\theta(g)=(g^{-1})^{t}$ (this is
the involution corresponding to split real form
$\mathrm{SL}_{2}(\mathbb{R})$). Then $K=\mathrm{SO}_{2}(\mathbb{C})$. Write

$\mathrm{Irr}(G)=\\{\tau_{m}\mid
m=0,1,2,...\\},\qquad\mathrm{Irr}(K)=\\{\chi_{n}\mid n\in\mathbb{Z}\\}$

(here $\tau_{m}$ is the irreducible with highest weight $m$ and $\chi_{n}$ is
the degree-$n$ character of $\mathrm{SO}_{2}(\mathbb{C})$). We have the
following branching rules

$\tau_{m}|_{K}=\chi_{-2m}+\chi_{-2m+2}+...+\chi_{2m}.$

From Proposition 4.0.1, we deduce

$\mathbb{C}[\mathcal{N}]|_{\widetilde{G}}=\sum_{m=0}^{\infty}\tau_{2m}q^{m}$

Also

$[\wedge(\mathfrak{k})]=\chi_{0}-\chi_{0}q.$

So by Corollary 6.0.2

$\displaystyle\mathbb{C}[\mathcal{N}_{\theta}]|_{\widetilde{K}}$
$\displaystyle=\mathbb{C}[\mathcal{N}]|_{\widetilde{K}}\otimes[\wedge(\mathfrak{k})]$
$\displaystyle=(\sum_{m=0}^{\infty}(\chi_{-2m}...+\chi_{2m})q^{m})\otimes(\chi_{0}-\chi_{0}q)$
$\displaystyle=\sum_{m=0}^{\infty}((\chi_{-2m}+...+\chi_{2m})-(\chi_{-2m+2}+...+\chi_{2m-2}))q^{m}$
$\displaystyle=\sum_{m=0}^{\infty}(\chi_{2m}+\chi_{-2m})q^{m}$

##  7\. Branching to $K$

In this section, we will use Corollary 6.0.2 to compute
$\mathbb{C}[\mathcal{N}_{\theta}]|_{\widetilde{K}}$ as a formal integer
combination of classes of the form $I(\Gamma)q^{n}$.

Suppose $(H,\gamma,\Phi^{+})$ is a continued Langlands parameter (see
Definition 2.0.1) and let $\chi\in K_{f}(K)$. It is easy to compute the tensor
product $[I(H,\gamma,\Phi^{+})]|_{K}\otimes\chi$ as a representation of $K$.

######  Lemma 7.0.1 (Lem 12.13, [Vog07]).

Choose a finite multiset $S_{H}(\chi)$ in $X^{*}(H)$ such that

$\chi|_{H^{\theta}}=\sum_{\mu\in S_{H}(\chi)}\mu|_{H^{\theta}}$

Then there is an identity in $K_{a}(K)$

$[I(H,\gamma,\Phi^{+})]|_{K}\otimes\chi=\sum_{\mu\in
S_{H}(\chi)}[I(H,\gamma+\mu,\Phi^{+})]$

We will use Lemma 7.0.1 (together with Zuckerman’s character formula for the
trivial representation) to compute
$\mathbb{C}[\mathcal{N}_{\theta}]|_{\widetilde{K}}$ in terms of the classes
$I(\Gamma)q^{n}$. For an arbitrary class $\chi\in K_{f}(K)$, it may be
difficult to find a multiset $S_{H}(\chi)$ as in Lemma 7.0.1. Fortunately, in
our setting, $\chi$ is _not_ arbitrary. For the problem at hand, we will need
to compute $S_{H}(\chi)$ in the following two cases:

  1. (1)

$\chi$ is the restriction to $K$ of an irreducible representation
$\tau_{\lambda}$ of $G$.

  2. (2)

$\chi$ is the class $\wedge^{n}(\mathfrak{k})$.

First, suppose $\chi=\tau_{\lambda}|_{K}$. By the Weyl character formula
(4.0.1), we have

$\tau_{\lambda}|_{H}=\sum_{\mu\in X^{*}(H)}\mathcal{M}(\lambda,\mu)e^{\mu}$

So we can take

$S_{H}(\tau_{\lambda})=\\{\mathcal{M}(\lambda,\mu)e^{\mu}\mid\mu\in
X^{*}(H)\\}$

(the coefficients $\mathcal{M}(\lambda,\mu)$ above denote the multiset
multiplicities. We will use similar notation below).

Next, suppose $\chi=\mathfrak{k}$. Choose a subset
$\Delta_{\mathbb{C}}^{\prime}\subset\Delta_{\mathbb{C}}$ such that for each
$\alpha\in\Delta_{\mathbb{C}}$, exactly one of $\\{\alpha,\theta\alpha\\}$
appears in $\Delta^{\prime}_{\mathbb{C}}$. Then there is a decomposition of
$\mathfrak{k}$ into weight spaces for $H^{\theta}$

$\mathfrak{k}\simeq\bigoplus_{\alpha\in\Delta_{c}}\mathfrak{g}_{\alpha}\oplus\bigoplus_{\alpha\in\Delta_{\mathbb{C}}^{\prime}}(1+d\theta)\mathfrak{g}_{\alpha}\oplus\bigoplus_{\alpha\in\Delta^{+}_{\mathbb{R}}}(1+d\theta)\mathfrak{g}_{\alpha}$

So we can take

$S_{H}(\mathfrak{k})=\\{e^{\alpha}\mid\alpha\in\Delta_{c}\\}\cup\\{e^{\alpha}\mid\alpha\in\Delta_{\mathbb{C}}^{\prime}\\}\cup\\{|\Delta_{\mathbb{R}}^{+}|e^{0}\\}$

More generally

$S_{H}(\wedge^{n}\mathfrak{k})=\\{\sum R\mid R\subseteq S_{H}(\mathfrak{k}),\
|R|=n\\}$

Zuckerman’s character formula for the trivial representation (see [Vog81, Thm
9.4.16]) can be interpreted as an identity in $K_{a}(K)$

$\mathrm{triv}=\sum_{H}\sum_{\Delta^{+}}(-1)^{\ell(\Delta^{+})}[I(H,\rho_{i\mathbb{R}},\Delta_{i\mathbb{R}}^{+})]|_{K}$

The outer sum runs over $K$-conjugacy classes of $\theta$-stable maximal tori
$H\subset G$ and the inner sum over $W(K,H^{\theta})$-conjugacy classes of
positive systems $\Delta^{+}\subset\Delta(G,H)$. The integer
$\ell(\Delta^{+})$ is the codimension of the $K$-orbit on the flag variety
containing the Borel subalgebra $\mathfrak{b}\subset\mathfrak{g}$
corresponding to the positive system $\Delta^{+}$. Now assume that
$G_{\mathbb{R}}$ is split modulo center. Using Corollary 6.0.2, we obtain an
identity in $K_{a}(\widetilde{K})$

$\mathbb{C}[\mathcal{N}_{\theta}]|_{\widetilde{K}}=\left(\sum_{H}\sum_{\Delta^{+}}(-1)^{\ell(\Delta^{+})}[I(H,\rho_{i\mathbb{R}},\Delta_{i\mathbb{R}}^{+})]|_{K}\right)\otimes\mathbb{C}[\mathcal{N}]|_{\widetilde{K}}\otimes[\wedge(\mathfrak{k})]$

Using Proposition 4.0.1 and Lemma 7.0.1, we can rewrite the right hand side in
terms of classes of the form $[I(\Gamma)]q^{n}$

######  Theorem 7.0.2.

Assume $G_{\mathbb{R}}$ is split modulo center. Then there is an identity in
$K_{a}(\widetilde{K})$

$\mathbb{C}[\mathcal{N}_{\theta}]|_{\widetilde{K}}=\sum_{H}\sum_{\Delta^{+}}\sum_{\lambda\in\Lambda^{+}}\sum_{\mu\in\Lambda}\sum_{R\subseteq
S_{H}(\mathfrak{k})}(-1)^{\ell(\Delta^{+})+|R|}\mathcal{M}(\lambda,\mu)\mathcal{M}_{q}(\lambda,0)[I(H,\rho_{i\mathbb{R}}+\mu+|R|,\Delta_{i\mathbb{R}}^{+})]q^{|R|}$

Note that the terms on the right are not final. We can rewrite the sum in
terms of final parameters using the Hecht-Schmid identities (this sort of
thing is easy to do in atlas).

## References

  * [AVTV20] J. Adams, M. Van Leeuwen, P. Trapa and D. Vogan  “Unitary representations of real reductive groups”  In _Asterisque_ , 2020
  * [AV19] J. Adams and D. Vogan  “Associated Varieties for Real Reductive Groups”  In _Pure Appl. Math. Q._ 0.0, 2019
  * [CG10] N. Chriss and V. Ginzburg  “Representation Theory and Complex Geometry”  Birkhäuser Boston, 2010
  * [GR70] H. Grauert and O. Riemenschneider  “Verschwindungssatze fur analytische Kohomologiegruppen auf komplexen Raumen”  In _Invent. Math._ 11, 1970
  * [Kos63] B. Kostant  “Lie Group Representations on Polynomial Rings”  In _Amer. J. Math._ 85.3, 1963
  * [Kos69] B. Kostant  “On the existence and irreducibility of certain series of representations”  In _Bull. Amer. Math. Soc._ 75.4 American Mathematical Society, 1969, pp. 627–642
  * [KR71] B. Kostant and S. Rallis  “Orbits and representations associated with symmetric spaces”  In _Amer. J. Math._ 93, 1971
  * [Lus83] G. Lusztig  “Singularities, character formulas, and a q-analog of weight multiplicities”  In _Astérisque_ 101-102, 1983
  * [Mat87] H. Matsumura  “Commutative Ring Theory”, Cambridge Studies in Advanced Mathematics  Cambridge University Press, 1987  DOI: 10.1017/CBO9781139171762
  * [McG89] W. McGovern  “Rings of regular functions on nilpotent orbits and their covers”  In _Invent. Math._ 97.1, 1989, pp. 209–217
  * [SV11] W. Schmid and K. Vilonen  “Hodge Theory and Unitary Representations of Reductive Lie Groups”  In _Fronteirs of Mathematical Sciences_ International Press, 2011, pp. 397–420
  * [SV80] B. Speh and D. Vogan  “Reducibility of generalized principal series representations”  In _Acta Math._ 145, 1980, pp. 227–299
  * [Tho87] R. Thomason  “Algebraic K-theory of group scheme actions”  In _Algebraic topology and algebraic K-theory_ 113 Ann. of Math. Stud., 1987
  * [Vog81] D. Vogan  “Representations of real reductive Lie groups” 15, Progr. Math.  Birkhäuser, 1981
  * [Vog91] D. Vogan  “Associated varieties and unipotent representations”  In _Harmonic analysis on reductive groups_ 101, Progr. Math.  Birkhäuser, 1991, pp. 315–388
  * [Vog07] D. Vogan  “Branching to a maximal compact subgroup”  In _Harmonic Analysis, Group Representations, Automorphic Forms, and Invariant Theory: In Honour of Roger E. Howe_ 12, Lecture Note Series, Institute for Mathematical Sciences, National University of Singapore, 2007


[affiliation=1 ⋆]XuLi [affiliation=1,2 ⋆]QiruiWang[affiliation=1 ⋆]XiaoyuLiu

# MaskSR: Masked Language Model for Full-band Speech Restoration

###### Abstract

Speech restoration aims at restoring high quality speech in the presence of a
diverse set of distortions. Although several deep learning paradigms have been
studied for this task, the power of the recently emerging language models has
not been fully explored. In this paper, we propose MaskSR, a masked language
model capable of restoring full-band 44.1 kHz speech jointly considering
noise, reverb, clipping, and low bandwidth. MaskSR works with discrete
acoustic tokens extracted using a pre-trained neural codec. During training,
MaskSR is optimized to predict randomly masked tokens extracted from the high
quality target speech, conditioned on the corrupted speech with various
distortions. During inference, MaskSR reconstructs the target speech tokens
with efficient iterative sampling. Extensive experiments show that MaskSR
obtains competitive results on both the full-band speech restoration task and
also on sub-tasks compared with a wide range of models.

###### keywords:

Speech restoration, Language model

##  1 Introduction

Speech restoration aims at restoring high quality speech from a corrupted
input signal considering a diverse set of distortions [1, 2, 3, 4, 5, 6, 7].
Compared with conventional denoising and dereverberation, the diverse nature
of the distortions (not just the quantity) makes this task much more
challenging. Regression models succeed in removing noise and reverb [8, 9,
10], but they cannot address tasks that are generative in nature, such as
bandwidth extension, packet loss concealment, etc. To ease the task, a two-
stage paradigm that employs separately trained models is widely adopted, in
which one suppresses noise, and another one generates missing speech [1, 2, 3,
4, 5]. Another variant jointly trains the two stages [6], but the success of
the speech generation stage heavily relies on the previous stage that employs
auxiliary losses to suppress the distortions. Thus, the power of generative
models as a unified framework that addresses the considered distortions all at
once has not been fully explored.††⋆ Equal contribution. Work done during
Qirui’s internship.

Recently, language models (LMs) have gained popularity in audio and image
synthesis due to their scalability, ease of training, and unification of
different modalities as discrete tokens [11, 12, 13, 14, 15]. Several works
also show that LMs can translate noisy speech tokens to clean tokens end-to-
end, providing an elegant framework [16, 17, 18], but only limited to
denoising. In addition, to our knowledge, previous speech denoising LMs work
with limited sampling rates up to 24 kHz [17]. Therefore, the capability of
LMs remains unknown for full-band speech restoration in the presence of a
diverse set of distortions, all considered under a single generative
framework.

In this work, we propose MaskSR, a full-band 44.1 kHz111Both 44.1 and 48 kHz
have been termed as full-band in previous research [8, 19]. For brevity, we do
not distinguish the two in this work. speech restoration system that performs
denoising, dereverberation, declipping, and bandwidth extension holistically.
As shown in Figure 1, MaskSR consists of a (frozen) pre-trained neural audio
codec to (de)tokenize the high quality target speech, a speech encoder to
encode a corrupted speech signal, and an LM conditioned on the encoded
corrupted speech to predict the masked acoustic tokens of the target speech.
During inference, MaskSR predicts the target speech tokens with efficient
iterative sampling. MaskSR obtains strong results on both the contributed
full-band speech restoration task evaluated with a blend of the studied
distortions, and also on individual tasks compared with a wide range of
models.

##  2 Method

Figure 1: Training workflow of MaskSR.

###  2.1 Neural Audio Tokenizer

We use the (frozen) pre-trained Descript Audio Codec (DAC) as our speech
(de)tokenizer [20]. DAC is a state-of-the-art auto-encoder. In the training
stage of MaskSR, the DAC encoder projects a 44.1 kHz high quality target
speech signal to $T$ frames in a latent space with a reduced sampling rate of
$\sim$86 Hz. Each frame is then tokenized by 9 residual vector quantizers
(RVQs) [21]. Each RVQ quantizes the error of the previous one with a codebook
size 1024. Thus, a waveform becomes a $9\times{T}$ codegram. During inference,
the DAC decoder detokenizes a codegram predicted by the LM to a waveform. DAC
is pre-trained to accurately reconstruct the unquantized waveform. Note that
although MaskSR can be divided into two stages: DAC and the rest, it's
fundamentally different than the previous two-stage systems that split the
removal of various distortions across cascaded models. Here, DAC only creates
a compact discrete space that encodes enough acoustic details of the full-band
speech to be restored, and it's the rest of the system that performs the
restoration jointly considering all the distortions.

###  2.2 Speech Encoder

The speech encoder first computes the power-law compressed magnitude STFT
spectrogram $X^{0.3}$ given a corrupted speech signal $x$ using a window
length and hop length of 2048 and 512 samples, respectively. The hop length is
consistent with that of the DAC encoder to align the STFT and DAC frames along
time. Next, a multi-layer perceptron (MLP) followed by a stack of self-
attention transformer blocks map the STFT features to $d$ dimensional
embeddings compatible with the DAC space, so that the speech encoder and the
LM could be jointly optimized.

###  2.3 Masked Language Model

Inspired by [22, 23], we extend the Masked Generative Image Transformer
(MaskGIT) [24] originated from modeling 1-D sequences to 2-D codegrams. During
training, after obtaining a $9\times{T}$ codegram from a full-band target
speech, we randomly mask a subset of tokens by replacing them with a special
mask token. Then, we use 9 learnable embedding tables each with 1025 entries
(including the mask token) to embed the 9 codebooks, respectively, each
resulting in a $T\times{d}$ tensor. Inspired by [22], we aggregate the
codebooks by summing the 9 tensors, and also sum the resulting embeddings with
the speech encoder representation of the corrupted speech. The summation of
the codebook embeddings keeps the sequence length unchanged despite of
multiple codebooks, thus minimizes the system complexity. The LM uses a stack
of self-attention transformer blocks to model the aggregated sequences. Due to
masking, the LM is free to learn from all the positions in the codegram, as
opposed to autoregressive LMs [12, 16]. Finally, an output layer consisting of
9 1024-dim softmax classifiers computes the logit scores of the tokens from
the 9 codebooks. The LM and the speech encoder are jointly optimized with a
cross entropy loss only applied to the masked positions.

During inference, starting from a fully masked codegram, we conduct iterative
sampling as in [24] to gradually generate the target speech tokens. In each
iteration, the LM predicts 1024-dim token probability distributions in all the
masked positions in parallel, given the tokens generated from previous
iterations. We sample a token in each masked position from the predicted
distribution, and re-mask a subset of the sampled tokens with low logit
scores. The percentage of re-masking is controlled by a cosine schedule. We
add Gaussian noise to the logit scores before ranking them to increase
diversity, and the variance linearly decreases from 4 to 0 throughout
inference. We perform 40 iterations until the codegram is fully reconstructed.

In addition, we use classifier-free guidance formulated for LMs [15, 25] on
the speech encoder representation of the corrupted speech to prevent the
generated speech from deviating too far from the original speaker voice.
During training, we randomly replace the speech encoder output 10% of the time
with a learnable embedding repeated $T$ times, and during inference, the logit
scores $l_{g}$ of the tokens are computed as:

$l_{g}=(1+w)l_{c}-wl_{u}$

where $l_{c}$ and $l_{u}$ are the conditional and unconditional logits,
respectively. A larger guidance $w\geq 0$ improves the speaker identity
preservation at the cost of slightly more residual noise.

###  2.4 Other Codebook Modeling Strategies

Since efficiently and effectively modeling multiple codebooks is crucial to
the system performance, we compare the parallel strategy in MaskSR with
another 2 representative variants.

SoundStorm [26] exploits the hierarchical nature of the RVQ, noting that the
low level codebooks help predicting the higher level ones. For each training
sample, only the tokens from a randomly selected codebook are randomly masked
and predicted in the MaskGIT fashion, whereas all the lower codebooks are
assumed available, and all the higher level codebooks are fully masked (but
not predicted). During inference, the codebooks are reconstructed
hierarchically, with the first one based on the MaskGIT iterative sampling,
and the rest simply taking the tokens with the highest probabilities.
SoundStorm only requires running the LM 8 more times (with 9 codebooks)
compared with MaskSR, thus only modestly increases inference time. But we
observe that the first codebook, which encodes the most salient speech
patterns, is not well modeled (Sec. 4.1).

UniAudio [16] is a recent LM that runs autoregressively along both the time
and codebook dimensions. Thus, it is much slower than MaskSR and SoundStorm
during inference. Also, the causal transformers in UniAudio only have access
to the past tokens which may hurt the modeling capability whereas masked LMs
do not have such a limitation.

##  3 Experimental Setup

Table 1: Full-band 44.1 kHz speech restoration results on the SR and ALL-GSR
test sets

System | Model size | SR clean test set for bandwidth extension | ALL-GSR test set with all 4 studied distortions
---|---|---|---
DNSMOS $\uparrow$ | SESQA $\uparrow$ | LSD $\downarrow$ | Spk Sim $\uparrow$ | DNSMOS $\uparrow$ | SESQA $\uparrow$ | LSD $\downarrow$ | Spk Sim $\uparrow$
SIG | BAK | OVL | SIG | BAK | OVL
Unprocessed | - | 3.413 | 4.025 | 3.107 | 2.577 | 2.889 | 0.808 | 2.961 | 2.857 | 2.393 | 2.598 | 2.014 | 0.901
Target-DAC | - | 3.472 | 4.044 | 3.174 | 3.488 | 0.837 | 0.933 | 3.455 | 3.981 | 3.143 | 3.533 | 0.827 | 0.931
NSNet2 | 2.8 M | 2.947 | 4.077 | 2.584 | 2.933 | 2.868 | 0.741 | 3.001 | 3.983 | 2.749 | 3.010 | 2.545 | 0.867
VoiceFixer | 111 M | 3.401 | 4.039 | 3.109 | 3.339 | 1.044 | 0.737 | 3.298 | 3.969 | 3.002 | 3.396 | 1.019 | 0.781
SoundStorm | 55 M | 3.423 | 4.001 | 3.117 | 3.426 | 1.062 | 0.812 | 3.395 | 3.973 | 3.085 | 3.485 | 1.171 | 0.833
UniAudio | 55 M | 3.415 | 4.022 | 3.110 | 3.447 | 1.036 | 0.792 | 3.403 | 4.026 | 3.117 | 3.538 | 1.363 | 0.815
MaskSR-S | 55 M | 3.442 | 4.017 | 3.135 | 3.430 | 0.978 | 0.822 | 3.430 | 3.982 | 3.123 | 3.541 | 1.201 | 0.845
MaskSR-M | 145 M | 3.440 | 4.021 | 3.136 | 3.467 | 0.959 | 0.832 | 3.445 | 3.971 | 3.128 | 3.531 | 1.191 | 0.853

###  3.1 Datasets

Training set We use $\sim$800 hours of publicly available clean speech
including the `read speech' and VCTK [27] subsets provided by the 2022 DNS
Challenge [28], and also the AISHELL-1 dataset [29]. We use 181 hours of noise
and 60 k room impulse responses (RIRs) also from [28]. All speech, noise, and
RIRs are recorded with 48 kHz or 44.1 kHz sampling rates, and we downsample
the data from 48 kHz to 44.1 kHz to be compatible with DAC. We consider 4
types of distortions as in [1]: noise, reverb, clipping, low bandwidth, and
create 44.1 kHz corrupted speech on the fly using the open-source pipeline in
[1], resulting in distorted samples with an SNR in $[-5,20]$ dB, clipped
between $[0.1,0.5]$, and a bandwidth from 1 kHz to 22.05 kHz.

Full-band test sets We use the 44.1 kHz open-source SR and ALL-GSR test sets
used by [1] to evaluate full-band models. SR contains clean data with
bandwidth between 1 kHz and 8 kHz, targeting at bandwidth extension only. ALL-
GSR contains a blend of the 4 studied distortions. Overlapping speakers that
also appear in VCTK are excluded from the training set.

Wide-band test sets To compare extensively with the majority of models that
only perform denoising and/or dereverberation at 16 kHz, we consider the 2020
DNS Challenge [30] test sets, including the synthetic data with and without
reverb, and the real recordings. To run MaskSR, we upsample the input speech
to 44.1 kHz, and downsample the output back to 16 kHz.

###  3.2 Implementation Details

We use the pre-trained DAC released in [20] as our speech tokenizer. A small
version MaskSR-S uses an embedding dimension $d=512$, sinusoidal positional
encoding, and there are 6 and 8 transformer blocks in the speech encoder and
the LM, respectively, each with 16 attention heads, an MLP with a hidden
dimension $4d$, and pre-norm. A medium size MaskSR-M uses $d=768$ and 12
transformer blocks in the LM. SoundStorm and UniAudio share the same overall
system (Figure 1) and model size as MaskSR-S to fairly compare codebook
modeling methods. We use the official UniAudio implementation in [16]. All
models are trained on 3 sec speech segments for 800 k steps on 4 A100 GPUs
with a learning rate of 0.0001 using the Adam optimizer. We use a batch size
256 for MaskSR-M and 64 for other models. During inference, we decode each
non-overlapping 3 sec window with 40 and 48 iterations for MaskSR and
SoundStorm, respectively, and 2331 iterations ($9\times{259}$) for UniAudio.

###  3.3 Baseline Models

Full-band models In addition to the 3 LMs, we consider 2 full-band models:
VoiceFixer [1] and NSNet2 [31]. VoiceFixer is a strong 2-stage speech
restoration model targeting at the same 4 distortions as in our work. NSNet2
is a regression-only model provided as the DNS Challenge baseline, performing
denoising and dereverberation. We use the model checkpoints from [1, 28]. The
released VoiceFixer was trained on a different dataset, but re-training
VoiceFixer on our data did not obtain better results on the full-band test
sets. Thus, we stick with the official checkpoint to report the results.

Wide-band models MaskSR is compared with a collection of models specializing
in denoising on the 16 kHz wide-band test sets. We obtain the released DEMUCS
and FRCRN checkpoints from [32, 8] as strong regression candidates. We use the
results reported in Wang et al. [18] for SGMSE [33], StoRM [34], and SELM
[18]. The former two are diffusion models and the third is a recent speech
enhancement LM. All the models are trained on datasets comparable to ours. In
addition, DEMUCS also jointly performs dereverberation.

Unprocessed and Target-DAC refer to the corrupted input speech and DAC-
processed target speech, respectively. Target-DAC is an upper bound for the
LM-based models studied in this work that employ DAC as the tokenizer.

###  3.4 Evaluation Metrics

It's a known fact that standard metrics such as PESQ, SI-SNR cannot accurately
assess generative models due to lack of waveform alignment [6, 35]. We rely on
the following metrics instead, and resample the generated speech if necessary.

DNSMOS and SESQA are reference-free perceptual quality estimators [36, 37]
capable of evaluating generative models aiming to fix similar distortions [6,
18, 38]. SESQA works with 48 kHz and DNSMOS works with 16 kHz. We use the
public DNSMOS [36] and our in-house SESQA trained on the data and model
configurations as described in [37].

Log-Spectral Distance (LSD) [39] is a common metric to measure bandwidth
extension. LSD supports 44.1 kHz. We use the public implementation in [1].

Speaker Similarity (Spk Sim) is the speaker cosine similarity between the
ground truth and the processed speech. We use the public WeSpeaker [40] to
compute similarity at 16 kHz.

Subjective Listening (MOS) We ask 14 expert listeners to rate the overall
generated speech quality on a 1–5 scale, and report the mean opinion scores
based on 40 samples from the full-band ALL-GSR test set that covers the 4
studied distortions and their combinations. Samples are available on our demo
page222https://masksr.github.io/MaskSR/.

##  4 Results

Table 4: Wide-band 16 kHz denoising/dereverberation results on the DNS
Challenge test sets. The SGMSE, StoRM, and SELM results are reported in [18].
On the `With Reverb' test set, only MaskSR and DEMUCS perform joint denoising
and dereverberation while other models only perform denoising (see Sec. 4.2).

System | Model type | With Reverb | Without Reverb | Real Recordings
---|---|---|---|---
DNSMOS $\uparrow$ | Spk Sim $\uparrow$ | DNSMOS $\uparrow$ | Spk Sim $\uparrow$ | DNSMOS $\uparrow$
SIG | BAK | OVL | SIG | BAK | OVL | SIG | BAK | OVL
Unprocessed | - | 1.760 | 1.497 | 1.392 | 0.941 | 3.392 | 2.618 | 2.483 | 0.969 | 3.053 | 2.509 | 2.255
DEMUCS | Regression | 2.856 | 3.897 | 2.553 | 0.762 | 3.575 | 4.153 | 3.345 | 0.956 | 3.263 | 4.027 | 2.988
FRCRN | Regression | 2.934 | 2.924 | 2.279 | 0.935 | 3.578 | 4.133 | 3.335 | 0.970 | 3.370 | 3.977 | 3.037
SGMSE | Diffusion | 2.730 | 2.741 | 2.430 | - | 3.501 | 3.710 | 3.137 | - | 3.297 | 2.894 | 2.793
StoRM | Diffusion | 2.947 | 3.141 | 2.516 | - | 3.514 | 3.941 | 3.205 | - | 3.410 | 3.379 | 2.940
SELM | LM | 3.160 | 3.577 | 2.695 | - | 3.508 | 4.096 | 3.258 | - | 3.591 | 3.435 | 3.124
MaskSR-S | LM | 3.524 | 4.016 | 3.223 | 0.816 | 3.575 | 4.082 | 3.307 | 0.926 | 3.398 | 4.011 | 3.103
MaskSR-M | LM | 3.531 | 4.065 | 3.253 | 0.827 | 3.586 | 4.116 | 3.339 | 0.929 | 3.430 | 4.025 | 3.136

###  4.1 Full-band 44.1 kHz Speech Restoration

Figure 2: Effects of guidance on the overall DNSMOS (green) and speaker
similarity scores (red).

First, we show the effects of the guidance level $w$ on a held-out dev set.
Figure 2 shows that a larger $w$ yields the peak speaker similarity at $w=2$
due to more alignment with the input speech, but DNSMOS decreases due to more
residual noise. This shows the complementary nature of the two scores. We use
$w=2$ to report all the results without tuning on the test sets.

In Table 1, on the SR clean test set, since DNSMOS is not sensitive to
bandwidth extension, we mainly rely on the other 3 scores. We see that both
MaskSR models achieve leading bandwidth extension performance. They are better
than the two-stage VoiceFixer, and the regression-based NSNet2, which cannot
perform this task. On the ALL-GSR test set with a blend of all the 4 studied
distortions, both MaskSR also obtain competitive results, which outperform
VoiceFixer in terms of all scores except for LSD. This indicates the strong
capability of end-to-end speech restoration in the discrete space.

Figure 3: Token classification accuracy of codebook 1 (left) and 3 (right)
from MaskSR-S (blue) and SoundStorm (orange). Other codebooks follow the trend
of codebook 3.

We notice that SoundStorm, in terms of all results except for the ALL-GSR LSD,
does not outperform MaskSR-S. In Figure 3, we compare the two models in terms
of the token accuracy from codebook 1 and 3. Due to the hierarchical codebook
modeling (Sec. 2.4), SoundStorm yields modestly higher accuracy for codebook
2–9 (represented by codebook 3) than that of MaskSR-S, at the cost of
significantly lower codebook 1 accuracy, the codebook that encodes the most
salient speech pattern, such as speaker identity. This is because that no
other codebooks are available when predicting codebook 1. The much lower
codebook 1 accuracy may lead to the consistently worse speaker similarity
scores in Table 1, whereas the modestly higher codebook 2–9 accuracy does not
consistently translate to better LSD that reflects the generated high
frequency details. Thus, the fully parallel codebook modeling in MaskSR
provides overall better performance. In addition, we also measure the average
runtime (over 20 runs) of the studied LMs on an A100 GPU. Table 2 shows that
MaskSR-S is slightly faster than SoundStorm, and significantly faster than the
autoregressive UniAudio, which also does not yield better quality.

Table 2: Average runtime (sec) of different language models System | Sequence length (sec)
---|---
4 | 8 | 12 | 16
UniAudio | 44.86 | 66.08 | 87.54 | 112.02
SoundStorm | 3.05 | 4.25 | 5.55 | 6.85
MaskSR-S | 2.72 | 4.10 | 4.92 | 5.22
Table 3: Subjective MOS scores with 95% confidence intervals

Unprocessed | Target | NSNet2 | VoiceFixer | MaskSR-M
---|---|---|---|---
1.96 $\pm$ 0.08 | 4.66 $\pm$ 0.05 | 2.50 $\pm$ 0.09 | 3.53 $\pm$ 0.09 | 4.36 $\pm$ 0.07

Table 3 reports the subjective listening results based on the 40 samples from
the ALL-GSR test set. The MOS reflects the overall speech quality. MaskSR-M
significantly outperforms other systems, showing superior capability to
restore high quality full-band speech from diverse distortions.

###  4.2 Wide-band 16 kHz Denoising and Dereverberation

In Table 4, on the `With Reverb' test set, since only MaskSR and DEMUCS [32]
perform joint noise and reverb suppression while other models only suppress
noise, this partially contributes to the higher DNSMOS for MaskSR and DEMUCS.
But comparing only these two, MaskSR still outperforms DEMUCS by a large
margin. Meanwhile, since the ground truth contains reverb, but the outputs of
MaskSR and DEMUCS do not, that leads to lower speaker similarity scores
relative to other denoising-only models. On the other two test sets without
noticeable reverb, despite the fact that MaskSR is trained to address a
diverse set of distortions, it still achieves competitive denoising results
compared to various specialized models.

##  5 Conclusion

In this work, we proposed a full-band speech restoration system that addressed
a diverse set of distortions holistically based on masked LMs. The system
showed promising results on both the full-band speech restoration task
evaluated with a blend of the studied distortions, and also on individual
tasks. We also shed insights into the effects of codebook modeling on the
studied task, and improved speaker identity preservation using classifier-free
guidance. Our future work includes further improving the generated speech
quality and intelligibility by exploring semantic tokens [18].

## References

  * [1] H. Liu, Q. Kong, Q. Tian, Y. Zhao, D. Wang, C. Huang, and Y. Wang, ``VoiceFixer: Toward General Speech Restoration with Neural Vocoder,'' _arXiv preprint arXiv:2109.13731_ , 2021.
  * [2] M. Liu, S. Lv, Z. Zhang, R. Han, X. Hao, X. Xia, L. Chen, Y. Xiao, and L. Xie, ``Two-stage Neural Network for ICASSP 2023 Speech Signal Improvement Challenge,'' in _ICASSP_ , 2023, pp. 1–2.
  * [3] M. Liu, Z. Chen, X. Yan, Y. Lv, X. Xia, C. Huang, Y. Xiao, and L. Xie, ``RaD-Net: A Repairing and Denoising Network for Speech Signal Improvement,'' _arXiv preprint arXiv:2401.04389_ , 2024.
  * [4] W. Liu, Y. Shi, J. Chen, W. Rao, S. He, A. Li, Y. Wang, and Z. Wu, ``Gesper: A Restoration-Enhancement Framework for General Speech Reconstruction,'' in _INTERSPEECH_ , 2023, pp. 4044–4048.
  * [5] N. Kandpal, O. Nieto, and Z. Jin, ``Music Enhancement via Image Translation and Vocoding,'' in _ICASSP_ , 2022, pp. 3124–3128.
  * [6] J. Serrà, S. Pascual, J. Pons, R. O. Araz, and D. Scaini, ``Universal Speech Enhancement with Score-based Diffusion,'' _arXiv preprint arXiv:2206.03065_ , 2022.
  * [7] Y. Koizumi, H. Zen, S. Karita, Y. Ding, K. Yatabe, N. Morioka, Y. Zhang, W. Han, A. Bapna, and M. Bacchiani, ``Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representations,'' in _WASPPA_ , 2023, pp. 1–5.
  * [8] S. Zhao, B. Ma, K. N. Watcharasupat, and W.-S. Gan, ``FRCRN: Boosting Feature Representation Using Frequency Recurrence for Monaural Speech Enhancement,'' in _ICASSP_ , 2022, pp. 9281–9285.
  * [9] X. Hao, X. Su, R. Horaud, and X. Li, ``FullSubNet: A Full-band and Sub-band Fusion Model for Real-time Single-channel Speech Enhancement,'' in _ICASSP_ , 2021, pp. 6633–6637.
  * [10] A. Li, W. Liu, X. Luo, G. Yu, C. Zheng, and X. Li, ``A Simultaneous Denoising and Dereverberation Framework with Target Decoupling,'' in _INTERSPEECH_ , 2021, pp. 2801–2805.
  * [11] F. Kreuk, G. Synnaeve, A. Polyak, U. Singer, A. Défossez, J. Copet, D. Parikh, Y. Taigman, and Y. Adi, ``AUDIOGEN: Textually Guided Audio Generation,'' in _ICLR_ , 2023.
  * [12] J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y. Adi, and A. Défossez, ``Simple and Controllable Music Generation,'' _NeurIPS_ , vol. 36, 2024.
  * [13] S. Deshmukh, B. Elizalde, R. Singh, and H. Wang, ``Pengi: An Audio Language Model for Audio Tasks,'' _NeurIPS_ , vol. 36, 2024.
  * [14] C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li _et al._ , ``Neural Codec Language Models Are Zero-shot Text to Speech Synthesizers,'' _arXiv preprint arXiv:2301.02111_ , 2023.
  * [15] H. Chang, H. Zhang, J. Barber, A. Maschinot, J. Lezama, L. Jiang, M.-H. Yang, K. P. Murphy, W. T. Freeman, M. Rubinstein _et al._ , ``MUSE: Text-To-Image Generation via Masked Generative Transformers,'' in _ICML_ , 2023, pp. 4055–4075.
  * [16] D. Yang, J. Tian, X. Tan, R. Huang, S. Liu, X. Chang, J. Shi, S. Zhao, J. Bian, X. Wu _et al._ , ``UniAudio: An Audio Foundation Model Toward Universal Audio Generation,'' _arXiv preprint arXiv:2310.00704_ , 2023.
  * [17] X. Wang, M. Thakker, Z. Chen, N. Kanda, S. E. Eskimez, S. Chen, M. Tang, S. Liu, J. Li, and T. Yoshioka, ``SpeechX: Neural Codec Language Model as a Versatile Speech Transformer,'' _arXiv preprint arXiv:2308.06873_ , 2023.
  * [18] Z. Wang, X. Zhu, Z. Zhang, Y. Lv, N. Jiang, G. Zhao, and L. Xie, ``SELM: Speech Enhancement Using Discrete Tokens and Language Models,'' in _ICASSP_ , 2024.
  * [19] H. Liu, X. Liu, Q. Kong, Q. Tian, Y. Zhao, D. Wang, C. Huang, and Y. Wang, ``VoiceFixer: A Unified Framework for High-Fidelity Speech Restoration,'' in _INTERSPEECH_ , 2022, pp. 4232–4236.
  * [20] R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, ``High-fidelity Audio Compression with Improved RVQGAN,'' _NeurIPS_ , vol. 36, 2024.
  * [21] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, ``SoundStream: An End-to-end Neural Audio Codec,'' _IEEE/ACM TASLP_ , vol. 30, pp. 495–507, 2021.
  * [22] J. Serrà, D. Scaini, S. Pascual, D. Arteaga, J. Pons, J. Breebaart, and G. Cengarle, ``Mono-to-stereo Through Parametric Stereo Generation,'' _ISMIR_ , 2023.
  * [23] H. F. Garcia, P. Seetharaman, R. Kumar, and B. Pardo, ``VampNet: Music Generation via Masked Acoustic Token Modeling,'' _ISMIR_ , 2023.
  * [24] H. Chang, H. Zhang, L. Jiang, C. Liu, and W. T. Freeman, ``MaskGIT: Masked Generative Image Transformer,'' in _CVPR_ , 2022, pp. 11 315–11 325.
  * [25] R. Sheffer and Y. Adi, ``I Hear Your True Colors: Image Guided Audio Generation,'' in _ICASSP_ , 2023, pp. 1–5.
  * [26] Z. Borsos, M. Sharifi, D. Vincent, E. Kharitonov, N. Zeghidour, and M. Tagliasacchi, ``SoundStorm: Efficient Parallel Audio Generation,'' _arXiv preprint arXiv:2305.09636_ , 2023.
  * [27] C. Veaux, J. Yamagishi, and K. MacDonald, ``CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit,'' 2017.
  * [28] H. Dubey, V. Gopal, R. Cutler, A. Aazami, S. Matusevych, S. Braun, S. E. Eskimez, M. Thakker, T. Yoshioka, H. Gamper _et al._ , ``ICASSP 2022 Deep Noise Suppression Challenge,'' in _ICASSP_ , 2022, pp. 9271–9275.
  * [29] H. Bu, J. Du, X. Na, B. Wu, and H. Zheng, ``AISHELL-1: An Open-source Mandarin Speech Corpus and a Speech Recognition Baseline,'' in _O-COCOSDA_ , 2017, pp. 1–5.
  * [30] C. K. Reddy, V. Gopal, R. Cutler, E. Beyrami, R. Cheng, H. Dubey, S. Matusevych, R. Aichner, A. Aazami, S. Braun, P. Rana, S. Srinivasan, and J. Gehrke, ``The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results,'' in _INTERSPEECH_ , 2020, pp. 2492–2496.
  * [31] S. Braun and I. Tashev, ``Data Augmentation and Loss Normalization for Deep Noise Suppression,'' in _International Conference on Speech and Computer_ , 2020, pp. 79–86.
  * [32] A. Défossez, G. Synnaeve, and Y. Adi, ``Real Time Speech Enhancement in the Waveform Domain,'' in _INTERSPEECH_ , 2020, pp. 3291–3295.
  * [33] J. Richter, S. Welker, J.-M. Lemercier, B. Lay, and T. Gerkmann, ``Speech Enhancement and Dereverberation with Diffusion-based Generative Models,'' _IEEE/ACM TASLP_ , 2023.
  * [34] J.-M. Lemercier, J. Richter, S. Welker, and T. Gerkmann, ``StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation,'' _IEEE/ACM TASLP_ , 2023.
  * [35] W. A. Jassim, J. Skoglund, M. Chinen, and A. Hines, ``WARP-Q: Quality Prediction for Generative Neural Speech Codecs,'' in _ICASSP_ , 2021, pp. 401–405.
  * [36] C. K. Reddy, V. Gopal, and R. Cutler, ``DNSMOS: A Non-intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors,'' in _ICASSP_ , 2021, pp. 6493–6497.
  * [37] J. Serrà, J. Pons, and S. Pascual, ``SESQA: Semi-supervised Learning for Speech Quality Assessment,'' in _ICASSP_ , 2021, pp. 381–385.
  * [38] S. Pascual, J. Serrà, and J. Pons, ``Adversarial Auto-encoding for Packet Loss Concealment,'' in _WASPAA_ , 2021, pp. 71–75.
  * [39] A. Erell and M. Weintraub, ``Estimation Using Log-spectral-distance Criterion for Noise-robust Speech Recognition,'' in _ICASSP_ , 1990, pp. 853–856.
  * [40] H. Wang, C. Liang, S. Wang, Z. Chen, B. Zhang, X. Xiang, Y. Deng, and Y. Qian, ``Wespeaker: A Research and Production Oriented Speaker Embedding Learning Toolkit,'' in _ICASSP_ , 2023, pp. 1–5.

Table 5: ALL-GSR full-band speech restoration results using different input
features to encode the corrupted speech

Input feature | Model size | DNSMOS $\uparrow$ | SESQA $\uparrow$ | LSD $\downarrow$ | Spk Sim $\uparrow$
---|---|---|---|---|---
SIG | BAK | OVL
DAC | 54 M | 3.179 | 3.915 | 2.871 | 3.440 | 1.286 | 0.803
Waveform | 59 M | 3.377 | 3.973 | 3.068 | 3.573 | 1.276 | 0.837
STFT (MaskSR) | 55 M | 3.430 | 3.982 | 3.123 | 3.541 | 1.201 | 0.845

##  Appendix A Input Speech Representation

Previous speech denoising LMs [16, 17] encode the input corrupted speech as
discrete tokens extracted from a pre-trained audio tokenizer. Since these
models target at translating various input modalities (such as text and audio)
to a target audio under a unified multi-task framework, it's convenient to
encode all types of input signals as discrete tokens. However, for the
dedicated speech restoration task, it's unclear whether such representation of
the corrupted speech is optimal or not. The lossy compression caused by the
neural audio tokenizer could hurt the integrity of the crucial information in
the input speech, such as low level speaker characteristics, high frequency
details, etc. Therefore, it might be beneficial to use lossless
transformations to encode the input speech. To study the effects of the input
features, we consider 3 options.

DAC We extract the $9\times{T}$ codegram from the input corrupted speech using
the pre-trained DAC tokenizer. Then, following the same method to embed the
target speech codegram (Sec. 2.3), we obtain the summation of the codebook
embeddings from the 9 learnable embedding tables. Since the corrupted and the
target speech share the same DAC space, we directly sum their embeddings
without using a transformer-based speech encoder to further transform the
corrupted speech. For a fair comparison, we use 14 transformer blocks in the
LM with approximately the same total model capacity as other variants which do
employ a speech encoder.

STFT This is the adopted method in MaskSR. As detailed in Sec. 2.2, we compute
the power-law compressed magnitude STFT spectrogram given a corrupted speech
signal using a window length and hop length of 2048 and 512 samples,
respectively. Next, an MLP followed by a stack of 6 self-attention transformer
blocks map the STFT features to 512-dim embeddings, which are summed with
those of the masked target speech. An LM consisting of 8 transformer blocks is
used to predict the masked target speech tokens.

Waveform We replace the STFT by a learnable 2048-dim 1-D convolution followed
by a ReLU activation function and a layer normalization. To be consistent with
STFT, the convolution also uses a kernel length and stride of 2048 and 512
samples, respectively. After the layer normalization, the 2048-dim latent
features of the corrupted speech signal are projected down to 512-dim
embeddings by a fully connected layer followed by 6 self-attention transformer
blocks. The LM also uses 8 transformer blocks as in the STFT model. Compared
with DAC, both STFT and waveform are lossless representations of the input
speech.

We train the 3 systems on the dataset described in Sec. 3.1, and evaluate them
on the full-band ALL-GSR test set. We optimize the classifier-free guidance
level $w$ for each of them during inference on a held-out dev set, and choose
$w=0.3$ for the DAC model and $w=2$ for the other two variants.

From Table 5, it can be seen that there is a noticeable gap between the system
that uses DAC to encode the input speech and the other two variants that
employ raw features. Although the LM component in the DAC-based model is
larger than those in the other two systems (14 vs. 8 transformer blocks), the
quality of its generated speech is lagging. This shows that the discrete DAC
token is not the optimal feature representation to enable high quality speech
restoration. On the other hand, the STFT model performs better than the
waveform model in terms of most metrics. Empirical listening finds that the
advantage of the STFT model is larger on the dereverberation task as it
generates more natural speech with less over-suppression of the target speech
and better speaker voice preservation. Thus, these results provide the
supporting evidence for adopting STFT in MaskSR.