# The holomorphic discrete series contribution to the generalized Whittaker
Plancherel formula II. Non-tube type groups

Jan Frahm  Department of Mathematics, Aarhus University, Ny Munkegade 118,
8000 Aarhus C, Denmark<EMAIL_ADDRESS>,  Gestur Ólafsson  Department of
Mathematics, Louisiana State University, Baton Rouge, LA 70803, USA
<EMAIL_ADDRESS>and  Bent Ørsted  Department of Mathematics, Aarhus
University, Ny Munkegade 118, 8000 Aarhus C, Denmark<EMAIL_ADDRESS>Dedicated to the memory of Gerrit van Dijk

###### Abstract.

For every simple Hermitian Lie group $G$, we consider a certain maximal
parabolic subgroup whose unipotent radical $N$ is either abelian (if $G$ is of
tube type) or two-step nilpotent (if $G$ is of non-tube type). By the
generalized Whittaker Plancherel formula we mean the Plancherel decomposition
of $L^{2}(G/N,\omega)$, the space of square-integrable sections of the
homogeneous vector bundle over $G/N$ associated with an irreducible unitary
representation $\omega$ of $N$. Assuming that the central character of
$\omega$ is contained in a certain cone, we construct embeddings of all
holomorphic discrete series representations of $G$ into $L^{2}(G/N,\omega)$
and show that the multiplicities are equal to the dimensions of the lowest
$K$-types.
The construction is in terms of a kernel function which can be explicitly
defined using certain projections inside a complexification of $G$. This
kernel function carries all information about the holomorphic discrete series
embedding, the lowest $K$-type as functions on $G/N$, as well as the
associated Whittaker vectors.

###### 2010 Mathematics Subject Classification:

Primary 22E46; Secondary 43A85.

Part of the research in this paper was carried out within the online research
community on Representation Theory and Noncommutative Geometry sponsored by
the American Institute of Mathematics. The first author was partially
supported by a research grant from the Villum Foundation (Grant No. 00025373).
The second author was partially supported by Simons grant 586106.

## Introduction

Gerrit van Dijk worked on several aspects of representation theory and
harmonic analysis, in particular using special functions and distributions on
Lie groups. With his younger colleagues he developed the notion of canonical
representations due to Gelfand et al. as concrete models of unitary
representations; giving the explicit decomposition of these into irreducibles
amounts in effect to calculating tensor products of unitary irreducible
representations or the restriction of such to symmetric subgroups. The groups
in question include the classical rank one semisimple Lie groups, such as
$G=\operatorname{SU}(n,1)$ with the corresponding Riemannian symmetric space
$\mathcal{D}=G/K$ of Hermitian type, namely the open unit ball in complex
Euclidian $n$-space; here the boundary sphere is also the Shilov boundary and
of the form $G/P$ for a parabolic subgroup $P$ with nilradical $N$ a
Heisenberg group if $n>1$.

In the present paper we shall consider similar groups of Hermitian type, non-
tube type, and various models of unitary representations. Even though unitary
representations are often considered up to unitary equivalence, for certain
problems it is convenient to work with concrete models; this is the case for
the above mentioned problems studied by Gerrit van Dijk, namely branching
laws, i.e. finding the explicit decomposition of a given irreducible
representation restricted to a subgroup. We shall also use some of the tools
in his study, namely reproducing kernel Hilbert spaces, interwining operators
given as kernel operators, and explicit distribution vectors in unitary
representations.

With notation as in the example above, we shall find explicit irreducible
invariant subspaces of $L^{2}(G/N,\omega)$ where $\omega$ is an infinite-
dimensional irreducible unitary representation of $N$ defining a homogeneous
vector bundle over $X=G/N$; these will be equivalent to holomorphic discrete
series representations of $G$. In effect this gives a so-called Whittaker
model of these representations corresponding to the Siegel parabolic subgroup
$P$ of potential use in the study of automorphic representations and number
theory. Thus, our aim in this paper is to

  * •

Find explicit Whittaker distribution vectors in the holomorphic discrete
series for $G$ with values in $\omega$,

  * •

Find explicit intertwining kernel operators between the standard model on
$\mathcal{D}=G/K$ and the Whittaker model on $X=G/N$ for these
representations,

  * •

Find an explicit expression for the lowest $K$-types on $X$,

and to understand links between these problems.

Let us state our results in more detail.

### The inclusion $G\subseteq P^{+}K_{\mathbb{C}}N_{\mathbb{C}}$

Let $G$ be a simple Lie group of Hermitian type, i.e. the associated
Riemannian symmetric space $\mathcal{D}=G/K$ carries a $G$-invariant complex
structure. Denote by $\mathfrak{p}^{+}$ the holomorphic tangent space at the
base point $o=eK\in\mathcal{D}$ and by $\mathfrak{p}^{-}$ the antiholomorphic
tangent space. We assume that $G$ is contained as a real form inside a simply-
connected complex Lie group $G_{\mathbb{C}}$. Let
$P^{\pm}=\exp\mathfrak{p}^{\pm}$ denote the connected subgroups of
$G_{\mathbb{C}}$ with Lie algebras $\mathfrak{p}^{\pm}$ and write
$K_{\mathbb{C}}$ for the complexification of $K$ in $G_{\mathbb{C}}$.

For groups of tube type, we constructed in our previous work [2] the
holomorphic discrete series contribution to the Plancherel decomposition of
$L^{2}(G/N,\psi)$, where $N$ is the unipotent radical of some maximal
parabolic subgroup and $\psi$ a certain non-degenerate character on $N$. One
approach to this problem was to show the inclusion $G\subseteq
P^{+}K_{\mathbb{C}}N_{\mathbb{C}}$, where $N_{\mathbb{C}}$ is the connected
complexification of $N$ inside $G_{\mathbb{C}}$. In order to extend these
results to non-tube type groups, we first investigate for which maximal
parabolic subgroups $P=MAN$ this inclusion holds.

######  Theorem A (see Theorem 2.3).

For every simple Hermitian Lie group $G$ there exists a unique standard
maximal parabolic subgroup $P=MAN$ such that $G\subseteq
P^{+}K_{\mathbb{C}}N_{\mathbb{C}}$.

### Discrete spectrum in the generalized Whittaker Plancherel formula

We fix the parabolic subgroup $P=MAN$ from Theorem A. The unipotent radical
$N$ is either abelian (if $\mathcal{D}$ is of tube type) or two-step nilpotent
(if $\mathcal{D}$ is of non-tube type). We fix an irreducible unitary
representation $\omega$ of $N$ whose central character is contained in a
certain cone in the Lie algebra of the center of $N$ (see Section 4 for
details). Note that $\omega$ is a character if $N$ is abelian and infinite-
dimensional otherwise. By the generalized Whittaker Plancherel formula we mean
the Plancherel formula for the induced representation
$\operatorname{Ind}_{N}^{G}(\omega)$ which can be realized on
$L^{2}(G/N,\omega)$, the $L^{2}$-sections of the homogeneous vector bundle
over $G/N$ associated with $\omega$. Our main result is the following:

######  Theorem B (see Corollary 5.9).

Every holomorphic discrete series representation of $G$ occurs discretely in
$L^{2}(G/N,\omega)$ with multiplicity equal to the dimension of its highest
weight space.

Here, the highest weight space of an irreducible unitary representation
$(U,\mathcal{H})$ is the subspace $\mathcal{H}^{\mathfrak{p}^{+}}$ annihilated
by $\mathfrak{p}^{+}$. It forms an irreducible representation
$(\pi,\mathcal{V}_{\pi})$ of $K$ which determines $(U,\mathcal{H})$ uniquely.
We therefore write $(U_{\pi},\mathcal{H}_{\pi})$ for the holomorphic discrete
series representation with highest weight $\pi$, and we realize it on a space
$\mathcal{H}_{\pi}$ of holomorphic $\mathcal{V}_{\pi}$-valued functions on
$\mathcal{D}$ (see Section 3 for details). Note that $\mathcal{V}_{\pi}$ is
also the lowest $K$-type of $\mathcal{H}_{\pi}$.

### Lowest $K$-types and Whittaker vectors via kernel functions

The proof of Theorem B is by constructing explicit embeddings

$\mathcal{H}_{\pi}\otimes\overline{\mathcal{V}_{\pi}}\to
L^{2}(G/N,\omega),\quad F\otimes\overline{\eta}\mapsto T_{\pi,\eta}F.$

The maps $T_{\pi,\eta}:\mathcal{H}_{\pi}\to L^{2}(G/N,\omega)$
($\eta\in\mathcal{V}_{\pi}$) are given in terms of their integral kernels.

In view of Theorem A, every element $g\in G$ decomposes into
$g=p^{+}_{N}(g)k_{\mathbb{C},N}^{+}(g)n_{\mathbb{C}}^{+}(g)\in
P^{+}K_{\mathbb{C}}N_{\mathbb{C}}$ (not uniquely, but all formulas will be
independent of the choice of decomposition). Using this decomposition, we
define for every $\eta\in\mathcal{V}_{\pi}$ a function
$\Psi_{\pi,\eta}:G\times\mathcal{D}\to\operatorname{Hom}(\mathcal{F},\mathcal{V}_{\pi})$
by its adjoint:

$\Psi_{\pi,\eta}(x,g\cdot o)^{*}:\mathcal{V}_{\pi}\to\mathcal{F},\\\
\xi\mapsto\omega(n_{\mathbb{C}}^{+}(g^{-1}x))^{-1}m\Big{(}\pi(k_{\mathbb{C},N}^{+}(g^{-1}x))^{-1}\circ
j_{\pi}(g,o)^{-*}\xi\otimes\overline{\eta}\Big{)}\qquad(x,g\in
G,\xi\in\mathcal{V}_{\pi}),$

where $j_{\pi}(g,z)\in\operatorname{GL}(\mathcal{V}_{\pi})$ ($g\in G$,
$z\in\mathcal{D}$) is the cocycle used to define the holomorphic discrete
series action on $\mathcal{D}$ (see (3.2)) and
$m:\mathcal{V}_{\pi}\otimes\overline{\mathcal{V}}_{\pi}\to\mathcal{F}$ is a
certain matrix coefficient map (see (5.4)). This construction is discussed in
detail in Section 5.4, but the main point is that the definition of $\Psi$ is
completely explicit when using the embedding $G\subseteq
P^{+}K_{\mathbb{C}}N_{\mathbb{C}}$.

The operator $T_{\pi,\eta}:\mathcal{H}_{\pi}\to L^{2}(G/N,\omega)$
($\eta\in\mathcal{V}_{\pi}$) is most easily characterized by its adjoint
$T_{\pi,\eta}^{*}:L^{2}(G/N,\omega)\to\mathcal{H}_{\pi}$ which has $\Psi(x,z)$
as integral kernel:

$T_{\pi,\eta}^{*}f(z)=\int_{G/N}\Psi_{\pi,\eta}(x,z)f(x)\,d(xN)\qquad(f\in
L^{2}(G/N,\omega),z\in\mathcal{D}).$

The function $\Psi$ carries all information about the embedding of the
associated holomorphic discrete series representation into $L^{2}(G/N,\omega)$
in the following sense:

######  Theorem C (see Proposition 5.5, Lemma 5.7 and Proposition 5.10).

  1. (1)

The map

$\mathcal{H}_{\pi}\otimes\overline{\mathcal{V}_{\pi}}\to
L^{2}(G/N,\omega),\quad F\otimes\overline{\eta}\mapsto T_{\pi,\eta}F$

constructs the holomorphic discrete series contribution in Theorem B.

  2. (2)

The embedding $T_{\pi,\eta}$ restricted to the lowest $K$-type
$\mathcal{V}_{\pi}$ is given by

$T_{\pi,\eta}\xi(x)=\Psi_{\pi,\eta}(x,o)^{*}\xi\qquad(\xi\in\mathcal{V}_{\pi},x\in
G).$

  3. (3)

Identifying distribution vectors on $\mathcal{H}_{\pi}$ with antiholomorphic
$\mathcal{V}_{\pi}^{\vee}$-valued functions on $\mathcal{D}$ (see Section 5.5
for details), the Whittaker vector
$W\in\operatorname{Hom}_{N}(U_{\pi}^{\infty}|_{N},\omega)$ associated with the
embedding
$T_{\pi,\eta}\in\operatorname{Hom}_{G}(U_{\pi},\operatorname{Ind}_{N}^{G}(\omega))$
by Frobenius reciprocity is given by the
$\mathcal{V}_{\pi}^{\vee}\otimes\mathcal{F}=\operatorname{Hom}(\mathcal{V}_{\pi},\mathcal{F})$-valued
antiholomorphic function

$W(z)=\Psi_{\pi,\eta}(e,z)^{*}\qquad(z\in\mathcal{D}).$

Items (2) and (3) show that the kernel $\Psi_{\pi,\eta}$ contains both
information about the Whittaker vector (when restricted to
$\\{e\\}\times\mathcal{D}$) and about the lowest $K$-type embedding (when
restricted to $G\times\\{o\\}$).

### Relation to other work

This paper extends parts of the results in our previous work [2] to the case
of non-tube type groups. In [2] we also give explicit formulas for Whittaker
vectors in different models of the holomorphic discrete series. This should
also be possible for non-tube type groups using the $L^{2}$-model obtained by
Ding in [1]. Moreover, in [2] we show that the holomorphic discrete series
contribution to $L^{2}(G/N,\omega)$ is given by the boundary values of
functions in a Hardy space of holomorphic sections on a complex domain in
$G_{\mathbb{C}}/N_{\mathbb{C}}$ containing $G/N$ in its boundary. We expect
that similar results can be obtained in the more general case using the same
methods.

### Structure of the paper

In Section 1 we recall some basic structure theory for Hermitian Lie groups
and the associated bounded symmetric domains. This is used in Section 2 to
show Theorem A. Section 3 is about the standard realization of holomorphic
discrete series representations on a space of holomorphic functions on the
bounded domain. In Section 4 we introduce the irreducible unitary
representations $\omega$ of $N$ that define the Whittaker space
$L^{2}(G/N,\omega)$ and realize them on a Fock space of holomorphic functions
which are square integrable with respect to a Gaussian measure. Finally,
Section 5 provides the construction of the embeddings $T_{\pi,\eta}$ of the
holomorphic discrete series into $L^{2}(G/N,\omega)$ and the proofs of
Theorems B and C.

##  1\. Bounded symmetric domains

In this section we recall some basic standard facts about bounded symmetric
domains. For more details we refer the reader to [3, 4, 5, 6, 9, 10].

###  1.1. The Harish-Chandra realization of bounded domains

In this article we assume that $G$ is a connected simple Lie group with Lie
algebra $\mathfrak{g}$. We assume that $G$ is contained as a closed subgroup
in a simply connected complex Lie group $G_{\mathbb{C}}$ with Lie algebra
$\mathfrak{g}_{\mathbb{C}}$. Let
$\sigma_{\mathfrak{g}}:\mathfrak{g}_{\mathbb{C}}\to\mathfrak{g}_{\mathbb{C}}$
be the conjugation with respect to $\mathfrak{g}$,
$\sigma_{\mathfrak{g}}(X+iY)=X-iY$ for $X,Y\in\mathfrak{g}$. Then
$\sigma_{\mathfrak{g}}$ can be lifted to a homomorphism
$\sigma_{G}:G_{\mathbb{C}}\to G_{\mathbb{C}}$ such that $G\subseteq
G_{\mathbb{C}}^{\sigma_{G}}$ is open.

Let $\theta_{\mathfrak{g}}:\mathfrak{g}\to\mathfrak{g}$ be a Cartan involution
which we extend to a complex-linear involution
$\mathfrak{g}_{\mathbb{C}}\to\mathfrak{g}_{\mathbb{C}}$. It lifts to an
involution $\theta_{G}:G_{\mathbb{C}}\to G_{\mathbb{C}}$ such that
$K=G^{\theta_{G}}$ is a maximal compact subgroup of $G$ with complexification
$K_{\mathbb{C}}=G_{\mathbb{C}}^{\theta_{G}}$. If the role of $G$ or
$\mathfrak{g}$ is clear from the context, we simply write $\theta$ for both
involutions.

Denote by $\mathfrak{g}=\mathfrak{k}\oplus\mathfrak{p}$ the corresponding
Cartan decomposition of $\mathfrak{g}$. We assume that $G/K$ is a bounded
symmetric domain. This happens if and only if the center $\mathfrak{z}$ of
$\mathfrak{k}$ is non-zero, and in that case it is one-dimensional. There
exists an element $z_{0}\in\mathfrak{z}$, unique up to sign, such that
$\operatorname{ad}(z_{0})$ has eigenvalues $0$, $\pm i$ on
$\mathfrak{g}_{\mathbb{C}}$. The zero-eigenspace is
$\mathfrak{k}_{\mathbb{C}}$. We write

$\mathfrak{p}^{\pm}=\\{X\in\mathfrak{g}_{\mathbb{C}}\,:\,[z_{0},X]=\pm iX\\}.$

Then $\mathfrak{p}_{\mathbb{C}}=\mathfrak{p}^{+}\oplus\mathfrak{p}^{-}$ and
$\mathfrak{p}^{\pm}$ are abelian Lie algebras and the corresponding analytic
subgroups $P^{\pm}=\exp\mathfrak{p}^{\pm}\subseteq G_{\mathbb{C}}$ are
normalized by $K_{\mathbb{C}}$. The set $P^{+}K_{\mathbb{C}}P^{-}$ is open and
dense in $G_{\mathbb{C}}$ and the multiplication map

$P^{+}\times K_{\mathbb{C}}\times P^{-}\to
G_{\mathbb{C}},\quad(p^{+},k,p^{-})\mapsto p^{+}kp^{-}$

is a holomorphic diffeomorphism into its image. For $g\in
P^{+}K_{\mathbb{C}}P^{-}$ we write $g=p^{+}(g)k_{\mathbb{C}}(g)p^{-}(g)$. We
have $G\subseteq P^{+}K_{\mathbb{C}}P^{-}$, $G\cap K_{\mathbb{C}}P^{-}=K$ and
the map

$\iota:G/K\to\mathfrak{p}^{+},\quad
gK\mapsto\log_{\mathfrak{p}^{+}}(p^{+}(g))$

is a well defined $G$-equivariant embedding. Its image
$\mathcal{D}=\iota(G/K)$ is a bounded circular domain in the complex vector
space $\mathfrak{p}^{+}$, the Harish-Chandra realization of $G/K$, and we
write

$g\cdot Z=\iota(g\exp Z)\qquad(g\in G,Z\in\mathcal{D})$

for the corresponding action of $G$ on $\mathcal{D}$.

Embedding $\mathfrak{p}^{+}$ into $G_{\mathbb{C}}/K_{\mathbb{C}}P^{-}$ by
$Z\mapsto\exp(Z)K_{\mathbb{C}}P^{-}$, we can consider the action of
$G_{\mathbb{C}}$ on $\mathcal{D}$ and $\mathfrak{p}^{+}$. Clearly, $P^{+}$
acts on $\mathfrak{p}^{+}$ by translation $\exp(X)\cdot Z=Z+X$ and
$K_{\mathbb{C}}$ acts linearly by $k\cdot Z=\operatorname{Ad}(k)Z$.

###  1.2. Strongly orthogonal roots

Let $\mathfrak{t}$ be a Cartan subalgebra of $\mathfrak{g}$ containing
$z_{0}$, then $\mathfrak{t}\subseteq\mathfrak{k}$. Let
$\Delta(\mathfrak{g}_{\mathbb{C}},\mathfrak{t}_{\mathbb{C}})$ be the roots of
$\mathfrak{t}_{\mathbb{C}}$ in $\mathfrak{g}_{\mathbb{C}}$ and for
$\alpha\in\Delta(\mathfrak{g}_{\mathbb{C}},\mathfrak{t}_{\mathbb{C}})$ write
$\mathfrak{g}_{\mathbb{C}}^{\alpha}$ for the corresponding root space. We
define subsets

$\displaystyle\Delta_{c}$
$\displaystyle=\\{\alpha\in\Delta(\mathfrak{g}_{\mathbb{C}},\mathfrak{t}_{\mathbb{C}})\,:\,\mathfrak{g}_{\mathbb{C}}^{\alpha}\subseteq\mathfrak{k}_{\mathbb{C}}\\}=\\{\alpha\in\Delta(\mathfrak{g}_{\mathbb{C}},\mathfrak{t}_{\mathbb{C}})\,:\,\alpha(z_{0})=0\\}$
$\displaystyle\Delta_{n}$
$\displaystyle=\\{\alpha\in\Delta(\mathfrak{g}_{\mathbb{C}},\mathfrak{t}_{\mathbb{C}})\,:\,\mathfrak{g}_{\mathbb{C}}^{\alpha}\subseteq\mathfrak{p}_{\mathbb{C}}\\}=\\{\alpha\in\Delta(\mathfrak{g}_{\mathbb{C}},\mathfrak{t}_{\mathbb{C}})\,:\,\alpha(z_{0})\not=0\\}$
$\displaystyle\Delta_{n}^{+}$
$\displaystyle=\\{\alpha\in\Delta(\mathfrak{g}_{\mathbb{C}},\mathfrak{t}_{\mathbb{C}})\,:\,\mathfrak{g}_{\mathbb{C}}^{\alpha}\subseteq\mathfrak{p}^{+}\\}=\\{\alpha\in\Delta_{n}\,:\,\alpha(z_{0})=i\\}$

and call the roots in $\Delta_{c}$ compact and the roots in $\Delta_{n}$ non-
compact. Choose a positive system of roots
$\Delta^{+}(\mathfrak{g}_{\mathbb{C}},\mathfrak{t}_{\mathbb{C}})$ such that
$\Delta_{n}^{+}\subseteq\Delta^{+}(\mathfrak{g}_{\mathbb{C}},\mathfrak{t}_{\mathbb{C}})$.
Then
$\Delta_{c}^{+}:=\Delta^{+}(\mathfrak{g}_{\mathbb{C}},\mathfrak{t}_{\mathbb{C}})\cap\Delta_{c}$
is a positive system in $\Delta_{c}$ and
$\Delta^{+}(\mathfrak{g}_{\mathbb{C}},\mathfrak{t}_{\mathbb{C}})=\Delta_{n}^{+}\cup\Delta_{c}^{+}$.

Let $\gamma_{1},\ldots,\gamma_{r}$ be a maximal set of (long) strongly
orthogonal roots in $\Delta_{n}^{+}$ constructed in the usual way:
$\gamma_{1}$ is the maximal root in $\Delta_{n}^{+}$ and $\gamma_{j+1}$ the
maximal root in $\Delta_{n}^{+}$ strongly orthogonal to
$\gamma_{1},\ldots,\gamma_{j}$. Let $h_{j}\in i\mathfrak{t}$ be the
corresponding co-root vector. In particular, $\gamma_{i}(h_{j})=2\delta_{ij}$.
Let $\mathfrak{c}=\sum_{j=1}^{r}\mathbb{R}ih_{j}\subseteq\mathfrak{t}$ and
$\mathfrak{c}_{\mathbb{C}}$ its complexification. Abusing notation, we also
write $\gamma_{i}$ for its restriction to $\mathfrak{c}_{\mathbb{C}}$. Moore’s
Theorem describes the roots of $\mathfrak{c}_{\mathbb{C}}$ in
$\mathfrak{g}_{\mathbb{C}}$, i.e. the restrictions of roots in
$\Delta(\mathfrak{g}_{\mathbb{C}},\mathfrak{t}_{\mathbb{C}})$ to
$\mathfrak{c}_{\mathbb{C}}$. Here, we call a restricted root compact resp.
non-compact if it is the restriction of a compact resp. non-compact root.

######  Theorem 1.1 (Moore’s Theorem, first version).

For the set $\Delta(\mathfrak{g}_{\mathbb{C}},\mathfrak{c}_{\mathbb{C}})$ of
restricted roots there are two possibilities:

  1. (1)

$\mathcal{D}$ is of tube type. In this case

$\qquad\displaystyle\Delta(\mathfrak{g}_{\mathbb{C}},\mathfrak{c}_{\mathbb{C}})=\\{\pm\gamma_{i}:1\leq
i\leq r\\}\cup\\{\tfrac{1}{2}(\pm\gamma_{i}\pm\gamma_{j})\,:\,1\leq i<j\leq
r\\},$

the roots $\pm\gamma_{i}$ and $\pm\frac{1}{2}(\gamma_{i}+\gamma_{j})$ are non-
compact and the roots $\pm\frac{1}{2}(\gamma_{i}-\gamma_{j})$ are compact.

  2. (2)

$\mathcal{D}$ is of non-tube type. In this case

$\qquad\quad\displaystyle\Delta(\mathfrak{g}_{\mathbb{C}},\mathfrak{c}_{\mathbb{C}})=\\{\pm\tfrac{1}{2}\gamma_{i},\pm\gamma_{i}:1\leq
i\leq r\\}\cup\\{\tfrac{1}{2}(\pm\gamma_{i}\pm\gamma_{j})\,:\,1\leq i<j\leq
r\\},$

the roots $\pm\gamma_{i}$, $\pm\tfrac{1}{2}\gamma_{i}$ and
$\pm\frac{1}{2}(\gamma_{i}+\gamma_{j})$ are non-compact and the roots
$\pm\tfrac{1}{2}\gamma_{i}$ and $\pm\frac{1}{2}(\gamma_{i}-\gamma_{j})$ are
compact.

###  1.3. The $\operatorname{\mathfrak{su}}(1,1)$-embeddings

Let for the moment

$G=\operatorname{SU}(1,1)=\left\\{\begin{pmatrix}a&b\\\
\bar{b}&\bar{a}\end{pmatrix}\,:\,|a|^{2}-|b|^{2}=1\right\\}\subseteq\operatorname{SL}(2,\mathbb{C})$

with Lie algebra

$\mathfrak{g}=\operatorname{\mathfrak{su}}(1,1)=\left\\{\begin{pmatrix}ix&z\\\
\bar{z}&-ix\end{pmatrix}\,:\,x\in\mathbb{R},z\in\mathbb{C}\right\\}\subseteq\operatorname{\mathfrak{sl}}(2,\mathbb{C}).$

We have $\sigma_{G}(g)=I_{1,1}g^{-*}I_{1,1}$ and
$\sigma_{\mathfrak{g}}(X)=-I_{1,1}X^{*}I_{1,1}$, with the notation
$I_{1,1}=\operatorname{diag}(1,-1)$ and $X^{*}=\overline{X}^{\top}$. We choose
$\theta_{G}(g)=I_{1,1}gI_{1,1}$ as a Cartan involution, then
$\theta_{\mathfrak{g}}(X)=I_{1,1}XI_{1,1}$,
$\mathfrak{k}=\\{\operatorname{diag}(ix,-ix)\,:\,x\in\mathbb{R}\\}$ and
$K=\\{\operatorname{diag}(\gamma,\gamma^{-1})\,:\,|\gamma|=1\\}$.

Let

$z_{0}=\begin{pmatrix}\frac{i}{2}&0\\\ 0&-\frac{i}{2}\end{pmatrix},\quad
e_{0}=\begin{pmatrix}0&1\\\ 0&0\end{pmatrix},\quad f_{0}=\begin{pmatrix}0&0\\\
1&0\end{pmatrix},$

then

$\mathfrak{k}_{\mathbb{C}}=\mathbb{C}z_{0},\quad\mathfrak{p}^{+}=\mathbb{C}e_{0},\quad\mathfrak{p}^{-}=\mathbb{C}f_{0}$

and

$h_{0}=[e_{0},f_{0}]=\begin{pmatrix}1&0\\\ 0&-1\end{pmatrix}=-2iz_{0}.$

We have

$P^{+}K_{\mathbb{C}}P^{-}=\left\\{g=\begin{pmatrix}a&b\\\ c&d\end{pmatrix}:ad-
bc=1,d\not=0\right\\}$

Identifying $K_{\mathbb{C}}\simeq\mathbb{C}^{*}$ and
$\mathfrak{p}^{+}=\mathbb{C}e_{0}\simeq\mathbb{C}$ we have for a matrix $g$ as
above

$k_{\mathbb{C}}(g)=\frac{1}{d}\quad\text{and}\quad p^{+}(g)=\frac{b}{d}.$
(1.1)

Therefore, the Harish-Chandra embedding is given by

$\iota:G/K\to\mathfrak{p}^{+},\quad\begin{pmatrix}a&b\\\
\bar{b}&\bar{a}\end{pmatrix}K\mapsto\frac{b}{\bar{a}}\in\mathcal{D}_{0}=\\{z\in\mathbb{C}\,:\,|z|<1\\}$

and the corresponding action is the usual action by fractional linear
transformations:

$g\cdot z=\frac{az+b}{cz+d}\qquad\mbox{for }g=\begin{pmatrix}a&b\\\
c&d\end{pmatrix},z\in\mathcal{D}_{0}.$

Now to the general case. For every $1\leq j\leq r$ the root spaces
$\mathfrak{g}_{\mathbb{C}}^{\pm\gamma_{j}}$ are one-dimensional and we choose
$e_{j}\in\mathfrak{g}_{\mathbb{C}}^{\gamma_{j}}$, so that with
$f_{j}=\sigma_{\mathfrak{g}}(e_{j})\in\mathfrak{g}_{\mathbb{C}}^{-\gamma_{j}}$
we have $[e_{j},f_{j}]=h_{j}$ with $h_{j}$ as in Section 1.1. Let
$x_{j}=e_{j}+f_{j}\in\mathfrak{g}$.

For $j=1,\ldots,r$ define
$\varphi_{j}:\mathfrak{sl}(2,\mathbb{C})\to\mathfrak{g}_{\mathbb{C}}$ by

$h_{0}=\begin{pmatrix}1&0\\\ 0&-1\end{pmatrix}\mapsto h_{j},\quad
e_{0}=\begin{pmatrix}0&1\\\ 0&0\end{pmatrix}\mapsto e_{j}\quad\text{and}\quad
f_{0}=\begin{pmatrix}0&0\\\ 1&0\end{pmatrix}\mapsto f_{j}.$

Then $\varphi_{j}$ is a Lie algebra homomorphism such that
$\varphi_{j}\circ\theta_{\operatorname{\mathfrak{su}}(1,1)}=\theta_{\mathfrak{g}}\circ\varphi_{j}$
and
$\varphi_{j}\circ\sigma_{\operatorname{\mathfrak{su}}(1,1)}=\sigma_{\mathfrak{g}}\circ\varphi_{j}$.
As $\operatorname{SL}(2,\mathbb{C})$ is simply connected, there exists a
unique homomorphism, which we will denote by the same letter,
$\varphi_{j}:\operatorname{SL}(2,\mathbb{C})\to G_{\mathbb{C}}$ such that
$\varphi_{j}(\exp X)=\exp(\varphi_{j}(X))$ and
$\varphi_{j}\circ\theta_{\operatorname{SU}(1,1)}=\theta_{G}\circ\varphi_{j}$
as well as
$\varphi_{j}\circ\sigma_{\operatorname{SU}(1,1)}=\sigma_{G}\circ\varphi_{j}$.
It follows in particular that $\varphi_{j}(\operatorname{SU}(1,1))\subseteq
G$. As the roots $\gamma_{1},\ldots\gamma_{r}$ are strongly orthogonal it
follows that the algebras
$\mathfrak{g}_{\mathbb{C}}^{j}=\varphi_{j}(\mathfrak{sl}(2,\mathbb{C}))$
commute pairwise.

######  Lemma 1.2.

The subspace $\mathfrak{a}=\bigoplus_{j=1}^{r}\mathbb{R}x_{j}$ is maximal
abelian in $\mathfrak{p}$.

For $\operatorname{SU}(1,1)$ we let

$c_{0}=\exp\left(-\frac{\pi}{4}(e_{0}-f_{0})\right)=\frac{1}{\sqrt{2}}\begin{pmatrix}1&-1\\\
1&1\end{pmatrix}\in\operatorname{SL}(2,\mathbb{R})$

and

$c_{j}=\varphi_{j}(c_{0})=\exp\left(-\frac{\pi}{4}(e_{j}-f_{j})\right).$

Then simple matrix multiplication shows that
$\operatorname{Ad}(c_{0})h_{0}=e_{0}+f_{0}=x_{0}$ and hence
$\operatorname{Ad}(c_{j})h_{j}=x_{j}$. Let $c=c_{1}\cdots c_{r}$. Then we have

######  Lemma 1.3.

$\operatorname{Ad}(c)\mathfrak{c}=\mathfrak{a}$.

Define
$\lambda_{j}=\gamma_{j}\circ\operatorname{Ad}(c)^{-1}\in\mathfrak{a}^{*}$.
Then Moore’s Theorem implies:

######  Theorem 1.4 (Moore’s Theorem, second version).

For the set $\Delta(\mathfrak{g},\mathfrak{a})$ of restricted roots there are
two possibilities:

  1. (1)

$\mathcal{D}$ is of tube type. In this case

$\qquad\displaystyle\Delta(\mathfrak{g},\mathfrak{a})=\\{\pm\lambda_{i}:1\leq
i\leq r\\}\cup\\{\tfrac{1}{2}(\pm\lambda_{i}\pm\lambda_{j})\,:\,1\leq i<j\leq
r\\}.$

  2. (2)

$\mathcal{D}$ is of non-tube type. In this case

$\qquad\quad\displaystyle\Delta(\mathfrak{g},\mathfrak{a})=\\{\pm\tfrac{1}{2}\lambda_{i},\pm\lambda_{i}:1\leq
i\leq r\\}\cup\\{\tfrac{1}{2}(\pm\lambda_{i}\pm\lambda_{j})\,:\,1\leq i<j\leq
r\\}.$

In the case where $\mathcal{D}$ is of non-tube type, the root spaces
$\mathfrak{g}^{\frac{1}{2}\lambda_{j}}$ have a complex structure. For this,
let

$E=\sum_{j=1}^{r}\varphi_{j}\begin{pmatrix}i&-i\\\
i&-i\end{pmatrix}\in\bigoplus_{j=1}^{r}\mathfrak{g}^{\lambda_{j}},\qquad\mbox{and}\qquad
F=\theta(E)=\sum_{j=1}^{r}\varphi_{j}\begin{pmatrix}i&i\\\
-i&-i\end{pmatrix}\in\bigoplus_{j=1}^{r}\mathfrak{g}^{-\lambda_{j}}.$ (1.2)

######  Lemma 1.5 (see [11, Lemma 2.2.4] and [9, Chapter III, §2 and §3]).

The map $J:=\operatorname{ad}(E)\circ\theta=\theta\circ\operatorname{ad}(F)$
on $\bigoplus_{j=1}^{r}\mathfrak{g}^{\frac{1}{2}\lambda_{j}}$ has the
following properties:

  1. (1)

$J^{2}=-\operatorname{id}$,

  2. (2)

$J(\mathfrak{g}^{\frac{1}{2}\lambda_{j}})=\mathfrak{g}^{\frac{1}{2}\lambda_{j}}$
for all $1\leq j\leq r$,

  3. (3)

For every $1\leq j\leq r$, the complexified root space
$\mathfrak{g}^{\frac{1}{2}\lambda_{j}}_{\mathbb{C}}$ decomposes as

$\mathfrak{g}^{\frac{1}{2}\lambda_{j}}_{\mathbb{C}}=\mathfrak{g}^{\frac{1}{2}\lambda_{j},+}_{\mathbb{C}}\oplus\mathfrak{g}^{\frac{1}{2}\lambda_{j},-}_{\mathbb{C}},$

where

$\mathfrak{g}^{\frac{1}{2}\lambda_{j},\pm}_{\mathbb{C}}=\mathfrak{g}_{\mathbb{C}}^{\frac{1}{2}\lambda_{j}}\cap(\mathfrak{k}_{\mathbb{C}}\oplus\mathfrak{p}^{\pm})=\\{Z\in\mathfrak{g}_{\mathbb{C}}^{\frac{1}{2}\lambda_{j}}:JZ=\pm
iZ\\}.$

##  2\. The inclusion $G\subseteq P^{+}K_{\mathbb{C}}N_{\mathbb{C}}$

In this section we show that there is exactly one choice of a maximal
parabolic subgroup $P=LN\subseteq G$ such that $G\subseteq
P^{+}K_{\mathbb{C}}N_{\mathbb{C}}$. This embedding will be fundamental in our
construction of Whittaker vectors in Section 5.4. For tube type domains, this
inclusion was obtained earlier in [2]. The proof uses
$\operatorname{SU}(1,1)$-reduction by first proving the inclusion for
$G=\operatorname{SU}(1,1)$ and then using the commuting homomorphisms
$\varphi_{j}$, $j=1,\ldots,r$, to prove it for the general case. The proof
further provides us with the explicit decomposition of elements in a maximally
split torus which turns out to be important to prove the $L^{2}$-condition in
Theorem 5.8.

###  2.1. The case $G=\operatorname{SU}(1,1)$

Let $G=\operatorname{SU}(1,1)\subseteq\operatorname{SL}(2,\mathbb{C})$. We use
the notation from the last section and set $x_{0}=e_{0}+f_{0}$ and
$\mathfrak{a}=\mathbb{R}x_{0}$. Then we define

$\displaystyle A$
$\displaystyle=\exp\mathbb{R}x_{0}=\\{a_{t}:t\in\mathbb{R}\\},$ $\displaystyle
a_{t}$ $\displaystyle=\exp tx_{0}=\begin{pmatrix}\cosh t&\sinh t\\\ \sinh
t&\cosh t\end{pmatrix},$ $\displaystyle N$
$\displaystyle=\\{n_{x}:x\in\mathbb{R}\\},$ $\displaystyle n_{x}$
$\displaystyle=\exp x\begin{pmatrix}i&-i\\\
i&-i\end{pmatrix}=\begin{pmatrix}1+ix&-ix\\\ ix&1-ix\end{pmatrix}.$

The natural complexification of $N$ inside
$G_{\mathbb{C}}=\operatorname{SL}(2,\mathbb{C})$ is

$N_{\mathbb{C}}=\\{n_{z}:z\in\mathbb{C}\\},\qquad
n_{z}=\begin{pmatrix}1+iz&-iz\\\ iz&1-iz\end{pmatrix}.$

######  Theorem 2.1 (see [2, Lemma 4.1 and Theorem 4.2]).

A matrix

$g=\begin{pmatrix}a&b\\\ c&d\end{pmatrix}\in\operatorname{SL}(2,\mathbb{C})$

is contained in $P^{+}K_{\mathbb{C}}N_{\mathbb{C}}$ if and only if $c+d\neq
0$. More precisely, writing $p_{z}^{+}=\exp ze_{0}$ and $k_{\theta}=\exp\theta
z_{0}$, we have $g=p_{z}^{+}k_{\theta}n_{w}$ with

$e^{i\theta/2}=\frac{1}{c+d},\qquad w=\frac{-ic}{c+d}\qquad\text{and}\qquad
z=\frac{(a+b)(c+d)-1}{(c+d)^{2}}.$

In particular, $G\subseteq P^{+}K_{\mathbb{C}}N_{\mathbb{C}}$ and
$P^{+}K_{\mathbb{C}}N_{\mathbb{C}}$ is dense in $G_{\mathbb{C}}$.

######  Corollary 2.2.

For every $t\in\mathbb{R}$ we have $a_{t}=p^{+}_{z}k_{2it}n_{w}$ with

$z=1-e^{-2t}\qquad\mbox{and}\qquad w=\frac{1}{2i}(1-e^{-2t}).$

###  2.2. The general case

We now prove Theorem 2.1 for the general case. We use the notation from
Sections 1.2 and 1.3.

For $j=1,\ldots,r$ let $y_{j}=x_{1}+\cdots+x_{j}$. Then $y_{j}$ determines a
maximal parabolic subalgebra

$\mathfrak{p}_{j}=\mathfrak{l}_{j}\oplus\mathfrak{n}_{j}=\mathfrak{m}_{j}\oplus\mathbb{R}y_{j}\oplus\mathfrak{n}_{j}$

in $\mathfrak{g}$ given by

$\mathfrak{l}_{j}=\mathfrak{z}_{\mathfrak{g}}(y_{j})\quad\text{and}\quad\mathfrak{n}_{j}=\bigoplus_{\alpha(y_{j})>0}\mathfrak{g}^{\alpha}.$

In particular, it follows from Theorem 1.4 that
$\mathfrak{g}^{\lambda_{k}}\subseteq\mathfrak{n}_{j}$ if and only if $k\leq
j$. Denote by $P_{j}=L_{j}N_{j}$ the corresponding maximal parabolic subgroup
of $G$.

######  Theorem 2.3.

Let $j=1,\ldots,r$, then $G\subseteq P^{+}K_{\mathbb{C}}N_{j\mathbb{C}}$ if
and only if $j=r$.

###### Proof.

Write $P=P_{j}$ and $L=L_{j}$, $N=N_{j}$ for short. The set
$P^{+}K_{\mathbb{C}}N_{\mathbb{C}}$ is left $K_{\mathbb{C}}$-invariant and
right $(K\cap L)_{\mathbb{C}}N_{\mathbb{C}}$-invariant. We have $G=KLN$ and
$L=(K\cap L)A(K\cap L)$, so

$G=KA(K\cap L)N.$

It therefore suffice to show that $A\subseteq
P^{+}K_{\mathbb{C}}N_{\mathbb{C}}$ if and only if $j=r$. As
$\mathfrak{a}=\bigoplus_{k=1}^{r}\mathbb{R}x_{k}$, we have $A\subseteq
P^{+}K_{\mathbb{C}}N_{\mathbb{C}}$ if and only if $\exp tx_{k}\in
P^{+}K_{\mathbb{C}}N_{\mathbb{C}}$ for all $t\in\mathbb{R}$ and
$k=1,\ldots,r$. But $\operatorname{SU}(1,1)$-reduction shows that

$\exp
tx_{k}=\exp((1-e^{-2t})e_{k})\exp(-th_{k})\exp\left(\frac{1}{2i}(1-e^{-2t})\varphi_{k}\begin{pmatrix}i&-i\\\
i&-i\end{pmatrix}\right).$

By definition we have

$\varphi_{k}\begin{pmatrix}i&-i\\\
i&-i\end{pmatrix}\in\mathfrak{g}^{\lambda_{k}}.$

Thus, $\exp tx_{k}\in N_{\mathbb{C}}$ for all $t\in\mathbb{R}$ if and only if
$\mathfrak{g}^{\lambda_{k}}\subseteq\mathfrak{n}$. By the considerations above
this is true for all $1\leq k\leq r$ if and only if $j=r$. ∎

From now on we write $N_{\mathbb{C}}$ for $N_{r\mathbb{C}}$.

######  Remark 2.4.

Note that by applying $\sigma_{G}$ to the decomposition $G\subseteq
P^{+}K_{\mathbb{C}}N_{\mathbb{C}}$ and using that $\sigma_{G}(P^{+})=P^{-}$,
$\sigma_{G}(K_{\mathbb{C}})=K_{\mathbb{C}}$ and
$\sigma_{G}(N_{\mathbb{C}})=N_{\mathbb{C}}$, it also follows that $G\subseteq
P^{-}K_{\mathbb{C}}N_{\mathbb{C}}$.

In contrast to the case where $\mathcal{D}$ is of tube type, the
multiplication map $P^{+}\times K_{\mathbb{C}}\times N_{\mathbb{C}}\to
G_{\mathbb{C}}$ is not bijective. In fact, by Lemma 1.5 the intersection
$K_{\mathbb{C}}P^{+}\cap N_{\mathbb{C}}$ is non-trivial if and only if
$\mathcal{D}$ is of non-tube type, and in this case

$K_{\mathbb{C}}P^{+}\cap
N_{\mathbb{C}}=\exp\Bigg{(}\bigoplus_{j=1}^{r}\mathfrak{g}_{\mathbb{C}}^{\frac{1}{2}\lambda_{j},+}\Bigg{)}.$

Taking this intersection into account, we obtain:

######  Corollary 2.5.

The set $P^{+}K_{\mathbb{C}}N_{\mathbb{C}}$ is open in $G_{\mathbb{C}}$ and
the multiplication map

$(P^{+}\times K_{\mathbb{C}})\times_{K_{\mathbb{C}}P^{+}\cap
N_{\mathbb{C}}}N_{\mathbb{C}}\to
P^{+}K_{\mathbb{C}}N_{\mathbb{C}},\quad[p^{+},k,n]\mapsto p^{+}kn$

is biholomorphic.

######  Remark 2.6.

For $g\in P^{+}K_{\mathbb{C}}N_{\mathbb{C}}$ write

$g=p^{+}_{N}(g)k^{+}_{\mathbb{C},N}(g)n^{+}_{\mathbb{C}}(g)\in
P^{+}K_{\mathbb{C}}N_{\mathbb{C}}$

for some choice of decomposition. Whenever using this notation, the expression
under consideration will be independent of the chosen representatives.
Further note that if we decompose

$g=p^{-}_{N}(g)k^{-}_{\mathbb{C},N}(g)n^{-}_{\mathbb{C}}(g)\in
P^{-}K_{\mathbb{C}}N_{\mathbb{C}},$

then applying $\sigma$ yields

$p^{+}_{N}(g)=\sigma(p^{-}_{N}(g)),\qquad
k^{+}_{\mathbb{C},N}(g)=\sigma(k^{-}_{\mathbb{C},N}(g)),\qquad
n^{+}_{\mathbb{C}}(g)=\sigma(n^{-}_{\mathbb{C}}(g)).$ (2.1)

From the explicit calculation in the proof of Theorem 2.3 we further obtain:

######  Corollary 2.7.

For $j=1,\ldots,r$ and $t\in\mathbb{R}$ we have

$\displaystyle k_{\mathbb{C},N}(\exp tx_{j})$ $\displaystyle=\exp(-th_{j}),$
(2.2) $\displaystyle n_{\mathbb{C}}(\exp tx_{j})$
$\displaystyle=\exp\left(\frac{1}{2i}(1-e^{-2t})\varphi_{j}\begin{pmatrix}i&-i\\\
i&-i\end{pmatrix}\right).$ (2.3)

##  3\. The holomorphic discrete series

In this section we recall the realization of holomorphic discrete series
representations on spaces of holomorphic functions on the bounded domain
$\mathcal{D}$. We refer to [7, 8, 11] for more details.

We start by defining the universal cocycle and kernel, see [9, pp 64]. For
that we let $g\in G_{\mathbb{C}}$ and $z,w\in\mathfrak{p}^{+}$. Then, whenever
defined, which in particular is true for $g\in G$ and $z,w\in\mathcal{D}$, we
set, using the notation from Section 1.1:

  * (J)

(_universal cocycle_) $J(g,z):=k_{\mathbb{C}}(g\exp z)$,

  * (K)

(_universal kernel_)
$K(z,w)=k_{\mathbb{C}}(\exp(-\sigma_{\mathfrak{g}}(w))\exp(z))$.

The following relations are easy to verify:

######  Lemma 3.1.

For $k\in K_{\mathbb{C}}$ define $k^{*}=\sigma_{G}(k)^{-1}$. Then the
following holds true for $a,b\in G_{\mathbb{C}}$, $z,w\in\mathfrak{p}^{+}$ and
$k\in K_{\mathbb{C}}$:

  1. (1)

$J(k,z)=k$,

  2. (2)

$J(ab,z)=J(a,b\cdot z)J(b,z)$,

  3. (3)

$K(z,o)=K(o,w)=e$,

  4. (4)

$K(z,w)=K(w,z)^{*}$.

Moreover, if $a\in G$ we further have

  1. (5)

$J(a,w)^{*}K(a\cdot z,a\cdot w)J(a,z)=K(z,w)$,

  2. (6)

$K(a\cdot o,a\cdot o)=J(a,o)^{-*}J(a,o)^{-1}$.

Now fix a finite-dimensional holomorphic representation
$(\pi,\mathcal{V}_{\pi})$ of $K_{\mathbb{C}}$. We endow $\mathcal{V}_{\pi}$
with an inner product that makes $\pi$ unitary, so that

$\pi(k)^{*}=\pi(k^{*})\qquad(k\in K_{\mathbb{C}}).$ (3.1)

Let

$j_{\pi}(g,z)=\pi(J(g,z))^{-1}\qquad\mbox{and}\qquad
K_{\pi}(z,w)=\pi(K(z,w))^{-1},$ (3.2)

whenever $g\in G$ and $z,w\in\mathcal{D}$. Then by Lemma 3.1 (2), the formula

$U_{\pi}(g)F(z)=j_{\pi}(g^{-1},z)F(g^{-1}z)\qquad(g\in
G,F\in\mathcal{O}(\mathcal{D},\mathcal{V}_{\pi}),z\in\mathcal{D})$

defines a representation of $G$ on
$\mathcal{O}(\mathcal{D},\mathcal{V}_{\pi})$, the space of
$\mathcal{V}_{\pi}$-valued holomorphic functions on $\mathcal{D}$. By Lemma
3.1 (5) it preserves the inner product

$\langle F_{1},F_{2}\rangle_{\pi}=\int_{\mathcal{D}}\langle
K_{\pi}(z,z)^{-1}F_{1}(z),F_{2}(z)\rangle\,d^{*}z,$

where $d^{*}z$ denotes a $G$-invariant measure on $\mathcal{D}$. This defines
a unitary representation $(U_{\pi},\mathcal{H}_{\pi})$, but
$\mathcal{H}_{\pi}$ could be zero. A necessary and sufficient condition for
$\mathcal{H}_{\pi}\neq\\{0\\}$ can be given in terms of the highest weight
$\mu_{\pi}\in\mathfrak{t}_{\mathbb{C}}^{*}$ of $\pi$:

######  Theorem 3.2 (Harish-Chandra).

Let
$\rho=\frac{1}{2}\sum_{\alpha\in\Delta^{+}(\mathfrak{g}_{\mathbb{C}},\mathfrak{t}_{\mathbb{C}})}\alpha$.
Then $\mathcal{H}_{\pi}\neq\\{0\\}$ if and only if

$\langle\mu+\rho,\alpha\rangle<0,\quad\text{for all }\alpha\in\Delta_{n}^{+}.$
(3.3)

In this case, $(U_{\pi},\mathcal{H}_{\pi})$ is irreducible and belongs to the
discrete series.

The representations $(U_{\pi},\mathcal{H}_{\pi})$ are called _holomorphic
discrete series representations_. Moreover, the Hilbert space
$\mathcal{H}_{\pi}$ is a reproducing kernel Hilbert space with reproducing
kernel $K_{\pi}(z,w)$ (after suitable normalization of the measure $d^{*}z$).

Note that the condition (3.3) only depends on the restriction of $\mu_{\pi}$
to $\mathfrak{c}_{\mathbb{C}}$. Writing
$\mu_{\pi}|_{\mathfrak{c}_{\mathbb{C}}}=-\frac{1}{2}\sum_{j=1}^{r}\mu_{\pi,j}\gamma_{j}$
and $\rho|_{\mathfrak{c}_{\mathbb{C}}}=\frac{1}{2}\sum_{j=1}^{r}\rho_{j}$, we
find that (3.3) is equivalent to

$\mu_{\pi,j}\geq\rho_{j}\qquad\mbox{for all }1\leq j\leq r.$ (3.4)

In this notation, the action of the torus $\exp\mathfrak{c}_{\mathbb{C}}$ on a
highest weight vector $\xi\in\mathcal{V}_{\pi}$ is given by

$\exp\left(\sum_{j=1}^{r}t_{j}h_{j}\right)\xi=\left(\prod_{j=1}^{r}e^{-\mu_{\pi,j}t_{j}}\right)\xi.$
(3.5)

######  Example 3.3 (The case of $G=\operatorname{SU}(1,1)$).

We identify $K_{\mathbb{C}}$ with $\mathbb{C}^{*}$ by $k_{\theta}\mapsto
e^{\theta}$ and $\mathfrak{p}^{+}$ with $\mathbb{C}$ by $ze_{0}\mapsto e_{0}$.
For

$g=\begin{pmatrix}a&b\\\ c&d\end{pmatrix}\in\operatorname{SL}(2,\mathbb{C})$

and $z,w\in\mathbb{C}$, $\gamma\in\mathbb{C}^{*}$ we have

$g\exp ze_{0}=\begin{pmatrix}a&b\\\ c&d\end{pmatrix}\begin{pmatrix}1&z\\\
0&1\end{pmatrix}=\begin{pmatrix}a&az+b\\\ c&cz+d\end{pmatrix}.$

Thus $J(g,z)=(cz+d)^{-1}$ by (1.1). Moreover, for $z,w\in\mathbb{C}$ we find

$\exp(-\bar{w}f_{0})\exp(ze_{0})=\begin{pmatrix}1&0\\\
-\bar{w}&1\end{pmatrix}\begin{pmatrix}1&z\\\
0&1\end{pmatrix}\begin{pmatrix}1&z\\\ -\bar{w}&1-z\bar{w}\end{pmatrix},$

so, again by (1.1), we have $K(z,w)=(1-z\bar{w})^{-1}$.

The finite-dimensional irreducible representation of
$\widetilde{K}\simeq\mathbb{R}$ are the characters $t\mapsto e^{i\lambda t}$.
We then have

$j_{\lambda}(g,z)=(cz+d)^{-\lambda}\quad\text{and}\quad
K_{\lambda}(z,w)=(1-z\bar{w})^{-\lambda}.$

Note that

$j_{\lambda}(a_{t},z)=(\sinh(t)z+\cosh(t))^{-\lambda},\quad
j_{\lambda}(a_{t},0)=\cosh(t)^{-\lambda}.$

The condition (3.3) is equivalent to $\lambda>1$.

##  4\. Fock space representations

We recall the construction of the irreducible unitary representations of the
nilpotent group $N$. Recall from Section 2.2 that
$\mathfrak{n}=\mathfrak{n}_{\frac{1}{2}}\oplus\mathfrak{n}_{1}$, where

$\mathfrak{n}_{\frac{1}{2}}=\bigoplus_{j=1}^{r}\mathfrak{g}^{\frac{1}{2}\lambda_{j}},\qquad\mbox{and}\qquad\mathfrak{n}_{1}=\bigoplus_{1\leq
j\leq k\leq r}\mathfrak{g}^{\frac{1}{2}(\lambda_{j}+\lambda_{k})}$

with
$[\mathfrak{n}_{\frac{1}{2}},\mathfrak{n}_{\frac{1}{2}}]=\mathfrak{n}_{1}$ and
$[\mathfrak{n}_{\frac{1}{2}}\oplus\mathfrak{n}_{1},\mathfrak{n}]=\\{0\\}$.
Hence, $\mathfrak{n}$ is abelian if $\mathcal{D}$ is of tube-type and two-step
nilpotent otherwise. In particular, $\mathfrak{n}_{1}$ is the center of
$\mathfrak{n}$. It follows that for every non-trivial central character
$\psi:N\to S^{1}$ there exists a unique irreducible unitary representation
$\omega_{\psi}$ of $N$ such that
$\omega_{\psi}(n)=\psi(n)\cdot\operatorname{id}$ for all
$n\in\exp(\mathfrak{n}_{1})$. If $\mathfrak{n}$ is abelian, then
$\omega_{\psi}=\psi$ is simply a unitary character of $N$, and if
$\mathfrak{n}$ is two-step nilpotent the representation $\omega_{\psi}$ is
infinite-dimensional.

We realize $\omega_{\psi}$ explicitly on a Fock space $\mathcal{F}$ of
holomorphic functions on $\mathfrak{n}_{\frac{1}{2}}$, with the convention
that $\mathcal{F}=\mathbb{C}$ if $\mathfrak{n}_{\frac{1}{2}}=\\{0\\}$. More
precisely, let $J$ be the complex structure on $\mathfrak{n}_{\frac{1}{2}}$
given by Lemma 1.5. The Killing form $\kappa$ on $\mathfrak{g}$ gives rise to
an inner product on the real vector space $\mathfrak{n}$:

$(z|w)=\kappa(z,\theta w)\qquad(z,w\in\mathfrak{n}).$

Using the complex structure $J$, we define a Hermitian form on
$\mathfrak{n}_{\frac{1}{2}}$:

$\langle z,w\rangle=\kappa(z,\theta w)-i\kappa(Jz,\theta
w)\qquad(z,w\in\mathfrak{n}_{\frac{1}{2}}).$

It is easy to see that this form is indeed Hermitian with respect to the
complex structure induced by $J$. The Fock space $\mathcal{F}$ is defined by

$\mathcal{F}=\left\\{\zeta\in\mathcal{O}(\mathfrak{n}_{\frac{1}{2}}):\int_{\mathfrak{n}_{\frac{1}{2}}}|\zeta(w)|^{2}e^{-|w|^{2}}\,dw<\infty\right\\}$

with the obvious norm, where $dw$ denotes Lebesgue measure on
$\mathfrak{n}_{\frac{1}{2}}$ normalized such that $e^{-|w|^{2}}\,dw$ is a
probability measure. $\mathcal{F}$ is a reproducing kernel Hilbert space with
reproducing kernel given by

$K_{\mathcal{F}}(z,w)=e^{\langle
z,w\rangle}\qquad(z,w\in\mathfrak{n}_{\frac{1}{2}}).$ (4.1)

The group $N$ acts on $\mathcal{F}$ by

$\displaystyle\omega(\exp(z))\zeta(w)$ $\displaystyle=e^{\langle
w,z\rangle-\frac{1}{2}|z|^{2}}\zeta(w-z)$
$\displaystyle(z\in\mathfrak{n}_{\frac{1}{2}}),$ (4.2)
$\displaystyle\omega(\exp(x))\zeta(w)$ $\displaystyle=e^{2i(x|E)}\zeta(w)$
$\displaystyle(x\in\mathfrak{n}_{1}),$ (4.3)

where $\zeta\in\mathcal{F}$ and $w\in\mathfrak{n}_{\frac{1}{2}}$ and $E$ as in
(1.2). Here, we normalize the Killing form such that $(E|E)=r$. Note that
$K\cap L$ leaves $E$ invariant.

On the dense subspace $\mathcal{F}_{0}$ spanned by functions of the form

$\zeta(z)=p(z)e^{\langle z,w\rangle}\qquad(z\in\mathfrak{n}_{\frac{1}{2}}),$

where $p$ is a polynomial and $w\in\mathfrak{n}_{\frac{1}{2}}$, this
representation extends to the complexification $N_{\mathbb{C}}$ of $N$. Recall
that

$\mathfrak{n}_{\frac{1}{2},\mathbb{C}}=\mathfrak{n}_{\frac{1}{2},\mathbb{C}}^{+}\oplus\mathfrak{n}_{\frac{1}{2},\mathbb{C}}^{-}\qquad\mbox{with
}\mathfrak{n}_{\frac{1}{2},\mathbb{C}}^{\pm}=\\{\tfrac{1}{2}(z\mp
iJz):z\in\mathfrak{n}_{\frac{1}{2}}\\}$

and put

$n_{z}^{\pm}=\exp\left(\frac{1}{2}(z\mp
iJz)\right)\in\exp(\mathfrak{n}_{\frac{1}{2},\mathbb{C}}^{\pm})\qquad(z\in\mathfrak{n}_{\frac{1}{2}}).$
(4.4)

It follows that for $\zeta\in\mathcal{F}_{0}$ and
$z,w\in\mathfrak{n}_{\frac{1}{2}}$:

$\displaystyle\omega(n_{z}^{+})\zeta(w)$ $\displaystyle=\zeta(w-z),$ (4.5)
$\displaystyle\omega(n_{z}^{-})\zeta(w)$ $\displaystyle=e^{\langle
w,z\rangle}\zeta(w).$ (4.6)

In particular,

$\omega(n_{z}^{+})^{*}=\omega(n_{z}^{-})^{-1}\qquad(z\in\mathfrak{n}_{\frac{1}{2}}).$
(4.7)

The compact group $K\cap L$ acts on $\mathfrak{n}_{\frac{1}{2}}$,
$\mathfrak{n}_{\frac{1}{2},\mathbb{C}}$ and
$\mathfrak{n}_{\frac{1}{2},\mathbb{C}}^{\pm}$ by the adjoint representation.
We have

$\omega(n_{\operatorname{Ad}(k)z}^{\pm})=\tau(k)\circ\omega(n_{z}^{\pm})\circ\tau(k)^{-1}\qquad(k\in
K\cap L,z\in\mathfrak{n}_{\frac{1}{2}}),$

where $\tau$ is the unitary representation of $K\cap L$ on $\mathcal{F}$ given
by

$\tau(k)\zeta(z)=\zeta(\operatorname{Ad}(k)^{-1}z)\qquad(k\in K\cap
L,\zeta\in\mathcal{F},z\in\mathfrak{n}_{\frac{1}{2}}).$

##  5\. Whittaker models for holomorphic discrete series

We fix the irreducible unitary representation $(\omega,\mathcal{F})$ of $N$
constructed in the previous section. For a holomorphic discrete series
representation $(U_{\pi},\mathcal{H}_{\pi})$ we study three types of objects:

  * •

(Whittaker vectors)
$W\in\operatorname{Hom}_{N}(\mathcal{H}_{\pi}^{\infty},\mathcal{F})$

  * •

(Intertwining operators)
$T\in\operatorname{Hom}_{G}(\mathcal{H}_{\pi}^{\infty},C^{\infty}(G/N,\omega))$

  * •

(Kernels)
$\Psi:G\times\mathcal{D}\to\operatorname{Hom}(\mathcal{F},\mathcal{V}_{\pi})$

Using explicit kernels, we embed holomorphic discrete series representations
$(U_{\pi},\mathcal{H}_{\pi})$ into $L^{2}(G/N,\omega)$, the space of
$L^{2}$-sections of the homogeneous vector bundle $G\times_{N}\mathcal{F}$
over $G/N$ induced from $(\omega,\mathcal{F})$.

###  5.1. Whittaker vectors vs. intertwining operators

Let $W\in\operatorname{Hom}_{N}(\mathcal{H}_{\pi}^{\infty},\mathcal{F})$ be a
Whittaker functional. Then we obtain a continuous $G$-equivariant embedding

$\displaystyle T:\mathcal{H}_{\pi}^{\infty}\to C^{\infty}(G/N,\omega)=\\{f\in
C^{\infty}(G,\mathcal{F}):f(gn)=\omega(n)^{-1}f(g)\mbox{ for }g\in G,n\in
N\\},$ $\displaystyle
T(F)(x)=W(U_{\pi}(x)^{-1}F)\qquad(F\in\mathcal{H}_{\pi}^{\infty},x\in G).$

On the other hand, every continuous $G$-equivariant linear operator
$T:\mathcal{H}_{\pi}^{\infty}\to C^{\infty}(G/N,\omega)$ gives rise to a
Whittaker functional
$W\in\operatorname{Hom}_{N}(\mathcal{H}_{\pi}^{\infty},\mathcal{F})$ by

$W(F)=T(F)(e)\qquad(F\in\mathcal{H}_{\pi}^{\infty}).$ (5.1)

This gives the Frobenius reciprocity isomorphism

$\operatorname{Hom}_{N}(\mathcal{H}_{\pi}^{\infty},\mathcal{F})\simeq\operatorname{Hom}_{G}(\mathcal{H}_{\pi}^{\infty},C^{\infty}(G/N,\omega)),\quad
W\mapsto T.$

###  5.2. From intertwining operators to kernels

To every
$T\in\operatorname{Hom}_{G}(\mathcal{H}_{\pi}^{\infty},C^{\infty}(G/N,\omega))$
we can now associate a kernel function
$\Psi:G\times\mathcal{D}\to\operatorname{Hom}(\mathcal{F},\mathcal{V}_{\pi})$
in the following way. For every $z\in\mathcal{D}$ and
$\xi\in\mathcal{V}_{\pi}$, the reproducing kernel $K_{z}\xi$ is contained in
$\mathcal{H}_{\pi}^{\infty}$, and we can put

$\Psi^{*}(x,z)\xi=T(K_{z,\xi})(x)=T(U_{\pi}(x)^{*}K_{z,\xi})(e)\qquad(x\in
G,z\in\mathcal{D},\xi\in\mathcal{V}_{\pi}).$

This defines a map
$\Psi^{*}:G\times\mathcal{D}\to\operatorname{Hom}(\mathcal{V}_{\pi},\mathcal{F})$.
Since $K_{z,\xi}$ is antiholomorphic in $z\in\mathcal{D}$, the function
$\Psi^{*}(x,z)$ is continuous in $x\in G$ and antiholomorphic in
$z\in\mathcal{D}$. Taking the adjoint of $\Psi^{*}(x,z)$, we obtain

$\Psi:G\times\mathcal{D}\to\operatorname{Hom}(\mathcal{F},\mathcal{V}_{\pi}),\quad\Psi(x,z):=\Psi^{*}(x,z)^{*}.$

######  Lemma 5.1.

The function $\Psi(x,z)$ is continuous in $x\in G$ and holomorphic in
$z\in\mathcal{D}$. Moreover, it has the following invariance:

  1. (1)

$\Psi(xn,z)=\Psi(x,z)\circ\omega(n)$ for $n\in N$.

  2. (2)

$\Psi(gx,z)=j_{\pi}(g^{-1},z)\circ\Psi(x,g^{-1}z)$ for all $g,x\in G$ and
$z\in\mathcal{D}$.

###### Proof.

The proof is the same as in [2, Lemma 4.4]. ∎

By the previous lemma, the kernel $\Psi(x,z)$ is uniquely determined by its
values at $(x,o)$ for $x\in G$. We therefore let

$\widetilde{\Psi}:G\to\operatorname{Hom}(\mathcal{F},\mathcal{V}_{\pi}),\quad\widetilde{\Psi}(x)=\Psi(x,o),$

noting that

$\Psi(x,g\cdot o)=j_{\pi}(g,o)^{-1}\circ\widetilde{\Psi}(g^{-1}x)\qquad(x,g\in
G).$ (5.2)

Using the equivariance properties from before, we can express the values of
$\widetilde{\Psi}$ in terms of the decomposition $G\subseteq
P^{-}K_{\mathbb{C}}N_{\mathbb{C}}$ (see Remark 2.4).

######  Lemma 5.2.

Let $g\in G$ and write $g=p^{-}kn\in P^{-}K_{\mathbb{C}}N_{\mathbb{C}}$. Then
the following identity holds in
$\operatorname{Hom}(\mathcal{F}_{0},\mathcal{V}_{\pi})$ with $\mathcal{F}_{0}$
as in Section 4:

$\widetilde{\Psi}(g)=\pi(k)\circ\widetilde{\Psi}(e)\circ\omega(n).$

###### Proof.

The same as in [2, Remark 4.5 (2)]. ∎

It follows that $\widetilde{\Psi}$ is uniquely determined by
$\widetilde{\Psi}(e)=\Psi(e,o)\in\operatorname{Hom}(\mathcal{F},\mathcal{V}_{\pi})$.
The latter space is infinite-dimensional in the case where $\mathcal{D}$ is of
non-tube type. However, by Lemma 5.2 there is some restriction on
$\widetilde{\Psi}(e)$: for $n\in N_{\mathbb{C}}\cap(K_{\mathbb{C}}P^{-})$ we
have

$\widetilde{\Psi}(e)\circ\omega(n)=\widetilde{\Psi}(n)=\pi(n)\circ\widetilde{\Psi}(e),$

where we view $(\pi,\mathcal{V}_{\pi})$ as a representation of
$K_{\mathbb{C}}P^{-}$ by extending it trivially to $P^{-}$. It follows that
$A=\widetilde{\Psi}(e)$ belongs to
$\operatorname{Hom}_{N_{\mathbb{C}}\cap(K_{\mathbb{C}}P^{-})}(\mathcal{F}_{0},\mathcal{V}_{\pi})$.

######  Lemma 5.3.

Every
$A\in\operatorname{Hom}_{N_{\mathbb{C}}\cap(K_{\mathbb{C}}P^{-})}(\mathcal{F}_{0},\mathcal{V}_{\pi})$
is of the form

$A\zeta=\int_{\mathfrak{n}_{\frac{1}{2}}}\zeta(w)\pi(n_{w}^{-})\eta\cdot
e^{-|w|^{2}}\,dw\qquad(\zeta\in\mathcal{F})$

for some $\eta\in\mathcal{V}_{\pi}$.

###### Proof.

Taking adjoints, we have

$\omega(n)^{*}\circ A^{*}=A^{*}\circ\pi(n)^{*}\qquad(n\in
N_{\mathbb{C}}\cap(K_{\mathbb{C}}P^{-})).$

Since $N_{\mathbb{C}}\cap
K_{\mathbb{C}}P^{-}=\\{n_{z}^{-}:z\in\mathfrak{n}_{\frac{1}{2}}\\}$ (see Lemma
1.5 and (4.4)), using (4.1) and (4.5) we find

$\displaystyle A^{*}\xi(z)$ $\displaystyle=[\omega(n_{z}^{+})^{-1}\circ
A^{*}\xi](0)=[\omega(n_{z}^{-})^{*}\circ A^{*}\xi](0)$
$\displaystyle=[A^{*}\circ\pi(n_{z}^{-})^{*}\xi](0)=\langle
A^{*}\circ\pi(n_{z}^{-})^{*}\xi,\mathbf{1}\rangle=\langle\xi,\pi(n_{z}^{-})\circ
A\mathbf{1}\rangle.$

Writing $\eta=A\mathbf{1}$ it follows that

$\langle
A\zeta,\xi\rangle=\langle\zeta,A^{*}\xi\rangle=\int_{\mathfrak{n}_{\frac{1}{2}}}\zeta(w)\langle\pi(n_{z}^{-})\eta,\xi\rangle
e^{-|w|^{2}}\,dw$

and the claim follows. ∎

######  Corollary 5.4.

$\widetilde{\Psi}(e)\in\operatorname{Hom}(\mathcal{F}_{0},\mathcal{V}_{\pi})$
is given by

$\widetilde{\Psi}(e)\zeta=\int_{\mathfrak{n}_{\frac{1}{2}}}\zeta(w)\pi(n_{w}^{-})\eta\cdot
e^{-|w|^{2}}\,dw\qquad(\zeta\in\mathcal{F})$

for some $\eta\in\mathcal{V}_{\pi}$. In particular, the space
$\operatorname{Hom}_{N}(\mathcal{H}_{\pi}^{\infty},\mathcal{F})$ of Whittaker
vectors on $\mathcal{H}_{\pi}$ is of dimension at most
$\dim\mathcal{V}_{\pi}$.

To show that the dimension is actually equal to $\dim\mathcal{V}_{\pi}$, we
construct in Section 5.4 kernel functions $\Psi$ explicitly and show that they
give rise to Whittaker vectors.

###  5.3. The case of unitary intertwining operators

We assume for a moment that $T$ extends to an embedding
$T:\mathcal{H}_{\pi}\to L^{2}(G/N,\omega)$. Then its adjoint
$T^{*}:L^{2}(G/N,\omega)\to\mathcal{H}_{\pi}$ is an integral operator with
integral kernel $\Psi(x,z)$:

######  Proposition 5.5.

If $T:\mathcal{H}_{\pi}\to L^{2}(G/N,\omega)$ is a $G$-equivariant continuous
linear map and $\Psi$ as in Section 5.2, then its adjoint
$T^{*}:L^{2}(G/N,\omega)\to\mathcal{H}_{\pi}$ is given by

$T^{*}f(z)=\int_{G/N}\Psi(x,z)f(x)\,d(xN)\qquad(f\in
L^{2}(G/N,\omega),z\in\mathcal{D}).$

###### Proof.

For $f\in L^{2}(G/N,\omega)$ we have

$\displaystyle\langle T^{*}f(z),\xi\rangle$ $\displaystyle=\langle
T^{*}f,K_{z,\xi}\rangle=\langle f,T(K_{z,\xi})\rangle$
$\displaystyle=\int_{G/N}\langle f(x),\Psi(x,z)^{*}\xi\rangle\,d(xN)$
$\displaystyle=\int_{G/N}\langle\Psi(x,z)f(x),\xi\rangle\,d(xN).\qed$

######  Corollary 5.6.

If $T:\mathcal{H}_{\pi}\to L^{2}(G/N,\omega)$ is a $G$-equivariant continuous
linear map and $\Psi$ as in Section 5.2, then

$TF(x)=\int_{\mathcal{D}}\Psi(x,z)^{*}K_{\pi}(z,z)^{-1}F(z)\,d^{*}z\qquad(F\in\mathcal{H}_{\pi},x\in
G).$

###### Proof.

For $f\in L^{2}(G/N,\omega)$ we have, in view of the previous proposition:

$\displaystyle\langle TF,f\rangle$ $\displaystyle=\langle
F,T^{*}f\rangle=\int_{\mathcal{D}}\langle
K_{\pi}(z,z)^{-1}F(z),T^{*}f(z)\rangle\,d^{*}z$
$\displaystyle=\int_{\mathcal{D}}\int_{G/N}\langle
K_{\pi}(z,z)^{-1}F(z),\Psi(x,z)f(x)\rangle\,d(xN)\,d^{*}z$
$\displaystyle=\int_{G/N}\left\langle\int_{\mathcal{D}}\Psi(x,z)^{*}K_{\pi}(z,z)^{-1}F(z)d^{*}z,f(x)\right\rangle\,d(xN).\qed$

###  5.4. From kernels to unitary intertwining operators

Now let $\eta\in\mathcal{V}_{\pi}$ and define
$A_{\eta}\in\operatorname{Hom}(\mathcal{F},\mathcal{V}_{\pi})$ by

$A_{\eta}\zeta=\int_{\mathfrak{n}_{\frac{1}{2}}}\zeta(w)\pi(n_{w}^{-})\eta\cdot
e^{-|w|^{2}}\,dw\qquad(\zeta\in\mathcal{F}).$

Note that since $\pi(n_{w}^{-})\eta$ is an antiholomorphic polynomial in $w$,
the integral converges for all $\zeta\in\mathcal{F}$. In view of the proof of
Lemma 5.3, the adjoint
$A_{\eta}^{*}\in\operatorname{Hom}(\mathcal{V}_{\pi},\mathcal{F})$ is given by

$A_{\eta}^{*}\xi(z)=\langle\xi,\pi(n_{z}^{-})\circ\eta\rangle=\langle\pi(n_{z}^{+})^{-1}\xi,\eta\rangle\qquad(\xi\in\mathcal{V}_{\pi},z\in\mathfrak{n}_{\frac{1}{2}}).$

We define
$\widetilde{\Psi}_{\pi,\eta}:G\to\operatorname{Hom}(\mathcal{F}_{0},\mathcal{V}_{\pi})$
at $s=p^{-}_{N}(s)k_{\mathbb{C},N}^{-}(s)n_{\mathbb{C}}^{-}(s)\in
P^{-}K_{\mathbb{C}}N_{\mathbb{C}}$ by

$\widetilde{\Psi}_{\pi,\eta}(s)=\pi(k_{\mathbb{C},N}^{-}(s))\circ
A_{\eta}\circ\omega(n_{\mathbb{C}}^{-}(s)).$

By the $(N_{\mathbb{C}}\cap K_{\mathbb{C}}P^{-})$-equivariance of $A_{\eta}$,
this is well-defined (cf. Remark 2.6) and gives rise to a kernel function
$\Psi_{\pi,\eta}:G\times\mathcal{D}\to\operatorname{Hom}(\mathcal{F}_{0},\mathcal{V}_{\pi})$
by (5.2). For later purpose, we note the following formula for the adjoint
$\Psi_{\pi,\eta}^{*}:G\times\mathcal{D}\to\operatorname{Hom}(\mathcal{V}_{\pi},\mathcal{F})$
which follows from (2.1), (3.1) and (4.7):

$\displaystyle\Psi_{\pi,\eta}(x,g\cdot o)^{*}$
$\displaystyle=\widetilde{\Psi}_{\pi,\eta}(g^{-1}x)^{*}\circ
j_{\pi}(g,o)^{-*}$ $\displaystyle=\omega(n_{\mathbb{C}}^{-}(g^{-1}x))^{*}\circ
A_{\eta}^{*}\circ\pi(k_{\mathbb{C},N}^{-}(g^{-1}x))^{*}\circ
j_{\pi}(g,o)^{-*}$
$\displaystyle=\omega(n_{\mathbb{C}}^{+}(g^{-1}x))^{-1}\circ
A_{\eta}^{*}\circ\pi(k_{\mathbb{C},N}^{+}(g^{-1}x))^{-1}\circ
j_{\pi}(g,o)^{-*}.$ (5.3)

Using the notation

$m:\mathcal{V}_{\pi}\otimes\overline{\mathcal{V}}_{\pi}\to\mathcal{F},\quad
m(\xi\otimes\overline{\eta})(z)=\langle\pi(n_{z}^{+})^{-1}\xi,\eta\rangle\qquad(\xi,\eta\in\mathcal{V}_{\pi},z\in\mathfrak{n}_{\frac{1}{2}}),$
(5.4)

so that $m(\xi\otimes\overline{\eta})$ is a polynomial and hence contained in
$\mathcal{F}_{0}$, this can be written as

$\Psi_{\pi.,\eta}(x,g\cdot
o)^{*}\xi=\omega(n_{\mathbb{C}}^{+}(g^{-1}x))^{-1}m\Big{(}\pi(k_{\mathbb{C},N}^{+}(g^{-1}x))^{-1}\circ
j_{\pi}(g,o)^{-*}\xi\otimes\overline{\eta}\Big{)}.$ (5.5)

In particular,
$\Psi_{\pi,\eta}(x,z)^{*}\in\operatorname{Hom}(\mathcal{V}_{\pi},\mathcal{F})$
so $\Psi_{\pi,\eta}(x,z)\in\operatorname{Hom}(\mathcal{F},\mathcal{V}_{\pi})$
instead of $\operatorname{Hom}(\mathcal{F}_{0},\mathcal{V}_{\pi})$.

In view of Corollary 5.6, we define (whenever the integral converges)

$T_{\pi,\eta}:\mathcal{H}_{\pi}^{\infty}\to C^{\infty}(G/N,\omega),\quad
T_{\pi,\eta}F(x)=\int_{\mathcal{D}}\Psi_{\pi,\eta}(x,z)^{*}K_{\pi}(z,z)^{-1}F(z)\,d^{*}z.$

In order to show that $T_{\pi,\eta}$ defines an intertwining operator
$\mathcal{H}_{\pi}\to L^{2}(G/N,\omega)$, it suffices to show that
$T_{\pi,\eta}F\in L^{2}(G/N,\omega)$ for $F$ contained in the lowest $K$-type.
Recall that the lowest $K$-type of $\mathcal{H}_{\pi}$ is spanned by the
functions $z\mapsto K_{\pi}(z,o)\xi=\xi$ for $\xi\in\mathcal{V}_{\pi}$.

######  Lemma 5.7.

Identifying $\xi\in\mathcal{V}_{\pi}$ with the constant function
$z\mapsto\xi$, the restriction of $T_{\pi,\eta}$ to $\mathcal{V}_{\pi}$ is
given by

$T_{\pi,\eta}\xi(x)=\Psi_{\pi,\eta}(x,o)^{*}\xi=\omega(n_{\mathbb{C}}^{+}(x))^{-1}\circ
A_{\eta}^{*}\circ\pi(k_{\mathbb{C},N}^{+}(x))^{-1}\xi\qquad(\xi\in\mathcal{V}_{\pi},x\in
G).$

###### Proof.

The first identity follows from the reproducing kernel property of $K_{\pi}$
and the second identity is a special case of (5.3). ∎

In particular, for $x=k_{1}ak_{2}$ with $k_{1}\in K$, $a\in A$ and $k_{2}\in
K\cap L$ we find

$T_{\pi,\eta}\xi(x)=\omega(k_{2}^{-1}n_{\mathbb{C}}^{+}(a)k_{2})^{-1}\circ
A_{\eta}^{*}\circ\pi(k_{1}k_{\mathbb{C},N}^{+}(a)k_{2})^{-1}\xi.$

By $\operatorname{SU}(1,1)$-reduction, $n_{\mathbb{C}}^{-}(a)$ is contained in
the center of $N_{\mathbb{C}}$ for $a\in A$. Since $K\cap L$ leaves
$E\in\mathfrak{n}_{1}$ invariant, it follows from (4.3) that
$\omega(k_{2}^{-1}n_{\mathbb{C}}^{+}(a)k_{2})=\omega(n_{\mathbb{C}}^{+}(a))$.
This implies:

$T_{\pi,\eta}\xi(x)=\omega(n_{\mathbb{C}}^{+}(a))^{-1}\circ
A_{\eta}^{*}\circ\pi(k_{2})^{-1}\circ\pi(k_{\mathbb{C},N}^{+}(a))^{-1}\circ\pi(k_{1})^{-1}\xi.$

######  Theorem 5.8.

$T_{\pi,\eta}\xi\in L^{2}(G/N,\omega)$ for some resp. all
$\xi\in\mathcal{V}_{\pi}$ if and only if $\mathcal{H}_{\pi}$ belongs to the
holomorphic discrete series.

###### Proof.

We use the integral formulas

$\int_{G/N}\varphi(x)\,d(xN)=\int_{K}\int_{L}\varphi(kl)l^{2\rho_{\mathfrak{n}}}\,dl\,dk$

and

$\int_{L}\varphi(l)\,dl=\int_{K\cap L}\int_{K\cap
L}\int_{A}\varphi(k_{1}ak_{2})w(a)\,da\,dk_{1}\,dk_{2},$

where

$w(\exp
H)=\prod_{\alpha\in\Delta^{+}(\mathfrak{l},\mathfrak{a})}(\sinh\alpha(H))^{\dim\mathfrak{l}^{\alpha}}.$

With this, we find

$\int_{G/N}\|T_{\pi,\eta}\xi\|^{2}\,d(gN)=\int_{K}\int_{A}\int_{K\cap
L}\|\omega(n_{\mathbb{C}}^{+}(a))^{-1}\circ\\\
A_{\eta}^{*}\circ\pi(k_{2})^{-1}\circ\pi(k_{\mathbb{C},N}^{+}(a))^{-1}\circ\pi(k_{1})^{-1}\xi\|^{2}\,a^{2\rho_{n}}w(a)\,dk_{2}\,da\,dk_{1}.$

By Schur’s orthogonality relations this equals up to a constant

$\int_{A}\|\omega(n_{\mathbb{C}}^{+}(a))^{-1}\circ
A_{\eta}^{*}\circ\pi(k_{\mathbb{C},N}^{+}(a))^{-1}\xi\|^{2}a^{2\rho_{\mathfrak{n}}}w(a)\,da.$
(5.6)

Now, by the $\operatorname{SU}(1,1)$-reduction in Corollary 2.7 we find for
$a=\exp\sum_{j=1}^{r}t_{j}x_{j}$:

$\displaystyle k_{\mathbb{C},N}^{+}(a)$
$\displaystyle=\exp\left(-\sum_{j=1}^{r}t_{j}h_{j}\right),$ $\displaystyle
n_{\mathbb{C}}^{+}(a)$
$\displaystyle=\exp\left(\frac{1}{2i}\sum_{j=1}^{r}(1-e^{-2t_{j}})\varphi_{j}\begin{pmatrix}i&-i\\\
i&-i\end{pmatrix}\right).$

By (4.3) we have

$\omega(n_{\mathbb{C}}^{+}(a))^{-1}=\exp\left(-\sum_{j=1}^{r}(1-e^{-2t_{j}})\right).$

Moreover, if $\xi\in\mathcal{V}_{\pi}$ is a vector in the highest restricted
weight space, then by (3.5):

$\pi(k_{\mathbb{C},N}^{+}(a))^{-1}\xi=\left(\prod_{j=1}^{r}e^{-\mu_{\pi,j}t_{j}}\right)\xi.$

It follows that (5.6) equals

$\int_{t_{1}>\cdots>t_{r}}\prod_{j=1}^{r}\exp\left(-2(1-e^{-2t_{j}})-2\mu_{\pi,j}t_{j}+2\rho_{\mathfrak{n},j}t_{j}\right)\cdot
w\left(\exp\sum_{j=1}^{r}t_{j}x_{j}\right)\,dt_{1}\cdots dt_{r}.$

Because of the superexponential decay as $t_{j}\to-\infty$, it is sufficient
to study the integral over $t_{1}>\cdots>t_{r}>0$. In this region, we can
estimate $w(a)\sim a^{2\rho_{\mathfrak{l}}}$, so the integral under
consideration becomes

$\int_{t_{1}>\cdots>t_{r}>0}\prod_{j=1}^{r}\exp\left(-2(1-e^{-2t_{j}})-2\mu_{\pi,j}t_{j}+2\rho_{\mathfrak{n},j}t_{j}+2\rho_{\mathfrak{l},j}t_{j}\right)\,dt_{1}\cdots
dt_{r}.$

This integral clearly converges if and only if
$\mu_{\pi,j}>\rho_{\mathfrak{n},j}+\rho_{\mathfrak{l},j}=\rho_{j}$ for all
$1\leq j\leq r$, which is precisely condition (3.4). Finally,
$T_{\pi,\eta}\xi\in L^{2}(G/N,\omega)$ holds for $\xi$ in the highest
restricted weight space if and only if it holds for all
$\xi\in\mathcal{V}_{\pi}$ by irreducibility. ∎

This shows that every $T_{\pi,\eta}$ extends to $\mathcal{H}_{\pi}\to
L^{2}(G/N,\omega)$. Because $\mathcal{H}_{\pi}$ is irreducible and unitary,
this extension must be unitary (up to a scalar). As shown in Section 5.3,
every unitary intertwining operator $\mathcal{H}_{\pi}\to L^{2}(G/N,\omega)$
must be of this form, so we have shown:

######  Corollary 5.9.

For every $\eta\in\mathcal{V}_{\pi}$ the operator $T_{\pi,\eta}$ extends to a
$G$-equivariant unitary (up to a scalar) operator

$T_{\pi,\eta}:\mathcal{H}_{\pi}\to L^{2}(G/N,\omega).$

Moreover, the map

$\bigoplus_{\pi}\mathcal{H}_{\pi}\otimes\overline{\mathcal{V}_{\pi}}\to
L^{2}(G/N,\omega),\quad\mathcal{H}_{\pi}\otimes\overline{\mathcal{V}_{\pi}}\ni
F\otimes\overline{\eta}\mapsto T_{\pi,\eta}F$

is an embedding onto the holomorphic discrete series contribution to the
Plancherel decomposition of $L^{2}(G/N,\omega)$.

###  5.5. Back to Whittaker vectors

Every unitary intertwining operator $T_{\pi,\eta}:\mathcal{H}_{\pi}\to
L^{2}(G/N,\omega)$ constructed in the previous section restricts to an
operator $\mathcal{H}_{\pi}^{\infty}\to C^{\infty}(G/N,\omega)$, so the
inequality in Corollary 5.4 is actually an equality. We now identify the
corresponding Whittaker vectors with $\mathcal{F}$-valued antiholomorphic
functions on $\mathcal{D}$.

As in [2, Lemma 2.4], we identify a Whittaker vector
$W\in\operatorname{Hom}_{N}(\mathcal{H}_{\pi}^{\infty},\mathcal{F})$ with an
antiholomorphic function
$\Pi:\mathcal{D}\to\mathcal{V}_{\pi}^{\vee}\otimes\mathcal{F}$ in the sense
that

$W(F)=\int_{\mathcal{D}}\Pi(z)\left(K_{\pi}(z,z)^{-1}F(z)\right)\,d^{*}z\qquad(F\in\mathcal{H}_{\pi}^{\infty}).$

Let $\Pi_{\pi,\eta}$ denote the function associated with the Whittaker vector
which arises from the intertwiner $T_{\pi,\eta}$ (in the sense of Section
5.1), i.e.

$T_{\pi,\eta}(F)(e)=\int_{\mathcal{D}}\Pi_{\pi,\eta}(z)\left(K_{\pi}(z,z)^{-1}F(z)\right)\,d^{*}z\qquad(F\in\mathcal{H}_{\pi}^{\infty}).$
(5.7)

######  Proposition 5.10.

As an element of
$\operatorname{Hom}(\mathcal{V}_{\pi},\mathcal{F})\simeq\mathcal{V}_{\pi}^{\vee}\otimes\mathcal{F}$,
the following identity holds, where $z=g\cdot o$:

$\Pi_{\pi,\eta}(z)=\Psi(e,z)^{*}=\omega(n_{\mathbb{C}}^{+}(g^{-1}))^{-1}\circ
A_{\eta}^{*}\circ\pi(k_{\mathbb{C},N}^{+}(g^{-1}))^{-1}\circ
j_{\pi}(g,o)^{-*}.$

###### Proof.

The first identity is immediately clear from (5.1), Corollary 5.6 and (5.7).
The second one is a special case of (5.3). ∎

## References

  * [1] Hongming Ding, _Weighted Laplace transforms and Bessel functions on Hermitian symmetric spaces_ , Trans. Amer. Math. Soc. 351 (1999), no. 10, 4205–4243.
  * [2] Jan Frahm, Gestur Ólafsson, and Bent Ørsted, _The holomorphic discrete series contribution to the generalized Whittaker Plancherel formula_ , to appear in J. Funct. Anal., available at arXiv:2203.14784.
  * [3] Adam Korányi and Joseph A. Wolf, _Generalized Cayley transformations of bounded symmetric domains_ , Amer. J. Math. 87 (1965), 899–939.
  * [4] by same author, _Realization of hermitian symmetric spaces as generalized half-planes_ , Ann. of Math. (2) 81 (1965), 265–288.
  * [5] Ottmar Loos, _Bounded symmetric domains and Jordan pairs_ , Univ. of California, Irvine, 1977.
  * [6] Calvin C. Moore, _Compactifications of symmetric spaces. II. The Cartan domains_ , Amer. J. Math. 86 (1964), 358–378.
  * [7] Karl-Hermann Neeb, _Holomorphy and convexity in Lie theory_ , De Gruyter Expositions in Mathematics, vol. 28, Walter de Gruyter & Co., Berlin, 2000.
  * [8] Gestur Ólafsson, _Analytic continuation in representation theory and harmonic analysis_ , Global analysis and harmonic analysis (Marseille-Luminy, 1999), Sémin. Congr., vol. 4, Soc. Math. France, Paris, 2000, pp. 201–233.
  * [9] Ichirô Satake, _Algebraic structures of symmetric domains_ , Kanô Memorial Lectures, vol. 4, Iwanami Shoten, Tokyo; Princeton University Press, Princeton, NJ, 1980.
  * [10] Henrik Schlichtkrull, _One-dimensional $K$-types in finite-dimensional representations of semisimple Lie groups: a generalization of Helgason’s theorem_, Math. Scand. 54 (1984), no. 2, 279–294. MR 757468
  * [11] Michèle Vergne and Hugo Rossi, _Analytic continuation of the holomorphic discrete series of a semi-simple Lie group_ , Acta Math. 136 (1976), no. 1-2, 1–59.