# Localization operators associated with the windowed Opdam–Cherednik
transform on modulation spaces

Anirudha Poria  Department of Mathematics, Bar-Ilan University, Ramat-Gan
5290002, Israel<EMAIL_ADDRESS>
###### Abstract.

In this paper, we study a class of pseudodifferential operators known as time-
frequency localization operators, which depend on a symbol $\varsigma$ and two
windows functions $g_{1}$ and $g_{2}$. We first present some basic properties
of the windowed Opdam–Cherednik transform. Then, we use modulation spaces
associated with the Opdam–Cherednik transform as appropriate classes for
symbols and windows, and study the boundedness and compactness of the
localization operators associated with the windowed Opdam–Cherednik transform
on modulation spaces. Finally, we show that these operators are in the
Schatten–von Neumann class.

###### Key words and phrases:

Opdam–Cherednik transform; windowed Opdam–Cherednik transform; modulation
spaces; localization operators; Schatten–von Neumann class; compact operators.

###### 2010 Mathematics Subject Classification:

Primary 47G30; Secondary 44A15, 42B35, 47B10.

Research supported by ERC Starting Grant No. 713927.

##  1\. Introduction

Time-frequency localization operators are a mathematical tool to define a
restriction of functions to a region in the time-frequency plane to extract
time-frequency features. The role of these operators to localize a signal
simultaneously in time and frequency domains, this can be seen as the
uncertainty principle. The localization operators were introduced and studied
by Daubechies [9, 10, 11], Ramanathan and Topiwala [30], and extensively
investigated in [16, 33, 35]. This class of operators occurs in various
branches of pure and applied mathematics and has been studied by many authors.
Localization operators are recognized as an important new mathematical tool
and have found many applications to time-frequency analysis, quantum
mechanics, the theory of differential equations, and signal processing (see
[7, 22, 23, 18, 30, 35]). They are also known as anti-Wick operators, wave
packets, Toeplitz operators, or Gabor multipliers (see [23, 5, 8, 16]). For a
detailed study of the theory of localization operators, we refer to the series
of papers of Wong [6, 34, 36, 37], and also the book of Wong [35].
Considerable attention has been devoted to study the localization operators to
new contexts. For instance, localization operators were investigated in [3]
for the wavelet transform associated to the Riemann–Liouville operator, in [4]
for the windowed Hankel transform and in [26] for the k-Hankel wavelet
transform. However, upto our knowledge, the localization operators have not
been studied for the Opdam–Cherednik transform on modulation spaces. In this
paper, we attempt to study the localization operators for the windowed
Opdam–Cherednik transform on modulation spaces associated with the
Opdam–Cherednik transform.

The motivation to study the localization operators for the windowed
Opdam–Cherednik transform on modulation spaces arises from the classical
results on these operators for the short-time Fourier transform. Since the
last decade modulation spaces have found to be very fruitful in various
current trends (e.g., pseudo-differential operators, partial differential
equations, etc..) of investigation and have been widely used in several fields
in analysis, physics and engineering. Localization operators have implications
in two main areas: quantum mechanics and signal analysis, and modulation
spaces are widely used in these areas. We hope that the study of these
operators on the modulation spaces makes a significant impact in these areas.
Another important motivation to study the Jacobi–Cherednik operators arises
from their relevance in the algebraic description of exactly solvable quantum
many-body systems of Calogero–Moser–Sutherland type (see [12, 20]) and they
provide a useful tool in the study of special functions with root systems (see
[13, 19]). These describe algebraically integrable systems in one dimension
and have gained considerable interest in mathematical physics. Other
motivation for the investigation of the Jacobi–Cherednik operator and the
Opdam–Cherednik transform is to generalize the previous subjects which are
bound with physics. For a more detailed discussion, we refer to [25].

Time-frequency localization operators were defined using the Schrödinger
representation and the short-time Fourier transform, which suggests studying
these operators as a part of time-frequency analysis. Modulation spaces were
used as the appropriate function spaces for understanding these operators, as
these spaces are associated to the short-time Fourier transform. However,
without using the short-time Fourier transform, we define the localization
operator using the windowed Opdam–Cherednik transform. As the harmonic
analysis associated with the Opdam–Cherednik transform has known remarkable
development (see [2, 24, 27, 28, 31]), the natural question to ask whether
there exists the equivalent of the theory of localization operators in the
framework of the Opdam–Cherednik transform. In this paper, we mainly concern
the windowed Opdam–Cherednik transform under the setting of the
Opdam–Cherednik transform. To measure the time-frequency concentration of
functions and distributions, we use norms and function spaces that are
associated with the Opdam–Cherednik transform. Here, we consider the
modulation spaces associated with the Opdam–Cherednik transform, as the
standard modulation spaces are not suited to this transform. Our main aim in
this paper is to expose and study the boundedness and compactness of the
localization operators associated with the windowed Opdam–Cherednik transform
under suitable conditions on symbols and windows and show that these operators
are in the Schatten–von Neumann class.

The paper is organized as follows. In Section 2, we recall some basic facts
about the Jacobi–Cherednik operator and we discuss the main results for the
Opdam–Cherednik transform. In Section 3, we discuss the modulation spaces
associated with the Opdam–Cherednik transform. In Section 4, we define and
study the windowed Opdam–Cherednik transform. Also, we present some basic
properties of the windowed Opdam–Cherednik transform and show that this
transform shares many properties with the short-time Fourier transform. In
particular, we prove Plancherel’s formula, orthogonality relation, and provide
a reconstruction formula. Finally, in Section 5, we use modulation spaces
associated with the Opdam–Cherednik transform as appropriate classes for
symbols and windows, and study the boundedness and compactness of the
localization operators associated with the windowed Opdam–Cherednik transform
on modulation spaces. Also, we show that these operators are in the
Schatten–von Neumann class.

##  2\. Harmonic analysis and the Opdam–Cherednik transform

In this section, we collect the necessary definitions and results from the
harmonic analysis related to the Opdam–Cherednik transform. The main
references for this section are [2, 24, 27, 28, 31]. However, we will use the
same notation as in [21, 29].

Let $T_{\alpha,\beta}$ denote the Jacobi–Cherednik differential–difference
operator (also called the Dunkl–Cherednik operator)

$T_{\alpha,\beta}f(x)=\frac{d}{dx}f(x)+\Big{[}(2\alpha+1)\coth
x+(2\beta+1)\tanh x\Big{]}\frac{f(x)-f(-x)}{2}-\rho f(-x),$

where $\alpha,\beta$ are two parameters satisfying
$\alpha\geq\beta\geq-\frac{1}{2}$ and $\alpha>-\frac{1}{2}$, and
$\rho=\alpha+\beta+1$. Let $\lambda\in{\mathbb{C}}$. The Opdam hypergeometric
functions $G^{\alpha,\beta}_{\lambda}$ on $\mathbb{R}$ are eigenfunctions
$T_{\alpha,\beta}G^{\alpha,\beta}_{\lambda}(x)=i\lambda
G^{\alpha,\beta}_{\lambda}(x)$ of $T_{\alpha,\beta}$ that are normalized such
that $G^{\alpha,\beta}_{\lambda}(0)=1$. The eigenfunction
$G^{\alpha,\beta}_{\lambda}$ is given by

$G^{\alpha,\beta}_{\lambda}(x)=\varphi^{\alpha,\beta}_{\lambda}(x)-\frac{1}{\rho-i\lambda}\frac{d}{dx}\varphi^{\alpha,\beta}_{\lambda}(x)=\varphi^{\alpha,\beta}_{\lambda}(x)+\frac{\rho+i\lambda}{4(\alpha+1)}\sinh
2x\;\varphi^{\alpha+1,\beta+1}_{\lambda}(x),$

where
$\varphi^{\alpha,\beta}_{\lambda}(x)={}_{2}F_{1}\left(\frac{\rho+i\lambda}{2},\frac{\rho-i\lambda}{2};\alpha+1;-\sinh^{2}x\right)$
is the hypergeometric function.

For every $\lambda\in{\mathbb{C}}$ and $x\in\mathbb{R}$, the eigenfunction
$G^{\alpha,\beta}_{\lambda}$ satisfy

$|G^{\alpha,\beta}_{\lambda}(x)|\leq
C\;e^{-\rho|x|}e^{|\text{Im}(\lambda)||x|},$

where $C$ is a positive constant. Since $\rho>0$, we have

$|G^{\alpha,\beta}_{\lambda}(x)|\leq C\;e^{|\text{Im}(\lambda)||x|}.$ (2.1)

Let us denote by $C_{c}(\mathbb{R})$ the space of continuous functions on
$\mathbb{R}$ with compact support. The Opdam–Cherednik transform is the
Fourier transform in the trigonometric Dunkl setting, and it is defined as
follows.

######  Definition 2.1.

Let $\alpha\geq\beta\geq-\frac{1}{2}$ with $\alpha>-\frac{1}{2}$. The
Opdam–Cherednik transform $\mathcal{H}_{\alpha,\beta}(f)$ of a function $f\in
C_{c}(\mathbb{R})$ is defined by

${\mathcal{H}}_{\alpha,\beta}(f)(\lambda)=\int_{\mathbb{R}}f(x)\;G^{\alpha,\beta}_{\lambda}(-x)\;A_{\alpha,\beta}(x)dx\quad\text{for
all }\lambda\in{\mathbb{C}},$

where $A_{\alpha,\beta}(x)=(\sinh|x|)^{2\alpha+1}(\cosh|x|)^{2\beta+1}$. The
inverse Opdam–Cherednik transform for a suitable function $g$ on $\mathbb{R}$
is given by

${\mathcal{H}}_{\alpha,\beta}^{-1}(g)(x)=\int_{\mathbb{R}}g(\lambda)\;G^{\alpha,\beta}_{\lambda}(x)\;d\sigma_{\alpha,\beta}(\lambda)\quad\text{for
all }x\in\mathbb{R},$

where

$d\sigma_{\alpha,\beta}(\lambda)=\left(1-\dfrac{\rho}{i\lambda}\right)\dfrac{d\lambda}{8\pi|C_{\alpha,\beta}(\lambda)|^{2}}$

and

$C_{\alpha,\beta}(\lambda)=\dfrac{2^{\rho-i\lambda}\Gamma(\alpha+1)\Gamma(i\lambda)}{\Gamma\left(\frac{\rho+i\lambda}{2}\right)\;\Gamma\left(\frac{\alpha-\beta+1+i\lambda}{2}\right)},\quad\lambda\in{\mathbb{C}}\setminus
i\mathbb{N}.$

The Plancherel formula is given by

$\int_{\mathbb{R}}|f(x)|^{2}A_{\alpha,\beta}(x)dx=\int_{\mathbb{R}}{\mathcal{H}}_{\alpha,\beta}(f)(\lambda)\overline{{\mathcal{H}}_{\alpha,\beta}(\check{f})(-\lambda)}\;d\sigma_{\alpha,\beta}(\lambda),$
(2.2)

where $\check{f}(x):=f(-x)$. Since the hypergeometric functions
$G^{\alpha,\beta}_{\lambda}$ satisfy
$G^{\alpha,\beta}_{\lambda}(tx)=G^{\alpha,\beta}_{\lambda t}(x)$, for every
$\lambda\in{\mathbb{C}}$ and $x,t\in\mathbb{R}$, using the definition of
$\mathcal{H}_{\alpha,\beta}$, we obtain
$\mathcal{H}_{\alpha,\beta}(\check{f})(-\lambda)=\mathcal{H}_{\alpha,\beta}(f)(\lambda)$.
Therefore, we can rewrite the Plancherel formula (2.2) as follows:

$\int_{\mathbb{R}}|f(x)|^{2}A_{\alpha,\beta}(x)dx=\int_{\mathbb{R}}\left|\mathcal{H}_{\alpha,\beta}(f)(\lambda)\right|^{2}d\sigma_{\alpha,\beta}(\lambda).$
(2.3)

Let $L^{p}(\mathbb{R},A_{\alpha,\beta})$ (resp.
$L^{p}(\mathbb{R},\sigma_{\alpha,\beta})$), $p\in[1,\infty]$, denote the
$L^{p}$-spaces corresponding to the measure $A_{\alpha,\beta}(x)dx$ (resp.
$d|\sigma_{\alpha,\beta}|(x)$). The Schwartz space
${\mathcal{S}}_{\alpha,\beta}(\mathbb{R})=(\cosh
x)^{-\rho}{\mathcal{S}}(\mathbb{R})$ is defined as the space of all
differentiable functions $f$ such that

$\sup_{x\in\mathbb{R}}\;(1+|x|)^{m}e^{\rho|x|}\left|\frac{d^{n}}{dx^{n}}f(x)\right|<\infty,$

for all $m,n\in{\mathbb{N}}_{0}={\mathbb{N}}\cup\\{0\\}$, equipped with the
obvious seminorms. The Opdam–Cherednik transform
${\mathcal{H}}_{\alpha,\beta}$ and its inverse
${\mathcal{H}}_{\alpha,\beta}^{-1}$ are topological isomorphisms between the
space ${\mathcal{S}}_{\alpha,\beta}(\mathbb{R})$ and the space
${\mathcal{S}}(\mathbb{R})$ (see [31], Theorem 4.1).

The generalized translation operator associated with the Opdam–Cherednik
transform is defined by [1]

$\tau_{x}^{(\alpha,\beta)}f(y)=\int_{\mathbb{R}}f(z)\;{d\mu}_{\;x,y}^{(\alpha,\beta)}(z),$
(2.4)

where ${d\mu}_{\;x,y}^{(\alpha,\beta)}$ is given by

${d\mu}_{\;x,y}^{(\alpha,\beta)}(z)=\begin{cases}\mathcal{K}_{\alpha,\beta}(x,y,z)\;A_{\alpha,\beta}(z)\;dz&\text{if}\;\;xy\neq
0\\\ d\delta_{x}(z)&\text{if}\;\;y=0\\\
d\delta_{y}(z)&\text{if}\;\;x=0\end{cases}$ (2.5)

and

$\displaystyle\mathcal{K}_{\alpha,\beta}{}$
$\displaystyle(x,y,z)=M_{\alpha,\beta}|\sinh x\cdot\sinh y\cdot\sinh
z|^{-2\alpha}\int_{0}^{\pi}g(x,y,z,\chi)_{+}^{\alpha-\beta-1}$
$\displaystyle\times\left[1-\sigma^{\chi}_{x,y,z}+\sigma^{\chi}_{x,z,y}+\sigma^{\chi}_{z,y,x}+\frac{\rho}{\beta+\frac{1}{2}}\coth
x\cdot\coth y\cdot\coth
z(\sin\chi)^{2}\right]\times(\sin\chi)^{2\beta}\;d\chi,$

where

$M_{\alpha,\beta}=\frac{\Gamma(\alpha+1)}{\sqrt{\pi}\Gamma(\alpha-\beta)\Gamma\left(\beta+\frac{1}{2}\right)},$

if $x,y,z\in\mathbb{R}\setminus\\{0\\}$ satisfy the triangular inequality
$||x|-|y||<|z|<|x|+|y|$, and $\mathcal{K}_{\alpha,\beta}(x,y,z)=0$ otherwise.
Here

$\sigma^{\chi}_{x,y,z}=\begin{cases}\frac{\cosh x\cdot\cosh y-\cosh
z\cdot\cos\chi}{\sinh x\cdot\sinh y}&\text{if}\;\;xy\neq 0\\\
0&\text{if}\;\;xy=0\end{cases}\quad\text{for}\;x,y,z\in\mathbb{R},\;\chi\in[0,\pi],$

$g(x,y,z,\chi)=1-\cosh^{2}x-\cosh^{2}y-\cosh^{2}z+2\cosh x\cdot\cosh
y\cdot\cosh z\cdot\cos\chi$, and

$g_{+}=\begin{cases}g&\text{if}\;\;g>0\\\ 0&\text{if}\;\;g\leq 0.\end{cases}$

The kernel $\mathcal{K}_{\alpha,\beta}(x,y,z)$ satisfies the following
symmetry properties:

$\mathcal{K}_{\alpha,\beta}(x,y,z)=\mathcal{K}_{\alpha,\beta}(y,x,z)$,
$\mathcal{K}_{\alpha,\beta}(x,y,z)=\mathcal{K}_{\alpha,\beta}(-z,y,-x)$,
$\mathcal{K}_{\alpha,\beta}(x,y,z)=\mathcal{K}_{\alpha,\beta}(x,-z,-y)$.

Next, we recall some basic properties of $\tau_{x}^{(\alpha,\beta)}$ from [1].
For every $x,y\in\mathbb{R}$, we have

$\tau_{x}^{(\alpha,\beta)}f(y)=\tau_{y}^{(\alpha,\beta)}f(x),$ (2.6)

and

${\mathcal{H}}_{\alpha,\beta}(\tau_{x}^{(\alpha,\beta)}f)(\lambda)=G_{\lambda}^{\alpha,\beta}(x)\;{\mathcal{H}}_{\alpha,\beta}(f)(\lambda),$
(2.7)

for $f\in C_{c}(\mathbb{R})$.

If $f\in L^{1}(\mathbb{R},A_{\alpha,\beta})$, then

$\displaystyle\int_{\mathbb{R}}\tau_{x}^{(\alpha,\beta)}f(y)\;A_{\alpha,\beta}(y)\;dy$
$\displaystyle=$
$\displaystyle\int_{\mathbb{R}}\left(\int_{\mathbb{R}}f(z)\;{d\mu}_{\;x,y}^{(\alpha,\beta)}(z)\right)\;A_{\alpha,\beta}(y)\;dy$
(2.8) $\displaystyle=$
$\displaystyle\int_{\mathbb{R}}\left(\int_{\mathbb{R}}f(z)\;\mathcal{K}_{\alpha,\beta}(x,y,z)\;A_{\alpha,\beta}(z)\;dz\right)\;A_{\alpha,\beta}(y)\;dy$
$\displaystyle=$
$\displaystyle\int_{\mathbb{R}}f(z)\left(\int_{\mathbb{R}}\;\mathcal{K}_{\alpha,\beta}(x,-z,-y)\;A_{\alpha,\beta}(y)\;dy\right)\;A_{\alpha,\beta}(z)\;dz$
$\displaystyle=$
$\displaystyle\int_{\mathbb{R}}f(z)\left(\int_{\mathbb{R}}\;\mathcal{K}_{\alpha,\beta}(x,-z,y)\;A_{\alpha,\beta}(y)\;dy\right)\;A_{\alpha,\beta}(z)\;dz$
$\displaystyle=$
$\displaystyle\int_{\mathbb{R}}f(z)\left(\int_{\mathbb{R}}\;{d\mu}_{\;x,-z}^{(\alpha,\beta)}(y)\right)\;A_{\alpha,\beta}(z)\;dz$
$\displaystyle=$
$\displaystyle\int_{\mathbb{R}}f(y)\;A_{\alpha,\beta}(y)\;dy\qquad\qquad\left(\text{since}\int_{\mathbb{R}}\;{d\mu}_{\;x,-z}^{(\alpha,\beta)}(y)=1\right).\qquad\qquad$

For every $f\in L^{p}(\mathbb{R},A_{\alpha,\beta})$ and every
$x\in\mathbb{R}$, the function $\tau_{x}^{(\alpha,\beta)}f$ belongs to the
space $L^{p}(\mathbb{R},A_{\alpha,\beta})$ and

$\left\|\tau_{x}^{(\alpha,\beta)}f\right\|_{L^{p}(\mathbb{R},A_{\alpha,\beta})}\leq
C_{\alpha,\beta}\left\|f\right\|_{L^{p}(\mathbb{R},A_{\alpha,\beta})},$ (2.9)

where $C_{\alpha,\beta}$ is a positive constant.

The convolution product associated with the Opdam–Cherednik transform is
defined for two suitable functions $f$ and $g$ by [1]

$(f*_{\alpha,\beta}g)(x)=\int_{\mathbb{R}}\tau_{x}^{(\alpha,\beta)}f(-y)\;g(y)\;A_{\alpha,\beta}(y)\;dy$

and

${\mathcal{H}}_{\alpha,\beta}(f*_{\alpha,\beta}g)={\mathcal{H}}_{\alpha,\beta}(f)\;{\mathcal{H}}_{\alpha,\beta}(g).$
(2.10)

##  3\. Modulation spaces associated with the Opdam–Cherednik transform

The modulation spaces were introduced by Feichtinger [14, 15], by imposing
integrability conditions on the short-time Fourier transform (STFT) of
tempered distributions. More specifically, for $x,w\in\mathbb{R}$, let $M_{w}$
and $T_{x}$ denote the operators of modulation and translation. Then, the STFT
of a function $f$ with respect to a window function
$g\in{\mathcal{S}}(\mathbb{R})$ is defined by

$V_{g}f(x,w)=\langle
f,M_{w}T_{x}g\rangle=\int_{\mathbb{R}}f(t)\overline{g(t-x)}e^{-2\pi
iwt}dt,\quad(x,y)\in\mathbb{R}^{2}.$

Here we are interested in modulation spaces with respect to measure
$A_{\alpha,\beta}(x)dx$.

######  Definition 3.1.

Fix a non-zero window $g\in\mathcal{S}(\mathbb{R})$, and $1\leq
p,q\leq\infty$. Then the modulation space
$M^{p,q}(\mathbb{R},A_{\alpha,\beta})$ consists of all tempered distributions
$f\in\mathcal{S^{\prime}}(\mathbb{R})$ such that $V_{g}f\in
L^{p,q}(\mathbb{R}^{2},A_{\alpha,\beta})$. The norm on
$M^{p,q}(\mathbb{R},A_{\alpha,\beta})$ is

$\displaystyle\|f\|_{M^{p,q}(\mathbb{R},A_{\alpha,\beta})}$ $\displaystyle=$
$\displaystyle\|V_{g}f\|_{L^{p,q}(\mathbb{R}^{2},A_{\alpha,\beta})}$
$\displaystyle=$
$\displaystyle\bigg{(}\int_{\mathbb{R}}\bigg{(}\int_{\mathbb{R}}|V_{g}f(x,w)|^{p}A_{\alpha,\beta}(x)dx\bigg{)}^{q/p}A_{\alpha,\beta}(w)dw\bigg{)}^{1/q}<\infty,$

with the usual adjustments if $p$ or $q$ is infinite. If $p=q$, then we write
$M^{p}(\mathbb{R},A_{\alpha,\beta})$ instead of
$M^{p,p}(\mathbb{R},A_{\alpha,\beta})$. Also, we denote by
$M^{p}(\mathbb{R},\sigma_{\alpha,\beta})$ the modulation space corresponding
to the measure $d|\sigma_{\alpha,\beta}|(x)$ and $M^{p}(\mathbb{R})$ the
modulation space corresponding to the Lebesgue measure $dx$.

The definition of $M^{p,q}(\mathbb{R},A_{\alpha,\beta})$ is independent of the
choice of $g$ in the sense that each different choice of $g$ defines an
equivalent norm on $M^{p,q}(\mathbb{R},A_{\alpha,\beta})$. Each modulation
space is a Banach space. For $p=q=2$, we have that
$M^{2}(\mathbb{R},A_{\alpha,\beta})=L^{2}(\mathbb{R},A_{\alpha,\beta}).$ For
other $p=q$, the space $M^{p}(\mathbb{R},A_{\alpha,\beta})$ is not
$L^{p}(\mathbb{R},A_{\alpha,\beta})$. In fact for $p=q>2$, the space
$M^{p}(\mathbb{R},A_{\alpha,\beta})$ is a superset of
$L^{2}(\mathbb{R},A_{\alpha,\beta})$. Here, we collect some basic properties
and inclusion relations of modulation spaces with respect to
$A_{\alpha,\beta}$. We define the space of special windows
$\mathcal{S}_{\mathcal{C}}(\mathbb{R})$ by

$\displaystyle\mathcal{S}_{\mathcal{C}}\left(\mathbb{R}\right)=\left\\{f\in
L^{2}(\mathbb{R}):\;f=V_{g}^{*}F=\iint_{\mathbb{R}^{2}}F(x,w)\;M_{w}T_{x}g\;dx\;dw,\right.$
$\displaystyle\left.\qquad\qquad\qquad\qquad\qquad\text{where}\;F\in
L^{\infty}(\mathbb{R}^{2})\;\text{and}\;\hbox{supp\,}F\;\text{is
compact}\right\\}.$

Then
$\mathcal{S}_{\mathcal{C}}\left(\mathbb{R}\right)\subseteq\mathcal{S}\left(\mathbb{R}\right)$
and $\mathcal{S}_{\mathcal{C}}\left(\mathbb{R}\right)$ is dense in
$M^{1}(\mathbb{R},A_{\alpha,\beta})$ (see [18], Lemma 11.4.1). Let $B$ be a
Banach space of tempered distributions with the following properties: (1) $B$
is invariant under time-frequency shifts, and
$\left\|T_{x}M_{w}f\right\|_{B}\leq C\|f\|_{B}$ for all $f\in B$, (2)
$M^{1}(\mathbb{R},A_{\alpha,\beta})\cap B\neq\\{0\\}$. Then
$M^{1}(\mathbb{R},A_{\alpha,\beta})$ is embedded in $B$ (see [18], Theorem
12.1.9). Also, $M^{p}(\mathbb{R},A_{\alpha,\beta})$ is invariant under time-
frequency shifts and
$\left\|T_{x}M_{w}f\right\|_{M^{p}(\mathbb{R},A_{\alpha,\beta})}\leq
C\|f\|_{M^{p}(\mathbb{R},A_{\alpha,\beta})}$. Since
$\mathcal{S}_{\mathcal{C}}\left(\mathbb{R}\right)\subseteq
M^{1}(\mathbb{R},A_{\alpha,\beta})\cap M^{p}(\mathbb{R},A_{\alpha,\beta})$,
using Corollary 12.1.10 from [18], we obtain the following inclusions

$\mathcal{S}(\mathbb{R})\subset M^{1}(\mathbb{R},A_{\alpha,\beta})\subset
M^{2}(\mathbb{R},A_{\alpha,\beta})=L^{2}(\mathbb{R},A_{\alpha,\beta})\subset
M^{\infty}(\mathbb{R},A_{\alpha,\beta})\subset\mathcal{S^{\prime}}(\mathbb{R}).$

In particular, we have $M^{p}(\mathbb{R},A_{\alpha,\beta})\hookrightarrow
L^{p}(\mathbb{R},A_{\alpha,\beta})$ for $1\leq p\leq 2$, and
$L^{p}(\mathbb{R},A_{\alpha,\beta})\hookrightarrow
M^{p}(\mathbb{R},A_{\alpha,\beta})$ for $2\leq p\leq\infty$. Furthermore, the
dual of a modulation space is also a modulation space, if $p<\infty$,
$q<\infty$,
$(M^{p,q}(\mathbb{R},A_{\alpha,\beta}))^{{}^{\prime}}=M^{p^{\prime},q^{\prime}}(\mathbb{R},A_{\alpha,\beta})$,
where $p^{\prime},\;q^{\prime}$ denote the dual exponents of $p$ and $q$,
respectively. We refer to Gröchenig’s book [18] for further properties and
uses of modulation spaces.

##  4\. The windowed Opdam–Cherednik transform

Let $g\in L^{2}(\mathbb{R},A_{\alpha,\beta})$ and $\xi\in\mathbb{R}$, the
modulation operator of $g$ associated with the Opdam–Cherednik transform is
defined by

${\mathcal{M}}^{(\alpha,\beta)}_{\xi}g={\mathcal{H}}^{-1}_{\alpha,\beta}\left(\sqrt{\tau_{\xi}^{(\alpha,\beta)}|{\mathcal{H}}_{\alpha,\beta}(g)|^{2}}\right).$
(4.1)

Then, for every $g\in L^{2}(\mathbb{R},A_{\alpha,\beta})$ and
$\xi\in\mathbb{R}$, by using the Plancherel formula (2.2) and the translation
invariance of the Plancherel measure $d\sigma_{\alpha,\beta}$, we obtain

$\left\|{\mathcal{M}}^{(\alpha,\beta)}_{\xi}g\right\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}=\left\|g\right\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}.$
(4.2)

Now, for a non-zero window function $g\in L^{2}(\mathbb{R},A_{\alpha,\beta})$
and $(x,\xi)\in\mathbb{R}^{2}$, we consider the function
$g_{x,\xi}^{(\alpha,\beta)}$ defined by

$g_{x,\xi}^{(\alpha,\beta)}=\tau_{x}^{(\alpha,\beta)}{\mathcal{M}}^{(\alpha,\beta)}_{\xi}g.$
(4.3)

For any function $f\in L^{2}(\mathbb{R},A_{\alpha,\beta})$, we define the
windowed Opdam–Cherednik transform by

${\mathcal{W}}^{(\alpha,\beta)}_{g}(f)(x,\xi)=\int_{\mathbb{R}}f(s)\;\overline{g_{x,\xi}^{(\alpha,\beta)}(-s)}\;A_{\alpha,\beta}(s)\;ds,\quad(x,\xi)\in\mathbb{R}^{2},$
(4.4)

which can be also written in the form

${\mathcal{W}}^{(\alpha,\beta)}_{g}(f)(x,\xi)=\left(f*_{\alpha,\beta}\overline{{\mathcal{M}}^{(\alpha,\beta)}_{\xi}g}\right)(x).$
(4.5)

We define the measure $A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta}$ on
$\mathbb{R}^{2}$ by

$d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi)=A_{\alpha,\beta}(x)dx\;d\sigma_{\alpha,\beta}(\xi).$
(4.6)

The windowed Opdam–Cherednik transform satisfies the following properties.

######  Proposition 4.1.

Let $g\in L^{2}(\mathbb{R},A_{\alpha,\beta})$ be a non-zero window function.
Then we have

$(1)$ $($Plancherel’s formula$)$ For every $f\in
L^{2}(\mathbb{R},A_{\alpha,\beta})$,

$\left\|{\mathcal{W}}^{(\alpha,\beta)}_{g}(f)\right\|_{L^{2}(\mathbb{R}^{2},\;A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}=\|f\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\;\|g\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}.$
(4.7)

$(2)$ $($Orthogonality relation$)$ For every $f,h\in
L^{2}(\mathbb{R},A_{\alpha,\beta})$, we have

$\iint_{\mathbb{R}^{2}}{\mathcal{W}}^{(\alpha,\beta)}_{g}(f)(x,\xi)\;\overline{{\mathcal{W}}^{(\alpha,\beta)}_{g}(h)(x,\xi)}\;d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi)=\|g\|^{2}_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\int_{\mathbb{R}}f(s)\overline{h(s)}\;A_{\alpha,\beta}(s)\;ds.$
(4.8)

$(3)$ $($Reproducing kernel Hilbert space$)$ The space
${\mathcal{W}}^{(\alpha,\beta)}_{g}(L^{2}(\mathbb{R},A_{\alpha,\beta}))$ is a
reproducing kernel Hilbert space in
$L^{2}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$ with
kernel function $K_{g}$ defined by

$\displaystyle K_{g}((x^{\prime},\xi^{\prime});(x,\xi))$ $\displaystyle=$
$\displaystyle\frac{1}{\|g\|^{2}_{L^{2}(\mathbb{R},A_{\alpha,\beta})}}\;\left(g_{x,\xi}^{(\alpha,\beta)}(-\;\cdot)*_{\alpha,\beta}\overline{{\mathcal{M}}^{(\alpha,\beta)}_{\xi^{\prime}}g}\right)(x^{\prime})$
(4.9) $\displaystyle=$
$\displaystyle\frac{1}{\|g\|^{2}_{L^{2}(\mathbb{R},A_{\alpha,\beta})}}\;{\mathcal{W}}^{(\alpha,\beta)}_{g}\left(g_{x,\xi}^{(\alpha,\beta)}(-\;\cdot)\right)(x^{\prime},\xi^{\prime}).$

Furthermore, the kernel is pointwise bounded

$\left|K_{g}((x^{\prime},\xi^{\prime});(x,\xi))\right|\leq
C_{\alpha,\beta},\quad\text{for
all}\;(x,\xi);(x^{\prime},\xi^{\prime})\in\mathbb{R}^{2},$ (4.10)

where $C_{\alpha,\beta}$ is a positive constant.

###### Proof.

(1) Using Plancherel’s formula (2.3), and relations (2.10) and (4.5), we
obtain

$\displaystyle\iint_{\mathbb{R}^{2}}\left|{\mathcal{W}}^{(\alpha,\beta)}_{g}(f)(x,\xi)\right|^{2}\;d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi)$
$\displaystyle=\iint_{\mathbb{R}^{2}}\left|\left(f*_{\alpha,\beta}\overline{{\mathcal{M}}^{(\alpha,\beta)}_{\xi}g}\right)(x)\right|^{2}\;A_{\alpha,\beta}(x)dx\;d\sigma_{\alpha,\beta}(\xi)$
$\displaystyle=\iint_{\mathbb{R}^{2}}\left|{\mathcal{H}}_{\alpha,\beta}\left(f*_{\alpha,\beta}\overline{{\mathcal{M}}^{(\alpha,\beta)}_{\xi}g}\right)(\lambda)\right|^{2}\;d\sigma_{\alpha,\beta}(\lambda)\;d\sigma_{\alpha,\beta}(\xi)$
$\displaystyle=\iint_{\mathbb{R}^{2}}\left|{\mathcal{H}}_{\alpha,\beta}(f)(\lambda)\right|^{2}\;\left|{\mathcal{H}}_{\alpha,\beta}\left(\overline{{\mathcal{M}}^{(\alpha,\beta)}_{\xi}g}\right)(\lambda)\right|^{2}\;d\sigma_{\alpha,\beta}(\lambda)\;d\sigma_{\alpha,\beta}(\xi)$
$\displaystyle=\iint_{\mathbb{R}^{2}}\left|{\mathcal{H}}_{\alpha,\beta}(f)(\lambda)\right|^{2}\;\tau^{(\alpha,\beta)}_{\xi}\left|{\mathcal{H}}_{\alpha,\beta}(g)\right|^{2}(\lambda)\;d\sigma_{\alpha,\beta}(\lambda)\;d\sigma_{\alpha,\beta}(\xi)$
$\displaystyle=\int_{\mathbb{R}}\left|{\mathcal{H}}_{\alpha,\beta}(f)(\lambda)\right|^{2}\;\int_{\mathbb{R}}\tau^{(\alpha,\beta)}_{\lambda}\left|{\mathcal{H}}_{\alpha,\beta}(g)\right|^{2}(\xi)\;d\sigma_{\alpha,\beta}(\xi)\;d\sigma_{\alpha,\beta}(\lambda)$
$\displaystyle=\int_{\mathbb{R}}\left|{\mathcal{H}}_{\alpha,\beta}(f)(\lambda)\right|^{2}d\sigma_{\alpha,\beta}(\lambda)\;\int_{\mathbb{R}}\left|{\mathcal{H}}_{\alpha,\beta}(g)(\xi)\right|^{2}d\sigma_{\alpha,\beta}(\xi)$
$\displaystyle=\|f\|^{2}_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\;\|g\|^{2}_{L^{2}(\mathbb{R},A_{\alpha,\beta})}.$

(2) Using the polarization identity and Plancherel’s formula (4.7), we obtain
the result.

(3) From relations (4.4) and (4.8), we obtain that

$\displaystyle{\mathcal{W}}^{(\alpha,\beta)}_{g}(f)(x,\xi)$
$\displaystyle=\frac{1}{\|g\|^{2}_{L^{2}(\mathbb{R},A_{\alpha,\beta})}}\iint_{\mathbb{R}^{2}}{\mathcal{W}}^{(\alpha,\beta)}_{g}(f)(x^{\prime},\xi^{\prime})\;\overline{{\mathcal{W}}^{(\alpha,\beta)}_{g}\left(g_{x,\xi}^{(\alpha,\beta)}(-\;\cdot)\right)(x^{\prime},\xi^{\prime})}\;d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x^{\prime},\xi^{\prime})$
$\displaystyle=\left\langle{\mathcal{W}}^{(\alpha,\beta)}_{g}(f),\;K_{g}((\cdot,\cdot);(x,\xi))\right\rangle_{L^{2}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})},$

where
$\langle\cdot,\cdot\rangle_{L^{2}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}$
denotes the inner product of
$L^{2}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$ and
$K_{g}$ is the function on $\mathbb{R}^{2}$ defined by

$\displaystyle K_{g}((x^{\prime},\xi^{\prime});(x,\xi))$ $\displaystyle=$
$\displaystyle\frac{1}{\|g\|^{2}_{L^{2}(\mathbb{R},A_{\alpha,\beta})}}\;{\mathcal{W}}^{(\alpha,\beta)}_{g}\left(g_{x,\xi}^{(\alpha,\beta)}(-\;\cdot)\right)(x^{\prime},\xi^{\prime})$
$\displaystyle=$
$\displaystyle\frac{1}{\|g\|^{2}_{L^{2}(\mathbb{R},A_{\alpha,\beta})}}\;\left(g_{x,\xi}^{(\alpha,\beta)}(-\;\cdot)*_{\alpha,\beta}\overline{{\mathcal{M}}^{(\alpha,\beta)}_{\xi^{\prime}}g}\right)(x^{\prime}).$

Also, for every $(x,\xi)\in\mathbb{R}^{2}$, by using Plancherel’s formula
(4.7), we obtain

$\left\|K_{g}((\cdot,\cdot);(x,\xi))\right\|_{L^{2}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\leq
C_{\alpha,\beta}.$

Finally, by the Cauchy–Schwarz inequality, for every
$(x,\xi);(x^{\prime},\xi^{\prime})\in\mathbb{R}^{2}$ we have

$\displaystyle|K_{g}((x^{\prime},\xi^{\prime});(x,\xi))|$ $\displaystyle\leq$
$\displaystyle\frac{1}{\|g\|^{2}_{L^{2}(\mathbb{R},A_{\alpha,\beta})}}\int_{\mathbb{R}}\left|g_{x,\xi}^{(\alpha,\beta)}(-s)\right|\;\left|g_{x^{\prime},\xi^{\prime}}^{(\alpha,\beta)}(-s)\right|\;A_{\alpha,\beta}(s)\;ds$
$\displaystyle\leq$
$\displaystyle\frac{1}{\|g\|^{2}_{L^{2}(\mathbb{R},A_{\alpha,\beta})}}\;\left\|g_{x,\xi}^{(\alpha,\beta)}\right\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\left\|g_{x^{\prime},\xi^{\prime}}^{(\alpha,\beta)}\right\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}=C_{\alpha,\beta}.$

This proves that the kernel $K_{g}\in
L^{2}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$ and is
bounded. ∎

######  Theorem 4.2.

Let $g\in L^{2}(\mathbb{R},A_{\alpha,\beta})$ be a non-zero window function.
Then for every $f\in L^{2}(\mathbb{R},A_{\alpha,\beta})$, we have

$(1)$

$\left\|{\mathcal{W}}^{(\alpha,\beta)}_{g}(f)\right\|_{L^{\infty}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\leq
C_{\alpha,\beta}\|f\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\;\|g\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}.$
(4.11)

$(2)$ The function ${\mathcal{W}}^{(\alpha,\beta)}_{g}(f)\in
L^{p}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$,
$p\in[2,\infty)$ and

$\left\|{\mathcal{W}}^{(\alpha,\beta)}_{g}(f)\right\|_{L^{p}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\leq
C_{\alpha,\beta}\|f\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\;\|g\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}.$
(4.12)

###### Proof.

(1) Using the Cauchy–Schwarz inequality, and relations (4.2) and (4.3), we get

$\left|{\mathcal{W}}^{(\alpha,\beta)}_{g}(f)(x,\xi)\right|\leq\|f\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\;\left\|g_{x,\xi}^{(\alpha,\beta)}\right\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\leq
C_{\alpha,\beta}\|f\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\;\|g\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}.$

Therefore,

$\left\|{\mathcal{W}}^{(\alpha,\beta)}_{g}(f)\right\|_{L^{\infty}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\leq
C_{\alpha,\beta}\|f\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\;\|g\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}.$

(2) Using Plancherel’s formula (4.7), relation (4.11) and the Riesz–Thorin
interpolation theorem (see [32]), we obtain the result. ∎

######  Theorem 4.3 (Reconstruction formula).

Let $g\in L^{2}(\mathbb{R},A_{\alpha,\beta})$ be a non-zero positive window
function. Then for every $f\in L^{2}(\mathbb{R},A_{\alpha,\beta})$, we have

$f(\cdot)=\frac{1}{\|g\|^{2}_{L^{2}(\mathbb{R},A_{\alpha,\beta})}}\iint_{\mathbb{R}^{2}}{\mathcal{W}}^{(\alpha,\beta)}_{g}(f)(x,\xi)\;g_{x,\xi}^{(\alpha,\beta)}(-\;\cdot)\;d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi),$

weakly in $L^{2}(\mathbb{R},A_{\alpha,\beta})$.

###### Proof.

Using relation (4.8) and applying Fubini’s theorem, for every $h\in
L^{2}(\mathbb{R},A_{\alpha,\beta})$ we obtain

$\displaystyle\int_{\mathbb{R}}f(s)\overline{h(s)}\;A_{\alpha,\beta}(s)\;ds$
$\displaystyle=$
$\displaystyle\frac{1}{\|g\|^{2}_{L^{2}(\mathbb{R},A_{\alpha,\beta})}}\iint_{\mathbb{R}^{2}}{\mathcal{W}}^{(\alpha,\beta)}_{g}(f)(x,\xi)\;\overline{{\mathcal{W}}^{(\alpha,\beta)}_{g}(h)(x,\xi)}\;d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi)$
$\displaystyle=$
$\displaystyle\frac{1}{\|g\|^{2}_{L^{2}(\mathbb{R},A_{\alpha,\beta})}}\int_{\mathbb{R}}\left(\iint_{\mathbb{R}^{2}}{\mathcal{W}}^{(\alpha,\beta)}_{g}(f)(x,\xi)\;g_{x,\xi}^{(\alpha,\beta)}(-s)\;d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi)\right)\;\overline{h(s)}\;A_{\alpha,\beta}(s)\;ds,$

which completes the proof. ∎

##  5\. Localization operators for the windowed Opdam–Cherednik transform

In this section, we define the localization operators for the windowed
Opdam–Cherednik transform and we show that these operators are bounded. Also,
we prove that localization operators are compact and in the Schatten–von
Neumann class.

######  Definition 5.1.

Let $\varsigma\in
L^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})\cup
L^{\infty}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$. The
localization operator for the windowed Opdam–Cherednik transform associated
with the symbol $\varsigma$ and two window functions $g_{1}$ and $g_{2}$, is
denoted by $\mathfrak{L}_{g_{1},g_{2}}(\varsigma)$, and defined on
$L^{2}(\mathbb{R},A_{\alpha,\beta})$, by

$\mathfrak{L}_{g_{1},g_{2}}(\varsigma)(f)(y)=\iint_{\mathbb{R}^{2}}\varsigma(x,\xi)\;{\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)(x,\xi)\;{g_{2}}_{x,\xi}^{(\alpha,\beta)}(-y)\;d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi),\quad
y\in\mathbb{R}.$ (5.1)

Also, it is useful to rewrite the definition of
$\mathfrak{L}_{g_{1},g_{2}}(\varsigma)$ in a weak sense as, for every $f,h\in
L^{2}(\mathbb{R},A_{\alpha,\beta})$

$\left\langle\mathfrak{L}_{g_{1},g_{2}}(\varsigma)(f),h\right\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}=\iint_{\mathbb{R}^{2}}\varsigma(x,\xi)\;{\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)(x,\xi)\;\overline{{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(h)(x,\xi)}\;d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi).$
(5.2)

We denote by $\mathcal{B}(L^{p}(\mathbb{R},A_{\alpha,\beta}))$, $1\leq
p\leq\infty$, the space of all bounded linear operators from
$L^{p}(\mathbb{R},A_{\alpha,\beta})$ into itself. In particular,
$\mathcal{B}(L^{2}(\mathbb{R},A_{\alpha,\beta}))$ denote the C∗-algebra of
bounded linear operator $\mathcal{A}$ from
$L^{2}(\mathbb{R},A_{\alpha,\beta})$ into itself, equipped with the norm

$\|\mathcal{A}\|_{\mathcal{B}(L^{2}(\mathbb{R},A_{\alpha,\beta}))}=\sup_{\|f\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\leq
1}\|\mathcal{A}(f)\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}.$

Next, we define the Schatten–von Neumann class $S_{p}$. For a compact operator
$\mathcal{A}\in\mathcal{B}(L^{2}(\mathbb{R},A_{\alpha,\beta}))$, the
eigenvalues of the positive self-adjoint operator
$|\mathcal{A}|=\sqrt{\mathcal{A}^{*}\mathcal{A}}$ are called the singular
values of $\mathcal{A}$ and denoted by
$\\{s_{n}(\mathcal{A})\\}_{n\in\mathbb{N}}$. For $1\leq p<\infty$, the
Schatten–von Neumann class $S_{p}$ is defined to be the space of all compact
operators whose singular values lie in $\ell^{p}$. $S_{p}$ is equipped with
the norm

$\|\mathcal{A}\|_{S_{p}}=\left(\sum_{n=1}^{\infty}(s_{n}(\mathcal{A}))^{p}\right)^{1/p}.$

For $p=\infty$, the Schatten–von Neumann class $S_{\infty}$ is the class of
all compact operators with the norm
$\|\mathcal{A}\|_{S_{\infty}}:=\|\mathcal{A}\|_{\mathcal{B}(L^{2}(\mathbb{R},A_{\alpha,\beta}))}$.
In particular, for $p=1$, we define the trace of an operator $\mathcal{A}$ in
$S_{1}$ by

$tr(\mathcal{A})=\sum_{n=1}^{\infty}\langle\mathcal{A}v_{n},v_{n}\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})},$

where $\\{v_{n}\\}_{n}$ is any orthonormal basis of
$L^{2}(\mathbb{R},A_{\alpha,\beta})$. Moreover, if $\mathcal{A}$ is positive,
then

$tr(\mathcal{A})=\|\mathcal{A}\|_{S_{1}}.$

A compact operator $\mathcal{A}$ on the Hilbert space
$L^{2}(\mathbb{R},A_{\alpha,\beta})$ is called the Hilbert–Schmidt operator,
if the positive operator $\mathcal{A}^{*}\mathcal{A}$ is in the trace class
$S_{1}$. Then for any orthonormal basis $\\{v_{n}\\}_{n}$ of
$L^{2}(\mathbb{R},A_{\alpha,\beta})$, we have

$\|\mathcal{A}\|_{HS}^{2}:=\|\mathcal{A}\|_{S_{2}}^{2}=\|\mathcal{A}^{*}\mathcal{A}\|_{S_{1}}=tr(\mathcal{A}^{*}\mathcal{A})=\sum_{n=1}^{\infty}\|\mathcal{A}v_{n}\|^{2}_{L^{2}(\mathbb{R},A_{\alpha,\beta})}.$

###  5.1. Boundedness and compactness of localization operators

In this subsection, we consider window functions $g_{1},g_{2}\in
M^{1}(\mathbb{R},A_{\alpha,\beta})$ and establish the following boundedness
and compactness results of localization operators.

######  Proposition 5.2.

Let $\varsigma\in
L^{\infty}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$ and
$g_{1},g_{2}\in M^{1}(\mathbb{R},A_{\alpha,\beta})$. Then the localization
operator $\mathfrak{L}_{g_{1},g_{2}}(\varsigma)$ is in
$\mathcal{B}(L^{2}(\mathbb{R},A_{\alpha,\beta}))$ and we have

$\left\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\right\|_{\mathcal{B}(L^{2}(\mathbb{R},A_{\alpha,\beta}))}\leq\|\varsigma\|_{L^{\infty}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;\|g_{1}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}.$

###### Proof.

For every $f,h\in L^{2}(\mathbb{R},A_{\alpha,\beta})$, using Hölder’s
inequality, we obtain

$\displaystyle\left|\left\langle\mathfrak{L}_{g_{1},g_{2}}(\varsigma)(f),h\right\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\right|$
$\displaystyle\leq\iint_{\mathbb{R}^{2}}|\varsigma(x,\xi)|\;\left|{\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)(x,\xi)\right|\;\left|{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(h)(x,\xi)\right|\;d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi)$
$\displaystyle\leq\left\|\varsigma\right\|_{L^{\infty}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\left\|{\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)\right\|_{L^{2}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\left\|{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(h)\right\|_{L^{2}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}.$

Using Plancherel’s formula (4.7), we get

$\displaystyle\left|\left\langle\mathfrak{L}_{g_{1},g_{2}}(\varsigma)(f),h\right\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\right|$
$\displaystyle\leq\left\|\varsigma\right\|_{L^{\infty}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\|f\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{1}\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\;\|h\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}.$

Since $M^{1}(\mathbb{R},A_{\alpha,\beta})\subset
L^{2}(\mathbb{R},A_{\alpha,\beta})$, we have

$\|g_{1}\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\leq\|g_{1}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\quad\text{and}\quad\|g_{2}\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\leq\|g_{2}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}.$

Hence,

$\left\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\right\|_{\mathcal{B}(L^{2}(\mathbb{R},A_{\alpha,\beta}))}\leq\|\varsigma\|_{L^{\infty}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;\|g_{1}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}.$

∎

######  Proposition 5.3.

Let $\varsigma\in
M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$ and
$g_{1},g_{2}\in M^{1}(\mathbb{R},A_{\alpha,\beta})$. Then the localization
operator $\mathfrak{L}_{g_{1},g_{2}}(\varsigma)$ is in
$\mathcal{B}(L^{2}(\mathbb{R},A_{\alpha,\beta}))$ and we have

$\left\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\right\|_{\mathcal{B}(L^{2}(\mathbb{R},A_{\alpha,\beta}))}\leq\|\varsigma\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;\|g_{1}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}.$

###### Proof.

Let $f,h\in L^{2}(\mathbb{R},A_{\alpha,\beta})$. Since
$M^{\infty}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$ is
the dual space of
$M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$, we have

$\displaystyle\left|\left\langle\mathfrak{L}_{g_{1},g_{2}}(\varsigma)(f),h\right\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\right|$
$\displaystyle\leq\iint_{\mathbb{R}^{2}}|\varsigma(x,\xi)|\;\left|{\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)(x,\xi)\;\overline{{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(h)(x,\xi)}\right|\;d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi)$
$\displaystyle\leq\|\varsigma\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\left\|{\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)\cdot\overline{{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(h)}\right\|_{M^{\infty}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}.$
(5.3)

Since the definition of
$M^{\infty}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$ is
independent of the choice of the window $g$, we estimate the STFT of
${\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)\cdot\overline{{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(h)}$
with respect to some $g\in\mathcal{S}(\mathbb{R}^{2})\setminus\\{0\\}$ with
$\|g\|_{L^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\leq
1$. Also, we can write the STFT as the convolution $V_{g}f(x,\xi)=e^{-2\pi
ix\cdot\xi}(f*M_{\xi}g^{*})(x)$, where ∗ is the involution
$g^{*}(x)=\overline{g(-x)}$. Since
$M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})\subset
M^{\infty}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$,
using Young’s convolution inequality, Hölder’s inequality, and Plancherel’s
formula (4.7), we obtain

$\displaystyle\left\|{\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)\cdot\overline{{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(h)}\right\|_{M^{\infty}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}$
$\displaystyle\leq\left\|{\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)\cdot\overline{{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(h)}\right\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}$
$\displaystyle=\left\|V_{g}\left({\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)\cdot\overline{{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(h)}\right)\right\|_{L^{1}(\mathbb{R}^{4},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}$
$\displaystyle=\left\|\left({\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)\cdot\overline{{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(h)}\right)*M_{\xi}g^{*}\right\|_{L^{1}(\mathbb{R}^{4},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}$
$\displaystyle\leq\left\|{\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)\cdot\overline{{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(h)}\right\|_{L^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\left\|M_{\xi}g^{*}\right\|_{L^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}$
$\displaystyle\leq\left\|{\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)\right\|_{L^{2}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\left\|{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(h)\right\|_{L^{2}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}$
$\displaystyle=\|f\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{1}\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\;\|h\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}$
$\displaystyle\leq\|f\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\;\|h\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{1}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}.$
(5.4)

Thus from (5.1) and (5.1), we get

$\left\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\right\|_{\mathcal{B}(L^{2}(\mathbb{R},A_{\alpha,\beta}))}\leq\|\varsigma\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;\|g_{1}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}.$

∎

######  Proposition 5.4.

Let $\varsigma\in
M^{\infty}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$ and
$g_{1},g_{2}\in M^{1}(\mathbb{R},A_{\alpha,\beta})$. Then the localization
operator $\mathfrak{L}_{g_{1},g_{2}}(\varsigma)$ is in
$\mathcal{B}(L^{2}(\mathbb{R},A_{\alpha,\beta}))$ and we have

$\left\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\right\|_{\mathcal{B}(L^{2}(\mathbb{R},A_{\alpha,\beta}))}\leq\|\varsigma\|_{M^{\infty}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;\|g_{1}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}.$

###### Proof.

For every $f,h\in L^{2}(\mathbb{R},A_{\alpha,\beta})$, using the duality
between $M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$
and $M^{\infty}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$,
we deduce that

$\displaystyle\left|\left\langle\mathfrak{L}_{g_{1},g_{2}}(\varsigma)(f),h\right\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\right|$
$\displaystyle\leq\iint_{\mathbb{R}^{2}}|\varsigma(x,\xi)|\;\left|{\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)(x,\xi)\;\overline{{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(h)(x,\xi)}\right|\;d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi)$
$\displaystyle\leq\left\|\varsigma\right\|_{M^{\infty}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\left\|{\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)\cdot\overline{{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(h)}\right\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}.$

Now, using the estimate obtained in (5.1), we get

$\displaystyle\left|\left\langle\mathfrak{L}_{g_{1},g_{2}}(\varsigma)(f),h\right\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\right|$
$\displaystyle\leq\left\|\varsigma\right\|_{M^{\infty}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\|f\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\;\|h\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{1}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}.$

Hence,

$\left\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\right\|_{\mathcal{B}(L^{2}(\mathbb{R},A_{\alpha,\beta}))}\leq\|\varsigma\|_{M^{\infty}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;\|g_{1}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}.$

∎

######  Theorem 5.5.

Let $\varsigma\in
M^{p}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$,
$1<p<\infty$ and $g_{1},g_{2}\in M^{1}(\mathbb{R},A_{\alpha,\beta})$. Then,
for fixed $\varsigma\in
M^{p}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$ the
operator $\mathfrak{L}_{g_{1},g_{2}}$ can be uniquely extended to a bounded
linear operator on $L^{2}(\mathbb{R},A_{\alpha,\beta})$, such that

$\left\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\right\|_{\mathcal{B}(L^{2}(\mathbb{R},A_{\alpha,\beta}))}\leq\|\varsigma\|_{M^{p}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;\|g_{1}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}.$

###### Proof.

Let $1<p<\infty$. For every $\varsigma\in
M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})\cap
M^{\infty}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$,
using Proposition 5.3, Proposition 5.4, the fact that the modulation spaces
$M^{p,q}$ interpolate exactly like the corresponding mixed-norm spaces
$L^{p,q}$ and the Riesz–Thorin interpolation theorem (see [32]), we obtain

$\left\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\right\|_{\mathcal{B}(L^{2}(\mathbb{R},A_{\alpha,\beta}))}\leq\|\varsigma\|_{M^{p}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;\|g_{1}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}.$

Let $\varsigma\in
M^{p}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$ and
$\\{\varsigma_{n}\\}_{n\geq 1}$ be a sequence of functions in
$M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})\cap
M^{\infty}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$ such
that $\varsigma_{n}\to\varsigma$ in
$M^{p}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$ as
$n\to\infty$. Hence for every $n,k\in\mathbb{N}$, we have

$\left\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma_{n})-\mathfrak{L}_{g_{1},g_{2}}(\varsigma_{k})\right\|_{\mathcal{B}(L^{2}(\mathbb{R},A_{\alpha,\beta}))}\leq\|\varsigma_{n}-\varsigma_{k}\|_{M^{p}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;\|g_{1}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}.$

Therefore, $\\{\mathfrak{L}_{g_{1},g_{2}}(\varsigma_{n})\\}_{n\geq 1}$ is a
Cauchy sequence in $\mathcal{B}(L^{2}(\mathbb{R},A_{\alpha,\beta}))$. Let
$\\{\mathfrak{L}_{g_{1},g_{2}}(\varsigma_{n})\\}_{n\geq 1}$ converges to
$\mathfrak{L}_{g_{1},g_{2}}(\varsigma)$. Then the limit
$\mathfrak{L}_{g_{1},g_{2}}(\varsigma)$ is independent of the choice of
$\\{\varsigma_{n}\\}_{n\geq 1}$ and we obtain

$\displaystyle\left\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\right\|_{\mathcal{B}(L^{2}(\mathbb{R},A_{\alpha,\beta}))}$
$\displaystyle=$
$\displaystyle\lim_{n\to\infty}\left\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma_{n})\right\|_{\mathcal{B}(L^{2}(\mathbb{R},A_{\alpha,\beta}))}$
$\displaystyle\leq$
$\displaystyle\lim_{n\to\infty}\|\varsigma_{n}\|_{M^{p}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;\|g_{1}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}$
$\displaystyle=$
$\displaystyle\|\varsigma\|_{M^{p}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;\|g_{1}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}.$

This completes the proof. ∎

######  Theorem 5.6.

Let $\varsigma\in
M^{p}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$, $1\leq
p<\infty$ and $g_{1},g_{2}\in M^{1}(\mathbb{R},A_{\alpha,\beta})$. Then the
localization operator
$\mathfrak{L}_{g_{1},g_{2}}(\varsigma):L^{2}(\mathbb{R},A_{\alpha,\beta})\to
L^{2}(\mathbb{R},A_{\alpha,\beta})$ is compact.

###### Proof.

Assume that $\varsigma\in
M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$. Let
$\\{v_{n}\\}_{n}$ be an orthonormal basis for
$L^{2}(\mathbb{R},A_{\alpha,\beta})$. Since
$M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})\subset
L^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$, using
Fubini’s theorem and Parseval’s identity, we obtain

$\displaystyle\sum_{n=1}^{\infty}\left\langle\mathfrak{L}_{g_{1},g_{2}}(\varsigma)(v_{n}),v_{n}\right\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}$
$\displaystyle=\sum_{n=1}^{\infty}\iint_{\mathbb{R}^{2}}\varsigma(x,\xi)\langle
v_{n},{g_{1}}_{x,\xi}^{(\alpha,\beta)}(-\;\cdot)\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\langle{g_{2}}_{x,\xi}^{(\alpha,\beta)}(-\;\cdot),v_{n}\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi)$
$\displaystyle\leq\iint_{\mathbb{R}^{2}}\varsigma(x,\xi)\left(\sum_{n=1}^{\infty}\langle
v_{n},{g_{1}}_{x,\xi}^{(\alpha,\beta)}(-\;\cdot)\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\langle{g_{2}}_{x,\xi}^{(\alpha,\beta)}(-\;\cdot),v_{n}\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\right)$
$\displaystyle\quad\times\;d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi)$
$\displaystyle\leq\frac{1}{2}\iint_{\mathbb{R}^{2}}\varsigma(x,\xi)\left(\sum_{n=1}^{\infty}|\langle
v_{n},{g_{1}}_{x,\xi}^{(\alpha,\beta)}(-\;\cdot)\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}|^{2}+\sum_{n=1}^{\infty}|\langle{g_{2}}_{x,\xi}^{(\alpha,\beta)}(-\;\cdot),v_{n}\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}|^{2}\right)$
$\displaystyle\quad\times\;d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi)$
$\displaystyle=\frac{1}{2}\|\varsigma\|_{L^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;(\|g_{1}\|^{2}_{L^{2}(\mathbb{R},A_{\alpha,\beta})}+\|g_{2}\|^{2}_{L^{2}(\mathbb{R},A_{\alpha,\beta})})$
$\displaystyle\leq\frac{1}{2}\|\varsigma\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;(\|g_{1}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})}+\|g_{2}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})}).\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad$

Therefore, the operator $\mathfrak{L}_{g_{1},g_{2}}(\varsigma)$ is in $S_{1}$.
Next, assume that $\varsigma\in
M^{p}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$. We
consider a sequence of functions $\\{\varsigma_{n}\\}_{n\geq 1}$ in
$M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})\cap
M^{\infty}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$ such
that $\varsigma_{n}\to\varsigma$ in
$M^{p}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$ as
$n\to\infty$. Then, using Theorem 5.5, we get

$\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma_{n})-\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\|_{\mathcal{B}(L^{2}(\mathbb{R},A_{\alpha,\beta}))}\leq\|\varsigma_{n}-\varsigma\|_{M^{p}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;\|g_{1}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\to
0,$

as $n\to\infty$. Hence,
$\mathfrak{L}_{g_{1},g_{2}}(\varsigma_{n})\to\mathfrak{L}_{g_{1},g_{2}}(\varsigma)$
in $\mathcal{B}(L^{2}(\mathbb{R},A_{\alpha,\beta}))$ as $n\to\infty$. From the
above, we obtain that $\\{\mathfrak{L}_{g_{1},g_{2}}(\varsigma_{n})\\}_{n\geq
1}$ is a sequence of linear operators in $S_{1}$ and hence compact, so
$\mathfrak{L}_{g_{1},g_{2}}(\varsigma)$ is compact. ∎

###  5.2. Boundedness of $\mathfrak{L}_{g_{1},g_{2}}(\varsigma)$ on $S_{p}$

In this subsection, we prove that the localization operator
$\mathfrak{L}_{g_{1},g_{2}}(\varsigma)$ is in $S_{p}$ and provide an upper
bound of the norm $\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\|_{S_{p}}$. We
begin with the following proposition.

######  Proposition 5.7.

Let $\varsigma\in
M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$ and
$g_{1},g_{2}\in M^{1}(\mathbb{R},A_{\alpha,\beta})$. Then the localization
operator
$\mathfrak{L}_{g_{1},g_{2}}(\varsigma):L^{2}(\mathbb{R},A_{\alpha,\beta})\to
L^{2}(\mathbb{R},A_{\alpha,\beta})$ is in $S_{1}$ and

$\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\|_{S_{1}}\leq
2\|\varsigma\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;(\|g_{1}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})}+\|g_{2}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})}).$

###### Proof.

If $\varsigma\in
M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$, then from
the first part of the proof of Theorem 5.6, the operator
$\mathfrak{L}_{g_{1},g_{2}}(\varsigma)$ is in $S_{1}$. Now, to prove the
estimate, assume that $\varsigma$ is non-negative and in
$M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$. Then
$(\mathfrak{L}_{g_{1},g_{2}}(\varsigma)^{*}\mathfrak{L}_{g_{1},g_{2}}(\varsigma))^{1/2}=\mathfrak{L}_{g_{1},g_{2}}(\varsigma)$
(see [4, 35]). Let $\\{v_{n}\\}_{n}$ be an orthonormal basis for
$L^{2}(\mathbb{R},A_{\alpha,\beta})$ consisting of eigenvalues of
$(\mathfrak{L}_{g_{1},g_{2}}(\varsigma)^{*}\mathfrak{L}_{g_{1},g_{2}}(\varsigma))^{1/2}:L^{2}(\mathbb{R},A_{\alpha,\beta})\to
L^{2}(\mathbb{R},A_{\alpha,\beta})$. Then by using the estimate obtained in
the first part of the proof of Theorem 5.6, we get

$\displaystyle\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\|_{S_{1}}$
$\displaystyle=$
$\displaystyle\sum_{n=1}^{\infty}\left\langle(\mathfrak{L}_{g_{1},g_{2}}(\varsigma)^{*}\mathfrak{L}_{g_{1},g_{2}}(\varsigma))^{1/2}(v_{n}),v_{n}\right\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}$
(5.5) $\displaystyle=$
$\displaystyle\sum_{n=1}^{\infty}\left\langle\mathfrak{L}_{g_{1},g_{2}}(\varsigma)(v_{n}),v_{n}\right\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}$
$\displaystyle\leq$
$\displaystyle\frac{1}{2}\|\varsigma\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;(\|g_{1}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})}+\|g_{2}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})}).$

Next, assume that $\varsigma\in
M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$ is an
arbitrary real-valued function. We can write
$\varsigma=\varsigma_{+}-\varsigma_{-}$, where
$\varsigma_{+}=\max(\varsigma,0)$ and $\varsigma_{-}=-\min(\varsigma,0)$.
Then, using relation (5.5), we obtain

$\displaystyle\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\|_{S_{1}}$
$\displaystyle=$
$\displaystyle\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma_{+})-\mathfrak{L}_{g_{1},g_{2}}(\varsigma_{-})\|_{S_{1}}$
(5.6) $\displaystyle\leq$
$\displaystyle\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma_{+})\|_{S_{1}}+\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma_{-})\|_{S_{1}}$
$\displaystyle\leq$
$\displaystyle\frac{1}{2}(\|g_{1}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})}+\|g_{2}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})})(\|\varsigma_{+}\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}+\|\varsigma_{-}\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})})$
$\displaystyle\leq$
$\displaystyle(\|g_{1}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})}+\|g_{2}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})})\|\varsigma\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}.$

Finally, assume that $\varsigma\in
M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$ is a
complex-valued function. Then, we can write
$\varsigma=\varsigma_{1}+i\varsigma_{2}$, where $\varsigma_{1},\varsigma_{2}$
are the real and imaginary parts of $\varsigma$ respectively. Then, using
relation (5.6), we obtain

$\displaystyle\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\|_{S_{1}}$
$\displaystyle=$
$\displaystyle\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma_{1})+i\mathfrak{L}_{g_{1},g_{2}}(\varsigma_{2})\|_{S_{1}}$
$\displaystyle\leq$
$\displaystyle\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma_{1})\|_{S_{1}}+\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma_{2})\|_{S_{1}}$
$\displaystyle\leq$
$\displaystyle(\|g_{1}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})}+\|g_{2}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})})(\|\varsigma_{1}\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}+\|\varsigma_{2}\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})})$
$\displaystyle\leq$ $\displaystyle
2\|\varsigma\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}(\|g_{1}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})}+\|g_{2}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})}).$

This completes the proof. ∎

######  Theorem 5.8.

Let $\varsigma\in
M^{p}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$, $1\leq
p\leq\infty$ and $g_{1},g_{2}\in M^{1}(\mathbb{R},A_{\alpha,\beta})$. Then the
localization operator
$\mathfrak{L}_{g_{1},g_{2}}(\varsigma):L^{2}(\mathbb{R},A_{\alpha,\beta})\to
L^{2}(\mathbb{R},A_{\alpha,\beta})$ is in $S_{p}$ and

$\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\|_{S_{p}}\leq
2^{1/p}\|\varsigma\|_{M^{p}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;(\|g_{1}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})}+\|g_{2}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})})^{1/p}.$

###### Proof.

The proof follows from Proposition 5.4, Proposition 5.7 and by interpolation
theorems (see [35], Theorems 2.10 and 2.11). ∎

Next, we improve the constant given in the previous proposition and also we
provide a lower bound of the norm
$\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\|_{S_{1}}$, more precisely we have
the following.

######  Theorem 5.9.

Let $\varsigma\in
M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$ and
$g_{1},g_{2}\in M^{1}(\mathbb{R},A_{\alpha,\beta})$. Then the localization
operator $\mathfrak{L}_{g_{1},g_{2}}(\varsigma)$ is in $S_{1}$ and we have

$\displaystyle\frac{2}{\|g_{1}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})}+\|g_{2}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})}}\;\|\tilde{\varsigma}\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}$
$\displaystyle\leq\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\|_{S_{1}}\leq\frac{1}{2}(\|g_{1}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})}+\|g_{2}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})})\;\|\varsigma\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})},$

where $\tilde{\varsigma}$ is given by
$\tilde{\varsigma}(x,\xi)=\left\langle\mathfrak{L}_{g_{1},g_{2}}(\varsigma)({g_{1}}_{x,\xi}^{(\alpha,\beta)}),{g_{2}}_{x,\xi}^{(\alpha,\beta)}\right\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}$.

###### Proof.

Since $\varsigma\in
M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$, by
Proposition 5.7, $\mathfrak{L}_{g_{1},g_{2}}(\varsigma)$ is in $S_{1}$. Using
the canonical form of compact operators (see [35], Theorem 2.2), we obtain

$\mathfrak{L}_{g_{1},g_{2}}(\varsigma)(f)=\sum_{n=1}^{\infty}s_{n}(\mathfrak{L}_{g_{1},g_{2}}(\varsigma))\langle
f,v_{n}\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}u_{n},$ (5.7)

where $\\{s_{n}(\mathfrak{L}_{g_{1},g_{2}}(\varsigma))\\}_{n}$ are the
positive singular values of $\mathfrak{L}_{g_{1},g_{2}}(\varsigma)$,
$\\{v_{n}\\}_{n}$ is an orthonormal basis for the orthogonal complement of the
null space of $\mathfrak{L}_{g_{1},g_{2}}(\varsigma)$ consisting of
eigenvectors of $|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)|$ and
$\\{u_{n}\\}_{n}$ is an orthonormal set in
$L^{2}(\mathbb{R},A_{\alpha,\beta})$. Then we have

$\sum_{n=1}^{\infty}\langle\mathfrak{L}_{g_{1},g_{2}}(\varsigma)(v_{n}),u_{n}\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}=\sum_{n=1}^{\infty}s_{n}(\mathfrak{L}_{g_{1},g_{2}}(\varsigma))=\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\|_{S_{1}}.$

Now, using Fubini’s theorem, Cauchy–Schwarz’s inequality, and Bessel’s
inequality, we get

$\displaystyle\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\|_{S_{1}}$
$\displaystyle=$
$\displaystyle\sum_{n=1}^{\infty}\langle\mathfrak{L}_{g_{1},g_{2}}(\varsigma)(v_{n}),u_{n}\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}$
$\displaystyle=$
$\displaystyle\sum_{n=1}^{\infty}\iint_{\mathbb{R}^{2}}\varsigma(x,\xi)\;{\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(v_{n})(x,\xi)\;\overline{{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(u_{n})(x,\xi)}\;d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi)$
$\displaystyle\leq$
$\displaystyle\iint_{\mathbb{R}^{2}}|\varsigma(x,\xi)|\left(\sum_{n=1}^{\infty}|{\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(v_{n})(x,\xi)|^{2}\right)^{1/2}\left(\sum_{n=1}^{\infty}|{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(u_{n})(x,\xi)|^{2}\right)^{1/2}$
$\displaystyle\;\times\;d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi)$
$\displaystyle\leq$
$\displaystyle\|\varsigma\|_{L^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;\|{g_{1}}_{x,\xi}^{(\alpha,\beta)}\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\;\|{g_{2}}_{x,\xi}^{(\alpha,\beta)}\|_{L^{2}(\mathbb{R},A_{\alpha,\beta})}$
$\displaystyle\leq$
$\displaystyle\|\varsigma\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;\|g_{1}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}$
$\displaystyle\leq$
$\displaystyle\frac{1}{2}(\|g_{1}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})}+\|g_{2}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})})\;\|\varsigma\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}.$

Next, we show that $\tilde{\varsigma}\in
M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$. Using
formula (5.7), we obtain

$\displaystyle|\tilde{\varsigma}(x,\xi)|$ $\displaystyle=$
$\displaystyle\left|\left\langle\mathfrak{L}_{g_{1},g_{2}}(\varsigma)({g_{1}}_{x,\xi}^{(\alpha,\beta)}),{g_{2}}_{x,\xi}^{(\alpha,\beta)}\right\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\right|$
$\displaystyle=$
$\displaystyle\left|\sum_{n=1}^{\infty}s_{n}(\mathfrak{L}_{g_{1},g_{2}}(\varsigma))\left\langle{g_{1}}_{x,\xi}^{(\alpha,\beta)},v_{n}\right\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\left\langle
u_{n},{g_{2}}_{x,\xi}^{(\alpha,\beta)}\right\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\right|$
$\displaystyle\leq$
$\displaystyle\frac{1}{2}\sum_{n=1}^{\infty}s_{n}(\mathfrak{L}_{g_{1},g_{2}}(\varsigma))\left(\left|\left\langle{g_{1}}_{x,\xi}^{(\alpha,\beta)},v_{n}\right\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\right|^{2}+\left|\left\langle{g_{2}}_{x,\xi}^{(\alpha,\beta)},u_{n}\right\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\right|^{2}\right).$

Now, using Plancherel’s formula (4.7) and Fubini’s theorem, we get

$\displaystyle\|\tilde{\varsigma}\|_{L^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}$
$\displaystyle=$
$\displaystyle\iint_{\mathbb{R}^{2}}|\tilde{\varsigma}(x,\xi)|\;d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi)$
(5.8) $\displaystyle\leq$
$\displaystyle\frac{1}{2}\sum_{n=1}^{\infty}s_{n}(\mathfrak{L}_{g_{1},g_{2}}(\varsigma))\left(\iint_{\mathbb{R}^{2}}\left|\left\langle{g_{1}}_{x,\xi}^{(\alpha,\beta)},v_{n}\right\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\right|^{2}d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi)\right.$
$\displaystyle\;\;\left.+\iint_{\mathbb{R}^{2}}\left|\left\langle{g_{2}}_{x,\xi}^{(\alpha,\beta)},u_{n}\right\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\right|^{2}d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi)\right)$
$\displaystyle\leq$
$\displaystyle\frac{1}{2}(\|g_{1}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})}+\|g_{2}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})})\sum_{n=1}^{\infty}s_{n}(\mathfrak{L}_{g_{1},g_{2}}(\varsigma))$
$\displaystyle=$
$\displaystyle\frac{1}{2}(\|g_{1}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})}+\|g_{2}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})})\;\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\|_{S_{1}}.$

Also, we have

$\displaystyle\|\tilde{\varsigma}\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}=\|V_{g}\tilde{\varsigma}\|_{L^{1}(\mathbb{R}^{4},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}$
$\displaystyle=$
$\displaystyle\|\tilde{\varsigma}*M_{\xi}g^{*}\|_{L^{1}(\mathbb{R}^{4},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}$
(5.9) $\displaystyle\leq$
$\displaystyle\|\tilde{\varsigma}\|_{L^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})},$

where we obtain the estimate with respect to some
$g\in\mathcal{S}(\mathbb{R}^{2})\setminus\\{0\\}$ with
$\|g\|_{L^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\leq
1$. Finally, using (5.8) and (5.9), we get

$\|\tilde{\varsigma}\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\leq\frac{1}{2}(\|g_{1}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})}+\|g_{2}\|^{2}_{M^{1}(\mathbb{R},A_{\alpha,\beta})})\;\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\|_{S_{1}}.$

This completes the proof of the theorem. ∎

###  5.3. $M^{p}(\mathbb{R},A_{\alpha,\beta})$ Boundedness

In this section, we prove that the localization operators
$\mathfrak{L}_{g_{1},g_{2}}(\varsigma):M^{p}(\mathbb{R},A_{\alpha,\beta})\to
M^{p}(\mathbb{R},A_{\alpha,\beta})$ are bounded. We begin with the following
propositions.

######  Proposition 5.10.

Let $\varsigma\in
M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$, $g_{1}\in
M^{p^{\prime}}(\mathbb{R},A_{\alpha,\beta})$ and $g_{2}\in
M^{p}(\mathbb{R},A_{\alpha,\beta})$, for $1\leq p\leq\infty$. Then the
localization operator
$\mathfrak{L}_{g_{1},g_{2}}(\varsigma):M^{p}(\mathbb{R},A_{\alpha,\beta})\to
M^{p}(\mathbb{R},A_{\alpha,\beta})$ is a bounded linear operator, and we have

$\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\|_{\mathcal{B}(M^{p}(\mathbb{R},A_{\alpha,\beta}))}\leq\|\varsigma\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;\|g_{1}\|_{M^{p^{\prime}}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{M^{p}(\mathbb{R},A_{\alpha,\beta})}.$

###### Proof.

Let $f\in M^{p}(\mathbb{R},A_{\alpha,\beta})$, $1\leq p\leq\infty$ and $g\in
M^{p^{\prime}}(\mathbb{R},A_{\alpha,\beta})$. Then from Hölder’s inequality
and the relation (4.4), we have

$\left|{\mathcal{W}}^{(\alpha,\beta)}_{g}(f)(x,\xi)\right|\leq\|f\|_{M^{p}(\mathbb{R},A_{\alpha,\beta})}\;\|g\|_{M^{p^{\prime}}(\mathbb{R},A_{\alpha,\beta})}.$
(5.10)

For every $f\in M^{p}(\mathbb{R},A_{\alpha,\beta})$ and $h\in
M^{p^{\prime}}(\mathbb{R},A_{\alpha,\beta})$, using the relations (5.2) and
(5.10), we obtain

$\displaystyle\left|\left\langle\mathfrak{L}_{g_{1},g_{2}}(\varsigma)(f),h\right\rangle\right|$
$\displaystyle\leq\iint_{\mathbb{R}^{2}}|\varsigma(x,\xi)|\;\left|{\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)(x,\xi)\right|\left|{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(h)(x,\xi)\right|\;d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi)$
$\displaystyle\leq\|\varsigma\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;\|f\|_{M^{p}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{1}\|_{M^{p^{\prime}}(\mathbb{R},A_{\alpha,\beta})}\;\|h\|_{M^{p^{\prime}}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{M^{p}(\mathbb{R},A_{\alpha,\beta})}.$

Hence,

$\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\|_{\mathcal{B}(M^{p}(\mathbb{R},A_{\alpha,\beta}))}\leq\|\varsigma\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;\|g_{1}\|_{M^{p^{\prime}}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{M^{p}(\mathbb{R},A_{\alpha,\beta})}.$

∎

Next, we obtain an $M^{p}(\mathbb{R},A_{\alpha,\beta})$-boundedness result
using the Schur technique. The estimate obtained for the norm
$\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\|_{\mathcal{B}(M^{p}(\mathbb{R},A_{\alpha,\beta}))}$
is different from the previous Proposition.

######  Proposition 5.11.

Let $\varsigma\in
M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$ and
$g_{1},g_{2}\in M^{1}(\mathbb{R},A_{\alpha,\beta})\cap
L^{\infty}(\mathbb{R},A_{\alpha,\beta})$. Then there exists a bounded linear
operator
$\mathfrak{L}_{g_{1},g_{2}}(\varsigma):M^{p}(\mathbb{R},A_{\alpha,\beta})\to
M^{p}(\mathbb{R},A_{\alpha,\beta})$, $1\leq p\leq\infty$ such that

$\displaystyle\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\|_{\mathcal{B}(M^{p}(\mathbb{R},A_{\alpha,\beta}))}$
$\displaystyle\leq\max(\|g_{1}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\|g_{2}\|_{L^{\infty}(\mathbb{R},A_{\alpha,\beta})},\|g_{1}\|_{L^{\infty}(\mathbb{R},A_{\alpha,\beta})}\|g_{2}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})})\;\|\varsigma\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}.$

###### Proof.

Let $\mathcal{K}$ be the function defined on $\mathbb{R}^{2}$ by

$\mathcal{K}(y,z)=\iint_{\mathbb{R}^{2}}\varsigma(x,\xi)\;\overline{{g_{1}}_{x,\xi}^{(\alpha,\beta)}(-z)}\;{g_{2}}_{x,\xi}^{(\alpha,\beta)}(-y)\;d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi).$
(5.11)

Then we define

$\mathfrak{L}_{g_{1},g_{2}}(\varsigma)(f)(y)=\int_{\mathbb{R}}\mathcal{K}(y,z)f(z)\;A_{\alpha,\beta}(z)\;dz.$

Now, using Fubini’s theorem, for any $z\in\mathbb{R}$, we obtain

$\displaystyle\int_{\mathbb{R}}|\mathcal{K}(y,z)|\;A_{\alpha,\beta}(y)\;dy$
$\displaystyle\leq\int_{\mathbb{R}}\left(\iint_{\mathbb{R}^{2}}|\varsigma(x,\xi)|\left|\overline{{g_{1}}_{x,\xi}^{(\alpha,\beta)}(-z)}\right|\left|{g_{2}}_{x,\xi}^{(\alpha,\beta)}(-y)\right|d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi)\right)A_{\alpha,\beta}(y)\;dy$
$\displaystyle\leq\|g_{1}\|_{L^{\infty}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\;\|\varsigma\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})},$

and for any $y\in\mathbb{R}$, we obtain

$\int_{\mathbb{R}}|\mathcal{K}(y,z)|\;A_{\alpha,\beta}(z)\;dz\leq\|g_{1}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{L^{\infty}(\mathbb{R},A_{\alpha,\beta})}\;\|\varsigma\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}.$

Thus using Schur’s lemma (see [17]), we conclude that, for $1\leq
p\leq\infty$,
$\mathfrak{L}_{g_{1},g_{2}}(\varsigma):M^{p}(\mathbb{R},A_{\alpha,\beta})\to
M^{p}(\mathbb{R},A_{\alpha,\beta})$ is a bounded linear operator, and we have

$\displaystyle\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\|_{\mathcal{B}(M^{p}(\mathbb{R},A_{\alpha,\beta}))}$
$\displaystyle\leq\max(\|g_{1}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\|g_{2}\|_{L^{\infty}(\mathbb{R},A_{\alpha,\beta})},\|g_{1}\|_{L^{\infty}(\mathbb{R},A_{\alpha,\beta})}\|g_{2}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})})\;\|\varsigma\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}.$

∎

######  Remark 5.12.

From the Proposition 5.11, we conclude that the bounded linear operator on
$M^{p}(\mathbb{R},A_{\alpha,\beta})$, $1\leq p\leq\infty$, obtained in
Proposition 5.10 is actually the integral operator on
$M^{p}(\mathbb{R},A_{\alpha,\beta})$ with the kernel $\mathcal{K}$ given by
(5.11).

######  Theorem 5.13.

Let $\varsigma\in
M^{p}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$, $1\leq
p\leq\infty$ and $g_{1},g_{2}\in M^{1}(\mathbb{R},A_{\alpha,\beta})$. Then the
localization operator $\mathfrak{L}_{g_{1},g_{2}}(\varsigma)$ is in
$\mathcal{B}(M^{q}(\mathbb{R},A_{\alpha,\beta}))$, $1\leq q\leq\infty$ and we
have

$\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\|_{\mathcal{B}(M^{q}(\mathbb{R},A_{\alpha,\beta}))}\leq\|\varsigma\|_{M^{p}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;\|g_{1}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}.$

###### Proof.

Let $f\in M^{q}(\mathbb{R},A_{\alpha,\beta})$ and $h\in
M^{q^{\prime}}(\mathbb{R},A_{\alpha,\beta})$, where $q^{\prime}$ is the
conjugate exponent of $q$. Using the duality between the modulation spaces
$M^{p}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$ and
$M^{p^{\prime}}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$,
we obtain

$\displaystyle\left|\left\langle\mathfrak{L}_{g_{1},g_{2}}(\varsigma)(f),h\right\rangle\right|$
$\displaystyle\leq$
$\displaystyle\iint_{\mathbb{R}^{2}}|\varsigma(x,\xi)|\;\left|{\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)(x,\xi)\;\overline{{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(h)(x,\xi)}\right|\;d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi)$
(5.12) $\displaystyle\leq$
$\displaystyle\left\|\varsigma\right\|_{M^{p}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\left\|{\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)\cdot\overline{{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(h)}\right\|_{M^{p^{\prime}}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}.$

Since the definition of
$M^{p^{\prime}}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$
is independent of the choice of the window $g$, we estimate the STFT of
${\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)\cdot\overline{{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(h)}$
with respect to some $g\in\mathcal{S}(\mathbb{R}^{2})\setminus\\{0\\}$ with
$\|g\|_{L^{p^{\prime}}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\leq
1$. Using Young’s convolution inequality and Hölder’s inequality, we get

$\displaystyle\left\|{\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)\cdot\overline{{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(h)}\right\|_{M^{p^{\prime}}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}$
$\displaystyle=\left\|V_{g}\left({\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)\cdot\overline{{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(h)}\right)\right\|_{L^{p^{\prime}}(\mathbb{R}^{4},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}$
$\displaystyle=\left\|\left({\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)\cdot\overline{{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(h)}\right)*M_{\xi}g^{*}\right\|_{L^{p^{\prime}}(\mathbb{R}^{4},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}$
$\displaystyle\leq\left\|{\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)\cdot\overline{{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(h)}\right\|_{L^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\left\|M_{\xi}g^{*}\right\|_{L^{p^{\prime}}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}$
$\displaystyle\leq\left\|{\mathcal{W}}^{(\alpha,\beta)}_{g_{1}}(f)\right\|_{M^{q}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\left\|{\mathcal{W}}^{(\alpha,\beta)}_{g_{2}}(h)\right\|_{M^{q^{\prime}}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}$
$\displaystyle=\left\|f*_{\alpha,\beta}\overline{\mathcal{M}^{(\alpha,\beta)}_{\xi}g_{1}}\right\|_{M^{q}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\left\|h*_{\alpha,\beta}\overline{\mathcal{M}^{(\alpha,\beta)}_{\xi}g_{2}}\right\|_{M^{q^{\prime}}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}$
$\displaystyle\leq\|f\|_{M^{q}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{1}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\;\|h\|_{M^{q^{\prime}}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}.$
(5.13)

Thus from (5.12) and (5.3), we obtain

$\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\|_{\mathcal{B}(M^{q}(\mathbb{R},A_{\alpha,\beta}))}\leq\|\varsigma\|_{M^{p}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}\;\|g_{1}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}.$

∎

###  5.4. Compactness of $\mathfrak{L}_{g_{1},g_{2}}(\varsigma)$ for symbols
in $M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$

In this section, we prove that the localization operators
$\mathfrak{L}_{g_{1},g_{2}}(\varsigma):M^{p}(\mathbb{R},A_{\alpha,\beta})\to
M^{p}(\mathbb{R},A_{\alpha,\beta}),\;1<p<\infty$ are compact for symbols
$\varsigma$ in
$M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$. Let us
start with the following proposition.

######  Proposition 5.14.

Let $\varsigma\in
M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$ and
$g_{1},g_{2}\in M^{1}(\mathbb{R},A_{\alpha,\beta})\cap
L^{\infty}(\mathbb{R},A_{\alpha,\beta})$. Then, for fixed $1<p<\infty$, the
localization operator
$\mathfrak{L}_{g_{1},g_{2}}(\varsigma):M^{p}(\mathbb{R},A_{\alpha,\beta})\to
M^{p}(\mathbb{R},A_{\alpha,\beta})$ is compact.

###### Proof.

Let $\\{f_{n}\\}_{n\in\mathbb{N}}$ be a sequence of functions in
$M^{p}(\mathbb{R},A_{\alpha,\beta})$ such that $f_{n}\rightharpoonup 0$ weakly
in $M^{p}(\mathbb{R},A_{\alpha,\beta})$ as $n\to\infty$. It is sufficient to
prove that
$\lim\limits_{n\to\infty}\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)(f_{n})\|_{M^{p}(\mathbb{R},A_{\alpha,\beta})}=0$.
From the relation (5.1), we obtain

$\displaystyle\left|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)(f_{n})(y)\right|$
$\displaystyle\leq\iint_{\mathbb{R}^{2}}|\varsigma(x,\xi)|\left|\left\langle
f_{n},{g_{1}}_{x,\xi}^{(\alpha,\beta)}(-\;\cdot)\right\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\right|\left|{g_{2}}_{x,\xi}^{(\alpha,\beta)}(-y)\right|\;d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi).$
(5.14)

Since $f_{n}\rightharpoonup 0$ weakly in $M^{p}(\mathbb{R},A_{\alpha,\beta})$,
we deduce that

$\lim_{n\to\infty}|\varsigma(x,\xi)|\left|\left\langle
f_{n},{g_{1}}_{x,\xi}^{(\alpha,\beta)}(-\;\cdot)\right\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\right|\left|{g_{2}}_{x,\xi}^{(\alpha,\beta)}(-y)\right|=0,\quad\text{for
all}\;\;x,y,\xi\in\mathbb{R}.$ (5.15)

Moreover, as $f_{n}\rightharpoonup 0$ weakly in
$M^{p}(\mathbb{R},A_{\alpha,\beta})$ as $n\to\infty$, then there exists a
constant $C>0$ such that $\|f_{n}\|_{M^{p}(\mathbb{R},A_{\alpha,\beta})}\leq
C$. Hence, for all $x,y,\xi\in\mathbb{R}$, we obtain

$|\varsigma(x,\xi)|\left|\left\langle
f_{n},{g_{1}}_{x,\xi}^{(\alpha,\beta)}(-\;\cdot)\right\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\right|\left|{g_{2}}_{x,\xi}^{(\alpha,\beta)}(-y)\right|\leq
C|\varsigma(x,\xi)|\;\|g_{1}\|_{M^{\infty}(\mathbb{R},A_{\alpha,\beta})}\left|{g_{2}}_{x,\xi}^{(\alpha,\beta)}(-y)\right|.$
(5.16)

Further, using Fubini’s theorem, we get

$\displaystyle\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)(f_{n})\|_{M^{p}(\mathbb{R},A_{\alpha,\beta})}$
$\displaystyle=\left\|V_{g}\left(\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\left(f_{n}\right)\right)\right\|_{L^{p}\left(\mathbb{R}^{2},A_{\alpha,\beta}\right)}$
$\displaystyle=\left\|\left(\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\left(f_{n}\right)\right)*M_{\xi}g^{*}\right\|_{L^{p}\left(\mathbb{R}^{2},A_{\alpha,\beta}\right)}$
$\displaystyle\leq\left\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)\left(f_{n}\right)\right\|_{L^{1}\left(\mathbb{R},A_{\alpha,\beta}\right)}\left\|M_{\xi}g^{*}\right\|_{L^{p}\left(\mathbb{R},A_{\alpha,\beta}\right)}$
$\displaystyle\leq\int_{\mathbb{R}}\iint_{\mathbb{R}^{2}}|\varsigma(x,\xi)|\left|\left\langle
f_{n},{g_{1}}_{x,\xi}^{(\alpha,\beta)}(-\;\cdot)\right\rangle_{L^{2}(\mathbb{R},A_{\alpha,\beta})}\right|\left|{g_{2}}_{x,\xi}^{(\alpha,\beta)}(-y)\right|\;d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi)\;A_{\alpha,\beta}(y)\;dy$
$\displaystyle\leq
C\;\|g_{1}\|_{M^{\infty}(\mathbb{R},A_{\alpha,\beta})}\iint_{\mathbb{R}^{2}}|\varsigma(x,\xi)|\int_{\mathbb{R}}\left|{g_{2}}_{x,\xi}^{(\alpha,\beta)}(-y)\right|\;A_{\alpha,\beta}(y)dy\;d(A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})(x,\xi)$
$\displaystyle\leq
C\;\|g_{1}\|_{L^{\infty}(\mathbb{R},A_{\alpha,\beta})}\;\|g_{2}\|_{M^{1}(\mathbb{R},A_{\alpha,\beta})}\;\|\varsigma\|_{M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})}<\infty.$
(5.17)

Thus, using the Lebesgue dominated convergence theorem and the relations (5.4)
– (5.4), we obtain that

$\lim\limits_{n\to\infty}\|\mathfrak{L}_{g_{1},g_{2}}(\varsigma)(f_{n})\|_{M^{p}(\mathbb{R},A_{\alpha,\beta})}=0.$

This completes the proof. ∎

######  Theorem 5.15.

Let $\varsigma\in
M^{1}(\mathbb{R}^{2},A_{\alpha,\beta}\otimes\sigma_{\alpha,\beta})$, $g_{1}\in
M^{p^{\prime}}(\mathbb{R},A_{\alpha,\beta})$ and $g_{2}\in
M^{p}(\mathbb{R},A_{\alpha,\beta})$, for $1<p<\infty$. Then the localization
operator
$\mathfrak{L}_{g_{1},g_{2}}(\varsigma):M^{p}(\mathbb{R},A_{\alpha,\beta})\to
M^{p}(\mathbb{R},A_{\alpha,\beta})$ is compact.

###### Proof.

Let $p^{\prime}$ be the conjugate exponent of $p$. We first show that the
conclusion of the previous proposition holds for $p^{\prime}$. The operator
$\mathfrak{L}_{g_{1},g_{2}}(\varsigma):M^{p^{\prime}}(\mathbb{R},A_{\alpha,\beta})\to
M^{p^{\prime}}(\mathbb{R},A_{\alpha,\beta})$ is the adjoint of the operator
$\mathfrak{L}_{g_{1},g_{2}}(\overline{\varsigma}):M^{p}(\mathbb{R},A_{\alpha,\beta})\to
M^{p}(\mathbb{R},A_{\alpha,\beta})$, which is compact by the previous
proposition. Hence, by the duality properties of modulation spaces,
$\mathfrak{L}_{g_{1},g_{2}}(\varsigma):M^{p^{\prime}}(\mathbb{R},A_{\alpha,\beta})\to
M^{p^{\prime}}(\mathbb{R},A_{\alpha,\beta})$ is compact. Finally, using the
interpolation of the compactness on $M^{p}(\mathbb{R},A_{\alpha,\beta})$ and
on $M^{p^{\prime}}(\mathbb{R},A_{\alpha,\beta})$, the proof is complete. ∎

## Acknowledgments

The author is deeply indebted to Prof. Nir Lev for several fruitful
discussions and generous comments. The author wishes to thank the anonymous
referees for their helpful comments and suggestions that helped to improve the
quality of the paper.

## References

  * [1] J.-P. Anker, F. Ayadi and M. Sifi, Opdam’s hypergeometric functions: product formula and convolution structure in dimension 1, Adv. Pure Appl. Math. 3(1):11–44 (2012).
  * [2] N.B. Andersen, Real Paley–Wiener theorems and Roe’s theorem associated with the Opdam–Cherednik transform, J Math Anal Appl. 427(1):47–59 (2015).
  * [3] C. Baccar and N.B. Hamadi, Localization operators of the wavelet transform associated to the Riemann–Liouville operator, Internat. J. Math. 27(4):1650036, 20 pp. (2016).
  * [4] C. Baccar, N.B. Hamadi and H. Herch, Time-frequency analysis of localization operators associated to the windowed Hankel transform, Integral Trans Spec Funct. 27(3):245–258 (2016).
  * [5] F.A. Berezin, Wick and anti-Wick operator symbols, Math USSR Sb. 15(4):577–606 (1971).
  * [6] P. Boggiatto and M.W. Wong, Two-wavelet localization operators on $L^{p}(\mathbb{R}^{d})$ for the Weyl-Heisenberg group, Integr. Equ. Oper. Theory 49:1–10 (2004).
  * [7] E. Cordero and K. Gröchenig, Time-frequency analysis of localization operators, J Funct Anal. 205(1):107–131 (2003).
  * [8] A. Córdoba and C. Fefferman, Wave packets and Fourier integral operators, Commun Partial Differ Equ. 3(11):979–1005 (1978).
  * [9] I. Daubechies, Time-frequency localization operators: A geometric phase space approach, IEEE Trans. Inf. Theory 34(4):605–612 (1988).
  * [10] I. Daubechies, The wavelet transform, time-frequency localization and signal analysis, IEEE Trans. Inf. Theory 36(5):961–1005 (1990).
  * [11] I. Daubechies and T. Paul, Time-frequency localisation operators – a geometric phase space approach: II. The use of dilations, Inverse Probl. 4(3):661–680 (1988).
  * [12] J.F. van Diejen and L. Vinet, Calogero–Moser–Sutherland Models, CRM Series in Mathematical Physics, Springer, New York (2000).
  * [13] C.F. Dunkl, Hankel transforms associated to finite reflection groups, Contemp Math. 138:123–138 (1992).
  * [14] H.G. Feichtinger, Modulation spaces on locally compact abelian groups, In: Wavelets and their Applications (M. Krishna, R. Radha, S. Thangavelu (eds.)), pp. 1–56, Allied Publishers, New Delhi (2003).
  * [15] H.G. Feichtinger and K. Gröchenig, Gabor frames and time–frequency analysis of distributions, J Funct Anal. 146(2):464–495 (1997).
  * [16] H.G. Feichtinger and K. Nowak, A first survey of Gabor multipliers, in: H.G. Feichtinger and T. Strohmer (Eds.), Advances in Gabor Analysis, Birkhäuser, Boston (2002).
  * [17] G.B. Folland, Introduction to partial differential equations, 2nd edition, Princeton University Press, Princeton, NJ (1995).
  * [18] K. Gröchenig, Foundations of Time–Frequency Analysis, Birkhäuser, Boston (2001).
  * [19] G.J. Heckman, An elementary approach to the hypergeometric shift operators of Opdam, Invent Math. 103:341–350 (1991).
  * [20] K. Hikami, Dunkl operators formalism for quantum many-body problems associated with classical root systems, Phys Soc Japan. 65:394–401 (1996).
  * [21] T.R. Johansen, Remarks on the inverse Cherednik-Opdam transform on the real line, arXiv:1502.01293v1 (2015).
  * [22] F. De Mari, H.G. Feichtinger and K. Nowak, Uniform eigenvalue estimates for time-frequency localization operators, J London Math Soc. 65(3):720–732 (2002).
  * [23] F. De Mari and K. Nowak, Localization type Berezin–Toeplitz operators on bounded symmetric domains, J Geom Anal. 12(1):9–27 (2002).
  * [24] H. Mejjaoli, Spectral theorems associated with the Jacobi–Cherednik operator, Bull Sci Math. 138(3):416–439 (2014).
  * [25] H. Mejjaoli and K. Trimèche, Qualitative uncertainty principles for the generalized Fourier transform associated to a Dunkl type operator on the real line, Anal Math Phys. 6:141–162 (2016).
  * [26] H. Mejjaoli and K. Trimèche, k-Hankel two-wavelet theory and localization operators, Integral Trans Spec Funct. 31(8):620–644 (2020).
  * [27] E.M. Opdam, Harmonic analysis for certain representations of graded Hecke algebras, Acta Math. 175(1):75–121 (1995).
  * [28] E.M. Opdam, Lecture notes on Dunkl operators for real and complex reflection groups, In: Mem Math Soc Japan 8, Tokyo (2000).
  * [29] A. Poria, Uncertainty principles for the Opdam–Cherednik transform on modulation spaces, Integral Trans Spec Funct. 32(3):191–206 (2021).
  * [30] J. Ramanathan and P. Topiwala, Time-frequency localization via the Weyl correspondence, SIAM J. Math. Anal. 24(5):1378–1393 (1993).
  * [31] B. Schapira, Contributions to the hypergeometric function theory of Heckman and Opdam: sharp estimates, Schwartz space, heat kernel, Geom Funct Anal. 18:222–250 (2008).
  * [32] E.M. Stein, Interpolation of linear operators, Trans. Amer. Math. Soc. 83:482–492 (1956).
  * [33] M.W. Wong, Localization operators, Seoul National University Research Institute of Mathematics Global Analysis Research Center, Seoul (1999).
  * [34] M.W. Wong, Localization operators on the Weyl-Heisenberg group, In: Geometry, Analysis and Applications (R.S. Pathak (Ed.)), pp. 303–314, World-Scientific (2001).
  * [35] M.W. Wong, Wavelet transforms and localization operators, Operator Theory: Advances and Applications Vol. 136, Birkhäuser, Basel (2002).
  * [36] M.W. Wong, $L^{p}$ boundedness of localization operators associated to left regular representations, Proc Amer Math Soc. 130(10):2911–2919 (2002).
  * [37] M.W. Wong, Localization operators on the affine group and paracommutators, Progress in Analysis, pp. 663–669, World Scientific (2003).