###### Proof.

The claim follows by an adaptation of the proof given in [19] (cf. also
techniques of [8, the proof of Lemma 1.2]). ∎

We formulate the next corollary concerning the restricted collision operator
(below the primes in $S^{\prime}$ and $I^{\prime}$ refer to their arguments
$\omega^{\prime}$ and $E^{\prime}$, respectively). For simplicity, we assume
that the restricted collision operator $K_{r}$ is of the form
$K_{r}=K_{r}^{1}$ but the claims of the below Corollary 5.13 are valid also
when $K_{r}$ is of the form $K_{r}=K_{r}^{2}$.

###### Corollary 5.13

A. Suppose that $s\geq 1$ and that $\sigma^{1}:\mathbb{R}^{3}\times
S^{\prime}\times S\times I^{\prime}\times I\to\mathbb{R}$ is a measurable
function such that

$\displaystyle\sigma^{1}\in L^{2}(S\times
I,W^{\infty,((s-1)/2,0,0)}(\mathbb{R}^{3}\times
S^{\prime}\times{I^{\prime}}^{\circ}))$
$\displaystyle\omega^{\prime}\cdot\nabla_{x}\sigma^{1}\in L^{2}(S\times
I,W^{\infty,((s-1)/2,0,0)}(\mathbb{R}^{3}\times
S^{\prime}\times{I^{\prime}}^{\circ})).$ (203)

Then $K_{r}^{1}:{\mathcal{H}}^{((s-1)/2,0,0)}(\mathbb{R}^{3}\times S\times
I^{\circ})\to H^{(s/2,0,0)}(\mathbb{R}^{3}\times S\times I^{\circ})$ given by

$\displaystyle(K_{r}^{1}\psi)(x,\omega,E)=\int_{S^{\prime}\times
I^{\prime}}\sigma^{1}(x,\omega^{\prime},\omega,E^{\prime},E)\psi(x,\omega^{\prime},E^{\prime})d\omega^{\prime}dE^{\prime}$

is bounded.

B. Suppose that $G$ is convex and bounded. Furthermore, suppose that
$\sigma^{1}:G\times S^{\prime}\times S\times I^{\prime}\times I\to\mathbb{R}$
is a measurable function and that

$\displaystyle\sigma^{1}\in L^{2}(S\times I,L^{\infty}(G\times
S^{\prime}\times I^{\prime}))$
$\displaystyle\omega^{\prime}\cdot\nabla_{x}\sigma^{1}\in L^{2}(S\times
I,L^{\infty}(G\times S^{\prime}\times I^{\prime}))$
$\displaystyle\sigma^{1}\in L^{2}(S\times I,L^{\infty}(\Gamma^{\prime}\times
I^{\prime}))$ (204)

(here $\Gamma^{\prime}=(\partial G)\times S^{\prime}$). Then
$K_{r}^{1}:\widetilde{W}^{2}(G\times S\times I)\to H^{(1/2,0,0)}(G\times
S\times I)$ is bounded.

###### Proof.

Define for a fixed $(\omega,E)\in S\times I$ a mapping
$\Psi_{\omega,E}:G\times S^{\prime}\times I^{\prime}\to\mathbb{R}$ by

$\displaystyle\Psi_{\omega,E}(x,\omega^{\prime},E^{\prime}):=\sigma^{1}(x,\omega^{\prime},\omega,E^{\prime},E)\psi(x,\omega^{\prime},E^{\prime}).$

Noting that

$\displaystyle(K_{r}^{1}\psi)(x,\omega,E)=({\mathcal{M}}\Psi_{\omega,E})(x)$

the proof follows from Theorem 5.12. We omit details. ∎

Consider the transport problem

$\displaystyle T\psi=f,\ \psi_{|\Gamma_{-}}=g,\ \psi(.,.,E_{m})=0.$ (205)

Let $u:=\psi-L_{-}g$. Assuming that the incoming data $g$ is smooth enough and
that the compatibility condition $g(.,.,E_{m})=0$ holds the problem (205) is
equivalent to

$\displaystyle Tu=f-T(L_{-}g),\ u_{|\Gamma_{-}}=0,\ u(.,.,E_{m})=0.$ (206)

Hence it suffices to consider only the problem of the form

$\displaystyle Tu=f,\ u_{|\Gamma_{-}}=0,\ u(.,.,E_{m})=0$ (207)

for which the homogeneous inflow boundary condition $u_{|\Gamma_{-}}=0$ holds.
Assume (for simplicity) that the transport operator $T$ is of the form

$\displaystyle Tu=-a{\frac{\partial u}{\partial
E}}+\omega\cdot\nabla_{x}u+\Sigma u-K_{r}u$

where $a$ and $\Sigma$ are positive constants and where $K_{r}=K_{r}^{2}$.

Let the assumptions of Theorem 5.1 and Remark 5.10 be valid. Due to (136)
(here we use notations of section 5.1)

$\displaystyle
u=\widetilde{P}_{C,0}^{-1}K_{r,C}u+\widetilde{P}_{C,0}^{-1}f_{C}.$ (208)

By iteration we obtain from (208)

$\displaystyle
u=\widetilde{P}_{C,0}^{-1}K_{r,C}u+\widetilde{P}_{C,0}^{-1}f_{C}$
$\displaystyle=\widetilde{P}_{C,0}^{-1}K_{r,C}\big{(}\widetilde{P}_{C,0}^{-1}K_{r,C}u+\widetilde{P}_{C,0}^{-1}f_{C}\big{)}+\widetilde{P}_{C,0}^{-1}f_{C}$
$\displaystyle=(\widetilde{P}_{C,0}^{-1}K_{r,C})^{2}u+(\widetilde{P}_{C,0}^{-1}K_{r,C})\widetilde{P}_{C,0}^{-1}f_{C}+\widetilde{P}_{C,0}^{-1}f_{C}$
$\displaystyle=(\widetilde{P}_{C,0}^{-1}K_{r,C})^{3}u+(\widetilde{P}_{C,0}^{-1}K_{r,C})^{2}\widetilde{P}_{C,0}^{-1}f_{C}+(\widetilde{P}_{C,0}^{-1}K_{r,C})\widetilde{P}_{C,0}^{-1}f_{C}+\widetilde{P}_{C,0}^{-1}f_{C}.$
(209)

The iteration can be continued in the similar way.

The iteration formula (5.3) together with the smoothing property of collision
operator can be used to prove the regularity of solutions (cf. [8]). The idea
is as follows:

Let $f_{C}\in H^{(\kappa,0,1)}(G\times S\times I^{\circ}),\ 0<\kappa\leq 1/2$.
Then $f_{C}\in H^{(\kappa,0,0)}(G\times S\times I^{\circ})$ and we are able to
deduce (under some additional assumptions) that

$\displaystyle w:=\widetilde{P}_{C,0}^{-1}f_{C}\in H^{(\kappa,0,0)}(G\times
S\times I^{\circ}).$ (210)

This is based on the following decomposition. In virtue of (4.2) the solution
$w$ is

$\displaystyle w(x,\omega,E)=\int_{0}^{\alpha(x,\omega,E)}e^{-\Sigma s}\cdot
f_{C}(x-s\omega,\omega,E+as)ds$ (211)

where $\alpha(x,\omega,E):=\min\\{\eta(E),t(x,\omega)\\},\
\eta(E)={{E_{m}-E}\over a}$. We decompose

$\displaystyle w(x,\omega,E)-w(y,\omega,E)$
$\displaystyle=\int_{0}^{\alpha(x,\omega,E)}e^{-\Sigma
s}\cdot\big{(}f_{C}(x-s\omega,\omega,E+as)-f_{C}(y-s\omega,\omega,E+as)\big{)}ds$
$\displaystyle+\Big{(}\int_{0}^{\alpha(x,\omega,E)}e^{-\Sigma
s}f_{C}(y-s\omega,\omega,E+as)ds-\int_{0}^{\alpha(y,\omega,E)}e^{-\Sigma
s}f_{C}(y-s\omega,\omega,E+as)ds\Big{)}$
$\displaystyle=:v_{1}(x,y,\omega,E)+v_{2}(x,y,\omega,E).$ (212)

Note that

$\displaystyle
v_{2}(x,y,\omega,E)=-\int_{\alpha(x,\omega,E)}^{\alpha(y,\omega,E)}e^{-\Sigma
s}\cdot f_{C}(y-s\omega,\omega,E+as)ds$

It suffices to show that

$\displaystyle
Q_{j}:=\int_{I}\int_{S}\int_{G}\int_{G}{{|v_{j}(x,y,\omega,E)|^{2}}\over{|x-y|^{3+2\kappa}}}dydxd\omega
dE<\infty,\ j=1,2.$

The term $Q_{1}$ can be seen to be finite by fairly standard estimations. The
term $Q_{2}$ requires more specific consideration and we do not analyse it
here (see [8, the proof of Lemma 1.12]).

Furthermore, due to Remark 5.10 $u\in H^{(0,0,1)}(G\times S\times I^{\circ})$
and so ${\frac{\partial u}{\partial E}}\in L^{2}(G\times S\times I)$. This
implies that

$\displaystyle\omega\cdot\nabla_{x}u=a{\frac{\partial u}{\partial E}}-\Sigma
u+K^{2}_{r,C}u+f_{C}\in L^{2}(G\times S\times I)$ (213)

and so $u\in W^{2}(G\times S\times I)$. In addition, $u_{|\Gamma_{-}}=0$ and
then $u\in\widetilde{W}^{2}(G\times S\times I)$. In virtue of Corollary 5.13
$K_{r,C}u\in H^{(1/2,0,0)}(G\times S\times I^{\circ})$. Hence similarly to
(210) (under appropriate additional assumptions)
$\widetilde{P}_{C,0}^{-1}K_{r,C}u\in H^{(\kappa,0,0)}(G\times S\times
I^{\circ})$ and so by (5.3)

$\displaystyle
u=\widetilde{P}_{C,0}^{-1}K_{r,C}u+\widetilde{P}_{C,0}^{-1}f_{C}\in
H^{(\kappa,0,0)}(G\times S\times I^{\circ}).$ (214)

Higher order regularity up to certain (fractional) order can be obtained by
using higher order iteration (5.3). In the case where $G=\mathbb{R}^{3}$ the
solution is

$\displaystyle u(x,\omega,E)=\int_{0}^{\eta(E)}e^{-\Sigma s}\cdot
f_{C}(x-s\omega,\omega,E+as)ds.$

and the method can be founded fairly straightforwardly on Corollary 5.13/Part
A. In this case only the repeating of the first iteration formula (5.3) is
sufficient. For example, when we know that
$u\in{\mathcal{H}}^{(\kappa,0,0)}(\mathbb{R}^{3}\times S\times I^{\circ})$ we
can under appropriate assumptions conclude by Corollary 5.13/Part A (by
choosing $s=2\kappa+1$) that $K_{r,C}u\in
H^{(2\kappa,0,0)}(\mathbb{R}^{3}\times S\times I^{\circ})$ and so

$\displaystyle
u=\widetilde{P}_{C,0}^{-1}K_{r,C}u+\widetilde{P}_{C,0}^{-1}f_{C}\in
H^{(2\kappa,0,0)}(\mathbb{R}^{3}\times S\times I^{\circ})$

where we additionally used the (quite easily verifiable) fact that for $s\geq
0$

$\displaystyle w=\widetilde{P}_{C,0}^{-1}f\in H^{(s,0,0)}(\mathbb{R}^{3}\times
S\times I^{\circ})\ {\rm when}\ f\in H^{(s,0,0)}(\mathbb{R}^{3}\times S\times
I^{\circ}).$

In the case where $G\subset\mathbb{R}^{3}$ is bounded the method needs more
subtle treatise because the escape time mapping $t(x,\omega)$ is included in
the solution formula (cf. [8] where regularity with respect to $x$ up to order
$1-\epsilon$ is achieved).

Suitability of this method to more general transport equations is potential.

##  6\. Some outlines for regularity eminating from general theory of first
order PDEs

###  6.1. Transport problem in rectified coordinates

In some cases the transport problem in the so called rectified coordinates is
useful. At first we settle how the problem

$\displaystyle T\psi=a(x,E){\frac{\partial\psi}{\partial
E}}+c(x,E)\Delta_{S}\psi+d(x,\omega,E)\cdot\nabla_{S}\psi+\omega\cdot\nabla_{x}\psi+\Sigma\psi-
K_{r}\psi$ $\displaystyle\psi_{|\Gamma_{-}}=g,\ \psi(.,.,E_{m})=0$ (215)

locally changes when the boundary is rectified.

The boundary of a sufficiently regular domain $G\subset\mathbb{R}^{3}$ can be
rectified in the following sense. The boundary $\partial G$ of $G$ is said to
be in the class $C^{k}$, or shortly $\partial G\in C^{k}$, (see e.g. [20],
section 1.2) if for any $y\in\partial G$ there exists an open neighbourhood
$U_{y}\subset\mathbb{R}^{3}$ of $y$ and a diffeomorphism $h:U_{y}\to D_{a,b}$
such that

$\displaystyle h\in C_{*}^{k}(U_{y},D_{a,b}),\ h^{-1}\in
C_{*}^{k}(D_{a,b},U_{y}),$ $\displaystyle h^{-1}(D_{+,a,b})=G\cap U_{y},\
h^{-1}(D_{0,a,b})=(\partial G)\cap U_{y}$ (216)

where

$\displaystyle C_{*}^{k}(U,U^{\prime}):=\\{f\in C^{k}(U,U^{\prime})|\ {\rm
for\ which}\ \left\|f\right\|_{C_{*}^{k}(U,U^{\prime})}:=\sum_{|\alpha|\leq
k}\left\|\partial_{x}^{\alpha}f\right\|_{L^{\infty}(U,U^{\prime})}<\infty\\},$
$\displaystyle D_{a,b}:=\\{z=(z^{\prime},z_{3})\in\mathbb{R}^{3}|\
\left\|z^{\prime}\right\|<a,\ |z_{3}|<b\\},$ $\displaystyle D_{+,a,b}:=\\{z\in
D_{a,b}|\ z_{3}>0\\},$ $\displaystyle D_{0,a,b}:=\\{z\in D_{a,b}|\
z_{3}=0\\}.$

Firstly we consider in which way the equation $T\psi=f$ changes under the
diffeomorphism $h:U_{y}\to D_{a,b}$. We assume that the coefficients $a,\ c,\
d,\ \Sigma,\ \sigma^{j},\ j=1,2,3$ are regular enough up to the boundary. Let
$(y,\omega,E)\in\Gamma$ and let $\phi:D_{+,a,b}\to\mathbb{R}$ be a mapping

$\displaystyle\phi(z,\omega,E):=(\psi\circ
h^{-1})(z,\omega,E):=\psi(h^{-1}(z),\omega,E).$

Then $\psi=\phi\circ h$ in $G\cap U_{y}$ when we define $(\phi\circ
h)(x,\omega,E):=\phi(h(x),\omega,E)$. Since $h$ does not depend on $\omega$
and $E$ we have for $(z,\omega,E)\in V_{+}:=D_{+,a,b}\times S\times I$

$\displaystyle(a{\frac{\partial\psi}{\partial E}})\circ h^{-1}=(a\circ
h^{-1}){\frac{\partial\phi}{\partial E}},\ (c\Delta_{S}\psi)\circ
h^{-1}=(c\circ h^{-1})\Delta_{S}\phi$
$\displaystyle(d\cdot\nabla_{S}\psi)\circ h^{-1}=(d\circ
h^{-1})\cdot\nabla_{S}\phi,\ (\Sigma\psi)\circ h^{-1}=(\Sigma\circ
h^{-1})\phi$

and

$\displaystyle((K_{r}\psi)\circ h^{-1})(z,\omega,E)$
$\displaystyle=\int_{S^{\prime}\times
I^{\prime}}\sigma^{1}(h^{-1}(z),\omega^{\prime},\omega,E^{\prime},E)\phi(z,\omega^{\prime},E^{\prime})d\omega^{\prime}dE^{\prime}$
$\displaystyle+\int_{S^{\prime}}\sigma^{2}(h^{-1}(z),\omega^{\prime},\omega,E)\phi(z,\omega^{\prime},E)d\omega^{\prime},$
$\displaystyle+\int_{I^{\prime}}\int_{0}^{2\pi}\sigma^{3}(h^{-1}(z),E^{\prime},E)\phi(z,\gamma(E^{\prime},E,\omega)(s),E^{\prime})dsdE^{\prime}$
$\displaystyle=:(\widetilde{K}_{r}\phi)(z,\omega,E).$ (217)

Furthermore,

$\displaystyle{\frac{\partial\psi}{\partial x_{j}}}={\frac{\partial(\phi\circ
h)}{\partial x_{j}}}=\left\langle(\nabla_{z}\phi)\circ h,{\frac{\partial
h}{\partial x_{j}}}\right\rangle$

and so

$\displaystyle\omega\cdot\nabla_{x}\psi=\sum_{j=1}^{3}\omega_{j}\left\langle(\nabla_{z}\phi)\circ
h,{\frac{\partial h}{\partial x_{j}}}\right\rangle.$

Hence

$\displaystyle(\omega\cdot\nabla_{x}\psi)\circ
h^{-1}=\left\langle\nabla_{z}\phi,\sum_{j=1}^{3}\omega_{j}{\frac{\partial
h}{\partial x_{j}}}\circ
h^{-1}\right\rangle=\tilde{b}(z,\omega)\cdot\nabla_{z}\phi$ (218)

where

$\displaystyle\tilde{b}(z,\omega):=\sum_{j=1}^{3}\omega_{j}{\frac{\partial
h}{\partial x_{j}}}\circ
h^{-1}=(\omega\cdot\nabla_{x}h_{1},\omega\cdot\nabla_{x}h_{2},\omega\cdot\nabla_{x}h_{3})\circ
h^{-1}.$ (219)

The equation $T\psi=f$ holds in $(G\cap U_{y})\times S\times I$ if and only if
$(T\psi)\circ h^{-1}=f\circ h^{-1}$ in $V_{+}$ that is,

$\displaystyle\big{(}a{\frac{\partial\psi}{\partial E}}\big{)}\circ
h^{-1}+(c\Delta_{S}\psi)\circ h^{-1}+(d\cdot\nabla_{S}\psi)\circ
h^{-1}+(\omega\cdot\nabla_{x}\psi)\circ h^{-1}$
$\displaystyle+(\Sigma\psi)\circ h^{-1}-(K_{r}\psi)\circ h^{-1}=f\circ h^{-1}$

which by the above computations is equivalent to

$\displaystyle\tilde{a}(z,E){\frac{\partial\phi}{\partial
E}}+\tilde{c}(z,E)\Delta_{S}\phi+\tilde{d}(z,\omega,E)\cdot\nabla_{S}\phi+\tilde{b}(x,\omega)\cdot\nabla_{z}\phi+\tilde{\Sigma}\phi-\widetilde{K}_{r}\phi=\tilde{f}$
(220)

where $\tilde{b}(z,E)$ and $\widetilde{K}_{r}$ are given by (219), (6.1),
respectively and where

$\displaystyle\tilde{a}(z,E):=a(h^{-1}(z),E),\
\tilde{c}(z,E):=c(h^{-1}(z),E),\
\tilde{d}(z,\omega,E):=d(h^{-1}(z),\omega,E),$
$\displaystyle\tilde{\Sigma}(z,\omega,E):=\Sigma(h^{-1}(z),\omega,E),\
\tilde{f}(z,\omega,E):=f(h^{-1}(z),\omega,E).$

Secondly, we consider the inflow boundary condition. We find that (recall
(6.1))

$\displaystyle(\partial G)\cap U_{y}=\\{y^{\prime}\in U_{y}|\
h_{3}(y^{\prime})=0\\}$

and so (when we fix $h_{3}$ such that $-\nabla_{x}h_{3}$ is pointing outwards
on $(\partial G)\cap U_{y}$).

$\displaystyle\nu(y^{\prime})=-{{\nabla_{x}h_{3}(y^{\prime})}\over{\left\|\nabla_{x}h_{3}(y^{\prime})\right\|}}.$

Then

$\displaystyle\omega\cdot\nu(y^{\prime})<0\Leftrightarrow\omega\cdot\nabla_{x}h_{3}(y^{\prime})>0.$
(221)

Let $n:=(0,0,-1)$ (=the outward pointing unit normal on on $D_{0,a,b}$) and
let $\overline{\sigma}$ be the surface measure
$D_{0,a,b}\subset\mathbb{R}^{3}$. Then we find that

$\displaystyle\tilde{b}\cdot
n=\sum_{j=1}^{3}((\omega\cdot\nabla_{x}h_{j})\circ
h^{-1})n_{j}=-(\omega\cdot\nabla_{x}h_{3})\circ h^{-1}$

and so by (221)

$\displaystyle\omega\cdot\nu(y^{\prime})<0\Leftrightarrow\omega\cdot\nabla_{x}h_{3}(y^{\prime})>0\Leftrightarrow\tilde{b}_{3}(z,\omega)>0\Leftrightarrow\tilde{b}(z,\omega)\cdot
n(z)=-\tilde{b}_{3}(z,\omega)<0$ (222)

where $y^{\prime}=h^{-1}(z)$. Note that the initial condition leaves
$\phi(.,.,E_{m})=0$.

Let

$\displaystyle{R}_{0,a,b,-}^{\prime}:=\\{(z,\omega)\in D_{0,a,b}\times S|\
\tilde{b}_{3}(z,\omega)<0\\},\ {R}_{0,a,b,-}:=R_{0,a,b,-}^{\prime}\times I.$

As a conclusion we see that the problem (6.1) is _locally around $\Gamma$_
equivalent to the problem

$\displaystyle\widetilde{T}\phi:=\tilde{a}(z,E){\frac{\partial\phi}{\partial
E}}+\tilde{c}(z,E)\Delta_{S}\phi+\tilde{d}(z,\omega,E)\cdot\nabla_{S}\phi+\tilde{b}(x,\omega)\cdot\nabla_{z}\phi+\tilde{\Sigma}\phi-\widetilde{K}_{r}\phi=\tilde{f}$
$\displaystyle\phi_{|R_{0,a,b,-}}=\tilde{g},\ \phi(.,.,E_{m})=0$ (223)

where

$\displaystyle\tilde{g}(z,\omega,E):=g(h^{-1}(z),\omega,E).$

We see that the problem is again of variable multiplicity.

###### Remark 6.1

Using the local results (that is, results for patches $G\cap U_{y}$), the
global results (that is, results for the whole $G$) can be retrieved by
applying the usual _partition of unity process_ ([58, section 1.2]). Above we
treated only localization around $\Gamma$. Using the standard localization and
partition of unity methods only these patches are interesting. The interior
patches can be easily handled.

###  6.2. On tangential regularity

The solutions of the characteristic initial boundary value problems (these are
problems for which the kernel of the boundary matrix is non-zero) do not have
necessarily full Sobolev regularity (e.g. [39, Examples 2.1, 2.2]). However,
the problems whose boundary matrix has a constant rank (problems with constant
multiplicity) the so called tangential (or co-normal) Sobolev regularity can
be shown under relevant assumptions (e.g. [43], [38]). For the problems with
variable multiplicity even the tangential regularity is not necessarily
generally valid but for some problems it may be possible. If one applies the
techniques by [43] the difficulties in proving the tangential regularity for
problems with the variable multiplicity arise in the validity of formulas (19)
and (20) therein (the proof of Theorem 6). As mentioned above, our problem has
a variable multiplicity and the obstacles of the validity of [43, (19), (20)]
can be seen by direct computations. In the sequel we formulate some details of
tangential regularity concerning for transport problems under consideration.

A smooth vector field
$F=\sum_{j=1}^{3}a_{j}\partial_{x_{j}}:\overline{G}\times S\times I\to
T(\mathbb{R}^{3})=\mathbb{R}^{3}$ is _tangential (or co-normal) with respect
to $\Gamma=(\partial G)\times S\times I$_ if (here $a:=(a_{1},a_{2},a_{3})$)

$\displaystyle\left\langle a,\nu\right\rangle=0\ {\rm on}\ \Gamma.$

Similarly, we define tangential (or co-normal) vector fields with respect to
$\Gamma_{\pm}$ and tangential (or co-normal) vector fields
$F:\overline{D}_{+,a,b}\times S\times I\to T(\mathbb{R}^{3})$ with respect to
$D_{0,a,b}\times S\times I$.

We find that $\\{\partial_{z_{1}},\partial_{z_{2}},z_{3}\partial_{z_{3}}\\}$
forms a basis for the space of tangential vector fields
$F:\eta\in\overline{D}_{+,a,b}\times S\times I\to T(\mathbb{R}^{3})$ with
respect to $D_{0,a,b}\times S\times I$. This follows from the fact that for
any smooth $\eta\in\overline{D}_{+,a,b}\times S\times I$ obeying
$\eta(z_{1},z_{2},0,\omega,E)=0$ we have

$\displaystyle\eta(z_{1},z_{2},z_{3},\omega,E)=z_{3}\int_{0}^{1}{\frac{\partial\eta}{\partial
z_{3}}}(z_{1},z_{2},tz_{3},\omega,E)dt.$

This implies that the space of tangential vector fields with respect to
$\Gamma$ has a natural local basis (near $\Gamma$) as well which can be seen
as follows. We find that for $\phi=\psi\circ h^{-1}$ and for $j=1,2$

$\displaystyle{\frac{\partial\phi}{\partial
z_{j}}}=\left\langle(\nabla_{x}\psi)\circ
h^{-1},{\frac{\partial(h^{-1})}{\partial z_{j}}}\right\rangle.$

Let $A_{h}$ and $A_{h^{-1}}$ be the Jacobian matrices of $h$ and $h^{-1}$,
respectively. Since

$\displaystyle
A_{h^{-1}}=\left(\begin{matrix}{\frac{\partial(h^{-1})}{\partial
z_{1}}}&{\frac{\partial(h^{-1})}{\partial
z_{2}}}&{\frac{\partial(h^{-1})}{\partial z_{3}}}\\\
\end{matrix}\right)=A_{h}^{-1}\circ h^{-1}=(({\rm det}(A_{h}))^{-1}\circ
h^{-1})\left(\begin{matrix}A_{1}&A_{2}&A_{3}\\\ \end{matrix}\right)\circ
h^{-1}$

where

$\displaystyle
A_{1}=\left(\begin{matrix}\partial_{2}h_{2}\partial_{3}h_{3}-\partial_{2}h_{3}\partial_{3}h_{2}\\\
\partial_{1}h_{2}\partial_{3}h_{3}-\partial_{1}h_{3}\partial_{3}h_{2}\\\
\partial_{1}h_{2}\partial_{2}h_{3}-\partial_{1}h_{3}\partial_{2}h_{2}\\\
\end{matrix}\right),A_{2}=\left(\begin{matrix}\partial_{3}h_{1}\partial_{2}h_{3}-\partial_{3}h_{3}\partial_{2}h_{1}\\\
\partial_{1}h_{1}\partial_{3}h_{3}-\partial_{1}h_{3}\partial_{3}h_{1}\\\
\partial_{2}h_{1}\partial_{1}h_{3}-\partial_{2}h_{3}\partial_{1}h_{1}\\\
\end{matrix}\right),A_{3}=\left(\begin{matrix}\partial_{2}h_{1}\partial_{3}h_{2}-\partial_{2}h_{2}\partial_{3}h_{1}\\\
\partial_{3}h_{1}\partial_{1}h_{2}-\partial_{3}h_{2}\partial_{1}h_{1}\\\
\partial_{1}h_{1}\partial_{2}h_{2}-\partial_{1}h_{2}\partial_{2}h_{1}\\\
\end{matrix}\right)$

we see that

$\displaystyle{\frac{\partial(h^{-1})}{\partial z_{j}}}=(({\rm
det}(A_{h}))^{-1}\circ h^{-1})A_{j}\circ h^{-1}.$

Hence

$\displaystyle{\frac{\partial\phi}{\partial z_{j}}}=(({\rm
det}(A_{h}))^{-1}\circ h^{-1})\left\langle(\nabla_{x}\psi)\circ
h^{-1},A_{j}\circ h^{-1}\right\rangle$ $\displaystyle=(({\rm
det}(A_{h}))^{-1}\circ
h^{-1})\left\langle\nabla_{x}\psi,A_{j}\right\rangle\circ h^{-1}.$ (224)

Furthermore, since

$\displaystyle\nu=-{{\nabla_{x}h_{3}}\over{\left\|\nabla_{x}h_{3}\right\|}}$

we see that on $\Gamma$

$\displaystyle-\left\|\nabla_{x}h_{3}\right\|\left\langle\nu,A_{1}\right\rangle$
$\displaystyle=-\big{(}\partial_{1}h_{3}\partial_{2}h_{2}\partial_{3}h_{3}-\partial_{1}h_{3}\partial_{2}h_{3}\partial_{3}h_{2}+\partial_{2}h_{3}\partial_{1}h_{2}\partial_{3}h_{3}-\partial_{2}h_{3}\partial_{1}h_{3}\partial_{3}h_{2}$
$\displaystyle+\partial_{3}h_{3}\partial_{1}h_{2}\partial_{2}h_{3}-\partial_{3}h_{3}\partial_{1}h_{3}\partial_{2}h_{2}\big{)}=0.$
(225)

Similarly

$\displaystyle\left\|\nabla_{x}h_{3}\right\|\left\langle\nu,A_{2}\right\rangle=0$
(226)

on $\Gamma$ and so $F_{j}:=({\rm det}(A_{h}))^{-1}\left\langle
A_{j},\nabla_{x}\right\rangle,\ j=1,2$ are tangential vector fields with
respect to $\Gamma$. Finally, we analogously find that

$\displaystyle z_{3}{\frac{\partial\phi}{\partial z_{3}}}=(h_{3}(({\rm
det}(A_{h}))^{-1})\circ
h^{-1})\left\langle\nabla_{x}\psi,A_{3}\right\rangle\circ h^{-1}.$ (227)

Since $h_{3}(y)=0$ on $\Gamma$ we see that also $F_{3}:=h_{3}({\rm
det}(A_{h}))^{-1}\left\langle A_{3},\nabla_{x}\right\rangle$ is a tangential
vector field with respect to $\Gamma$. The above computations show that
$\\{F_{1},F_{2},F_{3}\\}:=\\{\left\langle A_{1},\nabla_{x}\right\rangle,\
\left\langle A_{2},\nabla_{x}\right\rangle,\ h_{3}\left\langle
A_{3},\nabla_{x}\right\rangle\\}$ forms (near $\Gamma$) a local basis of
tangential vector fields with respect to $\Gamma$.

We define

$\displaystyle H^{(m,0,0)}(G\times S\times I^{\circ};\Gamma):=L^{2}(S\times
I,H^{m}(G;\Gamma))$ $\displaystyle:=\\{\psi\in L^{2}(G\times S\times I)|\
F^{\alpha}\psi\in L^{2}(G\times S\times I)\ {\rm for}\ |\alpha|\leq m\\}$

where $F^{\alpha}:=F_{1}^{\alpha_{1}}F_{2}^{\alpha_{2}}F_{3}^{\alpha_{3}}$ and
$F_{1},\ F_{2},\ F_{3}$ are tangential vector fields with respect to $\Gamma$.
Similarly we define tangential Sobolev spaces $H^{(m,0,0)}(G\times S\times
I^{\circ};\Gamma_{\pm})$ and $H^{(m,0,0)}(D_{+,a,b,}\times S\times
I^{\circ};D_{a,b,0}\times S\times I)$. Note that $H^{(m,0,0)}(G\times S\times
I^{\circ};\Gamma)\subset L^{2}(S\times I,H^{m}_{\rm loc}(G))$. Moreover, we
find that $\psi\in H^{(m,0,0)}(G\times S\times I^{\circ};\Gamma)$ if and only
if locally

$\displaystyle\left\|Z^{\alpha}(\psi\circ
h^{-1})\right\|_{L^{2}(D_{+,a,b}\times S\times I)}<\infty\ {\rm for\ all}\
|\alpha|\leq m$ (228)

where
$Z^{\alpha}:=\partial_{z_{1}}^{\alpha_{1}}\partial_{z_{2}}^{\alpha_{2}}(z_{3}\partial_{z_{3}})^{\alpha_{3}}$
for $\alpha=(\alpha_{1},\alpha_{2},\alpha_{3})\in\mathbb{N}_{0}^{3}$. The
condition (228)means that locally $\phi=\psi\circ h^{-1}\in
H^{(m,0,0)}(D_{+,a,b,}\times S\times I^{\circ};D_{0,a,b}\times S\times I)$.
$H^{(m,0,0)}(G\times S\times I^{\circ};\Gamma)$ can be equipped with a natural
norm by glueing together local patches. Actually, it can be equipped with an
inner product and it is a Hilbert space. We omit details.

###### Example 6.2

In the case of open ball $G=B(0,1)$ we can choose (locally) in a neighbourhood
$e_{3}=(0,0,1)$

$\displaystyle h(x)=(x_{1},x_{2},1-|x|^{2}).$

The local inverse of $h$ is
$h^{-1}(z)=(z_{1},z_{2},\sqrt{1-z_{3}-z_{1}^{2}-z_{2}^{2}})$. We have

$\displaystyle\partial_{1}h=(1,0,-2x_{1}),\ \partial_{2}h=(0,1,-2x_{2}),\
\partial_{3}h=(0,0,-2x_{3})$

and so (recall that in this case $\nu(x)=x$),

$\displaystyle
A_{1}=(\partial_{2}h_{2}\partial_{3}h_{3}-\partial_{2}h_{3}\partial_{3}h_{2},\partial_{1}h_{2}\partial_{3}h_{3}-\partial_{1}h_{3}\partial_{3}h_{2},\partial_{1}h_{2}\partial_{2}h_{3}-\partial_{1}h_{3}\partial_{2}h_{2})=2(-x_{3},0,x_{1}),$
$\displaystyle
A_{2}=(\partial_{3}h_{1}\partial_{2}h_{3}-\partial_{3}h_{3}\partial_{2}h_{1},\partial_{1}h_{1}\partial_{3}h_{3}-\partial_{1}h_{3}\partial_{3}h_{1},\partial_{2}h_{1}\partial_{1}h_{3}-\partial_{2}h_{3}\partial_{1}h_{1})=2(0,-x_{3},x_{2}),$
$\displaystyle
A_{3}=(\partial_{2}h_{1}\partial_{3}h_{2}-\partial_{2}h_{2}\partial_{3}h_{1},\partial_{3}h_{1}\partial_{1}h_{2}-\partial_{3}h_{2}\partial_{1}h_{1},\partial_{1}h_{1}\partial_{2}h_{2}-\partial_{1}h_{2}\partial_{2}h_{1})=(0,0,1).$

Hence the (local) basis of tangential vector fields near $\Gamma$ with respect
to $\Gamma$ is $\\{-x_{3}\partial_{x_{1}}+x_{1}\partial_{x_{3}},\
-x_{3}\partial_{x_{2}}+x_{2}\partial_{x_{3}},\
(1-|x|^{2})\partial_{x_{3}}\\}$.

The below Example 6.5 verifies that the higher order tangential regularity is
not necessarily valid for the present transport problems. We need the next
lemma.

###### Lemma 6.3

Let $G=B(0,1)$ and let $k\in\mathbb{N}$ and $q>0$. The integral

$\displaystyle\int_{G\times S}{{(\omega\cdot x)^{k}}\over{((\omega\cdot
x)^{2}+1-|x|^{2})^{q}}}d\omega dx$ (229)

convergences if and only if $k-2q>-3$.

###### Proof.

Noting that $\nu(y)=y,\ \tau_{-}(y,\omega)=2|\omega\cdot y|$ and that on
$\Gamma_{-}^{\prime}$

$\displaystyle(\omega\cdot(y+t\omega))^{2}+1-|y+t\omega|^{2}=(\omega\cdot
y)^{2}$

we have by Lemma 4.1

$\displaystyle\int_{G\times S}{{(\omega\cdot x)^{k}}\over{((\omega\cdot
x)^{2}+1-|x|^{2})^{q}}}d\omega
dx=\int_{\Gamma_{-}^{\prime}}\int_{0}^{2|\omega\cdot
y|}{{(\omega\cdot(y+t\omega))^{k}}\over{((\omega\cdot(y+t\omega))^{2}+1-|y+t\omega|^{2})^{q}}}|\omega\cdot
y|dtd\sigma(y)d\omega$
$\displaystyle=\int_{\Gamma_{-}^{\prime}}\int_{0}^{2|\omega\cdot
y|}{{(\omega\cdot y+t)^{k}}\over{(\omega\cdot y)^{2q}}}|\omega\cdot
y|dtd\sigma(y)d\omega$ $\displaystyle=\sum_{j=0}^{k}{k\choose
j}\int_{\Gamma_{-}^{\prime}}\int_{0}^{2|\omega\cdot y|}{{(\omega\cdot
y)^{j}t^{k-j}}\over{(\omega\cdot y)^{2q}}}|\omega\cdot y|dtd\sigma(y)d\omega$
$\displaystyle=\sum_{j=0}^{k}{k\choose
j}{1\over{k-j+1}}\int_{\Gamma_{-}^{\prime}}{{(\omega\cdot y)^{j}(2|\omega\cdot
y|)^{k-j+1}}\over{(\omega\cdot y)^{2q}}}|\omega\cdot y|d\sigma(y)d\omega$
$\displaystyle=\sum_{j=0}^{k}{k\choose
j}{1\over{k-j+1}}2^{k-j+1}(-1)^{j-2q}\int_{\Gamma_{-}^{\prime}}|\omega\cdot
y|^{k-2q+2}d\sigma(y)d\omega$ (230)

where we additionally used that $|\omega\cdot y|=-\omega\cdot y$ on
$\Gamma_{-}^{\prime}$.

Consider the integral of the form $\int_{\Gamma_{-}^{\prime}}|\omega\cdot
y|^{s}d\sigma(y)d\omega$. Let

$\displaystyle\Gamma_{-,\omega}^{\prime}:=\\{y\in\partial G|\ \omega\cdot
y<0\\},\ S_{\tau,\omega}:=\\{y\in\partial G|\ \omega\cdot y=\tau\\}.$

Note that each $\Gamma_{-,\omega}$ is a hemisphere and $S_{\tau,\omega}$ is a
circle whose radius is $\sqrt{1-\tau^{2}}$. Moreover, let
$d\ell_{\tau,\omega}$ be the differential path length on $S_{\tau,\omega}$. In
virtue of the Fubin’s Theorem (see [48], section 3.2)

$\displaystyle\int_{\Gamma_{-}^{\prime}}|\omega\cdot
y|^{s}d\sigma(y)d\omega=\int_{S}\int_{\Gamma_{-,\omega}}|\omega\cdot
y|^{s}d\sigma(y)d\omega=\int_{S}\int_{0}^{1}\int_{S_{\tau,\omega}}|\omega\cdot
y|^{s}d\ell_{\tau,\omega}(y){1\over{\sqrt{1-\tau^{2}}}}d\tau d\omega$
$\displaystyle=\int_{S}\int_{0}^{1}\int_{S_{\tau,\omega}}\tau^{s}{1\over{\sqrt{1-\tau^{2}}}}d\ell_{\tau,\omega}(y)d\tau
d\omega=2\pi\int_{S}\int_{0}^{1}\tau^{s}\sqrt{1-\tau^{2}}{1\over{\sqrt{1-\tau^{2}}}}d\tau
d\omega=8\pi\int_{0}^{1}\tau^{s}d\tau$ (231)

and so the integral $\int_{\Gamma_{-}^{\prime}}|\omega\cdot
y|^{s}d\sigma(y)d\omega$ convergences if and only if $s>-1$. Hence the
integral (6.2) convergences if and only if

$\displaystyle k-2q+2>-1\Leftrightarrow k-2q>-3$

as desired. ∎

###### Remark 6.4

In the previous lemma it is not essential that $k\in\mathbb{N}$ since the
integral $\int_{0}^{2|\omega\cdot y|}(\omega\cdot y+t)^{k}dt$ can be computed
for any $k\in\mathbb{R}$. However, the computations above are sufficient for
our needs. Especially we find that the integral $\int_{G\times
S}{1\over{((\omega\cdot x)^{2}+1-|x|^{2})^{q}}}d\omega dx$ convergences if and
only if $q<{3\over 2}$. Since $|\omega\cdot y+t|\leq|\omega\cdot y|+t$ we also
see by the above proof that the integral $\int_{G\times S}{{|\omega\cdot
x|^{k}}\over{((\omega\cdot x)^{2}+1-|x|^{2})^{q}}}d\omega dx$ convergences if
$k-2q>-3$.

###### Example 6.5

Let $G=B(0,1)\subset\mathbb{R}^{3}$ and consider the mono-kinetic problem
(given in [47, Example 7.4])

$\displaystyle\omega\cdot\nabla\psi+\psi=1,\ \psi_{|\Gamma_{-}}=0.$

The solution of the problem is

$\displaystyle\psi=1-e^{-t(x,\omega)},$

where

$\displaystyle t(x,\omega)=x\cdot\omega+\sqrt{(x\cdot\omega)^{2}+1-|x|^{2}}.$

We find that

$\displaystyle{\frac{\partial\psi}{\partial
x_{j}}}=e^{-t(x,\omega)}{\frac{\partial t}{\partial
x_{j}}}=e^{-t(x,\omega)}\omega_{j}+e^{-t(x,\omega)}{{(x\cdot\omega)\omega_{j}-x_{j}}\over{\big{(}(x\cdot\omega)^{2}+1-|x|^{2}\big{)}^{1/2}}}$

and then

$\displaystyle\nabla_{x}\psi=e^{-t(x,\omega)}\omega+e^{-t(x,\omega)}{{(x\cdot\omega)\omega-x}\over{\big{(}(x\cdot\omega)^{2}+1-|x|^{2}\big{)}^{1/2}}}.$

Let $F_{j}=\left\langle A_{j},\nabla_{x}\right\rangle,\ j=1,2.$ be the basis
tangent vectors deduced in Example 6.2. Recall that

$\displaystyle A_{1}(x)=2(-x_{3},0,x_{1}),\ A_{2}(x)=2(0,-x_{3},x_{2}).$

We find that

$\displaystyle
F_{1}\psi=2\left\langle(-x_{3},0,x_{1}),e^{-t(x,\omega)}\omega\right\rangle$
$\displaystyle+2\left\langle(-x_{3},0,x_{1}),e^{-t(x,\omega)}{{(x\cdot\omega)\omega-x}\over{\big{(}(x\cdot\omega)^{2}+1-|x|^{2}\big{)}^{1/2}}}\right\rangle$
$\displaystyle=2e^{-t(x,\omega)}(-x_{3}\omega_{1}+x_{1}\omega_{3})$
$\displaystyle+2{{e^{-t(x,\omega)}}\over{\big{(}(x\cdot\omega)^{2}+1-|x|^{2}\big{)}^{1/2}}}\big{(}(x\cdot\omega)(-x_{3}\omega_{1}+x_{1}\omega_{3})-(-x_{1}x_{3}+x_{3}x_{1})\big{)}$
$\displaystyle=2e^{-t(x,\omega)}u_{1}+2e^{-t(x,\omega)}u_{2}$

where

$\displaystyle u_{1}:=(-x_{3}\omega_{1}+x_{1}\omega_{3}),\
u_{2}:={1\over{\big{(}(x\cdot\omega)^{2}+1-|x|^{2}\big{)}^{1/2}}}(x\cdot\omega)(-x_{3}\omega_{1}+x_{1}\omega_{3}).$

Furthermore,

$\displaystyle F_{1}^{2}\psi=F_{1}(F_{1}\psi)=\left\langle
A_{1},\nabla_{x}(F_{1}\psi)\right\rangle=2\left\langle
A_{1},\nabla_{x}(e^{-t(x,\omega)}u_{1}+e^{-t(x,\omega)}u_{2})\right\rangle$
$\displaystyle=2\left\langle
A_{1},e^{-t(x,\omega)}\nabla_{x}u_{1}-u_{1}e^{-t(x,\omega)}\nabla_{x}t\right\rangle+2\left\langle
A_{1},e^{-t(x,\omega)}\nabla_{x}u_{2}-u_{2}e^{-t(x,\omega)}\nabla_{x}t\right\rangle$
$\displaystyle=:2U_{1}+2U_{2}$

where

$\displaystyle U_{1}:=\left\langle
A_{1},e^{-t(x,\omega)}\nabla_{x}u_{1}-u_{1}e^{-t(x,\omega)}\nabla_{x}t\right\rangle,$
$\displaystyle U_{2}:=\left\langle
A_{1},e^{-t(x,\omega)}\nabla_{x}u_{2}-u_{2}e^{-t(x,\omega)}\nabla_{x}t\right\rangle.$

Since $A_{1}=(-x_{3},0,x_{1}),\ \nabla_{x}u_{1}=(\omega_{3},0,-\omega_{1})$
and
$\nabla_{x}t=\omega+{{(x\cdot\omega)\omega-x}\over{\big{(}(x\cdot\omega)^{2}+1-|x|^{2}\big{)}^{1/2}}}$
we have for $U_{1}$

$\displaystyle
U_{1}=-e^{-t(x,\omega)}(\omega_{1}x_{1}+\omega_{3}x_{3})-u_{1}e^{-t(x,\omega)}\big{(}(-\omega_{1}x_{3}+\omega_{3}x_{1})+{{(\omega\cdot
x)(-\omega_{1}x_{3}+x_{1}\omega_{3})}\over{((x\cdot\omega)^{2}+1-|x|^{2})^{1/2}}}\big{)}.$

Hence

$\displaystyle|U_{1}|\leq(\left\|x\right\|+\left\|x\right\|^{2})+\left\|x\right\|^{2}{{|\omega\cdot
x|}\over{((x\cdot\omega)^{2}+1-|x|^{2})^{1/2}}}$ (232)

where we used that $e^{-t(x,\omega)}\leq 1$,
$|x_{1}\omega_{1}+x_{3}\omega_{3}|\leq\left\|x\right\|$ and
$|-x_{3}\omega_{1}+x_{1}\omega_{3}|\leq\left\|x\right\|$.

Furthermore,

$\displaystyle(\nabla_{x}u_{2})(x,\omega)$
$\displaystyle={1\over{\big{(}(x\cdot\omega)^{2}+1-|x|^{2}\big{)}^{1/2}}}\big{(}(-x_{3}\omega_{1}+x_{1}\omega_{3})\omega+(x\cdot\omega)(\omega_{3},0,-\omega_{1})\big{)}$
$\displaystyle-(x\cdot\omega)(-x_{3}\omega_{1}+x_{1}\omega_{3}){{(\omega\cdot
x)\omega-x}\over{\big{(}(x\cdot\omega)^{2}+1-|x|^{2}\big{)}^{3/2}}}$

and so (recall that
$\nabla_{x}t=\omega+{{(x\cdot\omega)\omega-x}\over{\big{(}(x\cdot\omega)^{2}+1-|x|^{2}\big{)}^{1/2}}}$)

$\displaystyle
U_{2}=e^{-t(x,\omega)}\big{(}\left\langle(-x_{3},0,x_{1}),\nabla_{x}u_{2}\right\rangle-
u_{2}\left\langle(-x_{3},0,x_{1}),\nabla_{x}t\right\rangle$
$\displaystyle=e^{-t(x,\omega)}(-x_{3}\omega_{1}+x_{1}\omega_{3})^{2}{1\over{\big{(}(x\cdot\omega)^{2}+1-|x|^{2}\big{)}^{1/2}}}$
$\displaystyle-e^{-t(x,\omega)}(x\cdot\omega)(\omega_{3}x_{3}+\omega_{1}x_{1}){1\over{\big{(}(x\cdot\omega)^{2}+1-|x|^{2}\big{)}^{1/2}}}$
$\displaystyle-e^{-t(x,\omega)}(x\cdot\omega)(-x_{3}\omega_{1}+x_{1}\omega_{3}){{(\omega\cdot
x)(-x_{3}\omega_{1}+x_{1}\omega_{3})}\over{\big{(}(x\cdot\omega)^{2}+1-|x|^{2}\big{)}^{3/2}}}$
$\displaystyle-e^{-t(x,\omega)}{1\over{\big{(}(x\cdot\omega)^{2}+1-|x|^{2}\big{)}^{1/2}}}(\omega\cdot
x)(-x_{3}\omega_{1}+x_{1}\omega_{3})^{2}$
$\displaystyle-e^{-t(x,\omega)}{1\over{\big{(}(x\cdot\omega)^{2}+1-|x|^{2}\big{)}^{1/2}}}(x\cdot\omega)^{2}(-x_{3}\omega_{1}+x_{1}\omega_{3})^{2}{{1}\over{\big{(}(x\cdot\omega)^{2}+1-|x|^{2}\big{)}^{1/2}}}$
$\displaystyle=U_{2,1}+U_{2,2}+U_{2,3}+U_{2,4}+U_{2,5}.$

By Lemma 6.3 we find that(we omit the details)

$\displaystyle U_{1},\ U_{2,j}\in L^{2}(G\times S\times I),\ j=1,2,3,4,5.$
(233)

Hence $F_{1}^{2}\psi\in L^{2}(G\times S\times I)$.

The third order derivative $F_{1}^{3}\psi$ contains terms like

$\displaystyle
e^{-t(x,\omega)}(x\cdot\omega)(-x_{3}\omega_{1}+x_{1}\omega_{3})^{3}{{1}\over{\big{(}(x\cdot\omega)^{2}+1-|x|^{2}\big{)}^{3/2}}},$
$\displaystyle
e^{-t(x,\omega)}(x\cdot\omega)^{3}(-x_{3}\omega_{1}+x_{1}\omega_{3})^{3}{{1}\over{\big{(}(x\cdot\omega)^{2}+1-|x|^{2}\big{)}^{5/2}}}$

and we conjecture that $F_{1}^{3}\psi\not\in L^{2}(G\times S\times I)$.

The above Example 6.5 suggests that the tangential regularity might not be
valid for some classes of transport problems considered in this paper.
Actually, we see that the techniques (for tangential regularity of problems
with constant multiplicity) by [43] do not seem to work since after the above
localization (section 6.1), for example, the problem

$\displaystyle\omega\cdot\nabla_{x}\psi+\Sigma\psi-K_{r}\psi=f,\
\psi_{|\Gamma_{-}}=g,\ \psi(.,.,E_{m})=0$ (234)

becomes

$\displaystyle\tilde{b}(x,\omega)\cdot\nabla_{z}\phi+\tilde{\Sigma}\phi-\widetilde{K}_{r}\phi=\tilde{f},\
\phi_{|\widetilde{\Gamma}_{-}}=\tilde{g},\ \phi(.,.,E_{m})=0.$ (235)

The equation (235) is equivalent to

$\displaystyle\tilde{b}_{3}(z,\omega){\frac{\partial\phi}{\partial
z_{3}}}=-\tilde{b}_{1}(z,\omega){\frac{\partial\phi}{\partial
z_{1}}}-\tilde{b}_{2}(x,\omega){\frac{\partial\phi}{\partial
z_{2}}}-\tilde{\Sigma}\phi+\widetilde{K}_{r}\phi+\tilde{f}$ (236)

Here (we decompose $z=(z^{\prime},z_{3})$)

$\displaystyle\tilde{b}_{3}(z,\omega)=\tilde{b}_{3}(z^{\prime},0,\omega)+z_{3}G(z^{\prime},z_{3})$

where
$\tilde{b}_{3}(z^{\prime},0,\omega)=(\omega\cdot\nabla_{x}h_{3})(h^{-1}(z^{\prime},0))$
is not (necessarily) non-zero in a neighbourhood of
$0\in\\{z\in\mathbb{R}^{3}|\ z_{3}=0\\}$. Hence the crucial representation
[43, (19)] is not necessarily valid.

###  6.3. A result for tangential regularity

For some problems with variable multiplicity higher order tangential
regularity can be shown when the data is appropriately restricted. One such
result can be found in [39] which we bring up shortly below.

We need the following spaces. Using the notations of [39] let
$D_{\infty}(G\times S,\Gamma_{-}^{\prime})$ be the subspace of
$C^{\infty}(\overline{G}\times S)$ such that ${\rm supp}\
(u)\cap(\Gamma_{-}^{\prime}\cup\Gamma_{0}^{\prime})=\emptyset$ for $u\in
D_{\infty}(G\times S,\Gamma_{-}^{\prime})$. Moreover, let $X_{k}(G\times
S,\Gamma_{-}^{\prime})$ be the completion of $D_{\infty}(G\times
S,\Gamma_{-}^{\prime})$ with respect to $H^{(k,0)}(G\times S)$-norm where the
space $H^{(k,0)}(G\times S)$ is similarly defined as the spaces
$H^{(m_{1},m_{2},m_{3})}(G\times S\times I^{\circ})$ in section 2.2.

Consider the following transport problem with homogeneous inflow boundary
values (that is, $g=0$)

$\displaystyle T\psi=a{\frac{\partial\psi}{\partial
E}}+\omega\cdot\nabla_{x}\psi+\Sigma\psi-K_{r}\psi=f,\ \psi_{|\Gamma_{-}}=0,\
\psi(.,E_{m})=0.$ (237)

For simplicity, we assume that $G$ is of the form

$\displaystyle G=\\{x\in U|\ r(x)>0\\}$

where

$\displaystyle\overline{G}\subset U\subset\circ\mathbb{R}^{3},\ r\in
C^{1}(U),\ \partial G=\\{x\in U|\ r(x)=0\\},\ \nabla_{x}r(x)\not=0,\
x\in\partial G,$ $\displaystyle\omega^{*}\ ({\rm Hess}\ r(x))\ \omega\ {\rm
is\ negative}\ {\rm for}\ (x,\omega)\in\Gamma_{0}.$ (238)

Then $\nu=-{{\nabla_{x}r(x)}\over{|\nabla_{x}r(x)|}}$ and we assume that
$-\nabla_{x}r$ is pointing outwards on $\partial G$. Using again the notations
by [39] we take

$\displaystyle A_{j}(x,\omega)=\omega_{j},\
\nu(x,\omega)=\nu(x)=-{{\nabla_{x}r(x)}\over{|\nabla_{x}r(x)|}},\
A_{b}(x,\omega):=\sum_{j=1}^{3}A_{j}(x,\omega)\nu_{j}(x,\omega)=(\omega\cdot\nu),$
$\displaystyle\Omega:=G\times S,\ r(x,\omega):=r(x),\
h(x,\omega)=h_{\pm}(x,\omega):=(\omega\cdot\nabla_{x}r)(x),$ $\displaystyle
M(x,\omega):=\begin{cases}\mathbb{R}&{\rm on}\ \Gamma_{+}\cup\Gamma_{0}\\\ 0,\
&{\rm on}\ \Gamma_{-}\end{cases}.$

We see that

$\displaystyle
A_{r}(x,\omega):=\sum_{j=1}^{3}(\partial_{x_{j}}r)A_{j}(x,\omega)=\nabla_{x}r(x)\cdot\omega=-h(x,\omega)A(x,\omega)$

where

$\displaystyle A(x,\omega):=-1.$

Furthermore,

$\displaystyle
A_{h}(x,\omega):=\sum_{j=1}^{3}(\partial_{x_{j}}h)A_{j}(x,\omega)=\omega\cdot\nabla_{x}h=\omega\cdot\nabla_{x}(\nabla_{x}r(x)\cdot\omega))$
$\displaystyle=\sum_{k=1}^{3}\omega_{k}\partial_{k}(\omega\cdot\nabla_{x}r)=\sum_{k=1}^{3}\sum_{j=1}^{3}\omega_{k}\omega_{j}\partial_{k}\partial_{j}r=\omega^{*}\
({\rm Hess}\ r(x))\ \omega.$

and so we find by (6.3) that $A(x,\omega)$ and $A_{h}(x,\omega)$ are
simultaneously negative on $\Gamma_{0}$ which is a basic assumption of [39].
This assumption enables to compensate the degeneracy of the boundary matrix
$A_{b}(x,\omega)$ on $\Gamma_{0}$ (see [39, Lemma 3.4]).

We conjecture that the following regularity result is true:

Suppose that $K_{r}=K_{r}^{2}$ (recall section 3.1) and that the assumptions
(27), (3.1) are valid. Furthermore, assume that

$\displaystyle-{\frac{\partial a}{\partial E}}(x,E)\geq q_{1}>0,\ {\rm a.e.},$
(239) $\displaystyle-a(x,E_{0})\geq q_{3}>0,\ -a(x,E_{m})\geq q_{3}>0,\ {\rm
a.e.}$ (240) $\displaystyle|a(x,E)|\geq q_{4}>0\ {\rm a.e.}.$ (241)

Let $k\in\mathbb{N}_{0}$ and assume that

$\displaystyle a\in W^{\infty,(k,1)}(G\times I),\ \Sigma\in
W^{\infty,(k,0,1)}(G\times S\times I^{\circ}),$ (242)
$\displaystyle\int_{S^{\prime}}\left\|\sigma^{2}\right\|_{W^{\infty,(k,1)}(G\times
I)}(.,\omega^{\prime},\omega,.)d\omega^{\prime}\leq M_{1},$
$\displaystyle\int_{S^{\prime}}\left\|\sigma^{2}\right\|_{W^{\infty,(k,1)}(G\times
I)}(.,\omega^{\prime},\omega,.)d\omega^{\prime}\leq M_{2},$ (243)

for a.e. $\omega\in S$. Let

$\displaystyle f\in C(I,X_{k+1}(G\times S,\Gamma_{-}^{\prime})).$ (244)

Then the solution of the problem (237) belongs to $C^{1}(I,X_{k}(G\times
S,\Gamma_{-}^{\prime})$.

For the proof see [39, Proposition 5.2, Theorem 5.5]. The result, however does
not follow straightforwardly from [39] since the collision operator $K_{r}$
(which is a partial integral operator) is included. The presence of $S$ also
causes some extra handling.

It is possible to choose the function $h$ and $h_{\pm}$ in a more subtle way.
Note that the above result is not in contradiction with Example 6.5 since we
assume that $f\in C(I,X_{k+1}(G\times S,\Gamma_{-}^{\prime}))$ which is not
valid for data $f=1$.

###### Example 6.6

For the unit ball $G=B(0,1)\subset\mathbb{R}^{3}$ we may choose
$r(x)=1-|x|^{2}$. Then $\nu(x)=x$, $h(x,\omega)=-2(\omega\cdot x)$,
$\omega^{*}\ ({\rm Hess}\ r(x))\ \omega=A_{h}(x,\omega)=-2|\omega|^{2}=-2$.
Hence the above assumptions for $r$ and $h=h_{\pm}$ are valid.

###### Remark 6.7

In certain _weighted co-normal Sobolev spaces_ higher order (tangential)
regularity can be achieved with less restrictive data. Here weighted co-normal
Sobolev spaces are spaces of the form (see [40])

$\displaystyle X^{k}_{s,r}(G\times S\times
I^{\circ};\Gamma):=\cap_{j=0}^{k}\theta_{+}^{s+k-j}\theta_{-}^{r+k-j}H^{j}(G\times
S\times I^{\circ};\Gamma)$

equipped with the natural inner product

$\displaystyle\left\langle\psi,v\right\rangle_{X^{k}_{s,r}(G\times S\times
I;\Gamma})=\sum_{j=0}^{k}\left\langle\theta_{+}^{-s-k+j}\theta_{-}^{-r-k+j}\psi,\theta_{+}^{-s-k+j}\theta_{-}^{-r-k+j}v\right\rangle_{H^{j}(G\times
S\times I^{\circ};\Gamma)}$

where $k\in\mathbb{N}_{0}$ and $s,\ r\in\mathbb{R}$ and where $\theta_{+},\
\theta_{-}$ are appropriate weight functions. The weight functions are
emerging from the representations of $G$ and $\Gamma_{\pm}$ (that is, they
emerge from $r$ and $h_{\pm}$ when $G=\\{x\in U|\
r(x)>0\\},\Gamma_{\pm}=\\{(y,\omega)\in\Gamma|\ h_{\pm}(y,\omega)>0\\}$).
Under relevant assumptions the solutions of the problem

$\displaystyle T\psi=a{\frac{\partial\psi}{\partial
E}}+\omega\cdot\nabla_{x}\psi+\Sigma\psi-K_{r}\psi=f,\ \psi_{|\Gamma_{-}}=0,\
\psi(.,E_{m})=0$ (245)

can be shown to be for some $s,\ r>C(k)>0$ in

$\displaystyle\psi\in X^{k}_{-s,r}(G\times S\times I^{\circ};\Gamma)$ (246)

(see [40, Theorem 2.4]). The condition (246) means that

$\displaystyle\theta_{+}^{s-k+j}\theta_{-}^{-r-k+j}\psi\in H^{j}(G\times
S\times I^{\circ};\Gamma),\ j=0,...,k.$

The factor $\theta_{+}^{s-k+j}\theta_{-}^{-r-k+j}$ ”eliminates singularities”.

###### Remark 6.8

In the theory of PDEs one knows Lopatinski-Shapiro theory and related more
general _Kreiss-Lopatinski theory_ within which the existence, stability and
regularity of solutions can be analysed as well (see e.g. [58, Chapters 12,
13] for elliptic operators and for symmetric hyperbolic systems e.g. [34],
[35]). Conventionally sufficient algebraic criteria for Kreiss-Lopatinski
conditions have been formulated by using the symbols of PDE operators but
there exist also certain apriori estimates which guarantee these conditions.
After exponential shift $\phi=e^{CE}\psi$ the estimates like (186), (189) may
be useful in the study of uniform Kreiss-Lopatinski conditions (e.g. [34,
Chapter 2]). We omit here all formulations.

##  7\. Remarks on transport operators from microlocal perspective

_Transport operator interpreted as a pseudo-differential operator_. Consider
the equation

$\displaystyle T\psi:=-a(x,E){\frac{\partial\psi}{\partial E}}$
$\displaystyle+c(x,E)\Delta_{S}\psi+d(x,\omega,E)\cdot\nabla_{S}\psi$
$\displaystyle+\omega\cdot\nabla_{x}\psi+\Sigma(x,\omega,E)\psi-K_{r}\psi=f$
(247)

where we assume (for simplicity) that the restricted collision operator
$K_{r}$ is of the form

$\displaystyle(K_{r}\psi)(x,\omega,E)$
$\displaystyle=\int_{S^{\prime}}\int_{I^{\prime}}\sigma^{1}(x,\omega^{\prime},\omega,E^{\prime},E)\psi(x,\omega^{\prime},E^{\prime})dE^{\prime}d\omega^{\prime}$
$\displaystyle+\int_{S^{\prime}}\sigma^{2}(x,\omega^{\prime},\omega,E)\psi(x,\omega^{\prime},E)d\omega^{\prime}=:(K_{r}^{1}\psi)(x,\omega,E)+(K_{r}^{2}\psi)(x,\omega,E).$
(248)

Denote more shortly $U:=G\times S\times I^{\circ},\ z:=(x,\omega,E)\in U$. Let
$\zeta:=(\xi,\eta,\tau)$ be the cotangent variable on $T_{z}^{*}(U)$ .
Assuming that the cross-sections $\sigma^{1},\ \sigma^{2}$ obey certain
criteria the partial integral operator $K_{r}$ can be expressed as a sum
$K_{r}=K_{r}(z,D)+R$ where $K_{r}(z,D)$ is a pseudo-differential operator
whose symbol, say $k_{r}(z,\zeta)$, belongs to the _classical symbol class_
$S_{1,0}^{0}(U)$ and $R$ is a smoothing operator (cf. [27, Chapter 7]). In the
sequel the operator $R$ is meaningless and so we omit it. Hence assuming that
the coefficients are smooth the operator $T$ is a pseudo-differential operator
$T(z,D)$ on $U$ whose symbol

$\displaystyle t_{0}(z,\zeta)={\rm i}\
a(x,E)\tau-c(x,E)\sum_{i,j}^{2}g^{ij}\eta_{i}\eta_{j}-{\rm i}\
d(z)\cdot\eta-{\rm i}\ (\omega\cdot\xi)+\Sigma(z)-k_{r}(z,\zeta)$

belongs to the class $S_{1,0}^{2}(U)$ where $g^{ij}$ are the elements of the
inverse matrix
$\big{(}\left\langle{{\partial}\over{\partial\omega_{i}}}_{|\omega},{{\partial}\over{\partial\omega_{j}}}_{|\omega}\right\rangle\big{)}^{-1}.$
The principal symbol of $T(z,D)$ is

$\displaystyle t_{0}(z,\zeta)=-c(x,E)\sum_{i,j}^{2}g^{ij}\eta_{i}\eta_{j}$

or

$\displaystyle t_{0}(z,\zeta)={\rm i}\ a(x,E)\tau-{\rm i}\ d(z)\cdot\eta-{\rm
i}\ (\omega\cdot\xi)$

depending on whether $c\equiv 0$ or not. We remark that the principal symbol
$t_{0}(z,\zeta)$ is well-defined on the cotangent bundle $T^{*}(U)$ (that is,
it is defined in the coordinate free way). (Above $\xi_{j}=(\xi\cdot
dx)(\partial_{x_{j}}),\ \eta_{j}=(\eta\cdot d\omega)(\partial_{\omega_{j}}),\
\tau=(\tau\cdot dE)(\partial_{E})$ for $\zeta\cdot dz=\xi\cdot dx+\eta\cdot
d\omega+\tau dE\in T^{*}(U)$). Denote the differential part by
$P(z,D\psi:=-a{\frac{\partial\psi}{\partial
E}}+c\Delta_{S}\psi+d\cdot\nabla_{S}\psi+\omega\cdot\nabla_{x}\psi+\Sigma\psi$.
Then $T(z,D)=P(z,D-K_{r}(z,D)$. We find that $P(z,D)$ is properly supported
(as a partial differential operator). Moreover, $K_{r}(z.D)$ is properly
supported if $K_{r}^{1}=0$.

_Wavefront set in the interior_. The transport problems have a limited
regularity of solutions (in spite of regularity of data and domains). There
are some general methods along which one is able to study the formation and
location of singularities. One of them is the microlocal study of
distributional solutions. The basic concepts of microlocal analysis are the
wavefront sets and propagation of singularities along bicharacteristic curves.
We shortly recall some basics of them.

To start with we define the wavefront set in the case where the domain, say
$V$, is an open subset of $\mathbb{R}^{N}$. Let $(x_{0},\xi_{0})\in
T^{*}(V)\sim V\times\mathbb{R}^{N},\ \xi_{0}\not=0$. An open neighbourhood
$\mathcal{C}_{\xi_{0}}\subset T_{x_{0}}^{*}(V)$ of $\xi_{0}$ is called a
_conical neighbourhood_ if $\xi\in\mathcal{C}_{\xi_{0}}$ implies that
$t\xi\in\mathcal{C}_{\xi_{0}}$ for all $t>0$. The $C^{\infty}$ _wavefront set
$WF(u)$_ of a distribution $u\in D^{\prime}(V)$ is the closed subset of
$T^{*}(V)$ defined (via its complement) as follows (e.g. [24], [42] and [15]
(where a lot of results and literature regarding to microlocal analysis and
its consequences are exposed; results are given mainly without proofs):

Let $(x_{0},\xi_{0})\in T^{*}(V),\ \xi_{0}\not=0$. Then $(x_{0},\xi_{0})$ is
not in $WF(u)$ if and only if there exists a function $\varphi\in
C_{0}^{\infty}(V)$ for which $\varphi(x_{0})\not=0$ and an open conical
neighbourhood $\mathcal{C}_{\xi_{0}}$ of $\xi_{0}$, such that for any $N>0$
there exists $C_{N}>0$ such that

$\displaystyle|\mathcal{F}(\varphi u)(\xi)|\leq
C_{N}(1+|\xi|)^{-N},\xi\in\mathcal{C}_{\xi_{0}}$

where $\mathcal{F}$ denotes the Fourier transform.

One of the equivalent definitions of wavefront sets is:

$(x_{0},\xi_{0})\in T^{*}(V),\ \xi_{0}\not=0$ is not in $WF(u)$ if and only if
there exists a symbol $a(x,\xi)\in S_{1,0}^{m}(V)$ such that
$a_{0}(x,\xi)\not=0$ and $a(x,D)(\varphi u)\in C^{\infty}(V)$ where
$a_{0}(x,\xi)$ is the principal symbol of $a(x,\xi)$ and $\varphi$ is as in
the above definition.

A distribution on a manifold (like on $U$) can be defined in various
(coordinate free) ways. The definition of the wavefront set $WF(\psi)\subset
T^{*}(U)$ in the case where the domain is the manifold $U=G\times S\times
I^{\circ}$ can be reverted to the above definition by using pullbacks along
coordinate charts. The other possibility is to employ the equivalent
definition noted above after the definition of $WF(u)\subset T^{*}(V)$. We
omit exact definitions.

It can be shown that

$\displaystyle\pi({WF}(\psi))={\rm sing\ supp}(\psi)$ (249)

where $\pi$ is the natural projection $\pi:T^{*}(U)\to U$ and ${\rm sing\
supp}(\psi)$ is the singular support of $\psi$ (that is, the complement of the
open set where $\psi$ is $C^{\infty}$).

The _characteristic set (or variety)_ of $T:=T(z,D)$ is

$\displaystyle{\rm char}(T):=\\{(z,\zeta\cdot dz)\in T^{*}(U)|\
t_{0}(z,\zeta)=0\\}.$

In the case where $T(z,D)$ is properly supported $T(z,D)\psi$ is defined for
any $\psi\in D^{\prime}(U)$ (recall that $T(z,D)$ is properly supported if
e.g. $K_{r}^{1}=0$). Assuming that $T(z,D)$ is properly supported or
$\psi\in\mathcal{E}^{\prime}(U)$ (here $\mathcal{E}^{\prime}(U)$ is the space
of distributions with compact support) the basic result is

$\displaystyle WF(T\psi)\subset WF(\psi)\subset WF(f)\cup{\rm char}(T)$ (250)

where $f:=T\psi$. The inclusions (250) tell us especially that possible
wavefront points (that is, microlocal singularities) of the distributional
solutions $\psi$ of the equation $T\psi=f$ lie on ${\rm char}(T)$ when $f$ is
smooth.

The _bicharacteristic curve_ of $T(z,D)$ emanating from
$(z_{0},\zeta_{0})\in{\rm char}(T)$ is the solution
$\alpha=(z,\zeta):\Delta\to{\rm char}(T)$ of the Hamiltonian system of
differential equations which in local coordinates is (here
$z^{\prime}={{dz}\over{ds}},\ \zeta^{\prime}={{d\zeta}\over{ds}}$)

$\displaystyle\begin{cases}&z^{\prime}=\nabla_{\zeta}t_{0}(z,\zeta)\\\
&\zeta^{\prime}=-\nabla_{z}t_{0}(z,\zeta),\\\
&(z(0),\zeta(0))=(z_{0},\zeta_{0})\end{cases}.$ (251)

The solutions curves $\alpha(s)=(z(s),\zeta(s)),\ s\in\Delta$ of (251) give
the Hamiltonian flows that is, integral curves of the Hamiltonian vector field
$H_{t_{0}}:=\nabla_{\zeta}t_{0}\cdot{{\partial}\over{\partial
z}}-\nabla_{z}t_{0}\cdot{{\partial}\over{\partial\zeta}}$ with $t_{0}$. The
projection curves $\tilde{\alpha}=\pi\circ\alpha:\Delta\to U$ are called
_characteristic curves_.

The operator $T(z,D)$ is of _principal type_ if its principal symbol
$t_{0}(z,\zeta)$ is homogeneous in $\zeta$ and
$\nabla_{\zeta}(z,\zeta)\not=0,\ \zeta\not=0$ on ${\rm char}(T)$.

Suppose that $T(z,D)$ is properly supported, of principal type and that
$t_{0}(z,\zeta)$ is real valued. Then the bicharacteristic curve preserves
microlocal singularities in the following sense:

Suppose that $\psi\in D^{\prime}(U)$ is a distributional solution of
$T(z,D)\psi=f.$ Furthermore, assume that $(z_{0},\zeta_{0})\in
WF(\psi)\setminus WF(f)$. Then the whole bicharacteristic curve
$\alpha(s)=(z(s),\zeta(s)),\ s\in\Delta$ through $(z_{0},\zeta_{0})$ is
included in $WF(\psi)\setminus WF(f)$ (that is, possible the singularity
$(z_{0},\zeta_{0})$ propagates along the bicharacteristic curve when $f$ is
smooth). This result holds without the assumption ’$T(z,D)$ is properly
supported’ if $\psi$ has a compact support in $U$.

The above definition gives the notion of $C^{\infty}$ wavefront set. There
exists also other type of wavefront sets such as _Sobolev wavefront sets_.
Relevant function spaces in our case would be the anisotropic Sobolev spaces
$H^{(s_{1},s_{2},s_{3})}(G\times S\times I^{\circ})$ (given in section 2.2
above). Basic results for Sobolev wavefront sets have counterparts to the
above formulations.

_Wavefront set up to the boundary_. Consider the initial boundary value
problem

$\displaystyle T(z,D)\psi=f,\ {B\psi}_{|\partial U}=g,\ \psi(.,.,E_{\rm m})=0$
(252)

where $B$ is a partial differential boundary operator. We assume that
_$\partial U$ is non-characteristic for $T(z,D)$_ that is,
$t_{0}(z,n(z))\not=0$ for $z\in(\partial U)_{r}$ where $n(z)$ is the unit
normal vector on the regular part $(\partial U)_{r}$ of $\partial U$. When the
boundary values are included the micro-localization must be studied on the so
called _(compressed) $b$-cotangent bundle_ (e.g. [33], [57, sections 1 and 2],
[42]). Omitting all details we only mention that the $b$-cotangent bundle
${}^{b}T^{*}(\overline{U})$ is the dual bundle of the bundle, say
${}^{b}T(\overline{U})$, formed by the tangential vector fields.

A distribution $\psi\in D^{\prime}(U)$ is _extendable_ if it can be extended
to a distribution $\tilde{\psi}\in D^{\prime}(\mathbb{R}^{3}\times
S\times\mathbb{R})$ and if $\tilde{\psi}_{|\partial U}\in D^{\prime}(\partial
U)$ is well-defined. The relevant microlocal concept to analyse boundary value
problems is the so called $C^{\infty}$ _(compressed) b-wavefront set_
${}^{b}WF(\psi)$ of an extendable distribution $\psi\in D^{\prime}(U)$. We
omit its definition but remark that

${}^{b}WF(\psi)\subset(T^{*}(U)\setminus\\{0\\})\cup(T^{*}(\partial
U)\setminus\\{0\\}),\ T^{*}(\overline{U})\subset\ ^{b}T^{*}(\overline{U}).$

Let $\iota$ be the natural imbedding $T^{*}(\overline{U})\to\
^{b}T^{*}(\overline{U})$. Suppose that $T(z,D)$ is properly supported. The
basic result is that

${}^{b}WF(\psi)\subset\ ^{b}WF(f)\cup WF(g)\cup\widetilde{\rm char}(T)$

where $\widetilde{\rm char}(T):=\iota({\rm char}(T))$ and $f=T\psi$.

Suppose that $T$ is properly supported, of principal type, $t_{0}(z,\zeta)$ is
real valued and that data is smooth. Then wavefront singularities propagate
along the so called _broken bicharacteristic curve_. The broken
bicharacteristic curve $\alpha:\Delta\to\widetilde{\rm char}(T)$ obeys a kind
of broken Hamiltonian system. In the interior $U$ the requirement for the
broken bicharacteristic curve reduces to (251). When the bicharacteristic
curve encounters the boundary $\partial U$ it may reflect or retract back to
$U$ or it may glide on the boundary. For example, the geometric nature of
reflection and gliding are different and so one needs the concept of a broken
bicharacteristic curve.

The above standard assumptions concerning for microlocal analysis are not
necessarily valid for all transport problems. For example,

A. The boundary $\Gamma=(\partial G)\times S\times I^{\circ}$ (and hence
either $\partial U$) is not generally non-characteristic for the transport
operator $T(z,D)$. This implies (among others) that $\iota({\rm
char}(T))\not\subset\ ^{b}T^{*}(U)$ where $\iota$ is the imbedding given
above. That is why the $b$-cotangent bundles ${}^{b}T^{*}(U)$ are not
necessarily sufficient structures to analyse inflow boundary value problems.
For example, $\Gamma$ is not non-characteristic for
$T(z,D)=\omega\cdot\nabla_{x}+\Sigma-K_{r}$ since $t_{0}(z,\nu(x))=-{\rm i}\
\omega\cdot\nu(x)=0$ when $\omega\perp\nu(x)$. Because ${\frac{\partial
t_{0}}{\partial\xi_{j}}}={\rm i}\omega_{j}$ and $\left\|\omega\right\|=1$ we
see that $T(z,D)$ is of principal type. $t_{0}(z,\zeta)$ is not real-valued.

B. The transport operators are not necessarily of principal type. This implies
that the principal part operator, say $T_{0}(z,D)$, does not dominate lower
order pseudo-differential operators. In addition, the Hamiltonian flow may
trap in a compact set $A\subset U$. For example, the principal part of the
operator $T(z,D)=-a{{\partial}\over{\partial
E}}+c\Delta_{S}+d\cdot\nabla_{S}+\omega\cdot\nabla_{x}+\Sigma-K_{r}$ is
$t_{0}(z,\zeta)=-c\sum_{i,j}^{2}g^{ij}\eta_{i}\eta_{j}$. So, $t_{0}(z,\zeta)$
is real-valued. Furthermore, ${\frac{\partial
t_{0}}{\partial\xi_{j}}}={\frac{\partial t_{0}}{\partial\tau}}=0,\ j=1,2,3$
and $({\frac{\partial t_{0}}{\partial\eta_{1}}},{\frac{\partial
t_{0}}{\partial\eta_{2}}})=-c(2g^{11}\eta_{1}+g^{21}\eta_{2}+g^{12}\eta_{2},2g^{22}\eta_{2}+g^{12}\eta_{1}+g^{21}\eta_{1})$.
Hence $T(z,D)$ is not of principal type since ${\rm
char}(T)=\\{(\xi,\eta,\tau)|\ \eta=0\\}$ and so
$\nabla_{\zeta}t_{0}(z,\zeta)=0$ on ${\rm char}(T)$. There exists variant
conditions instead of principal type property. For instance, they assume that
the differential $dt_{0}=(\nabla_{z}t_{0})\cdot dz+(\nabla_{\zeta}t_{0})\cdot
d\zeta$ and the fundamental 1-form $\zeta\cdot dz=\xi\cdot dx+\eta\cdot
d\omega+\tau dE$ are linearly independent on ${\rm char}(T)$. They are not
necessarily valid for transport problems as well.

The assumption that $t_{0}(z,\zeta)$ must be real-valued can be in some cases
avoided by considering $-{\rm i}T(z,D)$ instead of $T(z,D)$. For example, the
principal symbol of $T(z,D)\psi=-{{\partial}\over{\partial
E}}+\omega\cdot\nabla_{x}+\Sigma$ is $t_{0}(z,\zeta)={\rm i}\tau-{\rm
i}(\omega\cdot\xi)$. The principal symbol of $-{\rm i}T(z,D)$ is
$t_{0}(z,\zeta)=-\tau+\omega\cdot\xi$ which is real-valued. By computing a
bicharacteristic curve of the operator $-{\rm i}T(z,D)$ and its projection
along $U$ we see that the corresponding characteristic curve is
$\tilde{\alpha}(s)=(x_{0}-\omega_{0}s,\omega_{0},E_{0}+s),$ as wellknown.

Finally we remark that emerging and propagation of singularities can be
studied by using different methods. For example, in [30] one has considered
(without microlocal methods) the _propagation of discontinuities_ along the
characteristic curves in non-convex domains. Moreover, one shows therein that
the discontinuities emanate at the non-convex part of the (grazing) boundary
$\Gamma$.

##  8\. Discussion

In the global case $G=\mathbb{R}^{3}$ the regularity of solutions of initial
value transport problems may increase according to data (see [53]). In the
case where $G$ is not the whole space the relevant transport problems contain
the inflow boundary condition. This condition causes that the corresponding
boundary value operator has a so called variable multiplicity on $\Gamma$. It
is well-known from the general viewpoint of the first order PDE-systems that
solutions of these kind of initial boundary value problems posses only limited
regularity. In the case of initial inflow boundary value transport problems
the limit of regularity with respect to $x$ (and $\omega$-variables) seems to
be $3/2$ (see Example 4.11 above). In the case where the transport equation
does not contain the slowing down term $a{\frac{\partial\psi}{\partial E}}$
the regularity with respect to $E$-variable is not necessarily limited (see
section 4.1.3 above).

The regularity results exposed in this paper may be largely generalized for
fractional indexes. The relevant spaces within which the results can be
formulated are then fractional anisotropic Sobolev-Slobodevskij spaces
$H^{(s_{1},s_{2},s_{3})}(G\times S\times I^{\circ})$ (cf. Remark 4.10 above).
Under relevant assumptions the solution of the problem (1), (2), (3) lies in
$H^{(s_{1},s_{2},s_{3})}(G\times S\times I^{\circ})$ for certain indexes
$s_{j}>1$. The restricted collision operator $K_{r}$ may be more general than
that of (162) and the assumption (168) is superfluous. The analysis requires
further study and techniques but we think that the scales
$H^{(s_{1},s_{2},s_{3})}(G\times S\times I^{\circ})$ are relevant for more
general formulations.

In the section 5.1 the decomposition of the solution as $\psi=u+L_{-}g,\ u\in
D(\widetilde{P}_{C,0})$ is not necessary for the use of Neumann series in
regularity analysis. For instance, consider the situation given in section 4.2
and assume that the coefficient $a$ is a positive constant. After exponential
shift the problem therein is (here $f_{C}$ and $g_{C}$ are as in section 5.1)

$\displaystyle{\mathbb{P}}_{C}\phi:=-a{\frac{\partial\phi}{\partial
E}}+\omega\cdot\nabla_{x}\phi+(\Sigma+Ca)\phi=f_{C},\
\phi_{|\Gamma_{-}}=g_{C},\ \phi(.,.,E_{m})=0.$

In virtue of (4.2), (4.2) the inverse ${\mathbb{P}}_{C}^{-1}$ (for $C$ large
enough) is

$\displaystyle{\mathbb{P}}_{C}^{-1}(f_{C},g_{C})=\int_{0}^{\min\\{\eta(E),t(x,\omega)\\}}e^{-\int_{0}^{s}(Ca+\Sigma)(x-\tau\omega,\omega)d\tau}f_{C}(x-s\omega,\omega,E+as)ds$
$\displaystyle+H(\eta(E)-t(x,\omega))e^{-\int_{0}^{t(x,\omega)}(Ca+\Sigma)(x-s\omega,\omega)ds}{g_{C}}(x-t(x,\omega)\omega,\omega,E+at(x,\omega))$
(253)

where as above $\eta(E)={{E_{m}-E}\over a}$. Working instead of the space
$H_{-,0}^{(1,0,0)}(G\times S\times I^{\circ})$ in more appropriate spaces
(which we do not conjecture here) one is able to retrieve regularity results
by using similar techniques as above in section 5.2. Applying this kind of
layout the condition (168) becomes unnecessary. We also notice that the
knowledge of explicit formulas is not essential in Neumann series approach.
The key is the a priori estimate like (LABEL:sc-5) for which one can retrieve
sufficient criteria without knowing the solution explicitly. The Lebesgue
index $p$ can also be chosen more generally $p\in[1,\infty]$. The case $p=2$
is, however most important as to approximation and numerical analysis. The
considerations require further research.

As we mentioned above only a limited the Sobolev regularity of solutions can
be achieved for transport problems. Nevertheless, in appropriately weighted
Sobolev spaces the regularity of arbitrary order is possible. One of them (for
tangential regularity) we depicted in Remark 6.7. Additionally, we mention
[46, Theorem 2.2] which gives, among others sufficient criteria under which
for any $k\in\mathbb{N}_{0}$ the solution of the mono-kinetic problem

$\displaystyle\omega\cdot\nabla_{x}\psi+(\Sigma+\lambda)\psi={f}(x,\omega),\
{\psi}_{|\Gamma_{-}}={0}.$ (254)

belongs to $m^{-k}\eta_{+}^{-s}\eta_{-}^{\tau}H^{k}(G,L^{2}(S))$ for $s,\
\tau$ large enough. Here $m$ and $\eta$ are appropriate weight functions. The
property $\psi\in m^{-k}\eta_{+}^{-s}\eta_{-}^{\tau}H^{k}(G,L^{2}(S))$ implies
the interior Sobolev regularity of solutions since it can be seen that
$m^{k}\eta_{+}^{s}\eta_{-}^{-\tau}>0$ in any compact subset of $G\times S$.
The reference contains results which can be applied to $E$-dependent problems
as well.

Acknowledgement. The author thanks Dr. Petri Kokkonen for useful discussions
and improvements while preparing the paper.

## References

  * [1] Agoshkov, V. _Boundary Value Problems for Transport Equations_ , Springer Science +Business Media, 1998
  * [2] Alexandre, R., Y. Morimoto, S. Ukai, C.-J. Xu and T. Yang. _Global existence and full regularity of the Boltzmann equation without angular cutoff_. Comm. in Math. Phys. 304, Article number 304, 2011
  * [3] Alonso R. and W. Sun. _The radiative transfer equation in the forward-peaked regime_ , arXiv:1411.0163v1 [math.AP], 2014
  * [4] Aubin, J.-P. _Applied functional analysis_ , John Wiley and Sons, 1979
  * [5] Bouchut F. _Hypoelliptic regularity in kinetic equations_ , J. Math. Pures Appl. 81, pp. 1135-1159, 2002
  * [6] Case, K.M. and P.F. Zweifel. _Existence and uniqueness theorems for neutron transport equation_. J. Math. Phys. 4 (11), pp. 1376-1385, 1963
  * [7] Cessenat, M. _Théorémes de trace $L^{p}$ pour des espaces de fonctions de la neutronique_, C.R. Acad. Sc. Paris, t. 299, Serie 1, number 16, 1984
  * [8] Chen, J-Kun, Ping-Han Chuang, Chun-Hsiung Hsia and Jhe-Kuan Su. _A revisit of the velocity averaging lemma: On the regularity of stationary Boltzmann equation in a bounded convex domain_. arXiv:2011.00180v1 [math.AP] 31 Oct 2020
  * [9] Chen Y. and L. He. _Smoothing estimates for Boltzmann equation with full-range interactions. Spatially inhomogeneous case_ , Arch. Rational Mech. Anal. 203, pp. 343-377, 2012
  * [10] Choulli M. and P. Stefanov. _An inverse boundary value problem for stationary transport equation_. Osaka J. Math. 36 (1), 1998, pp. 87-104
  * [11] Chupin, L. _Fokker-Planck equation in bounded domain_. Annales de L’Institut Fourier 60 (1), pp. 217-255, 2010
  * [12] Dautray, R., Lions, J-L. _Mathematical Analysis and Numerical Methods for Science and Technology_ , Vol. 6. Evolution Problems II, Springer, 1999.
  * [13] Degond, P. _Global existence of smooth solutions for the Vlasov-Fokker-Planck equation in $1$ and $2$ space dimensions_. Ann. scient. Ec. Norm. Sup. $4^{e}$ serie 19 (4), pp. 519-542, 1986
  * [14] Egger, H. and Schlottbom, M.. _An $L^{p}$-theory for stationary radiative transfer_. Appl. Anal. 93 (6), pp. 1283-1296, 2014
  * [15] Egorov, Yu.V and M.A. Shubin (Eds.). _Partial Differential Equations IV: Microlocal Analysis and Hyperbolic Equations_. Encyclopaedia of Mathematical Sciences, Springer, 1993
  * [16] Frank, M., M. Herty and A.N. Sandjo. _Optimal radiotherapy treatment planning governed by kinetic equations_ , Math. Mod. Meth. Appl. Sci. 20 (4), 2010, pp. 661-678
  * [17] Frank, M., Goudon, T. _On a generalized Boltzmann equation for non-classical particle transport_ , Kinet. Relat. Models 3 (2010) 395-407.
  * [18] Friedrichs, K. O., _Symmetric positive linear differential equations_ , Comm. Pure Appl. Math. 11, pp. 333-418, 1958.
  * [19] F. Golse, P-L. Lions, B. Perthame and R. Sentis. _Regularity of moments of the solution of a transport equation_ , J. Funct. Anal. 76. pp. 110-125, 1988
  * [20] Grisvard, P. _Elliptic problems in nonsmooth domains_ , vol. 24 of Monographs and Studies (Mathematics), Pitman, 1985
  * [21] Guo, Y., C. Kim, D. Tonon and A. Trescases. _BV-regularity of the Boltzmann equation in non-convex domains_ , Archive for Rational Mechanics and Analysis 220 (3), 2016
  * [22] Guo, Y., C. Kim, D. Tonon and A. Trescases. _Regularity of the Boltzmann equation in convex domains_ , Inventiones Mathematicae 207 (1), pp. 115-290, 2017
  * [23] Halmos P.R. and Sunder,V.S. . _Bounded Integral Operators on $L^{2}$ Spaces_, Springer 1978
  * [24] Hansen S., J. Hilgert and W. Paravicini. _Microlocal Analysis_. Lecture notes, 2010
  * [25] Herty, M., Jörres, C. and Sandjo, A.N. _Optimization of a model Fokker-Planck equation_. Kinetic and Related Models 5 (3), pp.485-503, 2012
  * [26] Hörmander, L. _The Analysis of Linear Partial Differential Operators III_ , Springer, 1985
  * [27] Hsiao, G.C. and Wendland, W.L. _Boundary Integral Equations_ , Springer, 2008.
  * [28] Jörres, C. _Numerical Methods for Boltzmann Transport Equations in Radiotherapy Treatment Planning_. Dissertation, RWTH Aachen University, 2015
  * [29] Kawagoe, D. _Regularity of solutions to the stationary transport equation with the incoming boundary data_. Thesis in Kyoto University, 2018
  * [30] Kim, C. _Formation and propagation of discontinuity for Boltzmann equation in non-convex domains_. arXiv: 1007.1997v1 [math.AP] 12 Jul 2010
  * [31] Lax, P.D. and Phillips, R.S.. _Local boundary conditions for dissipative symmetric linear differential operators_ , Comm. Pure Appl. Math. 13, pp. 427-455, 1960
  * [32] Le Bris, C. and Lions, P.-L. . _Existence and uniqueness of solutions to Fokker-Planck type equations with irregular coefficients_. Comm. Partial Diff. Equations 33, pp. 1272-1317, 2008
  * [33] Melrose, R.B. and J. Sjöstrand. _Singularities of boundary value problems I_. Comm. Pure Appl. Math., Vol. 31, pp. 593- 617, 1978
  * [34] Metivier, G. _Small viscosity and boundary layer methods, theory, stability analysis and applications_ , Modelling and Simulation in Science, Engineering and Technology, Birkhäuser, 2003
  * [35] Metivier, G. and K. Zumbrun. _Hyperbolic boundary value problems for symmetric systems with variable multiplicities_ , J. Diff. Equations 211, pp. 61-134, 2005
  * [36] Mokhtar-Kharroubi, M. _$W^{p,1}$ regularity in transport theory_, Math. Models. Methods. Appl. Sci. 1 (4), pp. 477-499, 1991
  * [37] Morando, A., P. Secchi and P. Trebeschi. _Characteristic initial boundary value problems for symmetrizable systems_. Rend. Sem. Mat. Univ. Torino Vol. 67, pp. 229-245, 2009
  * [38] Morando, A., P. Secchi and P. Trebeschi. _Regularity of solutions to characteristic initial-boundary value problems for symmetrizable systems_. J. of Hyperbolic Differential Equations, 6 (4), pp. 753-808, 2009
  * [39] Nishitani, T. and M. Takayama. _A characteristic initial boundary value problem for a symmetric positive system_. Hokkaido Math. J. 25, pp. 167-182, 1996
  * [40] Nishitani, T. and M. Takayama. _Characteristic initial boundary value problems for symmetric hyperbolic systems_. Osaka J. Math. 35, pp. 629-657, 1998
  * [41] Pazy, A. _Semigroups of Linear Operators and Applications to Partial Differential Equations_ , Springer, 1983.
  * [42] Payne, K.R. _Interior regularity of the Dirichlet problem for the Tricomi equation_. J. Math. Anal. Appl. 199, pp. 271-292, 1995
  * [43] Rauch, J. _Symmetric positive systems with boundary characteristic of constant multiplicity_. Trans. Amer. Math. Soc. 291 (1), pp. 167-187, 1985
  * [44] Rauch, J. and F. Massey. _Differentiability of solutions to hyperbolic initial-boundary value problems_. Trans. Amer. Math. Soc. 189, pp. 303-318, 1974
  * [45] Rauch, J. _Hyperbolic Partial Differential Equations and Geometric Optics_ , 2012
  * [46] Takayama, M. _Regularity of solutions to non-uniformly characteristic boundary value problems for symmetric systems_. Reprint in Kyoto University (on-line), 2002
  * [47] Tervo, J. and Kokkonen, P. _On existence of $L^{1}$-solutions for coupled Boltzmann transport equation and radiation therapy treatment planning_, ArXiv: 1406.3228v2 $[$ math.OC $]$ 1 Feb 2017
  * [48] Tervo, J., Kokkonen, P., Frank, M. and Herty, M. _On Existence of $L^{2}$-solutions of Coupled Boltzmann Continuous Slowing Down Transport Equation System_, arXiv:1603.05534v3 $[$ math.OC $]$ 5 Apr 2018
  * [49] Tervo, J., Kokkonen, P. , Frank, M. and Herty, M. _On existence of solutions for Boltzmann Continuous Slowing Down transport Equation_ , J. Math. Anal. Appl. 460 :1, 271-301, 2018
  * [50] Tervo, J., Kokkonen, P. , Frank, M. and Herty, M. _On approximative linear Boltzmann transport equation for charged particle transport_. Math. Models Methods Appl. Sci. 14:28, pp. 2905-2939, 2018.
  * [51] Tervo, J. _On linear hypersingular Boltzmann transport equation and its variational formulation_. arXiv:1808.09631v3 $[$ math.AP $]$, 2019.
  * [52] Tervo, J. and M- Herty. _On approximation of a hyper-singular transport operator and existence of solutions_. Methods and Applications in Analysis 27 (1). pp. 125-152, 2020
  * [53] Tervo, J. _On global existence and regularity of solutions for a transport problem related to charged particles_. J. Computational and Theoretical Transport, DOI: 10.1080/23324309.2020.1851722 (on-line), 2021
  * [54] Tian, R. _Existence, uniqueness and regularity property of solutions to Fokker-Planck type equations_. J. Math. Anal. Appl. 2 (1), pp. 53-63, 2014
  * [55] Zeghal, A. _Sobolev regularity in neutron transport theory_ , ISRH Math. Anal., pp. 1-22, 2012
  * [56] Ziemer, W.P. _Weakly Differentiable Functions_. Sobolev Spaces and Functions of Bounded Variation. Springer, 1989
  * [57] Vazy, A. _Propagation of singularities for the wave equation on manifolds with corners_. Annals of Math. 168, pp. 749-812, 2008
  * [58] Wloka, J. _Partial Differential Equations_. Cambridge University Press, 1987