$\\{\Sigma_{t}\\}_{t\in[0,T)}$ starting from $\hat{\Gamma}_{\epsilon}$ remains
mean convex. Moreover, $H_{\Sigma_{t}}\geq ce^{-n(\mu+1)t}y^{\mu}$ for some
$c>0$ and $t\in[0,T)$.

###### Proof.

By direct computation (see, e.g., [Hui86]),
$\frac{d}{dt}H_{\Sigma_{t}}=\Delta_{\Sigma_{t}}H_{\Sigma_{t}}+(|A_{\Sigma_{t}}|^{2}-n)H_{\Sigma_{t}}$.
We also have $\frac{d}{dt}(y^{\mu})=\Delta_{\Sigma_{t}}(y^{\mu})+n\mu
y^{\mu}-\mu(\mu+1)y^{\mu}(1-(\mkern
1.5mu\overline{\mkern-1.5mu\textbf{n}\mkern-1.5mu}\mkern
1.5mu_{\Sigma_{t}}\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1})^{2})$.
Hence, $\left(\frac{d}{dt}-\Delta_{\Sigma_{t}}\right)\left(e^{-n\mu
t}y^{\mu}\right)\leq 0$, and
$\left(\frac{d}{dt}-\Delta_{\Sigma_{t}}\right)\left(e^{nt}H_{\Sigma_{t}}\right)=|A_{\Sigma_{t}}|^{2}H_{\Sigma_{t}}$.
Combining these quantities, we get for any $c>0$,

$\displaystyle\left(\frac{d}{dt}-\Delta_{\Sigma_{t}}\right)\left(ce^{-n\mu
t}y^{\mu}-e^{nt}H_{\Sigma_{t}}\right)\leq-|A_{\Sigma_{t}}|^{2}e^{nt}H_{\Sigma_{t}}\leq|A_{\Sigma_{t}}|^{2}\left(ce^{-n\mu
t}y^{\mu}-e^{nt}H_{\Sigma_{t}}\right).$

At $t=0$, we know from direct computation that

$\displaystyle H_{\hat{\Gamma}_{\epsilon}}$
$\displaystyle=-(\Delta_{\Gamma}+|A_{\Gamma}|^{2}-n)(\epsilon f)+a(p,h,\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern 1.5muh,\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern 1.5mu^{2}h)\epsilon
f+y\left\langle\textbf{b}(p,h,\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern 1.5muh,\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu^{2}h),\nabla\epsilon f\right\rangle$
$\displaystyle=\epsilon\big{(}\lambda_{1}f+a(p,h,\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern 1.5muh,\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu^{2}h)f+y\left\langle\textbf{b}(p,h,\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern 1.5muh,\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern 1.5mu^{2}h),\nabla
f\right\rangle\big{)}$

where $|a|+|\textbf{b}|\leq C(|h|+|\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern 1.5muh|+|\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern 1.5mu^{2}h|)$ and
$h=\frac{\epsilon f}{y}$. Hence, by Proposition 8.1, there is a small
$\epsilon$, so that $\hat{\Gamma}_{\epsilon}$ is mean convex and $ce^{-n\mu
t}y^{\mu}-e^{nt}H_{\Sigma_{t}}\leq 0$ holds at $t=0$ for some $c>0$. On the
other hand, since $\Gamma$ is a $C^{3}$ hypersurface up to the ideal boundary,
the perturbed hypersurface $\hat{\Gamma}_{\epsilon}$ is $C^{2}$ up to the
ideal boundary. Hence, $|A_{\hat{\Gamma}_{\epsilon}}|=\mathcal{O}(y)$. By the
pseudo-locality theorem [CY07, Theorem 7.5], we see that for any $T_{0}<T$,
there are constants $C(T_{0})>0$ and $\epsilon=\epsilon(T_{0})>0$, so that

$\sup_{0\leq t\leq
T_{0},p\in\Sigma_{t}\cap\\{y\leq\epsilon\\}}|A_{\Sigma_{t}}|(p)\leq C(T_{0}).$

Since $\bigcup_{0\leq t\leq
T_{0}}\big{(}\Sigma_{t}\cap\\{y\geq\epsilon\\}\big{)}\times\\{t\\}$ is
compact, we see that $\sup_{0\leq t\leq T_{0}}|A_{\Sigma_{t}}|<\infty$.

Then, by a non-compact maximum principle (e.g., one can use a variant of the
monotonicity formula introduced in [Ber21] and modify the proof of [EH89,
Corollary 1.1]), $ce^{-n\mu t}y^{\mu}-e^{nt}H_{\Sigma_{t}}\leq 0$ for
$t\in[0,T)$. ∎

######  Lemma 8.4.

Let $\\{\Sigma_{t}\\}_{t}$ be a mean curvature flow which satisfies
$H_{\Sigma_{t}}\geq ce^{-n(\mu+1)t}y^{\mu}$ for some $c>0$ and
$\mu\in\mathbb{R}^{1}$. Suppose the inital hypersurface $\Sigma_{0}$ is
$C^{2}$-asymptotic to the ideal boundary and meets the ideal boundary
orthogonally. Then, for any $T_{1}<\infty$ with $T_{1}\leq T$(the maximal
existing time), there is a constant $C_{1}=C_{1}(n,T_{1},\Sigma_{0})>0$, so
that $|A_{\Sigma_{t}}|^{2}\leq C_{1}y^{-2\mu}e^{4nt}H_{\Sigma_{t}}^{2}$ for
any $p\in\Sigma_{t}$ and $t<T_{1}$.

###### Proof.

When it is clear from the context, we will omit the subscripts and simply
write $A$ (resp. $H$) for $A_{\Sigma_{t}}$ (resp. $H_{\Sigma_{t}}$). By
[Hui86, Corollary 3.5],

$\frac{d}{dt}|A|^{2}=\Delta|A|^{2}-2|\nabla
A|^{2}+2|A|^{2}(|A|^{2}-n)+4n(|A|^{2}-\frac{1}{n}H^{2}).$

After computation, we see that

$\displaystyle\frac{d}{dt}\left(\frac{|A|^{2}}{H^{2}}-\frac{1}{n}\right)=$
$\displaystyle\Delta\left(\frac{|A|^{2}}{H^{2}}-\frac{1}{n}\right)+4n\left(\frac{|A|^{2}}{H^{2}}-\frac{1}{n}\right)+\frac{4}{H^{3}}\left\langle\nabla(|A|^{2}),\nabla
H\right\rangle$ $\displaystyle-\frac{6|A|^{2}}{H^{4}}|\nabla
H|^{4}-\frac{2|\nabla A|^{2}}{H^{2}}.$

Following the computation in [BW17, Proof of Proposition 3.3],

$\displaystyle\frac{4}{H^{3}}\left\langle\nabla(|A|^{2}),\nabla
H\right\rangle\leq\frac{2}{H}\left\langle\nabla\left(\frac{|A|^{2}}{H^{2}}\right),\nabla
H\right\rangle+\frac{6|A|^{2}|\nabla H|^{2}}{H^{4}}+\frac{2|\nabla
A|^{2}}{H^{2}}.$

Hence, the function $g=\frac{|A|^{2}}{H^{2}}e^{-4nt}-\frac{1}{n}e^{-4nt}$
satisfies $\frac{d}{dt}g\leq\Delta g+\frac{2}{H}\left\langle\nabla g,\nabla
H\right\rangle$. From the assumptions on $\Sigma_{0}$, we see that
$|A_{\Sigma_{0}}|=\mathcal{O}(y)$ as $y=y(p)\to 0$. Since $T_{1}<\infty$, the
pseudo-locality theorem [CY07, Theorem 7.5] implies that there is an
$\epsilon>0$, so that

(8.2) $\sup_{t<T_{1},p\in\Sigma_{t}\cap\\{y\leq 2\epsilon\\}}|A|\leq C.$

Together with Lemma 8.3,

(8.3) $\sup_{0\leq t<T_{1},p\in\Sigma_{t}\cap\\{y\leq\epsilon\\}}g(t,p)\leq
Cy^{-2\mu}e^{2n(\mu+1)T_{1}}.$

On the other hand, we can use a maximum principle for $g$ on
$W_{\epsilon}=\Big{(}\bigcup_{0\leq
t<T_{1}}\Gamma_{t}\times\\{t\\}\Big{)}\cap\\{y\geq\epsilon\\}$ to see that $g$
is bounded above by the maximum of $g$ over the parabolic boundary of
$W_{\epsilon}$. Hence,

(8.4) $\sup_{0\leq t<T_{1},p\in\Sigma_{t}\cap\\{y\geq\epsilon\\}}g(t,p)\leq
Cy^{-2\mu}e^{2n(\mu+1)T_{1}}.$

Combining (8.3) and (8.4), we get the desired bound. ∎

######  Lemma 8.5.

Let $2\leq n\leq 6$ and $\\{\Sigma_{t}\\}_{t\in[0,T)}$ be the same flow as in
Lemma 8.4. If we further assume that
$\lambda_{\mathbb{H}}[\Sigma_{0}]<\lambda[\mathbb{S}^{n-1}\times\mathbb{R}^{1}]$,
then the maximal time of existence is $T=\infty$.

###### Proof.

Assume for now $T<\infty$. From (8.2), we see that

$\lim_{t\to T^{-}}\sup_{p\in\Sigma_{t}}|A_{\Sigma_{t}}|=\lim_{t\to
T^{-}}\sup_{p\in\Sigma_{t}\cap\\{y\geq\epsilon\\}}|A_{\Sigma_{t}}|=\infty.$

Pick a sequence $\\{p_{i}\\}\in\Sigma_{t_{i}}$ with $t_{i}\to T$ and
$|A_{\Sigma_{t_{i}}}|(p_{i})=\sup_{p\in\Sigma_{t_{i}}\cap\\{y\geq\delta\\}}|A_{\Sigma_{t_{i}}}|$.
Without loss of generality, we may assume that $p_{i}\to\widetilde{p}$.
Consider the exponential map
$\text{exp}_{\widetilde{p}}:\,\mathbb{R}^{n+1}\to\mathbb{H}^{n+1}$. It is
readily checked that $\text{exp}_{\widetilde{p}}$ is a diffeomorphism. Let
$(\mathbb{R}^{n+1},g)$ be the pull-back of
$(\mathbb{H}^{n+1},g_{\mathbb{H}^{n+1}})$ under the exponential map. With a
slight abuse of notions, assume $\\{\Sigma_{t}\\}$ is also a flow in
$(\mathbb{R}^{n+1},g)$. Let
$\\{\hat{\Sigma}^{\lambda}_{s}\\}_{s\in[-T\lambda^{-2},0)}=\\{\frac{1}{\lambda}\Sigma_{T+\lambda^{2}s}\\}$.
Here, the dilation is understood in $\mathbb{R}^{n+1}$. One can check that
$\\{\hat{\Sigma}_{s}^{\lambda}\\}$ is a mean curvature flow in
$(\mathbb{R}^{n+1},g^{\lambda})$, where
$g^{\lambda}_{ij}(\textbf{x})=g(\lambda\textbf{x})(\partial_{i},\partial_{j})=g_{ij}(\lambda\textbf{x})$.
By [Ber21],

$\displaystyle\int_{\Sigma_{t}}\Phi^{0,\textbf{0}}_{n}(t-T,p)d\mathcal{H}^{n}_{\mathbb{H}}(p)$

is monotone decreasing in $t\in[0,T)$. After re-writing it into an integral
over $\hat{\Sigma}^{\lambda}_{s}$ in $\mathbb{R}^{n+1}$, we have

(8.5)
$\int_{\Sigma_{t}}\Phi^{0,\textbf{0}}_{n}(t-T,p)d\mathcal{H}^{n}_{\mathbb{H}}(p)=\int_{\hat{\Sigma}^{\lambda}_{s}}\lambda^{n}K_{n}(-\lambda^{2}s,\lambda\textbf{y})\sqrt{\text{det}\,g^{\lambda}(\textbf{y})}d\mathcal{H}^{n}(\textbf{y}).$

Since $g^{\lambda}\to g_{\mathbb{R}^{n+1}}$ in
$C^{\infty}_{loc}(\mathbb{R}^{n+1})$ as $\lambda\to 0$ and the arguments in
[Ton19, Chapter 3] are local, one can show with minor modifications that for
any sequence $\lambda_{k}\to 0^{+}$, there is a subsequence
$\\{\lambda_{k_{i}}\\}\subset\\{\lambda_{k}\\}$, so that
$\\{\hat{\Sigma}^{\lambda_{i}}_{s}\\}$ converges to a self-shrinking Brakke’s
flow $\\{\mu_{s}\\}_{s<0}$ in $(\mathbb{R}^{n+1},g_{\mathbb{R}^{n+1}})$. By
the nature of convergence,
$\lambda[\mu_{s}]<\lambda[\mathbb{S}^{n-1}\times\mathbb{R}^{1}]<2$.

By Lemma 8.4 and the Brakke’s regularity theorem, on the regular part of
$\mu_{-1}$, $|A_{\mu_{-1}}|\leq CH_{\mu_{-1}}$. It then follows from [CIIW13,
Proposition 5.1] that $\\{\mu_{s}\\}$ is actually smooth and hence, by [CM12,
Theorem 0.17], is either $\mathbb{S}^{n}$ or $\mathbb{R}^{n}$. Since
$\Sigma_{t}$ doesn’t have closed component, $\mu_{s}$ must be a plane. This
implies $\\{\Sigma_{t}\\}$ is smooth at $(\widetilde{p},T)$, which contradicts
our assumption. ∎

###### Proof of Theorem 8.2.

Combining Lemma 8.3, Lemma 8.4 and Lemma 8.5, we see that for a sufficiently
small $\epsilon$, the mean curvature flow $\\{\Sigma_{t}\\}$ starting from
$\hat{\Gamma}_{\epsilon}$ exists for all $t\geq 0$ and remains mean convex. We
first show that as $t\to\infty$,
$\Sigma_{t}\xrightarrow{C^{\infty}_{loc}}\Gamma^{\prime}$ for some stable
minimal hypersurface $\Gamma^{\prime}$ that is $C^{k+1,\alpha}$-asymptotic to
$M$.

Since $\Sigma_{t}$ is mean convex, there is a one-parameter family of open
sets $\\{U_{t}\\}_{t\in[0,\infty)}$ so that $\partial U_{t}=\Sigma_{t}$ and
$U_{t}\subset U_{t^{\prime}}$ if $0\leq t^{\prime}\leq t$. Let
$K=\bigcap_{t\geq 0}\text{cl}(U_{t})$. For any $\tau_{i}\to\infty$, consider
the flow $\\{\Sigma^{i}_{t}\\}:=\\{\Sigma_{t+\tau_{i}}\\}$. By the compactness
of Brakke’s flow, there is a subsequence $\\{\Sigma^{i_{k}}_{t}\\}$ converging
to a Brakke’s flow $\\{\nu_{t}\\}_{t\in[0,\infty)}$. For each $t\geq 0$,
$\mathcal{H}^{n}_{\mathbb{H}}\mathbin{\vrule
height=6.88889pt,depth=0.0pt,width=0.55974pt\vrule
height=0.55974pt,depth=0.0pt,width=5.59721pt}\Sigma^{i_{k}}_{t}\to\nu_{t}$
implies $\text{Spt}(\nu_{t})\subset\partial K$. On the other hand, the upper
semicontinuity of the Gaussian density implies $\partial
K\subset\text{Spt}(\nu_{t})$. So, $\text{Spt}(\nu_{t})=\partial K$ for all
$t\geq 0$. Furthermore, there is a subsequence of $\\{\chi_{U_{t+i_{k}}}\\}$
that converges as Caccioppoli sets. So, we see that $\nu_{s}$ is a
multiplicity one rectifiable Radon measure. So,
$\nu_{s}=\mathcal{H}^{n}_{\mathbb{H}}\mathbin{\vrule
height=6.88889pt,depth=0.0pt,width=0.55974pt\vrule
height=0.55974pt,depth=0.0pt,width=5.59721pt}\partial K$ for all $s\geq 0$ and
$\nu_{s}$ is thus stationary.

By the nature of convergence and the monotonicity formula [Ber21],
$\lambda_{\mathbb{H}}[\nu_{s}]<\lambda[\mathbb{S}^{n-1}\times\mathbb{R}^{1}]$.
It then follows from this entropy bound and [Whi97] that the singular part of
$\nu_{0}$ is at most of Hausdorff dimension $n-3$. As the mean curvature
vector of $\Sigma_{t}$ points toward $\nu_{0}$, we see that
$\\{\Sigma_{t}\\}_{t}$ serves as a foliation of $\nu_{0}$. So, by [SW89],
$\nu_{0}$ is locally one-sided minimizing. By [SS81],
Spt($\nu_{0}$)$:=\Gamma^{\prime}$ is regular. Since
$\lambda_{\mathbb{H}}[\Gamma^{\prime}]<\infty$, the regularity of
$\Gamma^{\prime}$ at the ideal boundary follows in the same manner as in the
proof of Proposition 7.3. As $\Gamma^{\prime}$ is smooth, [Whi05] implies that
$\Sigma_{t}\xrightarrow{C^{\infty}_{loc}}\Gamma^{\prime}$.

Fix $\epsilon>0$, so that $\Gamma\cap\\{y\leq 2\epsilon\\}$ and
$\Gamma^{\prime}\cap\\{y\leq 2\epsilon\\}$ are both $C^{k,\alpha}$ isotopic to
$M\times(0,2\epsilon]$. By shrinking $\epsilon$ if necessary, we see from
[Yao, Lemma 5.1] that $\Sigma_{t}\cap\\{y\leq 2\epsilon\\}$ is a graph over
$M\times(0,2\epsilon]$ for all $t$. Choose a transverse section v on
$\Gamma^{\prime}$ with $\textbf{v}(p)$ perpendicular to $\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}$ on
$\Gamma^{\prime}\cap\\{y=\epsilon\\}$. Then, for $T_{0}>0$ sufficiently large,
$\Sigma_{T_{0}}\cap\\{y\geq\epsilon\\}$ is a v-graph over
$\Gamma^{\prime}\cap\\{y\geq\epsilon\\}$. Let $\widetilde{\Sigma}_{T_{0}}$
coincide with $\Sigma_{T_{0}}$ on $\\{y\geq\epsilon\\}$ and extends to $M$ in
a $C^{k,\alpha}$ way. We can further require that $\widetilde{\Sigma}_{T_{0}}$
and $\Gamma^{\prime}$ are $C^{k,\alpha}$ isotopic. Now, partition $[0,T_{0}]$
into sub-intervals $[s_{k},s_{k+1}]$, ($k=1,2,\cdots,N$). Since
$\Sigma_{t}\cap\\{y\leq 2\epsilon\\}$ is a graph over $M\times(0,2\epsilon]$,
we may choose a trasnverse section $\textbf{v}_{k}$ over
$\Sigma_{s_{k}}\cap\\{y\geq\epsilon\\}$ with $\textbf{v}_{k}$ perpendicular to
$\mkern 1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}$
on $\Sigma_{s_{k}}\cap\\{y=\epsilon\\}$ and that
$\Sigma_{s_{k}}\cap\\{y\geq\epsilon\\}$ can be expressed as a
$\textbf{v}_{k+1}$-graph over $\Sigma_{s_{k+1}}\cap\\{y\geq\epsilon\\}$. So,
$\Sigma_{s_{k}}\cap\\{y\geq\epsilon\\}$ is isotopic to $\Sigma_{s_{k\pm
1}}\cap\\{y\geq\epsilon\\}$. Finally, extend
$\Sigma_{s_{k}}\cap\\{y\geq\epsilon\\}$ to $M$ in a $C^{k,\alpha}$ way with
the extension of $\Sigma_{s_{k}}$ isotopic to that of $\Sigma_{s_{k\pm 1}}$.
Thus, we have shown that $\Sigma_{0}$ and $\Gamma^{\prime}$ are $C^{k,\alpha}$
isotopic. The isotopy of $\Gamma$ and $\Sigma_{0}$ can be constructed in an
obvious way.

Finally, we show that if there are minimal hypersurfaces $\Gamma_{1}$ and
$\Gamma_{2}$ with $\Gamma_{1}\preceq\Gamma\preceq\Gamma_{2}$, then
$\Gamma^{\prime}$ is also trapped between $\Gamma_{1}$ and $\Gamma_{2}$. By
Proposition 7.6, the distance between $\Gamma_{1}$ and $\Gamma_{2}$ at $y=r$
is bounded below by $cr^{n+1}$ for some constant $c>0$. By Proposition 8.1, we
can shrink $\epsilon$ to make $\hat{\Gamma}_{\epsilon}$ satisfy
$\Gamma_{1}\preceq\hat{\Gamma}_{\epsilon}\preceq\Gamma_{2}$. By the maximum
principle, $\Gamma_{1}\preceq\Sigma_{t}\preceq\Gamma_{2}$ for all $t\geq 0$.
Hence, $\Gamma^{\prime}$ is also trapped between $\Gamma_{1}$ and
$\Gamma_{2}$. ∎

Following the arguments in Lemma 8.4, Lemma 8.5 and the proof of Theorem 8.2,
we have also proved the following:

######  Theorem 8.6.

Assume $2\leq n\leq 6$, $\alpha\in(0,1)$ and $k\geq 2$. Let $\Sigma$ be a
hypersurface that is $C^{k,\alpha}$-asymptotic to an $(n-1)$-dimensional
$C^{k,\alpha}$ closed submanifold $M\subset\mathbb{R}^{n}\times\\{0\\}$ and
meets $M$ orthogonally. If
$\lambda_{\mathbb{H}}[\Sigma]<\lambda[\mathbb{S}^{n-1}\times\mathbb{R}^{1}]$,
and $\Sigma$ satisfies $H_{\Sigma}\geq cy^{\mu}$ for some $c>0$ and
$\mu\in\mathbb{R}^{1}$, then $\Sigma$ is $C^{k,\alpha}$ isotopic to a stable
minimal hypersurface $\Sigma^{\prime}$ that is also $C^{k,\alpha}$-asymptotic
to $M$.

##  9\. Topological Uniqueness for Minimal Hypersurfaces with Small Entropy

In this final section, we prove Theorem 1.3. Let $2\leq n\leq 6$,
$\alpha\in(0,1)$, $k\geq 4$ and $m=\min\\{k,n\\}$ throughout this section. Let
$M$ be a $C^{k+1,\alpha}$ closed submanifold in
$\partial_{\infty}\mathbb{H}^{n+1}$ with
$\lambda_{c}[M]<\text{Vol}(\mathbb{S}^{n-1})\lambda[\mathbb{S}^{n-1}\times\mathbb{R}^{1}]$.
After suitable transformations, we assume that
$M\subset\mathbb{R}^{n}\times\\{0\\}$. By a minor modification of Proposition
2.2, Proposition 2.3, we can apply [Sma65] to get
$M^{\prime}\in\mathcal{M}^{k+1,\alpha}$ being a regular value of $\Pi$ with
$M^{\prime}$ close to $M$ in the $C^{k+1,\alpha}$ topology in
$\mathbb{R}^{n}\times\\{0\\}$.

We first prove Theorem 1.3 when $M$ is a regular value of $\Pi$. As a starting
point, we prove

######  Proposition 9.1.

Let $\ell\geq 2$ and $M\in\mathcal{E}^{\ell,\alpha}$ be a regular value of
$\Pi$. Then, there are only finitely many stable minimal hypersurfaces in
$\mathcal{M}^{\ell,\alpha}$ that are $C^{\ell,\alpha}$-asymptotic to M.

###### Proof.

Let
$\mathcal{S}:=\\{\Gamma\in\mathcal{M}^{\ell,\alpha}:\Pi(\Gamma)=M,\,\Gamma\text{
is stable}\\}$. If $\mathcal{S}$ contains infinitely many elements, we choose
$\\{\Gamma_{j}\\}_{j=1}^{\infty}\subset\mathcal{S}$ which are distinct to each
other. By Proposition 7.3, we get a convergent subsequence. With a slight
abuse of notions, we assume that $\Gamma_{j}$ converges (in the
$C^{\infty}_{loc}$ sense) to a stable minimal hypersurface
$\widetilde{\Gamma}\in\mathcal{S}$. We claim that $\Gamma_{j}$ converges to
$\widetilde{\Gamma}$ in the $C^{\ell,\frac{\alpha}{2}}$ topology in $\mkern
1.5mu\overline{\mkern-1.5mu\mathbb{R}\mkern-1.5mu}\mkern 1.5mu^{n+1}_{+}$. Fix
$p\in M$. Without loss of generality, assume
$p=(\textbf{0},0)\in\mathbb{R}^{n}\times\\{0\\}$ and
$\textbf{n}_{M}(p)=(0,\cdots,0,1,0)$. Since the argument in the proof of
Proposition 6.12 only relies on the geometry of ideal boundary $M$, there is a
$\delta>0$, so that for any $j$,

$\Gamma_{j}\cap(-\delta,\delta)^{n}\times[0,\delta)=\\{(\textbf{x}^{\prime},x_{n},y):x_{n}=u_{j}(\textbf{x}^{\prime},y),(\textbf{x}^{\prime},y)\in(-\delta,\delta)^{n-1}\times[0,\delta)\\},$

where $u_{j}\in
C^{\ell,\alpha}\Big{(}(-\delta,\delta)^{n-1}\times[0,\delta)\Big{)}$ with
$u_{j}(\textbf{x}^{\prime},0)=\varphi(\textbf{x}^{\prime})$ parametrize
$M\cap\Big{(}(-\delta,\delta)^{n-1}\times\\{0\\}\Big{)}$. By [Lin89b, Theorem
5.2],

$\|u_{j}\|_{C^{\ell,\alpha}\big{(}(-\frac{\delta}{2},\frac{\delta}{2})^{n-1}\times[0,\frac{\delta}{2})\big{)}}\leq
C(n,\ell,\alpha,\delta)\left(1+\|\varphi\|_{C^{\ell,\alpha}\big{(}(-\delta,\delta)^{n-1}\times[0,\delta)\big{)}}\right).$

This implies every subsequence of $u_{j}$ has a sub-subsequence that converges
in the
$C^{\ell,\frac{\alpha}{2}}_{loc}\big{(}(-\frac{\delta}{2},\frac{\delta}{2})^{n-1}\times[0,\frac{\delta}{2})\big{)}$
sense to a function $\widetilde{u}$. Since
$\Gamma_{j}\xrightarrow{C^{\infty}_{loc}}\widetilde{\Gamma}$, the graph of
$\widetilde{u}$ coincides with $\widetilde{\Gamma}$. Hence,
$\Gamma_{j}\to\widetilde{\Gamma}$ in the $C^{\ell,\frac{\alpha}{2}}$-topology
of $\mkern 1.5mu\overline{\mkern-1.5mu\mathbb{R}\mkern-1.5mu}\mkern
1.5mu^{n+1}_{+}$.

On the other hand, since $M$ is a regular value, $\Pi$ maps a neighborhood of
$\widetilde{\Gamma}$ in $\mathcal{M}^{\ell,\frac{\alpha}{2}}$
diffeomorphically to a neighborhood of $M$ in
$\mathcal{E}^{\ell,\frac{\alpha}{2}}$, which contradicts the fact that
$\Gamma_{k}\to\widetilde{\Gamma}$ in $C^{\ell,\frac{\alpha}{2}}$-topology of
$\mkern 1.5mu\overline{\mkern-1.5mu\mathbb{R}\mkern-1.5mu}\mkern
1.5mu^{n+1}_{+}$. ∎

######  Proposition 9.2.

Let $M\in\mathcal{E}^{k+1,\alpha}$ be a regular value of $\Pi$ with
$\lambda_{\mathbb{H}}[M]<\mathrm{Vol}(\mathbb{S}^{n-1})\lambda[\mathbb{S}^{n-1}\times\mathbb{R}^{1}]$.
Then, all the minimal hypersurfaces with $C^{k+1,\alpha}$-asymptotic boundary
$M$ are $C^{k,\alpha}$-isotopic to each other.

###### Proof.

By Theorem 8.2, any unstable minimal hypersurface is isotopic to a stable one.
So, it suffices to show that all the stable minimal hypersurfaces are isotopic
to each other.

Fix a stable minimal hypersurface $\Gamma$ that is $C^{k+1,\alpha}$-asymptotic
to $M$. By Proposition 7.4, there is a maximal stable minimal hypersurface
$\Gamma_{G}$ with $\Gamma\preceq\Gamma_{G}$. Let
$\mathcal{S}_{\Gamma,\Gamma_{G}}:=\\{\Gamma^{\prime}\in\mathcal{M}^{k+1,\alpha}:\Pi(\Gamma^{\prime})=M,\Gamma\preceq\Gamma^{\prime}\preceq\Gamma_{G}\\}$
and $\Gamma_{1}$ be a minimal element in $\mathcal{S}_{\Gamma,\Gamma_{G}}$.
That is, there’s no $\widetilde{\Gamma}\in\mathcal{S}_{\Gamma,\Gamma_{G}}$
with $\Gamma\preceq\widetilde{\Gamma}\preceq\Gamma_{1}$. Since $M$ is a
regular value, $\Gamma$ and $\Gamma_{1}$ are both strictly stable. So, Theorem
1.1 implies the existence of a third minimal hypersurface
$\Gamma^{\prime\prime}$ with
$\Gamma\preceq\Gamma^{\prime\prime}\preceq\Gamma_{1}$. Obviously,
$\Gamma^{\prime\prime}$ is unstable. By Theorem 8.2, we see that
$\Gamma^{\prime\prime}$ is $C^{k,\alpha}$ isotopic to both $\Gamma_{1}$ and
$\Gamma$. Replace $\Gamma$ by $\Gamma_{1}$, and repeat the same argument to
$\mathcal{S}_{\Gamma_{1},\Gamma_{G}}$. Since there are only finitely many
stable minimal hypersurfaces, an induction argument would imply $\Gamma$ is
eventually isotopic to $\Gamma_{G}$. Since $\Gamma$ is arbitrary chosen, we
have proved that all the stable minimal hypersurfaces are isotopic. ∎

###### Proof of Theorem 1.3.

Notice that all the unstable minimal hypersurfaces with
$C^{k+1,\alpha}$-asymptotic boundary $M$ are isotopic to stable ones. It
suffices to show that, by appropriately choosing $M^{\prime}$ which is a
regular value of $\Pi$, any stable minimal hypersurface $\Gamma$ with
$\Pi(\Gamma)=M$ is $C^{m,\alpha}$-isotopic to another minimal hypersurface
$\Gamma^{\prime}$ with $\Pi(\Gamma^{\prime})=M^{\prime}$.

When $\Gamma$ is strictly stable, the map $\Pi$ is a local diffeomorphism from
a neighborhood $\mathcal{U}$ of $\Gamma$ in $\mathcal{M}^{m,\alpha}$ to a
neighborhood of $M$ in $\mathcal{E}^{m,\alpha}$. Let $\psi\in C^{m,\alpha}(M)$
be a function with

$M_{\psi}=\big{\\{}\textbf{x}(p)+\psi(p)\nu_{M}(p):\,p\in M\big{\\}},$

where $\nu_{M}$ is a unit normal of $M$ in $\mathbb{R}^{n}\times\\{0\\}$. By
[And82, Theorem 3], there is a minimal hypersurface $\Upsilon_{\psi}$
asymptotic to $M_{\psi}$. If $\psi$ is sufficiently small in $C^{2}$-norm, by
[HL87, Theorem 2.2], there is a $\delta>0$ independent of $\psi$, so that for
any $p\in M_{\psi}$,
$\big{(}B^{n}_{\delta}(p)\times[0,\delta)\big{)}\cap\Upsilon_{\psi}$ can be
locally written as a graph over a hyperplane that is parallel to
$\textbf{e}_{n+1}$. Similar to the proof of Proposition 4.1, we choose an
approximate minimal hypersurface asymptotic to $M_{\psi}$ by letting
$\Gamma_{\psi}:=\\{\textbf{x}(p)+\mathcal{E}(\psi)\textbf{n}_{\Gamma}(p):\
p\in\Gamma\\}$ coincide with $\Upsilon_{\psi}$ in
$\\{y\leq\frac{\delta}{4}\\}$ and coincide with $\Gamma$ in
$\\{y\geq\frac{\delta}{2}\\}$. We can patch them together by a cut-off
function in $y$ since both of them can be expressed as graphs near the ideal
boundary. We see that the mean curvature of $\Gamma_{\psi}$, $H_{\psi}$, is in
$y^{\mu_{0}}\Lambda^{m-2,\alpha}_{0}$ for any $\mu_{0}>0$. To apply the theory
of uniformly degenerate operators, we assume $\mu_{0}\in(0,n)$. Let
$\Gamma_{\psi,\phi}:=\\{\textbf{x}(p)+(\mathcal{E}(\psi)+\phi)y(p)\mkern
1.5mu\overline{\mkern-1.5mu\textbf{n}\mkern-1.5mu}\mkern 1.5mu_{\Gamma}(p):\
p\in\Gamma\\}$ for $\phi\in y^{\mu_{0}}\Lambda_{0}^{m,\alpha}(\Gamma)$. Note
that for $\mu_{0}$ close to $n$, $y^{\mu_{0}+1}\Lambda^{\ell,\alpha}_{0}$ is
continuously embedded into $C^{\ell,\alpha}(\mkern
1.5mu\overline{\mkern-1.5mu\Gamma\mkern-1.5mu}\mkern 1.5mu)$ whenever
$\ell\leq n$. Since $\phi\in y^{\mu_{0}}\Lambda^{m,\alpha}_{0}$, we see that
$\Gamma_{\psi,\phi}:=\\{\textbf{x}(p)+(\mathcal{E}(\psi)+\phi)y(p)\mkern
1.5mu\overline{\mkern-1.5mu\textbf{n}\mkern-1.5mu}\mkern 1.5mu_{\Gamma}(p):\
p\in\Gamma\\}$ is $C^{m,\alpha}$ up to the ideal boundary.

The mean curvature of $\Gamma_{\psi,\phi}$, $H_{\psi,\phi}$, is in
$y^{\mu_{0}}\Lambda^{m-2,\alpha}_{0}$. Hence, $H_{\cdot,\cdot}$ maps a
neighborhood of $(0,0)$ in $C^{m,\alpha}(M)\times
y^{\mu_{0}}\Lambda_{0}^{m,\alpha}(\Gamma)$ to
$y^{\mu_{0}}\Lambda_{0}^{m-2,\alpha}(\Gamma)$. By the implicit function
theorem, there exists a map $\mathcal{K}$ mapping from a neighborhood of the
origin in $C^{m,\alpha}(M)$ to $y^{\mu_{0}}\Lambda_{0}^{m,\alpha}(\Gamma)$,
such that $H_{\psi,\mathcal{K}(\psi)}\equiv 0$. Now, pick $\psi$ so that
$M_{\psi}=M^{\prime}\in\mathcal{U}$ is a regular value. Then,
$t\mapsto\Gamma^{i}_{t\psi,\mathcal{K}(t\psi)}$ is a $C^{m,\alpha}$ isotopy
between $\Gamma$ and another minimal hypersurface
$\Gamma_{\psi,\mathcal{K}(\psi)}$.

Now, assume $\Gamma$ is weakly stable. Write $\Gamma$ into disjoint components
$\Gamma=\sqcup\Gamma^{i}$. Let $M^{i}=\partial_{\infty}\Gamma^{i}$. We proceed
as in the preceding paragraph with $\Gamma$ replaced by $\Gamma^{i}$. Hence,
for functions $\psi\in C^{m,\alpha}(\Gamma^{i})$ and $\phi\in
y^{\mu_{0}}\Lambda_{0}^{m,\alpha}$, there are a closed submanifold
$M^{i}_{\psi}$ with

$M^{i}_{\psi}=\big{\\{}\textbf{x}(p)+\psi(p)\nu_{M^{i}}(p):\,p\in
M^{i}\big{\\}}$

and an approximate minimal hypersurface
$\Gamma^{i}_{\psi,\phi}:=\\{\textbf{x}(p)+(\mathcal{E}(\psi)+\phi)y(p)\mkern
1.5mu\overline{\mkern-1.5mu\textbf{n}\mkern-1.5mu}\mkern
1.5mu_{\Gamma^{i}}(p):\ p\in\Gamma^{i}\\}$ with the mean curvature,
$H_{\psi,\phi}$, lying in $y^{\mu_{0}}\Lambda^{m-2,\alpha}_{0}$. If
$\Gamma^{i}$ is strictly stable, the isotopy follows from the preceding
paragraph.

If $\Gamma^{i}$ is weakly stable, since $\lambda=0$ corresponds to the first
eigenvalue of $\Gamma^{i}$, we know that
$\text{Ker}(L_{\Gamma^{i}})=\text{Span}\\{f\\}$. By Proposition 8.1, $f$ is
positive with $\frac{1}{C}y^{n}\leq f\leq Cy^{n}$. Let
$X_{\Gamma^{i}}:=\\{\phi\in
y^{\mu_{0}}\Lambda^{m,\alpha}_{0}:\,\int_{\Gamma^{i}}\phi
fd\mathcal{H}^{n}_{\mathbb{H}}=0\\}$. Since $f\in
y^{n}\Lambda^{m,\alpha}_{0}$, the integral of $\phi f$ over $\Gamma^{i}$ is
well defined. Moreover, $X_{\Gamma^{i}}$ is a closed subspace of
$y^{\mu_{0}}\Lambda^{m,\alpha}_{0}$. Let

$P_{\Gamma^{i}}:\,y^{\mu_{0}}\Lambda_{0}^{m-2,\alpha}\to
X_{\Gamma^{i}},\,g\mapsto
g-\frac{1}{\|f\|_{L^{2}(\Gamma^{i})}^{2}}\left\langle
f,g\right\rangle_{L^{2}(\Gamma^{i})}f$

be the projection map. It is readily checked that $P_{\Gamma^{i}}$ is a
bounded linear map. By implicit function theorem, there exists a map
$\mathcal{K}$ mapping from a neighborhood of the origin in
$C^{m,\alpha}(M^{i})$ to $X_{\Gamma^{i}}$, such that $P_{\Gamma^{i}}\circ
H_{\psi,\mathcal{K}(\psi)}\equiv 0$. Now, pick $\psi$ so that $M^{i}_{\psi}$
is a regular value. Then, $t\mapsto\Gamma^{i}_{t\psi,\mathcal{K}(t\psi)}$ is a
$C^{m,\alpha}$ isotopy between $\Gamma^{i}$ and
$\Gamma^{i}_{\psi,\mathcal{K}(\psi)}$, where $m=\min\\{k,n\\}$. By Lemma 7.2,
we can assume
$\lambda_{\mathbb{H}}[\Gamma^{i}_{\psi,\mathcal{K}(\psi)}]<\lambda[\mathbb{S}^{n-1}\times\mathbb{R}^{1}]$
by choosing $\psi$ sufficiently small in $C^{m,\alpha}$-norm. Since
$P_{\Gamma^{i}}\circ H_{\psi,\mathcal{K}(\psi)}=0$, the mean curvature
$H_{\psi,\mathcal{K}(\psi)}=cf$ for some constant $c\in\mathbb{R}^{1}$. Pick
an appropriate direction of normal vector, we assume that $c>0$. Applying
Theorem 8.6, we get an isotopy between $\Gamma^{i}_{\psi,\mathcal{K}(\psi)}$
and a minimal hypersurface $\widetilde{\Gamma}^{i}$ with
$\partial_{\infty}\widetilde{\Gamma}^{i}=M^{i}_{\psi}$.

In either case, we have proved that $\Gamma$ is isotopic to another minimal
hypersurface $\Gamma^{\prime}$ with
$M^{\prime}=\partial_{\infty}\Gamma^{\prime}$ being a regular value. This
finishes the proof Theorem 1.3. ∎

##  Appendix A Computations for non-compactly supported vector fields

Let $\textbf{Y}=\alpha\textbf{N}_{\epsilon}+\textbf{V}\in\mathcal{Y}^{-}$ and
$\\{\Phi_{t}(p)\\}$ be the $1$-parameter family of diffeomorphisms generated
by Y throughout the appendix. Assume $\|\textbf{Y}\|_{\mathcal{Y}^{-}}\leq
M_{0}$.

######  Lemma A.1.

There is a constant $C=C(\Gamma_{-},\Omega^{\prime},M_{0})$, so that

(A.1) $\displaystyle|\mathbf{Y}(p)-(\alpha\mathbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1})\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}|\leq
C(\mathbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1})^{2},$
(A.2) $\displaystyle|\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu\mathbf{Y}-\alpha\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}\otimes\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}|\leq
C\mathbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}\text{ and
}|\mkern 1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu^{2}\mathbf{Y}|\leq C$

###### Proof.

From the estimate on V, it suffices to show that
$|\textbf{N}_{\epsilon}-(\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1})\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}|\leq
C(\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1})^{2}$,
that $|\mkern 1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu\textbf{N}_{\epsilon}-\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}\otimes\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}|\leq
C\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}$ and that
$|\mkern 1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu^{2}\textbf{N}_{\epsilon}|\leq C$. The estimate $|\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu^{2}\textbf{N}_{\epsilon}|\leq C$ follows from the fact that $\Gamma_{-}$
is $C^{3}$-asymptotic. In
$\text{cl}(\Omega^{\prime})\cap\\{y<\frac{\epsilon}{2}\\}$,
$\textbf{N}_{\epsilon}=(\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1})\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu^{\top}_{n+1}\circ\Pi$. So, near the ideal boundary, we have

$\displaystyle|\textbf{N}_{\epsilon}-(\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1})\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}|=(\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1})|\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu^{\top}_{n+1}\circ\Pi-\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}|.$

Since $\mkern 1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu^{\top}_{n+1}=\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}$ on
$\mkern 1.5mu\overline{\mkern-1.5mu\Gamma_{-}\mkern-1.5mu}\mkern
1.5mu\cap\\{y=0\\}$, (A.1) follows easily. On the other hand, near the ideal
boundary, we have

$\displaystyle\mkern 1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu\textbf{N}_{\epsilon}=\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}\otimes\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}+(\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1})\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern 1.5mu(\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu^{\top}_{n+1}\circ\Pi).$

Since $\Gamma_{-}$ is $C^{3}$-asymptotic, $\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu^{\top}_{n+1}\circ\Pi$ is $C^{2}$ up to the ideal boundary. So, $|\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern 1.5mu(\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu^{\top}_{n+1}\circ\Pi)|\leq C$. This completes the proof. ∎

######  Lemma A.2.

Let $0\leq t\leq T<\infty$. There is a constant
$C=C(\Gamma_{-},\Omega^{\prime},M_{0},T)$, so that

(A.3) $\displaystyle\left|\frac{\mathbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}-e^{-\alpha t}\right|$ $\displaystyle\leq
C\mathbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}$ (A.4)
$\displaystyle|\mkern 1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu\Phi_{t}-I_{n+1}-(e^{\alpha t}-1)\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}\otimes\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}|$
$\displaystyle\leq C\mathbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}\text{ and
}|\mkern 1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu^{2}\Phi_{t}|\leq C.$

###### Proof.

Since $\frac{d}{dt}\Phi_{t}(p)=\textbf{Y}(\Phi_{t}(p))$, using (A.1), one sees
that

$\displaystyle\left|\frac{d}{dt}\left(\frac{\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}\right)\right|\leq C\left|\frac{\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}}\right|.$

So,

(A.5) $|\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}|\leq
C\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}$

Similarly, one can compute that

$\displaystyle\frac{d}{dt}\left(\frac{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}\right)=-\frac{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{(\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1})^{2}}\textbf{Y}(\Phi_{t}(p))\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}=-\alpha\frac{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}+\mathcal{O}(\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}),$

where in the last step, we used (A.1). This implies that

$\left|\frac{d}{dt}\left(e^{\alpha t}\frac{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}\right)\right|\leq C(\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}).$

Taking the integral and noticing that $\Phi_{0}$ is the identity map, we get
(A.3). To prove (A.4), notice that $\frac{d}{dt}\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern 1.5mu\Phi_{t}=\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu\frac{d}{dt}\Phi_{t}=\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu(\textbf{Y}(\Phi_{t}(p)))=\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu\textbf{Y}\circ\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern 1.5mu\Phi_{t}$. Combining
this with (A.2),

$\displaystyle\left|\frac{d}{dt}\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu\Phi_{t}-\alpha(\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}\otimes\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1})\circ\mkern 1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu\Phi_{t}\right|\leq C\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}$

That is,

(A.6) $\displaystyle\left|\frac{d}{dt}\left(e^{-\alpha tA}\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu\Phi_{t}\right)\right|\leq C\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}e^{-\alpha
tA},$

where $A=\mkern 1.5mu\overline{\mkern-1.5mu\mathbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}\otimes\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}$. As a
result, $e^{-\alpha tA}=\sum_{k=0}^{\infty}(-\alpha
tA)^{k}/k!=I_{n+1}+(e^{-\alpha t}-1)A$. Taking the integral in (A.6),

$\displaystyle|\mkern 1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu\Phi_{t}-I_{n+1}-(e^{\alpha t}-1)\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}\otimes\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}|\leq
C\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}\leq
C\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1},$

where in the last step, we used (A.5). Finally, we show $|\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern 1.5mu^{2}\Phi_{t}|\leq
C$. Since

$\displaystyle\frac{d}{dt}\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern 1.5mu^{2}\Phi_{t}=\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu^{2}\frac{d}{dt}\Phi_{t}=\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu^{2}\textbf{Y}(\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern 1.5mu\Phi_{t},\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern 1.5mu\Phi_{t})+\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu\textbf{Y}\circ\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern 1.5mu^{2}\Phi_{t},$

we have $|\frac{d}{dt}\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern 1.5mu^{2}\Phi_{t}|\leq
C(1+|\mkern 1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu^{2}\Phi_{t}|)$. This implies $|\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern 1.5mu^{2}\Phi_{t}|\leq
C$. ∎

######  Lemma A.3.

$\mathcal{J}^{E}\Phi_{t}(p,\mathbf{v})\in\mathfrak{Y}$. Moreover,
$\|\mathcal{J}^{E}\Phi_{t}(p,\mathbf{v})\|_{\mathfrak{Y}}\leq C$, where
$C=C(\Gamma_{-},\Omega^{\prime},M_{0},T)$.

###### Proof.

Since $\mathfrak{Y}$ is an algebra and
$\mathcal{J}^{E}\Phi_{t}=\mathcal{J}\Phi_{t}\left(\frac{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}\right)^{n}$, it suffices to show that
$\mathcal{J}\Phi_{t}\in\mathfrak{Y}$ and $\frac{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}\in\mathfrak{Y}$. By direct computations,

$\displaystyle(\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1})\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu\left(\frac{(\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1})}{\left(\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}\right)}\right)$ $\displaystyle=\frac{(\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1})}{\left(\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}\right)}\left(\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}-(\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1})\frac{\mkern 1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}}\right)$
$\displaystyle=-\frac{(\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1})}{\left(\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}\right)}\int_{0}^{t}\big{(}\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}\big{)}\frac{d}{d\tau}\Big{(}\frac{\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu\Phi_{\tau}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\Phi_{\tau}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}\Big{)}d\tau.$

Notice that

$\displaystyle\frac{d}{d\tau}\Big{(}\frac{\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu\Phi_{\tau}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\Phi_{\tau}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}\Big{)}=\frac{\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu\textbf{Y}(\Phi_{\tau}(p))\circ\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu\Phi_{\tau}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\Phi_{\tau}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}-\frac{\mkern 1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu\Phi_{\tau}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\big{(}\Phi_{\tau}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}\big{)}^{2}}\big{(}\textbf{Y}(\Phi_{\tau}(p))\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}\big{)}.$

From (A.1), (A.2), (A.3) and (A.4), one sees that

(A.7) $\left|(\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1})\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu\left(\frac{(\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1})}{\left(\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}\right)}\right)\right|\leq C\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}.$

Write

(A.8) $\frac{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}-e^{-\alpha t}=\phi_{y_{1},\delta}(\frac{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}-e^{-\alpha
t})+(1-\phi_{y_{1},\delta})(\frac{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}-e^{-\alpha t}).$

Denote the second term on the right hand side of (A.8) by $g_{y_{1}}$.
Combining (A.3) and (A.7), one sees that $\|g_{y_{1}}\|_{\mathfrak{X}}\leq
C(\Gamma_{-},\Omega^{\prime},M_{0},T)y_{1}$. Hence,
$\frac{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}-e^{-\alpha t}$ can be approximated by compactly supported
functions in $\mathfrak{X}$-norm. This implies $\frac{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}\in\mathfrak{Y}$.

We next prove $\mathcal{J}\Phi_{t}(p,\textbf{v})\in\mathfrak{Y}$. For
$\textbf{v}\in\mathbb{S}^{n}$, we choose a local orthogonal frame
$\\{\tau_{1}(\textbf{v}),\cdots,\tau_{n}(\textbf{v})\\}$ near v such that
$\tau_{i}$ is smooth and
$|\nabla_{\mathbb{S}^{n}}\tau_{i}|+|\nabla^{2}_{\mathbb{S}^{n}}\tau_{i}|\leq
C$, where $C$ is a constant independent of $p$. Combining (A.1), (A.2) and
(A.4), one can prove in a similar way that

(A.9) $\displaystyle\mathcal{J}\Phi_{t}-\sqrt{1+(e^{\alpha t}-1)(1-(\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}\cdot\textbf{v})^{2})}=\phi_{y_{1},\delta}\left(\mathcal{J}\Phi_{t}-\sqrt{1+(e^{\alpha
t}-1)(1-(\mkern 1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}\cdot\textbf{v})^{2})}\right)+h_{y_{1}},$

where $h_{y_{1}}$ satisfies $\|h_{y_{1}}\|_{\mathfrak{X}}\leq
C(\Gamma_{-},\Omega^{\prime},M_{0},T)y_{1}$. This implies
$\mathcal{J}\Phi_{t}\in\mathfrak{Y}$. ∎

######  Lemma A.4.

The map $t\mapsto\mathcal{J}^{E}\Phi_{t}(p,\textbf{v})$ is in
$C^{1}([0,\infty);\mathfrak{Y})$. Moreover,

$\frac{d}{dt}\big{|}_{t=0}\mathcal{J}^{E}\Phi_{t}=\mkern
1.5mu\overline{\mkern-1.5mu\mathrm{div}\mkern-1.5mu}\mkern
1.5mu\mathbf{Y}-Q_{\mathbf{Y}}-\frac{n}{y}\mathbf{Y}\cdot\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1},$

where $Q_{\mathbf{Y}}(p,\mathbf{v})=\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu_{\mathbf{v}}\mathbf{Y}(p)\cdot\mathbf{v}$.

###### Proof.

To show the differentiability of the map
$t\mapsto\mathcal{J}^{E}\Phi_{t}(p,\textbf{v})$, it suffices to show the
differentiability of the maps $t\mapsto\frac{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}}$ and
$t\mapsto\mathcal{J}\Phi_{t}(p,\textbf{v})$.

Fix $t_{0}\in[0,\infty)$. Notice that

$\displaystyle\frac{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}-\frac{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\Phi_{t_{0}}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}=-\int_{t_{0}}^{t}\frac{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{(\Phi_{\tau}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1})^{2}}\left(\textbf{Y}(\Phi_{\tau}(p))\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}\right)d\tau.$

Let

$g_{t_{0},t}(p)=\frac{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{(\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1})^{2}}\left(\textbf{Y}(\Phi_{t}(p))\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}\right)-\frac{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{(\Phi_{t_{0}}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1})^{2}}\left(\textbf{Y}(\Phi_{t_{0}}(p))\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}\right).$

To show the differentiability of $t\mapsto\frac{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}}$, it
suffices to show $\|g_{t_{0},t}\|_{\mathfrak{Y}}\to 0$ as $t\to t_{0}$.
Firstly, we have $\|\phi_{y_{1},\delta}g_{t_{0},t}\|_{\mathfrak{X}}\to 0$ as
$t\to t_{0}$ for any fixed $y_{1}>\delta>0$. Then, from (A.1)-(A.4),
$\|(1-\phi_{y_{1},\delta})g_{t_{0},t}\|_{\mathfrak{Y}}\leq Cy_{1}$. So,
$\|g_{t_{0},t}\|_{\mathfrak{Y}}\to 0$ is proved.

Similarly, if we let $\\{\tau_{1}(\textbf{v}),\cdots,\tau_{n}(\textbf{v})\\}$
be a local orthogonal frame in $\mathbb{S}^{n}$ near v, then

$\mathcal{J}\Phi_{t}(p,\textbf{v})-\mathcal{J}\Phi_{t_{0}}(p,\textbf{v})=\int_{t_{0}}^{t}\frac{1}{2\mathcal{J}\Phi_{r}}\frac{d}{dr}\left(\det(\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu_{\tau_{i}}\Phi_{r}\cdot\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu_{\tau_{j}}\Phi_{r})\right)dr.$

Since $\frac{d}{dr}\left(\det(\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu_{\tau_{i}}\Phi_{r}\cdot\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu_{\tau_{j}}\Phi_{r})\right)$ is a polynomial of $(\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu_{k}\textbf{Y}\cdot\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu_{\tau_{j}}\Phi_{r})(\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu_{\tau_{i}}\Phi_{r}\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{k})$ ($1\leq
i,j\leq n$ and $1\leq k\leq n+1$). So, the differentiability of
$t\mapsto\mathcal{J}\Phi_{t}(p,\textbf{v})$ is equivalent to the following

$\displaystyle\|(\mkern 1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu_{k}\textbf{Y}\cdot\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu_{\tau_{j}}\Phi_{t})(\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu_{\tau_{i}}\Phi_{t}\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{k})-(\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu_{k}\textbf{Y}\cdot\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu_{\tau_{j}}\Phi_{t_{0}})(\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu_{\tau_{i}}\Phi_{t_{0}}\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{k})\|_{\mathfrak{X}}\to 0;$
$\displaystyle\|\frac{1}{\mathcal{J}\Phi_{t}(p,\textbf{v})}-\frac{1}{\mathcal{J}\Phi_{t_{0}}(p,\textbf{v})}\|_{\mathfrak{X}}\to
0\text{ as }t\to t_{0}.$

They can be verified using a cut-off function similar to the proof of the
differentiability of $t\mapsto\frac{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}}$.
Finally, letting $t_{0}=0$, one sees that

$\frac{d}{dt}\Big{|}_{t=0}\frac{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}=\frac{\textbf{Y}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}},\
\frac{d}{dt}\Big{|}_{t=0}\mathcal{J}\Phi_{t}(p,\textbf{v})=\sum_{i=1}^{n}\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu_{\tau_{i}}\textbf{Y}\cdot\tau_{i}=\mkern
1.5mu\overline{\mkern-1.5mu\text{div}\mkern-1.5mu}\mkern
1.5mu(\textbf{Y})-\mkern 1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu_{\textbf{v}}\textbf{Y}\cdot\textbf{v}.$

This implies $\frac{d}{dt}\big{|}_{t=0}\mathcal{J}^{E}\Phi_{t}=\mkern
1.5mu\overline{\mkern-1.5mu\mathrm{div}\mkern-1.5mu}\mkern
1.5mu\mathbf{Y}-Q_{\mathbf{Y}}-\frac{n}{y}\mathbf{Y}\cdot\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{e}\mkern-1.5mu}\mkern 1.5mu_{n+1}$. ∎

######  Lemma A.5.

Let $\psi\in\mathfrak{Y}$. There is a constant
$C=C(\Gamma_{-},\Omega^{\prime},M_{0},T)$, so that for $0\leq t\leq T$,
$\|\Phi_{t}^{\\#}\psi\|_{\mathfrak{Y}}\leq C\|\psi\|_{\mathfrak{Y}}$.

###### Proof.

Let $\psi=f+g$ with $\|f\|_{\mathfrak{X}}+\|y^{-n}g\|_{\infty}\leq
2\|\psi\|_{\mathfrak{Y}}$. By definition,
$\Phi_{t}^{\\#}\psi=\Phi_{t}^{\\#}f+\Phi_{t}^{\\#}g=f(\Phi_{t}(p),\nabla_{\textbf{v}}\Phi_{t}(p))+g(\Phi_{t}(p),\nabla_{\textbf{v}}\Phi_{t}(p))$.
So,

(A.10)
$\displaystyle\|y^{-n}\Phi_{t}^{\\#}g\|_{\infty}\leq\left\|\frac{\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}{\textbf{x}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1}}\right\|_{\infty}^{n}\|(\Phi_{t}(p)\cdot\mkern
1.5mu\overline{\mkern-1.5mu\textbf{e}\mkern-1.5mu}\mkern
1.5mu_{n+1})^{-n}\Phi_{t}^{\\#}g\|_{\infty}\leq C\|y^{-n}g\|_{\infty},$

where in the last step, we used (A.5). Similarly, using (A.3), (A.4) and
(A.5), we have

$\displaystyle\|y\nabla_{\mathbb{R}^{n+1}_{+}}(\Phi_{t}^{\\#}f)\|_{\infty}$
$\displaystyle\leq
C(\|y\nabla_{\mathbb{R}^{n+1}_{+}}f\|_{\infty}+\|\nabla_{\mathbb{S}^{n}}f\|_{\infty});$
$\displaystyle\|\nabla_{\mathbb{S}^{n}}(\Phi_{t}^{\\#}f)\|_{\infty}$
$\displaystyle\leq C\|\nabla_{\mathbb{S}^{n}}f\|_{\infty};$
$\displaystyle\|\nabla_{\mathbb{S}^{n}}\nabla_{\mathbb{S}^{n}}(\Phi_{t}^{\\#}f)\|_{\infty}$
$\displaystyle\leq
C(\|\nabla_{\mathbb{S}^{n}}f\|_{\infty}+\|\nabla_{\mathbb{S}^{n}}\nabla_{\mathbb{S}^{n}}f\|_{\infty});$
$\displaystyle\|y\nabla_{\mathbb{R}^{n+1}_{+}}\nabla_{\mathbb{S}^{n}}(\Phi_{t}^{\\#}f)\|_{\infty}$
$\displaystyle\leq
C(\|y\nabla_{\mathbb{R}^{n+1}_{+}}\nabla_{\mathbb{S}^{n}}f\|_{\infty}+\|\nabla_{\mathbb{S}^{n}}\nabla_{\mathbb{S}^{n}}f\|_{\infty}).$

This implies $\|\Phi_{t}^{\\#}f\|_{\mathfrak{X}}\leq C\|f\|_{\mathfrak{X}}$.
Combining this with (A.10), we see that

$\|\Phi_{t}^{\\#}\psi\|_{\mathfrak{Y}}\leq\|\Phi_{t}^{\\#}f\|_{\mathfrak{X}}+\|y^{-n}\Phi_{t}^{\\#}g\|_{\infty}\leq
C(\|f\|_{\mathfrak{X}}+\|y^{-n}g\|_{\infty})\leq 2C\|\psi\|_{\mathfrak{Y}}.$

∎

##  Appendix B Proof of Proposition 6.3

A proof to Proposition 6.3 is included here. We adapt the argument in [BW,
Section 8].

###### Proof of Proposition 6.3.

It suffices to prove the existence of a $\delta_{0}$, such that
$\max_{\tau\in[0,1]}E_{rel}[\Gamma_{\tau},\Gamma_{-}]\geq\delta_{0}$, since if
we swap $\Gamma_{-}$ with $\Gamma_{+}$, it can be proved in a similar way that
$\max_{\tau\in[0,1]}E_{rel}[\Gamma_{\tau},\Gamma_{+}]\geq\delta_{0}$, and this
implies $\max_{\tau\in[0,1]}E_{rel}[\Gamma_{\tau},\Gamma_{-}]\geq
E_{rel}[\Gamma_{+},\Gamma_{-}]+\delta_{0}$.

For any $U\in\mathcal{C}(\Gamma^{\prime}_{-},\Gamma^{\prime}_{+})$, let
$V_{U}=U\Delta U_{\Gamma_{-}}$ be the symmetric difference. Since
$\tau\mapsto\mathbf{1}_{U_{\tau}}$ is continuous in $L^{1}_{loc}$ and
$\text{Vol}_{\mathbb{H}}\big{(}(U_{\Gamma_{+}}\backslash
U_{\Gamma_{-}})\cap\\{y\leq\epsilon\\}\big{)}$ tends to $0$ as $\epsilon\to
0$, we see that $\text{Vol}_{\mathbb{H}}(V_{U_{\tau}})$ is a continuous
function in $\tau\in[0,1]$. Let $M_{0}=\text{Vol}_{\mathbb{H}}(V_{U_{1}})$ and

$\mathcal{C}_{m}=\\{U\in\mathcal{C}(\Gamma^{\prime}_{-},\Gamma^{\prime}_{+}):\,\text{Vol}_{\mathbb{H}}(V_{U})=m\\}.$

By Proposition 6.2, there is a sweep-out from $\Gamma_{-}$ to $\Gamma_{+}$.
So, $\mathcal{C}_{m}$ is non-empty for each $m\in[0,M_{0}]$. We will show that
for sufficiently small $m_{0}$, there is a
$\delta_{0}=\delta_{0}(\Gamma_{-},\Gamma_{+},\Gamma^{\prime}_{-},\Gamma^{\prime}_{+})$,
so that $E_{rel}[\partial^{*}U,\Gamma_{-}]\geq\delta_{0}$ for all
$U\in\mathcal{C}_{m_{0}}$. And this proves the proposition.

Fix $m\in[0,M_{0}]$. Let $\\{U^{i}\\}_{i=1}^{\infty}$ be a sequence of
Caccioppoli sets in $\mathcal{C}_{m}$, which minimizes the functional
$E_{rel}[\partial^{*}\cdot,\Gamma_{-}]$. By Proposition 2.1, there exists an
$E_{0}=E_{0}(\Gamma_{-},\Omega)$, so that
$E_{rel}[\partial^{*}U,\Gamma_{-}]\geq-E_{0}$ for all
$U\in\mathcal{C}(\Gamma^{\prime}_{-},\Gamma^{\prime}_{+})$. So,
$E_{m,\inf}:=\lim_{i\to\infty}E_{rel}[\partial^{*}U^{i},\Gamma_{-}]>-\infty$.
If $E_{m,\inf}=\infty$, we take $m_{0}$ to be this specific $m$ and
$\delta_{0}$ to be arbitrary positive real number. So, we assume that
$E_{m,\inf}\in(-\infty,\infty)$. In this case, by the compactness of BV-
functions, there is a subsequence of $\\{U^{i}\\}$ converging in $L^{1}_{loc}$
to a Caccioppoli set $U_{0}$. Then, $\text{Vol}_{\mathbb{H}}(V_{U_{0}})=m$ and
$E_{rel}[\partial^{*}U_{0},\Gamma_{-}]=E_{m,\inf}$. From Proposition 4.1, we
know that there is a mean-convex foliation,
$\\{\Gamma_{-}^{s}\\}_{s\in[-1,1]}$, of $\Gamma_{-}$ that covers $\mkern
1.5mu\overline{\mkern-1.5mu\mathcal{N}_{\epsilon}\mkern-1.5mu}\mkern
1.5mu(\Gamma_{-})$ for some $\epsilon>0$. We make a distinction between two
cases.

  * (1)

If $\mkern 1.5mu\overline{\mkern-1.5mu\partial^{*}U_{0}\mkern-1.5mu}\mkern
1.5mu\subset\mkern
1.5mu\overline{\mkern-1.5mu\mathcal{N}_{\epsilon}\mkern-1.5mu}\mkern
1.5mu(\Gamma_{-})$, let $U_{+}=U_{0}\cup U_{\Gamma_{-}}$ and $U_{-}=U_{0}\cap
U_{\Gamma_{-}}$. In $U_{+}\backslash U_{\Gamma_{-}}$, consider a vector field
$\textbf{X}_{+}$ that is equal to $n_{\Gamma_{-}^{s}}$ on $\Gamma_{-}^{s}$.
Since $U_{+}\backslash U_{\Gamma_{-}}\subset\mkern
1.5mu\overline{\mkern-1.5mu\mathcal{N}_{\epsilon}\mkern-1.5mu}\mkern
1.5mu(\Gamma_{-})$, $\textbf{X}_{+}$ is well defined in $U_{+}\backslash
U_{\Gamma_{-}}$. Direct computation shows
$\text{div}(\textbf{X}_{+})=H_{\Gamma_{-}^{s}}\geq 0$. Applying the divergence
theorem to $\phi_{y_{1},\delta}\textbf{X}_{+}$,

(B.1) $\displaystyle
E_{rel}[\partial^{*}U_{+},\Gamma_{-};\phi_{y_{1},\delta}]\geq$
$\displaystyle\int_{\partial^{*}U_{+}}\left\langle\phi_{y_{1},\delta}\textbf{X}_{+},n_{\partial^{*}U_{+}}\right\rangle
d\mathcal{H}^{n}_{\mathbb{H}}-\int_{\Gamma_{-}}\left\langle\phi_{y_{1},\delta}\textbf{X}_{+},n_{\Gamma_{-}}\right\rangle
d\mathcal{H}^{n}_{\mathbb{H}}$ $\displaystyle=$
$\displaystyle\int_{U_{+}\backslash
U_{\Gamma_{-}}}\text{div}(\phi_{y_{1},\delta}\textbf{X}_{+})d\text{Vol}_{\mathbb{H}}\geq\int_{U_{+}}\left\langle\nabla\phi_{y_{1},\delta},\textbf{X}_{+}\right\rangle
d\text{Vol}_{\mathbb{H}}$ $\displaystyle\geq$
$\displaystyle-\int_{U_{+}\backslash U_{\Gamma_{-}}\cap\\{y_{1}-\delta\leq
y\leq y_{1}\\}}\frac{C}{\delta}yd\text{Vol}_{\mathbb{H}}\geq-Cy_{1}.$

Letting $y_{1}$, $\delta\to 0$, we get
$E_{rel}[\partial^{*}U_{+},\Gamma_{-}]\geq 0$. Similarly, we can prove
$E_{rel}[\partial^{*}U_{-},\Gamma_{-}]\geq 0$. Notice that
$E_{rel}[\partial^{*}U_{+},\Gamma_{-}]+E_{rel}[\partial^{*}U_{-},\Gamma_{-}]=E_{rel}[\partial^{*}U_{0},\Gamma_{-}]$.
Hence, $E_{rel}[\partial^{*}U_{0},\Gamma_{-}]\geq 0$. The equality holds only
if $U_{0}\Delta U_{\Gamma_{-}}$ has measure zero. Since
$\text{Vol}_{\mathbb{H}}(V_{U_{0}})=m>0$, the equality can not hold. Thus,
$E_{rel}[\partial^{*}U_{0},\Gamma_{-}]>0$.

  * (2)

If $\mkern 1.5mu\overline{\mkern-1.5mu\partial^{*}U_{0}\mkern-1.5mu}\mkern
1.5mu\backslash\mkern
1.5mu\overline{\mkern-1.5mu\mathcal{N}_{\epsilon}\mkern-1.5mu}\mkern
1.5mu(\Gamma_{-})\neq\emptyset$, then $\big{(}\mkern
1.5mu\overline{\mkern-1.5mu\partial^{*}U_{0}\mkern-1.5mu}\mkern
1.5mu\backslash\mkern
1.5mu\overline{\mkern-1.5mu\mathcal{N}_{\epsilon}\mkern-1.5mu}\mkern
1.5mu(\Gamma_{-})\big{)}\subset U^{c}_{\Gamma_{-}}$ since
$U_{\Gamma_{-}}\backslash U_{\Gamma^{\prime}_{-}}\subset\mkern
1.5mu\overline{\mkern-1.5mu\mathcal{N}_{\epsilon}\mkern-1.5mu}\mkern
1.5mu(\Gamma_{-})$. Since $\Omega^{\prime}$ is a bounded in
$\mathbb{R}^{n+1}_{+}$, there is a $y_{0}>0$, so that
$V_{U_{0}}\subset\\{y\leq y_{0}\\}$. Let $\mkern
1.5mu\overline{\mkern-1.5mud\mkern-1.5mu}\mkern 1.5mu_{\Gamma_{-}}$ be the
distance to $\Gamma_{-}$ in Euclidean topology. By the coarea formula,

$\frac{1}{y_{0}}\int_{\frac{\epsilon}{4}}^{\frac{\epsilon}{2}}\mathcal{H}^{n}_{\mathbb{H}}(V_{U_{0}}\cap\\{\mkern
1.5mu\overline{\mkern-1.5mud\mkern-1.5mu}\mkern
1.5mu_{\Gamma_{-}}=r\\})dr\leq\text{Vol}(V_{U_{0}})=m.$

By the mean value theorem, we can pick an
$\epsilon_{1}\in(\frac{\epsilon}{4},\frac{\epsilon}{2})$, so that
$\mathcal{H}^{n}_{\mathbb{H}}(V_{U_{0}}\cap\\{\mkern
1.5mu\overline{\mkern-1.5mud\mkern-1.5mu}\mkern
1.5mu_{\Gamma_{-}}=\epsilon_{1}\\})\leq\frac{8y_{0}m}{\epsilon}$. Denote
$\text{cl}(\Omega^{\prime})\backslash\mkern
1.5mu\overline{\mkern-1.5mu\mathcal{N}\mkern-1.5mu}\mkern
1.5mu_{\epsilon_{1}}(\Gamma_{-})$ by $W_{\epsilon_{1}}$. Let
$U^{\prime}\in\mathcal{C}(\Gamma^{\prime}_{-},\Gamma^{\prime}_{+})$ satisfy
that $\mathcal{H}^{n}_{\mathbb{H}}(\partial^{*}U^{\prime}\cap
W_{\epsilon_{1}})$ minimizes the hyperbolic area among all the sets in
$\mathcal{C}(\Gamma^{\prime}_{-},\Gamma^{\prime}_{+})$ that coincide with
$U_{0}$ in $\mkern 1.5mu\overline{\mkern-1.5mu\mathcal{N}\mkern-1.5mu}\mkern
1.5mu_{\epsilon_{1}}(\Gamma_{-})\cap\text{cl}(\Omega^{\prime})$. We claim that
$U^{\prime}\subset\text{cl}(U_{\Gamma_{+}})$. Otherwise, an calibration
argument similar to (B.1) would imply
$E_{rel}[\partial^{*}U^{\prime},\Gamma_{-}]>E_{rel}[\partial^{*}(U^{\prime}\cap
U_{\Gamma_{+}}),\Gamma_{-}]$, which gives a contradiction. Then, the maximal
principle [SW89] implies that once $\partial^{*}U^{\prime}$ touches
$\Gamma_{+}$, it coincides with $\Gamma_{+}$ in a neighborhood. So,
$\partial^{*}U^{\prime}$ is stationary in $W_{\epsilon_{1}}$. If
$\text{Spt}(\partial^{*}U^{\prime})\subset\mkern
1.5mu\overline{\mkern-1.5mu\mathcal{N}_{\epsilon}\mkern-1.5mu}\mkern
1.5mu(\Gamma_{-})$, then applying case (1) to $\partial^{*}U^{\prime}$, we get
$E_{rel}[\partial^{*}U_{0},\Gamma]\geq
E_{rel}[\partial^{*}U^{\prime},\Gamma_{-}]>0$. Otherwise, we pick a point
$p_{0}\in\text{Spt}(\partial^{*}U^{\prime})\backslash\mkern
1.5mu\overline{\mkern-1.5mu\mathcal{N}_{\epsilon}\mkern-1.5mu}\mkern
1.5mu(\Gamma_{-})$. Then, $B_{\frac{\epsilon}{2}}(p_{0})\subset
W_{\epsilon_{1}}$. The monotonicity formula [Sim84, Section 17] implies
$\mathcal{H}^{n}_{\mathbb{H}}(\partial^{*}U^{\prime}\cap
W_{\epsilon_{1}})\geq\gamma_{0}$, where $\gamma_{0}=\gamma_{0}(\epsilon,n)$.
Hence, by letting $U^{\prime\prime}=U^{\prime}\cap\mkern
1.5mu\overline{\mkern-1.5mu\mathcal{N}\mkern-1.5mu}\mkern
1.5mu_{\epsilon_{1}}(\Gamma_{-})$, we have

$\displaystyle E_{rel}[\partial^{*}U^{\prime\prime},\Gamma_{-}]$
$\displaystyle\leq
E_{rel}[\partial^{*}U^{\prime},\Gamma_{-}]+\mathcal{H}^{n}_{\mathbb{H}}(V_{U_{0}}\cap\\{\mkern
1.5mu\overline{\mkern-1.5mud\mkern-1.5mu}\mkern
1.5mu_{\Gamma_{-}}=r\\})-\mathcal{H}^{n}_{\mathbb{H}}(\partial^{*}U^{\prime}\cap
W_{\epsilon_{1}})$ $\displaystyle\leq
E_{rel}[\partial^{*}U_{0},\Gamma_{-}]+\frac{8y_{0}m}{\epsilon}-\gamma_{0}.$

$E_{rel}[\partial^{*}U^{\prime\prime},\Gamma_{-}]\geq 0$ follows from a
calibration argument used in case (1). By choosing $m_{0}=m$ sufficiently
small, we see that $E_{rel}[\partial^{*}U_{0},\Gamma_{-}]>0$.

So, in either case, there is an $m_{0}\in[0,M_{0}]$, such that
$\delta_{0}=E_{rel}[\partial^{*}U_{0},\Gamma_{-}]=E_{m_{0},\inf}>0$. ∎

## References

  * [AM10] Spyridon Alexakis and Rafe Mazzeo.  Renormalized area and properly embedded minimal surfaces in hyperbolic 3-manifolds.  Communications in Mathematical Physics, 297(3):621–651, 2010.
  * [And82] Michael T. Anderson.  Complete minimal varieties in hyperbolic space.  Inventiones mathematicae, 69(3):477–494, Oct 1982.
  * [And83] Michael T. Anderson.  Complete minimal hypersurfaces in hyperbolic n-manifolds.  Commentarii Mathematici Helvetici, 58(1):264–290, Dec 1983.
  * [BCW] Jacob Bernstein, Letian Chen, and Lu Wang.  Existence of monotone morse flow lines of the expander functional.  In preparation.
  * [Ber21] Jacob Bernstein.  Colding minicozzi entropy in hyperbolic space.  Nonlinear Analysis, 210:112401, 2021.
  * [Ber22] Jacob Bernstein.  A sharp isoperimetric property of the renormalized area of a minimal surface in hyperbolic space.  Proc. Amer. Math. Soc., 2022.
  * [Bry88] Robert L. Bryant.  Surfaces in conformal geometry, volume 48 of Proc. Sympos. Pure Math. American Mathematical Society, Providence, RI, 1988.
  * [BS88] Colin Bennett and Robert Sharpley.  Interpolation of Operators, volume 129 of Pure and Applied Mathematics.  Academic Press, Boston, MA, 1988.
  * [BW] Jacob Bernstein and Lu Wang.  A mountain-pass theorem for asymptotically conical self-expanders.  Peking Math. J. To Appear.
  * [BW17] Jacob Bernstein and Lu Wang.  A topological property of asymptotically conical self-shrinkers of small entropy.  Duke Math. J., 166(3):403–435, 2017.
  * [BW18] Jacob Bernstein and Lu Wang.  An integer degree for asymptotically conical self-expanders.  https://arxiv.org/abs/1807.06494, 2018.  Preprint.
  * [BW19a] Jacob Bernstein and Lu Wang.  Smooth compactness for spaces of asymptotically conical self-expanders of mean curvature flow.  Int. Math. Res. Not. IMRN, 2021(12):9016–9044, 2019.
  * [BW19b] Jacob Bernstein and Lu Wang.  Topological uniqueness for self-expanders of small entropy.  https://arxiv.org/abs/1902.02642, 2019.  Preprint.
  * [BW20] Jacob Bernstein and Lu Wang.  Closed hypersurfaces of low entropy in $\mathbb{R}^{4}$ are isotopically trivial.  arXiv: Differential Geometry, 2020.
  * [BW21] Jacob Bernstein and Lu Wang.  The space of asymptotically conical self-expanders of mean curvature flow.  Math. Ann., 380:175–230, 2021.
  * [BW22] Jacob Bernstein and Lu Wang.  Relative expander entropy in the presence of a two-sided obstacle and applications.  Advances in Mathematics, 399:108284, 2022.
  * [CIIW13] Tobias Holck Colding, Tom Ilmanen, William P. Minicozzi II, and Brian White.  The round sphere minimizes entropy among closed self-shrinkers.  Journal of Differential Geometry, 95(1):53 – 69, 2013.
  * [CL03] Tobias Holck Colding and Camillo De Lellis.  The min–max construction of minimal surfaces.  Surveys in differential geometry, 8:75–107, 2003.
  * [CM12] Tobias Holck Colding and William P. Minicozzi II.  Generic mean curvature flow i: generic singularities.  Ann. of Math. (2), 175(2):755–833, 2012.
  * [Cos06] Baris Coskunuzer.  Generic uniqueness of least area planes in hyperbolic space.  Geometry & Topology, 10:401–412, 2006.
  * [Cos11] Baris Coskunuzer.  On the number of solutions to the asymptotic plateau problem.  Journal of Gökova Geometry Topology, 5:1–19, 2011.
  * [Cos14] Baris Coskunuzer.  Asymptotic plateau problem: a survey.  2014\.
  * [CY07] Binglong Chen and Le Yin.  Uniqueness and pseudolocality theorems of the mean curvature flow.  Communications in Analysis and Geometry, 15:435–490, 2007.
  * [DLR18] Camillo De Lellis and Jusuf Ramic.  Min-max theory for minimal hypersurfaces with boundary.  Annales de l’Institut Fourier, 68(5):1909–1986, 2018.
  * [DM88] E. B. Davies and Nikolaos Mandouvalos.  Heat kernel bounds on hyperbolic space and kleinian groups.  Proceedings of The London Mathematical Society, pages 182–208, 1988\.
  * [dOS98] Geraldo de Oliveira and Marc Soret.  Complete minimal surfaces in hyperbolic space.  Mathematische Annalen, 311(3):397–419, Jul 1998.
  * [EH89] Klaus Ecker and Gerhard Huisken.  Mean curvature evolution of entire graphs.  Ann. of Math., 130(3):453–471, 1989.
  * [Gab97] David Gabai.  On the geometric and topological rigidity of hyperbolic 3-manifolds.  Journal of the American Mathematical Society, 10(1):37–74, 1997\.
  * [Giu84] Enrico Giusti.  Minimal Surfaces and Functions of Bounded Variation.  Monographs in Mathematics. Birkhäuser Boston, 1984.
  * [GT83] D. Gilbarg and N.S. Trudinger.  Elliptic Partial Differential Equations of Second Order.  Die Grundlehren der mathematischen Wissenschaften : a series of comprehensive studies in mathematics. Springer-Verlag, 1983.
  * [HL87] Robert Hardt and Fang-Hua Lin.  Regularity at infinity for area-minimizing hypersurfaces in hyperbolic space.  Inventiones mathematicae, 88(1):217–224, 1987.
  * [HL11] Qing Han and Fang-Hua Lin.  Elliptic partial differential equations.  Courant lecture notes. Courant Institute of Mathematical Sciences, Robotics Lab, New York University, 2nd ed. edition, 2011.
  * [Hui86] Gerhard Huisken.  Contracting convex hypersurfaces in riemannian manifolds by their mean curvature.  Inventiones mathematicae, 84(3):463–480, Oct 1986.
  * [HW15] Zheng Huang and Biao Wang.  Counting minimal surfaces in quasi-fuchsian three-manifolds.  Transactions of the American Mathematical Society, 367(9):6063–6083, 2015.
  * [KZ18] Daniel Ketover and Xin Zhou.  Entropy of closed surfaces and min-max theory.  Journal of Differential Geometry, 110(1):31 – 71, 2018.
  * [Lin89a] Fang-Hua Lin.  Asymptotic-behavior of area-minimizing currents in hyperbolic space.  Communications on Pure and Applied Mathematics, 42(3):229–242, April 1989.
  * [Lin89b] Fang-Hua Lin.  On the dirichlet problem for minimal graphs in hyperbolic space.  Inventiones mathematicae, 96(3):593–612, 1989.
  * [LT13] Camillo De Lellis and Dominik Tasnady.  The existence of embedded minimal hypersurfaces.  Journal of Differential Geometry, 95(3):355 – 388, 2013.
  * [LY82] Peter Li and Shing-Tung Yau.  A new conformal invariant and its applications to the willmore conjecture and the first eigenvalue of compact surfaces.  Inventiones mathematicae, 69(2):269–291, Jun 1982.
  * [Maz91] Rafe Mazzeo.  Elliptic theory of differential edge operators i.  Communications in Partial Differential Equations, 16(10):1615–1664, 1991.
  * [Mil15] J. Milnor.  Lectures on the H-Cobordism Theorem.  Princeton Legacy Library. Princeton University Press, 2015.
  * [MN14] Fernando C. Marques and André Neves.  Min-max theory and the willmore conjecture.  Annals of Mathematics, 179(2):683–782, 2014.
  * [MW13] Francisco Martin and Brian White.  Properly embedded, area-minimizing surfaces in hyperbolic 3-space.  arXiv: Differential Geometry, 2013.
  * [Ngu21] Manh Tien Nguyen.  Weighted monotonicity theorems and applications to minimal surfaces in hyperbolic space.  https://arxiv.org/abs/2105.12625, 2021.  Preprint.
  * [Pit14] J.T. Pitts.  Existence and Regularity of Minimal Surfaces on Riemannian Manifolds. (MN-27).  Mathematical Notes. Princeton University Press, 2014.
  * [RW99] C. Robin Graham and Edward Witten.  Conformal anomaly of submanifold observables in ads/cft correspondence.  Nuclear Physics B, 546(1):52–64, 1999.
  * [Sim84] Leon Simon.  Lectures on Geometric Measure Theory.  Proceedings of the Centre of Mathematical Analysis, Australian National University. Australian National University, Centre of Mathematical Analysis, Canberra, 1984.
  * [Sma65] S. Smale.  An infinite dimensional version of sard’s theorem.  American Journal of Mathematics, 87(4):861–866, 1965.
  * [Smi83] Francis R. Smith.  On the existence of embedded minimal 2-spheres in the 3-sphere, endowed with an arbitrary metric.  Bulletin of the Australian Mathematical Society, 28(1):159–160, 1983.
  * [SS81] Richard M. Schoen and Leon Simon.  Regularity of stable minimal hypersurfaces.  Communications on Pure and Applied Mathematics, 34:741–797, 1981\.
  * [SW89] Bruce Solomon and Brian White.  A strong maximum principle for varifolds that are stationary with respect to even parametric elliptic functionals.  Indiana Univ. Math. J., 38(3):683–691, 1989.
  * [Ton96] Yoshihiro Tonegawa.  Existence and regularity of constant mean curvature hypersurfaces in hyperbolic space.  Mathematische Zeitschrift, 221(1):591–615, 1996.
  * [Ton19] Yoshihiro Tonegawa.  Brakke’s Mean Curvature Flow: An Introduction.  Springer Briefs in Mathematics. Springer, Singapore, 2019.
  * [Tre06] Francois Treves.  Topological Vector Spaces, Distributions and Kernels.  Dover books on mathematics. Dover Publications, 2006.
  * [Whi91] Brian White.  Existence of smooth embedded surfaces of prescribed genus that minimize parametric even elliptic functionals on 3-manifolds.  Journal of Differential Geometry, 33(2):413–443, 1991.
  * [Whi97] Brian White.  Stratification of minimal surfaces, mean curvature flows, and harmonic maps.  Journal für die reine und angewandte Mathematik, 488:1–36, 1997\.
  * [Whi05] Brian White.  A local regularity theorem for mean curvature flow.  Annals of mathematics, ISSN 0003-486X, Vol. 161, Nº 3, 2005, pags. 1487-1519, 161, 05 2005.
  * [Yao] Junfu Yao.  Relative entropy of hypersurfaces in hyperbolic space.  In preparation.


Jean Pachebat<EMAIL_ADDRESS>
Sergey Ivanov<EMAIL_ADDRESS>
Criteo AI Lab, Paris, France

# High-Order Optimization of Gradient Boosted Decision Trees

###### Abstract

Gradient Boosted Decision Trees (GBDTs) are dominant machine learning
algorithms for modeling discrete or tabular data. Unlike neural networks with
millions of trainable parameters, GBDTs optimize loss function in an additive
manner and have a single trainable parameter per leaf, which makes it easy to
apply high-order optimization of the loss function. In this paper, we
introduce high-order optimization for GBDTs based on numerical optimization
theory which allows us to construct trees based on high-order derivatives of a
given loss function. In the experiments, we show that high-order optimization
has faster per-iteration convergence that leads to reduced running time. Our
solution can be easily parallelized and run on GPUs with little overhead on
the code. Finally, we discuss future potential improvements such as automatic
differentiation of arbitrary loss function and combination of GBDTs with
neural networks.

##  1 Introduction

Gradient boosted decision trees (GBDT) [Friedman et al.(2000)Friedman, Hastie,
and Tibshirani] are state-of-the-art ML models for tabular datasets. Many
variants of GBDTs have been proposed in the recent years that achieve top
performance in classification [He et al.(2014)He, Pan, Jin, Xu, Liu, Xu, Shi,
Atallah, Herbrich, Bowers, et al., Richardson et al.(2007)Richardson,
Dominowska, and Ragno], regression [Ivanov and Prokhorenkova(2021), Chen et
al.(2022)Chen, Mueller, Ioannidis, Adeshina, Wang, Goldstein, and Wipf], and
ranking [Gulin et al.(2011)Gulin, Kuralenok, and Pavlov, Ustimenko et
al.(2019)Ustimenko, Vorobev, Gusev, and Serdyukov] tasks with applications
where data often contain missing or noisy features, complex relationships, and
heterogeneity such as in recommender systems, information retrieval, and many
others [Chen et al.(2021b)Chen, Wang, Zhao, Zhang, Guo, Shen, Qu, Lu, Xu, Xu,
et al., Qin et al.(2021)Qin, Yan, Zhuang, Tay, Pasumarthi, Wang, Bendersky,
and Najork, Islam et al.(2020)Islam, Liu, Li, Liu, and Kang, Ozcaglar et
al.(2019)Ozcaglar, Geyik, Schmitz, Sharma, Shelkovnykov, Ma, and Buchanan,
Zhang and Haghani(2015)].

Let’s consider a dataset
$\mathcal{D}=\left\\{\left({{\mathbf{x}_{i}},{y_{i}}}\right)|i\in\left\\{{1,...,n}\right\\}\right\\}$,
where ${\mathbf{x}_{i}}\in{\mathbb{R}^{m}}$ and ${y_{i}}\in\mathbb{R}$. A GBDT
model is a sum of $K$ additive functions, each of which represents a decision
tree:

$\hat{y}_{i}=\phi(\mathbf{x}_{i})=\sum^{K}_{k=1}f_{k}(\mathbf{x}_{i}),\ \
f_{k}\in\mathcal{F},$ (1)

where
$\mathcal{F}=\\{f(\mathbf{x})=w_{q(\mathbf{x})}\\}(q:\mathbb{R}^{m}\rightarrow
T,w\in\mathbb{R}^{T})$ is a space of regression trees. Each regression tree
$f_{k}$ maps $m$-dimensional feature vector $\mathbf{x}$ to one of its leaves
(or regions) with index $q(\mathbf{x})$ that has a corresponding weight value
$w_{q(\mathbf{x})}$.

One of the most important differences between GBDT models and other tree-based
models such as Random Forest [Breiman(2001)] or Extremely Randomized Trees
[Geurts et al.(2006)Geurts, Ernst, and Wehenkel] is that GBDT model learns the
trees in an iterative fashion by adding a new tree with respect to the
performance of previous trees, while the latter models grow trees
independently from each other. In particular, in GBDT the loss objective
$\mathcal{L}^{(t)}$ at every iteration $t$ and regularization term $\Omega$ is
minimized by adding a new tree $f_{t}$:

$\mathcal{L}^{(t)}=\sum_{i=1}^{n}l(y_{i},\hat{y_{i}}^{(t-1)}+f_{t}(\mathbf{x}_{i}))+\Omega(f_{t})$
(2)

The regularization term $\Omega$ often penalizes the complexity of the tree
function and we consider it to be $\Omega(f_{t})=\frac{1}{2}\lambda\|w\|^{2}$,
where $\lambda$ is a hyperparameter.

In the seminal paper [Friedman(2001)] Friedman proposed to learn new trees
such that they produce the steepest-descent step direction given by the first-
order gradient of the loss function with respect to the last prediction of the
model:

$f_{t}=-\partial_{\hat{y}^{(t-1)}}l(y_{i},\hat{y}^{(t-1)})$ (3)

Thus, each new tree gets an updated version of the data
$\left\\{\left({{\mathbf{x}_{i}},{\tilde{y}_{i}}}\right)|i\in\left\\{{1,...,n}\right\\}\right\\}$,
where $\tilde{y}_{i}=\partial_{\hat{y}^{(t-1)}}l(y_{i},\hat{y}^{(t-1)})$ is a
“pseudoresponse” that the tree is fitting.

More than a decade after Chen and Guestrin proposed XGBoost [Chen and
Guestrin(2016)] that popularized second-order optimization of gradient
boosting. Instead of fitting the first-order gradient, each new tree
decomposes the loss function using second-order approximation and derives the
closed-form formulas for assigning the leaf weights and building a regression
tree.

Let $g^{(k)}_{i}$ be the $k$-th order derivative of the loss function with
respect to the last model’s prediction $\hat{y}^{(t-1)}$, i.e.
$g^{(k)}_{i}=\partial^{k}_{\hat{y}^{(t-1)}}l(y_{i},\hat{y}^{(t-1)})$. Then the
Eq. 2 can be approximated by second-order Taylor expansion:

$\mathcal{L}^{(t)}\simeq\sum_{i=1}^{n}[l(y_{i},\hat{y}^{(t-1)})+g^{(1)}_{i}f_{t}(\mathbf{x}_{i})+\frac{1}{2}g^{(2)}_{i}f_{t}^{2}(\mathbf{x}_{i})]+\Omega(f_{t})$
(4)

Let’s first assume that the tree structure is given to us and we want to find
the leaf weights that minimize the Eq. 4. Let
$I_{j}=\\{i|q(\mathbf{x}_{i})=j\\}$ be a set of all data instances that fall
into the leaf $j$. Then, removing the term that does not depend on the tree
$f_{t}$ and regrouping the data for each leaf we obtain:

$\begin{split}\tilde{\mathcal{L}}^{(t)}&=\sum^{n}_{i=1}[g^{(1)}_{i}f_{t}(\mathbf{x}_{i})+\frac{1}{2}g^{(2)}_{i}f_{t}^{2}(\mathbf{x}_{i})]+\frac{1}{2}\lambda\sum^{T}_{j=1}w_{j}^{2}\\\
&=\sum^{T}_{j=1}[(\sum_{i\in I_{j}}g^{(1)}_{i})w_{j}+\frac{1}{2}(\sum_{i\in
I_{j}}g^{(2)}_{i}+\lambda)w_{j}^{2}]\end{split}$ (5)

For each leaf $j$ let’s denote $G^{(k)}_{j}=\sum_{i\in I_{j}}g^{(k)}_{i}$ be
the sum of $k$-th order gradients of data instances that fall into that leaf.
Then the minimum of Eq. 5 for each leaf $j$ is given by

$w^{*}_{j}=-\frac{G^{(1)}_{j}}{G^{(2)}_{j}+\lambda},$ (6)

and by plugging it back to Eq.5 we get the optimal loss value:

$\mathcal{L}^{*}_{j}=-\frac{1}{2}\sum^{T}_{j=1}\frac{[G^{(1)}_{j}]^{2}}{G^{(2)}_{j}+\lambda}.$
(7)

To date, the power of second-order GBDT models [Ustimenko and
Prokhorenkova(2021), Sprangers et al.(2021)Sprangers, Schelter, and de Rijke,
Prokhorenkova et al.(2018)Prokhorenkova, Gusev, Vorobev, Dorogush, and Gulin,
Ke et al.(2017)Ke, Meng, Finley, Wang, Chen, Ma, Ye, and Liu, Chen and
Guestrin(2016)] has been demonstrated across a range of tasks and baselines
including modern neural networks [Grinsztajn et al.(2022)Grinsztajn, Oyallon,
and Varoquaux, Borisov et al.(2021)Borisov, Leemann, Seßler, Haug, Pawelczyk,
and Kasneci]. However, to the best of our knowledge, there are no works that
study high-order gradient information of the loss function during
optimization. In this work, we consider a high-order Taylor expansion of the
loss function (Eq. 2) and derive closed-form formulas for arbitrary order
gradient statistics and compare the results of these higher-order methods in
the experiments.

##  2 High-Order Optimization of GBDT

We start by noting that XGboost decomposition of the loss function Eq. 4
provides two advantages. First, it allows us to derive optimal leaf weights
for arbitrary differentiable loss functions. Second, the closed-form Eq. 6
provides a greedy algorithm to construct a tree. However, the second-order
Taylor approximation may lead to inaccurate estimation of the loss function
and therefore longer convergence. Next, we show an example of the optimal leaf
weights that use third-order approximation, which is followed by the general
$k$-th order methods.

###  2.1 Cubic Optimization of GBDT

Let’s start with a third-order Taylor expansion of the loss Eq. 2 (after
removing constant terms):

$\begin{split}\tilde{\mathcal{L}}^{(t)}&=\sum^{T}_{j=1}[G^{(1)}_{j}w_{j}+\frac{1}{2}(G^{(2)}_{j}+\lambda)w_{j}^{2}+\frac{1}{6}G^{(3)}_{j}w_{j}^{3}]\end{split}$
(8)

Here $G^{(k)}_{j}=\sum_{i\in I_{j}}g^{(k)}_{i}$ be the sum of $k$-th order
gradients of data instances in the leaf $j$. By equating the derivative to
zero, we can find the optimal weights $w_{j}$ for each leaf $j$:

$w^{*}_{j}=-\frac{G^{(2)}_{j}+\lambda}{G^{(3)}_{j}}\left(1\pm\sqrt{1-\frac{2G^{(1)}_{j}G^{(3)}_{j}}{(G^{(2)}_{j}+\lambda)^{2}}}\right)$
(9)

Note that at the minimum the first-order gradients $G^{(1)}_{j}$ are zero and
therefore for a small enough $\sum_{i\in I_{j}}g^{(1)}_{i}$ we can use an
expansion
$1-\sqrt{1-\alpha}=\frac{\alpha}{2}+\frac{\alpha^{2}}{8}+O(\alpha^{3})$ to
simplify the terms of this equation . The optimal weights become:

$w^{*}_{j}=-\frac{G^{(1)}_{j}}{G^{(2)}_{j}+\lambda}\left(1+\frac{G^{(1)}_{j}G^{(3)}_{j}}{2(G^{(2)}_{j}+\lambda)^{2}}\right)$
(10)

By plugging the weights back into the Eq. 8 we can compute the optimal loss
value:

$\begin{split}\mathcal{L}^{*}_{j}=-\frac{[G^{(1)}_{j}]^{2}}{G^{(2)}_{j}+\lambda}(\frac{1}{2}+\frac{1}{6}\varepsilon+2\varepsilon^{3}+\frac{2}{3}\varepsilon^{4})\end{split}$
(11)

Here, $\varepsilon=\dfrac{G^{(1)}_{j}G^{(3)}_{j}}{G^{(2)}_{j}+\lambda}$. By
comparing it to the Eq. 7 we can observe additional terms in Eq. 11 which
correct the loss value estimation. Similarly, to XGBoost approach we can
design an efficient greedy algorithm (Alg. 1) that estimate the goodness of
each split by computing the loss reduction score
$\mathcal{L}_{split}=\mathcal{L}_{node}–(\mathcal{L}_{left}+\mathcal{L}_{right})$
and then selecting the split with the maximum score. To make it efficient, the
algorithm first sorts the data on that node based on the feature values and
then computes the loss based on the Eq. 11. Similar algorithms can be designed
for arbitrary order $k$.

###  2.2 Householder Optimization of GBDT

Recap that the GBDT model is an additive process of adding new trees, each of
which approximates some function of the loss objective
$\phi(\mathbf{x})\leftarrow\phi(\mathbf{x})+f_{t}(\mathbf{x})$. In the case of
Friedman’s first-order method $f_{t}(\mathbf{x})$ approximates the gradient
Eq. 3. In the second and third-order methods, each tree $f_{t}(\mathbf{x})$
approximates functions that involve higher-order gradient statistics, given by
Eqs. 6 and 9. Furthermore, under some conditions of the regularity of the loss
function, Householder [Householder(1970)] gave the general formula for
arbitrary high order:

$\phi(\mathbf{x})\leftarrow\phi(\mathbf{x})+(p+1)\left(\frac{(1/g)^{(p)}}{(1/g)^{(p+1)}}\right)_{x_{n}},$
(12)

where $g$ is the gradient of the loss function and $(1/g)^{(p)}$ is the
derivative of order $p$ of inverse of $g$. The convergence has an order
$(p+2)$. For example, for $p=0$ Eq. 12 results in Newton-Raphson update of
XGBoost (Eq. 6). For $p=1$, Householder equation gives an update of order 3,
also known as Halley’s method [Boyd(2013)]:

$w^{*}_{j}=-\frac{G^{(1)}_{j}}{G^{(2)}_{j}+\lambda}\left(1-\frac{G^{(1)}_{j}G^{(3)}_{j}}{2(G^{(2)}_{j}+\lambda)^{2}}\right)^{-1}$
(13)

Given the approximation $(1-\alpha)^{-1}=1+\alpha+O(\alpha^{2})$ for small
$\alpha$, we can recover third-order update in Eq. 9. Finally, for $p=2$ we
obtain the following fourth-order update:

$\begin{split}w^{*}_{j}=&-G^{(1)}_{j}\left(\frac{(G^{(2)}_{j}+\lambda)^{2}-G^{(1)}_{j}G^{(3)}_{j}/2}{(G^{(2)}_{j}+\lambda)^{3}-G^{(1)}_{j}(G^{(2)}_{j}+\lambda)G^{(3)}_{j}+G^{(1)}_{j}G^{(4)}_{j}/6}\right).\end{split}$
(14)

##  3 Experiments

In the experiments, we use Eqs. 6, 13, 14 for the second, third, and fourth-
order models respectively. Our implementation is based on the open-source
package Py-Boost 111https://github.com/jpachebat/py_boost. We used four binary
classification datasets [Grinsztajn et al.(2022)Grinsztajn, Oyallon, and
Varoquaux] for which we optimize binary cross-entropy loss. For each dataset,
we use the validation part to select the best hyperparameters and use the test
part to measure the accuracy. To compare models’ efficiency we first trained
the second-order model (GBDT-2) for 10000 trees and then recorded the best
test accuracy it achieves. Then for each model, we measured the running time
it takes to reach 99% of the best accuracy so that the small perturbations in
the loss value can be discarded. The training was performed on GPU, with 120GB
memory, 7.8Gbps, and 2 GPUs. We kept the maximum depth of 6 for trees. In
contrary to second order, higher order GBDTs are more sensitive to
regularization factor $\lambda$, which we selected using hyperparameter search
in the range [$10^{0},\ldots,10^{6}$].

Convergence of validation accuracy is presented in the Fig. 1. We can see that
the third (GBDT-3) and fourth (GBDT-4) methods have much faster convergence
initially. However, as we train longer, the accuracy of GBDT-2 becomes
comparable to those of the higher-order methods (see Appendix B). On the other
hand, the running time to reach 99% optimal accuracy for higher-order is
significantly smaller as shown in Table 1. For example, on Epsilon dataset the
relative difference (Gap) between GBDT-2 model and GBDT-3 model is 73%.

[Epsilon] [Higgs]

Figure 1: Test accuracy for high-order GBDT models. Table 1: Running time to
achieve 99% of the optimal test accuracy for high-order GBDT models. Gap % is
the relative difference w.r.t. GBDT-2 accuracy (the smaller, the better).

|  | Higgs | Epsilon | Covertype | MiniBooNE
---|---|---|---|---|---
| Dataset | Time (s.) | Gap % | Time (s.) | Gap % | Time (s.) | Gap % | Time (s.) | Gap %
| GBDT-2 |  8.98 $\pm$ 0.01 | 0 |  94.80 $\pm$ 0.01 | 0 |  32.08 $\pm$ 0.01 | 0 |  1.35 $\pm$ 0.01 | 0
Ours | GBDT-3 |  4.93 $\pm$ 0.00 | -45 |  25.00 $\pm$ 0.01 | -73 |  18.46 $\pm$ 0.00 | -42 |  0.59 $\pm$ 0.01 | -56
GBDT-4 |  4.7 $\pm$ 0.1 | -47 |  29.17 $\pm$ 0.01 | -69 |  23.09 $\pm$ 0.00 | -28 |  0.48 $\pm$ 0.01 | -64

##  4 Conclusion

In this work, we proposed a high-order optimization framework to learn GBDT
model, which has not been explored in the context of gradient boosting and may
lead to many improvements to the existing GBDT algorithms: faster convergence,
automatic regularization of the step size, and better optima. There are many
exciting future directions for this research. Our framework is developed for
arbitrary differentiable loss objectives; however, the user still has to
provide manually-derived gradients in order to compute the optimal weights.
Recent interest in automatic and symbolic differentiation [Baydin et
al.(2018)Baydin, Pearlmutter, Radul, and Siskind] can come to the rescue,
especially in the case when the loss objective is highly non-linear and the
optimization order $p$ is high. Note, however, that automatic differentiation
additionally increases the running time so there is a trade-off between the
efficiency of higher-order optimization and its versatility. Going one step
further, we can implement a GBDT model directly in popular frameworks such as
PyTorch [Paszke et al.(2019)Paszke, Gross, Massa, Lerer, Bradbury, Chanan,
Killeen, Lin, Gimelshein, Antiga, et al.] or TensorFlow [Abadi et
al.(2016)Abadi, Barham, Chen, Chen, Davis, Dean, Devin, Ghemawat, Irving,
Isard, et al.] and leverage automatic differentiation, GPU acceleration,
distributed training, and many other features (see [Sprangers et
al.(2021)Sprangers, Schelter, and de Rijke] for an example). Finally, training
GBDT model with neural networks in an end-to-end fashion has recently
attracted attention [Ivanov and Prokhorenkova(2021), Chen et al.(2021a)Chen,
Mueller, Ioannidis, Adeshina, Wang, Goldstein, and Wipf] and is worth studying
in the context of high-order optimization.

## References

  * [Abadi et al.(2016)Abadi, Barham, Chen, Chen, Davis, Dean, Devin, Ghemawat, Irving, Isard, et al.] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al.  $\\{$TensorFlow$\\}$: a system for $\\{$Large-Scale$\\}$ machine learning.  In _12th USENIX symposium on operating systems design and implementation (OSDI 16)_ , pages 265–283, 2016.
  * [Baydin et al.(2018)Baydin, Pearlmutter, Radul, and Siskind] Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind.  Automatic differentiation in machine learning: a survey.  _Journal of Marchine Learning Research_ , 18:1–43, 2018\.
  * [Borisov et al.(2021)Borisov, Leemann, Seßler, Haug, Pawelczyk, and Kasneci] Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci.  Deep neural networks and tabular data: A survey.  _arXiv preprint arXiv:2110.01889_ , 2021.
  * [Boyd(2013)] John P Boyd.  Finding the zeros of a univariate equation: proxy rootfinders, chebyshev interpolation, and the companion matrix.  _SIAM review_ , 55(2):375–396, 2013.
  * [Breiman(2001)] Leo Breiman.  Random forests.  _Machine learning_ , 45(1):5–32, 2001.
  * [Chen et al.(2021a)Chen, Mueller, Ioannidis, Adeshina, Wang, Goldstein, and Wipf] Jiuhai Chen, Jonas Mueller, Vassilis N Ioannidis, Soji Adeshina, Yangkun Wang, Tom Goldstein, and David Wipf.  Convergent boosted smoothing for modeling graph data with tabular node features.  _arXiv preprint arXiv:2110.13413_ , 2021a.
  * [Chen et al.(2022)Chen, Mueller, Ioannidis, Adeshina, Wang, Goldstein, and Wipf] Jiuhai Chen, Jonas Mueller, Vassilis N. Ioannidis, Soji Adeshina, Yangkun Wang, Tom Goldstein, and David Wipf.  Does your graph need a confidence boost? convergent boosted smoothing on graphs with tabular node features.  In _International Conference on Learning Representations_ , 2022.
  * [Chen et al.(2021b)Chen, Wang, Zhao, Zhang, Guo, Shen, Qu, Lu, Xu, Xu, et al.] Mingcheng Chen, Zhenghui Wang, Zhiyun Zhao, Weinan Zhang, Xiawei Guo, Jian Shen, Yanru Qu, Jieli Lu, Min Xu, Yu Xu, et al.  Task-wise split gradient boosting trees for multi-center diabetes prediction.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, pages 2663–2673, 2021b.
  * [Chen and Guestrin(2016)] Tianqi Chen and Carlos Guestrin.  Xgboost: A scalable tree boosting system.  In _Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining_ , pages 785–794, 2016.
  * [Friedman et al.(2000)Friedman, Hastie, and Tibshirani] Jerome Friedman, Trevor Hastie, and Robert Tibshirani.  Additive logistic regression: a statistical view of boosting (with discussion and a rejoinder by the authors).  _The annals of statistics_ , 28(2):337–407, 2000\.
  * [Friedman(2001)] Jerome H Friedman.  Greedy function approximation: a gradient boosting machine.  _Annals of statistics_ , pages 1189–1232, 2001.
  * [Geurts et al.(2006)Geurts, Ernst, and Wehenkel] Pierre Geurts, Damien Ernst, and Louis Wehenkel.  Extremely randomized trees.  _Machine learning_ , 63(1):3–42, 2006.
  * [Grinsztajn et al.(2022)Grinsztajn, Oyallon, and Varoquaux] Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux.  Why do tree-based models still outperform deep learning on tabular data?  _arXiv preprint arXiv:2207.08815_ , 2022.
  * [Gulin et al.(2011)Gulin, Kuralenok, and Pavlov] Andrey Gulin, Igor Kuralenok, and Dimitry Pavlov.  Winning the transfer learning track of yahoo!’s learning to rank challenge with yetirank.  Proceedings of Machine Learning Research. PMLR, 25 Jun 2011.
  * [He et al.(2014)He, Pan, Jin, Xu, Liu, Xu, Shi, Atallah, Herbrich, Bowers, et al.] Xinran He, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi, Antoine Atallah, Ralf Herbrich, Stuart Bowers, et al.  Practical lessons from predicting clicks on ads at facebook.  In _Proceedings of the eighth international workshop on data mining for online advertising_ , pages 1–9, 2014.
  * [Householder(1970)] A.S. Householder.  _The Numerical Treatment of a Single Nonlinear Equation_.  International series in pure and applied mathematics. McGraw-Hill, 1970\.
  * [Islam et al.(2020)Islam, Liu, Li, Liu, and Kang] Md Zahidul Islam, Jixue Liu, Jiuyong Li, Lin Liu, and Wei Kang.  Evidence weighted tree ensembles for text classification.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 1737–1740, 2020\.
  * [Ivanov and Prokhorenkova(2021)] Sergei Ivanov and Liudmila Prokhorenkova.  Boost then convolve: Gradient boosting meets graph neural networks.  _arXiv preprint arXiv:2101.08543_ , 2021.
  * [Ke et al.(2017)Ke, Meng, Finley, Wang, Chen, Ma, Ye, and Liu] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu.  Lightgbm: A highly efficient gradient boosting decision tree.  _Advances in neural information processing systems_ , 30, 2017.
  * [Ozcaglar et al.(2019)Ozcaglar, Geyik, Schmitz, Sharma, Shelkovnykov, Ma, and Buchanan] Cagri Ozcaglar, Sahin Geyik, Brian Schmitz, Prakhar Sharma, Alex Shelkovnykov, Yiming Ma, and Erik Buchanan.  Entity personalized talent search models with tree interaction features.  In _The World Wide Web Conference_ , pages 3116–3122, 2019.
  * [Paszke et al.(2019)Paszke, Gross, Massa, Lerer, Bradbury, Chanan, Killeen, Lin, Gimelshein, Antiga, et al.] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32, 2019.
  * [Prokhorenkova et al.(2018)Prokhorenkova, Gusev, Vorobev, Dorogush, and Gulin] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin.  Catboost: unbiased boosting with categorical features.  _Advances in neural information processing systems_ , 31, 2018.
  * [Qin et al.(2021)Qin, Yan, Zhuang, Tay, Pasumarthi, Wang, Bendersky, and Najork] Zhen Qin, Le Yan, Honglei Zhuang, Yi Tay, Rama Kumar Pasumarthi, Xuanhui Wang, Michael Bendersky, and Marc Najork.  Are neural rankers still outperformed by gradient boosted decision trees?  In _International Conference on Learning Representations_ , 2021.
  * [Richardson et al.(2007)Richardson, Dominowska, and Ragno] Matthew Richardson, Ewa Dominowska, and Robert Ragno.  Predicting clicks: estimating the click-through rate for new ads.  In _Proceedings of the 16th international conference on World Wide Web_ , pages 521–530, 2007.
  * [Sprangers et al.(2021)Sprangers, Schelter, and de Rijke] Olivier Sprangers, Sebastian Schelter, and Maarten de Rijke.  Probabilistic gradient boosting machines for large-scale probabilistic regression.  In _Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining_, pages 1510–1520, 2021.
  * [Ustimenko and Prokhorenkova(2021)] Aleksei Ustimenko and Liudmila Prokhorenkova.  Sglb: Stochastic gradient langevin boosting.  In _International Conference on Machine Learning_ , pages 10487–10496. PMLR, 2021.
  * [Ustimenko et al.(2019)Ustimenko, Vorobev, Gusev, and Serdyukov] Aleksei Ustimenko, Aleksandr Vorobev, Gleb Gusev, and Pavel Serdyukov.  Learning to select for a predefined ranking.  In _International Conference on Machine Learning_ , pages 6477–6486. PMLR, 2019.
  * [Zhang and Haghani(2015)] Yanru Zhang and Ali Haghani.  A gradient boosting method to improve travel time prediction.  _Transportation Research Part C: Emerging Technologies_ , 58:308–324, 2015.

##  Appendix A Third-Order Split Selection Algorithm

Algorithm 1 Selecting optimal split for a tree node

Input: $I$, instances of the node

$score\leftarrow 0$
$G^{(i)}\leftarrow\sum_{i\in I}g^{(1)}_{i}$, $G^{(2)}\leftarrow\sum_{i\in
I}g^{(2)}_{i}$, $G^{(3)}\leftarrow\sum_{i\in I}g^{(3)}_{i}$
for  _$k=1$ to $m$ _ do

$G^{(i)}_{left}\leftarrow 0$ for $i=1,2,3$
for  _$j$ in sorted($I$, by $\mathbf{x}_{jk}$)_ do

$G^{(i)}_{left}\leftarrow G_{left}+g^{(i)}_{j}$ for $i=1,2,3$
$G^{(i)}_{right}\leftarrow G^{(i)}-G_{left}^{(i)}$ for $i=1,2,3$
Compute  $\mathcal{L}_{node}$, $\mathcal{L}_{left}$, $\mathcal{L}_{right}$ Eq.
11
$score\leftarrow\max(score,\mathcal{L}_{node}–(\mathcal{L}_{left}+\mathcal{L}_{right}))$

end for

end for

Output: Split with max score

##  Appendix B Full traning curves

[Epsilon] [Higgs] [MiniBooNE] [covertype]

Figure 2: Test accuracy for high-order GBDT models (full training curves).