Next we show the non-vanishing. By our choice of $\chi_h,\chi_\theta$, we have $\Hom_{D^\times(\bQ_v)}\left(\Pi_v,\bC\right)\neq 0$. By <cit.>, we can pick $\sT'_v\in \cO_L\left[D^\times(\bQ_v)\right]$ and $\varphi'_v\in \pi^D_v$ such that the linear functional

\begin{equation}
\sL_{F'_{\tau,v}}:\Pi_v\lra \bC, \quad\quad \sL_{F'_{\tau,v}}(F_v)=\int_{\bQ^\times_v\backslash D^\times(\bQ_v)} \big<\Pi_v(g)F_v,F'_{\tau,v}\big>_v\,dg
\end{equation}

is nonzero and spans the one-dimensional space $\Hom_{D^\times(\bQ_v)}\left(\Pi_v,\bC\right)$. What we need to show is that there exists $\sT_v\breve{h}_{\tau,v}\in\pi^D_{h_\tau,v}$ and $\varphi_v\in\pi^D_v$ invariant under the level group $K_v$ defined in (<ref>) such that $\sL_{F'_{\tau,v}}\left(\breve{\theta}^{\bar{\lambda}}_{\tau,v}\otimes\varphi_v\otimes\sT_v\breve{h}_{\tau,v}\right)\neq 0$. Since $\breve{\theta}^{\bar{\lambda}}_{\tau,v}$ is invariant under $K_v$, it suffices to show that there exists $\sT_v\breve{h}_{\tau,v}\in \pi^D_{h_\tau,v}$ invariant under $K_v$ such that the linear functional

\begin{equation}\label{eq:Lpi}
\sL_{F'_{\tau,v}}\left(\breve{\theta}^{\bar{\lambda}}_{\tau,v}\otimes\bdot\otimes\sT_v\breve{h}_{\tau,v}\right): \pi_v\lra \bC
\end{equation}

is nonzero. There are two cases.

Case 1: $v\neq q$, $D^\times(\bQ_v)\cong\GL_2(\bQ_v)$. Let $\varphi_v$ denote the prime number corresponding to $v$. Recall that at the beginning of <ref>, we have fixed the positive integer $\fs$ and totally imaginary element $\delta$ in $\cK$, and $\U(2)$ is the unitary group for the skew-Hermitian form $\zeta_0=\delta\begin{pmatrix}\fs\\&1\end{pmatrix}$. By our assumption on $\fs$, there exists $\fr\in \cO^\times_{\cK,v}$ such that $\Nm_{\cK_v/\bQ_v}(\fr)=-\fs$. We fix the following isomorphism

\begin{align*}
\ffi:\GU(2)(\bQ_v)&\stackrel{\cong}{\lra}\GU(1,1)(\bQ_v),
&g&\longmapsto \begin{pmatrix}\frac{\delta}{2}&\frac{\delta\fr}{2}\\1&-\fr\end{pmatrix}g\begin{pmatrix}\frac{\delta}{2}&\frac{\delta\fr}{2}\\1&-\fr\end{pmatrix}^{-1},
\end{align*}

where $\GU(1,1)$ denotes the unitary group attached to the skew-Hermitian form $\begin{pmatrix}&1\\-1\end{pmatrix}.$ Let $K^{\GL_2}_v=\ffi(K_v)\cap \GL_2(\bZ_v)$. By our assumptions in <ref>, $v\neq 2$. One can check that

\begin{equation*}
\end{equation*}

where $\epsilon_v=\begin{cases}0,&v\text{ inert}\\ 1,&v\text{ ramified}\end{cases}$.

Given a character $\chi_v:\bQ^\times_v\ra\bC^\times$, we denote also by $\chi_v$ the character $B(\bQ_v)\ra\bC^\times$, $\begin{pmatrix}a&b\\0&d\end{pmatrix}\mapsto \chi_v(ad^{-1})$, and $\Ind_{B}^{\GL_2}(\chi_v)$ the (normalized) induction of $\chi_v$. By our assumption on $\pi$, $\pi_v$ is either isomorphic to $\Ind_{B}^{\GL(2)}(\chi_v)$ for an unramified character $\chi_v\neq |\cdot|^{\pm 1/2}_{\bQ_v}$ or is isomorphic to the irreducible sub-representation of $\Ind_{B}^{\GL(2)}(\chi_v)$ with $\chi_v=|\cdot|^{-1/2}_{\bQ_v}$. Let $\psi_1=\be_v$, the additive character of $\bQ_v$ defined in (<ref>), $\psi_2=\bar{\be}_v$, and $W_{1}$ (resp. $W_{2}$) be the Kirillov model of $\pi^D_{\theta^{\bar{\lambda}}_\tau,v}$ (resp. $\pi^D_{h_\tau,v}$) with respect to $\psi_1$ (resp. $\psi_2$). We have

\begin{equation}\label{eq:indF}
\begin{aligned}
\Hom_{\GL_2(\bQ_v)}\left(W_1\otimes W_2,\tilde{\pi}\right)&\cong \Hom_{\GL_2(\bQ_v)}\left(W_1\otimes W_2,\Ind^{\GL_2}_B (\chi^{-1}_v)\right)\\
&\cong \Hom_{B(\bQ_v)}\left(W_1\otimes W_2,\chi^{-1}_v|\cdot|^{1/2}_{\bQ_v}\right),
\end{aligned}
\end{equation}

where the second isomorphism is deduced from the Frobenius reciprocity, and when $\chi_v=|\cdot|^{-1/2}_{\bQ_v}$, the first isomorphism is because by our choice of $\chi_\theta,\chi_h$, $W_1\not\cong \tilde{W}_2$. Our choice of $\chi_\theta,\chi_h$ also implies that $W_1,W_2$ are supercuspidal, so they consist of Schwartz functions on $\bQ^\times_v$ on which the action $B(\bQ_v)\subset\GL_2(\bQ_v)$ has the formula

\begin{align}
\label{eq:Kirillov} \begin{pmatrix}a&b\\0&d\end{pmatrix}\cdot  w_j(x)&=\omega_{\pi_j}(d)\,\psi_j(bd^{-1}x)\,w_j(ad^{-1}x), &w_j\in W_j,
\end{align}

where $\omega_{\pi_j}$ is the central character of $\pi_j$. By the way we extend $\bfh$ and $\bmtheta$ in <ref>, we know that $\omega_{\pi_1}\omega_{\pi_2}=\triv$. It is easy to see that the map

\begin{equation}
\sL(w_1,w_2)= \int_{\bQ^\times_v} w_1(x)w_2(x)\cdot\chi_v(x)|x|^{-1/2}_{\bQ_v}\,d^\times x, \quad\quad w_1\in W_1,\,w_2\in W_2,
\end{equation}

is a nonzero element in (<ref>), which is one dimensional, so it equals the image of $\sL_{F'_{\tau,v}}\left(\breve{\theta}^{\bar{\lambda}}_{\tau,v}\otimes\bdot\otimes\bdot\right)$ in (<ref>) up to a nonzero scalar. Hence, showing the existence of $\breve{h}_{\tau,v}\in \pi^D_{h_\tau,v}$ invariant under $K_v$ such that (<ref>) is nonzero reduces to showing the existence of $\pzy_2\in W_2$ invariant under $K^{\GL_2}_v$ such that $\sL(\pzy_1,\pzy_2)\neq 0$, where $\pzy_1\in W_1$ is the vector invariant under$K^{\GL_2}_v$  corresponding to $\breve{\theta}^{\bar{\lambda}}_{\tau,v}\in \pi^D_{\theta^{\bar{\lambda}}_\tau,v}$.

For a character $\vartheta:\bZ^\times_v\ra\bC^\times$, put

\[
\pzy_{1,\vartheta}=\int_{\bZ^\times_v}\vartheta^{-1}(a)\pzy_1(ax)\,d x,
\]

the projection of $\pzy_1$ to the $\vartheta$-eigenspace for the action of $\diag(\bZ^\times_v,1)$. Let $\vartheta^*$ be the extension of $\vartheta$ to $\bQ^\times_v$ taking value $1$ at $\varpi_v$. For each integer $m$, put define $w^m_{j,\vartheta}\in W_j$, $j=1,2$, as

\[
\]

Then we have $\pzy_{1,\vartheta}=\sum_m c_m w^m_{1,\vartheta}$ with $c_m\in\bC$ finitely many nonzero, and $\sL(\pzy_1,w^m_{2,\vartheta^{-1}})=\sL(\pzy_{1,\vartheta},w^m_{2,\vartheta^{-1}})$. We can take a $\vartheta$ of conductor $\leq \varpi^{c_v}_v$ such that $\pzy_{1,\vartheta}\neq 0$. This $\pzy_{1,\vartheta}$ is still invariant under $K^{\GL_2}_v$. By the formula (<ref>) and (1) of Lemma <ref> proved below, we deduce that

\[
\pzy_{1,\vartheta}=\sum_{m=-(c_v+\epsilon_v)}^{c_v-n(w^0_{1,\vartheta})}c_m w^m_{1,\vartheta}, \quad\quad c_m\in\bC,
\]

with $n(w^0_{1,\vartheta}$ defined as in (<ref>). Take $-(c_v+\epsilon_v)\leq m_0\leq c_v-n(w^0_{1,\vartheta})$ such that $c_{m_0}\neq 0$. Then

\[
\sL(\pzy_1,w^{m_0}_{2,\vartheta^{-1}})=c_{m_0}\sL(w^{m_0}_{1,\vartheta},w^{m_0}_{2,\vartheta^{-1}})\neq 0,
\]

and by the formula (<ref>), we know that $w^{m_0}_{2,\vartheta^{-1}}$ is invariant under $\begin{pmatrix}1+\varpi^{c_v}_v\bZ_v&\varpi^{c_v+\epsilon_v}_v\bZ_v\\0&1+\varpi^{c_v}_v\end{pmatrix}$, and by (2) of Lemma <ref>, we know that

\[
n(w^{m_0}_{2,\vartheta^{-1}})=m_0+c(\pi_2\otimes\vartheta^{*-1})\leq c_v-n(w^0_{1,\vartheta})+c(\pi_2\otimes\vartheta^{*-1})=c_v-c(\pi_1\otimes\vartheta^*)+c(\pi_2\otimes\vartheta^{*-1}).
\]

By our choice of $\chi_\theta,\chi_h$ and the formula (<ref>), we have $c(\pi_1\otimes\vartheta^*)\geq c(\pi_2\otimes\vartheta^{*-1})$. Hence, $ n(w^{m_0}_{2,\vartheta^{-1}})\geq c_v$, and $w^{m_0}_{2,\vartheta^{-1}}$ is invariant under $\begin{pmatrix}1&0\\ \varpi^{c_v}_v\bZ_v&1\end{pmatrix}$. Therefore, $\pzy_2=w^{m_0}_{2,\vartheta^{-1}}$ is the desired vector in $W_2$ invariant under $K^{\GL_2}_v$ such that $\sL(\pzy_1,\pzy_2)\neq 0$.

Case 2: $v=q$, $D^\times(\bQ_q)$ modulo its center is compact and $\pi^D_v$ is one-dimensional isomorphic to $\chi_q\circ \Nm_D$, where $\chi_q$ is the unramified quadratic character in condition (5) in the choice of $\chi_\theta,\chi_h$ at the beginning of <ref>, and $\Nm_D$ denotes the norm of $D$. By our choice of $\chi_\theta,\chi_h$, we have

\[
\pi^D_{\theta^{\bar{\lambda}}_\tau,q}\otimes \chi_q\circ\Nm_D\cong \tilde{\pi}^D_{h_\tau,q}.
\]

It follows that there exists $\sT_q\in\cO_L\left[D^\times(\bQ_v)\right]$ such that (<ref>) is nonzero. Since $K_v$ is a normal subgroup of $D^\times(\bQ_q)$, $\sT_q\breve{h}_{\tau,q}$ is still invariant under $K_v$.

Let $v$ be a place of $\bQ$ non-split in $\cK$ and $\sigma:\cK^\times_v\ra \bQ^\times_v$ be a character nontrivial on the $\ker \Nm_{\cK_v/\bQ_v}$. Let $\pi_\sigma$ be the dihedral supercuspidal representation of $\GL_2(\bQ_v)$ obtained by the theta lift of $\sigma$. (See <cit.> for the precise definition.) Then the conductor of $\pi_\sigma$ equals

\begin{equation}\label{eq:cond-dih}
2\cond(\sigma), &v\text{ inert,}\\
\cond(\sigma)+1, &v\text{ ramified}.
\end{array}\right.
\end{equation}

by <cit.>. Let $W$ be the Kirillov model of $\pi_\sigma$ with respect to the additive character $\be_v^{\pm 1}$ of $\bQ_v$. For $w\in W$, define

\begin{equation}\label{eq:nw}
n(w)=\min\left\{n\in\bZ: \text{$w$ is fixed by $\begin{pmatrix}1&0\\ \varpi^{n}_v\bZ_v&1\end{pmatrix}$}\right\}
\end{equation}

Let $\vartheta$ be a character of $\bZ^\times_v$ and $\vartheta^*$ be its extension to $\bQ^\times_v$ taking value $1$ at $\varpi_v$. For each integer $m$, define $w^m_\vartheta\in W$ as

\[
\]

Suppose that $\ord_v(\cond(\sigma))>\ord_v(\cond(\sigma|_{\bQ^\times_v}))$.

* Let $w=\sum\limits_{m=l_1}^{l_2} c_m w^m_\vartheta$ with $c_{l_2}\neq 0$. Then $n(w)=n(w^{l_2}_\vartheta)=n(w^0_\vartheta)+l_2$.

* We have

\begin{align*}
m+2\cond(\sigma\cdot\vartheta^{*-1}\circ \Nm_{\cK_v/\bQ_v}), &\text{$v$ inert},\\
m+\cond(\sigma\cdot\vartheta^{*-1}\circ\Nm_{\cK_v/\bQ_v})+1,  &\text{$v$ ramified}.
\end{array}\right.
\end{align*}

(1) It is easy to see that $\begin{pmatrix}\varpi^l_v\\&1\end{pmatrix}\cdot w^m_\vartheta =w^{m-l}_\vartheta$. Thus, $n(w^m_\vartheta)=n(w^0_\vartheta)+m$. It follows that $n(w^m_\vartheta)\leq n(w^{l_2}_\vartheta)$ for all $m\leq l_2$, so $n(w)\leq n(w^{l_2}_\vartheta)$. On the other hand, if $n(w)\leq n(w^{l_2}_\vartheta)-1$, then $n(w^{l_2}_\vartheta)=n\Big(c^{-1}_{l_2}(w-\sum\limits_{l_1}^{l_2-1}c_m w^m_\vartheta)\Big)\leq n(w^{l_2}_\vartheta)-1$. This is a contradiction. Hence,  $n(w)= n(w^{l_2}_\vartheta)$.

(2) Let $W'$ the representation of $\GL_2(\bQ_v)$ whose underlying $\bC$-vector space equals that of $W$ with the action of $\GL_2(\bQ_v)$ defined as the action of $\GL_2(\bQ_v)$ on $W$ twisted by $\vartheta\circ\det$, i.e. for $g\in \GL_2(\bQ_v)$ and $w\in W'$, the action of $g$ on $w$ gives $\vartheta^{-1}(\det g)(g\cdot w)$, where $g\cdot w$ denotes the action of $g$ on $w$ viewed as an element in $W$. Then $W'\cong \pi_\sigma\otimes\vartheta^{*-1}$. Suppose that $w'\in W'$ is the new vector, then $\begin{pmatrix}\bZ^\times_v&\bZ_v\\0&d\end{pmatrix}$, $d\in\bZ^\times_v$, acts on it by the scalar $\sigma\vartheta^{-2}(d)$, and we have $w'=\sum\limits_{m=l_1}^{l_2}c_m w^m_\vartheta$ with $c_{l_1}\neq 0$ for some $l_1\geq 0$.  Then

\begin{equation}\label{eq:Kcond}
n(w^0_\vartheta)\geq c(\pi_\sigma\otimes\vartheta^{*-1})=n(w')=n(w^0_\vartheta)+l_2,
\end{equation}

where the first inequality and the middle equality follows from the definition of new vectors and conductors of $\GL_2(\bQ_v)$-representations, and the right equality follows from (1). (<ref>) implies that $l_1=l_2=0$, $w^0_\vartheta$ is the new vector in $W'$ and $ n(w^0_\vartheta)=c(\pi_\sigma\otimes\vartheta^{*-1})$. Since $n(w^m_\vartheta)=n(w^0_\vartheta)+m$, we see that $n(w^m_\vartheta)=m+c(\pi_\sigma\otimes\vartheta^{*-1})$.

§ PROOF OF GREENBERG–IWASAWA MAIN CONJECTURE

§.§ The Klingen Eisenstein ideal

Let $\cM_{\so}=\Hom_{\Lambda_\so}\left(\pV^*_\so,\Lambda_\so\right)$ be the $\cO_L\llb T_\so(\bZ_p)\rrb$-module of semi-ordinary families on $\GU(3,1)$ of tame level $K^p_f$ (defined in (<ref>)) as in Theorem <ref>. Identifying $T_\so(\bZ_p)$ with $\bZ^\times_p\times\bZ^\times_p\simeq \cO^\times_{\cK,p}$, we have the map

\begin{equation}\label{eq:TGamma}
T_\so(\bZ_p)\lra \cO^\times_{\cK,p}\lra \Gamma_\cK,
\end{equation}

with kernel $\mu_p\times\mu_p$. Denote by $\glsuseri{pVxi}$ (resp. $\glsuseri{Mxi}$) the sub-$\cO_L\llb T_\so(\bZ_p)\rrb$-module of $e_\so \pV$ (resp. $\cM_\so$) on which $\mu_p\times\mu_p$ acts through $\xi_{p\adic}$. We make $T_\so(\bZ_p)$ act on  $\pV_{\so,\,\xi}$ and $\cM_{\so,\,\xi}$ through its usual action multiplied by the character $\xi^{-1}_{p\adic}$. Then this action factors through the image of $T_\so(\bZ_p)$ in $\Gamma_\cK$ under the map (<ref>). Let $\glsuserii{pVxi}\subset \pV_{\so,\,\xi}$, $\glsuserii{Mxi}\subset\cM_{\so,\,\xi}$ be the cuspidal part. We consider the $\cO_L\llb\Gamma_\cK\rrb$-modules

\begin{align*}
\glsuseri{MxiGamma}&=\cM_{\so,\,\xi}\otimes_{\cO_L\llb T_\so(\bZ_p)\rrb} \hat{\cO}^\ur_L\llb\Gamma_\cK\rrb,
&\glsuserii{MxiGamma}&=\cM^0_{\so,\,\xi}\otimes_{\cO_L\llb T_\so(\bZ_p)\rrb} \hat{\cO}^\ur_L\llb\Gamma_\cK\rrb.
\end{align*}

The Klingen Eisenstein family $\bfE^\Kling_\varphi$ constructed in <ref> belongs to $\Meas\left(\Gamma_\cK,e_\so V_{\GU(3,1),\,\xi}\wh{\otimes}\hat{\cO}^\ur_L\right)$. By the natural pairing between $V_{\GU(3,1),\so}$ and the space $\pV^*_\so$ in Theorem <ref>, we see that there is a natural map

\begin{align*}
\Meas\left(\Gamma_\cK,e_\so V_{\GU(3,1),\,\xi}\wh{\otimes}\hat{\cO}^\ur_L\right)\times  (\pV_{\so,\,\xi})^*\lra \Meas\left(\Gamma_\cK,\hat{\cO}^\ur_L\right)\simeq \hat{\cO}^\ur_L\llb\Gamma_\cK\rrb,
\end{align*}

and it induces a $\hat{\cO}^\ur_L\llb\Gamma_\cK\rrb$-linear Hecke-equivariant map

\begin{align*}
\Meas\left(\Gamma_\cK,e_\so V_{\GU(3,1),\,\xi}\wh{\otimes}\hat{\cO}^\ur_L\right)\lra \Hom_{\cO_L\llb T_\so(\bZ_p)\rrb}\left((\pV_{\so,\,\xi}^*,\hat{\cO}^\ur_L\llb\Gamma_\cK\rrb\right)=\cM_{\so,\,\xi,\Gamma_\cK}.
\end{align*}

Therefore, we can view $\bfE^\Kling_\varphi$ as an element in $\cM_{\so,\,\xi,\Gamma_\cK}$. Let $\bT$ be the abstract algebra generated by the unramified Hecke algebras away from $\Sigma\cup\{p,\ell,\ell'\}$ and $\bU_p$-operators at $p$. Then $\bfE^\Kling_\varphi$ is an eigen-family for the action of $\bT$, and we denote by $\lambda_{\mr{Eis},\pi,\xi}:\bT\ra \hat{\cO}^\ur_L\llb\Gamma_\cK\rrb$ the corresponding eigen-system.

Let $\gls{bT}$ be the reduced Hecke algebra generated by  $\bT$ acting on $\cM^0_{\so,\,\xi,\Gamma_\cK}$. Let $\gls{IEis}\subset \bT^0_{\so,\,\xi,\Gamma_\cK}$ be the ideal generated by the images of $T-\lambda_{\mr{Eis},\pi,\xi}(T)$ in $\bT^0_{\so,\,\xi,\Gamma_\cK}$, $T\in\bT$. We define the Klingen Eisenstein ideal as

\[
\begin{tikzcd}[column sep=large]
\gls{cEpi}=\ker\Big(\hat{\cO}^\ur_L\llb\Gamma_\cK\rrb\arrow[r,"\text{structure}","\text{map}"']& \bT^0_{\so,\,\xi,\Gamma_\cK}/\cI_{\mr{Eis},\pi,\xi}\Big).
\end{tikzcd}
\]

It measures the congruences between the Hecke eigen-systems in $\bT^0_{\so,\,\xi,\Gamma_\cK}$ and the Hecke eigen-system $\lambda_{\mr{Eis},\pi,\xi}$ attached to $\bfE^\Kling_\varphi$.

Let $\varphi$ be as in Proposition <ref>. Let $P$ be a height one prime ideal of $\hat{\cO}^\ur_L\llb\Gamma_\cK\rrb$ such that $P\cap \hat{\cO}^\ur_L=(0)$. Then

\[
\ord_P\left(\cL^{\Sigma\,\cup\{\ell,\ell'\}}_{\pi,\cK,\xi}\cL^{\Sigma\,\cup\{\ell,\ell'\}}_{\xi,\bQ}\right)\leq \ord_P(\cE_{\pi,\xi}),
\]

(where $\cL^{\Sigma\,\cup\{\ell,\ell'\}}_{\xi,\bQ},\cL^{\Sigma\,\cup\{\ell,\ell'\}}_{\pi,\cK,\xi}\in\hat{\cO}^\ur_L\llb\Gamma_\cK\rrb$ are the $p$-adic $L$-function satisfying the interpolation property in (<ref>)(<ref>)).

By Theorem <ref>, we have the fundamental exact sequence

\[
0\lra\cM^0_{\so,\,\xi,\Gamma_\cK}\lra \cM_{\so,\,\xi,\Gamma_\cK}\lra \bigoplus_{g\in
C(K^p_f)} M_{\GU(2)}\left(K^p_{f,g}K_{p,0},\hat{\cO}^\ur_L\right)\otimes_{\bZ_p}\bZ_p\llb \Gamma_\cK\rrb \lra 0.
\]

Combining it with Theorem <ref> on the degenerate Fourier–Jacobi coefficients of $\bfE^\Kling_\varphi$, we deduce that there exists $\bfE'\in \cM_{\so,\,\xi,\Gamma_\cK}$ such that

\[
\bm{F}=\bfE^\Kling_\varphi-\cL^{\Sigma\,\cup\{\ell,\ell'\}}_{\xi,\bQ}\cL^{\Sigma\,\cup\{\ell,\ell'\}}_{\pi,\cK,\xi}\cdot \bfE'\in \cM^0_{\so,\,\xi,\Gamma_\cK}.
\]

Let $P$  be a height one prime ideal of $\hat{\cO}^\ur_L\llb\Gamma_\cK\rrb$ such that

\[
P\cap \hat{\cO}^\ur_L=(0), \quad \ord_P\left(\cL^{\Sigma\,\cup\{\ell,\ell'\}}_{\xi,\bQ}\cL^{\Sigma\,\cup\{\ell,\ell'\}}_{\pi,\cK,\xi}\right)=m\geq 1.
\]
By Proposition <ref>, for $\beta=1$, there exists $g\in\U(2)(\bA_\cK)$ and $u\in \bigotimes_{v\in\Sigma_\ns\cup\{\ell'\}}\U(1)(\bQ_v)$ such that $l_{\theta^J_1}\left(\bfE^\Kling_{\varphi,\beta,u}\right)(g)\notin P$. Then $l_{\theta^J_1}\left(\bm{F}_{\beta,u}\right)(g)\notin P$, and we have the $\hat{\cO}^\ur_\cK\llb\Gamma_\cK\rrb_P$-linear map

\begin{align}
\label{eq:Ipi1}\left(\bT^0_{\so,\,\xi,\Gamma_\cK}\right)_P&\lra \hat{\cO}^\ur_\cK\llb\Gamma_\cK\rrb_P/P^m\hat{\cO}^\ur_\cK\llb\Gamma_\cK\rrb_P,
\end{align}

which sends the image of $T\in\bT$ in $\left(\bT^0_{\so,\,\xi,\Gamma_\cK}\right)_P$ to

\begin{equation*}%\label{eq:Ipi2}
\frac{l_{\theta^J_1}\left((T\cdot \bm{F})_{\beta,u}\right)(g)}{l_{\theta^J_1}\left(\bm{F}_{\beta,u}\right)(g)}\equiv \frac{l_{\theta^J_1}\left((T\cdot \bm{E}^\Kling_\varphi)_{\beta,u}\right)(g)}{l_{\theta^J_1}\left(\bm{E}^\Kling_{\varphi,\beta,u}\right)(g)}\equiv \lambda_{\mr{Eis},\pi,\xi}(T) \mod P^m.
\end{equation*}

We see that (<ref>) factors through $\left(\bT^0_{\so,\,\xi,\Gamma_\cK}\right)_P/\cI_{\mr{Eis},\pi,\xi} \left(\bT^0_{\so,\,\xi,\Gamma_\cK}\right)_P$. It follows that the composition

\begin{equation}\label{eq:Ipi3}
\begin{tikzcd}[column sep=large]
\hat{\cO}^\ur_\cK\llb\Gamma_\cK\rrb_P\arrow[r,"\text{structure}", "\text{map}"']& \left(\bT^0_{\so,\,\xi,\Gamma_\cK}\right)_P\arrow[r,"\eqref{eq:Ipi1}"]& \hat{\cO}^\ur_\cK\llb\Gamma_\cK\rrb_P/P^m\hat{\cO}^\ur_\cK\llb\Gamma_\cK\rrb_P.
\end{tikzcd}
\end{equation}

factors through $\hat{\cO}^\ur_\cK\llb\Gamma_\cK\rrb_P/\cE_{\pi,\xi}\hat{\cO}^\ur_\cK\llb\Gamma_\cK\rrb_P$. By the definition of (<ref>), we also know that the composition (<ref>) is the natural projection. Hence, we deduce that $\cE_{\pi,\xi} \hat{\cO}^\ur_\cK\llb\Gamma_\cK\rrb_P\subset P^m \hat{\cO}^\ur_\cK\llb\Gamma_\cK\rrb_P$, and

\[
\ord_P\left(\cE_{\pi,\xi}\right)\leq m= \ord_P\left(\cL^{\Sigma\,\cup\{\ell,\ell'\}}_{\xi,\bQ}\cL^{\Sigma\,\cup\{\ell,\ell'\}}_{\pi,\cK,\xi}\right).
\]

§.§ The main theorem

We prove the following partial results towards Conjecture <ref>.

Let $\cK$ be an imaginary quadratic extension of $\bQ$ and $p\geq 3$ be a prime split in $\cK$. Let $\pi$ and $\xi$ be as in <ref>.

* Let $\Gamma^+_\cK\subset\Gamma_\cK$ be the rank one $\bZ_p$-module on which the complex conjugation acts by $+1$. As fractional ideals of $\pzR=\cO^\ur_L\llb\Gamma_\cK\rrb \otimes_{\cO_L\llb \Gamma^+_{\cK}\rrb}\mr{Frac}\left(\cO_L\llb \Gamma^+_\cK\rrb\right)$,

\begin{equation}\label{eq:ideal-incl}
\big(\cL_{\pi,\cK,\xi}\big)\supset \mr{char}_\pzR \big(X_{\pi,\cK,\xi}\otimes _{\cO_L\llb \Gamma^+_{\cK}\rrb}\pzR\big).
\end{equation}

See <ref> for the definition of the $p$-adic $L$-function $\cL_{\pi,\cK,\xi}\in \mr{Frac}\left(\cO^\ur_L\llb \Gamma_\cK\rrb\right)$, the $\cO_L\llb\Gamma_\cK\rrb$-module $X_{\pi,\cK,\xi}$, and the characteristic ideals.

* If $\xi$ satisfies $ \xi_{\fp}\not\equiv\xi_{\bar{\fp}}$ modulo the maximal ideal of $\cO_L$, then

\[
\cL_{\pi,\cK,\xi}\in \cO^\ur_L\llb \Gamma_\cK\rrb
\]

and if it further satisfies:

– $\xi|_{\bA^\times_\bQ}=\omega^2$ with $\omega:\bQ^\times\backslash\bA^\times_\bQ\ra\bC^\times$ the Teichmüller character, and $k_0\equiv 0 \mod 2(p-1)$,

– for every finite place of $\bQ$ non-split in $\cK$, $\epsilon_v\left(\frac{1}{2},\mr{BC}(\pi)\times\xi_0\right)=1$,

– the conductor of $\xi$ is only divisible by primes split in $\cK/\bQ$,

\[
\big(\cL_{\pi,\cK,\xi}\big)\supset \mr{char}_{\cO^\ur_L\llb\Gamma_\cK\rrb \otimes_{\bZ} \bQ}\big(X_{\pi,\cK,\xi}\otimes_{\bZ}\bQ\big)
\]

(1) First, relaxing the conditions at $v\in\Sigma\cup\{\ell,\ell'\}$ in the definition of $\Sel_{\pi,\cK,\xi}$ in (<ref>) and $X_{\pi,\cK,\xi}$ in (<ref>), we define the $\Sigma\cup\{\ell,\ell'\}$-primitive Selmer  group:

\begin{align*}
\gls{Selpartial}=\ker\left\{H^1\big(\cK,T_{\pi,\cK,\xi}\otimes_{\cO_L}\cO_L\llb\Gamma_\cK\rrb\big)
\lra \prod_{\substack{\fv \neq\fp\\ \fv\,\nmid \text{ places in $\Sigma\cup\{\ell,\ell'\}$}}}H^1\big(I_\fv,T_{\pi,\cK,\xi}\otimes_{\cO_L}\cO_L\llb\Gamma_\cK\rrb^*\big)\right\}.
\end{align*}

\[
\gls{Xpartial}:=\Hom_{\bZ_p}\left(\Sel^{\Sigma\,\cup\{\ell,\ell'\}}_{\pi,\cK,\xi},\,\bQ_p/\bZ_p\right).
\]

We can assume that the module $X_{\pi,\cK,\xi}$ is $\cO_L\llb \Gamma_{\cK}\rrb$-torsion (since otherwise its characteristic ideal is defined to be $(0)$ and the inclusion (<ref>) automatically holds). By the same argument as in the proof of <cit.>, which uses the fact that the sizes of the unramified cohomology groups at primes
outside $p$ are controlled by the local Euler factors of the $p$-adic $L$-functions (<cit.>), it suffices to prove the inclusion

\begin{equation*}
\left(\cL^{\Sigma\,\cup\{\ell,\ell'\}}_{\pi,\cK,\xi}\right)\supset \mr{char}_\pzR \left(X^{\Sigma\,\cup\{\ell,\ell'\}}_{\pi,\cK,\xi}\otimes _{\cO_L\llb \Gamma^+_{\cK}\rrb}\pzR\right)
\end{equation*}

as fractional ideals of $\pzR=\cO^\ur_L\llb\Gamma_\cK\rrb \otimes_{\cO_L\llb \Gamma^+_{\cK}\rrb}\mr{Frac}\left(\cO_L\llb \Gamma^+_\cK\rrb\right)$.

We reduce to show that given a height one prime ideal $P\subset \cO^\ur_L\llb \Gamma_\cK\rrb$ such that $P\cap \cO^\ur_L\llb\Gamma^+_\cK\rrb=(0)$, we have the inequality

\begin{equation}\label{eq:incl}
\ord_P\left(\cL^{\Sigma\cup\{\ell,\ell'\}}_{\pi,\cK,\xi}\right)\leq \mr{length}_{\cO^\ur_L\llb\Gamma_\cK\rrb_P}\left(\left(X^{\Sigma\cup\{\ell,\ell'\}}_{\pi,\cK,\xi}\right)_P\right),
\end{equation}

which, by Theorem <ref>, is implied by

\begin{equation}\label{eq:cE-incl}
\ord_P\left(\cE_{\pi,\xi}\right)\leq \mr{length}_{\cO^\ur_L\llb\Gamma_\cK\rrb_P}\left(\left(X^{\Sigma\cup\{\ell,\ell'\}}_{\pi,\cK,\xi}\right)_P\right).
\end{equation}

We have the following setup.

* $A_0=\cO^{\ur}_L\llb\Gamma_\cK\rrb$ and $A=\hat{A}_{0,P}$.

* $R_0=\bT^0_{\so,\xi,\Gamma_\cK}$, a prime ideal $Q\subset R$ such that $Q\cap A_0=P$, and $R=\hat{R}_{0,Q}$.

* $G=\Gal(\ol{\bQ}/\bQ)$ and $H=\Gal(\ol{\bQ}/\cK)$.

The following data gives us the setup (1)-(5) on <cit.>.

* Let $\nu:H\ra A^\times_0$ be the trivial character.

* $\chi=\xi\Psi_\cK:G\ra A^\times_0$, with $\chi\not\equiv\chi^{-c}$ modulo the maximal ideal of $A_0$ by the choice of $\xi$,

* $\rho:G\ra \mr{Aut}_A(V)$, a Galois representation obtained from the two-dimensional Galois representation $\rho_\pi:H\ra\GL(V_\pi)$ with $V=T_\pi\otimes_{\cO_L} A$ and $T_\pi\subset V_\pi$ a $G$-stable lattice.

* $\sigma:G\ra \mr{Aut}_{R\otimes_A F}(M)$, a Galois representation on $M=(R\otimes_A \mr{Frac}(A))^4$ obtained as the pseudo-representation associated to $R_0$ as in <cit.>

* $I=\cE_{\pi,\xi}A_P$ and $J=\ker(R\ra A/I)$ where the map $R\ra A$ is defined as the map (<ref>) and is surjective, and $A/I=R/J$.

Moreover, the data satisfies the properties (6)-(9) on <cit.>. Properties (6)-(8) are checked in the same way as loc.cit.

Property (9) is about the irreducibility of $\sigma$. It requires that for each $\mr{Frac}(A)$-algebra homomorphism $\lambda:R\otimes_A\mr{Frac(A)}\ra K$, $K$ a finite extension of $\mr{Frac}(A)$, the representation $\sigma_\lambda:G\ra \GL_4(K)$ obtained from $\sigma$ via $\lambda$ is either absolutely irreducible or contains an absolutely irreducible two-dimensional sub-representation whose trace is congruent to $\chi+\chi^{-c}$ mod $I$. It can be checked as follows. Let $\pzT:G\ra A$ be the pseudo representation giving rise to $\sigma$. Suppose that $\sigma$ does not satisfy (9). Then $\pzT=\delta_1+\delta_2+\delta_3$ with $\delta_1,\delta_2$ one-dimensional pseudo-character and $\delta_3$ a two-dimensional pseudo-representation with irreducible residual representation is $\Tr\bar{\rho}_\pi$.  Take an arithmetic point $x:\mathbb{T}^0_{\so,\xi,\Gamma_\cK}\ra\ol{\bQ}_p$ corresponding to an automorphic representation $\Pi$ of $\U(3,1)(\bA_\bQ)$ generated by a classical semi-ordinary cuspidal automorphic form fixed by $K^0_{p,1}$ of weight $(0,0,t^+;t^-)$, $0\gg t^+\gg -t^-$. We can associate a $2$-dimensional (irreducible) Galois representation $\rho_{3,x}:G\ra\GL_2(\ol{\bQ}_p)$ to the specialization of $\delta_3$ at $x$. As in <cit.>, a twist of $\rho_{3,x}$ descends to $\Gal(\ol{\bQ}/\bQ)$, which we denote by $\rho'_{3,x}:\Gal(\ol{\bQ}/\bQ)\ra\GL_2(\ol{\bQ}_p)$.

If $\pi$ is ordinary at $p$, then $\Pi$ is ordinary at $p$. By the argument in <cit.>, we know that $\rho'_{3,x}$ is modular and $\Pi$ is CAP, but the condition on the weights $(0,0,t^+;t^-)$ excludes this possibility (<cit.>).

If $\pi$ is not ordinary at $p$, then $\bar{\rho}_\pi|_{G_{\cK,\fp}}\cong\bar{\rho}_\pi|_{G_{\bQ,p}}$ is irreducible by [4]. Since $\bar{\rho}_\Pi\cong \bar{\rho}_\pi|_G\oplus \text{two characters}$, we deduce that the semi-simplification of $\rho_\Pi|_{G_{\cK,\fp}}$ is not a direct sum of four characters. Identify $\U(3,1)(\bQ_p)$ with $\GL_4(\bQ_p)$ and let $Q\subset\GL_4$ be a parabolic subgroup with its Levi subgroup isomorphic to $\GL_2\times\GL_1\times\GL_1$. As in the proof of Proposition <ref>, it follows from the theory of Jacquet modules that $\Pi_p$ is isomorphic to a subquotient of $\Ind^{\GL_4(\bQ_p)}_{Q(\bQ_p)}\sigma\boxtimes\chi\boxtimes\chi'$ with  $\chi,\chi'$ characters of $\bQ^\times_p$  and $\sigma$ an irreducible admissible representation of $\GL_2(\bQ_p)$ containing a nonzero vector fixed by the Iwahori subgroup. Since the semi-simplification of $\rho_\Pi|_{G_{\cK,p}}$ is not a direct sum of four characters, $\sigma$ is not Steinberg or a twist of Steinberg, and must be unramified. Hence, $\Pi_p$ is isomorphic to a subquotient of $\Ind^{\GL_4(\bQ_p)}_{B(\bQ_p)}\chi_1\boxtimes\chi_2\boxtimes\chi_3\boxtimes\chi_4$ with $\chi_1,\chi_2,\chi_3,\chi_4$ unramified characters of $\bQ^\times_p$. Let $\alpha_j=\chi_j(p)$, $1\leq j\leq 4$. We can assume that $\val_p(\alpha_1)\leq \val_p(\alpha_2)\leq \val_p(\alpha_3)\leq \val_p(\alpha_4)$. The semi-ordinarity of $\Pi$ at $p$ implies that

\begin{equation}\label{eq:alpha34}
\begin{aligned}
\val_p(\alpha_1)+\val_p(\alpha_2)&=0,
\end{aligned}
\end{equation}

and the integrality of the operator $U^+_{p,1}$ (defined in (<ref>)) implies that

\begin{equation}\label{eq:alpha1}
\val_p(\alpha_1),\val_p(\alpha_2)\geq -\frac{1}{2}.
\end{equation}

Thanks to the condition $0\gg t^+\gg-t^-$, (<ref>) and (<ref>) imply that $\alpha_i\alpha^{-1}_j\neq p^{\pm 1}$ unless $\{i,j\}=\{1,2\}$ and $\val_p(\alpha_1)=-\frac{1}{2}$, $\val_p(\alpha_2)=\frac{1}{2}$. However, $\val_p(\alpha_1)=-\frac{1}{2}$, $\val_p(\alpha_2)=\frac{1}{2}$ imply that $\Pi$ is ordinary at $p$ and this contradicts that $\rho_\Pi|_{G_{\cK,\fp}}$ is not a direct sum of four characters. Therefore, we have  $\alpha_i\alpha^{-1}_j\neq p^{\pm 1}$ for all $i,j$, so  $\Ind^{\GL_4(\bQ_p)}_{Q(\bQ_p)}\sigma\boxtimes\chi\boxtimes\chi'$ is irreducible and $\Pi_p$ is unramified. It follows that $\rho_\Pi|_{G_{\cK,\fp}}$ is crystalline, so $\rho'_{3,x}|_{G_{\bQ,p}}$ is crystalline. Also, $\bar{\rho}'_{3,x}\equiv\bar{\rho}_\pi$ is not induced from a Galois character for $\bQ\left(\sqrt{(-1)^{(p-1)/2}p}\right)$, because its restriction to $I_p$ has semi-simplification $\omega^i_2\oplus\omega^{pi}_2$ with $\omega_2$ the fundamental character of level $2$ and $i\equiv 1\mod (p-1)$ (since $\rho_\pi|_{G_{\bQ,p}}$ is crystalline of weight $(0,1)$), and if $\bar{\pi}$ was induced from a Galois character for $\bQ\left(\sqrt{(-1)^{(p-1)/2}p}\right)$, $i$ must be a multiple of $\frac{p+1}{2}$, which is impossible when $p$ is odd. Thus, we can apply [24], we deduce that $\rho'_{3,x}$ is modular. Then $\Pi$ is CAP, i.e. it has the same system of Hecke eigenvalues as a Klingen-type
Eisenstein series). This is impossible thanks to the condition on the weights $(0,0,t^+;t^-)$ by <cit.>. Property (9) is verified.

\begin{align*}
\ker\left\{H^1\left(\bQ,\cO^{\ur}_L\llb\Gamma_\cK\rrb,\xi\Psi_\cK\circ \Nm\right)\ra H^1\left(G_{\cK,\fp},\xi\Psi_\cK\circ \Nm\right)\right\},\,\bQ_p/\bZ_p\right).
\end{align*}

With the input (1)-(5) satisfying the properties (6)-(9), applying the lattice construction <cit.>, one proves that if $P\subset\cO^\ur_L\llb\Gamma_\cK\rrb$ is a height one prime such that $\ord_P\left(\cE_{\pi,\xi}\right)>0$ and $\left(X_{\xi\circ\Nm}\otimes_\bZ\bQ)\right)_P=0$, then the inequality (<ref>) holds. It follows that for all height one prime ideals $P\subset \cO^\ur_L\llb\Gamma_\cK\rrb$ with $P\cap \cO^\ur_L\llb\Gamma^+_\cK\rrb=(0)$, the desired inequality (<ref>) holds.

(The congruence ideal $\cE_{\pi,\xi}$ is bounded by the product  $\cL^{\Sigma\,\cup\{\ell,\ell'\}}_{\xi,\bQ}\cL^{\Sigma\,\cup\{\ell,\ell'\}}_{\pi,\cK,\xi}$, and is expected to give a bound for the product of the characteristic ideals of $X^{\Sigma\,\cup\{\ell,\ell'\}}_{\xi\circ\Nm}$ and $X^{\Sigma\,\cup\{\ell,\ell'\}}_{\pi,\cK,\xi}$. Although, the characteristic ideals of $X^{\Sigma\,\cup\{\ell,\ell'\}}_{\xi\circ\Nm}$ is known to be generated by $\cL^{\Sigma\,\cup\{\ell,\ell'\}}_{\xi,\bQ}$ by the Iwasawa Main Conjecture proved in [32], the lattice construction argument in [35, 40] only proves the desired bound for $\ord_P\left(\mr{char}_{\cO^\ur_L\llb\Gamma_\cK\rrb}\left(X^{\Sigma\,\cup\{\ell,\ell'\}}_{\pi,\cK,\xi}\right)\right)$ when $\left(X_{\xi\circ\Nm}\otimes_\bZ\bQ)\right)_P=0$. In [40], height one primes in $\cO^\ur_L\llb\Gamma^+_\cK\rrb\otimes_\bZ\bQ$ do not need to be excluded thanks to Lemma 9.3 loc.cit and the fact that $\cL_{\pi,\cK,\xi}$ is not contained in any height one prime ideal in $\cO^\ur_L\llb\Gamma_\cK\rrb$ by [18]. However, in our case with $\pi$ fixed instead of moving in a family, we cannot prove an analogue of Lemma 9.3  loc. cit.)

(2) Let $\sigma_\xi$ be the irreducible cuspidal automorphic representation of $\GL_2(\bA_\bQ)$ generated by a CM form associated to $\xi$, and $\rho_{\sigma_\xi}:\Gal(\ol{\bQ}/\bQ)\ra\GL_2(\ol{\bQ}_p)$ be the Galois representation associated to $\sigma_\xi$. Then $\rho_{\sigma_\xi}\cong \Ind^\bQ_\cK \rho_{\cK,\xi}$. The condition $\xi_\fp\not\equiv\xi_{\bar{\fp}}$ modulo the maximal ideal of $\cO_L$ implies that

(irred)      $\rho_{\sigma_\xi}$ is residually absolutely irreducible,

(dist)       the characters of $\rho_{\sigma_\xi}|_{G_{\bQ,p}}$ on the diagonal are distinct.

Let $\bm{g}$ be the ordinary $\cO^\ur_L\llb\Gamma_\cK\rrb$-adic CM family passing through the CM form associated to $\xi$. As in <cit.>, one can construct a $p$-adic $L$-function $\gls{LpiHida}$ by taking the product of Hida's Rankin–Selberg $p$-adic $L$-functions for the family $\bm{g}$ and the newform $f\in \pi$ and  Katz's $p$-adic $L$-function restricted to the line of interpolating special values at $1$. Note that although Hida's construction assumes that $\bm{g}$ and $\pi$ are both ordinary, only the ordinarity of $\bm{g}$ is used. By using <cit.> and <cit.> (which relates the Petersson norm with the adjoint $L$-value at $1$), one can check that $\cL^{\mr{Hida}}_{\pi,\cK,\xi}\in \cO^\ur_L\llb\Gamma_\cK\rrb$ and the specializations of $\cL^{\mr{Hida}}_{\pi,\cK,\xi}$ and our $\cL_{\pi,\cK,\xi}$ agree at a Zariski dense subsets of the weight space. Hence, $\cL^{\mr{Hida}}_{\pi,\cK,\xi}=\cL_{\pi,\cK,\xi}$. Then under the conditions (dist) and (dist), by using the Iwasawa Main conjecture proved in [20, 21, 14], one can check that $\cL^{\mr{Hida}}_{\pi,\cK,\xi}\in \cO^\ur_L\llb\Gamma_\cK\rrb$. Hence,  $\cL_{\pi,\cK,\xi}\in \cO^\ur_L\llb\Gamma_\cK\rrb$.

With further conditions that $\xi|_{\bA^\times_\bQ}=\omega^2$ and $k_0\equiv 0\mod 2(p-1)$ and conditions on the epsilon factors and the conductors of $\xi$, one can use the vanishing of the anticyclotomic $\mu$-invariant proved in [18] to deduce that the $\cL_{\pi,\cK,\xi}$ is not divisible by any height one prime ideal in $\cO^\ur_L\llb\Gamma^+_\cK\rrb$. Thus, the inclusion in (1) implies the inclusion in (2).

§ INDEX

[1]
Fabrizio Andreatta, Adrian Iovita, and Vincent Pilloni.
$p$-adic families of Siegel modular cuspforms.
Ann. of Math. (2), 181(2):623–697, 2015.

[2]
Riccardo Brasca.
Eigenvarieties for cuspforms over PEL type Shimura varieties with
dense ordinary locus.
Canad. J. Math., 68(6):1227–1256, 2016.

[3]
William Casselman.
Introduction to the theory of admissible representaions of $p$-adic
Unpublished notes distributed by P. Sally.

[4]
Bas Edixhoven.
The weight in Serre's conjectures on modular forms.
Invent. Math., 109(3):563–594, 1992.

[5]
Ellen Eischen and Zheng Liu.
The Archimedean facotr for $p$-adic $l$-functions for unitary
Preprint available at <https://arxiv.org/abs/2006.04302>.

[6]
Ellen Eischen and Xin Wan.
$p$-adic Eisenstein series and $L$-functions of certain cusp
forms on definite unitary groups.
J. Inst. Math. Jussieu, 15(3):471–510, 2016.

[7]
Paul B. Garrett.
Integral representations of Eisenstein series and $L$-functions.
In Number theory, trace formulas and discrete groups
(Oslo,1987), pages 241–264. Academic Press, Boston, MA, 1989.

[8]
Wee Teck Gan and Atsushi Ichino.
Formal degrees and local theta correspondence.
Invent. Math., 195(3):509–672, 2014.

[9]
Ralph Greenberg.
Iwasawa theory and $p$-adic deformations of motives.
In Motives (Seattle, WA, 1991), volume 55 of Proc.
Sympos. Pure Math., pages 193–223. Amer. Math. Soc., Providence, RI, 1994.

[10]
Ralph Greenberg and Vinayak Vatsal.
On the Iwasawa invariants of elliptic curves.
Invent. Math., 142(1):17–63, 2000.

[11]
Michael Harris.
Eisenstein series on Shimura varieties.
Ann. of Math., 119(1):59–94, 1984.

[12]
Haruzo Hida.
On $p$-adic $L$-functions of ${\rm GL}(2)\times {\rm GL}(2)$
over totally real fields.
Ann. Inst. Fourier (Grenoble), 41(2):311–391, 1991.

[13]
Haruzo Hida.
Control theorems of coherent sheaves on Shimura varieties of PEL
J. Inst. Math. Jussieu, 1(1):1–76, 2002.

[14]
Haruzo Hida.
Anticyclotomic main conjectures.
Doc. Math., (Extra Vol.):465–532, 2006.

[15]
Michael Harris, Kai-Wen Lan, Richard Taylor, and Jack Thorne.
On the rigid cohomology of certain Shimura varieties.
Res. Math. Sci., 3:Paper No. 37, 308, 2016.

[16]
Ming-Lun Hsieh.
On the non-vanishing of Hecke $L$-values modulo $p$.
Amer. J. Math., 134(6):1503–1539, 2012.

[17]
Ming-Lun Hsieh.
Eisenstein congruence on unitary groups and Iwasawa main
conjectures for CM fields.
J. Amer. Math. Soc., 27(3):753–862, 2014.

[18]
Ming-Lun Hsieh.
Special values of anticyclotomic Rankin-Selberg $L$-functions.
Doc. Math., 19:709–767, 2014.

[19]
Ming-Lun Hsieh.
Hida families and $p$-adic triple product $L$-functions.
American Journal of Mathematics, to appear. Preprint, arXiv:1705.02717, 2019.

[20]
Haruzo Hida and Jacques Tilouine.
Anti-cyclotomic Katz $p$-adic $L$-functions and congruence
Ann. Sci. École Norm. Sup. (4), 26(2):189–259, 1993.

[21]
Haruzo Hida and Jacques Tilouine.
On the anticyclotomic main conjecture for CM fields.
Invent. Math., 117(1):89–147, 1994.

[22]
Atsushi Ichino.
Trilinear forms and the central values of triple product
Duke Math. J., 145(2):281–307, 2008.

[23]
Dimitar Jetchev, Christopher Skinner, and Xin Wan.
The Birch and Swinnerton-Dyer formula for elliptic curves of
analytic rank one.
Camb. J. Math., 5(3):369–434, 2017.

[24]
Mark Kisin.
Moduli of finite flat group schemes, and modularity.
Ann. of Math. (2), 170(3):1085–1180, 2009.

[25]
Shin-ichi Kobayashi.
Iwasawa theory for elliptic curves at supersingular primes.
Invent. Math., 152(1):1–36, 2003.

[26]
Kai-Wen Lan.
Arithmetic compactifications of PEL-type Shimura varieties,
volume 36 of London Mathematical Society Monographs Series.
Princeton University Press, Princeton, NJ, 2013.

[27]
Kai-Wen Lan.
Boundary strata of connected components of positive characteristincs.
Algebra Number Theory, 9(9):1955–2054, 2015.
Appendix to Families of nearly ordinary Eisenstein series on
unitary groups by Xin Wan.

[28]
Kai-Wen Lan.
Compactifications of PEL-type Shimura varieties and Kuga
families with ordinary loci.
World Scientific Publishing Co. Pte. Ltd., Hackensack, NJ, 2018.

[29]
Erez M. Lapid and Stephen Rallis.
On the local factors of representations of classical groups.
In Automorphic representations, $L$-functions and
applications: progress and prospects, volume 11 of Ohio State Univ.
Math. Res. Inst. Publ., pages 309–359. de Gruyter, Berlin, 2005.

[30]
Zheng Liu and Giovanni Rosso.
Non-cuspidal Hida theory for Siegel modular forms and trivial
zeros of $p$-adic ${L}$-functions.
Math. Ann., 378(1):153–231, 2020.

[31]
Colette Mœ glin and Jean-Loup Waldspurger.
Décomposition spectrale et séries d'Eisenstein, volume
113 of Progress in Mathematics.
Birkhäuser Verlag, Basel, 1994.
Une paraphrase de l'Écriture. [A paraphrase of Scripture].

[32]
B. Mazur and A. Wiles.
Class fields of abelian extensions of ${\bf Q}$.
Invent. Math., 76(2):179–330, 1984.

[33]
Dipendra Prasad.
Trilinear forms for representations of $\mathrm {GL}(2)$ and local
$\epsilon $-factors.
Compositio Mathematica, 75(1):1–46, 1990.

[34]
Ralf Schmidt.
Some remarks on local newforms for $\rm GL(2)$.
J. Ramanujan Math. Soc., 17(2):115–147, 2002.

[35]
Christopher Skinner and Eric Urban.
The Iwasawa Main Conjectures for $GL\sb 2$.
Invent. Math., 195(1):1–277, 2014.

[36]
Eric Urban.
Selmer groups and the Eisenstein-Klingen ideal.
Duke Math. J., 106(3):485–525, 2001.

[37]
Eric Urban.
Groupes de Selmer et fonctions $L$ $p$-adiques pour les
représentations modulaires adjointes.
preprint, 2006.

[38]
Xin Wan.
Iwasawa main conjecture for supersingular elliptic curves and bsd
Preprint available at <https://arxiv.org/abs/1411.6352>, 2014.

[39]
Xin Wan.
Families of nearly ordinary Eisenstein series on unitary groups.
Algebra Number Theory, 9(9):1955–2054, 2015.
With an appendix by Kai-Wen Lan.

[40]
Xin Wan.
Iwasawa main conjecture for Rankin–Selberg $p$-adic
Algebra & Number Theory, 14(2):383–483, 2020.


# Receptive Field Broadening and Boosting for Salient Object Detection

Mingcan Ma12, Changqun Xia2$*$, Chenxi Xie12, Xiaowu Chen1, Jia
Li12111Correspondence should be addressed to Changqun Xia and Jia Li. URL:
http://cvteam.net.

###### Abstract

Salient object detection requires a comprehensive and scalable receptive field
to locate the visually significant objects in the image. Recently, the
emergence of visual transformers and multi-branch modules has significantly
enhanced the ability of neural networks to perceive objects at different
scales. However, compared to the traditional backbone, the calculation process
of transformers is time-consuming. Moreover, different branches of the multi-
branch modules could cause the same error back propagation in each training
iteration, which is not conducive to extracting discriminative features. To
solve these problems, we propose a bilateral network based on transformer and
CNN to efficiently broaden local details and global semantic information
simultaneously. Besides, a Multi-Head Boosting (MHB) strategy is proposed to
enhance the specificity of different network branches. By calculating the
errors of different prediction heads, each branch can separately pay more
attention to the pixels that other branches predict incorrectly. Moreover,
Unlike multi-path parallel training, MHB randomly selects one branch each time
for gradient back propagation in a boosting way. Additionally, an Attention
Feature Fusion Module (AF) is proposed to fuse two types of features according
to respective characteristics. Comprehensive experiments on five benchmark
datasets demonstrate that the proposed method can achieve a significant
performance improvement compared with the state-of-the-art methods.

Figure 1: Comparison of the two types of models. From left to right, it shows
the receptive field, features, frequency domain signals and efficiency.
ResNet-18 (He et al. 2015) and Swin-Transformer (Liu et al. 2021b) are
selected as the basic network to extract details and deep semantics
respectively.

## Introduction

Recently, CNN-based methods have made great progress in the SOD field due to
the powerful feature representation. However, due to the limitations of the
convolutional receptive field, it is difficult for the CNN networks to break
through the existing performance. The emergence of visual transformers has
broken the current visual attention mechanism and can understand images from
an overall perspective. However, the price of a larger receptive field is more
computational cost, especially when the input image resolution is larger.

Actually, CNN and transformers have their own distinctiveness. As shown in
Fig. 1, we observe that: 1. Features are different in the frequency domain.
CNN features reveal mostly high frequency signals, while transformer features
contain mostly low-frequency signals. 2. The computational cost increases at
different rates as the input resolution increases. Two types of models can
both run fast on low-resolution images, but when the resolution is large, the
calculation cost of Transformer is much higher than that of CNN. 3. There are
differences in regard to the receptive field mechanism. CNN expands the
receptive field from the local to the whole image through a bottom-up
calculation method, while Transformer obtains global information directly from
the overall perspective.

Based on the observation of these distinctiveness, we further analyzed the
feasibility of complementarity between CNN and transformers from three
aspects: 1. The signal difference in frequency domain indicates that the
feature characteristics of the two models can be complementary. 2. The
processing difference in attention mechanism reflects that their ways of
understanding images can be complementary. 3. The difference in processing
speed reveals that the input sizes can be complementary. Therefore, we
consider using a transformer to extract global semantic information and CNN to
extract detailed information together. In this way, transformer does not
require a large input size and CNN does not require a deeper convolutional
network, which can achieve a better balance of computational costs and
performance. To this end, we propose to build an effective bilateral network
to balance two kinds of requirements. The transformer model with low-
resolution input is used to obtain low-frequency semantic information, and the
lightweight CNN with high-resolution input is used to analyze high-frequency
details. It is worth noting that the characteristic of the CNN models is that
they can establish a perfect channel connection but the spatial receptive
field is insufficient, while the transformer models can establish a full-space
connection but the channel connection is relatively weak. So, Attention Fusion
Module (AF) is proposed to explore a better integration strategy of two types
of features.

Figure 2: A logical comparison diagram between the proposed Multi-Head
Boosting and ASPP (Chen et al. 2017). The colored squares represent dilated
convolutions with different atrous rates. The blue arrow represents forward
propagation, and the orange arrow represents the only branch randomly selected
during back propagation.

However, when the input resolution becomes smaller, the transformer’s global
attention mechanism degenerates into an association between regions, which
cannot capture enough signal range. Therefore, the goal of increasing the
receptive field cannot be totally dependent on the transformer branch. To
enhance the receptive field of CNN across regions, multiscale modules such as
ASPP (Chen et al. 2017) and PPM (Zhao et al. 2017) are good choices due to the
strong ability of extracting features of different scales in different
branches. But remarkably, the current multi-branch module adopts the strategy
of parallel training and synchronous error back propagation. This method is
not conducive to extracting discriminative features separately for different
branches.

Therefore, we put forward a novel strategy named Multi-Head Boosting (MHB) as
shown in Figure 2. MHB mainly has the following improvements: 1. Forward
propagation activates all paths, and the error back propagation process only
activates a random branch. 2. A separate prediction head is set for each
branch, and each branch is back-propagated by pixel weighting to focus on the
error parts of other branches. With MHB, we can effectively construct the
association between different regions. The combination of bilateral network
and MHB can effectively enhance the receptive field of the network and thus
achieve higher SOD performance.

Finally, we design ablation experiments to verify the effectiveness of the
proposed bilateral framework and multi-head boosting strategy. The results of
comprehensive comparison experiments show that our method has greatly exceeded
the current state-of-the-art methods. Our main contributions can be summarized
as follows:

  * •

We propose to construct an effective bilateral network based on CNN and
transformer. The lightweight CNN is responsible for fast detail extraction at
high resolution, and the transformer model can efficiently generate global
correlation features at low resolution input.

  * •

We propose an attention feature fusion module (AF) to fuse the features of the
two types of models. AF considers the advantages of the two features in space
and channel dimensions to strengthen each other.

  * •

We propose a novel multi-head boosting strategy, which makes each branch pay
more attention to the pixel position of each other’s prediction error and
breaks the synchronization relationship of the error back-propagation process.

  * •

The proposed method achieves a great performance improvement over the state-
of-the-art methods. In particular, The MAE metric of our method reach 0.021 on
ECSSD and 0.040 on DUT-OMRON, which is 29% and 20% lower than the previous
methods.

Figure 3: Framework diagram of the proposed method. This figure shows the
training process where the random number $X$ takes a value of $4$. The feature
fusion module in the figure accepts the input of the bilateral module with the
corresponding number. The prediction result of $H_{4}$ is used for gradient
back propagation, other branches do not calculate the gradient, and their
results are used to calculate the error weight.

## Related Work

### Efficient Two-Stream Network Designs

Computer vision tasks often need to extract detailed information and semantic
information of an image. Detailed information does not depend on the depth of
the network but requires larger input resolution, while semantic information
is just the opposite. In order to balance these two processes, the dual-branch
network is proposed to obtain these two features separately, such as BiSeNet
(Yu et al. 2018). This method can better balance efficiency and performance.
However, the backbones used by the two branches are often similar
convolutional neural networks, which often brings a lot of redundancy. For
example, StdcNet (Fan et al. 2021) propose that better results can be obtained
without a separate detail branch. In fact, this two-stream fusion idea is not
only suitable for the fusion between convolutional neural networks. Recently,
more and more visual backbones have been proposed, such as ResNet (He et al.
2015), MobileNet (Howard et al. 2017), StdcNet (Fan et al. 2021), ShuffleNet
(Zhang et al. 2018), Swin Transformer (Liu et al. 2021b), T2T-VIT (Yuan et al.
2021), etc. These models contain efficient lightweight CNN and high
computational cost networks based on visual self-attention. Compared with
fusing different convolutional neural networks, we propose to fuse CNN and
transformer frameworks with different but complementary mechanisms.

### Multi-scale perception

The multi-scale perception ability of neural networks often depends on the
range of the receptive field. The multi-branch module is an effective method
to enhance the receptive field of convolutional neural networks. For example,
Pyramid Pooling Moudle (Zhao et al. 2017) and Atrous Spatial Pyramid Pooling
(Chen et al. 2017) enhance the local scale perception ability of the network
by designing multi-branch modules with different receptive fields. PoolNet
(Liu et al. 2019) has used PPM for SOD for the first time and achieved good
results. BaNet (Su et al. 2019) proposes to improve ASPP and achieve the
latest new capabilities at the time. The above methods can effectively improve
the expressive ability of the network, but there are still shortcomings. The
original intention of the multi-branch model is to allow different branches to
capture specific features. However, most of the previous methods use a
synchronous error back propagation strategy and seldom consider the
specificity of the branch separately. In contrast, We use a similar multi-
model boosting idea to achieve mutual enhancement of multiple branches. Each
branch calculates the error separately and weights the error areas of the
other branches.

### Accurate Detection of Salient Objects

In the past three years, the upper limit of SOD performance has been
continuously broken by the latest methods. For example, F3Net (Wei, Wang, and
Huang 2019) proposes Fusion, Feedback and Focus to detect salient objects in
2019 and has achieved the best performance at the time. The MAE reach 0.053 on
the DUT-OMRON dataset. In 2020, LDF (Wei et al. 2020) decouples pixels based
on the distance from the edge and iteratively optimizes the predicted maps,
and its MAE on the same dataset reaches 0.051. In 2021, PA-KRN (Xu et al.
2021) proposes a strategy of positioning first, then segmentation, and its MAE
reaches 0.050. VST (Liu et al. 2021a) uses the transformer framework for SOD
for the first time, and its MAE reach 0.058. From the above results, it can be
seen that the performance of SOD is getting better and better, but the
improvement is very slow. We explore the differences between CNN and
transformers in terms of features, computational complexity, and attention
mechanisms. Based on this, we design a bilateral network and an Attention
Feature Fusion Module to balance the two models and propose a Multi-Head
Boosting method to compensate for the lack of association between regions. Due
to the complementary combination, our method has achieved good results in both
performance and efficiency. In particular, the MAE metric of our method on
ECSSD has reached 0.040.

## Proposed Method

In this section, we will introduce the details of the composition of the
bilateral network, the Attention Feature Fusion Module, and the Multi-Head
Boosting strategy. The training process of the model is shown in Fig. 3. The
verification process only needs to add up the prediction results of each
branch and then pass the activation function.

### Bilateral Network

As can be seen from Fig. 1, CNN and transformers have both distinctiveness and
complementarity. Therefore, we propose a bilateral network as shown in Fig. 3
to fully utilize the complementary advantages. The bilateral network contains
a semantic branch and a detail branch. The semantic branch makes full use of
the global view of the transformer model to extract global semantic features
and enhance the receptive field of the overall network. The detail branch is
responsible for extracting high frequency details. The ingenuity of the
bilateral network is that it can take advantage of the two types of models at
the lowest computational cost. The complex semantic branch is mainly
responsible for extracting global attention information so it can be
calculated under low-resolution input. The detail branch only requires a
lightweight CNN as basic network. Therefore, the combination of transformer
and CNN can efficiently balance semantics, details, and their calculation
cost.

Figure 4: Illustration of the Attention Feature Fusion Module. X and Y
represent the output feature of the transformer model and the output feature
of CNN, respectively.

Specifically, we choose ResNet-18 (He et al. 2015) to build the detail branch.
In order to enrich the detailed information extracted by this branch, we
remove the first down-sampling operation to obtain more detailed features at
high resolution. In this way, the resolution of the features output by each
layer is $1/2$, $1/4$, $1/8$, and $1/16$ of the initial resolution. As for the
semantic branch, we utilize Swin Transformer (Liu et al. 2021b) to extract
global features. This branch first samples the image to a resolution of
$56\times 56$, and then passes through the self-attention modules of the
transformer in turn.

The working process of proposed bilateral network can be summarized as
follows: Input the $352\times 352$ resolution image into the lightweight CNN
to obtain multi-level detailed features $R_{1}$,$R_{2}$,$R_{3}$ and $R_{4}$.
Besides, the input image is down-sampled to a resolution of $56\times 56$.
Then we input it to the semantic branch to obtain features $T_{1}$, $T_{2}$,
$T_{3}$, and $T_{4}$ and reshape the features. Finally, Use the attention
feature fusion module $AF_{i}$ to fuse $T_{i}$ and $R_{i}$, where
$i\in\\{1,2,3,4\\}$.

### Attention Feature Fusion Module

Unlike the features of CNN, the features of transformer model are composed of
vectors stretched by pixel blocks. Given the transformer features, we first
restore the previous positional relationship of the vector to get feature $X$
of Fig. 4. The transformer model calculates the correlation between all patch
blocks in the space through the vector inner product, while CNN establishes
the connection between all channels in the local space. The former constructs
better spatial correlation, while the latter has stronger channel correlation.
In order to better integrate the two types of features, we design an Attention
Feature Fusion Module (AF). AF uses the respective characteristics of the two
types of features to enhance the expression of each other. The specific
process can be expressed as:

$\displaystyle F_{mid}$
$\displaystyle=\mathcal{C}_{br}(\mathcal{U}(\mathcal{R}_{ca}(Y)\otimes
X)ⓒ(\mathcal{R}_{sa}(X)\otimes Y)),$ (1) $\displaystyle F_{out}$
$\displaystyle=\mathcal{C}_{br}(\mathcal{U}(\mathcal{C}(X))\oplus
F_{mid}\oplus\mathcal{C}(Y)),$ (2)

where $X$ and $Y$ correspond to the features in Fig. 4. $\mathcal{R}_{sa}$ and
$\mathcal{R}_{ca}$ represent the operations corresponding to the crossed
arrows in the figure. $\mathcal{U}$ represents the upsampling operation.
$\mathcal{C}$ represents convolution, and the subscript $br$ represents a
Batch Normalization and a ReLU activation. $\oplus$, $\otimes$ represent
pixel-wise addition and multiplication operations, respectively. $ⓒ$ denotes
feature concatenation operation. $F_{mid}$ is the intermediate result,
$F_{out}$ is the final output.

Specifically, the feature $X$ obtained by transformer contains rich semantic
information, while the low level details are lost. CNN feature $Y$ is obtained
by convolution, the spatial receptive field is smaller, but the correlation
between channels is stronger. Therefore, we use the features of $X$ to select
the spatial Features of $Y$, and use the channel information of $Y$ to enhance
$X$. Finally, the final result is obtained by fusing all the information
through the residual structure.

### Multi-Head Boosting

The design idea of Multi-Head Boosting (MHB) is to improve the overall effect
of the multi-branch module by decoupling the training process and adding
branch complementary losses. It can be roughly divided into two parts: multi-
path decoupling and boosting loss. Next, we will introduce these two processes
in detail.

Multi-path decoupling aims to decouple multi-branch modules trained in
parallel into multiple models. The significance is that the training process
of different branches can be isolated, thereby reducing the gradient
correlation between each branch and enhancing the scale specificity of
different branches. ASPP integrates multi-channel features in the module,
single-channel training cannot be performed directly, therefore we cannot use
multi-channel features in the testing process. Our solution is to delay
feature merging to the final result layer. In this way, the purpose of single-
channel training and multi-channel testing can be realized.

Figure 5: Comparison of PR curves between our method and the 14 latest
methods.

Boosting Loss aims to enhance the complementarity between branches. As shown
in Fig. 3, the forward propagation process will activate all branches and
obtain $N$ predicted maps $\\{P_{i}|i=1,2,3,...,N\\}$. The random number $X$
is used to determine which branch performs error back propagation, and the
prediction errors of other branches are regarded as weight. In this way, each
branch will pay attention to the mispredicted parts of other branches. The
calculation process of weight can be expressed as:

$\displaystyle W_{x}=\sum_{i=1}^{N}\mathcal{L}_{bce}(P_{i},g)(i\neq X)+1,$ (3)

where $g$ represents the ground-truth map. $\mathcal{L}_{bce}$ denotes the
pixel-level binary cross-entropy error, which can be expressed as:

$\displaystyle\mathcal{L}_{bce}(p,g)=-((g\otimes
log(p))\oplus(\overline{g}\otimes log(\overline{p}))),$ (4)

where $p$ denotes predicted map. $log$ represents a pixel-by-pixel logarithmic
operation. $\overline{g}$ and $\overline{p}$ denote the inversion operation
pixel by pixel. The boosting loss can be calculated according to the
calculated weight. It can be expressed as:

$\displaystyle\mathcal{L}_{b}(p,g,w)=\mathcal{L}_{wbce}(p,g,w)+\mathcal{L}_{wiou}(p,g,w),$
(5)

where $\mathcal{L}_{wbce}$ and $\mathcal{L}_{wiou}$ represent weighted binary
cross entropy loss and weighted intersection of union loss. They can be
expressed as:

$\displaystyle\mathcal{L}_{wbce}(p,g,w)=\frac{\mathcal{S}(\mathcal{L}_{bce}(p,g)\otimes
w)}{\mathcal{S}(w)},$ (6)
$\displaystyle\mathcal{L}_{wiou}(p,g,w)=1-\frac{\mathcal{S}(p\otimes g\otimes
w)}{\mathcal{S}((p\oplus g)\otimes w)-\mathcal{S}(p\otimes g\otimes w)},$ (7)

where $\mathcal{S}$ represents the operation of summing all pixels. Other
symbols are consistent with the previous description.

In general, MHB decomposes the training process of multi-scale modules so that
multiple branches receive training with random samples and relieves the mutual
influence of the error back propagation process. BL further enhances the
complementarity between each branch. The experiments prove that MHB and BL can
effectively improve SOD performance.

### Loss Function

We utilize the sum of Binary Cross Entropy (BCE) and Intersection of Union
(IoU) as the loss function, which is widely used in LDF (Wei et al. 2020),
etc. In addition to the final prediction map, we will also supervise the
output features of each Convolution Block (CB) in Fig. 4. The total loss
function can be expressed as:

$\displaystyle\mathcal{L}_{tot}=\sum_{i=1}^{M}\mathcal{L}_{bi}(F_{i},g)+\mathcal{L}_{b}(P_{4},g,W_{x}),$
(8)

where $F_{i}$ represents the prediction map of each CB module, and $P_{4}$
represents the final result. $M$ denotes the number of CB modules.
$\mathcal{L}_{bi}$ can be expressed as:

$\displaystyle\mathcal{L}_{bi}(p,g)=\mathcal{L}_{wbce}(p,g,W_{1})+\mathcal{L}_{wiou}(p,g,W_{1}),$
(9)

where $W_{1}$ represents a matrix with all $1$ values, which means that the
auxiliary loss is not weighted.

## Experiment

### Experiment setting

The evaluation indicators are mainly as follows: Mean Absolute Error (MAE),
max and mean F-measure ($F_{\beta}^{*},F_{\beta}^{m}$) (Yang et al. 2013),
Maximum enhanced-alignment measure ($E_{\xi}$) (Fan et al. 2018), structure
measure ($S_{m}$) (Fan et al. 2017) and precision-recall (PR).

The experiment involves the following datasets: DUT-OMRON (Yang et al. 2013)
(5168), ECSSD (Yan et al. 2013) (1000), HKU-IS (Li and Yu 2015) (4447),
PASCAL-S (Li et al. 2014) (850), DUTS-TE (Wang et al. 2017) (5019), DUTS-TR
(Wang et al. 2017) (10553). We choose DUTS-TR as the training set and other
datasets as the test set.

Table 1: Quantitative comparison table with the latest methods on multiple
indicators, including the max and mean F-measure ($F_{\beta}^{*}$ and
$F_{\beta}^{m}$ the larger the better), MAE (the smaller the better),
E-measure ($E_{\xi}$, the larger the better), and S-measure ($S_{m}$, the
larger the better) . The best and second best results are marked in red, and
blue, respectively.

| ECSSD (1000) | HKU-IS (4447) | DUTS-TE (5019) | DUT-OMRON (5168) | PASCAL-S (850)
---|---|---|---|---|---
Method | $F_{\beta}^{*}\uparrow$ | $F_{\beta}^{m}\uparrow$ | $mae\downarrow$ | $E_{\xi}\uparrow$ | $S_{m}\uparrow$ | $F_{\beta}^{*}\uparrow$ | $F_{\beta}^{m}\uparrow$ | $mae\downarrow$ | $E_{\xi}\uparrow$ | $S_{m}\uparrow$ | $F_{\beta}^{*}\uparrow$ | $F_{\beta}^{m}\uparrow$ | $mae\downarrow$ | $E_{\xi}\uparrow$ | $S_{m}\uparrow$ | $F_{\beta}^{*}\uparrow$ | $F_{\beta}^{m}\uparrow$ | $mae\downarrow$ | $E_{\xi}\uparrow$ | $S_{m}\uparrow$ | $F_{\beta}^{*}\uparrow$ | $F_{\beta}^{m}\uparrow$ | $mae\downarrow$ | $E_{\xi}\uparrow$ | $S_{m}\uparrow$
$BASNet_{19}$ | .942 | .880 | .037 | .921 | .916 | .928 | .895 | .032 | .946 | .909 | .860 | .791 | .048 | .884 | .866 | .805 | .756 | .056 | .869 | .836 | .857 | .775 | .076 | .847 | .832
$PoolNet_{19}$ | .944 | .914 | .039 | .924 | .922 | .933 | .896 | .032 | .949 | .910 | .880 | .811 | .040 | .889 | .878 | .808 | .746 | .056 | .863 | .828 | .869 | .823 | .074 | .850 | .847
$CPD_{19}$ | .939 | .917 | .037 | .925 | .918 | .925 | .891 | .034 | .944 | .905 | .865 | .805 | .043 | .887 | .869 | .797 | .747 | .056 | .866 | .825 | .864 | .824 | .072 | .849 | .842
$BANet_{19}$ | .945 | .880 | .035 | .928 | .916 | .931 | .895 | .032 | .950 | .909 | .872 | .791 | .040 | .892 | .866 | .803 | .756 | .059 | .860 | .836 | .870 | .775 | .070 | .855 | .832
$GateNet_{20}$ | .945 | .916 | .040 | .924 | .920 | .933 | .899 | .033 | .949 | .915 | .888 | .807 | .040 | .889 | .885 | .818 | .746 | .055 | .862 | .838 | .875 | .825 | .068 | .852 | .852
$U2Net_{20}$ | .951 | .892 | .033 | .924 | .928 | .935 | .896 | .031 | .948 | .916 | .873 | .792 | .045 | .886 | .874 | .823 | .761 | .054 | .871 | .847 | .862 | .772 | .076 | .841 | .836
$DFI_{20}$ | .949 | .920 | .035 | .924 | .927 | .934 | .902 | .031 | .951 | .920 | .886 | .814 | .039 | .892 | .887 | .818 | .752 | .055 | .865 | .839 | .885 | .837 | .065 | .857 | .861
$MINet_{20}$ | .947 | .924 | .033 | .927 | .925 | .935 | .909 | .029 | .953 | .919 | .884 | .828 | .037 | .898 | .884 | .810 | .755 | .055 | .865 | .833 | .865 | .835 | .064 | .852 | .851
$GCPANet_{20}$ | .948 | .919 | .035 | .920 | .927 | .938 | .898 | .031 | .949 | .920 | .888 | .817 | .038 | .891 | .891 | .812 | .748 | .056 | .860 | .839 | .876 | .833 | .061 | .850 | .861
$ITSDNet_{20}$ | .947 | .895 | .034 | .927 | .925 | .934 | .899 | .031 | .952 | .917 | .883 | .804 | .041 | .895 | .885 | .821 | .756 | .061 | .863 | .840 | .876 | .792 | .064 | .853 | .856
$LDF_{20}$ | .950 | .930 | .034 | .925 | .924 | .939 | .914 | .027 | .954 | .919 | .898 | .855 | .034 | .910 | .892 | .820 | .773 | .051 | .873 | .838 | .874 | .843 | .059 | .865 | .856
$PFSNet_{21}$ | .952 | .932 | .031 | .928 | .930 | .943 | .919 | .026 | .956 | .924 | .896 | .847 | .036 | .902 | .892 | .823 | .774 | .055 | .875 | .842 | .875 | .837 | .063 | .856 | .854
$VST_{21}$ | .951 | .920 | .033 | .918 | .932 | .942 | .900 | .029 | .953 | .928 | .890 | .818 | .037 | .892 | .896 | .825 | .756 | .058 | .861 | .850 | .875 | .829 | .061 | .837 | .865
$PAKRN_{21}$ | .953 | .931 | .032 | .924 | .928 | .943 | .920 | .027 | .955 | .923 | .907 | .865 | .033 | .916 | .900 | .834 | .793 | .050 | .885 | .853 | .873 | .838 | .066 | .857 | .852
OURS | .964 | .949 | .022 | .932 | .941 | .953 | .936 | .020 | .965 | .933 | .920 | .890 | .025 | .925 | .910 | .838 | .804 | .040 | .878 | .847 | .890 | .862 | .050 | .866 | .867

We use NVDIA GTX 2080Ti to train our network. The input resolution of the
detail branch is $352\times 352$, and the input resolution of the first
transformer block is $56\times 56$. The maximum learning rate of the backbone
is 0.004, and the other parts are expanded by ten times. And the learning rate
will first increase and then decrease when training our model. The
optimization method uses Stochastic Gradient Descent. Batch size is set to 26,
and the epoch is set to 32. The data augmentation methods involves multi-scale
training, random flipping and cropping. The prediction results do not need any
post-processing.

### Methods of Comparison

The experimental process involves the state-of-the-art methods of the last
three years. Four of them in 2019 include BANet (Su et al. 2019), BASNet (Qin
et al. 2019), PoolNet (Liu et al. 2019) and CPD (Wu, Su, and Huang 2019).
Seven methods in 2020, including LDF (Wei et al. 2020), MINet (Pang et al.
2020), GCPANet (Chen et al. 2020), GateNet (Zhao et al. 2020), DFI (Liu, Hou,
and Cheng 2020), ITSDNet (Zhou et al. 2020) and U2Net (Qin et al. 2020). And
there methods published in 2021: PAKRN (Xu et al. 2021), VST (Liu et al.
2021a) and PFSNet (Ma, Xia, and Li 2021). The evaluation process uses a
unified evaluation code to evaluate the published saliency map.

Table 2: Ablation experiments of the proposed method. Bi is the abbreviation
of Bilateral. The last line is the final result of our method.

| ECSSD | DUTS-TE | DUT-OMRON
---|---|---|---
Method | $F_{\beta}^{m}\uparrow$ | $mae\downarrow$ | $F_{\beta}^{m}\uparrow$ | $mae\downarrow$ | $F_{\beta}^{m}\uparrow$ | $mae\downarrow$
Detail branch | .867 | .061 | .716 | .067 | .644 | .094
Semantic branch | .848 | .033 | .845 | .033 | .780 | .050
Bilateral | .930 | .027 | .858 | .029 | .788 | .047
Bi+AF | .940 | .025 | .869 | .028 | .790 | .045
Bi+AF+MBD2 | .941 | .025 | .874 | .027 | .798 | .045
Bi+AF+MBD3 | .946 | .024 | .882 | .028 | .799 | .044
Bi+AF+MBD4 | .947 | .023 | .888 | .026 | .801 | .041
Bi+AF+MBD5 | .946 | .024 | .886 | .027 | .800 | .042
Bi+AF+ASPP | .941 | .025 | .864 | .028 | .793 | .044
Bi+AF+MBD4+BL | .949 | .022 | .890 | .025 | .804 | .040

### Performance Comparison.

The verification results prove that the proposed method achieves a
breakthrough in performance compared with the latest method. First of all, it
can be seen from the PR curve in Fig. 5 that the PR curve of the previous
methods has not improved significantly, but our method is significantly better
than other methods. In addition, the performance comparison results in Tab. 1
can also verify the effectiveness of the method in this paper. Our method has
mostly surpassed the previous method in five commonly used indicators. And the
performance improvement is very significant. Taking the ECSSD dataset as an
example, the MAE indicator reached $0.033$ in 2020, and it has been reduced to
$0.031$ in 2021, a reduction of only $6\%$. But our method reached $0.022$, an
decrease of $29\%$ compared to last year. The experimental data on DUTS-TE
shows that the MAE metric of LDF (Wei et al. 2020) in 2020 has reached
$0.034$, while the result in 2021 is basically unchanged. But our method
reduces the MAE indicator to $0.025$, which is a $24\%$ reduction. From the
results of F-measure, the proposed method greatly improves the F-measure value
calculated at different thresholds.

Figure 6: Visual comparison between the proposed method and the state-of-the-
art methods. Figure 7: Visualization of prediction results of different
prediction heads in MBD. H1, H4 represent the predicted maps of the first and
fourth branches, respectively.

### Ablation Studies

In order to verify the innovations of this paper one by one, we conduct
ablation experiments on the proposed Bilateral Network, Attention Fusion
Module AF, Multi-Branch Decoupling, and Boosting Loss. The experimental
details are shown in Tab. 2.

#### Bilateral Network.

The results of the first three rows of Tab. 2 verify the effectiveness of the
bilateral network. The detailed branch represents a framework composed of a
lightweight CNN and FPN (Lin et al. 2017) decoder. Here, the CNN network
selects ResNet-18 (He et al. 2015), and the input resolution is set to
$352\times 352$. The semantic branch selects Swin Transformer (Liu et al.
2021b) as basic network and utilizes the same decoder. The input resolution of
semantic branch is set to $56\times 56$. It can be seen from the experimental
results that the performance of a pure CNN or transformer model is not ideal.
The lightweight CNN lacks deep semantic information and the transformer model
with low-resolution input lacks high-frequency details. When we combine the
two models into a bilateral network, the performance can be greatly improved.

#### Attention Fusion Module (AF)

AF improves their performance through cross-attention compensation. The
performance of the fourth row in Tab. 2 is significantly improved than that of
the third row. On different datasets, the F-measure metric can be improved by
more than 1 percentage point. And the Average Absolute Error shows a downward
trend. Experiments have proved that the bilateral network can achieve higher
performance under the blessing of AF.

#### Multi-Branch Decoupling (MBD).

Tab. 2 also verifies the effect of multi-branch decoupling and includes an
ablation experiment of the number of branches $N$. The data in the fourth row
to the eighth row shows the change in performance as the number of branches
increases. When the number of branches increases to 4, the performance is
close to saturation. Therefore, we finally chose to use four branches. In
addition, the effect of directly adding ASPP to the bilateral network in the
table is not obvious. This verifies the role of MBD in the serial training of
multi-branch sub-networks.

#### Boosting Loss (BL).

In order to verify the effect of BL, we analyze it from both quantitative and
qualitative perspectives. The last row in Tab. 2 shows the results of BL
acting on the 4 branches. Compared to the model without BL, the average
performance has improved and basically exceeded the best result of the
ablation test. Fig. 7 shows the visualization results of the first and fourth
branches. It is easy to see that different branches can extract different
detailed features, and the fusion result can effectively integrate the results
of different branches.

#### Visual comparison.

The latest method and the visualization results of the proposed method are
shown in Fig. 6. It is easy to see that the existing methods mainly have the
following problems: 1. Some objects are missing in the prediction result when
there are multiple salient objects. 2. Incomplete prediction of a single
object. 3. Improper handling of details generates noise. Among the methods
compared, VST is based on the transformer model, while other methods are based
on CNN. We observe that VST can handle problems 1 and 2 better, but the
segmentation details are often not ideal.As shown in the second and fourth
rows of the figure, the CNN model is difficult to deal with problem 1 and
problem 2 due to the limited receptive field. It can be seen from the third
column that our model can effectively combine the advantages of the two types
of models. This also verifies the expansion and boosting effect of our method
on the receptive field. The results of the first two rows verify that the
method in this paper can effectively solve the problem of missing objects. The
results in the third and fourth rows show that our method has advantages in
detail processing. And the data in the last row shows that even if the object
is large, our model can still fully predict and accurately segment the
results. In short, the visualization results reflect that the proposed method
can better alleviate the problems caused by the receptive field.

## Conclusion

In this paper, we rethink the role of the receptive field in SOD and propose a
bilateral network based on vision transformer and lightweight CNN to broaden
the receptive field of the network. in order to better integrate the
characteristics of the bilateral network, we design an attention feature
fusion module based on the characteristics of the two types of features. In
the proposed bilateral network, transformer branches input low-resolution
images to efficiently generate semantic information. However, this operation
causes the global attention mechanism of transformer branch to degenerate into
an association mechanism between pixel regions. To enhance the attention
between these regions, we propose a Multi-Head Boosting strategy to compensate
for the loss of the global receptive field. Experiments show that our method
can achieve impressive results under various evaluation indicators on multiple
benchmark datasets.

## References

  * Chen et al. (2017) Chen, L.-C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; and Yuille, A. L. 2017.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  _IEEE transactions on pattern analysis and machine intelligence_ , 40(4): 834–848.
  * Chen et al. (2020) Chen, Z.; Xu, Q.; Cong, R.; and Huang, Q. 2020.  Global context-aware progressive aggregation network for salient object detection.  _arXiv preprint arXiv:2003.00651_.
  * Fan et al. (2017) Fan, D.; Cheng, M.; Liu, Y.; Li, T.; and Borji, A. 2017.  A new way to evaluate foreground maps.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 245484557.
  * Fan et al. (2018) Fan, D.-P.; Gong, C.; Cao, Y.; Ren, B.; Cheng, M.-M.; and Borji, A. 2018.  Enhanced-alignment measure for binary foreground map evaluation.  _arXiv preprint arXiv:1805.10421_.
  * Fan et al. (2021) Fan, M.; Lai, S.; Huang, J.; Wei, X.; Chai, Z.; Luo, J.; and Wei, X. 2021.  Rethinking BiSeNet For Real-time Semantic Segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 9716–9725.
  * He et al. (2015) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2015.  Deep Residual Learning for Image Recognition.  arXiv:1512.03385.
  * Howard et al. (2017) Howard, A. G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; and Adam, H. 2017.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _arXiv preprint arXiv:1704.04861_.
  * Li and Yu (2015) Li, G.; and Yu, Y. 2015.  Visual saliency based on multiscale deep features.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 5455–5463.
  * Li et al. (2014) Li, Y.; Hou, X.; Koch, C.; Rehg, J. M.; and Yuille, A. L. 2014.  The secrets of salient object segmentation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 280–287.
  * Lin et al. (2017) Lin, T.-Y.; Dollár, P.; Girshick, R.; He, K.; Hariharan, B.; and Belongie, S. 2017.  Feature pyramid networks for object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2117–2125.
  * Liu, Hou, and Cheng (2020) Liu, J.-J.; Hou, Q.; and Cheng, M.-M. 2020.  Dynamic Feature Integration for Simultaneous Detection of Salient Object, Edge and Skeleton.  _arXiv preprint arXiv:2004.08595_.
  * Liu et al. (2019) Liu, J.-J.; Hou, Q.; Cheng, M.-M.; Feng, J.; and Jiang, J. 2019.  A Simple Pooling-Based Design for Real-Time Salient Object Detection.  In _IEEE CVPR_.
  * Liu et al. (2021a) Liu, N.; Zhang, N.; Wan, K.; Han, J.; and Shao, L. 2021a.  Visual Saliency Transformer.  _arXiv preprint arXiv:2104.12099_.
  * Liu et al. (2021b) Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; and Guo, B. 2021b.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.  arXiv:2103.14030.
  * Ma, Xia, and Li (2021) Ma, M.; Xia, C.; and Li, J. 2021.  Pyramidal Feature Shrinking for Salient Object Detection.  35: 2311–2318.
  * Pang et al. (2020) Pang, Y.; Zhao, X.; Zhang, L.; and Lu, H. 2020.  Multi-Scale Interactive Network for Salient Object Detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 9413–9422.
  * Qin et al. (2020) Qin, X.; Zhang, Z.; Huang, C.; Dehghan, M.; Zaiane, O. R.; and Jagersand, M. 2020\.  U2-Net: Going deeper with nested U-structure for salient object detection.  _Pattern Recognition_ , 106: 107404.
  * Qin et al. (2019) Qin, X.; Zhang, Z.; Huang, C.; Gao, C.; Dehghan, M.; and Jagersand, M. 2019.  BASNet: Boundary-Aware Salient Object Detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Su et al. (2019) Su, J.; Li, J.; Zhang, Y.; Xia, C.; and Tian, Y. 2019.  Selectivity or Invariance: Boundary-aware Salient Object Detection.  In _ICCV_.
  * Wang et al. (2017) Wang, L.; Lu, H.; Wang, Y.; Feng, M.; Wang, D.; Yin, B.; and Ruan, X. 2017.  Learning to detect salient objects with image-level supervision.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 136–145.
  * Wei, Wang, and Huang (2019) Wei, J.; Wang, S.; and Huang, Q. 2019.  F3Net: Fusion, feedback and focus for salient object detection.  _arXiv preprint arXiv:1911.11445_.
  * Wei et al. (2020) Wei, J.; Wang, S.; Wu, Z.; Su, C.; Huang, Q.; and Tian, Q. 2020.  Label decoupling framework for salient object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 13025–13034.
  * Wu, Su, and Huang (2019) Wu, Z.; Su, L.; and Huang, Q. 2019.  Cascaded partial decoder for fast and accurate salient object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 3907–3916.
  * Xu et al. (2021) Xu, B.; Liang, H.; Liang, R.; and Chen, P. 2021.  Locate Globally, Segment Locally: A Progressive Architecture With Knowledge Review Network for Salient Object Detection.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, 3004–3012.
  * Yan et al. (2013) Yan, Q.; Xu, L.; Shi, J.; and Jia, J. 2013.  Hierarchical saliency detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1155–1162.
  * Yang et al. (2013) Yang, C.; Zhang, L.; Lu, H.; Ruan, X.; and Yang, M.-H. 2013.  Saliency detection via graph-based manifold ranking.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 3166–3173.
  * Yu et al. (2018) Yu, C.; Wang, J.; Peng, C.; Gao, C.; Yu, G.; and Sang, N. 2018.  BiSeNet: Bilateral Segmentation Network for Real-Time Semantic Segmentation.  In Ferrari, V.; Hebert, M.; Sminchisescu, C.; and Weiss, Y., eds., _Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIII_ , volume 11217 of _Lecture Notes in Computer Science_ , 334–349. Springer.
  * Yuan et al. (2021) Yuan, L.; Chen, Y.; Wang, T.; Yu, W.; Shi, Y.; Jiang, Z.; Tay, F. E.; Feng, J.; and Yan, S. 2021.  Tokens-to-token vit: Training vision transformers from scratch on imagenet.  _arXiv preprint arXiv:2101.11986_.
  * Zhang et al. (2018) Zhang, X.; Zhou, X.; Lin, M.; and Sun, J. 2018.  Shufflenet: An extremely efficient convolutional neural network for mobile devices.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 6848–6856.
  * Zhao et al. (2017) Zhao, H.; Shi, J.; Qi, X.; Wang, X.; and Jia, J. 2017.  Pyramid Scene Parsing Network.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 6230–6239.
  * Zhao et al. (2020) Zhao, X.; Pang, Y.; Zhang, L.; Lu, H.; and Zhang, L. 2020.  Suppress and balance: A simple gated network for salient object detection.  _arXiv preprint arXiv:2007.08074_.
  * Zhou et al. (2020) Zhou, H.; Xie, X.; Lai, J.-H.; Chen, Z.; and Yang, L. 2020.  Interactive Two-Stream Decoder for Accurate and Fast Saliency Detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 9141–9150.