# Scotogenic top-quark FCNC decays

Chuan-Hung Chen<EMAIL_ADDRESS>Department of Physics, National
Cheng-Kung University, Tainan 70101, Taiwan  Physics Division, National Center
for Theoretical Sciences, Taipei 10617, Taiwan.  Takaaki Nomura
<EMAIL_ADDRESS>College of Physics, Sichuan University, Chengdu 610065,
China

###### Abstract

Flavor-changing neutral current (FCNC) top-quark decays are highly suppressed
due to the Glashow-Iliopoulos-Maiani mechanism in the standard model (SM). If
$t\to qh,qV$ with $V=g,\gamma,Z$ are all induced via quantum loop levels, then
we investigate the effect that can enhance the top-FCNC up to the sensitivity
designed at the high-luminosity (HL) LHC. Inspired by the mechanism of the
scotogenic neutrino mass, we extend the SM by including $Z_{2}$-odd colored
fermions when a $Z_{2}$ discrete is imposed. The results show that by taking
$BR(t\to ug)\lesssim 0.61\times 10^{-4}$ recently measured by ATLAS as an
input, $t\to q\gamma$ can be indirectly bounded to be $BR(t\to
q\gamma)\lesssim 3.2\times 10^{-6}$, which is below the expected sensitivity
at the HL LHC. After taking potential constraints from various experiments
into account, the obtained branching ratios for the loop-induced $t\to qh$ and
$t\to qZ$ decays can be $O(10^{-4})$, which falls within the sensitivity at
the HL LHC.

##  I Introduction

Flavor-changing neutral currents (FCNCs) are suppressed at the tree level and
can be induced via quantum corrections in the standard model (SM). However,
loop-induced top-quark FCNCs are highly suppressed in the SM due to the
Glashow-Iliopoulos-Maiani (GIM) mechanism Glashow:1970gm . As a result, the
branching ratios (BRs) for the $t\to q(g,\gamma,Z,h)$ decays with $q=u,c$ in
the SM are of the order of $10^{-12}-10^{-17}$ AguilarSaavedra:2004wm ;
Abbas:2015cua ; Balaji:2020qjg , and the results are far below the LHC
sensitivities.

The expected sensitivities in the high-luminosity (HL) LHC with an integrated
luminosity of 3ab-1 at $\sqrt{s}=14$ TeV are expected to be $2.4-5.8\times
10^{-5}$ for $t\to qZ$ ATLAS:2019pcn , $1.5\times 10^{-4}$ for $t\to qh$
ATLAS:2016qxw , and $0.9-7\times 10^{-5}$ for $t\to q\gamma$
collaboration:2017gbu . The event that the top-FCNC is found at the $10^{-5}$
level definitely indicates a new physics effect. Thus, the top-quark flavor-
changing processes can serve as good candidates for investigating new physics
effects. To explore the new physics effects in the rare top-quark decays,
various extensions of the SM, which can reach the HL LHC sensitivity, were
proposed in Abraham:2000kx ; Eilam:2001dh ; AguilarSaavedra:2002kr ;
Gaitan:2017tka ; Shen:2017oel ; Chiang:2018oyd ; Oyulmaz:2018irs ;
Chen:2018lze ; Arroyo-Urena:2019qhl ; Shi:2019epw ; Liu:2020kxt ; Hou:2020ciy
; Bie:2020sro ; Gutierrez:2020eby ; Liu:2021crr ; Cai:2022xha ; Hernandez-
Juarez:2022kjx ; Badziak:2017wxn ; Dey:2016cve .

The top-FCNC effects not only can be detected via the new top-quark decay
channels, but also can be used to produce more top-quark events, such as
$gq\to t$. Using the single-top production, the upper limits on the BR using
139 fb-1 at $\sqrt{s}=13$ TeV in ATLAS are shown as $BR(t\to ug)<0.61\times
10^{-4}$ and $BR(t\to cg)<3.7\times 10^{-4}$ ATLAS:2021amo .

If all mentioned top-FCNC processes are enhanced via quantum loop effects, the
intermediate states in the loops may carry the quantum number that the SM
particles do not possess. A known example is the radiative neutrino mass in a
scotogenic model Ma:2006km , where the mediated particles in the loop are
$Z_{2}$-parity odd, whereas the SM particles are $Z_{2}$-parity even.
Interestingly, in addition to the explanation of the neutrino mass, the
predicting dark matter (DM) candidate can fit the DM relic density that was
observed by the Planck collaboration Planck:2018vyg .

If we have $Z_{2}$-odd colorless fermions, it is reasonable that there also
exist $Z_{2}$-odd colored fermions, which are similar to the quarks in the SM.
Hereafter, we call these particles as $Z_{2}$-odd quarks. Based on this
assumption, when we make a minimal extension of the Ma-model proposed in
Ma:2006km , we investigate if the top-FCNC processes can be enhanced up to the
sensitivities of the HL LHC. As the minimal requirement, no new local gauge
symmetry is considered, and the number of including $Z_{2}$-odd quarks is as
less as possible. Since a new local gauge symmetry introduces a new gauge
coupling and gauge boson(s), which are not directly related to the top-FCNC
processes, thus, we only focus on the SM gauge symmetry.

Because the $t\to q\gamma(h)$ decay involves the structure of a dipole
(scalar) current, the chirality in the initial and final quarks has to be
different. In addition, the left-handed and right-handed top quarks are
$SU(2)$ doublet and singlet in the SM, respectively. Therefore, to avoid the
chirality suppression of $m^{2}_{t}/\Lambda^{2}$, where $\Lambda$ is the mass
scale of the new heavy particle, the candidate of new colored fermions should
be chosen in such a way that the chirality suppression can be overcome.
Without introducing extra new $Z_{2}$-odd scalar field with the exception of
the inert Higgs doublet, we find that the possibly minimal representations for
the colored fermions in $SU(2)_{L}\times U(1)_{Y}$ gauge symmetry are the
vector-like doublet and singlet. Vector-like particles are used because their
gauge anomaly can be evaded.

Three scalar bosons exist in an inert Higgs doublet, namely, inert charged
Higgs, scalar, and pesudoscalar, where the lightest neutral inert scalar can
be the DM candidate. Due to the suppression factor of $m^{2}_{t}/\Lambda^{2}$,
which arises from the chirality flip of top-quark, the contributions from the
neutral inert scalars to $t\to qh$ are small; therefore, the $t\to qh$
processes are dominated by the inert charged-Higgs. Although $t\to q\gamma$
can avoid the chirality suppression, since the loop-induced effect is
associated with the mixing between $Z_{2}$-odd doublet and singlet quark,
which is of order of $v/\Lambda$ with $v$ being the vacuum expectation value
of Higgs field, $t\to q\gamma$ dominated by the inert charged Higggs in the
model cannot be enhanced up to the sensitivity of HL LHC. For the $t\to qZ$
processes, both neutral and charged scalars can have significant
contributions.

The remainder of this paper is organized as follows: We introduce the model
and derive the relevant Yukawa and gauge couplings in Section II. Based on the
obtained couplings, we formulate the decay amplitudes and BRs for the studied
top decays in Section III. Then, we discuss various possible constraints,
which include radiative $B$-meson decay, oblique parameters, Higgs production,
and DM detections, in Section IV. We analyze and discuss the numerical results
in detail in Section V. Finally, we summarize the study in Section VI.

##  II  Model and the new couplings

To enhance the top-quark FCNC processes through radiative corrections without
introducing a new gauge symmetry in a scotogenic mechanism, we impose a
$Z_{2}$ discrete symmetry in the model and extend the SM, including a new
Higgs doublet $(\Phi_{I})$, three singlet Majorana fermions $N_{Ri}$ ($i=$1-3)
and vector-like quark doublet ($Q_{4}$) and singlet ($B^{\prime}$), where the
introduced particles are $Z_{2}$-odd and the SM particles are $Z_{2}$-even
under the $Z_{2}$ transformation. The representations and charge assignments
of new particles in $SU(2)_{L}\times U(1)_{Y}\times Z_{2}$ are shown in Table.
1. The $Z_{2}$ odd Majorana fermions $N_{R}$s are necessary to generate
neutrino mass by scotogenic mechanism. In this work we do not discuss neutrino
mass generation since it is exactly the same as original Ma-model Ma:2006km .
Due to the unbroken $Z_{2}$ symmetry, the neutral component of the inert Higgs
doublet and the lightest Majorana fermion can be the DM candidate
Barbieri:2006dq . Since the inert Higgs doublet is directly related to the
top-FCNC process, in this study, we take the lightest neutral component of
$\Phi_{I}$ as the DM candidate. We derive the relevant couplings for top-quark
FCNC in the following discussions.

Table 1: Representations and charge assignments for new particles. |  $SU(3)_{C}$ |  $SU(2)_{L}$ |  $U(1)_{Y}$ |  $Z_{2}$
---|---|---|---|---
$Q_{4}$ | $3$ | $2$ | $1/3$ | $-1$
$B^{\prime}$ | $3$ | $1$ | $-2/3$ | $-1$
$\Phi_{I}$ | $1$ | $2$ | $1/2$ | $-1$
$N_{Ri}$ | $1$ | $1$ | $0$ | $-1$

###  II.1 Yukawa couplings and scalar potential

Based on the charge assignments shown in Table 1, the new Yukawa interactions
can be written as

$\displaystyle-{\cal L}_{Y}$
$\displaystyle=y_{B^{\prime}}\overline{Q_{4L}}HB^{\prime}_{R}+\tilde{y}_{B^{\prime}}\overline{Q_{4R}}HB^{\prime}_{L}+\overline{Q_{4L}}{\bf
Y}^{d}_{I}\Phi_{I}d_{R}+\overline{Q_{4L}}{\bf
Y}^{u}_{I}\widetilde{\Phi_{I}}u_{R}$ $\displaystyle+\overline{Q_{L}}{\bf
Y}^{B^{\prime}}_{I}\Phi_{I}B^{\prime}_{R}+m_{4Q}\overline{Q_{4L}}Q_{4R}+m_{B^{\prime}}\overline{B^{\prime}_{L}}B^{\prime}_{R}+H.c.\,,$
(1)

where the flavor indices are suppressed;
$\widetilde{\Phi_{I}}=i\tau_{2}\Phi^{*}_{I}$ and $\tau_{2}$ is the Pauli
matrix, and $H$ is the SM Higgs doublet. In this work, we considered that the
new Yukawa couplings are real. Since Eq. (1) does not involve the SM quark
mass diagonalization, the up- and down-type quarks can be taken as the
physical states, and the flavor mixing effects are absorbed into the Yukawa
couplings. The doublet components of $Q_{4}$, $\Phi_{I}$, and $H$ are taken as

$Q_{4}=\left(\begin{array}[]{c}T\\\ B\\\
\end{array}\right)\,,~{}~{}\Phi_{I}=\left(\begin{array}[]{c}H^{+}_{I}\\\
\frac{1}{\sqrt{2}}(H_{I}+iA_{I})\\\
\end{array}\right)\,,~{}~{}H=\left(\begin{array}[]{c}G^{+}\\\
\frac{1}{\sqrt{2}}(v+h+iG^{0})\\\ \end{array}\right)\,.$ (2)

Thus, the first two terms in Eq. (1) lead to the mixture between $B$ and
$B^{\prime}$, and the associated mass matrix is expressed as

$(\bar{B}^{\prime},\bar{B})_{L}\left(\begin{array}[]{cc}m_{B^{\prime}}&\frac{\tilde{y}_{B^{\prime}}v}{\sqrt{2}}\\\
\frac{y_{B^{\prime}}v}{\sqrt{2}}&m_{4Q}\\\
\end{array}\right)\left(\begin{array}[]{c}B^{\prime}\\\ B\\\
\end{array}\right)_{R}\,.$ (3)

In general, we need bi-unitary transformation to diagonalize the mass matrix
in Eq. (3). In order to simplify the analysis, we take
$\tilde{y}_{B^{\prime}}=y_{B^{\prime}}$ and the $2\times 2$ real symmetric
matrix can be diagonalized by an $SO(2)$ transformation, where the eigenvalues
and eigenstates can be obtained as

$\displaystyle m_{B_{1(2)}}$
$\displaystyle=\frac{m_{4Q}+m_{B^{\prime}}}{2}\mp\frac{1}{2}\sqrt{(m_{4Q}-m_{B^{\prime}})^{2}+2y^{2}_{B^{\prime}}v^{2}}\,,$
$\displaystyle\left(\begin{array}[]{c}B_{1}\\\ B_{2}\\\ \end{array}\right)$
$\displaystyle=\left(\begin{array}[]{cc}\cos\theta&-\sin\theta\\\
\sin\theta&\cos\theta\\\
\end{array}\right)\left(\begin{array}[]{c}B^{\prime}\\\ B\\\
\end{array}\right)\,,$ (10) $\displaystyle\tan 2\theta$
$\displaystyle=\frac{\sqrt{2}y_{B^{\prime}}v}{m_{4Q}-m_{B^{\prime}}}\,,$ (11)

where $\theta$ is the mixing angle of $B$ and $B^{\prime}$. We have taken
$B_{1}$ as the lightest $Z_{2}$-odd quark. The resulting Yukawa couplings are
then written as

$\displaystyle{\cal L}_{Y}$
$\displaystyle\supset\bar{B}_{1}\left(-s_{\theta}{\bf
Y}^{u}_{I}P_{R}-c_{\theta}{\bf
Y}^{B^{\prime}}_{I}P_{L}\right)uH^{-}_{I}+\bar{B}_{2}\left(c_{\theta}{\bf
Y}^{u}_{I}P_{R}-s_{\theta}{\bf Y}^{B^{\prime}}_{I}P_{L}\right)uH^{-}_{I}$
$\displaystyle-\frac{1}{\sqrt{2}}\bar{T}{\bf
Y}^{u}_{I}P_{R}u(H_{I}-iA_{I})-\frac{1}{\sqrt{2}}(c_{\theta}\bar{B}_{1}+s_{\theta}\bar{B}_{2}){\bf
Y}^{B^{\prime}}_{I}P_{L}d(H_{I}-iA_{I})$ $\displaystyle-\bar{T}\frac{{\bf
Y}^{d}_{I}}{\sqrt{2}}P_{R}\,d\,H^{+}_{I}-\left(-s_{\theta}\bar{B}_{1}+c_{\theta}\bar{B}_{2}\right)\frac{{\bf
Y}^{d}_{I}}{\sqrt{2}}P_{R}\,d\left(H_{I}+iA_{I}\right)+H.c.\,,$ (12)

with $c_{\theta}(s_{\theta})=\cos\theta(\sin\theta)$. Using the
parametrization, the Higgs couplings to $B_{1}$ and $B_{2}$ can be written as

$\displaystyle{\cal L}_{hBB}$
$\displaystyle=-\frac{1}{\sqrt{2}}h(\bar{B}_{1},\bar{B}_{2}){\bf
y}^{h}\left(\begin{array}[]{c}B_{1}\\\ B_{2}\\\ \end{array}\right)$ (15)
$\displaystyle=-\frac{y_{B^{\prime}}}{\sqrt{2}}h(\bar{B}_{1},\bar{B}_{2})\left(\begin{array}[]{cc}-s_{2\theta}&c_{2\theta}\\\
c_{2\theta}&s_{2\theta}\\\ \end{array}\right)\left(\begin{array}[]{c}B_{1}\\\
B_{2}\\\ \end{array}\right)\,,$ (20)

with $c_{2\theta}(s_{2\theta})=\cos 2\theta(\sin 2\theta)$. The $hB_{i}B_{i}$
coupling, which will lead to the $t\to qh$ decay at the loop level, can be
induced from the mixing terms of $B_{1}$ and $B_{2}$. We note that there is no
$hTT$ coupling at the tree level in the model.

The masses of the inert scalars and their couplings to the Higgs are
determined by the scalar potential, which can be written as Ma:2006km ;
Barbieri:2006dq

$\displaystyle V(H,\Phi_{I})$
$\displaystyle=\mu^{2}_{1}H^{\dagger}H+\mu^{2}_{2}\Phi^{\dagger}_{I}\Phi_{I}+\lambda_{1}(H^{\dagger}H)^{2}+\lambda_{2}(\Phi^{\dagger}_{I}\Phi_{I})^{2}$
$\displaystyle+\lambda_{3}(H^{+}H)(\Phi^{\dagger}_{I}\Phi_{I})+\lambda_{4}(H^{+}\Phi_{I})(\Phi^{\dagger}_{I}H)+\left[\frac{1}{2}\lambda_{5}(H^{\dagger}\Phi_{I})^{2}+H.c.\right]\,.$
(21)

We can obtain the mass squares of $H^{\pm}_{I}$, $H_{I}$, and $A_{I}$ as

$\displaystyle m^{2}_{H_{I}}$
$\displaystyle=\mu^{2}_{2}+\frac{\lambda_{L}v^{2}}{2}\,,~{}m^{2}_{A_{I}}=\mu^{2}_{2}+\frac{\lambda_{A}v^{2}}{2}\,,~{}m^{2}_{H^{\pm}_{I}}=\mu^{2}_{2}+\frac{\lambda_{3}v^{2}}{2}\,,$
(22)

with $\lambda_{L(A)}=\lambda_{3}+\lambda_{4}\pm\lambda_{5}$. The mass
difference between $H_{I}$ and $A_{I}$ depends on the $\lambda_{5}$ parameter.
The Higgs couplings to $(H^{\pm}_{I},H_{I},A_{I})$ can be determined as

$\displaystyle
hH^{+}_{I}H^{-}_{I}:\lambda_{3}\,v\,,~{}~{}hH_{I}H_{I}:\lambda_{L}v\,,~{}~{}hA_{I}A_{I}:\lambda_{A}v.$
(23)

In addition to the minimal conditions, i.e., $\partial V/\partial h(H_{I})=0$,
the vacuum stability is controlled by the copositivity criteria in the
dimension-4 terms of scalar potential, and the stable conditions are yielded
as Barbieri:2006dq ; Klimenko:1984qx ; Kannike:2012pe

$\displaystyle\lambda_{1,2}\geq
0\,,~{}\lambda_{3}+2\sqrt{\lambda_{1}\lambda_{2}}\geq
0\,,~{}\lambda_{3}+\lambda_{4}-|\lambda_{5}|+2\sqrt{\lambda_{1}\lambda_{2}}\geq
0\,.$ (24)

If we take the mass ordering to be $m_{H^{\pm}_{I}}>m_{A_{I}}>m_{H_{I}}$, then
$\lambda_{4,5}<0$, which can be expressed as

$|\lambda_{4}|=\frac{1}{v^{2}}\left(2m^{2}_{H^{\pm}_{I}}-m^{2}_{H_{I}}-m^{2}_{A_{I}}\right)\,,~{}|\lambda_{5}|=\frac{m^{2}_{A_{I}}-m^{2}_{H_{I}}}{v^{2}}\,.$
(25)

To obtain the potential bounded from below, where the conditions in Eq. (24)
are satisfied, we require $\lambda_{3}>0$. Thus, some cancellation occurs in
the $hS_{I}S_{I}$ ($S_{I}=H_{I},A_{I})$ coupling. In addition, in Eq. (12),
$S_{I}Tu$ couplings are only associated with the right-handed up-type SM
quarks. As a result, the loop-induced $t\to q(h,\gamma)$ processes through the
$S_{I}Tu$ couplings are suppressed by $m^{2}_{t}/m^{2}_{T}$, and they are
negligible if $m_{T}\sim O(1)$ TeV. Hence, when the heavy quark and inert
Higgs masses are fixed, the main parameters that affect the rare top decays in
the model are the Yukawa couplings ${\bf Y}^{B^{\prime}}_{I}$ and ${\bf
Y}^{u}_{I}$, the mixing angle $\theta$, and the $\lambda_{3}$ parameter that
is directly related to the $hH^{-}_{I}H^{+}_{I}$ coupling.

###  II.2 Gauge couplings to new fermions and scalars

To study the $t\to qV$ ($V=\gamma,Z$) processes, we also need the gauge
couplings of the photon and $Z$-boson to the inert Higgses and to the
$Z_{2}$-odd quarks, where the gauge couplings from the kinetic terms of
$H_{I}$, $Q_{4}$, and $B^{\prime}$ are written as

$\displaystyle{\cal L}_{\rm
gauge}\supset(D_{\mu}\Phi_{I})^{\dagger}D^{\mu}\Phi_{I}+\overline{Q_{4}}i\not{D}Q_{4}+\overline{B^{\prime}}i\not{D^{\prime}}B^{\prime}\,.$
(26)

The covariant derivatives for the doublet and singlet fields are taken as

$\displaystyle D^{\mu}$
$\displaystyle=\partial^{\mu}+ig\vec{T}\cdot\vec{W}^{\mu}+ig^{\prime}\frac{Y}{2}B_{\mu}\,,$
$\displaystyle D^{\prime\mu}$
$\displaystyle=\partial^{\mu}+ig^{\prime}\frac{Y}{2}B_{\mu}\,,$ (27)

where $W^{i\mu}$ and $B^{\mu}$ are the gauge fields of $SU(2)_{L}$ and
$U(1)_{Y}$, respectively. Using the standard notations for the photon and
$Z$-boson, which are defined as

$\displaystyle A_{\mu}$ $\displaystyle=c_{W}B_{\mu}+s_{W}W^{3}_{\mu}\,,$
$\displaystyle Z_{\mu}$ $\displaystyle=-s_{W}B_{\mu}+c_{W}W^{3}_{\mu}\,.$ (28)

The neutral gauge couplings to the inert Higgses are obtained as

$\displaystyle{\cal L}_{\rm VS_{I}S_{I}}\supset
i\left(eA^{\mu}+\frac{gc_{2W}}{2c_{W}}Z^{\mu}\right)(H^{+}_{I}\partial_{\mu}H^{-}_{I}-H^{-}_{I}\partial_{\mu}H^{+}_{I})+\frac{g}{2c_{W}}Z^{\mu}(A_{I}\partial
H_{I}-H_{I}\partial_{\mu}A_{I})\,,$ (29)

where $c_{W}(s_{W})=\cos\theta_{W}(\sin\theta_{W})$, and $\theta_{W}$ is the
Weinberg’s angle. Since the quark $T$ doesn’t mix with the SM up-type quarks,
its gauge couplings to the photon and the $Z$ boson can be simply written as

$-eQ_{t}\bar{T}\gamma_{\mu}TA^{\mu}-\frac{g}{c_{W}}\left(\frac{1}{2}-Q_{t}s^{2}_{W}\right)\bar{T}\gamma_{\mu}TZ^{\mu}\,.$
(30)

Although the photon couplings to $B_{1}$ and $B_{2}$ are flavor-conserved,
since the $B$ and $B^{\prime}$ quarks belong to different $SU(2)_{L}$
representations and are mixed via the Yukawa couplings, their $Z$ gauge
couplings allow flavor-changing and are expressed as

$\displaystyle{\cal L}_{VBB}$
$\displaystyle=-eQ_{b}\sum_{i}\bar{B}_{i}\gamma_{\mu}B_{i}A^{\mu}-\frac{g}{c_{W}}(\bar{B}_{1},\bar{B}_{2})\gamma_{\mu}{\bf
C}^{Z}\left(\begin{array}[]{c}B_{1}\\\ B_{2}\\\ \end{array}\right)Z^{\mu}\,,$
(33)

with

${\bf
C}^{Z}=\left(\begin{array}[]{cc}-s^{2}_{\theta}/2-Q_{b}s^{2}_{W}&s_{\theta}c_{\theta}\\\
s_{\theta}c_{\theta}&-c^{2}_{\theta}/2-Q_{b}s^{2}_{W}\\\
\end{array}\right)\,.$ (34)

Because the charged gauge boson $W^{\pm}$ only couples to the doublet quarks,
the $W^{\pm}$ coupling can be found as

${\cal
L}_{WTB}=-\frac{g}{\sqrt{2}}\bar{T}\gamma_{\mu}\left(-s_{\theta}B_{1}+c_{\theta}B_{2}\right)W^{+\mu}+H.c.$
(35)

##  III  Loop-induced decay amplitudes for $t\to q(h,V)$

In this section, we derive the decay amplitudes for the $t\to q(h,V)$ decays.
Because the calculations for $t\to qg$ are similar to those for $t\to
q\gamma$, and their BRs can be approximately related by $BR(t\to qg)\sim
C_{F}(\alpha_{s}/\alpha)BR(t\to q\gamma)$ with $C_{F}=4/3$ in the model, where
$\alpha=e^{2}/4\pi$ and $\alpha_{s}=g^{2}_{s}/4\pi$. In this study, we only
focus on the $t\to q\gamma$ analysis.

###  III.1  $t\to q(h,\gamma)$ decays

In the model, the $t\to q(h,\gamma)$ processes can be induced from the loops
with the mediation of $H^{\pm}_{I}$ and $S_{I}$. The Feynman diagrams are
sketched in Fig. 1, where the emitted dashed lines can be the Higgs or the
photon. Since there is no $hTT$ coupling, the $t\to qh$ decay cannot be
produced from Fig. 1(d). In Fig. 1(b), the flavor-changing between $B_{1}$ and
$B_{2}$ only occurs in the Higgs coupling.

Figure 1:  Feynman diagrams used to induce the $t\to q(h,\gamma)$ processes,
where the emitted dashed lines can be the Higgs or photon.

The effective interactions for $t\to q(h,\gamma)$ can be written as

${\cal L}_{t\to
q(h,\gamma)}=-C^{h}_{L}\bar{q}P_{L}t\,h-C^{h}_{R}\bar{q}P_{R}t\,h+\frac{eB^{\gamma}_{L}}{m_{t}}\bar{q}i\sigma_{\mu\nu}\epsilon^{\mu*}_{\gamma}k^{\nu}P_{L}t+\frac{eB^{\gamma}_{R}}{m_{t}}\bar{q}i\sigma_{\mu\nu}\epsilon^{\mu*}_{\gamma}k^{\nu}P_{R}t\,.$
(36)

Following the Feynman diagrams shown in Fig. 1 and the couplings obtained in
Eq. (12) and Eqs. (29-33), the effective Wilson coefficients with $\chi=L,R$
can be written as

$\displaystyle C^{h}_{\chi}$
$\displaystyle=\sum_{i}\left(C^{ha}_{i\chi}+\sum_{j}C^{hb}_{ij\chi}\right)+C^{hc}_{\chi}\,,$
$\displaystyle B^{\gamma}_{\chi}$ $\displaystyle=\sum_{i}\left(B^{\gamma
a}_{i\chi}+B^{\gamma b}_{i\chi}\right)+B^{\gamma d}_{\chi}\,.$ (37)

The contributions to $t\to qh$ from each diagram can be formulated as

$\displaystyle C^{ha}_{iL}$
$\displaystyle=\frac{\lambda_{3}v}{(4\pi)^{2}m_{t}}C^{q}_{iR}\int^{1}_{0}dx_{1}\int^{x_{1}}_{0}dx_{2}\frac{C^{t}_{iR}r_{it}x_{2}+\sqrt{r_{it}}C^{t}_{iL}}{D^{a}_{h}(r_{iH^{\pm}_{I}},r_{it},r_{ih})}\,,$
$\displaystyle C^{ha}_{iR}$
$\displaystyle=\frac{\lambda_{3}v}{(4\pi)^{2}m_{t}}C^{q}_{iL}\int^{1}_{0}dx_{1}\int^{x_{1}}_{0}dx_{2}\frac{C^{t}_{iL}r_{it}x_{2}+C^{t}_{iR}\sqrt{r_{it}}}{D^{a}_{h}(r_{iH^{\pm}_{I}},r_{it},r_{ih})}\,,$
$\displaystyle C^{hb}_{ijL}$
$\displaystyle=-\frac{y^{h}_{ij}}{(4\pi)^{2}}\int^{1}_{0}dx_{1}\int^{x_{1}}_{0}dx_{2}\frac{C^{q}_{jR}}{D^{b}_{h}(r_{jH^{\pm}_{I}},r_{jt},r_{ji},r_{jh})}\left[C^{t}_{iR}(r_{jt}x_{2}(1-x_{1})\right.$
$\displaystyle\left.+r_{jh}(x_{1}-x_{2})-r_{ji})+C^{t}_{iL}\sqrt{r_{jt}}(\sqrt{r_{jt}}x_{2}-(1-x_{2}))\right]\,,$
$\displaystyle C^{hb}_{ijR}$
$\displaystyle=-\frac{y^{h}_{ij}}{(4\pi)^{2}}\int^{1}_{0}dx_{1}\int^{x_{1}}_{0}dx_{2}\frac{C^{q}_{jL}}{D^{b}_{h}(r_{jH^{\pm}_{I}},r_{jt},r_{ji},r_{jh})}\left[C^{t}_{iL}(r_{jt}x_{2}(1-x_{1})\right.$
$\displaystyle\left.+r_{jh}(x_{1}-x_{2})-r_{ji})+C^{t}_{iR}\sqrt{r_{jt}}(\sqrt{r_{jt}}x_{2}-(1-x_{2}))\right]\,,$
$\displaystyle C^{hc}_{L}$
$\displaystyle=\frac{Y^{u}_{Iq}Y^{u}_{I3}y_{t}}{(4\pi)^{2}}\frac{v}{m_{t}}\int^{1}_{0}dx_{1}\int^{x_{1}}_{0}dx_{2}\,x_{2}\left[\frac{\lambda_{3}+\lambda_{4}+2\lambda_{5}}{D^{a}_{h}(y_{H_{I}},y_{t},y_{h})}+\frac{\lambda_{3}+\lambda_{4}+-2\lambda_{5}}{D^{a}_{h}(y_{A_{I}},y_{t},y_{h})}\right]\,,$
(38)

where $r_{if}=m^{2}_{f}/m^{2}_{B_{i}}$, $y_{f}=m^{2}_{f}/m^{2}_{T}$; the
Yukawa couplings $C^{f}_{i\chi}$ are

$\displaystyle C^{f}_{1R}$
$\displaystyle=-s_{\theta}Y^{u}_{If}\,,~{}C^{f}_{1L}=-c_{\theta}Y^{B^{\prime}}_{If}\,,$
$\displaystyle C^{f}_{2R}$
$\displaystyle=c_{\theta}Y^{u}_{If}\,,~{}C^{f}_{2L}=-s_{\theta}Y^{B^{\prime}}_{If}\,,$
(39)

and the denominators are defined by

$\displaystyle D^{a}_{h}(x,y,z)$
$\displaystyle=1-(1-x)x_{1}-y(1-x_{1})x_{2}-z(x_{1}-x_{2})x_{2}\,,$
$\displaystyle D^{b}_{h}(w,x,y,z)$
$\displaystyle=x_{1}+w(1-x_{1})-(x-y+1)x_{2}+zx^{2}_{2}+(x-z)x_{1}x_{2}\,.$
(40)

The results for the $t\to q\gamma$ decay from each Feynman diagram are
obtained as

$\displaystyle B^{\gamma a}_{iL}$
$\displaystyle=-\frac{1}{(4\pi)^{2}}\int^{1}_{0}dx_{1}\int^{x_{1}}_{0}dx_{2}\frac{1-x_{1}}{D^{a}_{\gamma}(r_{iH^{\pm}_{I}},r_{it})}C^{q}_{iR}\left[C^{t}_{iR}r_{it}x_{1}+C^{t}_{iL}\sqrt{r_{it}}\right]\,,$
$\displaystyle B^{\gamma a}_{iR}$
$\displaystyle=-\frac{1}{(4\pi)^{2}}\int^{1}_{0}dx_{1}\int^{x_{1}}_{0}dx_{2}\frac{1-x_{1}}{D^{a}_{\gamma}(r_{iH^{\pm}_{I}},r_{it})}C^{q}_{iL}\left[C^{t}_{iL}r_{it}x_{1}+C^{t}_{iR}\sqrt{r_{it}}\right]\,,$
$\displaystyle B^{\gamma b}_{iL}$
$\displaystyle=\frac{Q_{b}}{(4\pi)^{2}}\int^{1}_{0}dx_{1}\int^{x_{1}}_{0}dx_{2}\frac{1}{D^{b}_{\gamma}(r_{iH^{\pm}_{I}},r_{it})}C^{q}_{iR}\left[C^{t}_{iR}r_{it}(1-x_{1})x_{2}+C^{t}_{iL}\sqrt{r_{it}}x_{1}\right]\,,$
$\displaystyle B^{\gamma b}_{iR}$
$\displaystyle=\frac{Q_{b}}{(4\pi)^{2}}\int^{1}_{0}dx_{1}\int^{x_{1}}_{0}dx_{2}\frac{1}{D^{b}_{\gamma}(r_{iH^{\pm}_{I}},r_{it})}C^{q}_{iL}\left[C^{t}_{iL}r_{it}(1-x_{1})x_{2}+C^{t}_{iR}\sqrt{r_{it}}x_{1}\right]\,,$
$\displaystyle B^{\gamma d}_{L}$
$\displaystyle=\frac{Q_{t}Y^{u}_{Iq}Y^{u}_{I3}}{2(4\pi)^{2}}y_{t}\int^{1}_{0}dx_{1}\int^{x_{1}}_{0}dx_{2}\,x_{2}(1-x_{1})\left[\frac{1}{D^{b}_{\gamma}(y_{H_{I}},y_{t})}+\frac{1}{D^{b}_{\gamma}(y_{A_{I}},y_{t})}\right]\,,$
(41)

with

$\displaystyle D^{a}_{\gamma}(x,y)$
$\displaystyle=1-(1-x)x_{1}-y(1-x_{1})x_{2}\,,$ $\displaystyle
D^{b}_{\gamma}(x,y)$ $\displaystyle=x_{1}+x(1-x_{1})-y(1-x_{1})x_{2}\,.$ (42)

It can be seen that $C^{hc}_{L}$ and $B^{\gamma d}_{L}$, which arise from Fig.
1(c) and (d), are much smaller than other contributions because of the
suppression factor $y_{t}=m^{2}_{t}/m^{2}_{T}$. To illustrate the smallness of
$C^{hc}_{L}$ and $B^{\gamma d}_{L}$, we take $m_{H_{I}}=65$ GeV,
$m_{A_{I},H^{\pm}_{I}}=100$ GeV, $m_{B_{1},T(B)}=(1,1.5)$ TeV,
$c_{\theta}=0.78$, and $Y^{u(B^{\prime})}_{Iq,I3}=2$; as a result,
$C^{hb}_{12R}\approx 3.2\times 10^{-3}$, $C^{hc}_{L}\approx-9.6\times
10^{-6}$, $B^{\gamma a}_{1L}\approx-1.1\times 10^{-3}$, and $B^{\gamma
d}_{L}\approx 2.1\times 10^{-5}$. Hence, the contributions from $C^{hc}_{L}$
and $B^{\gamma d}_{L}$ can be neglected.

Using the obtained effective Wilson coefficients, the BRs for $t\to
q(h,\gamma)$ can be estimated by the following relations

$\displaystyle Br(t\to qh)$
$\displaystyle=\frac{m_{t}}{32\pi\Gamma_{t}}\left(1-\frac{m^{2}_{h}}{m^{2}_{t}}\right)^{2}\left(|C^{h}_{L}|^{2}+|C^{h}_{R}|^{2}\right)\,,$
$\displaystyle Br(t\to q\gamma)$ $\displaystyle=\frac{\alpha
m_{t}}{4\Gamma_{t}}\left(|B^{\gamma}_{L}|^{2}+|B^{\gamma}_{R}|^{2}\right)\,,$
(43)

where $\Gamma_{t}$ is the top-quark width. Since the top-quark decay is
dominated by the $t\to Wb$ process, for the numerical estimation, we take the
next-to-leading order SM result for $\Gamma_{t}$, which is given as
Jezabek:1988iv ; PDG

$\Gamma_{t}=\frac{G_{F}m^{3}_{t}}{8\pi\sqrt{2}}\left(1-\frac{m^{2}_{W}}{m^{2}_{t}}\right)^{2}\left(1+2\frac{m^{2}_{W}}{m^{2}_{t}}\right)\left[1-\frac{2\alpha_{s}}{3\pi}\left(\frac{2\pi^{2}}{3}-\frac{5}{2}\right)\right]\,.$
(44)

###  III.2  $t\to qZ$ decay

The $t\to qZ$ decay can arise from Fig. 1(a), (b), and (d) using $Z$ instead
of $h/\gamma$. Moreover, the $t\to qZ$ decay can be produced via the
$ZH_{I}A_{I}$ coupling, where the associated Feynman diagram is shown in Fig.
2.

Figure 2:  Feynman diagram mediated by $H_{I}$ and $A_{I}$ for the $t\to qZ$
process.

The decay amplitudes for the $t\to qZ$ decay can be formulated as

$\displaystyle{\cal L}_{t\to qZ}$
$\displaystyle=\bar{u}_{q}\gamma_{\mu}\left(A^{Z}_{L}P_{L}+A^{Z}_{R}P_{R}\right)tZ^{\mu}+\frac{1}{m_{t}}\bar{u}_{q}i\sigma_{\mu\nu}k^{\nu}\left(B^{Z}_{L}P_{L}+B^{Z}_{R}P_{R}\right)tZ^{\mu}\,.$
(45)

Because of the massive $Z$ boson, the $t\to qZ$ decay involves the vector
currents in addition to the tensor currents. Following the Feynman diagrams
and using the couplings given in Eq. (12) and Eqs. (29-33), the effective
Wilson coefficients can be summarized as

$\displaystyle A^{Z}_{\chi}$
$\displaystyle=\sum_{i}\left(A^{Za}_{i\chi}+\sum_{j}A^{Zb}_{ij\chi}\right)+A^{Zd}_{\chi}+A^{Ze}_{\chi}\,,$
$\displaystyle B^{Z}_{\chi}$
$\displaystyle=\sum_{i}\left(B^{Za}_{i\chi}+\sum_{j}B^{Zb}_{ij\chi}\right)+B^{Zd}_{\chi}+B^{Ze}_{\chi}\,.$
(46)

The contributions from each Feynman diagram are shown as

$\displaystyle A^{Za}_{iL}$
$\displaystyle=-\frac{g(c^{2}_{W}-s^{2}_{W})}{2c_{W}(4\pi)^{2}}\int^{1}_{0}dx_{1}\int^{x_{1}}_{0}dx_{2}\frac{1-x_{1}}{D^{a}_{h}(r_{iH^{\pm}_{I}},r_{it},r_{iZ})}C^{q}_{iL}\left(C^{t}_{iL}r_{it}x_{2}+C^{t}_{iR}\sqrt{r_{it}}\right)\,,$
$\displaystyle B^{Za}_{iL}$
$\displaystyle=-\frac{g(c^{2}_{W}-s^{2}_{W})}{2c_{W}(4\pi)^{2}}\int^{1}_{0}dx_{1}\int^{x_{1}}_{0}dx_{2}\frac{1-x_{1}}{D^{a}_{h}(r_{iH^{\pm}_{I}},r_{it},r_{iZ})}C^{q}_{iR}\left(C^{t}_{iR}r_{it}x_{2}+C^{t}_{iL}\sqrt{r_{it}}\right)\,,$
$\displaystyle X^{Zb}_{ij\chi}$
$\displaystyle=-\frac{gC^{Z}_{ij}}{c_{W}(4\pi)^{2}}\int^{1}_{0}dx_{1}\int^{x_{1}}_{0}dx_{2}\frac{\tilde{X}^{Z}_{ij\chi}}{D^{b}_{h}(r_{jH^{\pm}_{I}},r_{jt},r_{ji},r_{jZ})}\,,$
$\displaystyle A^{Zd}_{R}$
$\displaystyle=-\frac{g(1-2Q_{t}s^{2}_{W})}{4c_{W}(4\pi)^{2}}Y^{u}_{Iq}Y^{u}_{I3}\int^{1}_{0}dx_{1}\int^{x_{1}}_{0}dx_{2}\left(\frac{1+y_{Z}x_{2}(x_{1}-x_{2})}{D^{b}_{h}(y_{H_{I}},y_{t},1,y_{Z})}+\frac{1+y_{Z}x_{2}(x_{1}-x_{2})}{D^{b}_{h}(y_{A_{I}},y_{t},1,y_{Z})}\right)\,,$
$\displaystyle B^{Zd}_{L}$
$\displaystyle=\frac{g(1-2Q_{t}s^{2}_{W})}{4c_{W}(4\pi)^{2}}y_{t}Y^{u}_{Iq}Y^{u}_{I3}\int^{1}_{0}dx_{1}\int^{x_{1}}_{0}dx_{2}\left(\frac{x_{2}(1-x_{1})}{D^{b}_{h}(y_{H_{I}},y_{t},1,y_{Z})}+\frac{x_{2}(1-x_{1})}{D^{b}_{h}(y_{A_{I}},y_{t},1,y_{Z})}\right)\,,$
$\displaystyle A^{Ze}_{R}$
$\displaystyle=\frac{gy_{t}}{4c_{W}(4\pi)^{2}}Y^{u}_{Iq}Y^{u}_{I3}\int^{1}_{0}dx_{1}\int^{x_{1}}_{0}dx_{2}\left(\frac{x_{2}(1-x_{1})}{D^{e}_{Z}(y_{H_{I}},y_{A_{I}},y_{t},y_{Z})}+\frac{x_{2}(1-x_{1})}{D^{e}_{Z}(y_{A_{I}},y_{H_{I}},y_{t},y_{Z})}\right)\,,$
(47)

$A^{Za}_{iR}=B^{Za}_{iL}$, $B^{Za}_{iR}=A^{Za}_{iL}$,
$A^{Zd}_{L}=B^{Zd}_{R}=0$, and $B^{Ze}_{L}=A^{Ze}_{R}$ with

$\displaystyle\tilde{A}^{Z}_{ijL}$
$\displaystyle=C^{q}_{jL}\left[C^{t}_{iL}\left(\sqrt{r_{ji}}+r_{jZ}x_{2}(x_{1}-x_{2})\right)+C^{t}_{iR}\sqrt{r_{jt}}(1-x_{1})\right]\,,$
$\displaystyle\tilde{A}^{Z}_{ijR}$
$\displaystyle=C^{q}_{jR}\left[C^{t}_{iR}\left(\sqrt{r_{ji}}+r_{jZ}x_{2}(x_{1}-x_{2})\right)+C^{t}_{iL}\sqrt{r_{jt}}(1-x_{1})\right]\,,$
$\displaystyle\tilde{B}^{Z}_{ijL}$
$\displaystyle=-C^{q}_{jR}\left[C^{t}_{iR}r_{jt}x_{2}(1-x_{1})+C^{t}_{iL}\left(\sqrt{r_{jt}r_{ji}}x_{2}+\sqrt{r_{jt}}(x_{1}-x_{2})\right)\right]\,,$
$\displaystyle\tilde{B}^{Z}_{ijR}$
$\displaystyle=-C^{q}_{jL}\left[C^{t}_{iL}r_{jt}x_{2}(1-x_{1})+C^{t}_{iR}\left(\sqrt{r_{jt}r_{ji}}x_{2}+\sqrt{r_{jt}}(x_{1}-x_{2})\right)\right]\,,$
$\displaystyle D^{e}_{Z}(w,x,y,z)$
$\displaystyle=1+(1-x)x_{1}-(y-w-x)x_{2}+zx^{2}_{2}+yx_{1}x_{2}\,,$ (48)

where $X^{Zb}_{ij\chi}$ and $\tilde{X}^{Z}_{ij\chi}$, which involve the $Z$
flavor-changing couplings, are the contributions from Fig. 1(b). From Eq.
(47), it can be seen that $A^{Ze}_{R}$ from Fig. 2 is suppressed by
$m^{2}_{t}/m^{2}_{T}$. According to earlier analysis, the contribution is
small and can be neglected. Moreover, the $Z$-penguin induced from Fig. 1(d)
is comparable with that induced from Fig. 1(b). Using the obtained
$A^{Z}_{\chi}$ and $B^{Z}_{\chi}$, the BR for the $t\to qZ$ decay can be
written as

$\displaystyle Br(t\to qZ)$
$\displaystyle=\frac{1}{\Gamma_{t}}\frac{G_{F}m_{t}^{3}}{4\sqrt{2}\pi}\frac{c^{2}_{W}}{g^{2}}\left(1-\frac{m^{2}_{Z}}{m^{2}_{t}}\right)^{2}\sum_{\chi=L,R}\left[|A^{Z}_{\chi}+B^{Z}_{\chi^{\prime}}|^{2}\left(1+\frac{2m^{2}_{Z}}{m^{2}_{t}}\right)\right.$
$\displaystyle\left.+\left(|B^{Z}_{\chi^{\prime}}|^{2}-2Re(A^{Z}_{\chi}+B^{Z}_{\chi^{\prime}})B^{Z*}_{\chi^{\prime}})\right)\left(1-\frac{m^{2}_{Z}}{m^{2}_{t}}\right)\right]\,,$
(49)

where $B^{Z}_{\chi^{\prime}}$ denotes that when $A^{Z}_{\chi}=A^{Z}_{L(R)}$,
$B^{Z}_{\chi^{\prime}}=B^{Z}_{R(L)}$.

##  IV Constraints

In this section, we discuss the possible constraints from the flavor physics,
DM data, and oblique parameters.

###  IV.1 $b\to q^{\prime}\gamma$ and $\Delta B=2$

The similar Feynman diagrams for $t\to q\gamma$ can be applied to the $B\to
X_{q^{\prime}}\gamma$ ($q^{\prime}=d,s$) decays, where the current
measurements are $BR(B\to X_{s}\gamma)=(3.49\pm 0.19)\times 10^{-4}$ and
$BR(B\to X_{d}\gamma)=(9.2\pm 3.0)\times 10^{-6}$ PDG . If there are large
effects contributing to the radiative $B$ decays, then the current data may
provide a serious bound. From Eq. (12), it can be seen that the involving
Yukawa couplings for the $b\to q^{\prime}\gamma$ decay are ${\bf
Y}^{B^{\prime}}_{I}$ and ${\bf Y}^{d}_{I}$. To understand the influence from
${\bf Y}^{B^{\prime}}_{I}$ and ${\bf Y}^{d}_{I}$, we write the effective
interaction for $b\to q^{\prime}\gamma$ from the new physics as

$\displaystyle{\cal L}_{b\to q^{\prime}\gamma}$
$\displaystyle=\frac{G_{F}V_{tb}V^{*}_{tq^{\prime}}}{\sqrt{2}}\left(C^{\rm
NP}_{7R}{\cal O}_{7R}+C^{\rm NP}_{7L}{\cal O}_{7L}\right)\,,$
$\displaystyle{\cal O}_{7\chi}$
$\displaystyle=\frac{em_{b}}{4\pi^{2}}\overline{d_{q^{\prime}}}\sigma_{\mu\nu}P_{\chi}bF^{\mu\nu}\,.$
(50)

The $C^{\rm NP}_{7R,7L}$ mediated by $S_{I}$ and $H^{\pm}_{I}$ can be
expressed as

$\displaystyle C^{\rm NP,q^{\prime}}_{7R}$
$\displaystyle=-\frac{\sqrt{2}Y^{B^{\prime}}_{Iq^{\prime}}Y^{B^{\prime}}_{I3}}{G_{F}V_{tb}V^{*}_{tq^{\prime}}}\sum^{2}_{i=1}\frac{Q_{b}\zeta^{2}_{i}}{16m^{2}_{B_{i}}}\left(J(r_{iH_{I}})+J(r_{iA_{I}})\right)\,,$
$\displaystyle C^{\rm NP,q^{\prime}}_{7L}$
$\displaystyle=-\frac{\sqrt{2}Y^{d}_{Iq^{\prime}}Y^{d}_{I3}}{G_{F}V_{tb}V^{*}_{tq^{\prime}}}\left[\frac{1}{16m^{2}_{T}}\left(J^{\prime}(y_{H^{\pm}_{I}})Q_{t}J(y_{H^{\pm}_{I}})\right)\right.$
$\displaystyle\left.-\sum^{2}_{i=1}\frac{Q_{b}\xi^{2}_{i}}{16m^{2}_{B_{i}}}\left(J(r_{iH_{I}})+J(r_{iA_{I}})\right)\right]\,,$
$\displaystyle J(a)$
$\displaystyle=\frac{1-5a-2a^{2}}{6(1-a)^{3}}-\frac{a^{2}\ln
a}{2(1-a)^{4}}\,,$ $\displaystyle J^{\prime}(a)$
$\displaystyle=\frac{2+5a-a^{2}}{12(1-a)^{3}}+\frac{a\ln a}{2(1-a)^{4}}\,,$
(51)

with $\zeta_{1}=c_{\theta}$, $\zeta_{2}=s_{\theta}$, $\xi_{1}=-s_{\theta}$,
and $\xi_{2}=c_{\theta}$. Using $|V_{ts}|=0.04$, $|V_{td}|=0.0088$,
$s_{\theta}=1/\sqrt{2}$, ${\bf Y}^{B^{\prime}}_{I}={\bf Y}^{d}_{I}=2$, and
$m_{B_{1},B_{2},T}=(1,1.5,1.4)$ TeV, we obtain $C^{\rm
NP,s}_{7R,7L}\sim(-0.051,\,0.033)$ and $C^{\rm
NP,d}_{7R,7L}\sim(0.23,\,-0.15)$, where the SM result is $C^{\rm
SM}_{7R,7L}\sim(-0.3,\,0)$ Buchalla:1995vs . It can be seen that due to
$|V_{td}|/|V_{ts}|\sim 0.22$, the values of $C^{\rm NP,d}_{7\chi}$ are much
larger than those of $C^{\rm NP,s}_{7\chi}$. Although the value of $C^{\rm
NP,d}_{7R}$ is close to that of $C^{\rm SM}_{7R}$, the contribution from
${\cal O}_{7R}$ operator can be diminished due to the opposite sign in $C^{\rm
NP,d}_{7R}$ and $C^{\rm SM}_{7R}$; thus, the dominant contribution to $B\to
X_{d}\gamma$ is from ${\cal O}_{7L}$. Since $|C^{\rm NP,d}_{7L}|$ is less than
$|C^{\rm SM}_{7R}|$, ${\bf Y}^{B^{\prime}(d)}_{I}\sim 2$ are still allowed
when the constraint from the $B\to X_{d}\gamma$ process is taken into account.
Hence, $B\to X_{q^{\prime}}\gamma$ do not provide severe constraints on the
parameters $Y^{B^{\prime}}_{Iq^{\prime}}$, which are related to the $t\to
q(h,V)$ decays.

In addition to the radiative $b$ decays, the Yukawa couplings ${\bf
Y}^{B^{\prime}}_{I}$ and ${\bf Y}^{d}_{I}$ can also contribute to the $\Delta
B=2$ process through box diagrams mediated by $H_{I}$, $A_{I}$, and
$H^{\pm}_{I}$. Since the Yukawa couplings to $S_{I}$ involves left-handed and
right-handed couplings, for simplicity, we write the Yukawa couplings as

$\displaystyle{\cal
L}_{Y}\supset-\eta_{S_{I}}\bar{B}_{i}(C^{i}_{Lf}P_{L}+C^{i}_{Rf}P_{R})fS_{I}+H.c.\,,$
(52)

where $\eta_{H_{I}}=1$, $\eta_{A_{I}}=-\sqrt{-1}$,
$C^{i}_{Lf}=\zeta_{i}Y^{B^{\prime}}_{If}/\sqrt{2}$, and
$C^{i}_{Lf}=-\xi_{i}Y^{d}_{If}/\sqrt{2}$. In order to simplify the expression
and show the possible destruction between ${\bf Y}^{B^{\prime}}_{I}$ and ${\bf
Y}^{d}_{I}$, we take $m_{F}\equiv m_{T}\sim m_{B_{1}}\sim m_{B_{2}}$ and
neglect the small ratios $m^{2}_{S_{I},H^{\pm}_{I}}/m^{2}_{B_{1}}$; as a
result, the effective Lagrangian for $\Delta B=2$ can be written as

$\displaystyle{\cal L}_{\Delta B=2}$
$\displaystyle\sim-\frac{1}{2(4\pi)^{2}}\sum^{2}_{i,j=1}\frac{1}{m^{2}_{B_{i}}}\left[C^{i}_{Lq^{\prime}}C^{i}_{L3}C^{j}_{Lq^{\prime}}C^{j}_{L3}(\bar{q}^{\prime}\gamma_{\mu}P_{L}b)^{2}+C^{i}_{Rq^{\prime}}C^{i}_{R3}C^{j}_{Rq^{\prime}}C^{j}_{R3}(\bar{q}^{\prime}\gamma_{\mu}P_{R}b)^{2}\right.$
$\displaystyle\left.+\left(C^{i}_{Lq^{\prime}}C^{i}_{L3}C^{j}_{Rq^{\prime}}C^{j}_{R3}+R\leftrightarrow
L\right)(\bar{q}^{\prime}\gamma_{\mu}P_{L}b)(\bar{q}^{\prime}\gamma^{\mu}P_{R}b)\right]-\frac{Y^{d}_{Iq^{\prime}}Y^{d}_{I3}}{8(4\pi)^{2}m^{2}_{T}}(\bar{q}^{\prime}\gamma_{\mu}P_{R}b)^{2}\,.$
(53)

Using $\sum_{i}\zeta^{2}_{i}=\sum_{i}\xi^{2}_{i}=1$ and the hadronic matrix
elements of $B_{q^{\prime}}$ for various effective operators that were
obtained in Ref. Buras:2001ra , the transition matrix element for
$B_{q^{\prime}}$ mixing is obtained as

$\displaystyle\langle B_{q^{\prime}}|H_{\rm eff}|\bar{B}_{q^{\prime}}\rangle$
$\displaystyle\sim\frac{f^{2}_{B_{q^{\prime}}}m_{B_{q^{\prime}}}}{6(4\pi)^{2}m^{2}_{F}}\left[\left((Y^{B^{\prime}}_{Iq^{\prime}}Y^{B^{\prime}}_{I3})^{2}+\frac{5}{4}(Y^{d}_{Iq^{\prime}}Y^{d}_{I3})^{2}\right)P^{\rm
VLL}_{1}\right.$
$\displaystyle\left.+2Y^{B^{\prime}}_{Iq^{\prime}}Y^{B^{\prime}}_{I3}Y^{d}_{Iq^{\prime}}Y^{d}_{I3}\,P^{LR}_{1}\right]\,,$
(54)

where $P^{VLL}_{1}$ and $P^{LR}_{1}$ are the nonperturbative hadronic effects
of $(\bar{q}^{\prime}\gamma_{\mu}P_{R(L)}b)^{2}$ and
$(\bar{q}^{\prime}\gamma_{\mu}P_{L}b)(\bar{q}^{\prime}\gamma^{\mu}P_{R}b)$,
respectively, and their values are $P^{VLL}_{1}\approx 0.84$ and
$P^{LR}_{1}\approx-1.62$ Buras:2001ra . It can be seen that because
$P^{VLL}_{1}$ and $P^{LR}_{1}$ are opposite in sign, the contributions from
${\bf Y}^{B^{\prime}}_{I}$ to $\Delta B=2$ can be diminished by those from
${\bf Y}^{d}_{I}$. To numerically exhibit the cancellation, we take
$\hat{w}=Y^{d}_{Iq^{\prime}}Y^{d}_{I3}$ as a variable, fix
$Y^{B^{\prime}}_{Iq^{\prime}}=Y^{B^{\prime}}_{I3}=2$, and turn Eq. (54) to be
a quadratic equation as

$\frac{5}{4}P^{VLL}_{1}\hat{w}^{2}+8P^{RL}_{1}\hat{w}+16P^{VLL}_{1}=0\,.$ (55)

By solving the quadratic equation, the values of
$Y^{d}_{Iq^{\prime}}Y^{d}_{I3}$, which lead to small $B_{q^{\prime}}$ mixing,
can then be found. Indeed, two solutions to Eq. (55) exist and are obtained
as: $\hat{w}\approx 1.143$ and $\hat{w}\approx 11.2$. Based on the analysis,
it is known that the strict constraint from the $\Delta B=2$ process can be
avoided when the left-handed and right-handed current couplings are
simultaneously considered.

###  IV.2  Oblique parameters

Since the $Z_{2}$-odd quark $B$ in doublet $Q_{4}$ mixes with the singlet
$B^{\prime}$, the mixing effect leads to the mass difference between $T$ and
$B$. In Eq. (11), it can be seen that the mass splitting within the vector-
like quark doublet can be expressed as $\delta m_{Q_{4}}=|m_{T}-m_{B_{2}}|$
and is dictated by $vy_{B^{\prime}}/\sqrt{2}$. This mass splitting contributes
to the electroweak oblique parameters, where the current measurements with
$U=0$ are given as PDG

$S=0.02\pm 0.07\,,~{}T=0.06\pm 0.06\,.$ (56)

Therefore, the precision measurements of electroweak oblique parameters
Peskin:1991sw  may constrain $y_{B^{\prime}}$ or the mixing angle $\theta$.
Take the constraints into account, following the results in Lavoura:1992np ;
Chen:2017hak , we write the oblique correction to the $T$ parameter as

$\displaystyle\Delta T_{Q_{4}}$ $\displaystyle=\frac{N_{c}}{8\pi
s^{2}_{W}c^{2}_{W}}\left[s^{2}_{\theta}\Theta_{+-}(z_{T},z_{B_{1}})+c^{2}_{\theta}\Theta_{+-}(z_{T},z_{B_{2}})-s^{2}_{\theta}c^{2}_{\theta}\Theta_{+-}(z_{B_{1}},z_{B_{2}})\right]\,,$
(57)

where $N_{c}=3$ is the color number, $z_{f}=m^{2}_{f}/m^{2}_{Z}$, and
$\Theta_{+-}(a,b)=\theta_{+}(a,b)+\theta_{-}(a,b)$, with

$\displaystyle\theta_{+}(a,b)$
$\displaystyle=a+b-\frac{2ab}{a-b}\ln\left(\frac{a}{b}\right)\,,$
$\displaystyle\theta_{-}(a,b)$
$\displaystyle=2\sqrt{ab}\left(\frac{a+b}{a-b}\ln\left(\frac{a}{b}\right)-2\right)\,.$
(58)

When $y_{B^{\prime}}=0$, it can be seen that $s_{\theta}=0$ and
$m_{T}=m_{B_{2}}=m_{4Q}$. Due to $\theta_{\pm}(a,a)=0$, we obtain $\Delta
T=0$.

The correction to the $S$ parameter can be expressed as

$\displaystyle\Delta S_{Q_{4}}$
$\displaystyle=\frac{N_{c}}{\pi}\left[s^{2}_{\theta}\Psi_{+-}(z_{T},z_{B_{1}})+c^{2}_{\theta}\Psi_{+-}(z_{T},z_{B_{2}})-s^{2}_{\theta}c^{2}_{\theta}\chi_{+-}(z_{B_{1}},z_{B_{2}})\right]\,,$
(59)

where $\Psi_{+-}(a,b)=\Psi_{+}(a,b)+\Psi_{-}(a,b)$ and
$\chi_{+-}(a,b)=\chi_{+}(a,b)+\chi_{-}(a,b)$, with

$\displaystyle\Psi(a,b)$
$\displaystyle=\frac{1}{3}-\frac{1}{9}\ln\left(\frac{a}{b}\right)\,,~{}\Psi_{-}(a,b)=-\frac{a+b}{6\sqrt{ab}}\,,$
$\displaystyle\chi_{+}(a,b)$
$\displaystyle=\frac{5(a^{2}+b^{2})-22ab}{9(a-b)^{2}}+\frac{3ab(a+b)-a^{3}-b^{3}}{3(a-b)^{3}}\ln\left(\frac{a}{b}\right)\,,$
$\displaystyle\chi_{-}(a,b)$
$\displaystyle=-\sqrt{ab}\left[\frac{a+b}{6ab}-\frac{a+b}{(a-b)^{2}}+\frac{2ab}{(a-b)^{3}}\ln\left(\frac{a}{b}\right)\right]\,.$
(60)

Similar to $\Delta T_{Q_{4}}$, when $y_{B^{\prime}}=0$, due to
$\Psi_{+-}(a,a)=0$ and $\chi_{\pm}(a,a)=0$, we obtain $\Delta S_{Q_{4}}=0$.
With $c_{\theta}=0.8$, $\Delta S_{Q_{4}}\sim 0.01$, which is much smaller than
$\Delta T_{Q_{4}}$. Thus, we only take the $T$ parameter as the potential
constraint.

The mass splittings among $H^{\pm}_{I}$, $H_{I}$, and $A_{I}$ also contribute
to the $T$ parameter. Following the results shown in Barbieri:2006dq , the
correction of the inert Higgs doublet to the $T$ parameter is expressed as

$\displaystyle\Delta T_{\Phi_{I}}=\frac{1}{16\pi
s^{2}_{W}c^{2}_{W}}\left(\theta_{+}(z_{H^{\pm}_{I}},z_{H_{I}})+\theta_{+}(z_{H^{\pm}_{I}},z_{A_{I}})-\theta_{+}(z_{H_{I}},z_{A_{I}}))\right)\,.$
(61)

###  IV.3 Higgs production and $h\to\gamma\gamma$

From Eqs. (20) and (23), the SM Higgs has extra couplings to $H^{\pm}_{I}$ and
$B_{i}$, where the former can induce the $h\gamma\gamma$ effective coupling,
and the latter can generate $h\gamma\gamma$ and $hgg$ effective couplings.
With the precision measurements for the Higgs production and Higgs decay to
diphoton, the new physics effect could be strictly bounded. To show the new
physics effect, the signal strength for $pp\to h\to\gamma\gamma$ is defined as

$\displaystyle\mu_{\gamma\gamma}=\frac{\sigma(pp\to h)}{\sigma(pp\to h)^{\rm
SM}}\frac{BR(h\to\gamma\gamma)}{BR(h\to\gamma\gamma)^{\rm SM}}\,,$ (62)

where the measurements from ATLAS and CMS at $\sqrt{s}=13$ TeV are given as
$1.10\pm 0.10$ using $139$ fb-1 ATLAS:2020pvn  and $1.03^{+0.11}_{-0.09}$
using $137$ fb-1 CMS:2020omd , respectively.

Figure 3:  Feynman diagram for $h\to(\gamma\gamma,gg)$ induced by
$H^{\pm}_{I}$ and $B_{i}$.

The loop-induced effective interactions for $h\gamma\gamma$ and $hgg$ can be
parameterized as

$\displaystyle{\cal L}_{hVV}$
$\displaystyle=\frac{\alpha}{4\pi}\frac{a_{\gamma\gamma}}{m_{h}}hF_{\mu\nu}F^{\mu\nu}+\frac{\alpha_{s}}{4\pi}\frac{a_{gg}}{m_{h}}hG^{a}_{\mu\nu}G^{a\mu\nu}\,,$
(63)

where the Feynman diagrams mediated by $H^{\pm}_{I}$ and $B_{i}$ are shown in
Fig. 3. The resulting $a_{\gamma\gamma}$ and $a_{gg}$ in the model are
obtained as

$\displaystyle a_{\gamma\gamma}$
$\displaystyle=\frac{gm_{h}}{2m_{W}}\left(a^{\rm
SM}_{2\gamma}+\frac{\lambda_{3}vm_{W}}{gm^{2}_{H^{\pm}_{I}}}F_{0}(\tau_{H^{\pm}_{I}})+N_{c}Q^{2}_{b}\sum_{i}\frac{y^{h}_{ii}v}{\sqrt{2}m_{t}}F_{1/2}(\tau_{B_{i}})\right)\,,$
$\displaystyle a_{gg}$ $\displaystyle=\frac{gm_{h}}{4m_{W}}\left(a^{\rm
SM}_{2g}+\sum_{i}\frac{y^{h}_{ii}v}{\sqrt{2}m_{t}}F_{1/2}(\tau_{B_{i}})\right)\,$
(64)

where $a^{\rm SM}_{2\gamma}\approx 6.51-i0.02$ and $a^{\rm
SM}_{2g}\approx-0.69$ are the SM results;
$y^{h}_{11}=-s_{2\theta}y_{B^{\prime}}$ and
$y^{h}_{22}=s_{2\theta}y_{B^{\prime}}$, and the functions $F_{0}$ and
$F_{1/2}$ are given as

$\displaystyle F_{0}(\tau)$ $\displaystyle=\tau(1-\tau f(\tau))\,,$
$\displaystyle F_{1/2}(\tau)$ $\displaystyle=-2\tau(1+(1-\tau)f(\tau))\,,$
(65)

with $\tau=4m^{2}_{f}/m^{2}_{h}$ and $f(\tau)=(\arcsin(1/\sqrt{\tau}))^{2}$.
Considering that we focus on the case with
$2m_{H^{\pm}_{I}},2m_{B_{i}}>m_{h}$, the on-shell condition in the loop
propagators is not available.

When $m_{B_{i}}\gg m_{h}$, $F_{1/2}(\tau)\to-4/3$. Because
$y^{h}_{11}=-y^{h}_{22}$, the contributions to the $hgg$ effective coupling
from the $B_{1}$ and $B_{2}$ quarks are canceled each other; that is, the
Higgs production through the gluon-gluon fusion is the same as the SM. Thus,
the signal strength for $pp\to h\to\gamma\gamma$ can be simplified as

$\mu_{\gamma\gamma}\approx\frac{BR(h\to\gamma\gamma)}{BR(h\to\gamma\gamma)^{\rm
SM}}\approx\left|1+\frac{\lambda_{3}vm_{W}}{gm^{2}_{H_{I}}a^{\rm
SM}_{2\gamma}}F_{0}(\tau_{H^{\pm}_{I}})\right|^{2}\,,$ (66)

where the new physics on the Higgs width $\Gamma_{h}$ is assumed to be small
and neglected in $\mu_{\gamma\gamma}$. To suppress the invisible Higgs decay
$h\to S_{I}S_{I}$ and to have $\Gamma_{h}\approx\Gamma^{\rm SM}_{h}$, we
simply take $m_{h}<2m_{S_{I}}$ in the model. Using $m_{H^{\pm}_{I}}=100$ GeV,
the $H^{\pm}_{I}$ effect on $\mu_{\gamma\gamma}$ can be estimated as
$-0.197\lambda_{3}$. If we take the allowed range of $\mu_{\gamma\gamma}$ to
be $0.8<\mu_{\gamma\gamma}<1.2$, then $\lambda_{3}$ is limited to be
$\lambda_{3}<0.5$. Hence, the $\lambda_{3}$ parameter can be bounded by the
$h\to\gamma\gamma$ measurement.

###  IV.4 DM direct detection

In the inert Higgs doublet model, although there is no $ZH_{I}H_{I}$ coupling
at the tree level, the nonvanizing $ZH_{I}A_{I}$ coupling will contribute to
the DM-nucleon scattering. To satisfy the DM direct detection experiments, the
$Z$-mediated $H_{I}n\to A_{I}n$ process has to be suppressed, where $n$
denotes the nucleon. The process can be kinematically forbidden by requiring
$m_{A_{I}}-m_{H_{I}}$ to be larger than the kinetic energy of the DM, where
the typical energy is tens of keV. In the study, we take
$m_{A_{I}}-m_{H_{I}}>1$ GeV.

The spin-independent (SI) DM-nucleon scattering can occur via the trilinear
coupling $hH_{I}H_{I}$ shown in Eq. (23). The Higgs-mediated cross-section can
be formulated as Barbieri:2006dq

$\displaystyle\sigma^{\rm
SI}_{h}=\frac{\mu^{2}_{H_{I}n}}{4\pi}\left|\frac{\lambda_{L}}{m_{H_{I}}m^{2}_{h}}\right|^{2}f^{2}_{n}m^{2}_{n}\,,$
(67)

where $\mu_{H_{I}n}=m_{H_{I}}m_{n}/(m_{H_{I}}+m_{n})$ is the DM-nucleon
reduced mass, and $f_{n}\approx 0.3$ is the nucleon matrix element. With
$m_{n}=0.94$ GeV and $m_{H_{I}}=70$ GeV, we have

$\sigma^{\rm SI}_{h}\approx 2.0\times
10^{-42}(\lambda_{3}-|\lambda_{4}|-|\lambda_{5}|)^{2}~{}\text{cm${}^{2}$}\,.$
(68)

To satisfy the XENON1T upper limit of $7\times 10^{-47}$ cm2 Aprile:2018dbl ,
$\lambda_{3}-|\lambda_{4}|-|\lambda_{5}|<5\times 10^{-3}$ is required; that
is, $\lambda_{3}\sim|\lambda_{4}|+|\lambda_{5}|$. As mentioned earlier,
$\lambda_{3}$ is bounded by $\mu_{\gamma\gamma}$. Therefore, the magnitude
$|\lambda_{4}|$ can be bounded by the DM direct detection. With the mass
ordering of $m_{H^{\pm}_{I}}>m_{A_{I}}>m_{H_{I}}$, we obtain
$|\lambda_{5}|<|\lambda_{4}|$.

##  V  Numerical analysis and discussions

Many free parameters are involved in the phenomenological analysis, such as
Yukawa couplings ${\bf Y}^{u}_{I}$ and ${\bf Y}^{B^{\prime}}_{I}$, the inert
scalar masses $m_{H_{I},A_{I},H^{\pm}_{I}}$, the $Z_{2}$-odd quark masses
$m_{B_{1},B_{2},T}$, and the parameter $\lambda_{3}$ in the scalar potential.
Before discussing their influence on the rare top decays, we first determine
the allowed ranges for the free parameters and then use the constrained
parameter values to analyze the implications on the rare top decays. For the
numerical analysis, the current experimental upper limits are taken as PDG

$\displaystyle BR(t\to q\gamma)$ $\displaystyle<1.8\times 10^{-4}\,,$
$\displaystyle BR(t\to qZ)$ $\displaystyle<5\times 10^{-4}\,,$ $\displaystyle
BR(t\to uh)$ $\displaystyle<1.2\times 10^{-3}\,,$ $\displaystyle BR(t\to ch)$
$\displaystyle<1.1\times 10^{-3}\,.$ (69)

###  V.1 Parameter choices and constraints

As stated earlier, the radiative $B$ decay cannot severely bound the Yukawa
couplings. Thus, we employ the perturbative unitarity constraint, and the
upper limits of the Yukawa couplings are required to be $|{\bf
Y}^{u(B^{\prime})}_{I}|<2\sqrt{2\pi}$ Castillo:2013uda . To obtain the mass
upper limit of the $Z_{2}$-odd quarks, we apply the similar constraints for
the stop and sbottom with the $R$-parity conserving supersymmetry, where using
the data with an integrated luminosity of 139 fb-1 at $\sqrt{s}=13$ TeV
ATLAS:2021hza , the mass below $1$ TeV has been excluded by ATLAS when the
neutralino mass is below 100 GeV. Thus, for the parameter scan, we assume
$m_{B_{1}}<m_{B_{2}}$ and take the mass regions for $m_{B_{1,2}}$ and $m_{T}$
to be

$m_{B_{1}}\in(1000,1200)~{}\text{GeV}\,,~{}m_{B_{2},T}\in(1000,2000)~{}\text{GeV}\,.$
(70)

If $H_{I}$ is the DM candidate in the inert Higgs model, then $m_{H_{I}}\sim
m_{W}$ can fit the observed DM relic abundance Barbieri:2006dq . It has been
studied that the direct bounds from colliders on $m_{H_{I},A_{I}}$ are not
strict, and the bound on $m_{H^{\pm}_{I}}$, which converts from the SUSY
search at LEP, is $m_{H^{\pm}_{I}}>70-90$ GeV Pierce:2007ut ; Lundstrom:2008ai
; Merchand:2019bod ; Belanger:2021lwd . Therefore, the mass regions of
$H_{I}$, $A_{I}$, and $H^{\pm}_{I}$ are taken as

$m_{H_{I},A_{I},H^{\pm}_{I}}\in(70,120)~{}\text{GeV}\,.$ (71)

The mass regions in Eqs. (70) and (71) are only used to set the boundaries of
the scanned parameters. Their mass differences are dictated by the oblique $T$
parameters, as shown in Eqs. (57) and (61). To understand the $T$-parameter
constraints, we perform the parameter scan based on the chosen parameter
regions. The correlation of $m_{B_{2}}-m_{T}$ and $m_{B_{2}}-m_{B_{1}}$ is
shown in Fig. 4(a), where $5\times 10^{6}$ random sampling points are used.
The allowed region for $m_{B_{2}}-m_{T}$ is limited within 200 GeV and that
for $m_{B_{2}}-m_{B_{1}}$ is wider; that is, the mass splitting in the same
representation is strictly constrained. Therefore, it is appropriate if we
take $m_{B_{2}}-m_{T}=100$ GeV for the top decay analysis. Similarly, the
$T$-parameter constraint on $m_{A_{I}}-m_{H_{I}}$ and $m_{H^{\pm}_{I}-H_{I}}$
is shown in Fig. 4(b), where the mass ordering
$m_{H^{\pm}}>m_{A_{I}}>m_{H_{I}}$ is applied. Because the chosen ranges of
$m_{H_{I},A_{I},H^{\pm}_{I}}$ in Eq. (71) are not broad, the constraint is not
significant. As $m_{B_{2}}-m_{T}$ is related to the parameter $y_{B^{\prime}}$
or $s_{\theta}$, the correlation between $y_{B^{\prime}}$ and $s_{\theta}$
under the $T$-parameter constraint is shown in Fig. 4(c). When
$m_{B_{2}}-m_{T}$ is limited, the allowed $y_{B^{\prime}}$ parameter is
bounded by $|y_{B}^{\prime}|\lesssim 1.5$, where the maximum $s_{\theta}$ can
still reach $|s_{\theta}|_{\rm max}\sim 1/\sqrt{2}$. Using Eq. (66), the
strength signal for the Higgs to diphoton as a function of $m_{H^{\pm}_{I}}$
and $\lambda_{3}$ is shown in Fig. 4(d). If the uncertainty of the observed
strength signal is $10\%$ of the SM result, then the $\lambda_{3}$ value is
limited to be approximately $0.2$ when $m_{H^{\pm}}=90$ GeV is used.

Figure 4:  Plots (a)-(c) show the constraints from the oblique $T$-parameter,
and plot (d) is the constraint from $\mu_{\gamma\gamma}$.

###  V.2  BRs for $t\to q(h,\gamma)$ and $t\to qZ$

After analyzing the constraints of the parameters, in this subsection, we
numerically calculate and discuss the $S_{I}$\- and $H^{\pm}_{I}$-mediated
contributions to the $t\to q(h,\gamma)$ and $t\to qZ$ decays. As stated
before, many parameters are involved in the processes. We can use the
parameter scan to comprehend the influence of various parameters. Before
scanning the parameters, we need to examine whether the $t\to qV$ and $t\to
qh$ decays can be simultaneously enhanced to the current experimental upper
limits.

For the purpose of illustration, we use the formulas given in Eqs. (43) and
(49) and show the contours for $BR(t\to q\gamma)$ (in units of $10^{-6}$) and
$BR(t\to qZ)$ (in units of $10^{-4}$) in Fig. 5(a) and (b), respectively,
where the dashed lines denote $BR(t\to qh)$. The parameter values are taken as
$m_{H_{I},A_{I},H^{\pm}_{I}}=(65,70,90)$ GeV, $m_{B_{1},B_{2},T}=(1,1.5,1.4)$
TeV, $\lambda_{3}=0.2$, $y_{B^{\prime}}=1.0$, and
$Y^{B^{\prime}}_{Iq,I3}=(4,5)$. As shown in the plots, the BRs for $t\to
q(h,\gamma,Z)$ with the chosen parameter values can reach the levels of
$(10^{-3},10^{-6},10^{-4})$, where with the exception of $t\to q\gamma$, $t\to
q(h,Z)$ can reach the current upper bounds that are shown in Eq. (69). Based
on the results, although the used Yukawa couplings are lower than the upper
limit from the perturbative unitarity, $BR(t\to q\gamma)$ of $O(10^{-6})$
inevitably has to reply on the large Yukawa couplings. To illustrate the
situation with small Yukawa couplings, we fix $Y^{B^{\prime}}_{Iq,I3}=(2,3)$
and show the regions for $BR(t\to qh)\geq 1.5\times 10^{-4}$ and $BR(t\to
qZ)\geq 1.5\times 10^{-5}$ as a function of $Y^{u}_{Iq,I3}$ in Fig. 6. In
these chosen regions, $BR(t\to q\gamma)$ is far below $10^{-6}$.

Figure 5: Contours for (a) $BR(t\to q\gamma)$ and (b) $BR(t\to qZ)$ as a
function of $Y^{u}_{Iq}$ and $Y^{u}_{I3}$, where the dashed lines are $BR(t\to
qh)$ and $Y^{B^{\prime}}_{Iq,I3}=(4,5)$ are used. The other taken parameter
values can be found in the text. Figure 6: Regions for $BR(t\to qh)\geq
1.5\times 10^{-4}$ and $BR(t\to qZ)\geq 1.5\times 10^{-5}$ as a function for
$Y^{u}_{Iq}$ and $Y^{u}_{I3}$, where $Y^{B^{\prime}}_{Iq}=2$, and
$Y^{B^{\prime}}_{I3}=3$ are used.

In the following, we discuss the influence of various parameters on the rare
top decays in detail. To reduce the number of scanned parameters, the
parameters that are insensitive to the studying phenomena are fixed as follows

$m_{H_{I},A_{I},H^{\pm}_{I}}=(65,70,90)~{}\text{GeV}\,,~{}\lambda_{3}=0.2\,,$
(72)

where the values are chosen to satisfy the constraints obtained earlier. Thus,
the involving parameters are the $Z_{2}$-odd quark masses $m_{B_{1},B_{2},T}$
and Yukawa couplings $y_{B^{\prime}}$, $Y^{B^{\prime},u}_{I1,I3}$, and their
scanning regions are chosen as

$\displaystyle m_{B_{1}}$ $\displaystyle\in[1000,1300]~{}{\rm
GeV}\,,~{}m_{T}\in[1000,2000]~{}{\rm GeV}\,,~{}m_{B_{2}}=m_{T}+100\,,$
$\displaystyle y_{B^{\prime}}$
$\displaystyle\in(-1.5,1.5)\,,~{}Y^{B^{\prime}}_{Iq,I3}\in(-3,3)\,,~{}Y^{u}_{Iq,I3}\in(-3,3)\,.$
(73)

According to the ATLAS results of $BR(t\to ug)\lesssim 0.61\times 10^{-4}$
ATLAS:2021amo , we can indirectly bound the $t\to q\gamma$ to be $BR(t\to
q\gamma)\lesssim 3.2\times 10^{-6}$ using $BR(t\to q\gamma)\sim BR(t\to
q\gamma)\alpha/(C_{F}\alpha_{s})$ in the model, where the upper limit is
smaller than the current experimental upper limit. Therefore, it is sufficient
to use the small Yukawa couplings for the scan. In addition, in order to show
the resulting BRs that can reach the sensitivities at the HL LHC, we require
the obtained BRs for the rare top decays to be

$\displaystyle 10^{-5}$ $\displaystyle<BR(t\to qh)<10^{-3}\,,$ $\displaystyle
2\times 10^{-5}$ $\displaystyle<BR(t\to qZ)<5\times 10^{-4}\,.$ (74)

However, the radiative top decay is required to be $BR(t\to q\gamma)>0.3\times
10^{-6}$. Since we have taken small Yukawa couplings, $BR(t\to q\gamma)$
cannot reach the current upper limit, i.e., $O(10^{-4})$. Thus, it is not
necessary to set the upper value for $BR(t\to q\gamma)$.

We show the scatter plots for the Yukawa couplings of $Y^{B^{\prime}}_{Iq,I3}$
and $Y^{u}_{Iq,I3}$, which fit the ranges given in Eqs. (73) and (74), in Fig.
7(a) and (b). Hence, $|Y^{B^{\prime},u}_{Iq,I3}|<1$ are disfavored. The
correlation between $y_{B^{\prime}}$ and $m_{B_{1}}$ is shown in Fig. 7(c).
From the numerical results, $|y_{B^{\prime}}|\lesssim 0.5$ is excluded. In
addition, the correlation between $m_{B_{1}}$ and $m_{B_{2}}$ can be found in
Fig. 7(d).

Figure 7:  Correlations of Yukawa couplings and $m_{B_{1,2}}$ that fit the
taken ranges shown in Eqs. (73) and (74).

After knowing the constrained parameter ranges, in the following, we discuss
the implications on the $t\to q(\gamma,h,Z)$ decays. Because $t\to q\gamma$
and $t\to qh$ have similar chirality structures in decay amplitudes and arise
from the similar Feynman diagrams, we analyze their numerical results
together. When neglecting the small contributions from Fig. 1(c) and (d), as
stated earlier, $t\to q\gamma,qh$ decays can arise from Fig. 1(a) and (b).
Compared to the $H^{\pm}\gamma$ coupling, the $B_{i}B_{i}\gamma$ coupling has
a suppression factor from the electric charge of $B_{i}$; thus, the $t\to
q\gamma$ decay indeed is dominated by Fig. 1(a). By contrast, because the
$H^{\pm}_{I}h$ coupling is $\lambda_{3}$ and is limited by the
$h\to\gamma\gamma$ measurement, the dominant contribution to $t\to qh$ is from
Fig. 1(b). Hence, the resulting $BR(t\to q\gamma)$ and $BR(t\to qh)$ as a
function of $m_{B_{1}}$ are shown in Fig. 8(a) and (b). In the chosen Yukawa
couplings, the BR for $t\to q\gamma$ can only maximally reach of the
$O(10^{-7})$, which is consistent with the indirect bound from the $t\to qg$
measurement. The dependence of $B(t\to qh)$ with respect to $m_{B_{2}}$ is
shown in Fig. 8(c), where $BR(t\to qh)$ increases as $m_{B_{2}}$ increases.
The reason for the behavior can be understood as the relaxed cancellation
between $B_{1}$ and $B_{2}$ when the mass of $B_{2}$ increases; that is, the
$B_{1}$ becomes the dominant effect if $m_{B_{2}}\gg m_{B_{1}}$. In Fig. 8(d),
we show the scatter plot for the correlation between $BR(t\to q\gamma)$ and
$BR(t\to qh)$. The dashed lines in Fig. 8 denote the sensitivity of the HL
LHC.

Figure 8:  (a)-(b) $BR(t\to q\gamma,qh)$ as a function of $m_{B_{1}}$; (c)
$BR(t\to qh)$ as a function of $m_{B_{2}}$, and (d) correlation between
$BR(t\to q\gamma)$ and $BR(t\to qh)$.

In addition to the dipole operators, unlike the $t\to q\gamma$ decay, $t\to
qZ$ also involves the vectorial types of operators in the decay amplitude,
which are not associated with the chirality flip in the quark currents, so
they are the dominant effect. To observe the influence of the new physics
effects on $t\to qZ$, we show the scatter plots for the correlations of
$BR(t\to qZ)$ with $BR(t\to q\gamma)$ and with $BR(t\to qh)$ in Fig. 9(a) and
(b), respectively, where the dashed lines are the sensitivity that the HL LHC
is planning to reach. With the exception of the $t\to q\gamma$ decay, the BRs
for the $t\to qh$ and $t\to qZ$ decays in the model can reach the level of
$O(10^{-4})$.

Figure 9:  Correlation of $BR(t\to qZ)$ with (a) $BR(t\to q\gamma)$ and (b)
$BR(t\to qh)$.

##  VI Summary

In this study, we investigated the potential effects that can enhance the top-
FCNC processes, where the processes are highly suppressed in the SM. If we
assume that $t\to qh$ and $t\to qZ$ are induced via quantum loop diagrams, the
intermediate states in the loop may have different properties from the SM
particles. Inspired from the mechanism of the scotogenic neutrino mass, we
consider that the new particles carry an extra $Z_{2}$-odd parity, whereas the
SM particles are $Z_{2}$-even.

To retain the basic element in the radiative neutrino mass Ma:2006km , in
addition to the $Z_{2}$ discrete symmetry, we can extend the SM by including
one inert Higgs doublet, one vector-like $Z_{2}$-odd doublet quark, and one
vector-like $Z_{2}$-odd singlet quark.

The potential constraints from the experimental observations are taken into
account, such as the oblique parameters, Higgs to diphoton decay, and DM
direct detection. Although the $b\to s\gamma$ decay and $\Delta B=2$ process
can be induced in the model, their effects can be small.

According to the recent ATLAS’s measurement that shows $BR(t\to qg)\lesssim
O(10^{-4})$, we found that the indirect bound on $t\to q\gamma$ in the model
is $BR(t\to q\gamma)\lesssim 3.2\times 10^{-6}$, which is lower than the
expected sensitivity at the HL LHC. With the exception of $t\to q\gamma$, the
branching ratios for the loop-induced $t\to qh$ and $t\to qZ$ decays in the
model can be of the order of $10^{-4}$ and can be tested at the HL LHC.

## Acknowledgments

This work was supported in part by the Ministry of Science and Technology,
Taiwan under the Grant No. MOST-110-2112-M-006-010-MY2 (C. H. Chen). The work
was supported in part by the Fundamental Research Funds for the Central
Universities (T. N.).

## References

  * (1) S. L. Glashow, J. Iliopoulos and L. Maiani, Phys. Rev. D 2, 1285 (1970).
  * (2) J. A. Aguilar-Saavedra, Acta Phys. Polon. B 35, 2695 (2004) [hep-ph/0409342].
  * (3) G. Abbas, A. Celis, X. Q. Li, J. Lu and A. Pich, JHEP 1506, 005 (2015) [arXiv:1503.06423 [hep-ph]].
  * (4) S. Balaji, Phys. Rev. D 102 (2020) no.11, 113010 [arXiv:2009.03315 [hep-ph]].
  * (5) [ATLAS], ATL-PHYS-PUB-2019-001.
  * (6) [ATLAS], ATL-PHYS-PUB-2016-019.
  * (7) [CMS], CERN-LHCC-2017-023.
  * (8) K. J. Abraham, K. Whisnant, J. M. Yang and B. L. Young, Phys. Rev. D 63, 034011 (2001) [hep-ph/0007280].
  * (9) G. Eilam, A. Gemintern, T. Han, J. M. Yang and X. Zhang, Phys. Lett. B 510, 227 (2001) [hep-ph/0102037].
  * (10) J. A. Aguilar-Saavedra, Phys. Rev. D 67, 035003 (2003) Erratum: [Phys. Rev. D 69, 099901 (2004)] [hep-ph/0210112].
  * (11) U. K. Dey and T. Jha, Phys. Rev. D 94 (2016) no.5, 056011 [arXiv:1602.03286 [hep-ph]]. Copy to ClipboardDownload
  * (12) R. Gaitan, R. Martinez, J. H. M. de Oca and E. A. Garces, Phys. Rev. D 98, no. 3, 035031 (2018) [arXiv:1710.04262 [hep-ph]].
  * (13) M. Badziak and K. Harigaya, Phys. Rev. Lett. 120 (2018) no.21, 211803 [arXiv:1711.11040 [hep-ph]].
  * (14) J. F. Shen, Y. Q. Li and Y. B. Liu, Phys. Lett. B 776, 391-395 (2018) [arXiv:1712.03506 [hep-ph]].
  * (15) C. W. Chiang, U. K. Dey and T. Jha, Eur. Phys. J. Plus 134, no.5, 210 (2019) [arXiv:1807.01481 [hep-ph]].
  * (16) K. Y. Oyulmaz, A. Senol, H. Denizli, A. Yilmaz, I. Turk Cakir and O. Cakir, Eur. Phys. J. C 79, no.1, 83 (2019) [arXiv:1811.01074 [hep-ph]].
  * (17) C. H. Chen and T. Nomura, Eur. Phys. J. C 79, no.8, 644 (2019) [arXiv:1812.05904 [hep-ph]].
  * (18) M. A. Arroyo-Ureña, R. Gaitán, E. A. Herrera-Chacón, J. H. Montes de Oca Y. and T. A. Valencia-Pérez, JHEP 07, 041 (2019) [arXiv:1903.02718 [hep-ph]].
  * (19) L. Shi and C. Zhang, Chin. Phys. C 43, no.11, 113104 (2019) [arXiv:1906.04573 [hep-ph]].
  * (20) Y. B. Liu and S. Moretti, Phys. Rev. D 101, no.7, 075029 (2020) [arXiv:2002.05311 [hep-ph]].
  * (21) W. S. Hou, T. H. Hsu and T. Modak, Phys. Rev. D 102, no.5, 055006 (2020) [arXiv:2008.02573 [hep-ph]]. Bie:2020sro
  * (22) S. Y. Bie, G. L. Liu and W. Wang, Chin. Phys. C 45, no.1, 013106 (2021) [arXiv:2009.04858 [hep-ph]].
  * (23) P. Gutierrez, R. Jain and C. Kao, Phys. Rev. D 103, no.11, 115020 (2021) [arXiv:2012.09209 [hep-ph]].
  * (24) Y. Liu, B. Yan and R. Zhang, Phys. Lett. B 827, 136964 (2022) [arXiv:2103.07859 [hep-ph]].
  * (25) F. M. Cai, S. Funatsu, X. Q. Li and Y. D. Yang, [arXiv:2202.08091 [hep-ph]].
  * (26) A. I. Hernández-Juárez and G. Tavares-Velasco, [arXiv:2203.16819 [hep-ph]].
  * (27) G. Aad et al. [ATLAS], [arXiv:2112.01302 [hep-ex]].
  * (28) E. Ma, Phys. Rev. D 73, 077301 (2006) [hep-ph/0601225].
  * (29) N. Aghanim et al. [Planck], Astron. Astrophys. 641, A6 (2020) [erratum: Astron. Astrophys. 652, C4 (2021)] [arXiv:1807.06209 [astro-ph.CO]].
  * (30) R. Barbieri, L. J. Hall and V. S. Rychkov, Phys. Rev. D 74, 015007 (2006) [arXiv:hep-ph/0603188 [hep-ph]].
  * (31) A. Pierce and J. Thaler, JHEP 08, 026 (2007) [arXiv:hep-ph/0703056 [hep-ph]].
  * (32) E. Lundstrom, M. Gustafsson and J. Edsjo, Phys. Rev. D 79, 035013 (2009) [arXiv:0810.3924 [hep-ph]].
  * (33) M. Merchand and M. Sher, JHEP 03, 108 (2020) [arXiv:1911.06477 [hep-ph]].
  * (34) G. Belanger, A. Mjallal and A. Pukhov, Phys. Rev. D 105, no.3, 035018 (2022) [arXiv:2108.08061 [hep-ph]].
  * (35) K. G. Klimenko, Theor. Math. Phys. 62, 58-65 (1985).
  * (36) K. Kannike, Eur. Phys. J. C 72, 2093 (2012) [arXiv:1205.3781 [hep-ph]].
  * (37) M. Jezabek and J. H. Kuhn, Nucl. Phys. B 314, 1-6 (1989)
  * (38) P. A. Zyla et al. (Particle Data Group), Prog. Theor. Exp. Phys. 2020, 083C01 (2020).
  * (39) G. Buchalla, A. J. Buras and M. E. Lautenbacher, Rev. Mod. Phys. 68, 1125-1144 (1996) [arXiv:hep-ph/9512380 [hep-ph]].
  * (40) A. J. Buras, S. Jager and J. Urban, Nucl. Phys. B 605 (2001), 600-624 [arXiv:hep-ph/0102316 [hep-ph]].
  * (41) M. E. Peskin and T. Takeuchi, Phys. Rev. D 46, 381 (1992).
  * (42) L. Lavoura and J. P. Silva, Phys. Rev. D 47, 2046 (1993).
  * (43) C. Y. Chen, S. Dawson and E. Furlan, Phys. Rev. D 96, no.1, 015006 (2017) [arXiv:1703.06134 [hep-ph]].
  * (44) [ATLAS], ATLAS-CONF-2020-026.
  * (45) [CMS], CMS-PAS-HIG-19-015.
  * (46) E. Aprile et al. [XENON Collaboration], Phys. Rev. Lett. 121, no. 11, 111302 (2018) [arXiv:1805.12562 [astro-ph.CO]].
  * (47) A. Castillo, R. A. Diaz and J. Morales, Int. J. Mod. Phys. A 29, no.18, 1450085 (2014) [arXiv:1309.0831 [hep-ph]].
  * (48) G. Aad et al. [ATLAS], JHEP 04, 165 (2021) [arXiv:2102.01444 [hep-ex]].