#
Modular Origin of Mass Hierarchy:
Froggatt-Nielsen like Mechanism

Hitomi Kuranaga1, Hiroshi Ohki1, and Shohei Uemura2
1Department of Physics, Nara Women’s University, Nara 630-8506, Japan
2CORE of STEM, Nara Women’s University, Nara 630-8506, Japan

( Abstract
We study Froggatt-Nielsen (FN) like flavor models with modular symmetry. The
FN mechanism is a convincing solution to the flavor puzzle in quark sector.
The FN mechanism requires an extra $U(1)$ gauge symmetry which is broken at
high energy. Alternatively, in the framework of modular symmetry the modular
weights can play the role of the FN charges of the extra $U(1)$ symmetry.
Based on the FN-like mechanism with modular symmetry we present new flavor
models for quark sector. Assuming that the three generations have a common
representation under modular symmetry, our models simply reproduce the FN-like
Yukawa matrices. We also show that the realistic mass hierarchy and mixing
angles, which are related each other through the modular parameters and a
scalar vev, can be realized in models with several finite modular groups (and
their double covering groups) without unnatural hierarchical parameters. )

##  1 Introduction

The origin of the flavor structure of the Standard Model (SM) is one of the
most challenging problems in the particle physics. The observed mass
eigenvalues of the matter fields have a large hierarchy, which is more than
$10^{6}$ among them. This hierarchy can not be explained in the framework of
the SM, since the Yukawa couplings are free parameters in the SM. Thus we
require a beyond the SM mechanism naturally reproducing it.

The modular symmetry is a recently proposed solution for the flavor puzzle [1,
2]. In this model, the action is assumed to be invariant under the
(inhomogeneous) modular group $\Gamma\simeq PSL(2,\mathbb{Z})$, which is the
quotient group of $SL(2,\mathbb{Z})$ divided by its center $\\{I,-I\\}$. In
this model, coupling constants are no longer free parameters, but modular
forms. The modular forms are specific holomorphic functions of the complex
parameter known as the modulus. They form unitary representations of the
quotient group of the modular group: $\Gamma_{N}=\Gamma/\Gamma(N)$.
$\Gamma(N)$ is known as a principle congruence subgroup of $\Gamma$.
$\Gamma_{N}$ is isomorphic to a non-Abelian discrete group. In particular, it
is isomorphic to a finite group when the level $N$ is lower than 6 [3].
Therefore coupling constants, as well as the dynamical fields, transform as
unitary representations of the non-Abelian finite group in this model. This is
attractive for the particle phenomenologist since discrete symmetries are well
known candidate solutions for the flavor puzzle, especially for the lepton
flavor structure, [4, 5, 6].111For review, see [7, 8] and other approaches
including continuous flavor symmetry and the GUT are shortly reviewed in [9].
Indeed, various modular invariant models which successfully reproduce the SM
have been constructed in these years, e.g., for $\Gamma_{2}$ [10, 11, 12, 13],
$\Gamma_{3}$ [1, 2, 10, 13, 14, 15, 16, 17, 18, 19, 20], $\Gamma_{4}$ [13, 16,
21, 22, 23, 24], $\Gamma_{5}$ [13, 24, 25], $\Gamma_{7}$ [26], and the double
covering groups of the modular groups [13, 27, 28, 29, 30, 31, 32, 33]. A
combined symmetry of the modular symmetry with the conventional flavor
symmetries or CP-symmetry is also considered in [34, 35, 36, 37, 38, 39, 40,
41].

The Froggatt-Nielsen (FN) mechanism is another well-known possible solution
for the flavor puzzle [42]. In the original FN mechanism, the matter fields
such as the left-handed quarks $Q_{i}$ and the right-handed quarks $u_{i}^{c}$
and $d_{i}^{c}$ are assumed to be charged under an extra gauge group denoted
by $U(1)_{FN}$, which prohibits the tree level Yukawa coupling except for the
top quark. An extra scalar field $\phi$, which is a trivial singlet under the
SM gauge group, is introduced to spontaneously break $U(1)_{FN}$. Since the
effective Yukawa couplings are given by higher order couplings suppressed by
its vacuum expectation value $(\langle\phi\rangle/\Lambda)^{n}$, where $n$ is
a difference of $U(1)_{FN}$ charges between generations, the mass hierarchy is
controlled by the $U(1)_{FN}$ charges of the quark fields. It is interesting
that in the FN model the mass ratios are also related to the mixing angles, so
that it naturally explains a realistic mass hierarchy and the mixing angles of
the quark sector simultaneously if $\langle\phi\rangle/\Lambda$ is chosen to
the Cabibbo angle [9].

We consider FN-like mechanism in the framework of modular symmetry. In
analogous to the FN mechanism, the modular weights of the fermion fields play
the role of $U(1)_{FN}$ charges. An extra SM singlet $\phi$ with a negative
modular weight is introduced, which compensates the modular weights of the
fermion fields. The effective couplings are given by higher dimensional
operators suppressed by powers of $(\phi/\Lambda)^{n}$, where $n$ is a
difference of modular weights between generations. The Yukawa coupling itself
is also controlled by the modular weights of the fermion fields, since the
modular forms are classified by modular weights. This FN-like mechanism based
on modular symmetry has been recently considered to explain the large mass
hierarchy in [24, 43, 44, 45].222 Another approach to the mass hierarchy by
the residual modular symmetry can be found in [13, 46]. We note that while in
the previous model the mass hierarchy originates from powers of the weighton
vev, the resulting quark mass matrices do not fully simulate the FN-like
structure, so that the relationship between two origins of mass hierarchy and
the small mixing angles becomes more subtle than that of the FN model.

In this paper, we present new flavor models for quark sector based on the FN-
like mechanism with modular symmetry. In our models, it is assumed that the
three generations of singlet quarks have a common representation under the
modular symmetry, so that we see the same order of suppression factors
appearing in each column or each row of the Yukawa matrices, where the two
different suppression factors (the exponentially suppressed factor in the
modular forms and a power suppression by $\phi/\Lambda$) are incorporated,
respectively. Thus our models can simply reproduce the characteristic
structure of the FN-like mass matrices. We illustrate this mechanism in models
with different finite modular groups of
$\Gamma_{3}^{\prime},\Gamma_{4}^{\prime}$ and $\Gamma_{5}^{\prime}$ in detail.
Following Ref. [43], we analyze an approximate expression for the mass ratios
and the mixing angles, where a hierarchical mass structure which relates to
the mixing angles is easily obtained with a suitable choice of the modular
parameters and the singlet vev. We then numerically confirm this mechanism by
a fit analysis, where we find parameter sets for the realistic mass hierarchy
and mixing angles with $\mathcal{O}(1)$ coefficients. The validity of the
approximate estimation and the stability against changes of free parameters
are also numerically investigated through the parameter dependence of the
results.

This paper is organized as follows. In section 2, we briefly review the
modular symmetry. We also explain possible two origins of the hierarchy for
the modular symmetry; one is the hierarchy among the modular forms, and the
other is FN mechanism. In section 3, we consider Froggatt-Nielsen like
superpotential with the modular group of level 3. In this model, hierarchical
mass matrix is realized by cooperation between the above two possible origins.
In section 4, we generalize the previous model to the modular group of higher
levels. In section 5, we investigate our models statistically. Section 6 is
devoted to the conclusion. We also review the modular forms of level 3, 4, and
5 in Appendix.

##  2 Modular symmetry and the Froggatt-Nielsen mechanism

In this section, we briefly review the modular symmetry and the Froggatt-
Nielsen mechanism. We also introduce our notations mostly based on [1].

The modular symmetry is a recently proposed model building framework. In this
model, the action is assumed to include a complex parameter known as the
complex structure modulus $\tau\in\mathbb{H}$. The complex structure
parameterizes the geometry of torus. Torus is invariant under the linear
fractional transformation,

$\displaystyle\tau\rightarrow\gamma\tau=\frac{a\tau+b}{c\tau+d},$ (1)

where $\gamma=\begin{pmatrix}a&b\\\ c&d\end{pmatrix}\in SL(2,\mathbb{Z})$
since $\tau$ and $\gamma\tau$ generate the same lattice. It is obvious that
both $\gamma$ and $-\gamma$ equivalently act on $\tau$. Thus the torus is
invariant under $PSL(2,\mathbb{Z})\equiv SL(2,\mathbb{Z})/\mathbb{Z}_{2}$.
This is the (inhomogeneous) modular group $\Gamma$. The modular group is
generated by two generators,

$\displaystyle S=\begin{pmatrix}0&1\\\ -1&0\end{pmatrix}~{}~{}{\rm
and}~{}~{}T=\begin{pmatrix}1&1\\\ 0&1\end{pmatrix},$ (2)

and these generators satisfying the following relations

$\displaystyle S^{2}=(ST)^{3}=I,$ (3)

where $I$ is the identity. In this paper, we abuse elements of
$SL(2,\mathbb{Z})$ to denote the corresponding elements of the modular group
$SL(2,\mathbb{Z})/\mathbb{Z}_{2}$. For instance, $S$ is the element of
$SL(2,\mathbb{Z})$ originally, but the same symbol also denotes the
equivalence class in the modular group. The modular group acts on the
effective action. For instance, coupling constants such as Yukawa couplings
depend on the moduli $\tau$. They transform under the modular group through
the modular transformation of $\tau$. To construct a modular invariant action,
coupling constants should form representations of the modular group, and such
functions are known as the modular forms.

Before considering modular forms, we introduce the principal congruence
subgroup of level $N$ in $\Gamma$, which is usually represented by
$\Gamma(N)$. $\Gamma(N)$ is given by

$\displaystyle\Gamma(N)=\left\\{\begin{pmatrix}a&b\\\
c&d\end{pmatrix}\in\Gamma\;\middle|\;a=d=1~{}{\rm and}~{}b=c=0~{}{\rm
mod}~{}N\right\\}.$ (4)

The modular forms of level $N$ and weight $k$ are holomorphic functions
satisfying the following transformation

$\displaystyle f(\gamma\tau)=(c\tau+d)^{k}f(\tau),$ (5)

for any $\gamma$ in $\Gamma(N)$. The prefactor $(c\tau+d)^{k}$ is so-called
automorphy factor. Since linear combinations of the modular forms of level $N$
and weight $k$ are also modular forms of level $N$ and weight $k$, they form a
linear space, which is denoted by $\mathcal{M}_{k}(\Gamma(N))$.
$\mathcal{M}_{k}(\Gamma(N))$ is finite dimensional. The modular forms form
unitary representations of the quotient group $\Gamma_{N}=\Gamma/\Gamma(N)$ up
to the automorphy factor,

$\displaystyle f_{i}(\gamma\tau)=(c\tau+d)^{k}\rho(\gamma)_{ij}f_{j}(\tau),$
(6)

where $\\{f_{i}\\}$ is a basis of $\mathcal{M}_{k}(\Gamma_{N})$, and $\rho$ is
a unitary representation of $\Gamma_{N}$. The generators of $\Gamma_{N}$ are
satisfying the following relations,333We abuse elements of
$SL(2,\mathbb{Z})_{2}$ to denote elements of $\Gamma_{N}$ too.

$\displaystyle S^{2}=(ST)^{3}=T^{N}=I.$ (7)

$\Gamma_{N}$ is isomorphic to the non-Abelian finite group when $N$ is smaller
than 6: $\Gamma_{2}\simeq S_{3},\Gamma_{3}\simeq A_{4},\Gamma_{4}\simeq
S_{4},\Gamma_{5}\simeq A_{5}$ [3]. Note that the definition of the automorphy
factor has ambiguity since the modular group is divided by its center. Hence
the modular forms are well-defined only if the modular weight is even. If we
consider the double covering group of the modular group instead of the usual
modular group, modular forms of odd modular weights can be defined as well.
The double covering group of the modular group is known as homogeneous modular
group $\Gamma^{\prime}$. To distinguish $\Gamma$ from $\Gamma^{\prime}$,
$\Gamma$ is called inhomogeneous modular group. $\Gamma^{\prime}$ is nothing
but $SL(2,\mathbb{Z})$ itself, and there are no ambiguities of sign of $c$ and
$d$. The principal congruence subgroup of level $N$ in the homogeneous modular
group, and its quotient group $\Gamma_{N}^{\prime}$ is similarly obtained as

$\displaystyle\Gamma_{N}^{\prime}=\Gamma^{\prime}/\Gamma^{\prime}(N),$ (8)

where $\Gamma^{\prime}(N)$ is a subgroup of $\Gamma^{\prime}$ whose element is
equivalent to $I$ mod $N$. The generator of $\Gamma_{N}^{\prime}$ are
satisfying the similar relations:

$\displaystyle S^{2}=R,~{}~{}(ST)^{3}=I,~{}~{}T^{N}=I,~{}~{}TR=RT,~{}R^{2}=I.$
(9)

$\Gamma_{N}^{\prime}$ is isomorphic to non-Abelian finite group when $N<6$,
too: $\Gamma_{2}^{\prime}=\Gamma_{2}\simeq S_{3},\Gamma_{3}^{\prime}\simeq
T^{\prime},\Gamma_{4}^{\prime}\simeq S_{4}^{\prime}$ and
$\Gamma_{5}^{\prime}\simeq A_{5}^{\prime}$ [27]. The modular symmetry is
originally inspired by string compactifications [49, 50, 51, 52, 53, 54]. From
the stringy perspective, the coupling constants and dynamical fields transform
under the homogeneous modular group rather than the inhomogeneous modular
group [36, 35, 55, 56, 57, 58, 59] and metaplectic group [60]. Hence we
consider modular symmetric model based on $\Gamma^{\prime}$ in this paper.

Throughout this paper, we assume global supersymmetry. The matter fields such
as quarks are denoted by chiral superfields. We assume that they transform as
the modular forms of level $N$ and weight $k$,

$\displaystyle\gamma~{}:~{}\Phi_{i}\rightarrow(c\tau+d)^{k_{i}}\rho_{\Phi}(\gamma)_{ij}\Phi_{j},$
(10)

where $\Phi_{i}$ is a matter field and $\rho_{\Phi}(\gamma)$ is a unitary
matrix. The action of chiral superfields $\Phi_{i}$ is given by two functions,
the Kähler potential $K$ and the superpotential $W$,

$\displaystyle\mathcal{S}=\int d^{4}xd^{2}\theta
d^{2}\bar{\theta}K(\Phi^{i},\bar{\Phi}^{i},\tau,\bar{\tau})+\int
d^{4}xd^{2}\theta W(\Phi^{i},\tau)+(h.c.).$

The typical modular invariant action is given as444This form of Kähler
potential is given by string compactifications at tree level. The general form
of the modular invariant Kähler potential includes modular forms and other
couplings. Such additional terms may affect results [61], but we ignore their
effects in the present paper for simplicity.

$\displaystyle K$ $\displaystyle=\sum_{i}\frac{\Phi_{i}\bar{\Phi}_{i}}{{\rm
Im\,}\tau^{-k_{i}}},$ (11) $\displaystyle W$
$\displaystyle=\sum(f_{i_{1}i_{2}...i_{n}}(\tau)\Phi_{i_{1}}\Phi_{i_{2}}...\Phi_{i_{n}})_{\bf
1},$ (12)

where $f_{i_{1}i_{2}...i_{n}}(\tau)$ is a modular form of weight $k$
satisfying

$\displaystyle k+k_{i_{1}}+...+k_{i_{n}}=0,$ (13)

so that the modular weight of the superpotential is zero.555From local
supersymmetry, the modulus $\tau$ is a vacuum expectation value of a dynamical
field, and the Kähler potential includes the kinetic term of the moduli field:
$\displaystyle K=-h\ln(\tau+\bar{\tau}).$ This term implies that the modular
weight of the superpotential is $h$ rather than zero, and the modular
invariant condition (13) is changed to $k+k_{i_{1}}+...+k_{i_{n}}=h$. However
it is always possible to cancel it by shifting the modular weight of the
chiral superfields. Therefore we assume (13) throughout this paper.  The
modular weight plays a similar role of the charge of $U(1)$ gauge symmetry.
$(f_{i_{1}i_{2}...i_{n}}(\tau)\Phi_{i_{1}}\Phi_{i_{2}}...\Phi_{i_{n}})_{\bf
1}$ denotes the trivial singlet component of the tensor product of the chiral
superfields and the modular form. We consider canonically normalized couplings
rather than the holomorphic couplings. The canonically normalized field
$\tilde{\Phi}_{i}$ is given by

$\displaystyle\tilde{\Phi}_{i}={\rm Im\,}\tau^{k_{i}/2}\Phi_{i},$ (14)

and the canonically normalized $n$-point coupling
$\tilde{f}_{i_{1}i_{2}...i_{n}}$is given by

$\displaystyle\tilde{f}_{i_{1}i_{2}...i_{n}}={\rm
Im\,}\tau^{-\frac{k_{1}+k_{2}+...+k_{n}}{2}}f_{i_{1}i_{2}...i_{n}}(\tau)={\rm
Im\,}\tau^{k/2}f_{i_{1}i_{2}...i_{n}}(\tau),$ (15)

where $k$ is the modular weight of $f_{i_{1}i_{2}...i_{n}}$ itself.

In order to be invariant under modular symmetry, the allowed Yukawa couplings
are also given in term of the modular forms, which are classified by modular
weights. In the followings, we discuss the properties of the modular forms.

###  2.1 Hierarchy in the modular forms

The modular forms naturally have a hierarchy. To illustrate this point
clearly, we consider the modular forms of level 3 as a concrete example. The
modular forms of level 3 and weight $k$ form a $k+1$ dimensional linear space
$\mathcal{M}_{k}(\Gamma(3))$ [27, 28]. The modular forms of weight 1 are given
by

$\displaystyle\hat{e}_{1}(\tau)\equiv\frac{\eta^{3}(3\tau)}{\eta(\tau)},~{}~{}~{}~{}\hat{e}_{2}(\tau)\equiv\frac{\eta^{3}(\tau/3)}{\eta(\tau)},$
(16)

where $\eta(\tau)$ is the Dedekind eta function,

$\displaystyle\eta(\tau)=q^{1/24}\prod_{n=1}^{\infty}(1-q^{n}),~{}~{}~{}~{}q\equiv
e^{2\pi i\tau}.$

Note that since we consider the homogeneous modular group, odd weight modular
forms can be defined. The modular transformations of $\hat{e}_{1}$ and
$\hat{e}_{2}$ are given by

$\displaystyle\begin{cases}\hat{e}_{1}(\tau+1)=e^{i2\pi/3}\hat{e}_{1}(\tau),~{}~{}\hat{e}_{2}(\tau+1)=3(1-e^{i2\pi/3})\hat{e}_{1}(\tau)+\hat{e}_{2}(\tau),\\\
\hat{e}_{1}(-1/\tau)=3^{-3/2}(-i\tau)\hat{e}_{2}(\tau),~{}~{}\hat{e}_{2}(-1/\tau)=3^{3/2}(-i\tau)\hat{e}_{1}(\tau),\end{cases}$
(17)

and we can check that the action of the modular group is closed in this space.
Since the modular forms form representations of $\Gamma_{3}^{\prime}$, they
can be decomposed to the irreducible representations of
$\Gamma_{3}^{\prime}\simeq T^{\prime}$. The irreducible representations of
$T^{\prime}$ are as follows,666Their matrix representations are summarized in
Appendix A.

$\displaystyle{\bf
1,1^{\prime},1^{\prime\prime},2,2^{\prime},2^{\prime\prime},3}.$ (18)

$\hat{e}_{1},\hat{e}_{2}$ form ${\bf 2}$ of $T^{\prime}$ by

$\displaystyle Y^{(1)}_{\bf 2}(\tau)=\begin{pmatrix}Y_{1}(\tau)\\\
Y_{2}(\tau)\end{pmatrix}=\begin{pmatrix}\sqrt{2}e^{i7\pi/12}\hat{e}_{1}(\tau)\\\
\hat{e}_{1}(\tau)+\frac{1}{3}\hat{e}_{2}(\tau)\end{pmatrix}.$ (19)

The modular forms of higher weights are constructed by the tensor products of
the modular forms of weight 1, e.g., the modular forms of weight 2 are given
by

$\displaystyle Y_{\bf 3}^{(2)}=(Y_{\bf 2}^{(1)}\otimes Y_{\bf 2}^{(1)})_{\bf
3}=(e^{i\pi/6}Y_{2}^{2},\sqrt{2}e^{i7\pi/12}Y_{1}Y_{2},Y_{1}^{2})^{t},$ (20)

and they form ${\bf 3}$ of $T^{\prime}$. We can obtain the higher weight
modular forms in a similar way.

The modular forms have $q$-expansion. $Y_{\bf 2}^{(1)}(\tau)$ is expanded as

$\displaystyle Y_{1}(\tau)$
$\displaystyle=\sqrt{2}e^{i7\pi/12}q^{1/3}(1+q+2q^{2}+2q^{4}+q^{5}+2q^{6}+\cdots),$
$\displaystyle Y_{2}(\tau)$
$\displaystyle=1/3+2q+2q^{3}+2q^{4}+2q^{7}+2q^{9}+\cdots.$ (21)

Thus $Y_{1}\ll Y_{2}$ for large ${\rm Im\,}\tau$. This is a general feature
for the modular forms which transform as ${\bf 2}$ of $T^{\prime}$. The matrix
representation of $\rho_{\bf 2}(T)$ are given by

$\displaystyle\rho_{\bf 2}(T)=\begin{pmatrix}\omega&0\\\
0&1\end{pmatrix},~{}~{}(\omega=e^{2\pi i/3}).$ (22)

It is clear that $q^{1/M}$ transforms under $T$ as

$\displaystyle q^{1/M}\rightarrow e^{2\pi i(\tau+1)/M}=e^{2\pi i/M}q^{1/M}.$
(23)

Thus the modular forms which transform as ${\bf 2}$ of $T^{\prime}$ must have
the $q$-expansion of the following form,

$\displaystyle Y_{\bf 2}^{(k)}=\begin{pmatrix}Y_{{\bf 2},1}^{(k)}\\\ Y_{{\bf
2},2}^{(k)}\end{pmatrix}=\begin{pmatrix}q^{1/3}\sum_{n\in\mathbb{N}}C_{n}^{(k)}q^{n}\\\
\sum_{n\in\mathbb{N}}D_{n}^{(k)}q^{n}\end{pmatrix},$ (24)

where $C_{n}^{(k)}$ and $D_{n}^{(k)}$ are coefficients independent of $\tau$.
Thus we can generally approximate the modular forms of ${\bf 2}$ as

$\displaystyle Y_{\bf 2}^{(k)}\sim(q^{1/3},1)^{T},$ (25)

for large ${\rm Im\,}\tau$. More precisely, the leading terms of the
$q$-expansions are not uniquely determined by the matrix representations of
$T$. They always have ambiguity of integer powers of $q$. Thus the modular
forms are approximated as

$\displaystyle Y_{{\bf 2},1}^{(k)}\sim q^{1/3+M_{1}},~{}~{}Y_{{\bf
2},2}^{(k)}\sim q^{M_{2}},$ (26)

where $M_{1}$ and $M_{2}$ are appropriate integer numbers. To determine the
correct hierarchy, we must calculate the explicit forms of the modular forms.
Nevertheless we obtain the hierarchical values in either case since the powers
of $q$ of the leading terms can not be the same. The matrix representation of
$T$ for other representations are given in Appendix A. The modular forms of
the other representations have $q$-expansion of

$\displaystyle Y_{\bf 1}^{(k)}$ $\displaystyle=\sum
C_{n}^{(k)}q^{n},~{}~{}Y_{\bf 1^{\prime}}^{(k)}=q^{1/3}\sum
C_{n}^{(k)}q^{n},~{}~{}Y_{\bf 1^{\prime\prime}}^{(k)}=q^{2/3}\sum
C_{n}^{(k)}q^{n},$ $\displaystyle Y_{\bf 2^{\prime}}^{(k)}$
$\displaystyle=\begin{pmatrix}q^{2/3}\sum C_{n}^{(k)}q^{n}\\\ q^{1/3}\sum
D_{n}^{(k)}q^{n}\end{pmatrix},~{}~{}Y_{\bf
2^{\prime\prime}}^{(k)}=\begin{pmatrix}\sum C_{n}^{(k)}q^{n}\\\ q^{2/3}\sum
D_{n}^{(k)}q^{n}\end{pmatrix},~{}~{}Y_{\bf 3}^{(k)}=\begin{pmatrix}\sum
C_{n}^{(k)}q^{n}\\\ q^{1/3}\sum D_{n}^{(k)}q^{n}\\\ q^{2/3}\sum
E_{n}^{(k)}q^{n}\end{pmatrix},$ (27)

and they are approximated by

$\displaystyle Y_{\bf 1}^{(k)}$ $\displaystyle\sim 1,~{}~{}Y_{\bf
1^{\prime}}^{(k)}\sim q^{1/3},~{}~{}Y_{\bf 1^{\prime\prime}}^{(k)}\sim
q^{2/3},$ $\displaystyle Y_{\bf 2^{\prime}}^{(k)}$
$\displaystyle\sim\begin{pmatrix}q^{2/3}\\\ q^{1/3}\end{pmatrix},~{}~{}Y_{\bf
2^{\prime\prime}}^{(k)}\sim\begin{pmatrix}1\\\
q^{2/3}\end{pmatrix},~{}~{}Y_{\bf 3}^{(k)}\sim\begin{pmatrix}1\\\ q^{1/3}\\\
q^{2/3}\end{pmatrix}.$ (28)

Therefore the modular forms naturally have large hierarchy for large Im
$\tau$.

This feature is general for the modular group of other levels. We summarize
the result of $\Gamma_{4}^{\prime}$ in Appendix B, and that of
$\Gamma_{5}^{\prime}$ in Appendix C.

###  2.2 Froggatt-Nielsen mechanism

The Froggatt-Nielsen (FN) mechanism is a well-known candidate solution for the
flavor puzzle. In FN mechanism, one introduces an extra Abelian gauge group
$U(1)_{FN}$, and assumes that the matter fields such as the left-handed quark
field $Q_{i}$ and the right-handed quark fields $u_{i}^{c}$, $d_{i}^{c}$ are
charged under $U(1)_{FN}$. We assume that Higgs fields $H_{u}$ and $H_{d}$ are
neutral under $U(1)_{FN}$. The Yukawa couplings are prohibited by $U(1)_{FN}$
unless the sum of $U(1)_{FN}$ charges of the corresponding fields are
canceled. We introduce an extra scalar field $\phi$ which has the $U(1)_{FN}$
charge of $-1$. We also assume that $\phi$ is the trivial singlet under the SM
gauge group. After the $U(1)_{FN}$ is spontaneously broken down at high energy
scale $\Lambda$, $\phi$ obtains vacuum expectation value $\braket{\phi}$ and
the effective superpotential for the quarks would be given by

$\displaystyle
W_{u}=\sum_{i,j}C^{(u)}_{ij}\tilde{\phi}^{f_{u_{i}}+f_{Q_{j}}}Q_{j}H_{u}u_{i}^{c},~{}~{}W_{d}=\sum_{i,j}C^{(d)}_{ij}\tilde{\phi}^{f_{d_{i}}+f_{Q_{j}}}Q_{j}H_{d}d_{i}^{c},$

where $\tilde{\phi}=\braket{\phi}/\Lambda$, and $f_{Q_{i}}$ and
$f_{u_{i},d_{i}}$ are the charge of the quarks respectively. $C^{(u)}_{ij}$
and $C^{(d)}_{ij}$ are free parameters supposed to be order 1. If
$\tilde{\phi}$ is sufficiently small (and the $U(1)_{FN}$ charges of the
quarks are large enough), effective Yukawa couplings much less than 1 are
obtained.

A typical example for the quark charges reproducing the observed mass
eigenvalues and the mixing angles is given by

$\displaystyle f_{Q_{1}}$ $\displaystyle=3,f_{Q_{2}}=2,f_{Q_{3}}=0,$
$\displaystyle f_{u_{1}^{c}}$
$\displaystyle=3,f_{u_{2}^{c}}=2,f_{u_{3}^{c}}=0,$ $\displaystyle
f_{d_{1}^{c}}$ $\displaystyle=2,f_{d_{2}^{c}}=1,f_{d_{3}^{c}}=1.$

In this case, Yukawa couplings of order 1 are prohibited except for the top
quark, and the light quark mass terms are given by nonrenormalizable higher
order terms. The quark mass matrix is given by [47]

$\displaystyle
M_{u}\sim\begin{pmatrix}\tilde{\phi}^{6}&\tilde{\phi}^{5}&\tilde{\phi}^{3}\\\
\tilde{\phi}^{5}&\tilde{\phi}^{4}&\tilde{\phi}^{2}\\\
\tilde{\phi}^{3}&\tilde{\phi}^{2}&\tilde{\phi}^{0}\end{pmatrix}\frac{v_{u}}{\sqrt{2}},~{}~{}M_{d}\sim\begin{pmatrix}\tilde{\phi}^{5}&\tilde{\phi}^{4}&\tilde{\phi}^{4}\\\
\tilde{\phi}^{4}&\tilde{\phi}^{3}&\tilde{\phi}^{3}\\\
\tilde{\phi}^{2}&\tilde{\phi}^{1}&\tilde{\phi}^{1}\end{pmatrix}\frac{v_{d}}{\sqrt{2}}.$
(29)

The left-handed quarks $Q_{i}$ are on the left side of the mass matrix, and
the right-handed quarks $u_{i}^{c},d_{i}^{c}$ are on the right side in our
notation [48]. The mass matrix is diagonalized by two unitary matrices
$V_{L}^{f}$ and $V_{R}^{f}$ as
$V_{L}^{f}M_{f}V_{R}^{f\dagger}=M_{f}^{diagonal}$, and the Cabbibo-Kobayashi-
Maskawa (CKM) matrix is given by $V_{CKM}=V_{L}^{u}V_{L}^{d\dagger}$. The mass
eigenvalues of (29) are approximated by
$m_{u}\sim\tilde{\phi}^{6}v_{u},m_{c}\sim\tilde{\phi}^{4}v_{u}$ and $m_{t}\sim
v_{u}$ for the up sector, and they are given as
$m_{d}\sim\tilde{\phi}^{5}v_{d},m_{s}\sim\tilde{\phi}^{3}v_{d}$ and
$m_{b}\sim\tilde{\phi}v_{u}$ for the down sector.

In terms of the mixing angles, we consider two-flavor model at first for
simplicity. Suppose that the mass matrix of the up and down quarks are given
by

$\displaystyle M_{u}=\begin{pmatrix}\epsilon_{1}\epsilon_{2}&\epsilon_{2}\\\
\epsilon_{1}&1\end{pmatrix},~{}~{}M_{d}=\begin{pmatrix}\delta_{1}\delta_{2}&\delta_{2}\\\
\delta_{1}&1\end{pmatrix},$

where $\epsilon_{i},\delta_{j}$ are small parameters. The eigenvalues of
$M_{u}$ are obtained by the following eigenvalue equations,

$\displaystyle\det(M_{u}M_{u}^{\dagger}-\lambda I)=0.$

We obtain

$\displaystyle\lambda=0,(1+\epsilon_{1}^{2})(1+\epsilon_{2}^{2}).$

The diagonalizing matrix is given by

$\displaystyle(V_{L}^{u})^{\dagger}=\begin{pmatrix}1&\epsilon_{2}\\\
-\epsilon_{2}&1\end{pmatrix}.$

The calculation for the down sector is completely parallel, and we obtain the
CKM matrix

$\displaystyle
V_{L}^{u}(V_{L}^{d})^{\dagger}=\begin{pmatrix}1-\epsilon_{2}\delta_{2}&\delta_{2}-\epsilon_{2}\\\
\epsilon_{2}-\delta_{2}&1+\epsilon_{2}\delta_{2}\end{pmatrix}$ (30)

Thus the mixing angle is approximated by the difference between the
$\epsilon_{2}$ and $\delta_{2}$. For three-flavor model, the mixing angles
$\theta_{IJ}$ is approximately given by the mass matrix focusing on the
corresponding two quarks, that is $\begin{pmatrix}M_{II}&M_{IJ}\\\
M_{JI}&M_{JJ}\end{pmatrix}$. Hence we obtain

$\displaystyle\theta_{12}\sim\frac{M_{12}}{M_{22}},~{}~{}\theta_{23}\sim\frac{M_{23}}{M_{33}},~{}~{}\theta_{13}\sim\frac{M_{13}}{M_{33}}.$
(31)

The approximated CKM matrix of (29) is written as

$\displaystyle V_{CKM}\sim\begin{pmatrix}1&\tilde{\phi}&\tilde{\phi}^{3}\\\
\tilde{\phi}&1&\tilde{\phi}^{2}\\\
\tilde{\phi}^{3}&\tilde{\phi}^{2}&1\end{pmatrix}.$ (32)

It is interesting that the observed values of CKM matrix are given by

$\displaystyle\theta_{12}\equiv\theta_{C}\sim 0.2,~{}~{}\theta_{23}\sim
0.03,~{}~{}\theta_{13}\sim 0.003,$ (33)

and the realistic mixing angles are realized if $\tilde{\phi}$ is equivalent
to the Cabbibo angle $\theta_{C}$.

##  3 Froggatt-Nielsen like mechanism with $\Gamma_{3}^{\prime}$

The FN-like mechanism based on the modular symmetry is first proposed in [43],
and similar idea is also proposed in [24]. In this model, the extra $U(1)$
symmetry is not required, but a similar role is played by the modular group.
In this section, we explain this idea. We also explain a difference between
our model and the previous ones.

We consider a modular symmetric models with $\Gamma_{3}^{\prime}=T^{\prime}$
at first. It is straightforward to generalize the idea to the modular group of
another level. Suppose that the quark fields
$Q_{i}=(U_{L}^{i},D_{L}^{i})^{t},u_{i}^{c},d_{i}^{c}$ have modular weight of
$k_{Q_{i},u_{j},d_{k}}$, and their representations under the finite modular
group are denoted by $\rho_{Q},\rho_{u},\rho_{d}$, respectively. We can set
the modular weights of the Higgs fields to zero without loss of generality
since they can be absorbed by shifting the modular weight of the other fields.
We also assume that the Higgs fields are the trivial singlet of the modular
group. The tree-level superpotential is given by

$\displaystyle
W_{u}=\sum(Y_{ij}^{(-k_{Q_{i}}-k_{u_{j}})}(\tau)Q_{i}u_{j}^{c})_{\bf
1}H_{u},~{}~{}W_{d}=\sum(Y_{ij}^{(-k_{Q_{i}}-k_{d_{j}})}(\tau)Q_{i}d_{j}^{c})_{\bf
1}H_{d},$ (34)

where the Yukawa couplings $Y_{ij}^{(-k_{Q_{i}}-k_{u_{j}})}$ and
$Y_{ij}^{(-k_{Q_{i}}-k_{d_{j}})}$ are the modular forms of weight
$-k_{Q_{i}}-k_{u_{j}}$ and $-k_{Q_{i}}-k_{d_{j}}$.

weight $k$ | representations
---|---
1 | 2
2 | 3
3 | ${\bf 2},{\bf 2^{\prime\prime}}$
4 | ${\bf 1,1^{\prime},3}$
5 | ${\bf 2,2^{\prime},2^{\prime\prime}}$
6 | ${\bf 1},2\times{\bf 3}$
⋮ | ⋮
Table 1: Irreducible decomposition of $\mathcal{M}_{k}(\Gamma(3))$.

Suppose $\rho_{Q}={\bf 3}$ and $\rho_{u}=\rho_{d}={\bf 1}$ of $T^{\prime}$,
and their modular weights are given by

$\displaystyle k_{Q_{1}}=k_{Q_{2}}=k_{Q_{3}}=k_{Q}=0,$ $\displaystyle
k_{u_{1}}=+1,~{}k_{u_{2}}=-3,~{}k_{u_{3}}=-6,$ $\displaystyle
k_{d_{1}}=0,~{}k_{d_{2}}=-3,~{}k_{d_{3}}=-5,$

where odd modular weights to the quark fields are allowed if we consider the
double covering group of the modular group. In this case, the Yukawa couplings
must be 3. We show the modular forms of level 3 and weight $k<7$ in Table 1.
The tree level couplings are prohibited except for $u_{3}^{c}$ because of the
absence of the triplet modular forms for the odd weights. Hence the modular
invariant superpotential is obtained as

$\displaystyle W_{u}=\left[\gamma_{u}\left((Y^{(6)}_{{\bf
3},I}+r_{u}Y^{(6)}_{{\bf 3},II})Q\right)_{\bf
1}u_{3}^{c}\right]H_{u},~{}~{}W_{d}=0,$ (35)

where $Y^{(6)}_{{\bf 3},I}$ and $Y^{(6)}_{{\bf 3},II}$ denote the two modular
forms of weight 6 which transform as the triplet under $T^{\prime}$.
$\gamma_{u}$ and $r_{u}$ are arbitrary coefficients supposed to be order 1.
The explicit form of $Y^{(6)}_{{\bf 3},I}$ and $Y^{(6)}_{{\bf 3},II}$ are
given by (99). $\gamma_{u}$ is real and $r_{u}$ is complex since the phase of
$\gamma_{u}$ can be absorbed by redefinition of $u_{3}^{c}$. This
superpotential would corresponds to the Yukawa couplings of the top quark. The
other couplings are given as non-renormalizable higher order couplings. We
introduce a new chiral superfield $\phi$ whose modular weight is $-1$. This
$\phi$ is called weighton since it carries the unit of modular weight [43]. We
assume that $\phi$ is the trivial singlet under both $T^{\prime}$ and the SM
gauge group. After breaking the modular symmetry, $\phi$ develops its vev, and
the effective superpotential should be given by

$\displaystyle W_{u}^{tri}$
$\displaystyle=\left[\alpha_{u}\tilde{\phi}^{3}(Y^{(2)}_{{\bf 3}}Q)_{\bf
1}u_{1}^{c}+\beta_{u}\tilde{\phi}(Y^{(4)}_{{\bf 3}}Q)_{\bf
1}u_{2}^{c}+\gamma_{u}((Y^{(6)}_{{\bf 3},I}+r_{u}Y^{(6)}_{{\bf 3},II})Q)_{\bf
1}u_{3}^{c}\right]H_{u},$ $\displaystyle~{}~{}W_{d}^{tri}$
$\displaystyle=\left[\alpha_{d}\tilde{\phi}^{2}(Y^{(2)}_{{\bf 3}}Q)_{\bf
1}d_{1}^{c}+\beta_{d}\tilde{\phi}(Y^{(4)}_{{\bf 3}}Q)_{\bf
1}d_{2}^{c}+\gamma_{d}\tilde{\phi}((Y^{(6)}_{{\bf 3},I}+r_{d}Y^{(6)}_{{\bf
3},II})Q)_{\bf 1}d_{3}^{c}\right]H_{d},$ (36)

where $\tilde{\phi}$ denotes $\braket{\phi}/\Lambda$ and $\Lambda$ is the
cutoff scale. $\alpha_{f},\beta_{f},\gamma_{f}$ and $r_{f}$, where $f$ denotes
$u$ or $d$, are order 1 parameters. The superscript $tri$ explicitly indicates
that the left-handed quarks form the triplet. In our basis, the irreducible
decomposition of ${\bf 3}\otimes{\bf 3}$ is given by777The Clebsch-Gordon (CG)
coefficients of $T^{\prime}$ in our notation are summarized in Appendix A.

$\displaystyle\begin{pmatrix}Y^{(k)}_{1}\\\ Y^{(k)}_{2}\\\ Y^{(k)}_{3}\\\
\end{pmatrix}_{\bf 3}\otimes\begin{pmatrix}Q_{1}\\\ Q_{2}\\\ Q_{3}\\\
\end{pmatrix}_{\bf
3}=\left(Y^{(k)}_{1}Q_{1}+Y^{(k)}_{2}Q_{3}+Y^{(k)}_{3}Q_{2}\right)_{\bf
1}+\cdots.$ (37)

If Im $\tau$ is large, the modular forms $Y^{(k)}_{1,2,3}$ are approximated by
$q^{0,1/3,2/3}$ respectively (see (27)). Then the superpotential is
approximated for large Im $\tau$ as

$\displaystyle W_{u}^{tri}\sim\Big{[}$
$\displaystyle\alpha_{u}\tilde{\phi}^{3}(Q_{1}+q^{1/3}Q_{3}+q^{2/3}Q_{2})u_{1}^{c}+\beta_{u}\tilde{\phi}(Q_{1}+q^{1/3}Q_{3}+q^{2/3}Q_{2})u_{2}^{c}$
$\displaystyle+\gamma_{u}(Q_{1}+q^{1/3}Q_{3}+q^{2/3}Q_{2})u_{3}^{c}\Big{]}H_{u},$
$\displaystyle~{}~{}W_{d}^{tri}\sim\Big{[}$
$\displaystyle\alpha_{d}\tilde{\phi}^{2}(Q_{1}+q^{1/3}Q_{3}+q^{2/3}Q_{2})d_{1}^{c}+\beta_{d}\tilde{\phi}(Q_{1}+q^{1/3}Q_{3}+q^{2/3}Q_{2})d_{2}^{c}$
$\displaystyle+\gamma_{d}\tilde{\phi}(Q_{1}+q^{1/3}Q_{3}+q^{2/3}Q_{2})d_{3}^{c}\Big{]}H_{d},$
(38)

and the mass matrix is approximated by

$\displaystyle M^{tri}_{u}\sim$
$\displaystyle\begin{pmatrix}\tilde{\phi}^{3}&\tilde{\phi}^{1}&\tilde{\phi}^{0}\\\
\tilde{\phi}^{3}q^{2/3}&\tilde{\phi}^{1}q^{2/3}&\tilde{\phi}^{0}q^{2/3}\\\
\tilde{\phi}^{3}q^{1/3}&\tilde{\phi}^{1}q^{1/3}&\tilde{\phi}^{0}q^{1/3}\end{pmatrix}\frac{v_{u}}{\sqrt{2}},~{}~{}M^{tri}_{d}\sim\begin{pmatrix}\tilde{\phi}^{2}&\tilde{\phi}^{1}&\tilde{\phi}^{1}\\\
\tilde{\phi}^{2}q^{2/3}&\tilde{\phi}^{1}q^{2/3}&\tilde{\phi}^{1}q^{2/3}\\\
\tilde{\phi}^{2}q^{1/3}&\tilde{\phi}^{1}q^{1/3}&\tilde{\phi}^{1}q^{1/3}\end{pmatrix}\frac{v_{d}}{\sqrt{2}}.$
(39)

We can always exchange the indices of the quark fields freely. We redefine the
left-handed quark fields $Q_{i}$ as

$\displaystyle Q_{1}\rightarrow Q_{3},~{}~{}Q_{2}\rightarrow
Q_{1},~{}~{}Q_{3}\rightarrow Q_{2},$ (40)

and the mass matrix is rewritten as

$\displaystyle
M_{u}^{tri}\sim\begin{pmatrix}\tilde{\phi}^{3}q^{2/3}&\tilde{\phi}^{1}q^{2/3}&\tilde{\phi}^{0}q^{2/3}\\\
\tilde{\phi}^{3}q^{1/3}&\tilde{\phi}^{1}q^{1/3}&\tilde{\phi}^{0}q^{1/3}\\\
\tilde{\phi}^{3}&\tilde{\phi}^{1}&\tilde{\phi}^{0}\end{pmatrix}\frac{v_{u}}{\sqrt{2}},~{}~{}M_{d}^{tri}\sim\begin{pmatrix}\tilde{\phi}^{2}q^{2/3}&\tilde{\phi}^{1}q^{2/3}&\tilde{\phi}^{1}q^{2/3}\\\
\tilde{\phi}^{2}q^{1/3}&\tilde{\phi}^{1}q^{1/3}&\tilde{\phi}^{1}q^{1/3}\\\
\tilde{\phi}^{2}&\tilde{\phi}^{1}&\tilde{\phi}^{1}\end{pmatrix}\frac{v_{d}}{\sqrt{2}}.$
(41)

The determinants of these two matrices are proportional to
$\tilde{\phi}^{4}q$, and the largest eigenvalue is of order of
$\tilde{\phi}^{0}q^{0}$. Thus we have a natural hierarchy. It is the same as
the FN mechanism (29). If $q^{1/3}\sim\tilde{\phi}^{3/2}$, we obtain the FN-
like mass matrix. Thus we expect that these mass matrices reproduce the mass
hierarchies and the mixing angles of the quarks.

Note that the right-handed quarks $u_{i}^{c}$ (and $d_{i}^{c}$) must be the
same representation to realize FN-like mass matrix. For example, suppose
$u_{2}^{c}$ is assigned to ${\bf 1^{\prime\prime}}$, and $u_{3}^{c}$ to ${\bf
1^{\prime}}$, and the other right-handed quarks are the trivial singlet 1. In
this case, the effective superpotential of the up sector is changed to

$\displaystyle W_{u}^{\prime}$
$\displaystyle=\left[\alpha_{u}\tilde{\phi}^{3}(Y^{(2)}_{{\bf 3}}Q)_{{\bf
1}}u_{1}^{c}+\beta_{u}\tilde{\phi}(Y^{(4)}_{{\bf 3}}Q)_{\bf
1^{\prime}}u_{2}^{c}+\gamma_{u}((Y^{(6)}_{{\bf 3},I}+r_{u}Y^{(6)}_{{\bf
3},II})Q)_{\bf 1^{\prime\prime}}u_{3}^{c}\right]H_{u}.$ (42)

The CG coefficients in terms of $({\bf 3}\times{\bf 3})_{\bf
1^{\prime\prime}}$, and $({\bf 3}\times{\bf 3})_{\bf 1^{\prime}}$ are
summarized in (96). The mass matrix is approximated by

$\displaystyle
M_{u}^{\prime}\sim\begin{pmatrix}\tilde{\phi}^{3}&\tilde{\phi}^{1}q^{1/3}&\tilde{\phi}^{0}q^{2/3}\\\
\tilde{\phi}^{3}q^{2/3}&\tilde{\phi}^{1}&\tilde{\phi}^{0}q^{1/3}\\\
\tilde{\phi}^{3}q^{1/3}&\tilde{\phi}^{1}q^{2/3}&\tilde{\phi}^{0}\end{pmatrix}\frac{v_{u}}{\sqrt{2}}.$
(43)

The eigenvalues of (43) are approximated by
$\tilde{\phi}^{3}v_{u},\tilde{\phi}^{1}v_{u}$, and $\tilde{\phi}^{0}v_{u}$ in
the limit of ${\rm Im\,}\tau\rightarrow+\infty$. It may reproduce the small
values of the mixing angles since it is close to diagonal matrix. Indeed, in
the previous modular symmetric models, the mass matrices are the same form as
(43). In this case, however, the source of the mass hierarchy and that of the
Cabbibo angles are independent each other [43]. On the other hand in the case
of the “FN-like” mass matrices in (41) they are related each other through the
moduli parameters and the singlet vev. As it will be discussed later, the
empirical relations between mass ratios and the mixing angles in (31) can also
be realized with $\mathcal{O}(1)$ coefficients. For our purpose it is
important to assume that the right-handed quarks are the same representation
under modular symmetry.

To construct a mass matrix similar to (41), we have a restriction on the
modular weights too. Since the components of the modular forms are aligned in
the same order at each row in our model, the weight of the Yukawa couplings of
each generation must be different to realize the full rank matrix. The modular
forms of weight higher than 5 are required at least. The modular forms of
modular weight higher than 5 is also required for the CP-phase. The complex
phases of the modular forms do not affect the CP-phase for large ${\rm
Im\,}\tau$, since the mass matrix is approximated by (41), and the phase of
$q$, i.e., ${\rm Re\,}\tau$, can be absorbed by field redefinition of $Q_{i}$.
The phase of coefficients $\alpha_{f},\beta_{f}$ and $\gamma_{f}$, as well as
that of $\phi$ are also absorbed by $u_{i}$ and $d_{j}$. Hence $r_{u}$ and
$r_{d}$ in (36) are the only source of CP-violation in our model.888More
generally, ${\rm Im\,}\tau=+\infty$ is a point invariant under
$\tau\rightarrow-\tau^{*}$, which is the generalized CP-transformation for the
modular symmetry [62, 63]. Thus the superpotential must include explicit
breaking term for large ${\rm Im\,}\tau$. Such a CP-phase appears if
$\mathcal{M}_{k}(\Gamma(3))$ has multiple triplets. It is satisfied when the
modular weight is higher than 5.

We also comment on the sub-leading terms. Suppose that the leading Yukawa term
is given by

$\displaystyle W\supset\tilde{\phi}^{k}(Y^{(\ell)}_{\bf 3}Q)_{\bf
1}f^{c}_{j}H_{f},~{}~{}~{}~{}f=u~{}{\rm or}~{}d,$ (44)

then we have sub-leading couplings,

$\displaystyle\Delta W=\alpha_{f}^{\prime}\tilde{\phi}^{k+2}(Y_{\bf
3}^{(\ell+2)}Q)_{\bf 1}f^{c}_{j}H_{f}.$ (45)

The sub-leading term is suppressed by $\tilde{\phi}^{2}$ compared to the
leading term because the weight of the modular forms which is 3 of
$T^{\prime}$ must be even and positive. As shown in the following analysis,
$\tilde{\phi}$ is of order $10^{-2}$ for the realistic models, and we can omit
the sub-leading terms. This also implies that the smallest value of $k$ (the
power of $\tilde{\phi}$ in the leading Yukawa term) should be lower than 2 in
general. The only exception is the Yukawa term in which the modular weight of
the Yukawa coupling is the lowest, that is, $\ell=2$ in the case of $Y_{\bf
3}^{(\ell)}$. In this case it is possible to assign an arbitrary positive
power of $\tilde{\phi}$ at the leading order, since there is no modular form
with lower modular weight $\ell<2$ for $Y_{\bf 3}^{(\ell)}$ (See Table 1).

Finally, we obtain the general superpotential of our FN-like model with
$\Gamma_{3}^{\prime}$:

$\displaystyle W_{u}^{tri}$
$\displaystyle=\left[\alpha_{u}\tilde{\phi}^{I}(QY^{(k_{1})}_{\bf 3})_{\bf
1}u_{1}^{c}+\beta_{u}\tilde{\phi}^{J}(QY^{(k_{2})}_{\bf 3})_{\bf
1}u_{2}^{c}+\gamma_{u}\tilde{\phi}^{K}(QY^{(k_{3})}_{\bf 3})_{\bf
1}u_{3}^{c}\right]H_{u},$ $\displaystyle W_{d}^{tri}$
$\displaystyle=\left[\alpha_{d}\tilde{\phi}^{L}(QY^{(\ell_{1})}_{\bf 3})_{\bf
1}d_{1}^{c}+\beta_{d}\tilde{\phi}^{M}(QY^{(\ell_{2})}_{\bf 3})_{\bf
1}d_{2}^{c}+\gamma_{d}\tilde{\phi}^{N}(QY^{(\ell_{3})}_{\bf 3})_{\bf
1}d_{3}^{c}\right]H_{d},$ (46)

where $I,J,K,L,M,N$ and $k_{i},\ell_{i}$ are integer numbers satisfying the
following conditions:

$\displaystyle-I+k_{Q}+k_{1}+k_{u_{1}}$ $\displaystyle=0,$
$\displaystyle-J+k_{Q}+k_{2}+k_{u_{2}}$ $\displaystyle=0,$
$\displaystyle-K+k_{Q}+k_{3}+k_{u_{3}}$ $\displaystyle=0,$
$\displaystyle-L+k_{Q}+\ell_{1}+k_{d_{1}}$ $\displaystyle=0,$
$\displaystyle-M+k_{Q}+\ell_{2}+k_{d_{2}}$ $\displaystyle=0,$
$\displaystyle-N+k_{Q}+\ell_{3}+k_{d_{3}}$ $\displaystyle=0.$ (47)

As mentioned above, if $k_{i}$ or $\ell_{i}$ is equal to 2, the corresponding
capital index can be arbitrary integer number, otherwise the capital indices
must be 0 or 1. $(k_{1},k_{2},k_{3})=(\ell_{1},\ell_{2},\ell_{3})=(2,4,6)$ is
the smallest weight full rank model, which has the smallest number of free
parameters because the dimension of $\mathcal{M}_{k}(\Gamma(3))$ monotonically
increases as the weight $k$ increases.

###  3.1 Models with the singlet left-handed quarks

We construct a similar model by exchanging the representations of $Q_{i}$ and
$(u_{i}^{c},d_{i}^{c})$. Suppose that $Q_{i}$ are the trivial singlet of
$T^{\prime}$, and $u_{i}^{c}$ and $d_{i}^{c}$ form the triplet of
$T^{\prime}$, then we obtain the following superpotential:

$\displaystyle W_{u}^{sing}$
$\displaystyle=\left[\alpha_{u}\tilde{\phi}^{I}Q_{1}(Y^{(k_{1})}_{\bf
3}u^{c})_{\bf 1}+\beta_{u}\tilde{\phi}^{J}Q_{2}(Y^{(k_{2})}_{\bf 3}u^{c})_{\bf
1}+\gamma_{u}\tilde{\phi}^{K}Q_{3}(Y^{(k_{3})}_{\bf 3}u^{c})_{\bf
1}\right]H_{u},$ $\displaystyle W_{d}^{sing}$
$\displaystyle=\left[\alpha_{d}\tilde{\phi}^{L}Q_{1}(Y^{(\ell_{1})}_{\bf
3}d^{c})_{\bf 1}+\beta_{d}\tilde{\phi}^{M}Q_{2}(Y^{(\ell_{2})}_{\bf
3}d^{c})_{\bf 1}+\gamma_{d}\tilde{\phi}^{N}Q_{3}(Y^{(\ell_{3})}_{\bf
3}d^{c})_{\bf 1}\right]H_{d},$ (48)

where $u^{c}=(u_{1}^{c},u_{2}^{c},u_{3}^{c})^{t}$ and
$d^{c}=(d_{1}^{c},d_{2}^{c},d_{3}^{c})^{t}$, which form 3 of $T^{\prime}$. In
this case, the powers of $\tilde{\phi}$ and the modular weights of the fields
satisfy the following conditions,

$\displaystyle-I+k_{Q_{1}}+k_{1}+k_{u}$ $\displaystyle=0,$
$\displaystyle-J+k_{Q_{2}}+k_{2}+k_{u}$ $\displaystyle=0,$
$\displaystyle-K+k_{Q_{3}}+k_{3}+k_{u}$ $\displaystyle=0,$
$\displaystyle-L+k_{Q_{1}}+\ell_{1}+k_{d}$ $\displaystyle=0,$
$\displaystyle-M+k_{Q_{2}}+\ell_{2}+k_{d}$ $\displaystyle=0,$
$\displaystyle-N+k_{Q_{2}}+\ell_{3}+k_{d}$ $\displaystyle=0.$ (49)

The mass matrices are approximated as

$\displaystyle
M_{u}^{sing}\sim\begin{pmatrix}\tilde{\phi}^{I}q^{2/3}&\tilde{\phi}^{I}q^{1/3}&\tilde{\phi}^{I}\\\
\tilde{\phi}^{J}q^{2/3}&\tilde{\phi}^{J}q^{1/3}&\tilde{\phi}^{J}\\\
\tilde{\phi}^{K}q^{2/3}&\tilde{\phi}^{K}q^{1/3}&\tilde{\phi}^{K}\end{pmatrix}\frac{v_{u}}{\sqrt{2}},~{}~{}M_{d}^{sing}\sim\begin{pmatrix}\tilde{\phi}^{L}q^{2/3}&\tilde{\phi}^{L}q^{1/3}&\tilde{\phi}^{L}\\\
\tilde{\phi}^{M}q^{2/3}&\tilde{\phi}^{M}q^{1/3}&\tilde{\phi}^{M}\\\
\tilde{\phi}^{N}q^{2/3}&\tilde{\phi}^{N}q^{1/3}&\tilde{\phi}^{N}\end{pmatrix}\frac{v_{d}}{\sqrt{2}}.$
(50)

We also obtain the hierarchical mass matrix. In fact, this mass matrix is the
transposed matrix of the previous one in (41).

$(k_{1},k_{2},k_{3})=(\ell_{1},\ell_{2},\ell_{3})=(2,4,6)$ is the model with
the lowest weight modular forms. In this case, the constraints (49) implies

$\displaystyle I-L=J-M=K-N=-k_{d}+k_{u}.$ (51)

###  3.2 Numerical analysis of mass ratios and the mixing angles of
$\Gamma_{3}^{\prime}$ models

The origin of the modular symmetry is the geometrical symmetry of the extra
dimensions. Hence we should evaluate the Yukawa couplings at the
compactification scale. We assume that the compactification scale is the GUT
scale ($2\times 10^{16}$ GeV). The Yukawa couplings at high energy scale
receives quantum corrections, and they are given by solving the
renormalization group equation. They depend on the physics beyond the standard
model. In this paper, we assume a minimal SUSY breaking scenario with
$\tan\beta=5$ [64, 65]. At the GUT scale, the Yukawa couplings are calculated
as

$\displaystyle y_{u}^{obs}$ $\displaystyle=(2.92\pm 1.81)\times
10^{-6},~{}~{}y_{c}^{obs}=(1.43\pm 0.100)\times
10^{-3},~{}~{}y_{t}^{obs}=0.534\pm 0.0341.$ $\displaystyle y_{d}^{obs}$
$\displaystyle=(4.81\pm 1.06)\times 10^{-6},~{}~{}y_{s}^{obs}=(9.52\pm
1.03)\times 10^{-5},~{}~{}y_{b}^{obs}=(6.95\pm 0.175)\times 10^{-3},$

We explicitly show $1\sigma$ interval for every observable. In the following
analysis, we concentrate on the ratios of the Yukawa couplings rather than the
Yukawa couplings themselves, since the overall factor is irrelevant to our
study. The ratios of the Yukawa couplings are calculated as

$\displaystyle y_{u}^{obs}/y_{t}^{obs}$ $\displaystyle=(5.47\pm 3.41)\times
10^{-6},~{}~{}y_{c}^{obs}/y_{t}^{obs}=(2.68\pm 0.254)\times 10^{-3},~{}~{}$
$\displaystyle y_{d}^{obs}/y_{b}^{obs}$ $\displaystyle=(6.92\pm 1.54)\times
10^{-4},~{}~{}y_{s}^{obs}/y_{b}^{obs}=(1.37\pm 0.152)\times 10^{-2},$
$\displaystyle y_{b}^{obs}/y_{t}^{obs}$ $\displaystyle=(1.30\pm 0.0893)\times
10^{-2}.$ (52)

Similarly, the mixing angles and $CP$-phase consistent with the experimental
results at the GUT scale are given by

$\displaystyle\theta_{12}^{obs}$ $\displaystyle=0.22736\pm
0.00142,~{}~{}\theta_{23}^{obs}=0.03585\pm 0.00670,$
$\displaystyle\theta_{13}^{obs}$ $\displaystyle=0.003145\pm
0.000490,~{}~{}\delta_{CP}^{obs}=1.206\pm 0.108.$

Our notation of the mixing angles and the CP-phase is based on the PDG [48].
The quark sector has 9 observables to fit.

In this section, we analyze the mass hierarchy and the mixing angles of our
FN-like model. The superpotential with $\Gamma_{3}^{\prime}$ are summarized in
(46) and (48). We consider the superpotential with the lowest weight modular
forms.

#### Model with triplet left-handed quarks

First we consider the FN-like model based on the superpotential of (46).
Before investigating the numerical analysis, we should study the structure of
the mass matrix analytically. The physical mass matrix is given in terms of
the canonically normalized Yukawa couplings in (15) as

$\displaystyle M_{u}=$
$\displaystyle\begin{pmatrix}\alpha_{u}\tilde{\phi}^{I}Y_{3}^{(2)}&\beta_{u}\tilde{\phi}^{J}Y_{3}^{(4)}&\gamma_{u}\tilde{\phi}^{K}(Y_{3,I}^{(6)}+r_{u}Y_{3,II}^{(6)})\\\
\alpha_{u}\tilde{\phi}^{I}Y_{2}^{(2)}&\beta_{u}\tilde{\phi}^{J}Y_{2}^{(4)}&\gamma_{u}\tilde{\phi}^{K}(Y_{2,I}^{(6)}+r_{u}Y_{2,II}^{(6)})\\\
\alpha_{u}\tilde{\phi}^{I}Y_{1}^{(2)}&\beta_{u}\tilde{\phi}^{J}Y_{1}^{(4)}&\gamma_{u}\tilde{\phi}^{K}(Y_{1,I}^{(6)}+r_{u}Y_{1,II}^{(6)})\\\
\end{pmatrix}\begin{pmatrix}{\rm Im\,}\tau^{1}&&\\\ &{\rm Im\,}\tau^{2}&\\\
&&{\rm Im\,}\tau^{3}\end{pmatrix}\frac{v_{u}}{\sqrt{2}},$ $\displaystyle
M_{d}=$
$\displaystyle\begin{pmatrix}\alpha_{d}\tilde{\phi}^{L}Y_{3}^{(2)}&\beta_{d}\tilde{\phi}^{M}(Y_{3,I}^{(6)}+r_{d}Y_{3,II}^{(6)})&\gamma_{d}\tilde{\phi}^{N}Y_{3}^{(4)}\\\
\alpha_{d}\tilde{\phi}^{L}Y_{2}^{(2)}&\beta_{d}\tilde{\phi}^{M}(Y_{2,I}^{(6)}+r_{d}Y_{2,II}^{(6)})&\gamma_{d}\tilde{\phi}^{N}Y_{2}^{(4)}\\\
\alpha_{d}\tilde{\phi}^{L}Y_{1}^{(2)}&\beta_{d}\tilde{\phi}^{M}(Y_{1,I}^{(6)}+r_{d}Y_{1,II}^{(6)})&\gamma_{d}\tilde{\phi}^{N}Y_{1}^{(4)}\\\
\end{pmatrix}\begin{pmatrix}{\rm Im\,}\tau^{1}&&\\\ &{\rm Im\,}\tau^{3}&\\\
&&{\rm Im\,}\tau^{2}\end{pmatrix}\frac{v_{d}}{\sqrt{2}},$ (53)

where we note that in order to clearly see a FN-like hierarchy we redefine the
indices of the quark fields as

$\displaystyle Q_{1}\rightarrow Q_{3},~{}~{}Q_{2}\rightarrow
Q_{1},~{}~{}Q_{3}\rightarrow Q_{2},~{}~{}d_{2}\leftrightarrow d_{3},$ (54)

where we implicitly assume the following conditions999We can consider other
possibilities with different powers. We find the model in (53) with the
conditions (55) the best in our numerical analysis.

$\displaystyle I\geq J\geq K,\quad$ $\displaystyle\quad L\geq M\geq N.$ (55)

Thus the largest Yukawa coupling for the down sector would be
$\gamma_{d}\tilde{\phi}^{N}Y_{1}^{(4)}$, while that for the up sector
$\gamma_{u}\tilde{\phi}^{K}(Y^{(6)}_{1,I}+r_{u}Y^{(6)}_{1,II})$. Using
explicit $q$-expansions in Appendix A we obtain an approximate estimation of
the mixing angles for large ${\rm Im\,}\tau$ as

$\displaystyle\theta_{12}$
$\displaystyle\sim\left|\frac{Y^{(4)}_{3}}{Y^{(4)}_{2}}-\frac{Y^{(6)}_{3,I}+r_{d}Y^{(6)}_{3,II}}{Y^{(6)}_{2,I}+r_{d}Y^{(6)}_{2,II}}\right|\sim
15\left|\frac{r_{d}+\frac{1}{5}}{r_{d}+\frac{1}{2}}q^{1/3}\right|,$
$\displaystyle\theta_{23}$
$\displaystyle\sim\left|\frac{Y^{(6)}_{2,I}+r_{u}Y^{(6)}_{2,II}}{Y^{(6)}_{1,I}+r_{u}Y^{(6)}_{1,II}}-\frac{Y^{(4)}_{2}}{Y^{(4)}_{1}}\right|\sim
12\left|(1+r_{u})q^{1/3}\right|,~{}~{}$ $\displaystyle\theta_{13}$
$\displaystyle\sim\left|\frac{Y^{(6)}_{3,I}+r_{u}Y^{(6)}_{3,II}}{Y^{(6)}_{1,I}+r_{u}Y^{(6)}_{1,II}}-\frac{Y^{(4)}_{3}}{Y^{(4)}_{1}}\right|\sim
72\left|(1-r_{u})q^{2/3}\right|.$ (56)

We see that the approximate mixing angles do not depend on $\tilde{\phi}$,
$\alpha_{u,d},\beta_{u,d}$, and $\gamma_{u,d}$. The first relation implies
$15|q^{1/3}|\sim\theta_{C}\sim 10^{-1}$, which is satisfied when ${\rm
Im\,}\tau\sim 2.4$. The other two conditions are rewritten as

$\displaystyle\theta_{23}$
$\displaystyle\sim\frac{4|1+r_{u}|}{5}\theta_{C}\sim 8|1+r_{u}|\times
10^{-2},$ $\displaystyle\theta_{13}$
$\displaystyle\sim\frac{8|1-r_{u}|}{25}\theta_{C}^{2}\sim 3|1-r_{u}|\times
10^{-3}.$

Then we can realize the observed values for $r_{u}=\mathcal{O}(1)$. We also
obtain a natural hierarchical structure for the mass ratios which are
suppressed by powers of $\theta_{C}$ and $\tilde{\phi}$ as

$\displaystyle y_{u}/y_{t}$ $\displaystyle\sim\frac{162}{25}{\rm
Im\,}\tau^{-2}\frac{\alpha_{u}}{\gamma_{u}}\tilde{\phi}^{I-K}\theta_{C}^{2}\sim
1\times 10^{-2}\frac{\alpha_{u}}{\gamma_{u}}\tilde{\phi}^{I-K},$
$\displaystyle y_{c}/y_{t}$ $\displaystyle\sim\frac{18}{5}{\rm
Im\,}\tau^{-1}\frac{\beta_{u}}{\gamma_{u}}\tilde{\phi}^{J-K}\theta_{C}\sim
2\times 10^{-1}\frac{\beta_{u}}{\gamma_{u}}\tilde{\phi}^{J-K},$ (57)

for the up sector, and

$\displaystyle y_{d}/y_{b}$ $\displaystyle\sim\frac{162}{225}{\rm
Im\,}\tau^{-1}\frac{\alpha_{d}}{\gamma_{d}}\tilde{\phi}^{L-N}\theta_{C}^{2}\sim
3\times 10^{-3}\frac{\alpha_{d}}{\gamma_{d}}\tilde{\phi}^{L-N},$
$\displaystyle y_{s}/y_{b}$
$\displaystyle\sim\frac{8}{9}\frac{|r_{d}+\frac{1}{2}|\beta_{d}}{\gamma_{d}}\tilde{\phi}^{M-N}\theta_{C}\sim
2\times
10^{-2}\frac{|r_{d}+\frac{1}{2}|\beta_{d}}{\gamma_{d}}\tilde{\phi}^{M-N},~{}~{}$
$\displaystyle y_{b}/y_{t}$ $\displaystyle\sim 9{\rm
Im\,}\tau^{-1}\frac{\gamma_{d}}{\gamma_{u}}\tilde{\phi}^{N-K}\sim
4\frac{\gamma_{d}}{\gamma_{u}}\tilde{\phi}^{N-K},$ (58)

for the down sector, where we use ${\rm Im\,}\tau\sim 2.4$. We solve these
equations under the conditions (55). For simplicity we assume
$\tilde{\phi}=10^{-2}$ for the later estimation. We then find a solution of
$I=2,J=1,K=0,M=1,N=1$, and $L=1$ or 2, which can reproduce the observed mass
ratios in (52) with $\mathcal{O}(1)$ coefficients of
$\frac{\alpha_{u,d}}{\gamma_{u,d}},\frac{\beta_{u,d}}{\gamma_{u,d}},\frac{\gamma_{d}}{\gamma_{u}}$,
and $r_{u,d}$.

To confirm our analysis we construct an explicit model which satisfies the
above conditions. The representations and modular weights of the quark fields
are summarized in Table 2. The mass matrix is given by (53) with
$I=2,J=1,K=0,$ and $L=2,M=N=1$.

| $Q_{1,2,3}$ | $u_{1}^{c}$ | $u_{2}^{c}$ | $u_{2}^{c}$ | $d_{1}^{c}$ | $d_{2}^{c}$ | $d_{3}^{c}$ | $H_{u}$ | $H_{d}$ | $\phi$
---|---|---|---|---|---|---|---|---|---|---
$\Gamma_{3}^{\prime}=T^{\prime}$ | ${\bf 3}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$
$k_{I}$ | $-2$ | $2$ | $-1$ | $-4$ | $2$ | $-3$ | $-1$ | $0$ | $0$ | $-1$
Table 2: The representations and the modular weights of the chiral
superfields.

We set ${\rm Re\,}\tau=0$ since the complex phase factor in the modular forms
is negligibly small for large ${\rm Im\,}\tau$. We also assume absolute value
of $r_{f}$ is 1 at first. Thus we have 8 free parameters:
$\alpha_{u}/\gamma_{u}$, $\beta_{u}/\gamma_{u}$, $r_{u}$,
$\alpha_{d}/\gamma_{d}$, $\beta_{d}/\gamma_{d}$, $r_{d}$, ${\rm Im\,}\tau$ and
$\gamma_{d}/\gamma_{u}$. $\tilde{\phi}$ is not counted as a d.o.f since it is
absorbed by the coefficients. The best fit parameters in our search are given
by

$\displaystyle\alpha_{u}/\gamma_{u}=$ $\displaystyle
0.1904,~{}~{}\beta_{u}/\gamma_{u}=2.427,~{}~{}r_{u}=e^{-2.764i},$
$\displaystyle\alpha_{d}/\gamma_{d}=$ $\displaystyle
4.946,~{}~{}\beta_{d}/\gamma_{d}=0.5614,~{}~{}r_{d}=e^{2.462i},$
$\displaystyle\gamma_{d}/\gamma_{u}=$ $\displaystyle
0.3400,~{}~{}\tau=2.406i,~{}~{}\tilde{\phi}=1\times 10^{-2}.$ (59)

The most hierarchical parameter is $\beta_{u}/\alpha_{u}=12.7$ in this
parameter set. The FN-like mass matrices are successfully obtained as,

$\displaystyle|M_{u}|=$ $\displaystyle\gamma_{u}\begin{pmatrix}3.846\times
10^{-9}&3.932\times 10^{-6}&4.464\times 10^{-5}\\\ 1.979\times
10^{-7}&6.743\times 10^{-5}&2.193\times 10^{-3}\\\ 5.090\times
10^{-6}&1.734\times 10^{-3}&1.911\times 10^{-2}\\\
\end{pmatrix}\frac{v_{u}}{\sqrt{2}},$ $\displaystyle|M_{d}|=$
$\displaystyle\gamma_{d}\begin{pmatrix}9.991\times 10^{-8}&2.661\times
10^{-7}&1.620\times 10^{-6}\\\ 5.140\times 10^{-6}&1.188\times
10^{-5}&2.778\times 10^{-5}\\\ 1.322\times 10^{-4}&1.073\times
10^{-4}&7.147\times 10^{-4}\\\ \end{pmatrix}\frac{v_{d}}{\sqrt{2}}.$ (60)

As the result, we obtain the following mass eigenvalues and mixing angles,

$\displaystyle y_{u}/y_{t}$ $\displaystyle=4.19\times
10^{-6},~{}~{}y_{c}/y_{t}=2.67\times 10^{-3},~{}~{}y_{d}/y_{b}=7.69\times
10^{-4},~{}~{}y_{s}/y_{b}=1.56\times 10^{-2},$ $\displaystyle y_{b}/y_{t}$
$\displaystyle=1.30\times
10^{-2},~{}~{}\theta_{12}=0.227,~{}~{}\theta_{23}=0.0300,~{}~{}\theta_{13}=0.00074,~{}~{}\delta_{CP}=1.20.$
(61)

In this case, though we have only 8 free parameters, all mixing angles and
mass eigenvalues except for the mixing angles $\theta_{13}$ are reproduced
within $2\sigma$ range of the observed values. In this case, $\chi^{2}$, which
is given by

$\displaystyle\chi^{2}=\sum_{x\in
Observables}\left(\frac{x-x^{obs}}{\sigma_{x}}\right)^{2},$ (62)

where
$Observables=\\{y_{u}/y_{t},y_{c}/y_{t},y_{d}/y_{b},y_{s}/y_{b},y_{b}/y_{t},\sigma_{12},\sigma_{23},\sigma_{13},\delta_{CP}\\}$,
is estimated as $\chi^{2}\sim 27$.

If we relax the restriction on $|r|$, we can realize the observed values more
precisely, while the number of free parameters is more than observed values. A
benchmark value is obtained as

$\displaystyle\alpha_{u}/\gamma_{u}=$ $\displaystyle
0.3320,~{}~{}\beta_{u}/\gamma_{u}=2.159,~{}~{}r_{u}=0.8888e^{-2.649i},$
$\displaystyle\alpha_{d}/\gamma_{d}=$ $\displaystyle
2.976,~{}~{}\beta_{d}/\gamma_{d}=3.665,~{}~{}r_{d}=0.6941e^{-3.375i},$
$\displaystyle\gamma_{d}/\gamma_{u}=$ $\displaystyle
0.2509,~{}~{}\tau=2.448i,~{}~{}\tilde{\phi}=1\times 10^{-2}.$

The most hierarchical parameter is $\beta_{u}/\gamma_{d}=8.61$ in this
parameter set, and all the coefficients can be the same order. The mass matrix
is given by

$\displaystyle|M_{u}|=$ $\displaystyle\gamma_{u}\begin{pmatrix}5.720\times
10^{-9}&3.035\times 10^{-6}&5.688\times 10^{-5}\\\ 3.214\times
10^{-7}&5.685\times 10^{-5}&7.260\times 10^{-4}\\\ 9.030\times
10^{-6}&1.597\times 10^{-3}&2.013\times 10^{-2}\\\
\end{pmatrix}\frac{v_{u}}{\sqrt{2}},$ $\displaystyle|M_{d}|=$
$\displaystyle\gamma_{d}\begin{pmatrix}5.127\times 10^{-8}&1.755\times
10^{-6}&1.406\times 10^{-6}\\\ 2.881\times 10^{-6}&1.248\times
10^{-5}&2.634\times 10^{-5}\\\ 8.095\times 10^{-5}&7.376\times
10^{-4}&7.399\times 10^{-4}\\\ \end{pmatrix}\frac{v_{d}}{\sqrt{2}}.$ (63)

We obtain the following mass eigenvalues and mixing angles

$\displaystyle y_{u}/y_{t}$ $\displaystyle=4.22\times
10^{-6},~{}~{}y_{c}/y_{t}=2.68\times 10^{-3},~{}~{}y_{d}/y_{b}=6.96\times
10^{-4},~{}~{}y_{s}/y_{b}=1.38\times 10^{-2},$ $\displaystyle y_{b}/y_{t}$
$\displaystyle=1.30\times
10^{-2},~{}~{}\theta_{12}=0.227,~{}~{}\theta_{23}=0.0355,~{}~{}\theta_{13}=0.00314,~{}~{}\delta_{CP}=1.21.$

In this case, all the observables are within $0.4\sigma$ range, and
$\chi^{2}\sim 0.1$. Hence we can realize the realistic values without
hierarchical parameters.

#### Model with singlet left-handed quarks

Here we consider a model with singlet left-handed quarks based on the
superpotential of (48). Changing the flavor indices of the right-handed quarks
for both up and down sectors as $f_{1}^{c}\rightarrow
f_{3}^{c},f_{3}^{c}\rightarrow f_{2}^{c}$, and $f_{2}^{c}\rightarrow
f_{1}^{c}$, we obtain

$\displaystyle M_{u}=$ $\displaystyle\begin{pmatrix}{\rm Im\,}\tau^{1}&&\\\
&{\rm Im\,}\tau^{2}&\\\ &&{\rm
Im\,}\tau^{3}\end{pmatrix}\begin{pmatrix}\alpha_{u}\tilde{\phi}^{I}Y_{3}^{(2)}&\beta_{u}\tilde{\phi}^{J}Y_{3}^{(4)}&\gamma_{u}\tilde{\phi}^{K}(Y_{3,I}^{(6)}+r_{u}Y_{3,II}^{(6)})\\\
\alpha_{u}\tilde{\phi}^{I}Y_{2}^{(2)}&\beta_{u}\tilde{\phi}^{J}Y_{2}^{(4)}&\gamma_{u}\tilde{\phi}^{K}(Y_{2,I}^{(6)}+r_{u}Y_{2,II}^{(6)})\\\
\alpha_{u}\tilde{\phi}^{I}Y_{1}^{(2)}&\beta_{u}\tilde{\phi}^{J}Y_{1}^{(4)}&\gamma_{u}\tilde{\phi}^{K}(Y_{1,I}^{(6)}+r_{u}Y_{1,II}^{(6)})\\\
\end{pmatrix}^{t}\frac{v_{u}}{\sqrt{2}},$ $\displaystyle M_{d}=$
$\displaystyle\begin{pmatrix}{\rm Im\,}\tau^{1}&&\\\ &{\rm Im\,}\tau^{2}&\\\
&&{\rm
Im\,}\tau^{3}\end{pmatrix}\begin{pmatrix}\alpha_{d}\tilde{\phi}^{L}Y_{3}^{(2)}&\beta_{d}\tilde{\phi}^{M}Y_{3}^{(4)}&\gamma_{d}\tilde{\phi}^{N}(Y_{3,I}^{(6)}+r_{d}Y_{3,II}^{(6)})\\\
\alpha_{d}\tilde{\phi}^{L}Y_{2}^{(2)}&\beta_{d}\tilde{\phi}^{M}Y_{2}^{(4)}&\gamma_{d}\tilde{\phi}^{N}(Y_{2,I}^{(6)}+r_{d}Y_{2,II}^{(6)})\\\
\alpha_{d}\tilde{\phi}^{L}Y_{1}^{(2)}&\beta_{d}\tilde{\phi}^{M}Y_{1}^{(4)}&\gamma_{d}\tilde{\phi}^{N}(Y_{1,I}^{(6)}+r_{d}Y_{1,II}^{(6)})\\\
\end{pmatrix}^{t}\frac{v_{d}}{\sqrt{2}},$ (64)

which is the transposed matrix of (53). In the case of the lowest weight
modular forms, we have additional conditions of (51), which implies $I-J=L-M$
and $J-K=M-N$. Thus the same powers of $\tilde{\phi}$ arise in the mass ratios
for both the up and down sectors. In fact, an approximate expression of the
mass ratios are given by

$\displaystyle y_{u}/y_{t}$ $\displaystyle\sim
2\left(\frac{1}{3}\right)^{-6}{\rm
Im\,}\tau^{-2}\frac{\alpha_{u}}{\gamma_{u}}\tilde{\phi}^{I-K}q^{2/3},$
$\displaystyle y_{c}/y_{t}$ $\displaystyle\sim
2\left(\frac{1}{3}\right)^{-3}{\rm
Im\,}\tau^{-1}\frac{\beta_{u}}{\gamma_{u}}\tilde{\phi}^{J-K}q^{1/3},$
$\displaystyle y_{d}/y_{b}$ $\displaystyle\sim
2\left(\frac{1}{3}\right)^{-6}{\rm
Im\,}\tau^{-2}\frac{\alpha_{d}}{\gamma_{d}}\tilde{\phi}^{I-K}q^{2/3},~{}~{}$
$\displaystyle y_{s}/y_{b}$ $\displaystyle\sim
2\left(\frac{1}{3}\right)^{-3}{\rm
Im\,}\tau^{-1}\frac{\beta_{d}}{\gamma_{d}}\tilde{\phi}^{J-K}q^{1/3},~{}~{}$
$\displaystyle y_{b}/y_{t}$ $\displaystyle\sim\frac{\gamma_{d}}{\gamma_{u}}.$
(65)

Therefore unnatural hierarchical coefficients are inevitable to obtain the
realistic mass hierarchy.

| $Q_{1}$ | $Q_{2}$ | $Q_{3}$ | $u_{1,2,3}^{c}$ | $d_{1,2,3}^{c}$ | $H_{u}$ | $H_{d}$ | $\phi$
---|---|---|---|---|---|---|---|---
$\Gamma_{3}^{\prime}=T^{\prime}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 3}$ | ${\bf 3}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$
$k_{I}$ | $0$ | $-3$ | $-6$ | $0$ | $0$ | $0$ | $0$ | $-1$
Table 3: The representation of the quark and the Higgs fields.

We show an explicit model. The modular weights and representations for the
best fit model are summarized in Table 3. The mass matrices are given by (64)
with $I=L=2,J=M=1,K=N=0$. The best fit values are given by

$\displaystyle\alpha_{u}/\gamma_{u}=$ $\displaystyle 7.851\times
10^{-3},~{}~{}\beta_{u}/\gamma_{u}=0.1294,~{}~{}r_{u}=1.203e^{-2.369i},$
$\displaystyle\alpha_{d}/\gamma_{d}=$ $\displaystyle
1.967,~{}~{}\beta_{d}/\gamma_{d}=0.4558,~{}~{}r_{d}=2.398e^{-4.534i},$
$\displaystyle\gamma_{d}/\gamma_{u}=$ $\displaystyle 9.617\times
10^{-3}~{}~{}\tau=1.573i,~{}~{}\tilde{\phi}=1\times 10^{-2},$

where hierarchical coefficients are required. We obtain the mass matrix,

$\displaystyle|M_{u}|=$ $\displaystyle\gamma_{u}\begin{pmatrix}3.395\times
10^{-9}&3.053\times 10^{-8}&1.373\times 10^{-7}\\\ 2.935\times
10^{-6}&8.824\times 10^{-6}&3.936\times 10^{-5}\\\ 7.375\times
10^{-4}&2.166\times 10^{-3}&5.359\times 10^{-3}\\\
\end{pmatrix}\frac{v_{u}}{\sqrt{2}},$ $\displaystyle|M_{d}|=$
$\displaystyle\gamma_{d}\begin{pmatrix}8.507\times 10^{-7}&7.651\times
10^{-6}&3.441\times 10^{-5}\\\ 1.034\times 10^{-5}&3.110\times
10^{-5}&1.387\times 10^{-4}\\\ 1.298\times 10^{-3}&5.611\times
10^{-3}&5.387\times 10^{-3}\\\ \end{pmatrix}\frac{v_{d}}{\sqrt{2}}.$ (66)

The mixing angles and the mass ratios are calculated as

$\displaystyle y_{u}/y_{t}$ $\displaystyle=5.74\times
10^{-6},~{}~{}y_{c}/y_{t}=2.70\times 10^{-3},~{}~{}y_{d}/y_{b}=7.97\times
10^{-4},~{}~{}y_{s}/y_{b}=1.38\times 10^{-2},$ $\displaystyle y_{b}/y_{t}$
$\displaystyle=1.30\times
10^{-2},~{}~{}\theta_{12}=0.227,~{}~{}\theta_{23}=0.00612,~{}~{}\theta_{13}=0.00306,~{}~{}\delta_{CP}=1.11,$

where $\chi^{2}\sim 20$. Even though we relax the constraints on $|r|$, we can
not realize the observed values in this model.

As shown above, in the case of the lowest modular weight forms, hierarchical
parameters should be required even if we consider other modular groups of
higher levels. Thus singlet left-handed quark model is not suitable for our
purpose. Hereafter we only consider the models where the left-handed quarks
form a triplet of the modular group.

##  4 Froggatt-Nielsen like mechanism with the modular groups of higher
levels.

It is straightforward to generalize this mechanism to the modular group of the
other levels. The only requirement is the existence of triplet modular forms
which have hierarchical components. This condition is satisfied for the
modular group of level $N\geq 3$.

###  4.1 FN-like mechanism with the modular group of level 4

The algebra of $\Gamma_{4}^{\prime}$ is summarized in Appendix B.
$\Gamma_{4}^{\prime}$ is isomorphic to $S_{4}^{\prime}\simeq
SL(2,\mathbb{Z}_{4})$. It has four triplet representations, ${\bf 3},$
$\hat{\bf 3}$, ${\bf 3^{\prime}}$ and $\hat{\bf 3}^{\prime}$. The matrix
representations of $T$ in this algebra are summarized in Table 8 in Appendix
B. The triplet modular forms are approximated as

$\displaystyle Y^{(k)}_{\bf 3}\sim\begin{pmatrix}q^{1/2}\\\ q^{3/4}\\\
q^{1/4}\end{pmatrix},~{}~{}Y^{(k)}_{\hat{\bf 3}}\sim\begin{pmatrix}q^{1/4}\\\
q^{2/4}\\\ 1\end{pmatrix},~{}~{}Y^{(k)}_{{\bf
3}^{\prime}}\sim\begin{pmatrix}1\\\ q^{1/4}\\\
q^{3/4}\end{pmatrix},~{}~{}Y^{(k)}_{\hat{\bf
3}^{\prime}}\sim\begin{pmatrix}q^{3/4}\\\ 1\\\ q^{2/4}\end{pmatrix},$ (67)

for large Im $\tau$. Thus the FN-like hierarchical mass matrix can be realized
in the similar way. We consider four classes of the modular invariant
superpotentials:

$\displaystyle W_{1}$
$\displaystyle=\left[\alpha_{f}\tilde{\phi}^{I}(QY^{(k_{1})}_{\bf 3})_{\bf
1}f_{1}^{c}+\beta_{f}\tilde{\phi}^{J}(QY^{(k_{2})}_{\bf 3})_{\bf
1}f_{2}^{c}+\gamma_{f}\tilde{\phi}^{K}(QY^{(k_{3})}_{\bf 3})_{\bf
1}f_{3}^{c}\right]H_{f},$ $\displaystyle W_{2}$
$\displaystyle=\left[\alpha_{f}\tilde{\phi}^{I}(QY^{(k_{1})}_{\bf\hat{3}})_{\bf
1}f_{1}^{c}+\beta_{f}\tilde{\phi}^{J}(QY^{(k_{2})}_{\bf\hat{3}})_{\bf
1}f_{2}^{c}+\gamma_{f}\tilde{\phi}^{K}(QY^{(k_{3})}_{\bf\hat{3}})_{\bf
1}f_{3}^{c}\right]H_{f},$ $\displaystyle W_{3}$
$\displaystyle=\left[\alpha_{f}\tilde{\phi}^{I}(QY^{(k_{1})}_{\bf
3^{\prime}})_{\bf 1}f_{1}^{c}+\beta_{f}\tilde{\phi}^{J}(QY^{(k_{2})}_{\bf
3^{\prime}})_{\bf 1}f_{2}^{c}+\gamma_{f}\tilde{\phi}^{K}(QY^{(k_{3})}_{\bf
3^{\prime}})_{\bf 1}f_{3}^{c}\right]H_{f},$ $\displaystyle W_{4}$
$\displaystyle=\left[\alpha_{f}\tilde{\phi}^{I}(QY^{(k_{1})}_{\bf\hat{3}^{\prime}})_{\bf
1}f_{1}^{c}+\beta_{f}\tilde{\phi}^{J}(QY^{(k_{2})}_{\bf\hat{3}^{\prime}})_{\bf
1}f_{2}^{c}+\gamma_{f}\tilde{\phi}^{K}(QY^{(k_{3})}_{\bf\hat{3}^{\prime}})_{\bf
1}f_{3}^{c}\right]H_{f}.$ (68)

$f$ denotes the flavor $u$ or $d$, and $\phi$ is the trivial singlet carrying
modular weight $-1$, i.e., weighton. We can assign $f_{i}^{c}$ to the trivial
singlet ${\bf 3}$ of $\Gamma_{4}^{\prime}$ without loss of generality.
$Q_{i}^{c}$ is assigned to ${\bf 3}$ for $W_{i}$, ${\bf\hat{3}^{\prime}}$ for
$W_{2}$, ${\bf 3^{\prime}}$ for $W_{3}$, and ${\bf\hat{3}}$ for $W_{4}$. The
CG coefficients of the tensor products of the triplets are given by (101). We
obtain the mass matrix,

$\displaystyle
M_{1,2,3,4}\sim\begin{pmatrix}\tilde{\phi}^{I}Y_{1}^{(k_{1})}&\tilde{\phi}^{J}Y_{1}^{(k_{2})}&\tilde{\phi}^{K}Y_{1}^{(k_{3})}\\\
\tilde{\phi}^{I}Y_{3}^{(k_{1})}&\tilde{\phi}^{J}Y_{3}^{(k_{2})}&\tilde{\phi}^{K}Y_{3}^{(k_{3})}\\\
\tilde{\phi}^{I}Y_{2}^{(k_{1})}&\tilde{\phi}^{J}Y_{2}^{(k_{2})}&\tilde{\phi}^{K}Y_{2}^{(k_{3})}\end{pmatrix}\frac{v_{f}}{\sqrt{2}},$
(69)

where $Y_{i}^{(k_{j})}$ is the $i$-th component of the corresponding triplet
modular form of weight $k_{j}$. The mass matrix is approximated as

$\displaystyle
M_{1}\sim\begin{pmatrix}\tilde{\phi}^{I}q^{1/2}&\tilde{\phi}^{J}q^{1/2}&\tilde{\phi}^{K}q^{1/2}\\\
\tilde{\phi}^{I}q^{1/4}&\tilde{\phi}^{J}q^{1/4}&\tilde{\phi}^{K}q^{1/4}\\\
\tilde{\phi}^{I}q^{3/4}&\tilde{\phi}^{J}q^{3/4}&\tilde{\phi}^{K}q^{3/4}\end{pmatrix}\frac{v_{f}}{\sqrt{2}},~{}~{}M_{2}\sim\begin{pmatrix}\tilde{\phi}^{I}q^{1/4}&\tilde{\phi}^{J}q^{1/4}&\tilde{\phi}^{K}q^{1/4}\\\
\tilde{\phi}^{I}&\tilde{\phi}^{J}&\tilde{\phi}^{K}\\\
\tilde{\phi}^{I}q^{1/2}&\tilde{\phi}^{J}q^{1/2}&\tilde{\phi}^{K}q^{1/2}\end{pmatrix}\frac{v_{f}}{\sqrt{2}},$
$\displaystyle
M_{3}\sim\begin{pmatrix}\tilde{\phi}^{I}&\tilde{\phi}^{J}&\tilde{\phi}^{K}\\\
\tilde{\phi}^{I}q^{3/4}&\tilde{\phi}^{J}q^{3/4}&\tilde{\phi}^{K}q^{3/4}\\\
\tilde{\phi}^{I}q^{1/4}&\tilde{\phi}^{J}q^{1/4}&\tilde{\phi}^{K}q^{1/4}\end{pmatrix}\frac{v_{f}}{\sqrt{2}},~{}~{}M_{4}\sim\begin{pmatrix}\tilde{\phi}^{I}q^{3/4}&\tilde{\phi}^{J}q^{3/4}&\tilde{\phi}^{K}q^{3/4}\\\
\tilde{\phi}^{I}q^{1/2}&\tilde{\phi}^{J}q^{1/2}&\tilde{\phi}^{K}q^{1/2}\\\
\tilde{\phi}^{I}&\tilde{\phi}^{J}&\tilde{\phi}^{K}\end{pmatrix}\frac{v_{f}}{\sqrt{2}},$
(70)

for large ${\rm Im\,}\tau$. We obtain the FN-like mass matrix and hierarchical
mass eigenvalues.

We can choose the superpotential of the up sector and that of the down sector
from $W_{1}$, $W_{2}$, $W_{3}$ and $W_{4}$ individually. Thus we have 16
classes of FN-like models with $\Gamma_{4}^{\prime}$ in principal. However, we
find that it is difficult to obtain the observed mixing angles if we use a
different type of the superpotential for each sector. As shown in (67) the
position of the largest component is different for each representation. If we
use a different representation for each sector, we can not obtain FN-like
Yukawa matrices for both sectors simultaneously, where the order of the
contribution to a mixing angle from each sector could be different, and one of
the mixing angles may become large.101010 More precisely, ${\bf 3}$ and
${\bf\hat{3}}$ have the same order with different numerical factors in
$q$-expansion (See Appendix B), so that we could obtain the FN-like Yukawa
matrices for a model using ${\bf 3}$ and ${\bf\hat{3}}$. We find, however,
that this model is not realistic. In fact, one can check that the modular
forms of ${\bf\hat{3}}$ of weight $k$ and the modular forms of of 3 of weight
$k+3$ are linearly dependent for $k=1,3,5$.

We have the same constraints on the power of $\tilde{\phi}$ as those in
$\Gamma_{3}^{\prime}$. Namely, the powers of $\tilde{\phi}$ are 0 or 1 in
general, but they can be arbitrary positive integer if the corresponding
Yukawa coupling is the lowest weight modular form. The lowest weight is 1 for
${\bf\hat{3}}$, 2 for ${\bf 3^{\prime}}$, 3 for ${\bf\hat{3}^{\prime}}$ and 4
for ${\bf 3}$.

### Lowest weight models

The superpotential $W_{i}$ has several free parameters. They are proportional
to the number of the triplet modular forms in the superpotential. In order to
minimize the number of the free parameters we consider the model with the
lowest weight modular forms. They are given by

$\displaystyle W_{1}$
$\displaystyle=\left[\alpha_{f}\tilde{\phi}^{I}(QY^{(4)}_{\bf 3})_{\bf
1}f_{1}^{c}+\beta_{f}\tilde{\phi}^{J}(QY^{(6)}_{\bf 3})_{\bf
1}f_{2}^{c}+\gamma_{f}\tilde{\phi}^{K}(Q(Y^{(8)}_{{\bf
3},I}+r_{f}Y^{(8)}_{{\bf 3},II}))_{\bf 1}f_{3}^{c}\right]H_{f},$
$\displaystyle W_{2}$
$\displaystyle=\left[\alpha_{f}\tilde{\phi}^{I}(QY^{(1)}_{\bf\hat{3}})_{\bf
1}f_{1}^{c}+\beta_{f}\tilde{\phi}^{J}(QY^{3)}_{\bf\hat{3}})_{\bf
1}f_{2}^{c}+\gamma_{f}\tilde{\phi}^{K}(Q(Y^{(5)}_{{\bf\hat{3}},I}+r_{f}Y^{(5)}_{{\bf\hat{3}},II}))_{\bf
1}f_{3}^{c}\right]H_{f},$ $\displaystyle W_{3}$
$\displaystyle=\left[\alpha_{f}\tilde{\phi}^{I}(QY^{(2)}_{\bf
3^{\prime}})_{\bf 1}f_{1}^{c}+\beta_{f}\tilde{\phi}^{J}(QY^{(4)}_{\bf
3^{\prime}})_{\bf 1}f_{2}^{c}+\gamma_{f}\tilde{\phi}^{K}(Q(Y^{(6)}_{{\bf
3^{\prime}},I}+r_{f}Y^{(6)}_{{\bf 3^{\prime}},II})_{\bf
1}f_{3}^{c}\right]H_{f},$ $\displaystyle W_{4}$
$\displaystyle=\left[\alpha_{f}\tilde{\phi}^{I}(QY^{(3)}_{\bf\hat{3}^{\prime}})_{\bf
1}f_{1}^{c}+\beta_{f}\tilde{\phi}^{J}(QY^{(5)}_{\bf\hat{3}^{\prime}})_{\bf
1}f_{2}^{c}+\gamma_{f}\tilde{\phi}^{K}(Q(Y^{(7)}_{{\bf\hat{3}}^{\prime},I}+r_{f}Y^{(7)}_{{\bf\hat{3}}^{\prime},II}))_{\bf
1}f_{3}^{c}\right]H_{f}.$ (71)

The total number of the free parameters are the same as that of the
superpotential of $\Gamma_{3}^{\prime}$. $\alpha_{f},\beta_{f},\gamma_{f}$ are
real numbers, and $r_{f}$ is a complex number. Thus we have five real
parameters for each sector. $I$ can be an arbitrary positive integer number,
but $J$ and $K$ are restricted to $0$ or $1$.

###  4.2 FN-like mechanism with the modular group of level 5

The algebra of $\Gamma_{5}^{\prime}$ is summarized in Appendix C.
$\Gamma_{5}^{\prime}$ is isomorphic to $A_{5}^{\prime}\simeq
SL(2,\mathbb{Z}_{5})$ having two triplets ${\bf 3}$ and ${\bf 3^{\prime}}$.
Their matrix representations are shown in Table 10. The triplet modular forms
are approximated as

$\displaystyle Y^{(k)}_{\bf 3}\sim\begin{pmatrix}1\\\ q^{1/5}\\\
q^{4/5}\end{pmatrix},~{}~{}Y^{(k)}_{{\bf 3}^{\prime}}\sim\begin{pmatrix}1\\\
q^{2/5}\\\ q^{3/5}\end{pmatrix},$ (72)

for large Im $\tau$. We can construct the FN-like model in the same way. We
have two classes of the FN-like superpotential,

$\displaystyle W_{1}$
$\displaystyle=\left[\alpha_{f}\tilde{\phi}^{I}(QY^{(k_{1})}_{\bf 3})_{\bf
1}f_{1}^{c}+\beta_{f}\tilde{\phi}^{J}(QY^{(k_{2})}_{\bf 3})_{\bf
1}f_{2}^{c}+\gamma_{f}\tilde{\phi}^{K}(QY^{(k_{3})}_{\bf 3})_{\bf
1}f_{3}^{c}\right]H_{f},$ $\displaystyle W_{2}$
$\displaystyle=\left[\alpha_{f}\tilde{\phi}^{I}(QY^{(k_{1})}_{\bf
3^{\prime}})_{\bf 1}f_{1}^{c}+\beta_{f}\tilde{\phi}^{J}(QY^{(k_{2})}_{\bf
3^{\prime}})_{\bf 1}f_{2}^{c}+\gamma_{f}\tilde{\phi}^{K}(QY^{(k_{3})}_{\bf
3^{\prime}})_{\bf 1}f_{3}^{c}\right]H_{f},$ (73)

with the triplet left-handed quarks. We assume that $f_{i}^{c}$ is the trivial
singlet, and $Q_{i}$ form a triplet of $\Gamma_{5}^{\prime}$. The CG
coefficients are summarized in (103). The mass matrices are approximated as

$\displaystyle
M_{1}\sim\begin{pmatrix}\tilde{\phi}^{I}&\tilde{\phi}^{J}&\tilde{\phi}^{K}\\\
\tilde{\phi}^{I}q^{4/5}&\tilde{\phi}^{J}q^{4/5}&\tilde{\phi}^{K}q^{4/5}\\\
\tilde{\phi}^{I}q^{1/5}&\tilde{\phi}^{J}q^{1/5}&\tilde{\phi}^{K}q^{1/5}\end{pmatrix}\frac{v_{f}}{\sqrt{2}},~{}~{}M_{2}\sim\begin{pmatrix}\tilde{\phi}^{I}&\tilde{\phi}^{J}&\tilde{\phi}^{K}\\\
\tilde{\phi}^{I}q^{3/5}&\tilde{\phi}^{J}q^{3/5}&\tilde{\phi}^{K}q^{3/5}\\\
\tilde{\phi}^{I}q^{2/5}&\tilde{\phi}^{J}q^{2/5}&\tilde{\phi}^{K}q^{2/5}\end{pmatrix}\frac{v_{f}}{\sqrt{2}},$

for large ${\rm Im\,}\tau$. Thus we obtain FN-like hierarchical eigenvalues
for both cases. We have 2 possibilities of the representations of the Yukawa
couplings both for the up and down sectors. We note, however, that since the
tensor product of ${\bf 3}$ and ${\bf 3}^{\prime}$ has no singlet, the Yukawa
couplings and $Q$ should be the same representation. Therefore we have 2
possibilities of either ${\bf 3}$ or ${\bf 3^{\prime}}$ for both up and down
sectors.

We have the same constraints on the power of $\tilde{\phi}$ as in the previous
models. In this case, the lowest weight is $2$ both for ${\bf 3}$ and ${\bf
3^{\prime}}$.

### Lowest weight models

The superpotential including the lowest weight modular forms are given by

$\displaystyle W_{1}$
$\displaystyle=\left[\alpha_{f}\tilde{\phi}^{I}(QY^{(2)}_{\bf 3})_{\bf
1}f_{1}^{c}+\beta_{f}\tilde{\phi}^{J}(QY^{(4)}_{\bf 3})_{\bf
1}f_{2}^{c}+\gamma_{f}\tilde{\phi}^{K}(Q(Y^{(6)}_{{\bf
3},I}+r_{f}Y^{(6)}_{{\bf 3},II}))_{\bf 1}f_{3}^{c}\right]H_{f},$
$\displaystyle W_{2}$
$\displaystyle=\left[\alpha_{f}\tilde{\phi}^{I}(QY^{(2)}_{\bf
3^{\prime}})_{\bf 1}f_{1}^{c}+\beta_{f}\tilde{\phi}^{J}(QY^{(4)}_{\bf
3^{\prime}})_{\bf 1}f_{2}^{c}+\gamma_{f}\tilde{\phi}^{K}(Q(Y^{(6)}_{{\bf
3^{\prime}},I}+r_{f}Y^{(6)}_{{\bf 3^{\prime}},II})_{\bf
1}f_{3}^{c}\right]H_{f}.$ (74)

The number of the free parameters are the same as that of the superpotential
of $\Gamma_{3}^{\prime}$ and $\Gamma_{4}^{\prime}$.
$\alpha_{f},\beta_{f},\gamma_{f}$ are real, and $r_{f}$ is a complex number.
We have five real parameters for each sector.

###  4.3 Realistic models without hierarchical parameters with
$\Gamma_{4}^{\prime}$ and $\Gamma_{5}^{\prime}$

In this subsection, we analyze the model with the modular group of level 4 and
5. We show some typical models for illustration purpose.

#### Yukawa couplings of 3 representation in $\Gamma_{4}^{\prime}$

We consider the FN-like mechanism based on the superpotential $W_{1}$ in (71),
i.e., the Yukawa couplings are ${\bf 3}$ in $\Gamma_{4}^{\prime}$. We obtain
the mass matrix

$\displaystyle M_{u}=$
$\displaystyle\begin{pmatrix}\alpha_{u}\tilde{\phi}^{I}Y_{2}^{(4)}&\beta_{u}\tilde{\phi}^{J}Y_{2}^{(6)}&\gamma_{u}\tilde{\phi}^{K}(Y_{2,I}^{(8)}+r_{u}Y_{2,II}^{(8)})\\\
\alpha_{u}\tilde{\phi}^{I}Y_{1}^{(4)}&\beta_{u}\tilde{\phi}^{J}Y_{1}^{(6)}&\gamma_{u}\tilde{\phi}^{K}(Y_{1,I}^{(8)}+r_{u}Y_{1,II}^{(8)})\\\
\alpha_{u}\tilde{\phi}^{I}Y_{3}^{(4)}&\beta_{u}\tilde{\phi}^{J}Y_{3}^{(6)}&\gamma_{u}\tilde{\phi}^{K}(Y_{3,I}^{(8)}+r_{u}Y_{3,II}^{(8)})\\\
\end{pmatrix}\begin{pmatrix}{\rm Im\,}\tau^{2}&&\\\ &{\rm Im\,}\tau^{3}&\\\
&&{\rm Im\,}\tau^{4}\end{pmatrix}\frac{v_{u}}{\sqrt{2}},$ $\displaystyle
M_{d}=$
$\displaystyle\begin{pmatrix}\alpha_{d}\tilde{\phi}^{L}Y_{2}^{(4)}&\beta_{d}\tilde{\phi}^{M}(Y_{2,I}^{(8)}+r_{d}Y_{2,II}^{(8)})&\gamma_{d}\tilde{\phi}^{N}Y_{2}^{(6)}\\\
\alpha_{d}\tilde{\phi}^{L}Y_{1}^{(4)}&\beta_{d}\tilde{\phi}^{M}(Y_{1,I}^{(8)}+r_{d}Y_{1,II}^{(8)})&\gamma_{d}\tilde{\phi}^{N}Y_{1}^{(6)}\\\
\alpha_{d}\tilde{\phi}^{L}Y_{3}^{(4)}&\beta_{d}\tilde{\phi}^{M}(Y_{3,I}^{(8)}+r_{d}Y_{3,II}^{(8)})&\gamma_{d}\tilde{\phi}^{N}Y_{3}^{(6)}\\\
\end{pmatrix}\begin{pmatrix}{\rm Im\,}\tau^{2}&&\\\ &{\rm Im\,}\tau^{4}&\\\
&&{\rm Im\,}\tau^{3}\end{pmatrix}\frac{v_{d}}{\sqrt{2}}.$ (75)

We assume $I\geq J\geq K$ and $L\geq M\geq N$ to obtain a FN-like matrix. We
also study other possibilities, but the above mass matrix is the best one. The
mixing angles are approximated by

$\displaystyle\theta_{12}$
$\displaystyle\sim\left|\frac{Y^{(6)}_{2}}{Y^{(6)}_{1}}-\frac{Y^{(8)}_{2,I}+r_{d}Y^{(8)}_{2,II}}{Y^{(8)}_{1,I}+r_{d}Y^{(8)}_{1,II}}\right|\sim\frac{13}{\sqrt{2}}\left|\left(1+\frac{\sqrt{10}}{13r_{d}}\right)q^{1/4}\right|\sim
9\left|(1+0.2r_{d}^{-1})q^{1/4}\right|,$ $\displaystyle\theta_{23}$
$\displaystyle\sim\left|\frac{Y^{(8)}_{1,I}+r_{u}Y^{(8)}_{1,II}}{Y^{(8)}_{3,I}+r_{u}Y^{(8)}_{3,II}}-\frac{Y^{(6)}_{1}}{Y^{(6)}_{3}}\right|\sim
4\sqrt{5}\left|\left(r_{u}-\sqrt{\frac{2}{5}}\right)q^{1/4}\right|\sim
9\left|(r_{u}-0.6)q^{1/4}\right|,$ $\displaystyle\theta_{13}$
$\displaystyle\sim\left|\frac{Y^{(8)}_{2,I}+r_{u}Y^{(8)}_{2,II}}{Y^{(8)}_{3,I}+r_{u}Y^{(8)}_{3,II}}-\frac{Y^{(6)}_{2}}{Y^{(6)}_{3}}\right|\sim
16\sqrt{10}\left|\left(r_{u}+\frac{1}{2}\sqrt{\frac{5}{2}}\right)q^{2/4}\right|\sim
51\left|(r_{u}+0.8)q^{2/4}\right|.$

The first condition implies $9|q^{1/4}|\sim\theta_{C}\sim 10^{-1}$, which
implies ${\rm Im\,}\tau\sim 2.8$. The remaining conditions are rewritten as

$\displaystyle\theta_{23}\sim\left|r_{u}-0.6\right|\times
10^{-1},~{}~{}\theta_{13}\sim 6\left|r_{u}+0.8\right|\times 10^{-3}.$

and the realistic mixing angles are realized naturally with $|r_{u}-0.6|\sim
10^{-1}$. Then the mass ratios are approximated as

$\displaystyle y_{u}/y_{t}$
$\displaystyle\sim\frac{16\sqrt{10}}{3}\left(\frac{\sqrt{2}}{13}\right)^{2}{\rm
Im\,}\tau^{-2}\frac{\alpha_{u}}{\gamma_{u}}\tilde{\phi}^{I-K}\theta_{C}^{2}\sim
3\times 10^{-4}\frac{\alpha_{u}}{\gamma_{u}}\tilde{\phi}^{I-K},$
$\displaystyle y_{c}/y_{t}$ $\displaystyle\sim\frac{16\sqrt{10}}{39}{\rm
Im\,}\tau^{-1}\frac{\beta_{u}}{\gamma_{u}}\tilde{\phi}^{J-K}\theta_{C}\sim
5\times 10^{-2}\frac{\beta_{u}}{\gamma_{u}}\tilde{\phi}^{J-K},$ $\displaystyle
y_{d}/y_{b}$ $\displaystyle\sim\frac{16}{(13)^{2}}{\rm
Im\,}\tau^{-1}\frac{\alpha_{d}}{\gamma_{d}}\tilde{\phi}^{L-N}\theta_{C}^{2}\sim
3\times 10^{-4}\frac{\alpha_{d}}{\gamma_{d}}\tilde{\phi}^{L-N},$
$\displaystyle y_{s}/y_{b}$ $\displaystyle\sim\frac{6|r_{d}|}{13}{\rm
Im\,}\tau\frac{\beta_{d}}{\gamma_{d}}\tilde{\phi}^{M-N}\theta_{C}\sim 1\times
10^{-1}\frac{\beta_{d}|r_{d}|}{\gamma_{d}}\tilde{\phi}^{M-N},$ $\displaystyle
y_{b}/y_{t}$ $\displaystyle\sim\frac{4}{3}\sqrt{\frac{5}{2}}{\rm
Im\,}\tau^{-1}\frac{\gamma_{d}}{\gamma_{u}}\tilde{\phi}^{N-K}\sim 7\times
10^{-1}\frac{\gamma_{d}}{\gamma_{u}}\tilde{\phi}^{N-K}.$ (76)

These conditions imply $\tilde{\phi}\sim 2\times 10^{-2}$,
$I=J=1,K=0,L=M=N=1$, and all the coefficients are $\mathcal{O}(1)$. Thus we
can naturally reproduce the observed values.

To confirm the above analysis, we construct an explicit example. The
representations and the modular weights of the quark fields are summarized in
Table 4. The mass matrix is given by (75) with $I=2,J=1,K=0,$ and $L=2,M=N=1$.

| $Q_{1,2,3}$ | $u_{1}^{c}$ | $u_{2}^{c}$ | $u_{2}^{c}$ | $d_{1}^{c}$ | $d_{2}^{c}$ | $d_{3}^{c}$ | $H_{u}$ | $H_{d}$ | $\phi$
---|---|---|---|---|---|---|---|---|---|---
$\Gamma_{4}^{\prime}$ | ${\bf 3}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$
$k_{I}$ | $-1$ | $-1$ | $-4$ | $-7$ | $-1$ | $-6$ | $-4$ | $0$ | $0$ | $-1$
Table 4: The representations and the modular weights of the quarks. This is an
explicit model for superpotential with Yukawa coupling of ${\bf 3}$ in
$S_{4}^{\prime}$.

The best fit parameter in our analysis is given by

$\displaystyle\alpha_{u}/\gamma_{u}=$ $\displaystyle
0.2514,~{}~{}\beta_{u}/\gamma_{u}=1.968,~{}~{}r_{u}=e^{-0.3176i},$
$\displaystyle\alpha_{d}/\gamma_{d}=$ $\displaystyle
0.8793,~{}~{}\beta_{d}/\gamma_{d}=0.3907,~{}~{}r_{d}=e^{0.1469i},$
$\displaystyle\gamma_{d}/\gamma_{u}=$ $\displaystyle
0.3906,~{}~{}\tau=2.851i,~{}~{}\tilde{\phi}=4\times 10^{-2}.$ (77)

The largest hierarchy comes from $\beta_{u}/\beta_{d}=12.9$. We set ${\rm
Re\,}\tau=0$ and $|r_{f}|=1$ again. We obtain the following hierarchical mass
matrices,

$\displaystyle|M_{u}|=$ $\displaystyle\gamma_{u}\begin{pmatrix}1.423\times
10^{-7}&1.984\times 10^{-5}&1.191\times 10^{-3}\\\ 8.865\times
10^{-7}&4.945\times 10^{-4}&1.344\times 10^{-2}\\\ 2.76\times
10^{-5}&7.702\times 10^{-3}&1.323\times 10^{-1}\\\
\end{pmatrix}\frac{v_{u}}{\sqrt{2}},$ $\displaystyle|M_{d}|=$
$\displaystyle\gamma_{d}\begin{pmatrix}4.977\times 10^{-8}&1.877\times
10^{-5}&1.009\times 10^{-5}\\\ 3.101\times 10^{-6}&2.100\times
10^{-4}&2.513\times 10^{-4}\\\ 9.658\times 10^{-5}&2.068\times
10^{-3}&3.914\times 10^{-3}\end{pmatrix}\frac{v_{d}}{\sqrt{2}}.$ (78)

The mass eigenvalues and mixing angles are given as

$\displaystyle y_{u}/y_{t}$ $\displaystyle=5.45\times
10^{-6},~{}~{}y_{c}/y_{t}=2.68\times 10^{-3},~{}~{}y_{d}/y_{b}=6.53\times
10^{-4},~{}~{}y_{s}/y_{b}=1.69\times 10^{-2},$ $\displaystyle y_{b}/y_{t}$
$\displaystyle=1.30\times
10^{-2},~{}~{}\theta_{12}=0.227,~{}~{}\theta_{23}=0.0430,~{}~{}\theta_{13}=0.00203,~{}~{}\delta_{CP}=1.31.$
(79)

The parameter which is the most apart from the observed value is
$\theta_{13}$, and $(\theta_{13}-\theta_{13}^{obs})/\sigma_{13}\sim
2.3\sigma$. We obtain $\chi^{2}\sim 12$, and almost all the parameters are
within $2\sigma$ range.

Relaxing the restriction on $|r_{f}|$, we can find parameter set which
reproduce observed values more precisely. A benchmark is given by

$\displaystyle\alpha_{u}/\gamma_{u}=$ $\displaystyle
1.130,~{}~{}\beta_{u}/\gamma_{u}=4.334,~{}~{}r_{u}=1.040e^{-0.2224i},$
$\displaystyle\alpha_{d}/\gamma_{d}=$ $\displaystyle
2.193,~{}~{}\beta_{d}/\gamma_{d}=0.7258,~{}~{}r_{d}=0.8893e^{0.1228i},$
$\displaystyle\gamma_{d}/\gamma_{u}=$ $\displaystyle
0.6361,~{}~{}\tau=2.901i,~{}~{}\tilde{\phi}=2\times 10^{-2}.$

and the mot hierarchical term comes from $\alpha_{u}/\beta_{d}=9.4$. Thus all
the coefficients are the same order. We obtain the mass matrices

$\displaystyle|M_{u}|=$ $\displaystyle\gamma_{u}\begin{pmatrix}1.287\times
10^{-8}&1.791\times 10^{-5}&1.027\times 10^{-3}\\\ 8.671\times
10^{-7}&4.825\times 10^{-4}&1.259\times 10^{-2}\\\ 2.920\times
10^{-5}&8.123\times 10^{-3}&1.289\times 10^{-1}\\\
\end{pmatrix}\frac{v_{u}}{\sqrt{2}},$ $\displaystyle|M_{d}|=$
$\displaystyle\gamma_{d}\begin{pmatrix}2.499\times 10^{-8}&1.339\times
10^{-5}&4.132\times 10^{-6}\\\ 1.683\times 10^{-6}&1.563\times
10^{-4}&1.113\times 10^{-4}\\\ 5.668\times 10^{-5}&1.871\times
10^{-3}&1.874\times 10^{-3}\\\ \end{pmatrix}\frac{v_{d}}{\sqrt{2}}.$ (80)

The mass eigenvalues and mixing angles are given by

$\displaystyle y_{u}/y_{t}$ $\displaystyle=5.70\times
10^{-6},~{}~{}y_{c}/y_{t}=2.68\times 10^{-3},~{}~{}y_{d}/y_{b}=6.92\times
10^{-4},~{}~{}y_{s}/y_{b}=1.37\times 10^{-2},~{}~{}$ $\displaystyle
y_{b}/y_{t}$ $\displaystyle=1.30\times
10^{-2},~{}~{}\theta_{12}=0.227,~{}~{}\theta_{23}=0.0359,~{}~{}\theta_{13}=0.00314,~{}~{}\delta_{CP}=1.21.$
(81)

Thus all parameters are included within $0.1\sigma$ interval, and
$\chi^{2}<0.01$.

#### Yukawa couplings of ${\bf\hat{3}^{\prime}}$ in $\Gamma_{4}^{\prime}$

The $q$-expansions of the modular forms of ${\bf\hat{3}^{\prime}}$ in
$\Gamma_{4}^{\prime}$ are different from the modular forms of ${\bf 3}$. If
the superpotential is given by $W_{4}$ in (71), the approximated mass matrix
is given by $M_{4}$ in (70). This matrix is quite interesting because the
mixing angles are approximated by

$\displaystyle\theta_{12}\sim|q^{1/4}|,~{}~{}\theta_{23}\sim|q^{2/4}|,~{}~{}\theta_{13}\sim|q^{3/4}|.$
(82)

These relations are nothing but approximate mixing angles predicted in the FN
mechanism. Thus this model seems to be the most promising candidate. We
consider the model with the following mass matrices,

$\displaystyle M_{u}=$
$\displaystyle\begin{pmatrix}\alpha_{u}\tilde{\phi}^{I}Y_{1}^{(3)}&\beta_{u}\tilde{\phi}^{J}Y_{1}^{(5)}&\gamma_{u}\tilde{\phi}^{K}(Y_{1,I}^{(7)}+r_{u}Y_{1,II}^{(7)})\\\
\alpha_{u}\tilde{\phi}^{I}Y_{3}^{(3)}&\beta_{u}\tilde{\phi}^{J}Y_{3}^{(5)}&\gamma_{u}\tilde{\phi}^{K}(Y_{3,I}^{(7)}+r_{u}Y_{3,II}^{(7)})\\\
\alpha_{u}\tilde{\phi}^{I}Y_{2}^{(3)}&\beta_{u}\tilde{\phi}^{J}Y_{2}^{(5)}&\gamma_{u}\tilde{\phi}^{K}(Y_{2,I}^{(7)}+r_{u}Y_{2,II}^{(7)})\\\
\end{pmatrix}\begin{pmatrix}{\rm Im\,}\tau^{3/2}&&\\\ &{\rm
Im\,}\tau^{5/2}&\\\ &&{\rm
Im\,}\tau^{7/2}\end{pmatrix}\frac{v_{u}}{\sqrt{2}},$ $\displaystyle M_{d}=$
$\displaystyle\begin{pmatrix}\alpha_{d}\tilde{\phi}^{L}Y_{1}^{(3)}&\beta_{d}\tilde{\phi}^{M}(Y_{1,I}^{(7)}+r_{d}Y_{1,II}^{(7)})&\gamma_{d}\tilde{\phi}^{N}Y_{1}^{(5)}\\\
\alpha_{d}\tilde{\phi}^{L}Y_{3}^{(3)}&\beta_{d}\tilde{\phi}^{M}(Y_{3,I}^{(7)}+r_{d}Y_{3,II}^{(7)})&\gamma_{d}\tilde{\phi}^{N}Y_{3}^{(5)}\\\
\alpha_{d}\tilde{\phi}^{L}Y_{2}^{(3)}&\beta_{d}\tilde{\phi}^{M}(Y_{2,I}^{(7)}+r_{d}Y_{2,II}^{(7)})&\gamma_{d}\tilde{\phi}^{N}Y_{2}^{(5)}\\\
\end{pmatrix}\begin{pmatrix}{\rm Im\,}\tau^{3/2}&&\\\ &{\rm
Im\,}\tau^{7/2}&\\\ &&{\rm
Im\,}\tau^{5/2}\end{pmatrix}\frac{v_{d}}{\sqrt{2}}.$ (83)

The precise $q$-expansion of the modular forms are summarized in Appendix B.
The mixing angles are estimated as

$\displaystyle\theta_{12}$
$\displaystyle\sim\left|\frac{Y^{(5)}_{1}}{Y^{(5)}_{3}}-\frac{Y^{(7)}_{1,I}+r_{d}Y^{(7)}_{1,II}}{Y^{(7)}_{3,I}+r_{d}Y^{(7)}_{3,II}}\right|\sim\frac{22\sqrt{2}}{3}\left|1+\frac{14}{22\sqrt{37}}\frac{1}{r_{d}}\right||q^{1/4}|\sim
10\left|1+0.1r_{d}^{-1}\right||q^{1/4}|,$ $\displaystyle\theta_{23}$
$\displaystyle\sim\left|\frac{Y^{(7)}_{3,I}+r_{u}Y^{(7)}_{3,II}}{Y^{(7)}_{2,I}+r_{u}Y^{(7)}_{2,II}}-\frac{Y^{(5)}_{3}}{Y^{(5)}_{2}}\right|\sim
12\sqrt{37}\left|r_{u}-\frac{1}{\sqrt{37}}\right||q^{2/4}|\sim
73\left|r_{u}-0.2\right||q^{2/4}|,$ $\displaystyle\theta_{13}$
$\displaystyle\sim\left|\frac{Y^{(7)}_{1,I}+r_{u}Y^{(7)}_{1,II}}{Y^{(7)}_{2,I}+r_{u}Y^{(7)}_{2,II}}-\frac{Y^{(5)}_{1}}{Y^{(5)}_{2}}\right|\sim
24\sqrt{74}\left|r_{u}+\frac{5}{\sqrt{37}}\right||q^{3/4}|\sim
206\left|r_{u}+0.8\right||q^{3/4}|.$

The first relation implies $10|q^{1/4}|\sim\theta_{C}\sim 10^{-1}$, which
implies ${\rm Im\,}\tau\sim 2.9$. Substituting this relation, we obtain

$\displaystyle\theta_{23}\sim
0.7\left|r_{u}-0.2\right|\theta_{C}^{2},~{}~{}\theta_{13}\sim
0.2\left|r_{u}+0.8\right|\theta_{C}^{3}.$ (84)

Thus we obtain approximate relation of the mixing angles. The mass ratios are
approximated as

$\displaystyle y_{u}/y_{t}$ $\displaystyle\sim\frac{36\sqrt{74}}{11^{3}}{\rm
Im\,}\tau^{-2}\frac{\alpha_{u}}{\gamma_{u}}\tilde{\phi}^{I-K}\theta_{C}^{3}\sim
3\times 10^{-5}\frac{\alpha_{u}}{\gamma_{u}}\tilde{\phi}^{I-K},$
$\displaystyle y_{c}/y_{t}$ $\displaystyle\sim\frac{9\sqrt{37}}{242}{\rm
Im\,}\tau^{-1}\frac{\beta_{u}}{\gamma_{u}}\tilde{\phi}^{J-K}\theta_{C}^{2}\sim
8\times 10^{-4}\frac{\beta_{u}}{\gamma_{u}}\tilde{\phi}^{J-K},$ $\displaystyle
y_{d}/y_{b}$ $\displaystyle\sim\frac{216}{11^{3}\sqrt{2}}{\rm
Im\,}\tau^{-1}\frac{\alpha_{d}}{\gamma_{d}}\tilde{\phi}^{L-N}\theta_{C}^{3}\sim
4\times 10^{-5}\frac{\alpha_{d}}{\gamma_{d}}\tilde{\phi}^{L-N},$
$\displaystyle y_{s}/y_{b}$ $\displaystyle\sim\frac{81}{242}{\rm
Im\,}\tau\frac{\beta_{d}}{\gamma_{d}}|r_{d}|\tilde{\phi}^{M-N}\theta_{C}^{2}\sim
1\times 10^{-2}\frac{\beta_{d}}{\gamma_{d}}|r_{d}|\tilde{\phi}^{M-N},$
$\displaystyle y_{b}/y_{t}$ $\displaystyle\sim\frac{\sqrt{37}}{3}{\rm
Im\,}\tau^{-1}\frac{\gamma_{d}}{\gamma_{u}}\tilde{\phi}^{N-K}\sim 7\times
10^{-1}\frac{\gamma_{d}}{\gamma_{u}}\tilde{\phi}^{N-K}.$ (85)

These relations imply $\tilde{\phi}\sim 2\times 10^{-2}$, $I=1,J=K=0$ and
$L=M=N=1.$

To confirm the above estimation, we construct an explicit examples. The
representations and the modular weights of the fields are summarized in Table
5. This model generates the mass matrix with $I=2,J=1,K=0,L=2,M=N=1$.

| $Q_{1,2,3}$ | $u_{1}^{c}$ | $u_{2}^{c}$ | $u_{2}^{c}$ | $d_{1}^{c}$ | $d_{2}^{c}$ | $d_{3}^{c}$ | $H_{u}$ | $H_{d}$ | $\phi$
---|---|---|---|---|---|---|---|---|---|---
$\Gamma_{4}^{\prime}$ | ${\bf\hat{3}}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$
$k_{I}$ | $-1$ | $0$ | $-3$ | $-6$ | $0$ | $-5$ | $-3$ | $0$ | $0$ | $-1$
Table 5: The representations and the modular weights of the quarks. This is an
explicit model for superpotential with Yukawa coupling of
${\bf\hat{3}^{\prime}}$ in $S_{4}^{\prime}$.

In this model, we can not find a parameter set whose $\chi^{2}<50$ with
$|r_{f}|=1$. We relax this constraint. The best fit parameters in our analysis
are given by

$\displaystyle\alpha_{u}/\gamma_{u}=$ $\displaystyle
1.516,~{}~{}\beta_{u}/\gamma_{u}=2.908,~{}~{}r_{u}=0.4535e^{2.051i},$
$\displaystyle\alpha_{d}/\gamma_{d}=$ $\displaystyle
3.287,~{}~{}\beta_{d}/\gamma_{d}=0.9015,~{}~{}r_{d}=0.5514e^{0.5518i},$
$\displaystyle\gamma_{d}/\gamma_{u}=$ $\displaystyle
0.3402,~{}~{}\tau=2.250i,~{}~{}\tilde{\phi}=3\times 10^{-2}.$

The largest hierarchy is $\beta_{u}/\beta_{d}=8.6$ among them. The mass matrix
is given by

$\displaystyle|M_{u}|=$ $\displaystyle\gamma_{u}\begin{pmatrix}2.588\times
10^{-6}&2.632\times 10^{-4}&3.347\times 10^{-3}\\\ 2.352\times
10^{-5}&1.196\times 10^{-3}&4.197\times 10^{-2}\\\ 2.303\times
10^{-3}&1.171\times 10^{-1}&1.490\times
10^{-0}\end{pmatrix}\frac{v_{u}}{\sqrt{2}},$ $\displaystyle|M_{d}|=$
$\displaystyle\gamma_{d}\begin{pmatrix}5.610\times 10^{-6}&1.861\times
10^{-4}&9.051\times 10^{-5}\\\ 5.099\times 10^{-5}&1.380\times
10^{-3}&4.114\times 10^{-4}\\\ 4.994\times 10^{-3}&4.030\times
10^{-2}&4.027\times 10^{-2}\\\ \end{pmatrix}\frac{v_{d}}{\sqrt{2}}.$ (86)

The mixing angles and the mass ratios are calculated as

$\displaystyle y_{u}/y_{t}$ $\displaystyle=5.70\times
10^{-6},~{}~{}y_{c}/y_{t}=2.68\times 10^{-3},y_{d}/y_{b}=6.92.\times
10^{-4},~{}~{}y_{s}/y_{b}=1.37\times 10^{-2},~{}~{}$ $\displaystyle
y_{b}/y_{t}$ $\displaystyle=1.30\times
10^{-2},~{}~{}\theta_{12}=0.227,~{}~{}\theta_{23}=0.359,~{}~{}\theta_{13}=0.00314,~{}~{}\delta_{CP}=1.21,$
(87)

and we can realize all the parameters within $0.1\sigma$ range.
$\chi^{2}<0.01$.

#### Yukawa couplings of ${\bf 3^{\prime}}$ in $\Gamma_{5}^{\prime}$

We investigate the model based on $W_{2}$ in (74). The explicit form of the
$q$-expansion of the modular forms are summarized in Appendix C. We also
assume the lightest up-type quark (up-quark) corresponds to the modular form
of weight 4, and the second lightest up-type quark (charm quark) corresponds
to the modular form of weight 2. Thus the mass matrix is given by

$\displaystyle M_{u}=$
$\displaystyle\begin{pmatrix}\beta_{u}\tilde{\phi}^{I}Y_{3}^{(4)}&\alpha_{u}\tilde{\phi}^{J}Y_{3}^{(2)}&\gamma_{u}\tilde{\phi}^{K}(Y_{3,I}^{(6)}+r_{u}Y_{3,II}^{(6)})\\\
\beta_{u}\tilde{\phi}^{I}Y_{2}^{(4)}&\alpha_{u}\tilde{\phi}^{J}Y_{2}^{(2)}&\gamma_{u}\tilde{\phi}^{K}(Y_{2,I}^{(6)}+r_{u}Y_{2,II}^{(6)})\\\
\beta_{u}\tilde{\phi}^{I}Y_{1}^{(4)}&\alpha_{u}\tilde{\phi}^{J}Y_{1}^{(2)}&\gamma_{u}\tilde{\phi}^{K}(Y_{1,I}^{(6)}+r_{u}Y_{1,II}^{(6)})\\\
\end{pmatrix}\begin{pmatrix}{\rm Im\,}\tau^{2}&&\\\ &{\rm Im\,}\tau^{1}&\\\
&&{\rm Im\,}\tau^{3}\end{pmatrix}\frac{v_{u}}{\sqrt{2}},$ $\displaystyle
M_{d}=$
$\displaystyle\begin{pmatrix}\alpha_{d}\tilde{\phi}^{L}Y_{3}^{(2)}&\beta_{d}\tilde{\phi}^{M}Y_{3}^{(4)}&\gamma_{d}\tilde{\phi}^{N}(Y_{3,I}^{(6)}+r_{d}Y_{3,II}^{(6)})\\\
\alpha_{d}\tilde{\phi}^{L}Y_{2}^{(2)}&\beta_{d}\tilde{\phi}^{M}Y_{2}^{(4)}&\gamma_{d}\tilde{\phi}^{N}(Y_{2,I}^{(6)}+r_{d}Y_{2,II}^{(6)})\\\
\alpha_{d}\tilde{\phi}^{L}Y_{1}^{(2)}&\beta_{d}\tilde{\phi}^{M}Y_{1}^{(4)}&\gamma_{d}\tilde{\phi}^{N}(Y_{1,I}^{(6)}+r_{d}Y_{1,II}^{(6)})\\\
\end{pmatrix}\begin{pmatrix}{\rm Im\,}\tau^{1}&&\\\ &{\rm Im\,}\tau^{2}&\\\
&&{\rm Im\,}\tau^{3}\end{pmatrix}\frac{v_{d}}{\sqrt{2}}.$ (88)

The approximate mixing angles are given by

$\displaystyle\theta_{12}\sim\frac{23}{14}|q^{1/5}|,~{}~{}\theta_{23}\sim
40\sqrt{2}\left|\left(\frac{1}{r_{u}}-\frac{1}{r_{d}}\right)q^{2/5}\right|,~{}~{}\theta_{13}\sim
120\sqrt{2}\left|\left(\frac{1}{r_{u}}-\frac{1}{r_{d}}\right)q^{3/5}\right|.$
(89)

Thus $2|q^{1/5}|\sim\theta_{C}\sim 10^{-1}$, which implies ${\rm Im\,}\tau\sim
2.4$. We obtain

$\displaystyle\theta_{23}$ $\displaystyle\sim
2\left|\frac{1}{r_{u}}-\frac{1}{r_{d}}\right|\times
10^{-1},~{}~{}\theta_{13}\sim
4\left|\frac{1}{r_{u}}-\frac{1}{r_{d}}\right|\times 10^{-2}.$

In this case we can realize the observed values if
$|\frac{1}{r_{u}}-\frac{1}{r_{d}}|\sim 10^{-1}$. On the other hand, conditions
for the mass eigenvalues are given by

$\displaystyle y_{u}/y_{t}$
$\displaystyle\sim\frac{255}{\sqrt{12}|r_{u}|}\left(\frac{14}{23}\right)^{3}\theta_{C}^{3}{\rm
Im\,}\tau^{-1}\frac{\alpha_{u}}{\gamma_{u}}\tilde{\phi}^{I-K}\sim 7\times
10^{-3}\frac{\alpha_{u}}{\gamma_{u}|r_{u}|}\tilde{\phi}^{I-K},$ $\displaystyle
y_{c}/y_{t}$
$\displaystyle\sim\frac{5}{|r_{u}|}\left(\frac{14}{23}\right)^{2}\theta_{C}^{2}{\rm
Im\,}\tau^{-2}\frac{\beta_{u}}{\gamma_{u}}\tilde{\phi}^{J-K}\sim 3\times
10^{-3}\frac{\beta_{u}}{\gamma_{u}|r_{u}|}\tilde{\phi}^{J-K},$ $\displaystyle
y_{d}/y_{b}$
$\displaystyle\sim\frac{10}{|r_{d}|}\left(\frac{14}{23}\right)^{3}\theta_{C}^{3}{\rm
Im\,}\tau^{-2}\frac{\alpha_{d}}{\gamma_{d}}\tilde{\phi}^{L-N}\sim 4\times
10^{-4}\frac{\alpha_{d}}{\gamma_{d}|r_{d}|}\tilde{\phi}^{L-N},$ $\displaystyle
y_{s}/y_{b}$
$\displaystyle\sim\frac{70}{\sqrt{12}|r_{d}|}\left(\frac{14}{23}\right)^{2}\theta_{C}^{2}{\rm
Im\,}\tau^{-1}\frac{\beta_{d}}{\gamma_{d}}\tilde{\phi}^{M-N}\sim 3\times
10^{-2}\frac{\beta_{d}}{\gamma_{d}|r_{d}|}\tilde{\phi}^{M-N},$ $\displaystyle
y_{b}/y_{t}$
$\displaystyle\sim\frac{\gamma_{d}}{\gamma_{u}}\tilde{\phi}^{N-K}.$ (90)

These conditions imply $\tilde{\phi}\sim 10^{-2}$, $I=1,J=K=0$ and $L=M=N=1$.
We can realize the observed values naturally. An explicit model is given by
Table 6, which generates the mass matrix of (88) with $I=1,J=K=0,L=M=N=1$.

| $Q_{1,2,3}$ | $u_{1}^{c}$ | $u_{2}^{c}$ | $u_{2}^{c}$ | $d_{1}^{c}$ | $d_{2}^{c}$ | $d_{3}^{c}$ | $H_{u}$ | $H_{d}$ | $\phi$
---|---|---|---|---|---|---|---|---|---|---
$A_{5}^{\prime}$ | ${\bf 3}^{\prime}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$ | ${\bf 1}$
$k_{I}$ | $-1$ | $-2$ | $-1$ | $-5$ | $0$ | $-2$ | $-4$ | $0$ | $0$ | $-1$
Table 6: The representations and the modular weights of the quarks. This is an
explicit model for superpotential with Yukawa coupling of ${\bf 3^{\prime}}$
in $A_{5}^{\prime}$.

In this model, we can not find a parameter set which reproduces the mixing
angles and the mass ratios whose $\chi^{2}<30$ with the constraints of
$|r_{f}|=1$. Thus we relax this constraint. One benchmark value is given by

$\displaystyle\alpha_{u}/\gamma_{u}=$ $\displaystyle
0.1089,~{}~{}\beta_{u}/\gamma_{u}=0.4899,~{}~{}r_{u}=1.344e^{-3.500i},$
$\displaystyle\alpha_{d}/\gamma_{d}=$ $\displaystyle
1.199,~{}~{}\beta_{d}/\gamma_{d}=2.026,~{}~{}r_{d}=1.719e^{-2.979i},$
$\displaystyle\gamma_{d}/\gamma_{u}=$ $\displaystyle
0.7421~{}~{}\tau=2.560i,~{}~{}\tilde{\phi}=1\times 10^{-2}.$ (91)

We obtain

$\displaystyle|M_{u}|$ $\displaystyle=\gamma_{u}\begin{pmatrix}1.195\times
10^{-4}&2.798\times 10^{-3}&7.022\times 10^{-1}\\\ 8.026\times
10^{-4}&3.490\times 10^{-2}&4.461\times 10^{+0}\\\ 5.046\times
10^{-2}&3.072\times 10^{+0}&7.811\times 10^{+1}\\\
\end{pmatrix}\frac{v_{u}}{\sqrt{2}},$ $\displaystyle|M_{d}|$
$\displaystyle=\gamma_{d}\begin{pmatrix}6.849\times 10^{-5}&2.223\times
10^{-3}&7.251\times 10^{-3}\\\ 8.542\times 10^{-4}&1.493\times
10^{-2}&4.166\times 10^{-2}\\\ 7.518\times 10^{-2}&9.385\times
10^{-1}&9.987\times 10^{-1}\\\ \end{pmatrix}\frac{v_{d}}{\sqrt{2}}.$ (92)

The mixing angles and mass ratios are obtained as

$\displaystyle y_{u}/y_{t}$ $\displaystyle=4.20\times
10^{-6},~{}~{}y_{c}/y_{t}=2.67\times 10^{-3},y_{d}/y_{b}=6.69\times
10^{-4},~{}~{}y_{s}/y_{b}=1.41\times 10^{-2},~{}~{}$ $\displaystyle
y_{b}/y_{t}$ $\displaystyle=1.30\times
10^{-2},~{}~{}\theta_{12}=0.227,~{}~{}\theta_{23}=0.0367,~{}~{}\theta_{13}=0.00304,~{}~{}\delta_{CP}=1.20.$
(93)

In this case $\chi^{2}\sim 0.3$ and realistic observed values are reproduced.

##  5 Stability of parameters

In the previous section we find realistic models which reproduce the observed
quark mass ratios and mixing angles with $\mathcal{O}(1)$ parameters. Our
approximate estimations show that the mixing angles depend only on ${\rm
Im\,}\tau$ and $r_{u,d}$. We expect that they are rather stable prediction in
our model, if the overall coefficients are $\mathcal{O}(1)$. We study the
validity of the approximations used in our models and the stability of the
results against changes of the free parameters. For these purposes, we
investigate the coefficient dependence of the results. In the followings we
use the previous model of Table 6 in $\Gamma_{5}^{\prime}$ as an example.

(a) Distribution of $\theta_{12}$.

(b) Distribution of $\theta_{23}$.

(c) Distribution of $\theta_{13}$.

(d) Distribution of $\delta_{CP}$.

Figure 1: Distribution of the mixing angles and CP-phase. The blue bars denote
the distributions of the mixing angles and the CP-phase. The distributions
have been normalized already. The orange regions denote the $2\sigma$ ranges
of the corresponding value.

(a) Distribution of $y_{u}/y_{t}$.

(b) Distribution of $y_{c}/y_{t}$.

(c) Distribution of $y_{d}/y_{b}$.

(d) Distribution of $y_{s}/y_{b}$.

Figure 2: Distribution of the mass ratios. The blue bars denote the
distributions of the mass ratios. The distributions have been normalized
already. The orange region denotes the $2\sigma$ region of the corresponding
value.

To study the stability, we see the distributions of the results, where we
randomly generate four free parameters of
$(\alpha_{u}/\gamma_{u},\beta_{u}/\gamma_{u},\alpha_{d}/\gamma_{d},\beta_{d}/\gamma_{d})$.
Each parameter is generated as

$\displaystyle
x=\exp(p\log(10^{1/2}))x_{best},~{}~{}~{}~{}x\in\\{\alpha_{u}/\gamma_{u},\beta_{u}/\gamma_{u},\alpha_{d}/\gamma_{d},\beta_{d}/\gamma_{d}\\},$
(94)

where $p$ follows the uniform distribution with minimum $-1$ and maximum $1$.
$x_{best}$ is the best fit value of the coefficients (91). Therefore each of
free coefficients fluctuates within the range of
$\frac{x_{best}}{\sqrt{10}}<x<\sqrt{10}x_{best}$ to keep the same order of
magnitude. Distributions of the mixing angles and the CP-phase are shown in
Figure 1. From the figure we find that the realistic values of the mixing
angles are realized without fine-tuning of the free coefficients. Especially
$\theta_{12}$ as well as $\theta_{23}$ are localized around the observed
values, where approximately half of the configurations reproduces the observed
value of $\theta_{12}$ within $2\sigma$ range, and all the configurations
reproduce the observed value of $\theta_{23}$. While some configurations are
out of $2\sigma$ range for $\theta_{13}$, its order is realized without any
fine-tuning as we expected. $\delta_{CP}$ is also naturally realized, and more
than half of the configurations can reproduce the observed value.

Figure 2 shows the distributions of the mass ratios. Figures 2 and 2 show that
$y_{u}/y_{t}$ and $y_{c}/y_{t}$ are uniformly distributed as we expected. On
the other hand, the mass ratios of the down sectors in Figures 2 and 2 do not
follow the uniform distribution. It is consistent with the fact that $M_{d}$
is less hierarchical than $M_{u}$ and the largest components of the mass
matrix is exchangeable if the coefficients are significantly apart from the
best fit point (See also Figure 3 for singular behavior). However, the mass
ratios are localized around the observed values even for $M_{d}$. Thus the
physical parameters are stable under perturbations of the coefficients.

The parameter dependence of the mixing angles and the mass eigenvalues are
shown in Figure 3. The mass ratios in up sector and down sector depend on
$|r_{u}|$ and $|r_{d}|$, respectively. We also find that $\theta_{12}$
drastically changes, although it is independent of $|r_{u,d}|$ in our
approximation (89). This is because large $|r_{d}|$ can exchange the largest
component of $M_{d}$ as well and our approximation is no longer valid in such
regions. In fact, Figure 3 shows that $\theta_{12}$ have a peak around
$|r_{d}|\sim 3$, and Figure 3 shows that $y_{s}$ and $y_{d}$ get closer at the
same point, where our approximation becomes invalid and the mixing angles are
unstable. Apart from such region, the order of all the mixing angles are
correctly reproduced. We also show the $\alpha_{u,d}$ dependence in
comparison. In contrast to $r_{u,d}$, the mixing angles are almost independent
of $\alpha_{u}$ and $\alpha_{d}$, which is consistent with the approximation
in (89).

Similar results can be obtained in the models with $\Gamma_{3}^{\prime}$ and
$\Gamma_{4}^{\prime}$. Therefore, in any cases of our FN-like models the
approximate estimation is valid and useful to analyze the relation between the
mass ratios and the mixing angles, especially for the up-type mass matrix due
to its large hierarchy. On the other hand, the down-type mass matrix is less
hierarchical and the mixing angles may fluctuate depending on the coefficients
in the down sectors. Nevertheless all of our models can reproduce the correct
orders of the mixing angles and the mass ratios under perturbations of the
$\mathcal{O}(1)$ coefficients.

(a) $|r_{d}|$ dependence of mass ratios.

(b) $|r_{d}|$ dependence of mixing angles.

(c) $|r_{u}|$ dependence of mass ratios.

(d) $|r_{u}|$ dependence of mixing angles.

(e) $\alpha_{d}$ dependence of mass ratios.

(f) $\alpha_{d}$ dependence of mixing angles.

(g) $\alpha_{u}$ dependence of mass ratios.

(h) $\alpha_{u}$ dependence of mixing angles.

Figure 3: Parameter dependence of the mass ratios, mixing angles and CP-phase.
We show $|r_{u,d}|$ dependence since the mixing angles are explicitly depend
on them at the leading order. We also show $\alpha_{d}$ and $\alpha_{u}$
dependence of the parameters for comparison purpose. The observables (except
for the corresponding mass eigenvalues) are almost independent of $\alpha_{u}$
and $\alpha_{d}$.

##  6 Conclusion

We have studied the mass hierarchy and the mixing angles in quark sector based
on the FN-like mechanism in the framework of the modular symmetry. We have
assumed that each of singlet quarks have a common representation under the
modular symmetry but with a different modular weight, which is important to
obtain the CP-phase as well as a full rank mass matrix. We have introduced a
scalar $\phi$ with a negative modular weight. The allowed Yukawa couplings are
then suppressed by powers of the scalar vev ($\langle\phi\rangle/\Lambda$) due
to the FN-like mechanism, and the mass hierarchy is originated from powers of
$\langle\phi\rangle/\Lambda$ and the hierarchical modular forms. Using
$q$-expansions of the modular forms we have analyzed an approximate expression
of the Yukawa matrices, where the same order of suppression factors appears in
each column or each row of the Yukawa matrices, so that our models can simply
realize the FN-like matrices for the up sector and down sector simultaneously.
We have illustrated this mechanism in models with different finite modular
groups of $\Gamma_{3}^{\prime},\Gamma_{4}^{\prime}$ and $\Gamma_{5}^{\prime}$
in detail. As the result, all of our models can reproduce the correct orders
of observed mass ratios and mixing angles by choosing the modular parameters
and the singlet vev. The best model is constructed in the model with
$\Gamma_{4}^{\prime}$, where the Yukawa couplings and the left-handed quarks
$Q$ have to be the same representation to obtain small mixing angles. In this
model, we have approximately reproduced all the 9 observables by tuning 8
parameters, for which we do not require any unnatural hierarchical
coefficients. A statistical investigation has also been carried out to study
the stability of our results. We have shown that the approximate estimation is
valid and our results are stable against perturbations of the $\mathcal{O}(1)$
coefficients, especially for the up sector due to its large hierarchical
structure.

Throughout this paper, we have not studied the lepton sector, while it is
interesting to see if the lepton mass spectra and the neutrino mixing angles
can be realized based on this models. We assume that $\phi$ is stabilized and
is safely decoupled. To stabilize the weighton, we require superpotetial in
terms of $\phi$. The superpotential of $\phi$ is modular invariant as well.
Thus it is also restricted by the weight and representations. We should
investigate its vacuum structure to see if $\phi$ can develop our desired
value of vev. The potential of $\phi$ is also interesting from the
phenomenological point of view. $\phi$ may be related to beyond the SM
physics, such as SUSY breaking, soft-terms, and $\mu$-terms. The deviation
between Majonara mass scale and GUT scale may be originated from the VEV of
$\phi$, too. We assume $\phi$ is the trivial singlet of the finite modular
group. $\phi$ can be non-trivial singlet such as ${\bf\hat{1}}$ in
$S_{4}^{\prime}$, and such $\phi$ possibly change the phenomenology. In this
case $\phi^{4}$ is the trivial singlet, and it looks like a flavon of the
usual $\mathbb{Z}_{4}$ flavor symmetric models. Such a flavon may be important
for physics around the standard model [66]. The stringy origin of our model is
also unclear and it should be investigated. However, these topics are beyond
our scope in this paper, and we will study them elsewhere.

## Acknowledgments

H. O. is supported in part by JSPS KAKENHI Grants No. 17K14309, No. 18H03710,
and No. 21K03554. The authors also thank the Yukawa Institute for Theoretical
Physics at Kyoto University, where this work was initiated during the
YITP-W-20-08 on “Progress in Particle Physics 2020”.

##  Appendix A The modular group of level 3

In this appendix, we briefly review the modular forms of level 3 and develop
our notation. Complete explanation is not our purpose. We would provide a
minimal toolkit necessary for our analysis.

$\Gamma_{3}^{\prime}$ is isomorphic to $T^{\prime}$. $T^{\prime}$ has seven
irreducible representations,

$\displaystyle{\bf
1,1^{\prime},1^{\prime\prime},2,2^{\prime},2^{\prime\prime},3}.$ (95)

$T^{\prime}$ is generated by $S$, $T$ and $R$. Thus it is sufficient to study
the matrix representations of these two elements. The matrix representations
of $T^{\prime}$ have ambiguities. In this paper, we follow the notation of
[27]. The matrix representations are summarized in Table 7.

r | $S$ | $T$ | $R$
---|---|---|---
${\bf 1}$ | 1 | 1 | 1
${\bf 1}^{\prime}$ | 1 | $\omega$ | 1
${\bf 1}^{\prime\prime}$ | 1 | $\omega^{2}$ | 1
${\bf 2}$ | $-\frac{1}{\sqrt{3}}\begin{pmatrix}i&\sqrt{2}e^{i\pi/12}\\\ -\sqrt{2}e^{-i\pi/12}&i\end{pmatrix}$ | $\begin{pmatrix}\omega&0\\\ 0&1\end{pmatrix}$ | $-I_{2\times 2}$
${\bf 2}^{\prime}$ | $-\frac{1}{\sqrt{3}}\begin{pmatrix}i&\sqrt{2}e^{i\pi/12}\\\ -\sqrt{2}e^{-i\pi/12}&i\end{pmatrix}$ | $\omega\begin{pmatrix}\omega&0\\\ 0&1\end{pmatrix}$ | $-I_{2\times 2}$
${\bf 2}^{\prime\prime}$ | $-\frac{1}{\sqrt{3}}\begin{pmatrix}i&\sqrt{2}e^{i\pi/12}\\\ -\sqrt{2}e^{-i\pi/12}&i\end{pmatrix}$ | $\omega^{2}\begin{pmatrix}\omega&0\\\ 0&1\end{pmatrix}$ | $-I_{2\times 2}$
${\bf 3}$ | $-\frac{1}{3}\begin{pmatrix}-1&2&2\\\ 2&-1&2\\\ 2&2&-1\end{pmatrix}$ | $\begin{pmatrix}1&0&0\\\ 0&\omega&0\\\ 0&0&\omega^{2}\end{pmatrix}$ | $I_{3\times 3}$
Table 7: The matrix representations of $S$ and $T$. $\omega=e^{2\pi i/3}$. Our
notation is base on [27].

The irreducible decomposition of the tensor product of the singlets are
trivial:

$\displaystyle{\bf 1}\otimes{\bf 1}={\bf 1}^{\prime}\otimes{\bf
1}^{\prime\prime}={\bf 1},$ $\displaystyle{\bf 1}\otimes{\bf 1}^{\prime}={\bf
1}^{\prime\prime}\otimes{\bf 1}^{\prime\prime}={\bf 1}^{\prime},$
$\displaystyle{\bf 1}\otimes{\bf 1}^{\prime\prime}={\bf 1}^{\prime}\otimes{\bf
1}^{\prime}={\bf 1}^{\prime\prime}.$

We also study the irreducible decomposition of the tensor product including
${\bf 3}$, i.e., Clebsh-Gordon coefficients. It is given by

$\displaystyle\begin{pmatrix}\alpha_{1}\\\ \alpha_{2}\\\ \alpha_{3}\\\
\end{pmatrix}_{\bf 3}\otimes\begin{pmatrix}\beta_{1}\\\ \beta_{2}\\\
\beta_{3}\\\ \end{pmatrix}_{\bf 3}=$
$\displaystyle\left(\alpha_{1}\beta_{1}+\alpha_{2}\beta_{3}+\alpha_{3}\beta_{2}\right)_{\bf
1}+\left(\alpha_{1}\beta_{2}+\alpha_{2}\beta_{1}+\alpha_{3}\beta_{3}\right)_{\bf
1^{\prime}}+\left(\alpha_{1}\beta_{3}+\alpha_{2}\beta_{2}+\alpha_{3}\beta_{1}\right)_{\bf
1^{\prime\prime}}$
$\displaystyle+\begin{pmatrix}2\alpha_{1}\beta_{1}-\alpha_{2}\beta_{3}-\alpha_{3}\beta_{2}\\\
2\alpha_{3}\beta_{3}-\alpha_{1}\beta_{2}-\alpha_{2}\beta_{1}\\\
2\alpha_{2}\beta_{2}-\alpha_{1}\beta_{3}-\alpha_{3}\beta_{1}\end{pmatrix}_{{\bf
3}S}+\begin{pmatrix}\alpha_{2}\beta_{3}-\alpha_{3}\beta_{2}\\\
\alpha_{1}\beta_{2}-\alpha_{2}\beta_{1}\\\
\alpha_{3}\beta_{1}-\alpha_{1}\beta_{3}\\\ \end{pmatrix}_{{\bf 3}A}.$ (96)

### Modular forms

The modular forms of level 3 and weight $k$ are given by tensor products of
the modular forms of level 3 and weight 1 (19). In this appendix, we consider
modular forms which are ${\bf 3}$ of $T^{\prime}$. The complete set of the
modular forms whose weights are lower than 6 can be found in [27]. The modular
forms of weight 2 are given by the tensor product,

$\displaystyle Y_{\bf 3}^{(2)}=(Y_{\bf 2}^{(1)}\otimes Y_{\bf 2}^{(1)})_{{\bf
3}S}=\begin{pmatrix}e^{i\pi/6}Y_{2}^{2}\\\ \sqrt{2}e^{i\pi 7/12}Y_{1}Y_{2}\\\
Y_{1}^{2}\\\ \end{pmatrix}\sim\begin{pmatrix}\frac{1}{9}e^{\pi i/6}\\\
-\frac{2}{3}e^{\pi i/6}q^{1/3}\\\ -2e^{\pi i/6}q^{2/3}\end{pmatrix}.$ (97)

We can obtain the modular forms of higher weights as

$\displaystyle Y_{\bf 3}^{(4)}=\begin{pmatrix}\sqrt{2}e^{i\pi
7/12}Y_{1}^{3}Y_{2}-e^{i\pi/3}Y_{2}^{4}\\\ -Y_{1}^{4}-(1-i)Y_{1}Y_{2}^{3}\\\
3e^{i\pi/6}Y_{1}^{2}Y_{2}^{2}\end{pmatrix}\sim\begin{pmatrix}\frac{1}{81}e^{4\pi
i/3}\\\ \frac{2}{27}e^{4\pi i/3}q^{1/3}\\\ \frac{2}{3}e^{4\pi
i/3}q^{2/3}\end{pmatrix}$ (98)

and

$\displaystyle Y_{{\bf 3},I}^{(6)}$
$\displaystyle=\begin{pmatrix}-2(1-i)Y_{1}^{3}Y_{2}^{3}+iY_{2}^{6}\\\
-4e^{i\pi/6}Y_{1}^{4}Y_{2}^{2}-(1-i)e^{i\pi/6}Y_{1}Y_{2}^{5}\\\
2\sqrt{2}e^{i\pi
7/12}Y_{1}^{5}Y_{2}+e^{i\pi/3}Y_{1}^{2}Y_{2}^{4}\end{pmatrix}\sim\frac{i}{81}\begin{pmatrix}\frac{1}{9}\\\
-\frac{2}{3}q^{1/3}\\\ -2q^{2/3}\end{pmatrix},$ $\displaystyle Y_{{\bf
3},II}^{(6)}$
$\displaystyle=\begin{pmatrix}Y_{1}^{6}-2(1-i)Y_{1}^{3}Y_{2}^{3}\\\
e^{i\pi/6}Y_{1}^{4}Y_{2}^{2}-2(1-i)e^{i\pi/6}Y_{1}Y_{2}^{5}\\\
-4e^{i\pi/3}Y_{1}^{2}Y_{2}^{4}+(1+i)e^{i\pi/3}Y_{1}^{5}Y_{2}\end{pmatrix}\sim\frac{i}{81}\begin{pmatrix}-24q\\\
-\frac{4}{3}q^{1/3}\\\ 8q^{2/3}\end{pmatrix}.$ (99)

These expansions are consistent with (27).

##  Appendix B The modular forms of level 4

$\Gamma_{4}^{\prime}$ is isomorphic to $S_{4}^{\prime}\simeq
SL(2,\mathbb{Z}_{4})$, which is a double covering group of $S_{4}$.
$S_{4}^{\prime}$ has the following irreducible representations,

$\displaystyle{\bf
1,\hat{1},1^{\prime},\hat{1}^{\prime},2,\hat{2},3,\hat{3},3^{\prime},\hat{3}^{\prime}}.$
(100)

Our notation follows [29]. The matrix representations are summarized in Table
8.

r | $S$ | $T$ | $R$
---|---|---|---
${\bf 1}$ | 1 | 1 | 1
$\hat{\bf 1}$ | i | $-i$ | $-1$
${\bf 1}^{\prime}$ | $-1$ | $-1$ | $1$
$\hat{\bf 1}^{\prime}$ | $-i$ | $i$ | $-1$
${\bf 2}$ | $\frac{1}{2}\begin{pmatrix}-1&\sqrt{3}\\\ -\sqrt{3}&1\end{pmatrix}$ | $\begin{pmatrix}1&0\\\ 0&-1\end{pmatrix}$ | $I_{2\times 2}$
$\hat{\bf 2}$ | $\frac{i}{2}\begin{pmatrix}-1&\sqrt{3}\\\ -\sqrt{3}&1\end{pmatrix}$ | $\begin{pmatrix}-i&0\\\ 0&i\end{pmatrix}$ | $-I_{2\times 2}$
${\bf 3}$ | $-\frac{1}{2}\begin{pmatrix}0&\sqrt{2}&\sqrt{2}\\\ \sqrt{2}&-1&1\\\ \sqrt{2}&1&-1\end{pmatrix}$ | $\begin{pmatrix}-1&0&0\\\ 0&-i&0\\\ 0&0&i\end{pmatrix}$ | $I_{3\times 3}$
$\hat{\bf 3}$ | $-\frac{i}{2}\begin{pmatrix}0&\sqrt{2}&\sqrt{2}\\\ \sqrt{2}&-1&1\\\ \sqrt{2}&1&-1\end{pmatrix}$ | $\begin{pmatrix}i&0&0\\\ 0&-1&0\\\ 0&0&1\end{pmatrix}$ | $-I_{3\times 3}$
${\bf 3}^{\prime}$ | $\frac{1}{2}\begin{pmatrix}0&\sqrt{2}&\sqrt{2}\\\ \sqrt{2}&-1&1\\\ \sqrt{2}&1&-1\end{pmatrix}$ | $\begin{pmatrix}1&0&0\\\ 0&i&0\\\ 0&0&-i\end{pmatrix}$ | $I_{3\times 3}$
$\hat{\bf 3}^{\prime}$ | $\frac{i}{2}\begin{pmatrix}0&\sqrt{2}&\sqrt{2}\\\ \sqrt{2}&-1&1\\\ \sqrt{2}&1&-1\end{pmatrix}$ | $\begin{pmatrix}-i&0&0\\\ 0&1&0\\\ 0&0&-1\end{pmatrix}$ | $-I_{3\times 3}$
Table 8: The matrix representations of the generators of
$\Gamma_{4}^{\prime}$. Our notation is base on [29].

The complete table of the irreducible decomposition of the tensor products,
and their CG coefficients also be found in [29]. We summarize necessary part
here. $\hat{\bf 1}$ corresponds to $e^{i\pi/2}$. $\hat{\bf 1}^{2}={\bf
1}^{\prime}$, and $({\bf 1}^{\prime})^{2}={\bf 1}$. Thus the irreducible
decomposition of the tensor product of the singlets are trivial:

$\displaystyle{\bf 1}\otimes{\bf 1}=\hat{\bf 1}\otimes\hat{\bf
1}^{\prime}={\bf 1}^{\prime}\otimes{\bf 1}^{\prime}={\bf 1},~{}~{}{\bf
1}\otimes{\bf 1}^{\prime}=\hat{\bf 1}\otimes\hat{\bf 1}=\hat{\bf
1}^{\prime}\otimes\hat{\bf 1}^{\prime}={\bf 1}^{\prime},$ $\displaystyle{\bf
1}\otimes\hat{\bf 1}={\bf 1}^{\prime}\otimes\hat{l\bf 1}^{\prime}=\hat{\bf
1},~{}~{}{\bf 1}\otimes\hat{\bf 1}^{\prime}={\bf 1}^{\prime}\otimes\hat{\bf
1}=\hat{\bf 1}^{\prime},$

We also require the irreducible decomposition of the tensor products including
the triplets, and their Clebsh-Gordon coefficients. They are classified to two
cases. For the first case, the irreducible decomposition are given by

$\displaystyle{\bf 3}\otimes{\bf 3}$ $\displaystyle={\bf 1}\oplus{\bf
2}\oplus{\bf 3}\oplus{\bf 3}^{\prime},$ $\displaystyle{\bf 3}\otimes\hat{\bf
3}$ $\displaystyle=\hat{\bf 1}\oplus\hat{\bf 2}\oplus\hat{\bf 3}\oplus\hat{\bf
3}^{\prime},$ $\displaystyle{\bf 3}^{\prime}\otimes{\bf 3}^{\prime}$
$\displaystyle={\bf 1}\oplus{\bf 2}\oplus{\bf 3}\oplus{\bf 3}^{\prime},$
$\displaystyle{\bf 3}^{\prime}\otimes\hat{\bf 3}^{\prime}$
$\displaystyle=\hat{\bf 1}\oplus\hat{\bf 2}\oplus\hat{\bf 3}\oplus\hat{\bf
3}^{\prime},$ $\displaystyle\hat{\bf 3}\otimes\hat{\bf 3}$ $\displaystyle={\bf
1}\oplus{\bf 2}\oplus{\bf 3}\oplus{\bf 3}^{\prime},$

and the CG-coefficients for these five cases are given as follows

$\displaystyle\begin{pmatrix}\alpha_{1}\\\ \alpha_{2}\\\ \alpha_{3}\\\
\end{pmatrix}\otimes\begin{pmatrix}\beta_{1}\\\ \beta_{2}\\\ \beta_{3}\\\
\end{pmatrix}=$
$\displaystyle\frac{1}{\sqrt{3}}\left(\alpha_{1}\beta_{1}+\alpha_{2}\beta_{3}+\alpha_{3}\beta_{2}\right)+\frac{1}{\sqrt{2}}\begin{pmatrix}(2\alpha_{1}\beta_{1}-\alpha_{2}\beta_{3}-\alpha_{3}\beta_{2})/\sqrt{3}\\\
\alpha_{2}\beta_{2}+\alpha_{3}\beta_{3}\end{pmatrix}$
$\displaystyle+\frac{1}{\sqrt{2}}\begin{pmatrix}\alpha_{3}\beta_{3}-\alpha_{2}\beta_{2}\\\
\alpha_{1}\beta_{3}+\alpha_{3}\beta_{1}\\\
-\alpha_{1}\beta_{2}-\alpha_{2}\beta_{1}\\\
\end{pmatrix}+\frac{1}{\sqrt{2}}\begin{pmatrix}\alpha_{3}\beta_{2}-\alpha_{2}\beta_{3}\\\
\alpha_{2}\beta_{1}-\alpha_{1}\beta_{2}\\\
\alpha_{1}\beta_{3}-\alpha_{3}\beta_{1}\\\ \end{pmatrix}.$ (101)

For the second case, the irreducible decomposition of the tensor products are
given by

$\displaystyle{\bf 3}\otimes{\bf 3}^{\prime}$ $\displaystyle={\bf
1}^{\prime}\oplus{\bf 2}\oplus{\bf 3}\oplus{\bf 3}^{\prime},$
$\displaystyle{\bf 3}\otimes\hat{\bf 3}^{\prime}$ $\displaystyle=\hat{\bf
1}^{\prime}\oplus\hat{\bf 2}\oplus\hat{\bf 3}\oplus\hat{\bf 3}^{\prime},$
$\displaystyle{\bf 3}^{\prime}\otimes\hat{\bf 3}$ $\displaystyle=\hat{\bf
1}^{\prime}\oplus\hat{\bf 2}\oplus\hat{\bf 3}\oplus\hat{\bf 3}^{\prime},$
$\displaystyle\hat{\bf 3}\otimes\hat{\bf 3}$ $\displaystyle={\bf
1}^{\prime}\oplus{\bf 2}\oplus{\bf 3}\oplus{\bf 3}^{\prime},$
$\displaystyle\hat{\bf 3}^{\prime}\otimes\hat{\bf 3}^{\prime}$
$\displaystyle={\bf 1}^{\prime}\oplus{\bf 2}\oplus{\bf 3}\oplus{\bf
3}^{\prime},$

and the CG-coefficients are summarized by

$\displaystyle\begin{pmatrix}\alpha_{1}\\\ \alpha_{2}\\\ \alpha_{3}\\\
\end{pmatrix}\otimes\begin{pmatrix}\beta_{1}\\\ \beta_{2}\\\ \beta_{3}\\\
\end{pmatrix}=$
$\displaystyle\frac{1}{\sqrt{3}}\left(\alpha_{1}\beta_{1}+\alpha_{2}\beta_{3}+\alpha_{3}\beta_{2}\right)+\frac{1}{\sqrt{2}}\begin{pmatrix}\alpha_{2}\beta_{2}+\alpha_{3}\beta_{3}\\\
(-2\alpha_{1}\beta_{1}+\alpha_{2}\beta_{3}+\alpha_{3}\beta_{2})/\sqrt{3}\end{pmatrix}$
$\displaystyle+\frac{1}{\sqrt{2}}\begin{pmatrix}\alpha_{3}\beta_{2}-\alpha_{2}\beta_{3}\\\
\alpha_{2}\beta_{1}-\alpha_{1}\beta_{2}\\\
\alpha_{1}\beta_{3}-\alpha_{3}\beta_{1}\\\
\end{pmatrix}+\frac{1}{\sqrt{2}}\begin{pmatrix}\alpha_{3}\beta_{3}-\alpha_{2}\beta_{2}\\\
\alpha_{1}\beta_{3}+\alpha_{3}\beta_{1}\\\
-\alpha_{1}\beta_{2}-\alpha_{2}\beta_{1}\\\ \end{pmatrix}.$ (102)

### Modular forms

The modular forms of level 4 and weight $k$ are given by tensor products of
the modular forms of level 4 and weight 1. The modular forms of weight 1 form
$\hat{\bf 3}$ of $S_{4}^{\prime}$, and they are given by

$\displaystyle
Y_{\bf\hat{3}}^{(1)}(\tau)=\begin{pmatrix}\sqrt{2}\varepsilon\theta\\\
\varepsilon^{2}\\\ -\theta^{2}\end{pmatrix},$

where $\varepsilon$ and $\theta$ are given by

$\displaystyle\varepsilon(\tau)\equiv\frac{2\eta^{2}(4\tau)}{\eta(2\tau)},~{}~{}\theta(\tau)\equiv\frac{\eta^{5}(2\tau)}{\eta^{2}(\tau)\eta^{2}(4\tau)},$

where $\eta(\tau)$ is the Dedekind eta function. $\varepsilon(\tau)$ and
$\theta(\tau)$ are expanded by

$\displaystyle\varepsilon(\tau)$
$\displaystyle=2\sum_{k=1}^{\infty}q^{\frac{(2k-1)^{2}}{4}}=2q^{\frac{1}{4}}+2q^{\frac{9}{4}}+2q^{\frac{25}{4}}+...$
$\displaystyle\theta(\tau)$
$\displaystyle=1+2\sum_{k=1}^{\infty}q^{\frac{(2k)^{2}}{4}}=1+2q^{1}+2q^{4}+2q^{9}+...$

$\varepsilon(\tau)\sim 2q^{1/4}$ and $\theta(\tau)\sim 1$ for large ${\rm
Im\,}\tau$. The modular forms of higher weights are constructed by their
tensor products. We summarize the irreducible decompositions of
$\mathcal{M}_{k}(\Gamma(4))$ in Table 9.

weight $k$ | representations
---|---
1 | ${\bf\hat{3}}$
2 | ${\bf 2,3^{\prime}}$
3 | ${\bf\hat{1}^{\prime},\hat{3},\hat{3}^{\prime}}$
4 | ${\bf 1,2,3,3^{\prime}}$
5 | ${\bf\hat{2}},2\times{\bf\hat{3}},{\bf\hat{3}^{\prime}}$
6 | ${\bf 1,1^{\prime},2,2^{\prime},3},2\times{\bf 3^{\prime}}$
⋮ | ⋮
Table 9: Irreducible decomposition of $\mathcal{M}_{k}(\Gamma(4))$.

We concentrate on the modular forms which are triplets of $S_{4}^{\prime}$
since our Yukawa couplings are triplets.111111The complete set of the modular
forms whose weights are lower than 8 can be found in [29]. The modular forms
of weight 2 are given by

$\displaystyle Y_{3^{\prime}}^{(2)}(\tau)=(Y_{\hat{3}}^{(1)}\otimes
Y_{\hat{3}}^{(1)})_{3^{\prime}}=\begin{pmatrix}\frac{1}{\sqrt{2}}(\theta^{4}-\epsilon^{4})\\\
-2\epsilon\theta^{3}\\\
-2\epsilon^{3}\theta\end{pmatrix}\sim\begin{pmatrix}\frac{1}{\sqrt{2}}\\\
-4q^{1/4}\\\ -16q^{3/4}\end{pmatrix}.$

The modular forms of weight 3 are given by

$\displaystyle Y_{\hat{\bf 3}}^{(3)}(\tau)=$
$\displaystyle\begin{pmatrix}\varepsilon\theta^{5}+\varepsilon^{5}\theta\\\
\frac{1}{2\sqrt{2}}(5\varepsilon^{2}\theta^{4}-\varepsilon^{6})\\\
\frac{1}{2\sqrt{2}}(\theta^{6}-5\varepsilon^{4}\theta^{2})\end{pmatrix}\sim\begin{pmatrix}2q^{1/4}\\\
\frac{10}{\sqrt{2}}q^{2/4}\\\
\frac{1}{2\sqrt{2}}\end{pmatrix},~{}~{}Y_{\hat{\bf
3}^{\prime}}^{(3)}(\tau)=\frac{1}{2}\begin{pmatrix}-4\sqrt{2}\varepsilon^{3}\theta^{3}\\\
\theta^{6}+3\epsilon^{4}\theta^{2}\\\
-3\varepsilon^{2}\theta^{4}-\varepsilon^{6}\end{pmatrix}\sim\begin{pmatrix}-16\sqrt{2}q^{3/4}\\\
1/2\\\ -6q^{2/4}\end{pmatrix}.$

The modular forms of weight 4 are given by

$\displaystyle Y_{{\bf 3}}^{(4)}(\tau)=$
$\displaystyle\frac{3}{2\sqrt{2}}\begin{pmatrix}\sqrt{2}(\varepsilon^{2}\theta^{6}-\varepsilon^{6}\theta^{2})\\\
\varepsilon^{3}\theta^{5}-\varepsilon^{7}\theta\\\
-\varepsilon\theta^{7}+\varepsilon^{5}\theta^{3}\end{pmatrix}\sim\frac{3}{2\sqrt{2}}\begin{pmatrix}4\sqrt{2}q^{2/4}\\\
8q^{3/4}\\\ -2q^{1/4}\end{pmatrix},$ $\displaystyle Y_{{\bf
3}^{\prime}}^{(4)}(\tau)=$
$\displaystyle\begin{pmatrix}\frac{1}{4}(\theta^{8}-\varepsilon^{8})\\\
\frac{1}{2\sqrt{2}}(\varepsilon\theta^{7}+7\varepsilon^{5}\theta^{3})\\\
\frac{1}{2\sqrt{2}}(7\varepsilon^{3}\theta^{5}+\varepsilon^{7}\theta)\end{pmatrix}\sim\begin{pmatrix}\frac{1}{4}\\\
\frac{1}{\sqrt{2}}q^{1/3}\\\ \frac{28}{\sqrt{2}}q^{3/4}\end{pmatrix}.$

The modular forms of weight 5 are given by

$\displaystyle Y_{\hat{\bf 3},I}^{(5)}(\tau)=$
$\displaystyle\begin{pmatrix}\frac{6\sqrt{2}}{\sqrt{5}}\varepsilon^{5}\theta^{5}\\\
\frac{3}{8\sqrt{5}}(5\varepsilon^{2}\theta^{8}+10\varepsilon^{6}\theta^{4}+\varepsilon^{10})\\\
-\frac{3}{8\sqrt{5}}(\theta^{10}+10\varepsilon^{5}\theta^{6}+5\varepsilon^{8}\theta^{2})\end{pmatrix}\sim\begin{pmatrix}\frac{192\sqrt{2}}{\sqrt{5}}q^{5/4}\\\
\frac{15}{2\sqrt{5}}q^{2/4}\\\ -\frac{3}{8\sqrt{5}}\end{pmatrix},$
$\displaystyle~{}~{}Y_{\hat{\bf 3},II}^{(5)}(\tau)=$
$\displaystyle\begin{pmatrix}\frac{3}{4}(\varepsilon\theta^{9}-2\varepsilon^{5}\theta^{5}+\varepsilon^{9}\theta)\\\
\frac{3}{\sqrt{2}}(-\varepsilon^{2}\theta^{8}+\varepsilon^{6}\theta^{4})\\\
\frac{3}{\sqrt{2}}(-\varepsilon^{4}\theta^{6}+\varepsilon^{8}\theta^{2})\end{pmatrix}\sim\begin{pmatrix}\frac{3}{2}q^{1/4}\\\
-\frac{12}{\sqrt{2}}q^{2/4}\\\ -\frac{48}{\sqrt{2}}q\end{pmatrix},$
$\displaystyle Y_{\hat{\bf 3}^{\prime}}^{(5)}(\tau)=$
$\displaystyle\begin{pmatrix}2(\varepsilon^{3}\theta^{7}+\varepsilon^{7}\theta^{3})\\\
\frac{1}{4\sqrt{2}}(\theta^{10}-14\varepsilon^{4}\theta^{6}-3\varepsilon^{8}\theta^{2})\\\
\frac{1}{4\sqrt{2}}(3\varepsilon^{2}\theta^{8}+14\varepsilon^{6}\theta^{4}-\varepsilon^{10})\end{pmatrix}\sim\begin{pmatrix}16q^{3/4}\\\
\frac{1}{4\sqrt{2}}\\\ \frac{3}{\sqrt{2}}q^{2/4}\end{pmatrix}.$

The modular forms of weight 6 are given by

$\displaystyle Y_{{\bf 3}}^{(6)}(\tau)=$
$\displaystyle\begin{pmatrix}\frac{3}{2}(\varepsilon^{2}\theta^{10}-\varepsilon^{10}\theta^{2})\\\
\frac{3}{4\sqrt{2}}(5\varepsilon^{3}\theta^{9}-6\varepsilon^{7}\theta^{5}+\varepsilon^{11}\theta)\\\
\frac{3}{4\sqrt{2}}(\varepsilon\theta^{11}-6\varepsilon^{5}\theta^{7}+5\varepsilon^{9}\theta^{3})\end{pmatrix}\sim\begin{pmatrix}6q^{2/4}\\\
\frac{30}{\sqrt{2}}q^{3/4}\\\ \frac{3}{2\sqrt{2}}q^{1/4}\end{pmatrix},$
$\displaystyle Y_{{\bf 3}^{\prime},I}^{(6)}(\tau)=$
$\displaystyle\begin{pmatrix}-\frac{3}{8\sqrt{13}}(\theta^{12}-3\varepsilon^{4}\theta^{8}+3\varepsilon^{8}\theta^{4}-\varepsilon^{12})\\\
\frac{3\sqrt{2}}{\sqrt{13}}(3\varepsilon^{5}\theta^{7}+\varepsilon^{9}\theta^{3})\\\
\frac{3\sqrt{2}}{\sqrt{13}}(\varepsilon^{3}\theta^{9}+3\varepsilon^{7}\theta^{5})\\\
\end{pmatrix}\sim\begin{pmatrix}-\frac{3}{8\sqrt{13}}\\\
\frac{36\sqrt{2}}{\sqrt{13}}q^{5/4}\\\ \frac{24\sqrt{2}}{\sqrt{13}}q^{3/4}\\\
\end{pmatrix},$ $\displaystyle Y_{{\bf 3}^{\prime},II}^{(6)}(\tau)=$
$\displaystyle\begin{pmatrix}3(\varepsilon^{4}\theta^{8}-\varepsilon^{8}\theta^{4})\\\
-\frac{3}{4\sqrt{2}}(\varepsilon^{5}\theta^{11}+2\varepsilon^{5}\theta^{7}-3\varepsilon^{9}\theta^{3})\\\
\frac{3}{4\sqrt{2}}(3\varepsilon^{3}\theta^{9}-2\varepsilon^{7}\theta^{5}-\varepsilon^{11}\theta)\\\
\end{pmatrix}\sim\begin{pmatrix}48q\\\ -\frac{24}{\sqrt{2}}q^{5/4}\\\
\frac{18}{\sqrt{2}}q^{3/4}\\\ \end{pmatrix}.$

For $k=7$, we have

$\displaystyle Y_{\hat{\bf 3}^{\prime},I}^{(7)}(\tau)=$
$\displaystyle\begin{pmatrix}\frac{3}{4\sqrt{37}}(7\varepsilon^{3}\theta^{9}+50\varepsilon^{7}\theta^{7}+7\varepsilon^{11}\theta^{3})\\\
-\frac{3}{4\sqrt{74}}(\theta^{14}+14\varepsilon^{4}\theta^{10}+49\varepsilon^{8}\theta^{6})\\\
\frac{3}{4\sqrt{74}}(49\varepsilon^{6}\theta^{8}+14\varepsilon^{10}\theta^{4}+\varepsilon^{14})\\\
\end{pmatrix}\sim\begin{pmatrix}\frac{42}{\sqrt{37}}q^{3/4}\\\
-\frac{3}{4\sqrt{74}}\\\ \frac{2352}{\sqrt{74}}q^{6/4}\\\ \end{pmatrix},$
$\displaystyle Y_{\hat{\bf 3}^{\prime},II}^{(7)}(\tau)=$
$\displaystyle\begin{pmatrix}\frac{9}{4}(\varepsilon^{3}\theta^{11}-2\varepsilon^{7}\theta^{7}+\varepsilon^{11}\theta^{3})\\\
\frac{9}{4\sqrt{2}}(\varepsilon^{4}\theta^{10}-2\varepsilon^{8}\theta^{6}+\varepsilon^{12}\theta^{2})\\\
-\frac{9}{4\sqrt{2}}(\varepsilon^{2}\theta^{12}-2\varepsilon^{6}\theta^{8}+\varepsilon^{10}\theta^{4})\\\
\end{pmatrix}\sim\begin{pmatrix}18q^{3/4}\\\ \frac{36}{\sqrt{2}}q\\\
-\frac{9}{\sqrt{2}}q^{2/4}\\\ \end{pmatrix}.$

For $k=8$, we have

$\displaystyle Y_{{\bf 3},I}^{(8)}(\tau)=$
$\displaystyle\begin{pmatrix}9\sqrt{\frac{2}{5}}(\varepsilon^{6}\theta^{10}-\varepsilon^{10}\theta^{6})\\\
\frac{9}{16\sqrt{5}}(5\varepsilon^{3}\theta^{13}+5\varepsilon^{7}\theta^{9}-9\varepsilon^{11}\theta^{5}-\varepsilon^{15}\theta)\\\
-\frac{9}{16\sqrt{5}}(\varepsilon\theta^{15}+9\varepsilon^{5}\theta^{11}-5\varepsilon^{9}\theta^{7}-5\varepsilon^{13}\theta^{3})\\\
\end{pmatrix}\sim\begin{pmatrix}576\sqrt{\frac{2}{5}}q^{6/4}\\\
\frac{9\sqrt{5}}{2}q^{3/4}\\\ -\frac{9}{8\sqrt{5}}q^{1/4}\\\ \end{pmatrix},$
$\displaystyle~{}~{}Y_{{\bf 3},II}^{(8)}(\tau)=$
$\displaystyle\begin{pmatrix}-\frac{9}{8}(\varepsilon^{2}\theta^{14}-2\varepsilon^{6}\theta^{10}+3\varepsilon^{10}\theta^{6}-\varepsilon^{14}\theta^{2})\\\
\frac{9}{2\sqrt{2}}(\varepsilon^{3}\theta^{13}-2\varepsilon^{7}\theta^{9}+\varepsilon^{11}\theta^{5})\\\
-\frac{9}{2\sqrt{2}}(\varepsilon^{5}\theta^{11}-2\varepsilon^{9}\theta^{7}+\varepsilon^{13}\theta^{3})\\\
\end{pmatrix}\sim\begin{pmatrix}-\frac{9}{2}q^{2/4}\\\
\frac{36}{\sqrt{2}}q^{3/4}\\\ -\frac{144}{\sqrt{2}}q^{5/4}\\\ \end{pmatrix}.$

These $q$-expansions are consistent with (67).

##  Appendix C The modular forms of level 5

$\Gamma_{5}^{\prime}$ is isomorphic to $A_{5}^{\prime}\simeq
SL(2,\mathbb{Z}_{5})$, which is a double covering group of $A_{5}$.
$A_{5}^{\prime}$ has the following irreducible representations,

$\displaystyle{\bf
1,\hat{2},\hat{2}^{\prime},3,3^{\prime},4,\hat{4},5,\hat{6}}.$

Our notation follows [31]. We concentrate on the triplet representations.
Their matrix representations are summarized in Table 8.

r | $S$ | $T$ | $R$
---|---|---|---
${\bf 1}$ | 1 | 1 | 1
${\bf 3}$ | $\frac{1}{\sqrt{5}}\begin{pmatrix}1&-\sqrt{2}&-\sqrt{2}\\\ -\sqrt{2}&-(\sqrt{5}+1)/2&(\sqrt{5}-1)/2\\\ -\sqrt{2}&(\sqrt{5}-1)/2&-(\sqrt{5}+1)/2\end{pmatrix}$ | $\begin{pmatrix}1&0&0\\\ 0&\sigma&0\\\ 0&0&\sigma^{4}\end{pmatrix}$ | $I_{3\times 3}$
${\bf 3}^{\prime}$ | $\frac{1}{\sqrt{5}}\begin{pmatrix}-1&\sqrt{2}&\sqrt{2}\\\ \sqrt{2}&(-\sqrt{5}+1)/2&(\sqrt{5}+1)/2\\\ \sqrt{2}&(\sqrt{5}+1)/2&(-\sqrt{5}+1)/2\end{pmatrix}$ | $\begin{pmatrix}1&0&0\\\ 0&\sigma^{2}&0\\\ 0&0&\sigma^{3}\end{pmatrix}$ | $I_{3\times 3}$
Table 10: The matrix representations of the generators of
$\Gamma_{5}^{\prime}$. $\sigma=e^{2i\pi/5}$. We concentrate on the modular
forms of 3-dimensional representations, and omit the other representations in
this table. Our notation is base on [31].

The complete table of the irreducible decomposition of the tensor products,
and their CG coefficients also can be found in [31]. The irreducible
decomposition of the tensor products of the triplets are given by

$\displaystyle{\bf 3}\otimes{\bf 3}$ $\displaystyle={\bf 1}\oplus{\bf
3}\oplus{\bf 5},$ $\displaystyle{\bf 3}^{\prime}\otimes{\bf 3}^{\prime}$
$\displaystyle={\bf 1}\oplus{\bf 3}^{\prime}\oplus{\bf 5},$ $\displaystyle{\bf
3}\otimes{\bf 3}^{\prime}$ $\displaystyle={\bf 4}\oplus{\bf 5},$

and their CG-coefficients are given by

$\displaystyle\begin{pmatrix}\alpha_{1}\\\ \alpha_{2}\\\ \alpha_{3}\\\
\end{pmatrix}_{\bf 3}\otimes\begin{pmatrix}\beta_{1}\\\ \beta_{2}\\\
\beta_{3}\\\ \end{pmatrix}_{\bf 3}=$
$\displaystyle\frac{1}{\sqrt{3}}\left(\alpha_{1}\beta_{1}+\alpha_{2}\beta_{3}+\alpha_{3}\beta_{2}\right)+\frac{1}{\sqrt{2}}\begin{pmatrix}\alpha_{2}\beta_{3}-\alpha_{3}\beta_{2}\\\
\alpha_{1}\beta_{2}-\alpha_{2}\beta_{1}\\\
\alpha_{3}\beta_{1}-\alpha_{1}\beta_{3}\\\
\end{pmatrix}+\frac{1}{\sqrt{6}}\begin{pmatrix}2\alpha_{1}\beta_{1}-\alpha_{2}\beta_{3}-\alpha_{3}\beta_{2}\\\
-\sqrt{3}(\alpha_{1}\beta_{2}+\alpha_{2}\beta_{1})\\\
\sqrt{6}\alpha_{2}\beta_{2}\\\ \sqrt{6}\alpha_{3}\beta_{3}\\\
\sqrt{3}(\alpha_{1}\beta_{3}+\alpha_{3}\beta_{1})\\\ \end{pmatrix},$
$\displaystyle\begin{pmatrix}\alpha_{1}\\\ \alpha_{2}\\\ \alpha_{3}\\\
\end{pmatrix}_{\bf 3^{\prime}}\otimes\begin{pmatrix}\beta_{1}\\\ \beta_{2}\\\
\beta_{3}\\\ \end{pmatrix}_{\bf 3^{\prime}}=$
$\displaystyle\frac{1}{\sqrt{3}}\left(\alpha_{1}\beta_{1}+\alpha_{2}\beta_{3}+\alpha_{3}\beta_{2}\right)+\frac{1}{\sqrt{2}}\begin{pmatrix}\alpha_{2}\beta_{3}-\alpha_{3}\beta_{2}\\\
\alpha_{1}\beta_{2}-\alpha_{2}\beta_{1}\\\
\alpha_{3}\beta_{1}-\alpha_{1}\beta_{3}\end{pmatrix}+\frac{1}{\sqrt{6}}\begin{pmatrix}2\alpha_{1}\beta_{1}-\alpha_{2}\beta_{3}-\alpha_{3}\beta_{2}\\\
\sqrt{6}\alpha_{3}\beta_{3}\\\
-\sqrt{3}(\alpha_{1}\beta_{2}+\alpha_{2}\beta_{1})\\\
\sqrt{3}(\alpha_{1}\beta_{3}+\alpha_{3}\beta_{1})\\\
\sqrt{6}\alpha_{2}\beta_{2}\end{pmatrix},$
$\displaystyle\begin{pmatrix}\alpha_{1}\\\ \alpha_{2}\\\ \alpha_{3}\\\
\end{pmatrix}_{\bf 3}\otimes\begin{pmatrix}\beta_{1}\\\ \beta_{2}\\\
\beta_{3}\\\ \end{pmatrix}_{\bf 3^{\prime}}=$
$\displaystyle\frac{1}{\sqrt{3}}\begin{pmatrix}\sqrt{2}\alpha_{2}\beta_{1}+\alpha_{3}\beta_{2}\\\
-\sqrt{2}\alpha_{1}\beta_{2}-\alpha_{3}\beta_{3}\\\
-\sqrt{2}\alpha_{1}\beta_{3}-\alpha_{2}\beta_{2}\\\
\sqrt{2}\alpha_{3}\beta_{1}+\alpha_{2}\beta_{3}\end{pmatrix}+\frac{1}{\sqrt{3}}\begin{pmatrix}\sqrt{3}\alpha_{2}\beta_{1}+\alpha_{3}\beta_{2}\\\
\alpha_{2}\beta_{1}-\sqrt{2}\alpha_{3}\beta_{2}\\\
\alpha_{1}\beta_{2}-\sqrt{2}\alpha_{3}\beta_{3}\\\
\alpha_{1}\beta_{3}-\sqrt{2}\alpha_{2}\beta_{2}\\\
\alpha_{3}\beta_{1}-\sqrt{2}\alpha_{2}\beta_{3}\\\ \end{pmatrix}.$ (103)

### Modular forms

The modular forms of level 5 and weight $k$ are given by tensor products of
the modular forms of level 5 and weight 1. The modular forms of weight 1 form