aainstitutetext: Department of Physics, POSTECH,
Pohang 37673, Koreabbinstitutetext: Asia Pacific Center for Theoretical
Physics,
67 Cheongam-ro, Nam-gu, Pohang 37673, Koreaccinstitutetext: School of Physics,
University of Electronic Science and Technology of China,
No. 2006 Xiyuan Ave, West Hi-Tech Zone, Chengdu, Sichuan 611731, China

# Topological vertex for 6d SCFTs with $\mathbb{Z}_{2}$-twist

Hee-Cheol Kim a Minsung Kim c Sung-Soo Kim

###### Abstract

We compute the partition function for 6d $\mathcal{N}=1$ $SO(2N)$ gauge
theories compactified on a circle with $\mathbb{Z}_{2}$ outer automorphism
twist. We perform the computation based on 5-brane webs with two O5-planes
using topological vertex with two O5-planes. As representative examples, we
consider 6d $SO(8)$ and $SU(3)$ gauge theories with $\mathbb{Z}_{2}$ twist. We
confirm that these partition functions obtained from the topological vertex
with O5-planes indeed agree with the elliptic genus computations.

##  1 Introduction

In Kim:2019dqn , it was proposed that 5-brane webs Aharony:1997ju ;
Aharony:1997bh  for 6d $\mathcal{N}=(1,0)$ superconformal field theories
(SCFTs) with $SO(N)$ gauge symmetry coupled to a tensor multiplet on a circle
Heckman:2015bfa . Such 5-brane webs are constructed with two O5-planes whose
separation can be naturally identified with the Kaluza-Klein (KK) momentum
Hayashi:2015vhy ; Kim:2017jqn . Given a 5-brane web, one can compute the
prepotential Witten:1996qb ; Intriligator:1997pq  for the corresponding
theories on Coulomb branch. It was checked Kim:2019dqn  that the prepotentials
obtained from the proposed 5-branes webs indeed agree with the prepotentials
that one can compute the triple intersection numbers in their geometric
descriptions Jefferson:2018irk ; Bhardwaj:2019fzv .

Of particular significance to this 5-brane construction of 6d SCFTs with
$SO(N)$ gauge symmetry is a realization of such 6d SCFTs with $\mathbb{Z}_{2}$
twist, providing a new perspective on RG flows on Higgs branches of D-type
conformal matters Heckman:2013pva ; DelZotto:2014hpa . With non-zero
holonomies turned on, one introduces a light charged scalar mode carrying non-
zero KK momentum along the 6d circle, giving rise to new Higgs branch
associated with a vev of the light mode. In particular, RG flows from
Higgsings on two O5-planes leads to 5-brane webs for twisted compactifications
of 6d theories.

In this paper, we compute $\mathbb{R}^{4}\times T^{2}$ partition functions for
6d theories with $\mathbb{Z}_{2}$ twist, based on 5-brane configurations
proposed in Kim:2019dqn . As representative examples, we consider two 6d SCFTs
with $\mathbb{Z}_{2}$ twist: one is the 6d $SO(8)$ gauge theory with
$\mathbb{Z}_{2}$ twist and the other is the 6d $SU(3)$ theory with
$\mathbb{Z}_{2}$ twist. The $SO(8)$ theory is obtained through the Higgsing
sequence from the 6d $SO(10)$ gauge theory with two hypermultiplets in the
fundamental representation (flavors) to the $SO(9)$ gauge theory with a flavor
and then to the $SO(8)$ theory with $\mathbb{Z}_{2}$ twist, while the $SU(3)$
theory is obtained through the Higgsing sequence from 6d $G_{2}$ gauge theory
with a flavor to the $SU(3)$ theory with $\mathbb{Z}_{2}$ twist, whose 5-brane
configuration yields 5d $SU(3)_{9}$ gauge theory with the Chern-Simons level
$9$, as predicted in Jefferson:2017ahm ; Jefferson:2018irk ; Hayashi:2018lyv .

As a main tool of computation, we implement topological vertex formalism
Aganagic:2003db ; Iqbal:2007ii ; Awata:2008ed  with an O5-plane Kim:2017jqn
applying these Higgsing sequences. We check the obtained results against the
elliptic genus Haghighat:2013tka ; Kim:2014dza ; Haghighat:2014vxa ;
Gadde:2015tra . computations by applying the Higgsings leading to the twisted
theories or directly twisting.

The organization of the paper is as follows. In section 2, we discuss the
construction of 5-brane webs for the 6d $SO(8)$ gauge theory with
$\mathbb{Z}_{2}$ twist and compute the partition function using topological
vertex based on 5-brane webs as well as using the ADHM method. In a similar
way, the 6d $SU(3)$ gauge theory with $\mathbb{Z}_{2}$ twist is discussed in
section 3. We then summarize the results and discuss some subtle issues in
section 4. In appendices A and B, we discuss properties of $\mathbb{Z}_{2}$
twisting of 6d theories and the perturbative partition function from the
perspective of twisted affine Lie algebras, and also provide various
identities that are useful in actual computations.

While completing this paper, we became aware of Hayashi:2020hhb  which has
some overlap with this paper.

##  2 $SO(8)$ theory with $\mathbb{Z}_{2}$ twist

We first consider twisted compactification of 6d pure $SO(8)$ gauge theory on
$-4$ curve. As proposed in Kim:2019dqn , a 5-brane configuration for the
twisted compactification of the 6d pure $SO(8)$ gauge theory can be obtained
from a Higgsing of 6d $SO(10)$ gauge theory with two hypermultiplets in the
fundamental representation ($SO(10)+2\mathbf{F}$) on $-4$ curves. Before we
discuss twisting, let us recall the standard Higgsing, which is to give the
vev’s to two fundamental scalars charged under a color D5-brane. This leads to
the 6d $SO(8)$ gauge theory on $-4$ curve. To realize twisted
compactification, one instead gives independent vev’s to two individual
scalars. Namely, first give a vev to one fundamental scalar to yield the 6d
$SO(9)$ gauge theory with a fundamental hypermultiplet ($SO(9)+1\mathbf{F}$),
and then give a different vev to another fundamental scalar carrying a unit KK
momentum. This leads to the 6d $SO(8)$ gauge theory on $-4$ curve with
${\mathbb{Z}}_{2}$ twist. In this section, we first review construction of
5-brane configuration leading to the twisted compactification of the 6d pure
$SO(8)$ gauge theory on $-4$ curve, and then compute the partition function of
the twisted $SO(8)$ gauge theory on $\mathbb{R}^{4}\times T^{2}$, using
topological vertex Aganagic:2003db  on a 5-brane web with two O5-planes
Kim:2017jqn . We compare our result with Higgsing or twisting of the elliptic
genus from the ADHM construction using localization technique, introduced in
Benini:2013nda ; Benini:2013xpa .

Figure 1: A 5-brane web for 6d $SO(10)$ theory with two fundamentals. Two
fundamental hypermultiplets are represented by two external D5-branes ending
on a D7-brane (black dot) and the monodromy cuts of D7-branes point the
outward directions.

###  2.1 5-brane web for 6d $SO(8)$ gauge theory with $\mathbb{Z}_{2}$ twist

Let us begin with a 5-brane configuration for the 6d $SO(10)$ gauge theory
with two fundamental hypermultiplets on $-4$ curve. It is depicted in Figure
1, where there are two O5-planes whose separation naturally gives a
compactification direction associated with 6d circle. Two fundamental
hypermultiplets are denoted by two D5-branes ending on D7-branes. Their masses
are $m_{1},m_{2}$. The W-bosons $W_{i}$ of the theory are denoted by wiggly
lines connecting color D5-branes in Figure 1. The masses of the W-bosons are
given by

$\displaystyle m_{W_{1}}$ $\displaystyle=2\phi_{0}-\phi_{2}\ ,$
$\displaystyle\quad m_{W_{2}}$ $\displaystyle=2\phi_{1}-\phi_{2}\ ,$
$\displaystyle m_{W_{3}}$ $\displaystyle=2\phi_{2}-\phi_{0}-\phi_{1}-\phi_{3}\
,$ $\displaystyle m_{W_{4}}$
$\displaystyle=2\phi_{3}-\phi_{2}-\phi_{4}-\phi_{5}\ ,$ $\displaystyle
m_{W_{5}}$ $\displaystyle=2\phi_{4}-\phi_{3}\ ,$ $\displaystyle\quad
m_{W_{6}}$ $\displaystyle=2\phi_{5}-\phi_{3}\ ,$ (1)

where $\phi_{i}$ ($i=1,\cdots,5$) are the Coulomb branch vev and $\phi_{0}$ is
the tensor branch vev. As it is a configuration for the 6d theory on a circle,
one can see that the W-bosons masses are consistent with the affine Cartan
matrix111The affine Cartan matrix is defined as
$\mathcal{C}_{ij}=\displaystyle
2\frac{(\alpha_{i},\alpha_{j})}{(\alpha_{j},\alpha_{j})}$ with the simple
roots $\alpha_{i}$ of affine Lie algebras.  of untwisted affine Lie algebra
$D_{5}^{(1)}$,

$\displaystyle
m_{W_{i}}=\big{(}\mathcal{C}_{D_{5}^{(1)}}\big{)}_{ij}\,\phi_{j-1}\
,\qquad\quad\mathcal{C}_{D_{5}^{(1)}}=\begin{pmatrix}[r]2&0&-1&0&0&0\\\
0&2&-1&0&0&0\\\ -1&-1&2&-1&0&0\\\ 0&0&-1&2&-1&-1\\\ 0&0&0&-1&2&0\\\
0&0&0&-1&0&2\end{pmatrix}.$ (2)

(a)

(b)

(c)

(d)

Figure 2: (a) Another 5-brane web of 6d $SO(10)$ theory with two flavors after
applying a series of flop transitions in Figure 1. (b) The bottom part of (a)
after applying Hanany-Witten transition. The black circle and black zigzag
line denote D7-brane and corresponding monodromy cut, respectively. (c)
Putting a D7-brane and a color D5-brane to the orientifold plane on the
bottom. A D7-brane is split into two half D7-branes and generates
$\widetilde{\mathrm{O}5}^{-}$ plane. The half D7-branes and corresponding half
monodromy cuts are denoted by blue dots and blue zigzag lines. The half
D5-brane is denoted by violet line. (d) A 5-brane configuration after Higgsing
away two half D5-branes between D7-branes and then taking two half D7-branes
to the left and right infinities, respectively.

As discussed, to twist, we perform the Higgsing such that we first bring a
D7-brane down to one of the O5-planes to obtain $SO(9)$ gauge theory with a
fundamental as depicted in Figure 2, and then push up the remaining D7-brane
to the other O5-plane to give a different vev to the remaining scalar, as in
Figure 3. More precisely, the Higgsing from a $SO(10)$ gauge theory with a
flavor to a $SO(9)$ gauge theory is achieved as follows: Through successive
flop transitions, one can bring a D7-brane near the bottom O5-plane as
depicted in Figure 2(2(a)). Then one brings the D7-brane inside the Coulomb
branch of $SO(10)$ gauge theory, where the D7-brane becomes floating as in
Figure 2(2(d)), since the flavor D5-brane is annihilated due to the Hanany-
Witten transition. We give a vev to the flavor which locates the flavor
7-brane and a color D5-brane on an O5-plane. That is to set the masses of the
flavor hypermultiplet and the associated W-boson $\phi_{1}-\phi_{0}$ to zero,

$\displaystyle m_{1}=0\ ,\qquad\phi_{1}=\phi_{0}\ .$ (3)

We note that when it is placed on an O5-plane as depicted in Figure 2(2(d))
and Figure 2(2(d)), a D7-brane (black dot) is split into two half D7-branes
(blue dots), creating a half D5-brane between two half D7-branes stuck on an
O5-plane, which hence turns the O5-plane between these split half D7-branes to
an $\widetilde{{\rm O5}}^{-}$-plane Evans:1997hk ; Giveon:1998sr ; Feng:2000eq
; Bertoldi:2002nn . In doing so, a new Higgs branch opens up such that two
half color D5-branes (or a full color D5-brane) are suspended between two half
D7-branes on the $\widetilde{{\rm O5}}^{-}$-plane which can be Higgsed away,
resulting in a 5-brane configuration for a $SO(9)$ gauge theory. After moving
half D7-branes away from each other to the opposite along the orientifold
plane, one gets a 5-brane configuration with an $\widetilde{{\rm O5}}$-plane
as in Figure 2(2(d)).

(a)

(b)

Figure 3: (a) The 5-brane web obtained by applying Figure 2(b)-(d) to the
bottom part of Figure 2(a). (b) Flop transition of (a).

Figure 3(3(a)) is the resulting 5-brane web diagram describing the 6d $SO(9)$
gauge theory on a circle with a fundamental hypermultiplet of mass $m_{2}$,
implementing (3). Here we have redefined the scalars $\phi_{i}$

$\displaystyle\phi_{0}\to\phi_{4}\ ,\qquad\phi_{2}\to\phi_{3}\
,\qquad\phi_{3}\to\phi_{2}\ ,\qquad\phi_{4}\to\phi_{0}\
,\qquad\phi_{5}\to\phi_{1}\ ,$ (4)

to make the masses of the W-bosons form the affine Cartan matrix of untwisted
affine Lie algebra $B_{4}^{(1)}$,

$\displaystyle m_{W_{1}}$ $\displaystyle=2\phi_{0}-\phi_{2}\ ,$ $\displaystyle
m_{W_{2}}$ $\displaystyle=2\phi_{1}-\phi_{2}\ ,$ $\displaystyle m_{W_{3}}$
$\displaystyle=2\phi_{2}-\phi_{3}-\phi_{0}-\phi_{1}\ ,$ $\displaystyle
m_{W_{4}}$ $\displaystyle=2\phi_{3}-2\phi_{4}-\phi_{2}\ ,$ $\displaystyle
m_{W_{5}}$ $\displaystyle=2\phi_{4}-\phi_{3}\ ,$ (5)

or more explicitly,

$\displaystyle
m_{W_{i}}=\big{(}\mathcal{C}_{B_{4}^{(1)}}\big{)}_{ij}\,\phi_{j-1}\
,\qquad\quad\mathcal{C}_{B_{4}^{(1)}}=\begin{pmatrix}[r]2&0&-1&0&0\\\
0&2&-1&0&0\\\ -1&-1&2&-1&0\\\ 0&0&-1&2&-2\\\ 0&0&0&-1&2\end{pmatrix}.$ (6)

We further Higgs the remaining fundamental hypermultiplet. As the 5-brane
configuration given in Figure 3 has two different types of orientifold planes,
O5- and $\widetilde{\rm O5}$-planes, we have two ways of Higgsing the
remaining hypermultiplet. In the previous Higgsing $SO(10)+2\mathbf{F}\to
SO(9)+1\mathbf{F}$, the Higgsing procedure converts an O5-plane to an
$\widetilde{\rm O5}$-plane or vice versa. In a similar fashion, if one brings
a flavor D5-brane (or equivalently a flavor D7-brane) down to the
$\widetilde{\rm O5}$-plane, then it is to perform the same Higgsing, and hence
it would yield a 5-brane web for a pure $SO(8)$ theory on a circle. This is
the standard Higgsing $SO(9)+1\mathbf{F}\to SO(8)$, hence corresponding to
untwisted compactification $SO(10)+2\mathbf{F}\to SO(9)+1\mathbf{F}\to SO(8)$.
Indeed, shifting $\phi_{3}\to\phi_{3}+\phi_{4}$ straightforwardly yields the
W-boson masses forming the affine Cartan matrix of untwisted affine Lie
algebra $D_{4}^{(1)}$, as expected,

$\displaystyle m_{W_{1}}$ $\displaystyle=2\phi_{0}-\phi_{2}\ ,$ $\displaystyle
m_{W_{2}}$ $\displaystyle=2\phi_{1}-\phi_{2}\ ,$ $\displaystyle m_{W_{3}}$
$\displaystyle=2\phi_{2}-\phi_{3}-\phi_{4}-\phi_{0}-\phi_{1}\ ,$
$\displaystyle m_{W_{4}}$ $\displaystyle=2\phi_{3}-\phi_{2}\ ,$ $\displaystyle
m_{W_{5}}$ $\displaystyle=2\phi_{4}-\phi_{2}\ .$ (7)

Figure 4: The 5-brane web of $SO(8)$ theory with $\mathbb{Z}_{2}$ twist.

Now, let us take the flavor D5-brane and one of color D5-branes to the ${\rm
O5}$-plane instead. One can do the same kind of Higgsing as done for the
previous the $SO(10)$ theory to the $SO(9)$ theory, which then makes a 5-brane
configuration with two $\widetilde{\rm O5}$-planes. As discussed in
Kim:2019dqn , this is the Higgsing that one Higgses the $SO(9)$ theory with
one fundamental hypermultiplet by giving a vev to a scalar field carrying
Kaluza-Klein momentum which leads to the $SO(8)$ gauge theory with a
$\mathbb{Z}_{2}$ twist. Namely, we first revive the 6d circle radius $R$ in
Figure 3(3(a)) by introducing Kaluza-Klein momentum, which can be done by
redefining Kähler parameters to be

$\displaystyle 2\phi_{0}-\phi_{2}\to 2\phi_{0}-\phi_{2}+\frac{1}{R}\
,\quad\qquad\phi_{1}-\phi_{0}\to\phi_{1}-\phi_{0}-\frac{1}{2R}\ ,$ (8)

where other parameters are unaltered in Figure 3(3(a)). By successive flop
transitions, the fundamental hypermultiplet can be placed near the O5-plane
located on the top in Figure 3(3(a)). This yields a 5-brane configuration in
Figure 3(3(b)). We then Higgs by setting

$\displaystyle m_{2}=\frac{1}{2R}\
,\qquad\qquad\phi_{1}=\phi_{0}+\frac{1}{2R}\ .$ (9)

The resulting 5-brane configuration for the 6d $SO(8)$ gauge theory with a
$\mathbb{Z}_{2}$ twist is depicted in Figure 4. Here the W-boson masses are
given by

$\displaystyle m_{W_{1}}$ $\displaystyle=2\phi_{0}-\phi_{2}+\frac{1}{R}\ ,$
$\displaystyle m_{W_{2}}$
$\displaystyle=2\phi_{2}-2\phi_{0}-\phi_{3}-\frac{1}{2R}\ ,$ $\displaystyle
m_{W_{3}}$ $\displaystyle=2\phi_{3}-2\phi_{4}-\phi_{2}\ ,$ $\displaystyle
m_{W_{4}}$ $\displaystyle=2\phi_{4}-\phi_{3}\ ,$ (10)

which form the affine Cartan matrix of twisted affine Lie algebra
$D_{4}^{(2)}$,

$\displaystyle\mathcal{C}_{D_{4}^{(2)}}=\left(\begin{array}[]{rrrr}2&-1&0&0\\\
-2&2&-1&0\\\ 0&-1&2&-2\\\ 0&0&-1&2\end{array}\right).$ (15)

###  2.2 Partition function from 5-brane webs

We now compute the partition function based on the 5-brane configuration for
the 6d $SO(8)$ gauge theory with a $\mathbb{Z}_{2}$ twist, using topological
vertex with an O5-plane Kim:2017jqn . With an O5-plane, we can easily realize
a 5d $SO(N)$ or $Sp(N)$ gauge theory with hypermultiplets in the fundamental
representation. For $SO(N)$ gauge theories, we can also depict 5-brane
configurations for hypermultiplets in the spinor representation Zafrir:2015ftn
. Our strategy to carry out topological vertex computation with O5-planes is
first to introduce auxiliary spinor hypermultiplets and then to decouple them
after the computation.

(a)

(b)

(c)

Figure 5: (a) Couple two spinors after Higgsing Figure 2(c). (b) Moving two
half D7-branes and corresponding monodromy cuts to far left. (c) Applying
generalized flop transition to spinors.

(a)

(b)

Figure 6: A 5-brane web diagram for $\mathbb{Z}_{2}$ twisted compactification
of 6d $SO(8)$ theory with four auxiliary spinors.

To introduce four auxiliary spinors, we start with a 5-brane configuration
before twisting. Namely, we consider 5-brane web for 6d $SO(10)$ gauge theory
with two hypermultiplets in the fundamental representation as well as four
auxiliary hypermultiplets in the spinor representation. One can imagine that
given a 5-brane web for 6d $SO(10)$ gauge theory with two fundamentals as in
Figure 2(2(a)), one introduces each spinor as a charge conserving distant
5-brane on the left and on the right of the main web for the $SO(10)$ gauge
theory. On the bottom O5-plane, they are a 5-brane of charge $(2,-1)$ on the
left and a 5-brane of charge $(2,1)$ on the right. On the top O5-plane, they
are a 5-brane of charge $(2,-1)$ on the left and a 5-brane of charge $(2,-1)$
on the right. Higgsing toward $SO(8)$ gauge theory with a twist is
straightforward except there are additional half D5-branes introduced along
O5-planes due to Hanany-Witten transitions when taking half D7-brane to
infinity. For instance, in Figure 5, Higgsing from $SO(10)$ gauge theory with
a fundamental is depicted in the presence of two spinors, a distant 5-brane of
charge $(1,-1)$ due to the monodromy of D7-brane (fundamental hyper) on the
left and another distant 5-brane of charge $(2,1)$ on the right of the bottom
O5-plane. Figure 5(5(a)) is a configuration when the Higgsing is performed,
where there are two half D7-branes are on the orientifold plane. Unlike the
previous case where we took each half D7-brane to the opposite directions to
realize an $\widetilde{\rm O5}$-plane, this time, for computation ease, we
take two half D7-branes to the same direction, to the left, to keep the
orientifold plane an O5-plane. This is depicted in Figure 5(5(b)), where half
D5-branes are created due to the Hanany-Witten transitions. Taking the spinor
5-branes closer to the center of the O5-plane, they undergo “generalized flop”
transitions Hayashi:2017btw  but still preserve the charge conservation as
giving in Figure 5(5(c)). Repeating the same procedure on the top O5-plane,
one finds a 5-brane web for the 6d $SO(8)$ gauge theory with four auxiliary
spinors with a $\mathbb{Z}_{2}$ twist, as in Figure 6. Again, to obtain the
partition function for the 6d $SO(8)$ gauge theory with a $\mathbb{Z}_{2}$
twist, we first compute the partition function based on the 5-brane web for
the 6d $SO(8)$ gauge theory with four spinors with the twist in Figure 6, and
then decouple the spinors by taking all the masses of the spinor matter to
infinity.

To perform the topological vertex method with an O5-plane, we assign arrows,
Young diagrams $\lambda,\mu_{i},\nu_{i}$ and Kähler parameters $Q_{i}$ on the
edges as in Figure 7. The topological string partition function $Z$ can be
computed by evaluating the edge factor and the vertex factor:

$\displaystyle
Z=\sum_{\lambda,\mu,\nu,\cdots}\quantity(\prod\mathrm{Edge\>Factor})\quantity(\prod\mathrm{Vertex\>Factor})\
.$ (16)

The Edge Factor is given by the product of Kähler parameters and the framing
factor of the associated Young diagram
$\lambda=(\lambda_{1},\lambda_{2},\cdots,\lambda_{\ell(\lambda)})$,

$\displaystyle(-Q)^{|\lambda|}f^{n}_{\lambda}\ ,$ (17)

where the framing factor is defined as

$\displaystyle
f_{\lambda}=(-1)^{\absolutevalue{\lambda}}g^{\frac{1}{2}(\norm*{\lambda^{t}}^{2}-\norm*{\lambda}^{2})}=f_{\lambda^{t}}^{-1}\
,$ (18)

with

$\displaystyle\absolutevalue{\lambda}=\sum_{i=1}^{\ell(\lambda)}\lambda_{i}\
,\qquad\norm{\lambda}^{2}=\sum_{i=1}^{\ell(\lambda)}\lambda_{i}^{2}\ .$ (19)

Here, $g=e^{-\epsilon}$ is the $\Omega$-deformation parameter that is
unrefined $\epsilon_{1}=-\epsilon_{2}=\epsilon$, and $\lambda^{t}$ denotes the
transposed Young diagram which corresponds to the conjugate partition of
$\lambda$. The power of the framing factor $n$ is defined with the incoming
and outgoing edges connected to $Q$ and their antisymmetric product, for
instance, in Figure 7, $n=u_{i}\wedge v_{2}=\det(u_{1},v_{2})$.

Figure 7: The example of 5-brane web where $\lambda,\mu_{i},\nu_{i}$ are Young
diagram associated with edges, $Q$ is a Kähler parameter associated with edge
$\lambda$, and $u_{i},v_{i}$ are $(p,q)$-charges of four edges.

The Vertex Factor is given by

$\displaystyle
C_{\lambda\mu\nu}=g^{\frac{1}{2}(-\norm*{\mu^{t}}^{2}+\norm*{\mu}^{2}+\norm*{\nu}^{2})}\tilde{Z}_{\nu}(g)\sum_{\eta}s_{\lambda^{t}/\eta}(g^{-\rho-\nu})s_{\mu/\eta}(g^{-\rho-\nu^{t}})\
,$ (20)

where $\lambda,\mu,\nu$ are the Young diagrams assigned to three out-going
edges connected to a given vertex, arranged in the clockwise order such that
the last Young diagram $\nu$ is assigned to the preferred direction which is
associated with instanton sum. We note that
$C_{\lambda\mu\nu}=C_{\mu\nu\lambda}=C_{\nu\lambda\mu}$ for unrefined vertex
factors. When the orientation of an edge is reversed, the corresponding Young
diagram is transposed. Here, $\tilde{Z}_{\nu}(g)$ is defined by

$\displaystyle\tilde{Z}_{\nu}(g)$
$\displaystyle=\tilde{Z}_{\nu^{t}}(g)=\prod_{i=1}^{\ell(\nu)}\prod_{j=1}^{\nu_{i}}\quantity(1-g^{\nu_{i}+\nu_{j}^{t}-i-j+1})^{-1},$
(21)

where $s_{\lambda/\eta}(x)$ is skew Shur functions and
$\rho=(-\frac{1}{2},-\frac{3}{2},-\frac{5}{2},\cdots)$. When summing skew Shur
functions, one needs to repeatedly use the Cauchy identities

$\displaystyle\sum_{\lambda}Q^{\absolutevalue{\lambda}}s_{\lambda/\eta_{1}}(g^{-\rho-\nu_{1}})s_{\lambda/\eta_{2}}(g^{-\rho-\nu_{2}})$
$\displaystyle\quad=\mathcal{R}_{\nu_{2}\nu_{1}}(Q)^{-1}\sum_{\lambda}Q^{\absolutevalue{\eta_{1}}+\absolutevalue{\eta_{2}}-\absolutevalue{\lambda}}s_{\eta_{2}/\lambda}(g^{-\rho-\nu_{1}})s_{\eta_{1}/\lambda}(g^{-\rho-\nu_{2}})\
,$ (22)
$\displaystyle\sum_{\lambda}Q^{\absolutevalue{\lambda}}s_{\lambda/\eta_{1}^{t}}(g^{-\rho-\nu_{1}})s_{\lambda^{t}/\eta_{2}}(g^{-\rho-\nu_{2}})$
$\displaystyle\quad=\mathcal{R}_{\nu_{2}\nu_{1}}(-Q)\sum_{\lambda}Q^{\absolutevalue{\eta_{1}}+\absolutevalue{\eta_{2}}-\absolutevalue{\lambda}}s_{\eta_{2}^{t}/\lambda}(g^{-\rho-\nu_{1}})s_{\eta_{1}/\lambda^{t}}(g^{-\rho-\nu_{2}})\
,$ (23)

where

$\displaystyle\mathcal{R}_{\lambda\mu}(Q)$
$\displaystyle=\prod^{\infty}_{i,j=1}\left(1-Q\,g^{i+j-\mu_{i}-\lambda_{j}-1}\right).$
(24)

See also Appendix B for various other Cauchy identities and other related
special functions.

Now, we assign the Young diagrams and Kähler parameters to a 5-brane web
diagram for the 6d $SO(8)$ gauge theory with four auxiliary spinors as in
Figure 8. We compute the partition function based on this 5-brane web with two
O5-planes as follows: First we reflect the 5-brane web with respect to each
O5-plane, which creates a mirror image of the original 5-brane web. We also
cut the 5-brane configuration by half so that we can glue over the color
D5-branes associated with Young diagrams $\mu_{1},\mu_{2},\mu_{3}$. As we have
mirror images, we can choose a convenient fundamental 5-brane configuration,
which contains both the original and the mirror images such that when crossing
the O5-plane, 5-brane webs are smoothly connected. Together with the mirror
images, we then see that we can extract two pieces of strip diagrams given in
Figure 9.

Figure 8: 5-brane web diagram for $SO(8)$ theory with a $\mathbb{Z}_{2}$ twist
where we first couple four auxiliary spinors and then later decouple all the
spinors after obtaining the partition function. In this way, one can compute
the partition function for 6d $SO(8)$ theory with a $\mathbb{Z}_{2}$ twist.

(a)

(b)

Figure 9: Strip diagrams for 6d $SO(8)$ gauge theory with $\mathbb{Z}_{2}$
twist, which is coupled to 4 auxiliary spinors. They are the left part (a) and
the right part (b) of Figure 8. We use mirror images to make each part as a
form of a strip diagram.

We note here in Figure 9 that unlike the edge factors $\mu_{i}$ following the
conventional rule, the edge factors of $\nu_{i}$ should be treated carefully.
When $\nu_{i}$ are reflected over the orientifold planes, they get the
additional factor $(-1)^{\absolutevalue{\nu_{i}}}f_{\nu_{i}}^{\pm 1}$, where
$\pm 1$ is determined from the orientation of the edges. For the vertex
factor, the assigned arrows associated with Young diagrams are reversed when
they are reflected over, which makes the ordering of the corresponding Young
diagrams counterclockwise. It also follows from the identity Kim:2017jqn

$\displaystyle
C_{\lambda\mu\nu}=(-1)^{|\lambda|+|\mu|+|\nu|}f_{\lambda}^{-1}(g)f_{\mu}^{-1}(g)f_{\nu}^{-1}(g)C_{\mu^{t}\lambda^{t}\nu^{t}}\
,$ (25)

that the reversal of the Young diagrams is translated into the transposition
of the Young diagrams, and in turn, it is equivalent to changing the direction
of the arrows.

The topological string partition function is then obtained by gluing two strip
diagrams in Figure 9, by summing up the Young diagrams $\mu_{i}$ and $\nu_{i}$
which are associated with the preferred direction

$\displaystyle Z$
$\displaystyle=\sum_{\mu_{i},\nu_{i}}Q_{1}^{\absolutevalue{\nu_{1}}}Q_{3}^{\absolutevalue{\nu_{2}}}Q_{5}^{\absolutevalue{\nu_{3}}}Q_{7}^{\absolutevalue{\nu_{4}}}(-Q_{B}Q_{10})^{\absolutevalue{\mu_{1}}}(-Q_{B})^{\absolutevalue{\mu_{2}}}(-Q_{B}Q_{9})^{\absolutevalue{\mu_{3}}}$
$\displaystyle\qquad\quad\times
f_{\mu_{1}}^{2}f_{\mu_{3}}^{-2}f_{\nu_{1}}^{3}f_{\nu_{2}}f_{\nu_{3}}f_{\nu_{4}}^{3}Z^{\rm
strip}_{\mathrm{left}}(\mu_{1},\mu_{2},\mu_{3},\nu_{1},\nu_{2})Z^{\rm
strip}_{\mathrm{right}}(\mu_{1},\mu_{2},\mu_{3},\nu_{3},\nu_{4})\ ,$ (26)

where two strip parts are as follows: With the shorthand notation
$Q_{i,j,k,\cdots}=Q_{i}Q_{j}Q_{k}\cdots$, the left strip part, in Figure
9(9(a)), is given by

$\displaystyle Z_{\mathrm{left}}^{\rm
strip}(\mu_{1},\mu_{2},\mu_{3},\nu_{1},\nu_{2})$
$\displaystyle=g^{\frac{1}{2}(\norm*{\mu_{1}}^{2}+\norm*{\mu_{2}}^{2}+\norm*{\mu_{3}}^{2}+2\norm*{\nu_{1}}^{2}+2\norm*{\nu_{2}^{t}}^{2})}\tilde{Z}_{\mu_{1}}(g)\tilde{Z}_{\mu_{2}}(g)\tilde{Z}_{\mu_{3}}(g)\tilde{Z}_{\nu_{1}}(g)^{2}\tilde{Z}_{\nu_{2}}(g)^{2}$
$\displaystyle~{}~{}\times\frac{\mathcal{R}_{\emptyset\emptyset}(Q_{1,2,3,4,9,10})\mathcal{R}_{\emptyset\mu_{1}}(Q_{3,4,9,10})\mathcal{R}_{\emptyset\mu_{2}}(Q_{3,4,9})\mathcal{R}_{\emptyset\mu_{3}}(Q_{3,4})\mathcal{R}_{\emptyset\nu_{1}}(Q_{1,1,2,3,4,9,10})}{\mathcal{R}_{\emptyset\mu_{1}^{t}}(Q_{1,2})\mathcal{R}_{\emptyset\mu_{2}^{t}}(Q_{1,2,10})\mathcal{R}_{\emptyset\mu_{3}^{t}}(Q_{1,2,9,10})\mathcal{R}_{\emptyset\nu_{1}^{t}}(Q_{2,3,4,9,10})\mathcal{R}_{\emptyset\nu_{2}}(Q_{1,2,3,3,4,9,10})\mathcal{R}_{\mu_{1}\mu_{2}^{t}}(Q_{10})}$
$\displaystyle~{}~{}\times\frac{\mathcal{R}_{\emptyset\nu_{2}^{t}}(Q_{1,2,4,9,10})\mathcal{R}_{\mu_{1}^{t}\nu_{1}^{t}}(Q_{2})\mathcal{R}_{\mu_{2}^{t}\nu_{1}^{t}}(Q_{2,10})\mathcal{R}_{\mu_{3}^{t}\nu_{1}^{t}}(Q_{2,9,10})\mathcal{R}_{\mu_{1}\nu_{2}^{t}}(Q_{4,9,10})\mathcal{R}_{\mu_{2}\nu_{2}^{t}}(Q_{4,9})}{\mathcal{R}_{\mu_{1}\mu_{3}^{t}}(Q_{9,10})\mathcal{R}_{\mu_{2}\mu_{3}^{t}}(Q_{9})\mathcal{R}_{\mu_{1}^{t}\nu_{1}}(Q_{1,1,2})\mathcal{R}_{\mu_{2}^{t}\nu_{1}}(Q_{1,1,2,10})\mathcal{R}_{\mu_{3}^{t}\nu_{1}}(Q_{1,1,2,9,10})\mathcal{R}_{\mu_{1}\nu_{2}}(Q_{3,3,4,9,10})}$
$\displaystyle~{}~{}\times\frac{\mathcal{R}_{\mu_{3}\nu_{2}^{t}}(Q_{4})\mathcal{R}_{\nu_{1}\nu_{1}}(Q_{1,1})\mathcal{R}_{\nu_{2}\nu_{2}}(Q_{3,3})\mathcal{R}_{\nu_{1}^{t}\nu_{2}}(Q_{2,3,3,4,9,10})\mathcal{R}_{\nu_{1}\nu_{2}^{t}}(Q_{1,1,2,4,9,10})}{\mathcal{R}_{\mu_{2}\nu_{2}}(Q_{3,3,4,9})\mathcal{R}_{\mu_{3}\nu_{2}}(Q_{3,3,4})\mathcal{R}_{\nu_{1}\nu_{2}}(Q_{1,1,2,3,3,4,9,10})\mathcal{R}_{\nu_{1}^{t}\nu_{2}^{t}}(Q_{2,4,9,10})}\
,$ (27)

and the right strip part, in Figure 9(9(b)), is given by

$\displaystyle Z_{\mathrm{right}}^{\rm
strip}(\mu_{1},\mu_{2},\mu_{3},\nu_{3},\nu_{4})$
$\displaystyle=g^{\frac{1}{2}(\norm*{\mu_{1}^{t}}^{2}+\norm*{\mu_{2}^{t}}^{2}+\norm*{\mu_{3}^{t}}^{2}+2\norm*{\nu_{3}^{t}}^{2}+2\norm*{\nu_{4}}^{2})}\tilde{Z}_{\mu_{1}}(g)\tilde{Z}_{\mu_{2}}(g)\tilde{Z}_{\mu_{3}}(g)\tilde{Z}_{\nu_{3}}(g)^{2}\tilde{Z}_{\nu_{4}}(g)^{2}$
$\displaystyle~{}~{}\times\frac{\mathcal{R}_{\emptyset\emptyset}(Q_{5,6,7,8,9,10})\mathcal{R}_{\emptyset\mu_{1}^{t}}(Q_{5,6})\mathcal{R}_{\emptyset\mu_{2}^{t}}(Q_{5,6,10})\mathcal{R}_{\emptyset\mu_{3}^{t}}(Q_{5,6,9,10})\mathcal{R}_{\emptyset\nu_{3}^{t}}(Q_{6,7,8,9,10})}{\mathcal{R}_{\emptyset\mu_{1}}(Q_{7,8,9,10})\mathcal{R}_{\emptyset\mu_{2}}(Q_{7,8,9})\mathcal{R}_{\emptyset\mu_{3}}(Q_{7,8})\mathcal{R}_{\emptyset\nu_{3}}(Q_{5,5,6,7,8,9,10})\mathcal{R}_{\emptyset\nu_{4}^{t}}(Q_{5,6,8,9,10})\mathcal{R}_{\mu_{1}\mu_{2}^{t}}(Q_{10})}$
$\displaystyle~{}~{}\times\frac{\mathcal{R}_{\emptyset\nu_{4}}(Q_{5,6,7,7,8,9,10})\mathcal{R}_{\mu_{1}^{t}\nu_{3}^{t}}(Q_{6})\mathcal{R}_{\mu_{2}^{t}\nu_{3}^{t}}(Q_{6,10})\mathcal{R}_{\mu_{3}^{t}\nu_{3}^{t}}(Q_{6,9,10})\mathcal{R}_{\mu_{1}\nu_{4}^{t}}(Q_{8,9,10})\mathcal{R}_{\mu_{2}\nu_{4}^{t}}(Q_{8,9})}{\mathcal{R}_{\mu_{1}\mu_{3}^{t}}(Q_{9,10})\mathcal{R}_{\mu_{2}\mu_{3}^{t}}(Q_{9})\mathcal{R}_{\mu_{1}^{t}\nu_{3}}(Q_{5,5,6})\mathcal{R}_{\mu_{2}^{t}\nu_{3}}(Q_{5,5,6,10})\mathcal{R}_{\mu_{3}^{t}\nu_{3}}(Q_{5,5,6,9,10})\mathcal{R}_{\mu_{1}\nu_{4}}(Q_{7,7,8,9,10})}$
$\displaystyle~{}~{}\times\frac{\mathcal{R}_{\mu_{3}\nu_{4}^{t}}(Q_{8})\mathcal{R}_{\nu_{3}\nu_{3}}(Q_{5,5})\mathcal{R}_{\nu_{4}\nu_{4}}(Q_{7,7})\mathcal{R}_{\nu_{3}^{t}\nu_{4}}(Q_{6,7,7,8,9,10})\mathcal{R}_{\nu_{3}\nu_{4}^{t}}(Q_{5,5,6,8,9,10})}{\mathcal{R}_{\mu_{2}\nu_{4}}(Q_{7,7,8,9})\mathcal{R}_{\mu_{3}\nu_{4}}(Q_{7,7,8})\mathcal{R}_{\nu_{3}\nu_{4}}(Q_{5,5,6,7,7,8,9,10})\mathcal{R}_{\nu_{3}^{t}\nu_{4}^{t}}(Q_{6,8,9,10})}\
.$ (28)

As we are computing the partition function of a 6d theory on the
$\Omega$-background which is compactified on a circle with a twist, it is a
partition function on $\mathbb{R}^{4}\times T^{2}$. Hence, it can be compared
with the elliptic genus of 6d theories. To this end, we expand the partition
function (2.2) as

$\displaystyle
Z=Z_{\mathrm{pert}}\bigg{(}1+\sum_{n=1}^{\infty}u^{n}Z_{n}\bigg{)}\ ,$ (29)

where $Z_{\mathrm{pert}}$ is the perturbative part and $Z_{n}$ are the
$n$-string elliptic genus, and $u$ is the string fugacity which is given by a
product of Kähler parameters.

We proceed with the computation by expressing the Kähler parameters in terms
of physical parameters. First we eliminate the dependence of $\phi_{0}$ along
the W-bosons. For that, we perform the following shifts

$\displaystyle\phi_{2}\to\phi_{2}+2\phi_{0}+\frac{1}{2R}\
,\quad\phi_{3}\to\phi_{3}+2\phi_{0}+\frac{1}{2R}\
,\quad\phi_{4}\to\phi_{4}+\phi_{0}+\frac{1}{4R}\ ,$ (30)

which yield

$\displaystyle-\log\alpha_{1}=2\phi_{4}-\phi_{3}\
,\quad\qquad-\log\alpha_{2}=2\phi_{3}-2\phi_{4}-\phi_{2}\ ,$
$\displaystyle-\log\alpha_{3}=2\phi_{2}-\phi_{3}\
,\quad\qquad-\log\alpha_{4}=-\phi_{2}+\frac{\tau}{2}\ ,$ $\displaystyle-\log
Q_{B}=\phi_{2}+\phi_{3}+4\phi_{0}+\tau\ ,$ (31)

where $\alpha_{i}$ and $Q_{B}$ are the Kähler parameters assigned in Figure 8.
Here, we have identified that $1/R$ as $\tau$, since it parametrizes the KK-
momentum. One can see the Cartan matrix of $\mathfrak{so}(7)$ from the
W-bosons in (2.2), which is the invariant subalgebra under order two outer
automorphism of $\mathfrak{so}(8)$ algebra. The string fugacity $u$ is
$Q_{B}\alpha_{1}^{-2}\alpha_{2}^{-2}\alpha_{3}^{-1}\alpha_{4}$, and in terms
of $u$,

$\displaystyle-\log Q_{B}=-\log u+\phi_{2}+\phi_{3}-\frac{\tau}{2}\ .$ (32)

As the perturbative part of the partition function $Z_{\mathrm{pert}}$ is the
zeroth order in $u$, we obtain $Z_{\mathrm{pert}}$ by setting
$\mu_{i}=\emptyset$ in (2.2). It is convenient to separate out the spinor
matter parts, as we will decouple them. To this end, we first sum over
$\nu_{i}$ and express the Kähler parameters in terms of the $\alpha_{i}$
associated with the W-bosons, e.g., $Q_{2}=\alpha_{4}Q_{1}^{-1}$, etc. Then we
see that $Z_{\mathrm{pert}}$ is written as a function of $g$, $\alpha_{i}$ and
$Q_{1}$, $Q_{3}$, $Q_{5}$, $Q_{7}$ that are the Kähler parameters associated
with the spinor matter. Next, we take the plethystic logarithm, defined in
Appendix B, and expand $Z_{\mathrm{pert}}$ in terms of $\alpha_{i}$ and
$Q_{i}$. The shift (2.2) then yields

$\displaystyle Z_{\mathrm{pert}}$
$\displaystyle=\operatorname{PE}\quantity[\frac{2g}{(1-g)^{2}}\quantity(\chi_{\Delta_{+}}^{\mathfrak{so}(7)}+(\chi_{\mathbf{7}}^{\mathfrak{so}(7)}-1)q^{1/2}+\cdots)]\
,$ (33)

where $\operatorname{PE}$ stands for the plethystic exponential,

$\displaystyle\operatorname{PE}[f(x)]=\exp(\sum_{n=1}^{\infty}\frac{1}{n}f(x^{n}))\
,$ (34)

and $q=e^{-\tau}$, the terms $\cdots$ are the terms depend on the spinor
matter. Here the characters for the positive roots and the fundamental
representation of $\mathfrak{so}(7)$ algebra take the form

$\displaystyle\chi_{\Delta_{+}}^{\mathfrak{so}(7)}$
$\displaystyle=x_{2}+\frac{x_{2}^{2}}{x_{3}}+x_{3}+\frac{x_{3}}{x_{2}}+\frac{x_{2}x_{3}}{x_{4}^{2}}+\frac{x_{3}^{2}}{x_{2}x_{4}^{2}}+\frac{x_{4}^{2}}{x_{2}}+\frac{x_{4}^{2}}{x_{3}}+\frac{x_{2}x_{4}^{2}}{x_{3}}\
,$ (35) $\displaystyle\chi_{\mathbf{7}}^{\mathfrak{so}(7)}$
$\displaystyle=1+x_{2}+\frac{1}{x_{2}}+\frac{x_{2}}{x_{3}}+\frac{x_{3}}{x_{2}}+\frac{x_{3}}{x_{4}^{2}}+\frac{x_{4}^{2}}{x_{3}}\
,$ (36)

where $x_{i}=e^{-\phi_{i}}$. We note that the $\cdots$ terms in (33) denote
the terms that involve $Q_{1}$, $Q_{3}$, $Q_{5}$ and $Q_{7}$ which we will
decouple. In the 5d limit where the KK-momentum becomes very large, the states
with KK-momenta are truncated. Thus, only $q^{0}$ terms in $Z_{\mathrm{pert}}$
survive. It is then easy to see that that $Z_{\mathrm{pert}}$ involves the
contribution from the positive roots of $\mathfrak{so}(7)$ which is expected
on the Coulomb branch where masses of W-bosons are all positive. This thus
shows that the corresponding 5d theory is an $SO(7)$ gauge theory, as
expected.

The partition function for 6d self-dual strings can be expanded in terms of
the string fugacity

$\displaystyle
Z_{\mathrm{string}}=\frac{Z}{Z_{\mathrm{pert}}}=1+uZ_{1}+u^{2}Z_{2}+\cdots\ ,$
(37)

where $Z_{n}$ is the $n$-string elliptic genus. Again, having in mind the
decoupling of the spinor contributions, when summing over Young diagrams
(2.2), we express it as a function of $\alpha_{i}$ and $Q_{1}$, $Q_{3}$,
$Q_{5}$, $Q_{7}$. To decouple the spinors, one needs to go through a flop
transition, where masses of the spinors are proportional to $-\log$ of
$Q_{1}^{-1}$, $Q_{3}^{-1}$, $Q_{5}^{-1}$, and $Q_{7}^{-1}$. We take each mass
of the spinors to infinite or the corresponding Kähler parameters to zero
while keeping $\alpha_{1},\alpha_{4}$ finite. So, after decoupling the
spinors, we get the desired $n$-string elliptic genus.

To organize $n$-string elliptic genus, we expand $Z_{n}$ in terms of the
Kähler parameters associated with the W-bosons, given in (2.2). In particular,
using $\alpha_{4}=q^{1/2}(\alpha_{1}\alpha_{2}\alpha_{3})^{-1}$, we expand
$Z_{n}$ in terms of $q$ and $\alpha_{1}$. The one-string elliptic genus
$Z_{1}$ is expressed as

$\displaystyle Z_{1}$
$\displaystyle=\\!\Bigg{(}\\!-\frac{g\alpha_{2}^{3}\alpha_{3}^{2}\Big{(}\alpha_{3}(\alpha_{3}+\\!1)\alpha_{2}^{2}+\\!(\alpha_{3}^{2}-6\alpha_{3}+1)\alpha_{2}\\!+\\!\alpha_{3}\\!+\\!1\Big{)}}{(1-g)^{2}(1-\alpha_{2})^{2}(1-\alpha_{3})^{2}(1-\alpha_{2}\alpha_{3})}\alpha_{1}^{3}+O(\alpha_{1}^{4})\Bigg{)}q^{-1/2}$
$\displaystyle~{}~{}+\\!\Bigg{(}-\frac{4g\alpha_{2}^{2}\alpha_{3}^{2}\Big{(}(\alpha_{3}^{2}-\alpha_{3}+1)\alpha_{2}^{2}-(\alpha_{3}+1)\alpha_{2}+1\Big{)}}{(1-g)^{2}(1-\alpha_{2})^{2}(1-\alpha_{3})^{2}(1-\alpha_{2}\alpha_{3})^{2}}\alpha_{1}^{2}$
$\displaystyle\quad~{}~{}-\frac{4g\alpha_{2}^{3}\alpha_{3}^{2}\Big{(}\alpha_{3}(\alpha_{3}\\!+\\!1)\alpha_{2}^{2}+(\alpha_{3}^{2}\\!-\\!6\alpha_{3}\\!+\\!1)\alpha_{2}\\!+\\!\alpha_{3}\\!+\\!1\Big{)}}{(1-g)^{2}(1-\alpha_{2})^{2}(1-\alpha_{3})^{2}(1-\alpha_{2}\alpha_{3})^{2}}\alpha_{1}^{3}+\\!O(\alpha_{1}^{4})\\!\Bigg{)}+\\!O(q^{1/2}).$
(38)

The two-string elliptic genus $Z_{2}$ is rather complicated, so we write only
the leading terms in $\alpha_{1}$:

$\displaystyle Z_{2}$
$\displaystyle=\Bigg{(}\Bigg{[}\frac{g^{5}\alpha_{2}^{6}\alpha_{3}^{4}}{(1\\!-\\!g)^{4}(1\\!+\\!g)^{2}(1\\!-\\!\alpha_{2})^{2}(1\\!-\\!\alpha_{3})^{2}}\quantity(\frac{1}{(1\\!-\\!g\alpha_{2})^{2}(g-\alpha_{3})^{2}}+\frac{1}{(g\\!-\\!\alpha_{2})^{2}(1-g\alpha_{3})^{2}})$
$\displaystyle+\\!\frac{g^{4}\alpha_{2}^{6}\alpha_{3}^{5}}{(1-g)^{4}(1-\alpha_{2})^{2}(1-\alpha_{2}\alpha_{3})^{2}(g-\alpha_{3})^{2}(1-\alpha_{3}g)^{2}}$
$\displaystyle+\\!\frac{g^{5}\alpha_{2}^{6}\alpha_{3}^{6}}{(1\\!-\\!g)^{4}(1\\!+\\!g)^{2}(1\\!-\\!\alpha_{3})^{2}(1\\!-\\!\alpha_{2}\alpha_{3})^{2}}\quantity(\frac{1}{(g\\!-\\!\alpha_{3})^{2}(g\\!-\\!\alpha_{2}\alpha_{3})^{2}}\\!+\\!\frac{1}{(1\\!-\\!\alpha_{3}g)^{2}(1\\!-\\!\alpha_{2}\alpha_{3}g)^{2}}\\!)$
$\displaystyle+\\!\frac{g^{4}\alpha_{2}^{7}\alpha_{3}^{4}}{(1-g)^{4}(g-\alpha_{2})^{2}(1-\alpha_{2}g)^{2}(1-\alpha_{3})^{2}(1-\alpha_{2}\alpha_{3})^{2}}$
$\displaystyle+\\!\frac{g^{4}\alpha_{2}^{7}\alpha_{3}^{5}}{(1-g)^{4}(1-\alpha_{2})^{2}(1-\alpha_{3})^{2}(g-\alpha_{2}\alpha_{3})^{2}(1-\alpha_{2}\alpha_{3}g)^{2}}$
$\displaystyle+\\!\frac{g^{5}\alpha_{2}^{8}\alpha_{3}^{4}}{(1\\!-\\!g)^{4}(1\\!+\\!g)^{2}(1\\!-\\!\alpha_{2})^{2}(1\\!-\\!\alpha_{2}\alpha_{3})^{2}}\\!\quantity(\\!\frac{1}{(g\\!-\\!\alpha_{2})^{2}(g\\!-\\!\alpha_{2}\alpha_{3})^{2}}\\!+\\!\frac{1}{(1\\!-\\!g\alpha_{2})^{2}(1\\!-\\!g\alpha_{2}\alpha_{3})^{2}}\\!)\\!\Bigg{]}\\!\alpha_{1}^{6}$
(39) $\displaystyle+O(\alpha_{1}^{7})\Bigg{)}q^{-1}+O(q^{-1/2})\ .$ (40)

In the following subsections, we will compare this obtained result with the
elliptic genus computations in two different ways: First, as explained through
5-brane webs, we perform the Higgsing $SO(10)+2\mathbf{F}\to
SO(9)+1\mathbf{F}\to SO(8)$ with $\mathbb{Z}_{2}$ twist. Secondly, we apply
the $\mathbb{Z}$ twist directly to the elliptic genus of the 6d $SO(8)$
theory.

###  2.3 Partition function from the Higgsing sequence

We now compute the partition function of the $SO(8)$ theory with
$\mathbb{Z}_{2}$ twist from direct Higgsing of the partition function for the
6d $SO(10)$ gauge theory with two fundamental hypermultiplets. The
perturbative part is given by

$\displaystyle
Z_{\mathrm{pert}}^{\mathrm{gauge}}=\operatorname{PE}\quantity[\frac{2g}{(1-g)^{2}}\quantity(\chi_{\Delta_{+}}^{\mathfrak{so}(10)}+q\chi_{\Delta_{-}}^{\mathfrak{so}(10)})\frac{1}{1-q}]\
,$ (41)

where $\chi_{\Delta_{+}}^{\mathfrak{so}(10)}$ and
$\chi_{\Delta_{-}}^{\mathfrak{so}(10)}$ are the positive and negative root
parts of the character for the adjoint representation of $\mathfrak{so}(10)$,
respectively. More explicitly, for instance,
$\chi_{\Delta_{+}}^{\mathfrak{so}(10)}=\sum_{\alpha}e^{-\alpha}$ for the
positive roots $\alpha$, which can be also expressed in terms of $\phi_{i}$ in
Figure 2(2(a)).

To realize the Higgsing $SO(10)+2F\to SO(9)+1F$, let us consider the
contribution of the first hypermultiplet of mass $m_{1}$ to the perturbative
part, which is given by

$\displaystyle
Z_{\mathrm{pert}}^{\mathrm{hyper1}}=\operatorname{PE}\quantity[-\frac{g}{(1-g)^{2}}\quantity(M_{1}\chi_{\mathbf{10}}^{\mathfrak{so}(10)}+\frac{q}{M_{1}}\chi_{\mathbf{10}}^{\mathfrak{so}(10)})\frac{1}{1-q}]\
,$ (42)

where $\chi_{\mathbf{10}}^{\mathfrak{so}(10)}$ is the character for the
fundamental representation $\mathbf{10}$ of $\mathfrak{so}(10)$, and
$M_{1}=e^{-m_{1}}$ is the fugacity for the mass of the fundamental
hypermultiplet $m_{1}$. To Higgs the first hypermultiplet, we recover the
affine node $\phi_{0}$ using Figure 2(2(a)) and impose (3). If one changes the
variables as in Figure 3(3(a)) and eliminate the affine node $\phi_{0}$, one
can find that the perturbative part

$\displaystyle
Z_{\mathrm{pert}}^{\mathrm{gauge}}Z_{\mathrm{pert}}^{\mathrm{hyper1}}\to\operatorname{PE}\quantity[\frac{2g}{(1-g)^{2}}\quantity(\chi_{\Delta_{+}}^{\mathfrak{so}(9)}+q\chi_{\Delta_{-}}^{\mathfrak{so}(9)})\frac{1}{1-q}]\
,$ (43)

up to the Cartan part, where $\chi_{\Delta_{\pm}}^{\mathfrak{so}(9)}$ are the
positive and negative parts of the characters for the adjoint representation
of $\mathfrak{so}(9)$, respectively. This reflects that the theory becomes the
$SO(9)$ gauge theory as in Figure 3(a).

Higgsing the second hypermultiplet of mass $m_{2}$ needs more caution because
of $1/R$ factor in Figure 3(b). From string tensions associated with the
hypermultiplet in Figure 3(b), one can read that the contribution of the
second hypermultiplet after Higgsing the first hypermultiplet becomes

$\displaystyle Z_{\mathrm{pert}}^{\mathrm{hyper2}}\to$
$\displaystyle\operatorname{PE}\bigg{[}-\frac{g}{(1-g)^{2}}\bigg{(}M_{2}\quantity(\frac{x_{1}}{q}+\frac{x_{1}}{x_{2}}+\frac{x_{2}}{x_{3}}+\frac{x_{3}}{x_{4}^{2}}+1+\frac{x_{4}^{2}}{x_{3}}+\frac{x_{3}}{x_{2}}+\frac{x_{2}}{x_{1}}+\frac{q}{x_{1}})$
$\displaystyle+\frac{q}{M_{2}}\quantity(\frac{x_{1}}{q}+\frac{x_{1}}{x_{2}}+\frac{x_{2}}{x_{3}}+\frac{x_{3}}{x_{4}^{2}}+1+\frac{x_{4}^{2}}{x_{3}}+\frac{x_{3}}{x_{2}}+\frac{x_{2}}{x_{1}}+\frac{q}{x_{1}})\bigg{)}\frac{1}{1-q}\bigg{]}\
,$ (44)

where $x_{i}=e^{-\phi_{i}}$ for $\phi_{i}$ in Figure 3(b). To Higgs the second
hypermultiplet, we reintroduce the affine node $\phi_{0}$ again and impose
(9). Lastly, shifting variables (30) gives the perturbative part

$\displaystyle
Z_{\mathrm{pert}}^{\mathrm{gauge}}Z_{\mathrm{pert}}^{\mathrm{hyper1}}Z_{\mathrm{pert}}^{\mathrm{hyper2}}\to\operatorname{PE}\quantity[\frac{2g}{(1-g)^{2}}\quantity((\chi_{\Delta_{+}}^{\mathfrak{so}(7)}-2)+(\chi_{\mathbf{7}}^{\mathfrak{so}(7)}+1)q^{1/2}+\cdots)\\!],$
(45)

which is the same result as (33) up to the Cartan part.

Next, consider the instanton part. The 6d $SO(8+2p)$ gauge theories on $-4$
curve have $Sp(p)\times Sp(p)$ flavor symmetry. In the 2d worldsheet theory on
a self-dual string introduced in Haghighat:2014vxa , both $SO(8+2p)$ gauge
symmetry and $Sp(p)$ flavor symmetries become flavor symmetry. The gauge
symmetry in 2d theory is $Sp(n)=USp(2n)$ where $n$ is string number. The field
contents in this theory can be written as $\mathcal{N}=(0,2)$ multiplets. The
charged fields are vector multiplet $V$, three Fermi multiplets
$\Lambda^{\Phi},\Lambda^{Q},\Lambda^{\tilde{Q}}$, and four chiral multiplets
$B,\tilde{B},Q,\tilde{Q}$. Their charges under the symmetries on a worldsheet
are summarized in Table 1.

| $Sp(n)$ | $SO(8+2p)$ | $Sp(p)_{1}$ | $Sp(p)_{2}$ | $U(1)_{\epsilon_{1}}$ | $U(1)_{\epsilon_{2}}$
---|---|---|---|---|---|---
$V$ | $\mathbf{adj}$ | $\mathbf{1}$ | $\mathbf{1}$ | $\mathbf{1}$ | $0$ | $0$
$\Lambda^{\Phi}$ | $\mathbf{adj}$ | $\mathbf{1}$ | $\mathbf{1}$ | $\mathbf{1}$ | $-1$ | $-1$
$B$ | $\mathbf{antisymm}$ | $\mathbf{1}$ | $\mathbf{1}$ | $\mathbf{1}$ | $1$ | $0$
$\tilde{B}$ | $\mathbf{antisymm}$ | $\mathbf{1}$ | $\mathbf{1}$ | $\mathbf{1}$ | $0$ | $1$
$Q$ | $\mathbf{n}$ | $\mathbf{8+2p}$ | $\mathbf{1}$ | $\mathbf{1}$ | $1/2$ | $1/2$
$\tilde{Q}$ | $\bar{\mathbf{n}}$ | $\overline{\mathbf{8+2p}}$ | $\mathbf{1}$ | $\mathbf{1}$ | $1/2$ | $1/2$
$\Lambda^{Q}$ | $\mathbf{n}$ | $\mathbf{1}$ | $\mathbf{p}$ | $\mathbf{1}$ | $0$ | $0$
$\Lambda^{\tilde{Q}}$ | $\bar{\mathbf{n}}$ | $\mathbf{1}$ | $\mathbf{1}$ | $\bar{\mathbf{p}}$ | $0$ | $0$
Table 1: The $\mathcal{N}=(0,2)$ multiplets and symmetries on worldsheet in 6d
$SO(8+2p)$ gauge theory.

The $U(1)_{\epsilon_{1}}$ and $U(1)_{\epsilon_{2}}$ are Cartans of $SO(4)$
rotating transverse $\mathbb{R}^{4}$ to worldsheet.

Using the field contents in Table 1, one can calculate the elliptic genus of
strings in the 6d $SO(10)$ theory using localization technique Benini:2013xpa
. The $n$-string elliptic genus is expressed as

$\displaystyle Z_{n}=\frac{1}{(2\pi
i)^{n}}\frac{1}{\absolutevalue{\mathrm{Weyl}\quantity[G]}}\oint
Z_{\mathrm{1-loop}}\ ,$ (46)

where $G$ is the gauge group on the 2d worldsheet. The contour integral is
performed using the Jeffrey-Kirwan (JK) residue prescription in
1993alg.geom..7001J .

The one-loop determinant $Z_{\mathrm{1-loop}}$ is the product of the
contributions from each multiplets. The $\mathcal{N}=(0,2)$ vector multiplet
contribution in $Z_{\mathrm{1-loop}}$ is given as a product of all the roots
$\alpha$ of $G$ of rank $r$,

$\displaystyle
Z_{\mathrm{vec}}=\quantity(\frac{2\pi\eta^{2}}{i})^{r}\prod_{\alpha}\frac{i\theta_{1}(\alpha)}{\eta}\prod_{j=1}^{r}du_{j}\
,$ (47)

where $\eta$ is the Dedekind eta function, $\theta_{1}(x)$ is the Jacobi theta
function. See Appendix B for the definition and some useful properties of the
theta function. The $\mathcal{N}=(0,2)$ chiral and Fermi multiplet
contributions are given as

$\displaystyle
Z_{\mathrm{chiral}}=\prod_{\rho}\frac{i\eta}{\theta_{1}(\rho)},\qquad
Z_{\mathrm{Fermi}}=\prod_{\rho}\frac{i\theta_{1}(\rho)}{\eta}\ ,$ (48)

where $\rho$ is the weight vector of the representation of each multiplet in
gauge and flavor symmetries.

In the case of the $SO(10)$ theory, the 2d gauge group is $Sp(n)$. Note that
in orthogonal basis $\\{e_{i}\\}_{i=1}^{n}$ of $\mathfrak{sp}(n)$ algebra, the
root vectors are given by $\pm e_{i}\pm e_{j}$, the weight vectors of the
fundamental and the antisymmetric representations are $\pm e_{i}$ and $\pm
e_{i}\pm e_{j}(i\neq j$), respectively. Denote chemical potential parameters
of $\mathfrak{sp}(n)$ gauge algebra be $u_{i}$ and $\mathfrak{so}(10)$,
$\mathfrak{sp}(1)_{1}$, $\mathfrak{sp}(1)_{2}$ flavor symmetry algebras be
$v_{i}$, $m_{1}$ and $m_{2}$, respectively. Then the $n$-string elliptic genus
from (46) is

$\displaystyle Z_{n}^{SO(10)}$ $\displaystyle=\frac{1}{(2\pi
i)^{n}}\frac{1}{\absolutevalue{\mathrm{Weyl}[Sp(n)]}}\oint\prod_{I=1}^{n}du_{I}\cdot\quantity(\prod_{I=1}^{n}\frac{i^{2}\theta_{1}(\pm
2u_{I})}{\eta^{2}})\quantity(\prod_{I<J}^{n}\frac{i^{4}\theta_{1}(\pm u_{I}\pm
u_{J})}{\eta^{4}})$
$\displaystyle\quad\times\quantity(\frac{i\theta_{1}(-2\epsilon_{+})}{\eta})^{n}\quantity(\prod_{I=1}^{n}\frac{i^{2}\theta_{1}(-2\epsilon_{+}\pm
2u_{I})}{\eta^{2}})\quantity(\prod_{I<J}^{n}\frac{i^{4}\theta_{1}(-2\epsilon_{+}\pm
u_{I}\pm u_{J})}{\eta^{4}})$
$\displaystyle\quad\times\quantity(\frac{i^{2}\eta^{2}}{\theta_{1}(\epsilon_{1,2})})^{n}\quantity(\prod_{I<J}^{n}\frac{i^{8}\eta^{8}}{\theta_{1}(\epsilon_{1,2}\pm
u_{I}\pm
u_{J})})\quantity(\prod_{I=1}^{n}\prod_{J=1}^{5}\frac{i^{4}\eta^{4}}{\theta_{1}(\epsilon_{+}\pm
u_{I}\pm v_{J})})$
$\displaystyle\quad\times\quantity(\prod_{I=1}^{n}\frac{i^{4}\theta_{1}(\pm
u_{I}+m_{1})\theta_{1}(\pm u_{I}+m_{2})}{\eta^{4}})\ ,$ (49)

where $\epsilon_{\pm}=\frac{\epsilon_{1}\pm\epsilon_{2}}{2}$ and we use
shorthand notation $\theta_{1}(\pm
2u_{i})=\theta_{1}(2u_{i})\theta_{1}(-2u_{i})$,
$\theta_{1}(\epsilon_{1,2})=\theta_{1}(\epsilon_{1})\theta_{1}(\epsilon_{2})$,
etc.222For example, $\theta_{1}(\epsilon_{1,2}\pm u_{I}\pm
u_{J})=\\!\displaystyle\prod_{n=1}^{2}\\!\theta_{1}(\epsilon_{n}\\!+u_{I}+u_{J})\theta_{1}(\epsilon_{n}\\!+u_{I}-u_{J})\theta_{1}(\epsilon_{n}\\!-u_{I}+u_{J})\theta_{1}(\epsilon_{n}\\!-u_{I}-u_{J})$.

To evaluate the integral (2.3), one needs to find residues of the integrand.
However, not all poles contributes to elliptic genus. The contributing
residues are called Jeffrey-Kirwan (JK) residue, determined as follows.
Suppose the singular hyperplanes $H_{n}$ are
$\sum_{I=1}^{r}a_{I}^{(n)}u_{I}+\cdots=0$. Fix an auxiliary vector $\zeta$ in
the space which contains vectors $a^{(n)}$. If $\zeta$ can be written as
linear combination of $a^{(n_{1})},\cdots,a^{(n_{l})}$ with positive
coefficients, then the singularities $H_{1},\cdots,H_{l}$ contribute to the
integral.

We explicitly calculate the one string and two string elliptic genus to
compare with topological vertex calculation. For $n=1$, the singular
hyperplanes are

$\displaystyle\pm u_{1}\pm v_{J}+\epsilon_{+}=0\ .$ (50)

If one takes an auxiliary vector $\zeta=(1)$, only singular hyperplanes
$+u_{1}\pm v_{J}+\epsilon_{+}=0$ contribute to the elliptic genus. Using

$\displaystyle\frac{1}{2\pi
i}\oint_{u_{1}=0}\frac{du}{\theta_{1}(u_{1})}=\frac{1}{2\pi\eta^{3}}\
,\qquad\theta_{1}(-x)=-\theta_{1}(x)\ ,$ (51)

the one-string elliptic genus is

$\displaystyle Z_{1}^{SO(10)}$
$\displaystyle=-\frac{\eta^{12}}{2}\sum_{I=1}^{5}\Bigg{(}\frac{\theta_{1}(2\epsilon_{+}+2v_{I})\theta_{1}(4\epsilon_{+}+2v_{I})\theta_{1}(\epsilon_{+}+v_{I}\pm
m_{1})\theta_{1}(\epsilon_{+}+v_{I}\pm
m_{2})}{\theta_{1}(\epsilon_{1,2})\prod_{J\neq I}^{5}\theta_{1}(v_{I}\pm
v_{J})\theta_{1}(2\epsilon_{+}+v_{I}\pm v_{J})}$
$\displaystyle\qquad\qquad\qquad\qquad+(v_{I}\to-v_{I})\Bigg{)}.$ (52)

For $n=2$, the singular hyperplanes are

$\displaystyle\pm u_{I}\pm u_{2}+\epsilon_{1,2}=0,\qquad\pm u_{I}\pm
v_{J}+\epsilon_{+}=0\ ,$ (53)

for $I=1,2$ and $J=1,\cdots,5$. We choose an auxiliary vector $\zeta$ as in
Figure 10, so that the contributing poles are

$\displaystyle(i)\left\\{\begin{array}[]{l}-u_{1}\pm v_{I}+\epsilon_{+}=0\\\
-u_{1}-u_{2}+\epsilon_{1,2}=0\end{array}\right.$
$\displaystyle(ii)\left\\{\begin{array}[]{l}-u_{1}\pm v_{I}+\epsilon_{+}=0\\\
u_{1}-u_{2}+\epsilon_{1,2}=0\end{array}\right.$ (58)
$\displaystyle(iii)\left\\{\begin{array}[]{l}-u_{1}\pm m_{I}+\epsilon_{+}=0\\\
-u_{2}\pm m_{J}+\epsilon_{+}=0\\\ \end{array}\right.$
$\displaystyle(iv)\left\\{\begin{array}[]{l}-u_{1}+u_{2}+\epsilon_{1,2}=0\\\
-u_{1}-u_{2}+\epsilon_{1,2}=0\end{array}\right.$ (63)
$\displaystyle(v)\left\\{\begin{array}[]{l}-u_{1}+u_{2}+\epsilon_{1,2}=0\\\
-u_{2}\pm v_{I}+\epsilon_{+}=0\\\ \end{array}\right.$
$\displaystyle(vi)\left\\{\begin{array}[]{l}u_{2}\pm v_{I}+\epsilon_{+}=0\\\
-u_{1}-u_{2}+\epsilon_{1,2}=0\end{array}\right.\ .$ (68)

Figure 10: A choice of an auxiliary vector $\zeta$ for JK-residue calculation
of two-string elliptic genus.

The residue around poles (58)(i) is

$\displaystyle\mathrm{Res}_{1}$
$\displaystyle=\frac{\eta^{24}}{8}\sum_{I=1}^{5}\bigg{[}\frac{\theta_{1}(3\epsilon_{+}-\epsilon_{-})\theta_{1}(2\epsilon_{-}-2v_{I})\theta_{1}(2\epsilon_{1}-2v_{I})\theta_{1}(4\epsilon_{+}+2v_{I})\theta_{1}(\epsilon_{2}+2v_{I})}{\theta_{1}(\epsilon_{1})\theta_{1}(\epsilon_{2})^{2}\theta_{1}(2\epsilon_{1})\theta_{1}(2\epsilon_{-})\theta_{1}(2v_{I})}$
$\displaystyle\qquad\quad\times\\!\\!\frac{\theta_{1}(3\epsilon_{+}-\epsilon_{-}+2v_{I})\theta_{1}(\epsilon_{+}+v_{I}\pm
m_{1,2})\theta_{1}(\epsilon_{-}-v_{I}\pm m_{1,2})}{\prod_{J\neq
I}^{5}\theta_{1}(v_{I}\pm v_{J})\theta_{1}(\epsilon_{1}-v_{I}\pm
v_{J})\theta_{1}(\epsilon_{2}+v_{I}\pm v_{J})\theta_{1}(2\epsilon_{+}+v_{I}\pm
v_{J})}$ $\displaystyle\qquad\qquad+(v_{I}\to-
v_{I})\bigg{]}+(\epsilon_{1}\leftrightarrow\epsilon_{2},\>\epsilon_{-}\to-\epsilon_{-})\
.$ (69)

The poles (58)(ii) give

$\displaystyle\mathrm{Res}_{2}$
$\displaystyle=-\frac{\eta^{24}}{8}\sum_{I=1}^{5}\bigg{[}\frac{\theta_{1}(3\epsilon_{+}+\epsilon_{-}+2v_{I})\theta_{1}(4\epsilon_{+}+2\epsilon_{-}+2v_{I})\theta_{1}(5\epsilon_{+}+\epsilon_{-}+2v_{I})}{\theta_{1}(\epsilon_{1,2})\theta_{1}(2\epsilon_{1})\theta_{1}(2\epsilon_{-})}$
$\displaystyle\qquad\quad\times\\!\\!\frac{\theta_{1}(6\epsilon_{+}\\!+2\epsilon_{-}+2v_{I})\theta_{1}(\epsilon_{+}\\!+v_{I}\pm
m_{1,2})\theta_{1}(2\epsilon_{+}\epsilon_{-}\\!+v_{I}\pm
m_{1,2})}{\prod_{J\neq I}^{5}\theta_{1}(v_{I}\pm
v_{J})\theta_{1}(\epsilon_{1}\\!+v_{I}\pm
v_{J})\theta_{1}(2\epsilon_{+}\\!+v_{I}\pm
v_{J})\theta_{1}(3\epsilon_{+}\\!+\epsilon_{-}\\!+v_{I}\pm v_{J})}$
$\displaystyle\qquad\qquad+(v_{I}\to-
v_{I})\bigg{]}+(\epsilon_{1}\leftrightarrow\epsilon_{2},\>\epsilon_{-}\to-\epsilon_{-})\
,$ (70)

and the poles (58)(iii) contribute

$\displaystyle\mathrm{Res}_{3}$ $\displaystyle=\frac{\eta^{24}}{8}\sum_{I\neq
J}^{5}\bigg{[}\frac{\theta_{1}(2\epsilon_{+}+2v_{I,J})\theta_{1}(4\epsilon_{+}+2v_{I,J})\theta_{1}(4\epsilon_{+}+v_{I}+v_{J})}{\theta_{1}(\epsilon_{1,2})^{2}\theta_{1}(v_{I}+v_{J})\theta_{1}(\epsilon_{1,2}+v_{I}+v_{J})\theta_{1}(\epsilon_{1,2}\pm(v_{I}-v_{J}))}$
$\displaystyle\qquad\quad\times\\!\\!\frac{\theta_{1}(\epsilon_{+}+v_{I,J}\pm
m_{1,2})}{\theta_{1}(3\epsilon_{+}\pm\epsilon_{-}+v_{I}+v_{J})\prod_{K\neq
I,J}^{5}\theta_{1}(v_{I,J}\pm v_{K})\theta_{1}(2\epsilon_{+}+v_{I,J}\pm
v_{K})}$ $\displaystyle\quad\qquad+(v_{I}\to-v_{I})+(v_{J}\to-
v_{J})+(v_{I}\to-v_{I},\>v_{J}\to-v_{J})\bigg{]},$ (71)

where
$\theta_{1}(2\epsilon_{+}+2v_{I,J})=\theta_{1}(2\epsilon_{+}+2v_{I})\theta_{1}(2\epsilon_{+}+2v_{J})$.
The residue of poles (58)(iv) is zero. The cases (58)(v) and (vi) are
$u_{1}\leftrightarrow u_{2}$ and $u_{1}\to u_{2},\>u_{2}\to-u_{1}$ of the case
(ii), respectively. These transformations do not change the residues. As a
result, two string elliptic genus reads

$\displaystyle
Z_{2}^{SO(10)}=\mathrm{Res}_{1}+3\times\mathrm{Res}_{2}+\mathrm{Res}_{3}\ .$
(72)

We now Higgs this $SO(10)$ elliptic genera. We only consider the unrefined
case, $\epsilon_{1}=-\epsilon_{2}$. The brane web construction gives hints for
Higgsing procedure. In the elliptic genus calculated from the field theory
construction, there are $\mathfrak{so}(10)$ parameters $v_{i}$. They are
written in the orthogonal basis, so first convert it to fundamental weight
basis (Dynkin basis) and recover an affine node:

$\displaystyle v_{1}=\phi_{1}-\phi_{0},\qquad
v_{2}=-\phi_{0}-\phi_{1}+\phi_{2},$ $\displaystyle
v_{3}=\phi_{3}-\phi_{2},\qquad v_{4}=-\phi_{3}+\phi_{4}+\phi_{5},\qquad
v_{5}=\phi_{5}-\phi_{4}$ (73)

Then we can apply the Higgsing conditions (3) and (9). Lastly, we shift
variables (30) to compare with topological vertex calculation. The result is

$\displaystyle v_{1}=0,\quad v_{2}=-2\phi_{4}+\phi_{3},\quad
v_{3}=\phi_{2}-\phi_{3},\quad v_{4}=-\phi_{2},\quad v_{5}=\frac{\tau}{2}$ (74)

Note that $v_{2}$, $v_{3}$, $v_{4}$ form the orthogonal basis of
$\mathfrak{so}(7)$ algebra. Thus, by redefinition of the variables, the
Higgsing sets the parameters as

$\displaystyle m_{1}=0\ ,\qquad m_{2}=\frac{\tau}{2}\ ,\qquad
v_{4}=\frac{\tau}{2}\ ,\qquad v_{5}=0\ .$ (75)

Substituting (75) into (2.3) and (72) and taking the unrefined limit
$\epsilon_{1}=-\epsilon_{2}$, we find that the one-string elliptic genus is
given by

$\displaystyle
Z_{1}=\\!\frac{\eta^{12}}{2\theta_{1}(\epsilon_{-})^{2}}\sum_{I=1}^{3}\\!\quantity(\frac{\theta_{1}(2v_{I})^{2}}{\theta_{1}(v_{I})^{2}\theta_{1}(v_{I}\\!-\\!\frac{\tau}{2})\theta_{1}(v_{I}\\!+\\!\frac{\tau}{2})\prod_{J\neq
I}^{3}\theta_{1}(v_{I}\pm v_{J})^{2}}+(v_{I}\\!\to\\!-v_{I})),$ (76)

and the two-string elliptic genus is given by

$\displaystyle
Z_{2}=\mathrm{Res}_{1}+3\times\mathrm{Res}_{2}+\mathrm{Res}_{3}\ ,$ (77)

where

$\displaystyle\mathrm{Res}_{1}$
$\displaystyle=\frac{\eta^{24}}{8}\sum_{I=1}^{3}\bigg{[}\frac{\theta_{1}(\epsilon_{-}-2v_{I})^{2}\theta_{1}(2\epsilon_{-}-2v_{I})^{2}}{\theta_{1}(\epsilon_{-})^{2}\theta_{1}(2\epsilon_{-})^{2}\theta_{1}(v_{I})^{2}\theta_{1}(\epsilon_{-}-v_{I})^{2}\theta_{1}(\pm
v_{I}+\frac{\tau}{2})\theta_{1}(\pm(\epsilon_{-}-v_{I})+\frac{\tau}{2})}$
$\displaystyle\qquad\qquad\times\prod_{J\neq
I}^{3}\frac{1}{\theta_{1}(\epsilon_{-}-v_{I}-v_{J})^{2}\theta_{1}(v_{I}\pm
v_{J})^{2}\theta_{1}(\epsilon_{-}-v_{I}+v_{J})^{2}}+(v_{I}\to-v_{I})\bigg{]}$
$\displaystyle\qquad+(\epsilon_{-}\to-\epsilon_{-})\ ,$ (78)
$\displaystyle\mathrm{Res}_{2}$
$\displaystyle=\frac{\eta^{24}}{8}\sum_{I=1}^{3}\bigg{[}\frac{\theta_{1}(\epsilon_{-}+2v_{I})^{2}\theta_{1}(2\epsilon_{-}+2v_{I})^{2}}{\theta_{1}(\epsilon_{-})^{2}\theta_{1}(2\epsilon_{-})^{2}\theta_{1}(v_{I})^{2}\theta_{1}(\epsilon_{-}+v_{I})^{2}\theta_{1}(\pm
v_{I}+\frac{\tau}{2})\theta_{1}(\pm(\epsilon_{-}+v_{I})+\frac{\tau}{2})}$
$\displaystyle\qquad\qquad\times\prod_{J\neq
I}^{3}\frac{1}{\theta_{1}(v_{I}\pm v_{J})^{2}\theta_{1}(\epsilon_{-}+v_{I}\pm
v_{J})^{2}}+(v_{I}\to-v_{I})\bigg{]}$
$\displaystyle\quad+(\epsilon_{-}\to-\epsilon_{-})\ ,$ (79)
$\displaystyle\mathrm{Res}_{3}$ $\displaystyle=\frac{\eta^{24}}{8}\sum_{I\neq
J}^{3}\bigg{[}\frac{\theta_{1}(2v_{I,J})^{2}}{\theta_{1}(\epsilon_{-})^{4}\theta_{1}(v_{I,J})^{2}\theta_{1}(\epsilon_{-}\pm
v_{I}\pm v_{J})^{2}\theta_{1}(\pm v_{I,J}+\frac{\tau}{2})\prod_{K\neq
I,J}^{3}\theta_{1}(v_{I,J}\pm v_{K})^{2}}$
$\displaystyle\qquad\qquad+(v_{I}\to-v_{I})+(v_{J}\to-v_{J})+(v_{I}\to-
v_{I},\>v_{J}\to-v_{J})\bigg{]}.$ (80)

Since $v_{i}$ are the $\mathfrak{so}(7)$ parameters, we change the variables

$\displaystyle v_{1}$ $\displaystyle=-\log\alpha_{1}\ ,\qquad
v_{2}=-\quantity(\log\alpha_{1}+\log\alpha_{2})\ ,$ (81) $\displaystyle v_{3}$
$\displaystyle=-\quantity(\log\alpha_{1}+\log\alpha_{2}+\log\alpha_{3})\ .$
(82)

and expand $Z_{1}$ and $Z_{2}$ in terms of $q$ and $\alpha_{1}$ to compare
with topological vertex calculation in the previous subsection. By an explicit
calculation, we see that two results are in complete agreement.

###  2.4 Partition function from a direct twisting of the $SO(8)$ theory

There is yet another way to obtaining the partition function of the $SO(8)$
theory with $\mathbb{Z}_{2}$ twist from the field theory construction. The
perturbative part of the partition function of the 6d $SO(8)$ theory is given
by

$\displaystyle
Z_{\mathrm{pert}}^{SO(8)}=\operatorname{PE}\quantity[\frac{2g}{(1-g)^{2}}\quantity(\chi_{\Delta_{+}}^{\mathfrak{so}(8)}+q\chi_{\Delta_{-}}^{\mathfrak{so}(8)})\frac{1}{1-q}]\
,$ (83)

where $\chi_{\Delta_{\pm}}^{\mathfrak{so}(8)}$ are the positive and negative
parts of the character for the adjoint representation of $\mathfrak{so}(8)$,

$\displaystyle\chi_{\Delta_{+}}^{\mathfrak{so}(8)}=\sum_{i<j}^{4}\quantity(x_{i}x_{j}+\frac{x_{i}}{x_{j}}),\qquad\chi_{\Delta_{-}}^{\mathfrak{so}(8)}=\sum_{i<j}^{4}\quantity(\frac{1}{x_{i}x_{j}}+\frac{x_{j}}{x_{i}})\
,$ (84)

for the fugacities $x_{i}$ of the orthonormal basis of $\mathfrak{so}(8)$.

The elliptic genus of the 6d pure $SO(8)$ theory can be obtained from the 2d
worldsheet theory with matters in Table 1, with $p=0$. As there is no Fermi
multiplet $\Lambda^{Q}$ and $\Lambda^{\tilde{Q}}$ for $p=0$, the $n$-string
elliptic genus is given by

$\displaystyle Z_{n}^{SO(8)}$ $\displaystyle=\frac{1}{(2\pi
i)^{n}}\frac{1}{\absolutevalue{\mathrm{Weyl}[Sp(n)]}}\oint\prod_{I=1}^{n}du_{I}\cdot\quantity(\prod_{I=1}^{n}\frac{i^{2}\theta_{1}(\pm
2u_{I})}{\eta^{2}})\quantity(\prod_{I<J}^{n}\frac{i^{4}\theta_{1}(\pm u_{I}\pm
u_{J})}{\eta^{4}})$
$\displaystyle\quad\times\quantity(\frac{i\theta_{1}(-2\epsilon_{+})}{\eta})^{n}\quantity(\prod_{I=1}^{n}\frac{i^{2}\theta_{1}(-2\epsilon_{+}\pm
2u_{I})}{\eta^{2}})\quantity(\prod_{I<J}^{n}\frac{i^{4}\theta_{1}(-2\epsilon_{+}\pm
u_{I}\pm u_{J})}{\eta^{4}})$
$\displaystyle\quad\times\quantity(\frac{i^{2}\eta^{2}}{\theta_{1}(\epsilon_{1,2})})^{n}\quantity(\prod_{I<J}^{n}\frac{i^{8}\eta^{8}}{\theta_{1}(\epsilon_{1,2}\pm
u_{I}\pm
u_{J})})\quantity(\prod_{I=1}^{n}\prod_{j=1}^{4}\frac{i^{4}\eta^{4}}{\theta_{1}(\epsilon_{+}\pm
u_{I}\pm v_{J})}).$ (85)

$v_{J}$ are the chemical potentials for the $SO(8)$ symmetry and they can be
regarded as being in the orthogonal basis of $SO(8)$.

Let the eight states in the fundamental representation be $\ket{\pm e_{i}}$
where $i=1,2,3,4$. They satisfy $H^{j}\ket{\pm e_{j}}=\pm\delta_{ij}\ket{\pm
e_{j}}$ for the orthonormal basis $H^{j}$ of the Cartan subalgebra of
$\mathfrak{so}(8)$. The order two outer automorphism $\sigma$ of
$\mathfrak{so}(8)$ algebra exchanges the simple root $\alpha_{3}$ and
$\alpha_{4}$. In terms of the orthonormal basis of the Cartan subalgebra,
$\sigma(H^{4})=-H^{4}$. Thus, the invariant elements are $\pm H^{1}$, $\pm
H^{2}$, $\pm H^{3}$ and $H^{4}-H^{4}$, corresponding to the states $\ket{\pm
e_{1}}$, $\ket{\pm e_{2}}$, $\ket{\pm e_{3}}$ and $\ket{e_{4}}+\ket{-e_{4}}$.
On the other hand, $H^{4}-(-H^{4})$ has the eigenvalue $-1$, which corresponds
to the state $\ket{e_{4}}-\ket{-e_{4}}$. Upon compactification, $\ket{e_{4}}$
and $\ket{-e_{4}}$ are identified so that the invariant states $\ket{\pm
e_{1}}$, $\ket{\pm e_{2}}$, $\ket{\pm e_{3}}$ and $\ket{0}$ form the
fundamental representation of the invariant algebra $\mathfrak{so}(7)$ under
the order two outer automorphism. The state of eigenvalue $-1$ becomes a
singlet and gets additional half KK-momentum shift Tachikawa:2011ch .

Consequently, we change $(x_{4}+\frac{1}{x_{4}})\sum_{i=1}^{3}x_{i}$ to
$(1+q^{1/2})\sum_{i=1}^{3}x_{i}$ in $\chi_{\Delta_{+}}^{\mathfrak{so}(8)}$ and
$(v_{4}+\frac{1}{v_{4}})\sum_{i=1}^{3}\frac{1}{v_{i}}$ to
$(1+q^{-1/2})\sum_{i=1}^{3}\frac{1}{v_{i}}$ in
$\chi_{\Delta_{-}}^{\mathfrak{so}(8)}$. After this replacement, the
perturbative part becomes

$\displaystyle
Z_{\mathrm{pert}}=\operatorname{PE}\quantity[\frac{2g}{(1-g)^{2}}\quantity(\chi_{\Delta_{+}}^{\mathfrak{so}(7)}+\quantity(\chi_{\mathbf{7}}^{\mathfrak{so}(7)}-1)q^{1/2}+\cdots)]\
,$ (86)

which agrees with the perturbative part obtained from 5-brane webs in (33).
For the instanton partition function, we replace $\pm v_{4}$ in (2.4) to
$\frac{\tau}{2}$ and $0$. The elliptic genus is then expressed as

$\displaystyle Z_{n}$ $\displaystyle=\frac{1}{(2\pi
i)^{n}}\frac{1}{\absolutevalue{\mathrm{Weyl}[Sp(n)]}}\oint\prod_{I=1}^{n}du_{I}\cdot\quantity(\prod_{I=1}^{n}\frac{i^{2}\theta_{1}(\pm
2u_{I})}{\eta^{2}})\quantity(\prod_{I<J}^{n}\frac{i^{4}\theta_{1}(\pm u_{I}\pm
u_{J})}{\eta^{4}})$
$\displaystyle\quad\times\quantity(\frac{i\theta_{1}(-2\epsilon_{+})}{\eta})^{n}\quantity(\prod_{I=1}^{n}\frac{i^{2}\theta_{1}(-2\epsilon_{+}\pm
2u_{I})}{\eta^{2}})\quantity(\prod_{I<J}^{n}\frac{i^{4}\theta_{1}(-2\epsilon_{+}\pm
u_{I}\pm u_{J})}{\eta^{4}})$
$\displaystyle\quad\times\quantity(\frac{i^{2}\eta^{2}}{\theta_{1}(\epsilon_{1,2})})^{n}\quantity(\prod_{I<J}^{n}\frac{i^{8}\eta^{8}}{\theta_{1}(\epsilon_{1,2}\pm
u_{I}\pm
u_{J})})\quantity(\prod_{I=1}^{n}\prod_{J=1}^{3}\frac{i^{4}\eta^{3}}{\theta_{1}(\epsilon_{+}\pm
u_{I}\pm v_{J})})$
$\displaystyle\quad\times\prod_{I=1}^{n}\frac{i^{4}\eta^{3}}{\theta_{1}(\epsilon_{+}\pm
u_{I})\theta_{1}(\epsilon_{+}\pm u_{I}+\frac{\tau}{2})}\ .$ (87)

For $n=1$, the singular hyperplanes are

$\displaystyle\pm u_{1}+\epsilon_{+}\pm v_{J}=0\ ,\qquad\pm
u_{1}+\epsilon_{+}=0\ ,\qquad\pm u_{1}+\epsilon_{+}+\frac{\tau}{2}=0\ .$ (88)

Take an auxiliary vector $\zeta=(1)$. The residues of the last two singular
hyperplanes are zero since $\theta_{1}(k\tau)=0$ for $k\in\mathbb{Z}$. Thus,
only poles $+u_{1}+\epsilon_{+}\pm v_{j}=0$ contribute to the one string
elliptic genus. One can find that the JK-residue sum for $Z_{1}$ with the
unrefined limit $\epsilon_{1}=-\epsilon_{2}$ gives the same expression as
(76). A similar calculation for the $n=2$ case leads to (77) as well.

##  3 $SU(3)$ theory with $\mathbb{Z}_{2}$ twist

Figure 11: A 5-brane web for 6d $G_{2}$ gauge theory with one fundamental.

The 6d $SU(3)$ gauge theory with $\mathbb{Z}_{2}$ twist can be obtained by
Higgsing from the 6d $G_{2}$ gauge theory with a hypermultiplet in the
fundamental representation. The Higgsing procedure can be realized through
5-brane webs as given in Kim:2019dqn . It is also worthy of noting that from
the perspective of 5d SCFTs, it is 5d pure $SU(3)$ gauge theory at the Chern-
Simons level $\kappa=9$ Jefferson:2018irk . The corresponding 5-brane web is
given in Hayashi:2018lyv . In order not to be confused with this 5d $SU(3)$
theory, we denote it by $SU(3)_{9}$. In this section, we compute the partition
function for the 6d $SU(3)$ gauge theory with $\mathbb{Z}_{2}$ twist from
5-brane webs using topological vertex and also from performing a Higgsing on
the elliptic genera of self-dual strings in the 6d $G_{2}$ theory.

###  3.1 5-brane web for 6d $SU(3)$ gauge theory with $\mathbb{Z}_{2}$ twist

We first review Type IIB 5-brane construction of the 6d pure $SU(3)$ gauge
theory with $\mathbb{Z}_{2}$ twist. We start from the 6d $G_{2}$ gauge theory
with one fundamental hypermultiplet, whose 5-brane configuration is given in
Figure 11333The 5-brane web given in Figure 11 is obtained by taking monodromy
cut of D7-branes on O5-planes differently compared with the 5-brane web given
in Kim:2019dqn . One can think of it as an $SL(2,\mathbb{Z})$ transformed web
diagram., where the mass of the fundamental hyper is denoted by $m$ and the
masses of W-bosons $W_{i}$ are expressed in terms of the scalar fields as

$\displaystyle m_{W_{1}}=2\phi_{0}-\phi_{2}\ ,\qquad
m_{W_{2}}=2\phi_{1}-\phi_{2}\ ,\qquad m_{W_{3}}=-\phi_{0}-3\phi_{1}+2\phi_{2}\
,$ (89)

which form the affine Cartan matrix of untwisted affine Lie algebra
$G_{2}^{(1)}$.

To obtain the $SU(3)$ gauge theory with $\mathbb{Z}_{2}$ twist from the
$G_{2}$ gauge theory with a fundamental, we Higgs the hypermultiplet of the
$G_{2}$ gauge theory by giving vevs to the scalar fields which carry KK-
momentum. To this end, we redefine the scalars as

$\displaystyle 2\phi_{0}-\phi_{2}\to 2\phi_{0}-\phi_{2}+\frac{1}{R}\
,\qquad\phi_{1}-\phi_{0}\to\phi_{1}-\phi_{0}-\frac{1}{2R}\ ,$ (90)

while other parameters in Figure 11 remain unaltered. This recovers the 6d
circle radius $R$ in the brane configuration. The Higgsing procedure to the 6d
$SU(3)$ gauge theory with $\mathbb{Z}_{2}$ twist is then similar to the
Higgsing explained through Figure 2(b)-(d) in the previous section. We first
bring the flavor D7-brane down to the lower O5-plane as depicted in Figure 12.
The Higgsing condition is then given by

$\displaystyle m=\frac{1}{2R},\qquad\phi_{1}=\phi_{0}+\frac{1}{2R}\ .$ (91)

Figure 12: Lowering the flavor brane in Figure 11 through a series of flop
transitions, so as to apply the Higgsing procedure discussed in Figure
2(b)-(d).

The first condition in (91) comes from the flavor D7-brane put on the lower
$\mathrm{O}5$-plane. As discussed in the previous section, this D7-brane is
brought to the lower chamber of the Coulomb branch and then is put to the
$\mathrm{O}5^{-}$-plane on the bottom. It follows that this full D7-brane is
split into two half D7-branes generating an
$\widetilde{\mathrm{O}5}^{-}$-plane in between these two half D7-branes. The
second condition in (91) comes from putting the lower color D5-brane on the
$\mathrm{O}5^{-}$-plane in the middle, which also becomes two half D5-branes.
Through Higgsing, one of half D5-branes remains intact, while the other half
D5-brane is recombined onto the two half D7-branes on the
$\mathrm{O}5^{-}$-plane. So there are effectively three half D5-branes in
between two half D7-branes so that the two half D5-branes can be Higgsed away.

Now, by taking each half D7-brane in the opposite directions, leaving the half
D7 monodromy cut being presented from the right to the left, one obtains a
5-brane configuration given in Figure 13. The masses of W-bosons after the
Higgsing are then given by

$\displaystyle m_{W_{1}}=2\phi_{2}-4\phi_{0}-\frac{3}{2R}\ ,\qquad
m_{W_{2}}=\frac{1}{R}+2\phi_{0}-\phi_{2}\ ,$ (92)

which form the affine Cartan matrix of twisted affine Lie algebra
$A_{2}^{(2)}$.

Figure 13: A 5-brane web for 6d $SU(3)$ theory with $\mathbb{Z}_{2}$ twist.
The half monodromy cut extends from the right to the left.

###  3.2 Partition function from 5-brane webs

With this 5-brane web, we are ready to compute the topological string
partition function for the 6d $SU(3)$ gauge theory with $\mathbb{Z}_{2}$
twist. Like 5-brane configurations for the $SO(8)$ theory with
$\mathbb{Z}_{2}$ twist in the previous section, this 5-brane configuration is
not smoothly connected to its reflected image over the orientifold plane. In
order to make a desirable 5-brane configuration where the original 5-brane and
its reflected mirror image are smoothly connected, we introduce an auxiliary
spinor hypermultiplet on the 5-brane configuration, as done in the previous
section. It allows us then to apply topological vertex formulation with an
$O5$-plane to compute the partition function for the 6d $SU(3)$ gauge theory
with $\mathbb{Z}_{2}$ twist, followed by taking the decoupling limit of the
newly introduced hypermultiplet. For instance, see Figure 14. The lower part
of the 5-brane configuration for 6d $SU(3)$ gauge theory with $\mathbb{Z}_{2}$
twist is given in Figure 14(a) where we introduced a hypermultiplet444From
$G_{2}$ gauge theory point of view, this hypermultiplet is a fundamental hyper
which originates from a hypermultiplet in the spinor representation of an
$SO(7)$ gauge theory. on the left hand side of the web configuration. Here two
half D7-branes are still in the middle of the brane web which are denoted by
blue dots and their monodromy cuts are the wiggly lines pointing to the left.
For computational ease, we take these half D7-branes to the far left instead
of taking them in the opposite directions. This makes a 5-brane configuration
with only O5-plane, as depicted in Figure 14(b). When taking all the half
D7-branes to the left, half D5-branes are created as a result of the Hanany-
Witten transitions, which is depicted in Figure 14(b), where half D5-branes
are solid lines in violet. In actual computation, they can be viewed as
follows: one full external D5-brane (or two half D5-branes) is connected to
the right $(0,1)$ and $(1,1)$ 5-branes, and an additional full D5-brane is
connected to the left $(0,1)$ and $(1,1)$ 5-branes.

(a)

(b)

Figure 14: (a) Coupling a spinor $(-1,1)$ 5-brane to $SU(3)$ theory with
$\mathbb{Z}_{2}$ twist. (b) Applying the generalized flop transition and
moving the half D7-branes and monodromy cuts to the right. Several half
D5-branes are created due to the Hanany-Witten transition.

For the upper part of the 5-brane web in Figure 13, the 5-brane configuration
is not smoothly connected to its mirror image as well. One may again couple
another hypermultiplet as in Figure 15. It is also possible to use the
symmetry of the 5-brane web. Instead of computing the partition function based
on Figure 15, we use the following trick. We recycle the lower part of 5-brane
configuration with a hyper for the upper part computation. In other words,
when implementing the topological vertex for 5-brane configuration on the
upper orientifold plane, we re-use the lower part configuration with a hyper
as if it is a suitable 5-brane configuration for the upper part and perform
the partition function computation by properly gluing 5-branes. For instance,
as in Figure 16, we first consider strip diagrams. For the upper part, in
particular, we use the strip diagram of the lower part and glue the upper and
lower part together. In the figure, the two edges marked with
${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}||}$
symbols are identified. This trick is not harmful, as the point of introducing
extra hypermultiplets is to arrange the 5-brane configuration in a way that we
can apply the topological vertex. Moreover, we decouple the contribution of
these newly introduced hypermultiplets.

Figure 15: A brane web of $SU(3)$ theory with $\mathbb{Z}_{2}$ twist coupled
with two spinors.  Figure 16: Strip diagrams for 5-brane webs for $SU(3)$
theory with $\mathbb{Z}_{2}$ twist given in Figure 15. The left part (a) and
the right part (b) of the web diagram in Figure 15. $Z_{\mathrm{left}}$ is
based on (a) and $Z_{\mathrm{right}}$ is based on (b). The edges denoted with
${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}||}$ are
identified.

It follows then that the the strip diagram on Figure 16(a) yields

$\displaystyle Z_{\mathrm{left}}(\mu_{1},\mu_{2},\nu_{1},\nu_{2})$ (93)
$\displaystyle=g^{\frac{1}{2}(\norm*{\mu_{1}}^{2}+\norm*{\mu_{2}}^{2}+2\norm*{\nu_{1}}^{2}+2\norm*{\nu_{2}^{t}}^{2})}\tilde{Z}_{\mu_{1}}(g)\tilde{Z}_{\mu_{2}}(g)\tilde{Z}_{\nu_{1}}(g)^{2}\tilde{Z}_{\nu_{2}}(g)^{2}\frac{\mathcal{R}_{\emptyset\mu_{1}}(Q_{3,4,5})\mathcal{R}_{\emptyset\mu_{1}^{t}}(Q_{1,2})}{\mathcal{R}_{\emptyset\emptyset}(Q_{1,2,3,4,5})}$
$\displaystyle\quad\times\frac{\mathcal{R}_{\emptyset\mu_{2}}(Q_{4,5})\mathcal{R}_{\emptyset\mu_{2}^{t}}(Q_{1,2,3})\mathcal{R}_{\nu_{1}\emptyset}(Q_{1,1,2,3,4,5})\mathcal{R}_{\emptyset\nu_{2}}(Q_{1,2,3,4,5,5})\mathcal{R}_{\mu_{1}^{t}\nu_{1}^{t}}(Q_{2})\mathcal{R}_{\mu_{2}^{t}\nu_{1}^{t}}(Q_{2,3})}{\mathcal{R}_{\emptyset\nu_{1}^{t}}(Q_{2,3,4,5})\mathcal{R}_{\emptyset\nu_{2}^{t}}(Q_{1,2,3,4})\mathcal{R}_{\mu_{1}\mu_{2}^{t}}(Q_{3})\mathcal{R}_{\mu_{1}^{t}\nu_{1}}(Q_{1,1,2})}$
$\displaystyle\quad\times\frac{\mathcal{R}_{\mu_{1}\nu_{2}^{t}}(Q_{3,4})\mathcal{R}_{\mu_{2}\nu_{2}^{t}}(Q_{4})\mathcal{R}_{\nu_{1}\nu_{1}}(Q_{1,1})\mathcal{R}_{\nu_{1}^{t}\nu_{2}}(Q_{2,3,4,5,5})\mathcal{R}_{\nu_{2}\nu_{2}}(Q_{5,5})\mathcal{R}_{\nu_{1}\nu_{2}^{t}}(Q_{1,1,2,3,4})}{\mathcal{R}_{\mu_{2}^{t}\nu_{1}}(Q_{1,1,2,3})\mathcal{R}_{\mu_{1}\nu_{2}}(Q_{3,4,5,5})\mathcal{R}_{\mu_{2}\nu_{2}}(Q_{4,5,5})\mathcal{R}_{\nu_{1}\nu_{2}}(Q_{1,1,2,3,4,5,5})\mathcal{R}_{\nu_{1}^{t}\nu_{2}^{t}}(Q_{2,3,4})},$

where we used the Kähler parameters $Q_{i}$ assigned in Figure 15 with the
convention that $Q_{i,j,\cdots}=Q_{i}Q_{j}\cdots$.

The strip on Figure 16(b) can be treated in a similar way except that the
upper and lower edges marked with
${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}||}$ symbol
are identified.

The topological string partition function for the strip in Figure 16(b) is a
bit involving but can be obtained in a straightforward manner following
Haghighat:2013gba ; macdonald_symmetric_1995 . One needs to repeatedly use
various Cauchy identities in Appendix B. The contribution of the strip diagram
in Figure 16(b) then reads

$\displaystyle
Z_{\mathrm{right}}(\mu_{1},\mu_{2})=g^{\frac{1}{2}(\norm*{\mu_{1}^{t}}^{2}+\norm*{\mu_{2}^{t}}^{2})}\tilde{Z}_{\mu_{1}}(g)\tilde{Z}_{\mu_{2}}(g)\times$
$\displaystyle\quad\times\prod_{m=1}^{\infty}\frac{1}{1-q^{m}}\frac{1}{\mathcal{R}_{\emptyset\emptyset}(q^{2m})^{2}\mathcal{R}_{\emptyset\emptyset}(q^{2m-1/2})^{2}\mathcal{R}_{\emptyset\emptyset}(q^{2m-1})^{2}\mathcal{R}_{\emptyset\emptyset}(q^{2m-3/2})^{2}}$
$\displaystyle\quad\times\frac{1}{\mathcal{R}_{\emptyset\mu_{1}}(q^{2m-2}Q_{3,4,5})\mathcal{R}_{\emptyset\mu_{1}}(q^{2m-1}Q_{3,4,5})\mathcal{R}_{\emptyset\mu_{1}}(q^{2m}Q_{1,2}^{-1})\mathcal{R}_{\emptyset\mu_{1}}(q^{2m-1}Q_{1,2}^{-1})}$
$\displaystyle\quad\times\frac{1}{\mathcal{R}_{\emptyset\mu_{1}^{t}}(q^{2m-2}Q_{1,2})\mathcal{R}_{\emptyset\mu_{1}^{t}}(q^{2m-1}Q_{1,2})\mathcal{R}_{\emptyset\mu_{1}^{t}}(q^{2m-2}Q_{1,1,2,2,3,4,5})\mathcal{R}_{\emptyset\mu_{1}^{t}}(q^{2m-1}Q_{1,1,2,2,3,4,5})}$
$\displaystyle\quad\times\frac{1}{\mathcal{R}_{\emptyset\mu_{2}}(q^{2m-2}Q_{4,5})\mathcal{R}_{\emptyset\mu_{2}}(q^{2m-1}Q_{4,5})\mathcal{R}_{\emptyset\mu_{2}}(q^{2m}Q_{1,2,3}^{-1})\mathcal{R}_{\emptyset\mu_{2}}(q^{2m-1}Q_{1,2,3}^{-1})}$
$\displaystyle\quad\times\frac{1}{\mathcal{R}_{\emptyset\mu_{2}^{t}}(q^{2m}Q_{4,5}^{-1})\mathcal{R}_{\emptyset\mu_{2}^{t}}(q^{2m-1}Q_{1,2,3})\mathcal{R}_{\emptyset\mu_{2}^{t}}(q^{2m-1}Q_{4,5}^{-1})\mathcal{R}_{\emptyset\mu_{2}^{t}}(q^{2m-2}Q_{1,2,3})}$
$\displaystyle\quad\times\frac{1}{\mathcal{R}_{\mu_{1}\mu_{1}^{t}}(q^{2m})\mathcal{R}_{\mu_{1}\mu_{1}^{t}}(q^{2m-1})\mathcal{R}_{\mu_{2}\mu_{2}^{t}}(q^{2m})\mathcal{R}_{\mu_{2}\mu_{2}^{t}}(q^{2m-1})}$
$\displaystyle\quad\times\frac{1}{\mathcal{R}_{\mu_{1}^{t}\mu_{2}}(q^{2m}Q_{3}^{-1})\mathcal{R}_{\mu_{1}^{t}\mu_{2}}(q^{2m-1}Q_{3}^{-1})\mathcal{R}_{\mu_{1}\mu_{2}^{t}}(q^{2m-1}Q_{3})\mathcal{R}_{\mu_{1}\mu_{2}^{t}}(q^{2m-2}Q_{3})}\
,$ (94)

where $q=(Q_{1}Q_{2}Q_{3}Q_{4}Q_{5})^{2}$.

The full topological string partition function $Z$ is obtained by combining
$Z_{\mathrm{left}}$ and $Z_{\mathrm{right}}$,

$\displaystyle
Z=\sum_{\mu_{i},\nu_{i}}(-Q_{B})^{\absolutevalue{\mu_{1}}+\absolutevalue{\mu_{2}}}Q_{1}^{\absolutevalue{\nu_{1}}}Q_{5}^{\absolutevalue{\nu_{2}}}f_{\mu_{1}}f_{\mu_{2}}^{-1}f_{\nu_{1}}^{3}f_{\nu_{2}}Z_{\mathrm{left}}(\mu_{1},\mu_{2},\nu_{1},\nu_{2})Z_{\mathrm{right}}(\mu_{1},\mu_{2}).$
(95)

It is convenient to further rescale the scalars in Figure 13 as

$\displaystyle\phi_{2}\to\phi_{2}+2\phi_{0}+\frac{3}{4R}\ ,$ (96)

which gives that the Kähler parameters are expressed in terms of physical
parameters

$\displaystyle-\log Q_{1}Q_{2}$ $\displaystyle=-\log
Q_{4}Q_{5}=-\phi_{2}+\frac{\tau}{4}\ ,$ $\displaystyle-\log Q_{3}$
$\displaystyle=2\phi_{2}\ ,$ (97) $\displaystyle-\log Q_{B}$
$\displaystyle=\log u+2\phi_{2}\ ,$ (98)

where $u$ is the string fugacity. We note that the relation
$q=(Q_{1}Q_{2}Q_{3}Q_{4}Q_{5})^{2}=e^{-\tau}$ is not changed under the
rescaling. With $A=e^{-\phi_{2}}$, we can write the Kähler parameters $Q_{2}$
and $Q_{4}$ as $Q_{2}=q^{1/4}A^{-1}Q_{1}^{-1}$ and
$Q_{4}=q^{1/4}A^{-1}Q_{5}^{-1}$ where $Q_{1},Q_{5}$ are the Kähler parameters
associated with the spinors which we will decouple.

The perturbative partition function $Z_{\mathrm{pert}}$ is obtained by setting
$\mu_{1}=\mu_{2}=\emptyset$ in (95) and by summing up over the Young diagrams
$\nu_{i}$, as a function of $g$, $A$, $q$, $Q_{1}$ and $Q_{5}$. As an
expansion of $q$, we find the perturbative part is given by

$\displaystyle
Z_{\mathrm{pert}}\\!=\operatorname{PE}\\!\bigg{[}\frac{2gA^{2}}{(1\\!-g)^{2}}\\!+\frac{g(2A^{-1}\\!+2A)q^{1/4}}{(1-g)^{2}}\\!+\frac{4gq^{1/2}}{(1\\!-g)^{2}}\\!+\frac{2g(A\\!+A^{-1})q^{3/4}}{(1-g)^{2}}\\!+\cdots\\!\bigg{]},$
(99)

where the $\cdots$ denotes the terms involving $Q_{1}$ and $Q_{5}$ which we
will decouple later. This is expected result as in Appendix A. If we take the
large radius limit corresponding to $q\to 0$, the states depending on KK-
momentum will be truncated, and as a result, only $q^{0}$ term will remain.

The partition function for 6d self-dual strings $Z_{\mathrm{string}}$ is
obtained by summing up over all Young diagrams in (95), expand about $q$ and
$A$, and also by decoupling the auxiliary spinor hypermultiplets that we
introduced for computational ease, $Q_{1},Q_{5}\to\infty$ as discussed in the
previous section,

$\displaystyle
Z_{\mathrm{string}}=\frac{Z}{Z_{\mathrm{pert}}}=1+uZ_{1}+u^{2}Z_{2}+\cdots,$
(100)

where $Z_{n}$ correspond to the $n$-string elliptic genus,

$\displaystyle Z_{1}$
$\displaystyle=\frac{2gA^{2}}{(1-g)^{2}(1-A^{2})^{2}}+\frac{2gA(1+A^{2})}{(1-g)^{2}(1-A^{2})^{2}}q^{1/4}+\frac{6gA^{2}}{(1-g)^{2}(1-A^{2})^{2}}q^{1/2}$
$\displaystyle\quad+\frac{6gA(1+A^{2})}{(1-g)^{2}(1-A^{2})^{2}}q^{3/4}+\frac{6A^{4}g+2A^{2}(2g^{2}+7g+2)+6g}{(1-g)^{2}(1-A^{2})^{2}}q+O(q^{5/4})\
,$ $\displaystyle Z_{2}$
$\displaystyle=\frac{g^{4}A^{4}\Big{(}A^{4}(3g^{2}+2g+3)-2A^{2}(g^{2}+6g+1)+3g^{2}+2g+3\Big{)}}{(1-g)^{4}(1+g)^{2}(1-A^{2})^{2}(A^{2}-g)^{2}(1-A^{2}g)^{2}}$
$\displaystyle\quad+\frac{2g^{3}A^{3}(A^{2}+1)\big{(}2A^{4}g+A^{2}(g^{2}-6g+1)+2g\big{)}}{(1-g)^{4}(1-A^{2})^{2}(A^{2}-g)^{2}(1-A^{2}g)^{2}}q^{1/4}$
$\displaystyle\quad+\frac{A^{2}g^{3}\Big{(}A^{4}(3g^{2}+2g+3)-2A^{2}(g^{2}+6g+1)+3g^{2}+2g+3\Big{)}}{(1-g)^{4}(1+g)^{2}(1-A^{2})^{2}(A^{2}-g)^{2}(1-A^{2}g)^{2}}\times$
$\displaystyle\qquad\times\big{(}A^{4}g+2A^{2}(g+1)^{2}+g\big{)}q^{1/2}+O(q^{3/4})\
.$ (101)

Though we presented the result of one- and two-string elliptic genus only,
higher $n$-string elliptic genus can be computed in a straightforward manner.

###  3.3 Elliptic genus by Higgsing 6d $G_{2}+1\mathbf{F}$

In this subsection, we compute the elliptic genus to cross-check the partition
for the $SU(3)$ theory with $\mathbb{Z}_{2}$ twist obtained from 5-brane webs
in the previous subsection. As discussed, the $SU(3)$ theory with
$\mathbb{Z}_{2}$ twist can be obtained by Higgsing the 6d $G_{2}$ gauge theory
with one fundamental hypermultiplet ($G_{2}+1\mathbf{F}$). We hence start by
computing the elliptic genus for the 6d $G_{2}$ gauge theory with one
fundamental hypermultiplet and apply the Higgsing that leads to the $SU(3)$
theory with $\mathbb{Z}_{2}$ twist.

The perturbative part of the partition function comes from the contributions
from the vector multiplet and the hypermultiplet,
$Z_{\mathrm{pert}}^{G_{2}+1\mathbf{F}}=Z_{\mathrm{pert}}^{\mathrm{gauge}}Z_{\mathrm{pert}}^{\mathrm{hyper}}$.
As we have the correspond 5-brane web in Figure 12, we can readily obtain the
perturbative part. The vector multiplet contribution to the perturbative part
for $G_{2}+1\mathbf{F}$ takes the following form

$\displaystyle
Z_{\mathrm{pert}}^{\mathrm{gauge}}=\operatorname{PE}\quantity[\frac{2g}{(1-g)^{2}}\quantity(\chi_{\Delta_{+}}^{G_{2}}+q\chi_{\Delta_{-}}^{G_{2}})\frac{1}{1-q}]\
,$ (102)

where $\chi_{\Delta_{\pm}}^{G_{2}}$ are the positive and negative parts from
the characters for the adjoint representation of $G_{2}$. The hypermultiplet
contribution to the perturbative part is given by

$\displaystyle
Z_{\mathrm{pert}}^{\mathrm{hyper}}\\!=\\!\operatorname{PE}\\!\quantity[-\frac{g}{(1-g)^{2}}\quantity(M\\!+\\!\frac{q}{M})\quantity(\frac{x_{1}}{q}\\!+\frac{x_{1}}{x_{2}}\\!+\frac{x_{2}}{x_{1}^{2}}\\!+1\\!+\frac{x_{1}^{2}}{x_{2}}\\!+\frac{x_{2}}{x_{1}}\\!+\frac{q}{x_{1}})\frac{1}{1-q}],$
(103)

where $M=e^{-m}$ and $x_{i}=e^{-\phi_{i}}$. These perturbative part can be
easily computed from the web diagram in Figure 12.

To perform the Higgsing leading to the $SU(3)$ gauge theory with
$\mathbb{Z}_{2}$ twist, we give a vev to the hypermultiplet carrying KK
momentum. For that, we recover the affine node $\phi_{0}$ as in Figure 12, and
impose the Higgsing conditions (91). By rescaling the scalars (96), we obtain
the perturbative part of the partition function for the $SU(3)$ gauge theory
with $\mathbb{Z}_{2}$ twist:

$\displaystyle
Z_{\mathrm{pert}}\\!=\\!\operatorname{PE}\\!\bigg{[}\frac{2g}{(1-g)^{2}}$
$\displaystyle\Big{(}\\!(A^{2}\\!-\\!2)+(A\\!+\\!A^{-1})q^{1/4}+q^{1/2}+(A\\!+\\!A^{-1})q^{3/4}+\\!\cdots\\!\Big{)}\bigg{]},$
(104)

where $A=x_{2}=e^{-\phi_{2}}$. This result is the same as that obtained from
the topological vertex calculation (99) up to the Cartan parts.

We now compute the elliptic genus of the 6d $G_{2}$ theory with one
fundamental hypermultiplet, following Kim:2018gjo . The symmetries of the 2d
worldsheet theory on self-dual strings are $U(n)$ gauge symmetry for string
number $n$, $SU(3)$ global symmetry which is enhanced to $G_{2}$ in IR,
$U(1)_{\sf J}$ and $SU(2)_{l}$ global symmetries. Here, for $SU(2)_{R}$
R-symmetry and $SO(4)=SU(2)_{l}\times SU(2)_{r}$ rotation symmetry of the
transverse $\mathbb{R}^{4}$, the charge ${\sf J}$ is identified as the sum of
the $SU(2)_{r}$ charge and the $SU(2)_{R}$ charge, ${\sf J}=J_{r}+J_{R}$. The
worldsheet fields can be written in $\mathcal{N}=(0,2)$ multiplets. There are
the vector multiplet $V$, the Fermi multiplets $\lambda$,
$(\hat{\lambda},\check{\lambda})$, $(\Psi,\bar{\Psi})$, and the chiral
multiplets $(q,\tilde{q})$, $(a,\tilde{a})$, $(\phi_{i},\phi_{4})$,
$(b,\tilde{b})$. Their charges under the symmetries in the 2d worldsheet
theory are summarized in Table 2.

| $U(n)$ | $SU(3)$ | $SU(2)_{l}$ | $U(1)_{\sf J}$
---|---|---|---|---
$V$ | $\mathbf{adj}$ | $\mathbf{1}$ | $\mathbf{1}$ | $0$
$\lambda$ | $\mathbf{adj}$ | $\mathbf{1}$ | $\mathbf{1}$ | $-1$
$(q,\tilde{q})$ | $(\mathbf{n},\bar{\mathbf{n}})$ | $(\bar{\mathbf{3}},\mathbf{3})$ | $\mathbf{1}$ | $1/2$
$(a,\tilde{a})$ | $\mathbf{adj}$ | $\mathbf{1}$ | $\mathbf{2}$ | $1/2$
$(\phi_{i},\phi_{4})$ | $\bar{\mathbf{n}}$ | $(\bar{\mathbf{3}},\mathbf{1})$ | $\mathbf{1}$ | 1/2
$(b,\tilde{b})$ | $\overline{\mathbf{anti}}$ | $\mathbf{1}$ | $\mathbf{2}$ | $1/2$
$(\hat{\lambda},\check{\lambda})$ | $\overline{\mathbf{sym}}$ | $\mathbf{1}$ | $\mathbf{1}$ | $(0,-1)$
$(\Psi,\tilde{\Psi})$ | $(\mathbf{n},\bar{\mathbf{n}})$ | $\mathbf{1}$ | $\mathbf{1}$ | $0$
Table 2: The $\mathcal{N}=(0,2)$ multiplets and symmetries in ADHM formalism
of 6d $G_{2}$ gauge theory.

Using the matter content in Table 2, one can write down the $n$-string
elliptic genus:

$\displaystyle Z_{n}$ $\displaystyle=\frac{1}{n!}\frac{1}{(2\pi
i)^{n}}\oint\prod_{I=1}^{n}du_{I}\cdot\Big{(}\frac{2\pi\eta^{2}}{i}\Big{)}^{n}\quantity(\prod_{I\neq
J}^{n}\frac{i\theta_{1}(u_{I}-u_{J})}{\eta})\quantity(\prod_{I,J=1}^{n}\frac{i\theta_{1}(-2\epsilon_{+}+u_{I}-u_{J})}{\eta})$
$\displaystyle\quad\times\quantity(\prod_{I=1}^{n}\prod_{J=1}^{3}\frac{i^{2}\eta^{2}}{\theta_{1}(\epsilon_{+}\pm(u_{I}-v_{J}))})\quantity(\prod_{I,J=1}^{n}\frac{i^{2}\eta^{2}}{\theta_{1}(\epsilon_{1,2}+u_{I}-u_{J})})$
$\displaystyle\quad\times\quantity(\prod_{I=1}^{n}\prod_{J=1}^{3}\frac{i\eta}{\theta_{1}(\epsilon_{+}-u_{I}-v_{J})}\frac{i\eta}{\theta_{1}(\epsilon_{+}-u_{I})})\quantity(\prod_{I<J}^{n}\frac{i\eta}{\theta_{1}(\epsilon_{1,2}-u_{I}-u_{J})})$
$\displaystyle\quad\times\quantity(\prod_{I\leq
J}^{n}\frac{i\theta_{1}(u_{I}+u_{J})}{\eta}\frac{i\theta_{1}(-2\epsilon_{+}+u_{I}+u_{J})}{\eta})\quantity(\prod_{I=1}^{n}\frac{i^{2}\theta_{1}(\pm
u_{I}+m)}{\eta^{2}}).$ (105)

Here, $u_{I}$ are the $U(n)$ parameters, $\eta$ is the Dedekind eta function,
$v_{I}$ are the $SU(3)$ parameters subject to $v_{1}+v_{2}+v_{3}=0$. First
consider the $n=1$ case. We choose an auxiliary vector $\zeta=(1)$. Then
contributing poles are from $\epsilon_{+}+u-v_{J}=0$. The contour integral
converts to the JK-residue sum. We thus obtain the one-string elliptic genus
for the $G_{2}$ gauge theory with one hypermultiplet

$\displaystyle
Z_{1}^{G_{2}+1\mathbf{F}}\\!=\\!\sum_{I=1}^{3}\frac{\eta^{6}\,\theta_{1}(2v_{I}-4\epsilon_{+})\,\theta_{1}\\!(m\pm(\epsilon_{+}\\!-\\!v_{I}))}{\theta_{1}(\epsilon_{1,2})\,\theta_{1}(2\epsilon_{+}\\!-\\!v_{I})\displaystyle{\prod_{J\neq
I}}\theta_{1}(v_{I}\\!-\\!v_{J})\,\theta_{1}(2\epsilon_{+}\\!-\\!v_{I}\\!+v_{J})\,\theta_{1}(2\epsilon_{+}\\!-\\!v_{J})},$
(106)

where the $SU(3)$ condition $v_{1}+v_{2}+v_{3}=0$ is used. For the $n=2$ case,
we choose an auxiliary vector $\zeta$ as in Figure 17.

Figure 17: An auxiliary vector $\zeta$ for the JK-residue calculation of two-
string elliptic genus of the $G_{2}$ gauge theory with one fundamental
hypermultiplet.

The poles which survive in the JK-residue sum are

$\displaystyle(i)\left\\{\begin{array}[]{l}\epsilon_{+}+u_{2}-v_{I}=0\\\
\epsilon_{+}+u_{1}-v_{J}=0\end{array}\right.$
$\displaystyle(ii)\left\\{\begin{array}[]{l}\epsilon_{+}+u_{1}-v_{J}=0\\\
\epsilon_{1,2}+u_{1}-u_{2}=0\end{array}\right.$
$\displaystyle(iii)\left\\{\begin{array}[]{l}\epsilon_{1,2}+u_{2}-u_{1}=0\\\
\epsilon_{+}+u_{1}-v_{J}=0\ .\end{array}\right.$ (113)

The first poles $(i)$ in (113) give

$\displaystyle\mathrm{Res}_{1}$ $\displaystyle=\sum_{I\neq
J}^{3}\frac{\eta^{12}\theta_{1}(2v_{I,J}-4\epsilon_{+})\theta_{1}(\epsilon_{+}-v_{I,J}\pm
m)}{2\,\theta_{1}\\!(\epsilon_{1,2})^{2}\,\theta_{1}\\!(2\epsilon_{+}\pm
v_{I,J})\,\theta_{1}\\!(\epsilon_{1,2}\pm v_{I,J})}$ (114)
$\displaystyle\quad\\!\times\\!\\!\prod_{K\neq
I,J}\\!\frac{\theta_{1}(4\epsilon_{+}+v_{K})}{\theta_{1}\\!(2\epsilon_{+}+v_{K})\,\theta_{1}(3\epsilon_{+}\pm\epsilon_{-}\\!+\\!v_{K})\,\theta_{1}\\!(v_{I,J}\\!-v_{K})\,\theta_{1}\\!(2\epsilon_{+}\\!-v_{I,J}\\!+v_{K})}.$

The second poles $(ii)$ give

$\displaystyle\mathrm{Res}_{2}$
$\displaystyle=\\!\frac{\eta^{12}}{2}\sum_{I=1}^{3}\\!\Bigg{(}\\!\frac{\theta_{1}(5\epsilon_{+}\\!+\\!\epsilon_{-}\\!-2v_{I})\theta_{1}(6\epsilon_{+}\\!+\\!2\epsilon_{-}\\!-\\!2v_{I})\theta_{1}(\epsilon_{+}\\!-\\!v_{I}\pm\\!m)\theta_{1}(2\epsilon_{+}\\!+\\!\epsilon_{-}\\!-\\!v_{I}\\!\pm
m)}{\theta_{1}(\epsilon_{1,2})\theta_{1}(2\epsilon_{1})\theta_{1}(-2\epsilon_{-})\theta_{1}(2\epsilon_{+}-v_{I})\theta_{1}(3\epsilon_{+}+\epsilon_{-}-v_{I})}$
$\displaystyle\quad\qquad\times\prod_{J\neq
I}\frac{1}{\theta_{1}(v_{I}-v_{J})\theta_{1}(\epsilon_{1}-v_{I}+v_{J})\theta_{1}(2\epsilon_{+}-v_{I}+v_{J})\theta_{1}(3\epsilon_{+}+\epsilon_{-}-v_{I}+v_{J})}$
$\displaystyle\quad\qquad\quad\times\frac{1}{\theta_{1}(2\epsilon_{+}+v_{J})\theta_{1}(3\epsilon_{+}+\epsilon_{-}+v_{J})}+(\epsilon_{1}\leftrightarrow\epsilon_{2},\>\epsilon_{-}\to-\epsilon_{-})\bigg{)}\
.$ (115)

Notice that the third poles (iii) can be obtained by $u_{1}\leftrightarrow
u_{2}$ in $(ii)$. It follows that the residue for (iii) is the same as that
for $(ii)$, $\mathrm{Res}_{3}=\mathrm{Res}_{2}$. The two-string elliptic genus
of the $G_{2}$ theory with one hypermultiplet is then given by

$\displaystyle
Z_{2}^{G_{2}+1\mathbf{F}}=\mathrm{Res}_{1}+2\times\mathrm{Res}_{2}\ .$ (116)

We now Higgs the 6d $G_{2}$ theory. The procedure is similar to that for the
$SO(10)$ case in the previous section. As the elliptic genus results are
written in terms of the $SU(3)$ parameters, rather than the $G_{2}$
parameters, some additional implementation is required for the proper Higgs
from the 6d $G_{2}$ theory to the $SU(3)$ theory with $\mathbb{Z}_{2}$ twist.

We begin by identifying the relation between the $SU(3)$ parameters $v_{i}$
and the $G_{2}$ fundamental weights. It follows from the embedding

$\displaystyle G_{2}$ $\displaystyle\supset SU(3)$ (117) $\displaystyle{\bf
7}$ $\displaystyle={\bf 1+3+\bar{3}}\,$ (118)

that the character for the fundamental representation of $G_{2}$ is expressed
in terms of the $SU(3)$ characters, $1+\sum_{I}(e^{v_{i}}+e^{-v_{I}})$, and
the parameter map between $SU(3)$ parameters $v_{i}$ and the $G_{2}$
fundamental weights is $v_{1}\to\phi_{1}$, $v_{2}\to\phi_{1}-\phi_{2}$ and
$v_{3}\to\phi_{2}-2\phi_{1}$. By applying the Higgsing conditions from the
5-brane web diagram (91) as well as the reparameterization (96), one finds
that the proper Higgsing from the $G_{2}$ theory to the $SU(3)$ gauge theory
with $\mathbb{Z}_{2}$ twist is given by

$\displaystyle v_{1}\to\frac{\tau}{2},\qquad
v_{2}\to-\phi_{2}-\frac{\tau}{4},\qquad v_{3}\to\phi_{2}-\frac{\tau}{4},\qquad
m\to\frac{\tau}{2}.$ (119)

Substituting these into the one-string elliptic genus for the $G_{2}$ theory
given in (106), we get the one-string elliptic genus for the $SU(3)$ gauge
theory with $\mathbb{Z}_{2}$ twist:

$\displaystyle
Z_{1}=\frac{2\,\eta^{6}\,\theta_{1}(-2\phi_{2}+\frac{\tau}{2})}{\theta_{1}(\epsilon_{-})^{2}\theta_{1}(2\phi_{2})^{2}\theta_{1}(\frac{\tau}{2})\theta_{1}(-\phi_{2}+\frac{\tau}{4})\theta_{1}(\phi_{2}-\frac{3\tau}{4})}\
.$ (120)

It follows that two-string elliptic genus for the $SU(3)$ gauge theory with
$\mathbb{Z}_{2}$ twist is given by

$\displaystyle Z_{2}$ $\displaystyle=\frac{\eta^{12}\,\theta_{1}(\pm
2\phi_{2}+\frac{\tau}{2})}{\theta_{1}(\epsilon_{-})^{4}\theta_{1}(\epsilon_{-}\pm
2\phi_{2})^{2}\theta_{1}(\pm\phi_{2}+\frac{\tau}{4})\theta_{1}(\pm\epsilon_{-}+\frac{\tau}{2})\theta_{1}(\pm\phi_{2}+\frac{3\tau}{4})}$
$\displaystyle\quad+\bigg{(}\frac{2\eta^{12}\theta_{1}(\epsilon_{-}-2\phi_{2}+\frac{\tau}{2})\theta_{1}(2\epsilon_{-}-2\phi_{2}+\frac{\tau}{2})}{\theta_{1}(\epsilon_{-})^{2}\theta_{1}(2\epsilon_{-})^{2}\theta_{1}(2\phi_{2})^{2}\theta_{1}(\epsilon_{-}-2\phi_{2})^{2}\theta_{1}(-\phi_{2}+\frac{\tau}{4})\theta_{1}(\epsilon_{-}-A+\frac{\tau}{4})}$
$\displaystyle\qquad\quad\times\frac{1}{\theta_{1}(\frac{\tau}{2})\theta_{1}(\epsilon_{-}+\frac{\tau}{2})\theta_{1}(-\phi_{2}+\frac{3\tau}{4})\theta_{1}(\epsilon_{-}-\phi_{2}+\frac{3\tau}{4})}+(\epsilon_{-}\to-\epsilon_{-})\bigg{)}.$
(121)

It is straightforward to see that these one- and two-string elliptic genera,
(120) and (3.3), agree with the $Z_{1}$ and $Z_{2}$ obtained from topological
vertex given in (3.2), by double expanding (120) and (3.3) in terms of $q$ and
$A$ and also taking the unrefined limit.

##  4 Conclusion

In this paper, we computed the partition functions of 6d $SO(8)$ and $SU(3)$
gauge theories with $\mathbb{Z}_{2}$ outer automorphism twist in two different
perspectives. One is to use their Type IIB 5-brane webs where 6d conformal
matter theories of D-type gauge symmetry can be engineered as 5-brane webs
with two $\mathrm{O}5$-planes. Among various RG flows, we discussed the
Higgsing procedure on 5-brane webs giving rise to the 6d $SO(8)$ and $SU(3)$
gauge theories with $\mathbb{Z}_{2}$ twist, from the 6d $SO(10)$ gauge theory
with two flavors and from the 6d $G_{2}$ gauge theory with a flavor,
respectively. We computed the partition functions of these theories following
the topological vertex formalism in the presence of $\mathrm{O}5$-planes
developed in Kim:2017jqn , by introducing and decoupling spinor matter fields
to implement the topological vertex as done Hayashi:2018bkd . The other is to
directly apply the outer automorphism twists on the elliptic genera for the
$SO(10)$ and $G_{2}$ gauge theories, by Higgsing the hypermultiplets with
proper KK-momentum dependence. We checked that the partition functions based
on topological vertex with O5-planes agree with the elliptic genera after
Higgsings. As the elliptic genera are fully refined, we compared them in the
unrefined limit as well as by double expanding each in terms of the KK
momentum and the Coulomb branch parameters.

When computing the elliptic genus for the $SU(3)$ theory with $\mathbb{Z}_{2}$
twist in section 3.3, one may wonder whether one can apply the twisting
directly on the elliptic genus for the $SU(3)$ theory, implementing the KK-
momentum shifts as done for the $SO(8)$ case in section 2.4. In the case of
$SO(8)$ theory with $\mathbb{Z}_{2}$ twist, the order two outer automorphism
maps fundamental representation to itself, and hence KK-momentum shifts can be
easily understood. For $\mathfrak{su}(3)$ algebra, however, the fundamental
representation $\mathbf{3}$ maps to the anti-fundamental representation
$\bar{\mathbf{3}}$ under the order two outer automorphism. From the ADHM
perspective, there are one chiral multiplet which transforms as $\mathbf{3}$
and two chiral multiplets which transform as $\bar{\mathbf{3}}$, and hence it
is not clear how to perform the automorphism twist on these $SU(3)$ states in
the ADHM construction. It would be good if a more systematic study along this
direction is carried out so that it would be even applicable to order three
outer automorphism twist.

Yet as another independent check, we computed, in Kim:2020hhh , the BPS
spectrum of the 6d theories with $\mathbb{Z}_{2}$ twist using the bootstrap
method via the Nakajima-Yoshioka’s blowup equations, which provides fully
refined partition functions. We checked our results against the BPS spectrum
from the blowup formula and found that two results completely match.

## Acknowledgements

We would like to thank Kimyeong Lee, Kaiwen Sun, Xing-Yue Wei, and Futoshi
Yagi for useful discussions. S.K. thanks APCTP, KIAS, and POSTECH for
hospitality for his visit. The research of HK and MK is supported by the POSCO
Science Fellowship of POSCO TJ Park Foundation and the National Research
Foundation of Korea (NRF) Grant 2018R1D1A1B07042934.

##  Appendix A Twisted boundary condition and affine Lie algebras

Consider a 6d gauge theory with a simple Lie algebra $\mathfrak{g}$ and
compactification of the theory on a circle $S^{1}$ of radius $R$. For the
gauge fields $A_{\mu}=A_{\mu}^{a}T^{a}$, where $T^{a}$ lie in the adjoint
representation of $\mathfrak{g}$, we impose the periodic boundary condition

$\displaystyle A_{\mu}(x^{i},x^{6}+2\pi R)=A_{\mu}(x^{i},x^{6})\ .$ (122)

Here, $i=1,2,\cdots,5$ and $x^{6}$ is the coordinate along $S^{1}$. Fourier
expansion of the gauge fields preserving the periodic boundary condition takes
the form

$\displaystyle
A_{\mu}(x^{i},x^{6})=A_{\mu}^{a}(x^{i},x^{6})T^{a}=\sum_{n\in\mathbb{Z}}e^{i\frac{nx^{6}}{R}}A_{\mu,n}^{a}(x^{i})T^{a}=\sum_{n\in\mathbb{Z}}A_{\mu,n}^{a}(x^{i})T^{a}_{n},$
(123)

where $T^{a}_{n}=T^{a}e^{inx^{6}/R}$. The new basis $T^{a}_{n}$ satisfies

$\displaystyle\commutator{T^{a}_{n}}{T^{b}_{m}}=f^{abc}\,T^{c}_{n+m}\ ,$ (124)

where $f^{abc}$ is the structure constant of $\mathfrak{g}$,
$\commutator*{T^{a}}{T^{b}}=f^{abc}T^{c}$. The basis $\\{T^{a}_{n}\\}$ with
possible central extensions generates an untwisted affine Lie algebra. For a
simple Lie algebra of type $X_{\ell}$, the untwisted affine Lie algebra
constructed using the periodic boundary condition (122) is denoted by
$X_{\ell}^{(1)}$, and their Dynkin diagram is shown in Figure 18. The black
circles are affine nodes. Excluding it gives the Dynkin diagram of
corresponding simple Lie algebra $X_{\ell}$.

Figure 18: Dynkin diagrams of untwisted affine Lie algebras, where the black
node $\bullet$ represents the affine node.

Instead of the periodic boundary condition, one can impose a twisted boundary
condition

$\displaystyle A_{\mu}(x^{i},x^{6}+2\pi R)=\sigma(A_{\mu}(x^{i},x^{6}))\ ,$
(125)

where $\sigma$ is a finite order automorphism of $\mathfrak{g}$. If $\sigma$
is an order $m$ automorphism, i.e., $\sigma^{m}=1$, then the Fourier expansion
of the gauge fields preserving the twisted boundary condition (125) is

$\displaystyle
A_{\mu}(x^{i},x^{6})=\sum_{n,k}e^{i\frac{x^{6}}{R}(n+\frac{k}{m})}A_{\mu,n,k}^{a}(x^{i})T^{a}=\sum_{n,k}A_{\mu,n,k}^{a}T^{a}_{n+k/l}\
,$ (126)

where $T^{a}_{n+k/m}=T^{a}e^{i\frac{x^{6}}{R}(n+\frac{k}{m})}$. The
commutation relation of new basis is given by

$\displaystyle\commutator{T^{a}_{n+k/m}}{T^{b}_{n^{\prime}+k^{\prime}/m}}=f^{abc}T^{c}_{n+n^{\prime}+(k+k^{\prime})/m}\
.$ (127)

If $\sigma$ is a conjugation, it is called an inner automorphism. In this
case, the resultant algebra is the same as untwisted affine Lie algebra
Fuchs:1992nq . If $\sigma$ is not an inner automorphism, it is called an outer
automorphism and the resultant algebra becomes different from untwisted affine
Lie algebras. The new algebra generated by $\\{T^{a}_{n+m/l}\\}$ with possible
central extensions is called a twisted affine Lie algebra.

An outer automorphism can be viewed as a graph automorphism of Dynkin
diagrams. Only the simple Lie algebras of types $A_{n}$, $D_{n}$ and $E_{6}$
have nontrivial outer automorphisms. $A_{n}$ has an $\mathbb{Z}_{2}$ Dynkin
diagram automorphism which exchanges the simple roots $\alpha_{i}$ and
$\alpha_{n+1-i}$ as in Figure 19(a) and Figure 19(b). $D_{n}$ has an order two
outer automorphism which exchanges the simple roots $\alpha_{n-1}$ and
$\alpha_{n}$ as in Figure 19(c). The exceptional algebra $E_{6}$ also has an
order two outer automorphism shown in Figure 19(e). The $D_{4}$ algebra
additionally has an order three automorphism as Figure 19(d). For these simple
Lie algebras $X_{\ell}$, twisted affine Lie algebras associated with order
$r=2,3$ diagram automorphism are denoted by $X_{\ell}^{(r)}$, and their Dynkin
diagrams are given in Figure 20. Note that unlike the untwisted case,
excluding the affine node does not yield $X_{\ell}$ algebra. For example,
deleting the affine node from $A_{2\ell}^{(2)}$ and $D_{\ell+1}^{(2)}$
algebras give the Dynkin diagram of $C_{\ell}$ and $B_{\ell}$ algebras,
respectively.

Figure 19: Dynkin diagrams and graph outer automorphisms of simple Lie
algebras of type (a) $A_{2n+1}$, (b) $A_{2n}$, (c) $D_{n}$, (d) $D_{4}$ and
(e) $E_{6}$. Figure 20: Dynkin diagrams of twisted affine Lie algebras.

Since a 6d gauge theory compactified on a circle naturally has an affine Lie
algebra structure, its perturbative spectrum can be read off from the
representation theory of Lie algebras. One can write down the perturbative
part of the partition function by using either affine root system or by
decomposing the representation of 6d gauge algebra into the representation of
the invariant subalgebra under the automorphism. For more details, see
Kac:1990gs ; Kim:2004xx  and also Appendix A of Kim:2020hhh .

We list some relevant results of the perturbative part of the partition
function $Z_{\rm pert}$, used in this paper.
$(i)$ The untwisted compactification: for a 6d gauge theory with gauge algebra
$\mathfrak{g}$, the W-boson contribution to the perturbative part is given by,
in the unrefined limit,

$\displaystyle
Z_{\mathrm{pert}}=\operatorname{PE}\quantity[\frac{2g}{(1-g)^{2}}\frac{1}{1-q}\quantity(\chi_{\Delta_{+}}^{\mathfrak{g}}+q\,\chi_{\Delta_{-}}^{\mathfrak{g}})]\
,$ (128)

where $\chi_{\Delta_{\pm}}^{\mathfrak{g}}$ are the character associated with
the positive/negative roots, respectively. The twisted compactifications that
we discussed in the main text are $SO(8)$ gauge theory with $\mathbb{Z}_{2}$
twist and $SU(3)$ gauge theory with $\mathbb{Z}_{2}$ twist. Their invariant
subalgebras carry fractional KK-momentum. These fractional momenta contribute
to the perturbative part.

$(ii)$ $SO(8)$ gauge theory with $\mathbb{Z}_{2}$ twist: the adjoint
representation of $\mathfrak{so}(8)$ decomposes into the adjoint and the
fundamental representations of $\mathfrak{so}(7)$. The adjoint representation
of $\mathfrak{so}(7)$ carries integer KK charge, while the fundamental
representation carries half-integer KK charge,

$\displaystyle D_{4}$ $\displaystyle\to B_{3}$ $\displaystyle\mathbf{28}$
$\displaystyle\to\mathbf{21}_{0}\oplus\mathbf{7}_{1/2}\ ,$ (129)

where the subscript denotes the KK charge. From this, one finds that the
perturbative contribution to the partition function for $SO(8)$ gauge theory
with $\mathbb{Z}_{2}$ twist is given by

$\displaystyle
Z_{\mathrm{pert}}=\operatorname{PE}\quantity[\frac{2g}{(1-g)^{2}}\frac{1}{1-q}\quantity(\chi_{\Delta_{+}}^{\mathfrak{so}(7)}+q^{1/2}\chi_{\mathbf{7}}^{\mathfrak{so}(7)}+q\,\chi_{\Delta_{-}}^{\mathfrak{so}(7)})]\,.$
(130)

In the dimensional reduction limit where $R\to 0$, all the states with non-
zero KK-momentum are truncated so that only the adjoint representation of
$\mathfrak{so}(7)$ algebra remains. We note that the invariant subalgebra
$\mathfrak{so}(7)$ can be readily read off, as it it nothing but the algebra
obtained by removing the affine node of the Dynkin diagram of $D_{4}^{(2)}$.

$(ii)$ $SU(3)$ gauge theory with $\mathbb{Z}_{2}$ twist: the adjoint
representation of $\mathfrak{su}(3)$ decomposes into the representations of
$\mathfrak{su}(2)$ as

$\displaystyle A_{2}$ $\displaystyle\to A_{1}$ $\displaystyle\mathbf{8}$
$\displaystyle\to\mathbf{3}_{0}\oplus\mathbf{2}_{1/4}\oplus\mathbf{2}_{3/4}\oplus\mathbf{1}_{1/2}\,.$
(131)

The perturbative contribution to the partition function for $SU(3)$ gauge
theory with $\mathbb{Z}_{2}$ twist is then given by

$\displaystyle
Z_{\mathrm{pert}}=\operatorname{PE}\quantity[\frac{2g}{(1-g)^{2}}\frac{1}{1-q}\quantity(\chi_{\Delta_{+}}^{\mathfrak{su}(2)}+(q^{1/4}+q^{3/4})\chi_{\mathbf{2}}^{\mathfrak{su}(2)}+q\,\chi_{\Delta_{-}}^{\mathfrak{su}(2)})]\
,$ (132)

up to the Cartan part. Again, in dimensional reduction limits, only the
adjoint representation of $\mathfrak{su}(2)$ survives, and this
$\mathfrak{su}(2)$ is the algebra obtained by removing the affine node of the
Dynkin diagram of $A_{2}^{(2)}$.

##  Appendix B Special functions

The Plethystic exponential is defined by

$\displaystyle\operatorname{PE}[f(x)]=\exp(\sum_{n=1}^{\infty}\frac{1}{n}f(x^{n}))\
.$

Its inverse function, Plethystic logarithm, is given by

$\displaystyle\operatorname{PL}\quantity[f(x)]=\operatorname{PE}^{-1}\quantity[f(x)]=\sum_{n=1}^{\infty}\frac{\mu(n)}{n}\log
f(x^{n})\ ,$ (133)

where $\mu(n)$ is the Möbius function defined by

$\displaystyle\mu(n)=\left\\{\begin{array}[]{ll}(-1)^{p}&~{}\text{if $n$ is a
square-free positive integer with $p$ prime factors,}\\\
0&~{}\text{if~{}}n\text{ has a squared prime factor.}\end{array}\right.$ (136)

In topological vertex formalism, we use the following special functions for
integer partitions
$\lambda=(\lambda_{1},\lambda_{2},\cdots,\lambda_{\ell(\lambda)})$ and
$\mu=(\mu_{1},\mu_{2},\cdots,\mu_{\ell(\mu)})$,

$\displaystyle\tilde{Z}_{\nu}(g)$
$\displaystyle=\tilde{Z}_{\nu^{t}}(g)=\prod_{i=1}^{\ell(\nu)}\prod_{j=1}^{\nu_{i}}\quantity(1-g^{\nu_{i}+\nu_{j}^{t}-i-j+1})^{-1},$
(137) $\displaystyle\mathcal{R}_{\lambda\mu}(Q)$
$\displaystyle=\mathcal{M}(Q)^{-1}\mathcal{N}_{\lambda^{t}\mu}(Q)\ ,$ (138)
$\displaystyle\mathcal{M}(Q)$
$\displaystyle=\prod_{i,j=1}^{\infty}(1-Qg^{i+j-1})^{-1}=\operatorname{PE}\quantity[\frac{gQ}{(1-g)^{2}}]\
,$ (139) $\displaystyle\mathcal{N}_{\lambda\mu}(Q)$
$\displaystyle=\quantity[\prod_{i=1}^{\ell(\lambda)}\prod_{j=1}^{\lambda_{i}}\quantity(1-Qg^{\lambda_{i}+\mu^{t}_{j}-i-j+1})]\quantity[\prod_{i=1}^{\ell(\mu)}\prod_{j=1}^{\mu_{i}}\quantity(1-Qg^{-\lambda^{t}_{j}-\mu_{i}+i+j-1})]\
,$ (140)

where $g$ the unrefined $\Omega$-deformation parameter, $\nu^{t}$ means
transposed partition of $\nu$, and $Q$ is a Kähler parameter. In practical
calculation, the following Cauchy identities macdonald_symmetric_1995  are
also handy:

$\displaystyle\sum_{\lambda}Q^{\absolutevalue{\lambda}}s_{\lambda/\eta_{1}}(\mathbf{x})s_{\lambda/\eta_{2}}(\mathbf{y})$
$\displaystyle=\prod_{i,j}\frac{1}{1-Qx_{i}y_{j}}\sum_{\lambda}Q^{\absolutevalue{\lambda}}s_{\eta_{2}/\lambda}(\mathbf{x})s_{\eta_{1}/\lambda}(\mathbf{y})\
,$ (141)
$\displaystyle\sum_{\lambda}Q^{\absolutevalue{\lambda}}s_{\lambda/\eta_{1}^{t}}(\mathbf{x})s_{\lambda^{t}/\eta_{2}}(\mathbf{y})$
$\displaystyle=\prod_{i,j}\quantity(1+Qx_{i}y_{j})\sum_{\lambda}Q^{\absolutevalue{\lambda}}s_{\eta_{2}^{t}/\lambda}(\mathbf{x})s_{\eta_{1}/\lambda^{t}}(\mathbf{y})\
.$ (142)

When $\mathbf{x}=g^{-\rho-\nu_{1}}$ and $\mathbf{y}=g^{-\rho-\nu_{2}}$ for
$\rho=(-\frac{1}{2},-\frac{3}{2},-\frac{5}{2},\cdots)$, as discussed in the
main text,

$\displaystyle\sum_{\lambda}Q^{\absolutevalue{\lambda}}s_{\lambda/\eta_{1}}(g^{-\rho-\nu_{1}})s_{\lambda/\eta_{2}}(g^{-\rho-\nu_{2}})=\mathcal{R}_{\nu_{2}\nu_{1}}(Q)^{-1}\sum_{\lambda}Q^{\absolutevalue{\eta_{1}}+\absolutevalue{\eta_{2}}-\absolutevalue{\lambda}}s_{\eta_{2}/\lambda}(g^{-\rho-\nu_{1}})s_{\eta_{1}/\lambda}(g^{-\rho-\nu_{2}}),$
$\displaystyle\sum_{\lambda}Q^{\absolutevalue{\lambda}}s_{\lambda/\eta_{1}^{t}}(g^{-\rho-\nu_{1}})s_{\lambda^{t}/\eta_{2}}(g^{-\rho-\nu_{2}})=\mathcal{R}_{\nu_{2}\nu_{1}}(-Q)\sum_{\lambda}Q^{\absolutevalue{\eta_{1}}+\absolutevalue{\eta_{2}}-\absolutevalue{\lambda}}s_{\eta_{2}^{t}/\lambda}(g^{-\rho-\nu_{1}})s_{\eta_{1}/\lambda^{t}}(g^{-\rho-\nu_{2}}).$

We also note that skew Schur function $s_{\lambda/\mu}$ is zero unless
$\lambda\supset\mu$.

For a periodic strip diagram given in Figure 16, the explicit form of such
periodic vertex can be obtained using the method in Haghighat:2013gba ;
macdonald_symmetric_1995 . Using the definition of the edge factor and the
vertex factor, one needs to evaluate the equation of the form

$\displaystyle\mathcal{G}(\mathbf{x}_{1},\\!\cdots\\!,\mathbf{x}_{4},\mathbf{y}_{1},\\!\cdots\\!,\mathbf{y}_{4})$
$\displaystyle=\sum_{\lambda_{i},\eta_{i}}q_{1}^{\absolutevalue{\lambda_{1}}}q_{2}^{\absolutevalue{\lambda_{2}}}q_{3}^{\absolutevalue{\lambda_{3}}}q_{4}^{\absolutevalue{\lambda_{4}}}s_{\lambda_{1}/\eta_{1}}(\mathbf{x}_{1})s_{\lambda_{1}/\eta_{4}}(\mathbf{y}_{1})s_{\lambda_{2}/\eta_{1}}(\mathbf{x}_{2})$
$\displaystyle~{}~{}\times
s_{\lambda_{2}/\eta_{2}}(\mathbf{y}_{2})s_{\lambda_{3}/\eta_{2}}(\mathbf{x}_{3})s_{\lambda_{3}/\eta_{3}}(\mathbf{y}_{3})s_{\lambda_{4}/\eta_{3}}(\mathbf{x}_{4})s_{\lambda_{4}/\eta_{4}}(\mathbf{y}_{4}).$
(143)

A successive use of the Cauchy identities (141) yields

$\displaystyle\mathcal{G}(\mathbf{x}_{1},\cdots,\mathbf{y}_{4})=\mathcal{F}(\mathbf{x}_{1},\cdots,\mathbf{y}_{4})\mathcal{G}(Q\mathbf{x}_{1},\cdots,Q\mathbf{x}_{4},Q\mathbf{y}_{1},\cdots,Q\mathbf{y}_{4}),$
(144)

where $Q=q_{1}q_{2}q_{3}q_{4}$ and
$\mathcal{F}(\mathbf{x}_{1},\cdots,\mathbf{y}_{4})=F(\mathbf{x}_{1},\cdots,\mathbf{y}_{4})F(Q^{1/2}\mathbf{x}_{1},\cdots,Q^{1/2}\mathbf{y}_{4})$
for

$\displaystyle F(\mathbf{x}_{1},\cdots,\mathbf{y}_{4})$ (145)
$\displaystyle=\prod\Bigg{[}\frac{1}{\left(1-q_{1}\mathbf{x}_{1}\mathbf{y}_{1}\right)\left(1-q_{2}\mathbf{x}_{2}\mathbf{y}_{2}\right)\left(1-q_{3}\mathbf{x}_{3}\mathbf{y}_{3}\right)\left(1-q_{4}\mathbf{x}_{4}\mathbf{y}_{4}\right)\left(1-q_{1,2}\mathbf{y}_{1}\mathbf{y}_{2}\right)}$
$\displaystyle\times\frac{1}{\left(1-q_{1,4}\mathbf{x}_{1}\mathbf{x}_{4}\right)\left(1-q_{2,3}\mathbf{x}_{2}\mathbf{y}_{3}\right)\left(1-q_{3,4}\mathbf{x}_{3}\mathbf{y}_{4}\right)\left(1-q_{1,2,3}\mathbf{y}_{1}\mathbf{y}_{3}\right)\left(1-q_{1,2,4}\mathbf{y}_{2}\mathbf{x}_{4}\right)}$
$\displaystyle\times\frac{1}{\left(1-q_{1,3,4}\mathbf{x}_{3}\mathbf{x}_{1}\right)\left(1-q_{2,3,4}\mathbf{x}_{2}\mathbf{y}_{4}\right)\left(1-Q\mathbf{x}_{2}\mathbf{x}_{1}\right)\left(1-Q\mathbf{y}_{2}\mathbf{x}_{3}\right)\left(1-Q\mathbf{x}_{4}\mathbf{y}_{3}\right)\left(1-Q\mathbf{y}_{1}\mathbf{y}_{4}\right)}\Bigg{]}.$

Here, $q_{i,j,\cdots}=q_{i}q_{j}\cdots$ and
$\prod(1-Q\mathbf{x}_{i}\mathbf{y}_{j})=\prod_{m,n}\big{(}1-(\mathbf{x}_{i})_{m}(\mathbf{y}_{j})_{n}Q\big{)}$
for integer partitions $\mathbf{x}_{i}$ and $\mathbf{y}_{j}$. After repeating
this procedure for $m$ times, we need to change $\mathbf{x}_{i}$ and
$\mathbf{y}_{j}$ to $Q^{m}\mathbf{x}_{i}$ and $Q^{m}\mathbf{y}_{j}$,
respectively, and hence,

$\displaystyle\mathcal{G}(\mathbf{x}_{1},\cdots,\mathbf{y}_{4})=\quantity[\prod_{m=0}^{n}\mathcal{F}(Q^{m}\mathbf{x}_{1},\cdots,Q^{m}\mathbf{y}_{4})]\mathcal{G}(Q^{n+1}\mathbf{x}_{1},\cdots,Q^{n+1}\mathbf{y}_{4}).$
(146)

Under the condition $Q\to 0$ as $n\to\infty$ which is used for deriving the
Cauchy identities, the only contributing terms in
$\mathcal{G}(Q^{m}\mathbf{x}_{1},\cdots,Q^{m}\mathbf{y}_{4})$ are
$\lambda_{i}=\eta_{j}$ for all $i$ and $j$:

$\displaystyle\lim_{n\to\infty}\mathcal{G}(Q^{n}\mathbf{x}_{1},\cdots,Q^{n}\mathbf{y}_{4})=\sum_{\lambda}Q^{\absolutevalue{\lambda}}=\prod_{m=1}^{\infty}\frac{1}{1-Q^{m}}\
.$ (147)

Hence, it follows that

$\displaystyle\mathcal{G}(\mathbf{x}_{1},\cdots,\mathbf{y}_{4})=\prod_{n=1}^{\infty}\frac{\mathcal{F}(Q^{n-1}\mathbf{x}_{1},\cdots,Q^{n-1}\mathbf{y}_{4})}{1-Q^{n}}\
.$ (148)

In localization computation, the Dedekind eta function $\eta$ and the Jacobi
theta function $\theta_{1}(x)$ are defined as follows: For the complex
structure $\tau$ of a torus, $q=e^{2\pi i\tau}$,

$\displaystyle\eta$
$\displaystyle=q^{\frac{1}{24}}\prod_{n=1}^{\infty}(1-q^{n})\ ,$ (149)
$\displaystyle\theta_{1}(x)$
$\displaystyle=-iq^{\frac{1}{8}}y^{\frac{1}{2}}\prod_{n=1}^{\infty}(1-q^{n})(1-yq^{n})(1-y^{-1}q^{n-1})\
,$ (150)

where $y=e^{2\pi ix}$. They satisfy

$\displaystyle\theta_{1}(-x)=-\theta_{1}(x),\quad\theta_{1}(n\tau)=0\quad(n\in\mathbb{Z})\
,$ $\displaystyle\frac{1}{2\pi
i}\oint_{u=0}\frac{du}{\theta_{1}(u)}=\frac{1}{2\pi\eta^{3}}\ ,$ (151)

where $\oint_{u=0}$ means that the integral contour is taken around $u=0$ so
that only the residue at $u=0$ contributes. These properties are useful when
we evaluate the JK-residue.

## References

  * (1) H.-C. Kim, S.-S. Kim and K. Lee, _Higgsing and Twisting of 6d $D_{N}$ gauge theories_, 1908.04704.
  * (2) O. Aharony and A. Hanany, _Branes, superpotentials and superconformal fixed points_ , _Nucl.Phys._ B504 (1997) 239 [hep-th/9704170].
  * (3) O. Aharony, A. Hanany and B. Kol, _Webs of (p,q) five-branes, five-dimensional field theories and grid diagrams_ , _JHEP_ 9801 (1998) 002 [hep-th/9710116].
  * (4) J. J. Heckman, D. R. Morrison, T. Rudelius and C. Vafa, _Atomic Classification of 6D SCFTs_ , 1502.05405.
  * (5) H. Hayashi, S.-S. Kim, K. Lee, M. Taki and F. Yagi, _More on 5d descriptions of 6d SCFTs_ , _JHEP_ 10 (2016) 126 [1512.08239].
  * (6) S.-S. Kim and F. Yagi, _Topological vertex formalism with O5-plane_ , _Phys. Rev._ D97 (2018) 026011 [1709.01928].
  * (7) E. Witten, _Phase transitions in M theory and F theory_ , _Nucl. Phys._ B471 (1996) 195 [hep-th/9603150].
  * (8) K. A. Intriligator, D. R. Morrison and N. Seiberg, _Five-dimensional supersymmetric gauge theories and degenerations of Calabi-Yau spaces_ , _Nucl.Phys._ B497 (1997) 56 [hep-th/9702198].
  * (9) P. Jefferson, S. Katz, H.-C. Kim and C. Vafa, _On Geometric Classification of 5d SCFTs_ , _JHEP_ 04 (2018) 103 [1801.04036].
  * (10) L. Bhardwaj, P. Jefferson, H.-C. Kim, H.-C. Tarazi and C. Vafa, _Twisted Circle Compactifications of 6d SCFTs_ , 1909.11666.
  * (11) J. J. Heckman, D. R. Morrison and C. Vafa, _On the Classification of 6D SCFTs and Generalized ADE Orbifolds_ , _JHEP_ 1405 (2014) 028 [1312.5746].
  * (12) M. Del Zotto, J. J. Heckman, A. Tomasiello and C. Vafa, _6d Conformal Matter_ , _JHEP_ 1502 (2015) 054 [1407.6359].
  * (13) P. Jefferson, H.-C. Kim, C. Vafa and G. Zafrir, _Towards Classification of 5d SCFTs: Single Gauge Node_ , 1705.05836.
  * (14) H. Hayashi, S.-S. Kim, K. Lee and F. Yagi, _Dualities and 5-brane webs for 5d rank 2 SCFTs_ , _JHEP_ 12 (2018) 016 [1806.10569].
  * (15) M. Aganagic, A. Klemm, M. Marino and C. Vafa, _The Topological vertex_ , _Commun.Math.Phys._ 254 (2005) 425 [hep-th/0305132].
  * (16) A. Iqbal, C. Kozcaz and C. Vafa, _The Refined topological vertex_ , _JHEP_ 0910 (2009) 069 [hep-th/0701156].
  * (17) H. Awata and H. Kanno, _Refined BPS state counting from Nekrasov’s formula and Macdonald functions_ , _Int.J.Mod.Phys._ A24 (2009) 2253 [0805.0191].
  * (18) B. Haghighat, C. Kozcaz, G. Lockhart and C. Vafa, _Orbifolds of M-strings_ , _Phys. Rev._ D89 (2014) 046003 [1310.1185].
  * (19) J. Kim, S. Kim, K. Lee, J. Park and C. Vafa, _Elliptic Genus of E-strings_ , 1411.2324.
  * (20) B. Haghighat, A. Klemm, G. Lockhart and C. Vafa, _Strings of Minimal 6d SCFTs_ , _Fortsch.Phys._ 63 (2015) 294 [1412.3152].
  * (21) A. Gadde, B. Haghighat, J. Kim, S. Kim, G. Lockhart and C. Vafa, _6d String Chains_ , 1504.04614.
  * (22) H. Hayashi and R.-D. Zhu, _More on topological vertex formalism for 5-brane webs with O5-plane_ , 2012.13303.
  * (23) F. Benini, R. Eager, K. Hori and Y. Tachikawa, _Elliptic genera of two-dimensional N=2 gauge theories with rank-one gauge groups_ , _Lett. Math. Phys._ 104 (2014) 465 [1305.0533].
  * (24) F. Benini, R. Eager, K. Hori and Y. Tachikawa, _Elliptic Genera of 2d ${\mathcal{N}}$ = 2 Gauge Theories_, _Commun. Math. Phys._ 333 (2015) 1241 [1308.4896].
  * (25) N. J. Evans, C. V. Johnson and A. D. Shapere, _Orientifolds, branes, and duality of 4-D gauge theories_ , _Nucl. Phys._ B505 (1997) 251 [hep-th/9703210].
  * (26) A. Giveon and D. Kutasov, _Brane dynamics and gauge theory_ , _Rev. Mod. Phys._ 71 (1999) 983 [hep-th/9802067].
  * (27) B. Feng and A. Hanany, _Mirror symmetry by O3 planes_ , _JHEP_ 11 (2000) 033 [hep-th/0004092].
  * (28) G. Bertoldi, B. Feng and A. Hanany, _The Splitting of branes on orientifold planes_ , _JHEP_ 04 (2002) 015 [hep-th/0202090].
  * (29) G. Zafrir, _Brane webs and $O5$-planes_, _JHEP_ 03 (2016) 109 [1512.08114].
  * (30) H. Hayashi, S.-S. Kim, K. Lee and F. Yagi, _Discrete theta angle from an O5-plane_ , _JHEP_ 11 (2017) 041 [1707.07181].
  * (31) L. C. Jeffrey and F. C. Kirwan, _Localization for nonabelian group actions_ , in _eprint arXiv:alg-geom/9307001_ , p. 7001, July, 1993.
  * (32) Y. Tachikawa, _On S-duality of 5d super Yang-Mills on $S^{1}$_, _JHEP_ 11 (2011) 123 [1110.0531].
  * (33) B. Haghighat, A. Iqbal, C. Kozcaz, G. Lockhart and C. Vafa, _M-Strings_ , _Commun. Math. Phys._ 334 (2015) 779 [1305.6322].
  * (34) I. G. Macdonald, _Symmetric Functions and Hall Polynomials_. Oxford University Press, 2 ed., 1995.
  * (35) H.-C. Kim, J. Kim, S. Kim, K.-H. Lee and J. Park, _6d strings and exceptional instantons_ , 1801.03579.
  * (36) H. Hayashi, S.-S. Kim, K. Lee and F. Yagi, _5-brane webs for 5d $\mathcal{N}$ = 1 G2 gauge theories_, _JHEP_ 03 (2018) 125 [1801.03916].
  * (37) H.-C. Kim, M. Kim, S.-S. Kim and K.-H. Lee, _Bootstrapping BPS spectra of 5d/6d field theories_ , 2101.00023.
  * (38) J. Fuchs, _Affine Lie algebras and quantum groups: An Introduction, with applications in conformal field theory_. Cambridge University Press, 3, 1995\.
  * (39) V. G. Kac, _Infinite dimensional Lie algebras_. Cambridge University Press, 3 ed., 1990.
  * (40) S. Kim, K.-M. Lee, H.-U. Yee and P. Yi, _The $N=1^{*}$ theories on $R^{1+2}\times S^{1}$ with twisted boundary conditions_, _JHEP_ 08 (2004) 040 [hep-th/0403076].