aainstitutetext: Department of Physics and Center for Field Theory & Particle
Physics, Fudan University, 20005, Songhu Road, 200438 Shanghai,
Chinabbinstitutetext: Institut de Mathématiques de Bourgogne, Université de
Bourgogne Franche-Comté,
9 avenue Alain Savary, Dijon, France

# Instantons on Young diagrams with matters

Yanyan Chen a Jiaqun Jiang a Satoshi Nawata b and Yilu Shao
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We present the unrefined instanton partition functions of various 5d gauge
theories with matter beyond the fundamental representation as sums over Young
diagrams. By using these explicit expressions, we verify a range of identities
among the instanton partition functions predicted by Higgsing procedures of
fivebrane web diagrams and representation theory.

††preprint:

##  1 Introduction

One of the ultimate goals of quantum field theory is to obtain exact results,
including non-perturbative effects. In recent decades, significant progress
has been made in understanding the non-perturbative nature of quantum field
theory. Starting with the seminal work by Seiberg and Witten Seiberg:1994aj ;
Seiberg:1994rs , deep insights have been gained into supersymmetric theories
with eight supercharges. In particular, Nekrasov performed the first instance
of supersymmetric localization for instanton partition functions
Nekrasov:2002qd , providing exact results for certain observables in the
Seiberg-Witten theory, such as the low-energy effective prepotential.

More remarkably, the fixed points of equivariant actions on the instanton
moduli spaces are classified by a set of Young diagrams, and the partition
function is expressed as a sum over Young diagrams. Nekrasov’s exact result
has far-reaching consequences in both physics and mathematics. For example,
the expression as a sum over Young diagrams is important because it provides a
connection to the topological vertex AKMV ; Awata:2005fa ; IKV . It is also
particularly useful in the context of the AGT relation Alday:2009aq , where
the instanton partition function is identified with conformal blocks of a 2d
CFT.

These developments were exclusively made in the gauge groups of $A$ type until
recent years. However, closed-form expressions of the unrefined instanton
partition functions for pure Yang-Mills theory with gauge groups of $BCD$ type
were recently obtained as sums over Young diagrams Nawata:2021dlk .
Additionally, the connection to the topological vertex Hayashi:2020hhb  of an
O5-plane was uncovered.

In this paper, we push forward the research direction in Nawata:2021dlk ,
aiming to obtain unrefined instanton partition functions with a hypermultiplet
beyond the fundamental representation as sums over 2d and 4d Young diagrams.
We combine two approaches to studying 5d supersymmetric gauge theories with
matters beyond the fundamental representation. The first approach is the ADHM
description, which has been studied in previous works such as Nekrasov-
Shadchin ; Shadchin:2005mx ; Kim:2012gu ; Kim:2012qf ; Hwang:2014uwa . The
second approach is the fivebrane construction, which has been extensively
studied in Bergman:2014kza ; Bergman:2015dpa ; Zafrir:2015ftn ;
Hayashi:2016jak ; Kim-Yagi ; Kim:2018gjo ; Hayashi:2018bkd ; Hayashi:2018lyv ;
Cheng:2018wll ; Hayashi:2019yxj ; Kim:2021cua ; Kim:2022dbr . Our main results
are as follows: first, we demonstrate that the poles of the JK residue
integral arising from the ADHM descriptions can be classified by 2d and 4d
Young diagrams in the unrefined limit. Second, we verify identities of the
instanton partition functions predicted by Higgsing procedures of fivebrane
web diagrams and representation theory.

The paper is structured as follows. In §2, we investigate the
$\operatorname{SU}(N)$ gauge theory with (anti-)symmetric hypermultiplet, and
we represent the unrefined instanton partition functions as sums over 2d Young
diagrams. In §3, we study the $\operatorname{SO}(N)$ gauge theory with
(conjugate) spinor hypermultiplet and similarly express the unrefined
instanton partition functions as sums over 2d Young diagrams. In §4, we
consider the $\mathcal{N}=1^{*}$ $\operatorname{SO}(N)$ gauge theory, and in
§5, we investigate the $\operatorname{Sp}(N)$ gauge theory with antisymmetric
hypermultiplet. In these latter cases, the unrefined instanton partition
functions are represented as sums over 4d Young diagrams. In all of these
cases, we verify the identities of the instanton partition functions predicted
by Higgsing of fivebrane webs and representation theory.

##  2 $\operatorname{SU}(N)$ gauge group with (anti-)symmetric hypermultiplet

The instanton partition function can be obtained from the equivariant Chern
characters of the universal bundle $\mathcal{E}$ over the instanton moduli
space Nekrasov:2002qd . Let $p$ be a generic element of the equivariant torus
action on the universal bundle $\mathcal{E}$. The equivariant Chern characters
for (rank-two) symmetric and antisymmetric representations of
$\operatorname{SU}(N)$ can be obtained by

$\displaystyle\mathrm{Ch}_{p}^{\text{sym }}(\mathcal{E})$
$\displaystyle=\mathrm{Ch}_{p}\left(\mathrm{Sym}^{2}\mathcal{E}\right)=\frac{1}{2}\left[(\mathrm{Ch}_{p}(\mathcal{E}))^{2}+\mathrm{Ch}_{p^{2}}(\mathcal{E})\right],$
$\displaystyle\mathrm{Ch}_{p}^{\text{ant }}(\mathcal{E})$
$\displaystyle=\mathrm{Ch}_{p}\left(\wedge^{2}\mathcal{E}\right)=\frac{1}{2}\left[(\mathrm{Ch}_{p}(\mathcal{E}))^{2}-\mathrm{Ch}_{p^{2}}(\mathcal{E})\right]~{}.$

The equivariant index of the Dirac operator on the universal bundle takes the
following form

$\operatorname{Ind}_{q}=\sum_{\alpha}\epsilon_{\alpha}\mathrm{e}^{w_{a}}=\int_{\mathbb{C}^{2}}\operatorname{Ch}_{q}(\mathcal{E})\mathrm{Td}_{q}\left(\mathbb{C}^{2}\right)_{t}~{}.$

Then, the contributions to the integrands of the 5d instanton partition
function can be read off by mapping

$\sum_{\alpha}\epsilon_{\alpha}\mathrm{e}^{w_{\alpha}}\mapsto\prod_{\alpha}\operatorname{sh}^{\epsilon_{\alpha}}(w_{\alpha})~{}.$

We refer the reader to Nekrasov:2002qd ; Losev:2003py ; Shadchin:2005mx  for
the details. Note that we use the notation that

$\operatorname{sh}(x):=\mathrm{e}^{\frac{x}{2}}-\mathrm{e}^{-\frac{x}{2}}~{},\qquad\operatorname{ch}(x):=\mathrm{e}^{\frac{x}{2}}+\mathrm{e}^{-\frac{x}{2}}~{}.$
(1)

In this way, we can write down the contour integral expressions of the
instanton partition functions

$Z_{\operatorname{SU}(N),k,\kappa}^{\textrm{rep}}=\frac{1}{k!2^{k}}\oint_{\textrm{JK}}\prod_{I=1}^{k}\frac{d\phi_{I}}{2\pi
i}\cdot
e^{\kappa\sum_{I=1}^{k}\phi_{I}}\mathcal{I}_{\operatorname{SU}(N),k}^{\textrm{vec}}\mathcal{I}_{\operatorname{SU}(N),k}^{\textrm{rep}}$
(2)

where the integrands are given by

$\displaystyle\mathcal{I}_{\operatorname{SU}(N),k}^{\textrm{vec}}=$
$\displaystyle\frac{\prod_{I\neq
J}\operatorname{sh}(\phi_{I}-\phi_{J})\cdot\prod_{I,J}\operatorname{sh}(2\epsilon_{+}-\phi_{I}+\phi_{J})}{\prod_{I,J}\operatorname{sh}(\epsilon_{1,2}+\phi_{I}-\phi_{J})\prod_{I=1}^{k}\prod_{s=1}^{N}\operatorname{sh}(\epsilon_{+}\pm(\phi_{I}-a_{s}))}$
(3) $\displaystyle\mathcal{I}_{\operatorname{SU}(N),k}^{\textrm{fund}}(m)=$
$\displaystyle\prod_{I=1}^{k}\operatorname{sh}(\phi_{I}+m)$ (4)
$\displaystyle\mathcal{I}_{\operatorname{SU}(N),k}^{\textrm{sym}}(m)=$
$\displaystyle\prod_{I=1}^{k}\operatorname{sh}{(2\phi_{I}+m\pm\epsilon_{-})}\prod_{s=1}^{N}\operatorname{sh}{(\phi_{I}+a_{s}+m)}\prod_{I<J}^{k}\frac{\operatorname{sh}{(\phi_{I}+\phi_{J}+m\pm\epsilon_{-})}}{\operatorname{sh}{(-\epsilon_{+}\pm(\phi_{I}+\phi_{J}+m))}}$
(5) $\displaystyle\mathcal{I}_{\operatorname{SU}(N),k}^{\textrm{anti}}(m)=$
$\displaystyle\prod_{I=1}^{k}\frac{\prod_{s=1}^{N}\operatorname{sh}{(\phi_{I}+a_{s}+m)}}{\operatorname{sh}{(-\epsilon_{+}\pm(2\phi_{I}+m))}}\prod_{I<J}^{k}\frac{\operatorname{sh}{(\phi_{I}+\phi_{J}+m\pm\epsilon_{-})}}{\operatorname{sh}{(-\epsilon_{+}\pm(\phi_{I}+\phi_{J}+m))}}~{}.$
(6)

Note that $\kappa$ is the 5d Chern-Simons level. For a gauge group
$\operatorname{SU}(2N+1)$ of even rank, the level $\kappa$ needs to be a half-
odd-integer due to the parity anomaly associated with the (anti-)symmetric
matter. The residue computation is carried out by the Jeffrey-Kirwan
prescription jeffrey1995localization ; Benini:2013xpa . With (anti-)symmetric
hypermultiplet, it is difficult to classify poles fully and obtain closed-form
expressions at the refined level. Nonetheless, one can check that the
following isomorphisms of representations of gauge Lie algebras111We perform
the following change of variables on the Coulomb branch parameters of the
instanton partition function from the orthogonal basis of
$\operatorname{SU}(4)$ to that of $\operatorname{SO}(6)$:
$A_{1}\to(A_{1}A_{2}A_{3})^{\frac{1}{2}},\quad
A_{2}\to(A_{1}A_{2}^{-1}A_{3}^{-1})^{\frac{1}{2}},\quad
A_{3}\to(A_{1}^{-1}A_{2}A_{3}^{-1})^{\frac{1}{2}},\quad
A_{4}\to(A_{1}^{-1}A_{2}^{-1}A_{3})^{\frac{1}{2}}~{}.$

$\displaystyle(\mathfrak{su}(4),{\mathbf{AS}})\cong(\mathfrak{so}(6),\mathbf{V})\quad\longrightarrow\quad$
$\displaystyle
Z_{\operatorname{SU}(4),k,\kappa=0}^{\textrm{anti}}=Z_{\operatorname{SO}(6),k}^{\textrm{vect}}~{},$
(7)
$\displaystyle(\mathfrak{su}(3),{\mathbf{AS}})\cong(\mathfrak{su}(3),\mathbf{F})\quad\longrightarrow\quad$
$\displaystyle
Z_{\operatorname{SU}(3),k,\kappa=\frac{1}{2}}^{\textrm{anti}}(m)=Z_{\operatorname{SU}(3),k,\kappa=\frac{1}{2}}^{\textrm{fund}}(m\to-m)~{}.$

We verify these identities up to 4-instanton by performing JK residues
explicitly at the refined level.

In Bergman:2015dpa , 5d $\operatorname{SU}(N)$ gauge theory with one
(anti-)symmetric hypermultiplet is constructed from fivebranes with an
O7-plane in Type IIB theory. (See Figure 1.)

Figure 1: (Left-panel) $\operatorname{SU}(4)$ gauge theory with a symmetric
hypermultiplet.
(Right-panel) $\operatorname{SU}(4)$ gauge theory with an anti-symmetric
hypermultiplet.

Once we align D5-branes in the brane configuration with an O7+-plane
(symmetric hypermultiplet), we can higgs the middle NS5-brane, yielding the
brane configuration for pure $\operatorname{SO}(N)$ gauge theory. This
operation can be seen at the level of the instanton partition functions as

$Z_{\operatorname{SU}(N),2k,\kappa=\frac{1}{2}(N\bmod
2)}^{\textrm{sym}}(a_{N-i+1}=-a_{i},m=\epsilon_{+})=(-1)^{k(N+1)}Z_{\operatorname{SO}(N),k}^{\textrm{pure}}~{}.$
(8)

Note that $2k$-instanton on the left-hand side coincides with $k$-instanton on
the right-hand side. Also, when $N$ is odd, we set $a_{\frac{N+1}{2}}=0$ in
the left-hand side.

Similarly, we can perform the same operation on the brane configuration with
an O7--plane (anti-symmetric hypermultiplet), which results in the brane
configuration for pure $\operatorname{Sp}(N)$ gauge theory. This operation
amounts to the identities of the instanton partition functions

$\displaystyle Z_{\operatorname{SU}(2N),k,\kappa=N\bmod
2}^{\textrm{anti}}(a_{N-i+1}=$ $\displaystyle-
a_{i},m=\epsilon_{+})=(-1)^{k(N+1)+\lceil\frac{k}{2}\rceil}Z_{\operatorname{Sp}(N),k,\theta=0}^{\textrm{pure}}~{},$
(9) $\displaystyle Z_{\operatorname{SU}(2N),k,\kappa=N+1\bmod
2}^{\textrm{anti}}(a_{N-i+1}=$ $\displaystyle-
a_{i},m=\epsilon_{+})=(-1)^{k(N+1)+\lceil\frac{k}{2}\rceil}Z_{\operatorname{Sp}(N),k,\theta=\pi}^{\textrm{pure}}~{},$

where the discrete $\theta$-angle of the $\operatorname{Sp}(N)$ theory is
determined by the Chern-Simons level and the rank of the gauge group.

Although pole classification is hard at the refined level, non-trivial JK-
poles are classified by $N$-tuples
$\vec{\lambda}=(\lambda^{(1)},\ldots,\lambda^{(N)})$ of 2d Young diagrams with
the total number of boxes $\sum_{s=1}^{N}|\lambda^{(s)}|=k$ in the _unrefined_
limit $\epsilon_{1}=-\epsilon_{2}=\hbar$. We specify a pole location
associated with a content $x=(x_{1},x_{2})\in\lambda^{(s)}$ as

$\phi_{s}(x)=a_{s}+(x_{1}-x_{2})\hbar\ .$ (10)

By writing the Nekrasov factor Nekrasov:2002qd

$N_{s,t}(x):=a_{s}-a_{t}-\hbar(a_{\lambda^{(s)}}(x)+l_{\lambda^{(t)}}(x)+1)\
,$ (11)

the residue sum for a symmetric hypermultiplet is expressed as

$\displaystyle Z_{\operatorname{SU}(N),k,\kappa}^{\textrm{sym}}$ (12)
$\displaystyle=$
$\displaystyle(-1)^{kN+\lceil\frac{k}{2}\rceil}\sum_{\vec{\lambda}}\prod_{s=1}^{N}\prod_{x\in\lambda^{(s)}}e^{\kappa\phi_{s}(x)}\frac{\operatorname{sh}(2\phi_{s}(x)+m\pm\hbar)\cdot\prod_{t=1}^{N}\operatorname{sh}(\phi_{s}(x)+a_{t}+m)}{\prod_{t=1}^{N}\operatorname{sh}^{2}(N_{s,t}(x))}$
$\displaystyle\times\prod_{s\leq
t}^{N}\prod_{\begin{subarray}{c}x\in\lambda^{(s)},y\in\lambda^{(t)}\\\
x<y\end{subarray}}\frac{\operatorname{sh}(\phi_{s}(x)+\phi_{t}(y)+m\pm\hbar)}{\operatorname{sh}^{2}(\phi_{s}(x)+\phi_{t}(y)+m)}$

Here we define a total ordering on the boxes of 2d Young diagrams:

$\lambda^{(s)}\ni x<y\in\lambda^{(t)}\quad\text{ if
}\left\\{\begin{array}[]{l}s<t\\\ s=t,x_{1}<y_{1}\\\
s=t,x_{1}=y_{1},x_{2}<y_{2}\end{array}\right.~{}.$ (13)

Once we take the specialization (8) at the unrefined level, we obtain an
expression as a Young diagram sum for the pure $\operatorname{SO}(N)$
instanton partition function, which is different from the one in
Nawata:2021dlk .

For an anti-symmetric matter, the JK-poles are classified by $(N+4)$-tuples of
2d Young diagrams with the total number of boxes
$\sum_{s=1}^{N+4}|\lambda^{(s)}|=k$ where the four additional effective
Coulomb branch parameters are

$a_{N+j}=-\frac{m}{2}(+\pi i)~{},\ \frac{\hbar-m}{2}(+\pi i)~{},$ (14)

for $j=1,2,3,4$. Then, the residue sums are expressed as

$\displaystyle
Z_{\operatorname{SU}(N),k,\kappa}^{\textrm{anti}}(A_{N+1}=e^{\frac{m}{2}},A_{N+2}=-e^{\frac{m}{2}},A_{N+3}=e^{-\frac{\hbar-m}{2}},A_{N+4}=-e^{-\frac{\hbar-m}{2}})$
(15) $\displaystyle=$
$\displaystyle(-1)^{kN+\lceil\frac{k}{2}\rceil}\sum_{\vec{\lambda}}C^{\textrm{anti}}_{\vec{\lambda},\vec{A}}\prod_{s=1}^{N+4}\prod_{x\in\lambda^{(s)}}e^{\kappa\phi_{s}(x)}\frac{\operatorname{sh}^{2}(2\phi_{s}(x)+m-\hbar)\cdot\prod_{t=1}^{N}\operatorname{sh}(\phi_{s}(x)+a_{t}+m)}{\prod_{t=1}^{N+4}\operatorname{sh}^{2}(N_{s,t}(x))}$
(16) $\displaystyle\qquad\qquad\qquad\times\prod_{s\leq
t}^{N+4}\prod_{\begin{subarray}{c}x\in\lambda^{(s)},y\in\lambda^{(t)}\\\
x<y\end{subarray}}\frac{\operatorname{sh}(\phi_{s}(x)+\phi_{t}(y)+m\pm\hbar)}{\operatorname{sh}^{2}(\phi_{s}(x)+\phi_{t}(y)+m)}$
(17)

where $\phi_{s}(x)$ and $N_{s,t}(x)$ are given in (10) and (11). Here, the
non-trivial multiplicity constants

$C^{\textrm{anti}}_{\vec{\lambda},\vec{A}}=\prod_{s=N+1}^{N+4}C^{\textrm{anti}}_{\lambda^{(s)},A_{s}}~{}.$
(18)

are involved due to higher-order poles coming from the additional effective
Coulomb branch parameters (14). These multiplicity constants are in principle
determined by the orders of poles and the determinant of JK vectors that
define a cone. Here we conjecture these constants as follows:
$C^{\textrm{anti}}_{\lambda^{(s)}=\emptyset,A_{s}}=1$, and

$\begin{split}{\raisebox{-42.67912pt}{\includegraphics[width=142.26378pt]{pictures/C-anti1}}}\quad\leadsto\quad
C^{\textrm{anti}}_{\lambda^{(s)},A_{s}=\pm e^{-\frac{m}{2}}}=(-2)^{j}\\\
{\raisebox{-42.67912pt}{\includegraphics[width=142.26378pt]{pictures/C-anti2}}}\quad\leadsto\quad
C^{\textrm{anti}}_{\lambda^{(s)},A_{s}=\pm
e^{-\frac{\hbar-m}{2}}}=(-2)^{j}\end{split}$ (19)

Once we take the specialization (9) at the unrefined level, we obtain an
expression as Young diagram sums for the pure $\operatorname{Sp}(N)$ instanton
partition function (both $\theta=0$ and $\theta=\pi$), which takes a different
presentation from the one in Nawata:2021dlk .

Since the relation to the instanton partition functions for gauge groups of
$BCD$ comes from the fivebrane webs with an O7-plane here, the difference in
the expressions of the partition functions from Nawata:2021dlk  may be due to
the distinction between O7-planes and O5-planes in the topological vertex
formalism. Further investigation from this perspective could be worthwhile.

The relationship between the instanton partition functions for the symmetric
and antisymmetric hypermultiplet, as viewed from the perspective of fivebrane
systems with an O-plane, will be explored in the upcoming work Chengdu . This
will also help to deepen our understanding of the additional Coulomb branch
parameters (14) and the multiplicity coefficients (19).

##  3 $\operatorname{SO}(n)$ gauge group with spinor hypermultiplet

A Type IIB brane configuration for a hypermultiplet in the spinor
representation of a 5d SO($n$) gauge theory is given in Zafrir:2015ftn  by
using O5-planes. To introduce two hypermultiplets in the spinor
representation, a “leg” fivebrane needs to be introduced on the right side.
For instance, we draw the fivebrane diagrams for SO(7) gauge group with one
spinor, and SO(8) gauge theory with spinor and conjugate spinor in Figure 2.
As the figures show, the diagram for a hypermultiplet in the (conjugate)
spinor representation can be intuitively interpreted as the “Sp(0)” gauge
theory. In other words, roughly speaking, the theory can be regarded as an
“SO($n$)-Sp(0) quiver” gauge theory.

Figure 2: (Left-panel) $\operatorname{SO}(7)$ gauge theory with a spinor
matter.
(Right-panel) $\operatorname{SO}(8)$ gauge theory with spinor and conjugate
spinor matters.

Instantons are created by D1-branes that are suspended by fivebranes in the
middle. On the other hand, hypermultiplet particles in the spinor
representation are created by D1’-branes that are suspended by “leg”
fivebranes. As illustrated in Figure 3, for SO($n$) gauge theory with $n\leq
6$, the “leg” fivebranes do not intersect. For $n=7,8$, the two “leg”
fivebranes are parallel to each other. In fact, from this viewpoint of quantum
mechanics on D1-D1’-branes, the SO($n$) instanton moduli spaces with spinor
matter are described using the ADHM approach in Kim:2018gjo . The results are
reviewed in Appendix B.

When $k$ D1-branes create $k$ instantons and $j$ D1’-branes create $j$
hypermultiplet particles, we can view the system as
$\operatorname{Sp}(k)$-$\mathrm{O}(j)$ quiver gauge theory from the
perspective of quantum mechanics on the D1-D1’-branes. (See Figures 6 and 7.)
The gauge group O($j$) has two connected components, $\mathrm{O}(j)_{\pm}$,
which leads to generally two contributions,
$Z^{\operatorname{SO}(n)}_{k,j,\pm}$, to the partition function for a
(conjugate) spinor matter. These contributions can be obtained by performing
JK residue integrals, as detailed in Appendix B. However, for
$n=\textrm{odd}$, $Z^{\operatorname{SO}(n)}_{k,\textrm{even},-}$ and
$Z^{\operatorname{SO}(n)}_{k,\textrm{odd},+}$ vanish due to fermionic zero
modes. The full partition function for a spinor hypermultiplet can still be
written as

$Z_{\textrm{full},\operatorname{SO}(n)}^{\textrm{spinor}}=\sum_{k,j=0}^{\infty}\mathfrak{q}^{k}e^{-jm}\frac{Z^{\operatorname{SO}(n)}_{k,j,+}+Z^{\operatorname{SO}(n)}_{k,j,-}}{2}$
(20)

where $m$ is the mass of the hypermultiplet. On the other hand, for a
conjugate spinor hypermultiplet, the full partition function is

$Z_{\textrm{full},\operatorname{SO}(n)}^{\textrm{conj}}=\sum_{k,j=0}^{\infty}\mathfrak{q}^{k}e^{-jm}(-1)^{j}\frac{Z^{\operatorname{SO}(n)}_{k,j,+}+(-1)^{\mathrm{sign}(j)}Z^{\operatorname{SO}(n)}_{k,j,-}}{2}$
(21)

where $\mathrm{sign}(j)=0$ for $j=0$, and $\mathrm{sign}(j)=1$ for $j>0$. For
$n<9$, the full partition function $Z_{\textrm{full}}$ contains not only
instanton contributions, but also spurious contributions.

Figure 3: D1’-branes (blue) create hypermultiplet particles in the spinor
representation.

As analyzed in Kim:2018gjo , to extract genuine instanton contributions, we
must remove spurious contributions. Below, we focus only on the spinor
hypermultiplet, but the story is parallel for the conjugate spinor (just
insert appropriate signs as in (21)). For $n\leq 6$, the partition function at
$k=0$ is given by

$\displaystyle\sum_{j=0}^{\infty}e^{-jm}\frac{Z^{\operatorname{SO}(n)}_{0,j,+}+Z^{\operatorname{SO}(n)}_{0,j,-}}{2}=Z_{\textrm{pert}}\equiv\textrm{PE}\left[e^{-m}\frac{Z^{\operatorname{SO}(n)}_{0,1,+}+Z^{\operatorname{SO}(n)}_{0,1,-}}{2}\right]$
(22)

where PE is the plethystic exponent (77). Consequently, the full partition
function factorizes as

$\displaystyle
Z_{\textrm{full},\operatorname{SO}(n)}^{\textrm{spinor}}=\widehat{Z}_{\operatorname{SO}(n)}^{\textrm{spinor}}(\mathfrak{q},\epsilon_{1,2},m)Z_{\textrm{pert}}(\epsilon_{1,2},m)$
(23)

where we extract the genuine instanton contribution

$\widehat{Z}_{\operatorname{SO}(n)}^{\textrm{spinor}}(\mathfrak{q},\epsilon_{1,2},m)=\sum_{k=0}^{\infty}\mathfrak{q}^{k}\widehat{Z}_{\operatorname{SO}(n),k}^{\textrm{spinor}}(\epsilon_{1,2},m)~{},\quad\widehat{Z}_{\operatorname{SO}(n),k}^{\textrm{spinor}}(\epsilon_{1,2},m)=\sum_{j=0}^{\infty}e^{-jm}\widehat{Z}_{\operatorname{SO}(n),k,j}^{\textrm{spinor}}~{}.$
(24)

For the cases of $n\leq 6$, the contributions from hypermultiplet particles
more than instanton numbers vanish as $\widehat{Z}_{k,j}=0$ for $j>k$ for both
the spinor and the conjugate spinor.

For $n=7,8$, the partition function at $k=0$ is given by

$\displaystyle\sum_{j=0}^{\infty}e^{-jm}\frac{Z^{\operatorname{SO}(n)}_{0,j,+}+Z^{\operatorname{SO}(n)}_{0,j,-}}{2}=$
$\displaystyle Z_{\textrm{pert}}Z_{\textrm{extra}}$ (25) $\displaystyle=$
$\displaystyle\textrm{PE}\left[e^{-m}\frac{Z^{\operatorname{SO}(n)}_{0,1,+}+Z^{\operatorname{SO}(n)}_{0,1,-}}{2}\right]\mathrm{PE}\left[-\frac{1}{2}e^{-2m}\frac{\operatorname{ch}2\epsilon_{+}}{\operatorname{sh}\epsilon_{1,2}}\right]~{},$

where $Z_{\textrm{extra}}$ is the contribution from D1’-branes running away
along the two parallel fivebranes. (See the middle of Figure 3.) Thus, the
full partition function factorizes as

$Z_{\textrm{full},\operatorname{SO}(n)}^{\textrm{spinor}}=\widehat{Z}_{\operatorname{SO}(n)}^{\textrm{spinor}}(\mathfrak{q},\epsilon_{1,2},m)Z_{\textrm{pert}}(\epsilon_{1,2},m)Z_{\text{extra
}}(\epsilon_{1,2},m)~{},$ (26)

where
$\widehat{Z}_{\operatorname{SO}(n)}^{\textrm{spinor}}(\mathfrak{q},\epsilon_{1,2},m)$
takes the form (24). In the cases of $n=7,8$, for both the spinor and the
conjugate spinor, the instanton partition functions are subject to
$\widehat{Z}_{k,2k-n}=\widehat{Z}_{k,n}$ so that $\widehat{Z}_{k,j}=0$ for
$j>2k$.

For $n\geq 9$, we cannot express the contribution purely from the
hypermultiplet in the form of a plethystic exponent, unlike in previous
examples (22) and (25). This is because the leg fivebranes meet at a certain
point in this case. Such a fivebrane configuration has been studied in
Hayashi:2019yxj . However, we believe that the full partition functions (20)
and (21) can still be applied to these fivebrane web configurations even for
$n\geq 9$, although we leave a more detailed investigation of this issue for
future work.

As we learned in Nawata:2021dlk , the instanton partition function of pure
Yang-Mills theory for gauge groups of $BCD$ type can be written as sums over
2d Young diagrams at the unrefined level. Since the SO($n$) theory with a
spinor matter can be roughly regarded as an “SO($n$)-Sp(0) quiver” gauge
theory, we expect that its instanton partition function can also be expressed
as a sum over 2d Young diagrams in the unrefined limit. In the following, we
will demonstrate that this is indeed the case. Moreover, we can generalize the
method in this section to the SO-Sp linear quiver gauge theories.

###  3.1 $\operatorname{SO}(2N+1)$ with one spinor

The integral expressions for the partition functions are given in Appendix
B.1. For a gauge group of $B$ type, the contributions from
$\mathrm{O}(2\ell)_{-}$ and $\mathrm{O}(2\ell+1)_{+}$ sectors vanish due to
fermionic zero modes. As a result, we have a single contribution,
$Z^{\operatorname{SO}(2N+1)}_{k,j}$, for each hypermultiplet particle number
$j$. This is another way to see that there is no distinction between the
spinor representation and its conjugate for $\operatorname{SO}(n)$ with odd
$n$.

As in Nawata:2021dlk  and we also discussed below (15), there are four
effective Coulomb branch parameters so that even the “Sp(0)” gauge theory
contributes to the unrefined partition function. Hence, for
$Z^{\operatorname{SO}(2N+1)}_{k,2\ell(+1)}$, the non-trivial JK poles in the
unrefined limit are classified $(N+4)$-tuples $\vec{\lambda}$ of 2d Young
diagrams where the numbers of boxes are subject to

$\sum_{s=1}^{N}|\lambda^{(s)}|=k~{},\qquad\sum_{s=N+1}^{N+4}|\lambda^{(s)}|=\ell~{}.$
(27)

The JK residue sums are expressed as

$\displaystyle
Z^{\operatorname{SO}(2N+1)}_{k,2\ell}(A_{N+1}=q^{\frac{1}{2}},A_{N+2}=-q^{\frac{1}{2}},A_{N+3}=1,A_{N+4}=-1;q)=(-1)^{k+\ell}$
(28)
$\displaystyle\sum_{\vec{\lambda}}C^{\operatorname{Sp}}_{\vec{\lambda},\vec{A}}\prod_{s=1}^{N}\prod_{x\in\lambda^{(s)}}\frac{\operatorname{sh}^{4}2\phi_{s}(x)}{\operatorname{sh}^{2}(\phi_{s}(x))\prod\limits_{t=1}^{N}\operatorname{sh}^{2}(N_{s,t}(x))\,\operatorname{sh}^{2}(\phi_{s}(x)+a_{t})}\prod_{s\leq
t}^{N}\prod_{\begin{subarray}{c}x\in\lambda^{(s)}\\\ y\in\lambda^{(t)}\\\
x<y\end{subarray}}\frac{\operatorname{sh}^{4}(\phi_{s}(x)+\phi_{t}(y))}{\operatorname{sh}^{2}(\phi_{s}(x)+\phi_{t}(y)\pm\hbar)}$
(29)
$\displaystyle\prod_{s=N+1}^{N+4}\prod_{x\in\lambda^{(s)}}\frac{\operatorname{sh}^{2}(\phi_{s}(x))\,\operatorname{sh}^{4}(2\phi_{s}(x))\prod\limits_{t=1}^{N}\operatorname{sh}(\pm\phi_{s}(x)+a_{t})}{\prod\limits_{t=N+1}^{N+4}\operatorname{sh}^{2}(N_{s,t}(x))\,\operatorname{sh}^{2}(\phi_{s}(x)+a_{t})}\prod_{N+1\leq
s\leq t}^{N+4}\prod_{\begin{subarray}{c}x\in\lambda^{(s)}\\\
y\in\lambda^{(t)}\\\
x<y\end{subarray}}\frac{\operatorname{sh}^{4}(\phi_{s}(x)+\phi_{t}(y))}{\operatorname{sh}^{2}(\phi_{s}(x)+\phi_{t}(y)\pm\hbar)}$
(30)
$\displaystyle\prod_{s=1}^{N}\prod_{x\in\lambda^{(s)}}\prod_{t=N+1}^{N+4}\prod_{y\in\lambda^{(t)}}\frac{\operatorname{sh}(\hbar\pm\phi_{s}(x)\pm\phi_{t}(y))}{\operatorname{sh}(\pm\phi_{s}(x)\pm\phi_{t}(y))}$
(31) $\displaystyle
Z^{\operatorname{SO}(2N+1)}_{k,2\ell+1}(A_{N+1}=q^{\frac{1}{2}},A_{N+2}=-q^{\frac{1}{2}},A_{N+3}=1,A_{N+4}=-q;q)=(-1)^{k+\ell+1}\frac{\prod_{t=1}^{3}\operatorname{ch}a_{t}}{\operatorname{sh}^{2}\hbar}$
(32)
$\displaystyle\sum_{\vec{\lambda}}C^{\operatorname{Sp}}_{\vec{\lambda},\vec{A}}\prod_{s=1}^{N}\prod_{x\in\lambda^{(s)}}\frac{\operatorname{sh}^{4}2\phi_{s}(x)}{\operatorname{sh}^{2}(\phi_{s}(x))\prod\limits_{t=1}^{N}\operatorname{sh}^{2}(N_{s,t}(x))\,\operatorname{sh}^{2}(\phi_{s}(x)+a_{t})}\prod_{s\leq
t}^{N}\prod_{\begin{subarray}{c}x\in\lambda^{(s)}\\\ y\in\lambda^{(t)}\\\
x<y\end{subarray}}\frac{\operatorname{sh}^{4}(\phi_{s}(x)+\phi_{t}(y))}{\operatorname{sh}^{2}(\phi_{s}(x)+\phi_{t}(y)\pm\hbar)}$
(33)
$\displaystyle\prod_{s=N+1}^{N+4}\prod_{x\in\lambda^{(s)}}\frac{\operatorname{sh}^{2}(\phi_{s}(x))\,\operatorname{sh}^{4}(2\phi_{s}(x))\prod\limits_{t=1}^{N}\operatorname{sh}(\pm\phi_{s}(x)+a_{t})}{\prod\limits_{t=N+1}^{N+4}\operatorname{sh}^{2}(N_{s,t}(x))\,\operatorname{sh}^{2}(\phi_{s}(x)+a_{t})}\prod_{N+1\leq
s\leq t}^{N+4}\prod_{\begin{subarray}{c}x\in\lambda^{(s)}\\\
y\in\lambda^{(t)}\\\
x<y\end{subarray}}\frac{\operatorname{sh}^{4}(\phi_{s}(x)+\phi_{t}(y))}{\operatorname{sh}^{2}(\phi_{s}(x)+\phi_{t}(y)\pm\hbar)}$
(34)
$\displaystyle\prod_{s=1}^{N}\prod_{x\in\lambda^{(s)}}\prod_{t=N+1}^{N+4}\prod_{y\in\lambda^{(t)}}\frac{\operatorname{sh}(\hbar\pm\phi_{s}(x)\pm\phi_{t}(y))}{\operatorname{sh}(\pm\phi_{s}(x)\pm\phi_{t}(y))}\frac{\operatorname{ch}(\hbar\pm\phi_{s}(x))}{\operatorname{ch}(\pm\phi_{s}(x))}$
(35)

where $\phi_{s}(x)$ and $N_{s,t}(x)$ are given in (10) and (11), and
$C^{\operatorname{Sp}}_{\vec{\lambda},\vec{A}}$ is a constant

$C^{\operatorname{Sp}}_{\vec{\lambda},\vec{A}}=\prod_{s=N+1}^{N+4}C^{\operatorname{Sp}}_{\lambda^{(s)},A_{s}}~{}.$
(36)

These coefficients are the same as those that appear in the pure
$\operatorname{Sp}(N)$ instanton partition function in Nawata:2021dlk :
$C_{\lambda^{(s)}=\emptyset,A_{s}}=1$, and

$\begin{split}C^{\operatorname{Sp}}_{\lambda^{(s)},A_{s}=\pm 1,\pm
q^{\frac{1}{2}}}&=\frac{2^{2j-1}}{\binom{2j-1}{j-1}}\qquad\textrm{ where $j$
is the number of rows with $\lambda_{i}^{(s)}\geq i$}~{},\\\
C^{\operatorname{Sp}}_{\lambda^{(s)},A_{s}=\pm
q}&=\frac{2^{2j}}{\binom{2j+1}{j}}\qquad\textrm{ where $j$ is the number of
rows with $\lambda_{i}^{(s)}\geq i+1$}~{}.\end{split}$ (37)

(See also (19) for the illustration by Young diagrams.) Note that these
multiplicity coefficients are still conjectural.

As a simple check, we can compare the result with (Kim:2018gjo, , (2.29)),
which are obtained from SU(4) instanton partition function by ingenious
representation theoretic methods. We check the match upto 5-instanton.

### New formula of $G_{2}$ instanton

Higssing of $\operatorname{SO}(7)+1\textbf{S}$ leads to pure $G_{2}$ gauge
theory Kim:2018gjo ; Hayashi:2018bkd ; Nawata:2021dlk . Therefore, if we set
$m=0$, $A_{3}=A_{1}A_{2}$ in the instanton partition function of
$\operatorname{SO}(7)+1\textbf{S}$, we obtain a new formula of $G_{2}$
instanton partition function. We check it against Kim:2018gjo ;
Hayashi:2018bkd ; Nawata:2021dlk  upto 4-instanton.

###  3.2 $\operatorname{SO}(2N)$ with one (conjugate) spinor

The integral expressions of the partition functions are written in Appendix
B.2. For a gauge group of $D$ type, the two sectors from $\mathrm{O}(j)_{\pm}$
make non-trivial contributions. This is another way to see that the spinor
representation and its conjugate are not isomorphic for $\operatorname{SO}(n)$
for general even $n$.

Hence, the non-trivial JK poles in the unrefined limit are classified
$(N+4)$-tuples $\vec{\lambda}$ of 2d Young diagrams where the numbers of boxes
for $Z^{\operatorname{SO}(2N)}_{k,2\ell,+}$ and
$Z^{\operatorname{SO}(2N)}_{k,2\ell+1,\pm}$ are subject to

$\sum_{s=1}^{N}|\lambda^{(s)}|=k~{},\qquad\sum_{s=N+1}^{N+4}|\lambda^{(s)}|=\ell~{},$
(38)

whereas those for $Z^{\operatorname{SO}(2N)}_{k,2\ell,-}$ is required to be

$\sum_{s=1}^{N}|\lambda^{(s)}|=k~{},\qquad\sum_{s=N+1}^{N+4}|\lambda^{(s)}|=\ell-1~{}.$
(39)

Then, using the multiplicity coefficients in (37), the JK residue sum of each
sector is expressed as

$\begin{split}&Z^{\operatorname{SO}(2N)}_{k,2\ell,+}(A_{N+1}=q^{\frac{1}{2}},A_{N+2}=-q^{\frac{1}{2}},A_{N+3}=1,A_{N+4}=-1;q)=2^{\text{sign}(\ell)}\\\
&\sum_{\vec{\lambda}}C^{\operatorname{Sp}}_{\vec{\lambda},\vec{A}}\prod_{s=1}^{N}\prod_{x\in\lambda^{(s)}}\frac{\operatorname{sh}^{4}(2\phi_{s}(x))}{\prod\limits_{t=1}^{N}\operatorname{sh}^{2}(N_{s,t}(x))\,\operatorname{sh}^{2}(\phi_{s}(x)+a_{t})}\prod_{s\leq
t}^{N}\prod_{\begin{subarray}{c}x\in\lambda^{(s)}\\\ y\in\lambda^{(t)}\\\
x<y\end{subarray}}\frac{\operatorname{sh}^{4}(\phi_{s}(x)+\phi_{t}(y))}{\operatorname{sh}^{2}(\phi_{s}(x)+\phi_{t}(y)\pm\hbar)}\\\
&\prod_{s=N+1}^{N+4}\prod_{x\in\lambda^{(s)}}\frac{\operatorname{sh}^{4}(2\phi_{s}(x))\prod\limits_{t=1}^{N}\operatorname{sh}(\pm\phi_{s}(x)+a_{t})}{\prod\limits_{t=N+1}^{N+4}\operatorname{sh}^{2}(N_{s,t}(x))\,\operatorname{sh}^{2}(\phi_{s}(x)+a_{t})}\prod_{N+1\leq
s\leq t}^{N+4}\prod_{\begin{subarray}{c}x\in\lambda^{(s)}\\\
y\in\lambda^{(t)}\\\
x<y\end{subarray}}\frac{\operatorname{sh}^{4}(\phi_{s}(x)+\phi_{t}(y))}{\operatorname{sh}^{2}(\phi_{s}(x)+\phi_{t}(y)\pm\hbar)}\\\
&\prod_{s=1}^{N}\prod_{x\in\lambda^{(s)}}\prod_{t=N+1}^{N+4}\prod_{y\in\lambda^{(t)}}\frac{\operatorname{sh}(\hbar\pm\phi_{s}(x)\pm\phi_{t}(y))}{\operatorname{sh}(\pm\phi_{s}(x)\pm\phi_{t}(y))}\end{split}$
(40)
$\begin{split}&Z^{\operatorname{SO}(2N)}_{k,2\ell,-}(A_{N+1}=q^{\frac{1}{2}},A_{N+2}=-q^{\frac{1}{2}},A_{N+3}=q,A_{N+4}=-q;q)=\Bigg{[}(-1)^{k}2\frac{\prod_{s=1}^{N}\operatorname{sh}(2a_{s})}{\operatorname{sh}^{2}(\hbar)\operatorname{sh}^{2}(2\hbar)}\Bigg{]}^{\text{sign}(\ell)}\\\
&\sum_{\vec{\lambda}}C^{\operatorname{Sp}}_{\vec{\lambda},\vec{A}}\prod_{s=1}^{N}\prod_{x\in\lambda^{(s)}}\frac{\operatorname{sh}^{4}(2\phi_{s}(x))}{\prod\limits_{t=1}^{N}\operatorname{sh}^{2}(N_{s,t}(x))\,\operatorname{sh}^{2}(\phi_{s}(x)+a_{t})}\prod_{s\leq
t}^{N}\prod_{\begin{subarray}{c}x\in\lambda^{(s)}\\\ y\in\lambda^{(t)}\\\
x<y\end{subarray}}\frac{\operatorname{sh}^{4}(\phi_{s}(x)+\phi_{t}(y))}{\operatorname{sh}^{2}(\phi_{s}(x)+\phi_{t}(y)\pm\hbar)}\\\
&\Bigg{[}\prod_{s=N+1}^{N+4}\prod_{x\in\lambda^{(s)}}\frac{\operatorname{sh}^{4}(2\phi_{s}(x))\prod\limits_{t=1}^{N}\operatorname{sh}(\pm\phi_{s}(x)+a_{t})}{\prod\limits_{t=N+1}^{N+4}\operatorname{sh}^{2}(N_{s,t}(x))\,\operatorname{sh}^{2}(\phi_{s}(x)+a_{t})}\prod_{N+1\leq
s\leq t}^{N+4}\prod_{\begin{subarray}{c}x\in\lambda^{(s)}\\\
y\in\lambda^{(t)}\\\
x<y\end{subarray}}\frac{\operatorname{sh}^{4}(\phi_{s}(x)+\phi_{t}(y))}{\operatorname{sh}^{2}(\phi_{s}(x)+\phi_{t}(y)\pm\hbar)}\\\
&\prod_{s=1}^{N}\prod_{x\in\lambda^{(s)}}\frac{\operatorname{sh}(2\hbar\pm
2\phi_{s}(x))}{\operatorname{sh}^{2}(2\phi_{s}(x))}\prod_{t=N+1}^{N+4}\prod_{y\in\lambda^{(t)}}\frac{\operatorname{sh}(\hbar\pm\phi_{s}(x)\pm\phi_{t}(y))}{\operatorname{sh}(\pm\phi_{s}(x)\pm\phi_{t}(y))}\Bigg{]}^{\text{sign}(\ell)}\end{split}$
(41)
$\begin{split}&Z^{\operatorname{SO}(2N)}_{k,2\ell+1,+}(A_{N+1}=q^{\frac{1}{2}},A_{N+2}=-q^{\frac{1}{2}},A_{N+3}=q,A_{N+4}=-1;q)=(-1)^{k+1}\frac{\prod_{s=1}^{N}\operatorname{sh}(a_{s})}{\operatorname{sh}^{2}(\hbar)}\\\
&\sum_{\vec{\lambda}}C^{\operatorname{Sp}}_{\vec{\lambda},\vec{A}}\prod_{s=1}^{N}\prod_{x\in\lambda^{(s)}}\frac{\operatorname{sh}^{4}(2\phi_{s}(x))}{\prod\limits_{t=1}^{N}\operatorname{sh}^{2}(N_{s,t}(x))\,\operatorname{sh}^{2}(\phi_{s}(x)+a_{t})}\prod_{s\leq
t}^{N}\prod_{\begin{subarray}{c}x\in\lambda^{(s)}\\\ y\in\lambda^{(t)}\\\
x<y\end{subarray}}\frac{\operatorname{sh}^{4}(\phi_{s}(x)+\phi_{t}(y))}{\operatorname{sh}^{2}(\phi_{s}(x)+\phi_{t}(y)\pm\hbar)}\\\
&\prod_{s=N+1}^{N+4}\prod_{x\in\lambda^{(s)}}\frac{\operatorname{sh}^{4}(2\phi_{s}(x))\prod\limits_{t=1}^{N}\operatorname{sh}(\pm\phi_{s}(x)+a_{t})}{\prod\limits_{t=N+1}^{N+4}\operatorname{sh}^{2}(N_{s,t}(x))\,\operatorname{sh}^{2}(\phi_{s}(x)+a_{t})}\prod_{N+1\leq
s\leq t}^{N+4}\prod_{\begin{subarray}{c}x\in\lambda^{(s)}\\\
y\in\lambda^{(t)}\\\
x<y\end{subarray}}\frac{\operatorname{sh}^{4}(\phi_{s}(x)+\phi_{t}(y))}{\operatorname{sh}^{2}(\phi_{s}(x)+\phi_{t}(y)\pm\hbar)}\\\
&\prod_{s=1}^{N}\prod_{x\in\lambda^{(s)}}\frac{\operatorname{sh}(\hbar\pm\phi_{s}(x))}{\operatorname{sh}^{2}(\phi_{s}(x))}\prod_{t=N+1}^{N+4}\prod_{y\in\lambda^{(t)}}\frac{\operatorname{sh}(\hbar\pm\phi_{s}(x)\pm\phi_{t}(y))}{\operatorname{sh}(\pm\phi_{s}(x)\pm\phi_{t}(y))}\end{split}$
(42)
$\begin{split}&Z^{\operatorname{SO}(2N)}_{k,2\ell+1,-}(A_{N+1}=q^{\frac{1}{2}},A_{N+2}=-q^{\frac{1}{2}},A_{N+3}=1,A_{N+4}=-q;q)=-\frac{\prod_{s=1}^{N}\operatorname{ch}(a_{s})}{\operatorname{sh}^{2}(\hbar)}\\\
&\sum_{\vec{\lambda}}C^{\operatorname{Sp}}_{\vec{\lambda},\vec{A}}\prod_{s=1}^{N}\prod_{x\in\lambda^{(s)}}\frac{\operatorname{sh}^{4}(2\phi_{s}(x))}{\prod\limits_{t=1}^{N}\operatorname{sh}^{2}(N_{s,t}(x))\,\operatorname{sh}^{2}(\phi_{s}(x)+a_{t})}\prod_{s\leq
t}^{N}\prod_{\begin{subarray}{c}x\in\lambda^{(s)}\\\ y\in\lambda^{(t)}\\\
x<y\end{subarray}}\frac{\operatorname{sh}^{4}(\phi_{s}(x)+\phi_{t}(y))}{\operatorname{sh}^{2}(\phi_{s}(x)+\phi_{t}(y)\pm\hbar)}\\\
&\prod_{s=N+1}^{N+4}\prod_{x\in\lambda^{(s)}}\frac{\operatorname{sh}^{4}(2\phi_{s}(x))\prod\limits_{t=1}^{N}\operatorname{sh}(\pm\phi_{s}(x)+a_{t})}{\prod\limits_{t=N+1}^{N+4}\operatorname{sh}^{2}(N_{s,t}(x))\,\operatorname{sh}^{2}(\phi_{s}(x)+a_{t})}\prod_{N+1\leq
s\leq t}^{N+4}\prod_{\begin{subarray}{c}x\in\lambda^{(s)}\\\
y\in\lambda^{(t)}\\\
x<y\end{subarray}}\frac{\operatorname{sh}^{4}(\phi_{s}(x)+\phi_{t}(y))}{\operatorname{sh}^{2}(\phi_{s}(x)+\phi_{t}(y)\pm\hbar)}\\\
&\prod_{s=1}^{N}\prod_{x\in\lambda^{(s)}}\frac{\operatorname{ch}(\hbar\pm\phi_{s}(x))}{\operatorname{ch}^{2}(\phi_{s}(x))}\prod_{t=N+1}^{N+4}\prod_{y\in\lambda^{(t)}}\frac{\operatorname{sh}(\hbar\pm\phi_{s}(x)\pm\phi_{t}(y))}{\operatorname{sh}(\pm\phi_{s}(x)\pm\phi_{t}(y))}\end{split}$
(43)

###  3.3 Isomorphisms of representations

In the cases of $\operatorname{SO}(4)$, $\operatorname{SO}(5)$ and
$\operatorname{SO}(6)$, the (conjugate) spinor representation is isomorphic to
the fundamental representation of
$\operatorname{SU}(2)\times\operatorname{SU}(2)$, $\operatorname{Sp}(2)$ and
$\operatorname{SU}(4)$, respectively. This can be seen at the level of the
unrefined instanton partition functions as follows, and we check the following
identities up to 5-instanton.

$\bullet\quad(\mathfrak{so}(4),\mathbf{S})\cong(\mathfrak{su}(2)\oplus\mathfrak{su}(2),\mathbf{F}\oplus\emptyset)$

$\widehat{Z}_{\operatorname{SO}(4)}^{\textrm{spinor}}=Z_{\operatorname{SU}(2),\kappa=\frac{1}{2}}^{\textrm{fund}}(A\to\sqrt{{A_{1}}{A_{2}}},\mathfrak{q}\to-\mathfrak{q}M^{-1/2})Z_{\operatorname{SU}(2),\kappa=0}^{\textrm{pure}}(A\to\sqrt{\frac{A_{1}}{A_{2}}})Z_{\operatorname{U}(1)}~{},$
(44)

$\bullet\quad(\mathfrak{so}(4),\mathbf{C})\cong(\mathfrak{su}(2)\oplus\mathfrak{su}(2),\emptyset\oplus\mathbf{F})$

$\widehat{Z}_{\operatorname{SO}(4)}^{\textrm{conj}}=Z_{\operatorname{SU}(2),\kappa=0}^{\textrm{pure}}(A\to\sqrt{{A_{1}}{A_{2}}})Z_{\operatorname{SU}(2),\kappa=\frac{1}{2}}^{\textrm{fund}}(A\to\sqrt{\frac{A_{1}}{A_{2}}},\mathfrak{q}\to-\mathfrak{q}M^{-1/2})Z_{\operatorname{U}(1)}~{},$
(45)

$\bullet\quad(\mathfrak{so}(5),\mathbf{S})\cong(\mathfrak{sp}(2),\mathbf{F})$

$M^{\frac{k}{2}}\widehat{Z}_{\operatorname{SO}(5),k}^{\textrm{spinor}}=Z_{\operatorname{Sp}(2),k}^{\textrm{fund}}(A_{1}\to\sqrt{A_{1}A_{2}},A_{2}\to\sqrt{\frac{A_{1}}{A_{2}}})~{},$
(46)

$\bullet\quad(\mathfrak{so}(6),\mathbf{S})\cong(\mathfrak{su}(4),\mathbf{F})$

$M^{\frac{k}{2}}\widehat{Z}_{\operatorname{SO}(6),k}^{\textrm{spinor}}=(-1)^{k}Z_{\operatorname{SU}(4),k,\kappa=\frac{1}{2}}^{\textrm{fund}}~{}.$
(47)

$\bullet\quad(\mathfrak{so}(6),\mathbf{C})\cong(\mathfrak{su}(4),\mathbf{F})$

$M^{\frac{k}{2}}\widehat{Z}_{\operatorname{SO}(6),k}^{\textrm{conj}}=Z_{\operatorname{SU}(4),k,\kappa=-\frac{1}{2}}^{\textrm{fund}}(M\to
M^{-1})~{}.$ (48)

Note that the U(1) factor is given by

$Z_{\operatorname{U}(1)}=\operatorname{PE}\left[-\frac{\mathfrak{q}\operatorname{ch}\epsilon_{+}}{2\operatorname{sh}\epsilon_{1,2}}\right]~{}.$
(49)

Also, we need to shift the 5d Chern-Simons level by $\pm\frac{1}{2}$ for
$\operatorname{SU}(N)$ gauge theory with one fundamental because of the parity
anomaly.

In particular, (46) implies that the top ($j=k$) and bottom ($j=0$) components
of the hypermultiplet particles are equal to

$\displaystyle\widehat{Z}_{k,0}^{\operatorname{SO}(5)}=$ $\displaystyle
Z_{\operatorname{Sp}(2),k,\theta=0}^{\textrm{pure}}(A_{1}\to\sqrt{A_{1}A_{2}},A_{2}\to\sqrt{\frac{A_{1}}{A_{2}}})~{},$
(50) $\displaystyle\widehat{Z}_{k,k}^{\operatorname{SO}(5)}=$ $\displaystyle
Z_{\operatorname{Sp}(2),k,\theta=\pi}^{\textrm{pure}}(A_{1}\to\sqrt{A_{1}A_{2}},A_{2}\to\sqrt{\frac{A_{1}}{A_{2}}})~{}.$

Also, (47) and (48) indicate that

$\widehat{Z}_{k,0}^{\operatorname{SO}(6)}=(-1)^{k}Z_{\operatorname{SU}(4),k,\kappa=0}^{\textrm{pure}}~{},\qquad\widehat{Z}_{k,k}^{\operatorname{SO}(6)}=Z_{\operatorname{SU}(4),k,\kappa=1}^{\textrm{pure}}~{}.$
(51)

Certainly, of most importance is the triality of $\operatorname{SO}(8)$. The
$\operatorname{SO}(8)$ instanton partition functions enjoy the triality
Kim:2019uqw

${(\mathfrak{so}(8),\textbf{S})}$${(\mathfrak{so}(8),\textbf{V})}$${(\mathfrak{so}(8),\textbf{C})}$c.o.v.c.o.v.c.o.v.

where the change of variables for the instanton partition functions is given
by

$A_{1}\to\sqrt{A_{1}A_{2}A_{3}A_{4}}~{},\quad
A_{2}\to\sqrt{\frac{A_{1}A_{2}}{A_{3}A_{4}}}~{},\quad
A_{3}\to\sqrt{\frac{A_{1}A_{3}}{A_{2}A_{4}}}~{},\quad
A_{4}\to\sqrt{\frac{A_{2}A_{3}}{A_{1}A_{4}}}$ (52)

We check the triality upto 5-instanton.

##  4 $\operatorname{SO}(n)$ $\mathcal{N}=1^{*}$ theory

The maximally supersymmetric 5d $\operatorname{SO}(n)$ gauge theory can be
realized as a worldvolume theory on D4-branes with an O4--plane in Type IIA
theory. The scalar field in the adjoint (antisymmetric) hypermultiplet
represents the fluctuation of a D4-brane along the transverse direction to its
worldvolume. When we turn on the $\Omega$-background transverse to the
D4-branes/O4-plane, the adjoint hypermultiplet acquires a mass, giving rise to
the $\operatorname{SO}(n)$ $\mathcal{N}=1^{*}$ theory.

The equivariant index for the adjoint hypermultiplet is described in detail in
Shadchin:2005mx . The integrand of the JK residue integral for the instanton
partition function of the $\operatorname{SO}(n)$ $\mathcal{N}=1^{*}$ theory is
given by

$\displaystyle\mathcal{I}_{\operatorname{SO}(n),k}^{\textrm{adj}}$ (53)
$\displaystyle=$
$\displaystyle\frac{1}{k!2^{k}}\prod_{I=1}^{k}\frac{\operatorname{sh}{(2\epsilon_{+})}\operatorname{sh}(\pm
2\phi_{I})\operatorname{sh}(\pm
2\phi_{I}+2\epsilon_{+})}{\operatorname{sh}{(\epsilon_{1,2})}\operatorname{sh}^{\chi}(\epsilon_{+}\pm\phi_{I})\,\prod_{s=1}^{\lfloor\frac{n}{2}\rfloor}\operatorname{sh}{(\pm\phi_{I}\pm
a_{s}+\epsilon_{+})}}\prod_{I<J}^{k}\frac{\operatorname{sh}{(\pm\phi_{I}\pm\phi_{J})}\operatorname{sh}{(\pm\phi_{I}\pm\phi_{J}+2\epsilon_{+})}}{\operatorname{sh}{(\pm\phi_{I}\pm\phi_{J}+\epsilon_{1,2})}}$
(54) $\displaystyle\cdot\prod_{I=1}^{k}\frac{\operatorname{sh}{(\pm
m-\epsilon_{-})}\operatorname{sh}^{\chi}(m\pm\phi_{I})\prod_{s=1}^{\lfloor\frac{n}{2}\rfloor}\operatorname{sh}{(\pm\phi_{I}\pm
a_{s}+m)}}{\operatorname{sh}{(\pm m-\epsilon_{+})}\operatorname{sh}{(\pm
2\phi_{I}\pm
m-\epsilon_{+})}}\prod_{I<J}^{k}\frac{\operatorname{sh}{(\pm\phi_{I}\pm\phi_{J}\pm
m-\epsilon_{-})}}{\operatorname{sh}{(\pm\phi_{I}\pm\phi_{J}\pm
m-\epsilon_{+})}}$ (55)

where $n\equiv\chi\bmod 2$.

At the unrefined level, $\hbar$ and $m$ play a similar role in the JK residue
integral. This is natural because they are the parameters for the
$\Omega$-background. Indeed, poles are classified by
$(\lfloor\frac{n}{2}\rfloor+4)$-tuples of 4d young diagrams with the total
number of boxes $k$. We specify a pole location associated with a content
$x=(x_{1},x_{2},x_{3},x_{4})\in\lambda^{(s)}$ as

$\phi_{s}(x)=a_{s}+(x_{1}-x_{2})\hbar+(x_{3}-x_{4})m~{}.$ (56)

Note that $a_{s}$ ($s=1,\ldots,\lfloor\frac{n}{2}\rfloor$) are the Coulomb
branch parameters of the $\operatorname{SO}(n)$ gauge theory. Similarly to
(14), the other four effective Coulomb branch parameters are given by

$a_{\lfloor\frac{n}{2}\rfloor+j}=\frac{m}{2}(+\pi i)~{},\
\frac{\hbar-m}{2}(+\pi i)~{},$ (57)

for $j=1,2,3,4$. For the purpose of later use, we define

$\displaystyle\widetilde{Z}_{k}(\vec{A};q)$ (58) $\displaystyle=$
$\displaystyle\sum_{\vec{\lambda}}C^{\textrm{4d}}_{\vec{\lambda},\vec{A}}\Bigg{(}\frac{\operatorname{sh}(m\pm\hbar)\,\operatorname{sh}^{2}(0)}{\operatorname{sh}^{2}(\hbar)\,\operatorname{sh}^{2}(m)}\Bigg{)}^{k}$
(59)
$\displaystyle\prod_{s=1}^{\|\vec{\lambda}\|}\prod_{x\in\lambda^{(s)}}\frac{\operatorname{sh}^{4}(2\phi_{s}(x))\,\operatorname{sh}^{2}(2\phi_{s}(x)+m-\hbar)\,\operatorname{sh}^{2}(2\phi_{s}(x)-m+\hbar)\prod_{t=1}^{N}\operatorname{sh}(\phi_{s}(x)\pm
a_{t}\pm
m)}{\prod_{t=1}^{\|\vec{\lambda}\|}\operatorname{sh}^{2}(\phi_{s}(x)\pm
a_{t})}$ (60) $\displaystyle\prod_{s\leq
t}^{\|\vec{\lambda}\|}\prod_{\begin{subarray}{c}x\in\lambda^{(s)},\,y\in\lambda^{(t)}\\\
x<y\end{subarray}}\frac{\operatorname{sh}^{4}(\phi_{s}(x)\pm\phi_{t}(y))\,\operatorname{sh}(\phi_{s}(x)\pm\phi_{t}(y)\pm
m\pm\hbar)}{\operatorname{sh}^{2}(\phi_{s}(x)\pm\phi_{t}(y)\pm\hbar)\,\operatorname{sh}^{2}(\phi_{s}(x)\pm\phi_{t}(y)\pm
m)}$

where $\|\vec{\lambda}\|$ is the number of 4d Young diagrams in
$\vec{\lambda}$, and the total ordering on the boxes is the natural extension
of (13). In this case, $\|\vec{\lambda}\|=\lfloor\frac{n}{2}\rfloor+4$, and
the instanton partition function is

$Z_{\operatorname{SO}(n),k}^{\textrm{adj}}=\widetilde{Z}_{k}(\vec{A},A_{N+1}=e^{-\frac{m}{2}},A_{N+2}=-e^{-\frac{m}{2}},A_{N+3}=e^{-\frac{\hbar-m}{2}},A_{N+4}=-e^{-\frac{\hbar-m}{2}};q)$
(61)

where $N=\lfloor\frac{n}{2}\rfloor$. As in (18), the non-trivial multiplicity
constants are involved due to higher-order poles coming from the additional
effective Coulomb branch parameters (57). We do _not_ know how to determine
these multiplicity constants for general 4d Young diagrams. However, some of
the multiplicity constants $C^{\textrm{4d}}_{\vec{\lambda},\vec{A}}$ are
listed in Appendix C.

Since the adjoint representation $\boldsymbol{6}$ of $\mathfrak{so}(4)$ is
isomorphic to the direct sum $(\boldsymbol{3,1})\oplus(\boldsymbol{1,3})$ of
the adjoint representations of $\mathfrak{su}(2)\oplus\mathfrak{su}(2)$, we
have the following identity at the unrefined level

$Z_{\operatorname{SO}(4)}^{\textrm{adj}}(A_{1},A_{2},M)=Z_{\operatorname{SU}(2)}^{\textrm{adj}}(A=(A_{1}A_{2})^{\frac{1}{2}},M)Z_{\operatorname{SU}(2)}^{\textrm{adj}}(A=(A_{1}/A_{2})^{\frac{1}{2}},M)Z_{\textrm{extra}}$
(62)

where

$Z_{\textrm{extra}}=\mathrm{PE}\left[\frac{\mathfrak{q}(2+3M+2M^{2})(q-M)(1-Mq)}{(1-\mathfrak{q})M(1+M)^{2}(1-q)^{2}}\right]~{}.$
(63)

Also, the adjoint representation of $\mathfrak{so}(6)$ is isomorphic to that
of $\mathfrak{su}(4)$, which leads to

$Z_{\operatorname{SO}(6)}^{\textrm{adj}}(A_{1},A_{2},A_{3},M)=Z_{\operatorname{SU}(4)}^{\textrm{adj}}(A_{1},A_{2},A_{3},M)Z^{\prime}_{\textrm{extra}}$
(64)

where

$Z^{\prime}_{\textrm{extra}}=\mathrm{PE}\left[\frac{\mathfrak{q}(1+M+M^{2})(q-M)(1-Mq)}{(1-\mathfrak{q})M(1+M)^{2}(1-q)^{2}}\right]~{}.$
(65)

We check these identities upto 5-instanton.

##  5 $\operatorname{Sp}(N)$ gauge group with antisymmetric hypermultiplet

If we replace the O4--plane by an O8--plane in the brane configuration
described in the previous section, the gauge group becomes
$\operatorname{Sp}(N)$, and the fluctuation of a D4-brane along the transverse
directions to its worldvolume on the O8--plane describes the scalar field in
the antisymmetric hypermultiplet. Hence, the D4-O8- system in Type IIA theory
gives rise to the $\operatorname{Sp}(N)$ gauge theory with the antisymmetric
hypermultiplet.

This brane system has close connections to Type I’ theory. From the viewpoint
of Type I’ theory, the instanton quantum mechanics is described in
(Kim:2012gu, , Appendix D). Using the instanton quantum mechanics, we can
write down the instanton partition functions as JK contour integral formulas
Shadchin:2005mx ; Kim:2012gu ; Hwang:2014uwa . The results are summarized in
Appendix B.3.

Again, the gauge group $\mathrm{O}(k)$ of the instanton quantum mechanics has
two components, which results in two contributions
$Z^{\textrm{anti}}_{\operatorname{Sp}(N),k,\pm}$ to the partition function.
Then, the $\theta$-angle can be distinguished through the following equations:

$Z_{\operatorname{Sp}(N),k,\theta=0}^{\textrm{anti}}=\frac{Z^{\textrm{anti}}_{\operatorname{Sp}(N),k,+}+Z^{\textrm{anti}}_{\operatorname{Sp}(N),k,-}}{2}~{},\quad
Z_{\operatorname{Sp}(N),k,\theta=\pi}^{\textrm{anti}}=(-1)^{k}\frac{Z^{\textrm{anti}}_{\operatorname{Sp}(N),k,+}-Z^{\textrm{anti}}_{\operatorname{Sp}(N),k,-}}{2}~{}.$
(66)

In the unrefined limit, the JK poles are classified using 4d Young diagrams,
and a pole location at a content $x\in\lambda^{(s)}$ is as in (56).
Additionally, all the residues can be expressed in a single universal formula,
as shown in (58). This is similar to the relationship between the instanton
partition functions of pure Yang-Mills theory for $\operatorname{SO}(n)$ and
$\operatorname{Sp}(N)$ gauge groups Nawata:2021dlk . This relationship still
holds when including an antisymmetric hypermultiplet. However, there are more
effective Coulomb branch parameters than (61), depending on the instanton
number and plus/minus sector:

$\bullet$ (even,$+$) sector

$Z^{\textrm{anti}}_{\operatorname{Sp}(N),k=2\ell,+}=\widetilde{Z}_{\ell}(\vec{A};q)$
(67)

where there are 8 additional effective Coulomb branch parameters
($j=1,\ldots,8$)

$a_{N+j}=\frac{\hbar}{2}(+\pi i)~{},\ \frac{m}{2}(+\pi i)~{},\
\frac{\hbar-m}{2}(+\pi i)~{},\ 0(+\pi i)~{}.$

$\bullet$ (odd,$+$) sector

$Z^{\textrm{anti}}_{\operatorname{Sp}(N),k=2\ell+1,+}=\frac{1}{2\operatorname{sh}^{2}(\hbar)\operatorname{sh}^{2}(m)}\prod^{N}_{t=1}\frac{\operatorname{sh}(a_{t}\pm
m)}{\operatorname{sh}^{2}(a_{t})}\prod_{s=1}^{\|\vec{\lambda}\|}\prod_{x\in\lambda^{(s)}}\operatorname{sh}(\phi_{s}(x)\pm
m\pm\hbar)\widetilde{Z}_{\ell}(\vec{A};q)$ (68)

where there are 9 additional effective Coulomb branch parameters
($j=1,\ldots,9$)

$a_{N+j}=\frac{\hbar}{2}(+\pi i)~{},\ \frac{m}{2}(+\pi i)~{},\
\frac{\hbar-m}{2}(+\pi i)~{},\ \pi i~{},\ \hbar~{},\ m~{}.$

$\bullet$ (even,$-$) sector

$Z^{\textrm{anti}}_{\operatorname{Sp}(N),k=2\ell,+}=\frac{\operatorname{ch}(m\pm\hbar)}{\operatorname{sh}^{2}(\hbar)\operatorname{sh}^{2}(2\hbar)\operatorname{sh}^{2}(m)\operatorname{sh}^{2}(2m)}\prod^{N}_{t=1}\frac{\operatorname{sh}(a_{t}\pm
m)\operatorname{ch}(a_{t}\pm
m)}{\operatorname{sh}^{2}(2a_{t})}\widetilde{Z}_{\ell-1}(\vec{A};q)$ (69)

where there are 10 additional effective Coulomb branch parameters
($j=1,\ldots,10$)

$a_{N+j}=\frac{\hbar}{2}(+\pi i)~{},\ \frac{m}{2}(+\pi i)~{},\
\frac{\hbar-m}{2}(+\pi i)~{},\ \hbar(+\pi i)~{},\ m(+\pi i)~{}.$

$\bullet$ (odd,$-$) sector

$Z^{\textrm{anti}}_{\operatorname{Sp}(N),k=2\ell+1,+}=\frac{1}{2\operatorname{sh}^{2}(\hbar)\operatorname{sh}^{2}(m)}\prod^{N}_{t=1}\frac{\operatorname{ch}(a_{t}\pm
m)}{\operatorname{ch}^{2}(a_{t})}\prod_{s=1}^{\|\vec{\lambda}\|}\prod_{x\in\lambda^{(s)}}\operatorname{ch}(\phi_{s}(x)\pm
m\pm\hbar)\widetilde{Z}_{\ell}(\vec{A};q)$ (70)

where there are 9 additional effective Coulomb branch parameters
($j=1,\ldots,9$)

$a_{N+j}=\frac{\hbar}{2}(+\pi i)~{},\ \frac{m}{2}(+\pi i)~{},\
\frac{\hbar-m}{2}(+\pi i)~{},\ 0~{},\ \hbar+\pi i~{},\ m+\pi i~{}.$

Note that the multiplicity coefficients are involved for the additional
Coulomb branch parameters due to the presence of higher-order poles in (58),
and we do not know how to determine them for general 4d Young diagrams.
Nonetheless, we list some of them in Appendix C.

We can verify the isomorphisms of representations by comparing the instanton
partition functions of different theories. For example, the antisymmetric
representation of $\operatorname{Sp}(1)$ is trivial, so it decouples from the
gauge theory and does not affect the instanton dynamics. Thus, the instanton
partition function for the $\operatorname{Sp}(1)$ theory with an antisymmetric
hypermultiplet is equal to the pure $\operatorname{Sp}(1)$ instanton partition
function

$\displaystyle Z_{\operatorname{Sp}(1),\theta=0}^{\textrm{anti}}=$
$\displaystyle
Z_{\operatorname{Sp}(1),\theta=0}^{\textrm{pure}}Z_{\textrm{D0}}~{},$ (71)
$\displaystyle Z_{\operatorname{Sp}(1),\theta=\pi}^{\textrm{anti}}=$
$\displaystyle Z_{\operatorname{Sp}(1),\theta=\pi}^{\textrm{pure}}$ (72)

up to the 8 translational zero modes of a single D0-brane

$Z_{\textrm{D0}}=\operatorname{PE}\Big{[}\frac{\mathfrak{q}}{\operatorname{sh}({\epsilon_{1,2}})\operatorname{sh}({m\pm\epsilon_{+}})}\Big{]}~{}.$
(73)

On the other hand, the antisymmetric representation of $\operatorname{Sp}(2)$
is isomorphic to the vector representation of $\operatorname{SO}(5)$, so the
instanton partition functions should obey the relation

$Z_{\operatorname{Sp}(2),\theta=0}^{\textrm{anti}}Z_{\operatorname{U}(1)}=Z_{\operatorname{SO}(5)}^{\textrm{vect}}Z_{\textrm{D0}}$
(74)

where the U(1) instanton partition function is given in (49). We have checked
(71) and (74) upto 8-instanton at the unrefined level, respectively.

Figure 4: Fivebrane web configuration of $\operatorname{Sp}(2)$ gauge theory
with massless antisymmetric hypermultiplet in Type IIB theory where the left-
panel represents $\theta=0$ while the right-panel illustrates $\theta=\pi$.

Once we take T-duality from the brane system in Type IIA theory, we can
construct 5d $\operatorname{Sp}(N)$ theories with an antisymmetric
hypermultiplet by fivebrane web with two O7--planes in Type IIB theory
Bergman:2015dpa ; Hayashi:2018lyv ; Cheng:2018wll ; Hayashi:2019yxj . The
brane configuration can be further deformed by splitting an O7--plane into two
$(p,q)$-sevenbranes Sen:1996vd  and performing Hanany-Witten transitions
Hanany:1996ie . In the massless limit of the antisymmetric matter
($m=\epsilon_{+}$ at the refined level), the resulting fivebrane web takes the
form of a product of pure $\operatorname{Sp}(1)$ theories Cheng:2018wll , as
shown in Figure 4. This brane moves and the Higgsing procedure can also be
verified at the level of instanton partition functions as

$\displaystyle
Z_{\operatorname{Sp}(2),\theta=0}^{\textrm{anti}}(m=\epsilon_{+})=$
$\displaystyle
Z_{\operatorname{Sp}(1),\theta=0}^{\textrm{pure}}(A_{1})Z_{\operatorname{Sp}(1),\theta=0}^{\textrm{pure}}(A_{2})Z_{\textrm{D0}}$
(75) $\displaystyle
Z_{\operatorname{Sp}(2),\theta=\pi}^{\textrm{anti}}(m=\epsilon_{+})=$
$\displaystyle
Z_{\operatorname{Sp}(1),\theta=\pi}^{\textrm{pure}}(A_{1})Z_{\operatorname{Sp}(1),\theta=\pi}^{\textrm{pure}}(A_{2})~{}.$
(76)

In order to move forward with the research presented in this paper, it is
important to fully determine the multiplicity constants in (58). One potential
approach to this problem is to consider the blowup equation on
$\mathbb{C}^{2}_{\pm\hbar}\times\mathbb{C}^{2}_{\pm m}$, as the equivariant
parameters $\hbar$ and $m$ play a similar role in this context.

While (58) currently yields non-trivial rational functions through the
cancellation of Sh(0) in the numerator with poles in the denominator, it would
be valuable to find a closed-form expression similar to the Nekrasov factor
(11) for 4d Young diagrams. This could provide deeper insights and simplify
calculations furthermore.

###### Acknowledgements.

We are grateful to Sung-Soo Kim, Xiaobin Li, and Futoshi Yagi for the valuable
discussion on the related topic Chengdu . Their help was essential in the
development of this paper, particularly the results in §2 which stemmed from
the collaboration with them. We would also like to thank Jiaqi Guo and Rui-
Dong Zhu for the discussion. Additionally, we extend our sincere gratitude to
the anonymous referee of JHEP who carefully reviewed our paper. S.N. is
indebted to Southwest Jiaotong University for the hospitality where a part of
the work is carried out. The research of S.N. is supported by the National
Science Foundation of China under Grant No.12050410234 and Shanghai Foreign
Expert grant No. 22WZ2502100.

##  Appendix A Notations and definitions

In this appendix, we provide a summary of the notations and definitions used
throughout the paper.

We denote a 2d Young diagram by
$\lambda=\left(\lambda_{1},\lambda_{2},\cdots\right)$, which is a sequence of
non-negative integers such that $\lambda_{i}\geq\lambda_{i+1}$ and
$|\lambda|=\sum_{i}\lambda_{i}<\infty$. We write the transposition of
$\lambda$ by $\lambda^{t}$. The arm length $a_{\lambda}(x)$ of a box at
$x=(x_{1},x_{2})$ in the Young diagram is the number of boxes to the right of
the box, and the leg length $l_{\lambda}(x)$ is the number of boxes below it.
This is illustrated in the following figure:

$x_{2}$$x_{1}$$x$$a_{\lambda}(x)$$l_{\lambda}(x)$ Figure 5: Arm and leg length
of a box $x=(x_{1},x_{2})$ in a Young diagram.

We also introduce the notation for 5d instanton partition functions. The
moduli spaces of instantons receive equivariant actions of
$\operatorname{SO}(2)_{\epsilon_{1}}\times\operatorname{SO}(2)_{\epsilon_{1}}$
of the space-time. Also, the maximal torus
$\prod_{s=1}^{{\textrm{rank}G}}\operatorname{U}(1)_{a_{s}}$ of the gauge group
$G$ and that $\operatorname{U}(1)_{m}$ of the flavor group act on the
instanton moduli spaces. Consequently, 5d instanton partition functions are
rational functions of variables

$q=e^{-\epsilon_{1}}~{},\quad t=e^{\epsilon_{2}}~{},\quad
A_{s}=e^{-a_{s}}~{},\quad M=e^{-m}.$

The parameters $A_{s}$ are Coulomb branch parameters, and the parameter $M$ is
the mass parameter of a hypermultiplet. We also use the notation
$2\epsilon_{\pm}=\epsilon_{1}\pm\epsilon_{2}$.

In this paper, we mainly focus on the unrefined limit
$\epsilon_{1}=-\epsilon_{2}=\hbar$, and we write $q=e^{-\hbar}$. We denote a
$k$-instanton partition function with a hypermultiplet in a representation of
$G$ by $Z_{G,k}^{\textrm{rep}}$. This paper obtains expressions of
$k$-instanton partition functions summed over $P$-tuples of Young diagrams,
which is denoted by

$\vec{\lambda}=(\lambda^{(1)},\ldots,\lambda^{(P)})~{},$

where the total number of boxes satisfy

$k=\sum_{s=1}^{\|\vec{\lambda}\|}|\lambda^{(s)}|~{}.$

Here we also denote the number of Young diagrams by $\|\vec{\lambda}\|=P$. The
total instanton partition function is expressed as a generating function with
instanton counting parameter $\mathfrak{q}$

$Z_{G}^{\textrm{rep}}=\sum_{k=0}^{\infty}\mathfrak{q}^{k}Z_{G,k}^{\textrm{rep}}(A_{i};q)~{}.$

The perturbative partition function is often written in terms of the
plethystic exponent

$\textrm{PE}\left[f(x,y,\cdots)\right]\equiv\exp\left[\sum_{d=1}^{\infty}\frac{1}{d}f(x^{d},y^{d},\cdots)\right]$
(77)

which brings the single particle index $f$ to the multi-particle index.

With these notations and definitions, we can now proceed with the analysis
presented in the main text of the paper.

##  Appendix B Integral expressions of instanton partition functions

To be self-contained, in this Appendix, we provide a summary of the partition
functions of supersymmetric quantum mechanics on instanton moduli spaces, as
studied in Kim:2018gjo ; Kim:2012gu . For more details, we refer the reader to
the original references.

###  B.1 $\operatorname{SO}(2N+1)$ with spinor hypermultiplet

The brane system for $\operatorname{SO}(2N+1)$ gauge theory with spinor
hypermultiplet is schematically drawn in Figure 2. The instanton quantum
mechanics can be derived from open strings in this brane system.
$\mathcal{N}=4$ supersymmetric quantum mechanics on $k$ D1-branes and $j$
D1’-branes is given in Figure 6. The formal $\operatorname{SO}(1)$ comes from
the half D5-brane on the $\widetilde{\mathrm{O}5}^{+}$-plane, on the leg part
of fivebranes in Figure 2. For 5d gauge groups of $B$ type, the contributions
from $\mathrm{O}(2\ell)_{-}$ and $\mathrm{O}(2\ell+1)_{+}$ sectors vanish due
to the fermionic zero modes.

Figure 6: The solid line represents a hypermultiplet while the dashed line
represents a Fermi multiplet.

Hence, the integrand of the supersymmetric quantum mechanics for $k$
instantons and $2\ell$ hypermultiplet particles is given by

$\displaystyle\mathcal{I}^{\operatorname{SO}(2N+1)}_{k,2\ell}$ (78)
$\displaystyle=$
$\displaystyle\frac{1}{2^{k}\,k!}\frac{\operatorname{sh}^{k}(2\epsilon_{+})}{\operatorname{sh}^{k}(\epsilon_{1,2})}\prod_{I=1}^{k}\frac{\operatorname{sh}(2\epsilon_{+}\pm
2\phi_{I})\,\operatorname{sh}(\pm
2\phi_{I})}{\operatorname{sh}(\epsilon_{+}\pm\phi_{I})\prod_{s=1}^{N}\operatorname{sh}(\epsilon_{+}\pm\phi_{I}\pm
a_{s})}\prod_{I>J}\frac{\operatorname{sh}(2\epsilon_{+}\pm\phi_{I}\pm\phi_{J})\,\operatorname{sh}(\pm\phi_{I}\pm\phi_{J})}{\operatorname{sh}(\epsilon_{1,2}\pm\phi_{I}\pm\phi_{J})}$
(79)
$\displaystyle\frac{1}{2^{\ell}\,\ell!}\frac{\operatorname{sh}^{\ell}(2\epsilon_{+})}{\operatorname{sh}^{\ell}(\epsilon_{1,2})}\prod_{i=1}^{\ell}\frac{\operatorname{sh}(\pm\chi_{i})\prod_{s=1}^{N}\operatorname{sh}(\pm\chi_{i}+a_{s})}{\operatorname{sh}(\epsilon_{1,2}\pm
2\chi_{i})}\prod_{i>j}\frac{\operatorname{sh}(2\epsilon_{+}\pm\chi_{i}\pm\chi_{j})\,\operatorname{sh}(\pm\chi_{i}\pm\chi_{j})}{\operatorname{sh}(\epsilon_{1,2}\pm\chi_{i}\pm\chi_{j})}$
(80)
$\displaystyle\prod_{I=1}^{k}\prod_{i=1}^{\ell}\frac{\operatorname{sh}(\epsilon_{-}\pm\phi_{I}\pm\chi_{i})}{\operatorname{sh}(-\epsilon_{+}\pm\phi_{I}\pm\chi_{i})}\
.$ (81)

Here $\phi_{I}$ $(I=1,\cdots,k)$ are $\operatorname{Sp}(k)$ gauge fugacities,
and $\chi_{i}$ $(i=1,\cdots,\ell)$ are $\mathrm{O}(2\ell)$ gauge fugacities so
that they are integrated by the JK prescription. On the other hand, the
integrand for $k$ instantons and $(2\ell+1)$ hypermultiplet particles is given
by

$\displaystyle\mathcal{I}^{\operatorname{SO}(2N+1)}_{k,2\ell+1}$ (82)
$\displaystyle=$
$\displaystyle\frac{1}{2^{k}\,k!}\frac{\operatorname{sh}^{k}(2\epsilon_{+})}{\operatorname{sh}^{k}(\epsilon_{1,2})}\prod_{I=1}^{k}\frac{\operatorname{sh}(2\epsilon_{+}\pm
2\phi_{I})\,\operatorname{sh}(\pm
2\phi_{I})}{\operatorname{sh}(\epsilon_{+}\pm\phi_{I})\prod_{s=1}^{N}\operatorname{sh}(\epsilon_{+}\pm\phi_{I}\pm
a_{s})}\prod_{I>J}\frac{\operatorname{sh}(2\epsilon_{+}\pm\phi_{I}\pm\phi_{J})\,\operatorname{sh}(\pm\phi_{I}\pm\phi_{J})}{\operatorname{sh}(\epsilon_{1,2}\pm\phi_{I}\pm\phi_{J})}$
(83)
$\displaystyle\frac{1}{2^{\ell}\,\ell!}\frac{\operatorname{sh}^{\ell}(2\epsilon_{+})}{\operatorname{sh}^{\ell+1}(\epsilon_{1,2})}\prod_{i=1}^{\ell}\frac{\operatorname{ch}(2\epsilon_{+}\pm\chi_{i})\,\operatorname{sh}(\pm
2\chi_{i})\prod_{s=1}^{N}\operatorname{sh}(\pm\chi_{i}+a_{s})}{\operatorname{ch}(\epsilon_{1,2}\pm\chi_{i})\,\operatorname{sh}(\epsilon_{1,2}\pm
2\chi_{i})}\cdot\prod_{s=1}^{N}\operatorname{ch}(a_{s})$ (84)
$\displaystyle\prod_{i>j}\frac{\operatorname{sh}(2\epsilon_{+}\pm\chi_{i}\pm\chi_{j})\,\operatorname{sh}(\pm\chi_{i}\pm\chi_{j})}{\operatorname{sh}(\epsilon_{1,2}\pm\chi_{i}\pm\chi_{j})}\prod_{I=1}^{k}\frac{\operatorname{ch}(\epsilon_{-}\pm\phi_{I})}{\operatorname{ch}(-\epsilon_{+}\pm\phi_{I})}\prod_{i=1}^{\ell}\frac{\operatorname{sh}(\epsilon_{-}\pm\phi_{I}\pm\chi_{i})}{\operatorname{sh}(-\epsilon_{+}\pm\phi_{I}\pm\chi_{i})}\
.$ (85)

###  B.2 $\operatorname{SO}(2N)$ with (conjugate) spinor hypermultiplet

The brane system for $\operatorname{SO}(2N)$ gauge theory with (conjugate)
spinor hypermultiplet is shown in Figure 2. This brane system allows us to
derive the instanton quantum mechanics from open strings. In Figure 7, we see
the $\mathcal{N}=4$ supersymmetric quantum mechanics on $k$ D1-branes and $j$
D1’-branes. In this case, there are non-trivial contributions from the two
connected components of the gauge group $\mathrm{O}(j)$.

Figure 7: The solid line represents a hypermultiplet while the dashed line
represents a Fermi multiplet.

To account for each contribution, we write down an integrand of the partition
function for each one:

$\begin{split}&\mathcal{I}^{\operatorname{SO}(2N)}_{k,2\ell,+}=\\\
&\frac{1}{2^{k}\,k!}\frac{\operatorname{sh}^{k}(2\epsilon_{+})}{\operatorname{sh}^{k}(\epsilon_{1,2})}\prod_{I=1}^{k}\frac{\operatorname{sh}(2\epsilon_{+}\pm
2\phi_{I})\,\operatorname{sh}(\pm
2\phi_{I})}{\prod\limits_{s=1}^{N}\operatorname{sh}(\epsilon_{+}\pm\phi_{I}\pm
a_{s})}\prod_{I>J}\frac{\operatorname{sh}(2\epsilon_{+}\pm\phi_{I}\pm\phi_{J})\,\operatorname{sh}(\pm\phi_{I}\pm\phi_{J})}{\operatorname{sh}(\epsilon_{1,2}\pm\phi_{I}\pm\phi_{J})}\\\
&\frac{2^{\text{sign}(\ell)}}{2^{\ell}\,\ell!}\frac{\operatorname{sh}^{\ell}(2\epsilon_{+})}{\operatorname{sh}^{\ell}(\epsilon_{1,2})}\prod_{i=1}^{\ell}\frac{\prod\limits_{s=1}^{N}\operatorname{sh}(\pm\chi_{i}+a_{s})}{\operatorname{sh}(\epsilon_{1,2}\pm
2\chi_{i})}\prod_{i>j}\frac{\operatorname{sh}(2\epsilon_{+}\pm\chi_{i}\pm\chi_{j})\,\operatorname{sh}(\pm\chi_{i}\pm\chi_{j})}{\operatorname{sh}(\epsilon_{1,2}\pm\chi_{i}\pm\chi_{j})}\\\
&\prod_{I=1}^{k}\prod_{i=1}^{\ell}\frac{\operatorname{sh}(\epsilon_{-}\pm\phi_{I}\pm\chi_{i})}{\operatorname{sh}(-\epsilon_{+}\pm\phi_{I}\pm\chi_{i})}\end{split}$
(86) $\begin{split}&\mathcal{I}^{\operatorname{SO}(2N)}_{k,2\ell,-}=\\\
&\frac{1}{2^{k}\,k!}\frac{\operatorname{sh}^{k}(2\epsilon_{+})}{\operatorname{sh}^{k}(\epsilon_{1,2})}\prod_{I=1}^{k}\frac{\operatorname{sh}(2\epsilon_{+}\pm
2\phi_{I})\,\operatorname{sh}(\pm
2\phi_{I})}{\prod\limits_{s=1}^{N}\operatorname{sh}(\epsilon_{+}\pm\phi_{I}\pm
a_{s})}\prod_{I>J}\frac{\operatorname{sh}(2\epsilon_{+}\pm\phi_{I}\pm\phi_{J})\,\operatorname{sh}(\pm\phi_{I}\pm\phi_{J})}{\operatorname{sh}(\epsilon_{1,2}\pm\phi_{I}\pm\phi_{J})}\\\
&\Bigg{[}\frac{\prod\limits_{s=1}^{N}\operatorname{sh}(2a_{s})}{2^{\ell-1}\,(\ell-1)!}\frac{\operatorname{ch}(2\epsilon_{+})}{\operatorname{sh}(2\epsilon_{1,2})\operatorname{sh}(\epsilon_{1,2})}\prod_{i=1}^{\ell-1}\frac{\operatorname{sh}(2\epsilon_{+})\,\operatorname{sh}(4\epsilon_{+}\pm
2\chi_{i})\,\operatorname{sh}(\pm
2\chi_{i})\prod\limits_{s=1}^{N}\operatorname{sh}(\pm\chi_{i}+a_{s})}{\operatorname{sh}(\epsilon_{1,2})\,\operatorname{sh}(2\epsilon_{1,2}\pm
2\chi_{i})\,\operatorname{sh}(\epsilon_{1,2}\pm 2\chi_{i})}\\\
&\prod_{i>j}\frac{\operatorname{sh}(2\epsilon_{+}\pm\chi_{i}\pm\chi_{j})\,\operatorname{sh}(\pm\chi_{i}\pm\chi_{j})}{\operatorname{sh}(\epsilon_{1,2}\pm\chi_{i}\pm\chi_{j})}\prod_{I=1}^{k}\frac{\operatorname{sh}(2\epsilon_{-}\pm
2\phi_{I})}{\operatorname{sh}(-2\epsilon_{+}\pm
2\phi_{I})}\prod_{i=1}^{\ell-1}\frac{\operatorname{sh}(\epsilon_{-}\pm\phi_{I}\pm\chi_{i})}{\operatorname{sh}(-\epsilon_{+}\pm\phi_{I}\pm\chi_{i})}\Bigg{]}^{\text{sign}(\ell)}\end{split}$
(87) $\begin{split}&\mathcal{I}^{\operatorname{SO}(2N)}_{k,2\ell+1,+}=\\\
&\frac{1}{2^{k}\,k!}\frac{\operatorname{sh}^{k}(2\epsilon_{+})}{\operatorname{sh}^{k}(\epsilon_{1,2})}\prod_{I=1}^{k}\frac{\operatorname{sh}(2\epsilon_{+}\pm
2\phi_{I})\,\operatorname{sh}(\pm
2\phi_{I})}{\prod\limits_{s=1}^{N}\operatorname{sh}(\epsilon_{+}\pm\phi_{I}\pm
a_{s})}\prod_{I>J}\frac{\operatorname{sh}(2\epsilon_{+}\pm\phi_{I}\pm\phi_{J})\,\operatorname{sh}(\pm\phi_{I}\pm\phi_{J})}{\operatorname{sh}(\epsilon_{1,2}\pm\phi_{I}\pm\phi_{J})}\\\
&\frac{\prod\limits_{s=1}^{N}\operatorname{sh}(a_{s})}{2^{\ell}\,\ell!}\frac{\operatorname{sh}^{\ell}(2\epsilon_{+})}{\operatorname{sh}^{\ell+1}(\epsilon_{1,2})}\prod_{i=1}^{\ell}\frac{\operatorname{sh}(2\epsilon_{+}\pm\chi_{i})\,\operatorname{sh}(\pm\chi_{i})\prod\limits_{s=1}^{N}\operatorname{sh}(\pm\chi_{i}+a_{s})}{\operatorname{sh}(\epsilon_{1,2}\pm\chi_{i})\,\operatorname{sh}(\epsilon_{1,2}\pm
2\chi_{i})}\\\
&\prod_{i>j}\frac{\operatorname{sh}(2\epsilon_{+}\pm\chi_{i}\pm\chi_{j})\,\operatorname{sh}(\pm\chi_{i}\pm\chi_{j})}{\operatorname{sh}(\epsilon_{1,2}\pm\chi_{i}\pm\chi_{j})}\prod_{I=1}^{k}\frac{\operatorname{sh}(\epsilon_{-}\pm\phi_{I})}{\operatorname{sh}(-\epsilon_{+}\pm\phi_{I})}\prod_{i=1}^{\ell}\frac{\operatorname{sh}(\epsilon_{-}\pm\phi_{I}\pm\chi_{i})}{\operatorname{sh}(-\epsilon_{+}\pm\phi_{I}\pm\chi_{i})}\end{split}$
(88) $\begin{split}&\mathcal{I}^{\operatorname{SO}(2N)}_{k,2\ell+1,-}=\\\
&\frac{1}{2^{k}\,k!}\frac{\operatorname{sh}^{k}(2\epsilon_{+})}{\operatorname{sh}^{k}(\epsilon_{1,2})}\prod_{I=1}^{k}\frac{\operatorname{sh}(2\epsilon_{+}\pm
2\phi_{I})\,\operatorname{sh}(\pm
2\phi_{I})}{\prod\limits_{s=1}^{N}\operatorname{sh}(\epsilon_{+}\pm\phi_{I}\pm
a_{s})}\prod_{I>J}\frac{\operatorname{sh}(2\epsilon_{+}\pm\phi_{I}\pm\phi_{J})\,\operatorname{sh}(\pm\phi_{I}\pm\phi_{J})}{\operatorname{sh}(\epsilon_{1,2}\pm\phi_{I}\pm\phi_{J})}\\\
&\frac{\prod\limits_{s=1}^{N}\operatorname{ch}(a_{s})}{2^{\ell}\,\ell!}\frac{\operatorname{sh}^{\ell}(2\epsilon_{+})}{\operatorname{sh}^{\ell+1}(\epsilon_{1,2})}\prod_{i=1}^{\ell}\frac{\operatorname{ch}(2\epsilon_{+}\pm\chi_{i})\,\operatorname{ch}(\pm\chi_{i})\prod\limits_{s=1}^{N}\operatorname{sh}(\pm\chi_{i}+a_{s})}{\operatorname{ch}(\epsilon_{1,2}\pm\chi_{i})\,\operatorname{sh}(\epsilon_{1,2}\pm
2\chi_{i})}\\\
&\prod_{i>j}\frac{\operatorname{sh}(2\epsilon_{+}\pm\chi_{i}\pm\chi_{j})\,\operatorname{sh}(\pm\chi_{i}\pm\chi_{j})}{\operatorname{sh}(\epsilon_{1,2}\pm\chi_{i}\pm\chi_{j})}\prod_{I=1}^{k}\frac{\operatorname{ch}(\epsilon_{-}\pm\phi_{I})}{\operatorname{ch}(-\epsilon_{+}\pm\phi_{I})}\prod_{i=1}^{\ell}\frac{\operatorname{sh}(\epsilon_{-}\pm\phi_{I}\pm\chi_{i})}{\operatorname{sh}(-\epsilon_{+}\pm\phi_{I}\pm\chi_{i})}\end{split}$
(89)

###  B.3 $\operatorname{Sp}(N)$ with antisymmetric hypermultiplet

Let us recall the supersymmetric quantum mechanics of the $k$ instanton moduli
space of pure $\operatorname{Sp}(N)$ gauge theory Nekrasov-Shadchin . This
system is described by an $\mathrm{O}(k)$ gauge theory with a (rank-two)
symmetric hypermultiplet and $2N$ fundamental half-hypermultiplets, which have
$\operatorname{Sp}(N)$ flavor symmetry.

The introduction of a 5d antisymmetric hypermultiplet modifies the field
content of the instanton quantum mechanics. Specifically, it introduces an
additional symmetric hypermultiplet and $2N$ fundamental half-Fermi
multiplets. (This is illustrated in Figure 8.) The equivariant index for the
5d antisymmetric hypermultiplet has previously been calculated in
(Shadchin:2005mx, , (5.14)). For more details on the field content of the
instanton quantum mechanics, see (Kim:2012gu, , Appendix D).

Figure 8: The solid line represents a hypermultiplet while the dashed line
represents a Fermi multiplet.

The partition function receives two contributions due to the $\mathrm{O}(k)$
gauge group. These contributions can be calculated by performing the JK
residue integrals of the following integrands:

$\displaystyle\mathcal{I}_{k=2\ell+\chi}^{+}$ (90) $\displaystyle=$
$\displaystyle\left(\frac{1}{\operatorname{sh}{(\epsilon_{1,2})}\,\prod_{s=1}^{N}\operatorname{sh}{(\pm
a_{s}+\epsilon_{+})}}\cdot\prod_{I=1}^{\ell}\frac{\operatorname{sh}{(\pm\phi_{I})}\operatorname{sh}{(\pm\phi_{I}+2\epsilon_{+})}}{\operatorname{sh}{(\pm\phi_{I}+\epsilon_{1,2})}}\right)^{\chi}$
(91)
$\displaystyle\cdot\prod_{I=1}^{\ell}\frac{\operatorname{sh}{(2\epsilon_{+})}}{\operatorname{sh}{(\epsilon_{1,2})}\operatorname{sh}(\pm
2\phi_{I}+\epsilon_{1,2})\,\prod_{s=1}^{N}\operatorname{sh}{(\pm\phi_{I}\pm
a_{s}+\epsilon_{+})}}\prod_{I<J}^{\ell}\frac{\operatorname{sh}{(\pm\phi_{I}\pm\phi_{J})}\operatorname{sh}{(\pm\phi_{I}\pm\phi_{J}+2\epsilon_{+})}}{\operatorname{sh}{(\pm\phi_{I}\pm\phi_{J}+\epsilon_{1,2})}}$
(92) $\displaystyle\cdot\left(\frac{\prod_{s=1}^{N}\operatorname{sh}{(m\pm
a_{s})}}{\operatorname{sh}{(\pm
m-\epsilon_{+})}}\prod_{I=1}^{\ell}\frac{\operatorname{sh}{(\pm\phi_{I}\pm
m-\epsilon_{-})}}{\operatorname{sh}{(\pm\phi_{I}\pm
m-\epsilon_{+})}}\right)^{\chi}\prod_{I=1}^{\ell}\frac{\operatorname{sh}{(\pm
m-\epsilon_{-})}\prod_{s=1}^{N}\operatorname{sh}{(\pm\phi_{I}\pm
a_{s}+m)}}{\operatorname{sh}{(\pm m-\epsilon_{+})}\operatorname{sh}{(\pm
2\phi_{I}\pm m-\epsilon_{+})}}$ (93)
$\displaystyle\cdot\prod_{I<J}^{\ell}\frac{\operatorname{sh}{(\pm\phi_{I}\pm\phi_{J}\pm
m-\epsilon_{-})}}{\operatorname{sh}{(\pm\phi_{I}\pm\phi_{J}\pm
m-\epsilon_{+})}}$ $\begin{split}&\mathcal{I}_{k=2\ell+1}^{-}\\\
=&\frac{1}{\operatorname{sh}{(\epsilon_{1,2})}\,\prod_{s=1}^{N}\operatorname{ch}{(\pm
a_{s}+\epsilon_{+})}}\cdot\prod_{I=1}^{\ell}\frac{\operatorname{ch}{(\pm\phi_{I})}\operatorname{ch}{(\pm\phi_{I}+2\epsilon_{+})}}{\operatorname{ch}{(\pm\phi_{I}+\epsilon_{1,2})}}\\\
&\cdot\prod_{I=1}^{\ell}\frac{\operatorname{sh}{(2\epsilon_{+})}}{\operatorname{sh}{(\epsilon_{1,2})}\operatorname{sh}{(\pm
2\phi_{I}+\epsilon_{1,2})}\,\prod_{s=1}^{N}\operatorname{sh}{(\pm\phi_{I}\pm
a_{s}+\epsilon_{+})}}\prod_{I<J}^{\ell}\frac{\operatorname{sh}{(\pm\phi_{I}\pm\phi_{J})}\operatorname{sh}{(\pm\phi_{I}\pm\phi_{J}+2\epsilon_{+})}}{\operatorname{sh}{(\pm\phi_{I}\pm\phi_{J}+\epsilon_{1,2})}}\\\
&\cdot\frac{\prod_{s=1}^{N}\operatorname{ch}{(m\pm
a_{s})}}{\operatorname{sh}{(\pm
m-\epsilon_{+})}}\cdot\prod_{I=1}^{\ell}\frac{\operatorname{ch}{(\pm\phi_{I}\pm
m-\epsilon_{-})}}{\operatorname{ch}{(\pm\phi_{I}\pm
m-\epsilon_{+})}}\frac{\operatorname{sh}{(\pm
m-\epsilon_{-})}\prod_{s=1}^{N}\operatorname{sh}{(\pm\phi_{I}\pm
a_{s}+m)}}{\operatorname{sh}{(\pm m-\epsilon_{+})}\operatorname{sh}{(\pm
2\phi_{I}\pm
m-\epsilon_{+})}}\cr&\cdot\prod_{I<J}^{\ell}\frac{\operatorname{sh}{(\pm\phi_{I}\pm\phi_{J}\pm
m-\epsilon_{-})}}{\operatorname{sh}{(\pm\phi_{I}\pm\phi_{J}\pm
m-\epsilon_{+})}}\end{split}$ (94) $\displaystyle\mathcal{I}_{k=2\ell}^{-}$
(95) $\displaystyle=$ $\displaystyle\frac{\operatorname{ch}{(\pm
m-\epsilon_{-})}\prod_{s=1}^{N}\operatorname{sh}{(2m\pm
2a_{s})}}{\operatorname{sh}{(\pm m-\epsilon_{+})}\,\operatorname{sh}{(\pm
2m-2\epsilon_{+})}}$ (96)
$\displaystyle\cdot\frac{\operatorname{ch}{(2\epsilon_{+})}}{\operatorname{sh}{(\epsilon_{1,2})}\,\operatorname{sh}{(2\epsilon_{1,2})}\,\prod_{s=1}^{N}\operatorname{sh}{(\pm
2a_{s}+2\epsilon_{+})}}\cdot\prod_{I=1}^{\ell-1}\frac{\operatorname{sh}{(\pm
2\phi_{I})}\operatorname{sh}{(\pm
2\phi_{I}+4\epsilon_{+})}}{\operatorname{sh}{(\pm
2\phi_{I}+2\epsilon_{1,2})}}$ (97)
$\displaystyle\cdot\prod_{I=1}^{\ell-1}\frac{\operatorname{sh}{(2\epsilon_{+})}}{\operatorname{sh}{(\epsilon_{1,2})}\operatorname{sh}{(\pm
2\phi_{I}+\epsilon_{1,2})}\,\prod_{s=1}^{N}\operatorname{sh}{(\pm\phi_{I}\pm
a_{s}+\epsilon_{+})}}\prod_{I<J}^{\ell-1}\frac{\operatorname{sh}{(\pm\phi_{I}\pm\phi_{J})}\operatorname{sh}{(\pm\phi_{I}\pm\phi_{J}+2\epsilon_{+})}}{\operatorname{sh}{(\pm\phi_{I}\pm\phi_{J}+\epsilon_{1,2})}}$
(98) $\displaystyle\cdot\prod_{I=1}^{\ell-1}\frac{\operatorname{sh}{(\pm
2\phi_{I}\pm 2m-2\epsilon_{-})}}{\operatorname{sh}{(\pm 2\phi_{I}\pm
2m-2\epsilon_{+})}}\frac{\operatorname{sh}{(\pm
m-\epsilon_{-})}\prod_{s=1}^{N}\operatorname{sh}{(\pm\phi_{I}\pm
a_{s}+m)}}{\operatorname{sh}{(\pm m-\epsilon_{+})}\operatorname{sh}{(\pm
2\phi_{I}\pm
m-\epsilon_{+})}}\cdot\prod_{I<J}^{\ell-1}\frac{\operatorname{sh}{(\pm\phi_{I}\pm\phi_{J}\pm
m-\epsilon_{-})}}{\operatorname{sh}{(\pm\phi_{I}\pm\phi_{J}\pm
m-\epsilon_{+})}}$

##  Appendix C Multiplicity coefficients

In this Appendix, we list some of the multiplicity coefficients
$C^{\textrm{4d}}_{\vec{\lambda},\vec{A}}$ in (58). The additional effective
Coulomb branch parameters that appear in (61) and (67)–(70) are

$a_{N+j}=\frac{\hbar}{2}(+\pi i)~{},\ \frac{m}{2}(+\pi i)~{},\
\frac{\hbar-m}{2}(+\pi i)~{},\ 0(+\pi i)~{},\ \hbar(+\pi i)~{},\ m(+\pi
i)~{}.$ (99)

In order to express the multiplicity coefficients
$C^{\textrm{4d}}_{\lambda,A}$ that appear in these parameters, we need to
introduce some notation for 4d Young diagrams. As in (56), the $(x_{1},x_{2})$
direction is associated with the equivariant parameter $\hbar$, while the
$(x_{3},x_{4})$ direction is associated with the equivariant parameter $m$. We
represent an $(x_{1},x_{2})$ Young sub-diagram by drawing it explicitly, and
we assign an $(x_{3},x_{4})$ Young sub-diagram to each box of the
$(x_{1},x_{2})$ Young sub-diagram by monotonically decreasing positive
integers $\mu=(\mu_{1},\mu_{2},\ldots)$. Note that we write the transposition
of a 2d Young diagram $\mu$ by $\mu^{t}$. For example, all the contents
$x=(x_{1},x_{2},x_{3},x_{4})\in\lambda$ of the following presentation of a 4d
Young diagram $\lambda$ are given by

$\displaystyle\raisebox{-7.11317pt}{\includegraphics[width=45.52458pt]{pictures/4dYT08}}\leadsto\quad$
$\displaystyle\\{(1,1,1,1),(1,2,1,1),(1,1,1,2)\\}$ (100)
$\displaystyle\raisebox{-7.11317pt}{\includegraphics[width=45.52458pt]{pictures/4dYT11}}\leadsto\quad$
$\displaystyle\\{(1,1,1,1),(1,2,1,1),(1,1,2,1)\\}$ (101)
$\displaystyle\raisebox{-7.11317pt}{\includegraphics[width=22.76228pt]{pictures/4dYT07}}\leadsto\quad$
$\displaystyle\\{(1,1,1,1),(1,1,1,2),(1,1,2,1)\\}$ (102)

Although we obtain multiplicity coefficients for 4d Young diagrams with total
4 boxes, they are quite complicated. Therefore, we provide a list of
multiplicity coefficients for 4d Young diagrams with up to total 3 boxes that
correspond to non-trivial residues. Note that residues are trivial upto
7-instanton for 4d Young diagrams that are not listed here. For 4 boxes, the
reader can refer to the Mathematica file provided on the arXiv page.

$\bullet$ $C^{\textrm{4d}}_{\lambda,A=\pm
q^{\frac{1}{2}}}\quad(a=\frac{\hbar}{2}(+\pi i)$):

$\displaystyle\raisebox{-7.11317pt}{\includegraphics[width=22.76228pt]{pictures/4dYT01}}\leadsto
2;\qquad\raisebox{-17.07182pt}{\includegraphics[width=22.76228pt]{pictures/4dYT05}}\leadsto
2;$
$\displaystyle\raisebox{-7.11317pt}{\includegraphics[width=22.76228pt]{pictures/4dYT07}}\leadsto
2;\qquad\raisebox{-7.11317pt}{\includegraphics[width=59.75095pt]{pictures/4dYT13}}\leadsto
2;\qquad\raisebox{-15.6491pt}{\includegraphics[width=39.83368pt]{pictures/4dYT14}}\leadsto
2;\qquad\raisebox{-25.6073pt}{\includegraphics[width=22.76228pt]{pictures/4dYT15}}\leadsto
2;$

$\bullet$ $C^{\textrm{4d}}_{\lambda,A=\pm
M^{\frac{1}{2}}}\quad(a=\frac{m}{2}(+\pi i)$):

$\displaystyle\raisebox{-7.11317pt}{\includegraphics[width=22.76228pt]{pictures/4dYT01}}\leadsto-2;\qquad\raisebox{-7.11317pt}{\includegraphics[width=22.76228pt]{pictures/4dYT03}}\leadsto
2;$
$\displaystyle\raisebox{-15.6491pt}{\includegraphics[width=39.83368pt]{pictures/4dYT14}}\leadsto-2;\qquad\raisebox{-7.11317pt}{\includegraphics[width=22.76228pt]{pictures/4dYT10}}\leadsto-2;\qquad\raisebox{-7.11317pt}{\includegraphics[width=22.76228pt]{pictures/4dYT06}}\leadsto-2;\qquad\raisebox{-7.11317pt}{\includegraphics[width=22.76228pt]{pictures/4dYT07}}\leadsto-2;$

$\bullet$
$C^{\textrm{4d}}_{\lambda,A=\pm\left(\frac{q}{M}\right)^{\frac{1}{2}}}\quad(a=\frac{\hbar-m}{2}(+i\pi)$):

$\displaystyle\raisebox{-7.11317pt}{\includegraphics[width=42.67912pt]{pictures/4dYT04}}\leadsto-1;\qquad\raisebox{-7.11317pt}{\includegraphics[width=22.76228pt]{pictures/4dYT03}}\leadsto-1;$

$\bullet$ $C^{\textrm{4d}}_{\lambda,A=\pm 1}\quad(a=0(+i\pi))$:

$\displaystyle\raisebox{-8.5359pt}{\includegraphics[width=42.67912pt]{pictures/4dYT04}}\leadsto
2;\qquad\raisebox{-17.07182pt}{\includegraphics[width=22.76228pt]{pictures/4dYT05}}\leadsto
2;\qquad\raisebox{-7.11317pt}{\includegraphics[width=22.76228pt]{pictures/4dYT03}}\leadsto
2;\qquad\raisebox{-7.11317pt}{\includegraphics[width=22.76228pt]{pictures/4dYT02}}\leadsto
2;$
$\displaystyle\raisebox{-7.11317pt}{\includegraphics[width=22.76228pt]{pictures/4dYT10}}\leadsto-2;\qquad\raisebox{-7.11317pt}{\includegraphics[width=22.76228pt]{pictures/4dYT06}}\leadsto-2;\qquad\raisebox{-7.11317pt}{\includegraphics[width=59.75095pt]{pictures/4dYT13}}\leadsto
2;\qquad\raisebox{-28.45274pt}{\includegraphics[width=22.76228pt]{pictures/4dYT15}}\leadsto
2;$
$\displaystyle\raisebox{-8.5359pt}{\includegraphics[width=42.67912pt]{pictures/4dYT11}}\leadsto-1;\qquad\raisebox{-7.11317pt}{\includegraphics[width=42.67912pt]{pictures/4dYT08}}\leadsto-1;\qquad\raisebox{-17.07182pt}{\includegraphics[width=22.76228pt]{pictures/4dYT12}}\leadsto-1;\qquad\raisebox{-17.07182pt}{\includegraphics[width=22.76228pt]{pictures/4dYT09}}\leadsto-1;$

$\bullet$ $C^{\textrm{4d}}_{\lambda,A=\pm q}\quad(a=\hbar(+i\pi))$:

$\displaystyle\raisebox{-7.11317pt}{\includegraphics[width=22.76228pt]{pictures/4dYT01}}\leadsto
1;\qquad\raisebox{-17.07182pt}{\includegraphics[width=22.76228pt]{pictures/4dYT05}}\leadsto
1;\qquad\raisebox{-8.5359pt}{\includegraphics[width=42.67912pt]{pictures/4dYT04}}\leadsto
2;$
$\displaystyle\raisebox{-28.45274pt}{\includegraphics[width=22.76228pt]{pictures/4dYT15}}\leadsto
1;\qquad\raisebox{-7.11317pt}{\includegraphics[width=22.76228pt]{pictures/4dYT07}}\leadsto
2;\qquad\raisebox{-15.6491pt}{\includegraphics[width=39.83368pt]{pictures/4dYT14}}\leadsto
2;$

$\bullet$ $C^{\textrm{4d}}_{\lambda,A=\pm M}\quad(a=m(+i\pi))$:

$\displaystyle\raisebox{-7.11317pt}{\includegraphics[width=22.76228pt]{pictures/4dYT01}}\leadsto-1;\qquad\raisebox{-7.11317pt}{\includegraphics[width=22.76228pt]{pictures/4dYT03}}\leadsto
1;\qquad\raisebox{-7.11317pt}{\includegraphics[width=22.76228pt]{pictures/4dYT02}}\leadsto-2;\qquad\raisebox{-7.11317pt}{\includegraphics[width=22.76228pt]{pictures/4dYT10}}\leadsto-1;\qquad\raisebox{-7.11317pt}{\includegraphics[width=22.76228pt]{pictures/4dYT07}}\leadsto
2;$

##  Appendix D Gopakumar-Vafa invariants for $D$-type singularities

In this appendix, we will list the Gopakumar-Vafa (GV) invariants
Gopakumar:1998ii ; Gopakumar:1998jq  from $\operatorname{SO}(2N)$ instanton
partition functions.

Compactification of M-theory on a Calabi-Yau manifold leads to a five-
dimensional (5d) supersymmetric theory with eight supercharges, known as
_geometric engineering_ Katz:1996fh ; Katz:1997eq . In particular, when a
Calabi-Yau manifold is a bundle in which an ALE space (a blowup of an
$ADE$-orbifold singularity $\mathbb{C}^{2}/\Gamma$) is fibered over
$\boldsymbol{P}^{1}$, it gives rise to 5d pure Yang-Mills theory for a gauge
group of $ADE$ type. In this construction, $W$-bosons arise as M2-branes
wrapped on two-cycles in the ALE space, while instantons appear by wrapping
M2-branes on the base $\boldsymbol{P}^{1}$.

Through geometric engineering, instantons are related to the Gopakumar-Vafa
(GV) invariants Gopakumar:1998ii ; Gopakumar:1998jq  for a Calabi-Yau
manifold. These invariants are graded dimensions of BPS states that arise from
M2-branes wrapping two-cycles of a Calabi-Yau threefold in M-theory, and they
can be defined using techniques in algebraic geometry pandharipande2010stable
; Maulik:2016rip . As a result, the GV invariants are a set of integers, and
the closed topological string partition function for a Calabi-Yau threefold
becomes the GV generating function. Therefore, geometric engineering relates
the GV invariants to instantons in the 5d supersymmetric theory.

The relationship between the closed topological string partition function and
the instanton partition functions has been studied extensively in the case of
gauge groups of $A$ type Iqbal:2003ix ; Iqbal:2003zz ; Eguchi:2003sj . Since
the $\operatorname{SO}(2N)$ instanton partition function has been obtained in
Nawata:2021dlk , we can read off GV invariants of the ALE space with $D$-type
singularity fibered over $\boldsymbol{P}^{1}$ from it.

To understand this relationship, we first recall the GV expansion of the free
energy of a closed topological string amplitude, which is given by spins as
follows:

$F=\sum_{C\in
H_{2}(X,\mathbb{Z})}\sum_{d=1}^{\infty}\sum_{j_{L}}(-1)^{2j_{L}}n_{C}^{j_{L}}e^{-kT_{C}}\left(\frac{q^{-2j_{L}d}+\cdots+q^{+2j_{L}d}}{d\left(q^{d/2}-q^{-d/2}\right)^{2}}\right),\quad
q=e^{ig_{s}}.$ (103)

Here, $n_{C}\in\mathbb{Z}$ are invariants associated to the two-cycle $C\in
H_{2}(X,\mathbb{Z})$ of a Calabi-Yau manifold $X$. This partition function has
the following infinite product form:

$\displaystyle Z\left(\omega,g_{s}\right)$ $\displaystyle=\exp(F)\ $
$\displaystyle=\prod_{C\in
H_{2}(X,\mathbb{Z})}\prod_{j_{L}}\prod_{p=-j_{L}}^{+j_{L}}\prod_{m=0}^{\infty}\left(1-q^{2p+m+1}Q^{C}\right)^{(-1)^{2j_{L}+1}(m+1)n_{C}^{j_{L}}}~{},$
(104)

where $Q^{C}=e^{-\int_{C}\omega}$ is the Kähler parameter associated to the
holomorphic curve $C$ in $H_{2}(X,\mathbb{Z})$.

###  D.1 $\operatorname{SO}(4)$

The Kähler parameters for $\operatorname{SO}(4)$ instanton partition function
are $Q$ and $Q_{1,2}$. Their relations with the Coulomb branch parameters are

$A_{s}:=\prod_{j=1}^{s}Q_{j}.$ (105)

We can choose such a basis that $Q=\mathfrak{q}$ and $k,a,b\in\mathbb{Z}_{+}$,
then $k$ is identified with instanton number

$Z_{\rm
inst.}=\prod_{k,a,b\in\mathbb{Z}_{+}}\prod_{j_{L}}\prod_{p=-j_{L}}^{+j_{L}}\prod_{m=0}^{\infty}\left(1-q^{2p+m+1}Q^{k}Q_{1}^{a}Q_{2}^{b}\right)^{(-1)^{2j_{L}+1}(m+1)n_{k,a,b}^{j_{L}}}$
(106)

The GV invariants at $k=1$ can be expressed as follows:

$\displaystyle n^{0}_{1,0,0}$ $\displaystyle=1,$ (107) $\displaystyle
n^{0}_{1,0,p}$ $\displaystyle=n^{0}_{1,2p,p}=2p.$

For the $k=2$ case, we have read off the GV invariants from the instanton
partition function up to $Q_{1}^{14}$ and $Q_{2}^{7}$. These results are
summarized in Table 1.

$j_{L}$ | $a,b$ | $n$ | $a,b$ | $n$ | $j_{L}$ | $a,b$ | $n$ | $a,b$ | $n$
---|---|---|---|---|---|---|---|---|---
2 | 0,7 | 18 | 14,7 | 18 |  1/2 | 0,6 | -46 | 12,6 | -46
3/2 | 0,6 | -15 | 12,6 | -15 | 0,7 | -76 | 14,7 | -76
0,7 | -32 | 14,7 | -32 |  0 | 0,3 | 6 | 6,3 | 6
1 | 0,5 | 12 | 10,5 | 12 | 0,4 | 14 | 8,4 | 14
0,6 | 26 | 12,6 | 26 | 0,5 | 34 | 10,5 | 34
0,7 | 58 | 14,7 | 58 | 0,6 | 58 | 12,6 | 58
1/2 | 0,5 | -20 | 10,5 | -20 | 0,7 | 100 | 14,7 | 100
Table 1: GV invariants $n_{2,a,b}^{j_{L}}$ for $\operatorname{SO}(4)$, $k=2$
case.

From Table 1, we can see that the GV invariants of $\operatorname{SO}(4)$ have
a symmetry

$n^{j_{L}}_{k,0,p}=n^{j_{L}}_{k,2p,p}$ (108)

for $k=1,2$. We conjecture that this relation holds for arbitrary $k$.

#### Relation with $\operatorname{SU}(2)$

The isomorphism $\mathfrak{so}(4)=\mathfrak{su}(2)\oplus\mathfrak{su}(2)$
leads to

$Z_{\mathrm{SO}(4)}=Z_{\mathrm{SU}(2)}\left(Q_{1}\to
Q_{1}Q^{1/2}_{2}\right)Z_{\mathrm{SU}(2)}\left(Q_{1}\to
Q^{1/2}_{2}\right)Z_{\mathrm{U}(1)}\text{, }$ (109)

Recall that

$\displaystyle Z_{\operatorname{SO}(4)}=$
$\displaystyle\exp[\sum_{d=1}^{\infty}\sum_{k,a,b\in\mathbb{Z}}\sum_{j_{L}}\frac{n^{j_{L}}_{k,a,b}Q^{dk}Q_{1}^{da}Q_{2}^{db}(q^{-2j_{L}d}+...+q^{2j_{L}d})}{d(q^{d/2}-q^{-d/2})^{2}}\Big{]}$
(110) $\displaystyle Z_{\operatorname{SU}(2)}=$
$\displaystyle\exp[\sum_{d=1}^{\infty}\sum_{k,a\in\mathbb{Z}}\sum_{j_{L}}\frac{n^{j_{L}}_{k,a}Q^{dk}Q_{1}^{2d(k+a)}(q^{-2j_{L}d}+...+q^{2j_{L}d})}{d(q^{d/2}-q^{-d/2})^{2}}\Big{]}$
$\displaystyle Z_{\operatorname{U}(1)}=$
$\displaystyle\exp[\sum_{d=1}^{\infty}\sum_{k\in\mathbb{Z}}\sum_{j_{L}}\frac{n^{j_{L}}_{k}Q^{dk}(q^{-2j_{L}d}+...+q^{2j_{L}d})}{d(q^{d/2}-q^{-d/2})^{2}}\Big{]}$

Substituting them into (109), we can identify the relations between GV
invariants of $\operatorname{SO}(4)$ and $\operatorname{SU}(2)$ (also
$\operatorname{U}(1)$) as

$\displaystyle n^{\operatorname{SO}(4)}_{j_{L},k,0,p}=$ $\displaystyle
n^{\operatorname{SO}(4)}_{j_{L},k,2p,p}=n^{\operatorname{SU}(2)}_{j_{L},k,p-k}$
(111) $\displaystyle n^{\operatorname{SO}(4)}_{j_{L},k,0,0}=$ $\displaystyle
n^{\operatorname{U}(1)}_{j_{L},k}$

where GV invariants of $\operatorname{U}(1)$ are just
$n^{\operatorname{U}(1)}_{0,1}=1$ and all others are $0$.

###  D.2 $\operatorname{SO}(6)$

The Kähler parameters are now $Q$ and $Q_{1,2,3}$.

$Z^{\operatorname{SO}(6)}_{\rm
inst.}=\prod_{k,a,b,c\in\mathbb{Z}_{+}}\prod_{j_{L}}\prod_{p=-j_{L}}^{+j_{L}}\prod_{m=0}^{\infty}\left(1-q^{2p+m+1}Q^{k}Q_{1}^{a}Q_{2}^{b}Q_{3}^{c}\right)^{(-1)^{2j_{L}+1}(m+1)n_{k,a,b,c}^{j_{L}}}.$
(112)

For the $k=1$ case, we computed the GV invariants up to $Q_{1}^{6}$ and
$Q_{2,3}^{3}$. The results are summarized in Table 2.

$j_{L}$ | $a,b,c$ | $n$ | $a,b,c$ | $n$ | $a,b,c$ | $n$ | $a,b,c$ | $n$
---|---|---|---|---|---|---|---|---
0 | 0,0,0 | 2 | 2,2,1 | 8 | 6,3,1 | 10 | 0,1,3 | 14
0,1,0 | 2 | 4,2,1 | 6 | 2,1,2 | 10 | 2,1,3 | 14
2,1,0 | 2 | 0,0,2 | 6 | 0,2,2 | 12 | 0,2,3 | 18
0,2,0 | 4 | 0,1,2 | 10 | 2,2,2 | 16 | 2,2,3 | 24
4,2,0 | 4 | 0,3,0 | 6 | 4,2,2 | 12 | 4,2,3 | 18
0,0,1 | 4 | 6,3,0 | 6 | 0,3,2 | 12 | 0,3,3 | 20
0,1,1 | 6 | 0,3,1 | 10 | 2,3,2 | 18 | 2,3,3 | 30
2,1,1 | 6 | 2,3,1 | 6 | 4,3,2 | 18 | 4,3,3 | 30
0,2,1 | 6 | 4,3,1 | 6 | 6,3,2 | 12 | 6,3,3 | 20
Table 2: GV invariants $n_{1,a,b,c}^{j_{L}}$ for $\operatorname{SO}(6)$, $k=1$
case.

#### Relation with $\operatorname{SU}(4)$

The isomorphism $\mathfrak{so}(6)=\mathfrak{su}(4)$ leads to

$Z_{\mathrm{SO}(6)}\left(A_{1},A_{2},A_{3}\right)=Z_{\mathrm{SU}(4)}\left(A_{1},A_{2},A_{3}\right).$
(113)

And given

$\displaystyle Z_{\operatorname{SO}(6)}=$
$\displaystyle\exp[\sum_{d=1}^{\infty}\sum_{k,a,b,c\in\mathbb{Z}}\sum_{j_{L}}\frac{n^{j_{L}}_{k,a,b,c}Q^{dk}Q_{1}^{da}Q_{2}^{db}Q_{3}^{dc}(q^{-2j_{L}d}+...+q^{2j_{L}d})}{d(q^{d/2}-q^{-d/2})^{2}}\Big{]}$
(114) $\displaystyle Z_{\operatorname{SU}(4)}=$
$\displaystyle\exp[\sum_{d=1}^{\infty}\sum_{k,a,b,c\in\mathbb{Z}}\sum_{j_{L}}\frac{n^{j_{L}}_{k,a,b,c}Q^{dk}Q_{1}^{2da}Q_{2}^{d(a+c)}Q_{3}^{db}(q^{-2j_{L}d}+...+q^{2j_{L}d})}{d(q^{d/2}-q^{-d/2})^{2}}\Big{]}$

We can conclude that by simple changes of variables, the GV invariants of
$\operatorname{SO}(6)$ are related to those of $\operatorname{SU}(4)$ via

$n^{\operatorname{SU}(4)}_{j_{L},k,a,b,c}=n^{\operatorname{SO}(6)}_{j_{L},k,2a,a+c,b}\Longleftrightarrow
n^{\operatorname{SO}(6)}_{j_{L},k,a,b,c}=n^{\operatorname{SU}(4)}_{j_{L},k,a/2,c,b-a/2}$
(115)

###  D.3 $\operatorname{SO}(8)$

The first non-trivial case is the $D_{4}$ gauge group. The Kähler parameters
are $Q$ and $Q_{1,2,3,4}$.

$Z_{\rm
inst.}=\prod_{k,a,b,c,d\in\mathbb{Z}_{+}}\prod_{j_{L}}\prod_{p=-j_{L}}^{+j_{L}}\prod_{m=0}^{\infty}\left(1-q^{2p+m+1}Q^{k}Q_{1}^{a}Q_{2}^{b}Q_{3}^{c}Q_{4}^{d}\right)^{(-1)^{2j_{L}+1}(m+1)n_{k,a,b,c,d}^{j_{L}}}$
(116)

For the $k=1$ case, we computed the GV invariants up to $Q_{1}^{4}$ and
$Q_{2,3,4}^{2}$. The results are summarized in Table 3.

$j_{L}$ | $a,b,c,d$ | $n$ | $a,b,c,d$ | $n$ | $a,b,c,d$ | $n$ | $a,b,c,d$ | $n$
---|---|---|---|---|---|---|---|---
0 | 0,0,0,0 | 2 | 0,0,2,0 | 6 | 2,2,1,1 | 10 | 2,1,1,2 | 6
0,1,0,0 | 2 | 0,1,2,0 | 10 | 4,2,1,1 | 6 | 2,2,1,2 | 6
2,1,0,0 | 2 | 2,1,2,0 | 10 | 0,0,2,1 | 10 | 0,0,2,2 | 12
0,2,0,0 | 4 | 0,2,2,0 | 12 | 0,1,2,1 | 16 | 0,1,2,2 | 18
4,2,0,0 | 4 | 2,2,2,0 | 16 | 2,1,2,1 | 16 | 2,1,2,2 | 18
0,0,1,0 | 4 | 4,2,2,0 | 12 | 0,2,2,1 | 18 | 0,2,2,2 | 18
0,1,1,0 | 6 | 0,0,0,1 | 2 | 2,2,2,1 | 30 | 4,2,2,2 | 18
2,1,1,0 | 6 | 0,0,1,1 | 6 | 4,2,2,1 | 18 | 2,2,2,2 | 32
0,2,1,0 | 6 | 0,1,1,1 | 8 | 0,0,0,2 | 4 |  |
2,2,1,0 | 8 | 2,1,1,1 | 8 | 0,0,1,2 | 6 |  |
4,2,1,0 | 6 | 0,2,1,1 | 6 | 0,1,1,2 | 6 |  |
Table 3: GV invariants $n_{1,a,b,c,d}^{j_{L}}$ for $\operatorname{SO}(8)$,
$k=1$ case.

## References

  * (1) N. Seiberg and E. Witten, _Monopoles, duality and chiral symmetry breaking in N=2 supersymmetric QCD_ , _Nucl. Phys. B_ 431 (1994) 484 [hep-th/9408099].
  * (2) N. Seiberg and E. Witten, _Electric - magnetic duality, monopole condensation, and confinement in N=2 supersymmetric Yang-Mills theory_ , _Nucl. Phys. B_ 426 (1994) 19 [hep-th/9407087].
  * (3) N.A. Nekrasov, _Seiberg-Witten prepotential from instanton counting_ , _Adv. Theor. Math. Phys._ 7 (2003) 831 [hep-th/0206161].
  * (4) M. Aganagic, A. Klemm, M. Mariño and C. Vafa, _The Topological Vertex_ , _Commun. Math. Phys._ 254 (2005) 425 [hep-th/0305132].
  * (5) H. Awata and H. Kanno, _Instanton counting, Macdonald functions and the moduli space of D-branes_ , _JHEP_ 05 (2005) 039 [hep-th/0502061].
  * (6) A. Iqbal, C. Kozcaz and C. Vafa, _The Refined topological vertex_ , _JHEP_ 10 (2009) 069 [hep-th/0701156].
  * (7) L.F. Alday, D. Gaiotto and Y. Tachikawa, _Liouville Correlation Functions from Four-dimensional Gauge Theories_ , _Lett.Math.Phys._ 91 (2010) 167 [0906.3219].
  * (8) S. Nawata and R.-D. Zhu, _Instanton counting and O-vertex_ , _JHEP_ 09 (2021) 190 [2107.03656].
  * (9) H. Hayashi and R.-D. Zhu, _More on topological vertex formalism for 5-brane webs with O5-plane_ , _JHEP_ 04 (2021) 292 [2012.13303].
  * (10) N. Nekrasov and S. Shadchin, _ABCD of instantons_ , _Commun. Math. Phys._ 252 (2004) 359 [hep-th/0404225].
  * (11) S. Shadchin, _On certain aspects of string theory/gauge theory correspondence_ , Ph.D. thesis, 2, 2005, [hep-th/0502180].
  * (12) H.-C. Kim, S.-S. Kim and K. Lee, _5-dim Superconformal Index with Enhanced $E_{n}$ Global Symmetry_, _JHEP_ 10 (2012) 142 [1206.6781].
  * (13) H.-C. Kim, J. Kim and S. Kim, _Instantons on the 5-sphere and M5-branes_ , 1211.0144.
  * (14) C. Hwang, J. Kim, S. Kim and J. Park, _General instanton counting and 5d SCFT_ , _JHEP_ 07 (2015) 063 [1406.6793].
  * (15) O. Bergman and G. Zafrir, _Lifting 4d dualities to 5d_ , _JHEP_ 04 (2015) 141 [1410.2806].
  * (16) O. Bergman and G. Zafrir, _5d fixed points from brane webs and O7-planes_ , _JHEP_ 12 (2015) 163 [1507.03860].
  * (17) G. Zafrir, _Brane webs and $O5$-planes_, _JHEP_ 03 (2016) 109 [1512.08114].
  * (18) H. Hayashi and G. Zoccarato, _Partition functions of web diagrams with an O7 --plane_, _JHEP_ 03 (2017) 112 [1609.07381].
  * (19) S.-S. Kim and F. Yagi, _Topological vertex formalism with O5-plane_ , _Phys. Rev._ D97 (2018) 026011 [1709.01928].
  * (20) H.-C. Kim, J. Kim, S. Kim, K.-H. Lee and J. Park, _6d strings and exceptional instantons_ , _Phys. Rev. D_ 103 (2021) 025012 [1801.03579].
  * (21) H. Hayashi, S.-S. Kim, K. Lee and F. Yagi, _5-brane webs for 5d $\mathcal{N}$ = 1 G2 gauge theories_, _JHEP_ 03 (2018) 125 [1801.03916].
  * (22) H. Hayashi, S.-S. Kim, K. Lee and F. Yagi, _Dualities and 5-brane webs for 5d rank 2 SCFTs_ , _JHEP_ 12 (2018) 016 [1806.10569].
  * (23) S. Cheng and S.-S. Kim, _Refined topological vertex for a 5D Sp(N) gauge theories with antisymmetric matter_ , _Phys. Rev. D_ 104 (2021) 086004 [1809.00629].
  * (24) H. Hayashi, S.-S. Kim, K. Lee and F. Yagi, _Rank-3 antisymmetric matter on 5-brane webs_ , _JHEP_ 05 (2019) 133 [1902.04754].
  * (25) H.-C. Kim, M. Kim and S.-S. Kim, _Topological vertex for 6d SCFTs with $\mathbb{Z}_{2}$-twist_, _JHEP_ 03 (2021) 132 [2101.01030].
  * (26) S.-S. Kim and X.-Y. Wei, _Refined topological vertex with ON-planes_ , _JHEP_ 08 (2022) 006 [2201.12264].
  * (27) A.S. Losev, A. Marshakov and N.A. Nekrasov, _Small instantons, little strings and free fermions_ , in _From Fields to Strings: Circumnavigating Theoretical Physics: A Conference in Tribute to Ian Kogan_ , pp. 581–621, 2, 2003 [hep-th/0302191].
  * (28) L.C. Jeffrey and F.C. Kirwan, _Localization for nonabelian group actions_ , _Topology_ 34 (1995) 291 [alg-geom/9307001].
  * (29) F. Benini, R. Eager, K. Hori and Y. Tachikawa, _Elliptic Genera of 2d ${\mathcal{N}}$ = 2 Gauge Theories_, _Commun. Math. Phys._ 333 (2015) 1241 [1308.4896].
  * (30) S.-S. Kim, X. Li, S. Nawata and F. Yagi, _Higgsing and tuning D-branes with O-planes_ , _Work in progress_ .
  * (31) J. Kim, S.-S. Kim, K.-H. Lee, K. Lee and J. Song, _Instantons from Blow-up_ , _JHEP_ 11 (2019) 092 [1908.11276].
  * (32) A. Sen, _F theory and orientifolds_ , _Nucl. Phys. B_ 475 (1996) 562 [hep-th/9605150].
  * (33) A. Hanany and E. Witten, _Type IIB superstrings, BPS monopoles, and three-dimensional gauge dynamics_ , _Nucl. Phys. B_ 492 (1997) 152 [hep-th/9611230].
  * (34) R. Gopakumar and C. Vafa, _M theory and topological strings. 1._ , hep-th/9809187.
  * (35) R. Gopakumar and C. Vafa, _M theory and topological strings. 2._ , hep-th/9812127.
  * (36) S.H. Katz, A. Klemm and C. Vafa, _Geometric engineering of quantum field theories_ , _Nucl. Phys. B_ 497 (1997) 173 [hep-th/9609239].
  * (37) S. Katz, P. Mayr and C. Vafa, _Mirror symmetry and exact solution of 4-D N=2 gauge theories: 1._ , _Adv. Theor. Math. Phys._ 1 (1998) 53 [hep-th/9706110].
  * (38) R. Pandharipande and R. Thomas, _Stable pairs and BPS invariants_ , _Journal of the American Mathematical Society_ 23 (2010) 267 [0711.3899].
  * (39) D. Maulik and Y. Toda, _Gopakumar-Vafa invariants via vanishing cycles_ , _Inventiones mathematicae_ 213 (2018) 1017 [1610.07303].
  * (40) A. Iqbal and A.-K. Kashani-Poor, _Instanton counting and Chern-Simons theory_ , _Adv. Theor. Math. Phys._ 7 (2003) 457 [hep-th/0212279].
  * (41) A. Iqbal and A.-K. Kashani-Poor, _SU(N) geometries and topological string amplitudes_ , _Adv. Theor. Math. Phys._ 10 (2006) 1 [hep-th/0306032].
  * (42) T. Eguchi and H. Kanno, _Topological strings and Nekrasov’s formulas_ , _JHEP_ 12 (2003) 006 [hep-th/0310235].