# This is the title

p-ADIC WELCH BOUNDS AND p-ADIC ZAUNER CONJECTURE
K. MAHESH KRISHNA
Post Doctoral Fellow

Statistics and Mathematics Unit

Indian Statistical Institute, Bangalore Centre

Karnataka 560 059, India

Email<EMAIL_ADDRESS>
Date:

Abstract: Let $p$ be a prime. For $d\in\mathbb{N}$, let $\mathbb{Q}_{p}^{d}$
be the standard $d$-dimensional p-adic Hilbert space. Let $m\in\mathbb{N}$ and
$\text{Sym}^{m}(\mathbb{Q}_{p}^{d})$ be the p-adic Hilbert space of symmetric
m-tensors. We prove the following result. Let $\\{\tau_{j}\\}_{j=1}^{n}$ be a
collection in $\mathbb{Q}_{p}^{d}$ satisfying (i)
$\langle\tau_{j},\tau_{j}\rangle=1$ for all $1\leq j\leq n$ and (ii) there
exists $b\in\mathbb{Q}_{p}$ satisfying $\sum_{j=1}^{n}\langle
x,\tau_{j}\rangle\tau_{j}=bx$ for all $x\in\mathbb{Q}^{d}_{p}.$ Then

(1) $\displaystyle\max_{1\leq j,k\leq n,j\neq
k}\\{|n|,|\langle\tau_{j},\tau_{k}\rangle|^{2m}\\}\geq\frac{|n|^{2}}{\left|{d+m-1\choose
m}\right|}.$

We call Inequality (1) as the p-adic version of Welch bounds obtained by Welch
[IEEE Transactions on Information Theory, 1974]. Inequality (1) differs from
the non-Archimedean Welch bound obtained recently by M. Krishna as one can not
derive one from another. We formulate p-adic Zauner conjecture.

Keywords: p-adic number field, p-adic Hilbert space, Welch bound, Zauner
conjecture.

Mathematics Subject Classification (2020): 12J25, 46S10, 47S10, 11D88.

##  1\. Introduction

In 1974 Prof. L. Welch proved the following result [86].

######  Theorem 1.1.

[86] (Welch Bounds) Let $n>d$. If $\\{\tau_{j}\\}_{j=1}^{n}$ is any collection
of unit vectors in $\mathbb{C}^{d}$, then

$\displaystyle\sum_{1\leq j,k\leq
n}|\langle\tau_{j},\tau_{k}\rangle|^{2m}=\sum_{j=1}^{n}\sum_{k=1}^{n}|\langle\tau_{j},\tau_{k}\rangle|^{2m}\geq\frac{n^{2}}{{d+m-1\choose
m}},\quad\forall m\in\mathbb{N}.$

In particular,

$\displaystyle\sum_{1\leq j,k\leq
n}|\langle\tau_{j},\tau_{k}\rangle|^{2}=\sum_{j=1}^{n}\sum_{k=1}^{n}|\langle\tau_{j},\tau_{k}\rangle|^{2}\geq\frac{n^{2}}{{d}}.$

Further,

$\displaystyle\text{({Higher order Welch bounds})}\quad\max_{1\leq j,k\leq
n,j\neq
k}|\langle\tau_{j},\tau_{k}\rangle|^{2m}\geq\frac{1}{n-1}\left[\frac{n}{{d+m-1\choose
m}}-1\right],\quad\forall m\in\mathbb{N}.$

In particular,

$\displaystyle\text{({First order Welch bound})}\quad\max_{1\leq j,k\leq
n,j\neq k}|\langle\tau_{j},\tau_{k}\rangle|^{2}\geq\frac{n-d}{d(n-1)}.$

It is impossible to list all applications of Theorem 1.1. A few are: in the
study of root-mean-square (RMS) absolute cross relation of unit vectors [73],
frame potential [10, 19, 15], correlations [72], codebooks [30], numerical
search algorithms [88, 87], quantum measurements [75], coding and
communications [82, 78], code division multiple access (CDMA) systems [54,
55], wireless systems [70], compressed/compressive sensing [80, 83, 35, 32, 7,
81, 74, 2], ‘game of Sloanes’ [48], equiangular tight frames [79], equiangular
lines [63, 24, 34, 47], digital fingerprinting [62] etc.

Theorem 1.1 has been improved/different proofs were given in [20, 26, 85, 84,
25, 71, 45, 31, 78]. In 2021 M. Krishna derived continuous version of Theorem
1.1 [56]. In 2022 M. Krishna obtained Theorem 1.1 for Hilbert C*-modules [58],
Banach spaces [57] and non-Archimedean Hilbert spaces [59].

In this paper we derive p-adic Welch bounds (Theorem 2.3). We formulate p-adic
Zauner conjecture (Conjecture 3.3).

##  2\. p-adic Welch bounds

Let $p$ be a prime. For $d\in\mathbb{N}$, let $\mathbb{Q}_{p}^{d}$ be the
standard p-adic Hilbert space equipped with the inner product

$\displaystyle\langle(a_{j})_{j=1}^{d},(b_{j})_{j=1}^{d}\rangle\coloneqq\sum_{j=1}^{d}a_{j}b_{j},\quad\forall(a_{j})_{j=1}^{d},(b_{j})_{j=1}^{d}\in\mathbb{Q}_{p}^{d}.$

Let $I_{\mathbb{Q}_{p}^{d}}$ be the identity operator on $\mathbb{Q}_{p}^{d}$.
Note that $\mathbb{Q}_{p}^{d}$ is not a non-Archimedean Hilbert space as it
does not satisfies Equation (2) in [59] (see Page 40, [67]). We refer [51, 28,
29, 52, 1] for more on p-adic Hilbert spaces.

######  Theorem 2.1.

(First Order p-adic Welch Bound) Let $p$ be a prime and $n,d\in\mathbb{N}$. If
$\\{\tau_{j}\\}_{j=1}^{n}$ is any collection in $\mathbb{Q}_{p}^{d}$ such that
there exists $b\in\mathbb{Q}_{p}$ satisfying

$\displaystyle\sum_{j=1}^{n}\langle x,\tau_{j}\rangle\tau_{j}=bx,\quad\forall
x\in\mathbb{Q}^{d}_{p},$

then

$\displaystyle\max_{1\leq j,k\leq n,j\neq
k}\left\\{\left|\sum_{l=1}^{n}\langle\tau_{l},\tau_{l}\rangle^{2}\right|,|\langle\tau_{j},\tau_{k}\rangle|^{2}\right\\}\geq\frac{1}{|d|}\left|\sum_{j=1}^{n}\langle\tau_{j},\tau_{j}\rangle\right|^{2}.$

In particular, if $\langle\tau_{j},\tau_{j}\rangle=1$ for all $1\leq j\leq n$,
then

$\displaystyle\text{{(First order p-adic Welch bound)}}\quad\max_{1\leq
j,k\leq n,j\neq
k}\\{|n|,|\langle\tau_{j},\tau_{k}\rangle|^{2}\\}\geq\frac{|n|^{2}}{|d|}.$

###### Proof.

Define $S_{\tau}:\mathbb{Q}_{p}^{d}\ni x\mapsto\sum_{j=1}^{n}\langle
x,\tau_{j}\rangle\tau_{j}\in\mathbb{Q}_{p}^{d}$. Then

$\displaystyle
bd=\operatorname{Tra}(bI_{\mathbb{Q}_{p}^{d}})=\operatorname{Tra}(S_{\tau})=\sum_{j=1}^{n}\langle\tau_{j},\tau_{j}\rangle,$
$\displaystyle
b^{2}d=\operatorname{Tra}(b^{2}I_{\mathbb{Q}_{p}^{d}})=\operatorname{Tra}(S^{2}_{\tau})=\sum_{j=1}^{n}\sum_{k=1}^{n}\langle\tau_{j},\tau_{k}\rangle\langle\tau_{k},\tau_{j}\rangle.$

Therefore

$\displaystyle\left|\sum_{j=1}^{n}\langle\tau_{j},\tau_{j}\rangle\right|^{2}$
$\displaystyle=|\operatorname{Tra}(S_{\tau})|^{2}=|bd|^{2}=|d||b^{2}d|=|d|\left|\sum_{j=1}^{n}\sum_{k=1}^{n}\langle\tau_{j},\tau_{k}\rangle\langle\tau_{k},\tau_{j}\rangle\right|$
$\displaystyle=|d|\left|\sum_{l=1}^{n}\langle\tau_{l},\tau_{l}\rangle^{2}+\sum_{j,k=1,j\neq
k}^{n}\langle\tau_{j},\tau_{k}\rangle\langle\tau_{k},\tau_{j}\rangle\right|$
$\displaystyle\leq|d|\max_{1\leq j,k\leq n,j\neq
k}\left\\{\left|\sum_{l=1}^{n}\langle\tau_{l},\tau_{l}\rangle^{2}\right|,|\langle\tau_{j},\tau_{k}\rangle\langle\tau_{k},\tau_{j}\rangle|\right\\}$
$\displaystyle=|d|\max_{1\leq j,k\leq n,j\neq
k}\left\\{\left|\sum_{l=1}^{n}\langle\tau_{l},\tau_{l}\rangle^{2}\right|,|\langle\tau_{j},\tau_{k}\rangle|^{2}\right\\}.$

Whenever $\langle\tau_{j},\tau_{j}\rangle=1$ for all $1\leq j\leq n$,

$\displaystyle|n|^{2}\leq|d|\max_{1\leq j,k\leq n,j\neq
k}\\{|n|,|\langle\tau_{j},\tau_{k}\rangle|^{2}\\}.$

∎

We next derive higher order p-adic Welch bounds. For this, we need the
following result.

######  Theorem 2.2.

[22, 14] If $\mathcal{V}$ is a vector space of dimension $d$ and
$\text{Sym}^{m}(\mathcal{V})$ denotes the vector space of symmetric m-tensors,
then

$\displaystyle\text{dim}(\text{Sym}^{m}(\mathcal{V}))={d+m-1\choose
m},\quad\forall m\in\mathbb{N}.$

######  Theorem 2.3.

(Higher Order p-adic Welch Bounds) Let $p$ be a prime and
$n,d,m\in\mathbb{N}$. If $\\{\tau_{j}\\}_{j=1}^{n}$ is any collection in
$\mathbb{Q}_{p}^{d}$ such that there exists $b\in\mathbb{Q}_{p}$ satisfying

$\displaystyle\sum_{j=1}^{n}\langle x,\tau_{j}^{\otimes
m}\rangle\tau_{j}^{\otimes m}=bx,\quad\forall
x\in\text{Sym}^{m}(\mathbb{Q}_{p}^{d}),$

then

$\displaystyle\max_{1\leq j,k\leq n,j\neq
k}\left\\{\left|\sum_{l=1}^{n}\langle\tau_{l},\tau_{l}\rangle^{2m}\right|,|\langle\tau_{j},\tau_{k}\rangle|^{2m}\right\\}\geq\frac{1}{\left|{d+m-1\choose
m}\right|}\left|\sum_{j=1}^{n}\langle\tau_{j},\tau_{j}\rangle^{m}\right|^{2}.$

In particular, if $\langle\tau_{j},\tau_{j}\rangle=1$ for all $1\leq j\leq n$,
then

$\displaystyle\text{{(Higher order p-adic Welch bound)}}\quad\max_{1\leq
j,k\leq n,j\neq
k}\\{|n|,|\langle\tau_{j},\tau_{k}\rangle|^{2m}\\}\geq\frac{|n|^{2}}{\left|{d+m-1\choose
m}\right|}.$

###### Proof.

Define $S_{\tau}:\text{Sym}^{m}(\mathbb{Q}_{p}^{d})\ni
x\mapsto\sum_{j=1}^{n}\langle x,\tau_{j}^{\otimes m}\rangle\tau_{j}^{\otimes
m}\in\text{Sym}^{m}(\mathbb{Q}_{p}^{d})$. Then Then

$\displaystyle
b\operatorname{dim(\text{Sym}^{m}(\mathbb{Q}_{p}^{d}))}=\operatorname{Tra}(bI_{\text{Sym}^{m}(\mathbb{Q}_{p}^{d})})=\operatorname{Tra}(S_{\tau})=\sum_{j=1}^{n}\langle\tau_{j}^{\otimes
m},\tau_{j}^{\otimes m}\rangle,$ $\displaystyle
b^{2}\operatorname{dim(\text{Sym}^{m}(\mathbb{Q}_{p}^{d}))}=\operatorname{Tra}(b^{2}I_{\text{Sym}^{m}(\mathbb{Q}_{p}^{d})})=\operatorname{Tra}(S^{2}_{\tau})=\sum_{j=1}^{n}\sum_{k=1}^{n}\langle\tau_{j}^{\otimes
m},\tau^{\otimes m}_{k}\rangle\langle\tau^{\otimes m}_{k},\tau^{\otimes
m}_{j}\rangle.$

Therefore

$\displaystyle\left|\sum_{j=1}^{n}\langle\tau_{j},\tau_{j}\rangle^{m}\right|^{2}=\left|\sum_{j=1}^{n}\langle\tau_{j}^{\otimes
m},\tau_{j}^{\otimes
m}\rangle\right|^{2}=|\operatorname{Tra}(S_{\tau})|^{2}=\left|b\operatorname{dim(\text{Sym}^{m}(\mathbb{Q}_{p}^{d}))}\right|^{2}$
$\displaystyle=\left|\operatorname{dim(\text{Sym}^{m}(\mathbb{Q}_{p}^{d}))}\right|\left|b^{2}\operatorname{dim(\text{Sym}^{m}(\mathbb{Q}_{p}^{d}))}\right|$
$\displaystyle=\left|\operatorname{dim(\text{Sym}^{m}(\mathbb{Q}_{p}^{d}))}\right|\left|\sum_{j=1}^{n}\sum_{k=1}^{n}\langle\tau_{j}^{\otimes
m},\tau_{k}^{\otimes m}\rangle\langle\tau_{k}^{\otimes m},\tau_{j}^{\otimes
m}\rangle\right|$ $\displaystyle=\left|{d+m-1\choose
m}\right|\left|\sum_{j=1}^{n}\sum_{k=1}^{n}\langle\tau_{j}^{\otimes
m},\tau_{k}^{\otimes m}\rangle\langle\tau_{k}^{\otimes m},\tau_{j}^{\otimes
m}\rangle\right|$ $\displaystyle=\left|{d+m-1\choose
m}\right|\left|\sum_{j=1}^{n}\sum_{k=1}^{n}\langle\tau_{j},\tau_{k}\rangle^{m}\langle\tau_{k},\tau_{j}\rangle^{m}\right|$
$\displaystyle=\left|{d+m-1\choose
m}\right|\left|\sum_{l=1}^{n}\langle\tau_{l},\tau_{l}\rangle^{2m}+\sum_{j,k=1,j\neq
k}^{n}\langle\tau_{j},\tau_{k}\rangle^{m}\langle\tau_{k},\tau_{j}\rangle^{m}\right|$
$\displaystyle\leq\left|{d+m-1\choose m}\right|\max_{1\leq j,k\leq n,j\neq
k}\left\\{\left|\sum_{l=1}^{n}\langle\tau_{l},\tau_{l}\rangle^{2m}\right|,|\langle\tau_{j},\tau_{k}\rangle^{m}\langle\tau_{k},\tau_{j}\rangle^{m}|\right\\}$
$\displaystyle=\left|{d+m-1\choose m}\right|\max_{1\leq j,k\leq n,j\neq
k}\left\\{\left|\sum_{l=1}^{n}\langle\tau_{l},\tau_{l}\rangle^{2m}\right|,|\langle\tau_{j},\tau_{k}\rangle|^{2m}\right\\}.$

Whenever $\langle\tau_{j},\tau_{j}\rangle=1$ for all $1\leq j\leq n$,

$\displaystyle|n|^{2}\leq\left|{d+m-1\choose m}\right|\max_{1\leq j,k\leq
n,j\neq k}\\{|n|,|\langle\tau_{j},\tau_{k}\rangle|^{2m}\\}.$

∎

######  Remark 2.4.

Conditions given in the Theorem 2.3 says that the operator $S_{\tau}$ in the
proof of Theorem 2.3 is diagonalizable. Thus Theorem 2.3 is restrictive as the
hypothesis is stronger than that of Theorem 2.3 in [59]. However, note that
the field $\mathbb{Q}_{p}$ does not satisfies the Equation (2) in [59] (see
[67]) and hence neither the results in this paper can be derived from the
results in [59] nor the results in [59] can be derived from the results in
this paper.

######  Remark 2.5.

Theorems 2.1 and 2.3 hold by replacing $\mathbb{Q}_{p}^{d}$ by a
$d$-dimensional p-adic Hilbert space over any non-Archimedean (complete)
valued field (such as $\mathbb{C}_{p}$).

##  3\. p-adic Zauner Conjecture and open problems

Using Theorem 2.1 we ask the following question.

######  Question 3.1.

Given a prime $p$, for which $(d,n)\in\mathbb{N}\times\mathbb{N}$, there exist
vectors $\tau_{1},\dots,\tau_{n}\in\mathbb{Q}_{p}^{d}$ satisfying the
following.

  1. (i)

$\langle\tau_{j},\tau_{j}\rangle=1$ for all $1\leq j\leq n$.

  2. (ii)

There exists $b\in\mathbb{Q}_{p}$ satisfying

$\displaystyle\sum_{j=1}^{n}\langle x,\tau_{j}\rangle\tau_{j}=bx,\quad\forall
x\in\mathbb{Q}^{d}_{p}.$

  3. (iii)

$\displaystyle\max_{1\leq j,k\leq n,j\neq
k}\\{|n|,|\langle\tau_{j},\tau_{k}\rangle|^{2}\\}=\frac{|n|^{2}}{|d|}.$

We can formulate a strong form of Question 3.1 as follows.

######  Question 3.2.

Given a prime $p$, for which $(d,n)\in\mathbb{N}\times\mathbb{N}$, there exist
vectors $\tau_{1},\dots,\tau_{n}\in\mathbb{Q}_{p}^{d}$ satisfying the
following.

  1. (i)

$\langle\tau_{j},\tau_{j}\rangle=1$ for all $1\leq j\leq n$.

  2. (ii)

There exists $b\in\mathbb{Q}_{p}$ satisfying

$\displaystyle\sum_{j=1}^{n}\langle x,\tau_{j}\rangle\tau_{j}=bx,\quad\forall
x\in\mathbb{Q}^{d}_{p}.$

  3. (iii)

$\displaystyle\max_{1\leq j,k\leq n,j\neq
k}\\{|n|,|\langle\tau_{j},\tau_{k}\rangle|^{2}\\}=\frac{|n|^{2}}{|d|}.$

  4. (iv)

$\|\tau_{j}\|=1$ for all $1\leq j\leq n$.

Why Question 3.2 is different than Question 3.1? Reason is that unlike non-
Archimedean Hilbert spaces, in p-adic Hilbert spaces, norm is not defined as
$\sqrt{|\langle\cdot,\cdot\rangle|}.$ A particular case of Question 3.1 is the
following p-adic version of Zauner conjecture (see [6, 4, 90, 76, 36, 69, 3,
12, 5, 53, 39, 13, 46, 11, 61, 56] for Zauner conjecture in Hilbert spaces,
[58] for Zauner conjecture in Hilbert C*-modules, [57] for Zauner conjecture
in Banach spaces and [59] for Zauner conjecture in non-Archimedean Hilbert
spaces).

######  Conjecture 3.3.

(p-adic Zauner Conjecture) Let $p$ be a prime. For each $d\in\mathbb{N}$,
there exist vectors $\tau_{1},\dots,\tau_{d^{2}}\in\mathbb{Q}_{p}^{d}$
satisfying the following.

  1. (i)

$\langle\tau_{j},\tau_{j}\rangle=1$ for all $1\leq j\leq d^{2}$.

  2. (ii)

There exists $b\in\mathbb{Q}_{p}$ satisfying

$\displaystyle\sum_{j=1}^{d^{2}}\langle
x,\tau_{j}\rangle\tau_{j}=bx,\quad\forall x\in\mathbb{Q}^{d}_{p}.$

  3. (iii)

$\displaystyle|\langle\tau_{j},\tau_{k}\rangle|^{2}=|n|,\quad\forall 1\leq
j,k\leq d^{2},j\neq k.$

Question 3.2 gives the following Zauner conjecture.

######  Conjecture 3.4.

(p-adic Zauner Conjecture - strong form) Let $p$ be a prime. For each
$d\in\mathbb{N}$, there exist vectors
$\tau_{1},\dots,\tau_{d^{2}}\in\mathbb{Q}_{p}^{d}$ satisfying the following.

  1. (i)

$\langle\tau_{j},\tau_{j}\rangle=1$ for all $1\leq j\leq d^{2}$.

  2. (ii)

There exists $b\in\mathbb{Q}_{p}$ satisfying

$\displaystyle\sum_{j=1}^{d^{2}}\langle
x,\tau_{j}\rangle\tau_{j}=bx,\quad\forall x\in\mathbb{Q}^{d}_{p}.$

  3. (iii)

$\displaystyle|\langle\tau_{j},\tau_{k}\rangle|^{2}=|n|,\quad\forall 1\leq
j,k\leq d^{2},j\neq k.$

  4. (iv)

$\|\tau_{j}\|=1$ for all $1\leq j\leq d^{2}$.

We recall the definition of Gerzon’s bound which allows us to remember
companions to Welch bounds in Hilbert spaces.

######  Definition 3.5.

[48] Given $d\in\mathbb{N}$, define Gerzon’s bound

$\displaystyle\mathcal{Z}(d,\mathbb{K})\coloneqq\left\\{\begin{array}[]{cc}d^{2}&\quad\text{if}\quad\mathbb{K}=\mathbb{C}\\\
\frac{d(d+1)}{2}&\quad\text{if}\quad\mathbb{K}=\mathbb{R}.\\\
\end{array}\right.$

######  Theorem 3.6.

[48, 87, 64, 77, 17, 23, 44, 68] Define $\mathbb{K}=\mathbb{R}$ or
$\mathbb{C}$ and $m\coloneqq\operatorname{dim}_{\mathbb{R}}(\mathbb{K})/2$. If
$\\{\tau_{j}\\}_{j=1}^{n}$ is any collection of unit vectors in
$\mathbb{K}^{d}$, then

  1. (i)

(Bukh-Cox bound)

$\displaystyle\max_{1\leq j,k\leq n,j\neq
k}|\langle\tau_{j},\tau_{k}\rangle|\geq\frac{\mathcal{Z}(n-d,\mathbb{K})}{n(1+m(n-d-1)\sqrt{m^{-1}+n-d})-\mathcal{Z}(n-d,\mathbb{K})}\quad\text{if}\quad
n>d.$

  2. (ii)

(Orthoplex/Rankin bound)

$\displaystyle\max_{1\leq j,k\leq n,j\neq
k}|\langle\tau_{j},\tau_{k}\rangle|\geq\frac{1}{\sqrt{d}}\quad\text{if}\quad
n>\mathcal{Z}(d,\mathbb{K}).$

  3. (iii)

(Levenstein bound)

$\displaystyle\max_{1\leq j,k\leq n,j\neq
k}|\langle\tau_{j},\tau_{k}\rangle|\geq\sqrt{\frac{n(m+1)-d(md+1)}{(n-d)(md+1)}}\quad\text{if}\quad
n>\mathcal{Z}(d,\mathbb{K}).$

  4. (iv)

(Exponential bound)

$\displaystyle\max_{1\leq j,k\leq n,j\neq
k}|\langle\tau_{j},\tau_{k}\rangle|\geq 1-2n^{\frac{-1}{d-1}}.$

Theorem 3.6 and Theorem 2.1 give the following problem.

######  Question 3.7.

Whether there is a p-adic version of Theorem 3.6? In particular, does there
exists a version of

  1. (i)

p-adic Bukh-Cox bound?

  2. (ii)

p-adic Orthoplex/Rankin bound?

  3. (iii)

p-adic Levenstein bound?

  4. (iv)

p-adic Exponential bound?

We already wrote that Welch bounds have applications in study of equiangular
lines. We wish to formulate equiangular line problem for p-adic Hilbert
spaces. For the study of equiangular lines in Hilbert spaces we refer [60, 50,
40, 8, 16, 27, 37, 9, 49, 65, 43, 38, 18, 66, 89], quaternion Hilbert spaces
we refer [33], octonion Hilbert spaces we refer [21], finite dimensional
vector spaces over finite fields we refer [41, 42], for Banach spaces we refer
[57] and for non-Archimedean Hilbert spaces we refer [59].

######  Question 3.8.

(p-adic Equiangular Line Problem) Let $p$ be a prime. Given
$a\in\mathbb{Q}_{p}$, $d\in\mathbb{N}$ and $\gamma>0$, what is the maximum
$n=n(p,a,d,\gamma)\in\mathbb{N}$ such that there exist vectors
$\tau_{1},\dots,\tau_{n}\in\mathbb{Q}_{p}^{d}$ satisfying the following.

  1. (i)

$\langle\tau_{j},\tau_{j}\rangle=a$ for all $1\leq j\leq n$.

  2. (ii)

$|\langle\tau_{j},\tau_{k}\rangle|^{2}=\gamma$ for all $1\leq j,k\leq n,j\neq
k$.

In particular, whether there is a p-adic Gerzon bound?

Question 3.8 can be easily lifted to formulate question of p-adic regular
$s$-distance sets.

## References

  * [1] Sergio Albeverio, Jose Manuel Bayod, Cristina Perez-Garcia, Roberto Cianci, and Andrew Khrennikov.  Non-Archimedean analogues of orthogonal and symmetric operators and $p$-adic quantization.  Acta Appl. Math., 57(3):205–237, 1999.
  * [2] W. O. Alltop.  Complex sequences with low periodic correlations.  IEEE Trans. Inform. Theory, 26(3):350–354, 1980.
  * [3] D. M. Appleby.  Symmetric informationally complete-positive operator valued measures and the extended Clifford group.  J. Math. Phys., 46(5):052107, 29, 2005.
  * [4] Marcus Appleby, Ingemar Bengtsson, Steven Flammia, and Dardo Goyeneche.  Tight frames, Hadamard matrices and Zauner’s conjecture.  J. Phys. A, 52(29):295301, 26, 2019.
  * [5] Marcus Appleby, Steven Flammia, Gary McConnell, and Jon Yard.  SICs and algebraic number theory.  Found. Phys., 47(8):1042–1059, 2017.
  * [6] Marcus Appleby, Steven Flammia, Gary McConnell, and Jon Yard.  Generating ray class fields of real quadratic fields via complex equiangular lines.  Acta Arith., 192(3):211–233, 2020.
  * [7] Waheed U. Bajwa, Robert Calderbank, and Dustin G. Mixon.  Two are better than one: fundamental parameters of frame coherence.  Appl. Comput. Harmon. Anal., 33(1):58–78, 2012.
  * [8] Igor Balla, Felix Draxler, Peter Keevash, and Benny Sudakov.  Equiangular lines and spherical codes in Euclidean space.  Invent. Math., 211(1):179–212, 2018.
  * [9] Alexander Barg and Wei-Hsuan Yu.  New bounds for equiangular lines.  In Discrete geometry and algebraic combinatorics, volume 625 of Contemp. Math., pages 111–121. Amer. Math. Soc., Providence, RI, 2014.
  * [10] John J. Benedetto and Matthew Fickus.  Finite normalized tight frames.  Adv. Comput. Math., 18(2-4):357–385, 2003.
  * [11] Ingemar Bengtsson.  The number behind the simplest SIC-POVM.  Found. Phys., 47(8):1031–1041, 2017.
  * [12] Ingemar Bengtsson.  SICs: some explanations.  Found. Phys., 50(12):1794–1808, 2020.
  * [13] Ingemar Bengtsson and Karol Zyczkowski.  On discrete structures in finite Hilbert spaces.  arXiv:1701.07902v1 [quant-ph], 26 January 2017.
  * [14] Cristiano Bocci and Luca Chiantini.  An introduction to algebraic statistics with tensors, volume 118 of Unitext.  Springer, Cham, 2019.
  * [15] Bernhard G. Bodmann and John Haas.  Frame potentials and the geometry of frames.  J. Fourier Anal. Appl., 21(6):1344–1383, 2015.
  * [16] Boris Bukh.  Bounds on equiangular lines and on related spherical codes.  SIAM J. Discrete Math., 30(1):549–554, 2016.
  * [17] Boris Bukh and Christopher Cox.  Nearly orthogonal vectors and small antipodal spherical codes.  Israel J. Math., 238(1):359–388, 2020.
  * [18] A. R. Calderbank, P. J. Cameron, W. M. Kantor, and J. J. Seidel.  $Z_{4}$-Kerdock codes, orthogonal spreads, and extremal Euclidean line-sets.  Proc. London Math. Soc. (3), 75(2):436–480, 1997.
  * [19] Peter G. Casazza, Matthew Fickus, Jelena Kovačević, Manuel T. Leon, and Janet C. Tremain.  A physical interpretation of tight frames.  In Harmonic analysis and applications, Appl. Numer. Harmon. Anal., pages 51–76. Birkhäuser Boston, Boston, MA, 2006.
  * [20] Ole Christensen, Somantika Datta, and Rae Young Kim.  Equiangular frames and generalizations of the Welch bound to dual pairs of frames.  Linear Multilinear Algebra, 68(12):2495–2505, 2020.
  * [21] Henry Cohn, Abhinav Kumar, and Gregory Minton.  Optimal simplices and codes in projective spaces.  Geom. Topol., 20(3):1289–1357, 2016.
  * [22] Pierre Comon, Gene Golub, Lek-Heng Lim, and Bernard Mourrain.  Symmetric tensors and symmetric tensor rank.  SIAM J. Matrix Anal. Appl., 30(3):1254–1279, 2008.
  * [23] John H. Conway, Ronald H. Hardin, and Neil J. A. Sloane.  Packing lines, planes, etc.: packings in Grassmannian spaces.  Experiment. Math., 5(2):139–159, 1996.
  * [24] G. Coutinho, C. Godsil, H. Shirazi, and H. Zhan.  Equiangular lines and covers of the complete graph.  Linear Algebra Appl., 488:264–283, 2016.
  * [25] S. Datta, S. Howard, and D. Cochran.  Geometry of the Welch bounds.  Linear Algebra Appl., 437(10):2455–2470, 2012.
  * [26] Somantika Datta.  Welch bounds for cross correlation of subspaces and generalizations.  Linear Multilinear Algebra, 64(8):1484–1497, 2016.
  * [27] D. de Caen.  Large equiangular sets of lines in Euclidean space.  Electron. J. Combin., 7:Research Paper 55, 3, 2000.
  * [28] Toka Diagana.  Non-Archimedean linear operators and applications.  Nova Science Publishers, Inc., Huntington, NY, 2007.
  * [29] Toka Diagana and Francois Ramaroson.  Non-Archimedean operator theory.  SpringerBriefs in Mathematics. Springer, Cham, 2016.
  * [30] Cunsheng Ding and Tao Feng.  Codebooks from almost difference sets.  Des. Codes Cryptogr., 46(1):113–126, 2008.
  * [31] M. Ehler and K. A. Okoudjou.  Minimization of the probabilistic $p$-frame potential.  J. Statist. Plann. Inference, 142(3):645–659, 2012.
  * [32] Yonina C. Eldar and Gitta Kutyniok, editors.  Compressed sensing : Theory and application.  Cambridge University Press, Cambridge, 2012.
  * [33] Boumediene Et-Taoui.  Quaternionic equiangular lines.  Adv. Geom., 20(2):273–284, 2020.
  * [34] Matthew Fickus, John Jasper, and Dustin G. Mixon.  Packings in real projective spaces.  SIAM J. Appl. Algebra Geom., 2(3):377–409, 2018.
  * [35] Simon Foucart and Holger Rauhut.  A mathematical introduction to compressive sensing.  Applied and Numerical Harmonic Analysis. Birkhäuser/Springer, New York, 2013.
  * [36] Hoang M.C. Fuchs, C.A. and B.C Stacey.  The SIC question: History and state of play.  Axioms, 6(3), 2017.
  * [37] Alexey Glazyrin and Wei-Hsuan Yu.  Upper bounds for $s$-distance sets and equiangular lines.  Adv. Math., 330:810–833, 2018.
  * [38] Chris Godsil and Aidan Roy.  Equiangular lines, mutually unbiased bases, and spin models.  European J. Combin., 30(1):246–262, 2009.
  * [39] Gilad Gour and Amir Kalev.  Construction of all general symmetric informationally complete measurements.  J. Phys. A, 47(33):335302, 14, 2014.
  * [40] Gary Greaves, Jacobus H. Koolen, Akihiro Munemasa, and Ferenc Szollosi.  Equiangular lines in Euclidean spaces.  J. Combin. Theory Ser. A, 138:208–235, 2016.
  * [41] Gary R. W. Greaves, Joseph W. Iverson, John Jasper, and Dustin G. Mixon.  Frames over finite fields: basic theory and equiangular lines in unitary geometry.  Finite Fields Appl., 77:Paper No. 101954, 41, 2022.
  * [42] Gary R. W. Greaves, Joseph W. Iverson, John Jasper, and Dustin G. Mixon.  Frames over finite fields: equiangular lines in orthogonal geometry.  Linear Algebra Appl., 639:50–80, 2022.
  * [43] Gary R. W. Greaves, Jeven Syatriadi, and Pavlo Yatsyna.  Equiangular lines in low dimensional Euclidean spaces.  Combinatorica, 41(6):839–872, 2021.
  * [44] John I. Haas, Nathaniel Hammen, and Dustin G. Mixon.  The Levenstein bound for packings in projective spaces.  Proceedings, volume 10394, Wavelets and Sparsity XVII, SPIE Optical Engineering+Applications, San Diego, California, United States of America, 2017.
  * [45] Marina Haikin, Ram Zamir, and Matan Gavish.  Frame moments and Welch bound with erasures.  2018 IEEE International Symposium on Information Theory (ISIT), pages 2057–2061, 2018.
  * [46] Pawel Horodecki, Lukasz Rudnicki, and Karol Zyczkowski.  Five open problems in theory of quantum information.  arXiv:2002.03233v2 [quant-ph], 21 December 2020.
  * [47] Joseph W. Iverson and Dustin G. Mixon.  Doubly transitive lines I: Higman pairs and roux.  J. Combin. Theory Ser. A, 185:Paper No. 105540, 47, 2022.
  * [48] John Jasper, Emily J. King, and Dustin G. Mixon.  Game of Sloanes: best known packings in complex projective space.  Proc. SPIE 11138, Wavelets and Sparsity XVIII, 2019.
  * [49] Zilin Jiang and Alexandr Polyanskii.  Forbidden subgraphs for graphs of bounded spectral radius, with applications to equiangular lines.  Israel J. Math., 236(1):393–421, 2020.
  * [50] Zilin Jiang, Jonathan Tidor, Yuan Yao, Shengtong Zhang, and Yufei Zhao.  Equiangular lines with a fixed angle.  Ann. of Math. (2), 194(3):729–743, 2021.
  * [51] G. K. Kalisch.  On $p$-adic Hilbert spaces.  Ann. of Math. (2), 48:180–192, 1947.
  * [52] Andrew Khrennikov.  The ultrametric Hilbert-space description of quantum measurements with a finite exactness.  Found. Phys., 26(8):1033–1054, 1996.
  * [53] Gene S. Kopp.  SIC-POVMs and the Stark Conjectures.  Int. Math. Res. Not. IMRN, (18):13812–13838, 2021.
  * [54] J. Kovacevic and A. Chebira.  Life beyond bases: The advent of frames (part I).  IEEE Signal Processing Magazine, 24(4):86–104, 2007.
  * [55] J. Kovacevic and A. Chebira.  Life beyond bases: The advent of frames (part II).  IEEE Signal Processing Magazine, 24(5):115–125, 2007.
  * [56] K. Mahesh Krishna.  Continuous Welch bounds with applications.  arXiv:2109.09296v1 [FA], 20 September 2021.
  * [57] K. Mahesh Krishna.  Discrete and continuous Welch bounds for Banach spaces with applications.  arXiv:2201.00980v1 [math.FA] 4 January, 2022.
  * [58] K. Mahesh Krishna.  Modular Welch bounds with applications.  arXiv:2201.00319v1 [OA] 2 January, 2022.
  * [59] K. Mahesh Krishna.  Non-Archimedean Welch bounds and non-Archimedean Zauner conjecture.  arXiv.submit/4467018 [math. FA], 28 August, 2022.
  * [60] P. W. H. Lemmens and J. J. Seidel.  Equiangular lines.  J. Algebra, 24:494–512, 1973.
  * [61] M. Maxino and Dustin G. Mixon.  Biangular Gabor frames and Zauner’s conjecture.  In Wavelets and Sparsity XVIII. 2019.
  * [62] Dustin G. Mixon, Christopher J. Quinn, Negar Kiyavash, and Matthew Fickus.  Fingerprinting with equiangular tight frames.  IEEE Trans. Inform. Theory, 59(3):1855–1865, 2013.
  * [63] Dustin G. Mixon and James Solazzo.  A short introduction to optimal line packings.  College Math. J., 49(2):82–91, 2018.
  * [64] K.K. Mukkavilli, A. Sabharwal, E. Erkip, and B. Aazhang.  On beamforming with finite rate feedback in multiple-antenna systems.  IEEE Transactions on Information Theory, 49(10):2562–2579, 2003\.
  * [65] A. Neumaier.  Graph representations, two-distance sets, and equiangular lines.  Linear Algebra Appl., 114/115:141–156, 1989.
  * [66] Takayuki Okuda and Wei-Hsuan Yu.  A new relative bound for equiangular lines and nonexistence of tight spherical designs of harmonic index 4.  European J. Combin., 53:96–103, 2016.
  * [67] C. Perez-Garcia and W. H. Schikhof.  Locally convex spaces over non-Archimedean valued fields, volume 119 of Cambridge Studies in Advanced Mathematics.  Cambridge University Press, Cambridge, 2010.
  * [68] R. A. Rankin.  The closest packing of spherical caps in $n$ dimensions.  Proc. Glasgow Math. Assoc., 2:139–144, 1955.
  * [69] Joseph M. Renes, Robin Blume-Kohout, A. J. Scott, and Carlton M. Caves.  Symmetric informationally complete quantum measurements.  J. Math. Phys., 45(6):2171–2180, 2004.
  * [70] C. Rose, S. Ulukus, and R. D. Yates.  Wireless systems and interference avoidance.  EEE Transactions on Wireless Communications, 1(3):415–428, 2002\.
  * [71] Moshe Rosenfeld.  In praise of the Gram matrix.  In The mathematics of Paul Erdős, II, volume 14 of Algorithms Combin., pages 318–323. Springer, Berlin, 1997.
  * [72] Dilip V. Sarwate.  Bounds on crosscorrelation and autocorrelation of sequences.  IEEE Trans. Inform. Theory, 25(6):720–724, 1979.
  * [73] Dilip V. Sarwate.  Meeting the Welch bound with equality.  In Sequences and their applications (Singapore, 1998), Springer Ser. Discrete Math. Theor. Comput. Sci., pages 79–102. Springer, London, 1999.
  * [74] Karin Schnass and Pierre Vandergheynst.  Dictionary preconditioning for greedy algorithms.  IEEE Trans. Signal Process., 56(5):1994–2002, 2008.
  * [75] A. J. Scott.  Tight informationally complete quantum measurements.  J. Phys. A, 39(43):13507–13530, 2006.
  * [76] A. J. Scott and M. Grassl.  Symmetric informationally complete positive-operator-valued measures: a new computer study.  J. Math. Phys., 51(4):042203, 16, 2010.
  * [77] Mojtaba Soltanalian, Mohammad Mahdi Naghsh, and Petre Stoica.  On meeting the peak correlation bounds.  IEEE Transactions on Signal Processing, 62(5):1210–1220, 2014.
  * [78] Thomas Strohmer and Robert W. Heath, Jr.  Grassmannian frames with applications to coding and communication.  Appl. Comput. Harmon. Anal., 14(3):257–275, 2003.
  * [79] Mátyás A. Sustik, Joel A. Tropp, Inderjit S. Dhillon, and Robert W. Heath, Jr.  On the existence of equiangular tight frames.  Linear Algebra Appl., 426(2-3):619–635, 2007.
  * [80] Yan Shuo Tan.  Energy optimization for distributions on the sphere and improvement to the Welch bounds.  Electron. Commun. Probab., 22:Paper No. 43, 12, 2017.
  * [81] Joel A. Tropp.  Greed is good: algorithmic results for sparse approximation.  IEEE Trans. Inform. Theory, 50(10):2231–2242, 2004.
  * [82] Joel A. Tropp, Inderjit S. Dhillon, Robert W. Heath, Jr., and Thomas Strohmer.  Designing structured tight frames via an alternating projection method.  IEEE Trans. Inform. Theory, 51(1):188–209, 2005.
  * [83] M. Vidyasagar.  An introduction to compressed sensing, volume 22 of Computational Science & Engineering.  Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2020.
  * [84] Shayne Waldron.  Generalized Welch bound equality sequences are tight frames.  IEEE Trans. Inform. Theory, 49(9):2307–2309, 2003.
  * [85] Shayne Waldron.  A sharpening of the Welch bounds and the existence of real and complex spherical $t$-designs.  IEEE Trans. Inform. Theory, 63(11):6849–6857, 2017.
  * [86] L. Welch.  Lower bounds on the maximum cross correlation of signals.  IEEE Transactions on Information Theory, 20(3):397–399, 1974.
  * [87] Pengfei Xia, Shengli Zhou, and Georgios B. Giannakis.  Achieving the Welch bound with difference sets.  IEEE Trans. Inform. Theory, 51(5):1900–1907, 2005.
  * [88] Pengfei Xia, Shengli Zhou, and Georgios B. Giannakis.  Correction to: “Achieving the Welch bound with difference sets” [IEEE Trans. Inform. Theory 51 (2005), no. 5, 1900–1907].  IEEE Trans. Inform. Theory, 52(7):3359, 2006.
  * [89] Wei-Hsuan Yu.  New bounds for equiangular lines and spherical two-distance sets.  SIAM J. Discrete Math., 31(2):908–917, 2017.
  * [90] Gerhard Zauner.  Quantum designs: foundations of a noncommutative design theory.  Int. J. Quantum Inf., 9(1):445–507, 2011.


# THE MSXF TTS SYSTEM FOR ICASSP 2022 ADD CHALLENGE

###### Abstract

This paper presents our MSXF TTS system for Task 3.1 of the Audio Deep
Synthesis Detection (ADD) Challenge 2022. We use an end to end text to speech
system, and add a constraint loss to the system when training stage. The end
to end TTS system is VITS, and the pre-training self-supervised model is
wav2vec 2.0. And we also explore the influence of the speech speed and volume
in spoofing. The faster speech means the less the silence part in audio, the
easier to fool the detector. We also find the smaller the volume, the better
spoofing ability, though we normalize volume for submission. Our team is
identified as C2, and we got the fourth place in the challenge.

Index Terms—  fake speech,VITS, wav2vec2, speech representation, TTS

##  1 Introduction

This paper describes the zkj team’s submissions for Task 3.1 of the Audio Deep
Synthesis Detection (ADD) Challenge 2022[1],which is an IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP) 2022 Signal
Processing Grand Challenge. The purpose of the Challenge is to build
innovative new technologies that can further accelerate and foster research on
detecting deep synthesis and manipulated audios. The ADD 2022 includes three
tracks, the track 3 contains two tasks: track 3.1 Generation task and track
3.2 Detection task. We only participant track 3.1, which aims to generate fake
audios that can fool the fake detection model.

To generate fake audios, the text to speech or voice cloning system can be
used. Though there are methods like Adversarial Voice Conversion[2], it needs
to generate adversarial examples against a neural-network-based spoofing
detector by white-box. But the task requires that the text content and speaker
identity of generated audio is constrained to ensure the submitted sample are
generated by deep fake methods. So only using Voice Conversion is not suitable
for text input required and the spoofing detector is also not known. Also
considering the speech recognition and speaker recognition technologies would
be applied for screening each submitted sample, we need the system to provide
high MOS score. Finally, we choose to build a multi-speaker text to speech
system.

There are many successful multi-speaker text to speech system, like [3] [4]
[5] etc. Which one to be chosen needs to be determined. Consider our purpose
is fooling the fake detection model, so we need the distribution of the
generated audio is the same as the train data. The less gap between the fake
and real audio the better ability to fool the detector. So we need a GAN
vocoder like hifi-gan [6] which generates waveform like the train data. And
then we use The full end to end text to speech system to prevent the two
stages system producing gap between acoustic model and vocoder, Finally, we
choose conditional variational autoencoder with adversarial learning for end
to end text to speech(VITS) [7], which gets MOS score almost the same as
ground truth. We are also inspired by the method [8] that uses speaker
verification as a constraint to improve white-box spoofing attack. But we
replace speaker verification with the wav2vec 2.0 [9] for black-box spoofing.
For using the wav2vec 2.0 which is a self-supervised learning of speech
representation, we follow the research [10][11]. It shows which layer of
wav2vec 2.0 is better for a specific category of feature to choose for feature
extraction for downstream tasks.

The main contributions of this paper are summarized as follows:

(1)we use wav2vec 2.0 to improve speaker identity of the VITS, it shows better
result than origin VITS system.

(2)we demonstrate the VITS system is better than fastspeech2+hifi-gan system
for fooling the detector such as [12]. And found the suitable model settings
for VITS.

(3)we find that the talking speed of speech can influence the detector EER on
test data. We also find that the volume of the audio can influence the result,
but we follow the Challenge guide to set it as the original train data’s
volume.

This paper is organized as follows: Section 2 describes our system. Section 3
introduces the setup of the datasets for the challenge, experiments. Section 4
gives the result and analysis. Finally, Section 5 gives the conclusions.

##  2 SYSTEM DESCRIPTION

Our system is made up of two modules: text to phoneme front-end module, end to
end phoneme to speech module.

###  2.1 Text to Phoneme Front-end Module

Though texts and pinyin are both provided in the training corpus, we just need
to convert the challenge texts to phoneme for synthesis. We do text
normalization (TN) first to convert the number and symbols to right pinyin.
Here we use rule based method to do TN. Then we use g2pM[13] to process
polyphonic characters. And finally the rest Chinese characters in texts are
converted to pinyin with 5 tones by a grapheme-to-phoneme (G2P) toolkit named
pypinyin. Our system don’t predict prosodic structure because we believe
multi-speaker system with enough training data can learn it well, There is
also another reason that some data of the AISHELL-3 don’t read prosodic
structure rightly.

###  2.2 End to End Phoneme to Speech Module

The end to end phoneme to speech module, we use VITS, which’s input is phoneme
width tones and output audio waveform. we keep the VITS model settings in the
paper[7] and change the sample rate to 16K Hz.

To improve the spoofing ability, when training the VITS model, we add a
constraint loss to its generate loss. The constraint loss is a MSE loss of the
fake and real features that are extracted by wav2vec 2.0. As the
papers[10][11] give the experiments that the first layer of the wav2vec 2.0
got good result. So when train stage, use the first layer of wav2vec 2.0 to
extract features of fake and real audio for constraint loss calculating. The
wav2vec 2.0 are trained using the open source data WenetSpeech[14], a 10000+
hours multi-domain mandarin corpus.

##  3 Experiments

###  3.1 Datasets

For the audio fake game generation task, participants are allowed to generate
deepfake audio based on the AIShell-3 dataset. The AIShell-3 corpus contains
roughly 85 hours of speech recordings spoken by 218 native Mandarin Chinese
speakers. Speech is recorded by a high-quality microphone in a typical room
with room reverberation and some small background noise. We use the sox tools
to down the sample rate to 16K Hz. We use one data for each speaker for
validation, and all rest data for train. 10 speakers ID from AIShell-3 dataset
are listed as the evaluation speaker ID, and 500 lines text for each speaker
synthesis.

And for evaluation, we use RawGAT-ST [12] for detecting. The RawGAT-ST use the
ASVspoof 2017 eval data for training.

###  3.2 Model Settings

All models used in experiments are adjusted for 16K Hz sample rate. And keeps
other parameters the same as original papers. And we train a fastspeech2+hifi-
gan two stages model for comparing with VITS. Here we don’t use MFA to
generate the alignments which the fastspeech2 model needed. We use the
alignment learning framework in paper[15] instead of MFA, which improves
alignment convergence speed of existing attention-based mechanisms, simplifies
the training pipeline, and makes the models more robust to errors on long
utterances.

##  4 Results

We first test VITS and fastspeech2+hifi-gan with 2000 samples from 100
speakers in AISHELL-3 before the organizers give the evaluation speakers. From
the test, We find volume has huge influence on EER. So we only show the result
after normalize the volume, the VITS got 33.722 EER on RawGAT-ST while the
fastspeech2+hifi-gan got 17.342 EER. It demonstrates the end to end VITS
system more efficient on spoofing than the fs2+hifi-gan two stages system.

After got the evaluation speakers and texts, we do other experiments on the
evaluation test, the result show in Table 1. We can see the same VITS
parameters, only add wav2vec2.0 feature loss will let EER improve more than
one point. And the VITS has a stochastic duration predictor, the noise scale
parameter means a scale factor which multiplied to the standard deviation of
the prior distribution. The noise scale d means the standard deviation of
input noise of the stochastic duration predictor. The two parameters influence
the phoneme duration, the smaller the scale, the shorter the phoneme’s
duration. We can see that shorter duration means speech faster, and have less
silence part, the fake ability will improve. But when parameter set to zero,
the EER will go down, and too fast will influence the MOS, so finally we use
the parameters that ns=0.1, nsd=0.3 which the EER is 34.810.

models | EER(RawGAT-ST)
---|---
VITS(ns=0.667,nsd=0.8) | 30.681
VITS(ns=0.667,nsd=0.8)+wav2vec | 31.819
VITS(ns=1.0,nsd=0.3)+wav2vec | 20.000
VITS(ns=0.1,nsd=0.3)+wav2vec | 34.810
VITS(ns=0.0,nsd=0.3)+wav2vec | 33.000
VITS(ns=0.1,nsd=1.0)+wav2vec | 34.030
Table 1: Tests on the 10 evaluation speakers and texts. ns means the noise
scale factor, nsd means noise standard deviation of the duration stochastic
duration predictor. wav2vec means using wav2vec2.0 for VITS generate loss.

##  5 Conclusions

In this work, we build an end to end multi-speaker text to speech system. We
use wav2vec2.0 as a constraint for VITS generate loss, which improve the
ability of spoofing. Then we find the speed of speech influence the detector
too, the less the silence part in audio, the easier to fool the detector. We
also find volume has a huge influence in spoofing, the smaller the volume, the
better spoofing ability, though we normalize volume for submission. All the
work will make us to create a better detector for voice anti-spoofing in the
future.

##  6 Acknowledgements

This work is supported by Mashang Consumer Finance Co., Ltd.

## References

  * [1] “ADD CHALLENGE 2022 offical website,” http://addchallenge.cn/,  Accessed: 2022-01-21.
  * [2] Yi-Yang Ding, Li-Juan Liu, Yu Hu, and Zhen-Hua Ling,  “Adversarial Voice Conversion Against Neural Spoofing Detectors,”  in Proc. Interspeech 2021, 2021, pp. 816–820.
  * [3] Mingjian Chen, Xu Tan, Yi Ren, Jin Xu, Hao Sun, Sheng Zhao, and Tao Qin,  “MultiSpeech: Multi-Speaker Text to Speech with Transformer,”  in Proc. Interspeech 2020, 2020, pp. 4024–4028.
  * [4] Mingjian Chen, Xu Tan, Bohan Li, Yanqing Liu, Tao Qin, Sheng Zhao, and Tie-Yan Liu,  “Adaspeech: Adaptive text to speech for custom voice,”  ArXiv, vol. abs/2103.00993, 2021.
  * [5] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu,  “Fastspeech 2: Fast and high-quality end-to-end text to speech,”  6 2020.
  * [6] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae,  “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,”  Advances in Neural Information Processing Systems, vol. 2020-December, 10 2020.
  * [7] Jaehyeon Kim, Jungil Kong, and Juhee Son,  “Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,”  6 2021.
  * [8] Zexin Cai, Chuxiong Zhang, and Ming Li,  “From speaker verification to multispeaker speech synthesis, deep transfer with feedback constraint,”  Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, vol. 2020-October, pp. 3974–3978, 5 2020\.
  * [9] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  6 2020.
  * [10] Jui Shah, Yaman Kumar Singla, Changyou Chen, and Rajiv Ratn Shah,  “What all do audio transformer models hear? probing acoustic representations for language delivery and its structure,”  1 2021.
  * [11] Hyeong-Seok Choi, Juheon Lee, Wansoo Kim, Jie Hwan Lee, Hoon Heo, and Kyogu Lee,  “Neural analysis and synthesis: Reconstructing speech from self-supervised representations,” .
  * [12] Hemlata Tak, Jee weon Jung, Jose Patino, Madhu Kamble, Massimiliano Todisco, and Nicholas Evans,  “End-to-end spectro-temporal graph attention networks for speaker verification anti-spoofing and speech deepfake detection,”  pp. 1–8, 9 2021.
  * [13] Kyubyong Park and Seanie Lee,  “g2pm: A neural grapheme-to-phoneme conversion package for mandarinchinese based on a new open benchmark dataset,”  in INTERSPEECH, 2020.
  * [14] “WenetSpeech a 10000+ hours multi-domain mandarin corpus for speech recognition,” https://wenet.org.cn/WenetSpeech/,  Accessed: 2022-01-21.
  * [15] Rohan Badlani, Adrian Ła´ncucki Ła´ncucki, Kevin J Shih, Rafael Valle, Wei Ping, and Bryan Catanzaro,  “One tts alignment to rule them all,”  8 2021.


# Can Open Domain QA Models Answer Visual Knowledge Questions?

Jiawen Zhang Abhijit Mishra Avinesh P.V.S
Siddharth Patwardhan Sachin Agarwal

###### Abstract

The task of Outside Knowledge Visual Question Answering (OKVQA) requires an
automatic system to answer natural language questions about pictures and
images using external knowledge. We observe that many visual questions, which
contain deictic referential phrases referring to entities in the image, can be
rewritten as “non-grounded” questions and can be answered by existing text-
based question answering systems. This allows for the reuse of existing text-
based Open Domain QA Systems for visual question answering. In this work, we
propose a potentially data-efficient approach that reuses existing systems for
(a) image analysis, (b) question rewriting, and (c) text-based question
answering to answer such visual questions. Given an image and a question
pertaining to that image (a visual question), we first extract the entities
present in the image using pre-trained object and scene classifiers. Using
these detected entities, the visual questions can be rewritten so as to be
answerable by open domain QA systems. We explore two rewriting strategies: (1)
an unsupervised method using BERT (Devlin et al. 2018) for masking and
rewriting, and (2) a weakly supervised approach that combines adaptive
rewriting and reinforcement learning techniques to use the implicit feedback
from the QA system. We test our strategies on the publicly available OKVQA
dataset (Marino et al. 2019) and obtain a competitive performance with state-
of-the-art models while using only 10% of the training data.

##  1 Introduction

Within the realm of question answering systems, Visual Question Answering
(VQA) seeks to answer questions pertaining to given pictures or images.
Broadly, VQA can be categorized into three types — (a) direct (e.g., “How many
slices of pizza are there?”), (b) visual common sense reasoning (e.g., “Why is
the person running?”), (c) outside knowledge (OK) (e.g., “Where does this food
originate from?”). In this work, we focus on studying “outside knowledge”
queries, which require external knowledge from either free text documents
(e.g., Wikipedia), knowledge graphs (e.g., DBpedia), or large language models
(e.g., BERT) to answer questions about contents of the image.

Existing approaches on OKVQA tackle this task by combining text and image
representations for a joint QA system (Su et al. 2020; Lu et al. 2019).
However, they requires large amounts of high-quality multi-modal training data
containing images, a question to each image that draws upon external
knowledge, and some human-annotated answers for each question. Such high-
quality data is often harder to collect (Marino et al. 2019) than individual
datasets used for well-known problems in computer vision and NLP (e.g., entity
extraction from images, text-based QA). Furthermore, according to Jain et al.
(2021), a surprisingly large fraction of the queries in the available OKVQA
dataset (Marino et al. 2019) does not need external knowledge to answer.
Instead, some are independent of the image, some depend on speculation (e.g.,
“Can you guess what it is?”), some require object recognition (e.g. “What type
of plant is it?”) or are otherwise answerable from the image alone. To
simplify human annotation requirements, the OKVQA dataset treats the task as a
classification task, selecting from among the most frequent answers in the
training set rather than obtaining the answers from an external source (such
as, a knowledge graph or spans from a document corpus like Wikipedia).

Question |  How tall is this animal on average?
---|---
(1) Objects |  giraffe, stone, tree, park
(1) Rewrite |  How tall is giraffe on average?
(2) TextQA |  15 feet
Figure 1: Example OKVQA image and question describing steps needed to solve
using a text-based model

Since several pre-trained models are already available for image analysis (He
et al. 2016; Iandola et al. 2016; Wu et al. 2019) and for text-based question
answering (Roberts, Raffel, and Shazeer 2020; Lewis et al. 2020; Chen et al.
2017), a potential solution to mitigate the data scarcity problem is to
leverage such pre-trained models. We propose a potentially data-efficient
approach that reuses existing systems for (a) image analysis, (b) question
rewriting, and (c) text-based question answering to answer such visual
questions.

Given an image and a question pertaining to that image (a visual question), we
first extract the entities present in the image using pre-trained object and
scene classifiers. Using these detected entities, the visual questions can be
rewritten so as to be answerable by open domain QA systems. We explore two
rewriting strategies using unsupervised and weakly supervised techniques that
do not require any additional data for training a rewriter.

In the unsupervised approach, given a question and a list of entities as
input, we find all possible replacements of an entity with a span and leverage
existing language models like BERT (Devlin et al. 2018) and GPT-3 (Brown et
al. 2020) to select the semantically correct entity. While this method solely
depends on the pre-trained language models’ syntactic ability to choose the
best rewrite, the reformulated questions may not necessarily follow the same
vocabulary and syntactic structure that is seen by the text-based QA model
during training. Thus, for the second rewriting strategy, our goal is to adapt
to the input style of the text QA model. In this, we train a sequence-to-
sequence-based rewriter such as BART (Lewis et al. 2019) implicitly, without
the need of a labeled dataset of original and rewritten questions, with the
help of reinforcement learning (RL). We leverage feedback from the QA system
and correctness of the answer candidates to reward the RL.

We test the above two strategies on a portion of the publicly available OKVQA
dataset (Marino et al. 2019) that contains only knowledge-oriented visual
questions. We reuse T5-based closed-book question answering system (Roberts,
Raffel, and Shazeer 2020) as our text-based model and BART as our rewriter
model for implementing the two strategies mentioned above. We choose the T5
model purely because it is easy to deploy, does not depend on external
resources or models (such as retrievers in other pipelines), and delivers
state-of-the-art performance on knowledge-oriented question answering
datasets. Furthermore, experimental results reveal that our proposed rewrite-
based pipelines have an exact match and semantic-similarity-based scores
competitive to the other end-to-end VQA models.

The primary contributions of this work are the following:

  * •

We explore two rewriting strategies, a QA model agnostic (unsupervised) and a
QA model-driven (weakly supervised), to transform deictic knowledge-oriented
visual questions into direct questions better understood by text-based QA
models.

  * •

Unlike end-to-end outside knowledge VQA models that try to capture images,
questions, and external knowledge through a single model with a limited number
of parameters, our approach is modular. As a result, it can reuse richer
state-of-the-art individual models for image label extraction, rewriting, and
question answering.

  * •

Our QA model-based strategy uses an architecture of adaptive rewriting with
reinforcement learning techniques using fractionally lesser data (1/10th)
compared to state-of-the-art systems.

##  2 Related Work

In this section, we discuss related work to our research concerning: (1) Text-
based QA Models, (2) Visual Question Answering, (3) Adaptive rewriting
techniques, and (4) Reinforcement learning for rewriting.

###  2.1 Text-based QA Models

Open-domain question answering has been predominantly based on the retrieve-
then-read based paradigm (Zhu et al. 2021). Excerpts from large-scale web
documents (e.g., Wikipedia) or knowledge representations from massive KGs are
first retrieved and then comprehended by machine-learned models to yield
potential answers from the extracted passages. Knowledge-oriented question
answering, a type of open-domain question answering, often draws on two types
of information (Fu et al. 2020). The first relies on retrieving many documents
from a structured or unstructured knowledge source, such as Wikipedia. The
response is then extracted by conditioning on the query and the retrieved
passages. REALM (Guu et al. 2020) and ORQA (Lee, Chang, and Toutanova 2019),
for example, have demonstrated promising results when combining masked
language models with a differential retriever. The second type relies on
implicit knowledge stored in model parameters. For example, T5 (Roberts,
Raffel, and Shazeer 2020) distributes knowledge in its parameters in a
possibly inexplicable way using a large language model pre-trained on
unstructured text, which delivers well on knowledge-oriented questions without
having to mine knowledge from external sources. There have also been hybrid
models that combine parametric and non-parametric (i.e., retrieval-based)
memories, allowing information to be assessed, changed, and expanded. For
example, Retrieval-Augmented-Generation (RAG) (Lewis et al. 2020), enables it
to update internal knowledge based on newly obtained data.

Figure 2: End-to-end flow with entity extraction and input adaptation

###  2.2 Visual Question Answering

A growing interest in multi-modal research and the advent of high-capacity
deep neural learning models has made visual question answering a trending
research topic. Wu et al. (2017) provide a summary of VQA models and datasets,
where they focus on deep learning models that perform reasonably well on
public datasets such as VQA (Antol et al. 2015) and OKVQA (Marino et al.
2019). Early approaches to VQA combined recurrent networks with CNNs to
integrate textual and visual data, transformer-based models to highlight the
most relevant image region (Ben-younes et al. 2017; Wang et al. 2020), and
zero-shot learning to counter the lack of labeled examples (Demirel, Cinbis,
and Ikizler-Cinbis 2019). None of these approaches, however, are built to use
external knowledge, thus they can’t manage situations where the image doesn’t
reflect all of the information needed to answer the query. Some research
studies (Cao et al. 2021; Ziaeefard and Lécué 2020) have tackled knowledge-
based visual queries, but they exclusively deal with structured knowledge
sources represented as subject-relation-object or visual concept-relation-
attribute triplets. Therefore, the capacity to answer open-ended and complex
questions is limited.

Most relevant to us, Jain et al. (2021) have proposed to treat OKVQA as a task
of fusing structured data from the image with the unstructured text rather
than a visual recognition problem. The idea is to transform the multi-modal
input (image + text) to a text-only input so that the text-based QA model can
directly interpret and answer (Figure 1 shows a sample). The text-only version
of the original question is constructed by: (a) Selecting a referential
expression (e.g., “this animal”), (b) Substituting the expression with the
most appropriate entity from the image, and (c) Searching and augmenting
additional knowledge with the help of a search engine. For selection and
substitution, Jain et al. (2021) use supervised learning approaches for
predicting referential spans and selecting appropriate entities for
substitution. This approach, however, would be harder to scale OKVQA systems
to unseen domains and question types. Moreover, the output of such a rewriter
may not always align with the input vocabulary of the text-based QA models
used for answering. In our work, we emphasize using unsupervised and weakly
supervised methods that extract grammatical and contextual cues from the
visual questions to reformulate them.

###  2.3 Adaptive Rewriting Techniques

Since fine-tuning the large pre-trained model could be expensive if there are
many downstream tasks, Houlsby et al. (2019) propose a new transfer mechanism:
adding only a few trainable parameters with adapter module per task so that
new tasks can be added without revisiting previous ones. Adapters serve the
same purpose as fine-tuning but do it by stitching in layers to the main pre-
trained model, and updating the weights of these new layers while freezing the
weights of the pre-trained model. Seq2Seq based adapters (rewriters) are not
new and have been applied to tasks like accent adaptation (Khandelwal et al.
2020) and response generation (Madotto et al. 2020), but not for adapting to
knowledge-based visual QA. Based on the idea of adapter, Pfeiffer et al.
(2020) introduces Adapterhub111https://adapterhub.ml, a new framework built on
top of the HuggingFace library. These adapter architectures adapt well to
tasks based on top of generic encoders like BERT, but they can not easily
adapt to different input styles of QA models.

###  2.4 Reinforcement Learning for rewriting

Reinforcement learning has been successfully applied to various NLP tasks,
including summarization (Paulus, Xiong, and Socher 2018), question answering
(Hua et al. 2020), and many more. Only recently, Reinforcement Learning (RL)
has been applied for query rewriting. Buck et al. (2017) propose an approach
to rewrite a jeopardy style query (declarative) to a question answering system
that is often trained on questions (interrogative). The reformulation system
is trained end-to-end to maximize answer quality using policy gradient. The
policy gradient approach has also been used for tasks like machine translation
(Wu et al. 2018), which has shown decent performance improvement even when
less labeled data is available.

Figure 3: QA Model Agnostic Rewriting with Mask-Replace-Rank based approach
using BERT

##  3 Method

Inspired by the idea of adapter (Houlsby et al. 2019), our solution is to
adaptively rewrite the query and produce a ”non-grounded” question, which can
be answered by the text-based QA system. As shown in Figure 2, given an input
question (e.g., How tall is this animal on average?) and an image of two
giraffes, we first extract the top $E$ entities from the image using off-the-
shelf entity extractors such as SqueezeNet (Iandola et al. 2016) for object
entities and Scene Classifier (Zhou et al. 2017) for scene labels. Those
entities are then provided as input to a rewriter along with the original
question with the objective of rewriting the question into an independent
form. We take a simple approach of concatenating the entities with the
question (e.g. how tall is this animal on average . giraffe . stone . tree .
park), and allowing the model to learn the rewrite transformations. Our
adaptive rewriting model is a generative model that learns to rewrite this
input into an “independent” question (e.g. how tall is giraffe on average?).

The rewritten question is then passed on to the text-based QA model, to obtain
an answer. Adaptive rewriting offers the following advantages. (1) since we
directly use an off-the-shelf QA system, the core QA model does not have to be
retrained on this kind of input and is allowed to remain agnostic of the
changes in the input structure. (2) Adapter rewriting module can be
added/removed from the execution pipelines on-demand. This allows for a
modular system that is easy to debug and analyze. (3) the predicted answer is
more interpretable, since we can see the rewritten that resulted in the final
answer.

As mentioned in Section 1, we explore two approaches for adaptive rewriting:
QA Model Agnostic Rewriting and QA Model Aware Rewriting. We reiterate that
these approaches do not require additional labeled training data of original
and rewritten questions.

###  3.1 QA Model Agnostic Adaptive Rewriting

For model agnostic rewriting, we implement a fairly straightforward Mask-
Replace-Rank approach (Havens and Stal 2019). As shown in Figure 3, given a
question “how tall is this animal on average?”, we extract all N-grams
($N={1,2,3}$) from the question, and generate the masked candidates by
replacing the N-grams with the $\mathrm{<MASK>}$ tokens. For each entity
extracted from the image, a pre-trained BERT model (BERT-large (Devlin et al.
2018)) is used to compute the probability of replacing the mask with the
entity ($P_{bert}$(entity $|$ MASKed N-grams, context)). Entities are ranked
according to the language model probability of the token sequence and the top
scoring token sequence is selected as the rewrite. In our example, $P$(giraffe
$|$ this animal, context) returns the highest score since language model
prefers the sequence “how tall is giraffe on average?” to other candidate
sequences. This approach leverages the fill-in-the-blanks nature of the BERT
model without requiring additional training data.

###  3.2 QA Model Aware Adaptive Rewriting

While the prior approach relies primarily on identifying the more probable
token sequences for that language, we expect that including some signal from
the question answering task would help the model sort out the more ambiguous
cases. In this approach, the model-aware method uses the gold-standard answers
and the QA model to learn to rewrite. We use a generative sequence-to-sequence
rewriter that takes inputs over vocabulary $V_{0}$ and produces an output
sequence over vocabulary $V_{1}$, which in practice is the same as the
vocabulary of the black-box QA model. To begin with, this rewriter can be any
generic, pre-trained sequence to sequence model (such as BART), and is fine-
tuned with noisy inputs (Morris et al. 2020) that simulate our use-case
scenarios.

The difference between a traditional sequence-to-sequence model and the
rewriter is that the rewriter is not retrained with direct supervision of
gold-standard sentences provided as output labels. It is rather fine-tuned
based on the learning signals (gradients) from the black-box text-based QA
model. This way of training would not only help train the rewriter train with
less data, but it will also enable it to adapt to different QA systems during
training. Since most of the submodules in the architecture are frozen, the
rewriter should learn faster and mitigate the problem of catastrophic
forgetting during adaptation.

Figure 4: End-to-end flow with Policy Gradient Optimization

We use BART (Lewis et al. 2019) for our rewriter, keeping its input and output
vocabulary shared and same as the text-based question answering model (T5).
The rewriter is initialized with a pre-trained version of BART trained on a
large corpus with a denoising auto-encoding objective. The model then
undergoes a reinforcement learning process, comprised of three steps: (a)
Exploration (b) Reward Computation (c) Optimization (Figure 4).

#### Exploration:

In this step we allow the rewriter to produce a set of candidates. In
reinforcement learning settings for sequence-to-sequence networks, beam-search
decoding is often employed to generate candidates. Our BART decoder generates
top $K$ candidates ($K=3$) through beam search with a beam width of $5$.
Initially, the pre-trained version of BART (which is trained with an auto-
encoding objective) may not offer a lot of variations with beam search and may
end up copying the input. So, to allow some syntactic variations, we consider
the following strategy:

  1. 1.

We prepare several temporary inputs following a similar method as Section
3.1). We first extract all N-grams (e.g., $N={1,2,3}$) from the question and
replace them with entities from the image. For each entity, we filter the top
$T$ candidates (in our work, $T=10$) using a GPT-2 language model (Radford et
al. 2019), using the likelihood (log probability) of the rewrites.

  2. 2.

We then pass these $T$ temporary rewrites through the BART encoder-decoder,
which generates $K$ candidates for each temporary rewrite. For one entity the
total number of candidates is $T*K$.

  3. 3.

For each entity, we compute the average reward of the generated $T*K$
candidates following the steps in Section 3.2. or optimization, we only use
the candidate lot that has the maximum average reward.

#### Reward Computation:

Each candidate output is given to the QA model which predicts the candidate
answer. For $T*K$ candidates we compute the average rewards ($\mathcal{R}$) as
follows:

$\mathcal{R}=\frac{1}{|S|}\sum_{i=1}^{|S|}cos(e(y),e(\hat{y_{i}}))$ (1)

where, $S$ represents the $T*K$ candidates, $y$ and $\hat{y}$ are the gold
standard and predicted answers respectively, $e(.)$ is the embedding
representation of the answers, and $cos(.)$ is the cosine similarity between
two embeddings. As the answers may have multiple words, for embedding
extraction, we consider a sentence embedder such as BERT.

The rationale behind choosing this reward function is two-fold: (a) it is
continuous and offers smooth policy gradients (a) it indicates the degree with
which the predicted answers are semantically similar to the gold-standard
answers. With this design, the rewriter output “how stone this animal on
average” should receive low rewards, as it does not have adequate information
to obtain a correct answer.

#### Optimization:

In the optimization step, the rewriter model learns to exploit the candidates
that maximize the average reward ($\mathcal{R}$). In a traditional setting for
BART, for $B$ training instances of inputs (concatenated question and
entities) and rewrites, $\\{x^{i},r^{i}\\}_{i=1}^{B}$, the learning objective
(L) is to maximize the likelihood of a target $r^{i}$ given ${x^{i}}$, which
is similar to minimizing the cross entropy between the target distribution and
the predicted output distribution. For training the network, the gradient of
the negative cross-entropy loss is considered to update the model parameters.
The gradient is given by:

$\nabla_{\theta}L(\theta)=\nabla_{\theta}\sum_{i=1}^{B}r^{i}log~{}P_{M_{\theta}}(\hat{r^{i}}|x^{i})$
(2)

where $M_{\theta}$ is the BART model with parameters $\theta$. In our
reinforcement learning setting, for optimizing the reward that comes from an
external source, we use the policy gradient mechanism (Williams 1992). The
generator of BART, operating with a policy of
$P_{M_{\theta}}(\hat{r^{i}}|x^{i})$, producing an output $\hat{r^{i}}$ with an
expected reward ($\mathcal{R}$) computed using Equation 1, will thus have the
following gradient:

$\begin{split}\nabla_{\theta}RL(\theta)&=\nabla_{\theta}\mathbb{E}_{\hat{r_{i}}\sim
P_{M_{\theta}}(\hat{r^{i}}|x^{i})}[\mathcal{R}_{r^{i}}]\\\
&=\mathbb{E}[\mathcal{R}_{r^{i}})\nabla_{\theta}log(P_{M_{\theta}}(\hat{r^{i}}|x^{i}))]\end{split}$
(3)

where $RL$ is the modified learning objective which has to be maximized. This
way of optimizing ensures that at the end of the training, the BART model
learns to translate the concatenated input to a rewritten by dropping entities
and performing necessary re-ordering. For implementation, we use cross-entropy
loss for the policy in Equation 3, by computing the cross entropy between the
candidates and the input, using the input as reference. This helps in
penalizing too many edits (e.g., adding or dropping too many terms in the
rewrite).

##  4 Experimental Setup

In this section, we describe the setup for our experiments, including the
dataset, evaluation metrics, baselines and training details that we will use
to assess the effectiveness our approach.

###  4.1 Dataset

We test our method on the OKVQAS3 (Jain et al. 2021) dataset, which annotates
four types of questions in the OKVQA: (1) question which require detecting
objects and subsequent reasoning over an external knowledge source to arrive
at the answer, (2) question which require reading text from the image (OCR)
(and no other information) to answer, (3) question which are based on personal
opinion or speculation, (4) Other. Since we focus on rewriting knowledge-based
queries, we take only the questions annotated as the first type, which have
1,643 questions in the training dataset and 999 questions in the test dataset.
We further deleted the questions that require spatial reasoning (e.g., the
person on the left). After the data processing work, we eventually have 1,010
training examples and 363 test examples (Table 1).

Dataset | #Train | #Test
---|---|---
OKVQA | 9,009 | 5,046
OKVQAS3 | 1,643 | 999
Our Final Dataset | 1,010 | 363
Table 1: Statistics of the Knowledge VQA Datasets

###  4.2 Systems for Comparision

We compare our approach with five existing VQA methods222We obtained
predictions for OKVQA (Marino et al. 2019) from the author along with two
additional baselines: one uses a concatenated version (without rewrite) as
input to the T5 model and the other fine-tunes T5 with those concatenated
inputs. The VQA baselines are:

  * •

MUTAN (Ben-Younes et al. 2017): A tensor-based Tucker decomposition to
efficiently parametrize bilinear interactions between visual and textual
representations.

  * •

MUTAN+AN (Marino et al. 2019): An attention-based version of MUTAN that fuses
MUTAN and ArticleNet (MUTAN + AN) as a knowledge-base baseline.

  * •

BAN (Kim, Jun, and Zhang 2018): Bilinear Attention Networks that uses a co-
attention mechanism between the question and image features.

  * •

BAN+AN (Marino et al. 2019): An attention-version of BAN that concatenates the
output of the memory network with the last BAN hidden state.

  * •

QOnly (Marino et al. 2019): An neural network with 3 hidden layers that only
takes question features.

###  4.3 Evaluation Metrics

To evaluate the QA system, we employ three widely used metrics for QA systems
(a) Exact match (EM), (b) Bert Similarity Score (BS), (c) Human evaluation
(HE)

  * •

EM: We calculate the exact match score between the golden answers and the
predicted answers, and take the average across all answers.

  * •

BS: We use SentenceBERT to calculate the similarity between the golden answers
and the predicted answers and take the average score across all answers.

  * •

HE: We manually evaluate the quality of the answers and based on the question
and the image. The result should be either correct or incorrect. We asked four
annotators to help us grade the 363 test data for all baselines. They are
provided with a UI and guidelines (Section 4.4).

###  4.4 Grading Guidelines for Human Evaluation

There are 363 examples to be graded. For each system output, the human
annotator marks whether the answer and rewritten output given by the system
are correct or not (binary). Each example contains an image, a question about
the image, a list of gold standard answers from human annotators, and answers
from eight Question Answering systems. While evaluating the answers, note that
for certain questions multiple answers are possible. Moreover, some of the
answers may not appear in the gold standard answer list. So, in some cases
they mark correct vs. incorrect according to their best judgment. If required,
they are allowed to look up the question on a Web search engine. Certain
questions require the answers to be “numbers” (such as calorie content in food
or year of invention, etc.). The predicted answer is considered to be correct
if it is within a small window of the correct answer (e.g., 19th century for
1890).

###  4.5 Model and Training Details

For BERT-based scoring and ranking, we use
FitBERT333https://github.com/writerai/fitbert and GPT based scoring is done
through LMScorer444https://github.com/simonepri/lm-scorer. The model
optimization was carried out with a batch size of 16 for 9,600 steps. We
allocated 1 GPU and 32 CPUs for all of our experiments, and on average for
each experiment, we used 8 percent of the allocated GPU, 60 percent of the
allocated CPUs, and 35 percent of the GPU memory.

##  5 Results and Analysis

Models | Train Data | EM | BS | HE
---|---|---|---|---
VQA Models |
MUTAN+AN | 9,009 | 0.28 | 0.70 | 0.45
BAN | 9,009 | 0.29 | 0.70 | 0.47
BAN+AN | 9,009 | 0.29 | 0.70 | 0.43
MUTAN | 9,009 | 0.30 | 0.69 | 0.43
QOnly | 9,009 | 0.16 | 0.62 | 0.24
Baselines |
Concatenated Input | - | 0.32 | 0.71 | 0.54
Fine-Tuned T5 | 9,009 | 0.30 | 0.71 | 0.48
Our Methods |
Model Agnostic | - | 0.31 | 0.70 | 0.67
Model Aware | 1,010 | 0.29 | 0.69 | 0.67
Table 2: Evaluation of answers and rewrites using three different metrics: EM:
Exact Match, BS: Bert Similarity, HE: Human Evaluation

Image |  Question |  Extracted Entities | Model |  Rewritten Outputs (Inputs to T5) | Answers
---|---|---|---|---|---
|  what do these animals eat? |  zebras, tree, park | Gold |  what do zebras eat? | grass, plants, leaves
Concat |  what do these animals eat . zebras . tree . park | wild plants
Fine-Tuned T5 |  - | hay
Model Agnostic |  what do zebras eat? | leaves
Model Aware |  what do zebras eat? | wild plants
MUTAN |  - | plant
MUTAN+AN |  - | grass
BAN |  - | grass
BAN+AN |  - | grass
|  |  | QOnly |  - | meat
|  what famous founding father was known for his association with this object?  |  kite, sky, beach | Gold |  what famous founding father was known for his association with kite? | Benjamin Franklin
Concat |  what famous founding father was known for his association with this object . kite . sky . beach | Benjamin Franklin
Fine-Tuned T5 |  - | Benjamin Franklin
Model Agnostic |  what famous founding father was known for his association with kite? | Benjamin Franklin
Model Aware |  what famous founding father was known for his association with kite? | Benjamin Franklin
MUTAN |  - | kite
MUTAN+AN |  - | shawn white
BAN |  - | beach boy
BAN+AN |  - | Benjamin Franklin
QOnly |  - | wright
|  what conditions are necessary for this sport?  |  skiing, dogsled, ski slope  | Gold |  what conditions are necessary for skiing? | snow, snowfall
Concat |  what conditions are necessary for this sport . skiing . dogsled . ski slope | snow
Fine-Tuned T5 |  - | ice
Model Agnostic |  what conditions are necessary for dogsled? (comment: bad rewrite but generates good answer) | snow
Model Aware |  what conditions are necessary for ski slope? (comment: bad rewrite but generates good answer) | snowfall
MUTAN |  - | snow
MUTAN+AN |  - | snow
BAN |  - | snow
BAN+AN |  - | snow
QOnly |  - | ski

Table 3: Examples outputs from different methods.

Out of 363 test data, Table 2 shows the results for different comparison
systems using Exact match (EM), BERT similarity scores (BS), and human
evaluation (HE). We also present some anecdotal examples in Table 5. Except
for the QOnly model, which has much lower accuracy, our methods achieve
competitive results on exact match and BERT similarity scores in comparison
with other VQA models. Note that those VQA models are trained on significantly
more human labeled data. It is also worth noting that the exact match results
contain many false negatives, because the metric assumes that the model
generates the same answers as the gold standard answers. Nevertheless, we can
get other good answers such as the synonyms of the golden answers. Similarly,
BERT Similarity Score results can have false positives, because the answers
that are of the same type as the golden answers will be encouraged but they
might not be correct. We thus report answer correctness assessment by humans
as a reliable metric. Based on human evaluation results, our model agnostic
method and model aware methods seem to do better than the existing systems. We
have several interesting findings:

  * •

The difference between the exact match accuracy and human evaluation is larger
for our methods than the comparison system because our methods are more likely
to generate synonyms of the golden answers, which can be falsely ignored by
the exact match metric. T5 has a larger vocabulary size because it is trained
on at least 409K questions across three datasets: Natural Questions
(Kwiatkowski et al. 2019), WebQuestions (Berant et al. 2013) and TriviaQA
(Joshi et al. 2017). By contrast, those systems are only trained on 9,009 data
from a single dataset OKVQA labeled by five MTurk workers.

  * •

The fine-tuned T5 baseline is lower than the concatenated input across all
three metrics, which shows that fine-tuning does not help on this task.

  * •

Good rewrites usually generate good answers, but bad/grammatically incorrect
rewrites (e.g., last row in Table 5) may not lead to good answers.

##  6 Conclusion

In this paper, we explore question rewrite strategies for knowledge-oriented
VQA. We contend that for certain types of VQA question rewriting, a text-based
question answering system can lower the needs for training data. Unsupervised
and weakly-supervised reinforcement learning-based question rewriting
strategies are explored in this work. We demonstrate effective VQA using an
existing pre-trained text-based QA model. Experiments on these types of
questions on a public dataset illustrate that this technique is competitive
with other end-to-end VQA approaches.

For future work, we will investigate other types of text-based QA systems
within this framework. Since entity extraction and disambiguation are key
steps, we plan to train an entity extractor with a larger label ontology for
visual objects. Additionally, we would like to investigate the impact of
visual features, such as image position, bounding box information, relative
positions of objects, etc. on the quality of the entity extractor.

## References

  * Antol et al. (2015) Antol, S.; Agrawal, A.; Lu, J.; Mitchell, M.; Batra, D.; Zitnick, C. L.; and Parikh, D. 2015.  Vqa: Visual question answering.  In _Proceedings of the IEEE international conference on computer vision_ , 2425–2433.
  * Ben-younes et al. (2017) Ben-younes, H.; Cadene, R.; Cord, M.; and Thome, N. 2017.  MUTAN: Multimodal Tucker Fusion for Visual Question Answering.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_.
  * Ben-Younes et al. (2017) Ben-Younes, H.; Cadene, R.; Cord, M.; and Thome, N. 2017.  Mutan: Multimodal tucker fusion for visual question answering.  In _Proceedings of the IEEE international conference on computer vision_ , 2612–2620.
  * Berant et al. (2013) Berant, J.; Chou, A.; Frostig, R.; and Liang, P. 2013.  Semantic parsing on freebase from question-answer pairs.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , 1533–1544.
  * Brown et al. (2020) Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Chess, B.; Clark, J.; Berner, C.; McCandlish, S.; Radford, A.; Sutskever, I.; and Amodei, D. 2020.  Language Models are Few-Shot Learners.  In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M. F.; and Lin, H., eds., _Advances in Neural Information Processing Systems_ , volume 33, 1877–1901. Curran Associates, Inc.
  * Buck et al. (2017) Buck, C.; Bulian, J.; Ciaramita, M.; Gajewski, W.; Gesmundo, A.; Houlsby, N.; and Wang, W. 2017.  Ask the right questions: Active question reformulation with reinforcement learning.  _arXiv preprint arXiv:1705.07830_.
  * Cao et al. (2021) Cao, Q.; Li, B.; Liang, X.; Wang, K.; and Lin, L. 2021.  Knowledge-Routed Visual Question Reasoning: Challenges for Deep Representation Embedding.  _IEEE Transactions on Neural Networks and Learning Systems_.
  * Chen et al. (2017) Chen, D.; Fisch, A.; Weston, J.; and Bordes, A. 2017.  Reading wikipedia to answer open-domain questions.  _arXiv preprint arXiv:1704.00051_.
  * Demirel, Cinbis, and Ikizler-Cinbis (2019) Demirel, B.; Cinbis, R. G.; and Ikizler-Cinbis, N. 2019.  Image captioning with unseen objects.  _arXiv preprint arXiv:1908.00047_.
  * Devlin et al. (2018) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Fu et al. (2020) Fu, B.; Qiu, Y.; Tang, C.; Li, Y.; Yu, H.; and Sun, J. 2020.  A survey on complex question answering over knowledge base: Recent advances and challenges.  _arXiv preprint arXiv:2007.13069_.
  * Guu et al. (2020) Guu, K.; Lee, K.; Tung, Z.; Pasupat, P.; and Chang, M.-W. 2020.  Realm: Retrieval-augmented language model pre-training.  _arXiv preprint arXiv:2002.08909_.
  * Havens and Stal (2019) Havens, S.; and Stal, A. 2019.  Use BERT to Fill in the Blanks.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Houlsby et al. (2019) Houlsby, N.; Giurgiu, A.; Jastrzebski, S.; Morrone, B.; De Laroussilhe, Q.; Gesmundo, A.; Attariyan, M.; and Gelly, S. 2019.  Parameter-efficient transfer learning for NLP.  In _International Conference on Machine Learning_ , 2790–2799. PMLR.
  * Hua et al. (2020) Hua, Y.; Li, Y.-F.; Haffari, G.; Qi, G.; and Wu, T. 2020.  Few-Shot Complex Knowledge Base Question Answering via Meta Reinforcement Learning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 5827–5837. Online: Association for Computational Linguistics.
  * Iandola et al. (2016) Iandola, F. N.; Han, S.; Moskewicz, M. W.; Ashraf, K.; Dally, W. J.; and Keutzer, K. 2016.  SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and¡ 0.5 MB model size.  _arXiv preprint arXiv:1602.07360_.
  * Jain et al. (2021) Jain, A.; Kothyari, M.; Kumar, V.; Jyothi, P.; Ramakrishnan, G.; and Chakrabarti, S. 2021.  Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answering.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’21, 2491–2498. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450380379.
  * Joshi et al. (2017) Joshi, M.; Choi, E.; Weld, D. S.; and Zettlemoyer, L. 2017.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.  _arXiv preprint arXiv:1705.03551_.
  * Khandelwal et al. (2020) Khandelwal, K.; Jyothi, P.; Awasthi, A.; and Sarawagi, S. 2020.  Black-box Adaptation of ASR for Accented Speech.  In _INTERSPEECH_.
  * Kim, Jun, and Zhang (2018) Kim, J.-H.; Jun, J.; and Zhang, B.-T. 2018.  Bilinear attention networks.  _arXiv preprint arXiv:1805.07932_.
  * Kwiatkowski et al. (2019) Kwiatkowski, T.; Palomaki, J.; Redfield, O.; Collins, M.; Parikh, A.; Alberti, C.; Epstein, D.; Polosukhin, I.; Devlin, J.; Lee, K.; et al. 2019.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7: 453–466.
  * Lee, Chang, and Toutanova (2019) Lee, K.; Chang, M.-W.; and Toutanova, K. 2019.  Latent retrieval for weakly supervised open domain question answering.  _arXiv preprint arXiv:1906.00300_.
  * Lewis et al. (2019) Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; and Zettlemoyer, L. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_.
  * Lewis et al. (2020) Lewis, P.; Perez, E.; Piktus, A.; Petroni, F.; Karpukhin, V.; Goyal, N.; Küttler, H.; Lewis, M.; Yih, W.-t.; Rocktäschel, T.; et al. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _arXiv preprint arXiv:2005.11401_.
  * Lu et al. (2019) Lu, J.; Batra, D.; Parikh, D.; and Lee, S. 2019.  ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks.  In Wallach, H.; Larochelle, H.; Beygelzimer, A.; d'Alché-Buc, F.; Fox, E.; and Garnett, R., eds., _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Madotto et al. (2020) Madotto, A.; Lin, Z.; Bang, Y.; and Fung, P. 2020.  The Adapter-Bot: All-In-One Controllable Conversational Model.  _arXiv preprint arXiv:2008.12579_.
  * Marino et al. (2019) Marino, K.; Rastegari, M.; Farhadi, A.; and Mottaghi, R. 2019.  OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Morris et al. (2020) Morris, J.; Lifland, E.; Yoo, J. Y.; Grigsby, J.; Jin, D.; and Qi, Y. 2020.  TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , 119–126. Online: Association for Computational Linguistics.
  * Paulus, Xiong, and Socher (2018) Paulus, R.; Xiong, C.; and Socher, R. 2018.  A Deep Reinforced Model for Abstractive Summarization.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net.
  * Pfeiffer et al. (2020) Pfeiffer, J.; Rücklé, A.; Poth, C.; Kamath, A.; Vulić, I.; Ruder, S.; Cho, K.; and Gurevych, I. 2020.  Adapterhub: A framework for adapting transformers.  _arXiv preprint arXiv:2007.07779_.
  * Radford et al. (2019) Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I.; et al. 2019\.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8): 9.
  * Roberts, Raffel, and Shazeer (2020) Roberts, A.; Raffel, C.; and Shazeer, N. 2020.  How Much Knowledge Can You Pack Into the Parameters of a Language Model?  In _Empirical Methods in Natural Language Processing (EMNLP)_.
  * Su et al. (2020) Su, W.; Zhu, X.; Cao, Y.; Li, B.; Lu, L.; Wei, F.; and Dai, J. 2020.  VL-BERT: Pre-training of Generic Visual-Linguistic Representations.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Wang et al. (2020) Wang, Y.; Joty, S.; Lyu, M. R.; King, I.; Xiong, C.; and Hoi, S. C. 2020.  Vd-bert: A unified vision and dialog transformer with bert.  _arXiv preprint arXiv:2004.13278_.
  * Williams (1992) Williams, R. J. 1992.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  _Machine learning_ , 8(3): 229–256.
  * Wu et al. (2018) Wu, L.; Tian, F.; Qin, T.; Lai, J.; and Liu, T.-Y. 2018.  A study of reinforcement learning for neural machine translation.  _arXiv preprint arXiv:1808.08866_.
  * Wu et al. (2017) Wu, Q.; Teney, D.; Wang, P.; Shen, C.; Dick, A.; and van den Hengel, A. 2017.  Visual question answering: A survey of methods and datasets.  _Computer Vision and Image Understanding_ , 163: 21–40.
  * Wu et al. (2019) Wu, Y.; Kirillov, A.; Massa, F.; Lo, W.-Y.; and Girshick, R. 2019.  Detectron2.  https://github.com/facebookresearch/detectron2.
  * Zhou et al. (2017) Zhou, B.; Lapedriza, A.; Khosla, A.; Oliva, A.; and Torralba, A. 2017.  Places: A 10 million Image Database for Scene Recognition.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
  * Zhu et al. (2021) Zhu, F.; Lei, W.; Wang, C.; Zheng, J.; Poria, S.; and Chua, T.-S. 2021.  Retrieving and reading: A comprehensive survey on open-domain question answering.  _arXiv preprint arXiv:2101.00774_.
  * Ziaeefard and Lécué (2020) Ziaeefard, M.; and Lécué, F. 2020.  Towards Knowledge-Augmented Visual Question Answering.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , 1863–1873.


* [55] P. J. Forrester, _Large deviation eigenvalue density for the soft edge Laguerre and Jacobi $\beta$-ensembles_, J. Phys. A 45 (2012), 145201.
  * [56] P. J. Forrester, S. N. Majumdar and G. Schehr, _Non-intersecting Brownian walkers and Yang–Mills theory on the sphere_ , Nucl. Phys. B 844 (2011), 500–526.
  * [57] P. J. Forrester and E. M. Rains, _Matrix averages relating to Ginibre ensembles_ , J. Phys. A 42 (2009), 385205.
  * [58] P. J. Forrester and N. S. Witte, _Asymptotic forms for hard and soft edge general $\beta$ conditional gap probabilities_, Nuclear Phys. B 859, (2012), 321–340.
  * [59] Y. V. Fyodorov, _On statistics of bi-orthogonal eigenvectors in real and complex Ginibre ensembles: combining partial Schur decomposition with supersymmetry_ , Comm, Math. Phys. 363 (2018), 579–603.
  * [60] S. Ghosh and A. Nishry, _Point processes, hole events, and large deviations: random complex zeros and Coulomb gases_ , Constr. Approx. 48 (2018), 101–136.
  * [61] J. Grela, _Diffusion method in Random Matrix Theory_ , J. Phys. A 49 (2016), 015201.
  * [62] A. Guionnet and J. Huang, _Asymptotics of rectangular spherical integrals_ , J. Funct. Anal. 285 (2023), 110144.
  * [63] A. Hassel A and S. Zelditch, _Determinants of. Laplacians in exterior domains_ , Int. Math. Res. Not. 1999 (1999), 971–1004.
  * [64] H. Hedenmalm, _Soft Riemann-Hilbert problems and planar orthogonal polynomials_ , Comm. Pure Appl. Math. (to appear), arXiv:2108.05270.
  * [65] H. Hedenmalm and A. Wennman, _Off-spectral analysis of Bergman kernels_ , Comm. Math. Phys. 373 (2020), 1049–1083.
  * [66] H. Hedenmalm and A. Wennman, Planar orthogonal polynomials and boundary universality in the random normal matrix model, Acta Math. 227 (2021), 309–406.
  * [67] J. Jalowy, _The Wasserstein distance to the circular law_ , Ann. Inst. H. Poincaré Probab. Stat. 59 (2023), 2285–2307.
  * [68] B. Jancovici, G. Manificat and C. Pisani, _Coulomb systems seen as critical systems: finite-size effects in two dimensions_ , J. Stat. Phys. 76 (2019), 307–329.
  * [69] K. Johansson and F. Viklund, _Coulomb gas and the Grunsky operator on a Jordan domain with corners_ , arXiv:2309.00308.
  * [70] N.-G. Kang and N. Makarov, _Gaussian free field and conformal field theory_ , Astérisque 353 (2013), viii+136.
  * [71] M. Katori, M. Izumi and N. Kobayashi, _Two Bessel bridges conditioned never to collide, double Dirichlet series, and Jacobi theta function_ , J. Stat. Phys. 131 (2008), 1067–1083.
  * [72] E. Katzav and I. P. Castillo, _Large deviations of the smallest eigenvalue of the Wishart–Laguerre ensemble_ , Phys. Rev. E 82 (2010), 040104.
  * [73] S. Klevtsov, _Lowest Landau level on a cone and zeta determinants_ , J. Phys. A 50 (2017), 234003.
  * [74] I. V. Krasovsky, _Correlations of the characteristic polynomials in the Gaussian unitary ensemble or a singular Hankel determinant_ , Duke Math. J. 139 (2007), 581–619.
  * [75] T. Krüger, S.-Y. Lee and M. Yang, _Local statistics in normal matrix models with merging singularity_ , arXiv:2306.12263.
  * [76] S. Kumar, _Recursion for the smallest eigenvalue density of $\beta$-Wishart–Laguerre ensemble_, J. Stat. Phys. 175 (2019), 126–149.
  * [77] G. Lambert, _Maximum of the characteristic polynomial of the Ginibre ensemble_ , Comm. Math. Phys. 378 (2020), 943–985.
  * [78] G. Lambert, D. Ostrovsky and N. Simm, _Subcritical multiplicative chaos for regularized counting statistics from random matrix theory_ , Comm. Math. Phys. 360 (2018), 1–54.
  * [79] G. Lawler, O. Schramm and W. Werner, _Conformal restriction: the chordal case_ , J. Amer. Math. Soc. 16 (2003), 917–955.
  * [80] G. Lawler and W. Werner, _The Brownian loop soup_ , Probab. Theory Related Fields 128 (2004), 565–588.
  * [81] T. Leblé and S. Serfaty, _Large deviation principle for empirical fields of log and Riesz gases_ , Invent. Math. 210 (2017), 645–757.
  * [82] S.-Y. Lee and N. Makarov, _Topology of quadrature domains_ , J. Amer. Math. Soc. 29 (2016), 333–369.
  * [83] S.-Y. Lee and M. Yang, _Discontinuity in the asymptotic behavior of planar orthogonal polynomials under a perturbation of the Gaussian weight_ , Comm. Math. Phys. 355 (2017), 303–338.
  * [84] S.-Y. Lee and M. Yang, _Planar orthogonal polynomials as Type II multiple orthogonal polynomials_ , J. Phys. A 52 (2019), 275202.
  * [85] S.-Y. Lee and M. Yang, _Strong asymptotics of planar orthogonal polynomials: Gaussian weight perturbed by finite number of point charges_ , Comm. Pure Appl. Math. 76 (2023), 2888–2956.
  * [86] D.-Z. Liu and L. Zhang, _Phase transition of eigenvalues in deformed Ginibre ensembles_ , arXiv:2204.13171.
  * [87] S. N. Majumdar and G. Schehr, _Top eigenvalue of a random matrix: Large deviations and third order phase transition_ , J. Stat. Mech. 2014 (2014) P01012.
  * [88] S. N. Majumdar and M. Vergassola, _Large Deviations of the Maximum Eigenvalue for Wishart and Gaussian Random Matrices_ , Phys. Rev. Lett. 102 (2009), 060601.
  * [89] A. Martínez-Finkelshtein and G. L. F. Silva, _Critical measures for vector energy: asymptotics of nondiagonal multiple orthogonal polynomials for a cubic weight_ , Adv. Math. 349 (2019), 246–315.
  * [90] S. M. Nishigaki and A. Kamenev, _Replica treatment of non-Hermitian disordered Hamiltonians_ , J. Phys. A 35, (2002), 4581.
  * [91] F. W. J. Olver, D. W. Lozier, R. F. Boisvert, and C. W. Clark, eds. _NIST Handbook of Mathematical Functions_ , Cambridge: Cambridge University Press, 2010.
  * [92] E. Peltola and Y. Wang, _Large deviations of multichordal SLE 0+, real rational functions, and zeta-regularized determinants of Laplacians_, J. Eur. Math. Soc. 26 (2024), 469–535.
  * [93] A. Perret and G. Schehr, _Finite $N$ corrections to the limiting distribution of the smallest eigenvalue of Wishart complex matrices_, Random Matrices Theory Appl. 5 (2016), 1650001.
  * [94] A. M. Polyakov, _Quantum geometry of bosonic strings_ , Phys. Lett. B 103 (1981), 207–210.
  * [95] R. Rhodes and V. Vargas, _Gaussian multiplicative chaos and applications: a review_ , Probab. Surv. 11 (2014), 315–392.
  * [96] E. B. Saff and V. Totik, _Logarithmic Potentials with External Fields_ , Grundlehren der Mathematischen Wissenschaften, Springer-Verlag, Berlin, 1997.
  * [97] M. Sakai, _Regularity of a boundary having a Schwarz function_ , Acta Math. 166 (1991), 263–297.
  * [98] G. Schehr, S. N. Majumdar, A. Comtet and P. J. Forrester, _Reunion probability of $N$ vicious walkers: typical and large fluctuations for large $N$_, J. Stat. Phys. 150 (2013) 491–530.
  * [99] A. Serebryakov and N. Simm, _Schur function expansion in non-Hermitian ensembles and averages of characteristic polynomials_ , arXiv:2310.20686.
  * [100] A. Serebryakov and N. Simm, _Characteristic polynomials of random truncations: moments, duality and asymptotics_ , With an Appendix by G. Dubach. Random Matrices Theory Appl. 12 (2023), 2250049.
  * [101] S. Serfaty, _Gaussian fluctuations and free energy expansion for Coulomb gases at any temperature_ , Ann. Inst. Henri Poincaré Probab. Stat. 59 (2023), 1074–1142.
  * [102] L. A. Takhtajan and L.-P., Teo, _Weil-Petersson metric on the universal Teichmüller space_ , Mem. Amer. Math. Soc. 183 (2006), no. 861.
  * [103] G. Téllez and P J. Forrester, _Exact finite-size study of the 2D OCP at $\Gamma=4$ and $\Gamma=6$_, J. Stat. Phys. 97 (1999), 489–521.
  * [104] P. Vivo, S. N. Majumdar and O. Bohigas, _Large deviations of the maximum eigenvalue in Wishart random matrices_ , J. Phys. A 40 (2007), 4317–4337.
  * [105] Y. Wang, _The energy of a deterministic Loewner chain: reversibility and interpretation via SLE 0+_, J. Eur. Math. Soc. 21 (2019), 1915–1941.
  * [106] Y. Wang, _Equivalent descriptions of the Loewner energy_ , Invent. Math. 218 (2019), 573–621.
  * [107] C. Webb, _The characteristic polynomial of a random unitary matrix and Gaussian multiplicative chaos–the $L^{2}$-phase_, Electron. J. Probab. 20 (2015), 1–21.
  * [108] C. Webb and M. D. Wong, _On the moments of the characteristic polynomial of a Ginibre random matrix_ , Proc. Lond. Math. Soc. 118 (2019), 1017–1056.
  * [109] P. Wiegmann and A. Zabrodin, _Dyson gas on a curved contour_ , J. Phys. A 55 (2022), 165202.
  * [110] T. Wirtz, M. Kieburg and T. Guhr, _Limiting statistics of the largest and smallest eigenvalues in the correlated Wishart model_ , Europhys. Lett. 109 (2015), 20005.
  * [111] T. Wirtz and T. Guhr, _Distribution of the smallest eigenvalue in the correlated Wishart model_ , Phys. Rev. Lett. 111 (2013), 094101.
  * [112] T. Wirtz and T. Guhr, _Distribution of the smallest eigenvalue in complex and real correlated Wishart ensembles_ , J. Phys. A 47 (2014), 075004.
  * [113] A. Zabrodin and P. Wiegmann, _Large- $N$ expansion for the 2D Dyson gas_, J. Phys. A 39 (2006), 8933–8964.