# On a conjecture of Elliott concerning a probabilistic variant of
Titchmarsh’s divisor problem

Ofir Gorodetsky  Mathematical Institute, University of Oxford, Oxford, OX2
6GG, UK<EMAIL_ADDRESS>and  Lasse Grimmelt  Mathematical
Institute, University of Oxford, Oxford, OX2 6GG, UK
<EMAIL_ADDRESS>
###### Abstract.

Elliott and Halberstam proved that $\sum_{p<n}2^{\omega(n-p)}$ is asymptotic
to $\phi(n)$. In analogy to the Erdős–Kac Theorem, Elliott conjectured that if
one restricts the summation to primes $p$ such that $\omega(n-p)\leq 2\log\log
n+\lambda(2\log\log n)^{1/2}$ then the sum will be asymptotic to
$\phi(n)\int_{-\infty}^{\lambda}e^{-t^{2}/2}dt/\sqrt{2\pi}$. We show that this
conjecture follows from the Bombieri–Vinogradov theorem. We further prove a
related result involving Poisson–Dirichlet distribution, employing deeper
lying level of distribution results of the primes.

This project has received funding from the European Research Council (ERC)
under the European Union’s Horizon 2020 research and innovation programme
(grant agreement No 851318).

##  1\. Introduction and motivation

Let $\omega(m)=\sum_{p\mid m}1$ be the prime divisor function. Elliott and
Halberstam proved that [EH66]

(1.1) $\sum_{p<n}2^{\omega(n-p)}=\phi(n)+O\left(\frac{n}{\log n}(\log\log
n)^{2}\right)$

holds for $n\geq 10$. Here and later $p$ denotes a prime and $\phi$ is Euler’s
totient function. Elliott gave a talk at the meeting held in Urbana–Champaign,
June 5–7, 2014, in memory of Paul and Felice Bateman, and Heini Halberstam,
where he revisited his works with Halberstam. A telegraphic representation of
the talk was published, which contains the following conjecture [Ell15, Conj.
B]: for each real $\lambda$,

(1.2) $\sum_{\begin{subarray}{c}p<n\\\ \omega(n-p)\leq 2\log\log
n+\lambda(2\log\log
n)^{1/2}\end{subarray}}2^{\omega(n-p)}\to\frac{\phi(n)}{\sqrt{2\pi}}\int_{-\infty}^{\lambda}e^{-u^{2}/2}du$

as $n\to\infty$. In this note we establish the conjecture.

######  Theorem 1.1.

The asymptotic relation (1.2) holds for each real $\lambda$.

Theorem 1.1 can be restated as follows. Let $X_{n}$ be a random variable
taking the value $m\in\\{1,2,\ldots,n\\}$ with probability proportional to
$2^{\omega(m)}\mathbf{1}_{n-m\textup{ a prime}}$. Then

(1.3) $\frac{\omega(X_{n})-2\log\log n}{(2\log\log
n)^{1/2}}\overset{d}{\longrightarrow}N(0,1),\qquad n\to\infty,$

where $N(0,1)$ is the standard gaussian distribution and the arrow indicates
convergence in distribution. The proof of Theorem 1.1 will follow from

######  Proposition 1.2.

Fix $k,B\geq 1$. For any squarefree $R\in[\exp((\log\log n)^{2}),n^{1/100}]$
we have

(1.4) $\sum_{\begin{subarray}{c}a\mid R\\\ \omega(a)\leq k\\\
(a,n)=1\end{subarray}}\bigg{|}\mathbb{P}(a\textup{ divides
}X_{n})-\frac{2^{\omega(a)}}{a}\bigg{|}\ll_{k,B}\frac{n}{\phi(n)}\frac{\log
R}{\log n}\sum_{\begin{subarray}{c}a\mid R\\\ \omega(a)\leq k\\\
(a,n)=1\end{subarray}}\frac{1}{a}+\frac{1}{\log^{B}n}.$

The reduction of Theorem 1.1 to Proposition 1.2 is explained in §3. The proof
of Proposition 1.2 is given in §4 and uses the classical tools utilized in
[EH66] (the Brun–Titchmarsh and Bombieri–Vinogradov Theorems). Proposition 1.2
has little content when $\log R\asymp\log n$. This is remedied by the
following proposition, proved using recent work on primes in arithmetic
progression [BGLM21].

######  Proposition 1.3.

There is an absolute constant $\eta>0$ with the following property. For every
$k\geq 1$,

(1.5) $\sum_{\begin{subarray}{c}1\leq a\leq n^{\eta}\\\ \omega(a)\leq k\\\
(a,n)=1\end{subarray}}\mu^{2}(a)\bigg{|}\mathbb{P}(a\textup{ divides
}X_{n})-\frac{2^{\omega(a)}}{a}+\frac{2D_{a}}{\phi(a)\log
n}\bigg{|}\ll_{k}\frac{(\log\log n)^{C_{k}}}{\log n}$

where

(1.6) $D_{a}:=\sum_{d\mid a^{\infty}}g(d)\log d,\qquad
g(d):=\prod_{p^{k}\mid\mid d}(-1)^{k-1}(p-2)(p-1)^{-k}.$

Since $\sum_{d\mid a^{\infty}}|g(d)|=2^{\omega(a)-\mathbf{1}_{2\mid a}}$,

(1.7) $D_{a}\leq\sum_{p\mid a}\sum_{i\geq 1}\log(p^{i})\sum_{p^{i}\mid\mid
d\mid a^{\infty}}|g(d)|\leq 2^{\omega(a)}\sum_{p\mid a}\sum_{i\geq
1}\log(p^{i})|g(p^{i})|\ll 2^{\omega(a)}\log a.$

Proposition 1.3 is proved in §5. The lower order term in (1.5) leads to the
following application. Given an integer $m>1$ we write it as
$m=\prod_{i=1}^{\Omega(m)}p_{i}(m)$ where $\Omega(m)$ is the number of prime
factors of $m$ (multiplicity counted) and $p_{1}(m)\geq\ldots\geq
p_{\Omega(m)}$. We define

$V(m):=\left(\frac{\log p_{i}(m)}{\log m}\right)_{i\geq 1}$

where $p_{i}(m):=1$ for $i>\Omega(m)$. Let $\textup{PD}(\theta)$ be the
Poisson–Dirichlet distribution with parameter $\theta$ (see Chapter 1 of
[Bil99] for its definition).

######  Corollary 1.4.

If $V(X_{n})$ tends in distribution to $\textup{PD}(\theta)$ for some
$\theta>0$ then $\theta=2$.

Let $\widetilde{X}_{n}$ be the random variable taking the value
$m\in\\{1,2,\ldots,n\\}$ with probability proportional to $2^{\omega(m)}$. As
observed by Elliott [Ell15],

(1.8) $\frac{\omega(\widetilde{X}_{n})-2\log\log n}{(2\log\log
n)^{1/2}}\overset{d}{\longrightarrow}N(0,1),\qquad n\to\infty.$

See §6 for a self-contained proof of (1.8). As shown in [EG19],
$V(\widetilde{X}_{n})$ tends to $\textup{PD}(2)$. We expect Theorem 1.1 and
Corollary 1.4 to hold because $X_{n}$ is distributed like $\widetilde{X}_{n}$,
only with support restricted to shifted primes which are believed to behave in
most ways like random integers.

The structure of the paper is as follows. In the next short section we show
how Corollary 1.4 follows from Proposition 1.3. Afterwards in §3 we reduce the
conjecture of Elliot we prove, Theorem 1.1 to Proposition 1.2. In §4 we show
how Proposition 1.2 follows from the classical Bombieri–Vinogradov theorem and
the Brun–Titchmarsh inequality. At this point the proof of Theorem 1.1 is
complete. In the penultimate §5 we go beyond the Bombieri–Vinogradov range by
employing a result in [BGLM21], based on spectral theory of automorphic forms,
to prove Proposition 1.3. As mentioned in the last paragraph, we conclude the
paper with a short proof of (1.8).

##  2\. Proof of Corollary 1.4

We now show how Corollary 1.4 follows from Proposition 1.3. Suppose
$V(X_{n})\overset{d}{\longrightarrow}\textup{PD}(\theta)$, and in particular
$V(X_{n})_{i}\overset{d}{\longrightarrow}\textup{PD}(\theta)_{i}$ for all $i$.
Since $2^{\omega(n-p)}=n^{o(1)}$, we have $\mathbb{P}(X_{n}\leq T)\ll
T/n^{1+o(1)}$, so that $\log X_{n}/\log n$ tends to $1$ in probability. This
implies
$\widetilde{V}(X_{n})_{i}\overset{d}{\longrightarrow}\textup{PD}(\theta)_{i}$
for $\widetilde{V}(X_{n})_{i}:=\log p_{i}(X_{n})/\log n$. By [Han09, Thm.
2.1],

$\lim_{n\to\infty}\mathbb{E}\sum_{i=1}^{\infty}\mathbf{1}_{\widetilde{V}(X_{n})_{i}\in[a,b]}=\theta\int_{a}^{b}\frac{(1-t)^{\theta-1}}{t}dt$

for all $0<a<b<1$ (the sum appears infinite but can be truncated at $i=\lfloor
k/a\rfloor$ since $\sum_{i\geq 1}\widetilde{V}(X_{n})_{i}\leq 1$). This is
simplified as

(2.1) $\lim_{n\to\infty}\sum_{p\in[n^{a},n^{b}]}\sum_{k\geq
1}\mathbb{P}(p^{k}\text{ divides
}X_{n})=\theta\int_{a}^{b}\frac{(1-t)^{\theta-1}}{t}dt.$

Since $\mathbb{P}(d\text{ divides }X_{n},\,X_{n}\leq T)\ll T/(n^{1+o(1)}d)$,
(2.1) is equivalent to

(2.2) $\lim_{n\to\infty}\sum_{n^{a}\leq p\leq n^{b}}\mathbb{P}(p\text{ divides
}X_{n})=\theta\int_{a}^{b}\frac{(1-t)^{\theta-1}}{t}dt.$

By Proposition 1.3 with $k=1$ we have

(2.3) $\lim_{n\to\infty}\sum_{n^{a}\leq p\leq n^{b}}\mathbb{P}(p\text{ divides
}X_{n})=2\int_{a}^{b}\frac{1-t}{t}dt$

by Mertens’ theorems, as long as $b<\eta$. This is consistent with (2.2) only
if $\theta=2$.

##  3\. Reduction

We now reduce Theorem 1.1 to Proposition 1.2. In other words, we make precise
the well-known principle (see for example [GS07]) that Erdős–Kac type results
can be deduced from level of distribution information.

Let

(3.1) $C_{k}:=\mathbb{E}X^{k},\qquad C^{\prime}_{k}:=\mathbb{E}|X|^{k},$

where $X\sim N(0,1)$. It is known that for (1.3) to hold it suffices to show
that

(3.2) $\mathbb{E}\left(\frac{\omega(X_{n})-2\log\log n}{(2\log\log
n)^{1/2}}\right)^{k}\to C_{k},\qquad n\to\infty,$

holds for every integer $k\geq 1$ [Dur19, Ch. 3.3.5]. To approach (3.2) we use

######  Proposition 3.1.

[GS07] Let $f\colon\mathbb{N}\to\mathbb{R}_{\geq 0}$ be an arithmetic function
and $\mathcal{P}$ be a squarefree integer. Let
$h\colon\mathbb{N}\to\mathbb{R}_{\geq 0}$ be a multiplicative function with
$h(p)\leq p$ for $p|\mathcal{P}$. For $a|\mathcal{P}$ define $r_{a}$ via

$\sum_{m\leq n:\,a\mid m}f(m)=\frac{h(a)}{a}\sum_{m\leq n}f(m)+r_{a}.$

Let

$\mu_{\mathcal{P}}=\sum_{p\mid\mathcal{P}}\frac{h(p)}{p},\qquad\sigma^{2}_{\mathcal{P}}=\sum_{p\mid\mathcal{P}}\frac{h(p)}{p}\left(1-\frac{h(p)}{p}\right).$

Uniformly for $1\leq k\leq\sigma_{\mathcal{P}}^{2/3}$ we have

(3.3) $\sum_{m=1}^{n}f(m)\big{(}\sum_{p\mid\mathcal{P}}\mathbf{1}_{p\mid
m}-\mu_{\mathcal{P}}\big{)}^{k}=\sigma_{\mathcal{P}}^{k}\sum_{m=1}^{n}f(m)\big{(}C_{k}+O((k^{3/2}\sigma_{\mathcal{P}}^{-1})^{1+\mathbf{1}_{2\mid
k}}C^{\prime}_{k})\big{)}+O\bigg{(}\mu_{\mathcal{P}}^{k}\sum_{\begin{subarray}{c}\omega(a)\leq
k\\\ a\mid\mathcal{P}\end{subarray}}|r_{a}|\bigg{)},$

where $C_{k}$, $C^{\prime}_{k}$ are given by (3.1).

For $f\equiv 1$ Proposition 3.1 is [GS07, Prop. 3] and the proof works the
same way for general $f\geq 0$. We derive from Proposition 3.1 the following
criterion, to be proved below.

######  Corollary 3.2.

For every sufficiently large $n$ let $Y_{n}$ be a random variable taking
values in $\\{1,2,\ldots,n\\}$ and set

$I_{n}:=(\exp(\exp(\log^{1/3}\log n)),n^{\exp(-\log^{1/3}\log n)}),\quad
T:=n^{1/\log^{1/3}\log n},\quad\mathcal{P}:=\prod_{p\in I_{n}}p.$

Fix $k\geq 1$. Suppose that

(3.4) $\sum_{\begin{subarray}{c}\omega(a)\leq k\\\
a\mid\prod_{p<T}p/\mathcal{P}\end{subarray}}\mathbb{P}(a\textup{ divides
}Y_{n})=o\big{(}(\log\log n)^{k/2}\big{)}$

as $n\to\infty$, and that for some multiplicative function
$g\colon\mathbb{N}\to[0,1]$,

(3.5) $\sum_{\begin{subarray}{c}\omega(a)\leq k\\\
a\mid\mathcal{P}\end{subarray}}\bigg{|}\mathbb{P}(a\textup{ divides
}Y_{n})-g(a)\bigg{|}=o\big{(}(\log\log n)^{-k/2}\big{)}$

as $n\to\infty$. If $\sum_{p\leq x}g(p)=\theta\log\log x+O(1)$ for some
$\theta>0$ and $\sum_{p\leq x}g^{2}(p)=O(1)$ then

$\mathbb{E}\bigg{(}\frac{\omega(Y_{n})-\theta\log\log n}{(\theta\log\log
n)^{1/2}}\bigg{)}^{k}\to C_{k}$

as $n\to\infty$.

###### Proof.

We consider a given $n$ and apply Proposition 3.1 with
$f(m)=\mathbb{P}(Y_{n}=m)$, $h(m)=g(m)/m$. Define

(3.6) $\omega_{\mathcal{P}}(n)=\sum_{p\mid\mathcal{P}}\mathbf{1}_{p\mid
n},\quad\mu=\sum_{p\mid\mathcal{P}}g(p),\quad\sigma^{2}=\sum_{p\mid\mathcal{P}}g(p)(1-g(p)),\quad
R_{a}=\mathbb{P}(a\textup{ divides }Y_{n})-g(a).$

We find that

(3.7)
$\sum_{m=1}^{n}\mathbb{P}(Y_{n}=m)\left(\frac{\omega_{\mathcal{P}}(m)-\mu}{\sigma}\right)^{k}=\sum_{m=1}^{n}\mathbb{P}(Y_{n}=m)\big{(}C_{k}+O(\sigma^{-1-\mathbf{1}_{2\mid
k}})\big{)}+O\bigg{(}\frac{\mu^{k}}{\sigma^{k}}\sum_{\begin{subarray}{c}\omega(a)\leq
k\\\ a\mid\mathcal{P}\end{subarray}}|R_{a}|\bigg{)}$

holds for every fixed $k\geq 1$. By our assumptions on $g$ we have

(3.8) $\mu,\sigma^{2}=\theta\log\log n+O(\log^{1/3}\log n).$

We express (3.7) as

(3.9)
$\displaystyle\mathbb{E}\left(\frac{\omega_{\mathcal{P}}(Y_{n})-\mu}{\sigma}\right)^{k}-C_{k}\ll(\log\log
n)^{-1/2}+(\log\log n)^{k/2}\sum_{\begin{subarray}{c}\omega(a)\leq k\\\
a\mid\mathcal{P}\end{subarray}}|R_{a}|.$

By (3.5), the right-hand side of (3.9) goes to $0$ as $n\to\infty$. We have
the identity

$\frac{\omega(Y_{n})-\theta\log\log n}{(\theta\log\log
n)^{1/2}}=A_{1}A_{2}+A_{3}+A_{4}+A_{5}$

for

(3.10) $\displaystyle A_{1}$ $\displaystyle=\frac{\sigma}{(\theta\log\log
n)^{1/2}},\qquad A_{2}=\frac{\omega_{\mathcal{P}}(Y_{n})-\mu}{\sigma},\qquad
A_{3}=\frac{\mu-\theta\log\log n}{(\theta\log\log n)^{1/2}},$ (3.11)
$\displaystyle A_{4}$ $\displaystyle=\frac{\sum_{p\in[2,T)\setminus
I_{n}}\mathbf{1}_{p\mid Y_{n}}}{(\theta\log\log n)^{1/2}},\qquad
A_{5}=\frac{\sum_{p\in[T,n]}\mathbf{1}_{p\mid Y_{n}}}{(\theta\log\log
n)^{1/2}}$

for $T$ and $I_{n}$ as in the statement of the corollary. By (3.8), the
(constant) random variable $A_{1}$ and $A_{3}$ tend to $1$ and $0$,
respectively. We have observed $\mathbb{E}A_{2}^{k}\to C_{k}$. An integer $m$
can have at most $\log m/\log T$ prime factors of size at least $T$, implying
$A_{5}\ll(\log\log n)^{-1/6}$ and so $\mathbb{E}A_{5}^{k}\to 0$ as
$n\to\infty$. To handle $A_{4}$ we use (3.4) which yields

(3.12) $\mathbb{E}A_{4}^{k}=\frac{\sum_{p_{1},\ldots,p_{k}\in[2,T)\setminus
I_{n}}\mathbb{P}([p_{1},\ldots,p_{k}]\textup{ divides }Y_{n})}{(\theta\log\log
n)^{k/2}}\ll\frac{\sum_{\begin{subarray}{c}\omega(a)\leq k\\\
a\mid\prod_{p<T}p/\mathcal{P}\end{subarray}}\mathbb{P}(a\text{ divides
}Y_{n})}{(\log\log n)^{k/2}}=o(1).$

We have shown $\mathbb{E}|A_{3}+A_{4}+A_{5}|^{k}\to 0$ and
$\mathbb{E}(A_{1}A_{2})^{k}\to C_{k}$. By Hölder, (3.2) follows. ∎

Corollary 3.2 may also be proved by generalizing an argument of Billingsley
[Bil69]. To deduce Theorem 1.1 from Corollary 3.2 we apply it with
$Y_{n}=X_{n}$, $g(m)=2^{\omega(m)}/m$ and $\theta=2$. This reduces matters to
Proposition 1.2 with $R=\mathcal{P}$ and $R=\prod_{p<T}p/\mathcal{P}$ in the
notation of Corollary 3.2. The condition $(a,n)=1$ in Proposition 1.2 is
harmless since, uniformly for $a$ with $(a,n)>1$, $\mathbb{P}(a\textup{
divides }X_{n})\ll n^{-1+o(1)}$.

##  4\. Proof of Proposition 1.2

We have $\mathbb{P}(a\textup{ divides }X_{n})=A_{a}(n)/A_{1}(n)$ where

(4.1) $A_{a}(n):=\sum_{p<n:\,a\mid n-p}2^{\omega(n-p)}.$

Using (1.1) we see that Proposition 1.2 will follow if we show

(4.2) $\sum_{\begin{subarray}{c}a\mid R\\\ \omega(a)\leq k\\\
(a,n)=1\end{subarray}}\bigg{|}A_{a}(n)-\phi(n)\frac{2^{\omega(a)}}{a}\bigg{|}\ll_{k,B}n\frac{\log
R}{\log n}\sum_{\begin{subarray}{c}a\mid R\\\ \omega(a)\leq
k\end{subarray}}\frac{1}{a}+\frac{n}{\log^{B}n}.$

We write $2^{\omega(n-p)}$ as a divisor sum using $2^{\omega(m)}=\sum_{d\mid
m}\mu^{2}(d)$ and $\mu^{2}(d)=\sum_{e^{2}\mid d}\mu(e)$ and separate to
different ranges:

$2^{\omega(n)}=\sum_{n=ds}\mu^{2}(d)=\sum_{\begin{subarray}{c}n=ds\\\ d\leq
D\end{subarray}}\mu^{2}(d)+\sum_{\begin{subarray}{c}n=ds\\\
D<s<n/D\end{subarray}}\mu^{2}(d)+\sum_{\begin{subarray}{c}n=e^{2}ds\\\ s\leq
D\\\ e\leq V\end{subarray}}\mu(e)+\sum_{\begin{subarray}{c}n=e^{2}ds\\\ s\leq
D\\\ e>V\end{subarray}}\mu(e),$

with $D,V\in[1,n]$ to be determined later. We split $A_{a}(n)$ accordingly:

(4.3) $\displaystyle A_{a}(n)$ $\displaystyle=\sum_{\begin{subarray}{c}p<n\\\
a\mid n-p\end{subarray}}\bigg{(}\sum_{\begin{subarray}{c}n-p=ds\\\ d\leq
D\end{subarray}}\mu^{2}(d)+\sum_{\begin{subarray}{c}n-p=ds\\\
D<s<(n-p)/D\end{subarray}}\mu^{2}(d)+\sum_{\begin{subarray}{c}n-p=e^{2}ds\\\
s\leq D\\\ e\leq
V\end{subarray}}\mu(e)+\sum_{\begin{subarray}{c}n-p=e^{2}ds\\\ s\leq D\\\
e>V\end{subarray}}\mu(e)\bigg{)}$ (4.4)
$\displaystyle=:S_{0}(a)+S_{1}(a)+S_{2}(a)+S_{3}(a).$

writing $\pi(x;q,b)$ for the number of primes in $[1,x]$ that are congruent to
$b$ modulo $q$, we have

(4.5) $\displaystyle S_{0}(a)$ $\displaystyle=\sum_{d\leq
D}\mu^{2}(d)\pi(n-1;[d,a],n),\qquad
S_{1}(a)=\sum_{D<s<(n-2)/D}\sum_{\begin{subarray}{c}p<n-Ds\\\ p\equiv
n([a,s])\end{subarray}}\mu^{2}((n-p)/s),$ (4.6) $\displaystyle S_{2}(a)$
$\displaystyle=\sum_{\begin{subarray}{c}s\leq D\\\ e\leq
V\end{subarray}}\mu(e)\pi(n-1;[se^{2},a],n),\qquad
S_{3}(a)=\sum_{\begin{subarray}{c}s\leq D\\\
e>V\end{subarray}}\mu(e)\pi(n-1;[se^{2},a],n).$

Write
$E(m;q,a)=\pi(m;q,a)-\tfrac{\textup{Li}(m)\mathbf{1}_{(q,a)=1}}{\varphi(a)}$
and accordingly $S_{i}(a)=M_{i}(a)+E_{i}(a)$ ($i=0,2$) for

(4.7) $\displaystyle M_{0}(a)$
$\displaystyle=\textup{Li}(n-1)\sum_{\begin{subarray}{c}d\leq D\\\
(d,n)=1\end{subarray}}\frac{\mu^{2}(d)}{\phi([d,a])},\qquad
E_{0}(a)=\sum_{d\leq D}\mu^{2}(d)E(n-1;[d,a],n),$ (4.8) $\displaystyle
M_{2}(a)$ $\displaystyle=\textup{Li}(n-1)\sum_{\begin{subarray}{c}s\leq D\\\
e\leq V\\\ (se,n)=1\end{subarray}}\frac{\mu(e)}{\phi([se^{2},a])},\qquad
E_{2}(a)=\sum_{\begin{subarray}{c}s\leq D\\\ e\leq
V\end{subarray}}\mu(e)E(n-1;[se^{2},a],n).$

We first observe some basic properties of $\phi$. We have

(4.9) $\phi(n_{1})\phi(n_{2})=\phi((n_{1},n_{2}))\phi([n_{1},n_{2}])$

and

(4.10) $\phi(n_{1})\phi(n_{2})\leq\phi(n_{1}n_{2})$

for every $n_{1},n_{2}\geq 1$. We also have

(4.11) $\sum_{n\leq A}1/\phi(n)=C\log A+O(1)$

as shown in [MV07, p. 43]. The estimate

(4.12) $\sum_{n>A}1/\phi(n^{2})\ll 1/A$

holds for each $A>0$, since it is equivalent to $\sum_{A\leq
n<2A}\tfrac{n^{2}}{\phi(n^{2})}\ll A$ which follows from [MV07, Thm. 2.14].
The following lemmas estimate the contributions of $S_{i}$ to (4.2) and will
be proved below.

######  Lemma 4.1.

Fix $k\geq 1$. Suppose $D,V\in[1,n]$ satisfy $D\geq Rn^{1/4}$ and $DV^{2}R\leq
n^{3/4}$. Then

(4.13) $\displaystyle\sum_{\begin{subarray}{c}a\mid R\\\ \omega(a)\leq
k\end{subarray}}|S_{1}(a)|+|S_{3}(a)|\ll_{k}\bigg{(}\frac{n}{V}+\frac{n}{\log
n}\log(2+n/D^{2})\bigg{)}\sum_{\begin{subarray}{c}a\mid R\\\ \omega(a)\leq
k\end{subarray}}\frac{1}{a}.$

######  Lemma 4.2.

Suppose $DV^{2}R\ll_{B}\sqrt{n}(\log n)^{-B}$ for every $B>0$. Then for every
$B>0$,

$\sum_{a\leq R}|E_{0}(a)|+|E_{2}(a)|\ll_{B}n(\log n)^{-B}.$

######  Lemma 4.3.

Suppose $D,V\in[1,n]$. Then for $1\leq a\leq n$ with $\omega(a)\leq k$,

$M_{2}(a)-M_{0}(a)\ll_{k}\frac{n}{aV}+\frac{n}{a\log n}.$

######  Lemma 4.4.

Uniformly for $a\leq n^{1/100}$ and $X\geq n^{1/3}$ with $(a,n)=1$,
$\mu^{2}(a)=1$ and $\omega(a)\leq k$,

(4.14) $\sum_{\begin{subarray}{c}m\leq X\\\
(m,n)=1\end{subarray}}\frac{\mu^{2}(m)}{\phi([m,a])}=\frac{2^{\omega(a)}}{a}\left(\frac{\phi(n)}{n}\log
X+O_{k}((\log\log n)^{2})\right)-\frac{\phi(n)}{n}\frac{D_{a}}{\phi(a)}.$

Note that $M_{0}(a)$ coincides with $\textup{Li}(n-1)$ times the left-hand
side of (4.14) with $X=D$. Applying the last 4 lemmas with $V=\log n$ and
$X=D=\sqrt{n}/R^{2}$ we obtain

(4.15) $\sum_{\begin{subarray}{c}a\mid R\\\ \omega(a)\leq k\\\
(a,n)=1\end{subarray}}\bigg{|}A_{a}(n)-\phi(n)\frac{2^{\omega(a)}}{a}+\mathrm{Li}(n-1)\frac{2D_{a}\phi(n)}{n\phi(a)}\bigg{|}\ll_{k,B}\frac{n}{\log
n}\sum_{\begin{subarray}{c}a\mid R\\\ \omega(a)\leq
k\end{subarray}}\frac{1}{a}+\frac{n}{\log^{B}n}$

for $R\in[\exp((\log\log n)^{2}),n^{1/100}]$. The contribution of the lower
order term $\textup{Li}(n-1)2D_{a}\phi(n)/(n\phi(a))$ can be absorbed in the
error using (1.7), giving (4.2).

###  4.1. Proof of Lemma 4.1

In estimating $S_{1}(a)$ we use the Brun–Titchmarsh Theorem:

(4.16) $\sum_{\begin{subarray}{c}a\mid R\\\ \omega(a)\leq
k\end{subarray}}|S_{1}(a)|\leq\sum_{\begin{subarray}{c}a\mid R\\\
\omega(a)\leq k\\\
D<s<\frac{n}{D}\end{subarray}}\pi(n-1;[a,s],n)\ll\frac{n}{\log
n}\sum_{\begin{subarray}{c}a\mid R\\\ \omega(a)\leq k\\\
D<s<\frac{n}{D}\end{subarray}}\frac{1}{\phi([a,s])}.$

The theorem saves $\log n$ since $[a,s]<Rn/D\leq n^{3/4}$ by assumption. Let
$g=(a,s)$ change the order of summation:

(4.17) $\displaystyle\sum_{\begin{subarray}{c}a\mid R\\\ \omega(a)\leq k\\\
D<s<\frac{n}{D}\end{subarray}}\frac{1}{\phi([a,s])}=\sum_{g}\sum_{\begin{subarray}{c}a\mid
R\\\ \omega(a)\leq k\\\ g\mid
a\end{subarray}}\frac{1}{\phi(a)}\sum_{\begin{subarray}{c}D<s<\frac{n}{D}\\\
g\mid s\end{subarray}}\frac{\phi(g)}{\phi(s)}\ll\sum_{\begin{subarray}{c}g\mid
a\mid R\\\ \omega(a)\leq
k\end{subarray}}\frac{\log(2+n/D^{2})}{\phi(a)}\ll_{k}\log(2+n/D^{2})\sum_{\begin{subarray}{c}a\mid
R\\\ \omega(a)\leq k\end{subarray}}\frac{1}{a}.$

In the first equality we used (4.9) and in the first inequality we used
(4.10)–(4.11). To estimate $S_{3}(a)$ we use the Brun–Titchmarsh Theorem when
$e\leq V\log n$ and otherwise $\pi(n-1,m,n)\leq n/m$:

(4.18) $\sum_{\begin{subarray}{c}a\mid R\\\ \omega(a)\leq
k\end{subarray}}|S_{3}(a)|\ll\frac{n}{\log n}\sum_{\begin{subarray}{c}a\mid
R\\\ \omega(a)\leq k\end{subarray}}\sum_{\begin{subarray}{c}s\leq D\\\
e>V\end{subarray}}\frac{1}{\phi([se^{2},a])}+n\sum_{\begin{subarray}{c}a\mid
R\\\ \omega(a)\leq k\end{subarray}}\sum_{\begin{subarray}{c}s\leq D\\\ e>V\log
n\end{subarray}}\frac{1}{\phi([se^{2},a])}.$

The theorem saves $\log n$ because $[se^{2},a]\leq D(V\log n)^{2}R\leq
n^{3/4}\log^{2}n$ by assumption. Let $g=(a,se^{2})$ and observe there are
coprime $g_{1}$ and $g_{2}$ such that $g=g_{1}g_{2}$, $g_{1}\mid s$ and
$g_{2}\mid e$. It follows that

(4.19) $\displaystyle\sum_{\begin{subarray}{c}a\mid R\\\ \omega(a)\leq
k\end{subarray}}\sum_{\begin{subarray}{c}s\leq D\\\
e>V^{\prime}\end{subarray}}\frac{1}{\phi([se^{2},a])}\leq\sum_{\begin{subarray}{c}g_{1}g_{2}\mid
a\mid R\\\ \omega(a)\leq k\end{subarray}}\sum_{g_{1}\mid s\leq
D}\frac{\phi(g_{1})}{\phi(s)}\sum_{g_{2}\mid
e>V^{\prime}}\frac{\phi(g_{2})}{\phi(e^{2})\phi(a)}\leq\frac{\log
n}{V^{\prime}}\sum_{\begin{subarray}{c}g_{1}g_{2}\mid a\mid R\\\ \omega(a)\leq
k\end{subarray}}\frac{1}{\phi(a)}\ll_{k}\frac{\log
n}{V^{\prime}}\sum_{\begin{subarray}{c}a\mid R\\\ \omega(a)\leq
k\end{subarray}}\frac{1}{a}$

for every $V^{\prime}\geq 1$. Here we used (4.9)–(4.12). Applying (4.19) with
$V^{\prime}=V,V\log n$ we are done.

###  4.2. Proof of Lemma 4.2

By Cauchy–Schwarz and the Bombieri–Vinogradov Theorem [EH66, Lemma],

$\sum_{a\leq R}|E_{0}(a)|\leq\sum_{b\leq
DR}\tau(b)^{2}|E(n-1;b,n)|\ll_{B}n(\log n)^{-B}$

where $b$ stands for $[d,a]$ and $\tau$ is the divisor function. Similarly,
letting $b:=[se^{2},a]$,

$\sum_{a\leq R}|E_{2}(a)|\leq\sum_{b\leq
DV^{2}R}\tau(b)^{3}|E(n-1;b,n)|\ll_{B}n(\log n)^{-B}.$

###  4.3. Proof of Lemma 4.3

We complete the $e$-range in $M_{2}(a)$, obtaining

(4.20) $\displaystyle M_{2}(a)$
$\displaystyle=\textup{Li}(n-1)\sum_{\begin{subarray}{c}e\geq 1\\\ s\leq D\\\
(se,n)=1\end{subarray}}\frac{\mu(e)}{\phi([se^{2},a])}+M_{2,1}(n)$

where, by (4.9)–(4.10),

$M_{2,1}(n)\ll\frac{n}{\log n}\sum_{\begin{subarray}{c}s\leq D\\\
e>V\end{subarray}}\frac{\mu^{2}(e)}{\phi([se^{2},a])}\leq\frac{n}{\phi(a)\log
n}\sum_{\begin{subarray}{c}s\leq D\\\
e>V\end{subarray}}\frac{\mu^{2}(e)\phi((se^{2},a))}{\phi(s)\phi(e^{2})}.$

Since $a$ is squarefree, if we let $g=(se^{2},a)$, then there are coprime
$g_{1},g_{2}$ (not necessarily unique) such that $g=g_{1}g_{2}$, $g_{1}\mid e$
and $g_{2}\mid s$, and so (using (4.11)–(4.12))

$M_{2,1}(n)\ll\frac{n}{\phi(a)\log n}\sum_{g_{1}g_{2}\mid a}\sum_{g_{1}\mid
e>V}\frac{\mu^{2}(e)\phi(g_{1})}{\phi(e^{2})}\sum_{g_{2}\mid s\leq
D}\frac{\phi(g_{2})}{\phi(s)}\ll_{k}\frac{n}{aV}$

when $\omega(a)\leq k$. Now write

$\textup{Li}(n-1)\sum_{\begin{subarray}{c}e\geq 1\\\ s\leq D\\\
(se,n)=1\end{subarray}}\frac{\mu(e)}{\phi([se^{2},a])}=M_{0}(a)+M_{2,2}(a)$

where

$M_{2,2}(a)=\textup{Li}(n-1)\sum_{\begin{subarray}{c}D<se^{2}\leq De^{2}\\\
(se,n)=1\end{subarray}}\frac{\mu(e)}{\phi([se^{2},a])}\ll\frac{n}{\log
n}\sum_{D<se^{2}\leq De^{2}}\frac{\mu^{2}(e)}{\phi([se^{2},a])}.$

Using the same $g$, $g_{1}$ and $g_{2}$ as before,

(4.21) $\displaystyle\sum_{D<se^{2}\leq
De^{2}}\frac{\mu^{2}(e)}{\phi([se^{2},a])}\leq\frac{1}{\phi(a)}\sum_{e\geq
1}\frac{1}{\phi(e^{2})}\sum_{D/e^{2}<s\leq
D}\frac{\phi(g)}{\phi(s)}\leq\frac{1}{\phi(a)}\sum_{g_{1}g_{2}\mid
a}\sum_{g_{1}\mid
e}\frac{\mu^{2}(e)\phi(g_{1})}{\phi(e^{2})}\sum_{\begin{subarray}{c}D/e^{2}<s\leq
D\\\ g_{2}\mid s\end{subarray}}\frac{\phi(g_{2})}{\phi(s)}\ll_{k}\frac{1}{a}$

where we used (4.9)–(4.12).

###  4.4. Proof of Lemma 4.4

For $a$ coprime to $n$ let

$f_{n,a}(m):=\frac{\mu^{2}(m)\phi(a)}{\phi([m,a])}\mathbf{1}_{(m,n)=1}$

which is multiplicative in $m$. Elliott and Halberstam established [EH66, p.
202]

(4.22) $\sum_{m\leq X}f_{n,1}(m)=\frac{\phi(n)}{n}\log X+O((\log\log n)^{2})$

for $n\leq X^{5}$. We have $f_{n,a}=f_{n,1}*g_{a}$ where
$g_{a}(m)=g(m)\mathbf{1}_{m\mid a^{\infty}}$ and $g$ is defined in (1.6).
Hence, by (4.22),

(4.23) $\displaystyle\sum_{\begin{subarray}{c}m\leq X\\\
(m,n)=1\end{subarray}}\frac{\mu^{2}(m)}{\phi([m,a])}=\sum_{m\leq
X}\frac{f_{n,a}(m)}{\phi(a)}=\frac{\phi(n)}{n\phi(a)}\sum_{d\leq
X/n^{1/5}}g_{a}(d)\log(X/d)+O\left(\frac{E}{\phi(a)}\right)$

for

(4.24) $E:=\sum_{d\leq X}|g_{a}(d)|(\log\log
n)^{2}+\sum_{d>X/n^{1/5}}|g_{a}(d)|\log d.$

To bound the first sum in $E$ we use $\sum_{d\geq
1}|g_{a}(d)|=2^{\omega(a)-\mathbf{1}_{2\mid a}}\ll_{k}1$. We turn to the
second sum in $E$. If $d$ is in the support of $g_{a}$ then $d$ is odd and

$|g_{a}(d)|\leq\prod_{p\mid d}p\prod_{p^{i}\mid\mid
d}p^{-i}\prod_{p^{i}\mid\mid
d}\left(1+\frac{1}{p-1}\right)^{i-1}\leq\frac{a}{d}\prod_{p^{i}\mid\mid
d}\left(1+\frac{1}{p-1}\right)^{i-1}\leq\frac{a}{d}(\tfrac{3}{2})^{\log_{3}d}.$

If $\omega(a)\leq k$ then there are $\ll_{k}(\log T)^{k}$ integers
$d\in[T,2T)$ with $d\mid a^{\infty}$, so

(4.25) $\sum_{d\in[T,2T)}|g_{a}(d)|\ll_{k}\frac{a(\log
T)^{k}}{T}T^{\log(3/2)/\log 3}.$

This implies that under our assumptions, $E/\phi(a)\ll_{k}(\log\log n)^{2}/a$.
With the same $E$ as in (4.24), we express the sum in the right-hand side of
(4.23) as

$\sum_{d\leq X/n^{1/5}}g_{a}(d)\log(X/d)=\log X\prod_{p\mid
a}\big{(}\sum_{i=0}^{\infty}g_{a}(p^{i})\big{)}-\sum_{d\geq 1}g_{a}(d)\log
d+O(E)$

times $\phi(n)/(n\phi(a))$. The $p$-product is $2^{\omega(a)}\phi(a)/a$ and
the $d$-sum is $D_{a}$.

##  5\. Proof of Proposition 1.3

As in the proof of Proposition 1.2, we work with $A_{a}(n)$ (defined in (4.1))
and reduce the lemma to proving

(5.1) $\sum_{\begin{subarray}{c}1\leq a\leq n^{\eta}\\\ \omega(a)\leq k\\\
(a,n)=1\end{subarray}}\mu^{2}(a)\bigg{|}A_{a}(n)-\phi(n)\bigg{(}\frac{2^{\omega(a)}}{a}-\frac{2D_{a}}{\phi(a)\log
n}\bigg{)}\bigg{|}\ll_{k}\frac{n}{\log n}(\log\log n)^{O_{k}(1)}.$

We write $2^{\omega(n-p)}$ as a divisor sum, separate the contribution of
large square divisors, and apply divisor switching. We have

$2^{\omega(n)}=\sum_{n=ds}\mu^{2}(d)=\sum_{n=e^{2}ds}\mu(e)=\sum_{\begin{subarray}{c}n=e^{2}ds\\\
e>V\end{subarray}}\mu(e)+\sum_{\begin{subarray}{c}n=e^{2}ds\\\ e\leq V\\\
d\leq D\end{subarray}}\mu(e)+\sum_{\begin{subarray}{c}n=e^{2}ds\\\ e\leq V\\\
s<n/(e^{2}D)\end{subarray}}\mu(e)$

with the choices $V=n^{2\eta}$ and $D=n^{1/2+2\eta}$. Then, given $a\geq 1$,
we split $A_{a}(n)$ accordingly:

$A_{a}(n)=\sum_{\begin{subarray}{c}p<n\\\
a|n-p\end{subarray}}\bigg{(}\sum_{\begin{subarray}{c}n-p=e^{2}ds\\\
e>V\end{subarray}}\mu(e)+\sum_{\begin{subarray}{c}n-p=e^{2}ds\\\ e\leq V\\\
d\leq D\end{subarray}}\mu(e)+\sum_{\begin{subarray}{c}n-p=e^{2}ds\\\ e\leq
V\\\
s<(n-p)/(e^{2}D)\end{subarray}}\mu(e)\bigg{)}=:S_{0}(a)+S_{1}(a)+S_{2}(a),$

so that

(5.2) $\displaystyle S_{0}(a)$
$\displaystyle=\sum_{\begin{subarray}{c}V<e\leq\sqrt{n}\\\ d\leq
n\end{subarray}}\mu(e)\pi(n-1;[de^{2},a],n),\qquad
S_{1}(a)=\sum_{\begin{subarray}{c}e\leq V\\\ d\leq
D\end{subarray}}\mu(e)\pi(n-1;[de^{2},a],n),$ (5.3) $\displaystyle S_{2}(a)$
$\displaystyle=\sum_{\begin{subarray}{c}e\leq V\\\
s<(n-2)/(e^{2}D)\end{subarray}}\mu(e)\pi(n-se^{2}D-1;[se^{2},a],n).$

Write
$E(m;q,a)=\pi(m;q,a)-\frac{\textup{Li}(m)\mathbf{1}_{(q,a)=1}}{\varphi(a)}$
and accordingly

$\displaystyle S_{i}(a)$ $\displaystyle=N_{i}(a)+E_{i}(a)\qquad(i=1,2)$

for

(5.4) $\displaystyle N_{1}(a)$
$\displaystyle=\textup{Li}(n-1)\sum_{\begin{subarray}{c}e\leq V\\\ d\leq D\\\
(de,n)=1\end{subarray}}\frac{\mu(e)}{\phi([de^{2},a])},\qquad
E_{1}(a)=\sum_{\begin{subarray}{c}e\leq V\\\ d\leq D\\\
(de,n)=1\end{subarray}}\mu(e)E(n-1;[de^{2},a],n),$ (5.5) $\displaystyle
N_{2}(a)$ $\displaystyle=\sum_{\begin{subarray}{c}e\leq V\\\
d<(n-2)/(e^{2}D)\\\
(de,n)=1\end{subarray}}\frac{\mu(e)\textup{Li}(n-de^{2}D-1)}{\phi([de^{2},a])},\qquad
E_{2}(a)=\sum_{\begin{subarray}{c}e\leq V\\\ d<(n-2)/(e^{2}D)\\\
(de,n)=1\end{subarray}}\mu(e)E(n-de^{2}D-1;[de^{2},a],n).$

The rest of the proof is separated into four lemmas, proved below, which
together establish (4.2).

######  Lemma 5.1.

We have $\sum_{a\leq n^{\eta}}|S_{0}(a)|\ll n^{1-\frac{\eta}{2}}$.

######  Lemma 5.2.

For every $B>0$, $\sum_{a\leq n^{\eta}}|E_{1}(a)|\ll_{B}n(\log n)^{-B}$.

######  Lemma 5.3.

For every $B>0$, $\sum_{a\leq n^{\eta}}|E_{2}(a)|\ll_{B}n(\log n)^{-B}$.

######  Lemma 5.4.

Fix $k\geq 1$. Suppose $a\leq n^{\eta}$, $\omega(a)\leq k$ and $\mu^{2}(a)=1$.
We have

(5.6) $\displaystyle
N_{1}(a)+N_{2}(a)=\phi(n)\bigg{(}\frac{2^{\omega(a)}}{a}-\frac{2D_{a}}{\phi(a)\log
n}\bigg{)}+O_{k}\bigg{(}\frac{n(\log\log n)^{2}}{a\log n}\bigg{)}.$

###  5.1. Proof of Lemma 5.1

The bound $\pi(n-1;q,n)\leq n/q$ gives

$\left|S_{0}(a)\right|\leq\sum_{\begin{subarray}{c}V<e\leq\sqrt{n}\\\ d\leq
n\end{subarray}}\sum_{\begin{subarray}{c}p<n\\\
[de^{2},a]|n-p\end{subarray}}1\leq\sum_{\begin{subarray}{c}e>V\\\ d\leq
n\end{subarray}}\frac{n}{[de^{2},a]}\leq\sum_{\begin{subarray}{c}e>V\\\ d\leq
n\end{subarray}}\frac{n}{de^{2}}\ll\frac{n\log n}{V}$

which implies the result.

###  5.2. Proof of Lemma 5.3

Letting $d:=[se^{2},a]$, we have

$\displaystyle\sum_{a\leq n^{\eta}}|E_{2}(a)|=\sum_{a\leq
n^{\eta}}\bigg{|}\sum_{\begin{subarray}{c}e\leq V\\\
s<(n-2)/(e^{2}D)\end{subarray}}\mu(e)E(n-se^{2}D-1;[se^{2},a],n)\bigg{|}\leq\sum_{\begin{subarray}{c}d\leq
n^{1/2-\eta}\\\ (d,n)=1\end{subarray}}\tau^{2}(d)\max_{y\leq
n}\left|E(y;d,n)\right|.$

The conclusion follows by applying Cauchy–Schwarz and the Bombieri–Vinogradov
Theorem [EH66, Lemma].

###  5.3. Proof of Lemma 5.2

As $E_{1}(a)$ requires us to break the $1/2$-barrier in the level of
distribution, it is the more delicate error term. In the previous two lemmas,
$\eta$ can be close to $1$, but not here. We have

$\displaystyle\sum_{a\leq n^{\eta}}|E_{1}(a)|$ $\displaystyle=\sum_{a\leq
n^{\eta}}\bigg{|}\sum_{e\leq V}\mu(e)\sum_{d\leq
D}E(n-1;[de^{2},a],n)\bigg{|}$ $\displaystyle=\sum_{a\leq
n^{\eta}}\bigg{|}\sum_{\begin{subarray}{c}e\leq V\\\
b\mid(ae)^{\infty}\end{subarray}}\mu(e)\sum_{\begin{subarray}{c}d\leq D\\\
b\mid d\\\ (d/b,ae)=1\end{subarray}}E(n-1;[de^{2},a],n)\bigg{|}$
$\displaystyle=\sum_{a\leq n^{\eta}}\bigg{|}\sum_{\begin{subarray}{c}e\leq
V\\\
b\mid(ae)^{\infty}\end{subarray}}\mu(e)\sum_{\begin{subarray}{c}d^{\prime}\leq
D/b\\\ (d^{\prime},ae)=1\end{subarray}}E(n-1;d^{\prime}[be^{2},a],n)\bigg{|},$

where we substituted $d^{\prime}=db$. Let $E_{1,1}$ denote the contribution to
the above with $b\leq J=n^{2\eta}$ and $E_{1,2}$ the remainder. For $E_{1,1}$
we apply the triangle inequality to arrive at

$\displaystyle E_{1,1}\leq\sum_{a\leq n^{\eta}}\sum_{e\leq
V}\sum_{\begin{subarray}{c}b\leq J\\\
b\mid(ae)^{\infty}\end{subarray}}\mu^{2}(e)\bigg{|}\sum_{\begin{subarray}{c}d^{\prime}\leq
D/b\\\ (d^{\prime},ae)=1\end{subarray}}E(n-1;d^{\prime}[be^{2},a],n)\bigg{|}.$

To make the summation range of $d^{\prime}$ independent of the other
variables, we apply a standard finer-than-dyadic decomposition on the
summation over $d^{\prime}$ (see e.g. the proof of [ABL21, Thm. 1.1]). It thus
suffices to estimate

$\displaystyle E_{1,1,D^{\prime}}=\sum_{a\leq n^{\eta}}\sum_{e\leq
V}\sum_{\begin{subarray}{c}b\leq J\\\
b\mid(ae)^{\infty}\end{subarray}}\mu^{2}(e)\bigg{|}\sum_{\begin{subarray}{c}d\leq
D^{\prime}\\\
(d^{\prime},ae)=1\end{subarray}}E(n-1;d^{\prime}[be^{2},a],n)\bigg{|}$

for any $D^{\prime}\leq D$. If $[be^{2},a]=g$ then $a$, $b$ and $e$ divide $g$
and the condition $(d^{\prime},ae)=1$ is equivalent to $(d^{\prime},g)=1$.
Thus

$E_{1,1,D^{\prime}}\leq\sum_{g\leq
n^{8\eta}}\tau(g)^{3}\bigg{|}\sum_{\begin{subarray}{c}d^{\prime}\leq
D^{\prime}\\\ (d^{\prime},g)=1\end{subarray}}E(n-1;d^{\prime}g,n)\bigg{|}.$

We quote [BGLM21, Prop. 6.1] specialised to $a_{2}=c=c_{0}=1$.

######  Proposition 5.5.

There exists some absolute constant $\varpi>0$ such that the following holds.
Let $x\geq 2$, $d,C\in\mathbb{N},a\in\mathbb{Z}\setminus\\{0\\}$ such that

(5.7) $\displaystyle Q\leq x^{1/2+\varpi},\quad|a|\leq
x^{1+\varpi},\quad|\lambda_{d}|\ll\tau(d)^{C}$

hold. Then for any $B>0$ we have

(5.8) $\displaystyle\sum_{d\leq
x^{\varpi}}\lambda_{d}\sup_{\begin{subarray}{c}w\bmod d\\\
(w,d)=1\end{subarray}}\Big{|}\sum_{\begin{subarray}{c}q\leq Q\\\
(q,ad)=1\end{subarray}}\Big{(}\sum_{\begin{subarray}{c}n\leq x\\\ n\equiv
a\bmod q\\\ n\equiv w\bmod
d\end{subarray}}\Lambda(n)-\frac{1}{\phi(qd)}\sum_{\begin{subarray}{c}n\leq
x\\\ (n,qd)=1\end{subarray}}\Lambda(n)\Big{)}\Big{|}\ll_{C,B}x(\log x)^{-B}.$

As commented after [BGLM21, Prop. 6.1], this holds for the characteristic
function of the primes replacing the von Mangoldt function and so implies
$E_{1,1,D^{\prime}}\ll_{B}n(\log n)^{-B}$. To prove Lemma 5.2, it remains to
estimate $E_{1,2}$ and we claim

(5.9) $\displaystyle E_{1,2}\ll n^{1-\eta/2}.$

By the trivial bound $E(n-1;q,a)\ll 1+n/q$ we have

$E_{1,2}\ll n\sum_{\begin{subarray}{c}a\leq n^{\eta}\\\ e\leq V\\\
d^{\prime}<D/J\end{subarray}}\frac{1}{d^{\prime}}\sum_{\begin{subarray}{c}J<b\leq
D/d^{\prime}\\\ b\mid(ae)^{\infty}\end{subarray}}\frac{1}{[be^{2},a]}.$

Since $[be^{2},a]=be^{2}a/(be^{2},a)$, by dyadic decomposition to get (5.9) it
suffices to show

(5.10) $\sum_{\begin{subarray}{c}a\sim A\\\ e\sim
V\end{subarray}}\sum_{\begin{subarray}{c}b\sim J\\\
b\mid(ae)^{\infty}\end{subarray}}(be^{2},a)\ll(2+\log
AV)^{C_{\varepsilon}}J^{\varepsilon}A^{2}V$

for all $\varepsilon>0$. Here $C_{\varepsilon}>0$ depends only on
$\varepsilon>0$ (later occurrences of $C_{\varepsilon}$ might have a different
value), $a\sim A$ means $a\in[A,2A)$ and $e\sim V$, $b\sim J$ should be
interpreted in the same way. Estimate (5.10) implies $E_{1,2}\ll
n^{1+\eta}(\log n)^{C_{\varepsilon}}/J^{1-\varepsilon}\ll
n^{1-\frac{\eta}{2}}$ and so (5.9) if $\varepsilon$ is sufficiently small.

To demonstrate (5.10), let $h=(be^{2},a)$. If $h\mid be^{2}$ then
$h=h_{1}h_{2}$ for some (not necessarily unique) $h_{1}\mid b$ and $h_{2}\mid
e^{2}$. This implies $h^{\prime}_{2}\mid e$ for
$h^{\prime}_{2}=\prod_{p^{k}\mid\mid h_{2}}p^{\lceil k/2\rceil}$. We can bound
(5.10) by

$\leq\sum_{h_{1}h_{2}\mid a\sim
A}h_{1}h_{2}\sum_{\begin{subarray}{c}h^{\prime}_{2}\mid e\sim V\\\ b\sim J\\\
h_{1}\mid b\mid(ae)^{\infty}\end{subarray}}1\leq\sum_{h_{1}h_{2}\mid a\sim
A}h_{1}h_{2}\sum_{\begin{subarray}{c}e^{\prime}\sim V/h^{\prime}_{2}\\\
b^{\prime}\sim J/h_{1}\\\
b^{\prime}\mid(ae^{\prime})^{\infty}\end{subarray}}1.$

By Rankin’s trick,

$\sum_{b\leq B,\,b\mid m^{\infty}}1\leq B^{\varepsilon}\prod_{p\mid
m}(1-p^{-\varepsilon})^{-1}\leq B^{\varepsilon}\exp(C_{\varepsilon}\sum_{p\mid
m}p^{-\varepsilon})\leq B^{\varepsilon}\exp(C_{\varepsilon}\omega(m))$

for every $\varepsilon>0$. Hence

$\sum_{\begin{subarray}{c}e^{\prime}\sim V/h^{\prime}_{2}\\\ b^{\prime}\sim
J/h_{1}\\\
b^{\prime}\mid(ae^{\prime})^{\infty}\end{subarray}}1\ll\sum_{e^{\prime}\sim
V/h^{\prime}_{2}}(J/h_{1})^{\varepsilon}\exp(C_{\varepsilon}(\omega(a)+\omega(e^{\prime})))\ll(J/h_{1})^{\varepsilon}V(2+\log
V)^{C_{\varepsilon}}\exp(C_{\varepsilon}\omega(a))$

and so, since $h_{1}h_{2}\ll A$,

(5.11) $\displaystyle\sum_{\begin{subarray}{c}a\sim A\\\ e\sim
V\end{subarray}}\sum_{\begin{subarray}{c}b\sim J\\\
b\mid(ae)^{\infty}\end{subarray}}(be^{2},a)$ $\displaystyle\ll(2+\log
V)^{C_{\varepsilon}}J^{\varepsilon}V\sum_{h_{1}h_{2}\mid a\sim
A}h_{1}^{1-\varepsilon}h_{2}\exp(C_{\varepsilon}\omega(a))$ (5.12)
$\displaystyle\ll(2+\log V)^{C_{\varepsilon}}J^{\varepsilon}AV\sum_{a\sim
A}\exp(C_{\varepsilon}\omega(a))\tau(a)^{2}\ll(2+\log
AV)^{C_{\varepsilon}}J^{\varepsilon}A^{2}V.$

###  5.4. Proof of Lemma 5.4

Let $1\leq a\leq n^{\eta}$ be squarefree. The expression $N_{1}(a)$ matches
$M_{2}(a)$ defined in (4.8), so

$N_{1}(a)=\textup{Li}(n-1)\sum_{\begin{subarray}{c}d\leq D\\\
(d,n)=1\end{subarray}}\frac{\mu^{2}(d)}{\phi([d,a])}+O\left(\frac{n}{a\log
n}\right)$

by Lemma 4.3 and our choice of $V$. The $d$-sum is estimated in Lemma 4.4,
giving

$N_{1}(a)=\phi(n)\left(\frac{2^{\omega(a)}}{a}\frac{\log D}{\log
n}-\frac{D_{a}}{\phi(a)\log n}\right)+O_{k}\left(\frac{n(\log\log
n)^{2}}{a\log n}\right)$

where we used (1.7). It remains to deal with $N_{2}(a)$. By completing the $e$
range we have

(5.13) $\displaystyle N_{2}(a)=\sum_{\begin{subarray}{c}e\geq 1\\\
d<(n-2)/(e^{2}D)\\\
(de,n)=1\end{subarray}}\frac{\mu(e)\textup{Li}(n-de^{2}D-1)}{\phi([de^{2},a])}+O\left(\frac{n}{V}\right).$

We now use $\textup{Li}(n-de^{2}D-1)=\textup{Li}(n-1)+O(de^{2}D/\log n)$ to
decompose $N_{2}(a)$ as

$N_{2}(a)=N_{2,1}(a)+N_{2,2}(a)+O\left(\frac{n}{V}\right)$

where

(5.14) $\displaystyle
N_{2,1}(a)=\textup{Li}(n-1)\sum_{\begin{subarray}{c}de^{2}<(n-2)/D\\\
(de,n)=1\end{subarray}}\frac{\mu(e)}{\phi([de^{2},a])}=\textup{Li}(n-1)\sum_{\begin{subarray}{c}m<(n-2)/D\\\
(m,n)=1\end{subarray}}\frac{\mu^{2}(m)}{\phi([m,a])}$

and

(5.15) $\displaystyle N_{2,2}(a)\ll\frac{D}{\log
n}\sum_{\begin{subarray}{c}de^{2}<(n-2)/D\\\
(de,n)=1\end{subarray}}\frac{\mu^{2}(e)de^{2}}{\phi([de^{2},a])}\leq\frac{D}{\log
n}\sum_{m<n/D}\frac{m2^{\\#\\{p^{2}\mid m\\}}}{\phi([m,a])}.$

We let $g=(a,m)$ in the right-hand side of (5.15) and use (4.9)–(4.10) to find

$N_{2,2}(a)\ll\frac{D}{\phi(a)\log n}\sum_{g\mid a}g\sum_{g\mid
m<n/D}\frac{(m/g)2^{\\#\\{p^{2}\mid m\\}}}{\phi(m/g)}.$

Let $S(x):=\sum_{m\leq X}(m/\phi(m))2^{\\#\\{p^{2}\mid m\\}}$. This sum is
$\ll X$ [MV07, Thm. 2.14]. Hence

$N_{2,2}(a)\ll\frac{D}{\phi(a)\log n}\sum_{g\mid
a}g2^{\omega(g)}S\left(\frac{n}{Dg}\right)\ll\frac{n}{\phi(a)\log
n}3^{\omega(a)}\ll_{k}\frac{n}{a\log n}.$

To conclude we apply Lemma 4.4 with $X=n/D$ to estimate $N_{2,1}(a)$,
obtaining

$N_{2}(a)=\phi(n)\left(\frac{2^{\omega(a)}}{a}\frac{\log(n/D)}{\log
n}-\frac{D_{a}}{\phi(a)\log n}\right)+O_{k}\left(\frac{n(\log\log
n)^{2}}{a\log n}\right).$

##  6\. Elementary proof of (1.7)

Let $X\sim N(0,1)$. By Lévy’s continuity theorem [Dur19, Thm, 3.3.17], (1.8)
is equivalent to

$\mathbb{E}e^{it\left(\frac{\omega(\widetilde{X}_{n})-2\log\log n}{(2\log\log
n)^{1/2}}\right)}=\big{(}\sum_{m\leq n}2^{\omega(m)}\big{)}^{-1}\sum_{m\leq
n}2^{\omega(m)}e^{it\left(\frac{\omega(m)-2\log\log n}{(2\log\log
n)^{1/2}}\right)}\to\mathbb{E}e^{itX}=e^{-\frac{t^{2}}{2}},\qquad
t\in\mathbb{R},$

a relation that is readily verified using the Selberg–Delange method, as
observed by Elliott. We give an elementary proof of (1.8) using Corollary 3.2.
One only needs to verify (3.4) and (3.5), both of which follow from the next
lemma, whose prof is a streamlined version of the arguments in [EG19, Lems.
3.4, 3.15].

######  Lemma 6.1.

For positive integers $1\leq a\leq n$ such that $\omega(a)\leq k$ we have

$\mathbb{P}(a\textup{ divides
}\widetilde{X}_{n})=\frac{2^{\omega(a)}}{a}\bigg{(}1-\frac{\log a}{\log
n}+O_{k}\bigg{(}\frac{1}{\log n}+\sum_{p\mid a}\frac{1}{p}\bigg{)}\bigg{)}.$

###### Proof.

Let $S_{a}(n):=\sum_{a\mid r\leq n}2^{\omega(r)}$. We have

(6.1) $\displaystyle S_{a}(n)$ $\displaystyle=\sum_{m\leq
n/a}2^{\omega(am)}=\sum_{d\mid a^{\infty}}2^{\omega(ad)}\sum_{m^{\prime}\leq
n/(ad),\,(m^{\prime},a)=1}2^{\omega(m^{\prime})}=A+B$

where $d$ stands for $(m,a^{\infty})$, $A$ is the contribution of $d=1$ and
$B$ is the contribution of $d>1$. We have $0\leq B\leq\sum_{1<d\mid
a^{\infty}}2^{\omega(ad)}S_{1}(n/(ad))$ by dropping the coprimality condition.
For $A$ we have

$A=2^{\omega(a)}(\sum_{m^{\prime}\leq n/a}2^{\omega(m^{\prime})}-\sum_{1<d\mid
a^{\infty}}2^{\omega(d)}\sum_{m^{\prime\prime}\leq
n/(ad),\,(m^{\prime\prime},a)=1}2^{\omega(m^{\prime\prime})})$

and so, by dropping the coprimality condition $(m^{\prime\prime},a)=1$,

$0\leq 2^{\omega(a)}\sum_{m\leq n/a}2^{\omega(m)}-A\leq\sum_{1<d\mid
a^{\infty}}2^{\omega(a)+\omega(d)}\sum_{m\leq n/(ad)}2^{\omega(m)}.$

We have proved

(6.2) $\left|S_{a}(n)-2^{\omega(a)}S_{1}(n/a)\right|\leq\sum_{1<d\mid
a^{\infty}}(2^{\omega(ad)}+2^{\omega(a)+\omega(d)})S_{1}(n/(da)).$

We divide (6.2) by $S_{1}(n)$ and use $S_{1}(n)=\zeta(2)^{-1}n\log
n(1+O(1/\log n))$ [MV07, p. 42]. ∎

######  Acknowledgements.

We thank Brad Rodgers for discussions around Corollary 1.4 and useful
suggestions, and an anonymous referee for useful comments which led us to a
simplified proof of Elliott’s conjecture using Proposition 1.2.

## References

  * [ABL21] Edgar Assing, Valentin Blomer, and Junxian Li.  Uniform Titchmarsh divisor problems.  Adv. Math., 393:51, 2021.  Id/No 108076.
  * [BGLM21] Valentin Blomer, Lasse Grimmelt, Junxian Li, and Simon L Rydin Myerson.  Additive problems with almost prime squares.  arXiv preprint arXiv:2111.01601, 2021.  To appear in GAFA.
  * [Bil69] P. Billingsley.  On the central limit theorem for the prime divisor function.  Am. Math. Mon., 76:132–139, 1969.
  * [Bil99] Patrick Billingsley.  Convergence of probability measures. Wiley Ser. Probab. Stat. Chichester: Wiley, 2nd ed. edition, 1999.
  * [Dur19] Rick Durrett.  Probability. Theory and examples, volume 49 of Camb. Ser. Stat. Probab. Math. Cambridge: Cambridge University Press, 5th edition edition, 2019.
  * [EG19] Dor Elboim and Ofir Gorodetsky.  Multiplicative arithmetic functions and the generalized Ewens measure.  arXiv preprint arXiv:1909.00601, 2019.
  * [EH66] P. D. T. A. Elliott and H. Halberstam.  Some applications of Bombieri’s theorem.  Mathematika, 13:196–203, 1966.
  * [Ell15] P. D. T. A. Elliott.  Additive and multiplicative functions from Nottingham to Urbana-Champaign.  Int. J. Number Theory, 11(5):1357–1366, 2015.
  * [GS07] Andrew Granville and K. Soundararajan.  Sieving and the Erdős-Kac theorem.  In Equidistribution in number theory, an introduction, volume 237 of NATO Sci. Ser. II Math. Phys. Chem., pages 15–27. Springer, Dordrecht, 2007.
  * [Han09] Kenji Handa.  The two-parameter Poisson-Dirichlet point process.  Bernoulli, 15(4):1082–1116, 2009.
  * [MV07] Hugh L. Montgomery and Robert C. Vaughan.  Multiplicative number theory. I. Classical theory, volume 97 of Cambridge Studies in Advanced Mathematics.  Cambridge University Press, Cambridge, 2007.


# Multi-Industry Simplex :
A Probabilistic Extension of GICS

Maksim Papenkov1,2,111Corresponding Author<EMAIL_ADDRESS>Chris
Meredith1,3 Claire Noel1
Jai Padalkar1 Temple Hendrickson1 Daniel Nitiutomo1 $\&$ Thomas Farrell1

(1O’Shaughnessy Asset Management
2Columbia University, Department of Computer Science
3Cornell University, Johnson College of Business
)

###### Abstract

Accurate industry classification is a critical tool for many asset management
applications. While the current industry gold-standard GICS (Global Industry
Classification Standard) has proven to be reliable and robust in many
settings, it has limitations that cannot be ignored. Fundamentally, GICS is a
single-industry model, in which every firm is assigned to exactly one group -
regardless of how diversified that firm may be. This approach breaks down for
large conglomerates like Amazon, which have risk exposure spread out across
multiple sectors. We attempt to overcome these limitations by developing MIS
(Multi-Industry Simplex), a probabilistic model that can flexibly assign a
firm to as many industries as can be supported by the data. In particular, we
utilize topic modeling, an natural language processing approach that utilizes
business descriptions to extract and identify corresponding industries. Each
identified industry comes with a relevance probability, allowing for high
interpretability and easy auditing, circumventing the black-box nature of
alternative machine learning approaches. We describe this model in detail and
provide two use-cases that are relevant to asset management - thematic
portfolios and nearest neighbor identification. While our approach has
limitations of its own, we demonstrate the viability of probabilistic industry
classification and hope to inspire future research in this field.

Keywords \- Industry Classification ; Probabilistic Machine Learning ; Natural
Language Processing

##  1 Research Motivation

###  1.1 Problem

Is Amazon a bookstore? Perhaps in 1998 it was, but even then - it wasn’t just
a bookstore, rather it was a _digital_ bookstore (making it _both_ a retailer,
and a fledgling internet company). But what about now? Amazon is still
technically a bookstore, but it’s also a cloud computing platform, a film
studio, a grocer, and so much more. Amazon is gradually evolving into a
conglomerate of conglomerates, leaving investors with a predicament - if they
are holding a stock with such heterogeneous risk exposure, how can they manage
it within a complex portfolio?

###  1.2 Existing Solution (and its Limitations)

For such a task, most practitioners turn to GICS (Global Industry
Classification Standard), which assigns each firm to exactly _one_ industry -
generally determined by the sub-business of a firm that generates the highest
revenue [1]). The more diversified a conglomerate becomes however, the less
its total risk can be accurately attributed to just a single sub-business.
When asset managers operate off of such incomplete information, they expose
their portfolio to misrepresentation risk.

If we label Amazon as a “retail” stock - we introduce _misrepresentation
risk_. If we then instead label it as a “tech” stock, we then just introduce
alternative _misrepresentation risk_ \- there’s no way around it. As large-cap
firms continue to diversify, misrepresentation risk becomes increasingly more
difficult for investors to ignore - and a critical demand for a multi-industry
classification tool reveals itself. While GICS has proven to be a robust and
reliable system for many decades, the ubiquity of multi-sector diversification
for large-cap firms requires an honest consideration of GICS’ key limitations
:

  * •

One-Dimensional :  A firm can only belong to _one_ industry.

  * •

Statically-Defined : Industry definitions are rarely updated and may become
_outdated_ over time.

  * •

Opaque :  Firms are _manually_ classified (by committee) after an IPO, with
rare label re-assignments.

Asset managers need a better tool ; we provide one by leveraging recent
innovations in artificial intelligence.

###  1.3 New Solution (and its Value Proposition)

We introduce the Multi-Industry Simplex (MIS) as a probabilistic extension of
GICS. Rather than assign each firm to exactly _one_ industry, we now assign it
to _multiple_ industries - in proportion to their relevance (estimated by a
model). MIS provides a rigorously-constructed, data-oriented solution that
provides asset managers with a better tool for representing risk, with the
following specific benefits :

  * •

Multi-Dimensional : A firm can belong to _multiple_ industries.

  * •

Dynamically-Defined : Industry definitions are flexible and can easily
_evolve_ with new data.

  * •

Internally-Consistent : Firms are classified _simultaneously_ , guaranteeing
model homogeneity.

MIS is a text-based classification tool that leverages _natural language
processing_ to identify the key industries that a firm is involved in. We
discuss the input data in section 2, and describe the model architecture in
section 3. While our method has limitations of its own, we argue that a
probabilistic framework is the natural evolution for industry classification
systems - which can provide significant positive impact to various asset
management applications.

##  2 Principals for Robust Probabilistic Industry Classification

Before we dive into the methodology for MIS, we first establish heuristics for
robust probabilistic industry classification such that our model best
satisfies investor needs by identifying a critical requirement :

  * •

Explainable Fail States : data is noisy, markets are wild, and with full
certainty - at some point every single model will invariably fail. Investors
generally understand this, and reluctantly accept it as a necessary price for
participation. What investors generally do _not_ tolerate however, is an
_unexplainable_ fail state in which the asset manager is not able to attribute
model failure to any particular model defect that can potentially be
circumvented with further research and model iteration.

We believe that this is the single most significant barrier to the utilization
of advanced artificial intelligence techniques in asset management. Though a
model may be correct $99\%$ of the time, if an asset manager is unable to
explain the $1\%$ of events in which they lose a significant amount of money
for a client, they are at a high risk of losing investor confidence, which
ultimately hurts their business.

###  2.1 Model Risk Management

MIS, along with other probabilistic industry classification methods, are not
simply models for the sake of prediction - rather they are proper _risk
management_ tools, and must be held to the highest standard of quality, as
model failure can lead to the very rapid erosion of an investor’s wealth (e.g.
Long Term Capital Management). To manage model risk, we require three specific
criteria :

  * •

Numerical Stability : small perturbations in input data must yield negligible
prediction variation.

  * •

Replicability : model predictions should be approximately invariant to
initialization.

  * •

Interpretability : each model parameter must be plainly relatable to a real-
world attribute (later on in the paper we’ll see that every single parameter
of MIS can be plainly interpreted as a conditional probability of some object
belong to a category).

Many modern innovations in regularization reliably ensure that most models
satisfy the first two criteria, but the third is far more difficult to obtain,
particularly for _black-box_ models such as neural networks, which have been
utilized for probabilistic industry classification in innovative and exciting
papers such as [2, 3, 4, 5].

###  2.2 Practical Risk of Black-Box Models

While neural networks (such as large language models like GPT [6] and LLaMa
[7]) have revolutionized the ways in which we utilize natural language data
for real-world applications, they often present a significant risk :
_unexplainable fail states_. It is widely understood that this is simply the
cost of high-precision modeling, but it is important to recognize that this
risk is much easier to ignore in applications where the implication of model
failure is much less severe - such as text summarization, chatbots, and
machine translation. We argue that many investors are simply not willing to
tolerate such risks when the cost of failure is a significant loss in their
wealth.

###  2.3 Relative Benefit of Clear-Box Models

Though in many regards the architecture of MIS may be considered more _naive_
than an LLM (as we’ll demonstrate in later sections), we argue that the
mechanical transparency of the underlying probability model provides an asset
manager with a clear framework to understand and explain why a model fails -
which would then allow for quick iteration to address any issues that are
discovered. In the context of risk management, model simplicity is often an
asset, not a liability.

##  3 Data

At a high-level, MIS extracts high-importance keywords from a large collection
of text to identify the most relevant industries corresponding to a firm. As
with any data-oriented endeavor, the reliability of this industry
classification method is highly dependent on the quality of data sourcing and
the robustness of the text pre-processing methodology. _Better curated input
data invariably leads to a more useful model._

###  3.1 Sourcing

To improve the probability of correctly associating keywords with relevant
industries, we require comprehensive documents with high word-counts to
provide sufficient evidence for classification. In practice, it is recommended
to pool together text data from multiple sources to ensure coverage of all
relevant industries that a firm is involved in. Examples of appropriate texts
include :

  * •

10-K Business Descriptions : self-disclosed descriptions of high-relevance
business ventures.

  * •

Analyst-Written Business Descriptions : third-party descriptions of business
ventures.

  * •

Earnings Call Transcripts : discussions on existing and emerging ventures of a
firm.

Beyond this, we can also supplement our dataset with news headlines and social
media feeds - though such data sources provide less structure and require
additional considerations for stable integration. In practice, we generally
concatenate multiple separate documents into a single object. As we’ll
demonstrate later, the relative weighting of a document in the model is
proportionate to the number of words : a twenty-paragraph business description
will have much more influence on classification than a ten-word news headline.

Fundamental Data Assumptions : all documents are _comprehensive_ (they cover
_all_ non-trivial business ventures of a firm) and are _proportionately-
representative_ (business-ventures are mentioned in proportion to their
relevance to a firm; if something is discussed in greater detail - it must
have higher relevance). We can reasonably assume that documents such as 10-K
business descriptions satisfy these assumptions by default, but for exotic
text datasets like social media feeds this must be verified by the
practitioner.

###  3.2 Bag-of-Words Representation (BOW)

In principal, MIS identifies industries by counting keywords - which thus
requires keywords to be easily extractable from our text. For this, we require
the bag-of-words (BOW) representation, which is a permutation-invariant
multiset of words. Simply, we care about the frequency with which a word
appears in a document, but not the particular ordering. Consider the example
below :

$\text{``Amazon sells many
books.''}\Rightarrow\begin{cases}\\{\text{\;``Amazon''\;,\; ``sells''\;,\;
``many''\;,\; ``books.''\;}\\}\\\\[5.0pt]
\\{\text{\;``books.''\;,\;``many''\;,\;``sells''\;,\;
``Amazon''\;}\\}\\\\[5.0pt]
\\{\text{\;``many''\;,\;``Amazon''\;,\;``books.''\;,\;
``sells''\;}\\}\end{cases}$

The primary limitation of the BOW representation is that it is unable to
utilize within-document context to extract meaning from text, unlike more
sophisticated embedding schemes utilized by modern neural-net based NLP models
like GPT [6].

Fundamental BOW Limitation : _induced semantic ambiguity_ , which is the
inability to distinguish a particular usage of a word in the absence of
context. To make this less abstract, consider for example the word “tissue”
without additional descriptive information. On its own - it is unclear whether
this word corresponds to a materials industry (as in “paper tissue”) or to a
healthcare industry (as in “human tissue”). Thus, we are required to apply
text pre-processing measures to perform semantic disambiguation.

###  3.3 Text Pre-Processing

To mitigate the influence of noise and ambiguity on industry classification,
we must apply various _text pre-processing_ techniques to construct a
semantically-unambiguous BOW, which is simply a collection of words such that
each word is logically associated with only _one_ industry.

The goal of text pre-processing is to homogenize our input data, which reduces
the complexity of our vocabulary and thus reduces the dimensionality of our
model. This includes the following set of tools :

  * •

Normalization : remove punctuation and convert all letters to lower-case.

  * •

Stemming : standardize a word by reducing it to its root ( “movies”
$\Rightarrow$ “movie” ).

  * •

N-Gram Construction : form compound words ( “machine” + “learning”
$\Rightarrow$ “machine-learning” ).

  * •

Stopword Removal : eliminate non-descriptive words ( “and” ).

Consider the following example which fully pre-processes an entire sentence :

$\text{``Amazon provides machine learning tools and makes
movies."}\Rightarrow\big{\\{}\text{\;``machine-
learning''\;,\;``movie''\;}\big{\\}}$

Further, we can also apply lemmatization which maps a set of words to a single
keyword that represents aggregate meaning. These can either be synonyms or
sub-cases of a general term. For example :

$\begin{matrix}\text{``clean-energy''}\\\\[2.5pt] \text{``green-
energy''}\\\\[2.5pt] \text{``green-
power''}\end{matrix}\;\;\Bigg{\\}}\Rightarrow\text{``renewable-energy''}$

When building a text pre-processor for a large collection of documents, it is
helpful to encode these various steps in a semantic tree which summarizes how
all of the various tools operate together to homogenize our vocabulary.
Consider the following illustrative example, which constructs the keyword
“movie” :

“movie”“movies”“motion-picture”“cinema”“cinematic”+[ “motion”“picture”
]“pictures” Figure 1: Partial Semantic-Tree for Keyword = “movie”

For this example we construct a bigram (“motion” + “picture” $\Rightarrow$
“motion-picture”), we apply stemming (“cinematic” $\Rightarrow$ “cinema”) and
we apply lemmatization (“cinema” $\Rightarrow$ “movie”). In practice, a full
semantic tree may be vast with hundreds of nodes that account for subtle
nuances in language, though regardless of scale - the logic doesn’t need to be
more complicated that what we demonstrate above.

We recommend that such semantic trees are built by hand, to ensure validity
and thoroughness. We emphasize that _a domain expert’s ability to manipulate
the input text governs the nature of the industries that emerge from the
probabilistic model_. Language is subjective, and a practitioner’s biases are
reflected in the model - though we argue that this isn’t a bug, It’s a
feature. The goal of MIS is not to blindly adhere to a black-box modelling
approach, but rather to leverage human expertise to guide the model towards a
solution that best meets an asset manager’s specific needs.

##  4 Model

The modeling framework for MIS utilizes Bayes Theorem, which incorporates
information contained in dataset $\mathbf{X}$ and prior parameter
$\bm{\theta}$ to estimate a _posterior_ distribution as :

$\displaystyle\underbrace{\mathbb{P}\big{(}\bm{\theta}\mid\mathbf{X}\big{)}}_{\text{posterior}}\propto\underbrace{\mathbb{P}\big{(}\mathbf{X}\mid\bm{\theta}\big{)}}_{\text{likelihood}}\;\;\underbrace{\mathbb{P}\big{(}\bm{\theta}\big{)}}_{\text{prior}}$
(1)

For the remainder of this section, we assume that the reader is familiar with
basic Bayesian learning theory.

###  4.1 Mixture Model

We motivate the architecture of _mixed-membership_ models by first describing
_mixture_ models, which represent an observation as a distribution across
several groups222Mixture models can be thought of as probabilistic
generalizations of the classical machine learning tool _K-Means Clustering_..
The core machinery of a mixture-model is the categorical distribution, which
represents random variable $\mathbf{z}$ as of one of $K$-many possible values
based on probability distribution $\bm{\theta}\in\triangle^{K}$ (which we call
a “simplex”) :

$\mathbf{z}\sim\text{Categorical}_{K}\big{(}\mathbf{z}\mid\bm{\theta}\big{)}:\begin{cases}\mathbb{P}\big{(}z=1\mid\bm{\theta}\big{)}=\theta_{1}\\\\[5.0pt]
\dots\\\\[5.0pt]
\mathbb{P}\big{(}z=K\mid\bm{\theta}\big{)}=\theta_{K}\end{cases}$ (2)

As an illustrative example, with $K=2$ we can represent the fairness of a
coin, and with $K=6$ we can represent the fairness of dice. For our purposes,
we’ll use this framework to model the probability that a firm belongs to one
of $K$-many possible industries.

To make this a proper probabilistic model, we parameterize the randomness of
parameter $\bm{\theta}$ with a Dirichlet distribution, which captures the
_uncertainty of our uncertainty_ as :

$\bm{\theta}\sim\text{Dirichlet}_{K}\big{(}\bm{\theta}\mid\bm{\beta}\big{)}=\frac{\Gamma\big{(}\sum_{i=1}^{K}\beta_{i}\big{)}}{\prod_{i=1}^{K}\Gamma\big{(}\beta_{i}\big{)}}\cdot\prod_{i=1}^{K}\theta_{i}^{\beta_{i}-1}$
(3)

We combine these components to construct a mixture model defined by the
following generative process, where $N$ is the total number of firms (each
denoted as $\mathbf{y}_{n}$ to avoid confusion with $\mathbf{x}_{n}$ in the
next section).

$\displaystyle\mathbf{y}_{n}$
$\displaystyle\sim\mathbb{P}\big{(}\mathbf{y}_{n}\mid\bm{\mu}_{z_{n}}\big{)}$
(4) $\displaystyle\mathbf{z}_{n}$
$\displaystyle\sim\text{Categorical}_{K}\big{(}\mathbf{z}_{n}\mid\bm{\theta}\big{)}$
(5) $\displaystyle\bm{\theta}$
$\displaystyle\sim\text{Dirichlet}_{K}\big{(}\bm{\theta}\mid\bm{\beta}\big{)}$
(6)

To be clear, $\mathbf{z}_{n}\in\\{1,2,\dots,K\\}$ is a group index, so
observation $\mathbf{y}_{n}$ simply belongs to the group-$\mathbf{z}_{n}$.
Here $\bm{\mu}_{z_{n}}$ is a parameter that describes the nature of that
particular group, though we leave it abstract for generality. We summarize the
full architecture with the following graphical model :

$\mathbf{y}_{n}$$z_{n}$$\bm{\mu}_{k}$$\bm{\theta}$$\beta$$N$$K$ Figure 2:
Mixture Model

Once this model is fit, we can then estimate the probability of group
membership for a particular observation
$\mathbb{P}(\mathbf{z}_{n}\mid\mathbf{x}_{n})$ using Bayes Theorem, which may
yield something like the following (assuming $K=2$) :

$\displaystyle\mathbb{P}\big{(}\mathbf{z}_{n}\mid\mathbf{y}_{n}\big{)}=\big{[}0.95,0.05\big{]}\iff\text{95\%
probability $\mathbf{y}_{n}$ belongs to group-1, 5\% otherwise}$

We are now equipped to generalize to a more sophisticated model architecture.

###  4.2 Mixed-Membership Model (Latent Dirichlet Allocation)

MIS utilizes two-levels of mixture-models by positing the following
architecture :

  * •

Each firm is an _industry-mixture_.

  * •

Each industry is a _word-mixture_.

A _mixed-membership model_ is used for data in which each observation is a
collection of elements (in this case a BOW) such that each element belongs to
a group. Mixed-membership models have prolific applications to various fields
including document classification [8], social network analysis [9], and
genetics [10]. When mixed-membership models are applied to text, they are
commonly referred to as “topic models”, the simplest of which is _Latent
Dirichlet Allocation (LDA)_. Some previous work has been done in the field of
applying LDA to business descriptions [11], but it is still a nascent area of
study.

Before diving into the math, we provide a high-level overview of the MIS flow
of information as :

“Amazon is a conglomeratefirm with business ventures inonline retail, cloud
computing,digital media, as well as… ”“retail” x 20“cloud” x 15“movie” x 10…
_E-Commerce_ ($40\%$)_Cloud Computing_ ($30\%$)_Movies_ ($20\%$)…Business
Description(Clean) Bag-of-WordsIndustry-Mixture Figure 3: Industry-Mixture
Identification Process

First we begin with a raw business description, then we convert it to a clean
BOW via text pre-processing, and finally the model estimates an _industry-
mixture_. Each industry is a _word-mixture_ of the following form :

$\displaystyle\emph{E-Commerce}=\big{[}\;\text{``retail''}(30\%)\;,\;\text{``online''}(28\%)\;,\;\text{``supply-
chain''}(25\%)\;,\;...\big{]}$ (7)

To formally define the model architecture, we require three dimensionalities :

  * •

$M$ : total number of firms; each business description has $N_{m}$-many words.

  * •

$K$ : total number of industries that are modeled (specified by the user).

  * •

$V$ : total vocabulary size in pre-processed text.

With these, we define the following random variables :

  * •

$\mathbf{x}_{m}$ : the $m$-th firm’s business description represented as a BOW
(e.g. $\mathbf{x}_{1}=\\{\text{``bank'', ``finance'', ...}\\}$).

  * •

$\mathbf{x}_{m,n}$ : the $n$-th word in the $m$-th firm’s BOW business
description.

  * •

$\mathbf{z}_{m,n}$ : the industry-index for word $\mathbf{x}_{m,n}$ (such that
$\mathbf{z}_{m,n}\in\\{1,2,\dots,K\\})$.

  * •

$\bm{\theta}_{m}\in\triangle^{K}$ : the industry-mixture of the $m$-th firm.

  * •

$\bm{\phi}_{k}\in\triangle^{V}$ : the word-mixture of the $k$-th industry.

We thus now define MIS using the Latent Dirichlet Allocation generative
process :

$\displaystyle\mathbf{x}_{m,n}$
$\displaystyle\sim\text{Categorical}_{V}\big{(}\mathbf{x}_{m,n}\mid\bm{\phi}_{z_{m,m}}\big{)}$
(8) $\displaystyle\mathbf{z}_{m,n}$
$\displaystyle\sim\text{Categorical}_{K}\big{(}\mathbf{z}_{m,n}\mid\bm{\theta}_{m}\big{)}$
(9) $\displaystyle\bm{\theta}_{m}$
$\displaystyle\sim\mathbb{P}\big{(}\text{Industry}_{k}\mid\text{Firm}_{m}\big{)}=\text{Dirichlet}_{K}\big{(}\bm{\theta}_{m}\mid\bm{\beta}\big{)}$
(10) $\displaystyle\bm{\phi}_{k}$
$\displaystyle\sim\mathbb{P}\big{(}\text{Word}_{k}\mid\text{Industry}_{m}\big{)}=\text{Dirichlet}_{V}\big{(}\bm{\phi}_{k}\mid\bm{\alpha}\big{)}$
(11)

The appeal of this architecture is that it fits both the industry-mixtures and
the word-mixtures simultaneously, yielding an internally-consistent
representation of a market. As each parameter in this model is simply a
probability of something belonging to a particularly group, this model is
fully interpretable. We discuss the hyperparameters $\bm{\alpha}$ and
$\bm{\beta}$ in section 3.5.

We summarize the full-architecture with the following graphical model :

$\mathbf{x}_{m,n}$$\mathbf{z}_{m,n}$$\bm{\phi}_{k}$$\bm{\alpha}$$\bm{\theta}_{m}$$\bm{\beta}$Word-
Mixturefor Topic-kIndustry-Mixturefor Firm-mWord-nfor Firm-mIndex-nfor Firm-
mWord-MixtureHyperparamIndustry-MixtureHyperparam$N_{m}$$M$$K$ Figure 4:
Mixed-Membership Model (Latent Dirichlet Allocation)

###  4.3 Model Evaluation

A useful tool for evaluating the quality of a mixed-membership model is
_perplexity_ , which measures goodness-of-fit. A lower perplexity indicates a
better sample fit, which may provide guidance for calibrating the text pre-
processor and for hyperparameter tuning. Formally, we define perplexity for
estimated $\bm{\theta}$ and $\bm{\phi}$ as :

$\text{Perp}\big{(}\bm{\theta}_{1:M},\bm{\phi}_{1:K}\;;\mathbf{x}_{1:M}\big{)}=\exp\bigg{(}-\frac{\sum_{m=1}^{M}\log\mathbb{P}\big{(}\bm{\theta}_{1:M},\bm{\phi}_{1:K}\;;\mathbf{x}_{m}\big{)}}{\sum_{m=1}^{M}N_{m}}\bigg{)}$
(12)

For this we use $\text{count}(w_{m,v})$ as the count of the $v$-th word of the
vocabulary for the $m$-th firm :

$\log\mathbb{P}\big{(}\bm{\theta}_{1:M},\bm{\phi}_{1:K}\;;\mathbf{x}_{m}\big{)}=\sum_{v=1}^{V}\text{count}\big{(}w_{m,v}\big{)}\cdot\log\Bigg{[}\sum_{k=1}^{K}\bm{\phi}_{k,v}\cdot\bm{\theta}_{m,k}\Bigg{]}$
(13)

Perplexity is a crude measure, and at times a higher numerical perplexity may
correspond to a less human-interpretable model, so care must be applied when
relying on this metric. Another tool for model evaluation is _coherence_[12,
13], though this is beyond the scope of this paper. We can directly compare
the perplexity between two models to evaluate if one is a better fit than the
other, with a lower score being more optimal.

###  4.4 Optimal Model

To build intuitions, we present this technical aside on the nature of the
joint log-likelihood for LDA :

$\log\mathbb{P}\big{(}\bm{\theta}_{1:M},\bm{\phi}_{1:K}\;;\mathbf{x}_{1:M}\big{)}=\sum_{k=1}^{K}\log\mathbb{P}\big{(}\bm{\phi}_{k}\big{)}+\sum_{m=1}^{M}\bigg{[}\log\mathbb{P}\big{(}\bm{\theta}_{m}\big{)}+\sum_{i=1}^{N_{m}}\big{(}\log\bm{\theta}_{m,z_{m,i}}+\log\bm{\phi}_{z_{m,i},w_{m,i}}\big{)}\bigg{]}$
(14)

Observe that there are precisely two mechanisms via which we can improve this
function :

  * •

For a firm, choose high-likelihood industries that maximize
$\mathbb{P}\big{(}\text{industry}=z_{m,i}\mid\text{firm}=m\big{)}$.

  * •

For an industry, choose high-likelihood keywords that maximize
$\mathbb{P}\big{(}\text{word}=w_{m,i}\mid\text{industry}=z_{m,i}\big{)}$.

The key insight here is that these two objectives are _mutually exclusive_ ,
thus during inference we balance a delicate trade-off [14]. By selecting fewer
industries for a firm, we require them to represent a broader set of words By
selecting fewer words for an industry, we require more industries to represent
a single firm.

These competing goals encourage _peaky_ distributions to form for both
$\bm{\theta}$s and $\bm{\phi}$s, which causes both the posteriors for the
industry-mixtures and the word-mixtures to appear approximately sparse. Thus
most firms will be part of a small number of industries, and most industries
will be defined by a small number of words. This significantly helps with
interpretability in practice.

###  4.5 Posterior Inference via Gibbs Sampling

There exist various robust approaches for approximating the posterior of a
mixed-membership model, including Gibbs-Sampling [15], Expectation-
Maximization [16], and Variational Inference [17]. For simplicity, here we
will only focus on Gibbs Sampling for LDA, as it is mechanically transparent
and provides useful intuitions for understanding how the optimal solution is
obtained.

Gibbs Sampling is a flavor of Monte Carlo Markov Chain (MCMC), a simulation-
based technique that generates a sequence of samples that asymptotically
converges to the true posterior. Here, we present the simplest form of the
algorithm to develop intuitions around the underlying math [18, 19].

For this algorithm, we require _prior_ values for $\bm{\alpha}^{\text{prior}}$
and $\bm{\beta}^{\text{prior}}$ to be specified by the user. In practice,
these can either be set to constant values (producing a symmetric prior), or
can be calibrated by hand in order to coerce the emergence of certain topics.
Further, the user must also specify the number of industries $K$ and the total
number of samples $S$.

Algorithm 1 $\text{Gibbs Sampler for Latent Dirichlet Allocation
}\big{(}\mathbf{X},K,\bm{\beta}^{\text{prior}},\bm{\alpha}^{\text{prior}},S\big{)}$
:

1:$\\{\bm{\theta}_{1:M},\bm{\phi}_{1:K}\\}^{(0)}\sim\text{Random}(\cdot\big{)}$
// Initialize word-mixtures and industry-mixtures.

2:for $s\in\\{1,2,\dots,S\\}$ do// Repeat process for $S$-many samples.

3: for $m\in\\{1,2,\dots,M\\}$ do // Per each $m$-th firm :

4: for $n\in\\{1,2,\dots,N_{m}\\}$ do // Per each $n$-th word for the $m$-th
firm :

5:
$\mathbf{z}_{m,n}^{(s)}\sim\mathbb{P}\big{(}\text{Industry}_{k}\mid\text{Firm}_{m}\big{)}$//
- probabilistically assign each word to an industry.

6: end for

7: end for

8: for $k\in\\{1,2,\dots,K\\}$ do // Per each $k$-th industry:

9:
$\bm{\beta}_{k}^{(s)}\leftarrow\text{count}\big{[}\mathbf{z}_{1}^{(s)},\dots,\mathbf{z}_{m}^{(s)}\big{]}$
// - count word-$v$ assigned to industry-$k$ across all firms.

10:
$\bm{\beta}_{k}^{\text{new}}\leftarrow\bm{\beta}_{k}^{\text{prior}}+\bm{\beta}_{k}^{(s)}$
// - update parameter of probability model.

11:
$\bm{\phi}_{k}^{(s)}\sim\text{Dirichlet}\big{(}\bm{\beta}_{k}^{\text{new}})$
// - sample a new word-mixture.

12: end for

13: for $m\in\\{1,2,\dots,M\\}$ do // Per each $m$-th firm :

14:
$\bm{\alpha}_{m}^{(s)}\leftarrow\text{count}\big{[}\mathbf{z}_{m}^{(s)}\big{]}$//
- count all words assigned to each $k$-th industry.

15:
$\bm{\alpha}_{m}^{\text{new}}\leftarrow\bm{\alpha}_{m}^{\text{prior}}+\bm{\alpha}_{m}^{(s)}$
// - update parameter of the probability model.

16:
$\bm{\theta}_{m}^{(s)}\sim\text{Dirichlet}_{K}\big{(}\bm{\alpha}_{m}^{\text{new}}\big{)}$
// - sample a new industry-mixture.

17: end for

18:end for

19:return $\\{\bm{\theta}_{1:M},\bm{\phi}_{1:K}\\}^{(0,\dots,S)}$// Return the
estimated samples.

In practice, we discard the first $S^{*}$ many of the returned samples as part
of the _burn-in_ process, and could take the mean of the remainder as our best
estimate for the optimal value.

We highlight that fitting such a model simply amounts to _counting_ the number
of words assigned to each industry, and then iteratively re-calibrating until
we arrive at a stable solution. In general, higher values for $S$ and $S^{*}$
yield a better model, and in practice our only limitation to convergence is
hardware. We note that there exist more elaborate extensions of this algorithm
in the literature, though they are beyond the scope of this paper [20, 21, 22,
23]. As this architecture is clearly very simple, we emphasize that text pre-
processing is the primary mechanism for influencing the optimal solution of
this model.

##  5 Minimal Illustrative Example

Consider a collection of hypothetical pre-processed BOW business descriptions
for three fictional firms :

Nile $\displaystyle=\big{\\{}\text{``retail'', ``cloud'', ``movie'',
``movie'', ``retail'', ``cloud``}\big{\\}}$ Tallmart
$\displaystyle=\big{\\{}\text{``retail'', ``retail'', ``store''}\big{\\}}$
CloudFilms $\displaystyle=\big{\\{}\text{``movie'', ``cinema'', ``web-app'',
``cloud''}\big{\\}}$

Suppose we fit an MIS model on this data with $K=3$ to generate the following
industry representations :

$\displaystyle\emph{Retail}:\phi_{1}$
$\displaystyle=\mathbb{P}\big{(}\text{word}\mid\text{industry}=1\big{)}=\big{[}\;\underset{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}{\text{``cinema''}}}{0.01},\underset{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\text{``cloud''}}{0.02},\underset{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\text{``movie''}}{0.01},\underset{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\text{``retail''}}{\textbf{0.61}},\underset{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\text{``store''}}{\textbf{0.33}},\underset{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\text{``web-
app''}}{0.02}\;\big{]}$ $\displaystyle\emph{Movie}:\phi_{2}$
$\displaystyle=\mathbb{P}\big{(}\text{word}\mid\text{industry}=2\big{)}=\big{[}\;\underset{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}{\text{``cinema''}}}{\textbf{0.31}},\underset{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\text{``cloud''}}{0.01},\underset{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\text{``movie''}}{\textbf{0.64}},\underset{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\text{``retail''}}{0.02},\underset{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\text{``store''}}{0.01},\underset{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\text{``web-
app''}}{0.01}\;\big{]}$ $\displaystyle\emph{Tech}:\phi_{3}$
$\displaystyle=\mathbb{P}\big{(}\text{word}\mid\text{industry}=3\big{)}=\big{[}\;\underset{\text{``cinema''}}{0.02},\underset{\text{``cloud''}}{\textbf{0.45}},\underset{\text{``movie''}}{0.02},\underset{\text{``retail''}}{0.02},\underset{\text{``store''}}{0.01},\underset{\text{``web-
app''}}{\textbf{0.48}}\;\big{]}$

The names of the topics are intuited from the maximal values of each
distribution. _Retail_ is dominated by “retail” and “store”, _Movie_ is
dominated by “movie” and “cinema”, and _Tech_ is dominated by “cloud” and
“web-app”. Functionally, we can ignore all of the other values, as they are
arbitrarily small.

MIS also provides us with the following _industry embeddings_ :

$\displaystyle\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}{\theta_{1}}$
$\displaystyle=\mathbb{P}\big{(}\text{industry}\mid\text{firm =
}\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}{\text{Nile}}$
$\displaystyle\big{)}=\big{[}\;\underset{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}{\emph{Retail}}}{\textbf{0.34}},\underset{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}{\emph{Movie}}}{\textbf{0.33}},\underset{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}{\emph{Tech}}}{\textbf{0.33}}\;\big{]}$
$\displaystyle\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{\theta_{2}}$
$\displaystyle=\mathbb{P}\big{(}\text{industry}\mid\text{firm =
}\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{\text{Tallmart}}$
$\displaystyle\big{)}=\big{[}\;\underset{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}{\emph{Retail}}}{\textbf{0.98}},\underset{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}{\emph{Movie}}}{0.01},\underset{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}{\emph{Tech}}}{0.01}\;\big{]}$
$\displaystyle\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\theta_{3}}$
$\displaystyle=\mathbb{P}\big{(}\text{industry}\mid\text{firm =
}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\text{CloudFilms}}$
$\displaystyle\big{)}=\big{[}\;\underset{\emph{Retail}}{0.01},\underset{\emph{Movie}}{\textbf{0.66}},\underset{\emph{Tech}}{\textbf{0.33}}\;\big{]}$

Within this simplified three-industry market, we see that Nile is a
_conglomerate_ with membership dispersed across multiple industries, while
Tallmart is a _pure play_ with membership concentrated in only one. CloudFilms
is in the middle. We can visualize this data as follows :

$\phi_{1}$
---
_Retail_
$\phi_{2}$
---
_Movie_
$\phi_{3}$
---
_Tech_
$\theta_{2}$$\theta_{1}$$\theta_{3}$ Figure 5: Minimal Example of a Multi-
Industry Simplex

Notice that in the above space, Tallmart
($\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{\theta_{2}}$)
is physically closer to Nile
($\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}{\theta_{1}}$)
than it is to CloudFilms
($\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\theta_{3}}$).
This is a mathematically rigorous way of demonstrating that Tallmart is _more
similar_ to Nile than it is to CloudFilms, on the basis of text-similarity.

To put things into perspective, for real-world applications, we would have
thousands firms in our universe, each business description would have several
thousand words, and the number of industries could be over 100. At that point,
visualizing the data becomes much more tricky, but all of the intuitions that
we build in this low-dimensional example generalize naturally to a higher-
dimensional space.

##  6 Applications

Utilizing a universe of over 5000 firms, we calibrate and estimate MIS with 97
industries333Due to observed instability, we elected to remove all real estate
and REIT firms from our sample. This instability is primarily attributed to
semantic ambiguity in the business descriptions, and exceedingly high
quantities of false-positives in the model. We hope to resolve this issue in
future iterations of the model.. We emphasize that what we illustrate here is
just a single implementation of what an MIS model can look like, and different
choices in data sourcing and semantic tree construction will yield varying
outputs. Our goal here is to motivate and justify the general methodology,
rather than to fixate on a specific model instance.

We represent the model-estimated industries with the following network, where
each node corresponds to an MIS-industry. Node size is proportionate to
probability of incidence and color corresponding to the most common GICS-
sector of firms within that MIS-industry. An edge between two nodes indicates
that two industries have non-trivial co-occurrence, meaning that these
industries are linked within the data :

Figure 6: MIS Industry Network, Colored by Most-Associated GICS Sector

With this plot we emphasize two important points :

  * •

At a high-level, MIS has a lot in common with GICS, particularly in terms of
_global structure_. Within the plot we clearly observe a “sector” of health-
oriented firms, a “sector” of finance-related firms, and so on. Here, we
demonstrate that MIS doesn’t attempt to reinvent the wheel.

  * •

At a more detailed-level however, MIS represents a more nuanced (and perhaps a
more _realistic_) representation of how industries relate to one another in a
market. The abundance of edges between nodes highlights just how
interconnected various industries are, which shouldn’t be surprising from an
economic standpoint. Thus, we argue that MIS is a _better_ wheel.

The particular MIS model instance that we show in this paper uses a large
collection of private and public text datasets that are not all available
before 2021, and because of this a proper historical backtest is not possible.
Despite this limitation, we highlight several use-cases that are relevant to
asset managers and discuss future directions of research as we strive towards
improving the credibility of this methodology.

###  6.1 Examples of MIS “Sectors”

To best highlight the differences between MIS and GICS, we illustrate several
collections of MIS-industries that are intended to evoke the structure of
GICS-sectors.

First, consider the following commodities-related MIS industries, with the top
three associated keywords :

OIL DRILLING | : | “oil-drilling” ($82\%$) | “gas-pipeline” ($9\%$) | “fracking” ($7\%$)
---|---|---|---|---
METAL | : | “metal” ($46\%$) | “ferrous” ($37\%$) | “nonferrous” ($14\%$)
WOOD | : | “wood” ($64\%$) | “timber” ($27\%$) | “lumber” ($10\%$)
MEAT | : | “meat” ($49\%$) | “red-meat” ($18\%$) | “poultry” ($16\%$)
MARIJUANA | : | “marijuana” ($54\%$) | “cannabis” ($44\%$) | “t.h.c.” ($2\%$)
Table 1: Subset of MIS Commodities Industries

The first four MIS-industries have direct analogues to GICS-industries, while
_marijuana_ is something that is completely unrepresented in GICS.

As a second example, consider several healthcare-oriented MIS-industries :

ONCOLOGY : | “oncology” ($85\%$) | “carcinoma” ($10\%$) | “lymphoma” ($9\%$)
---|---|---|---
MICROBIOLOGY : | “microbiology” ($58\%$) | “virology” ($18\%$) | “bacteriology” ($15\%$)
GENOMICS : | “genomics” ($66\%$) | “gene-editing” ($23\%$) | “d.n.a.” ($10\%$)
SURGERY : | “surgery” ($72\%$) | “organ-transplant” ($17\%$) | “orthopedics” ($12\%$)
CARDIOLOGY : | “cardiology” ($65\%$) | “heart-attack” ($22\%$) | “heart” ($11\%$)
Table 2: Subset of MIS Healthcare Industries

Unlike GICS, which simply partitions all of healthcare into broad categories
like _biotechnology_ and _pharmaceuticals_ , MIS is able to leverage the
specificity of medical language to identify precisely which areas of health a
firm is working in, providing a much richer description of the relevant sub-
industries.

As a final example, consider some of the technology industries that MIS is
able to discover :

A.I. : | “artificial-intelligence” ($60\%$) | “computer-vision” ($23\%$) | “n.l.p.” ($15\%$)
---|---|---|---
STREAMING : | “media-streaming” ($49\%$) | “television” ($21\%$) | “music-streaming” ($14\%$)
BLOCKCHAIN : | “blockchain” ($56\%$) | “cryptocurrency” ($44\%$) | -
ROBOTICS : | “robotics” ($59\%$) | “factory-automation” ($31\%$) | -
CAD : | “c.a.d.” ($52\%$) | “3d-print” ($48\%$) | -
Table 3: Subset of MIS Tech Industries

Here, we are again able to construct industries that are completely
unrepresented in GICS (see section 5.3 for more discussion on this). As a
final note here, we emphasize that the emergence of these precise industries
is a function of the text sources and pre-processing choices made by the
practitioner. Richer input text and more elaborate pre-processing can lead to
more robust topics, and these should only be considered for illustrative
purposes.

###  6.2 Nearest Neighbors

We can utilize MIS for nearest neighbor identification, finding "most similar"
firms on the basis of text. We utilize Hellinger similarity as a “correlation”
measure between the descriptions of two firms :

$\displaystyle\text{similarity}\big{(}\text{firm}_{i},\text{firm}_{j}\big{)}=1-\frac{1}{\sqrt{2}}\Big{\|}\sqrt{\bm{\theta}_{i}}-\sqrt{\bm{\theta}_{j}}\Big{\|}_{2}$
(15)

As an illustrative example, consider 20 Amazon neighbors with a Hellinger
similarity greater than 0.25 (sorted by size). Each node represents one of
these neighboring firms, where color corresponds to GICS sector and size
corresponds to firm size. An edge between nodes indicates that two firms are
“neighbors”.

Figure 7: Nearest Neighbors for Amazon

MIS identifies four major industries that Amazon is a part of : retail
(Amazon’s primary business), cloud computing (Amazon Web Services), media
streaming (Amazon Prime Video), and artificial intelligence. Amazon’s largest
neighbors span _four_ GICS sectors - information technology, communications
services, consumer discretionary, and consumer staples - highlighting its high
level of diversification.

As a second example, consider Apple, which at the time of writing is the
world’s largest publicly traded firm.

Figure 8: Nearest Neighbors for Apple

Again, MIS is able to identify four major industries for a highly diversified
firm : consumer tech (iPhone, iPad, Mac), artificial intelligence, media
streaming (Apple Music, Apple TV+), and payment technology (Apple Pay). As
with Amazon, the nearest neighbor list similarly represents multiple GICS
sectors.

###  6.3 Thematic Portfolios

We can also utilize MIS for constructing thematic portfolios, which have been
popular in recent years as a means of investing in innovative technologies.
For illustrative example, we construct a thematic portfolio by identifying all
firms with a probability of inclusion in an industry (aka “theme”) exceeding
5$\%$. The portfolio weight is a function of firm size ($s_{i}$) and
probability of inclusion, defined by the following formula :

$w_{i}\propto\sqrt{s_{i}}\cdot\mathbb{P}\big{(}\text{industry}\mid\text{firm}_{i}\big{)}$
(16)

As an example, consider an _artificial intelligence_ portfolio. Again, each
nodes represents a firm, where color corresponds to GICS sector and node size
corresponds to firm size. The positions of the nodes are determined by tSNE
[24] such that more-similar firms are positioned more closely together.

Figure 9: Extraction of AI Firms from Relevant GICS Sectors

The ten largest firms represent three sectors, highlighting just how sector-
agnostic _artificial intelligence_ really is. Beyond that, the long tail of
smaller names also includes firms in industrials, healthcare, and financials -
producing a portfolio that would not be possible to construct systematically
using GICS.

As a second example, consider a _robotics_ portfolio :

Figure 10: Extraction of AI Firms from Relevant GICS Sectors

As above, this portfolio represents multiple GICS sectors, including a cluster
of surgical robots (healthcare), oil drilling robots (energy), industrial
robots, and several firms in information technology. Without a tool like MIS,
the only way to construct such a portfolio would be to do it entirely by hand,
thus highlighting the significant utility of our model to investors who want
to allocate to emerging industries.

##  7 Conclusion

What we present here is a proof-of-concept. Probabilistic industry
classification is still a highly nascent area of research, with many exciting
potential improvements and useful applications. MIS is merely a starting point
for a much larger conversation on the various applications of big data and
advanced artificial intelligence systems for improving the quality of services
that asset managers can provide.

###  7.1 Current Model Limitations

In the spirit of intellectual honesty and academic transparency, we believe
that it is critically important for us to address key limitations of our
process as we iterate towards better solutions :

  * •

Bias : as mentioned in section 2.3, MIS involves text pre-processing via the
construction of semantic trees, which must be done by hand and thus reflects
the subjective perspectives of the practitioner.

  * •

Incompleteness : as described in section 3.2, MIS leverages a mixed-membership
model that categorizes the individual words of the underlying text. Thus, the
model is only able to identify industries that are explicitly mentioned in the
input text data.

  * •

Noise : as demonstrated in section 3.4, MIS utilizes posterior approximation
for estimating industry-mixtures, which introduces variation due to randomness
in the optimization process.

Despite such limitations, which are unfortunately unavoidable for many
artificial intelligence pipelines, we find that MIS estimated industry-
mixtures are incredibly useful. Further, when sufficient care is applied to
data sourcing, text pre-processing, and model regularization - the negative
effects of these limitations can be significantly mitigated, providing a
reliable tool for asset management applications.

###  7.2 Future Research Directions

As with any emerging technology, there exist many obvious low-hanging-fruit
for additional investigation. Here, we highlight three potential extensions
that we are particularly excited about :

  * •

LLM Text Pre-Processing : though we have been critical of LLMs as the primary
machinery for industry classification, we believe that there exist many
interesting ways that they can be used for low-impact tasks such as text pre-
processing, in which they may help to de-noise unruly documents.

  * •

Private Equity Proxies : beyond simply identifying the nearest neighbors of a
publicly traded firm, we can use MIS to identify industry profiles for
privately held firms, using only a text-based description. This allows for
approximate risk modeling for firms that do not provide quantitative data
publicly.

  * •

Risk Modeling : many popular fundamental risk models use GICS to stratify
systematic risk by industry. By replacing GICS with MIS it may be possible to
improve the quality of fit for such models.

We intend to explore each of these direction of research in the coming years,
and hope to inspire industry peers to develop innovations in similar
technologies.

###  7.3 Closing Remarks

Though GICS has proven immensely useful for asset managers over the past
several decades, the advent of the artificial intelligence has paved the way
for more robust data-oriented approaches that can circumvent the limitations
of existing systems. Amazon is so much more than a bookstore, and MIS provides
a rigorous framework to describe it.

## References

  * [1] MSCI, “The global industry classification standard (gics),” 2023.
  * [2] D. Vamvourellis, M. Toth, S. Bhagat, D. Desai, D. Mehta, and S. Pasquali, “Company similarity using large language models,” 2023.
  * [3] M. Rizinski, A. Jankov, V. Sankaradas, E. Pinsky, I. Miskovski, and D. Trajanov, “Company classification using zero-shot learning,” 2023.
  * [4] S. Wood, R. Muthyala, Y. Jin, Y. Qin, N. Rukadikar, H. Gao, and A. Rai, “Automated industry classification with deep learning,” in 2018 IEEE 12th International Conference on Semantic Computing (ICSC), pp. 64–70, 2018\.
  * [5] D. Wu, Q. Wang, and D. L. Olson, “Industry classification based on supply chain network information using graph neural networks,” Applied Soft Computing, vol. 132, p. 109849, 2023.
  * [6] OpenAI, “Gpt-4 technical report,” 2023.
  * [7] H. Touvron and et al., “Llama: Open and efficient foundation language models,” 2023.
  * [8] D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent dirichlet allocation,” Journal of Machine Learning Research, vol. 3, pp. 993–1022, 2003.
  * [9] Y. Cha and J. Cho, “Social-network analysis using topic models,” Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 565–574, 2013.
  * [10] J. K. Pritchard, M. Stephens, and P. Donnelly, “Inference of population structure using multilocus genotype data,” Genetics, vol. 155, no. 2, pp. 945–959, 2000.
  * [11] F. Fang, K. Dutta, and A. Datta, “Lda-based industry classification,” Thirty Fourth International Conference on Information Systems, Milan, 2013.
  * [12] D. Newman, J. Lau, K. Grieser, and T. Baldwin, “Automatic evaluation of topic coherence,” Human Language Technologies : The 2010 Annual Conference of the North American Chapter of the Association for Computational Statistics, pp. 100–108, 2010.
  * [13] A. Hinneburg and A. Both, “Exploring the space of topic coherence measures,” WSDM ’15 : Proceedings of the Eigth ACM International Conference on Web Search and Data Mining, pp. 399–408, 2015.
  * [14] D. Blei, “Mixed-membership models,” Lecture Notes for Columbia University, 2015.
  * [15] S. Geman, D. Geman, K. Abend, T. Harley, and L. Kanal, “Stochastic relaxation, gibbs distributions and the bayesian restoration of images,” Journal of Applied Statistics, vol. 20, no. 5, pp. 25–62, 1993.
  * [16] D. Blei, A. Ng, and M. I. Jordan, “Latent dirichlet allocation,” Journal of Machine Learning Research, vol. 3, pp. 993–1022, 2003.
  * [17] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe, “Variational inference: A review for statisticians,” Journal of the American Statistical Association, vol. 112, pp. 859–877, apr 2017.
  * [18] T. Griffiths and M. Steyvers, “Finding scientific topics,” Proceedings of the National Academy of Science, vol. 101, pp. 5228–5235, 2004.
  * [19] Y. Yang, S. Pan, and Y. Song, “Improving topic model stability for effective document exploration,” Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI-16), pp. 4223–4227, 2016\.
  * [20] Y. W. Teh, M. I. Jordan, M. Beal, and D. Blei, “Hierarchical dirichlet process,” Advances of the American Statistical Association, vol. 101, pp. 1566–1581, 2006.
  * [21] D. Blei and J. Lafferty, “Correlated topic models,” Advances in Neural Information Processing Systems 18 (NIPS), 2005.
  * [22] D. Blei, T. Griffiths, M. I. Jordan, and J. Tenenbaum, “Hierarchical topic models and the nested chinese restaurant process,” Advances in Neural Information Processing Systems 16 (NIPS), 2003.
  * [23] W. Li and A. McCallum, “Pachinko allocation : Dag-structure mixture models of topic correlations,” Proceedings of the 23rd International Conference on Machine Learning, 2006.
  * [24] L. v. d. Maaten and G. Hinton, “Visualizing data using t-sne,” Journal of Machine Learning Research, vol. 9, no. Nov, pp. 2579–2605, 2008.