interior}:\quad(\pm f)|_{Q}\in\mathcal{BA}_{\tau\sqrt{d}C}(Q).$

Step 3 (Sets with Barron boundary as in [9]): For $x\in\mathbb{R}^{d}$ and
$j\in\\{1,\dots,d\\}$, let us write
$x^{(j)}:=(x_{1},\dots,x_{j-1},x_{j+1},\dots,x_{d})\in\mathbb{R}^{d-1}$.
Furthermore, for $x,y\in\mathbb{R}^{d}$, let us write $x\leq y$ if $x_{i}\leq
y_{i}$ for all $i\in\\{1,\dots,d\\}$. In [9, Definition 3.3], given $B>0$ and
$a,b\in\mathbb{R}^{d}$ with $a\leq b$ and setting
$Q=[a,b]:=\\{x\in\mathbb{R}^{d}\colon a\leq x\leq b\\}$ and
$Q^{(j)}:=[a^{(j)},b^{(j)}]$, a function $F:Q\to\\{0,1\\}$ is said to belong
to the class $\mathcal{BH}_{B}(Q)$ of “Barron horizon functions” if

$\exists\,j\in\\{1,\dots,d\\},\theta\in\\{\pm 1\\},\text{ and
}f\in\mathcal{BA}_{B}\bigl{(}[a^{(j)},b^{(j)}]\bigr{)}:\quad\forall\,x\in
Q:\quad F(x)=\mathds{1}_{\theta x_{j}\leq f(x^{(j)})}.$

Finally, a compact set $\Omega\subset\mathbb{R}^{d}$ is said to belong to the
set $\mathcal{BB}_{B,M}(\mathbb{R}^{d})$ of sets with “Barron class boundary”,
if there exist (closed, axis-aligned, non-degenerate) rectangles
$Q_{1},\dots,Q_{M}\subset\mathbb{R}^{d}$ with pairwise disjoint interiors and
with $\Omega\subset\bigcup_{i=1}^{M}Q_{i}$ and such that
$\mathds{1}_{Q_{i}\cap\Omega}\in\mathcal{BH}_{B}(Q_{i})$ for all
$i\in\\{1,\dots,M\\}$.

Step 4 (Completing the proof): Let $\Omega\subset[0,1]^{d}$ be compact with
$\mathds{1}_{\Omega}\in\mathcal{C}\ell_{B(C)}(d,M)$. That is, there exist
(closed, axis-aligned, non-degenerate) rectangles
$Q_{1},\dots,Q_{M}\subset[0,1]^{d}$ with pairwise disjoint interiors and with
$\Omega\subset\bigcup_{i=1}^{M}Q_{i}$ and such that for each
$i\in\\{1,\dots,M\\}$, there exist $f_{i}\in H_{B(C)}$ and a permutation
matrix $P_{i}\in\mathbb{R}^{d\times d}$ such that

$\mathds{1}_{\Omega}=f_{i}\circ
P_{i}\qquad\text{or}\qquad\mathds{1}_{\Omega}=1-f_{i}\circ
P_{i}\quad\text{almost everywhere on }Q_{i}.$

Fix $i\in\\{1,\dots,M\\}$. Since $f_{i}\in H_{B(C)}$, there exists
$b:[0,1]^{d-1}\to[0,1]$ satisfying $b=b_{i}\in B(C)$ and
$f_{i}(x)=\mathds{1}_{B(x_{1},\dots,x_{d-1})\leq x_{d}}$ for all
$x\in[0,1]^{d}$.

Given a permutation $\pi\in S_{d}$ and $x\in\mathbb{R}^{d}$, define
$x^{\pi}\in\mathbb{R}^{d}$ by $x^{\pi}_{i}=x_{\pi^{-1}(i)}$ and note
$x^{\pi\circ\sigma}=(x^{\sigma})^{\pi}$ for permutations $\pi,\sigma\in
S_{d}$. We can then choose $\pi\in S_{d}$ satisfying $P_{i}x=x^{\pi}$ for all
$x\in\mathbb{R}^{d}$. Set $\widetilde{\pi}:=\pi\circ\sigma$, where $\sigma\in
S_{d}$ is the permutation that swaps $d$ and $j:=\pi^{-1}(d)$ and leaves the
rest of $\\{1,\dots,d\\}$ fixed. Note
$\widetilde{\pi}(d)=\pi(\sigma(d))=\pi(\pi^{-1}(d))=d$, meaning that
$\widetilde{\pi}$ can be seen as an element of $S_{d-1}$. Let
$\widetilde{P}\in\mathbb{R}^{(d-1)\times(d-1)}$ be the associated permutation
matrix satisfying $\widetilde{P}z=z^{\widetilde{\pi}}$ for all
$z\in\mathbb{R}^{d-1}$.

Given $x\in[0,1]^{d}$, let $y:=x^{\sigma^{-1}}$, noting that
$x^{\pi}=x^{\widetilde{\pi}\circ\sigma^{-1}}=(x^{\sigma^{-1}})^{\widetilde{\pi}}=y^{\widetilde{\pi}}$
and $x_{d}^{\pi}=x_{\pi^{-1}(d)}=x_{j}$, and furthermore

$(y_{1},\dots,y_{d-1})=(x_{1},\dots,x_{j-1},x_{d},x_{j+1},\dots,x_{d-1})=S(x_{1},\dots,x_{j-1},x_{j+1},\dots,x_{d})=Sx^{(j)}$

for a suitable permutation matrix $S\in\mathbb{R}^{(d-1)\times(d-1)}$.
Directly from Definition 2.1, it follows that
$\widetilde{b}:=b\circ\widetilde{P}\circ S\in B(C)$. Hence,

$\begin{split}(f_{i}\circ
P_{i})(x)&=f_{i}(x^{\pi})=\mathds{1}_{b(x_{1}^{\pi},\dots,x_{d-1}^{\pi})\leq
x_{d}^{\pi}}=\mathds{1}_{b(y_{1}^{\widetilde{\pi}},\dots,y_{d-1}^{\widetilde{\pi}})\leq
x_{j}}\\\ &=\mathds{1}_{b\circ\widetilde{P}(y_{1},\dots,y_{d-1})\leq
x_{j}}=\mathds{1}_{b\circ\widetilde{P}\circ S(x^{(j)})\leq
x_{j}}=\mathds{1}_{\widetilde{b}(x^{(j)})\leq
x_{j}}=\mathds{1}_{-x_{j}\leq-\widetilde{b}(x^{(j)})}\qquad\forall\,x\in[0,1]^{d}.\end{split}$

Now, if $\mathds{1}_{\Omega}=f_{i}\circ P_{i}$ almost everywhere on $Q_{i}$,
then we see that $\mathds{1}_{\Omega\cap
Q_{i}}(x)=\mathds{1}_{-x_{j}\leq-\widetilde{b}(x^{(j)})}$ for _all(!)_ $x\in
Q_{i}$, as can be seen by compactness of $\Omega$ and continuity of
$\widetilde{b}$. Furthermore, as seen in Step 2, we have
$(-\widetilde{b})|_{Q_{i}^{(j)}}\in\mathcal{BA}_{\tau\sqrt{d}C}\bigl{(}Q_{i}^{(j)}\bigr{)}$
and thus $\mathds{1}_{\Omega\cap
Q_{i}}\in\mathcal{BH}_{\tau\sqrt{d}C}(Q_{i})$.

If otherwise $\mathds{1}_{\Omega}=1-f_{i}\circ P_{i}$ almost everywhere on
$Q_{i}$, then

$\mathds{1}_{\Omega\cap Q_{i}}(x)=1-\mathds{1}_{\widetilde{b}(x^{(j)})\leq
x_{j}}=\mathds{1}_{x_{j}<\widetilde{b}(x^{(j)})}=\mathds{1}_{x_{j}\leq\widetilde{b}(x^{(j)})}$

for _almost_ all $x\in Q_{i}$. By compactness of $\Omega$ and continuity of
$\widetilde{b}$, this implies $\mathds{1}_{\Omega\cap
Q_{i}}(x)=\mathds{1}_{x_{j}\leq\widetilde{b}(x^{(j)})}$ for _all_ $x\in
Q_{i}$. As above, we thus see $\mathds{1}_{\Omega\cap
Q_{i}}\in\mathcal{BH}_{\tau\sqrt{d}C}(Q_{i})$. Overall, we have shown as
claimed that $\Omega\in\mathcal{BB}_{\tau\sqrt{d}C,M}(\mathbb{R}^{d})$. ∎

###  A.3 Equivalence of convergence in $L^{1}$ and in measure on the set of
probability densities

We show here that the topology of convergence in measure coincides with the
topology induced by $\|\cdot\|_{L^{1}(\lambda)}$ on the set $\mathcal{D}$ of
probability density functions. Since both topologies are metrizable (see e.g.
[25, Theorem 6.7] for convergence in measure), it suffices to show that the
convergent sequences are the same. It is well-known (see e.g. [25, Remark
6.11]) that convergence in $L^{1}$ implies convergence in measure. Conversely,
suppose towards a contradiction that there exists a sequence
$(p_{n})_{n\in\mathbb{N}}\subset\mathcal{D}$ and $p\in\mathcal{D}$ satisfying
$p_{n}\to p$, but $\|p_{n}-p\|_{L^{1}(\lambda)}\not\to 0$. Passing to a
subsequence, we can assume that $\|p_{n}-p\|_{L^{1}(\lambda)}\geq\varepsilon$
for all $n\in\mathbb{N}$ and some $\varepsilon>0$. By passing to a further
subsequence, we can assume that $p_{n}\to p$ almost everywhere; see e.g. [25,
Corollary 6.13]. Now, [23, Lemma 1.34] shows because of
$\|p_{n}\|_{L^{1}(\lambda)}=1=\|p\|_{L^{1}(\lambda)}$ for all $n$ that
$\|p_{n}-p\|_{L^{1}(\lambda)}\to 0$, in contradiction to
$\|p_{n}-p\|_{L^{1}(\lambda)}\geq\varepsilon$ for all $n\in\mathbb{N}$.
$\square$

###  A.4 The relation between Kullback-Leibler and Hellinger distance

Recall the elementary estimate $1+x\leq e^{x}$ for $x\in\mathbb{R}$. Hence,
$y\leq e^{y-1}$ for $y\in\mathbb{R}$. For $y>0$, we can take the logarithm to
obtain $\ln(y)\leq y-1$ and hence $-\ln(y)\geq 1-y$.

Hence,

$\displaystyle D(p\parallel q)$
$\displaystyle=2\cdot\int-\ln(\sqrt{q}/\sqrt{p})\cdot p\,d\lambda\geq
2\cdot\int p\cdot\big{(}1-\sqrt{q}/\sqrt{p}\big{)}\,d\lambda$
$\displaystyle=2-2\int\sqrt{q}\sqrt{p}\,d\lambda,$

since $\int p\,d\lambda=1$. Finally, note because of $\int p\,d\lambda=1=\int
q\,d\lambda$ that

$\displaystyle d_{H}^{2}(p,q)$
$\displaystyle=\int(\sqrt{p}-\sqrt{q})^{2}\,d\lambda=\int
p+q-2\sqrt{p}\sqrt{q}\,d\lambda$
$\displaystyle=2-2\int\sqrt{p}\sqrt{q}\,d\lambda.$

Overall, this implies that $d_{H}^{2}(p,q)\leq D(p\parallel q)$ and in
particular that $D(p\parallel q)\geq 0$.

###  A.5 Proof of Theorem 5.6

###### Proof.

In this proof, we show how the result stated above can be derived from [24,
Theorem A.2]. The first difference between the present setting and the one in
[24] is that we consider binary classification with values in $\\{0,1\\}$,
whereas [24] consider values in $\\{\pm 1\\}$. Therefore, we define $g:=2h-1$
and $\mathcal{F}_{m}:=\\{2f_{m}^{\ast}-1\colon
f_{m}^{\ast}\in\mathcal{F}_{m}^{\ast}\\}$.

Furthermore, with $X\sim U([0,1]^{d})$, we let $\mathrm{Pr}$ be the image
measure of $(X,g(X))$ on $[0,1]^{d}\times\\{\pm 1\\}$; that is, for each Borel
set $A\subset[0,1]^{d}\times\\{\pm 1\\}$, we have

$\mathrm{Pr}(A)=\boldsymbol{\lambda}\big{(}\bigl{\\{}x\in[0,1]^{d}\colon\bigl{(}x,g(x)\bigr{)}\in
A\bigr{\\}}\big{)}.$

It is then straightforward to see for any measurable $f:[0,1]^{d}\to[-1,1]$
that the $\phi$-risk defined in [24, Column 5] as

$\mathcal{E}_{\phi}(f)=\mathbb{E}_{(X,Y)\sim\mathrm{Pr}}\bigl{[}\phi(Y\,f(X))\bigr{]}\quad\text{satisfies}\quad\mathcal{E}_{\phi}(f)=\mathcal{E}_{\phi,\boldsymbol{\lambda},h}(\tfrac{1+f}{2}).$
(A.3)

Furthermore, directly from the definitions it follows that
$\mathcal{E}_{\phi}(g)=0$, so that we can choose $f_{\phi}^{\ast}=g$ in the
notation of [24]. Thus, the _excess $\phi$-risk_ of a measurable
$f:[0,1]^{d}\to[-1,1]$ as defined in [24, Column 5] is given by
$\mathcal{E}_{\phi}(f,f_{\phi}^{\ast})=\mathcal{E}_{\phi}(f)-\mathcal{E}_{\phi}(f_{\phi}^{\ast})=\mathcal{E}_{\phi}(f)$.
Next, the _Bayes classifier_ $C^{\ast}:[0,1]^{d}\to\\{\pm 1\\}$ associated to
the given problem is easily seen to be given by $C^{\ast}(x)=g(x)$, since this
classifier has risk
$\mathbb{E}_{(X,Y)\sim\mathrm{Pr}}\big{[}\mathds{1}_{C^{\ast}(X)\neq
Y}\big{]}\\!=\\!0$. Finally, the _conditional class probability_
$\eta:[0,1]^{d}\\!\to\\![0,1]$, $\eta(x)=\mathrm{Pr}(Y=1\mid X=x)$ in our
setting satisfies $\eta(x)=\mathds{1}_{\Omega}(x)\in\\{0,1\\}$.

We now verify the conditions of [24, Theorem A.2] in the setting from above:

_i)_ _Condition (N)_ from [24], i.e., the so-called _Tsybakov noise condition_
should be satisfied, which requires that there exist $C^{\prime}>0$ and
$q\in[0,\infty]$ satisfying

$\mathrm{Pr}(\\{(X,Y)\colon|2\,\eta(X)-1|\leq t\\})\leq C^{\prime}\cdot
t^{q}\qquad\forall\,t\in(0,\infty).$

In our setting, this is satisfied with $C^{\prime}=1$ and $q=\infty$, since
$\eta(X)\in\\{0,1\\}$ and hence $|2\eta(X)-1|=1$, so that

$\mathrm{Pr}(\\{(X,Y)\colon|2\,\eta(X)-1|\leq
t\\})=\begin{cases}0=t^{\infty},&\text{if }t<1,\\\ 1\leq t^{\infty},&\text{if
}t\geq 1.\end{cases}$

_ii)_ _Condition (A2)_ in [24] is satisfied in our setting, since the
considerations from above combined with the assumptions of the theorem imply
that for each $m\in\mathbb{N}$ there exists a function
$f_{m}^{\ast}\in\mathcal{F}_{m}$ satisfying
$\mathcal{E}_{\phi,\boldsymbol{\lambda},h}(f_{m}^{\ast})\leq a_{n}$, so that
$f_{m}:=2\,f_{m}^{\ast}-1\in\mathcal{F}_{m}$ satisfies

$\mathcal{E}_{\phi}(f_{m},f_{\phi}^{\ast})=\mathcal{E}_{\phi}(f_{m})=\mathcal{E}_{\phi,\boldsymbol{\lambda},h}\bigl{(}\tfrac{1+f_{m}}{2}\bigr{)}=\mathcal{E}_{\phi,\boldsymbol{\lambda},h}(f_{m}^{\ast})\leq
a_{m},$

where also $a_{m}=\mathcal{O}(m^{-a_{0}})$ with $a_{0}>0$.

_iii)_ _Condition (A3)_ in [24] is satisfied for $F_{n}:=1$, since $0\leq
f_{m}^{\ast}\leq 1$ for all $f_{m}^{\ast}\in\mathcal{F}_{m}^{\ast}$ and hence
$\|f_{m}\|_{L^{\infty}}\leq 1=F_{n}$ for all $f_{m}\in\mathcal{F}_{m}$.

_iv)_ Regarding _condition (A5)_ in [24], we first note that [24, Lemma A.2]
shows that _condition (A4)_ is satisfied with $\nu=\frac{q}{q+1}=1$, since
$q=\infty$ (see above). Furthermore, it is well-known that the so-called
_bracketing entropy_ (see [24, Column 18] for a definition)
$H_{B}(\delta,\mathcal{F}_{m},\|\cdot\|_{L^{2}})$ is bounded by the
$L^{\infty}$ covering entropy
$V_{\mathcal{F}_{m},\|\cdot\|_{L^{\infty}}}(\delta/2)$; see for instance [24,
Equation A.1] or [42, Lemma 2.1]. Therefore, setting
$\delta_{m}:=4\delta_{m}^{\ast}$, we see by definition of
$\mathcal{F}_{m}=2\mathcal{F}_{m}^{\ast}-1$ and elementary properties of
entropy numbers that

$\displaystyle H_{B}(\delta_{m},\mathcal{F}_{m},\|\cdot\|_{L^{2}})$
$\displaystyle\leq
V_{\mathcal{F}_{m},\|\cdot\|_{L^{\infty}}}(\delta_{m}/2)\leq
V_{\mathcal{F}_{m}^{\ast},\|\cdot\|_{L^{\infty}}}(\delta_{m}/4)=V_{\mathcal{F}_{m}^{\ast},\|\cdot\|_{L^{\infty}}}(\delta_{m}^{\ast})$
$\displaystyle\leq C\cdot m\cdot\delta_{m}^{\ast}=4C\cdot
m\cdot\delta_{m}=4C\cdot m\cdot\bigl{(}\delta_{m}/F_{m}\bigr{)}^{2-\nu},$

so that _condition (A5)_ from [24] is satisfied with $c_{3}=4C$.

Finally, as in [24, Theorem A.2], we let
$\varepsilon_{m}=\sqrt{\max\\{a_{m},\delta_{m}\\}}$, where we note because of
$\delta_{m}\asymp\delta_{m}^{\ast}$ that
$\varepsilon_{m}\asymp\sqrt{\varepsilon_{m}^{\ast}}$. Hence, recalling that
$q=\infty$, we see in view of Equation 5.3 that

$m\cdot(\varepsilon_{m}^{2}/F_{m})^{(q+2)/(q+1)}=m\cdot\varepsilon_{m}^{2}\asymp
m\cdot\varepsilon_{m}^{\ast}\gtrsim m^{\iota}\gtrsim\ln^{2}(m),$

as required in [24, Theorem A.2].

Overall, we can thus apply [24, Theorem A.2]. By assumption of the current
theorem, $f_{S}\in\mathcal{F}_{m}^{\ast}$ satisfies Equation 5.2 with
$\mathcal{H}=\mathcal{F}_{m}^{\ast}$. Note that if the training sample $S$ is
as in the current theorem, then $T:=(X,g(X_{i}))_{i=1}^{m}$ satisfies
$T\overset{iid}{\sim}\mathrm{Pr}$; furthermore, the map $S\mapsto T$ is a
bijection. In view of the obvious generalization of Equation A.3 to the
_empirical_ risk, this easily shows that $f_{T,m}:=2f_{S}-1\in\mathcal{F}_{m}$
minimizes the empirical $\phi$-risk $\mathcal{E}_{\phi,m}$ (with respect to
the training sample $T$) as defined in [24, Equation (2.2)]. Thus, [24,
Theorem A.2] shows for the true (excess) risk

$\displaystyle\mathcal{E}(f_{T,m},C^{\ast})$
$\displaystyle=\mathrm{Pr}\big{(}\\{(X,Y)\colon
Y\neq\operatorname{sign}(f_{T,m}(X))\\}\big{)}-\mathrm{Pr}\big{(}\\{(X,Y)\colon
Y\neq C^{\ast}(X)\\}\big{)}$ $\displaystyle=\mathrm{Pr}\big{(}\\{(X,Y)\colon
Y\neq\operatorname{sign}(f_{T,m}(X))\\}\big{)}$
$\displaystyle=\boldsymbol{\lambda}\big{(}\\{x\in[0,1]^{d}\colon
g(x)\neq\operatorname{sign}(f_{T,m}(x))\\}\big{)}$
$\displaystyle=\boldsymbol{\lambda}\big{(}\big{\\{}x\in[0,1]^{d}\,\,\colon\,\,2h(x)-1\neq\operatorname{sign}\bigl{(}2f_{S}(x)-1\bigr{)}\big{\\}}\big{)}$

of $f_{T,m}$ that

$\mathbb{E}_{S}\big{[}\boldsymbol{\lambda}\big{(}\big{\\{}x\in[0,1]^{d}\,\,\colon\,\,\operatorname{sign}\bigl{(}2f_{S}(x)-1\bigr{)}\neq
2h(x)-1\big{\\}}\big{)}\big{]}\\\
=\mathbb{E}_{T}[\mathcal{E}(f_{T,m},C^{\ast})]\lesssim\varepsilon_{m}^{2}\asymp\varepsilon_{m}^{\ast},$

where the implied constant is independent of $g$ (and hence of $h$) and only
depends on the constants appearing in conditions (A2)–(A5) and (N) verified
above. These constants in turn only depend on those appearing in Conditions
1–3 of Theorem 5.6. We note that it is never clearly stated in [24] what the
interpretation of $\operatorname{sign}(0)$ is. However, the proof of [24,
Theorem A.2] relies on [39, Theorem 2.31], and the convention in [39] is
$\operatorname{sign}(0)=1$; see immediately below [39, Equation (2.2)]. ∎

## References

  * [1] H. Amann and J. Escher.  Analysis. III.  Birkhäuser Verlag, Basel, 2009.
  * [2] F. Bach.  Breaking the curse of dimensionality with convex neutral networks.  J. Mach. Learn. Res., 18:Paper No. 19, 53, 2017.
  * [3] A. R. Barron.  Neural net approximation.  In Proc. 7th Yale Workshop on Adaptive and Learning Systems, volume 1, pages 69–72, 1992.
  * [4] A. R. Barron.  Universal approximation bounds for superpositions of a sigmoidal function.  IEEE Transactions on Information theory, 39(3):930–945, 1993.
  * [5] A. R Barron.  Approximation and estimation bounds for artificial neural networks.  Machine learning, 14(1):115–133, 1994.
  * [6] J. Berner, P. Grohs, and A. Jentzen.  Analysis of the generalization error: Empirical risk minimization over deep artificial neural networks overcomes the curse of dimensionality in the numerical approximation of Black–Scholes partial differential equations.  SIAM Journal on Mathematics of Data Science, 2(3):631–657, 2020\.
  * [7] J. Berner, P. Grohs, G. Kutyniok, and P. Petersen.  The modern mathematics of deep learning.  arXiv preprint arXiv:2105.04026, 2021.
  * [8] E. J. Candès.  Ridgelets: theory and applications.  PhD thesis, Department of Statistics, Stanford University, 1998.  https://statweb.stanford.edu/~candes/publications/downloads/Thesis.pdf.
  * [9] A. Caragea, P. Petersen, and F. Voigtlaender.  Neural network approximation and estimation of classifiers with classification boundary in a Barron class.  arXiv preprint arXiv:2011.09363, 2020.
  * [10] D. L. Cohn.  Measure theory.  Birkhäuser Advanced Texts: Basler Lehrbücher. [Birkhäuser Advanced Texts: Basel Textbooks]. Birkhäuser/Springer, New York, second edition, 2013.
  * [11] R. A. DeVore and V. N. Temlyakov.  Nonlinear approximation by trigonometric sums.  J. Fourier Anal. Appl., 2(1):29–48, 1995.
  * [12] L. Devroye, L. Györfi, and G. Lugosi.  A probabilistic theory of pattern recognition, volume 31.  Springer Science & Business Media, 2013.
  * [13] W. E, C. Ma, and L. Wu.  A priori estimates of the population risk for two-layer neural networks.  Commun. Math. Sci., 17(5):1407–1425, 2019.
  * [14] W. E, C. Ma, L. Wu, and S. Wojtowytsch.  Towards a Mathematical Understanding of Neural Network-Based Machine Learning: What We Know and What We Don’t.  CSIAM Transactions on Applied Mathematics, 1(4):561–615, 2020.
  * [15] W. E and S. Wojtowytsch.  On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamics.  arXiv preprint arXiv:2007.15623, 2020.
  * [16] W. E and S. Wojtowytsch.  Representation formulas and pointwise properties for Barron functions.  arXiv preprint arXiv:2006.05982, 2020.
  * [17] I. Goodfellow, Y. Bengio, and A. Courville.  Deep learning.  MIT press, 2016.
  * [18] L. Grafakos.  Classical Fourier analysis, volume 249 of Graduate Texts in Mathematics.  Springer, New York, third edition, 2014.
  * [19] P. Grohs.  Optimally sparse data representations.  In Harmonic and applied analysis, Appl. Numer. Harmon. Anal., pages 199–248. Birkhäuser/Springer, Cham, 2015.
  * [20] P. Grohs and F. Voigtlaender.  Proof of the theory-to-practice gap in deep learning via sampling complexity bounds for neural network approximation spaces.  arXiv preprint arXiv:2104.02746, 2021.
  * [21] M. Imaizumi and K. Fukumizu.  Deep neural networks learn non-smooth functions effectively.  In The 22nd International Conference on Artificial Intelligence and Statistics, pages 869–878. PMLR, 2019.
  * [22] M. Imaizumi and K. Fukumizu.  Advantage of deep neural networks for estimating functions with singularity on curves.  arXiv preprint arXiv:2011.02256, 2020.
  * [23] O. Kallenberg.  Foundations of modern probability, volume 99 of Probability Theory and Stochastic Modelling.  Springer, Cham, 2021.
  * [24] Y. Kim, I. Ohn, and D. Kim.  Fast convergence rates of deep neural networks for classification.  Neural Networks, 138:179–197, 2021.
  * [25] A. Klenke.  Probability theory.  Universitext. Springer, London, second edition, 2014.  A comprehensive course.
  * [26] J. M. Klusowski and A. R. Barron.  Risk bounds for high-dimensional ridge function combinations including neural networks.  arXiv preprint arXiv:1607.01434, 2016.
  * [27] J. M. Klusowski and A. R. Barron.  Approximation by combinations of ReLU and squared ReLU ridge functions with $\ell^{1}$ and $\ell^{0}$ controls.  IEEE Trans. Inform. Theory, 64(12):7649–7656, 2018.
  * [28] D. Krieg and M. Sonnleitner.  Random points are optimal for the approximation of Sobolev functions.  arXiv preprint arXiv:2009.11275, 2020.
  * [29] Y. Makovoz.  Uniform approximation by neural networks.  Journal of Approximation Theory, 95(2):215–228, 1998.
  * [30] E. Mammen and A. B. Tsybakov.  Smooth discrimination analysis.  The Annals of Statistics, 27(6):1808–1829, 1999.
  * [31] M. Mohri, A. Rostamizadeh, and A. Talwalkar.  Foundations of machine learning.  MIT press, 2018.
  * [32] C. Muscalu and W. Schlag.  Classical and multilinear harmonic analysis. Vol. I, volume 137 of Cambridge Studies in Advanced Mathematics.  Cambridge University Press, Cambridge, 2013.
  * [33] P. Petersen and F. Voigtlaender.  Optimal approximation of piecewise smooth functions using deep ReLU neural networks.  Neural Networks, 108:296–330, 2018.
  * [34] W. Rudin.  Real and complex analysis.  McGraw-Hill Book Co., New York, third edition, 1987.
  * [35] J. Schmidhuber.  Deep learning in neural networks: An overview.  Neural networks, 61:85–117, 2015.
  * [36] J. Schmidt-Hieber.  Nonparametric regression using deep neural networks with ReLU activation function.  The Annals of Statistics, 48(4):1875–1897, 2020.
  * [37] S. Shalev-Shwartz and S. Ben-David.  Understanding machine learning: From theory to algorithms.  Cambridge university press, 2014.
  * [38] J. W Siegel and J. Xu.  Sharp bounds on the approximation rates, metric entropy, and $n$-widths of shallow neural networks.  arXiv preprint arXiv:2101.12365, 2021.
  * [39] I. Steinwart and A. Christmann.  Support Vector Machines.  Information Science and Statistics. Springer, New York, 2008.
  * [40] C. J. Stone.  Optimal global rates of convergence for nonparametric regression.  Ann. Statist., 10(4):1040–1053, 1982.
  * [41] A. B. Tsybakov.  Optimal aggregation of classifiers in statistical learning.  The Annals of Statistics, 32(1):135–166, 2004.
  * [42] S. A. Van de Geer.  Applications of empirical process theory, volume 91.  Cambridge University Press Cambridge, 2000.
  * [43] R. Vershynin.  High-dimensional probability, volume 47 of Cambridge Series in Statistical and Probabilistic Mathematics.  Cambridge University Press, Cambridge, 2018.
  * [44] Y. Yang and A. Barron.  Information-theoretic determination of minimax rates of convergence.  Ann. Statist., 27(5):1564–1599, 1999.


# Scoreformer: A Surrogate Model For Large-Scale Prediction of Docking Scores

Álvaro Ciudad  Adrián Morales-Pastor  Laura Malo  Isaac Filella-Mercè  Victor
Guallar  Alexis Molina

###### Abstract

In this study, we present ScoreFormer, a novel graph transformer model
designed to accurately predict molecular docking scores, thereby optimizing
high-throughput virtual screening (HTVS) in drug discovery. The architecture
integrates Principal Neighborhood Aggregation (PNA) and Learnable Random Walk
Positional Encodings (LRWPE), enhancing the model’s ability to understand
complex molecular structures and their relationship with their respective
docking scores. This approach significantly surpasses traditional HTVS methods
and recent Graph Neural Network (GNN) models in both recovery and efficiency
due to a wider coverage of the chemical space and enhanced performance. Our
results demonstrate that ScoreFormer achieves competitive performance in
docking score prediction and offers a substantial 1.65-fold reduction in
inference time compared to existing models. We evaluated ScoreFormer across
multiple datasets under various conditions, confirming its robustness and
reliability in identifying potential drug candidates rapidly.

Machine Learning, ICML

##  1 Introduction

Virtual screening (VS) is a key computational technique in drug discovery,
utilized for evaluating the potential binding affinity of numerous molecules
with target proteins. The main aim of VS is to identify molecules with
potential interaction capabilities, thereby streamlining the drug discovery
process and reducing the need for extensive, costly experimental assays
(Lavecchia & Di Giovanni, 2013). HTVS represents an advanced and more
efficient form of VS. Characterized by its ability to rapidly assess vast
molecular libraries, HTVS significantly contributes to identifying potential
biological interactions.

The evolution of HTVS has been significantly influenced by the advancement of
combinatorial chemistry, enabling the generation of extensive and diverse
compound libraries. Previously, drug discovery relied on smaller and less
diverse libraries, limiting the scope of potential candidates. The integration
of combinatorial chemistry with HTVS allows for the utilization of larger
compound libraries, alongside advanced computational evaluation techniques.
This integration has accelerated the process of identifying suitable drug
candidates by enabling rapid and precise assessment of ligand affinity across
these vast libraries. Such efficiency is crucial in modern drug discovery as
it can considerably shorten the hit-finding phase and enhance the overall hit
rate. As a result, there’s an increasing need for methodologies that can
swiftly and reliably navigate this expanded chemical space (Sadybekov &
Katritch, 2023).

In this context, we present ScoreFormer, a graph transformer model developed
to accurately predict molecular docking scores, thereby optimizing the
efficiency of HTVS. ScoreFormer is designed to interpret molecular structures
and their relationship with docking scores. Our rigorous evaluation
demonstrates that ScoreFormer not only achieves state-of-the-art accuracy in
hit identification and docking score prediction but also offers a significant
reduction in inference time compared to existing methods. Moreover, we also
present L-ScoreFormer, a smaller version of the original model, focused on the
general prediction of docking scores with improved efficiency. Both approaches
make efficient and reliable tools for HTVS, capable of handling diverse
datasets and conditions. The datasets, including scores, SMILES, and poses,
used in our testing across multiple systems, with and without docking
constraints, will be made available at Zenodo.

##  2 Related work

HTVS applied to relatively large libraries has been a key practice in drug
discovery for some time. Within HTVS, traditional methodologies, such as Glide
(Friesner et al., 2004) and rDock (Ruiz-Carmona et al., 2014), employ
specialized scoring functions and search-based protocols in their algorithms.
These are designed to prioritize speed, enabling the assessment of larger
compound collections than typically seen in standard docking campaigns.
However, with the ever-increasing size of modern compound libraries, these
traditional docking approaches are becoming less suitable for the task at
hand.

Advancements in HTVS have been greatly influenced by the introduction of
surrogate models. In a study conducted by Agrawal et al. (2019), a progressive
docking approach incorporating molecular features and Morgan fingerprints
achieved a 50-fold speed increase and a 90-fold enrichment compared to whole
library docking and random sampling, respectively. Graff et al. (2021) and
Graff et al. (2022) utilized Bayesian optimization and a design space pruning
framework on the MolPAL platform, attaining a notable top-10k recovery rate of
67.7% while significantly reducing computational demands. Moreover, Yang et
al. (2021)’s active learning framework, which integrates a Graph Convolutional
Network(GCN)-based surrogate model, successfully identified 80% of
experimentally confirmed hits, reducing computational costs by a factor of 14.
These metrics emphasize the substantial enhancements in efficiency and
accuracy these methods bring to HTVS.

GNNs have become increasingly effective in molecular learning due to their
ability to model the complex, non-Euclidean structure of molecular data.
Molecules, represented as graphs with atoms as nodes and bonds as edges, align
well with GNN architectures, which effectively capture node interactions. The
work by Hosseini et al. (2022) showcases the use of GNNs in enhancing docking
score prediction, integrating machine learning-based surrogate docking with
traditional methods to improve screening efficiency. Their introduction of
FiLMv2, a novel GNN architecture, has shown remarkable performance
improvements, including a 9.496-fold increase in molecule screening speed with
respect to DOCK (Ewing et al., 2001) and a recall error rate below 3%,
achieving significant speed increases and low recall error rates in molecule
screening, thereby advancing chemical docking tasks.

However, the use of virtual nodes in GNNs, including FiLMv2, to capture global
information can lead to increased computational complexity and prolonged
training and inference times, presenting a challenge in HTVS. Additionally,
virtual nodes risk overshadowing important local interactions and molecular
topologies crucial for precise molecular property predictions due to the
absence of learnable weights. To mitigate these issues, we propose the
adoption of an attention mechanism, as an alternative to virtual nodes. This
approach efficiently processes global information and reduces computational
demands, while preserving the ability to capture essential local variations in
molecular structures, as shown in the next section.

##  3 Methods

###  3.1 ScoreFormer Architecture

The ScoreFormer architecture is grounded in the graph transformer framework as
introduced in Rampášek et al. (2022). This choice is motivated by the
significance of long-range interactions in molecular regression tasks, a
finding supported by ablation studies in Hosseini et al. (2022).
Traditionally, a virtual node strategy from Gilmer et al. (2017) facilitates
long-range interaction by introducing an extra node connected to every other
node. However, our analysis (see Section 4.3) indicates that this approach
substantially increases inference time. To mitigate this, we integrate an
attention mechanism capable of modeling similar long-range interactions but
with learnable parameters, resulting in enhanced performance.

To enhance the aggregation process from a node’s neighborhood, in the message
passing component of the graph transformer layers, we employ Principal
Neighborhood Aggregation (PNA) as proposed by Corso et al. (2020). The PNA
mechanism is defined as:

$\mathbf{X}_{i}^{(t+1)}=\text{U}^{(t)}\left(\mathbf{X}_{i}^{(t)},\underset{(j,i)\in
E}{\bigoplus}\left(\text{M}^{(t)}\left(\mathbf{X}_{i}^{(t)},E_{j\rightarrow
i},\mathbf{X}_{j}^{(t)}\right)\right)\right)$

(1)

where $\mathbf{X}_{i}^{(t)}$ is the node feature matrix at layer $t$,
$E_{j\rightarrow i}$ the edge feature of (j,i) if present, $\bigoplus$ are the
set of aggregator and scalers defined in the original paper and $M^{t}$ and
$U^{t}$ are neural networks at layer $t$.

Furthermore, to better capture positional information within the graph
structure, we incorporate Learnable Random Walk Positional Encodings (LRWPE)
from Dwivedi et al. (2022). These encodings are integrated at each layer of
the PNA as follows:

$\mathbf{x}_{i}^{(t+1)}=\text{PNA}\left(\mathbf{x}_{i}^{(t)}\oplus\mathbf{p}_{i}^{(t)},E_{\mathcal{N}(i)\rightarrow
i},\mathcal{N}(i)\right)$ (2)

$\mathbf{p}_{i}^{(t+1)}=\text{PNA}\left(\mathbf{p}_{i}^{(t)},\mathcal{N}(i)\right)$
(3)

where $\mathbf{p}_{i}^{(t)}$ is the learnable positional encoding for node $v$
at layer $t$, $\oplus$ denotes concatenation as the operation for combining
node features and positional encodings and $\mathcal{N}(i)$ is defined as the
neighbouring nodes of $i$. The positional encodings are learned layer-wise
with another PNA component.

The LRWPE are optimized simultaneously with the GNN parameters through a task-
specific loss function, ensuring that both the feature and positional
information are appropriately leveraged to improve the model’s performance on
molecular regression tasks. This dual optimization is a key feature of our
architecture, distinguishing ScoreFormer from traditional graph transformer
models by providing a more nuanced and positionally-aware representation of
graph-structured data. A schematic representation of the ScoreFormer
architecture is shown in Figure 1.

Figure 1: Schematic representation of the architecture used in the ScoreFormer
model.

For improved efficiency, we also developed L-ScoreFormer, a streamlined
variant of the ScoreFormer architecture, designed for efficiency and reduced
overfitting risk. Developed with an emphasis on minimizing computational costs
while maintaining performance, this model features a parameter-reduced
structure. Its optimization, achieved through multiple objective trials using
Optuna (Akiba et al., 2019), focuses on balancing parameter reduction with
model efficacy. This approach ensures that L-ScoreFormer retains the core
capabilities of the original ScoreFormer, yet operates more efficiently and
with a lower risk of overfitting to specific data distributions.

###  3.2 Datasets

Our evaluation of ScoreFormer and FiLMv2 involved seven datasets, including
one from the ZINC database with 128 million molecules docked to the dopamine
D4 receptor (Lyu et al., 2019). We further utilized six datasets from the ZINC
20 database (Irwin et al., 2020), choosing 500k molecules with molecular
weights between 200 and 500 Dalton. These molecules were docked against three
protein systems, Dopamine D4 receptor, HIV-1 protease, and Cyclin-dependent
kinase 2 (CDK2), using the Glide software. To assess model performance in
different scenarios, we applied both constrained and unconstrained docking
protocols, offering a comparative analysis of more realistic and idealized
docking conditions. Further details are provided in Appendix A.

###  3.3 Model evaluation

In the following experiments, model performance was assessed by training the
model using an 80-10-10 data split protocol. Inference epochs were selected
based on F1 performance in a validation set composed of 10% of the dataset.
Final performance metrics are obtained on a holdout test set using the
remaining 10% of the dataset. This procedure was repeated for each of the
protein systems and each docking protocol.

Performance was measured using the metrics employed in Hosseini et al. (2022),
along with additional metrics for a more comprehensive comparison. Further
details are in Appendix C.

##  4 Experiments

In evaluating both ScoreFormer architectures, our approach was three-folded,
focusing on its generalization capabilities for a range of docking scores,
efficacy in identifying promising drug candidates, and computational speed
relative to leading models in the field. We utilized standard regression
metrics to assess its predictive accuracy and specialized metrics for its
precision in identifying potential high-affinity compounds. Additionally, we
compared its processing speed against the currently available top-performing
architecture. We also evaluated the generalization capabilities of ScoreFormer
by performing inference on out-of-distribution molecules. Finally, we further
conducted ablation studies to assess the contributions of LRWPE and PNA
convolution layers to the performance of ScoreFormer, see Appendix G.

###  4.1 Prediction of docking scores

In implementing a graph transformer model like ScoreFormer, one of our primary
objectives was to achieve superior generalization in predicting docking
scores. This capability can be relevant in drug discovery campaigns, where
there is a significant interest in compounds across different ranges of
docking scores. Such comprehensive predictive ability is beneficial not only
for HTVS tasks but also for parallel applications like Quantitative Structure-
Activity Relationships (QSAR) (Hansch & Fujita, 1964).

To this end, we evaluated ScoreFormer’s performance using standard regression
metrics. According to the results in Appendix D, L-ScoreFormer demonstrates
enhanced predictive power across a broad spectrum of docking scores,
surpassing FiLMv2 implementation in both $R^{2}$ and Pearson. We hypothesize
that this improvement is attributable to a reduction in overfitting, a common
challenge in models trained with a focus on specific topologies, particularly
under the WMSE loss used during training. This diminished tendency for
overfitting in L-ScoreFormer suggests that a leaner model structure can be
more effective in generalizing across different molecular structures and
docking scenarios.

Table 1: Metrics for FiLMv2, ScoreFormer, and L-ScoreFormer for the seven
datasets chosen. Best performance metrics are highlighted in bold.

|  |  |  | WMSE | RES | $AURTC_{0.01}$ | $AURTC_{0.001}$ | $R_{0.1,0.01}$ | $R_{0.1,0.001}$
---|---|---|---|---|---|---|---|---|---
target | engine | settings | model |  |  |  |  |  |
d4 | dock | unconstrained | FiLMv2 | 0.399 | 0.754 | 0.644 | 0.721 | 0.886 | 1.000
ScoreFormer | 0.388 | 0.755 | 0.646 | 0.715 | 0.886 | 0.976
L-ScoreFormer | 0.398 | 0.763 | 0.647 | 0.691 | 0.874 | 0.952
glide | unconstrained | FiLMv2 | 0.383 | 0.813 | 0.706 | 0.802 | 0.901 | 1.000
ScoreFormer | 0.358 | 0.820 | 0.721 | 0.789 | 0.909 | 1.000
L-ScoreFormer | 0.389 | 0.817 | 0.711 | 0.764 | 0.868 | 1.000
constrained | FiLMv2 | 0.349 | 0.817 | 0.760 | 0.749 | 0.912 | 0.951
ScoreFormer | 0.321 | 0.845 | 0.795 | 0.803 | 0.924 | 0.951
L-ScoreFormer | 0.344 | 0.826 | 0.763 | 0.750 | 0.905 | 0.951
cdk2 | glide | unconstrained | FiLMv2 | 0.602 | 0.707 | 0.613 | 0.655 | 0.760 | 0.875
ScoreFormer | 0.603 | 0.698 | 0.603 | 0.624 | 0.762 | 0.917
L-ScoreFormer | 0.629 | 0.707 | 0.599 | 0.622 | 0.738 | 0.875
constrained | FiLMv2 | 0.510 | 0.737 | 0.632 | 0.702 | 0.801 | 0.935
ScoreFormer | 0.519 | 0.746 | 0.653 | 0.735 | 0.826 | 0.968
L-ScoreFormer | 0.508 | 0.758 | 0.675 | 0.744 | 0.804 | 1.000
hiv | glide | unconstrained | FiLMv2 | 0.629 | 0.674 | 0.537 | 0.643 | 0.690 | 0.918
ScoreFormer | 0.640 | 0.691 | 0.562 | 0.695 | 0.712 | 0.980
L-ScoreFormer | 0.614 | 0.696 | 0.559 | 0.682 | 0.675 | 0.939
constrained | FiLMv2 | 0.450 | 0.711 | 0.588 | 0.657 | 0.749 | 0.875
ScoreFormer | 0.439 | 0.725 | 0.599 | 0.678 | 0.771 | 0.875
L-ScoreFormer | 0.455 | 0.711 | 0.580 | 0.644 | 0.719 | 0.800
all | all | all | FiLMv2 | 0.475 | 0.745 | 0.640 | 0.704 | 0.814 | 0.936
ScoreFormer | 0.467 | 0.754 | 0.654 | 0.720 | 0.827 | 0.952
L-ScoreFormer | 0.477 | 0.754 | 0.648 | 0.700 | 0.798 | 0.931

###  4.2 Molecular hit recovery

ScoreFormer’s ability to recover molecular hits, crucial for its application
in prospective HTVS campaigns, was thoroughly evaluated using the metrics
stated in Appendix C. The analysis, as illustrated in Table 1, demonstrated
ScoreFormer’s superior performance in accurately identifying and prioritizing
compounds with high binding affinity across various datasets and conditions.
This proficiency is especially evident when comparing its performance to
FiLMv2, a benchmark model in the field. ScoreFormer consistently outperformed
FiLMv2 across multiple recovery metrics. Due to its increase in the number of
parameters, the model is able to capture the topological features responsible
for good docking scores along the different systems and constrain set. These
results highlight ScoreFormer’s robustness and reliability in hit recovery.
Notably, L-ScoreFormer, also displayed commendable performance, often closely
following the FiLMv2 architecture. This observation underscores the
effectiveness of both of our architectures in molecular hit recovery tasks.
The consistent results of both ScoreFormer and L-ScoreFormer across diverse
screening scenarios reinforce the versatility of our approach. See further
results at Appendix E.

###  4.3 Inference speed

The inference speed of ScoreFormer, a critical factor in HTVS, was evaluated
using a dataset of 500,000 molecules from the DOCK database. Conducted on a
single A30 GPU, this assessment was tailored to reflect the real-world demands
of computational drug discovery. To focus purely on the model’s core
computational capabilities, we excluded pre-processing steps from the
evaluation, as these are commonly cached in practical applications, for both
ScoreFormer and the FiLMv2 model.

The results, as shown in Table 2, highlight a significant enhancement in
L-ScoreFormer’s processing efficiency compared to FiLMv2. Achieving a
1.86-fold increase in speed, L-ScoreFormer processed samples at a rate of
2468.4 per second, effectively halving the estimated processing time for 128
million samples (DOCK dataset) to 14.40 hours from FiLMv2’s 26.85 hours.
ScoreFormer, also outperformed previous benchmarks by a factor of 1.652. This
marked speed improvement is not achieved through a reduction of trainable
parameters but by the replacement of the virtual node in FiLMv2 by an
attention mechanism. This highlights ScoreFormer’s architectural efficiency
while demonstrating the model’s capacity to handle large-scale datasets
rapidly.

Table 2: Speed performance of the three models tested in this study. Speed is
indicated as samples predicted per second and time needed for predicting 128
million docked compounds. The number of learnable parameters is also added for
comparison.
Model | Samples/s | 128M time (h)
---|---|---
FiLMv2 | 1323.942 | 26.850
ScoreFormer | 2186.828 | 16.259
L-ScoreFormer | 2468.404 | 14.404
Model | Speedup | # parameters
FiLMv2 | 1.000 | 102977
ScoreFormer | 1.652 | 5398273
L-ScoreFormer | 1.864 | 147457

###  4.4 Generalization across the chemical space

To evaluate the generalization capabilities of ScoreFormer, we conducted two
benchmarks with molecules outside the training distribution. These benchmarks
are necessary due to the combinatorial nature of the training databases, which
may lead to an unfair assessment of the generalization capabilities of this
kind of methods.

The first benchmark uses molecules generated by generative models for the
evaluation set. As shown in Filella-Merce et al. (2023), generative models can
escape densely populated patent spaces, such as the one of CDK2. This results
in a set of molecules with novel chemistry, which can be used to evaluate
chemical motif generalization. The evaluation set included molecules with a
maximum tanimoto similarity of 0.3 to the training dataset of the generative
model. Docking scores were calculated using the same constrained protocol
reported in this study and we evaluated all three previous models, comparing
them with metrics obtained by ScoreFormer on non-generated molecules. All
metrics showed a decline, but ScoreFormer and L-ScoreFormer performed better
or equal to FiLMv2, demonstrating better generalization capabilities for out-
of-distribution molecules. 111Due to low overlap of the new docking scores
with the docking scores of the original training distribution, we do not
report the regression metrics values, as all the models under-perform in this
task. However, we analyzed molecular hit recovery capability, as this can
still be inferred from available results. Results are summarized in Table 3
and full results can be found in Appendix H.

The second benchmark focuses on creating custom training and testing splits
based upon molecular weight, to assess molecular size generalization, and its
results can also be found at Appendix H.

Table 3: Metrics obtained on generated molecules. Reference configuration
corresponds to constrained ScoreFormer results on CDK2 on Table 1. Best
values, excluding the reference, are highlighted in bold.
Configuration | RES | $AURTC_{0.01}$ | $AURTC_{0.001}$
---|---|---|---
ScoreFormer | 0.458 | 0.344 | 0.359
L-ScoreFormer | 0.449 | 0.358 | 0.336
FiLMv2 | 0.431 | 0.333 | 0.314
Reference | 0.746 | 0.653 | 0.735

##  5 Conclusions

In this study, we introduce ScoreFormer and its variant L-ScoreFormer, graph
transformer models designed to overcome limitations in current approaches for
surrogate models in HTVS. ScoreFormer’s architecture, incorporating an
attention mechanism, effectively manages global and local molecular
information, leading to superior performance in identifying high-affinity
compounds. L-ScoreFormer, with fewer parameters, shows strong performance in
general docking score prediction, reducing overfitting risks. A key feature is
the improved inference speed, with a 1.86-fold increase, vital for high-
throughput virtual screening. Future work will focus on applying ScoreFormer
in active learning contexts, integrating explicability modules, and adopting
uncertainty estimation methods, aiming to further enhance its utility in drug
discovery and contribute to the advancement of computational chemistry with
more reliable, efficient, and interpretable methods.

## References

  * Agrawal et al. (2019) Agrawal, V., Gentile, F., Hsing, M., Ban, F., and Cherkasov, A.  Progressive docking-deep learning based approach for accelerated virtual screening.  In _International Conference on Artificial Neural Networks_ , pp. 737–740. Springer, 2019.
  * Akiba et al. (2019) Akiba, T., Sano, S., Yanase, T., Ohta, T., and Koyama, M.  Optuna: A next-generation hyperparameter optimization framework.  In _The 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pp. 2623–2631, 2019.
  * Bettayeb et al. (2007) Bettayeb, K., Tirado, O. M., Marionneau-Lambot, S., Ferandin, Y., Lozach, O., Morris, J. C., Mateo-Lozano, S., Drueckes, P., Schächtele, C., Kubbutat, M. H. G., Liger, F., Marquet, B., Joseph, B., Echalier, A., Endicott, J. A., Notario, V., and Meijer, L.  Meriolins, a new class of cell death inducing kinase inhibitors with enhanced selectivity for cyclin-dependent kinases.  _Cancer Res_ , 67(17):8325–8334, September 2007.
  * Corso et al. (2020) Corso, G., Cavalleri, L., Beaini, D., Liò, P., and Veličković, P.  Principal neighbourhood aggregation for graph nets.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Dwivedi et al. (2022) Dwivedi, V. P., Luu, A. T., Laurent, T., Bengio, Y., and Bresson, X.  Graph neural networks with learnable structural and positional representations.  In _International Conference on Learning Representations_ , 2022.
  * Ewing et al. (2001) Ewing, T. J., Makino, S., Skillman, A. G., and Kuntz, I. D.  Dock 4.0: search strategies for automated molecular docking of flexible molecule databases.  _Journal of computer-aided molecular design_ , 15:411–428, 2001.
  * Filella-Merce et al. (2023) Filella-Merce, I., Molina, A., Orzechowski, M., Díaz, L., Zhu, Y. M., Mor, J. V., Malo, L., Yekkirala, A. S., Ray, S., and Guallar, V.  Optimizing drug design by merging generative ai with active learning frameworks.  _arXiv preprint arXiv:2305.06334_ , 2023.
  * Friesner et al. (2004) Friesner, R. A., Banks, J. L., Murphy, R. B., Halgren, T. A., Klicic, J. J., Mainz, D. T., Repasky, M. P., Knoll, E. H., Shelley, M., Perry, J. K., Shaw, D. E., Francis, P., and Shenkin, P. S.  Glide: A new approach for rapid, accurate docking and scoring. 1. method and assessment of docking accuracy.  _Journal of Medicinal Chemistry_ , 47(7):1739–1749, 2004.
  * Gilmer et al. (2017) Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E.  Neural message passing for quantum chemistry.  In _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pp. 1263–1272. PMLR, 06–11 Aug 2017.
  * Graff et al. (2021) Graff, D. E., Shakhnovich, E. I., and Coley, C. W.  Accelerating high-throughput virtual screening through molecular pool-based active learning.  _Chemical science_ , 12(22):7866–7881, 2021.
  * Graff et al. (2022) Graff, D. E., Aldeghi, M., Morrone, J. A., Jordan, K. E., Pyzer-Knapp, E. O., and Coley, C. W.  Self-focusing virtual screening with active design space pruning.  _Journal of Chemical Information and Modeling_ , 62(16):3854–3862, 2022.
  * Hansch & Fujita (1964) Hansch, C. and Fujita, T.  p-$\sigma$-$\pi$ analysis. a method for the correlation of biological activity and chemical structure.  _Journal of the American Chemical Society_ , 86(8):1616–1626, 1964.
  * Hosseini et al. (2022) Hosseini, R., Simini, F., Clyde, A., and Ramanathan, A.  Deep surrogate docking: Accelerating automated drug discovery with graph neural networks.  _arXiv preprint arXiv:2211.02720_ , 2022.
  * Irwin et al. (2020) Irwin, J. J., Tang, K. G., Young, J., Dandarchuluun, C., Wong, B. R., Khurelbaatar, M., Moroz, Y. S., Mayfield, J., and Sayle, R. A.  Zinc20—a free ultralarge-scale chemical database for ligand discovery.  _Journal of chemical information and modeling_ , 60(12):6065–6073, 2020.
  * Lavecchia & Di Giovanni (2013) Lavecchia, A. and Di Giovanni, C.  Virtual screening strategies in drug discovery: a critical review.  _Curr Med Chem_ , 20(23):2839–2860, 2013.
  * Li et al. (2015) Li, Y., Zhang, J., Gao, W., Zhang, L., Pan, Y., Zhang, S., and Wang, Y.  Insights on structural characteristics and ligand binding mechanisms of cdk2.  _International Journal of Molecular Sciences_ , 16(5):9314–9340, 2015.
  * Lu et al. (2021) Lu, C., Wu, C., Ghoreishi, D., Chen, W., Wang, L., Damm, W., Ross, G. A., Dahlgren, M. K., Russell, E., Von Bargen, C. D., Abel, R., Friesner, R. A., and Harder, E. D.  OPLS4: Improving force field accuracy on challenging regimes of chemical space.  _J. Chem. Theory Comput._ , 17(7):4291–4300, July 2021.
  * Lyu et al. (2019) Lyu, J., Wang, S., Balius, T. E., Singh, I., Levit, A., Moroz, Y. S., O’Meara, M. J., Che, T., Algaa, E., Tolmachova, K., et al.  Ultra-large library docking for discovering new chemotypes.  _Nature_ , 566(7743):224–229, 2019.
  * Madhavi Sastry et al. (2013) Madhavi Sastry, G., Adzhigirey, M., Day, T., Annabhimoju, R., and Sherman, W.  Protein and ligand preparation: parameters, protocols, and influence on virtual screening enrichments.  _Journal of Computer-Aided Molecular Design_ , 27(3):221–234, Mar 2013.
  * Rampášek et al. (2022) Rampášek, L., Galkin, M., Dwivedi, V. P., Luu, A. T., Wolf, G., and Beaini, D.  Recipe for a General, Powerful, Scalable Graph Transformer.  _Advances in Neural Information Processing Systems_ , 35, 2022.
  * Ruiz-Carmona et al. (2014) Ruiz-Carmona, S., Alvarez-Garcia, D., Foloppe, N., Garmendia-Doval, A. B., Juhos, S., Schmidtke, P., Barril, X., Hubbard, R. E., and Morley, S. D.  rdock: a fast, versatile and open source program for docking ligands to proteins and nucleic acids.  _PLoS Comput. Biol._ , 10(4):e1003571, April 2014.
  * Rutenber et al. (1996) Rutenber, E. E., McPhee, F., Kaplan, A. P., Gallion, S. L., Hogan, J. C., Craik, C. S., and Stroud, R. M.  A new class of hiv-1 protease inhibitor: The crystallographic structure, inhibition and chemical synthesis of an aminimide peptide isostere.  _Bioorganic & Medicinal Chemistry_, 4(9):1545–1558, 1996.
  * Sadybekov & Katritch (2023) Sadybekov, A. V. and Katritch, V.  Computational approaches streamlining drug discovery.  _Nature_ , 616(7958):673–685, Apr 2023.
  * Søndergaard et al. (2011) Søndergaard, C. R., Olsson, M. H. M., Rostkowski, M., and Jensen, J. H.  Improved treatment of ligands and coupling effects in empirical calculation and rationalization of pka values.  _Journal of Chemical Theory and Computation_ , 7(7):2284–2295, 2011.
  * Wang et al. (2017) Wang, S., Wacker, D., Levit, A., Che, T., Betz, R. M., McCorvy, J. D., Venkatakrishnan, A. J., Huang, X.-P., Dror, R. O., Shoichet, B. K., and Roth, B. L.  D(4) dopamine receptor high-resolution structures enable the discovery of selective agonists.  _Science_ , 358(6361):381–386, October 2017.
  * Yang et al. (2021) Yang, Y., Yao, K., Repasky, M. P., Leswing, K., Abel, R., Shoichet, B. K., and Jerome, S. V.  Efficient exploration of chemical space with docking and deep learning.  _Journal of Chemical Theory and Computation_ , 17(11):7106–7119, 2021.

##  Appendix A Molecular docking in CDK2, D4 and HIV-1.

###  A.1 System preparation

All systems were prepared with the same protocol and the structures were the
following: CDK2 with PDB id 3BHV (Bettayeb et al., 2007), D4 with PDB id 5WIU
(Wang et al., 2017) and HIV-1 with PDB id 3AID (Rutenber et al., 1996).

Protonation states were assigned at pH 7.4 using the Protein Preparation
Wizard (Madhavi Sastry et al., 2013). Hydrogen bonds were optimized with
PROPKA (Søndergaard et al., 2011) and a restrained minimization using the atom
force field OPLS4 (Lu et al., 2021) was applied to each system.

###  A.2 Glide docking

Ligand dockings were performed using Schrödinger’s Glide software, we used the
Standard Precision (SP) protocol with a fast sampling of 500 poses kept per
ligand for the initial phase of docking, 40 best poses kept per ligand for
energy minimization and only one pose selected for each compound after
minimization. Docking grids were centered into the centroid of the
corresponding native ligand with an inner box of 15 Å for D4 and HIV-1 and 10Å
for CDK2. Hydrogen-bond constraints were imposed to improve the applicability
of the poses generated. Constraints selected per system:

  1. 1.

CDK2: hydrogen bond constraint to the central donor of the CDK2 hinge region
(LEU83) (Li et al., 2015).

  2. 2.

D4: hydrogen bond constraint to bias the selection of the compounds to
selective agonists (ASP115) (Wang et al., 2017).

  3. 3.

HIV-1: hydrogen bond constraint to the highly conserved aspartate in an
aspartic protease (ASP25) active site to mimic ligand interactions of known
inhibitors (Rutenber et al., 1996).

Docking was also performed on the unconstrained systems.

##  Appendix B Hyperparameters

Table 4: Hyperparameters for Scoreformer model.
group | parameter | value
---|---|---
Training parameters | Batch size | 512
Learning rate | 0.001
Optimizer | Adam
GPS parameters | Activation | ReLU
Activation PE | Tanh
Dropout | 0.100
Dropout Attention | 0.500
Dropout PE | 0.100
Num conv layers | 4
Residual weight | 1
LPSE PNA parameters | Aggregators | mean, min, max, std
Hidden dim | 128
MLP activation | NA
MLP activation PE | NA
Pre Aggregation FC Layers | 1
Post Aggregation FC Layers | 1
Scalers | identity, amplification, attenuation
Towers | 4
RW length | 9
Table 5: Hyperparameters for L-Scoreformer model.
group | parameter | value
---|---|---
Training parameters | Batch size | 64
Learning rate | 1.4126e-05
Optimizer | Adam
GPS parameters | Activation | ELU
Activation PE | ReLU
Dropout | 0.245
Dropout Attention | 0.432
Dropout PE | 0.188
Num conv layers | 4
Residual weight | 0.035
LPSE PNA parameters | Aggregators | mul, add, sum, mean, moment4, moment5
Hidden dim | 24
MLP activation | ELU
MLP activation PE | ReLU
Pre Aggregation FC Layers | 3
Post Aggregation FC Layers | 2
Scalers | linear, inverse_linear
Towers | 1
RW length | 2

##  Appendix C Evaluation metrics

###  C.1 Exponentially weighted mean squared error (W-MSE)

This metric was used as the loss function during training. It modifies a
preexisting loss function, in this case, the mean squared error in Equation 5,
to increase the weight of samples with low target values in Equation 4. In the
context of molecular docking, we are specifically interested in accurately
predicting the docking score of good binders, which are given lower values.
This metric allows the training process to pay less attention to poor binders
and hence becomes a more valuable tool for identifying good binders

$\sum_{i=0}^{N}e^{-\alpha y_{i}}\cdot l(z_{i},y_{i})$ (4) $l(z,y)=(z-y)^{2}$
(5)

###  C.2 Regression enrichment surface score

This score corresponds to the volume under the surface defined as the recall
of an estimator using two thresholds ($\zeta$ , $\sigma$) for binarizing the
target variable and the model outputs. The thresholds are expressed as the
fraction of the samples to be considered as positive instances and hence the
surface is bounded from 0 to 1 in both axes. The volume is computed with
logarithmically scaled axes which results in the higher impact of samples with
low target and predicted values in the overall metric.

###  C.3 Area under recall threshold curve ($AURTC_{\zeta}$)

The recall threshold curve is defined as a slice of the RES by fixing the
threshold $\zeta$ (the ratio of the model predictions to be considered
positive). As in Hosseini et al. (2022), we report AURTC using two $\zeta$
values for all experiments: 0.01 and 0.001.

###  C.4 Parametrized recall ($R_{\zeta,\sigma}$)

The parametrized recall is the fraction of retrieved positive examples by the
model when using two thresholds i.e. $\zeta$ and $\sigma$, for dividing the
true labels and the predicted labels into positive and negative examples. It
corresponds to the value of the RES at a given point. As in the original
publication, we report R using two pairs of thresholds for all experiments:
0.1 and 0.01; 0.1 and 0.001.

###  C.5 Other metrics

Beyond the metrics listed, model performance was also evaluated using the
$R^{2}$ and Pearson’s correlation coefficient.

##  Appendix D Regression results

Table 6: Correlation metrics for the different models and targets. Best
performing models are highlighted in bold.
|  |  |  | Pearson | $R^{2}$
---|---|---|---|---|---
target | engine | settings | model |  |
d4 | dock | unconstrained | FiLMv2 | 0.725 | 0.466
ScoreFormer | 0.736 | 0.472
L-ScoreFormer | 0.750 | 0.522
glide | unconstrained | FiLMv2 | 0.751 | 0.520
ScoreFormer | 0.777 | 0.558
L-ScoreFormer | 0.765 | 0.559
constrained | FiLMv2 | 0.714 | 0.451
ScoreFormer | 0.748 | 0.517
L-ScoreFormer | 0.729 | 0.479
cdk2 | glide | unconstrained | FiLMv2 | 0.596 | 0.247
ScoreFormer | 0.585 | 0.228
L-ScoreFormer | 0.616 | 0.322
constrained | FiLMv2 | 0.556 | 0.176
ScoreFormer | 0.542 | 0.154
L-ScoreFormer | 0.577 | 0.260
hiv | glide | unconstrained | FiLMv2 | 0.675 | 0.383
ScoreFormer | 0.669 | 0.394
L-ScoreFormer | 0.673 | 0.350
constrained | FiLMv2 | 0.645 | 0.337
ScoreFormer | 0.651 | 0.351
L-ScoreFormer | 0.637 | 0.308
all | all | all | FiLMv2 | 0.666 | 0.368
ScoreFormer | 0.673 | 0.382
L-ScoreFormer | 0.678 | 0.400

##  Appendix E Performance plots

Figure 2: Performance metrics by model type. Bar height indicate the mean
performance across different targets, docking engines and docking settings.
The metrics reported are those used for the evaluation of docking score
prediction and hit recovery.

Figure 3: wMSE grouped by docking settings, target and model type. Data is
only shown for the docking engine glide as is the only one presenting the two
different types of docking settings.

##  Appendix F Pipeline

In this section, we describe the key features of the pipeline used to develop
a functional surrogate model tailored to specific targets. An overview of the
pipeline is shown in Figure 4. For each target of interest, a unique dataset
is generated by calculating the docking scores for a set of molecules. These
molecules are represented as molecular graphs, which encode atom types and
their connections. Additionally, each atom’s representation is augmented with
the RWPE which stored for convenience. These graph representations, along with
the docking scores, are used to train the model. Crucially, information about
the target’s structure, sequence, or the molecule’s binding pose is not
directly included as input. Instead, the model is designed to infer which
molecular features are critical for binding affinity and how they influence
this for a specific target. Employing a target-specific approach offers
superior performance compared to target-agnostic models and, although it
requires generating and training a new dataset and model for each target, it
uses significantly fewer resources than conducting a full-scale HTVS. This
makes target-specific pipelines an efficient balance between speed and
precision.

Figure 4: Schematic representation of the pipeline used to generate a
functional model able to predict docking scores of new molecules.

##  Appendix G Ablation studies

To assess the impact of various components on model performance, we conducted
three ablation studies. In the first study, we replaced the PNA network with
two well-known GNNs, namely Graph Convolutional Network (GCN) and Graph
Attention Network v2 (GATv2). In the second study, we removed the LRWPE from
the ScoreFormer architecture. Lastly, we combined these two modifications,
training a GPS graph transformer without LRWPE and using either GCN or GATv2
instead of the PNA network.

We evaluated these architectures using the D4 dataset, with docking scores
calculated via Glide under the unconstrained protocol. The evaluation metrics
included the average scores from the best 10 steps for each metric, as shown
in Table 7, with the highest score for each metric highlighted in bold.

Our analysis focused on two metrics for score prediction — Pearson correlation
and R² — and one metric for hit recovery, wMSE, evaluated on both the training
and testing sets. Replacing the PNA layer with either GCN or GATv2 led to a
decrease in performance across all analyzed metrics. Similarly, removing the
LRWPE resulted in a comparable decline in all validation set metrics, though
the wMSE for the training set remained unaffected. This indicates that LRWPE
might have a regularization effect, helping to prevent overfitting and
ensuring consistent performance across different datasets. Combining both
ablations caused an even more significant performance drop across all metrics.

Collectively, these findings underscore the importance of the proposed
components in enhancing the accuracy of docking score predictions.

Table 7: Metrics obtained by ScoreFormer and ablated architectures.
Model | Pearson | $R^{2}$ | train wMSE | test wMSE
---|---|---|---|---
ScoreFormer | 0.779 | 0.564 | 0.269 | 0.341
GCN | 0.762 | 0.531 | 0.309 | 0.364
GATv2 | 0.761 | 0.526 | 0.297 | 0.364
No RWPE | 0.760 | 0.531 | 0.269 | 0.366
No RWPE GCN | 0.749 | 0.494 | 0.315 | 0.378
No RWPE GATv2 | 0.747 | 0.487 | 0.304 | 0.380

##  Appendix H Generalization studies

Here we include the complete table showing the results of the unseen chemistry
benchmark using molecules obtained from a generative model.

Table 8: Extended metrics obtained on generated molecules. Reference
configuration corresponds to constrained ScoreFormer results on Table 1. Best
values, excluding the reference, are highlighted in bold.
Configuration | RES | $AURTC_{0.01}$ | $AURTC_{0.001}$ | $R_{0.1,0.01}$ | $R_{0.1,0.001}$
---|---|---|---|---|---
ScoreFormer | 0.458 | 0.344 | 0.359 | 0.441 | 0.466
L-ScoreFormer | 0.449 | 0.358 | 0.336 | 0.441 | 0.466
FiLMv2 | 0.431 | 0.333 | 0.314 | 0.441 | 0.466
Reference | 0.746 | 0.653 | 0.735 | 0.826 | 0.968

Apart from the unseen chemistry benchmark, we conducted another which involved
testing the generalization capabilities of our models across different
molecular sizes. This test involved generating specific training and
evaluation sets based on a molecular weight threshold and then inferring on
the remaining testing molecules, with a higher molecular weight. Specifically,
ScoreFormer was trained on the 90% of molecules with the lowest molecular
weight and evaluated on the 10% of molecules with the highest molecular
weight. To prevent any possible information leakage during evaluation,
molecules in the validation set used for architecture optimization were
excluded. The results, displayed in table 9 showed a slight degradation in
performance compared to the model trained with a random selection of
molecules. However, the model still captured feature patterns associated with
binding affinities. The most significant degradation was observed in the
weighted Mean Squared Error (wMSE) and the R-squared (R²) coefficient. Despite
this, Pearson’s correlation coefficient and metrics related to hit recovery
were over 80% of those achieved by the random split approach, indicating that
ScoreFormer can generalize to out-of-distribution molecules, at least, when
there is a combinatorial relationship between the evaluation and training
sets.

Table 9: Inference evaluation metrics on the molecular weight generalization
benchmark.
Split | wMSE | RES | $AURTC_{0.01}$ | $AURTC_{0.001}$ | $R_{0.1,0.01}$ | $R_{0.1,0.001}$ | Pearson | $R^{2}$
---|---|---|---|---|---|---|---|---
Weight | 0.697 | 0.719 | 0.610 | 0.656 | 0.753 | 0.917 | 0.627 | 0.33
Random | 0.383 | 0.813 | 0.706 | 0.802 | 0.901 | 1.000 | 0.751 | 0.52

Both of these benchmarks together allow us to evaluate the generalization
capabilities of our models in various out-of-distribution situations and
demonstrate significant robustness across them.


# Label distribution learning via label correlation grid

###### Abstract

Label distribution learning can characterize the polysemy of an instance
through label distributions. However, some noise and uncertainty may be
introduced into the label space when processing label distribution data due to
artificial or environmental factors. To alleviate this problem, we propose a
Label Correlation Grid (LCG) to model the uncertainty of label relationships.
Specifically, we compute a covariance matrix for the label space in the
training set to represent the relationships between labels, then model the
information distribution (Gaussian distribution function) for each element in
the covariance matrix to obtain an LCG. Finally, our network learns the LCG to
accurately estimate the label distribution for each instance. In addition, we
propose a label distribution projection algorithm as a regularization term in
the model training process. Extensive experiments verify the effectiveness of
our method on several real benchmarks.

Index Terms—  label distribution learning, covariance matrix, label
correlation grid (LCG)

##  1 Introduction

In contrast to the traditional single-label learning and multi-label learning,
label distribution learning (LDL) [1] aims to characterize instance polysemy
with labels of different description degrees. Specifically, LDL extends
discrete labels into the form of a continuous probability distribution
(description degree). The descriptive degree is deemed as the relevance of the
label to the sample, and the sum of the descriptive degree of all labels is 1,
showing the relationship between the instance and the sample. For example, a
person’s facial expression [2, 3, 4] may involve multiple emotions, including
happiness, sadness, surprise, etc. LDL can convey such sophisticated and
ambiguous semantic information.

Since researchers built label distribution datasets by using manual annotation
and label enhancement algorithms [5, 6, 7, 8], the label space of the dataset
is usually disturbed by the introduction of uncertainty and noise. On the one
hand, the manually annotated datasets have a high degree of subjectivity and
ambiguity. On the other hand, the inductive bias of the label enhancement
algorithm cannot guarantee the accuracy of the label set. The uncertainty of
the label set further affects the modeling of label relations. To clear this
obstacle, we model the uncertainty of label relations to boost the performance
of the LDL algorithm.

Inspired by the idea of modeling the implicit label distribution proposed by
[9], our proposed model also implicitly characterizes the label relationship’s
uncertainty. So far, we develop a novel approach to learning label
distributions via a label correlation grid (LCG), which can model the
relationships between labels and learn the implied uncertainty between pairs
of labels. In particular, we first use an MLP (multi-layer perceptron) to
extract the deep semantics of the input information in order to perform
learning label distribution. Then, we use the predicted label distribution to
compute a covariance matrix between the labels, with the help of which we
generate an LCG. Each cell in the grid is generated by sampling the element
values in the covariance matrix using a Gaussian distribution function. Note
that this grid is constrained by the grid generated from the label space of
the samples with the help of the $L_{1}$ norm.

In addition, since deep networks are prone to overfitting when learning
tabular data, we propose a label distribution projection algorithm as a
regularization term that performs on datasets with numerous labels. The label
distribution projection algorithm aims to extract the depth semantics of the
label space (in this paper, we use KPCA [10]), similar to the perceptual loss
[11], which improves the generalization ability of the whole model. Compared
to traditional LDL algorithms, our model can execute acceleration on GPU
(millisecond level speed on one inference), which is unparalleled competitive
in terms of efficiency and accuracy. Extensive experiments (including the
standard accuracy experiment and the noise immunity experiment) have confirmed
the effectiveness of our approach. Our main contribution includes three folds:
i): We propose a new method to learn label distributions via a label
correlation grid that models the uncertainty of the label relationship. ii):
We design a robust depth model that combines efficiency (ms level) and
accuracy on a tabular dataset. iii): To alleviate the overfitting of the
model, we design a label distribution projection algorithm on the dataset with
a large number of labels.

Fig. 1: Our architecture. Our algorithm aims to regress the label distribution
of a sample using an MLP, where there are three learning targets, one is to
learn the true label distribution, one is to learn the projection of the true
label distribution, and one is to learn the LCG.

##  2 related work

Geng et al. [1] first proposed a new machine learning paradigm: LDL, which
focuses on the relative importance of each label to an instance. More
researchers have since devoted themselves to LDL [12, 13, 14]. In these
researches, label correlation is usually applied to boost the generalization
ability of the model. LDL-LCLR [15] proposed to utilize both the global and
local relevance among labels to train an LDL algorithm. Wang et al. [16] tried
the underlying manifold structure of label distribution to encode the
correlations among labels. Jia et al. [17] introduced a ranking loss in the
LDL model to maintain the label ranking relationship. Qian et al. [18]
proposed an algorithm to generate the label correlations that uses
neighborhood granularity to explore feature similarity while generating the
label correlation with the correlation coefficient. In contrast to the
explicit modeling learning algorithm described above, we focus on implicitly
representing correlations between labels, while taking into account the
uncertainty factor carried by the dataset itself.

##  3 PROPOSED METHOD

The architecture of our method is shown in Fig. 1. MLP learns the label
distribution based on the deep semantics of the input features. Among them, we
conduct three sub-learning targets, a key target is to learn the true label
distribution by using $L_{1}$ loss, followed by learning the LCG and another
one (label distribution projection loss) is to learn the deep semantics of the
label distribution. Of these, the elements of the LCG come from the Gaussian
distribution and the depth semantics of label distribution are from KPCA.

Notations. Given a particular example, the goal of our method is to learn the
degree to which each label describes that instance. Input matrix $X\in
R^{m\times n}$, where $m$ is the number of instances and $n$ is the dimension
of features. We define the i-th instance in the dataset as $x_{i}$. The label
distribution is defined $Y\in R^{m\times t}$, then the j-th label is defined
$y_{j}$. For each instance $x_{i}$, we define its label distribution
$D_{i}=\left\\{{d_{x_{i}}^{y_{1}},d_{x_{i}}^{y_{2}},...,d_{x_{i}}^{y_{j}}}\right\\}$
where $d_{x_{i}}^{y_{j}}$ is the degree of description of the label $y_{j}$
for the sample $x_{i}$. The $d_{x_{i}}^{y_{j}}$ is constrained by
${{d_{x_{i}}^{y_{j}}}}\in[0,1]$ and $\sum_{j=1}^{j}{d_{x_{i}}^{y_{j}}}=1$.

Network Architecture. Early studies introduced neural networks into LDL, such
as AA-BP [1]. We refer to this network architecture to regress a label
distribution. Firstly, the input instance $x_{i}$ is fed into an MLP with two
linear layers. In the first hidden layer, we used 256 neurons to learn the
latent features. The number of linear layer neurons in the last layer is the
number of labels. Note that the number of neurons is set to the power of 2 in
favor of acceleration on the GPU shader. Two non-linear layers (sigmoid and
softmax) are added after the hidden layer and the output layer, respectively.
The MLP has the ability to learn the latent semantic and global information of
the input examples and generates accurate label distribution values. The
predicted label description degree $\tilde{d_{x_{i}}^{y_{j}}}$ of each
instance $x_{i}$ is

$\displaystyle\tilde{d_{x_{i}}^{y_{j}}}=\frac{\text{exp}(\text{MLP}(x_{i}))}{\sum_{k=1}^{j}{\text{exp}(\text{MLP}(x_{k}))}}.$
(1)

We optimize the MLP by minimizing the $L_{1}$ distance between the predicted
label distribution and the ground truth as follows:

$\displaystyle
L_{\text{1}}\left(d_{i},\tilde{d_{i}}\right)=\sum_{i=1}^{T}\left|\left|{d_{x_{i}}^{y_{j}}-\tilde{d_{x_{i}}^{y_{j}}}}\right|\right|,$
(2)

where $\tilde{d_{i}}$ is the predicted label distribution and $d_{i}$ is the
true label distribution, and T denotes the number of labels. The following
describes some key regularization terms.

Label Correlation Grid. Implicit representation [9] has nice properties due to
its powerful expressiveness and for this reason, we develop the scheme of
implicitly representing label relations. To robustly model label uncertainty,
we build a tensor that can accommodate high-dimensional information. Summing
up the above thesis, we propose an LCG $y\\_grid\in R^{t\times t\times t}$.
$y\\_grid$ is initially composed of a covariance matrix computed from the
label space, where each element is sampled by a Gaussian function to create a
vector. The Gaussian distribution has two hyperparameters and we use each
element of the covariance matrix as the mean and the variance is set fixed at
0.5. We also try other values {0.1, 0.2, … ,0.9} as the variance, but the
results are not good as seen from the extensive experimental results. The LCG
is shown as:

$\bNiceMatrix[name=a]y_{1}^{1}&\pagecolor{blue!20}y_{1}^{2}\cdots y_{1}^{t}\\\
y_{2}^{1}y_{2}^{2}\cdots y_{2}^{t}\\\ \vdots\vdots\ddots\vdots\\\
y_{t}^{1}y_{t}^{2}\cdots
y_{t}^{t}\qquad\bNiceMatrix[name=b]q_{1},q_{2},...,\mathchoice{\leavevmode\hbox
to6.6pt{\vbox to6.25pt{\pgfpicture\makeatletter\hbox{\hskip 3.302pt\lower
0.2pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-3.302pt}{2.14442pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\displaystyle
q_{l}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to6.6pt{\vbox to6.25pt{\pgfpicture\makeatletter\hbox{\hskip 3.302pt\lower
0.2pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-3.302pt}{2.14442pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\textstyle q_{l}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to4.65pt{\vbox to4.4pt{\pgfpicture\makeatletter\hbox{\hskip 2.32413pt\lower
0.2pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.32413pt}{1.58888pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\scriptstyle q_{l}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to3.68pt{\vbox to3.54pt{\pgfpicture\makeatletter\hbox{\hskip 1.84183pt\lower
0.2pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-1.84183pt}{1.58888pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\scriptscriptstyle
q_{l}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}\leavevmode\hbox
to1.5pt{\vbox to3.2pt{\pgfpicture\makeatletter\hbox{\hskip
1.19998pt\lower-1.59998pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} {{}{}}{{}}
{{{}}{{}}}{{}}{{{}}{{}}}{}{{}}{}{{}}{}{{}{}{}{}{}}\pgfsys@beginscope\pgfsys@invoke{
}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\pgfsys@color@rgb@stroke{1}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{1}{0}{0}\pgfsys@invoke{
}\definecolor[named]{pgffillcolor}{rgb}{1,0,0}{}{{
{\pgfsys@beginscope\pgfsys@setlinewidth{0.32pt}\pgfsys@setdash{}{0.0pt}\pgfsys@roundcap\pgfsys@roundjoin{}
{}{}{} {}{}{}
\pgfsys@moveto{-1.19998pt}{1.59998pt}\pgfsys@curveto{-1.09998pt}{0.99998pt}{0.0pt}{0.09999pt}{0.29999pt}{0.0pt}\pgfsys@curveto{0.0pt}{-0.09999pt}{-1.09998pt}{-0.99998pt}{-1.19998pt}{-1.59998pt}\pgfsys@stroke\pgfsys@endscope}}
}{}{}{{}}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@curveto{0.0pt}{0.0pt}{0.0pt}{0.0pt}{0.45999pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
}{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-0.45999pt}{0.0pt}\pgfsys@invoke{
}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}{{}}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}$

in which $y_{i}^{j}=[q_{1},q_{2},...,q_{l}]$ indicates the information between
the i-th label and the j-th label. The final LCG is optimized by calculating
the $L_{1}$ distance. A large number of papers use Gaussian priors for
uncertainty estimation of unknown information. So we use the Gaussian process
regression [19] to generate the description information.

Label Distribution Projection Algorithm. To boost the performance of the
model, we propose a label distribution projection algorithm as a
regularization term in the network training process. We use a sub-MLP to
aggregate feature information into a lower dimension in the network. The sub-
MLP consists of one linear layer with sigmoid and softmax as activation
functions. For example, the 68-dimensional raw label distribution is mapped to
a the 32-dimensional label distribution space by sub-MLP for the Human Gene
dataset. The formulaic expression form is as follows:

$\displaystyle\tilde{d_{x_{i}}^{y_{j}}}^{\prime}=\text{softmax}(\text{sub-
MLP}(d_{x_{i}}^{y_{j}})).$ (3)

For true low-dimensional label distributions, we use Kernel-based principal
component analysis (KPCA) to project the raw true label distribution. KPCA
draws on a nonlinear approach to extract the abstract semantics of the label
distribution, and in addition, has the effect of reducing noise. The lower
label distribution is calculated as ${d_{x_{i}}^{y_{j}}}^{\prime}$. We use
$L_{1}$ to minimize the output of the network and the true low-dimensional
label distribution.

Loss Function. The network parameters are optimized by combining $L_{1}$,
projection loss ($L_{ldp}$) and label correlation grid loss ($L_{lcg}$).
Combined with the content of the above, our target function is constructed as
follows:

$\displaystyle Loss=\lambda_{1}\times L_{1}+\lambda_{2}\times
L_{ldp}+\lambda_{3}\times L_{lcg},$ (4)

where $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ are parameters. In the
four label distribution datasets, we select $\lambda_{1}$, $\lambda_{2}$ and
$\lambda_{3}$ to be 1, 0.1 and 0.05, respectively.

Table 1: Statistics of four real-world datasets. Dataset | Examples | Features | Labels
---|---|---|---
Human Gene | 17892 | 36 | 68
Natural Scene | 2000 | 294 | 9
Yeast-alpha | 2465 | 24 | 18
Movie | 7755 | 1869 | 5
Table 2: Parameters on the four datasets. Dataset | Batch size | Learning rate | Epoch | LDP loss
---|---|---|---|---
Human Gene | 500 | 0.0005 | 400 | 32
Natural Scene | 500 | 0.0005 | 300 | -
Yeast-alpha | 500 | 0.0005 | 400 | 16
Movie | 2000 | 0.0005 | 100 | -

##  4 EXPERIMENTS

Datasets. To validate the efficiency of our model, we conduct experiments on
four LDL datasets. Details of the datasets from Wang et al. [16] are presented
in Table 1. The datasets involve different types, including biological
experiment datasets, natural scene datasets, movie category datasets, and
large-scale biomedical research datasets. The evaluation is based on six
indicators proposed by [1], including Chebyshev distance (Chebyshev), Clark
distance (Clark), Canberra distance (Canberra), KL divergence (K-L), Cosine
similarity (Cosine), and Intersection similarity (Intersection).

Comparative Algorithms. We compared the results of our model with five LDL
algorithms: DDH-LDL [20], BFGSLLD [1], LDL-LRR [17], LDLSF [21] and LALOT
[22]. Except for DDH-LDL, the indicators for all datasets of the models are
from [17]. Our experiments in DDH-LDL are based on the tips in the code
website corresponding to the article.

Table 3: Experimental results on four datasets and the best results are
bolded.

Dataset | Algorithm | Chebyshev $\downarrow$ | Clark $\downarrow$ | Canberra $\downarrow$ | K-L $\downarrow$ | Cosine $\uparrow$ | Intersection $\uparrow$
---|---|---|---|---|---|---|---
Movie | Ours | 0.1145$\pm$0.0073 | 0.5054$\pm$0.0027 | 0.9746$\pm$0.0055 | 0.1178$\pm$0.0029 | 0.9446$\pm$0.0092 | 0.8384$\pm$0.0090
DDH-LDL | 0.1543$\pm$0.0634 | 0.6427$\pm$0.2526 | 1.2094$\pm$0.0051 | 0.1451$\pm$0.0127 | 0.9026$\pm$0.0048 | 0.7915$\pm$0.0913
LDL-LRR | 0.1151$\pm$0.0016 | 0.5235$\pm$0.0065 | 0.9996$\pm$0.0093 | 0.0982$\pm$.0031 | 0.9353$\pm$0.0020 | 0.8355$\pm$0.0023
BFGS-LLD | 0.1397$\pm$0.0223 | 0.5856$\pm$0.0494 | 1.1309$\pm$0.1098 | 0.1378$\pm$0.0373 | 0.9108$\pm$.0227 | 0.8053$\pm$0.0263
LALOT | 0.2854$\pm$0.0903 | 1.431$\pm$0.9006 | 2.9023$\pm$0.8996 | 3.7216$\pm$0.9542 | 0.7021$\pm$0.1415 | 0.6015$\pm$0.2114
LDLSF | 0.1271$\pm$0.0027 | 0.6227$\pm$0.0112 | 1.0166$\pm$0.0158 | 0.1954$\pm$0.0257 | 0.9220$\pm$0.0028 | 0.8180$\pm$0.0034
Yeast-alpha | Ours | 0.0133$\pm$0.0025 | 0.2100$\pm$0.0083 | 0.6708$\pm$0.2624 | 0.0062$\pm$0.0194 | 0.9947$\pm$0.0099 | 0.9626$\pm$0.0056
DDH-LDL | 0.1627$\pm$0.0529 | 0.5228$\pm$0.1937 | 1.0236$\pm$0.3634 | 0.1177$\pm$0.0638 | 0.8794$\pm$0.0936 | 0.8128$\pm$0.9355
LDL-LRR | 0.0134$\pm$0.0002 | 0.2093$\pm$0.0031 | 0.6791$\pm$0.0100 | 0.0054$\pm$0.0001 | 0.9946$\pm$0.0001 | 0.9625$\pm$0.0005
BFGS-LLD | 0.0135$\pm$0.0001 | 0.2110$\pm$0.0030 | 0.6865$\pm$0.0106 | 0.0055$\pm$0.0001 | 0.9949$\pm$0.0001 | 0.9629$\pm$0.0006
LALOT | 0.0165$\pm$0.0003 | 0.2608$\pm$0.0043 | 0.8544$\pm$0.0150 | 0.0084$\pm$0.0003 | 0.9917$\pm$0.0003 | 0.9526$\pm$0.0008
LDLSF | 0.0139$\pm$.0002 | 0.2164$\pm$0.0021 | 0.6874$\pm$0.0111 | 0.0058$\pm$0.0001 | 0.9943$\pm$0.0001 | 0.9613$\pm$.00004
Natural Scene | Ours | 0.3298$\pm$0.0020 | 1.8424$\pm$0.0004 | 4.3583$\pm$0.0125 | 2.332$\pm$0.00178 | 0.7607$\pm$0.0075 | 0.5749$\pm$0.0078
DDH-LDL | 0.4051$\pm$0.0022 | 2.1792$\pm$0.0037 | 5.5492$\pm$1.3133 | 1.6338$\pm$0.4791 | 0.5603$\pm$0.0067 | 0.4228$\pm$0.0055
LDL-LRR | 0.2997$\pm$0.0070 | 2.4268$\pm$0.0106 | 5.7143$\pm$0.0126 | 0.7314$\pm$0.0070 | 0.7532$\pm$0.0044 | 0.5670$\pm$0.0022
BFGS-LLD | 0.3511$\pm$0.0158 | 2.4756$\pm$0.0217 | 5.8445$\pm$0.0932 | 0.9067$\pm$0.0556 | 0.6765$\pm$0.0211 | 0.4778$\pm$0.0196
LALOT | 0.3873$\pm$0.0033 | 2.4999$\pm$0.0030 | 5.9962$\pm$0.0251 | 1.2529$\pm$0.0145 | 0.5473$\pm$0.0046 | 0.3600$\pm$0.0029
LDLSF | 0.3196$\pm$0.0092 | 2.4622$\pm$0.0159 | 6.6739$\pm$0.0139 | 0.9981$\pm$0.0328 | 0.7297$\pm$0.0074 | 0.5397$\pm$0.0074
Human gene | Ours | 0.0528$\pm$0.0074 | 2.1000$\pm$1.245 | 14.3673$\pm$0.1252 | 0.2271$\pm$0.0347 | 0.8352$\pm$0.0082 | 0.7849$\pm$0.0085
DDH-LDL | 0.0597$\pm$0.0752 | 2.8830$\pm$0.8514 | 19.9391$\pm$0.9627 | 0.3815$\pm$0.3870 | 0.7313$\pm$0.8355 | 0.6878$\pm$0.8845
LDL-LRR | 0.0532$\pm$0.0011 | 2.1114$\pm$0.0122 | 13.5681$\pm$0.0025 | 0.2365$\pm$0.0049 | 0.8346$\pm$0.0020 | 0.7844$\pm$0.0014
BFGS-LLD | 0.0539$\pm$0.0009 | 2.1270$\pm$0.0141 | 14.5633$\pm$0.1107 | 0.2398$\pm$0.0038 | 0.8328$\pm$0.0018 | 0.7828$\pm$0.0014
LALOT | 0.0573$\pm$0.0012 | 4.0703$\pm$3.6917 | 17.8198$\pm$3.7167 | 0.2956$\pm$0.0059 | 0.8055$\pm$0.0024 | 0.7578$\pm$0.0016
LDLSF | 0.0533$\pm$0.0009 | 2.1295$\pm$0.0209 | 14.5681$\pm$0.0055 | 0.2395$\pm$0.0054 | 0.8332$\pm$0.0028 | 0.7828$\pm$0.0022

Experimental Setting. For our algorithm, we put the customized selection of
parameters for each dataset in Table 2. The parameters that may be used
include batch size, learning rate, and epoch. Since the label space of the
three datasets Natural Scene, Movie is small, there is not enforced LDP loss
function in the training phase. Here, 32 indicates that we squeeze the raw
label dimension from 68 to 32 using KPCA, followed immediately by
normalization using softmax. Besides, we use the PyTorch framework and AdamW
optimizer to train and test a deep network on the GPU shader.

Results and Discussion. The experimental results for each dataset are
summarized in Table 3, where we take the results of the 10 times 5-fold cross-
validation. The experimental results are reported in the form of
“mean$\pm$std”. $\uparrow$ indicates that the higher the value of the metric,
the better. $\downarrow$ indicates that the smaller the value of the metric,
the better.

Our algorithm performs optimally on six parameters across the four datasets,
particularly on the metric Clark. Our approach also achieves competitive
results in the remaining metrics. This is mainly due to the ability of the
algorithm to expand label relationships and mine feature semantic information.
Besides, we observe that algorithms that exploit label relations, like LDL-LRR
and ours, generally outperform other algorithms, which demonstrates the
importance of label relations in label distribution. Our method has a better
performance compared to LDL-LRR by taking into account the uncertainty of the
label relationship. Although our modeling has an expensive cost, a GPU
acceleration can be enforced in the PyTorch to alleviate this problem. We
evaluate the inference speed of our model on CPU (AMD 8-core processor) and
GPU (RTX3080) as 22ms and 1ms, respectively.

Ablation Studies. We conduct ablation studies to demonstrate the effectiveness
of LCG and label distribution projection algorithms on dataset Human gene. The
results of simple ablation experiments are shown in Table 4. w/o grid is an
algorithm for removing the LCG and w/o $L_{dpa}$ is defined when the loss term
$L_{dpa}$ is removed. The variation of the data illustrates the effectiveness
of the LCG and the label distribution projection algorithms. We conduct 10
times 5-fold cross-validation on the dataset of the ablation studies. The gray
markers represent optimal performance.

Noise Interference. To verify the robustness of our algorithm against the
noise and uncertainty in label space, we experimented on the dataset Movie
with Gaussian noise. We synthesize a dataset on Movie by Gaussian function to
enforce different degrees of noise, and the labels with noise are finally
normalized by softmax. For label space on the dataset, the variance of the
Gaussian distribution function is selected from (0.1, 0.2, …, 1).

Table 4: Results on Ablation studies.

Measures | Ours | w/o grid | w/o $L_{dpa}$
---|---|---|---
Chebyshev $\downarrow$ | 0.0528$\pm$0.0074 | 0.5346$\pm$0.0021 | 0.5349$\pm$0.0132
Clark $\downarrow$ | 2.1000$\pm$1.2455 | 2.1011$\pm$1.1352 | 2.1073$\pm$0.0047
Canberra $\downarrow$ | 14.3673$\pm$0.1252 | 14.3798$\pm$0.1368 | 14.3689$\pm$0.0054
K-L $\downarrow$ | 0.2271$\pm$0.0347 | 0.2274$\pm$0.0052 | 0.2283$\pm$0.0032
Cosine $\uparrow$ | 0.8352$\pm$0.0082 | 0.8350$\pm$0.0086 | 0.8313$\pm$0.0861
Intersection $\uparrow$ | 0.7849$\pm$0.0085 | 0.7845$\pm$0.0074 | 0.7841$\pm$0.0015

It is widely believed that the higher the variance, the more dispersed the
data, and the more noise. We conduct 10 times 5-fold cross-validation on the
synthetic dataset. Our algorithm yield 0.1257$\pm$0.0071 (Chebyshev),
0.5713$\pm$0.0027 (Clark), 1.0854$\pm$0.0055 (Canberra), 0.1452$\pm$0.0033
(K-L), 0.9250$\pm$0.0092 (Cosine), 0.8203$\pm$0.0037 (Intersection)
respectively on the data set (the variance is 1) with Gaussian noise has a
variance of 1 and mean of 0. However, the accuracy of the label distribution
predicted by other algorithms decreases by 50% on average, and our model is
still very competitive.

##  5 CONCLUSION

Ambiguity and uncertainty of label space in LDL become major challenges,
limiting the performance of LDL models. In this paper, we propose a new
algorithm for label distribution by using an LCG. In standard evaluation
experiments, our method achieves competitive results, especially on the metric
of vector similarity rather than distance. Furthermore, our method has a
striking performance on the mini-test of noise resistance, thanks to the gains
from uncertainty modeling. Our model can execute GPU acceleration on
processing LDL data, balancing accuracy and efficiency.

## References

  * [1] Xin Geng,  “Label distribution learning,”  TKDE, 2016.
  * [2] Zengqun Zhao, Qingshan Liu, and Feng Zhou,  “Robust lightweight facial expression recognition network with label distribution training,”  in AAAI, 2021.
  * [3] Shijing Si, Jianzong Wang, Junqing Peng, and Jing Xiao,  “Towards speaker age estimation with label distribution learning,”  in ICASSP, 2022.
  * [4] Jingying Chen, Chen Guo, Ruyi Xu, Kun Zhang, Zongkai Yang, and Honghai Liu,  “Toward children’s empathy ability analysis: Joint facial expression recognition and intensity estimation using label distribution learning,”  TII, 2022.
  * [5] Xingyu Zhao, Yuexuan An, Ning Xu, and Xin Geng,  “Fusion label enhancement for multi-label learning,”  in IJCAI, 2022.
  * [6] Qinghai Zheng, Jihua Zhu, Haoyu Tang, Xinyuan Liu, Zhongyu Li, and Huimin Lu,  “Generalized label enhancement with sample correlations,”  TKDE, 2020.
  * [7] Ning Xu, Jun Shu, Yun-Peng Liu, and Xin Geng,  “Variational label enhancement,”  in ICML, 2020.
  * [8] Ning Xu, Yun-Peng Liu, and Xin Geng,  “Label enhancement for label distribution learning,”  TKDE, 2021.
  * [9] Zhuoran Zheng and Xiuyi Jia,  “Label distribution learning via implicit distribution representation,”  arXiv preprint arXiv:2209.13824, 2022.
  * [10] Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller,  “Nonlinear component analysis as a kernel eigenvalue problem,”  Neural Computation, 1998.
  * [11] Leon A Gatys, Alexander S Ecker, and Matthias Bethge,  “Image style transfer using convolutional neural networks,”  in CVPR, 2016.
  * [12] Deyu Zhou, Xuan Zhang, Yin Zhou, Quan Zhao, and Xin Geng,  “Emotion distribution learning from texts,”  in EMNLP, 2016.
  * [13] Ying Zhou, Hui Xue, and Xin Geng,  “Emotion distribution recognition from facial expressions,”  in ACM MM, 2015.
  * [14] Jing Wang and Xin Geng,  “Classification with label distribution learning,”  in IJCAI, 2019.
  * [15] Tingting Ren, Xiuyi Jia, Weiwei Li, and Shu Zhao,  “Label distribution learning with label correlations via low-rank approximation,”  in IJCAI, 2019.
  * [16] Jing Wang and Xin Geng,  “Label distribution learning by exploiting label distribution manifold,”  TNNLS, 2021.
  * [17] Xiuyi Jia, Xiaoxia Shen, Weiwei Li, Yunan Lu, and Jihua Zhu,  “Label distribution learning by maintaining label ranking relation,”  TKDE, 2021.
  * [18] Wenbin Qian, Yinsong Xiong, Jun Yang, and Wenhao Shu,  “Feature selection for label distribution learning via feature similarity and label correlation,”  Information Sciences, 2022.
  * [19] Andrew Gordon Wilson, David A Knowles, and Zoubin Ghahramani,  “Gaussian process regression networks,”  arXiv preprint arXiv:1110.4411, 2011.
  * [20] Zhen Zhang, Lei Zhu, Yaping Li, and Yang Xu,  “Deep discrete hashing for label distribution learning,”  IEEE Signal Process. Lett., 2022.
  * [21] Tingting Ren, Xiuyi Jia, Weiwei Li, Lei Chen, and Zechao Li,  “Label distribution learning with label-specific features,”  in IJCAI, 2019.
  * [22] Peng Zhao and Zhi-Hua Zhou,  “Label distribution learning by optimal transport,”  in AAAI, 2018.