# Generating Unbiased Pseudo-labels via a Theoretically Guaranteed Chebyshev
Constraint to Unify Semi-supervised Classification and Regression

Jiaqi Wu1, Junbiao Pang1, Qingming Huang2

###### Abstract

Both semi-supervised classification and regression are practically challenging
tasks for computer vision. However, semi-supervised classification methods are
barely applied to regression tasks. Because the threshold-to-pseudo label
process (T2L) in classification uses confidence to determine the quality of
label. It is successful for classification tasks but inefficient for
regression tasks. In nature, regression also requires unbiased methods to
generate high-quality labels. On the other hand, T2L for classification often
fails if the confidence is generated by a biased method. To address this
issue, in this paper, we propose a theoretically guaranteed constraint for
generating unbiased labels based on Chebyshev’s inequality, combining multiple
predictions to generate superior quality labels from several inferior ones. In
terms of high-quality labels, the unbiased method naturally avoids the
drawback of T2L. Specially, we propose an Unbiased Pseudo-labels network (UBPL
network) with multiple branches to combine multiple predictions as pseudo-
labels, where a Feature Decorrelation loss (FD loss) is proposed based on
Chebyshev constraint. In principle, our method can be used for both
classification and regression and can be easily extended to any semi-
supervised framework, e.g. Mean Teacher, FixMatch, DualPose. Our approach
achieves superior performance over SOTAs on the pose estimation datasets
Mouse, FLIC and LSP, as well as the classification datasets CIFAR10/100 and
SVHN.

## Introduction

Classification and regression are two important tasks in computer vision that
have achieved great success in numerous real-world applications (Krizhevsky,
Hinton et al. 2009; He et al. 2016; Sun et al. 2019; Wang, Wang, and Yuille
2013, 2016), especially in supervised learning scenarios that utilize large
amounts of high-quality labeled data. However, labeled data is difficult to
obtain in many domains, and manually labeling data is a time-consuming and
costly task. For example, in medical tasks, multiple human experts need to
perform time-consuming analysis to label samples. Compared to labeled data,
unlabeled data is more accessible and less costly. Therefore, semi-supervised
learning (SSL), which can leverage a large amount of unlabeled data to improve
the performance of learning with a limited amount of labeled data, has
attracted a lot of attention.

Figure 1: Error and confidence curves of two pseudo-labels generated by Mean
Teacher (Tarvainen and Valpola 2017), a typical semi-supervised classification
method, in the pose estimation task.

The Pseudo-labeling (PL) method is an essential type of semi-supervised
classification methods. The PL methods (Xie et al. 2020; Sohn et al. 2020;
Zhang et al. 2021; Wang et al. 2022) is to use the pseudo-labels generated by
the model to train itself during training. Thus the PL method relies on the
confidence of the pseudo-label, and the pseudo-label with strong confidence
score is selected out and added to the training data set through the
threshold-to-pseudo label process (T2L). In the classification task, the T2L
is justified. This is because confidence, which is the probability of a
model’s prediction, can be viewed as how confident the model is about the
outcome of the prediction, which can effectively assess the quality of the
prediction.

Nonetheless, semi-supervised classification methods are barely applied to
regression tasks, since the original T2L in classification is not suitable for
regression tasks. Take pose estimation as an example. In pose estimation
tasks, the dominant approach is to use heatmaps to predict keypoints. Compared
to classification, the confidence in the heatmap represents only the
possibility that a location of the heatmap is a keypoint, not the quality of
that prediction. As shown in Fig. 1, the error of the generated pseudo-label
does not decrease when the confidence of the pseudo-label increases. This
indicates that the SSL method is not unbiased in the regression task, and the
generated confidence scores do not effectively assess the quality of the
pseudo-labels. Therefore, for the regression task, semi-supervised
classification methods fail to generate high-quality pseudo-labels.

The ”non-unbiased” problem also exists in classification. Currently, semi-
supervised classification methods focus more on how to improve the dynamic
control strategy for confidence thresholds, hoping to select out high-quality
pseudo-labels through more reasonable dynamic thresholds. However, these
methods all ignore the point of whether the confidence itself is plausible,
after all, the confidence is also predicted by the method. If the method
itself is biased, the confidence of the generated predictions may be
inaccurate, resulting in inaccurate pseudo-labels selected by T2L. Therefore,
”non-unbiased” can not be ignored in the semi-supervised classification task.

To solve the ”non-unbiased” problem, we propose a theoretically guaranteed
constraint for generating unbiased labels based on Chebyshev’s inequality,
encouraging combining multiple predictions to generate superior quality labels
from several inferior ones. Specially, we propose an Unbiased Pseudo-labels
network (UBPL network) with multiple branches to combine multiple predictions
as pseudo-labels, where a Feature Decorrelation loss (FD loss) is proposed
based on Chebyshev constraint. In principle, our method can be used for both
classification and regression and can be easily extended to any semi-
supervised framework, e.g. Mean Teacher (Tarvainen and Valpola 2017), FixMatch
(Sohn et al. 2020), DualPose (Xie et al. 2021). Our approach achieves superior
performance over SOTAs on the pose estimation datasets Mouse, FLIC (Sapp and
Taskar 2013) and LSP (Johnson and Everingham 2011), as well as the
classification datasets CIFAR10/100 (Krizhevsky, Hinton et al. 2009) and SVHN
(Netzer et al. 2011).

The regression project code and Mouse dataset are publicly available on
https://github.com/Qi2019KB/UBPL-PoseEstimation. And, the classification
project code is publicly available on https://github.com/Qi2019KB/UBPL-
Classification

The key contributions of this work are:

  * •

We propose a theoretically guaranteed constraint for generating unbiased
labels based on Chebyshev’s inequality, which encourages combining multiple
predictions to generate superior quality labels from several inferior ones.
Benefiting from this, semi-supervised classification methods can be well
applied to regression tasks, which implies unifying semi-supervised
classification and regression.

  * •

We propose a Feature Decorrelation loss (FD loss) based on the Chebyshev
constraint to maximize the diversity. The FD loss effectively improves the
accuracy and sustainability of the generated pseudo-labels.

## Related Work

### Semi-supervised Classification

Semi-supervised learning (SSL) can achieve better training results by using a
limited amount of labeled data and a large amount of unlabeled data. Because
it can effectively reduce the need for labeled data, SSL is well suited for
real-world application scenarios. There have been a number of excellent
results in semi-supervised classification.

The first type is the Pseudo-label (PL) method. The PL method trains itself
during training using pseudo-labels generated by the model. In general, the PL
method uses the threshold-to-pseudo label process (T2L) to select out high-
quality pseudo-labels to add to the training set. FixMatch (Sohn et al. 2020)
is a landmark that greatly improves the performance of SSL. FixMatch uses the
predictions of weakly augmented samples as pseudo-labels to supervise the
predictions of strongly augmented samples, thereby obtaining an additional
supervised signal from unlabeled samples. FlexMatch (Zhang et al. 2021)
proposed the idea of adaptive the threshold based on FixMatch. FreeMatch (Wang
et al. 2022) adaptively adjusts the threshold based on the learning state of
the model and uses an adaptive class fairness regularization penalty to
encourage the model to make diversified predictions in the early training
phase. These methods are mainly improved from the perspective of threshold in
T2L, focusing on how to select better pseudo-labels.

Another Type is the Consistency Regularization (CR) method (Sajjadi,
Javanmardi, and Tasdizen 2016; Laine and Aila 2016; Tarvainen and Valpola
2017; Miyato et al. 2018; Bachman, Alsharif, and Precup 2014; Ke et al. 2019).
The CR method encourages the model to produce approximate predictions for
different perturbations in the same sample and minimizes the prediction
difference to train the model. Such perturbation includes data enhancement
(Sajjadi, Javanmardi, and Tasdizen 2016), adversarial sample (Miyato et al.
2018), model perturbation (Bachman, Alsharif, and Precup 2014), etc. A typical
approach to CR methods is the teacher-student structure (Tarvainen and Valpola
2017), which uses the predictions of the teacher to supervise the students
through consistency constraints. Similar to the PL method, since the target is
generated by the model itself, the target is imprecise. The CR method also
requires T2L to select high-quality pseudo-labels, whose confidence predicted
from teacher is higher than that from student.

Although the above methods all use T2L to generate pseudo-labels, there are
still shortcomings. This is because, when confidence thresholds are used for
T2L, it is not considered whether the confidence scores generated by the bias
method are accurate or sufficiently reflective of the quality of the pseudo-
labels.

### Semi-supervised Regression

At present, the research on semi-supervised regression method is much less
than semi-supervised classification. Since regression methods predict true-
valued outputs, whereas classification methods typically predict within a
finite number of categories, regression is far more difficult than
classification. However, this difficulty is more prominent in semi-supervised
regression tasks, which makes the confidence and the corresponding prediction
result more imprecise. As a result, the pseudo-labels selected with T2L are
also less accurate.

Take pose estimation as an example. DualPose (Xie et al. 2021) is a milestone
in the field of semi-supervised pose estimation. DualPose addresses the
problem of model collapsing in 2D human pose estimation by the consistent
regularization loss and proposes to solve the problem by constructing pairs of
samples with different prediction difficulties. However, the confidence of a
heatmap only indicates the possibility that a certain location in the heatmap
is a keypoint, not the quality of the prediction. As a consequence, if a
method is biased, the pseudo-label selected by T2L cannot provide an
additional effective supervision signal due to the inaccurate generated
pseudo-label.

To further confirm this, we use Mean Teacher for semi-supervised pose
estimation, and the confidence and error of the pseudo-labels generated by
Teacher are shown in Fig. 1. As the confidence of the pseudo-label increases,
the error of the pseudo-label does not decrease. These results indicate that
the SSL method is not unbiased in the regression task, and the generated
confidence scores are not effective in evaluating the quality of the pseudo-
labels. Therefore, regression also requires unbiased methods to generate high-
quality labels.

### Multi-branch structure in SSL

Several works have applied multi-branch structure to SSL. (Ke et al. 2019)
finds that, under the influence of EMA mechanism, the model parameters of Mean
Teacher and student become more and more similar, which leads to the
convergence of their predictions and the loss of the supervision effectiveness
of consistency constraints. To this end, in (Ke et al. 2019), an additional
student is added to reduce this impact. Also considering that the model
parameters of teacher and student tend to be consistent, (Tang et al. 2021)
proposes a two-teacher structure, and EMA is used to update the parameters of
two teachers alternately, so as to increase the difference between the two
teachers’ parameters. However, both approaches have drawbacks. Although the
additional student in (Ke et al. 2019) reduces the risk of parameter
consistent, compared with teacher, student has lower prediction accuracy and
stability, and is more likely to introduce noise, reducing the accuracy of
predictions. In (Tang et al. 2021), although the diversity has been improved
to some extent, the trend of parameter consistent between teacher and student
has not changed substantially.

Considering these, we propose a network with multiple branches to combine
multiple predictions as pseudo-labels, where a Feature Decorrelation loss (FD
loss) is proposed based on Chebyshev constraint.

## Method

### Theoretically Guaranteed Chebyshev Constraint

To solve the ”non-unbiased” problem, we defined a theoretically guaranteed
chebyshev constraint.

Suppose we use $T$ predictors $H=\\{h_{1},h_{2},\cdots,h_{t}\\}$ to build the
ensemble architecture. The prediction of each predictor is
$h_{1}(x),\cdots,h_{t}(x)$, as $h_{1},\cdots,h_{t}$, and the ensemble
prediction is $\overline{H}(x)$, as $\overline{H}$. The aim of ensemble
learning is to make the ensemble prediction $\overline{H}$ as close as
possible to the optimal prediction $H^{*}(x)$, as $H^{*}$.

One of the intuitions of ensemble learning is that when the predictors are
trained synchronously using the same data set, they can be considered to be
close to the optimal prediction $H^{*}$ at a similar optimization rate.
Therefore, we assume that:

$h_{1}-H^{*}\approx\cdots,h_{2}-H^{*}\approx\overline{H}-H^{*},$ (1)

Based on (1), we can easily get:

$\mathbb{E}(H^{*})=\frac{1}{T}\sum_{t=1}^{T}h_{t}=\overline{H},$ (2)

According to Chebyshev inequality, we define that the error between ensemble
prediction $\overline{H}$ and optimal prediction $H^{*}$ can be approximately
expressed as:

$\displaystyle err$ $\displaystyle=P\\{|\overline{H}-H^{*}|\\}$ (3)
$\displaystyle=P\\{|H^{*}-\mathbb{E}(H^{*})|\geq\varepsilon\\}$
$\displaystyle\leq\frac{1}{\varepsilon^{2}}\text{var}(H^{*})$

Since $\text{var}(H^{*})\leq\text{var}(\overline{H})$, then:

$\displaystyle err$
$\displaystyle\leq\frac{1}{\varepsilon^{2}}\text{var}(\overline{H})$ (4)
$\displaystyle\leq\frac{1}{\varepsilon^{2}}\text{var}\\{\frac{\sum_{t=1}^{T}h_{t}}{T}\\}$
$\displaystyle\leq\frac{1}{\varepsilon^{2}}\frac{1}{T^{2}}\left[\underbrace{\sum_{t=1}^{T}\text{var}(h_{t})}_{\text{var}(H)}+\underbrace{\sum_{t=1}^{T}\sum_{t=1,j\neq
t}^{T}2\text{covar}(h_{t},h_{j})}_{\text{covar}(H)}\right],$

According to (4), factors affecting the ensemble prediction error include
stability of each predictor ($\text{var}(H)$) and diversity of ensemble
($\text{covar}(H)$).

(a) Stability The variance represents sensitivity of the predictor itself to
different augmentation (or perturbation) of the same sample. Small variance
implies the small prediction error, which means that each predictor needs to
be stable, which is consistent with (4).

(b) Diversity Correlations between predictors model ensemble diversity. As
shown in (4), the lower the correlation between the predictors, the smaller
the ensemble prediction error, which is consistent with the effect of
diversity.

### Overview

Figure 2: Overview of the proposed Unbiased Pseudo-labels network (UBPL
network).

We propose an Unbiased Pseudo-labels network (UBPL network) with multiple
branches to combine multiple predictions as pseudo-labels, so as to obtain the
better supervised signals from unlabeled data.

The UBPL network, shown in Fig. 2, consists of two branches with the same
structure (only the initialization strategies for the model parameters
differ). Each branch is a semi-supervised learning process. This is a general
structure, so the SSL method here can be any of the mainstream SSL methods
including Mean Teacher, FixMatch, etc.

In each branch, we input two different data augmentation samples, a hard
prediction sample generated by strong data augmentation and an easy prediction
sample generated by weak data augmentation. The output predictions of
unlabeled samples from the two branches are then filtered and combined, and
the final ensemble predictions are used as pseudo-labels to supervise the
training of the two branches separately.

In addition, considering the problem of model homogenization, we propose a
Feature Decorrelation Loss (FD Loss) based on Chebyshev constraints to
maximize diversity and maintain the difference between branches.

### Feature Decorrelation Loss

Figure 3: The structure of Feature Discrepancy loss (FD loss). To compute
feature correlations between two branches, we output features for
classification prediction. In addition, to reduce the amount of computation
and improve the computational robustness of feature correlation, we perform
Average Pooling on the output features. Then, the FD loss is used to minimize
the two corresponding features.

Based on the above discussion, we propose a Feature Decorrelation loss (FD
loss) is proposed based on Chebyshev constraint.

The features extracted by the model can be considered as salient information
discovered by the model from the input sample in order to accurately predict
this sample. If the two models find different salient information for the same
sample, one can argue that the two models have different concerns. The
difference in concerns reflects the low correlation between the two models. It
is also reflected in (4).

We use the labeled data to calculate the FD loss. Moreover we reshape the
feature ($C*H*W$), which is outputted from the AvgPool layer in the model of
each branch, to $C*(H*W)$. $C$, $H$, $W$ are the channel, height, width of
feature, respectively. Thus the FD loss $L_{fd}$ is as follows:

$L_{fd}=\frac{1}{N_{B}}\frac{1}{C}\beta_{fd}\sum_{i=1}^{N_{B}}\sum_{c=1}^{C}\text{cov}(g_{br_{1}}\big{(}\big{[}\omega(x_{i}^{l})\big{]}_{c}\big{)},g_{br_{2}}\big{(}\big{[}\omega(x_{i}^{l})\big{]}_{c}\big{)})),$
(5)

where $g_{br_{1}}$ and $g_{br_{2}}$ are the features outputted from the
AvgPool layer of models $f_{br_{1}}$ and $f_{br_{2}}$ of two branches,
respectively. $\big{[}\cdot\big{]}_{c}$ is the $c$ channel’s value of feature.
Due to the small value of the FD loss, we introduce an expansion factor
$\beta_{fd}$ to balance the FD loss with the other loss functions.

### Total Loss

So, the total loss of each branch is as follows:

$L_{br}=\lambda_{ssl}L_{ssl}+\lambda_{pse}L_{pse}+\lambda_{fd}L_{fd},$ (6)

where $L_{ssl}$ is the loss of the SSL method in the branch, which varies
depending on the SSL method used.

### SSL Method Selection

In general, we can decompose the semi-supervised method into two parts. One is
a supervised learning process based on labeled data. The other part is the
unsupervised learning process based on unlabeled d (some are also called self-
supervised learning processes). Consistent regularization methods and pseudo-
labeling methods are typical representatives of the unsupervised learning
process, and their essence is to obtain additional trusted supervised signals
from unlabeled samples to improve the training performance of semi-supervised
learning. Thus, we can express the loss function of the semi-supervised method
$L_{ssl}$ uniformly as follows:

$L_{ssl}=L_{sup}+L_{unsup},$ (7)

where $L_{sup}$ is the loss of supervised learning process, and $L_{unsup}$ is
the loss of unsupervised learning process.

In a follow-up experiment, we will combine Mean Teacher, FixMatch, DualPose,
respectively, and perform in both classification and pose estimation.

## Experiments on Pose Estimation

We perform semi-supervised pose estimation experiments to validate the
effectiveness of our method on semi-supervised regression tasks.

### Implementation Details

Method | Mouse
---|---
30/100 | 30/200
MSE |<EMAIL_ADDRESS>| MSE |<EMAIL_ADDRESS>
Supervised | 9.59 $\pm$ 0.05 | 0.264 $\pm$ 0.005 | 8.45 $\pm$ 0.12 | 0.266 $\pm$ 0.012
Mean Teacher (Tarvainen and Valpola 2017) | 4.60 $\pm$ 0.14 | 0.591 $\pm$ 0.007 | 3.92 $\pm$ 0.00 | 0.678 $\pm$ 0.000
Mean Teacher + UBPL |  4.39 $\pm$ 0.01 |  0.617 $\pm$ 0.003 |  3.65 $\pm$ 0.00 |  0.719 $\pm$ 0.001
DualPose (Xie et al. 2021) | 4.74 $\pm$ 0.00 | 0.575 $\pm$ 0.002 | 3.90 $\pm$ 0.02 | 0.683 $\pm$ 0.002
DualPose + UBPL |  4.59 $\pm$ 0.02 |  0.579 $\pm$ 0.004 |  3.75 $\pm$ 0.07 |  0.686 $\pm$ 0.001
Method | FLIC
30/100 | 30/200
MSE |<EMAIL_ADDRESS>| MSE |<EMAIL_ADDRESS>
Supervised | 50.70 $\pm$ 1.07 | 0.441 $\pm$ 0.009 | 54.03 $\pm$ 0.06 | 0.396 $\pm$ 0.001
Mean Teacher (Tarvainen and Valpola 2017) | 43.14 $\pm$ 0.04 | 0.516 $\pm$ 0.005 | 48.67 $\pm$ 0.31 | 0.449 $\pm$ 0.001
Mean Teacher + UBPL |  40.29 $\pm$ 0.28 |  0.529 $\pm$ 0.002 |  47.03 $\pm$ 0.59 |  0.450 $\pm$ 0.003
DualPose (Xie et al. 2021) | 41.89 $\pm$ 0.67 | 0.514 $\pm$ 0.009 | 46.83 $\pm$ 0.52 | 0.453 $\pm$ 0.003
DualPose + UBPL |  41.51 $\pm$ 0.08 |  0.520 $\pm$ 0.003 |  46.18 $\pm$ 0.56 |  0.454 $\pm$ 0.001
Method | LSP
100/500 | 200/500
MSE |<EMAIL_ADDRESS>| MSE |<EMAIL_ADDRESS>
Supervised | 52.22 $\pm$ 0.81 | 0.102 $\pm$ 0.001 | 47.02 $\pm$ 0.71 | 0.150 $\pm$ 0.004
Mean Teacher (Tarvainen and Valpola 2017) | 49.56 $\pm$ 0.03 |  0.146 $\pm$ 0.000 | 41.61 $\pm$ 0.54 |  0.220 $\pm$ 0.001
Mean Teacher + UBPL |  45.48 $\pm$ 0.27 | 0.130 $\pm$ 0.003 |  39.60 $\pm$ 0.01 | 0.189 $\pm$ 0.001
DualPose (Xie et al. 2021) | 46.35 $\pm$ 0.46 |  0.131 $\pm$ 0.000 |  39.81 $\pm$ 0.10 |  0.183 $\pm$ 0.001
DualPose + UBPL |  45.73 $\pm$ 0.62 | 0.129 $\pm$ 0.002 | 39.82 $\pm$ 0.07 | 0.182 $\pm$ 0.003
Table 1: The MSE and PCK of semi-supervised pose estimation on Mouse, FLIC and
LSP datasets. ’Supervised’ denotes training a single Pose Model (Stacked
Hourglass (Newell, Yang, and Deng 2016)) in supervised manner with only
labeled data. ’+ UBPL’ denotes using our UBPL network. ’30/100’ denotes the
data count is 100 with 30 labeled data.

#### Dataset

Our experiments are mainly performed on the following three datasets: FLIC
(Sapp and Taskar 2013), LSP (Johnson and Everingham 2011) dataset, and our own
collected Mouse dataset. FLIC and LSP datasets are open-source human pose
datasets commonly used in human pose estimation experiments. Moreover, to
present our approach from more perspectives, we introduce the Mouse dataset.
It is experimental data on the sniffing movements of laboratory mice captured
in a realistic experimental environment.

#### Details

Figure 4: Error and accuracy curves for ’MT’ and ’MT+UBPL’ on the Mouse
(30/200).

We conduct experiments on the Mouse, FLIC and LSP dataset and extensively
study the training performance under various labeled data amounts. In all
experiments, the pose model is Stacked Hourglass (Newell, Yang, and Deng
2016).

For fair comparison, the same hyper-parameters are used in all experiments,
except that the data augmentation varies depending on the semi-supervised
method used. Specifically, all experiments are performed with an Adam with
weight decay 0.The learning rate is 0.00025. The batch size is 4 with 2
labeled data. The confidence threshold $\tau$ is 0.95. The decay value for the
Exponential Moving Average (EMA) (Tarvainen and Valpola 2017) is 0.999.

To verify the universality of UBPL, we conduct semi-supervised pose estimation
experiments using both Mean Teacher and DualPose as SSL methods.

In terms of data augmentation, Mean Teacher generates two augmented sample
pairs with similar prediction difficulty. The data enhancement methods used
include: random rotation ($+/-30$ degrees), random scaling ($0.75-1.25$) and
random horizontal flip.

And, DualPose uses two data enhancement strategies of varying strength to
generate a set of sample pairs with varying prediction difficulty. The strong
data augmentation include: random rotation ($+/-30$ degrees), random scaling
($0.75-1.25$) and random horizontal flip. And the weak data augmentation
include: random rotation ($+/-5$ degrees), random scaling ($0.95-1.05$) and
random horizontal flip.

$\lambda_{ssl}$ is 10, $\lambda_{pse}$ is 10. $L_{fd}$ is 1.

Due to hardware limitations, all experiments were trained for 100 epochs. We
used a fixed random seed (1388) to obtain accurate and reliable experimental
results.

### Comparison with Existing Methods

The MSE and PCK of our method on Mouse, FLIC, LSP datasets are shown in Table.
1. The results show that our approach achieves better results on most of the
baseline datasets. This is especially the case when the total sample count is
small. These results show that our approach improves the performance of
existing semi-supervised methods by making towards unbiased predictions to
provide more accurate and informative pseudo-labels.

As shown in Fig. 4, our UBPL performs better throughout the training period in
the pose estimation task, indicating that UBPL is towards unbiased and can
generate better pseudo-labels.

### Ablation Studies

Method | Mouse (30/100)
---|---
MSE |<EMAIL_ADDRESS>
Supervised | 9.59 $\pm$ 0.05 | 0.264 $\pm$ 0.005
Mean Teacher | 4.60 $\pm$ 0.14 | 0.591 $\pm$ 0.007
Mean Teacher + UBPL (noFDL) | 4.55 $\pm$ 0.03 | 0.608 $\pm$ 0.000
Mean Teacher + UBPL |  4.39 $\pm$ 0.01 |  0.617 $\pm$ 0.003
DualPose | 4.74 $\pm$ 0.00 | 0.575 $\pm$ 0.002
DualPose + UBPL (noFDL) | 4.63 $\pm$ 0.03 | 0.576 $\pm$ 0.003
DualPose + UBPL |  4.59 $\pm$ 0.02 |  0.579 $\pm$ 0.004
Method | Mouse (30/200)
MSE |<EMAIL_ADDRESS>
Supervised | 8.45 $\pm$ 0.12 | 0.266 $\pm$ 0.012
Mean Teacher | 3.92 $\pm$ 0.00 | 0.678 $\pm$ 0.000
Mean Teacher + UBPL (noFDL) | 3.63 $\pm$ 0.02 | 0.715 $\pm$ 0.004
Mean Teacher + UBPL |  3.65 $\pm$ 0.00 |  0.719 $\pm$ 0.001
DualPose | 3.90 $\pm$ 0.02 | 0.683 $\pm$ 0.002
DualPose + UBPL (noFDL) | 3.81 $\pm$ 0.03 | 0.684 $\pm$ 0.001
DualPose + UBPL |  3.75 $\pm$ 0.07 |  0.686 $\pm$ 0.001
Table 2: Results of the ablation experiment in pose estimation. Where, ’UBPL
(noFDL)’ means using our UBPL network and not using the Feature Decorrelation
Loss (FD Loss).

The ablation results of our method on the semi-supervised pose estimation task
are shown in Table. 2. Without the use of Feature Decorrelation Loss (FD
loss), UBPL can effectively improve the performance of semi-supervised
methods. The results show that our approach can provide more accurate and
informative pseudo-labels. When FD loss is used, the performance of the semi-
supervised method is further improved. It shows that FD loss effectively
improves diversity and makes the pseudo-labels generated by ensemble
prediction more accurate.

## Experiments on Classification

Further, we apply our method to semi-supervised classification tasks to
validate the generality of our method.

### Implementation Details

Figure 5: Error rate curve for ’FixMatch’ and ’FixMatch+UBPL’ on the CIFAR10 (40 labels). Method | CIFAR10 | CIFAR100 | SVHN
---|---|---|---
40 | 250 | 4000 | 400 | 2500 | 10000 | 40 | 250 | 1000
FixMatch (Sohn et al. 2020) | 43.69 | 12.21 | 10.28 | 81.15 | 46.42 | 32.20 | 89.71 | 79.94 | 77.86
FixMatch + UBPL | 27.27 | 11.44 | 9.29 | 80.88 | 44.96 | 31.37 | 90.00 | 79.00 | 78.36
Table 3: The error rate of semi-supervised classification on CIFAR10, CIFAR100
and SVHN datasets. ’Mean Teacher’ and ’FixMatch’ denote the use of them only,
respectively. ’+ UBPL’ denotes using our UBPL network.

We conduct experiments on the CIFAR-10/100 (Krizhevsky, Hinton et al. 2009)
dataset and SVHN dataset and extensively study the training performance under
various labeled data amounts. In all experiments, the classification model is
Wide ResNet (WRN) (Zagoruyko and Komodakis 2016). On the CIFAR-10 dataset, the
widen-factor of WRN is 2. On the CIFAR-100 dataset, due to hardware
limitations, the widen-factor of WRN is 6.

For a fair comparison, the same hyper-parameters are used for all experiments.
Specifically, all experiments are performed with an standard Stochastic
Gradient Descent (SGD) optimizer with momentum 0.9 (Sutskever et al. 2013;
Polyak 1964) and Nesterov momentum (Dozat 2016) enabled.The learning rate is
0.03. The batch size of labeled data is 32, the batch size of labeled data is
224 (i.e., $\mu=7$). The confidence threshold $\tau$ is 0.95. We used the same
decay value (0.999) for the experiment involving the Exponential Moving
Average (EMA) (Tarvainen and Valpola 2017). Data augmentation for labeled
samples consists of random horizontal flips and random cropping. Weak data
augmentation for unlabeled samples includes random horizontal flipping and
random clipping. Strong data augmentation adds RandAugment (Cubuk et al. 2020)
to weak data augmentation, where n is 2 and m is 10.

$\lambda_{ssl}$ is 10, $\lambda_{pse}$ is 10. $L_{fd}$ is 1000.

Due to hardware limitations, all experiments were trained for 50 epochs with
1024 iterations within each epoch. We used a fixed random seed (1388) to
obtain accurate and reliable experimental results.

### Comparison with Existing Methods

The classification error rate on the CIFAR-10/100 and SVHN dataset is shown in
Table 3. The results show that our UBPL achieve better performance than
FixMatch on most benchmark datasets. Our approach achieved better performance
for tasks with extremely limited marker data (on CIFAR10 with 40 labeled data,
on CIFAR100 with 400 labeled data).

As shown in Fig. 5, our UBPL performs better throughout the training period in
the classification task, indicating that UBPL is towards unbiased and can
generate better pseudo-labels.

### Ablation Studies

Method | CIFAR10
---|---
40 | 250 | 4000
FixMatch (Sohn et al. 2020) | 43.69 | 12.21 | 10.28
FixMatch + UBPL (noFDL) | 30.06 | 11.68 | 9.43
FixMatch + UBPL | 27.27 | 11.43 | 9.21
Method | CIFAR100
400 | 2500 | 10000
FixMatch | 81.15 | 46.42 | 32.20
FixMatch + UBPL (noFDL) | 81.03 | 45.21 | 31.32
FixMatch + UBPL | 80.85 | 44.93 | 30.93
Table 4: Results of the ablation experiment in classification. Where, ’UBPL
(noFDL)’ means using our UBPL network and not using the Feature Decorrelation
Loss (FD Loss).

The ablation results of the proposed method on the semi-supervised
classification task are shown in Table. 4. Without using FD loss, UBPL can
improve the performance of semi-supervised methods to some extent. Moreover,
the performance of the semi-supervised method is further improved when FD loss
is introduced. It is shown that FD loss effectively improves the performance.

## Limitations

Although our method performs better throughout the training period in both
classification and regression tasks, it still suffers from shortcomings.
Multi-model ensemble training is slow and consumes a lot of resources, which
has certain limitations in practical applications. In the future, we will
focus on how to improve training efficiency and reduce resource footprint.

## Conclusion

In this paper, we first discuss the T2L Dilemma that arises when semi-
supervised classification methods are directly applied to regression tasks.
Then, we propose a theoretically guaranteed constraint for generating towards
unbiased labels based on Chebyshev’s inequality, combining multiple
predictions to generate superior quality labels from several inferior ones.
Specially, we propose an Unbiased Pseudo-labels network (UBPL network) with
multiple branches to combine multiple predictions as pseudo-labels, where a
Feature Decorrelation loss (FD loss) is proposed based on Chebyshev
constraint. We conduct extensive experiments to validate the effectiveness of
our method and show that it can be applied to both semi-supervised
classification and regression. Our approach can be easily extended to any
semi-supervised framework such as Mean Teacher, FixMatch, DualPose.

## References

  * Bachman, Alsharif, and Precup (2014) Bachman, P.; Alsharif, O.; and Precup, D. 2014.  Learning with pseudo-ensembles.  _Advances in neural information processing systems_ , 27.
  * Cubuk et al. (2020) Cubuk, E. D.; Zoph, B.; Shlens, J.; and Le, Q. V. 2020.  Randaugment: Practical automated data augmentation with a reduced search space.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , 702–703.
  * Dozat (2016) Dozat, T. 2016.  Incorporating nesterov momentum into adam.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Johnson and Everingham (2011) Johnson, S.; and Everingham, M. 2011.  Learning effective human pose estimation from inaccurate annotation.  In _CVPR 2011_ , 1465–1472. IEEE.
  * Ke et al. (2019) Ke, Z.; Wang, D.; Yan, Q.; Ren, J.; and Lau, R. W. 2019.  Dual student: Breaking the limits of the teacher in semi-supervised learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 6728–6736.
  * Krizhevsky, Hinton et al. (2009) Krizhevsky, A.; Hinton, G.; et al. 2009.  Learning multiple layers of features from tiny images.
  * Laine and Aila (2016) Laine, S.; and Aila, T. 2016.  Temporal ensembling for semi-supervised learning.  _arXiv preprint arXiv:1610.02242_.
  * Miyato et al. (2018) Miyato, T.; Maeda, S.-i.; Koyama, M.; and Ishii, S. 2018.  Virtual adversarial training: a regularization method for supervised and semi-supervised learning.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(8): 1979–1993.
  * Netzer et al. (2011) Netzer, Y.; Wang, T.; Coates, A.; Bissacco, A.; Wu, B.; and Ng, A. Y. 2011.  Reading digits in natural images with unsupervised feature learning.
  * Newell, Yang, and Deng (2016) Newell, A.; Yang, K.; and Deng, J. 2016.  Stacked hourglass networks for human pose estimation.  In _European conference on computer vision_ , 483–499.
  * Polyak (1964) Polyak, B. T. 1964.  Some methods of speeding up the convergence of iteration methods.  _Ussr computational mathematics and mathematical physics_ , 4(5): 1–17.
  * Sajjadi, Javanmardi, and Tasdizen (2016) Sajjadi, M.; Javanmardi, M.; and Tasdizen, T. 2016.  Regularization with stochastic transformations and perturbations for deep semi-supervised learning.  _Advances in neural information processing systems_ , 29.
  * Sapp and Taskar (2013) Sapp, B.; and Taskar, B. 2013.  Modec: Multimodal decomposable models for human pose estimation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 3674–3681.
  * Sohn et al. (2020) Sohn, K.; Berthelot, D.; Carlini, N.; Zhang, Z.; Zhang, H.; Raffel, C. A.; Cubuk, E. D.; Kurakin, A.; and Li, C.-L. 2020.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  _Advances in neural information processing systems_ , 33: 596–608.
  * Sun et al. (2019) Sun, K.; Xiao, B.; Liu, D.; and Wang, J. 2019.  Deep high-resolution representation learning for human pose estimation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 5693–5703.
  * Sutskever et al. (2013) Sutskever, I.; Martens, J.; Dahl, G.; and Hinton, G. 2013.  On the importance of initialization and momentum in deep learning.  In _International conference on machine learning_ , 1139–1147. PMLR.
  * Tang et al. (2021) Tang, Y.; Chen, W.; Luo, Y.; and Zhang, Y. 2021.  Humble teachers teach better students for semi-supervised object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 3132–3141.
  * Tarvainen and Valpola (2017) Tarvainen, A.; and Valpola, H. 2017.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  _Advances in neural information processing systems_ , 30.
  * Wang, Wang, and Yuille (2013) Wang, C.; Wang, Y.; and Yuille, A. L. 2013.  An approach to pose-based action recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 915–922.
  * Wang, Wang, and Yuille (2016) Wang, C.; Wang, Y.; and Yuille, A. L. 2016.  Mining 3d key-pose-motifs for action recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2639–2647.
  * Wang et al. (2022) Wang, Y.; Chen, H.; Heng, Q.; Hou, W.; Fan, Y.; Wu, Z.; Wang, J.; Savvides, M.; Shinozaki, T.; Raj, B.; et al. 2022.  Freematch: Self-adaptive thresholding for semi-supervised learning.  _arXiv preprint arXiv:2205.07246_.
  * Xie et al. (2020) Xie, Q.; Luong, M.-T.; Hovy, E.; and Le, Q. V. 2020.  Self-training with noisy student improves imagenet classification.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 10687–10698.
  * Xie et al. (2021) Xie, R.; Wang, C.; Zeng, W.; and Wang, Y. 2021.  An empirical study of the collapsing problem in semi-supervised 2d human pose estimation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 11240–11249.
  * Zagoruyko and Komodakis (2016) Zagoruyko, S.; and Komodakis, N. 2016.  Wide residual networks.  _arXiv preprint arXiv:1605.07146_.
  * Zhang et al. (2021) Zhang, B.; Wang, Y.; Hou, W.; Wu, H.; Wang, J.; Okumura, M.; and Shinozaki, T. 2021\.  Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling.  _Advances in Neural Information Processing Systems_ , 34.


# Opening Amyloid-Windows to the Secondary Structure of Proteins: The
Amyloidogenecity Increases Tenfold Inside Beta-Sheets

Kristóf Takács<EMAIL_ADDRESS>Bálint Varga<EMAIL_ADDRESS>Viktor Farkas<EMAIL_ADDRESS>András Perczel
<EMAIL_ADDRESS>Vince Grolmusz<EMAIL_ADDRESS>PIT
Bioinformatics Group, Eötvös University, H-1117 Budapest, Hungary  Uratim
Ltd., H-1118 Budapest, Hungary  ELKH-ELTE Protein Modeling Research Group,
H-1117 Budapest, Hungary  Laboratory of Structural Chemistry and Biology,
Eötvös University, H-1117, Budapest, Hungary

###### Abstract

Methods from artificial intelligence (AI), in general, and machine learning,
in particular, have kept conquering new territories in numerous areas of
science. Most of the applications of these techniques are restricted to the
classification of large data sets, but new scientific knowledge can seldom be
inferred from these tools. Here we show that an AI-based amyloidogenecity
predictor can strongly differentiate the border- and the internal hexamers of
$\beta$-pleated sheets when screening all the Protein Data Bank-deposited
homology-filtered protein structures. Our main result shows that more than 30%
of internal hexamers of $\beta$ sheets are predicted to be amyloidogenic,
while just outside the border regions, only 3% are predicted as such. This
result may elucidate a general protection mechanism of proteins against
turning into amyloids: if the borders of $\beta$-sheets were amyloidogenic,
then the whole $\beta$ sheet could turn more easily into an insoluble amyloid-
structure, characterized by periodically repeated parallel $\beta$-sheets. We
also present that no analogous phenomenon exists on the borders of
$\alpha$-helices or randomly chosen subsequences of the studied protein
structures.

## Introduction

The amyloid status of proteins are studied for a long time because of its
relevance in human diseases, including neurodegenerative ailments [1, 2, 3,
4], amyloidoses [5] and other diseases, listed in Table 1 of [2]. Recently,
the amyloids gained increased attention as possible antiviral agents [6], or
in one of the first successes in the human in vivo CRISPR-Cas9-based gene
editing therapy of the transthyretin amyloidosis [7].

The insoluble amyloid state differs from the unstructured protein aggregates;
it has a well-defined structure characterized by repetitive parallel
$\beta$-sheets [8, 9].

Numerous globular proteins may turn into insoluble amyloid structures in
certain pH and temperature combinations. It is observed that the transition to
the amyloid state is similar to the contagious prion-formation: it is related
to a core-forming structure [2] for starting the amyloid-transition, and the
subsequent propagation of misfolded, insoluble, parallel $\beta$-sheets [10,
11]. The amyloid propagation is related to the spatially accessible,
neighboring $\beta$-pleated sheet secondary structural elements of proteins
[12, 3, 4].

Naturally occurring, globular, soluble proteins need to possess certain
structural properties which prevent them from turning into insoluble amyloid
aggregates. Experimental studies, together with in silico methods able to
predict aggregation-prone regions (APRs) in protein sequences. In the
literature, certain residues acting as “gatekeepers” are described as
preventing those conformational changes to amyloid structures: aspartic acid
and glycine in bacterial curlin subunits CsgA and CsgB [13], lysine in
transthyretin [14] or in peptides Trpzip1 and Trpzip2 [15].

Our hypothesis is that the borders of $\beta$-sheets in globular, soluble
proteins in general (and not only in the specific examples listed above) need
to be protected against turning to amyloid structures; otherwise the whole
polypeptide chain bordering the $\beta$-sheet subsequences would be
transformed to amyloids. We examine this hypothesis in the present work by
screening the starting and ending subsequences, or in other terminology, the
prefixes, and suffixes of the $\beta$-sheets of the homology-filtered Protein
Data Bank (PDB) entries.

### Previous work

In [8], a clean geometrical formulation was given for the characterization of
the amyloid-like structures in length, distance, and parallelism of their
$\beta$-sheets, and by the search of these geometric properties, an
automatically updated list of the PDB-entries was published and maintained at
the address https://pitgroup.org/amyloid/.

In [9], the amyloid-precursor molecules of https://pitgroup.org/amyloid/ were
classified by the prefixes of their parallel $\beta$-sheet subsequences, and
these prefixes were also searched for in the whole Protein Data Bank [16]. The
remarkable findings include the major histocompatibility complexes MHC-1 and
MHC-2, the p53 tumor suppressor protein, and an anti-coagulant peptide
molecule.

With a prediction tool, based on artificial intelligence, in [17], we
partitioned all the possible hexapeptides into two classes: “amyloidogenic”
and “non-amyloidogenic” with more than 84% correctness. The prediction tool,
which applies a linear Support Vector Machine (SVM) [18], was trained by the
Waltz dataset of 514 amyloidogenic and 901 non-amyloidogenic hexapeptides [19,
20]. The corresponding web-based tool is available as the Budapest Amyloid
Predictor at https://pitgroup.org/bap.

As a surprising application of the Budapest Amyloid Predictor, numerous
amyloid- and non-amyloid hexapeptide patterns were identified in [21]. For
example, it was shown that for all independent amino acid substitutions for
the positions marked by “x”, the CxFLWx or FxFLFx patterns describe
amyloidogenic hexamers, while the patterns PxDxxx or xxKxEx describe non-
amyloidogenic hexamers. Note that each pattern with two x’s describes
$20^{2}=400$, while those with four x’s describe $20^{4}=160,000$ different
hexapeptides succinctly if for the positions x, we can substitute the 20 amino
acid residues.

In the present work, we study the prefixes and suffixes of the $\beta$-sheet
subsequences of the PDB-deposited structures and apply the Budapest Amyloid
Predictor to the hexamers of the border regions on these $\beta$-sheets. As we
show below, the number of the amyloidogenic hexapeptides of the border regions
of the $\beta$-sheets is one-tenth of the same number in the inner parts of
those sequences by screening the homology-filtered Protein Data Bank entries.
We believe that this finding is a very strong evidence for our hypothesis.

We remark that applying the artificial intelligence-based Budapest Amyloid
Predictor for the 64 million ($=20^{6}$) possible hexapeptides is a crucial
step here: the largest experimental dataset that labels the hexapeptides by
their amyloidogenic propensity, the Waltz dataset, contains only 1415
hexapeptides [19, 20], while we analyzed the prefixes and suffixes of more
than 110,000 $\beta$-sheets from the homology-filtered PDB. Gaining
experimental data for that many hexapeptides is intractable today.

## Methods

For the statistical analysis involving the entries of the Protein Data Bank
(PDB) [16], usually some “non-redundant” homology-filtered PDB-subsets are
applied because otherwise, the multiple depositions of more important or more
researched protein structures (e.g., several thousand copies of the HIV-1
protease) would impact the results. In this study, we have made use of the
representative polypeptide chains of the 30% homology-filtered set, which was
available at the URL https://www.rcsb.org/pdb/rest/representatives?cluster=30
(downloaded on May 21, 2020).

Technical note for reproducibility of the present work: At the time of writing
this section, we were informed that the administrators of the RCSB PDB have
removed the link target indicated above, but one can still generate the
similar non-redundant set with an advanced search feature of the RCSB PDB,
namely https://search.rcsb.org/#group-by-context, by choosing 30% sequence
identity.

The structure of almost all representative members of this non-redundant set
was identified in a soluble state, so almost all proteins in the set are
water-soluble (see Figure S3 in the supporting material).

From the representative sequences of the non-redundant set, we have identified
the maximal contiguous subsequences corresponding to $\beta$-sheets by
applying the “SHEET” records in the PDB file. In other words, all the non-
expandable contiguous $\beta$-sheet subsequences were identified. Those
subsequences whose lengths are less than 6 were discarded. This way, we have
identified as many as 117,467 subsequences.

In the next step, we listed the six-residue long sliding windows through the
borders of these 117,467 $\beta$-sheet subsequences, as it is visualized in
Figure 1.

Figure 1: A visualization of the sliding prefix windows of $\beta$-sheet
units, mapping their border regions. A section of the polypeptide sequence of
a protein is drawn at the top of the figure, listed in the standard N-terminal
through C-terminal order. The green shaded subsequence represents the
$\beta$-sheet region, identified from the “SHEET” record of the PDB file. The
rows, annotated by $k=0,1,\ldots,6$, correspond to the sliding windows of
length 6 residues, when the window slides to the left, and in row, labeled by
$k=i$, exactly $i$ residues are outside the green-shaded $\beta$-sheet region,
for $i=0,1,2,\ldots,6$. In the case of sliding suffix windows (Figure S1 in
the supporting material), the windows move to the right, and, similarly to the
prefix case, $k=0$ corresponds to the window entirely inside the right end of
the green subsequence, and $k=6$ to the configuration where the length-6
window is just outside to the right of the green sequence in its entirety.

The six residue-long windows map the border regions of the $\beta$-sheets,
identified from the “SHEET” record of the PDB file. For each contiguous
$\beta$-sheet section, we consider the sliding windows of its left border
(Figure 1), called prefixes, and the seven sliding windows at the right
border, called suffixes. That is, for each of the 117,467 unique subsequences,
we have listed the seven prefix and the seven suffix sequences (for
$k=0,1,2,3,4,5,6$), and applied the Budapest Amyloid Predictor [17] at
https://pitgroup.org/bap for assigning one of the “amyloidogenic” or “non-
amyloidogenic” labels for each of the seven length-6 windows at both ends of
the 117,467 subsequences.

Next, we have drawn a diagram, Figure 2, which contains at the $x$ axis the
$k$ value (i.e., the length of the window over (or outside) the $\beta$-sheet
sequence (colored by green in Figure 1). On the $y$ axis, we give the
percentage of the predicted amyloidogenic length-6 windows, computed by the
Budapest Amyloid Predictor. In other words, the $y$ value gives the fraction
of the hexapeptides with exactly $k$ residues outside the $\beta$-sheet
region, predicted to be amyloidogenic.

Figure 2: The percentage of the length-6 prefix windows (left panel) and
suffix windows (right panel) with $k$ positions outside the $\beta$-sheet
region, predicted to be amyloidogenic by the Budapest Amyloid Predictor. The
$x$ axis shows the $k$ value (i.e., the length of the window over (or outside)
the $\beta$-sheet sequence (colored by green in Figure 1 and S1, resp.), and
on the $y$ axis, the percentage of the amyloidogenic length-6 windows are
given. In the left panel, the minimum value is 2.54% at $k=4$, at $k=6$ the
percentage is 2.89, the maximum is taken at $k=0$ with 32.9%. In the right
panel, the minimum value is 1.79% at $k=5$, at $k=6$ the percentage is 2.81,
the maximum is taken at $k=0$ with 32.4%. Both graphs show that by crossing
the borders of the $\beta$-sheet-regions, the amyloidocity percentage of the
hexapeptide windows drops to less than 1/10 of the starting value. Or, in
other words, more than 10 times more hexapeptide windows are amyloidogenic
just inside the $\beta$-sheet region than just outside of it.

## Discussion and results

The rapid development of artificial intelligence tools may open up new
insights into the structural properties of biological macromolecules. For
example, the largest available hexapeptide database, the Waltz dataset [19,
20] today contains 1415 hexapeptides, labeled as “amyloidogenic” or “non-
amyloidogenic”, based on different types of experimental data. The total
number of all the hexapeptides from the biologically most relevant 20 amino
acids is $20^{6}=64000000$. Therefore, it is improbable that experimental
evidence could be gained for the amyloid-forming properties of all of these
hexapeptides.

By artificial intelligence tools, however, we were able to build the Budapest
Amyloid Predictor, which predicts the amyloidocity of any hexapeptide from
these 64 million with an 84% success rate [17], https://pitgroup.org/bap.

The BAP predictor makes it possible to assign amyloidogenecity labels to any
hexapeptides, not only the experimentally identified ones. Consequently, we
may screen all the given secondary structures deposited in the PDB and compute
the prediction for any hexapeptides in them. These actions were not possible
without highly developed artificial intelligence tools.

Our results are summarized in Figure 2.

Figure 2 shows a monotonic tenfold increase of the amyloidogenecity of the
hexapeptides, described by the sliding windows, for positions entirely in the
$\beta$-sheet sequences ($k=0$), relative to the other positions, when the
largest part of the windows are outside of the $\beta$-sheet region $k=4,5,6$.
This observation validates our hypothesis, described in the Introduction,
namely, the prefixes and the suffixes of the $\beta$-sheet subsequences need
to be protected against turning into amyloid structures Otherwise, the whole
polypeptide chain bordering the $\beta$-sheet subsequences would be
transformed to amyloids.

### The $\alpha$-helix control

For a simple control of our hypothesis, we also prepared the sliding prefix-
and suffix windows for the $\alpha$-helices, found by screening all the
homology-filtered PDB entries. As the supporting Figure S2 shows in the
Supporting Material, no similar observation can be done in the case of the
$\alpha$-helices.

### The random control

For another simple control, let us consider random (contiguous) subsequences
of the PDB-deposited proteins instead of subsequences of $\beta$-sheets or
$\alpha$-helices and evaluate their prefixes or suffixes for amyloidogenecity.
The following reasoning shows that in this case, the amylodocity percentage of
prefixes or suffixes of different $k$ values are almost the same (the “almost”
adjective is explained below). We consider prefixes here; the proof for
suffixes is analogous.

We refer to Figure S4 in the Supporting Material for visualizing the
reasoning.

First, assume that we consider only fixed-length subsequences (of length
$\ell\geq 6$). Let us consider a hexapeptide, which is not too close to the
right end of the protein sequence, more exactly, whose first position is at
least 6+$\ell$ residues from the right end (denoted by yellow in the first row
of Fig. S4).

It is easy to see that the very same (yellow) hexapeptide may be a prefix-
window with $k=0$ for the green, with $k=1$ for the blue, with $k=2$ for the
red, or with $k=3$ for the brown subsequence of length $\ell$. That is, if a
hexapeptide is not too close to the right end of a protein chain, then it
appears with $k=1,2,3,4,5,6$ for different length $\ell$ sequences. Let us
remark that the amyloidocity of a hexapeptide depends only on its sequence, so
there would not be any difference of prefix-amyloidocities for different $k$
values, unlike in the case of $\beta$-sheets in Fig. 2. This reasoning works
also for random subsequences of different lengths, and also for suffixes.
Small variability of the appearance percentage of a given hexapeptide may
occur only if it is close to the right end of the protein-sequence and,
consequently, cannot appear with larger $k$ or $\ell$ values.

## Conclusions

Here we screened all the prefix- and suffix sequences of length-6 of the
$\beta$-sheets in the homology-filtered Protein Data Bank and have found that
by crossing the boundaries of the $\beta$-sheet sequences the amyloidogenic
ratio of the hexapeptides, characterized by the length-6 sliding windows,
moving from outside to inside of the $\beta$-sheet sequence, increases more
than tenfolds.

No similar phenomenon can be observed in the case of $\alpha$-helices or
random subsequences. We strongly believe that this fact shows the validity of
our hypothesis, namely, the prefixes and the suffixes of the $\beta$-sheet
subsequences need to be strongly protected against turning into amyloid
structures, otherwise the whole polypeptide chain, bordering the $\beta$-sheet
subsequences would possibly be transformed into amyloids.

## Data availability

The Budapest Amyloid Predictor webserver is available freely at
https://pitgroup.org/bap.

## Funding

KT, BV, and VG were partially supported by the NKFI-127909 grant of the
National Research, Development and Innovation Office of Hungary. KT, BV, AP,
VF, and VG were partially supported by the ELTE Thematic Excellence Programme
(Szint+) subsidized by the Hungarian Ministry for Innovation and Technology.

## Author Contribution

AP, VF, and VG initiated the study and evaluated results; KT collected &
identified the sequences to be analyzed, generated the sliding windows and
analyzed the results, created the figures, and provided the control studies;
BV assigned the amyloidogenecity labels to hexapeptides, VG has overseen the
work and wrote the first version of the paper. AP, VF, and VG secured funding
& improved the manuscript.

## Conflicting interest

The authors declare no conflicting interests.

## References

  * Nelson et al. [2005] Rebecca Nelson, Michael R Sawaya, Melinda Balbirnie, Anders Ø Madsen, Christian Riekel, Robert Grothe, and David Eisenberg.  Structure of the cross-$\beta$ spine of amyloid-like fibrils.  _Nature_ , 435(7043):773, 2005.
  * Eisenberg and Jucker [2012] David Eisenberg and Mathias Jucker.  The amyloid state of proteins in human diseases.  _Cell_ , 148(6):1188–1203, 2012.
  * Horvath et al. [2019] Daniel Horvath, Dora Menyhard, and Andras Perczel.  Protein aggregation in a nutshell: The splendid molecular architecture of the dreaded amyloid fibrils.  _Current Protein and Peptide Science_ , 20(11):1077–1088, 2019.
  * Taricska et al. [2020] Nora Taricska, Daniel Horvath, Dora K Menyhard, Hanna Akontz-Kiss, Masahiro Noji, Masatomo So, Yuji Goto, Toshimichi Fujiwara, and Andras Perczel.  The route from the folded to the amyloid state: Exploring the potential energy surface of a drug-like miniprotein.  _Chemistry A European Journal_ , 26:1968–1978, February 2020.  ISSN 1521-3765.  doi: 10.1002/chem.201903826.  URL https://doi.org/10.1002/chem.201903826.
  * Hazenberg [2013] Bouke P. C. Hazenberg.  Amyloidosis:: A clinical overview.  _RHEUMATIC DISEASE CLINICS OF NORTH AMERICA_ , 39(2):323–345, May 2013.  ISSN 0889-857X.  doi: 10.1016/j.rdc.2013.02.012.
  * Michiels et al. [2020] Emiel Michiels, Kenny Roose, Rodrigo Gallardo, Ladan Khodaparast, Laleh Khodaparast, Rob van der Kant, Maxime Siemons, Bert Houben, Meine Ramakers, Hannah Wilkinson, Patricia Guerreiro, Nikolaos Louros, Suzanne J F Kaptein, Lorena Itati Ibanez, Anouk Smet, Pieter Baatsen, Shu Liu, Ina Vorberg, Guy Bormans, Johan Neyts, Xavier Saelens, Frederic Rousseau, and Joost Schymkowitz.  Reverse engineering synthetic antiviral amyloids.  _Nature Communications_ , 11:2832, June 2020.  ISSN 2041-1723.  doi: 10.1038/s41467-020-16721-8.
  * Gillmore et al. [2021] Julian D. Gillmore, Ed Gane, Jorg Taubel, Justin Kao, Marianna Fontana, Michael L. Maitland, Jessica Seitzer, Daniel O’Connell, Kathryn R. Walsh, Kristy Wood, Jonathan Phillips, Yuanxin Xu, Adam Amaral, Adam P. Boyd, Jeffrey E. Cehelsky, Mark D. McKee, Andrew Schiermeier, Olivier Harari, Andrew Murphy, Christos A. Kyratsous, Brian Zambrowicz, Randy Soltys, David E. Gutstein, John Leonard, Laura Sepp-Lorenzino, and David Lebwohl.  Crispr-cas9 in vivo gene editing for transthyretin amyloidosis.  _New England Journal of Medicine_ , 385(0):493–502, 2021.  doi: 10.1056/NEJMoa2107454.  URL https://doi.org/10.1056/NEJMoa2107454.
  * Takács et al. [2019] Kristóf Takács, Bálint Varga, and Vince Grolmusz.  Pdb _amyloid: an extended live amyloid structure list from the PDB.  _FEBS Open Bio_ , 9(1):185–190, 2019.
  * Takacs and Grolmusz [2021] Kristof Takacs and Vince Grolmusz.  On the border of the amyloidogenic sequences: Prefix analysis of the parallel beta sheets in the PDB_Amyloid collection.  _Journal of Integrative Bioinformatics_ , 2021.  URL https://doi.org/10.1515/jib-2020-0043.
  * Ban et al. [2006] Tadato Ban, Keiichi Yamaguchi, and Yuji Goto.  Direct observation of amyloid fibril growth, propagation, and adaptation.  _Accounts of chemical research_ , 39:663–670, September 2006.  ISSN 0001-4842.  doi: 10.1021/ar050074l.
  * Knowles et al. [2011] Tuomas P. J. Knowles, Duncan A. White, Adam R. Abate, Jeremy J. Agresti, Samuel I. A. Cohen, Ralph A. Sperling, Erwin J. De Genst, Christopher M. Dobson, and David A. Weitz.  Observation of spatial propagation of amyloid assembly from single nuclei.  _Proceedings of the National Academy of Sciences of the United States of America_ , 108:14746–14751, September 2011.  ISSN 1091-6490.  doi: 10.1073/pnas.1105555108.
  * Chiti and Dobson [2017] Fabrizio Chiti and Christopher M. Dobson.  Protein misfolding, amyloid formation, and human disease: A summary of progress over the last decade.  _Annual review of biochemistry_ , 86:27–68, June 2017.  ISSN 1545-4509.  doi: 10.1146/annurev-biochem-061516-045115.
  * Wang et al. [2010] Xuan Wang, Yizhou Zhou, Juan-Jie Ren, Neal D Hammer, and Matthew R Chapman.  Gatekeeper residues in the major curlin subunit modulate bacterial amyloid fiber biogenesis.  _Proceedings of the National Academy of Sciences_ , 107(1):163–168, 2010.
  * Sant’Anna et al. [2014] Ricardo Sant’Anna, Carolina Braga, Nathalia Varejão, Karinne M Pimenta, Ricardo Graña-Montes, Aline Alves, Juliana Cortines, Yraima Cordeiro, Salvador Ventura, and Debora Foguel.  The importance of a gatekeeper residue on the aggregation of transthyretin.  _Journal of Biological Chemistry_ , 289(41):28324–28337, 2014.
  * Markiewicz et al. [2014] Beatrice N Markiewicz, Rolando Oyola, Deguo Du, and Feng Gai.  Aggregation gatekeeper and controlled assembly of trpzip $\beta$-hairpins.  _Biochemistry_ , 53(7):1146–1154, 2014.
  * Berman et al. [2000] H.M. Berman, J. Westbrook, Z. Feng, G. Gilliland, T.N. Bhat, H. Weissig, I.N. Shindyalov, and P.E. Bourne.  The Protein Data Bank.  _Nucleic Acids Research_ , 28:235–242, 2000.
  * [17] Laszlo Keresztes, Evelin Szogi, Balint Varga, Viktor Farkas, Andras Perczel, and Vince Grolmusz.  The Budapest Amyloid Predictor and its applications.  _Biomolecules_ , 11(4):500.  URL https://doi.org/10.3390/biom11040500.
  * Cortes and Vapnik [1995] Corinna Cortes and Vladimir Vapnik.  Support-vector networks.  _Machine Learning_ , 20(3):273–297, 1995.
  * Beerten et al. [2015] Jacinte Beerten, Joost Van Durme, Rodrigo Gallardo, Emidio Capriotti, Louise Serpell, Frederic Rousseau, and Joost Schymkowitz.  WALTZ-DB: a benchmark database of amyloidogenic hexapeptides.  _Bioinformatics (Oxford, England)_ , 31:1698–1700, May 2015\.  ISSN 1367-4811.  doi: 10.1093/bioinformatics/btv027.
  * Louros et al. [2020] Nikolaos Louros, Katerina Konstantoulea, Matthias De Vleeschouwer, Meine Ramakers, Joost Schymkowitz, and Frederic Rousseau.  WALTZ-DB 2.0: an updated database containing structural information of experimentally determined amyloid-forming peptides.  _Nucleic Acids Research_ , 48:D389–D393, January 2020.  ISSN 1362-4962.  doi: 10.1093/nar/gkz758.
  * Keresztes et al. [2022] Laszlo Keresztes, Evelin Szogi, Balint Varga, Viktor Farkas, Andras Perczel, and Vince Grolmusz.  Succinct amyloid and non-amyloid patterns in hexapeptides.  _ACS Omega_ , 2022.  URL https://doi.org/10.1021/acsomega.2c02513.

## Appendix: Supplementary material

We present here Supplementary Figures for our work.

Figure S1 is the suffix-version of Figure 1 in the main text: it visualizes
the sliding suffix windows of length 6.

Figure S2 is the $\alpha$-helix version of Figure 2 in the main text; it
yields a control for our main result.

Figure S3 visualizes the percentage of the water-soluble proteins in the non-
redundant PDB set we worked with.

Figure S4 explains the random control of our main result.

Figure S1: A visualization of the sliding suffix windows of $\beta$-chains,
mapping their border regions. A section of the polypeptide sequence of a
protein is drawn at the top of the figure, listed in the standard N-terminal
through C-terminal order. The green shaded subsequence represents the
$\beta$-sheet region, identified from the “SHEET” record of the PDB file. The
rows, annotated by $k=0,1,\ldots,6$, correspond to the sliding windows of
length 6 residues, when the window slides to the left, and in row, labeled by
$k=i$, exactly $i$ residues are outside the green-shaded $\beta$-sheet region,
for $i=0,1,2,\ldots,6$. Figure S2: The percentage of the amyloidogenic
length-6 prefix windows (left panel) and suffix windows (right panel) with $k$
positions outside the $\alpha$-helix region, predicted by the Budapest Amyloid
Predictor. The $x$ axis shows the $k$ value (i.e., the length of the window
over (or outside) the $\alpha$-helix sequence, and on the $y$ axis, the
percentage of the amyloidogenic length-6 windows are given. The difference
between the small and large $k$ values is much less than in the $\beta$-sheet
case.  Figure S3: The distribution of the protein-structure identifications
methods in the non-redundant set we have analyzed. The left panel contains the
overall distribution of the methods and the right panel refines the NMR
methods, differentiating soluble and solid-state methods. Clearly, almost all
proteins were water soluble in the set. Figure S4: Random subsequence control
for our main result. Consider the yellow hexapeptide in the first row of the
figure. It is easy to see that the same (yellow) hexapeptide may be a prefix-
window with $k=0$ for the green, with $k=1$ for the blue, with $k=2$ for the
red, or with $k=3$ for the brown subsequence of length $\ell$. That is, if a
hexapeptide is not too close to the right end of a protein chain, then it
appears with $k=1,2,3,4,5,6$ for different length $\ell$ sequences. Let us
remark that the amyloidocity of a hexapeptide depends only on its sequence, so
there would not be any difference of prefix-amyloidocities for different $k$
values, unlike in the case of $\beta$-sheets in Fig. 2.