# Learning to Detect Noisy Labels Using Model-Based Features

Zhihao Wang$\;{}^{14}$, Zongyu Lin111Code is available at
https://github.com/Rafa-zy/SENT$\;{}^{2}$, Peiqi Liu3, Guidong Zheng3, Junjie
Wen3
Xianxin Chen1, Yujun Chen1, Zhilin Yang$\;{}^{12}$
1Recurrent AI, 2Tsinghua University, 3China Merchants Bank, 4Meta
<EMAIL_ADDRESS>{linzongy21<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>The authors have contributed equally to
this work. The work was conducted while the author was an intern at Recurrent
AI. Corresponding Author.

###### Abstract

Label noise is ubiquitous in various machine learning scenarios such as self-
labeling with model predictions and erroneous data annotation. Many existing
approaches are based on heuristics such as sample losses, which might not be
flexible enough to achieve optimal solutions. Meta learning based methods
address this issue by learning a data selection function, but can be hard to
optimize. In light of these pros and cons, we propose SENT (Selection-Enhanced
Noisy label Training) that does not rely on meta learning while having the
flexibility of being data-driven. SENT transfers the noise distribution to a
clean set and trains a model to distinguish noisy labels from clean ones using
model-based features. Empirically, on a wide range of tasks including text
classification and speech recognition, SENT improves performance over strong
baselines under the settings of self-training and label corruption.111Code is
available at https://github.com/Rafa-zy/SENT

##  1 Introduction

State-of-the-art deep neural networks require large amounts of annotated
training data. Though the success of large pre-training models (Devlin et al.,
2018) alleviates such requirements, high-quality labeled data are still
crucial to obtain the best performance on downstream tasks. However, it is
extremely expensive to acquire large-scale annotated data for every new task.

To overcome the challenge of rigorous data requirements, recent works utilize
weak labels for supervision, including heuristic rules (Augenstein et al.,
2016; Bach et al., 2019; Awasthi et al., 2020), large-scale datasets with
cheap but noisy labels (Li et al., 2017; Lee et al., 2018);, and self-training
(Park et al., 2020; Wang et al., 2020). In self-training, one trains a model
on a labeled dataset and then predicts on a large amount of unlabeled data.
Then the clean labeled data and pseudo-labeled data are combined to further
train the model.

The aforementioned sources of supervision share two important characteristics:
First, they are learning from noisy labels. Second, the noise is dependent on
data features. Thus, they could be unified into the framework of noisy label
learning, for which numerous approaches have been proposed to reduce the
negative impact of noise. However, there are three problems in prior work on
noisy label learning: First, many existing approaches are based on heuristics
such as sample losses which are not flexible enough (Han et al., 2018; Yu et
al., 2019; Zhou et al., 2020; Song et al., 2019); Second, many previous works
require prior knowledge of the noise distribution of the dataset to adjust the
hyperparameters, which is often not available in real-world applications (Song
et al., 2019, 2020). Third, meta learning based methods avoid previous
problems but suffer from optimization difficulties (Ren et al., 2018; Zheng et
al., 2021) such as longer training time, heavy hyper-parameter tuning and an
unstable convergence process. To address the above problems, we propose a
simple data-driven approach which does not rely on meta learning while being
flexible.

Our contributions are summarized as follows:

  * •

We propose a simple yet effective de-noising approach which avoids the
optimization difficulty of meta learning while enjoying the flexibility of
being data-driven.

  * •

We unify the settings of both self-training and label corruption into a noisy
label learning framework and demonstrate the effectiveness of our approach
under both settings.

  * •

Our approach improves performance over state-of-the-art baselines on a wide
range of datasets, including text classification and automatic speech
recognition (ASR). Last but not least, our approach achieves even larger gains
on few-shot learning.

##  2 Related Work

###  2.1 Self-training

Self-training is a powerful learning method that enables models to learn from
huge amounts of unlabeled data by generating weak labels through either the
teacher model predictions or heuristic rules. Self-training has been shown to
be effective in many scenarios, including image classification (Yalniz et al.,
2019), text classification (Li et al., 2019), machine translation (Wu et al.,
2019), etc. However, noise contained in weak labels could largely hinge the
performance of self-training.

Recently, Xie et al. (2020) improved the performance of self-training on image
classfication by injecting noise to the student model, which is called
NoisyStudent. Park et al. (2020) customized NoisyStudent on automatic speech
recognition. One problem related with self-training is error propagation (Zou
et al., 2019); in other words, pseudo labelling on unlabeled data might bring
noise to the training set which leads to the degradation of further training.
Most previous work simply set a fixed threshold to filter samples with low
confidence (Sohn et al., 2020; Xie et al., 2020). Wang et al. (2020) used meta
learning for adaptive sample re-weighting to mitigate error propagation from
noisy pseudo-labels. Zhang et al. (2021) used a curriculum learning approach
to re-weight unlabeled data according to the model’s learning status. In our
work, we alleviate the error propagation from another perspective, by learning
a selection model using model-based features based on a clean dataset.

###  2.2 Noisy Label Learning

Learning from noisy labels has long been a research area. One of the most
classical works is to add a noise adaptation layer on top of the main model to
learn a label transition matrix for label correction (Goldberger et al.,
2017). Bootstrapping (Reed et al., 2014) introduces the notion of perceptual
consistency that a model predicts correct labels for noisy samples before
overfitting to noisy labels. Co-Teaching (Han et al., 2018) and Co-Teaching+
(Yu et al., 2019) train two networks while each network selects its small-loss
samples as clean samples for its peer network. However, the aforementioned
approaches only deal with class dependent noise (CDN) and make a strong
assumption that noise distribution is independent of each instance, which is
not flexible enough for many cases.

SEAL (Chen et al., 2020) goes beyond previous work to consider instance
dependent noise (IDN), which is more realistic and common than CDN on real
world datasets. SELFIE (Song et al., 2019) uses a hybrid approach that selects
refurbishable samples based on the entropy of model predictions and then
refurbishes the labels with model predictions. RoCL (Zhou et al., 2020)
utilizes curriculum learning that starts with easy and clean samples and
gradually moves to data with pseudo labels produced by a time-ensemble.
However, both SELFIE and RoCL require prior knowledge of the noise
distribution of the dataset and manual adjustment for hyperparameters. To
avoid such efforts, meta learning is introduced to learn selection and
refurbishment.

Learning to Re-weight (Ren et al., 2018) is a meta learning algorithm that
learns to assign weights to training examples based on their gradient
directions. Meta-weight-net (Shu et al., 2019) parameterizes the reweighting
function as a multi-layer perceptron network. Meta Label Correction (Zheng et
al., 2021) trains the target model with corrected labels generated by a label
correction model trained on clean validation data which is jointly trained by
solving a bi-level optimization problem. These meta learning algorithms afford
a large degree of flexibility by directly optimizing a reliable objective.
However, meta learning based models are known to be sensitive to
hyperparameter tuning and the quality of support data (Agarwal et al., 2021)
and suffer from optimization difficulties as they are trained by propagating
second-order gradients (Hospedales et al., 2020).

The major differences between our approach and previous methods are as
follows. Compared with meta-learning based models, our approach do not suffer
from optimization difficulties. Compared with models with CDN assumptions, our
approach can handle the IDN settings. Compared with other state-of-the-art IDN
methods such as SELFIE and RoCL, the selection strategy in our approach is
learnable with model-based features. And our approach does not require the
prior knowledge of noise distribution. Last but not least, we unify the
settings of self-training and label corruption in the framework of noisy label
learning and conduct extensive experiments on both settings.

Figure 1: The pipeline of our general framework. Firstly, we are given a noisy
training set and a small clean set for selection (also as development set).
Secondly, we transfer the noise from the training set to the selection set.
Thirdly, we compute predefined signals on both sets and train our selection
model S using the selection set. Fourthly, We apply the trained selection
model on the training set to distinguish clean samples for training model M.
Finally, we repeat step 3 and 4.

##  3 Method

Now we present our approach, SENT (Selection Enhanced Noisy label Training),
that learns to select a subset from a noisy dataset and only uses the selected
subset for training to reduce label noise. The core idea is to transfer the
noise distribution so that both clean and noisy labels are available on a data
subset. A selection model is trained to distinguish clean labels from noisy
ones and then applied to selecting a clean subset from a noisy dataset.

Formally, given a noisy training dataset $D_{train}=\\{(x_{i},y_{i})|1\leq
i\leq N\\}$ that is corrupted following some unknown noise distribution $P$,
our approach is to learn a selection strategy $f$ to select a clean subset
$D^{\prime}_{train}=\\{x_{i}|y_{i}=y_{i}^{*},i=1,2,...,N\\}$ for training.
Here $(x_{i},y_{i})$ is a training sample, $y_{i}$ is the noisy label,
$y_{i}^{*}$ is the corresponding unknown true label, and $N$ is the size of
training set. Let model $M$ be our main model which is trained on the noisy
dataset $D_{train}$ and performs certain tasks such as text classification,
speech recognition and others. We will also have a selection model $S$ trained
on a small clean development dataset $D_{select}$. The task of $S$ is to learn
the selection strategy $f$. There are two main stages in our approach: noise
transfer and selection learning.

###  3.1 Noise Transfer

Now we describe the noise transfer stage. Given the corrupted training set
$D_{train}$, we learn the unknown noise distribution $P$ and transfer the
noise to $D_{select}$. We train a model $M$ on $D_{train}$ till full
convergence, which predicts a (noisy) label given a text input. $M$ is now
assumed to have learned to capture the unknown distribution $P$ by its
parameters. Then we will use the model $M$ to get noisy labels on $D_{select}$
by making predictions. Here we argue that the noise on $D_{select}$ is
approximately following the noise distribution $P$. Formally, now the
development set is
$D_{select}=\\{(x_{i}^{select},y_{i}^{select},y_{i}^{select*})\\}$, where
$(x_{i}^{select},y_{i}^{select*})$ is the original clean development sample,
and $y_{i}^{select}$ is the noisy label predicted by $M$.

###  3.2 Selection Learning

We model the selection learning stage as a binary classification task. On
$D_{select}$, a model $S$ is trained to classify whether a label is clean or
noisy. In our approach, the model $S$ is constructed as a multi-layer
perceptron with one hidden layer, which uses a pre-calculated 5-dimensional
feature vector as the input and outputs a binary classification probability.
Given a sample from the development set
${(x_{i}^{select},y_{i}^{select},y_{i}^{select*})}$, the selection training
sample is defined as $({sg}_{i},{sr}_{i})$, where ${sg}_{i}$ is a
5-dimensional feature vector for the $i$-th sample. We will discuss how to
compute the 5-dimensional feature in later sections. Meanwhile, ${sr}_{i}$ is
the corresponding true selection result, defined as
${sr}_{i}=\mathbb{I}_{[{y_{i}^{select}=y_{i}^{select*}}]}$. In other words, if
the $i$-th sample has a clean label, ${sr}_{i}=1$, and otherwise zero. Let
$P_{select}(sr_{i}|sg_{i})$ denote the probability given by the selection
model $S$. The loss function for each sample can be written as:

$L_{i}^{select}=-\log P_{select}({sr}_{i}|sg_{i})$ (1)

###  3.3 Model-Based Features

Now we discuss how to compute the 5-dimensional feature $sg_{i}$ for each
selection sample $x_{i}^{select}$.

The first feature is called the instant loss. Given a sample
${(x_{i},y_{i})}$, let $\hat{P}(y_{i}|x_{i})$ be the predicted probability
from the main model $M$. The instant loss (IL) for the sample is defined as:

$\text{IL}_{i}=-log\hat{P}(y_{i}|x_{i})$ (2)

Intuitively, a larger instant loss indicates a possibly noisier sample,
because the model $M$ has low confidence in predicting the label. However, as
training proceeds, the model will overfit some of the noisy labels. As a
result, the instant loss will decrease for noisy samples as well. To address
this issue, following Zhou et al. (2020), we additionally use an exponential
moving average (EMA) loss to better differentiate noisy and clean samples. In
the $t$-th training epoch, the EMA loss (EMAL) for the $i$-th sample is
defined as follows:

$\text{EMAL}_{i}^{t}=\left\\{\begin{array}[]{lr}\gamma\text{EMAL}_{i}^{t-1}+(1-\gamma)\text{IL}_{i},&t\geq
1\\\ \text{IL}_{i},&t=0\end{array}\right.$ (3)

where $\gamma\in[0,1]$ is a discounting factor. Intuitively, a larger EMAL
represents a possibly noisier sample as the model has lower confidence in the
training history.

Song et al. (2019) has shown that the entropy of model prediction is a strong
indicator to differentiate noisy and clean samples as noisy samples tend to
have a larger entropy. Here we adopt two entropy signals as additional
features: Instant Entropy (IE) and History Entropy (HE). The calculation of IE
follows Song et al. (2019). Let $\hat{y_{i}}^{t}$ be the predicted label of
the $i$-th sample at epoch $t$ and let
$H_{i}(t)=\\{\hat{y_{i}}^{0},\hat{y_{i}}^{1},....,\hat{y_{i}}^{t}\\}$ be the
prediction history of first $t$ epochs. Then we formulate an empirical
distribution

$\tilde{P}(y|x_{i},t)=\frac{1}{t}\sum_{y^{\prime}\in
H_{i}(t)}\mathbb{I}_{[y=y^{\prime}]}$

which equals the ratio of prediction $y$ in the first $t$ epochs. The IH
feature at the $t$-th epoch is computed as

$\text{IE}_{i}=(1/\tau)\times-\tilde{P}(\hat{y}_{i}^{t}|x_{i},t)\log\tilde{P}(\hat{y}_{i}^{t}|x_{i},t)$
(4)

where $\tau=-\log(1/k)$ is a normalizing factor with $k$ being the number of
labels. The HE feature at the $t$-th epoch is computed as

$\text{HE}_{i}=(1/\tau)\times\sum_{y}-\tilde{P}(y|x_{i},t)\log\tilde{P}(y|x_{i},t)$
(5)

We explore another informative feature inspired by (Han et al., 2019) who
discovered that distances between low level features and high level features
in a convolution model are larger on noisy samples than on clean samples. We
find this holds for transformer models. Thus, we adopt the cosine similarity
between the hidden states of the first layer and the last layer as another
feature. Formally, FLS (First Last Similarity) is defined as:

$\text{FLS}_{i}=\text{normalize}\left(\sum_{j}\cos(\mathbf{h}_{ij}^{F},\mathbf{h}_{ij}^{L})\right)$
(6)

where $\mathbf{h}_{ij}^{F}$ and $\mathbf{h}_{ij}^{L}$ represent the hidden
states of the $j$-th token in the first and last layers respectively, and
$\cos$ refers to cosine similarity. We normalize the FLS feature into the
range of $[0,1]$.

Finally, we concatenate the above all features to be the input for the
selection model $S$ as follows,

$sg_{i}=[\text{EMAL}_{i},\text{IL}_{i},\text{HE}_{i},\text{IE}_{i},\text{FLS}_{i}]$
(7)

where $[,]$ denotes concatenation.

###  3.4 Overall Training Procedure

1: Initialization: $M$;$S$;$D_{train}$;$D_{select}$; total_epochs;
pretrain_epochs

2: Train $M$ on $D_{train}$. {Learn noise distribution $P$}

3: Infer on $D_{select}$ using $M$. {Noise Transfer}

4: epoch = 0; Reinitialize M;

5: Pretrain $M$ for pretrain_epochs;

6: while epoch $<$ total_epochs do

7: Infer signals $[\text{IL},\text{EMAL},\text{HE},\text{IE},\text{FLS}]$ on
$D_{train}$ and $D_{select}$ using $M$;

8: while not early stopping $S$ do

9: Train $S$ on $D_{select}$;

10: end while

11: Select a clean subset $D^{\prime}_{train}$ out of $D_{train}$ using $S$;

12: Train $M$ on $D^{\prime}_{train}$;

13: end while

13: $M$

Algorithm 1 SENT

Now we show the training procedure with pseudo code in Algorithm 1 and
illustrations in Figure 1. In the first box of Figure 1, we clarify our data
setting, where a clean set $D_{select}$ and a corrupted set $D_{train}$ are
the input to our method. In the second box, we learn the noise distribution
$P$ of the training set by fitting model $M$ on $D_{train}$ and then transfer
$P$ to $D_{select}$. After this step, we will have both noisy and clean labels
on $D_{select}$. This box corresponds to lines 2 to 4 in Algorithm 1.

Next, we will move on to the repeated training stage for model $S$ and model
$M$, which is shown in box 3 and 4 of the figure. First, as illustrated in box
3, we use model $M$ to infer the aforementioned signals on $D_{train}$ and
$D_{select}$. Then we will use the signals on $D_{select}$ as the input to
train the selection model $S$. Since $D_{select}$ has both clean and noisy
labels, we can easily get the training targets $sr$ as mentioned in Section
3.2 for the selection model $S$. Once we are done with selection learning,
given the inferred signals on $D_{train}$, we can use $S$ to predict and
select clean samples from $D_{train}$ to get $D^{\prime}_{train}$. Then we
will train $M$ on the clean subset $D^{\prime}_{train}$ and repeat above
steps. $D^{\prime}_{train}$ is not guaranteed to be entirely clean but is
expected to be cleaner than $D_{train}$. This repeated training phase
corresponds to the code from lines 6 to lines 13 in Algorithm 1.

###  3.5 Adaptation to Self-Training

Our above framework can be directly applied to learning scenarios with noisy
labels. In this section, we will further discuss how to adapt this method to
self-training, a classic semi-supervised learning paradigm. Generally, it
trains the teacher model on labeled data to infer pseudo labels on unlabeled
data and add them back to the original training set. Then a student model is
trained on the combined data. After that, the student becomes a new teacher
model and the above process is repeated. Obviously, the process of pseudo
labelling will bring noise since it cannot ensure 100% accuracy on the
predicted labels. Therefore, it fits the nature of our proposed framework.
Specifically, the noise distribution $P$ in self-training is known because the
source of noise is the teacher model. Thus, it is natural to directly leverage
the teacher model to infer noisy labels on $D_{select}$.

Figure 2: The pipeline of our general system applied to the classical self-
training paradigm.

The whole pipeline of adapting our framework to classic self-training is
displayed in Figure 2, and the corresponding algorithm is shown in Algorithm 2
in Appendix. After training the teacher model on labeled data $L$, we use the
teacher model to infer on the unlabeled data $U$ and the dev set $D_{select}$.
This is followed by training the selection model based on the signals of
$D_{select}$. Then, we utilize the selection model to predict on the unlabeled
data $U$ to judge whether to choose each pseudo-labeled sample or not. Then,
we train the student model on the combination of original labeled data and
selected samples from unlabeled data. The above procedure is repeated as in
classical self-training.

##  4 Experiments

| IMDB | SMS | SMS* | TREC | TREC* | YOUTUBE | AGNEWS
---|---|---|---|---|---|---|---
Train | 250 | 69 | 61 | 68 | 60 | 77 | 60
Test | 25000 | 500 | 392 | 500 | 500 | 392 | 7600
Unlabeled | 24500 | 4502 | 4948 | 4965 | 5409 | 1409 | 11876
Dev_eval | 126 | 250 | 31 | 251 | 30 | 40 | 32
Dev_select | 124 | 250 | 31 | 249 | 32 | 38 | 32
Table 1: Statistics of text classifcation datasets. Method | TREC | TREC* | SMS | SMS* | AGNEWS | IMDB | YOUTUBE
---|---|---|---|---|---|---|---
Supervised | 83.4 | 84.2 | 97.8 | 97.5 | 78.1 | 85.1 | 92.9
Co-teaching+ | 81.8 | 79.0 | 98.2 | 98.4 | 82.8 | 86.8 | 92.1
L2R | 80.8 | 86.0 | 98.6 | 97.2 | 84.2 | 84.7 | 92.9
SELF | 81.0 | 81.2 | 98.4 | 98.6 | 82.4 | 83.5 | 92.9
Self-train | 84.0 | 84.8 | 97.9 | 98.4 | 82.4 | 87.0 | 93.6
Self-train (thres) | 84.2 | 87.8 | 97.9 | 98.4 | 83.3 | 85.6 | 93.1
Noisy Student | 85.0 | 88.6 | 98.4 | 99.0 | 85.0 | 87.7 | 93.9
Ours | 85.0 | 86.6 | 99.0 | 99.2 | 83.4 | 88.3 | 94.4
Ours + noisy | 85.0 | 89.2 | 99.0 | 99.2 | 86.0 | 89.0 | 95.2
Table 2: Results for text classification in the self-training setting. We
compare our approach with baselines under the BERT-based models.

###  4.1 Experimental Setup

####  4.1.1 Overview

To verify the effectiveness of SENT, we conduct extensive experiments on text
classification and automatic speech recognition (ASR) benchmarks.

We use text classification tasks to evaluate the self-training setting. We
perform finetuning based on the BERT (Devlin et al., 2018) model. We use ASR
tasks to evaluate the label corruption setting. Our approach and other
baselines are built on top of an encoder-decoder transformer network. Details
of model configuration can be found in Appendix A.2.

####  4.1.2 Datasets

For text classification in the self-training setting, we evaluate our
framework on the following five benchmark datasets: question classification
TREC-6 (Li and Roth, 2002), spam classification of SMS messages (Almeida et
al., 2011), spam classification of YouTube comments (Alberto et al., 2015),
AG’s news topic classification dataset (Zhang et al., 2015), and sentiment
classification on IMDB movie reviews (Maas et al., 2011). Our data splits
follow previous work (Karamanolakis et al., 2021). Related details are shown
in Table 1. For the SMS and TREC datasets, we consider two separate versions.
The datasets with $*$ have smaller development sets $D_{eval}$ and
$D_{select}$ while the ones without $*$ have larger developments. Smaller
development sets are more challenging for noisy label learning because
selection learning has to perform on a smaller clean set. We use these two
separate versions to test the robustness of our approach.

For ASR in the label corruption learning setting, we use AISHELL-1 (Bu et al.,
2017) as the benchmark. Following prior work, we model IDN using DNNs’
prediction error (Du and Cai, 2015; Menon et al., 2018). Specifically, we
train three small transformer models to corrupt the training set to different
corruption levels: hard, medium, easy. The higher the error rate, the harder
the corrupted dataset. In the following experiments of SENT, the prior noise
information (i.e. how the training set is corrupted) is assumed to be unknown.
Related statistics are shown in Table 8 in Appendix.

####  4.1.3 Evaluation

For text classification, we report micro F1 for SMS and accuracy for the rest
of the datasets. For ASR, we follow Bu et al. (2017) to use the character
error rate (CER) for evaluation.

Since our approach relies on an additional clean set $D_{select}$, we split
the normal development set into two halves. We use one half as $D_{select}$
for selection learning and the other half $D_{eval}$ for standard model
tuning, so as to set up fair comparison with the baselines.

####  4.1.4 Baselines

For text classification, we compare with the following baselines: (a)
“Supervised” refers to supervised learning using only labeled data; (b) “self-
train” is standard self training that utilizes both labeled and unlabeled data
for iterative training; (c) “self-train (thres)” means self-training that uses
the development set to select a threshold of confidence score for filtering
pseudo-labeled data; (d) “noisy student” (Xie et al., 2020) adds dropout noise
to the student model in self training; (e) “co-teaching+” (Yu et al., 2019)
uses two neural networks to select small-loss samples for each other and
applies a disagreement strategy; (f) “L2R” (Ren et al., 2018) learns to re-
weight noisy labels via meta-learning (g) “SELF” (Nguyen et al., 2019)
utilizes self-ensemble predictions to progressively remove noisy labels. We
also evaluate the performance when combining noisy student and our method,
denoted as “ours + noisy”.

In the experiments of ASR, we include the following baselines: Vanilla (a
naive encoder-decoder transformer network), Co-Teaching+ (Yu et al., 2019),
L2R (Ren et al., 2018), RoCL (Zhou et al., 2020) and SELFIE (Song et al.,
2019). The details of these baselines can be found in Appendix A.1.1.

###  4.2 Experimental Results

Model | Hard | Medium | Easy
---|---|---|---
Vanilla | 32.63 | 21.82 | 16.10
Co-Teaching+ | 32.08 | 21.67 | 15.11
L2R | 30.43 | 20.07 | 15.15
RoCL | 27.16 | 17.87 | 14.95
SELFIE | 27.31 | 19.10 | 13.98
Ours | 26.91 | 17.67 | 13.47
Table 3: Comparison with baselines on AISHELL-1 test set in the label
corruption setting. We use CER as the metric of performance. A lower CER
indicates a better model.

Data | Method | Pse-Acc. | Sel-Pre. | Sel-Rec. | #Selected
---|---|---|---|---|---
IMDB | Self-train | 85.7 | 85.7 | 100.0 | 24500
Self-train (thres) | 85.2 | 94.3 | 58.1 | 12870
Ours | 88.1 | 97.9 | 47.5 | 7742
YOUTUBE | Self-train | 95.0 | 95.0 | 100.0 | 1409
Self-train (thres) | 95.7 | 96.1 | 98.7 | 1386
Ours | 94.6 | 96.9 | 94.1 | 1294
SMS* | Self-train | 98.1 | 98.1 | 100.0 | 4948
Self-train (thres) | 96.0 | 96.0 | 98.6 | 4880
Ours | 98.3 | 98.3 | 100.0 | 4897
TREC* | Self-train | 77.4 | 77.4 | 100.0 | 4965
Self-train (thres) | 77.2 | 77.2 | 99.8 | 4944
Ours | 78.3 | 81.2 | 67.1 | 4897
AGNEWS | Self-train | 84.4 | 84.4 | 100.0 | 11876
Self-train (thres) | 83.4 | 83.6 | 99.7 | 11826
Ours | 84.3 | 88.0 | 85.6 | 9728

Table 4: The key metrics of pseudo labeling and sample selection during the
self-training. We report the accuracy of pseudo labeling, the precision and
recall of sample selection, and the number of selected samples.

####  4.2.1 Results in Text Classification.

As shown in Table 2, the self-training baseline improves text classification
performance. In comparison, our selection approach can stably lift the
performance, which shows that our selection model has learned an informative
selection strategy. Last, although using SENT alone outperforms self-train and
noisy student, our approach can be combined with the noisy student approach to
achieve even better performance. Overall, this combined approach achieves the
best performance among all the datasets we consider.

####  4.2.2 Results in ASR

Table 3 shows that Co-Teaching+ is not able to handle our setting as it only
achieves similar result to the vanilla model. L2R is effective on our problem
setting with improved performance. However, meta learning based L2R
underperforms RoCL and SELFIE. Compared with above baselines, our approach
consistently excels on all error levels, which demonstrates the effectiveness
of our approach.

###  4.3 Empirical Analysis

##### Case Study: Key Metrics During Self-Training

In order to gain a deeper understanding of how our method improves over the
traditional self-training methods, we investigate some of the key metrics
regarding pseudo labelling and selection performance. We consider self-train,
self-train (thres), and our model. Specifically, we display the accuracy of
the pseudo labelling on unlabeled data (Pse-Acc.), the precision of sample
selection (Sel-Pre.), the recall of sample selection (Sel-Rec.) and the number
of selected samples in the best round (i.e., the training round that achieves
the best performance in the repeated self-training process). As can be seen in
Table 4, our approach achieves a better pseudo labeling accuracy. This is
because our approach obtains a more balanced tradeoff between selection
precision and selection recall compared to self training. Because of a more
rigorous selection model, our approach tends to only select samples with a
higher probability of having clean labels; i.e., increasing the selection
precision. This is also reflected in small decrease in the number of selected
samples. We believe this is crucial for mitigating error propagation (Xie et
al., 2020) and thus for better performance.

##### Ablation Study: Substituting with Simpler Models

To further test the robustness of our approach, we substitute the base model
from BERT to simple multi-layer percetrons (MLPs) without pretraining. As show
in Table 5, the performance will decrease after applying simpler MLPs.
However, our approach remains effective compared to the other baselines. The
relative gain and absolute improvement from “Supervised” to our approach is
still significant.

Method | TREC | SMS | YOUTUBE
---|---|---|---
Supervised | 66.5 | 93.3 | 91.0
Co-teaching+ | 66.8 | 98.3 | 93.3
L2R | 66.4 | 98.0 | 93.3
SELF | 66.3 | 98.1 | 92.3
Self-train | 71.1 | 95.1 | 92.5
Self-train (thres) | 70.1 | 98.1 | 92.1
Noisy Student | 68.9 | 98.1 | 92.1
Ours | 70.3 | 98.2 | 93.3
Ours+Noisy | 70.0 | 98.4 | 93.4

Table 5: Comparison with baselines under the MLP models. Signal | Hard | Medium | Easy
---|---|---|---
All | 26.91 | 17.67 | 13.47
-EMAL | 27.31 | 17.95 | 13.85
-IL | 28.56 | 19.32 | 14.94
-HE | 29.32 | 19.88 | 15.01
Table 6: Ablation experiments for features on AISHELL-1. ’All’ is all features. Each line removes one feature based on the last line. Signal | IMDB | YT | TREC | TREC* | SMS | SMS* | AG
---|---|---|---|---|---|---|---
All | 85.9 | 95.2 | 85.0 | 89.2 | 99.0 | 99.2 | 85.1
-FLS | 88.2 | 92.9 | 83.6 | 82.0 | 99.0 | 99.0 | 84.1
-EMAL | 89.0 | 92.1 | 84.2 | 86.8 | 99.0 | 99.0 | 86.0
-IL | 87.8 | 92.6 | 83.4 | 85.8 | 99.0 | 98.8 | 82.5
-HE | 88.0 | 93.4 | 82.6 | 79.8 | 98.4 | 98.0 | 79.6
Table 7: Ablation experiments for features on text classification. ’All’ is
all signals. Each line removes one feature from the previous line. YT
represents YouTube dataset, and AG represents AG News dataset. Figure 3:
Evaluation on SMS under the few-shot setting.

##### Ablation Study: Features

We study the effects of the model-based features we introduced in Section 3.3
with an ablation. Note that we did not use FLS for ASR because the the first
layer and the last layer have different lengths. As shown in Table 6 and Table
7, all of the features contribute to the final performance. Among the
features, adding the instant loss (IL) feature results in the most relative
gain for performance.

Also, we do similar experiments on text classification tasks. As seen in Table
7, we can do basic search on feature engineering to improve the final
performance. Among all signals, IL leads to the greatest relative gain of the
performance. But across all the datasets, each signal has its own role and
contributes to our final performance.

##### Performance Under Few-shot Setting

LST (Li et al., 2019) has shown that self-training paradigm can be customized
on few-shot classification. Here, we also investigate the effectiveness of our
method when applying to the few-shot setting. Specifically, we evaluate on
selected text classification datasets (i.e., YOUTUBE, SMS and IMDB). Figure 5
showcases that generally self-train performs better than “Supervised” (using
only labeled data), while our model achieves the best performance in most
cases, indicating the robustness of our method. More results are displayed in
Appendix A.3.2.

##  5 Conclusions

In this paper, we propose SENT to address the problem of label noises.
Compared with meta learning based models, our selection model is trained with
full supervision using cross entropy loss which facilitates the convergence
process. In the meantime, we model IDN noise without the prior knowledge of
noise distribution. We also unify the setting of self-training and label
corruption in the framework of noisy label learning and conduct extensive
experiments on both settings.

##  6 Limitations

Although our framework has been proved to be effective under the setting of
self-training and label corruption on text classification and speech
recognition tasks, adapting our approach to more sequence-level tasks such as
named entity recognition (NER) and machine translation will also be
interesting. Besides, the selection model in our framework is feature-based.
These features are very informative but might be limited in terms of
expressivity. This reserves the space for further improvement to learn more
data-driven features under our framework.

## References

  * Agarwal et al. (2021) Mayank Agarwal, Mikhail Yurochkin, and Yuekai Sun. 2021.  On sensitivity of meta-learning to support data.  _Advances in Neural Information Processing Systems_ , 34.
  * Alberto et al. (2015) Túlio C Alberto, Johannes V Lochter, and Tiago A Almeida. 2015.  Tubespam: Comment spam filtering on youtube.  In _2015 IEEE 14th International conference on machine learning and applications (ICMLA)_ , pages 138–143. IEEE.
  * Almeida et al. (2011) Tiago A Almeida, José María G Hidalgo, and Akebo Yamakami. 2011.  Contributions to the study of sms spam filtering: new collection and results.  In _Proceedings of the 11th ACM symposium on Document engineering_ , pages 259–262.
  * Augenstein et al. (2016) Isabelle Augenstein, Tim Rocktäschel, Andreas Vlachos, and Kalina Bontcheva. 2016.  Stance detection with bidirectional conditional encoding.  _arXiv preprint arXiv:1606.05464_.
  * Awasthi et al. (2020) Abhijeet Awasthi, Sabyasachi Ghosh, Rasna Goyal, and Sunita Sarawagi. 2020.  Learning from rules generalizing labeled exemplars.
  * Bach et al. (2019) Stephen H Bach, Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Cassandra Xia, Souvik Sen, Alex Ratner, Braden Hancock, Houman Alborzi, et al. 2019.  Snorkel drybell: A case study in deploying weak supervision at industrial scale.  In _Proceedings of the 2019 International Conference on Management of Data_ , pages 362–375.
  * Bu et al. (2017) Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. 2017.  Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline.  In _2017 20th Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech I/O Systems and Assessment (O-COCOSDA)_ , pages 1–5. IEEE.
  * Chen et al. (2020) Pengfei Chen, Junjie Ye, Guangyong Chen, Jingwei Zhao, and Pheng-Ann Heng. 2020\.  Beyond class-conditional assumption: A primary attempt to combat instance-dependent label noise.  _arXiv preprint arXiv:2012.05458_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Du and Cai (2015) Jun Du and Zhihua Cai. 2015.  Modelling class noise with symmetric and asymmetric distributions.  In _Twenty-Ninth AAAI Conference on Artificial Intelligence_.
  * Goldberger et al. (2017) J Goldberger, E Ben-Reuven, et al. 2017.  Deep neural—networks using a noise adaptation layer.  In _Proc. 5th Int. Conf. Learn. Represent.(ICLR) Conf. Track_ , 2014, pages 1–9.
  * Han et al. (2018) Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. 2018.  Co-teaching: Robust training of deep neural networks with extremely noisy labels.  _arXiv preprint arXiv:1804.06872_.
  * Han et al. (2019) Jiangfan Han, Ping Luo, and Xiaogang Wang. 2019.  Deep self-learning from noisy labels.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5138–5147.
  * Hospedales et al. (2020) Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. 2020.  Meta-learning in neural networks: A survey.  _arXiv preprint arXiv:2004.05439_.
  * Hu et al. (2016) Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. 2016.  Harnessing deep neural networks with logic rules.  _arXiv preprint arXiv:1603.06318_.
  * Karamanolakis et al. (2021) Giannis Karamanolakis, Subhabrata Mukherjee, Guoqing Zheng, and Ahmed Hassan Awadallah. 2021.  Self-training with weak supervision.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 845–863, Online. Association for Computational Linguistics.
  * Lee et al. (2018) Kuang-Huei Lee, Xiaodong He, Lei Zhang, and Linjun Yang. 2018.  Cleannet: Transfer learning for scalable image classifier training with label noise.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5447–5456.
  * Li et al. (2017) Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. 2017.  Webvision database: Visual learning and understanding from web data.  _arXiv preprint arXiv:1708.02862_.
  * Li and Roth (2002) Xin Li and Dan Roth. 2002.  Learning question classifiers.  In _COLING 2002: The 19th International Conference on Computational Linguistics_.
  * Li et al. (2019) Xinzhe Li, Qianru Sun, Yaoyao Liu, Qin Zhou, Shibao Zheng, Tat-Seng Chua, and Bernt Schiele. 2019.  Learning to self-train for semi-supervised few-shot classification.  _Advances in Neural Information Processing Systems_ , 32:10276–10286.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  * Menon et al. (2018) Aditya Krishna Menon, Brendan Van Rooyen, and Nagarajan Natarajan. 2018.  Learning from binary labels with instance-dependent noise.  _Machine Learning_ , 107(8):1561–1595.
  * Nguyen et al. (2019) Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. 2019.  Self: Learning to filter noisy labels with self-ensembling.  _arXiv preprint arXiv:1910.01842_.
  * Park et al. (2020) Daniel S Park, Yu Zhang, Ye Jia, Wei Han, Chung-Cheng Chiu, Bo Li, Yonghui Wu, and Quoc V Le. 2020.  Improved noisy student training for automatic speech recognition.  _arXiv preprint arXiv:2005.09629_.
  * Ratner et al. (2017) Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. 2017.  Snorkel: Rapid training data creation with weak supervision.  In _Proceedings of the VLDB Endowment. International Conference on Very Large Data Bases_ , volume 11, page 269. NIH Public Access.
  * Reed et al. (2014) Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. 2014.  Training deep neural networks on noisy labels with bootstrapping.  _arXiv preprint arXiv:1412.6596_.
  * Ren et al. (2018) Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. 2018.  Learning to reweight examples for robust deep learning.  In _International Conference on Machine Learning_ , pages 4334–4343. PMLR.
  * Shu et al. (2019) Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. 2019\.  Meta-weight-net: Learning an explicit mapping for sample weighting.  _arXiv preprint arXiv:1902.07379_.
  * Sohn et al. (2020) Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. 2020.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  _arXiv preprint arXiv:2001.07685_.
  * Song et al. (2019) Hwanjun Song, Minseok Kim, and Jae-Gil Lee. 2019.  Selfie: Refurbishing unclean samples for robust deep learning.  In _International Conference on Machine Learning_ , pages 5907–5915. PMLR.
  * Song et al. (2020) Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. 2020.  Learning from noisy labels with deep neural networks: A survey.  _arXiv preprint arXiv:2007.08199_.
  * Wang et al. (2020) Yaqing Wang, Subhabrata Mukherjee, Haoda Chu, Yuancheng Tu, Ming Wu, Jing Gao, and Ahmed Hassan Awadallah. 2020.  Adaptive self-training for few-shot neural sequence labeling.  _arXiv preprint arXiv:2010.03680_.
  * Wu et al. (2019) Lijun Wu, Yiren Wang, Yingce Xia, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2019\.  Exploiting monolingual data at scale for neural machine translation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4207–4216.
  * Xie et al. (2020) Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. 2020.  Self-training with noisy student improves imagenet classification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10687–10698.
  * Yalniz et al. (2019) I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. 2019.  Billion-scale semi-supervised learning for image classification.  _arXiv preprint arXiv:1905.00546_.
  * Yu et al. (2019) Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor W. Tsang, and Masashi Sugiyama. 2019.  How does disagreement help generalization against label corruption?
  * Zhang et al. (2021) Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. 2021.  Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling.  _arXiv preprint arXiv:2110.08263_.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  _Advances in neural information processing systems_ , 28:649–657.
  * Zheng et al. (2021) Guoqing Zheng, Ahmed Hassan Awadallah, and Susan Dumais. 2021.  Meta label correction for noisy label learning.  In _Proceedings of the 35th AAAI Conference on Artificial Intelligence_.
  * Zhou et al. (2020) Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. 2020.  Robust curriculum learning: From clean label detection to noisy label self-correction.  In _International Conference on Learning Representations_.
  * Zou et al. (2019) Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. 2019.  Confidence regularized self-training.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5982–5991.

##  Appendix A Appendix

We provide details of our datasets (Section A.1) and experimental results
(Section A.2).

Stats | OriginalTrain | HardTrain | MediumTrain | EasyTrain | OriginalDev | OriginalTest
---|---|---|---|---|---|---
CER | $NA$ | 37.11 | 26.19 | 16.14 | $NA$ | $NA$
MaxLen | 44 | 82 | 89 | 44 | 35 | 37
AvgLen | 14.41 | 14.30 | 14.35 | 14.41 | 14.33 | 14.60
Num | 120098 | 120098 | 120098 | 120098 | 14326 | 7176
Table 8: Statistics of AISHELL-1. ‘Worse’,’Normal’,’Better’ are three notation
we use to represent different error levels. ’NA’ means not char error rate
because original datasets are assumed to be clean. ’Num’ represents the data
volume of each set.

###  A.1 Details of Implementation

1: Initialization:Labeled data $L$, Unlabeled data $U$, total dev data
$D_{total}$, dev_select data $D_{select}$;

2: Initialize selected_train_set $L_{chosen}=L$;

3: Initialize the teacher model $M_{teacher}$, selection model $S$;

4: Train the teacher model $M_{teacher}$ on $L_{chosen}$;

5: Infer on $U,D_{select}$ using $M_{teacher}$;

6: Infer signals $sg$ =
$[\text{IL},\text{EMAL},\text{HE},\text{IE},\text{FLS}]$ on $U,D_{select}$
using $M_{teacher}$ and get signals $sg_{unlabeled},sg_{select};$

7: Train $S$ on $D_{select}$;

8: $P_{select}$ = $S(U,sg_{unlabeled})$;

9: $sr=argmax(P_{select})$;

10: $L_{chosen}=L\cup U[sr]$;

11: Train the student model $M_{student}$ on $L_{chosen}$;

12: Make the student a teacher and go back to step 5 and repeat.

Algorithm 2 Adaptation of SENT on classical self-training.

####  A.1.1 Baselines

In the first experiment of text classification, we also take the same
baselines which consider rules and report the same results as utilized in
ASTRA ( Karamanolakis et al. (2021)). (a) Majority predicts the majority vote
of the rules with ties resolved by predicting a random class. (b)
Snorkel+Labeled (Ratner et al. (2017)) trains classifiers using weakly-labeled
data with a generative model. . (c) L2R (Ren et al. (2018)) learns to re-
weight noisy labels from rules by meta learning. (d) PosteriorReg (Hu et al.
(2016)) uses rules as soft constraints for training by posterior
regularization. (e) ImplyLoss (Awasthi et al. (2020)) learns both instance-
specific and rule specific weights by minimizing an implication loss (h) ASTRA
(Karamanolakis et al. (2021)) introduces an rule attention network to leverage
multiple sources of weak supervision with trainable weights to compute soft
weak labels for unlabeled data.

For ASR baselines, Vanilla is a naive encoder-decoder transformer network
without any denoising moduels. All following baselines and our approache are
build on top of the vanilla model. (b) Co-Teaching+ trains two networks with
each network selecting its small-loss samples as clean samples for its peer
network. (c) L2R is the same as mentioned before. (d) RoCL starts learning
with easy and clean samples and gradually moves to learn noisy-labeled data
with pseudo labels produced by a time-ensemble of the model and data
augmentations.(e) SELFIE selects refurbishable samples based on the entropy of
model predictions and refurbs the samples with model predictions.

###  A.2 Details of Experiments

For ASR models, the transformer model contains 12 layers of encoder and 6
layers of decoder. For each transformer block, the number of heads in the
multiheadattention module is 4. The dimension of the encoder and decoder input
is 256. The dimension of the feedforward network is 2048. We use
80-dimensional filter bank coefficient as input features. The hyper parameter
for training is show in Table 9. Batch_size_in_s2 means the maximum allowed
length of audio in seconds in one batch. History_length represents the maximum
allowed length for stored history predicted labels. These history predictions
are used to calculate entropy.

It should be noted that in both text classification as ASR tasks, we split the
total development set into dev_select and dev_eval, where dev_select is used
to train the selection model and dev_eval is used to evaluate the selection
model.

HP | HardTrain | MediumTrain | EasyTrain
---|---|---|---
Max_LR | 5e-4 | 3e-4 | 3e-4
Min_LR | 5e-6 | 1e-6 | 1e-6
Warmup_step | 20000 | 20000 | 20000
Max_steps | 80000 | 50000 | 50000
Batch_size | 300 | 300 | 300
Batch_size_in_s2 | 500 | 500 | 500
History_length | 18 | 12 | 12
Table 9: Hyperparameter for SENT on ASR. HP means hyperparameters.

There are three details worth to noice in ASR: a) we perform an additional
correction module for ASR. The correction module has the same architecture as
the selection module. Correction module takes the same signal as selection
module, and it outputs three weights which sum to one. The weights are
assigned to the noisy labels (NL), model predicted probabilities (Pred),
accumulated corrected labels respectively. The corrected label(CL) at T-th
epoch is:

$CL^{T}=w_{1}*NL+w_{2}*Pred+w_{3}*CL^{T-1}$ (8)

after correction, we will perform the normal selection module to select clean
labels from corrected labels. b) Since ASR is sequence level problem, we can
not correct and select each token independently which would ignore the word
dependencies. We first align the predicted word sequence to the noisy target
sequence accoding to their longest common sequence. Then we will correct and
select the token that are not common in both sequence. c) As ASR is a
generation problem and the length of input and output is different, we do not
extract FSL as a feature for our approach.

###  A.3 Details of Ablation Study

Model | Hard | Medium | Easy
---|---|---|---
Vanilla | 30.06 | 20.21 | 14.42
Co-Teaching+ | 29.67 | 20.02 | 13.71
L2R | 27.79 | 19.08 | 14.12
RoCL | 24.50 | 16.34 | 13.22
Selfie | 24.40 | 17.22 | 13.01
ours | 23.97 | 16.01 | 11.96
Table 10: Comparison with other baselines on dev set of AISHELL-1

####  A.3.1 The Influence of Selection Threshold On The Final Performance

In practice, we find that choosing a proper threshold for selection model
might have some influence on the final performance. In detail, we choose FX-
score as the target to choose the threshold which yields best FX-score on the
dev_eval set, and use this threshold to select the samples from the unlabeled
data based on the output of the selection model. We investigate the influence
on final performance by changing the X of FX-score on YOUTUBE and SMS. The
computation of this metric is displayed as follow:

$FX-score=\frac{(1+X^{2})*precision*recall}{X^{2}*preision+recall}$ (9)

Noted that this metric becomes F1-score if we set X as 1. The X measures the
preference of precision to recall. If X approaches 0, it becomes precision. If
X approaches infinite, it becomes recall. As shown in Figure 4, the
performance gradually decreases as the X grows, which implicitly indicates
that precision matters more than recall when we are going to select samples
from unlabeled data.

Figure 4: The influence of X of FX-score as the selection threshold on the
final performance.

####  A.3.2 The Performance Under The Few-shot Setting

We also investigate the performance of IMDB and YOUTUBE under the few-shot
learning setting. The results are shown in Figure 5.

(a) YOUTUBE

(b) IMDB

Figure 5: Evaluation on Youtube, SMS and IMDB under the few-shot setting.

####  A.3.3 Case Study: Signal Difference On Train and Development Sets.

We show the difference of signals on training and development sets in Figure
6, 7, 8. We can see that the all signals show close statistics on both sets.
This indicates that our noise transfer approach holds and has a good
performance. This phenomena exists in all three error levels.

Figure 6: Mean of signals on hard level training and development set Figure 7:
Mean of signals on medium level training and development set Figure 8: Mean of
signals on easy level training and development set

####  A.3.4 Influence of Signals On AISHELL-1 Dev

We show the influence of different signals on AISHELL-1 development set in
Table 11. We can see that the tendency is same as the tendency on the test
set.

Signal | Hard | Medium | Easy
---|---|---|---
All | 23.97 | 16.01 | 11.96
-EMAL | 24.35 | 16.43 | 12.45
-IL | 25.48 | 17.55 | 13.71
-HE | 25.89 | 18.01 | 13.99
Table 11: Ablation experiments for signals on dev set of AISHELL-1. Each line
is removing one signal from previous line.


$\alpha=[1,1,1,1,2,1,1,4,4,3],\beta=[4,4,4,4,4,4,4,4,4,2]$.

#### Model Parameters

  1. 1.

Discriminator and generator models for the GAN, MO-PaDGAN, and DTAI-GAN are 2
and 3 hidden-layer neural networks respectively with 128 parameters and leaky
ReLU activation with slope coefficient $\alpha=0.2$.

  2. 2.

All PaDGAN and DTAI-GAN training parameters are identical to [60] except
$\gamma_{0}=5$ and $\gamma_{1}=0.5$

  3. 3.

The latent vectors are four-dimensional

  4. 4.

Discriminator and generator optimizers are Adam [96] with learning rate
$1e-4$.

  5. 5.

The models are trained over 50000 randomized batches of 8 samples.

  6. 6.

MO-PaDGAN and DTAI-GAN used differentiable performance surrogate regressors
for performance evaluations of generated designs. DTAI-GAN used a
differentiable constraint satisfaction surrogate classifier for constraint
satisfaction likelihoods. Performance prediction and constraint satisfaction
prediction networks were optimal neural networks identified through Bayesian
optimization in [12].


#  Primordial black holes in interferometry

L. Stodolsky,
Max-Planck-Institut für Physik (Werner-Heisenberg-Institut)
Föhringer Ring 6, 80805 München, Germany

###### Abstract

If there is a population of black holes distributed randomly in space, light
rays passing in their vicinity will acquire random phases. In the “two-slit”
model of an interferometer this can, for a high density of black holes, lead
to a diffusion in the phase difference between the two arms of the
interferometer and thus to a loss of coherence or “visibility”in
interferometric observations. Hence the existence of “fringe constrast” or
“visibility” in interferometric observations can be used to put a limit on the
possible presence of black holes along the flight path.

We give a formula for this effect and consider its application, particularly
for observations in cosmology. Under the assumption that the dark matter
consists of primordial black holes, we consider sources at high z, up to the
CMB. While the strongest results are for the CMB as the most remote source,
more nearby sources at high $z$ lead to similar effects.

The effect increases with the baseline, and in the limiting case of the CMB we
find that with earth-size baselines a non-zero “visibility” would limit the
mass of possible primordial black holes, to approximately $M/M_{\odot}\leq
10^{-1}$. Although such limits would not appear to be as strong as those
obtained, say from microlensing, they involve a much different methodology and
are dominated by very early times (see Table 1). Longer baselines lead to more
stringent limits and in principle with extreme lengths, the method could
possibly find positive evidence for primordial black holes. In this case,
however, all other kinds of phase averaging would have to be constrained or
eliminated.

##  1 Introduction

When there are scattering centers between a source of radiaton and an
observer, the associated deflections lead to a spread of the incoming
directions. The usual concern with this effect is the intrinsic limitation on
the angular resolution it presents for the observer. But if the nature of the
intervening medium or scattering centers is unknown, it can also be used to
obtain information on these.

In a recent paper[1] we discussed how this could be used to limit, or perhaps
to discover primordial black hole dark matter. With the CMB as the source [2]
and assuming the dark matter composed of black holes, we found that there is a
limit on the angular resolution
$\delta\alpha_{lim}\approx{3}\times{10^{-7}}\sqrt{M/M_{\odot}}~{}rad$.
Combining this with the milliradian resolution demonstrated [3] in the
observations of “acoustic peaks ” one concludes that the (average) mass $M$ of
possible black holes is constrained to about $M/M_{\odot}\leq 10^{7}$.

The scattering process in question is the gravitational deflection of photons,
as in the bending of light by the sun,

$\delta\alpha=\frac{4GM}{b}=\frac{2r_{s}}{b}={6.3}\times{10^{-13}}(ly/{b})\,(M/M_{\odot})\,,$
(1)

with $\delta\alpha$ the deflection angle in radians, $b$ the impact parameter
of the ray or photon, and $r_{s}=2GM$ the schwarzchild radius associated with
$M$, the mass of the black hole or gravitating object (c=1 units, ly=light
year, $M_{\odot}$=solar mass).

At present the most likely explanation for the dark matter seems to be a new
kind of elementary particle, which might be detected in the laboratory.
especially by the cryogenic technique [4], But black holes are an interesting
alternative [5]. It would evidently be interesting to be able to extend our
argument to smaller angles, and so to eliminate, or conceivably to discover,
smaller mass primordial black holes.

However, there appears to be a difficulty. There are very likely no very fine,
prominent features on the CMB, analogous to the “acoustic peaks”, but
subtending very small angles. How then, could one exhibit a high resolution,
or the lack of it, when there are no fine feaures to observe?

##  2 Principle

Here we would like to propose an solution to this question, and to apply it to
the possible primordial black holes at the CMB. As briefly alluded to in [1],
the answer lies in examining the coherence of the wavefront at large
separations. If instead of a perfect plane wave from some source, one has an
incoherent sum of slightly differing incoming directions, the phases at widely
separated points will become uncorrelated or incoherent. That is, the
coherence over the wavefront at large separations is closely connected to the
angular spread, the limiting resolution, of the incoming waves.

In long baseline interferometry, which is used in astronomy to achieve high
angular resolution, one has a form of the two-slit interference arrangement
[6]. The high resolution arises from a long baseline $B$ between two “slits”
or receivers so that
$\delta\alpha_{min}\approx\frac{1}{2\pi}\frac{\lambda}{B},$ with $\lambda$ the
wavelength of the radiation. For long baseline interferometry to function it
is evidently necessary that the two paths from the source to the “slits” be
coherent. If the two rays which should ultimately interfere encounter
different perturbations or scatterings along the way to the receivers, there
will be a loss of interference and a breakdown of the method, giving the
instrinsic limiting resolution.

When $B$ is small and the two paths traverse essentially the same environments
they will be coherent. As $B$ increases, at some point this will no longer be
true and interference breaks down. In the following we examine how this would
happen due to the presence of black holes along the flight path from the CMB.
The resulting loss of “visibility” or “fringes”, or the absence of such loss,
can then be used as a limit or an indication for the black holes.

##  3 Calculation

The basis for the calculation will be the formula for the phase acquired by a
photon or light ray traversing a weak gravitational field. In addition to the
plane wave term in the phase, their is an extra gravitational phase $\phi_{G}$
which according to general arguments [7] is

$\phi_{G}=\tfrac{1}{2}\int h_{\mu\nu}k^{\mu}dx^{\nu}$ (2)

where $h_{\mu\nu}$ is a small deviation from an overall metric tensor, $k$ the
four-momentum or wave vector of the photon, and the integration is along the
path of the ray.

We need the difference in phases $\delta\phi_{G}$ for two parallel rays 1,2 in
the vicinity of a black hole, with impact parameters $b_{1},b_{2}$.
Integrating the difference according to Eq 2 with the Schwarzschild metric and
taking the difference $\delta b=b_{1}-b_{2}$ to be small compared to that for
the pair as a whole, $b$, one finds,

$\delta\phi_{G}=\omega r_{s}\delta
b\int_{-\infty}^{\infty}dz\,\frac{d}{db}\frac{1}{\sqrt{z^{2}+b^{2}}}=2\omega
r_{s}\frac{\delta b}{b}\,,$ (3)

where $r_{s}$ is the Schwarzschild radius of the body, and $\omega=k$ the
frequency of the radiation. The phase difference increases linearly with the
ray separation $\delta b$. The assumption $\delta b/b<<1$ seems justified
since while $\delta b$ will be given by the baseline, and so be on the of
order of earth or perhaps solar system size, we anticpate $b$ on the order of
light years or more.

There is a close relation between the classic bending formula Eq 1 and Eq 3.
When $b_{1}$ and $b_{2}$ lie along a radius vector, this calculation for
$\delta\phi_{G}$ is the same as needed for the bending of a wavefront, where
the scattering angle is given by the transverse derivative of the phase. That
is, for a scattering process with long range forces and smoothly varying phase
$\phi(b)$ one has for the bending of a plane wave
$\delta\alpha=k_{T}/k=\frac{1}{\omega}\frac{d\phi(b)}{db}$ where $k_{T}$ is
the transverse momentum to the photon. In Eq 3 this amounts to dividing by
$\omega\,\delta b$ which yields, in fact, Eq 1.

##  4 Random $\delta\phi_{G}$

A pair of rays starting at a point on the source and arriving at the two
‘slits’ or receivers will experience different fields and so aquire different
phases along their paths. If the dark matter is made of black holes, presumed
random in their locations, this will lead to a random $\delta\phi_{G}$, which.
if it is large enough, will lead to loss of coherence or “visibility” in the
interference pattern.

In view of the small values of the factors in Eq 3, $\delta\phi_{G}$ from a
single passage near a black hole will be small. However with many repeated
passages, which would occur if the dark matter is made of black holes, it is
possible that a substantial $\delta\phi_{G}$ will acccumulate.

The sign of $\delta\phi_{G}$ in a single passage depends on the orientation of
the ray pair with respect to the radius vector from the black hole, sometimes
ray 1 will be closer to the black hole and sometimes ray 2, so that the
average value of $\delta\phi_{G}$ will be zero. However the accumulated
$\delta\phi_{G}$ will fluctuate, with the character of a random walk. In such
problems the expected magnitude of the quantity, is characterized by the
variance, here $\overline{(\delta\phi_{G})^{2}}$ and it is this quantity that
we wish to study.

When the two rays are combined in the detector, the $\delta\phi_{G}$ will lead
to fluctuating factors such as $cos(\delta\phi_{G})$ multiplying the
interference term. The significance of $\overline{(\delta\phi_{G})^{2}}$ may
be seen with a gaussian probability distribution $P_{gauss}$ [8] for
$\delta\phi_{G}$:

$\overline{cos(\delta\phi_{G})}=\int
d(\delta\phi_{G})\,cos(\delta\phi_{G})\,P_{gauss}=e^{-\tfrac{1}{2}\overline{(\delta\phi_{G})^{2}}}$
(4)

When $\overline{(\delta\phi_{G})^{2}}$ becomes larger than one, interferences
are “washed out”. We shall use this condition,

$\overline{(\delta\phi_{G})^{2}}>1$ (5)

to estimate when coherence is lost and the effects of the possible black holes
would become significant.

##  5 Calculation of $\overline{(\delta\phi_{G})^{2}}$

To calculate the total variance we take the individaul black holes to be
uncorrelated and use the principle of the addition of variances. We do this by
calculating the variance from one encounter as given by Eq 3 and then multiply
by the number of enounters along the flight path.

These enounters will take place in the general background spacetime, which we
take to be that of the FRW metric given by $ds^{2}=dt^{2}-a^{2}d{\bf r}^{2}$
with expansion factor $a=(t/t_{now})^{2/3}$, $t_{now}={2.9}\times{10^{17}}s$
[8]. In this connection one notes that Eq 2 is a scalar in general relativity,
being the contraction of two tensors. Thus we may use the evaluation Eq 3,
carried out for an ambient flat space, in the local frames of black holes
along the flight path and simply add the various $(\delta\phi_{G})^{2}$. Of
course the parameters needed, such as the frequency, and the separation of the
two rays, will depend on the location in the general background spacetime.

There is a difficulty, however, associated with the long ranged nature of the
gravitational field. In order to define the probability of an “encounter” one
must assign some range or cross section to the field around each constituent,
here the black holes. But for the gravitational field there is no such
parameter, the force has an infinite range. We shall handle this problem by
assuming that the picture of a single, independent constituents holds up to
some distance $b_{max}$, with a corresponding cross section $\sigma=\pi
b_{max}^{2}$. This seems a reasonable procedure since the contributions to
$\overline{(\delta\phi_{G})^{2}}$ are largely at small $b$. The $b_{max}$
parameter can be plausibly be taken to be the typical distance between black
holes, We shall see below that the choice of $b_{max}$ enters into the final
result only logarithmically.

###  5.1 Number of Encounters

With this understanding, we first consider the number of black holes
encountered in an interval of cosmic time $dt$. Along the flight path, the
number of black holes per unit area in a time interval $dt$ is
$d\rho_{2}=\rho(t)\,dt$, where $\rho(t)$ is the number density. Associating
with each black hole a “cross section ” $\sigma$, one has for the number of
encounters $dN$ in a time $dt$

$dN=\sigma d\rho_{2}=\rho(t)\,\sigma\,dt$ (6)

###  5.2 Single Passage

For the $\overline{(\delta\phi_{G}^{2})}_{single}$ associated with a single
encounter, we assume the $b$ for the ray pair to be uniformly distributed
across the area of $\sigma$, integrate the square of Eq 3 over this area and
divide by the area.

Angular Factor: First we consider the angular factor involved in $\delta b$,
which is the origin of the fluctuating sign. Let $\bf b_{1}$ and $\bf b_{2}$
be the respective vectors from the black hole to the points of closest
approach, and $\bf d=(b_{1}-b_{2})$ the vector connecting them. Then for the
$\delta b$ needed in Eq 3 one finds

$\delta b={b_{1}-b_{2}}=2{\bf d}\cdot\frac{\bf b}{2b}=d\,cos\theta\,,$ (7)

with $\theta$ the angle between $\bf d$ and the radius vector $\bf
b=\tfrac{1}{2}(b_{1}+b_{2})$. This relation follows from writing $\bf
b_{1},b_{2}=b\pm\tiny\frac{1}{2}d$, squaring, taking the difference, and using
$b_{1}+b_{2}\approx 2b$. For the nearby universe and the line of sight
perpendicular to the baseline, $d$ is simply the baseline $B$.(In the
following we take the direction of observation to be a right angles to the
baseline, otherwise a trignometric factor can be included. ) For the angular
integration one thus obtains

$\int_{0}^{2\pi}d\theta(\delta b)^{2}=\pi d^{2}$ (8)

Integration Over b: For the remaining $b$ integration one now has

$\overline{(\delta\phi_{G}^{2})}_{single}=\frac{1}{\sigma}4\pi(\omega
r_{s}d)^{2}\int_{b_{min}}^{b_{max}}\frac{db}{b}=\frac{1}{\sigma}4\pi(\omega
r_{s}d)^{2}ln(\frac{b_{max}}{b_{min}})\,,$ (9)

with $b_{min},b_{max}$ some smallest, largest, impact parameters.

###  5.3 General Formula

Eq 9 must now be multiplied by the number of encounters in an interval of
cosmic time.

$\displaystyle d\overline{(\delta\phi_{G}^{2})}$ $\displaystyle=$
$\displaystyle dN\times\overline{(\delta\phi_{G}^{2})}_{single}$
$\displaystyle=$ $\displaystyle dt\rho\,\sigma\frac{1}{\sigma}4\pi(\omega
r_{s}d)^{2}ln(\frac{b_{max}}{b_{min}})$ $\displaystyle=$ $\displaystyle
dt\,4\pi\rho\,\,(\omega
r_{s}d)^{2}ln\bigl{(}\frac{b_{max}}{b_{min}}\bigr{)}\,.$ (11)

As anticipated, $\sigma$ has cancelled and the $b$ limits are only in the
logarithm. The appearance of the logarithm is an interesting feature of the
present problem, in contrast to the more familiar situation of particles or
photons passing through ordinary neutral matter. There one deals with the
shielded coulomb field or the short ranged nuclear force, while here one has
the unshielded, long range, gravitational force.

It remains to insert values for $b_{max},b_{min}$. For $b_{max}$ we take the
distance between the black holes, namely $\rho^{-1/3}$. Beyond this distance
one must consider the summed fields from many objects, leading to smooth and
relatively weak fields. For the lower limit we take $b_{min}=r_{s}$.

The final result is thus

$\displaystyle\overline{(\delta\phi_{G}^{2})}$ $\displaystyle=$ $\displaystyle
4\pi\,r_{s}^{2}\int
dt\,\rho(t)\biggl{(}\omega(t)d(t)\biggr{)}^{2}ln\bigl{(}\frac{\rho^{-1/3}(t)}{r_{s}}\bigr{)}$
$\displaystyle=$ $\displaystyle 16\pi^{3}\,r_{s}^{2}\int
dt\,\rho(t)\biggl{(}d(t)/\lambda(t)\biggr{)}^{2}ln\bigl{(}\frac{\rho^{-1/3}(t)}{r_{s}}\bigr{)}$

writing $\omega d=2\pi d/\lambda$, and where we have explicitly indicated
those quantities which might depend on $t$.

Parameters: For quantitative estimates, we shall use light years (ly) for
lengths and solar masses ($M_{\odot}$) for mass. Thus the number density is
$\rho(t)=\rho_{o}(t)(M_{\odot}/M)/ly^{3})$ for a dark matter mass density
$\rho_{o}(t)M_{\odot}/ly^{3}$ , where $\rho_{o}(t)$ is a dimensionless density
parameter. We will be assuming that a well defined mass or average mass can be
assigned to the black holes, if this is not true the arguments may have to be
reconsidered. Furthermore, anticipating the use of earth-size baselines and
microwave wavelengths, as in ref.[9], we introduce a length parameter

$d_{o}={1.0}\times{10^{4}}\,km$ and a wavelength parameter
$\lambda_{o}={1.0}\times{10^{-1}}cm$. Collecting all factors and inserting
$r_{s}=3.0\,km(M/M_{\odot})={3.2}\times{10^{-13}}(M/M_{\odot})ly$ as in Eq 1,
Eq 5.3 becomes

$\displaystyle\overline{(\delta\phi_{G}^{2})}$
$\displaystyle=({5.1}\times{10^{-3}})(M/M_{\odot})\int
dt/y\,\rho_{o}(t)\biggl{(}\frac{d/d_{o}}{\lambda/\lambda_{o}}\biggr{)}^{2}ln\bigl{(}\frac{\rho^{-1/3}(t)}{r_{s}}\bigr{)}$
$\displaystyle=$ $\displaystyle({5.1}\times{10^{-3}})(M/M_{\odot})\int
dt/y\,\rho_{o}(t)\biggl{(}\frac{d/d_{o}}{\lambda/\lambda_{o}}\biggr{)}^{2}\biggl{(}(1/3)ln\rho_{o}(t)-(2/3)ln(M/M_{\odot})+29)\biggr{)}$
$\displaystyle=$ $\displaystyle({1.5}\times{10^{-1}})(M/M_{\odot})\int
dt/y\,\rho_{o}(t)\biggl{(}\frac{d/d_{o}}{\lambda/\lambda_{o}}\biggr{)}^{2}\biggl{(}1+(0.011)ln\rho_{o}(t)-(0.023)ln(M/M_{\odot})\biggr{)}$

( Since we have $c=1$ units and are dealing with light rays, one may use a
distance label $r$ instead of $t$. In that case one writes $\int dt/y\to\int
dr/ly$). We also note that with this notation the dimensionless quantity $\int
dt/y\,\,\rho_{o}$ is simply the coefficient in the mass column density; that
is, the mass column density is $\int dt/y\,\,\rho_{o}\times M_{\odot}/ly^{2}$.

Because of the small coefficient of the $ln\,\rho_{o}(t)$ term, it may be
possible to replace it by some typical value and move it outside the integral.
In that case Eq 5.3 becomes

$\displaystyle\overline{(\delta\phi_{G}^{2})}$ $\displaystyle\approx$
$\displaystyle C\,\int
dt/y\,\rho_{o}(t)\biggl{(}\frac{d(t)/d_{o}}{\lambda(t)/\lambda_{o}}\biggr{)}^{2}$
(14) $\displaystyle C$ $\displaystyle=$
$\displaystyle({1.5}\times{10^{-1}})(M/M_{\odot})\biggl{(}1+(0.011)ln\rho^{typ}_{o}-(0.023)ln(M/M_{\odot})\biggr{)}$

As should be expected, the effect vanishes with $M\to 0$, as the mediun
becomes smooth and less “lumpy”. We note that a tendency of the black holes to
“clump” around the time of the CMB would amount to an increase of their mass
for the given amount of dark matter and tendentially increase the effect.

##  6 CMB

We now consider the application of the formula to rays traveling from the CMB
to the Milky Way. For the density we take

$\rho_{0}\approx\frac{1}{a^{3}(t)}({1.0}\times{10^{-9}})\,,$ (15)

as follows from taking the present cosmological dark matter energy density at
1/4 the critical density [8], and applying the cosmological expansion factor
$a(t)=(t/t_{now})^{2/3}$.

To simplify the integral we replace the time dependent $ln\,a(t)$ arising in
$ln\,\rho_{o}(t)$ by its value at $t_{cmb}$, namely $-7.0$. Since $ln\,a(t)$
varies from zero to this value in the time integration, this simplification
gives a small underestimate of the total integral. This gives
$ln\,\rho^{typ}=0.3$ and one then has for $C$ in Eq 14
$C=({1.5}\times{10^{-1}})(M/M_{\odot})\bigl{(}1-(0.023)ln(M/M_{\odot})\bigr{)}$
and so

$\overline{(\delta\phi_{G}^{2})}\approx{1.5}\times{10^{-10}}(M/M_{\odot})\bigl{(}1-(0.023)ln(M/M_{\odot})\bigr{)}\int
dt/y\,\frac{1}{a^{3}(t)}\biggl{(}\frac{d(t)/d_{o}}{\lambda(t)/\lambda_{o}}\biggr{)}^{2}$
(16)

Since $\int dt$ will be at least $10^{10}$ years,
$\overline{(\delta\phi_{G}^{2})}$ can easily be greater than one, and satisfy
the condition Eq 5 .

###  6.1 Time Integration

We are thus left with the integral

$I=\int_{t_{cmb}}^{t_{now}}dt/y\,\frac{1}{a^{3}(t)}\biggl{(}\frac{d(t)/d_{o}}{\lambda(t)/\lambda_{o}}\biggr{)}^{2}$
(17)

to evaluate.

Frequency Shift: In the rest frame of a black hole at time $t$, one has the
frequency or wavelength red shifted, so that if $\omega$ is the frequency of
observation, one has $\omega\to\omega/a(t)$ or $\lambda\to a(t)\lambda$.

Ray Separation: The remaining question is the ray separation $d(t)$. The rays
must originate from a common point to interfere. In euclidean space, without
general relativistic considerations, one would have
$d(t)=\frac{B}{t_{now}-t_{cmb}}(t-t_{cmb})$, where we chose the constants to
give $d=0$ at emission and $B$ at the baseline of the detector.

To transfer this reasoning to the general relativistic situation, we use the
“conformal time” $\eta(t)=3t_{now}x^{1/3}$, and introduce the dimensionless
time variable $x$ so that

$x=t/t_{now}~{}~{}~{}~{}~{}a=x^{2/3}~{}~{}~{}~{}~{}\eta=3\,t_{now}x^{1/3}~{}~{}~{}~{}~{}x_{cmb}={2.7}\times{10^{-5}}~{}~{}~{}~{}~{}$
(18)

In terms of $\eta$ one has $ds^{2}=a^{2}(d\eta^{2}-d{\bf r}^{2})$. Light rays
travel in “straight lines” in these coordinates, so for the coordinate
separation, one has the euclidean result as before,
$d_{coord}(\eta)=\frac{B}{\eta_{now}-\eta_{cmb}}(\eta-\eta_{cmb}).$ However,
we require the physical separation so $a(t)d_{coord}(t)$ should be used for
$d$. The $a$ factors cancel then in the ratio $d/\lambda$ and the integral
becomes

$\displaystyle I$ $\displaystyle=$
$\displaystyle(t_{now}/y)\biggl{(}\frac{B/d_{o}}{\lambda/\lambda_{o}}\biggr{)}^{2}\int_{x_{cmb}}^{1}dx\,\frac{1}{x^{2}}\biggl{(}\frac{\eta-\eta_{cmb}}{\eta_{now}-\eta_{cmb}}\biggr{)}^{2}$
$\displaystyle=$
$\displaystyle(t_{now}/y)\biggl{(}\frac{B/d_{o}}{\lambda/\lambda_{o}}\biggr{)}^{2}\int_{x_{cmb}}^{1}dx\,\frac{1}{x^{2}}\biggl{(}\frac{x^{1/3}-x^{1/3}_{cmb}}{1-x^{1/3}_{cmb}}\biggr{)}^{2}$

In Eq 6.1 we have a convienient separation into factors depending on the
observational arrangenment ($B,\lambda$) and a “cosmological integral”
$\int_{x_{cmb}}^{1}...$ independent of the setup.

###  6.2 CMB Result

Examination of the integrand shows that with
$x^{1/3}_{cmb}={3.0}\times{10^{-2}}$ from Eq 18 , it is strongly peaked around
$x\sim 10^{-4}$. Since $x_{cmb}={2.7}\times{10^{-5}}$, this shows that the
effect would come predominantely close to the CMB and that a positive
observation would support the idea that the black holes are primordial. For
the integration one finds the integrated value $\int_{x_{cmb}}^{1}...=32$. As
in [1], there is an enhancement at early times, but due to the requirement of
the convergence of the rays at their origin. to a much lower power. Combining
with Eq 16 one has finally,

$\displaystyle\overline{(\delta\phi_{G}^{2})}$ $\displaystyle=$
$\displaystyle({4.5}\times{10^{-9}})\bigl{(}M/M_{\odot}\bigr{)}(t_{now}/y)\biggl{(}\frac{B/d_{o}}{\lambda/\lambda_{o}}\biggr{)}^{2}$
$\displaystyle=$
$\displaystyle({4.1}\times{10^{1}})\bigl{(}M/M_{\odot}\bigr{)}\biggl{(}\frac{B/d_{o}}{\lambda/\lambda_{o}}\biggr{)}^{2}\biggl{(}1-({2.3}\times{10^{-2}})\,ln(M/M_{\odot})\biggr{)}\,,$

after inserting $t_{now}={2.9}\times{10^{17}}sec={0.92}\times{10^{10}}y$.

###  6.3  Sources More Nearby

We have chosen to evaluate the effect for a source at the CMB, since this is
at the greatest distance available. However interferometry on the CMB [10]
has, until now, not been with the earth-sized baselines as used by the EHT
project [9].

It may thus be of interest to inquire as to the size of the effect for more
nearby sources, say for $z\leq 10$, where there can be localized sources. To
do this, we must evaluate the integral $\int_{x_{cmb}}^{1}...$ in Eq 6.1 with
$x_{cmb}$ replaced by the $x$ of the source, which we shall call $x_{o}$.

In the Table we show the results for a few values of $x_{o}$. The first two
columns gives the ‘distance’ of the source expressed as $x_{o}$ or
equivalently the redshift factor $z_{o}$. The second column gives the location
of the peak of the integrand, in terms of its $x$ or $z$. Finally, the last
column shows the value of the integral $\int_{x_{o}}^{1}...$. For comparison
the last row shows the CMB result from the subsetion above.

The integral is given by
$\int_{x_{o}}^{1}...=(1/1-x_{0}^{1/3})^{2}(x_{0}^{-1/3}-3+3\,x_{0}^{1/3}-x_{o}^{2/3}).$

“Distance”$x_{o}$ | ”Distance” $z_{o}$ | Peak Location $x,z$ | $\int_{x_{o}}^{1}...$
---|---|---|---
$0.032$ | $8.9$ | $0.1,~{}~{}~{}~{}3.6$ | $1.4$
0.08 | 4.4 | 0.25 1.5 | 1.3
$0.10$ | $3.6$ | $0.35~{}~{}~{}~{}1.0$ | $1.2$
${2.7}\times{10^{-5}}$ | ${1.1}\times{10^{3}}$ | $10^{-4}~{}~{}~{}~{}460$ | 32
Table 1: Values of the “cosmological integral’ $\int_{x_{o}}^{1}...$ in Eq 6.1
for some sources in the range $1<z<10$, and finally in the last row, for the
CMB. The first column gives the scaled time variable $x_{o}$ for the source,
the second column its redshift factor $z_{o}$ and the third column where the
integral has its peak in terms of these quantities. One notes that while the
effect is largest for the CMB, by about a factor 20, the other estimates are
not enormously smaller.

One sees that the effect for such nearer sources would be reduced by a factor
of only about twenty or thirty compared to the CMB, despite the very great
differences in “distance”.

##  7 Galactic Contributions

On their way to the earth, rays from the CMB or other sources will also pass
through the galaxy, first presumably through a large, extended, but tenuous
dark halo, and then through some part of the more dense inner galaxy. Can
these traversals make a significant contribution to the above estimates?

A thorough answer to this question would require the use of detailed models of
the dark matter distribution in the Milky Way and its halo, and refered to a
particular line-of-sight. However a crude estimate suggests the answer is
negative, at least for the CMB.

First for the inner galaxy, the estimated dark matter density in the vicinity
of the earth [8] is $\rho_{o}={3.0}\times{10^{-4}}.$ (Units of
$M_{\odot}/ly^{3}$). Taking a typical path length of $10^{4}$ly at this
density, on finds from Eq 14 that the numerical coefficient in front of Eq 6.2
becomes ${4.5}\times{10^{-1}}$ instead of ${4.1}\times{10^{1}}$, about two
orders smaller.

For the dark halo, the density presumably drops off quickly beyond some radius
$R_{inner}$. If we crudely take the density beyond this to vary as $\sim
1/R^{2},$ as one would have with a constant rotation curve, then for the
column depth integration through the halo one has
$\rho_{inner}\int_{R_{inner}}^{\infty}dR(R/R_{inner})^{-2}=\rho_{inner}R_{inner},$
qualitatively the same or perhaps somewhat larger than that for the inner
galaxy.

It thus appears that the galactic contributions to
$\overline{(\delta\phi_{G}^{2})}$ are one or two orders less than the
cosmological Eq 6.2. In the event of a possible positive signal from more
nearby soures, as in subsection 6.3, it might be necessary to disentangle
these from the local, galactic ones.

These argument concerning the galactic contribution are of course quite
schematic, necessitating more detailed estimates. In this connection, we have
recently [2] estimated the limits arising from models of the ‘spike’ of dark
matter believed to surround the supermassive black hole in M87, using the
arguments of ref. [1], and the angular resolution of the EHT results [9].

##  8 Conclusions

The examination of “visibility” at long baselines provides a very high
sensitivity method for studying an intervening medium. We have appplied the
method to the question of primordial black holes in cosmology.

Interferometric observations on the CMB have been considered and carried out
over the last decades [10]. If observations with earth-size baselines, as in
[9], reveal coherence, a nonzero “visibility”, then according to Eq 6.2 and Eq
5, primordial black holes down to the $(M/M_{\odot})\sim 10^{-1}$ range are
ruled out. Longer baselines would of course access smaller $M$. An earth-moon
baseline, $\sim{4}\times{10^{5}}\,km$, for example, leads to $(M/M_{\odot})$
in the $\sim 10^{-4}$ range.

These values are similar but generally weaker than limits found in work of the
“macho search ” or “microlensing” type [11]. But the interferometric method
would provide a different approach to the question, and also refer to a quite
different epochs in cosmology. The microlensing work uses background stars in
the nearby universe, Andromeda or the Milky Way bulge, while the main
contribution to Eq 6.2, as said, is at very high redshift, close to the CMB.

In subsection 6.3 we have discussed applying the method to sources in the
range $1<z<10$, leading to somewhat smaller effects. This region could involve
different technologies and would cover a different region in cosmology.

##  9 Remarks

Conceivably, the method could also be used to find positive evidence for
primordial black holes. This would be evidently a more difficult matter than
the simple setting of limits, since all kinds of other phase averaging,
instrumental as well as natural, would have to be eliminated or controlled. In
this respect, the simple frequency dependence of Eq 6.2, which originates in
the achromatic behavior of photons in the gravitational field, might be
helpful.

Our discussion has been for the ‘two slit’ configuration, which is the
paradigm of astronomical interferometry [6], while contemporary long baseline
interferometry often usues many baselines together. But our essential effect,
due to intefering rays traversing different fields, should be dominated by the
largest baseline. For the more complex , multi-baseline arrangements it would
be interesting to examine possible improvements in sensitivity and statistics,
but the “two slit” estimate for the sensitivity to potential black holes
should remain roughly correct,

We stress that our considerations, or equivalently those concerning the
resolution, should not be confused with the question of the power spectrum of
the CMB. For the present purposes the role of the CMB is simply to provide a
remote “source” and we are not concerned with the nature of the CMB itself.
This point is discussed in section 5.5 of [1].

## References

  * [1] L. Stodolsky Mod.Phys.Let.36 A, no.11 (2021), arXiv:1912.01325.
  * [2] Although in ref [1] the origin of the rays is taken at the CMB, one may also consider sources in the nearby universe where a high density of dark matter is expected. In “Limits on primordial black holes from M87”, J. Silk and L. Stodolsky , Phys. Rev. D 105 063506,(2022), arXiv 2201.03591, we apply the argument to the dark matter “spike” expected to surround a Super-Massive-Black Hole, exploiting the high angular resolution of the EHT observations of M87.
  * [3] Multipoles up to $l\sim 10^{3}$ are discussed in N. Aghanim et al. [Planck Collaboration], arXiv:1807.06207 [astro-ph.CO].
  * [4] For recents results see A. H. Abdelhameed et al. [CRESST Collaboration], arXiv:1904.00498 [astro-ph.CO], Phys. Rev. D 100, 102002 (2019). For history and background , A. Drukier and L. Stodolsky, Phys. Rev. D 30, 2295 (1984). doi:10.1103/PhysRevD.30.2295; M. W. Goodman and E. Witten, Phys. Rev. D 31, 3059 (1985). doi:10.1103/PhysRevD.31.3059. A general introduction is in L. Stodolsky, Phys. Today 44N8, 24 (1991), doi:10.1063/1.881266.
  * [5] B. Carr and J. Silk, Mon. Not. Roy. Astron. Soc. 478, no. 3, 3756 (2018) doi:10.1093/mnras/sty1204 [arXiv:1801.00672 [astro-ph.CO]]. See also B. J. Carr, Astrophys. J. 201, 1 (1975), doi:10.1086/153853, P. H. Frampton, Mod. Phys. Lett. A 33, no. 31, 1850176 (2018) doi:10.1142/S0217732318501766 [arXiv:1802.05678 [gr-qc]], B. Carr, S. Clesse, J. Garcia-Bellido and F. Kuhnel, arXiv:1906.08217 [astro-ph.CO].
  * [6] A general introduction to stellar interferometry and its history is A. R. Hajian and J. T. Armstrong, A Sharper View of the Stars, Scientific American, March 2001,pg 49.
  * [7] L. Stodolsky, “Matter and Light Wave Interferometry in Gravitational Fields,” Gen. Rel. Grav. 11, 391 (1979). doi:10.1007/BF00759302
  * [8] We take statistical formulas and parameters from the Particle Data Group booklet 2016, extracted from C. Patrignani et al. [Particle Data Group], “Review of Particle Physics,” Chin. Phys. C 40, no. 10, 100001 (2016). doi:10.1088/1674-1137/40/10/100001. Note that our definition of $a(t)$ introduces a factor 3/2 between $t_{now}$ and the hubble constant.
  * [9] K. Akiyama et al. [Event Horizon Telescope Collaboration], Astrophys. J. 875, no. 1, L1 (2019) doi:10.3847/2041-8213/ab0ec7 [arXiv:1906.11238 [astro-ph.GA]],
  * [10] M. J. White, J. E. Carlstrom and M. Dragovan, “Interferometric observation of cosmic microwave background anisotropies,” Astrophys. J. 514, 12 (1999) doi:10.1086/306911 [astro-ph/9712195]. J. Kovac, E. M. Leitch, C. Pryke, J. E. Carlstrom, N. W. Halverson and W. L. Holz “Detection of polarization in the cosmic microwave background using DASI,” Nature 420 (2002), 772-787 doi:10.1038/nature01269 [arXiv:astro-ph/0209478 [astro-ph]]. C. Dickinson, “CMB interferometry,” PoS RTS2012 (2012), 034 doi:10.22323/1.163.0034 [arXiv:1212.1729 [astro-ph.CO]].
  * [11] B. Paczynski, Astrophys. J. 304, 1 (1986). doi:10.1086/164140. For limits in the solar mass range see R. A. Allsman et al. [Macho Collaboration], Astrophys. J. 550, L169 (2001) doi:10.1086/319636 [astro-ph/0011506]. H. Niikura, M. Takada, S. Yokoyama, T. Sumi and S. Masaki, “Constraints on Earth-mass primordial black holes from OGLE 5-year microlensing events,” Phys. Rev. D 99, no.8, 083503 (2019) doi:10.1103/PhysRevD.99.083503 [arXiv:1901.07120 [astro-ph.CO]]. and H. Niikura, M. Takada, N. Yasuda, R. H. Lupton, T. Sumi, S. More, T. Kurita, S. Sugiyama, A. More and M. Oguri, et al. “Microlensing constraints on primordial black holes with Subaru/HSC Andromeda observations,” Nature Astron. 3, no.6, 524-534 (2019) doi:10.1038/s41550-019-0723-1 [arXiv:1701.02151 [astro-ph.CO]].