L[1]>m#1 C[1]>m#1 R[1]>m#1

# OLKAVS: An Open Large-Scale Korean Audio-Visual Speech Dataset

Jeongkyun Park, Jung-Wook Hwang, Kwanghee Choi, Seung-Hyun Lee, Jun Hwan Ahn,
Rae-Hong Park, Hyung-Min Park This work was partly supported by National
Information Society Agency (NIA) grant (AI HUB voice and motion data - lip
voice) and Institute of Information & communications Technology Planning &
evaluation (IITP) grant funded by the Korea government (MSIT) (No.
2022-0-00989, Development of artificial intelligence technology for multi-
speaker dialog modeling). The audio-visual dataset collection project was
proposed based on the National Information Society Agency (NIA)’s request for
proposal (AI HUB voice and motion data - lip voice), funded by the Korea
government (MSIT). After obtaining consent from the participants, the dataset
was collected and published on https://aihub.or.kr, an official portal that
discloses various AI data. All of this was done under the supervision of the
NIA.J. Park is with the Department of Artificial Intelligence, Sogang
University, Seoul 04107, Republic of Korea (e-mail: park32323@gmail.com).J.-W.
Hwang, K. Choi, S.-H. Lee, R.-H. Park, and H.-M. Park are with the Department
of Electronic Engineering, Sogang University, Seoul 04107, Republic of Korea
(e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>hpark@sogang.ac.kr).R.-H. Park
is also with ICT Convergence Disaster/Safety Research Institute, Sogang
University, Seoul 04107, Republic of Korea.H.-M. Park is also with the
Department of Artificial Intelligence, Sogang University, Seoul 04107,
Republic of Korea.J. H. Ahn is with Mindslab Inc., Gyeonggi-do 13493, Republic
of Korea (e-mail: jay@mindslab.ai).

###### Abstract

Inspired by humans comprehending speech in a multi-modal manner, various
audio-visual datasets have been constructed. However, most existing datasets
focus on English, induce dependencies with various prediction models during
dataset preparation, and have only a small number of multi-view videos. To
mitigate the limitations, we recently developed the Open Large-scale Korean
Audio-Visual Speech (OLKAVS) dataset, which is the largest among publicly
available audio-visual speech datasets. The dataset contains 1,150 hours of
transcribed audio from 1,107 Korean speakers in a studio setup with nine
different viewpoints and various noise situations. We also provide the pre-
trained baseline models for two tasks, audio-visual speech recognition and lip
reading. We conducted experiments based on the models to verify the
effectiveness of multi-modal and multi-view training over uni-modal and
frontal-view-only training. We expect the OLKAVS dataset to facilitate multi-
modal research in broader areas such as Korean speech recognition, speaker
recognition, pronunciation level classification, and mouth motion analysis.

###### Index Terms:

Audio-visual speech datasets, multi-view datasets, lip reading, audio-visual
speech recognition, deep learning

##  I Introduction

Humans comprehend speech in a multi-modal manner in real-world communication,
not only relying on listening to utterances but also reading speakers’ faces
and lips to improve perception [1, 2]. Inspired by this, many audio-visual
speech datasets have been introduced, capturing both utterances and talking
faces in a synced manner (refer to Table I for more details). These datasets
enable the research on various speech-related applications, such as noise-
robust speech recognition [3], applications for the deaf and hard-of-hearing
via lip reading [4, 5], mouth motion analysis [6], and speaker recognition
[7]. To further embolden future research, it is essential to develop datasets
with more diverse situations, such as various speakers [8, 5, 9], languages
[8, 10, 7], sentences [5, 9], and multiple viewpoints of the head [6, 11].

Despite the increasing attention to the audio-visual speech domain, existing
datasets have a number of limitations. 1) First and foremost, most datasets
only focus on English alone, with a few exceptions in Mandarin [8, 12],
Cantonese [13], and German [10]. Notably, VoxCeleb2 [7] contains multilingual
speech, but there are no transcriptions available. 2) Many large-scale
datasets often rely on repurposing publicly available videos on the web to
increase their sizes, such as TV programs [5, 11], parliament records [10],
Youtube channels [7], and TED videos [9]. However, even though the data
directly reflects real-world usages, they introduce inherent dependency on
various prediction models within the data pre-processing pipeline. For
example, many employ face detection models inside the pipeline, providing
videos composed of cropped faces with the original videos unavailable [10, 5,
11, 9, 12]. Furthermore, in some cases, their sources prohibit public use due
to copyright concerns [4, 14]. 3) Many datasets with multi-view videos on a
speaker are too small for modern data-hungry deep learning models, often with
less than 100 hours of speech [15, 6, 16]. The notable exception is the MV-LRS
dataset with 777 hours of various face rotations [11]. However, as the dataset
was constructed from web videos, it only shows one view at a time, unlike
other datasets that offer synchronous multi-view videos for a single speaker.

TABLE I:  Summarized statistics of various audio-visual datasets. The largest values are written in bold. Hours, #Utt., and #Subj. of LRS2-BBC and LRS3-TED are approximations that can be slightly larger due to potential overlaps in the dataset. Refer to Fig. 1 for the explanation of frontal, diagonal, and cross views.  Database | Hours | #Utt. | #Subj. | Views | Language | Content
---|---|---|---|---|---|---
AVLetters [17] | - | 780 | 10 | frontal | English | isolated alphabets (A-Z)
CUAVE [18] | - | 7.9K | 36 | frontal | English | isolated/connected digits
GRID [19] | 28.3 | 34K | 34 | frontal | English | command sentences
OuluVS [20] | - | 817 | 20 |  | frontal
---
English |  | phrases
---
OuluVS2 [6] | - | 1.5K | 53 |  | 5 views
---
(horizontal)
English |  | continuous digits,
---
phrases,
TIMIT sentences
AVICAR [15] | - | 59K | 86 |  | 4 views
---
(horizontal, i.e.,
left, center, right)
English |  | isolated digits/letters,
---
phone numbers,
TIMIT sentences
LRW [4] | 173 | 539K | - | frontal | English | words
LRS2-BBC [14] | 224.5 | 144K | - | frontal | English | sentences
LRS3-TED [9] | 438 | 152K | 9,545 | frontal | English | sentences
MV-LRS [11] | 777.2 | 504K | - |  | 5 views
---
(horizontal)
English | sentences
VoxCeleb1 [21] | 352 | 154K | 1,251 | frontal | English | speaker info. only
VoxCeleb2 [7] | 2,442 | 1.1M | 6,112 | frontal | Multilingual | speaker info. only
GLips [10] | 81 | 250K | - | frontal | German | words
CI-AVSR [13] | 8.3 | 5K | 30 | frontal | Cantonese | command sentences
LRW-1000 [8] | 508 | 718K | - | frontal | Mandarin | words
MISP2021 [12] | 141.2 | - | 263 |  | 3 distances
---
(far, middle, near)
Mandarin | sentences
Jo et al. [16] | - | 168 | 1 |  | 7 views
---
(horizontal)
Korean | syllable, consonant, vowel
Lee and Park [22] | - | 4.5K | 56 | frontal | Korean | digits, words
Lee et al. [23] | - | 3.2K | 9 | frontal | Korean | words
OLKAVS |  | Video:
---
5,750
Audio:
1,150
2.5M | 1,107 |  | 9 views
---
(frontal
\+ diagonal
\+ cross)
Korean |  | sentences
---

To mitigate the limitations, we developed a new audio-visual speech dataset,
Open Large-scale Korean Audio-Visual Speech (OLKAVS) dataset. We gathered a
total of 1,150 hours of audio from 1,107 Korean speakers in a studio setup
with corresponding Korean transcriptions. We recorded a speaker from 9
viewpoints: frontal, horizontal (up and down), vertical (left and right), and
diagonal (upper left, upper right, lower left, and lower right). Both scripted
and spontaneous speech data are available in the dataset. We emphasize that,
to the best of our knowledge, OLKAVS is the largest dataset regarding the
number of hours compared to existing audio-visual datasets, especially for the
multi-view datasets. OLKAVS can facilitate the research on a multi-modal
approach to various tasks such as audio-visual speech recognition and lip
reading. To further assist future research, we make the pre-trained baseline
models for both tasks publicly available.111The dataset and the pre-trained
models of audio-visual speech recognition and lip reading with their training
scripts are available at
https://aihub.or.kr/aihubdata/data/view.do?currMenu=115&topMenu=100&aihubDataSe=realm&dataSetSn=538
and https://github.com/IIP-Sogang/olkavs-avspeech, respectively.

In this paper, we aim to provide a complete guideline for the dataset. We
first compare OLKAVS with the existing audio-visual speech datasets in Section
II. A brief overview of the recent notable datasets is provided in Table I.
Then, we enumerate the details of how the dataset was constructed in Section
III. We describe how the speakers and the speech contents are chosen
(Subsection III-A), the recording environment (Subsection III-B), data post-
processing (Subsection III-C), and potential use cases of OLKAVS (Subsection
III-D).

Section IV provides the details for the pre-trained baseline models trained on
the OLKAVS dataset for two downstream tasks, speech recognition and lip
reading. For the speech recognition task (Subsection IV-B), we analyze the
effectiveness of two modalities, vision and audio, by injecting varying levels
of synthesized speech corruption and comparing multi-modal and uni-modal
models. Our results indicate that using both modalities brings the best of
both worlds, enjoying the low error rate comparable to the audio-only model on
the low noise regime, whereas being consistently better than both vision-only
and audio-only baselines even if noise level increases. We additionally
demonstrate the usefulness of leveraging multiple views compared to using only
the frontal data in the lip reading task (Subsection IV-C).

##  II Related Works

In this section, we briefly review the publicly available audio-visual speech
datasets, summarized in Table I. Decades of research on lip reading led to the
construction of multiple datasets, albeit the limited sample size of classical
datasets, e.g., AVLetters [17], CUAVE [18], OuluVS [20] and GRID [19]. All
four datasets contain less than 50 speakers with limited vocabulary. AVLetters
and CUAVE datasets contain utterances of the English alphabet and 10 digits,
respectively, while GRID and OuluVS datasets include short sentences but
suffer from vocabulary limitations.

AVICAR [15] and OuluVS2 [6] consider full sentences with more various
vocabularies. The vocabularies of both datasets are more than a thousand
words. Interestingly, AVICAR and OuluVS2 also provide multi-view videos from
four and five different angles, respectively.

However, the datasets above have a fundamental limitation of containing only a
predefined or limited set of words. To develop larger datasets for word
utterances and reflect real-world speech with unrestricted vocabulary, LRW
proposed a data collection pipeline to label TV news footage automatically.
The pipeline-based approach has influenced numerous follow-up sentence-based
audio-visual speech datasets: LRS2-BBC [5, 14], LRS3-TED [9], and MV-LRS [11].
Whereas LRS2-BBC and LRS3-TED differ only by their data source and dataset
size, MV-LRS primarily focused on providing multi-view video, dividing the
data into five categories based on the face angle to facilitate the multi-view
evaluation.

Moreover, most of the datasets mentioned above contain English speech; only a
few attempts have been made to collect non-English speech. CI-AVSR dataset
[13] covers the Cantonese spoken commands in the car environments, similar to
AVICAR. Other non-English datasets, such as LRW-1000 [8], GLips [10], and
MISP2021 [12], leverage the automated pipeline similar to LRW [4]. LRW-1000
and MISP2021 contain Mandarin word and sentence utterances, respectively,
whereas GLips used the German parliament footage as the source video.
VoxCeleb2 [7] contains multilingual speech in more than 100 languages, but
there are no transcriptions available in the dataset.

Finally, we discuss several audio-visual datasets containing Korean speech. Jo
et al. [16] collected predefined syllables of a single speaker with 7 views.
Also, Lee and Park [22] and Lee et al. [23] collected predefined word
utterances, such as digits and city names, of 56 and 9 speakers, respectively.
Unfortunately, the size of all the datasets is too minuscule to support deep
learning-driven models; moreover, some are not publicly available. Hence, we
are highly motivated to build the open large-scale Korean audio-visual speech
dataset, OLKAVS.

TABLE II:  Script topics of the OLKAVS dataset. Only the topics marked with an asterisk (*) were presented to the teenagers, as other topics might be challenging to understand fully.  Topic | Hours | Ratio (%) | Keyword examples
---|---|---|---
Health/Diet* | 497 | 8.7 | Disease, Hospital, Medicine, Healthcare, Side Effects, Workout, Treatment
Daily | 484 | 8.5 | Weather, Hobby, Habit, Self-management, Shopping, Leisure, Club, Recollection
Job/Work | 480 | 8.4 | Recruiting, Job, Company, Industries, Employment, Labor, Foundation
Financial | 445 | 7.7 | Price, Cost, Tax, Investment, Stock, Real Estate, Deposit, Saving, Loan, Fund, Interest Rate, Exchange Rate
Human Relations | 413 | 7.2 | Family, Friends, Couple, Quarrel, Reconciliation
Pet | 409 | 7.1 | Dog, Cat, Adoption, Pet Disease, Memories with Pet, Pet Accessory
Sports/Leisure* | 406 | 7.1 | Player, Watching a Game, Olympic, Paralympic, National Team, Rule, Water Sports, Climbing
Food* | 399 | 6.9 | Restaurants, Ingredients, Recipe, Local Foods, Fruits, Vegetables, Fishes, Korean Foods, World Foods, Delivery, Meal Kit
Education/School* | 394 | 6.8 | Examination, Subjects, Major, Quiz, Grades, Entrance, Graduation, School, Teacher, Professor, Mid Term, Club
Vacation | 384 | 6.7 | Trip, Travel, Destination, Accommodation, Photo, Holiday, Beach, Attraction, Christmas
Art/Culture* | 388 | 6.7 | Book, Music, Broadcast, Movie, Drama, Celeb, Singer, Actor, Show, Museum, Musical, Poem, Writer, Exhibition
Transportation* | 379 | 6.6 | Driving, Car, Bus, Taxi, Subway, Train, Airplane, Bicycle, Scooter, Ticket Reservation, Public Transit
Social Issues | 357 | 6.2 | Social Issue, News, Human Right, Journalism, Global Warming, Environment, Animal Welfare
| Ceremonial
---
Occasions
307 | 5.3 | Wedding, Invitation, Visitor, Presents, Funeral, Condolences, Rite

##  III OLKAVS Dataset

In this section, we describe our dataset preparation process in detail. OLKAVS
contains 1,150 and 5,750 hours of audio and video in total, respectively,
where we recorded five videos per single audio. To the best of our knowledge,
this is the largest audio-visual speech dataset, especially for multi-view
datasets.

###  III-A Scripts and Speakers

####  III-A1 Designing the script

As shown in Table II, the OLKAVS dataset contains 14 different topics, where
we constructed a set of sentences based on their corresponding keywords. Each
topic has at least 2,000 sentences, so the total sentences exceed 28,000. Each
sentence is written in the Korean language. Example sentences are shown in
Table III. Then, randomly chosen sentences from each topic are combined to
build a script with an average of 2,750 characters (5 minutes long on
average). Generated scripts are provided to the speakers when recording.

TABLE III:  A sample sentence per topic and a sample guideline given to
speakers for the OLKAVS dataset. Only the expert speakers received guidelines
for spontaneous speech, whereas scripts were given to the average speakers
(refer to Subsection III-A for more details). English translations are
provided in the table for ease of understanding, but the speakers were only
supplied with Korean sentences and guidelines.

Script (for scripted speech) | 주제 (Topic) | 스크립트 (Script)
---|---|---
| 건강/다이어트
---
(Health/Diet)
| 딸이 약을 제대로 챙겨 먹지 않아서 건강 상태가 더 악화됐어.
---
(My daughter’s health condition worsened because she didn’t take her medicine
properly.)
| 일상생활
---
(Daily)
| 이번에 새로 산 바지가 한 번 빨았는데 줄어들어서 너무 작아졌어요.
---
(The new pants I bought were washed once, but they shrunked too small.)
| 일/직장/직업
---
(Job/Work)
| 요즘 직장 생활은 위아래로 부딪혀서 업무에 집중하기 힘들어.
---
(These days, work life is bumping up and down, so it’s hard to work properly.)
| 경제/재테크
---
(Financial)
| 지금 달러가 강세라고 하던데 그럼 내가 뭘 해야지 이득을 볼 수 있는 거야?
---
(They say the dollar is strong right now, so what should I do to benefit?)
| 인간관계
---
(Human Relations)
| 남편이 나를 너무 못살게 굴어서 결혼생활이 불행해.
---
(My husband treats me so badly that my marriage is unhappy.)
| 반려동물
---
(Pet)
| 동물 병원에서 일하다 보니 나도 반려동물을 키우고 싶어져.
---
(Working at a veterinary hospital makes me want to have a pet too.)
| 스포츠/레저
---
(Sports/Leisure)
| 넌 제일 좋아하는 축구 선수가 누구야?
---
(Who is your favorite soccer player?)
| 음식
---
(Food)
| 오늘 집에서 스테이크를 미디엄 레어로 구워 먹었어.
---
(I had a steak cooked medium rare at home today.)
| 진학/학교
---
(Education/School)
| 나는 벌써 휴학 신청서를 제출했어.
---
(I have already submitted my application for leave of absence.)
| 휴가
---
(Vacation)
| 부다페스트는 야경이 정말 아름다웠어.
---
(Budapest was really beautiful at night.)
| 문화/예술
---
(Art/Culture)
| 내가 원하는 날짜에 뮤지컬 공연 예약 예매했어.
---
(I made a reservation for a musical performance on the date I want.)
| 교통수단
---
(Transportation)
| 나는 자가용을 사용하는 게 편해.
---
(I’m comfortable using my own car.)
| 사회/시사
---
(Social Issues)
| 어 그래서 우리나라도 지금 제주도로 난민이 꽤 들어오고 있어.
---
(Uh, that’s why there are quite a few refugees coming to Jeju Island.)
| 관혼상제
---
(Ceremonial Occasions)
| 내일 아는 분 결혼식에 가는데 어떻게 입고 가는 게 좋을까요?
---
(What should I wear tommorrow for the wedding of my aquaintance?)
Guideline (for spontaneous speech) | 주제 (Topic) | 스포츠/레저 (Sports/leisure)
| 발화 주제
---
(Topic sentence)
| 이스포츠의 정식 스포츠 등록에 대해 어떻게 생각하시나요?
---
(What’s your opinion about adoption of E-Sports as an official event
in the Asian Games?)
| 관련 키워드
---
(Keywords)
| 이스포츠, 스포츠, 아시안게임
---
(E-sports, Sports, Asian Games)
| 관련 뉴스
---
(Related News)
| e스포츠는 왜 아직 스포츠가 되지 못했을까
---
(Why hasn’t eSports become a sport yet?)

####  III-A2 Speaker distribution

A total of 1,107 speakers participated in constructing the OLKAVS dataset. The
speakers were recruited while considering the demography of biological sex and
age. The numbers of males and females are balanced with 555 males and 552
females. The age of speakers ranges from 10 to 60 and above, following its
distribution within Korea. The age groups are defined with a bin size of 10
years, where their statistics are supplied in Table IV. Note that the
teenagers were specifically asked to speak on six topics only as shown in
Table II (Health/Diet, Sports/Leisure, Food, Education/School, Art/Culture,
and Transportation), which are relatively easier to understand for younger
people.

TABLE IV: Age distribution of speakers in the OLKAVS dataset Age $a$ | # of speakers | Speaker ratio (%)
---|---|---
$10\leq a<20$ | 109 | 10
$20\leq a<30$ | 286 | 26
$30\leq a<40$ | 280 | 25
$40\leq a<50$ | 259 | 23
$50\leq a<60$ |  94 |  8
$a\geq 60$ |  79 |  7

####  III-A3 Expert speakers

30% of the speakers are speech experts, such as announcers, actors, would-be
announcers, and would-be actors. 50% of the experts’ speeches were done
spontaneously based on the related news article. The example guideline is
shown in Table III. However, the spontaneous speech was too demanding for the
average speaker, hence not collected.

####  III-A4 Usage rights

To avoid potential privacy issues, all the speakers formally consented to
using portrait rights and disclosing personal information.

###  III-B Recording Environment

####  III-B1 Multi-view video

Five different views were simultaneously recorded by GoPro Hero7 cameras. We
attached a Tablet PC with cameras for script prompts. Fig. 1 shows two
different types of recorded views, cross views (frontal, up, down, left, and
right) and diagonal views (frontal, upper left, upper right, lower left, and
lower right). Different from pre-existing datasets [6, 11, 16] that only
consider horizontal views, i.e., left, frontal, and right, OLKAVS dataset also
considers the vertical variants, i.e., upper left, upper center, upper right,
lower left, lower center, and lower right.

(a) Cross views

(b) Diagonal views

Figure 1:  Video samples from the OLKAVS dataset. 5 different views were
synchronously recorded for each recording session. (a) and (b) show two types
of views and denote their labels. The amount of data with view "A" doubles
than the others because the view is common in both recording sessions.

####  III-B2 Audio with various noise scenarios

To simulate outdoor recording conditions inside the studio environment, we
played the pre-recorded noise inside the studio. We constructed six different
noise scenarios: no noise, indoor ambiance, indoor noise (laughing and
clapping), traffic noise, construction-site noise, and natural outdoor noise
(flowing water and animal sounds). All the noise audios were collected from
the corresponding real-world situations.

###  III-C Data Cleansing and Post-processing

####  III-C1 Transcription

After recording, we transcribed all the scripted and spontaneous speech. Even
though the scripts were given to the speakers, some of the speakers
mispronounced a few words or spoke pause fillers such as ‘um,’ ‘ah,’ or ‘err,’
where everything was transcribed afterward.

####  III-C2 Data cleansing

We manually filtered out the problematic recordings by the following failure
criteria: the entire lip or facial region is not recorded, the filming
environment is too dark, the noise level is too high, or videos and the audio
are not in sync. We also trimmed every audio-visual pair to have the same
length.

####  III-C3 Data splits

The OLKAVS dataset provides predefined splits of train, validation, and
evaluation with a ratio of 8:1:1. The dataset is split based on speaker ID so
that no speaker co-occurs on different splits. Only the train and validation
sets are made publicly available.

####  III-C4 Facial landmarks

For ease of use, we also provide the bounding box coordinates of the lip and
the face. We used the face alignment estimation module from Bulat and
Tzimiropoulos [24]. Facial landmarks were extracted every 3 seconds and
linearly interpolated for the frames inbetween.

####  III-C5 File formats

Videos are stored in MP4 format with the FHD (1,920$\times$1,080) resolution
and 10-Mbps bitrates. Audios are encoded in WAV format with stereo channels
and 48-kHz sampling rates. Transcriptions are in JSON format, including all
the details of file formats, speaker information on ID, age, and gender,
whether the speaker is an expert, and the utterances’ start and end times.

###  III-D Additional Use Cases

We expect the OLKVAS dataset to further facilitate various research outside
audio-visual speech recognition and lip reading, especially leveraging its
large-scale data.

####  III-D1 Korean speech recognition

Not only the size of OLKAVS greatly outplays the previous audio-visual speech
datasets [16, 22, 23], but also comparable to the audio-only Korean speech
dataset [25]. Also, our pre-trained audio-visual speech recognition model can
be useful when fine-tuned to other languages [10].

####  III-D2 Speaker understanding

As OLKAVS contains speaker identity information of 1,107 speakers, it can be
used for speaker recognition tasks [21, 7]. Also, the metadata on whether the
speaker is an expert can be used as a binary label for language fluency for
automatic pronunciation level classification [26].

####  III-D3 Robust lip reading

OLKAVS is multi-view, so it contains more challenging real-world lip-reading
scenarios than the front-only case. One can also utilize the multi-view to
improve lip reading performance further [27], which we demonstrate in
Subsection IV-C.

##  IV Experiments

In this section, we provide the baselines of the OLKAVS dataset for the two
common tasks: audio-visual speech recognition and multi-view lip reading. The
former was conducted to analyze the effectiveness of multi-modal training over
uni-modal training on changing noise levels, whereas the latter focused on
utilizing multi-view data for increasing lip reading performance over frontal-
view-only data.

###  IV-A Data Pre-processing

Videos were cropped with the bounding boxes of the lips that are provided in
the dataset. Then, to train and evaluate sentence-wise, we split the audio-
visual data based on the timestamps in the transcripts. Also, we modified the
Korean transcripts to be more prediction-friendly: removing punctuations and
converting numbers into corresponding words. We tokenized the transcripts into
grapheme units of 21 vowels, 19 initial consonants, and 27 final consonants.
Refer to [28] for more details on pre-processing the Korean transcripts for
speech recognition.

###  IV-B Speech Recognition

####  IV-B1 Architecture

As shown in Fig. 2, we used the hybrid conformer architecture of [3] for the
audio-visual model (AV-model), which is the state-of-the-art model on LRS3-TED
[9]. The architecture is the cascade of a ResNet-like convolutional network
and the conformer as the encoder for both image sequences and audio waveforms
with the hybrid decoder. For the audio-only (A-model) and visual-only
(V-model) variants, following [3], we simply dropped the corresponding encoder
and the fusion module.

Figure 2:  Hybrid conformer architecture for audio-visual speech recognition.

However, to reduce the required GPU memory size, we modified the original
architecture by adopting the lip reading visual front-end module from [14],
pre-trained with several English lip reading datasets [4, 11, 14], which is
frozen during training. To make the feature dimension equal, we attached a
learnable linear layer between the visual front-end and encoder, converting
512-dimensional features to 256 dimensions.

Both audio and visual front-ends used the modified ResNet-18 architecture
[29]. For the first layer, 1D- and 3D-convolution layer is used for raw
waveform inputs and video inputs, respectively, which is followed by the
residual network. A simple multi-layer perceptron (MLP) was used to fuse the
audio and visual features. The encoder and the decoder consisted of 12
conformer layers and 6 transformer layers, respectively, of which multi-head
attentions used 8 heads and 256 feature dimensions.

####  IV-B2 Training details

We used the same settings for all the models, AV-model, A-model, and V-model,
following the original setting of Ma et al. [3] where they used the weighted
sum of two losses, the connectionist temporal classification (CTC) loss and
cross-entropy (CE) loss. However, for simplicity, we used the simple average
of two losses. We used the Adam optimizer with the learning rate of 0.0001
with the Noam learning rate scheduler, similar to the original setting. Due to
the memory constraints, we used different batch sizes of 8, 12, and 16 for AV-
model, A-model, and V-model, respectively.

####  IV-B3 Evaluation

For the baseline evaluation (Table V), we evaluated all three models, AV-
model, A-model, and V-model, on the publicly available validation set. The
validation set contains various noises, as mentioned in Subsection III-B.

For the noise robustness experiments (Fig. 3), we varied the noise levels
explicitly by mixing the validation data with babble noise from NOISEX-92
[30]. To make the signal-to-noise ratio (SNR) consistent across different
audio samples, we used the subset of the validation data with no noise to be
mixed with the babble noise. The performance was measured in terms of
character error rate (CER), word error rate (WER), and space-normalized word
error rate (sWER) [25].

TABLE V: Baseline performances of A-model, V-model, and AV-model on the OLKAVS validation set Model (modality) | Error rates (%)
---|---
CER | WER | sWER
A-model (Audio-only) |  5.63 | 11.61 |  9.37
V-model (Visual-only) | 36.53 | 49.97 | 51.71
AV-model (Both) |  5.64 | 12.05 |  9.45
Figure 3:  CERs of the AV-model on a subset of validation data with no noise
in the OLKAVS dataset, corrupted by babble noise in NOISEX-92 to simulate
different noise conditions.

####  IV-B4 Results

We provide the speech recognition baselines in Table V and the noise
robustness comparison in Fig. 3. Table V shows that the performance of the
A-model and the AV-model is comparable, differing only by 0.2%. The V-model
achieved 36.53% in CER, which is the first published result for Korean lip
reading at the sentence level.

TABLE VI: CERs (%) of lip reading on a subset of validation data with no noise in the OLKAVS dataset Model View | Upper | Medium-high | Lower
---|---|---|---
left | center | right | left | center(frontal) | right | left | center | right
F-model (Frontal only) |  |  |  |  | 55.00 |  |  |  |
All-model (All views) | 54.65 | 61.53 | 56.53 | 46.63 | 44.86 | 46.16 | 42.59 | 44.77 | 42.96

For the noise robustness experiment, Fig. 3 shows that the AV-model can
effectively utilize both audio and visual data, balancing between two
modalities on varying noise levels to provide better performance than uni-
modal models. As visual data would be less helpful in a low noise condition
(SNR = 15 dB), while audio data would be too corrupted in the opposite end
(SNR = $-$5 dB), the A-model and the V-model act as the hard baseline for the
AV-model in each case, respectively. The AV-model provided better or
comparable performances on both extremes while showing superior performance in
the middle, indicating that utilizing both audio and visual modalities is
effective in speech recognition tasks.

###  IV-C Multi-view Lip Reading

####  IV-C1 Architecture

We trained the visual front-end module from scratch instead of utilizing the
pre-trained module as in the previous experiment (Subsection IV-B).
Specifically, we used the original hybrid conformer architecture [3] without
any modifications in this experiment.

####  IV-C2 Training details

We used the same settings as in Subsection IV-B except the batch size of 6 was
used.

####  IV-C3 Evaluation

To verify the effectiveness of utilizing all the views to improve the lip-
reading performance [27], we compared the lip-reading model trained only with
the frontal view (F-model) and using all the views (All-model). Additionally,
All-model’s performance on various views was compared. We used the subset of
validation data with no noise for evaluation.

####  IV-C4 Results

Table VI demonstrates that the frontal lip reading performance of the All-
model (44.86%) was superior to that of the F-model (55.00%) with an 18%
relative reduction in CERs, which reaffirmed the results of [27] that multi-
view training is superior to that of the frontal-view-only.

We also compared the performance of the All-model on various views to analyze
the difficulty of lip reading from each view. Frontal lip reading in the
medium-high level performed better than the other medium-high views, which
differs from the results in [27]. We suspect that frontal data being twice as
large compared to the other views caused the model to be biased toward the
frontal data. Indeed, lip reading of the centered views performed worse than
that of the right and left views in each of the upper and lower views,
confirming the results in [27]. Also, as the lower angle showed lip movement
better, its performance was better than that of the upper angle.

##  V Conclusion

In this paper, we introduced a novel audio-visual speech dataset, the Open
Large-scale Korean Audio-Visual Speech (OLKAVS) dataset, which is the largest
publicly available audio-visual speech dataset, to the best of our knowledge.
It contains 1,150 hours of transcribed audio from 1,107 Korean speakers with
nine different viewpoints and various noise situations. We also provided pre-
trained models for audio-visual speech recognition and multi-view lip reading,
which serves as a future research baseline and a good starting point for fine-
tuning. Our experiments demonstrated that using both multi-modal and multi-
view data during training improved upon the uni-modal and frontal-view-only
data. Finally, we emphasize that the OLKAVS dataset can be utilized for
broader research areas outside of audio-visual speech recognition and lip
reading.

## References

  * Iezzoni et al. [2004] L. I. Iezzoni, B. L. O’Day, M. Killeen, and H. Harker, “Communicating about health care: Observations from persons who are deaf or hard of hearing,” _Annals of Internal Medicine_ , vol. 140, no. 5, pp. 356–362, 2004.
  * Sumby and Pollack [1954] W. H. Sumby and I. Pollack, “Visual contribution to speech intelligibility in noise,” _The Journal of the Acoustical Society of America_ , vol. 26, no. 2, pp. 212–215, 1954.
  * Ma et al. [2021] P. Ma, S. Petridis, and M. Pantic, “End-to-end audio-visual speech recognition with conformers,” in _Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_ , 2021, pp. 7613–7617.
  * Chung and Zisserman [2016] J. S. Chung and A. Zisserman, “Lip reading in the wild,” in _Proc. Asian Conference on Computer Vision_ , 2016, pp. 87–103.
  * Chung et al. [2017] J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, “Lip reading sentences in the wild,” in _Proc. IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 6447–6456.
  * Anina et al. [2015] I. Anina, Z. Zhou, G. Zhao, and M. Pietikäinen, “OuluVS2: A multi-view audiovisual database for non-rigid mouth motion analysis,” in _Proc. IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG)_ , vol. 1, 2015, pp. 1–5.
  * Chung et al. [2018] J. S. Chung, A. Nagrani, and A. Zisserman, “VoxCeleb2: Deep speaker recognition,” in _Proc. Interspeech_. ISCA, 2018, pp. 1086–1090.
  * Yang et al. [2019] S. Yang, Y. Zhang, D. Feng, M. Yang, C. Wang, J. Xiao, K. Long, S. Shan, and X. Chen, “LRW-1000: A naturally-distributed large-scale benchmark for lip reading in the wild,” in _Proc. IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG)_ , 2019, pp. 1–8.
  * Afouras et al. [2018] T. Afouras, J. S. Chung, and A. Zisserman, “LRS3-TED: a large-scale dataset for visual speech recognition,” _arXiv preprint arXiv:1809.00496_ , 2018\.
  * Schwiebert et al. [2022] G. Schwiebert, C. Weber, L. Qu, H. Siqueira, and S. Wermter, “A multimodal German dataset for automatic lip reading systems and transfer learning,” in _Proc. Thirteenth Language Resources and Evaluation Conference_. European Language Resources Association, 2022, pp. 6829–6836.
  * Chung and Zisserman [2017] J. S. Chung and A. Zisserman, “Lip reading in profile,” in _Proc. British Machine Vision Conference (BMVC)_. BMVA Press, 2017.
  * Chen et al. [2022] H. Chen, J. Du, Y. Dai, C. H. Lee, S. M. Siniscalchi, S. Watanabe, O. Scharenborg, J. Chen, B. C. Yin, and J. Pan, “Audio-visual speech recognition in MISP2021 challenge: Dataset release and deep analysis,” in _Proc. Interspeech_. ISCA, 2022, pp. 1766–1770.
  * Dai et al. [2022] W. Dai, S. Cahyawijaya, T. Yu, E. J. Barezi, P. Xu, C. T. Yiu, R. Frieske, H. Lovenia, G. Winata, Q. Chen _et al._ , “CI-AVSR: A Cantonese audio-visual speech dataset for in-car command recognition,” in _Proc. Thirteenth Language Resources and Evaluation Conference_ , 2022, pp. 6786–6793.
  * Afouras et al. [2022] T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, “Deep audio-visual speech recognition,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 12, pp. 8717–8727, 2022.
  * Lee et al. [2004] B. Lee, M. Hasegawa-Johnson, C. Goudeseune, S. Kamdar, S. Borys, M. Liu, and T. Huang, “AVICAR: Audio-visual speech corpus in a car environment,” in _Proc. International Conference on Spoken Language Processing_ , 2004.
  * Jo et al. [2000] C.-W. Jo, R. Goecke, and B. Millar, “Collection of Korean audio-video speech data,” _Speech Sciences_ , vol. 7, no. 1, pp. 5–15, 2000.
  * Matthews et al. [2002] I. Matthews, T. F. Cootes, J. A. Bangham, S. Cox, and R. Harvey, “Extraction of visual features for lipreading,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 24, no. 2, pp. 198–213, 2002.
  * Patterson et al. [2002] E. K. Patterson, S. Gurbuz, Z. Tufekci, and J. N. Gowdy, “CUAVE: A new audio-visual database for multimodal human-computer interface research,” in _Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_ , vol. 2, 2002, pp. 2017–2020.
  * Cooke et al. [2006] M. Cooke, J. Barker, S. Cunningham, and X. Shao, “An audio-visual corpus for speech perception and automatic speech recognition,” _The Journal of the Acoustical Society of America_ , vol. 120, no. 5, pp. 2421–2424, 2006.
  * Zhao et al. [2009] G. Zhao, M. Barnard, and M. Pietikainen, “Lipreading with local spatiotemporal descriptors,” _IEEE Transactions on Multimedia_ , vol. 11, no. 7, pp. 1254–1265, 2009.
  * Nagrani et al. [2020] A. Nagrani, J. S. Chung, W. Xie, and A. Zisserman, “VoxCeleb: Large-scale speaker verification in the wild,” _Computer Speech & Language_, vol. 60, p. 101027, 2020.
  * Lee and Park [2008] J.-S. Lee and C. H. Park, “Robust audio-visual speech recognition based on late integration,” _IEEE Transactions on Multimedia_ , vol. 10, no. 5, pp. 767–779, 2008.
  * Lee et al. [2019] S.-W. Lee, J.-H. Yu, S. M. Park, and K.-B. Sim, “Visual speech recognition of Korean words using convolutional neural network,” _International Journal of Fuzzy Logic and Intelligent Systems_ , vol. 19, no. 1, pp. 1–9, 2019\.
  * Bulat and Tzimiropoulos [2017] A. Bulat and G. Tzimiropoulos, “How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks),” in _Proc. International Conference on Computer Vision_ , 2017, pp. 1021–1030.
  * Bang et al. [2020] J.-U. Bang, S. Yun, S.-H. Kim, M.-Y. Choi, M.-K. Lee, Y.-J. Kim, D.-H. Kim, J. Park, Y.-J. Lee, and S.-H. Kim, “Ksponspeech: Korean spontaneous speech corpus for automatic speech recognition,” _Applied Sciences_ , vol. 10, no. 19, p. 6936, 2020.
  * Kobayashi and Wilson [2020] A. Kobayashi and I. Wilson, “Using deep learning to classify English native pronunciation level from acoustic information,” in _Proc. SHS Web of Conferences_ , vol. 77, 2020, p. 02004.
  * Lee et al. [2017] D. Lee, J. Lee, and K.-E. Kim, “Multi-view automatic lip-reading using neural network,” in _Proc. Asian Conference on Computer Vision_ , 2017, pp. 290–302.
  * Lee and Chang [2019] M.-H. Lee and J.-H. Chang, “Korean speech recognition based on grapheme,” _The Journal of the Acoustical Society of Korea_ , vol. 38, no. 5, pp. 601–606, 2019.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proc. IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.
  * Varga and Steeneken [1993] A. Varga and H. J. Steeneken, “Assessment for automatic speech recognition: II. NOISEX-92: A database and an experiment to study the effect of additive noise on speech recognition systems,” _Speech Communication_ , vol. 12, no. 3, pp. 247–251, 1993.


(<ref>) and (<ref>) yield
_D R_P(f_,,t,) - R_P^*     ≥    σ^2/n ·1/4C^5 S(β)

≥    c̃”/4C^5 σ^2 min{(+ t^-1)^-2/n, (+ t^-1)^-1/q/n, I(n)/n } .

§.§ Equivalences of norms and eigenvalues

Later, we will use concentration inequalities for kernel matrix eigenvalues proved for specific kernels, which we then want to transfer to other kernels with equivalent RKHSs. In this subsection, we show that this is possible.

Let $n \geq 1$ and let $\bfK, \tilde \bfK \in \bbR^{n \times n}$ be symmetric. For $C \geq 1$, we say that $\bfK$ and $\tilde \bfK$ are $C$-equivalent if their ordered eigenvalues satisfy
C^-1 λ_i() ≤λ_i() ≤Cλ_i()

for all $i \in [n]$.
Moreover, we say that two norms $\|\cdot\|_A$, $\|\cdot\|_B$ on a vector space $V$ are $C$-equivalent if
C^-1 _A ≤_B ≤C_A

for all $\bfv \in V$.

Let $n \geq 1$ and let $\bfK, \tilde \bfK \in \bbR^{n \times n}$ be symmetric. Then, $\bfK$ and $\tilde \bfK$ are $C$-equivalent iff the Moore-Penrose pseudoinverses $\bfK^+$ and $\tilde \bfK^+$ are $C$-equivalent.

This follows from the fact that if $\bfK$ has eigenvalues $\lambda_1, \hdots, \lambda_n$, then $\bfK^+$ has eigenvalues $1/\lambda_1, \hdots, 1/\lambda_n$, where we define $1/0 \equalDef 0$. (A detailed proof would be a bit technical due to the sorting of eigenvalues.)

Let $k: \calX \times \calX \to \bbR$ be a kernel on a set $\calX$.
Then, for any $\bfy \in \bbR^n$,

\begin{equation*}
\bfy^\top k(\bfX, \bfX)^+ \bfy = \|f_{k, \bfy}^*\|_{\calH_k}^2~,
\end{equation*}

where $\calH_k$ is the RKHS associated with $k$ and $f_{k, \bfy}^*$ is the minimum-norm regression solution
f_k, ^*         _f ∈B f__k^2,

B         {f ∈_k |∑_i=1^n (f(_i) - y_i)^2 = inf_f̃ ∈_k ∑_i=1^n (f̃(_i) - y_i)^2} .

It is well-known that $f_{k, \bfy}^*(\bfx) = \sum_{i=1}^n \alpha_i k(\bfx, \bfx_i)$, where $\bfalpha \equalDef \bfK^+ \bfy$ <cit.>. We then have
f_k, ^*__k^2     =     ⟨∑_i=1^n α_i k(_i, ·), ∑_j=1^n α_j k(_j, ·) ⟩__k = ∑_i=1^n ∑_j=1^n α_i _ij α_j

=     ^⊤^+ ^+ = ^⊤^+  ,

where the last step follows from a standard identity for the Moore-Penrose pseudoinverse <cit.>.

Let $k, \tilde k: \calX \times \calX \to \bbR$ be kernels such that their RKHSs are equal as sets and the corresponding RKHS-norms are $C$-equivalent as defined in <Ref>. Then, for any $n \geq 1$ and any $\bfx_1, \hdots, \bfx_n \in \calX$, the corresponding kernel matrices $k(\bfX, \bfX), \tilde k(\bfX, \bfX)$ are $C^2$-equivalent.

Let $i \in [n]$. For $\bfy \in \bbR^n$ we have, using the notation of <Ref>:
^⊤k(, )^+     =     f_k, ^*__k^2 ≥C^-2 f_k, ^*__k̃^2 ≥C^-2 f_k̃, ^*__k̃^2 = C^-2 ^⊤k̃(, )^+  .

Now, by the Courant-Fischer-Weyl theorem,
λ_i(k(, )^+)     =     sup_V: V = i inf_y ∈V: y_2 = 1 y^⊤k(, )^+ y

≥    C^-2 sup_V: V = i inf_y ∈V: y_2 = 1 y^⊤k̃(, )^+ y

=     C^-2 λ_i(k̃(, )^+) .

By switching the roles of $k$ and $\tilde k$, we obtain that $k(\bfX, \bfX)^+$ and $\tilde k(\bfX, \bfX)^+$ are $C^2$-equivalent. By <Ref> $k(\bfX, \bfX)$ and $\tilde k(\bfX, \bfX)$ are then also $C^2$-equivalent.

To prove <Ref> for arbitrary input distributions $P_X$ with lower and upper bounded densities, we need the following theorem investigating the corresponding eigenvalues of the integral operator.

Let $k: \calX \times \calX \to \bbR$ be a kernel and let $\mu, \nu$ be finite measures on $\calX$ whose support is $\calX$ such that $\nu$ has an lower and upper bounded density w.r.t. $\mu$. Then, $\lambda_i(T_{k, \nu}) = \Theta(\lambda_i(T_{k, \mu}))$.

Let $p$ be such an upper bounded density, that is, $\diff \nu = p \diff \mu$ and there exist $c, C > 0$ such that $c \leq p(\bfx) \leq C$ for all $\bfx \in \calX$. For $f \in L_2(\nu)$, we have

\begin{equation*}
\|p \cdot f\|_{L_2(\mu)}^2 = \int f^2 p^2 \diff \mu \leq C \int f^2 p \diff \mu = C \int f^2 \diff \nu = C\|f\|_{L_2(\nu)}^2~.
\end{equation*}

Hence, the linear operator

\begin{equation*}
A: L_2(\nu) \to L_2(\mu), f \mapsto p \cdot f
\end{equation*}

is well-defined and continuous. It is also easily verified that $A$ is bijective. Moreover, we have

\begin{equation*}
\langle Af, Af \rangle_{L_2(\mu)} = \int f^2 p^2 \diff \mu \geq c \int f^2 p \diff \mu = c \int f^2 \diff \nu = c \langle f, f\rangle_{L_2(\nu)}~.
\end{equation*}

\begin{equation*}
\langle f, T_{k, \nu} f\rangle_{L_2(\nu)} = \int \int p(\bfx) f(\bfx) k(\bfx, \bfx') f(\bfx') p(\bfx') \diff \mu(\bfx) \diff \mu(\bfx') = \langle Af, T_{k, \mu} Af\rangle_{L_2(\mu)}~.
\end{equation*}

Since $T_{k, \mu}$ and $T_{k, \nu}$ are compact, self-adjoint, and positive, we can use the Courant-Fischer minmax principle for operators <cit.> to obtain
λ_i(T_k, ν)     =     max_V ⊆L_2(ν)  subspace
V = i min_f ∈V ∖{0} ⟨f, T_k, ν f⟩_L_2(ν)/⟨f, f⟩_L_2(ν)

≥    c max_V ⊆L_2(ν)  subspace
V = i min_f ∈V ∖{0} ⟨Af, T_k, μ Af⟩_L_2(μ)/⟨Af, Af⟩_L_2(μ)

=     c max_Ṽ ⊆L_2(ν)  subspace
Ṽ = i min_g ∈Ṽ ∖{0} ⟨g, T_k, μ g⟩_L_2(μ)/⟨g, g⟩_L_2(μ)

=     c λ_i(T_k, μ) .

Here, we have used that since $A$ is bijective, the subspaces $AV$ for $\dim(V) = i$ are exactly the $i$-dimensional subspaces of $L_2(\mu)$. Our calculation above shows that $\lambda_i(T_{k, \mu}) \leq O(\lambda_i(T_{k, \nu}))$. Since $\diff \mu = \frac{1}{p} \diff \nu$ with the lower and upper bounded density $1/p$, we can reverse the roles of $\nu$ and $\mu$ to also obtain $\lambda_i(T_{k, \nu}) \leq O(\lambda_i(T_{k, \mu}))$, which proves the claim.

Let $\calH_1$ and $\calH_2$ be two RKHSs with $\calH_1\subset \calH_2$. Then there exists a constant $C>0$ such that
$\|f\|_{\calH_2}\le C\|f\|_{\calH_1}$.

Let $I:\calH_1 \to \calH_2$ be the inclusion map, i.e. $I_h := h$ for all $h\in \calH_1$.
Obviously, $I$ is linear and we need to show that $I$ is bounded.
To this end, let $(h_n)_{n\geq 1} \subset \calH_1$ be a sequence such that there exist $h\in \calH_1$ and $g\in \calH
_2$ with $h_n \to h$ in $\calH_1$ and $Ih_n \to g$ in $\calH_2$. This implies $h_n \to h$ pointwise and $h_n = Ih_n\to g$ pointwise,
which in turn gives $h=g$. The closed graph theorem, see e.g. <cit.>, then shows that $I$ is bounded.

§.§ Kernel matrix eigenvalue bounds

For upper bounds on the eigenvalues of kernel matrices, we use the following result:

For $m \geq 1$, we have
_ ∑_i=m^n λ_i(k(, )/n) ≤∑_i=m^∞λ_i(T_k) .

Theorem 7.29 in [Steinwart and Christmann, 2008] shows that

\begin{equation}
\bbE_{D \sim \mu^n} \sum_{i=m}^\infty \lambda_i(T_{k, D}) \leq \sum_{i=m}^\infty \lambda_i(T_{k, \mu})~,
\end{equation}

where $T_{k, \mu}: L_2(\mu) \to L_2(\mu), f \mapsto \int k(x, \cdot) f(x) \diff \mu(x)$ is the integral operator corresponding to the measure $\mu$ and $T_{k, D}$ is the corresponding discrete version thereof. We set $\mu \equalDef P_X$ and need to show that $k(\bfX, \bfX)/n$ has the same eigenvalues as $T_{k, D}$ if $D$ and $\calX$ contain the same data points $\bfx_1, \hdots, \bfx_n$. Consider a fixed $D$. Then, we can write $T_{k, D}(f) = n^{-1} AB f$, where
A: ^n →L_2(D),     ↦    ∑_i=1^n v_i k(_i, ·)

B: L_2(D) →^n, f     ↦    (f(_1), , f(_n))^⊤ .

Then, $k(\bfX, \bfX)/n$ is the matrix representation of $n^{-1}BA$ with respect to the standard basis of $\bbR^n$. But $AB$ and $BA$ have the same non-zero eigenvalues, which means that
∑_i=m^n λ_i(k(, )/n) = ∑_i=m^∞λ_i(T_k, D) ,

from which the claim follows.

To obtain a lower bound, we want to leverage the lower bound by [Buchholz, 2022] for a certain radial basis function kernel with data generated from an open subset of $\bbR^d$. However, we want to consider different kernels and distributions on the whole sphere. The following theorem bridges the gap by going to subsets of the data on a sphere cap, projecting them to $\bbR^d$, and using the kernel equivalence results from <Ref>:

Let $k$ be a kernel on $\bbS^d$ such that its RKHS $\calH_k$ is equivalent to a Sobolev space $H^s(\bbS^d)$ with smoothness $s > d/2$. Moreover, let $P_X$ be a probability distribution on $\bbS^d$ with lower and upper bounded density. Let the rows of $\bfX \in \bbR^{n \times d}$ are drawn independently from $P_X$. Then, for $\varepsilon \in (0, 1/20)$, there exists a constant $c > 0$ and $n_0 \in \bbN$ such that for all $n \geq n_0$,
λ_m(k(, )/n) ≥cn^-2s/d

holds with probability $\geq 4/5$ for all $m \in \bbN$ with $1 \leq m \leq (1-11\eps)n$.

We can choose a suitably large sphere cap $T$ such that $P_X(T) \geq 1-\eps$. Define the conditional distribution $P_T(\cdot) \equalDef P_X(\cdot | T)$. Out of the points $\bfX = (\bfx_1, \hdots, \bfx_n)$, we can consider the submatrix $\bfX_T = (\bfx_{i_1}, \hdots, \bfx_{i_N})^\top$ of the points lying in $T$. Conditioned on $N$, these points are i.i.d. samples from $P_T$. Moreover, by applying Markov's inequality to a Bernoulli distribution, we obtain $N \geq (1-10\eps)n$ with probability $\geq 9/10$. We fix a value of $N \geq (1-10\eps)n$ in the following and condition on it.

We denote the centered unit ball in $\bbR^d$ by $B_1(\bbR^d)$. Using a construction as in <Ref>, we can transport $k$ and $P_T$ from $T$ to the unit ball $B_1(\bbR^d)$ using a rescaled stereographic projection feature map $\phi$, such that we obtain a kernel $k_\phi$ and a distribution $P_\phi = (P_T)_\phi$ on $B_1(\bbR^d)$ that generate the same distribution of kernel matrices as $k$ with $P_T$, and such that $\calH_{k_\phi} \cong H^s(B_1(\bbR^d))$. The rows of $\bfX_\phi \equalDef \phi(\bfX_T)$ are i.i.d. samples from $P_\phi$. Moreover, we know that $P_\phi$ has an lower and upper bounded density w.r.t. the Lebesgue measure on $B_1(\bbR^d)$.

In order to apply the results from [Buchholz, 2022], we define a translation-invariant reference kernel on $\bbR^d$ through the Fourier transform

\begin{equation*}
\hatkref (\xi) = (1 + |\xi|^2)^{-2s}~,
\end{equation*}

see Eq. (3) in [Buchholz, 2022]. The RKHS of $\kref$ on $\bbR^d$ is equivalent to the Sobolev space $H^s(\bbR^d)$. Therefore, the RKHS of $\kref|_{B_1(\bbR^d), B_1(\bbR^d)}$ is $H^s(B_1(\bbR^d))$, cf. the remarks in <Ref> and <Ref>.

Now, let $1 \leq m \leq (1-11\eps)n$, which implies

\begin{equation*}
1 \leq m \leq (1-11\eps)n \leq (1-\eps)(1-10\eps)n \leq (1-\eps)N~.
\end{equation*}

We apply Theorem 12 by [Buchholz, 2022] with bandwidth $\gamma = 1$ and $\alpha = 2s$ to $\lambda_m$ and obtain with probability at least $1-2/N$:
λ_m((_ϕ, _ϕ))^-1     ≤    c_3 (N^2(α-d)/d/(N-m)^(α-d)/d + 1) ≤c_3 (N^2(α-d)/d/(N)^(α-d)/d + 1)

≤    c_4 (n^α/d - 1 + 1)

as long as $N$ is large enough such that $(1-\eps)N < N - 32\ln(N)$, which is the case if $n$ is large enough. Here, the constant $c_3$ from [Buchholz, 2022] does not depend on $N$ or $m$, but only on $\alpha$, $d$, and the upper and lower bounds on the density, which in our case depend on $\eps$ through the choice of $T$.
Since $\alpha = 2s > d$, we have $n^{\alpha/d - 1} > 1$ and therefore
λ_m((_ϕ, _ϕ)/n) ≥c_5 n^-α/d = c_5 n^-2s/d .

Now, we want to translate this to the kernel $k$. Since the RKHSs of $k_\phi$ and $\kref$ on $B_1(\bbR^d)$ are both equivalent to $H^s(B_1(\bbR^d))$, the kernels themselves are $C$-equivalent for some constant $C \geq 1$ as defined in <Ref>. Therefore, <Ref> shows that the corresponding kernel matrices are $C^2$-equivalent, which implies
λ_m((_ϕ, _ϕ)/n) ≥c_5 C^-2 n^-2s/d .

By Cauchy's interlacing theorem, we therefore have
λ_m((, )/n) ≥λ_m((_T, _T)/n) = λ_m((_ϕ, _ϕ)/n) ≥c_5 C^-2 n^-2s/d .

Denoting the event where $\lambda_m(\ksq(\bfX, \bfX)/n) \geq c_5 C^{-2} n^{-2s/d}$ by $A$, we thus have
P(A)     =     P(A|N≥(1-10)n) P(N≥(1-10)n) ≥9/10 P(A|N ≥(1-10)n)

=     9/10 ∑_N̂ = ⌈(1-10)n⌉^n P(N=N̂|N ≥(1-10)n) P(A|N=N̂)

≥    9/10 ∑_N̂ = ⌈(1-10)n⌉^n P(N=N̂|N ≥(1-10)n) (1-2/N)

≥    9/10 (1 - 2/(1-10)n) ∑_N̂ = ⌈(1-10)n⌉^n P(N=N̂|N ≥(1-10)n)

=     9/10 (1 - 2/(1-10)n) ≥4/5 ,

where the last step holds for sufficiently large $n$.

§.§ Spectral lower bound for dot-product kernels on the sphere

An application of the spectral generalization bound in <Ref> requires a lower bound on eigenvalues of the kernel matrix $\ksq(\bfX, \bfX)$. To achieve this, we need to understand the properties of the convolution kernel $\ksq$. Since the eigenvalues of $T_{\ksq, P_X}$ are the squared eigenvalues of $T_{k, P_X}$, one might hope that if $\calH_k$ is equivalent to a Sobolev space $H^s$, then $\calH_{\ksq}$ is equivalent to a Sobolev space $H^{2s}$. Unfortunately, this is not the case in general, as $\calH_{\ksq}$ might be a smaller space that involves additional boundary conditions [Schaback, 2018]. However, perhaps since the sphere is a manifold without boundary, the desired characterization of $\calH_{\ksq}$ holds for dot-product kernels on the sphere:

Let $k$ be a dot-product kernel on $\bbS^d$ such that its RKHS $\calH_k$ is equivalent to a Sobolev space $H^s(\bbS^d)$ with smoothness $s > d/2$, and let $P_X$ be a distribution on $\bbS^d$ with lower and upper bounded density. Then, the RKHS $\calH_{\ksq}$ of the kernel

\begin{equation*}
\ksq: \bbS^d \times \bbS^d \to \bbR, \ksq(\bfx, \bfx') \equalDef \int k(\bfx, \bfx'') k(\bfx'', \bfx') \diff P_X(\bfx'')
\end{equation*}

is equivalent to the Sobolev space $H^{2s}(\bbS^d)$.

(, ')     =     ∫k(, ”) k(”, ') (^d)(”) .

For the corresponding integral operator, we have
T_, (^d)     =     T_k, (^d)^2 .

This means that the corresponding eigenvalues are the squares of the eigenvalues of the corresponding integral operator of $k$. Especially, we obtain the Mercer representations
k(, ')     =     ∑_l=0^∞μ_l ∑_i=1^N_l, d Y_l,i() Y_l,i(') ,

(, ')     =     ∑_l=0^∞μ_l^2 ∑_i=1^N_l, d Y_l,i() Y_l,i(') ,

where <Ref> yields $\mu_l = \Theta((l+1)^{-2s})$, hence $\mu_l^2 = \Theta((l+1)^{-4s})$ and hence $\calH_{\ksqunif} \cong H^{2s}(\bbS^d)$.

Next, we show the equality of the ranges of the integral operators:

\begin{equation*}
R(T_{k, \calU(\bbS^d)}) = R(T_{k, P_X})~.
\end{equation*}

Let $p_X$ be a density of $P_X$ w.r.t. the uniform distribution $\calU(\bbS^d)$. If $f \in R(T_{k, \calU(\bbS^d)})$, there exists $g \in L_2(\calU(\bbS^d))$ with $f = T_{k, \calU(\bbS^d)}g$. But then, since $p_X$ is lower bounded, we have $g/p_X \in L_2(P_X)$ and therefore

\begin{equation*}
f = T_{k, P_X} (g/p_X) \in R(T_{k, P_X})~.
\end{equation*}

An analogous argument shows that $R(T_{k, P_X}) \subseteq R(T_{k, \calU(\bbS^d)})$ since $p_X$ is upper bounded.

The equality of the ranges yields for the RKHSs (as sets)
_ = R(T_k, (^d)) = R(T_k, P_X) = _ ,

Applying <Ref> twice then shows $\calH_{\ksq} \cong H^{2s}(\bbS^d)$.

Step 0: Preparation.
Since the Sobelev space $H^{2s}(\bbS^d)$ is dense in the space of continuous functions $\bbS^d\to \bbR$, the kernel
$k$ is universal. Applying <cit.>
for the least squares loss thus shows that $k$ is strictly positive definite. If we have mutually distinct
$\bfx_1,\dots,\bfx_n$, the corresponding Gram matrix
$k((\bfx_i, \bfx_j))_{i,j=1}^n$ is therefore invertible. Now, our assumptions on
$P$ guarantee that $\bfX$ consists almost surely of mutually distinct observations,
and therefore $k(\bfX, \bfX)$ is almost surely invertible.

By <Ref>, we know that
_D _P(f_,,t,) - _P^*     ≥    σ^2/n ∑_i=1^n _ λ_i((, )/n)(1 - e^-2t(λ_i(k(, )/n) + ))^2/(λ_i(k(, )/n) + )^2

≥    σ^2/n ∑_i=1^n _ λ_i((, )/n)(1 - e^-2C^-1n^2s/d(λ_i(k(, )/n) + 0))^2/(λ_i(k(, )/n) + Cn^-2s/d)^2

≥    c_n σ^2

for a suitable constant $c_n > 0$ depending on $n$ but not on $\sigma^2, t, \rho$, since the kernel matrix eigenvalues are nonzero almost surely. It is therefore sufficient to show the desired statement (with $c$ independent of $n, \sigma^2, t, \rho$) for sufficiently large $n$.

In the following, we assume $n \geq 40$ and set $\eps \equalDef 1/100$.

Step 1: Eigenvalue decay for the integral operator. From <Ref>, we know that
λ_i(T_k, (^d)) = Θ(i^-2s/d) .

Therefore, by <Ref>, we know that
λ_i(T_k, P_X) = Θ(i^-2s/d) .

Step 2: Eigenvalue upper bound. Next, we want to upper-bound suitable eigenvalues of the form $\lambda_i(k(\bfX, \bfX)/n)$ using <Ref>. Using Step 1, we derive

\begin{equation*}
\sum_{i=m}^\infty \lambda_i(T_{k, P_X}) \leq C_1 \sum_{i=m}^\infty i^{-2s/d} \leq C_2 \int_m^\infty x^{-2s/d} \diff x = C_3 m^{1-2s/d}
\end{equation*}

with constants independent of $m \geq 1$. For sufficiently large $n$, we can choose $m \in \bbN_{\geq 1}$ such that $\eps n \leq m \leq 2 \eps n$. Then, <Ref> yields

\begin{equation*}
\bbE_{\bfX} \sum_{i=m}^n \lambda_i(k(\bfX, \bfX)/n) \leq \sum_{i=m}^\infty \lambda_i(T_k) \leq C_3 m^{1-2s/d} \leq C_4 n^{1-2s/d}~.
\end{equation*}

Since $\bbE_{\bfX} \lambda_i(k(\bfX, \bfX)/n)$ is decreasing with $i$, we have for $i \geq 4\eps n \geq 2m$:

\begin{equation*}
\bbE_{\bfX} \lambda_i(k(\bfX, \bfX)/n) \leq C_4 n^{1-2s/d} / m \leq C_5 n^{-2s/d} \leq C_6 \lambda_i(T_{k, P_X})~.
\end{equation*}

Step 3: Eigenvalue lower bounds. From <Ref>, we know that $\calH_{\ksq} \cong H^{2s}(\bbS^d)$. Therefore, we can apply <Ref> to both $k$ and $\ksq$ and obtain for sufficiently large $n$ and suitable constants $c_1, c_2 > 0$ that
λ_i(k(, )/n)     ≥    c_1 n^-2s/d

λ_i((, )/n)     ≥    c_2 n^-4s/d

individually hold with probability $\geq 4/5$ for all $i \in \bbN$ with $1 \leq i \leq (1-11\eps)n$. By the union bound, both bounds hold at the same time with probability $\geq 3/5$.

Step 4: Final result. Now, using the value of $m$ from Step 2, consider an index $i$ with $2m \leq i \leq (1-11\eps) n$. Since $2m \leq 4\eps n$ and $\eps = 1/100$, there are at least $n/2$ such indices. By combining Step 3 and Step 1, we have
λ_i(k(, )/n)     ≥    c_3 λ_i(T_k, P_X)

λ_i((, )/n)     ≥    c_4 λ_i(T_k, P_X)^2

with probability $\geq 3/5$. By applying Markov's inequality to Step 2, we obtain

\begin{equation*}
\lambda_i(k(\bfX, \bfX)/n) \leq 10C_6 \lambda_i(T_{k, P_X})
\end{equation*}

with probability $\geq 9/10$. Therefore, by the union bound, all three inequalities hold simultaneously with probability $\geq 1/2$. Moreover, for $q = 2s/d$, we have $\lambda_i(T_{k, P_X}) \geq c_5 i^{-q}$ by Step 1. We can thus apply the first lower bound from <Ref> to obtain
_D R_P(f_,,t,) - R_P^*     ≥    c' σ^2 1/1 + (ρ+ t^-1)n^2s/d ·|(n)|/n

≥    c' σ^2 1/1 + (Cn^-2s/d + Cn^-2s/d)n^2s/d ·n/2/n

=     c'/2 + 2C σ^2 .

§ PROOF OF <REF>

Here we denote the solution of kernel ridge regression on $D$ with the kernel function $k$ and regularization parameter $\reg > 0$ as

\[
\hat{f}_\reg^k(\bfx)= k(\bfx,\bfX) \left(k(\bfX, \bfX)+\reg \bfI\right)^{-1} \bfy,
\]

and write $\hat{f}_0^k(\bfx)=k(\bfx,\bfX) k(\bfX, \bfX)^{+} \bfy$ for the minimum-norm interpolant in the RKHS of $k$.

While <Ref> states that overfitting kernel ridge regression using Sobolev kernels is always inconsistent as long as the derivatives remain bounded by the derivatives of the minimum-norm interpolant of the fixed kernel (Assumption (N)), here we show that consistency over a large class of distributions is achievable by designing a kernel sequence, which can have Sobolev RKHS, that consists of a smooth component for generalization and a spiky component for interpolation.

Recall that $\kti$ denotes any universal kernel function for the smooth component, and $\check{k}_\gamma$ denotes the kernel function of the spiky component with bandwidth $\gamma$. Then we define the $\reg$-regularized spiky-smooth kernel with spike bandwidth $\gamma$ as

\[
k_{\reg,\gamma}(\bfx,\bfx') = \kti(\bfx,\bfx') + \reg\cdot \check{k}_{\gamma}(\bfx,\bfx').
\]

Let $B_t(\bfx)\equalDef \{\bfy\in\bbR^d \mid |\bfx-\bfy|\le t\}$ denote the Euclidean ball of radius $t\geq 0$ around $\bfx\in\bbR^d$.

(D2) There exists a constant $\beta_X>0$ and a continuous function $\phi:[0,\infty)\to[0,1]$ with $\phi(0)=0$ such that $P_X(B_t(\bfx))\le \phi(t) =O(t^{\beta_X})$ for all $\bfx\in\Omega$ and all $t\geq 0$.

The kernel $\check{k}_\gamma$ of the spiky component should fulfill the following weak assumption on its decay behaviour. For example, Laplace, Matérn, and Gaussian kernels all fulfill Assumption (SK).

(SK) There exists a function $\eps:(0,\infty)\times [0,\infty)\to [0,1]$ such that for any bandwidth $\gamma>0$ and any $\delta>0$ it holds that

(i) $\eps(\gamma,0)=1$,

(ii) $\eps(\gamma,\delta)$ is monotonically increasing in $\gamma$,

(iii) For all $\bfx,\bfy\in\Omega$, if $|\bfx-\bfy|\geq \delta \quad\text{ then }\quad |\check{k}_\gamma(\bfx,\bfy)|\le \eps(\gamma,\delta),$

(iv) For any rates $\beta_X, \beta_k>0$ there exists a rate $\beta_\gamma>0$ such that, if $\delta_n= \Omega(n^{-\beta_X})$ and $\gamma_n=O(n^{-\beta_\gamma})$, then $\eps(\gamma_n,\delta_n)=O(n^{-\beta_k})$.

. Assume that the training set $D$ consists of $n$ i.i.d. pairs $(\bfx,y)\sim P$ such that the marginal $P_X$ fulfills (D2) and $\bbE y^2<\infty$. Let the kernel components satisfy:

* $\kti$ denotes an arbitrary universal kernel, and $\reg_n\to 0$ and $n\reg_n^4\to \infty$.

* $\kch$ denotes a kernel function that fulfills Assumption (SK) with a sequence of positive bandwidths $(\gamma_n)$ fulfilling $\gamma_n=O(\exp(-\beta n))$ for some arbitrary $\beta>0$.

Then the minimum-norm interpolant of the $\reg_n$-regularized spiky-smooth kernel sequence $k_n\equalDef k_{\reg_n,\gamma_n}$
is consistent for $P$.

Under stronger assumptions on $\phi$ and $\eps$ in assumptions (D2) and (SK), the spike bandwidths $\gamma_n$ can be chosen to converge to $0$ at a much slower rate. For example, if we choose $\check{k}_\gamma$ to be the Laplace kernel, choosing bandwidths $0<\gamma_n\le\frac{\delta}{\beta \ln n}$ yields, for separated points $|\bfx-\bfy|\geq \delta$,
\[
\kch(\bfx,\bfy)\le \exp{\left( -\frac{\delta}{\gamma_n} \right)} \le n^{-\beta}.
\]

For probability measures with upper bounded Lebesgue density, we can choose $\delta_n=n^{-\frac{2+\alpha}{d}}$ and $\beta=\frac{9}{4}+\frac{\alpha}{2}$, for any fixed $\alpha>0$, in the proof of <Ref>. Hence the Laplace kernel only requires a slow bandwidth decay rate of $\gamma_n=\Omega\left(\frac{n^{-\frac{2+\alpha}{d}}}{\alpha \ln(n)}\right)$, where $\alpha>0$ arbitrary. For the Gaussian kernel an analogous argument yields $\gamma_n=\Omega\left(\frac{n^{-\frac{4+2\alpha}{d}}}{\alpha \ln(n)}\right)$. The larger the dimension $d$, the slower the required bandwidth decay.

If one does not care about continuous kernels, one could simply take a Dirac kernel as the spike and then obtain consistency for all atom-free $P_X$. However, we need a continuous kernel to be able to translate it to an activation function for the NTK. Beyond kernel regression, the spike component $\check{k}_\gamma$ does not even need to be a kernel, it just needs to fulfill Assumption (SK) or a similar decay criterion. Then one could still use the 'quasi minimum-norm estimator' $\bfx\mapsto(\kti + \reg_n \kch)(\bfx,\bfX) \cdot(\Kti + \reg_n \Kch)^{+} \bfy$.

Without resorting to kernel sequences as we do, there seems to be no rigorous proof showing that ridgeless kernel regression can be consistent in fixed dimension. In future work, can an analytical expression of such a kernel be found? According to the semi-rigorous results in Mallinar et al., 2022 a spectral decay like $\lambda_k=\Theta(k^{-1}\cdot\log^{\alpha}(k))$, $\alpha>1$ could lead to such a kernel.

Given any universal kernel, <cit.> implies universal consistency of kernel ridge regression if $\reg_n\to 0$ and $n\reg_n^4\to\infty$. Hence, for any $\eps>0$ it holds that
\[
\lim_{n\to\infty} P^n\left(D\in(\bbR^d \times \bbR)^n \mid \; R_P(\hat{f}_{\reg_n}^{\kti}) - R_P(f_P^*)=\bbE_{\bfx} (\hat{f}_{\reg_n}^{\kti}(\bfx)-f^*_P(\bfx))^2 \geq (\eps/2)^2\right)=0.
\]

Due to the triangle inequality in $L_2(P_X)$, we know
R_P(f̂_0^k_n)-R_P(f_P^*)=_ (f̂_0^k_n()-f^*_P())^2
≤    ( (_ (f̂_0^k_n()-f̂__n^())^2)^1/2 + (_ (f̂__n^()-f^*_P())^2)^1/2 )^2.

It is left to show that $k_n$ fulfills

\[
\lim_{n \to \infty} P^n\left(D\in(\bbR^d \times \bbR)^n \mid \; \bbE_{\bfx} (\hat{f}_0^{k_n}(\bfx)-\hat{f}_{\reg_n}^{\kti}(\bfx))^2 \geq (\eps/2)^2\right) = 0.
\]

For this purpose we decompose the above difference into the difference of $\Kch\equalDef \kch(\bfX,\bfX)$ and $\bfI_n$ and a remainder term depending on $\kch$. We denote the $2$-operator norm by $\|\cdot\|$ and the Euclidean norm in $\bbR^n$ by $|\cdot|$. For any $\bfx\in\bbR^d$ it holds that
|f̂_0^k()-f̂__n^k̃()|    ≤    |(+ _n )(,) ·(+ _n )^-1 - (,)·(+_n _n)^-1|

≤   |(,) ((+ _n )^-1-(+_n _n)^-1)|

+ _n ·|(,) (+_n )^-1 |

≤    (,) ·(+ _n )^-1-(+_n _n)^-1 ·||

+ _n ·(,)·(+_n )^-1·||.

Consequently we get
_ (f̂_0^k()-f̂__n^k̃())^2 ≤     2     _(,)^2 ·(+ _n )^-1-(+_n _n)^-1^2 ·||^2 .........

+     2     ^2_n ·_(,)^2·(+_n )^-1^2·||^2.

We now bound the individual terms in Eq.  (<ref>) and (<ref>). To this end, fix any $\alpha>0$.

Bounding Eq. (<ref>):

Since we assumed $y_i$ i.i.d. and $\bbE y_1^2 <\infty$, the Markov inequality implies, with $b_n=\bbE y_1^2 \cdot n^{\alpha}$,
\[
P(|\bfy|^2\geq b_n n)\le \frac{\bbE y_1^2}{b_n}= n^{-\alpha}.
\]

Stated differently, with probability at least $1-n^{-\alpha}$ it holds that $|\bfy|^2\le \bbE y_1^2\cdot n^{1+\alpha}$.

In order to bound the spectrum of $\Kch$, <Ref> implies that there exists a positive sequence $\delta_\alpha(n)=n^{-\frac{2+\alpha}{\beta_X}}$ such that with probability at least $1-O(n^{-\alpha})$ it holds that

\[
\min_{i,j\in [n]: i\neq j} |\bfx_i-\bfx_j| \geq \delta_{\alpha}(n).
\]

Since $(\gamma_n)$ fulfills $\gamma_n=O(n^{-\beta_\gamma})$ for any $\beta_\gamma>0$, by Assumption (SK) there exists a sequence $\eps_n=o(\reg_n n^{-2-\frac{\alpha}{2}})$ such that $\eps(\gamma_n,\delta_\alpha(n))\le \eps_n$.
Assumption (SK) further implies that whenever $\min_{i,j\in [n]: i\neq j} |\bfx_i-\bfx_j| \geq \delta_\alpha(n)$ it holds that $(\Kch)_{ii}=1$ and $0\le (\Kch)_{ij}\le \eps(\gamma_n,\delta_\alpha(n))\le\eps_n$ for $i \neq j$.
Then Gershgorin's theorem [Gerschgorin, 1931] implies that for all eigenvalues of $\Kch$
\[
|\lambda_i(\Kch)-1|\le (n-1)\eps_n \text{ for all } i\in [n].
\]

This in turn implies
\[
\|\Kch-\bfI_n\|\le (n-1)\eps_n,\qquad \lmax(\Kch)\le 1+(n-1)\eps_n,\qquad \lmin(\Kch)\geq 1-(n-1)\eps_n.
\]

Using $\|(\Kti+\reg_n \bfI_n)^{-1}\|\le \frac{1}{\lmin(\Kti)+\reg_n}\le \reg_n^{-1}$ and $\|\Kch-\bfI_n\|\le (n-1)\eps_n$, <Ref> implies
(+ _n )^-1-(+_n _n)^-1    ≤    (+_n _n)^-1^2 ·_n -_n/1-(+_n _n)^-1·_n -_n

≤    _n^-1 (n-1)_n/1- (n-1)_n.

Using $|\kti(\bfx,\bfX_i)|\le 1$ for all $i\in [n]$ yields the naive bound $\|\kti(\bfx,\bfX)\|^2\le n$.

Combining all terms in Eq. (<ref>) yields its convergence to $0$ as the product satifies the rate $O(n^{4+\alpha}\reg_n^{-2} \eps_n^2)=o(1)$ with probability at least $1-2n^{-\alpha}$.

Bounding Eq. (<ref>):

The analysis below is restricted to the event of probability at least $1-2n^{-\alpha}$, on which the bound on Eq. (<ref>) holds.

Since $(n-1)\eps_n\to 0$, for any $C>1$ it holds for $n$ large enough,

\[
\reg_n \cdot \|(\Kti+\reg_n \Kch)^{-1}\|\le \frac{\reg_n}{\lmin(\Kti)+\reg_n(1-(n-1)\eps_n)} \le \frac{1}{(1-(n-1)\eps_n)}\le C.
\]

Finally we show $\sup_{\bfx'\in\bbR^d} \bbE_{\bfx} \kch(\bfx,\bfx')^2\le 2 n^{-(2+\alpha)}$ for $n$ large enough.

Fix an arbitrary $\bfx'\in\bbR^d$. Then by construction of $\delta_\alpha(n)$ and $\eps_n$ it holds that
_ (,')^2    ≤    1·P_X({∈^d:   (,')^2≥_n^2})+_n^2

≤    P_X({∈^d:   |-'|<δ_α(n)}) +_n^2

≤    ϕ(δ_α(n)) + _n^2≤n^-(2+α) + _n^2.

Since $\eps_n^2=o(\reg^2_n n^{-4-\alpha})$, we get $\bbE_{\bfx} \kch(\bfx,\bfx')^2 \le 2 n^{-(2+\alpha)}$ for $n$ large enough.

Combining all terms in Eq. (<ref>) yields its convergence to $0$ with the rate $O(n^{-(2+\alpha)} \cdot 1 \cdot n^{1+\alpha})=O(n^{-1})$ with probability at least $1-2n^{-\alpha}$, which concludes the proof.

§.§ Auxiliary results for the proof of <Ref>

The distributional Assumption (D2) immediately implies that the training points are separated with high probability.

Assume (D2) is fulfilled with $\beta_X>0$. Then with probability at least $1-O(n^{-\alpha})$,

\[
\min_{i,j\in [n]: i\neq j} |\bfx_i-\bfx_j| \geq n^{-\frac{2+\alpha}{\beta_X}}.
\]

For any $i\in [n]$, the union bound implies

\[
P(\min_{j\in [n]: i\neq j} |\bfx_i-\bfx_j| \le \delta) = P\left(\bigcup_{j\in [n]: j\neq i} \{ \bfx_j \in B_{\delta}(\bfx_i) \}\right) \le (n-1) \phi(\delta).
\]

Another union bound yields
\[
P(\min_{i,j\in [n]: i\neq j} |\bfx_i-\bfx_j| \le \delta) \le n(n-1) \phi(\delta).
\]

Choosing $\delta_\alpha(n)=n^{-\frac{2+\alpha}{\beta_X}}$ yields $\phi(\delta_{\alpha}(n))=O(\frac{1}{n^{2+\alpha}})$, which concludes the proof.

The following lemma bounds $\|\bfA^{-1}-\bfB^{-1}\|$ via $\|\bfA^{-1}\|$ and $\|\bfA-\bfB\|$. Similar results can for example be found in <cit.>.

Let $\bfA,\bfB\in\bbR^{n\times n}$ be invertible matrices and let $\|\cdot\|$ be a submultiplicative matrix norm with $\|\bfI_n\|=1$. If $\bfA$ and $\bfB$ fulfill $\|\bfA^{-1}\| \|\bfA-\bfB\|<1$, then it holds that

\[
\|\bfB^{-1}-\bfA^{-1}\|\le \frac{\|\bfA^{-1}\|^2\cdot \|\bfA-\bfB\|}{1-\|\bfA^{-1}\|\cdot\|\bfA-\bfB\|}~.
\]

Because of $\|\bfA^{-1}(\bfA-\bfB)\|\le \|\bfA^{-1}\| \|\bfA-\bfB\|<1$ we get

\[
\|\bfI-\bfA^{-1}(\bfA-\bfB)\|\geq 1-\|\bfA^{-1}\| \|\bfA-\bfB\|.
\]

Writing $\bfB=\bfA(\bfI-\bfA^{-1}(\bfA-\bfB))$ yields $\bfB^{-1}=(\bfI-\bfA^{-1}(\bfA-\bfB))^{-1} \bfA^{-1}$ which implies
\[
\|\bfB^{-1}\| \le \frac{\|\bfA^{-1}\|}{1-\|\bfA^{-1}\|\|\bfA-\bfB\|}.
\]

Now write $\bfB^{-1}-\bfA^{-1}=\bfA^{-1}(\bfA-\bfB)\bfB^{-1}$ to get
\[
\|\bfB^{-1}-\bfA^{-1}\| \le \|\bfA^{-1}\| \|\bfA-\bfB\| \|\bfB^{-1}\|.
\]

Combining the last two inequalities concludes the proof.

§.§ RKHS norm bounds

Here we show that if $\kti$ and $\check{k}_\gamma$ have RKHS equivalent to some Sobolev space $H^s$, $s>d/2$, then the RKHS of the spiky-smooth kernel $k_{\reg,\gamma}$ is also equivalent to $H^s$, for any fixed $\reg,\gamma>0$. Hence all members of the spiky-smooth kernel sequence may have RKHS equivalent to a Sobolev space $H^s$ and are individually inconsistent due to <Ref>; yet the sequence is consistent. This shows that when arguing about generalization properties based on RKHS equivalence, the constants matter and the narrative that depth does not matter in the NTK regime as in Bietti and Bach, 2021 is too simplified.

The following proposition states that the sum of kernels with equivalent RKHS yields an RKHS that is equivalent to the RKHS of the summands. For example, the spiky-smooth kernel with Laplace components possesses an RKHS equivalent to the RKHS of the Laplace kernel.

Let $\calH_1$ and $\calH_2$ denote the RKHS of $k_1$ and $k_2$ respectively. If $\calH_1=\calH_2$
then the RKHS $\calH$ of $k=k_1+k_2$ fulfills $\calH=\calH_1$. Moreover, if
$C\geq 1$ is a constant with $\frac{1}{C} \|f\|_{\calH_2}\le \|f\|_{\calH_1}\le C\|f\|_{\calH_2}$, then we have $\frac{1}{\sqrt{2} C}\|f\|_{\calH_1}\le \|f\|_{\calH}\le \|f\|_{\calH_1}$.

The RKHS of $k=k_1+k_2$ is given by $\calH=\calH_1+\calH_2$ with norm

\|f\|^2_\calH=\min\{\|f_1\|^2_{\calH_1}+\|f_2\|^2_{\calH_2} : \; f=f_1+f_2, f_1\in \calH_1, f_2\in\calH_2\}\, .

To see this we consider the map $\Phi:X\to \calH_1 \times \calH_2$ defined by
$\Phi(\bfx) := (\Phi_1(\bfx,\cdot), \Phi_2(\bfx, \cdot))$ for all $\bfx \in X$, where $X$ is the set, the spaces $\calH_i$ live on and $\Phi_i(\bfx) := k_i(\bfx,\cdot)$.
The reproducing property of $k_1$ and $k_2$ immediately ensures that $\Phi$ is a feature map of $k_1+k_2$ and
<Ref> then shows

\begin{align*}
\calH
&= \bigl\{  \langle w,\Phi(\cdot)\rangle_{\calH_1 \times \calH_2} : w\in \calH_1\times \calH_2 \bigr\} \\
&= \bigl\{  \langle w_1,\Phi_1(\cdot) \rangle_{\calH_1} + \langle w_2,\Phi_2(\cdot) \rangle_{\calH_2}: w_1\in \calH_1, w_2\in \calH_2 \bigr\}
= \calH_1 + \calH_2
\end{align*}

as well as the formula for the norm on $\calH$.
Now let $f\in \calH$. Considering the decomposition $f=f_1+0$ then gives $\|f\|_{\calH}\le \|f\|_{\calH_1}$.
Moreover, for $f=f_1+f_2$ with $f_i\in\calH_i$ we have

\[
\|f\|_{\calH_1} \leq \|f_1\|_{\calH_1} + \|f_2\|_{\calH_1} \leq \|f_1\|_{\calH_1} + C\|f_2\|_{\calH_2} \leq \sqrt 2 C \bigl( \|f_1\|^2_{\calH_1} + \|f_2\|^2_{\calH_1}  \bigr)^{1/2}\, .
\]

Taking the infimum over all decomposition then yields the estimate
$\|f\|_{\calH_1} \leq {\sqrt{2} C}\|f\|_{\calH}$.

§ SPIKY-SMOOTH ACTIVATION FUNCTIONS INDUCED BY GAUSSIAN COMPONENTS

Here we explore the properties of the NNGP and NTK activation functions induced by spiky-smooth kernels with Gaussian components.

To offer some more background, it is well-known that NNGPs and NTKs on the sphere $\bbS^d$ are dot-product kernels, i.e., kernels of the form $k_d(\bfx, \bfx') = \kappa(\langle \bfx, \bfx' \rangle)$, where the function $\kappa$ has a series representation $\kappa(t) = \sum_{i=0}^\infty b_i t^i$ with $b_i \geq 0$ and $ \sum_{i=0}^\infty b_i<\infty$. The function $\kappa$ is independent of the dimension $d$ of the sphere. Conversely, all such kernels can be realized as NNGPs or NTKs <cit.>.

As dot-product kernel $k(\bfx,\bfy)=\kappa(\langle\bfx,\bfy\rangle)$ on the sphere, the Gaussian kernel has the simple analytic expression,
\[
\kappa^{Gauss}_\gamma(z)=\exp\left(\frac{2(z-1)}{\gamma}\right),
\]

with Taylor expansion
\[
\kappa^{Gauss}_\gamma(z)= \sum_{i=0}^\infty \underbrace{\frac{2^i}{\gamma^i i!} \exp(-2/\gamma)}_{b_i^{Gauss}} \; z^i.
\]

For spiky-smooth kernels $k=\kti+\reg\check{k}_\gamma$ with Gaussian components $\kti$ and $\check{k}_\gamma$ of width $\tilde{\gamma}$ and $\gamma$ respectively, we get Taylor series coefficients
b_i=  exp(-2/γ̃)/i! (2/γ̃)^i + exp(-2/γ)/i! (2/γ)^i.

Now <Ref> states that as soon as $\kappa$ induces a dot-product kernel for every input dimension $d$, then the dot-product kernels can be written as the NNGP kernel of a 2-layer fully-connected network without biases and with the induced activation function

\[
\phi^\kappa_{NNGP}(x)=\sum_{i=0}^\infty s_i b_i^{1/2} h_i(x),
\]

or as the NTK of a 2-layer fully-connected network without biases and with the induced activation function

\[
\phi^\kappa_{NTK}(x) = \sum_{i=0}^\infty s_i \left(\frac{b_i}{i+1}\right)^{1/2} h_i(x),
\]

where $h_i$ denotes the $i$-th Probabilist's Hermite polynomial normalized such that $\|h_i\|_{L_2(\calN(0,1))}=1$ and $s_i\in\{-1,+1\}$ are arbitrarily chosen for all $i\in\bbN$.

Now we can study the induced activation functions if we know the kernel's Taylor coefficients $(b_i)_{i\in \bbN_0}$. If infinitely many $b_i>0$, then infinitely many activation functions induce the same dot-product kernel, with different choices of the signs $s_i$. For alternating signs $s_i=-1$ if $i$ is odd and $s_i=1$ if $i$ is even, the symmetry property $h_i(-x)=(-1)^i h_i(x)$ of the Hermite polynomials implies
\[
\phi_{NNGP,+-}(x)=\phi_{NNGP,+}(-x),\qquad \phi_{NTK,+-}(x)=\phi_{NTK,+}(-x).
\]

To form an orthonormal basis of $L_2(\calN(0,1))$ the unnormalized Probabilist's Hermite polynomials $He_i$ have to be normalized by $h_i(x)=\frac{1}{\sqrt{i!}}He_i(x)$. We can use the identity $\exp(xt-\frac{t^2}{2})=\sum_{i=0}^\infty He_i(x)\frac{t^i}{i!}$ with $t=\sqrt{{2}/{\gamma}}$ to analytically express the NNGP activation of the Gaussian kernel with all $s_i=+1$ as the exponential function
ϕ^Gauss_NNGP(x) = exp(-1/γ) ∑_i=0^∞1/i! (2/γ)^i/2 h_i(x) = exp( (2/γ)^1/2·x - 2/γ).

For $\phi_{NNGP}(x)=\sum_{i=0}^\infty s_i\sqrt{b_i} h_i(x)$, we get $\|\phi\|^2_{L_2(\calN(0,1))} = \sum_{i=0}^\infty b_i$ invariant to the choice $\{s_i\}_{i\in\bbN}$, which yields
ϕ^Gauss_NNGP^2_L_2((0,1))    =    exp(-2/γ) ∑_i=0^∞1/i! (2/γ)^i = 1,

because $\{h_i\}_{i\in\bbN_0}$ is an ONB of $L_2(\calN(0,1))$. Analogously,
ϕ^Gauss_NTK^2_L_2((0,1))    =    exp(-2/γ) ∑_i=0^∞1/(i+1)! (2/γ)^i
=    exp(-2/γ) γ/2 ∑_i=1^∞1/i! (2/γ)^i = γ/2(1-exp(-2/γ)).

This implies that the average amplitude of NNGP activation functions does not depend on $\gamma$, while the average amplitude of NTK activation functions decays with $\gamma\to 0$.

By the fact $h_n'(x)=\sqrt{n} h_{n-1}(x)$, we know that any activation function $\phi(x)= \sum_{i=0}^\infty s_i a_i h_i(x)$ has the derivative $\phi'(x)=\sum_{i=0}^\infty s_i a_{i+1} \sqrt{i+1}\cdot h_i(x)$ as long as $\sum_{i=0}^\infty |a_{i+1}\sqrt{i+1} |<\infty$.

The following proposition formalizes the additive approximation $\phi^k\approx \phi^{\kti}+\reg^{1/2} \phi^{\check{k}_{\gamma}}$, and quantifies the necessary scaling of $\gamma$ for any demanded precision of the approximation.

Fix $\tilde{\gamma},\reg>0$ arbitrary. Let $k=\tilde k +\reg \check{k}_\gamma$ denote the spiky-smooth kernel where  $\tilde k$ and $\check{k}_\gamma$ are Gaussian kernels of bandwidth $\tilde{\gamma}$ and $\gamma$, respectively. Assume that we choose the activation functions $\phi^{k}_{NTK}$, $\phi^{\tilde{k}}_{NTK}$ and $\phi^{\check{k}_\gamma}_{NTK}$ as in <Ref> with same signs $\{s_i\}_{i\in\bbN}$. Then, for $\gamma>0$ small enough, it holds that
ϕ^k_NTK - (ϕ^k̃_NTK+√()·ϕ^ǩ_γ_NTK) ^2_L_2((0,1))    ≤    2^1/2 γ^3/2 exp(-1/γ) + 4πγ(1+γ̃)/γ̃,

ϕ^k_NNGP - (ϕ^k̃_NNGP+√()·ϕ^ǩ_γ_NNGP) ^2_L_2((0,1))    ≤    2^3/2 γ^1/2 exp(-1/γ) + 8πγ(1+γ̃)/γ̃^2.

Let $b_{i,\gamma}=\frac{2^i}{\gamma^i i!}\exp(-2/\gamma)$ denote the Taylor coefficients of the Gaussian kernel. All considered infinite series converge absolutely.
ϕ^γ̃,γ,_NNGP - (ϕ^γ̃_NNGP+√()·ϕ^γ_NNGP) ^2_L_2((0,1))

=      ∑_i=0^∞s_i √(b_i,γ̃ +b_i,γ) h_i(x) - ∑_i=0^∞s_i (√(b_i,γ̃) + √(b_i,γ)) h_i(x) ^2_L_2((0,1))

=        ∑_i=0^∞(  √(b_i,γ̃ + b_i,γ) - (√(b_i,γ̃) + √(b_i,γ)))^2

≤       2∑_i=0^I (√(b_i,γ̃ + b_i,γ) - b^1/2_i,γ̃)^2_(I) + 2 ∑_i=0^I b_i,γ_(II) + 2∑_i=I+1^∞(√(b_i,γ̃ + b_i,γ) - ^1/2 b^1/2_i,γ) ^2_(III) + 2∑_i=I+1^∞b_i,γ̃_(IV),

for any $I\in \bbN$.
To bound $(I)$ observe
∑_i=0^I (√(b_i,γ̃ + b_i,γ) - b^1/2_i,γ̃)^2    =    ∑_i=0^I ( b_i,γ + 2b_i,γ̃(1- √(1+b_i,γ/b_i,γ̃)) )≤∑_i=0^I b_i,γ.

An analogous calculation for $(III)$ yields
\[
\sum_{i=I+1}^\infty (\sqrt{b_{i,\tilde{\gamma}} + \reg b_{i,\gamma}} - \reg^{1/2} b^{1/2}_{i,\gamma}) ^2 \le \sum_{i=I+1}^\infty b_{i,\tilde{\gamma}}.
\]

So overall we get the bound
ϕ^γ̃,γ,_NNGP - (ϕ^γ̃_NNGP+√()·ϕ^γ_NNGP) ^2_L_2((0,1)) ≤4 ∑_i=0^I b_i,γ +4∑_i=I+1^∞b_i,γ̃.

Now, defining $c\equalDef 2/\gamma$,
\[
\sum_{i=0}^I b_{i,\gamma} = \exp(-c) \sum_{i=0}^I \frac{1}{i!} c^i=\frac{\Gamma(I+1,c)}{I!},
\]

where $\Gamma(k+1,c)$ denotes the upper incomplete Gamma function. Choosing $I=\lfloor \frac{c}{2\pi} \rfloor$, () yields, for $c\geq 121$,
Γ(I+1,c)/I!    ≤    exp(-c) (c+(I+1)!^1/I)^I+1/(I+1)! ·(I+1)!^1/I ≤exp(-c) (c+I)^I+1/(I+1)!^(I+1)/I

≤    exp(-c) (c+I)^I+1/(2π(I+1))^1/2 (I+1/e)^(I+1)^2/I

≤    1/√(2π(I+1)) exp(-c+(I+1)( ln(c+I)-ln(I+1)+1 ))

≤   1/√(c) exp(-c+(c/2π+1)( ln(2π+1/2πc)-ln(c/2π)+1 ))

≤   1/√(c) exp(-c+(c/2π+1)( ln(2π+1)+1 ))≤exp(-c/2)/√(c),

where we used $(I+1)!^{1/I} \le I$ for $I\geq 3$ in the first line, Stirling's approximation in the second line, and $(\frac{c}{2\pi}+1)\big( \ln(2\pi+1)+1 \big)\le c/2$ for $c\geq 121$ in the last line.

It is obvious that
\[
\sum_{i=I+1}^\infty b_{i,\tilde{\gamma}} \to 0, \quad\text{for } I=\lfloor \frac{c}{2\pi} \rfloor \to \infty.
\]

To quantify the rate of convergence, we use the bound $\Gamma(I+1,c_0)\geq e^{-c_0} I! (1+c_0/(I+1))^I$, which follows from applying Jensen's inequality to $\Gamma(I+1,c_0)=e^{-c_0} I! \bbE(1+c_0/G)^I$, where $G\sim \Gamma(I+1,1)$ and $\bbE G=I+1$. Defining $c_0=2/\tilde{\gamma}$, it holds that
∑_i=I+1^∞b_i,γ̃ ≤1-Γ(I+1,c_0)/I! ≤1- e^-c_0 (1+c_0/I+1)^I ≤1- e^-c_0 (1+c_0/I+1)^I+1.

Taking the first two terms of the Laurent series expansion of $n\mapsto \left(1+\frac{c_0}{n}\right)^{n}$ about $n=\infty$ yields $\left(1+\frac{c_0}{I+1}\right)^{I+1}>e^{c_0} (1-\frac{c_0^2}{2(I+1)})$ for $I$ large enough (where we demand $\gamma\in o(\tilde\gamma^2)$), thus

∑_i=I+1^∞b_i,γ̃    ≤    1- e^-c_0 (1+c_0/I+1)^I+1 ·(1+c_0/I+1)^-1

≤    c_0/(I+1)+c_0^2/(2(I+1))/1+c_0/(I+1)≤c_0/I+1 + c_0^2/2(I+1)≤4π/γ̃c + 4π/γ̃^2 c.

Plugging (<ref>) and (<ref>) into (<ref>) yields, for $\gamma\le 1/61$,
\[
\|\phi^{\tilde{\gamma},\gamma,\reg}_{NNGP} - (\phi^{\tilde{\gamma}}_{NNGP}+\sqrt{\reg}\cdot\phi^{\gamma}_{NNGP}) \|^2_{L_2(\calN(0,1))} \le 2^{3/2} \reg \gamma^{1/2} \exp\left(-\frac{1}{\gamma}\right) + \frac{8\pi \gamma(1+\tilde{\gamma})}{\tilde{\gamma}^2}.
\]

For the NTK we get
ϕ^γ̃,γ,_NTK - (ϕ^γ̃_NTK+√()·ϕ^γ_NTK) ^2_L_2((0,1))

=      ∑_i=0^∞s_i √(b_i,γ̃ +b_i,γ/i+1) h_i(x) - ∑_i=0^∞s_i (√(b_i,γ̃/i+1) + √(b_i,γ/i+1)) h_i(x) ^2_L_2((0,1))

=        ∑_i=0^∞1/i+1 ( √(b_i,γ̃ + b_i,γ) - (√(b_i,γ̃) + √(b_i,γ)))^2.

We can proceed exactly as for the NNGP, but choose $I=\lfloor \frac{c}{2\pi} \rfloor-1$ to get
\[
\sum_{i=0}^I \frac{b_{i,\gamma}}{i+1} = \exp(-c) \sum_{i=0}^I \frac{c^i}{(i+1)!} = \frac{\exp(-c)}{c} \left( \sum_{i=0}^{I+1} \frac{c^i}{i!} -1 \right) \le \frac{\exp(-c/2)}{c^{3/2}} - \frac{\exp(-c)}{c},
\]

and replace (<ref>) with
∑_i=I+1^∞b_i,γ̃/i+1    =    exp(-c_0)/c_0 ∑_i=I+2^∞ c_0^i/i! ≤1/I+2+c_0/2(I+2)≤πγ(1+γ̃)/γ̃.

§ ADDITIONAL EXPERIMENTAL RESULTS

We provide Python code to reproduce all of our experimental results at

Our implementations rely on  [Paszke et al., 2019] for neural networks and  [Johansson et al., 2023] for high-precision calculations.

§.§ Experimental details of <Ref>

For the kernel experiment (<Ref>a), we used the Laplace kernel with bandwidth $0.4$ and the spiky-smooth kernel (<ref>) with Laplace components with $\reg=1,\tilde{\gamma}=1,\gamma=0.01$.

For the neural network experiment (<Ref>b,c) we initialize 2-layer networks with NTK parametrization [Jacot et al., 2018] and He initialization [He et al., 2015]. Using the antisymmetric initialization trick from Zhang et al., 2020 doubles the network width from $10000$ to $20000$ and helps to prevent errors induced by the random initialization function. We train the network with stochastic gradient descent of batch size $1$ over the $15$ training samples with learning rate $0.04$ for $2500$ epochs. Training with gradient descent and learning rate $0.4$ produces similar results. We use the spiky-smooth activation function given by $x\mapsto ReLU(x)+0.01 \cdot (\sin(100x)+\cos(100x))$, which corresponds to $x\mapsto ReLU(x)+\omega_{NTK}(x,\frac{1}{5000})$, including both even and uneven Hermite coefficients.

§.§ Disentangling signal from noise in neural networks with spiky-smooth activation functions

Since our spiky-smooth activation function has the additive form $\sigma_{spsm}(x)=ReLU(x)+ \omega_{NTK}(x;\frac{1}{5000})$, we can dissect the learned neural network
f_spsm()=_2·σ_spsm(_1·+_1)+b_2=f_ReLU() + f_spikes()

into its $ReLU$-component
\[
f_{ReLU}(\bfx)=\bfW_2\cdot ReLU(\bfW_1\cdot \bfx+\bfb_1)+b_2,\]

and its spike component
\[
f_{spikes}(\bfx)=\bfW_2\cdot \omega_{NTK}(\bfW_1\cdot \bfx+\bfb_1;\frac{1}{5000}).
\]

If the analogy to the spiky-smooth kernel holds and $f_{spikes}$ fits the noise in the labels while having a small $L_2$-norm, then $f_{ReLU}$ would have learned the signal in the data. Indeed <Ref> demonstrates that this simple decomposition is useful to disentangle the learned signal from the spike component in our setting. The figure also suggests that the oscillations in the activations of the hidden layer constructively interfere to interpolate the training points, while the differing frequencies and phases approximately destructively interfere on most of the remaining covariate support. <Ref> shows some of the functions generated by the hidden layer neurons of the spike component $f_{spikes}$. Both the phases and frequencies vary. Destructive interference in sums of many oszillations occurs, for example, under a uniform phase distribution.

a. The $ReLU$-component $f_{ReLU}$ (blue) and the full spiky-smooth network $f_{spsm}$ (orange) of the learned neural network from <Ref>. b. The spike component $f_{spikes}$ of the learned neural network from <Ref> against the label noise in the training set, derived by subtracting the signal from the training points. Observe that the $ReLU$-component has learned the signal, while the spike component has fitted the noise in the data while regressing to $0$ between data points.

Here we plot the functions learned by $12$ random hidden layer neurons of the spike component network $f_{spikes}$ corresponding to <Ref>.

An exciting direction of future work will be to understand when and why the neural networks with spiky-smooth activation functions learn the target function well, and when the decomposition into $ReLU$- and spike component succeeds to disentangle the noise from the signal. Particular challenges will be to design architectures and learning algorithms that provably work on complex data sets and to determine their statistical convergence rates. A different line of work could evaluate whether there exist useful spike components for deep and narrow networks beyond the pure infinite-width limit. Maybe for deep architectures is suffices to apply spiky-smooth activation functions only between the penultimate and the last layer.

Of course an analogous additive decomposition exists for the minimum-norm interpolant $\hat{f}_0^k$ of the spiky-smooth kernel,
f̂_0^k()= (+ _n )(,) ·(+ _n )^-1 = f_signal() + f_spikes(),

\[
f_{signal}(\bfx) = \kti(\bfx,\bfX) \cdot(\Kti + \reg_n \Kch)^{-1} \bfy, \qquad f_{spikes}(\bfx) = \reg_n \kch(\bfx,\bfX) \cdot(\Kti + \reg_n \Kch)^{-1} \bfy.
\]

We plot the results in <Ref>. Observe that the spikes $f_{spikes}$ regress to $0$ more reliably than in the neural network.

Although spiky-smooth estimators can be consistent, any method that interpolates noise cannot be adversarially robust. The signal component $f_{signal}$ may be a simple correction towards robust estimators. <Ref> suggests that the signal components of spiky-smooth estimators behave more robustly than ReLU networks or minimum-norm interpolants of Laplace kernels in terms of finite-sample variance.

a. The signal component $f_{signal}$ (blue) and the full minimum-norm interpolant $\hat{f}_0^k$ (orange) of the spiky-smooth kernel from <Ref>. b. The spike component $f_{spikes}$ of the spiky-smooth kernel from <Ref> against the label noise in the training set, derived by subtracting the signal from the training points.

§.§ Repeating the finite-sample experiments

We repeat the experiment from <Ref> $100$ times, both randomizing with respect to the training set and with respect to neural network initialization.

For the kernels (<Ref>a), observe that all minimum-norm kernel interpolants are biased towards $0$. While the Laplace kernel and the signal component (<ref>) of the spiky-smooth kernel have similar averages, the spiky-smooth kernel has a slightly larger bias. However, both the spiky-smooth kernel as well as its signal component produces lower variance estimates than the Laplace kernel.

Considering the trained neural networks (<Ref>b), the ReLU networks are approximately unbiased, but have large variance. The neural networks with spiky-smooth activation function as well as the extracted signal network (<ref>) are similar on average: They are slighly biased towards $0$, but have much smaller variance than the ReLU networks.

We repeat the experiment from <Ref> $100$ times and report the mean values (lines). Confidence bands denote the interval between the empirical $2.5\%$- and $97.5\%$-quantiles from the $100$ independent runs.

The training curves (<Ref>c) offer similar conclusions as <Ref>: While the ReLU networks harmfully overfit over the course of training, the neural networks with spiky-smooth activation function quickly overfit to $0$ training error with monotonically decreasing test error, which on average is almost optimal, already with only $15$ training points. The spiky-smooth networks have smaller confidence bands, indicating increased robustness compared to the ReLU networks. If the ReLU networks would be early-stopped with perfect timing, they would generalize similarly well as the networks with spiky-smooth activation function.

§.§ Spiky-smooth activation functions

In <Ref> we plot the 2-layer NTK activation functions induced by spiky-smooth kernels with Gaussian components, where $\kti$ has bandwidth $1$, and in the first figure $\reg=1$ while in the second figure $\reg=0.1$.

The 2-layer NTK activation functions for Gaussian-Gaussian spiky-smooth kernels with varying $\gamma$ (columns) with $k_{max}=1000$, $\kti$ has bandwidth $1$, $\reg=1$. Top: all $s_i = +1$, middle: $+,+,-,-,+,+,...$, bottom: Random $+1$ and $-1$. Although the activation function induced by the spiky-smooth kernel is not exactly the sum of the activation functions induced by its components, this approximation is accurate because the spike components approximately live in a subspace of higher frequency in the Hermite basis orthogonal to the low-frequency subspace of the smooth component.

Same as above but $\reg=0.1$. The high-frequency fluctuations are much smaller compared to <Ref>.

In <Ref> we plot the corresponding 2-layer NNGP activation functions with $\rho=1$. In contrast to the NTK activation functions the amplitudes of the fluctuations only depends on $\reg$ and not on $\gamma$. Our intuition is the following: Since the first layer weights are not learned in case of the NNGP, the first layer cannot learn constructive interference, so that the oscillations in the activation function need to be larger.

The additive approximation $\phi^k\approx \phi^{\kti}+\reg^{1/2} \phi^{\check{k}_{\gamma}}$ remains accurate in all considered cases (<Ref>).

Same as above but NNGP and $\reg=1$. As expected from the isolated spike plots: Spikes essentially add fluctuations that increase in frequency and stay constant in amplitude for $\gamma\to 0$, $\reg$ regulates the amplitude.

§.§ Isolated spike activation functions

<Ref> is the equivalent of <Ref> for the NNGP.

By plotting the NTK activation components corresponding to Gaussian spikes $\phi^{\check{k}_{\gamma}}$ with varying choices of the signs $s_i$ in <Ref>, we observe the following properties:

* All $s_i=+1$ leads to exponentially exploding activation functions, cf. Eq. (<ref>).

* If the signs $s_i$ alternate every second $i$, i.e. $s_i=+1$ iff $\lfloor\frac{i}{2}\rfloor$ even, $\phi^{\check{k}_{\gamma}}$ is approximately a single shifted $\sin$-curve with increasing frequency and decreasing/constant amplitude for NTK/NNGP activation functions, cf. Eq. (<ref>).

* If $s_i$ is chosen uniformly at random, with high probability, $\phi^{\check{k}_{\gamma}}$ both oscillates at a high frequency around $0$ and explodes for $|x|\to\infty$.

Same as <Ref> but for the NNGP. In contrast to the NTK, the amplitudes of the oscillations in a. do not shrink with $\gamma\to0$. Otherwise the behaviour is analogous. For example, the Hermite coefficients peak at $2/\gamma$. The squared coefficients sum to $1$ (Eq. (<ref>)).

The spike activation components of 2-layer NTK for Gaussian spikes with varying $\gamma$ (columns), $k_{max}=1000$, top: all $s_i=+1$, middle: signs alternate every second index, bottom: 3 draws from uniformly random signs. With $\gamma\to0$, the amplitude shrinks, while the frequency increases.

<Ref> visualizes NNGP activation functions induced by Gaussian spikes with varying bandwidth $\gamma$. Observe similar behaviour as for the NTK but amplitudes invariant to $\gamma$ as predicted by Eq. (<ref>). For smaller $\gamma$ the explosion of (all+) activation functions starts at larger $x$, but appears sharper as can be seen in the analytic expression (<ref>).

<Ref> resembles <Ref> but plotted on a larger range to visualize the exploding behaviour for $|x|\to\infty$.

Spike activation components as in <Ref>, but for the NNGP with $x$ between $[-4,4]$.

Spike NTK activation components as in <Ref> but with $x$ between $[-4,4]$. The all+ NTK activation explodes exponentially. While random sign activations explode as well, $++--$-activations remain stable $\sin$-fluctuations with slowly decaying amplitude for $|x|\to\infty$.

§.§ Additive decomposition and $\sin$-fit

Here we quantify the error of the $\sin$-approximation (<ref>) of Gaussian NTK activation components.
The additive decomposition $\phi^k\approx \phi^{\kti}+\reg^{1/2} \phi^{\check{k}_{\gamma}}$ quickly becomes accurate in the limit $\gamma\to 0$ (<Ref>), the $\sin$-approximation seems to converge pointwise at rate $\Theta(|x|\gamma)$, where a good approximation can be expected when $|x|\ll 1/\gamma$.
The error at large $|x|$ arises because the spike component decays for $|x|\to\infty$. For $O(1)$ inputs, we conjecture that this inaccuracy does not dramatically affect the test error of neural networks when $\gamma$ is chosen to be small.

The isolated NTK spike activation function (orange), the difference between spiky-smooth and smooth activation function (blue) and a fitted sin-curve (<ref>) (green). All curves roughly align, in particular for $\gamma\to0$.

The error of the additive decomposition $\phi^k\approx \phi^{\kti}+\reg^{1/2} \phi^{\check{k}_{\gamma}}$ (blue) and the sin-fit (<ref>) (orange) for the NTK. While the additive decomposition makes errors of order $10^{-3},10^{-4},10^{-9},10^{-15}$ (from left to right) in the domain $[-4,4]$, the sin-fit is increasingly inaccurate for $|x|\to\infty$, and increasingly accurate for $\gamma\to0$.

Now we evaluate the approximation quality of the $\sin$-fits (<ref>) and (<ref>) to the isolated spike activation components $\phi^{\check{k}_\gamma}$. The NNGP oscillating activation function $\phi^{\check{k}_\gamma}$ of a Gaussian spike component is extremely well approximated by Eq. (<ref>). Both for the NNGP and for the NTK, the approximations become increasingly accurate with smaller bandwidths $\gamma\to0$ (<Ref>). Again the approximation quality suffers for $|x|\to\infty$, since $\phi^{\check{k}_\gamma}$ slowly decay to $0$ for $|x|\to\infty$.

Absolute error between the oscillating activation function $\phi^{\check{k}_\gamma}$ of a Gaussian spike component and the approximation a. Eq. (<ref>) for the NNGP and b. Eq. (<ref>) for the NTK with varying bandwidth $\gamma$.

§.§ Spiky-smooth kernel hyper-parameter selection

To understand the empirical performance of spiky-smooth kernels on finite data sets, we generate i.i.d. data where $\bfx\sim \calU(\bbS^{d})$ and
\[
y=\bfx_1 + \bfx_2^2 + \sin(\bfx_3) + \prod_{i=1}^{d+1} \bfx_i + \eps,
\]

with $\eps\sim \calN(0,\sigma^2)$ independent of $\bfx$ and evaluate the least squares excess risk of the minimum-norm interpolant. <Ref> shows that

* the smaller the spike bandwidth $\gamma$, the better. At some point, the improvement saturates,

* $\reg$ should be carefully tuned, it has large impact. As with $\gamma\to 0$ ridgeless regression with the spiky-smooth kernel approximates ridge regression with $\kti$ and regularization $\reg$, simply choose the optimal regularization $\reg^{opt}$ of ridge regression.

* The spiky-smooth kernel with Gaussian components exhibits catastrophic overfitting, when $\gamma$ is too large (cf. [Mallinar et al., 2022]), the Laplace kernel is more robust with respect to $\gamma$.

* With sufficiently thin spikes and properly tuned $\reg$, spiky-smooth kernels with Gaussian components outperform the Laplace counterparts.

We repeat the experiment in <Ref> with a slightly more complex generating function and come to the same conclusions.

Least squares excess risk for spiky-smooth kernel ridgeless regression with Laplace components (left) and Gaussian components (right), with $n=1000, d=2$, estimated on $10000$ independent test points, $\sigma^2=0.5, \tilde{\gamma}=1$. The smaller the spike bandwidth $\gamma$, the better. Properly tuning $\reg$ is important.

Same as <Ref> but with the more complex generating function $y=|\bfx_1| + \bfx_2^2 + \sin(2\pi \bfx_3) + \prod_{i=1}^{d+1} \bfx_i + \eps$. The errors are larger compared to <Ref> and the optimal values of $\rho$ are smaller, but the conceptual conclusions remain the same.