Kangwook Jang1∗††thanks: * These authors contributed equally to this work.,
Sungnyun Kim2∗, Se-Young Yun2, Hoirin Kim1

# Recycle-and-Distill: Universal Compression Strategy for Transformer-based
Speech SSL Models with Attention Map Reusing and Masking Distillation

###### Abstract

Transformer-based speech self-supervised learning (SSL) models, such as
HuBERT, show surprising performance in various speech processing tasks.
However, huge number of parameters in speech SSL models necessitate the
compression to a more compact model for wider usage in academia or small
companies. In this study, we suggest to reuse attention maps across the
Transformer layers, so as to remove key and query parameters while retaining
the number of layers. Furthermore, we propose a novel masking distillation
strategy to improve the student model's speech representation quality. We
extend the distillation loss to utilize both masked and unmasked speech frames
to fully leverage the teacher model's high-quality representation. Our
universal compression strategy yields the student model that achieves phoneme
error rate (PER) of 7.72% and word error rate (WER) of 9.96% on the SUPERB
benchmark.

Index Terms: speech self-supervised learning, model compression, attention map
reusing, masking distillation

##  1 Introduction

Transformer-based speech SSL models [1, 2, 3] have been actively studied in
speech processing field [4] as SSL arises as a successful representation
learning approach in recent years [5, 6, 7, 8]. Especially for wav2vec 2.0
[9], HuBERT [10], and wavLM [11], all of which are inherited from BERT [12],
show surprising performance in automatic speech recognition (ASR), comparable
to supervised learning approaches [13, 14]. Since the versatility of speech
SSL becomes also crucial, the above models have been further explored in
various applications including automatic speaker verification (ASV) [15] or
emotion recognition (ER) [16].

However, these models have huge number of parameters and are trained for very
long time, which makes it hard for the resource-limited groups to train their
own models. For instance, wav2vec 2.0 Large with 317M parameters should be
pretrained for more than 290 days on a single V100 GPU [9] on LibriSpeech
dataset [17]. This necessitates us to build a compressed model that allows
much more parameter-efficient training and lower computational overhead.

Knowledge distillation (KD) [18] is a common model compression technique where
a smaller student model is being trained by distilling the knowledge from a
teacher model. Prior efforts in distilling large-scale speech SSL models have
been made with reducing the number of Transformer layers or shrinking their
width. DistilHuBERT [19] is distilled in a way of predicting multi-layer
outputs of HuBERT, with most of the Transformer layers removed. FitHuBERT
[20], instead of removing the layers, suggests cutting down the width of
attention and feed-forward network (FFN) in each Transformer layer.
LightHuBERT [21] creates a prunable supernet through distillation and conducts
architecture search to make a small student.

Despite the effectiveness of previous approaches in mitigating the performance
drop by compression, they still face several issues. (1) Wide and shallow
students [19, 22] still exhibit degradation on content-related downstream
tasks. (2) Layer-to-layer (L2L) distillation is proved to be effective [20,
22], however, it is counter-intuitive in terms of compression since every
layer's parameters are required. (3) Pruning by architecture search [21]
prepares an additional teacher-sized supernet using 32 GPUs, which is not end-
to-end (E2E) and cannot be easily trained by resource-limited groups.

We suggest reusing attention maps across the student's Transformer layers,
which is inspired by previous works [23, 24] that claimed the similarity
between attention maps. Attention map reusing enables us to remove key and
query parameters in certain Transformer layers, making it unnecessary to
retain all layer parameters for L2L distillation. Furthermore, we can reinvest
the saved parameters to other parts of Transformer.

We also propose a masking with L2L distillation for better speech
representation quality of our student model. Masking speech frames is a widely
used technique in speech SSL models [9, 10], trained by predicting the masked
representation. This technique has been simply applied to distilling HuBERT
[21], but not in the L2L manner. Our novel masking distillation scheme aims to
fully leverage the teacher's representation by extending the distillation loss
to both masked and unmasked speech frames. We emphasize that our scheme is an
E2E fashion and enhances the general quality of speech representation,
especially in content- and semantics-related tasks.

Combining our two approaches described (Fig. 1), we reinvest the saved
parameters from attention map reusing to FFN, and create our flagship model,
ARMHuBERT (Attention map Reused Mask HuBERT). As evaluated on the SUPERB
benchmark [25], ARMHuBERT achieves overall score [11] of 78.1, the state-of-
the-art E2E distillation. It also reaches 7.72% PER in phoneme recognition
(PR), and 9.96% WER in ASR.

##  2 Preliminaries

Figure 1: Our compression strategy involves reusing the attention map of the
previous layer and extending the distillation process to masked (red arrow)
and unmasked (blue arrow) representations. The input masked frames are
identical for both teacher and student.

###  2.1 Transformer-based Speech SSL Models

Recent dominant SSL models in speech field are wav2vec 2.0 [9], HuBERT [10],
and wavLM [11], where these three model structures are identical except for
detailed level. Specifically, they share 12 or 24 Transformer [26] layers and
7-layer 1D-CNN. Their pretraining schemes are based on masked prediction,
estimating the codewords by output representation of the masked frames.
Despite the superiority and scalability of speech SSL models, large number of
parameters and their computational overhead make it difficult to train these
models. We thus implement the model compression on HuBERT and wavLM, the two
dominant SSL models in speech, to demonstrate the effectiveness of our
compression strategy.

###  2.2 SUPERB Benchmark

The beginning of speech SSL models focused on content-related downstream tasks
such as ASR or PR [27, 28], however, their versatility to other tasks has been
recognized as crucial recently [11]. In this context, SUPERB benchmark [25]
has been proposed to evaluate the generalizability of speech SSL models,
covering the aspects of content, speaker, semantics, and paralinguistics. We
evaluate our representation against the SUPERB benchmark to verify the
generalizability of our student model. The SUPERB downstream tasks include PR,
ASR, keyword spotting (KS), query-by-example spoken term detection (QbE),
speaker identification (SID), ASV, speaker diarization (SD), intent
classification (IC), slot filling (SF), and ER.

##  3 Methodology

###  3.1 Attention Map Reusing

Table 1: Evaluation results on SUPERB benchmark. Metrics include parameter
size in million, PER%, WER% (w/o language model), accuracy (Acc%), maximum
term weighted value (MTWV), equal error rate (EER%), diarization error rate
(DER%), F1 score (F1%), and concept error rate (CER%). ``Overall'' denotes the
average scoring of all tasks proposed in [11]. LightHuBERT [21] works by two-
stage training, where HuBERT-sized supernet needs to be trained first, thus
not compared with E2E distillation models. ARMHuBERT-S and ARMwavLM-S with
960h distillation are trained for 100 epochs.

|  |  | Content | Speaker | Semantics | Paral.
---|---|---|---|---|---|---
| Params | Overall | PR | ASR | KS | QbE | SID | ASV | SD | IC | SF | ER
Models |  Millions $\downarrow$ | Score $\uparrow$ | PER $\downarrow$ | WER $\downarrow$ | Acc $\uparrow$ | MTWV $\uparrow$ | Acc $\uparrow$ | EER $\downarrow$ | DER $\downarrow$ | Acc $\uparrow$ | F1 $\uparrow$ | CER $\downarrow$ | Acc $\uparrow$
Baselines |  |  |  |  |  |  |  |  |  |  |  |  |
FBANK [25] | 0 | 40.5 | 82.01 | 23.18 | 8.63 | 0.0058 | 8.5E-4 | 9.56 | 10.55 | 9.1 | 69.64 | 52.94 | 35.39
HuBERT Base [10] | 94.70 | 80.8 | 5.41 | 6.42 | 96.30 | 0.0736 | 81.42 | 5.11 | 5.88 | 98.34 | 88.53 | 25.20 | 64.92
wavLM Base [11] | 94.70 | 81.9 | 4.84 | 6.21 | 96.79 | 0.0870 | 84.51 | 4.69 | 4.55 | 98.63 | 89.38 | 22.86 | 65.94
LightHuBERT $a_{\text{Small}}$ [21] |  94.7 $\rightarrow$ 27.00 | 79.1 | 6.60 | 8.33 | 96.07 | 0.0764 | 69.70 | 5.42 | 5.85 | 98.23 | 87.58 | 26.90 | 64.12
960h distillation – # params: 26.4M $\sim$ 31.6M |  |  |  |  |  |  |  |  |
FitW2V2 [20] | 31.63 | 76.5 | 12.22 | 11.44 | 96.04 | 0.0475 | 64.71 | 6.65 | 6.44 | 93.38 | 86.65 | 29.40 | 62.35
3-L One-Pred [22] | 30.58 | 76.8 | 13.34 | 12.23 | 96.69 | 0.0489 | 75.71 | 6.48 | 6.56 | 94.15 | 82.89 | 34.65 | 63.95
12-L Half-L2L [22] | 26.87 | 77.6 | 10.67 | 10.96 | 97.24 | 0.0604 | 69.52 | 6.13 | 6.81 | 96.97 | 86.11 | 30.93 | 63.24
MaskHuBERT (ours) | 26.64 | 77.8 | 7.30 | 9.77 | 96.36 | 0.0664 | 62.83 | 5.38 | 6.79 | 97.05 | 87.31 | 27.10 | 62.37
ARMHuBERT (ours) | 26.45 | 78.1 | 7.72 | 9.96 | 96.88 | 0.0635 | 65.03 | 5.68 | 7.10 | 97.07 | 87.59 | 26.06 | 62.86
960h distillation – # params: 22.4M $\sim$ 23.5M |  |  |  |  |  |  |  |  |
DistilHuBERT [19] | 23.49 | 75.9 | 16.27 | 13.37 | 95.98 | 0.0511 | 73.54 | 8.55 | 6.19 | 94.99 | 82.57 | 35.59 | 63.02
FitHuBERT [20] | 22.49 | 74.5 | 13.32 | 12.09 | 96.27 | 0.0489 | 55.71 | 8.00 | 6.84 | 91.25 | 84.06 | 32.46 | 59.82
ARMHuBERT-S (ours) | 22.39 | 77.5 | 8.63 | 10.82 | 96.82 | 0.0720 | 63.76 | 5.58 | 7.01 | 97.02 | 86.34 | 29.02 | 62.96
ARMwavLM-S (ours) | 22.39 | 78.9 | 7.42 | 10.03 | 97.01 | 0.0741 | 71.29 | 5.99 | 7.11 | 97.76 | 87.41 | 26.97 | 64.54
100h distillation |  |  |  |  |  |  |  |  |
FitW2V2 [20] | 22.49 | 73.1 | 16.50 | 14.77 | 94.68 | 0.0380 | 51.65 | 7.43 | 6.94 | 90.03 | 81.95 | 34.74 | 62.87
FitHuBERT [20] | 22.49 | 74.5 | 14.05 | 12.66 | 96.23 | 0.0579 | 54.24 | 7.88 | 7.19 | 94.20 | 83.41 | 34.00 | 61.67
ARMHuBERT-S (ours) | 22.39 | 76.8 | 9.17 | 11.83 | 96.01 | 0.0569 | 66.48 | 5.92 | 6.23 | 95.97 | 83.89 | 33.29 | 63.29
ARMwavLM-S (ours) | 22.39 | 77.0 | 8.33 | 11.37 | 96.30 | 0.0579 | 65.40 | 6.38 | 7.41 | 96.76 | 84.89 | 31.95 | 63.41

Attention map reusing is a technique for substituting the present layer's
attention map with the previous one, which has been covered in several domains
[23, 29]. Prior works [23, 24] have pointed out the similarity of the
attention maps across heads and layers in pretrained Transformer models, such
as BERT [12] and ViT [30]. We leverage this property by reusing the attention
maps to compress the student model. Alternatively, we can reassign the amount
of parameters saved by attention map reusing, without increasing the total
number of parameters.

In Transformer's multi-head self-attention (MHSA) module [26], the input
$x\in\mathbb{R}^{n\times d}$ with the sequence length $n$ is transformed to
$H$ independent queries, keys, and values by transformation matrices
$W_{h,k},W_{h,q}\in\mathbb{R}^{d\times d_{k}}$, and
$W_{h,v}\in\mathbb{R}^{d\times d_{v}}$, respectively, for each head $h$. Here,
$d_{k}$, $d_{v}$, and $d$ are the width of the keys, values, and model,
respectively.

$\begin{array}[]{cc}K_{h}=W_{h,k}x,&K_{h}\in\mathbb{R}^{n\times d_{k}},\\\
Q_{h}=W_{h,q}x,&Q_{h}\in\mathbb{R}^{n\times d_{k}},\\\
V_{h}=W_{h,v}x,&V_{h}\in\mathbb{R}^{n\times d_{v}}\end{array}$ (1)

Then, key and query are multiplied along the width axis to obtain a scaled
dot-product attention map, $A_{h}\in\mathbb{R}^{n\times n}$. Linear
combinations of the attention map and value for each head are concatenated,
and then projected to the original width.

$\displaystyle
A_{h}=\textit{softmax}\big{(}Q_{h}K^{\top}_{h}/\sqrt{d_{k}}\big{)},$ (2)
$\displaystyle\textit{MHSA}(x)=\big{[}A_{1}V_{1},\dots,A_{H}V_{H}\big{]}W_{o},\,\,W_{o}\in\mathbb{R}^{Hd_{v}\times
d}$ (3)

Attention map reusing is to replace $A_{h}$ with the previous layer's one. For
instance, if we reuse the $k$-th previous attention map on the current layer
$\ell$, the ReuseMHSA module is

$\begin{array}[]{c}\textit{ReuseMHSA}(x)=\big{[}{A_{1}^{\ell-k}}{V_{1}^{\ell}},\dots,{A_{H}^{\ell-k}}{V_{H}^{\ell}}\big{]}\,{W_{o}^{\ell}}.\\\
\end{array}$ (4)

Accordingly, computing $K_{h}$ and $Q_{h}$ can be omitted, reducing the number
of multiplications and additions by $(2nd^{2}+n^{2}d)$. Assuming $d/H$ $=$
$d_{v}$ $=$ $d_{k}$, the omitted computation accounts for half of the original
computation for MHSA, which is $(4nd^{2}+2n^{2}d)$. As a result, less
parameters and multiply-accumulates (MACs) are required as more ReuseMHSA
modules are employed (see Sec. 5.1).

###  3.2 Masking Distillation

Attention map reusing has reduced the number of parameters, however, it may
affect the representation quality of the student model. To improve the
student's representation learning, we offer a novel masking distillation
scheme that leverages the teacher's representation knowledge in a more
sophisticated way.

Speech frame masking involves learning representation through masked
prediction, where the model learns to represent masked frames accurately based
on other unmasked frames. LightHuBERT [21], inspired by data2vec [8], has
first applied the masking strategy to distilling HuBERT. In this approach, the
teacher model guides the representation of masked frames. Let $\mu(x)$ be the
masked input, and $f^{t}$ and $f^{s}$ the teacher and student model. Then, the
masked loss function becomes

$\mathcal{L}(x)=\frac{1}{|M|}\sum_{i\in
M}\big{\lVert}f^{t}_{i}(x)-f^{s}_{i}(\mu(x))\big{\rVert}_{2}$ (5)

where $f_{i}$ is the $i$-th frame of the speech representation, and $M$ is the
set of the masked frames.

In addition to the masked part loss (eq. 5), we suggest to employ an unmasked
loss since the teacher model can provide high-quality representation even on
the unmasked frames. However, if the masking process removes essential frames,
distilling the intact form of $f^{t}(x)$ can leak such essential knowledge
that should have been removed. This induces biased predictions of the student,
as it learns information that cannot be inferred from the masked input.

To prevent this, we make the teacher model receive the same masked input as
the student does when distilling the unmasked part. Hence, the entire
distillation loss becomes

$\displaystyle\mathcal{L}(x)$
$\displaystyle=\sum_{\ell}\alpha_{\ell}\big{[}\mathcal{L}_{m,\ell}(x)+\mathcal{L}_{u,\ell}(x)\big{]}$
$\displaystyle=\sum_{\ell}\frac{\alpha_{\ell}}{|M|}\sum_{i\in
M}\big{\lVert}f^{t}_{i,\ell}(x)-f^{s}_{i,\ell}(\mu(x))\big{\rVert}_{2}$ (6)
$\displaystyle+\sum_{\ell}\frac{\alpha_{\ell}}{n-|M|}\sum_{i\notin
M}\big{\lVert}f^{t}_{i,\ell}(\mu(x))-f^{s}_{i,\ell}(\mu(x))\big{\rVert}_{2}$

where $\alpha_{\ell}$ is the layerwise coefficient. $\mathcal{L}_{m,\ell}$ and
$\mathcal{L}_{u,\ell}$ represent masked loss and unmasked loss of the
$\ell$-th layer, respectively.

In summary, our novel masking distillation strategy appropriately guides the
student's knowledge acquisition, by distilling not only the masked
representation of unmasked data but also the unmasked representation of masked
data (see Fig. 1). In Sec. 5.2, we investigate the strength of our masking
strategy compared to other types of losses.

##  4 Results

###  4.1 Implementation Details

We distilled the two dominant Transformer-based speech SSL models, HuBERT Base
[10] and wavLM Base [11], that are pretrained on LibriSpeech 960 hours dataset
[17]. Our student model consists of 12 layers of Transformers as the teachers,
while the detailed design mostly follows FitHuBERT [20]: width of attention
and FFN reduced and linear projections adopted at each layer. The layerwise
coefficients $\alpha_{\ell}$ are set to 0.1 except for the last layer, where
it is set to 1. Unless specified, the LibriSpeech [17] dataset is distilled
for 200 epochs with effective batch size of 72 including gradient
accumulation.

Reuse pattern We employ an alternating reuse pattern for the attention maps,
whereby the attention map of an even-numbered Transformer layer is repeated by
that of the previous odd-numbered layer. We denote this pattern as 2by6, our
default setting. We examine other reuse patterns in Sec. 5.1 in terms of
performance, number of parameters, and MACs.

Model description To verify our masking distillation strategy, we first build
a student model, MaskHuBERT, which employs masking distillation only.
MaskHuBERT has the width of (attention, FFN) as (480, 640). Then, 2by6 reuse
pattern is applied to MaskHuBERT, leading to 10.3% of parameter reduction. We
extend this model to two options: ARMHuBERT and ARMHuBERT-S. ARMHuBERT is a
reinvested version of MaskHuBERT, where the saved parameters from attention
map reusing are reassigned to FFN, resulting in increased width of (480, 864).
ARMHuBERT-S is a reduced version to match the parameters with previous works,
having the width of (432, 816). To establish the universality of our strategy,
we introduce ARMwavLM-S that is structurally identical to ARMHuBERT-S, with
the only change in teacher from HuBERT to wavLM.

###  4.2 SUPERB Benchmark Results

In Table 1, we evaluate our student models on SUPERB benchmark [25]. We follow
the default fine-tuning recipes, including a learning rate scheduler, with the
learning rate scaled to 10$\times$ in SID task. MaskHuBERT outperforms 12-L
Half-L2L, the previous state-of-the-art E2E distillation method, with less
parameters used. Our observation indicates that incorporating our masking
strategy into the L2L distillation [20, 22] results in enhancing the student's
representation quality. Especially, MaskHuBERT highly improves the
performances in content- and semantics-related tasks.

ARMHuBERT achieves a better overall score of 78.1 with less parameters than
MaskHuBERT. Despite the removal of certain attention parameters, increasing
the FFN width contributes to better quality of speech representation,
achieving 7.72% PER and 9.96% WER. We find out that ARMHuBERT shows promising
improvements when compared to MaskHuBERT in SF and SID tasks, exhibiting a
similar level of performance in other tasks. In the end, the number of
parameters and MACs in ARMHuBERT have decreased to 28% and 30% of the teacher
model, HuBERT Base [10], respectively.

In a smaller parameter group, ARMHuBERT-S, the param-eter-reduced version,
outperforms DistilHuBERT and FitHuBERT by a large margin. Specifically,
ARMHuBERT-S also shows the outstanding results in content- and semantics-
related tasks, which means the consistency of the representations produced by
MaskHuBERT and ARMHuBERT-S. In addition, the result that ARMwavLM-S surpasses
ARMHuBERT-S implies the universality of our strategy: without any
modifications of the student model structure, replacing with a superior
teacher model creates a better student. The results of the LibriSpeech [17]
100h distillation are also consistent with the formerly demonstrated results.

##  5 Discussions

Table 2: Performance comparisons of various reusing patterns. Parameter size
(M) and MACs (G) are additionally measured. The width of (attention, FFN) for
each model is (432, 816), while ``-up'' suffix denotes more parameters
assigned to FFN to match with 2by6. Masking is not applied here.

pattern | reused layers |  params | MACs |  WER $\downarrow$ |  EER $\downarrow$ |  F1 $\uparrow$ |  CER $\downarrow$
---|---|---|---|---|---|---|---
6by2 | {1,7} | 20.90 | 423 | 13.52 | 6.30 | 83.69 | 34.92
3by4 | {1,4,7,10} | 21.65 | 437 | 12.37 | 5.67 | 83.29 | 33.60
6by2-up | {1,7} | 22.39 | 440 | 13.18 | 5.89 | 83.07 | 34.79
3by4-up | {1,4,7,10} | 22.39 | 445 | 12.39 | 6.06 | 83.79 | 33.49
2by6 | {1,3,5,7,9,11} | 22.39 | 450 | 12.18 | 5.95 | 84.91 | 32.29
None | - | 24.64 | 490 | 11.94 | 5.87 | 84.78 | 31.38

In this section, we explore which layer's attention map should be reused in
other layers and how to implement the masking distillation. Unless specified,
we have conducted the distillations on 100-hour of LibriSpeech [17] and
evaluated on the ASR, ASV, and SF tasks of the SUPERB benchmark [25].

###  5.1 Where to Reuse

Table 2 summarizes the performance depending on various attention map reusing
patterns, and in general, the 2by6 pattern performs the best. Other reuse
patterns have reduced the Transformer's representation capacity due to overly
frequent reusing. Assigning more parameters to FFN (-up) still has limit in
terms of the performance gain. Comparing to no reuse pattern applied, the
performance decrease of 2by6 is small, but it takes advantages in 9.13% and
8.16% reduction of parameters and MACs, respectively. We note that the number
of MACs in a single reuse MHSA module (eq. 4) is reduced by half, from 13.2G
to 6.6G.

###  5.2 How to Mask

Masking strategy Table 3 shows the efficacy of our masking strategy. We first
eliminated the loss function on the unmasked frames ($\mathcal{L}_{u,\ell}$),
making it equivalent to the L2L version of the LightHuBERT [21] distillation
loss. This approach severely damaged performances, particularly in the ASR and
ASV tasks. Next, we modified the unmasked loss function to distill from the
unmasked input, i.e., only $f^{t}(x)$ being distilled to the student. This
also led to degraded performance in most tasks, revealing that our unmasked
loss with masked input properly guides the knowledge acquisition without
imposing biased predictions.

Masking ratio High value of masking ratio can lead to a student model
producing good representation, as it has less information to infer with [10,
31]. However, it can also make the learning process more difficult. In Table
4, we examine the optimal masking ratios for each training set. For
LibriSpeech 960h [17], both ratios of 0.4 and 0.8 produce excellent results.
On the other hand, for the 100h dataset, ratio of 0.4 produces the best
results overall. This implies that lower masking ratio is preferred in low-
resource distillation setting. Accordingly, in our main experiments, we have
used the ratios of 0.8 and 0.4 for the 960h and 100h distillation,
respectively.

Table 3: Ablation study on our masking strategy.

methods |  WER $\downarrow$ |  EER $\downarrow$ |  F1 $\uparrow$ |  CER $\downarrow$
---|---|---|---|---
MaskHuBERT-100h | 11.56 | 5.87 | 84.31 | 32.28
[–] distil. unmasked part | 13.23 | 7.96 | 82.78 | 33.53
[–] distil. from masked input | 11.65 | 6.09 | 84.29 | 31.41

Table 4: Performance comparisons with different masking ratios. ``sch''
indicates linear scheduling of the ratio as 0.4 to 0.8.

models | ratio |  WER $\downarrow$ |  EER $\downarrow$ |  F1 $\uparrow$ |  CER $\downarrow$
---|---|---|---|---|---
MaskHuBERT-960h | 0.4 | 9.75 | 5.58 | 86.94 | 26.79
MaskHuBERT-960h | 0.8 | 9.77 | 5.38 | 87.31 | 27.10
MaskHuBERT-100h | 0.4 | 11.56 | 5.87 | 84.31 | 32.28
MaskHuBERT-100h | 0.6 | 11.99 | 6.18 | 83.42 | 33.31
MaskHuBERT-100h | 0.8 | 12.74 | 6.56 | 83.68 | 33.82
MaskHuBERT-100h | sch | 12.07 | 6.29 | 83.84 | 33.50

##  6 Conclusion and Future Work

In summary, we have proposed the universal compression strategy which involves
attention map reusing and novel masking distillation. Our parameter-reinvested
model, ARMHuBERT, achieves great performance in content- and semantics-related
tasks. Our strategy can be applied to any Transformer-based speech SSL models,
and contributes to enhancing the general quality of speech representation.
Future work can focus on further improving our model on speaker-related tasks.

##  7 Acknowledgements

The study was supported by Korea Health Technology R&D Project through the
Korea Health Industry Development Institute funded by the Ministry of Health
and Welfare, Republic of Korea (HR18C0016).

## References

  * [1] A. T. Liu, S.-w. Yang, P.-H. Chi, P.-c. Hsu, and H.-y. Lee, ``Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 6419–6423.
  * [2] P.-H. Chi, P.-H. Chung, T.-H. Wu, C.-C. Hsieh, Y.-H. Chen, S.-W. Li, and H.-y. Lee, ``Audio albert: A lite bert for self-supervised learning of audio representation,'' in _Spoken Language Technology Workshop (SLT)_. IEEE, 2021, pp. 344–350.
  * [3] A. T. Liu, S.-W. Li, and H.-y. Lee, ``Tera: Self-supervised learning of transformer encoder representation for speech,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 2351–2366, 2021.
  * [4] S. Liu, A. Mallol-Ragolta, E. Parada-Cabaleiro, K. Qian, X. Jing, A. Kathan, B. Hu, and B. W. Schuller, ``Audio self-supervised learning: A survey,'' _Patterns_ , vol. 3, no. 12, p. 100616, 2022.
  * [5] T. Mikolov, K. Chen, G. Corrado, and J. Dean, ``Efficient estimation of word representations in vector space,'' _arXiv preprint arXiv:1301.3781_ , 2013\.
  * [6] M. Caron, P. Bojanowski, A. Joulin, and M. Douze, ``Deep clustering for unsupervised learning of visual features,'' in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 132–149.
  * [7] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, ``A simple framework for contrastive learning of visual representations,'' in _International conference on machine learning_. PMLR, 2020, pp. 1597–1607.
  * [8] A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, ``Data2vec: A general framework for self-supervised learning in speech, vision and language,'' in _International Conference on Machine Learning_. PMLR, 2022, pp. 1298–1312.
  * [9] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, ``wav2vec 2.0: A framework for self-supervised learning of speech representations,'' _Advances in Neural Information Processing Systems_ , vol. 33, pp. 12 449–12 460, 2020.
  * [10] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``Hubert: Self-supervised speech representation learning by masked prediction of hidden units,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [11] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao _et al._ , ``Wavlm: Large-scale self-supervised pre-training for full stack speech processing,'' _IEEE Journal of Selected Topics in Signal Processing_ , vol. 16, no. 6, pp. 1505–1518, 2022.
  * [12] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, ``Bert: Pre-training of deep bidirectional transformers for language understanding,'' in _Proceedings of NAACL-HLT_ , 2019, pp. 4171–4186.
  * [13] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu _et al._ , ``Conformer: Convolution-augmented transformer for speech recognition,'' in _Proc. Interspeech_ , 2020.
  * [14] S. Kim, A. Gholami, A. E. Shaw, N. Lee, K. Mangalam, J. Malik, M. W. Mahoney, and K. Keutzer, ``Squeezeformer: An efficient transformer for automatic speech recognition,'' in _Advances in Neural Information Processing Systems_ , 2022.
  * [15] Y. Wang, A. Boumadane, and A. Heba, ``A fine-tuned wav2vec 2.0/hubert benchmark for speech emotion recognition, speaker verification and spoken language understanding,'' _arXiv preprint arXiv:2111.02735_ , 2021.
  * [16] L. Pepino, P. Riera, and L. Ferrer, ``Emotion Recognition from Speech Using wav2vec 2.0 Embeddings,'' in _Proc. Interspeech_ , 2021, pp. 3400–3404.
  * [17] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, ``Librispeech: an asr corpus based on public domain audio books,'' in _2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2015, pp. 5206–5210.
  * [18] G. Hinton, O. Vinyals, J. Dean _et al._ , ``Distilling the knowledge in a neural network,'' _arXiv preprint arXiv:1503.02531_ , vol. 2, no. 7, 2015\.
  * [19] H.-J. Chang, S.-w. Yang, and H.-y. Lee, ``Distilhubert: Speech representation learning by layer-wise distillation of hidden-unit bert,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 7087–7091.
  * [20] Y. Lee, K. Jang, J. Goo, Y. Jung, and H. Kim, ``Fithubert: Going thinner and deeper for knowledge distillation of speech self-supervised learning,'' in _Proc. Interspeech_ , 2022.
  * [21] R. Wang, Q. Bai, J. Ao, L. Zhou, Z. Xiong, Z. Wei, Y. Zhang, T. Ko, and H. Li, ``Lighthubert: Lightweight and configurable speech representation learning with once-for-all hidden-unit bert,'' in _Proc. Interspeech_ , 2022.
  * [22] T. Ashihara, T. Moriya, K. Matsuura, and T. Tanaka, ``Deep versus wide: An analysis of student architectures for task-agnostic knowledge distillation of self-supervised speech models,'' in _Proc. Interspeech_ , 2022.
  * [23] T. Xiao, Y. Li, J. Zhu, Z. Yu, and T. Liu, ``Sharing attention weights for fast transformer,'' in _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI)_ , 2019, pp. 5292–5298.
  * [24] S. Bhojanapalli, A. Chakrabarti, A. Veit, M. Lukasik, H. Jain, F. Liu, Y.-W. Chang, and S. Kumar, ``Leveraging redundancy in attention with reuse transformers,'' _arXiv preprint arXiv:2110.06821_ , 2021.
  * [25] S. wen Yang, P.-H. Chi, Y.-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin, T.-H. Huang, W.-C. Tseng, K. tik Lee, D.-R. Liu, Z. Huang, S. Dong, S.-W. Li, S. Watanabe, A. Mohamed, and H. yi Lee, ``SUPERB: Speech Processing Universal PERformance Benchmark,'' in _Proc. Interspeech 2021_ , 2021, pp. 1194–1198.
  * [26] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, ``Attention is all you need,'' _Advances in neural information processing systems_ , vol. 30, 2017.
  * [27] A. v. d. Oord, Y. Li, and O. Vinyals, ``Representation learning with contrastive predictive coding,'' _arXiv preprint arXiv:1807.03748_ , 2018\.
  * [28] S. Schneider, A. Baevski, R. Collobert, and M. Auli, ``wav2vec: Unsupervised pre-training for speech recognition,'' in _Proc. Interspeech_ , 2019.
  * [29] K. Shim, J. Choi, and W. Sung, ``Understanding the role of self attention for efficient speech recognition,'' in _International Conference on Learning Representations_ , 2022.
  * [30] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , ``An image is worth 16x16 words: Transformers for image recognition at scale,'' in _International Conference on Learning Representations_ , 2021\.
  * [31] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, ``Masked autoencoders are scalable vision learners,'' in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 16 000–16 009.


###### Abstract

We introduce a novel extension to the Klinkahmer-vacuum defect model by
incorporating a fifth spatial coordinate, resulting in a comprehensive five-
dimensional wormhole space-time. This extension preserves its status as a
vacuum solution to the field equations in five-dimensions. We delve into the
behavior of geodesic equations in the proximity of this wormhole, shedding
light on its intriguing properties.

A new five-dimensional vacuum-defect wormhole space-time

Faizuddin<EMAIL_ADDRESS><EMAIL_ADDRESS>
Department of Physics, University of Science & Technology Meghalaya, Ri-Bhoi,
793101, India
(Dated: )

keywords: Higher-dimensional gravity; Modified theories of gravity; exact
solutions

pacs numbers: 04.50.-h; 04.50.Kd; 04.20.Jb

##  1 Introduction

Einstein’s four-dimensional theory of general relativity was extended to five
dimensions by Kaluza and Klein ss6 ; ss7  in an effort to unify gravitation
and electromagnetism. This innovative concept proposed that electromagnetism
could be incorporated by introducing an additional, compactified dimension
within the framework of four-dimensional space-time. In this model, the
formerly spatial fifth dimension became an integral part of the unified
theory.

The five-dimensional model employs a metric tensor denoted as $g_{AB}$, with a
corresponding line element given by $dS^{2}=g_{AB}\,dx^{A}\,dx^{B}$, where the
indices $A,B$ run through values $0,123,4$ to represent time, space, and the
extra dimension. This framework encompasses a four-dimensional manifold
characterized by a line element $ds^{2}=g_{\mu\nu}\,dx^{\mu}\,dx^{\nu}$, where
indices $\mu,\nu$ take on values $0,123$ to denote the four-dimensional space-
time. Within this context, the null-geodesics in the five-dimensional space-
time, given by $dS^{2}=0$, describe the motion of massive objects in the four-
dimensional space-time, where $ds^{2}<0$ ss1 ; ss2 ; ss3 . This connection
between null-geodesics and massive object motion necessitates the introduction
of a relation such as fifth coordinate $x^{4}=\chi=\chi(s)$, involving the
proper time in the four-dimensional space-time, to account for the embedding
of the four-dimensional manifold into the five-dimensional case. When the
four-dimensional component $g_{\mu\nu}$ within the five-dimensional space-time
is explicitly independent of the fifth dimension, specifically
$g_{\mu\nu}=g_{\mu\nu}(x^{\alpha})$ and $\partial g_{\mu\nu}/\partial\chi=0$,
the fifth coordinate assumes the role of particle mass, and this formulation
adheres to the weak equivalence principle ss3 ; ss4 . This also implies that
the fifth force is absent, leaving the accelerations exactly as they are in
four-dimensional theory. Null geodesics in five-dimensional manifolds has been
studied in ss1 .

The vacuum Einstein’s field equations for space-time plus extra dimension are
given in terms of the Ricci tensor by $R_{AB}=0$ (where $A,B=0,123,4$). This
is the simplest type of higher-dimensional field equations in general
relativity. This condition is relevant to the space-time-matter (STM), but can
be applied to some other scenarios. There are a number of known solutions that
embed four-dimensional manifolds with or without cosmological constant and/or
spherically symmetric character ss8 , whereby the extended version is known to
agree with observation ss8 ; ss9 ; ss10 . A few examples are static wormhole
solutions within the framework of five-dimensions Kaluza-Klein gravity in the
presence of a massless ghost four-dimensional scalar field ss11 , five-
dimensional Einstein equations with four-dimensional de Sitter-like expansion
ss12 , five-dimensional Gödel-type space-time ss13 ; ss15 , FRW cosmological
model ss14 , cosmological implications of nonseparable five-dimensional vacuum
field equations ss16 , spherically symmetric dyonic wormhole-like space-time
in five-dimensional Kaluza-Klein theory ss17 , axisymmetric regular
multiwormhole ss20  and stationary solutions ss18 ; ss19  in five-dimensional
general relativity theory, wormholes in $(4+1)$ gravity ss21 , a regular
vacuum solution in 5D gravity ss22 , classical and quantized aspects of
dynamics in five-dimensional relativity ss23 , five-dimensional black holes
ss24 .

In searching for a new five-dimensional solution to vacuum field equations, we
must keep in mind a known vacuum solution in four-dimensional theory. Our
investigation is based on a four-dimensional vacuum-defect wormhole metric
presented in FRK2  given by the following line-element (see also related work
FRK )

$ds^{2}|^{4D}_{\mbox{vacuum-
defect}}=-dt^{2}+\Big{(}1+\frac{b^{2}}{\xi^{2}}\Big{)}^{-1}\,d\xi^{2}+(\xi^{2}+b^{2})\,(d\theta^{2}+\sin^{2}\theta\,d\phi^{2}),$
(1)

where $-\infty<(t,\xi)<+\infty$ and other coordinates are in the usual ranges.
This four-dimensional wormhole metric is not only Ricci flat, $R_{\mu\nu}=0$
but also the Riemann flat, $R_{\mu\nu\alpha\beta}=0$.

Our primary objective in this study is to construct a novel five-dimensional
vacuum-defect wormhole model, which serves as a natural extension or
generalization of the existing vacuum-defect wormhole (1). Particular interest
given around a solution which are not only the Ricci flat, $R_{AB}$ but also
Riemann flat, $R_{ABCD}=0$ in five-dimensions. Notably, we focus on
maintaining the vacuum nature of the five-dimensional solution by excluding
any incorporation of a mass parameter, $M$, in contrast to the approach taken
in the work FRK3 , where a higher-dimensional extension of the vacuum-defect
wormhole was achieved by introducing a mass parameter $M$ linked to the
standard $4D$ Schwarzschild metric.

##  2 Analysis of five-dimensional vacuum-defect wormhole

As stated above, we aim to extend the four-dimensional Klinkhamer-vacuum
defect wormhole (1) into the higher-dimensional theory, especially in the
five-dimensional manifold. Therefore, the line-element describing this five-
dimensional wormhole metric in the chat $(t,\xi,\theta,\phi,\chi)$ is given by
$(c=1=G=\hbar)$

$\displaystyle ds^{2}|^{5D}_{\mbox{vacuum-defect}}=ds^{2}|^{4D}_{\mbox{vacuum-
defect}}+F(\xi,\theta)\,d\chi^{2},$ (2)

where the four-dimensional part $ds^{2}|^{4D}_{\mbox{vacuum-defect}}$ is given
by (1), $\chi$ is the fifth coordinate of extra dimensions which is a spatial
with range $\chi\in(0,\infty)$, and the function
$F(\xi,\theta)=D(\xi)\,H(\theta)$ with $D(\xi)$ and $H(\theta)$ are unknown
functions.

For the five-dimensional metric (2), the Einstein tensor $G_{AA}\neq 0$ with
an additional non-zero non-diagonal term given by

$\displaystyle
G_{\theta\xi}=G_{\xi\theta}=\frac{H^{\prime}}{4\,H}\,\Big{(}\frac{2\,\xi}{\xi^{2}+b^{2}}-\frac{D^{\prime}}{D}\Big{)},$
(3)

where $H^{\prime}$ and $D^{\prime}$ denotes derivative w. r. t. $\theta$ and
$\xi$, respectively.

Solving for vacuum field equations and considering first $G_{\theta\xi}=0$,
from equation (3), we obtain

$D(\xi)=c_{1}\,(\xi^{2}+b^{2}),$ (4)

where $c_{1}>0$ is an arbitrary constant choosing unity here for simplicity.

With this function $D(\xi)$, the five-dimensional line-element (2) now becomes

$\displaystyle
ds^{2}=-dt^{2}+\Big{(}1+\frac{b^{2}}{\xi^{2}}\Big{)}^{-1}\,d\xi^{2}+(\xi^{2}+b^{2})\,\Big{(}d\theta^{2}+\sin^{2}\theta\,d\phi^{2}+H(\theta)\,d\chi^{2}\Big{)},$
(5)

Still space-time (5 is a non-vacuum solution of the field equations. The non-
zero components of the Einstein tensor $G_{AB}$ for this metric (5) are

$\displaystyle G_{tt}=R/2\quad,\quad
G_{\xi\xi}=-\frac{\xi^{2}\,R}{2}\quad,\quad
G_{\theta\theta}=1+\frac{H^{\prime}}{2\,H}\,\cot\theta\quad,\quad
G_{\phi\phi}=\frac{\sin^{2}\theta}{4\,H}\,\Big{(}-H^{\prime
2}+4\,H^{2}+2\,H\,H^{\prime\prime}\Big{)}.$ (6)

where $R$ is the Ricci scalar given by

$\displaystyle R=\frac{-8\,H^{2}+H^{\prime
2}-2\,H\,(H^{\prime}\,\cot\theta+H^{\prime\prime})}{2\,(\xi^{2}+b^{2})\,H^{2}}.$
(7)

For vacuum field equations, the Einstein tensor, $G_{AB}=0$, and the Ricci
scalar. $R=0$. From equations (6) and (7), we obtain

$\displaystyle 1+\frac{H^{\prime}}{2\,H}\,\cot\theta=0\quad,\quad H^{\prime
2}=4\,H^{2}+2\,H\,H^{\prime\prime}\quad,\quad H^{\prime
2}=8\,H^{2}+2\,H\,(H^{\prime}\,\cot\theta+H^{\prime\prime}).$ (8)

Simplification of the above equations result the function $H(\theta)$ of the
following form

$H(\theta)=c_{2}\,\cos^{2}\theta,$ (9)

where $c_{2}>0$ is a constant choosing unity here for simplicity.

Therefore, the final form of the five-dimensional vacuum wormhole space-time
(5) using the function (9) is of the following

$\displaystyle ds^{2}|^{5D}_{\mbox{vacuum-defect}}$ $\displaystyle=$
$\displaystyle ds^{2}|^{4D}_{\mbox{vacuum-
defect}}+(\xi^{2}+b^{2})\,\cos^{2}\theta\,d\chi^{2}$ (10) $\displaystyle=$
$\displaystyle-
dt^{2}+\Big{(}1+\frac{b^{2}}{\xi^{2}}\Big{)}^{-1}d\xi^{2}+(\xi^{2}+b^{2})\Big{(}d\theta^{2}+\sin^{2}\theta\,d\phi^{2}+\cos^{2}\theta\,d\chi^{2}\Big{)}.$

We see that the four-dimensional part of the metric tensor $g_{AB}$ doesn’t
depend on the extra coordinate $\chi$, and hence, $\partial
g_{\mu\nu}/\partial\chi=0$, where $g_{\mu\nu}$ is the metric tensor of the
four-dimensional manifold given by (1). Thus, the metric $C_{5}$ of (10) is in
fact a pure-canonical form. One can calculate the Kretschmann scalar curvature
for this five-dimensional vacuum-defect wormhole space-time (10) and it will
be zero, $\mathcal{K}=R^{ABCD}\,R_{ABCD}=0$. In addition, this metric not only
Ricci-flat, $R_{AB}=0$ but also the Riemann-flat, $R_{ABCD}=0$. Hence, there
is a set of coordinates in which metric $C_{5}$ of (10) becomes Minkowski
space $M_{5}$ in five-dimensions:

$ds^{2}|^{5D}_{\mbox{Minkowski}}=-dT^{2}+dX^{2}+dY^{2}+dZ^{2}+dW^{2}$ (11)

The precise transformations between $(T,X,Y,Z,W)$ and
$(t,\xi,\theta,\phi,\chi)$ are complicated. However, in four-dimensional cases
such as for the metric (1), we have the following transformation

$X=\sqrt{\xi^{2}+b^{2}}\,\sin\theta\,\cos\phi,\quad
Y=\sqrt{\xi^{2}+b^{2}}\,\sin\theta\,\sin\phi,\quad
Z=\sqrt{\xi^{2}+b^{2}}\,\cos\theta$ (12)

that gives us the Minkowski space $M_{4}$ as

$ds^{2}|^{4D}_{\mbox{Minkowski}}=-dT^{2}+dX^{2}+dY^{2}+dZ^{2}.$ (13)

We see that this new five-dimensional wormhole space-times admits three
constants of motions associated with three killing vectors denoted as
$X_{t}=\partial_{t}$, $X_{\phi}=\partial_{\phi}$, and
$X_{\chi}=\partial_{\chi}$. The norm of the last killing vector given by

$X^{2}_{\chi}=|\partial_{\chi}\bullet\partial_{\chi}|=g_{\chi\chi}=(\xi^{2}+b^{2})\,\cos^{2}\theta>0$
(14)

is spacelike and finite at $\xi=0$ as well as on the axis of rotation
$\theta=0,\pi$. Furthermore, we also see that for $\chi=const$ hypersurface,
the wormhole space-time (10) reduces to the vacuum-defect defect metric (1).
That means the four-dimensional Klinkhamer-defect wormhole is embedded in a
five-dimensional space $M_{5}$ with the metric given by (10).

Now, we study the geodesics motions of test particles around this new five-
dimensional vacuum wormhole space-time (10). The geodesic equations are given
by

$\displaystyle\ddot{t}=0,$ (15)
$\displaystyle\ddot{\xi}=\frac{b^{2}}{\xi\,(b^{2}+\xi^{2})}\,\dot{\xi}^{2}+\frac{\xi^{2}+b^{2}}{\xi}\,\dot{\theta}^{2}+\frac{(\xi^{2}+b^{2})\,\sin^{2}\theta}{\xi}\,\dot{\phi}^{2}+\frac{(\xi^{2}+b^{2})\,\cos^{2}\theta}{\xi}\,\dot{\chi}^{2},$
(16)
$\displaystyle\ddot{\theta}=\frac{2\,\xi}{\xi^{2}+b^{2}}\,\dot{\xi}\,\dot{\theta}+\frac{1}{2}\,\sin
2\theta\,(\dot{\phi}^{2}-\dot{\chi}^{2}),$ (17)
$\displaystyle\ddot{\phi}=-\frac{2\,\xi}{\xi^{2}+b^{2}}\,\dot{\xi}\,\dot{\phi}-2\,\cot\theta\,\dot{\theta}\,\dot{\phi},$
(18)
$\displaystyle\ddot{\chi}=2\,\Big{(}-\frac{\xi}{\xi^{2}+b^{2}}\,\dot{\xi}+\tan\theta\,\dot{\theta}\Big{)}\,\dot{\chi}.$
(19)

where dot represents derivative w. r. t. affine parameter $\tau$.

The first derivative of last two equations results

$\dot{\phi}(\tau)=\frac{c_{3}}{(\xi^{2}+b^{2})\,\sin^{2}\theta},\quad\dot{\chi}(\tau)=\frac{c_{4}}{(\xi^{2}+b^{2})\,\cos^{2}\theta},$
(20)

where $c_{3},c_{4}$ are arbitrary positive constants. Substituting Eq. (20)
into the equations (16)-(17), one will get the second-order equations of
$\xi,\theta$ whose solutions are little bit complicated.

On the other hand, in the particular case $\theta=const\neq\pi/2$, from
geodesics equations (16)-(19), we obtain

$\displaystyle\dot{\phi}(\tau)=\dot{\chi}(\tau)=\frac{c_{0}}{\xi^{2}+b^{2}},\quad\ddot{\xi}=\frac{b^{2}\,\dot{\xi}^{2}+c^{2}}{\xi\,(\xi^{2}+b^{2})}.$
(21)

where $c_{0}$ is an arbitrary constant.

The null-path in five-dimensions with $ds^{2}|^{5D}_{\mbox{vacuum-defect}}=0$
in the pure-canonical metric (10) corresponds to the time-like path in four-
dimensions with $ds^{2}|^{4D}_{\mbox{vacuum-
defect}}=-(\xi^{2}+b^{2})\,\cos^{2}\theta\,d\chi^{2}<0$ of a massive object
ss1 ; ss2 ; ss3 . This means that a particle in four-dimensional metric
behaves as photon-like in five-dimensional metric or we can say all objects in
four-dimensional metric travel along null paths in five-dimensional metric. It
is interesting to note here that in the equatorial plane defined by
$\theta=\pi/2$, both the four-dimensional metric (1) and the five-dimensional
metric (10) shows the same geometrical structure

$\displaystyle
ds^{2}|_{\theta=\pi/2}=-dt^{2}+\Big{(}1+\frac{b^{2}}{\xi^{2}}\Big{)}^{-1}\,d\xi^{2}+(\xi^{2}+b^{2})\,d\phi^{2}.$
(22)

##  3 Conclusions

This newly developed five-dimensional vacuum-defect wormhole metric serves as
an extension of the established four-dimensional Klinkhamer-vacuum defect
model. It’s worth highlighting that upon scrutinizing the geometric
characteristics of both vacuum-defect wormhole models, as denoted by equations
(1) and (10), a striking similarity becomes evident, particularly when
considering in the equatorial plane defined by $\theta=\pi/2$. Furthermore, an
intriguing observation arises when investigating our wormhole model on
surfaces where $\chi$ remains constant. One can see that the well-established
Klinkhamer-vacuum defect metric is embedded in the five-dimensional wormhole
metric.

## References

  * (1) T. Kaluza, Sitz. Preuss. Akad. Wiss. 33, 966 (1921). Int. J. Mod. Phys. D 27 (2018) 1870001.
  * (2) O. Klein, Z. Phys. 37, 895 (1926).
  * (3) S. S. Seahra and P. S. Wesson, Gen. Relat. Gravit. 33 (2001) 1731.
  * (4) D. Youm, Mod. Phys. Lett. A 16 (2001) 2371.
  * (5) P. S. Wesson, Class. Quantum Grav. 19 (2002) 2825.
  * (6) C. M. Will, Theory and Experiment in Gravitational Physics, Cambridge University Press, Cambridge (2018).
  * (7) P. S. Wesson, Space-Time-Matter, World Scientific, Singapore (1999).
  * (8) D. Kalligas, P. S. Wesson and C. W. F. Everitt, Astrophys. J 439, 548 (1995).
  * (9) J. P. de Leon, Gen. Relat. Gravit. 20, 539 (1988).
  * (10) V. Dzhunushaliev and V. Folomeev, Mod. Phys. Lett. A 29, 1450025 (2014).
  * (11) T. Liko and P. S. Wesson, J. Math. Phys. 46, 062504 (2005).
  * (12) M. J. Reboucas and A. F. F. Teixeira, J. Math. Phys. 39 (1998) 2180.
  * (13) J. B. Fonseca-Neto, C. Romero and F. Dahia, Mod. Phys. Lett. A 21, 525 (2006).
  * (14) P. S. Wesson, H. Liu and S. S. Seahra, Astron. Astrophys. 358, 425 (2000).
  * (15) T. Fukui, S S. Seahra and P. S Wesson, J. Math. Phys. 42, 5196 (2001).
  * (16) C.-M. Chen, Class. Quantum Grav. 18, 4179 (2001).
  * (17) G. Clément, Gen. Relat. Gravit. 16, 477 (1984).
  * (18) G. Clément, Gen. Relat. Gravit. 18, 137 (1986).
  * (19) G. Clément, Gen. Relat. Gravit. 18, 861 (1986).
  * (20) A. G. Agnese, A. P. Billyard, H. Liu and P. S. Wesson, Gen. Relat. Gravit. 31, 527 (1999).
  * (21) V. Dzhunushaliev and H.-J. Schmidt, arXiv:gr-qc/9908076.
  * (22) P. S Wesson, Class. Quantum Grav. 19, 2825 (2002).
  * (23) S. S. Seahra and P. S. Wesson, J. Math. Phys. 44, 5664 (2003).
  * (24) F. R. Klinkhamer, Acta Phys. Polon. B 54, 7-A3 (2023).
  * (25) F. R. Klinkhamer, Acta Phys. Polon. B 54, 5-A3 (2023).
  * (26) F. R. Klinkhamer, arXiv: 2307.12876 [gr-qc].