# DocKylin: A Large Multimodal Model for
Visual Document Understanding with
Efficient Visual Slimming

Jiaxin Zhang 1, Wentao Yang 1,∗ Songxuan Lai 2 Zecheng Xie 2,† Lianwen Jin 1,†
1South China University of Technology 2Huawei Cloud
Equal Contribution. † Corresponding Author. This work was conducted during
Jiaxin Zhang and Wentao Yang’s internship at Huawei Cloud. Emails:
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Current multimodal large language models (MLLMs) face significant challenges
in visual document understanding (VDU) tasks due to the high resolution, dense
text, and complex layouts typical of document images. These characteristics
demand a high level of detail perception ability from MLLMs. While increasing
input resolution improves detail perception, it also leads to longer sequences
of visual tokens, increasing computational costs and straining the models’
ability to handle long contexts. To address these challenges, we introduce
DocKylin, a document-centric MLLM that performs visual content slimming at
both the pixel and token levels, thereby reducing token sequence length in VDU
scenarios. DocKylin utilizes an Adaptive Pixel Slimming (APS) preprocessing
module to perform pixel-level slimming, increasing the proportion of
informative pixels. Moreover, DocKylin incorporates a novel Dynamic Token
Slimming (DTS) module to conduct token-level slimming, filtering essential
tokens and removing others to create a compressed, adaptive visual sequence.
Experiments demonstrate DocKylin’s promising performance across various VDU
benchmarks. Notably, both the proposed APS and DTS are parameter-free,
facilitating easy integration into existing MLLMs, and our experiments
indicate their potential for broader applications.

##  1 Introduction

With the substantial growth in data and model sizes, as well as alignment with
human preferences, current Large Language Models (LLMs) [45, 65, 63, 2, 62,
73] have demonstrated remarkable capabilities in reasoning, common sense
understanding, expertise in specialized knowledge, zero/few-shot learning, and
instruction following, shedding light on the potential for Artificial General
Intelligence. Multimodal Large Language Models (MLLMs) aim to enhance these
powerful LLMs by endowing them with multimodal capabilities, enabling the
integration of diverse modalities beyond text alone.

Although there has been significant progress in the development of MLLMs,
their performance in visual document understanding (VDU) tasks remains
suboptimal [36, 78]. A primary reason for this is their support for input
resolutions of only up to 448x448, which is inadequate for high-resolution,
fine-grained document images. Numerous efforts [76, 37, 29, 70, 3, 74, 32, 72]
have recently been made to increase the input resolution to address this
limitation and achieved notable improvements. However, these efforts still
face several challenges: 1) Increasing the resolution also escalates the
redundant regions within documents, which results in an increased number of
redundant visual tokens, complicating the task for LLMs to identify the
correct answers. 2) Some methods [29, 70, 72, 3] employ a fixed rectangular
resolution, which can lead to text distortion given the typically large aspect
ratios of document images. Moreover, scaling smaller images to larger ones
results in inefficiencies. 3) To manage the extended sequences of visual
tokens resulting from high resolution, most methods [29, 76, 37, 3] employ a
fixed compression ratio or extract a predetermined number of tokens. While
this approach may be suitable for natural scene images, it is less effective
for document scenarios where content density varies significantly. For
example, images of scientific papers are information-dense, whereas images of
identification cards are comparatively sparse.

To tackle these challenges, we propose a series of solutions. Firstly, we
propose a parameter-free Adaptive Pixel Slimming (APS) preprocessing technique
that utilizes gradient information to identify and eliminate redundant regions
within document images, thereby reducing the proportion of redundant pixels
and enhancing computational efficiency. Secondly, we adopt a more flexible
image encoding strategy by capping the maximum number of pixels instead of
fixing resolutions. This allows for variable resolutions and aspect ratios.
Our lightweight Swin encoder [38] processes high-resolution document images
holistically, ensuring feature coherence. Thirdly, we propose a Dynamic Token
Slimming (DTS) method based on k-means clustering [39], which is also
parameter-free. DTS filters useful tokens from a vast pool of visual tokens,
obtaining visual sequences with reduced and dynamic lengths. Incorporating
these innovations, we developed our DocKylin, which demonstrates superior
performance across multiple VDU benchmarks. Ablation studies further confirm
the effectiveness of some of these designs and highlight their potential
applicability to other MLLMs or scenarios.

##  2 Related Works

###  2.1 Text-centric MLLMs

Text is crucial for understanding images, but the ability of existing MLLMs in
this regard is generally subpar [36, 51, 61]. Therefore, there has been a
surge of recent work studying how to enhance the perception, recognition, and
understanding abilities of text images in MLLMs. Some approaches [58, 23]
leverage additional Optical Character Recognition (OCR) engines to recognize
text in images and input it into MLLMs. Others [29, 37, 3, 12, 13, 74, 70]
enhance the fine-grained perceptual capabilities of MLLMs by maintaining the
high resolution of the input images. This includes strategies such as
splitting the image into small pieces to accommodate the VIT’s requirement for
fixed and small-size inputs and utilizing visual encoders that are more
suitable for text images. Authors of [14, 69] explore learning text grounding
for MLLMs, while [16, 60] investigate the use of more superior and larger
scale instruction data to boost performance. Although these efforts have
significantly improved the performance of MLLMs on text-centric tasks, there
is still substantial room for improvement.

###  2.2 Visual Token Compression in MLLMs

Maintaining high-resolution input images is a straightforward and effective
method to enhance the performance of MLLMs on text images. However, increased
resolutions inevitably lead to a longer sequence of visual tokens. There are
various methods for compressing the length of visual sequences in current
MLLMs. One of the earlier and commonly used approaches [11, 27, 1] involves
utilizing a group of learnable query tokens to extract information in a cross-
attention manner. Another method [7, 76] involves concatenating adjacent
tokens along the channel dimension to reduce sequence length. Additionally,
convolutional neural networks (CNNs) have also been proven effective for
visual token compression [6, 16]. However, all these methods either extract
tokens of a fixed length or compress them at a fixed ratio. This approach is
unsuited for document images, where the content density can vary
significantly. A token sequence that is too short or a compression ratio that
is too high can result in an inadequate representation of some document
images. Conversely, longer token sequences or lower compression rates can
reduce computational efficiency.

Recent studies [5] have shown that the optimal length of visual token
sequences varies for each sample. Consequently, some recent approaches have
explored the adaptive compression of visual sequences. Most of these methods
[34, 49, 8, 31] dynamically discard nonessential tokens using attention scores
between visual tokens or between visual tokens and class tokens. However,
considering the differences between text and natural objects with complete
semantics, this paradigm may not be suitable for text images. For text-centric
images, HRVDA[72] employs text position annotations to train a separate visual
token filtering network, which requires introducing additional parameters
training procedures. Compared to the methods above, we explore alternative
dynamic compression schemes specifically for document images, which include
pixel-level redundancy removal and unsupervised clustering for filtering out
nonessential tokens.

##  3 Methodology

Figure 1: The overall architecture of our DocKylin model.

As illustrated in Fig. 1, similar to many multimodal large language models
(MLLMs), our architecture also incorporates an image encoder, an MLP layer
that aligns the visual modality to the language modality, and a LLM. Building
on this framework, DocKylin includes the following enhancements: Images are
preprocessed by the Adaptive Pixel Slimming module before being fed to the
image encoder, which removes redundant regions and reduces resolution while
increasing the proportion of informative pixels. The visual encoder takes the
resulting image as input and maintains the aspect ratio. The visual tokens
obtained by the visual encoder and MLP layer are handled by our Dynamic Token
Slimming (DTS) module to remove nonessential tokens to enhance both
performance and efficiency. We provide a detailed description of these
improvements in the following.

###  3.1 Adaptive Pixel Slimming

Perceiving textual information in images typically requires maintaining a high
resolution of input, especially for text-rich document images. This poses
significant challenges for current resource-intensive MLLMs. To address this
issue, existing methods employ strategies such as dividing the images into
patches for processing [37, 74], employing visual encoders that support high
resolution [70], leveraging OCR results as additional inputs [58, 23], or
applying compression transformations to the images beforehand [13].

Here, we propose a novel approach called Adaptive Pixel Slimming (APS) to
efficiently reduce the resolution of document images, which is orthogonal to
the approaches mentioned above. APS is based on the observation that, as
illustrated in Fig. 2, while document images usually exhibit high resolution,
they commonly contain a significant proportion of redundant regions. These
regions not only increase the computational burden on the visual encoder but
may also pose challenges [37, 72] to the reasoning capabilities of the LLMs by
the increased redundant visual tokens. APS is thus designed to identify and
remove these regions. Specifically, as shown in Fig. 2, during the gradient
extraction stage, we utilize the Sobel operator to extract the gradient map
$I_{G}$ from the original image $I_{S}$. Regions with low gradient values are
considered smooth and devoid of significant information. Therefore, in the
redundant region determination stage, we identify the redundant regions in the
horizontal and vertical directions by locating areas with consistently low
gradient values. For instance, to identify redundant regions in the image’s
horizontal direction: if the sum of gradient values for a specific row is
below a predefined threshold, we classify that row as redundant. A sequence of
contiguous redundant rows constitutes a redundant region in the horizontal
direction. The computation for the vertical direction follows a similar
procedure. The final image can be obtained by removing the identified
redundant rows and columns. For more details on the algorithm and parameter
settings, please refer to the Appendix.

Table 1 presents the reduction in image resolution after applying our APS
module, which demonstrates that APS significantly decreases the resolution of
images. Some visual results are available in the Appendix. It is worth noting
that APS is primarily designed for document images, where redundant regions
are commonly distinctly defined. In contrast, elements in natural scene images
tend to be more complex, making it challenging to identify extensive redundant
regions. Consequently, APS generally has a limited impact on natural scene
tasks. In the subsequent experimental section of our paper, we demonstrate
that the application of APS consistently improves performance on multiple
document image benchmarks without adversely affecting tasks involving natural
scenes.

Figure 2: Schematic representation of the Adaptive Pixel Slimming module. It effectively reduces the resolution of document images by removing redundant regions. Table 1: Changes in image resolution before and after the application of Adaptive Pixel Slimming, as well as the resulting reduction in pixel count. Datasets | Adaptive Pixel Slimming | Avg. Resolution | Reduction
---|---|---|---
DocVQA | ✗ |  1748$\times$2088 | -
✓ |  1531$\times$1524 | 36.0%
InfoVQA | ✗ |  1181$\times$2541 | -
✓ |  1120$\times$2193 | 18.1%

###  3.2 More Flexible Image Encoding

Most existing MLLMs [29] utilize a fixed rectangular resolution, such as
224$\times$224 or 336$\times$336\. This approach distorts the original aspect
ratio of the images, leading to text deformation and inefficiencies when
upscaling images smaller than the preset resolution. While some methods [74,
76] allow for more flexible aspect ratios and resolutions, they often employ a
patch-based processing strategy. Such a strategy may be suitable for natural
scene images where the continuity and context of visual elements are less
critical. However, it is less effective for document images [37] that contain
dense text. Segmenting such images can disrupt the textual content, severing
lines of text or splitting words and sentences across different patches. This
discontinuity can impair the ability to accurately recognize and interpret
textual information essential for effective document image understanding.

We employ a more flexible approach that can accommodate varying resolutions
and aspect ratios without the need for image patching. Specifically, for an
image obtained by APS with the size of H$\times$W, we set a maximum allowable
number of pixels, MAX_SIZE. If the product of H and W is less than MAX_SIZE,
the image will not be resized. Otherwise, a scaling factor
$r=\mathrm{Int}(\sqrt{\frac{\mathrm{MAX\\_SIZE}}{H\times W}})$ is calculated,
and then the image is resized to rH$\times$rW. Here $\mathrm{Int}()$ denotes
the floor function to ensure that the resulting image does not exceed the
maximum allowable pixels. One advantage of setting a maximum number of pixels
rather than fixed dimensions is the ability to support more varied resolutions
and preserve the original aspect ratio. Additionally, this manner avoids the
loss of image details due to resolution reduction as much as possible without
increasing computational resource consumption. For example, if we set the
maximum pixels number MAX_SIZE as 960$\times$960, we can accommodate an input
resolution of 1280$\times$720 without alteration. In contrast, a fixed-
dimension strategy would require resizing the image to 960$\times$960,
resulting in a 25% compression of the long side. Both strategies consume
comparable computational resources since the number of visual tokens remains
consistent.

Although ViT pre-trained with CLIP [48] demonstrates impressive generalization
capabilities for natural scene images, they perform suboptimally on dense
document images [28]. Furthermore, their self-attention mechanism, which has a
computational complexity that scales quadratically with image resolution,
makes it challenging to support high-resolution inputs. Consequently, we opt
to use the more efficient Swin Transformer [38] as our visual encoder, which
has been pre-trained on full-text recognition tasks [22]. It enables us to
process entire images as single inputs.

Figure 3: Schematic diagram of the proposed Similarity Weighted Aggregation
module. The aggregation module aggregates nonessential tokens into essential
tokens through similarity weighted summation.

###  3.3 Dynamic Token Slimming

Although the APS module has been effective in removing some redundant regions,
it is limited to processing entire rows or columns and is ineffective for more
complex scenarios. In contrast, our Dynamic Token Slimming (DTS) method offers
a more flexible approach to eliminate redundant visual content at the token
level. The design of the DTS is based on the premise that a well-trained
visual encoder should be capable of effectively distinguishing between
essential and nonessential regions within an image, with the features
corresponding to different regions exhibiting sufficient discriminative
properties. We propose to separate these two types of tokens directly through
clustering and then aggregate nonessential tokens into essential tokens. We
next detail the proposed DTS, which comprises two main processes: Dual-center
K-Means Clustering and Similarity Weighted Aggregation.

Algorithm 1 Dual-center K-Means Clustering

1:

2:visual tokens from Image Encoder: $V=\mathbb{R}^{L\times D}$

3:visual tokens from MLP Layer:
$V^{\prime}=\text{MLP}(V)\in\mathbb{R}^{L\times D^{\prime}}$

4:CMX (calculate min similarity)

5:$R=50$

6:$\text{cluster1, cluster2}=K\text{-means}(V,\text{center\\_num}=2)$

7:max_similarities = []

8:for $\text{token in }V^{\prime}$:

9: $\text{max\\_similarities}.\text{append}(\text{CMX}(\text{token, other
tokens in }V^{\prime}))$

10:$\text{top-R\\_similar\\_tokens}=\text{select\\_top}(V^{\prime},\text{max\\_similarities},R)$

11:num1 =num_of_same_tokens(top-R_similar_tokens, cluster1)

12:num2 =num_of_same_tokens(top-R_similar_tokens, cluster2)

13:if num1>num2:

14: essential_tokens = cluster2

15: nonessential_tokens = cluster1

16:else:

17: essential_tokens = cluster1

18: nonessential_tokens = cluster2

19:return essential_tokens, nonessential_tokens

Dual-center K-Means Clustering. To distinguish tokens into essential and
nonessential types without introducing additional parameters, we apply K-Means
clustering [39] with two centers to the output sequence from the visual
encoder, as illustrated in Fig 1. However, after obtaining the two clusters,
it is unclear which cluster contains the essential tokens and which contains
the non-essential tokens. Drawing on findings from [37], nonessential tokens
typically lack uniqueness and are often similar to other tokens. Therefore, we
identify potential nonessential tokens by evaluating the similarity of each
visual token with all other ones. The cluster with fewer of these nonessential
tokens is determined to be the one containing the essential tokens. For more
implementation details, please refer to Algorithm 1.

Similarity Weighted Aggregation. After obtaining essential and nonessential
tokens, how to handle the nonessential tokens becomes a crucial issue.
Inspired by [24, 30, 71], to mitigate potential clustering errors and prevent
information loss, we do not discard the nonessential tokens outright. Instead,
we aggregate them into the essential tokens to minimize the potential loss of
effective information while enhancing the model’s processing efficiency. As
illustrated in Fig. 3, for each nonessential token from the clustering step,
we find the most similar essential token. Each essential token may correspond
to zero or multiple nonessential tokens. If an essential token has
corresponding nonessential tokens, we perform a similarity weighted summation
of these tokens to obtain an aggregated token.

Specifically, after determining the set of essential tokens
$\mathcal{E}=\\{\boldsymbol{e}_{1},\boldsymbol{e}_{2},\boldsymbol{e}_{3},\ldots,\boldsymbol{e}_{N}\\}$
and the set of nonessential tokens
$\mathcal{\overline{E}}=\\{\overline{\boldsymbol{e}}_{1},\overline{\boldsymbol{e}}_{2},\overline{\boldsymbol{e}}_{3},\ldots,\overline{\boldsymbol{e}}_{\overline{N}}\\}$,
we aggregate nonessential tokens $\overline{\boldsymbol{e}}_{j}$ into
essential tokens $\boldsymbol{\boldsymbol{e}}_{i}$ by similarity weighted
summation:

$\footnotesize\hat{\boldsymbol{e}}_{i}=w_{i}\boldsymbol{e}_{i}+\sum_{\overline{\boldsymbol{e}}_{j}\in\mathcal{\overline{E}}}w_{j}\overline{\boldsymbol{e}}_{j},$
(1)

where $w_{i}$ and $w_{j}$ are the weight for the $i$-th essential and $j$-th
nonessential token, and $\hat{\boldsymbol{e}}_{i}$ is the aggregated token. We
use the similarity between $e_{i}$ and $\overline{e}_{j}$ to determine $w_{i}$
and $w_{j}$. Specifically, we first define the similarity matrix between
essential tokens and nonessential tokens:

$\footnotesize
c_{i,j}=\frac{\boldsymbol{e}_{i}^{T}\overline{\boldsymbol{e}}_{j}}{\|\boldsymbol{e}_{i}\|\|\overline{\boldsymbol{e}}_{j}\|}.$
(2)

Next, we obtain a mask matrix to ensure that each nonessential token is
aggregated only with the most similar essential token:

$\footnotesize
m_{i,j}=\begin{cases}1,&i=\mathop{\arg\max}\limits_{\overline{\boldsymbol{e}}_{j}\in\mathcal{\overline{E}}}c_{i,j},\\\
0,&\mathrm{otherwise},\end{cases}.$ (3)

Finally, we can obtain aggregating weight $w_{j}$ and $w_{i}$ by:

$\footnotesize
w_{j}=\frac{\exp(c_{i,j})m_{i,j}}{\sum_{\overline{\boldsymbol{e}}_{j}\in\mathcal{\overline{E}}}\exp(c_{i,j})m_{i,j}+\mathrm{e}}$
(4)

and

$\footnotesize
w_{i}=\frac{e}{\sum_{\overline{\boldsymbol{e}}_{j}\in\mathcal{\overline{E}}}\exp(c_{i,j})m_{i,j}+\mathrm{e}}.$
(5)

The pseudo-code for the specific implementation can be found in Algorithm 3 in
the appendix.

Table 2: Datasets used in pre-training and instruction tuning phase. Stage |  Datasets
---|---
Pre-training |  CASIA-HWDB [33], CCpdf [66], ChartQA [40], DeepForm [57], DocVQA [42], DVQA [20], FigureQA [21], InfoVQA [41], KLC [54], OCRVQA [44], PlotQA [43], PubTabNet [79], RVL-CDIP [15], SCUT-HCCDoc [77], SynthDog [22], TAT-QA [80], VisualMRC [59], WildReceipt [56]
Instruction tuning |  ChartQA [40], DeepForm [57], DocILE [52], DocVQA [42], DVQA [20], InfoVQA [41], InsuranceQA [55], KLC [54], OCR-TROSD[64], TAT-QA [80], VisualMRC [59], WildReceipt [56], WTQ [46]

##  4 Experiments

###  4.1 Implementation Details

Our model underwent two training stages: pre-training and instruction tuning.
Training data used in each phase are listed in Table 2. During the pre-
training stage, we trained only the visual encoder and the MLP, keeping the
LLM parameters fixed. In this phase, the model learns to perform full-text
recognition on document images and convert charts and tables into markdown
formats [16]. The goal is to enable the visual encoder to comprehend document
content and align the visual features with the language space through the MLP.

During the pre-training phase, the model was trained for 450,000 iterations
with a batch size of 8. The learning rate decayed from 1e-4 to 1e-5 using
cosine annealing, which took approximately 40 A800 GPU-days. In the
instruction tuning phase, the model underwent 300,000 iterations with the same
batch size and learning rate as in the pre-training phase. This phase further
consumed approximately 24 A800 GPU-days. Our Adaptive Pixel Slimming method is
employed in both phases, while the Dynamic Token Slimming (DTS) is used only
during the instruction tuning phase. The maximum allowed number of pixels,
MAX_SIZE, is set to 1728$\times$1728\. We initialize our visual encoder with
Donut-Swin (0.07B) [22], pre-trained on the reading texts task, and use
Qwen-7B-Chat [2] as our LLM.

###  4.2 Comparisons on VDU Benchmarks

We first compare our model with several existing general-purpose multimodal
large language models (MLLMs). As these models sometimes tend to provide
detailed responses, following [78], we adopt accuracy as our metric: a
response is considered correct if it contains the complete answer. The
results, as shown in Table 3, demonstrate that DocKylin achieves a significant
advantage over the competing models. This highlights that VDU tasks remain
challenging for current general-purpose MLLMs. Note that descriptions of all
these benchmarks are included in the Appendix.

We also compare our model with several current text-centric MLLMs. The
results, presented in Table 4, use the same metrics as those employed in the
original benchmarks. Detailed descriptions of the specific metrics used for
each benchmark can be found in the Appendix. Our method demonstrates
advantages over these text-centric MLLMs, achieving leading performance across
multiple benchmarks. Notably, our visual encoder is more lightweight than
those used by the compared methods.

Table 3: Quantitative accuracy (%) comparison of DocKylin with existing general-purpose MLLMs on document image benchmarks. Accuracy proposed in [78] is adopted as the metric for all benchmarks. Methods | Visual Encoder (Params.) |  DocVQAval |  InfoVQAval | ChartQA | FUNSD | SROIE | POIE
---|---|---|---|---|---|---|---
mPLUG-Owl [75] | CLIP-L (0.3B) | 7.4 | 20.0 | 7.9 | 0.2 | 0.1 | 0.3
InstructBLIP [11] | CLIP-G (2B) | 4.5 | 16.4 | 5.3 | 0.2 | 0.6 | 1.0
LLaVA1.5 [35] | CLIP-L (0.3B) | 8.5 | 14.7 | 9.3 | 0.2 | 1.7 | 2.5
Qwen-VL [3] | CLIP-G (2B) | 48.1 | 23.9 | 53.4 | 23.9 | 34.5 | 20.6
Monkey [29] | CLIP-G (2B) | 50.1 | 25.8 | 54.0 | 24.1 | 41.9 | 19.9
InternVL [9] | IViT (6B) | 28.7 | 23.6 | 45.6 | 6.5 | 26.4 | 25.9
InterLM-XComposer2 [12] | CLIP-L (0.3B) | 39.7 | 28.6 | 51.6 | 15.3 | 34.2 | 49.3
DocKylin (ours) | Swin (0.07B) | 65.1 | 34.8 | 58.7 | 25.5 | 49.5 | 36.1
Table 4: Quantitative comparison of DocKylin with existing text-centric MLLMs on document-oriented benchmarks. The metrics used for each benchmark are consistent with those proposed in the original papers. Methods | Visual Encoder (Params.) |  DocVQAtest |  InfoVQAtest | ChartQA | VisualMRC | DeepForm | WTQ
---|---|---|---|---|---|---|---
Donut [22] | Swin (0.07B) | 67.5 | 11.6 | 41.8 | 93.9 | 61.6 | 18.8
Pix2Struct[26] | - | 76.6 | 40.0 | 58.6 | - | - | -
mPLUG-Doc [75] | CLIP-L (0.3B) | 62.2 | 38.2 | 57.4 | 188.8 | 42.6 | 26.9
UReader [74] | CLIP-L (0.3B) | 65.4 | 42.2 | 65.7 | 221.7 | 49.5 | 29.4
DocPeida [13] | Swin (0.1B) | 47.1 | 15.2 | 46.9 | - | - | -
HRVDA [72] | Swin | 72.1 | 43.5 | 67.6 | 211.5 | 63.2 | 31.2
Vary [70] |  ViTDet-B & CLIP-L (0.4B) | 76.3 | - | 66.1 | - | - | -
TextMonkey [37] | CLIP-G (2B) | 71.5 | - | 65.5 | - | 61.6 | 30.6
TextHawk [76] | SigLIP-SO (0.4B) | 76.4 | 50.6 | 66.6 | - | - | 34.7
LayoutLLM-LLaMA2 [17] | LayoutLMv3-L (0.3B) | 74.3 | - | - | - | - | -
DocKylin (ours) | Swin (0.07B) | 77.3 | 46.6 | 66.8 | 319.9 | 64.2 | 32.4
Figure 4: Visualization from DocKylin. The redundant pixels and nonessential
tokens identified by Adaptive Pixel Slimming and Dynamic Token Slimming are
highlighted in light blue.

Some visual results from DocKylin on these benchmarks are presented in Fig. 4.
The first row shows the results of redundant regions detected by APS, with
corresponding text explaining the resulting reduction in resolution. The
second row displays the nonessential tokens removed by DTS, with accompanying
text illustrating its effect on compressing the visual sequence. These results
demonstrate their effectiveness in accurately retaining important regions
while significantly reducing the resolution and the length of the visual
sequence.

Table 5: Comparisions with current SOTA parameter-free visual token compression methods. Accuracy proposed in [78] is adopted as the metric for all benchmarks. Model | Compression | Adaptive |  DocVQAval |  InfoVQAval | SROIE
---|---|---|---|---|---
Avg. len |  Acc $\uparrow$ | Avg. len |  Acc $\uparrow$ | Avg. len |  Acc $\uparrow$
LLaVA1.5 [35] | - | - | 576 | 8.5 | 576 | 14.7 | 576 | 1.7
Random | ✗ | 300 | 5.4 | 300 | 14.3 | 300 | 0.5
Random | ✗ | 200 | 4.2 | 200 | 14.2 | 200 | 0
FastV (K=2, R=50%) [8] | ✗ | 306 | 8.4 | 306 | 14.4 | 306 | 2.0
PruMerge [49] | ✓ | 123 | 5.3 | 123 | 13.8 | 121 | 0.4
PruMerge+ [49] | ✓ | 232 | 5.2 | 233 | 14.0 | 229 | 0.3
APS+DTS (ours) | ✓ | 298 | 9.7 | 270 | 14.7 | 296 | 3.1

To further demonstrate the superiority of our method, we compared it with
other parameter-free visual token compression methods, using LLaVA1.5 [35] as
our baseline model (without retraining). The results in Table 5 indicate that
our APS and DTS achieve performance improvements with shorter sequence lengths
compared to the baseline model, significantly outperforming existing SOTA
compression methods. It is worth noting that these results are achieved
without retraining. Typically, after retraining, models can better adapt to
the new mode of compressed token sequences, leading to further performance
improvements [49, 8].

###  4.3 Ablation Studies and Potential Broader Applications

We validate the effectiveness of APS and DTS on DocKylin and give the results
in Table 6. Note that, settings without DTS in Table 6 refer to retraining a
new model from scratch without employing DTS in the instruction tuning phase,
rather than simply removing DTS from the already trained DocKylin model. The
results demonstrate that both APS and DTS effectively reduce token sequence
length and enhance computational efficiency, while also improving performance.
This aligns with findings from previous studies [37], which suggest that
excessively long sequences due to high resolution may degrade performance.
This degradation likely occurs because the visual sequences contain too many
redundant tokens, placing greater demands on the model’s ability to handle
long contexts and making the learning more challenging. Notably, the APS and
DTS methods can also be used online during the training phase to accelerate
training. Specifically, applying the DTS method during the instruction tuning
phase can reduce training time by approximately 15%.

Table 6: Ablation studies on DocKylin regarding Adaptive Pixel Slimming and Dynamic Token Slimming. The latency and training time are normalized to the results in the first row. Accuracy proposed in [78] is adopted as the metric for all benchmarks. APS | DTS | Avg. Token Len | Training Time |  DocVQAval
---|---|---|---|---
Acc. | Latency
✗ | ✗ | 2770 | 1 | 63.1 | 1
✓ | ✗ | 2198 | 0.97 | 64.5 | 0.81
✓ | ✓ | 1250 | 0.86 | 65.1 | 0.61
Table 7: The enhancement brought to other MLLMs by Adaptive Pixel Slimming. Relative improvements are highlighted in bold. Accuracy proposed in [78] is adopted as the metric for all benchmarks. Methods | Resolution | TextVQA |  DocVQAval |  InfoVQAval | SROIE | FUNSD
---|---|---|---|---|---|---
LLaVA1.5 [35] |  336$\times$336 | 38.7 | 8.5 | 14.7 | 1.7 | 0.2
LLaVA1.5 [35] \+ APS |  336$\times$336 |  40.2 (+4.1%) |  10.7 (+27.4%) |  14.7 (+0%) |  3.7 (+118%) |  0.92 (+360%)
Qwen-VL [3] |  448$\times$448 | 62.1 | 48.1 | 23.9 | 34.5 | 20.6
Qwen-VL [3] \+ APS |  448$\times$448 |  62.4 (+0.4%) |  51.2 (+6.4%) |  24.7 (+4.1%) |  40.0 (+15.9%) |  24.3 (+17.9%)
Monkey [29] |  896$\times$896 | 64.3 | 50.1 | 25.8 | 41.9 | 24.1
Monkey [29] \+ APS |  896$\times$896 |  65.2 (+1.4%) |  56.3 (+12.4%) |  27.5 (+6.6%) |  47.0 (+12.2%) |  27.3 (+13.3%)
Figure 5: Effects of applying DTS to natural scene images.

Since APS is a parameter-free method that requires no additional training or
modification of model architecture, it is easily applicable to other MLLMs to
enhance their performance on high-resolution document images. To demonstrate
its effectiveness, we apply APS to various existing MLLM models (employing it
before the image processing module of these models). As shown in Table 7 of
our experimental results, APS consistently improves document image tasks
across a range of MLLMs with different resolutions. Additionally, results on
TextVQA demonstrate that APS does not lead to performance degradation in tasks
involving scene images.

The redundant regions in document images are often simple, such as plain
backgrounds, distinguishing these regions is relatively easy. This raises the
question of whether DTS’s effectiveness is due to this simplicity. Although
this paper primarily explores VDU tasks, we are also interested in the
generalizability of DTS. Therefore, we conducted some experiments on natural
scene images. Specifically, we retrained a scene text image full-text
recognition model using some open-source scene text data [10, 53, 50, 68],
performing only the first pre-training phase. We then applied DTS to the
trained model and tested it on the TextVQA dataset. Some visual results are
shown in Fig. 5. It can be seen that DTS is effective in distinguishing
nonessential regions even in the complex backgrounds of natural scene images.

##  5 Limitations

While DocKylin exhibits outstanding performance in visual document
understanding (VDU) tasks, it currently lacks versatility for more general
tasks, such as question answering and captioning in natural scenes. This
limitation primarily stems from our current image encoder, which has not
undergone extensive image-text pair pre-training and also possesses
insufficient parameter capacity. Future work could explore scaling up the Swin
Transformer and conducting pre-training similar to that of the current CLIP
ViT. Alternatively, a dual-encoder approach [32, 70] could be employed, where
a flexible, lightweight Swin Transformer is specifically used for extracting
fine textual details within images, while a larger-capacity ViT model is
utilized for capturing more general content. The combined image features from
both encoders could then be integrated into the LLM.

##  6 Conclusions

To address the issues of high input resolution and excessively long token
sequences in current MLLMs when counters with VDU tasks, we propose the
DocKylin model. It includes a parameter-free Adaptive Pixel Slimming (APS)
preprocessing module that removes redundant regions from document images,
effectively reducing their resolution. The model employs a flexible image
encoding approach to adapt to the varying resolutions and aspect ratios of
document image scenarios. Moreover, we propose a parameter-free token
compression method, Dynamic Token Slimming (DTS), which clusters and selects
essential tokens from a long visual sequence and employs aggregation to avoid
potential information loss.

Experiments demonstrate that DocKylin achieves leading performance across
multiple VDU benchmarks. APS and DTS effectively shorten the length of visual
tokens, enhancing both computational efficiency and model performance.
Additionally, we have validated the potential of applying these methods to
other MLLMs or other scenarios beyond document images.

## References

  * [1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  Advances in neural information processing systems, 35:23716–23736, 2022.
  * [2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.  Qwen technical report.  arXiv preprint arXiv:2309.16609, 2023.
  * [3] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.  Qwen-vl: A frontier large vision-language model with versatile abilities.  arXiv preprint arXiv:2308.12966, 2023.
  * [4] Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Minesh Mathew, CV Jawahar, Ernest Valveny, and Dimosthenis Karatzas.  Icdar 2019 competition on scene text visual question answering.  In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1563–1570. IEEE, 2019.
  * [5] Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee.  Matryoshka multimodal models.  arXiv preprint arXiv:2405.17430, 2024.
  * [6] Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh.  Honeybee: Locality-enhanced projector for multimodal llm.  arXiv preprint arXiv:2312.06742, 2023.
  * [7] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny.  Minigpt-v2: large language model as a unified interface for vision-language multi-task learning.  arXiv preprint arXiv:2310.09478, 2023.
  * [8] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang.  An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models.  arXiv preprint arXiv:2403.06764, 2024.
  * [9] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al.  Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.  arXiv preprint arXiv:2312.14238, 2023.
  * [10] Chee Kheng Ch’ng and Chee Seng Chan.  Total-text: A comprehensive dataset for scene text detection and recognition.  In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 1, pages 935–942. IEEE, 2017.
  * [11] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  Advances in Neural Information Processing Systems, 36, 2024.
  * [12] Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al.  Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd.  arXiv preprint arXiv:2404.06512, 2024.
  * [13] Hao Feng, Qi Liu, Hao Liu, Wengang Zhou, Houqiang Li, and Can Huang.  Docpedia: Unleashing the power of large multimodal model in the frequency domain for versatile document understanding.  arXiv preprint arXiv:2311.11810, 2023.
  * [14] Hao Feng, Zijian Wang, Jingqun Tang, Jinghui Lu, Wengang Zhou, Houqiang Li, and Can Huang.  Unidoc: A universal large multimodal model for simultaneous text detection, recognition, spotting and understanding.  arXiv preprint arXiv:2308.11592, 2023.
  * [15] Adam W Harley, Alex Ufkes, and Konstantinos G Derpanis.  Evaluation of deep convolutional nets for document image classification and retrieval.  In 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pages 991–995. IEEE, 2015.
  * [16] Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al.  mplug-docowl 1.5: Unified structure learning for ocr-free document understanding.  arXiv preprint arXiv:2403.12895, 2024.
  * [17] Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei.  Layoutlmv3: Pre-training for document ai with unified text and image masking.  In Proceedings of the 30th ACM International Conference on Multimedia, pages 4083–4091, 2022.
  * [18] Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and CV Jawahar.  Icdar2019 competition on scanned receipt ocr and information extraction.  In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1516–1520. IEEE, 2019.
  * [19] Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran.  Funsd: A dataset for form understanding in noisy scanned documents.  In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 2, pages 1–6. IEEE, 2019.
  * [20] Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan.  Dvqa: Understanding data visualizations via question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648–5656, 2018.
  * [21] Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio.  Figureqa: An annotated figure dataset for visual reasoning.  arXiv preprint arXiv:1710.07300, 2017.
  * [22] Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park.  Ocr-free document understanding transformer.  In European Conference on Computer Vision, pages 498–517. Springer, 2022.
  * [23] Geewook Kim, Hodong Lee, Daehee Kim, Haeji Jung, Sanghee Park, Yoonsik Kim, Sangdoo Yun, Taeho Kil, Bado Lee, and Seunghyun Park.  Visually-situated natural language understanding with contrastive reading model and frozen large language models.  In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
  * [24] Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, et al.  Spvit: Enabling faster vision transformers via latency-aware soft token pruning.  In European conference on computer vision, pages 620–640. Springer, 2022.
  * [25] Jianfeng Kuang, Wei Hua, Dingkang Liang, Mingkun Yang, Deqiang Jiang, Bo Ren, and Xiang Bai.  Visual information extraction in the wild: practical dataset and end-to-end solution.  In International Conference on Document Analysis and Recognition, pages 36–53. Springer, 2023.
  * [26] Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova.  Pix2struct: Screenshot parsing as pretraining for visual language understanding.  In International Conference on Machine Learning, pages 18893–18912. PMLR, 2023.
  * [27] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  In International conference on machine learning, pages 19730–19742. PMLR, 2023.
  * [28] Xin Li, Yunfei Wu, Xinghua Jiang, Zhihao Guo, Mingming Gong, Haoyu Cao, Yinsong Liu, Deqiang Jiang, and Xing Sun.  Enhancing visual document understanding with contrastive learning in large visual-language models.  arXiv preprint arXiv:2402.19014, 2024.
  * [29] Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai.  Monkey: Image resolution and text label are important things for large multi-modal models.  arXiv preprint arXiv:2311.06607, 2023.
  * [30] Youwei Liang, GE Chongjian, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie.  Evit: Expediting vision transformers via token reorganizations.  In International Conference on Learning Representations, 2021.
  * [31] Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji.  Boosting multimodal large language models with visual tokens withdrawal for rapid inference.  arXiv preprint arXiv:2405.05803, 2024.
  * [32] Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al.  Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models.  arXiv preprint arXiv:2311.07575, 2023.
  * [33] Cheng-Lin Liu, Fei Yin, Da-Han Wang, and Qiu-Feng Wang.  Casia online and offline chinese handwriting databases.  In 2011 international conference on document analysis and recognition, pages 37–41. IEEE, 2011.
  * [34] Haogeng Liu, Quanzeng You, Xiaotian Han, Yongfei Liu, Huaibo Huang, Ran He, and Hongxia Yang.  Visual anchors are strong information aggregators for multimodal large language model.  arXiv preprint arXiv:2405.17815, 2024.
  * [35] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning.  arXiv preprint arXiv:2310.03744, 2023.
  * [36] Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, et al.  On the hidden mystery of ocr in large multimodal models.  arXiv preprint arXiv:2305.07895, 2023.
  * [37] Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai.  Textmonkey: An ocr-free large multimodal model for understanding document.  arXiv preprint arXiv:2403.04473, 2024.
  * [38] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021.
  * [39] Stuart Lloyd.  Least squares quantization in pcm.  IEEE transactions on information theory, 28(2):129–137, 1982.
  * [40] Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque.  Chartqa: A benchmark for question answering about charts with visual and logical reasoning.  In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, 2022.
  * [41] Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar.  Infographicvqa.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697–1706, 2022.
  * [42] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar.  Docvqa: A dataset for vqa on document images.  In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021.
  * [43] Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar.  Plotqa: Reasoning over scientific plots.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1527–1536, 2020.
  * [44] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty.  Ocr-vqa: Visual question answering by reading text in images.  In 2019 international conference on document analysis and recognition (ICDAR), pages 947–952. IEEE, 2019.
  * [45] OpenAI.  Chatgpt: Get answers. find inspiration. be more productive., 2022.  Accessed: 2024-05-10.
  * [46] Panupong Pasupat and Percy Liang.  Compositional semantic parsing on semi-structured tables.  arXiv preprint arXiv:1508.00305, 2015.
  * [47] Panupong Pasupat and Percy Liang.  Compositional semantic parsing on semi-structured tables.  arXiv preprint arXiv:1508.00305, 2015.
  * [48] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [49] Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan.  Llava-prumerge: Adaptive token reduction for efficient large multimodal models.  arXiv preprint arXiv:2403.15388, 2024.
  * [50] Baoguang Shi, Cong Yao, Minghui Liao, Mingkun Yang, Pei Xu, Linyan Cui, Serge Belongie, Shijian Lu, and Xiang Bai.  Icdar2017 competition on reading chinese text in the wild (rctw-17).  In 2017 14th iapr international conference on document analysis and recognition (ICDAR), volume 1, pages 1429–1434. IEEE, 2017.
  * [51] Yongxin Shi, Dezhi Peng, Wenhui Liao, Zening Lin, Xinhong Chen, Chongyu Liu, Yuyi Zhang, and Lianwen Jin.  Exploring ocr capabilities of gpt-4v (ision): A quantitative and in-depth evaluation.  arXiv preprint arXiv:2310.16809, 2023.
  * [52] Štěpán Šimsa, Milan Šulc, Michal Uřičář, Yash Patel, Ahmed Hamdi, Matěj Kocián, Matyáš Skalickỳ, Jiří Matas, Antoine Doucet, Mickaël Coustaty, et al.  Docile benchmark for document information localization and extraction.  In International Conference on Document Analysis and Recognition, pages 147–166. Springer, 2023.
  * [53] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach.  Towards vqa models that can read.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019.
  * [54] Tomasz Stanisławek, Filip Graliński, Anna Wróblewska, Dawid Lipiński, Agnieszka Kaliska, Paulina Rosalska, Bartosz Topolski, and Przemysław Biecek.  Kleister: key information extraction datasets involving long documents with complex layouts.  In International Conference on Document Analysis and Recognition, pages 564–579. Springer, 2021.
  * [55] AI Studio.  Insurance text visual qa, 2022.
  * [56] Hongbin Sun, Zhanghui Kuang, Xiaoyu Yue, Chenhao Lin, and Wayne Zhang.  Spatial dual-modality graph reasoning for key information extraction.  arXiv preprint arXiv:2103.14470, 2021.
  * [57] S Svetlichnaya.  Deepform: Understand structured documents at scale, 2020.
  * [58] Ryota Tanaka, Taichi Iki, Kyosuke Nishida, Kuniko Saito, and Jun Suzuki.  Instructdoc: A dataset for zero-shot generalization of visual document understanding with instructions.  arXiv preprint arXiv:2401.13313, 2024.
  * [59] Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida.  Visualmrc: Machine reading comprehension on document images.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13878–13888, 2021.
  * [60] Jingqun Tang, Chunhui Lin, Zhen Zhao, Shu Wei, Binghong Wu, Qi Liu, Hao Feng, Yang Li, Siqi Wang, Lei Liao, et al.  Textsquare: Scaling up text-centric visual instruction tuning.  arXiv preprint arXiv:2404.12803, 2024.
  * [61] Jingqun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, Chunhui Lin, Wanqing Li, Mohamad Fitri Faiz Bin Mahmood, Hao Feng, Zhen Zhao, et al.  Mtvqa: Benchmarking multilingual text-centric visual question answering.  arXiv preprint arXiv:2405.11985, 2024.
  * [62] InternLM Team.  Internlm: A multilingual language model with progressively enhanced capabilities, 2023.
  * [63] The Vicuna Team.  Vicuna: An open-source chatbot impressing gpt-4 with 90  Accessed: 2024-05-10.
  * [64] Tianchi.  Ocr text reading order sorting dataset, 2020.
  * [65] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [66] Michał Turski, Tomasz Stanisławek, Karol Kaczmarek, Paweł Dyda, and Filip Graliński.  Ccpdf: Building a high quality corpus for visually rich documents from web crawl data.  In International Conference on Document Analysis and Recognition, pages 348–365. Springer, 2023.
  * [67] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015.
  * [68] Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie.  Coco-text: Dataset and benchmark for text detection and recognition in natural images.  arXiv preprint arXiv:1601.07140, 2016.
  * [69] Yonghui Wang, Wengang Zhou, Hao Feng, Keyi Zhou, and Houqiang Li.  Towards improving document understanding: An exploration on text-grounding via mllms.  arXiv preprint arXiv:2311.13194, 2023.
  * [70] Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang.  Vary: Scaling up the vision vocabulary for large vision-language models.  arXiv preprint arXiv:2312.06109, 2023.
  * [71] Siyuan Wei, Tianzhu Ye, Shen Zhang, Yao Tang, and Jiajun Liang.  Joint token pruning and squeezing towards more aggressive compression of vision transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2092–2101, 2023.
  * [72] Siyuan Wei, Tianzhu Ye, Shen Zhang, Yao Tang, and Jiajun Liang.  Hrvda: High-resolution visual document assistant.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
  * [73] Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al.  Baichuan 2: Open large-scale language models.  arXiv preprint arXiv:2309.10305, 2023.
  * [74] Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al.  Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model.  arXiv preprint arXiv:2310.05126, 2023.
  * [75] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al.  mplug-owl: Modularization empowers large language models with multimodality.  arXiv preprint arXiv:2304.14178, 2023.
  * [76] Ya-Qi Yu, Minghui Liao, Jihao Wu, Yongxin Liao, Xiaoyu Zheng, and Wei Zeng.  Texthawk: Exploring efficient fine-grained perception of multimodal large language models.  arXiv preprint arXiv:2404.09204, 2024.
  * [77] Hesuo Zhang, Lingyu Liang, and Lianwen Jin.  Scut-hccdoc: A new benchmark dataset of handwritten chinese text in unconstrained camera-captured documents.  Pattern Recognition, 108:107559, 2020.
  * [78] Shuo Zhang, Biao Yang, Zhang Li, Zhiyin Ma, Yuliang Liu, and Xiang Bai.  Exploring the capabilities of large multimodal models on dense text.  arXiv preprint arXiv:2405.06706, 2024.
  * [79] Xu Zhong, Elaheh ShafieiBavani, and Antonio Jimeno Yepes.  Image-based table recognition: data, model, and evaluation.  In European conference on computer vision, pages 564–580. Springer, 2020.
  * [80] Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua.  Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance.  arXiv preprint arXiv:2105.07624, 2021.

## Appendix

##  Appendix A Benchmark Datasets Evaluation Metrics

DocVQA [42], InfoVQA [41], ChartQA [40], WTQ [47], and VisualMRC [59] are all
datasets designed for visual document image question answering. DocVQA
encompasses a variety of document types, featuring a rich diversity of
questions and data types. InfoVQA focuses on infographics that contain an
abundance of visual elements. ChartQA and WTQ specifically target question
answering for chart data and tabular data, respectively. VisualMRC, on the
other hand, places a greater emphasis on developing natural language
understanding and generation capabilities. DeepForm [57], FUNSD [19], SROIE
[18], and POIE [25] are key information extraction datasets where the model is
required to identify the value corresponding to a given key.

Widely used text-centric benchmarks employ diverse metrics to better capture
the distinct characteristics among various datasets. As depicted in Table 4
within the main text, both DocVQA [42] and InfoVQA [41] utilize ANLS [4] as
their metric. The results reported are evaluated on their respective official
websites. ChartQA [40] is evaluated by accuracy relaxed accuracy [43], while
VisualMRC [59] relies on CIDEr [67]. DeepForm [57] is measured by F1 score,
and WTQ [46] evaluated by accuracy.

##  Appendix B More Details and Results of Adaptive Pixel Slimming

Algorithm 2 Adaptive Pixel Slimming

1:

2:Source document image $I_{S}\in\mathbb{R}^{H\times W\times 3}$

3:SIZE=2048

4:$t_{n}=50$

5:$t_{c}=10$

6:ICR(Identify continuous regions where the values exceed $t_{v}$ and the
count of consecutive occurrences is greater than $t_{c}$.)

7:Sobelx = abs(Sobel($I_{S}$, direction=$x$))

8:Sobely = abs(Sobel($I_{S}$, direction=$y$))

9:Initialize $I_{G}\in\mathbb{R}^{H\times W}$

10:$I_{G}(i,j)$ = max(Sobel${}_{x}(i,j)$, Sobel${}_{y}(i,j)$)

11:$I_{G}[I_{G}<t_{n}]=0$

12:$I_{G}$ = resize($I_{G}$, (SIZE, SIZE))

13:sumy = sum($I_{G}(i,j)$, along $y$ direction) $\in\mathbb{R}^{SIZE}$

14:blocksy = ICR(sumy,$t_{v}$,$t_{c}$)

15:blocksy = normalize_to_original_size(blocksy, (H, W), SIZE)

16:sumx = sum($I_{G}(i,j)$, along $x$ direction) $\in\mathbb{R}^{SIZE}$

17:blocksx = ICR(sumx,$t_{v}$,$t_{c}$)

18:blocksx = normalize_to_original_size(blocksx, (H, W), SIZE)

19:$I_{t}$ = remove_blocks(blocksy, blocksx, $I_{S}$)

20:return $I_{t}$

The specific computation method and parameter settings for Adaptive Pixel
Slimming are detailed in Algorithm 2, which provides additional details beyond
the main text: 1. The 2D gradient map is obtained by taking the maximum value
from the gradient maps in the x and y directions. This approach aims to ensure
the accuracy of redundancy detection while minimizing false positives. Using
the maximum value rather than the average prevents the removal of regions with
features in a single direction, such as table lines. 2. Noise removal is
performed by setting gradient values below a threshold $t_{n}$ to zero,
thereby reducing interference from background texture noise. 3. Before
identifying redundant regions based on the value threshold $t_{v}$, we
normalize the gradient map to a fixed resolution. This normalization ensures
that the threshold is robust to varying resolutions.

In Fig. 6, we present some results of Adaptive Pixel Slimming, demonstrating
its effectiveness in removing redundant regions from document images and
reducing image resolution. The impact on natural scene images is minimal.

Figure 6: Additional visual results of Adaptive Pixel Slimming on various
types of data. The redundant regions identified by Adaptive Pixel Slimming are
highlighted on the corresponding gradient maps. The resolution for some images
is indicated below the respective images.

##  Appendix C More Details of DTS

Algorithm 3 Similarity Weighted Aggregation

1:

2:essential_tokens

3:nonessential_tokens

4:FMS (find the most similar token)

5:mappings = {}

6:for token in nonessential_tokens:

7: most_similar_essential_token = FMS(token, tokens in essential_tokens )

8: mappings[token] = most_similar_essential_token

9:updated_essential_tokens = []

10:for token in essential_tokens:

11: similar_nonessential_tokens = []

12: for key,value in mappings:

13: if token == value:

14: similar_nonessential_tokens.append(key)

15: if len(similar_nonessential_tokens)==0:

16: updated_essential_tokens.append(token)

17: else:

18: updated_essential_tokens.append(weighted_sum(token,
similar_nonessential_tokens))

19:return updated_essential_tokens

##  Appendix D Declaration of Potential Harmful Content

While our model performs well on many tasks, we acknowledge that, like all
MLLMs, due to the breadth of its training data and the diversity of its
content, our model may occasionally generate inaccurate, misleading, or
harmful content.

We hereby declare that, despite our efforts to mitigate this risk by selecting
appropriate training data and adjusting model parameters, we cannot entirely
eliminate the possibility of the MLLM producing harmful content. Therefore,
when referring to this paper or using the model proposed herein, please be
aware that the generated text may contain uncertainties and risks. We
encourage readers to exercise caution and independently assess the accuracy
and credibility of the content generated by the MLLM.


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

IJCAI 2023 Workshop on Deepfake Audio Detection and Analysis (DADA 2023),
August 19, 2023, Macao, S.A.R

[<EMAIL_ADDRESS>]

[ ]

[ ]

[ ]

[ ]

# The defender’s perspective on automatic speaker verification: An overview

Haibin Wu  Jiawen Kang  Lingwei Meng  Helen Meng  Hung-yi Lee

(2022)

###### Abstract

Automatic speaker verification (ASV) plays a critical role in security-
sensitive environments. Regrettably, the reliability of ASV has been
undermined by the emergence of spoofing attacks, such as replay and synthetic
speech, as well as adversarial attacks and the relatively new partially fake
speech. While there are several review papers that cover replay and synthetic
speech, and adversarial attacks, there is a notable gap in a comprehensive
review that addresses defense against adversarial attacks and the recently
emerged partially fake speech. Thus, the aim of this paper is to provide a
thorough and systematic overview of the defense methods used against these
types of attacks.

###### keywords:

Automatic speaker verification Replay and synthetic speech Adversarial attack
Partially fake speech Review

##  1 Introduction

The past few years have witnessed significant advances in ASV, and this
technique is now widely integrated into daily life, including voice activation
in smartphones and e-banking authentication. However, ASV is serious
vulnerable to malicious spoofing attacks includes tactics such as replay and
synthetic speech, adversarial attacks and recently emerged partially fake
speech.

While there are several review papers that cover replay and synthetic speech
[1, 2, 3, 4], and adversarial attacks [5], there is a notable gap in a
comprehensive review that addresses defense methods against adversarial
attacks and the recently emerged partially fake speech. The objective of this
thesis is to provide a thorough and systematic overview of the defense methods
used against these two types of attacks. It is hoped that they will inspire
further researches within the ASV community.

##  2 Attacks

###  2.1 Partially fake speech

The first Audio Deep Synthesis Detection challenge (ADD 2022) [6] releases a
kind of brand new attack, known as the partially fake speech attack [7]. The
ASVspoof challenge [1, 2, 3, 4] focuses on generating spoofing speech in its
entirety, ignoring the scenario of partially fake speech, where small fake
clips are hidden within a piece of real speech. The generation of partially
fake audio involves the insertion of only small clips of synthetic speech into
the real speech as shown in Figure 1, resulting in even more stealthy fake
speech containing a significant amount of the genuine user’s audio.

Figure 1: The partially fake audio generation process. A small clip is
selected from the user’s utterance, the content is recognized using Automatic
Speech Recognition (ASR), and the recognized content is modified to manipulate
the meaning of the entire speech. The fake clip is then generated using Text-
to-Speech (TTS) or Voice Conversion (VC), and inserted into the genuine
utterance to generate the partially fake speech.

Previous studies [7, 8] have shown that it is challenging to differentiate
between partially fake and genuine audios by directly using existing state-of-
the-art countermeasure models fostered by the ASVspoof challenge [1, 2, 3, 4].
These countermeasure models address the problem of identifying whether an
entire audio utterance is genuine or fabricated. However, they are not
equipped to identify anomalous regions within a single utterance.

###  2.2 Adversarial attacks

Figure 2: A tiny adversarial noise is added to the original wave to get the
adversarial one to fool the ASV falsely accept.

Speaker verification models are also subject to adversarial attacks [9, 5, 10]
as shown in Figure 2. Kreuk et al. [11] are among the pioneers in studying the
susceptibility of ASV models to adversarial attacks. Additionally, even the
current state-of-the-art ASV models, including i-vector [12] and x-vector [13]
systems, are not immune to adversarial attacks. [14] conducts a pioneering
effort in exposing the adversarial weakness of countermeasure models and [15]
further enhances the transferability of adversarial attacks through model
ensemble.

##  3 Defense methods

###  3.1 Tackle partially fake speech attacks

Figure 3: The two categories of methods to tackle partially fake speech
attacks. The black and red parts of the utterance are real and fake,
respectively. The first approach, illustrated in the blue block, focuses on
detecting the transition boundaries between the genuine and fake segments. The
second method, depicted in the orange block, endeavors to distinguish between
genuine and fake short segments.

Partially fake speech attacks are generated as shown in Figure 1. As this kind
of attack is brand new, there have been only a few initiatives to handle this
attack, and we categorize these efforts into two categories as shown in Figure
3: transition boundary detection [16, 17, 18] and segment level classification
[7, 8, 19, 20].

####  3.1.1 SSL-based feature extractor

Before delving into the two main approaches, let’s first examine the feature
engineering aspect of the task. Lv et al. [21] are the pioneers in utilizing
self-supervised learning (SSL) models to tackle partially fake speech attacks.
Rather than using traditional acoustic features, they instead adopt XLS-R
[22], a self-supervised learning model, as the feature extractor. Their method
[21], which involved simply adding a lightweight prediction head on top of the
XLS-R model and fine-tuning the large XLS-R model, ultimately achieved first
place out of 33 international teams in the ADD challenge [6].

Their efforts [21] have taught us a valuable lesson - the acoustic features
extracted by a fine-tuned self-supervised learning model can be incredibly
helpful for detecting partially fake speech. It’s worth noting that the two
main approaches introduced below can also harness the power of self-supervised
learning models, provided there are sufficient computing resources available.

####  3.1.2 Transition boundary detection

[16] is the first to introduce the transition boundary detection task for
partially fake audio detection. The transition boundaries contain artifacts,
such as discontinuity in speech and inconsistencies in ambient noise. Inspired
by the extraction-based question-answering models [23] used in natural
language processing (NLP), we refer to the boundary detection task as a
question-answering or fake span discovery proxy task. In this task, the model
is required to answer the question “where is the fake clip?" in a piece of
partially fake audio. Extraction-based question-answering models in NLP
typically take a question and a passage as input, construct representations
for the passage and the question, match the question and passage embeddings,
and then output the start and end positions of the answer within the passage.
In our case, the passage is the partially fake utterance, and the answer is
the start and end time of the fake clip. As depicted in the blue block of
Figure 3, when the model is presented with a boundary frame between a real
(black) and a fake (red) clip, it should predict “1". Conversely, when the
model is presented with a non-boundary frame, it should predict “0". By
training the model on the question-answering proxy task, the model can learn
to find the concatenation boundaries with discontinuity and identify fake
clips within an utterance, thus improving its ability to distinguish between
audios with and without fake clips. The proposed method placed the second out
of 33 international teams in the ADD challenge [6], even without the
assistance of self-supervised learning features.

Wang et al. [18] divide the entire utterance into several chunks, and
extracted acoustic features from each chunk to feed into the deep learning
model. The model is then tasked with determining whether a boundary exists
within the given chunk by predicting “1" if the chunk contains a boundary, or
a “0" if it does not. Through training, the model gains the ability to
identify clues such as speech discontinuity or inconsistencies in ambient
noise, allowing it to effectively highlight potential boundaries.

Cai et al. [17] propose to introduce the self-supervised learning model for
frame-level boundary detection to detect partially fake speech. They modify
the method in [16] to further boost the detection performance: 1). Instead of
solely focusing on transition boundaries that indicate inconsistency and
discontinuity, [17] proposes setting nearby frames of the boundaries as
boundaries to increase robustness. 2). [17] employs wav2vec 2.0 [24], a self-
supervised learning model as feature extractor and also fine-tunes the feature
extractor during training. Utilizing the features from wav2vec 2.0 improves
the performance by a relative 58.25% compared to traditional acoustic features
extracted by digital signal processing front-ends.

The main takeaway message from this subsection is that the transition
boundaries can serve as a useful cue to identify partially fake audio, as it
indicates discontinuity and inconsistency in speech. By tasking models with
detecting these boundaries, they can learn to identify these cues and detect
partially fake speech.

####  3.1.3 Segment level classification

The goal of segment level classification is to distinguish between genuine and
fake segments. The short segments have different time resolutions, ranging
from 1 frame (around 20 ms) to the entire utterance. Segments that only
contain genuine speech will be labeled as “1", while all other segments will
be labeled as “0" as shown in the orange block of Figure 3. Zhang et al. [8]
do the initial attempt to conduct segment level classification for partially
fake speech detection with a fixed time resolution. In their subsequent works
[19], they propose to train the countermeasure model by both the utterance
level classification and segment level classification. To further boost the
countermeasure’s performance, they [20] introduce the self-supervised learning
models [25, 24] as the front-end feature extractor, and enable the model to
learn segment level classification with different time resolutions, ranging
from 1 frame to the entire utterance.

The time resolution used in segment level classification is a crucial
hyperparameter for training. If the segment’s frame number is too small, the
model may not extract enough information to distinguish between genuine and
fake segments. On the other hand, if the frame number is too large, the
proportion of fake frames may be too small, resulting in fake frames being
dominated by genuine frames. Enabling the model learn from different time
resolutions [20] is a reasonable solution to bypass the hyperparameter search.
Note that in Figure 1, the inserted red clip can be from other genuine users.
The segment level classification [8, 19, 20] does not consider this condition
into account as in their produced dataset, the inserted clips are always fake.

Figure 4: The timeline of defense methods for ASV against adversarial attacks.

###  3.2 Defense against adversarial attacks

We propose to classify the defense methods into three categories and the
timeline for related works is shown in Figure 4. 1). Model enhancement focuses
on developing robust models during the training phase by modifying the models’
internals, making the attackers difficult to find effective adversarial
examples. 2). Adversarial sample purification aims to alleviate the
superficial adversarial noise and transform adversarial samples into genuine
samples. 3). Adversarial sample detection aims to distinguish between
adversarial and genuine samples, allowing the identification and removal of
adversarial samples.

####  3.2.1 Model enhancement

[26, 27, 28] adopt adversarial training to alleviate the vulnerability of ASV
against adversarial attacks. Wu et al. [29] also investigate improving the
adversarial robustness for countermeasures by adversarial training.

Model enhancement methods involve modifying the model’s parameters, and they
can usually work together with purification and detection methods.

####  3.2.2 Adversarial sample purification

Previous efforts for purification can be classified into 5 categories: Lossy
pre-processing, adding noise, generative method, denoising method and
filtering.

The “Lossy pre-processing" approach treats adversarial perturbations as
redundant information and discards it to improve the model’s adversarial
robustness. Chen et al. [30] consider adversarial perturbations as redundant
information and use lossy speech compression techniques to mitigate these
perturbations. Quantization [31, 30] involves rounding each audio sample point
to the nearest integer multiple of a factor $q$, which can impact the fragile
adversarial perturbations. Chen et al. [30] propose to do k-means [32] on the
acoustic features to get clusters of acoustic features, and use the clusters
to represent the acoustic features.

The “adding noise" approach aims to disrupt and neutralize adversarial
perturbations, by introducing additional noise, typically Gaussian. Randomized
smoothing [31, 33, 30, 34] involves adding random Gaussian noise to the input
utterances before sending them to the ASV to counter the adversarial
perturbations. [35] adopts to the idea of “voting for the right answer" to
prevent risky decisions of ASV in blind spot areas. To achieve this, they
samples the neighbors of a given utterance by random sampling using Gaussian
noise, and allow the neighbors to vote on whether the utterance should be
accepted by the ASV model or not, rather than relying solely on the prediction
of the single utterance. Olivier et al. [36] is an enhanced version by adding
Gaussian noise to the high-frequency region rather than the entire utterance.

The “Denosing method" treats adversarial noise as a specific kind of noise and
aims to estimate and eliminate it. Chang et al. [33] suggest using a denoising
algorithm tailored for Gaussian noise and they contend that the denoising
algorithm can also cleanse the adversarial noise. Zhang et al. [37] propose to
employ an adversarial separation network, which is trained using the
adversarial-genuine data pairs, to estimate and purify the adversarial noise.
This method requires prior knowledge of adversarial sample generation.

The “generative method" approach typically involve training a generative model
to model the genuine data manifolds and using this model to pull the
adversarial samples towards the genuine data manifolds. Wu et al. [38] propose
the SSLM-based reconstruction to alleviate the superficial adversarial noise
and maintain key information for genuine samples. They [38] utilize the self-
supervised learning models to extract key features from the adversarial
samples, and do reconstruction to pull the inputs to the genuine data
manifold. Joshi et al. [34] use the encoder of a VAE [39] to project testing
data onto a latent posterior that aligns with the genuine manifold. They then
use the decoder to re-generate the input data based on the hidden embedding
sampled by the latent posterior, thereby purifying superficial adversarial
noise. Joshi et al. [34] borrow the DefenseGAN from computer vision [40]. The
DefenseGAN projects the testing data, either adversarial or genuine, into the
low-dimensional manifold of genuine data to get the hidden embeddings and then
re-generate the testing data by the generator using such embeddings.

“Filtering", also known as local smoothing, helps smooth and alleviate the
superficial adversarial perturbations. Local smoothing involves applying
Gaussian, mean, and median filters to the waveform to purify the adversarial
noise. [31, 30] and [29] utilize local smoothing to defend ASV and
countermeasures, respectively.

####  3.2.3 Adversarial sample detection

The detection methods can be classified into two categories based on whether
they require prior knowledge about adversarial sample generation: attack-
dependent or attack-independent detection methods.

The attack-dependent methods usually leverage the deep learning models to
implicitly find cues to differentiate between specific kinds of adversarial
samples and genuine samples using both adversarial and genuine data. Li et al.
[41] propose to train a detector using the binary classification loss to
distinguish the adversarial and genuine samples. They find their detector is
unable to detect unseen adversarial samples derived by other adversarial
attack algorithms that are not used during training. Based on that different
kinds of adversarial samples attain different attack signatures, Villalba et
al. [42] propose to train an x-vector [13] system to extract the bottleneck
features as the attack signatures using various types of adversarial samples.
After training the x-vector system, attack signatures will be extracted for
different types of attacks. During inference, the testing utterance is
inputted, and the x-vector feature extractor will extract the hidden
embeddings. These embeddings are then compared with the enrolled attack
signatures to determine whether the testing utterance is an adversarial sample
or not. To further improve the performance of the attack signature extractor,
Joshi et al. [43] propose training the attack signature extractor using
adversarial perturbations instead of adversarial examples. They argue that the
adversarial perturbations eliminate redundant information from the adversarial
samples. They then train an adversarial perturbation estimator to extract
adversarial perturbations from the input utterance and use the attack
signature extractor to extract hidden features to detect the adversarial
samples.

Attack-independent methods treat the detection of adversarial samples as an
anomaly detection problem. Genuine data samples always exhibit some properties
that are absent or different for adversarial samples. Therefore, attack-
independent detection methods can exploit the inconsistency of these internal
properties to distinguish between adversarial and genuine samples. Wu et al.
[38] leverage the ASV score difference before and after putting the testing
utterance into SSLMs as an indicator to differentiate between adversarial and
genuine samples. Specifically, for genuine samples, the ASV score difference
before and after putting the utterance into SSLMs is small, while for
adversarial samples, the difference is large. Peng et al. [44] propose to
detect adversarial samples using twin ASV models, including one premier model
that is exposed to attackers and is fragile under adversarial attack, and one
mirror model that is robust to adversarial attacks and cannot be accessed by
attackers. When a genuine sample is inputted, both the premier and mirror
models produce similar predictions. However, when an adversarial sample is
inputted, the models produce different predictions. Peng et al. [44] leverage
the score inconsistency between genuine and adversarial samples to detect
adversarial samples. Wu et al. [45] utilize the vocoders to re-synthesize the
input utterance and find that the difference between the ASV scores for the
original and re-synthesized utterance is a good indicator for discrimination
between genuine and adversarial samples. To be specific, the score difference
for adversarial samples is large, while it is small for genuine samples. Chen
et al. [46] utilize two kinds of hand-crafted masks to detect adversarial
samples: they mask parts of the input speech features. They claim the masked
parts contain less speaker information and won’t affect the ASV scores for
genuine samples two much, but will greatly impact the adversarial samples. By
comparing the absolute difference of scores before and after masking, they are
able to detect adversarial examples. The two masks used are MLFB-H, which
masks the high frequencies of LogFBank, and MLFB-D, which masks the time-
frequency bins whose absolute values of their one-order difference along the
frequency axis are smaller than a threshold. Chen et al. [47] further enhance
the detection performance by learning such mask matrix by a deep recurrent
networks, rather than using hand-crafted masks.

##  4 Future directions

For the future directions of partially fake speech attacks: 1). Data
collection. The collection of data is a crucial component in developing an
effective defense system against partially fabricated speech. Only 100k
utterances are collected by [6] for partially fake detection and the
transition boundaries are not stealthy enough. To this end, there exists a
pressing need to investigate the generation of more data with discreet
transition boundaries, while carefully considering the linguistic and acoustic
characteristics involved. This undertaking is of great significance and
warrants further exploration. 2). Reduce training efforts. The state-of-the-
art (SOTA) methodology for partially fake speech detection involves the fine-
tuning of the entire SSLMs. The SSLM in [21] is with 2 billion parameters,
which presents a challenge for academic researchers when attempting to fine-
tune the model. Several works have emerged that offer promising avenues for
minimizing training efforts while maximizing the benefits of SSLMs, including
linear probing, adapter, and prompt techniques. Exploring these approaches may
significantly enhance the efficiency of adopting SSLMs for partially fake
speech detection. 3). Model compression. The current state-of-the-art
detection method relies heavily on large-scale SSLMs. The parameter number of
the SSLM used in [21] is 2 billion parameters. Therefore, investigating
approaches to reduce the model size is a crucial research endeavor. This issue
warrants considerable attention as it has significant implications for the
scalability, computational efficiency, and generalizability of partially fake
speech detection systems.

The re-synthesis-based adversarial sample detection methods achieves the SOTA
[45, 46, 47]. An effective audio re-synthesis method for adversarial sample
detection must possess two critical properties. Firstly, the score variations
between the original and re-synthesized utterances should be minimal for
genuine samples. Secondly, the score variations between the original and re-
synthesized utterances for adversarial samples should be substantial.
Investigating approaches for refining the design of audio re-synthesis methods
to further optimize these properties represents a valuable research direction.
By enhancing the efficacy of the audio re-synthesis method, it would be
possible to improve the reliability and accuracy of detection systems.

##  5 Conclusion

This paper reviews the defense methods against adversarial attacks and
partially fake speech attacks that have recently emerged. We hope the
comprehensive review and comparisons can inspire future works to boost the
robustness of ASV. Further investigation is needed to explore future
directions as in Section 4

## References

  * Yamagishi et al. [2021] J. Yamagishi, et al.,  Asvspoof 2021: accelerating progress in spoofed and deepfake speech detection,  arXiv preprint arXiv:2109.00537 (2021).
  * Todisco et al. [2019] M. Todisco, et al.,  Asvspoof 2019: Future horizons in spoofed and fake audio detection,  arXiv preprint arXiv:1904.05441 (2019).
  * Kinnunen et al. [2017] T. Kinnunen, et al.,  The asvspoof 2017 challenge: Assessing the limits of replay spoofing attack detection (2017).
  * Wu et al. [2015] Z. Wu, et al.,  Asvspoof 2015: the first automatic speaker verification spoofing and countermeasures challenge,  in: Sixteenth Annual Conference of the ISCA, 2015\.
  * Tan et al. [2022] H. Tan, et al.,  Adversarial attack and defense strategies of speaker recognition systems: A survey,  Electronics 11 (2022) 2183.
  * Yi et al. [2022] J. Yi, et al.,  Add 2022: the first audio deep synthesis detection challenge,  in: IEEE ICASSP, 2022.
  * Yi et al. [2021] J. Yi, et al.,  Half-truth: A partially fake audio detection dataset,  in: Interspeech, 2021, pp. 1654–1658.
  * Zhang et al. [2021] L. Zhang, et al.,  An initial investigation for detecting partially spoofed audio,  arXiv preprint arXiv:2104.02518 (2021).
  * Abdullah et al. [2021] H. Abdullah, et al.,  Sok: The faults in our asrs: An overview of attacks against automatic speech recognition and speaker identification systems,  in: IEEE SP, 2021, pp. 730–747.
  * Das et al. [2020] R. K. Das, et al.,  The attacker’s perspective on automatic speaker verification: An overview,  arXiv preprint arXiv:2004.08849 (2020).
  * Kreuk et al. [2018] F. Kreuk, et al.,  Fooling end-to-end speaker verification with adversarial examples,  in: IEEE ICASSP, 2018, pp. 1962–1966.
  * Li et al. [2020] X. Li, et al.,  Adversarial attacks on gmm i-vector based speaker verification systems,  in: IEEE ICASSP, 2020, pp. 6579–6583.
  * Villalba et al. [2020] J. Villalba, et al.,  x-vectors meet adversarial attacks: Benchmarking adversarial robustness in speaker verification,  ISCA Interspeech (2020) 4233–4237.
  * Liu et al. [2019] S. Liu, et al.,  Adversarial attacks on spoofing countermeasures of automatic speaker verification,  in: IEEE ASRU, 2019, pp. 312–319.
  * Zhang et al. [2020] Y. Zhang, et al.,  Black-box attacks on spoofing countermeasures using transferability of adversarial examples.,  in: ISCA Interspeech, 2020, pp. 4238–4242.
  * Wu et al. [2022] H. Wu, et al.,  Partially fake audio detection by self-attention-based fake span discovery,  in: IEEE ICASSP, IEEE, 2022, pp. 9236–9240.
  * Cai et al. [2022] Z. Cai, et al.,  Waveform boundary detection for partially spoofed audio,  arXiv preprint arXiv:2211.00226 (2022).
  * Wang et al. [2022] L. Wang, et al.,  Synthetic voice detection and audio splicing detection using se-res2net-conformer architecture,  arXiv preprint arXiv:2210.03581 (2022).
  * Zhang et al. [2021] L. Zhang, et al.,  Multi-task learning in utterance-level and segmental-level spoof detection,  arXiv preprint arXiv:2107.14132 (2021).
  * Zhang et al. [2022] L. Zhang, et al.,  The partialspoof database and countermeasures for the detection of short fake speech segments embedded in an utterance,  IEEE/ACM Transactions on Audio, Speech, and Language Processing (2022).
  * Lv et al. [2022] Z. Lv, et al.,  Fake audio detection based on unsupervised pretraining models,  in: IEEE ICASSP, IEEE, 2022, pp. 9231–9235.
  * Babu et al. [2021] A. Babu, et al.,  Xls-r: Self-supervised cross-lingual speech representation learning at scale,  arXiv preprint arXiv:2111.09296 (2021).
  * Allam and Haggag [2012] A. M. N. Allam M. H. Haggag,  The question answering systems: A survey,  IJRRIS 2 (2012).
  * Baevski et al. [2020] A. Baevski, et al.,  wav2vec 2.0: A framework for self-supervised learning of speech representations,  Advances in neural information processing systems 33 (2020) 12449–12460.
  * Chen et al. [2022] S. Chen, et al.,  Wavlm: Large-scale self-supervised pre-training for full stack speech processing,  IEEE Journal of Selected Topics in Signal Processing 16 (2022) 1505–1518.
  * Jati et al. [2021] A. Jati, et al.,  Adversarial attack and defense strategies for deep speaker recognition systems,  Computer Speech & Language 68 (2021) 101199.
  * Du et al. [2020] T. Du, et al.,  Sirenattack: Generating adversarial audio for end-to-end acoustic systems,  in: ACM ASIACCS, 2020.
  * Wang et al. [2019] Q. Wang, et al.,  Adversarial regularization for end-to-end robust speaker verification.,  in: Interspeech, 2019.
  * Wu et al. [2020] H. Wu, et al.,  Defense against adversarial attacks on spoofing countermeasures of asv,  arXiv preprint arXiv:2003.03065 (2020).
  * Chen et al. [2022] G. Chen, et al.,  Towards understanding and mitigating audio adversarial examples for speaker recognition,  IEEE TDSC (2022).
  * Chen et al. [2019] G. Chen, et al.,  Who is real bob? adversarial attacks on speaker recognition systems,  arXiv preprint arXiv:1911.01840 (2019).
  * Hartigan and Wong [1979] J. A. Hartigan M. A. Wong,  Algorithm as 136: A k-means clustering algorithm,  Journal of the royal statistical society. series c (applied statistics) 28 (1979) 100–108.
  * Chang et al. [2021] L.-C. Chang, et al.,  Defending against adversarial attacks in speaker verification systems,  in: IEEE IPCCC, 2021.
  * Joshi et al. [2021] S. Joshi, et al.,  Adversarial attacks and defenses for speaker identification systems,  arXiv preprint arXiv:2101.08909 (2021).
  * Wu et al. [2021] H. Wu, et al.,  Voting for the right answer: Adversarial defense for speaker verification,  arXiv preprint arXiv:2106.07868 (2021).
  * Olivier et al. [2021] R. Olivier, et al.,  High-frequency adversarial defense for speech and audio,  in: IEEE ICASSP, 2021.
  * Zhang et al. [2020] H. Zhang, et al.,  Adversarial separation network for speaker recognition.,  in: Interspeech, 2020.
  * Wu et al. [2021] H. Wu, et al.,  Improving the adversarial robustness for speaker verification by self-supervised learning,  IEEE/ACM Transactions on Audio, Speech, and Language Processing 30 (2021) 202–217.
  * Kingma and Welling [2013] D. P. Kingma M. Welling,  Auto-encoding variational bayes,  arXiv preprint arXiv:1312.6114 (2013).
  * Samangouei et al. [2018] P. Samangouei, et al.,  Defense-gan: Protecting classifiers against adversarial attacks using generative models,  arXiv preprint arXiv:1805.06605 (2018).
  * Li et al. [2020] X. Li, et al.,  Investigating robustness of adversarial samples detection for automatic speaker verification,  arXiv preprint arXiv:2006.06186 (2020).
  * Villalba et al. [2021] J. Villalba, et al.,  Representation learning to classify and detect adversarial attacks against speaker and speech recognition systems,  arXiv preprint arXiv:2107.04448 (2021).
  * Joshi et al. [2022] S. Joshi, et al.,  Advest: Adversarial perturbation estimation to classify and detect adversarial attacks against speaker identification,  arXiv preprint arXiv:2204.03848 (2022).
  * Peng et al. [2021] Z. Peng, et al.,  Pairing weak with strong: Twin models for defending against adversarial attack on speaker verification.,  in: Interspeech, 2021.
  * Wu et al. [2022] H. Wu, et al.,  Adversarial sample detection for speaker verification by neural vocoders,  in: IEEE ICASSP, 2022.
  * Chen et al. [2022a] X. Chen, et al.,  Masking speech feature to detect adversarial examples for speaker verification,  in: IEEE APSIPA ASC, 2022a.
  * Chen et al. [2022b] X. Chen, et al.,  Lmd: A learnable mask network to detect adversarial examples for speaker verification,  arXiv preprint arXiv:2211.00825 (2022b).