# InfiMM-HD: A Leap Forward in High-Resolution Multimodal Understanding

Haogeng Liu  Quanzeng You  Xiaotian Han  Yiqi Wang  Bohan Zhai  Yongfei Liu
Yunzhe Tao  Huaibo Huang  Ran He  Hongxia Yang

###### Abstract

Multimodal Large Language Models (MLLMs) have experienced significant
advancements recently. Nevertheless, challenges persist in the accurate
recognition and comprehension of intricate details within high-resolution
images. Despite being indispensable for the development of robust MLLMs, this
area remains underinvestigated. To tackle this challenge, our work introduces
InfiMM-HD, a novel architecture specifically designed for processing images of
different resolutions with low computational overhead. This innovation
facilitates the enlargement of MLLMs to higher-resolution capabilities.
InfiMM-HD incorporates a cross-attention module and visual windows to reduce
computation costs. By integrating this architectural design with a four-stage
training pipeline, our model attains improved visual perception efficiently
and cost-effectively. Empirical study underscores the robustness and
effectiveness of InfiMM-HD, opening new avenues for exploration in related
areas. Codes and models can be found https://huggingface.co/Infi-MM/infimm-hd

Machine Learning, ICML, Multimodal Large Language Models

##  1 Introduction

Figure 1: InfiMM-HD showcases superior performance across various tasks,
thanks to its enhanced training pipeline and high-resolution inputs,
consistently outperforming recent methods in downstream tasks. Figure 2:
Example outputs by InfiMM-HD, highlighting the model’s adeptness in fine-
grained visual perception.

The landscape of Multimodal Large Language Models (MLLMs) has been
revolutionized by integrating pretrained vision encoders with Large Language
Models (LLMs) (Han et al., 2023; Wang et al., 2024; Han et al., 2024), a trend
exemplified by developments in Flamingo (Alayrac et al., 2022), BLIP-2 (Li et
al., 2023b), LLaVA (Liu et al., 2023b), MiniGPT-4 (Zhu et al., 2023a) and etc.
MLLMs can exhibit emergent vision-language capabilities (Yang et al., 2023).
For example, they can write codes according to images, convert plots in images
into Markdown format tables, and perform web browsing, by leveraging combined
single-modal capabilities from pretrained vision encoders and LLMs.

Effective integration of pretrained vision encoders with Large Language Models
in MLLMs relies on carefully designed vision-language bridging modules. These
modules address two critical aspects: the transformation and alignment of
visual tokens to LLM-compatible formats, and the subsequent utilization of
these transformed tokens.

For the transformation and alignment of visual tokens, models such as Flamingo
(Alayrac et al., 2022) and BLIP-2 (Li et al., 2023b) employ Perceiver-
Resampler/Q-Former techniques for transforming visual tokens, offering
flexibility and nuanced alignment with language counterparts but at the cost
of higher computational demands and potential information loss (Cha et al.,
2023). Conversely, models such as LLaVA and MiniGPT-v2 take a different route,
using simpler Multi-Layer Perceptron (MLP) approaches. While this reduces
computational complexity and the number of learnable parameters, it might not
capture the full intricacies of visual data.

The integration phase of visual and language tokens is equally vital.
Flamingo-style architectures employ cross-attention mechanisms, facilitating
intricate interactions between token types without expanding the token
sequence length. This approach efficiently manages the computational load.
LLaVA-style models, however, use a direct concatenation method, which, while
straightforward, increases token sequence length and computational complexity.

Though different, both architecture utilized low image resolution due to
pretrained Vision Transformer (ViT) encoders (Jiang et al., 2023; Radford et
al., 2021; He et al., 2021). Low resolution suffices for basic image-level
semantic understanding but falls short for detailed, region-level analysis.
Recent efforts (Wang et al., 2023; Li et al., 2023c; Lin et al., 2023) aim to
enable MLLMs to handle higher-resolution images. However, significant
challenges remain, primarily because the computational demands tend to
increase quadratically in relation to the sequence length for larger images.
For instance, increasing the image resolution from $224\times 224$
(Dosovitskiy et al., 2021) to $448\times 448$ multiplies the self-attention
computation by 16 times.

To address these challenges, we introduce InfiMM-HD, an innovative MLLM
architecture designed for processing high-resolution image. InfiMM-HD
innovatively merges methodologies from both Flamingo and LLaVA styles in
MLLMs. For the transformation and alignment phase, it adopts an MLP-based
approach, akin to LLaVA, effectively transforming and aligning visual tokens
into formats compatible with LLMs. This strategy balances computational
efficiency with precise processing. In the integration phase, InfiMM-HD
utilizes a cross-attention mechanism, reminiscent of Flamingo-style MLLMs, to
seamlessly incorporate visual token features with language tokens. This
approach mitigates the computational challenges associated with longer token
sequences as previously mentioned. Notably, while the exploration of high-
resolution image input capabilities in Flamingo-style MLLMs is still an
emerging area of research, InfiMM-HD marks a significant, pioneering
advancement in this domain, blending the best of both worlds to enhance MLLM
performance with high-resolution visual inputs.

To overcome the resolution constraints of pretrained vision encoders, InfiMM-
HD is strategically trained in four stages, enhancing resolution handling
while maintaining vision-language alignment. Initially, the model is
pretrained with $224\times 224$ resolution images for efficient visual-
language alignment. Subsequently, it continues pretraining with interpolated
positional embeddings for $448\times 448$ images from multiple datasets,
keeping the LLMs frozen. This is followed by training with full-resolution
images, resized to the nearest multiple of $448\times 448$, added with 2D
positional embeddings and crop to multiple subimages. In the final stage, the
model undergoes visual instruction fine-tuning, freezing the vision encoder
and making LLM trainable to enhance instruction-following capability. This
structured training approach is crucial for the model’s adaptability and
performance across various input resolutions. The contributions of our work
can be summarized as follows:

  * •

We present InfiMM-HD, a pioneering MLLM that employs an MLP-based approach for
visual token transformation and alignment, coupled with a Flamingo-style
cross-attention mechanism for enhanced and efficient integration of
transformed visual and language tokens. It is uniquely designed to seamlessly
process high-resolution image inputs.

  * •

We present a four-stage training pipeline that effectively achieves a high-
resolution Multimodal Large Language Model with reduced training cost, from
initial low-resolution pretraining stage, to continue pretraining stage for
knowledge injection and alignment, to dynamic resolution adaption stage for
high resolution adoption and finally go through visual instruction fine-tuning
stage.

  * •

Experiments conducted across diverse benchmarks showcase the remarkable
proficiency of our model in the realm of visual perception. Additionally,
comprehensive ablation studies underscore the distinctive superiority of our
design within the context of cross-attention-style Multimodal Language Model
architectures.

##  2 Related Work

The advent of Large Language Models (LLMs) has catalyzed the development of
MLLMs. Flamingo (Alayrac et al., 2022) integrates pretrained language models
into the MLLM paradigm, employing a gated-cross attention mechanism to fuse
visual information into textual sequences. In contrast, BLIP-2 (Li et al.,
2023b), MiniGPT4 (Zhu et al., 2023a), and LLaVA (Liu et al., 2023b) propose a
paradigm shift, transforming visual signals into soft tokens and directly
incorporating them into language models. Shikra (Chen et al., 2023)
concentrates on referential dialogue. OtterHD (Li et al., 2023a) fine-tunes
Fuyu-8B (Bavishi et al., 2023) with instructional guidance, enabling ViT-free
MLLMs.

Despite the progress we have seen, some problems still exist. (Zhai et al.,
2023) points out that misalignment between visual representation and language
causes hallucination. (Zhang et al., 2023) reveals that enhancing the input
resolution will significantly increase MLLM’s Optical Character Recognition
(OCR) ability. More and more experiments suggest the presence of an
information bottleneck in contemporary vision encoders (Tong et al., 2024;
Zhai et al., 2023). The resolution of the image stands out as a critical
factor that constrains the capacity for visual processing. The study by (Tong
et al., 2024) highlights that contemporary MLLMs still face systematic
challenges, particularly in aligning visual and textual modalities
effectively.

There are some works trying to solve the problem. SPHINX (Lin et al., 2023),
introduced by Lin et al., employs a multi-layer perception (MLP) to establish
connections between visual signals and language. This model leverages multiple
vision encoders, amalgamating their output features to construct a robust
visual representation. To deal with high resolution image input, SPHINX breaks
down input high-resolution images into smaller sub-images and then concatenate
visual tokens directly with text tokens. It introduces limitations related to
increasing input sequence lengths for the Large Language Model (LLM).

The Monkey model (Li et al., 2023c) addresses this challenge by incorporating
a shared resampler. This approach involves compressing each input subimage
using the resampling technique from Flamingo (Alayrac et al., 2022) and
directly concatenate the visual tokens with text sequence, effectively
upscaling the input image resolution to $1344\times 896$. However, the
reliance on learnable queries within the perceiver architecture for extracting
and compressing information from the raw output of the vision encoder raises
concerns about the model’s adaptability across diverse application scenarios.

Figure 3: Visualization of the distribution of image sizes from the LLaVA 665k
dataset indicates a predominant clustering of resolutions between 500-700,
mixed with some high-resolution examples. Dynamic resolution utilization
during training is key for efficient resource management. Figure 4: The
architectural framework of InfiMM-HD is outlined, where POS(i, j) represents
the positional embedding of local patches, with (i, j) indicating their
position within the overall image. The model progresses through various
training stages, each characterized by selectively training different modules.
A detailed explanation of this strategic approach will be elaborated in the
following sections. Figure 5: The four stages of InfiMM-HD training pipeline.
Each stage is characterized by distinct trainable modules, datasets, and the
resolution of images input to ViT. Our experimental findings confirm the
efficacy of our approach, demonstrating the benefits of progressively
transitioning from 224x224 to high-resolution images.

We assert that the information compression process should intricately
intertwine with contextual instructions, allowing for discernment of pertinent
details essential for instruction completion. We introduce InfiMM-HD, which
establishes connections between vision and language through cross attention
mechanism. This departure from the reliance on learnable queries aims to
enhance the adaptability and applicability of the model across a broader
spectrum of scenarios-more detailed vision perception. Besides, it enables the
model to consume high-resolution images at lower cost than previously proposed
methods.

##  3 Methods

In this section, we introduce InfiMM architecture and propose a training
pipeline for elevating MLLM’s input image resolution with reduced cost. To the
best of our knowledge, we are the pioneers in achieving HD MLLM using the
Flamingo-style architecture.

###  3.1 Model Architecture

Figure 6: Illustration of data augmentation: Introducing randomly generated
characters with diverse colors into arbitrary regions of the image.
Corresponding questions are then generated to complement the original query.

The proposed model consists of three components: a Vision Transformer Encoder,
a Gated Cross Attention Module, and a Large Language Model. The comprehensive
architecture is elucidated in Figure 4. While the illustration showcases a
single image, it is imperative to note that follow flamingo’s design, our
module can also deal with multiple image as input. Following prior work (Li et
al., 2023b; Wang et al., 2023), for the Vision Transformer, we employ
EVA2-CLIP2-E (Sun et al., 2023), utilizing the output from the penultimate
layer as the extracted vision features. The Gated Cross Attention Module
leverages text hidden states as queries, and vision features as keys and
values. Different from the gating methodology introduced in Flamingo (Alayrac
et al., 2022), we incorporate an element-wise $\tanh$ gating mechanism for
activation. The language model in this study is instantiated using Vicuna
(Chiang et al., 2023).

To ensure an effective assimilation of visual information, the Gated Cross
Attention Module is strategically inserted every four layers between the
decoder layers of Large Language Model. This decision stems from our empirical
observation that inserting the module every two layers results in
approximately 50% of the gates weights near 0, rendering the cross-attention
module ineffective. The model showcases an impressive aggregate of 18 billion
parameters, intricately allocated among three key components: the Vision
Transformer (4.4 billion), the Large Language Model (12.9 billion), and the
Gated Cross Attention Module (approximately 0.7 billion).

During this investigation, we depart from the conventional paradigm of the
LLaVA-style structure. This deviation is essential due to its compromised
compatibility with high-resolution images, as demonstrated in previous studies
(Li et al., 2023c; Lin et al., 2023). Notably, the processing of an image with
dimensions $1344\times 1344$ yields an extensive token sequence comprising
9217 tokens when employing a patch size of 14. Despite the capability of Large
Language Models (LLMs) to accommodate sequence lengths up to 32k, the
utilization of 9k tokens per image inherently imposes constraints on the
performance of Multimodal Language Models (MLLMs), particularly in scenarios
involving multiple images. This limitation is consequential, underscoring the
necessity for alternative architectural considerations. These considerations
aim to tackle the challenges posed by high-resolution image processing within
the context of contemporary language models. We adopt a cross-attention module
for the integration of visual information at a reduced dimensionality of 768.
This method, in contrast to the LLaVA-style architecture, incurs significantly
lower computational costs while accommodating extended sequences. Meanwhile,
our experiments demonstrate its effective of extracting visual information.

Table 1: Details on the training data of CPT and DRA.

Task Dataset Samples Image Caption COCO Caption (Chen et al., 2015) 205k
TextCaps (Sidorov et al., 2020) 55k VizWiz Caption (Gurari et al., 2020) 55k
General VQA VQAV2 (Antol et al., 2015) 443k OKVQA (Marino et al., 2019) 9k
VizWiz VQA (Gurari et al., 2018) 20k GQA (Hudson & Manning, 2019) 471k A-OKQA
(Schwenk et al., 2022) 17k Text-oriented VQA TextVQA (Singh et al., 2019) 34k
OCRVQA (Mishra et al., 2019) 166k STVQA (Biten et al., 2019) 26k DocVQA
(Mathew et al., 2021) 63k LLaVAR (Zhang et al., 2023) 16k Region Description
VG (Krishna et al., 2017) 429k Total - 2.00m

###  3.2 Training Details

We have established a four-stage training procedures for improving MLLMs’
capability of processing high-resolution images, as shown in Figure 5. These
stages are denoted as the Pretraining (PT), Continue Pretraining (CPT),
Dynamic Resolution Adaption (DRA), and Instruction Finetuning (IFT).

Pretraining Stage (PT): This stage is mainly for initially aligning vision
features and language features. During this stage, both the Vision Transformer
(ViT) and Large Language Model (LLM) are frozen, only the Gated Cross
Attention module is trainable. In this stage, all of the images are resized to
$224\times 224$ to keep the low training cost.

Continue Pretraining Stage (CPT): In this stage, we employ bilinear
interpolation of positional embedding to extend the ViT’s capability to
process image of resolution $448\times 448$. The ViT and Gated Cross Attention
modules are trainable. Training datasets mainly focus on image captioning and
visual question-answering tasks. Detailed information about training datasets
is listed in Table 1.

Dynamic Resolution Adaption (DRA): In Figure 3, the outcomes illustrating the
sizes of images in the LLaVA-665k dataset (Liu et al., 2023a). Upon meticulous
examination of the dataset, it becomes apparent that not all images exhibit
resolutions reaching up to 1344. In contrast to conventional practices of
uniformly resizing images to a fixed resolution, such an approach incurs
unnecessary computational costs, while dynamic image resolution may be cost
friendly. To facilitate dynamic resolution inputs, we incorporate the 2D
position embedding method proposed in (Wang & Liu, 2021) for individual sub-
images. We adopts dynamic input image resolution, ranging from $448\times 448$
to $1344\times 1344$, during training. Subsequently, the resized image is
divided into sub-images of $448\times 448$. We also keep an original image
thumbnail of $448\times 448$. Finally we use ViT to extract features from each
sub-image and original image thumbnail, concatenated directly to form the
final vision feature. We use the same training datasets as CPT stage, and keep
both the ViT and Gated Cross Attention modules trainable.

Instruction Finetuning Stage (IFT): In this final stage, our goal is to make
the model better follow user instructions without losing high-resolution
visual perception capability. Thus, we keep the ViT frozen, but let the Gated
Cross Attention modules and LLM trainable.

The proposed four-stage training pipeline is key to stabilize training while
elevate input image resolution gradually.

##  4 Experiments

In this section, we first discuss about experiment setup. Then we show main
results of InfiMM-HD and list series of ablation studies to prove the
importance of our proposed modules in InfiMM-HD.

###  4.1 Setup

Training Dataset. For Pretrainin (PT) stage, training data includes both
image-text pairs and interleaved image-text. Image-text pair data includes
140M samples filtered from LAION-2B (Li et al., 2023b), COYO (Byeon et al.,
2022), and Laion-coco (Schuhmann et al., ). Interleaved image-text data is
sampled from MMC4 (Zhu et al., 2023b) and OBELISIC (Laurençon et al., 2023)
randomly with $50\%$ for training.

The datasets utilized in the Continue Pretraining (CPT) and Dynamic Resolution
Adaption (DRA) stages are enumerated in Table 1. During the Instruction
Finetuning (IFT) stage, we amalgamate datasets from LLaVA-665k (Liu et al.,
2023a), LLaVAR (Zhang et al., 2023), TextVQA (Singh et al., 2019), and
ScienceQA (Lu et al., 2022). This fusion is motivated by the fact that the
original LLaVA-665k dataset lacks text-oriented samples. Consequently, we
supplement this deficiency by incorporating additional data from these diverse
sources.

In the IFT stage, we primarily utilize the LLaVA-665k dataset (Liu et al.,
2023a) by default. Alongside, we incorporate additional datasets such as
TextVQA, LLAVAR, and ScienceQA to enrich our instruction tuning dataset.

Text-oriented Data Augmentation. Due to a scarcity of text-oriented data, we
employ a straightforward but effective data augmentation methodology.
Concretely, this process involves the incorporation of scene-irrelevant
characters into images, followed by the generation of corresponding questions.
In this context, we randomly introduce two sets of characters, each positioned
at the left-top and right-bottom corners of the image, respectively, with
distinct colors. The questions generated include inquiries such as “What
character is situated at the left-top of the image?”, “What is the color of
the characters located at the right-bottom of the image?”, and “How many
characters are present in the left-top region of the image?”. It is imperative
to note that this data augmentation technique is exclusively applied to the
GQA dataset, resulting in the generation of approximately 100,000 question-
answer pairs. Remarkably, we observe the effectiveness of this augmentation
approach in enhancing our model’s proficiency in text recognition. Examples
can be found in Figure 6.

Training Details. The training process was facilitated through the utilization
of deepspeed (Aminabadi et al., 2022), and the FusedAdam optimizer was
selected to orchestrate optimization tasks. Additional intricacies related to
the experimental configurations are thoroughly delineated in the accompanying
Appendix 8.

Table 2: Results on general VQA task. The table exclusively presents the
performance of our generalist model, showcasing its superiority compared with
various models.

Model LLM In-house data OKVQA IconVQA GQA VQAv2 ScienceQAimg Flamingo-80B
(Alayrac et al., 2022) - ✓ 50.6 - - 56.3 - Palm-E-12B (Driess et al., 2023) -
✓ 60.1 - - 77.7 - Qwen-VL (Bai et al., 2023) Qwen-7B ✓ 58.6 - 59.3 79.5 67.1
Qwen-VL-Chat (Bai et al., 2023) Qwen-7B ✓ 56.6 - 57.5 78.2 68.2 CogVLM (Wang
et al., 2023) Vicuna-7B ✓ 58.9 - - 83.4 - Monkey (Li et al., 2023c) Qwen-7B ✓
61.3 - 60.7 80.3 69.4 BLIP-2 (Li et al., 2023b) Vicuna-13B $\times$ 45.9 40.6
41.0 - - Shikra (Chen et al., 2023) Vicuna-13B $\times$ 47.2 - - 77.4 - mPLUG-
Owl2(Ye et al., 2023) LLaMA2-7B $\times$ 57.7 - 56.1 79.4 68.7 LLaVA 1.5 (Liu
et al., 2023a) Vicuna-13B $\times$ - - 63.3 80.0 71.6 Sphinx-2K (Lin et al.,
2023) LLaMA2-13B $\times$ 62.6 50.5 63.1 80.7 70.6 InfiMM-HD Vicuna-13B
$\times$ 65.5 51.3 63.5 82.0 83.6

Evaluation. We evaluate InfiMM-HD across a diverse array of VQA tasks. For
general VQA tasks, we leverage benchmarks such as OKVQA (Marino et al., 2019),
VQAV2 (Antol et al., 2015), GQA (Hudson & Manning, 2019), and ScienceQA (Lu et
al., 2022). These datasets, while not demanding advanced detail visual
perception capabilities from the model, effectively gauge models’ ability to
understand general scenes and follow user instructions. Moreover, to
scrutinize our model’s fine-grained detail perception capability, we
incorporate text-oriented VQA datasets, including TextVQA (Singh et al.,
2019), STVQA (Biten et al., 2019), and OCRVQA (Mishra et al., 2019). We assess
the logical reasoning capabilities of our model by employing newly introduced
benchmarks, including MM-VET (Yu et al., 2023), MME (Fu et al., 2023), MMbench
(Liu et al., 2023c), InfiMM-Eval (Han et al., 2023), and MMMU (Yue et al.,
2023). Notably, the MMMU (Yue et al., 2023) presents challenging tasks that
demand advanced subject knowledge and deliberate reasoning at a collegiate
level. These tasks span diverse fields such as physics, chemistry, and
biology. The MM-VET benchmark assesses the integrated capabilities of models.

Table 3: Evaluation results for text-oriented Visual Question Answering (VQA)
task. For STVQA, Monkey randomly samples data from the train set for
evaluation.

Model Res In-house data TextVQA OCRVQA STVQA Qwen-VL-Chat(Bai et al., 2023)
$448\times 448$ ✓ 61.5 70.5 - Monkey (Li et al., 2023c) $1344\times 768$ ✓
67.6 - 67.7 UniDoc (Gu et al., 2022) $224\times 224$ $\times$ 40.7 34.5 30.8
DocPedia (Feng et al., 2023) $2560\times 2560$ $\times$ 60.2 57.2 45.5 BLIP-2
(Li et al., 2023b) $224\times 224$ $\times$ - 40.6 - LLaVA1.5 (Liu et al.,
2023a) $336\times 336$ $\times$ 48.5 - - Sphinx-2K (Lin et al., 2023)
$768\times 768$ $\times$ 61.2 67.8 - InfiMM-HD (all are piexl-only) dynamic
$\times$ 70.7 66.0 67.0

###  4.2 Main Results

We present evaluation results of general VQA and text-oriented VQA tasks in
this section.

Table 3 presents results for general VQA tasks. It is essential to underscore
that the scope of evaluations in OKVQA, GQA, and VQAv2 extends beyond the mere
assessment of models’ visual perception capabilities, they also critically
examine the models’ ability to utilize prior knowledge effectively, thereby
providing a more comprehensive evaluation of models’ overall functionality.
Additionally, ScienceQA (Lu et al., 2022), which comprises 21,000 multimodal
multiple-choice questions covering a wide range of scientific subjects,
represents a significant expansion of the benchmarking landscape. In these
varied tasks, our model has shown remarkable effectiveness, indicating
significant performance improvements. By outperforming its closest competitor
by an average margin of 3.88%, our model not only showcases its superior
capacity for integrating multimodal information but also its proficiency in
leveraging extensive prior knowledge to navigate through a diverse array of
questions successfully.

In addition to the general VQA assessment, we further explore our model’s
detailed visual perception capability by evaluating on text-oriented datasets,
including TextVQA, OCRVQA and STVQA, as demonstrated in Figure 7. Quantitative
results are outlined in Table 2. These results underscore the effectiveness of
our proposed high-resolution model in comprehending intricate textual details
within images.

We also evaluate InfiMM-HD on recently proposed MLLMs evaluation benchmarks,
including MMMU, MM-Vet, InfiMM-Eval, MMB, MME, and POPE. Compared with
previous VQA datasets, these datasets include more comprehensive evaluation
aspects of MLLMs, requiring more complex reasoning capabilities. Evaluation
results are outlined in Table 4. It is noteworthy that no single model excels
across all benchmarks, with each model exhibiting its unique strengths and
limitations. Our proposed model demonstrates commendable overall performance,
highlighting its adaptability and competence across diverse disciplines.

Table 4: Results obtained from benchmarks intricately designed for MLLMs with
heightened complexity.

Model POPE MMEP MMEC MMB MM-VET InfiMM-Eval MMMU (val) BLIP-2 (Li et al.,
2023b) 85.3 1293.8 - - 22.4 - - Shikra (Chen et al., 2023) - - - 58.8 - - -
LLaVA 1.5 (Liu et al., 2023a) 85.9 1531.3 295.4 67.7 35.4 32.62 36.4 Qwen-VL-
Chat (Bai et al., 2023) - 1487.5 360.7 60.6 - 37.39 35.9 Sphinx-2K (Lin et
al., 2023) 87.2 1470.6 326.8 65.9 40.2 - 32.9 Ours 87.9 1472.3 329.4 71.6 38.9
37.42 37.6

Table 5: Evaluation Results for models trained with different input
resolutions. Here dynamic means the model supports resolution ranging from
$448\times 448$ to $1344\times 1344$. During inference, we don’t limit
resolution to 1344.

Resolution GQA VQAv2 OCRVQA DocVQA TextVQA $224\times 224$ 60.7 78.7 57.6 25.6
50.0 $448\times 448$ 61.3 80.5 58.7 44.9 64.1 dynamic 63.5 82.0 66.0 55.1 70.7

Table 6: Ablation study Results for position embedding. w/o PE means removing
the positional embedding.

Resolution GQA VQAv2 OCRVQA DocVQA TextVQA dynamic (w/o PE) 63.3 81.6 65.4
53.0 70.3 dynamic 63.5 82.0 66.0 55.1 70.7

Table 7: Evaluation Results for models trained with different resolution. PC
here means the model has perceriver resampler.

Configuration GQA VQAv2 DocVQA TextVQA $224\times 224$ 60.7 78.7 25.6 50.0
$224\times 224$ (PC) 57.7 79.0 25.2 48.9 $448\times 448$ 61.3 80.5 44.9 64.1
$448\times 448$ (PC) 56.9 79.5 30.7 56.0

###  4.3 Ablation Study

To elucidate the direct impact of input image resolution on model performance,
we conducted an ablation study. In this investigation, different resolution
inputs underwent an identical set of four training stages. To ensure
experimental equitability, we conduct the same model training without DRA by
two epochs on the multi-task datasets. The results are presented in Table 5.
An observable pattern suggests that elevating the input images’ resolution
boosts the efficacy of the model, especially in tasks necessitating an
understanding of textual nuances, such as TextVQA (Singh et al., 2019) and
DocVQA (Mathew et al., 2021). In contrast, for general VQA tasks, a coarse
scene overview is often sufficient for accurate question-answering,
underscoring the context-dependent influence of resolution on model efficacy.

Intuitively, as we cropped the input image into subimages, to maintain the
spatial information, it is important adding a position embedding for each
subimage. To figure out its impact, we carried out ablation study on 2D
position embedding, with the results listed in Table 6. The findings suggest
that removing the position embedding slightly influences model performance.
But on DocVQA, it faces apparently degradation. This phenomenon may be
attributed to the fact that DocVQA predominantly comprises documents, where
the correspondence between various components holds significance, directly
reflected through spatial information.

In our model, the perceiver resampler is removed compared with the origin
Flamingo. To figure out its impact, we investigated the significance of the
perceiver resampler with ablation study. A comparison with models
incorporating the perceiver resampler is presented in Table 7. As the table
indicates, the perceiver resampler would become an information bottleneck,
constraining the model’s performance improvement with increased resolution.

###  4.4 Limitations

This study introduces an enhancement to MLLMs to effectively process high-
resolution images. Results marks significant advancements in various
dimensions. Despite these achievements, certain limitations persist. In
practice, the model exhibits deficiencies in tasks oriented towards text
comprehension. Our ongoing efforts involve exploring more effective modal
alignment strategies while augmenting the dataset to enhance overall model
performance.

##  5 Conclusions

In this work, we present InfiMM-HD, an improvement over Flamingo-style MLLM
designed for processing high-resolution input images. Our approach leverages a
cross-attention mechanism to seamlessly integrate visual information with
language model in a low-dimensional space. To address the formidable
computational demands associated with high-resolution images, we partition the
input high-resolution image into smaller sub-images, each subjected to
individual processing using a shared Vision Transformer (ViT) specifically
tailored for relatively lower resolutions. Additionally, we establish a four-
stage training pipeline to construct the proposed model, ensuring low
computational costs are incurred. The proposed model is thus characterized by
its comprehensive design and an emphasis on minimizing computational
resources.

##  6 Broader Impact

Our model, despite its capabilities, may encounter challenges, including the
generation of inaccurate information and susceptibility to perceptual
illusions. Furthermore, akin to many machine learning models, it may manifest
biases influenced by underlying value systems. Recognizing these potential
issues is crucial for ensuring the responsible and ethical deployment of such
technologies.

## References

  * Alayrac et al. (2022) Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Aminabadi et al. (2022) Aminabadi, R. Y., Rajbhandari, S., Zhang, M., Awan, A. A., Li, C., Li, D., Zheng, E., Rasley, J., Smith, S., Ruwase, O., and He, Y.  Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale, 2022.
  * Antol et al. (2015) Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., and Parikh, D.  Vqa: Visual question answering.  In _International Conference on Computer Vision (ICCV)_ , 2015.
  * Bai et al. (2023) Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J.  Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023.
  * Bavishi et al. (2023) Bavishi, R., Elsen, E., Hawthorne, C., Nye, M., Odena, A., Somani, A., and Taşırlar, S.  Introducing our multimodal models, 2023.  URL https://www.adept.ai/blog/fuyu-8b.
  * Biten et al. (2019) Biten, A. F., Tito, R., Mafla, A., Gomez, L., Rusinol, M., Valveny, E., Jawahar, C., and Karatzas, D.  Scene text visual question answering.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 4291–4301, 2019.
  * Byeon et al. (2022) Byeon, M., Park, B., Kim, H., Lee, S., Baek, W., and Kim, S.  Coyo-700m: Image-text pair dataset.  https://github.com/kakaobrain/coyo-dataset, 2022.
  * Cha et al. (2023) Cha, J., Kang, W., Mun, J., and Roh, B.  Honeybee: Locality-enhanced projector for multimodal llm, 2023.
  * Chen et al. (2023) Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., and Zhao, R.  Shikra: Unleashing multimodal llm’s referential dialogue magic, 2023.
  * Chen et al. (2015) Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., and Zitnick, C. L.  Microsoft coco captions: Data collection and evaluation server, 2015.
  * Chiang et al. (2023) Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.  URL https://lmsys.org/blog/2023-03-30-vicuna/.
  * Dosovitskiy et al. (2021) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N.  An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
  * Driess et al. (2023) Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., Huang, W., Chebotar, Y., Sermanet, P., Duckworth, D., Levine, S., Vanhoucke, V., Hausman, K., Toussaint, M., Greff, K., Zeng, A., Mordatch, I., and Florence, P.  Palm-e: An embodied multimodal language model, 2023.
  * Feng et al. (2023) Feng, H., Liu, Q., Liu, H., Zhou, W., Li, H., and Huang, C.  Docpedia: Unleashing the power of large multimodal model in the frequency domain for versatile document understanding, 2023.
  * Fu et al. (2023) Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., Wu, Y., and Ji, R.  Mme: A comprehensive evaluation benchmark for multimodal large language models, 2023.
  * Gu et al. (2022) Gu, J., Kuen, J., Morariu, V. I., Zhao, H., Barmpalios, N., Jain, R., Nenkova, A., and Sun, T.  Unified pretraining framework for document understanding, 2022.
  * Gurari et al. (2018) Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., and Bigham, J. P.  Vizwiz grand challenge: Answering visual questions from blind people.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 3608–3617, 2018.
  * Gurari et al. (2020) Gurari, D., Zhao, Y., Zhang, M., and Bhattacharya, N.  Captioning images taken by people who are blind.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVII 16_ , pp. 417–434. Springer, 2020.
  * Han et al. (2023) Han, X., You, Q., Liu, Y., Chen, W., Zheng, H., Mrini, K., Lin, X., Wang, Y., Zhai, B., Yuan, J., Wang, H., and Yang, H.  InfiMM-Eval: Complex Open-Ended Reasoning Evaluation For Multi-Modal Large Language Models.  _arXiv e-prints_ , art. arXiv:2311.11567, November 2023.  doi: 10.48550/arXiv.2311.11567.
  * Han et al. (2023) Han, X., You, Q., Liu, Y., Chen, W., Zheng, H., Mrini, K., Lin, X., Wang, Y., Zhai, B., Yuan, J., Wang, H., and Yang, H.  Infimm-eval: Complex open-ended reasoning evaluation for multi-modal large language models, 2023.
  * Han et al. (2024) Han, X., Wang, Y., Zhai, B., You, Q., and Yang, H.  Coco is ”all” you need for visual instruction fine-tuning, 2024.
  * He et al. (2021) He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R.  Masked autoencoders are scalable vision learners, 2021.
  * Hudson & Manning (2019) Hudson, D. A. and Manning, C. D.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 6700–6709, 2019.
  * Jiang et al. (2023) Jiang, D., Liu, Y., Liu, S., Zhang, X., Li, J., Xiong, H., and Tian, Q.  From clip to dino: Visual encoders shout in multi-modal large language models, 2023.
  * Krishna et al. (2017) Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _International journal of computer vision_ , 123:32–73, 2017.
  * Laurençon et al. (2023) Laurençon, H., Saulnier, L., Tronchon, L., Bekman, S., Singh, A., Lozhkov, A., Wang, T., Karamcheti, S., Rush, A. M., Kiela, D., Cord, M., and Sanh, V.  Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023.
  * Li et al. (2023a) Li, B., Zhang, P., Yang, J., Zhang, Y., Pu, F., and Liu, Z.  Otterhd: A high-resolution multi-modality model, 2023a.
  * Li et al. (2023b) Li, J., Li, D., Savarese, S., and Hoi, S.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023b.
  * Li et al. (2023c) Li, Z., Yang, B., Liu, Q., Ma, Z., Zhang, S., Yang, J., Sun, Y., Liu, Y., and Bai, X.  Monkey: Image resolution and text label are important things for large multi-modal models, 2023c.
  * Lin et al. (2023) Lin, Z., Liu, C., Zhang, R., Gao, P., Qiu, L., Xiao, H., Qiu, H., Lin, C., Shao, W., Chen, K., Han, J., Huang, S., Zhang, Y., He, X., Li, H., and Qiao, Y.  Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models, 2023.
  * Liu et al. (2023a) Liu, H., Li, C., Li, Y., and Lee, Y. J.  Improved baselines with visual instruction tuning, 2023a.
  * Liu et al. (2023b) Liu, H., Li, C., Wu, Q., and Lee, Y. J.  Visual instruction tuning, 2023b.
  * Liu et al. (2023c) Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., Chen, K., and Lin, D.  Mmbench: Is your multi-modal model an all-around player?, 2023c.
  * Lu et al. (2022) Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A.  Learn to explain: Multimodal reasoning via thought chains for science question answering.  In _The 36th Conference on Neural Information Processing Systems (NeurIPS)_ , 2022.
  * Marino et al. (2019) Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R.  Ok-vqa: A visual question answering benchmark requiring external knowledge.  In _Proceedings of the IEEE/cvf conference on computer vision and pattern recognition_ , pp. 3195–3204, 2019.
  * Mathew et al. (2021) Mathew, M., Karatzas, D., and Jawahar, C.  Docvqa: A dataset for vqa on document images.  In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_ , pp. 2200–2209, 2021.
  * Mishra et al. (2019) Mishra, A., Shekhar, S., Singh, A. K., and Chakraborty, A.  Ocr-vqa: Visual question answering by reading text in images.  In _2019 international conference on document analysis and recognition (ICDAR)_ , pp. 947–952. IEEE, 2019.
  * Radford et al. (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I.  Learning transferable visual models from natural language supervision, 2021.
  * (39) Schuhmann, C., Köpf, A., Vencu, R., Coombes, T., and Beaumont, R.  Laion coco: 600m synthetic captions from laion2b-en.  https://laion.ai/blog/laion-coco/.
  * Schwenk et al. (2022) Schwenk, D., Khandelwal, A., Clark, C., Marino, K., and Mottaghi, R.  A-okvqa: A benchmark for visual question answering using world knowledge.  In _European Conference on Computer Vision_ , pp. 146–162. Springer, 2022.
  * Sidorov et al. (2020) Sidorov, O., Hu, R., Rohrbach, M., and Singh, A.  Textcaps: a dataset for image captioning with reading comprehension.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16_ , pp. 742–758. Springer, 2020.
  * Singh et al. (2019) Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M.  Towards vqa models that can read.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 8317–8326, 2019.
  * Sun et al. (2023) Sun, Q., Fang, Y., Wu, L., Wang, X., and Cao, Y.  Eva-clip: Improved training techniques for clip at scale, 2023.
  * Tong et al. (2024) Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., and Xie, S.  Eyes wide shut? exploring the visual shortcomings of multimodal llms, 2024.
  * Wang et al. (2023) Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., Song, X., Xu, J., Xu, B., Li, J., Dong, Y., Ding, M., and Tang, J.  Cogvlm: Visual expert for pretrained language models, 2023.
  * Wang et al. (2024) Wang, Y., Chen, W., Han, X., Lin, X., Zhao, H., Liu, Y., Zhai, B., Yuan, J., You, Q., and Yang, H.  Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasoning, 2024.
  * Wang & Liu (2021) Wang, Z. and Liu, J.-C.  Translating math formula images to latex sequences using deep neural networks with sequence-level training.  _International Journal on Document Analysis and Recognition (IJDAR)_ , 24(1-2):63–75, 2021.
  * Yang et al. (2023) Yang, Z., Li, L., Lin, K., Wang, J., Lin, C.-C., Liu, Z., and Wang, L.  The dawn of lmms: Preliminary explorations with gpt-4v(ision), 2023.
  * Ye et al. (2023) Ye, Q., Xu, H., Ye, J., Yan, M., Hu, A., Liu, H., Qian, Q., Zhang, J., Huang, F., and Zhou, J.  mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration, 2023.
  * Yu et al. (2023) Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., and Wang, L.  Mm-vet: Evaluating large multimodal models for integrated capabilities, 2023.
  * Yue et al. (2023) Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang, W., Sun, H., Su, Y., and Chen, W.  Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi, 2023.
  * Zhai et al. (2023) Zhai, B., Yang, S., Xu, C., Shen, S., Keutzer, K., and Li, M.  Halle-switch: Controlling object hallucination in large vision language models, 2023.
  * Zhang et al. (2023) Zhang, Y., Zhang, R., Gu, J., Zhou, Y., Lipka, N., Yang, D., and Sun, T.  Llavar: Enhanced visual instruction tuning for text-rich image understanding, 2023.
  * Zhu et al. (2023a) Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M.  Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023a.
  * Zhu et al. (2023b) Zhu, W., Hessel, J., Awadalla, A., Gadre, S. Y., Dodge, J., Fang, A., Yu, Y., Schmidt, L., Wang, W. Y., and Choi, Y.  Multimodal C4: An open, billion-scale corpus of images interleaved with text.  _arXiv preprint arXiv:2304.06939_ , 2023b.

##  Appendix A Training Configuration

Figure 7: Example of a TextVQA sample, exemplifying its intricate scrutiny of
nuanced visual perception.

We report the detailed training hyper-parameters and settings of InfiMM-HD in
Table 8. Our model was trained with 64 NVIDIA A100-SXM-80GB. The pretraining
stage cost about 72 hours. (If only use the image-text pair, it costs 30
hours.) The remaining three stages cost less than 21 hours.

Table 8: Details of the training Configuration. For the second stage, we
utilize bilinear interpolation to extend the origin ViT to support resolution
448. IT means image text pair. And IIT means interleaved image text sequence.

Configuration Pretraining Continue Pretraining Dynamic Resolution Adaption
Instruction Finetuning ViT init. EVA2-CLIP2-E (res 224) EVA2-CLIP2-E (res 448)
ViT from 2nd-stage ViT from 3rd-stage LLM init. Vicuna-13b Vicuna-13b
Vicuna-13b Vicuna-13b Gated cross-attention init. random InfiMM-HD 1st stage
InfiMM-HD 2nd stage InfiMM-HD 3rd stage Image resolution 224 448
dynamic(448-1344) dynamic(448-1344) ViT sequence length 257 1024 1025 1025 LLM
sequence length 32 (IT);384 (IIT) 128 128 1024 Optimizer AdamW Optimizer
hyperparameter $\beta_{1}=0.9,\beta_{2}=0.95,eps=1e^{-8}$
$\beta_{1}=0.9,\beta_{2}=0.999,eps=1e^{-5}$ Peak learning rate $1e^{-4}$
$1e^{-5}$ $1e^{-5}$ $5e^{-6}$ Minimum learning rate $1e^{-4}$ $1e^{-6}$
$1e^{-6}$ $5e^{-7}$ Learning rate schedule cosine decay Weight decay 0.1
Gradient clip 1.0 Training steps 120k 8k 8k 11k warm steps 6k 400 400 550
Global batch size 10240 (IT);768 (IIT) 256 256 64 Gradient accumulation steps
2 1 1 2 Gradient ACC. 2 1 1 2 Numerical precision bfloat16 Gradient
checkpointing $\times$ $\times$ $\checkmark$ $\times$ Deepspeed Zero Stage 2
Training resource 64 NVIDIA A100-SXM-80GB

##  Appendix B Summary of Evaluation Benchmarks

We provided a detailed summary of evaluation benchmarks we used and their
corresponding metrics in Table 9. Note that ANLS means Average Normalized
Levenshtein Similarity.

Table 9: Details on the test dataset.

Task Dataset Description Split Metric General VQA VQAV2 VQA on natural images
test-dev VQA Score($\uparrow$) OKVQA VQA on natural images but need world
knowledge val VQA Score($\uparrow$) IconQA Abstract diagram understanding and
visual language reasoning test EM($\uparrow$) GQA VQA on scene understandig
and reasoning test-dev EM($\uparrow$) ScienceQA Multimodal multi choice VQA on
science filed test Accuracy($\uparrow$) Text-oriented VQA TextVQA VQA about
text in natural scene val VQA Score($\uparrow$) OCRVQA VQA on images of book
covers val EM($\uparrow$) STVQA VQA covering reading and reasoning about text
test ANLS($\uparrow$) DocVQA VQA on images from documents test
ANLS($\uparrow$) Other Benchmarks MME Evalutaion for MLLM on perception and
cognition Perception and Cognition Accuracy($\uparrow$) MM-VET Dialog style
VQA on integrated ability test GPT-4 score($\uparrow$) MMbench Comprehensive
evalutaion with multi choice VQA test Accuracy($\uparrow$) score($\uparrow$)
POPE Object hallucination in MLLM adversial F1 score($\uparrow$) InfiMM
Complex Open-ended Reasoning test GPT-4 score($\uparrow$) MMMU College-level
multi choice VQA val Accuracy($\uparrow$)


# Fractional Concepts in Neural Networks: Enhancing Activation and Loss
Functions

Zahra Alijani
Institute for Research and Applications of Fuzzy Modeling
University of Ostrava
30\. dubna 22, 701 03 Ostrava
<EMAIL_ADDRESS>
& Vojtech Molek
Institute for Research and Applications of Fuzzy Modeling
University of Ostrava
30\. dubna 22, 701 03 Ostrava
<EMAIL_ADDRESS>
###### Abstract

The paper presents a method for using fractional concepts in a neural network
to modify the activation and loss functions. The methodology allows the neural
network to define and optimize its activation functions by determining the
fractional derivative order of the training process as an additional
hyperparameter. This will enable neurons in the network to adjust their
activation functions to match input data better and reduce output errors,
potentially improving the network’s overall performance.

Figure 1: Fractional derivative of Mish and sigmoid activation functions.

##  1 Introduction

Fractional derivatives have been studied in the context of activation
functions [21, 20], which can morph activation functions from one to another
in a continuous manner. The activation function is a fundamental element
within a neural network, introducing essential nonlinearity required to model
complex relations in machine learning tasks, including classification and
regression.

In this paper, we emphasize the importance of selecting appropriate activation
and loss functions. Over the years, various neural network architectures have
surfaced, each accompanied by its distinctive set of activation
functions—ranging from the sigmoid, radial basis function (RBF), ReLU [5],
Softplus [4], swish [16], Mish [14], and many more. However, the current
practice predominantly relies on manual or heuristic selection of activation
functions, often leading to an exhaustive trial-and-error methodology and
frequent retraining of networks to find the optimal configuration.

Fractional calculus, on the other hand, opens doors to entire families of
functions by naturally extending the original function through fractional
derivatives. Here, the user only needs to specify the original function, while
the fractional derivative is automatically adjusted. This elegant approach
holds particularly true for activation functions. In contrast, for loss
functions, the fractional derivative requires manual configuration.

However, the application of fractional activation functions comes with its
unique set of challenges. One prominent challenge is the potential increase in
computational complexity associated with fractional derivatives, which may
impact the efficient implementation and execution of neural networks.
Additionally, fractional activation functions demand careful tuning of
specific hyperparameters, which can be intricate and time-consuming.
Furthermore, the research on fractional activation functions remains
relatively limited, leaving the practical effectiveness of these functions
still an open question.

All of our code is publicly available111https://gitlab.com/irafm-
ai/frac_calculus_01.

##  2 Fractional Calculus

Fractional derivatives and integrals are used in neural networks to define a
numerical trainable parameter value that automatically selects the optimal
activation function for a particular task. This approach can be helpful when
it is difficult to manually select the best activation function, as fractional
derivatives and integrals allow for a more flexible and adaptive selection
process. Furthermore, using fractional calculus can also improve the
performance of neural networks by introducing more complex and nuanced
mathematical models.

###  2.1 Fractional Derivatives

Fractional calculus is a powerful mathematical tool to model various complex
engineering and real-world systems. The three popular fundamental definitions
of fractional derivatives are [7] :

  * •

Grünwald-Letnikov [15] derivative of fractional order $a\in\mathcal{R}^{+}$ :
It is defined as

$D^{a}f(t)=\lim_{h\to
0}\frac{1}{h^{a}}\sum_{n=0}^{[\frac{t-a}{h}]}(-1)^{n}C_{n,a}f(t-nh),$ (1)

where, $[x]$ is the integer part of $x$ and $C_{n,a}$ is the binomial
coefficient.

  * •

Riemann–Liouville derivative of fractional order $a\in\mathcal{R}^{+}$: It is
defined as

$D^{a}f(t)=\frac{1}{\Gamma(n-a)}\frac{d^{n}}{dt^{n}}\int_{a}^{t}\frac{f(\mu)}{(t-\mu)^{a-n+1}}d\mu,$
(2)

for $n-1<a<n,n\in\mathcal{Z}^{+}$ and $\Gamma(.)$ is the Gamma function (we
will recall it shortly).

  * •

Caputo derivative of fractional order $a\in\mathcal{R}^{+}$: It is defined as

$D^{a}f(t)=\frac{1}{\Gamma(n-a)}\int_{a}^{t}\frac{f^{n}(\mu)}{(t-\mu)^{a-n+1}}d\mu,$
(3)

for $n-1<a<n,n\in\mathcal{Z}^{+}$ where $f^{n}(\mu)$ is the $n$-th order
derivative of the function $f(t)$.

It should be noted that for an initially relaxed function, all three
definitions coincide. The Caputo definition, though more restrictive, is
widely employed, as it allows the use of physical initial conditions.

The Gamma function, denoted by $\Gamma(z)$, is a generalization of the
factorial operator and is used to define the fractional derivative in
fractional calculus. The Gamma function is defined as [1]:

$\Gamma(z)=\int_{0}^{\infty}t^{(z-1)}e^{-t}dt.$ (4)

The Gamma function is defined for non-negative integers as $\Gamma(n)=(n-1)!$,
and for other non-negative values of $z$ it can be computed by [1]:

$\Gamma(z)=\frac{e^{-\gamma
z}}{z}\prod_{k=1}^{\infty}((1+\frac{z}{k})^{-1}e^{\frac{z}{k}}).$ (5)

Here, $\gamma$ is the Euler-Mascheroni constant $(\gamma=0.57721..)$. The
fractional derivative, represented above, can be modified by replacing the
factorial with the gamma function. The definition provided in the following
statement represents the fractional derivative of the function $f(x)=x^{k}$
for $k,x\geq 0$.

$D^{a}x^{k}=\frac{\Gamma(k+1)}{\Gamma(k+1-a)}x^{k-a}.$ (6)

The Gamma function allows for the definition of the fractional derivative for
non-integer values of $k$, while the factorial is only defined for integers.
This allows for a more general and flexible formulation of the fractional
derivative. In machine learning, fractional derivatives and integrals have
been used in various ways. For example, they have been used to design
activation functions, as discussed in [21]. They will also be used in the
design of loss functions, where they can capture more complex patterns in
data.

##  3 Grouping Activation Functions by Using Fractional Calculus

It is possible to group certain activation functions into families using
fractional calculus. By analyzing the fractional derivative of an activation
function, it is possible to mathematically generate other activation functions
that belong to the same family.

It is worth noting that not all activation functions can be grouped into
families using fractional calculus. The step function is an example of an
activation function that is computationally efficient but is a poor choice for
an activation function due to discontinuity and flat derivations. Other
available activation functions, such as sigmoid, ReLU, and tanh, each with its
own strengths and weaknesses, provide detailed information on the various
options [19].

It also depends on the type of neural network and the problem you are trying
to solve. It is also important to note that fractional calculus is a
relatively new area of research and is still being fully understood. Its use
in neural networks is still experimental and not widely used.

Calculating the fractional derivative of some activation functions is
challenging and may not have a simple closed-form expression. However, there
are approximation methods, i.e., the Grünwald-Letnikov [15] or other
specialized techniques, that are generally used to calculate fractional
derivatives. These methods involve approximating the fractional derivative
using finite differences or numerical integration.

###  3.1 Fractional Mish

Mish is mathematically defined as:

$f(x)=x\tanh(\ln(1+e^{x})))=x.\frac{(e^{x}+1)^{2}-1}{(e^{x}+1)^{2}+1}.$

Some traditional activation functions, such as sigmoid and tanh, suffer from
the saturation problem, where the gradients become very small for significant
inputs. The Mish activation (and its fractional version) helps mitigate this
issue by having bounded angles for substantial positive and negative inputs,
leading to more stable training. The Mish activation has been reported to
perform well in particular machine learning tasks, such as image
classification. The fractional Mish can be computed as:

$D^{a}f(x)=\lim_{h\to
0}\frac{1}{h^{a}}\sum_{n=0}^{\infty}(-1)^{n}\frac{\Gamma(a+1)(x-nh)}{\Gamma(n+1)\Gamma(1-n+a)}.\frac{(e^{x-nh}+1)^{2}-1)}{((e^{x-nh}+1)^{2}+1)}.$
(7)

The Mish activation function combines the linear and non-linear properties of
the identity and hyperbolic tangent functions, respectively. It has been shown
to improve the training performance of deep neural networks in some cases.

###  3.2 Fractional sigmoid

The sigmoid activation function takes an input value and maps it into a value
between 0 and 1. One characteristic of the sigmoid function is its tendency to
saturate as the input becomes large or small. The saturation behavior leads to
the vanishing gradient problem, where the gradients approach zero during
backpropagation, hindering the learning process. The fractional sigmoid
activation function can be implemented using the soft plus function or
directly by applying the fractional derivative to the sigmoid function. The
sigmoid function is defined as:

$f(x)=\frac{1}{1+e^{-x}}$

and the fractional sigmoid is defined as:

$D^{a}f(x)=\lim_{h\to
0}\frac{1}{h^{a}}\sum_{n=0}^{\infty}(-1)^{n}\frac{\Gamma(a+1)}{\Gamma(n+1)\Gamma(1-n+a)(1+e^{-x+nh})}$
(8)

where $a$ is the fractional derivative order. By adjusting the value of $a$,
we can control the shape and smoothness of the activation function.

Sigmoid is an outdated activation function primarily used in the last layer of
networks to squeeze an output into $[0,1]$ interval. Fractionality makes
sigmoid usable and alleviates gradient vanishing, albeit it does not remove it
completely.

##  4 Grouping Loss Functions by Using Fractional Calculus

The term ”fractional” suggests that these loss functions may involve
fractional exponents or weights, which can be used to fine-tune the
contribution of individual data points or classes. Fractional loss functions
can have a more complex form than traditional loss functions and may combine
various components to achieve specific objectives. Here is an example of a
loss function that utilizes fractional calculus in a time series prediction
problem:

$L(y,\hat{y})=\int_{t_{1}}^{t_{2}}w(t)D^{a}(y(t)-\hat{y}(t))dt$

In this equation, $y(t)$ represents the actual label or target value,
$\hat{y}(t)$ represents the predicted value by the model, $a$ is the
fractional derivative order, $D^{a}$ represents the fractional derivative
operator, and $w(t)$ is a weight function that assigns weights to different
time points or data points. The integral is taken over the interval
$[t_{1},t_{2}]$, which means the specified time interval. It is important to
note that the specific choice and formulation of the loss function depends on
the problem at hand and the desired properties or characteristics that need to
be incorporated. Despite having a trainable parameter in the calculation of
the fractional activation function, for the loss function, $a$ is a non-
trainable parameter.

###  4.1 Fractional Mean Squared Error

Machine learning aims to find a model that can accurately predict the output
given the input data. This is typically done by minimizing a loss function,
which measures the difference between expected and actual output. The model
parameters are then adjusted to reduce the loss function. A standard loss
function used in many machine learning algorithms is the mean squared error
(MSE), which is defined as:

$\text{MSE}(y,\hat{y})=\frac{1}{n}\sum(y-\hat{y})^{2}.$ (9)

$y$ is the proper output, $\hat{y}$ is the predicted output, and n is the
number of samples. The goal is to minimize the MSE by adjusting the model
parameters. One example of a loss function with a fractional derivative is the
fractional mean squared error (FMSE), which is defined as:

$\text{FMSE}(y,\hat{y},a)=\frac{1}{n}\sum\big{(}(y-\hat{y})^{2})^{\frac{a}{2}},$
(10)

where $a$ is the order of the fractional derivative. When $a=1$, the FMSE is
equivalent to the MSE. However, when $a$ is a non-integer value, the FMSE
captures more complex relationships between the input and output.

###  4.2 Fractional Cross-Entropy Loss

The fractional cross-entropy loss is a loss function with a fractional
derivative that can be used in classification problems to improve the model’s
accuracy. The fractional cross-entropy loss function is defined as:

$\text{FCrossEntropy}(y,\hat{y},a)=-\frac{1}{n}\sum(ya\log(\hat{y})+(1-y)a\log(1-\hat{y}))$
(11)

One potential limitation of using loss functions with fractional derivatives
is that they may require more computational resources and may be more
challenging to optimize than standard loss functions. Additionally, the choice
of the order of the fractional derivative may not be evident in all cases and
may require some trial and error.

###  4.3 Fractional Huber Loss

The fractional Huber loss is a modified version of the Huber loss [8] that
incorporates a fractional derivative to capture more complex relationships
between variables. Huber loss is a robust function combining squared error for
minor errors and absolute errors for significant ones. By adjusting the ”a”
value, the fractional Huber loss can adapt to different levels of non-
linearity and robustness in the data. Additionally, the delta parameter allows
control over the point at which the loss function transitions from quadratic
to linear. The fractional Huber loss is particularly useful when the data
contain outliers or extreme values, as it provides a robust alternative to the
standard squared error loss. A fractional version of the Huber loss can be
defined as:

$\text{FHuberLoss}(y,\hat{y},a)=\left\\{{\begin{array}[]{l}\frac{1}{2}((y-\hat{y})^{2})^{\frac{a}{2}},\hskip
14.22636pt\text{if}\hskip 14.22636pt|y-\hat{y}|\leq\delta,\\\
\delta(|y-\hat{y}|)^{\frac{a}{2}}-\delta^{2}{(\frac{1-a}{2})},\hskip
14.22636pt\text{otherwise}.\end{array}}\right.$ (12)

Here $n$ is the number of samples, a is the order of the fractional
derivative, and $\delta$ is a threshold determining the point at which the
loss transitions from quadratic to linear.

##  5 Activation function results

In our experimental study, we evaluated the performance of three activation
functions: Mish, swish, and sigmoid, on the CIFAR-10 dataset [10]. These
evaluations were conducted across four distinct network architectures: a
simple CNN, ResNet 18 and 50 [6], and an architecture resembling
EfficientNetB0 [18]. For detailed insights into the architecture specifics, we
encourage readers to refer to our repository. We should note that we utilized
the fractional swish implementation in [20], but, despite our efforts, we
could not surpass the performance of the original swish activation function.

#### Setup:

We use the common CIFAR-10 setting. SGD optimizer with a five epochs warm-up
and decreasing learning rate at 30%, 60%, and 80% milestones. Data are fed to
the network in batches of 128 images augmented with padded random cropping,
horizontal flipping, and normalization. The network is trained for 200 epochs
using a cross-entropy with label smoothing. There is an exception: ResNet-50
fractional Mish is trained with 96 batch size due to memory consumption. We
track the best test accuracy and the fractional derivative order of each
activation function during training. All activation functions have
hyperparameters $h=1$ and $\sum_{n=0}^{3}$ (see function definitions). Sec. 7
discusses these two constants in further detail.

Act fc | Simple CNN | ResNet-18 | ResNet-50 | EffNet
---|---|---|---|---
[%] | [%] | [%] |
ReLU | 75.97 | 95.38 | 95.42 | 94.40
Sig | 65.69 | 82.01 | 56.49 | 84.16
Frac sig | 10.00 | 94.39 | 90.77 | 92.21
swish | 77.53 | 94.31 | 94.62 | 93.92
Frac swish | 77.40 | 93.42 | 93.82 | 92.97
Mish | 77.67 | 94.22 | 94.90 | 93.55
Frac Mish | 77.46 | 94.07 | 94.06 | 93.34
Table 1: CIFAR-10 test accuracies of different model/activation function
combinations.

Table 1 displays the CIFAR-10 test accuracies for various combinations of
models and activation functions. The results for Fractional Mish and Swish are
somewhat underwhelming, as they do not exhibit a significant improvement over
the original activation functions. On the other hand, the Fractional Sigmoid
outperforms the original function notably, except in the case of the simple
CNN model, where the entire model fails to converge completely. It’s
noteworthy that Sigmoid performs significantly worse with ResNet-50 compared
to both ResNet-18 and EfficientNetB0. To ensure that this behavior is not
solely due to seeding issues, we conducted an additional run without seeding
or deterministic algorithms, resulting in a final accuracy of 43.44%. One
plausible explanation for this discrepancy is the vanishing gradient issue,
particularly as network depth increases.

Figures 2(a) and 2(b) provide insights into the evolution of fractional
derivative orders for Fractional Mish and Sigmoid. Notably, most fractional
derivative orders for these activation functions tend to converge toward
similar values. In the case of Fractional Sigmoid, convergence is observed
toward its second derivative, whereas Fractional Mish remains closely aligned
with the original function.

(a) Fractional sigmoid.

(b) Fractional Mish.

Figure 2: Evolution of fractional derivative order during the training of
ResNet-18 on CIFAR-10.

##  6 Loss function results

In this set of experiments, we explore the effectiveness of three distinct
loss functions: fractional cross-entropy, fractional mean squared error (MSE),
and fractional Huber loss. Fractional cross-entropy is evaluated on
CIFAR-100LT [2] and ImageNet [17], while fractional MSE and fractional Huber
loss are tested using the UTKFace regression dataset [22].

#### Fractional cross-entropy and CIFAR-100LT:

We use Adam optimizer with a decreasing learning rate at 60%, 80%, and 92.5%
milestones. Data are fed to the network in batches of 50 images augmented with
translation, horizontal flipping, and normalization. The network is trained
for 200 epochs. Learning rates vary between experiments because fractionality
adds a power term to the cross-entropy equation. In Fig. 3(a), we present a
visual representation of the impact of fractional derivative order and
learning rate on the final test accuracy. Lower learning rates tend to benefit
from fractional derivative orders less than 1, whereas higher learning rates
exhibit improved performance with fractional derivative orders greater than 1.

#### Fractional cross-entropy and ImageNet:

We use Adam optimizer with a decreasing learning rate at 33% and 66%
milestones. Data are fed to the network in batches of 256 images augmented
with random resized cropping, horizontal flipping, and normalization. The
network is trained for 90 epochs. As illustrated in Fig. 3(b), the model’s
accuracy with a fixed learning rate of 2e-3 exhibited minimal changes and
peaked at 0.6.

(a) ResNet-18 trained on CIFAR-100LT with different fractional derivative
order.

(b) ResNet-50 trained on ImageNet with different fractional derivative order

Figure 3: Effect of the fractional derivative order of cross-entropy and
learning rate on test accuracy.

#### Fractional Huber/MSE loss and UTKFace:

We use the same experiment setting as CIFAR-10 in sec. 5 except for using Lion
optimizer. We chose Lion over SGD due to its performance in this particular
experiment. We run training with different fractional derivative orders. Table
2 summarizes the lowest losses achieved across different network
architectures. It’s important to note that, as the loss function itself is
being tuned, we did not employ it for reporting test loss values. Instead, we
utilized the L1 loss (Mean Absolute Error - MAE) to assess the test
performance. The fractional loss function served the sole purpose of training
during this experiment.

Loss function | Simple CNN | ResNet-18
---|---|---
MSE | $8.702$ | $6.142$
Fractional MSE |  $8.319$ (1.6) |  $5.982$ (0.9)
Huber loss | $8.678$ | $6.046$
Fractional Huber loss |  $8.174$ (3.4) |  $6.042$ (1)
Table 2: UTKFace test MAE of Baseline MSE and Huber loss and their fractional
variants. The numbers in parentheses are fractional derivative orders.

##  7 Practical challenges

As we transition from the theoretical framework of fractional calculus to
practical implementation, we encounter several specific challenges, which
we’ll address in this section.

#### Computational hyperparameters:

One of the key challenges lies in setting appropriate computational
hyperparameters, specifically the value of $h$ and the number of sigma
iterations. To illustrate this, let’s begin with a fundamental derivative
theorem:

$L=\lim_{h\to 0}\frac{f(b+h)-f(b)}{h}$ (13)

and its application to a sigmoid function. We must substitute $lim_{h\to 0}$
with $h$ equal to a scalar value. Fig. 4 shows the effect of increasing $h$.
With increasing $h$, the derivative graph shifts to the right. This is true
for both eq. 13 and eq. 8.

Figure 4: Left: first derivative of sigmoid using eq. 13. Right: first
derivative of sigmoid using fractional derivative eq. 8. The first derivative
of a sigmoid function with variable $h$.

In Eq.8, we have the option to increase the number of sigma iterations to
obtain a more precise approximation (see Fig.5). However, this approach
introduces its own set of challenges. Increasing the number of iterations
results in slower computation, potential issues related to type overflow due
to the factorial function $\Gamma(1+n)=n!$, the occurrence of negative integer
arguments for the gamma function, and problems associated with memory
consumption.

Modern deep learning frameworks, like PyTorch, construct a computational graph
to track values relevant to gradient computations. In this context, each sigma
iteration becomes a part of the computational graph, causing memory
consumption to grow rapidly (see Table 3).

Act fc | Simple CNN | ResNet-18 | ResNet-50 | EffNet
---|---|---|---|---
[MiB] | [MiB] | [MiB] | [MiB]
ReLU | 1 867 | 2 961 | 6 081 | 5 017
Sig | 1 867 | 2 963 | 6 075 | 5 165
Frac sig | 1 897 | 5 623 | 20 503 | 15 775
swish | 1 867 | 3 203 | 7 533 | 6 045
Frac swish | 1 893 | 4 597 | 14 685 | 11 585
Mish | 1 867 | 3 131 | 7 541 | 6 237
Frac Mish | 1 955 | 10 517 | OOM | 26 059
Table 3: Memory consumption of different activation functions/models using
CIFAR-10.

The problem of a negative, integral gamma function argument comes from the
term $\Gamma(1-n+a)$ when $a\in\mathcal{N}^{+}$; therefore, it is impossible
to increase the number of iterations of the sigma beyond a certain point. In
general, $n<1+a$ therefore, we impose the restriction that $a\in[0.1,1.9]$ and
$\sum_{n=0}^{3}$. To maintain the constraint on $a$ within the interval
$[0.1,1.9]$, we employ a leaky piecewise-linear function:

$f(x)=\begin{cases}0.1+(x-0.1)*0.01,&x<=0.1\\\ x,&0.1<x<1.9\\\
1.9+(x-1.9)*0.01,&x>=1.9\end{cases}.$

This formula enables the argument $x$ to move outside the $[0.1,1.9]$ range
when necessary. The slope or leakiness factor of 0.01 allows the fractional
derivative order to recover from deviating outside these bounds.

Figure 5: Different number of sigma iteration and h values. Graphs compare the
sigmoid’s first derivative (blue) and 1.1 derivative (green).

All the problems mentioned above can be relaxed using different definitions
for fractional derivatives, which is possible in particular cases. For
example, fractional derivation of swish function eq. (28) in [20]:
$D^{a}g(x)=g(x)+a\sigma(x)(1-g(x))$.

##  8 Ablation study

#### Optimizers:

One of the aspects we explored is the impact of optimizers on the training
process of fractional derivative activation functions. These functions exhibit
changes in shape during training, and it’s unclear whether stochastic gradient
descent (SGD) allows them to converge to the final fractional derivative order
quickly enough. To test this convergence speed hypothesis, we experimented
with various optimizers, including Lion [3], AdaBound [13], Adam [9], AdamW
[12], and RAdam [11].

SGD | Lion | AdaBound | Adam | AdamW | RAdam
---|---|---|---|---|---
$[\%]$ | [%] | [%] | [%] | [%] | [%]
94.45 | 92.01 | 91.15 | 92.52 | 92.72 | 92.19
Table 4: Test accuracy of ResNet-18 trained on CIFAR-10 using different
optimizing algorithms.

Table 4 displays the test accuracy of ResNet-18 trained on CIFAR-10 using
different optimizing algorithms. We adhered to the optimal values recommended
in the respective optimizer papers and for the experiments’ general settings,
as outlined in Section 5.

The activation functions in question are notably sensitive to the initial
value of the fractional derivative order. To identify the optimal initial
settings for both the fractional sigmoid and fractional Mish, we utilized 20%
of the CIFAR-10 dataset and the Optuna framework for a search.

(a) Fractional Mish.

(b) Fractional sigmoid

Figure 6: CIFAR-10 test accuracy with respect to the initial fraction
derivative of the factional Mish and sigmoid activation functions. Trained on
20% of data.

As shown in Fig.6, these results align with Fig.2. The fractional sigmoid
tends to converge close to the second derivative, while the fractional Mish
converges towards the original function.

#### Block activation functions:

In the ResNet architecture, layers are organized into blocks. We investigated
the impact of using a unique activation function with a fractional derivative
order per layer compared to a shared activation function with a fractional
order per block (meaning the same activation function shape is employed within
a block). To find a difference, we used ResNet18 and trained it on CIFAR-10
with the following result:

| Per layer | Per block
---|---|---
| [%] | [%]
Frac Sig | 94.29 | 94.39
Frac Mish | 94.05 | 94.07
Table 5: Fractional derivative order per layer versus per block.

As Table 5 demonstrates, the difference between employing a fractional
derivative order per layer and per block is insignificant. Consequently, we
have adopted the per block variant for all our experiments.

##  9 Conclusion

This paper introduces a novel technique for integrating fractional concepts
into neural networks, enabling the modification of activation and loss
functions. We leverage fractional derivatives to create families of activation
and loss functions. In this framework, the fractional derivative order becomes
a trainable parameter for activation functions, while it remains non-trainable
for loss functions. The potential for shaping activation functions offers
promise, particularly in situations where conventional functions, such as
sigmoid, prove suboptimal.

However, we have encountered various challenges and obstacles throughout our
exploration. The computation of fractional derivatives using the Grünwald-
Letnikov method results in infinite-series representations, necessitating a
trade-off between approximation accuracy and computational feasibility.
Additionally, alternative calculation variants often involve multiple
variables, making it challenging to integrate fractional derivatives into
neural networks seamlessly.

The experimental results presented in our study underscore the potential
benefits of fractional sigmoid and the fractional loss functions. Fractional
sigmoid exhibits the ability to address some of the limitations encountered by
the conventional sigmoid function. Moreover, the fractional loss functions
consistently outperform their non-fractional counterparts, although they
require the manual selection of fractional derivative orders.

## Acknowlegement

The work of Zahra Alijani was partially supported by project AIMet4AI No.
CZ.02.1.01/0.0/0.0/17_049/0008414.

## References

  * [1] Milton Abramowitz and Irene A Stegun.  Handbook of mathematical functions with formulas, graphs, and mathematical tables. national bureau of standards applied mathematics series 55\. tenth printing.  page 253–266, 1972.
  * [2] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma.  Learning imbalanced datasets with label-distribution-aware margin loss.  Advances in neural information processing systems, 32, 2019.
  * [3] Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, et al.  Symbolic discovery of optimization algorithms.  arXiv preprint arXiv:2302.06675, 2023.
  * [4] Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia.  Incorporating second-order functional knowledge for better option pricing.  Advances in neural information processing systems, 13, 2000.
  * [5] Kunihiko Fukushima.  Cognitron: A self-organizing multilayered neural network.  Biological cybernetics, 20(3-4):121–136, 1975.
  * [6] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [7] R Herrmann.  Fractional calculus-an introduction for physicists world scientific publishing, 2011.
  * [8] Peter J Huber.  Robust estimation of a location parameter.  In Breakthroughs in statistics: Methodology and distribution, pages 492–518. Springer, 1992.
  * [9] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [10] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [11] Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han.  On the variance of the adaptive learning rate and beyond.  arXiv preprint arXiv:1908.03265, 2019.
  * [12] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [13] Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun.  Adaptive gradient methods with dynamic bound of learning rate.  arXiv preprint arXiv:1902.09843, 2019.
  * [14] Diganta Misra.  Mish: A self regularized non-monotonic activation function.  arXiv preprint arXiv:1908.08681, 2019.
  * [15] Manuel Duarte Ortigueira and Fernando Coito.  From differences to derivatives.  Fractional Calculus and Applied Analysis, 7(4):459, 2004.
  * [16] Prajit Ramachandran, Barret Zoph, and Quoc V Le.  Searching for activation functions.  arXiv preprint arXiv:1710.05941, 2017.
  * [17] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  International Journal of Computer Vision (IJCV), 115(3):211–252, 2015.
  * [18] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International conference on machine learning, pages 6105–6114. PMLR, 2019.
  * [19] Paul Werbos.  Beyond regression: New tools for prediction and analysis in the behavioral sciences.  PhD thesis, Committee on Applied Mathematics, Harvard University, Cambridge, MA, 1974.
  * [20] Julio Zamora, Anthony D Rhodes, and Lama Nachman.  Fractional adaptive linear units.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8988–8996, 2022.
  * [21] Julio Zamora Esquivel, Adan Cruz Vargas, Rodrigo Camacho Perez, Paulo Lopez Meyer, Hector Cordourier, and Omesh Tickoo.  Adaptive activation functions using fractional calculus.  In Proceedings of the IEEE/CVF international conference on computer vision workshops, pages 0–0, 2019.
  * [22] Zhifei Zhang, Yang Song, and Hairong Qi.  Age progression/regression by conditional adversarial autoencoder.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5810–5818, 2017.


The Tunka-Grande scintillation array: current results

Anna L. Ivanova1,2,$\star$, I. Astapov3, P. Bezyazeekov2, E. Bonvech4, A.
Borodin5, N. Budnev2, A. Bulan4, D. Chernov4, A. Chiavassa6, A. Dyachok2, A.
Gafarov2, A. Garmash1,7, V. Grebenyuk5,8, E. Gress2, O. Gress2, T. Gress2, A.
Grinyuk5, O. Grishin2, A. D. Ivanova2, N. Kalmykov4, V. Kindin3, S. Kiryuhin2,
R. Kokoulin3, K. Kompaniets3, E. Korosteleva3, V. Kozhin3, E. Kravchenko1,7,
A. Kryukov4, L. Kuzmichev4,2, A. Lagutin9, M. Lavrova5, Y. Lemeshev2, B.
Lubsandorzhiev10,4, N. Lubsandorzhiev4, A. Lukanov10, D. Lukyantsev2, S.
Malakhov2, R. Mirgazov2, R. Monkhoev2, E. Osipova4, A. Pakhorukov2, L.
Pankov2, A. Pan5, A. Panov4, A. Petrukhin3, I. Poddubnyi2, D. Podgrudkov4, V.
Poleschuk2, V. Ponomareva2, E. Popova4, E. Postnikov4, V. Prosin4, V.
Ptuskin11, A. Pushnin2, R. Raikin9, A. Razumov4, G. Rubtsov10, E. Ryabov2, Y.
Sagan5,8, V. Samoliga2, A. Satyshev5, A. Silaev4, A. Silaev (junior)4, A.
Sidorenkov10, A. Skurikhin4, A. Sokolov1, L. Sveshnikova4, V. Tabolenko2, L.
Tkachev5,8, A. Tanaev2, M. Ternovoy2, R. Togoo12, N. Ushakov10, A.
Vaidyanathan1, P. Volchugov4, N. Volkov9, D. Voronin10, A. Zagorodnikov2, D.
Zhurov2,13 and I. Yashin3

1 Novosibirsk State University, Pirogova str. 2, Novosibirsk, 630090 Russia

2 Irkutsk State University, Karl Marx str. 1, Irkutsk, 664003 Russia

3 National Research Nuclear University MEPhI (Moscow Engineering Physics
Institute), Kashirskoe hwy 31, Moscow, 115409 Russia

4 Lomonosov Moscow State University, Skobeltsyn Institute of Nuclear Physics,
Leninskie gory 1 (2), GSP-1, Moscow, 119991 Russia

5 Joint Institute for Nuclear Research, Joliot-Curie str. 6, Dubna, 141980
Russia

6 Dipartimento di Fisica Generale Universiteta di Torino and INFN, Via P.
Giuria 1, Turin, 10125 Italy

7 Budker Institute of Nuclear Physics SB RAS, Ac. Lavrentiev Avenue 11,
Novosibirsk, 630090 Russia

8 Dubna State University, Universitetskaya str. 19, Dubna, 141982 Russia

9 Altai State University, pr. Lenina 61, Barnaul, 656049 Russia

10 Institute for Nuclear Research RAS, prospekt 60-letiya Oktyabrya 7a,
Moscow, 117312 Russia

11 IZMIRAN, Kaluzhskoe hwy 4, Troitsk, Moscow, 4108840 Russia

12 Institute of Physics and Technology Mongolian Academy of Sciences,
Enkhtaivan av. 54B, Ulaanbaatar, 210651 Mongolia

13 Irkutsk National Research Technical University, Lermontov str. 83, Irkutsk,
664074 Russia

*<EMAIL_ADDRESS>
21st International Symposium on Very High Energy Cosmic Ray Interactions
(ISVHE- CRI 2022)
Online, 23-27 May 2022 10.21468/SciPostPhysProc.?

## Abstract

The Tunka-Grande experiment is a scintillation array with about 0.5 km2
sensitive area at Tunka Valley, Siberia, for measuring charged particles and
muons in extensive air showers (EASs). Tunka-Grande is optimized for cosmic
ray studies in the energy range 10 PeV to about 1 EeV, where exploring the
composition is of fundamental importance for understanding the transition from
galactic to extragalactic origin of cosmic rays. This paper attempts to
provide a synopsis of the current results of the experiment. In particular,
the reconstruction of the all-particle energy spectrum in the range of 10 PeV
to 1 EeV based on experimental data from four observation seasons is
presented.

###### Contents

  1. 1 Introduction
  2. 2 Five seasons of Tunka-Grande scintillation array operation
  3. 3 EAS and CR parameters reconstruction technique
  4. 4 Estimating the accuracy of the EAS and CR parameters reconstruction
  5. 5 Energy spectrum
  6. 6 Gamma-hadron discrimination
  7. 7 Conclusion

##  1 Introduction

An important task of modern astrophysics is to study cosmic rays in the energy
range from 10 PeV to about 1 EeV, where exploring the composition is of
fundamental importance for understanding the transition from galactic to
extragalactic origin of cosmic rays. Studies in this energy range are also
interesting from the point of view of search for gamma rays with energy more
than 100 TeV.

The Tunka-Grande scintillation facility described in reference [1] conducts
scientific research in the fields of high-energy gamma-ray astronomy and
cosmic ray physics by detecting the charged and muon components of EAS. It is
located in Tunka Valley, 50 km of the Lake Baikal and together with four other
independent arrays (TAIGA-HiSCORE and Tunka-133 Cherenkov arrays, TAIGA-Muon
scintillation array, and network of Imaging Atmospheric Cherenkov Telescopes
TAIGA-IACT) forms the Tunka experimental complex.

##  2 Five seasons of Tunka-Grande scintillation array operation

During the four seasons from 2017 to 2021, there were 691 days of Tunka-Grande
operation. The array trigger condition was a coincidence of any three surface
detectors within 5 µs. During this period, about 3,409,000 triggering events
were detected on the Tunka-Grande area over 9100 h of operation. The mean
trigger rate is about 0.1 Hz.

Only successfully reconstructed events with a zenith angle $\theta\leq 35^{o}$
and core position in a circle around the centre of the array with a radius R <
350 m were selected. The threshold energy of 100 persent registration
efficiency for chosen area and zenith angles is 10 PeV. The number of selected
events with energies above 10 PeV was about 240,000. Approximately 2000 events
from them had energies above 100 PeV.

In the first observation season from 2016 to 2017, Tunka-Grande operated by
Tunka-133 trigger. There were 475 h of joint operation and about 77,000
events.

The data from the first season were used to analyze joint events and assess
the quality of the Tunka-Grande reconstruction technique. Unlike the weather-
independent Tunka-Grande, only clear moonless nights are used for the correct
events reconstruction by Tunka-133 Cherenkov array. This condition reduced the
time suitable for joint events analysis to 357 hours and the number of joint
events to 25,000. About 6000 events of them had core position in the circle
with radius of 350 m, zenith angle of up to $35^{o}$ and energy above 10 PeV.

##  3 EAS and CR parameters reconstruction technique

The EAS and CR parameters reconstruction procedure includes several stages.

At the first stage, EAS pulses main parameters (amplitude, pulse area,
registration time) are determined, calibration procedures are carried out, and
the most probable energy deposit per charged particle, necessary for further
reconstruction of the particle density in the detectors of the array, is
determined. Then density of particles in ground and underground detectors, the
initial values of the core position, the arrival direction of the EAS, the
total number of particles of the electron-photon and muon components are
calculated.

The shower arrival direction is determined by fitting the measured pulse front
delay using the curved shower front formula, which was obtained at the
KASCADE-Grande experiment[2].

An important element of the EAS and CR parameters reconstruction procedure is
the function of the lateral distribution of particles (LDF). The lateral
distribution of charged particles is described using the specific EAS-MSU
function[3]. Greisen function[3] is used for muons.

Next, the shower core coordinates, number of muons and charged particles, and
slope of the LDF are refined by minimizing the functional using independent
variables. In addition, the age of the shower and the density of charged
particles and muons at a distance of 200 m from EAS core position are
determined.

As a measure of energy, we use the charged particles density at a core
distance of 200 m – $\rho_{200}$ parameter rescaled relative to the measured
zenith angle $\theta$ for atmospheric depth from sea level for the Tunka
Valley $x_{0}=960g/sm^{2}$ and obtained from experimental data average value
of absorption path length $\lambda=260g/sm^{2}:\rho_{200}(\theta=0)$[1]. The
$\rho_{200}(\theta=0)$ conversion to energy is carried out according to
formula:

$E_{0}=10^{15.99}\cdot(\rho_{200}(\theta=0))^{0.84}$ (1)

Correlation $\rho_{200}(\theta=0)$ with primary energy was determined using
the experimental results of the Tunka-133 Cherenkov array. The
$\rho_{200}(\theta=0)$ value was calculated from the data of the Tunka-Grande
scintillation array, and the energy value was taken from the data of the
Tunka-133 Cherenkov array. This technique is based on a comparison of
experimental data and practically does not depend on the hadron interaction
model.

##  4 Estimating the accuracy of the EAS and CR parameters reconstruction

The Tunka-133 Cherenkov array[4] demonstrate good performance and high
reliability of equipment. Therefore, to determine the accuracy of the
reconstruction of the EAS parameters measured with the Tunka-Grande array, the
EAS parameters reconstructed from Tunka-133 experimental data are used.

The search for joint events was performed within the time range of minus plus
10 microseconds in showers with zenith angles of up to $35^{o}$, detected in a
circle with $R<350$ m.

The average difference between Tunka-Grande and Tunka-133 zenith angles
doesn’t exceed than $0.03^{o}$, for azimuth angles it is also less than
$0.03^{o}$, the standard deviations are $\sigma_{\theta},\sigma_{\phi}\approx
1.3^{o}$. The average differences between Tunka-Grande and Tunka-133 shower
core coordinates don’t exceed 2 meters, the standard deviation is
$\sigma=16.4$ m for X-coordinate and is $\sigma=17$ m for Y-coordinate.

The distribution of the difference between the EAS arrival directions and
distribution of the difference between the EAS core positions measured by the
Tunka-Grande and by the Tunka-133 are shown in the Fig. 1.

Figure 1: The accuracy of the arrival direction (a) and core position (b)
reconstruction by the Tunka-Grande array in comparison with data of Tunka-133
array[4]

The thresholds for parameters $\psi$ and $R$ on the $68\%$ \- confidence
level, within which $68\%$ events are located,were taken by us as angular
resolution and core position resolution of the Tunka-Grande array,
respectively. The $\psi_{68}$ is equal to $2.3^{o}$ and $R_{68}$ is equal to
$26$ m.

The energy resolution is $36\%$ for events with $E\geq 10$ PeV. It becomes
better with increasing energy and doesn’t exceed $15\%$ for energies above 100
PeV.

##  5 Energy spectrum

To reconstruct the differential energy spectrum the number of events inside
each bin of 0.1 in $\lg E_{0}$ was calculated. The differential uncorrected
intensity is obtained by dividing this number by the selected effective area,
the selected effective solid angle, the observation time, and the energy bin
width. We have selected events with zenit angle $\theta\leq 35^{o}$ and core
position in a circle with a radius of 350 meters.

The spectrum of Tunka-Grande (Fig. 2, a) shows several features, deviating
from a single power law. At an energy of about 20 PeV, the power law index
changes from $\gamma=3.18$ to $\gamma=3.00$. And the spectrum becomes much
steeper with $\gamma=3.25$ above 100 PeV.

The spectrum is compared with the results from Tunka-133[4], KASCADE-
Grande[5], TALE[6] and Ice Top[7] facilities (Fig. 2, b). It demonstrates good
agreement with data of large terrestrial facilities.

Figure 2: Differential primary cosmic-ray energy spectrum, a \- with a fit of
a doubly broken power law, b \- comparison to other experimental results. The
vertical lines show the value of the statistical errors.

##  6 Gamma-hadron discrimination

Nowadays the usual technique for searches for primary $\gamma$ in EASs is to
discriminate gamma-ray primaries from the hadronic background by identifying
muon-poor or even muon-less EASs. For this a computer simulation of the Tunka-
Grande detectors operation [8, 9] was done with help of CORSIKA(QGSJET-
II-04,GHEISHA interaction models) and Geant4 softwares.

The Fig. 3, a shows the distribution of the muon number in underground Tunka-
Grande detectors versus $\rho_{200}$ parameter. Blue circles - experiment data
($N_{\mu}$ and $\rho_{200}$ are calculated from EAS events detected by Tunka-
Grande array). Green circles - data of simulated Eass, initiated by gamma
quantum. The events without any detected muons are plotted with
$lg(N_{\mu})=-1$ to be visible at the logarithmic axis. Lower red line
indicates the selection criteria. There is no excess of events consistent with
a gamma-ray signal seen in the data. Hence, we assume that all events below
the selection line are primary $\gamma$-rays and set upper limits on the
gamma-ray fraction of the cosmic rays. An upper limit on the fraction of the
$\gamma$ ray with respect to the cosmic ray flux was determined by formula
with 90$\%$ C.L.(Feldman – Cousins) $N_{90}=2.44$, $\epsilon_{\gamma}\sim
0.5$:

$\frac{I_{\gamma}}{I_{CR}}<\frac{N_{90}}{N_{tot}\epsilon_{\gamma}}\left(\frac{E_{CR}}{E_{\gamma}}\right)^{-\beta+1}$
(2)

Figure 3: a \- Scatter plot of the number of muons in underground detectors
vs. $\rho_{200}$ parameter. Green circles - simulated EASs from $\gamma$, blue
circles - experiment, red line - selection criteria. b \- Gamma-hadron
discrimination. Yellow $\nabla$ \- EAS MSU[10], grey $\nabla$ \- KASCADE
Grande[11], blue $\nabla$ \- this work, black $\nabla$ \- PAO[12], green
$\nabla$ \- TA[13]

Fig. 3, b displays the measurements on the gamma-ray fraction as a function of
the energy, including this work, for the energy range of 10 PeV up to 100 EeV.

##  7 Conclusion

As a result of the "Tunka-Grande-Tunka-133" joint events analysis, it was
found that for energies above 10 PeV, the angular resolution of Tunka-Grande
array is $\psi_{68}\leq 2.3^{o}$, the core position resolution - $R_{68}\leq
26$ m, the energy resolution - $E_{68}\leq 36\%$ .

Based on 4 measurement seasons All-particle energy spectrum was reconstructed
and the 90$\%$ C.L. upper limits to the diffuse flux of ultrahigh energy gamma
rays for energies above 30 PeV were determined.

## Acknowledgements

The work was performed at the UNU “Astrophysical Complex of MSU-ISU”
(agreement EB 075-15-2021-675). The work is supported by RFBR (grants
19-52-44002, 19-32-60003), the RSF(grants 19-72-20067(Section 2)), the Russian
Federation Ministry of Science and High Education (projects FZZE-2020-0017,
FZZE-2020-0024, and FSUS-2020-0039).

## References

  * [1] A.L. Ivanova et al, Tunka-Grande array for high-energy gamma-ray astronomy and cosmic-ray physics: preliminary results, PoS ICRC2021 395, 361 (2021), 10.22323/1.395.0361.
  * [2] R. Glasstetter et al, Shower Size Reconstruction at KASCADE-Grande, 29th ICRC Pune 6, 293-296 (2005).
  * [3] R.D. Monkhoev et al, The Tunka-Grande experiment: Status and prospects, Bull. Russ. Acad. Sci. 81, 4, 468-470 (2017), 10.3103/S1062873817040311.
  * [4] N.M. Budnev et al, The primary cosmic-ray energy spectrum measured with the Tunka-133 array, Astroparticle Physics 117, 102406 (2020), 10.1016/j.astropartphys.2019.102406.
  * [5] W. D. Apel et al, The spectrum of high-energy cosmic rays measured with KASCADE-Grande, Astroparticle Physics 36, 183 (2012), 10.1016/j.astropartphys.2012.05.023.
  * [6] R.U. Abbasi et al, The Cosmic-Ray Energy Spectrum between 2 PeV and 2 EeV Observed with the TALE detector in monocular mode, Astrophysical Journal 865, 74 (2018), 10.3847/1538-4357/aada05.
  * [7] K. Rawlins [IceCube Collaboration], Cosmic ray spectrum and composition from three years of IceTop and IceCube, J. Phys. Conf. Ser. 718, 5, 052033 (2016), 10.1088/1742-6596/718/5/052033.
  * [8] M. Ternovoy et al, Simulation of the Tunka-Grande, TAIGA-Muon and TAIGA-HiSCORE arrays for a search of astrophysical gamma quanta with energy above 100 TeV, J. Phys. Conf. Ser. 1847, 012047 (2021), 10.1088/1742-6596/1847/1/012047.
  * [9] R.D. Monkhoev et al, Geant4 simulation of the Tunka-Grande experiment, J. Phys. Conf. Ser. 2103, 012001 (2021), 10.1088/1742-6596/2103/1/012001.
  * [10] Yu. A. Fomin et al, Constraints on the flux of eV cosmic photons from $\sim$ $(10^{16}-10^{17.5})$ the EAS–MSU muon data, http://arxiv.org/abs/1702.08024v1.
  * [11] W.D. Apel et al, KASCADE-Grande limits on the isotropic diffuse gamma-ray flux between 100 TeV and 1 EeV, http://arxiv.org/abs/1710.02889v1.
  * [12] P. Savina et al, A search for ultra-high-energy photons at the Pierre Auger Observatory exploiting air-shower Universality, PoS ICRC2021 395, 373 (2021), 10.22323/1.395.0373.
  * [13] M. C. Chantell et al, Constraints on the diffuse photon flux with energies above $10^{18}$ eV using the surface detector of the Telescope Array experiment, Astroparticle Physics 110, p.8-14 (2019), 10.1016/j.astropartphys.2019.03.003.