# File Fragment Classification using Light-Weight Convolutional Neural
Networks

Mustafa Ghaleb, Kunwar Saaim, Muhamad Felemban , Saleh Al-Saleh, Ahmad Al-
Mulhem M. Ghaleb, M. Felemban, S. Al-Saleh, and A. Al-Mulehm are in the
Department of Computer Engineering and the Interdisciplainry Research Center
for Intelligent Secure Systems at KFUPM, Dhahran, Saudi ArabiaK. Saaim is with
the Department of Computing Science, University of Alberta

###### Abstract

In digital forensics, file fragment classification is an important step toward
completing file carving process. There exist several techniques to identify
the type of file fragments without relying on meta-data, such as using
features like header/footer and N-gram to identify the fragment type.
Recently, convolutional neural network (CNN) models have been used to build
classification models to achieve this task. However, the number of parameters
in CNNs tends to grow exponentially as the number of layers increases. This
results in a dramatic increase in training and inference time. In this paper,
we propose light-weight file fragment classification models based on depthwise
separable CNNs. The evaluation results show that our proposed models provide
faster inference time with comparable accuracy as compared to the state-of-art
CNN based models. In particular, our models were able to achieve an accuracy
of 79% on the FFT-75 dataset with nearly 100K parameters and 164M FLOPs, which
is 4x smaller and 6x faster than the state-of-the-art classifier in the
literature.

###### Index Terms:

IEEE, IEEEtran, journal, LaTeX, paper, template.

##  I Introduction

Digital forensics is the science of collecting digital evidence to investigate
cybercrimes and cyberattacks. The process of digital forensics includes
preservation, identification, and extraction of files and data from damaged or
compromised machines. There are several digital forensics techniques and tools
that facilitate this process [38, 31]. Often, attackers attempt to wipe out
any evidence that incriminates them, for example, by formatting the hard disk
[6]. In such scenarios, traditional recovery methods based on the file system
meta-data are deemed ineffective.

To overcome this challenge, digital investigators use file carving to
reconstruct files based on their contents [16, 28, 30]. File carving process
recovers damaged data files from blocks of raw binary data without using any
meta-data in the file system. There are several file carving techniques to
reconstruct files fully or partially using a variety of techniques including
header/footer matching [39], probabilistic measures [3], and n-gram analysis
[49].

Without meta-data, it is challenging to identify the type of a carved file.
The problem is even more challenging when file carving is carried out on
fragmented files. File fragmentation usually occurs when there is not enough
contiguous space to write a large file on the hard disk. Two tasks can be
recognized in fragmented file carving: selecting a candidate sequence of file
blocks and classifying the type of the fragment [30]. The latter is called
file fragment classification.

There exist several approaches for file fragment classification in the
literature [50, 33, 5, 49, 39, 9, 29, 13, 1, 4, 53, 48, 18]. For example,
Sceadan tool determines the type of bulk data based file content [5].
Recently, deep learning has been used to identify the type of file fragment.
For example, Gray-scale [9] and FiFTy [33] tools illustrate how CNNs can be
efficiently used for file fragment classification. Although CNN is the most
used deep learning model for classification, we argue that existing models can
be further improved in terms of performance and accuracy by modifying the
architecture of the neural network. We particularly focus on light-weight CNNs
to be able to carry out file fragment classification on resource-limited
devices.

A major drawback of CNNs is the exponential increase in the number of
parameters as the number of layers increases, i.e., the model depth increases.
Such an increase in the number of parameters increases the complexity of the
model and, accordingly, increases the training and inference time. On the
other hand, deep CNN models provide better accuracy. Therefore, it is
important to find a balance between inference speed and accuracy by optimizing
the CNN structure, i.e., number of layers, CNN filters size, and number of
channels. To overcome this challenge, researchers proposed a computationally
cheaper model called depthwise separable convolution [23, 45, 46], in which
the number of parameters in the model is reduced dramatically.

In our previous paper [40], we proposed a light-weight file fragment
classification based on convolutional neural network, called Depthwise
Separable Convolutional (DSC). In this paper, we propose two additional
models: Depthwise Separable Convolutional with Squeeze-and-Excitation (DSC-SE)
and Modified Depthwise Separable Convolutional (M-DSC). We show that the
proposed models reduce the number of parameters as compared to CNN. Therefore,
the process of file fragment classification can be carried out without the
need of specialized hardware, e.g., GPUs and TPUs. To evaluate the performance
of our models, we compared the results of the proposed models with FiFTy [33]
and a baseline Recurrent Neural Networks (RNN) [21]. We have trained the
models using FFT-75 dataset [32], which includes six scenarios. The results of
Scenario 1 show that our model achieves an accuracy that is comparable to
FiFTy using 6.3x less floating-point operations (FLOPs) on 4096 bytes fragment
and 87x less FLOPs on 512 bytes fragments. Furthermore, our models achieve
79.27% accuracy with around 100K parameters and 164M FLOPs, while FiFTy
achieves 77.04% accuracy with more than 400K parameters and 1 GFLOPs. Finally,
we show that our models outperform FiFty in terms of inference time
(milliseconds per block) by being 15x faster on GPU than FiFTy and 105x faster
than baseline RNN.

The rest of this paper is organized as follows. In Section 2, we provide the
necessary background for this work. In Section 3, we present our proposed
models. In Section 4, we discuss the experiments and evaluation results. In
Section 5, we discuss the state-of-the-art of file fragment classification.
Finally, we draw conclusions from our findings and suggest potential
directions for future research in Section 6.

##  II Preliminaries

###  II-A File carving

File carving is the process of extracting files from storage media (such as
hard drives) based on their content instead of using the API or meta-data of a
file system [30]. Such process is often used when the underlying file system
is damaged or unknown. File carving can be complicated when files are
fragmented by the operating system. Therefore, file carving process includes
carving file fragments, identifying each fragment’s type, ordering the
fragments, and assembling the file. There are several file carving tools and
techniques that are used in practice, including header/footer matching and
file structure-based carving[35, 36]. However, there are few challenges that
need to be addressed: 1) improving the performance of file carving, as the
average digital forensics case size nowadays is in terabytes, and 2) improving
the overall accuracy of file carving.

###  II-B Convolution Neural Network

CNN is a type of artificial neural network that is widely used for object
detection and image classification. CNN contains one or more convolution
layers that can extract features from the inputs and predict an output based
on the extracted features [14]. A convolution filter, also known as a kernel,
is a 2D matrix that is used to perform the convolution operation on images.
Mathematically, the convolution operation is the dot product between the
kernel and the layer’s input. Various kernels can be used to extract various
features from the input data points. In general, kernel filters are designed
to be smaller than the input to allow for the sharing of kernel weights across
input dimensions. In file fragment classification, the input to the CNN is
either a vector of 4096 or 512 dimensions, equivalent to fragments of 4KB or
512 bytes, respectively. The input is then transmitted to an embedding layer
that converts the input vector into a continuous (4096,32) or (512,32)
dimensional vector, dependent on the fragment size. The embedding layer is a
preliminary step to prepare the input vectors for subseque CNN layers. The
output of the embedding layer is fed into a convolution layer with kernels of
any size. Several convolution layers can be stacked [33]. As a result of
stacking standard convolution layers when developing a Deep Neural Network
(DNN), in which the model can have a large number of parameters.

###  II-C Depthwise Separable Convolution

(a) Standard Convolution

(b) Depthwise Convolution

(c) Pointwise Convolution

Figure 1: Standard convolution in (a) is factorized into two layers: Depthwise
Convolution in (b) and pointwise in (c).

The separable convolution model was first introduced by Sifre and Mallat [42].
Since then, several models have been proposed based on separable convolution,
including Xception-net [10], Mobilenets [23], Effecienet [44], and Shufflenet
[52]. The automatic feature extraction behaviour of standard convolution is
achieved by smaller and faster depthwise separable convolution. Depthwise
separable convolution is a form of factorized convolutions in which the normal
convolution kernel is split into two parts: one depthwise convolution and a
$1\times 1$ convolution, called pointwise convolution. The splitting of
convolution into two layers drastically reduce the number of parameters, and
hence the computation time required for training and inference. The depthwise
convolution applies a single filter to each input channel. The pointwise
convolution then applies a $1\times 1$ convolution to combine the outputs of
the depthwise convolution [23]. Figure 1(a) depicts a standard one-dimensional
convolution operation that takes $D\textsubscript{F}\times 1\times M$ feature
maps as input and produces $D\textsubscript{F}\times 1\times N$ feature maps,
where DF is length of the input feature, M is the number of input channels,
and N is the number of output channels. In convolutional network, the number
of parameters with kernel size K is $D\textsubscript{K}\times 1\times N\times
M$. Subsequently, the total computation cost is

$D\textsubscript{F}\cdot N\cdot M\cdot D\textsubscript{K}$ (1)

Unlike CNNs, depthwise separable convolution factorizes the convolution layer
into two layers: depthwise convolution depicted in figure 1(b) and pointwise
convolution shown in figure 1(c). The depthwise convolution layer with one
filter per input channel has $D\textsubscript{K}\times 1\times M$ parameters
with K sized kernel and M number of channels. Therefore, depthwise convolution
has a computation cost of:

$D\textsubscript{F}\cdot M\cdot D\textsubscript{K}$ (2)

On the other hand, pointwise convolution uses a linear combination of filtered
feature maps with the help of $1\times 1$ convolution. The cost of $1\times 1$
convolution is the same as standard convolution having a kernel size of 1,
i.e., $D\textsubscript{F}\times N\times M$. Consequently, the total
computation cost of depthwise separable convolution is:

$D_{F}\cdot M\cdot D_{K}+D_{F}\cdot N\cdot M$ (3)

As a result, the total reduction in the computation cost when using depthwise
separable convolution is:

$\frac{D_{F}\cdot M\cdot D_{K}+D_{F}\cdot N\cdot M}{D_{F}\cdot N\cdot M\cdot
D_{K}}$

(4)

$=\scalebox{1.3}{$\frac{1}{N}+\frac{1}{D_{K}}$}$ (5)

The value of N ranges from 32 to 1024 and DK for one-dimensional convolution
can be between 9 and 27. Therefore, the reduction in computation cost is
between 85% and 95%.

In this regard, several models have been created using depthwise separable
convolution for the purpose of image classification. Such models achieve high
accuracy while still enabling speedy inference time. [23, 41, 10].

##  III Light-weight CNNs for File Fragment Classification

Figure 2: Network Architectures. a) DSC, b) DSC-SE, and c) M-DSC

In this section, we describe the proposed architectures of the proposed light-
weight CNNs models for file fragment classification. Previous file carving
techniques mostly rely on statistical feature extraction [26], SVMs [5], CNN
[33], RNN [18], and Fully Connected Neural Networks (FCNN) [48]. RNNs are
exponentially slow, while FCNNs and CNNs require a deep model to achieve good
performance. Our objective is to reduce the inference time of file fragment
classification while maintaining high accuracy. In this regard, several models
have been developed based on depthwise separable convolution for image
classification tasks. These models can achieve fast inference time while
maintaining high accuracy [23, 41, 10]. Compared to FCNNs, CNNs, and RNNs, it
was shown that depthwise separable convolution typically requires less
computation time and fewer parameters [23].

Figure 2 illustrates the overall architectures of the proposed models. All
models consist of an embedding layer and a standard 1D convolution block. The
embedding layer transforms a file fragment byte ranging from 0 to 255 into a
dense continuous vector of 32 dimensions. The embedding layers is used to
compress the input feature dimension into a smaller space. Without embedding
layer, each byte value would be represented by a 256-dimensional sparse
vector. With multiple layers of standard and depthwise separable convolutions,
features are automatically derived by non-linearly transforming inputs. In
order to extract essential features automatically for classification, the
dense vector representation is then passed to a standard 1D convolution block
and multiple depthwise separable inception blocks or squeeze-and-excitation
blocks. To get the final features for classification, the output of the last
depthwise separable convolution block is averaged along the spatial dimension.
In the following sections, we provide more details about each model.

###  III-A Depthwise Separable Convolution (DSC)

In DSC, the output of the 1D convolutional layer is passed to a hardswish
activation function, which adds non-linearity to the model. The Hardswish
function [22] is as the following.

$Hardswish(x)=\begin{cases}0&ifx\leq-3,\\\ x&ifx\geq+3,\\\
\frac{x\cdot(x+3)}{6}&otherwise\end{cases}$ (6)

The output is then processed by three inception blocks, which are composed of
multiple parallel convolutional and pooling layers, designed to capture both
local and global information from the input. After the inception blocks, the
output is passed to an average pooling layer to reduce the spatial dimension,
followed by a final 1D convolutional layer that performs feature map reduction
and generates the final prediction. The class probabilities are determined
from feature vectors using $1\times 1$ convolution followed by softmax
activation function [12].

Inception Block. Figure 3 depicts the architecture of a typical inception
block. The previous layer’s output is passed to three parallel depthwise
separable convolutions and one $1\times 1$ convolution. In our model, the
kernel size of the depthwise separable convolution layer is 11, 19, and 27,
respectively, with strides of 1, 1, 1, and 4, respectively. Kernel sizes were
chosen empirically based on previous state-of-the-art network [33]. A batch
normalization layer follows each of the convolution layer. All the depthwise
separable convolution layer output are summed and then sub-sampled using a
max-pooling layer with size 4. The output of the $1\times 1$ convolution
branch is directly added to the output of the max-pooling layer. If the number
of the input and output channels is the same, then the $1\times 1$ convolution
layer is replaced with the direct addition of the output of max-pooling layer.

Figure 3: Inception Block Figure 4: Modified Inception Block Figure 5:
Squeeze-and-Excitation Block

###  III-B Depthwise Separable Convolution with Squeeze and Excitation (DSC-
SE)

DSC-SE is based on DSC but with the addition of Squeeze-and-Excitation (SE)
[24] blocks after each inception block. SE block is a type of attention
mechanism to improve CNNs performance. SE block allows the network to focus on
the most important features in each channel of the feature maps, which is
useful in tasks such as image classification, segmentation, and object
detection. SE block works by first squeezing the spatial information from the
feature maps into a single channel-wise global descriptor, and then using a
fully connected layer to calculate a set of channel-wise weights. These
weights are then multiplied element-wise with the feature maps. This, in turn,
assigns higher weights to the channels that contain the most important
information for the task. Recently, SE blocks have been integrated into the
inception architecture resulting in an improved performance in various tasks
[24]. The core principle behind SE blocks is to give greater importance to the
features that contribute more to the final prediction. Moreover, SE blocks are
computationally efficient and easy to implement. Figure 5 shows the structure
of the SE block in detail.

###  III-C Modified Depthwise Separable Convolution (M-DSC)

M-DSC architecture is also based on DSC but with several modifications.
Instead of the first 1D convolutional layer, M-DSC uses a depthwise
convolutional layer that performs convolutions independently on each channel
of the input. The hardswish activation is replaced with the rectified linear
unit (ReLU). Instead of batch normalization, group normalization was used to
normalize the activation of multiple channels at once [25, 51]. The modified
inception block used in our proposed models is shown in Figure 4. This results
in reducing the memory and computation overhead of batch normalization.
Finally, after the average pooling and before the final 1D convolutional
layer, a dropout layer is added to prevent overfitting by randomly setting a
fraction of the activations to zero during training.

In summary, the three architectures differ in the way of handling activation
functions, normalization, and feature recalibration, with the aim to improve
the performance and efficiency of the model. DSC-SE adds SE blocks to the
inception blocks to improve feature recalibration, while M-DSC replaces the 1D
convolutional layer with a depthwise convolutional layer, replaces hardswish
with ReLU, and introduces group normalization and dropout to improve
performance and efficiency.

##  IV Performance Evaluation

In this section, we present the results of evaluating the performance of our
proposed model against baseline models based on RNN and FiFTy [33].

###  IV-A Dataset

TABLE I: Grouping of different file types Grouping | Files
---|---
Archive | apk, jar, msi, dmg, 7z, bz2, deb, gz,
| pkg, rar, rpm, xz, zip
Audio | aiff, flac, m4a, mp3, ogg, wav, wma
Bitmap | jpg, tiff, heic, bmp, gif, png
Executable | exe, mach-o, elf, dll
Human-readable | md, rtf, txt, tex, json, html, xml, log,
| csv
Office | doc, docx, key, ppt, pptx, xls, xlsx
Published | djvu, epub, mobi, pdf
Raw | arw, cr2, dng, gpr, nef, nrw, orf, pef, raf,
| rw2, 3fr
Vector | ai, eps, psd
Video | mov, mp4, 3gp, avi, mkv, ogv, webm
Miscellaneous | pcap, ttf, dwg, sqlite

To evaluate the performance of our models, we used the dataset provided by
Mittal et al. in [32], which contains a balanced number of files per class.
Other datasets, e.g., [15], are highly imbalanced with 20 files-types
comprising 99.3% of the dataset and remaining 0.7% belonging to 43 file types.
The dataset used is composed of 75 types of files that are organized into 6
different scenarios. The Scenarios are described as follows in [33]:

  1. 1.

(All; 75 classes): All file types are separate classes; this is the most
generic case and can be aggregated into more specialized use-cases.

  2. 2.

(Use-specific; 11 classes): File types are grouped into 11 classes according
to their use; this information may be useful for elaborate, hierarchical
classification, or for determining the primary use of an unknown device.

  3. 3.

(Media Carver - Photos & Videos; 25 classes): Every file type tagged as a
bitmap (6), RAW photo (11), or video (7) is considered as a separate class;
all remaining types are grouped into one other class.

  4. 4.

(Coarse Photo Carver; 5 classes): Separate classes for different photographic
types: JPEG, 11 RAW images, 7 videos, 5 remaining bitmaps are grouped into one
separate class per category; all remaining types are grouped into one other
class.

  5. 5.

(Specialized JPEG Carver; 2 classes): JPEG is a separate class and the
remaining 74 file types are grouped into one other class; this scenario is
designed for analyzing disk images from generic devices.

  6. 6.

(Camera-Specialized JPEG Carver; 2 classes): JPEG is a separate class, and the
remaining photographic/video types (11 RAW images, 3GP, MOV, MKV, TIFF and
HEIC) are grouped into one other class; this scenario is designed for
analyzing SD cards from digital cameras.

###  IV-B Baseline Model

We implemented a variation of RNN, i.e., Long Short Term Memory (LSTM) based
model, as a baseline model for performance comparison based on the work in
[18]. It was observed that the low number of learnable parameters hindered
their effectiveness in classifying 75 types of files. The model specifications
are as follows. File fragment byte sequences are fed into a 32-dimensional
embedding layer followed by bidirectional and unidirectional LSTM layers. The
LSTM layer has 128 neurons in each of its two layers for 512-byte fragments
and 128 and 256 neurons in its bidirectional and unidirectional layers for 4
KB fragments, respectively. A softmax dense layer was used to produce the
output labels.

###  IV-C Experimental setup

TABLE II: Comparison of results of five models on all 75 file types

Model Neural Network Block Size # Params Accuracy Speed[ms/block]† Speed
[min/GB]† DSC Depthwise Separable CNN 4096 103,083 78.45 2.567 0.055 512
103,083 65.89 3.531 0.382 DSC-SE Depthwise Separable CNN with SE 4096 105,515
79.27 3.908 0.064 512 105,515 66.33 3.473 0.471 M-DSC Modified Depthwise
Separable CNN 4096 85,291 78.68 2.606 0.057 512 85,291 64.03 2.776 0.378 FiFTy
1-D CNN 4096 449,867 77.04 38.189 1.366 512 289,995 65.66 38.67 3.052 Baseline
RNN LSTM 4096 717,643 70.51 268.58 36.375 512 379,851 67.5 126.54 33.431 †
Computed on Nvidia Titan X

All of our experiments were conducted on a dual Intel Xeon CPU E5-2620 CPUs @
2.40 GHz (12 physical cores, 24 logical cores), 192 GB RAM, and a single
Nvidia Titan X GPU, running on Ubuntu 20.04 Operating System. Pytorch 1.5.0
was used for neural network design. We used automated hyper-parameter tuning
for learning rate, optimizer choice, and activation function using TPE [7]
implemented through Optuna [2]. We found out that Adam optimizer [27] and
Hardswish [22] are the best optimizer and activation function, respectively.
We did not perform tuning of convolution kernel size; we used the kernel sizes
from previous work [33] that proved to be effective. The kernel sizes for
different branches of the inception block were taken as 11,19, and 27.

The accuracy for different networks was calculated as follows:

$Accuracy=\frac{TP+TN}{TP+TN+FP+FN}\\\ $ (7)

where TP is True Positive, TN is True Negative, FP is False Positive, and FN
is False Negative.

To achieve higher accuracy from smaller models, we pretrained our network on
corresponding fragment size dataset. We tried to leverage the performance of
transfer learning in gaining higher accuracy [37]. In particular, to develop a
4096-byte fragment, the model was pretrained on 512-byte fragment dataset and
vice-versa. We found that 6-8% accuracy was increased when pretraining was
done using 512-byte fragment data for developing the 4096-byte fragment model.
However, similar performance was not achieved when pretraining was done
4096-byte fragment data for the 512-byte fragment model.

###  IV-D Results

TABLE III: Comparison between FiFTy and our models in terms of inference time
and model Parameters

Scenario Fragment #param Inf. Time (CPU)[ms/block] Inf. Time (GPU)[ms/block]
Size DSC DSC-SE M-DSC FiFTy DSC DSC-SE M-DSC FiFTy DSC DSC-SE M-DSC FiFTy 1
4096 103,083 105,515 85,291 449,867 18.595 23.291 17.338 121.476 2.567 3.908
2.606 38.189 512 103,083 105,515 85,291 289,995 6.958 8.216 6.279 75.551 3.531
3.473 2.776 38.673 2 4096 94,827 97,259 77,035 597,259 22.587 18.023 17.214
92.324 2.763 3.300 2.610 29.826 512 94,827 97,259 77,035 269,323 7.827 6.657
6.346 89.344 3.239 3.504 2.910 29.809 3 4096 96,633 99,065 78,841 453,529
17.068 18.860 17.361 102.808 3.284 3.182 2.632 37.286 512 96,633 99,065 78,841
690,073 6.066 6.613 6.300 99.710 3.078 3.499 2.708 35.361 4 4096 94,053 96,485
76,261 684,485 17.149 20.477 17.537 100.117 2.524 3.197 2.608 40.176 512
94,053 96,485 76,261 474,885 6.029 6.533 6.285 79.346 2.620 3.258 2.677 35.965
5 4096 93,666 96,098 75,847 138,386 22.427 16.362 17.474 99.855 2.524 3.362
2.769 39.262 512 93,666 96,098 75,847 336,770 5.967 6.327 6.275 79.831 2.581
3.259 2.873 42.413 6 4096 93,666 96,098 75,847 666,242 18.418 23.000 17.279
98.536 2.530 3.389 2.641 34.931 512 93,666 96,098 75,847 242,114 6.133 8.373
6.398 89.104 2.612 3.274 2.702 38.982

In general, our models perform better than FiFTy in inference time with no
significant loss of accuracy. The results are summarised in Table II. In
particular, we observe that our models outperform both FiFTy and RNN
(Baseline) in terms of accuracy and inference time for scenario 1 (75 file
types). In particular, FiFTy and RNN models are 25x and 660x, respectively,
slower than our proposed models when running on 1 GB data of 4096-byte
fragments. Using 512-byte fragments, FiFTy is 8x slower, and RNNs is 87x
slower than our model when running on 1 GB of data. It was observed that RNN
model performs well with 512-byte fragments due to their ability to handle
short sequences. In contrast, when the fragment size increases to 4096-bytes,
the performance deteriorates due to the vanishing gradient problem of RNNs
[20]. While RNN model has higher accuracy on the 512-byte fragments, RNN is
not practical because it takes relatively longer inference time.

Number of parameters and inference time. Table III summarizes the number of
neural network parameters in the proposed models compared to FiFTy. For all
six scenarios and both fragment sizes, our models have far fewer parameters
than FiFTy. In addition, Table III compares the inference times for each of
the six scenarios on both the GPU and CPU. Our models outperform FiFTy by a
large margin on GPU and CPU in term of inference time for all six scenarios.
We observed a 5x reduction in inference time for 4096-byte file fragments and
a 9x reduction for 512-byte file fragments compared to FiFTy on CPU. For GPU,
the time reduction is more than 9x for both fragment types.

Number of FLOPS. To provide hardware-independent metrics, we calculated the
floating-point operations in our models and FiFTy. In Table IV, we provide the
results of the comparison. Compared to FiFTy, our models have 6.3x fewer FLOPS
for 4096-byte fragments and 87x fewer for 512-byte fragments. In Scenario 1,
for example, with a fragment size of 4096-bytes, DSC has 164.88, DSC-SE has
164.96, and M-DSC has 89.07 MFLOPs, while FiFTy has 1047.59 MFLOPs. These
results demonstrate the effectiveness of our proposed models, as they require
fewer computations to make predictions while still maintaining high accuracy.
This makes our models more suitable for real-world applications where
computational resources are limited.

TABLE IV: Comparison between FiFTy and our models for floating point
operations (Mega FLOPs)

Scenario Fragment FLOPs size DSC DSC-SE M-DSC FiFTy 1 4096 164.88 164.96 89.07
1047.59 512 20.63 20.64 11.15 1801.71 2 4096 164.86 164.95 89.05 1327.90 512
20.61 20.63 11.13 918.06 3 4096 164.86 164.95 89.05 647.78 512 20.61 20.63
11.14 3579.57 4 4096 164.86 164.95 89.05 2378.51 512 20.61 20.63 11.13 1576.71
5 4096 164.86 164.95 89.05 488.37 512 20.61 20.63 11.13 2330.48 6 4096 164.86
164.95 89.05 1126.00 512 20.61 20.63 11.13 611.30

Accuracy. The confusion matrix of file types grouped by use-cases for Scenario
1 is plotted in Figure 6 (The grouping is listed in Table I). Most of the
misclassification occurred in archived file group due to other file types
embedded in them. The accuracy of the proposed models and FiFTy is listed in
Table V. In general, it can be observed that the accuracy of our models is
comparable to FiFTy. However, our models have faster inference time.

TABLE V: Comparison between FiFTy and our models in terms of accuracy

Scenario #of Fragment Accuracy files Size DSC DSC-SE M-DSC FiFTy 1 75 4096
78.45 79.27 78.68 77.04 512 65.89 66.33 64.04 65.66 2 11 4096 85.7 87.10 85.33
89.91 512 75.84 74.99 72.18 78.97 3 25 4096 93.06 93.32 91.87 94.64 512 80.79
80.79 78.53 87.97 4 5 4096 94.17 94.61 92.66 94.03 512 87.14 87.32 83.41 90.30
5 2 4096 99.28 99.37 99.24 99.12 512 98.94 98.96 98.94 99.07 6 2 4096 99.59
99.69 99.62 99.59 512 98.76 98.65 98.87 99.23

(a) DSC (78.45%)

(b) DSC-SE (79.27%)

(c) M-DSC (78.68%)

(d) FiFTy (77.04%)

(e) Baseline (70.51%)

(f) DSC (65.89%)

(g) DSC-SE (66.33%)

(h) M-DSC (64.04%)

(i) FiFTy (65.66%)

(j) Baseline (67.5%)

Figure 6: Confusion matrices for Our models, FiFTy[33] and Baseline RNN model
on 4 KB fragments in (a)-(e) and 512 bytes in (f)-(j). Due to large number of
classes(75), classes belonging to same super class were clustered into one.

###  IV-E Discussion

In summary, the results show that all three models have similar inference
times, with minimal variance between runs. This is in contrast to the results
reported in FiFTy [33], where the inference time of FiFTy can be significantly
larger. Our models are optimized for efficient inference by utilizing cutting-
edge techniques to minimize computational overhead and reduce latency. As a
result, we were able to achieve inference times that are comparable to state-
of-the-art models while still maintaining high accuracy. This demonstrates the
effectiveness of our proposed models in real-world applications where fast and
efficient inference is crucial.

Similar to previous work [33, 9], we observe that files with high entropy are
difficult to classify because there is no statistical trace that the
convolutional kernel can extract. Moreover, many files are container types
that contain other files as embedded objects, e.g., pdf files that can contain
embedded jpg images. As a result, classifiers behave erratically. Finally,
similar file types with different format, e.g., ppt, pptx, and key, are
misclassified amongst themselves. However, doc and docx are not affected by
this as docx uses XML whereas doc is stored as binary.

##  V Related Work

Several techniques have been proposed for identifying and classifying file
fragments. The techniques range from fundamental methods utilizing magic
numbers and file headers [11] to more advanced methods using machine learning
and deep learning [33, 5]. Generally, file carving methods can be broadly
grouped into three categories: 1) statistical methods, 2) machine learning
methods, and 3) deep learning methods. Statistical methods. Karresand et al.
proposed Oscar for determining the probable file type of binary data fragment
[26]. Oscar creates vector models based on the Byte Frequency Distribution
(BFD) of 4096-byte-sized fragments from different file types. A vector
distance between the mean and standard deviation of the segment to be
classified and the centroids is calculated. If the distance is lower than a
threshold value, the segment is classified as a modeling file. Sceadan is a
prominent open-source tool for file carving that utilizes a wide range of
statistical features including uni-grams, bi-grams, entropy, mean byte value,
and longest streak, among others [5]. Ten separate input vectors are generated
from these statistical features, which are then divided into four sets: uni-
grams, bi-grams, all global features aside from n-grams, and a subset of
global features. These sets are provided as inputs to the classification
models. Ahmed et al. [1] proposed an approach to identify file type based on
its content. To reduce the computation time for identification, two techniques
are applied. First, a subset of features based on occurrence frequency is
selected. Second, file blocks of a 100-byte size are sampled. It was observed
that this classifier performed well n low entropy file fragments (such as
plain-text files, uncompressed images, etc.) and failed on high entropy file
fragments(such as compressed files, binary executables, etc.).

Machine learning methods In [29], Li et al. proposed a Support Vector Machine
(SVM) model that is trained using a feature vector which is based on the
histogram of the data bytes. The SVM is utilized for binary classification,
where one class is fixed as JPEG and the other class varies, including DLL,
EXE, PDF, and MP3. Similarly, Fitzgerald et al. [13] presented an SVM model
for fragment classification, where file fragments were treated as a bag of
bytes represented by a feature vector that included uni-gram and bi-gram
counts, as well as statistical measures such as entropy. Zheng et al. [53]
proposed an SVM-based approach that used BFD or histogram and entropy as
feature vectors. Amirani et al. [4] proposed a hierarchical combination of
Principal Component Analysis (PCA) and Multi-layer Perceptron (MLP) for
feature extraction. The extracted features were fed to the classifier. PCA
selected N1 number of features from BFD of the raw features. These N1 features
were used to train an auto-associative neural network that extracts N2
features (N2$<$N1). Bhatt et al. [8] proposed a hierarchical classification
approach for file fragment classification utilizing SVM as base classifiers.
It was observed that the features used in the study are not adequate for
complex file types, as they do not characterize the membership relationship
between the file fragment and its parent class. The authors suggest refining
the hierarchy branch for complex files and engineering features that better
capture the membership relationship of a file fragment with its parent file
type. Haque and Tozal [17] proposed Byte2Vec, a novel feature generation model
that extends the word2vec concept to map file fragments to dense vector
representations. Byte2Vec generates vector representations utilizing the Skip-
gram model, and a k-Nearest Neighbors (kNN) classifier is trained on these
representations to identify fragment types. Byte2Vec corpus model works for
various block sizes and file types. Using a publicly available dataset, the
authors combine Byte2Vec with the kNN algorithm (Byte2Vec+kNN) for feature
extraction and classification.

Deep learning methods Wang et al. [50] proposed a one layer convolution with
multiple kernel sizes. The raw byte values were converted to their
corresponding binary representation and fed to the embedding layer to convert
the binary value to a continuous dense vector. The authors studied 20 file
types from the GovDocs1 dataset. Chen et al. [9] took a different approach and
converted the 4096-byte file fragment to a 64x63 gray-scale image. The
intuition was that data fragments from different files would have different
texture features, reflected in the gray-scale image. The gray-scale images
were fed to a deep CNN network like VGG [43] with many convolutions and max-
pool layers followed by dense classification. Hiester [18] compared feed-
forward, recurrent and convolutional neural networks with input fragments in
the form of binary representation (fragment bits). The investigated file types
were JPEG, GIF, XML and CSV. On these easily distinguishable file types, only
recurrent networks gave satisfactory results. They emphasized lossless
representation for achieving high accuracy, but this binary representation
increases the input’s dimensionality.

Mittal et al. [33] proposed FiFTy, an open-source convolutional neural network
for file fragment type identification. An open-source dataset was also
developed by them, which is reported to be the largest open source dataset for
file fragment classification. A compact neural network was developed that used
trainable embedding space and convolutional neural networks. When compared
with neural network-based classifier [33, 50, 9] our model achieves better
accuracy on largest number of file types. We have only compared our models
with deep learning based methods as they can be accelerated using GPUs. Our
models far exceed recurrent neural network based classifier [18] by being 660x
faster. Compared to machine learning based classifier [5], which takes 9
min/GB, our model takes 0.05 min/GB.

##  VI Conclusion

In this paper, we proposed light-weight file fragment classification models
based on depthwise separable CNNs. The objective is to develop a model with
better inference time as compared to the best-of-the-art model, without
compromising on accuracy. In particular, we were able to design classification
models with around 100K parameters. The models were evaluated using FFT-75
dataset that includes different scenarios. The evaluation results show the
proposed models perform faster than the state-of-the-art file fragment
classification model for all scenarios without dramatically degrading the
accuracy. Several improvements can be made to increase the classification
accuracy for classes with high misclassification rate. Without hardware
constraints, neural architecture search [54, 34] is the best method to design
an architecture for data-specific models. Moreover, for simple scenarios like
classifying JPEGs against other file types, redundant connections in the
neural network can be removed using in-network distillation [19, 47].

##  VII Acknowledgement

The authors would like to acknowledge the Interdisciplinary Research Center
for Intelligent Secure Systems at KFUPM.

## References

  * [1] Ahmed, I., Lhee, K.S., Shin, H.J., Hong, M.P.: Fast content-based file type identification. Advances in Digital Forensics VII IFIP Advances in Information and Communication Technology p. 65–75 (2011). https://doi.org/10.1007978-3-642-24212-0_5
  * [2] Akiba, T., Sano, S., Yanase, T., Ohta, T., Koyama, M.: Optuna: A next-generation hyperparameter optimization framework. In: Proceedings of the 25rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2019)
  * [3] Alghafli, K., Yeun, C.Y., Damiani, E.: Techniques for measuring the probability of adjacency between carved video fragments: The vidcarve approach. IEEE Transactions on Sustainable Computing (2019)
  * [4] Amirani, M.C., Toorani, M., Mihandoost, S.: Feature-based type identification of file fragments. Security and Communication Networks 6, 115–128 (2013)
  * [5] Beebe, N.L., Maddox, L.A., Liu, L., Sun, M.: Sceadan: using concatenated n-gram vectors for improved file and data type classification. IEEE Transactions on Information Forensics and Security 8(9), 1519–1530 (2013)
  * [6] Bennett, D.: The challenges facing computer forensics investigators in obtaining information from mobile devices for use in criminal investigations. Information Security Journal: A Global Perspective 21(3), 159–168 (2012)
  * [7] Bergstra, J.S., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Shawe-Taylor, J., Zemel, R.S., Bartlett, P.L., Pereira, F., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 24, pp. 2546–2554. Curran Associates, Inc. (2011)
  * [8] Bhatt, M., Mishra, A., Kabir, M.W.U., Blake-Gatto, S., Rajendra, R., Hoque, M.T., Ahmed, I.: Hierarchy-based file fragment classification. Machine Learning and Knowledge Extraction 2(3), 216–232 (2020)
  * [9] Chen, Q., Hui, L.C., Liu, D., Zhang, E., Liao, Q., Jiang, Z.L., Fang, J., Yiu, S., Xi, G., Li, R.e.a.: File fragment classification using grayscale image conversion and deep learning in digital forensics. 2018 IEEE Security and Privacy Workshops (SPW) (2018). https://doi.org/10.1109/spw.2018.00029
  * [10] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. pp. 1800–1807 (07 2017). https://doi.org/10.1109/CVPR.2017.195
  * [11] Darwin, I.F.: Libmagic (2008), ftp://ftp.astron.com/pub/file
  * [12] Dunne, R.A., Campbell, N.A.: On the pairing of the softmax activation and cross-entropy penalty functions and the derivation of the softmax activation function. In: Proc. 8th Aust. Conf. on the Neural Networks, Melbourne. vol. 181, p. 185. Citeseer (1997)
  * [13] Fitzgerald, S., Mathews, G., Morris, C., Zhulyn, O.: Using nlp techniques for file fragment classification. Digital Investigation 9 (2012). https://doi.org/10.1016/j.diin.2012.05.008
  * [14] Fukushima, K.: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol. Cybern. 36, 193–202 (1980)
  * [15] Garfinkel, S., Farrell, P., Roussev, V., Dinolt, G.: Bringing science to digital forensics with standardized forensic corpora. Digital Investigation 6 (2009). https://doi.org/10.1016/j.diin.2009.06.016
  * [16] Garfinkel, S.L.: Carving contiguous and fragmented files with fast object validation. digital investigation 4, 2–12 (2007)
  * [17] Haque, M.E., Tozal, M.E.: Byte embeddings for file fragment classification. Future Generation Computer Systems 127, 448–461 (2022)
  * [18] Hiester, L.: File fragment classification using neural networks with lossless representations (2018)
  * [19] Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
  * [20] Hochreiter, S.: The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 6(02), 107–116 (1998)
  * [21] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation 9(8), 1735–1780 (1997)
  * [22] Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., et al.: Searching for mobilenetv3. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 1314–1324 (2019)
  * [23] Howard, A., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications (04 2017)
  * [24] Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7132–7141 (2018)
  * [25] Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning. pp. 448–456. pmlr (2015)
  * [26] Karresand, M., Shahmehri, N.: Oscar - file type identification of binary data in disk clusters and ram pages. In: SEC (2006)
  * [27] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [28] Lei, Z.: Forensic analysis of unallocated space. Ph.D. thesis, UOIT (2011)
  * [29] Li, Q., Ong, A., Suganthan, P., Thing, V.: A novel support vector machine approach to high entropy data fragment classification (01 2010)
  * [30] Lin, X.: File carving. In: Introductory Computer Forensics, pp. 211–233. Springer (2018)
  * [31] Marziale, L., Richard III, G.G., Roussev, V.: Massive threading: Using gpus to increase the performance of digital forensics tools. digital investigation 4, 73–81 (2007)
  * [32] Memon, G.M.P.K.N.: File fragment type (fft) - 75 dataset (2019). https://doi.org/10.21227/kfxw-8084, http://dx.doi.org/10.21227/kfxw-8084
  * [33] Mittal, G., Korus, P., Memon, N.: Fifty: large-scale file fragment type identification using convolutional neural networks. IEEE Transactions on Information Forensics and Security 16, 28–41 (2020)
  * [34] Pham, H., Guan, M.Y., Zoph, B., Le, Q.V., Dean, J.: Efficient neural architecture search via parameter sharing. arXiv preprint arXiv:1802.03268 (2018)
  * [35] Poisel, R., Tjoa, S.: A comprehensive literature review of file carving. In: 2013 International conference on availability, reliability and security. pp. 475–484. IEEE (2013)
  * [36] Povar, D., Bhadran, V.: Forensic data carving. In: International Conference on Digital Forensics and Cyber Crime. pp. 137–148. Springer (2010)
  * [37] Pratt, L.Y.: Discriminability-based transfer between neural networks. In: Advances in neural information processing systems. pp. 204–211 (1993)
  * [38] Rafique, M., Khan, M.: Exploring static and live digital forensics: Methods, practices and tools. International Journal of Scientific & Engineering Research 4(10), 1048–1056 (2013)
  * [39] Richard III, G.G., Roussev, V.: Scalpel: A frugal, high performance file carver. In: DFRWS. Citeseer (2005)
  * [40] Saaim, K.M., Felemban, M., Alsaleh, S., Almulhem, A.: Light-weight file fragments classification using depthwise separable convolutions. In: ICT Systems Security and Privacy Protection: 37th IFIP TC 11 International Conference, SEC 2022, Copenhagen, Denmark, June 13–15, 2022, Proceedings. pp. 196–211. Springer (2022)
  * [41] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (6 2018)
  * [42] SIfre, L., Mallat, S.: Rigid-motion scattering for texture classification (03 2014)
  * [43] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014), http://arxiv.org/abs/1409.1556
  * [44] Sze, V., Chen, Y.H., Yang, T.J., Emer, J.S.: Efficient processing of deep neural networks: A tutorial and survey. Proceedings of the IEEE 105(12), 2295–2329 (2017)
  * [45] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Computer Vision and Pattern Recognition (CVPR) (2015), http://arxiv.org/abs/1409.4842
  * [46] Tan, M., Le, Q.V.: Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.11946 (2019)
  * [47] Tian, Y., Krishnan, D., Isola, P.: Contrastive representation distillation. arXiv preprint arXiv:1910.10699 (2019)
  * [48] Vulinović, K., Ivković, L., Petrović, J., Skračić, K., Pale, P.: Neural networks for file fragment classification. In: 2019 42nd International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO). pp. 1194–1198. IEEE (2019)
  * [49] Wang, F., Quach, T.T., Wheeler, J., Aimone, J.B., James, C.D.: Sparse coding for n-gram feature extraction and training for file fragment classification. IEEE Transactions on Information Forensics and Security 13(10), 2553–2562 (2018)
  * [50] Wang, Y., Su, Z., Song, D.: File fragment type identification with convolutional neural networks. Proceedings of the 2018 International Conference on Machine Learning Technologies - ICMLT 18 (2018). https://doi.org/10.1145/3231884.3231889
  * [51] Wu, Y., He, K.: Group normalization. In: Proceedings of the European conference on computer vision (ECCV). pp. 3–19 (2018)
  * [52] Zhang, X., Zhou, X., Lin, M., Sun, J.: Shufflenet: An extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6848–6856 (2018)
  * [53] Zheng, N., Wang, J., Wu, T., Xu, M.: A fragment classification method depending on data type. 2015 IEEE International Conference on Computer and Information Technology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure Computing; Pervasive Intelligence and Computing pp. 1948–1953 (2015)
  * [54] Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 8697–8710 (2018)