# Multimedia Distribution Process Tracking for Android and iOS

Yu-Min Jeon  9th Best of the Best program, KITRI
Seoul, South Korea
Email<EMAIL_ADDRESS>Won-Mu Heo  School of Computer Software,
Daegu Catholic University,
South Korea
Email<EMAIL_ADDRESS>Jong-Min Kim  Department of Computer
Science and Engineering,
Korea University, South Korea
Email<EMAIL_ADDRESS>Kyounggon Kim  Center of Excellence in Cybercrime
and Digital Forensics,
Naif Arab University for Security Sciences,
Riyadh, Kingdom of Saudi Arabia
Email<EMAIL_ADDRESS>
###### Abstract

The crime of illegally filming and distributing images or videos worldwide is
increasing day by day. With the increasing penetration rate of smartphones,
there has been a rise in crimes involving secretly taking pictures of people’s
bodies and distributing them through messengers. However, little research has
been done on these related issue. The crime of distributing media using the
world’s popular messengers, WhatsApp and Telegram, is continuously increasing.
It is also common to see criminals distributing illegal footage through
various messengers to avoid being caught in the investigation network. As
these crimes increase, there will continue to be a need for professional
investigative personnel, and the time required for criminal investigations
will continue to increase. In this paper, we propose a multimedia forensic
method for tracking footprints by checking the media information that changes
when images and videos shot with a smartphone are transmitted through instant
messengers. We have selected 11 of the world’s most popular instant messengers
and two secure messengers. In addition, we selected the most widely used
Android and iOS operating systems for smartphones. Through this study, we were
able to confirm that it is possible to trace footprints related to the
distribution of instant messengers by analyzing transmitted images and videos.
Thus, it was possible to determine which messengers were used to distribute
the video when it was transmitted through multiple messengers.

##  I Introduction

As the penetration rate of smartphones has increased, crimes such as
simulation and fraud targeting on Online Social Networks (OSNs) using
smartphones have also evolved. Patel et al. conducted research indicating that
social media and messaging platforms are currently being used as tools for
cybercrime. Therefore, further research on OSNs and messaging platform is
considered important [1].

Existing research on criminal investigation focuses on artifact extraction
from specific apps, text mining, and image quality analysis. While existing
studies allow for the retrieval of records transmitted through artifacts in a
specific app, there is insufficient research on identifying the source and
path of media, such as images and videos, when they are not recorded on the
device. This gap shows that current methods may be inadequate for responding
to the distribution of illegally filmed footage, which is a type of crime. The
spread of pornography is particularly concerning, as it can be easily shared
in messenger chat rooms. According to the [2] report, a recent study found
that more than 42% of adults use two or more social media platforms in their
daily lives. As a result, there is a high likelihood that such content will be
shared again on other applications.

Given that it is impossible for investigators to manually check each image or
video, it is necessary to identify the source and route of the media through
media analysis. In this study, we examined a method for investigating the
source of an image received through a messenger on a mobile device. This
information is crucial for investigators to trace the origin of media found on
a suspect’s mobile device. This is because the media found on the device was
either received from someone else or obtained from elsewhere, unless it was
filmed using the device itself.

In the initial stages of this study, we considered four questions: i) What
media artifacts are left behind in a specific app?; ii) Do artifacts differ
for each smartphone?; iii) Is it possible to identify the platform through
which media has been transmitted?; iv) How could the alteration of media
evidence through transfers affect criminal investigations?

To the best of our knowledge, there is no other study that analyzes the media
changes that occur when a specific messenger is passed through multiple
messenger platforms. The contributions of this paper are as follows:

  1. 1.

First, we have classified encoded media artifacts for eleven of the most
popular messengers and two secure messengers.

  2. 2.

Second, we analyzed the changes in status when media encoded once in a
messenger passes through another messenger.

  3. 3.

Third, we suggested investigative methods based on the analysis results.

The remainder of this paper is structured as follows: Section 2 introduces
existing research, Section 3 presents our research methodology, Section 4
evaluates the study results, and finally, Section 5 summarizes the conclusions
and discusses future directions.

##  II Related work

Existing Instant Messenger (IM)-related studies have mainly focused on
artifact analysis, image quality analysis, and text mining research that
analyzes text in messengers [3]. Over the past 10 years, with the increasing
popularity of social networking services (SNS) and messaging apps, many
studies on artifact analysis have been conducted [4]. These studies have
successfully identified artifacts for specific apps, such as an account
information, text messages, photos, calls, and chats, especially on Android
compared to iOS.

[5] showed that user configurations containing information related to user
accounts, follower accounts, and close friend accounts were identified for
Instagram. [6] showed evidence collected, including text, photo, video
transmission, and calls from Facebook Messenger. Similar studies have been
conducted in other applications such as Discord ([7]), LINE ([8, 9, 10, 11]),
WhatsApp ([12, 13]), Telegram ([14, 15, 16, 17, 18]), KakaoTalk ([19, 20]),
Facebook ([21]), Skype ([22, 23]), Signal ([24, 25]), WeChat ([26, 27]), QQ
([28, 29]), ChatON ([30]), Kik ([31, 32]), and Wickr ([33, 34, 35]).

As the studies mentioned above show, artifact analysis for a specific app is
currently in progress. However, since there are many different messaging apps
and most people use several types of messengers, further research is needed.
[36] addressed that the issue of message transmission of PNG images, which is
secured using end-to-end encryption and compressed according to predefined
rules. This study analyzed PNG image compression and alpha channel by
comparing and analyzing PNG images that were sent through WhatsApp and PNG
images before transmission. However, it did not deal with rhetorical meaning.

On the other hand, [37] built a web system for WhatsApp that uses text mining
methods to provide information about instructions related to the intervention
of others. The method used for message identification is to calculate the
weight of each word for the transmitted document using TF-IDF and cosine
similarity.

##  III Methodology

Through the analysis of media disseminated via instant messengers, we proposed
a methodology to identify footprints related to the characteristic “the media
is distributed from a certain messenger”. We conducted scenarios assuming that
the media shot on the smartphone was distributed through the application. In
addition, there are several processes involved in dissemination of media, such
as cloud, drive, USB, and software, and the specific dissemination process is
unknown. However, by checking the media artifacts left by a specific
messenger, it is possible to track the distributed messenger through
footprinting.

We used MediaInfo version 20.09 as a media analysis tool. iOS supports both
high-efficiency formats and high-compatibility formats. However, since the
high-efficiency format requires an OS version of 11 or higher and an iPhone 7
or higher, we conducted the experiment by selecting a high-compatibility
format to further generalize the experimental results. For Android, we
experimented with two methods: the default 4:3 (12MP) 4032x3024 resolution
mainly used in the new Galaxy S series and the default 16:9 (16MP) 5312x2988
resolution mainly used in the old Galaxy Note series. The original information
on image media is shown in Table I.

TABLE I: Original image information Distribution |  iOS |  Android
---|---|---
Extension |  JPG |  JPG
Format |  JPEG |  JPEG
Resolution |  4032x3024 |  4032x3024 5312x2988

We set the camera options in iOS to 1080p HD – 30fps, and on Android to FHD
1920x1080 resolution. The main distinguishing features between iOS and Android
are the extensions and Codec IDs. For extensions, iOS uses the MOV extension
and Android uses the mp4 extension. Codec ID are presented in the form of
ftypqt for iOS and ftypmp42 for Android at the File Signature using HxD, as
shown in Figure 1. Additionally, each video has a specific file size; for
instance, a 30-second video is about 50-60 MB in size. The video format
profile can be divided into High (high quality), Main (medium quality), and
Baseline (low quality). Table II shows the original information on video
media.

(a)

(b)

Figure 1: Codec ID (Above iOS, Below Android) TABLE II: Original video information Distribution |  iOS |  Android
---|---|---
Extension |  MOV |  mp4
Format |  MPEG-4 |  MPEG-4
Format Profile |  QuickTime |  Base Media Version 2
Codec ID |  qt |  mp42 (isom/mp42)
Video Format Profile |  High@L4 |  High@L4
Resolution |  1920x1080 |  1920x1080

##  IV Experiment

We have summarized the changes made to images and videos sent from a total of
13 instant messengers for Android and iOS, which are smartphone operating
systems. In this study, KakaoTalk and Facebook Messenger were selected as the
N-th messengers, and media information that was changed when media was
transmitted to the rest of the messengers as the N+1st messenger was further
analyzed.

###  IV-A Experiment environment

For the media artifact analysis, we have conducted experiment on smartphones
(Android, iOS), which are devices typically used for instant messengers. Table
III shows the targeted test devices and versions.

TABLE III: Experimental device and version Manufacurer |  Model name |  OS Version
---|---|---
Apple |  A1687 |  14.2
Apple |  A1905 |  14.2
SAMSUNG |  SM-N916L |  6.0.1
SAMSUNG |  SM-G906K |  6.0.1
SAMSUNG |  SM-G930S |  8.0.0

In this study, we analyzed the changes in media on 13 platforms, including
popular messengers like WhatsApp, Facebook Messenger, and Telegram, as well as
some secure and lesser-known platforms. The mobile messenger applications used
in the experiment are listed in Table IV.

TABLE IV: Mobile messenger application information Application name |  Version (iOS) |  Version (Android)
---|---|---
KakaoTalk |  9.1.2 |  9.1.3
Facebook Messenger |  291.2 |  291.2.0.22.114
Facebook |  297.0 |  297.0.0.36.116
Instagram |  167.0 |  167.1.0.25.120
WhatsApp |  2.20.121 |  2.20.205.16
WeChat |  7.0.18 |  7.0.17
Telegram |  7.2.1 |  7.2.1
Skype |  8.66.76 |  8.66.0.76
Discord |  49.0 |  49.16
NateOn |  4.0.5 |  4.0.8
LINE |  10.19.0 |  10.20.1
Signal |  3.22.0 |  4.78.5
Wickr Me |  5.66.10 |  5.66.8
TABLE V: Software and web browser information Application name |  Version (Software) |  Browser (Web)
---|---|---
KakaoTalk |  3.1.9.2626 |
WhatsApp |  2.2047.11 |
WeChat |  3.0.0.57 |
Telegram |  2.4.7 |
Skype |  15.66.77.0 |
Discord |  0.0.308 |
NateOn |  7.0.3.1 |
LINE |  6.4.0.2388 |
Signal |  1.37.2 |
Wickr Me |  5.65.4 |
Facebook Messenger |  |  Chrome
Facebook |  |  Chrome
Instagram |  |  Chrome

Facebook Messenger, Facebook, and Instagram are primarily used in a PC
environment through web browsers rather than software. As a result, we
downloaded media from 10 messengers using software and from 3 messengers
through a web browser. The software and web browser information used in the
study are presented in Table V.

TABLE VI: Change image in instant messengers Application name | Image Option | iOS | Android (4032x3024) | Android (5312x2988)
---|---|---|---|---
Width | Length | Width | Length | Width |  Length
KakaoTalk | General | 960x720 | 720x960 | 960x720 | 720x960 | 960x540 |  540x960
High | 1440x1080 | 1080x1440 | 1440x1080 | 1080x1440 | 1440x810 |  810x1440
Facebook Messenger | Default | Indistinguishable |  | 2048x1152
---
3984x2241
1151x2048 2240x3984
Facebook | Default | 960x720 | 720x960 | Indistinguishable
Instagram | Indistinguishable
WhatsApp | Default | 1600x1200 | 1200x1600 | 1600x1200 | 1200x1600 | 1328x747 |  747x1328
WeChat | General | 1440x1080 | 1080x1440 | 1080x1440 | 1080x1440 | 1920x1080 |  1080x1920
Telegram | General | 1280x960 | 960x1280 | 1280x960 | 960x1280 | 1280x960 |  960x1280
Skype | Default | 2048x1536 | 1536x2048 | 2016x1512 | 1512x2016 | 1992x1121 |  1121x1992
Discord | Indistinguishable
NateOn | Minimal | Indistinguishable | 664x374 |  374x664
Standard | 1328x747 |  747x1328
LINE | General | 1478x1108 | 1108x1478 | 1477x1108 | 1108x1477 | 1706x960 |  960x1706
Standard | 2365x1774 | 1774x2365
Signal | General | 1536x1152 | 1152x1536 | Indistinguishable | 4096x2304 |  2304x4096
Wickr Me | Indistinguishable

###  IV-B Image change when sending messenger

Among the media information of the image, only the resolution of the image
transmitted through the messenger can be tracked through footprinting. The
reason is that when comparing Android image and iOS image media information,
the extension and format are the same, unlike video media. Thus, the only way
to differentiate them is by resolution. For each messenger platform, we
conducted experiments on all options such as normal quality, high definition,
and original. As a result of the experiment, there are many platforms that can
be tracked through footprinting using a specific resolution for each instant
messenger. However, there are several platforms where the original image is
transmitted without changing the resolution, or the resolution is changed
irregularly, making it impossible to track the footprint.

In the main text, we exluded the results of images with irregularly changed
resolution or transmitted as a general original image. There are also cases
where the error range of the image resolution result is about ±10, but these
are very rare. Table VI summarizes the image resolution changes for the 13
messenger platforms that were the subject of the experiment.

4.2.1. KakaoTalk
KakaoTalk offers three image quality options when transmitting images: normal,
high quality, and original. Through our experiments, we found that normal and
high-quality options, except for the original, can be traced through
footprinting using a specific resolution. However, the original option has the
same resolution as the normal original image, making it impossible to track
through the footprint. Furthermore, since the resolution of media transmitted
from iOS and Android (4032x3024) is the same, it is impossible to distinguish
between the operating systems.

4.2.2. Facebook Messenger
In Facebook Messenger, there is no option to select image quality, thus only
basic image quality is available. For both iOS and Android (4032x3024), the
resolution changes irregularly or is the same as the normal original image
resolution, making it impossible to track through footprints. However, since
Android (5312x2988) uses a specific resolution, it can be tracked by
footprint. The resolution result has two outcomes, which is different from
other messengers. Although the result depends on the process of media
dissemination, if the same result as shown in Table VII is obtained, it can be
identified as media transmitted through Facebook Messenger.

4.2.3. Facebook
Facebook does not have a quality option, so only the default image quality
exists. In iOS, tracking through footprints is possible using a specific
resolution. However, in Android, the resolution changes irregularly, making
tracking through footprints impossible. From the resolution result, it can be
seen that the normal resolution of KakaoTalk and the default resolution of
Facebook are the same. When the same resolution appears, it is possible to
distinguish it by a specific platform through an additional file size. The
file size of KakaoTalk general definition media is about 100KB, and the file
size of Facebook basic definition media is about 50KB. The error range of the
two files is about ±10KB.

4.2.4. Instagram
Instagram does not have a quality option, so only the default image quality
exists. Additionally, both iOS and Android have irregular resolutions, making
tracking through footprints impossible.

4.2.5. WhatsApp
WhatsApp does not have a quality option, so only the default image quality
exists. Both iOS and Android use specific resolutions to enable tracking
through footprints. Additionally, the resolution of the media transferred from
iOS and Android (4032x3024) is the same, making it impossible to distinguish
the operating system.

4.2.6. WeChat
WeChat allows to choose between two options, normal quality and full quality
to transmit. The general image quality can be traced through footprinting
using a specific resolution. However, the overall image quality cannot be
distinguished because it is the same as the general original image resolution,
and tracking through footprinting is impossible. Through the resolution result
analysis, it can be seen that the WeChat normal resolution and KakaoTalk high-
definition resolution produce the same result. As described above, when the
same resolution is obtained, it is possible to distinguish a specific platform
through an additional file size.

The file size of KakaoTalk high-definition media is about 500KB, and the file
size of WeChat normal-definition media is about 200KB. The error range of the
two files is about ±100KB. There are cases where the results may differ
depending on the dissemination process, but it is very rare.

4.2.7. Telegram
Telegram can be transmitted by selecting two options: normal quality and
original. General image quality can be tracked through footprinting using a
specific resolution. However, since the original option is the same as the
normal original image resolution and cannot be distinguished, it is impossible
to trace through the footprint. Also, since the media transferred from iOS and
Android have the same resolution, it is impossible to distinguish between
operating systems.

4.2.8. Skype
Skype has no picture quality options, thus only basic picture quality exists.
Both iOS and Android use specific resolutions to enable tracking through
footprints.

4.2.9. Discord
In Discord, when downloading an image, it can be chosen between the preview
option and the original option. In the preview option, both iOS and Android
change resolution irregularly, thus tracking through footprints is impossible.
And since the original option produces the same result as the normal original
image resolution, tracking through both options is impossible through
footprints.

4.2.10. NateOn
When sending an image in NateOn, iOS users can choose between two options:
normal image quality and original image quality, while Android users can
select from three options: minimum quality, standard quality, and original
quality. For both iOS and Android with a resolution of 4032x3024, the
resolution changes irregularly or is the same as the normal original image
resolution, making tracking through footprints impossible. However, Android
with a resolution of 5312x2988 uses a specific resolution, allowing it to be
tracked through footprints.

4.2.11. LINE
In LINE, it is possible to select from three options: normal quality, standard
quality, and original quality. Since both iOS and Android use specific
resolutions for normal and standard quality, tracking through footprint is
possible. However, since the original image quality is the same as the general
original image resolution, it is impossible to track through footprinting.

4.2.12. Signal
Signal only has basic image quality as it does not provide quality options.
Both iOS and Android allow tracking through footprints using specific
resolutions (5312x2988). However, tracking through footprint is impossible in
Android (4032x3024) as it’s the same as the original image resolution.

4.2.13. Wickr Me
In Wickr Me, only basic image quality exists as there is no option for image
quality. Additionally, tracking through footprints is impossible as both iOS
and Android have the same resolution as the normal original image.

TABLE VII: iOS video changes in instant messengers

| Application
---
name
| Video
---
Quality
Extension | Format Profile | Codec ID |  | Video Format
---
Profile
Width | Length | Encoder |  | Additional
---
Information
KakaoTalk | General | mp4 | Base Media | isom (isom/iso2/avc1/mp41) |<EMAIL_ADDRESS>| 720x404 | 404x720 | Lavf57.56.101 |
Base Media Version 2 | mp42 (isom/mp41/mp42) |  |
High | mp4 | Base Media | isom (isom/iso2/mp41) | Main@L4@Main | 1920x1080 | 1920x1080 | Lavf57.83.100 |
Base Media Version 2 | mp42 (isom/mp42) | Baseline@L4 |  |
| Facebook
---
Messenger
Default | mp4, MOV | Base Media | isom (isom/iso2/avc1/mp41) |<EMAIL_ADDRESS>| 1280x720 | 720x1280 | Lavf58.20.100 |
Facebook | Default | mp4 | Movie name
Instagram | Default | mp4, MOV | Base Media | isom (isom/iso2/avc1/mp41) | Baseline@L3 |  | 480x270
---
640x360
| 480x854
---
640x1138
Lavf58.20.100 |
WhatsApp | Default | mp4 | Base Media Version 2 | mp42 (mp42/isom) |<EMAIL_ADDRESS>| 848x480 | 848x480 |  |
WeChat | Default | mp4 | Base Media Version 2 | mp42 (isom/mp41/mp42) |<EMAIL_ADDRESS>| 960x544 | 544x960 |  | Movie More
Telegram | 1080p | MOV | Base Media Version 2 | mp42 (isom/mp41/mp42) | High@L4 | 1920x1072 | 1072x1920 |  |
720p |<EMAIL_ADDRESS>| 1280x720 | 720x1280 |  |
480p |<EMAIL_ADDRESS>| 848x464 | 464x848 |  |
360p | High@L3 | 640x352 | 352x640 |  |
240p |<EMAIL_ADDRESS>| 480x256 | 256x480 |  |
Skype | Default | MOV | Base Media Version 2 | mp42 (isom/mp41/mp42) |<EMAIL_ADDRESS>| 1280x720 | 720x1280 |  |
Discord | Default | MOV | QuickTime | qt |<EMAIL_ADDRESS>| 960x540 | 960x540 |  |
NateOn | Default | mp4 | Base Media Version 2 | mp42 (isom/mp41/mp42) | High@L4 | 1920x1080 | 1920x1080 |  | Recorded date
LINE | Default | mp4 | Base Media Version 2 | mp42 (isom/mp41/mp42) | High@L4 | 960x540 | 960x540 |  | Movie name
Signal | Default | mp4 | Base Media Version 2 | mp42 (isom/mp41/mp42) | Main@L3 | 568x320 | 568x320 |  |
Wickr Me | Default | MOV | QuickTime | qt | Main@L3 | 568x320 | 568x320 |  |

TABLE VIII: Android video changes in instant messengers

| Application
---
name
| Video
---
Quality
Extension | Format Profile | Codec ID |  | Video Format
---
Profile
Width | Length | Encoder |  | Additional
---
Information
KakaoTalk | General | mp4 | Base Media | isom (isom/iso2/avc1/mp41) | Baseline@L3 | 852x480 | 852x480 | Lavf57.56.101 |
Base Media Version 2 | mp42 (isom/mp41/mp42) |<EMAIL_ADDRESS>|  |
High | mp4 | Base Media | isom (isom/iso2/mp41) | Main@L4@Main | 1920x1080 | 1920x1080 | Lavf57.83.100 |
Base Media Version 2 | mp42 (isom/mp42) | Baseline@L4 |  |
| Facebook
---
Messenger
Default | mp4 | Base Media | isom (isom/iso2/avc1/mp41) |  | Main@L3
---
Main@L4
Resolution irregularity | Lavf58.20.100 |
Facebook | Default | mp4 | Baseline@L3 | 400x224 | 224x400 | Lavf56.40.101 | Movie name
Instagram | Default | mp4 | Base Media | isom (isom/iso2/avc1/mp41) | Baseline@L3 |  | 480x270
---
640x360
| 480x852
---
640x1136
Lavf58.20.100 |
WhatsApp | Indistinguishable
WeChat | Default | mp4 | Base Media Version 2 | mp42 (isom/mp41/mp42) |<EMAIL_ADDRESS>| 960x544 | 544x960 |  | Copyright
Telegram | High | mp4 | Base Media | isom (isom/iso2/avc1/mp41) |  | High@L4
---
Baseline@L4
1920x1080 | 1080x1920 |  |
Medium |<EMAIL_ADDRESS>| 1280x720 | 720x1280 |  |
| Between
---
medium and low
<EMAIL_ADDRESS>| 854x480 | 480x854 |  |
Low |<EMAIL_ADDRESS>| 480x270 | 270x480 |  |
Skype | Default | mp4 | Base Media Version 2 | mp42 (isom/mp42) |<EMAIL_ADDRESS>| 1280x720 | 720x1280 |  |
Discord | Indistinguishable
NateOn | Indistinguishable
LINE | Default | mp4 | Base Media Version 2 | mp42 (isom/mp42) |<EMAIL_ADDRESS>| 960x540 | 540x960 |  |
Signal | Default | mp4 | Base Media Version 2 | mp42 (isom/mp42) |<EMAIL_ADDRESS>| 1280x720 | 720x1280 |  |
Wickr Me | Indistinguishable

###  IV-C Video change when sending messenger

When transmitting a video through a messenger, we confirmed that it is
possible to track it through footprinting by analyzing various media
information such as extension, format profile, codec ID, video format profile,
resolution, encoder, and specific information. Most of the file sizes were
small, and the resolution changed during the encoding process on the messenger
platform. Some messenger platforms can provide encoder and specific
information for each platform. By synthesizing specific resolution, encoder
information, media information, and others for each instant messenger, it is
possible to distinguish artifacts of a specific messenger platform and track
them through footprinting. However, we also found a few platforms where media
information was consistent with the original video, making tracking through
footprinting impossible. We analyzed the video using the default resolution of
1920x1080 for both iOS and Android, excluding results that were the same as
the original video in the main text. Table VII and Table VIII summarize the
changes in video for the 13 instant messenger platforms that were tested.

4.3.1. KakaoTalk
KakaoTalk can be transmitted by selecting between two options: normal quality
and high quality. Although the results may differ depending on the
dissemination process, the disseminated media always has one of the two
results. If an encoder is used, different encoder information can be checked
for each image quality, and a specific encoder unique to KakaoTalk has been
confirmed. Once encoded files are redistributed within the messenger, there is
little change in the files. If an encoder is used, KakaoTalk’s specific
encoder, ‘Lavf57.56.101’ for normal quality and ‘Lavf57.83.100’ for high
quality, can be confirmed. When distributing through KakaoTalk, artifacts such
as resolution and other media information can be aggregated and tracked
through footprinting.

4.3.2. Facebook Messenger
In Facebook Messenger, there is no video quality option, thus only basic video
quality exists. Facebook Messenger has ‘Lavf58.20.100’ information, which is
an encoder common to Facebook and Instagram. In addition, iOS changes
resolution regularly, while Android changes resolution irregularly. When
distributing through Facebook Messenger, Android cannot specify the
resolution. However, it can be specified by combining other media information,
and tracking through footprint is possible. Then, iOS aggregates artifacts
such as resolution and other media information, and based on the results,
tracking through footprint is possible.

4.3.3. Facebook
Facebook does not have video quality options, thus only default image quality
exists. As with KakaoTalk described above, the results vary depending on the
dissemination process, however distributed videos always produce one of two
results. Facebook has the same encoder information as Facebook Messenger and
Instagram, and Facebook’s specific information such as ‘Movie name’ leaves
information related to the post number. When distributing via Facebook,
artifacts such as resolution and other media information can be aggregated and
traced through footprint based on the results.

4.3.4. Instagram
Instagram does not have video quality options, thus only default image quality
exists. Instagram has the same encoder information as Facebook Messenger or
Facebook. As a result, Facebook Messenger, Facebook, and Instagram have the
same encoder information but can be classified as specific platforms by
comprehensively checking media information such as codec ID and video format
profile. When distributing through Instagram, artifacts such as resolution and
other media information can be aggregated and tracked through footprinting.

4.3.5. WhatsApp
WhatsApp does not have video quality options, thus only default quality
exists. iOS can be identified as a WhatsApp platform, and therefore tracked
through footprints. However, since Android has similar media information to
general original video, it is impossible to distinguish it as a specific
platform, and therefore impossible to track it through footprints. When
distributing via WhatsApp, only iOS can aggregate artifacts such as resolution
and other media information and track them through footprints.

4.3.6. WeChat
WeChat does not have video quality options, thus only basic image quality
exists. Both iOS and Android versions of WeChat have specific information
unique to WeChat that can be checked. In iOS, information related to the
WeChat version called “Movie More” remains, and in Android, information
related to copyright called “Copyright” remains. When distributing through
WeChat, artifacts such as resolution and other media information can be
aggregated and tracked through footprinting.

4.3.7. Telegram
For Telegram, there are five video quality options for iOS: 1080p, 720p, 480p,
360p, and 240p, and four video quality options for Android: high quality,
medium quality, medium-low quality, and low quality. In Telegram, only high-
definition horizontal videos have a video format profile of High on Android,
while the video format profile for the rest of the high-definition vertical
videos and the medium-, low-mid, and low-quality videos is the baseline. On
Android, high-definition vertical videos have a video format profile of the
baseline. As a result, Telegram has different resolutions and profiles for
each video quality, and the distributed video quality can be checked. When
disseminating videos through Telegram, artifacts such as resolution and other
media information can be aggregated and traced through footprinting.

4.3.8. Skype
Skype does not have video quality options, thus only basic quality exists.
Skype can be identified as specific platforms for both iOS and Android.
Therefore, when distributing through Skype, artifacts such as resolution and
other media information can be aggregated and tracked through footprinting.

4.3.9. Discord
Discord does not offer video quality options, thus only basic quality is
available. It is possible to classify iOS as a specific platform, but Android
transmits the original video without any modifications, making tracking
through footprints impossible. Therefore, when distributing through Discord,
only iOS can aggregate artifacts such as resolution and other media
information for tracking through footprinting.

4.3.10. NateOn
In NateOn, there are two video quality options: normal video quality and
original video quality in iOS, and only basic video quality without options
exists in Android. Both options can be distinguished by the specific platform
in iOS, but in Android, the original video is transmitted as it is and cannot
be distinguished, and tracking through footprinting is also impossible.
NateOn’s iOS-specific information will leave information related to the date
the video was recorded called ‘Recorded date’. When distributing through
NateOn, only iOS can aggregate artifacts such as resolution and other media
information and track them through footprinting.

4.3.11. LINE
In LINE, there are no video quality options, thus only basic video quality
exists. iOS can be classified as a specific platform regardless of the
dissemination process, but for Android, it depends on whether it can be
distinguished or not based on the dissemination process. In this text, we only
consider the case where the distinction is possible for Android. LINE videos
on iOS contain specific information called ‘Movie name’. When distributing
through LINE, Android media information can be tracked through footprinting
only if it can be identified by the LINE platform. Additionally, iOS can
aggregate artifacts such as resolution and other media information to enable
tracking through footprinting.

4.3.12. Signal
Signal does not offer video quality options, therefore only basic video
quality is available. Additionally, both iOS and Android platforms can be
identified specifically when using Signal. When distributing media through
Signal, information such as resolution and other artifacts can be aggregated
and tracked through footprinting.

4.3.13. Wickr Me
In Wickr Me, there are no video quality options, thus only basic video quality
exists. iOS can be identified as a specific platform, thus tracking through
footprints is possible. However, since Android transmits the original video as
it is and cannot distinguish it, tracking through footprints is also
impossible. When distributing through Wickr Me, only iOS can aggregate
artifacts such as resolution and other media information and track them
through footprints.

TABLE IX: Video media changes from messenger to messenger (KakaoTalk / iOS)

| Application
---
name
Extension | Format Profile | Codec ID |  | Video Format
---
Profile
Width | Length | Encoder |  | Additional
---
Information
Facebook Messenger | mp4 | Base Media | isom (isom/iso2/avc1/mp41) | Main@L3 | 720x404 | 404x720 | Lavf58.20.100 |
Base Media Version 2 | mp42 (isom/mp41/mp42) |<EMAIL_ADDRESS>| 720x404 | 720x404 |  |
Facebook | Base Media | isom (isom/iso2/avc1/mp41) | Main@L3 | 640x358 | 358x640 | Lavf58.20.100 | Movie name
Instagram | Indistinguishable
WhatsApp | mp4 | Base Media Version 2 | mp42 (mp42/isom) | Baseline@L3 | 720x404 | 720x404 |  |
WeChat | mp42 (isom/mp41/mp42) | High@L3 | 404x720 |  | Movie More
Telegram | Indistinguishable
Skype | mp4 | Base Media Version 2 | mp42 (isom/mp41/mp42) |<EMAIL_ADDRESS>| 720x404 | 720x404 |  |
Discord |  |
NateOn |  |
LINE | High@L4 | 404x720 |  | Line Video
Signal |<EMAIL_ADDRESS>| 480x268 |  |
Wickr Me | QuickTime | qt |<EMAIL_ADDRESS>|  |

TABLE X: Video media changes from messenger to messenger (KakaoTalk / Android)

| Application
---
name
Extension | Format Profile | Codec ID |  | Video Format
---
Profile
Width | Length | Encoder |  | Additional
---
Information
| Facebook
---
Messenger
mp4 | Base Media | isom (isom/iso2/avc1/mp41) | Main@L3 | 852x480 | 480x852 | Lavf58.20.100 |
Facebook | Movie name
Instagram | Indistinguishable
WhatsApp | mp4 | Base Media Version 2 | mp42 (mp42/isom) |<EMAIL_ADDRESS>| 852x480 | 852x480 |  |
WeChat |<EMAIL_ADDRESS>| 480x852 |  |
Telegram | Base Media | isom (isom/iso2/avc1/mp41) |<EMAIL_ADDRESS>|  | Copyright
Skype | Base Media Version 2 | mp42 (isom/mp42) |  |
Discord | 852x480 |  |
NateOn |  |
LINE |<EMAIL_ADDRESS>| 404x720 |  |
Signal | Indistinguishable
Wickr Me | mp4 | Base Media Version 2 | mp42 (isom/mp42) |<EMAIL_ADDRESS>| 852x480 | 852x480 |  |

###  IV-D Video media switching when sending multiple messages via messenger

In the previous section, we identified the artifacts of an instant messenger
by tracking footprints when video media was distributed once. In this section,
we analyze each messenger artifact when it’s transmitted from one messenger to
another, focusing on normal and basic image quality only. We chose KakaoTalk
and Facebook Messenger, two popular messengers in the world, as the N-th
distribution sites, and the rest of the messengers as the N+1 distribution
sites. Based on the study’s results, we investigate whether it’s possible to
track the N-th and N+1 distribution messengers through footprints.

4.4.1. N-th messenger KakaoTalk
This section focuses on the changes that occur when KakaoTalk is the N-th
messenger, and the remaining messengers are the N+1 messenger. The experiment
resulted in three cases. First, it’s possible to partially check the media
information of both the N-th and N+1st messengers, enabling tracking through
footprinting. Second, the media information of the N-th messenger is
overwritten with that of the N+1st messenger, making tracking impossible.
Finally, there’s a case where multiple final media information results are the
same, narrowing down the investigation network’s scope. The study excluded
results that couldn’t be traced through footprinting. Table IX and Table X
present the changes in the N-th messenger for KakaoTalk in the iOS and Android
environments.

TABLE XI: Video media changes from messenger to messenger (Facebook Messenger
/ iOS)

| Application
---
name
Extension | Format Profile | Codec ID |  | Video Format
---
Profile
Width | Length | Encoder |  | Additional
---
Information
KakaoTalk | Indistinguishable
Facebook | mp4 | Base Media | isom (isom/iso2/avc1/mp41) |<EMAIL_ADDRESS>| 1280x720 | 720x1280 | Lavf58.20.100 | Movie name
Instagram | Indistinguishable
WhatsApp | mp4 | Base Media Version 2 | mp42 (mp42/isom) |<EMAIL_ADDRESS>| 1280x720 | 720x1280 |  |
WeChat | Indistinguishable
Telegram | Indistinguishable
Skype | MOV | Base Media | isom (isom/iso2/avc1/mp41) |<EMAIL_ADDRESS>| 1280x720 | 720x1280 | Lavf58.20.100 |
Discord | QuickTime | qt | 960x540 | 540x960 |
NateOn | mp4 | Base Media Version 2 | isom (isom/mp41/mp42) | 1280x720 | 720x1280 |  |
LINE | Indistinguishable
Signal | Indistinguishable
Wickr Me | MOV | QuickTime | qt | Main@L3 | 568x320 | 320x568 | Lavf58.20.100 |

TABLE XII: Video media changes from messenger to messenger (Facebook Messenger
/ Android)

| Application
---
name
Extension | Format Profile | Codec ID |  | Video Format
---
Profile
Width | Length | Encoder
KakaoTalk | mp4 | Base Media | isom (isom/iso2/avc1/mp41) | Main@L4 | 1920x1080 | 1080x1920 | Lavf58.20.100
Instagram
WhatsApp
Telegram
Discord
NateOn
LINE
Wickr Me
Facebook | Indistinguishable
WeChat
Skype
Signal

4.4.2. N-th messenger Facebook Messenger
This section analyzes the changes that occur when passing through Facebook
Messenger as the N-th messenger, and the remaining messengers as the N+1st
messenger. The experiment resulted in three cases, similar to section 4.4.1.
First, it’s possible to partially check the media information of both the N-th
and N+1st messengers, enabling tracking through footprinting. Second, the
media information of the N-th messenger is overwritten with that of the N+1st
messenger, making tracking impossible. Finally, there’s a case where multiple
final media information results are the same, narrowing down the investigation
network’s scope. Unlike the KaKaoTalk case, there are cases where only the
investigation network’s scope can be narrowed in the Android environment for
Facebook Messenger. Some messengers also produced results that couldn’t be
traced through footprinting. Table XI and Table XII present the changes in the
N-th messenger for Facebook Messenger in the iOS and Android environments.

##  V Conclusion

We analyzed the experimental results related to footprint tracking through
media information analysis when sending images and video media in instant
messengers. The study confirmed that instant messengers work differently on
iOS and Android. Furthermore, when a messenger was transferred from one
messenger to another, it was possible to check which messengers were passed
due to the artifacts of the transmitted messengers remaining. From a digital
forensic perspective, we drew the following conclusions:

When an original or similar image exists, the distribution site of the
transmitted image can be determined through resolution comparison. Each
instant messenger has a different resolution for transmitted images, except
for messengers whose resolution is unchanged or irregular. Additionally, while
some messengers have the same resolutions on both iOS and Android, others have
different resolutions. Thus, it’s possible to track the distribution site of
the image through footprinting.

In the case where there is an original or a similar video, the distribution of
the transmitted video can be checked through media information comparison. In
the video transmitted from instant messengers, different media information
remains for each messenger, except for the messenger in which the media
information is not changed. In addition, unlike images, videos exist only when
the media information of iOS and Android is different for each messenger.
Therefore, it is possible to estimate the operating system of the distributed
smartphone, and it is possible to track the footprint in relation to the
distribution site of the video.

When a messenger is transmitted to another messenger, the distribution of the
transmitted videos can be confirmed or estimated by checking the media
information, and the scope of the investigation network can be narrowed. We
conducted an experiment with KakaoTalk and Facebook Messenger as the N-th
messengers, and the remaining messengers as the N+1st messenger. Finally, we
concluded three key findings based on our experiment.

  1. 1.

The media information of the N-th messenger and the media information of the
N+1st messenger remain partially. Therefore, it is possible to accurately
check which messengers have been passed through and track the distribution of
the transmitted videos through footprinting.

  2. 2.

The media information of the N-th messenger is overwritten with the media
information of the N+1st messenger. Therefore, it is impossible to confirm
exactly which messengers have been passed, and tracking through footprints is
not possible.

  3. 3.

This is a case where the final media information result is the same in many
cases. In this case, it is impossible to confirm exactly which messengers have
been passed through, however, it is possible to narrow the scope of the
investigation network.

We hope that this study will assist digital forensic investigators in swiftly
conducting criminal investigations pertaining to media distribution by
synthesizing the information presented and tracking the channels through which
the media was disseminated.

## References

  * [1] P. Patel, K. Kannoorpatti, B. Shanmugam, S. Azam, and K. C. Yeo, “A theoretical review of social media usage by cyber-criminals,” in _2017 International Conference on Computer Communication and Informatics (ICCCI)_. IEEE, 2017, pp. 1–6.
  * [2] R. Roshan, “Multiple social media accounts of single user,” https://medium.com/@rajroshan26j/multiple-social-media-accounts-of-single-user-71f2f1cde31d, 2020, [Online; accessed 28-September-2020].
  * [3] J. Son, Y. W. Kim, D. B. Oh, and K. Kim, “Forensic analysis of instant messengers: Decrypt signal, wickr, and threema,” _Forensic Science International: Digital Investigation_ , vol. 40, p. 301347, 2022.
  * [4] M. G. Campana and F. Delmastro, “Mydigitalfootprint: An extensive context dataset for pervasive computing applications at the edge,” _Pervasive and Mobile Computing_ , vol. 70, p. 101309, 2021.
  * [5] C. Alisabeth and Y. R. Pramadi, “Forensic analysis of instagram on android,” in _IOP Conference Series: Materials Science and Engineering_ , vol. 1007, no. 1. IOP Publishing, 2020, p. 012116\.
  * [6] M. S. Chang and C. P. Yen, “Evidence gathering of facebook messenger on android,” _International Journal of Network Security_ , vol. 22, no. 5, pp. 828–837, 2020.
  * [7] M. Motyliński, Á. MacDermott, F. Iqbal, M. Hussain, and S. Aleem, “Digital forensic acquisition and analysis of discord applications,” in _2020 International Conference on Communications, Computing, Cybersecurity, and Informatics (CCCI)_. IEEE, 2020, pp. 1–7.
  * [8] M. S. Chang and C. Y. Chang, “Line messenger forensics on windows 10,” _Journal of Computers_ , vol. 30, no. 1, pp. 114–125, 2019.
  * [9] I. Riadi, A. Fadlil, and A. Fauzan, “Evidence gathering and identification of line messenger on android device,” _Int. J. Comput. Sci. Inf. Secur.(IJCSIS)_ , vol. 16, no. 5, pp. 201–205, 2018.
  * [10] M. S. Chang and C. Y. Chang, “Forensic analysis of line messenger on android,” _Journal of Computers_ , vol. 29, no. 1, pp. 11–20, 2018.
  * [11] I. Riadi, S. Sunardi, and M. E. Rauli, “Live forensics analysis of line app on proprietary operating system,” _Kinetik: Game Technology, Information System, Computer Network, Computing, Electronics, and Control_ , pp. 305–314, 2019\.
  * [12] H. A. Ghannam, “Forensic analysis of artifacts of giant instant messaging “whatsapp” in android smartphone,” _Journal of Applied Information, Communication and Technology_ , vol. 5, no. 2, pp. 63–72, 2018.
  * [13] D. Orr and A. Castro, “Whatsapp messenger on the android platform: A forensic examination of a physical device,” _Forensic Science Methods and Tech_ , vol. 1, pp. 4–19, 2018.
  * [14] C. Anglano, M. Canonico, and M. Guazzone, “Forensic analysis of telegram messenger on android smartphones,” _Digital Investigation_ , vol. 23, pp. 31–49, 2017.
  * [15] G. B. Satrya, P. T. Daely, and M. A. Nugroho, “Digital forensic analysis of telegram messenger on android devices,” in _2016 International Conference on Information & Communication Technology and Systems (ICTS)_. IEEE, 2016, pp. 1–7.
  * [16] D. Hintea, A. Sangins, and R. Bird, “Forensic analysis of the telegram instant messenger application on android devices,” in _European Conference on Cyber Warfare and Security. Academic Conferences International Limited_ , vol. 12, 2018, pp. 217–223.
  * [17] J. Gregorio, A. Gardel, and B. Alarcos, “Forensic analysis of telegram messenger for windows phone,” _Digital Investigation_ , vol. 22, pp. 88–106, 2017.
  * [18] J. Gregorio, B. Alarcos, and A. Gardel, “Forensic analysis of telegram messenger desktop on macos,” _vol_ , vol. 6, pp. 39–48, 2018.
  * [19] J. Choi, J. Park, and H. Kim, “Forensic analysis of the backup database file in kakaotalk messenger,” in _2017 IEEE International Conference on Big Data and Smart Computing (BigComp)_. IEEE, 2017, pp. 156–161.
  * [20] J. Yoon and Y. Park, “Forensic analysis of chatting messenger service in kakaotalk and comparison study of kakaotalk and whatsapp artifacts,” _Journal of the Korea Institute of Information and Communication Engineering_ , vol. 20, no. 4, pp. 777–785, 2016.
  * [21] A. K. Agrawal, A. Sharma, and P. Khatri, “Digital forensic analysis of facebook app in virtual environment,” in _2019 6th International Conference on Computing for Sustainable Global Development (INDIACom)_. IEEE, 2019, pp. 660–664.
  * [22] M. I. Al-Saleh and Y. A. Forihat, “Skype forensics in android devices,” _International Journal of Computer Applications_ , vol. 78, no. 7, 2013.
  * [23] M. Nicoletti and M. Bernaschi, “Forensic analysis of microsoft skype for business,” _Digital Investigation_ , vol. 29, pp. 159–179, 2019.
  * [24] S. M. Judge, _Mobile Forensics: Analysis of the Messaging Application Signal_. University of Central Oklahoma, 2018.
  * [25] S. Krishnapriya, V. Priyanka, and S. S. Kumar, “Forensic extraction and analysis of signal application in android phones,” in _2021 International Conference on Forensics, Analytics, Big Data, Security (FABS)_ , vol. 1. IEEE, 2021, pp. 1–6.
  * [26] S. Wu, Y. Zhang, X. Wang, X. Xiong, and L. Du, “Forensic analysis of wechat on android smartphones,” _Digital investigation_ , vol. 21, pp. 3–10, 2017\.
  * [27] L. Zhang, F. Yu, and Q. Ji, “The forensic analysis of wechat message,” in _2016 Sixth International Conference on Instrumentation & Measurement, Computer, Communication and Control (IMCCC)_. IEEE, 2016, pp. 500–503.
  * [28] Q. Li, D. Du, Z. Yang, B. Liu, and J. Yan, “A flexibility method of qq forensic in android system,” in _2017 IEEE 2nd Advanced Information Technology, Electronic and Automation Control Conference (IAEAC)_. IEEE, 2017, pp. 855–861.
  * [29] Y. Gao and T. Cao, “Memory forensics for qq from a live system.” _J. Comput._ , vol. 5, no. 4, pp. 541–548, 2010.
  * [30] A. Iqbal, A. Marrington, and I. Baggili, “Forensic artifacts of the chaton instant messaging application,” in _2013 8th International Workshop on Systematic Approaches to Digital Forensics Engineering (SADFE)_. IEEE, 2013, pp. 1–6.
  * [31] O. S. Adebayo, S. A. Sulaimon, O. Osho, and S. M. Abdulhamid, “Forensic analysis of kik messenger on android devices.” Second International Engineering Conference, Federal University of …, 2017.
  * [32] K. M. Ovens and G. Morison, “Forensic analysis of kik messenger on ios devices,” _Digital Investigation_ , vol. 17, pp. 40–52, 2016.
  * [33] T. Mehrotra and B. Mehtre, “Forensic analysis of wickr application on android devices,” in _2013 IEEE International Conference on Computational Intelligence and Computing Research_. IEEE, 2013, pp. 1–6.
  * [34] G. Kim, S. Kim, M. Park, Y. Park, I. Lee, and J. Kim, “Forensic analysis of instant messaging apps: Decrypting wickr and private text messaging data,” _Forensic Science International: Digital Investigation_ , vol. 37, p. 301138, 2021.
  * [35] M. Azhar and T. E. A. Barton, “Forensic analysis of secure ephemeral messaging applications on android platforms,” in _International Conference on Global Security, Safety, and Sustainability_. Springer, 2017, pp. 27–41.
  * [36] F. Anwar, A. Fadlil, and I. Riadi, “Image quality analysis of png images on whatsapp messenger sending,” _Telematika_ , vol. 14, no. 1, pp. 1–12, 2021\.
  * [37] A. Marfianto and I. Riadi, “Whatsapp messenger forensic analysis based on android using text mining method,” _International Journal of Cyber Security and Digital Forensics (IJCSDF)_ , vol. 7, no. 3, pp. 319–327, 2018.


# Composition based crystal materials symmetry prediction using machine
learning with enhanced descriptors

Yuxin Li
School of Mechanical Engineering
Guizhou University
Guiyang China 550025
&Rongzhi Dong, Wenhui Yang
School of Mechanical Engineering
Guizhou University
Guiyang China 550025
&Jianjun Hu
Department of Computer Science and Engineering
University of South Carolina
Columbia, SC, 29201, USA
<EMAIL_ADDRESS>
Corresponding author: J.H. (webpage)

###### Abstract

Geometric information such as the space groups and crystal systems plays an
important role in the properties of crystal materials. Prediction of crystal
system and space group thus has wide applications in crystal material property
estimation and structure prediction. Previous works on experimental X-ray
diffraction (XRD) and density functional theory (DFT) based structure
determination methods achieved outstanding performance, but they are not
applicable for large-scale screening of materials compositions. There are also
machine learning models using Magpie descriptors for composition based
material space group determination, but their prediction accuracy only ranges
between 0.638 and 0.907 in different kinds of crystals. Herein, we report an
improved machine learning model for predicting the crystal system and space
group of materials using only the formula information. Benchmark study on a
dataset downloaded from Materials Project Database shows that our random
forest models based on our new descriptor set, achieve significant performance
improvements compared with previous work with accuracy scores ranging between
0.712 and 0.961 in terms of space group classification. Our model also shows
large performance improvement for crystal system prediction. Trained models
and source code are freely available at https://github.com/Yuxinya/SG_predict

_K_ eywords crystal system prediction $\cdot$ space group prediction $\cdot$
materials informatics $\cdot$ machine learning

##  1 Introduction

According to the degree of geometric form symmetry, the crystals can be
divided into different crystal systems and space groups[1, 2]. Determining the
symmetry information, particularly the space group, provides wide applications
for crystal material property prediction[3] and crystal structure
prediction[4, 5]. Recently, we proposed the method of knowledge-rich approach
for crystal structure prediction[6, 7, 8, 9], which is inspired by the recent
advances in protein structure prediction (PSP)[10, 11] which predicts protein
structures using the predicted distance matrix. In our approach, various
global optimization algorithms[6] such as genetic algorithm[7], neural
networks[8], differential evolution algorithm[9] have been used to reconstruct
the crystal structure atom coordinates. However, it is indispensable to
provide the space group information for a given materials composition before
predicting their structures.

Several machine learning models have been proposed to predict the space groups
of crystals. Several studies developed machine learning approaches for space
group classification by the X-ray diffraction (XRD)[12] data of materials.
Suzuki et al.[13] emphasised on demonstrating the potential of simple machine
learning techniques suitable for knowledge discovery and real-world
experiments. Their tree-ensemble-based machine learning model works with over
90% accuracy for crystal system classification based on powder X-ray
diffraction patterns. Park et al.[14] developed three convolutional neural
networks (CNNs) for the space group, extinction group and crystal system
classification of 150,000 powder XRD patterns, which returned test accuracies
of 81.14, 83.83 and 94.99$\%$ respectively. Vecsei et al.[15] studied the
problem of space group determination from powder X-ray diffraction patterns by
using fully connected neural networks and convolutional neural networks and
then tested those two models on the orher database. Oviedo et al.[16] proposed
a supervised machine learning framework for rapid crystal structure
identification of novel materials from thin-film XRD measurements. Chakraborty
et al.[17] performed augmentation of thin filmed X-ray diffraction patterns
and developed a high accuracy model for lattice classification from X-ray
diffraction. Zaloga et al.[18] identified crystal systems and symmetry space
groups by full-profile X-ray diffraction patterns using convolutional neural
networks and explored the factors that affect the classification performance.
Ziletti et al.[19] represented crystals by calculating a diffraction image,
then constructed a deep learning neural network model for classification which
achieves robust performance even in the presence of highly defective
structures.

Although the XRD based symmetry prediction algorithms can achieve good
performance in space group classification, they have several limitations. The
performance of XRD based methods is frequently influenced by low-quality X-ray
diffraction data[20]. Moreover, it is time-consuming to acquire and analyze
XRD data to recognize the crystal structure for each material[16]. There are
other machine learning models, different from XRD based methods, applied for
space groups and crystal systems classification. Liu et al.[21] trained a
convolutional neural network model by atomic pair distribution function(PDF)
for 45 most heavily represented space groups with an accuracy of 0.70.
Kaufmann et al.[22] used a machine learning–based approach and developed a
general methodology for rapid and autonomous identification of the crystal
symmetry from electron backscatter diffraction(EBSD) patterns. However, those
methods are inconvenient for predicting thousands of space groups since the
input pictures of materials should be provided. Theoretically, given the
chemical composition of a material, computational prediction of its crystal
structure is possible[23]. Several studies determine crystal structures by
combining global optimization with DFT calculations[23]. These methods have
demonstrated successes in a variety of cases. However, the DFT based methods
generally require thousands of CPU hours and can only be applied to predict
structures of relative small systems[19], which is not suitable for large-
scale material space group determination.

Recently, there emerged several crystal structure prediction methods that
start with a seed structure generated by the symmetry-restricted procedure [5,
8, 24]. In these algorithms, usually for a given composition, a space group is
specified to generate some random structures that satisfy the symmetry
constraints of the space group. There is thus a need to predict the space
group for a given composition. Several machine learning algorithms have been
proposed for material crystal systems and space groups prediction quickly
using composition information alone[25, 26]. In order to get the best
classification performance, Zhao et al.[25] uses two machine learning
algorithms, random forest and multiple layer perceptron neural network models,
combined with three kinds of descriptors, atom vector, one-hot encoding and
Magpie, for the crystal system and space group classification. For the
Material Project database they used, there are only 18 space groups selected,
each has more than 1000 samples for multi-class classification which achieve a
performance of 0.652 and 0.637 in terms of the F1-scores. Liang et al.[26]
proposed CryspNet for crystal lattice type, space group and lattice constants
prediction using deep neural networks. However, the classification accuracy
scores only range from 0.638 and 0.907 in the fourteen crystal categories they
divided.

In this work, we present an improved machine learning model for the crystal
system and space group prediction by inputting the formulas of the crystal
materials. Magpie descriptors are used as the basic descriptor set which is
combined with a new descriptor set that we proposed, to train our machine
learning models. To ensure that our classification algorithm can predict all
types of crystal materials, we trained and validated the models on all kinds
of entries from the Material Project Database of September 2020.

Compared with previous works based only on the Magpie descriptors, the
addition of the new descriptors enables our models to achieve significantly
improved performance on crystal system and space group prediction. For
example, the space group prediction of the cubic crystal system, which
consists of 18,325 materials, has an accuracy score of 0.961. Our models can
simultaneously make crystal system and space group predictions for a large
number of hypothetical materials, which can make contribution to our
knowledge-rich approach for crystal structure prediction. Our algorithm is
also useful for downstream tasks such as the exploration of the structure and
properties of new materials.

Our contributions can be summarized as follows:

  * •

We propose a new descriptor set for crystal system and space group prediction
of crystal materials which achieves significant performance improvements
compared to prior studies. We also identified what features contribute most in
our experiments for the classification performance

  * •

We remove the duplication caused by the isomer, and build ML models for multi-
class classification for space group and crystal system of crystal materials.

  * •

We build ML models for multi-label classification for space group prediction
and crystal system prediction.

  * •

We conduct extensive experiments with different machine learning algorithms.
Our experiments show that our algorithm based on random forest achieves high
performance in crystal system and space group prediction.

##  2 Materials and Methods

###  2.1 Datasets

Table 1:  lattice parameter relationships for materials of different lattice systems.  Crystal system | Edge lengths | Axial angles | Space groups | Amount
---|---|---|---|---
Cubic | $a=b=c$ | $\alpha=\beta=\gamma=90$ | 195-230 | 18324
Hexagonal | $a=b$ | $\alpha=\beta=90,\gamma=120$ | 168-194 | 9243
Trigonal | $a=b\neq c$ | $\alpha=\beta=90,\gamma=120$ | 143-167 | 11086
Tetragonal | $a=b\neq c$ | $\alpha=\beta=\gamma=90$ | 75-142 | 14654
Orthorhombic | $a\neq b\neq c$ | $\alpha=\beta=\gamma=90$ | 16-74 | 26800
Monoclinic | $a\neq c$ | $\alpha=\gamma=90,\beta\neq 90$ | 3-15 | 29872
Triclinic | all other cases | all other cases | 1-2 | 15297

There are 125,276 inorganic material items used in our experiment, which are
extracted from the Materials Project[27], an extensive material database which
includes the properties of all known inorganic materials. According to the
degree of geometric forms symmetry, those crystals are divided into seven
categories, namely cubic crystal system, hexagonal crystal system, tetragonal
crystal system, trigonal crystal system, orthorhombic crystal system,
monoclinic crystal system and triclinic crystal system. The amount of those
crystal systems and some details of geometric forms symmetry are shown in
Table 1. For those crystals, they are divided into 230 different combinations
of symmetrical elements. Each space group has a unique crystal system
corresponding to it.

(a) The relationship of molecular formula quantity and crystal system
quantity.

(b) The relationship of molecular formula quantity and space group quantity.

Figure 1: The statistics of the formulas which have isomers.

However, for a given crystal composition, there may be multiple structural
isomers which share the same chemical formula. There are a total of 10,412
unrepeated formulas that have isomers in the dataset we used. As is shown in
Figure1, we show several statistics of these 10,412 formulas. We can see that
there are some formulas that have only one crystal system and even one space
group even though they have isomers. This is because the isomers of above
formulas belong to the same crystal system or space group. In addition, there
are no more than two crystal systems for most materials formula isomers, but
there is a formula that has even 7 crystal systems. For material formulas with
isomers, more than half of the formulas have two space groups. In order to
make each formula match only one crystal system and space group, we calculate
the retain score by formula1, which is also used in cryspnet[26], and keep the
material structure with the highest score among the isomers while dropping out
other materials. Then, our machine learning model has a unique crystal system
or space group target for a formula to achieve multi-class classification.
Moreover, we also trained machine learning models for multi-label
classification [25] for the 10,412 crystals which have isomers.

$Score\left(f,s\right)=\frac{Abundance\left(f,s\right)}{E_{hull}\left(f,s\right)+\alpha}$
(1)

In this formula, $\alpha$ is tunable for balancing the formation energy term
and the abundance count from the Materials Project dataset, we set it as 0.1
in our work, $s$ and $f$ represent the space group and chemical formula. The
$Abundance(s,f)$ means the number of records which have the same formula and
space group. The $E_{hull}(s,f)$ is used to find the lowest formation energy
above the convex hull by the given composition and space group. For the
formulas that have isomers, we choose the material with the highest
$Score(s,f)$ for our benchmark experiments. Then, there are 102,528 materials
used in our experiment for multiclass classification.

###  2.2 Descriptors

Magpie descriptors[28] are a set of composition based materials attributes
that calculate the statistics of stoichiometric attributes, elemental
properties, electronic structure attributes and ionic compound attributes. It
has been widely used for building machine learning models for composition
based materials property predictions. In our work, the elemental property
statistics have been used as the baseline data set for space group and crystal
system prediction. There are 22 kinds of features in Magpie element property
statistics including Atomic Number, Mendeleev Number, Atomic Weight, Melting
Temperature, Periodic Table Row and Column, Covalent Radius,
Electronegativity, the number of Valence e in each Orbital(s, p. d, f, total),
the number of unilled er in each orbital (s, p. d, f, total), Ground State
Volume, Ground State Band Gap Energy, Ground State Magnetic Moment, and the
Space Group Number of elements. The main features of the Magpie feature set
are obtained by calculating the mean, average deviation, range, mode, minimum,
and maximum of above elemental properties (weighted by the fraction of each
element in the composition) to transform raw materials data into a form
compatible with machine learning. We also add some other Magpie properties,
which are used in cryspnet[26] for structure information prediction, include
Stoichiometry p-norm (p=0,2,3,5,7), Elemental Fraction, Fraction of Electrons
in each Orbital, Band Center, Ion Property (possible to form ionic compound,
ionic charge) to improve classification performance.

Table 2: Descriptors. Element Property statistics of Magpie | Additional Predictors of Magpie | Added Predictors in this work
---|---|---
Atomic Number | Stoichiometry p-norm (p=0,2,3,5.7) | Total Atom Number
Mendeleev Number | Elemental Fraction | Maximum Atom Number
Atomic Weight | Fraction of Electrons in each Orbital | Minimum Atom Number
Melting Temperature | Band Center | Average Atom Number
Periodic Table Row and Column |  | lon Property (possible to form
---
ionic compound, ionic charge)
Specific Value
Covalent Radius |  | Atom Number Variance
Electronegativity |  |
| The number of Valence e in
---
each Orbital(s, p. d, f, total)
|
| The number of unilled e in
---
each orbital (s, p. d, f, total)
|
Ground State Volume |  |
Ground State Band Gap Energy |  |
Ground State Magnetic Moment |  |
Space Group Number |  |

Additionally, we propose a set of new descriptors to improve crystal systems
and space groups prediction performance. Our descriptors are related to the
number of various atoms in a crystal structure. For a crystal, the numbers of
atoms is different for different elements within a unit cell and our new
descriptor is obtained by (for an element) calculating the maximum atom
number, the minimum atom number, the total atom number, the average atom
number, the specific value (ratio of pretty formula to full formula) and the
atom number variance of all the elements within a crystal structure. All the
features used in our experiments are shown in table2

###  2.3  Machine learning models: Random forest(RF), Extreme Gradient
Boosting(XGBoost) and Deep Neural Networks(DNN)

In this study, we use ensemble machine learning algorithms and neural networks
to find out the best model for crystal system and space group prediction.

####  2.3.1 Random Forest

Random forest (RF)[29], a popular ensemble machine learning algorithm, is one
of the major bagging machine learning models. Its driving principle in
classification is to build several estimators independently and each estimator
gets the probability of possible output labels. Then the average of the
predicted probabilities of all estimators are calculated as outputs. Labels
with the highest average probability are used as the output results. In our
random forest classification models, we choose ’entropy’ as our criterion. The
two important hyper-parameters, the number of trees and max features, are set
to be 100 and 80 respectively. The max depth is None. The min samples leaf and
min samples split are sat as 1 and 2. This algorithm was implemented by the
Scikit-Learn[30] library in Python 3.6

####  2.3.2 Extreme Gradient Boosting

XGBoost[31] is an excellent boosting algorithm which is widely used in machine
learning classification tasks. As an integrated algorithm, xgboost has
outstanding performance in classification with excellent generalization
performance. In addition, it is optimized by a series of methods, such as
parallel processing, supporting regularization and using second-order Taylor
expansion for loss function. Similar to traditional boosting algorithms which
are composed of several weak algorithms, when training the XGBoost model, each
weak algorithm tries to correct the error of the previous algorithms. In our
work, we chose the ’gbtree’ as our booster whose max depth was set as 6. As an
important parameter, the number of the trees is set as 180. The learning rate,
alpha, gamma, lambda are set as 0.3, 0, 0 ,1 respectively. For triclinic,
which has only two kinds of space groups, the objective (used to specify the
learning task and the corresponding learning objective) and evaluation metrics
are set as softmax and multiclass logloss. For the rest experiments of the
XGBoost model, we choose the logistic and logloss as the objective and
evaluation metrics. We use the XGBoost library in Python 3.6 to implement this
algorithm.

####  2.3.3 Deep Neural Networks

Figure 2: Architecture of the deep neural network.

The deep neural networks, which plays an important role in material
performance prediction and discovery of new materials, is also used in our
work. The structure of our NN model, shown in Figure2, is composed of 7 fully
connected layers, and the numbers of the hidden nodes are 230, 178, 280, 244,
181, 192 respectively. Relu[32] is used as the action function for those
layers. And after each layer except the last one, two strategies, Dropout and
BatchNorm, are used to prevent overfitting. The loss function is set as the
categorical cross entropy loss. The Adam optimizer is used in model training.
The learning rate, epochs and batch size are set as 0.001, 2000 and 255
respectively. In order to prevent overfitting and improve performance of the
NN, we use the early stopping strategy where the monitor is ’loss’ and
patience is 30. Our NN model is implemented on TensorFlow2.4.

###  2.4 Evaluation criteria

We use K-fold cross-validation in our work to assess the performance of
classification models of different algorithms. The process of K-fold cross-
validation strategy is that it randomly splits the initial sample set into K
sub-sample sets, taking one of them as the test set, the rest as the training
set. After the initial sample set splitted into K sub-sample sets, the cross-
validation is repeated K times, each sub-sample set is verified once, then the
results of K times are averaged to finally obtain a single performance
estimate. The 10-fold cross-validation method is chosen to evaluate the
classification performance of different algorithms in our work. Due to the
uneven distribution of samples, the following performance criteria in this
work are used, including accuracy, Matthews correlation coefficient(MCC),
weighted precision, weighted recall and weighted F1 score[33, 34].

##  3 Results and Discussion

###  3.1 Multi-class classification

####  3.1.1 Prediction performance of multi-class classification

Table 3: The classification performance of RF algorithm for all data removed the duplication caused by isomers. | Accuracy | MCC | Precision | Recall | F1 score
---|---|---|---|---|---
Crystal system prediction | 0.816$\pm$0.005 | 0.779$\pm$0.006 | 0.818$\pm$0.005 | 0.816$\pm$0.005 | 0.816$\pm$0.005
Space group prediction | 0.729$\pm$0.004 | 0.721$\pm$0.004 | 0.734$\pm$0.004 | 0.729$\pm$0.004 | 0.725$\pm$0.004
Figure 3: Confusion matrix of crystal system classification.

We trained the multi-class prediction models using the data sets after
removing the duplications caused by isomers. Then we use the 10-fold cross-
validation evaluation approach to assess our model performance. The crystal
system and space group classification performance for all data are shown in
Table 3. Our random forest model achieves an accuracy score of 0.816 and F1
score of 0.812 for crystal systems prediction, and its confusion matrix is
shown in Figure 3 which shows that the more regular of the crystal material
structures, the better classification result the model has in general. For
space groups prediction with around 230 categories, the accuracy and MCC
scores are 0.729 and 0,721 by just using the formula as the input to the
machine learning models.

Table 4: The performance of space group classification in different crystal systems. | data set size | Accuracy | MCC | Precision | Recall | F1 score
---|---|---|---|---|---|---
Cubic | 17367 | 0.961$\pm$0.006 | 0.945$\pm$0.008 | 0.960$\pm$0.005 | 0.961$\pm$0.006 | 0.959$\pm$0.006
Hexagonal | 8201 | 0.909$\pm$0.008 | 0.888$\pm$0.010 | 0.908$\pm$0.008 | 0.909$\pm$0.008 | 0.906$\pm$0.008
Trigonal | 9429 | 0.824$\pm$0.012 | 0.797$\pm$0.014 | 0.823$\pm$0.013 | 0.824$\pm$0.012 | 0.818$\pm$0.012
Tetragonal | 12675 | 0.849$\pm$0.013 | 0.832$\pm$0.015 | 0.846$\pm$0.013 | 0.849$\pm$0.013 | 0.840$\pm$0.014
Orthorhombic | 22392 | 0.755$\pm$0.005 | 0.729$\pm$0.006 | 0.759$\pm$0.005 | 0.755$\pm$0.005 | 0.746$\pm$0.006
Monoclinic | 23024 | 0.712$\pm$0.009 | 0.647$\pm$0.011 | 0.715$\pm$0.010 | 0.712$\pm$0.009 | 0.703$\pm$0.010
Triclinic | 9440 | 0.835$\pm$0.013 | 0.665$\pm$0.026 | 0.835$\pm$0.013 | 0.835$\pm$0.013 | 0.834$\pm$0.013
All | 102528 | 0.729$\pm$0.004 | 0.721$\pm$0.004 | 0.734$\pm$0.004 | 0.729$\pm$0.004 | 0.725$\pm$0.004

The results of space group classification in each crystal system are shown in
Table 4, the accuracy score of our RF model reaches 0.961 in space group
classification for cubic materials. Although the crystal structures in the
cubic crystal system have the highest symmetry compared to other crystal
systems, the Cubic crystal system has 36 space groups, from the space group of
195 to 230. It is exciting to achieve such high space group classification
performance with so many categories. However, the classification performances
in Orthorhombic and Monoclinic for space group prediction are much lower with
accuracy scores neither reaching 0.8. This is because these two crystal
systems have complex crystal structures, and the input of our machine learning
models is only the crystal formula. It is difficult to get the complex crystal
structures information merely by the formulas of materials, which can however
be ameliorated with large dataset. While the top-1 accuracy scores may not be
ideal, it is possible to use the top-k prediction results of our models in
downstream tasks to improve the hit rate. For other crystal systems, the
performances in space groups classification have scores above 0.80 in terms of
accuracy, MCC and F1 scores.

(a) feature importance of crystal system classification.

(b) feature importance of space group classification.

Figure 4: Feature importance of crystal system and space group classification.

In order to find out which features most affect the performance of
classification, we calculated the feature importance scores to sort the top 20
features which are shown in Figure 4. We find that the specific value, minimum
atom number, total atom number, average atom number, maximum atom number and
atom number variance have major impact, which explains why our model is better
than previous Magpie features based machine learning methods.

####  3.1.2 Performance comparison with previous works

(a) Performance comparison in terms of Accuracy

(b) Performance comparison in terms of F1 score

Figure 5:  Performance comparison with previous works for space group
prediction. All those performances are trained by the same dataset. Note that
the results in RF[25] and Cryspnet[26] are based on our implementations of the
corresponding algorithms described in the literature.

(a) RF model with all descriptors in Table2.

(b) RF model with only the new descriptor set we proposed.

(c) RF model in reference[25].

(d) NN of cryspnet[26].

Figure 6: Performance of space group prediction for monoclinic materials using
different algorithms. All those performances are trained by the same dataset.
Note that the results in (c) and (d) are based on our implementations of the
corresponding algorithms described in the literature.

For multi-class classification, we select all kinds of space groups available
in Materials Project in order that our classification algorithm can predict
all types of crystal materials. However, in ref.[25], there are only 18 space
groups selected for experiment, each having more than 1000 compositions for
space group prediction. In addition, we use Formula 1 to tackle the duplicate
formula caused by isomers, but in paper[25], there is no information about the
isomer processing. In Magpie descriptors based space group classification,
compared with Cryspnet[26], we use random forest algorithm which has better
performance than the neural network[25]. And most importantly, we propose a
new descriptor set, which greatly improves the performance of our model
compared to previous works on space group prediction for inorganic
materials[25, 26]. As is shown in Figure 5, we make comparisons with two
previous works for space group prediction. Our model is the random forest in
this work, which is trained using all the descriptors in Table2. RF[25] is the
random forest algorithm trained by Element Property statistic descriptors of
Magpie. Cryspnet[26] is the neural networks framework of cryspnet trained by
Element Property statistic descriptors and additional Predictors of Magpie.
The three models are trained using the same dataset of materials project
processed by Formula 1. As we can see that compared with the previous works of
cryspnet and ref.[25], the prediction performance in terms of accuracy scores
are improved by about 0.14 and 0.16 respectively in the six crystal systems
except the triclinic. For F1 scores, the improvements are similar. Moreover,
the higher the symmetry of the crystal structures, the better prediction
performance our RF can achieve for space group classification in the three
algorithms.

In particular, we checked the space group prediction performance in the
Monoclinic system which has 13 space groups and relatively complex crystal
structures to explore some classification details. From the confusion matrices
shown in Figure 6, it can be found that the performance has obvious
improvements in crystal systems classification by using the descriptor we
proposed. The improvement of the space group number of 3 and 10 are obvious.
To illustrate the validity of our descriptor, we make classification by the
combination of random forest algorithm and the new descriptors we proposed,
the results are shown in Figure 6(b). The model even has a better performance
than the two previous works. This shows that the descriptors about the number
of atoms in the lattice are effective for space group classification.

####  3.1.3 Performance comparison of different algorithms

Various machine learning algorithms have been used for structural information
prediction. Here, we evaluate the performance of three powerful machine
learning methods, the DNN, RF, and XGBoost, in space group and crystal system
classification. As is shown in Figure 7, the RF model performances in both
accuracy and F1 score are slightly better than those of XGboost. And in our
experiment, which uses the physical and chemical properties as descriptors,
the performance of the two kinds of ensemble tree algorithms is better than
that of NN. In this work, RF has demonstrated better performance than NN in
making crystal systems and space groups classification, which is consistent
with the results in previous study[25]. However, there are many works
indicating that NN has excellent performance compared with other machine
learning models in the field of material informatics[16, 13]. Therefore, for
different problems, choosing suitable machine learning algorithms is needed to
achieve the best results.

(a) Performance comparison in terms of Accuracy

(b) Performance comparison in terms of F1 score

Figure 7:  Performance comparison of different algorithms in our work for
space group prediction

###  3.2 Multi-label classification

Considering the fact that some formulas can correspond to multiple structures
of different crystal systems or space groups, the prediction problem of
crystal systems and space groups can also be mapped as a multi-label
classifiation problem[25]. Here we train machine learning models for multi-
label classification for the 10,412 crystals which have isomers. The crystal
system classification performance is shown in Tabel 5 which has the recall
score of 0.706, the precision score of 0.813, and the F1 score is 0.751. For
space group classification, the performance is inferior to the crystal system
prediction. The precision, recall and F1 scores are 0.764, 0.452 and 0.547
respectively. This is because the crystal structures has 230 space groups,
which are far more diverse than the seven types of crystal systems.

Table 5: The performance of multi-label classification. | Precision | Recall | F1 score
---|---|---|---
Crystal system prediction | 0.813$\pm$0.011 | 0.706$\pm$0.013 | 0.751$\pm$0.010
Space group prediction | 0.764$\pm$0.020 | 0.452$\pm$0.018 | 0.547$\pm$0.017

##  4 Conclusion

Computational prediction of space groups and crystal systems plays an
important role in analyzing crystal material structures and their physical and
chemical properties and is useful for crystal structure prediction. While
there are various methods to classify the space groups and crystal systems of
crystal materials in previous works, this study proposes an efficient and
easy-to-use method to achieve more accurate classification by introducing a
new set of materials composition based descriptors. When combined with the
Magpie descriptors, our algorithms’ classification performance have improved
significantly compard to previous algorithms that use only the magpie
descriptors. Our trained models and source code are freely available at
https://github.com/Yuxinya/SG_predict, which should be helpful for downstream
works such as material property exploration, material screening, and crystal
structure prediction.

##  5 Contributions

Conceptualization, J.H. and Y.L.; methodology, Y.L. and J.H.; software, Y.L.
and J.H.; validation, Y.L. and J.H.;investigation, Y.L.,J.H., R.D., and W.Y.;
resources, J.H.; writing–original draft preparation, J.H. and Y.L.;
writing–review and editing, J.H and Y.L.; visualization, Y.L., R.D;
supervision, J.H.; funding acquisition, J.H

##  6 Data Availability

The data that support the findings of this study are openly available in
Materials Project database[27] at http://www.materialsproject.org.

##  7 Acknowledgement

Research reported in this work was supported in part by NSF under grant and
1940099 and 1905775. The views, perspective,and content do not necessarily
represent the official views of the NSF

## References

  * [1] Peter Paufler and Stanislav K Filatov.  Es fedorov promoting the russian-german scientific interrelationship.  Minerals, 10(2):181, 2020.
  * [2] Yuxin Li, Wenhui Yang, Rongzhi Dong, and Jianjun Hu.  Mlatticeabc: Generic lattice constant prediction of crystal materials using machine learning.  ACS Omega, 0(0):null, 0.
  * [3] Tian Xie and Jeffrey C Grossman.  Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties.  Physical review letters, 120(14):145301, 2018.
  * [4] Hoi Chun Po, Ashvin Vishwanath, and Haruki Watanabe.  Symmetry-based indicators of band topology in the 230 space groups.  Nature communications, 8(1):1–9, 2017.
  * [5] Nobuya Sato, Tomoki Yamashita, Tamio Oguchi, Koji Hukushima, and Takashi Miyake.  Adjusting the descriptor for a crystal structure search using bayesian optimization.  Physical Review Materials, 4(3):033801, 2020.
  * [6] Jianjun Hu, Wenhui Yang, Rongzhi Dong, Yuxin Li, Xiang Li, and Shaobo Li.  Contact map based crystal structure prediction using global optimization.  arXiv preprint arXiv:2008.07016, 2020.
  * [7] Jianjun Hu, Wenhui Yang, and Edirisuriya M Dilanga Siriwardane.  Distance matrix-based crystal structure prediction using evolutionary algorithms.  The Journal of Physical Chemistry A, 2020.
  * [8] Jianjun Hu, Yong Zhao, Wenhui Yang, Yuqi Song, Edirisuriya Siriwardane, Yuxin Li, and Rongzhi Dong.  Alphacrystal: Contact map based crystal structure prediction using deep learning.  arXiv preprint arXiv:2102.01620, 2021.
  * [9] Wenhui Yang, Edirisuriya M. Dilanga Siriwardane, Rongzhi Dong, Yuxin Li, and Jianjun Hu.  Crystal structure prediction of materials with high symmetry using differential evolution, 2021.
  * [10] Wei Zheng, Yang Li, Chengxin Zhang, Robin Pearce, SM Mortuza, and Yang Zhang.  Deep-learning contact-map guided protein structure prediction in casp13.  Proteins: Structure, Function, and Bioinformatics, 87(12):1149–1164, 2019.
  * [11] Andrew W Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, Augustin Žídek, Alexander WR Nelson, Alex Bridgland, et al.  Improved protein structure prediction using potentials from deep learning.  Nature, 577(7792):706–710, 2020.
  * [12] Cameron F Holder and Raymond E Schaak.  Tutorial on powder x-ray diffraction for characterizing nanoscale materials, 2019.
  * [13] Yuta Suzuki, Hideitsu Hino, Takafumi Hawai, Kotaro Saito, Masato Kotsugi, and Kanta Ono.  Symmetry prediction and knowledge discovery from x-ray diffraction patterns using an interpretable machine learning approach.  Scientific reports, 10(1):1–11, 2020.
  * [14] Woon Bae Park, Jiyong Chung, Jaeyoung Jung, Keemin Sohn, Satendra Pal Singh, Myoungho Pyo, Namsoo Shin, and K-S Sohn.  Classification of crystal structure using a convolutional neural network.  IUCrJ, 4(4):486–494, 2017.
  * [15] Pascal Marc Vecsei, Kenny Choo, Johan Chang, and Titus Neupert.  Neural network based classification of crystal symmetries from x-ray diffraction patterns.  Physical Review B, 99(24):245120, 2019.
  * [16] Felipe Oviedo, Zekun Ren, Shijing Sun, Charles Settens, Zhe Liu, Noor Titan Putri Hartono, Savitha Ramasamy, Brian L DeCost, Siyu IP Tian, Giuseppe Romano, et al.  Fast and interpretable classification of small x-ray diffraction datasets using data augmentation and deep neural networks.  npj Computational Materials, 5(1):1–9, 2019.
  * [17] Abhik Chakraborty and Raksha Sharma.  See deeper: Identifying crystal structure from x-ray diffraction patterns.  In 2020 International Conference on Cyberworlds (CW), pages 49–54. IEEE, 2020.
  * [18] Alexander N Zaloga, Vladimir V Stanovov, Oksana E Bezrukova, Petr S Dubinin, and Igor S Yakimov.  Crystal symmetry classification from powder x-ray diffraction patterns using a convolutional neural network.  Materials Today Communications, 25:101662, 2020.
  * [19] Angelo Ziletti, Devinder Kumar, Matthias Scheffler, and Luca M Ghiringhelli.  Insightful classification of crystal structures using deep learning.  Nature communications, 9(1):1–10, 2018.
  * [20] Yanchao Wang, Jian Lv, Li Zhu, and Yanming Ma.  Crystal structure prediction via particle-swarm optimization.  Physical Review B, 82(9):094116, 2010.
  * [21] C-H Liu, Yunzhe Tao, Daniel Hsu, Qiang Du, and Simon JL Billinge.  Using a machine learning approach to determine the space group of a structure from the atomic pair distribution function.  Acta Crystallographica Section A: Foundations and Advances, 75(4):633–643, 2019.
  * [22] Kevin Kaufmann, Chaoyi Zhu, Alexander S Rosengarten, Daniel Maryanovsky, Tyler J Harrington, Eduardo Marin, and Kenneth S Vecchio.  Crystal symmetry determination in electron diffraction using machine learning.  Science, 367(6477):564–568, 2020.
  * [23] Artem R Oganov and Colin W Glass.  Crystal structure prediction using ab initio evolutionary techniques: Principles and applications.  The Journal of chemical physics, 124(24):244704, 2006.
  * [24] In-Ho Lee and KJ Chang.  Crystal structure prediction in a continuous representative space.  Computational Materials Science, 194:110436, 2021.
  * [25] Yong Zhao, Yuxin Cui, Zheng Xiong, Jing Jin, Zhonghao Liu, Rongzhi Dong, and Jianjun Hu.  Machine learning-based prediction of crystal systems and space groups from inorganic materials compositions.  ACS omega, 5(7):3596–3606, 2020.
  * [26] Haotong Liang, Valentin Stanev, A Gilad Kusne, and Ichiro Takeuchi.  Cryspnet: Crystal structure predictions via neural networks.  Physical Review Materials, 4(12):123802, 2020.
  * [27] Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, et al.  Commentary: The materials project: A materials genome approach to accelerating materials innovation.  APL materials, 1(1):011002, 2013.
  * [28] Logan Ward, Ankit Agrawal, Alok Choudhary, and Christopher Wolverton.  A general-purpose machine learning framework for predicting properties of inorganic materials.  npj Computational Materials, 2(1):1–7, 2016.
  * [29] Leo Breiman.  Random forests.  Machine learning, 45(1):5–32, 2001.
  * [30] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al.  Scikit-learn: Machine learning in python.  the Journal of machine Learning research, 12:2825–2830, 2011.
  * [31] Tianqi Chen and Carlos Guestrin.  Xgboost: A scalable tree boosting system.  In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785–794, 2016.
  * [32] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems, 25:1097–1105, 2012.
  * [33] Kevin P Murphy.  Machine learning: a probabilistic perspective.  MIT press, 2012.
  * [34] Qiuling Tao, Pengcheng Xu, Minjie Li, and Wencong Lu.  Machine learning for perovskite materials design and discovery.  npj Computational Materials, 7(1):1–18, 2021.