# Malicious and Low Credibility URLs on Twitter during COVID-19

Sameera Horawalavithana  CSE, University of South Florida<EMAIL_ADDRESS>,
Ravindu De Silva  SCoRe Lab<EMAIL_ADDRESS>,  Mohamed Nabeel  Qatar
Computing Research Institute<EMAIL_ADDRESS>,  Charitha Elvitigala  SCoRe
Lab<EMAIL_ADDRESS>,  Primal Wijesekara  University of California,
Berkeley<EMAIL_ADDRESS>and  Adriana Iamnitchi  CSE, University of South
Florida<EMAIL_ADDRESS>
††copyright: none

##  1\. Key Findings

This study provides an in-depth analysis of a Twitter dataset around
AstraZeneca COVID vaccine development released as a part of Grand
Challenge111https://www.insna.org/nasn-2021-grand-challenge, North American
Social Network Conference, 2021. In brief, we show:

  1. (1)

the presence of malicious and low credibility information sources shared on
Twitter messages in multiple languages.

  2. (2)

the malicious URLs, often in shortened forms, are increasingly hosted in
content delivery networks and shared cloud hosting infrastructures not only to
improve reach but also to avoid being detected and blocked.

  3. (3)

potential signs of coordination to promote both malicious and low credibility
URLs on Twitter. We use a null model and several statistical tests to identify
meaningful coordination behavior.

##  2\. Analytic Approach

Our analysis identifies the patterns related to URLs and web domains cited on
Twitter messages. We use multiple features related to URL mentions including
user sharing activities, credibility of the information source, maliciousness
of the web domain, the article content, etc. to guide our analysis.

To understand the patterns of sharing URL information in Twitter
conversations, we only consider the Twitter messages that contain at least a
URL, which consist of 25% from the total messages. As shown in Figure 1, they
are shared between August 17 and September 10, 2020. We filtered out URLs to
Twitter itself that typically refer to other tweets. In addition, the external
links (e.g., a tweet mentioning a YouTube video, or an external website
domain) mentioned in the messages are pre-processed as following. The
shortened URLs are expanded, and HTML parameters are removed from the URLs.
The YouTube URLs are resolved to the base URL if they include a parameter
referencing a specific time in the video. We represent the URL by the parent
domain when multiple child domains exist (e.g., fr.sputniknews.com,
arabic.sputniknews.com etc. are renamed as sputniknews.com). This pre-
processing code of resolving URLs is publicly available (PNNL, 2018). In this
analysis, we do not differentiate among different message types (e.g., tweets,
retweets, replies and quotes), but consider all messages as URL mentions.

For additional analysis, we scrape the web page content pointed to an URL
using a Python library, Newspaper. We managed to scrape 22,856 articles. The
tool failed to extract content from some web domains mainly due to inactive
web pages and regulations enforced by the web domain.

##  3\. Results

We first show the basic characteristics of the URLs shared on Twitter (Section
3.1). Second, we report the characteristics of low credibility URLs (Section
3.2) and malicious URLs (Section 3.3). Finally, we show coordination patterns
within the groups of users who attempt to promote both low credibility and
malicious content (Section 3.4).

###  3.1. Basic Characteristics

The main characteristics of the dataset are listed in Table 1. The resulting
dataset consists of 85,064 Twitter messages.

Figure 1 shows a spike in URL mentions on September 9, 2020. This is mainly
due to the reports of the halt in AstraZeneca vaccine development following a
suspected adverse reaction of a trial participant. There are 23,620 (28%)
messages citing distinct 8,566 URLs on this particular day. The most popular
URLs point mainstream news articles, while the article published in
statnews.com received the highest number (1,158) of shares
222https://www.statnews.com/2020/09/08/astrazeneca-covid-19-vaccine-study-put-
on-hold-due-to-suspected-adverse-reaction-in-participant-in-the-u-k/.

According to the Langid (Lui and Baldwin, 2012), the majority of the articles
are in English (6,453) and Spanish (5,197) languages. Other articles are
written in Turkish (1,839), French (1,539), Portuguese (1,277), Russian (784),
Italian (772), Greek (708), Japanese (702), Croatian (447), etc. There are 274
and 165 YouTube videos shared on Twitter have the title written in English and
Spanish languages, respectively.

Table 1. Basic characteristics of Twitter URL Dataset. # Distinct URLs | 26,422
---|---
# Distinct Domains | 6,990
# Users | 40,287
# Messages | 85,064
Figure 1. Number of Twitter URL mentions over time. There is a known gap in
the data collection between August 27-31

###  3.2. Low Credibility Information Sources

In this section, we report how Twitter users react to low credibility
information sources. We group the web domains according to the classification
made by Media Bias/ Fact Check (MBFC) (mbf, 2020). We also cross-check these
domains with the list of web sites that publish False COVID-19 information as
identified by NewsGuard (new, 2021). We identify 6,860 (26%) URLs from 17 low
credibility information sources that are shared on 26,220 (31%) messages (as
shown in Table 2). sputniknews.com is the most popular web domain by the
number of mentions (22,251), the number of users (4,358) and the number of
URLs (6,638). This can be expected mainly due to a network of sputniknews.com
media outlets that publish articles in many languages. For example, 1,377 and
1,080 sputniknews.com articles are published in Turkish and French. However,
sputniknews.com URLs are not the most popular to be shared in the immediate
hours after their first appearance (as shown in Figure 2b). For example, the
median number of sputniknews.com URL mentions is 2 in an hour. sputniknews.com
is frequently described as a Russian propaganda outlet that spreads master
narratives in the Russia’s disinformation campaign (mbf, 2020).

We noticed that rt.com acquires its many mentions from very few URLs: 119 URLs
are cited in 3,225 messages during our observation period (Figure 2a). rt.com
URLs have the highest number of mentions in the first hour after publication
(as shown in Figure 2b) relative to the rest of popular web sites.

We also note that Twitter messages share the same article heading in the
messages when they cite the same URL. These users might promote certain topics
through massive repetition of messages via injecting URLs. For example, an
article published in zerohedge.com was in the Top-10 most popular URLs on the
day when AstraZeneza vaccine development halted
333https://www.zerohedge.com/markets/ft-confirms-astrazeneca-covid-19-vaccine-
caused-serious-spinal-issues-test-patient. However, this article tried to
build an alternative frame highlighting a statement by the US House Speaker
Nancy Pelosi about the issue instead of reporting the details of the main
event. This technique can spoof the Twitter trending algorithm to broadcast
their message towards a wider audience and to establish their preferred
narrative via artificial amplification.

Table 2. Twitter sharing characteristics for low credibility URLs as
identified by MBFC and NewsGuard (NG). We use the ✓ mark to reflect whether
the domain is being listed as a low credibility information source by the
respective fact checking organization.

Domain | MBFC | NG | # Mentions | # Users | # URLs
---|---|---|---|---|---
sputniknews.com | ✓ | ✓ | 22,251 | 4,358 | 6,638
rt.com | ✓ | ✓ | 3,225 | 1,922 | 119
zenith.news | ✓ | - | 117 | 6 | 62
zerohedge.com | ✓ | ✓ | 278 | 249 | 15
swarajyamag.com | ✓ | - | 228 | 105 | 5
oann.com | ✓ | ✓ | 50 | 48 | 4
childrenshealthdefense.org | ✓ | ✓ | 3 | 3 | 3
globalresearch.ca | ✓ | ✓ | 16 | 7 | 3
torontotoday.net | ✓ | - | 9 | 5 | 2
gnews.org | ✓ | ✓ | 4 | 4 | 2
truepundit.com | ✓ | ✓ | 29 | 29 | 1
needtoknow.news | ✓ | - | 5 | 5 | 1
oye.news | ✓ | ✓ | 1 | 1 | 1
gellerreport.com | ✓ | ✓ | 1 | 1 | 1
barenakedislam.com | ✓ | - | 1 | 1 | 1
wakingtimes.com | ✓ | ✓ | 1 | 1 | 1
thegatewaypundit.com | ✓ | ✓ | 1 | 1 | 1

|
---|---

(a) Top-10 shared domains

(b) Twitter lifespan of URLs by domain

Figure 2. Twitter sharing characteristics of most popular domains. Figure a)
shows the Top-10 domains by the number of distinct URLs. The size of the
markers in this plot are proportional to the number of URLs associated with
the domain. Figure b) shows the number of URL mentions posted at each hour
after the first appearance of URL in the respective domain. We count the
number of mentions for each URL in the respective hour, and calculate the
median number of mentions for the domain of the URL.

We also discovered URLs from two Russian news media domains, sputnik-
abkhazia.ru and moskva-tyt.ru that are shared by very few users. For example,
sputnik-abkhazia.ru is the fourth most popular web domain by the number of
URLs cited on Twitter messages. These URLs are shared solely by the Twitter
account controlled by the news media site. However, they gain limited
engagement with other users.

Table 3. The most popular domains hosting articles written in multiple
languages. sputniknews.com and youtube.com are among the most popular domains.

English
---
sputniknews.com
youtube.com
reuters.com
newsfilter.io
nytimes.com

Spanish
---
sputniknews.com
youtube.com
rt.com
reuters.com
wp.me

Turkish
---
sputniknews.com
youtube.com
ntv.com.tr
is.gd
sozcu.com.tr

French
---
sputniknews.com
cvitrolles.wordpress.com
youtube.com
francetvinfo.fr
lalibre.be

Portugese
---
sputniknews.com
brasil247.com
evsarteblog.wordpress.com
youtube.com
tvi24.iol.pt

Another objective is to compare and contrast the linguistic patterns used in
the misleading and credible news articles. We used the articles hosted at two
most popular domains sputniknews.com and reuters.com for this analysis. While
we know sputniknews.com publishes misleading, questionable content (mbf, 2020;
new, 2021), reuters.com is a credible mainstream news media outlet. To analyze
the linguistic patterns of the words used in the articles, we use LIWC to
count the words associated with different psychological categories. LIWC
(Tausczik and Pennebaker, 2010) is a linguistic dictionary that contains 4,500
words categorized to analyze psycholinguistic patterns in text. We use six
categories in the LIWC dictionary: Social processes, Affective processes,
Cognitive processes, Perceptual processes, Biological processes and
Relativity. Figure 3 shows the fraction of words used in the articles for each
LIWC category.

It is interesting to note the lack of words used in sputniknews.com articles
to cover the main discussion topics related to the biological processes
compared to what reported in reuters.com (Figure 3). We noticed
sputniknews.com articles used more words that are associated with social
processes (e.g., friends, family), affective processes (e.g., anxiety,
sadness), and cognitive processes (e.g., discrepancy, tentative) than what are
used in the articles published by reuters.com. This suggests that the
sputniknews.com articles used a different writing style in their articles.

Figure 3. Lexical patterns used in the articles hosted at sputniknews.com and
reuters.com

###  3.3. Malicious URLs

We use VirusTotal (VT) (VirusTotal, Subsidiary of Google, [n.d.]) to extract
the maliciousness of URLs. VT provides the state-of-the-art aggregated
intelligence for domains and URLs, and relies on more than 70 third-party
updated antivirus (AV) engines. For all distinct URLs in our collection, we
extract VT scan reports via querying the publicly available API. Each VT scan
report contains of the verdict from every AV engine, information related to
the URL such as first and last seen dates of the URL in the VT system, hosting
IP address, final redirected URL (if applicable), content length, etc. Each AV
engine in a VT report detects if the URL is malicious or not. We use the
number of engines that detect a URL as malicious as an indication of the
maliciousness of the URL.

In this study, we label a URL as malicious if at least one AV engine, that is
#VT $\geq$ 1, detects it as malicious. We identify 441 malicious URLs from the
collected VT reports. We observe that 25.75% of the malicious URLs utilize URL
shortening services with top 4 services being bit.ly, tinyurl.com, ow.ly and
goo.su whereas as only 0.97% of benign URLs utilize such services. This
observation is consistent with the trend that malicious actors increasingly
utilize shortening services to camouflage malicious URLs presenting non-
suspecting URLs to users (Group, 2020). We find that 30.80% of the domains
related to malicious URLs are ranked below 100K by Alexa (ale, 2019) (The
lower the rank value, the higher the popularity). This indicates the
concerning fact that malicious actors are able to reach a large user base
reaping a high return on investment for their attacks.

We further analyze the malicious URLs to identify related malicious URLs. To
this end, based on the lexical features in the literature and the hosting
features mentioned in Table 4, we cluster the malicious URLs using PCA/OPTICS
algorithm (Pedregosa et al., 2011). While lexical features identify
characteristics related to URLs themselves, hosting features, extracted from
Farsight Passive DNS (PDNS) data (Farsight Security, Inc., [n.d.]), capture
the characteristics of underlying hosting infrastructure. As shown in Figure
4, these features collectively identify 10 distinct malicious URL clusters. We
manually verified the accuracy of the top 2 clusters by checking the web page
content, registration information and domain certificate information. Our
observations suggest that attackers launch multiple attacks at the same time.
We also analyze the clusters based on the maliciousness of URLs. The
maliciousness of a URL can loosely be measured by #VT, the number of VT
positives. An interesting observation is that URLs belonging to different
maliciousness levels share similar lexical and hosting features.

We further analyze these malicious URLs in terms of where they are hosted. To
our surprise, we find that 80.04% of these malicious URLs are hosted in CDNs
such as Cloudflare and Akamai. While CDNs provide fast delivery of content
across the globe through their distributed computing infrastructure, we
believe a key reason why malicious actors utilize such services is to improve
attack agility and stay below the radar of malicious domain detection
mechanisms in place. This observation is further reinforced with the increased
utilization of public cloud computing infrastructure (33.5% of all malicious
URLs) sharing hosting IPs with tens of thousands of unrelated domains, which
are mostly benign. Such shared IPs are usually not blocked in practice due to
the collateral damage.

Table 4. Details of the hosting features for malicious URL clustering. Feature | Description
---|---
VT_Dur | URL duration in VT
PDNS_Dur | Domain duration in PDNS (Bilge et al., 2014)
#IPs | Number of hosting IPs during the study period
#Queries | Number of times the domain is accessed
#NSes | Number of Name servers
Is_NS | NS domain matches the domain name?
#SOAs | Number of administrative domains
Is_SOA | Admin domain matches the domain name?
#Domains | Number of domains hosted on the IP
#Queries_IP | Number of times the IP is accessed
ASN | Autonomous System Number
Org | Organization owning the ASN
Figure 4. Malicious URL clusters based on the lexical and hosting features.
Each point is a URL, and it is colored according to the cluster it belongs.

###  3.4. User Co-sharing Practices

As reported previously (Starbird et al., 2019; Pacheco et al., 2020), there
are signs of coordination to spread the disinformation content. In this work,
one of our assumptions is that the coordination is based on the content being
shared (i.e., URLs mentioned in the tweets). One of our objectives is to
measure the extent of coordination in URL sharing activities. We construct two
bipartite networks to quantify the amount of such coordination effort. The
first network connects an author to a low credibility URL mentioned in a
tweet, and the second network connects an author to a malicious URL mentioned
in a tweet. These networks would capture any suspicious behavior of promoting
a particular URL.

User-(low credibility) URLs Bipartite Network: We identify 6,860 low
credibility information URLs that are shared by 6,600 users in 26,220 Twitter
messages. The five (0.08%) most active users cite low credibility URLs in
6,404 (24%) messages. Their most preferred information source is
tr.sputniknews.com which is the Turkish outlet of the sputnik media network.

User-(malicious) URLs Bipartite Network: We identify 441 malicious URLs that
are shared by 357 users in 571 Twitter messages. While a user might share a
malicious URL unwittingly, it is suspicious to note users who share multiple
malicious URLs. Specifically, we identify 51 users who share more than 1
malicious URL, and 21 users who share more than 2 malicious URLs.

|
---|---

(a) Coordination to promote low credibility URLs (KS=0.6,p-value=0.001)

(b) Coordination to promote malicious URLs (KS=0.27,p-value=4.45$e^{-12}$)

Figure 5. Bipartite clustering coefficient of a) the user-(low credibility)
URL network, b) the user-(malicious) URL network. We also compare the network
clustering values with a random bipartite network using the Newman’s
configuration model (Newman, 2003). The deviation of the clustering values
from the random bipartite network shows potential coordination effort to
promote these URLs on Twitter.

Figures 5a and 5b show the distributions of bipartite clustering coefficients
(Latapy et al., 2008) for URLs in the two bipartite networks respectively.
Clustering coefficient values are higher for URLs when they are shared by a
group of users who engage with other URLs together. We compare the similar
clustering values which are calculated from the identical random bipartite
networks. We construct two random bipartite networks using the Newman’s
configuration model (Newman, 2003) for the comparison. Given the original
user-URL bipartite network, we match the two degree sequences in the users and
URLs in the random bipartite network. We noticed a significant deviation of
clustering coefficient values for URLs in both bipartite networks compared to
URLs in the random bipartite networks (as shown in Figures 5a and 5b). For
example, there are 4,320 (63%) low credibility URLs with a clustering
coefficient value greater than 0.8 than those number of URLs (217) in the
random bipartite network. In the malicious URL network, there are 121 URLs
with a clustering coefficient value greater than 0.8 that are promoted by the
same set of users (the expected number of URLs is 36 in the random network).

To confirm our observation, we also perform the Kolmogorov-Smirnov (KS) test
between the clustering coefficient values from the original and random
bipartite networks. KS-statistic values (statistically significant) are 0.6
and 0.27 for the low credibility URL network and malicious URL network,
respectively. This suggests the sustained effort of users to amplify both
misleading and malicious content. There may be different types of users (e.g.,
bots, cyborgs, paid activists) who amplify these URLs. That would remain as a
future work to identify these types of users.

##  4\. Conclusions

In times of crisis, whether political or health-related, online disinformation
is amplified by social media promotion of alternative media outlets (Starbird
et al., 2018). This study adds to the growing body of work (Ferrara et al.,
2020) that analyzes the misinformation activity during the COVID-19 crisis by
studying the sharing of URLs on Twitter between August 17 and September 10,
2020. Our contributions complement previous observations (Singh et al., 2020;
Group, 2020) in multiple ways. We discover a strong presence of malicious and
low credibility information sources shared on Twitter messages in multiple
languages. Not only that URLs from low credibility sources, as classified by
independent bodies such as NewsGuard and Media Bias/Fact Check, were present,
but many were shown to point to pages with malicious code. A significant
portion of these URLs (25%) were in shortened form (compared to under 1% of
the non-malicious URLs) and hosted on well-established, reputable content
delivery networks (such as Cloudflare and Akamai) in an attempt, we believe,
to avoid detection.

We also discovered potential signs of coordination to promote malicious and
low credibility URLs on Twitter. Specifically, we discovered unusual
clustering of user activity related to the sharing of such URLs. We use a null
model and several statistical tests to compare expected behavior with what we
suspect to be coordinated behavior as seen in this dataset.

This study is another datapoint in the continuous effort (Horawalavithana et
al., 2020) to unmask malicious strategies that exploit Twitter and other
social media platforms to promote shady objectives in times of crisis.

## References

  * (1)
  * ale (2019) 2019\.  Alexa Top 1M Sites.  https://www.alexa.com/topsites.  Accessed December 2019.
  * mbf (2020) 2020\.  Media Bias/Fact Check - Search and Learn the Bias of News Media.  https://mediabiasfactcheck.com/
  * new (2021) 2021\.  Coronavirus Misinformation Tracking Center.  https://www.newsguardtech.com/coronavirus-misinformation-tracking-center/
  * Bilge et al. (2014) L. Bilge, S. Sen, D. Balzarotti, E. Kirda, and C. Kruegel. 2014.  Exposure: A Passive DNS Analysis Service to Detect and Report Malicious Domains.  _ACM TISS_ 16, 4 (apr 2014), 14:1–14:28.
  * Farsight Security, Inc. ([n.d.]) Farsight Security, Inc. [n.d.].  DNS Database.  https://www.dnsdb.info/.  Accessed: 10-01-2021.
  * Ferrara et al. (2020) Emilio Ferrara, Stefano Cresci, and Luca Luceri. 2020\.  Misinformation, manipulation, and abuse on social media in the era of COVID-19.  _Journal of Computational Social Science_ 3, 2 (2020), 271–277.
  * Group (2020) Disinformation Research Group. 2020\.  Spanish-language vaccine news stories hosting malware disseminated via URL shorteners.  https://fas.org/disinfoblog/spanish-language-vaccine-news-stories-hosting-malware-disseminated-via-url-shorteners/
  * Horawalavithana et al. (2020) Sameera Horawalavithana, Kin Wai Ng, and Adriana Iamnitchi. 2020\.  Twitter Is the Megaphone of Cross-platform Messaging on the White Helmets. In _International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation_. Springer, 235–244.
  * Latapy et al. (2008) Matthieu Latapy, Clémence Magnien, and Nathalie Del Vecchio. 2008.  Basic notions for the analysis of large two-mode networks.  _Social networks_ 30, 1 (2008), 31–48.
  * Lui and Baldwin (2012) Marco Lui and Timothy Baldwin. 2012.  langid. py: An off-the-shelf language identification tool. In _Proceedings of the ACL 2012 system demonstrations_. 25–30.
  * Newman (2003) Mark EJ Newman. 2003\.  The structure and function of complex networks.  _SIAM review_ 45, 2 (2003), 167–256.
  * Pacheco et al. (2020) Diogo Pacheco, Pik-Mai Hui, Christopher Torres-Lugo, Bao Tran Truong, Alessandro Flammini, and Filippo Menczer. 2020.  Uncovering Coordinated Networks on Social Media.  _arXiv preprint arXiv:2001.05658_ (2020).
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine Learning in Python.  _Journal of Machine Learning Research_ 12 (2011), 2825–2830.
  * PNNL (2018) Pacific Northwest National Laboratory PNNL. 2018.  Socialsim.  https://github.com/pnnl/socialsim.
  * Singh et al. (2020) Lisa Singh, Leticia Bode, Ceren Budak, Kornraphop Kawintiranon, Colton Padden, and Emily Vraga. 2020\.  Understanding high-and low-quality URL Sharing on COVID-19 Twitter streams.  _Journal of Computational Social Science_ 3, 2 (2020), 343–366.
  * Starbird et al. (2019) Kate Starbird, Ahmer Arif, and Tom Wilson. 2019.  Disinformation As Collaborative Work: Surfacing the Participatory Nature of Strategic Information Operations.  _Proc. ACM Hum.-Comput. Interact._ 3, CSCW, Article 127 (Nov. 2019), 26 pages.  https://doi.org/10.1145/3359229
  * Starbird et al. (2018) Kate Starbird, Ahmer Arif, Tom Wilson, Katherine Van Koevering, Katya Yefimova, and Daniel Scarnecchia. 2018.  Ecosystem or echo-system? Exploring content sharing across alternative media domains. In _AAAI ICWSM_.
  * Tausczik and Pennebaker (2010) Yla R Tausczik and James W Pennebaker. 2010.  The psychological meaning of words: LIWC and computerized text analysis methods.  _Journal of language and social psychology_ 29, 1 (2010), 24–54.
  * VirusTotal, Subsidiary of Google ([n.d.]) VirusTotal, Subsidiary of Google. [n.d.].  VirusTotal – Free Online Virus, Malware and URL Scanner.  https://www.virustotal.com/.  Accessed: 14-01-2021.


# Coarse-to-Fine Cascaded Networks with Smooth Predicting for
Video Facial Expression Recognition

Fanglei Xue1,2, Zichang Tan3,4, Yu Zhu3,4, Zhongsong Ma1,2, Guodong Guo3,4
1University of Chinese Academy of Sciences, Beijing, China
2Key Laboratory of Space Utilization, Technology and Engineering Center for
Space Utilization,
Chinese Academy of Sciences, Beijing, China
3Institute of Deep Learning, Baidu Research, Beijing, China
4National Engineering Laboratory for Deep Learning Technology and Application,
Beijing, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>{tanzichang, zhuyu05,
<EMAIL_ADDRESS>This work was done when Fanglei Xue was an intern at
Institute of Deep Learning, Baidu Research.Corresponding author

###### Abstract

Facial expression recognition plays an important role in human-computer
interaction. In this paper, we propose the Coarse-to-Fine Cascaded network
with Smooth Predicting (CFC-SP) to improve the performance of facial
expression recognition. CFC-SP contains two core components, namely Coarse-to-
Fine Cascaded networks (CFC) and Smooth Predicting (SP). For CFC, it first
groups several similar emotions to form a rough category, and then employs a
network to conduct a coarse but accurate classification. Later, an additional
network for these grouped emotions is further used to obtain fine-grained
predictions. For SP, it improves the recognition capability of the model by
capturing both universal and unique expression features. To be specific, the
universal features denote the general characteristic of facial emotions within
a period and the unique features denote the specific characteristic at this
moment. Experiments on Aff-Wild2 show the effectiveness of the proposed CFSP.
We achieved 3rd place in the Expression Classification Challenge of the 3rd
Competition on Affective Behavior Analysis in-the-wild. The code will be
released at https://github.com/BR-IDL/PaddleViT.

##  1 Introduction

In the past years, facial expression recognition has attracted increasing
attention in the field of computer vision and psychology science [36, 35, 40,
12, 25, 6, 38, 28]. Facial expression can be regarded as the external signals
to reflect people’s psychological activities and intentions. Therefore, facial
expression recognition is particularly important in human-computer
interaction. Although this topic has been studied for many years, it is still
difficult to achieve accurate emotional analysis nowadays.

In order to promote the development of facial expression recognition, Kollias
et al. [37, 15, 19, 18, 14, 21, 17, 20, 16] organize the competition on
Affective Behavior Analysis in-the-wild (ABAW). The competition has been
successfully held two times, in conjunction with IEEE FG 2020 and ICCV 2021.
The third competition (ABAW3) will be held in conjunction with CVPR 2022.
Those competitions provide several large-scale in-the-wild benchmarks for
affective behavior analysis, e.g., Aff-Wild2 [19]. Aff-Wild2 contains 558
videos with around 2.8 million frames with rich annotations. Besides the
common annotations of facial expressions and Action Units (AUs), it also
provides the annotations of valence and arousal to study continuous emotions.

Figure 1: The architecture of our model. Given the input frames $I_{i-2}$, …
$I_{i}$, …, $I_{i+2}$, we first extract the features of these frames by the
Coarse Net and get the coarse results. If the predicted coarse result is
Negative, the Negative Net is utilized to predict the negative classes. The
smooth predicting (SP) module is designed to take multiple frame features
(denote as $f_{i}$) as input. And every feature is calculated by adding
themselves to the mean values of their adjacent features. After this, the
predicted result will be more smooth and stable.

The goal of our work is to study facial expression recognition based on Aff-
Wild2. One main challenge in facial expression recognition is the problem of
label ambiguity, which may be caused by two aspects: one is the ambiguity of
the expression itself, where some expressions are similar and difficult to
distinguish. The other is the labeling ambiguity caused by different people
having different understandings of different facial expressions, resulting in
inconsistent labeling. For example, ’Sadness’ is very similar to ’Disgust’,
and it is hard to distinguish them. To address this problem, we propose the
coarse-to-fine cascaded networks to obtain more reliable predictions in a
coarse-to-fine manner. Specifically, we first group several similar emotions
as a coarse category (including ’Anger’, ’Disgust’, ’Fear’ and ’Sadness’), and
this coarse category will be combined with other emotions to train a coarse
model at the first stage. In the second stage, an additional network is
further employed to distinguish those grouped emotions and then obtain fine
predictions. Moreover, inspired by the previous work [32], the feature
embedding of facial expression can be divided into two parts, namely universal
and unique parts. The universal part describes the general characteristic of
facial emotions, and the unique part describes the specific features of the
expressive component. The unique part can be denoted by the features of each
specific image. For the universal part, it is captured by using a proposed
Smooth Predicting (SP), which collects the features of nearby iterations and
outputs the averaging features. The final feature used for prediction is the
sum of the universal and unique parts. Furthermore, we train multiple models
based on various architectures (e.g., ResNet [10], Swin-Transformer [27]) and
improve the performance by using an ensemble of them.

The main contributions of our work can be summarized as follows:

  * •

We propose the Coarse-to-Fine Cascaded networks (CFC) to address the label
ambiguity problem in facial expression recognition.

  * •

We propose a Smooth Predicting (SP) to capture both universal and unique
features, which can effectively improve the performance of facial expression
recognition.

##  2 Related Works

In this section, we briefly review the related works in facial expression
recognition and affective behavior analysis.

Facial Expression Recognition. Facial expression recognition [36, 35, 40, 12,
25, 6, 38, 28], which is a classic research topic in computer vision and
psychology science, has been studied for many years. Most of the early methods
were based on hand-crafted features [7, 43, 42]. Recently, deep learning has
greatly advanced the development of facial expression recognition [36, 35, 40,
12, 6, 38, 31]. For example, Weng et al. [35] propose a multi-branch network
to capture both global and local features for facial expression recognition.
Xue et al. [36] propose a hybrid architecture with combined CNN and
transformer, which achieves state-of-the-art performance on multiple
benchmarks. Some researchers [34, 26, 24] propose to extract discriminative
features by attention mechanism, which is proven to be robust for occlusions.

Affective Behavior Analysis. This subsection mainly reviews the related works
in previous competitions of ABAW. Kuhnke et al. [22] propose a two-stream
network to utilize both vision and audio information for affective behavior
analysis. Considering the problem of incomplete labels, Deng et al. [1]
propose to learn from unlabeled data by a teacher-student scheme. Moreover,
Zhang et al. [40] a Prior Aided Streaming Network (PASN) for multi-task
expression recognition, including Categorical Emotions (CE), Action Units
(AU), and Valence Arousal (VA). In PASN, three tasks are placed in a serial
manner (AU$\rightarrow$CE$\rightarrow$VA), where the basic up-streaming task
would aid the recognition of down-streaming tasks. Jin et al. [12] propose a
multi-modal multi-task method for facial action unit and expression
Recognition.

Figure 2: Sketch of different backbone architectures. They have various
feature extract methods (convolution or self-attention), various lateral
connections (with or without skip connections), and various resolution streams
(gradually decrease or keep the same). We believe different architecture could
learn different features. Thus, we ensemble these four backbones in the coarse
net to generate robust prediction results and prevent over-fitting.

##  3 Method

###  3.1 Overview

The overall architecture of our proposed method is illustrated in Fig. 1. It
mainly consists of two parts: the Coarse-to-Fine Cascaded networks (CFC) on
the left and the Smooth Predicting (SP) module on the right. In the following,
we will introduce these two modules in detail.

###  3.2 Coarse-to-Fine Cascaded Networks

Facial expressions are generated by muscle movement in faces. Prior studies
[4] split expressions into seven basic categories to represent the common
emotions shared by all human beings. The EXPR task in ABAW3 aims to categorize
facial images into these seven expressions plus an ”other” category. But these
seven categories are not equally distributed, some expressions are very
similar to others, especially negative expressions. Specifically, four
negative expressions (anger, disgust, fear, and sadness) have very similar
muscle movements in the same facial areas [5]. Compared with other
expressions, like happiness and surprise, these expressions are more difficult
to predict correctly. On the other hand, most in-the-wild facial expression
datasets are collected from the Internet, where most people share their happy
times. Negative emotions are hard to obtain in real scenes [23], making
existing FER datasets imbalance. To address these two issues, we adopted the
coarse-fine strategy in [23].

To be specific, we utilize two stages to predict coarse and negative labels,
separately. Following [23], we combine four negative expressions (anger,
disgust, fear, and sadness) to one category (negative) in the first stage,
resulting in five classes in the coarse stage. If the coarse net predicts the
facial image into the negative class, the image will be fed into the negative
net to predict the negative expression. Otherwise, the coarse label will be
returned directly. In this manner, the low-frequency negative expression
samples are combined in the negative, making the coarse dataset more balance.
And the negative net only aims to distinguish negative expressions, which
share a similar difficulty.

In order to further enhance the robustness of our framework, we adopt the
model ensemble strategy to our coarse net. Specifically, IResNet-152 [2],
Swin-S [27], RepVGG [3] and HRNet [33] are adopted as backbone networks. As
illustrated in Fig. 3, there are significant differences among these
architectures. IResNet (a), Swin Transformer (b), and RepVGG (c) utilize
pooling or path merging layers to decrease the spatial resolution of feature
maps to reduce the computation cost and enlarge the receptive field, while
HRNet (d) remains the high-resolution representations and exchange features
across resolutions to obtain rich semantic features. (b) utilizes the self-
attention mechanism with shifted windows to explore relationships with
different areas while the other three models adopt the traditional convolution
method. (a) utilizes residual connection within stages while (b) and (c) do
not (c) has skip-connection in the training stage. (d) adopted more
connections among different resolutions to extract rich semantic features.
These different architectural designs could help different models learn
different features, and prevent the whole framework overfit to some noisy
features.

Figure 3: Illustration of our ensemble process. Feature logits are extracted
from each backbone network separately and produce the per-class confidential
scores by the softmax function. The scores are then summed up and generate the
final result with the argmax function.

###  3.3 Smooth Predicting

Benefit from the timeline in videos, we can find the process of muscle
movement, _e.g_. the process from neural to happiness. So the expression
change in videos should be very smooth. But current 2D FER methods only
consider the spatial information in facial images and ignore the relative
information among frames. They predict every frame independently, which may
lead to different results in the same expression clip. Videos in the ABAW3
dataset have a high frame rate (30 fps), thus it does not make sense for
expressions changing very frequently in adjacent frames.

To address this issue, we propose a smooth predicting module to capture both
universal and unique features of adjacent frames. As illustrated in the right
part of Fig. 1, the SP module takes a series of feature logits as input and
enhance every feature by adding the average information among a specific
window (5 in Fig. 1 as an example) to the original feature.

Mathematically, given $n$ adjacent frame images,
$I_{0},I_{1},...I_{i-1},I_{i},I_{i+1},...I_{n-1}$, and the windows size ($w$),
we first utilize the network to extract the features from every images
independently. After this, $n$ image features will be obtained, named
$f_{0},...f_{i},...f_{n-1}$. Then, our proposed SP module will update every
frame feature by:

$f_{i}^{{}^{\prime}}=f_{i}+\frac{\sum_{j=i-w/2}^{i+w/2}f_{j}}{w}$ (1)

where the $f_{i}^{{}^{\prime}}$ denotes the updated feature. The updated
feature consists of two-part: the unique current frame feature and the
universal feature among the given window size. We find that this will improve
the smoothness of model outputs as well as the performance.

Methods | F1 (%)
---|---
Baseline | 23
Ours baseline | 40
Ours baseline + C-F | 42
Ours baseline + SP | 43
Ours | 46
Table 1: The ablation studies to our method with and without C-F strategy and
SP module on the validation set of Aff-Wild2.

##  4 Experiments

In this section, we will first introduce some training details and then give
some experiment results based on the official validation set of the Aff-Wild2
[19].

###  4.1 Details

We only used the Aff-Wild2 [18] to train our models. All videos are processed
into frame images by FFmpeg, and the MTCNN [39] is utilized to detect and
align facial images. All the facial images are cropped into 112$\times$112
except for Swin Transformer, which is 224 $\times$ 224\. To alleviate the
over-fitting problem, we utilized several data augmentation methods following
[36]: random rotate and crop, random flip, colorjit, blur, and random erasing
[44]. The over-sample method proposed in [9] is also adopted to reduce the
influence of the class imbalance problem.

An ensemble of IR152 [2], Swin-S [27], and HRNet [33] is adopted as our coarse
net, and the RepVGG [3] is adopted as the negative nets. All these backbone
nets are pre-trained on the Ms-Celeb-1M [8].

The cross-entropy loss is applied for training. And the f1-score (%) on the
validation set is reported by default. The detail of hyperparameters could be
found on the corresponding config file in our released codes. All of the
experiments are trained on one NVIDIA V100 graphic card.

Coarse | Negative |
---|---|---
Swin-S | IR152 | HRNet | RepVGG | Swin-S | RepVGG | F1 (%)
$\checkmark$ |  |  |  |  | $\checkmark$ | 43.78
| $\checkmark$ |  |  |  | $\checkmark$ | 42.74
|  | $\checkmark$ |  |  | $\checkmark$ | 43.78
|  |  | $\checkmark$ |  | $\checkmark$ | 40.38
$\checkmark$ | $\checkmark$ |  |  |  | $\checkmark$ | 44.25
$\checkmark$ | $\checkmark$ |  |  | $\checkmark$ |  | 43.28
$\checkmark$ | $\checkmark$ |  |  | $\checkmark$ | $\checkmark$ | 43.17
$\checkmark$ |  | $\checkmark$ |  |  | $\checkmark$ | 43.28
$\checkmark$ | $\checkmark$ | $\checkmark$ |  |  | $\checkmark$ | 46.15
$\checkmark$ | $\checkmark$ | $\checkmark$ |  | $\checkmark$ | $\checkmark$ | 44.62
Table 2: Results of different ensemble strategies on the validation set of the
Aff-Wild2. The Swin-S, IR152, and HRNet ensembled as the coarse net, and the
RepVGG as the negative net performs the best.

###  4.2 Results

Backbone | Coarse | Negative
---|---|---
IR152 [2] | 50.82 | 44.99
Swin-S [27] | 51.52 | 46.70
HRNet [33] | 50.67 | 45.60
RepVGG [3] | 48.40 | 49.82
Table 3: Performance on coarse and negative tasks with different backbones on the validation set of Aff-Wild2. $w$ | F1 (%)
---|---
0 | 42.28
32 | 43.29
64 | 43.47
128 | 43.69
256 | 43.78
512 | 43.68
Table 4: Evaluation of different windows size $w$ in our method on the validation set of the Aff-Wild2. Rank | Teams | F1 (%)
---|---|---
1 | Netease Fuxi Virtual Human [41] | 35.87
2 | IXLAB [11] | 33.77
3 | AlphaAff (Ours) | 32.17
4 | HSE-NN [30] | 30.25
5 | PRL [29] | 28.6
6 | dgu [13] | 27.2
7 | USTC-NELSLIP | 21.91
| baseline [14] | 20.50
Table 5: Results of the test set of the Expression Classification Challenge.

Tab. 1 shows the ablation studies of our method on the validation set of Aff-
Wild2. By using a more powerful backbone and training the whole model in an
end-to-end manner, the baseline performance increased from 23 to 40. The C-F
strategy brings another 2 performance increment, indicating that the C-F
strategy could reduce the influence of the class imbalance problem and
distinguish negative expressions efficiently. The performance further
increased to 46 by applying the SP module to help the model capture both
universal and unique features along the timeline.

To illustrate the impact of different backbone architectures, we show the
coarse and negative results with different backbones in Fig. 3. As we can see,
Swin-S achieves the best performance (51.52) in the coarse task and a
comparable result (46.70) in the negative task, indicating the great impact
achieved by the self-attention mechanism. The self-attention allows the
network to use different patterns to extract different content even if they
are in the same positions which are very useful to classify highly varied
samples. The RepVGG achieves the best performance (49.82) in the negative
task, but its performance on the coarse task is the worst (48.40). The
performances on different architectures various on these two tasks, indicating
they could obtain different features. Therefore, it is intuitive to ensemble
them to produce robust results.

Performance with different backbone ensemble strategies is illustrated in Fig.
2. The IR152 and HRNet perform badly in the negative task, so they are not
considered in this experiment. The RepVGG performs worse in the coarse task,
resulting in bad performance (40.38) in the overall performance. To our
surprise, the Swin-S performs badly when ensemble in the negative net. When
ensemble with Swin-S, the overall performance will decrease from 0.11 to 1.53.
So the Swin-S, IR152, and HRNet are ensembled as the final coarse net, and the
RepVGG is utilized solely as the negative net.

We also conduct experiments with different $w$ in the SP module to evaluate
the impact of windows size. As is illustrated in Tab. 4, as the window size
becomes larger, the model could capture more universal features, and benefit
from this, it could generate smooth and stable predictions. The best result
(43.78) is obtained when the size of the window is 256, which is about 8
seconds in time. When the $w$ becomes larger than 256, the performance becomes
worse. This is because the expression may change in such a long period.

The final results on the test set are illustrated in Tab. 5, our method
achieved 3rd place on the leader board. Different from other methods, which
utilize other modalities (_e.g_. voice, spoken words), we only adopt frame
images as our input. We believe with the help of modality fusion modules
proposed in other methods, the performance could increase further.

##  5 Conclusion

In this paper, we have proposed the Coarse-to-Fine Cascaded Networks with
Smooth Predicting (CFC-SP) for video facial expression recognition. To address
the problem of label ambiguity, we first group several similar emotions to
form a coarse category. Then, the cascaded networks were employed to first
conduct a coarse classification and then perform a fine classification.
Moreover, we proposed a Smooth Predicting (SP) to further improve the
performance. Experiments on Aff-Wild2 have verified the effectiveness of the
proposed method.

## References

  * [1] Didan Deng, Zhaokang Chen, and Bertram E Shi.  Multitask emotion recognition with incomplete labels.  In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pages 592–599. IEEE, 2020.
  * [2] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690–4699, 2019.
  * [3] Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun.  RepVGG: Making VGG-style ConvNets Great Again.
  * [4] Paul Ekman.  Facial expression and emotion.  American psychologist, 48(4):384, 1993.
  * [5] Paul Ekman and Wallace V. Friesen.  Facial action coding system: a technique for the measurement of facial movement.  1978\.
  * [6] Amir Hossein Farzaneh and Xiaojun Qi.  Facial expression recognition in the wild via deep attentive center loss.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2402–2411, 2021.
  * [7] Guodong Guo and Charles R Dyer.  Learning from examples in the small sample case: face expression recognition.  IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 35(3):477–488, 2005.
  * [8] Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao.  Ms-celeb-1m: A dataset and benchmark for large-scale face recognition.  In European conference on computer vision, pages 87–102. Springer, 2016.
  * [9] Agrim Gupta, Piotr Dollar, and Ross Girshick.  LVIS: A dataset for large vocabulary instance segmentation.  Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019-June:5351–5359, aug 2019.
  * [10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [11] Jae-Yeop Jeong, Yeong-Gi Hong, Daun Kim, Yuchul Jung, and Jin-Woo Jeong.  Facial expression recognition based on multi-head cross attention network.  arXiv preprint arXiv:2203.13235, 2022.
  * [12] Yue Jin, Tianqing Zheng, Chao Gao, and Guoqiang Xu.  A multi-modal and multi-task learning method for action unit and expression recognition.  arXiv preprint arXiv:2107.04187, 2021.
  * [13] Jun-Hwa Kim, Namho Kim, and Chee Sun Won.  Facial expression recognition with swin transformer.  arXiv preprint arXiv:2203.13472, 2022.
  * [14] Dimitrios Kollias.  Abaw: Valence-arousal estimation, expression recognition, action unit detection & multi-task learning challenges.  arXiv preprint arXiv:2202.10659, 2022.
  * [15] Dimitrios Kollias, Attila Schulc, Elnar Hajiyev, and Stefanos Zafeiriou.  Analysing affective behavior in the first abaw 2020 competition.  In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pages 637–643. IEEE, 2020.
  * [16] Dimitrios Kollias, Viktoriia Sharmanska, and Stefanos Zafeiriou.  Face behavior a la carte: Expressions, affect and action units in a single network.  arXiv preprint arXiv:1910.11111, 2019.
  * [17] Dimitrios Kollias, Viktoriia Sharmanska, and Stefanos Zafeiriou.  Distribution matching for heterogeneous multi-task learning: a large-scale face study.  arXiv preprint arXiv:2105.03790, 2021.
  * [18] Dimitrios Kollias, Panagiotis Tzirakis, Mihalis A Nicolaou, Athanasios Papaioannou, Guoying Zhao, Björn Schuller, Irene Kotsia, and Stefanos Zafeiriou.  Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architectures, and beyond.  International Journal of Computer Vision, 127(6):907–929, 2019\.
  * [19] Dimitrios Kollias and Stefanos Zafeiriou.  Expression, affect, action unit recognition: Aff-wild2, multi-task learning and arcface.  arXiv preprint arXiv:1910.04855, 2019.
  * [20] Dimitrios Kollias and Stefanos Zafeiriou.  Affect analysis in-the-wild: Valence-arousal, expressions, action units and a unified framework.  arXiv preprint arXiv:2103.15792, 2021.
  * [21] Dimitrios Kollias and Stefanos Zafeiriou.  Analysing affective behavior in the second abaw2 competition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3652–3660, 2021.
  * [22] Felix Kuhnke, Lars Rumberg, and Jörn Ostermann.  Two-stream aural-visual affect analysis in the wild.  In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pages 600–605. IEEE, 2020.
  * [23] Hangyu Li, Nannan Wang, Xinpeng Ding, Xi Yang, and Xinbo Gao.  Adaptively learning facial expression representation via cf labels and distillation.  IEEE Transactions on Image Processing, 30:2016–2028, 2021.
  * [24] Jing Li, Kan Jin, Dalin Zhou, Naoyuki Kubota, and Zhaojie Ju.  Attention mechanism-based cnn for facial expression recognition.  Neurocomputing, 411:340–350, 2020.
  * [25] Shan Li, Weihong Deng, and JunPing Du.  Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 28, pages 2584–2593. IEEE, July 2017.
  * [26] Yong Li, Jiabei Zeng, Shiguang Shan, and Xilin Chen.  Occlusion aware facial expression recognition using cnn with attention mechanism.  IEEE Transactions on Image Processing, 28(5):2439–2450, 2018.
  * [27] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021.
  * [28] Ali Mollahosseini, Behzad Hasani, and Mohammad H Mahoor.  Affectnet: A database for facial expression, valence, and arousal computing in the wild.  IEEE Transactions on Affective Computing, 10(1):18–31, 2017.
  * [29] Kim Ngan Phan, Hong-Hai Nguyen, Van-Thong Huynh, and Soo-Hyung Kim.  Expression classification using concatenation of deep neural network for the 3rd abaw3 competition.  arXiv preprint arXiv:2203.12899, 2022.
  * [30] Andrey V Savchenko.  Frame-level prediction of facial expressions, valence, arousal and action units for mobile devices.  arXiv preprint arXiv:2203.13436, 2022.
  * [31] Jiahui She, Yibo Hu, Hailin Shi, Jun Wang, Qiu Shen, and Tao Mei.  Dive into ambiguity: Latent distribution mining and pairwise uncertainty estimation for facial expression recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6248–6257, June 2021.
  * [32] Jiawei Shi, Songhao Zhu, and Zhiwei Liang.  Learning to amend facial expression representation via de-albino and affinity.  arXiv preprint arXiv:2103.10189, 2021.
  * [33] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao.  Deep High-Resolution Representation Learning for Visual Recognition.  43(10):3349–3364.
  * [34] Kai Wang, Xiaojiang Peng, Jianfei Yang, Debin Meng, and Yu Qiao.  Region attention networks for pose and occlusion robust facial expression recognition.  IEEE Transactions on Image Processing, 29:4057–4069, 2020.
  * [35] Jun Weng, Yang Yang, Zichang Tan, and Zhen Lei.  Attentive hybrid feature with two-step fusion for facial expression recognition.  In 2020 25th International Conference on Pattern Recognition (ICPR), pages 6410–6416. IEEE, 2021.
  * [36] Fanglei Xue, Qiangchang Wang, and Guodong Guo.  TransFER: Learning Relation-aware Facial Expression Representations with Transformers.  In Proceedings of the IEEE International Conference on Computer Vision, Mar. 2021.
  * [37] Stefanos Zafeiriou, Dimitrios Kollias, Mihalis A Nicolaou, Athanasios Papaioannou, Guoying Zhao, and Irene Kotsia.  Aff-wild: Valence and arousal ‘in-the-wild’challenge.  In Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on, pages 1980–1987. IEEE, 2017.
  * [38] Jiabei Zeng, Shiguang Shan, and Xilin Chen.  Facial expression recognition with inconsistently annotated datasets.  In Proceedings of the European conference on computer vision (ECCV), pages 222–237, 2018.
  * [39] Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao.  Joint face detection and alignment using multitask cascaded convolutional networks.  IEEE Signal Processing Letters, 23(10):1499–1503, 2016.
  * [40] Wei Zhang, Zunhu Guo, Keyu Chen, Lincheng Li, Zhimeng Zhang, and Yu Ding.  Prior aided streaming network for multi-task affective recognitionat the 2nd abaw2 competition.  arXiv preprint arXiv:2107.03708, 2021.
  * [41] Wei Zhang, Zhimeng Zhang, Feng Qiu, Suzhen Wang, Bowen Ma, Hao Zeng, Rudong An, and Yu Ding.  Transformer-based multimodal information fusion for facial expression analysis.  arXiv preprint arXiv:2203.12367, 2022.
  * [42] Guoying Zhao, Xiaohua Huang, Matti Taini, Stan Z Li, and Matti PietikäInen.  Facial expression recognition from near-infrared videos.  Image and Vision Computing, 29(9):607–619, 2011.
  * [43] Guoying Zhao and Matti Pietikainen.  Dynamic texture recognition using local binary patterns with an application to facial expressions.  IEEE transactions on pattern analysis and machine intelligence, 29(6):915–928, 2007.
  * [44] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang.  Random erasing data augmentation.  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 13001–13008, 2020.


# Growth of groups of wind generated waves

F. Drullion1 and S.G. Sajjadi1,2 1 Department of Mathematics, ERAU, Florida,
USA.
2 Trinity College, University of Cambridge, UK.

###### Abstract

In this paper we demonstrate numerical computations of turbulent wind blowing
over group of waves that are growing in time. The numerical model adopted for
the turbulence model is based on differential second-moment model that was
adopted for growing idealized waves by Drullion & Sajjadi (2014). The results
obtained here demonstrate the formation of cat’s-eye which appear
asymmetrically over the waves within a group.

##  1 Introduction

The question of growth and decay of wind generated waves in the ocean has been
studied extensively but the interactions and energy transfer between the water
wave and the ambient wind is still not fully understood. Most of the studies
(experimental, numerical or analytical) consider monochromatic or idealized
waves profile. As it has been observed extensively that waves actually travel
in groups for which the profile changes as the wave is traveling. In this
study we are considering the influence of grouping on the growth of ocean
waves induced by wind. We are concentrating our efforts on the region around
the height were the real part of the complex wave speed is equal to the mean
flow velocity. This region called “critical layer” is at the center of Miles’
(1957) theory and Lighthill’s (1962) interpretation of growth waves. In this
region closed streamlines structures called “cat’s-eye” are developed. The
larger these structures are, the more disturbance of the wind flow above the
wave occurs. In some previous work, e.g. Drullion & Sajjadi (2014), a high-
Reynolds number stress closure model over a moving idealized wavy surface was
used to show that their size and position are dependent on the wave age and
wave steepness, which is in accordance with direct numerical simulations of
Sullivan et al.(2000). In this study we use the same Reynolds stress model
over different groups of waves to determine the height of the critical layer
and the overall shape and size.

##  2 Growth of waves within a group

In this paper we report computations of turbulent flow over two groups, in a
frame of reference moving with the wave, namely group1 and group2, for three
different wave ages: $c_{r}/U_{*}=0.98$, $c_{r}/U_{*}=3.5$, and
$c_{r}/U_{*}=7$. In the case of group1 the group amplitude is kept fixed, i.e.
non-growing waves. In contrast, we consider a case where the wave grows.

For group1 we adopt the following profile:

$z=a[\cos(kx)+\cos(k_{1}x)+\cos(k_{2}x)]$

where $a$ is the wave amplitude which we have chosen to be 0.0015 m, $k$ is
the wave number, $k_{1}=1+\sqrt{2}ak$ and $k_{2}=1-\sqrt{2}ak$. In both wave
groups the wavelength is $\lambda=0.0508$ m.

The profile for group2 is taken to be:

$z=a(t)[\cos(kx)+0.3\cos(k_{1}x)+0.3\cos(k_{2}x)]$

where am, the amplitude is set at 0.00165, $k_{1}$ and $k_{2}$ are the same as
for group1.

In our computations, the domain is taken to six wavelengths horizontally and
two wavelengths vertically. The groups only extend over four wavelengths (from
$x=0$ to $x=4\lambda$) and are surrounded by a flat surface. The latter
ensures the periodicity in boundary conditions in the $x$-direction. The
condition $U=U_{\lambda}$ is imposed for the top boundary condition, and for
the south boundary condition we prescribe the orbital velocity of the wave
groups.

The orbital velocities for the group1 (for $0\leq x\leq 4\lambda$) is given by

$\displaystyle u$ $\displaystyle=$ $\displaystyle-
c_{g}ak[\cos(kx)+k_{1}\cos(k_{1}x)+k_{2}\cos(k_{2}x)]-c_{g}$ $\displaystyle v$
$\displaystyle=$ $\displaystyle-
c_{g}ak[\sin(kx)+k_{1}\sin(k_{1}x)+k_{2}\sin(k_{2}x)]$

where $c_{g}$ is the group velocity. Similarly for the group2 the imposed
orbital velocities is taken to be

$\displaystyle u$ $\displaystyle=$ $\displaystyle-
c_{g}a(t)k[\cos(kx)+0.3k_{1}\cos(k_{1}x)+0.3k_{2}\cos(k_{2}x)]-c_{g}$
$\displaystyle v$ $\displaystyle=$ $\displaystyle-
c_{g}a(t)k[\sin(kx)+0.3k_{1}\sin(k_{1}x)+0.3k_{2}\sin(k_{2}x)]$

Note that, for the flat surfaces surrounding the group portion on the south
boundary, namely when $x<0$ and $x>4\lambda$, we impose the conditions
$u=-c_{g}$ and $v=0$.

For the growing wave groups, the computational mesh is regenerated every 50
time steps, where each time step consists of 500 iterations and is increased
as the waves become steeper. The growth factor for each wave within the group
is $e^{Kc_{i}t}$, where $K$ can be taken to be $k$, $k_{1}$ or $k_{2}$ and
$c_{i}=8c_{g}a/\lambda$.

Figure 1: Stream function contour plots of stream function over group1 wave
for three values of the wave age $c_{r}/U_{*}=0.98$ (top); 3.7 (middle), and 7
(bottom). Figure 2: Stream function contour plots of stream function over
group2 wave for three values of the wave age $c_{r}/U_{*}=0.98$ (top); 3.7
(middle), and 7 (bottom).  Figure 3: Stream function contour plots of stream
function over growing group1 wave for a single value of the wave age
$c_{r}/U_{*}=5.75$. Sallower wave (top) and steepest wave (bottom).

##  3 Results and conclusions

A computational turbulence model is used to compute the disturbed turbulent
air flow over two different kind of wave group which we have referred to as
group1 and group2.

Figure 1 shows the contour plot of the stream function for group1 as a
function for three wave ages $c_{r}/U_{*}=0.98,3.5$ and 7. As can be seen from
this set of figures, at lowest value of $c_{r}/U_{*}$ a cat’s-eye is formed
downstream of the last wave in a group. As the wave age increases the same
cat’s eye become smaller but lifts up and moves toward the peak of the last
wave, also formation of weaker cat’s eye over other waves, within the group,
become visible. At $c_{r}/U_{*}=7$ the cat’s-eye become stronger and move
further over the peak of the waves in the group.

In figure 2 we perform the same computation for the group2 for the same wave
ages. In this case no cat’s eye appear at $c_{r}/U_{*}=0.98$. However, at
$c_{r}/U_{*}=3.5$ two cat’s eye is formed; a stronger one in the lee of the
the first wave in the group, and a weaker one in the lee of the third wave in
a group, but unlike group1 no cat’s-eye is formed in the lee of the last wave
in the group.

In figure 3 we show the the result of computations for a growing wave group
group1 for one fixed value of $c_{r}/U_{*}=5.75$. As can bee seen from these
figures, as the wave steepens cat’s-eye are formed in each lee of the wave in
the group. As the wave grows so do the cat’s-eyes, and similar to our other
computations for monochromatic waves and bimodal Stokes waves (see also
Sullivan et al. (2000)) the critical height rises further up from the surface
of the waves. It is also evident that the flow become more asymmetrical which
shows how the air flow over the downwind part of the group is lower than over
the upwind part. We remark that as the wave steepens the number of iteration
must be increased in order to obtain a converged results. In what we have
reported there is some doubt as the steepness is increased whether the
solution for the steeper waves have completely converged. This is part of our
on going investigation.

We thus conclude that this asymmetry causes the critical layer height to be
lower over the downwind part. This is in line with the conclusion of our
earlier paper, (Sajjadi, Hunt and Drullion (2016)) that the positive growth of
the individual waves on the upwind part of the wave group exceeds the negative
growth on the downwind part. Hence, this leads to the critical layer group
effect producing a net horizontal force on the waves, in addition to the
sheltering effect.

## References

  * [Drullion & Sajjadi (2014)] Drullion, F. & Sajjadi, S.G. 2014 Interaction of wind with surface water waves. Neural, Parallel, and Sci. Comp. 22, 303–314.
  * [Lighthill (1962)] Lighthill, M.J. 1962 Physical interpretation of the theory of wind generated waves. J. Fluid Mech. 14, 385–398.
  * [Miles (1957)] Miles, J.W. 1957 On the generation of surface waves by shear flows. J. Fluid Mech. 3, 185–204.
  * [Sajjadi, Hunt & Drullion (2016)] Sajjadi, S. G., Hunt, J. C. R., Drullion, F. 2016 Growth of unsteady wave groups by shear flows. Proc. of IMA Conference on Turbulence, Waves and Mixing, Kings College Cambridge, U.K., July 2016, 79–84.
  * [sulmac (2000)] Sullivan, P.P., McWilliams, J.C. & Moeng, C.H. 2000 Simulation of turbulent flow over idealized water waves J. Fluid Mech., 404, 47–85.