# Impact of detecting clinical trial elements in exploration of COVID-19
literature

Simon Šuster  Fac. of Engineering and Information Tech.
The University of Melbourne
Melbourne, Australia
<EMAIL_ADDRESS>Karin Verspoor  School of Computing Technologies
RMIT University
Melbourne, Australia
<EMAIL_ADDRESS>Timothy Baldwin  Fac. of Engineering and
Information Tech.
The University of Melbourne
Melbourne, Australia
<EMAIL_ADDRESS>Jey Han Lau  Fac. of Engineering and Information
Tech.
The University of Melbourne
Melbourne, Australia
<EMAIL_ADDRESS>Antonio Jimeno Yepes  Fac. of Engineering and
Information Tech.
The University of Melbourne
Melbourne, Australia
<EMAIL_ADDRESS>David Martinez Iraola  Independent researcher
Melbourne, Australia
<EMAIL_ADDRESS>Yulia Otmakhova  Fac. of Engineering and
Information Tech.
The University of Melbourne
Melbourne, Australia
<EMAIL_ADDRESS>
###### Abstract

The COVID-19 pandemic has driven ever-greater demand for tools which enable
efficient exploration of biomedical literature. Although semi-structured
information resulting from concept recognition and detection of the defining
elements of clinical trials (e.g. PICO criteria) has been commonly used to
support literature search, the contributions of this abstraction remain poorly
understood, especially in relation to text-based retrieval. In this study, we
compare the results retrieved by a standard search engine with those filtered
using clinically-relevant concepts and their relations. With analysis based on
the annotations from the TREC-COVID shared task, we obtain quantitative as
well as qualitative insights into characteristics of relational and concept-
based literature exploration. Most importantly, we find that the relational
concept selection filters the original retrieved collection in a way that
decreases the proportion of unjudged documents and increases the precision,
which means that the user is likely to be exposed to a larger number of
relevant documents.

##  I Introduction

The outbreak of Coronavirus disease 2019 (COVID-19) has led to a vigorous
response from the global medical and AI communities, with efforts in the
fields of information retrieval and natural language processing revolving
around dataset construction and the development of tools for managing the
growing literature on the virus and related diseases [1]. The release of the
COVID-19 Open Research Dataset (CORD-19) [2] has stimulated the development of
a large number of tools, reviewed in [3], which can be divided into more
retrieval/QA-focused systems—returning a list of relevant documents for a
user-specified query [4, 5, 6, 7, 8]—and those that use domain knowledge to
organise and present information found in the literature [9, 10, 11, 12].

Broad clinically-relevant PICO categories derived from the structure of
clinical trials [13] have been used to enable structured search as well as the
visualisation of document content [10, 9, 14, 15, 11]. These categories
describe the Patient population enrolled (e.g. diabetics), the Interventions
studied (e.g. insulin) and to what they were Compared (e.g. placebo), and the
Outcomes measured (e.g. blood glucose levels).

Here, we aim to examine the impact of PICO content structuring on literature
exploration. Our analysis is based on the COVID-SEE system [11, 16] which
offers both simple document retrieval and diverse visualisations of the
retrieved document collection. We specifically focus on the information
presented in the relational concept selection of COVID-SEE, which adopts
Sankey diagrams to visually organise the medical concepts found in the
articles according to PICO categories. The aim of presenting the retrieved
documents in this way is to highlight salient P–I and I–O concept
relations111The comparator (‘C’) category is usually merged with interventions
(‘I’) due to their high similarity for the purposes of automatic PICO
labelling [17]. We also follow this practice here.. Figure 1 gives an example.

Figure 1: Sankey diagram of PICO concepts and relations for 35 articles
retrieved for what kinds of complications related to COVID-19 are associated
with diabetes (TREC Round 5, query no. 24). Selecting a link (e.g. Diabetes
Mellitus $\rightarrow$ Chloroquine) reveals papers that contain that relation
(the displayed relation in the example only contains one document). Links are
weighted by the number of documents containing the relevant relation.

P–I and I–O relations often cannot be extracted from the retrieved documents,
as not all articles are structured according to PICO and the identification of
PICO elements is imperfect. This means that the relational concept selection
shown in the visual summary is implicitly filtering a smaller subset of
documents from the retrieval results. The question we seek to answer is how
this affects the quality of the results. We approach this question by
analysing the results through an existing test collection of relevance
decisions, rather than carrying out a user study, which we leave for future
work. To obtain a ground truth for relevance of the retrieved documents, we
use the judgements for COVID-19-related literature obtained in the TREC-COVID
shared task [18].

Our findings reveal the following:

  * •

While the relational concept selection reduces the proportion of displayed
documents to around 15% of the base retrieval results, the proportion of
relevant documents actually increases. We find that this advantage comes from
the fact that the relational concept selection is more likely to show the
documents that received a judgment in the TREC-COVID test set (relevant or
irrelevant).

  * •

Based on manually rating a sample of unjudged documents, we find that most are
irrelevant, suggesting that PICO filtering boosts the overall quality of the
search results.

  * •

Furthermore, when calculating precision (the proportion of relevant documents
to all retrieved documents), we observe that the results vary substantially
across queries. This leads us to infer that queries which are more amenable to
PICO representation—based on a manual annotation of PICO elements in the
queries—also tend to correlate with higher precision in the relational concept
selection. In a similar vein, we find that the strength of a P–I/I–O relation
(i.e. the number of articles contained in that particular relation) is
positively correlated with the proportion of relevant documents included.
These results indicate that for medical queries with more easily identifiable
PICO elements, the use of these elements as a central organising principle in
the COVID-SEE system leads to users being exposed to more relevant results.

##  II Empirical setup

###  II-A Data

In our experiments, we use the CORD-19 dataset (v. 16 July 2020, totalling
around 192,000 documents). It is currently the most extensive publicly
available corpus of coronavirus-related publications from different sources,
including PubMed’s PMC open access corpus, research articles from a corpus
maintained by the WHO, and bioRxiv and medRxiv pre-prints [2].

#### PICO-concept annotation

To annotate PICO elements and biomedical concepts found within PICO phrases,
we follow a two-step procedure described in detail in [16, 11]. Briefly, in
the first step, a BiLSTM-CRF model trained on the EBM-NLP dataset [17] is used
to label textual spans with Population, Intervention, and Outcome categories.
On the same dataset, the model’s performance has been estimated at 0.78, 0.60
and 0.67 F1 for P, I and O categories, respectively. The next step consists of
applying MetaMap [19] to identify the terms that correspond to Medical Subject
Headings (MeSH) [20]. MeSH terms provide an established vocabulary for medical
articles indexed by PubMed. We keep only those MeSH terms that are found
within the PICO spans. The procedure is applied to all titles and abstracts in
our collection.

#### TREC-COVID test set

TREC-COVID is a TREC retrieval task addressing literature retrieval related to
the COVID-19 pandemic [18, 21]. The topics were developed based on searches
submitted to the National Library of Medicine and suggestions from researchers
on Twitter. They are representative of the high-level information needs
related to the pandemic. Each topic consists of three fields of increasing
granularity: a keyword-based query, a natural language question, and a longer
descriptive narrative. We use the intermediate-level natural language
questions as our queries. The first round of TREC-COVID introduced a set of 30
topics, and was based on the 10 April 2020 release of CORD-19. The most
recent, fifth, round includes an additional set of 20 topics, and exploits the
16 July 2020 release of CORD-19.

The relevance judgments in TREC-COVID were assigned by individuals with
biomedical expertise, based on a pooling approach in which only the top-ranked
results from different submissions are assessed. A document is judged as
‘relevant’, ‘partially relevant’, or ‘not relevant’. We consider the first two
labels as relevant in our experiments.

###  II-B Search and visualisation

In COVID-SEE, the CORD dataset is stored as a graph database (neo4j). Queries
are executed against the full-text index over the titles and abstracts of the
papers. We employ a very simple BM25 baseline which while not state of the art
represents a standard (strong) baseline for IR tasks [22, 23]. The search
capability in neo4j is powered by the Apache Lucene [24], with a default VSM
scoring function based on tf-idf. We limit the output of the IR search to 1000
hits.

The relational concept selection organises the identified MeSH concepts into
PICO categories, and shows how they interact in a Sankey diagram (Figure 1).
The diagram displays relations between pairs of concepts, where the relations
‘carry’ the corresponding documents, and the strength of a relation
corresponds to the number of documents in which that concept pair is attested.
The user interface for the relational concept selection supports varying
levels of conceptual match. All our experiments involve MeSH terms of
granularity 1, meaning that terms belonging to varying levels of the MeSH
hierarchy get mapped to a common level, which is the one of the 16 topmost
MeSH categories in combination with a tree descriptor. For example, Canada
(MeSH tree number: Z01.107.567.176) is mapped (truncated) to Geographic
Locations (Z01), and Hydroxychloroquine (D03.633.100.810.050.180.350) to
Heterocyclic Compounds (D03).

###  II-C Evaluation details

We evaluate the retrieval runs with the standard TREC script [25]. We report
precision, based on the retrieved documents $R$ that were either judged as
relevant (rel) or irrelevant (irrel), or were not judged at all in the TREC
test set (unj):
$\frac{|R^{\text{rel}}|}{|R^{\text{rel}}|+|R^{\text{irrel}}|+|R^{\text{unj}}|}$.
We additionally report a score that ignores the unjudged documents, referred
to as precisionjudg.

Note that there is no ranking in the relational concept selection, since
documents are shown without ordering or scoring. Hence, we do not use
evaluation metrics that are rank-dependent.

##  III Analysis

###  III-A Relevance of documents in the relational concept selection

The relational concept selection implicitly acts as a filter for the documents
retrieved by the IR engine, as some document abstracts do not contain any
identifiable P–I or I–O pairs. Overall, the median percentage of documents
(across all queries) containing PICO-typed concept relations after pre-
processing is 13% (with the minimum and maximum for a given query being 8% and
28%, respectively). To understand what documents are selected and the
retrieval quality of that selection, we carry out several experiments.

We first compare the precision scores of documents in the raw retrieval
results vs. documents in the filtered collection, across all 50 queries, as
detailed in Figure 2. The median score for documents in the relational concept
selection is higher (0.17) than for the raw search results (0.12). The three
queries with the highest precision are as follows, where the last query (28)
achieves the highest improvement over raw search results (0.28 $\rightarrow$
0.54):

  * (38)

What is the mechanism of inflammatory response and pathogenesis of COVID-19
cases?

  * (39)

What is the mechanism of cytokine storm syndrome on the COVID-19?

  * (28)

What evidence is there for the value of hydroxychloroquine in treating
Covid-19?

These queries can all be seen as precise statements about the required
Problem/Population (inflammatory response, cytokine storm syndrome, Covid-19,
respectively) and Intervention (hydroxychloroquine in query 28; the other two
queries lack this criterion).

Figure 2: A box plot comparing the precision (section II-C) between the result
sets retrieved for each query by the search engine (lucene) and the relational
concept selection. The numbers represent the TREC-COVID query identifiers.

###  III-B Role of unjudged documents

The calculation of precision (Section II-C) includes in the denominator the
unjudged documents $R^{unj}$. Since we have little a priori intuition about
the nature and the relevance of these documents, or whether presenting them to
the user can be favourable or not, we manually analysed a sample of them,
randomly selecting ten queries and picking for each the first 50 unjudged
documents.222Based on our document ranker. We then checked the abstract of
each document to decide whether it is relevant to the query. The results are
shown in Table I. Although some unjudged documents are relevant (cf. [26]),
for all but one query the precision of the unjudged documents is far below the
prior probability based on TREC-COVID judgements. We therefore infer that
including fewer unjudged documents in the results should be favourable. This
is in line with the view of the unjudged documents found in the literature
[27].

We observe that the increased precision of results in the relational concept
selection is largely due to including fewer unjudged documents: the proportion
of retrieved unjudged documents is high in the raw search results ($\mu$=0.74,
min=0.5, max=0.94, n=50), but somewhat lower in the relational selection
($\mu$=0.65, min=0.4, max=0.9, n=50). In practice, this means that the user
can expect to encounter—per 100 documents retrieved—9 unjudged documents less
in the relational view compared to the IR list.

While the number of hits is on average close to 1000 for the search engine,
the relational concept view typically captures only around 15% of the
retrieved articles ($\mu$=148 documents, min=78, max=278, n=50). The PICO view
filters from the IR search list without any particular order, and in a way
that is not guided by the original search ranking.

query | # rel. | P | Pjudg | $\Delta$
---|---|---|---|---
$2$ | $1$ | $0.02$ | $0.27$ | $-0.25$
$11$ | $3$ | $0.06$ | $0.17$ | $-0.11$
$15$ | $1$ | $0.02$ | $0.09$ | $-0.07$
$18$ | $10$ | $0.20$ | $0.89$ | $-0.69$
$19$ | $4$ | $0.08$ | $0.45$ | $-0.37$
$22$ | $13$ | $0.26$ | $0.56$ | $-0.3$
$26$ | $8$ | $0.16$ | $0.67$ | $-0.51$
$34$ | $8$ | $0.16$ | $0.06$ | $0.10$
$38$ | $20$ | $0.40$ | $0.85$ | $-0.45$
$47$ | $3$ | $0.06$ | $0.36$ | $-0.30$
TABLE I: Summary of manual annotations of relevance on top-50 unjudged
documents from TREC-COVID for each query. P refers to our estimated precision
among these unjudged documents in TREC-COVID; Pjudg is the precision obtained
on judged documents only (whether relevant or irrelevant, but excluding the
unjudged ones); $\Delta$ quantifies the difference in the reported precision
scores.

###  III-C Exploring the relational concept selection

#### The nature of queries and their fit to the relational selection

Given the design of COVID-SEE in which all P- and I-typed concepts (as well as
I- and O-typed concepts) are considered to be related if they co-occur within
the same abstract without any further constraints, there is no guarantee that
the P–I and I–O pairs truly correspond to the initial query. The query may not
clearly state any PICO criteria, or the abstract may introduce PICO elements
that are not mentioned in the query. Therefore, we would like to better
understand what proportion of PICO-typed MeSH terms (or nodes of the graph) in
the relational concept selection match the PICO criteria expressed in a query.
Since the TREC-COVID queries are not annotated with PICO criteria or MeSH
terms, we manually annotated all 30 queries from round 1 of the shared task
with this information.

The results in Table II show that the percentage of nodes that match the query
directly is generally low, on average around 7% for Population and
Intervention, and 12% for Outcome. We find a moderate correlation between the
proportions of Intervention concepts fitting the query and the precision
scores ($\rho$=0.53), but the number of queries with a clearly stated
Intervention is small ($n$=12).

type | $\mu$ | min | max | $n$
---|---|---|---|---
P | $7$ | 1 | 14 | 20
I | $7$ | 3 | 17 | 12
O | $12$ | 3 | 18 | 4
TABLE II: Percentage of PICO-typed MeSH concepts included in the relation
concept selection that are directly relevant to the query. $\mu$ is the mean
over $n$ queries (not all queries express PICO criteria).

#### Extent of grouping

We next explore to what extent the documents represented in the relational
concept selection are grouped (pooled) together under the same relation, and
whether more evidence of grouping suggests a higher precision or a better fit
to the query. Analysis (not shown) reveals that: (a) most relations are
sparse, only corresponding to a single document; (b) the ratio of relations
representing more than one document varies between 0.17 and 0.47, depending on
the query; and (c) for some queries, relations cover a large number of
documents, e.g. for query 7, some relations contain more than 25 documents,
whereas for query 8, fewer relations cover a large number of documents. The
correlation between the number of documents in a relation and the precision
scores for each relation (across all queries) reveals a slight positive
correlation, with $\rho$=0.22 on 85,287 relations.

##  IV Further discussion & conclusion

From this analysis, we can conclude that a query processing approach that
directly relies on matching a PICO-based structured query, such as the
semantic search available as an alternative search strategy in COVID-SEE as
well as in SciSight [9] and DOC Search [14], is unlikely to be effective for
the TREC-COVID queries. This reflects the fact that these queries are
generally quite open-ended [21], and in many cases do not clearly follow the
PICO structure. The finding is also consistent with previous research
examining the impact of explicitly using defined PICO criteria in search
strategies for systematic reviews, which found that the inclusion of some PICO
elements, specifically Outcome, significantly reduced recall [28].

However, our analysis of information presented in the selection of documents
based on PICO categories and medical concepts suggests some benefits of
integrating PICO. Specifically, the user is more likely to be exposed to more
relevant documents, and fewer unjudged documents, compared to the raw search
results. Additionally, the user is more likely to find the PICO concepts
related to the initial query in those relations that cover a larger number of
articles. These stronger relations then tend to represent to a greater extent
the articles relevant to the query. A limitation is that different queries
express the PICO concepts in various degrees, which affects the selection of
articles and their overall relevance.

## References

  * [1] M. Hutson, “Artificial-intelligence tools aim to tame the coronavirus literature,” Nature, June 2020.
  * [2] L. L. Wang, K. Lo, Y. Chandrasekhar, R. Reas, J. Yang, D. Burdick, D. Eide, K. Funk, Y. Katsis, R. M. Kinney, Y. Li, Z. Liu, W. Merrill, P. Mooney, D. A. Murdick, D. Rishi, J. Sheehan, Z. Shen, B. Stilson, A. D. Wade, K. Wang, N. X. R. Wang, C. Wilhelm, B. Xie, D. M. Raymond, D. S. Weld, O. Etzioni, and S. Kohlmeier, “CORD-19: The COVID-19 open research dataset,” in Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020, (Online), Association for Computational Linguistics, July 2020.
  * [3] L. L. Wang and K. Lo, “Text mining approaches for dealing with the rapidly expanding literature on COVID-19,” Briefings in Bioinformatics, 12 2020\.  bbaa296.
  * [4] CORD-19 Search, accessed February 11, 2021.
  * [5] Discovid, accessed February 11, 2021.
  * [6] Covidex, accessed February 11, 2021.
  * [7] CORD-19 Search, accessed February 11, 2021.
  * [8] COVID Scholar, accessed February 11, 2021.
  * [9] T. Hope, J. Portenoy, K. Vasan, J. Borchardt, E. Horvitz, D. S. Weld, M. A. Hearst, and J. D. West, “SciSight: Combining faceted navigation and research group detection for COVID-19 exploratory scientific search,” bioRxiv, 2020.
  * [10] B. Nye, A. Nenkova, I. Marshall, and B. C. Wallace, “Trialstreamer: Mapping and browsing medical evidence in real-time,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, (Online), pp. 63–69, Association for Computational Linguistics, July 2020.
  * [11] K. Verspoor, S. Šuster, Y. Otmakhova, S. Mendis, Z. Zhai, B. Fang, J. H. Lau, T. Baldwin, A. Jimeno Yepes, and D. Martinez, “Brief description of covid-see: The scientific evidence explorer for covid-19 related research,” in Advances in Information Retrieval (D. Hiemstra, M.-F. Moens, J. Mothe, R. Perego, M. Potthast, and F. Sebastiani, eds.), (Cham), pp. 559–564, Springer International Publishing, 2021.
  * [12] COVID-19 Navigator, accessed February 11, 2021.
  * [13] W. S. Richardson, M. C. Wilson, J. Nishikawa, R. S. Hayward, et al., “The well-built clinical question: a key to evidence-based decisions,” Acp j club, vol. 123, no. 3, pp. A12–3, 1995.
  * [14] DOC Search, accessed February 11, 2021.
  * [15] COVID-19 Living OVerview of Evidence, accessed February 11, 2021.
  * [16] K. Verspoor, S. Šuster, Y. Otmakhova, S. Mendis, Z. Zhai, B. Fang, J. H. Lau, T. Baldwin, A. J. Yepes, and D. Martinez, “Covid-see: Scientific evidence explorer for covid-19 related research,” arXiv preprint arXiv:2008.07880, 2020.
  * [17] B. Nye, J. J. Li, R. Patel, Y. Yang, I. Marshall, A. Nenkova, and B. Wallace, “A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), (Melbourne, Australia), pp. 197–207, Association for Computational Linguistics, July 2018.
  * [18] K. Roberts, T. Alam, S. Bedrick, D. Demner-Fushman, K. Lo, I. Soboroff, E. Voorhees, L. L. Wang, and W. R. Hersh, “TREC-COVID: Rationale and Structure of an Information Retrieval Shared Task for COVID-19,” Journal of the American Medical Informatics Association, 05 2020.  ocaa091.
  * [19] MetaMap - A Tool For Recognizing UMLS Concepts in Text, accessed February 11, 2021.
  * [20] MeSH (Medical Subject Headings), accessed February 11, 2021.
  * [21] E. Voorhees, T. Alam, S. Bedrick, D. Demner-Fushman, W. R. Hersh, K. Lo, K. Roberts, I. Soboroff, and L. L. Wang, “Trec-covid: Constructing a pandemic information retrieval test collection,” arXiv preprint arXiv:2005.04474, 2020.
  * [22] C. Kamphuis, A. P. de Vries, L. Boytsov, and J. Lin, “Which bm25 do you mean? a large-scale reproducibility study of scoring variants,” in European Conference on Information Retrieval, pp. 28–34, Springer, 2020.
  * [23] A. Trotman, A. Puurula, and B. Burgess, “Improvements to bm25 and language models examined,” in Proceedings of the 2014 Australasian Document Computing Symposium, pp. 58–65, 2014.
  * [24] Apache Lucene, accessed February 11, 2021.
  * [25] TREC evaluation script, accessed February 11, 2021.
  * [26] J. Zobel, “How reliable are the results of large-scale information retrieval experiments?,” in Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’98, (New York, NY, USA), p. 307–314, Association for Computing Machinery, 1998\.
  * [27] M. Sanderson, “Test collection based evaluation of information retrieval systems,” in Foundations and Trends in Information Retrieval, pp. 247–375, 2010.
  * [28] T. F. Frandsen, M. F. Bruun Nielsen, C. L. Lindhardt, and M. B. Eriksen, “Using the full pico model as a search tool for systematic reviews resulted in lower recall for some pico elements,” Journal of Clinical Epidemiology, vol. 127, pp. 69 – 75, 2020.


# Graph rewrites, from graphic lambda calculus, to chemlambda, to directed
interaction combinators

Marius Buliga

Institute of Mathematics, Romanian Academy
P.O. BOX 1-764, RO 014700
Bucureşti, Romania
<EMAIL_ADDRESS><EMAIL_ADDRESS>
(07.07.2020)

###### Abstract

Here I report about the modifications of and relations between graphic lambda
calculus, various formalisms which appeared under the name chemlambda and a
version of directed interaction combinators.

This is part of the study and experiments with the artificial chemistry
chemlambda and the relations with lambda calculus or interaction combinators,
as described in arXiv:2003.14332 and available from the entry page
chemlambda.github.io [12].

## Context

The chemlambda project context and relevant previous work are explained in
Section 4 (About this project) of [4] arXiv:2003.14332 See also, for more
mathematical background, the presentation [6] Emergent rewrites in knot theory
and logic.

Here I report about the modifications of the various formalisms which are
related to chemlambda. The online varsion of this article, which will be kept
up to date, is Graph rewrites, from emergent algebras to chemlambda. See also
the site of all chemlambda projects.

######

  1. 1 Graphic lambda calculus
  2. 2 Chemlambda v1 (chemical concrete machine)
  3. 3 Chemlambda v2
  4. 4 Interaction Combinators and directed interaction combinators (dirIC)

##  1 Graphic lambda calculus

Introduced in [3] arXiv:1305.5786, graphic lambda calculus has the following
doupbel push-out (DPO [15]) graph rewrites:

Graphic lambda calculus does not have an algorithm of application of rewrites.

There is an algorithm for conversion of untyped lambda terms into glc graphs.
Here are two examples of reductions:

  1. -

reduction of the Omega combinator

  2. -

reduction of the term SKK

Graphic lambda calculus is interesting because it can also represent the
graphical version of emergent algebras via decorated tangle diagrams explained
in arXiv:1103.6007. It can therefore be used to represent and compute (reduce)
differential calculus in metric spaces endowed with dilation structures
arXiv:0810.5042.

This is done via the identification of decorated crossings as

Another, different identification of undecorated crossings, which can be done
with GLC, uses the application and abstraction nodes

Under this identifications the equivalent of the beta rewrite in terms of
tangle diagrams becomes equivalent with the ”splice” and ”loop” graph
rewrites.

Both these identifications apply for a notion of tangle diagrams which differs
from the usual one. Namely we use tangle diagrams which are not constrained to
be planar graphs. More precisely, here a tangle diagram is an oriented port
graph in the sense of Bawden [1] [2], with nodes which are 4-valent and
oriented links. The nodes can be either undecorated, i.e. of the two kinds of
oriented crossings, or decorated, i.e. the two kinds of decorated crossings
are supplementary decorated with an element of a commutative group.

The difference from usual (oriented and with decorated crossings) tangle
diagrams is that as graphs these diagrams are not supposed to be planar. By
Kuratowski theorem, a graph is planar if it does not contain certain local
patterns. In the terms of local machines explained in [5] Alife properties of
directed interaction combinators vs. chemlambda we can’t detect with a local
machine the absence of a local pattern in a graph, only the presence of it.

In joint work with L.H. Kauffman [7] arXiv:1312.4333, in section 5 Kauffman
calls for a topological version of GLC, based exclusively on manipulations of
tangle diagrams.

##  2 Chemlambda v1 (chemical concrete machine)

Introduced in [9] arXiv:1309.6914, chemlambda v1 or the chemical concrete
machine is a modification of GLC which contains only local rewrites. It also
proposes for the first time to see the graphs as molecules and the rewrites as
chemical reactions mediated by rewriting enzymes. Beyond that, chemlambda v1
still lacks a clearly defined algorithm of rewriting.

The elements of chemlambda v1 are

The graphs are called molecules and the rewrites are mediated by enzymes.

The elements of GLC as translated into chemlambda are:

The rewrites are:

With these rewrites we can do some lambda calculus reductions, by using the
same parsing of lambda calculus terms to (chemlambda) graphs as in GLC. For
example the ENELSE lambda term.

A rewrite which is introduced later in the article is DISENTANGLE (as a
pattern it appears later as SHUFFLE)

With this supplementary rewrite I can prove that the BCKW combinators system

can be implemented, in the sense that the combinators can duplicate (example
here for B)

and that all needed reductions among B,C,K,W combinators can be done (example
here for K)

This graph rewrite system has conflicting rewrites, but it has no algorithm of
application of rewrites. Especially the status of DISENTANGLE is unclear. This
will be solved in chemlambda v2.

The first published (in the usual ways) article which contains the name
”chemlambda” is joint with L.H. Kauffman, in the ALIFE-14 conference,
arXiv:1403.8046. Here Kauffman pushes again for a more topological version
Kauffman’ slides [11] which mixes tangle diagrams with chemlambda nodes, in
order to be able to do computations (in the logical sense).The emergent
algebras part of GLC, which is still present in chemlambda, used such tangle
diagrams manipulations for differential calculus computations, instead. In
these articles the emergent algebras aspects and interest are hidden.

Together with Kauffman, we experimented a lot with the power of chemlambda v1.
Our essays on paper were not going too far, due to the lack of a program which
could make much easier such exploration. In the background this was because of
a lack of a clear rewrite application algorithm.

During this period I became aware of the existence of chemlambda quines. See
as an example the tedious manipulation of reductions for the predecessor of a
Church number taken from this chorasimilarity post

Meanwhile Kauffmann started to use Mathematica for doing chemlambda v1
reductions, as described in his slides [11].

He introduced the ”Arrow” 2-valent node, which is useful for easy application
of several rewrites in the same time.

##  3 Chemlambda v2

is an artificial chemistry, in the sense that it is a purely local graph
rewrite system together with an algorithm of applications which can be done by
local machines. This notion of artificial chemistry is proposed in [17]. See
arXiv:2005.06060 for more explanations.

Motivated by the first programming essays by Kauffman for chemlambda v1, I
wrote the needed programs for chemlambda v2, in awk and html+d3.js used for
visualization. The first chemlambda project site [13] contains demonstrations
made with these programs from the github repository chemlambda-gui [14].

The latest chemlambda project site, which contains several repositories from
various contributors, is this [12]. The project received help from several
contributors, or validators, therefore now we can reduce chemlambda molecules
in haskell, python or javascript, as you can see in the more recent
demonstrations.

Chemlambda can now be compared with Interaction Combinators [5] and there is a
lambda calculus to chemlambda parser and reducer [8].

The chemlambda v2 rewrites solve the problem of the curious DISENTANGLE
rewrite by introducing two fanout nodes (FO and FOE) instead of one and by a
modified set of graph rewrites. These are the following. We use the mol
notation for clarity and we describe the rewrites by giving the LHS and RHS
patterns:

and the COMB rewrites which eliminate Arrow nodes.

The DISENTANGLE rewrite is no longer needed, because it’s effect can be
achieved by the SHUFFLE move, which is a sequence of two chemlambda v2
rewrites: FO-FOE and FI-FOE.

##  4 Interaction Combinators and directed interaction combinators (dirIC)

For completeness, here are the graph rewrites of Lafont’ Interaction
combinators [16], as they appear in the chemlambda v2 js programs, chemistry
IC in chemistry.js:

FRIN and FROUT 1-valent nodes are introduced (as in chemlambda) for the free
half-edges. We use also the Arrow 2-valent node and the COMB rewrites for
eliminating them.

All LHS patterns, of chemlambda and IC, are made of two nodes connected
through their active ports. The main difference between chemlambda v2 rewrites
and IC rewrites is that several chemlambda nodes have two active ports, while
IC nodes have only one active port.

It is possible to modify the chemlambda v2 chemistry such that there are no
conflicts. We obtain a chemistry called ”dirIC”, or directed interaction
combinators, explained in [5] Alife properties of directed interaction
combinators vs. chemlambda, over the same set of nodes as chemlambda v2. With
the chemistry dirIC, there is a translation between IC nodes of Lafont to
chemlambda nodes:

As argued here [5], the existence of conflicting rewrites in chemlambda v2,
compared with dirIC, creates much more interesting alife behaviours.

## References

  * [1] A. Bawden, Connection graphs, In: Proceedings of the 1986 ACM conference on LISP and functional programming, 258-265, ACM Press, (1986)
  * [2] A. Bawden, Implementing Distributed Systems Using Linear Naming, A.I. Technical Report No. 1627, MIT, (1993)
  * [3] M. Buliga, Graphic lambda calculus, Complex Systems 22, 4, 311-360, (2013)
  * [4] M. Buliga, Artificial chemistry experiments with chemlambda, lambda calculus, interaction combinators, arXiv:2003.14332, (2020)
  * [5] M. Buliga, Artificial life properties of directed interaction combinators vs. chemlambda, arXiv:2005.06060 (2020)
https://mbuliga.github.io/quinegraphs/ic-vs-chem.html#icvschem

  * [6] M. Buliga, Emergent rewrites in knot theory and logic (2020)
https://mbuliga.github.io/novo/presentation.html

  * [7] M. Buliga, L.H. Kauffman, GLC actors, artificial chemical connectomes, topological issues and knots, arXiv:1312.4333, (2013)
  * [8] M. Buliga, Lambda calculus to chemlambda. (2019-2020),
https://mbuliga.github.io/quinegraphs/lambda2mol.html#lambdanote

  * [9] M. Buliga, Chemical concrete machine, Figshare, (2013), arXiv:1309.6914
  * [10] M. Buliga, L.H. Kauffman, Chemlambda, universality and self-multiplication, ALIFE 2014: The Fourteenth International Conference on the Synthesis and Simulation of Living Systems July 2014, MIT Press, 490-497, (2014)
  * [11] L.H. Kauffman, Chemlambda, universality and self-multiplication, slides (2014)
  * [12] M. Buliga, Chemlambda page, https://chemlambda.github.io/index.html (2020)
  * [13] M. Buliga, Chemlambda project http://chorasimilarity.github.io/chemlambda-gui/index.html (2015-2017)
  * [14] M. Buliga, Chemlambda-gui repository, Github (2015-2017)
https://github.com/chorasimilarity/chemlambda-gui/blob/gh-
pages/dynamic/README.md

  * [15] A. Corradini, U. Montanari, F. Rossi, H. Ehrig, R. Heckel, M. Loewe, Algebraic approaches to graph transformation, part I: Basic concepts and double push-out approach, Handbook of Graph Grammars and Computing by Graph Transformation, 163-245 (1997)
  * [16] Y. Lafont, Interaction Combinators, Information and Computation 137, 1, 69-101, (1997)
  * [17] K. Tomita, H. Kurokawa, S. Murata, Graph-rewriting automata as a natural extension of cellular automata. Chapter 14 in: Adaptive Networks, Understanding Complex Systems, T. Gross, H. Sayama (eds.), NECSI Cambridge/Massachusetts (2009)


# Affective Behaviour Analysis Using Pretrained Model with Facial Prior

Yifan Li⋆ 1Key Laboratory of Intelligent Information Processing of Chinese
Academy of Sciences (CAS), Institute of Computing Technology, CAS, Beijing
100190, China. 12University of the Chinese Academy of Science, Beijing 100049,
China.
22email<EMAIL_ADDRESS>
22email<EMAIL_ADDRESS>
22email: {hanhu<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Haomiao Sun⋆ 1Key Laboratory of Intelligent
Information Processing of Chinese Academy of Sciences (CAS), Institute of
Computing Technology, CAS, Beijing 100190, China. 12University of the Chinese
Academy of Science, Beijing 100049, China.
22email<EMAIL_ADDRESS>
22email<EMAIL_ADDRESS>
22email: {hanhu<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Zhaori Liu These authors contribute equally to
this work.1Key Laboratory of Intelligent Information Processing of Chinese
Academy of Sciences (CAS), Institute of Computing Technology, CAS, Beijing
100190, China. 12University of the Chinese Academy of Science, Beijing 100049,
China.
22email<EMAIL_ADDRESS>
22email<EMAIL_ADDRESS>
22email: {hanhu<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Hu Han 1Key Laboratory of Intelligent Information
Processing of Chinese Academy of Sciences (CAS), Institute of Computing
Technology, CAS, Beijing 100190, China. 12University of the Chinese Academy of
Science, Beijing 100049, China.
22email<EMAIL_ADDRESS>
22email<EMAIL_ADDRESS>
22email: {hanhu<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Shiguang Shan 1122

###### Abstract

Affective behavior analysis has aroused researchers’ attention due to its
broad applications. However, it is labor exhaustive to obtain accurate
annotations for massive face images. Thus, we propose to utilize the prior
facial information via Masked Auto-Encoder (MAE) pretrained on unlabeled face
images. Furthermore, we combine MAE pretrained Vision Transformer (ViT) and
AffectNet pretrained CNN to perform multi-task emotion recognition. We notice
that expression and action unit (AU) scores are pure and intact features for
valence-arousal (VA) regression. As a result, we utilize AffectNet pretrained
CNN to extract expression scores concatenating with expression and AU scores
from ViT to obtain the final VA features. Moreover, we also propose a co-
training framework with two parallel MAE pretrained ViTs for expression
recognition tasks. In order to make the two views independent, we randomly
mask most patches during the training process. Then, JS divergence is
performed to make the predictions of the two views as consistent as possible.
The results on ABAW4 show that our methods are effective, and our team reached
2nd place in the multi-task learning (MTL) challenge and 4th place in the
learning from synthetic data (LSD) challenge. Code is available
111https://github.com/JackYFL/EMMA_CoTEX_ABAW4.

###### Keywords:

Multi-task affective behaviour analysis, AU recognition, expression
recognition, VA regression, facial prior, MAE, ABAW4.

##  1 Introduction

Affective behaviour analysis such as facial expression recognition (EXPR)
[21], action unit (AU) recognition [24] and valence arousal (VA) regression
[23], raised much attention due to its wide application scenarios. With the
superior performance of deep learning, the traditional artificial designed
emotional representations are gradually replaced by the deep neural networks
(DNNs) extracted ones. While DNNs-based methods perform well in affective
behaviour analysis tasks, the limited size of the existing emotion benchmark
has hindered the recognition performance and generalization ability.

Although it costs much to obtain data with accurate emotion annotations, we
can acquire unlabeled face images more easily. To capitalize on the massive
unlabeled face images, we propose to learn facial prior knowledge using self-
supervised learning methods. In computer vision field, general self-supervised
learning methods can be summarized into two major directions, contrastive
learning based methods, e.g., MoCo [6], SimCLR [2], and generative methods,
e.g., BeiT [1], Masked Auto-Encoder (MAE) [5]. SSL-based methods have reached
great success, achieving even better performance compared with supervised
pretrained ones in some downstream tasks. Inspired by MAE, which randomly
masks image patches and utilizes vision transformer [4] (ViT) to reconstruct
pixels and learn the intrinsic relationships and discriminant representations
of patches, we propose to learn facial prior using MAE on massive unlabeled
face images. We suppose MAE could model the relationship of different face
components, which contains the prior about face structure and has a better
parameter initialization compared with the ImageNet pretrained model.

Based on MAE pretrained ViT, we propose a simple but effective framework
called Emotion Multi-Model Aggregation (EMMA, see Fig. 1) method for multi-
task affective behaviour analysis, i.e., EXPR, AU recognition and VA
regression. According to experiment results, we find that it’s easier to
overfit for VA regression task when finetuning MAE pretrained ViT for all
three tasks. As a result, we propose to use an extra convolutional neural
network (CNN) to extract features for VA regression tasks. We found that
expression scores could provide pure and intact features for VA regression.
Thus we choose to use a CNN (DAN [32]) pretrained on an EXPR benchmark
AffectNet that contains expression prior knowledge to extract VA features.
Furthermore, we also utilize the scores of both EXPR and AU recognition
extracted by ViT to aid VA regression. Note that we only finetune the linear
layer for VA regression to prevent overfitting.

Moreover, we also propose a masked Co-Training method for EXpression
recognition (masked CoTEX, see Fig. 2), in which we use MAE pretrained ViT as
the backbones of two views. To form two information-complementary views, we
randomly mask most patches of the face images. Inspired by MLCT [33] and MLCR
[24], we also use Jenson-Shannon (JS) divergence to constrain the predictions
from both views to be consistent to the average distribution of two
predictions. Different from MLCT or MLCR, we apply JS divergence to single-
label classification (e.g., expression) by modifying the entropy, and this
method is prepared for supervised learning.

The contributions of this paper can be summarized into the following aspects.

  * •

We use face images to pretrain MAE, which shows better performance than
ImageNet pretrained MAE for affective behaviour analysis related tasks.

  * •

We design EMMA, which uses ViT (pretrained on unlabeled face images) and CNN
(pretrained on expression benchmark) to extract features for multi-task
affective behaviour analysis. We also propose to use the concatenation of
expression scores extracted by CNN and ViT, and the AU scores extracted by ViT
to finetune the linear layer for VA regression.

  * •

We propose masked CoTEX for EXPR, which is a co-training framework utilizing
JS divergence on two random masked views to make them consistent. We find that
a large mask ratio could not only improve training speed and decrease memory
usage, but also increase accuracy. Furthermore, we also find that the accuracy
could be improved further by increasing the batch size.

##  2 Related Work

In this section, we first review studies on multi-task affective behaviour
analysis and then introduce related work on static image-based EXPR.

Multi-task Affective Behaviour Analysis. The MTL affective behaviour analysis
task refers to the simultaneous analysis of EXPR, AU recognition, and VA
regression, etc. Unlike the experimental setup in ABAW and ABAW2, where three
different tasks were completed independently, ABAW3 presents an integrated
metric and evaluates the performance of all three tasks simultaneously. Deng
[3] employs psychological prior knowledge for multi-task estimation, which
uses local features for AU recognition and merges the messages of different
regions for EXPR and VA. Jeong et al. [8] apply the knowledge distillation
technique for a better generalization performance and the domain adaptation
techniques to improve accuracy in target domains. Savchenko et al. [11] used a
lightweight EfficientNet model to develop a real-time framework and improve
performance by pre-training based on additional data. Unlike previous methods
for MTL, we utilize the facial prior information in both unlabeled face images
and expression benchmark to improve the performance.

Expression Recognition. The aim of expression recognition is to recognize
basic human expression categories. In the ABAW3 competition, some researchers
utilize multi-modal information such as audio and text to improve the model
performance and achieve a better ranking [37, 11]. However, these methods have
higher requirements for data collection. Therefore, it is worth exploring how
to use static images for a more generalized usage scenario. Jeong et al. [9]
use an affinity loss approach, which uses affinity loss to train a feature
extractor for images. In addition, they propose a multi-head attention network
in a coordinated manner to extract diverse attention for EXPR. Xue et al. [34]
propose a two-stage CFC network that separates negative and positive
expressions, and then distinguishes between similar ones. Phan et al. [25] use
a pretrained model RegNet [27] as a backbone and introduce the Transformer for
better modelling the temporal information. Different from the previous
methods, our masked CoTEX uses a masked co-training framework which consists
of two ViTs pretrained by MAE on unlabeled face images to achieve better
performance.

##  3 Proposed Method

In this section, we first formulate the problem, then introduce EMMA for
multi-task affective behaviour analysis and masked CoTEX for EXPR.

###  3.1 Formulation

Let $\mathcal{X}=\\{x_{i}\in\mathbb{R}^{C\times H\times W},i=1,2,...,N\\}$ and
$\mathcal{Y}$ denotes face images and according labels, respectively. For
multi-task affective behaviour analysis, the labels
$\mathcal{Y}=\\{y^{i}_{MTL},i=1,2,...,N\\}$ consists of three sub-task labels,
i.e.,

$y^{i}_{MTL}=\left[y^{i}_{VA}\in\mathbb{R}^{2},y^{i}_{EXP}\in\mathbb{Z},y^{i}_{AU}\in\mathbb{Z}^{12}\right],$

where $y^{i}_{VA},y^{i}_{EXP},y^{i}_{AU}$ indicate VA labels, EXP labels and
AU labels, respectively. $y^{i}_{VA}$ is a two dimension vector representing
valence and arousal in the range of $[-1,1]$. $y^{i}_{EXP}$ is an integer
ranging from 0 to 7 representing one of eight expression categories, i.e.,
neutral, anger, disgust, fear, happiness, sadness, surprise, and other.
$y^{i}_{AU}$ includes 12 AU labels, i.e., AU1, AU2, AU4, AU6, AU7, AU10, AU12,
AU15, AU23, AU24, AU25 and AU26. If a face is invisible due to large pose or
occlusion, the values of $y^{i}_{VA}$, $y^{i}_{EXP}$ and $y^{i}_{AU}$ can be
-5, -1 and 0, respectively. For EXPR task,
$\mathcal{Y}=\\{y^{i}_{EXP}\in\mathbb{Z},i=1,2,...,N\\}$. There are only six
expression categories in ABAW4[12] synthetic expression dataset, i.e., anger,
disgust, fear, happiness, sadness, and surprise.

Figure 1: EMMA framework for multi-task affective behaviour analysis.

###  3.2 EMMA

EMMA shown in Fig. 1 is a two-branch architecture, with one (MAE pretrained
ViT) for AU recognition and EXPR tasks, and the other (AffectNet pretrained
DAN) for VA recognition task. Assume MAE pretrained ViT and AffectNet
pretrained DAN can be denoted as $f_{ViT}$ and $f_{CNN}$, respectively. Given
an image $x_{i}$, we can obtain the AU and EXP prediction score
$f_{ViT}(x_{i})=\left[{p_{AU}(x_{i}),p_{EXP}^{1}(x_{i})}\right]$, and another
EXP prediction score $p_{EXP}^{2}(x_{i})=f_{CNN}(x_{i})$. Then the VA feature
can be obtained by concatenating three scores. Finally, we can obtain the VA
score $p_{VA}(x_{i})$ by passing the VA feature through a two-layer fully
connected layer. Since ViT and CNN are both easy to get overfitting when
finetuning the overall network for VA regression task, we stop the gradient of
$p_{VA}$ and only finetune the linear layer. The optimization objective
$\mathcal{L}$ can be expressed as:

$\mathcal{L}=\mathcal{L}_{AU}+\mathcal{L}_{VA}+\mathcal{L}_{EXP},$ (1)

where $\mathcal{L}_{AU}$, $\mathcal{L}_{VA}$, $\mathcal{L}_{EXP}$ indicate
losses for AU recognition, VA regression, and EXPR recognition, respectively.

For the AU recognition task, the training loss $\mathcal{L}_{AU}$ is the
binary cross-entropy loss, which is given by:

$\mathcal{L}_{AU}(p_{AU}(x_{i}),y_{AU}^{i})=-\frac{1}{L}\sum\limits_{j=0}^{L-1}\left[{y^{ij}_{AU}\log\sigma(p_{AU}^{j}({x_{i}}))+(1-y^{ij}_{AU})\log(1-\sigma(p_{AU}^{j}({x_{i}})))}\right],$
(2)

where L is the number of AUs, and $\sigma$ denotes sigmoid function:
$\sigma(x)=\frac{1}{{1+{e^{-x}}}}$.

For EXPR task, the training loss $\mathcal{L}_{EXP}$ is the cross-entropy
loss:

$\mathcal{L}_{EXP}=-\log{\rho^{y^{i}_{EXP}}},$ (3)

where $\rho^{y^{i}_{EXP}}$ is the softmax probability of the prediction score
$p_{EXP}(x_{i})$ indexed by the expression label $y^{i}_{EXP}$.

For VA regression task, we regard Concordance Correlation Coefficient (CCC,
see Eq. (9)) loss as training loss $\mathcal{L}_{VA}$:

$\mathcal{L}_{VA}=1-(CCC^{A}+CCC^{V}),$ (4)

where $CCC^{A}$ and $CCC^{V}$ are the $CCC$ of arousal and valence,
respectively.

Figure 2: Masked CoTEX framework for EXPR.

###  3.3 Masked CoTEX

Masked CoTEX (Fig. 2) is designed for EXPR, which is a co-training framework
with MAE pretrained ViT in each view. In order to make two views as
independent as possible, we randomly mask most patches during training process
to form two approximately information-complementary views. Masking random
patches in face images means we drop a few patches and put the rest patches
into the ViT. Given a face image $x_{i}$, we can obtain two expression scores
$p_{1}(x_{i})$ and $p_{2}(x_{i})$ by two parallel ViTs. JS divergence
$\mathcal{L}_{JS}$ is performed to make the predictions of two views
consistent. We use the average predictions from two views during inference
phase to further improve the performance. The overall optimization objective
is:

$\mathcal{L}=\lambda\mathcal{L}_{JS}+\mathcal{H}_{1}+\mathcal{H}_{2},$ (5)

$\lambda$ is the hyper-parameter to balance the influence of
$\mathcal{L}_{JS}$. $\mathcal{L}$ consists of two components, JS divergence
$\mathcal{L}_{JS}$ and cross-entropy loss $\mathcal{H}_{t},t=\\{1,2\\}$.

The JS divergence $\mathcal{L}_{JS}$ that constrains the predictions of two
views can be expressed as:

${{\cal
L}_{JS}}({\rho_{1}},{\rho_{2}})=H(m)-\frac{{H({\rho_{1}})+H({\rho_{2}})}}{2}=\frac{{{D_{KL}}(\left.{{\rho_{1}}}\right\|m)+{D_{KL}}(\left.{{\rho_{2}}}\right\|m)}}{2},$
(6)

where $\rho_{1}$ and $\rho_{2}$ indicate the softmax probabilities of
prediction scores, $m$ is the average of $\rho_{1}$ and $\rho_{2}$, and $H$ is
the entropy of the probability. JS divergence can also be denoted by the
average KL divergence which constrains two distributions similar to the
average distribution.

In each view, we use cross-entropy loss $\mathcal{H}$ to constrain:

$\mathcal{H}_{t}(\rho_{t},y_{i})=-log\rho^{y_{i}}_{t},t\in\\{1,2\\},$ (7)

where $y_{i}$ is the expression label of face image $x_{i}$.

##  4 Experiment

###  4.1 Benchmarks and Evaluation Metrics

The benchmark we used is provided by ABAW4 challenge [12], which includes two
sub-challenges, i.e., the multi-task learning (MTL) challenge and the learning
from synthetic data (LSD) challenge. All the face images are aligned and
cropped as 112$\times$112.

For MTL challenge, static version of Aff-Wild2 [12, 16, 20, 14, 19, 18, 17,
13, 35, 15] are utilized, which contains selected-specific frames from Aff-
Wild2. There are in total of around 220K images including training set (142K),
validation set (27K) and test set (51K). MTL challenge includes three sub-
tasks, i.e., VA regression, EXPR (including 8 categories) and AU recognition
(including 12 AUs). The MTL evaluation metrics $P_{MTL}$ consists of three
parts:

$P_{MTL}=\frac{1}{2}(CC{C_{V}}+CC{C_{A}})+\frac{1}{8}\sum\limits_{i=0}^{7}{F1_{EXPR}^{i}}+\frac{1}{{12}}\sum\limits_{i=0}^{11}{F1_{AU}^{i}},$
(8)

where $CCC$ and $F1$ are expressed as:

$CCC(\bm{x},\bm{y})=\frac{{2\rho{\sigma_{\bm{x}}}{\sigma_{\bm{y}}}}}{{\sigma^{2}_{\bm{x}}+\sigma^{2}_{\bm{y}}+{{({\mu_{\bm{x}}}-{\mu_{\bm{y}}})}^{2}}}},\;F1=2\cdot\frac{{precision\cdot
recall}}{{precision+recall}},$ (9)

where $\rho$ is the Pearson Coefficient, $\sigma_{\bm{x}}$ is the standard
deviation of vector $\bm{x}$, and $\mu_{\bm{x}}$ is the mean value of the
vector $\bm{x}$.

For the LSD challenge, the synthetic face images for the EXPR task (in a total
of six basic expressions) are generated from some specific frames in the
AffWild2. There are around 2770K synthetic face images for training, and 4.6K
and 106K real face images in validation and testing set, respectively. The LSD
evaluation metrics $P_{LSD}$ can be expressed as:

${P_{LSD}}=\frac{1}{6}\sum\limits_{i=0}^{5}{F1_{LSD}^{i}},$ (10)

which is the average of six expression F1 scores.

###  4.2 Training Details

All the data we used are the aligned and cropped version (112$\times$112)
provided by ABAW4. We resize the face images to 232$\times$232 (228$\times$228
for LSD task) and perform the random cropping to obtain the image size of
224$\times$224\. Furthermore, we also apply random color jitter and horizontal
flipping as data augmentation during the training process. We use ViT-base as
the backbone for both two methods, which is pretrained by MAE [5] on face
image benchmark CelebA [22]. We use AffectNet pretrained DAN [32] in EMMA. The
overall hyper-parameter settings follow common practice of supervised ViT
training, which is shown in Table 3.

Table 1: The F1 scores (in $\%$) comparison for multi-task affective behaviour analysis on ABAW4 validation set. All the results are recorded for the best final score. Method | AU F1 | VA CCC | EXPR F1 | Final score
---|---|---|---|---
Baseline (VGGface, linear evaluation) | - | - | - | 30
Clip (RN-50, finetune) | 30.36 | 22.60 | 18.86 | 71.82
ResNet50 (ImageNet pretrained, finetune) | 48.85 | 33.55 | 22.00 | 104.40
EmotionNet (ABAW3 SOTA, finetune) | 49.53 | 35.41 | 18.27 | 103.21
InceptionV3 (ImageNet pretrained, finetune) | 47.53 | 37.85 | 18.66 | 104.46
EfficientNet-b2 (AffectNet pretrained, finetune) | 50.60 | 41.74 | 28.19 | 120.53
ViT-base (MAE pretrained on CelebA, finetune) | 50.41 | 41.21 | 31.68 | 123.29
EMMA (MAE pretrained on CelebA, finetune) | 50.54 | 45.88 | 30.28 | 126.71
Table 2: The F1 score (in %) and macro accuracy (acc, in %) for LSD task on ABAW4 validation set. Method | Acc | F1
---|---|---
Baseline (ResNet-50, ImageNet pretrained) | 67.58 | 59.71
EfficientNet-b2 (ImageNet pretrained) | 67.71 | 63.33
EfficientNet-b2 (AffectNet pretrained) | 73.06 | 63.83
Clip (ViT-base) | 70.84 | 62.13
ViT-base (MAE pretrained on CelebA) | 71.95 | 64.24
CoTEX (mask ratio 0%, batch size 64) | 72.84 | 64.67
Masked CoTEX (mask ratio 75%, batch size 1024) | 75.77 | 70.68
Table 3: The training settings for EMMA and Masked CoTEX. Config | EMMA value | Masked CoTEX value
---|---|---
optimizer | AdamW | AdamW
base learning rate | 5e-4 | 5e-4
weight decay | 0.05 | 0.15
batch size | 100 | 1024
clip grad | 0.05 | 0.05
layer decay | 0.65 | 0.65
warm up epochs | 5 | 5
total epochs | 30 | 6
accumulated iterations | 4 | 4
drop path rate | 0.1 | 0.1
Table 4: The MTL and LSD results of top-5 teams on ABAW4 test set.

Team | Best Overall Score
---|---
Situ-RUCAIM3 | 143.61
Ours | 119.45
HSE-NN | 112.99
CNU Sclab | 111.35
STAR-2022 | 108.55
(a) MTL

Team | F1
---|---
HSE-NN | 37.18
PPAA | 36.51
IXLAB | 35.87
Ours | 34.83
HUST-ANT | 34.83
(b) LSD

Table 5: The ablation study for EMMA. All the models are pretrained with MAE. Method | AU F1 | VA CCC | EXPR F1 | Final score
---|---|---|---|---
ViT (ImageNet) | 47.67 | 32.61 | 24.65 | 104.92
ViT (CelebA) | 50.41 | 41.21 | 31.68 | 123.29
EMMA (ImageNet) | 45.38 | 42.57 | 27.02 | 114.96
EMMA (EmotioNet) | 49.00 | 47.11 | 22.51 | 118.62
EMMA (CelebA) | 50.54 | 45.88 | 30.28 | 126.71
EMMA (CelebA, $\mathcal{L}_{EXP}(p_{EXP}^{1}+p_{EXP}^{2})$) | 50.56 | 42.90 | 28.74 | 122.20
EMMA (different epochs ensemble) | 51.56 | 46.59 | 32.29 | 130.44
EMMA (different parameters ensemble) | 52.45 | 47.10 | 34.17 | 133.68
Table 6: The ablation study for Masked CoTEX. Method | Acc | F1
---|---|---
ViT-base (MAE pretrained on CelebA) | 71.95 | 64.24
CoTEX (mask ratio 0%, batch size 64) | 72.81 | 64.67
Masked CoTEX (mask ratio 50%, batch size 128) | 72.53 | 64.98
Masked CoTEX (mask ratio 75%, batch size 64) | 74.17 | 67.67
Masked CoTEX (mask ratio 75%, batch size 256) | 74.82 | 68.99
Masked CoTEX (mask ratio 75%, batch size 1024) | 75.77 | 70.68
Masked CoTEX (mask ratio 85%, batch size 1536) | 78.54 | 73.75
Masked CoTEX (mask ratio 90%, batch size 2560) | 81.18 | 78.62
Masked CoTEX (mask ratio 95%, batch size 4096) | 83.85 | 82.33

###  4.3 Recognition Results

The results for MTL and LSD task on ABAW4 validation set are shown in Table 1
and Table 2, respectively.

For MTL task, we compare EMMA with baseline (VGG-face) provided by ABAW4, Clip
[26] (RN-50), ResNet-50 [7], EmotionNet [3] (SOTA method in ABAW3 MTL task),
InceptionV3 [29], EfficientNet-b2 [30] and ViT-base [4]. We can see from Table
1 that EMMA achieves the best or competitive performance compared with other
methods especially for VA task. EMMA utilizes AffectNet pretrained CNN to
extract EXP score, and combines the AU and EXP score from MAE pretrained ViT,
which could provide pure and intact features for VA regression. Moreover, we
only finetune the linear layer for VA regression which is easy to overfit when
finetuning the entire network. Furthermore, the MAE pretrained ViT also
contributes to the improvement of the final result, since it could provide
more facial structure information which is a better initialization weight for
optimization.

For EXPR task, we compare EMMA with baseline (ResNet-50 pretrained on
ImageNet), EfficientNet-b2 [30] (pretrained on AffectNet), Clip [26] (ViT-
base), ViT-base (MAE pretrained on CelebA) and CoTEX. From Table 2 we can see
that masked CoTEX outperforms all the other methods. It is worth noting that a
large mask ratio and batch size is beneficial for improving the performance.

Table 4 shows the best test results of each team in ABAW4. Our team reached
2nd place in the MTL challenge. The Situ-RUCAIM3 team [36] outperforms our
proposed method due to the utilization of temporal information during the
testing phase and more pre-trained models. Meanwhile, our method achieved
fourth place out of ten teams in the LSD task. HSE-NN [28] uses a model pre-
trained with multi-task labels in an external training set, and IXLAB [10]
applies a bagging strategy to obtain better performance.

###  4.4 Ablation Study

We also perform an ablation study on ABAW4 validation set to investigate the
effectiveness of each component, which is shown in Table 5 for EMMA and Table
6 for masked CoTEX, respectively.

For EMMA, we can see that face images pretrained MAE has a better performance
than ImageNet pretrained one, which indicates that face images may contain
facial structure information that is beneficial for affective behaviour
analysis. Furthermore, the AffectNet pretrained CNN could provide facial prior
knowledge to improve VA regression performance. Moreover, we also explored the
influence of face image dataset. We perform experiments on EmotioNet [31],
which includes around 1000,000 face images. However, the performance drops
when using EmotioNet pretraind model compared with the CelebA pretrained one.
We think this may be caused by the quality of face images, such as the
resolution, the image noise in face images, etc. We also consider adding
$p_{EXP}^{2}$ for expression task, while the performance has dropped about 3%,
which is probably caused by the noise in expression logits. In the end, we
also attempt the ensemble technique to further improve the final performance,
and the results show that this technique is useful.

For masked CoTEX, the results show that a large mask ratio contributes to the
improvement of the performance. Since a large mask ratio can reduce memory
consumption, we can use a larger batch size. We notice that with the increase
of the batch size, the performance is improved accordingly. However, since the
training set is generated from the validation set, this improvement may not
reflect the actual performance on test set. In all our submissions for ABAW4
LSD challenge, CoTEX with 75% mask ratio ($\lambda$=0, the 4th submission)
achieved the highest F1 score which is 4th place in this challenge. But with
95% mask ratio (the 5th submission), the F1 score is even lower than the one
without masking(the 2nd submission). We suppose an excessively large mask
ratio may cause overfitting.

##  5 Conclusions

In this paper, we propose two approaches using pretrained models with facial
prior, namely EMMA and masked CoTEX, for the ABAW4 MTL and LSD challenges,
respectively. We find that the ViT pretrained by MAE using face images
performs better on emotion related tasks compared with the ImageNet pretrained
one. Moreover, we notice that the expression score is a pure and intact
feature for VA regression, which is prone to get overfitting when finetuning
the entire network. Furthermore, we propose a co-training framework, in which
two views are generated by randomly masking most patches. According to our
experiment results, we find that increasing mask ratio and batch size is
beneficial to improve the performance on the LSD validation set. In the
future, we can also attempt pretraining MAE on different benchmarks.

## References

  * [1] Bao, H., Dong, L., Wei, F.: Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254 (2021)
  * [2] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: Proc. ICML. pp. 1597–1607 (2020)
  * [3] Deng, D.: Multiple emotion descriptors estimation at the abaw3 challenge (2022)
  * [4] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [5] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proc. CVPR. pp. 16000–16009 (2022)
  * [6] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proc. CVPR. pp. 9729–9738 (2020)
  * [7] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proc. CVPR. pp. 770–778 (2016)
  * [8] Jeong, E., Oh, G., Lim, S.: Multitask emotion recognition model with knowledge distillation and task discriminator. arXiv preprint arXiv:2203.13072 (2022)
  * [9] Jeong, J.Y., Hong, Y.G., Kim, D., Jung, Y., Jeong, J.W.: Facial expression recognition based on multi-head cross attention network. arXiv preprint arXiv:2203.13235 (2022)
  * [10] Jeong, J.Y., Hong, Y.G., Oh, J., Hong, S., Jeong, J.W., Jung, Y.: Learning from synthetic data: Facial expression classification based on ensemble of multi-task networks. arXiv preprint arXiv:2207.10025 (2022)
  * [11] Kim, J.H., Kim, N., Won, C.S.: Facial expression recognition with swin transformer. arXiv preprint arXiv:2203.13472 (2022)
  * [12] Kollias, D.: Abaw: Valence-arousal estimation, expression recognition, action unit detection & multi-task learning challenges. In: Proc. CVPR. pp. 2328–2336 (2022)
  * [13] Kollias, D., Cheng, S., Pantic, M., Zafeiriou, S.: Photorealistic facial synthesis in the dimensional affect space. In: Proc. ECCVW. pp. 0–0 (2018)
  * [14] Kollias, D., Cheng, S., Ververas, E., Kotsia, I., Zafeiriou, S.: Deep neural network augmentation: Generating faces for affect analysis. IJCV 128(5), 1455–1484 (2020)
  * [15] Kollias, D., Nicolaou, M.A., Kotsia, I., Zhao, G., Zafeiriou, S.: Recognition of affect in the wild using deep neural networks. In: Proc. CVPRW. pp. 1972–1979. IEEE (2017)
  * [16] Kollias, D., Sharmanska, V., Zafeiriou, S.: Distribution matching for heterogeneous multi-task learning: a large-scale face study. arXiv preprint arXiv:2105.03790 (2021)
  * [17] Kollias, D., Tzirakis, P., Nicolaou, M.A., Papaioannou, A., Zhao, G., Schuller, B., Kotsia, I., Zafeiriou, S.: Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architectures, and beyond. IJCV pp. 1–23 (2019)
  * [18] Kollias, D., Zafeiriou, S.: Expression, affect, action unit recognition: Aff-wild2, multi-task learning and arcface. arXiv preprint arXiv:1910.04855 (2019)
  * [19] Kollias, D., Zafeiriou, S.: Va-stargan: Continuous affect generation. In: Int. Conf. Adv. Conc. Intel. Vis. Sys. pp. 227–238. Springer (2020)
  * [20] Kollias, D., Zafeiriou, S.: Affect analysis in-the-wild: Valence-arousal, expressions, action units and a unified framework. arXiv preprint arXiv:2103.15792 (2021)
  * [21] Li, S., Deng, W.: Deep facial expression recognition: A survey. IEEE TAC (2020)
  * [22] Liu, Z., Luo, P., Wang, X., Tang, X.: Large-scale celebfaces attributes (celeba) dataset. Retrieved August 15(2018), 11 (2018)
  * [23] Nicolaou, M.A., Gunes, H., Pantic, M.: Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space. IEEE TAC 2(2), 92–105 (2011)
  * [24] Niu, X., Han, H., Shan, S., Chen, X.: Multi-label co-regularization for semi-supervised facial action unit recognition. In: Proc. NeurIPS. pp. 909–919 (2019)
  * [25] Phan, K.N., Nguyen, H.H., Huynh, V.T., Kim, S.H.: Expression classification using concatenation of deep neural network for the 3rd abaw3 competition. arXiv preprint arXiv:2203.12899 (2022)
  * [26] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: Proc. ICML. pp. 8748–8763. PMLR (2021)
  * [27] Radosavovic, I., Kosaraju, R.P., Girshick, R., He, K., Dollár, P.: Designing network design spaces. In: Proc. CVPR. pp. 10428–10436 (2020)
  * [28] Savchenko, A.V.: Hse-nn team at the 4th abaw competition: Multi-task emotion recognition and learning from synthetic images. arXiv preprint arXiv:2207.09508 (2022)
  * [29] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proc. CVPR. pp. 2818–2826 (2016)
  * [30] Tan, M., Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: Proc. ICML. pp. 6105–6114. PMLR (2019)
  * [31] Wang, P., Wang, Z., Ji, Z., Liu, X., Yang, S., Wu, Z.: Tal emotionet challenge 2020 rethinking the model chosen problem in multi-task learning. In: Proc. CVPRW. pp. 412–413 (2020)
  * [32] Wen, Z., Lin, W., Wang, T., Xu, G.: Distract your attention: multi-head cross attention network for facial expression recognition. arXiv preprint arXiv:2109.07270 (2021)
  * [33] Xing, Y., Yu, G., Domeniconi, C., Wang, J., Zhang, Z.: Multi-label co-training. In: Proc. IJCAI. pp. 2882–2888 (2018)
  * [34] Xue, F., Tan, Z., Zhu, Y., Ma, Z., Guo, G.: Coarse-to-fine cascaded networks with smooth predicting for video facial expression recognition. In: Proc. CVPR. pp. 2412–2418 (2022)
  * [35] Zafeiriou, S., Kollias, D., Nicolaou, M.A., Papaioannou, A., Zhao, G., Kotsia, I.: Aff-wild: Valence and arousal ‘in-the-wild’challenge. In: Proc. CVPRW. pp. 1980–1987. IEEE (2017)
  * [36] Zhang, T., Liu, C., Liu, X., Liu, Y., Meng, L., Sun, L., Jiang, W., Zhang, F.: Emotion recognition based on multi-task learning framework in the abaw4 challenge. arXiv preprint arXiv:2207.09373 (2022)
  * [37] Zhang, W., Qiu, F., Wang, S., Zeng, H., Zhang, Z., An, R., Ma, B., Ding, Y.: Transformer-based multimodal information fusion for facial expression analysis. In: Proc. CVPRW. pp. 2428–2437 (2022)