# Commonsense Knowledge Base Completion with Structural and Semantic Context

Chaitanya Malaviya♢, Chandra Bhagavatula♢, Antoine Bosselut♢♣, Yejin Choi♢♣
♢Allen Institute for Artificial Intelligence
♣University of Washington
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Automatic KB completion for _commonsense_ knowledge graphs (e.g., ATOMIC and
ConceptNet) poses unique challenges compared to the much studied conventional
knowledge bases (e.g., Freebase). Commonsense knowledge graphs use free-form
text to represent nodes, resulting in orders of magnitude more nodes compared
to conventional KBs ( $\sim$18x more nodes in ATOMIC compared to Freebase
(FB15K-237)). Importantly, this implies significantly sparser graph structures
— a major challenge for existing KB completion methods that assume densely
connected graphs over a relatively smaller set of nodes.

In this paper, we present novel KB completion models that can address these
challenges by exploiting the structural and semantic context of nodes.
Specifically, we investigate two key ideas: (1) learning from local _graph
structure_ , using graph convolutional networks and automatic graph
densification and (2) _transfer learning_ from pre-trained language models to
knowledge graphs for enhanced contextual representation of knowledge. We
describe our method to incorporate information from both these sources in a
joint model and provide the first empirical results for KB completion on
ATOMIC and evaluation with ranking metrics on ConceptNet. Our results
demonstrate the effectiveness of language model representations in boosting
link prediction performance and the advantages of learning from local graph
structure (+1.5 points in MRR for ConceptNet) when training on subgraphs for
computational efficiency. Further analysis on model predictions shines light
on the types of commonsense knowledge that language models capture
well.111Code and dataset are available at github.com/allenai/ commonsense-kg-
completion.

##  1 Introduction and Motivation

Figure 1: Subgraph from ConceptNet illustrating semantic diversity of nodes.
Dashed blue lines represent potential edges to be added to the graph.

While there has been a substantial amount of work on KB completion for
conventional knowledge bases such as Freebase, relatively little work exists
for KB completion for _commonsense_ knowledge graphs such as ATOMIC (?) and
ConceptNet (?). The distinct goals of this paper are to identify unique
challenges in commonsense KB completion, investigate effective methods to
address these challenges, and provide comprehensive empirical insights and
analysis.

The key challenge in completing commonsense KGs is the _scale_ and _sparsity_
of the graphs. Unlike conventional KBs, commonsense KGs consist of nodes that
are represented by _non-canonicalized_ , _free-form_ text, as shown in Figure
1. For example, the nodes ”prevent tooth decay” and ”tooth decay” are
conceptually related, but _not_ equivalent, thus represented as distinct
nodes. This conceptual diversity and expressiveness of the graph, imperative
for representing commonsense, implies that the number of nodes is orders of
magnitude larger, and graphs are substantially sparser than conventional KBs.
For instance, an encyclopedic KB like FB15K-237 (?) has 100x the density of
ConceptNet and ATOMIC (node in-degrees visualized in Figure 2).

In this work, we provide empirical insights on how the sparsity of commonsense
KGs poses a challenge to existing KB completion models that implicitly assume
densely connected graphs. Figure 3 provides a brief preview of this evidence,
where the performance of ConvTransE (?), a high performing KB completion
model, degrades quickly as we reduce the graph density of FB15K-237.

This motivates a strong need for investigating novel approaches to KB
completion for commonsense KGs. We posit that new methods need to better
accommodate the implicit conceptual connectivity across all nodes — both
_structural_ and _semantic_ – beyond what is explicitly encoded in existing
commonsense KBs. Specifically, we investigate two key ideas: (1) learning from
local _graph structure_ , using graph convolutional networks and automatic
graph densification. and (2) _transfer learning_ from language models to
knowledge graphs to improve _contextual_ representation of nodes.

To integrate graph structure information, we present an approach based on
graph convolutional networks (GCN) (?) to contextualize a node’s
representation based on its local neighborhood. For transfer learning, we
present effective approaches to fine-tune pre-trained language models (?) to
commonsense KGs, essentially achieving transfer learning from _language_ to
_knowledge_. Our work shares the high-level spirit of recent work from ? (?)
that demonstrates the use of pre-trained LMs for reconstructing KB entries,
but we provide a more focused study specifically for commonsense KGs.
Empirical analysis leads to the observation that GCNs, although effective on
various densely connected graphs (?), are not as effective on commonsense KGs
out of the box, as sparse connections do not allow effective knowledge
propagation. Hence, we propose an approach for automatic graph densification
based on semantic similarity scores between nodes. Finally, we highlight
strategies necessary to train models using information from both the graph
structure and language models.

Our main contributions are highlighted below:

  1. 1.

Empirical insights about the unique challenges of commonsense KB completion
compared to conventional encyclopedic KB completion.

  2. 2.

Novel KB completion methods to model the implicit structural and semantic
context of knowledge beyond what is explicitly available in existing KGs.

  3. 3.

The _first_ empirical results on ATOMIC for KB completion and evaluation with
ranking metrics on ConceptNet.

  4. 4.

Analysis and insights on types of commonsense knowledge captured well by
language models.

In sum, our findings indicate that transfer learning is generally more
effective than learning from graph structure. Moreoever, we find that graph
structure can indeed provide complementary information which can boost
performance, especially when training with subgraphs for efficiency.

Figure 2: Heatmap showing the percentage of nodes with in-degree belonging to
the specified bins on the x-axis. The plot illustrates the sparseness of
commonsense KGs relative to a standard KB completion benchmark (FB15K-237).
Figure 3: Trend of decreasing KB Completion Scores with different values of
graph density (in log scale) for the FB15K-237 dataset with the ConvTransE
model.

##  2 Knowledge Graphs

There have been several efforts in building graph-structured representations
of commonsense (?; ?; ?; ?). We focus our experiments on two prominent
knowledge graphs: ConceptNet and ATOMIC. Statistics for both graphs are
provided in Table 1, along with FB15K-237 – a standard KB completion dataset.

Dataset | # Nodes | # Edges | # Relations | Density | Average In-Degree
---|---|---|---|---|---
ConceptNet-100K | 78088 | 100000 | 34 | 1.6e-5 | 1.25
Atomic | 256570 | 610536 | 9 | 9.0e-6 | 2.25
FB15K-237 (for scale) | 14505 | 272115 | 237 | 1.2e-3 | 16.98
Table 1:  Knowledge Graph Statistics (Training Set Only). Graph density is
calculated as $D={\frac{V}{N(N-1)}}$, where $N$ is the number of nodes and $V$
is the number of edges in the graph.

##### ConceptNet-100K222https://ttic.uchicago.edu/~kgimpel/commonsense.html:

CN-100K contains general commonsense facts about the world. This version (?)
contains the Open Mind Common Sense (OMCS) entries from ConceptNet (?). The
nodes in this graph contain 2.85 words on average. We used the original splits
from the dataset, and combined the two provided development sets to create a
larger development set. The development and test sets consisted of 1200 tuples
each.

##### ATOMIC333https://homes.cs.washington.edu/~msap/atomic/:

The ATOMIC knowledge graph contains social commonsense knowledge about day-to-
day events. The dataset specifies effects, needs, intents and attributes of
the actors in an event. The average phrase length of nodes (4.40 words) is
slightly higher than that of CN-100K. Multiple targets may exist for a source
entity and relation. The tuples in this graph may also contain a none target
in the case that the relation type does not necessitate an annotation. The
original dataset split was created to make the set of seed entities between
the training and evaluation splits mutually exclusive. Since the KB completion
task requires entities to be seen at least once, we create a new random
80-10-10 split for the dataset. The development and test sets consisted of 87K
tuples each.

##  3 Machine Commonsense Completion

We investigate two key ideas for performing completion of commonsense KGs – 1)
transfer learning from language to knowledge graphs and 2) learning from graph
structure. To address the challenge of sparsity of commonsense KGs, we enrich
the graph connectivity with synthetic semantic similarity links to enable the
effective use of GCNs. The overall architecture of our model is illustrated in
Figure 4.

### Problem Formulation

Given a knowledge graph $G=(N,V)$ where $N$ is the set of nodes and $V$ is the
set of edges, we consider a single training instance as the tuple
$v_{i}=(\mathbf{e}_{1},\mathbf{rel},\mathbf{e}_{2})$ with source entity
$\mathbf{e}_{1}$ represented in text as $\bar{\mathbf{e}_{1}}$, relation type
$\mathbf{rel}$ and target entity $\mathbf{e}_{2}$, represented as
$\bar{\mathbf{e}_{2}}$.444we use the terms tuple and edge synonymously. Here,
$v_{i}\in V$ and $\mathbf{e}_{1},\mathbf{e}_{2}\in N$. The objective of the KB
completion task is to maximize the score of a target entity $\mathbf{e}_{2}$
given a tuple prefix $(\mathbf{e}_{1},\mathbf{rel})$. Following previous work
(?), we also include inverse relations in our graph structure – for every edge
$(\mathbf{e}_{1},\mathbf{rel},\mathbf{e}_{2})$, we add an inverse edge
$(\mathbf{e}_{2},\mathbf{rel}^{-1},\mathbf{e}_{1})$.

### Transfer Learning from Text to Knowledge Graphs

Transfer learning from language to knowledge graphs has recently been shown to
be effective for commonsense knowledge graph construction (?). To transfer
from language to knowledge graphs for completion, we finetune BERT (?) with
the masked language modeling loss and obtain rich semantic representations of
nodes based on their text phrase. This allows BERT to be attuned to a KG’s
specific style of text. The input for finetuning is the list of unique phrases
used to represent nodes in the KG. The format of the input to the model is
[CLS] \+ $\bar{\mathbf{e}_{i}}$ \+ [SEP], where $\bar{\mathbf{e}_{i}}$ is the
natural language phrase represented by a node. We use representations of the
[CLS] token from the last layer of the BERT model as node representations in
our models. We represent the node embedding matrix obtained from the BERT
model as $T\in\mathbb{R}^{|N|\times M}$, where $M$ is the dimensionality of
BERT embeddings.

Figure 4: Model Architecture for machine commonsense completion.

### Learning from Graph Structure

Graph Convolutional Networks (GCNs) (?) are effective at incorporating
information from the local neighborhood of a node in the graph. A graph
convolutional encoder take as input a graph $G$, and encodes each node as a
$D$-dimensional embedding $h_{i}\in\mathbb{R}^{D}$ for all nodes
$\mathbf{e}_{i}\in N$. The GCN encoder operates by sending messages from a
node to its neighbors, optionally weighted by the relation type specified by
the edge. This operation occurs in multiple layers, incorporating information
multiple hops away from a node. The last layer’s representation is used as the
graph embedding of the node. Several variants (?; ?) of these models have been
proposed recently, all of which use the same underlying local neighborhood
aggregation mechanism. We choose to use a version of the GCN which allows us
to 1) parameterize the relation type corresponding to an edge and 2) account
for the importance of a node’s neighbor during aggregation. Given the graph
$G$ with $R$ relation types and a GCN with $L$ layers, the operation for
computing the node representation of a node $\mathbf{e}_{i}$ in layer $l+1$
is:

$h_{i}^{l+1}=\tanh\left(\sum_{r\in R}\sum_{j\in
J_{i}}\alpha_{r}\beta_{ij}^{l}W^{l}h_{j}^{l}+W_{0}^{l}h_{i}^{l}\right)$ (1)

where $J_{i}$ represents the neighbors of the node $\mathbf{e}_{i}$ in the
graph, and $W^{l}$ is a linear projection matrix specific to layer $l$. The
initial node representation $h_{i}^{0}$ is computed using an embedding layer.
The second term in Equation 1 represents the self-connection for the node, and
is used to propagate information from one layer to the next. $\alpha_{r}$ is
the weight for the relation type of the edge and $\beta_{i}^{l}$ is a vector
denoting the relative importance of each of $\mathbf{e}_{i}$’s neighbors:

$\beta_{i}^{l}=\textnormal{softmax}(\hat{\boldsymbol{\beta}_{i}^{l}})$ (2)

where each element of $\hat{\boldsymbol{\beta}_{i}^{l}}$ is computed as,

$\hat{\boldsymbol{\beta}_{ij}^{l}}=h_{i}^{l}h_{j}^{l}$ (3)

Here, $h_{i}^{l}$ and $h_{j}^{l}$ are the representation of a node
$\mathbf{e}_{i}$ and its neighbor $\mathbf{e}_{j}$. The output of the GCN is a
node embedding matrix $H\in\mathbb{R}^{|N|\times D}$.

#### Graph Densification

The sparsity of commonsense KGs makes it challenging for GCNs to perform
information propagation over a node’s neighborhood. To tackle this issue, we
add synthetic edges between nodes with semantically similar meanings to boost
the learning of graph embeddings. These edges form a new synthetic sim
relation and are only used for computing graph embeddings but not scored by
the decoder. To form these edges, we use the fine-tuned BERT model described
earlier to extract node representations and use these representations to
compute the cosine similarity between all pairs of nodes in the graph.

Upon computing the pairwise similarities, we use a hard threshold $\tau$ to
filter the pairs of nodes that are most similar. This threshold is computed
using different criteria for each graph, each prioritizing the precision of
these links. For CN-100K ($\tau=$ 0.95 results in 122,618 sim edges), we plot
the distribution of pairwise similarity values between all pairs of nodes and
select the top $\boldsymbol{\sigma}/2$ pairs of nodes to form these synthetic
links, where $\boldsymbol{\sigma}$ is the standard deviation of the normally-
distributed pairwise similarity distribution. For ATOMIC ($\tau=$ 0.98 results
in 89,682 sim edges), we obtain a pairwise similarity distribution that is not
normal, and hence use a threshold (measured up to 2 decimal points) that would
only increase up to 100K edges in the graph555There are possibly other ways to
choose the similarity threshold from a non-normal distribution, but we choose
this criterion for the sake of simplicity. Decreasing this threshold resulted
in minute drops in performance.. After this step, we obtain a set of edges
$V^{\prime}$, where $|V^{\prime}|>|V|$.

### Progressive Masking for Fusion

Models that use node embeddings from GCNs and BERT tend to overly rely on BERT
embeddings, rendering graph embeddings ineffective (we verify this using a
random permutation test (?) where we randomly shuffle graph embeddings in a
minibatch and observe little drop in performance). At the beginning of
training, graph embeddings are not informative whereas fine-tuned BERT
embeddings provide useful information – causing the model to safely ignore
graph embeddings. To prevent this issue, we randomly mask BERT embeddings
starting with an all-zeros mask at the beginning to an all-ones mask at the
end of 100 epochs666midway while training for 200 epochs.. The ratio of
dimensions masked is set as (epoch/100) for the first 100 epochs. This
strategy forces the model to rely on both sources of information. A similar
technique was used to enforce multimodal machine translation models to rely on
images by masking out tokens in the source (?).

### Convolutional Decoder

Convolutional models provide strong scores for KB completion (?; ?) and hence,
we use a convolutional decoder. Given an edge prefix
$(\mathbf{e}_{i},\mathbf{rel})$, graph embeddings $H\in\mathbb{R}^{|N|\times
D}$ from the GCN (§3) and BERT-based node embeddings
$T\in\mathbb{R}^{|N|\times M}$ (§3), the decoder produces a score for
$(\mathbf{e}_{i},\mathbf{rel},\mathbf{e}_{j})$ where
$\mathbf{e}_{i},\mathbf{e}_{j}\in N$. We use the convolutional decoder
ConvTransE (?), to score a tuple. This model is based on ConvE (?) but
additionally models the translational property of TransE (?).

The model uses one of the following as input node embeddings – (i) graph
embeddings $e_{i}=h_{i}$, (ii) BERT-based node embeddings $e_{i}=t_{i}$, or
(iii) a concatenation of both $e_{i}=[h_{i};t_{i}]$.777We experimented with
other fusion methods (e.g., summation, linear transformation of concatenated
representations, initializing GCN with BERT embeddings, tying initial graph
embeddings with BERT embeddings), and found that concatenation as a fusion
method works best. The model proceeds by first stacking the source node
embeddings $e_{i}$, and relation embeddings $e_{rel}$, which are randomly
initialized. The relation embeddings are chosen to have the same
dimensionality as the embedding dimension of $e_{i}$, so that the stacking
operation is possible. Assuming $C$ different kernels and $K$ as the width of
a kernel, the output of kernel $c$ is given by,

$\displaystyle
m_{c}(\mathbf{e}_{i},\mathbf{rel})[\eta]=\sum_{\tau=0}^{K-1}W_{c}(\tau,0)e_{i}(\eta+\tau)$
$\displaystyle+W_{c}(\tau,1)e_{rel}(\eta+\tau)$ (4)

We denote the output for the kernel $c$ to be $M_{c}\in\mathbb{R}^{|e_{i}|}$
and the concatenated output for all kernels to be $M\in\mathbb{R}^{C|e_{i}|}$.
Finally, the score for a tuple $(\mathbf{e}_{i},\mathbf{rel},\mathbf{e}_{j})$
is computed as,

$s(\mathbf{e}_{i},\mathbf{rel},\mathbf{e}_{j})=\sigma(M(e_{i},e_{rel})W_{conv}e_{j})$
(5)

where $W_{conv}\in\mathbb{R}^{C|e_{i}|\times|e_{i}|}$ is a bilinear projection
matrix and $\sigma$ is the sigmoid function. Upon computing scores for all
candidate tuples $s(\mathbf{e}_{i},\mathbf{rel},\mathbf{e}_{j})$, we use a
binary cross entropy loss to train the model. All target nodes found in the
training set are treated as positive instances while all non-existing target
nodes are treated as negative instances.

### Subgraph Sampling

As the graph size increases, it becomes computationally intensive to train
with the entire graph in memory. Specifically, it is intensive to perform
graph convolution over the entire graph and compute scores for all nodes in
the graph using the decoder. For instance, the model with GCN and BERT
representations for ATOMIC occupies $\sim$30GB memory and takes 8-10 days for
training on a Quadro RTX 8000 GPU. Hence, we sample a smaller subgraph for
training. We experiment with different sampling criteria and find that
sampling edges uniformly at random provides the best performance.888We tried
using random walk sampling, snowball sampling and uniform random sampling. For
graph densification, we link all pairs of nodes in the subgraph that cross the
semantic similarity threshold $\tau$.

##  4 Experimental Setup

### Evaluation Metrics

Following previous work on KB completion (?; ?), we use ranking metrics (HITS
and Mean Reciprocal Rank) for the purpose of evaluation. Similar to ? (?), we
filter out all remaining entities valid for an $(\mathbf{e}_{1},\mathbf{rel})$
pair (found across training + validation + test sets) from the ranking when
computing scores for a gold target entity. The scores are measured in both
directions, where we compute the ranking of $\mathbf{e}_{2}$ given
$(\mathbf{e}_{1},\mathbf{rel})$ and the ranking of $\mathbf{e}_{1}$ given
$(\mathbf{e}_{2},\mathbf{rel}^{-})$. We report the mean HITS and MRR scores
averaged across both directions. We note that there are problems with these
automatic metrics. Since commonsense KGs are highly sparse, several false
negative target entities appear at the top of the ranking. Hence, we also
perform human evaluation.

### Baselines

We compare several high performing KB completion models and a transformer-
based commonsense generation model.

##### DistMult

DistMult, proposed by ? (?), is an embedding-based method based on a bi-linear
product between the entities and a relation matrix, which is constrained to be
a diagonal matrix. The score function for a tuple is formulated as
$s(\mathbf{e}_{1},\mathbf{rel},\mathbf{e}_{2})=e_{1}^{T}W_{\mathbf{rel}}e_{2}$.

##### ComplEx

? (?) proposed the use of complex-valued embeddings for nodes and relations,
with the motivation that composition of complex embeddings can allow the model
to handle a large number of relations. The model uses a tri-linear dot product
as the scoring function.

##### ConvE

ConvE (?) is a convolution-based model that stacks the node embedding and
relation embedding for an entity prefix $(\mathbf{e}_{1},\mathbf{rel})$ and
reshapes the resulting tensor. The model performs 2D convolution upon this
reshaped tensor and projects the output to a vector with the same
dimensionality as the node embeddings.

##### ConvTransE

? (?) proposed the ConvTransE model that builds upon the ConvE model but
additionally models the translational properties of TransE. This model is
discussed in depth in Section 3.

##### COMeT

We adapt a commonsense generation model COMeT (?) for completion. COMeT
generates the target phrase for a given (source entity, relation) prefix.
Since COMeT was not trained using inverse relations, we only compute scores in
the forward direction. We use the COMeT model by ranking target nodes based on
the total and normalized negative log-likelihood score of each candidate
tuple. In the absence of a standard approach for calibrating generation log-
likelihoods, we report results using both metrics.999The total score can be
biased against longer sequences; the normalized score can be biased against
shorter sequences. The COMeT scores for ATOMIC have been computed on a smaller
evaluation set with 2000 tuples due to computational limitations (denoted by *
in table).

### Proposed Models

All our proposed models use the ConvTransE decoder. We experiment with using a
GCN as the encoder for obtaining node embeddings, these models are labeled
with the prefix GCN +. Models that utilize synthetic links in the graph are
labeled with the affix sim +. For models enriched with BERT representations,
the ConvTransE decoder takes phrase representations extracted from BERT (BERT
+) or a concatenation of graph embeddings and BERT embeddings (GCN + BERT +)
as input node embeddings.

### Training Regimen

We train all models for at least 200 epochs and continue training until the
MRR on the development set stops improving. The MRR is evaluated on the dev
set every 10 epochs for CN-100K and every 30 epochs for ATOMIC, and the model
checkpoint with the highest MRR is used for testing. Further details about
hyperparameter settings are specified in the supplemental material.

| CN-100K | ATOMIC
---|---|---
| MRR | Hits@1 | @3 | @10 | MRR | Hits@1 | @3 | @10
DistMult | 8.97 | 4.51 | 9.76 | 17.44 | 12.39 | 9.24 | 15.18 | 18.30
ComplEx | 11.40 | 7.42 | 12.45 | 19.01 | 14.24 | 13.27 | 14.13 | 15.96
ConvE | 20.88 | 13.97 | 22.91 | 34.02 | 10.07 | 8.24 | 10.29 | 13.37
ConvTransE | 18.68 | 7.87 | 23.87 | 38.95 | 12.94 | 12.92 | 12.95 | 12.98
COMeT-Normalized | 6.07 | 0.08 | 2.92 | 21.17 | 3.36* | 0.00* | 2.15* | 15.75*
COMeT-Total | 6.21 | 0.00 | 0.00 | 24.00 | 4.91* | 0.00* | 2.40* | 21.60*
BERT + ConvTransE | 49.56 | 38.12 | 55.5 | 71.54 | 12.33 | 10.21 | 12.78 | 16.20
GCN + ConvTransE | 29.80 | 21.25 | 33.04 | 47.50 | 13.12 | 10.70 | 13.74 | 17.68
sim + GCN + ConvTransE | 30.03 | 21.33 | 33.46 | 46.75 | 13.88 | 11.50 | 14.44 | 18.38
GCN + BERT + ConvTransE | 50.38 | 38.79 | 56.46 | 72.96 | 10.8 | 9.04 | 11.21 | 14.10
sim + GCN + BERT + ConvTransE | 51.11 | 39.42 | 59.58 | 73.59 | 10.33 | 8.41 | 10.79 | 13.86
Table 2:  KB Completion Results on CN-100K and ATOMIC with subgraph sampling. We present baselines in the top half of the graph and our implementations in the bottom half. ’*’ indicates difference in evaluation set described in Section 4. | CN-100K
---|---
| MRR | Hits@1 | @3 | @10
BERT + Conv.. | 52.25 | 41.04 | 58.46 | 73.50
GCN + Conv.. | 27.24 | 18.96 | 30.46 | 43.17
sim + GCN + Conv.. | 27.51 | 19.04 | 30.79 | 45.46
GCN + BERT + Conv.. | 50.80 | 39.29 | 57.33 | 72.66
sim + GCN + BERT + Conv.. | 50.17 | 38.71 | 57.08 | 72.00
Table 3:  KB Completion Results on CN-100K with full graph training. Conv.. is
ConvTransE.

##  5 Results and Discussion

### KB Completion Performance

We report our results on completion with subgraph sampling in Table 2. For
completeness, we also report results for CN-100K with the full graph in Table
3. The baseline models are trained with the full graph in memory. We attempt
to answer some pertinent research questions based on our results.

### What does BERT know about commonsense assertions?

Main Finding: BERT is proficient at capturing taxonomic relations and hence,
provides significant boosts for CN-100K but is not as effective for ATOMIC.

When BERT representations are incorporated into a model, performance
improvements ranging from $\sim$19-35 points on MRR are observed for CN-100K.
This indicates that BERT can provide crucial information for the model to
discriminate between target entities. We discuss the reasons for the
usefulness of BERT representations below.

Assertions of the style present in CN-100K are prevalent in large text
corpora, which indicates that masking of tokens during pre-training would have
enabled BERT to already gain a headstart on the task of KB completion. For
instance, consider a sentence ”John bought plastic cups for the party.” where
we mask one token to obtain ”John bought [MASK] cups for the party.” during
pre-training. Now when the BERT representations are used to score a candidate
tuple [cups, MadeOf, plastic], the model might compute a high score for this
tuple because the BERT model solved a similar task during pre-training due to
the masked LM objective. This result concurs with recent investigations on
retrieval of commonsense knowledge from language models (?; ?). Interestingly,
BERT representations are found to be less useful for ATOMIC, because of a
significantly larger number of nodes and more complex relation types (e.g.
oEffect – effect of event on others, xNeed – what actor might need to do
before event) in ATOMIC.

Finally, we attempt to use link prediction as a scaffold to find the
commonsense relations that BERT captures well. Specifically, we use the
BERT+ConvTransE model trained for CN-100K to find the top scoring relations
(by rank) in the evaluation set (presented in Supp:Figure 8). We observe that
BERT is most proficient at picking up on taxonomic relations such as MadeOf,
PartOf and additionally, also does well on temporal relations such as
HasPrerequisite and ReceivesAction. This can be explained by the higher
frequency of taxonomic knowledge present in large text corpora as opposed to
more complicated relations found in ATOMIC.

### How crucial is graph structure information?

Main Finding: Local graph structure information boosts performance when
training with subgraph sampling, but improvements fade when training with the
entire graph.

We note that in the absence of BERT representations, incorporating graph
embeddings provides strong improvements ($>$ +9 points on MRR for CN-100K and
+$\sim$0.2 points for ATOMIC) over the model without a GCN encoder (i.e.
GCN+ConvTransE is better than ConvTransE). Similarly in the presence of BERT
representations, the graph embeddings provide complementary information about
the local neighborhood of a node which boosts performance for CN-100K. When
training with the entire graph in memory, the improvements with using graph
embeddings fade away. However, it is important to note that, using graph
embeddings provides benefits when it is infeasible to train models with the
entire graph in memory.

We further verify the importance of graph embeddings when BERT-based
representations are used with a random permutation test (?). In this test, we
randomly shuffle the graph embeddings within each minibatch during inference
and note the drop in performance. For the sim+GCN+BERT+ConvTransE model, we
observe $\Delta$MRR drops of -7.24, -8.98 for CN-100K and ATOMIC,
respectively.

### Do similarity-induced edges help?

Main Finding: Similarity-induced edges boost learning of graph embeddings
resulting in improved performance.

We observe that when both graphs are trained with subgraph sampling but in the
absence of BERT representations, augmenting the graph with similarity-induced
edges provides improvements (sim+GCN+ConvTransE $>$ GCN+ConvTransE ). When
BERT representations are incorporated, the similarity-induced edges continue
to help for CN-100K but not for ATOMIC. The sim+GCN+BERT+ConvTransE model
achieves the best MRR for CN-100K. For ATOMIC, the sim+GCN+ConvTransE model
provides even stronger results than models which use BERT (+1.5 MRR points
over BERT+ConvTransE ). This allows us to conclude that augmenting commonsense
KGs with similarity-induced links can provide more context for computing graph
embeddings.

### Can we use generative models for ranking tuples?

Main Finding: Generative models cannot easily be repurposed to rank tuples for
KB completion.

Although generative models like COMeT have shown to produce diverse and
precise target phrases, our results from Table 2 indicate that it is non-
trivial to repurpose them to perform completion. This is partly due to the
problems associated with using log-likelihood as an estimate for the truth of
a tuple. Nonetheless, generative models such as COMeT have several merits.
These models are faster to train, require lower memory for storage and are
transductive in nature. However, we argue that reasoning models that rely on
KBs could benefit more from a discriminative approach towards commonsense
knowledge induction, one that would make the graph denser without adding new
nodes.

### Human Evaluation

Upon looking at model predictions, we note that false negative entities can
appear at the top of the ranking. For instance, when $\mathbf{e}_{1}$=PersonX
wins two tickets and $\mathbf{rel}$=xEffect and the gold target entity
$\mathbf{e}_{2}$=elated and excited, our model predicts elated and excited
separately in the top 10 candidate entities. Automated metrics fail to capture
this semantic similarity in predictions. Hence, we perform human evaluation by
presenting the top 10 predicted targets to 3 annotators on Amazon Mechanical
Turk (AMT), for a subset of the instances in the dev set. This subset was
created by sampling 200 instances randomly for CN-100K, and 450 instances (=50
instances for 9 relations) for ATOMIC. The annotators were asked to answer
whether a tuple is valid, where a valid tuple is meaningful and true. Our
results are reported in Figure 5. Samples of our model predictions with human
annotations are provided in the supplementary material. Human evaluation
results indicate that using graph embeddings computed with graph densification
in addition to BERT shows improvements.

Figure 5: Human Evaluation Results (Average % of valid tuples among top-10
target candidates for random subset of entity-relation pairs)

##  6 Related Work

A host of techniques (?; ?; ?; ?; ?) have been proposed for KB completion.
These can be classified into graph traversal based and embedding-based
approaches. Although embedding-based methods are more scalable, they usually
assume enough training instances for each relation and a relatively high
average degree for nodes to generalize well. Both these criteria are typically
not satisfied by commonsense KGs.

Among embedding-based methods, convolutional models have proven useful for
computing entity-relation feature representations (?; ?) and hence, we use a
convolutional decoder. Simultaneously, we make use of GCNs to incorporate a
node’s neighborhood into its representation. Much as we do, ? (?) use a GCN to
compute graph embeddings and the ConvTransE decoder to score tuples. Our work
differs from their model, as we account for the relative importance of
neighbors in our graph convolution operation, tackle the sparsity of the graph
structure by graph densification, and provide strategies to jointly train
using graph embeddings and BERT representations. Finally, the use of BERT
embeddings follows from work which showed that initializing node embeddings
with pre-trained GloVe vectors (?) improves KB completion performance (?).

Prior work on completing commonsense KBs (?; ?; ?) uses BiLSTMs to encode a
tuple and performs linear transformations to predict a score for binary
classification. We argue that commonsense KB completion models should be
trained and evaluated for ranking. A similar effort aimed at predicting new
knowledge in ConceptNet, using dimensionality reduction was made by ? (?).

We believe that completing commonsense KBs is bound to translate into
improvements on a range of tasks that rely on these KBs, such as information
retrieval (?), question answering (?; ?), and reading comprehension (?). ? (?)
provide a survey of the use of commonsense KBs in downstream tasks.

##  7 Conclusion

In this work, we show that existing KB completion models underperform with
commonsense knowledge graphs, due to the sparsity and scale of these graphs.
As our solution, we propose novel KB completion models, which are enriched by
structural and semantic context, obtained from GCNs and language model
representations. We describe a progressive masking strategy to efficiently
utilize information from both sources. Further, we show that augmenting the
graph with semantic similarity edges can help with completion. Our results
indicate that 1) BERT-based node representations can provide significant
improvements, especially when text in the graph is similar to pre-training
corpora; 2) graph embeddings can provide rich local context for encoding
nodes, and boost performance when training with subgraphs.

##  8 Acknowledgments

We thank the anonymous reviewers and the Mosaic team at AI2 for their
insightful comments and suggestions. We also thank the Amazon MTurk workers
who provided annotations to our human evaluation tasks. Finally, we thank the
Beaker team at AI2 for providing support with experiments. This research was
supported in part by NSF (IIS-1524371, IIS-1714566), DARPA under the CwC
program through the ARO (W911NF-15-1-0543), and DARPA under the MCS program
through NIWC Pacific (N66001-19-2-4031). Computations on beaker.org were
supported in part by credits from Google Cloud.

## References

  * [2018] Bauer, L.; Wang, Y.; and Bansal, M.  2018\.  Commonsense for generative multi-hop question answering tasks.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 4220–4230.  ACL.
  * [2013] Bordes, A.; Usunier, N.; Garcia-Duran, A.; Weston, J.; and Yakhnenko, O.  2013\.  Translating embeddings for modeling multi-relational data.  In Advances in neural information processing systems, 2787–2795.
  * [2019] Bosselut, A.; Rashkin, H.; Sap, M.; Malaviya, C.; Celikyilmaz, A.; and Choi, Y.  2019\.  Comet: Commonsense transformers for automatic knowledge graph construction.  In The 57th Annual Meeting of the Association for Computational Linguistics (ACL).
  * [2019] Caglayan, O.; Madhyastha, P.; Specia, L.; and Barrault, L.  2019\.  Probing the need for visual context in multimodal machine translation.  In NAACL 2019.  ACL.
  * [2014] Cambria, E.; Olsher, D.; and Rajagopal, D.  2014\.  Senticnet 3: a common and common-sense knowledge base for cognition-driven sentiment analysis.  In Twenty-eighth AAAI conference on artificial intelligence.
  * [2018] Dettmers, T.; Minervini, P.; Stenetorp, P.; and Riedel, S.  2018\.  Convolutional 2d knowledge graph embeddings.  In Thirty-Second AAAI Conference on Artificial Intelligence.
  * [2019] Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K.  2019\.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In NAACL 2019.
  * [2019] Feldman, J.; Davison, J.; and Rush, A. M.  2019\.  Commonsense knowledge mining from pretrained models.  In Conference on Empirical Methods in Natural Language Processing (EMNLP).
  * [2018] Fisher, A.; Rudin, C.; and Dominici, F.  2018\.  Model class reliance: Variable importance measures for any machine learning model class, from the ”rashomon” perspective.  arXiv preprint arXiv:1801.01489.
  * [2015] Guu, K.; Miller, J.; and Liang, P.  2015\.  Traversing knowledge graphs in vector space.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 318–327.  Lisbon, Portugal: ACL.
  * [2018] Jastrzebski, S.; Bahdanau, D.; Hosseini, S.; Noukhovitch, M.; Bengio, Y.; and Cheung, J.  2018\.  Commonsense mining as knowledge base completion? a study on the impact of novelty.  In Proceedings of the Workshop on Generalization in the Age of Deep Learning, 8–16.  ACL.
  * [2017] Kipf, T. N., and Welling, M.  2017\.  Semi-supervised classification with graph convolutional networks.  In ICLR.
  * [2012] Kotov, A., and Zhai, C.  2012\.  Tapping into knowledge base for concept feedback: leveraging conceptnet to improve search results for difficult queries.  In Proceedings of the fifth ACM international conference on Web search and data mining, 403–412.  ACM.
  * [1995] Lenat, D. B.  1995\.  Cyc: A large-scale investment in knowledge infrastructure.  Communications of the ACM 38(11):33–38.
  * [2016] Li, X.; Taheri, A.; Tu, L.; and Gimpel, K.  2016\.  Commonsense knowledge base completion.  In ACL, 1445–1455.
  * [2018] Musa, R.; Wang, X.; Fokoue, A.; Mattei, N.; Chang, M.; Kapanipathi, P.; Makni, B.; Talamadupula, K.; and Witbrock, M.  2018\.  Answering science exam questions using query reformulation with background knowledge.
  * [2018] Ostermann, S.; Roth, M.; Modi, A.; Thater, S.; and Pinkal, M.  2018\.  Semeval-2018 task 11: Machine comprehension using commonsense knowledge.  In Proceedings of The 12th International Workshop on Semantic Evaluation, 747–757.
  * [2014] Pennington, J.; Socher, R.; and Manning, C.  2014\.  Glove: Global vectors for word representation.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1532–1543.  ACL.
  * [2019] Petroni, F.; Rocktaschel, T.; Lewis, P.; Bakhtin, A.; Wu1, Y.; Miller, A. H.; and Riedel, S.  2019\.  Language models as knowledge bases?  In Conference on Empirical Methods in Natural Language Processing (EMNLP).
  * [2018] Saito, I.; Nishida, K.; Asano, H.; and Tomita, J.  2018\.  Commonsense knowledge base completion and generation.  In Proceedings of the 22nd Conference on Computational Natural Language Learning, 141–150.  ACL.
  * [2019] Sap, M.; LeBras, R.; Allaway, E.; Bhagavatula, C.; Lourie, N.; Rashkin, H.; Roof, B.; Smith, N. A.; and Choi, Y.  2019\.  Atomic: An atlas of machine commonsense for if-then reasoning.  In AAAI.
  * [2018] Schlichtkrull, M.; Kipf, T. N.; Bloem, P.; Van Den Berg, R.; Titov, I.; and Welling, M.  2018\.  Modeling relational data with graph convolutional networks.  In European Semantic Web Conference, 593–607.  Springer.
  * [2019] Shang, C.; Tang, Y.; Huang, J.; Bi, J.; He, X.; and Zhou, B.  2019\.  End-to-end structure-aware convolutional networks for knowledge base completion.  In AAAI.
  * [2013] Speer, R., and Havasi, C.  2013\.  Conceptnet 5: A large semantic network for relational knowledge.  In The People’s Web Meets NLP. Springer.  161–176.
  * [2008] Speer, R.; Havasi, C.; and Lieberman, H.  2008\.  Analogyspace: Reducing the dimensionality of common sense knowledge.  In AAAI, volume 8, 548–553.
  * [2019] Storks, S.; Gao, Q.; and Chai, J. Y.  2019\.  Commonsense reasoning for natural language understanding: A survey of benchmarks, resources, and approaches.  arXiv preprint arXiv:1904.01172.
  * [2015] Toutanova, K., and Chen, D.  2015\.  Observed versus latent features for knowledge base and text inference.  In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality, 57–66.  ACL.
  * [2016] Trouillon, T.; Welbl, J.; Riedel, S.; Gaussier, É.; and Bouchard, G.  2016\.  Complex embeddings for simple link prediction.  In ICML, 2071–2080.
  * [2018] Veličković, P.; Cucurull, G.; Casanova, A.; Romero, A.; Liò, P.; and Bengio, Y.  2018\.  Graph Attention Networks.  International Conference on Learning Representations.
  * [2015] Yang, B.; Yih, W.-t.; He, X.; Gao, J.; and Deng, L.  2015\.  Embedding entities and relations for learning and inference in knowledge bases.  In Proceedings of the International Conference on Learning Representations (ICLR).

##  9 Supplementary Material

### Hyperparameter Details

##### BERT Fine-tuning

We used a maximum sequence length of 64, batch size of 32, and learning rate
of 3e-5 to fine-tune the uncased BERT-Large model with the masked language
modeling objective. The warmup proportion was set to 0.1.

##### Baseline Models

To train the baseline models, we used the implementations provided
here.101010https://github.com/TimDettmers/ConvE We tuned the batch size and
learning rate for the baseline models from 128, 256, 512 and 0.001, 0.002,
0.003 and used the default values for other hyperparameters.

##### Our Implementations

The graph convolutional network used 2 layers (using more ({3,4,5}) layers did
not result in significant improvements) and an input and output embedding
dimension of 200. The message passing algorithm for the GCN-based models was
implemented using the Deep Graph Library (DGL). All embedding layers are
initialized with Glorot initialization. The graph batch size used for subgraph
sampling was 30000 edges. For the ConvTransE decoder, we used 500 channels, a
kernel size of 5 and a batch size of 128. Dropout was enforced at the feature
map layers, the input layer and after the fully connected layer in the
decoder, with a value of 0.2. The Adam optimizer was used for optimization
with a learning rate of 1e-4 and gradient clipping was performed with a max
gradient norm value of 1.0. We performed L2 weight regularization with a
weight of 0.1. We also used label smoothing with a value of 0.1.

### Example Predictions

Figure 6: Randomly sampled top-1 predictions from best model
(sim+GCN+ConvTransE+BERT) for CN-100K along with human annotations for
validity of tuples. Figure 7: Randomly sampled top-1 predictions from best
model (sim+GCN+ConvTransE) for ATOMIC along with human annotations for
validity of tuples. While the complete relation descriptions can be found in ?
(?), it is noted that ”x” refers to PersonX and ”o” refers to other actors
besides PersonX. Figure 8: Top scoring relations for BERT+ConvTransE (top) and
sim+GCN+ConvTransE (bottom).


11institutetext: University of Nottingham, Ningbo, China
11email<EMAIL_ADDRESS>22institutetext: Shanghai Maritime
University, Shanghai, China
22email<EMAIL_ADDRESS>
# DiffuSyn Bench: Evaluating Vision-Language Models on Real-World Complexities
with Diffusion-Generated Synthetic Benchmarks

Haokun Zhou 1⋆1⋆ 0009-0001-5955-285X Yipeng Hong These authors contributed
equally to this work.22 0009-0000-9896-5617

###### Abstract

This study assesses the ability of Large Vision-Language Models (LVLMs) to
differentiate between AI-generated and human-generated images. It introduces a
new automated benchmark construction method for this evaluation. The
experiment compared common LVLMs with human participants using a mixed dataset
of AI and human-created images. Results showed that LVLMs could distinguish
between the image types to some extent but exhibited a rightward bias, and
perform significantly worse compared to humans. To build on these findings, we
developed an automated benchmark construction process using AI. This process
involved topic retrieval, narrative script generation, error embedding, and
image generation, creating a diverse set of text-image pairs with intentional
errors. We validated our method through constructing two caparable benchmarks.
This study highlights the strengths and weaknesses of LVLMs in real-world
understanding and advances benchmark construction techniques, providing a
scalable and automatic approach for AI model evaluation.

###### Keywords:

Vision language model Benchmark Synthetic data Latent diffusion model

##  1 Introduction

Figure 1: This is a simple example that shows a brief framework for benchmark
generation process and provides a specific example from our benchmark,
showcasing the response from GPT-4V to our tests. The example image features a
modern office setting with a medieval knight working on a laptop, highlighting
the intentional anachronism. This example also reveals why GPT-4V may have
underperformed in the initial part of our test, potentially due to alignment
issues.

In recent years, remarkable advances in deep learning within the fields of
computer vision and natural language processing have propelled Large Vision-
Language Models (LVLMs) to the forefront as powerful tools for understanding
and interpreting visual and textual data. Despite their impressive
capabilities, the extent to which these models can accurately comprehend and
analyze real-world scenarios remains a subject of ongoing investigation. This
essay embarks on a dual-fold exploration: firstly, it assesses the capacity of
LVLMs to differentiate between AI-generated and human-originated images—a task
that probes their ability to interpret and understand nuanced visual
information. Secondly, it extends these insights to the systematic development
of a methodology for the automatic generation of benchmarks designed to
evaluate this ability.

The first part of our study focuses on a rigorous examination of LVLMs’
performance in identifying anomalies and errors within images, utilizing a
dataset comprising both AI-generated and human-created visuals. This
evaluation is critical for understanding the models’ proficiency in real-world
understanding, as well as their ability to discern subtle differences in image
quality and content.

Building upon these findings, the second part of the essay delves into the
development of an automatic benchmark construction methodology. This
innovative approach involves the use of AI techniques to generate and validate
benchmarks, providing a scalable and efficient means of testing LVLMs’
capabilities. By integrating automated processes, we aim to create robust and
reliable benchmarks that can be utilized to continuously assess and improve
the performance of LVLMs in various real-world tasks.

Through a combination of experimental evaluation and methodological
innovation, this essay provides a comprehensive overview of LVLMs’ abilities
and the potential for AI-driven benchmark construction. Our findings
contribute to the broader understanding of LVLMs’ real-world comprehension and
pave the way for future advancements in AI model evaluation and development.

##  2 Recognizing LVLMs’ ability to understand the real world

Recently, generative models have made significant strides in synthesizing
realistic images. Various text-to-image generative models, exemplified by
Latent Diffusion Models (LDM), have blurred the lines between computer-
generated and real-world images. Prominent models such as Midjourney[3],
DALL-E[1], and Stable Diffusion[2], built on denoising autoencoders, exhibit
an exceptional ability to produce lifelike images. Concurrently,
discriminative methods rooted in traditional computer vision classification
techniques, such as the CNN-based ResNet[4], transformer-based DeiT-S[5], and
Swin-T[6], have been explored for their effectiveness in distinguishing text-
to-image (T2I) creations in the existing literature[7]. Despite these
advances, a critical observation in this domain is the inherent limitation of
these models in replicating the fidelity of the real world. Unlike algorithm-
based feature extraction, humans possess an innate ability to discern AI-
generated images by identifying intrinsic errors or anomalies.

This observation has propelled our investigation into the capabilities of
Large Vision-Language Models (LVLMs), which have demonstrated impressive
performance across various tasks and exhibit human-like responses to images by
integrating visual and textual processing abilities.

##  3 Method

###  3.1 Overview

This experiment aims to evaluate the ability of Large Vision Language Models
(LVLMs) to distinguish between AI-generated and human-generated images and to
compare their performance with that of human participants. We utilize GPT-3.5
to process the LVLMs’ responses into binary data for analysis.

###  3.2 Image Dataset

The dataset includes 2000 images, evenly split between AI-generated images and
those created by human artists. The AI-generated images include blends
generated by several renowned diffusion models, including Stable Diffusion
2.1, Stable Diffusion 1.5, and DALL-E 3. The Stable Diffusion images are
sourced from Poloclub’s DiffusionDB dataset, while the DALL-E 3 images are
selected from LAION’s DALL-E 3 dataset. The human-generated images are sourced
from diverse and publicly available collections, selected to cover a wide
range of subjects and styles. All images are standardized to a resolution of
512x512 pixels to ensure uniformity.

###  3.3 Experiment Settings

For GPT-4V, we utilized the API endpoint specifically designated for the
GPT-4-1106-Vision-Preview, dated November 6, 2023. For the other models, we
employed a selection of publicly accessible models and corresponding
endpoints, conducting the experiments using a computational setup consisting
of two blocks of A100 cloud servers.

The methodology involved a meticulous process of prompt selection. Various
prompts were iteratively tested to ascertain their effectiveness in two key
areas: categorizing images using LVLMs and directing human participants to
perform the same task. For example, initial prompts that directly asked if an
image was AI-generated often led to refusals or biased answers, whereas
descriptive prompts about image quality yielded better results. The specific
prompts finalized for the experiment are detailed in the appendix of this
document.

A critical parameter in our experimental design was the ’temperature’ setting,
which was fixed at 0. This decision was driven by the necessity to minimize
variation in the model’s output, ensuring a more deterministic and stable
output, crucial for the reliability of the categorization process.

###  3.4 Procedure for LVLMs

The LVLMs were presented with each image in the dataset and tasked with
identifying any anomalies or errors indicative of AI generation. Directly
asking if the image is AI-generated or contains errors may result in a refusal
to answer or affect the result due to the alignment of the LVLM. Therefore,
following their analysis, the LVLMs provided a descriptive response for each
image. The responses from the LVLMs were then processed through
GPT-3.5-Turbo-1106, which was used to interpret the LVLMs’ descriptive answers
and convert them into binary outcomes: ’AI-generated’ or ’Human-generated.’
This conversion involves identifying specific patterns and keywords indicative
of AI generation. This step standardizes the results for statistical
comparison.

###  3.5 Procedure for Human Participants

Each human participant was presented with a randomized set of 200 images from
the dataset, consisting of 100 AI-generated images and 100 images from human
artists. They were instructed to categorize each image as either AI-generated
or human-generated, relying on their judgment and observation. Human subjects
were provided with the same prompt as the LVLMs, with no examples included and
no time constraints imposed, allowing for careful consideration. Before the
main task, participants underwent a brief training session (5 shots) with a
different set of images to familiarize them with the task. The experiment did
not include any subjects with knowledge related to NLP or CV, nor past
experience in using diffusion models.

###  3.6 Statistical Analysis

The performance of LVLMs and human participants will be compared using
statistical tests to evaluate the significance of their performance
differences.

##  4 Result and Analysis

###  4.1 Overview of Findings

Our study’s findings are delineated through two analytical representations: a
confusion matrix and a bar chart. These visual aids serve to elucidate the
discernment capabilities of Large Vision-Language Models (LVLMs) versus human
participants in classifying images as either AI-generated or human-originated.

###  4.2 Observation and analysis

The construction of a confusion matrix forms the cornerstone of our analysis,
partitioned into True Positives (TP), False Positives (FP), True Negatives
(TN), and False Negatives (FN). In this framework, TP represents correct
identifications of AI-generated images, while TN signifies accurate
classifications of human-generated images. Conversely, FP and FN denote
erroneous classifications. This structured approach enables a detailed
examination of error patterns within both LVLMs and human cohorts.

Our findings highlight a notable inclination among LVLMs to misclassify images
as human-generated, indicating a rightward bias. This tendency suggests a
deficiency in LVLMs’ comprehension of real-world textures and physical
attributes, posing challenges in discerning subtle nuances characteristic of
AI-generated imagery. However, we propose that this bias may be partly
attributable to systemic alignment biases. Further analysis reveals instances
where humans misclassify photographs lacking discernible AI-generated features
as AI images, leading to an elevated FN rate. Conversely, examination of raw
data unveils a predisposition in LVLMs, particularly GPT-4V, to reject AI-
origin attributions even in the presence of detectable AI-specific features or
when such features are erroneously identified. This disparity underscores the
existence of systemic alignment biases in LVLMs, potentially impeding accuracy
in positive identifications. (see Fig. 2)

Figure 2: showcases the performance comparison between different Large
Language Models (LLMs) and humans in the task of distinguishing between AI-
generated and human-generated images. The data is presented in six heatmaps,
each representing the performance of one LLM or human. The x-axis in each
heatmap denotes the predicted label (either "Guessed AI" or "Guessed Human"),
while the y-axis denotes the actual label (either "Actual AI" or "Actual
Human")..

###  4.3 Quantitative Analysis

In subsequent quantitative analyses, we employ accuracy and F1-score metrics
to evaluate prediction efficacy. Accuracy gauges the proportion of correctly
predicted instances relative to total predictions, while the F1-score,
encompassing precision and recall, offers a nuanced performance assessment.
Precision measures the ratio of true positive predictions to all positive
predictions, whereas recall quantifies true positive predictions relative to
all actual positive instances. Formally, the F1-score is defined as follows:

$\text{F1-score}=2\times\frac{\text{TP}}{2\times\text{TP}+\text{FP}+\text{FN}}$
(1)

Human observers achieved an exceptional 88.45% accuracy in differentiating
between AI and non-AI generated images, and have a similar correctness in both
tasks. In contrast, all language-based AI models (LVLMs) produced lower
overall correct rates, with the highest performing Fuyu-8B model achieving
only 66.1% correctness and the lowest ranking LLaVA-1.5-13B model with a mere
50.5% correctness. As for the F1-score, Cogvlm, Qwen, LLaVA, Fuyu, and GPT-4V
were 48.9%, 18.1%, 3.7%, 64.9%, and 28.0%, respectively, which was also
significantly different from the 88.1% obtained by humans. From the results,
the LVLMs show inefficient ability at the task of correctly recognizing AI-
generated images and demonstrate a significant difference from human ability.

Chi-square tests for independence corroborate LVLMs’ capacity to identify AI-
generated content beyond chance, except for the LLaVA-1.5-13B model. This
statistical significance underscores LVLMs’ innate ability to detect errors
indicative of AI-generated images, notwithstanding their inherent
limitations.(see Fig. 3)

Figure 3: compares the overall performance of humans and various models in
identifying AI-generated and non-AI-generated images. The y-axis shows
correctness percentages, while the x-axis lists the participants. The bars are
divided into segments: blue for correctly identified AI-generated images and
orange for correctly identified non-AI-generated images. The red dashed line
at 50% marks the threshold for random guessing.

##  5 Automatic Benchmark Construction and Evaluation Based on Our Findings

Expanding upon our thorough examination of LVLMs’ proficiency in real-world
understanding, it is essential to situate our discoveries within the broader
context of benchmark evaluations and to embark on the creation of a novel
benchmark tailored to our investigative task. Our findings bear particular
significance as the evaluation inherently revolves around AI image generation,
suggesting that benchmark construction for this task could conceivably rely
entirely on AI methodologies. Numerous benchmarks have previously focused on
testing LVLMs’ abilities across a wide range of areas, including MM-Vet[8],
HallusionBench[9], humor understanding benchmarks[10], and MMMU[11].
Additionally, benchmarks such as VisIT-Bench[12] and VASR[13] have
concentrated on visual reasoning and real-world comprehension.

In light of these precedents, we now endeavor to craft an innovative benchmark
framework that assesses LVLMs’ aptitude in discerning AI-generated images and
their ability to understand real-world contexts. Our observations suggest that
a synthetic benchmark, constructed through a fully automated process, could be
employed. This approach diverges from prior work, such as the use of
Photorealistic Unreal Graphics (PUG)[16] to construct complex scenes. Instead,
we propose leveraging AI-based methods to create benchmarks with broader
capabilities, thereby enhancing the robustness and relevance of our
evaluations.(see Fig. 4)

Figure 4: An automated bench-marking framework designed to assess the
proficiency of Large Vision-Language Models in identifying AI-generated images
and understanding real-world contexts.

##  6 Method

Current text-to-image models often face many limitations, such as the lack of
precise output control and ’illusions’ that can cause their generated images
to contain parts that are difficult to predict[14]. Additionally, they have
difficulty combining multiple concepts, which leads to mixing items with
different properties[15]. These problems can lead to unintended, problematic
images. To minimize these issues, we devised a unique procedure to generate a
dataset comprising prompts with embedded errors and their visual
representations. The process unfolds in several distinct stages, splitting the
assignment of tasks to four language models and a text-to-image conversion
model. Through this approach, we aim to challenge and assess the capabilities
of LVLM models in recognizing and interpreting errors with a scalable and
automatic method.

Topic Retrieval: The topic pool is a set of phrase entries which served as the
topics of the text-image pair. Initially, a program randomly selects a phrase
from the predefined topic pool and the same topic will be passed to both LM1
and LM2.

Narrative Script Generation: Utilizing GPT-4, we generate a detailed narrative
script based on the retrieved topic. This script is designed to be
comprehensive and contextually rich, setting the stage for the introduction of
errors.

Error Generation: Parallelly, another instance of GPT-4 is tasked with
generating a specific error related to the same topic. This error is intended
to introduce a deliberate flaw or inconsistency within the context of the
narrative script.

Prompt Synthesis with Embedded Error: Both the detailed narrative script and
the generated error are then combined by a third instance of GPT-4. This step
synthesizes a cohesive prompt that seamlessly integrates the narrative context
with the specified error.

Evaluation of Prompt: Due to the limited capabilities of the text-to-image
model, we found that in many cases, images do not efficiently perform the
prompts generated by the language model, e.g., ’the shadow of the tree faces
the sun’, which introduces noise to the dataset. We use a language model to
act as a judge, controlling the type of error and the proportion of the image
accounted for by the error, in order to increase the success rate of image
generation.

Image Generation: The synthesized prompt, now containing the embedded error,
is passed to DALL-E. Leveraging its text-to-image capabilities, DALL-E
generates an image that visually manifests the embedded error, creating a
direct visual representation of the narrative and error combination.

Benchmark Creation and Evaluation: Each text-image pair produced through this
process establishes a unique benchmark within our dataset. To automate the
evaluation of the model’s ability to recognize and interpret errors in images,
we use GPT-4, which compares pre-generated prompts to the model’s responses,
with individual scores ranging from 0 to 10, to quantify the accuracy of the
model’s responses.

The errors introduced into our benchmarks were systematically categorized into
three distinct types:

  * •

Biological Errors: These include anatomical anomalies, such as living beings
missing essential parts or possessing extraneous ones, and behaviors that defy
natural instincts or capabilities, exemplified by a fish capable of flight or
a human engaging in photosynthesis.

  * •

Mismatched Eras: This category captures errors involving anachronisms or the
juxtaposition of elements from disparate historical periods without logical
coherence, creating scenarios that challenge temporal understanding.

  * •

Logical Inconsistencies: Encompassing functionality flaws—such as a door that
leads to an impossible space—and incongruous object combinations that defy
logical grouping or use, this category tests the model’s ability to recognize
and rationalize physical and conceptual discrepancies.

Upon generating and meticulously reviewing potential benchmarks, we dropped
the noise and result a final set comprising 287, 289, and 272 text-and-image
pairs for each error category, respectively. Through this framework,
generation failures accounted for only 5.8% of all generated images, compared
to 28.1% in the dataset where our generation architecture was not introduced,
where prompts were generated by a single GPT4 and images were generated by the
Dall-E 3 model. At the same time, the bias of the benchmark is also
effectively controlled. When using a single LM, the three scenarios of
kitchen, living room, and office occupy more than 90% of the total dataset,
and even with higher temperature parameters, there are a lot of repetitive
themes such as the error of introducing traffic lights in the kitchen, whereas
after the introduction of our method, each of these scenarios accounts for no
more than 5%.

##  7 Benchmark Result

We tested four LVLM models—CogVLM, GPT-4V, LLaVA, and Qwen—on our benchmark.
The models were tasked with describing the errors present in the images, and
their outputs were compared to the error descriptions in the dataset, scoring
them on a scale of 0-10 via GPT-4. From our results, GPT-4V demonstrated
superiority in all three aspects compared to the other models. Specifically,
we find that GPT-4V consistently outperforms comparable models in three
categories: temporal error, biological error, and logical error.

Temporal Errors: Figure 3 shows the test results of the different models.
GPT-4V scored a total of 2031 points for temporal errors, while its closest
competitor, LLaVA, scored a total of 1347 points, indicating a large
performance gap between the two.

Biological Errors: In terms of biological errors, GPT-4V earned a total score
of 1268 compared to 796 for LLaVA and 657 for Qwen. This advantage suggests
that GPT-4V may have a deeper understanding of images as well as the real
world, enabling it to generate more accurate descriptions of image errors.

Logical Errors: Similarly, in terms of logical errors, GPT-4V’s total score of
177 exceeds LLaVA’s 164 and Qwen’s 121. This also validates our observation in
the previous section that GPT-4V’s lower scores on the pairwise error
discrimination task are mainly due to its stronger alignment, which leads to a
refusal to recognize the presence of definite errors in the image.

Overall, the data show that GPT-4V consistently performs better across all
error categories, confirming its superior capability in error recognition and
description. However, the performance of all models on data other than
temporal errors is poorer, which illustrates the fact that current LVLMs still
have major limitations in terms of the consistency of image details with
reality.(see Fig. 5)

Figure 5: The performance of four models across three types of errors, GPT4V
achieve outstanding scores among three aspects.

To further validate our benchmark generation method, we conducted additional
experimental evaluations. We created two other comparable benchmarks using
different methodologies. In one benchmark, we selected images with obvious
errors from a public diffusion model dataset and manually described the
errors. In the other, we used human-authored prompts containing errors and
employed the same image generation model to complete the dataset. We then
tested the same LVLMs on these new benchmarks and recorded their scores.

We tested the same LVLMs on these new benchmarks and recorded their scores, as
shown in Fig. 6. Comparing the original dataset with the human-authored
prompts benchmark, we observed no significant differences in LVLM scores.
However, in the comparison with the manually screened and labeled dataset, we
found that the scores of all the LVLMs were reduced, and upon observation of
the dataset, this was due to the limited control that Dall-E 3 has over the
generated images. Errors in images from public datasets often stem from the
diffusion model’s inherent limitations rather than the specified input prompt,
resulting in less obvious errors and smaller error areas. Despite these
differences, the ranking of LVLMs’ abilities remained consistent, with a
Spearman rank correlation of 0.93 (p < 0.001) between our synthetic benchmark
and the manually labeled benchmark. This consistency underscores the validity
of our benchmark in assessing LVLMs, despite the synthetic dataset being less
challenging.This shows our benchmark maintains relatively great assessment
validity while using automated synthesis methods, and can effectively
evaluating the real-world capabilities of LVLMs.

Figure 6: Benchmark Results Using Different Construction Methods: Average
performance metrics of four LVLMs across different error types (temporal,
biological, logical) are compared using three different benchmarks. The yellow
line represents the original synthetic benchmark, the orange line represents
the human-authored prompts benchmark, and the red line represents the manually
screened and labeled benchmark. LVLMs achieve similar results on yellow and
orange lines

##  8 Conclusion

The exploration into the proficiency of Large Vision-Language Models (LVLMs)
in discerning AI-generated images versus human-created ones has yielded
illuminating insights. Our comprehensive study, juxtaposing LVLMs against
human participants, reveals a pronounced disparity in their capabilities.
Human participants demonstrated exceptional accuracy, significantly
outperforming LVLMs, which exhibited inherent limitations in recognizing AI-
generated imagery.

Based on these findings, we developed a new benchmark that leverages the
unique ability of AI to generate hard-to-obtain images, and proposed a robust
framework to evaluate the misidentification ability of LVLMs using synthetic
data. This approach enhances the scalability and flexibility of the benchmark,
and making it resistant to data contamination. This ensures that the models
are rigorously tested on their true capabilities rather than their familiarity
with training data.

This study delineates the current landscape of LVLM proficiency, demonstrating
a novel approach for creating synthetic benchmarks on visio-linguistic
abilities. It provides valuable experiments and data to support subsequent
related research, paving the way for further advancements in the evaluation
and development of LVLMs. Our findings underscore the necessity for continuous
improvement in LVLMs to bridge the gap between human and machine capabilities
in image recognition and understanding.

## Limitations

While our methodology aims to leverage AI for generating a comprehensive and
diverse set of benchmarks, inherent limitations exist in both text-to-image
(T2I) models and large language models (LLMs). The T2I models sometimes
struggle to produce erroneous images following specific instructions, and LLMs
exhibit limited ability to understand this characteristic of T2I models and
craft clear, precise instructions for T2I models. This can introduce noise
into the synthetic data, leading to less pronounced or unobservable errors and
weakening the accuracy of the benchmarks.

To address this issue, we plan to enhance the precision of generated data and
further reduce noise. Our efforts include constructing more powerful LLM
agents combined with state-of-the-art T2I models, which we believe will help
alleviate these challenges. By improving the clarity and specificity of
instructions given to T2I models, and by refining the error generation
process, we aim to create benchmarks that are both highly accurate and
representative of real-world complexities. This will ultimately strengthen the
evaluation process and contribute to more robust and reliable performance
assessments of LVLMs.

## References

  * [1] OpenAI. (n.d.). Dall-E 3. https://openai.com/index/dall-e-3/
  * [2] Stability AI image models. Stability AI. (n.d.). https://stability.ai/stable-image
  * [3] Midjourney. (n.d.). Midjourney. https://www.midjourney.com/
  * [4] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/cvpr.2016.90
  * [5] Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jegou, H. (2021, July 1). Training data-efficient image transformers & distillation through attention. PMLR. https://proceedings.mlr.press/v139/touvron21a.html
  * [6] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021). Swin Transformer: Hierarchical vision transformer using shifted windows. 2021 IEEE/CVF International Conference on Computer Vision (ICCV). https://doi.org/10.1109/iccv48922.2021.00986
  * [7] Zhu, M., Chen, H., Yan, Q., Huang, X., Lin, G., Li, W., Tu, Z., Hu, H., Hu, J., & Wang, Y. (2023, June 24). Genimage: A million-scale benchmark for detecting AI-generated image. arXiv.org. https://arxiv.org/abs/2306.08571
  * [8] Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., & Wang, L. (2023, October 24). MM-vet: Evaluating large multimodal models for integrated capabilities. arXiv.org. https://arxiv.org/abs/2308.02490
  * [9] Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., Manocha, D., & Zhou, T. (2023, November 28). Hallusionbench: An advanced diagnostic suite for entangled language hallucination & visual illusion in large vision-language models. arXiv.org. https://arxiv.org/abs/2310.14566
  * [10] Hessel, J., Marasović, A., Hwang, J. D., Lee, L., Da, J., Zellers, R., Mankoff, R., & Choi, Y. (2023, July 6). Do androids laugh at Electric Sheep? humor “understanding” benchmarks from the New Yorker Caption Contest. arXiv.org. https://arxiv.org/abs/2209.06293
  * [11] Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang, W., … Chen, W. (2023, December 21). MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. arXiv.org. https://arxiv.org/abs/2311.16502
  * [12] Bitton, Y., Bansal, H., Hessel, J., Shao, R., Zhu, W., Awadalla, A., Gardner, J., Taori, R., & Schmidt, L. (2023, December 26). VisIT-Bench: A benchmark for vision-language instruction following inspired by real-world use. arXiv.org. https://arxiv.org/abs/2308.06595
  * [13] Bitton, Y., Yosef, R., Strugo, E., Shahaf, D., Schwartz, R., & Stanovsky, G. (2022, December 8). VASR: Visual analogies of situation recognition. arXiv.org. https://arxiv.org/abs/2212.04542
  * [14] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., Salimans, T., Ho, J., Fleet, D. J., & Norouzi, M. (2022, May 23). Photorealistic text-to-image diffusion models with deep language understanding. arXiv.org. https://arxiv.org/abs/2205.11487
  * [15] Feng, W., He, X., Fu, T.-J., Jampani, V., Akula, A., Narayana, P., Basu, S., Wang, X. E., & Wang, W. Y. (2023, February 28). Training-free structured diffusion guidance for compositional text-to-image synthesis. arXiv.org. https://arxiv.org/abs/2212.05032
  * [16] Bordes, F., Shekhar, S., Ibrahim, M., Bouchacourt, D., Vincent, P., & Morcos, A. S. (2023, December 13). Pug: Photorealistic and semantically controllable synthetic data for representation learning. arXiv.org. https://arxiv.org/abs/2308.03977


[a]Pia Bredt

# Automated NLO SM corrections for all colliders

Jürgen Reuter Pascal Stienemeier

###### Abstract

We summarize the status of automated NLO SM corrections for hadron and lepton
collider processes in the multi-purpose event generator WHIZARD. The focus
will be on NLO EW and QCD-EW mixed corrections at the LHC. Also, recent
progress on the inclusion of EW corrections in future lepton collider
processes and on the POWHEG-matched event generation in the NLO automated
setup will be discussed.

DESY-22-151

##  1 Introduction – Automated NLO corrections in WHIZARD

Monte-Carlo event generators in general are indispensable tools for simulating
physics in a universal manner with a predictive power at the level of
exclusive data. WHIZARD [1] is a multi-purpose generator for cross sections,
distributions and simulated event samples for lepton and hadron collider
processes covering SM and BSM physics. It provides an intrinsic tree-level
matrix element generator O’Mega [2] and a phase-space integrator with MPI-
based parallelization capabilities VAMP2 [3]. This automated framework
recently has been extended to the stage of accounting for complete
perturbative NLO corrections in the full SM, i. e. NLO QCD, EW and mixed
corrections. The regularization of infrared singularities is based on the FKS
subtraction scheme where one-loop virtual matrix elements are accessed by
generic interfaces to one-loop providers such as OpenLoops [4], RECOLA [5] and
GoSam [6]. The matching of NLO QCD event generation to parton showers in
WHIZARD happens via the POWHEG matching scheme.

We defer a detailed documentation of the complete NLO module, involving the
extension of the NLO QCD automated framework [7, 8], to a separate
publication. We summarize recent developments towards the NLO EW automated
setup as well as the POWHEG matching below.

##  2 NLO EW and SM mixed corrections at the LHC

Automated NLO EW corrections for LHC processes in general impose new
computational challenges in theoretical predictions. First of all, photon-
induced processes as well as QED IR-safety criteria, e. g. requiring photon
recombination, play a role already for pure EW processes. Concerning off-shell
vector boson processes of this class the WHIZARD+OpenLoops NLO EW framework is
validated with MG5_aMC@NLO [9]. Results of these checks for a selection of
benchmark processes including neutral- and charged-current processes (with and
without associated Higgs), VBF as well as single top plus jet processes are
listed in Table 1. The input parameters for the checks presented in this
section are chosen according to the setup in ref. [9], particularly,

$\sqrt{s}=13$ TeV,
$\mu_{R}=\mu_{F}=\frac{H_{T}}{2}=\frac{1}{2}\sum_{i}\sqrt{p_{T,i}^{2}+m_{i}^{2}}$,
$\alpha$ input scheme: $G_{\mu}$ CMS.

process | $\alpha^{n}$ |  MG5_aMC@NLO $\sigma_{\text{NLO}}^{\text{tot}}$ [pb] |  WHIZARD $\sigma_{\text{NLO}}^{\text{tot}}$ [pb] |  $\delta$ [%] | $\sigma^{\text{sig}}_{\text{NLO}}$
---|---|---|---|---|---
$pp\rightarrow$ |  |  | +OpenLoops |  |
$e^{+}\nu_{e}\mu^{-}\bar{\nu}_{\mu}$ | $\alpha^{4}$ | $0.52794(9)$ | $0.52816(9)$ | $+3.69$ | $1.69$
$e^{+}e^{-}\mu^{+}\mu^{-}$ | $\alpha^{4}$ | $0.012083(3)$ | $0.012078(3)$ | $-5.25$ | $1.26$
$He^{+}\nu_{e}$ | $\alpha^{3}$ | $0.064740(17)$ | $0.064763(6)$ | $-4.04$ | $1.24$
$He^{+}e^{-}$ | $\alpha^{3}$ | $0.013699(2)$ | $0.013699(1)$ | $-5.86$ | $0.32$
$Hjj$ | $\alpha^{3}$ | $2.7058(4)$ | $2.7056(6)$ | $-4.23$ | $0.27$
$tj$ | $\alpha^{2}$ | $105.40(1)$ | $105.38(1)$ | $-0.72$ | $0.74$
Table 1: Checks for a set of pure EW off-shell vector boson processes at the
LHC with
$\delta\equiv\sigma^{\text{tot}}_{\text{NLO}}/\sigma^{\text{tot}}_{\text{LO}}-1$

Furthermore, for processes with on-shell bosons $VV$, $VH$, $VVV$, $VVH$ and
$VHH$ with $V=W^{\pm},Z$ agreeing NLO EW cross section results with those of
MUNICH/MATRIX [10, 11] using OpenLoops could be achieved for relative MC
errors well-below sub-per-mille level.

For processes at $\mathcal{O}(\alpha_{s})$ and higher NLO fixed order
contributions possibly contain overlapping QCD-EW corrections demanding for
the subtraction of both, QED and QCD IR singularities. More precisely, both
correction types of possible IR splittings have to be considered in the
counterterms for all NLO contributions at fixed coupling powers except for
those at orders leading in $\alpha_{s}$ or $\alpha$.

|  |  $\sigma^{\text{tot}}$ [fb] | $\sigma^{\text{sig}}$ | rel. deviation
---|---|---|---|---
$pp\rightarrow t\bar{t}H$ | $\alpha_{s}^{m}\alpha^{n}$ | MUNICH+OpenLoops | WHIZARD+OpenLoops |  |
$\text{LO}_{21}$ | $\alpha_{s}^{2}\alpha$ |  $3.44865(1)\cdot 10^{2}$ | $3.4487(1)\cdot 10^{2}$ | $0.76$ | $0.003\%$
$\text{LO}_{12}$ | $\alpha_{s}\alpha^{2}$ |  $1.40208(2)\cdot 10^{0}$ | $1.4022(1)\cdot 10^{0}$ | $1.44$ | $0.011\%$
$\text{LO}_{03}$ | $\alpha^{3}$ |  $2.42709(1)\cdot 10^{0}$ | $2.4274(2)\cdot 10^{0}$ | $2.07$ | $0.011\%$
$\text{NLO}_{31}$ | $\alpha_{s}^{3}\alpha$ | $9.9656(4)\cdot 10^{1}$ | $9.968(4)\cdot 10^{1}$ | $0.62$ | $0.023\%$
$\text{NLO}_{22}$ | $\alpha_{s}^{2}\alpha^{2}$ | $6.209(1)\cdot 10^{0}$ | $6.208(2)\cdot 10^{0}$ | $0.20$ | $0.009\%$
$\text{NLO}_{13}$ | $\alpha_{s}\alpha^{3}$ | $1.7238(2)\cdot 10^{0}$ | $1.7232(5)\cdot 10^{0}$ | $1.24$ | $0.040\%$
$\text{NLO}_{04}$ | $\alpha^{4}$ | $1.5053(3)\cdot 10^{-1}$ | $1.5060(7)\cdot 10^{-1}$ | $1.00$ | $0.048\%$
Table 2: Comparison of all fixed order LO and NLO contributions to the cross
section of $pp\to t\bar{t}H$

By appropriate modifications of the FKS automated framework in WHIZARD the
contributions of the whole tower of coupling powers can be computed. By
interfacing OpenLoops process libraries the validation of all $\alpha_{s}$
leading and subleading NLO contributions of $pp\rightarrow
t\bar{t}(H/Z/W^{\pm})$ has been performed with reference results of
MUNICH/MATRIX. Results for $pp\to t\bar{t}H$ are exemplarily shown in Table 2.
Due to IR-safety criteria photons have to be treated on a democratic basis
with gluons in the jet definition. This yields a non-trivial phase-space cut
evaluation including photon recombination and jet clustering for processes
with jets and leptons in the final state. In this context, checks with
MG5_aMC@NLO for $pp\rightarrow e^{+}\nu_{e}j$ and $pp\rightarrow e^{+}e^{-}j$
are performed with results listed in Table 3.

$pp\rightarrow$ | $\alpha_{s}^{m}\alpha^{n}$ |  MG5_aMC@NLO $\sigma_{\text{NLO}}^{\text{tot}}$ [fb] |  WHIZARD+OpenLoops $\sigma_{\text{NLO}}^{\text{tot}}$ [fb] |  $\delta$ [%] | $\sigma^{\text{sig}}_{\text{NLO}}$
---|---|---|---|---|---
$e^{+}\nu_{e}j$ | $\alpha_{s}\alpha^{2}$ | $9.0475(8)\cdot 10^{5}$ | $9.0459(7)\cdot 10^{5}$ | $-1.11$ | $1.5$
$e^{+}e^{-}j$ | $\alpha_{s}\alpha^{2}$ | $1.4909(2)\cdot 10^{5}$ | $1.4908(2)\cdot 10^{5}$ | $-1.00$ | $0.4$
Table 3: Checks for cross sections at NLO EW for representative processes
requiring involved cuts

##  3 Lepton collider processes at NLO EW

Under certain conditions predictions for lepton collision processes at NLO EW
are reliable in a fixed order massive initial state approximation. Using NLO
EW accurate amplitudes from RECOLA this is automated in WHIZARD with adjusted
FKS phase space construction for massive initial-state emitters. This
framework is used in our recent study on NLO EW corrections to multi-boson
production at a future muon collider [12]. For a universal treatment of
collinear ISR effects in NLO calculations NLL accurate electron PDFs [13, 14]
\- implemented and validated in WHIZARD \- have to be applied. Embedding these
into the FKS framework is under development. Numerical pitfalls are posed by
the interplay of FKS ISR construction and the divergent behavior of the PDFs
in the asymptotic $z\to 1$ limit which is being handled by adequate phase-
space mappings.

##  4 POWHEG-matched and showered NLO event generation

The work on the POWHEG matching in WHIZARD started with earlier studies on
$e^{+}e^{-}\rightarrow t\bar{t}(H)$ precision calculations [15]. Aiming for a
generalization of the POWHEG matching to arbitrary processes the
implementation now has been extended to $pp$ processes with validation
completed for Drell-Yan and similar processes. Comparisons of $p_{T,e^{-}}$,
$m_{e^{+}e^{-}}$ and $y_{e^{-}}$ distributions for $pp\rightarrow e^{+}e^{-}$
with POWHEG-matched events from WHIZARD and POWHEG-BOX [16] and showered with
PYTHIA [17] are shown in Fig. 1.

Figure 1: Distributions at NLO QCD to $pp\rightarrow e^{+}e^{-}$ for the
validation of matched and showered events

##  5 Outlook

Future plans include the application of NLO-NLL electron PDFs in lepton
collision observables at NLO as well as efficiency improvements on the phase-
space generation for NLO EW calculations to high multiplicity processes at the
LHC. Furthermore, the automated POWHEG matching is supposed to be extended
accounting for NLO corrections in the full SM.

## Acknowledgements

This work was funded by the Deutsche Forschungsgemeinschaft under Germany’s
Excellence Strategy – EXC 2121 ”Quantum Universe” - 390833306. For cross-
checks with MUNICH and numerous advices on automated NLO EW corrections PB
wants to thank Stefan Kallweit.

## References

  * [1] W. Kilian, _et. al._ Eur. Phys. J. C 71 (2011), 1742 arXiv: 0708.4233 [hep-ph].
  * [2] M. Moretti, _et. al._ arXiv: hep-ph/0102195 [hep-ph].
  * [3] S. Brass, _et. al._ Eur. Phys. J. C 79 (2019) no.4, 344 arXiv: 1811.09711 [hep-ph].
  * [4] F. Buccioni, _et. al._ Eur. Phys. J. C 79 (2019) no.10, 866 arXiv: 1907.13071 [hep-ph].
  * [5] S. Actis, _et. al._ JHEP 04 (2013), 037 arXiv: 1211.6316 [hep-ph].
  * [6] G. Cullen, _et al._ Eur. Phys. J. C 74 (2014) no.8, 3001 arXiv: 1404.7096 [hep-ph].
  * [7] F. Bach, _et. al._ JHEP 03 (2018), 184 arXiv: 1712.02220 [hep-ph].
  * [8] B. Chokoufé Nejad, _et. al._ JHEP 12 (2016), 075 [arXiv:1609.03390 [hep-ph]].
  * [9] R. Frederix, _et. al._ JHEP 07 (2018), 185 arXiv: 1804.10017 [hep-ph].
  * [10] S. Kallweit, _et. al._ JHEP 04 (2015), 012 arXiv: 1412.5157 [hep-ph].
  * [11] L. Buonocore, _et. al._ Phys. Rev. D 103 (2021), 114012 arXiv: 2102.12539 [hep-ph].
  * [12] P. Bredt, _et. al._ arXiv: 2208.09438 [hep-ph].
  * [13] S. Frixione, JHEP 11 (2019), 158 arXiv: 1909.03886 [hep-ph].
  * [14] V. Bertone, _et. al._ JHEP 03 (2020), 135 arXiv: 1911.12040 [hep-ph].
  * [15] B. Chokoufe Nejad, _et. al._ PoS EPS-HEP2015 (2015), 317 arXiv: 1510.02739 [hep-ph].
  * [16] S. Alioli, _et. al._ JHEP 07 (2008), 060 arXiv: 0805.4802 [hep-ph].
  * [17] T. Sjöstrand, _et. al._ Comput. Phys. Commun. 191 (2015), 159-177 arXiv: 1410.3012 [hep-ph].


Log(hours) | -0.18*** | -0.19*** | -0.21*** | -0.19*** | -0.23*** | -0.25*** | -0.19*** | -0.12*** | -0.23*** | -0.28*** | -0.20*** | -0.27*** | -0.21*** | -0.11*** | -0.12***
| (0.020) | (0.020) | (0.019) | (0.022) | (0.025) | (0.022) | (0.021) | (0.024) | (0.027) | (0.021) | (0.024) | (0.026) | (0.024) | (0.022) | (0.026)
Log(hours)2 | 0.18*** | 0.19*** | 0.22*** | 0.17*** | 0.23*** | 0.25*** | 0.19*** | 0.12*** | 0.23*** | 0.28*** | 0.20*** | 0.27*** | 0.22*** | 0.10*** | 0.12***
| (0.022) | (0.022) | (0.022) | (0.024) | (0.028) | (0.025) | (0.024) | (0.027) | (0.030) | (0.023) | (0.026) | (0.028) | (0.026) | (0.024) | (0.028)
EEA | -0.00 | 0.00 | -0.00 | -0.00 | 0.00 | 0.00 | 0.00* | 0.00** | 0.00 | 0.00*** | 0.00 | 0.00*** | 0.00** | 0.00** | 0.00**
| (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001)
Non-EEA | -0.00 | -0.00 | -0.00 | -0.00 | -0.00 | 0.00 | -0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
| (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000)
Age | 0.02 | 0.13*** | 0.08** | 0.09*** | 0.02 | -0.00 | 0.05** | 0.03 | 0.04* | 0.05*** | 0.06*** | -0.01 | 0.03* | 0.05*** | 0.05**
| (0.032) | (0.031) | (0.032) | (0.030) | (0.023) | (0.018) | (0.022) | (0.022) | (0.020) | (0.019) | (0.018) | (0.015) | (0.015) | (0.018) | (0.024)
Experience | 0.07*** | 0.00 | 0.03 | 0.01 | 0.03* | 0.04*** | 0.02 | 0.04** | 0.03** | 0.01 | 0.02 | 0.05*** | 0.02* | 0.02* | 0.01
| (0.022) | (0.021) | (0.022) | (0.020) | (0.016) | (0.014) | (0.016) | (0.015) | (0.015) | (0.013) | (0.012) | (0.013) | (0.011) | (0.012) | (0.016)
Years educ | 0.03*** | 0.02** | 0.02** | 0.00 | 0.02** | 0.02*** | 0.01* | -0.00 | 0.01 | -0.00 | 0.00 | -0.00 | -0.01 | -0.02** | -0.01**
| (0.007) | (0.007) | (0.007) | (0.007) | (0.007) | (0.008) | (0.006) | (0.008) | (0.006) | (0.007) | (0.006) | (0.005) | (0.006) | (0.008) | (0.007)
Training | 0.00** | 0.00* | 0.00** | 0.00*** | 0.00** | 0.00 | -0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | -0.00 | -0.00 | -0.00*
| (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.001)
In couple | -0.00 | -0.00 | -0.00** | -0.00* | 0.00 | 0.00 | -0.00 | 0.00 | -0.00 | 0.00 | 0.00* | 0.00** | 0.00 | 0.00 | 0.00
| (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001)
With dep. children | 0.00** | 0.00 | 0.00 | 0.00** | 0.00 | 0.00 | 0.00 | 0.00* | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | -0.00
| (0.001) | (0.001) | (0.000) | (0.001) | (0.001) | (0.000) | (0.001) | (0.001) | (0.001) | (0.000) | (0.000) | (0.000) | (0.000) | (0.001) | (0.000)
In couple with dep. children | 0.00** | 0.00* | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00* | 0.01*** | 0.00* | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
| (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.002) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001)
Public sector | 0.00 | 0.00 | -0.00 | -0.00 | -0.00 | -0.00 | -0.00 | -0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00** | -0.00
| (0.000) | (0.000) | (0.000) | (0.000) | (0.001) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.001) | (0.000)
_Coefficient effect_ |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
Low gender sect. segregation | 0.01 | 0.00 | 0.01 | 0.00 | 0.03 | -0.02 | 0.02 | -0.02 | -0.03 | -0.03 | -0.04 | -0.03 | 0.00 | 0.02 | 0.06**
| (0.023) | (0.020) | (0.022) | (0.026) | (0.028) | (0.029) | (0.024) | (0.023) | (0.022) | (0.023) | (0.028) | (0.023) | (0.025) | (0.024) | (0.028)
Log(hours) | 2.98*** | 2.83*** | 3.09*** | 3.68*** | 1.94*** | 1.93*** | 1.66*** | 1.27*** | 1.84*** | 4.47*** | 2.33*** | 2.99*** | 2.43*** | 2.90*** | 1.17**
| (0.395) | (0.361) | (0.368) | (0.392) | (0.427) | (0.342) | (0.402) | (0.387) | (0.435) | (0.458) | (0.396) | (0.428) | (0.476) | (0.463) | (0.546)
Log(hours)2 | -1.72*** | -1.68*** | -1.86*** | -2.00*** | -1.31*** | -1.35*** | -1.17*** | -1.00*** | -1.22*** | -2.54*** | -1.48*** | -1.84*** | -1.60*** | -1.65*** | -1.04***
| (0.204) | (0.193) | (0.198) | (0.210) | (0.226) | (0.187) | (0.215) | (0.212) | (0.233) | (0.238) | (0.215) | (0.231) | (0.251) | (0.245) | (0.300)
EEA | -0.00 | -0.00 | 0.00 | -0.00 | 0.00 | -0.00 | 0.00 | 0.00 | -0.00 | 0.00 | -0.00 | 0.00 | -0.00 | -0.00 | 0.00
| (0.001) | (0.002) | (0.002) | (0.002) | (0.002) | (0.002) | (0.002) | (0.003) | (0.003) | (0.003) | (0.003) | (0.003) | (0.003) | (0.003) | (0.003)
Non-EEA | 0.00 | -0.00 | 0.00 | -0.00 | -0.00 | -0.01** | -0.00 | -0.00 | 0.00 | -0.00 | 0.00 | 0.00 | -0.00 | -0.00 | 0.00
| (0.002) | (0.002) | (0.002) | (0.002) | (0.002) | (0.002) | (0.003) | (0.003) | (0.003) | (0.003) | (0.003) | (0.003) | (0.003) | (0.003) | (0.003)
Age | -0.09 | -0.68 | -0.71 | -0.14 | 0.09 | 0.31 | -0.58 | 0.08 | -0.58 | -0.82 | -0.87 | 0.80 | -0.56 | -0.83 | 0.11
| (0.516) | (0.462) | (0.491) | (0.541) | (0.578) | (0.539) | (0.560) | (0.569) | (0.583) | (0.561) | (0.546) | (0.549) | (0.560) | (0.561) | (0.671)
Experience | -0.03 | 0.18 | 0.18 | 0.02 | -0.04 | -0.03 | 0.17 | -0.10 | 0.10 | 0.28 | 0.10 | -0.30 | 0.10 | 0.02 | 0.10
| (0.187) | (0.164) | (0.177) | (0.195) | (0.217) | (0.199) | (0.204) | (0.199) | (0.211) | (0.193) | (0.185) | (0.185) | (0.187) | (0.187) | (0.226)
Years educ | -0.02 | -0.08 | 0.13 | 0.19 | 0.20 | 0.59 | 0.84** | 0.34 | 1.04** | 0.68 | 0.89** | 1.14*** | 0.62 | 0.67 | 1.26**
| (0.318) | (0.316) | (0.332) | (0.358) | (0.394) | (0.382) | (0.402) | (0.415) | (0.429) | (0.429) | (0.435) | (0.436) | (0.459) | (0.477) | (0.539)
Training | -0.01 | -0.00 | 0.01 | 0.00 | 0.00 | -0.00 | 0.00 | 0.00 | 0.00 | -0.00 | 0.00 | 0.00 | 0.00 | 0.00 | -0.00
| (0.008) | (0.007) | (0.008) | (0.008) | (0.008) | (0.004) | (0.004) | (0.003) | (0.003) | (0.003) | (0.003) | (0.003) | (0.003) | (0.003) | (0.003)
In couple | 0.04*** | 0.02** | 0.05*** | 0.05*** | 0.01 | 0.02 | 0.05*** | 0.02* | 0.04*** | 0.04*** | 0.02 | 0.02 | 0.04*** | 0.02** | 0.02*
| (0.010) | (0.010) | (0.010) | (0.011) | (0.012) | (0.011) | (0.012) | (0.012) | (0.012) | (0.012) | (0.013) | (0.012) | (0.012) | (0.012) | (0.013)
With dep. children | 0.05*** | 0.04*** | 0.05*** | 0.05*** | 0.04*** | 0.02** | 0.05*** | 0.04*** | 0.06*** | 0.03** | 0.04*** | 0.02* | 0.03*** | 0.04*** | 0.02*
| (0.010) | (0.010) | (0.010) | (0.011) | (0.012) | (0.011) | (0.012) | (0.012) | (0.012) | (0.012) | (0.012) | (0.012) | (0.012) | (0.013) | (0.013)
In couple with dep. children | -0.01 | -0.01 | -0.01 | -0.01 | -0.00 | 0.00 | -0.03*** | -0.01 | -0.03*** | -0.02** | -0.01 | -0.00 | -0.01 | -0.02 | -0.01
| (0.009) | (0.008) | (0.009) | (0.010) | (0.010) | (0.009) | (0.010) | (0.010) | (0.010) | (0.010) | (0.011) | (0.010) | (0.010) | (0.011) | (0.011)
Public sector | -0.00 | 0.00 | -0.00 | 0.00 | -0.00 | -0.00 | -0.00 | -0.01** | -0.00 | -0.00 | -0.00 | 0.00 | 0.00 | 0.00 | -0.00
| (0.002) | (0.002) | (0.002) | (0.002) | (0.003) | (0.002) | (0.002) | (0.002) | (0.002) | (0.002) | (0.002) | (0.002) | (0.002) | (0.002) | (0.002)
_Interaction effect_ |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
Low gender sect. segregation | -0.01 | -0.00 | -0.00 | -0.00 | -0.01 | 0.01 | -0.01 | 0.01 | 0.01 | 0.02 | 0.02 | 0.01 | -0.00 | -0.01 | -0.02**
| (0.012) | (0.010) | (0.011) | (0.014) | (0.013) | (0.015) | (0.012) | (0.012) | (0.010) | (0.010) | (0.012) | (0.010) | (0.011) | (0.011) | (0.012)
Log(hours) | 0.27*** | 0.24*** | 0.26*** | 0.32*** | 0.19*** | 0.18*** | 0.14*** | 0.11*** | 0.16*** | 0.39*** | 0.20*** | 0.26*** | 0.20*** | 0.23*** | 0.08**
| (0.036) | (0.031) | (0.032) | (0.035) | (0.041) | (0.033) | (0.035) | (0.034) | (0.039) | (0.041) | (0.035) | (0.037) | (0.040) | (0.037) | (0.036)
Log(hours)2 | -0.30*** | -0.27*** | -0.30*** | -0.33*** | -0.24*** | -0.25*** | -0.19*** | -0.17*** | -0.21*** | -0.42*** | -0.25*** | -0.30*** | -0.26*** | -0.25*** | -0.13***
| (0.036) | (0.032) | (0.033) | (0.036) | (0.042) | (0.034) | (0.036) | (0.036) | (0.040) | (0.040) | (0.036) | (0.039) | (0.041) | (0.038) | (0.039)
EEA | 0.00 | 0.00 | -0.00 | 0.00 | -0.00 | 0.00 | -0.00 | -0.00 | 0.00 | -0.00 | 0.00 | -0.00 | 0.00 | 0.00 | -0.00
| (0.000) | (0.000) | (0.001) | (0.001) | (0.000) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001)
Non-EEA | -0.00 | 0.00 | -0.00 | 0.00 | 0.00 | -0.00 | 0.00 | -0.00 | -0.00 | 0.00 | -0.00 | -0.00 | 0.00 | 0.00 | -0.00
| (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.001)
Age | -0.01 | -0.05 | -0.05 | -0.01 | 0.00 | 0.01 | -0.03 | 0.00 | -0.02 | -0.03 | -0.03 | 0.03 | -0.02 | -0.03 | 0.00
| (0.037) | (0.036) | (0.038) | (0.034) | (0.027) | (0.022) | (0.027) | (0.026) | (0.024) | (0.023) | (0.022) | (0.019) | (0.018) | (0.021) | (0.029)
Experience | -0.00 | 0.03 | 0.03 | 0.00 | -0.00 | -0.00 | 0.02 | -0.01 | 0.01 | 0.02 | 0.01 | -0.02 | 0.01 | 0.00 | 0.01
| (0.026) | (0.026) | (0.027) | (0.024) | (0.020) | (0.015) | (0.020) | (0.019) | (0.018) | (0.016) | (0.016) | (0.015) | (0.014) | (0.016) | (0.021)
Years educ | -0.00 | -0.00 | 0.00 | 0.00 | 0.00 | 0.01 | 0.01 | -0.00 | 0.01 | -0.00 | 0.00 | -0.00 | -0.00 | -0.01 | -0.01*
| (0.006) | (0.003) | (0.003) | (0.001) | (0.005) | (0.005) | (0.005) | (0.001) | (0.005) | (0.003) | (0.004) | (0.004) | (0.004) | (0.006) | (0.008)
Training | -0.00 | -0.00 | 0.00 | 0.00 | 0.00 | -0.00 | -0.00 | 0.00 | 0.00 | -0.00 | 0.00 | 0.00 | -0.00 | -0.00 | 0.00
| (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.000)
In couple | 0.00*** | 0.00** | 0.01*** | 0.00*** | 0.00 | 0.00 | 0.01*** | 0.00 | 0.01*** | 0.00*** | 0.00 | 0.00 | 0.00*** | 0.00* | 0.00*
| (0.001) | (0.001) | (0.002) | (0.001) | (0.001) | (0.001) | (0.002) | (0.001) | (0.002) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.002)
With dep. children | -0.00** | -0.00 | -0.00 | -0.00** | -0.00 | -0.00 | -0.00 | -0.00** | -0.00 | -0.00 | -0.00 | -0.00 | -0.00 | -0.00* | 0.00
| (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001)
In couple with dep. children | -0.00 | -0.00 | -0.00 | -0.00 | -0.00 | 0.00 | -0.00*** | -0.00 | -0.01*** | -0.00* | -0.00 | -0.00 | -0.00 | -0.00 | -0.00
| (0.001) | (0.001) | (0.001) | (0.001) | (0.002) | (0.001) | (0.002) | (0.001) | (0.002) | (0.001) | (0.001) | (0.001) | (0.001) | (0.001) | (0.002)
Public sector | -0.00 | -0.00 | 0.00 | -0.00 | 0.00 | 0.00 | 0.00 | 0.00** | 0.00 | 0.00 | 0.00 | -0.00 | -0.00 | -0.00 | 0.00
| (0.000) | (0.000) | (0.000) | (0.000) | (0.001) | (0.000) | (0.000) | (0.001) | (0.001) | (0.001) | (0.001) | (0.000) | (0.001) | (0.001) | (0.000)
Observations | 17046 | 17745 | 16772 | 15574 | 13410 | 13840 | 13204 | 13218 | 13606 | 12994 | 12447 | 12874 | 12085 | 11588 | 10995

  * _Notes: Contribution of main socio-demographic characteristics, human capital attributes and sectoral indicators. Occupations (SOC) are omitted for layout reasons. Significance levels: pvalue <0.01 ***, pvalue<0.05 **, pvalue<0.1 *._

##  Appendix B Additional material: Figures

Figure B.1: Shift-share decomposition of employment, male sample

_Note: The graphs display the shift-share decomposition of male employment
(from Equation 3 by replacing female figures with male’s). The difference in
employment in the comparison year with respect to the base year (i.e., the
fiscal year 2005). The composition of male employment remained unchanged in
female-dominated sectors. As expected, the 2007-2009 crisis hit male-dominated
sectors more than female-dominated sectors. After the EA2020, male composition
in male-dominated sectors decreased and remained stable in female-dominated
for 2005. The Covid–19 outbreak did not arrest total male employment with
respect to previous years._

Figure B.2: KBO decomposition, pooled sample

_Note : The figure shows the evolution of the components of the KBO
decomposition and their sum over time for full sample. Women are contrasted to
men. The dashed line represents the _coefficient effect_ , the long-dashed
line the _endowment effect_ and the dotted line the part of the “unexplained”
component of the three-fold decomposition (or _interaction effect_). The
corresponding shadowed areas display the 95% confidence intervals. The solid
line is the sum of the three effects and reveals their overall contribution.
The wage difference between men and women is, on average, 0.2 logarithmic
points over time. Most of the gender pay gap (around three-fourths) can be
explained by differences in the estimated coefficients between genders. The
_coefficient effect_ on average quantifies indeed an increase of 0.16
logarithmic points in women’s wages when the male coefficients are applied to
female characteristics. In addition, this component displays a downward trend
after 2008. The _endowment effect_ would quantify an expected average increase
in women’s wage by around 0.05 points if they had male predictors levels.
Therefore, differences in observed characteristics account for one-fourth of
the gap.
Estimation note: Both models for women and men are estimated using the
Mincerian regression equation (with OLS). The shaded areas is the 95%
confidence intervals._