11institutetext: Huazhong University of Science and Technology
11email: {wenwenyu<EMAIL_ADDRESS>
22institutetext: Baidu Inc.
22email<EMAIL_ADDRESS>33institutetext: Tencent YouTu Lab
33email: {rechycao, huaangchen<EMAIL_ADDRESS>44institutetext:
Universitat Autónoma de Barcelona 55institutetext: Peking
University66institutetext: Harbin Institute of Technology77institutetext: CAS
Institute of Automation88institutetext: University of
Rochester99institutetext: Sea AI Lab

# ICDAR 2023 Competition on Structured Text Extraction from Visually-Rich
Document Images

Wenwen Yu Contributed equally.11 Chengquan Zhang⋆ 22 Haoyu Cao⋆ 33 Wei Hua 11
Bohan Li 22 Huang Chen 33 Mingyu Liu 11 Mingrui Chen 11 Jianfeng Kuang 11
Mengjun Cheng 55 Yuning Du 22 Shikun Feng 22 Xiaoguang Hu 22 Pengyuan Lyu 22
Kun Yao 22 Yuechen Yu 22 Yuliang Liu 11 Wanxiang Che 66 Errui Ding 22 Cheng-
Lin Liu 77 Jiebo Luo 88 Shuicheng Yan 99 Min Zhang 66 Dimosthenis Karatzas 44
Xing Sun 33 Jingdong Wang 22 Xiang Bai(✉) 11

###### Abstract

Structured text extraction is one of the most valuable and challenging
application directions in the field of Document AI. However, the scenarios of
past benchmarks are limited, and the corresponding evaluation protocols
usually focus on the submodules of the structured text extraction scheme. In
order to eliminate these problems, we organized the ICDAR 2023 competition on
Structured text extraction from Visually-Rich Document images (SVRD). We set
up two tracks for SVRD including Track 1: HUST-CELL and Track 2: Baidu-FEST,
where HUST-CELL aims to evaluate the end-to-end performance of Complex Entity
Linking and Labeling, and Baidu-FEST focuses on evaluating the performance and
generalization of Zero-shot / Few-shot Structured Text extraction from an end-
to-end perspective. Compared to the current document benchmarks, our two
tracks of competition benchmark enriches the scenarios greatly and contains
more than 50 types of visually-rich document images (mainly from the actual
enterprise applications). The competition opened on 30th December, 2022 and
closed on 24th March, 2023. There are 35 participants and 91 valid submissions
received for Track 1, and 15 participants and 26 valid submissions received
for Track 2. In this report we will presents the motivation, competition
datasets, task definition, evaluation protocol, and submission summaries.
According to the performance of the submissions, we believe there is still a
large gap on the expected information extraction performance for complex and
zero-shot scenarios. It is hoped that this competition will attract many
researchers in the field of CV and NLP, and bring some new thoughts to the
field of Document AI.

##  1 Introduction

In recent years, the domain of Document AI has gradually become a hot research
topic. As one of the most concerned Document AI technologies, structured text
extraction aims to capture text fields with specified semantic attributes from
complex visually-rich documents (VRDs). It is widely used in many applications
and services, such as customs information inspection, accounting in the
financial field, office automation, and so on.

In the past, several benchmarks have been established in the community, such
as FUNSD [7], CORD [12], XFUND [16], EPHOIE [14], etc., to measure relative
technical efforts. However, there are many imperfections in these benchmarks
and the corresponding evaluation protocols. The obvious shortcomings are as
follows: 1) The performance of structured text extraction is not evaluated
from the end-to-end perspective, but is disassembled into three independent
functional modules, namely text detection [11, 22, 19], text recognition [13,
17, 4] and entity labelling or linking) [15, 1, 10, 18], for respective
evaluation, which is not intuitive for downstream applications. 2) The
scenarios covered by the above benchmarks are relatively few, and only focus
on a certain receipt or form scenario, which is difficult to guide objectively
evaluate the effectiveness and robustness of the model.

Therefore, we propose a new structured text extraction competition benchmark
including two tracks, which covers the most abundant visually-rich document
images of scenarios and types as far as we know. The whole benchmark will
contain more than 50 document types and more than 100 semantic attributes of
text fields. In order to evaluate the performance of structured information
extraction from an end-to-end perspective, we will set up two tracks: (1) end-
to-end Complex Entity Linking and Labeling created by Huanzhong University of
Science and Technology (HUST-CELL), where we have designed entity linking and
labeling tasks. (2) end-to-end Few-shot Structured Text extraction created by
Baidu Company (Baidu-FEST) to explore the generalization and robustness of the
submitted models, where we have newly designed zero-shot and few-shot
structured text extraction tasks. The motivation and relevance to ICDAR
community including:

  * •

The intelligent analysis of visually-rich document images has always been an
important domain of concern for ICDAR community. Its core technologies include
text detection, recognition and named-entity recognition. Our proposed
competition is the first time to guide the evaluation of the effect and
generalization of structured text extraction scheme from an end-to-end
perspective, which is more valuable but more challenging.

  * •

This competition aims to further connect researchers from both the document
image understanding and NLP communities to bring more inspiration.

###  1.1 Competition Organization

ICDAR 2023 competition on SVRD is organized by a joint team, including
Huazhong University of Science and Technology, Baidu Inc., Tencent YouTu Lab,
Universitat Autónoma de Barcelona, Peking University, Harbin Institute of
Technology, CAS Institute of Automation, University of Rochester, and Sea AI
Lab.

We organize the SVRD competition on the Robust Reading Competition (RRC)
website 111https://rrc.cvc.uab.es/?ch=21, where provide corresponding download
links of the datasets, and user interfaces for participants and submission
page for their results 222Scores achieved using the ChatGPT large model
interface during the competition are temporarily excluded from the
leaderboard.. Great support has been received from the RRC web team.

##  2 Related Works

This section discusses most of the well-received visually-rich document
benchmarks as following:

In 2019, the Robust Reading Competition (RRC) web portal introduced a new
challenge, known as Scanned Receipts OCR and Information Extraction reading
competition which also commonly named as SROIE [6]. The main feature of SROIE
is that all the images are collected from scanned receipts. It contains of 626
receipts for training and 347 receipts for testing, and each receipt only
contains four predefined values: company, date, address, and total.

Meanwhile, EATEN [5] constructs a dataset of 1,900 real images in train ticket
scenarios, and it only has entity values annotation without OCR annotation.
CORD [12] consists of 1,000 Indonesian receipts, which contains images and
box/text annotations for OCR, and multi-level semantic labels for parsing,
which can be used to address various OCR and entity extraction tasks.

FUNSD [7] dataset was presented at the ICDAR workshop in 2019. FUNSD is a form
understanding benchmark with 199 real, fully annotated, scanned form images,
such as marketing, advertising, and scientific reports, which is split into
149 training samples and 50 testing samples. FUNSD dataset is suitable for a
variety of tasks including text detection, recognition, entity labelling, etc.
And its entity has only three types of semantic attributes including question,
answer and header.

XFUND [16] is an extended version of FUNSD that was proposed in 2021. It is
launched to introduce the multi-script structured text extraction problem. It
consists of human-labeled forms with key-value pairs in 7 languages (Chinese,
Japanese, Spanish, French, Italian, German, Portuguese). Each language
includes 199 forms, where the training set includes 149 forms, and the test
set includes 50 forms.

EPHOIE [14] was also released in 2021, and contains 1,494 images which are
collected and scanned from real examination papers of various schools in
China. There are 10 text entity types including Subject, Test Time, Name,
School, Examination Number, Seat Number, Class, Student Number, Grade, and
Score. As an overview, Table 1 lists the details of above datasets.

Table 1: Existing Visually-rich Document Images Datasets.

Dataset | Image number(Train/Test) | Language | Granularity | Document type | Year
---|---|---|---|---|---
SROIE | 973(626/347) | English | Word | Receipt | 2019
CORD | 1,000(800/200) | English | Word | Receipt | 2019
EATEN | 1,900(1,500/400) | Chinese | w/o OCR | Train Ticket | 2019
FUNSD | 199(149/50) | English | Word | Form | 2019
XFUND | 1,393(1,043/350) | 7 languages | Word/Line | Form | 2021
EPHOIE | 1,494(1,183/311) | Chinese | Character | Paper | 2021

##  3 Benchmark Description

###  3.1 Track 1: HUST-CELL

Our proposed HUST-CELL complexity goes over and above previous datasets in
four distinct aspects. First, we provide 30 categories of documents with more
than 4k documents, 2 times larger than the existing English and Chinese
datasets including SROIE (973), CORD (1,000), EATEN (1,900), FUNSD (199),
XFUNSD (1,393), and EPHOIE (1,494). Second, HUST-CELL contains 400+ diverse
keys and values. Third, HUST-CELL covers complex keys more challenging than
others, for instance, nested keys, fine-grained key-value pairs, multi-line
keys/values, long-tailed key-value pairs, as shown in Figure 1. Current state-
of-the-art Key Information Extraction (KIE) techniques [18, 20, 15, 3, 2] fail
to deal with such situations that are essential for a robust KIE system in the
real world. Fourth, our dataset comprises real-world documents reflecting
real-life diversity of content and the complexity of the background, e.g.
different fonts, noise, blur, seal.

In this regard, under the consideration of the importance and huge application
value of KIE, we propose to set up track 1 competition on complex entity
linking and labeling.

Our proposed HUST-CELL were collected from public websites and cover a variety
of scenarios, e.g., receipt, certificate, and license of various industries.
The language of the documents is mainly Chinese, along with a small portion of
English. The number of images collected for each specific scenario varies,
ranging from 10 to 300, with a long-tail distribution, which can avoid
introducing any bias towards specific real application scenarios. Due to the
complexity of the data source, the diversity of this dataset can be
guaranteed. To be able to use publicly, this data is collected from open
websites, and we delete images that contain private information for privacy
protection. Some examples are shown in Figure 1.

Figure 1: Samples of HUST-CELL collected from various scenarios.

The dataset is split into the training set and test set. The training set
consists of 2,000 images, which will be available to the participants along
with OCR and KIE annotations. The test set consists of 2,000 images, whose OCR
annotations and KIE annotations will not be released, but the online
evaluation server 333https://rrc.cvc.uab.es/?ch=21&com=mymethods&task=1 will
remain available for future usage of this benchmark.

###  3.2 Track 2: Baidu-FEST

Our proposed Baidu-FEST benchmark comes from the practical scenarios, mainly
including finance, insurance, logistics, customs inspection, and other fields.
Different applications have different requirements for text fields of
interest. In addition, the data collection methods in different scenarios may
be affected by different cameras and environments, thus the benchmark is
relatively rich and challenging.

Specifically, the benchmark contains about 11 kinds of synthetic business
documents for training, and 10 types of real visually-rich document images for
testing. The format of documents major consists of cards, receipts, and forms.
Each type of document provides about 60 images.

Each image in the dataset is annotated with text-field bounding boxes (bbox)
and the transcript, entity caption, entity id of each text bbox. Locations are
annotated as rectangles with four vertices, which are in clockwise order
starting from the top. Some examples of images and the corresponding
annotations are shown in Figure 2.

Figure 2: Some visually-rich document samples of Baidu-FEST.

##  4 Competition Tasks and Evaluation Protocols

Our proposed competition has two tracks totaling four main tasks.

###  4.1 Track 1: HUST-CELL

####  4.1.1 Task-1: E2E Complex Entity Linking

  * •

Task Description: This task aims to extract key-value pairs (entity linking)
from given images only, then save the key-value pairs of each image into a
JSON file. For the train set, both KIE annotation files for training and
human-checked OCR annotation files are provided. So the OCR annotation is
clean and can be used as the ground truth of the OCR task. The test set of
Task 1 will only provide images without any annotation including OCR and KIE.
It requires the method to accomplish both OCR and KIE tasks in an end-to-end
manner.

####  4.1.2 Task-2: E2E Complex Entity Labeling

  * •

Task Description: The end-to-end complex entity labeling is to extract texts
of a number of predefined key text fields from given images (entity labeling),
and save the texts for each image in a JSON file with required format. Task 2
has 13 predefined entities. For the train set, both KIE annotation files for
training and human-checked OCR annotation files are provided. So the OCR
annotation is clean and can be used as the ground truth of the OCR task. The
test set of Task 2 will only provide images without any annotation including
OCR and KIE. It requires the method to accomplish both OCR and KIE tasks in an
end-to-end manner.

###  4.2 Track 2: Baidu-FEST

####  4.2.1 Task-3: E2E Zero-shot Structured Text Extraction

  * •

Task Description: The zero-shot structured text extraction is to extract texts
of a number of key fields from given images, and save the texts for each image
in a JSON file with required format. Different from Task2, there is no
intersection between the scenarios of the provided training-set and the
scenarios of the provided test-set. Of course, the training data consists of
the real data provided by Track 1 and the synthetic data generated officially.
The caption_en and caption_ch in GT can be used as prompt to assist extraction
but it is not allowed to be modified.

####  4.2.2 Task-4: E2E Few-shot Structured Text Extraction

  * •

Task Description: The few-shot structured text extraction is to extract texts
of a number of key fields from given images, and save the texts for each image
in a JSON file with required format. Different from Task-3, the localization
information and transcript will be provided, but the total number of the
provided training-set will no more than five images for each scenario of the
provided test-set. The caption_en and caption_ch in GT can be used as prompt
to assist extraction but it is not allowed to be modified.

###  4.3 Evaluation Protocol

####  4.3.1 Task 1 Evaluation

For Task 1, the evaluation metrics include two parts:

Normalized edit distance. For each predicted kv-pair (key-value pair), if it
matched with GT kv-pair in the given image, the normalized edit distance (NED)
between the predicted kv-pair s1 and ground-truth kv-pair s2 will be
calculated as following:

$NED(s1,s2)=\left(\frac{ed\left(s1\\_k,s2\\_k\right)}{\max\left(\operatorname{len}\left(s1\\_k\right),\operatorname{len}\left(s2\\_k\right)\right)}+\frac{ed\left(s1\\_v,s2_{\\_}v\right)}{\max\left(\operatorname{len}\left(s1_{-}v\right),\operatorname{len}\left(s2\\_v\right)\right)}\right)/2$
(1)
$\operatorname{score}1=1-\sum_{i=1}^{n}\frac{NED\left(s_{i1},s_{i2}\right)}{n}$
(2)

where n denotes the number of matched kv-pairs (both the edit distance of key
and value are larger than a threshold simultaneously. $ed()$ denotes the edit
distance function. The calculated details refer to the following Matching
Protocol.). s1_k/s2_k, s1_v/s2_v indicate the content of key and value of the
kv-pair s1/s2, respectively. Note that for predicted kv-pairs that do not
matched in the GT of the given image, the edit distance will be calculated
between predicted kv-pairs and empty string.

Matching Protocol: Given the predicted kv-pair s1 and ground-truth kv-pair s2.
The matching protocol is calculated as following:

$\operatorname{Match}(\mathrm{s}1,\mathrm{~{}s}2)=\begin{cases}True,&ed(s1\\_k,s2\\_k)\leq
th\\_k~{}and~{}ed(s1\\_v,s2\\_v)\leq th\\_v\\\ False,&\text{other}\end{cases}$
(3) $th\\_k=max(factor\\_k*min(len(s1\\_k),len(s2\\_k)),0)$ (4)
$th\\_v=max(factor\\_v*min(len(s1\\_v),len(s2\\_v)),0)$ (5)

where $ed()$ denotes the edit distance function. the factor_k and factor_v are
set to 0.15, 0.15, respectively.

F-score. Considering all the predicted kv-pairs and all GT kv-pairs, the
F-score will be calculated as following:

$\text{ Precision }=\frac{N3}{N2}$ (6) $\text{ Recall }=\frac{N3}{N1}$ (7)
$\text{ score}2=\frac{2*\text{ Precision }*\text{ Recall }}{\text{ Precision
}+\text{ Recall }}$ (8)

where N1 denotes the number of kv-pairs that exists in the given image, N2
denotes the number of predicted kv-pairs, N3 denotes the number of perfectly
matched kv-pairs (both the edit distance of key and value are larger than a
threshold simultaneously. Specifically, the factor_k and factor_v in matching
protocol are set to 0.). The final score is the weighted score of score1 and
score2:

$\text{ score }=0.5*\text{ score}1+0.5*\text{ score}2$ (9)

The final weighted score will be used as submission ranking purpose for Task
1.

####  4.3.2 Task 2-4 Evaluation

For Task 2, Task 3, and Task 4, the evaluation metrics include two parts:

Normalized edit distance. For each predefined key text field, if it exists in
the given image, the normalized edit distance (NED) between predicted text s1
and ground-truth text s2 will be calculated as following:

$\begin{gathered}NED(s1,s2)=\frac{\text{ edit\\_distance }(s1,s2)}{\max(\text{
length }(s1),\operatorname{length}(s2))}\\\ \end{gathered}$ (10)

where n denotes the number of perfectly matched key text fields (both
entity_id and text are predicted correctly). Note that for predicted key text
fields that do not exist in the given image, the edit distance will be
calculated between predicted text and empty string. Then the $score1$ can be
calculated by Eq. 2.

F-score. Considering all the predicted key text fields and all the predefined
key text fields, the $score2$ (F-score) can also be calculated as Eq. 8. In
this scenarios, N1 denotes the number of key text fields that exists in the
given image, N2 denotes the number of predicted key text fields, N3 denotes
the number of perfectly matched key text fields (both entity_id and text are
predicted correctly). The final score is the weighted score of score1 and
score2:

$\text{ score }=0.8*\text{ score}1+0.2*\text{ score}2$ (11)

The final weighted score will be be used as submission ranking purpose for
Task 2, Task 3, and Task 4.

##  5 Submissions and Results

The competition attracted 50 participants and 117 submissions from academia
and industry, including 19 participants and 53 submissions for Task 1, 16
participants and 38 submissions for Task 2, 7 participants and 15 submissions
for Task 3 and 8 participants and 11 submissions for Task 4, which
demonstrated significant interest in this challenging task.

After the submission deadlines, we collected all submissions and evaluate
their performance through automated process with scripts developed by the RRC
web team. No feedback was given to the participants during the submission
process. If participants have multiple submissions, we pick the last
submission made before the final submission deadline for ranking. The winners
are determined for each task based on the score achieved by the corresponding
primary metric. The complete leaderboard is available on the online website
444https://rrc.cvc.uab.es/?ch=21&com=evaluation&task=1 for all tasks. However,
due to limited space, the results table presented below showcases a maximum of
10 top performers.

###  5.1 Task 1 Performance and Ranking

The result for Task 1 is presented on Table 2.

Table 2: Task-1: E2E Complex Entity Linking Results. ∗ means the reproducible
script has been submitted by participants and verified by organizers.

Rank | Method Name | Team Members | Insititute | Score1(NED) | Score2(F-score) | Score(Total)
---|---|---|---|---|---|---
1 | Super_KVer∗ |  Lele Xie, Zuming Huang, Boqian Xia, Yu Wang, Yadong Li, Hongbin Wang, Jingdong Chen | Ant Group | 49.93% | 62.97% | 56.45%
2 |  End-to-end document relationship extraction | Huiyan Wu, Pengfei Li, Can Li |  University of Chinese Academy of Sciences | 43.55% | 57.90% | 50.73%
3 | sample-3∗ |  Zhenrong Zhang, Lei Jiang, Youhui Guo, Jianshu Zhang, Jun Du |  University of Science and Technology of China, iFLYTEK AI Research | 42.52% | 56.68% | 49.60%
4 |  Pre-trained model based fullpipe pair extraction (opti_v3, no inf_aug)∗ |  Zening Lin, Teng Li, Wenhui Liao, Jiapeng Wang, Songxuan Lai, Lianwen Jin |  South China University of Technology; Huawei Cloud | 42.17% | 55.63% | 48.90%
5 | Meituan OCR V4∗ |  Jianqiang Liu, Kai Zhou, Chen Duan, Shuaishuai Chang, Ran Wei, Shan Guo | Meituan | 41.10% | 54.55% | 47.83%
6 | submit-trainall | hsy | - | 40.65% | 52.98% | 46.82%
7 | f2 | Zhi Zhang | cocopark | 41.07% | 50.82% | 45.94%
8 |  LayoutLM & STrucText Based Method | Wumin Hui, Mei Jiang | PKU & BUPT | 33.09% | 45.92% | 39.51%
9 | Layoutlmv3 |  Li Jie, Wang Wei, Li Songtao, Yang Yunxin, Chen Pengyu, Zhou Danya, Li Chao, Hu Shiyu, Zhang Yuqi, Xu Min, Zhao Yiru, Zhang Bin, Zhang Ruixue, Wang Di, Wang Hui, Xiang Dong | SPDB LAB | 29.81% | 41.45% | 35.63%
10 | Data Relation2 | - | - | 23.26% | 35.07% | 29.16%

The methods used by the top 3 submissions for Task 1 are presented below.

1st ranking method. “Ant Group” team apply an ensemble of both discriminative
and generative models. The former is a multimodal method which utilizes text,
layout and image, and they train this model with two different sequence
lengths, 2048 and 512 respectively. The texts and boxes are generated by
independent OCR models. The latter model is an end-to-end method which
directly generates K-V pairs for an input image.

2nd ranking method. “University of Chinese Academy of Sciences” team realized
end-to-end information extraction through OCR, NER and RE technologies. Text
information extracted by OCR and image information are jointly transmitted to
NER to identify key and value entities. RE module extracts entity pair
relationships through multi-classification. The training dataset is Hust-Cell.

3rd ranking method. “University of Science and Technology of China (USTC),
iFLYTEK AI Research” firstly perform key-value-background triplet
classification for each OCR bounding box using a PretrainedLM called GraphDoc
[21] which utilizes text, layout, and visual information simultaneously. Then
they use a detection model (DBNet [11]) to detect all the table cells in input
images and split images into table-images and non-table images. For table
images, they merge ocr boxes into table cells and then group all the left and
top keys for each value table cell as its corresponding key content. For non-
table images (including all text in non-table images and text outside tabel
cells in table images), they directly use a MLP model to predict all keys for
each value box.

###  5.2 Task 2 Performance and Ranking

The result for Task 2 is presented on Table 3.

Table 3: Task-2: E2E Complex Entity Labeling Results. ∗ means the reproducible
script has been submitted by participants and verified by organizers.

Rank | Method Name | Team Members | Insititute | Score1(NED) | Score2(F-score) | Score(Total)
---|---|---|---|---|---|---
1 | LayoutLMV3&StrucText∗ |  Minhui Wu, Mei Jiang, Chen Li, Jing Lv, Qingxiang Lin, Fan Yang | TencentOCR | 57.78% | 55.32% | 57.29%
2 | sample-3∗ |  Zhenrong Zhang, Lei Jiang, Youhui Guo, Jianshu Zhang, Jun Du |  University of Science and Technology of China, iFLYTEK AI Research | 47.15% | 41.91% | 46.10%
3 |  task 1 transfer learning LiLT + task3 transfer learning LiLT + LilLT + Layoutlmv3 ensemble∗ |  Hengguang Zhou, Zeyin Lin, Xingjian Zhao, Yue Zhang, Dahyun Kim, Sehwan Joo, Minsoo Khang, Teakgyu Hong | Deep SE x Upstage HK | 45.70% | 40.20% | 44.60%
4 | LayoutMask-v3∗ | Yi Tu | Ant Group | 44.79% | 42.53% | 44.34%
5 |  Pre-trained model based entity extraction (ro)∗ |  Zening Lin, Teng Li, Wenhui Liao, Jiapeng Wang, Songxuan Lai, Lianwen Jin |  South China University of Technology, Huawei Cloud | 44.98% | 40.06% | 43.99%
6 | EXO-brain for KIE |  Boqian Xia, Yu Wang, Yadong Li, Zuming Huang, Lele Xie, Jingdong Chen, Hongbin Wang | Ant Group | 44.02% | 39.63% | 43.14%
7 |  multi-modal based KIE through model fusion |  Jie Li, Wei Wang, Min Xu, Yiru Zhao, Bin Zhang, Pengyu Chen, Danya Zhou, Yuqi Zhang, Ruixue Zhang, Di Wang, Hui Wang, Chao Li, Shiyu Hu, Dong Xiang, Songtao Li, Yunxin Yang | SPDB LAB | 42.42% | 37.97% | 41.53%
8 | Aaaa | Li Rihong, Zheng Bowen |  Shenzhen Runnable Information Technology Co.,Ltd | 42.03% | 37.14% | 41.05%
9 | donut | zy | - | 41.64% | 37.65% | 40.84%
10 | Ant-FinCV | Tao Huang, Jie Wang, Tao Xu | Ant Group | 41.61% | 35.98% | 40.48%

The methods used by the top 3 submissions for Task 2 are presented below.

1st ranking method. “TencentOCR” team are mainly based on LayoutLMv3 and
StrucTextv1 model architecture. All training models are finetuned on large
pretrained models of LayoutLM and StrucText. During training and testing, they
did some preprocessings to merge and split some badly detected boxes. Since
entity label of kv-pair boxes are ignored, they used model trained on task1
images to predict kv relations of text boxes in Task 2 training/testing
images. Thus they added additional 2 classes of labels (question/answer) and
mapped original labels to new labels(other $\rightarrow$ question/answer) to
ease the difficulty of training. Similarly, During testing, they used kv-
prediction model to filter those text boxes with kv relations and used model
trained on Task 2 to predict entity label of the lefted boxes. In addition,
they combined predicted results of different models based on scores and rules
and did some postprocessings to merge texts with same entity label and
generated final output.

2nd ranking method. “University of Science and Technology of China (USTC),
iFLYTEK AI Research” team uses the GraphDoc [21] to perform bounding box
classification, which utilizes text, layout, and visual information
simultaneously.

3rd ranking method. “Deep SE x Upstage HK” team, for the OCR, uses a cascade
approach where the pipeline is broken up into text detection and text
recognition. For text detection, they use the CRAFT architecture with the
backbone changed to EfficientUNet-b3. For text recognition, they use the
ParSeq architecture with the visual feature extractor changed to SwinV2.
Regarding the parsing models, they trained both the LiLT and LayoutLMv3 models
on the Task2 dataset. For LiLT, they also conducted transfer learning on
either task1 or task3 before fine-tuning on Task 2 dataset. Finally, they take
an ensemble of these four models to get the final predictions.

###  5.3 Task 3 Performance and Ranking

The result for Task 3 is presented on Table 4.

Table 4: Task-3: E2E Zero-shot Structured Text Extraction Results. ∗ means the
reproducible script has been submitted by participants and verified by
organizers.

Rank | Method Name | Team Members | Insititute | Score1(NED) | Score2(F-score) | Score(Total)
---|---|---|---|---|---|---
1 |  sample-1∗ |  Zhenrong Zhang, Lei Jiang, Youhui Guo, Jianshu Zhang, Jun Du |  University of Science and Technology of China, iFLYTEK AI Research | 82.07% | 65.27% | 78.71%
2 |  LayoutLMv3∗ |  Minhui Wu, Mei Jiang, Chen Li, Jing Lv, Huiwen Shi | TencentOCR | 80.01% | 66.71% | 77.35%
3 |  KIE-Brain3∗ |  Boqian Xia, Yu Wang, Yadong Li, Ruyi Zhao, Zuming Huang, Lele Xie, Jingdong Chen, Hongbin Wang | Ant Group | 74.90% | 57.59% | 71.44%
4 | zero-shot-qa | - | - | 74.24% | 56.81% | 70.75%
5 | task3-2 | chengl | CMSS | 65.52% | 50.85% | 62.59%

The methods used by the top 3 submissions for Task 3 are presented below.

1st ranking method. “University of Science and Technology of China (USTC),
iFLYTEK AI Research” team first use OCR models (DBNet-det + SVTR-rec) to get
each bounding box coordinate and it’s text content of input images, then sort
boxes with manual rules, and concatenate all text content to a string
accroding to the box sequence. Result string of step 1 is fed into a seq2seq
model to directly predict target text content, which consists of 8 open-source
bert models, including chinese-roberta, chinese-lilt-roberta, chinese-pert,
chinese-lilt-pert, chinese-lert, chinese-lilt-lert, chinese-macbert and
chinese-lilt-macbert, and these models are trained using the UniLM
555https://github.com/microsoft/unilm/blob/master/s2s-ft/ toolbox. Data
augmentation including random text replacing and erasing, box random scaling
and shifting is used. As for english doc images, they directly use DocPrompt
666https://github.com/PaddlePaddle/PaddleNLP/blob/develop/model_zoo/ernie-
layout outputs as final result

2nd ranking method. “TencentOCR” team’s method is based on LayoutLMv3 and
StrucTextv1 model architecture. All training models are finetuned on large
pretrained models of LayoutLM and StrucText. During training and testing, we
did some preprocessings to merge and split some badly detected boxes. They
used models trained on task 1 images to predict key-value pairs in test images
and models trained on task 2 images to predict entity labels(title, date, etc)
for text boxes. Besides, they applied rule based post processing methods and
assembled results of different models to generate final outputs.

3rd ranking method. “Ant Group” team apply an ensemble of multi-task end-to-
end information extraction models. The document question answering task and
the document information extraction task are jointly realized, and the model
performance is improved. At the same time, this solution is an end-to-end
information extraction method and does not rely on external OCR.

###  5.4 Task 4 Performance and Ranking

The result for Task 4 is presented on Table 5.

Table 5: Task-4: E2E Few-shot Structured Text Extraction Results. ∗ means the
reproducible script has been submitted by participants and verified by
organizers.

Rank | Method Name | Team Members | Insititute | Score1(NED) | Score2(F-score) | Score(Total)
---|---|---|---|---|---|---
1 |  LayoutLMv3& StrucText ∗ |  Mei Jiang, Minhui Wu, Chen Li, Jing Lv, Haoxi Li, Lifu Wang, Sicong Liu | TencentOCR | 87.14% | 73.59% | 84.43%
2 | sample-1∗ |  Zhenrong Zhang, Lei Jiang, Youhui Guo, Jianshu Zhang, Jun Du |  University of Science and Technology of China, iFLYTEK AI Research | 85.24% | 69.68% | 82.13%
3 | task4-base | chengl | CMSS | 78.57% | 60.21% | 74.90%
4 | Fewshot-brain_v1∗ |  Boqian Xia, Yu Wang, Yadong Li, Hongbin Wang | Ant Group | 77.81% | 60.71% | 74.39%
5 |  Dao Xianghu light of TianQuan |  Kai Yang, Tingmao Lin, Ye Wang, Shuqiang Lin, Jian Xie, Bin Wang, Wentao Liu, Xiaolu Ding, Jun Zhu, Hongyan Pan, Jia Lv |  CCB Financial Technology Co. Ltd, China | 71.48% | 55.03% | 68.19%
6 | GRGBanking |  Liu Kaihang, Yue Xuyao, Xu Tianshi, Zhang Huajun, Liang Tiankai | GRGBanking | 45.44% | 35.83% | 43.52%

The methods used by the top 3 submissions for Task 4 are presented below.

1st ranking method. “TencentOCR” team’s methods are mainly based on LayoutLMv3
and StrucTextv1 model architecture. All training models are finetuned on large
pretrained models of LayoutLM and StrucText. During training and testing, they
did some preprocessings to merge and split some badly detected boxes. They
also trained our own ocr models including dc-convnet based detection and
ctc/eda based recognition. They applied merging methods to merge ocr results
from different sources. Based on predicted results of task 3, they also
introduced self-supervising training to train segment-based classification
models for folder 9 & folder 10 to predict entity labels. Similar to task 3,
they also did rule based post processing methods and assembled results of
different models to generate final outputs.

2nd ranking method. “University of Science and Technology of China (USTC),
iFLYTEK AI Research” team first use OCR models (DBNet-det + SVTR-rec) to get
each bounding box coordinate and it’s text content of input images, then sort
boxes with manual rules, and concatenate all text content to a string
accroding to the box sequence. Result string of step 1 is fed into a seq2seq
model to directly predict target text content, which consists of 8 open-source
bert models, including chinese-roberta, chinese-lilt-roberta, chinese-pert,
chinese-lilt-pert, chinese-lert, chinese-lilt-lert, chinese-macbert and
chinese-lilt-macbert, and these models are trained using the UniLM
777https://github.com/microsoft/unilm/blob/master/s2s-ft/ toolbox. Data
augmentation including random text replacing and erasing, box random scaling
and shifting is used. As for english doc images, they directly use DocPrompt
888https://github.com/PaddlePaddle/PaddleNLP/blob/develop/model_zoo/ernie-
layout outputs as final result.

3rd ranking method. “CMSS” team’s method firstly perform data cleaning, data
synthesis, and random augmentation based on the given data. For blurry images,
they utilize a variety of cascaded models to optimize the text detection
results. To further improve the accuracy of text recognition algorithm, they
use a neural network structure and remove special characters. Particularly,
they have found that the position of the text plays a significant role in the
structure, especially for structured text with key and value. To address this
issue, they optimize the model parameters based on the UIE-X basic model
(Unified Structure Generation for Universal Information Extraction) and
optimize the data pre-processing part to enhance the relationship features
between text lines. Finally, they merge the model inference results by
designing rules to achieve the best possible outcome.

##  6 Discussion

The competition saw a considerable number of submissions from both academic
and industrial participants, indicating a notable level of interest in the
topic of complex structured text extraction. The competition focused on two
tracks: Track 1 involved end-to-end complex entity linking and labeling, while
Track 2 introduced new zero/few-shot scenario tasks for structured text
extraction.

The leading approaches in Track 1 of the competition utilized ensemble models
that integrated multiple modalities, including text, layout, and image
information. These models were enhanced by incorporating techniques such as
OCR, NER, and RE. The prominence of multimodal models and ensemble techniques
highlights their significance. Nevertheless, there exists a notable
performance gap in achieving the necessary accuracy for information extraction
in complex scenarios for large-scale practical applications. The observed
performance gap is notable when comparing the top-performing F1 score of
55.32% for Task 2 to the state-of-the-art end-to-end method on other datasets.
Specifically, Kuang et al. [9] achieved an 85.87% F1 score on the SROIE
dataset and Donut [8] achieved an 84.1% F1 score on the CORD dataset in end-
to-end evaluations.

Track 2 of the SVRD competition focused on new zero/few-shot scenario tasks
for structured text extraction. Specifically, Task 3 required participants to
extract structured text from images that were not present in the training set,
whereas Task 4 allowed participants to perform structured text extraction
using a small number of provided labeled examples. The methods used for zero-
shot and few-shot learning involved various deep learning techniques, such as
pretrained large multimodal model and model ensemble. The top-performing
methods achieved 78.71% and 84.43% for Task 3 and Task 4, respectively.
Despite the commendable average performance of Task 3 and Task 4 across all 10
scenarios, the individual performance analysis of the champion method reveals
significantly low effectiveness in more challenging scenarios. Specifically,
the champion method attained a score of 53.25% in the Letter/Email scenario
and 46.2% in the Technical Report scenario for Task 3. In Task 4, the champion
method achieved a score of 59.23% in the Car Ticket scenario. These results
underscore the need for further research to enhance the performance of these
methods in structured text extraction for such challenging scenarios.

Despite the top-performing methods showcased promising results, a significant
performance gap remains for challenging scenarios in structured text
extraction. Advancements in developing robust techniques for structured text
extraction are crucial for the progress of Document AI. The SVRD competition
provided a platform for researchers in CV and NLP to collaborate and showcase
their expertise. The results highlight the potential of structured text
extraction for various document analysis applications, emphasizing the need
for continued research.

##  7 Conclusion

We hosted the SVRD competition, focused on key information extraction from
visually-rich document images, and provided new datasets, including HUST-CELL
and Baidu-FEST, and designed new evaluation protocols for our tasks. The
submissions indicated strong interest from academia and industry, but also
revealed significant challenges in achieving high performance for complex and
zero-shot scenarios. Key information extraction remains a difficult task with
potential for numerous document analysis applications. Future competitions
could expand on this topic with more challenging datasets and applications,
attracting researchers in CV and NLP and advancing the field of Document AI.

##  8 Acknowledgements

This competition is supported by the National Natural Science Foundation of
China (No.62225603, No.62206103, No.62206104). The organizers thank Sergi
Robles and the RRC web team for their tremendous support on the registration,
submission and evaluation jobs.

## References

  * [1] Appalaraju, S., Jasani, B., Kota, B.U., Xie, Y., Manmatha, R.: Docformer: End-to-end transformer for document understanding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 993–1003 (2021)
  * [2] Cao, H., Li, X., Ma, J., Jiang, D., Guo, A., Hu, Y., Liu, H., Liu, Y., Ren, B.: Query-driven generative network for document information extraction in the wild. In: Proceedings of the 30th ACM International Conference on Multimedia (2022)
  * [3] Cao, H., Ma, J., Guo, A., Hu, Y., Liu, H., Jiang, D., Liu, Y., Ren, B.: Gmn: Generative multi-modal network for practical document information extraction. In: North American Chapter of the Association for Computational Linguistics (2022)
  * [4] Fang, S., Xie, H., Wang, Y., Mao, Z., Zhang, Y.: Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7098–7107 (2021)
  * [5] Guo, H., Qin, X., Liu, J., Han, J., Liu, J., Ding, E.: Eaten: Entity-aware attention for single shot visual text extraction. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 254–259. IEEE (2019)
  * [6] Huang, Z., Chen, K., He, J., Bai, X., Karatzas, D., Lu, S., Jawahar, C.: Icdar2019 competition on scanned receipt ocr and information extraction. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 1516–1520. IEEE (2019)
  * [7] Jaume, G., Ekenel, H.K., Thiran, J.P.: Funsd: A dataset for form understanding in noisy scanned documents. In: ICDARW. vol. 2, pp. 1–6 (2019)
  * [8] Kim, G., Hong, T., Yim, M., Nam, J., Park, J., Yim, J., Hwang, W., Yun, S., Han, D., Park, S.: Ocr-free document understanding transformer. In: European Conference on Computer Vision (ECCV) (2022)
  * [9] Kuang, J., Hua, W., Liang, D., Yang, M., Jiang, D., Ren, B., Zhou, Y., Bai, X.: Visual information extraction in the wild: Practical dataset and end-to-end solution. In: ICDAR (2023)
  * [10] Li, Y., Qian, Y., Yu, Y., Qin, X., Zhang, C., Liu, Y., Yao, K., Han, J., Liu, J., Ding, E.: Structext: Structured text understanding with multi-modal transformers. In: Proceedings of the 29th ACM International Conference on Multimedia. pp. 1912–1920 (2021)
  * [11] Liao, M., Wan, Z., Yao, C., Chen, K., Bai, X.: Real-time scene text detection with differentiable binarization. In: Proceedings of the AAAI conference on artificial intelligence. vol. 34, pp. 11474–11481 (2020)
  * [12] Park, S., Shin, S., Lee, B., Lee, J., Surh, J., Seo, M., Lee, H.: Cord: a consolidated receipt dataset for post-ocr parsing. In: Workshop on Document Intelligence at NeurIPS 2019 (2019)
  * [13] Shi, B., Bai, X., Yao, C.: An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE transactions on pattern analysis and machine intelligence 39(11), 2298–2304 (2016)
  * [14] Wang, J., Liu, C., Jin, L., Tang, G., Zhang, J., Zhang, S., Wang, Q., Wu, Y., Cai, M.: Towards robust visual information extraction in real world: new dataset and novel solution. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 2738–2745 (2021)
  * [15] Xu, Y., Li, M., Cui, L., Huang, S., Wei, F., Zhou, M.: Layoutlm: Pre-training of text and layout for document image understanding. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 1192–1200 (2020)
  * [16] Xu, Y., Lv, T., Cui, L., Wang, G., Lu, Y., Florencio, D., Zhang, C., Wei, F.: Xfund: A benchmark dataset for multilingual visually rich form understanding. In: Findings of the Association for Computational Linguistics: ACL 2022. pp. 3214–3224 (2022)
  * [17] Yu, D., Li, X., Zhang, C., Liu, T., Han, J., Liu, J., Ding, E.: Towards accurate scene text recognition with semantic reasoning networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12113–12122 (2020)
  * [18] Yu, W., Lu, N., Qi, X., Gong, P., Xiao, R.: Pick: processing key information extraction from documents using improved graph learning-convolutional networks. In: 2020 25th International Conference on Pattern Recognition (ICPR). pp. 4363–4370. IEEE (2021)
  * [19] Zhang, C., Liang, B., Huang, Z., En, M., Han, J., Ding, E., Ding, X.: Look more than once: An accurate detector for text of arbitrary shapes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10552–10561 (2019)
  * [20] Zhang, P., Xu, Y., Cheng, Z., Pu, S., Lu, J., Qiao, L., Niu, Y., Wu, F.: Trie: end-to-end text reading and information extraction for document understanding. In: Proceedings of the 28th ACM International Conference on Multimedia. pp. 1413–1422 (2020)
  * [21] Zhang, Z., Ma, J., Du, J., Wang, L., Zhang, J.: Multimodal pre-training based on graph attention network for document understanding. IEEE Transactions on Multimedia (2022)
  * [22] Zhou, X., Yao, C., Wen, H., Wang, Y., Zhou, S., He, W., Liang, J.: East: an efficient and accurate scene text detector. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 5551–5560 (2017)


††thanks: Corresponding author

# Model-free inequality for data of Einstein-Podolsky-Rosen-Bohm experiments

Hans De Raedt<EMAIL_ADDRESS>Jülich Supercomputing Centre, Institute
for Advanced Simulation, Forschungszentrum Jülich, 52425 Jülich, Germany
Zernike Institute for Advanced Materials, University of Groningen, Nijenborgh
4, 9747AG Groningen, Netherlands  Mikhail I. Katsnelson  Radboud University,
Institute for Molecules and Materials, Heyendaalseweg 135, 6525AJ Nijmegen,
Netherlands  Manpreet S. Jattana  Jülich Supercomputing Centre, Institute for
Advanced Simulation, Forschungszentrum Jülich, 52425 Jülich, Germany  Modular
Supercomputing and Quantum Computing, Goethe University Frankfurt,
Kettenhofweg 139, 60325 Frankfurt am Main, Germany  Vrinda Mehta  Jülich
Supercomputing Centre, Institute for Advanced Simulation, Forschungszentrum
Jülich, 52425 Jülich, Germany  RWTH Aachen University, 52056 Aachen, Germany
Madita Willsch  Jülich Supercomputing Centre, Institute for Advanced
Simulation, Forschungszentrum Jülich, 52425 Jülich, Germany  Dennis Willsch
Jülich Supercomputing Centre, Institute for Advanced Simulation,
Forschungszentrum Jülich, 52425 Jülich, Germany  Kristel Michielsen  Jülich
Supercomputing Centre, Institute for Advanced Simulation, Forschungszentrum
Jülich, 52425 Jülich, Germany  RWTH Aachen University, 52056 Aachen, Germany
Fengping Jin  Jülich Supercomputing Centre, Institute for Advanced Simulation,
Forschungszentrum Jülich, 52425 Jülich, Germany

###### Abstract

We present a new inequality constraining correlations obtained when performing
Einstein-Podolsky-Rosen-Bohm experiments. The proof does not rely on
mathematical models that are imagined to have produced the data and is
therefore “model-free”. The new inequality contains the model-free version of
the well-known Bell-CHSH inequality as a special case. A violation of the
latter implies that not all the data pairs in four data sets can be reshuffled
to create quadruples. This conclusion provides a new perspective on the
implications of the violation of Bell-type inequalities by experimental data.

Einstein-Podolsky-Rosen-Bohm experiments, Bell’s theorem, Bell-Clauser-Horn-
Shimony-Holt inequalities, data analysis

The Einstein-Podolsky-Rosen thought experiment was introduced to question the
completeness of quantum theory [1]. Bohm proposed a modified version that
employs spin-1/2 objects instead of coordinates and momenta of a two-particle
system [2]. This modified version, which we refer to as the Einstein-Podolsky-
Rosen-Bohm (EPRB) experiment, has been the subject of many experiments [3, 4,
5, 6, 7, 8, 9, 10], primarily focusing on ruling out the model for the EPRB
experiment proposed by Bell [11, 12].

The essence of the EPRB thought experiment is shown and described in Fig. 1.
Motivated by the work of Bell [12] and Clauser et al. [13, 14], many EPRB
experiments [4, 5, 6, 7, 8, 9, 10] focus on demonstrating a violation of the
Bell-CHSH inequality [13, 12]. To this end, one performs four EPRB experiments
under conditions defined by the directions $(\mathbf{a},\mathbf{c})$,
$(\mathbf{a},\mathbf{d})$, $(\mathbf{b},\mathbf{c})$, and
$(\mathbf{b},\mathbf{d})$, yielding the data sets of pairs of discrete data

$\displaystyle{\cal D}_{s}$ $\displaystyle=$
$\displaystyle\\{(A_{s,n},B_{s,n})\,|\,A_{s,n},B_{s,n}=\pm
1\,;\,n=1,\ldots,N_{s}\\}\;,$ (1)

where $s=1,2,3,4$ labels the four alternative conditions and $N_{s}$ is the
number of pairs emitted by the source. Then one computes correlations
according to

$C_{s}=\frac{1}{N}\sum_{n=1}^{N}A_{s,n}B_{s,n}\;,$ (2)

where $N=\min(N_{1},N_{2},N_{3},N_{4})$.

Figure 1: (color online) Conceptual representation of the Einstein-Podolsky-
Rosen experiment [1] in the modified form proposed by Bohm [2]. A source
produces pairs of particles. The particles of each pair carry opposite
magnetic moments, implying that there is a correlation between the two
magnetic moments of each pair leaving the source. The magnetic field gradients
of the Stern-Gerlach magnets (cylinders) with their uniform magnetic field
component along the directions of the unit vectors $\mathbf{a}$ and
$\mathbf{c}$ divert each incoming particle into one of the two spatially
separated directions labeled by $+1$ and $-1$. The pair
$(\mathbf{a},\mathbf{c})$ represents the conditions, denoted by the subscript
“$1$”, under which the discrete data $(A_{1,n},B_{1,n})$ is collected. The
values of $A_{1,n}$ and $B_{1,n}$ correspond to the labels of the directions
in which the particles have been diverted. The result of this experiment is
the set of data pairs ${\cal
D}_{1}=\\{(A_{1,1},B_{1,1}),\ldots,(A_{1,N_{1}},B_{1,N_{1}})\\}$ where $N_{1}$
denotes the number of pairs emitted by the source. The alternative conditions
$(\mathbf{a},\mathbf{d})$, $(\mathbf{b},\mathbf{c})$, and
$(\mathbf{b},\mathbf{d})$ are labeled by subscripts “$2$”, “$3$”, and “$4$”,
respectively.

In general, each correlation $C_{s}$ may take values $-1$ or $+1$, independent
of the values taken by other contributions, yielding the trivial bound
$|C_{1}\mp C_{2}|+|C_{3}\pm C_{4}|\leq 4$. Without introducing a specific
model for the process generating the data, we can derive a nontrivial bound
that is sharper by exploiting the commutative property of addition.

Let $K_{\mathrm{max}}$ be the maximum number of quadruples
$(x_{k},y_{k},z_{k},w_{k})$ that can be found by searching for permutations
$P$, ${\widehat{P}}$, ${\widetilde{P}}$, and $P^{\prime}$ of
$\\{1,2,\ldots,N\\}$ such that for $k=1,\ldots,K_{\mathrm{max}}$,

$\displaystyle x_{k}$ $\displaystyle=A_{1,P(k)}=A_{2,{\widehat{P}}(k)}\;,$
$\displaystyle\;y_{k}$
$\displaystyle=A_{3,{\widetilde{P}}(k)}=A_{4,P^{\prime}(k)}\;,$ $\displaystyle
z_{k}$ $\displaystyle=B_{1,P(k)}=B_{3,{\widetilde{P}}(k)}\;,$
$\displaystyle\;w_{k}$
$\displaystyle=B_{2,{\widehat{P}}(k)}=B_{4,P^{\prime}(k)}\;.$ (3)

These quadruples are found by rearranging/reshuffling the data in ${\cal
D}_{1}$, ${\cal D}_{2}$, ${\cal D}_{3}$, and ${\cal D}_{4}$ without affecting
the value of the correlations $C_{1}$, $C_{2}$, $C_{3}$, and $C_{4}$.

Theorem: For any (real or computer or thought) EPRB experiment, the
correlations Eq. (2) computed from the four data sets ${\cal D}_{1}$, ${\cal
D}_{2}$, ${\cal D}_{3}$, and ${\cal D}_{4}$, must satisfy the model-free
inequalities

$\displaystyle{\cal C}_{\pm}=\left|C_{1}\mp C_{2}\right|+\left|C_{3}\pm
C_{4}\right|\leq 4-2{\Delta}\;,$ (4)

where $0\leq{\Delta}=K_{\mathrm{max}}/N\leq 1$.

Proof: We rewrite the correlations in Eq. (2) as

$\displaystyle C_{1}$
$\displaystyle=\frac{1}{N}\sum_{n=1}^{N}A_{1,{P}(n)}B_{1,{P}(n)},$
$\displaystyle\;C_{2}$
$\displaystyle=\frac{1}{N}\sum_{n=1}^{N}A_{2,{\widehat{P}}(n)}B_{2,{\widehat{P}}(n)}\;,$
$\displaystyle C_{3}$
$\displaystyle=\frac{1}{N}\sum_{n=1}^{N}A_{3,{\widetilde{P}}(n)}B_{3,{\widetilde{P}}(n)},$
$\displaystyle\;C_{4}$
$\displaystyle=\frac{1}{N}\sum_{n=1}^{N}A_{4,P^{\prime}(n)}B_{4,P^{\prime}(n)}\;.$
(5)

Obviously, reordering the terms of the sums does not change the value of the
sums themselves. As $|x_{k}|=|y_{k}|=|z_{k}|=|w_{k}|=1$, we (trivially) have
$|x_{k}z_{k}-x_{k}w_{k}|+|y_{k}z_{k}+y_{k}w_{k}|=2$.

Splitting each of the sums in Eq. (5) into a sum over
$k=1,\ldots,K_{\mathrm{max}}$ and the sum of the remaining terms, application
of the triangle inequality yields

$\displaystyle{\cal C}_{\pm}$ $\displaystyle\leq$
$\displaystyle\frac{2K_{\mathrm{max}}}{N}$ (6)
$\displaystyle+\frac{1}{N}\sum_{n=K_{\mathrm{max}}+1}^{N}\left(\left|A_{1,{P}(n)}B_{1,{P}(n)}\right|+\left|A_{2,{\widehat{P}}(n)}B_{2,{\widehat{P}}(n)}\right|\right.$
$\displaystyle\hbox
to56.9055pt{}\left.+\left|A_{3,{\widetilde{P}}(n)}B_{3,{\widetilde{P}}(n)}\right|+\left|A_{4,P^{\prime}(n)}B_{4,P^{\prime}(n)}\right|\right)$
$\displaystyle\leq$
$\displaystyle\frac{2K_{\mathrm{max}}}{N}+\frac{4(N-K_{\mathrm{max}})}{N}=4-2{\Delta}\;.$

QED.

In the same manner, one can prove a model-free version of the Clauser-Horne
inequality [14, 15], applicable to the data of the EPRB experiments reported
in Ref. 9, 10.

The choice represented by Eq. (3) is motivated by the EPRB experiment, see
Fig. 1. In general, other choices to define quadruples are possible and may
yield different values of the maximum fraction of quadruples.

We introduce the Bell-CHSH-like function

$\displaystyle{S}=\max_{p}\left|C_{p(1)}-C_{p(2)}+C_{p(3)}+C_{p(4)}\right|\;,$
(7)

where the maximum is over all permutations $p$ of $\\{1,2,3,4\\}$. The maximum
guarantees that we cover all possible expressions of the original Bell-CHSH
function [13, 16, 12]. By application of the triangle inequality, it directly
follows from Eq. (4) that, in the case of data collected by EPRB experiments,

$\displaystyle{S}\leq 4-2\Delta\;.$ (8)

The symbol ${\Delta}$ in Eqs. (4) and (8) quantifies structure in terms of
quadruples which can be created by relabeling the pairs of data in the sets
${\cal D}_{1}$, ${\cal D}_{2}$, ${\cal D}_{3}$, and ${\cal D}_{4}$. If
${\Delta}=0$, it is impossible to find a reshuffling that yields even one
quadruple. If ${\Delta}=1$, the four sets can be reshuffled such that they can
be viewed as being generated from $N$ quadruples. In the special case
${\Delta}=1$, we recover the model-free version of the Bell-CHSH inequality

$\displaystyle{S}\leq 2\;.$ (9)

Traditionally, Eq. (9) is proven by assuming that the data can be modeled by a
so-called “local realistic” (Bell) model [13, 16, 14, 15, 12]. However, this
proof does not extend to the much more general case of the data generated by
EPRB experiments, in contrast to the proof of Eq. (9) which holds for
experimental data.

The proof of the model-free inequalities Eqs. (4) and (8) only requires the
existence of a maximum number of quadruples, the actual value of this maximum
being irrelevant for the proof. However, it is instructive to write a computer
program that uses pseudo-random numbers to generate the data sets ${\cal
D}_{1}$, ${\cal D}_{2}$, ${\cal D}_{3}$, and ${\cal D}_{4}$ and finds the
number of quadruples. We have implemented the computer program in
Mathematica®.

Naively, finding the value of ${\Delta}$ seems to require ${\cal O}(N!^{4})$
operations. Fortunately, the problem of determining the fraction of quadruples
${\Delta}$ can be cast into an integer linear programming problem which is
readily solved by considering the associated linear programming problem with
real-valued unknowns. In practice, we solve the latter by standard
optimization techniques [17]. For all cases that we have studied, the solution
of the linear programming problem takes integer values only. Then the solution
of the linear programming problem is also the solution of the integer
programming problem.

The results of several numerical experiments using $N=1000000$ pairs per data
set can be summarized as follows:

  * •

If the $A$’s and $B$’s are generated in the form of quadruples, all items
taking random values $\pm 1$, the program returns ${\Delta}=1$,
$|C_{1}-C_{2}|+|C_{3}+C_{4}|=0.00063$, and
$|C_{1}+C_{2}|+|C_{3}-C_{4}|=0.0028$ such that Eq. (4) is satisfied.

  * •

If all $A$’s and $B$’s take independent random values $\pm 1$, the $C$’s are
approximately zero. We have $|C_{1}\mp C_{2}|+|C_{3}\pm C_{4}|\approx 0\leq
4-2{\Delta}$ where in our particular numerical experiment ${\Delta}=0.998$.

  * •

If the pairs $(A_{1,i},B_{1,i})$, $(A_{2,j},B_{2,j})$, $(A_{3,k}B_{3,k})$, and
$(A_{4,l}B_{4,l})$ are generated randomly with frequencies
$(1-c_{1}A_{1,i}B_{1,i})/4$, $(1-c_{2}A_{2,j}B_{2,j})/4$,
$(1-c_{3}A_{3,k}B_{3,k})/4$, and $(1-c_{4}A_{4,l}B_{4,l})/4$, respectively,
the simulation mimics the case of the correlation of two spin-1/2 objects in
the singlet state if we choose $c_{1}=-c_{2}=c_{3}=c_{4}=1/\sqrt{2}$. In this
particular case, quantum theory yields
$\max\left(|C_{1}-C_{2}|+|C_{3}+C_{4}|,|C_{1}+C_{2}|+|C_{3}-C_{4}|\right)=2\sqrt{2}\approx
2.83$ [18].

Generating four times one million independent pairs, we obtain
${\Delta}\approx 0.585$, ${S}=|C_{1}-C_{2}|+|C_{3}+C_{4}|\approx 2.83$ and
$4-2{\Delta}\approx 2.83$, demonstrating that the value of the quantum-
theoretical upper bound $2\sqrt{2}$ is reflected in the maximum fraction of
quadruples that one can create by reshuffling the data.

  * •

In the case of Bell’s model, slightly modified to comply with Malus’ law, we
have $C_{1}=-\mathbf{a}\cdot\mathbf{c}/2$ and ${S}\leq\sqrt{2}$. Choosing
$c_{1}=-c_{2}=c_{3}=c_{4}=1/(2\sqrt{2})$ and generating four times one million
independent pairs, we obtain $S=|C_{1}-C_{2}|+|C_{3}+C_{4}|\approx 1.42$ and
$4-2{\Delta}\approx 2.00$, as expected for Bell’s local realistic model.

Except for the first case, the numerical values of ${\Delta}$ quoted fluctuate
a little if we repeat the $N=1000000$ simulations with different random
numbers. In the third case, the simulations suggest that inequality Eq. (4)
can be saturated.

Suppose that the (post-processed) data of an EPRB laboratory experiment yield
${S}>2$, that is the data violate inequality Eq. (9). From Eq. (8), it follows
that ${\Delta}\leq 2-{S}/2<1$ if ${S}>2$. Therefore, if ${S}>2$ not all the
data in ${\cal D}_{1}$, ${\cal D}_{2}$, ${\cal D}_{3}$, and ${\cal D}_{4}$ can
be reshuffled such that they form quadruples only. Indeed, the data produced
by these experiments have to comply with Eq. (8) which follows from Eq. (4)
holding for data, and certainly do not have to comply with the original Bell-
CHSH inequality obtained from Bell’s model.

In other words, all EPRB experiments which have been performed and may be
performed in the future and which only focus on demonstrating a violation of
Eq. (9) merely provide evidence that not all contributions to the correlations
can be reshuffled to form quadruples (yielding ${\Delta}<1$). These violations
do not provide any clue about the nature of the physical processes that
produce the data.

More specifically, Eq. (4) holds for discrete data, irrespective of how the
data sets ${\cal D}_{1}$, ${\cal D}_{2}$, ${\cal D}_{3}$, and ${\cal D}_{4}$
were obtained. Inequality (4) shows that correlations of discrete data violate
the Bell-CHSH inequality Eq. (9) only if not all the pairs of data in Eq. (2)
can be reshuffled to create quadruples. The proofs of Eq. (4) and Eq. (9) do
not refer to notions such as “locality”, “realism”, “non-invasive
measurements”, “action at a distance”, “free will”, “superdeterminism”,
“(non)contextually”, “complementarity”, etc. Logically speaking, a violation
of Eq. (9) by experimental data cannot be used to argue about the relevance of
one or more of these notions to the process that generated the experimental
data.

The existence of the divide between the realm of experimental EPRB data and
mathematical models thereof is further supported by Fine’s theorem [19, 20].
Of particular relevance to the present discussion is the part of the theorem
that establishes the Bell-CHSH inequalities (plus compatibility) as being the
necessary and sufficient conditions for the existence of a joint distribution
of the four observables involved in these inequalities. This four-variable
joint distribution returns the pair distributions describing the four EPRB
experiments required to test for a violation of these inequalities. Fine’s
theorem holds in the realm of mathematical models only. Only in the
unattainable limit of an infinite number of measurements (that is by leaving
the realm of experimental data), and in the special case that the Bell-CHSH
inequalities hold, it may be possible to prove the equivalence between the
model-free inequality Eq. (9) and the Bell-CHSH inequality [13, 16, 14, 12].

A violation of the original (non model-free) Bell-CHSH inequality $S\leq 2$
may lead to a variety of conclusions about certain properties of the
mathematical model for which this inequality has been derived. However,
projecting these logically correct conclusions about the mathematical model,
obtained within the context of that mathematical model, to the domain of EPRB
laboratory experiments requires some care, as we now discuss.

The first step in this projection is to feed real-world, discrete data into
the original Bell-CHSH inequality $S\leq 2$ derived, not for discrete data as
we did by considering the case ${\Delta}=1$ in Eq. (8), but rather in the
context of some mathematical model, and to conclude that this inequality is
violated. Considering the discrete data for the correlations as given, it may
indeed be tempting to plug these rational numbers into an expression obtained
from some mathematical model. However, then it is no longer clear what a
violation actually means in terms of the mathematical model because the latter
(possibly by the help of pseudo-random number generators) may not be able to
produce these  experimental data at all. The second step is to conclude from
this violation that the mathematical model cannot produce the numerical values
of the correlations, implying that the mathematical model simply does not
apply and has to be replaced by a more adequate one or that one or more
premises underlying the mathematical model must be wrong. In the latter case,
the final step is to project at least one of these wrong premises to
properties of the world around us.

The key question is then to what extent the premises or properties of a
mathematical model can be transferred to those of the world around us. Based
on the rigorous analysis presented in this paper, the authors’ point of view
is that in the case of laboratory EPRB experiments, they cannot.

###### Acknowledgements.

We are grateful to Bart De Raedt for suggesting that finding the maximum
number of quadruples might be cast into an integer programming problem and for
making pertinent comments. We thank Koen De Raedt for many discussions and
continuous support. The work of M.I.K. was supported by the European Research
Council (ERC) under the European Union’s Horizon 2020 research and innovation
programme, grant agreement 854843 FASTCORR. M.S.J. acknowledges support from
the project OpenSuperQ (820363) of the EU Quantum Flagship. V.M., D.W. and
M.W. acknowledge support from the project Jülich UNified Infrastructure for
Quantum computing (JUNIQ) that has received funding from the German Federal
Ministry of Education and Research (BMBF) and the Ministry of Culture and
Science of the State of North Rhine-Westphalia.

## References

  * Einstein  _et al._ [1935] A. Einstein, A. Podolsky, and N. Rosen, Can quantum-mechanical description of physical reality be considered complete?, Phys. Rev. 47, 777 (1935).
  * Bohm [1951] D. Bohm,  _Quantum Theory_ (Prentice-Hall, New York, 1951).
  * Kocher and Commins [1967] C. A. Kocher and E. D. Commins, Polarization correlation of photons emitted in an atomic cascade, Phys. Rev. Lett. 18, 575 (1967).
  * Clauser and Shimony [1978] J. F. Clauser and A. Shimony, Bell’s theorem: Experimental tests and implications, Rep. Prog. Phys. 41, 1881 (1978).
  * Aspect  _et al._ [1982] A. Aspect, J. Dalibard, and G. Roger, Experimental test of Bell’s inequalities using time-varying analyzers, Phys. Rev. Lett. 49, 1804 (1982).
  * Weihs  _et al._ [1998] G. Weihs, T. Jennewein, C. Simon, H. Weinfurther, and A. Zeilinger, Violation of Bell’s inequality under strict Einstein locality conditions, Phys. Rev. Lett. 81, 5039 (1998).
  * Christensen  _et al._ [2013] B. Christensen, K. McCusker, J. Altepeter, B. Calkins, C. Lim, N. Gisin, and P. Kwiat, Detection-loophole-free test of quantum nonlocality, and applications, Phys. Rev. Lett. 111, 130406 (2013).
  * Hensen  _et al._ [2015] B. Hensen, H. Bernien, A. E. Dreau, A. Reiserer, N. Kalb, M. S. Blok, J. Ruitenberg, R. F. L. Vermeulen, R. N. Schouten, C. Abellan, W. Amaya, V. Pruneri, M. W. Mitchell, M. Markham, D. J. Twitchen, D. Elkouss, S. Wehner, T. H. Taminiau, and R. Hanson, Loophole-free Bell inequality violation using electron spins separated by 1.3 kilometres, Nature , 15759 (2015).
  * Giustina  _et al._ [2015] M. Giustina, M. A. M. Versteegh, S. Wengerowsky, J. Handsteiner, A. Hochrainer, K. Phelan, F. Steinlechner, J. Kofler, J.-A. Larsson, C. Abellán, W. Amaya, V. Pruneri, M. W. Mitchell, J. Beyer, T. Gerrits, A. E. Lita, L. K. Shalm, S. W. Nam, T. Scheidl, R. Ursin, B. Wittmann, and A. Zeilinger, Significant-loophole-free test of Bell’s theorem with entangled photons, Phys. Rev. Lett. 115, 250401 (2015).
  * Shalm  _et al._ [2015] L. K. Shalm, E. Meyer-Scott, B. G. Christensen, P. Bierhorst, M. A. Wayne, M. J. Stevens, T. Gerrits, S. Glancy, D. R. Hamel, M. S. Allman, K. J. Coakley, S. D. Dyer, C. Hodge, A. E. Lita, V. B. Verma, C. Lambrocco, E. Tortorici, A. L. Migdall, Y. Zhang, D. Kumor, W. H. Farr, F. Marsili, M. D. Shaw, J. A. Stern, C. Abellán, W. Amaya, V. Pruneri, T. Jennewein, M. W. Mitchell, P. G. Kwiat, J. C. Bienfang, R. P. Mirin, E. Knill, and S. W. Nam, Strong loophole-free test of local realism, Phys. Rev. Lett. 115, 250402 (2015).
  * Bell [1964] J. S. Bell, On the Einstein-Podolsky-Rosen paradox, Physics 1, 195 (1964).
  * Bell [1993] J. S. Bell,  _Speakable and Unspeakable in Quantum Mechanics_ (Cambridge University Press, Cambridge, 1993).
  * Clauser  _et al._ [1969] J. F. Clauser, M. A. Horn, A. Shimony, and R. A. Holt, Proposed experiment to test local hidden-variable theories, Phys. Rev. Lett. 23, 880 (1969).
  * Clauser and Horn [1974] J. F. Clauser and M. A. Horn, Experimental consequences of objective local theories, Phys. Rev. D 10, 526 (1974).
  * Eberhard [1993] P. H. Eberhard, Background level and counter efficiencies required for a loophole-free Einstein-Podolsky-Rosen experiment, Phys. Rev. A 47, R747 (1993).
  * Bell [1971] J. S. Bell, Introduction to the hidden-variable question, in  _Proceedings of the International School of Physics ‘Enrico Fermi’, course II: Foundations of Quantum Mechanics_ (New York, Academic, 1971) pp. 171–181.
  * Press  _et al._ [2003] W. H. Press, B. P. Flannery, S. A. Teukolsky, and W. T. Vetterling,  _Numerical Recipes_ (Cambridge University Press, Cambridge, 2003).
  * Cirel’son [1980] B. S. Cirel’son, Quantum generalizations of Bell’s inequality, Lett. Math. Phys. 4, 93 (1980).
  * Fine [1982a] A. Fine, Hidden variables, joint probability, and Bell inequalities, Phys. Rev. Lett. 48, 291 (1982a).
  * Fine [1982b] A. Fine, Joint distributions, quantum correlations, and commuting observables, J. Math. Phys. 23, 1306 (1982b).


#  $C\\!P$ violation in $B$ mesons

D. Manuzzi
on behalf of the LHCb collaboration,
with results from the Belle and Belle II collaborations  University of
Bologna, via Irnerio 46, Bologna, 40126, BO, Italy

###### Abstract

This proceeding reports a selection of recent experimental results concerning
$C\\!P$ violation in the sectors of ${B}^{0}$ and ${B}^{\pm}$ mesons. They
were published within the last two years by the Belle, Belle II and LHCb
collaborations. The first set of measurements is related to the determination
of the angles of the Unitarity Triangle. The second is connected to another
crucial test of the SM: the isospin sum rule of $B\rightarrow K\pi$ decays.
Thirdly, studies of $C\\!P$ violation in the decays charged $B$ mesons to
charmless three-body final states are presented. Finally, prospects for the
future experimental upgrades of this sector are summarised.

##  I Introduction

Several $C\\!P$-violating phenomena have been observed studying ${B}^{+}$ and
${B}^{0}$ mesons. Now, $C\\!P$ violation is well established in the decays of
both mesons PDG2020 . The $C\\!P$ violation in the ${B}^{0}$-${\kern
1.79993pt\overline{\kern-1.79993ptB}}{}^{0}$ mixing has not been observed yet.
However, the Standard Model (SM) predicts this effect to be suppressed below
the current experimental limit PDG2020 ; CPVmixingB . Instead, the $C\\!P$
violation in the interference between mixing and decay of neutral $B$ mesons
is well measured PDG2020 . These results strongly corroborate the SM and, in
particular, the flavour-mixing mechanism of quarks encoded by the Cabibbo-
Kobayashi-Maskawa matrix (CKM) CKM . The investigation of this sector is a
very active field. Indeed, the combination of multiple measurements provides
crucial SM consistency tests, which are very sensitive to the eventual
indirect effects of new physics (NP). Some examples come from the relations
called Unitarity Triangle (UT) and Isospin Sum Rule PDG2020 ; Gronau:2005kz .
Besides, the decays of $B$ mesons to more than two bodies have a rich
$C\\!P$-violation phenomenology, whose connection with the underlying
resonance contributions has to be scrutinised. This proceeding reviews the
most recent results on these topics. They have been obtained by the Belle,
Belle II, and LHCb collaborations Detectors . Due to the large number of
publications and the complexity of the analyses, only main aspects are
highlighted here. Finally, the programs for the future upgrades of the Belle
II and LHCb experiments are summarised, with a focus on the prospects for the
measurements introduced above.

##  II Updates to the UT angles

The SM assumes the unitarity of the CKM matrix. This condition implies various
relations among its elements. One of them is
${V_{{u}{d}}^{\phantom{\ast}}}{V_{{u}{b}}^{\ast}}+{V_{{c}{d}}^{\phantom{\ast}}}{V_{{c}{b}}^{\ast}}+{V_{{t}{d}}^{\phantom{\ast}}}{V_{{t}{b}}^{\ast}}=0$.
It can be represented as a triangle in the complex plane, whose angles are
111Two notations are available in the literature. After defining them both,
the one from the PDG PDG2020 , namely $(\alpha,\beta,\gamma)$, is used.

$\displaystyle\alpha$ $\displaystyle\equiv\phi_{2}\equiv$
$\displaystyle\arg\left[-({V_{{t}{d}}^{\phantom{\ast}}}{V_{{t}{b}}^{\ast}})/({V_{{u}{d}}^{\phantom{\ast}}}{V_{{u}{b}}^{\ast}})\right],$
$\displaystyle\beta$ $\displaystyle\equiv\phi_{1}\equiv$
$\displaystyle\arg\left[-({V_{{c}{d}}^{\phantom{\ast}}}{V_{{c}{b}}^{\ast}})/({V_{{t}{d}}^{\phantom{\ast}}}{V_{{t}{b}}^{\ast}})\right],$
$\displaystyle\gamma$ $\displaystyle\equiv\phi_{3}\equiv$
$\displaystyle\arg\left[-({V_{{u}{d}}^{\phantom{\ast}}}{V_{{u}{b}}^{\ast}})/({V_{{c}{d}}^{\phantom{\ast}}}{V_{{c}{b}}^{\ast}})\right].$

Each of them can be determined either with direct measurements or indirectly
through global fits of the CKM parameters, assuming unitarity. Any discrepancy
between direct and indirect measurements, or between measurements exploiting
different modes, would be a hint of NP. The status of the art of these
comparisons is reviewed by another talk at this conference UTfit .

Exploiting the interference between the Cabibbo-favoured ${b}\rightarrow{c}$
and the Cabibbo-suppressed ${b}\rightarrow{u}$ transitions, it is possible to
measure $\gamma$ with processes dominated by tree-level Feynman diagrams. This
is usually done by studying $B^{\pm}\rightarrow(D\rightarrow f_{D})h^{\pm}$
decays, where $h^{\pm}$ is a charged pion or kaon, and $f_{D}$ stands for a
final state shared by the decays of both ${D}^{0}$ and ${\kern
1.79993pt\overline{\kern-1.79993ptD}}{}^{0}$ mesons. In general, crucial
parameters for the achievable precision are the ratio between the magnitudes
of the amplitudes of the interfering modes ($r_{B}$) and the difference
between their strong phases ($\delta_{B}$). Depending on $f_{D}$, various
methods are available in the literature. The GLW GLW  method concerns the
cases when $f_{D}$ is a $C\\!P$ eigenstate (e.g. $f_{D}={{K}^{+}}{{K}^{-}}$).
The ADS ADS  method exploits the interference between Cabibbo-favoured
${D}^{0}$ decays and doubly-Cabibbo-suppressed ${\kern
1.79993pt\overline{\kern-1.79993ptD}}{}^{0}$ decays (e.g.
$f_{D}={{K}^{+}}{{\pi}^{-}}$). Another option is using ${D}^{0}$ and ${\kern
1.79993pt\overline{\kern-1.79993ptD}}{}^{0}$ decays $C\\!P$ self-conjugated
multi-body final states (e.g. $f_{D}={{K}^{0}_{\mathrm{S}}}h^{+}h^{-}$). In
this case, the $D$-decay phase space has to be considered. A detailed
description of the intermediate-resonance structure requires assumptions on
the $D$-decay amplitude model. This implies potentially large and difficult-
to-determine systematic uncertainties. To overcome this issue a model-
independent strategy, called BPGGSZ method BPGGSZbinned , was developed. It
utilises $C\\!P$-asymmetry measurements in disjoint regions (bins) of the
$D$-decay phase space. Such measurements are then related to $\gamma$ using
measurements of $D$-decay parameters available in the literature
BESIII:DdecayParameters .

The Belle and Belle II collaborations have recently published a joint $\gamma$
determination following the BPGGSZ method. The full decay chain is
${{B}^{\pm}}\rightarrow(D\rightarrow{{K}^{0}_{\mathrm{S}}}h^{+}h^{-})h^{-}$
where $h\in\\{\pi,K\\}$ and the integrated luminosity is $711\text{\,fb}^{-1}$
from Belle plus $128\text{\,fb}^{-1}$ from Belle II. The final result is
gammaBelleBelle2 :

$\displaystyle\gamma$ $\displaystyle=$ $\displaystyle(78.4\pm 11.4\pm 0.5\pm
1.0)^{\circ},$

where the uncertainties are statistical, systematic and due to external
inputs, respectively. This is the most precise determination of $\gamma$ at
the $B$-factories. The improvement compared to the previous Belle results is
equivalent to doubling the statistics. This was possible thanks to the
inclusion of the $D\rightarrow{{K}^{0}_{\mathrm{S}}}{{K}^{+}}{{K}^{-}}$ decay
mode and improvements in the ${K}^{0}_{\mathrm{S}}$ selection and background
suppression. A further decisive advantage is also taken from the inputs
updated by BESIII BESIII:DdecayParameters .

In 2021 the LHCb collaboration published the combination of all its direct
$\gamma$ measurements. It includes all the procedures cited above, but also
methods based on the measurements of time-dependent decay rates to exploit the
interference due to the mixing of ${B}_{({s})}^{0}$ and ${\kern
1.79993pt\overline{\kern-1.79993ptB}}{}_{({s})}^{0}$ mesons. The data were
collected during the Run1 and Run2 of the LHC, with a total integrated
luminosity of 9$\text{\,fb}^{-1}$. The combination follows a frequentist
approach gammaLHCbCombMethod , with the novel inclusion of inputs from the
charm sector. The final results is gammaLHCbComb :

$\displaystyle\gamma$ $\displaystyle=$
$\displaystyle(65.4^{+3.8}_{-4.2})^{\circ}.$

This is the most precise determination of $\gamma$ by a single experiment. It
is statistically dominated and agrees with the indirect determinations from
global CKM fits. In Fig. 1 the contributions to the LHCb $\gamma$ combination
coming from the ${B}^{0}$, ${B}^{+}$, and ${B}^{0}_{s}$ species are compared.
A $2\sigma$ tension between the results from ${B}^{0}$ and ${B}^{+}$ sectors
is observed.

Figure 1: One dimensional 1-CL profiles for the LHCb $\gamma$ combinations
using inputs from ${B}^{0}_{s}$ (light blue), ${B}^{0}$ (orange), ${B}^{+}$
mesons (red) and all species together (dark blue) gammaLHCbComb .

The new combination method permitted the simultaneous determination the
${D}^{0}$ mixing parameters. In particular, the precision on
$y_{D}\equiv\Delta\Gamma/\Gamma$ was improved by a factor of two with respect
to the previous world average.

Despite the release of this combination, the analysis of LHCb data is still
ongoing. A more recent $\gamma$ measurement, concerns
${B}^{\pm}\rightarrow(D\rightarrow
h^{+}h^{\prime-}{{\pi}^{0}})h^{\prime\prime\pm}$ decays, with
$h,h^{\prime},h^{\prime\prime}\in\\{\pi,K\\}$ gammaLHCbB2Dh . The case with
$h=h^{\prime}$ are treated as “quasi-GLW”, while the channels with $h\neq
h^{\prime}$ are “quasi-ADS”. In the first case, the
${D}\rightarrow{{\pi}^{+}}{{\pi}^{-}}{{\pi}^{0}}$ and
${D}\rightarrow{{K}^{+}}{{K}^{-}}{{\pi}^{0}}$ final states are admixtures of
$C\\!P$-even and $C\\!P$-odd eigenstates. In the second case, the interference
effects that are sensitive to $\gamma$ vary over the phase space of the $D$
decay due to the strong decays of intermediate resonances. To deal with that,
dilution factors determined by measurements of CLEO-$c$ and BESIII
gammaLHCbB2DhInputs ; BESIII:2021eud ; Malde:2015mha  are exploited. This
permits $\gamma$ to be calculated, combining eleven $C\\!P$ observables
described in Ref. gammaLHCbB2Dh . They are all measured with world best
precision by this analysis. The final results is gammaLHCbB2Dh

$\gamma=(56^{+24}_{-19})^{\circ}.$

It shall be used in future combinations to constrain the angle $\gamma$. The
total reported uncertainty is largely dominated by the statistical
uncertainty. In addition, the suppressed
${{{B}^{+}}}\rightarrow(D\rightarrow{{\pi}^{+}}{{K}^{-}}{{\pi}^{0}}){{K}^{+}}$
decay is observed for the first time, with a significance of almost
$8\,\sigma$.

The UT angle $\beta$ can be measured using decays of neutral $B$ mesons to
$C\\!P$ eigenstates, $f_{C\\!P}$, in common between ${B}^{0}$ and ${\kern
1.79993pt\overline{\kern-1.79993ptB}}{}^{0}$. In these cases, assuming
$C\\!PT$ invariance and negligible $C\\!P$ violation in the mixing, the time-
dependent $C\\!P$ symmetry can be written as222The notation for the time-
dependent $C\\!P$ asymmetries defined by the PDG is used here. Compared to the
one used in Ref. Belle:CPVB2KSKSKS , it implies $C_{f}\\!=\\!-\mathcal{A}$.

$\displaystyle\tfrac{\Gamma_{{{\kern
0.89996pt\overline{\kern-0.89996ptB}}{}^{0}}\rightarrow
f_{C\\!P}}(t)-\Gamma_{{{B}^{0}}\rightarrow f_{C\\!P}}(t)}{\Gamma_{{{\kern
0.89996pt\overline{\kern-0.89996ptB}}{}^{0}}\rightarrow
f_{C\\!P}}(t)+\Gamma_{{{B}^{0}}\rightarrow f_{C\\!P}}(t)}$ $\displaystyle=$
$\displaystyle+S_{f_{C\\!P}}\sin({\Delta m_{{d}}}t)$ $\displaystyle-
C_{f_{C\\!P}}\cos({\Delta m_{{d}}}t),$

where $\Delta m_{{d}}$ is the ${B}^{0}$-${\kern
1.79993pt\overline{\kern-1.79993ptB}}{}^{0}$ oscillation frequency, and the
$C\\!P$ violation in the decay and in the interference between mixing and
decay are encoded by the parameters $C_{f_{C\\!P}}$ and $S_{f_{C\\!P}}$,
respectively. When only one CKM phase is present in the amplitudes which
dominates the transition, the SM predicts $C_{f_{C\\!P}}=0$ and
$S_{f_{C\\!P}}=-\sin(2\beta)$. The theoretically cleanest process is
${{B}^{0}}\rightarrow{{J\mskip-3.0mu/\mskip-2.0mu\psi}}{{K}^{0}_{\mathrm{S}}}$,
which proceeds at tree level. However, also channels dominated by a loop-level
topology are possible. They are particularly interesting because NP in the
loops may have implications on the values of $C_{f_{C\\!P}}$ and
$S_{f_{C\\!P}}$. The Belle collaboration recently studied with its full
dataset (711 $\text{\,fb}^{-1}$) one of these cases: the
$B^{0}\rightarrow{{K}^{0}_{\mathrm{S}}}{{K}^{0}_{\mathrm{S}}}{{K}^{0}_{\mathrm{S}}}$
decay belle:CPVB2KSKSKS:penguin . The analysis utilises a three-dimensional
fit to extract signal and background yields. The latter is mainly due to
continuum background, namely
${e^{+}}{e^{-}}\rightarrow{{u}{\overline{u}}},{{d}{\overline{d}}},{{s}{\overline{s}}},{{c}{\overline{c}}}$
processes. It is suppressed with a Neural Network classifier. The classifier
output is also an observable of the fit. The other ones are the beam-energy-
constrained mass $M_{bc}=\sqrt{E^{2}_{\rm beam}-|\vec{p}_{B}|^{2}}$ and the
energy difference $\Delta E=E_{B}-E_{\rm beam}$. The time-dependent $C\\!P$
asymmetry is determined with a second fit. Besides the signal-to-background
fraction obtained in the first step, it use information about the flavour of
the $B$ meson and its decay time. Since neutral mesons produced by the process
${\mathchar 28935\relax{(4S)}}\rightarrow{{B}^{0}}{{\kern
1.79993pt\overline{\kern-1.79993ptB}}{}^{0}}$ are selected, the quark content
of the signal ${B}^{0}$ (${\kern 1.79993pt\overline{\kern-1.79993ptB}}{}^{0}$)
can be tagged when a flavour-specific decay of the companion ${\kern
1.79993pt\overline{\kern-1.79993ptB}}{}^{0}$ (${B}^{0}$) is reconstructed. The
time information derives from the difference, $\Delta t$, between the decay
times of the signal and tagging $B$, measured in their center-of-mass frame.
Fig. 2 displays the time-dependent $C\\!P$ asymmetry observed in data with the
results of the best fit superimposed. The determinations of this analysis are
Belle:CPVB2KSKSKS

$\displaystyle S_{3{{K}^{0}_{\mathrm{S}}}}$ $\displaystyle=$
$\displaystyle-0.71\pm 0.23{\rm(stat.)}\pm 0.05{\rm(syst.)},$ $\displaystyle
C_{3{{K}^{0}_{\mathrm{S}}}}$ $\displaystyle=$ $\displaystyle-\,0.12\pm
0.16{\rm(stat.)}\pm 0.05{\rm(syst.)}.$

They supersede the previous Belle results thanks to the improvements in the
${K}^{0}_{\mathrm{S}}$ selection, background suppression, vertex
reconstruction, and increased statistics. These values agree with the world
average for $-\sin(2\beta)$ HFLAV2019  and the null SM prediction for
$C_{3{{K}^{0}_{\mathrm{S}}}}$. The significance of $C\\!P$ violation in this
mode is determined to be $2.5\,\sigma$ away from $(0,0)$.

Figure 2: Time dependent $C\\!P$ asymmetry distribution obtained from
$B^{0}\rightarrow{{K}^{0}_{\mathrm{S}}}{{K}^{0}_{\mathrm{S}}}{{K}^{0}_{\mathrm{S}}}$
candidates (black dots) and curve coming from the result of the best fit (red
line) Belle:CPVB2KSKSKS .

The measurement of the UT angle $\alpha$ relies on
${b}\rightarrow{u}{\overline{u}}{d}$ transitions. Due to the Cabibbo
suppression of the tree-level topology, loop-level amplitudes may be sizeable.
Therefore, hadronic contributions, which are difficult to be precisely
measured, complicate the determination of $\alpha$ from $B$ decays. A possible
approach consists of combining measurements of decays related to isospin
symmetries. The complete set of ${B}\rightarrow\rho\rho$ isospin partners is
valuable for this purpose BelleII:B2rhorho:alphaMotivations . Belle II is
expected to be able to study them jointly and within the same experimental
environment. As a first step, it recently exploited the 190 $\text{\,fb}^{-1}$
data collected so far to analyse the
${{{B}^{+}}}\rightarrow{{\rho}^{+}}{{\rho}^{0}}$ decay. Relevant inputs for
the $\alpha$ determination are its branching ratio, $C\\!P$ asymmetry, and
longitudinal polarization fraction, $f_{L}$. The whole reconstructed decay
chain includes
${{\rho}^{+}}\rightarrow{{\pi}^{+}}({{\pi}^{0}}\rightarrow\gamma\gamma)$ and
${{\rho}^{0}}\rightarrow{{\pi}^{+}}{{\pi}^{-}}$ decays. To measure $f_{L}$,
the helicity angles between the momentum of the positive-charged pions and the
direction opposite to the ${{B}^{+}}$ momentum are considered. The large
continuum background is reduced with multivariate methods. The final fit for
the parameter extraction included 6D templates obtained from simulation and
corrected with calibration samples, as described in Ref. Belle-II:B2rhorho .
The raw charge asymmetry determined by the fit is then corrected for the
detection asymmetry due to instrumental effects. It is measured with a
${{D}^{+}}\rightarrow{{K}^{0}_{\mathrm{S}}}{{\pi}^{+}}$ control channel. The
final results are Belle-II:B2rhorho

$\displaystyle A_{C\\!P}({{{B}^{+}}}\rightarrow{{\rho}^{+}}{{\rho}^{0}})$
$\displaystyle=$ $\displaystyle-0.069\pm 0.068\pm 0.060,$
$\displaystyle\mathcal{B}({{{B}^{+}}}\rightarrow{{\rho}^{+}}{{\rho}^{0}})$
$\displaystyle=$ $\displaystyle(23.2^{+2.2}_{-2.1}\pm 2.7)\times 10^{-6},$
$\displaystyle f_{L}$ $\displaystyle=$ $\displaystyle
0.943^{+0.035}_{-0.033}\pm 0.027,$

where the first uncertainty is statistical and the second is systematic. The
determinations are consistent with the measurements available in the
literature PDG2020 . This is the first measurement of
$A_{C\\!P}({{{B}^{+}}}\rightarrow{{\rho}^{+}}{{\rho}^{0}})$ by Belle II. The
systematic uncertainties are mainly data-driven, hence they are expected to
decrease by exploiting the future datasets.

##  III Updates to the $\bm{K\pi}$ puzzle

Isospin relations would suggest the same value for the $C\\!P$ asymmetries of
${{{B}^{+}}}\rightarrow{{K}^{+}}{{\pi}^{0}}$ and
${{B}^{0}}\rightarrow{{K}^{+}}{{\pi}^{-}}$ decays. However, their world
averages, recently updated by LHCb LHCb:CPV:B2Kpi , show a $8\,\sigma$
discrepancy. This peculiarity constitutes the long-standing “$K\pi$ puzzle”.
As explained in Ref. Gronau:2005kz , a more accurate examination of this
anomaly leads to the following sum rule, which also concerns the $C\\!P$
asymmetries in the ${{B}^{0}}\rightarrow{{K}^{0}}{{\pi}^{0}}$ and
${{{B}^{+}}}\rightarrow{{K}^{0}}{{\pi}^{+}}$:

$\displaystyle A_{{C\\!P}}(B^{0}\rightarrow K^{+}\pi^{-})$
$\displaystyle\mathcal{B}(B^{0}\rightarrow K^{+}\pi^{-})\tau_{{{B}^{+}}}$
$\displaystyle+$ $\displaystyle A_{{C\\!P}}(B^{+}\rightarrow K^{0}\pi^{+})$
$\displaystyle\mathcal{B}(B^{+}\rightarrow K^{0}\pi^{+})\tau_{{B}^{0}}$
$\displaystyle-$ $\displaystyle 2A_{{C\\!P}}(B^{+}\rightarrow K^{+}\pi^{0})$
$\displaystyle\mathcal{B}(B^{+}\rightarrow K^{+}\pi^{0})\tau_{{B}^{0}}$
$\displaystyle-$ $\displaystyle 2A_{{C\\!P}}(B^{0}\rightarrow K^{0}\pi^{0})$
$\displaystyle\mathcal{B}(B^{0}\rightarrow K^{0}\pi^{0})\tau_{{{B}^{+}}}=0,$

where $\tau_{{{B}^{+}}}$ ($\tau_{{B}^{0}}$) is the lifetime of the ${{B}^{+}}$
(${B}^{0}$) meson. Deviations above the $1\%$ level from this crucial
condition are not allowed within the SM. The observable with higher
uncertainty is the $C\\!P$ asymmetry of the
${{B}^{0}}\rightarrow{{K}^{0}}{{\pi}^{0}}$ decay.

Figure 3: Projections on the $\Delta E$ and $\Delta t$ axes of the fit used to
determine $A_{C\\!P}({{B}^{0}}\rightarrow{{K}^{0}}{{\pi}^{0}})$. More details
in Ref. Belle-II:B2KSpi0 .

Currently, the sum-rule predicts
$A_{{C\\!P}}({{B}^{0}}\rightarrow{{K}^{0}}{{\pi}^{0}})=-0.138\pm 0.025$, using
the world averages of the other quantities HFLAV2019 . Instead, the
combination of the direct measurements by Belle and BaBar returns
$A_{{C\\!P}}({{B}^{0}}\rightarrow{{K}^{0}}{{\pi}^{0}})=0.01\pm 0.10$, strongly
requiring further inputs PDG2020 . The Belle II collaboration recently updated
this value using the 190 $\text{\,fb}^{-1}$ data collected so far. The
measurement analyses the decays of $B$ mesons to the ${K}^{0}_{\mathrm{S}}$
${\pi}^{0}$ final states. The signal $B$ vertex is obtained by projecting back
to the interaction region the flight direction of the ${K}^{0}_{\mathrm{S}}$
candidate. This provides a good approximation of the signal $B$ decay vertex
since both the transverse flight length of the ${B}^{0}$ meson and the
transverse size of the interaction region are small compared to the ${B}^{0}$
flight length along the boost direction. Another challenge is the suppression
of the continuum background. A Boost Decision Tree algorithm is trained for
this purpose. The ${{K}^{0}_{\mathrm{S}}}{{\pi}^{0}}$ final state cannot be
used to tag the $B$ flavour at the decay. Therefore, the time-dependent
$C\\!P$ asymmetry is considered. The tagging information is obtained from the
companion $B$ meson produced by the primary $\mathchar 28935\relax{(4S)}$
resonance. The values of the $\tau_{{B}^{0}}$, $\Delta m_{{d}}$, and
$S_{{{K}^{0}_{\mathrm{S}}}{{\pi}^{0}}}$ are fixed to their world average to
maximise the sensitivity on
$A_{{C\\!P}}({{B}^{0}}\rightarrow{{K}^{0}}{{\pi}^{0}})$. The $\Delta t$
resolution and bias are calibrated with
${{B}^{0}}\rightarrow{{J\mskip-3.0mu/\mskip-2.0mu\psi}}{{K}^{0}_{\mathrm{S}}}$
decays. Fig. 3 illustrates the projections of the fit on the $\Delta E$ and
$\Delta t$ axes. The final results are Belle-II:B2KSpi0

$\displaystyle A_{{C\\!P}}(B^{0}\rightarrow K^{0}\pi^{0})$ $\displaystyle=$
$\displaystyle-0.41^{+0.30}_{-0.32}\pm 0.09,$
$\displaystyle\mathcal{B}(B^{0}\rightarrow{{K}^{0}}\pi^{0})$ $\displaystyle=$
$\displaystyle(11.0\pm 1.2\pm 1.0)\times 10^{-6}$

This is the first determination of $A_{{C\\!P}}(B^{0}\rightarrow
K_{S}^{0}\pi^{0})$ using a time-dependent analysis at Belle II. The results
agree with the previous measurements HFLAV2019 ; Belle-
II:B2KSpi0:timeIntegrated .

##  IV $\bm{B}$ decays to charmless 3-body final states

A long-standing debate about the role of short- and long-distance
contributions to the generation of the strong-phase differences, needed for
direct $C\\!P$ violation to occur, is ongoing in literature. The three-body
decays of $B$ mesons offer a way to solve it Bediaga:2020qxg . Using Run1 data
LHCb showed evidence of global direct $C\\!P$-violation and high localised
$C\\!P$ asymmetries across the Dalitz plot in charmless 3-body $B$ decays
LHCb:2014mir . Amplitude analysis of
${{B}^{\pm}}\rightarrow{{\pi}^{\pm}}{{\pi}^{+}}{{\pi}^{-}}$ connected the
large $C\\!P$ violation to the interference between S- and P-waves. A similar
analysis related the ${{\pi}^{+}}{{\pi}^{-}}\leftrightarrow{{K}^{+}}{{K}^{-}}$
rescattering LHCb:B2hhh:prequels  to the large $C\\!P$ violation observed in
${{B}^{\pm}}\rightarrow{{\pi}^{\pm}}{{K}^{+}}{{K}^{-}}$ decays. LHCb has just
updated these results with the data collected during the Run2
(6$\text{\,fb}^{-1}$). The
${{B}^{\pm}}\rightarrow{{K}^{\pm}}{{\pi}^{+}}{{\pi}^{-}}$ and
${{B}^{\pm}}\rightarrow{{K}^{\pm}}{{K}^{+}}{{K}^{-}}$ are also included. The
raw asymmetries between the yields of the ${{B}^{+}}$ and ${{B}^{-}}$ mesons
are determined with a simultaneous fit to invariant mass spectra of the eight
final states. This permits the cross-feed backgrounds due to misidentification
of the final state mesons to be better handled. This background source is
particularly relevant because it peaks close to the signal. After that,
corrections for the experimental effect are applied. The asymmetry due to the
different detection efficiency of the charge conjugated final states is
estimated with simulated candidates, calibrated with data-driven techniques.
The proton-proton collisions of the LHC cause a slight difference between the
production rates of ${{B}^{+}}$ and ${{B}^{-}}$ mesons. This production
asymmetry is estimated as the difference between the raw asymmetry of the
${{B}^{\pm}}\rightarrow{{J\mskip-3.0mu/\mskip-2.0mu\psi}}{{K}^{\pm}}$ decay
and the world average of its $C\\!P$ asymmetry, which is treated as an
external input. The global $C\\!P$ asymmetries of the signal decay modes are
determined to be LHCb:B2hhh

$\displaystyle A_{C\\!P}(B^{\pm}\rightarrow K^{\pm}\pi^{+}\pi^{-})$
$\displaystyle=$ $\displaystyle+0.011\pm 0.002\pm 0.003\pm 0.003$
$\displaystyle A_{C\\!P}(B^{\pm}\rightarrow\pi^{\pm}K^{+}K^{-})$
$\displaystyle=$ $\displaystyle-0.144\pm 0.007\pm 0.003\pm 0.003$
$\displaystyle A_{C\\!P}(B^{\pm}\rightarrow K^{\pm}K^{+}K^{-})$
$\displaystyle=$ $\displaystyle-0.037\pm 0.002\pm 0.002\pm 0.003$
$\displaystyle A_{C\\!P}(B^{\pm}\rightarrow\pi^{\pm}\pi^{+}\pi^{-})$
$\displaystyle=$ $\displaystyle+0.080\pm 0.004\pm 0.003\pm 0.003$

where the uncertainties are statistical, systematic, and due to external
inputs, respectively. Large global $C\\!P$ violation is measured for the
latter three decay modes. In the last two cases, it is observed for the first
time. Besides, the analysis was replicated in different regions (bins) of the
Dalitz plot, as shown in Fig. 4.

Figure 4: $C\\!P$ asymmetry distribution in bins of the Dalitz plot for
$B^{\pm}\rightarrow K^{\pm}\pi^{+}\pi^{-}$ (top),
$B^{\pm}\rightarrow\pi^{\pm}K^{+}K^{-}$ (middle-top), $B^{\pm}\rightarrow
K^{\pm}K^{+}K^{-}$ (middle-bottom), and
$B^{\pm}\rightarrow\pi^{\pm}\pi^{+}\pi^{-}$ (bottom) decays LHCb:B2hhh .

The $C\\!P$ asymmetries are not uniformly distributed in the phase space, with
positive and negative $A_{C\\!P}$ appearing in the same charged B decay
channel. The results of the previous LHCb analysis are confirmed. Significant
$C\\!P$ violation is present for all the analysed channels in the
${{\pi}^{+}}{{\pi}^{-}}\leftrightarrow{{K}^{+}}{{K}^{-}}$ rescattering region,
namely between 1 and 2.25 $\text{\,Ge\kern-1.00006ptV}^{2}\\!/c^{4}$ in both
${\pi}^{+}$ ${\pi}^{-}$ and ${K}^{+}$ ${K}^{-}$ invariant masses. The
combination of the present $A_{C\\!P}$ results with the recent LHCb relative
branching fraction results for these decays shows good agreement with the
U-spin symmetry proposed in Ref. LHCb:B2hhh:Uspin .

##  V Prospects

Many of the $C\\!P$ violation observables in the $B$ sector are dominated by
statistical uncertainty. This is one of the motivations leading to the
foreseen upgrades of the LHCb and Belle II detectors. LHCb is expected to
increase its dataset to 23$\text{\,fb}^{-1}$ before the end of 2026, touching
50$\text{\,fb}^{-1}$ within 2032. In the same period, the target of Belle II
is collecting a total of 50$\text{\,ab}^{-1}$. Several studies have already
started for the next phase, setting the LHCb goal to 300$\text{\,fb}^{-1}$ and
the Belle II target to 250$\text{\,ab}^{-1}$. Details about the crucial
experimental challenges and the future physics programmes of both
collaborations are documented in Ref. Belle2:future ; LHCb:future . Table 5
summarises the expected uncertainties on the direct measurements of the UT
angles. LHCb and Belle II will provide comparable contributions to the
determination of $\sin 2\beta$. LHCb is expected to be more competitive for
the $\gamma$ measurement, whereas Belle II will achieve the most precise
determinations of $\alpha$. The comparison between the current and future
accuracy of the UT tests is illustrated in Fig. 5 by the plots from the
CKMfitter collaboration.

The further $C\\!P$ violation features cited above will be also deeply
investigated. Belle II is expected to dominate the measurement of
$A_{{C\\!P}}({{B}^{0}}\rightarrow{{K}^{0}}{{\pi}^{0}})$, testing the isospin
sum rule at $3\%$ level with $50\text{\,ab}^{-1}$. Instead, concerning $B$
decays to charmless three-body final states, LHCb is expected to collect far
larger yields with a much better signal-to-noise ratio.

In conclusion, this proceeding presented a selection of novel experimental
results about the $C\\!P$ violation in the $B$ sector. They are just examples
of the intense work ongoing in this field. It can provide powerful tests of
the SM consistency. They currently corroborate the SM, but a much higher level
of precision is expected for the years to come. This challenge will benefit
from the competition and complementarity of the LHCb and Belle II physic
programmes.

Dataset | $\bm{\alpha}$ | $\bm{\sin 2\beta}$ | $\bm{\gamma}$
---|---|---|---
Current world average | $4.8^{\circ}$ | $0.017$ | $3.5^{\circ}$
LHCb, 23$\text{\,fb}^{-1}$ |  | $0.006$ | $1.5^{\circ}$
Belle II, 50$\text{\,ab}^{-1}$ | $0.6^{\circ}$ | $0.005$ | $1.5^{\circ}$
LHCb, 300$\text{\,fb}^{-1}$ |  | $0.003$ | $0.35^{\circ}$
Belle II, 250$\text{\,ab}^{-1}$ | $0.3^{\circ}$ | $0.002$ | $0.8^{\circ}$
Table 1: Current and expected uncertainty on the direct determinations of the
UT angles Belle2:future ; LHCb:future ; HFLAV2019 .

Figure 5: Allowed parameter ranges for the UT parameters CKMfitter . The left
plot is the current status of the art. The right plot assumes the current
central values and includes the precision improvement foreseen for the future
for both experiment and theory inputs.

## References

  * (1) P. A. Zyla et al. (Particle Data Group), PTEP 2020, 083C01 (2020).
  * (2) M. Artuso, G. Borissov and A. Lenz, Rev. Mod. Phys. 88, 045002 (2016).
  * (3) N. Cabibbo, Phys. Rev. Lett., 10 (1963) 531. M. Kobayashi and T. Maskawa, Prog. Theor. Phys., 49 (1973) 652.
  * (4) M. Gronau, Phys. Lett. B 627, 82-88 (2005).
  * (5) A. Abashian et al. (Belle collab.), Nucl. Instrum. Methods Phys. Res., Sect. A 479, 117 (2002). T. Abe et al. (Belle II collab.), KEK Report 2010-1, arXiv:1011.0352. A. A. Alves Jr. et al. (LHCb collab.), JINST 3 (2008) S08005.
  * (6) M. Bona et al. (UTfit collab.), JHEP 10 (2006) 081, updates at: http://www.utfit.org/UTfit/
  * (7) M. Gronau and D. London, Phys. Lett. B253, 483 (1991). M. Gronau and D. Wyler, Phys. Lett. B265, 172 (1991).
  * (8) D. Atwood, I. Dunietz and A. Soni, Phys. Rev. Lett. 78, 3257 (1997), Phys. Rev. D63, 036005 (2001).
  * (9) A. Bondar, Phys. Rev. D70, 072003 (2004).
  * (10) A. Giri et al., Phys. Rev. D68, 054018 (2003).
  * (11) M. Ablikim et al. (BESIII collab.), Phys. Rev. D 101 (2020) 112002, Phys. Rev. D 102 (2020) 052008.
  * (12) F. Abudinén, L. Aggarwal et al. (Belle and Belle II collaborations), J. High Energ. Phys. 2022, 63 (2022).
  * (13) R. Aaji et al. (LHCb collab.), Phys. Lett. B 726 (2013) 151
  * (14) R. Aaji et al. (LHCb collab.), J. High Energ. Phys. 2021, 141 (2021).
  * (15) R. Aaji et al. (LHCb collab.), J. High Energ. Phys. 2022, 99 (2022).
  * (16) T. Evans et al., Phys. Lett. B 765, 402-403 (2017).
  * (17) M. Ablikim et al. (BESIII collab.), JHEP 05, 164 (2021).
  * (18) S. Malde et al. Phys. Lett. B 747, 9-17 (2015)
  * (19) Y. Grossman and M. P. Worah, Phys. Lett. B 395, 241 (1997).
  * (20) K. H. Kang et al. (Belle collab.), Phys. Rev. D 103, 032003.
  * (21) Y. S. Amhis et al. (HFLAV Group), arXiv:1909.12524 (2019); and updates at https://hflav.web.cern.ch/
  * (22) T. Bloomfield et al. (Belle collab.), Phys. Rev. D 105, 072007.
  * (23) F. Abudinén et al. (Belle II collab.), arXiv:2206.12362.
  * (24) M. Gronau and D. London, Phys. Rev. Lett. 65 (1990) 3381.
  * (25) R. Aaij et al. (LHCb collab.), JHEP 03 (2021), 075, Phys. Rev. Lett. 126 (2021) no.9, 091802.
  * (26) F. Abudinén et al. (Belle II collab.), arXiv:2206.07453.
  * (27) F. Abudinén et al. (Belle II collab.), BELLE2-CONF-PH-2021-001.
  * (28) R. Aaij et al. (LHCb collab.), Phys. Rev. D 90 (2014) no.11, 112004.
  * (29) I. Bediaga and C. Göbel, Prog. Part. Nucl. Phys. 114 (2020), 103808.
  * (30) R. Aaij et al. (LHCb collab.), Phys. Rev. Lett. 124 (2020) no.3, 031801, Phys. Rev. D 101 (2020) no.1, 012006, Phys. Rev. Lett. 123 (2019) no.23, 231802.
  * (31) R. Aaji et al. (LHCb collab.), arXiv:2206.07622.
  * (32) M. Gronau, J. L. Rosner et al., Phys. Lett. B564 (2003) 90, Phys. Lett. B726 (2013) 337, Phys. Rev. D89 (2014) 074043.
  * (33) F. Forti et al. (Belle II collab.), arXiv:2203.11349. D. Tonelli et al. (Belle II collab.), arXiv:2207.06307.
  * (34) R. Aaji et al. (LHCb collab.), CERN-LHCC-2021-012, arXiv:1808.08865.
  * (35) J. Charles et al. (CKMfitter Group), Eur. Phys. J. C41, 1-131 (2005), updates at: http://ckmfitter.in2p3.fr