example[Example][List of examples] prompt[Prompt][List of prompts]
equ[Equation][List of equations]

# Quality and Quantity of Machine Translation
References for Automated Metrics

Vilém Zouhar
ETH Zürich
<EMAIL_ADDRESS>&Ondřej Bojar
Charles University
<EMAIL_ADDRESS>
###### Abstract

Automatic machine translation metrics often use human translations to
determine the quality of system translations. Common wisdom in the field
dictates that the human references should be of very high quality. However,
there are no cost-benefit analyses that could be used to guide practitioners
who plan to collect references for machine translation evaluation.

We find that higher-quality references lead to better metric correlations with
humans at the segment-level. Having up to 7 references per segment and taking
their average helps all metrics. Interestingly, the references from vendors of
different qualities can be mixed together and improve metric success. Higher
quality references, however, cost more to create and we frame this as an
optimization problem: given a specific budget, what references should be
collected to maximize metric success. These findings can be used by evaluators
of shared tasks when references need to be created under a certain budget.

##  1 Introduction

††footnotetext: 0github.com/ufal/optimal-reference-translations

Machine translation systems are robustly evaluated through human annotation.
This is non-scaleable and non-replicable (Freitag et al., 2021a) for settings
such as shared tasks where number of teams submit automatic translations of
the same testset. Automatic metrics aim to provide a cheap and replicable
solution. Given the translation and possibly the source and reference
segments, they produce a score that correlates with what a human annotator
would predict. There is support and evidence for not using references (Lommel,
2016) in metrics, i.e. quality estimation (Specia and Shah, 2018; Rei et al.,
2021). Still, most of the commonly used metrics (Section 3) require human
reference translations (Freitag et al., 2023). These metrics work by comparing
either the overlap on the surface-level (e.g. BLEU, Papineni et al., 2002) or
semantic representations (e.g. Comet, Rei et al., 2020).

Humans also do not always arrive at the perfect translation and thus the
quality of the references themselves varies (Castilho et al., 2018). Given
some cases of very poor translations, such as non-translation,111Keeping the
text untouched in the source language. the reference-based metrics would
clearly fail. While low-quality references are known to decrease the metric
correlations (Freitag et al., 2023), the extent of this effect and
interactions with other phenomena remains unclear. Many automatic machine
translation metrics support multiple references for a single translation
natively or by using an average. The usefulness of multiple references is
disputed (Freitag et al., 2020) for state-of-the-art system evaluation.
Additionally, a professional experienced translator is likely to produce a
better translation than an average crowd-worker. However, the cost of a high-
quality human translation is likely also much higher. In this paper, we aim to
quantify the trade-off between reference quality, quantity and cost for
segment-level automated metric performance. We base our experiments on a
small-scale English$\rightarrow$Czech dataset with multiple references of
varying qualities.

We pose a series of research questions with immediate implications for
practitioners who evaluate machine translation systems. The short answers here
are only summaries and more elaborate discussions are in their respective
sections.

  *   * Q:

Are higher-quality references useful? (Sec. 4.1)

  * A:

Yes. Low-quality references should be avoided. Too much investment has
diminishing returns.

  * Q:

Are multiple references useful? (Sec. 4.2)

  * A:

Yes. Averaging or taking the maximum across reference improves the metrics.

  * Q:

How to allocate a reference budget? (Sec. 4.3)

  * A:

By not focusing exclusively on either quality or quantity of references but
their combination. This can be computed by Algorithm 1, given a list of
vendors and their attributes.

##  2 Related Work

Reference quality is known to affect machine translation evaluation. Freitag
et al. (2023) note that specifically very low-quality references reduce metric
success. This stands in contrast to the pre-neural machine translation era
where the reference quality did not play a role (Hamon and Mostefa, 2008).
This is likely caused by the much higher quality of systems being compared.
Vernikos et al. (2022) hypothesize that ambiguous references are the culprits
of metric success deterioration. Additionally, Freitag et al. (2020) study
preventing low-quality references in human translation campaigns.

The BLEU metric (Papineni et al., 2002) was intended to be used with multiple
metrics, which is not a common practice anymore. Nevertheless, newer and more
sophisticated methods exist to incorporate them (Qin and Specia, 2015). Our
results from Figure 1 confirm the older study of Finch et al. (2004) which
studies the effect of the reference count on metric performance. Finally,
multiple references can be used in training better machine translation systems
(Madnani et al., 2008; Zheng et al., 2018; Khayrallah et al., 2020; Mi et al.,
2020) or for analyzing model uncertainty (Ott et al., 2018) or evaluation
uncertainty (Zhang and Vogel, 2004, 2010; Fomicheva et al., 2020).

The budget allocation algorithm is reminiscent of active learning or data
selection. In machine translation, the research is limited to selecting
example for training (Haffari et al., 2009; González-Rubio et al., 2012; van
der Wees et al., 2017; Shi and Huang, 2020; Mendonça et al., 2023). By
comparison, we focus on algorihtmic data selection for higher-quality
evaluation.

Our work aims to complete similar works on practical advice on machine
translation evaluation. Kocmi et al. (2021) study the reliability of metrics
with the perspective of deployment decisions. We show that the configuration
of references can make these metrics stronger on weaker on segment-level.

##  3 Setup

To evaluate the effect of references on automatic machine translation
evaluation, we need data with controlled references and reference-based
metrics.

#### Optimal Reference Translations.

Zouhar et al. (2023) re-evaluated in detail a subset of the
English$\rightarrow$Czech testset from the News domain used in the WMT2020
campaign (Barrault et al., 2020). New data was created by translating the
original source in 4 different human settings ranging from generic translation
vendors to academic translatologists. This phase was followed up with a human
annotation and post-editing phase performed by 11 annotators of varying
professionalities.

Zouhar et al. (2023) study whether the human quality of the references is
really the highest achievable one. They stop short of evaluating the impact of
this on machine translation evaluation. We re-purpose their data and system
submissions from Barrault et al. (2020). We refer to the references, from
lowest to highest quality of the source, as R1, R2, R3, and R4. Specifically,
R1 to R2 come from standard translation vendors, R3 is high-quality
translation vendor, and R4 is the work of a translatologist. See more details
in Table 1.

#### Automated Metrics.

For the metrics, we use BLEU (Papineni et al., 2002), ChrF (Popović, 2015),
TER (Snover et al., 2006), Comet${}^{20}_{\textrm{}}$ (Rei et al., 2020), its
referenceless version Comet${}^{20}_{\textrm{QE}}$, and its updated iteration
Comet${}^{22}_{\textrm{}}$ (Rei et al., 2022), and BLEURT (Sellam et al.,
2020). We select those as a representative set of widely-used string-matching
and trainable metrics. We focus on and evaluate metric success at the segment-
level (“sentence”-level) by correlating the metric scores with human scores
using Kendall’s $\bm{\tau}$.222 $\bm{\tau}=(\textrm{\\#concordant
pairs}-\textrm{\\#discordant pairs})/\textrm{\\#pairs}$ Each translation
receives a human score and automated metric scores which are correlated. This
is the standard segment-level evaluation adopted by the WMT metrics shared
task (Freitag et al., 2021b, 2022, 2023).

Source segments & documents |  $160$ & $20$
---|---
Average source segment length |  $34$ tokens
Reference segments | $160{\times}4=640$
Reference post-editing | $160{\times}4{\times}11=7040$
Systems & system segments |  $13$ & $160{\times}13=2080$

Table 1: Overview of the used dataset.

##  4 Experiments

###  4.1 Reference Quality is Important

As stated in Section 3, we have access to four human translations of varying
quality. In Table 2 we show the metric success measured by correlation with
human scores. The metrics stay the same but the references they use are
changed. Across both string-matching and parametrized model-based metrics, R1,
the worst human translation, leads to the worst metric performance. The best
performance is achieved with R3, a standard professional translation. Notably,
it is not R4 which was created by professional translatologists and also the
most expensive. This can be explained by the presence of translation
shifts,333Deviation from the original segment structure or meaning. which
occur more frequently on this professionality level, but can negatively impact
the utility of the reference (Fomicheva, 2017). See Appendix C for an example
and analysis.

A simple way of improving a translation is to post-edit (refine) it, which is
cheaper that translating it from scratch (Daems and Macken, 2020; Zouhar et
al., 2021). We mark the post-edited version of a reference Rx as
Rx${}^{\textrm{PE}}$ and use them as references for the automated metrics in
Table 3. The post-editors are either laymen with knowledge of both languages,
students of translatology, or professional translators. While the proficiency
level plays a role, in most cases the post-edited translations serve as better
references.

Metric | R1 | R2 | R3 | R4
---|---|---|---|---
BLEU | 0.082  $\blacksquare$ | 0.103  $\blacksquare$ | 0.109  $\blacksquare$ | 0.103  $\blacksquare$
ChrF | 0.090  $\blacksquare$ | 0.125  $\blacksquare$ | 0.128  $\blacksquare$ | 0.123  $\blacksquare$
TER | 0.082  $\blacksquare$ | 0.092  $\blacksquare$ | 0.114  $\blacksquare$ | 0.105  $\blacksquare$
COMET20 | 0.172  $\blacksquare$ | 0.176  $\blacksquare$ | 0.185  $\blacksquare$ | 0.181  $\blacksquare$
COMET22 | 0.189  $\blacksquare$ | 0.195  $\blacksquare$ | 0.191  $\blacksquare$ | 0.192  $\blacksquare$
BLEURT | 0.159  $\blacksquare$ | 0.156  $\blacksquare$ | 0.199  $\blacksquare$ | 0.178  $\blacksquare$
Average | 0.129  $\blacksquare$ | 0.141  $\blacksquare$ | 0.154  $\blacksquare$ | 0.147  $\blacksquare$
Comet${}^{20}_{\textrm{QE}}$ | 0.171  $\blacksquare$

Table 2: Segment-level Kendall’s $\bm{\tau}$ between automated metrics and
human scores. The metrics are computed with respect to the four references.
The R3 translation yields the best results as the reference, despite not being
the optimal translation from the human perspective.

Proficiency | R1${}^{\textrm{PE}}$-R1 | R2${}^{\textrm{PE}}$-R2 | R3${}^{\textrm{PE}}$-R3 | R4${}^{\textrm{PE}}$-R4
---|---|---|---|---
Layman |  $\bm{\bm{+}}$0.019 |  $\bm{\bm{+}}$0.011 |  $\bm{\bm{+}}$0.011 |  $\bm{\bm{+}}$0.011
Student |  $\bm{\bm{+}}$0.009 |  $\bm{\bm{+}}$0.005 |  $\bm{\bm{+}}$0.001 |  $\bm{\bm{-}}$0.002
Professional |  $\bm{\bm{+}}$0.025 |  $\bm{\bm{+}}$0.011 |  $\bm{\bm{+}}$0.004 |  $\bm{\bm{+}}$0.002

Table 3: Difference between using original translations (in Table 2) and their
post-edited versions. The post-editing comes from translators on different
levels. The scores are averaged across all metrics; see Tables 8 and 10 for
per-metric breakdowns.  In most cases, using post-edited versions improves
metric performance.

###  4.2 Multiple References are Useful

The previous section provided an analysis of how individual references affect
metric performance. In many situations, however, multiple references are
available. While some metrics, such as BLEU, support multiple references
natively, one can also aggregate them using either segment-level averages or
maximums (i.e. compute multiple scores for each segment and take the average
or maximum). In Table 4 we consider three setups: (1) two high-quality
references, R3 and R4, (2) all human translations, or (3) all human post-
edited translations. Across all metrics, this segment-level aggregation
improves the correlation with humans, especially in the case of using the
original four human translations. Taking the maximum and not the average has
the advantage that there exists a specific reference which yields that
particular score. We include all subsets as references in Table 9.

Aggregation | R3 | R{3,4} | Rx | Rx${}^{\textrm{PE}}$
---|---|---|---|---
Average | 0.154  $\blacksquare$ | 0.159  $\blacksquare$ | 0.166  $\blacksquare$ | 0.164  $\blacksquare$
Max | 0.154  $\blacksquare$ | 0.155  $\blacksquare$ | 0.165  $\blacksquare$ | 0.167  $\blacksquare$

Table 4: Average performance of metrics with multiple references. See Table 9
for per-metric breakdown.  All aggregation methods improve the performance
over the best single one, R3.

To systematically study the effect of reference count on metric performance,
in Figure 1 we randomly sample $x$ references from the whole pool of original
and post-edited translations, irregardless of their quality. The biggest gains
in metric performance are achieved until seven references and further gains
are negligible.

Figure 1: Metric performance with multiple sampled references from the pool of
the original human translations and all their post-edited versions. Confidence
intervals using t-test indicate that there is 99% confidence of the mean (of
10 samples) being in the shaded area.  Biggest advantage is gained from at
least three references and taking their segment-level average.

###  4.3 Allocating a Budget for References

Usually, it is simple to gather many source sentences and let multiple systems
translate them. Evaluating all of them using human annotators is unattainable
but running automated metrics is not. However, these require references, which
are also costly. It remains unclear how many references and of quality level
to ask for given a budget.

#### Can references be mixed?

To assess what types of configurations of references can lead to an
improvement, we first validate if they can be mixed. For example if 75% of the
sources can have references from a cheaper vendor R1 and 25% from a higher-
quality but more expensive vendor R3. This is different from Table 4 where
each segment had exactly two references from the same two sources. In Appendix
B we show that using lower-quality references R1 leads to lower absolute
metric scores (e.g. BLEU = 24.2) as opposed to higher-quality ones R3 (e.g.
BLEU = 37.1). This holds across all metrics, which could disrupt the
correlation with humans. In Figure 2 we mix some of the references together
for evaluation. Despite the varying absolute scores of metrics under different
references (Appendix B), mixing of multiple references leads to an almost
perfectly linear combination of the endpoint metric performances. This is
counter-intuitive — the segment-level correlation should be lower because of
the mix of distributions. It also stands in contrast to Mirsarraf et al.
(2012), who documented a decade ago the need to normalize the references. A
positive conclusion is that if there is budget to only translate 10% of
segments with high quality, it can be used.

Figure 2: Metric performance with references (ordered by usefulness) from
mixed sources (e.g. 25% R1 and 75% R2; rightmost is 100% R3).  All metrics
benefit from higher-quality references, even when mixed.

#### Budget Allocation Algorithm.

In Appendix A we approach this task as an optimization problem. We provide a
heuristic-based stochastic algorithm to find an assignment of source segments
$S$ to be translated by vendors of different costs and qualities within a
specific budget. For the current dataset, we set the cost of a segment in R1,
R2, R3, and R4 to be 1, 1, 2, and 3, respectively. Their quality, a prior
knowledge verified by Section 4.1, were set to 1, 2, 4, and 3. The algorithm
in Appendix A contains a hyperparameter $\lambda$ that controls whether budget
will be allocated more towards having multiple references or more towards
having fewer but higher-quality references.

In Figure 3 we show ChrF and BLEU correlations when using the references from
our algorithm. The optimal preference between quality and quantity changes
with increasing budget. Using all of the budget on either increasing quality
or quantity would correspond to the bottom or top row, which are not optimal.
The best reference configurations for a particular budget, such as
$|S|{\times}4$, contain a mixture of references from R1, R2, R3, and R4 with
multiple references for some segments.

Figure 3: ChrF (left) and Comet${}^{20}_{\textrm{}}$ (right) correlations on
reference configurations created with a specific budget (x-axis) and quality-
quantity trade-off $\lambda$ (y-axis). $\bigstar$ marks the best value in each
column (fixed budget).

##  5 Conclusion

We showed that the quality of references is important for accurate automated
machine translation metrics. The relationships is not straightforward:
translatologists’ translations, despite being the peak human translation
quality, are not the best references. Rather, it is the standard commercial
professional translation that work best for current metrics. The improvements
apply both for string-matching metrics as well as to parametric model-based
ones. Taking the average over multiple references provides the biggest
benefit, with diminishing returns after 7 references. We also provided a
heuristic-based algorithm for finding a good configuration of references given
a budget, which surpasses optimizing solely for quantity or quantity.

#### Future work.

The dataset size prevents system-level investigations. Because there is little
point in evaluating segments that are easy to translate, a follow-up approach
could prioritize difficult-to translate segments. This is used by Isabelle et
al. (2017) for creating a challenge set. Future works should extend this
research by asking the question how many segments are needed to fulfill a
certain desideratum, such as effect size or metric accuracy.

## Limitations

We note the limitation of using only a small dataset and a single language
direction due to the costs of creating multiple rounds of high-quality
references. However, we are convinced the results hold in other scenarios
because the effect directions are the same across multiple metrics and setups.

## Acknowledgements

We extend our gratitude to Yasmin Moslem, Isabella Lai, Theia Vogel, Blanka
Sokołowska, Raj Dabre, Tom Kocmi, and Farhan Samir, who proofread this paper
in their free time. Ondřej Bojar received funding from Ministry of Education,
Youth and Sports of the Czech Republic LM2018101 LINDAT/CLARIAH-CZ and the
19-26934X grant of the Czech Science Foundation (NEUREM3).

Algorithm 1 Budget Allocation for References
Input: Source segments $S$, levels $L$, cost function
$\textsc{Cost}:L\rightarrow\mathbb{R}^{+}$, utility function
$\textsc{Util.}:L\rightarrow\mathbb{R}^{+}$,
tradeoff hyperparameter $\lambda\in[0,1]$, temperature $\tau>0$, budget
$B\in\mathbb{R}^{+}$.
Output: Assignment $R:L\rightarrow 2^{S}$.
Note: Figure 3 contains a patience mechanism, spans $\lambda\in[0,0.7]$ and
uses $\tau=0.5$.

1:$L\leftarrow\textsc{Sort}(L,\textsc{Cost})$

2:$R[L_{0}]\leftarrow S$ $\triangleright$ Assign everything to the cheapest
level.

3:$O\leftarrow R$

4:while $\sum_{l\in L}|R[l]|\cdot\textsc{$\textsc{Cost}$}(l)<B$ $\wedge$ no
exception do

5: $O\leftarrow R$

6: $a\sim\textsc{Sample}(\textsc{Promote}:\lambda,\textsc{Add}:1-\lambda)$
$\triangleright$ Selection action.

7: $X^{+}\leftarrow\\{\langle s,l\rangle|l\in L,s\in S\setminus R[l]\\}$
$\triangleright$ Samples that could be added to $R[l]$.

8: $X^{-}\leftarrow\\{\langle s,l\rangle|l\in L,s\in R[l]\\}$ $\triangleright$
Samples that could be removed from $R[l]$.

9: if $a=\textsc{Add}$ then

10: $x,l\sim\textsc{Sample}(\\{\langle
x,l\rangle:\frac{\sigma(\textsc{Util.}(l)-\textsc{Cost}(l))^{\nicefrac{{1}}{{\tau}}}}{Z}|x,l\in
X^{+}\\})$ $\triangleright$ Sample a segment to add.

11: $R[l]\leftarrow R[l]\cup\\{x\\}$

12: else if $a=\textsc{Promote}$ then
$\triangleright$ Sample a segment to add.

13: $x^{+},l^{+}\sim\textsc{Sample}(\\{\langle
x,l\rangle:\frac{\sigma(\textsc{Util.}(l)-\textsc{Cost}(l))^{\nicefrac{{1}}{{\tau}}}}{Z}|x,l\in
X^{+}\\})$
$\triangleright$ Sample where to move from.

14: $\,\\_\,\,\,,l^{-}\sim\textsc{Samp.}(\\{\langle
x,l\rangle:\frac{\sigma(\textsc{Cost}(l)-\textsc{Util.}(l))^{\nicefrac{{1}}{{\tau}}}}{Z}|x,l\in
X^{-},x{=}x^{+}\wedge\textsc{Util.}(l^{-}){<}\textsc{Util.}(l^{+})\\})$

15: $R[l^{+}]\leftarrow R[l^{+}]\cup\\{x^{+}\\}$

16: $R[l^{-}]\leftarrow R[l^{-}]\setminus\\{x^{+}\\}$

17: end if

18:end while

19:return O

Figure 4: Illustration of two operations from Algorithm 1. The initial state
is on the left. Then, a new segment is added to the $l_{3}$ level. Lastly, one
segment is moved from $l_{1}$ to $l_{3}$.

##  Appendix A Budget Allocation Algorithm

We formalize the problem with a segment cost $\textsc{Cost}(l)$ for a
reference on level $l{\in}L$ and the utility $\textsc{Util.}(l)$. The levels
might correspond to translation vendors which have costs and qualities. In
Section 4.3, we assumed
$\textsc{Cost}=\\{\textrm{R1:}\,1,\textrm{R2:}\,1,\textrm{R3:}\,2,\textrm{R4:}\,3\\}$
and
$\textsc{Util.}=\\{\textrm{R1:}\,1,\textrm{R2:}\,2,\textrm{R3:}\,4,\textrm{R4:}\,3\\}$.
Given a set of source sentences $S$, the goal is to divvy up the segments into
different levels. The selection should maximize performance of a particular
metric on a number of systems but needs to be under a fixed budget $B$, i.e.
$\sum_{l}|R_{l}|\cdot\textsc{Cost}(l)\leq B$. In our setup, to preserve fair
comparison, each segment needs to have at least one reference. This is because
the smaller the testset, the easier it is to achieve higher but spurious
correlations. Therefore, $\bigcup_{l\in L}R_{l}=S$. The formalization
explicitly allows for parts of the testset to be translated multiple times.

We present a heuristic algorithm with pseudocode in Algorithm 1 and
explanatory illustration of the two operations in Figure 4. The algorithm
continually applies one of the two operations until they can either no longer
be applied or the budget is reached. The algorithm will always terminate
because Add increases the cost and utility and Promote increases the utility
by at least $\min_{l\in L}\textsc{Cost}(l)$ and $\min_{l_{1},l_{2}\in
L}|\textsc{Util.}(l_{2})-\textsc{Util.}(l_{1})|$, respectively.

In addition to the metric correlations in Figure 3, we show the average number
of references per source segment in Figure 5. With focus on quality, each
segment has fewer references.

Figure 5: Average number of references per one segment allocated by Algorithm
1 with $\tau=0.5$ (top) and $\tau=10^{-3}$ (bottom).

##  Appendix B References and Metric Scores

To understand the effect of different metrics, we show the average raw scores
of each metric in Table 5. While it appears that the higher the raw score, the
better the metric performance (low score of R1 and high scores of R3 and R4),
this trend does not explain the improvements of using the post-edited
versions, e.g. as R1${}^{\textrm{PE}}$ over R1, or R3${}^{\textrm{PE}}$ over
R3. In fact, the post-edited versions always lead to lower raw scores. This
could be the result of further translations shifts as the post-edits are based
on a translation and not the source segment.

Metric | R1 | R2 | R3 | R4 | R1${}^{\textrm{PE}}$ | R3${}^{\textrm{PE}}$
---|---|---|---|---|---|---
BLEU | 24.2 | 31.5 | 37.1 | 27.3 | 23.9 | 31.0
ChrF | 55.7 | 60.3 | 63.0 | 56.1 | 54.5 | 58.4
TER | -63.3 | -53.0 | -48.7 | -59.4 | -64.1 | -58.9
COMET20 | 65.5 | 68.9 | 68.2 | 61.0 | 60.4 | 61.4
COMET22 | 84.6 | 84.9 | 84.8 | 83.6 | 83.6 | 83.7
BLEURT | 61.3 | 66.1 | 68.8 | 64.5 | 61.6 | 64.9

Table 5: Raw average scores across metrics and references. TER scores are
flipped to make higher numbers be better.  R3 serves as the best references
(Table 2). It also leads to the highest raw metric scores but this trend does
not hold overall for R3${}^{\textrm{PE}}$.

##  Appendix C Qualitative Analysis

In Table 6 we show a single source segment, system translation and multiple
references and the metric scores. BLEU ranges from 0 to 100 and both maximums
are almost∗ achieved just with a different reference. The best human
translation led to the lowest BLEU score because of a translation shift. This
is not surprising because BLEU operates on the surface-level. Unexpectedly, a
similar thing happens also with Comet${}^{20}_{\textrm{}}$, which uses a
distributed semantic representation of the segments. This shows that
parametric model-based metrics are not robust to changes in references. In
Table 7 the Comet${}^{20}_{\textrm{}}$ difference between references is large
due to some translators deciding to drop the verb “spolupracovat”
(collaborate), which changes the meaning and the system translation is
penalized.

Table 6: BLEU and Comet${}^{20}_{\textrm{}}$ scores of the source “Three
Scottish students named among Europe’s best” and the system translation “Tři
skotští studenti byli zařazeni mezi nejlepší v Evropě”. Both metrics are
multiplied by 100.  All references are good translations but the scores vary.

BLEU | Comet${}^{20}_{\textrm{}}$ | Reference
---|---|---
10 | 78 | K evropské špičce nově patří i tři skotští studenti
23 | 120 | Tři skotští studenti se umístili mezi nejlepšími v Evropě
23 | 121 | Tři skotští studenti mezi nejlepšími v Evropě
28 | 116 | Tři skotští studenti byli oceněni jako jedni z nejlepších v Evropě
28 | 115 | Tři skotští studenti byli jmenováni jako jedni z nejlepších v Evropě
28 | 114 | Tři skotští studenti byli vyhlášeni jako jedni z nejlepších v Evropě
32 | 117 | Tři skotští studenti byli jmenováni jednimi z nejlepších v Evropě
37 | 122 | Tři skotští studenti byli jmenováni mezi nejlepšími v Evropě
43 | 125 | Tři skotští studenti se zařadili mezi nejlepší v Evropě
43 | 121 | Tři skotští studenti patří mezi nejlepší v Evropě.
43 | 122 | Tři skotští studenti patří mezi nejlepší v Evropě
60 | 127 | Tři skotští studenti zařazeni mezi nejlepší v Evropě
100 | 131 | Tři skotští studenti byli zařazeni mezi nejlepší v Evropě

Table 7: BLEU and Comet${}^{20}_{\textrm{}}$ scores of the source “Sony,
Disney Back To Work On Third Spider-Man Film” and the system translation
“Disney se vrací, bude spolupracovat se Sony na třetím sólovém Spider-Man
filmu”. Both metrics are multiplied by 100.  Some references omit part of the
information and Comet${}^{20}_{\textrm{}}$ thus penalizes the system
translation.

BLEU | Comet${}^{20}_{\textrm{}}$ | Reference
---|---|---
4 | -42 | Sony a Disney točí třetí film o Spidermanovi
4 | -33 | Sony a Disney točí třetí film o Spider-Manovi
8 | -9 | Sony a Disney pracují na třetím filmu o Spider-Manovi
8 | -4 | Sony a Disney pokračují v práci na třetím filmu o Spider-Manovi
8 | 1 | Sony a Disney opět pracují na třetím filmu o Spider-Manovi
8 | 15 | Sony a Disney spolupracují na třetím filmu o Spider-Manovi
4 | 16 | Sony a Disney budou spolupracovat při natáčení třetího filmo o Spider-manovi
8 | 28 | Sony a Disney opět spolupracují na třetím filmu o Spider-Manovi
8 | 30 | Sony a Disney budou spolupracovat na třetím filmu o Spider-Manovi
8 | 35 | Sony a Disney budou opět spolupracovat na třetím filmu o Spider-Manovi
17 | 52 | Disney bude znovu spolupracovat se společností Sony na třetím filmu Spider-Man
10 | 64 | Disney bude se Sony dál pracovat na třetím filmu se Spider-Manem
50 | 73 | Disney bude spolupracovat se Sony na třetím sólovém filmu o Spider-Manovi
75 | 99 | Disney se vrací, bude spolupracovat se Sony na třetím filmu o Spider-Manovi
78 | 106 | Disney se vrací, bude spolupracovat se Sony na třetím sólovém filu Spider-Man.
79 | 108 | Disney se vrací, bude spolupracovat se Sony na třetím Spider-Man filmu
100 | 121 | Disney se vrací, bude spolupracovat se Sony na třetím sólovém Spider-Man filmu

##  Appendix D Implementation Details

BLEU, ChrF and TER used SacreBleu (Post 2018, v:2.3.1) at the segment-level.
For the trainable metrics, we use publicly available models.

  * BLEU: #:1|c:mixed|e:yes|tok:13a|s:exp

  * ChrF: #:1|c:mixed|e:yes|nc:6|nw:0|s:no

  * TER: #:1|c:lc|t:tercom|nr:no|pn:yes|as:no

  * COMET20: Unbabel/wmt20-comet-da

  * COMET22: Unbabel/wmt22-comet-da

  * BLEURT: BLEURT20

| Metric | R1${}^{\textrm{PE}}$-R1 | R2${}^{\textrm{PE}}$-R2 | R3${}^{\textrm{PE}}$-R3 | R4${}^{\textrm{PE}}$-R4
---|---|---|---|---|---
Layman PE | BLEU |  $\bm{\bm{+}}$0.019 |  $\bm{\bm{+}}$0.007 |  $\bm{\bm{+}}$0.009 |  $\bm{\bm{+}}$0.010
ChrF |  $\bm{\bm{+}}$0.027 |  $\bm{\bm{+}}$0.015 |  $\bm{\bm{+}}$0.016 |  $\bm{\bm{+}}$0.019
TER |  $\bm{\bm{+}}$0.026 |  $\bm{\bm{+}}$0.015 |  $\bm{\bm{+}}$0.013 |  $\bm{\bm{+}}$0.014
COMET20 |  $\bm{\bm{+}}$0.016 |  $\bm{\bm{+}}$0.011 |  $\bm{\bm{+}}$0.010 |  $\bm{\bm{+}}$0.009
COMET22 |  $\bm{\bm{+}}$0.008 |  $\bm{\bm{+}}$0.006 |  $\bm{\bm{+}}$0.007 |  $\bm{\bm{+}}$0.005
BLEURT |  $\bm{\bm{+}}$0.017 |  $\bm{\bm{+}}$0.015 |  $\bm{\bm{+}}$0.011 |  $\bm{\bm{+}}$0.010
Student PE | BLEU |  $\bm{\bm{+}}$0.010 |  $\bm{\bm{+}}$0.001 |  $\bm{\bm{-}}$0.004 |  $\bm{\bm{-}}$0.001
ChrF |  $\bm{\bm{+}}$0.011 |  $\bm{\bm{+}}$0.001 |  $\bm{\bm{+}}$0.000 |  $\bm{\bm{-}}$0.004
TER |  $\bm{\bm{+}}$0.003 |  $\bm{\bm{+}}$0.001 |  $\bm{\bm{+}}$0.005 |  $\bm{\bm{-}}$0.002
COMET20 |  $\bm{\bm{+}}$0.010 |  $\bm{\bm{+}}$0.003 |  $\bm{\bm{-}}$0.002 |  $\bm{\bm{-}}$0.002
COMET22 |  $\bm{\bm{+}}$0.002 |  $\bm{\bm{+}}$0.001 |  $\bm{\bm{+}}$0.000 |  $\bm{\bm{-}}$0.002
BLEURT |  $\bm{\bm{+}}$0.021 |  $\bm{\bm{+}}$0.022 |  $\bm{\bm{+}}$0.005 |  $\bm{\bm{-}}$0.004
Professional PE | BLEU |  $\bm{\bm{+}}$0.035 |  $\bm{\bm{+}}$0.011 |  $\bm{\bm{+}}$0.007 |  $\bm{\bm{-}}$0.000
ChrF |  $\bm{\bm{+}}$0.040 |  $\bm{\bm{+}}$0.010 |  $\bm{\bm{+}}$0.008 |  $\bm{\bm{+}}$0.004
TER |  $\bm{\bm{+}}$0.023 |  $\bm{\bm{+}}$0.014 |  $\bm{\bm{+}}$0.003 |  $\bm{\bm{-}}$0.002
COMET20 |  $\bm{\bm{+}}$0.016 |  $\bm{\bm{+}}$0.006 |  $\bm{\bm{+}}$0.000 |  $\bm{\bm{+}}$0.005
COMET22 |  $\bm{\bm{+}}$0.008 |  $\bm{\bm{+}}$0.002 |  $\bm{\bm{+}}$0.003 |  $\bm{\bm{+}}$0.005
BLEURT |  $\bm{\bm{+}}$0.027 |  $\bm{\bm{+}}$0.022 |  $\bm{\bm{+}}$0.005 |  $\bm{\bm{-}}$0.001

Table 8: Difference between using original translations (in Table 2) and post-
edited translations as references. Sections are divided based on who did the
post-editing (layman, translatology student, or professional translator). This
table expands on Table 3. Absolute scores of individual reference subsets are
in Table 10.

|  | R1 | R2 | R3 | R4 | R{1,2} | R{1,3} | R{1,4} | R{2,3} | R{2,4} | R{3,4} | R{1,2,3} | R{1,2,4} | R{1,3,4} | R{2,3,4} | Rx
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Average | BLEU | 0.082  $\blacksquare$ | 0.103  $\blacksquare$ | 0.109  $\blacksquare$ | 0.103  $\blacksquare$ | 0.109  $\blacksquare$ | 0.122  $\blacksquare$ | 0.114  $\blacksquare$ | 0.132  $\blacksquare$ | 0.124  $\blacksquare$ | 0.114  $\blacksquare$ | 0.136  $\blacksquare$ | 0.124  $\blacksquare$ | 0.125  $\blacksquare$ | 0.130  $\blacksquare$ | 0.134  $\blacksquare$
ChrF | 0.090  $\blacksquare$ | 0.125  $\blacksquare$ | 0.128  $\blacksquare$ | 0.123  $\blacksquare$ | 0.121  $\blacksquare$ | 0.135  $\blacksquare$ | 0.124  $\blacksquare$ | 0.148  $\blacksquare$ | 0.139  $\blacksquare$ | 0.135  $\blacksquare$ | 0.146  $\blacksquare$ | 0.135  $\blacksquare$ | 0.140  $\blacksquare$ | 0.147  $\blacksquare$ | 0.147  $\blacksquare$
TER | 0.082  $\blacksquare$ | 0.092  $\blacksquare$ | 0.114  $\blacksquare$ | 0.105  $\blacksquare$ | 0.095  $\blacksquare$ | 0.120  $\blacksquare$ | 0.107  $\blacksquare$ | 0.125  $\blacksquare$ | 0.117  $\blacksquare$ | 0.120  $\blacksquare$ | 0.121  $\blacksquare$ | 0.110  $\blacksquare$ | 0.123  $\blacksquare$ | 0.127  $\blacksquare$ | 0.124  $\blacksquare$
COMET20 | 0.172  $\blacksquare$ | 0.176  $\blacksquare$ | 0.185  $\blacksquare$ | 0.181  $\blacksquare$ | 0.181  $\blacksquare$ | 0.189  $\blacksquare$ | 0.185  $\blacksquare$ | 0.191  $\blacksquare$ | 0.183  $\blacksquare$ | 0.188  $\blacksquare$ | 0.190  $\blacksquare$ | 0.185  $\blacksquare$ | 0.190  $\blacksquare$ | 0.189  $\blacksquare$ | 0.189  $\blacksquare$
COMET22 | 0.189  $\blacksquare$ | 0.195  $\blacksquare$ | 0.191  $\blacksquare$ | 0.192  $\blacksquare$ | 0.195  $\blacksquare$ | 0.197  $\blacksquare$ | 0.194  $\blacksquare$ | 0.201  $\blacksquare$ | 0.197  $\blacksquare$ | 0.195  $\blacksquare$ | 0.200  $\blacksquare$ | 0.197  $\blacksquare$ | 0.197  $\blacksquare$ | 0.199  $\blacksquare$ | 0.199  $\blacksquare$
BLEURT | 0.159  $\blacksquare$ | 0.156  $\blacksquare$ | 0.199  $\blacksquare$ | 0.178  $\blacksquare$ | 0.171  $\blacksquare$ | 0.203  $\blacksquare$ | 0.183  $\blacksquare$ | 0.201  $\blacksquare$ | 0.180  $\blacksquare$ | 0.203  $\blacksquare$ | 0.201  $\blacksquare$ | 0.184  $\blacksquare$ | 0.204  $\blacksquare$ | 0.202  $\blacksquare$ | 0.202  $\blacksquare$
Average | 0.129  $\blacksquare$ | 0.141  $\blacksquare$ | 0.154  $\blacksquare$ | 0.147  $\blacksquare$ | 0.145  $\blacksquare$ | 0.161  $\blacksquare$ | 0.151  $\blacksquare$ | 0.166  $\blacksquare$ | 0.157  $\blacksquare$ | 0.159  $\blacksquare$ | 0.166  $\blacksquare$ | 0.156  $\blacksquare$ | 0.163  $\blacksquare$ | 0.166  $\blacksquare$ | 0.166  $\blacksquare$
Max | BLEU | 0.082  $\blacksquare$ | 0.103  $\blacksquare$ | 0.109  $\blacksquare$ | 0.103  $\blacksquare$ | 0.116  $\blacksquare$ | 0.122  $\blacksquare$ | 0.118  $\blacksquare$ | 0.135  $\blacksquare$ | 0.132  $\blacksquare$ | 0.111  $\blacksquare$ | 0.138  $\blacksquare$ | 0.137  $\blacksquare$ | 0.121  $\blacksquare$ | 0.135  $\blacksquare$ | 0.137  $\blacksquare$
ChrF | 0.090  $\blacksquare$ | 0.125  $\blacksquare$ | 0.128  $\blacksquare$ | 0.123  $\blacksquare$ | 0.133  $\blacksquare$ | 0.137  $\blacksquare$ | 0.129  $\blacksquare$ | 0.139  $\blacksquare$ | 0.146  $\blacksquare$ | 0.135  $\blacksquare$ | 0.144  $\blacksquare$ | 0.148  $\blacksquare$ | 0.140  $\blacksquare$ | 0.144  $\blacksquare$ | 0.147  $\blacksquare$
TER | 0.082  $\blacksquare$ | 0.092  $\blacksquare$ | 0.114  $\blacksquare$ | 0.105  $\blacksquare$ | 0.101  $\blacksquare$ | 0.124  $\blacksquare$ | 0.116  $\blacksquare$ | 0.132  $\blacksquare$ | 0.128  $\blacksquare$ | 0.117  $\blacksquare$ | 0.132  $\blacksquare$ | 0.130  $\blacksquare$ | 0.126  $\blacksquare$ | 0.135  $\blacksquare$ | 0.134  $\blacksquare$
COMET20 | 0.172  $\blacksquare$ | 0.176  $\blacksquare$ | 0.185  $\blacksquare$ | 0.181  $\blacksquare$ | 0.177  $\blacksquare$ | 0.184  $\blacksquare$ | 0.185  $\blacksquare$ | 0.180  $\blacksquare$ | 0.183  $\blacksquare$ | 0.183  $\blacksquare$ | 0.181  $\blacksquare$ | 0.183  $\blacksquare$ | 0.184  $\blacksquare$ | 0.181  $\blacksquare$ | 0.182  $\blacksquare$
COMET22 | 0.189  $\blacksquare$ | 0.195  $\blacksquare$ | 0.191  $\blacksquare$ | 0.192  $\blacksquare$ | 0.195  $\blacksquare$ | 0.191  $\blacksquare$ | 0.196  $\blacksquare$ | 0.191  $\blacksquare$ | 0.195  $\blacksquare$ | 0.191  $\blacksquare$ | 0.192  $\blacksquare$ | 0.197  $\blacksquare$ | 0.191  $\blacksquare$ | 0.192  $\blacksquare$ | 0.192  $\blacksquare$
BLEURT | 0.159  $\blacksquare$ | 0.156  $\blacksquare$ | 0.199  $\blacksquare$ | 0.178  $\blacksquare$ | 0.180  $\blacksquare$ | 0.199  $\blacksquare$ | 0.190  $\blacksquare$ | 0.188  $\blacksquare$ | 0.181  $\blacksquare$ | 0.193  $\blacksquare$ | 0.197  $\blacksquare$ | 0.192  $\blacksquare$ | 0.200  $\blacksquare$ | 0.188  $\blacksquare$ | 0.197  $\blacksquare$
Average | 0.129  $\blacksquare$ | 0.141  $\blacksquare$ | 0.154  $\blacksquare$ | 0.147  $\blacksquare$ | 0.150  $\blacksquare$ | 0.159  $\blacksquare$ | 0.156  $\blacksquare$ | 0.161  $\blacksquare$ | 0.161  $\blacksquare$ | 0.155  $\blacksquare$ | 0.164  $\blacksquare$ | 0.164  $\blacksquare$ | 0.160  $\blacksquare$ | 0.162  $\blacksquare$ | 0.165  $\blacksquare$

Table 9: Comparison using either a single or multiple references and taking
the average or maximum on segment-level. This table expands on Table 4.

|  | R1${}^{\textrm{PE}}$ | R2${}^{\textrm{PE}}$ | R3${}^{\textrm{PE}}$ | R4${}^{\textrm{PE}}$ | Rx${}^{\textrm{PE}}$ | R{1,1${}^{\textrm{PE}}$} | R{2,2${}^{\textrm{PE}}$} | R{3,3${}^{\textrm{PE}}$} | R{4,4${}^{\textrm{PE}}$} | R{x,x${}^{\textrm{PE}}$}
---|---|---|---|---|---|---|---|---|---|---|---
Layman PE | BLEU | 0.101  $\blacksquare$ | 0.111  $\blacksquare$ | 0.117  $\blacksquare$ | 0.113  $\blacksquare$ | 0.144  $\blacksquare$ | 0.092  $\blacksquare$ | 0.107  $\blacksquare$ | 0.113  $\blacksquare$ | 0.108  $\blacksquare$ | 0.140  $\blacksquare$
ChrF | 0.118  $\blacksquare$ | 0.139  $\blacksquare$ | 0.144  $\blacksquare$ | 0.142  $\blacksquare$ | 0.164  $\blacksquare$ | 0.106  $\blacksquare$ | 0.134  $\blacksquare$ | 0.137  $\blacksquare$ | 0.135  $\blacksquare$ | 0.159  $\blacksquare$
TER | 0.107  $\blacksquare$ | 0.107  $\blacksquare$ | 0.127  $\blacksquare$ | 0.119  $\blacksquare$ | 0.142  $\blacksquare$ | 0.099  $\blacksquare$ | 0.102  $\blacksquare$ | 0.123  $\blacksquare$ | 0.116  $\blacksquare$ | 0.139  $\blacksquare$
COMET20 | 0.188  $\blacksquare$ | 0.187  $\blacksquare$ | 0.195  $\blacksquare$ | 0.190  $\blacksquare$ | 0.198  $\blacksquare$ | 0.183  $\blacksquare$ | 0.184  $\blacksquare$ | 0.193  $\blacksquare$ | 0.188  $\blacksquare$ | 0.197  $\blacksquare$
COMET22 | 0.197  $\blacksquare$ | 0.201  $\blacksquare$ | 0.198  $\blacksquare$ | 0.197  $\blacksquare$ | 0.203  $\blacksquare$ | 0.195  $\blacksquare$ | 0.200  $\blacksquare$ | 0.196  $\blacksquare$ | 0.196  $\blacksquare$ | 0.202  $\blacksquare$
BLEURT | 0.176  $\blacksquare$ | 0.170  $\blacksquare$ | 0.210  $\blacksquare$ | 0.188  $\blacksquare$ | 0.209  $\blacksquare$ | 0.169  $\blacksquare$ | 0.165  $\blacksquare$ | 0.206  $\blacksquare$ | 0.186  $\blacksquare$ | 0.209  $\blacksquare$
Average | 0.148  $\blacksquare$ | 0.153  $\blacksquare$ | 0.165  $\blacksquare$ | 0.158  $\blacksquare$ | 0.177  $\blacksquare$ | 0.141  $\blacksquare$ | 0.149  $\blacksquare$ | 0.161  $\blacksquare$ | 0.155  $\blacksquare$ | 0.174  $\blacksquare$
Student PE | BLEU | 0.092  $\blacksquare$ | 0.104  $\blacksquare$ | 0.105  $\blacksquare$ | 0.102  $\blacksquare$ | 0.123  $\blacksquare$ | 0.089  $\blacksquare$ | 0.107  $\blacksquare$ | 0.108  $\blacksquare$ | 0.103  $\blacksquare$ | 0.130  $\blacksquare$
ChrF | 0.102  $\blacksquare$ | 0.126  $\blacksquare$ | 0.128  $\blacksquare$ | 0.119  $\blacksquare$ | 0.139  $\blacksquare$ | 0.097  $\blacksquare$ | 0.127  $\blacksquare$ | 0.130  $\blacksquare$ | 0.122  $\blacksquare$ | 0.144  $\blacksquare$
TER | 0.085  $\blacksquare$ | 0.093  $\blacksquare$ | 0.119  $\blacksquare$ | 0.103  $\blacksquare$ | 0.119  $\blacksquare$ | 0.084  $\blacksquare$ | 0.095  $\blacksquare$ | 0.119  $\blacksquare$ | 0.104  $\blacksquare$ | 0.123  $\blacksquare$
COMET20 | 0.182  $\blacksquare$ | 0.179  $\blacksquare$ | 0.183  $\blacksquare$ | 0.179  $\blacksquare$ | 0.186  $\blacksquare$ | 0.179  $\blacksquare$ | 0.179  $\blacksquare$ | 0.186  $\blacksquare$ | 0.181  $\blacksquare$ | 0.188  $\blacksquare$
COMET22 | 0.191  $\blacksquare$ | 0.196  $\blacksquare$ | 0.191  $\blacksquare$ | 0.189  $\blacksquare$ | 0.195  $\blacksquare$ | 0.191  $\blacksquare$ | 0.197  $\blacksquare$ | 0.193  $\blacksquare$ | 0.191  $\blacksquare$ | 0.197  $\blacksquare$
BLEURT | 0.180  $\blacksquare$ | 0.178  $\blacksquare$ | 0.203  $\blacksquare$ | 0.174  $\blacksquare$ | 0.199  $\blacksquare$ | 0.172  $\blacksquare$ | 0.170  $\blacksquare$ | 0.204  $\blacksquare$ | 0.177  $\blacksquare$ | 0.202  $\blacksquare$
Average | 0.139  $\blacksquare$ | 0.146  $\blacksquare$ | 0.155  $\blacksquare$ | 0.144  $\blacksquare$ | 0.160  $\blacksquare$ | 0.136  $\blacksquare$ | 0.146  $\blacksquare$ | 0.157  $\blacksquare$ | 0.146  $\blacksquare$ | 0.164  $\blacksquare$
Professional PE | BLEU | 0.118  $\blacksquare$ | 0.114  $\blacksquare$ | 0.115  $\blacksquare$ | 0.103  $\blacksquare$ | 0.127  $\blacksquare$ | 0.103  $\blacksquare$ | 0.113  $\blacksquare$ | 0.113  $\blacksquare$ | 0.104  $\blacksquare$ | 0.133  $\blacksquare$
ChrF | 0.131  $\blacksquare$ | 0.135  $\blacksquare$ | 0.136  $\blacksquare$ | 0.127  $\blacksquare$ | 0.146  $\blacksquare$ | 0.113  $\blacksquare$ | 0.133  $\blacksquare$ | 0.135  $\blacksquare$ | 0.126  $\blacksquare$ | 0.149  $\blacksquare$
TER | 0.105  $\blacksquare$ | 0.106  $\blacksquare$ | 0.116  $\blacksquare$ | 0.103  $\blacksquare$ | 0.118  $\blacksquare$ | 0.094  $\blacksquare$ | 0.102  $\blacksquare$ | 0.118  $\blacksquare$ | 0.104  $\blacksquare$ | 0.122  $\blacksquare$
COMET20 | 0.188  $\blacksquare$ | 0.182  $\blacksquare$ | 0.185  $\blacksquare$ | 0.186  $\blacksquare$ | 0.190  $\blacksquare$ | 0.183  $\blacksquare$ | 0.181  $\blacksquare$ | 0.189  $\blacksquare$ | 0.185  $\blacksquare$ | 0.191  $\blacksquare$
COMET22 | 0.198  $\blacksquare$ | 0.198  $\blacksquare$ | 0.195  $\blacksquare$ | 0.196  $\blacksquare$ | 0.199  $\blacksquare$ | 0.195  $\blacksquare$ | 0.199  $\blacksquare$ | 0.195  $\blacksquare$ | 0.195  $\blacksquare$ | 0.200  $\blacksquare$
BLEURT | 0.186  $\blacksquare$ | 0.178  $\blacksquare$ | 0.204  $\blacksquare$ | 0.176  $\blacksquare$ | 0.197  $\blacksquare$ | 0.177  $\blacksquare$ | 0.172  $\blacksquare$ | 0.206  $\blacksquare$ | 0.179  $\blacksquare$ | 0.202  $\blacksquare$
Average | 0.154  $\blacksquare$ | 0.152  $\blacksquare$ | 0.159  $\blacksquare$ | 0.149  $\blacksquare$ | 0.163  $\blacksquare$ | 0.144  $\blacksquare$ | 0.150  $\blacksquare$ | 0.159  $\blacksquare$ | 0.149  $\blacksquare$ | 0.166  $\blacksquare$

Table 10: Metric performance when using post-edited references also jointly
with their original versions (averaged at the segment-level). This table
expands on Tables 3 and 8.

## References

  * Barrault et al. (2020) Loïc Barrault, Magdalena Biesialska, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubešić, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. 2020.  Findings of the conference on machine translation.  In _Proceedings of the Fifth Conference on Machine Translation_ , 1–55. Association for Computational Linguistics.
  * Castilho et al. (2018) Sheila Castilho, Stephen Doherty, Federico Gaspari, and Joss Moorkens. 2018.  Approaches to human and machine translation quality assessment.  _Translation quality assessment: From principles to practice_.
  * Daems and Macken (2020) Joke Daems and Lieve Macken. 2020.  Post-editing human translations and revising machine translations: Impact on efficiency and quality.  In _Translation Revision and Post-editing_ , 50–70. Routledge.
  * Finch et al. (2004) Andrew M Finch, Yasuhiro Akiba, and Eiichiro Sumita. 2004.  How does automatic machine translation evaluation correlate with human scoring as the number of reference translations increases? In _LREC_.
  * Fomicheva (2017) Marina Fomicheva. 2017.  _The role of human reference translation in machine translation evaluation_.  Ph.D. thesis, Universitat Pompeu Fabra.
  * Fomicheva et al. (2020) Marina Fomicheva, Lucia Specia, and Francisco Guzmán. 2020.  Multi-hypothesis machine translation evaluation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 1218–1232.
  * Freitag et al. (2021a) Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021a.  Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation.  _Transactions of the Association for Computational Linguistics_ , 9:1460–1474.
  * Freitag et al. (2020) Markus Freitag, David Grangier, and Isaac Caswell. 2020.  BLEU might be guilty but references are not innocent.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , 61–71. Association for Computational Linguistics.
  * Freitag et al. (2023) Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023.  Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent.  In _Proceedings of the Eighth Conference on Machine Translation_ , 578–628. Association for Computational Linguistics.
  * Freitag et al. (2022) Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022.  Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust.  In _Proceedings of the Seventh Conference on Machine Translation_ , 46–68. Association for Computational Linguistics.
  * Freitag et al. (2021b) Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021b.  Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain.  In _Proceedings of the Sixth Conference on Machine Translation_ , 733–774. Association for Computational Linguistics.
  * González-Rubio et al. (2012) Jesús González-Rubio, Daniel Ortiz-Martínez, and Francisco Casacuberta. 2012.  Active learning for interactive machine translation.  In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_ , 245–254. Association for Computational Linguistics.
  * Haffari et al. (2009) Gholamreza Haffari, Maxim Roy, and Anoop Sarkar. 2009.  Active learning for statistical phrase-based machine translation.  In _Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics_ , 415–423. Association for Computational Linguistics.
  * Hamon and Mostefa (2008) Olivier Hamon and Djamel Mostefa. 2008.  The impact of reference quality on automatic MT evaluation.  In _Coling 2008: Companion volume: Posters_ , 39–42.
  * Isabelle et al. (2017) Pierre Isabelle, Colin Cherry, and George Foster. 2017.  A challenge set approach to evaluating machine translation.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , 2486–2496. Association for Computational Linguistics.
  * Khayrallah et al. (2020) Huda Khayrallah, Brian Thompson, Matt Post, and Philipp Koehn. 2020.  Simulated multiple reference training improves low-resource machine translation.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , 82–89. Association for Computational Linguistics.
  * Kocmi et al. (2021) Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021.  To ship or not to ship: An extensive evaluation of automatic metrics for machine translation.  In _Proceedings of the Sixth Conference on Machine Translation_ , 478–494. Association for Computational Linguistics.
  * Lommel (2016) Arle Lommel. 2016.  Blues for BLEU: Reconsidering the validity of reference-based MT evaluation.  _Translation Evaluation: From Fragmented Tools and Data Sets to an Integrated Ecosystem_ , 63.
  * Madnani et al. (2008) Nitin Madnani, Philip Resnik, Bonnie J. Dorr, and Richard Schwartz. 2008.  Are multiple reference translations necessary? investigating the value of paraphrased reference translations in parameter optimization.  In _Proceedings of the 8th Conference of the Association for Machine Translation in the Americas_ , 143–152.
  * Mendonça et al. (2023) Vânia Mendonça, Ricardo Rei, Luísa Coheur, and Alberto Sardinha. 2023.  Onception: Active Learning with Expert Advice for Real World Machine Translation.  _Computational Linguistics_ , 49(2):325–372.
  * Mi et al. (2020) Chenggang Mi, Lei Xie, and Yanning Zhang. 2020.  Improving adversarial neural machine translation for morphologically rich language.  _IEEE Transactions on Emerging Topics in Computational Intelligence_ , 4(4):417–426.
  * Mirsarraf et al. (2012) Mohammad Reza Mirsarraf, Kiyana Zolfaghar, Nazanin Dehghani, and Peyman Abolghasemi. 2012.  The impact of reference normalization on automatic mt evaluation.  In _6th International Symposium on Telecommunications_ , 811–816.
  * Ott et al. (2018) Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. 2018.  Analyzing uncertainty in neural machine translation.  In _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , 3956–3965. PMLR.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , 311–318. Association for Computational Linguistics.
  * Popović (2015) Maja Popović. 2015.  chrF: character n-gram F-score for automatic MT evaluation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , 392–395. Association for Computational Linguistics.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , 186–191. Association for Computational Linguistics.
  * Qin and Specia (2015) Ying Qin and Lucia Specia. 2015.  Truly exploring multiple references for machine translation evaluation.  In _Proceedings of the 18th Annual Conference of the European Association for Machine Translation_. European Association for Machine Translation.
  * Rei et al. (2022) Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022.  COMET-22: Unbabel-IST 2022 submission for the metrics shared task.  In _Proceedings of the Seventh Conference on Machine Translation_ , 578–585. Association for Computational Linguistics.
  * Rei et al. (2021) Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, André F. T. Martins, and Alon Lavie. 2021.  Are references really needed? unbabel-IST 2021 submission for the metrics shared task.  In _Proceedings of the Sixth Conference on Machine Translation_ , 1030–1040. Association for Computational Linguistics.
  * Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020.  COMET: A neural framework for MT evaluation.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , 2685–2702. Association for Computational Linguistics.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 7881–7892. Association for Computational Linguistics.
  * Shi and Huang (2020) Zhouxing Shi and Minlie Huang. 2020.  Robustness to modification with shared words in paraphrase identification.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , 164–171. Association for Computational Linguistics.
  * Snover et al. (2006) Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006.  A study of translation edit rate with targeted human annotation.  In _Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers_ , 223–231. Association for Machine Translation in the Americas.
  * Specia and Shah (2018) Lucia Specia and Kashif Shah. 2018.  Machine translation quality estimation: Applications and future perspectives.  _Translation quality assessment: from principles to practice_ , 201–235.
  * van der Wees et al. (2017) Marlies van der Wees, Arianna Bisazza, and Christof Monz. 2017.  Dynamic data selection for neural machine translation.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , 1400–1410. Association for Computational Linguistics.
  * Vernikos et al. (2022) Giorgos Vernikos, Brian Thompson, Prashant Mathur, and Marcello Federico. 2022.  Embarrassingly easy document-level MT metrics: How to convert any pretrained metric into a document-level metric.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , 118–128. Association for Computational Linguistics.
  * Zhang and Vogel (2004) Ying Zhang and Stephan Vogel. 2004.  Measuring confidence intervals for the machine translation evaluation metrics.  In _Proceedings of the 10th Conference on Theoretical and Methodological Issues in Machine Translation of Natural Languages_.
  * Zhang and Vogel (2010) Ying Zhang and Stephan Vogel. 2010.  Significance tests of automatic machine translation evaluation metrics.  _Machine Translation_ , 24:51–65.
  * Zheng et al. (2018) Renjie Zheng, Mingbo Ma, and Liang Huang. 2018.  Multi-reference training with pseudo-references for neural translation and text generation.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , 3188–3197. Association for Computational Linguistics.
  * Zouhar et al. (2021) Vilém Zouhar, Martin Popel, Ondřej Bojar, and Aleš Tamchyna. 2021.  Neural machine translation quality and post-editing performance.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , 10204–10214. Association for Computational Linguistics.
  * Zouhar et al. (2023) Vilém Zouhar, Věra Kloudová, Martin Popel, and Ondřej Bojar. 2023.  Evaluating optimal reference translations.


# Beauville-Voisin filtrations on zero cycles of moduli space of stable
sheaves on K3 surfaces

Zhiyuan Li  Zhiyuan Li, Shanghai Center for Mathematical Sciences, Fudan
University, Jiangwan Campus, Shanghai, 200438, China<EMAIL_ADDRESS>and  Ruxuan Zhang  Ruxuan Zhang, Shanghai Center for Mathematical Sciences,
Fudan University, Jiangwan Campus, Shanghai, 200438, China
<EMAIL_ADDRESS>
###### Abstract.

The Beauville-Voisin conjecture predicts the existence of a filtration on
projective hyper-Kähler manifolds opposite to the conjecture Bloch-Beilinson
filtration, called the Beauivlle-Voisin filtration. In [Voi16], Voisin has
introduced a filtration on zero cycles of an arbitrary projective hyper-Kähler
manifold. On moduli space of stable objects of a projective K3 surface, there
are other candidates constructed by Shen-Yin-Zhao, Barros-Flapan-Marian-
Silversmith in [SYZ20, BFMS19] and more recently by Vial in [Via20] from
different point of views. According to the work in [Via20], all of them are
proved to be equivalent except Voisin’s filtration. In this paper, we show
that Voisin’s filtration is the same as the other filtrations. As an
application, we prove a conjecture in [BFMS19].

###### Key words and phrases:

Beauville-Voisin filtration, moduli space of sheaves, hyper-Kähler

###### 2020 Mathematics Subject Classification:

14J28; 14F08; 14J42

##  1\. Introduction

On a projective K3 surface $X$, Beauville and Voisin [BV04] showed that $X$
carries a canonical zero-cycle class of degree 1,

$[o_{X}]\in\operatorname{CH}_{0}(X),$

where $o_{X}$ can be taken any point lying on a rational curve in $X$. The
intersections of divisor classes in $X$, as well as the second Chern class of
$X$, lie in $\mathbb{Z}\cdot[o_{X}]$. In [O’G13], O’Grady introduced a
filtration $\mathrm{S}_{\bullet}(X)$ on $\operatorname{CH}_{0}(X)$,

$\mathrm{S}_{i}(X)=\bigcup_{\deg([z])=i}\\{[z]+\mathbb{Z}\cdot[o_{X}]\\}$

for all effective zero-cycles $z$ of degree $i$, which gives another
characterization of the Beauville-Voisin class $[o_{X}]$. More generally,
Beauville-Voisin’s conjecture predicts that a similar picture also holds for
any projective hyper-Kähler manifold $M$ and there exists a filtration (called
Beauville-Voisin filtration) on the Chow ring viewed as an opposite to the
conjectural Bloch-Beilinson filtration. For zero cycles, Voisin has
constructed a filtration
$\mathrm{S}_{\bullet}\operatorname{CH}_{0}(M)\subseteq\operatorname{CH}_{0}(M)$
in [Voi16] defined by

$\mathrm{S}_{i}\operatorname{CH}_{0}(M):=\left<x\in M|~{}\dim O_{x}\geq
d-i\right>$

where $O_{x}$ is the orbit of $x$ under the rational equivalence.

In the case $M=M_{\sigma}(v)$ is the moduli space of $\sigma$-stable objects
in the derived category $\mathrm{D}^{b}(X)$ of a projective K3 surface $X$ for
some Bridgeland stability condition $\sigma$. Shen-Yin-Zhao discovered a
filtration on the derived category $\mathrm{D}^{b}(X)$, which gives another
natural filtration $\mathrm{S}_{\bullet}^{\rm SYZ}\operatorname{CH}_{0}(M)$,

$\mathrm{S}_{i}^{\rm
SYZ}\operatorname{CH}_{0}(M):=\left<E|~{}c_{2}(E)\in\mathrm{S}_{i}(X)\right>.$

They showed that there is an inclusion

$\mathrm{S}_{i}^{\rm
SYZ}\operatorname{CH}_{0}(M)\subseteq\mathrm{S}_{i}\operatorname{CH}_{0}(M),$

between the two filtrations. A natural question is whether the two filtration
coincides (cf. [SYZ20, Remark 2.6]). This has been confirmed on Hilbert scheme
of points on $X$ via using the density of constant cycle subvarieties. In this
paper, we give an affirmative answer to this question and the main result is

######  Theorem 1.1.

Let $M=M_{\sigma}(v)$ be the moduli space of $\sigma$-stable objects in
$\mathrm{D}^{b}(X)$ with a primitive Mukai vector $v$. Suppose $\sigma$ is
$v$-generic. Then we have

$\mathrm{S}_{\bullet}^{\rm
SYZ}\operatorname{CH}_{0}(M)=\mathrm{S}_{\bullet}\operatorname{CH}_{0}(M).$

As a corollary, we obtain the density of constant cycle subvarieties on $M$
(See Corollary 3.2). Finally, let us compare Voisin’s filtration with other
filtrations. In [BFMS19], Flapan, Marian and Silversmith have constructed two
filtrations $\mathrm{S}_{\bullet}^{\rm small}\operatorname{CH}_{0}(M)$ and
$\mathrm{S}^{\rm BFMS}_{\bullet}\operatorname{CH}_{0}(M)$ from the product
point of view, while Vial has found a so called co-radical filtration
$\mathrm{R}_{\bullet}\operatorname{CH}_{0}(M)$ from the co-algebra structure
on birational motives. Combined with the results in [BFMS19, Via20], our main
theorem indicates that

######  Theorem 1.2.

With the assumptions in Theorem 1.1, we have

$\mathrm{S}^{\mathrm{small}}_{\bullet}\operatorname{CH}_{0}(M)=\mathrm{S}^{\rm
BFMS}_{\bullet}\operatorname{CH}_{0}(M)=\mathrm{S}_{\bullet}\operatorname{CH}_{0}(M)=\mathrm{S}_{\bullet}^{\rm
SYZ}\operatorname{CH}_{0}(M)=\mathrm{R}_{\bullet}\operatorname{CH}_{0}(M).$
(1)

As a consequence, [BFMS19, Conjecture 4] holds.

### Acknowledgement

The authors are grateful to Junliang Shen, Qizheng Yin and Ziyu Zhang for
useful comments. The authors were supported by NSFC for Innovative Research
Groups (Grant No. 12121001) and partially supported by NSFC grant for General
Program (11771086).

##  2\. Degenerate loci of vector bundles

In this section, we recap Voisin’s work in [Voi15] which establish a relation
between the zero cycles on the moduli space of stable sheaves and the Hilbert
scheme of points. Let $(X,H)$ be a polarized K3 surface and let $M=M_{H}(v)$
be the moduli space of Gieseker $H$-stable sheaves on $X$ with a primitive
Mukai vector $v=(r,D,s)$. We may assume $H$ is $v$-generic so that $M$ is a
smooth projective hyper-Kähler variety of dimension

$v^{2}+2=D^{2}-2rs+2.$

Given a locally free sheaf $E\in M$, let
$d=\frac{1}{2}\operatorname{Ext}^{1}(E,E)$ and
$g=\dim\operatorname{Gr}(r-1,\mathrm{H}^{0}(X,E))$. As in [Voi15], if $E$ is
globally generated and $\mathrm{H}^{1}(X,E)=0$, we have an exact sequence for
a general $(r-1)$-dimensional sections of $\mathrm{H}^{0}(X,E)$

$0\rightarrow\mathcal{O}_{X}^{r-1}\xrightarrow{s}E\rightarrow I_{Z}\otimes
D\rightarrow 0$ (2)

where $Z$ is the degeneracy locus of $s$ consisting of $d+g$ distinct points.
Then the sequence defines a rational map

$\phi_{E}:\mathrm{G}_{E}\dashrightarrow X^{[d+g]},$

where $\mathrm{G}_{E}=\operatorname{Gr}(r-1,\mathrm{H}^{0}(X,E))$, sending $s$
to $I_{Z}\otimes D\subset X^{[d+g]}$.

######  Proposition 2.1.

Assume the condition $(\star)$

$E~{}\hbox{is globally generated
and}~{}\mathrm{H}^{i}(X,E)=0~{}\hbox{for}~{}i=1,2.$

holds. The morphism $\phi_{E}$ is generically injective. Moreover, for any two
stable sheaves $E,F\in M$ satisfying $(\star)$ as extensions of
$\mathcal{O}_{X}^{r-1}$ and $I_{Z}\otimes D$ for some $Z$, we have $E\cong F$.

###### Proof.

The generic injectivity of $\phi_{E}$ is already proved in [Voi15, Proposition
3.2]. It follows from the computation of $\operatorname{Hom}(E,I_{Z}\otimes
D)$. Indeed, apply the functor $\operatorname{Hom}(E,-)$ to the exact sequence
(2), one can obtain a long exact sequence:

$0\rightarrow\operatorname{Hom}(E,\mathcal{O}_{X}^{r-1})\rightarrow\operatorname{Hom}(E,E)\rightarrow\operatorname{Hom}(E,I_{Z}\otimes
D)\rightarrow\operatorname{Ext}^{1}(E,\mathcal{O}_{X}^{r-1})\to\ldots$

By the Serre duality, we have

$\operatorname{Hom}(E,\mathcal{O}_{X}^{r-1})=\operatorname{Ext}^{1}(E,\mathcal{O}_{X}^{r-1})=0.$

Since $E$ is simple, $\operatorname{Hom}(E,E)\cong\mathbb{C}$. It follows that

$\operatorname{Hom}(E,I_{Z}\otimes D)\cong\mathbb{C}.$

It implies that $\phi_{E}$ is injective on $s$ whenever we have the sequence
(2) for $s$. If we have another extension

$0\rightarrow\mathcal{O}_{X}^{r-1}\rightarrow F\rightarrow I_{Z}\otimes
D\rightarrow 0,$

after applying $\operatorname{Hom}(E,-)$, one can find that

$\operatorname{Hom}(E,F)\cong\operatorname{Hom}(E,I_{Z}\otimes
D)\cong\mathbb{C}.$

In particular, if $F$ is a stable sheaf, then $E\cong F$. ∎

More generally, one can extend this construction to a family of stable locally
free sheaves. Let $\mathcal{E}$ be a locally free sheaf on $B\times X$ for
some variety $B$ satisfying that for $E_{b}=\mathcal{E}|_{\\{b\\}\times X}$,
we have

$\mathrm{H}^{i}(X,E_{b})=0,~{}i=1,2$

for any $b\in B$. In particular, the spaces of sections
$\mathrm{H}^{0}(X,E_{b})$ are of the same dimension and
$\mathcal{F}:=\pi_{*}\mathcal{E}$ is a locally free sheaf on $B$, where
$\pi:X\times B\rightarrow B$ is the projection. Let
$\mathbf{G}_{B}:=\operatorname{Gr}^{r-1}(\mathcal{F})$ be the Grassmannian of
rank $r-1$ subbundles of $\mathcal{F}$ which fits into the following diagram:

${\mathbf{G}_{B}\times X}$${B\times
X}$${\mathbf{G}_{B}}$${B}$$\scriptstyle{p\times\operatorname{id}}$$\scriptstyle{q}$$\scriptstyle{\pi}$$\scriptstyle{p}$
(3)

There exists a universal locally free subsheaf $\mathcal{W}\hookrightarrow
p^{*}\mathcal{F}$ on $\mathbf{G}_{B}$. Pulling back to $\mathbf{G}_{B}\times
X$, one has a natural morphism:

$\sigma:q^{*}\mathcal{W}\rightarrow(p\times\operatorname{id})^{*}\mathcal{E}.$
(4)

Then the degeneracy locus of $\sigma$ in $\mathbf{G}_{B}\times X$ is:

$D_{k}(\sigma)=\\{(s,x)~{}|~{}\mathrm{rank}(\sigma(s,x))\leq k\\}.$

A general fiber of $D_{k}(\sigma)\rightarrow\mathbf{G}_{B}$ consists of $d+g$
distinct points in $X$ by Proposition 2.1. Then it defines a rational map
$\phi$:

${\phi_{B}:\mathbf{G}_{B}}$${X^{[d+g]}}$

The restriction of $\sigma$ at $\\{(b,s)\\}\times X\subset\mathbf{G}_{B}\times
X$ is exactly $\mathcal{O}_{X}^{r-1}\xrightarrow{s}E_{b}$ in Voisin’s
construction. In this way, one can obtain a rational map

$\phi_{B}:\mathbf{G}_{B}\dashrightarrow X^{[d+g]}$ (5)

whose restriction to the fiber $q^{-1}(E)\subseteq\mathbf{G}_{B}$ is
$\phi_{E}$.

##  3\. Proof of Main theorem

Let $v=(r,D,s)$ be a primitive Mukai vector with $r\geq 0$ and assume that $H$
is $v$-generic. We first consider the locally free sheaves in $M_{H}(v)$.

######  Theorem 3.1.

Let $E$ be a locally free sheaf in $M=M_{H}(v)$. Then
$c_{2}(E)\in\mathrm{S}_{i}(X)$ if there exists a constant cycle subvariety
$Y\subseteq M$ such that $[E]\in Y$ and $\dim Y\geq d-i$. In particular, if
$M$ consists of stable locally free sheaves, then $\mathrm{S}_{\bullet}^{\rm
SYZ}\operatorname{CH}_{0}(M)=\mathrm{S}_{\bullet}\operatorname{CH}_{0}(M)$.

###### Proof.

After tensoring line bundles $mH$ for sufficiently large $m$ without changing
$M$, we may assume that the sheaves in $M$ satisfy the condition $(\star)$ by
[Lan06, Theorem 0.1, Section 10.1]. There exists an $1\boxtimes\alpha$-twisted
universal sheaf on $X\times M$. We take a Severi-Brauer variety

$\nu:P\rightarrow M$

for the Brauer class $\alpha$. Then there is a universal sheaf $\mathcal{E}$
on $X\times P$ in the sense that for $x\in P$, the stalk $\mathcal{E}_{x}$ of
$\mathcal{E}$ is isomorphic to $E_{\nu(x)}$, where $E_{\nu(x)}$ is the sheaf
corresponding to $\nu(x)\in M$.

With the same notations in the previous section, for any subvariety
$Y\subseteq M$ containing $E$, take $B=\nu^{-1}(Y)$, we can apply relative
version of Voisin’s construction to obtain a rational map

$\phi_{\nu^{-1}(Y)}:\mathbf{G}_{\nu^{-1}(Y)}\dashrightarrow X^{[d+g]}$

By Proposition 2.1, we have

  1. (1)

the restriction of $\phi_{\nu^{-1}(Y)}$ to the fiber of
$p:\mathbf{G}_{\nu^{-1}(Y)}\rightarrow\nu^{-1}(Y)$ is generic injective and
hence $\phi_{\nu^{-1}(Y)}$ sends each fiber to a $g$-dimensional constant
cycle subvariety in $X^{[d+g]}$.

  2. (2)

for different $E^{\prime}\in Y$ and the images $\phi(p^{-1}(E^{\prime}))$ in
$X^{[d+g]}$ are disjoint if $E^{\prime}$ is locally free.

Moreover, for any two points $s_{1},s_{2}\in\nu^{-1}(E^{\prime})$, their
corresponding sheaves are the same and so are their images in $X^{[d+g]}$.
Then we have

$\mathrm{dim}~{}\phi(p^{-1}(Y))=\dim Y+g.$

Now, suppose $Y$ is a constant cycle subvariety on $M$ of dimension $\geq
d-i$. According to the main Theorem in [MZ20], two sheaves
$E^{\prime},E^{\prime\prime}\in M$ are rationally equivalent in
$\operatorname{CH}_{0}(M)$ if and only if $c_{2}(E^{\prime})$ and
$c_{2}(E^{\prime\prime})$ are rationally equivalent in
$\operatorname{CH}_{0}(X)$. For any point in $\mathbf{G}_{\nu^{-1}(Y)}$, its
image $I_{Z}\otimes D$ represent the same class in
$\operatorname{CH}_{0}(X^{[d+g]})$ as

$c_{2}(I_{Z}\otimes D)=c_{2}(E)$

in $\operatorname{CH}_{0}(X)$ because the points in $Y$ are constant in
$\operatorname{CH}_{0}(M)$. This implies that the Zariski closure of the image
$\phi(p^{-1}(Y))$ is a constant cycle subvariety on $X^{[d+g]}$. By [Voi15,
Theorem 2.1], we have

$c_{2}(I_{Z}\otimes D)\in\mathrm{S}_{i}(X)$

and hence $E\in\mathrm{S}_{i}\operatorname{CH}_{0}(M)$. For the last
statement, we have

$\mathrm{S}_{i}\operatorname{CH}_{0}(M)\subseteq\mathrm{S}_{i}^{\rm
SYZ}\operatorname{CH}_{0}(M)$

from the definition of $\mathrm{S}_{i}\operatorname{CH}_{0}(M)$. The assertion
then follows from the inclusion $\mathrm{S}_{i}^{\rm
SYZ}\operatorname{CH}_{0}(M)\subseteq\mathrm{S}_{i}\operatorname{CH}_{0}(M)$.
∎

### Proof of Theorem 1.1

We first deal with the case $M=M_{H}(v)$ with $r\geq 0$. By tensoring the
sheaf in $M$ with $mH$ for $m$ sufficiently large, we may always assume that

$D\cdot H>\max\\{4r^{2}+1,2r(v^{2}H^{2}-D^{2}H^{2}+(D\cdot
H)^{2})\\},~{}s>2d,\hbox{~{}if $r>0$}$ (6)

or

$s>\max\\{2d,\frac{(D\cdot H)^{2}}{2H^{2}}+1\\},\hbox{~{}if $r=0$}.$ (7)

Let $\mu_{H}$ be the slope function with respect to $H$. Under these
conditions, Yoshioka has shown in [Yos09, Theorem 1.7, Corollary 2.14] that
there is an isomorphism

$\Phi:M_{H}(v)\to M_{\widehat{H}}(\widehat{v})$

induced by a Fourier-Mukai transform such that for any stable sheaf $E\in M$,
$\Phi(E)$ is a $\mu_{\widehat{H}}$-stable sheaf of rank $s$ (See also [BZ19,
Theorem 3.4]). The conditions also ensures that the $\mu_{\widehat{H}}$-stable
sheaves in $M_{\widehat{H}}(\widehat{v})$ have sufficiently large rank and
have to be locally free (cf. [Yos01, Remark 3.2] and [HL97, Remark 6.1.9]). In
fact, there is an exact sequence

$0\rightarrow E\rightarrow E^{\vee\vee}\rightarrow Q\rightarrow 0$

with $E^{\vee\vee}$ being $\mu_{\widehat{H}}$-stable and $Q$ being a
$0$-dimensional sheaf whose support is of length $l$. Then
$v(E^{\vee\vee})^{2}=v(E)^{2}-2rl\geq-2$. Therefore $l=0$ if
$r>\frac{v(E)^{2}+2}{2}$. By Theorem 3.1, we have

$\mathrm{S}_{\bullet}^{\rm
SYZ}(M_{\widehat{H}}(\widehat{v}))=\mathrm{S}_{\bullet}(M_{\widehat{H}}(\widehat{v}))$

Finally, according to [SYZ20, Proposition 0.4], $\Phi$ induces an isomorphism
between the two filtartions

$\mathrm{S}_{\bullet}^{\rm
SYZ}\operatorname{CH}_{0}(M_{H}(v))\cong\mathrm{S}_{\bullet}^{\rm
SYZ}\operatorname{CH}_{0}(M_{\widehat{H}}(\widehat{v})).$

It follows that $\mathrm{S}_{\bullet}^{\rm
SYZ}\operatorname{CH}_{0}(M)=\mathrm{S}_{\bullet}\operatorname{CH}_{0}(M)$.

Now we return to the case $M=M_{\sigma}(v)$ is a Bridgeland moduli space.
According to [BZ19, Proposition 3.5], every Bridgeland moduli space is
isomorphic to a moduli space of Gieseker stable sheaves via a derived
equivalence. Then again by [SYZ20, Proposition 0.4], one can immediately get
the assertion.

######  Corollary 3.2.

For any $E\in M$ with dim $O_{E}=d-i$, the subvarieties of the orbit $O_{E}$
with dimension $d-i$ are dense in $M$.

###### Proof.

In [SYZ20], it has been shown that there is an incident variety

$R\subset\\{(E,\xi)~{}|~{}c_{2}(E)=[\mathrm{supp}(\xi)]+c\cdot
o_{X}\in\operatorname{CH}_{0}(X)\\}\subset M\times S^{[d]}$

such that the natural projections in the diagram below

${R}$${M}$${{X^{[d]}}}$$\scriptstyle{p}$$\scriptstyle{q}$

are both generically finite and surjective. Here, $c$ is a constant depending
on the Mukai vector of $E$. By Theorem 1.1, we have
$c_{2}(E)\in\mathrm{S}_{i}(X)$. Using the surjectivity of $p$, we can find
$\xi\in X^{[d]}$ such that $(E,\xi)\in R$. As
$c_{2}(\xi)\in\mathrm{S}_{i}(X)$, by [Voi15, Lemma 3.5], the subvarieties
$W_{\alpha}\subseteq O_{\xi}$ with dimension $d-i$ are dense in $X^{[d]}$.
Shrinking $R$ if necessary, we may assume $p$ and $q$ are finite. Then the
subvarieties $p\circ q^{-1}(W_{\alpha})$ are lying in the orbit $O_{E}$ of
dimension $d-i$ and they are clearly Zariski dense in $M$. ∎

######  Corollary 3.3.

The filtration $S_{\bullet}\operatorname{CH}_{0}(M)$ is invariant under
birational transform between hyper-Kähler varieties, i.e. if
$f:M=M_{\sigma}(v)\dashrightarrow M^{\prime}$ is a birational map, then the
induced isomorphism

$\operatorname{CH}_{0}(M^{\prime})\cong\operatorname{CH}_{0}(M)$

preserves the filration $S_{\bullet}$.

###### Proof.

The hyper-Kähler birational models of $M$ is controlled by its birational
ample cone. According to [BM14, Theorem 1.2], there is a continuous map

$\ell:{\rm Stab}^{\dagger}(X)\rightarrow{\operatorname{NS}(M_{\sigma}(v))},$

from a connected component of space of stability conditions on
$\mathrm{D}^{b}(X)$ to the Néron-Severi group such that the image of $\ell$ is
the birational ample cone and $M^{\prime}$ is isomorphic to a moduli space of
stable objects $M_{\sigma^{\prime}}(v)$ for a generic $\sigma^{\prime}$ in
some chamber $\mathcal{C}^{\prime}$, such that
$\ell(\mathcal{C}^{\prime})=f^{*}{\rm Amp}(M_{\sigma^{\prime}}(v))$ and
$\ell(\mathcal{C})={\rm Amp}(M_{\sigma}(v))$ for the chamber $\mathcal{C}$
containing $\sigma$. Moreover, up to an automorphism, the birational map
$f:M_{\sigma}(v)\dashrightarrow M_{\sigma^{\prime}(v)}$ is actually induced by
a derived (anti) autoequivalence.

Thus we can deduce the result by the $\mathrm{S}_{\bullet}=\mathrm{S}^{\rm
SYZ}_{\bullet}$ and that $\mathrm{S}^{\rm SYZ}_{\bullet}$ is independent of
modular interpretations.

∎

##  4\. Applications: Comparison with other filtrations

With the notations as before, on the moduli space $M=M_{H}(v)$ of stable
sheaves, there is a distinguished point $c_{M}=[E_{0}]\in M$ such that
$c_{2}(E_{0})=ko_{X}$ with $k=\frac{D^{2}}{2}+r-s$. The existence of $c_{M}$
is ensured by the surjectivity of the projection map from the incident variety
$R$ to $X^{[d]}$. Let us recall the filtrations defined in [BFMS19].

######  Definition 4.1.

Let $\Delta$ be the diagonal in the product $M\times M$ and we set

$\overline{\Delta}=\Delta-M\times c_{M}\in\operatorname{CH}_{2d}(M\times M)$
(8)

Then one can define

$\mathrm{S}^{\rm
BFMS}_{i}(\operatorname{CH}_{0}(M)):=\\{\alpha~{}|~{}\alpha\cdot\overline{\Delta}_{0,1}\ldots\cdot\overline{\Delta}_{0,i+1}=0\in\operatorname{CH}_{\ast}(M\times
M^{i+1})\\},$ (9)

where $\overline{\Delta}_{0,j}$ is the pull back of $\overline{\Delta}$ from
the $0$-th and $j$-th factors for $j=0,1,\cdots i+1$. It is useful to single
out the subgroup

$\mathrm{S}^{\mathrm{small}}_{i}(\operatorname{CH}_{0}(M))\subset\mathrm{S}^{\rm
BFMS}_{i}(\operatorname{CH}_{0}(M))$

generated by point classes $[F]\in\mathrm{S}_{i}^{\rm
BFMS}(\operatorname{CH}_{0}(M))$ with $F\in M$.

In [BFMS19] and [Via20], it has been proved that

$\mathrm{S}^{\mathrm{small}}_{\bullet}\operatorname{CH}_{0}(M)=\mathrm{S}^{\rm
BFMS}_{\bullet}\operatorname{CH}_{0}(M)=\mathrm{S}_{\bullet}^{\rm
SYZ}\operatorname{CH}_{0}(M)=\mathrm{R}_{\bullet}\operatorname{CH}_{0}(M),$
(10)

(cf. [Via20, Proposition A.6, Theorem 8.4]). Furthermore, for 0-cycles, the
termination of this filtration implies that for all $F\in M$

$[F]\cdot\overline{\Delta}_{0,1}\cdot\overline{\Delta}_{0,2}\ldots\overline{\Delta}_{0,d+1}=0\in\operatorname{CH}_{*}(M\times
M^{d+1}).$

Furthermore, they conjectured that a generalization of this equation for
higher dimensional constant cycle varieties is still valid. As a consequence
of Theorem 1.1, we confirm their conjecture.

######  Theorem 4.2.

[BFMS19, Conjecture 4] Let $Y\subseteq M$ be a constant cycle subvariety. Then

$[Y]\cdot\overline{\Delta}_{0,1}\cdot\overline{\Delta}_{0,2}\ldots\overline{\Delta}_{0,d+1}=0\in\operatorname{CH}_{*}(M\times
M^{d+1}).$

Here, $[Y]$ is pulled back as usual from the first $M$ factor.

###### Proof.

According to [BFMS19, Lemma 4], Conjecture holds if and only if

$\mathrm{S}_{\bullet}\operatorname{CH}_{0}(M)\subseteq\mathrm{S}^{\rm
BFMS}_{\bullet}\operatorname{CH}_{0}(M).$

The assertion follows directly from (10) (cf.[BFMS19, Lemma 3] and [Via20,
Theorem 8.4]). ∎

Combining the above results, one can conclude Theorem 1.2.

######  Remark 4.3.

It was asked in [SYZ20, Question 3.2] whether one has the following
equivalence

$[E]\in\mathrm{S}_{i}^{\rm SYZ}\operatorname{CH}_{0}(M)~{}{\rm
if~{}and~{}only~{}if}~{}c_{2}(E)\in\mathrm{S}_{i}(X).$

An affirmative answer of this question can be implied by the following
statement:

$[V_{E}]\cdot\overline{\Delta}_{0,1}\cdot\overline{\Delta}_{0,2}\ldots\overline{\Delta}_{0,d}\neq
0$ (11)

for a maximal constant cycle variety $V_{E}$ containing $E$. This is due to a
direct computation in [BFMS19, Remark 6]: if $E\in\mathrm{S}_{i}^{\rm
small}\operatorname{CH}_{0}(X)$, one must have dim $V_{E}\geq d-i$ assuming
(11). Then up to taking a derived equivalence preserving O’Grady’s filtration,
one can assume that $E$ is locally free and obtain
$c_{2}(E)\in\mathrm{S}_{i}(X)$ by applying Theorem 3.1.

In [BFMS19, Remark 6], it also claims that (11) holds if

$\mathrm{S}_{\bullet}^{\mathrm{small}}\operatorname{CH}_{0}(M)\subseteq\mathrm{S}_{\bullet}\operatorname{CH}_{0}(M).$

which is always true by Theorem 1.2. However, it seems to us that there exists
a gap in the proof: it’s unclear why there exists a constant cycle variety
$V_{E}$ containing $E$ with dim $V_{E}\geq n-i$ for $E\in\mathrm{S}_{i}^{\rm
small}\operatorname{CH}_{0}(M)\cap\mathrm{S}_{i}\operatorname{CH}_{0}(M)\setminus\mathrm{S}_{i-1}^{\rm
small}\operatorname{CH}_{0}(M)$. Hence the question seems still open.

## References

  * [BFMS19] Ignacio Barros, Laure Flapan, Alina Marian, and Rob Silversmith.  On product identities and Chow ring of holomorphic symplectic varieties.  arXiv:1912.13419, 2019.
  * [BM14] Arend Bayer and Emanuele Macrì.  MMP for moduli of sheaves on K3s via wall-crossing: nef and movable cones, Lagrangian fibrations.  Invent. Math., 198(3):505–590, 2014.
  * [BV04] Arnaud Beauville and Claire Voisin.  On the chow ring of a k3 surface.  Journal of Algebraic Geometry, 13(3):417–426, 2004.
  * [BZ19] Nero Budur and Ziyu Zhang.  Formality conjecture for K3 surfaces.  Compos. Math., 155(5):902–911, 2019.
  * [HL97] Daniel Huybrechts and Manfred Lehn.  The geometry of moduli spaces of sheaves.  Aspects of Mathematics, E31. Friedr. Vieweg & Sohn, Braunschweig, 1997\.
  * [Lan06] Adrian Langer.  Moduli spaces and castelnuovo-mumford regularity of sheaves on surfaces.  American journal of mathematics, 128(2):373–417, 2006.
  * [MZ20] Alina Marian and Xiaolei Zhao.  On the group of zero-cycles of holomorphic symplectic varieties.  Épijournal Géom. Algébrique, 4:Art. 3, 5, 2020.
  * [O’G13] Kieran G. O’Grady.  Moduli of sheaves and the Chow group of $K3$ surfaces.  J. Math. Pures Appl. (9), 100(5):701–718, 2013.
  * [SYZ20] Junliang Shen, Qizheng Yin, and Xiaolei Zhao.  Derived categories of $K3$ surfaces, O’Grady’s filtration, and zero-cycles on holomorphic symplectic varieties.  Compos. Math., 156(1):179–197, 2020.
  * [Via20] Charles Vial.  On the birational motive of hyper-kähler varieties.  arXiv:2010.00099, 2020.
  * [Voi15] Claire Voisin.  Rational equivalence of 0-cycles on $K3$ surfaces and conjectures of Huybrechts and O’Grady.  In Recent advances in algebraic geometry, volume 417 of London Math. Soc. Lecture Note Ser., pages 422–436. Cambridge Univ. Press, Cambridge, 2015.
  * [Voi16] Claire Voisin.  Remarks and questions on coisotropic subvarieties and 0-cycles of hyper-Kähler varieties.  In K3 surfaces and their moduli, volume 315 of Progr. Math., pages 365–399. Birkhäuser/Springer, [Cham], 2016.
  * [Yos01] Kota Yoshioka.  Moduli spaces of stable sheaves on abelian surfaces.  Math. Ann., 321(4):817–884, 2001.
  * [Yos09] Kota Yoshioka.  Stability and the Fourier-Mukai transform. II.  Compos. Math., 145(1):112–142, 2009.