# FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metrics for
Automatic Text Generation

Moussa Kamal Eddine  École Polytechnique, 2Linagora  Guokan Shang  Antoine
J.-P. Tixier  Michalis Vazirgiannis  École Polytechnique, 2Linagora

###### Abstract

††*Equal contribution

Fast and reliable evaluation metrics are key to R&D progress. While
traditional natural language generation metrics are fast, they are not very
reliable. Conversely, new metrics based on large pretrained language models
are much more reliable, but require significant computational resources. In
this paper, we propose FrugalScore, an approach to learn a fixed, low cost
version of any expensive NLG metric, while retaining most of its original
performance. Experiments with BERTScore and MoverScore on summarization and
translation show that FrugalScore is on par with the original metrics (and
sometimes better), while having several orders of magnitude less parameters
and running several times faster. On average over all learned metrics, tasks,
and variants, FrugalScore retains 96.8% of the performance, runs 24 times
faster, and has 35 times less parameters than the original metrics. We make
our trained metrics publicly
available111https://github.com/moussaKam/FrugalScore, to benefit the entire
NLP community and in particular researchers and practitioners with limited
resources.

##  1 Introduction

Automatic evaluation metrics are the only way to monitor the training of,
evaluate, and compare across models in a systematic, large-scale way, and are
thus a critical component of the research and development ecosystem in machine
learning. To get adopted in practice, evaluation metrics need to be both
reliable and affordable, i.e., fast and easy to compute.

While some metrics meet these criteria, such as precision and recall in
information retrieval, root mean square error in regression, etc., finding
suitable metrics is still an open problem in the field of Natural Language
Generation (NLG) Novikova et al. (2017).

Indeed, historical $n$-gram matching metrics such as ROUGE (Lin, 2004) for
summarization, BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie,
2005) for translation, while affordable, are not very reliable, as they are
based on surface-form matching only, i.e., lexical similarity, and have thus
no sense of semantic similarity. For instance, it makes little sense to use
ROUGE for the evaluation of abstractive summarization systems (which are
becoming the norm), or whenever the generated text paraphrases the original
text.

Following the advent of transfer learning in NLP, new NLG metrics based on
large pretrained language models have recently been proposed, such as
BERTScore Zhang et al. (2019) and MoverScore Zhao et al. (2019). By relying on
contextual embeddings, these metrics capture semantics and are therefore much
more reliable. However, due to the sheer size of the underlying models, these
metrics pose environmental issues Strubell et al. (2019), take time to
compute, and require access to significant computational resources, so they
are not accessible by everyone in the NLP community.

For example, we were not able to run some of the best variants of
BERTScore222From BERTScore’s authors: https://tinyurl.com/8cwyter2, based on
DeBERTa-Large and DeBERTa-XLarge He et al. (2020) on a 12GB GPU. Even when
enough GPU memory is available, relying on such large models is still
associated with extended runtimes, which can impede the progress of
experiments when used once or more per epoch for validation and monitoring
purposes.

To address this problem, we propose in this paper FrugalScore, an approach to
learn a lightweight version of BERTScore, MoverScore, and more generally any
metric based on a large pretrained language model. While our objectives are
clearly the same as that of model compression, and distillation in particular,
our method differs: we first sample sequence pairs, annotate these pairs with
the metric to be learned, and finally train a miniature model on the resulting
dataset.

Our contributions can be summarized as follows:

1) Our compact models have several orders of magnitude less parameters than
the original metrics and run several times faster, while retaining most of
their original performance. We even outperform the original metrics in some
cases333Hence the name FrugalScore, as frugal engineering is defined as
“achieving more with fewer resources”..

2) Our metrics are not only faster because of the much smaller amount of
parameters, but also because they require only one forward pass and do not
rely on any similarity function.

3) Regardless of how expensive the original metric is, querying our trained
metrics always has the same low, fixed cost. This decoupling is a major
advantage as the size of the pretrained language models has recently been
growing tremendously (e.g., Brown et al. (2020)).

##  2 Background

Related work falls into two categories: unsupervised and supervised metrics.

###  2.1 Unsupervised metrics

To address the limitations of ROUGE and BLEU, variants based on static word
embeddings (Mikolov et al., 2013) were developed, e.g., ROUGE-WE (Ng and
Abrecht, 2015), BLEU2VEC Tättar and Fishel (2017), and MEANT 2.0 (Lo, 2017).
While using word vectors is a progress over strict $n$-gram matching, static
embeddings are still very limited as they do not capture polysemy, i.e., the
fact that words have different meanings in different contexts.

More recently, the focus has shifted to harnessing the power of the
contextualized embeddings produced by large pretrained language models. For
instance, the Sentence Mover’s Similarity (Clark et al., 2019) represents
sentences as the average of their ELMo word embeddings (Peters et al., 2018)
and measures the minimum cost of transforming one summary into the other,
using a modified version of the Word Mover’s Distance (Kusner et al., 2015).
BERTr (Mathur et al., 2019) computes approximate recall based on the pairwise
cosine similarity between the BERT embeddings (Devlin et al., 2018) of the
words in automatic and reference translations. Mark-Evaluate (Mordido and
Meinel, 2020) is a family of metrics that consider contextualized word or
sentence embeddings derived from BERT as population samples, to evaluate
language generation with population estimation methods used in ecology.

Finally, the recently introduced BERTScore Zhang et al. (2019) and MoverScore
Zhao et al. (2019) are general-purpose NLG evaluation metrics that are
becoming widely used. The main difference between BERTScore and MoverScore
lies in the function used to compute the similarity between the
representations of the two sequences
$\mathbf{x}=\langle\mathbf{x_{1}},...,\mathbf{x_{k}}\rangle$ and
$\mathbf{y}=\langle\mathbf{y_{1}},...,\mathbf{y_{l}}\rangle$. We experimented
with these two metrics, so we provide more details about them in what follows.

BERTScore first computes the pairwise cosine similarity between the
representations of the tokens in each sequence, and uses greedy matching to
match each token to the most similar one in the other sequence. Given two pre-
normalized vector sequences $\mathbf{x}$ and $\mathbf{y}$, BERTScore computes:

$R_{BERT}=\frac{1}{|\mathbf{x}|}\sum_{\mathbf{x_{i}}\in
x}\max_{\mathbf{y}_{j}\in\mathbf{y}}\mathbf{\mathbf{x^{T}_{i}}\mathbf{y}_{j}}$
(1)

and:

$P_{BERT}=\frac{1}{|\mathbf{y}|}\sum_{\mathbf{y_{i}}\in
y}\max_{\mathbf{x}_{j}\in\mathbf{x}}\mathbf{\mathbf{y^{T}_{i}}\mathbf{x}_{j}}$
(2)

The F1-score is classically obtained as:

$F_{BERT}=2\frac{P_{BERT}R_{BERT}}{P_{BERT}+R_{BERT}}$ (3)

MoverScore uses an $n$-gram generalization of the Word Mover’s Distance (WMD)
Kusner et al. (2015) as their (dis)similarity function. More specifically,
they solve for the optimal transportation flow matrix
$F\in\mathbb{R}^{|\mathbf{x}|\times|\mathbf{y}|}$ between the two weighted
sequences of $n$-grams:

$WMD(\mathbf{x},\mathbf{y})=min_{F}\langle C,F\rangle$ (4) $s.t.\quad
F\mathbf{1}=f_{\mathbf{x}},~{}~{}F^{T}\mathbf{1}=f_{\mathbf{y}}$

Where $C$ is the transportation cost matrix ($C_{ij}$ is the Euclidean
distance between $x_{i}$ and $y_{j}$) and
$f_{x}\in\mathbb{R}^{|\mathbf{x}|}_{+}$ and
$f_{y}\in\mathbb{R}^{|\mathbf{y}|}_{+}$ are the $n$-gram weight vectors.

Note that by directly learning BERTScore’s and MoverScore’s full internal
mapping (from sequence pairs to final scalar scores), FrugalScore internalizes
their similarity functions. This does not only provide a speedup at inference
time, but also improves performance, as shown in section 5.

###  2.2 Supervised metrics

Related to our work are also supervised metrics, which are directly trained on
human evaluations. ROSE (Conroy and Dang, 2008) is a linear combination model
of different variants of ROUGE using canonical correlation. Beer (Stanojević
and Sima’an, 2014) is a learning-to-rank approach using word and character
n-gram matching, and token ordering, as features to maximize correlation with
human rankings of machine translation systems. S3 (Peyrard et al., 2017)
trains a regression model that takes the evaluation scores of several existing
metrics and many hand-crafted features as input, and learns the best
combination of them to approximate human summary judgments. DPMFcomb (Yu et
al., 2015) and Blend (Ma et al., 2017) are combined metrics incorporating a
vast amount of lexical, syntactic and semantic based translation evaluation
metrics using ranking and regression SVMs respectively. RUSE (Shimanaka et
al., 2018) evaluates machine translation with a neural regressor based on
universal sentence embeddings (e.g., InferSent (Conneau et al., 2017)). NUBIA
(Kane et al., 2020) consists of three modules: a feature extractor based on
RoBERTa (Liu et al., 2019) and GPT-2 (Radford et al., 2019) fine-tuned on
language evaluation tasks, an aggregator trained to predict the quality of the
hypothesis given the reference using the extracted features, and a calibrator
mapping all predictions between 0 and 1.

Differences. Like the aforementioned efforts, FrugalScore is a learned metric.
However, it does not rely on any intermediate or handcrafted features, and,
most importantly, it does not require training on human annotations.
Supervision in FrugalScore is conducted on a synthetic dataset, as a trick to
expose and learn the internal mapping of the unsupervised metrics to be
learned. Last but not least, unlike all aforementioned methods, compression is
central to FrugalScore, which is based on miniature versions of the models
used by the original metrics.

###  2.3 Differences with distillation

Knowledge distillation (Hinton et al., 2015) is the process of transferring
knowledge from a large teacher model to a smaller student model to accomplish
model compression (Buciluǎ et al., 2006). It was originally proposed in the
domain of computer vision and speech recognition, then successfully adapted to
NLP (Sanh et al., 2019). While FrugalScore, like distillation, focuses on
model compression, there is one major difference. Distillation was designed
for multi-class classification settings, and relies on a cross-entropy loss
over the softened probability distributions of the teacher and student. In our
case, we deal with a regression setting, and use the mean squared error
objective.

###  2.4 Differences with BLEURT

A work closely related to ours is BLEURT Sellam et al. (2020). However, there
are a number of significant differences with our approach. First, BLEURT
continues the pretraining of an already pretrained BERT-based model on a
synthetic dataset in a self-supervised way, whereas FrugalScore is directly
trained to learn the scores of the metric of interest, in a supervised
fashion.

Also, BLEURT’s synthetic dataset is made by perturbing Wikipedia sentences
with mask-filling, backtranslation, and word dropping, whereas we use other
data sources than Wikipedia such as summarization and translation datasets,
and only NLG models to induce perturbations.

When creating its synthetic dataset, BLEURT automatically annotates the
(original, perturbed) sequence pairs with numerical and categorical “signals”:
BLEU, ROUGE, BERTscore, backtranslation likelihood, textual entailment
(probability of three labels: entail, contradict, and neutral, given by BERT
fine-tuned on MNLI), and backtranslation flag. On the other hand, FrugalScore
simply and directly annotates the sequence pairs with the metric to be
learned.

After pretraining, BLEURT is fine-tuned on human judgments, in a way similar
to the supervised metrics described in subsection 2.2. BLEURT does not learn
to generate a scalar until that final fine-tuning phase, so it cannot be used
as a metric before that. Conversely, FrugalScore is trained from the start to
be a metric, and the fine-tuning phase is optional.

Also, BLEURT was designed for the evaluation of translation. The authors only
test whether it can be applied to a different task by experimenting on the
WebNLG (data-to-text) dataset (Gardent et al., 2017). Conversely, we focus on
learning general text similarity metrics (e.g., BERTscore and MoverScore), so
FrugalScore is task-agnostic by design.

Finally, and above all, the objective of FrugalScore is model compression,
whereas that of BLEURT is metric learning.

##  3 Our Approach

Developing FrugalScore requires three phases, one of which is optional.

Phase 1. We create a synthetic dataset (see subsection 3.1) by sampling pairs
of more or less related sequences and annotating them with the expensive
metrics to be learned. This is a one-time operation that does not need to be
repeated regardless of the model used in Phase 2.

Phase 2. We continue the pretraining (subsection 3.2) of a miniature
pretrained language model on the synthetic dataset built by Phase 1. Here, the
miniature model learns the internal mapping of the expensive metric, including
any similarity function applied to the representations. Note that a different
miniature is trained for each metric to be learned (we leave learning metric
combinations as future work).

The miniature can then be used in inference mode to generate scores for any
never-seen pair of sequences.

Phase 3 (optional). We fine-tune the miniature on human annotations, which, as
shown in section 6, can boost performance.

###  3.1 Synthetic Dataset

The objective here was to generate pairs of sequences mimicking the
(reference, candidate) pairs found in NLG datasets, which are usually
semantically related and in many cases paraphrasing one another. We sampled
our sequences from a variety of data sources, listed next.

Summarization. For each document in the well-known CNN/DailyMail dataset
Nallapati et al. (2016), our goal was to generate several summaries differing
in terms of structure and quality. To this purpose, we used different
pretrained seq2seq summarization models: BART-base and BART-large Lewis et al.
(2019), mBART Liu et al. (2020), and BARThez Eddine et al. (2020). BART is a
seq2seq autoencoder with a Transformer architecture.

The four models were fine-tuned for one epoch on 50k examples randomly sampled
from the training set of CNN/DM, and were used to generate summaries for the
whole training set of 287,112 documents, using greedy decoding.

Note that we kept the 50K documents used for fine-tuning in the final
generation pool, in order to create quality differences among summaries.
Indeed, models are expected to better summarize the documents used for
training than never-seen documents.

We also used the human reference summaries, so that in the end, each document
was associated with 5 summaries, resulting in 10 pairs of summaries per
document.

Backtranslation. We also generated paraphrases with backtranslation, by
sampling sentences from the OpenSubtitle English monolingual corpus Lison and
Tiedemann (2016), and translating them to French, Arabic and German with OPUS-
MT Tiedemann and Thottingal (2020), before translating them back to English.
We used OPUS-MT because of its ready-to-use checkpoints available for many
language pairs. We ended up with 4 variations for each sentence (including the
original one), resulting in 6 paraphrase pairs per sentence.

Denoising. To avoid bias towards summarization and translation, we also
generated pairs of related sequences such that the first element in the pair
was a Wikipedia segment and the second element was a BART-denoised version of
it Lewis et al. (2019).

More precisely, we sampled 2M segments from Wikipedia such that the number of
unigrams in these segments was uniformly distributed between 1 and 200. Our
assumption was that enforcing variations in sequence length would help the
learned metric to generalize.

We then applied BART’s text infilling and sentence permutation perturbation
strategies to each segment. That is, multiple text spans were sampled and
replaced with a [MASK] special token. The lengths of the spans were sampled
from a Poisson distribution ($\lambda=3$). $50\%$ of the tokens within the
input segment were masked and $20\%$ of the masked text was replaced with
random tokens (creating pathological examples to increase the robustness of
the learned metric). The sentences in the input segment were then shuffled.

We finally used a BART-Base
checkpoint444https://dl.fbaipublicfiles.com/fairseq/models/bart.base.tar.gz
from the Fairseq library Ott et al. (2019) to try to reconstruct the perturbed
versions of the original sequences, hence creating variants of them.

Annotating pairs. We sampled 4.5M sequence pairs uniformly from each
aforelisted source. These pairs were then annotated with the metrics to be
learned. Note that this is a one-time operation that does not need to be
repeated regardless of which models are trained downstream.

In this work, we experimented with two recent expensive NLG metrics that rely
on large pretrained language models, BERTScore Zhang et al. (2019) and
MoverScore Zhao et al. (2019), presented in section 2. However, it is
important to note that our method can be used with any other NLG metric.

Note that for BERTScore, we used the F-1 score $F_{BERT}$, as recommended by
the authors Zhang et al. (2019). For MoverScore, still following the authors
Zhao et al. (2019), we used the variant operating on unigrams and the IDF to
compute the vectors of weights.

| Metric | Model |  Scores (TAC) |  Runtime (TAC) |  Scores (WMT) |  Runtime (WMT) | Params
---|---|---|---|---|---|---|---
a | BERTScore | BERT-Tiny | 55.4/47.5 | 1m 27s | 37.6 | 1m 22s | 4.4M
b | BERTScore | BERT-Small | 61.6/51.5 | 2m 20s | 39.1 | 1m 42s | 29.1M
c | BERTScore | BERT-Medium | 62.7/52.4 | 2m 28s | 39.8 | 2m 04s | 41.7M
d | BERTScore | BERT-Base | 64.7/54.7 | 3m 28s | 41.9 | 2m 09s | 110M
e | BERTScore | RoBERTa-Large | 64.2/55.4 | 5m 17s | 43.2 | 3m 03s | 355M
f | BERTScore | DeBERTa-XLarge | 64.5/56.0 | 6m 20s | 44.5 | 3m 49s | 900M
g | MoverScore | BERT-Base |  66.5/55.4 | 301m 29s | 44.0 | 64m 32s | 110M
i | FrugalScored | BERT-Tiny | 64.9/53.5 | 1m 28s | 38.4 | 1m 18s | 4.4M
ii | FrugalScored | BERT-Small | 64.7/53.7 | 2m 29s | 41.3 | 1m 35s | 29.1M
iii | FrugalScored | BERT-Medium | 64.8/54.2 | 3m 41s | 41.9 | 1m 55s | 41.7M
iv | FrugalScoree | BERT-Tiny | 60.0/50.1 | 1m 28s | 37.5 | 1m 18s | 4.4M
v | FrugalScoree | BERT-Small | 64.1/53.8 | 2m 29s | 40.5 | 1m 35s | 29.1M
vi | FrugalScoree | BERT-Medium | 63.9/52.1 | 3m 41s | 41.7 | 1m 55s | 41.7M
vii | FrugalScoref | BERT-Tiny | 61.7/51.0 | 1m 28s | 38.0 | 1m 18s | 4.4M
viii | FrugalScoref | BERT-Small | 66.0/54.9 | 2m 29s | 41.5 | 1m 35s | 29.1M
ix | FrugalScoref | BERT-Medium | 65.5/54.9 | 3m 41s | 43.0 | 1m 55s | 41.7M
x | FrugalScoreg | BERT-Tiny |  67.3/55.1 | 1m 28s | 39.8 | 1m 18s | 4.4M
xi | FrugalScoreg | BERT-Small | 65.9/54.7 | 2m 29s | 42.8 | 1m 35s | 29.1M
xii | FrugalScoreg | BERT-Medium | 66.2/55.1 | 3m 41s | 43.6 | 1m 55s | 41.7M
Table 1:  Scores are summary-level (TAC) and segment-level (WMT) Pearson
correlations averaged over 2008 to 2011 for TAC (pyramid score/responsiveness)
and over all source languages for WMT-2019. Runtimes include preprocessing.
Subscripts refer to row labels and indicate which metric-model combination was
used to annotate pairs (e.g., for FrugalScored, it is row $d$, i.e.,
BERTScore-BERT-Base).

###  3.2 Metric Learning

We continue the pretraining of three BERT
miniatures555https://huggingface.co/google on our synthetic dataset: BERT-Tiny
($L=2$, $H=128$), BERT-Small ($L=4$, $H=512$) and BERT-Medium ($L=8$,
$H=512$), where $L$ is the number of layers and $H$ is the dimension of the
embedding space. These models have respectively 25 times, 3.78 times, and 2.64
times less parameters than BERT-base. The concept of BERT miniatures was
introduced by Turc et al. (2019) to test whether pretraining small models from
scratch was competitive to distilling very large models. The miniature models
have already been pretrained on masked language model and next sentence
prediction objectives.

We continue pretraining using the standard method introduced by Devlin et al.
(2018). We concatenate the two sequences $x=\langle x_{1},...,x_{k}\rangle$
and $y=\langle y_{1},...,y_{l}\rangle$ in a given pair, separating them with a
special [SEP] token. A special [CLS] token is also added at the beginning of
the resulting sequence. The sequence of contextualised embeddings
$\langle\mathbf{z_{[CLS]}},\mathbf{x_{1}},...\mathbf{x_{k}},\mathbf{z_{[SEP]}},\mathbf{y_{1}},...,\mathbf{y_{l}}\rangle$
is then obtained. We finally add a fully connected layer on top, that linearly
projects the $\mathbf{z_{[CLS]}}$ vector to a scalar $s$.

The model is trained to minimize the mean square error (MSE) loss between the
learned metric $s_{i}$ and the metric to be learned $\hat{s_{i}}$ (i.e., the
annotation of the pair):

$l=\frac{1}{N}\sum_{n=1}^{N}||s_{i}-\hat{s_{i}}||^{2}$ (5)

When pretraining is over, the models can be further fine-tuned on smaller
human-annotated datasets as shown in section 6, or directly used to generate
scores for unseen examples as shown in section 4.

Setup. We use a batch size of 32 and the Adam optimizer Kingma and Ba (2014)
with a learning rate of $3\times 10^{-5}$, linear decay, and a warm-up for 6%
of the total training steps, and we train the model for three epochs. We
conducted the pretraining on a single TITAN RTX GPU (24GB). It took 10, 24 and
33 hours, respectively for the tiny, small, and medium miniatures. We rely on
the Transformers library Wolf et al. (2019) for all pretraining and fine-
tuning experiments.

##  4 Experiments

In this section, FrugalScore is used in inference mode to generate scores
directly after pretraining, i.e., no fine-tuning is performed (see section 6
for fine-tuning results).

We evaluate on two text generation tasks: summarization and translation. We
use evaluation datasets containing (reference, candidate) sequence pairs
annotated with human scores assessing the quality of the candidates given the
references. We measure the effectiveness of FrugalScore by measuring the
Pearson correlation of its scores with the human judgments and comparing it to
that of the original metrics. We also take the number of parameters and the
runtime into account.

Text Summarization. We use 4 different multi-document summarization datasets
from the Text Analysis Conference (TAC)666https://tac.nist.gov/: TAC-2008,
TAC-2009, TAC-2010 and TAC-2011.

These datasets respectively contain 48, 44, 46 and 44 clusters of documents
and 58, 55, 43 and 51 systems are used to generate summaries. Each cluster
forms a topic to be summarized and has 4 reference summaries. There are
approximately 10k pairs in each dataset. Each pair is annotated with two human
judgment scores: the Pyramid Score Harnly et al. (2005) and the Responsiveness
Dang et al. (2008). The former measures the proportion of important semantic
units (SCUs) in the reference summaries captured by the system summary, while
the latter reflects the content coverage and the readability of each summary.

Machine Translation. Our evaluation corpus is from the
WMT-2019777http://www.statmt.org/wmt19/ shared task Li et al. (2019). We
consider all the to-English pairs: Chinese, Czech, German, Finnish, Russian,
Lithuanian and Kazakh to English. For each language, we use the test set that
contains several thousands of reference-candidate pairs annotated with human
ratings that assess the translation quality.

##  5 Results

Table 1 reports the results averaged over the 4 TAC datsets and the 7 WMT to-
English language pairs. Details are provided in Appendices A and B.

We benchmarked the metrics in terms of Pearson correlations with human scores,
runtimes, and numbers of parameters. We used two approaches to compute the
Pearson correlations: summary-level (or segment-level) and system-level.

In the former approach, a score is attributed to each of the output
candidates, while in the latter approach, one single overall score is
attributed to the system (by averaging its individual scores).

Rows a to c correspond to BERTScore with BERT miniatures as the underlying
model. They are simple baselines added for the sake of comparison, to see what
we get when BERTScore is used with the same number of parameters as
FrugalScore.

Rows d to g correspond to the expensive metrics that are learned by
FrugalScore (in the respective sections of the bottom half of the table). They
are BERTScore and MoverScore metrics where the underlying model is a large
pretrained language model: BERT-Base ($L=12$, $H=512$), RoBERTa-Large ($L=24$,
$H=1024$) (Liu et al., 2019), and DeBERTa-XLarge ($L=24$, $H=1536$) (He et
al., 2020).

Finally, rows i to xii correspond to FrugalScore. Subscripts refer to row
labels and indicate which metric-model combination was used to annotate pairs.
I.e., FrugalScored learned the metric of row $d$, i.e., BERTScore with BERT-
Base.

Note that FrugalScores were only created for the metrics relying on large
models (rows d to g). Rows a to c, as was already explained, are just for
sanity checking.

First, results show that all FrugalScores, regardless of which metric they
learned, significantly outperform the BERTScores with miniature models. These
results suggest that FrugalScore is a better approach than using an existing
metric with an already-compressed underlying model. The reason for this is
probably that in FrugalScore, the knowledge of the original unsupervised
metric (based on a large model) is explicitly transferred to the miniature via
the continuation of its pretraining on the synthetic dataset. That is, the
miniature is actually learning a metric. Whereas, on the other hand, plugging
a compressed version of a general-purpose language model into the original
unsupervised metric just makes it lose expressiveness and capacity.

Second, we can clearly see that FrugalScore retains most of the performance of
the original metric, while running several times faster and reducing the
number of parameters by several orders of magnitude. On average over all
metrics, tasks, and miniatures, FrugalScore retains 96.8% of the original
performance, runs 24 times faster, and has 35 times less parameters.

More precisely, on average across all metrics, FrugalScore-Tiny retains
97.7/94.7% of the original performance on TAC (pyramid score/responsiveness),
while running 54 times faster and having 84 times less parameters. Its small
and medium versions retain near full performance in terms of responsiveness
(98 and 97.7%) and even slightly outperform the original metrics in terms of
pyramid score, while at the same time reducing the runtime and the number of
parameters by 32 (resp. 21) and 13 (resp. 9) times.

On WMT, FrugalScore-Tiny retains 88.58% of the performance of the original
metrics, while running 14 times faster (and still having 84 times less
parameters), while the small and medium versions of FrugalScore retain 95.71 %
and 98.06% of the original performance while still offering a 32 times (resp.
21) speedup and having 13 times (resp. 9) less parameters, on average.

Interestingly, FrugalScore even improves the performance of the original
metrics in some cases. For example, on TAC, FrugalScoreg with BERT-Tiny (row
x) improves the performance of the original MoverScore metric based on BERT-
Base (row g) from 66.5 to 67.3 in terms of pyramid score, while reducing the
number of parameters by 25 and running 50 times faster. Other examples, also
for TAC with the pyramid score, include FrugalScoref with BERT-Small (row
viii, +1.5 point) and FrugalScoref with BERT-medium (row ix, +1 point).

Finally, the results of FrugalScore for different miniature sizes show that,
on WMT, using larger models always improves performance (e.g., row x
$\rightarrow$ xi $\rightarrow$ xii). But interestingly, on TAC, this
observation does not hold (e.g., row vi $\rightarrow$ viii $\rightarrow$ ix),
and sometimes, FrugalScore with the smallest miniature (BERT-Tiny) is superior
(e.g. rows i and x). This finding suggests that the impact of the pretrained
language model size is task-dependent.

To sum up, results clearly show the effectiveness of FrugalScore in learning a
cheaper, lighter, and faster version of the original metrics, while retaining
most of their original performance. The system-level correlations, provided in
Appendices C and D, corroborate these positive results.

|  Pretraining Continued | TAC-2008 | TAC-2009 | TAC-2010 | TAC-2011 | Average
---|---|---|---|---|---|---
TAC-08 | no | - | 67.70.57 | 66.10.18 | 63.60.36 | 65.8
yes | 74.40.13 | 71.30.04 | 67.30.13 | 71.0
TAC-09 | no | 61.40.41 | - | 66.90.24 | 62.70.55 | 63.7
yes | 65.80.25 | 70.70.32 | 66.00.18 | 67.5
TAC-10 | no | 59.70.47 | 67.30.7 | - | 62.40.47 | 63.1
yes | 64.70.19 | 74.30.24 | 67.20.11 | 68.7
TAC-11 | no | 57.61.39 | 64.71.03 | 66.50.66 | - | 62.9
yes | 63.90.31 | 72.00.44 | 71.60.44 | 69.2
Table 2:  Summary-level Pearson correlations with human judgments (Pyramid
scores), averaged over 3 runs (standard deviation as subscript). Rows
correspond to the training sets and columns to the test sets.

##  6 Fine-tuning on Human Annotations

We test two hypotheses in this section: (1) whether fine-tuning on a human-
annotated dataset is beneficial, and (2) when fine-tuning on human
annotations, whether continuing pretraining on our synthetic dataset is
useful.

Because we cannot use the same dataset for fine-tuning and evaluation, we
fine-tune a BERT-Small on each year of TAC 2008-2011 for 4 epochs, using two
other years as the validation set, and the remaining year as the test set. The
best epoch is selected based on validation performance. We use a batch size of
32 and a learning rate of 2e-5 that linearly decreases to zero. Finally, we
experiment with two scenarios: fine-tuning the miniature directly without
continuing its pretraining on our synthetic dataset, and fine-tuning it after
the pretraining continuation (with annotations generated by BERTScore-BERT-
Base).

Results. Results are reported in Table 2 in terms of summary-level Pearson
correlations with human evaluations (Pyramid), averaged over 3 runs with
different random seeds.

First, it is obvious that everywhere, continuing the pretraining on our
synthetic dataset leads to a significant boost in performance. This is in
accordance with Sellam et al. (2020), who found that pretraining was
beneficial even in a supervised setting.

Second, even if a direct comparison is not possible, we can remark when
looking at the TAC Pyramid score of row ii) in Table 1 (FrugalScored-BERT-
Small) that fine-tuning after pretraining seems very beneficial too. Indeed,
after fine-tuning, we reach on average 71, 67.5, 68.7, and 69.2 (depending on
the split), which represents overall a gain of 4.4 points over the non-fine-
tuned model (score of 64.7).

##  7 Impact of Data Sources

To test the importance of each data source introduced in subsection 3.1, we
created a training set containing sequence pairs uniformly and equally sampled
from each source. We annotated these pairs with the BERTScore-BERT-Base metric
and we used them to continue the pretraining of a BERT-Small miniature.

We also considered pairs drawn at random from the pairs generated with the
other strategies. The motivation for random pairs was to sample “negative
examples”, as seeing only “positive examples” (pairs of related sequences)
could bias the learned metric towards considering any two unrelated sequences
as similar.

We then continued the pretraining of the BERT-Small miniature four times,
excluding each time the pairs coming from a specific data source. We evaluated
the learned metric on TAC-2008 to 2011 and on WMT-2019. Figure 1 shows the
average improvements in the Pearson correlation with human judgments relative
to training a model on all sources. Note that when training on all four
sources, we sampled 30k pairs from each source (120k total), and when
excluding a source, we sampled 40k pairs from each source (120k total).

We can clearly see that excluding the random pairs improves performance while
excluding any of the other data sources decreases performance. In other words,
all our data sources are beneficial, and it is not necessary to add “negative
examples”. We hypothesise that this is due to the fact that NLG datasets
typically do not contain completely unrelated pairs of sentences.
Interestingly, the pairs generated with the backtranslation strategy have the
greatest impact on performance.

Figure 1: Relative improvement in Pearson correlation compared to a dataset
covering all sources. Left: TAC. Right: WMT.

##  8 Conclusion

We proposed FrugalScore, an approach to learn a fixed, low-cost version of any
expensive NLG evaluation metric. Experiments on summarization and translation
tasks show that our FrugalScore versions of BERTScore and MoverScore retain
most of the original performance in terms of the correlation with human
judgments, while running several times faster and having several orders of
magnitude less parameters. On average over all learned metrics, tasks, and
variants, FrugalScore retains 96.8% of the performance, runs 24 times faster,
and has 35 times less parameters than the original metrics.

## References

  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_.
  * Buciluǎ et al. (2006) Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil. 2006.  Model compression.  In _Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 535–541.
  * Clark et al. (2019) Elizabeth Clark, Asli Celikyilmaz, and Noah A Smith. 2019.  Sentence mover’s similarity: Automatic evaluation for multi-sentence texts.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2748–2760.
  * Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017.  Supervised learning of universal sentence representations from natural language inference data.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 670–680, Copenhagen, Denmark. Association for Computational Linguistics.
  * Conroy and Dang (2008) John M. Conroy and Hoa Trang Dang. 2008.  Mind the gap: Dangers of divorcing evaluations of summary content from linguistic quality.  In _Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008)_ , pages 145–152, Manchester, UK. Coling 2008 Organizing Committee.
  * Dang et al. (2008) Hoa Trang Dang, Karolina Owczarzak, et al. 2008.  Overview of the tac 2008 update summarization task.  In _TAC_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Eddine et al. (2020) Moussa Kamal Eddine, Antoine J-P Tixier, and Michalis Vazirgiannis. 2020.  Barthez: a skilled pretrained french sequence-to-sequence model.  _arXiv preprint arXiv:2010.12321_.
  * Gardent et al. (2017) Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017.  The WebNLG challenge: Generating text from RDF data.  In _Proceedings of the 10th International Conference on Natural Language Generation_ , pages 124–133, Santiago de Compostela, Spain. Association for Computational Linguistics.
  * Harnly et al. (2005) Aaron Harnly, Ani Nenkova, Rebecca Passonneau, and Owen Rambow. 2005.  Automation of summary evaluation by the pyramid method.  In _Recent Advances in Natural Language Processing (RANLP)_ , pages 226–232.
  * He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020.  Deberta: Decoding-enhanced bert with disentangled attention.  _arXiv preprint arXiv:2006.03654_.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_.
  * Kane et al. (2020) Hassan Kane, Muhammed Yusuf Kocyigit, Ali Abdalla, Pelkins Ajanoh, and Mohamed Coulibali. 2020.  NUBIA: NeUral based interchangeability assessor for text generation.  In _Proceedings of the 1st Workshop on Evaluating NLG Evaluation_ , pages 28–37, Online (Dublin, Ireland). Association for Computational Linguistics.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kusner et al. (2015) Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015.  From word embeddings to document distances.  In _International conference on machine learning_ , pages 957–966. PMLR.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_.
  * Li et al. (2019) Xian Li, Paul Michel, Antonios Anastasopoulos, Yonatan Belinkov, Nadir Durrani, Orhan Firat, Philipp Koehn, Graham Neubig, Juan Pino, and Hassan Sajjad. 2019\.  Findings of the first shared task on machine translation robustness.  _arXiv preprint arXiv:1906.11943_.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81.
  * Lison and Tiedemann (2016) Pierre Lison and Jörg Tiedemann. 2016.  Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 923–929.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _arXiv preprint arXiv:2001.08210_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Lo (2017) Chi-kiu Lo. 2017.  MEANT 2.0: Accurate semantic MT evaluation for any output language.  In _Proceedings of the Second Conference on Machine Translation_ , pages 589–597, Copenhagen, Denmark. Association for Computational Linguistics.
  * Ma et al. (2017) Qingsong Ma, Yvette Graham, Shugen Wang, and Qun Liu. 2017.  Blend: a novel combined mt metric based on direct assessment—casict-dcu submission to wmt17 metrics task.  In _Proceedings of the second conference on machine translation_ , pages 598–603.
  * Mathur et al. (2019) Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019.  Putting evaluation in context: Contextual embeddings improve machine translation evaluation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2799–2808.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_.
  * Mordido and Meinel (2020) Gonçalo Mordido and Christoph Meinel. 2020.  Mark-evaluate: Assessing language generation using population estimation methods.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 1963–1977, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016.  Abstractive text summarization using sequence-to-sequence rnns and beyond.  _arXiv preprint arXiv:1602.06023_.
  * Ng and Abrecht (2015) Jun-Ping Ng and Viktoria Abrecht. 2015.  Better summarization evaluation with word embeddings for ROUGE.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1925–1930, Lisbon, Portugal. Association for Computational Linguistics.
  * Novikova et al. (2017) Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017.  Why we need new evaluation metrics for nlg.  _arXiv preprint arXiv:1707.06875_.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  _arXiv preprint arXiv:1904.01038_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  * Peyrard et al. (2017) Maxime Peyrard, Teresa Botschen, and Iryna Gurevych. 2017.  Learning to score system summaries for better content selection evaluation. In _Proceedings of the Workshop on New Frontiers in Summarization_ , pages 74–84, Copenhagen, Denmark. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020.  Bleurt: Learning robust metrics for text generation.  _arXiv preprint arXiv:2004.04696_.
  * Shimanaka et al. (2018) Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. 2018.  RUSE: Regressor using sentence embeddings for automatic machine translation evaluation.  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , pages 751–758, Belgium, Brussels. Association for Computational Linguistics.
  * Stanojević and Sima’an (2014) Miloš Stanojević and Khalil Sima’an. 2014.  Beer: Better evaluation as ranking.  In _Proceedings of the Ninth Workshop on Statistical Machine Translation_ , pages 414–419.
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019.  Energy and policy considerations for deep learning in nlp.  _arXiv preprint arXiv:1906.02243_.
  * Tättar and Fishel (2017) Andre Tättar and Mark Fishel. 2017.  bleu2vec: the painfully familiar metric on continuous vector space steroids.  In _Proceedings of the Second Conference on Machine Translation_ , pages 619–622.
  * Tiedemann and Thottingal (2020) Jörg Tiedemann and Santhosh Thottingal. 2020.  OPUS-MT — Building open translation services for the World.  In _Proceedings of the 22nd Annual Conferenec of the European Association for Machine Translation (EAMT)_ , Lisbon, Portugal.
  * Turc et al. (2019) Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Well-read students learn better: On the importance of pre-training compact models.  _arXiv preprint arXiv:1908.08962_.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.
  * Yu et al. (2015) Hui Yu, Qingsong Ma, Xiaofeng Wu, and Qun Liu. 2015.  CASICT-DCU participation in WMT2015 metrics task.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 417–421, Lisbon, Portugal. Association for Computational Linguistics.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019\.  Bertscore: Evaluating text generation with bert.  _arXiv preprint arXiv:1904.09675_.
  * Zhao et al. (2019) Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. 2019.  Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance.  _arXiv preprint arXiv:1909.02622_.

##  Appendix A Detailed TAC Evaluation per Year

| Metric | Model | TAC-2008 | TAC-2009 | TAC-2010 | TAC-2011 |  Macro Avg. Score | Runtime | Params
---|---|---|---|---|---|---|---|---|---
a | BERTScore | BERT-Tiny | 52.1/44.4 | 62.2/51.9 | 54.6/49.9 | 52.7/43.6 | 55.4/47.5 | 1m 27s | 4.4M
b | BERTScore | BERT-Small | 56.0/47.8 | 70.0/54.6 | 61.1/54.5 | 59.1/49.2 | 61.6/51.5 | 2m 20s | 29.1M
c | BERTScore | BERT-Medium | 57.3/48.5 | 70.6/55.3 | 63.1/56.2 | 59.7/49.5 | 62.7/52.4 | 2m 28s | 41.7M
d | BERTScore | BERT-Base | 61.3/52.2 | 73.2/58.7 | 63.3/56.8 | 61.0/51.2 | 64.7/54.7 | 3m 28s | 110M
e | BERTScore | RoBERTa-Large | 56.4/50.9 | 71.1/58.3 |  69.1/61.4 | 60.3/50.8 | 64.2/55.4 | 5m 17s | 355M
f | BERTScore | DeBERTa-XLarge | 60.9/54.5 |  73.9/60.4 | 62.6/56.0 | 61.5/53.0 | 64.5/56.0 | 6m 20s | 900M
g | MoverScore | BERT-Base |  64.7/54.2 |  73.9/58.2 | 64.7/57.0 |  62.6/52.5 |  66.5/55.4 | 301m 29s | 110M
i | FrugalScored | BERT-Tiny | 60.9/50.0 | 72.5/56.4 | 64.8/57.5 | 61.4/50.0 | 64.9/53.5 | 1m 28s | 4.4M
ii | FrugalScored | BERT-Small | 61.9/51.8 | 73.0/57.3 | 62.6/55.8 | 61.3/50.0 | 64.7/53.7 | 1m 35s | 29.1M
iii | FrugalScored | BERT-Medium | 62.0/52.2 | 73.3/58.1 | 62.6/56.0 | 61.3/50.6 | 64.8/54.2 | 1m 55s | 41.7M
iv | FrugalScoree | BERT-Tiny | 54.8/46.4 | 66.8/54.2 | 61.8/53.1 | 56.4/46.7 | 60.0/50.1 | 1m 28s | 4.4M
v | FrugalScoree | BERT-Small | 59.1/49.6 | 72.7/55.7 |  68.1/59.8 | 63.0/50.1 | 64.1/53.8 | 2m 29s | 29.1M
vi | FrugalScoree | BERT-Medium | 57.9/48.4 | 71.8/54.4 | 65.7/57.0 | 60.3/48.5 | 63.9/52.1 | 3m 41s | 41.7M
vii | FrugalScoref | BERT-Tiny | 57.8/48.5 | 68.6/55.7 | 63.0/54.8 | 57.5/47.8 | 61.7/51.0 | 1m 28s | 4.4M
viii | FrugalScoref | BERT-Small | 60.1/51.0 | 73.5/57.5 | 67.3/59.5 | 63.1/51.7 | 66.0/54.9 | 2m 29s | 29.1M
ix | FrugalScoref | BERT-Medium | 59.0/50.3 | 73.3/57.4 | 67.2/60.2 | 62.4/51.5 | 65.5/54.9 | 3m 41s | 41.7M
x | FrugalScoreg | BERT-Tiny | 63.6/51.7 |  74.4/57.3 | 68.0/60.1 |  63.2/51.2 |  67.3/55.1 | 1m 28s | 4.4M
xi | FrugalScoreg | BERT-Small | 63.2/52.5 | 73.1/57.1 | 65.1/57.6 | 62.3/51.5 | 65.9/54.7 | 2m 29s | 29.1M
xii | FrugalScoreg | BERT-Medium |  63.8/53.2 | 73.6/57.7 | 65.3/57.5 | 62.1/51.8 | 66.2/55.1 | 3m 41s | 41.7M
Table 3: Summary-level Pearson correlation (pyramid score/responsiveness).

##  Appendix B Detailed WMT Evaluation per Language

| Metric | Model | de-en | fi-en | gu-en | kk-en | lt-en | ru-en | zh-en |  Macro Avg. Score | Runtime | Params
---|---|---|---|---|---|---|---|---|---|---|---|---
a | BERTScore | BERT-Tiny | 29.7 | 32.5 | 33.9 | 52.0 | 40.5 | 30.7 | 44.2 | 37.6 | 1m 22s | 4.4M
b | BERTScore | BERT-Small | 30.0 | 33.6 | 34.6 | 52.4 | 42.3 | 31.8 | 49.1 | 39.1 | 1m 42s | 29.1M
c | BERTScore | BERT-Medium | 30.8 | 34.4 | 35.2 | 52.8 | 42.8 | 32.4 | 50.3 | 39.8 | 2m 04s | 41.7M
d | BERTScore | BERT-Base | 32.8 | 37.4 | 37.1 | 54.0 | 44.7 | 33.7 | 53.7 | 41.9 | 2m 09s | 110M
e | BERTScore | RoBERTa-Large | 35.3 | 38.7 | 38.7 | 52.0 | 45.3 | 34.3 | 58.3 | 43.2 | 3m 03s | 355M
f | BERTScore | DeBERTa-XLarge | 37.6 | 39.2 | 40.3 | 53.4 | 47.3 | 35.7 | 57.8 | 44.5 | 3m 49s | 900M
g | MoverScore | BERT-Base | 36.5 | 39.1 | 39.3 | 55.0 | 46.5 | 35.6 | 56.0 | 44.0 | 64m 32s | 110M
i | FrugalScored | BERT-Tiny | 30.2 | 32.8 | 34.6 | 52.4 | 39.9 | 31.2 | 47.7 | 38.4 | 1m 18s | 4.4M
ii | FrugalScored | BERT-Small | 32.6 | 35.9 | 37.1 | 54.1 | 43.5 | 33.6 | 52.3 | 41.3 | 1m 35s | 29.1M
iii | FrugalScored | BERT-Medium | 32.9 | 37.0 | 37.4 | 54.4 | 44.3 | 34.1 | 53.2 | 41.9 | 1m 55s | 41.7M
iv | FrugalScoree | BERT-Tiny | 30.6 | 32.8 | 33.0 | 49.8 | 38.7 | 29.8 | 48.1 | 37.5 | 1m 18s | 4.4M
v | FrugalScoree | BERT-Small | 33.7 | 35.4 | 35.4 | 51.6 | 42.6 | 32.6 | 52.5 | 40.5 | 1m 35s | 29.1M
vi | FrugalScoree | BERT-Medium | 35.2 | 37.1 | 35.6 | 52.0 | 44.0 | 33.8 | 54.4 | 41.7 | 1m 55s | 41.7M
vii | FrugalScoref | BERT-Tiny | 30.8 | 33.1 | 34.4 | 50.8 | 39.4 | 30.4 | 47.1 | 38.0 | 1m 18s | 4.4M
viii | FrugalScoref | BERT-Small | 34.5 | 36.4 | 37.0 | 52.7 | 43.9 | 33.4 | 52.6 | 41.5 | 1m 35s | 29.1M
ix | FrugalScoref | BERT-Medium | 35.8 | 38.3 | 37.7 | 53.4 | 45.7 | 34.8 | 55.1 | 43.0 | 1m 55s | 41.7M
x | FrugalScoreg | BERT-Tiny | 33.0 | 34.0 | 36.2 | 53.6 | 40.5 | 32.7 | 48.6 | 39.8 | 1m 18s | 4.4M
xi | FrugalScoreg | BERT-Small | 35.6 | 37.4 | 38.9 | 55.0 | 44.8 | 34.8 | 52.8 | 42.8 | 1m 35s | 29.1M
xii | FrugalScoreg | BERT-Medium | 36.2 | 38.3 | 39.1 | 55.6 | 45.8 | 35.3 | 54.7 | 43.6 | 1m 55s | 41.7M
Table 4: Segment-level Pearson correlation.

##  Appendix C Detailed TAC Evaluation per Year (System Level)

| Metric | Model | TAC-2008 | TAC-2009 | TAC-2010 | TAC-2011 |  Macro Avg. Score | Runtime | Params
---|---|---|---|---|---|---|---|---|---
a | BERTScore | BERT-Tiny | 82.5/77.6 | 87.4/81.8 | 77.5/75.0 | 82.1/79.2 | 82.4/78.4 | 1m 27s | 4.4M
b | BERTScore | BERT-Small | 84.4/81.4 | 95.8/84.0 | 81.3/78.0 | 87.6/85.3 | 87.3/82.2 | 2m 20s | 29.1M
c | BERTScore | BERT-Medium | 86.3/82.7 | 96.0/84.6 | 84.0/80.6 | 87.8/85.5 | 88.5/83.3 | 2m 28s | 41.7M
d | BERTScore | BERT-Base | 90.6/87.5 | 96.5/87.5 | 83.7/80.9 | 88.3/86.4 | 89.8/85.6 | 3m 28s | 110M
e | BERTScore | RoBERTa-Large | 80.0/80.9 | 94.7/87.7 |  92.7/89.8 | 88.9/89.2 | 89.1/86.9 | 5m 17s | 355M
f | BERTScore | DeBERTa-XLarge | 88.0/89.8 |  97.5/89.8 | 85.7/84.0 |  90.7/91.8 | 90.5/88.9 | 6m 20s | 900M
g | MoverScore | BERT-Base |  95.4/89.5 | 96.9/85.9 | 85.7/84.0 | 88.6/86.0 |  91.7/86.3 | 301m 29s | 110M
i | FrugalScored | BERT-Tiny | 91.6/85.3 | 95.8/84.7 | 86.2/82.9 | 88.3/84.4 | 90.5/84.3 | 1m 28s | 4.4M
ii | FrugalScored | BERT-Small | 90.9/86.8 | 96.2/85.4 | 82.8/79.6 | 87.8/84.3 | 89.4/84.0 | 1m 35s | 29.1M
iii | FrugalScored | BERT-Medium | 90.6/87.0 | 96.6/86.3 | 82.5/79.6 | 87.6/84.9 | 89.3/84.5 | 1m 55s | 41.7M
iv | FrugalScoree | BERT-Tiny | 86.3/81.1 | 95.1/87.1 | 84.5/80.2 | 84.5/80.9 | 87.6/82.3 | 1m 28s | 4.4M
v | FrugalScoree | BERT-Small | 85.1/81.7 | 95.7/83.6 |  91.2/87.5 | 91.7/87.5 | 90.9/85.1 | 2m 29s | 29.1M
vi | FrugalScoree | BERT-Medium | 81.6/80.7 | 95.7/84.1 | 90.9/87.5 | 87.6/85.3 | 89.0/84.4 | 3m 41s | 41.7M
vii | FrugalScoref | BERT-Tiny | 89.7/84.5 | 95.3/87.6 | 85.1/81.4 | 84.8/81.2 | 88.7/83.7 | 1m 28s | 4.4M
viii | FrugalScoref | BERT-Small | 86.8/85.1 | 96.7/85.4 | 89.5/86.2 | 91.6/88.7 | 91.2/86.3 | 2m 29s | 29.1M
ix | FrugalScoref | BERT-Medium | 85.4/86.3 |  97.2/87.2 | 91.1/88.9 |  92.3/91.0 | 91.5/88.3 | 3m 41s | 41.7M
x | FrugalScoreg | BERT-Tiny |  93.7/86.1 | 96.2/83.9 | 90.1/87 | 89.4/84.8 |  92.3/85.5 | 1m 28s | 4.4M
xi | FrugalScoreg | BERT-Small | 93.2/87.6 | 96.4/84.2 | 85/81.7 | 87.9/84.9 | 90.6/84.6 | 2m 29s | 29.1M
xii | FrugalScoreg | BERT-Medium |  93.7/87.5 | 96.5/84.5 | 84.8/81.6 | 87.3/84.7 | 90.6/84.6 | 3m 41s | 41.7M
Table 5: System-level Pearson correlation (pyramid/responsiveness).

##  Appendix D Detailed WMT Evaluation per Language (System Level)

| Metric | Model | de-en | fi-en | gu-en | kk-en | lt-en | ru-en | zh-en |  Macro Avg. Score | Runtime | Params
---|---|---|---|---|---|---|---|---|---|---|---|---
a | BERTScore | BERT-Tiny | 74.1 | 97.9 | 93.1 | 99.77 | 87.9 | 94.5 | 91.7 | 91.3 | 1m 22s | 4.4M
b | BERTScore | BERT-Small | 82.6 | 97.5 | 88.2 | 99.87 | 95.3 | 96.4 | 93.0 | 93.3 | 1m 42s | 29.1M
c | BERTScore | BERT-Medium | 83.7 | 97.7 | 88.2 | 99.86 | 94.4 | 96.2 | 93.5 | 93.4 | 2m 04s | 41.7M
d | BERTScore | BERT-Base | 89.1 | 97.8 | 89.7 | 99.72 | 96.9 | 96.9 | 95.8 | 95.1 | 2m 09s | 110M
e | BERTScore | RoBERTa-Large | 94.0 | 98.4 | 98.1 | 98.00 | 96.1 | 91.0 | 98.2 | 96.3 | 3m 03s | 355M
f | BERTScore | DeBERTa-XLarge | 93.9 | 98.3 | 98.2 | 99.18 | 98.7 | 97.1 | 98.4 | 97.7 | 3m 49s | 900M
g | MoverScore | BERT-Base | 88.1 | 99.1 | 91.2 | 98.58 | 96.0 | 97.2 | 96.4 | 95.2 | 64m 32s | 110M
i | FrugalScored | BERT-Tiny | 81.1 | 98.6 | 94.4 | 99.80 | 92.2 | 95.4 | 93.8 | 93.6 | 1m 18s | 4.4M
ii | FrugalScored | BERT-Small | 86.5 | 98.5 | 93.6 | 99.82 | 95.9 | 97.1 | 94.7 | 95.2 | 1m 35s | 29.1M
iii | FrugalScored | BERT-Medium | 88.3 | 98.3 | 92.1 | 99.79 | 96.4 | 97.2 | 95.4 | 95.4 | 1m 55s | 41.7M
iv | FrugalScoree | BERT-Tiny | 80.2 | 97.7 | 94.9 | 99.73 | 86.4 | 94.6 | 93.7 | 92.5 | 1m 18s | 4.4M
v | FrugalScoree | BERT-Small | 83.9 | 98.0 | 95.2 | 99.79 | 92.4 | 97.0 | 95.1 | 94.5 | 1m 35s | 29.1M
vi | FrugalScoree | BERT-Medium | 88.1 | 97.9 | 93.0 | 99.78 | 94.9 | 97.8 | 96.1 | 95.4 | 1m 55s | 41.7M
vii | FrugalScoref | BERT-Tiny | 81.3 | 97.9 | 96.1 | 99.81 | 89.8 | 94.7 | 93.7 | 93.3 | 1m 18s | 4.4M
viii | FrugalScoref | BERT-Small | 85.8 | 97.7 | 96.2 | 99.85 | 95.3 | 97.3 | 95.7 | 95.4 | 1m 35s | 29.1M
ix | FrugalScoref | BERT-Medium | 89.9 | 97.9 | 90.8 | 99.85 | 97.6 | 97.8 | 96.9 | 95.8 | 1m 55s | 41.7M
x | FrugalScoreg | BERT-Tiny | 81.8 | 98.9 | 95.6 | 99.73 | 92.1 | 95.6 | 94.4 | 94.0 | 1m 18s | 4.4M
xi | FrugalScoreg | BERT-Small | 85.4 | 98.8 | 95.8 | 99.52 | 94.9 | 96.8 | 95.3 | 95.2 | 1m 35s | 29.1M
xii | FrugalScoreg | BERT-Medium | 87.0 | 98.8 | 93.5 | 99.29 | 95.6 | 97.0 | 95.9 | 95.3 | 1m 55s | 41.7M
Table 6: System-level Pearson correlation.


# How to decide whether two convex octahedra
are affinely equivalent using their natural
developments only

Victor Alexandrov
Sobolev Institute of Mathematics, Novosibirsk, Russia &
Department of Physics, Novosibirsk State University, Russia
<EMAIL_ADDRESS>
###### Abstract

Given two convex octahedra in Euclidean 3-space, we find conditions on their
natural developments which are necessary and sufficient for these octahedra to
be affinely equivalent to each other.

Keywords: affine transformation, octahedron, natural development,
Cayley–Menger determinant, spatial shape of a polyhedron

Mathematics subject classification (2010): 52C25, 52B10, 68U05

Dedicated to Professor Hellmuth Stachel on the occasion of his 80th birthday

##  1 Introduction

The question we shall address in this article is this:

######  Problem 1.

Given two convex octahedra in Euclidean 3-space, is it possible to decide
whether they are affinely equivalent to each other using their natural
developments only?

From our point of view, Problem 1 is similar to the problem of recognizing
congruent convex polyhedra whose solution is given by the famous Cauchy
rigidity theorem: If natural developments of two convex closed polyhedra in
Euclidean 3-space are isometric then these polyhedra are congruent to each
other. Initially, this theorem was proved by A.L. Cauchy in 1813 in [5]. An
extensive literature is devoted to the Cauchy rigidity theorem and its
generalizations, from which we mention the monograph [1], review article [6],
scientific article [8], and popular science book [7], where the reader can
find further references.

An analog of Problem 1 for general polyhedra in Euclidean 3-space is
considered in [2], where necessary conditions for the affine equivalence of
polyhedra are found. In this article, we present a similar approach as applied
to octahedra. This allows us to obtain both necessary and sufficient
conditions.

The main result of this article is Theorem 1 from Section 4.

We choose octahedra as the object of our study for two reasons. First, because
they are the simplest polyhedra in terms of combinatorial structure with no
trivalent vertices (the latter simplify the problem of recognition of affinely
equivalent polyhedra by their natural developments, see [2]). Second, because
historically octahedra played an intriguing role in the proof of the Cauchy
rigidity theorem (see [9, p. 446]) while our study is motivated by that
theorem.

##  2 Terminology and preliminaries

### Octahedron

We say that a polyhedral surface in $\mathbb{R}^{3}$ is an octahedron if it is
combinatorially equivalent to the regular convex octahedron in
$\mathbb{R}^{3}$ and any two adjacent faces do not lie in the same plane.

A straight line segment $xy$ is called a diagonal of an octahedron $P$ if $x$
and $y$ are vertices of $P$ and $xy$ is not an edge of $P$.

An octahedron $P$ is called convex if, for every its face $F$, the three
vertices of $P$ which are not incident to $F$, are contained in a single open
halfspace determined by $\mbox{aff\,}F$, the affine hull of $F$.

We leave the proofs of the following statements to the reader: every convex
octahedron $P$ is the boundary of a bounded convex set $B$ with nonempty
interior, i.e., of a convex body; moreover, $B$ contains every diagonal of
$P$.

### Natural development

The concept of a natural development of an octahedron is intuitively obvious
to anyone who has ever glued an octahedron from a set of its faces which are
cut out from cardboard. Avoiding unnecessary details, we give the following
more or less formal definition.

A set consisting of 8 triangles in the plane is called the natural development
$R$ of an octahedron $P$ if it is equipped with “gluing rules” and the
following conditions are fulfilled:

$\bullet$ $R$ is in one-to-one correspondence with the set of the faces of
$P$;

$\bullet$ each triangle of $R$ is congruent to the corresponding face of $P$;

$\bullet$ vertices of different triangles of $R$ are identified with each
other (or “are glued together”) if and only if they correspond to the same
vertex of $P$;

$\bullet$ sides of different triangles of $R$ are identified with each other
(or “are glued together”) if and only if they correspond to the same edge of
$P$.

### Cayley–Menger determinant

Let $M$ be an abstract set and $\rho:M\times M\to[0,+\infty)\subset\mathbb{R}$
be a map such that $\rho(x,y)=\rho(y,x)$ for all $x,y\in M$ and $\rho(x,y)=0$
if and only if $x=y$. Then the pair $(M,\rho)$ is called a semimetric space,
see [3, Definition 5.1, p. 7], and, for every
$\\{x_{0},x_{1},\dots,x_{k}\\}\subset M$, the expression

$\operatorname{cm}(x_{0},x_{1},\dots,x_{k})\stackrel{{\scriptstyle\textrm{def}}}{{=}}\left|\begin{array}[]{ccccc}0&1&1&\dots&1\\\
1&0&\rho^{2}(x_{0},x_{1})&\dots&\rho^{2}(x_{0},x_{k})\\\
1&\rho^{2}(x_{1},x_{0})&0&\dots&\rho^{2}(x_{1},x_{k})\\\ .&.&.&.&.\\\
1&\rho^{2}(x_{k},x_{0})&\rho^{2}(x_{k},x_{1})&\dots&0\end{array}\right|$

is called the Cayley–Menger determinant of $x_{0},x_{1},\dots,x_{k}$, see [3,
§ 40, p. 97].

If $M=\\{x_{0},x_{1},\dots,x_{n}\\}\subset\mathbb{R}^{n}$ and a metric $\rho$
in $M$ is induced by the Euclidean metric of $\mathbb{R}^{n}$ then
$\operatorname{vol}(x_{0},x_{1},\dots,x_{n})$, the $n$-dimensional volume of
the simplex with the vertices $x_{0},x_{1},\dots,x_{n}$, is related to the
Cayley–Menger determinant of $x_{0},x_{1},\dots,x_{n}$ by the following
formula, see [3, § 40, p. 98]:

$[\operatorname{vol}(x_{0},x_{1},\dots,x_{n})]^{2}=\frac{(-1)^{n+1}}{2^{n}n!}\operatorname{cm}(x_{0},x_{1},\dots,x_{n}).$
(1)

We say that a semimetric space $(M,\rho)$ embeds isometrically in
$\mathbb{R}^{n}$ if there is $f:M\to\mathbb{R}^{n}$ such that
$|f(x)-f(y)|=\rho(x,y)$ for every $x,y\in M$, where $|f(x)-f(y)|$ stands for
the Euclidean distance between $f(x),f(y)\in\mathbb{R}^{n}$.

Necessary and sufficient conditions for a semimetric space $(M,\rho)$ to embed
isometrically in $\mathbb{R}^{n}$ are given by the following

######  Theorem 1 (K. Menger, 1928).

For every semimetric space $M$, the following statements _(i)_ and _(ii)_ are
equivalent to each other:

_(i)_ $M$ embeds isometrically in $\mathbb{R}^{n}$, but not in
$\mathbb{R}^{n-1}$;

_(ii)_ there are $n+1$ points $x_{0},x_{1},\dots,x_{n}$ in $M$ such that

$(-1)^{k+1}\operatorname{cm}(x_{0},x_{1},\dots,x_{k})>0$ (2)

for all $k=1,2,\dots,n$ and

$\displaystyle\operatorname{cm}(x_{0},x_{1},\dots,x_{n},x)$
$\displaystyle\,=\,$ $\displaystyle 0,$ (3)
$\displaystyle\operatorname{cm}(x_{0},x_{1},\dots,x_{n},y)$
$\displaystyle\,=\,$ $\displaystyle 0,$ (4)
$\displaystyle\operatorname{cm}(x_{0},x_{1},\dots,x_{n},x,y)$
$\displaystyle\,=\,$ $\displaystyle 0$ (5)

for every pair of points $x,y$ of $M$.

The classic reference for Theorem 1 is [3, Theorem 42.2, p. 104]. A modern
exposition of Theorem 1 for the case of metric spaces may be found in [4].

##  3 Necessary conditions

Throughout this Section, we denote by $P$ and $P^{\prime}$ two fixed octahedra
in $\mathbb{R}^{3}$.

By $x_{i}$, $i=0,\dots,5$, denote the vertices of $P$. Throughout this
article, we fix their special enumeration, an example of which is shown in
Fig. 1. Such a special enumeration is obtained in the following way. Chose an
edge $e$ of $P$ arbitrarily and denote by $x_{0}$ and $x_{1}$ the vertices of
$P$ incident to $e$. For $i=2,3$, by $x_{i}$ denote a vertex of $P$ incident
to a face of $P$ containing $e$. By $x_{4}$ (resp., $x_{5}$) denote a vertex
of $P$ such that the straight line segment $x_{1}x_{4}$ (resp., $x_{0}x_{5}$)
is a diagonal of $P$.

Figure 1: Special enumeration of the vertices of an octahedron $P$

By $\langle x_{i},x_{j}\rangle$ denote the edge of $P$ incident to the
vertices $x_{i},x_{j}$ if such an edge exists. Similarly, by $\langle
x_{i},x_{j},x_{k}\rangle$ denote the face of $P$ incident to the vertices
$x_{i},x_{j},x_{k}$ if such a face exists.

Consider the set $V=\\{x_{0},\dots,x_{5}\\}$ endowed with a metric induced
from $\mathbb{R}^{3}$. For the convenience of further presentation, we denote
the Euclidean distance $|x_{i}-x_{j}|$ between $x_{i},x_{j}\in V$ by $d_{ij}$
if $x_{i}$ and $x_{j}$ are joined by an edge of $P$, and by $\delta_{ij}$
otherwise. In other words, we denote $|x_{i}-x_{j}|$ by $d_{ij}$ if this
distance is given to us directly in the natural development of $P$, and denote
it by $\delta_{ij}$ otherwise. We choose different notations $d_{ij}$ and
$\delta_{ij}$ because, when solving Problem 1 in Section 4, we consider
$\delta_{ij}$ as unknown quantities, the values of which have to be found in
the course of the solution.

Throughout Section 3, we denote by $x^{\prime}_{i}$, $i=0,\dots,5$, the
vertices of $P^{\prime}$ and assume that $P^{\prime}$ is affinely equivalent
to $P$ in the sense that there exists an affine transformation
$A:\mathbb{R}^{3}\to\mathbb{R}^{3}$ such that $A(P)=P^{\prime}$ and
$A(x_{i})=x^{\prime}_{i}$ for all $i=0,\dots,5$, i. e., that $A$ does not
change the numbering of the vertices. The Euclidean distance between the
vertices $x^{\prime}_{i}$ and $x^{\prime}_{j}$ of $P^{\prime}$ is denoted by
$d^{\prime}_{ij}$ if $x^{\prime}_{i}$ and $x^{\prime}_{j}$ are connected by an
edge of $P^{\prime}$ and by $\delta^{\prime}_{ij}$ otherwise.

### The first group of necessary conditions

Directly by its construction, the set $V=\\{x_{0},\dots,x_{5}\\}$ endowed with
the metric induced from $\mathbb{R}^{3}$ is a metric space which embeds
isometrically in $\mathbb{R}^{3}$, but not in $\mathbb{R}^{2}$. Hence, by
Theorem 1, the inequalities (2) and equalities (3)–(5) hold. Let us write them
down in detail.

The inequalities (2) take the form

$\displaystyle(-1)^{2}\operatorname{cm}(x_{0},x_{1})$ $\displaystyle\,>\,$
$\displaystyle 0,$ $\displaystyle(-1)^{3}\operatorname{cm}(x_{0},x_{1},x_{2})$
$\displaystyle\,>\,$ $\displaystyle 0,$
$\displaystyle(-1)^{4}\operatorname{cm}(x_{0},x_{1},x_{2},x_{3})$
$\displaystyle\,>\,$ $\displaystyle 0,$

i.e.,

$d^{2}_{01}>0,\qquad\left|\begin{array}[]{cccc}0&1&1&1\\\
1&0&d^{2}_{01}&d^{2}_{02}\\\ 1&d^{2}_{01}&0&d^{2}_{12}\\\
1&d^{2}_{02}&d^{2}_{12}&0\\\ \end{array}\right|<0,$ (6)
$\left|\begin{array}[]{ccccc}0&1&1&1&1\\\
1&0&d^{2}_{01}&d^{2}_{02}&d^{2}_{03}\\\
1&d^{2}_{01}&0&d^{2}_{12}&d^{2}_{13}\\\
1&d^{2}_{02}&d^{2}_{12}&0&\delta^{2}_{23}\\\
1&d^{2}_{03}&d^{2}_{13}&\delta^{2}_{23}&0\end{array}\right|>0.$ (7)

The inequalities (6) mean that $x_{0}\neq x_{1}$ and the area of the face
$\langle x_{0},x_{1},x_{2}\rangle$ is not equal to zero. The inequality (7)
means that the volume of the tetrahedron $\langle
x_{0},x_{1},x_{2},x_{3}\rangle$ with the vertices $x_{i}$, $i=0,\dots,3$ is
nonzero. According to our definition of an octahedron, (6)–(7) are
automatically satisfied.

Note that if we write (7) using only numerical data that is available to us
from the natural development of $P$ (i.e., using $d_{ij}$ only) then (7)
imposes a restriction on the possible values of the length $\delta_{23}$ of
the diagonal $x_{2}x_{3}$ of $P$.

The equality (3), when written for $P$, becomes
$\operatorname{cm}(x_{0},x_{1},x_{2},x_{3},x_{4}){=}0$, i.e.,

$\left|\begin{array}[]{cccccc}0&1&1&1&1&1\\\
1&0&d^{2}_{01}&d^{2}_{02}&d^{2}_{03}&d^{2}_{04}\\\
1&d^{2}_{01}&0&d^{2}_{12}&d^{2}_{13}&\delta^{2}_{14}\\\
1&d^{2}_{02}&d^{2}_{12}&0&\delta^{2}_{23}&d^{2}_{24}\\\
1&d^{2}_{03}&d^{2}_{13}&\delta^{2}_{23}&0&d^{2}_{34}\\\
1&d^{2}_{04}&\delta^{2}_{14}&d^{2}_{24}&d^{2}_{34}&0\end{array}\right|=0.$ (8)

The equality (4) takes the form
$\operatorname{cm}(x_{0},x_{1},x_{2},x_{3},x_{5})=0$, i.e.,

$\left|\begin{array}[]{cccccc}0&1&1&1&1&1\\\
1&0&d^{2}_{01}&d^{2}_{02}&d^{2}_{03}&\delta^{2}_{05}\\\
1&d^{2}_{01}&0&d^{2}_{12}&d^{2}_{13}&d^{2}_{15}\\\
1&d^{2}_{02}&d^{2}_{12}&0&\delta^{2}_{23}&d^{2}_{25}\\\
1&d^{2}_{03}&d^{2}_{13}&\delta^{2}_{23}&0&d^{2}_{35}\\\
1&\delta^{2}_{05}&d^{2}_{15}&d^{2}_{25}&d^{2}_{35}&0\end{array}\right|=0.$ (9)

Finally, the equality (5) becomes
$\operatorname{cm}(x_{0},x_{1},x_{2},x_{3},x_{4},x_{5})=0$, i.e.,

$\left|\begin{array}[]{ccccccc}0&1&1&1&1&1&1\\\
1&0&d^{2}_{01}&d^{2}_{02}&d^{2}_{03}&d^{2}_{04}&\delta^{2}_{05}\\\
1&d^{2}_{01}&0&d^{2}_{12}&d^{2}_{13}&\delta^{2}_{14}&d^{2}_{15}\\\
1&d^{2}_{02}&d^{2}_{12}&0&\delta^{2}_{23}&d^{2}_{24}&d^{2}_{25}\\\
1&d^{2}_{03}&d^{2}_{13}&\delta^{2}_{23}&0&d^{2}_{34}&d^{2}_{35}\\\
1&d^{2}_{04}&\delta^{2}_{14}&d^{2}_{24}&d^{2}_{34}&0&d^{2}_{45}\\\
1&\delta^{2}_{05}&d^{2}_{15}&d^{2}_{25}&d^{2}_{35}&d^{2}_{45}&0\end{array}\right|=0.$
(10)

The relations (7)–(10) form the first group of necessary conditions for the
affine equivalence of the octahedra $P$ and $P^{\prime}$. They must be
satisfied by the quantities $\delta_{05}$, $\delta_{14}$, and $\delta_{23}$,
which are the lengths of the diagonals of $P$ and which cannot be found
directly from the natural development of $P$.

### The second group of necessary conditions

This group of conditions expresses the fact that $P$ is convex.

Let us start with a detailed explanation of the construction of one of the
conditions of the second group of necessary conditions. To do this, consider
two tetrahedra $T_{3}=\langle x_{0},x_{1},x_{2},x_{3}\rangle$ and
$T_{4}=\langle x_{0},x_{1},x_{2},x_{4}\rangle$ in $\mathbb{R}^{3}$. They share
the common face $\langle x_{0},x_{1},x_{2}\rangle$. In principle, $T_{3}$ and
$T_{4}$ can be located relative to each other in two ways: either so that they
are contained in different closed halfspaces determined by the plane
containing the face $\langle x_{0},x_{1},x_{2}\rangle$, or so that they are
contained in one of these halfspaces. Obviously, $|x_{3}-x_{4}|$ (and hence
the numerical value of $d^{2}_{34}$) in the latter case is strictly less than
in the first one.

In (8), substitute $t$ instead of $d^{2}_{34}$ (the other $d_{ij}$’s and
$\delta_{14}$ are assumed to be fixed numbers borrowed from $P$). As a result,
we get

$\left|\begin{array}[]{cccccc}0&1&1&1&1&1\\\
1&0&d^{2}_{01}&d^{2}_{02}&d^{2}_{03}&d^{2}_{04}\\\
1&d^{2}_{01}&0&d^{2}_{12}&d^{2}_{13}&\delta^{2}_{14}\\\
1&d^{2}_{02}&d^{2}_{12}&0&t&d^{2}_{24}\\\
1&d^{2}_{03}&d^{2}_{13}&t&0&d^{2}_{34}\\\
1&d^{2}_{04}&\delta^{2}_{14}&d^{2}_{24}&d^{2}_{34}&0\end{array}\right|=0.$
(11)

Let us rewrite (11) in the form $At^{2}+Bt+C=0$, where $A$, $B$ and $C$ are
algebraic polynomials in $\delta_{14}$ and all variables $d_{ij}$ involved in
(11). It is easy to see that

$A=-\left|\begin{array}[]{cccc}0&1&1&1\\\ 1&0&d^{2}_{01}&d^{2}_{04}\\\
1&d^{2}_{01}&0&\delta^{2}_{14}\\\
1&d^{2}_{04}&\delta^{2}_{14}&0\end{array}\right|=-\operatorname{cm}(x_{0},x_{1},x_{4}).$

Hence, taking into account the formula (1), we get
$A={2^{2}}{2!}[\operatorname{vol}(x_{0},x_{1},x_{4})]^{2}>0$. Here
$\operatorname{vol}(x_{0},x_{1},x_{4})$ denotes the area of the triangle with
the vertices $x_{0},x_{1},x_{4}$, and the inequality holds true because
$\operatorname{vol}(x_{0},x_{1},x_{4})\neq 0$. The latter follows form the
definition of an octahedron given in Section 2.

Above, we mentioned a geometric argument based on the relative position of
$T_{3}$, $T_{4}$. It implies that the quadratic equation $At^{2}+Bt+C=0$ has
two different positive real roots. Let us denote them by $t_{1}$, $t_{2}$. For
definiteness, assume $0<t_{1}<t_{2}$. Then $t_{1}=d^{2}_{34}$, $B^{2}-4AC>0$,
$B=-A(t_{1}+t_{2})<-2At_{1}$, and $C=At_{1}t_{2}>At_{1}^{2}$. For positive
$t_{1},t_{2}$, each of the last two inequalities is equivalent to
$t_{1}<t_{2}$ and, thus, they are equivalent to each other. Hence, below we
can use only one of them, e.g., $C>At_{1}^{2}$ (or, what is the same,
$C>Ad^{4}_{34}$).

Since $C=At^{2}+Bt+C$ for $t=0$ and $At^{2}+Bt+C$ is given by the formula
(11), we can rewrite the inequality $C>Ad^{4}_{34}$ as

$\left|\begin{array}[]{cccccc}0&1&1&1&1&1\\\
1&0&d^{2}_{01}&d^{2}_{02}&d^{2}_{03}&d^{2}_{04}\\\
1&d^{2}_{01}&0&d^{2}_{12}&d^{2}_{13}&\delta^{2}_{14}\\\
1&d^{2}_{02}&d^{2}_{12}&0&0&d^{2}_{24}\\\
1&d^{2}_{03}&d^{2}_{13}&0&0&d^{2}_{34}\\\
1&d^{2}_{04}&\delta^{2}_{14}&d^{2}_{24}&d^{2}_{34}&0\end{array}\right|>-d^{4}_{34}\left|\begin{array}[]{cccc}0&1&1&1\\\
1&0&d^{2}_{01}&d^{2}_{04}\\\ 1&d^{2}_{01}&0&\delta^{2}_{14}\\\
1&d^{2}_{04}&\delta^{2}_{14}&0\end{array}\right|.$ (12)

The inequality (12) expresses the fact that the points $x_{3}$, $x_{4}$ are
contained in one of the two closed halfspaces determined by the plane
containing the face $\langle x_{0},x_{1},x_{2}\rangle$ of $P$. If we write
(12) using only distances given to us in the natural development of $P$ (i.e.,
using $d_{ij}$’s only) then (12) imposes restrictions on the possible values
of the length of the diagonal $x_{1}x_{4}$ (i.e., on $\delta_{14}$).

The inequality (12) belongs to the second group of necessary conditions for
the affine equivalence of the octahedra $P$ and $P^{\prime}$. The other
inequalities of that group are derived similarly to (12) according to the
following instructions:

$\bullet$ select one of the eight faces of $P$ and denote it by $\langle
x_{i_{0}},x_{i_{1}},x_{i_{2}}\rangle$ (it will play the same role as the face
$\langle x_{0},x_{1},x_{2}\rangle$ played in the above derivation of (12));

$\bullet$ select two of the three faces of $P$ incident to $\langle
x_{i_{0}},x_{i_{1}},x_{i_{2}}\rangle$ and denote them by $\langle
x_{i_{0}},x_{i_{1}},x_{i_{3}}\rangle$ and $\langle
x_{i_{0}},x_{i_{2}},x_{i_{4}}\rangle$ (they will play the same role as the
faces $\langle x_{0},x_{1},x_{3}\rangle$ and $\langle
x_{0},x_{2},x_{4}\rangle$ played in the above derivation of (12));

$\bullet$ apply the arguments described above for tetrahedra $T_{3}=\langle
x_{0},x_{1},x_{2},x_{3}\rangle$ and $T_{4}=\langle
x_{0},x_{1},x_{2},x_{4}\rangle$ to tetrahedra $\langle
x_{i_{0}},x_{i_{1}},x_{i_{2}},x_{i_{3}}\rangle$ and $\langle
x_{i_{0}},x_{i_{1}},x_{i_{2}},x_{i_{4}}\rangle$.

As a result, we get an inequality which is similar to (12). It expresses the
fact that the points $x_{i_{3}}$ and $x_{i_{4}}$ are contained in one of the
two closed halfspaces determined by the plane containing the face $\langle
x_{i_{0}},x_{i_{1}},x_{i_{2}}\rangle$ of $P$. It is easy to understand that
there will be $8\times 3=24$ of such inequalities. Together they guarantee the
convexity of $P$. We call them the second group of necessary conditions for
the affine equivalence of $P$ and $P^{\prime}$.

### The third group of necessary conditions

This group of necessary conditions is constructed from $P^{\prime}$ in the
same way as the first group was constructed from $P$. In other words, the
third group of necessary conditions consists of four relations, each of which
is obtained from (7)–(10) by replacing every $d_{ij}$ by $d^{\prime}_{ij}$ and
every $\delta_{ij}$ by $\delta^{\prime}_{ij}$.

### The fourth group of necessary conditions

This group is constructed from $P^{\prime}$ in the same way as the second
group was constructed from $P$. It consists of 24 inequalities. All together
they guarantee that $P^{\prime}$ is convex.

### The fifth group of necessary conditions

The relations of this group directly expresses the fact that $P$ and
$P^{\prime}$ are affine equivalent. They are constructed as follows.

Select any of the 12 edges of $P$. Denote it by $\langle
x_{i_{0}},x_{i_{1}}\rangle$. By $\langle x_{i_{0}},x_{i_{1}},x_{i_{2}}\rangle$
and $\langle x_{i_{0}},x_{i_{1}},x_{i_{3}}\rangle$ denote the two faces of $P$
incident to the edge $\langle x_{i_{0}},x_{i_{1}}\rangle$. According to our
definition of an octahedron, the tetrahedron $\langle
x_{i_{0}},x_{i_{1}},x_{i_{2}},x_{i_{3}}\rangle$ has nonzero 3-volume,
$\operatorname{vol}\langle x_{i_{0}},x_{i_{1}},x_{i_{2}},x_{i_{3}}\rangle$,
which is related to 3-volume of the tetrahedron $\langle
x^{\prime}_{i_{0}},x^{\prime}_{i_{1}},x^{\prime}_{i_{2}},x^{\prime}_{i_{3}}\rangle$,
$\operatorname{vol}\langle
x^{\prime}_{i_{0}},x^{\prime}_{i_{1}},x^{\prime}_{i_{2}},x^{\prime}_{i_{3}}\rangle$,
by the well-known formula
$\operatorname{vol}(x^{\prime}_{i_{0}},x^{\prime}_{i_{1}},x^{\prime}_{i_{2}},x^{\prime}_{i_{3}})=|\det
A|\operatorname{vol}(x_{i_{0}},x_{i_{1}},x_{i_{2}},x_{i_{3}})$. Squaring the
latter formula and using (1), we get

$\operatorname{cm}(x^{\prime}_{i_{0}},x^{\prime}_{i_{1}},x^{\prime}_{i_{2}},x^{\prime}_{i_{3}})=(\det
A)^{2}\operatorname{cm}(x_{i_{0}},x_{i_{1}},x_{i_{2}},x_{i_{3}})$

or, which is the same,

$\left|\begin{array}[]{ccccc}0&1&1&1&1\\\
1\vphantom{\int\limits^{1}}&0&{d^{\prime}}^{2}_{\\!\\!i_{0}i_{1}}&{d^{\prime}}^{2}_{\\!\\!i_{0}i_{2}}&{d^{\prime}}^{2}_{\\!\\!i_{0}i_{3}}\\\
1\vphantom{\int\limits^{1}}&{d^{\prime}}^{2}_{\\!\\!i_{0}i_{1}}&0&{d^{\prime}}^{2}_{\\!\\!i_{1}i_{2}}&{d^{\prime}}^{2}_{\\!\\!i_{1}i_{3}}\\\
1\vphantom{\int\limits^{1}}&{d^{\prime}}^{2}_{\\!\\!i_{0}i_{2}}&{d^{\prime}}^{2}_{\\!\\!i_{1}i_{2}}&0&{\delta^{\prime}}^{2}_{\\!\\!i_{2}i_{3}}\\\
1\vphantom{\int\limits^{1}}&{d^{\prime}}^{2}_{\\!\\!i_{0}i_{3}}&{d^{\prime}}^{2}_{\\!\\!i_{1}i_{3}}&{\delta^{\prime}}^{2}_{\\!\\!i_{2}i_{3}}&0\end{array}\right|=\alpha\left|\begin{array}[]{ccccc}0&1&1&1&1\\\
1\vphantom{\int\limits^{1}}&0&d^{2}_{i_{0}i_{1}}&d^{2}_{i_{0}i_{2}}&d^{2}_{i_{0}i_{3}}\\\
1\vphantom{\int\limits^{1}}&d^{2}_{i_{0}i_{1}}&0&d^{2}_{i_{1}i_{2}}&d^{2}_{i_{1}i_{3}}\\\
1\vphantom{\int\limits^{1}}&d^{2}_{i_{0}i_{2}}&d^{2}_{i_{1}i_{2}}&0&\delta^{2}_{i_{2}i_{3}}\\\
1\vphantom{\int\limits^{1}}&d^{2}_{i_{0}i_{3}}&d^{2}_{i_{1}i_{3}}&\delta^{2}_{i_{2}i_{3}}&0\end{array}\right|,$
(13)

where $\alpha=(\det A)^{2}>0$. All 12 equations forming the fifth group of
necessary conditions may be obtained from (13) with a suitable choice of
$x_{i_{k}}$, $k=0,1,2,3$.

##  4 The main result

In Section 4, we treat the relations which make up the five groups of
necessary conditions for $P$ and $P^{\prime}$ to be affinely equivalent to
each other as algebraic equations and inequalities in seven unknowns
$\delta_{05}$, $\delta_{14}$, $\delta_{23}$, $\delta^{\prime}_{05}$,
$\delta^{\prime}_{14}$, $\delta^{\prime}_{23}$, and $\alpha$, i.e., with
respect to those variables whose values cannot be directly found from the
natural developments $R$ and $R^{\prime}$ of $P$ and $P^{\prime}$. As we know
from Section 3, the coefficients of those algebraic equations and inequalities
are expressed in terms of the variables whose values can be directly found
from the natural developments $R$ and $R^{\prime}$ of $P$ and $P^{\prime}$,
i.e., through $\\{d_{ij}\\}$, the set of the lengths of the edges $\langle
x_{i},x_{j}\rangle$ of $R$, and $\\{d^{\prime}_{ij}\\}$, the set of the
lengths of the edges $\langle x^{\prime}_{i},x^{\prime}_{j}\rangle$ of
$R^{\prime}$.

The main result of this article is given by the following theorem:

######  Theorem 2.

Let $P$ and $P^{\prime}$ be two convex octahedra in $\mathbb{R}^{3}$, and let
$R$ and $R^{\prime}$ be their natural developments; then the following
statements are equivalent:

_(a)_ $P$ and $P^{\prime}$ are affinely equivalent to each other;

_(b)_ there are seven positive real numbers $\delta_{05}$, $\delta_{14}$,
$\delta_{23}$, $\delta^{\prime}_{05}$, $\delta^{\prime}_{14}$,
$\delta^{\prime}_{23}$, and $\alpha$ such that all five groups of necessary
conditions given in Section 3 are satisfied.

###### Proof.

The implication (a) $\Rightarrow$ (b) has been demonstrated in Section 3. One
should take the lengths of the corresponding diagonals of $P$ as
$\delta_{05}$, $\delta_{14}$, $\delta_{23}$, the lengths of the corresponding
diagonals of $P^{\prime}$ as $\delta^{\prime}_{05}$, $\delta^{\prime}_{14}$,
$\delta^{\prime}_{23}$, and the square of the determinant of the affine
transformation $A$ such that $P^{\prime}=A(P)$ as $\alpha$.

It remains to prove that (b) implies (a). By $\widetilde{x}_{i}$,
$i=0,\dots,5$, denote the vertices of $R$ which are enumerated in accordance
with the special enumeration of the vertices of $P$ introduced in Section 3
and depicted in Fig. 1.

Let us define a semimetric $\widetilde{\rho}$ on the set
$\widetilde{V}=\\{\widetilde{x}_{0},\dots,\widetilde{x}_{5}\\}$ by putting

$\widetilde{\rho}(\widetilde{x}_{i},\widetilde{x}_{j})\stackrel{{\scriptstyle\textrm{def}}}{{=}}\begin{cases}d_{ij},\text{
if }~{}\widetilde{x}_{i},\widetilde{x}_{j}\text{ are connected by an edge in
}R;\\\ \delta_{ij},\text{ otherwise}.\end{cases}$

Here $d_{ij}$ is the length of the edge
$\langle\widetilde{x}_{i},\widetilde{x}_{j}\rangle$ in $R$, and $\delta_{ij}$
is one of the positive real numbers $\delta_{05}$, $\delta_{14}$,
$\delta_{23}$ whose existence is asserted in (b).

It follows from (b) that the relations (7)–(10) are fulfilled. Thus, according
to Theorem 1, $(\widetilde{V},\widetilde{\rho})$ embeds isometrically in
$\mathbb{R}^{3}$. By $f:\widetilde{V}\to\mathbb{R}^{3}$ denote one of such
embeddings.

Let $\Delta$ be one of the eight triangles constituting the natural
development $R$. Suppose $\widetilde{x}_{i}$, $\widetilde{x}_{j}$, and
$\widetilde{x}_{k}$ are the vertices of $\Delta$. Since the second group of
necessary conditions described in Section 3 (and, in particular, the
inequality (11)) is assumed to be fulfilled, the set

$\bigl{(}\underset{m=0}{\overset{5}{\bigcup}}\\{f(\widetilde{x}_{m})\\}\bigr{)}\setminus\bigl{(}\\{f(\widetilde{x}_{i})\\}\cup\\{f(\widetilde{x}_{j})\\}\cup\\{f(\widetilde{x}_{k})\\}\bigr{)},$

is contained in one of the two closed halfspaces determined by the plane
containing the triangle $\langle
f(\widetilde{x}_{i}),f(\widetilde{x}_{j}),f(\widetilde{x}_{k})\rangle$. Hence,
$\langle
f(\widetilde{x}_{i}),f(\widetilde{x}_{j}),f(\widetilde{x}_{k})\rangle$ is a
face of the convex hull of the points $f(\widetilde{x}_{i})$, $i=0,\dots,5$.
This implies that the convex hull of the points $f(\widetilde{x}_{i})$,
$i=0,\dots,5$, is a convex octahedron whose vertices are provided with the
standard numbering as in Fig. 1. Let us denote this octahedron by $X$.

By $\widetilde{x}^{\prime}_{i}$, $i=0,\dots,5$, denote the vertices of
$R^{\prime}$ which are enumerated in such a way that $\widetilde{x}_{i}$ and
$\widetilde{x}^{\prime}_{i}$ correspond to each other according to the
combinatorial equivalence of $R$ and $R^{\prime}$. As before, we define a
semimetric $\widetilde{\rho}^{\prime}$ on the set
$\widetilde{V}^{\prime}=\\{\widetilde{x}^{\prime}_{0},\dots,\widetilde{x}^{\prime}_{5}\\}$
by putting

$\widetilde{\rho}^{\prime}(\widetilde{x}^{\prime}_{i},\widetilde{x}^{\prime}_{j})\stackrel{{\scriptstyle\textrm{def}}}{{=}}\begin{cases}d^{\prime}_{ij},\text{
if }~{}\widetilde{x}^{\prime}_{i},\widetilde{x}^{\prime}_{j}\text{ are
connected by an edge in }R^{\prime};\\\ \delta^{\prime}_{ij},\text{
otherwise}.\end{cases}$

Here $d^{\prime}_{ij}$ is the length of the edge
$\langle\widetilde{x}^{\prime}_{i},\widetilde{x}^{\prime}_{j}\rangle$ in
$R^{\prime}$, and $\delta^{\prime}_{ij}$ is one of the positive real numbers
$\delta^{\prime}_{05}$, $\delta^{\prime}_{14}$, $\delta^{\prime}_{23}$ whose
existence is asserted in (b).

According to Theorem 1 and the third group of necessary conditions,
$(\widetilde{V}^{\prime},\widetilde{\rho}^{\prime})$ embeds isometrically in
$\mathbb{R}^{3}$. By $f^{\prime}:\widetilde{V}^{\prime}\to\mathbb{R}^{3}$
denote one of such embeddings.

The forth group of necessary conditions implies that if
$\widetilde{x}^{\prime}_{i}$, $\widetilde{x}^{\prime}_{j}$, and
$\widetilde{x}^{\prime}_{k}$ are the vertices of any of the eight triangles
constituting the natural development $R^{\prime}$ then $\langle
f^{\prime}(\widetilde{x}^{\prime}_{i}),f^{\prime}(\widetilde{x}^{\prime}_{j}),f^{\prime}(\widetilde{x}^{\prime}_{k})\rangle$
is a face of the convex hull of the points
$f^{\prime}(\widetilde{x}^{\prime}_{i})$, $i=0,\dots,5$. Thus, the convex hull
of the points $f^{\prime}(\widetilde{x}^{\prime}_{i})$, $i=0,\dots,5$, is a
convex octahedron whose vertices are provided with the standard numbering as
in Fig. 1. Let us denote this octahedron by $X^{\prime}$.

Note that both the points $f(\widetilde{x}_{i})$, $i=0,\dots,3$ and the points
$f^{\prime}(\widetilde{x}^{\prime}_{i})$, $i=0,\dots,3$, are the vertices of
nondegenerate tetrahedra. Therefore, there is a unique affine transformation
$A:\mathbb{R}^{3}\to\mathbb{R}^{3}$ such that
$f^{\prime}(\widetilde{x}^{\prime}_{i})=A(f(\widetilde{x}_{i}))$ for all
$i=0,\dots,3$. According to (b), the fifth group of necessary conditions is
satisfied with some $\alpha>0$. Hence, $(\det A)^{2}=\alpha$.

Let us prove two more properties of $A$, namely,
$f^{\prime}(\widetilde{x}^{\prime}_{j})=A(f(\widetilde{x}_{j}))$, $j=4,5$.

Let $k=2\mbox{ or }3$, and $j=4\mbox{ or }5$. By definition, put

$\displaystyle\Delta_{k,j}$ $\displaystyle\,=\,$ $\displaystyle\langle
A(f(\widetilde{x}_{0})),A(f(\widetilde{x}_{1})),A(f(\widetilde{x}_{k})),A(f(\widetilde{x}_{j}))\rangle,$
$\displaystyle\Delta^{\prime}_{k,j}$ $\displaystyle\,=\,$
$\displaystyle\langle
f^{\prime}(\widetilde{x}^{\prime}_{0}),f^{\prime}(\widetilde{x}^{\prime}_{1}),f^{\prime}(\widetilde{x}^{\prime}_{k}),f^{\prime}(\widetilde{x}^{\prime}_{j})\rangle,$
$\displaystyle\delta_{k}$ $\displaystyle\,=\,$ $\displaystyle\langle
A(f(\widetilde{x}_{0})),A(f(\widetilde{x}_{1})),A(f(\widetilde{x}_{k}))\rangle=\langle
f^{\prime}(\widetilde{x}^{\prime}_{0}),f^{\prime}(\widetilde{x}^{\prime}_{1}),f^{\prime}(\widetilde{x}^{\prime}_{k})\rangle.$

Obviously, the tetrahedra $\Delta_{k,j}$ and $\Delta^{\prime}_{k,j}$ share the
common face $\delta_{k}$. Moreover, 3-volumes of $\Delta_{k,j}$ and
$\Delta^{\prime}_{k,j}$ are equal to each other:

$\displaystyle\operatorname{vol}(\Delta_{k,j})$
$\displaystyle=\operatorname{vol}(A(f(\widetilde{x}_{0})),A(f(\widetilde{x}_{1})),A(f(\widetilde{x}_{k})),A(f(\widetilde{x}_{j})))$
$\displaystyle=|\det
A|\operatorname{vol}(f(\widetilde{x}_{0}),f(\widetilde{x}_{1}),f(\widetilde{x}_{k}),f(\widetilde{x}_{j}))$
$\displaystyle=\sqrt{\alpha}\operatorname{vol}(f(\widetilde{x}_{0}),f(\widetilde{x}_{1}),f(\widetilde{x}_{k}),f(\widetilde{x}_{j}))$
$\displaystyle=\operatorname{vol}(f^{\prime}(\widetilde{x}^{\prime}_{0}),f^{\prime}(\widetilde{x}^{\prime}_{1}),f^{\prime}(\widetilde{x}^{\prime}_{k}),f^{\prime}(\widetilde{x}^{\prime}_{j}))$
$\displaystyle=\operatorname{vol}(\Delta^{\prime}_{k,j}).$

Therefore, the heights of $\Delta_{k,j}$ and $\Delta^{\prime}_{k,j}$ which are
treated as pyramids with the common base $\delta_{k}$ are equal to each other.
Denote this common height as $h_{k,j}$.

Recall that the affine hull of $\delta_{k}$ is denoted by
$\mbox{aff\,}\delta_{k}$. For $k=2,3$ and $j=4,5$, by $\tau_{k,j}$ denote the
plane which is parallel to $\mbox{aff\,}\delta_{k}$ and lies at distance
$h_{k,j}$ from $\mbox{aff\,}\delta_{k}$ in the closed halfspace which is
bounded by $\mbox{aff\,}\delta_{k}$ and contains both $\Delta_{k,j}$ and
$\Delta^{\prime}_{k,j}$. It follows from above that both points
$A(f(\widetilde{x}_{j}))$, $f^{\prime}(\widetilde{x}^{\prime}_{j})$ lie on
$\tau_{k,j}$.

Let $i=0\mbox{ or }1$. By $\sigma_{i}$ denote the affine hull of the three
points $A(f(\widetilde{x}_{i}))=f^{\prime}(\widetilde{x}^{\prime}_{i})$,
$A(f(\widetilde{x}_{2}))=f^{\prime}(\widetilde{x}^{\prime}_{2})$, and
$A(f(\widetilde{x}_{3}))=f^{\prime}(\widetilde{x}^{\prime}_{3})$. Arguing as
above, we see that the distance from either of the points
$A(f(\widetilde{x}_{i+4}))$ and $f^{\prime}(\widetilde{x}^{\prime}_{i+4})$ to
$\sigma_{i}$ is the same and these points lie in the same closed halfspace
bounded by $\sigma_{i}$. Hence, $A(f(\widetilde{x}_{i+4}))$ and
$f^{\prime}(\widetilde{x}^{\prime}_{i+4})$ lie on a plane parallel to
$\sigma_{i}$. Denote this plane by $\tau_{i}$.

Observe that both points $A(f(\widetilde{x}_{4}))$ and
$f^{\prime}(\widetilde{x}^{\prime}_{4})$ lie on each of the planes
$\tau_{2,4}$, $\tau_{3,4}$ and $\tau_{0}$, and hence are contained in their
intersection. However, the planes $\tau_{2,4}$, $\tau_{3,4}$ and $\tau_{0}$
are parallel to the planes $\mbox{aff\,}\delta_{2}$, $\mbox{aff\,}\delta_{3}$
and $\sigma_{0}$ respectively, while
$\mbox{aff\,}\delta_{2}\cap\mbox{aff\,}\delta_{3}\cap\sigma_{0}=\\{A(f(\widetilde{x}_{0}))\\}=\\{f^{\prime}(\widetilde{x}^{\prime}_{0})\\}$.
Hence, the intersection $\tau_{2,4}\cap\tau_{3,4}\cap\tau_{0}$ consists of
exactly one point, and this point is
$A(f(\widetilde{x}_{4}))=f^{\prime}(\widetilde{x}^{\prime}_{4})$.

Similarly, we check that
$\tau_{2,5}\cap\tau_{3,5}\cap\tau_{1}=\\{A(f(\widetilde{x}_{5}))\\}=\\{f^{\prime}(\widetilde{x}^{\prime}_{5})\\}$
which yields $A(f(\widetilde{x}_{5}))=f^{\prime}(\widetilde{x}^{\prime}_{5})$.

Thus, assuming (b) to be satisfied, we were able to isometrically embed the
natural developments $R$ and $R^{\prime}$ as convex affinely equivalent
octahedra $X$ and $X^{\prime}$ (since there is an affine transformation
$A:\mathbb{R}^{3}\to\mathbb{R}^{3}$ such that
$f^{\prime}(\widetilde{x}^{\prime}_{j})=A(f(\widetilde{x}_{j}))$ for all
$i=0,\dots,5$). On the other hand, $R$ is the natural development of both $P$
and $X$. Thus, the Cauchy rigidity theorem implies that $P$ and $X$ are
congruent. Similarly, $P^{\prime}$ and $X^{\prime}$ are congruent to each
other. Hence, $P$ and $P^{\prime}$ are affinely equivalent, and (b) implies
(a). ∎

##  5 Concluding remarks

$\bullet$ In Problem 1, it would be more natural to ask about the projective
(rather than affine) equivalence of convex octahedra. The point is that both
the property “to be convex” and the property “to be a polyhedron” are
projective invariant. Unfortunately, some arguments presented in this article
are not applicable to the study of the projective equivalence of octahedra. In
fact, the equations from the fifth group of necessary conditions obtained in
Section 3 are not valid for projective transformations. Here we need
qualitatively new ideas.

$\bullet$ The Cauchy rigidity theorem is the standard of elegance. Against
this background, our Theorem 2 looks especially clumsy. It is desirable to
find a solution to Problem 1 as elegant as the Cauchy rigidity theorem.

##  6 Acknowledgment

The study was carried out within the framework of the state contract of the
Sobolev Institute of Mathematics (project FWNF-2022-0006).

## References

  * [1] Alexandrov, A.D.: Convex polyhedra.  Berlin: Springer (2005)
  * [2] Alexandrov, V.: Recognition of affine-equivalent polyhedra by their natural developments. arXiv:2106.13659v2 [math.mg] (2021)
  * [3] Blumenthal, L.M.: Theory and applications of distance geometry. 2d edn.  New York: Chelsea Publishing Company (1970)
  * [4] Bowers, J.C., Bowers, Ph.L.: A Menger redux: embedding metric spaces isometrically in Euclidean space.  Am. Math. Mon. 124(7), 621–636 (2017).  DOI 10.4169/amer.math.monthly.124.7.621
  * [5] Cauchy, A.L.: Sur les polygones et polyèdres. Second Mémoire.  J. École Polytéchnique 9, 87–98 (1813)
  * [6] Connelly, R.: Rigidity.  In: P.M. Gruber and J.M. Wills (eds.), Handbook of convex geometry. Vol. A, pp. 223–271. Amsterdam: North-Holland (1993)
  * [7] Fuchs, D., Tabachnikov, S.: Mathematical omnibus. Thirty lectures on classic mathematics.  Providence: American Mathematical Society (2007)
  * [8] Rodríguez, L., Rosenberg, H.: Rigidity of certain polyhedra in $\mathbf{R}^{3}$.  Comment. Math. Helv. 75(3), 478–503 (2000).  DOI 10.1007/s000140050137
  * [9] Sabitov, I.Kh.: Algebraic methods for solution of polyhedra.  Russ. Math. Surv. 66(3), 445–505 (2011).  DOI 10.1070/RM2011v066n03ABEH004748