# Learning Translation Quality Evaluation on Low Resource Languages from Large
Language Models

Amirkeivan Mohtashami
Department of Computer Science, EPFL
Lausanne, Switzerland
<EMAIL_ADDRESS>
&Mauro Verzetti & Paul K. Rubenstein
Google
Zurich, Switzerland
{verzetti<EMAIL_ADDRESS>
Work done during an internship at Google.

###### Abstract

Learned metrics such as BLEURT have in recent years become widely employed to
evaluate the quality of machine translation systems. Training such metrics
requires data which can be expensive and difficult to acquire, particularly
for lower-resource languages. We show how knowledge can be distilled from
Large Language Models (LLMs) to improve upon such learned metrics without
requiring human annotators, by creating synthetic datasets which can be mixed
into existing datasets, requiring only a corpus of text in the target
language. We show that the performance of a BLEURT-like model on lower
resource languages can be improved in this way.

##  1 Introduction

A machine translation system is typically evaluated by comparing its output on
a given input sentence with one made by a professional translator. Concretely,
a test set is constructed by taking a corpus of sentences
$\\{a_{1},a_{2},\ldots,a_{n}\\}$ in language A and having a human translate
them to language B $\\{b_{1},b_{2},\ldots,b_{n}\\}$. A machine translation
system $T:A\rightarrow B$ is then evaluated by comparing $T(a_{i})$ and
$b_{i}$ using some metric $m\left(T(a_{i}),b_{i}\right)$.

Until recently, commonly used metrics such as BLEU (Papineni et al., 2002b)
and ROGUE (Lin, 2004) were generally based on number of co-occurring n-grams.
Advantages of such methods include that they are easy to interpret, do not
require learning from data, and have been shown to generally correlate with
human judgement when averaged over a corpus of sentences.

Nonetheless, these approaches fail when sentences are semantically similar but
differ significantly in phrasing. For example, although the sentences _the sky
is clear_ and _there are no clouds above_ have no words in common, their
meaning is similar and a good metric should assign a high score to such pairs
as well. To alleviate this issue, recent works such as BLEURT (Sellam et al.,
2020a) and COMET (Rei et al., 2020) have considered neural-network based
metrics which work based on semantic similarity, but come at the cost of
decreased interpretability and being more computationally intensive to run.

Such metrics can be learned by treating the problem of defining a metric
$m\left(T(a_{i}),b_{i}\right)$ as a regression problem, where a high score
should be returned if $T(a_{i})$ and $b_{i}$ are similar, and a low score if
they are different. This requires an annotated dataset of reference and
candidate sentences, along with scores provided by human annotators. Phrasing
the problem this way, $m$ is not a translation metric per se but rather a
sentence similarity metric which can be used to evaluate translation quality.

Although some public datasets for this purpose exist — notably the WMT
datasets released each year as part of the WMT Machine Translation Evaluation
task (Ma et al., 2019) — it is in general expensive to create such datasets as
they require expensive annotation by skilled humans. As such, available
datasets tend to be either very small or restricted to popular languages, and
therefore the performance of metrics trained with these datasets may
deteriorate on lower resource languages.

Fortunately, recent developments in large language models (LLMs) allow for new
possible solutions for many tasks including this one. Due to being trained on
internet-scale textual datasets, LLMs such as PaLM (Chowdhery et al., 2022)
and GPT-3 (Brown et al., 2020) have knowledge of a large number of languages;
the training sets of both PaLM and GPT-3 include more than 100 languages.
Furthermore, these models have demonstrated a remarkable ability to perform a
wide variety of tasks by selecting prompts with a few (or even zero) examples.

As such, LLMs could be used directly as a metric to evaluate machine
translation systems by asking the model to numerically rate how similar
sentence pairs are: Table 1 shows the correlation between the human scores and
those generated by PaLM using the method described in Section 3.3 which can be
seen is in par with the correlation of the scores generated by the BLEURT-like
baseline. While this would be significantly cheaper and faster than doing the
same with professional humans, cost and time may still be prohibitive since
inference with such models requires significant compute and is slow compared
to standard metrics.

Therefore, we propose another approach to use LLMs to create datasets of
sentence pairs with similarity ratings, which can be used to train smaller
neural metrics such as BLEURT. This means that the cost of querying the LLM is
incurred only once when the dataset is generated, and inference times and cost
of the resulting metric are unchanged. The LLM is used to create datasets for
languages for which such data is scarce or non-existent. In doing so,
knowledge of lower resource languages is distilled from the LLM into the
neural metric.

The contributions of this work are:

  * •

We demonstrate that LLMs can generate synthetic datasets for training sentence
similarity metrics.

  * •

We use this to construct to low resource language training data for
translation quality metrics, and demonstrate an improvement over prior art by
training a BLEURT-like model with this additional data.

| WMT | Internal
---|---|---
| English | Arabic | Urdu | Mongolian
SentenceBLEU | 0.316 | 0.061 | 0.069 | 0.025
BLEURT-like baseline | 0.592(0.001) | 0.210(0.001) | 0.255(0.001) | 0.123(0.001)
PaLM with scoring prompt | 0.541 | 0.163 | 0.262 | 0.105
Table 1: Spearman’s $\rho$ correlation between scores given by human raters
and automatic raters. This demonstrates that given the right prompting, PaLM
can be used to directly score sentence similarity. For the baseline the
average over 5 fine-tuning is reported along with the standard error in
parenthesis. See Section 4 for details about the baseline. SentenceBLEU refers
to the sacrebleu reference implementation (Post, 2018)

##  2 Related Works

In recent years, language models have considerably grown in size leading to a
remarkable increase in their capabilities and performance. GPT-3 (Brown et
al., 2020) and PaLM (Chowdhery et al., 2022) are current examples of the state
of the art with 175B and 540B parameters respectively. These models are
trained in an unsupervised manner over large corpus of text in a variety of
languages, contexts, and settings. In this paper we work with PaLM, though any
LLM could be used. (In particular, further advances in LLMs made made in the
future may translate into further gains in performance.)

Various learned sentence similarity metrics have been proposed in the
literature. BERTscore (Zhang et al., 2019) uses a pre-trained BERT model
(Devlin et al., 2018) to obtain token embeddings and uses the cosine
similarity between the embeddings of two sentences to compute their
similarity. Several recent methods also build upon pre-trained language models
but directly fine-tune them to solve the sentence similarity task as a
regression. For example, BLEURT (Sellam et al., 2020a) fine-tunes BERT first
over synthetic data labeled using traditional metrics such as BLEU (Papineni
et al., 2002a), followed by another fine-tuning over human labeled data. In a
follow-up Sellam et al. (2020b) note that the initial fine-tuning for multi-
lingual scenarios is not as benefical and can be ommitted. COMET (Rei et al.,
2020) is another learned metric for translation quality evaluation that also
takes into account the sentence in the source language. To avoid the need for
human-rated sentence pairs, Thompson & Post (2020) train PRISM in an
unsupervised manner to estimate how much a sentence can be considered a para-
phrased version of another sentence. Similarly, BARTScore (Yuan et al., 2021)
uses a pre-trained language model to estimate the probability of generating
one sentence to continue another sentence and uses the estimated value as a
surrogate for their similarity.

Several recent works also propose methods that are applicable to any learned
metric and can be used to improve their performance. For example Pu et al.
(2021) propose to train a smaller student model using a large amount of
unlabeled data over the specific set of target languages in order to both
improve the speed and the accuracy over those languages.

Existing works have explored the use of LLMs to generate synthetic datasets.
Schick & Schütze (2021) also consider generating data to train sentence
similarity metrics, but only for English and not for application to
translation quality evaluation. Rosenbaum et al. (2022) use LLMs to generate
structured data and a corresponding expression in natural language; while they
do consider a multilingual setting, they are restricted only to popular
languages.

The work presented here is unique in two regards: first, we demonstrate the
capability of LLMs to generate datasets for lower-resource languages; and
second we apply this to learning sentence similarity metrics for application
to translation quality evaluation. Given the near-universal lack of data for
lower-resource languages, our approach to transfer this knowledge from LLMs
can be useful to many tasks, even those already having a large amount of data
in high resource languages.

##  3 Dataset Generation

Our method to generate datasets assumes access only to a corpus of text in the
target language and access to a suitable LLM. In our experiments we use the
multilingual C4 dataset (Raffel et al., 2019) for this purpose, however any
other available sources could instead be used (books, news, government
documents or transcriptions from parliaments as examples).

We generate datasets in four steps. In the first step, we select sentences
from the text corpus. Since the corpus contains paragraphs extracted from the
web, there are many segments which do not constitute a complete sentence such
submission date, number of comments, etc. We apply basic filtering to remove
such segments. In the second step, we query PaLM to generate a second sentence
derived from this, creating a sentence pair. In the third step, we query PaLM
to score each of the sentence pairs generated in the first step according to
the similarity of the sentences. Finally, we apply heuristic filtering to the
generated triples to remove lower quality data.

###  3.1 Selecting Sentences from Corpus

For each record in multilingual C4 dataset (Raffel et al., 2019) we extract
the first sentence from each line in that record. We use an internal sentence
segmentation model to detect sentence boundaries. Subsequently, we filter out
sentences that do not start with an alphabet character or do not finish with a
punctuation mark. The resulting set of sentences are fed through the pipeline
to generate sentence pairs and then score the generated pairs.

###  3.2 Generating Sentence Pairs

To generate the sentence pair, we use PaLM to introduce alterations in a
sentence picked from our text corpus. Since the final model is a metric for
evaluating high-quality translation systems, we focus on generating altered
sentences that could be outputs of such systems. Therefore, we focus the main
instruction given to PaLM on the meaning of the generated alteration. In
particular, we ask PaLM to make alterations to achieve different objectives
such as rephrasing the sentence or changing its meaning as determined by the
prompt. While the instruction does not prohibit adding grammatical mistakes to
the sentence, however PaLM usually generates a grammatically correct sentence.

We also want the generated dataset to be diverse. Thus, to avoid biasing PaLM
to a specific change, e.g. only replacing a word with its synonym, we use a
zero-shot prompt, i.e. we describe the task without providing examples.

Similar to prior works (Brown et al., 2020; Chowdhery et al., 2022), we create
a template with a placeholder for the input sentence and use it to query PaLM.
The output is generated with deterministic zero-temperature decoding, whereby
the most probably next token is iteratively chosen. While testing various
templates, we observed that PaLM sometimes rephrases the given task in terms
similar to competitive programming tasks. We utilize this observation and
directly phrase our task in a competitive programming task template. Since
these tasks usually contain various constraints explained in the description,
using this template allows explaining the objective clearly and in our
experience reduces the chances of the model ignoring the given constraints.

We found it effective to follow chain-of-thought reasoning (Chowdhery et al.,
2022) by requesting an explanation of the changes to be made before producing
the final sentence. In the absence of this, we observed that in many cases the
generated sentences did not satisfy the given constraints. When not asking for
an explanation, we frequently observed that the model output follows the
generated sentence with a description that falsely asserts that the generated
sentence satisfies the given constraints. This description hints that the
model has not ignored the constraints completely but fails to apply them when
generating the sentence when not asked for a concrete explanation in advance.

We tested various prompts and based on the number of successfully generated
pairs, settled on the following prompts to generate our final dataset:

#### Prompts for Preserving the Meaning

We ask PaLM to re-write a sentence without using any of the words in the
original sentence while preserving the meaning. For the explanation, we either
ask it to provide a list of five differences between the input and the output
or ask it to provide three ideas used to change the input into the output.

#### Prompts for Changing Meaning

In this template, we ask PaLM to change a small number of words in the
original sentence to significantly change the meaning but keeping the context.
Similar to the first template, we ask for a list of five differences to force
the model to provide an explanation of the output.

Appendix Figure 1 shows the generic structure of the templates described
above.

###  3.3 Scoring Sentence Pairs

After generating a sentence pair, we query PaLM a second time and use it to
score the similarity of the two sentences in the pair. Note that a score can
also be guessed based on the template that was used for generation. For
example, the template can instruct PaLM to apply only minor changes. However,
we noticed that PaLM can violate such instructions quite often, making the
guessed score to be less reliable. Therefore, we decided to obtain a more
accurate score by querying PaLM again.

Similar to generation, we use a template with two placeholders (one for each
sentence in the pair) to query PaLM.However, we follow a different approach to
obtain the score than greedy decoding.

The template defines a discrete set of allowed scores: the integers
$\\{0,1,\ldots,4\\}$. We end the template such that the score would be the
natural continuation and compute the probability of each score as a
continuation. Next, we take a weighted average of these scores, with weighting
given by these probabilities. This approach allows obtaining the score in a
continuous scale instead of a discrete one and is more reliable as it can take
into account uncertainties of the model between two or more scores.

For this task, we tested several templates including both zero-shot and few-
shot templates, measuring the performance of each template according to the
correlation metrics between the generated scores and human scores over a
subset WMT Shared Metric Task dataset (Ma et al., 2019). The final template
was a few-shot template including an explanation of the task and the grading
scale, followed by examples from several languages. The general structure of
this template can be seen in Appendix Figure 2. The correlation of the
generated scores using this template with human scores over can be seen in
Table 1.

###  3.4 Final Cleaning

To ensure the quality of the generated sentence pairs, we select a subset of
the generated sentence pairs based on some heuristics:

  * •

Sentence Length: We ensure the reference sentence is neither too long or too
short. In particular, we filter out any sentence below 20 characters and above
300 characters.

  * •

Generated/Reference Length Ratio: We ensure that the generated sentence is not
much shorter or much longer than the reference. In particular, we allow pairs
where the ratio of the generated sentence’s length to the original sentence’s
length is at least $0.8$ and at most $2$.

  * •

Minimum Edit Distance: We ensure the two sentences have at least edit distance
$5$ in order to avoid too similar pairs. Note that this condition is relaxed
enough that most pairs with a noticeable change pass it. Thus, this condition
mainly filters cases where there are almost no changes, e.g. when the only
difference is a removed space. While such pairs could also be useful for
training, the ratio of such pairs among the pairs generated by PaLM is quite
high which can harm the quality of the final dataset. To avoid this, we filter
all such pairs from PaLM output. Note that pairs with identical sentences are
already present in the baseline training data as explained in more detail in
Section 4.

##  4 Models, Data and Experiments

#### Model

We train a BLEURT-like model, taking as a starting point the mT5-XL model with
3.7B parameters111Available at https://github.com/google-
research/multilingual-t5 which is pretrained using the multilingual C4 dataset
(Raffel et al., 2019). Similar to Sellam et al. (2020b) we omit the "mid-
training" step proposed in the original BLEURT paper and fine-tune pre-trained
model directly on rated sentence pairs.

#### Baseline Training Data

We obtained the dataset used to train the BLEURT-20222For additional details
see details available at https://github.com/google-research/bleurt model from
the authors which consists of the WMT Metrics Shared Task data with additional
augmentations.

#### Additional LLM-synthesized Training Data

We generate additional datasets in several languages by following the process
described in Section 3. Depending on the experiment, a combination of these
datasets are used as the final training set.

#### Test Data

The lack of data for lower-resource languages extends to the test data as
well, making it hard to evaluate new methods. In this work, we report our
result over a small internal dataset which contains a set of sentences
translated by human experts and different machine translation systems from
English to different target languages, with human-assigned scores of
translation quality obtained by comparing the human reference translations
with the machine translations. The test set for each language contains 1-2k
examples. We note that in this dataset each sentence pair is rated only by a
single rater, which may be a source of label noise. We expect a good sentence
similarity model to correlate with the human ratings and therefore, use the
correlation between our model’s score and the human scores as a metric.

Dataset | Language | BLEU | Baseline | Small | Medium | Large
---|---|---|---|---|---|---
WMT | All | 0.287 | 0.573(0.001) | 0.573(0.000) | 0.573(0.002) | 0.576(0.001)
Internal | Spanish | 0.130 | 0.321(0.001) | 0.316(0.001) | 0.312(0.001) | 0.315(0.002)
Mongolian | 0.025 | 0.142(0.002) | 0.152(0.005) | 0.149(0.004) | 0.156(0.003)
Amharic | 0.035 | 0.273(0.003) | 0.289(0.002) | 0.300(0.004) | 0.300(0.006)
Urdu | 0.069 | 0.261(0.001) | 0.260(0.002) | 0.264(0.002) | 0.279(0.003)
Belarusian | 0.051 | 0.214(0.003) | 0.212(0.005) | 0.201(0.006) | 0.199(0.005)
Punjabi | 0.080 | 0.184(0.004) | 0.171(0.002) | 0.185(0.004) | 0.194(0.006)
Macedonian | 0.159 | 0.265(0.004) | 0.278(0.004) | 0.268(0.004) | 0.266(0.003)
| Persian | 0.150 | 0.418(0.001) | 0.424(0.001) | 0.421(0.002) | 0.425(0.002)
| Arabic | 0.061 | 0.229(0.001) | 0.229(0.001) | 0.225(0.001) | 0.236(0.001)
Table 2: Pearson’s R correlation between the true score and the output of
models obtained from different sets of experiments. For each set, 5 models are
trained and the average is shown with standard error in parenthesis. BLEU
refers to the sacrebleu sentenceBLEU reference implementation (Post, 2018). A
colored cell at the intersection of a language and an experiment indicate that
the experiment includes the language.

#### Experiments

We generate a small dataset for multiple languages. The languages used along
with the size of the generated dataset is listed in Table 4. We perform 3 set
of experiments, adding a subset of the generated datasets to the baseline
dataset. In particular, the training data for the experiment sets are as
follows (refer to Section 4 for more details on the baseline dataset):

  * •

Small: Bsaeline data plus generated data for Spanish, Mongolian, and Amharic.

  * •

Medium: Small experiment data plus generated data for Urdu, Belarusian,
Punjabi, and Macedonian.

  * •

Large: Medium experiment data plus generated data for Arabic, and Persian.

In Appendix C, Table 4 shows which languages were included in each set.

We measure the correlation between the trained model’s score and the true
score. For the generated datasets, the true score is also generated by PaLM.
The results are listed in Table 2. Comparing the correlation between the large
experiments and the baseline, it can be seen that adding the generated data
helps improve the final performance across many languages. Furthermore, it can
be seen that adding data from more languages is important. For example,
performance on Urdu only improves after adding the Urdu dataset. Still, the
performance on some languages such as Amharic keeps improving as more data
from other languages is added.

We note that it is not always the case that adding our data in some language
improves the results for that language. This may be because of a distribution
shift between the test data and our synthesized data. It may also be due to
label noise in our generated data. Nonetheless, we see a broad trend of
improvement as our data is added.

##  5 Future Work & Conclusion

Our results demonstrate the efficacy of using large language models to
generate datasets for lower-resource languages. While we focus our attention
on the task of sentence similarity in this work, the same method can be
applied universally to other tasks facilitating the possibility of supporting
more languages. Some examples are summarization and sentiment detection.

Our method can also be further improved, for example by finding better
prompts. One possible approach is to use automatic prompt engineering which
could allow a more tuned prompt than the manually crafted prompts we used.

Finally, with the advancement of large language models, it is now possible to
use them to perform tasks without the need for a formal definition. We
demonstrate one such example in this work by asking PaLM to generate a
modified version of a given sentence. This ability could make it possible to
automate other tasks that are hard to describe formally.

## Acknowledgement

This work was done while Amirkeivan Mohtashami was doing an internship at
Google. We acknowledge and thank Google for supporting this work as well as
providing the necessary infrastructure to perform the experiments. We would
like to thank Aditya Siddhant, Thibault Sellam, Lotem Golany, Michael
Tschannen, Duc Dung Nguyen and Alex Tudor for support, feedback and helpful
discussions over the course of this work.

## References

  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are Few-Shot learners.  May 2020.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.  PaLM: Scaling language modeling with pathways.  April 2022.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  October 2018.
  * Lin (2004) Chin-Yew Lin.  ROUGE: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pp. 74–81, 2004.
  * Ma et al. (2019) Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham.  Results of the WMT19 metrics shared task: Segment-Level and strong MT systems pose big challenges.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pp. 62–90, Florence, Italy, August 2019\. Association for Computational Linguistics.
  * Papineni et al. (2002a) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  _40th Annual Meeting of the Association for Computational Linguistics_ , pp. 311–318, July 2002a.
  * Papineni et al. (2002b) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  BLEU: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pp. 311–318, 2002b.
  * Post (2018) Matt Post.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pp. 186–191, Belgium, Brussels, October 2018. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/W18-6319.
  * Pu et al. (2021) Amy Pu, Hyung Won Chung, Ankur P Parikh, Sebastian Gehrmann, and Thibault Sellam.  Learning compact metrics for MT.  October 2021.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified Text-to-Text transformer.  October 2019.
  * Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie.  COMET: A neural framework for MT evaluation.  September 2020.
  * Rosenbaum et al. (2022) Andy Rosenbaum, Saleh Soltan, Wael Hamza, Amir Saffari, Marco Damonte, and Isabel Groves.  CLASP: Few-Shot Cross-Lingual data augmentation for semantic parsing.  October 2022.
  * Schick & Schütze (2021) Timo Schick and Hinrich Schütze.  Generating datasets with pretrained language models.  April 2021.
  * Sellam et al. (2020a) Thibault Sellam, Dipanjan Das, and Ankur P Parikh.  BLEURT: Learning robust metrics for text generation.  April 2020a.
  * Sellam et al. (2020b) Thibault Sellam, Amy Pu, Hyung Won Chung, Sebastian Gehrmann, Qijun Tan, Markus Freitag, Dipanjan Das, and Ankur P Parikh.  Learning to evaluate translation beyond english: Bleurt submissions to the wmt metrics 2020 shared task.  _arXiv preprint arXiv:2010.04297_ , 2020b.
  * Thompson & Post (2020) Brian Thompson and Matt Post.  Automatic machine translation evaluation in many languages via zero-shot paraphrasing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 90–121, 2020.
  * Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu.  Bartscore: Evaluating generated text as text generation.  _Advances in Neural Information Processing Systems_ , 34:27263–27277, 2021.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.  BERTScore: Evaluating text generation with BERT.  April 2019.

##  Appendix A Other Correlation Metrics

There are various metrics to measure correlation of two sequences. We have
already reported the correlation of the trained models with the true score
using Pearson’s R metric. Here, we additionally report the correlation as
measured by Spearman’s $\rho$ and Kendall’s $\tau$ for compeleteness.

Dataset | Language | Baseline | Small | Medium | Large
---|---|---|---|---|---
WMT | All | 0.592(0.001) | 0.591(0.001) | 0.590(0.002) | 0.593
Internal | Spanish | 0.266(0.001) | 0.265(0.001) | 0.260(0.001) | 0.265
Mongolian | 0.123(0.002) | 0.137(0.003) | 0.134(0.002) | 0.140
Amharic | 0.287(0.003) | 0.295(0.002) | 0.303(0.004) | 0.295
Urdu | 0.255(0.001) | 0.249(0.002) | 0.245(0.003) | 0.260
Belarusian | 0.174(0.004) | 0.171(0.006) | 0.151(0.005) | 0.152
Punjabi | 0.116(0.003) | 0.121(0.002) | 0.126(0.005) | 0.150
Macedonian | 0.300(0.002) | 0.304(0.002) | 0.297(0.002) | 0.302
| Persian | 0.415(0.001) | 0.417(0.002) | 0.414(0.002) | 0.416
| Arabic | 0.210(0.001) | 0.210(0.002) | 0.208(0.001) | 0.213
Table 3: Spearman’s $\rho$ correlation between the true score and the output
of models obtained from different sets of experiments. For each set, 5 models
are trained and the average is shown with standard error in parenthesis.

##  Appendix B Structure of Prompt Templates

⬇

Abracadabra is the art of re-writing a sentence without using any of the words
in the original sentence while preserving the meaning. For example a beautiful
Abracadabra for "Hello" can be "Hi". A weak Abracadabra for "Hello" is "Bye".
Write a program to generate an Abracadabra for a given sentence.

Input

The input contains contains a single line which contains a sentence in
<LANGUAGE>.

Output

Output a single line containing the <LANGUAGE> Abracadabra for the given
sentence in <LANGUAGE>.

Sample Sections:

A) Sample Input

B) Five Differences Between the Sample Input Sentence and the Output Sentence

C) Sample Output

D) Additional Explanation

A) Sample Input

<SENTENCE>

B) Five Differences Between the Sample Input Sentence and the Output Sentence

Figure 1: Generic structure of the template for the prompt used to generate a
modified sentence with the same meaning. Similar template is used with a
different definition to obtain a modified sentence with a different meaning.
The term <SENTENCE> and <LANGUAGE> are replaced by the sentence to be modified
and its language respectively.

⬇

Task: Consider the pair of sentences and rate them on a scale of 0 to 4 based
on how similar they are, where 0 is least similar and 4 is most similar. The
sentences could be in any language.

An explanation of the ratings:

- 0: the hypothesis is incoherent, contains serious grammatical errors, words in different languages, or has meaning completely different from the reference.

- 1: the hypothesis has some relation to the reference, but has many errors.

- 2: the hypothesis has similar meaning to the reference, but contains grammatical errors.

- 3: the hypothesis has almost the same meaning as the reference, and is mostly grammatically correct.

- 4: the hypothesis has the same or very similar meaning as the reference, and is grammatically correct.

Reference: Xin chào tên bạn là gì?

Hypothesis: Xin chào tên bạn là gì?

Rating: 4

Reference: The dataset was too large to load into memory.

Hypothesis: All of the information together was impossible to load into
memory.

Rating: 2

<ADDITIONAL_EXAMPLES>

Reference: <SENTENCE_1>

Hypothesis: <SENTENCE_2>

Rating:

Figure 2: Generic structure of the template for the prompt used to score a
sentence pair. In the zero-shot setting only the explanation and no examples
are provided, though in practice we found providing more examples in a variety
of languages gave the best results.

##  Appendix C Generated Training Data Details

Dataset | Size ($\times$1000) | Baseline | Small | Medium | Large
---|---|---|---|---|---
Baseline | 595 | ✓ | ✓ | ✓ | ✓
Spanish | 15 |  | ✓ | ✓ | ✓
Mongolian | 4 |  | ✓ | ✓ | ✓
Amharic | 4 |  | ✓ | ✓ | ✓
Urdu | 9 |  |  | ✓ | ✓
Belarusian | 7 |  |  | ✓ | ✓
Punjabi | 2 |  |  | ✓ | ✓
Macedonian | 9 |  |  | ✓ | ✓
Arabic | 4 |  |  |  | ✓
Persian | 20 |  |  |  | ✓
Table 4: List of various datasets used for each set of experiments. The
dataset sizes are rounded down to the nearest thousand.


# Photonic Generation of Radar Signals with 30 GHz Bandwidth
and Ultra-High Time-Frequency Linearity

Ziqian Zhang  Yang Liu<EMAIL_ADDRESS>Benjamin J. Eggleton  Institute
of Photonics and Optical Science (IPOS), School of Physics, The University of
Sydney, NSW 2006, Australia  The University of Sydney Nano Institute (Sydney
Nano), The University of Sydney, NSW 2006, Australia

###### Abstract

Photonic generation of radio-frequency signals has shown significant
advantages over the electronic counterparts, allowing the high precision
generation of radio-frequency carriers up to the terahertz-wave region with
flexible bandwidth for radar applications. Great progress has been made in
photonics-based radio-frequency waveform generation. However, the approaches
that rely on sophisticated benchtop digital microwave components, such as
synthesizers and digital-to-analog converters have limited achievable
bandwidth and thus resolution for radar detections. Methods based on voltage-
controlled analog oscillators exhibit high time-frequency non-linearity,
causing degraded sensing precision. Here, we demonstrate, for the first time,
a photonic stepped-frequency (SF) waveform generation scheme enabled by MHz
electronics with a tunable bandwidth exceeding 30 GHz and intrinsic time-
frequency linearity. The ultra-wideband radio-frequency signal generation is
enabled by using a polarization-stabilized optical cavity to suppress intra-
cavity polarization-dependent instability; meanwhile, the signal’s high-
linearity is achieved via consecutive MHz acousto-optic frequency-shifting
modulation without the necessity of using electro-optic modulators that have
bias-drifting issues. We systematically evaluate the system’s signal quality
and imaging performance in comparison with conventional photonic radar schemes
that use high-speed digital electronics, confirming its feasibility and
excellent performance for high-resolution radar applications.

††preprint: APS/123-QED

Introduction.—Radar sensing has been progressing into the millimeter-wave
(MMW, 30-300 GHz) and terahertz-wave (THz-wave, 0.1-10 THz) regions to operate
with ultrawide bandwidth for growing demands of high spatial resolution
imaging in real-world applications, such as non-destructive testing,
automotive driving assistance, industrial quality inspection, and non-invasive
medical imaging [1, 2, 3]. However, the development of wideband radar has
posed significant challenges to conventional electronic technologies,
especially in the synthesis of ultra-broadband signals. In particular, the
bandwidth of direct digital synthesizers and digital-to-analog converters (two
commonly used and essential components for signal generation) are constrained
to a few gigahertz by the limited clock speed [4]. Moreover, these devices
have shown significant degradation in both efficiency and noise level when
approaching higher frequencies via multi-stage frequency up-conversion [5].
Additionally, multi-stage frequency multiplexing and spectrum stitching for
bandwidth broadening introduce noise and spectrum spur induced by device
nonlinearity and interference [6], which compromises overall sensing accuracy
and performance.

Photonics-assisted radar shows the potentials to overcome the drawbacks in the
electronic counterparts mentioned above, especially in bandwidth broadening
[7], low-noise up-conversion [8], and simple down-conversion and demodulation
[9]. However, existing photonic approaches for radar signal generation still
rely on bulky, high-speed benchtop electronics, or lossy and elaborately
biased electro-optic modulators (EOMs), limiting the achievable bandwidth,
long-term operating stability, and ultimately the practicality [10, 7].
Alternative approaches using dispersion-based time-stretch [11] and frequency-
sweeping light sources [12, 13] have also shown promising bandwidth capacity.
However, these techniques remain challenging to meet simultaneously the high
frequency-time linearity for accurate ranging without using pre-distorted
control signals for linearity compensation and the wide bandwidth for a fine
spatial resolution to cope with the real-world, resolution-demanding
applications.

In this paper, we demonstrate, for the first time, an MHz electronics-enabled
photonic synthesizing of stepped-frequency (SF) signal with tunable bandwidth
exceeding 30 GHz and an inherent high frequency-time linearity. Because of
using a polarization-maintaining optical cavity against the ambient
environment perturbation and the stable acousto-optic frequency-shifting
modulation without the bias-drifting issues from EOMs, the demonstrated system
reaches a signal-to-noise ratio (SNR) of above 34 dB in signal generation
comparable to those generated by high-end benchtop electronics.
Simultaneously, the constant frequency shift and cavity round-trip time ensure
an inherent high frequency-time linearity with a maximum deviation below 2.5
MHz throughout the bandwidth for accurate detection. We experimentally compare
a radar system using the demonstrated scheme with a conventional photonic
radar that relies on a high-speed waveform generator, clearly showing its
comparable performance in radar imaging. With the achieved bandwidth exceeding
30 GHz, this demonstrated approach provides a viable basis for ultra-wideband
microwave waveform synthesis for future ultra-high-resolution MMW and THz-wave
radar systems.

Figure 1: (a) Schematic of the demonstrated SF radar using optical frequency-
shifting modulation. (b) Principle comparisons of radar systems using linear-
frequency modulated (LFM or chirp) and SF signals. RF, radio-frequency; OS,
optical switch; OBPF, optical bandpass filter; EDFA, erbium-doped fiber
amplifier; AOM, acousto-optic modulator; FS, optical frequency shifter; OC,
optical coupler; PD, photodetector; DSP, digital signal processing; SF,
stepped-frequency; $\tau$, the time delay of the reflected detecting signal.

Experiments and Results.—The demonstrated SF waveforms are generated through
recirculating an optical rectangular pulsed signal originated from chopping a
continuous wave (CW) laser in a frequency-shifting loop (FSL) [14], diagrammed
in Fig. 1(a). An acousto-optic modulator (AOM) precisely shifts the pulse
frequency for each round-trip, offering inherent ultra-high frequency-time
linearity and ultra-fast frequency shifting. Moreover, the AOM shifted signal
avoids the generation of spectrum harmonic spurs, which is an outstanding
advantage over the approach using EOMs for single-sideband modulation (SSB)
that suffers from bias-drifting and parasitic harmonics [10, 15]. An optical
bandpass filter (OBPF) is used to determine and tune the bandwidth of the
optical and thus the RF SF signals to satisfy diverse resolution requirements
(range resolution is defined as $c/2B$, where $B$ is the synthesized bandwidth
of the signal). Meanwhile, an erbium-doped fiber amplifier (EDFA) compensates
for the optical modulation and propagation energy losses. It is worth
mentioning that the pulse dwelling time set by an optical switch (OS) should
be less than the round-trip time of the FSL to avoid inter-pulse cross-talk. A
simple optical-RF up-conversion is realized by mixing the optical SF signal
with a frequency-shifted CW laser signal in a photodetector (PD). Finally, a
coherent optical receiver will demodulate the received signal for ranging and
imaging [16].

Fig. 1(b) shows the principle of the SF signal, in comparison to linear
frequency modulated (LFM) signals, which are both widely used as pulse-
compression waveforms for radar sensing owing to their benefits of higher mean
powers while sustaining superior resolution [17]. The operation principles of
the SF signal and the LFM signal have fundamental but subtle differences;
intuitively, the SF signal is a discretely sampled version of the LFM signal,
as shown in Fig. 1(b). In radar applications, the SF radar encodes the time
delay ($\Delta\tau$) of the reflected signal as phase differences with respect
to the reference signals ($\omega_{SF}\propto 2\pi n\Delta f\Delta\tau$, where
$\Delta f$ is the incremental frequency, and $n=1,...,N,N$ is the total number
of frequency steps). The LFM radar forms a new oscillating frequency
proportional to the round-trip time $\Delta\tau$ ($\omega_{LFM}\propto 2\pi
k\Delta\tau t$, where $k$ is the chirp rate [18]), as depicted in Fig. 1(b).
Advantageously, SF radars require a much lower digitizing speed (down to only
one sample per step-time) than the LFM radar receivers bounded by the Nyquist
sampling theorem. This characteristic allows the photonic SF radars to
generate much less data while sustaining the same resolution, enabling low-
consumption edge computing for resolution-demanding applications such as hand
gesture recognition and object identification [19]. Moreover, the SF waveforms
have shown other advantages over the LFM signals. For instance, it possesses
an increased dynamic range due to each frequency step’s narrow instantaneous
noise bandwidth for signal processing while preserving the range resolution
and the overall bandwidth [3].

Figure 2: (a) Optical spectra showing the laser frequency and several
passbands of the OBPF from 3 to 18 GHz. (b) The baseband, RF-domain spectrum
of the SF signal with the measuring apparatus’s noise floor (NF). (c) A time-
domain plot of the SF signals with 250 steps, 100 MHz frequency shift, and a
total synthesized bandwidth of 25 GHz. (d) The time-frequency plot of the 25
GHz bandwidth SF signal using a 25 ns sliding time window. (e) SF linearity
analysis based on extracting the peak values of the time-frequency data with
one sample each frequency step and a constant sampling time interval. A first-
order polynomial fitting is applied to those data for calculating the
deviation.

Fig. 2 shows a typical waveform and spectrum of the generated SFCW signal to
investigate the quality and linearity. One tap of the CW laser mixes the
optical SF signal in a PD to generate the SF signal in the RF baseband. Fig.
2(b) plots the spectrum of the SF signal in the RF-domain as well as the noise
floor (input disconnected) of the measuring apparatus (Agilent E4448A). It
confirms the 25 GHz SF bandwidth, tantamount to a theoretical range resolution
of 6 mm. The temporal waves and the corresponding frequency-time relations are
presented in Fig. 2(c) and (d), respectively. These plots demonstrate that
each frequency step only contains a single oscillating frequency and is
progressively shifted after a specific dwell time (i.e., the round-trip time
of the FSL). The time-domain envelope roll-off at the higher frequency side is
mainly contributed by the roll-off of the OBPF, which is confirmed by the
spectral measurements in both the optical (Fig. 2(a)) and RF domain (Fig.
2(b)). A further linearity analysis is carried out by comparing the difference
between the experimental results (extracting the frequency of the time-domain
signal in each frequency step) and the corresponding values of the first-order
polynomial fitting, which is shown as the deviation in Fig. 2(e). The
linearity analysis reveals a ladder-like frequency-time feature with a maximum
deviation below 2.5 MHz throughout the 25 GHz bandwidth. The deviation is
overestimated and is limited by the resolution of the digital processing
within a finite time window. This feature ensures the high linearity of the SF
format, which is hard to achieve using laser sweeping or voltage-controlled
oscillators without an active feedback control loop or pre-distorted RF
signals generated from benchtop electronic units.

Figure 3: Bandwidth and signal-to-noise ratio (SNR) comparison between the
systems using single-mode fibers (SMFs) and polarization-maintaining fibers
(PMFs). (a) Time-frequency plot of the signal based on un-optimized FSL using
SMFs with a bandwidth of 12 GHz. (b) An insight of the 12 GHz signal. (c)
Time-frequency plot of the signal based on an optimized FSL using PMFs with a
bandwidth exceeding 30 GHz. (d) An insight of the 30 GHz signal. (e) SNR plots
of the SMF-based system at two frequency instances, i.e., 10-time
recirculation (blue) and 60-time recirculation (red) with SNRs around 17.75
dB. (f) SNR plots of the PMF-based system at the same frequency instances
showing an SNR of 34.10 dB at 60-time recirculation. 200-time recirculation is
plotted (yellow) with an SNR of 31.90 dB. (g) SNRs of the SF signal generated
using an AWG at the same frequency instances.

Bandwidth and frequency shift tunability (changing $\Delta f$) are prominent
metrics when deploying SF signals for various applications as two key sensing
parameters are decided accordingly, i.e., the range resolution, $c/2B$ and
ambiguity, $c/(2\times\Delta f)$. In principle, the demonstrated system can
achieve arbitrary bandwidth tuning by changing the passband and central
frequency of the OBPF, thereby enabling range resolutions down to mm level.
However, in practice, such broadband synthesizing is challenging for single-
mode fibers (SMFs), especially when polarization will be further scrambled in
the optical cavity, deteriorating the phase stability, signal coherence, and
ultimately the sensing performance. As shown in Fig. 3(a), the time-frequency
plot of the SF signal generated through an un-optimized FSL using SMF is
challenging to achieve a bandwidth of 12 GHz. Fig. 3(b) provides an insight
into the time-frequency plot, revealing the amplitude fluctuations across
different frequency steps, caused by the polarization and gain instability.

In contrast, the signal bandwidth and stability can be significantly increased
using polarization-maintaining fibers (PMFs) and components that can minimize
the polarization scrambling, and thus help achieve the ultra broad bandwidth
synthesizing over 30 GHz. Fig. 3(c) shows an $>$30 GHz bandwidth signal
synthesizing using PMFs against polarization scrambling caused by
polarization-dependent reflections originating from optical couplers and the
ambient environment, such as temperature and vibration. Compared to Fig. 3(b),
the optimized system also reduces the background noise by using an OS with a
high on-off extinction to suppress the seed pulse tails recirculating in the
FSL. As a result, Fig. 3(d) proved that the improved system performs a more
stable pulse-circulation with much less interference from inter-pulse
overlapping and noise accumulation.

To provide more insights into the signal quality, we compare the SNR of the
signal generated using different schemes. Fourier analysis using 25 ns, time-
domain signal clips between SMF-based system and PMF-based system are shown in
Fig. 3(e) and Fig. 3(f), respectively. The results are sampled at two
instances, i.e., 10-time (blue) and 60-time (red) recirculation, revealing a
more than 14 dB SNR improvement with an SNR of 34. In the meantime, the result
of 200-time recirculation in the optimized system exhibited in Fig. 3(f)
(yellow) also proves that the SNR does not obviously degrade as the pulse
recirculation time increases, indicating a low stability penalty from the FSL
noise accumulations. Moreover, we compared the SF signal generated between the
PM-based FSL and an arbitrary waveform generator (AWG, Keysight M8195A
65GSa/s). Signal quality analysis in Fig. 3(g) shows that the demonstrated
system sustains an SNR above 34 dB after 60-time recirculation in the PMF-
based FSL compared with 39 dB SNR from the high-end electronics. The results
also proved that the demonstrated system could replace high-speed, noisy
electronic synthesizers to generate ultra-broadband signals directly at the
carrier band for accurate MMW and THz-wave sensing.

Figure 4: (a) Schematic of a photonic radar system driven by a high-speed
signal generator, which is used to compare the imaging performance with the
demonstrated radar system. (b) The target for radar imaging with three-
cylinder objects mounted on a rotating platform. (c) The signals used for the
imaging performance comparisons. (d) 2D imaging results. EOM, electro-optic
modulator; CR, coherent receiver.

In order to demonstrate the competitive performance and feasibility, we
compare radar imaging performance based on the SF signal generators with those
using high-speed electronic AWGs as shown in Fig. 4. 2D imaging experiments
are conducted based on the inverse synthetic aperture radar (ISAR) technique.
Comparison experiments that use ultra-fast electronics (Keysight M8195A,
65GSa/s, $\sim$25 GHz analog bandwidth) for generating both SF and LFM signals
are carried out based on the schematic shown in Fig. 4(a) while keeping the
same bandwidth (5.76 GHz), step numbers (72 steps), repetition rate (25 us),
and RF radiation power ($\sim$ 10 dBm), as illustrated in Fig. 4(c). It should
be noted that the signal bandwidth in the demonstrations were chosen due to
the bandwidth availability of the RF antenna, without losing comparison
generality. Wideband-RF signals generated by the AWG modulate a CW laser
carrier through a single-sideband modulation before beating with a separate
optical carrier for optical-RF up-conversion. As shown in Fig. 4(b), three-
cylinder objects mounted on a rotating platform are used as the target. ISAR
imaging results in Fig. 4(d) are chosen at two particular instances for an
adequate performance illustration and comparison, proving that the
demonstrated system is reliable for imaging moving objects. Notably, the
demonstrated system successfully reconstructed ISAR images of the objects
without distinguishable differences from those based on the AWG. It should be
noted that a minor frequency step ($\Delta f$) can be achieved by cascading
two AOMs with the opposite frequency shift (e.g., a frequency shift from -10
to +10 MHz has been demonstrated in [20]) to achieve an unambiguous range over
100 meters, which is comparable to the 120 meters unambiguous range window
(1.25 MHz frequency shift) from the CARABAS system - a very early airborne
synthetic aperture radar that employed SF signals - mounted on an aircraft
[21].

Conclusions.—In conclusion, we demonstrated a photonic SF waveform generation
with a tunable bandwidth of $>$30 GHz and intrinsic high-linearity for high-
resolution and accurate radar detections. By stabilizing polarization in the
optical FSL loop, we significantly improve the SNR of the SF signals from 17
dB to 34 dB with a low noise accumulation penalty, allowing imaging
performance comparable to those using high-speed apparatus. The demonstrated
system enables high-resolution radar sensing and represents an attractive
combination of wideband signal synthesizing, high SNR, and reduced hardware
requirements. This work serves as a pilot study and experimental basis for
further chip-based radar integration using on-chip devices [22], opening the
door to future ultra-high-resolution, miniaturized, and mobile millimeter-wave
devices with prime performance and flexibility.

Funding: This work was funded by Australian Research Council Discovery Project
(DP200101893).

Data Availability Statement: The data that support the findings of this study
are available from the corresponding author upon reasonable request.

Author Contributions: Y.L. and B.J.E. conceived the project; Z.Z. and Y.L.
designed the system structure; Z.Z. conducted the experiments; Z.Z. and Y.L.
performed data analysis. Z.Z. wrote the manuscript with contribution from Y.L.
and B.J.E.; B.J.E. and Y.L. supervised the project.

## References

  * Duling and Zimdars [2009] I. Duling and D. Zimdars, Terahertz imaging: Revealing hidden defects, Nature Photonics 3, 630 (2009).
  * Gowen  _et al._ [2012] A. A. Gowen, C. O’Sullivan, and C. P. O’Donnell, Terahertz time domain spectroscopy and imaging: Emerging techniques for food process monitoring and quality control, Trends in Food Science and Technology 25, 40 (2012).
  * Caruso  _et al._ [2015] M. Caruso, M. Bassi, A. Bevilacqua, and A. Neviani, A 2-16 GHz 65 nm CMOS stepped-frequency radar transmitter with harmonic rejection for high-resolution medical imaging applications, IEEE Transactions on Circuits and Systems I: Regular Papers 62, 413 (2015).
  * Richards  _et al._ [2010] M. A. Richards, J. Scheer, W. A. Holm, and W. L. Melvin,  _Principles of modern radar_ (Citeseer, 2010).
  * Tonda-Goldstein  _et al._ [2006] S. Tonda-Goldstein, D. Dolfi, A. Monsterleet, S. Formont, J. Chazelas, and J. P. Huignard, Optical signal processing in radar systems, IEEE Transactions on Microwave Theory and Techniques 54, 847 (2006).
  * Serafino  _et al._ [2020] G. Serafino, S. Maresca, C. Porzi, F. Scotti, P. Ghelfi, and A. Bogoni, Microwave Photonics for Remote Sensing: From Basic Concepts to High-Level Functionalities, Journal of Lightwave Technology 38, 5339 (2020).
  * Pan and Zhang [2020] S. Pan and Y. Zhang, Microwave Photonic Radars, Journal of Lightwave Technology 38, 5450 (2020).
  * Kittlaus  _et al._ [2021] E. A. Kittlaus, D. Eliyahu, S. Ganji, S. Williams, A. B. Matsko, K. B. Cooper, and S. Forouhar, A low-noise photonic heterodyne synthesizer and its application to millimeter-wave radar, Nature Communications 12, 1 (2021).
  * Serafino  _et al._ [2019] G. Serafino, F. Scotti, L. Lembo, B. Hussain, C. Porzi, A. Malacarne, S. Maresca, D. Onori, P. Ghelfi, and A. Bogoni, Toward a new generation of radar systems based on microwave photonic technologies, Journal of Lightwave Technology 37, 643 (2019).
  * Fu  _et al._ [2013] Y. Fu, X. Zhang, B. Hraimel, T. Liu, and D. Shen, Mach-Zehnder: A review of bias control techniques for mach-zehnder modulators in photonic analog links, IEEE Microwave Magazine 14, 102 (2013).
  * Zhang and Yao [2014] J. Zhang and J. Yao, Time-stretched sampling of a fast microwave waveform based on the repetitive use of a linearly chirped fiber Bragg grating in a dispersive loop, Optica 1, 64 (2014).
  * Zhou  _et al._ [2016] P. Zhou, F. Zhang, Q. Guo, and S. Pan, Linearly chirped microwave waveform generation with large time-bandwidth product by optically injected semiconductor laser, Optics Express 24, 18460 (2016).
  * Hao  _et al._ [2018] T. Hao, Q. Cen, Y. Dai, J. Tang, W. Li, J. Yao, N. Zhu, and M. Li, Breaking the limitation of mode building time in an optoelectronic oscillator, Nature Communications 9, 10.1038/s41467-018-04240-6 (2018).
  * Zhang  _et al._ [2020] Z. Zhang, Y. Liu, M. Burla, and B. J. Eggleton, 5.6-GHz-Bandwidth Photonic Stepped-Frequency Radar using MHz-Level Frequency-Shifting Modulation, in  _2020 Conference on Lasers and Electro-Optics (CLEO)_ (2020) pp. 1–2.
  * Zhang  _et al._ [2021] Y. Zhang, C. Liu, Y. Zhang, K. Shao, C. Ma, L. Li, L. Sun, S. Li, and S. Pan, Multi-Functional Radar Waveform Generation Based on Optical Frequency-Time Stitching Method, Journal of Lightwave Technology 39, 458 (2021).
  * Kikuchi [2016] K. Kikuchi, Fundamentals of coherent optical fiber communications, Journal of Lightwave Technology 34, 157 (2016).
  * Ozdemir [2012] C. Ozdemir,  _Inverse Synthetic Aperture Radar Imaging with MATLAB Algorithms_ (John Wiley & Sons, Incorporated, Hoboken, UNITED STATES, 2012) pp. 57–60.
  * Levanon [2002] N. Levanon, Stepped-frequency pulse-train radar signal, IEE Proceedings: Radar, Sonar and Navigation , 297 (2002).
  * Jia  _et al._ [2021] Y. Jia, Y. Guo, G. Wang, R. Song, G. Cui, and X. Zhong, Multi-frequency and multi-domain human activity recognition based on SFCW radar using deep learning, Neurocomputing 444, 274 (2021).
  * Billault  _et al._ [2021] V. Billault, G. Arpison, V. Crozatier, V. Kemlin, L. Morvan, D. Dolfi, and H. G. De Chatellus, Coherent Optical Fiber Sensing Based on a Frequency Shifting Loop, Journal of Lightwave Technology 39, 4118 (2021).
  * Gustavsson  _et al._ [1990] A. Gustavsson, P. Frklind, H. Hellsten, T. Jonsson, B. Larsson, and G. Stenstrom, The Airborne VHF SAR System CARABAS, IEEE International Geoscience and Remote Sensing Symposium , 558 (1990).
  * Shao  _et al._ [2020] L. Shao, N. Sinclair, J. Leatham, Y. Hu, M. Yu, T. Turpin, D. Crowe, and M. Lončar, Integrated microwave acousto-optic frequency shifter on thin-film lithium niobate, Optics Express 28, 23728 (2020).