# What Makes Math Word Problems Challenging for LLMs?

KV Aditya Srivatsa
MBZUAI
Abu Dhabi, UAE
<EMAIL_ADDRESS>
Ekaterina Kochmar
MBZUAI
Abu Dhabi, UAE
<EMAIL_ADDRESS>

###### Abstract

This paper investigates the question of what makes math word problems (MWPs)
in English challenging for large language models (LLMs). We conduct an in-
depth analysis of the key linguistic and mathematical characteristics of MWPs.
In addition, we train feature-based classifiers to better understand the
impact of each feature on the overall difficulty of MWPs for prominent LLMs
and investigate whether this helps predict how well LLMs fare against specific
categories of MWPs.111Our code, data, and analysis are publicly available at
github.com/kvadityasrivatsa/analyzing-llms-for-mwps

What Makes Math Word Problems Challenging for LLMs?

KV Aditya Srivatsa MBZUAI Abu Dhabi, UAE<EMAIL_ADDRESS>Ekaterina Kochmar MBZUAI Abu Dhabi, UAE<EMAIL_ADDRESS>

##  1 Introduction

In recent years, large language models (LLMs) have not only demonstrated huge
potential across a range of core NLP tasks (Zhao et al., 2023; Brown et al.,
2020; Radford et al., 2019, inter alia), but also exhibited a number of
emergent abilities, such as an ability to solve mathematical puzzles Wei et
al. (2022). Math word problems (MWPs) have been proposed as a challenging
testbed for LLMs, as they test not only the ability of the models to deal with
purely mathematical expressions, but also their reasoning and natural language
understanding abilities (Wang and Lu, 2023; Cobbe et al., 2021; Patel et al.,
2021; Miao et al., 2020, inter alia). Experiments show that even quite
powerful LLMs are still challenged by MWPs Cobbe et al. (2021). At the same
time, most previous work has either focused on evaluation of LLMs’ performance
on MWPs or on changes in their behavior in response to progressive-hint
prompting, prompt paraphrasing or similar approaches Norberg et al. (2023);
Raiyan et al. (2023); Zheng et al. (2023); Zhu et al. (2023), while an in-
depth analysis of what exactly makes math problems challenging for LLMs is
lacking. We aim to address this knowledge gap.

Figure 1: A response from Llama2-70B to a lengthy math problem that involves
NLU challenges.

A recent study by Almoubayyed et al. (2023) demonstrates a strong connection
between reading skills and math outcomes in students. We hypothesize that
LLMs’ ability to solve MWPs correctly may similarly rely on: (1) the
linguistic complexity of the questions; (2) the conceptual complexity of the
tasks (e.g., the number of steps and types of math operations involved); and
(3) the amount of real-world knowledge required to solve the tasks. Supporting
this intuition, our preliminary analysis of the GSM8K dataset Cobbe et al.
(2021) suggests that relatively short questions with a small number of
described entities, a few calculation steps and a limited range of operators
involved in the solution (e.g., Mark is 7 years older than Amy, who is 15. How
old will Mark be in 5 years?) are typically answered correctly by a range of
LLMs. At the same time, long questions requiring real-world knowledge (e.g.,
how many cents there are in a dollar) and extended natural language
understanding (NLU) (e.g., interpretation of a lower price) pose challenges
for LLMs (see Figure 1).

In this paper, we formulate and investigate two research questions: (1) Which
characteristics of the input math word questions make them complex for an LLM?
and (2) Based on these characteristics, can we predict whether a particular
LLM will be able to solve specific input MWPs correctly?

##  2 Methodology

##### Data

We use the GSM8K dataset Cobbe et al. (2021), divided into $7,473$ training
and $1,319$ test instances, because of the high quality of human-generated
MWPs. This dataset contains a diverse set of problems in English with minimal
amount of recurring templates. Furthermore, the difficulty level of the
problems is tailored for LLMs, allowing for a wide variation in correctness
across models and question types, which is ideal for our feature-based
analysis.

##### Approach

We collect solution attempts from several LLMs to the questions from the GSM8K
training and test sets. Next, we train statistical classifiers on a filtered
subset of questions to predict if they are consistently solved correctly or
incorrectly across multiple runs of the models. Our approach is relatively
simple but it allows us to investigate which of the features are most
indicative of the challenges LLMs face in solving math problems.

##### LLMs

We select an array of open-source models for our experiments. We use Llama2
(13B and 70B) Touvron et al. (2023), Mistral-7B Jiang et al. (2023) as its
performance on math tasks has been found to match models several times its
size, and MetaMath-13B Yu et al. (2023) as it is fine-tuned on math QA data in
contrast to the other general-purpose models in the pool.

##### Features

We analyze and experiment with the features extracted from MWP questions and
their respective expected solutions. This way, the features remain grounded in
the dataset, allowing our approach to be applied to any LLM. The features are
broadly grouped into the following categories:222The complete list of features
extracted, their description and further statistics can be found in Appendix A

  1. 1.

Linguistic features focus on the phrasing of the question. These include the
length of the question, sophistication of the vocabulary, syntactic
complexity, instances of coreference, and overall readability. Note that the
linguistic features are only extracted from the question body as the phrasing
of the gold solution has no impact on the expected answer.

  2. 2.

Mathematical features cover the math arguments, operations, and reasoning
steps required to solve the questions. These include the number and diversity
of the math operations in the solution body. Arguments provided in the
question but not utilized in the solution also require mathematical reasoning
for them to be disregarded as noise. Note that while a question can be phrased
in many ways (affecting its linguistic features), the underlying math
operations and reasoning steps (thus, the mathematical features) remain
unchanged.

  3. 3.

Real-world knowledge & NLU based features indicate the amount of extraneous
information needed to solve the task that is not provided explicitly in the
question. This may include how many days there are in a month or the
interpretation of “half” as $1/2$.

##  3 Experiments

###  3.1 Solution Generation

To collect solution attempts from the LLMs, we use a simple task-specific
prompt (See Appendix B) to minimize any bias imposed on the model generation.
We query each LLM $5$ times on each question with varying generation seeds and
a temperature of $0.8$. A soft-matching strategy is then used to extract the
final answer from the solutions. Using each LLM’s attempted solutions, every
question is assigned a mean success rate using (# of correct answers) / (# of
solution attempts).

Model |  Success Rate ($N$=1,319 )
---|---
$\mu$ | $\sigma$
Llama2-13B | 0.3724 | 0.3681
Llama2-70B | 0.5609 | 0.3941
Mistral-7B | 0.3627 | 0.3309
MetaMath-13B | 0.6373 | 0.3816

Table 1: Success rates for solution attempts per LLM

###  3.2 Success Rate Prediction

We train and evaluate classifiers on their ability to predict for input test
questions whether they will be answered correctly or incorrectly by a specific
LLM. We also train and evaluate classifiers on the intersection set of
questions, which are either solved correctly by all or by none of the LLMs.

##### Models

We use Logistic Regression, Decision Tree, and Random Forest classifiers,
which allow us to extract relative feature importance with ease.

##### Data

For high confidence samples, we use the training and test subset from GSM8K
where the sampled success rate is either $1.0$ (always correct) or $0.0$
(never correct). The distribution of the LLM-specific splits is detailed in
Table 2.

##### Preprocessing & Optimization

We employ several preprocessing steps including dropping highly correlated
features, class-balancing, and feature scaling. We also perform a
hyperparameter search for each model to maximize performance on unseen data.
See Appendix C for more details.

##  4 Results

Class Distribution
---
Split | Class | Llama2-13b | Llama2-70b | Mistral-7b | MetaMath-13b | Intersection
Train | Always | 1102 (30.22%) | 2438 (61.36%) | 733 (24.06%) | 5162 (94.7%) | 205 (53.38%)
Never | 2545 (69.78%) | 1535 (38.64%) | 2313 (75.94%) | 289 (5.3%) | 179 (46.61%)
Total | 3647 | 3973 | 3046 | 5451 | 401
Test | Always | 188 (28.14%) | 427 (60.06%) | 111 (21.51%) | 528 (71.64%) | 31 (24.41%)
Never | 480 (71.86%) | 284 (39.94%) | 405 (78.49%) | 209 (28.36%) | 96 (75.59%)
Total | 668 | 711 | 516 | 737 | 135
Classification Performance
Classification Model | Llama2-13b | Llama2-70b | Mistral-7b | MetaMath-13b | Intersection
Acc. | Macro F1 | Acc. | Macro F1 | Acc. | Macro F1 | Acc. | Macro F1 | Acc. | Macro F1
Logistic Regression | 0.707 | 0.686 | 0.684 | 0.673 | 0.721 | 0.675 | 0.737 | 0.686 | 0.800 | 0.787
Decision Tree | 0.657 | 0.625 | 0.644 | 0.637 | 0.667 | 0.611 | 0.703 | 0.627 | 0.733 | 0.719
Random Forest | 0.767 | 0.724 | 0.717 | 0.707 | 0.814 | 0.738 | 0.744 | 0.549 | 0.815 | 0.799
RoBERTa-base | 0.816 | 0.771 | 0.756 | 0.738 | 0.838 | 0.743 | 0.701 | 0.415 | 0.811 | 0.781

Table 2: Class-wise distribution and classification results for different
LLMs. "Intersection" refers to questions always or never solved correctly by
all or any LLM, respectively. All classification results are mean values
across 5 runs with varying initialization seeds. The best results for feature-
based classifiers are highlighted in bold.

###  4.1 Success Rate Distribution

We report the mean success rates for each LLM on GSM8K’s test set in Table
1.333Our results generally align with those reported previously for these
models.  We observe that Llama2 13B and 70B follow the expected order of
scores along their respective parameter counts. Mistral-7B scores similar to
the 13B Llama2 model, and the additional fine-tuning allows MetaMath-13B to
outperform the other models (including the 70B Llama2). Figures 2(a) and 2(b)
respectively capture the number of questions always and never answered
correctly by each LLM. Overall, MetaMath-13B has the lowest number of
incorrectly and the highest number of correctly answered questions across the
tested LLMs.

(a) Always correct

(b) Never correct

Figure 2: Number of questions from the GSM8K-test (a) always and (b) never
answered correctly by each LLM. The rows in each figure correspond to
individual LLMs, with the counts on the right denoting the total number of
questions always (or never) answered correctly by each LLM. The counts at the
bottom denote the number of questions in each subset of LLMs.

###  4.2 Classification Results

To compare classifiers’ performance, we report the accuracy and macro-F1
scores for each classifier and LLM-specific test data split (see Table 2). We
observe that Random Forest outperforms other classifiers across most solution
sets.

At the same time, we also note that, due to significant class imbalance, this
task is not easy for the classifiers, with the best accuracy scores across LLM
splits being in the range of $71.7\%-81.4\%$. The small number of questions
always or never solved correctly by any LLM speaks to the models’ varying
capabilities (and potential points of brittleness). We include additional
analysis of the results in Appendix D.

For comparison, we also report the classification results for a fine-tuned
RoBERTa-base model Liu et al. (2019) for the same training and evaluation sets
(tuned on the question and gold solution as input text; see Appendix C for
more details) in Table 2. We note that the Transformer base classifier scores
on a par or a few points above the best statistical classifier, i.e., Random
Forest, suggesting that the proposed feature-based classifiers are not far
behind token-level contextual models for this task.

####  4.2.1 Feature Importance

The statistical classifiers used in our experiments allow us to estimate the
importance of each feature and its contribution to the classification
performance. We report the top $10$ features with the highest aggregate ranks
across LLM data splits and classifiers in Table 3. We use mean rank here as a
proxy for relative importance across features, and the respective standard
deviations indicate how spread out this importance is across classifiers and
queried LLMs. We observe that a greater number (Gx_op_unique_count) and
diversity (Gx_op_diversity) in math operations, and the use of infrequent
numerical tokens in the question and solution body (Qx_ &
Gx_mean_numerical_word_rank) impact the success rate. The list also contains
linguistic features based on the phrasing of the questions: longer questions
with a high number of noun phrases (Qx_np_count), mean syntactic depth
(Qx_constituency_tree_depth), and readability grade (Qx_flesch_kinkaid_grade)
are also difficult for LLMs to solve. Additionally, the need for extraneous
information (Gx_world_knowledge), such as conversion units for time, distance,
or weight, can make a question challenging. We also report value thresholds at
which each feature affects the success rate significantly: see the results of
the Student’s t-test and p-values in Table 7 in Appendix D.

Type | Feature Name |  Rank ($N$=23)
---|---|---
$\mu$ | $\sigma$
L | Qx_np_count | 1.2 | 0.45
M | Qx_mean_numerical_word_rank | 4 | 1.87
M | Gx_op_unique_count | 4 | 2.65
M | Gx_op_diversity | 4.4 | 2.30
M | Gx_mean_numerical_word_rank | 4.4 | 1.82
L | Qx_mean_word_rank | 5.6 | 1.82
L | Qx_flesch_kinkaid_grade | 6 | 1.87
W | Gx_world_knowledge | 7.8 | 2.28
L | Qx_constituency_tree_depth | 9.6 | 1.95
M | Gx_op’+’_count | 11.6 | 3.97

Table 3: Feature importance ranks across classification models and LLM-wise
data subsets.

####  4.2.2 Ablation Studies

To further measure the impact of each feature type, we report classification
scores along different feature-type subsets in Figure 3. We note that the
feature set with all types (L+M+W) is not optimal for classification. For
instance, the questions answered by Llama2-13B are best classified using only
mathematical features (M). The best-performing classifiers for Llama2-7B,
MetaMath-13B, and the intersection set either solely use linguistic features
(L) or both linguistic and math features (L+M), whereas the world knowledge &
NLU feature set if sufficient for Mistral-7B.

Figure 3: Results of the ablation studies across feature types (L –
Linguistic, M – Mathematical, W – World Knowledge & NLU). Each bar represents
the mean macro-F1 score over all three classifier models.

####  4.2.3 Impact of Linguistic Features

In order to better gauge the impact of linguistic features on the success
rate, we cluster questions by mathematical features. We fit a KMeans
clustering model444https://scikit-
learn.org/stable/modules/generated/sklearn.cluster.KMeans.html on all math
features for each question in the GSM8K training set with a target cluster
count of $100$. This helps group together questions from the data, wherein the
math features hardly vary within each question subset (or cluster). Thus,
variations in success rate across the questions within a cluster can be more
clearly attributed to other, i.e., linguistic types of features. We report
some notable Spearman correlation values between the linguistic feature values
within a cluster and the corresponding success rates in Table 4. The strong
and significant feature-wise negative correlations suggest that for a
relatively fixed set of math features, questions with greater length, nesting,
lexical rank, and reading grade become more challenging for LLMs to solve.
Note that this form of analysis on feature-based minimal pairs is extractive
in nature and may, to a certain extent, be restricted to the question types in
the GSM8K dataset. For a more exhaustive analysis for each feature, generative
approaches to furnish question paraphrases with the desired set of linguistic
features need to be employed.

Cluster | Feature | Spearman($\rho$)
---|---|---
ID | Size
09 | 27 | Qx_constituency_tree_depth | -0.64***
24 | 14 | Qx_mean_word_rank | -0.82***
63 | 62 | Qx_token_length | -0.42***
Qx_word_length | -0.43***
96 | 51 | Qx_flesch_kinkaid_grade | -0.51***

Table 4: Cluster-wise feature correlations. The cluster count represents the
number of questions included in the respective cluster. The p-value for all
reported correlation values is <0.001 (marked by ‘***’).

##  5 Conclusions

This work aims to identify what aspects of MWPs make them difficult for LLMs
to solve. To this end, we extract key features (spanning linguistic,
mathematical, and real-world knowledge & NLU-based aspects) to predict whether
several LLMs can reliably solve MWPs from GSM8K. We find that questions with a
high number and diversity of math operations using infrequent numerical tokens
are particularly challenging to solve. In addition, we show that lengthy
questions with low readability scores and those requiring real-world knowledge
are also seldom solved correctly. Our future work will rely on these findings
to make informed modifications to questions in order to study the impact on
LLMs’ reasoning and MWP-solving abilities. Figure 4 provides an example of an
informed modification, which leads to improved LLM performance.

Figure 4: Solution attempt by Llama2-70B on the question from Figure 1, with
the required real-world knowledge explicitly specified.

## Limitations

With the rapidly growing body of research on LLMs, this study necessarily has
several limitations, which we discuss below.

##### Limited set of LLMs tested

We consider it important to test and report results with a diverse set of
open-source LLMs, which motivated the selection of the specific models
included in this study. At the same time, we do not claim this study to be
comprehensive with respect to the range of LLMs tested and in future work, we
plan to include more LLMs in this research.

##### Limited number of classification models

As the main goal of this study is to identify aspects of the MWPs that make
them challenging for LLMs to solve, we have opted for a feature-based approach
and a range of traditional classification models as opposed to less
transparent but more powerful black-box algorithms. Our results show that the
prediction task is challenging for the traditional classifiers that we used,
and it is likely that these results can be improved with stronger
classification models.

##### Limitations of the dataset

In this work, we have focused on a single MWP dataset (GSM8K) due to its
unique properties, namely the high quality of the questions, high diversity of
the tasks (including linguistic diversity of the questions), and moderate
difficulty of the math problems covered Cobbe et al. (2021). At the same time,
we recognize that the results we report in this work may be limited in certain
ways to the dataset on which we report them. Our future work will apply this
approach to other available MWP datasets Kim et al. (2023); Wang and Lu
(2023); Patel et al. (2021); Miao et al. (2020) to verify the consistency of
the findings.

##### Impact studies

Finally, whilst we have identified aspects of the MWPs that make them
challenging for LLMs to solve, we admittedly presented only one example (see
Figure 4) where acting upon one of the identified aspects improves the output
of an LLM. While a thorough investigation of the impact of such informed
modifications is outside the scope of the current paper, such experiments will
follow in future work to demonstrate the practical usefulness of the
identified MWP aspects.

## Ethics Statement

We foresee no serious ethical implications from this study.

## Acknowledgments

We are grateful to the Campus Super Computing Center at MBZUAI for supporting
this work. We also thank the anonymous reviewers for their valuable feedback.

## References

  * Almoubayyed et al. (2023) Husni Almoubayyed, Stephen E Fancsali, and Steve Ritter. 2023.  Instruction-embedded assessment for reading ability in adaptive mathematics software.  In _LAK23: 13th International Learning Analytics and Knowledge Conference_ , pages 366–377.
  * Beeching et al. (2023) Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. 2023.  Open llm leaderboard.  https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_.
  * Flesch (1948) Rudolph Flesch. 1948.  A new readability yardstick.  _Journal of applied psychology_ , 32(3):221.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7b.
  * Kim et al. (2023) Jiwoo Kim, Youngbin Kim, Ilwoong Baek, JinYeong Bak, and Jongwuk Lee. 2023.  It ain’t over: A multi-aspect diverse math word problem dataset.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 14984–15011, Singapore. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Miao et al. (2020) Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020.  A diverse corpus for evaluating and developing English math word problem solvers.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 975–984, Online. Association for Computational Linguistics.
  * Norberg et al. (2023) Kole Norberg, Husni Almoubayyed, Stephen E. Fancsali, Logan De Ley, Kyle Weldon, April Murphy, and Steven Ritter. 2023.  Rewriting math word problems with large language models.  In _Proceedings of the Workshop on Empowering Education with LLMs - the Next-Gen Interface and Content Generation 2023 co-located with 24th International Conference on Artificial Intelligence in Education (AIED 2023), Tokyo, Japan, July 7, 2023_ , volume 3487 of _CEUR Workshop Proceedings_ , pages 163–172. CEUR-WS.org.
  * Patel et al. (2021) Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021.  Are NLP models really able to solve simple math word problems? In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2080–2094, Online. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raiyan et al. (2023) Syed Rifat Raiyan, Md Nafis Faiyaz, Shah Md. Jawad Kabir, Mohsinul Kabir, Hasan Mahmud, and Md Kamrul Hasan. 2023.  Math word problem solving by generating linguistic variants of problem statements.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)_ , pages 362–378, Toronto, Canada. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.
  * Wang and Lu (2023) Tianduo Wang and Wei Lu. 2023.  Learning multi-step reasoning by solving arithmetic tasks.  In _Proceedings of ACL_.
  * Wei et al. (2022) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022\.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_.
  * Yu et al. (2023) Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023.  Metamath: Bootstrap your own mathematical questions for large language models.
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_.
  * Zheng et al. (2023) Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. 2023.  Progressive-hint prompting improves reasoning in large language models.  _arXiv preprint arXiv:2304.09797_.
  * Zhu et al. (2023) Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, et al. 2023.  Promptbench: Towards evaluating the robustness of large language models on adversarial prompts.  _arXiv preprint arXiv:2306.04528_.

##  Appendix A Feature Details

Below, we describe the features used in our study and how they were extracted.

Type | Source | # | Feature Name | Range | $\mu$ | $\sigma$
---|---|---|---|---|---|---
L | Q | 1 | Qx_token_length | [12 – 239] | 66.05 | 24.384
Q | 2 | Qx_sentence_length | [1 – 13] | 3.431 | 1.201
Q | 3 | Qx_word_length | [9 – 184] | 45.885 | 17.832
Q | 4 | Qx_flesch_kinkaid_grade | [-1.9 – 26.3] | 4.236 | 2.468
Q | 5 | Qx_mean_word_rank | [3661.96 – 21929.96] | 10646.615 | 2110.891
Q | 6 | Qx_constituency_tree_depth | [5 – 31] | 10.803 | 2.798
Q | 7 | Qx_np_count | [3 – 74] | 18.034 | 7.488
Q | 8 | Qx_prp_count | [0 – 16] | 1.772 | 1.854
Q | 9 | Qx_coref_count | [0 – 16] | 0.462 | 1.283
M | Q | 10 | Qx_arg_count | [0 – 17] | 4.438 | 1.94
Q | 11 | Qx_word_arg_count | [0 – 14] | 1.091 | 1.397
Q | 12 | Qx_mean_numerical_word_rank | [259.0 – 29905.38] | 22643.319 | 3260.09
G | 13 | Gx_arg_count | [6 – 73] | 24.377 | 9.732
G | 14 | Gx_op‘+’_count | [0 – 12] | 1.06 | 1.212
G | 15 | Gx_op‘-’_count | [0 – 6] | 0.601 | 0.78
G | 16 | Gx_op‘*’_count | [0 – 8] | 1.369 | 1.183
G | 17 | Gx_op‘/’_count | [0 – 7] | 0.621 | 0.789
G | 18 | Gx_op‘(’_count | [0 – 4] | 0.026 | 0.187
G | 19 | Gx_op_unique_count | [0 – 6] | 2.284 | 0.93
G | 20 | Gx_op_diversity | [0.15 – 1.0] | 0.758 | 0.196
G | 21 | Gx_mean_numerical_word_rank | [22645.0 – 29915.0] | 28626.04 | 776.73
B | 22 | Gx_parameter_usage | [0.07 – 1.0] | 0.642 | 0.241
W | B | 23 | Gx_world_knowledge | [0 – 8] | 1.104 | 1.006

Table 5: Details of formulation and distribution (across GSM8K) for all
features included in the feature set. Each feature is of type: Linguistic (L),
Mathematical (M), or World Knowledge and NLU (W) and is sourced either from
the question body (Q), gold solution body (G), or both (B).

(1) Linguistic features (L) include $9$ features pertaining to the question
(Q) itself:

  * •

Qx_token_length: The number of tokens in the tokenized version of the question
body. We apply each LLM’s respective tokenizer from HuggingFace to extract
this feature.

  * •

Qx_sentence_length: The number of sentences detected in the question body. We
use the sentence_splitter Python library to extract this count.

  * •

Qx_word_length: The number of space-separated segments (words) in the question
body.

  * •

Qx_flesch_kinkaid_grade: The readability grade of the question body as per the
FKGL metric Flesch (1948). We use the textstat Python library to extract this
feature.

  * •

Qx_mean_word_rank: The mean vocabulary rank (in decreasing order of frequency)
of the tokens in the question body. We use the same tokenizer set used for
Qx_token_length.

  * •

Qx_constituency_tree_depth: The mean depth of the constituency tree across the
sentences in the question body. We use Stanford’s Stanza parsing library to
parse the questions.

  * •

Qx_np_count: Number of distinct noun phrases detected in the question body. We
extract this from the constituency parse collected from the Stanza parser.

  * •

Qx_prp_count: Number of prepositions in the question body. We use the part-of-
speech tags generated as part of the parse by Stanza.

  * •

Qx_coref_count: Number of pronominal or nominal instances of coreference in
the question body. We use Stanford’s CorefAnnotator to extract this feature.

(2) Mathematical features (M) include $12$ features pertaining to the question
(Q) and gold solution (G):

  * •

Qx_arg_count: The number of distinct numerical quantities (e.g., “3.5 hours
later” or “100 boxes”) in the question body. We use a Regexp pattern to detect
whole numbers, decimal point values, and quantities preceded by a negative
sign or dollar (and other currency) signs.

  * •

Qx_word_arg_count: The number of quantities mentioned in word-form (“three
times” or “half as much”) in the question body. We use a vocabulary of
frequently used word-form tokens and accommodate compound expressions (e.g.,
“twenty-two”).

  * •

Qx_mean_numerical_word_rank: The mean vocabulary rank of the numerical tokens
in the question body. We first isolate numerical tokens tokenized by
respective tokenizers, then aggregate their token rank.

  * •

Gx_arg_count: The number of distinct numerical quantities present as plain
text or on the left-hand side of equations in the gold solution. We use the
same Regexp pattern used for Qx_arg_count.

  * •

Gx_op{‘+’/ ‘-’/ ‘*’/ ‘/’/ ‘(’}_count: Number of times each listed math
operation is used in the gold solution. A simple Regexp pattern is applied to
extract these from within equations.

  * •

Gx_op_unique_count: The maximum number of times a single operation has been
used in the gold solution. For instance, “$3+4.5+7+1-2.7$” contains 3
instances of the ‘+’ operator.

  * •

Gx_op_diversity: Ratio of the number of unique math operators used to the
total number of operators in the gold solution. For instance, a question with
the consolidated math solution expression "$(2\times 12)\times 3=72$" contains
two arithmetic operations in total but only one unique operation type, i.e.,
‘$\times$,’ Gx_op_diversity$=1/2=0.5$.

  * •

Gx_mean_numerical_word_rank: The mean vocabulary rank of the numerical tokens
used on the left-hand side of equations in the gold solution. Extracted the
same way as Qx_mean_numerical_word_rank.

  * •

Bx_parameter_usage: The ratio of distinct arguments used in the gold solution
to that in the question body. A value lower than 1 indicates that one or more
arguments provided in the question were not required to solve the MWP
(potentially acting as distractors).

(3) World knowledge and NLU features (W) include:

  * •

Bx_world_knowledge: The number of distinct arguments on the left-hand-side of
equations in the gold solution, that are neither present in the question body
nor produced as intermediate results from any prior equations in the solution.
A non-zero value is interpreted as the use of a quantity (perhaps a conversion
factor, or the number of entities involved in computing mean) unspecified by
the question. The arguments were extracted from both sides using the same
Regexp policies used for previous features.

Table 5 shows further statistics on the features, including the range as well
as the mean and standard deviation of the values for each feature type.
Additionally, we report the Spearman correlation between all pairs in the
feature set in Figure 5.

Figure 5: Spearman correlation matrix between features. All correlation values
are marked with ‘*’, ‘**’, and ‘***’ if their corresponding p-values are less
than $0.05$, $0.01$, and $0.001$ respectively.

##  Appendix B Querying Details

###  B.1 Prompt Template

We use a simple task-specific prompt (see Figure 6) either prepended to the
question-prompt or specified as a system-prompt if an LLM input query format
requires so.

Figure 6: Prompt template used for solution generation across LLMs.

###  B.2 LLM Details

The exact large language models used in our experiments, along with their
reported performance on GSM8K according to the OpenLLM leaderboard Beeching et
al. (2023) are mentioned in Table 6. All LLMs and libraries used are open-
source. The license to use Meta’s Llama2 models was procured through due
process.

Model | HuggingFace Model Name | Pass@1
---|---|---
Llama2-13B | meta-llama/Llama-2-13b-chat-hf | 28.70
Llama2-70B | meta-llama/Llama-2-70b-chat-hf | 56.80
Mistral-7B | mistralai/Mistral-7B-Instruct-v0.2 | 40.03
MetaMath-13B | meta-math/MetaMath-13B-V1.0 | 72.30

Table 6: List of HuggingFace model variants and their respective reported
pass@1 (single run) accuracies on the GSM8K test set from the OpenLLM
leaderboard.

###  B.3 Implementation and Compute Resources Used

We use vLLM to load and query models. Models of parameter sizes 7B and 13B
were queried with a single NVIDIA A100 GPU. Llama2-70B was loaded and queried
using 4 A100 GPUs. Each query was set to a temperature of $0.8$, and a maximum
token length of $2000$. Each question was queried $5$ times by each LLM, with
a varying seed. Querying the entire GSM8K dataset ($8,793$ questions) took
approximately 1 hour for each LLM.

Feature | Llama2-13B | Llama2-70B | Mistral-7B | MetaMath-13B | Intersection
---|---|---|---|---|---
Thresh. | Diff. | T-val | Thresh. | Diff. | T-val | Thresh. | Diff. | T-val | Thresh. | Diff. | T-val | Thresh. | Diff. | T-val
Gx_num_arg_count | 51.102 | 0.313 | -8.328 | 53.273 | 0.380 | -7.747 | 55.082 | 0.283 | -5.452 | 59.061 | 0.438 | -5.558 | 57.735 | 0.330 | -5.977
Qx_np_count | 40.673 | 0.200 | -5.440 | 47.465 | 0.435 | -5.090 | 33.429 | 0.207 | -11.201 | 47.918 | 0.448 | -7.230 | 47.918 | 0.333 | -5.645
Gx_arg_count | 57.959 | 0.336 | -6.018 | 58.111 | 0.380 | -5.742 | 57.959 | 0.289 | -5.644 | 51.122 | 0.271 | -10.323 | 60.694 | 0.335 | -5.249
Qx_word_length | 91.143 | 0.219 | -7.944 | 97.384 | 0.300 | -8.468 | 19.714 | 0.233 | -7.016 | 101.857 | 0.275 | -9.267 | 116.143 | 0.251 | -5.328
Gx_op’-’_count | 2.082 | 0.198 | -7.683 | 2.061 | 0.246 | -9.009 | 2.082 | 0.181 | -7.660 | 3.061 | 0.339 | -7.348 | 3.061 | 0.280 | -6.383
Qx_prp_count | 7.184 | 0.215 | -5.957 | 8.081 | 0.313 | -6.322 | 8.163 | 0.235 | -5.475 | 7.184 | 0.167 | -6.039 | 8.163 | 0.233 | -6.834
Qx_sentence_length | 6.143 | 0.206 | -6.843 | 6.091 | 0.264 | -8.278 | 6.143 | 0.203 | -7.357 | 7.122 | 0.253 | -5.756 | 7.122 | 0.229 | -5.473
Gx_op’+’_count | 4.163 | 0.241 | -7.849 | 4.121 | 0.293 | -8.996 | 5.143 | 0.296 | -6.038 | 3.184 | 0.080 | -5.377 | 5.143 | 0.238 | -6.081
Qx_token_length | 123.184 | 0.211 | -8.937 | 126.646 | 0.258 | -9.557 | 113.918 | 0.214 | -12.560 | 132.449 | 0.232 | -9.953 | 118.551 | 0.219 | -14.125
Gx_op’*’_count | 4.082 | 0.211 | -6.135 | 4.040 | 0.220 | -6.043 | 4.082 | 0.184 | -5.831 | 5.061 | 0.268 | -5.423 | 5.061 | 0.250 | -5.295

Table 7: Feature-wise thresholds which reflect the greatest difference in the
corresponding mean success rate. For each feature, the optimal threshold
creates two sets of questions on either side, wherein the difference in the
corresponding mean success rates of the two sets is the greatest. We perform
Student’s $t$-tests on both sets to determine if this difference is
significant and report the corresponding $t$ values. All results reported in
the table have an absolute $t$-value >5 and a $p$-value <0.0001.

Classification Model | Llama2-13B | Llama2-70B | Mistral-7B | MetaMath-13B | Intersection
---|---|---|---|---|---
F1(Never) | F1(Always) | F1(Never) | F1(Always) | F1(Never) | F1(Always) | F1(Never) | F1(Always) | F1(Never) | F1(Always)
Logistic Regression | 0.8454 | 0.4968 | 0.5641 | 0.7862 | 0.8926 | 0.5081 | 0.0095 | 0.8354 | 0.8263 | 0.7184
Decision Tree | 0.8199 | 0.4532 | 0.4958 | 0.7431 | 0.8732 | 0.4592 | 0.0727 | 0.8373 | 0.732 | 0.6496
Random Forest | 0.8535 | 0.5049 | 0.6133 | 0.7824 | 0.8936 | 0.5161 | 0.019 | 0.8361 | 0.8439 | 0.7216

Table 8: Class-wise F1 scores for each classification model across LLM
solution splits.

##  Appendix C Training Details

###  C.1 Preprocessing

Before training classifiers, we perform the following steps on the feature
data:

  1. 1.

Pruning: For each training feature-set, we iteratively remove features with
high correlation with other features until no two columns in the data have an
absolute Spearman correlation higher than $0.5$.

  2. 2.

Scaling: We fit scikit-learn’s StandardScaler onto the train split to
normalize the mean and standard deviation of all features. We then apply the
same scaler on the test split features.

  3. 3.

Balancing: As many LLM solution splits either contain too few always-correct
or always-incorrect question samples, we use imblearn’s RandomOverSampling
tool to balance the proportions of the two classes in each run.

###  C.2 Hyperparameter Search

For each classifier model and LLM solution split pair, we conduct Bayesian
optimization on the ranges of key hyperparameters for each classifier. As the
objective function, we maximize the Macro-F1 score on a 15% held-out set of
GSM8K’s test split to prevent the models from over-fitting onto the train
samples.

###  C.3 Fine-tuning RoBERTa Classifier

We use HuggingFace’s Trainer module to tune a pre-trained RoBERTa-base
classifier on the same target data as the statistical classifiers in each
setting. The corresponding input text for training and evaluation was built by
concatenating the GSM8K question and gold-solution text for each sample, i.e.,
"<question> Question Body </question> <solution> Solution Body </solution>".
The model is trained for $3$ epochs with a peak learning rate of $2e-5$ and a
warmup ratio of $0.1$. On a Tesla P100, each training run took approximately
10-15 minutes.

##  Appendix D Results Analysis

###  D.1 Class-wise Classification Review

Table 8 reports the class-wise F1-scores (for questions that are always or
never solved correctly) for each classifier across LLM question splits. Though
there may be notable class imbalances among the two classes across splits, all
classifiers were trained with proportional oversampling. We see that for
relatively smaller pretrained models, i.e., Llama2-13B and Mistral-7B, F1
scores for always-incorrect questions are significantly higher than their
counterparts. Thus, the scores indicate that for smaller models, questions
answered incorrectly are more predictable. For larger models like Llama2-70B,
this difference is lower, with the always-correct questions being somewhat
more predictable. For the fine-tuned MetaMath-13B model, the small number of
questions that are never answered correctly, fail to provide a generalizable
sample for predicting on unseen data.

###  D.2 Feature Impact

We continue our discussion of feature importance (from Section 4.2.1) by
identifying pivot points for key features about which the corresponding
success rates for questions show a significant difference in mean values. We
perform Student’s $t$-tests on equally spaced thresholds along each feature
and report the thresholds which show the highest variation in mean success
rates in Table 7.

We see that across most LLMs, a significant rise in the mean success rate is
observed as the question contains, on average, more than $6$-$7$ sentences,
$90$-$115$ words, or $113$-$132$ tokens. We get a better idea of the kind of
questions Mistral-7B gets wrong more often than other models, as its threshold
($19.71$ words) for the number of words in the question body is substantially
lower than the average. Across features involving number of math operations,
the threshold for the fine-tuned MetaMath-13B model is either on a par or
higher than other models.


31 March 2020

# Electro-optically modulated lasers: important pulse energy and contrast
optimization derived from theoretical spectrum

Denis Marion  Université de Bordeaux, CNRS, CEA, CELIA (Centre Lasers Intenses
et Applications), UMR 5107, 33405 Talence, France  Corresponding author:
<EMAIL_ADDRESS>Jérôme Lhermite  Université de Bordeaux, CNRS,
CEA, CELIA (Centre Lasers Intenses et Applications), UMR 5107, 33405 Talence,
France

###### Abstract

In this article, we derive analytically the complex optical spectrum of a
pulsed laser source obtained when a frequency comb generated by phase
modulation is input into a synchronized intensity modulator. As the exact
knowledge of the spectrum allows to reshape it arbitrarily with an optical
spectrum processor, we show how this operation can lead to a 75% increase in
pulse energy and an enhancement of the extinction ratio by at least three
orders of magnitude in some cases. This method also enables large-factor rate-
multiplications of these versatile coherent sources using the Talbot effect
with negligible degradation of the signal.

###### doi:

https://dx.doi.org/10.1364/OL.391192

††journal: ol

Electro-optically modulated laser (EOML) setups are an emerging class of
pulsed laser sources delivering short light bursts at an ultra-high repetition
rate (RR, typically tens of GHz), usually tunable up to one or more octaves.
Contrarily to cavity-based mode-locked laser oscillators, these sources rely
on one or more phase modulators (PM) driven by a radio-frequency (RF)
oscillator to generate a wide optical spectrum. They are, in that sense, very
similar to electro-optic frequency combs. However, although it was not the
case in the earlier designs [1], a synchronized intensity modulator (IM) is
nowadays generally added, which is not the case for frequency combs used in
the spectral domain.

The spectral phase of an EOML is directly linked to the synchronization of all
the phase and intensity modulators. Neglecting the phase noise of the RF
oscillator as well as synchronization errors, it is definite, constant and
even [2]. As a direct consequence, the pulses emitted by these sources can
readily be compressed very close to their Fourier time-limited (FTL) duration.
Recent EOMLs deliver very short pulses ($\leqslant 350$ fs) without the need
for a nonlinear mode-locking process.

Historically, the first developments were carried out around 1550 nm, where
wideband RF modulators based on lithium niobate waveguides were first
available [3, 4, 5, 6, 7], but more recently new setups centered around 1 µm
were demonstrated [8, 9], giving access to the high average power of
ytterbium-doped fiber amplifiers (YDFA) and also allowing the nonlinear post-
compression of the pulses down to [9] 80 fs.

EOMLs have some drawbacks: they are relatively expensive as their total
spectral width is dictated by the number of PMs, and their extinction ratio
over the RR period, defined here as the peak-to-valley instantaneous power
ratio, is orders of magnitude higher than that of typical mode-locked lasers.
However, EOMLs gather many unique features. They are reliable all-fiber
sources without any need for nonlinear component and offer a very large span
of repetition rate in an unusual range (larger than 10 GHz typ.). Being based
on low spectral width laser emitters, they can also easily be tuned to the
desired central wavelength by changing the emitter and/or fine-tuned by
adjusting its temperature. Their spectral properties can be readily simulated,
as no population inversion nor nonlinear propagation needs be described [8]. A
more detailed review of technologies related to ultrahigh repetition rate
sources and electro-optic combs can be found elsewhere [10] and is beyond the
scope of this work.

The IM enhances the contrast over the period by at least one order of
magnitude, but it also modifies the spectrum of the frequency comb in a non
negligible way. Note that, to the best of our knowledge, such an analysis of
an EOML mixing phase and intensity modulations has never been presented
before.

The first part of this work is dedicated to the analytical calculation of the
complex spectrum. We then use the derived expressions to analyze the spectral
phase and show that a pulse shaper is generally essential to avoid detrimental
artefacts in the time domain. Lastly, we show how this derivation can be used,
in conjunction with such a device, to optimize greatly the time profile and
make it suitable for rate-multiplication by temporal Talbot effect. Throughout
this paper, we use the term "pulse shaper" to designate an optical spectrum
processor [11] based on a 2D spectral light modulator (SLM) allowing to shape
both the amplitude and the phase of the optical spectrum. Nowadays, such
devices with a spectral resolution of typically 20 GHz or less are
commercially available.

$E_{\text{in}}(t)$PM1…PMkIM$E_{\text{out}}(t)$RF osc $\omega_{m}$$\varphi$
Figure 1:  Generalized description of an EOML setup. PMj: synchronized phase
modulators, RF osc: radio-frequency oscillator delivering a perfect sine law
of pulsation $\omega_{m}$. The IM phase relative to the PMs is $\varphi$.

We restrict our approach to the EOML systems equivalent to that depicted in
Fig. 1. All PMs are supposed synchronized. We also make the hypothesis that
the optical electric field can be described by scalar theory.

The input field $E_{\text{in}}(t)$ is usually generated by a quasi-
monochromatic laser diode. Here, it is supposed of the form
$E_{\text{in}}(t)=E_{0}e^{\imath\omega_{0}t}$ where $\omega_{0}=2\pi
c/\lambda_{0}$ is the optical pulsation of the diode and $c$ is the speed of
light in vacuum. PMj adds an instantaneous phase modulation by the quantity

$\phi_{j}(t)=\pi\frac{V_{j}}{V_{\pi}}\sin\omega_{m}t$ (1)

where $V_{j}$ is the amplitude of the voltage on PMj and $V_{\pi}$ their half-
wavelength voltage, supposed common here for all the modulators (PMs and IM).
$\omega_{m}=2\pi/T_{m}$ is the modulation pulsation of all modulators. Since
the PMs of the setup have a null phase relative to the RF oscillator, the
scalar electric field at the output of PMk reads

$E_{\text{PM}}(t)=E_{0}e^{\imath\omega_{0}t}e^{\imath(\phi_{1}(t)+\dots+\phi_{k}(t))}=E_{0}e^{\imath\omega_{0}t}e^{\imath\beta\sin{\omega_{m}t}}$
(2)

where $\beta$ is the total phase modulation depth of the EOML:

$\beta=\pi\sum_{j=1}^{k}\frac{V_{j}}{V_{\pi}}$ (3)

The IM is usually a miniaturized Mach-Zehnder modulator embedded on a lithium
niobate waveguide. We may assume without loss of generality that its amplitude
transmission is in the form

$\tau(t)=\sin\left[\Phi_{0}+\Phi_{1}\sin(\omega_{m}t+\varphi)\right]$ (4)

where $\Phi_{0}$ is an offset controlled via the bias voltage of the IM and
$\Phi_{1}$ is the amplitude of modulation given by
$\Phi_{1}=V_{\text{IM}}/V_{\pi}$.

We recall the general Jacobi-Anger relation which gives the Fourier
decomposition of a purely phase-modulated signal:

$e^{\imath m\sin(\omega t+\psi)}=\sum_{n=-\infty}^{+\infty}J_{n}(m)e^{\imath
n\psi}e^{\imath n\omega t}$ (5)

where $J_{n}$ is the $n$-th Bessel function of the first kind. For the sake of
conciseness, in the following we omit the infinite summation bounds. Using
this relation and after some algebraic manipulations, the Fourier transform of
$\tau$ reads

$\hat{\tau}(\omega)=\sum_{n}\alpha_{n}(\Phi_{0},\varphi)J_{n}(\Phi_{1})\delta(\omega-n\omega_{m})$
(6)

where $\delta$ is the Dirac distribution and $\alpha_{n}(\Phi_{0},\varphi)$ is
defined as

$\displaystyle\alpha_{n}(\Phi_{0},\varphi)=\begin{cases}\sin\Phi_{0}e^{\imath
n\varphi}\ &\text{if $n$ is even}\\\ -\imath\cos\Phi_{0}e^{\imath n\varphi}\
&\text{if $n$ is odd}\end{cases}$ (7)

The scalar field at the output of the EOML writes
$E_{\text{out}}(t)=\tau(t)E_{\text{PM}}(t)$ or equivalently
$\hat{E}_{\text{out}}=\hat{\tau}*\hat{E}_{\text{PM}}$, where $*$ denotes the
convolution product. Using Eq. (5) we obtain

$\hat{E}_{\text{PM}}(\omega)=E_{0}\sum_{n}J_{n}(\beta)\delta_{n}(\omega)$ (8)

where $\delta_{n}(\omega)=\delta(\omega-\omega_{0}-n\omega_{m})$. From Eqs.
(6) and (8), the complex spectrum of an EOML can finally be written as

$\hat{E}_{\text{out}}(\omega)=E_{0}\sum_{n}\gamma_{n}\delta_{n}(\omega)$ (9)

where

$\gamma_{n}=\sum_{k}\alpha_{k}(\Phi_{0},\varphi)J_{k}(\Phi_{1})J_{n-k}(\beta)$
(10)

Although the spectrum of EOMLs may be approximated by Fourier simulations,
Eqs. (7) and (10) give a clear understanding of its complex features. They
also explicit the role of the IM and directly link its temporal parameters
(bias, amplitude, relative phase) with the shape and phase of the spectrum. In
the following, we name $E_{\text{FL}}$ the FTL function given by

$E_{\text{FL}}=E_{0}\sum_{n}|\gamma_{n}|e^{\imath n\omega_{m}t}$ (11)

$I_{\text{FL}}(t)=|E_{\text{FL}}(t)|^{2}$, and $\psi(\omega)$ the spectral
phase $\arg({\hat{E}_{\text{out}}(\omega)})$.

We illustrate the rest of this work with the curves obtained for an ideal EOML
($\varphi=0\ [\pi]$) modulated at $\beta=20\pi$, a value easily reachable with
current PMs and RF amplifiers [8, 9]. Taking different values for $\beta$
alters the spectrum and temporal profile of EOMLS, but doesn’t change the
validity of the following matters.

Figure 2: SPD (solid, black) and $\psi$ (dashed, red) of an ideal EOML
($\Phi_{0}=\Phi_{1}=\pi/4$, $\varphi=0$). Inset: $\psi-\psi_{2}$.

As is clearly visible on Fig. 2, the FWMH of the spectrum is very well
approximated by the value $2\beta$. Also, the unwrapped spectral phase of the
ideal EOML is purely even. This is verified taking $\varphi=0$ in Eq. (7).
Visually, the phase also appears to be mainly of $2^{\text{nd}}$ order. This
is consistent with the successful compression setups based on a chirped volume
Bragg grating (CVBG), a pulse shaper or a reflective grating in the literature
[2, 9, 12, 13]. In order to emulate a $2^{\text{nd}}$-order compression, we
carried out a least squares optimization with a polynomial of
2${}^{\text{nd}}$ degree. We name $E_{2}(t)$ the field obtained after
subtracting the fitted phase $\psi_{2}$ from $\hat{E}_{\text{out}}$, and
$I_{2}=|E_{2}|^{2}$.

The remainder $\psi-\psi_{2}$ is presented in the inset of Fig. 2: the ripples
clearly show that very high-order components are present in the spectral
phase, which prevent a compression down to the FTL duration, with an effect
both on the peak intensity and the pulse profile. Let us emphasize that static
compressors such as reflective gratings or even tailored CVBGs are not an
option in setups where $\beta$ and $\omega_{m}$ are adjustable, since $\psi$
and then $\psi_{2}$ change with these parameters.

Fig. 3 depicts the intensity over an entire period, for $E_{\text{FL}}$ as
well as $E_{2}$. It illustrates the main drawback of EOMLs: their extinction
ratio slowly reaches a plateau (here, about $10^{-3}$) one quadrant before and
after the main pulse, and stays at $10^{-7}$ only on a small time range about
one half-period after the pulse.

Figure 3: Intensity of an EOML (scaled to 1 in magnitude) over one modulation
period after a best-fit quadratic compression ($I_{2}$, dashed) or at Fourier
limit ($I_{\text{FL}}$, solid).

An important part of the residual phase can be corrected using a pulse shaper
[2, 12] provided $\omega_{m}/2\pi$ is superior to its frequency resolution. As
the experimental determination of $\psi$ is complex, published setups rely on
a step-by-step optimization of the programmed filter, hence leading to long
procedures and only local optima. Alternatively, thanks to Eq. (10), the
experimental SPD may be fitted against $(|\gamma_{n}|^{2})_{n}$ relative to
$\Phi_{0},\Phi_{1},\varphi$ and $\beta$. The resulting best-fit parameters
would give a more physically correct law
$\psi_{n}^{\text{fit}}=\text{arg}(\gamma_{n}^{\text{fit}})$ than what is
achieved with usual algorithms before resorting to line-by-line corrections.
This method could in the future prove faster and more accurate than the
arbitrary polynomial phase-fitting protocol.

The derivation of $\hat{E}_{\text{out}}$ given in Eq. (9) also enables a novel
spectral optimization method: it is possible to exactly invert the spectrum,
rendering it flat, and then to multiply it by a second arbitrary filter.
Hence, one may reshape the spectrum into a Gaussian curve. This Gaussian
reshaping method (GRM) consists in the multiplication of
$\hat{E}_{\text{out}}$ with the $H$ filter:

$H(\omega)=\sum_{n}\gamma_{n}^{-1}G_{\Delta}(\omega)\delta_{n}(\omega)\
\text{with}\
G_{\Delta}(\omega)=e^{-\frac{(\omega-\omega_{0})^{2}}{(\Delta\omega_{m})^{2}}}$
(12)

Modern pulse shapers reach dynamic attenuation ranges superior to 30 dB,
however, this finite attenuation may still alter the desired complex SPD
filter $F=|H^{2}|\cdot\text{arg}(H)$. In the following, special care is taken
to emulate a realistic device: first, only the spectral modes with a power
larger than 1% of the maximum are reshaped, to ensure that the transmitted
power remains sizable. The selected lines are reshaped with a dynamic range of
35 dB attenuation, while non-selected lines are put to 0. $F$ is then scaled
to 1 in magnitude so that no artificial gain may be introduced. Finally, all
lines of $|F|$ lower than 35 dB are supposed rerouted in the device and given
an 50 dB attenuation.

Let us emphasize that the GRM, contrary to a simple Gaussian filtering method
(GFM), completely cancels the ripples of the spectrum and gives a real-valued
Gaussian curve truncated at $\pm\beta$ values. The range of $\Delta$ is
experimentally limited by $\beta$: when $\Delta/\beta\ll 1$, a large portion
of the EOML spectrum is filtered out, hence leading to much longer pulses. On
the contrary for increasing $\Delta/\beta$, the resulting spectrum more and
more resembles a square function of width $2\beta$. The main pulse is
shortened but side lobes scaled as $\text{sinc}^{2}(\pi t/2\beta T_{m})$ lead
to a degradation of the extinction ratio. The range
$0.5\leqslant\Delta/\beta\leqslant 1.5$ seems typically advisable. For the
lower values of this interval, one may of course expect a non-negligible
increase in pulse duration (typ. a factor 2 for $\Delta/\beta=0.5$).

In order to assess the interest of GRM versus GFM in terms of peak power, we
normalize all time profiles by their energy, defined here as the integral
value of the intensity over one period $T_{m}$. This ensures a proper
comparison between the different cases without any dependence on the severity
of the filter. Experimentally, this would for instance correspond to a setup
where the EOML is launched after pulse shaping in a saturated amplifier. We
hereby name this operation "energy-normalization".

The case $\Delta/\beta=1.5$ is illustrated in Fig. 4 and presents an
interesting feature: the peak intensity after GRM is much higher and the pulse
duration unchanged. GRM hence increases the energy in the main peak by a
sizable amount of 75%. In comparison, GFM only increases the pulse energy by
14%.

Figure 4:  Intensity profiles of $I_{2}$ (reference, dashed), $I_{2}$ after
GFM (blue), or GRM (red). Both filters are computed with $\Delta/\beta=1.5$.
All profiles are energy-normalized and scaled so that the reference has a
magnitude of 1.

The potential of GRM in terms of extinction ratio is demonstrated in Fig. 5.
In the worst case ($\Delta/\beta=1.5$), an enhancement over $I_{\text{FL}}$ of
more than one order of magnitude is realized. For $\Delta/\beta=0.5$, the
extinction ratio drops between $10^{-6}$ (half-period after the pulse) and
$3\times 10^{-3}$ (first satellite peak), values that are comparable to those
of fiber mode-locked lasers [14].

Figure 5: Comparison of $I_{2}$ (reference, dashed), GFM (blue) and GRM (red)
profiles (all scaled to 1 in magnitude).

As many studies have already proven it [15, 16, 17, 18], it is possible to
multiply the repetition rate of a periodic optical signal with an integer
factor by carrying out phase-only line-by-line shaping. This remarkable
phenomenon is called the temporal Talbot effect. Starting from a periodic
pulsed signal with a flat spectral phase, adding to each mode $n$ the
quadratic phase term $\psi_{T}(n)=\pi n^{2}/q\ (q\in\mathbb{N})$, multiplies
the repetition rate by $q$.

If the rate-multiplication has already been achieved optically with a pulse
shaper [19], it has never, to the best of our knowledge, been tried on an
EOML. Indeed, we have seen above (cf Fig. 3) how its textured spectrum
provokes a slow time decrease of the intensity, precluding large
multiplication factors. Temporal self-imaging is also limited by the bandwidth
[15]: when $q\geqslant\beta$, the pedestals of the pulses overlap, leading to
important pulse-to-pulse variations and a decrease of the extinction ratio.
With a non-negligible residual phase, this defects happen for even smaller $q$
values, as small variations about $\psi_{T}$ induce sizable artefacts
temporally [19].

This perturbation is illustrated for EOMLs in Fig. 6, which shows the
intensity obtained when applying a Talbot phase $\psi_{T}$ with $q=20$ on
different EOML spectra. The reference case (Fig. 6a) presents the profile
given by $E_{2}$, as would be observed with only a quadratic compression. For
Fig. 6b, a GFM with $\Delta/\beta=0.85$ is applied on $\hat{E}_{2}$ before
applying $\psi_{T}$. Finally (Fig. 6c), one carries out the GRM of same width
$\Delta$ as b).

In all cases, the rate multiplication is evident, with 20 peaks in one period
$T_{m}$, but the reference signal shows a clear degradation. The GFM improves
partly the peak-to-peak stability and the extinction ratio. The GRM cures
these defects almost completely, with a peak-to-peak variation of 7% and an
extinction ratio about $5\times 10^{-3}$. By summing the energy dispatched in
the gray time-windows of Fig. 6, we can establish a total energy increase in
the peaks superior to 40% for the case of GRM, compared to the reference case
a). This is the consequence of the temporal redistribution of the energy by
proper spectral reshaping.

Figure 6: Talbot rate-multiplication ($q=20$) obtained in three different
cases: a) $\hat{E}_{2}$, b) GFM, c) GRM. b) and c) are computed with
$\Delta/\beta=0.85$; all profiles are energy-normalized, and scaled so that a)
has a magnitude of 1.

To conclude, we have derived the general expression of the complex spectrum of
EOMLs and characterized their time contrast, which is many orders of magnitude
above that of mode-locked lasers. We then showed how pulse shapers could be
put to interesting and optimal use: first, the exact knowledge of the spectrum
may improve the phase-fitting procedure. Second, the Gaussian reshaping method
may greatly improve both the peak power and the time contrast of EOMLs.
Finally, the GRM coupled to Talbot effect enables large rate-multiplication
factors.

As a prospective, we note that the presented results allow to envision very
agile sources with about 100 fs pulse duration and a very wide range of
repetition rate. Considering that modulators at 1500 nm may be driven at 80
GHz and, properly cascaded [9], can generate spectra typically up to
$\beta=50\pi$, EOMLs implemented with the described method would have a
repetition rate adjustable between 80 GHz and 5 THz ($q=63<\beta/2$). This new
class of laser sources should have interesting applications in diverse fields
such as telemetry, spectroscopy or THz generation.

## Acknowledgments

The authors wish to thank Prof. J.-C. Delagnes and Dr. G. Duchateau for
insightful discussions.

## Disclosures

The authors declare no conflicts of interest. This research did not receive
any specific grant from funding agencies in the public, commercial, or not-
for-profit sectors.

## References

  * [1] T. Kobayashi, H. Yao, K. Amano, Y. Fukushima, A. Morimoto, and T. Sueta, IEEE Journal of Quantum Electronics 24, 382 (1988).
  * [2] A. J. Metcalf, V. Torres-Company, D. E. Leaird, and A. M. Weiner, IEEE Journal of Selected Topics in Quantum Electronics 19, 231 (2013).
  * [3] H. Murata, A. Morimoto, T. Kobayashi, and S. Yamamoto, IEEE Journal of Selected Topics in Quantum Electronics 6, 1325 (2000).
  * [4] A. Ishizawa, T. Nishikawa, A. Mizutori, H. Takara, H. Nakano, T. Sogawa, A. Takada, and M. Koga, Optics express 19, 22402 (2011).
  * [5] A. Ishizawa, T. Nishikawa, A. Mizutori, H. Takara, A. Takada, T. Sogawa, and M. Koga, Optics Express 21, 29186 (2013).
  * [6] R. Wu, V. R. Supradeepa, C. M. Long, D. E. Leaird, and A. M. Weiner, Optics letters 35, 3234 (2010).
  * [7] K. Wang and C. Xu, Optics letters 36, 942 (2011).
  * [8] A. Aubourg, J. Lhermite, S. Hocquet, E. Cormier, and G. Santarelli, Opt. Lett. 40, 5610 (2015).
  * [9] A. J. Metcalf, C. D. Fredrick, R. C. Terrien, S. B. Papp, and S. A. Diddams, Optics Letters 44, 2673 (2019).
  * [10] V. Torres-Company and A. M. Weiner, Laser & Photonics Reviews 8, 368 (2014).
  * [11] M. A. F. Roelens, S. Frisken, J. A. Bolger, D. Abakoumov, G. Baxter, S. Poole, and B. J. Eggleton, J. Lightwave Technol. 26, 73 (2008).
  * [12] D. Marion, J. Lhermite, L. Pontagnier, A. Aubourg, P. Héricourt, G. Santarelli, and E. Cormier, “1 to 18 ghz tunable intra-burst repetition rate high-power picosecond fiber laser for ultrafast material processing,” in _Advanced Solid State Lasers,_ (Optical Society of America, 2018), pp. ATh5A–5.
  * [13] J. Lhermite, D. Marion, L. Pontagnier, P. Héricourt, G. Santarelli, and E. Cormier, “Millijoule level laser system delivering 1 to 18 ghz intra-train repetition rate picosecond pulses for materials processing,” in _The European Conference on Lasers and Electro-Optics,_ (Optical Society of America, 2019), p. cm_7_1.
  * [14] N. Stuart, T. Robinson, D. Hillier, N. Hopps, B. Parry, I. Musgrave, G. Nersisyan, A. Sharba, M. Zepf, and R. A. Smith, Opt. Lett. 41, 3221 (2016).
  * [15] H. G. De Chatellus, E. Lacot, W. Glastre, O. Jacquin, and O. Hugon, Physical Review A 88, 033828 (2013).
  * [16] J. A. Bolger, P. Hu, J. T. Mok, J. L. Blows, and B. J. Eggleton, Optics communications 249, 431 (2005).
  * [17] T. Jannson and J. Jannson, JOSA 71, 1373 (1981).
  * [18] J. Azaña, Electronics Letters 40, 449 (2004).
  * [19] J. Caraquitena, Z. Jiang, D. E. Leaird, and A. M. Weiner, Opt. Lett. 32, 716 (2007).