# Beyond Metrics: A Critical Analysis of the Variability in Large Language
Model Evaluation Frameworks

Marco AF Pimentel, Clément Christophe, Tathagata Raha, Prateek Munjal,
Praveen K Kanithi, Shadab Khan

M42
Abu Dhabi, UAE
{mpimentel, cchristophe, traha, pmunjal, pkanithi<EMAIL_ADDRESS>

###### Abstract

As large language models (LLMs) continue to evolve, the need for robust and
standardized evaluation benchmarks becomes paramount. Evaluating the
performance of these models is a complex challenge that requires careful
consideration of various linguistic tasks, model architectures, and
benchmarking methodologies. In recent years, various frameworks have emerged
as noteworthy contributions to the field, offering comprehensive evaluation
tests and benchmarks for assessing the capabilities of LLMs across diverse
domains. This paper provides an exploration and critical analysis of some of
these evaluation methodologies, shedding light on their strengths,
limitations, and impact on advancing the state-of-the-art in natural language
processing.

Figure 1: Performance disparities ($\delta$) of Llama2 (7B, 13B and 70B) and
Mistral-7B on various benchmark datasets (in 0-shot setting). $\delta$ values
are shown using lighter colors and they represent the variation observed in
the accuracy metric across the benchmarks due to different evaluation
frameworks.

##  1 Introduction

The rapid advancements in the field of natural language processing (NLP) have
been fueled by the development of increasingly sophisticated Large Language
Models (LLMs). From OpenAI’s GPT series [1] and Google’s BARD [2, 3] to a
plethora of open-source models such as Llama [4, 5], the Falcon series [6],
Jais [7] and ohers [8, 9, 10], these models showcase remarkable language
understanding and generation capabilities. As the abilities of LLMs continue
to expand, so does the demand for robust evaluation frameworks that can assess
their linguistic aptitude and generalization capabilities [11, 12, 13]. A
thorough evaluation is essential not only to benchmark the progress in model
development but also to identify and address potential biases, handle ethical
concerns, and ensure the responsible deployment of these models in real-world
applications [14, 15, 16, 17, 18].

However, evaluating LLMs poses a unique set of challenges due to their
parameter space, diverse applications, and sensitivity to the choice of
evaluation tasks. Traditional metrics often fall short in capturing the
nuances of language understanding, fluency, and context coherence exhibited by
these models [13, 19]. In addition, model performance is often affected by
minor implementation details. It is exceedingly challenging to anticipate that
results obtained from one codebase will seamlessly transfer to another [20].
Frequently, research papers fail to provide the necessary code and/or
sufficient details to replicate their evaluations fully. In response to this,
the research community has witnessed the emergence of multiple evaluation
frameworks for tracking the progress on specific NLP tasks across different
benchmarks and to encourage multi-task learning with diverse datasets. To
compare the performance of different LLMs, various leaderboards have also been
created to rank LLMs according to their performance metrics (or scores) on
existing or new evaluation benchmarks. For example, the GLUE [21] and
SuperGlue [22] benchmarks are widely used by language model developers. In
addition, BIG-bench [23], HELM [19], EleutherAI’s language model evaluation
harness [24] and OpenCompass [25] frameworks have been introduced to study the
capabilities of language models.

As the demand for rigorous evaluation benchmarks intensifies, it becomes
increasingly evident that the metrics employed are pivotal components. While
these benchmarks provide invaluable insights into the capabilities of language
models across diverse tasks, it is important to delve into the intricacies of
metric calculation (or computation) to discern the subtleties of model
behaviour. The significance of this aspect arises from the fact that metrics,
often treated as objective measures, are subject to specific assumptions and
methodologies during their calculation. These assumptions can significantly
influence the interpretation of model performance and, consequently, the
reliability of benchmark results. As the field progresses, the need for
transparent and standardized reporting of metric calculation procedures
becomes paramount to foster reproducibility and comparability across different
studies.

This paper aims to delve into the understanding of different metric
calculation methodologies employed by some of the most popular frameworks,
providing some details on how these nuances can affect the performance of LLMs
and the interpretability of evaluation results. Specifically, we describe an
investigation into the accuracy metrics employed by prominent evaluation
frameworks, detailing the methodology for their calculation. Our focus extends
to evaluating four widely recognized open-source language models across
question-answering datasets, which are comprised of those structured as
multiple-choice scenarios, with the aim of providing a better understanding of
their performance across these tasks.

##  2 Background

In this investigation, we focus on multiple-choice question tests, in which
for each question only one of the provided choices is the correct answer (see
Box 1). This represents a rather simple task, in contrast to open-ended
questions, for example. However, our examination reveals that even within this
seemingly simple problem formulation, significant variability exists, stemming
from subtle implementation details and disparities. Notably, the majority of
these disparities arise from differences in the methods used to select the
final answer option from the model’s output. To help us formulate the problem,
we focus on evaluating multiple-choice tasks using autoregressive language
models, such as GPT (Generative Pre-trained Transformer) like architectures,
which underpins the architecture of the models mentioned above.

Box 1: Example multiple-choice question with answer. One of the reasons that
the government discourages and regulates monopolies is that Choices: A.
producer surplus is lost and consumer surplus is gained. B. monopoly prices
ensure productive efficiency but cost society allocative efficiency. C.
monopoly firms do not engage in significant research and development. D.
consumer surplus is lost with higher prices and lower levels of output.
Correct answer: D

We frame mathematically the evaluation of multiple-choice questions (MCQs)
using LLMs as follows. Let $Q$ be the set of multiple-choice questions, each
denoted as $q_{i}$, where $i$ ranges from 1 to $N$, where $N$ represents the
total number of questions. For each question $q_{i}$, there exists a set of
answer choices $A_{i}=\\{a_{i1},a_{i2},...,a_{ik}\\}$, where $k$ is the number
of choices for question $q_{i}$, with $2\leq k\leq 5$. The goal is to assess
the model’s performance in selecting the correct answer from the provided
choices. Let $c_{i}$ be the correct answer for question $q_{i}$, where
$c_{i}\in A_{i}$ . The model’s predicted answer for $q_{i}$ is denoted as
$\hat{c}_{i}$. We note that ideally $\hat{c}_{i}\in A_{i}$, but it may not
always be the case; i.e., the model may generate an answer that is not
necessarily part of the set of choices $A_{i}$. From this point onward, to
simplify further our notation, we refer to a single question of a dataset as
$q$ and set of answer choices for that question as
$A=\\{a^{1},a^{2},...,a^{k}\\}$; i.e., we ommit the index $i$.

LLMs operate by accepting a textual input, i.e., a question/instruction (also
called a ’prompt’) which is segmented into tokens (typically, sub-words). Let
the input prompt be represented as $q_{0:m}$, where $m+1$ is the number of
tokens of the input prompt. From this tokenized input, the language model
generates a conditional probability distribution $P(q_{m+1}|q_{0:m})$ over the
vocabulary of tokens of the model for predicting the next token. This allows
LLMs to estimate the likelihood of any token as continuation of the input
prompt. By appending the selected token to the prompt and iterating this
process, the model generates subsequent tokens, enabling the creation of
entire sentences as continuations of the initial input prompt. Let
$q_{m+1:n_{k}}$ be the $k$-th possible continuation sequence generated by the
LLM, with $n_{k}-m$ tokens.

Hence, for model evaluation, two primary approaches emerge:

  * •

Token probability comparison: this involves obtaining the probabilities
$P(q_{m+1}|q_{0:m})$ for given sets of tokens and comparing these
probabilities for the predefined choices ($A$); i.e., the evaluation metric
can be constructed by assessing the relative likelihoods of various token
groups as continuations of the prompt.

  * •

Text generation comparison: one can obtain a text generation from the model
using the iterative process described above and then compare the generated
text to the various predefined possible choices; this provides a holistic
assessment of the model’s ability to generate coherent and contextually
relevant continuations.

Both token-level assessments and holistic text generation analyses underscore
popular evaluation frameworks. In the following sections, we detail three of
these frameworks and their implementations: OpenCompass, EleutherAI’s language
model evaluation harness, and Stanford University’s HELM.

###  2.1 OpenCompass

OpenCompass, an open-source evaluation framework for language
models111OpenCompass’ evaluation framework also supports multimodal models,
which is not within the scope of this investigation., uses the token
probability comparison approach for extracting the prediction of the model
[25]. Specifically, the probabilities predicted by the model for all possible
answers are compared, such that the probability for the option $a^{k}$ is
given by $P(q_{m+1}|q_{0:m})$; in the example above
$a^{k}\in\\{``A",``B",``C",``D"\\}$, i.e., the first letter corresponding to
each choice. We note that the framework uses perplexity as the key metric,
rather than relying solely on (log) likelihood.

In order to reduce the likelihood of the model generating responses that fall
outside the intended range of answers, a “few shots” approach is typically
used, in which the prompt is augmented with one or more instances/examples
with their correct answers as well (see Appendix B). In instances where the
model might have otherwise generated an unrelated word (or token), the
inclusion of a few shots attempts to ensure that the model is guided by known
examples and by better understanding of the expected behaviour. The approach
of incorporating a handful of examples in the prompt typically improves the
model’s performance and is a standard evaluation method, as demonstrated in
assessments such as MMLU, where five shots (i.e., five examples) are prepended
to each prompt for evaluation across various benchmarks [26].

###  2.2 LM Evaluation Harness

The evaluation harness framework from EleutherAI [24] also uses the token
probability comparison methodology. In this case, however, it computes the
likelihood of the entire continuation sequence using the process described
above (i.e., it uses the full answer sequence which contains the letter
followed by the text of the answer). In its simplest method, for each $k$
choice in $A$, the likelihood is determined using

$\sum_{j=m+1}^{n_{k}}\log P(q_{j}|q_{0:j-1}),$

where the aggregation of the probabilities is achieved by summing the log of
the individual probabilities for numerical stability222This approach is used
in [24] in all multiple choice tasks and tagged as “acc”.. Conceptually, this
entails determining the probability of a generated sequence, sampled from the
given prompt, incorporating the specific continuation (or choice) under
consideration (in this case, of the entire answer). The few-shot prompt is
generally similar to the approach described in the previous section. Although
straightforward, this method does not take into account (possible) differences
in length between the predefined choices that may be substantical (i.e.,
$n_{k}$ can vary substantially for each $k$). This can bias the model toward
favoring longer choices as they tend to have higher log probabilities (or
likelihoods).

In order to tackle this problem, the framework’s authors introduced a
normalization step in which the overall likelihood is divided by a measure of
the length of the answer sequence. This can be accomplished in two ways:
either using token-length normalization or byte-length normalization. In the
former, the normalized likelihood of the $k$-th option is determined using the
average log probability per token:

$\sum_{j=m+1}^{n_{k}}\log P(q_{j}|q_{0:j-1})/(n_{k}-m).$

However, the authors noted that this approach is not tokenization agnostic,
which means that two models with distinct tokenization procedures (and/or
vocabulary sizes), despite assigning the same log likelihood to every single
input string, may yield different token-length normalized log likelihoods.

The byte-length normalization approach attempts to normalize the likelihood by
computing the average log probability per character, which ensures that it is
tokenization agnostic333This approach is also used in [24] in all multiple
choice tasks and is tagged as “acc_norm”.. In this case, the normalized
likelihood of the $k$-th option is determined using

$\sum_{j=m+1}^{n_{k}}\log P(q_{j}|q_{0:j-1})/\sum_{j=m+1}^{n_{k}}L_{q_{j}},$

where $L_{q_{j}}$ is the number of bytes represented by the token $q_{j}$. In
practice, rather than using the number of bytes, the number of characters in
$a^{k}$ is used for normalization.

###  2.3 HELM

Another popular framework is the Holistic Evaluation of Language Models (or
HELM) project [19]. The method used for evaluating the model using MCQs in
HELM is different from the implementations described in the previous two
approaches. Instead of comparing the token probabilities from the given answer
choices, HELM utilizes the model’s next token output probabilities to generate
text. This generated text is then compared to the expected answer.
Specifically for MCQ tasks, HELM implements metric functions such as exact
match to assess the correspondence between the generated text and the correct
answer. This approach allows for a more natural evaluation of the model’s
understanding and ability to generate relevant responses, rather than simply
selecting the most probable option from a predefined set of choices.

While the evaluation methodology takes a distinct approach, the few-shot
prompt remains generally similar. However, for a given instance, we note that
if the model assigns the highest probability to a token that deviates from the
intended range of answers, even though it is not part of the set of choices,
the model’s response would be deemed incorrect, resulting in a lack of scoring
for that particular instance. In other words, the evaluation process hinges on
the model’s ability to prioritize the correct tokens within the specified
answer choices. In the event that the model, despite a generally similar few-
shot prompt, allocates the highest probability to a token outside the intended
range, the response in deemed incorrect. This is substantially different from
the methods described above, in which only the probabilities associated with
the given set of answers are included for computing the model’s performance.

We also note that frameworks such as HELM [19] and Langtest [27] attempt to
enhance the evaluation process of LLMs by offering a broader array of tasks
and metrics (in addition to accuracy). These frameworks offer a comprehensive
set of evaluation criteria that delves into various aspects of language
understanding and generation, providing additional metrics such as
calibration, robustness, fairness, bias, toxicity, and efficiency. Such
metrics are out of the scope of this study.

##  3 Methods

In this study, we focus on two prominent evaluation frameworks, OpenCompass
and Eval harness. As described above, these frameworks utilize a token-
probability comparison method to assess LLMs in the context of multiple-choice
question answering benchmarks. Our investigation centers on providing a
detailed account of the accuracy metrics obtained through these frameworks,
with particular attention to four widely recognized LLMs, for quantifying the
performance of such models. We analyse the results across four popular
benchmarks, with the goal of offering a better understanding of the
performance and variability in performance of the selected models within the
defined evaluation paradigms.

###  3.1 Evaluation tasks and datasets

In this section, we describe the test datasets used to evaluate the models on
different tasks. When selecting these datasets, preference was given to widely
acknowledged sources used across various domains, such as general and medical
contexts. Additionally, these encompass a range of context lengths and types
of reasoning. Table 1 includes a summary of the datasets used in this study.

Table 1: Question-answering evaluation datasets. Dataset name |  Count |  Dataset description
---|---|---
HellaSwag |  10,042 |  Common-sense reasoning about the physical world
MedQA |  1,273 |  General medical knowledge from medical board exams
MMLU |  14,042 |  Knowledge and problem-solving tasks covering various topics
OpenBookQA |  500 |  Multi-step, common-sense reasoning and text comprehension

HellaSwag. This dataset was introduced in 2019 to test commonsense natural
language inference about physical situations [28]. HellaSwag444”HellaSwag” is
short for Harder Endings, Longer contexts, and Low-shot Activities for
Situations With Adversarial Generations. employed "Adversarial Filtering", in
which the idea is to (machine) generate challenging incorrect answers for a
multi-choice test setting (see Box A1). These incorrect answers are dubbed
’adversarial endings’. Though humans score above 95.6% on HellaSwag, initial
state-of-the-art models struggled (with accuracies lower than 48% back in
2019).

MedQA. We included a domain-specific dataset, MedQA, originally designed for
addressing medical problems [29]. The dataset comprises free-form multiple-
choice question-answers which were sourced from professional medical board
exams (Box A2).

MMLU. The Measuring Multitask Language Understanding (MMLU) benchmark [26]
aimed to introduce a comprehensive assessment of LLMs across 57 subjects,
including elementary mathematics, US history, computer science, law, and
others (Box A3).

OpenBookQA. Mihaylov et al. (2018) [30] presented a question-answering dataset
that attempts to probe a deeper understanding of a topic (Box A4). Modeled
after open book exams for assessing human understanding of a subject, it
contains questions that require multi-step reasoning, use of additional common
and commonsense knowledge, and rich text comprehension.

###  3.2 Language model architectures

We employed two distinct family of language models, namely Llama2 [5] and
Mistral [31], to evaluate their performance within the context of the
OpenCompass and Eval harness benchmarks. These pretrained generative text
models have been shown to perform well across various benchmarks.

Llama2 models, part of the family of language models developed by Meta AI, are
a set of LLMs with varying size, ranging from 7 billion to 70 billion
parameters. It is an auto-regressive language model based on the transformer
decoder architecture with some notable differences from models like GPT-3. For
example, Llama2 employs the SwiGLU activation function rather than ReLU and
uses rotary positional embeddings in lieu of absolute learnable positional
embeddings [5]. The latest release of Llama2 also introduces architectural
refinements geared towards enhanced performance, extending the context length
to up to 4,096 tokens. Bigger models (70B) use Grouped-Query Attention (GQA)
to better leverage long sequences and improved inference scalability.

Mistral-7B (v0.1), introduced by Mistral AI, is a 7.3-billion parameter model
with a similar architecture to that of Llama2 [31]. It also uses grouped-query
attention which enhances the inference process by caching key and value
vectors for previously decoded tokens in the sequence, thereby reducing
processing time. In addition, it uses a sliding window-based attention
mechanism which replaces full attention, characterized by square compute cost.
In this mechanism, each token can attend to at most 4,096 tokens from the
preceding layer, resulting in a linear compute cost. This implementation
enhances Mistral-7B’s capability to handle long sequences (upto 32k), allowing
higher layers to access historical information beyond the 4,096-token context
window size.

In this investigation, from the family of Llama2 models we use the 7-billion
(Llama2-7B), 13-billion (Llama2-13B) and 70-billion-parameter models
(Llama2-70B) along with Mistral-7B.

###  3.3 Evaluation metrics

The assessment of the three models on the aforementioned datasets involves the
utilization of the accuracy metrics derived from OpenCompass and Eval harness.
The final accuracy for a dataset is determined by the percentage of questions
answered correctly. To determine this accuracy, we can follow different
methods to identify the correct option selected by a model (as described in
the previous section). Our focus centers on the accuracy metrics computed
using the different methods employed by different evaluation frameworks. We
attend to the following methods for this purpose:

  * •

OpenCompass’ accuracy (denoted OC accuracy): this approach invoves extracting
the model’s prediction by assessing the next token probabilities and
determining whether the selected choice (that with the highest likelihood) is
the correct answer (as described above and used in [25]);

  * •

Raw (unnormalized) accuracy (Raw accuracy): similarly to the previous method,
it involves comparing token probabilities, but in this case, the probabilities
of the full answers’ sequences are used to determine the correct option, using
the sum of the log likelihoods of all tokens; this corresponds to the method
used and reported in the eval harness’ framework (see section 2.2);

  * •

Token-normalized accuracy (T-norm accuracy): this method involves normalizing
the likelihhood for each answer (as obtained for the metric above) by dividing
the sum by the number of tokens to avoid giving too much "weight" to longer
answers (section 2.2);

  * •

Byte-normalized accuracy (B-norm accuracy): this method also attempts to avoid
biasing the model toward favoring longer choices by normalizing the likelihood
using the average log probability per character (using the number of
characters in the full answer sequence); as mentioned in section 2.2, it is
also used and reported in the eval harness’ framework.

The evaluation is carried out in a zero-shot setting, meaning that no examples
are added to the prompt. Also, to ensure consistency and mitigate the impact
of variations in prompts on model results, a standardized prompt design was
employed across the evaluation frameworks and methods. Refer to Appendix B for
further details on the adopted prompt design for each dataset.

##  4 Results

The performance of the models across selected benchmark datasets is summarized
in the Table 2, highlighting the accuracy metrics obtained within each one of
the four evaluation methods.

Table 2: 0-shot performance comparison of the different models on the selected benchmark datasets. Accuracy results (in %) are shown according to each of the four evaluation methods used for determining the correct answer. |  | HellaSwag | MedQA | MMLU | OpenBookQA
---|---|---|---|---|---
OC accuracy | Mistral-7B | 78.9 | 24.0 | 61.8 | 64.6
Llama2-7B | 74.0 | 23.3 | 42.6 | 32.0
Llama2-13B | 77.5 | 22.9 | 53.1 | 42.6
Llama2-70B | 82.3 | 25.1 | 66.5 | 61.2
Raw accuracy | Mistral-7B | 61.2 | 42.9 | 58.2 | 54.0
Llama2-7B | 57.1 | 28.4 | 38.2 | 28.4
Llama2-13B | 60.1 | 34.5 | 51.0 | 47.5
Llama2-70B | 64.8 | 50.8 | 65.8 | 65.4
T-norm accuracy | Mistral-7B | 80.0 | 39.8 | 54.1 | 56.2
Llama2-7B | 74.9 | 23.8 | 37.7 | 37.6
Llama2-13B | 78.3 | 30.8 | 48.3 | 47.8
Llama2-70B | 82.8 | 47.1 | 62.8 | 63.6
B-norm accuracy | Mistral-7B | 81.0 | 38.3 | 52.5 | 57.0
Llama2-7B | 76.0 | 25.4 | 36.5 | 37.8
Llama2-13B | 79.5 | 30.6 | 47.1 | 49.0
Llama2-70B | 83.8 | 47.0 | 61.4 | 60.6

For each evaluation scenario, we note that Llama2-70B consistently outperforms
other Llama2 (smaller) variants and Mistral-7B across all benchmark datasets.
However, within each benchmark dataset, a substantial variability in the
performance of the different models across the four methods is observed
(Figure 1). As examples, for MMLU, the performance of Mistral-7B ranges from
61.4% and 65.8%; while for HellaSwag, the performance of Llama2-70B fluctuates
between 64.8% and 83.8%.

We also note that the effect of the normalization methods (e.g., B-norm
accuracy) is not consistent across the benchmark datasets; while for
HellaSwag, the normalization-based accuracy metrics are higher than the raw-
based accuracy metric, which does not take into account the normalization of
the log likelihoods of the responses (according to their length), the opposite
behaviour is observed for other datasets (Table 2). To delve deeper into the
factors influencing correct option selection, we investigate the impact of
normalizing response likelihoods. For each question-answer pair, we assessed
the length of options and compared the length of the correct option with that
of the wrong options (this is set to be the median length of the wrong
options).

The Bland-Altman plot (depicted in black in Figure 2) illustrates the length
difference between right and wrong options for the (whole) HellaSwag dataset.
No significant inherent bias in the length of the correct options compared to
the length of the wrong options is observed (and the same is observed for the
other datasets too; figure not included in the manuscript). I.e., the lengths
of correct and wrong answer options of the records included in the benchmark
dataset are similar. Notably, the figure also shows the length difference (in
red) for those instances in which Mistral-7B incorrectly selected the wrong
option according to the unnormalized likelihood method (top panel) and B-norm
likelihood method (bottom panel).

Figure 3 shows the results of a similar analysis for the MedQA benchmark
dataset.

Figure 2: Bland-Altman plots (left) and frequency plots or histograms (left)
of the length difference between correct and wrong options for the HellaSwag
benchmark dataset. The length difference for the entire dataset is shown in
black (in both top and bottom panels). In the top panel, the length
differences for the instances in which Mistral-7B incorrectly selected the
wrong option in the unnormalized likelihood method (raw-based accuracy) are
overlayed in red. On the bottom panel, the length differences for the
instances in which Mistral-7B incorrectly selected the wrong option in the
B-norm accuracy method are overlayed in red.

Figure 3: Bland-Altman plots (left) and frequency plots or histograms (left)
of the length difference between correct and wrong options for the MedQA
benchmark dataset. The length difference for the entire dataset is shown in
black (in both top and bottom panels). In the top panel, the length
differences for the instances in which Llama2-70B incorrectly selected the
wrong option in the unnormalized likelihood scenario (raw-based accuracy) are
overlayed in red. On the bottom panel, the length differences for the
instances in which Llama2-70B incorrectly selected the wrong option in the
B-norm accuracy scenario are overlayed in red.

##  5 Discussion

In this paper, we aimed to explore the details of metric calculation
methodologies employed by prominent evaluation frameworks and their
implications on the performance assessment of LLMs and the subsequent
interpretability of evaluation results. Specifically, we scrutinized the
accuracy metrics, describing the intricacies of their calculation
methodologies. Extending our examination beyond theoretical considerations, we
directed our efforts towards the evaluation of widely recognized open-access
language models. This evaluation encompassed different question-answering
datasets, including those structured as multiple-choice scenarios.

The results of our analyses shed light on some critical aspects of the
evaluated LLMs and the associated methodologies. Notably, considering each
method individually, Llama2-70B consistently demostrates superior performance
across all benchmark datasets when compared to both its smaller variants and
Mistral-7B (Table 2). This has been observed in other studies, in which the
performance of Llama2-70B has been deemed to be superior in diverse evaluation
benchmark tasks, including question-answering tasks [5, 31].

Nevertheless, despite this overall superiority, the results from our analysis
demonstrate a substantial variability within each benchmark dataset across the
four evaluation methods (Figure 1). The observed fluctuations in performance,
ranging between 5% and 26%, highlight the sensitivity of the reported accuracy
metrics to the method-specific implementations.

In evaluation frameworks such as eval-harness by Eleuther AI [24], for
calculating accuracy of a model, the likelihood of the (complete) response
(for each option) is determined, and the option with the highest likelihood is
deemed to be the correct one. In order to avoid the introduction of a bias
toward favoring shorter answers, eval-harness has introduced a normalization
step, in which the overall likelihood for each choice is divided by a measure
of the length of the choice’s sequence. These different approaches, as
expected, produce different results.

In addition, Figure 2 highlights the impact of normalization in the selection
bias of the correct option by a model (in red) on HellaSwag. In the top panel
(without normalization of the response’s likelihood), a substantial bias is
evident, with the model preferring shorter answers; whereas, if normalization
of the response’s likelihood (bottom panel) is employed, the bias is
noticeably reduced, underscoring the effect of normalization on enhancing the
reliability of the response selection, and ultimately the model’s accuracy.
However, it is important to note that the impact of normalization on the bias
is not universally consistent across all datasets. In contrast to the observed
mitigation of bias in the HellaSwag dataset, the same analysis for other
datasets reveals that the normalization process seems to introduce a bias
rather than reducing it in the selection of the correct option. This can be
observed, for example, in the MedQA dataset (Figure 3). In this case, the
normalization of response likelihoods appears to contribute to a discernible
bias in the selection of the correct option.

The impact of the responses’ likelihood normalization methods on the accuracy
metrics reveals a lack of consistent behaviour across benchmark datasets.
While for some datasets, the normalization step appears to reduce the bias in
the choice selection and produce better performance results, for other the
normalization step seems to introduce a bias in the choice selection and,
hence, provide less accurate results.

This variation in performance across frameworks highlights the importance of
comprehensive benchmarking. The performance of an LLM is typically a function
of its architecture, the model’s training data and the extent to which it has
been fine-tuned to a particular domain of knowledge; i.e., one model may be
exceedingly good at answering medical questions, but it barely exceeds “Hello
World” in a Python programming test. Notwithstanding this, the results
obtained in this investigation, as also observed in [32], suggest that the
performance of an LLM is also highly dependent on the methodology used and
implementation details employed to evaluate them (even when using the same
benchmark dataset).

To conclude, evaluation frameworks play an important role in the assessment
and enhancement of LLMs. They are pivotal in addressing key challenges related
to performance evaluation, offering valuable insights for model development,
and enhancing transparency. By providing a multi-dimensional perspective on
LLMs, existing frameworks make substantial contributions to the advancement
and comprehension of these intricate AI systems. As the demand for rigorous
evaluation of LLMs intensifies, it becomes imperative not only to scrutinize
the metrics themselves but also to expose the methodologies of their
calculation. The diversity in evaluation frameworks also introduces a
considerable challenge, as each framework may employ distinct metrics and
computation methodologies tailored to its specific objectives. Understanding
the underlying assumptions and approaches during metric calculation is crucial
for interpreting and comparing the results effectively. Variability in metric
definitions across frameworks can introduce ambiguity, making it difficult to
draw meaningful cross-model and cross-study comparisons. Additionally, certain
metrics may inadvertently favor specific model characteristics or exhibit
sensitivity to dataset idiosyncrasies. Consequently, in the absence of
thorough evaluation frameworks, researchers must delve into the nuances of
metric definitions and computation procedures to grasp the full context of
reported results, ensuring an informed understanding of LLM performance. This
shift towards a more transparent and methodologically explicit evaluation
paradigm is pivotal for fostering reproducibility and advancing our collective
understanding of the capabilities and limitations of large language models.

## Acknowledgments

This study has been supported by M42.

## References

  * [1] OpenAI.  GPT-4 technical report, 2023.
  * [2] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.  Palm: Scaling language modeling with pathways, 2022.
  * [3] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence.  Palm-e: An embodied multimodal language model, 2023.
  * [4] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [5] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * [6] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Maitha Alhammadi, Mazzotta Daniele, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo.  The falcon series of language models: Towards open frontier models, 2023.
  * [7] Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, Alham Fikri Aji, Zhiqiang Shen, Zhengzhong Liu, Natalia Vassilieva, Joel Hestness, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Hector Xuguang Ren, Preslav Nakov, Timothy Baldwin, and Eric Xing.  Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models, 2023.
  * [8] Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal.  Pythia: A suite for analyzing large language models across training and scaling, 2023.
  * [9] Nolan Dey, Gurpreet Gosal, Zhiming, Chen, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, and Joel Hestness.  Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster, 2023.
  * [10] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  Journal of Machine Learning Research, 21(140):1–67, 2020.
  * [11] Yew Ken Chia, Pengfei Hong, Lidong Bing, and Soujanya Poria.  Instructeval: Towards holistic evaluation of instruction-tuned large language models, 2023.
  * [12] Joshua Maynez, Priyanka Agrawal, and Sebastian Gehrmann.  Benchmarking large language model capabilities for conditional generation, 2023.
  * [13] Sebastian Gehrmann, Elizabeth Clark, and Thibault Sellam.  Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text, 2022.
  * [14] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [15] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen.  A survey of large language models, 2023.
  * [16] Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen.  Large language models for information retrieval: A survey, 2023.
  * [17] Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Xiangji Huang.  A systematic study and comprehensive evaluation of chatgpt on benchmark datasets, 2023.
  * [18] Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, Lewis Ho, Divya Siddarth, Shahar Avin, Will Hawkins, Been Kim, Iason Gabriel, Vijay Bolina, Jack Clark, Yoshua Bengio, Paul Christiano, and Allan Dafoe.  Model evaluation for extreme risks, 2023.
  * [19] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda.  Holistic evaluation of language models, 2023.
  * [20] Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, Zhiqiang Shen, Xuguang Ren, Roberto Iriondo, Cun Mu, Zhiting Hu, Mark Schulze, Preslav Nakov, Tim Baldwin, and Eric P Xing.  LLM360: Towards fully transparent Open-Source LLMs, December 2023.
  * [21] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In Tal Linzen, Grzegorz Chrupała, and Afra Alishahi, editors, Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium, November 2018. Association for Computational Linguistics.
  * [22] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  SuperGlue: Learning feature matching with graph neural networks.  In CVPR, 2020.
  * [23] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartłomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-López, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoń, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michał Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan A. Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Miłkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2023.
  * [24] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.  A framework for few-shot language model evaluation, September 2021.
  * [25] OpenCompass Contributors.  Opencompass: A universal evaluation platform for foundation models.  https://github.com/open-compass/opencompass, 2023.
  * [26] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  Proceedings of the International Conference on Learning Representations (ICLR), 2021.
  * [27] John Snow Labs Inc.  Langtest: Deliver safe and effective language models, 2023.  Accessed: November, 2023.
  * [28] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  Hellaswag: Can a machine really finish your sentence?  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
  * [29] Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits.  What disease does this patient have? a large-scale open domain question answering dataset from medical exams.  Applied Sciences, 11(14):6421, 2021.
  * [30] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  In EMNLP, 2018.
  * [31] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mistral 7b, 2023.
  * [32] Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han.  Don’t make your llm an evaluation benchmark cheater, 2023.

## Appendix A: Datasets’ examples

An example instance from each one of the four datasets used in this study is
provided below.

Box A1: Example test record from the HellaSwag dataset. A bearded man is seen
speaking to the camera and making several faces. The man Correct answer: \-
then holds up a razor and begins shaving his face. Incorrect answers: \- then
switches off and shows himself via the washer and dryer rolling down a towel
and scrubbing the floor. \- then rubs and wipes down an individual’s face and
leads into another man playing another person’s flute. \- is then seen eating
food on a ladder while still speaking.

Box A2: Example question/answer from the MedQA dataset (the option in bold is
deemed to be the correct answer). A 23-year-old pregnant woman at 22 weeks
gestation presents with burning upon urination. She states it started 1 day
ago and has been worsening despite drinking more water and taking cranberry
extract. She otherwise feels well and is followed by a doctor for her
pregnancy. Her temperature is 97.7°F (36.5°C), blood pressure is 122/77 mmHg,
pulse is 80/min, respirations are 19/min, and oxygen saturation is 98% on room
air. Physical exam is notable for an absence of costovertebral angle
tenderness and a gravid uterus. Which of the following is the best treatment
for this patient? (A) Ampicillin. (B) Ceftriaxone (C) Ciprofloxacin (D)
Doxycycline (E) Nitrofurantoin

Box A3: Example test from MMLU dataset’s Microeconomics task (the option in
bold is deemed to be the correct answer). One of the reasons that the
government discourages and regulates monopolies is that (A) producer surplus
is lost and consumer surplus is gained. (B) monopoly prices ensure productive
efficiency but cost society allocative efficiency. (C) monopoly firms do not
engage in significant research and development. (D) consumer surplus is lost
with higher prices and lower levels of output.

Box A4: Example question/answer from the OpenBookQA dataset (the option in
bold is deemed to be the correct answer). Which of these would let the most
heat travel through? (A) a new pair of jeans. (B) a steel spoon in a
cafeteria. (C) a cotton candy at a store. (D) Doxycycline.

## Appendix B: Input prompts

In order to guarantee similar inputs to the model using either benchmark
framework, we standardized the prompt for each one of the datasets. We show
example prompts for each dataset.

Box B1: Example prompt input and outputs from MMLU dataset’s Microeconomics
task in Eval-harness. ———————————————————————————————————————————— One of the
reasons that the government discourages and regulates monopolies is that (A)
producer surplus is lost and consumer surplus is gained. (B) monopoly prices
ensure productive efficiency but cost society allocative efficiency. (C)
monopoly firms do not engage in significant research and development. (D)
consumer surplus is lost with higher prices and lower levels of output.
Answer: ———————————————————————————————————————————— Possible answers: (A)
producer surplus is lost and consumer surplus is gained. (B) monopoly prices
ensure productive efficiency but cost society allocative efficiency. (C)
monopoly firms do not engage in significant research and development. (D)
consumer surplus is lost with higher prices and lower levels of output.

Box B2: Example prompt input and outputs from MMLU dataset’s Microeconomics
task in OpenCompass. ———————————————————————————————————————————— One of the
reasons that the government discourages and regulates monopolies is that (A)
producer surplus is lost and consumer surplus is gained. (B) monopoly prices
ensure productive efficiency but cost society allocative efficiency. (C)
monopoly firms do not engage in significant research and development. (D)
consumer surplus is lost with higher prices and lower levels of output.
Answer: ———————————————————————————————————————————— Possible answers: A. B.
C. D.


#  On two-frequency quasi-periodic perturbations of systems close to two-
dimensional Hamiltonian ones with a double limit cycle

O. S. Kostromina

(Department of Differential Equations, Mathematical and Numerical Analysis,
Lobachevsky State University of Nizhny Novgorod,
prospekt Gagarina, 23, Nizhny Novgorod, 603950, Russia
<EMAIL_ADDRESS>
######

The problem of the effect of two-frequency quasi-periodic perturbations on
systems close to arbitrary nonlinear two-dimensional Hamiltonian ones is
studied in the case when the corresponding perturbed autonomous systems have a
double limit cycle. Its solution is important both for the theory of
synchronization of nonlinear oscillations and for the theory of bifurcations
of dynamical systems. In the case of commensurability of the natural frequency
of the unperturbed system with frequencies of quasi-periodic perturbation,
resonance occurs. Averaged systems are derived that make it possible to
ascertain the structure of the resonance zone, that is, to describe the
behavior of solutions in the neighborhood of individual resonance levels. The
study of these systems allows determining possible bifurcations arising when
the resonance level deviates from the level of the unperturbed system, which
generates a double limit cycle in a perturbed autonomous system. The
theoretical results obtained are applied in the study of a two-frequency
quasi-periodic perturbed pendulum-type equation and are illustrated by
numerical computations.

MSC2010: 34C15

Keywords: quasi-periodic perturbations, double limit cycle, resonances,
averaged systems.

##  1 Introduction

In recent years there has been considerable interest in the study of multi-
frequency quasi-periodic oscillations [1]–[13]. Such phenomena are widespread
in nature and technology, in almost all areas of physics (radiophysics,
electronics, laser physics, astrophysics, and many others), as well as in
biology, chemistry, and medicine (see the above papers as well as monographs
[14, 15, 16]). Systems with a quasi-periodic dynamics are primarily non-
autonomous system with two or more incommensurable frequencies. However,
physicists have revealed the presence of quasi-periodic oscillations in
autonomous systems (generators of quasi-periodic oscillations) [17, 18, 19].

Problems of the existence of quasi-periodic and almost periodic solutions, as
well as complex dynamics in systems close to Hamiltonian ones, were studied in
[6], [20]–[26]. The papers [20, 21] discussed the existence of quasi-periodic
and almost periodic solutions for the Duffing equation. The questions of the
existence of complex dynamics for the pendulum-type equation and the Duffing
equation under quasi-periodic perturbations were considered in [6], [22]–[26].
More general results (for ordinary differential equations that contain a small
parameter) on the existence of almost periodic solutions and integral
manifolds are presented in [27]. In contrast to these papers, papers [3, 7, 8,
11] are devoted to the problem of the effect of small nonconservative quasi-
periodic perturbations on systems close to arbitrary nonlinear two-dimensional
Hamiltonian ones with limit cycles. Based on research methods periodically
perturbed systems [28, 29], the structure of non-degenerate resonance zones is
revealed, conditions for the existence of quasi-periodic solutions in them are
found, the problem of synchronization of quasi-periodic oscillations is
solved, and the global behavior of solutions is analyzed. The theoretical
results obtained in these studies are illustrated by the example of Duffing-
type equations.

Let us consider the system

$\left\\{\begin{aligned} \dot{x}&=\frac{\partial H(x,y)}{\partial
y}+\varepsilon g(x,y,\omega_{1}t,\omega_{2}t),\\\ \dot{y}&=-\frac{\partial
H(x,y)}{\partial x}+\varepsilon
f(x,y,\omega_{1}t,\omega_{2}t),\end{aligned}\right.$ (1)

where $\varepsilon$ is a small positive parameter, Hamiltonian function H and
the functions $g$, $f$ are sufficiently smooth (analytic) and uniformly
bounded in $x,y$ within some domain $D\subset\mathbb{R}^{2}$ (or
$D\subset\mathbb{R}^{1}\times\mathbb{S}^{1}$) along with partial derivatives
up to the second order. In addition, functions $g$, $f$ are continuous and
quasi-periodic in $t$ uniformly with respect to $(x,y)\in D$ with
incommensurable frequencies $\omega_{1},\omega_{2}$.

Let us assume that the unperturbed system ($\varepsilon=0$) is nonlinear and
has a cell $D_{0}\in D$ filled with closed phase curves $H(x,y)=h$,
$h\in[h_{min},h_{max}]$, and not containing any equilibria and separatrices.
Let us suppose also that the perturbation is non-conservative, which is
equivalent to the condition: $g^{\prime}_{x}+f^{\prime}_{y}\not\equiv 0$.

Passing from variables $x$, $y$ to the action $I$ – angle $\theta$ variables
in $D_{0}$:

$x=X(I,\theta),\ y=Y(I,\theta),$ (2)

where $X$, $Y$ are functions periodic in $\theta$ with period $2\pi$, we
obtain the system of the form

$\left\\{\begin{aligned} \dot{I}&=\varepsilon
F(I,\theta,\theta_{1},\theta_{2}),\\\ \dot{\theta}&=\omega(I)+\varepsilon
G(I,\theta,\theta_{1},\theta_{2}),\\\ \dot{\theta}_{1}&=\omega_{1},\\\
\dot{\theta}_{2}&=\omega_{2},\end{aligned}\right.$ (3)

where $F(I,\theta,\theta_{1},\theta_{2})\equiv
f(X,Y,\theta_{1},\theta_{2})X^{\prime}_{\theta}-g(X,Y,\theta_{1},\theta_{2})Y^{\prime}_{\theta}$,
$G(I,\theta,\theta_{1},\theta_{2})\equiv-f(X,Y,\theta_{1},\theta_{2})X^{\prime}_{I}+g(X,Y,\theta_{1},\theta_{2})Y^{\prime}_{I}.$
Here $\omega(I)$ is the frequency of motion in closed phase curves (natural
frequency of the unperturbed system). We assume that the function $\omega(I)$
is monotonic and does not vanish on the interval
$(I_{min},I_{max})\equiv(I(h_{min}),I(h_{max}))$. The functions $F,G$ are
smooth enough in $I,\theta,\theta_{1},\theta_{2}$ within domain
$[I_{min},I_{max}]\times\mathbb{T}^{3}$, where $\mathbb{T}^{3}$ is a three-
dimensional torus.

It is said that in system (3) there is a resonance if:

$n\omega(I)=m_{1}\omega_{1}+m_{2}\omega_{2},$ (4)

where $n$, $m_{1}$ and $m_{2}$ are coprime integer numbers. We denote the
value of $I$, calculated from (4), by $I_{nm_{1}m_{2}}$. Accordingly, we will
refer to the unperturbed levels $I=I_{nm_{1}m_{2}}$ (closed phase curve
$H(x,y)=h_{nm_{1}m_{2}}$ of the unperturbed system) as the resonance levels.

Along with the system (1), let us consider the corresponding autonomous system

$\left\\{\begin{aligned} \dot{x}&=\frac{\partial H(x,y)}{\partial
y}+\varepsilon g_{0}(x,y),\\\ \dot{y}&=-\frac{\partial H(x,y)}{\partial
x}+\varepsilon f_{0}(x,y),\end{aligned}\right.$ (5)

where

$g_{0}(x,y)=\frac{1}{4\pi^{2}}\int\limits_{0}^{2\pi}\int\limits_{0}^{2\pi}g(x,y,\theta_{1},\theta_{2})d\theta_{1}d\theta_{2},\
f_{0}(x,y)=\frac{1}{4\pi^{2}}\int\limits_{0}^{2\pi}\int\limits_{0}^{2\pi}f(x,y,\theta_{1},\theta_{2})d\theta_{1}d\theta_{2}.$

Suppose that system (5) has a double limit cycle. This means that the
Poincaré–Pontryagin generating function has a double root $h=h_{*}$ (for more
details, see the next section).

The effect of quasi-periodic perturbation for a system of the form (5) with a
structurally stable limit cycle (in this case, the Poincaré–Pontryagin
generating function has a simple root $h=h_{0}$) was studied in [1, 3, 7, 8,
11]. The action of time-periodic perturbation for a system with a double limit
cycle was investigated in [30]. However, this study excluded the case of
parametric perturbations. A Duffing-type equation was considered as an
illustrative example.

The problem of the effect of quasi-periodic perturbations on a perturbed
autonomous system of the form (5), which has a double limit cycle, has not
been previously studied and is considered for the first time in this paper. We
study the rearrangements of phase portraits of the averaged system near the
bifurcation case, when the resonance level $I=I_{nm_{1}m_{2}}$ coincides with
the level $I=I_{*}$ in the neighborhood of which system (5) has a double limit
cycle. The solution to this problem is essential for the theory of
synchronization of oscillations, as well as the theory of bifurcations of
dynamical systems. This study largely follows the paper [30], as well as the
works [3, 7, 11].

As an illustrative example, we consider a pendulum-type equation of the form

$\ddot{x}+\sin{x}=\varepsilon[(-1+p_{1}\cos{3x}+p_{2}x\alpha(t))\dot{x}+p_{3}\alpha(t)],$
(6)

where $p_{1}$, $p_{2}>0$, $p_{3}>0$ are parameters, $\varepsilon$ is a small
positive parameter, $\alpha(t)=\cos{\omega_{1}t}\sin{\omega_{2}t}$. We suppose
that $\omega_{1}$, $\omega_{2}$ are incommensurable. The perturbation contains
a nonlinear parametric term $p_{2}x\dot{x}\alpha(t)$. The case when $p_{2}=0$
and the function $\alpha(t)$ is periodic was studied in [31].

##  2 Poincaré–Pontryagin generating function and double cycles in the
perturbed autonomous system

In cell $D_{0}$, pass to the action $I$ – angle $\theta$ variables in system
(5), writing this transformation in the form (2), and averaging the resulting
system over $\theta$, we obtain the system

$\dot{u}=\varepsilon B_{0}(u),$ (7)

where

$B_{0}(u)=\frac{1}{2\pi}\int\limits_{0}^{2\pi}[f_{0}(X,Y)X^{\prime}_{\theta}-g_{0}(X,Y)Y^{\prime}_{\theta}]\,d\theta.$
(8)

Here, $u=I+O(\varepsilon)$. The function $B_{0}(u)$ is called the
Poincaré–Pontryagin generating function.

It is known [28, 29], if $u=u_{0}$ is a simple equilibrium state
($B_{0}(u_{0})=0$, $B^{\prime}_{0}(u_{0})\neq 0$) of the averaged system (7),
then in system (5) for sufficiently small $\varepsilon$ this equilibrium state
corresponds to a rough limit cycle. Moreover, the limit cycle will be stable
if the equilibrium state is stable, that is, $B^{\prime}(u_{0})<0$, and
unstable otherwise. Let us denote

$B_{1}(u_{0})\equiv
B^{\prime}_{0}(u_{0})=\frac{1}{2\pi}\int\limits^{2\pi}_{0}\left({g_{0}}^{\prime}_{x}+{f_{0}}^{\prime}_{y}\right)_{\left|{\begin{subarray}{c}x=X(u_{0},\theta)\\\
y=Y(u_{0},\theta)\end{subarray}}\right.}d\theta.$

The double root $u=u_{*}$ of the equation $B_{0}(u)=0$ determines the
unperturbed level $I=I_{*}$ (closed phase curve $h=h_{*}$ of the unperturbed
system), from which a double limit cycle is generated under the action of the
perturbation. In this case, $B_{1}(u_{*})=0$, $B^{\prime}_{1}(u_{*})\neq 0$.

When non-autonomous disturbances affect an autonomous system, a limit cycle
may coincide with the resonance level.

##  3 Averaging system in the neighborhood of the individual resonance level

It is known [3, 7], in the neighborhood
$U_{\mu}=\\{(I,\theta):I_{nm_{1}m_{2}}-C\mu<I<I_{nm_{1}m_{2}}+C\mu,\
0\leqslant\theta<2\pi,\ C=\textrm{const}>0\\},\ \mu=\sqrt{\varepsilon}$ of the
individual resonance level $I=I_{nm_{1}m_{2}}$ (it will be called the
resonance zone) the system (2) reduces to an averaged system of the form

$\left\\{\begin{aligned} \dot{u}&=\mu
A(v;I_{nm_{1}m_{2}})+\mu^{2}P_{0}(v;I_{nm_{1}m_{2}})u+O(\mu^{3}),\\\
\dot{v}&=\mu
b_{1}u+\mu^{2}(b_{2}u^{2}+Q_{0}(v;I_{nm_{1}m_{2}}))+O(\mu^{3}),\end{aligned}\right.$
(9)

where

$A(v;I_{nm_{1}m_{2}})=\frac{1}{4\pi^{2}n^{2}}\int\limits_{0}^{2\pi
n}\int\limits_{0}^{2\pi
n}\left.F(I_{nm_{1}m_{2}},v+\frac{m_{1}\theta_{1}+m_{2}\theta_{2}}{n},\theta_{1},\theta_{2})\right.\,d\theta_{1}d\theta_{2},$
$P_{0}(v;I_{nm_{1}m_{2}})=\frac{1}{4\pi^{2}n^{2}}\int\limits_{0}^{2\pi
n}\int\limits_{0}^{2\pi
n}\left.F^{\prime}_{I}(I_{nm_{1}m_{2}},v+\frac{m_{1}\theta_{1}+m_{2}\theta_{2}}{n},\theta_{1},\theta_{2})\right.\,d\theta_{1}d\theta_{2},$
$Q_{0}(v;I_{nm_{1}m_{2}})=\frac{1}{4\pi^{2}n^{2}}\int\limits_{0}^{2\pi
n}\int\limits_{0}^{2\pi
n}\left.G(I_{nm_{1}m_{2}},v+\frac{m_{1}\theta_{1}+m_{2}\theta_{2}}{n},\theta_{1},\theta_{2})\right.\,d\theta_{1}d\theta_{2},$
$b_{1}=\omega^{\prime}(I_{nm_{1}m_{2}}),\
b_{2}=\omega^{\prime\prime}(I_{nm_{1}m_{2}})/2.$

The functions $A(v;I_{nm_{1}m_{2}})$, $P_{0}(v;I_{nm_{1}m_{2}})$ and
$Q_{0}(v;I_{nm_{1}m_{2}})$ are periodic in $v$ with the least period equal to
$2\pi/n$ [3, 7]. Therefore, the phase space of system (9) is the cylinder
$\\{v(mod(2\pi/n)),u\\}$. A simple stable (unstable) equilibrium state of the
system (9) corresponds to a stable (unstable) quasi-periodic resonance
solution with periods $2\pi n/\omega_{1}$, $2\pi n/\omega_{2}$, and two-
dimensional invariant tori with quasi-periodic motion in the original four-
dimensional system.

The functions $A(v;I_{nm_{1}m_{2}})$ and $P_{0}(v;I_{nm_{1}m_{2}})$ can be
represented as

$\begin{array}[]{cc}A(v;I_{nm_{1}m_{2}})=\widetilde{A}(v;I_{nm_{1}m_{2}})+B_{0}(I_{nm_{1}m_{2}}),P_{0}(v;I_{nm_{1}m_{2}})=\widetilde{P}_{0}(v;I_{nm_{1}m_{2}})+B_{1}(I_{nm_{1}m_{2}}),\\\
B_{0}(I_{nm_{1}m_{2}})=\left<A(v;I_{nm_{1}m_{2}})\right>_{v}=\displaystyle\frac{n}{2\pi}\int\limits_{0}^{2\pi/n}A(v;I_{nm_{1}m_{2}})\,dv,\\\
B_{1}(I_{nm_{1}m_{2}})=\left<P_{0}(v;I_{nm_{1}m_{2}})\right>_{v}=\displaystyle\frac{n}{2\pi}\int\limits_{0}^{2\pi/n}P_{0}(v;I_{nm_{1}m_{2}})\,dv.\end{array}$

Since we consider the case, when $I_{nm_{1}m_{2}}=I_{*}$, then
$B_{0}(I_{nm_{1}m_{2}})=B_{1}(I_{nm_{1}m_{2}})=0$, and therefore, it is
necessary to take into account the terms $O(\mu^{3})$. Neglecting the terms
$O(\mu^{4})$, we obtain the system

$\left\\{\begin{aligned}
\frac{du}{d\tau}&=\widetilde{A}(v;I_{nm_{1}m_{2}})+B_{0}(I_{nm_{1}m_{2}})+\mu\left[\widetilde{\sigma}(v;I_{nm_{1}m_{2}})+B_{1}(I_{nm_{1}m_{2}})\right]+\mu^{2}\left[\left(\widetilde{P}_{1}(v;I_{nm_{1}m_{2}})\right.\right.+\\\
&+\left.\left.B_{2}(I_{nm_{1}m_{2}})+\frac{b_{2}}{b_{1}}\frac{dQ_{0}(v;I_{nm_{1}m_{2}})}{dv}\right)u^{2}-\widetilde{P}_{0}(v;I_{nm_{1}m_{2}})\frac{Q_{0}(v;I_{nm_{1}m_{2}})}{b_{1}}\right],\\\
\frac{dv}{d\tau}&=b_{1}u+\mu
b_{2}u^{2}+\mu^{2}\left[b_{3}u^{3}+\left(Q_{1}(v;I_{nm_{1}m_{2}})-2\frac{b_{2}}{b_{1}}Q_{0}(v;I_{nm_{1}m_{2}})\right)u\right],\end{aligned}\right.$
(10)

where

$P_{1}(v;I_{nm_{1}m_{2}})=\frac{1}{8\pi^{2}n^{2}}\int\limits_{0}^{2\pi
n}\int\limits_{0}^{2\pi
n}\left.F^{\prime\prime}_{II}(I_{nm_{1}m_{2}},v+\frac{m_{1}\theta_{1}+m_{2}\theta_{2}}{n},\theta_{1},\theta_{2})\right.\,d\theta_{1}d\theta_{2},$
$Q_{1}(v;I_{nm_{1}m_{2}})=\frac{1}{4\pi^{2}n^{2}}\int\limits_{0}^{2\pi
n}\int\limits_{0}^{2\pi
n}\left.G^{\prime}_{I}(I_{nm_{1}m_{2}},v+\frac{m_{1}\theta_{1}+m_{2}\theta_{2}}{n},\theta_{1},\theta_{2})\right.\,d\theta_{1}d\theta_{2},$
$b_{3}=\omega^{\prime\prime\prime}(I_{nm_{1}m_{2}})/6.$

The functions $P_{1}(v;I_{nm_{1}m_{2}})$ can be represented as
$P_{1}(v;I_{nm_{1}m_{2}})=\widetilde{P}_{1}(v;I_{nm_{1}m_{2}})+B_{2}(I_{nm_{1}m_{2}})$,
$B_{2}(I_{nm_{1}m_{2}})=B^{\prime\prime}_{0}(I_{nm_{1}m_{2}})/2$.

Making the change $u\to u-\mu Q_{0}(v,I_{nm_{1}m_{2}})/b_{1}$ in system (10)
and passing to the slow time $\tau=\mu t$, we obtain the system

$\left\\{\begin{aligned}
\frac{du}{d\tau}&=\widetilde{A}(.)+B_{0}(I_{nm_{1}m_{2}})+\mu\left[\widetilde{\sigma}(.)+B_{1}(I_{nm_{1}m_{2}})\right]+\mu^{2}\left[\left(\widetilde{P}_{1}(.)+B_{2}(I_{nm_{1}m_{2}})\right.\right.+\\\
&+\left.\left.\frac{b_{2}}{b_{1}}\frac{dQ_{0}(.)}{dv}\right)u^{2}-\widetilde{P}_{0}(.)\frac{Q_{0}(.)}{b_{1}}\right],\\\
\frac{dv}{d\tau}&=b_{1}u+\mu
b_{2}u^{2}+\mu^{2}\left[b_{3}u^{3}+\left(Q_{1}(.)-2\frac{b_{2}}{b_{1}}Q_{0}(.)\right)u\right],\end{aligned}\right.$
(11)

where $(.)\equiv(v;I_{nm_{1}m_{2}})$,
$\widetilde{\sigma}(v;I_{nm_{1}m_{2}})=\widetilde{P}_{0}(v;I_{nm_{1}m_{2}})+dQ_{0}(v;I_{nm_{1}m_{2}})/dv$.
This system determines the topology of individual resonance zones up to terms
of order $\mu^{3}$. According to [3], a simple stable (unstable) equilibrium
state of the averaged system (11) corresponds to a stable (unstable) quasi-
periodic resonance solution of periods $\displaystyle\frac{2\pi
n}{\omega_{1}}$, $\displaystyle\frac{2\pi n}{\omega_{2}}$ in system (1), or a
two-dimensional stable (unstable) invariant torus in system (3). A rough limit
cycle of the averaged system (11) with frequency $\omega_{0}$ corresponds to a
quasi-periodic resonance solution of periods
$\displaystyle\frac{2\pi}{\omega_{0}}$, $\displaystyle\frac{2\pi
n}{\omega_{1}}$, $\displaystyle\frac{2\pi n}{\omega_{2}}$ in system (1), or a
three-dimensional invariant torus in system (3) of the same stability type as
the cycle.

An important role in the study of the structure of the neighborhood of
resonance levels $I=I_{nm_{1}m_{2}}$ is played by the function
$\sigma(v;I_{nm_{1}m_{2}})=\widetilde{\sigma}(v;I_{nm_{1}m_{2}})+B_{1}(I_{nm_{1}m_{2}})$,
$B_{1}(I_{nm_{1}m_{2}})=\left<\sigma(v;I_{nm_{1}m_{2}})\right>_{v}$. First of
all, it is necessary to find out whether it depends on $v$ or is constant [28,
29].

##  4 Study of the averaged system

Put $B_{0}(I_{nm_{1}m_{2}})=B_{1}(I_{nm_{1}m_{2}})=0$. Then system (11)
without taking into account the conservative term
$2\mu\displaystyle\frac{b_{2}}{b_{1}}\widetilde{A}(v;I_{nm_{1}m_{2}})\dot{v}$
is transformed to the form

$\ddot{v}-b_{1}\widetilde{A}(v;I_{nm_{1}m_{2}})=\mu\widetilde{\sigma}(v;I_{nm_{1}m_{2}})\dot{v}+\mu^{2}\left[M(v;I_{nm_{1}m_{2}})\dot{v}^{2}+N(v;I_{nm_{1}m_{2}})\right],$
(12)

where

$M(v;I_{nm_{1}m_{2}})=\frac{1}{b_{1}}\left[\widetilde{P}_{1}(.)+{Q_{1}}^{\prime}_{v}(.)+\frac{b_{2}}{b_{1}}\left(\widetilde{P}_{0}(.)+\widetilde{\sigma}(.)\right)+3\frac{b_{3}}{b_{1}}\widetilde{A}(.)\right]+\frac{1}{b_{1}}B_{2}(I_{nm_{1}m_{2}}),$
$N(v;I_{nm_{1}m_{2}})=-\widetilde{P}_{0}(.)Q_{0}(.)+\left(Q_{1}(.)-2\frac{b_{2}}{b_{1}}Q_{0}(.)\right)\widetilde{A}(.).$

re $(.)\equiv(v;I_{nm_{1}m_{2}})$.

Let us rewrite equation (12) as

$\left\\{\begin{aligned}
\dot{u}&=\widetilde{A}(v;I_{nm_{1}m_{2}})+\mu\widetilde{\sigma}(v;I_{nm_{1}m_{2}})u+\frac{\mu^{2}}{b_{1}}\left[M(v;I_{nm_{1}m_{2}})b_{1}^{2}u^{2}+N(v;I_{nm_{1}m_{2}})\right],\\\
\dot{v}&=b_{1}u\end{aligned}\right.$ (13)

Let us consider the case, when $\sigma(v;I_{nm_{1}m_{2}})=const$, therefore
$\widetilde{\sigma}=0$. In this case the system (13) is invariant under the
change $u\to-u,\tau\to-\tau$, therefore, the phase portrait of equation (12)
is symmetric about the $u_{2}$ axis. Phase space of system (13) is the
cylinder $\\{v(mod(2\pi/n)),u\\}$. System (13) is close to Hamiltonian with
the Hamiltonian function $\widetilde{H}(v,u)=b_{1}u^{2}/2-V(v)$,
$V(v)=\int\widetilde{A}(v;I_{nm_{1}m_{2}})dv$. Denote by $(v_{0},0)$ the
equilibrium state of the unperturbed system. System (13) has simple
equilibrium states of two types: the center, if
$b_{1}\widetilde{A}^{\prime}_{v}(v_{0})<0$, and the saddle, if
$b_{1}\widetilde{A}^{\prime}_{v}(v_{0})>0$. To establish the relative position
of the separatrices of the saddle $(v_{0},0)$ of the unperturbed system under
the action of the perturbation, we use the Melnikov formula [32]:
$\Delta_{\mu}=\mu^{2}\Delta_{1}+O(\mu^{3})$, which determines (up to terms of
order $\mu^{3}$) the distance between the perturbed separatrices. Making the
substitution $v=w+v_{0}$, moving the equilibrium state $(v_{0},0)$ of the
saddle type of the unperturbed system to the origin, we obtain

$\Delta_{1}=\frac{1}{b_{1}}\int\limits^{\infty}_{-\infty}\left[M(w+v_{0})b_{1}^{2}u^{2}+N(w+v_{0})\right]\frac{dw}{d\tau}d\tau,$
(14)

where $w(\tau)$, $u(\tau)$ is the solution of the unperturbed system on the
separatrix. From the integral of the unperturbed system we find

$u=\pm\sqrt{\frac{2}{b_{1}}\left(V(w,v_{0})-V(0,v_{0})\right)},\
V(w,v_{0})=\int\widetilde{A}(w+v_{0})dw.$ (15)

Substituting (15) into (14), we obtain

$\Delta_{1}=\frac{1}{b_{1}}\int\limits^{2\pi/n}_{0}\left[2b_{1}M(w+v_{0})\left(V(w,v_{0})-V(0,v_{0})\right)+N(w+v_{0})\right]dw.$
(16)

There are two possible cases: $\Delta_{1}=0$ and $\Delta_{1}\neq 0$. In the
first case, we have two separatrix loops of the saddle on the phase cylinder
(on the upper and lower half-cylinders). This is a bifurcation case. In the
second case, the separatrices are split, and the size of the gap depends on
the relationship between the amplitude of the function
$\widetilde{M}(v;I_{nm_{1}m_{2}})=\widetilde{P}_{1}(v;I_{nm_{1}m_{2}})+{Q_{1}}^{\prime}_{v}(v;I_{nm_{1}m_{2}})+\frac{b_{2}}{b_{1}}\widetilde{P}_{0}(v;I_{nm_{1}m_{2}})+3\frac{b_{3}}{b_{1}}\widetilde{A}(v;I_{nm_{1}m_{2}})$
and the value $B_{2}(I_{nm_{1}m_{2}})$. Possible phase portraits of system
(13) on the phase cylinder $\\{v(mod(2\pi/n)),u\\}$ are shown in Fig 1.

|  |
---|---|---
(a) $B_{2}(I_{nm_{1}m_{2}})\neq 0$ | (b) $B_{2}(I_{nm_{1}m_{2}})=0$ | (c) $B_{2}(I_{nm_{1}m_{2}})\neq 0$
Fig 1: Possible phase portraits of system (13) for $\widetilde{\sigma}=0$.

The case of impassable resonance (synchronization of oscillations) is shown in
Fig 1(b): for any initial conditions of the phase space of the averaged
system, the phase curves of the averaged system remain in a limited region of
the phase space. The case of partly passable resonance is shown in Figs 1(a)
and (c): in the phase space of the averaged system, there are sets of initial
conditions (inside the separatrix loop) that correspond to bounded phase
curves, as well as sets of initial conditions that correspond to phase curves
(outside the separatrix loop) that leave any bounded region of the phase space
when $\tau\to\infty$.

Let us consider the case of an alternating function
$\widetilde{\sigma}(v;I_{nm_{1}m_{2}})$. Possible phase portraits of system
(13) on the phase cylinder $\\{v(mod(2\pi/n)),u\\}$ in this case are shown in
Fig 2.

|
---|---
(a) $B_{2}(I_{nm_{1}m_{2}})=0$ | (b) $B_{2}(I_{nm_{1}m_{2}})\neq 0$
|
(c) $B_{2}(I_{nm_{1}m_{2}})\neq 0$ | (d) $B_{2}(I_{nm_{1}m_{2}})\neq 0$
Fig 2: Possible phase portraits of system (13) for an alternating function
$\widetilde{\sigma}(v;I_{nm_{1}m_{2}})$.

The structure of impassable resonance zone is shown in Figs 2(a) and (b): the
phase point tends to a stable equilibrium state at $\tau\to\infty$ for any
initial point (a); for any initial point on the lower half-cylinder and on an
unstable limit cycle located on the upper half-cylinder (b). The bifurcation
case, when the limit cycle merges into the separatrix loop on the upper half-
cylinder, is shown in Fig 2(c). The question of limit cycles can be solved
using the Poincaré–Pontryagin generating function, similarly to how it is done
for a perturbed autonomous system (5). The structure of partly passable
resonance zone is shown in Fig 2(d).

We assign $B_{0}(I_{nm_{1}m_{2}})=\mu\gamma_{1}$,
$B_{1}(I_{nm_{1}m_{2}})=\mu\gamma_{2}$. For nonzero parameters $\gamma_{1}$
and $\gamma_{2}$, possible phase portraits of system (11) for $n=3$ on the
phase cylinder $\\{v(mod(2\pi)),u\\}$ in the case of a constant sign function
$\sigma\neq 0$ are shown in Fig 3. The focus will be on the phase cylinder of
system (11) instead of the center, and there will also be an asymmetry in the
formation of limit cycles on the upper and lower half-cylinders of different
stability (shown in different colors). Therefore, a bifurcation case is
possible when one of the limit cycles (on the upper or lower phase half
cylinder) merges into the corresponding separatrix loop; as the parameter
changes further, it disappears.

|  |
---|---|---
(a) | (b) | (c)
|  |
(d) | (e) |
Fig 3: Possible phase portraits of system (11) for $n=3$ in the case of a constant sign function $\sigma\neq 0$. |  |
---|---|---
(a) | (b) | (c)
|  |
(d) | (e) |
Fig 4: Possible phase portraits of system (11) for $n=3$ in the case of an
alternating function $\sigma$.

In the case of an alternating function $\sigma$, the existence of a limit
cycle in the region of oscillatory motions is possible. Possible phase
portraits of system (11) for $n=3$ on the phase cylinder
$\\{v(mod(2\pi)),u\\}$ in the case of an alternating function $\sigma$ are
shown in Fig 4.

Cases of synchronization of oscillations are shown in Figs 3(c) and 4(c).
Figures 3(a), 3(e), 4(a) and 4(e) show the structures of partly passable
resonance zones. Bifurcation cases are shown in Figs 3(b), 3(d), 4(b) and
4(d).

In [30], the phenomenon of synchronization of oscillations was called
restricted. This is due to the possibility of the existence of a limit cycle
in the region of rotational motion of the pendulum; therefore, not all phase
points in the resonance zone tend to a stable periodic regime, which
corresponds to a stable equilibrium state of the averaged system.

##  5 Example for illustration

Let us consider the equation (6). The autonomous equation ($p_{2}=0$ and
$p_{3}=0$) was studied in the paper [31]. It was found that the
Poincaré–Pontryagin generating function for region of oscillatory motion
(filled with closed phase curves $\dot{x}^{2}/2-\cos{x}=h$, $h\in(-1,1)$) of
the unperturbed pendulum has the form

$\begin{split}B_{0}=B_{0}(k(h))&=\frac{8}{105\pi}\\{(1-k^{2})(105+(128k^{4}-80k^{2}+3)p_{1}){\bf
K}(k)+\\\ &+(-105+(2k^{2}-1)(128k^{4}-128k^{2}+3)p_{1}){\bf
E}(k)\\},\end{split}$ (17)

where ${\bf K}(k)$ and ${\bf E}(k)$ are complete elliptic integrals of the
first and second kind respectively; $k=\sqrt{(1+h)/2}\in(0,1)$ is their
module. In [31], we found the bifurcation value $p_{1}\approx-8.481$, for
which there is a double limit cycle. In this case, the Poincaré–Pontryagin
generating equation $B_{0}(k)=0$ has a double root $k=k_{*}\approx 0.759$.

Now we choose such $\omega_{1}$ and $\omega_{2}$ so that the level $k=k_{*}$
coincides with the resonance level $k=k_{311}$. From the resonance condition
$3\omega(k_{*})=\omega_{1}+\omega_{2}$, where
$\omega(k_{*})=\displaystyle\frac{\pi}{2{\bf K}(k_{*})}$, we find, for
example, $\omega_{1}=1$ and $\omega_{2}\approx 1.448$.

Let us calculate system (11) for our case. This needs to be calculated:

$\widetilde{A}(v)=\frac{1}{36\pi^{2}}\int\limits_{0}^{6\pi}\int\limits_{0}^{6\pi}(p_{2}xy+p_{3})\frac{\partial
x}{\partial\theta}\cos\theta_{1}\sin\theta_{2}\,d\theta_{1}\,d\theta_{2},$
(18)
$\widetilde{P_{0}}(v)=\frac{1}{36\pi^{2}}\int\limits_{0}^{6\pi}\int\limits_{0}^{6\pi}\frac{\partial}{\partial
I}\left((p_{2}xy+p_{3})\frac{\partial
x}{\partial\theta}\right)\cos\theta_{1}\sin\theta_{2}\,d\theta_{1}\,d\theta_{2},$
(19)
$Q_{0}(v)=-\frac{1}{36\pi^{2}}\int\limits_{0}^{6\pi}\int\limits_{0}^{6\pi}[(-1+p_{1}\cos{3x}+p_{2}x\cos\theta_{1}\sin\theta_{2})y+p_{3}\cos\theta_{1}\sin\theta_{2}]\frac{\partial
x}{\partial I}\,d\theta_{1}\,d\theta_{2},$ (20)
$\widetilde{P_{1}}(v)=\frac{1}{72\pi^{2}}\int\limits_{0}^{6\pi}\int\limits_{0}^{6\pi}\frac{\partial^{2}}{\partial
I^{2}}\left((p_{2}xy+p_{3})\frac{\partial
x}{\partial\theta}\right)\cos\theta_{1}\sin\theta_{2}\,d\theta_{1}\,d\theta_{2},$
(21)
$Q_{1}(v)=-\frac{1}{36\pi^{2}}\int\limits_{0}^{6\pi}\int\limits_{0}^{6\pi}\frac{\partial}{\partial
I}\left([(-1+p_{1}\cos{3x}+p_{2}x\cos\theta_{1}\sin\theta_{2})y+p_{3}\cos\theta_{1}\sin\theta_{2}]\frac{\partial
x}{\partial I}\right)\,d\theta_{1}\,d\theta_{2},$ (22)
$B_{1}(k)=\frac{dB_{0}(k)}{dI}=\frac{dB_{0}(k)}{dk}\frac{dk}{dh}\frac{dh}{dI}=\frac{\pi}{8k{\bf
K}(k)}\frac{dB_{0}(k)}{dk},\
B_{2}(k)=\frac{d^{2}B_{0}(k)}{2dI^{2}}=\frac{\pi}{16k{\bf
K}(k)}\frac{dB_{1}(k)}{dk},$ (23)
$\widetilde{\sigma}(v)=\frac{1}{36\pi^{2}}\int\limits_{0}^{6\pi}\int\limits_{0}^{6\pi}p_{2}x\cos\theta_{1}\sin\theta_{2}\,d\theta_{1}\,d\theta_{2},$
(24) $b_{1}=\frac{\pi}{8k{\bf K}(k)}\frac{d\omega(k)}{dk},\
b_{2}=\frac{\pi}{16k{\bf K}(k)}\frac{db_{1}(k)}{dk},\ b_{3}=\frac{\pi}{24k{\bf
K}(k)}\frac{db_{2}(k)}{dk},$ (25)

where $x(k,\theta)=2\arcsin{(k\textrm{s}n(2{\bf K}\theta/\pi,k))}$,
$y(k,\theta)=2k\textrm{c}n(2{\bf K}\theta/\pi,k)$,
$\theta=v+\displaystyle\frac{\theta_{1}+\theta_{2}}{3}$ ($\textrm{s}n(u,k)$ is
the Jacobi elliptic sine, $\textrm{c}n(u,k)$ is the Jacobi elliptic cosine).
Here and below we put $k=k_{*}=k_{311}$.

In calculating integrals (18), (19) and (21), using the well-known Fourier
series expansion for the Jacobi elliptic cosine [33], we obtain

$\displaystyle\frac{\partial
x}{\partial\theta}=\displaystyle\frac{y}{\omega}=\displaystyle\frac{4k{\bf
K}}{\pi}\textrm{c}n(2{\bf
K}\theta/\pi,k)=8\sum_{j=1}^{\infty}\displaystyle\frac{a^{j-1/2}}{1+a^{2j-1}}\cos(2j-1)\theta,$
$\displaystyle\frac{\partial}{\partial I}\left(\displaystyle\frac{\partial
x}{\partial\theta}\right)=\displaystyle\frac{\pi^{3}}{4k^{2}(1-k^{2}){\bf
K}^{3}}\sum_{j=1}^{\infty}\displaystyle\frac{(2j-1)a^{j-1/2}\left(1-a^{2j-1}\right)}{\left(1+a^{2j-1}\right)^{2}}\cos(2j-1)\theta,$
$\begin{split}\frac{\partial^{2}}{\partial I^{2}}\left(\frac{\partial
x}{\partial\theta}\right)&=\frac{\pi^{4}}{32k^{4}(1-k^{2})^{2}{\bf
K}^{5}}\left(((1+k^{2}){\bf K}-3{\bf
E})\sum_{j=1}^{\infty}\frac{(2j-1)a^{j-1/2}\left(1-a^{2j-1}\right)}{\left(1+a^{2j-1}\right)^{2}}\cos(2j-1)\theta\right.+\\\
&+\left.\frac{\pi^{2}}{4{\bf
K}}\sum_{j=1}^{\infty}\frac{(2j-1)^{2}a^{j-1/2}\left(1-6a^{2j-1}+a^{2(2j-1)}\right)}{\left(1+a^{2j-1}\right)^{3}}\cos(2j-1)\theta\right),\end{split}$

where $a=exp\left(-\pi\displaystyle\frac{{\bf K}(\sqrt{1-k^{2}})}{{\bf
K}(k)}\right)$. In calculating integrals (20) and (22), expansions for the
functions $\displaystyle\frac{\partial x}{\partial
I}=\displaystyle\frac{\pi\left(\textrm{s}n(.)\textrm{d}n(.)-\textrm{c}n(.)zn(.)\right)}{4k(1-k^{2}){\bf
K}}$ (here the dot denotes the arguments $(2{\bf K}\theta/\pi,k)$;
$\textrm{d}n(u,k)$ is the delta amplitude, $\textrm{z}n(u,k)$ is the Jacobi
zeta function) and $\displaystyle\frac{\partial}{\partial
I}\left(\displaystyle\frac{\partial x}{\partial I}\right)$ in a Fourier series

$\displaystyle\frac{\partial x}{\partial
I}=\displaystyle\frac{\pi^{3}}{4k^{2}(1-k^{2}){\bf
K}^{3}}\sum_{j=1}^{\infty}\displaystyle\frac{a^{j-1/2}\left(1-a^{2j-1}\right)}{\left(1+a^{2j-1}\right)^{2}}\sin(2j-1)\theta,$
$\begin{split}\frac{\partial}{\partial I}\left(\frac{\partial x}{\partial
I}\right)&=\frac{\pi^{4}}{32k^{4}(1-k^{2})^{2}{\bf K}^{5}}\left(((1+k^{2}){\bf
K}-3{\bf
E})\sum_{j=1}^{\infty}\frac{a^{j-1/2}\left(1-a^{2j-1}\right)}{\left(1+a^{2j-1}\right)^{2}}\sin(2j-1)\theta\right.+\\\
&+\left.\frac{\pi^{2}}{4{\bf
K}}\sum_{j=1}^{\infty}\frac{(2j-1)a^{j-1/2}\left(1-6a^{2j-1}+a^{2(2j-1)}\right)}{\left(1+a^{2j-1}\right)^{3}}\sin(2j-1)\theta\right)\end{split}$

were obtained. When calculating the integral (24), we have

$x=8\sum_{j=1}^{\infty}\displaystyle\frac{a^{j-1/2}}{(2j-1)\left(1+a^{2j-1}\right)}\sin(2j-1)\theta.$

Thus, calculating (18) – (25), we obtain an averaged system of the following
form

$\left\\{\begin{aligned}
\dot{u}&=p_{3}\widetilde{A_{1}}\sin{3v}+p_{2}\widetilde{A_{2}}\cos{3v}+B_{0}+\mu\left(p_{2}\widetilde{\sigma}\cos{3v}+B_{1}\right)u+\\\
&+\mu^{2}\left[\left(p_{3}\left(\widetilde{P_{11}}-\frac{3b_{2}}{b_{1}}Q_{01}\right)\sin{3v}\right.\right.+\left.\left.p_{2}\left(\widetilde{P_{12}}+\frac{3b_{2}}{b_{1}}Q_{02}\right)\cos{3v}+B_{2}\right)u^{2}\right.-\\\
&-\left.\frac{1}{b_{1}}\left(p_{3}\widetilde{P_{01}}\sin{3v}+p_{2}\widetilde{P_{02}}\cos{3v}\right)\left(p_{3}Q_{01}\cos{3v}+p_{2}Q_{02}\sin{3v}\right)\right],\\\
\dot{v}&=b_{1}u+\mu
b_{2}u^{2}+\mu^{2}\left[b_{3}u^{3}+\left(p_{3}\left(Q_{11}-\frac{2b_{2}}{b_{1}}Q_{01}\right)\cos{3v}+p_{2}\left(Q_{12}-\frac{2b_{2}}{b_{1}}Q_{02}\right)\sin{3v}\right)u\
\right],\end{aligned}\right.$ (26)

where

$\widetilde{A_{1}}=-2\displaystyle\frac{a^{3/2}}{1+a^{3}},\
\widetilde{A_{2}}=\displaystyle\frac{1}{4\pi\omega}\int_{0}^{2\pi}xy^{2}\sin{3\theta}d\theta,$
$\widetilde{P_{01}}=\displaystyle\frac{\pi^{3}}{16k^{2}(1-k^{2}){\bf
K}^{3}}\displaystyle\frac{3a^{3/2}(a^{3}-1)}{(1+a^{3})^{2}},\
\widetilde{P_{02}}=\displaystyle\frac{1}{4\pi\omega}\int_{0}^{2\pi}\displaystyle\frac{\partial}{\partial
I}\left(xy^{2}\right)\sin{3\theta}d\theta,$
$\widetilde{P_{11}}=\displaystyle\frac{\pi^{4}}{1024k^{4}(1-k^{2})^{2}{\bf
K}^{6}}\displaystyle\frac{3a^{3/2}}{(1+a^{3})^{3}}\left(4((1+k^{2}){\bf
K}-3{\bf E}){\bf K}(a^{6}-1)-3\pi^{2}(a^{6}-6a^{3}+1)\right),$
$\widetilde{P_{12}}=\displaystyle\frac{1}{8\pi\omega}\int_{0}^{2\pi}\displaystyle\frac{\partial^{2}}{\partial
I^{2}}\left(xy^{2}\right)\sin{3\theta}d\theta,\
Q_{01}=\displaystyle\frac{\widetilde{P_{01}}}{3},\
Q_{02}=\displaystyle\frac{1}{4\pi}\int_{0}^{2\pi}xy\displaystyle\frac{\partial
x}{\partial I}\cos{3\theta}d\theta,$
$Q_{11}=\displaystyle\frac{2\widetilde{P_{11}}}{3},\
Q_{12}=\displaystyle\frac{1}{4\pi}\int_{0}^{2\pi}\displaystyle\frac{\partial}{\partial
I}\left(xy\right)\displaystyle\frac{\partial x}{\partial
I}\cos{3\theta}d\theta,\
\widetilde{\sigma}=\displaystyle\frac{2a^{3/2}}{3(1+a^{3})},$
$B_{1}=-\displaystyle\frac{1}{15}\left[15+\left(128k^{4}-144k^{2}+31\right)p_{1}-\left(256k^{4}-256k^{2}+46\right)p_{1}\displaystyle\frac{{\bf
E}}{{\bf K}}\right],$
$\begin{split}B_{2}&=\displaystyle\frac{p_{1}\pi}{120k^{2}(1-k^{2}){\bf
K}^{3}}\left[\left(384k^{6}-656k^{4}+295k^{2}-23\right){\bf
K}^{2}-\left(768k^{6}-1280k^{4}+558k^{2}-46\right){\bf K}{\bf E}\right.-\\\
&-\left.\left(128k^{4}-128k^{2}+23\right){\bf E}^{2}\right],\end{split}$
$b_{1}=\frac{\pi^{2}}{16}\frac{\left(k^{2}-1\right){\bf K}+{\bf
E}}{k^{2}\left(k^{2}-1\right){\bf K}^{3}},\
b_{2}=-\frac{\pi^{3}}{256}\frac{\left(k^{2}-1\right){\bf
K}^{2}-2\left(k^{2}-2\right){\bf K}{\bf E}-3{\bf
E}^{2}}{k^{4}\left(k^{2}-1\right)^{2}{\bf K}^{5}},$
$b_{3}=\frac{\pi^{4}}{6144}\frac{\left(k^{4}+2k^{2}-3\right){\bf
K}^{3}-\left(2k^{4}+3k^{2}-13\right){\bf K}^{2}{\bf
E}+5\left(k^{2}-5\right){\bf K}{\bf E}^{2}+15{\bf
E}^{3}}{k^{6}\left(k^{2}-1\right)^{3}{\bf K}^{7}},$

$B_{0}$ is determined by formula (17).

Possible phase portraits of the averaged system (26) obtained using the WInSet
software [13], [14] are shown in Figures 1– 3.

##  6 Conclusion

The problem of the action of time-quasiperiodic perturbations on a self-
oscillating system plays an important role in the theory of oscillations. As
in the case of time-periodic perturbations, it leads to the phenomenon of
synchronization of oscillations, first studied by Andronov and Witt [36] on
the example of the Van der Pol equation in the quasi-linear case. Later, this
phenomenon was studied by Morozov and Shilnikov [37] for systems close to
arbitrary two-dimensional Hamiltonian ones (with time-periodic perturbation).

The existence of quasi-periodic solutions in systems close to Hamiltonian ones
has been considered in many papers. However, studies of the effect of quasi-
periodic perturbations on the self-oscillating system appeared for the first
time, apparently, in the papers of Morozov and his students [1, 3, 7, 8, 11].

In this paper, the problem of the effect of quasi-periodic perturbations on
systems close to arbitrary two-dimensional Hamiltonian in the case when
perturbed autonomous systems have a double limit cycle was studied. Averaged
systems are obtained for the case when the resonance level is close to the
level from which a double limit cycle appears under the influence of an
autonomous perturbation. These systems are not amenable to investigation in
the general case. This means the need to consider new examples and study the
behavior of solutions of these averaged systems in each case.

##  7 Acknowledgments

This research was supported by the Russian Foundation for Basic Research
(project No. 18-01-00306), the Russian Science Foundation (project No.
19-11-00280), and the Ministry of Science and Higher Education of Russian
Federation under grant No. 0729-2020-0036.

##  REFERENCES

  * [1] Morozov A.D., Dragunov T.N. On quasi-periodic perturbations of Duffing equation, Discontinuity, Nonlinearity, and Complexity, 2016, vol. 5, no. 4, pp. 377–386. https://doi.org/10.5890/DNC.2016.12.005
  * [2] Kuznetsov A.P., Kuznetsov S.P., Sedova Y.V. Pendulum system with an infinite number of equilibrium states and quasiperiodic dynamics, Nelineinaya Dinamika, 2016, vol. 12, no. 2, pp. 223–234. (In Russian). https://doi.org/10.20537/nd1602005
  * [3] Morozov A.D., Morozov K.E. Quasiperiodic perturbations of two-dimensional Hamiltonian systems, Differential Equations, 2017, vol. 53, no. 12, pp. 1557–1566. https://doi.org/10.1134/S0012266117120047
  * [4] Stankevich N.V., Kuznetsov A.P., Popova E.S., Seleznev E.P. Experimental diagnostics of multi-frequency quasiperiodic oscillations, Communications in Nonlinear Science and Numerical Simulation, 2017, vol. 43, pp. 200–210. https://doi.org/10.1016/j.cnsns.2016.07.007
  * [5] Truong T.Q., Tsubone T., Sekikawa M., Inaba N. Complicated quasiperiodic oscillations and chaos from driven piecewise-constant circuit: chenciner bubbles do not necessarily occur via simple phase-locking, Physica D, 2017, vol. 341, pp. 1–9. https://doi.org/10.1016/j.physd.2016.09.008
  * [6] Jiang T., Yang, Z., Jing, Z. Bifurcations and Chaos in the Duffing Equation with Parametric Excitation and Single External Forcing, International Journal of Bifurcation and Chaos, 2017, vol. 27, no. 08, p. 1750125-1-31. https://doi.org/10.1142/S0218127417501255
  * [7] Morozov A.D., Morozov K.E. On synchronization of quasiperiodic oscillations, Russian Journal of Nonlinear Dynamics, 2018, vol. 14, no. 3, pp. 367–376. https://doi.org/10.20537/nd180307
  * [8] Morozov A.D., Morozov K.E. Global dynamics of systems close to Hamiltonian ones under nonconservative quasi-periodic perturbation, Russian Journal of Nonlinear Dynamics, 2019, vol. 15, no. 2, pp. 187–198. https://doi.org/10.20537/nd190208
  * [9] Kuznetsov A.P., Kuznetsov S.P., Shchegoleva N.A., Stankevich N.V. Dynamics of coupled generators of quasiperiodic oscillations: Different types of synchronization and other phenomena, Physica D, 2019, vol. 398, pp. 1–12. https://doi.org/10.1016/j.physd.2019.05.014
  * [10] Guan Y., Gupta V., Wan M., Li L.K.B. Forced synchronization of quasiperiodic oscillations in a thermoacoustic system, Journal of Fluid Mechanics, 2019, vol. 879, pp. 390–421. https://doi.org/10.1017/jfm.2019.680
  * [11] Morozov A.D., Morozov K.E. On quasi-periodic parametric perturbations of Hamiltonian systems, Russian Journal of Nonlinear Dynamics, 2020, vol. 16, no. 2, pp. 369–378. https://doi.org/10.20537/nd200210
  * [12] Kolos M., Shahzadi M., Stuchlik Z. Quasi-periodic oscillations around Kerr-MOG black holes, The European Physical Journal C, 2020, vol. 80, no. 133, pp. 1–12. https://doi.org/10.1140/epjc/s10052-020-7692-5
  * [13] Sato M., Hyodo H., Biwa T., Delage R. Synchronization of thermoacoustic quasiperiodic oscillation by periodic external force, Chaos, 2020, vol. 30, no. 6, p. 063130. https://doi.org/10.1063/5.0004381
  * [14] Blekhman, I.I. Synchronization in science and technology, New York: ASME Press, 1988, 255 p.
  * [15] Pikovsky A., Rosenblum M., Kurths J. Synchronization: A universal concept in nonlinear science, England, Cambridge: Cambridge University Press, 2001. https://doi.org/10.1119/1.1475332
  * [16] Kuznetsov A.P., Sataev I.R., Stankevich N.V., Turukina L.V. Physics of quasiperiodic oscillations, Saratov: Publishing center ‘‘Science’’, 2013, 252 p. (In Russian).
  * [17] Anishchenko V., Nikolaev S., Kurths J. Winding number locking on a two-dimensional torus: Synchronization of quasiperiodic motions, Phys. Rev. E., 2006, vol. 73, p. 056202. https://doi.org/10.1103/PhysRevE.73.056202
  * [18] Kuznetsov A.P., Kuznetsov S.P., Stankevich N.V. A simple autonomous quasiperiodic self-oscillator, Communications in Nonlinear Science and Numerical Simulation, 2010, vol. 15, pp. 1676–1681. https://doi.org/10.1016/j.cnsns.2009.06.027
  * [19] Kuznetsov A.P., Kuanetsov S.P., Mosekilde E., Stankevich N.V. Generators of quasiperiodic oscillations with three-dimensional phase space, European Physical Journal Special Topics, 2013, vol. 222, pp. 2391–2398. https://doi.org/10.1140/epjst/e2013-02023-x
  * [20] Berger M.S., Chen Y. Y. Forced quasiperiodic and almost periodic oscillations of nonlinear Duffing equations, Nonlinear Analysis: Theory, Methods and Applications, 1992, vol. 19, no. 3, pp. 249–257. https://doi.org/10.1016/0362-546X(92)90143-3
  * [21] Liu B., You J. Quasiperiodic solutions of Duffings Equations, Nonlinear Analysis: Theory, Methods & Applications, 1998, vol. 33, no. 6, p. 645–655. https://doi.org/10.1016/S0362-546X(98)00662-2
  * [22] Grischenko A.D., Vavriv D.M., Dynamics of pendulum with a quasiperiodic perturbation, Technical Physics, 1997, vol. 42, no. 10, pp. 1115–1120. https://doi.org/10.1134/1.1258787
  * [23] Wang R.Q., Deng J., Jing Z.J. Chaos control in duffing system, Chaos, Solitons and Fractals, 2006, vol. 27, no. 1, pp. 249–257. https://doi.org/10.1016/j.chaos.2005.03.038
  * [24] Jing Z., Yang Z., Jiang T. Complex dynamics in Duffing-van der Pol equation, Chaos, Solitons and Fractals, 2006, vol. 27, no. 3, pp. 722–747. https://doi.org/10.1016/j.chaos.2005.04.044
  * [25] Jing Z.J., Huang J.C., Deng J. Complex dynamics in three-well Duffing system with two external forcings, Chaos, Solitons and Fractals, 2007, vol. 33, no. 3, pp.795–812. https://doi.org/10.1016/j.chaos.2006.03.071
  * [26] Ravichandran V., Chinnathambi V, Rajasekar S. Homoclinic bifurcation and chaos in Duffing oscillator driven by an amplitude-modulated force, Physica A: Statistical Mechanics and its Applications, 2007, vol. 376, pp. 223–236. https://doi.org/10.1016/j.physa.2006.11.003
  * [27] Hale J.K. Oscillations in nonlinear systems, Mineola, New York: Dover Publications, Inc., 2015, 192 p.
  * [28] Morozov A.D. Quasi-conservative systems: cycles, resonances and chaos, Singapore: World Sci, in ser. Nonlinear Science, ser. A, V. 30, 1998, 340 p. https://doi.org/10.1142/3238
  * [29] Morozov A.D. Resonance, cycles and chaos in quasi-conservative systems, Izhevsk: Regular and Chaotic Dynamics, Institute of Computer Science, 2005, 424 p. (In Russian).
  * [30] Morozov A.D., Mamedov E.A. On a double cycle and resonances, Regular and Chaotic Dynamics, 2012, vol. 17, no. 1, pp. 63–71. https://doi.org/10.1134/S1560354712010066
  * [31] Kostromina O.S. On limit cycles, resonance and homoclinic structures in asymmetric pendulum-type equation, Vestn. Udmurtsk. Univ. Mat. Mekh. Komp. Nauki, 2019, vol. 29, no. 2, pp. 228–244. https://doi.org/10.20537/vm190207
  * [32] Mel’nikov V.K. On stability of a center under periodic in time perturbations, Works of Moscow Math. Socity, 1963, vol. 12, pp. 3–53. (In Russian). http://mi.mathnet.ru/eng/mmo137
  * [33] Gradshteyn I.S., Ryzhik I.M. Table of integrals, series, and products, Academic Press, 2007, 1171 p.
  * [34] Morozov A.D., Dragunov T.N., Boykova S.A., Malysheva O.V. Invariant sets for Windows, World Scientific, 1999, 272 p. https://doi.org/10.1142/4220
  * [35] Morozov A.D., Dragunov T.N. Visualization and analysis of invariant sets of dynamical systems, Moscow–Izhevsk: Publishing House of the Institute of Computer Research, 2003, 303 p. (In Russian).
  * [36] Andronov A.A., Witt A.A. Zur theorie des mitnehmens von van der Pol, Archiv fur Elektrotechnik, 1930, vol. 24, no. 1, pp. 99–110. https://doi.org/10.1007/BF01659580
  * [37] Morozov A.D., Shil’nikov L.P. On nonconservative periodic systems close to two-dimensional Hamiltonian, Journal of Applied Mathematics and Mechanics, 1983, vol. 47, issue 3, pp. 327–334. https://doi.org/10.1016/0021-8928(83)90058-8