# Two Optimizers Are Better Than One: LLM Catalyst for Enhancing Gradient-
Based Optimization

Zixian Guo1,2 Ming Liu1 Zhilong Ji2 Jinfeng Bai2 Yiwen Guo4 Wangmeng Zuo1,3(✉)
1Harbin Institute of Technology 2Tomorrow Advancing Life
3Pazhou Lab, Guangzhou 4Independent Researcher
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Work done when
Zixian Guo was an intern at TAL.

###### Abstract

Learning a skill generally relies on both practical experience by doer and
insightful high-level guidance by instructor. _Will this strategy also work
well for solving complex non-convex optimization problems?_ Here, a common
gradient-based optimizer acts like a disciplined doer, making locally optimal
update at each step. Recent methods utilize large language models (LLMs) to
optimize solutions for concrete problems by inferring from natural language
instructions, akin to a high-level instructor. In this paper, we show that
these two optimizers are complementary to each other, suggesting a
collaborative optimization approach. The gradient-based optimizer and LLM-
based optimizer are combined in an interleaved manner. We instruct LLMs using
task descriptions and timely optimization trajectories recorded during
gradient-based optimization. Inferred results from LLMs are used as restarting
points for the next stage of gradient optimization. By leveraging both the
locally rigorous gradient-based optimizer and the high-level deductive LLM-
based optimizer, our combined optimization method consistently yields
improvements over competitive baseline prompt tuning methods. Our results
demonstrate the synergistic effect of conventional gradient-based optimization
and the inference ability of LLMs. The code is released at
https://github.com/guozix/LLM-catalyst.

##  1 Introduction

Recent developments of Large Language Models (LLMs) have demonstrated an
unprecedented ability to comprehend and generate human-like text, leading to
significant breakthroughs in natural language processing Touvron et al.
(2023a); Chowdhery et al. (2022). This has led to their adoption in various
advanced open-ended applications, where they are being instructed to
participate in dialogue (OpenAI et al., 2024), formulate and execute plans
(Gupta and Kembhavi, 2022; Gao et al., 2023), solve mathematical problems
(Yang et al., 2023), etc., rather than just performing rigid natural language
understanding tasks (Wang et al., 2019) like previous language models (Radford
et al., 2019; Devlin et al., 2018). The robust capability of LLMs in natural
language comprehension and the generation of more nuanced and contextually
relevant text provides a foundation for solving complex optimization problems
using LLMs.

Humans acquire skills through practical experience and external guidance from
instructors. Similarly, solving optimization problems relies on well-designed
algorithms incorporating prior knowledge, as well as meticulous procedural
implementation. We regard common gradient-based optimizer as disciplined doer.
These optimizers are effective in navigating the parameter space through
precise, incremental adjustments based on gradient information. However, their
local perspective often limits their ability to escape local optima and
discover more optimal solutions. LLMs can be used as an high-level instructor,
being capable of providing useful guidance for optimization, based on the
abundant internal knowledge acquired through massive pre-training, Recent
studies have proposed to utilize LLMs as strategy planner or optimizer on
concrete optimization tasks. For example, Eureka (Ma et al., 2024) trains
agents by reinforcement reward function designed by GPT-4, which can learn
various complex skills such as dexterous pen spinning. It shows that LLM can
guide the trend of optimized policy on a delicate level.

Employing LLMs for optimization offers unique advantages. The optimization is
conducted with natural language interactions, which contributes to two
charming properties. First, the implementation of the optimization is code-
free. All the steps in the process use dialog-like interactions with LLMs.
Secondly, LLMs generate instruction-related outputs by assembling task-related
semantic tokens which is hardly constrained by the local optima issue, which
is probably encountered by gradient-based optimization. On the other hand, LLM
returned solutions, generated by simply query and response interactions, may
not be as precise as the results optimized by rigorous step-by-step gradient
descent, especially under limited API calling budgets. The local carefulness
of gradient-based optimizer and diverse semantic exploration of LLM-based
optimizer are complementary to each other, suggesting a collaborative
optimization approach.

With this motivation, we propose a combined optimization method that adopts
LLMs as an optimizer (MaaO) as a catalyst for the conventional gradient-based
optimization. Our proposed optimization approach leverages both the locally
rigorous gradient-based optimizer in parameter space and high-level deductive
LLM-based optimizer in unconstrained vocabulary space for better performance.
To achieve a collaborative training based on the two optimizers, we interleave
the conventional training process of gradient-based optimization with
interactions with LLM. We first optimize the parameters for only dozens of
iterations by gradient optimizer. Then the optimized parameters in the
intermediate step, along with their loss and accuracy on the training set, are
provided as history trajectory clues for LLM to infer new candidates that are
potentially more effective, and, after grabbing the response from LLM, we use
the generated results as restarting points of the parameters for subsequent
gradient-based optimization iterations. The two optimizers are operated
alternately to optimize the parameter collaboratively. The final optimized
results are obtained by the gradient optimizer with a stable convergence. Our
proposed optimizing strategy only injects several API calls to LLM to the
conventional gradient-based training workflow. We instantiate the target
problem as prompt optimization and validate that our proposed combined
optimizing method leads to consistent improvements.

In summary, our contributions include:

  * •

We analyze the limitations of gradient-based optimization, especially the
entrapment in local optima, and attribute the issues to the limitation of
gradient-based optimizer in the short-sighted local perspective of the
parameter space.

  * •

We propose a novel optimization approach that combines the deductive LLM-based
optimizer in unconstrained vocabulary space with the disciplined gradient-
based optimizer in the parameter space for better optimization performance.

  * •

We test the effectiveness of the proposed combined optimization method on
prompt tuning tasks, and it achieves consistent improvements over existing
prompt tuning methods, validating the complementary effect of LLM-based and
gradient-based optimizers.

##  2 Related Work

###  2.1 LLMs and Optimization Problems

LLM’s superior capabilities in grasping intricate linguistic structures and
accurate language generation enable it to handle complex optimization
problems. Existing works have validated the effectiveness of LLM for solving
small-scale mathematical optimization problems (Yang et al., 2023), optimizing
prompts (Zhou et al., 2023; Pryzant et al., 2023; Guo et al., 2024; Liu et
al., 2023a; Fernando et al., 2023; Diao et al., 2023), searching for network
architectures (Chen et al., 2023; Zheng et al., 2023), hyperparameter
optimization (Chen et al., 2022) and discovering physical equations (Du et
al., 2024).

For prompt optimization, APE (Zhou et al., 2023) proposes a general framework
that searches for prompts in the vocabulary space by instructing large
language models with task information to obtain better solutions iteratively
from the responses of LLMs by analyzing previously found candidates. Based on
this framework, APO (Pryzant et al., 2023) proposes that editing prompts by
LLM is analogous to conducting gradient descent in the natural language
domain. They imitate the gradient-based learning by providing the failure
cases to LLM for a semantic "gradient" and updating the prompt in an opposite
semantic direction. EVOPROMPT (Guo et al., 2024) also connects LLM-based
optimization to traditional algorithms for better explainability. They
integrate LLM into the workflow of evolutionary algorithms by instructing LLM
to act like evolutionary operators to generate new candidate prompts. The
insight that LLM naturally enables an intelligent variation operator is also
revealed in LMC (Meyerson et al., 2024) and ELM (Lehman et al., 2022) on image
and code generation tasks. BBO(Liu et al., 2023a) searches prompts for vision-
language model by merely conversing with LLM following designed strategies and
achieves comparable results to white-box gradient-based prompt tuning.
However, the performance superiority only holds in one-shot setting. LLM can
not effectively optimize to gain more improvements based on more training
data.

The results achieved in these approaches demonstrate that LLMs have the
potential to be applied as a general-purpose optimizer to more tasks. Some of
the works explored the connection between LLM-based inference and existing
optimization algorithms (Pryzant et al., 2023; Guo et al., 2024). However, the
proposed optimization workflow is still largely based on the inherent ability
of LLM. It is still hard for LLM to make decisions as precise as deterministic
optimization algorithms, e.g., gradient-based optimizer. And the API calling
budget bounded by the high cost of operating super large-scale models also
limits the performance of LLM-based optimization.

###  2.2 Prompt Tuning for Pre-trained Models

Prompt tuning has emerged as a standard approach for the parameter-efficient
adaptation of pre-trained models, aimed at improving their performance in
various natural language processing Lester et al. (2021); Li and Liang (2021)
and vision-language Zhou et al. (2022b, a); Yao et al. (2024) tasks. Prompt-
based tuning of pre-trained models appends learnable embeddings to the
original sequence of the data for the input layer or intermediate layer. Fine-
tuning the lightweight parameters in the prompt yields comparable performance
even to full parameter fine-tuning and transferability (Vu et al., 2022; Su et
al., 2022) on various tasks (Lester et al., 2021; Li and Liang, 2021; Liu et
al., 2022). Despite its widespread adoption, the conventional prompt tuning
technique encounters challenges related to slow convergence and suboptimal
optimization Ding et al. (2022), which undermines the effectiveness of prompt
tuning in a wider and larger scale of pre-trained models and downstream tasks.
We attribute these issues to the complexity of the input embedding space of
the pre-trained model, making it challenging to optimize the prompt
effectively based on back-propagated gradients in this space.

##  3 Method

Figure 1: Overview of our proposed method. The bold arrows with different
color show the two collaborative optimizers of in our method. The thin arrows
show the workflow of MaaO which infer for promising candidate prompt in
vocabulary space for gradient-based optimizer.

In this section, we introduce our proposed combined optimization approach that
leverages both the local carefulness of gradient-based optimizer and the
flexible semantics exploration of LLM-based optimizer. The overview of our
method is shown in Figure 1. To elaborate on our proposed method, we
instantiate the problem in a prompt tuning scenario. We will describe the
general formulation of prompt tuning/optimization and the way of gradient-
based prompt tuning in Section 3.1. Next, we analyze the issues that occur in
the conventional gradient-based prompt tuning process in Section 3.2, and
attribute the problem to the characteristics of gradient-based optimizer that
is limited to the local view of the parameter space. Finally we introduce our
proposed optimization method that adopts LLM as a catalyst for gradient-based
prompt tuning in Section 3.3.

###  3.1 General Formulation of Prompt Tuning

In this part, we instantiate the task by prompt tuning for discriminative
tasks, i.e., classification. In a general situation, we consider a pre-trained
multi-modal model $\mathcal{E}$. Recognizing the classes of input images $I$
or texts $T$ can be achieved by classifying the representations
$\mathcal{E}\left(I,T\right)$, encoded by the pre-trained model. We denote the
task-specific classifier as $\mathcal{F}\left(\cdot\right)$. The prediction
can be obtained by
$p\left(\hat{y}|I,T\right)=\mathcal{F}\left(\mathcal{E}\left(I,T\right)\right)$.

To better adapt pre-trained models to various downstream tasks, prompt tuning
introduces learnable prompt tokens and formulates a task-specific input for
the pre-trained model. The learnable prompt tokens can be either continuous
vectors (Zhou et al., 2022b; Lester et al., 2021) in the textual embedding
space of the pre-trained model or discrete tokens (Diao et al., 2022; Deng et
al., 2022) sampled from the vocabulary. The prompt $P$ parameterized by
$\bm{\theta}$ is concatenated with the original input making up a task-
specific input. The adapted output can be formulated as
$p\left(\hat{y}|I,T;\bm{\theta}\right)=\mathcal{F}\left(\mathcal{E}\left(P_{\bm{\theta}},I,T\right)\right)$.
In common practice, the prompt tokens are learned through labeled few-shot
samples from target task datasets. The parameters of the prompt are optimized
by minimizing the loss function:

$\begin{split}\bm{\theta}^{*}=\mathop{\arg\min}_{\bm{\theta}}\mathcal{L}(y,I,T,\bm{\theta})=\mathop{\arg\min}_{\bm{\theta}}-\log
p(\hat{y}=y|I,T;\bm{\theta}).\end{split}$ (1)

According to this formulation, it is straightforward to use a standard
gradient-based optimizer to learn the parameters as is done in conventional
prompt tuning methods:

$\bm{\theta}_{t+1}=\bm{\theta}_{t}-\eta_{t}\nabla_{\bm{\theta}}\mathcal{L}(y,I,T,\bm{\theta}).$
(2)

###  3.2 Analysis on Issues of Gradient-Based Prompt Tuning

Figure 2: The result of gradient-based prompt optimization with different
prompt initialization. The shadow denotes the standard deviation of the
accuracy over three random seeds.

Although prompt tuning has become one of the most widely adopted parameter-
efficient fine-tuning methods for the adaptation of pre-trained models. The
optimization of the prompt still encounters challenges. The prompts converges
much slower than other parameters efficient fine-tuning methods, e.g., adapter
tuning or even full parameter fine-tuning (Ding et al., 2022), based on the
estimated gradients back-propagated through the entire pre-trained model.
Another main issue of prompt tuning is that the effectiveness of the learned
prompt is sensitive to its initialization values, suggesting that the
optimization of the prompt may easily entrapped in local optima due to the
complexity of the embedding space of the pre-trained model. Unfortunately, it
is challenging to carefully craft initial prompts for every downstream task.
To address this issue, Gu et al. (2022) propose to seek a satisfying
initialization point for the prompt. However, their method needs to inject
soft prompts into the pre-training stage, which limits its application to
scenarios where pre-training resource is limited.

To demonstrate the issues more specifically, we analyze some empirical results
of gradient-based prompt optimization performed on the one-shot training set
of EuroSAT (Helber et al., 2019). We fix the training set for all experiments
to eliminate the variance caused by data sampling. We run CoOp (Zhou et al.,
2022b) under three random initializations and show the results as indicated by
"Random Initialization" in Figure 2. It can be seen that even if we fix the
training samples, different random initialization value of the prompt can
still bring considerable standard deviation in the results of final learned
prompts, indicating a large performance gap (up to 9 percent of accuracy)
between different seeds. If we manually initialize the prompt as a prompt
template "a photo of a", which is used in Radford et al. (2021), the final
variance gets smaller but the absolute performance shows a slight decline.
Prior knowledge contained in manual prompts brings merits, providing better
results at the starting phase of the training, but lacks proper flexibility
for enhancement of final learned prompts. Our method adds marginal steps of
optimization based on the collaboration of gradient-based optimizer and MaaO
at the start of the training workflow, which results in both lower standard
deviation and better absolute performance.

The high sensitivity of prompt tuning results according to different
initialization values indicate the complexity of the input embedding space,
where gradient-based optimizer only leads to suboptimal converged parameters
based on gradient information in a short-sighted local perspective, hardly
taking the semantics of the prompt and the overall task information into
account. To mitigate the limitations of gradient-based optimizer, we leverage
LLM as an unconstrained vocabulary space prompt optimizer based on textual
semantic information of the task and previously found prompts.

Hi GPT, assume you are a prompt pattern learner. I have a list of text
templates with their corresponding loss values and accuracy. They are used for
image classification with CLIP model. The templates are arranged in descending
order based on their loss value on training samples, where lower loss
indicates better quality. Templates: a precise satellite view of Loss: 2.18
Accuracy: 20.0 Templates: a centered satellite photo of {}. (Mamual prompt to
inject prior knowledge.) Loss: 1.96 Accuracy: 30.0 Templates: a crisp high -
definition image of Loss: 1.85 Accuracy: 50.0 … (more optimized prompts and
scores) There are latent patterns that make the template good. Based on these
patterns, write your new template that is different from the old ones and has
a loss as low as possible. Here are some requirements \- Please reply with
only the template \- Keep every template under 10 words \- Generate 3
templates that potentially have better image classification performance

Figure 3: The instruction used to query GPT-3.5 and GPT-4.0 in an iteration of
optimizing the prompt using LLM. Algorithm 1 Combined Optimization Algorithm

1:Prompt $\bm{p}_{\bm{\theta}}$ parameterized with $\bm{\theta}$, training set
$\mathcal{D}$, loss function regarding the target pre-trained model and the
training set $f_{\mathcal{D}}(\cdot)$, number of optimization rounds $N$,
number of iterations $m$, $M$, embedding layer operater
$\mathcal{V\left(\cdot\right)}$ and token space projection operator
$\mathcal{V}^{-1}\left(\cdot\right)$, prompt candidates set $\mathcal{P}$.

2:Initialize: prompt $\bm{\theta}$ with random values,
$\mathcal{P}\leftarrow\varnothing$.

3:for $n=1$ to $N$ do

4: // Gradient-based optimization:

5: for $\tau=1$ to $m$ do

6: Update: $\bm{\theta}_{\tau}\leftarrow\bm{\theta}_{\tau-1}-$lr$\cdot\nabla
f(\bm{\theta}_{\tau-1})$

7: Record:
$\mathcal{P}\leftarrow\mathcal{P}\cup\\{(\bm{p}_{\bm{\theta}},-,-)\\}$

8: // Prompt evaluation:

9: for $\bm{p}_{\bm{\theta}}$ in $\mathcal{P}$ do

10: Discretize: $\hat{p}\leftarrow\mathcal{V}^{-1}(\bm{p}_{\bm{\theta}})$

11: Evaluate: $s\leftarrow f_{\mathcal{D}}(\hat{p})$

12: Record:
${\mathcal{P}}\leftarrow{\mathcal{P}}\cup\\{(\bm{p}_{\bm{\theta}},\hat{p},s)\\}$

13: // LLM-based optimization:

14: Sample:
$\\{\hat{p_{i}}\\}_{i=1}^{k}\leftarrow\text{TopK}_{s}(\hat{p}|(p_{\bm{\theta}},\hat{p},s)\in\mathcal{P})$

15: Generate:
$\widetilde{p}\leftarrow\text{LLM}(\texttt{Instruction}(\\{\hat{p_{i}}\\}_{i=1}^{k}))$

16: Reinitialize:
$\widetilde{\bm{p}}_{\bm{\theta}}\leftarrow\mathcal{V}(\widetilde{p})$,
$\mathcal{P}\leftarrow\varnothing$

17:// Gradient-based optimization:

18:Train the prompt parameter with gradient optimizer for $M$ iterations till
convergence.

19:Return the optimized prompt $\bm{p}^{*}_{\bm{\theta}}$

###  3.3 Catalyzing Prompt Tuning by Using LLM as a Prompt Optimizer

We propose to harness LLM as an optimizer (MaaO) to mitigate the issues of
gradient-based prompt tuning. We leverage the unconstrained inductive ability
of LLM in vocabulary space based on high-level semantic information of the
prompt to complement the gradient-based optimizer.

Our method optimizes the prompt by using the gradient optimizer and MaaO in an
alternating pattern. Specifically, we first update the parameter of the prompt
for minor steps of gradient-descent optimization and record the intermediate
learned prompts and corresponding fitness scores, which are evaluated on the
few-shot training samples. Then, we construct instruction for LLM with the
intermediate learned prompts as optimizing trajectory information. Taking the
instruction as input, LLM generates more promising candidate prompts for the
target model. Next, we reinitialize the parameter of the prompts with LLM-
generated prompts and restart the gradient-based training process for the next
round. After operating the above two optimizers alternately for few rounds, we
finally train the prompt to convergence using the gradient-based optimizer. In
the following, we will describe the components of MaaO and show the
combination of it with the gradient-based optimizer for a collaborative prompt
optimization.

Instruction construction. The gradient optimizer calculates updates based on
the current parameters and objective function. Information of the current
state of optimization should also be properly provided for LLM to infer from.
We collect the intermediate optimized prompt in the training trajectory of the
gradient-based optimizer and evaluate the performance corresponding to each
intermediate prompt as a fitness score, indicating how good or bad the prompt
performs. Considering that the accuracy may be not precise enough on few
samples, we employ loss as the indicator value. LLM is instructed to generate
prompts that potentially achieve better performance based on observed patterns
in top-performing candidates.

We also briefly define the role of LLM and explain the goal of optimization in
natural language, encouraging LLM to assemble task-related tokens when
constructing the prompt. Additional instructions to constrain the length and
number of the generated prompts are included for programmed processing. Figure
3 shows the instruction used in each optimization iteration of prompt using
GPT-3.5 and GPT-4.

Token space projection. Gradient-based prompt tuning typically optimizes soft
prompt tokens in the embedding space of the pre-trained model. However, it is
not feasible to directly provide the soft embedding vectors as input to the
LLM, which is proficient in responding to natural language with semantics. To
convert the soft prompt embedding to discrete words, we employ a reverse
process of word2vec (Mikolov et al., 2013) to project the embedding to the
matched vocabulary.

Given a pre-trained target model with token embedding layer
$\mathcal{V\left(\cdot\right)}$, textual inputs $\\{t_{i}\\}_{i=1}^{l}$ to the
model are first converted to vector sequence as
$\\{\bm{t}_{i}\\}_{i=1}^{l}=\\{\mathcal{V}(t_{i})|i\in[1,l]\\}$, before input
into the model. Gradient-based prompt tuning optimizes in the continuous
vector space for best prompt embeddings. We project the continuous prompt
vector $\bm{\hat{t}}_{i}$ to nearest discrete tokens by
$\hat{t_{i}}=\mathcal{V}^{-1}(\bm{\hat{t}}_{i})$, where $\mathcal{V}^{-1}$ is
defined as:

$\mathcal{V}^{-1}(\bm{\hat{t}}):=\mathop{\arg\min}_{\hat{t}\in\mathcal{S}}\ \
\|\mathcal{V}(\hat{t})-\bm{\hat{t}}\|_{2}.$ (3)

$\mathcal{S}$ denotes the dictionary of the pre-trained model. The projected
prompt is used to construct the instruction for LLM to infer for better prompt
candidates in the unconstrained semantic vocabulary space.

Integration with gradient-based optimizer. Gradient-based optimizer conduct
rigorous local-optimal update on the parameter based by back-propagated
gradient. MaaO infers for promising candidate prompts by analyzing and
generating semantic-related prompts based on currently found solutions. We
propose a cooperation workflow of the gradient-based optimizer and MaaO in
Algorithm 1.

We connect the two optimizers in two ways. First, the gradient optimizer
provides the LLM with the intermediate results in the prompt optimization
process, from which LLMs infer for more promising candidate prompts. The
generated prompts by LLMs assemble task-related semantic contents and provide
opportunities to break free from local optimal that may encountered in
gradient-based optimization. Second, we restart the gradient-based
optimization by using the prompts generated by the LLM optimizer as new
initial values of the gradient optimizer to obtain refined prompts based on
the LLM-generated ones. By optimizing the prompt based on the two optimizers
alternately, it guides the LLM to progressively exploit better prompts in a
more promising area of the search space near the previously found good
solutions. The gradient optimizer provides stable convergence for the final
learned prompts. Note that the overhead brought by our optimization algorithm
compared to the original gradient-based prompt tuning is only about dozens (at
most 30) of iterations using combined optimizer.

##  4 Experiments

Table 1: Few-shot prompt optimization results on downstream datasets.

Datasets | Settings | ResNet50 | ViT-B/16
---|---|---|---
CoOp | BBO | Ours(GPT-3.5) | Ours(GPT-4) | Ours(Llama7B) | TCP | Ours(GPT-3.5)
Eurosat | 1-shot | 50.58 | 49.0 | 56.27 | 56.74 | 55.38 | 65.04 | 63.06
4-shot | 69.65 | - | 71.17 | 72.55 | 73.39 | 72.42 | 77.35
8-shot | 72.74 | - | 74.33 | 75.99 | 76.79 | 77.71 | 79.49
16-shot | 83.57 | 51.4 | 83.77 | 85.07 | 83.95 | 84.43 | 86.18
Avg. | 69.14 |  | 71.39 | 72.59 | 72.38 | 74.90 | 76.52
DTD | 1-shot | 43.13 | 44.8 | 47.24 | 44.78 | 42.47 | 55.06 | 55.14
4-shot | 53.45 | - | 54.87 | 55.04 | 54.14 | 61.88 | 63.36
8-shot | 59.38 | - | 60.30 | 60.18 | 60.48 | 68.62 | 68.56
16-shot | 63.87 | 44.9 | 64.40 | 64.30 | 64.48 | 73.48 | 73.66
Avg. | 54.96 | - | 56.70 | 56.08 | 55.39 | 64.76 | 65.18
Caltech101 | 1-shot | 87.76 | 89.1 | 87.02 | 86.87 | 87.86 | 94.08 | 94.00
4-shot | 89.05 | - | 88.72 | 88.68 | 89.03 | 95.18 | 95.38
8-shot | 90.58 | - | 90.26 | 90.66 | 90.25 | 95.39 | 95.33
16-shot | 91.66 | 89.5 | 92.28 | 91.83 | 92.33 | 95.89 | 95.71
Avg. | 89.76 | - | 89.57 | 89.51 | 89.87 | 95.14 | 95.11
Oxford_Flowers | 1-shot | 69.09 | 67.2 | 71.62 | 71.28 | 72.31 | 85.80 | 87.05
4-shot | 87.00 | - | 89.38 | 88.51 | 88.93 | 94.72 | 95.09
8-shot | 90.19 | - | 91.15 | 90.77 | 90.41 | 96.14 | 96.31
16-shot | 93.88 | 67.4 | 94.42 | 94.02 | 94.49 | 97.47 | 97.54
Avg. | 85.04 | - | 86.64 | 86.15 | 86.54 | 93.53 | 94.00
Fgvc_Aircraft | 1-shot | 18.38 | 18.1 | 18.69 | 18.82 | 18.50 | 28.90 | 28.33
4-shot | 21.90 | - | 22.73 | 22.77 | 23.19 | 35.61 | 35.61
8-shot | 25.15 | - | 26.53 | 26.42 | 27.35 | 39.76 | 40.69
16-shot | 28.86 | 18.1 | 31.27 | 31.24 | 31.44 | 43.29 | 43.79
Avg. | 23.57 | - | 24.81 | 24.81 | 25.12 | 36.89 | 37.11
Food101 | 1-shot | 72.60 | 78.3 | 73.86 | 73.98 | 72.47 | 85.74 | 85.40
4-shot | 70.93 | - | 70.44 | 70.25 | 69.76 | 86.43 | 86.01
8-shot | 73.97 | - | 73.12 | 73.97 | 72.51 | 86.83 | 86.67
16-shot | 75.72 | 78.3 | 75.20 | 74.94 | 73.85 | 87.25 | 87.26
Avg. | 73.31 | - | 73.16 | 73.29 | 72.15 | 86.56 | 86.34
Stanford_Cars | 1-shot | 55.70 | 56.2 | 54.74 | 54.89 | 54.78 | 68.87 | 68.05
4-shot | 61.22 | - | 61.93 | 61.76 | 62.01 | 75.25 | 76.17
8-shot | 65.14 | - | 65.89 | 66.82 | 67.37 | 79.27 | 79.29
16-shot | 67.97 | 56.8 | 68.84 | 69.34 | 73.30 | 83.79 | 83.98
Avg. | 62.51 | - | 62.85 | 63.20 | 64.37 | 76.80 | 76.87
Oxford_Pets | 1-shot | 85.14 | 88.1 | 85.70 | 84.38 | 84.78 | 91.26 | 90.75
4-shot | 85.37 | - | 85.03 | 85.33 | 84.47 | 92.67 | 92.65
8-shot | 85.70 | - | 84.65 | 84.82 | 84.58 | 92.91 | 92.55
16-shot | 86.84 | 88.3 | 86.37 | 86.00 | 85.02 | 93.34 | 93.19
Avg. | 85.76 | - | 85.44 | 85.13 | 84.71 | 92.55 | 92.29
UCF101 | 1-shot | 62.60 | 60.2 | 61.85 | 61.07 | 62.34 | 73.44 | 72.69
4-shot | 68.75 | - | 68.25 | 69.18 | 68.27 | 80.93 | 80.77
8-shot | 72.26 | - | 72.69 | 72.26 | 72.58 | 83.18 | 83.40
16-shot | 74.91 | 60.5 | 74.82 | 75.96 | 74.95 | 85.25 | 85.13
Avg. | 69.63 | - | 69.40 | 69.62 | 69.54 | 80.70 | 80.50
SUN397 | 1-shot | 58.33 | 61.0 | 58.13 | 57.25 | 57.47 | 69.20 | 69.13
4-shot | 64.48 | - | 65.21 | 64.06 | 64.40 | 73.78 | 73.94
8-shot | 66.79 | - | 67.20 | 66.90 | 66.79 | 75.78 | 75.99
16-shot | 68.79 | 60.8 | 68.39 | 68.42 | 68.42 | 76.81 | 76.69
Avg. | 64.60 | - | 64.73 | 64.16 | 64.27 | 73.89 | 73.94

###  4.1 Experimental Setup

Implementation details and baselines. To comprehensively evaluate the
effectiveness of our method, we use GPT-3.5, GPT-4 (OpenAI et al., 2024), and
white-box LLM Llama2 (Touvron et al., 2023b) as the LLM-based optimizer. We
integrate MaaO into existing prompt tuning methods for language pre-trained
models, P-tuning (Liu et al., 2023b), Lester et al. (2021), and two methods
for vision-language model, CoOp (Zhou et al., 2022b), TCP (Yao et al., 2024).
CoOp pioneered prompt tuning for vision-language models, and TCP is one of the
recent state-of-the-art prompt tuning approaches. Both of the methods are
typical textual prompting methods for vision-language models. For a fair
comparison, we fix the original hyperparameter of previous methods, such as
pre-trained backbone and prompt module design, and only apply our method as
new optimization strategy. For the configuration of Algorithm 1, the number of
rounds $N$ is set as 3 and the iteration for the gradient optimizer $m$ is set
as 10. All experimental results are averaged over 3 random seeds. The
experiments are conducted on a V100 GPU. Detailed hyperparameter setting is
provided in the Appendix.

Datasets. For lauguage model, we conduct experiments over commonly-used pre-
trained model RoBERTa (Liu et al., 2019) on NLU tasks from SuperGLUE (Wang et
al., 2020) to test our methods. We apply our prompt optimization algorithm to
vision-language pretrained CLIP (Radford et al., 2021) for adaptation of image
classification tasks. We adopt commonly used 10 datasets to comprehensively
evaluate our method, including Caltech101 (Li et al., 2004), OxfordPets
(Parkhi et al., 2012), StanfordCars (Krause et al., 2013), Flowers102
(Nilsback and Zisserman, 2008), Food101 (Bossard et al., 2014), FGVCAircraft
(Maji et al., 2013), SUN397 (Xiao et al., 2010), UCF101 (Soomro et al., 2012),
DTD (Cimpoi et al., 2014), and EuroSAT (Helber et al., 2019). For each
dataset, labeled few-shot samples from each class are used as training data.

Figure 4: Interpretation of prompts optimized by LLM on EuroSAT dataset.

###  4.2 Main Results

Experiments on vision-language pre-trained models. We compare the results of
employing our proposed optimization method for existing prompting methods
(Zhou et al., 2022b; Yao et al., 2024). From Table 1, the results on "RN50"
backbone show that our integrated optimization outperforms existing gradient-
based prompt tuning methods at seven out of ten benchmark datasets and the
other tasks remain close to the baseline performance. Both black-box GPT and
open-sourced Llama2 achieve consistent improvements, demonstrating the
effectiveness of our combined optimization framework. TCP (Yao et al., 2024)
is one of the state-of-the-art prompt tuning approaches with a stronger
backbone. Although the absolute improvement inevitably decreases, our method
still brings stable improvements on six out of ten datasets.

We also compare with methods that optimize by LLM only, e.g., BBO (Liu et al.,
2023a). We list the results reported in BBO paper since the code has not been
released yet and our reproduced results can not match the results published in
the paper. Although BBO achieves a completely program-free prompt learning
method for CLIP, their performances in 16-shot setting are poor. Their method
largely relies on the inherent deductive ability of LLM, which can not make
good use of information in more training samples. Our method instruct LLMs
with intermediate results of gradient optimizer, enabling LLMs to exploit in a
more promising sub-region of the semantic space. The prompts generated by LLM
are further refined by the disciplined gradient-based optimizer to achieve
stable improvements.

Table 2: Results on SuperGLUE dev-set. (PT: P-tuning & Lester et al. (2021); Model: RoBERTa-Large). Methods | COPA | BoolQ | RTE | WiC | WSC
---|---|---|---|---|---
PT | 61.67 | 62.29 | 55.72 | 53.81 | 64.10
Ours | 68.67 | 63.09 | 58.00 | 55.85 | 63.46

Experiments on language models. We also test our method on pre-trained
language models and NLU tasks. From Table 2, our method with GPT-4 as
optimizer surpasses P-tuning (Liu et al., 2023b) and Lester et al. (2021) on
four out of five tasks from SuperGLUE, showing the superiority of our
optimization method over vanilla gradient-based prompt tuning.

In summary, the results indicate that our proposed combined optimization
approach, which leverages both the local precision of a gradient-based
optimizer and the flexible semantics exploration of an LLM-based optimizer, is
better than both single methods and outperforms each method individually.

Table 3: Ablation on the design of instruction. TD | MP | OT | EuroSAT | DTD | Oxford_Flowers
---|---|---|---|---|---
✗ | ✗ | ✗ | 70.33 | 55.42 | 85.71
✓ | ✓ | ✗ | 70.56 | 55.95 | 86.44
✓ | ✗ | ✓ | 71.26 | 56.98 | 86.25
✓ | ✓ | ✓ | 71.39 | 56.70 | 86.64

Interpretation of prompts optimized by LLM. To further analyze the
contribution of LLM-based optimizer in prompt optimization, we list the
prompts contained in the instruction and generated by LLM in Figure 4. In
round 1, the gradient-based optimizer tend to navigate around senseless prompt
tokens, e.g., "beh", "ila", etc. This phenomenon is alleviated after
leveraging LLM to infer for more meaningful prompts. In round 3, prompts with
both interpretability and low loss values are obtained by the collaboration of
gradient-based optimizer and LLM.

###  4.3 Ablation Study

Our method’s sensitivity to the choice of LLMs, prompt tuning baseline methods
and amount of training samples are already shown in Table 1. We provide
ablation results of three factors of our method in this section. The ablation
experiments are based on CoOp baseline, using GPT-3.5 as optimizer. More
ablation studies can be found in the Appendix.

Design of instruction. The instruction from which LLMs infer for new
candidates is influential on the results. We empirically analyze the effect of
each component in our instructions in Table 3. Task definition (TD) denotes
raw instruction defining the task information. Manual prompt (MP) means LLMs
are instructed with hand-crafted prompt template. Optimization trajectory (OT)
denotes the intermediate results from the gradient optimizer is provided. The
results in first line of Table 3 correspond to no LLM-based optimization,
serving as a stronger baseline than CoOp. Random noise is added to perturb the
prompts after each round of gradient-based optimization for the opportunity to
escape from local optima. The ablation results show that hand-crafted
template, providing prior knowledge of the prompt, and optimization
trajectory, providing timely semantic landscape of current optimized prompts,
are both important components for ideal performance.

Table 4: Ablation on the rounds of alternating optimization. $N$ | EuroSAT | DTD | Oxford_Flowers
---|---|---|---
1 | 72.51 | 56.33 | 84.45
2 | 71.69 | 56.13 | 84.82
3 | 71.39 | 56.70 | 86.64
4 | 71.50 | 56.41 | 86.89

Table 5: Ablation on the iterations of gradient-based optimizer. $m$ | EuroSAT | DTD | Oxford_Flowers
---|---|---|---
$10$ | 71.39 | 56.70 | 86.64
$10^{2}$ | 69.46 | 56.10 | 83.92
$10^{3}$ | 64.31 | 54.81 | 82.31

Rounds of alternating optimization. We analyze the effect of the alternating
rounds $N$ of the two optimizer on the result. Table 5 indicate that the
optimal round for each task varies. But more rounds involve more interactions
with LLM, providing more candidates prompts. The average performance improves
with more rounds generally. We choose 3 rounds as a proper value.

Iterations of gradient optimizer. We analyze the effect of the training
iterations $m$ of gradient optimizer in every round. From Table 5, we can see
that increasing training iterations leads to descending results. A longer
training process by gradient-based optimizer may not necessarily benefit the
final prompt tuning results. We find that more iterations of gradient-based
optimization may results in candidate prompts with less semantic diversity,
which is not good for proposing LLM to generate more promising candidate
prompts. Thus, we use a small training iteration for better performance and
efficiency of our optimization method.

##  5 Conclusion

In this paper, we proposed a combined optimization method that integrates LLM-
based optimizer with conventional gradient-based optimization. Our method
mitigates the inherent limitations of gradient-based optimization, such as
entrapment in local optima, by leveraging the inductive reasoning capabilities
of LLMs. By interleaving gradient-based optimization process with natural
language interactions with LLMs, we introduced a higher-level guidance for
optimization that effectively utilizes task descriptions and real-time
optimization trajectories. We validated our combined optimization strategy
through prompt tuning tasks, where the synergy between LLM-based optimizer and
gradient-based optimizer has consistently demonstrated improved performance
over competitive baselines. These results underscore the complementary effect
of LLM-based optimizer and conventional gradient-based optimization. Our
contributions inspire further exploration of the advantages of LLM-based
optimization over existing algorithms, paving the way for more effective
integration of LLM-based inference into conventional optimization workflows.

Limitations. Our proposed method can not be effectively employed for adapter-
based fine-tuning methods. Feasible solution of expressing optimization status
of adapter module to LLMs needs further design. We leave the application of
the proposed method to more wider range of optimization problems (e.g.,
adapters, LoRA) and algorithms (e.g., reinforce learning) as future work.

## References

  * Bossard et al. (2014) Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101–mining discriminative components with random forests.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13_ , pages 446–461. Springer, 2014.
  * Chen et al. (2023) Angelica Chen, David M. Dohan, and David R. So.  Evoprompting: Language models for code-level neural architecture search, November 2023.
  * Chen et al. (2022) Yutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Qiuyi Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski, Arnaud Doucet, Marc’aurelio Ranzato, Sagi Perel, and Nando de Freitas.  Towards learning universal hyperparameter optimizers with transformers, October 2022.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.  Palm: Scaling language modeling with pathways, October 2022.
  * Cimpoi et al. (2014) Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.  Describing textures in the wild.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3606–3613, 2014.
  * Deng et al. (2022) Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P Xing, and Zhiting Hu.  Rlprompt: Optimizing discrete text prompts with reinforcement learning.  _arXiv preprint arXiv:2205.12548_ , 2022.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Diao et al. (2022) Shizhe Diao, Zhichao Huang, Ruijia Xu, Xuechun Li, Yong Lin, Xiao Zhou, and Tong Zhang.  Black-box prompt learning for pre-trained language models.  _arXiv preprint arXiv:2201.08531_ , 2022.
  * Diao et al. (2023) Shizhe Diao, Pengcheng Wang, Yong Lin, and Tong Zhang.  Active prompting with chain-of-thought for large language models, May 2023.
  * Ding et al. (2022) Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Hai-Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juanzi Li, and Maosong Sun.  Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models, March 2022.
  * Du et al. (2024) Mengge Du, Yuntian Chen, Zhongzheng Wang, Longfeng Nie, and Dongxiao Zhang.  Llm4ed: Large language models for automatic equation discovery, May 2024.
  * Fernando et al. (2023) Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel.  Promptbreeder: Self-referential self-improvement via prompt evolution, September 2023.
  * Gao et al. (2023) Zhi Gao, Yuntao Du, Xintong Zhang, Xiaojian Ma, Wenjuan Han, Song-Chun Zhu, and Qing Li.  Clova: A closed-loop visual assistant with tool usage and update, December 2023.
  * Gu et al. (2022) Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang.  Ppt: Pre-trained prompt tuning for few-shot learning, March 2022.
  * Guo et al. (2024) Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang.  Connecting large language models with evolutionary algorithms yields powerful prompt optimizers, February 2024.
  * Gupta and Kembhavi (2022) Tanmay Gupta and Aniruddha Kembhavi.  Visual programming: Compositional visual reasoning without training, November 2022.
  * Helber et al. (2019) Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , 12(7):2217–2226, 2019.
  * Krause et al. (2013) Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In _Proceedings of the IEEE international conference on computer vision workshops_ , pages 554–561, 2013.
  * Lehman et al. (2022) Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Yeh, and Kenneth O. Stanley.  Evolution through large models, June 2022.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning, September 2021.
  * Li et al. (2004) Fei-Fei Li, Rob Fergus, and Pietro Perona.  Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories.  In _2004 conference on computer vision and pattern recognition workshop_ , pages 178–178. IEEE, 2004.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , January 2021.  doi: 10.18653/v1/2021.acl-long.353.
  * Liu et al. (2023a) Shihong Liu, Zhiqiu Lin, Samuel Yu, Ryan Lee, Tiffany Ling, Deepak Pathak, and Deva Ramanan.  Language models as black-box optimizers for vision-language models, November 2023a.
  * Liu et al. (2022) Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang.  P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks, March 2022.
  * Liu et al. (2023b) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang.  Gpt understands, too, October 2023b.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach, July 2019.
  * Ma et al. (2024) Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar.  Eureka: Human-level reward design via coding large language models, April 2024.
  * Maji et al. (2013) Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft.  _arXiv preprint arXiv:1306.5151_ , 2013.
  * Meyerson et al. (2024) Elliot Meyerson, Mark J. Nelson, Herbie Bradley, Adam Gaier, Arash Moradi, Amy K. Hoover, and Joel Lehman.  Language model crossover: Variation through few-shot prompting, May 2024.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space, September 2013.
  * Nilsback and Zisserman (2008) Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In _2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing_, pages 722–729. IEEE, 2008.
  * OpenAI et al. (2024) OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, C. J. Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph.  Gpt-4 technical report, March 2024.
  * Parkhi et al. (2012) Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar.  Cats and dogs.  In _2012 IEEE conference on computer vision and pattern recognition_ , pages 3498–3505. IEEE, 2012.
  * Pryzant et al. (2023) Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng.  Automatic prompt optimization with "gradient descent" and beam search, October 2023.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pages 8748–8763. PMLR, 2021.
  * Soomro et al. (2012) Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  _arXiv preprint arXiv:1212.0402_ , 2012.
  * Su et al. (2022) Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, and Jie Zhou.  On transferability of prompt tuning for natural language processing.  In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz, editors, _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3949–3969, Seattle, United States, July 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.naacl-main.290.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models, February 2023a.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models, July 2023b.
  * Vu et al. (2022) Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer.  Spot: Better frozen model adaptation through soft prompt transfer, March 2022.
  * Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding, February 2019.
  * Wang et al. (2020) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman.  Superglue: A stickier benchmark for general-purpose language understanding systems, February 2020.
  * Xiao et al. (2010) Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba.  Sun database: Large-scale scene recognition from abbey to zoo.  In _2010 IEEE computer society conference on computer vision and pattern recognition_ , pages 3485–3492. IEEE, 2010.
  * Yang et al. (2023) Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen.  Large language models as optimizers, December 2023.
  * Yao et al. (2024) Hantao Yao, Rui Zhang, and Changsheng Xu.  Tcp:textual-based class-aware prompt tuning for visual-language model, March 2024.
  * Zheng et al. (2023) Mingkai Zheng, Xiu Su, Shan You, Fei Wang, Chen Qian, Chang Xu, and Samuel Albanie.  Can gpt-4 perform neural architecture search?, August 2023.
  * Zhou et al. (2022a) Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16816–16825, 2022a.
  * Zhou et al. (2022b) Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  _International Journal of Computer Vision_ , 130(9):2337–2348, 2022b.
  * Zhou et al. (2023) Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba.  Large language models are human-level prompt engineers, March 2023.

##  Appendix A Appendix

###  A.1 More Experimental Details

Instructions used to query LLMs. The instruction used to query GPT-3.5 and
GPT-4 has been shown in Figure 3 of the main text. The instruction for
Llama2-7B-chat is provided in Figure 5.

The design of instruction for Llama2-7B is different from GPT-3.5 and GPT-4,
since we notice that the instruction following ability of Llama2-7B is weaker.
It is more likely to produce unexpected output. Even though we emphasized
desired way of responding to our query, the responses from Llama2-7B still
need proper post-processing to obtain the clean returned prompts.

System: You are a helpful, respectful and honest assistant capable of
proposing new prompts for users. User: Propose new prompts for user. Reply
with only the proposed short template, do not reply the loss and accuracy.
Keep every template under 8 words. Generate 3 templates that potentially have
better image recognition performance. I have a list of text templates with
their corresponding loss values and accuracy. They are used for image
classification with CLIP model. The templates are arranged in descending order
based on their loss value on training samples, where lower loss indicates
better quality. (Insert optimized prompts as optimization trajectories here.)

Figure 5: The instruction used to query Llama2-7B-chat in an iteration of
optimizing the prompt using LLM.

Detailed hyperparameter settings. The backbone models used by CoOp and TCP are
ResNet50 and ViT-B/16 respectively. The prompt length is set as 4 for both
CoOp and TCP. The training hyperparameters, such as epochs and learning rate,
are remained the same as the original methods. The number of training
iterations $M$ for Algorithm 1 is equal to the training iterations of the
original methods. We set the number of rounds $N$ as 3 and the iteration for
the gradient optimizer $m$ is set as 10 for CoOp, 30 for TCP. The prompt
length for NLU tasks is set as 8.

###  A.2 More Ablation Study

Distance function used for token space projection. The token space projection
operator in Eqn. 3 uses L2 distance to find the nearest discrete tokens for
continuous prompt embeddings. We also tried to using cosine similarity as
distance function. The results are provided in Table 6

Table 6: Ablation on distance function used for token space projection. Distance Function | EuroSAT | DTD | Oxford_Flowers
---|---|---|---
L2 | 71.39 | 56.70 | 86.64
Cosine | 71.36 | 56.19 | 87.09

Length of the prompt. We use a default prompt length 4 for our experiments. We
provide the result of our method with longer prompt in Table 7.

Table 7: Ablation on the length of the prompt. Prompt Length | EuroSAT | DTD | Oxford_Flowers
---|---|---|---
4 | 71.39 | 56.70 | 86.64
8 | 70.52 | 56.38 | 86.90


11institutetext: Skobeltsyn Institute of Nuclear Physics, Lomonosov Moscow
State University, Moscow, Russia

# Monitoring activities of satellite data processing services in real-time
with SDDS Live Monitor

Minh Duc Nguyen 11<EMAIL_ADDRESS>
###### Abstract

This work describes Live Monitor, the monitoring subsystem of SDDS â an
automated system for space experiment data processing, storage, and
distribution created at SINP MSU. Live Monitor allows operators and developers
of satellite data centers to identify errors occurred in data processing
quickly and to prevent further consequences caused by the errors. All
activities of the whole data processing cycle are illustrated via a web
interface in real-time. Notification messages are delivered to responsible
people via emails and Telegram messenger service. The flexible monitoring
mechanism implemented in Live Monitor allows us to dynamically change and
control events being shown on the web interface on our demands. Physicists,
whose space weather analysis models are functioning upon satellite data
provided by SDDS, can use the developed RESTful API to monitor their own
events and deliver customized notification messages by their needs.

##  1 Introduction

The ultimate goal of research in space weather is creating complex operational
magnetosphere-ionosphere models which allow predicting the radiation risk for
space satellites in different orbits, and also estimating the occurrence risk
of technological disasters due to magnetic storms and charged particle
precipitations. Currently, solutions to the problem involve collecting data
from all available satellites and ground stations, each of which measures a
limited number of parameters of space weather, and generating forecasts based
on the result of its analysis. Such task is challenging because data formats,
storage methods, datasets and values of measured parameters differ between
satellites. To achieve this goal an automated system called SDDS has been
created at SINP MSU. SDDS automatically connects to different data storage
servers of various satellites, downloads real-time data (both decoded txt-
files and binary telemetry) whenever available, decodes binary telemetry,
processes decoded data and stores it in a unified database.

Satellite data collected by SDDS is used as the primary source of input data
for space weather operational models created at SINP MSU. Since the
correctness and preciseness of space weather models depend much on the input
data, it is critical to be sure that data is correctly processed. It is also
important to detect errors in any step of the data processing cycle as soon as
possible so that they could be fixed quickly and their consequences could be
prevented in the future. Solution to the problem is monitoring all activities
of all components involved in data processing and sending alerts to
responsible people (satellite developers, system administrators, workers on
duty, etc.) so they could take measures in time. This paper is organized as
follows: in the second section, we consider several existing solutions to the
problem compared to our approach; in section 3, we give a more detailed view
of the overall architecture of the Live Monitor subsystem. A brief description
of the backend library is considered in section 4. Also in this section, we
explain how events are classified and how alerts are sent via different
mechanisms. Section 5 is dedicated to event representation on the web
interface. In section 6, we give a brief description of using Live Monitorâs
API to create a customized monitoring service. In conclusion, we give a short
resume of our completed work and describe our vision of the future
perspective.

##  2 Related works

Since SDDS is working on Linux, the first solution came to mind is using an
existing open source monitoring solution such as Zabbix zabbix , Nagios nagios
, or MMonit mmonit  to accomplish the task. But the detailed functional
analysis showed that these systems were mainly designed for monitoring IT
infrastructure (servers, routers, switches, etc.), system and network
services. Zabbix and Nagios support application monitoring but this function
is commercial. To monitor a custom service (or application) both Zabbix and
Nagios assume that one must write a wrapper which runs a number of tests to
check the service and produces standardized output data interpretable by the
interface of the solution. The server component of Zabbix or Nagios then call
this wrapper directly or via a client agent on a regular basis to check the
service. Such solutions are not suitable for SDDS because of several reasons.
Processing data of different satellites involves various components, the
components can change dynamically, and working states of each component also
differ. Writing a wrapper for each satellite would lead to a big amount of
source code to be maintained. Additional checks on a regular basis for too
many services would affect the overall performance of the operating system.
Thus, a lightweight event-driven monitoring mechanism would be better.

A better solution is requiring all components of SDDS to inform about their
current states in such a way that states could be treated uniformly. Each
event of each state can be logged as a record to a journal file. When the log
record is produced, we can use a message broadcasting server to either send
the record to a web interface to show it to an operator or deliver the record
via email or a messenger service directly to him. We have developed a library
in three major programming languages Python, PHP, and JavaScript solely for
this purpose.

While this approach can be applied perfectly to under development programs and
internal components of SDDS being maintained by us, the same is not true for
external programs that are used by SDDS to extract scientific data from the
raw binary data received from satellites. It is a real challenge because most
of these programs were written using different tools, both open source and
commercial, and programming languages. The explanation of this fact is that
data formats of different satellites differ and physicists use the tools they
know best to achieve their goals. Changing these programs is not reasonable,
mainly because either many of them were written such a long time ago that no
one knows exactly what was implemented inside, or they are such complicated,
so changes can lead to unexpected behaviors, resulting in erroneous data. To
overcome this difficulty, we have decided to run these programs inside a
wrapper which uses the developed library. Our final solution became the Live
Monitor subsystem that is used to monitor all activities of SDDS in real-time.
We have also developed a RESTful API so that physicists, whose space weather
models are working based on the data supplied by SDDS, can use it to create
their own monitoring scheme and deliver customized alerts to their customer.

##  3 Live Monitor’s architecture

Live monitor subsystem consists of the following components: a backend logging
library, RabbitMQ message broker, a RESTful API backend, a frontend UI
library. An illustration of the architecture is shown below in figure 1.

Figure 1: Live Monitor’s architecture

The backend logging library is a customization of the popular Python Logging
module. Every component of the SDDS system uses the library to inform about
its states while running. External programs are executed within a wrapper of
the SDDS global controller. When an external program crashes or returns an
error code, the wrapper informs about it using the library. Besides the
standard behavior which is writing short text messages in different log levels
to a log file, the library sends these text messages to the RabbitMQ message
broker [4] via a TCP socket. Messages are formatted using the STOMP protocol
[5]. RabbitMQ in turn broadcast received text messages to all active web
clients. When an error occurs during a certain state of data processing, the
library informs operators about the error by sending a corresponding error
description directly to the operators via the Telegram messenger service
and/or email. Notification features can be turned on and off by editing proper
configuration files.

Each data processing cycle is divided into stages. For example, a processing
cycle of data from Meteor-M2 satellites consists of the following stages:
connecting to data sources, downloading new raw data files, extracting
scientific binary data files from raw ones, processing binary data files,
adding processed data to the database, moving both raw and binary files to the
local storage. Processing stages of different satellites differ. Stages to be
displayed on the web interfaces of all components are customizable. One just
needs to define a list of all states in the component configuration file; the
library automatically does everything else. Since a satellite has a number of
instruments on board, the whole processing stage consists of sub-stages each
of which belongs to each instrument. An example is shown below in figure 2.

Figure 2: Live Monitor’s architecture

##  4 Live Monitor’s backend library

The primary goal of Live Monitorâs backend library is logging all events from
all components of SDDS as text messages to log files. If an SDDS’s component
needs to be monitored in real-time, one can change the proper property in the
component configuration file, and all logging messages will be passed to
RabbitMQ message broker and then delivered to all web clients. To monitor
external programs used for decoding scientific data from raw one, SDDS execute
them in a wrapper. The wrapper logs all start and stop execution points and
also error codes and output of these programs using the logging library.

The logging library supports four levels of logging messages: debug, info,
warning, and error. Error messages are logged when an error occurs during the
data processing cycle which could lead to incorrectly processed data or cause
a component failure. Warning messages are logged for minor errors that do not
affect the data correctness and normal functioning. Info messages are just
normal text descriptions of events during the data processing cycle. Debug
messages include diagnostic information that is helpful in failure
investigation. If the Live Monitor property is enabled in the configuration
file of a satellite, error messages will be passed directly to operators on
duty using the Telegram messenger service.

Another responsibility of Live Monitorâs backend is to control what should be
shown on the web interface during a data processing cycle of a satellite. When
a satellite is created, the satellite operator defines in the configuration
what stages the data processing cycle consists of and what instruments are
working on the satellite. A RESTful API has been developed to deliver this
kind of information to the frontend library flexibly and to allow remote work
with Live Monitor subsystem.

RabbitMQ was used as the message broker server to deliver short messages to
the frontend web client, primarily, because of its excellent documentation and
community support. Also, RabbitMQ provides libraries for all major programming
languages, which is a big advantage. In our case, the type of message
distribution is âpublish-subscribeâ with no guaranteed delivery. In this
scenario, RabbitMQ demonstrates an impressive performance and stability
according to the performance benchmark in the study stomp_performance . Also,
RabbitMQ requires less effort in configuration settings and implementation,
and itâs more reasonable compared with Apache ActiveMQ Apollo activemq  or
ZeroMQ zeromq .

##  5 Live Monitorâs frontend library

The main goal of the frontend library is to control how a data processing
cycle of a satellite should be shown on the web interface. When a user opens
the Live Monitorâs page of a satellite, the frontend library sends a request
to the backend according to the RESTful API to retrieve necessary information
of what should be shown. The answer from the backend is a JSON object that
consists of a number of stages and a number of instruments decoders involved
in the data decoding stage. After that, the frontend library establishes a
WebSocket connection with the RabbitMQ message broker. When a text message of
a stage arrives, the frontend library parses its content and changes the
visual appearance of the stage. Below is an example of the common âconnecting-
to-data-sourcesâ stage of each satellite.

Figure 3: Different states of the âconnecting-to-data-sourceâ stage of a
satellite

We use WebSocket protocol websocket  to deliver messages from RabbitMQ broker
to subscribed web clients because of its performance. The big advantage of
using WebSocket instead of AJAX long polling requests ajax  is that we can
push notifications to clients when an event occurs through a bi-directional
socket. It significantly reduces the workload of the server. In the case of
using AJAX long polling requests, each web client would have to poll the
broker regularly for new events. This approach leads to significant
performance overheads, especially when we have a large number of clients, due
to the unnecessary check data packets sending back and forth between the
broker and the clients.

We use Simple (or Streaming) Text Oriented Messaging Protocol (STOMP) as the
communication language between the broker and the clients. STOMP provides an
interoperable wire format so that STOMP clients can communicate with any STOMP
message broker to provide easy and widespread messaging interoperability among
many languages, platforms, and brokers. It suits our needs best. STOMP is a
very simple and easy to be implemented protocol, coming from the HTTP school
of design. It is very easy to write a client to get yourself connected. For
example, one can use Telnet to log into any STOMP broker and interact with it.
We have managed to write three STOMP clients (in Python, PHP, JavaScript) in
just a couple of hours and to integrate them to the RabbitMQ broker.

##  6 Customized monitoring service based on Live Monitor

At the moment of writing, Live Monitor is used mostly to monitor data
processing of satellites. However, it is also possible for one to create his
own monitoring service based on Live Monitorâs API.

First of all, one must be a registered user of our SDDS system to be
authorized to use Live Monitorâs API. To create a customized monitoring
service, a user must send a POST request with a proper JSON object body which
consists of a service name, all its stages, and component names involving in
each stage. When such request is received, Live Monitorâs backend checks the
request body for correctness. If the request is correct, the backend will
create a RabbitMQ channel dedicated only to delivering messages of the
requested service and a configuration file. As the answer, the user receives a
JSON object which consists of a URL to the page of the requested service, a
pair of login and password to be used in user’s program to deliver messages to
the RabbitMQ message broker. To control the behaviour of the service, one must
send proper requests to the Live Monitorâs backend. Currently, the following
operations are supported by Live Monitor:

  * •

create/delete a customized monitoring service;

  * •

switch a monitoring service on/off;

  * •

switch Telegram message delivery for a service on/off.

##  7 Conclusion

SDDS is the primary satellite data processing system being used at SINP MSU.
The Live Monitor subsystem of SDDS system has been deployed from the first day
of operation. During 12 months of operation Live Monitor helped us react to
any occurred event in data processing in time. Since all stages of data
processing are shown in details, Live Monitor allows us to identify and
localize the scope of a problem immediately and hence prevent or fix them
quickly. In future, we plan to support more operations to control the behavior
of monitoring services, such as manipulating a stage name or a component name
of a stage, controlling the monitoring behavior of each stage and component of
a service independently, and so on.

## Acknowledgments

We would like to thank Dr. Vladimir Kalegaev for helpful discussions on
satellite data processing and clear problem statements. Also, we would like to
thank Dr. Alexander Kryukov for his support in all aspects. This project is
supported by RSF grant #16-17-00098.

## References

  * (1) Zabbix SIA, _Zabbix Documentation_ (https://www.zabbix.com/documentation/3.0/manual, Version 3.0)
  * (2) Ethan Galstad, Nagios Core Development Team, and Community Contributors _Nagios Core Documentation_ (https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/4/en/index.html, October 4th, 2016)
  * (3) Tildeslash Ltd, _M/Monit User Manual_ (https://mmonit.com/documentation/mmonit_manual.pdf, Version 3.7.1)
  * (4) Pivotal Software, Inc. _RabbitMQ Server Documentation_ (https://www.rabbitmq.com/admin-guide.html, Version 3.6.9)
  * (5) _STOMP Protocol Specification_ (https://stomp.github.io/stomp-specification-1.2.html, Version 1.2)
  * (6) Hiram Chirino, _Stomp Benchmark_ (https://github.com/chirino/stomp-benchmark)
  * (7) The Apache Software Foundation, _ActiveMQ Apollo User Manual_ (https://activemq.apache.org/apollo/documentation/user-manual.html, Version 1.7.1)
  * (8) iMatix Corporation, _ZeroMQ - The Guide_ (http://zguide.zeromq.org/page:all, Version 2.2)
  * (9) Internet Engineering Task Force, _The WebSocket Protocol_ (https://tools.ietf.org/html/rfc6455, December 2011)
  * (10) J. J. Garrett, _Ajax: A New Approach to Web Applications_ , (https://web.archive.org/web/20080702075113/http://www.adaptivepath.com/ideas/essays/archives/000385.php, February 18th, 2005)