#  AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect,
and Learn

Difei Gao, Lei Ji, Luowei Zhou, Kevin Qinghong Lin, Joya Chen,
Zihan Fan, Mike Zheng Shou

Show Lab, National University of Singapore,
https://showlab.github.io/assistgpt/ Corresponding author.

###### Abstract

Recent research on Large Language Models (LLMs) has led to remarkable
advancements in general NLP AI assistants. Some studies have further explored
the use of LLMs for planning and invoking models or APIs to address more
general multi-modal user queries. Despite this progress, complex visual-based
tasks still remain challenging due to the diverse nature of visual tasks. This
diversity is reflected in two aspects: 1) Reasoning paths. For many real-life
applications, it is hard to accurately decompose a query simply by examining
the query itself. Planning based on the specific visual content and the
results of each step is usually required. 2) Flexible inputs and intermediate
results. Input forms could be flexible for in-the-wild cases, and involves not
only a single image or video but a mixture of videos and images, e.g., a user-
view image with some reference videos. Besides, a complex reasoning process
will also generate diverse multimodal intermediate results, e.g., video
narrations, segmented video clips, etc. To address such general cases, we
propose a multi-modal AI assistant, AssistGPT, with an interleaved code and
language reasoning approach called Plan, Execute, Inspect, and Learn (PEIL) to
integrate LLMs with various tools. Specifically, the Planner is capable of
using natural language to plan which tool in Executor should do next based on
the current reasoning progress. Inspector is an efficient memory manager to
assist the Planner to feed proper visual information into a specific tool.
Finally, since the entire reasoning process is complex and flexible, a Learner
is designed to enable the model to autonomously explore and discover the
optimal solution. We conducted experiments on A-OKVQA and NExT-QA benchmarks,
achieving state-of-the-art results. Moreover, showcases demonstrate the
ability of our system to handle questions far more complex than those found in
the benchmarks.

##  1 Introduction

Large language models (LLMs) [1, 2, 3, 4], especially ChatGPT [5], have made
remarkable progress in recent months, significantly advancing the field of
developing AI assistants. Despite these advances, a single LLM serving as an
AI assistant still exhibits inherent limitations in certain abilities, such as
understanding visual environments and comprehending complex tasks, which
restrict their utility in real-world applications. To address these
shortcomings, a promising solution is to explore the integration and
collaboration of multiple domain experts e.g., pretrained models or APIs, to
tackle complex tasks. Numerous efforts have been made in this direction. Some
works [6, 7, 8] utilize language as a bridge and transform the visual input
into pure texts using foundational visual models, such as captioner [9, 10,
11], object detectors [12, 13, 14], and OCR models [15, 16]. Subsequently, the
extracted texts are fed into LLMs for reasoning tasks like question-answering.
Nonetheless, as for complex visual scenarios such as a long-form video with
complicated scene switching, as shown in Fig. 1, the generated texts may go
well beyond the query requirements. This can lead to an abundance of
superfluous information while crucial details relevant to the query may be
omitted.

Figure 1: In-the-wild example of AssistGPT. AssistGPT can reason in an
interleaved language and code format. Given a query input and visual inputs,
AssistGPT plans the problem-solving path in language, using structured code to
call upon various powerful tools. The Inspector, part of the system, can
manage visual inputs and intermediate results, assisting the Planner to invoke
tools. Meanwhile, the Learner can assess the reasoning process and collect in-
context examples.

Some other concurrent works propose decomposing user queries into subtasks and
plan to sequentially call external models or APIs to answer them. Currently,
there are two branches of methods. The first one is language-based planning
[17, 18, 19, 20]. For instance, HuggingGPT and Chameleon [17, 19] propose
using an LLM as a controller, managing and organizing the cooperation of
expert models. Another branch of work is code-based planning [21, 22, 23].
ViperGPT [21] proposes to use Codex to write the Python code to call visual-
related APIs for handling multi-modal tasks. These approaches allow for
invoking the models only when necessary, which allows models to only output
useful information and optimize the use of computation resources.

Despite this progress, addressing high-level queries is still challenging.
Specifically, current questions in existing benchmarks usually directly imply
how to plan the reasoning. For example, for questions like "What is the red
object used for?", no matter what the image is, the reasoning steps are
relatively fixed, i.e., recognize the red object, then figure out its
function. However, for more complex questions, there could be diverse reason
paths. For example, for question _“How much black pepper should I use for 700g
beef?”_ in Fig. 2, the variations in the presentation of relevant information,
whether it’s in the form of subtitles, actions, text within videos, or a
combination of these, can result in distinct reasoning paths. Therefore, as
shown in Fig. 2, once a reason-only approach makes a mistake, it becomes
difficult for it to self-correct.

Figure 2: Comparison of PEIL and two mainstream reasoning methods in multi-
modal tasks.

Similar approaches are already proposed in the NLP field, such as ReAct [24]
and ToolFormer [25]. However, there is a unique challenge in multimodal tasks:
_How to handle non-textual intermediate result?_ For ReAct and ToolFormer, the
outputs of external models can be directly fed into the Planner and passed to
subsequent models. While the intermediate results obtained in multimodal tasks
usually are cropped regions for the image grounding module, and segmented
video clips for the temporal location module, as shown in Fig. 1. In complex
cases, it is hard for Planner to manage which information should be fed into
the next module.

In this paper, we propose a multi-modal AI Assistant system, named AssistGPT
(The design of our model’s icon is inspired by the HAL 9000 from the movie “A
Space Odyssey”, a fictional artificial intelligence character), with
interleaved language and code reasoning method, inheriting the advantages of
flexible reasoning in ReAct and robust tool invocation in Program-based
planning. Specifically, our system consists of four parts, Planner, Executor,
Inspector, and Learner. We show how our system works in Fig. 1. Similar to
ReAct, the Planner thinks about what needs to be done next based on the
current reasoning progress and invoking external models. What sets our method
apart is the use of formatted code to invoke external models. The Executor
wraps external tools into a uniform input and output format, allowing the tool
to be invoked with structural commands. Simultaneously, we have also proposed
an Inspector, which manages visual inputs and intermediate results during the
reasoning process. It provides the Planner with summaries and metadata of all
currently available visual materials. The combination of the Inspector and the
Executor allows the model to efficiently implement complex reasoning.
Moreover, it is challenging for the model to ensure correct reasoning in a
zero-shot scenario. The Planner might output invalid code or unreasonable
paths. To enable the system to continuously improve, we proposed the Learner,
which checks whether the prediction process is reasonable or judges the
correctness of the predicted results based on annotations. It allows the
system to try multiple times and record successful examples as in-context
examples.

The current version of AssistGPT integrates 10+ tools for different functions,
including image detection, captioning, region grounding, temporal grounding,
OCR Module, object enumeration, speech-to-text, etc. By combining these
functionalities, AssistGPT can accomplish a wide range of multi-modal tasks
which are still hard for existing systems.

In summary, our contributions are as follows: 1) We constructed a general
multimodal AI assistant that can accomplish diverse visual-related tasks with
the cooperation of multiple models. 2) We propose a new compositional
reasoning method that reasons over in an interleaved language and code manner.
A simple learning mechanism is also proposed to improve the AssistGPT’s
ability in planning. 3) We showcase AssistGPT’s capabilities not only the
benchmark results but also some realistic applications for processing complex
images and long-form videos, understanding high-level queries, and handling
flexible inputs.

##  2 Related Work

Multi-modal Systems. Prior to the advent of LLM, remarkable works were done to
design multimodal models for one or several specific tasks, such as focusing
on visual appearance [26, 27, 28, 29, 30, 31], visual-related knowledge [32,
33, 34, 35, 36, 37], action [38, 39, 40], ego-centric videos [41, 42, 43, 44],
instructional videos [45, 46, 47], scene text [48, 49, 50, 51], etc. They have
achieved commendable results in specific tasks, however, their
generalizability is relatively limited, making it challenging to address more
complex and diverse questions in real-world scenarios.

Recently, two types of strategies are proposed for developing a general multi-
modal system. One is pre-training LLM to support visual features as
conditional inputs. The representative models are GPT-4 [52], PaLM-E [53],
BLIP-2 [54], and Mini-GPT4 [55]. Despite these methods being capable of
directly processing multi-modal input, they still exhibit limitations in
addressing advanced functional needs, such as image spatial grounding, long-
form video grounding, and audio comprehension. Additionally, the computational
cost of scaling these models can be extremely high. The alternative strategy
aims to combine multiple models or APIs to accomplish complex multi-modal
reasoning. For instance, models like the Socratic model [6] and Visual ChatGPT
[8] achieve this by connecting ChatGPT with image generation models.
HuggingGPT [17] combines a variety of Huggingface models with LLMs. ViperGPT
[21] employs Codex [56] to call visual APIs via Python programming. Our
AssistGPT falls into the second category by combining and invoking various
modules for multi-modal reasoning, but we propose a new framework PEIL for
integrating external tools and models.

Compositional Reasoning. Compositional reasoning methods in the field of
visual question answering usually decompose questions into several subtasks,
each addressed by a specific module. This kind of method offers strong
interpretability due to its modular structure and the clear division of
responsibilities among the individual components. This idea was initially put
forward by [57]. Subsequently, [58, 59] introduced an end-to-end variant based
on LSTM and CNN. Traditional compositional reasoning methods are limited by
language models’ parsing capabilities, often requiring ground-truth question
decomposition or reinforcement learning for optimal module usage.

With the advent of LLMs, question decomposition can be accomplished remarkably
well in a zero-shot manner. Chain-of-thought prompts [60], Toolformer [25],
and ReAct [24] enable models to plan how to solve an NLP problem. HuggingGPT
[17] and ViperGPT [21] are multi-modal systems that use LLM to parse a
question into a series of reasoning steps. However, for complex queries, the
model needs to determine the subsequent steps based on not only questions but
also visual inputs or feedback from previously executed modules. MMReAct [61]
introduced the idea of ReAct to a multi-modal system to overcome it, while it
is still under development and hasn’t demonstrated its effectiveness on the
benchmark. Previous methods reason over either language reasoning or code, and
as stated in the introduction, both have certain shortcomings. Our work first
proposes an interleaved language and code reasoning manner which can better
handle general queries and complex visual inputs.

Learning Schemes for Modular System. Early modular models primarily employed
end-to-end Reinforcement Learning (RL) to train each module’s planning and
acting from scratch. While this approach is practical for lightweight models,
RL can introduce substantial overhead for systems where each module is an LLM.
Toolformer [25] proposes a self-supervised technique that optimizes planning
requiring only a handful of demonstrations for each API. Specifically,
Toolformer attempts various APIs to find successful examples and then fine-
tunes the model. In contrast, we propose a straightforward mechanism in the
multi-modal field, which can guide the system to retry and preserve the
successful explorations as in-context examples.

##  3 AssistGPT

Overview. AssistGPT is a general multi-modal AI assistant system that can
dynamically engage various tools in an interleaved language and code manner.
Specifically, given a general language query and reference images or videos as
inputs, the goal of AssistGPT is to generate the desired answer. As shown in
Fig. 3, AssistGPT is achieved by cooperation with four core modules: (a)
Planner, (b) Executor, (c) Inspector, and (d) Learner. The Planner $\S$ 3.1
aims to control the whole reasoning process, with the Executor $\S$ 3.2
supplying valuable feedback to Planner by executing external tools. The
Inspector $\S$ 3.3 manages the input and intermediate results and assists the
Planner in feeding proper content to the Executor. The Learner $\S$ 3.4 is
capable of assessing the system performance and record successful explorations
as in-context examples. In the following sections, we will go through each
module in detail.

Figure 3: Diagrammatic illustration of AssistGPT system. It consists of four
core modules:
⚫ Planner: control the whole reasoning process; ⚫ Executor: execute external
tool and return feedback to Planner; ⚫ Inspector: manage the input and
intermediate outcomes; ⚫ Learner: assess the system performance and record
successful trials as in-context examples.

###  3.1 Planner

The ⚫ Planner employs a highly intelligent LLM i.e., GPT-4 [52] as the central
brain to control the global reasoning planning. It begins the planning process
by taking inputs from three types of information: an Instruction Prompt
consisting of the [Tool Set Illustration] and [In-Context Example]111The
successful trials recorded by Learner, will be introduced later., Input Query,
and the Summary of Visual Inputs created by ⚫ Inspector.

Then it generates the appropriate output for the next step, which consist of
two parts: Thought: a language phrase indicates what should be done next.
While it doesn’t affect the module or API call directly, it aids the LLM
planning procedure. Action: a structural string obeys the pre-defined template
provided in the instructions. It specifies which external tool to call and
what arguments to input, e.g., caption("what color is the car?", visual[0]).

After each time ⚫ Executor call to an external tool, the tool returns outputs
in the form of natural language, which we refer to as Observation. If the tool
generates an intermediate outcome, e.g., a segmented video, our ⚫ Inspector
will store it and generate a Summary for it. Both the Observation and Summary
will be fed to the ⚫ Planner to guide the planning of the next step. The
following sections will introduce more details of Action, Observation, and
Summary.

Currently, we integrate 13 functional tools in AssistGPT to power multi-modal
assistance, as shown in Tab. 1. These modules can be mainly categorized into
three types:

  * •

Descriptor: To effectively comprehend and utilize the data derived from
intricate multimodal environments e.g., image, video, audio, and text, we
employ a variety of fundamental models as basic descriptors for perception.
These models, including (a) Image Caption, (b) Video Narration, (c) Object
Detection, (d) Text Detection, and (e) ASR Translation, enable us to extract
enough information from diverse source, thus enhancing our understanding of
the multimodal sceanrio.

  * •

Locator: As the saying goes, a picture is worth a thousand words. Images or
videos typically contain a wealth of information - objects, attributes,
actions, events, and so on. However, the abundance of information can
sometimes obstruct our problem-solving process. One crucial solution is to
pinpoint the most crucial and valuable information from the rich sea of
visual, textual, and audio data This part incorporates several modules such as
the (f) Region Ground, (g) Narration Ground, (h) Text Ground, and (i) Subtitle
Ground.

  * •

Reasoner: The initial two sets of tools primarily deal with collect and
identify of data, whereas the third set focuses on reasoning, utilizing the
extracted information and external knowledge. This part incorporates modules
such as (j) Knowledge Reason, (k) Narration Reason, (l) Subtitle reason, and
(m) Temporal Reason modules. These modules primarily utilize LLM at their core
by taking different types of information and prompts as inputs or a simple
program.

Table 1: Module used in AssistGPT. A module may have different models,
separated by a slash (/).

Module Usage | Core Model | Input | Output |  | Module Usage | Core Model | Input | Output
---|---|---|---|---|---|---|---|---
(a) Image Caption | BLIP series [9, 10, 11] | T, I | T |  | (h) Text Ground | Program + SSA [62, 63] | T, I | T, I
(b) Video Narration | BLIP series [9, 10, 11] | T, V | T |  | (i) Subtitle Ground | GPT [5] | T, Sub. | T, V
(c) Object Detection | G. Dino [64] / GLIP [13] | T, I | T |  | (j) Knowledge Reason | GPT [5] | T | T
(d) Text Detection | Google OCR | I | T |  | (k) Narration Reason | GPT [5] | T, Nar. | T
(e) ASR Translation | Whisper [65] | A | T |  | (l) Subtitle Reason | GPT [5] | T, Sub. | T
(f) Region Ground | OFA [66] | T, I | T, I |  | (m) Temporal Reason | Rule-based | T, V | T, V
(g) Narration Ground | GPT / CLIP [67] | T, Nar. | T, V |  |  I: Image, V: Video, T: Text, A: Audio, Nar.: Narration, Sub.: Subtitle

###  3.2 Executor

The ⚫ Executor takes the code generated by the ⚫ Planner as input, then call a
module to produce the output by carrying out three steps to obtain the final
result. These steps include validation check, module execution, and post-
processing, as shown in Fig. 3.

  * •

Validation Check: Even powerful LLM like GPT-4 can sometimes generate illegal
code. For example, an image caption module accept a long video as input. We
have designed a legality check for each module to determine whether the code
is executable. Moreover, if code includes errors, we do not interrupt the
entire reasoning process. Instead, we return an error message as the output
code to the ⚫ Planner, allowing it to optimize the planning process in real-
time.

  * •

Module Execution: We standard various modules or APIs into a unified interface
using the code-style template i.e., [Module_Name](<text_query>,
<visual_index>). Each module is designed to accept multiple text queries and
visual data (images or videos) as input. In each standarded module, we provide
instructions on its function and the requirements of the argument, which is
used for [Tool Set Illustration] in ⚫ Planner. Additionally, for the sake of
simplicity and accuracy in Planning, the generated code is simplified. Later,
a simple rule-based function will map it to the executable codes and then
execute it to obtain the final result.

  * •

Post-processing: For all modules, the generated results will be translated
into a language format to inform the ⚫ Planner about the outcome, as the
Observation part illustrated above. For instance, for the Narration Ground
module, the model will return whether it has found the relevant segment. If
so, output the start and end times of the segment. Additionally, many Ground-
related modules will send their segmented video or cropped image region to the
subsequent visual outcome manager i.e., ⚫ Inspector.

###  3.3 Inspector

The objective of the ⚫ Inspector is to manage the visual inputs provided by
the user and the intermediary results produced by our system to assist the ⚫
Planner in deciding which source should be directed to which module.
Specifically, the ⚫ Inspector records the metadata of each visual element,
which includes its type (image or video), source (provided by the user or
generated by the system), and a brief description of the content (obtained
from the caption model, or the title of an online video). For videos, there is
some additional metadata, such as the duration of the video, whether it
contains audio and subtitles. The ⚫ Inspector monitors the inputs from the
user and the outputs from the ⚫ Executor. As soon as a new visual element is
received, it appends the metadata, noted as Summary in the above, to the
reasoning history of the ⚫ Planner. With the cooperation of the ⚫ Planner, ⚫
Executor, and ⚫ Inspector, our system can generate answers to difficult
queries with complex visual inputs.

###  3.4 Learner

Despite the robot generalization capabilities of LLMs, they can still easily
encounter errors when dealing with multi-modal type queries. Thus, it is
essential for an AI assistant to have self-valuate mechanism. To achieve this
goal, we hope that the model can self-check the reasonableness of its output.
On the other hand, when ground truth is available, we intend to gather
successful prediction instances as in-context examples. Specifically,
AssistGPT will repeatedly attempt to provide the answer when the response is
not satisfactory until either passes the self-check, or correct answer is
given (when ground truth is available) or a predefined maximum number of
attempts is reached. ⚫ Learner includes an evaluator implemented by the LLM,
which operates in two modes: self-assessment and ground-truth comparison.
These modes are activated depending on the availability of ground truth, and
we discuss the two of them separately.

  * •

Self-assessment mode is activated when there is no user feedback or ground
truth available. It takes the reasoning trace and the results of each step as
input, allowing GPT to assess whether the reasoning is complete, consistent,
and adhere to the required format.

  * •

Ground-truth comparison mode is activated when annotators provide ground
truth. In this mode, GPT evaluates whether the AssistGPT’s prediction is
semantically consistent with the provided ground truth.

Furthermore, ⚫ Learner encourages to keep trying until it receives positive
feedback or reachs the maximum number of attempts. After conducting $N$ times
explorations, several outcomes may arise:

  * •

No adjustments required: If the model delivers the correct answer on its
initial attempt, this suggests that AssistGPT can well-solve the current
question effectively. Therefore, no improvement is required.

  * •

Plan Revision: If the model produces the correct answer after making $n$
attempts, where $1<n\leq N$, this implies that there is room for improving the
model’s planning capabilities. Therefore, we save the successful reasoning
trace to [In-Context Memory Bank]. Consequently, when the model comes across a
similar query in the future, it can use this as an in-context example.

  * •

Function Updates: If the model still fails to provide the correct answer even
after $N$ attempts, it is highly probable that the problem resides in a
specific module or API rather than the planning process. It may necessitate
incremental updates to the module. We will leave this for future work.

##  4 Experiments

###  4.1 Experimental Setting

Datasets. Our system is evaluated on A-OKVQA [33] and NExT-QA [68] benchmarks
designed to test comprehensive multimodal capabilities, including visual
facts, commonsense, temporal sequences, causality, etc. A-OKVQA [33] is an
innovative benchmark for knowledge-aware visual question answering with 25K
questions that demand a high-level comprehension of commonsense and world
knowledge. These questions in A-OKVQA go beyond the information contained in
the image and cannot be answered solely by querying a knowledge base. Besides,
the question is diverse, spanning a wide range of domains such as commonsense
reasoning, visually-grounded, knowledge-based, and physical understanding. In
our experiments, we assess the model performance under the in-context learning
setting on the validation set, which consists of 1,145 questions. NExT-QA [68]
is a benchmark for evaluating the AI system’s causal reasoning, temporal
action reasoning, and rich object interactions in video question answering.
NExT-QA has a total of 5,440 videos, with averaging 44 seconds in length, and
approximately 52K manually annotated question-answer pairs. In our
experiments, we assess the model performance under the in-context learning
setting on the validation set, which consists of 4,996 questions.

Implementation Details. In the following experiments, we use GPT-4 API
provided by OpenAI [52] as Planner. In the A-OKVQA experiments, we set Caption
Module as BLIP2 or InstructBLIP (abbreviated as Ins.BLIP), use the Gounding
Dino for the Object Detection model, and Google OCR for Text Detection. For
the NExT-QA experiments, our Video Narration Module is based on InstructBLIP
Vicuna-7B [11]. Our experiments are performed on 4 A5000 GPUs.

###  4.2 Quantitative Results

Table 2: Comparison of SOTAs on A-OKVQA dataset. D.A. and M.C. indicate direct
answer and multi-choice. ICL: In-context Learning. ZS: Zero-shot inference.

| Model | D. A. | M.C.
---|---|---|---
Sup. | LXMERT [29] | 30.7 | 51.4
KRISP [36] | 33.7 | 51.9
GPV-2 [69] | 48.6 | 60.3
InstructBLIP Vicuna-7B [11] | 64.0 | 75.7
ICL | PromptCap [70] | 56.3 | 73.2
AssistGPT (BLIP2 FlanT5XL [54]) | 42.6 | 73.7
AssistGPT (Ins.BLIP Vicuna-7B) | 44.3 | 74.7

Table 3: Ablation study of our AssistGPT on A-OKVQA dataset. Ins.BLIP used
here is the pre-trained version.

LLM | Model | D.A. | M.C.
---|---|---|---
- | Ins.BLIP | 13.4 | 53.8
| Ins.BLIP + GPT-4 | 27.9 | 55.2
AssistGPT | Reason only | 28.8 | 65.9
ReAct | 30.1 | 68.2
PIE | 32.4 | 72.4
PEIL w. Self-Check | 41.2 | 74.2
PEIL w. GT-Check | 44.3 | 74.7

Comparison with State-of-the-arts. From the results in Table 3, it can be seen
that in the multi-choice track, our two versions of AssistGPT (i.e., with
light-weight BLIP2 FlanT5XL and more powerful Ins.BLIP Vicuna-7B) achieve the
best among all current methods on in-context learning setting. It’s worth
noting that we use a pre-trained version of InstructBLIP, which performs at
53.3%, as shown in Table 3. When integrated into our system, it can enhance
its performance to the level of fine-tuning model. For direct answer
questions, while our performance on it may not match that of recently proposed
models, it is still comparable to previous supervised SOTA, like GPV-2 [69].

Our performance on direct answers did not surpass previous methods. The main
reason is that for open-ended questions, models relying on LLM tend to output
complete phrases rather than a single word as the final answer, even when we
prompt them to provide as concise an answer as possible. For instance, for a
given question "What flag is represented on the wall?", AssistGPT outputted
the answer, "United States flag", but the correct answer does not include the
word "flag", therefore it’s deemed incorrect. This type of error is very
common in AssistGPT. In the appendix, we show more examples to analyze the
failure cases. Moreover, compared to the SOTA method, PromptCap [70], it
specifically trained a caption model toward generating captions for A-OKVQA,
which is also the reason for its good performance, while our system is more
general.

From the results in Table 4, AssistGPT achieved higher performance than
recently proposed supervised methods, demonstrating the effectiveness of our
approach. We can see that our model’s performance mainly shows a more
promising improvement in Causal and Descriptive questions, mainly due to our
model continuously obtaining detailed information related to the question from
the videos. Moreover, our method does not perform well on temporal questions.
The main reason for this is that there are relatively few open-world temporal
grounding models available, and mainstream work still involves fine-tuning on
closed-world datasets. Therefore, we have to use the image captioner
InstructBLIP with GPT-4 to achieve temporal grounding. The effect is not as
good as that of fine-tuned models but has a more generalization ability.
Furthermore, its performance is also very close to recent concurrent work,
ViperGPT [21]. The ViperGPT is a little bit superior to ours, possibly because
it has designed a sophisticated rule-based method, iteratively checking
whether objects appear in the frame to perform temporal grounding.

Ablation Study. We have designed several variants of AssistGPT to test the
effectiveness of our proposed method. The most basic baseline is InstructBLIP
(note that all following models are using Vicuna-7B version), which is the
main source of visual information in AssistGPT. Since InstructionBLIP cannot
necessarily output the answer in the required format, we design a variant,
InstructionBLIP+GPT-4 allows GPT-4 to further refine the output of
InstructionBLIP. The Reason-only model directly plans all the steps the models
need to run, similar to previous works [17]. The ReAct model is capable of
executing language-based ReAct. However, without Inspector and Code-like
invocation forms, a subsequent model can only accept the output of the
previous model, which is similar to [61]. We also ablate the Learner, which
has three versions, PIE (i.e., w/o. Learner), PEIL w. Self-Check and PEIL w.
GT-Check.

From the results in Table 3, we can see that the Reason-only model, which
plans all the steps the models need to execute, showed a notable improvement
in D.A. and M.C. This indicates that integrating multiple models can enhance
model performance. The ReAct model, despite not having Inspector and Code-like
invocation forms, showed a further improvement in both metrics, surpassing the
Reason-only model. This suggests the effectiveness of ReAct manner. But
involving our interleaved language and code, i.e., PIE, brings a more
significant improvement on M.C. Finally, the two variants of PIE with partial
ablations, PEIL w. Self-Check and PEIL w. GT-Check, scored the highest on both
tracks, showing the effectiveness of the Learner. The Learner shows a more
significant improvement on D.A. tracks because models on D.A. often fail to
output extremely short answers as required by A-OKVQA. The Learner can
mitigate it by collecting in-context examples.

Table 4: Comparison of our AssistGPT with SOTAs on NExT-QA dataset.

| Method | Causal | Temporal | Descriptive | All
---|---|---|---|---|---
Sup. | HGA | 44.22 | 52.49 | 44.07 | 49.74
VQA-T [71] | 49.60 | 51.49 | 63.19 | 52.32
ATP [72] | 53.10 | 50.20 | 66.80 | 54.30
VGT [73] | 52.28 | 55.09 | 64.09 | 55.02
MIST [74] | 54.62 | 56.64 | 66.92 | 57.18
ICL | ViperGPT [21] | - | - | - | 60.00
AssistGPT | 60.02 | 51.38 | 67.26 | 58.36

###  4.3 Qualitative Results

Figure 4: Qualitative results on A-OKVQA (Question 1) and NExT-QA dataset
(Question 2).

In Fig. 4, we visualize some prediction cases from A-OKVQA (Question 1) and
NExT-QA (Question 2). From both examples, it can be seen that AssistGPT can
decompose the question into reasonable sub-tasks and then complete them step
by step, ultimately obtaining the final answer. Moreover, due to the
interleaved code and language reasoning method, the model can effectively
invoke the necessary content as input. From the reasoning process of Question
1, we can also see AssistGPT’s self-correction ability. When the visual model
output unsatisfactory results, AssistGPT can dynamically invoke other modules,
like the ground module to reason over another path. In addition, for Question
1, the model’s first attempt did not yield effective results, and it will
autonomously optimize the plan because it did not pass the self-check. In
addition, we also present the result of the reason-only baseline. It first
calls InstructBLIP to output a caption, then uses GPT-4 for inference. Since
the information from the caption does not meet the requirements, resulting
incorrect results. However, once the prediction fails, the model does not have
a way to self-optimize. It’s worth mentioning that the most significant
feature of our method is that it can solve more complex problems than those in
the benchmark, as the example in Fig. 1. We show more in-the-wild examples in
the Appendix.

##  5 Conclusions and Limitations

In this paper, we propose a novel multi-modal assistant system named AssistGPT
that leverages an interleaved code and language reasoning approach, namely
Plan, Execute, Inspect, and Learn (PEIL). This innovative system integrates
LLM with various tools to address the challenges posed by complex visual-based
tasks. Our experimental results on A-OKVQA and NExT-QA benchmarks demonstrate
AssistGPT’s effectiveness. Furthermore, we showcase our system’s ability in
handling diverse and intricate real-world scenarios. Our system also has some
limitations. Our approach does not propose an end-to-end updating solution,
which is crucial when the tools used make mistakes. Another limitation is that
the planning process requires an extensive explanation of tools, resulting in
a relatively large overhead, which could be improved by distilling a smaller
size planner.

## Appendix

In the appendix, we provide additional details for the main paper:

  * •

More discussion with existing modular systems in Sec. A

  * •

More details of AssistGPT in Sec. B.

  * •

More qualitative results of A-OKVQA in Sec. C.

  * •

More in-the-wild examples in Sec. D.

##  A Discussion with Existing LLM-driven Modular Systems

Table 5: Comparison of existing LLM-driven modular systems. We compare
existing methods from four dimensions: Task Focus, Reasoning Method, Source
Management (how they manage input and intermediate results), and whether they
have learning capabilities. The term "ReAct" in the table does not strictly
refer to using the ReAct [24], but rather it denotes planning and executing
concurrently.

Model | Task Focus | Reasoning | Source Management | Learning
---|---|---|---|---
NLP | Image | Video | Format | ReAct | Input format | Method
Toolformer [25] | ✓ | ✗ | ✗ | lang. & prog. | ✓ | text-only | - | ✓
WebGPT [75] | ✓ | ✗ | ✗ | program | ✓ | test-only | - | ✓
Visual ChatGPT [8] | ✗ | ✓ | ✗ | language | ✗ | multi. V. | Filename | ✗
ViperGPT [21] | ✗ | ✓ | ✓ | program | ✗ | single V. | Variable | ✗
VisProg [22] | ✗ | ✓ | ✗ | program | ✗ | single V. | Variable | ✗
MM-ReAct [61] | ✗ | ✓ | ✓ | language | ✓ | multi V. | Filename | ✗
Chameleon [19] | ✓ | ✓ | ✗ | language | ✗ | single V. | Cache update | ✗
HuggingGPT [17] | ✗ | ✓ | ✓ | language | ✗ | multi V. | Filename | ✗
AssistGPT (ours) | ✗ | ✓ | ✓ | lang. & prog. | ✓ | multi V. | Inspector | ✓

In Table 5, we compare the existing LLM-driven modular systems with our
AssistGPT from four perspectives:

Task Focus. From the perspective of Task Focus, there are currently three
works that can handle videos: Hugging GPT [17], MM-ReAct [61], and ViperGPT
[21]. Hugging GPT and MM-ReAct merely demonstrate their capabilities in
handling videos through a few simple examples (thus we mark them with orange
checkmarks ✓). For instance, Hugging GPT exhibits its video generation
feature, while MM-ReAct showcases its ability to perform tasks such as
summarization and localization based on subtitles. However, these methods have
not been validated on any benchmark. ViperGPT can handle questions based on
visual content. Compared to these works, AssistGPT is capable of dealing with
more complex and general video question-answering tasks, including
understanding subtitles, visual content, and OCR, and demonstrating long video
comprehension capabilities.

Reasoning. In terms of reasoning, existing Multi-modal models primarily adopt
a reason-only style, that is, directly deriving the solution steps based on
the question. This approach struggles with handling complex visual inputs, and
when the intermediate results don’t meet expectations, the model also finds it
hard to self-correct. MM-ReAct introduces the original ReAct for reasoning in
Multi-modal tasks, but due to the original ReAct’s inadequacy in dealing with
complex non-text intermediate results, its current planning scheme for
addressing video-related issues is basically two steps: extracting all
information from the video and then having an LLM answer the question. In
contrast, this paper proposes a more general Plan, Execute, Inspect, and Learn
(PEIL) reasoning scheme. In the case of complex videos, our interleaved
language and code reasoning approach allows for flexible language planning for
the next step, and structured code for invoking input and intermediate
results, thereby facilitating the handling of complex questions and visual
content.

Source Management. Handling complex input and a large number of intermediate
results is often crucial in complex reasoning processes. Current language-
based reasoning methods mainly use filenames to label resources. Chameleon
proposes an update mechanism with a cache that constantly updates the current
reasoning results. Program-based reasoning, on the other hand, uses variables
to store intermediate results. A deficiency of these methods is the inability
of the language-based Planner to quickly comprehend the content of visual
sources, which impedes the effective use of different sources to complete
different subtasks. As a result, existing work struggles to handle flexible
input and intermediate results. Even though some work supports multiple visual
sources as input, they are more often batch-processed for similar tasks, with
each source requiring similar operations. For instance, in HuggingGPT, the
task of calculating the sum of the number of zebras in several images involves
counting the number of zebras in each image. In contrast, our work introduces
the Inspector, which records the metadata and summary of each visual source
and provides it to the Planner for reasoning. This design can support complex
input. For example, a user view image that describes the current user’s
problem, and a reference video as a source of knowledge, AssistGPT can then
use these two different types of sources to jointly answer the user’s
question.

Learning. Most multi-modal modular systems lack the capability for continuous
optimization. This paper proposes a simple update mechanism that allows the
model to self-check the reasonableness of its output and ultimately continues
to collect in-context learning examples.

##  B More details of AssistGPT

Table 6: Invoke Commands and Illustrations to the Modules in AssistGPT.

Module | Invoke Command | Illustration
---|---|---
(a) Image Caption | caption(query, visual[i]) | extract the visual information in an image.
(b) Video Narration | video_narration(query, visual[i]) | output narration based on video’s visual information.
(c) Object Detection | object_detect(query, visual[i]) | detect required objects in an image.
(d) Text Detection | text_detect(None, visual[i]) | extract the OCR in an image.
(e) ASR Translation | asr(None, visual[i]) | transcribe audio to text.
(f) Region Ground | region_ground(query, visual[i]) | locate the queried region in an image.
(g) Narration Ground | narration_ground(query, visual[i]) | find the clip based on the narration of a video.
(h) Text Ground | text_ground(query, visual[i]) | find the location of a specific text in an image.
(i) Subtitle Ground | subtitle_ground(query, visual[i]) | find the clip based on the subtitle of a video.
(j) Knowledge Reason | knowledge_reason(query, []) | infer the answer based on the commonsense.
(k) Narration Reason | narration_reason(query, visual[i]) | infer the answer based on narration of a video.
(l) Subtitle Reason | subtitle_reason(query, visual[i]) | infer the answer based on subtitle of a video.
(m) Temporal Reason | temporal_reason(query, visual[i]) | find the clip based on temporal relationship words.

In Table 6, we show the invoke commands and illustration of each module in
AssistGPT. We provide more details of how each module is implemented.

  * •

Image Caption: The core model of this module is a text-conditioned captioning
model, e.g., BLIP2 [54], InstructBLIP [11], similar to an open-ended Visual
Question Answering model.

  * •

Video Narration: As the general video captioning models are not yet mature, we
currently use the image captioning model [54, 11] to accomplish this function.
Specifically, we sample image frames (1/3 FPS for current implementation) and
perform text-conditioned captioning on each frame. We employ text-conditioned
captioning because, if we use dense captioning, the output text will be
excessively abundant, making it difficult for subsequent models to utilize.
The Video Narration feature can also optionally read the OCR content within
the frames. The extracted OCR will be appended to the caption of each frame.

  * •

Object Detection: The main function of this module is to determine whether the
image contains the objects mentioned in the query and to address counting-
related questions. Thus, it contains an open-set object detection model, e.g.,
Grounding DINO [64], which can output the bounding boxes of relevant objects
based on the query. We also let the module calculate the number of related
objects.

  * •

Text Detection: This model is used to extract OCR from images, and the
extracted text is returned to the Planner. We use Google OCR to achieve this
purpose.

  * •

ASR Translation: This model is used to convert audio from a video into text.
We use OpenAI’s open-source ASR (Automatic Speech Recognition) model, Whisper
[65], to accomplish this. The detected ASR organizes timestamps and text in a
manner similar to subtitles. In the implementation, we automatically run this
module as soon as we receive a video with audio.

  * •

Region Ground: The purpose of this module is to find a specific area of an
image based on the query. We use the OFA-Large [66], which is fine-tuned on
RefCOCO, to achieve it.

  * •

Narration Ground: This model’s function is to find time segments related to
the query based on the video’s narration. We propose two implementations: 1)
We use GPT-4 [5], taking the video’s narration and query as prompts, to output
the timestamps of the time segments. 2) Another solution is using CLIP [67] to
do that. We can split the video into several segments, and calculate the
similarity between the frame in each segment and query. The time stamps of the
segment with the highest similarity will be outputted. In our preliminary
experiments, the first solution showed better interpretability and
generalization ability, so it was adopted in the benchmark evaluation.

  * •

Text Ground: The purpose of this model is to locate specific areas of an image
that correspond to a certain text. This capability can guide users in
identifying crucial information in complex, text-rich images, such as user
interfaces. The query format is text[:object_name], wherein text signifies the
text to be located, and object_name (which is optional) is used to locate the
text on a specific object, for instance, "menu: button". Specifically, the
model operates in two stages: 1) Based on the Optical Character Recognition
(OCR) detection results, the model identifies areas of the image that match
the text segment of the query. This is achieved by calculating the distance
between the query and the OCR extracted, and when the edit distance is below a
particular threshold, it is considered a match. 2) If more than one textual
area is identified, we further refine the results based on the object’s name.
We employ the Semantic Segment Anything (SSA) [63] to segment the image
semantically, identifying regions that match the object’s name mentioned in
the query.

  * •

Subtitle Ground: This model is similar to the narration grounding model, but
it uses the video’s subtitles as input instead of the narration. Thus, we also
use GPT-4 to achieve it.

  * •

Knowledge Reason: The purpose of this model is to enable the model to apply
external knowledge to answer questions. We currently do not connect to the
internet to retrieve knowledge, but use the knowledge that GPT-4 has itself
learned. Specifically, this model enables GPT-4 to use its own knowledge to
infer the answer based on the question and results of all previous reasoning
steps.

  * •

Narration Reason: The aim of this module is to infer some information based on
the visual content of the video. This module also uses GPT-4, taking the query
and the input video’s narration as prompts, to infer the answer.

  * •

Subtitle Reason: The aim of this module is to infer some information based on
the subtitle of the video. It is similar to Narration Reason, but takes the
input video’s subtitle and query as prompts, to infer the answer.

  * •

Temporal Reason: This model is able to find a video clip based on some
temporal relation words. The input to this module follows the following
format: temporal_word: time stamps, e.g., after: 3 - 6. Temporal relation
words include two types, one is absolute temporal relation words, such as in
the middle/beginning/end of the video. The second type is relative temporal
relation words, such as before and after. For the first type of words, we
divide the video into 5 segments and then output the time stamps of the
corresponding segment according to the temporal_word. For the second type, we
divide the video into 8 segments, and then, according to the input time
stamps, we output the time stamps of the segment before or after it. The
current hyperparameters, the division of video clips, are still preliminary.
It would be much better to use the model to divide them semantically, and then
perform temporal reasoning in the future.

##  C Qualitative Results in A-OKVQA

In Figure 5, we showcase a successful instance along with several failure
examples, illustrating the most frequent error patterns in A-OKVQA. As is
evident, AssistGPT can produce highly interpretable answer processes.
Moreover, even in cases where the questions are answered incorrectly, there
are relatively reasonable explanations provided. In the following, we
illustrate the common error patterns in detail:

  * •

Undesired output format: For Direct Answer questions, like Q2, the results of
AssistGPT are the same as the correct answers in meaning, but the expression
is different, which would be considered as incorrect under the existing
metrics.

  * •

Fine-grained recognition: The recognition of fine-grained categories of some
objects is still not well done by existing visual models, resulting in the
incorrect final answer. For example, AssistGPT didn’t successfully recognize
cough drops in Q3.

  * •

Pose-to-text: Currently, there are very few models that can map the fine-
grained pose or actions of people or animals to natural language. For example,
capturing the upward jump action of the cat in Q4 is a challenge. AssistGPT
currently does not incorporate a related model to grasp such information.
Instead, it makes prediction based on the surrounding objects in relation to
the cat.

  * •

Inconsistent reasoning: Despite AssistGPT having some self-error correction
mechanisms, it occasionally exhibits inconsistencies in its reasoning process,
which can lead to final inaccuracies. For instance, in Q5, the model initially
identifies the orange vehicle as a truck, but in subsequent steps, it is
referred to as a shuttle bus. Unfortunately, AssistGPT fails to detect this
inconsistency and does not proceed to make necessary corrections.

Figure 5: Reasoning process of AssistGPT on A-OKVQA. The choice colored with
green in the question indicates the ground truth.

##  D In-the-wild Prediction Examples

We show some examples of AssistGPT handling in-the-wild scenarios in Figure 6
and Figure 7. From various in-the-wild examples, it’s clear that AssistGPT can
adeptly handle a range of video types, be it dense, subtitled instructional
videos (Q2, Q3), or those featuring rich, visual content with sporadic on-
frame text (Q1, Q4, Q5). Impressively, when faced with high-level queries (Q2
and Q3), the model exhibits a capacity to strategically locate useful content,
accurately identify the correct responses, and offer comprehensive, multimodal
answers. A notable self-error correction capability is also evident during its
reasoning process, as demonstrated in Q2. Here, the narration model was unable
to generate meaningful narrations and, therefore, opted to utilize the
subtitle to answer the question.

Moreover, in Q5, we highlight that our model can effectively process multiple
video inputs serving different functions. This includes a User view image and
a couple of reference videos. It’s important to note that our model can
accommodate any number of inputs. Consequently, with the incorporation of a
YouTube video search function, the model could autonomously seek out several
reference videos and then cross-reference them to discern the user’s intent.

In summary, we want to emphasize that AssistGPT is a comprehensive multi-modal
assistant system, capable of managing a wide array of real-world application
queries that are far more complex and comprehensive than the samples provided
in benchmarks.

Figure 6: The reasoning process of AssistGPT when handling in-the-wild
questions. Figure 7: The reasoning process of AssistGPT when handling in-the-
wild questions.

## References

  * [1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [2] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, page 9, 2019.
  * [3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  NeurIPS, pages 1877–1901, 2020.
  * [4] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [5] OpenAI.  Introducing chatgpt.  OpenAI Blog, 09 2021.
  * [6] Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al.  Socratic models: Composing zero-shot multimodal reasoning with language.  arXiv preprint arXiv:2204.00598, 2022.
  * [7] Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, et al.  Language models with image descriptors are strong few-shot video-language learners.  arXiv preprint arXiv:2205.10747, 2022.
  * [8] Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan.  Visual chatgpt: Talking, drawing and editing with visual foundation models.  arXiv preprint arXiv:2303.04671, 2023.
  * [9] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In ICML, pages 12888–12900, 2022.
  * [10] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv preprint arXiv:2301.12597, 2023.
  * [11] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  arXiv preprint arXiv:2305.06500, 2023.
  * [12] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In ECCV, pages 213–229, 2020.
  * [13] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao.  Grounded language-image pre-training.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10965–10975, June 2022.
  * [14] Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang.  Grit: A generative region-to-text transformer for object understanding.  arXiv preprint arXiv:2212.00280, 2022.
  * [15] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang.  Git: A generative image-to-text transformer for vision and language.  arXiv preprint arXiv:2205.14100, 2022.
  * [16] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut.  Pali: A jointly-scaled multilingual language-image model, 2022.
  * [17] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang.  Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface.  arXiv preprint arXiv:2303.17580, 2023.
  * [18] Yingqiang Ge, Wenyue Hua, Jianchao Ji, Juntao Tan, Shuyuan Xu, and Yongfeng Zhang.  Openagi: When llm meets domain experts.  arXiv preprint arXiv:2304.04370, 2023.
  * [19] Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao.  Chameleon: Plug-and-play compositional reasoning with large language models.  arXiv preprint arXiv:2304.09842, 2023.
  * [20] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao.  Videochat: Chat-centric video understanding.  arXiv preprint arXiv:2305.06355, 2023.
  * [21] Dídac Surís, Sachit Menon, and Carl Vondrick.  Vipergpt: Visual inference via python execution for reasoning.  arXiv preprint arXiv:2303.08128, 2023.
  * [22] Tanmay Gupta and Aniruddha Kembhavi.  Visual programming: Compositional visual reasoning without training.  arXiv preprint arXiv:2211.11559, 2022.
  * [23] Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, Yun Wang, Linjun Shou, Ming Gong, and Nan Duan.  Taskmatrix.ai: Completing tasks by connecting foundation models with millions of apis.  arXiv preprint arXiv:2303.16434, 2023.
  * [24] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.  React: Synergizing reasoning and acting in language models.  arXiv preprint arXiv:2210.03629, 2022.
  * [25] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.  Toolformer: Language models can teach themselves to use tools.  arXiv preprint arXiv:2302.04761, 2023.
  * [26] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh.  Vqa: Visual question answering.  In Proceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015.
  * [27] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang.  Bottom-up and top-down attention for image captioning and visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077–6086, 2018.
  * [28] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  Advances in neural information processing systems, 32, 2019.
  * [29] Hao Tan and Mohit Bansal.  Lxmert: Learning cross-modality encoder representations from transformers.  arXiv preprint arXiv:1908.07490, 2019.
  * [30] Difei Gao, Ruiping Wang, Shiguang Shan, and Xilin Chen.  Learning to recognize visual concepts for visual question answering with structural label space.  IEEE Journal of Selected Topics in Signal Processing, 14(3):494–505, 2020.
  * [31] Difei Gao, Ruiping Wang, Shiguang Shan, and Xilin Chen.  Visual textbook network: Watch carefully before answering visual questions.  In BMVC, 2017.
  * [32] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.  Ok-vqa: A visual question answering benchmark requiring external knowledge.  In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019.
  * [33] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi.  A-okvqa: A benchmark for visual question answering using world knowledge.  arXiv, 2022.
  * [34] Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel.  Fvqa: Fact-based visual question answering.  IEEE transactions on pattern analysis and machine intelligence, 40(10):2413–2427, 2017.
  * [35] Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, and Jianfeng Gao.  Kat: A knowledge augmented transformer for vision-and-language.  arXiv preprint arXiv:2112.08614, 2021.
  * [36] Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach.  Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge-based vqa.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14111–14121, 2021.
  * [37] Difei Gao, Ruiping Wang, Shiguang Shan, and Xilin Chen.  Cric: A vqa dataset for compositional reasoning on vision and commonsense.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [38] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728–1738, 2021.
  * [39] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingjing Liu.  Less is more: Clipbert for video-and-language learning via sparse sampling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7331–7341, June 2021.
  * [40] Yuxuan Wang, Difei Gao, Licheng Yu, Weixian Lei, Matt Feiszli, and Mike Zheng Shou.  Geb+: A benchmark for generic event boundary captioning, grounding and retrieval.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXV, pages 709–725. Springer, 2022.
  * [41] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al.  Ego4d: Around the world in 3,000 hours of egocentric video.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995–19012, 2022.
  * [42] Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z XU, Difei Gao, Rong-Cheng Tu, Wenzhe Zhao, Weijie Kong, et al.  Egocentric video-language pretraining.  Advances in Neural Information Processing Systems, 35:7575–7586, 2022.
  * [43] Difei Gao, Ruiping Wang, Ziyi Bai, and Xilin Chen.  Env-qa: A video question answering benchmark for comprehensive understanding of dynamic environments.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1675–1685, 2021.
  * [44] Zhijian Hou, Wanjun Zhong, Lei Ji, Difei Gao, Kun Yan, Wing-Kwong Chan, Chong-Wah Ngo, Zheng Shou, and Nan Duan.  Cone: An efficient coarse-to-fine alignment framework for long video temporal grounding.  arXiv preprint arXiv:2209.10918, 2022.
  * [45] Benita Wong, Joya Chen, You Wu, Stan Weixian Lei, Dongxing Mao, Difei Gao, and Mike Zheng Shou.  Assistq: Affordance-centric question-driven task completion for egocentric assistant.  In European Conference on Computer Vision, pages 485–501. Springer, 2022.
  * [46] Weixian Lei, Difei Gao, Yuxuan Wang, Dongxing Mao, Zihan Liang, Lingmin Ran, and Mike Zheng Shou.  Assistsr: Task-oriented video segment retrieval for personal ai assistant.  In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 319–338, 2022.
  * [47] Joya Chen, Difei Gao, Kevin Qinghong Lin, and Mike Zheng Shou.  Affordance grounding from demonstration video to target image.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6799–6808, 2023.
  * [48] Ronghang Hu, Amanpreet Singh, Trevor Darrell, and Marcus Rohrbach.  Iterative answer prediction with pointer-augmented multimodal transformers for textvqa.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9992–10002, 2020.
  * [49] Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Florencio, Lijuan Wang, Cha Zhang, Lei Zhang, and Jiebo Luo.  Tap: Text-aware pre-training for text-vqa and text-caption.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8751–8761, 2021.
  * [50] Difei Gao, Ke Li, Ruiping Wang, Shiguang Shan, and Xilin Chen.  Multi-modal graph neural network for joint reasoning on vision and scene text.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [51] Stan Weixian Lei, Difei Gao, Jay Zhangjie Wu, Yuxuan Wang, Wei Liu, Mengmi Zhang, and Mike Zheng Shou.  Symbolic replay: Scene graph as prompt for continual learning on vqa task.  arXiv preprint arXiv:2208.12037, 2022.
  * [52] OpenAI.  Gpt-4 technical report, 2023.
  * [53] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al.  Palm-e: An embodied multimodal language model.  arXiv preprint arXiv:2303.03378, 2023.
  * [54] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv preprint arXiv:2301.12597, 2023.
  * [55] Deyao Zhu, Jun Chen, Xiaoqian Shen, xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023.
  * [56] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.  Evaluating large language models trained on code.  arXiv preprint arXiv:2107.03374, 2021.
  * [57] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein.  Neural module networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 39–48, 2016.
  * [58] Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko.  Learning to reason: End-to-end module networks for visual question answering.  In Proceedings of the IEEE international conference on computer vision, pages 804–813, 2017.
  * [59] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick.  Inferring and executing programs for visual reasoning.  In Proceedings of the IEEE international conference on computer vision, pages 2989–2998, 2017.
  * [60] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou.  Chain of thought prompting elicits reasoning in large language models.  arXiv preprint arXiv:2201.11903, 2022.
  * [61] Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang.  Mm-react: Prompting chatgpt for multimodal reasoning and action.  arXiv preprint arXiv:2303.11381, 2023.
  * [62] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  arXiv preprint arXiv:2304.02643, 2023.
  * [63] Jiaqi Chen, Zeyu Yang, and Li Zhang.  Semantic segment anything.  https://github.com/fudan-zvg/Semantic-Segment-Anything, 2023.
  * [64] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  arXiv preprint arXiv:2303.05499, 2023.
  * [65] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever.  Robust speech recognition via large-scale weak supervision.  arXiv preprint arXiv:2212.04356, 2022.
  * [66] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang.  Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.  CoRR, abs/2202.03052, 2022.
  * [67] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [68] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua.  Next-qa: Next phase of question-answering to explaining temporal actions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9777–9786, 2021.
  * [69] Amita Kamath, Christopher Clark, Tanmay Gupta, Eric Kolve, Derek Hoiem, and Aniruddha Kembhavi.  Webly supervised concept expansion for general purpose vision models.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXVI, pages 662–681. Springer, 2022.
  * [70] Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo.  Promptcap: Prompt-guided task-aware image captioning.  arXiv preprint arXiv:2211.09699, 2022.
  * [71] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid.  Just ask: Learning to answer questions from millions of narrated videos.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1686–1697, 2021.
  * [72] Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles.  Revisiting the" video" in video-language understanding.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2917–2927, 2022.
  * [73] Junbin Xiao, Pan Zhou, Tat-Seng Chua, and Shuicheng Yan.  Video graph transformer for video question answering.  In European Conference on Computer Vision, pages 39–58. Springer, 2022.
  * [74] Difei Gao, Luowei Zhou, Lei Ji, Linchao Zhu, Yi Yang, and Mike Zheng Shou.  Mist: Multi-modal iterative spatial-temporal transformer for long-form video question answering.  arXiv preprint arXiv:2212.09522, 2022.
  * [75] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al.  Webgpt: Browser-assisted question-answering with human feedback.  arXiv preprint arXiv:2112.09332, 2021.


On the Ternary Purely Exponential Diophantine Equation
$(ak)^{x}+(bk)^{y}=((a+b)k)^{z}$ with Prime Powers $a$ and $b$

Maohua Le

Institute of Mathematics
Lingnan Normal College
Zhanjiang, Guangdong 524048
China
<EMAIL_ADDRESS>

Gökhan Soydan
Department of Mathematics
Bursa Uludağ University
16059 Bursa
Türkiye
<EMAIL_ADDRESS>

###### Abstract

Let $k$ be a positive integer, and let $a,b$ be coprime positive integers with
$\min\\{a,b\\}>1$. In this paper, using a combination of some elementary
number theory techniques with classical results on the Nagell-Ljunggren
equation, the Catalan equation and some new properties of the Lucas sequence
(A000204 in OEIS), we prove that if $k>1$ and $a,b$ are both prime powers with
$\min\\{a,b\\}>2$, then the equation $(ak)^{x}+(bk)^{y}=((a+b)k)^{z}$ has only
one positive integer solution $(x,y,z)=(1,1,1)$. The above result partially
proves that Conjecture 1 presented in (Acta Arith. 2018, 184 (1): 37-49) is
true.

##  1 Introduction

Let $\mathbb{Z}$, $\mathbb{N}$, $\mathbb{P}$ be the sets of all integers,
positive integers and odd primes, respectively. Let $A,B,C$ be fixed positive
integers with $\min\\{A,B,C\\}>1$. In recent decades, the ternary purely
exponential Diophantine equation

$A^{x}+B^{y}=C^{z},\quad x,y,z\in\mathbb{N}$ (1)

has yielded very rich results, but some important problems about it are far
from being solved (see [7]). In recent 20 years, many authors have considered
equation (1) when $A,B$ and $C$ are Fibonacci A00045 or Lucas A000204 or Pell
A000129 numbers in OEIS [14] or when $A$ and $B$ are Fibonacci or Lucas or
Pell numbers (see [1, 2, 4, 6, 12, 13, 18]).

Let $k$ be a positive integer, and let $a,b$ be coprime positive integers with
$\min\\{a,b\\}>1$. In this paper, we discuss (1) for $(A,B,C)=(ak,bk,(a+b)k)$.
Then, (1) can be written as

$(ak)^{x}+(bk)^{y}=((a+b)k)^{z},\quad x,y,z\in\mathbb{N}.$ (2)

Obviously, for any $a,b$ and $k$, (2) has the solution $(x,y,z)=(1,1,1)$. A
solution $(x,y,z)$ of (2) with $(x,y,z)\neq(1,1,1)$ is called exceptional. In
2018, Sun and Tang [16] proved that if $k>1$, $\min\\{a,b\\}>2$ and $(x,y,z)$
is an exceptional solution of (2), then either $x>z>y$ or $y>z>x$. On this
basis, they further proved that if $k>1$ and
$(a,b)\in\\{(3,5),(5,8),(8,13),(13,21)\\}$, then (2) has no exceptional
solutions. In the same year, Yuan and Han [17] proposed the following
conjecture.

######  Conjecture 1.

For any $k$, if $\min\\{a,b\\}>3$, then (2) has no exceptional solutions.

The above conjecture is formally similar to Jeśmanowicz’ conjecture concerning
Pythagorean triples (see [5, 15]). So far, it has only been solved for some
very special cases. For example, the authors of [17] proved that if $a$ and
$b$ are squares with $b\equiv 4\pmod{8}$, then (2) has no solutions $(x,y,z)$
with $y>z>x$, in particular, if $a$ is a square and $b=4$, then Conjecture 1
is true. Afterwards, using Baker’s method, Le and Soydan [8] proved that if
$a$ and $b$ are squares with $a>64b^{3}$, then (2) has no solutions $(x,y,z)$
with $x>z>y$. It implies that if $a$ and $b$ are squares with $a>64b^{3}$ and
$b\equiv 4\pmod{8}$, then Conjecture 1 is true.

In this paper, using a combination of some elementary number theory techniques
with classical results on the Nagell-Ljunggren equation, the Catalan equation
and some new properties of the Lucas sequence A000204, we prove the following
result.

######  Theorem 2.

Let $r,s$ be positive integers, and let $p,q$ be distinct odd primes. If $k>1$
and $a,b$ satisfy one of the following conditions:

  1. (i)

$(a,b)=(2^{r},p^{s})$ with $r>1$; or

  2. (ii)

$(a,b)=(p^{r},2^{s})$ with $s>1$; or

  3. (iii)

$(a,b)=(p^{r},q^{s})$, then (2) has no solutions $(x,y,z)$ with $x>z>y$.

Since $a$ and $b$ are symmetric in (2), by the first result mentioned in [16],
we can obtain the following corollary from Theorem 2 immediately.

######  Corollary 3.

If $k>1$ and $a,b$ are prime powers, then Conjecture 1 is true.

Finally, we briefly analyze the effect of the above mentioned results. For any
enough large positive integer $N$, let $F(N)$ denote the number of pairs
$(a,b)$ such that $\max\\{a,b\\}\leq N$ and they have been proved to hold for
Conjecture 1 with $k>1$. Clearly, by the mentioned results in [8, 16, 17] and
Corollary 3, we have $F(N)=4$, $F(N)=\sqrt{N}$, $N>F(N)>\sqrt{N}$ and
$F(N)>N^{2}/(\log N)^{2}$, respectively.

##  2 Preliminaries

Let us now recall that if $\alpha$ and $\beta$ are roots of a quadratic
equation of the form $x^{2}-rx-s=0$ with nonzero coprime integers $r$ and $s$
and such that $\alpha/\beta$ is not a root of unity, then the sequence
$(u_{\ell})_{\ell\geq 0}$ of general term

$u_{\ell}=\dfrac{\alpha^{\ell}-{\beta}^{\ell}}{\alpha-{\beta}}\qquad{\text{for~{}all}}~{}\ell\geq
0$

is called a Lucas sequence A000204. It can also be defined inductively as
$u_{0}=0,~{}u_{1}=1$ and $u_{\ell+2}=r\cdot u_{\ell+1}+s\cdot u_{\ell}$. When
the case $\beta=1$, the following Lemmas 9, 10 and 11 are the three new
properties we proved about the Lucas sequence at the beginning of page 4.

######  Lemma 4 ([9]).

The equation

$\dfrac{X^{m}-1}{X-1}=Y^{n},\,\,\,X,Y,m,n\in\mathbb{N},\,\,\,X>1,\,\,Y>1,\,\,m>2,\,\,n>1$

has only the solution $(X,Y,m,n)=(3,11,5,2)$ with $2\mid n.$

######  Lemma 5 ([10]).

The equation

$X^{m}-Y^{n}=1,\,\,\,X,Y,m,n\in\mathbb{N},\,\,\,\min\\{X,Y,m,n\\}>1$

has only the solution $(X,Y,m,n)=(3,2,2,3)$.

Let $X,\ell,m,n$ be positive integers with $X>1$ and $\ell>1$, and let $p,q$
be odd primes. Three lemmas for divisibility are given directly below.

######  Lemma 6 ([11]).

If $X^{n}+1\equiv 0\pmod{X^{m}+1}$, then $m\mid n$ and $n/m$ is odd.

######  Lemma 7 ([3]).

Let $\ell$ be an odd prime. If $X\not\equiv 1\pmod{\ell}$, then every prime
divisor $p$ of $(X^{\ell}-1)/(X-1)$ satisfies

$p\equiv 1\pmod{2\ell}.$ (3)

If $X\equiv 1\pmod{\ell}$, then $\ell\mid\mid(X^{\ell}-1)/(X-1)$ and every
prime divisor $p$ of $(X^{\ell}-1)/\ell(X-1)$ satisfies (3).

######  Lemma 8 ([3]).

When $X\equiv 1\pmod{p}$, $p^{m}\mid\mid(X^{\ell}-1)/(X-1)$ if and only if
$p^{m}\mid\mid\ell$.

######  Lemma 9.

If

$\dfrac{X^{\ell}-1}{X-1}=p^{n},\,\,2\nmid\ell,$ (4)

then $\ell$ is an odd prime with (3).

###### Proof.

We now assume that $\ell$ is not an odd prime. Since $\ell>1$ and
$2\nmid\ell$, $\ell$ has an odd prime divisor $q$ with $q<\ell$. Then we have
$q\mid\ell$ and $\ell/q>1$. By (4), we get

$p^{n}=\dfrac{X^{\ell}-1}{X-1}=\left(\dfrac{X^{\ell/q}-1}{X-1}\right)\left(\dfrac{(X^{\ell/q})^{q}-1}{X^{\ell/q}-1}\right),$
(5)

where $(X^{\ell/q}-1)/(X-1)$ and $((X^{\ell/q})^{q}-1)/(X^{\ell/q}-1)$ are
positive integers greater than 1. Since $X>1$, by (5), we have

$\dfrac{X^{\ell/q}-1}{X-1}=p^{f},\,\,\dfrac{(X^{\ell/q})^{q}-1}{X^{\ell/q}-1}=p^{n-f},\,\,f\in\mathbb{N},\,\,f<n.$
(6)

Further, by the first and the second equalities of (6), we get
$X^{\ell/q}\equiv 1\pmod{p}$ and

$0\equiv
p^{n-f}\equiv\dfrac{(X^{\ell/q})^{q}-1}{X^{\ell/q}-1}\equiv(X^{\ell/q})^{q-1}+\cdots+X^{\ell/q}+1\equiv
q\pmod{p}.$ (7)

Since $p$ and $q$ are odd primes, by (7), we obtain $q=p$. Hence by Lemma 7,
we see from (6) that
$p\mid\mid((X^{\ell/p})^{p}-1)/(X^{\ell/p}-1)=((X^{\ell/q})^{q}-1)/(X^{\ell/q}-1)$
and

$p=\dfrac{(X^{\ell/p})^{p}-1}{X^{\ell/p}-1}=(X^{\ell/p})^{p-1}+\cdots
X^{\ell/p}+1>p,$ (8)

a contradiction. It implies that $\ell$ must be an odd prime. Moreover, using
Lemma 7 again, if $X\equiv 1\pmod{\ell}$, then from (4) we can get $\ell=p$
and $n=1$, which is the same contradiction as (8). Therefore, we have
$x\not\equiv 1\pmod{\ell}$ and $p$ satisfies (3). The lemma is proved. ∎

######  Lemma 10.

If

$X^{\ell}-1=2^{m}p^{n},$ (9)

then one of the following three conclusions holds.

  1. (i)

$(p,X,\ell,m,n)=(3,5,2,3,1),(3,7,2,4,1),(5,9,2,4,1),(5,3,4,4,1),(3,17,2,5,2)$
or
$(7,15,2,5,1)$.

  2. (ii)

$\ell=2,\,\,m\geq
6,\,\,n=1,\,\,X=2^{m-1}+\zeta,\,\,p=2^{m-2}-\zeta,\,\,\zeta\in\\{1,-1\\}.$
(10)

  3. (iii)

$\ell\,\,\text{is an odd
prime},\,\,\,X-1=2^{m},\,\,\dfrac{X^{\ell}-1}{X-1}=p^{n},\,\,p\equiv
1\pmod{2\ell}.$ (11)

###### Proof.

Obviously, by (9), we have $X>1$, $2\nmid X$ and $p\nmid X$. When $2\mid\ell$,
since $X^{\ell}\equiv 1\pmod{8}$, $\gcd(X^{\ell/2}+1,X^{\ell/2}-1)=2$ and $p$
is an odd prime, by (9), we have

$X^{\ell/2}+\zeta=2p^{n},\,\,X^{\ell/2}-\zeta=2^{m-1},\,\,m\geq
3,\,\,\zeta=((-1)^{(X-1)/2})^{\ell/2},$ (12)

where

$\zeta\in\\{1,-1\\}.$ (13)

If $3\leq m\leq 5$, then from (12) and (13) we can easily obtain the
conclusion (i). If $m\geq 6$, by Lemma 5, then from the second equality of
(12) we get $\ell=2$. In addition, eliminating $X^{\ell/2}$ from (12), we have

$p^{n}-2^{m-2}=\zeta.$ (14)

Since $m-2\geq 4$, using Lemma 5 again, we see from (14) that $n=1$.
Therefore, by (12), (13) and (14), we obtain (10) and conclusion (ii) is
proved.

When $2\nmid\ell$, since $\ell>1$ and $\dfrac{X^{\ell}-1}{X-1}$ is an odd
positive integer greater than 1, by (9), we have

$X-1=2^{m}p^{f},\,\,\dfrac{X^{\ell}-1}{X-1}=p^{n-f},\,\,f\in\mathbb{Z},\,\,0\leq
f<n.$ (15)

If $f>0$, then from the first equality of (15) we get $X\equiv 1\pmod{p}$.
Hence, applying Lemma 8 to the second equality of (15), we obtain
$p^{n-f}\mid\mid\ell.$ However, since $\ell\geq p^{n-f},$ we get
$p^{n-f}=(X^{\ell}-1)/(X-1)>\ell\geq p^{n-f},$ a contradiction. So, we have
$f=0$. Therefore, by (15), we get

$X-1=2^{m},\,\,\dfrac{X^{\ell}-1}{X-1}=p^{n}.$ (16)

Further, by Lemma 9, we see from the second equality of (16) that $\ell$ is an
odd prime with (3). Thus, we obtain (11) and the conclusion (iii) is proved.
The proof is completed. ∎

Using the same method as in the proof of Lemma 10, we can obtain the following
lemma without difficulty

######  Lemma 11.

If

$X^{\ell}-1=p^{m}q^{n},$ (17)

the one of the following six conclusions holds.

  1. (i)

$2\mid\ell,\,\,X^{\ell/2}+\zeta=p^{m},\,\,X^{\ell/2}-\zeta=q^{n},\,\,\zeta\in\\{1,-1\\}.$
(18)

  2. (ii)

$2\nmid\ell,\,\,X=2,\,\,2^{\ell}-1=p^{m}q^{n}.$ (19)

  3. (iii)

$\ell=p,\,\,m>1,\,\,X-1=p^{m-1},\,\,\dfrac{X^{p}-1}{X-1}=pq^{n},\,\,q\equiv
1\pmod{2p}.$ (20)

  4. (iv)

$\ell=q,\,\,n>1,\,\,X-1=q^{n-1},\,\,\dfrac{X^{q}-1}{X-1}=p^{m}q,\,\,p\equiv
1\pmod{2q}.$ (21)

  5. (v)

$\ell\,\,\text{is an odd
prime},\,\,X-1=p^{m},\,\,\dfrac{X^{\ell}-1}{X-1}=q^{n},\,\,q\equiv
1\pmod{2\ell}.$ (22)

  6. (vi)

$\ell\,\,\text{is an odd
prime},\,\,X-1=q^{n},\,\,\dfrac{X^{\ell}-1}{X-1}=p^{m},\,\,p\equiv
1\pmod{2\ell}.$ (23)

###### Proof.

Since $p$ and $q$ are distinct odd primes, by (17), we have $2\mid X$. When
$2\mid\ell$, since $\gcd(X^{\ell/2}+1,X^{\ell/2}-1)=1$, by (17), we can
directly obtain (18) and the conclusion (i) is proved. When $2\nmid\ell$ and
$X=2$, we see from (17) that (19) is clearly true and the conclusion (ii) is
proved. When $2\nmid\ell$ and $X>2$, by (17), we have

$\displaystyle
X-1=p^{f}q^{g},\,\,\dfrac{X^{\ell}-1}{X-1}=p^{m-f}q^{n-g},\,\,f,g\in\mathbb{Z},\,\,0\leq
f\leq m,$ (24) $\displaystyle 0\leq g\leq
n,\,\,(f,g)\neq(0,0)\,\,\text{or}\,\,(m,n).$

If $0<f<m$ and $0<g<n$, then from (24) we get $X\equiv 1\pmod{pq}$ and
$0\equiv p^{m-f}q^{n-g}\equiv(X^{\ell}-1)(X-1)\equiv
X^{\ell-1}+\cdots+X+1\equiv\ell\pmod{pq}$. It follows that $pq\mid\ell$ and
$(X^{pq}-1)/(X-1)\mid(X^{\ell}-1)(X-1).$ Hence, by the second equality of
(24), we have

$\displaystyle\dfrac{X^{pq}-1}{X-1}=\left(\dfrac{X^{p}-1}{X-1}\right)\left(\dfrac{(X^{p})^{q}-1}{X^{p}-1}\right)=p^{f^{\prime}}q^{g^{\prime}},\,\,f^{\prime},g^{\prime}\in\mathbb{Z},$
(25) $\displaystyle f^{\prime}\geq 0,\,\,g^{\prime}\geq
0,\,\,(f^{\prime},g^{\prime})\neq(0,0),$

where $(X^{p}-1)/(X-1)$ and $((X^{p})^{q}-1)/(X^{p}-1)$ are positive integers
greater than 1. Further, since $X\equiv 1\pmod{p}$, by Lemma 8, we get
$p\mid\mid(X^{p}-1)/(X-1)$. Furthermore, since $(X^{p}-1)/(X-1)>p,$ by (25),
we have

$\dfrac{X^{p}-1}{X-1}=pq^{g^{\prime\prime}},\,\,g^{\prime\prime}\in\mathbb{N}.$
(26)

Recall that $X\equiv 1\pmod{q}$. By (26), we get $0\equiv
pq^{g^{\prime\prime}}\equiv(X^{p}-1)/(X-1)\equiv X^{p-1}+\cdots+X+1\equiv
p\pmod{q}$ and $p=q$, a contradiction. Therefore, the case $0<f<m$ and $0<g<n$
is impossible.

If $0<f<m$ and $g=n$, then from (24) we get

$X-1=p^{f}q^{n},\,\,\dfrac{X^{\ell}-1}{X-1}=p^{m-f}.$ (27)

By the first equality of (27), we have $X\equiv 1\pmod{p}$. In addition, by
Lemma 8, we see from the second equality of (27) that $\ell$ is an odd prime
with $p\equiv 1\pmod{2\ell}$. But, since $X\equiv 1\pmod{p}$, we get $0\equiv
p^{m-f}\equiv(X^{\ell}-1)/(X-1)\equiv X^{\ell-1}+\cdots+X+1\equiv\ell\pmod{p}$
and $p=\ell$, a contradiction. Therefore, the case $0<f<m$ and $g=n$ is
impossible. Using the same method, we can eliminate the case $f=m$ and
$0<g<n$.

Recall that $(f,g)\neq(0,0)$ or $(m,n)$. According to the above analysis, we
are left with only the cases

$f=0,\,\,0<g\leq n$ (28)

and

$0<f\leq m,\,\,g=0$ (29)

that have not yet been discussed.

If (28) holds, by (24), then we have

$X-1=q^{g},\,\,\dfrac{X^{\ell}-1}{X-1}=p^{m}q^{n-g},\,\,g\in\mathbb{N},\,\,g\leq
n.$ (30)

When $g=n$, by (30), we get

$X-1=q^{n},\,\,\dfrac{X^{\ell}-1}{X-1}=p^{m}.$ (31)

Applying Lemma 9 to the second equality of (31), $\ell$ is an odd prime with
$p\equiv 1\pmod{2\ell}$. Hence, by (31), we get (23) and the conclusion (vi)
is proved.

When $g<n$, by the first equality of (30), we have $X\equiv 1\pmod{q}$. Hence,
by Lemma 8, we see from the second equality of (30) that
$q^{n-g}\mid\mid\ell$. It follows that

$\ell=q^{n-g}\ell_{1},\,\,\ell_{1}\in\mathbb{N},\,\,q\nmid\ell_{1}.$ (32)

Then we have

$\dfrac{X^{\ell}-1}{X-1}=\left(\dfrac{X^{\ell_{1}}-1}{X-1}\right)\prod_{j=1}^{n-g}\left(\dfrac{X^{q^{j}\ell_{1}}-1}{X^{q^{j-1}\ell_{1}}-1}\right),$
(33)

where $(X^{\ell_{1}}-1)/(X-1)$ and
$(X^{q^{j}\ell_{1}}-1)/(X^{q^{j-1}\ell_{1}}-1)$ $(j=1,\cdots,n-g)$ are
positive integers with

$q\nmid\dfrac{X^{\ell_{1}}-1}{X-1},\,\,q\mid\mid\dfrac{X^{q^{j}\ell_{1}}-1}{X^{q^{j-1}\ell_{1}}-1},\,\,j=1,\cdots,n-g.$
(34)

By (33) and (34), we get from the second equality of (30) that

$\dfrac{X^{\ell_{1}}-1}{X-1}=p^{m_{0}},\,\,m_{0}\in\mathbb{Z},\,\,m_{0}\geq 0$
(35)

and

$\dfrac{X^{q^{j}\ell_{1}}-1}{X^{q^{j-1}\ell_{1}}-1}=p^{m_{j}}q,\,\,m_{j}\in\mathbb{N},\,\,j=1,\cdots,n-g,$
(36)

where

$m_{0}+m_{1}+\cdots+m_{n-g}=m.$ (37)

If $n-g>1$, then from (36) we have $X^{q\ell_{1}}\equiv 1\pmod{p}$ and
$0\equiv p^{m_{2}}q\equiv(X^{q^{2}\ell_{1}}-1)/(X^{q\ell_{1}}-1)\equiv
X^{q\ell_{1}(q-1)}+\cdots+X^{q\ell_{1}}+1\equiv q\pmod{p}$, whence we get
$p=q$, a contradiction. So we have $n-g=1$. Further, if $n-g=1$ and
$\ell_{1}>1$, by (35) and (36), then $m_{0}$ is a positive integer,
$X^{\ell_{1}}\equiv 1\pmod{p}$, $0\equiv
p^{m_{1}}q\equiv(X^{q\ell_{1}}-1)/(X^{\ell_{1}}-1)\equiv
X^{\ell_{1}(q-1)}+\cdots+X^{\ell_{1}}+1\equiv q\pmod{p}$ and $p=q$, a
contradiction. Therefore, if (28) holds, then we have $n-g=1$ and
$\ell_{1}=1.$ By (30),(32),(33), (36) and (37), we get (21) and the conclusion
(iv) is proved.

Using the same method, as in the proof about the case (28), we can deduce that
if (29) holds, then there can only obtain (20) or (22), and the conclusions
(iii) and (v) are proved. To sum up, the proof is complete. ∎

For any positive integer $m$, let $\operatorname{rad}(m)$ denote the product
of all distinct prime divisors of $m$, and let $\operatorname{rad}(1)=1.$
Obviously, $\operatorname{rad}(m)$ is equal to the largest squarefree divisor
of $m$.

######  Lemma 12 ([16]).

If $k>1$ and $(x,y,z)$ is a solution of (2) with $x>z>y$, then we have

$\operatorname{rad}(k)\mid
b,\,\,b=b_{1}b_{2},\,\,b_{1},b_{2}\in\mathbb{N},\,\,b_{1}>1,\,\,\gcd(b_{1},b_{2})=1,$
$b_{1}^{y}=k^{z-y}$

and

$a^{x}k^{x-z}+b_{2}^{y}=(a+b)^{z}.$

By Lemma 12, we can obtain the following lemma immediately.

######  Lemma 13.

If $k>1$, $b$ is a prime power and $(x,y,z)$ is a solution of (2) with
$x>z>y$, then we have

$b^{y}=k^{z-y}$

and

$a^{x}k^{x-z}+1=(a+b)^{z}.$

##  3 Proof of Theorem 2

We now assume that $k>1$ and $(x,y,z)$ is a solution of (2) with $x>z>y$. We
will prove that this solution does not exist in the following three cases.

###  3.1 Case $(i)$ : $(a,b)=(2^{r},p^{s})$ with $r>1$.

Since $k>1$ and $p$ is an odd prime, Lemma 13, we have

$p^{sy}=k^{z-y}$ (38)

and

$2^{rx}k^{x-z}+1=(2^{r}+p^{s})^{z}.$ (39)

We see from (38) that $k$ is a power of $p$. So we have

$k^{x-z}=p^{t},\,\,t\in\mathbb{N}.$ (40)

Substituting (40) into (39), we get

$(2^{r}+p^{s})^{z}-1=2^{rx}p^{t}.$ (41)

We find from (41) that equation

$X^{\ell}-1=2^{m}p^{n},\,\,X,\ell,m,n\in\mathbb{N},\,\,X>1,\ell>1$ (42)

has a solution

$(X,\ell,m,n)=(2^{r}+p^{s},z,rx,t).$ (43)

Since $r\geq 2$ and $x>z>y\geq 1$, we have $x\geq 3$ and $rx\geq 6.$ Hence, by
Lemma 10, the solution (43) must satisfy the conclusion $(ii)$ or $(iii)$ in
this lemma.

When (43) satisfies the conclusion $(ii)$ in Lemma 10, by (10) and (43), we
have

$z=2,\,\,t=1,\,\,2^{r}+p^{s}=2^{rx-1}+\zeta,\,\,p=2^{rx-2}+\zeta,\,\,\zeta\in\\{1,-1\\}.$
(44)

Since $z=2$ and $z>y\geq 1$, we get $y=1$. Hence, by (38), we have

$k=p^{s}.$ (45)

Further, by (40), (43) and (45), we get
$p=p^{t}=k^{x-z}=p^{s(x-z)}=p^{s(x-2)}$, whence we obtain $s=1$ and $x=3$.
Therefore, by the third and fourth equalities of (44), we have

$2^{3r-1}+\zeta=2^{rx-1}+\zeta=2^{r}+p^{s}=2^{r}+p=2^{r}+(2^{rx-2}+\zeta)=2^{3r-2}+2^{r}+\zeta,$

whence we get $2^{r}=2^{3r-1}-2^{3r-2}=2^{3r-2}$ and $r=1$, a contradiction.

When (43) satisfies the conclusion $(iii)$ in Lemma 10, by (11) and (43), we
have

$z\,\,\text{is an odd
prime},\,\,2^{r}+p^{s}-1=2^{rx},\,\,\dfrac{(2^{r}+p^{s})^{z}-1}{2^{r}+p^{s}-1}=p^{t},\,\,p\equiv
1\pmod{2z}.$ (46)

By the second equality of (46), we get

$p^{s}=2^{rx}-2^{r}+1.$ (47)

If $t\geq s$, then from the third equality of (46) we have

$0\equiv
p^{t}\equiv\dfrac{(2^{r}+p^{s})^{z}-1}{2^{r}+p^{s}-1}=\dfrac{2^{rz}-1}{2^{r}-1}\,\,\pmod{p^{s}}.$

It implies that $(2^{rz}-1)/(2^{r}-1)$ is a positive integer satisfying

$\dfrac{2^{rz}-1}{2^{r}-1}\geq p^{s}.$ (48)

Since $x>z$, by (47) and (48), we have

$2^{r(z-1)+1}>\dfrac{2^{rz}-1}{2^{r}-1}\geq p^{s}=2^{rx}-2^{r}+1\geq
2^{r(z+1)}-2^{r}+1>2^{r(z+1)}-2^{r}$

and $r\geq 2r-1$, a contradiction. So we have $t<s$. Then, since $z>2$, by the
third equality of (46), we get

$p^{s}>p^{t}=\dfrac{(2^{r}+p^{s})^{z}-1}{2^{r}+p^{s}-1}>\dfrac{(2^{r}+p^{s})^{2}-1}{2^{r}+p^{s}-1}=2^{r}+p^{s}+1>p^{s},$

a contradiction. Thus, the theorem holds for this case.

###  3.2 Case $(ii)$ : $(a,b)=(p^{r},2^{s})$ with $s>1$.

By Lemma 13, we have

$2^{sy}=k^{z-y}$ (49)

and

$p^{rx}k^{x-z}+1=(p^{r}+2^{s})^{z}.$ (50)

We see from (49) that $k$ is a power of 2. So we have

$k^{x-z}=2^{t},\,\,t\in\mathbb{N}.$ (51)

Substituting (51) into (50), we get

$(p^{r}+2^{s})^{z}-1=2^{t}p^{rx}.$ (52)

We find from (52) that (42) has a solution

$(X,\ell,m,n)=(p^{r}+2^{s},z,t,rx).$ (53)

Since $rx\geq x\geq 3,$ by Lemma 10, the solution (53) only satisfies the
conclusion $(iii)$ in this lemma. Then, by (11) and (53), we have

$z\,\,\text{is an odd
prime},\,\,p^{r}+2^{s}-1=2^{t},\,\,\dfrac{(p^{r}+2^{s})^{z}-1}{p^{r}+2^{s}-1}=p^{rx},\,\,p\equiv
1\pmod{2z}.$ (54)

Further, since $rx\geq x>2$, applying Lemma 4 to the third equality of (54),
we get

$2\nmid x.$ (55)

By the first equality of (54), $z$ is an odd prime. Since $z>y\geq 1$, we have
$\gcd(y,z)=1$ and

$\gcd(y,z-y)=1.$ (56)

We see from (49) and (56) that $s\equiv 0\pmod{z-y}$,

$s=(z-y)s_{1},\,\,s_{1}\in\mathbb{N}$ (57)

and

$k=2^{s_{1}y}.$ (58)

Further, by (51) and (58), we have

$t=s_{1}y(x-z),$ (59)

whence we get

$y\mid t.$ (60)

By the second equality of (54), we have

$p^{r}+2^{s}=2^{t}+1$ (61)

and

$p^{r}\equiv-2^{s}\pmod{2^{t}+1}.$ (62)

Substituting (61) into the third equality of (54), we get

$p^{rx}=\dfrac{(2^{t}+1)^{z}-1}{(2^{t}+1)-1}=(2^{t}+1)^{z-1}+\cdots+(2^{t}+1)+1$
(63)

and

$p^{rx}\equiv 1\pmod{2^{t}+1}.$ (64)

Further, by (55), (62) and (64), we have

$2^{sx}+1\equiv 0\pmod{2^{t}+1}.$ (65)

Applying Lemma 6 to (65), we get $sx\equiv 0\pmod{t}$ and

$sx=tt_{1},\,\,t_{1}\in\mathbb{N},\,\,2\nmid t_{1}.$ (66)

Hence, by (57), (59) and (66), we have

$(z-y)x=y(x-z)t_{1}.$ (67)

Furthermore, by (56) and (67), we get

$y\mid x.$ (68)

Therefore, by (60), (63) and (68), we have

$1=(2^{t}+1)^{z}-2^{t}p^{rx}=(2^{t}+1)^{z}-(2^{t/y}p^{rx/y})^{y},$ (69)

where $2^{t/y}p^{rx/y}$ is a positive integer greater than 6. Since $z\geq 3,$
by Lemma 5, we see from (69) that

$y=1.$ (70)

Thus, by (57) and (70), we get

$s\geq z-1.$ (71)

On the other hand, we see from (61) that

$t>s$ (72)

and

$p^{r}\equiv 1\pmod{2^{s}}.$ (73)

Further, by (63), (72) and (73), we have

$1\equiv
p^{rx}\equiv\dfrac{(2^{t}+1)^{z}-1}{(2^{t}+1)-1}=(2^{t}+1)^{z-1}+\cdots+(2^{t}+1)+1\equiv
z\pmod{2^{s}}.$ (74)

Since $z>1$, by (74), we get

$z\geq 2^{s}+1.$ (75)

Therefore, the combination of (71) and (75) yields $s\geq z-1\geq 2^{s}$, a
contradiction. Thus, the theorem holds in this case.

###  3.3 Case $(iii)$ : $(a,b)=(p^{r},q^{s})$

By Lemma 13, we have

$q^{sy}=k^{z-y}$ (76)

and

$p^{rx}k^{x-z}+1=(p^{r}+q^{s})^{z}.$ (77)

We see from (76) that $k$ is a power of $q$. So we have

$k^{x-z}=q^{t},\,\,t\in\mathbb{N}.$ (78)

Substituting (78) into (77), we get

$(p^{r}+q^{s})^{z}-1=p^{rx}q^{t}.$ (79)

We find from (79) that the equation

$X^{\ell}-1=p^{m}q^{n},\,\,X,\ell,m,n\in\mathbb{N},\,\,X>1,\ell>1$ (80)

has a solution

$(X,\ell,m,n)=(p^{r}+q^{s},z,rx,t).$ (81)

Since $p^{r}+q^{s}\geq 8$, applying Lemma 11 to (80) and (81), we only need to
consider the following five subcases.

Subcase $(iii)-1$ :

$2\mid
z,\,\,(p^{r}+q^{s})^{z/2}+\zeta=p^{rx},\,\,(p^{r}+q^{s})^{z/2}-\zeta=q^{t},\,\,\zeta\in\\{1,-1\\}.$
(82)

Since $rx\geq 3$, by Lemma 5, we see from the second equality of (82) that
$z=2.$ So we have

$p^{r}+q^{s}+\zeta=p^{rx},\,\,p^{r}+q^{s}-\zeta=q^{t}.$ (83)

By the first equality of (83), we get

$q^{s}=p^{rx}-p^{r}-\zeta=p^{r}(p^{r(x-1)}-1)-\zeta\geq
p^{r}(p^{2r}-1)-1>p^{r}.$ (84)

However, by the second equality of (83), we have $t>s$ and

$p^{r}=q^{t}-q^{s}+\zeta=q^{s}(q^{t-s}-1)-1\geq q^{s}(q-1)-1>q^{s},$

which contradicts (84). Therefore, this subcase can be eliminated.

Subcase $(iii)-2$ :

$z=p,\,\,\,\,p^{r}+q^{s}-1=p^{rx-1},\,\,\dfrac{(p^{r}+q^{s})^{p}-1}{p^{r}+q^{s}-1}=pq^{t},\,\,q\equiv
1\pmod{2p}.$ (85)

If $t\geq s$, then from the third equality of (85) we get

$0\equiv
pq^{t}\equiv\dfrac{(p^{r}+q^{s})^{p}-1}{p^{r}+q^{s}-1}\equiv\dfrac{p^{rp}-1}{p^{r}-1}\pmod{q^{s}},$
(86)

where $(p^{rp}-1)/(p^{r}-1)$ is a positive integer. However, by (85) and (86),
we have

$\displaystyle 2p^{r(z-1)}=2p^{r(p-1)}$
$\displaystyle=p^{r(p-1)}\sum_{i=0}^{\infty}\dfrac{1}{2^{i}}>p^{r(p-1)}\sum_{j=0}^{p-1}\dfrac{1}{p^{r_{j}}}=\dfrac{p^{rp}-1}{p^{r}-1}$
$\displaystyle\geq q^{s}=p^{rx-1}-p^{r}+1\geq p^{r(z+1)-1}-p^{r}+1,$

whence we get

$\displaystyle p^{r}>p^{r}-1$ $\displaystyle\geq
p^{r(z+1)-1}-2p^{r(z-1)}=p^{r(z-1)}(p^{2r-1}-2)$ $\displaystyle\geq
p^{r(z-1)}(p-2)\geq p^{r(z-1)}\geq p^{r},$

a contradiction. So we have $t<s$. Then, since $p$ is an odd prime, by the
third equality of (85), we get

$pq^{s}>pq^{t}=\dfrac{(p^{r}+q^{s})^{p}-1}{p^{r}+q^{s}-1}\geq\dfrac{(p^{r}+q^{s})^{3}-1}{p^{r}+q^{s}-1}>(p^{r}+q^{s})^{2}>pq^{s},$

a contradiction. Thus, this subcase can be eliminated.

Subcase $(iii)-3$ :

$z=q,\,\,p^{r}+q^{s}-1=q^{t-1},\,\,\dfrac{(p^{r}+q^{s})^{q}-1}{p^{r}+q^{s}-1}=p^{rx}q,\,\,p\equiv
1\pmod{2q}.$ (87)

By (87), we have $p^{r}+q^{s}=q^{t-1}+1$ and

$p^{rx}q=\dfrac{(p^{r}+q^{s})^{q}-1}{p^{r}+q^{s}-1}=\dfrac{(q^{t-1}+1)^{q}-1}{q^{t-1}}=\sum_{i=1}^{q-1}\binom{q}{i}q^{(t-1)(i-1)},$

whence we get

$p^{rx}=1+\sum_{i=2}^{q-1}\binom{q}{i}q^{(t-1)(i-1)-1}.$ (88)

On the other hand, by the second equality of (87), we have

$s<t-1$ (89)

and

$p^{r}=q^{s}(q^{t-s-1}-1)+1.$ (90)

Hence, by (90), we get

$p^{rx}=1+\sum_{j=1}^{x}\binom{x}{j}(q^{s}(q^{t-s-1}-1))^{j}.$ (91)

The combination of (88) and (91) yields

$\sum_{i=2}^{q-1}\binom{q}{i}q^{(t-1)(i-1)-1}=\sum_{j=1}^{x}\binom{x}{j}(q^{s}(q^{t-s-1}-1))^{j}.$
(92)

Let $q^{\alpha}\mid\mid x$, where $\alpha$ is a nonnegative integer. Notice
that

$q^{t-1}\mid\mid\sum_{i=2}^{q-1}\binom{q}{i}q^{(t-1)(i-1)-1},\,\,q^{\alpha+s}\mid\mid\sum_{j=1}^{x}\binom{x}{j}(q^{s}(q^{t-s-1}-1))^{j}.$
(93)

By (92) and (93), we have $t-1=\alpha+s$. Further, by (89), we get
$\alpha=(t-1)-s>0.$ It implies that

$q\mid x.$ (94)

Recall that $z>y$ and $z=q$ is an odd prime. Hence, $y$ and $z$ satisfy (56).
By (56) and (76), we have $s\equiv 0\pmod{z-y}$ and (57). Further, by (76) and
(78) we get $k=q^{s_{1}y}$ and

$t=s_{1}y(x-z)=s_{1}y(x-q).$ (95)

Furthermore, by (94) and (95), we obtain

$q\mid t.$ (96)

Therefore, by (87), (94) and (96), we have

$\displaystyle p^{rx/q}q^{t/q}\in\mathbb{N},\,\,1$
$\displaystyle=(p^{r}+q^{s})^{q}-p^{rx}q^{t}=(p^{r}+q^{s})^{q}-(p^{rx/q}q^{t/q})^{q}$
$\displaystyle=(p^{r}+q^{s}-p^{rx/q}q^{t/q})\sum_{i=0}^{q-1}(p^{r}+q^{s})^{q-i-1}(p^{rx/q}q^{t/q})^{i}>1,$

a contradiction. Thus, this subcase can be eliminated.

Subcase $(iii)-4$ :

$z\,\,\text{is an odd
prime},\,\,p^{r}+q^{s}-1=p^{rx},\,\,\dfrac{(p^{r}+q^{s})^{z}-1}{p^{r}+q^{s}-1}=q^{t},\,\,q\equiv
1\pmod{2z}.$ (97)

We can eliminate this subcase by using the same method as in the proof of
Subcase $(iii)-2$. An outline of this process is given below. If $t\geq s,$
then from the second and third equalities of (97) we have

$0\equiv
q^{t}\equiv\dfrac{(p^{r}+q^{s})^{z}-1}{p^{r}+q^{s}-1}\equiv\dfrac{p^{rz}-1}{p^{r}-1}\pmod{q^{s}},$

whence we get $(p^{rz}-1)/(p^{r}-1)\geq q^{s}.$ So we have

$p^{r(z-1)+1}>\dfrac{p^{rz}-1}{p^{r}-1}\geq q^{s}=p^{rx}-p^{r}+1\geq
p^{r(z+1)}-p^{r}+1>p^{r(z+1)-1},$

a contradiction. It implies that $t<s$. Then, by (97), we get

$q^{s}>q^{t}=\dfrac{(p^{r}+q^{s})^{z}-1}{p^{r}+q^{s}-1}>p^{r}+q^{s}+1>q^{s},$

a contradiction. Thus, this subcase can be eliminated.

Subcase $(iii)-5$ :

$z\,\,\text{is an odd
prime},\,\,p^{r}+q^{s}-1=q^{t},\,\,\dfrac{(p^{r}+q^{s})^{z}-1}{p^{r}+q^{s}-1}=p^{rx},\,\,p\equiv
1\pmod{2z}.$ (98)

We see from the second equality of (98) that $t>s$. So we have
$p^{r}=q^{s}(q^{t-s}-1)+1$ and

$p^{r}\equiv 1\pmod{q^{s}}.$ (99)

By the second and third equalities of (98), we get

$p^{r}+q^{s}=q^{t}+1$ (100)

and

$p^{rx}=\dfrac{(p^{r}+q^{s})^{z}-1}{p^{r}+q^{s}-1}=\dfrac{(q^{t}+1)^{z}-1}{(q^{t}+1)-1}.$
(101)

Further, by (101), we have

$p^{rx}\equiv(q^{t}+1)^{z-1}+\cdots+(q^{t}+1)+1\equiv z\pmod{q^{t}}.$ (102)

Since $t>s$, by (99) and (102), we get $z\equiv 1\pmod{q^{s}}$. Furthermore,
since $z>1$, we have

$z\geq q^{s}+1.$ (103)

On the other hand, since $z$ is an odd prime with $z>y\geq 1$, $y$ and $z$
satisfy (56). Hence, by (56) and (76), $s$ satisfies (57), whence we can
obtain (60).

Since $p^{r}+q^{s}>3,$ applying Lemma 4 to the third equality of (98), we have

$2\nmid x.$ (104)

By (101), we have

$p^{rx}\equiv\dfrac{(q^{t}+1)^{z}-1}{q^{t}+1}\equiv(q^{t}+1)^{z-1}+\cdots+(q^{t}+1)+1\equiv
1\pmod{q^{t}+1}.$ (105)

Since $p^{r}\equiv-q^{s}\pmod{q^{t}+1}$ by (100), we get from (104) and (105)
that

$0\equiv p^{rx}-1\equiv(-q^{s})^{x}-1\equiv-(q^{sx}+1)\pmod{q^{t}+1}.$ (106)

Hence, by Lemma 6, we see from (106) that $x$ satisfies (66) and (67).
Further, by (56) and (67), $x$ satisfies (68). Therefore, by (60) and (68), we
get from the third equality of (98) that

$p^{rx/y}q^{t/y}\in\mathbb{N},\,\,(p^{r}+q^{s})^{z}-(p^{rx/y}q^{t/y})^{y}=1.$
(107)

Since $z\geq 3$ and $p^{r}+q^{s}>3$, by Lemma 5, we find from (107) that $y$
satisfies (70). Furthermore, by (57) and (70), $s$ satisfies (71). However,
the combination of (71) and (103) yields $s\geq z-1\geq q^{s}$, a
contradiction. Thus, this subcase can be eliminated.

To sum up, the theorem is proved.

##  4 Acknowledgments

We thank an anonymous referee for carefully reading our paper and for giving
such constructive comments which substantially helped improving the quality of
the paper.

## References

  * [1] H. Aboudja, M. Hernane, S. E. Rihane and A. Togbé, On perfect powers that are sums of two Pell numbers, Period. Math. Hungar. 82 (2021), 11–15.
  * [2] J. Bravo and F. Luca, Powers of two as sum of two Lucas numbers, J. Integer Seq. 17 (2014), Article 14.8.3.
  * [3] G. D. Birkhoff and H. S. Vandiver, On the integral divisors of $a^{n}-b^{n}$, Ann. of Math. 2 (1904), 173–180.
  * [4] M. Ddamulira and F. Luca, On the exponential Diophantine equation related to powers of two consecutive terms of Lucas sequences, Ramanujan J. 56 (2021), 651–684.
  * [5] L. Jeśmanowicz, Several remarks on Pythagorean numbers, Wiadom. Math. 1 (1955-1956), 196–202. (in Polish)
  * [6] N. H. Kohno and F. Luca, On the Diophantine equation $F_{n}^{x}+F_{n+1}^{x}=F_{m}^{y}$, Rocky Mountain J. Math. 45 (2015), 509–538.
  * [7] M. H. Le, R. Scott and R. Styer, A survey on the ternary purely exponential Diophantine equation $a^{x}+b^{y}=c^{z}$, Survey Math. Appl. 14 (2019), 109–140.
  * [8] M. H. Le and G. Soydan, A note on the exponential Diophantine equation $(A^{2}n)^{x}+(B^{2}n)^{y}=((A^{2}+B^{2})n)^{z}$, Glasnik Math. 55 (2020), 195–201.
  * [9] W. Ljunggren, Noen setninger om ubstemte likninger av formen $(x^{n}-1)/(x-1)=y^{q}$, Norsk Mat. Tidsskr. 25 (1943), 17–20.
  * [10] P. Mihǎilescu, Primary cyclotomic units and a proof of Catalan’s conjecture, J. Reine Angew. Math. 572 (2004), 167–195.
  * [11] T. Nagell, Introduction to Number Theory, New York, John Wiley & Sons, 1951.
  * [12] S. E. Rihane, B. Faye, F. Luca and A. Togbé, On the exponential Diophantine equation $P_{n}^{x}+F_{n+1}^{x}=P_{m}$, Turkish J. Math. 43 (2019), 1640–1649.
  * [13] Z. Siar, On the the exponential Diophantine equation $F_{n}^{x}\pm F_{m}^{x}=a$ with $a\in\\{F_{r},L_{r}\\}$, Int. J. Number Theory 19 (2023), 41–57.
  * [14] N. J. A. Sloane et al., The On-Line Encyclopedia of Integer Sequences, 2022. Available at https://oeis.org.
  * [15] G. Soydan, M. Demirci, I. N. Cangül and A. Togbé, On the conjecture of Jeśmanowicz, Int. J. Appl. Math. Stat. 56 (2017), 46–72.
  * [16] C. F. Sun and M. Tang, On the Diophantine equation $(an)^{x}+(bn)^{y}=(cn)^{z}$, Chinese Ann. Math. Ser. A 39 (2018), 87–94. (in Chinese)
  * [17] P. Z. Yuan and Q. Han, Jeśmanowicz’ conjecture and related equations, Acta Arith. 184 (2018), 37–49.
  * [18] Z. F. Zhang and A. Togbé, Perfect powers that are sums of two powers of Fibonacci numbers, Bull. Aust. Math. Soc. 99 (2019), 34–41.

2020 Mathematics Subject Classification: Primary 11D61; Secondary 11B39.

_Keywords:_ Lucas sequence, ternary purely exponential Diophantine equation,
Nagell-Ljunggren equation, Catalan equation, prime power base, elementary
number theory method.