[7]Equal contributions. Work was done during Zhenglin's visit to Westlake University.

The Sparse Mixture of Experts (SMoE) has been widely employed to enhance the efficiency of training and inference for Transformer-based foundational models, yielding promising results.
However, the performance of SMoE heavily depends on the choice of hyper-parameters, such as the number of experts and the number of experts to be activated (referred to as top-$k$), resulting in significant computational overhead due to the extensive model training by searching over various hyper-parameter configurations.
As a remedy, we introduce the () technique.
incorporates (1) a novel gating method that enables each token to automatically determine the number of experts to activate.
(2) An adaptive process automatically adjusts the number of experts during training.
Extensive numerical results across Vision, Language, and Vision-Language tasks demonstrate the effectiveness of our approach to achieve competitive performance compared to GMoE for vision and language tasks, and MoE-LLaVA for vision-language tasks, while maintaining efficiency by activating fewer parameters.
Our code is available at  <https://github.com/LINs-lab/DynMoE>.

§ INTRODUCTION

Illustration of performance fluctuation on various MoE settings.
We carried out experiments on GLUE benchmark [49], employing BERT-large [8] as backbone.
The $x$-axis represents the MoE settings, while the $y$-axis shows the performance on the COLA dataset.

The scalable nature of Transformer models [22] has gained remarkable successes across a spectrum of applications, ranging from language [1, 46, 47] and vision [23, 35] to cross-modality domains [32, 29, 28].
To further enhance performance while maintaining high efficiency, Sparse Mixture of Experts (SMoE) has emerged as a promising technique that significantly reduces computation costs during both training and inference stages [13, 24, 56], and has been shown to achieve comparable or superior performance compared to traditional dense models [25, 21, 7].

Despite its success, SMoE has an unavoidable drawback: the performance of SMoE heavily relies on the choice of hyper-parameters, such as the number of activated experts per token, referred as top-$k$, and the number of experts [6, 12, 53], denoted as $K$. As illustrated in Figure <ref>, the performance discrepancy of MoE models under various configurations can be approximately 1%-3%.
Notably, identifying the optimal hyper-parameter without a sufficient number of ablation studies is challenging.
As the size of the models continues to grow, this limitation could result in a significant waste of computational resources, and in turn, could hinder the efficiency of training MoE-based models in practice.

To tackle the above problems, the objective of this paper is to explore a novel training technique for MoE models, with the aim of addressing the following core question:

Is it possible to develop a MoE training strategy that can automatically determine the number of experts and the number of activated experts per token during the training process?

Hence, we introduce the () method, which addresses the aforementioned question through the introduction of two innovative components: (1) a top-any gating method that enables each token to autonomously determine the number of experts to activate, thereby allowing different tokens to activate varying numbers of experts; (2) an adaptive training process that dynamically adjusts the number of experts, increasing it when the current quantity is inadequate and removing redundant experts as necessary.
Additionally, we introduce a new auxiliary loss function specifically designed to encourage sparsity when employing the top-any gating approach. This loss encourages different experts to be diverse, rather than mandating that all experts be activated with the same frequency.
We summarize the contributions of this paper as follows:

* Introducing , a novel method frees the burden of pivotal hyper-parameter selection for MoE training, which is capable of autonomously determining the number of experts and the number of experts to be activated per token. We provide Tutel and DeepSpeed-MoE implementations for ease of practical usage.

* Conducting extensive empirical experiments across Vision, Language, and Vision-Language tasks.
The results illustrate that achieves comparable or superior performance compared to the well-tuned MoE settings.

§ RELATED WORKS

The Sparse Mixture of Experts (SMoE) approach [11, 44, 24] has been proven to effectively enhance the training and inference efficiency of foundational models. Contemporary studies primarily modify the MLP layer of transformer models into multiple expert models and employ a gating network to determine which expert to select. They only choose a subset of experts for each token during both training and inference [24, 13]. Recently, the SMoE structure has shown success in various research areas. For instance, GMoE [26] has demonstrated that SMoE can enhance generalization performance in vision tasks. Large Language Models (LLMs) have also employed MoE to simultaneously reduce training and inference costs while improving model performance [13, 21, 7, 42, 31]. However, most of these models employ standard SMoE structures and apply the SMoE to various tasks. Our paper focuses on improving the MoE training process, which can be easily integrated with these methods.

Recently, some attempts have been made to improve the architecture of MoE models. For example, researchers have investigated the benefits of sample-wise [41, 15] and token-wise [44, 43, 13] routing. Some studies introduce load balancing loss to ensure that the experts are activated an equal number of times [24, 13]. Expert choice routing [57] addresses load balance by allowing experts to choose tokens; however, this approach also suffers from dropped tokens. SoftMoE [37] uses a slot mechanism to simultaneously resolve the issues of load balance and dropped tokens. Nevertheless, these approaches also require pre-defined hyperparameters, such as the number of experts or the number of experts to be activated. In this paper, we tackle this problem by presenting , an algorithm that automatically determines the number of activated experts for each token and dynamically adds or removes experts during the training process. Furthermore, we introduce a new auxiliary loss function that ensures sparsity when utilizing the algorithm.

§ METHOD

Illustration of the top-any gating method. The input tokens pass through the gating weights $\mW_{g,e}$ corresponding to each expert $e$, obtaining the gating scores. These gating scores are then compared to the gates $\mG_e$ to determine if the subsequent expert will be activated. Finally, the expert outputs are combined to produce the output tokens.

In this section, we introduce the (), an algorithm capable of automatically determining the number of experts and the number of experts to be activated for both training and inference stages.
This is achieved through the incorporation of two crucial components:

(1) The top-any gating method (Figure <ref>), which models the gating mechanism as a multi-label classification problem, allowing tokens to decide the number of experts to be activated on their own.
This enables different tokens to activate varying numbers of experts, including the option to activate no experts.

(2) A carefully designed adaptive process that adds new experts when tokens choose to not activate any existing experts, and removes any surplus experts that have not been activated by any tokens.

The overall process is summarized in Algorithm <ref>.

§.§ Top-Any Gating

In this section, we present the superior gating method to eliminate the need for tuning the top-$k$ value.
We further improve the test-time inference procedure and introduce an additional auxiliary loss to prevent token dropping and boost efficiency.

Traditional top-$k$ gating and the limitations.
The traditional top-$k$ gating method uses the token embedding $\xx$ as inputs and uses an additional gating network $g$ to predict the scores that the input token embedding assigned to each expert.
Typically, given token $\xx \in \R^{d}$ as input, the gating process is defined as the follows [40, 20]:

\begin{align}
g(\xx) \in \R^{K} := \text{softmax}(\mW_g^{T} \xx) \,,
\end{align}

where $\mW_g \in \R^{d \times K}$ is the parameter of the gating network, and $K$ is the number of experts.
Then the output of the MoE layer is defined by

\begin{align}
\yy = \frac{1}{\sum_{e \in \text{Top-}k \left( g(\xx) \right) } g(\xx)_e } \sum_{e \in \text{Top-}k \left( g(\xx) \right) } g(\xx)_e E_e(\xx) \,,
\end{align}

where $ E_e(\xx) \in \R^{d} $ is the output of $e$-th expert given input $\xx$, and $g(\xx)_e$ is the $e$-th entry of $g(\xx)$.

Despite the considerable success of the top-$k$ gating method in enhancing training and inference efficiency, two limitations persist:

* The value of $k$ must be fine-tuned to optimize model performance.
As demonstrated in Figure <ref>, the performance of MoE models can vary significantly with different top-$k$ values.
This observation has also been noted in recent studies [6, 12, 53].
Consequently, substantial computational resources are needed to identify the optimal value of $k$.

* The top-$k$ gating approach assumes that each token must activate the same number of experts, which may not always hold in practice.
For instance, when considering different tasks, there could exist tokens shared by all tasks and those specific to certain tasks, i.e. different tokens could activate different numbers of experts.

Addressing the limitations of top-$k$ gating by tuning-free top-any gating.
To address the aforementioned limitations, we propose the top-any gating method, which does not require a pre-defined value of $k$ and allows different tokens to activate varying numbers of experts during both training and inference stages.

The design of the top-any gating method draws inspiration from the multi-label classification problem.
We consider each expert as an individual class and calculate the classification (gating) score for each class (expert) independently.
Subsequently, all classes (experts) with scores exceeding the threshold are deemed positive (activated).
In detail, given the expert representation matrix $\mW_g \in \R^{K \times d}$, where the $k$-th row of $\mW_g$ acts as the representation of expert $k$, and an input token $\xx \in \R^{d}$, the key steps of top-any gating can be formulated by the following equation:

\begin{align}
s(\xx) & = \frac{\left \langle \xx, \mW_{g} \right \rangle}{\norm{\xx} \norm{\mW_{g}}} \,, \label{equ:gating-sim} \\
g(\xx) & = \text{sign} \left( \sigma \left( s(\xx) \right) - \sigma( \mG ) \right) \,, \label{equ:gating-score}
\end{align}

where $\mW_g \in \R^{K \times d}$ and $\mG \in \R^{K}$.
To illustrate, we first compute the cosine similarities between the token and the expert representation matrix $\mW_g$ and obtain the similarity score $s(\xx) \in \R^{K}$.
Then the sigmoid function $\sigma$ is applied to the similarity score $s(\xx)$ to obtain the scores between $0$ and $1$.
Finally, experts with similarity scores greater than the trainable per-expert threshold $\mG$ are considered to activate experts for the token $\xx$.
It is important to note that the sign function does not support back-propagation, and thus we customize the back-propagation process of this part by directly copying the gradient of $g(\xx)$ to $\sigma \left( s(\xx) \right) - \sigma ( \mG )$ to effectively bypass the sign function.

Given the gating score $g(\xx) \in \R^{K}$, the number of activated experts is then defined by

\begin{align}
k := \text{sum} \left( g(\xx) \right)  \,, \label{equ:gating-k}
\end{align}

where $k$ represents the number of experts to be activated for token $\xx$.
The model output of the MoE layer with the top-any gating method can be derived as follows

\begin{align}
\yy = \frac{1}{k} \sum_{g(\xx)_e > 0} E_{e}(\xx) \,. \label{equ:gating-outputs}
\end{align}

Improving the top-any gating during test-time to prevent token dropping.
To facilitate the design of the adaptive expert number process, we did not impose a minimum value on $k$.
Consequently, some tokens may not activate any experts.
To address this issue, during model performance evaluation, we modify the top-any gating to enable top-$1$ gating for tokens that do not choose to activate any experts.
In detail, for the input token $\xx$ with $\text{sum}(g(\xx)) = 0$, the modified gating score $\tilde{g}(\xx)$ is obtained by

\begin{align}
\tilde{g}(\xx)_k =
\begin{split}
\left \{            \begin{array}{ll}
0               & k \not = \argmax_{k} \sigma (s(\xx)) \,, \\
\sigma (s(\xx)) & k = \argmax_{k} \sigma (s(\xx)) \,.
\end{array}
\right.
\end{split}
\end{align}

Guarding efficiency for top-any gating by auxiliary loss.
The primary goal of using MoE models is to improve the training and inference efficiency.
However, in the absence of a cap on the maximum number of activated experts, tokens might activate all experts, which is counterproductive to our primary goal.

Using an auxiliary loss as a regularization over experts may alleviate our issue.
However, existing auxiliary loss methods [24, 13, 51] are primarily designed to ensure load balancing across experts and thus cannot align with our objectives.
While activating all experts can indeed achieve load balancing, it contradicts our aim of improving efficiency by limiting the number of activated experts.
Therefore, we need a solution that not only ensures load balancing but also restricts the number of activated experts.

As a remedy, we propose a new auxiliary loss, namely sparse and simple gating loss, as shown in (<ref>).
The diversity loss and simplicity loss in (<ref>) work together to improve the efficiency of the model by addressing different aspects of the expert representations.
On one hand, the diversity loss encourages independence among the $\mW_g$ representations of various experts.
It serves two purposes: First, it prevents a high degree of similarity between experts, thereby enhancing the model's representational capacity;
Second, it guides tokens to avoid simultaneous activation of all experts, thereby promoting sparse gating for improved efficiency.
On the other hand, the simplicity loss normalizes $\mW_g$ to avoid excessively large values within the matrix, which helps maintain numerical stability and prevents overfitting due to extreme parameter values.
The detailed loss function is defined as follows:

\begin{align}
\textstyle
\cL = \underbrace{\norm{\mW_g^{T} \mW_g - \mI_K}_2}_{\emph{diversity loss}} + \underbrace{\frac{1}{K} \sum_{e=1}^{K} \norm{\ww_{g, e}}_2}_{\emph{simplicity loss}} \,, \label{equ:gating-loss}
\end{align}

where $\mI_K$ is the identity matrix with dimension $K$, and $\ww_{g, e} \in \R^{d}$ is the $e$-th element of $\mW_g$, indicating the representation of the $e$-th expert.

§.§ Adaptive Training Process

[1]
Input data $\xx$, initial gating network parameters $\mW_g$, $\mG$, and $\tau$, experts $E_1, \cdots, E_K$, start record routing flag $flag_{s}$, finish record routing flag $flag_{f}$.
MoE layer output $\yy$, auxiliary loss value.
Set routing flag $flag_{rout} = 1$.
Initialize routing records by $\mR_{\text{rout}} = \0_{K}$.
Initialize non-activate sample records $\mR_{\text{sam}} = \0_{d}$.
Get the gating outputs $g(\xx)$ and $\kk$ by Eq (<ref>) and (<ref>).
Get MoE layer output $\yy$ by Eq (<ref>).
Calculate auxiliary loss by Eq (<ref>).
$flag_{rout} = 1$
$\mR_{E} = \mR_{E} + \text{sum}(g(\xx), \text{dim}=0)$.
$\mR_{S} = \mR_{S} + \sum_{i=1}^{N} \1_{\kk_i = 0} \xx_i$
$flag_{rout} = 0$.
Exists $e$ that $\mR_{\text{E}}^{e} = \mathbf{0}$
Remove experts $e$.
$\mR_{\text{S}, e} \not = \mathbf{0}$
Add new expert $K + 1$ with expert representation $\mW_{g, K + 1} = \mR_{S} / \norm{\mR_{S}}$.

algorithmPseudo code of on each iteration and MoE layer.

In this section, we elaborate on the adaptive training process, which is designed to automatically determine the number of experts.
As illustrated in Figure <ref>, the adaptive process consists of three parts, namely
(1) Routing Recording: recording the routing results during training;
(2) Adding Experts: adding new experts when tokens choose not to activate any existing experts;
and (3) Removing Experts: removing experts that have not been chosen by any tokens.

Routing Recording.
To facilitate the removal and addition of experts, it is essential to track the routing status.
Specifically, we record two key pieces of information for each MoE layer: (1) For each expert $e$, we record the time at which expert $e$ is activated, denoted as $\mR_{E} \in \R^{K}$ (as shown in Line 9 of Algorithm <ref>).
(2) For input data that does not activate any expert, we compute the sum of their embeddings $\xx$ as $\mR_{S} \in \R^{d}$ (as outlined in Line 10 of Algorithm <ref>).
Note that this approach simplifies the expert addition process: by using the token embeddings to initialize the expert representation $\mW_g$, we can achieve a high similarity score between these tokens and the new experts, ensuring that the new expert will be activated by these tokens when added.

As demonstrated in Algorithm <ref>, we utilize $flag_{s}$ and $flag_{f}$ to determine when to start and stop routing recording.
Users can control these two flags as needed.

Adding Experts when there exist tokens that choose not to activate any experts.
We add new experts when the recorded $\mR_{S} \not = \mathbf{0}$, as some tokens do not activate any experts and $\mR_{S}$ is the sum of these tokens.
Therefore, given $K$ activated experts and new expert $K + 1$, we initialize $\mW_{g, K + 1} = \frac{\mR_{S}}{\norm{\mR_{S}}}$ and $\mG_{K+1} = \mathbf{0}$.

Removing Experts when there exist experts not activated by any token.
We remove experts when there is an expert $e$ such that $\mR_{E}^{e} = \mathbf{0}$ (as shown in Line 13 in Algorithm <ref>).

Elaboration on the adaptive training process.
We visualize the adaptive training process of , including record routing, experts adding, and experts removing.
The green strip connecting the token and the expert indicates records of a token routing to an expert.
The red arrow at the bottom part of the figure shows where and when expert addition and removal happens.

§ EXPERIMENTS

In this section, we carry out experiments to address the following questions:

* Q1: Can achieve competitive performance among different MoE settings? See <ref>.

* Q2: Can handle tasks with varying modalities and scales? See <ref>.

* Q3: Will the model trained by maintain sparsity to ensure efficiency? See <ref>.

* Q4: Can offer insights that could guide the design of MoE models? See <ref>.

§.§ Experiment Setup
To answer the above four questions, we conduct experiments on Vision, Language, and Vision-Language tasks. The details are shown in the following.

* Vision Task. For the vision tasks, we follow the same settings as in GMoE [26]. We employ the pre-trained ViT-S/16 [10] model and evaluate it on the DomainBed [16] benchmark.
Our experiments encompass four Domain Generalization datasets: PACS [27], VLCS [2], OfficeHome [48], and DomainNet [36].
All results are reported using the train-validation selection criterion.

* Language Task. The language tasks adhere to the same settings as those in MoEfication [56] and EMoE [38].
The MoE models are built upon the BERT-large [8] architecture using the MoEfication method and are fine-tuned on GLUE [49] tasks, which include COLA [50], QNLI [49], RTE [5], MNLI [52], and MRPC [9].

* Vision-Language Task. The vision-language tasks follows the setting in MoE-LLaVA [31], where we use StableLM-2-1.6B [4], Qwen-1.8B [3] and Phi-2-2.7B [19] as backbone language models, and use clip-vit-large-patch14-336 [39] as the vision encoder.
The models are evaluated on image understanding benchmarks including VQA-v2 [14], GQA [18], VisWiz [17], ScienceQA-IMG [34], TextVQA [45], POPE [30], MME [54], MMBench [33], LLaVA-Bench (in-the-Wild) [32], and MM-Vet [55]. Furthermore, we keep routing records in our model during testing time. For each benchmark, we collect the number of experts' activations per MoE layer and total processed tokens during testing.

§.§ A1: Achieves Competitive Performance among Various MoE Settings

In this section, we carry out experiments on the GLUE benchmark [49], varying the number of experts ($K$) and the value of top-$k$.
The results of these experiments can be observed in Figure <ref>.=-1

The performance of surpasses the average performance among various MoE settings.
As seen in Figure <ref>, we can observe that

* The performance of is higher than the average performance across different values of $K$ and top-$k$ in most tasks, indicating the competitive performance of .

* The performance fluctuates considerably with different $K$ and top-$k$ values, such as up to 3.0% on the RTE task and 1.3% on the COLA task.
overcomes this issue by not requiring pre-defined $K$ and top-$k$ values.

* The performance gain of specific $K$ and top-$k$ choice is not consistent among tasks.
For instance, the $K = 16, k = 4$ setting performs well on QNLI but poorly on MRPC.
In contrast, the always achieve competitive performance among tasks.

Performance of on language tasks.
We conduct experiments on the GLUE benchmark.
The $x$-axis represents MoE settings with varying $K$ and top-$k$ values.
The $y$-axis denotes the model's performance.
Dashed lines indicate the average performance across different settings, as well as the performance of .=-1

§.§ A2: Can Handle Vision, Language, and Vision-Language Tasks

In addition to Language tasks, we also conduct experiments on Vision and Vision-Language tasks to verify the performance of on different modalities and task scales.
The results can be found in Tables <ref>, and <ref>.

The effectiveness of remains consistent in both Vision and Vision-Language tasks.
We can observe the following: (1) outperforms well-tuned MoE [38] in Vision tasks. The performance difference between and  well tuned MoE in [26], falls within the range of random fluctuation. (2) When using StableLM-1.6B and Phi-2-2.7B as the backbone, the performance of -LLaVA surpasses that of MoE-LLaVA. (3) With Qwen-1.8B as the backbone, the performance of -LLaVA remains comparable to MoE-LLaVA. In this setting, the average top-$k$ of -LLaVA (avg $k = 1.86$) is also close to the MoE-LLaVA setting ($k=2$).

Performance of on vision tasks:
Our study investigates the performance of on vision tasks using the DomainBed benchmark, with ViT-small serving as the backbone model.
The effectiveness of GMoE is elucidated based on meticulously tuned results as presented in the previous works [26] and [38].
In our implementation of , we configure the maximum number of experts to $8$, with an initial setting of 6 experts.
The number of experts is dynamically adjusted in each iteration for .
We also report the performance of using Gshard loss [24] as the auxiliary loss.

Algorithms                                               PACS     VLCS     OfficeHome     DomainNet     Average

GMoE (in [26])                            88.1     80.2     74.2           48.7          72.8

GMoE (carefully tuned [38])           87.7     79.6     73.1           -             -

GMoE (with , Gshard Loss)                        88.4     79.4     73.6           47.4          72.2

GMoE (with , Diverse and Simple Gating Loss)     87.6     80.3     73.5           48.2          72.4

Performance of on vision-language tasks: Our study investigates the performance of -LLaVA on image understanding benchmarks.
Evaluation Benchmarks include VQA-v2; GQA; VisWiz; SQA$^I$ (ScienceQA-IMG); VQA$^T$ (TextVQA); POPE; MME; MMB (MMBench); LLaVA$^W$ (LLaVA-Bench (in-the-Wild)); MM-Vet.
For a fair comparison, we set the maximum number of experts to 4 for -LLaVA (the same as the number of experts in MoE-LLaVA) and set the initial number of experts to $2$.
$N_{A}$ indicates the number of activated parameters.

Algorithms                               $N_{A}$                        VQA$^{v2}$                      GQA                             VisWiz                          SQA$^I$                         VQA$^T$                         POPE                            MME                               MMB                             LLaVA$^W$                       MM-Vet

[r]LLaVA-1.5 (Vicuna-13B)     13B     80.0     63.3     53.6     71.6     61.3     85.9     1531.3     67.7     70.7     35.4

[r]LLaVA-1.5 (Vicuna-7B)      7B      78.5     62.0     50.0     66.8     58.2     85.9     1510.7     64.3     63.4     30.5

[r]LLaVA-Phi (Phi-2-2.7B)       2.7B                           71.4                            -                               35.9                            68.4                            48.6                            85.0                            1335.1                            59.8                            -                               28.9

Sparse (StableLM-1.6B)

($K=4,k=2$)     2.06B     76.7     60.3     36.2     62.6     50.1     85.7     1318.2     60.2     86.8     26.9

(avg $k = 1.25$)     1.75B     77.4     61.4     40.6     63.4     48.9     85.7     1300.9     63.2     86.4     28.1

Sparse (Qwen-1.8B)

($K=4,k=2$)     2.24B     76.2     61.5     32.6     63.1     48.0     87.0     1291.6     59.7     88.7     25.3

(avg $k = 1.86$)     2.19B     76.4     60.9     32.4     63.2     47.5     85.8     1302.4     61.3     89.2     24.2

Sparse (Phi-2-2.7B)

($K=4,k=2$)     3.62B     77.6     61.4     43.9     68.5     51.4     86.3     1423.0     65.2     94.1     34.3

(avg $k = 1.68$)     3.35B     77.9     61.6     45.1     68.0     51.8     86.0     1429.6     66.6     95.6     33.6

§.§ A3: Maintains Efficiency by Activating Less Parameters

In this section, we aim to demonstrate that although we did not enforce sparsity on the models, the trained models are still sparse, promising improved inference efficiency.

-LLaVA activates fewer parameters compared to MoE-LLaVA.
In Table <ref>, we display the number of activated parameters in the "$N_A$" column.
When using StabeLM-1.6B as the backbone, -LLaVA activates approximately 15.0$\%$ fewer parameters than MoE-LLaVA.
For Qwen-1.8B, -LLaVA activates about 2.2$\%$ fewer parameters than MoE-LLaVA.
For Phi-2-2.7B, -LLaVA activates about 7.5$\%$ fewer parameters than MoE-LLaVA.
In these three cases, the reduction in activated parameters does not compromise the model's performance.

Ablation studies on the value of top-$k$ during test.
In Table <ref>, we examine the performance of -LLaVA when using different top-$k$ values during the testing phase.
The results indicate that (1) The original -LLaVA outperforms other settings in most cases while activating the fewest number of parameters.
(2) Compared to the StableLM-1.6B backbone, -LLaVA trained with the Qwen-1.8B backbone sometimes favors activating two experts.
This observation aligns with the fact that -LLaVA also chooses to activate about $2$ experts (see Table <ref>).

Average top-$k$ activated experts of on vision-language benchmarks. We record average top-$k$ activated experts for each MoE layer when using StableLM-1.6B as the language model backbone.

Ablation studies on the value of top-$k$ during test.
We train the models using and set different values of top-$k$ during the test.
Training and evaluation settings are identical to that of Table <ref>.

Algorithms           $N_{A}$     VQA$^{v2}$     GQA      VisWiz     SQA$^I$     VQA$^T$     POPE     MME        MMB      LLaVA$^W$     MM-Vet

-LLaVA             1.75B       77.4           61.4     40.6       63.4        48.9        85.7     1300.9     63.2     86.4          28.1

-LLaVA ($k=2$)     2.06B       76.9           61.0     39.1       62.1        49.2        85.7     1320.4     62.4     73.6          28.2

-LLaVA ($k=3$)     2.47B       76.8           60.7     37.0       62.6        48.9        85.5     1306.9     62.5     74.0          26.8

-LLaVA ($k=4$)     2.89B       76.8           60.5     34.8       61.9        49.0        85.8     1321.9     61.9     75.8          27.8

-LLaVA             2.19B       76.2           61.5     32.6       63.1        48.0        87.0     1291.6     59.7     88.7          25.3

-LLaVA ($k=2$)     2.24B       76.2           60.8     33.8       62.2        47.7        87.5     1281.3     60.4     91.3          23.0

-LLaVA ($k=3$)     2.65B       76.2           60.5     32.2       62.9        48.1        88.4     1263.7     60.7     87.8          23.4

-LLaVA ($k=4$)     3.05B       75.7           60.0     31.6       62.8        48.3        88.1     1263.4     61.0     86.7          23.7

-LLaVA             3.35B       77.9           61.6     45.1       68.0        51.8        86.0     1429.6     66.6     95.6          33.6

-LLaVA ($k=2$)     3.62B       77.8           61.5     41.6       67.6        51.8        85.5     1433.5     66.8     95.1          32.7

-LLaVA ($k=3$)     4.46B       77.7           61.8     42.0       68.0        52.3        86.3     1438.1     66.8     94.3          30.8

-LLaVA ($k=4$)     5.30B       77.5           61.4     41.7       68.0        52.4        87.0     1431.5     66.5     95.8          32.8

[Activation frequency (Qwen)]

[Activation frequency (StableLM)]

[Activation frequency (Phi-2)]

Statistics of expert activation frequency in different layers.
We report the frequency of expert activations in various layers for the VQA task.
Larger circles indicate experts that are activated more frequently.

§.§ A4: MoE Structure is Required for Bottom Layer rather than Top Layer

Illustration of the number of activated experts for each layer.
In Figures <ref> and <ref>, we present the average top-$k$ of -LLaVA and the frequency of expert activation across various layers.
Our observations indicate that:
(1) In the top layer (the layer closest to the LM prediction head), tokens tend to select the same expert, while in the bottom layer, tokens activate all experts uniformly.
This suggests that there is no need to convert the top layer to MoE layer, whereas the bottom layer should be transformed into MoE layer.
(2) Different LLM backbones may exhibit distinct expert activation frequency patterns.
For the StableLM backbone, most MoE layers activate only one dominant expert, whereas for the Phi-2 backbone, experts are more likely to be activated uniformly.

§ CONCLUSION AND FUTURE WORKS

In this paper, we introduce , which automatically determines the number of experts and the number of experts to be activated. Our results demonstrate that achieves comparable or even superior performance across various MoE model settings while maintaining efficiency. This highlights 's potential to save researchers' time and computational resources when tuning these hyperparameters. Furthermore, our visualization results reveal interesting observations, such as the reduced number of experts required for the top layers. We believe these insights may inspire future advancements in MoE model design.
However, due to computational resource constraints, we did not test larger scale models. Additionally, the current adaptive process implementation keeps removed experts in a candidate pool, occupying GPU storage. Developing more efficient implementations in the future would be valuable.

[1]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
arXiv preprint arXiv:2303.08774, 2023.

[2]
Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiago H Falk, and Ioannis Mitliagkas.
Generalizing to unseen domains via distribution matching.
arXiv preprint arXiv:1911.00804, 2019.

[3]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu.
Qwen technical report.

[4]
Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, Meng Lee, Emad Mostaque, Michael Pieler, Nikhil Pinnaparju, Paulo Rocha, Harry Saini, Hannah Teufel, Niccolo Zanichelli, and Carlos Riquelme.
Stable lm 2 1.6b technical report.

[5]
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo.
The fifth pascal recognizing textual entailment challenge.
TAC, 7(8):1, 2009.

[6]
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al.
Unified scaling laws for routed language models.
In International conference on machine learning, pages 4057–4086. PMLR, 2022.

[7]
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al.
Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models.
arXiv preprint arXiv:2401.06066, 2024.

[8]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
Bert: Pre-training of deep bidirectional transformers for language understanding.
In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, 2019.

[9]
Bill Dolan and Chris Brockett.
Automatically constructing a corpus of sentential paraphrases.
In Third international workshop on paraphrasing (IWP2005), 2005.

[10]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.
An image is worth 16x16 words: Transformers for image recognition at scale.
In International Conference on Learning Representations, 2020.

[11]
David Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever.
Learning factored representations in a deep mixture of experts.
arXiv preprint arXiv:1312.4314, 2013.

[12]
Dongyang Fan, Bettina Messmer, and Martin Jaggi.
Towards an empirical understanding of moe design choices.
arXiv preprint arXiv:2402.13089, 2024.

[13]
William Fedus, Barret Zoph, and Noam Shazeer.
Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.
Journal of Machine Learning Research, 23(120):1–39, 2022.

[14]
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.
Making the v in vqa matter: Elevating the role of image understanding in visual question answering.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017.

[15]
Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam.
Hard mixtures of experts for large scale weakly supervised vision.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6865–6873, 2017.

[16]
Ishaan Gulrajani and David Lopez-Paz.
In search of lost domain generalization.
arXiv preprint arXiv:2007.01434, 2020.

[17]
Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham.
Vizwiz grand challenge: Answering visual questions from blind people.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617, 2018.

[18]
Drew A Hudson and Christopher D Manning.
Gqa: A new dataset for real-world visual reasoning and compositional question answering.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019.

[19]
Alyssa Hughes.
Phi-2: The surprising power of small language models.

[20]
Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, et al.
Tutel: Adaptive mixture-of-experts at scale.
Proceedings of Machine Learning and Systems, 5, 2023.

[21]
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al.
Mixtral of experts.
arXiv preprint arXiv:2401.04088, 2024.

[22]
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.
Scaling laws for neural language models.
arXiv preprint arXiv:2001.08361, 2020.

[23]
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.
Segment anything.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023.

[24]
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen.
Gshard: Scaling giant models with conditional computation and automatic sharding.
In International Conference on Learning Representations, 2020.

[25]
Bo Li, Yifei Shen, Jingkang Yang, Yezhen Wang, Jiawei Ren, Tong Che, Jun Zhang, and Ziwei Liu.
Sparse mixture-of-experts are domain generalizable learners.
In The Eleventh International Conference on Learning Representations, 2022.

[26]
Bo Li, Yifei Shen, Jingkang Yang, Yezhen Wang, Jiawei Ren, Tong Che, Jun Zhang, and Ziwei Liu.
Sparse mixture-of-experts are domain generalizable learners.
In The Eleventh International Conference on Learning Representations, 2023.

[27]
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales.
Deeper, broader and artier domain generalization.
In Proceedings of the IEEE international conference on computer vision, pages 5542–5550, 2017.

[28]
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.
Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.
In International conference on machine learning, pages 19730–19742. PMLR, 2023.

[29]
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.
Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.
In International conference on machine learning, pages 12888–12900. PMLR, 2022.

[30]
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen.
Evaluating object hallucination in large vision-language models.
arXiv preprint arXiv:2305.10355, 2023.

[31]
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan.
Moe-llava: Mixture of experts for large vision-language models.
arXiv preprint arXiv:2401.15947, 2024.

[32]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning.
Advances in neural information processing systems, 36, 2024.

[33]
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al.
Mmbench: Is your multi-modal model an all-around player?
arXiv preprint arXiv:2307.06281, 2023.

[34]
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan.
Learn to explain: Multimodal reasoning via thought chains for science question answering.
Advances in Neural Information Processing Systems, 35:2507–2521, 2022.

[35]
William Peebles and Saining Xie.
Scalable diffusion models with transformers.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023.

[36]
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.
Moment matching for multi-source domain adaptation.
In Proceedings of the IEEE/CVF international conference on computer vision, pages 1406–1415, 2019.

[37]
Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby.
From sparse to soft mixtures of experts.
arXiv preprint arXiv:2308.00951, 2023.

[38]
Zihan Qiu, Zeyu Huang, and Jie Fu.
Emergent mixture-of-experts: Can dense pre-trained transformers benefit from emergent modular structures?
arXiv preprint arXiv:2310.10908, 2023.

[39]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural language supervision.
In International conference on machine learning, pages 8748–8763. PMLR, 2021.

[40]
Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He.
Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale.
In International conference on machine learning, pages 18332–18346. PMLR, 2022.

[41]
Prajit Ramachandran and Quoc V Le.
Diversity and depth in per-example routing models.
In International Conference on Learning Representations, 2018.

[42]
Xiaozhe Ren, Pingyi Zhou, Xinfan Meng, Xinjing Huang, Yadao Wang, Weichao Wang, Pengfei Li, Xiaoda Zhang, Alexander Podolskiy, Grigory Arshinov, et al.
Pangu-$\{$$\backslash$Sigma$\}$: Towards trillion parameter language model with sparse heterogeneous computing.
arXiv preprint arXiv:2303.10845, 2023.

[43]
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby.
Scaling vision with sparse mixture of experts.
Advances in Neural Information Processing Systems, 34:8583–8595, 2021.

[44]
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean.
Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.
arXiv preprint arXiv:1701.06538, 2017.

[45]
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach.
Towards vqa models that can read.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019.

[46]
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.
Llama: Open and efficient foundation language models.
arXiv preprint arXiv:2302.13971, 2023.

[47]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.
Llama 2: Open foundation and fine-tuned chat models.
arXiv preprint arXiv:2307.09288, 2023.

[48]
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.
Deep hashing network for unsupervised domain adaptation.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5018–5027, 2017.

[49]
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.
Glue: A multi-task benchmark and analysis platform for natural language understanding.
In International Conference on Learning Representations, 2018.

[50]
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman.
Neural network acceptability judgments.
Transactions of the Association for Computational Linguistics, 7:625–641, 2019.

[51]
Xun Wu, Shaohan Huang, Wenhui Wang, and Furu Wei.
Multi-head mixture-of-experts.
arXiv preprint arXiv:2404.15045, 2024.

[52]
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan.
CLUE: A Chinese language understanding evaluation benchmark.
In Proceedings of the 28th International Conference on Computational Linguistics, pages 4762–4772, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics.

[53]
An Yang, Junyang Lin, Rui Men, Chang Zhou, Le Jiang, Xianyan Jia, Ang Wang, Jie Zhang, Jiamang Wang, Yong Li, et al.
M6-t: Exploring sparse expert models and beyond.
arXiv preprint arXiv:2105.15082, 2021.

[54]
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen.
A survey on multimodal large language models.
arXiv preprint arXiv:2306.13549, 2023.

[55]
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang.
Mm-vet: Evaluating large multimodal models for integrated capabilities.
arXiv preprint arXiv:2308.02490, 2023.

[56]
Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou.
Moefication: Transformer feed-forward layers are mixtures of experts.
In Findings of the Association for Computational Linguistics: ACL 2022, pages 877–890, 2022.

[57]
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al.
Mixture-of-experts with expert choice routing.
Advances in Neural Information Processing Systems, 35:7103–7114, 2022.

§ EXPERIMENT SETTINGS

We conduct experiments on Vision, Language, and Vision-Language tasks. The detailed experiment settings are shown in the following.

* Vision Task. For the vision tasks, we follow the same settings as in GMoE [26]. We employ the pre-trained ViT-S/16 [10] model and evaluate it on the DomainBed [16] benchmark.
Our experiments encompass four Domain Generalization datasets: PACS [27], VLCS [2], OfficeHome [48], and DomainNet [36].
All results are reported using the train-validation selection criterion.
We conduct all experiments on a single RTX 3090 GPU, and the reported results are averaged over three random seeds. For , we set the maximum number of experts to 8 and the initial number of experts to 6. The adaptive process is executed for each iteration.

* Language Task. The language tasks adhere to the same settings as those in MoEfication [56] and EMoE [38].
The MoE models are built upon the BERT-large [8] architecture using the MoEfication method and are fine-tuned on GLUE [49] tasks, which include COLA [50], QNLI [49], RTE [5], MNLI [52], and MRPC [9]. We conduct all experiments on a single RTX 3090 GPU, and the reported results are averaged over three random seeds. For , we set the maximum number of experts to 8 and the initial number of experts to 6. For each epoch, we begin recording routing at 1/3 of the epoch and complete recording routing and execute the adaptive process at 2/3 of the epoch.

* Vision-Language Task. The vision-language tasks follows the setting in MoE-LLaVA [31], where we use StableLM-2-1.6B [4], Qwen-1.8B [3] and Phi-2 [19] as backbone language models, and use clip-vit-large-patch14-336 [39] as the vision encoder. We conduct model training on 8 A100 (80G) GPUs, completing within 2 days, detailed hyper-parameters setting are shown in Table <ref>. The models are evaluated on image understanding benchmarks including VQA-v2 [14], GQA [18], VisWiz [17], ScienceQA-IMG [34], TextVQA [45], POPE [30], MME [54], MMBench [33], LLaVA-Bench (in-the-Wild) [32], and MM-Vet [55]. Furthermore, we keep routing records in our model during testing time. For each benchmark, we collect the number of experts' activations per MoE layer and total processed tokens during testing.

Detailed training hyper-parameters and configuration.

2*Config     3cModels
StableLM                                        Qwen                      Phi-2
Maximum experts             3c4
Deepspeed                   Zero2                                           Zero2                     Zero2_offload

Data                        3cLLaVA-Finetuning

Image resolution            3c336 $\times$ 336

Image encoder               3cCLIP-Large/336

Feature select layer        3c-2

Image projector             3cLinear layers with GeLU

Epoch                       3c1

Learning rate               3c2e-5

Learning rate schedule      3cCosine

Weight decay                3c0.0

Batch size per GPU          8                                               8                         4

GPU                         4 $\times$ A100 (80G)                           8 $\times$ A100 (80G)     8 $\times$ A100 (80G)

Precision                   3cBf16

§ ADDITIONAL EXPERIMENTS

In this section, we present the detailed results of our experiments on the GLUE benchmark [49] in Table <ref> and on the DomainNet dataset in Table <ref>. These results demonstrate that incorporating the specially designed diversity and simplicity loss significantly enhances the model's performance.

Performance of on language tasks: Our study investigates the performance of on language tasks using the GLUE [49] benchmark, with BERT-large serving as the backbone model. The baselines including traditional MoE methods with different number of experts $K$ and top-$k$. In our implementation of , we configure the maximum number of experts to 16, with an initial setting of 8 experts. The number of experts is dynamically adjusted in each epoch for .  The $-$ represents experiment failure, final results could not be obtained using Gshard loss.

Algorithms                COLA      MRPC      QNLI      MNLI      RTE       Average

MoE ($K = 8, k = 1$)      64.10     90.14     92.48     86.56     73.04     81.26

MoE ($K = 8, k = 2$)      64.51     90.19     92.39     86.70     74.85     81.73

MoE ($K = 8, k = 4$)      64.94     89.74     92.52     86.57     75.09     81.77

MoE ($K = 8, k = 8$)      64.03     89.36     92.46     86.61     74.37     81.37

MoE ($K = 16, k = 1$)     63.63     89.81     92.39     86.63     74.01     81.29

MoE ($K = 16, k = 2$)     64.71     90.18     92.53     86.73     72.32     81.29

MoE ($K = 16, k = 4$)     64.12     89.74     92.65     86.59     75.33     81.69

MoE ($K = 16, k = 8$)     64.37     90.35     92.49     86.51     73.53     81.45

, Gshard Loss     64.88     89.85     92.42     -         73.41     -

65.17     90.64     92.59     86.37     73.41     81.64

Detailed results on DomainNet dataset: We report the detailed test results on each domain of the DomainNet dataset.

Algorithms                                               clip     info     paint     quick     real     sketch     Average

GMoE (with , Gshard Loss)                        66.8     23.8     54.1      15.9      68.7     54.9       47.4

GMoE (with , Diverse and Simple Gating Loss)     68.0     24.4     55.4      16.6      69.5     55.1       48.2

§ ADDITIONAL VISUALIZATION RESULTS

§.§ Activation Frequency

We present the activation frequency of experts across various MoE layers and evaluation tasks using different backbones: StableLM-1.6B (Figures <ref> and <ref>), Qwen-1.8B (Figures <ref> and <ref>), and Phi-2-2.7B (Figures <ref> and <ref>).
The results suggest that compared to the StableLM-1.6B backbone, experts are more uniformly activated for models utilizing Qwen-1.8B and Phi-2-2.7B as backbone LLMs.

Comparing the performance efficiency of models.
The $x$-axis represents the number of activated parameters, while the $y$-axis shows the performance on the Visual Question Answering (VQA) task.

Activation frequency of experts on various MoE layers and evaluation tasks using StableLM as backbone.

Activation frequency of experts on various MoE layers and evaluation tasks using StableLM as backbone.

Activation frequency of experts on various MoE layers and evaluation tasks using Qwen as backbone.

Activation frequency of experts on various MoE layers and evaluation tasks using Qwen as backbone.

Activation frequency of experts on various MoE layers and evaluation tasks using Phi-2 as backbone.

Activation frequency of experts on various MoE layers and evaluation tasks using Phi-2 as backbone.

§.§ Average Top-$k$

In Figures <ref> and <ref> , we illustrate the average top-$k$ of models using Qwen and Phi-2 as backbone LLMs.

Average top-$k$ activated experts of on vision-language benchmarks, using Qwen as language backbone.

Average top-$k$ activated experts of on vision-language benchmarks, using Phi-2 as language backbone.

§.§ Layer-wise Expert Similarity Matrix

In Figures <ref>, <ref>, and <ref>, we illustrate the similarities between various expert representations, specifically, different rows of $\mW_g$ across multiple MoE layers. These comparisons utilize StableLM-1.6B, Qwen-1.8B, and Phi-2-2.7B as the backbone LLMs. The findings demonstrate that these expert representations are nearly orthogonal, suggesting that different experts capture diverse features, which could potentially enhance the model's capacity.

§.§ Visualization of $\mG$

In Figures <ref>, <ref>, and <ref>, we present the values of the learned threshold $\mG$, employing StableLM-1.6B, Qwen-1.8B, and Phi-2-2.7B as the backbone LLMs. The results reveal that for each MoE layer, there is one expert that is more readily activated. This observation is consistent with the design of Deepseek-MoE [7].

Layer-wise expert similarity matrix (StableLM). We record the experts' cosine similarity per layer during test time. It turns out the cosine similarity between experts is close to 0.

Layer-wise expert activation threshold (StableLM). Darker-colored experts are more likely to be activated compared to lighter-colored experts.

Layer-wise expert similarity matrix (Qwen). We record the experts' cosine similarity per layer during test time. It turns out the cosine similarity between experts is close to 0.

Layer-wise expert activation threshold (Qwen). Darker-colored experts are more likely to be activated compared to lighter-colored experts.

Layer-wise expert similarity matrix (Phi-2). We record the experts' cosine similarity per layer during test time. It turns out the cosine similarity between experts is close to 0.

Layer-wise expert activation threshold (Phi-2). Darker-colored experts are more likely to be activated compared to lighter-colored experts.


# Vertex-primitive $s$-arc-transitive digraphs admitting a Suzuki or Ree group

Lei Chen Michael Giudici Cheryl E. Praeger

Department of Mathematics and Statistics
The University of Western Australia
35 Stirling Highway, Perth WA 6009
Australia

<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

We study $G$-vertex-primitive and $(G,s)$-arc-transitive digraphs for almost
simple groups $G$ with socle ${}^{2}\mathrm{G}_{2}(3^{2n+1})$ or
$\mathrm{Sz}(2^{2n+1})$. It turns out that $s\leq 1$ for such digraphs. We
also construct examples with $s=1$ for each case.

##  1 Introduction

The property of $s$-arc-transitivity has been well-studied for many years.
Weiss [14] proved that finite undirected graphs that are not cycles can be at
most 7-arc-transitive. On the other hand, the third author [12] showed that
for each $s$ there are infinitely many finite $s$-arc-transitive digraphs that
are not $(s+1)$-arc-transitive.

However, vertex-primitive $s$-arc-transitive digraphs for large $s$ seem rare.
Though extensive attempts had been made to find a vertex-primitive $s$-arc-
transitive digraph for $s\geq 2$, no such examples were found until 2017 when
the second author, with Li and Xia, constructed an infinite family of $2$-arc
transitive examples in [5]. In [7], the second author and Xia asked the
following:

######  Question 1.1.

Is there an upper bound on $s$ for vertex-primitive $s$-arc-transitive
digraphs that are not directed cycles?

A group $G$ is said to be an _almost simple group_ if it has a unique _minimal
normal subgroup_ $T$ such that $T$ is a nonabelian simple group. This implies
(identifying $T$ with the group of inner automorphisms of $T$) that
$T\triangleleft G\leqslant\mathrm{Aut}(T)$. If an upper bound on $s$ in
Question 1.1 exists, then let us denote by $C$ the largest value of $s$ for
which there exists a $G$-vertex-primitive $(G,s)$-arc-transitive digraph with
$G$ an almost simple group. In [7], the second author and Xia proved that $C$
is also the least upper bound on $s^{\prime}$ for vertex-primitive
$s^{\prime}$-arc-transitive digraphs that are not directed cycles.

In [11], Pan, Wu and Yin proved that, if $G=$ Sm or $\mathrm{A}_{m}$, then
$s\leq 2$ except for one subcase left open, and in [6], Giudici, Li and Xia
proved that, if $\mathrm{PSL}_{n}(q)\leqslant
G\leqslant\mathrm{Aut}(\mathrm{PSL}_{n}(q))$, then $s\leq 2$. This paper
determines an upper bound $s$ for vertex-primitive $s$-arc-transitive digraphs
whose automorphism groups are almost simple Ree and Suzuki groups. We
juxtapose the Suzuki groups and the Ree groups in this paper as many
similarities can be found between these two kinds of exceptional simple
groups: (1) the Suzuki groups $\mathrm{Sz}(2^{2n+1})$ bear a relation to the
symplectic groups $\mathrm{Sp}_{4}(2^{2n+1})$ similar to that of the Ree
groups ${}^{2}\mathrm{G}_{2}(3^{2n+1})$ to $\mathrm{G}_{2}(3^{2n+1})$; (2) the
maximal subgroup types of the Suzuki groups and the Ree groups are fairly
similar; (3) the only outer automorphisms of the two groups are field
automorphisms. Hence we are able to apply similar arguments to both. Our main
result is as follows.

######  Theorem 1.2.

Let $s$ be a non-negative integer and let $\Gamma$ be a $G$-vertex-primitive
$(G,s)$-arc-transitive digraph, where $G$ is almost simple with socle
${}^{2}\mathrm{G}_{2}(3^{2n+1})$ or $\mathrm{Sz}(2^{2n+1})$. Then $s\leq 1$.

In the next paragraph we remind readers of some terms mentioned above.

A _digraph_ $\Gamma$ is a pair $(V,\to)$ such that $V$ is the set of vertices
and $\to$ is an anti-symmetric and irreflexive relation on $V$. For a non-
negative integer $s$, we call a sequence $v_{0},v_{1},\dots,v_{s}$ in $V$ an
_$s$ -arc_ if $v_{i}\to v_{i+1}$ for each $i\in\\{0,1,\dots,s-1\\}$. Note that
a 1-arc is simply called an _arc_. For $G\leqslant\mathrm{Aut}(\Gamma)$, we
say that $\Gamma$ is a $(G,s)$-arc-transitive digraph if $G$ acts transitively
on the set of $s$-arcs of $\Gamma$. We note that an $(s+1)$-arc-transitive
digraph is naturally $s$-arc-transitive if every $s$-arc extends to an
$(s+1)$-arc. A transitive subgroup $G\leqslant\mathrm{Sym}(\Omega)$ is said to
be primitive if it does not preserve any non-trivial partition of $\Omega$.
For $G\leqslant\mathrm{Aut}(\Gamma)$, we say that $\Gamma$ is _$G$ -vertex-
primitive_ if $G$ acts primitively on $V$. A digraph is said to be _finite_ if
$|V|$ is finite and all the digraphs we consider in this paper will be finite.

##  2 Preliminaries

###  2.1 Notation

We begin by defining some group theoretic notation:

For a group $X$, we denote by $\mathrm{Soc}(X)$ the socle of $X$, and by
$\Pi(X)$ the set of prime divisors of $|X|$. For a prime number $p$ and an
integer $n$, we denote by $n_{p}$ the $p-$part of $n$, which is the largest
power of $p$ dividing $n$.

The expression $n$ or $\mathrm{C}_{n}$ denotes a cyclic group of order $n$
while $[n]$ denotes an unspecified group of order $n$. The expression $p^{n}$
denotes an elementary abelian group of order $p^{n}$, that is, a direct
product of $n$ copies of $\mathrm{C}_{p}$.

Extensions of groups are written in one of the following ways: $A\times B$
denotes a direct product of $A$ and $B$; also $A:B$ denotes a semidirect
product of $A$ by $B$; and $A.B$ denotes an unspecified extension of $A$ by
$B$.

For groups $A$ and $B$ such that $B\leqslant A$, we denote by
$\mathrm{N}_{A}(B)$ the normaliser of $B$ in $A$, and $\mathrm{C}_{A}(B)$ the
centraliser of $B$ in $A$.

######  Lemma 2.1.

[6, Lemma 2.1] For any positive integer $n$ and prime $p$, we have
$(n!)_{p}<p^{\frac{n}{p-1}}$.

######  Definition 2.2.

Given integers $a,m\geq 2$, a prime $r$ is said to be a _primitive prime
divisor_ of $a^{m}-1$ if $r$ divides $a^{m}-1$ and does not divide $a^{i}-1$
for any $i<m$.

For $r$ a primitive prime divisor of $a^{m}-1$, we conclude by Fermat’s Little
Theorem that $r\equiv 1\pmod{m}$, and therefore $r>m$.

######  Lemma 2.3.

[1, Theorem IX.8.3] For $a,m\geq 2$, there exists a primitive prime divisor of
$a^{m}-1$ except when $(a,m)=(2,6)$, or $a+1$ is a power of $2$ and $m=2$.

###  2.2 Group factorisations

A factorisation of a group $G$ is an expression of $G$ as the product of two
subgroups $A$ and $B$ of $G$, where $A$ and $B$ are called factors. A proper
group factorisation occurs when neither $A$ nor $B$ equals $G$.

######  Definition 2.4.

A factorisation $G=AB$ is called a _homogeneous factorisation_ of $G$ if it is
proper and $A$ is isomorphic to $B$.

We now give two technical lemmas, which will be useful later.

######  Lemma 2.5.

Suppose that $G=\langle H,x\rangle$ with $H\triangleleft G$, and let $m$ be
the smallest positive integer such that $x^{m}\in H$. Suppose that $K\leqslant
G$ and $K=AB$ is a homogeneous factorisation such that $B=A^{t}$ for some
$t\in G$, and let $\pi:G\to G/H$ denote the natural projection map. Then
$\pi(A)=\pi(B)=\pi(K)$.

###### Proof.

Note that $G/H=\langle Hx\rangle$ has order $m$ by the minimality of $m$.
Since $\pi(A)=AH/H\leqslant G/H$ and $G/H$ is cyclic, we conclude that
$\pi(A)=\langle Hx^{j}\rangle$ for some divisor $j$ of $m$. So there exists
$a\in A$ such that $\pi(a)=Hx^{j}$. Thus $Ha=Hx^{j}$, so $a=hx^{j}$ for some
$h\in H$.

As $B=A^{t}$ with $t\in G$, we have $a^{t}\in B$ and
$\pi(a^{t})=\pi(t^{-1})\pi(a)\pi(t)=\pi(a)=Hx^{j}$ since $G/H$ is abelian.
Hence $\pi(B)\geqslant\pi(A)$. The same argument with $A$ and $B$ interchanged
and $t$ replaced by $t^{-1}$, gives that $\pi(A)\geqslant\pi(B)$. Hence
$\pi(A)=\pi(B)$, and so $\pi(K)=\pi(A)\pi(B)=\pi(A)=\pi(B)$. ∎

######  Lemma 2.6.

Suppose that $G=AB$ with $G=PSL_{2}(8):3$ such that $A,B$ are proper subgroups
of $G$. Then $|A|\neq|B|$.

###### Proof.

Suppose for a contradiction that there exists a factorisation $G=AB$ with
$G=\mathrm{PSL}_{2}(8):3$ and $|A|=|B|$. Then we deduce that
$|A|_{p}^{2}=|B|_{p}^{2}\geq|G|_{p}$ for any prime $p$. In particular,
$|A|_{2}\geq 2^{2}$, $|A|_{3}\geq 3^{2}$ and $|A|_{7}=7$. On the other hand,
since $A=A\cap\mathrm{PSL}_{2}(8)$ or $A=(A\cap\mathrm{PSL}_{2}(8)).3$. We
therefore conclude that $|A\cap\mathrm{PSL}_{2}(8)|$ is divisible by 2, 3 and
7. By [1, Corollary 5 and Table 10.7] there are no proper subgroups of
$\mathrm{PSL}_{2}(8)$ with order divisible by 2, 3 and 7, and hence
$A\cap\mathrm{PSL}_{2}(8)=\mathrm{PSL}_{2}(8)$. Similarly, we conclude that
$B\cap\mathrm{PSL}_{2}(8)=\mathrm{PSL}_{2}(8)$. However, since $|A|=|B|$, we
must have that $A=B=G$, which contradicts the fact that $A$ and $B$ are proper
subgroups of $G$. ∎

###  2.3 Arc-transitivity

We say that a group $G$ acts on a digraph $\Gamma$ if $G\leq Aut(\Gamma)$.
Here are two results in [6] and [7] that reveal some important properties for
an $s$-arc-transitive digraph $\Gamma$ where $s\geq 2$.

######  Lemma 2.7.

[7, Lemma 2.2] Let $\Gamma$ be a digraph, and $v_{0}\rightarrow
v_{1}\rightarrow v_{2}$ be a $2$-arc of $\Gamma$. Suppose that $G$ acts arc-
transitively on $\Gamma$. Then $G$ acts $2$-arc-transitively on $\Gamma$ if
and only if $G_{v_{1}}=G_{v_{0}v_{1}}G_{v_{1}v_{2}}$. Moreover, there exists
some $t\in G$ such that $(G_{v_{0}v_{1}})^{t}=G_{v_{1}v_{2}}$.

######  Lemma 2.8.

[6, Lemma 2.14] Let $\Gamma$ be a connected $G$-arc-transitive digraph with
arc $v\rightarrow w$. Let $g\in G$ such that $v^{g}=w$. Then $g$ normalises no
proper nontrivial normal subgroup of $G_{v}$.

We now set out the following hypothesis that we will use throughout the paper.

######  Hypothesis 2.9.

Let $\Gamma$ be a vertex-primitive $(G,s)$-arc-transitive digraph for some
$s\geq 2$, and let $u\to v\to w$ be a $2$-arc and $g\in G$ such that
$(u,v)^{g}=(v,w)$. Then by Lemma 2.7, $(G_{uv})^{g}=G_{vw}$ and
$G_{v}=G_{uv}G_{vw}$ is a homogeneous factorisation.

Note that necessary conditions for a digraph $\Gamma$ to be $G$-vertex-
primitive and $(G,2)$-arc-transitive are that $G_{v}$ is a maximal core-free
subgroup of $G$ and that $G_{v}$ admits a homogeneous factorisation.
Therefore, to disprove the 2-arc-transitivity of $G$ it suffices for us to
show that a maximal core-free subgroup $G_{v}$ does not have a homogeneous
factorisation.

We have the following corollary to Lemma 2.8.

######  Corollary 2.10.

Suppose that Hypothesis 2.9 holds. Then, for each prime $p$ dividing
$|G_{v}|$, $G_{v}$ has at least two subgroups of order $p$.

###### Proof.

Suppose for a contradiction that there exists a prime $p$ such that $G_{v}$
has a unique subgroup $Q$ of order $p$. Then $Q$ is normal in $G_{v}$. By
Hypothesis 2.9, $G_{v}=AB$ where $A=G_{uv}$ and $B=G_{vw}$, so $|G_{v}|$
divides $|A|\cdot|B|=|A|^{2}$ and hence $|A|_{p}=|B|_{p}\geq p$. Hence both of
$A$ and $B$ contain the unique subgroup $Q$ of order $p$, and as $A^{g}=B$, we
have $Q^{g}\leqslant B\leqslant G_{v}$ and therefore $Q^{g}=Q$, contradicting
Lemma 2.8. ∎

##  3 The small Ree groups

Suppose that $\Gamma$ is a $G$-vertex-primitive, $(G,s)$-arc-transitive
digraph such that $\mathrm{Soc}(G)={}^{2}\mathrm{G}_{2}(q)$ with $q=3^{2n+1}$,
for some $n\geq 1$ and $s\geq 1$. Since the action of $G$ on $\Gamma$ is
vertex-primitive, the vertex stabiliser $G_{v}$ is maximal in $G$ and does not
contain ${}^{2}\mathrm{G}_{2}(q)$. The following list of the maximal subgroups
of ${}^{2}\mathrm{G}_{2}(q)$ may be found in [16].

######  Theorem 3.1.

[16, Theorem 4.2] If $q=3^{2n+1}$ with $n\geq 1$, then the maximal subgroups
of ${}^{2}\mathrm{G}_{2}(q)$ are (up to conjugacy):
(i) $[q^{3}]:\mathrm{C}_{q-1}$,
(ii) $2\times\mathrm{PSL}_{2}(q)$,
(iii) $(2^{2}\times\mathrm{D}_{\frac{q+1}{2}}):3$,
(iv) $\mathrm{C}_{q-\sqrt{3q}+1}:6$,
(v) $\mathrm{C}_{q+\sqrt{3q}+1}:6$,
(vi) ${}^{2}\mathrm{G}_{2}(q_{0})$, where $q=q_{0}^{r}$ and $r$ is prime.

Since $\mathrm{Aut}({}^{2}\mathrm{G}_{2}(q))={}^{2}\mathrm{G}_{2}(q):(2n+1)$,
and ${}^{2}\mathrm{G}_{2}(3^{2n+1})\leq
G\leq\mathrm{Aut}({}^{2}\mathrm{G}_{2}(q))$, we have
$G={}^{2}\mathrm{G}_{2}(q):m$, for some divisor $m$ of $2n+1$, and a vertex-
stabiliser $G_{v}$ is maximal in $G$ and does not contain
${}^{2}\mathrm{G}_{2}(q)$. The subgroups of $G$ with these properties are the
following:

######  Corollary 3.2.

For $G={}^{2}G_{2}(3^{2n+1}):m$, where $m$ divides $2n+1$, the maximal
subgroups of $G$ not containing ${}^{2}G_{2}(3^{2n+1})$ are (up to conjugacy):
(i) $([q^{3}]:\mathrm{C}_{q-1}):m$,
(ii) $(2\times\mathrm{PSL}_{2}(q)):m$,
(iii) $((2^{2}\times\mathrm{D}_{\frac{q+1}{2}}):3).m$,
(iv) $(\mathrm{C}_{q-\sqrt{3q}+1}:6).m$,
(v) $(\mathrm{C}_{q+\sqrt{3q}+1}:6).m$,
(vi) ${}^{2}\mathrm{G}_{2}(q_{0}):m$, where $q=q_{0}^{r}$ and $r$ is prime.

For the rest of this section we assume that $s\geq 2$, and hence Hypothesis
2.9 holds for $G={}^{2}G_{2}(3^{2n+1}):m$, where $q=3^{2n+1}>3$ and $m$
divides $2n+1$, and we let $L=\mathrm{Soc}(G)={}^{2}\mathrm{G}_{2}(q)$. We
consider separately each of the possibilities for the maximal subgroup $G_{v}$
according to Corollary 3.2, and in each case derive a contradiction, hence
proving that $s\leq 1$.

We let $\pi$ be the natural projection map $\pi:G_{v}\rightarrow G_{v}/L_{v}$.
Note that since $G=LG_{v}$ we have $\pi(G_{v})\cong G/L\cong C_{m}$. We note
in particular that, by Hypothesis 2.9, $G_{v}$ has a homogeneous factorisation

$G_{v}=AB$ where $A=G_{uv}$ and $B=G_{vw}$ with $A^{g}=B$ for some $g\in G$.
(1)

This implies, first that $\Pi(A)=\Pi(B)=\Pi(G_{v})$, and secondly, by
Corollary 2.10, that for each prime $p$ dividing $|G_{v}|$, $G_{v}$ has at
least two subgroups of order $p$. We use these facts several times in our
arguments.

######  Lemma 3.3.

$G_{v}$ is not a Type (ii) subgroup of $G$.

###### Proof.

Suppose to the contrary that $G_{v}$ is a Type (ii) subgroup of $G$, and
consider the homogeneous factorisation $G_{v}=AB$ in (1), so
$\Pi(A)=\Pi(B)=\Pi(G_{v})$. Let $S$ and $T$ denote the subgroups of
$L_{v}=L\cap G_{v}$ isomorphic to 2 and $\mathrm{PSL}_{2}(q)$, respectively.
Then $L_{v}=S\times T$.

Note that, by Lemma 2.3, there exists $p\in\Pi(G_{v})$ such that $p$ is a
primitive prime divisor of $3^{2(2n+1)}-1$, which is greater than $2(2n+1)$.
Hence $|A|$ is divisible by $p$ and, in particular, $p$ divides the order of
$A_{1}:=A\cap T=A\cap\mathrm{PSL}_{2}(q)$. We also notice that

$\frac{|A||B|}{|A\cap B|}=|AB|=|G_{v}|$

and therefore,
$|A|_{3}^{2}\geq|G_{v}|_{3}=|\mathrm{PSL}_{2}(q)|_{3}m_{3}=3^{2n+1}m_{3}$.
Thus $|A|_{3}\geq 3^{\frac{2n+1}{2}}m_{3}^{\frac{1}{2}}$. However,
$m_{3}\leq(2n+1)_{3}\leq 3^{\frac{2n+1}{3}}\leq 3^{n}$, and
$|A|_{3}=|A_{1}|_{3}|\pi(A)|_{3}\leq|A_{1}|_{3}m_{3}$. Hence

$\displaystyle|A_{1}|_{3}$
$\displaystyle\geq\frac{|A|_{3}}{m_{3}}\geq\frac{3^{\frac{2n+1}{2}}m_{3}^{\frac{1}{2}}}{m_{3}}=3^{\frac{2n+1}{2}}m_{3}^{-\frac{1}{2}}\geq
3^{\frac{2n+1}{2}}3^{-\frac{n}{2}}=3^{\frac{n+1}{2}}>1.$

Thus $\\{3,p\\}\subseteq\Pi(A_{1})$. By [10, Theorem 4 and Table 10.3], there
are no proper subgroups of $\mathrm{PSL}_{2}(q)$ with order divisible by both
3 and $p$, and hence $A_{1}=\mathrm{PSL}_{2}(q)=T$. On the other hand, since
$A^{g}=B$, we have $T^{g}\leqslant A^{g}=B$. However, $T$ is the unique
subgroup in $G_{v}$ isomorphic to $\mathrm{PSL}_{2}(q)$, so this implies that
$T^{g}=T$, which is a contradiction to Lemma 2.8. ∎

######  Lemma 3.4.

$G_{v}$ is not a Type (iii) subgroup of $G$.

###### Proof.

Suppose for a contradiction that $G_{v}$ is a Type (iii) subgroup of $G$, and
again consider the homogeneous factorisation $G_{v}=AB$ in (1) which implies
that, for each $p$ dividing $|G_{v}|$, $G_{v}$ has more than one subgroup of
order $p$. We denote by $S$ and $T$ the normal subgroups of $L_{v}=L\cap
G_{v}$ isomorphic to $2^{2}$ and $\mathrm{D}_{\frac{q+1}{2}}$, respectively,
so that $L_{v}=(S\times T):3$.

By Lemma 2.3 there exists a primitive prime divisor $p$ of
$3^{2(2n+1)}-1=q^{2}-1$. Note that $p\neq 3$, and also $p$ divides $q+1$, and
$p$ is odd (as $p$ does not divide $q-1$). Hence
$p\in\Pi(\mathrm{D}_{\frac{q+1}{2}})\subseteq\Pi(G_{v})$. Since $p>2(2n+1)\geq
2m$ and $p\neq 3$, any subgroup $Q$ of $G_{v}$ of order $p$ must lie in $T$.
Since $T=\mathrm{D}_{\frac{q+1}{2}}$ is dihedral, this implies that $Q$ is the
unique subgroup of order $p$ in $T$ and hence in $G_{v}$. However, this
contradicts Corollary 2.10 and therefore the result follows. ∎

######  Lemma 3.5.

$G_{v}$ is neither a Type (iv) subgroup nor a Type (v) subgroup of $G$.

###### Proof.

Suppose for a contradiction that $G_{v}$ is a Type (iv) or (v) subgroup of
$G$. Recall, as discussed above, that for each prime $p$ dividing $|G_{v}|$,
$G_{v}$ has more than one subgroup of order $p$. We denote by $S$ and $T$ the
(unique) cyclic subgroups of $L_{v}=L\cap G_{v}$ of order $q\pm\sqrt{3q}+1$
and 6, respectively, so that $L_{v}=S:T$. Since $q\pm\sqrt{3q}+1$ is not
divisible by 2 or 3, we see that $|S|$ and $|T|$ are coprime.

By Lemma 2.5 we have that $\pi(A)=\pi(B)=\pi(G_{v})=\mathrm{C}_{m}$. Thus
$|A\cap L_{v}|=|B\cap L_{v}|$. Let $p$ be a prime dividing $|S|$. Then there
is a unique subgroup $Q_{p}\leqslant S$ of order $p$. We note that since $|S|$
and $|T|$ are coprime, $Q_{p}$ is the unique subgroup in $L_{v}$ of order $p$.

######  Claim 1.

$|A\cap L_{v}|_{p}=1$.

Suppose for a contradiction that $|A\cap L_{v}|_{p}\geq p$. Then $A\cap L_{p}$
has a subgroup of order $p$. This subgroup must be $Q_{p}$ as it is the unique
subgroup of order $p$ in $L_{v}$. On the other hand, since $|A\cap
L_{v}|=|B\cap L_{v}|$, we find that $Q_{p}\leqslant B\cap L_{v}$ as well. We
note that $A^{g}=B$, so $Q_{p}^{g}\leqslant(A\cap L_{v})^{g}=B\cap
L_{w}\leqslant B\cap L\leqslant G_{v}\cap L=L_{v}$. This implies that
$Q_{p}^{g}=Q_{p}$. However, this contradicts Lemma 2.8 and therefore Claim 1
holds.

By Claim 1 we conclude that $|A\cap L_{v}|\leq 6$. This implies that
$|A|=|A\cap L_{v}||\pi(A)|\leq 6m$. Suppose first that $n=1$. Then $m\leq 3$,
$q=3^{3}$ and $|A|\leq 18$. Thus $|G_{v}|$ is either divisible by
$37=q+\sqrt{3q}+1$ or by $19=q-\sqrt{3q}+1$. However, $|A|$ is divisible by
neither 37 nor 19 since $|A|\leq 18$. Hence $G_{v}$ does not have a
homogeneous factorisation when $n=1$. Thus $n\geq 2$ and so

$q-\sqrt{3q}+1=3^{2n+1}-3^{n+1}+1\geq 9(2n+1).$

Since $G_{v}=AB$, we have that $|A|\cdot|B|=|G_{v}|\cdot|A\cap B|$. However,

$\displaystyle|G_{v}|\cdot|A\cap B|$ $\displaystyle\geq(q\pm\sqrt{3q}+1)\cdot
6m\geq 9(2n+1)\cdot 6m>(6m)^{2}\geq|A|\cdot|B|.$

So we have a contradiction and the result follows. ∎

######  Lemma 3.6.

$G_{v}$ is not a Type (vi) subgroup of $G$.

###### Proof.

Suppose for a contradiction that $G_{v}$ is a Type (vi) subgroup of $G$, so
$G_{v}=H.m$, where $H=L_{v}=L\cap G_{v}={}^{2}\mathrm{G}_{2}(q_{0})$, with
$3^{2n+1}=q=q_{0}^{r}$ for some prime $r$, such that $r,m$ both divide $2n+1$.
Now $G_{v}$ has a homogeneous factorisation $G_{v}=AB$ where $A=G_{uv}$ and
$B=G_{vw}$ with $B=A^{g}$ for some $g\in G$. Let $X:=A\cap L_{v}$ and
$Y:=B\cap L_{v}$. It follows from Lemma 2.5 that
$\pi(A)=\pi(B)=\mathrm{C}_{m}$. We divide the analysis into two cases:

_Case $1$: $2n+1$ is not prime._ In this case $q_{0}=3^{(2n+1)/r}>3$. Let $C$
be the centraliser of $H={}^{2}\mathrm{G}_{2}(q_{0})$ in $G_{v}$. Then
$\mathrm{Aut}(H)\geq G_{v}/C=(AB)/C=(AC/C)(BC/C)\geq
HC/C\cong{}^{2}\mathrm{G}_{2}(q_{0})$. All the core-free factorisations of an
almost simple group with socle an exceptional group of Lie type are given in
[16, Theorem B], and it follows that $G_{v}/C$ does not have a core-free
factorisation since $q_{0}>3$. Hence ${}^{2}\mathrm{G}_{2}(q_{0})$ is
contained in one of $AC/C$ or $BC/C$. Without loss of generality, we may
assume that ${}^{2}\mathrm{G}_{2}(q_{0})\leq AC/C$. This together with the
fact that $H\cap C=1$ implies that $H\leqslant A$. On the other hand,
$H^{g}\leqslant A^{g}=B$, and since $H$ is the unique subgroup of $G_{v}$
isomorphic to ${}^{2}\mathrm{G}_{2}(q_{0})$, we conclude that $H^{g}=H$, which
contradicts Lemma 2.8.

_Case $2$: $2n+1$ is prime._ In this case, $m\in\\{1,2n+1\\}$ and $r=2n+1$, so
$q_{0}=3$ and $H=L_{v}=H^{\prime}:3$ with $H^{\prime}\cong L_{2}(8)$. If
$m=1$, then $AB$ is a homogeneous factorisation for
$H={}^{2}\mathrm{G}_{2}(3)=\mathrm{PSL}_{2}(8):3$, but no such factorisation
exists by Lemma 2.6. Hence $m=2n+1$ and we have
$\Pi(A)=\Pi(G_{v})=\Pi(H)\cup\\{m\\}=\\{2,3,7,m\\}$ (with possibly
$m\in\\{3,7\\}$).

Recall that $X=A\cap H=A\cap L_{v}$. Suppose that $\Pi(X\cap
H^{\prime})=\\{2,3,7\\}$. By [1, Corollary 5 and Table 10.7] there are no
proper subgroups of $\mathrm{PSL}_{2}(8)$ with order divisible by 2, 3 and 7,
and hence $X\cap H^{\prime}=H^{\prime}\cong\mathrm{PSL}_{2}(8)$ so
$H^{\prime}\leqslant A$. It follows that $(H^{\prime})^{g}\leqslant A^{g}=B$,
and since $H^{\prime}$ is the unique subgroup of $G_{v}$ isomorphic to
$\mathrm{PSL}_{2}(8)$, we conclude that $(H^{\prime})^{g}=H^{\prime}$,
contradicting Lemma 2.8. Thus $\Pi(X\cap H^{\prime})$ is a proper subset of
$\\{2,3,7\\}$.

Further, since $|G_{v}:H^{\prime}|=3m$ is odd, we have $|X\cap
H^{\prime}|_{2}=|A|_{2}\geq|G_{v}|_{2}^{1/2}=2^{3/2}$, and hence $|X\cap
H^{\prime}|_{2}\geq 2^{2}$, so $2\in\Pi(X\cap H^{\prime})$. If $\Pi(X\cap
H^{\prime})=\\{2\\}$ then, since $|A|/|X|$ divides $3m$, it follows that
$\Pi(A)\subseteq\\{2,3,m\\}$ and since $\Pi(A)=\\{2,3,7,m\\}$ we conclude that
$m=7$ and $3=|A|_{3}<3^{3/2}=|G_{v}|_{3}^{1/2}$, which is a contradiction.
Therefore $\Pi(X\cap H^{\prime})=\\{2,p\\}$ for some $p\in\\{3,7\\}$. If $p=3$
then $X\cap H^{\prime}$ is a subgroup of $H^{\prime}=L_{2}(8)$ of order
divisible by 12 and dividing $72$. However there are no such subgroups, see
for example [3, p. 6]. Therefore $\Pi(X\cap H^{\prime})=\\{2,7\\}$, and $X\cap
H^{\prime}$ is a subgroup of $H^{\prime}=L_{2}(8)$ of order divisible by 28.
It follows from [3, p. 6] that $X\cap H^{\prime}=[2^{3}]:7$ since this group
has no subgroups of index 2. The same argument gives $Y\cap
H^{\prime}\cong[2^{3}]:7$.

If the prime $m\neq 3$, then $|X|_{3}=|A|_{3}\geq|G_{v}|_{3}^{1/2}=3^{3/2}$ so
that $|X\cap H^{\prime}|_{3}\geq|X|_{3}/3\geq 3$, which is a contradiction.
Hence $m=3$. In this case, $G_{v}=L_{v}\times\langle
z\rangle\cong(\mathrm{PSL}_{2}(8):3)\times 3$, where $z$ is a field
automorphism of order 3. Since $AB=G_{v}=L_{v}\times\langle z\rangle$, we have
that $\mathrm{PSL}_{2}(8):3=L_{v}\cong G_{v}/\langle z\rangle=((A\langle
z\rangle/\langle z\rangle)(B\langle z\rangle/\langle z\rangle)$. Now $A\langle
z\rangle/\langle z\rangle$ has a normal subgroup $(X\cap H^{\prime})\langle
z\rangle/\langle z\rangle\cong[2^{3}]:7$, and similarly $B\langle
z\rangle/\langle z\rangle$ has a normal subgroup $[2^{3}]:7$. However, by [9,
Theorem A], there are no such factorisations of $\mathrm{PSL}_{2}(8):3$. This
completes the proof. ∎

######  Theorem 3.7.

Suppose that $\Gamma$ is a $G$-vertex-primitive $(G,s)$-arc-transitive digraph
such that $\mathrm{Soc}(G)={}^{2}\mathrm{G}_{2}(q)$ with $q=3^{2n+1}$, for
some $n\geq 1$. Then $s\leq 1$.

###### Proof.

Suppose for a contradiction that $s\geq 2$. Then the conditions of Hypothesis
2.9 hold with $\mathrm{Soc}(G)={}^{2}\mathrm{G}_{2}(q)$. Since $G$ acts
vertex-primitively on $\Gamma$, the vertex stabiliser $G_{v}$ is a maximal
subgroup of $G$ and so is given by Corollary 3.2. By Lemmas 3.3, 3.4, 3.5 and
3.6, $G_{v}$ cannot be of types (ii)–(vi). Hence $G_{v}$ is of type $(i)$.
However, in this case, $G$ acts 2-transitively on the set of right cosets of
$G_{v}$ in $G$, which implies that $\Gamma$ is an undirected graph,
contradicting it being a digraph. Hence the result follows. ∎

##  4 Suzuki Groups

Again, since the action of $G$ on $\Gamma$ is vertex-primitive, a vertex
stabiliser $G_{v}$ is maximal in $G$. The following list of the maximal
subgroups of $\mathrm{Sz}(q)$ may be found in the book [2].

######  Theorem 4.1.

[2, p 385] If $q=2^{2n+1}$ with $n\geq 1$, then the maximal subgroups of
$\mathrm{Sz}(q)$ are (up to conjugacy):
(i) $[q^{2}]:(q-1)$,
(ii) $\mathrm{D}_{2(q-1)}$,
(iii) $\mathrm{C}_{q+\sqrt{2q}+1}:4$,
(iv) $\mathrm{C}_{q-\sqrt{2q}+1}:4$,
(v) $\mathrm{Sz}(q_{0})$, where $q=q_{0}^{r}$, $r$ is prime and $q_{0}>2$.

Since $\mathrm{Aut}(\mathrm{Sz}(q))=\mathrm{Sz}(q):(2n+1)$ and
$\mathrm{Sz}(q)\leq G\leq\mathrm{Sz}(q):(2n+1)$, we have $G=\mathrm{Sz}(q):m$
for some divisor $m$ of $2n+1$, and a vertex stabiliser does not contain
$\mathrm{Sz}(q)$. The maximal such subgroups of $G$ are the following:

######  Corollary 4.2.

For $G=\mathrm{Sz}(q):m$, where $m$ divides $2n+1$, the maximal subgroups of
$G$ not containing $\mathrm{Sz}(q)$ are (up to conjugacy):
(i) $([q^{2}]:(q-1)).m$,
(ii) $\mathrm{D}_{2(q-1)}.m$,
(iii) $((q+\sqrt{2q}+1):4).m$,
(iv) $((q-\sqrt{2q}+1):4).m$,
(v) $\mathrm{Sz}(q_{0}).m$, where $q=q_{0}^{r}$, $r$ is prime, and $q_{0}>2$.

For the rest of this section we assume that Hypothesis 2.9 holds for
$G=\mathrm{Sz}(q):m$, where $q=2^{2n+1}$ and $m$ divides $2n+1$, and we let
$L=\mathrm{Soc}(G)=\mathrm{Sz}(q)$. Let $\pi:G_{v}\rightarrow G_{v}/L_{v}$ be
the natural projection map. Note that since $G=LG_{v}$ we have that
$\pi(G_{v})\cong G/L\cong C_{m}$.

We consider separately each of the possibilities for the maximal subgroup
$G_{v}$ according to Corollary 4.2. We note in particular that, by Hypothesis
2.9, $G_{v}$ has a homogeneous factorisation $G_{v}=AB$ where $A=G_{uv}$ and
$B=G_{vw}$ with $A^{g}=B$ for some $g\in G$. This implies, by Corollary 2.10,
that for each prime $p$ dividing $|G_{v}|$, $G_{v}$ has at least two subgroups
of order $p$. We use these facts several times in our arguments.

######  Lemma 4.3.

Suppose that Hypothesis 2.9 holds with $\mathrm{Soc}(G)=\mathrm{Sz}(q)$. Then
$G_{v}$ is not a Type (ii) subgroup of $G$.

###### Proof.

Suppose to the contrary that $G_{v}$ is a Type (ii) subgroup of $G$. Then
$L_{v}\cong D_{2(q-1)}$. By Lemma 2.3 there exists a primitive prime divisor
$p$ of $2^{2n+1}-1$, and as noted before Lemma 2.3, $p$ satisfies $p>2n+1\geq
m$. Hence $G_{v}$ has a unique subgroup $Q_{p}$ of order $p$, which is a
contradiction, as noted above. ∎

######  Lemma 4.4.

Suppose that Hypothesis 2.9 holds with $\mathrm{Soc}(G)=\mathrm{Sz}(q)$. Then
$G_{v}$ is neither a Type (iii) subgroup nor a Type (iv) subgroup of $G$.

###### Proof.

Suppose to the contrary that $G_{v}$ is a Type (iii) or Type (iv) subgroup of
$G$. As above we have $G_{v}=AB$ with $A^{g}=B$ for some $g\in G$. It follows
from Lemma 2.5 that $\pi(A)=\pi(B)=\pi(G_{v})\cong\mathrm{C}_{m}$, and hence
$|A\cap L_{v}|=|B\cap L_{v}|$.

Let $S$ and $T$ denote cyclic subgroups of $L_{v}=L\cap G_{v}$ of orders
$q\pm\sqrt{2q}+1$ and 4, respectively, such that $L_{v}=S:T$. Since
$q\pm\sqrt{2q}+1$ is an odd integer, the orders $|S|$ and $|T|$ are coprime.

Let $p$ be a prime dividing $|S|$, and note that the cyclic group $S$ has a
unique subgroup $Q_{p}$ of order $p$, and that $Q_{p}$ is the unique subgroup
of order $p$ in $L_{v}$. If $p$ divides $|A\cap L_{v}|$, then $A\cap L_{v}$
contains $Q_{p}$, and since $|A\cap L_{v}|=|B\cap L_{v}|$, also
$Q_{p}\leqslant B\cap L_{v}$. Moreover, since $A^{g}=B$, it follows that
$Q_{p}^{g}$ is also a subgroup of $B\cap L_{v}$ of order $p$, and so
$Q_{p}^{g}=Q_{p}$. However, this contradicts Lemma 2.8, and therefore $p$ does
not divide $|A\cap L_{v}|$. Since this holds for all primes $p$ dividing
$|S|$, we conclude that $|A\cap L_{v}|$ divides $|T|=4$.

Since $A/(A\cap L_{v})\cong AL_{v}/L_{v}\leq G_{v}/L_{v}\cong\mathrm{C}_{m}$,
it follows that $|A|$ divides $4m$, and hence $G_{v}=AB$ has order dividing
$|A|\cdot|B|=|A|^{2}$, which divides $16m^{2}$. On the other hand
$|G_{v}|=4m(q\pm\sqrt{2q}+1)$, and hence the odd integer $q\pm\sqrt{2q}+1$
divides $m$. This is impossible since $q\pm\sqrt{2q}+1\geq
2^{2n+1}-2^{n+1}+1>2n+1\geq m$, for all $n\geq 1$. This contradiction
completes the proof. ∎

######  Lemma 4.5.

Suppose that Hypothesis 2.9 holds with $\mathrm{Soc}(G)=\mathrm{Sz}(q)$. Then
$G_{v}$ is not a Type (v) subgroup of $G$.

###### Proof.

Suppose to the contrary that $G_{v}=\mathrm{Sz}(q_{0}).m$, where
$q=q_{0}^{r}$, for some prime $r$ dividing $2n+1$. As above we have $G_{v}=AB$
with $A^{g}=B$ for some $g\in G$. Let $C$ denote the centraliser of
$\mathrm{Sz}(q_{0})$ in $G_{v}$. Then

$\mathrm{Aut}(\mathrm{Sz}(q_{0}))\gtrsim
G_{v}/C=(AB)/C=(AC/C)(BC/C)\gtrsim\mathrm{Sz}(q_{0}).$

However, $G_{v}/C$ does not have a core-free factorisation by [10, Theorem B],
and therefore one of the factors, say $AC/C$, contains $\mathrm{Sz}(q_{0})$.
This, together with the fact that $L_{v}\cap C=1$, implies that
$\mathrm{Sz}(q_{0})=L_{v}\leqslant A$. Moreover, since $A^{g}=B$, we have
$L_{v}^{g}\leqslant A^{g}=B$. However $L_{v}$ is the only subgroup of $G_{v}$
isomorphic to $Sz(q_{0})$, and hence $L_{v}^{g}=L_{v}$. This contradicts Lemma
2.8, and completes the proof. ∎

Now we can collect all these results to prove the following result.

######  Theorem 4.6.

Suppose that $\Gamma$ is a $G$-vertex-primitive $(G,s)$-arc-transitive digraph
such that $\mathrm{Soc}(G)=\mathrm{Sz}(q)$ with $q=3^{2n+1}$ for some positive
integer $n$. Then $s\leq 1$.

###### Proof.

Suppose for a contradiction that $s\geq 2$. Then the conditions of Hypothesis
2.9 hold with $\mathrm{Soc}(G)={}^{2}\mathrm{Sz}(q)$. Since $G$ acts vertex-
primitively on $\Gamma$, the vertex stabiliser $G_{v}$ is a maximal subgroup
of $G$ and so is given by Corollary 4.2. By Lemmas 4.3, 4.4 and 4.5, $G_{v}$
is not of type $(ii)--(v)$ and so $G_{v}$ must be of type (i). However, in
this case $G$ acts 2-transitively on the set of right cosets of $G_{v}$ and so
$\Gamma$ is an undirected graph, contradicting it being a digraph. Hence the
result follows. ∎

Theorem 1.2 follows immediately from Theorems 3.7 and 4.6.

##  5 Examples

In this final section, we construct examples of vertex-primitive $(G,1)$-arc-
transitive digraphs with $\mathrm{Soc}(G)=\mathrm{Sz}(2^{2n+1})$ and
${}^{2}\mathrm{G}_{2}(3^{2n+1})$, respectively. The following is the mechanism
by which we construct the examples:

Let $G$ be a group, $H$ a subgroup of $G$ which does not contain
$\mathrm{Soc}(G)$, $V:=\\{Hz:z\in G$}, and let $g\in G$ such that
$g^{-1}\notin HgH$. We define a binary relation $\to\,$ on $V$ by

$Hx\to Hy$ if and only if $yx^{-1}\in HgH$ for any $x,y\in G$.

Then $(V,\to)$ is a digraph, which we denote by $\mathrm{Cos}(G,H,g)$. Since
$yg(xg)^{-1}=ygg^{-1}x=yx^{-1}$, right multiplication by elements of $G$
preserves the relation $\to\,$ and hence induces automorphisms of $(V,\to)$,
yielding a subgroup $\mathrm{R}_{H}(G)\cong G$ of
$\mathrm{Aut}(\mathrm{Cos}(G,H,g))$. Further the subgroup $\mathrm{R}_{H}(H)$
of right multplications by elements of $H$ is the stabiliser in
$\mathrm{R}_{H}(G)$ of the vertex $H$ of $(V,\to)$, and it follows from the
definition of the relation $\to\,$ that $\mathrm{R}_{H}(H)$ acts transitively
on the set of arcs $(H,Hx)$ beginning with $H$, since these arcs are precisely
those of the form $(H,Hgh)$ for $h\in H$. Thus $\mathrm{R}_{H}(G)$ acts arc-
transitively on $\mathrm{Cos}(G,H,g)$.

We aim to find a maximal subgroup $H\leqslant G$ and an element $g\in G$ such
that $g^{-1}\notin HgH$ to obtain a 1-arc-transitive digraph
$\mathrm{Cos}(G,H,g)$, for $G=\mathrm{Sz}(2^{2n+1})$ and
$G={}^{2}\mathrm{G}_{2}(3^{n+1})$.

###  5.1 A one-arc transitive digraph admitting a Suzuki group

Here $G=\mathrm{Sz}(q)$, where $q=2^{2n+1}$ with $n\geq 1$. Let $\Omega$
denote a set of size $q^{2}+1$ on which $G$ acts $2$-transitively. Let
$a,b\in\Omega$ with $a\neq b$. We define the following notation:

  * (i)

$L:=G_{a}$, so $L\cong[q^{2}]:(q-1)$;

  * (ii)

$K:=G_{a}\cap G_{b}$, so $K=\langle\kappa\rangle\cong q-1$, for some
$\kappa\in G$;

  * (iii)

$Q=[q^{2}]$, the normal Sylow $2$-subgroup of $L$, so $L=Q\rtimes K$;

  * (iv)

an involution $\tau\in\mathrm{N}_{G}(K)$, so
$\mathrm{N}_{G}(K)=\langle\kappa,\tau\,|\,\kappa^{q-1}=\tau^{2}=1,\tau\kappa\tau=\kappa^{-1}\rangle$,
see Theorem 4.1(ii);

  * (v)

an element $\rho\in Q$ of order 4.

We use this notation throughout this subsection, and also the following
results.

######  Lemma 5.1.

[13, Proposition 1] For any non-trivial element $x\in Q$, the centraliser
$\mathrm{C}_{G}(x)\leq Q$.

######  Lemma 5.2.

[13, p 108-109] The element $\tau$ satisfies $b=a^{\tau}$ and $a=b^{\tau}$, so
$\tau L\tau=G_{b}$, $L\cap\tau L\tau=K$, and $Q\cap\tau L\tau=\\{1\\}$.

We now give our construction.

######  Lemma 5.3.

Let $H:=\mathrm{N}_{G}(K)$ and $g:=\rho$. Then $g^{-1}\notin HgH$, and
$Cos(G,H,g)$ is a $(G,1)$-arc-transitive digraph.

###### Proof.

As we explained above, if $g^{-1}\notin HgH$, then $Cos(G,H,g)$ is a
$(G,1)$-arc-transitive digraph. So it is sufficient to prove that
$g^{-1}\notin HgH$. Suppose that this is not the case, that is, there exist
$x,y\in H=N_{G}(K)$ such that $\rho^{-1}=x\rho y$.

Note that $L=Q\rtimes K$ and $L\cap N_{G}(K)=K$, and also that $\rho\in Q<L$.
Thus if $x\in K$ then $y=\rho^{-1}x^{-1}\rho\in L$ and hence $y\in L\cap
N_{G}(K)=K$. Similarly if $y\in K$ then also $x\in K$. Thus $x,y$ are either
both in $K$, or both in $N_{G}(K)\setminus K$.

Suppose first that $x,y\in K$. Now $\rho\in Q$, and since $x\in K<L$ and $Q$
is a normal subgroup of $L$, it follows that $x\rho x^{-1}\in Q$, and also
$(x\rho x^{-1})(xy)=x\rho y=\rho^{-1}\in Q$. This implies that $xy\in Q$ and
hence $xy\in K\cap Q=\\{1\\}$. Thus $y=x^{-1}$, and so $\rho^{-1}=x\rho
x^{-1}$, which implies that $x^{2}\in C_{G}(\rho)$. By Lemma 5.1,
$\mathrm{C}_{G}(\rho)\leqslant Q$, so $x^{2}\in K\cap Q=\\{1\\}$. However,
$x\in K$ and $|K|=q-1$ is odd. Hence $x=1$, so $\rho^{-1}=x\rho x^{-1}=\rho$,
which contradicts the fact that $\rho$ has order $4$. Thus we must have
$x,y\in N_{G}(K)\setminus K$, and hence $x=\kappa^{i}\tau$ and
$y=\kappa^{j}\tau$, for some $i,j$. This implies that $\rho^{-1}=x\rho
y=\tau(\kappa^{-i}\rho\kappa^{j})\tau\in\tau L\tau$, and we also have
$\rho^{-1}\in Q$. Thus $\rho^{-1}\in Q\cap\tau L\tau$ and so, by Lemma 5.2,
$\rho^{-1}=1$, which is a contradiction. This completes the proof. ∎

###  5.2 A one-arc transitive digraph admitting a Ree group

Here $G={}^{2}\mathrm{G}_{2}(q)$, where $q=3^{2n+1}$ with $n\geq 1$. Although
several $(G,2)$-arc-transitive undirected graphs have been constructed, see
[4], we are interested in constructing a $(G,1)$-arc-transitive digraph with
$G$ acting primitively on the vertex set. Our treatment follows Wilson’s
description of the group $G$ given in his book [16, Section 4.5]. It is
different from some other constructions for these groups, say in [8], which
require knowledge of Lie algebras and algebraic groups. Wilson has an
elementary approach developed in [17] and [15], and we use the detailed
description given in [16, p 134-138].

Wilson [16] starts with a faithful $7$-dimensional representation of the group
$G={}^{2}\mathrm{G}_{2}(3^{2n+1})$ on a space $V$ over a field
$\mathbf{F}_{q}$ of order $q$. The space $V$ admits a $G$-invariant non-
degenerate symmetric bilinear form $f$ with an orthonormal basis
$\mathcal{B}:=\\{u_{0},u_{1},\ldots,u_{6}\\}$. He defines a second basis
$\mathcal{C}:=\\{v_{1},v_{2},\ldots,v_{7}\\}$ for $V$ by

$\begin{array}[]{llp{1cm}ll}v_{1}&=u_{3}+u_{5}+u_{6},&&v_{2}&=u_{1}+u_{2}+u_{4},\\\
v_{3}&=-u_{0}-u_{3}+u_{6},&&v_{4}&=u_{2}-u_{1},\\\
v_{5}&=-u_{0}+u_{3}-u_{6},&&v_{6}&=-u_{1}-u_{2}+u_{4},\\\
v_{7}&=-u_{3}+u_{5}-u_{6}.\end{array}$

He shows that the maps $\gamma$ and $\sigma$ given by:

$u_{i}^{\gamma}=\begin{cases}u_{i},&i=0,1,3\\\
-u_{i},&i=2,4,5,6\par\end{cases}$

and

$u_{i}^{\sigma}=\begin{cases}u_{i},&i=0,4,5\\\ -u_{i},&i=1,2,3,6\end{cases}$

are commuting involutions lying in the group $G$. Thus $G$ contains the
subgroup
$K=\langle\gamma,\sigma\rangle=\langle\gamma\rangle\times\langle\sigma\rangle\cong
2^{2}$. Moreover, we let $\delta:=\sigma\gamma$, find that

$u_{i}^{\delta}=\begin{cases}u_{i},&i=0,2,6\\\ -u_{i},&i=1,3,4,5\end{cases}$

Let $T:=N_{G}(K)$ and $H=C_{G}(\sigma)$. Note that by [4, Lemma 2.2], $T$ and
$H$ are maximal subgroups of $G$ and $T\cong(2^{2}\times D_{\frac{q+1}{2}}):3$
and $H\cong 2\times\mathrm{PSL}_{2}(q)$. Let us denote by $W$, $W_{1}$,
$W_{2}$ and $W_{3}$ the subspace $\langle u_{1},\ldots,u_{6}\rangle$, $\langle
u_{1},u_{3}\rangle$, $\langle u_{2},u_{6}\rangle$ and $\langle
u_{4},u_{5}\rangle$, respectively. For a subspace $U$ of $V$, we denote the
setwise stabiliser in $G$ of $U$ by $\mathrm{Stab}_{G}(U)$. We need the
following properties of $T$.

######  Lemma 5.4.

The subgroup $T=\mathrm{Stab}_{G}(\langle u_{0}\rangle)$, and $T$ leaves
invariant the subspace $W:=\langle u_{1},u_{2},\ldots,u_{6}\rangle$.

###### Proof.

Let $\mathrm{Fix}(K)$ be the subspace of fixed points of $K$ in $V$. Then
$\mathrm{Fix}(K)=\langle u_{0}\rangle$, by the definitions of $\gamma$ and
$\sigma$, and since $\mathcal{B}$ is an orthonormal basis it follows that
$\mathrm{Fix}(K)^{\perp}=W$. Since $T=N_{G}(K)$ normalises $K$, it follows
that $T$ leaves invariant both $\mathrm{Fix}(K)$ and
$\mathrm{Fix}(K)^{\perp}$. Finally since $T$ is maximal in $G$, it follows
that $T$ is equal to the full stabiliser of $\mathrm{Fix}(K)$. ∎

For any $x\in\\{\delta,\gamma,\sigma\\}$, let $C_{W}(x)=\\{u\in W|u^{x}=u\\}$.

######  Lemma 5.5.

With above notation, $T$ permutes $W_{1}$, $W_{2}$ and $W_{3}$. In particular,
the action of $T$ on $\\{W_{1},W_{2},W_{3}\\}$ is isomorphic to
$\mathrm{C}_{3}$

###### Proof.

Since $T$ normalises $K=\langle\sigma,\gamma\rangle$, we see that $T$ permutes
$\sigma,\gamma$ and $\delta$. We also note that $C_{W}(\gamma)=W_{1}$,
$C_{W}(\delta)=W_{2}$ and $C_{W}(\sigma)=W_{3}$. For $i\in\\{1,2,3\\}$,
$W_{i}=C_{W}(x)$ for some $x\in\\{\sigma,\gamma,\delta\\}$. Since $T$ leaves
$W$ invariant we have that $C_{W}(x)^{t}=C_{W}(x^{t})$ for each $x$. Thus
$W_{i}^{t}=C_{W}(x)^{t}=C_{W}(x^{t})$. Since
$x^{t}\in\\{\sigma,\gamma,\delta\\}$, we find that
$W_{i}^{t}\in\\{W_{1},W_{2},W_{3}\\}$. Moreover, we find that $C_{G}(K)\cong
2^{2}\times D_{\frac{q+1}{2}}$, so $[T:C_{G}(K)]=3$. We note that $C_{G}(K)$
acts trivially on $\\{W_{1},W_{2},W_{3}\\}$. This implies the kernel of the
action of $T$ on $\\{W_{1},W_{2},W_{3}\\}$ is of index 3, so the action is
isomorphic to $C_{3}$. Hence the result follows. ∎

In Wilson’s description [16, p 136], $G$ has a Borel subgroup $B$ such that
there exists $g\in B$ determined by its action on the basis $\mathcal{C}$ as
follows:

$\begin{array}[]{llp{1cm}ll}v_{1}&\mapsto v_{1}&&v_{2}&\mapsto v_{2}\\\
v_{3}&\mapsto v_{1}+v_{3}&&v_{4}&\mapsto v_{2}+v_{4}\\\ v_{5}&\mapsto
2v_{1}+v_{5}&&v_{6}&\mapsto v_{2}+2v_{4}+v_{6}\\\ v_{7}&\mapsto
v_{1}+v_{3}+2v_{5}+v_{7}.\end{array}$

It is easily checked that $g$ acts on $u_{i}$ by:

$\begin{array}[]{llp{1cm}ll}u_{0}&\mapsto u_{0}&&u_{1}&\mapsto u_{2}\\\
u_{2}&\mapsto u_{4}&&u_{3}&\mapsto u_{6}\\\ u_{4}&\mapsto u_{1}&&u_{5}&\mapsto
u_{3}\\\ u_{6}&\mapsto u_{5}.\end{array}$

In particular we find that $g\in T=N_{G}(K)$ and $W_{i}^{g}=W_{i+1}$ for all
$i\in\\{W_{1},W_{2},W_{3}\\}$.

######  Lemma 5.6.

Let $H$ and $g$ be as above. Then $g^{-1}\notin HgH$, and $Cos(G,H,g)$ is a
$(G,1)$-arc-transitive digraph.

###### Proof.

Let $g^{\prime}=g^{-1}$. If $g^{\prime}\notin HgH$, then $Cos(G,H,g)$ is a
$(G,1)$-arc-transitive digraph. So it is sufficient to prove that
$g^{\prime}\notin HgH$. Suppose that this is not the case. that is, there
exist $x,y\in H=C_{G}(\sigma)$ such that $g^{\prime}=x^{-1}gy$, or
equivalently, $xg=g^{\prime}y$. Let us denote by $E_{1}$ and $E_{-1}$ the
eigenspaces of $\sigma$ with eigenvalues 1 and $-1$ respectively. Indeed,
$E_{1}=\langle u_{0},u_{4},u_{5}\rangle$ and $E_{-1}=\langle
u_{1},u_{2},u_{3},u_{6}\rangle$.

######  Claim 2.

We have $u_{0}^{x},u_{0}^{y}\in\langle u_{0}\rangle$.

First we note that since $x,y\in H$, $x\sigma=\sigma x$ and $y\sigma=\sigma
y$. This implies that $u_{0}^{x\sigma}=u_{0}^{\sigma x}=u_{0}^{x}$. Thus
$u_{0}^{x}\in E_{1}$, so that
$u_{0}^{x}=\alpha_{0}u_{0}+\alpha_{4}u_{4}+\alpha_{5}u_{5}$ for some
$\alpha_{i}\in\mathbb{F}_{q}$. Similarly
$u_{0}^{y}=\beta_{0}u_{0}+\beta_{4}u_{4}+\beta_{5}u_{5}$ for some
$\beta_{i}\in\mathbb{F}_{q}$.
Now, we let $I=\\{0,4,5\\}$ and since $xg=g^{\prime}y$, we have:

$\sum_{i\in
I}\beta_{i}u_{i}=u_{0}^{y}=u_{0}^{g^{\prime}y}=u_{0}^{xg}=\sum_{i\in
I}\alpha_{i}u_{i}^{g}=\beta_{0}u_{0}+\beta_{4}u_{1}+\beta_{5}u_{3}.$ (2)

This implies that $\alpha_{0}=\beta_{0}$ and
$\alpha_{4}=\alpha_{5}=\beta_{4}=\beta_{5}=0$. Thus the claim is proved.
Let us consider the action of $x$ and $y$ on $W_{3}$. For any $v\in
W_{3}=\langle u_{4},u_{5}\rangle$, we have that $v^{x\sigma}=v^{\sigma
x}=v^{x}$ and $v^{y\sigma}=v^{\sigma y}=v^{y}$. Thus $v^{x},v^{y}\in
E_{1}=\langle u_{0},u_{4},u_{5}\rangle$. On the other hand, by Claim 2 we find
that $x,y\in T$. By Lemma 5.5 $W_{3}^{x},W_{3}^{y}\in\\{W_{1},W_{2},W_{3}\\}$.
Hence $v^{x},v^{y}\in E_{1}\cap W_{i}$ for some $i\in\\{1,2,3\\}$. Since the
only possible $i$ such that $E_{1}\cap W_{i}\neq\\{0\\}$ is 3, we deduce that
$W_{3}^{x}=W_{3}^{y}=W_{3}$. Hence $x$ and $y$ either swap or fix $W_{1}$ and
$W_{2}$. If $x$ and $y$ swap them, then $x,y$ acts as $(1,2)$ on
$\\{W_{1},W_{2},W_{3}\\}$. However, by Lemma 5.5, the action of $T$ on
$\\{W_{1},W_{2},W_{3}\\}$ is isomorphic to $\mathrm{C}_{3}$ and does not have
any element of order 2. Thus we deduce that $W_{i}^{x}=W_{i}^{y}=W_{i}$ for
$i=1,2$.
Now let us consider the action of $xg$ and $g^{\prime}y$ on $W_{1}$,

$W_{1}^{xg}=W_{1}^{g}=W_{2}$ (3)

while

$W_{1}^{g^{\prime}y}=W_{3}^{y}=W_{3}.$ (4)

This is a contradiction to $xg=g^{\prime}y$. Hence such $x$ and $y$ do not
exist and the result follows. ∎

## References

  * [1] N. Blackburn and B. Huppert, Finite groups II, Springer-Verlag, Berlin-New York, 1982.
  * [2] J. N. Bray, D. F. Holt and C. M. Roney-Dougal, The maximal subgroups of the low-dimensional finite classical groups, Cambridge University Press, Cambridge, 2013.
  * [3] J. H. Conway, R. T. Curtis, S. P. Norton, R. A. Parker, and R. A. Wilson, _Atlas of Finite Groups_ , Clarendon Press, Oxford, 1985.
  * [4] X. G. Fang and C. E. Praeger, Finite two-arc transitive graphs admitting a ree simple group, Comm. Algebra, 27 (1999) 3755–3769 .
  * [5] M. Giudici, C.H. Li and B. Xia, An infinite family of vertex-primitive 2-arc-transitive digraphs, J. Combin. Theory Ser. B, 127 (2017) 1–13.
  * [6] M. Giudici, C.H. Li and B. Xia, Vertex-primitive s-arc-transitive digraphs of linear groups, J. Math. Pure Appl. 223 (2019) 5455–5483.
  * [7] M. Giudici and B. Xia, Vertex-quasiprimitive 2-arc-transitive digraphs, Ars Math. Contempt., 14 (2018) 67–82.
  * [8] V. M. Levchuk and Y. N. Nuzhin, The structure of Ree groups, Alg. i Log., 24 (1985) 26–41,122.
  * [9] M. W. Liebeck, C. E. Praeger and J. Saxl, The maximal factorizations of the finite simple groups and their automorphism groups, Mem. Amer. Math. Soc., 86 (1990), no.432.
  * [10] M. W. Liebeck, C. E. Praeger and J. Saxl, Transitive subgroups of primitive groups, J. Algebra 234 (2000) 291–361.
  * [11] J. Pan, C. Wu, F. Yin, Vertex-primitive s-arc-transitive digraphs of alternating and symmetric groups, J. Algebra, 544 (2020) 75–91.
  * [12] C. E. Praeger, Highly arc-transitive digraphs, European J. Combin. 10 (1989) 281–292.
  * [13] M. Suzuki, On a class of doubly transitive groups, Ann. of Math. 75 (1962) 105–145.
  * [14] R. Weiss, The non-existence of 8-transitive graphs, Combinatorica 1 (1981) 309–311.
  * [15] R. A. Wilson, A new construction of the ree groups of type ${}^{2}G_{2}$, Proc. Edinburgh Math. Soc., 53 (2010) 531–542.
  * [16] R. A. Wilson, The Finite simple groups, Grad. Texts in Math., vol. 251, Springer-Verlag, London, 2009.
  * [17] R. A. Wilson, Another new approach to the small Ree groups, Arch. Math. 94 (2010) 501–510.