#  UNO Arena for Evaluating Sequential Decision-Making Capability of Large
Language Models

Zhanyue Qin1🖂, Haochuan Wang1, Deyuan Liu1, Ziyang Song1, Cunhang Fan2, Zhao
Lv2,
Jinlin Wu3, Zhen Lei3, Zhiying Tu1, Dianhui Chu1, Xiaoyan Yu3, Dianbo Sui1🖂
1 Harbin Institute of Technology
2 Anhui University
3 Institute of Automation, Chinese Academy of Sciences
<EMAIL_ADDRESS><EMAIL_ADDRESS>Dianbo Sui is the corresponding
author.

###### Abstract

Sequential decision-making refers to algorithms that take into account the
dynamics of the environment, where early decisions affect subsequent
decisions. With large language models (LLMs) demonstrating powerful capability
between tasks, we can’t help but ask: Can Current LLMs Effectively Make
Sequential Decisions? In order to answer this question, we propose the UNO
Arena based on the card game UNO to evaluate the sequential decision-making
capability of LLMs and explain in detail why we choose UNO. In UNO Arena, We
evaluate the sequential decision-making capability of LLMs dynamically with
novel metrics based Monte Carlo methods. We set up random players, DQN-based
reinforcement learning players, and LLM players (e.g. GPT-4, Gemini-pro) for
comparison testing. Furthermore, in order to improve the sequential decision-
making capability of LLMs, we propose the TuTri player, which can involves
having LLMs reflect their own actions wtih the summary of game history and the
game strategy. Numerous experiments demonstrate that the TuTri player achieves
a notable breakthrough in the performance of sequential decision-making
compared to the vanilla LLM player.

UNO Arena for Evaluating Sequential Decision-Making Capability of Large
Language Models

Zhanyue Qin1🖂, Haochuan Wang1, Deyuan Liu1, Ziyang Song1, Cunhang Fan2, Zhao
Lv2, Jinlin Wu3, Zhen Lei3, Zhiying Tu1, Dianhui Chu1, Xiaoyan Yu3, Dianbo
Sui1🖂 ††thanks: Dianbo Sui is the corresponding author. 1 Harbin Institute of
Technology 2 Anhui University 3 Institute of Automation, Chinese Academy of
Sciences<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

In artificial intelligence, sequential decision-making refers to algorithms
that take the dynamics of the world into consideration Frankish and Ramsey
(2014), and it can be described as a procedural approach to decision-making,
or as a step by step decision theory. Sequential decision-making has as a
consequence the intertemporal choice problem, where earlier decisions
influences the later available choices Amir (2014).

In recent years, Large language models (LLMs) are gaining increasing
popularity in both academia and industry, owing to their unprecedented
performances in various applications Chang et al. (2023), ranging from
chatbots to medical diagnoses Wang et al. (2023a) to robotics He et al.
(2022). From robots handle complex tasks Amiri et al. (2020) to
entrepreneurial action McMullen (2015), sequential decision-making permeates
diverse domains. Hence, an interesting question arises: Can Current LLMs
Effectively Make Sequential Decisions?

To answer this question, we need to design a benckmark to evaluate the
sequential decision-making ability of LLMs. However, evaluating LLMs’
abilities is not trivial. Many works have been proposed to test LLMs’
performances on either a large-scale static benchmark such as MMLU Hendrycks
et al. (2021), or with A/B tests judged by humans (Ganguli et al., 2023). One
common and evident limitation of these methods, however, is that the
environment for LLMs to be tested is static (Aiyappa et al., 2023; Zhou et
al., 2023), which can not reflect the domino effect in sequential decision-
making. Besides, data contamination Sainz et al. (2023); Zeng et al. (2024);
Xu et al. (2024), which means the inclusion of test data examples and labels
in the pre-training data, also challenges the efficacy of these static
benchmarks in differentiating model capabilities.

Figure 1: In this figure, (A) demonstrates the sequential decision-making
process in UNO Arena, (B) shows the execution process of the vanilla LLM
player, and (C) shows the execution process of the TuTri player. In fact, The
Module (D) and the Module (E) are completely identical.

Unlike static evaluation, dynamic evaluation by treating LLMs as game-playing
agents attracted more and more attention of researchers recently, such as
beauty contests and private-value second price auctions Guo et al. (2024a),
Warewolf Xu et al. (2023), Avalon Wang et al. (2023b); Light et al. (2023),
Leduc Hold’em Guo et al. (2023). However, current attempts do not account for
sequential decision-making, and these games are either challenging to evaluate
for intermediate results (such as Werewolf) or have too few decision points
per round (such as Leduc Hold’em). Meanwhile, we should also note that studies
of dynamically evaluating sequential decision-making capability in
reinforcement learning, such as games like Go Silver et al. (2017), Dou Di Zhu
You et al. (2019), and Mahjong Li et al. (2020). However, these games present
an excessively large action space. For instance, in Dou Di Zhu, players can
use any combination of their cards each round, posing significant challenges
for current LLMs Zhai et al. (2024).

Considering the above aspects, we make the following efforts in this paper:

First, we build UNO Arena to dynamically evaluate the sequential decision-
making capability of current LLMs. In UNO arena, we allow LLMs to participate
as players in the UNO game111https://en.wikipedia.org/wiki/Uno_(card_game),
aiming to play all the cards in their hand as quickly as possible. Compared to
games like Leduc Hold’em, which have fewer moves per game, UNO features an
average of dozens of moves per game, making it an ideal testbed for sequential
decision-making Pfann (2021). Additionally, unlike common games in
reinforcement learning, legal actions in the UNO Arena are limited, only
including drawing cards, playing cards, selecting colors to convert, and
choosing whether to challenge the wild draw four card. Furthermore, to monitor
the behaviours of LLMs in the UNO arena, we propose some real-time
quantitative evaluation metrics by leveraging the Monte Carlo method Kroese et
al. (2014) as the reference, which provide a window to observe the
intermediate results and various phenomenons (like domino effect) in LLMs’
sequential decision-making.

Second, based on the proposed UNO Arena, we set up a family of strong and
representative players. In detail, we first build the random player, which
makes decisions based on chance rather than a specific, consistent plan,
without considering the game’s current state or potential outcomes. Then, we
implement the reinforcement learning based player, which leverages DQN Mnih et
al. (2013) to develop sophisticated strategies for playing UNO. Finally, to
probing the capability of LLMs in sequential decision-making, we provides the
task description and then prompt LLMs, like GPT-4 Achiam et al. (2023) and
Gemini-pro Team et al. (2023), to generate their reasoning steps that lead to
the final action.

Third, to unleash the fully potential capability of LLMs in sequential
decision-making, we propose the TuTri player with reflection mechanism Shinn
et al. (2024), which can involves having LLMs analyze their own actions wtih
the game history and the game strategy. In detail, the proposed agent
framework consists of two key reflection modules: the game history reflection
module and the game strategy reflection module. In the game history reflection
module, we provide the statistical data of game history and then prompt the
LLMs to rethink their decision, which simulates the process of card
memorization by humans when playing UNO. In the game strategy reflection
module, LLMs further take into account the game strategy, like saving wild
draw four, and proceed to make the final decision, which simulates the use and
adherence to strategies by humans when playing UNO.

In the experiment, we comprehensively evaluate some mainstream LLMs’ ability
of sequential decision-making, including GPT-3.5 OpenAI (2022), GPT-4 Achiam
et al. (2023), Gemini-pro Team et al. (2023), Llama 2 Touvron et al. (2023),
and ChatGLM3 Du et al. (2021); Zeng et al. (2022). Our experiments show that
among these LLMs, GPT-4 is the most effective sequential decision-maker.

In summary, our contributions are as follows:

  * •

We propose a dynamic evaluation method named UNO Arena for assessing the
sequential decision-making capability of large language models (LLMs) based on
the card game UNO. This method supports the evaluation of 2-10 LLM players,
reinforcement learning players, or random players engaged in a single UNO
game.

  * •

We introduce multiple unique evaluation metrics based on the Monte Carlo
method for evaluating the sequential decision-making capabilities of players
in UNO Arena.

  * •

To improve the sequential decision-making capabilities of LLMs and enhance
their performance in the highly dynamic and complex UNO game, we have
developed the TuTri player and compared it horizontally with the vanilla LLM
player.

##  2 Related Work

Evaluate LLMs Dynamically with Game: LLMs has presented increasingly emerging
ability on game-playing Brookins and DeBacker (2023); Akata et al. (2023) in
recent development and iterations. Wang et al. (2023b) use the Avalon, which
contains elements of deception, to evaluate the capability of LLMs to
recognize and handle deceptive information. Gong et al. (2023) leverage the
CuisineWorld and Minecraft to assess the planning and emergency cooperation
capabilities of LLMs. Guo et al. (2024a) employ beauty contests and auction
games to evaluate the rationality, strategic reasoning capability, and
adherence to instructions of LLMs. Xu et al. (2023) use the game Werewolf to
evaluate the capability of LLMs to infer player roles. Despite evaluating with
game becoming a popular trend, exploring into sequential decision-making
capability is still of scarcity in current works.
Development of Agent Framework: LLM agents have been perceived as a promising
way to realizing Artificial General Intelligence(AGI) Xi et al. (2023) and
recently have shown emergent abilities to execute various tasks in complex
environment Wei et al. (2022). SiLLM Guo et al. (2024b) merges large language
models with synchronous machine translation, using policy decision agents and
translation agents. LLM-Vectorizer Taneja et al. (2024) uses multiple agents
to generate vectorized code by leveraging large language models and test-based
feedback. We tailored a special framework for UNO, featuring self-refinement
and iterative thinking.
Sequential Decision-Making Capability: Sequential decision-making refers to
the process of making a series of decisions over time, where each decision may
impact future choices and outcomes Amir (2014). Though certain algorithms or
reinforce learning provide solutions for some sequential decision-making
problems Littman (1996), LLM-based sequential decision-making are only
employed in limited field like recommendation Wang et al. (2023c). In our
work, we utilized UNO, which is not an easy one even for human Demaine et al.
(2014), to explore the sequential decision-making ability of LLMs. With
certain methods like integrating past experiences and expert advice or
demonstrations Chen et al. (2023), we made efforts to maximally leverage the
decision making ability as possible in a sequential manner.

##  3 The UNO Arena

In this section, we first provide a brief overview of the version of UNO we
adopt in the subsection §3.1. Then, we present the four different types of
players in the UNO arena in the subsection §3.2. Next, we detail how to use
Monte Carlo methods to determine whether a player has made an optimal decision
in subsection §3.3. In the end, we introduce our evaluation metrics in
subsection §3.4.

###  3.1 The UNO Game

We select the UNO as the foundation within our arena due to its widespread
popularity, simplicity and mathematical value. There are various versions of
the UNO game. In this section, we briefly introduce the rules of the version
we adopt in this work.

UNO Cards: A deck of UNO cards comprises a total of 108 cards. UNO cards are
divided into three types: number cards, function cards, and wild cards. A
number card is composed of a color (Red, Blue, Yellow and Green) and a number
(ranging from 0 to 9). A function card is composed of a color (Red, Blue,
Yellow and Green) and a function (Skip, Reverse, Draw Two). The wild cards has
no color and is only composed of Wild cards and Wild Draw Four cards. The
effects of the function cards and wild cards are shown in the Table 1.

UNO Process: First, deal each player 7 initial cards in clockwise order, then
continue drawing cards until a number card is drawn and set as the top card of
the initial discard pile. All players take rounds playing cards in clockwise
order(it will be reversed by a reverse card) until a player runs out of his
cards or the draw pile is exhausted, signaling the end of the game.

UNO Action: From the beginning to the end of the game, players continuously
take actions in UNO. In our work, UNO includes the following types of actions:

  * •

Select Card: When a player comes his playing round, they need to play a card
that matches the color, number, or function of the top card in the discard
pile, or play a Wild card. If they don’t have a card to play, they must draw
one card.

  * •

Select Color: After a player plays a Wild card or a Wild Draw Four card, they
need to change the color of the current top card to one of Red, Yellow, Blue
or Green.

  * •

Select ChallengeFlag: After a player’s previous opponent plays a Wild Draw
Four card, the player needs to decide whether to challenge the legality of the
previous opponent’s Wild Draw Four card.

For more details about the UNO games, please refer to Appendix A. The Figure 1
(A) shows the workflow diagrams of UNO Arena.

###  3.2 Players in the UNO Arena

In the UNO Arena, we initially involve three types of players: random player,
reinforcement learning based player, vanilla LLM player. To further unleash
the potential capability of LLMs in sequential decision-making, we propose
TuTri player, which involves reflection mechanism.

Card |  Sample |  Effect
---|---|---
Skip |  |  The next player in sequence misses a round.
Reverse |  |  Order of play switches directions (clockwise to counterclockwise, or vice versa).
Draw Two |  |  The next player in sequence draws two cards and misses a round.
Wild |  |  Player declares the next color to be matched (it can be used on any round even if the player has any card of matching color).
Wild Draw Four |  |  Player declares the next color to be matched. The next player in sequence draws four cards and misses a round. May be legally played if the player has cards of the current color.
Table 1: The effects of function and wild cards.

Random Player: As like its name suggests, the random player performs all
actions randomly, such as randomly selecting a regulative card to play when
it’s their turn. The random player can be considered the baseline of the UNO
Arena, mainly serving to maintain the flow of the UNO game. If some players
outperform the random player, we can infer that these players are consciously
playing UNO with an understanding of the game rules.

Reinforcement Learning Based Player: Previous research has sought
breakthroughs in UNO using reinforcement learning models Pfann (2021). We
built our reinforcement learning player with DQN Mnih et al. (2013) model
based on the open-source project RLcard Zha et al. (2019).

Vanilla LLM Player: During the vanilla LLM player’s turn, the game host
transmits all publicly available information through a prompt to the LLM. The
LLM then returns a JSON containing the decision and reasoning as required by
the prompt. The Figure 1 (B) shows the workflow diagrams of vanilla LLM
player.

TuTri Player: While LLMs do not always generate the best output on their first
try just as human Madaan et al. (2023), iterative feedback and refinement
could be a necessity for a better agent framework. Moreover, human-like
thinking patterns, such as introspective reflections foster divergent thinking
processes Zhang et al. (2023), inspires us to propose the TuTri player. This
advanced framework is designed to navigate the intricacies of UNO game play,
offering a more structured approach to strategic sequential decision-making.
The original decision for TuTri player is exactly the same as the vanilla LLM
player’s decision, after that are two additional reflection modules.

  * •

The Game History Reflection Module: In the module, we provide statistical
information about game history to TuTri player, and they are told to reflect
the action you just selected with these auxiliary information. Just like human
thinking when playing UNO, if there is a large number of green cards that have
already been played in the game’s history, it is very advantageous for players
to play a green card. LLM should output both reflection thoughts and updated
action.

  * •

The Game Strategy Reflection Module: In the module, we provide additional
useful game strategies to TuTri players, and they are again told to reflect
the action you just selected based on game strategies. For example, since wild
cards can be played at any situations and disrupt other players, saving the
wild cards in your hand as long as possible is a very useful game strategy.
LLM should output both reflection thoughts and updated action (the final
action).

It must be emphasized that the TuTri player should work in a conversational
manner, with exactly 3 times Q&A per round. Moveover, the TuTri players may
keep their original decision, in other words, literally updating the action is
not a necessity, nevertheless, the reflection process, instead of simple I-O
prompting of interaction, providing more opportunities for mistake correcting
and divergent thinking. The Figure 1(C) shows the workflow diagrams of TuTri
player.

###  3.3 Monte Carlo Simulation Method for Monitoring Players’ Behavior

In the game play, the change in each player’s winning rate after making a
decision is the key for tracking. In the classical combinatorial games, like
Nim Bouton (1901) or Wythoff’s Game Wythoff (1907), positions space are
limited and thus computationally affordable, while UNO is more intricate,
where positional space exponentially increases as cards number increases and
the calculation gets tougher Demaine et al. (2014).

To make a plausible ranking mechanism of the candidate decisions, we define
the concept of optimal decision, meaning the state transferred by the decision
from last state, has a highest winning rate concerning all subsequent
outcomes, and thus adopt Monte Carlo Simulation Mooney (1997) to calculate the
estimated winning rate.

Detailedly, with $S_{i}$ representing the state of the game after the $i$-th
step taken, $D_{i,j}$ representing the $j$-th legal decision candidates at
state $S_{i}$, $\mathcal{T}$ representing the state transfer function,
$\mathcal{E}$ representing the estimate function of state, thereby we have the
definition of the optimal decision $D_{i,opt}$ at the $i$-th step where

$opt=\mathop{\arg\max}_{j}\mathcal{E}(\mathcal{T}(S_{i-1},D_{i,j}))$ (1)

In calculation of $\mathcal{E}(S_{i})$, we massively randomly generate the
subsequent decision sequence $\\{D_{i+1},D_{i+2},\dots\\}$ and thus obtain the
subsequent state sequence $\\{S_{i+1},S_{i+2},\dots\\}$. Then
$\mathcal{E}(S_{i})$ is assigned to the ratio of number of sequences where the
player plays the state $S_{i-1}$ comes as the winner, to the total number of
sequences simulated.

As the times we simulate the subsequent sequence increases, the approximate
value $\mathcal{E}(S_{i})$ gets more precise, though we could not enumerate
all the possible situations. To balance the time expenditure and the precision
of the metrics, we control the simulation times in a certain range.
Additionally, a threshold parameter $p$ is set to identify critical decisions.
We say a decision $D:=D_{i}$ is critical if among its all candidate choices
$D_{j}$

$\max\mathcal{E}(\mathcal{T}(S,D_{j}))-\min\mathcal{E}(\mathcal{T}(S,D_{j}))\geq
p$ (2)

Actual decisions made on critical positions may have a huge effect on the
winning rate, which is consistent with the game nature.

###  3.4 Evaluation Metrics in the UNO Arena

In our work, we design three evaluation metrics, including WR, ODHR@K and
ADR@K, in conjunction with the UNO game to comprehensively evaluate the
sequential decision-making capability of LLMs. Among these metrics, ODHR@K and
ADR@K can off a glimpse into the intermediate results in the sequential
decision-making of LLMs.
Winning Rate (WR). WR denotes the proportion of player wins to total game
innings, and can be represented as:

$\text{WR}=\frac{N_{\text{Win}}}{N_{\text{{Game}}}}$ (3)

where $N_{\text{Win}}$ represents the total number of times the given player
has won, and $N_{\text{{Game}}}$ denotes the total game innings.
Optimal Decision Hit Rate at K Decision Points (ODHR@K) : This metric measure
the proportion of times players make the best decision to all decision times,
when facing K decision points:

$\text{ODHR@K}=\frac{\text{$N_{\text{Hit@K}}$}}{\text{$N_{\text{Decision@K}}$}}$
(4)

where $N_{\text{Hit@K}}$ is the number of times the player makes the optimal
decision when facing K optional decision points, and $N_{\text{Decision@K}}$
represents the total number of times the agent player makes decision when it
faces K optional decision points.
Average Decision Rank at K Decision Points (ADR@K). This metric looks at the
rank of output decision made by the player, and can be denoted as:

$\text{ADR@K}=\frac{\sum_{i=1}^{\text{$N_{\text{Decision@K}}$}}\text{Rank}(D_{i})}{\text{$N_{\text{Decision@K}}$}}$
(5)

where $\text{Rank}(D_{i})$ represents the rank from best to worst among all
legal decisions in its decision-making process, and $N_{\text{Decision@K}}$
represents the total number of times the agent player makes decision when it
faces K optional decision points.

For metrics ODHR@K and ADR@K, according to the characteristics of UNO, we only
focus on the situations where $K$ is equal to $2$, $3$ or $4$, because the
vast majority of decisions in UNO do not exceed 4 Pfann (2021).

##  4 Experiments

In this section, we first conduct preliminary experiments with vanilla LLM
players, RL players, and random players in subsection §4.1. Then, we have
multiple different LLM-based vanilla players compete in UNO Arena to identify
the best LLM in subsection §4.2. Next, we test the superiority of the TuTri
players compared to the vanilla LLM players in subsection §4.3. Finally, we
perform ablation experiments on the TuTri player in subsection §4.4.

To ensure the generalization of the experiments, we take the mainstream LLMs
mentioned in the introduction: (1) gpt-3.5-turbo-16k-0613 OpenAI (2022); (2)
gpt-4-1106-preview Achiam et al. (2023); (3) Gemini-pro Team et al. (2023);
(4) Llama-2-7b-chat Touvron et al. (2023); (5) ChatGLM3-6b Du et al. (2021);
Zeng et al. (2022).

###  4.1 1v1 UNO Arena between vanilla LLM players, RL players and random
players

In order to verify the rationality of using UNO Arena to evaluate the
sequential decision-making ability of LLMs, we first conduct experiments on
vanilla LLM players, RL players and random players in 1V1 UNO Arena. We
randomly generate 500 sets of UNO initial decks. Each vanilla LLM player or RL
player have to play with the random player in these 500 initial decks. In
addition, the random players are the first to play cards in all games. The
results are shown in Table 2.

Metrics | Vanilla LLM Players & RL Player with DNQ
---|---
GPT-3.5 | GPT-4 | Gemini-Pro | Llama 2 | ChatGLM3 | DNQ
WR ($\uparrow$) | 55.80 | 63.20 | 53.80 | 53.60 | 48.80 | 57.40
ODHR@2 ($\uparrow$) | 57.34 | 61.47 | 53.94 | 53.69 | 49.75 | 54.96
DAR@2 ($\downarrow$) | 1.427 | 1.385 | 1.461 | 1.463 | 1.503 | 1.450
ODHR@3 ($\uparrow$) | 32.15 | 39.30 | 34.42 | 33.84 | 34.45 | 35.98
DAR@3 ($\downarrow$) | 2.010 | 1.904 | 2.017 | 1.994 | 2.034 | 1.947
ODHR@4 ($\uparrow$) | 27.20 | 36.99 | 31.05 | 27.39 | 25.36 | 37.74
DAR@4 ($\downarrow$) | 2.399 | 2.142 | 2.331 | 2.436 | 2.460 | 2.247

Table 2: Statistical results of random player VS vanilla LLM player or RL
player with DNQ. The decision threshold $p$ for critical decision in ODHR@K
and ADR@K is 0.15. Bold indicates the best result, underline the second best
result, and the Table 3 below follows this pattern.

Metrics | Vanilla LLM Players
---|---
GPT-3.5 | GPT-4 | Gemini-Pro | Llama 2 | ChatGLM3
WR ($\uparrow$) | 22.80 | 24.20 | 20.40 | 20.00 | 15.60
ODHR@2 ($\uparrow$) | 52.57 | 54.77 | 49.88 | 54.08 | 50.52
DAR@2 ($\downarrow$) | 1.474 | 1.452 | 1.501 | 1.459 | 1.495
ODHR@3 ($\uparrow$) | 39.56 | 41.41 | 33.14 | 34.78 | 33.13
DAR@3 ($\downarrow$) | 1.889 | 1.885 | 2.034 | 1.978 | 2.043
ODHR@4 ($\uparrow$) | 26.75 | 29.03 | 25.74 | 24.90 | 25.04
DAR@4 ($\downarrow$) | 2.407 | 2.366 | 2.516 | 2.471 | 2.477

Table 3: Statistical results of competition among 5 vanilla LLM players in UNO
Arena. The decision threshold $p$ for critical decision in ODHR@K and ADR@K is
0.00.

LLM | WR ($\uparrow$) | ODHR@2 ($\uparrow$) | DAR@2 ($\downarrow$) | ODHR@3 ($\uparrow$) | DAR@3 ($\downarrow$) | ODHR@4 ($\uparrow$) | DAR@4 ($\downarrow$)
---|---|---|---|---|---|---|---
GPT-3.5 (vanilla) | 48.00 | 53.05 | 1.4695 | 34.97 | 1.9508 | 34.47 | 2.2340
GPT-3.5 (TuTri) | 52.50 (+4.50%) | 54.01 (+0.06%) | 1.4599 (-0.06%) | 43.13 (+8.16%) | 1.8563 (-4.73%) | 32.92 (-1.55%) | 2.2667 (+1.09%)
GPT-4 (vanilla) | 49.00 | 56.27 | 1.4373 | 39.38 | 1.9375 | 36.24 | 2.2140
GPT-4 (TuTri) | 51.00 (+2.00%) | 56.60 (+0.33%) | 1.4340 (-0.33%) | 40.14 (+0.76%) | 1.8592 (-3.92%) | 36.33 (+0.09%) | 2.1510 (-2.10%)
Gemini-pro (vanilla) | 44.00 | 50.62 | 1.4938 | 37.04 | 2.0159 | 25.44 | 2.4737
Gemini-pro (TuTri) | 56.50 (+12.50%) | 53.64 (+3.02%) | 1.4636 (-3.02%) | 34.13 (-2.91%) | 1.9461 (-3.49%) | 30.36 (+4.92%) | 2.3482 (-4.18%)
Llama 2 (vanilla) | 47.00 | 49.54 | 1.5046 | 33.11 | 1.9595 | 29.11 | 2.3944
Llama 2 (TuTri) | 54.00 (+7.00%) | 55.07 (+5.53%) | 1.4493 (-5.53%) | 37.31 (+4.20%) | 1.8507 (-5.44%) | 26.75 (-2.36%) | 2.4650 (+2.35%)
ChatGLM3 (vanilla) | 47.00 | 55.82 | 1.4418 | 29.05 | 2.0541 | 31.84 | 2.2935
ChatGLM3 (TuTri) | 54.00 (+7.00%) | 57.24 (+1.42%) | 1.4276 (-1.42%) | 39.51 (+10.46%) | 1.8642 (-9.50%) | 30.62 (-1.22%) | 2.4689 (+5.85%)

Table 4: Statistical results of vanilla LLM players VS TuTri players. The
decision threshold $p$ for critical decision in ODHR@K and ADR@K is 0.00. Red
annotations indicate favorable experimental results, while blue annotations
indicate unfavorable experimental results.

LLM | WR ($\uparrow$) | ODHR@2 ($\uparrow$) | DAR@2 ($\downarrow$) | ODHR@3 ($\uparrow$) | DAR@3 ($\downarrow$) | ODHR@4 ($\uparrow$) | DAR@4($\downarrow$)
---|---|---|---|---|---|---|---
Gemini-pro (TuTri) | 56.50 | 53.64 | 1.4636 | 34.13 | 1.9461 | 30.36 | 2.3482
Gemini-pro + $\textsc{TuTri}^{\prime}$ | 52.50 (-4.00%) | 54.33 (+0.69%) | 1.4567 (-0.69%) | 29.88 (-4.25%) | 2.0610 (+5.75%) | 31.25 (+0.89%) | 2.4219 (+2.46%)
Gemini-pro + $\textsc{TuTri}^{\prime\prime}$ | 53.50 (-3.00%) | 54.59 (+0.95%) | 1.4541 (-0.95%) | 31.95 (-2.18%) | 2.0384 (+4.62%) | 27.59 (-2.77%) | 2.3824 (-1.14%)

Table 5: Statistical results of the ablation study. Where
$\textsc{TuTri}^{\prime}$ represents the TuTri player which remove the game
history reflection module, and $\textsc{TuTri}^{\prime\prime}$ represents the
TuTri player which remove the game strategy reflection module. The decision
threshold $p$ for critical decision in ODHR@K and ADR@K is 0.15. Red
annotations indicate favorable experimental results, while blue annotations
indicate unfavorable experimental results.

From the Table 2, we can find that (1) Except for ChatGLM3, the WR of other
vanilla LLM players and RL players are all above 50.00%; (2) The performance
of GPT-4 is the best, and GPT-4 performs excellently on the 7 evaluation
metrics. Especially, the WR of GPT-4 is 63.20%, 13.20% higher than 50.00%.

###  4.2 5-players UNO Arena with 5 LLMs

To find the best LLM, we place 5 LLMs in a 5-players UNO Arena to compete
against each other. We fix the initial playing order of UNO Arena in the
sequence of GPT-3.5, GPT-4, Gemini-Pro, Llama 2, and ChatGLM3. We conduct
experiment on 200 decks generated randomly. All players are the vanilla LLM
players. The results are shown in Table 3.

From the Table 3, we can find that (1) GPT-4 has the best performance, with a
WR of 24.20%, 4.2% higher than the average (20.00%) and 1.4% higher than the
second highest ranked GPT-3.5. Not only that, GPT-4 also performs the best in
other 6 evaluation metrics; (2) ChatGLM3 has the worst performance, with a WR
of 15.60%, which is 4.4% lower than the average (20.00%) and 8.6% lower than
the highest ranked GPT-4. Not only that, ChatGLM3 also performs the worst in
ODHR@2, DAR@2, DAR@3, ODHR@4, and DAR@4.

Figure 2: The Pearson Correlation Heatmap among WR, ODHR@K (K=2,3,4), and
ADR@K (K=2,3,4). Figure 3: GPT-4 (the vanilla LLM player) real-time winning
rate variations on 4 decks.

###  4.3 Validation of the superiority of the TuTri player compared to the
vanilla LLM player

To verify that our TuTri player can improve the sequential decision-making
ability of LLMs, we compare the vanilla LLM players (baseline) with TuTri
players. We let 5 LLMs serve as the backend LLMs for both the vanilla LLM
players and TuTri players, and play two-players UNO Arena on 200 decks
generated randomly. The results are shown in Table 4.

From the Table 4, we can find that: (1) All LLMs (the TuTri player) are better
than LLM (the vanilla LLM player) on WR, ODHR@2, and DAR@3. Gemini-Pro (the
TuTri player) has a 12.50% higher than Gemini-Pro (the vanilla LLM player) on
WR; (2) For ODHR@3, except for Gemini-Pro which performed slightly worse
(-2.91%), the other 4 LLMs achieved good results. For ODHR@4 and DAR@4, GPT-4
and Gemini-Pro both performed well. It can be seen that the TuTri player based
on reflection can significantly improve its abilities of sequential decision-
making after two rounds of reflection on the summary of game history and the
game strategies. The experimental results strongly support the superiority of
our TuTri player based reflection over the vanilla LLM player.

###  4.4 Ablation studies on TuTri player

To illustrate the necessity of the two reflection modules in the TuTri player,
we conduct ablation study. We remove the game history reflection module and
the game strategy reflection module from the TuTri players, and conduct two-
players UNO Arena with vanilla LLM player respectively. The results are shown
in Table 5.

From the Table 5, we can find that: (1) After removing the game history
reflection module, the WR of decreased by 4%, the ODHR@3 of decreased 4.25%,
and the DAR@3 of increase by 5.75%. (2) After removing the game strategy
reflection module, the WR of decreased by 3%, the ODHR@3 of decreased 2.18%.
and the DAR@3 of increase by 4.62%. The game history holds significant
potential information for incomplete information games. Therefore, removing
the game history reflection module has a greater adverse impact on the TuTri
player.

##  5 Discussion

###  5.1 Further Exploration of ODHR@K and ADR@K

To better analyze the relationship between our unique evaluation metrics
(ODHR@K and ADR@K), and the evaluation metric WR, we conduct a Pearson
correlation analysis of the experimental results from the Table 2. The results
are shown in Figure 2. From the Figure 2, we can find that (1) WR shows a
positive correlation with ODHR@K (K=2,3,4), and simultaneously, WR shows a
negative correlation with ADR@K (K=2,3,4); (2) The strongest positive
correlation, reaching 0.85, exists between WR and ODHR@3, while the strongest
negative correlation, reaching -0.86, exists between WR and DAR@3. Overall,
our unique ODHR@K and ADR@K have a good correlation with WR, so they can serve
as reference evaluation metrics for evaluating LLMs in UNO Arena.

###  5.2 Case Study

In order to more intuitively see the advantages of LLM versus random player,
we conduct a case study. We utilized GPT-4 as the backend LLM for the vanilla
LLM player to engage in the game across 4 decks generated randomly, with the
random player plays first. We recorded all decision points (for both the
vanilla LLM player and the random player) and employed the Monte Carlo method
to calculate the real-time percentage change in winning rate for both sides
following each decision point. The results are shown in Figure 3.

From the Figure 3, we can find that: (1) In UNO Arena, winning rates fluctuate
significantly. For example, in deck 1, from round 28 to 29, the random
player’s winning rate dropped by 34.5%, while the vanilla LLM player by
34.67%; (2) Turning points, like rounds 44 and 67 in deck 3, show shifts in
dominance. Initially, the vanilla LLM player leads until round 44, then loses
advantage until round 67, before regaining control; (3) Brief game durations
occur, notably in deck 4, where the agent player consistently makes
exceptional decisions, steadily increasing its winning rate until achieving
victory. These findings underscore LLM’s adeptness at identifying crucial
decision junctures and exploiting its capabilities, highlighting its potential
in sequential decision-making scenarios.

##  6 Conclusion

In conclusion, LLMs possess the capability for sequential decision-making, as
evidenced by the experimental results of LLMs playing the UNO game. Our
proposed UNO Arena and unique evaluation metrics enable LLMs to compete with
each other in the same UNO Arena game, thereby providing a better dynamic
assessment of LLMs’ sequential decision-making abilities. Furthermore, we
propose that the TuTri player effectively addresses how to enhance LLMs’
sequential decision-making abilities for better performance in playing UNO
Arena.

## Limitations

The method of dynamically evaluating the sequential decision-making ability of
LLMs using the UNO Arena, as well as the TuTri player, is only applicable to
LLMs that support chat. The unique evaluation metrics, ODHR@K and ADR@K,
introduced in this paper are only applicable to games or tasks with a limited
action space.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Aiyappa et al. (2023) Rachith Aiyappa, Jisun An, Haewoon Kwak, and Yong-Yeol Ahn. 2023.  Can we trust the evaluation on chatgpt?
  * Akata et al. (2023) Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. 2023.  Playing repeated games with large language models.  _arXiv preprint arXiv:2305.16867_.
  * Amir (2014) Eyal Amir. 2014.  Reasoning and decision making.  _The Cambridge handbook of artificial intelligence_ , pages 191–212.
  * Amiri et al. (2020) Saeid Amiri, Mohammad Shokrolah Shirazi, and Shiqi Zhang. 2020.  Learning and reasoning for robot sequential decision making under uncertainty.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 2726–2733.
  * Bouton (1901) Charles L Bouton. 1901.  Nim, a game with a complete mathematical theory.  _The Annals of Mathematics_ , 3(1/4):35–39.
  * Brookins and DeBacker (2023) Philip Brookins and Jason Matthew DeBacker. 2023.  Playing games with gpt: What can we learn about a large language model from canonical strategic games?  _Available at SSRN 4493398_.
  * Chang et al. (2023) Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2023.  A survey on evaluation of large language models.  _ACM Transactions on Intelligent Systems and Technology_.
  * Chen et al. (2023) Liting Chen, Lu Wang, Hang Dong, Yali Du, Jie Yan, Fangkai Yang, Shuang Li, Pu Zhao, Si Qin, Saravan Rajmohan, et al. 2023.  Introspective tips: Large language model for in-context decision making.  _arXiv preprint arXiv:2305.11598_.
  * Demaine et al. (2014) Erik D Demaine, Martin L Demaine, Nicholas JA Harvey, Ryuhei Uehara, Takeaki Uno, and Yushi Uno. 2014.  Uno is hard, even for a single player.  _Theoretical Computer Science_ , 521:51–61.
  * Du et al. (2021) Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2021.  Glm: General language model pretraining with autoregressive blank infilling.  _arXiv preprint arXiv:2103.10360_.
  * Frankish and Ramsey (2014) Keith Frankish and William M Ramsey. 2014.  _The Cambridge handbook of artificial intelligence_.  Cambridge University Press.
  * Ganguli et al. (2023) Deep Ganguli, Nicholas Schiefer, Marina Favaro, and Jack Clark. 2023.  Challenges in evaluating AI systems.
  * Gong et al. (2023) Ran Gong, Qiuyuan Huang, Xiaojian Ma, Hoi Vo, Zane Durante, Yusuke Noda, Zilong Zheng, Song-Chun Zhu, Demetri Terzopoulos, Li Fei-Fei, et al. 2023.  Mindagent: Emergent gaming interaction.  _arXiv preprint arXiv:2309.09971_.
  * Guo et al. (2023) Jiaxian Guo, Bo Yang, Paul Yoo, Bill Yuchen Lin, Yusuke Iwasawa, and Yutaka Matsuo. 2023.  Suspicion-agent: Playing imperfect information games with theory of mind aware gpt-4.
  * Guo et al. (2024a) Shangmin Guo, Haoran Bu, Haochuan Wang, Yi Ren, Dianbo Sui, Yuming Shang, and Siting Lu. 2024a.  Economics arena for large language models.  _arXiv preprint arXiv:2401.01735_.
  * Guo et al. (2024b) Shoutao Guo, Shaolei Zhang, Zhengrui Ma, Min Zhang, and Yang Feng. 2024b.  Sillm: Large language models for simultaneous machine translation.  _arXiv preprint arXiv:2402.13036_.
  * He et al. (2022) Zexue He, Yu Wang, Julian McAuley, and Bodhisattwa Prasad Majumder. 2022.  Controlling bias exposure for fair interpretable predictions.  _arXiv preprint arXiv:2210.07455_.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021.  Measuring massive multitask language understanding.
  * Kroese et al. (2014) Dirk P Kroese, Tim Brereton, Thomas Taimre, and Zdravko I Botev. 2014.  Why the monte carlo method is so important today.  _Wiley Interdisciplinary Reviews: Computational Statistics_ , 6(6):386–392.
  * Li et al. (2020) Junjie Li, Sotetsu Koyamada, Qiwei Ye, Guoqing Liu, Chao Wang, Ruihan Yang, Li Zhao, Tao Qin, Tie-Yan Liu, and Hsiao-Wuen Hon. 2020.  Suphx: Mastering mahjong with deep reinforcement learning.  _arXiv preprint arXiv:2003.13590_.
  * Light et al. (2023) Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. 2023.  Avalonbench: Evaluating llms playing the game of avalon.
  * Littman (1996) Michael Lederman Littman. 1996.  _Algorithms for sequential decision-making_.  Brown University.
  * Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023.  Self-refine: Iterative refinement with self-feedback.  _arXiv preprint arXiv:2303.17651_.
  * McMullen (2015) Jeffery S McMullen. 2015.  Entrepreneurial judgment as empathic accuracy: A sequential decision-making approach to entrepreneurial action.  _Journal of Institutional Economics_ , 11(3):651–681.
  * Mnih et al. (2013) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_.
  * Mooney (1997) Christopher Z Mooney. 1997.  _Monte carlo simulation_.  116\. Sage.
  * OpenAI (2022) OpenAI. 2022.  Introducing chatgpt.  https://openai.com/blog/chatgpt.  Accessed: 2023-09-30.
  * Pfann (2021) Bernhard Pfann. 2021.  Tackling the uno card game with reinforcement learning.  _Towards Data Science: tackling-uno-card-game-with-reinforcement-learning_.
  * Sainz et al. (2023) Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023.  NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 10776–10787, Singapore. Association for Computational Linguistics.
  * Shinn et al. (2024) Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024.  Reflexion: Language agents with verbal reinforcement learning.  _Advances in Neural Information Processing Systems_ , 36.
  * Silver et al. (2017) David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. 2017.  Mastering the game of go without human knowledge.  _nature_ , 550(7676):354–359.
  * Taneja et al. (2024) Jubi Taneja, Avery Laird, Cong Yan, Madan Musuvathi, and Shuvendu K Lahiri. 2024.  Llm-vectorizer: Llm-based verified loop vectorizer.  _arXiv preprint arXiv:2406.04693_.
  * Team et al. (2023) Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Wang et al. (2023a) Sheng Wang, Zihao Zhao, Xi Ouyang, Qian Wang, and Dinggang Shen. 2023a.  Chatcad: Interactive computer-aided diagnosis on medical image using large language models.  _arXiv preprint arXiv:2302.07257_.
  * Wang et al. (2023b) Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. 2023b.  Avalon’s game of thoughts: Battle against deception through recursive contemplation.
  * Wang et al. (2023c) Yu Wang, Zhiwei Liu, Jianguo Zhang, Weiran Yao, Shelby Heinecke, and Philip S Yu. 2023c.  Drdt: Dynamic reflection with divergent thinking for llm-based sequential recommendation.  _arXiv preprint arXiv:2312.11336_.
  * Wei et al. (2022) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_.
  * Wythoff (1907) Willem A Wythoff. 1907.  A modification of the game of nim.  _Nieuw Arch. Wisk_ , 7(2):199–202.
  * Xi et al. (2023) Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023.  The rise and potential of large language model based agents: A survey.  _arXiv preprint arXiv:2309.07864_.
  * Xu et al. (2024) Cheng Xu, Shuhao Guan, Derek Greene, and M-Tahar Kechadi. 2024.  Benchmark data contamination of large language models: A survey.
  * Xu et al. (2023) Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. 2023.  Exploring large language models for communication games: An empirical study on werewolf.
  * You et al. (2019) Yang You, Liangwei Li, Baisong Guo, Weiming Wang, and Cewu Lu. 2019.  Combinational q-learning for dou di zhu.  _arXiv preprint arXiv:1901.08925_.
  * Zeng et al. (2022) Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022.  Glm-130b: An open bilingual pre-trained model.  _arXiv preprint arXiv:2210.02414_.
  * Zeng et al. (2024) Zhongshen Zeng, Pengguang Chen, Shu Liu, Haiyun Jiang, and Jiaya Jia. 2024.  Mr-gsm8k: A meta-reasoning revolution in large language model evaluation.
  * Zha et al. (2019) Daochen Zha, Kwei-Herng Lai, Yuanpu Cao, Songyi Huang, Ruzhe Wei, Junyu Guo, and Xia Hu. 2019.  Rlcard: A toolkit for reinforcement learning in card games.  _arXiv preprint arXiv:1910.04376_.
  * Zhai et al. (2024) Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, and Sergey Levine. 2024.  Fine-tuning large vision-language models as decision-making agents via reinforcement learning.
  * Zhang et al. (2023) Jintian Zhang, Xin Xu, and Shumin Deng. 2023.  Exploring collaboration mechanisms for llm agents: A social psychology view.  _arXiv preprint arXiv:2310.02124_.
  * Zhou et al. (2023) Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. 2023.  Don’t make your llm an evaluation benchmark cheater.

Appendix

##  Appendix A UNO Game

In this section of appendix, you will learn what UNO Game is, and in order to
facilitate the evaluation of LLMs with the UNO Game, we have made some slight
modifications to it.

###  A.1 Game Objective

In the modified UNO game, We simply set the game objective as to be the first
player to clear out the hand. Players play alternatively(2 players) or in
circle manner(3 or more players) and strive to achieve the unique goal. It
should be noted that if the cards in the deck are exhausted by players, the
player with the fewest number of cards in hand wins, so there may be multiple
winners in the same game.

###  A.2 UNO Cards

UNO comprises 3 categories of cards: number cards, function cards, and wild
cards. In total, UNO features 108 cards.

  * •

The Number Cards: the number cards can be expressed in the form of COLOR +
NUMBER, where COLOR is belong to the set $\\{Red,Blue,Yellow,Green\\}$, and
NUMBER is an integer from $0$ to $9$. It is important to note that there is
only one 0-number card per color, while there are two 1-9 number cards per
color. There are a total of 76 number cards.

  * •

The Function Cards: the function cards can be expressed in the form of COLOR +
FUNCTION, where COLOR is belong to the set $\\{Red,Blue,Yellow,Green\\}$, and
FUNCTION is belong to the set $\\{Skip,Reverse,DrawTwo\\}$. There are two
cards of the same COLOR for each FUNCTION. There are a total of 24 function
cards.

    * –

Skip: the player’s next player skips this round of play.

    * –

Reverse: the player reverses the order of play (from clockwise to
counterclockwise, or from counterclockwise to clockwise).

    * –

Draw Two: The player’s next player draws two cards and skips this round of
play.

  * •

The Wild Cards: the wild card includes 4 Black Wild cards and 4 Black Wild
Draw Four cards, totaling 8 cards.

    * –

Wild: the player selects one color from the COLOR set
$\\{Red,Blue,Yellow,Green\\}$ as the new color for the top card in the discard
pile.

    * –

Wild Draw Four: the player selects one color from the COLOR set
$\\{Red,Blue,Yellow,Green\\}$ as the new color for the top card in the discard
pile, and the player’s next player draws 4 cards.

###  A.3 Game Progress

First, deal each player 7 initial cards in clockwise order, then continue
drawing cards until a number card is drawn and set as the top card of the
initial discard pile. All players take rounds playing cards in clockwise
order(it will be reversed by a reverse card) until a player runs out of his
cards or the draw pile is exhausted, signaling the end of the game.

###  A.4 Legal Decision(Action)

In every round of the game, player in charge can using rules to match the top
card of the discard pile otherwise pick up a new card into hand. The rules, or
say, the legal decisions consists of several sorts:

  * •

Draw Card: If a player does not have any cards to play during their playing
round, they must draw a card, or their previous player used Draw Two or Wild
Draw Four cards to make the player draw multiple cards.

  * •

Select Card: In a player’s playing round, they need to play a card that
matches either the COLOR, NUMBER, or FUNCTION of the top card in the discard
pile, or play a Wild card(include Wild Draw Four card) to match. The card
played by the player then becomes the new top card.

  * •

Select Color: After selecting either Wild Card or Wild Draw Four Card, the
player needs to convert the color of the current top card to one of
$\\{Red,Blue,Yellow,Green\\}$.

  * •

Select ChallengeFlag: The use of the Wild Draw Four card may be illegal. After
a player plays a Wild Draw Four card, their next player can choose to
challenge its use. If the player who played the Wild Draw Four card still
holds non-Wild cards matching the color of the current top card, the use of
the Wild Draw Four card is illegal. Possible scenarios are as follows: (1) If
the player’s play is illegal and their next player challenges it, the player
must draw 4 cards, and their next player faces no penalty; (2) If the player’s
play is legal and their next player challenges it, the player’s next player
must draw 6 cards. (3) If the player’s next player does not challenge,
regardless of the legality of the player’s play, the player’s next player must
draw 4 cards. Note that Challenge is not a stand-alone action to complete
turns, it should be accompanied by a card draw or card match action.

##  Appendix B Prompt

Here is the prompt design for the entire experiment.

###  B.1 Select Card

The input1 prompt of the select card shared by the vanilla LLM player and the
TuTri player is shown in the Figure 4. The game history Rreflection module
prompt of the select card for the TuTri player is shown in the Figure 5. The
game strategy reflection module prompt of the select card for the TuTri player
is shown in the Fugure 6.

###  B.2 Select Color

The input1 prompt of the select color shared by the vanilla LLM player and the
TuTri player is shown in the Figure 7. The game history reflection module
prompt of the select color for the TuTri player is shown in the Figure 8. The
game strategy reflection module prompt of the select color for the TuTri
player is shown in the Fugure 9.

###  B.3 Select ChallengeFlag

The input1 prompt of the select challengeFlag shared by the vanilla LLM player
and the TuTri player is shown in the Figure 10. The game history reflection
module prompt of the select challengeFlag for the TuTri player is shown in the
Figure 11. The game strategy reflection module prompt of the select
challengeFlag for the TuTri player is shown in the Fugure 12.

Figure 4: The input1 prompt of the select card shared by the vanilla LLM
player and the TuTri player. Figure 5: The game history reflection module
prompt of the select card for TuTri player. Figure 6: The game strategy
reflection module prompt of the select card for TuTri player. Figure 7: The
input1 prompt of the select color shared by the vanilla LLM player and the
TuTri player. Figure 8: The game history reflection module prompt of the
select color for TuTri player. Figure 9: The game strategy reflection module
prompt of the select color for TuTri player. Figure 10: The input1 prompt of
the select challengeFlag shared by the vanilla LLM player and the TuTri
player. Figure 11: The game history reflection module prompt of the select
challengeFlag for TuTri player. Figure 12: The game strategy reflection module
prompt of the select challengeFlag for TuTri player.


* (30) M. Heesen, S. Macdonald, J. Ostner, and O. Schülke, “Ecological and social determinants of group cohesiveness and within‐group spatial position in wild Assamese macaques”, Ethology 121, 270 (2015).
  * (31) H. Sugiura, Y. Shimooka, and Y. Tsuji, “Variation in spatial cohesiveness in a group of Japanese macaques (Macaca fuscata)”, Int. J. Primatol. 32, 1348 (2011).
  * (32) M. Nishikawa, M. Suzuki, and D. S. Sprague, “Activity and social factors affect cohesion among individuals in female Japanese macaques: A simultaneous focal-follow study”, Am. J. Primatol. 76, 694 (2014).


††thanks: corresponding author, e-mail<EMAIL_ADDRESS>corresponding
author, e-mail<EMAIL_ADDRESS>deceased††thanks: deceased

A2 Collaboration at MAMI

# Photoproduction of the $\omega$ meson on the proton near threshold

I. I. Strakovsky  The George Washington University, Washington, DC 20052-0001,
USA  S. Prakhov  The George Washington University, Washington, DC 20052-0001,
USA  Institut für Kernphysik, University of Mainz, D-55099 Mainz,Germany
University of California Los Angeles, Los Angeles, California 90095-1547, USA
Ya. I. Azimov  Petersburg Nuclear Physics Institute, 188300 Gatchina, Russia
P. Aguar-Bartolomé  Institut für Kernphysik, University of Mainz, D-55099
Mainz,Germany  J. R. M. Annand  SUPA School of Physics and Astronomy,
University of Glasgow, Glasgow G12 8QQ, United Kingdom  H. J. Arends  Institut
für Kernphysik, University of Mainz, D-55099 Mainz,Germany  K. Bantawa  Kent
State University, Kent, Ohio 44242-0001, USA  R. Beck  Helmholtz-Institut für
Strahlen- und Kernphysik, University of Bonn, D-53115 Bonn, Germany  V.
Bekrenev  Petersburg Nuclear Physics Institute, 188300 Gatchina, Russia  H.
Berghäuser  II Physikalisches Institut, University of Giessen, D-3539 Giessen,
Germany  A. Braghieri  INFN Sezione di Pavia, I-27100 Pavia, Italy  W. J.
Briscoe  The George Washington University, Washington, DC 20052-0001, USA  J.
Brudvik  University of California Los Angeles, Los Angeles, California
90095-1547, USA  S. Cherepnya  Lebedev Physical Institute, 119991 Moscow,
Russia  R. F. B. Codling  SUPA School of Physics and Astronomy, University of
Glasgow, Glasgow G12 8QQ, United Kingdom  C. Collicott  Dalhousie University,
Halifax, Nova Scotia B3H 4R2, Canada  Saint Mary’s University, Halifax, Nova
Scotia B3H 3C3, Canada  S. Costanza  INFN Sezione di Pavia, I-27100 Pavia,
Italy  B. T. Demissie  The George Washington University, Washington, DC
20052-0001, USA  E. J. Downie  Institut für Kernphysik, University of Mainz,
D-55099 Mainz,Germany  The George Washington University, Washington, DC
20052-0001, USA  P. Drexler  II Physikalisches Institut, University of
Giessen, D-3539 Giessen, Germany  L. V. Fil’kov  Lebedev Physical Institute,
119991 Moscow, Russia  D. I. Glazier  SUPA School of Physics and Astronomy,
University of Glasgow, Glasgow G12 8QQ, United Kingdom  School of Physics,
University of Edinburgh, Edinburgh EH9 3JZ, United Kingdom  R. Gregor  II
Physikalisches Institut, University of Giessen, D-3539 Giessen, Germany  D. J.
Hamilton  SUPA School of Physics and Astronomy, University of Glasgow, Glasgow
G12 8QQ, United Kingdom  E. Heid  The George Washington University,
Washington, DC 20052-0001, USA  Institut für Kernphysik, University of Mainz,
D-55099 Mainz,Germany  D. Hornidge  Mount Allison University, Sackville, New
Brunswick E4L 1E6, Canada  I. Jaegle  Institut für Physik, University of
Basel, CH-4056 Basel, Switzerland  O. Jahn  Institut für Kernphysik,
University of Mainz, D-55099 Mainz,Germany  T. C. Jude  School of Physics,
University of Edinburgh, Edinburgh EH9 3JZ, United Kingdom  V. L. Kashevarov
Institut für Kernphysik, University of Mainz, D-55099 Mainz,Germany  Lebedev
Physical Institute, 119991 Moscow, Russia  I. Keshelashvili  Institut für
Physik, University of Basel, CH-4056 Basel, Switzerland  R. Kondratiev
Institute for Nuclear Research, 125047 Moscow, Russia  M. Korolija  Rudjer
Boskovic Institute, HR-10000 Zagreb, Croatia  M. Kotulla  II Physikalisches
Institut, University of Giessen, D-3539 Giessen, Germany  A. Koulbardis
Petersburg Nuclear Physics Institute, 188300 Gatchina, Russia  S. Kruglov
Petersburg Nuclear Physics Institute, 188300 Gatchina, Russia  B. Krusche
Institut für Physik, University of Basel, CH-4056 Basel, Switzerland  V. Lisin
Lebedev Physical Institute, 119991 Moscow, Russia  K. Livingston  SUPA School
of Physics and Astronomy, University of Glasgow, Glasgow G12 8QQ, United
Kingdom  I. J. D. MacGregor  SUPA School of Physics and Astronomy, University
of Glasgow, Glasgow G12 8QQ, United Kingdom  Y. Maghrbi  Institut für Physik,
University of Basel, CH-4056 Basel, Switzerland  D. M. Manley  Kent State
University, Kent, Ohio 44242-0001, USA  Z. Marinides  The George Washington
University, Washington, DC 20052-0001, USA  J. C. McGeorge  SUPA School of
Physics and Astronomy, University of Glasgow, Glasgow G12 8QQ, United Kingdom
E. F. McNicoll  SUPA School of Physics and Astronomy, University of Glasgow,
Glasgow G12 8QQ, United Kingdom  D. Mekterovic  Rudjer Boskovic Institute,
HR-10000 Zagreb, Croatia  V. Metag  II Physikalisches Institut, University of
Giessen, D-3539 Giessen, Germany  D. G. Middleton  Institut für Kernphysik,
University of Mainz, D-55099 Mainz,Germany  Mount Allison University,
Sackville, New Brunswick E4L 1E6, Canada  A. Mushkarenkov  INFN Sezione di
Pavia, I-27100 Pavia, Italy  B. M. K. Nefkens  University of California Los
Angeles, Los Angeles, California 90095-1547, USA  A. Nikolaev  Helmholtz-
Institut für Strahlen- und Kernphysik, University of Bonn, D-53115 Bonn,
Germany  R. Novotny  II Physikalisches Institut, University of Giessen, D-3539
Giessen, Germany  H. Ortega  Institut für Kernphysik, University of Mainz,
D-55099 Mainz,Germany  M. Ostrick  Institut für Kernphysik, University of
Mainz, D-55099 Mainz,Germany  P. B. Otte  Institut für Kernphysik, University
of Mainz, D-55099 Mainz,Germany  B. Oussena  The George Washington University,
Washington, DC 20052-0001, USA  Institut für Kernphysik, University of Mainz,
D-55099 Mainz,Germany  P. Pedroni  INFN Sezione di Pavia, I-27100 Pavia, Italy
F. Pheron  Institut für Physik, University of Basel, CH-4056 Basel,
Switzerland  A. Polonski  Institute for Nuclear Research, 125047 Moscow,
Russia  J. Robinson  SUPA School of Physics and Astronomy, University of
Glasgow, Glasgow G12 8QQ, United Kingdom  G. Rosner  SUPA School of Physics
and Astronomy, University of Glasgow, Glasgow G12 8QQ, United Kingdom  T.
Rostomyan  Institut für Physik, University of Basel, CH-4056 Basel,
Switzerland  S. Schumann  Institut für Kernphysik, University of Mainz,
D-55099 Mainz,Germany  M. H. Sikora  The George Washington University,
Washington, DC 20052-0001, USA  School of Physics, University of Edinburgh,
Edinburgh EH9 3JZ, United Kingdom  A. Starostin  University of California Los
Angeles, Los Angeles, California 90095-1547, USA  I. Supek  Rudjer Boskovic
Institute, HR-10000 Zagreb, Croatia  M. F. Taragin  The George Washington
University, Washington, DC 20052-0001, USA  C. M. Tarbert  School of Physics,
University of Edinburgh, Edinburgh EH9 3JZ, United Kingdom  M. Thiel  II
Physikalisches Institut, University of Giessen, D-3539 Giessen, Germany  A.
Thomas  Institut für Kernphysik, University of Mainz, D-55099 Mainz,Germany
M. Unverzagt  Institut für Kernphysik, University of Mainz, D-55099
Mainz,Germany  Helmholtz-Institut für Strahlen- und Kernphysik, University of
Bonn, D-53115 Bonn, Germany  D. P. Watts  School of Physics, University of
Edinburgh, Edinburgh EH9 3JZ, United Kingdom  D. Werthmüller  Institut für
Physik, University of Basel, CH-4056 Basel, Switzerland  F. Zehr  Institut für
Physik, University of Basel, CH-4056 Basel, Switzerland

###### Abstract

An experimental study of $\omega$ photoproduction on the proton was conducted
by using the Crystal Ball and TAPS multiphoton spectrometers together with the
photon tagging facility at the Mainz Microtron MAMI. The $\gamma p\to\omega p$
differential cross sections are measured from threshold to the incident-photon
energy $E_{\gamma}=1.40$ GeV ($W=1.87$ GeV for the center-of-mass energy) with
15-MeV binning in $E_{\gamma}$ and full production-angle coverage. The quality
of the present data near threshold gives access to a variety of interesting
physics aspects. As an example, an estimation of the $\omega N$ scattering
length $\alpha_{\omega p}$ is provided.

###### pacs:

12.40.Vv,13.60.Le,14.40.Be,25.20.Lj

##  I Introduction

Although Quantum Chromodynamics (QCD) is generally believed to govern strong
interactions, it still can be applied to particular problems only in terms of
specific model-dependent approaches, which can be distinguished by their
predictions for the resonance spectra. However, such predictions can only be
verified in collisions of a very restricted set of hadron pairs: a proton or a
bound neutron as a target and a stable or a weakly decaying hadron as a
projectile. Meanwhile, various constituent quark models (e.g., see Ref. CapRob
and references therein) predict a richer spectrum of hadron resonances than
have so far been observed in experiments PDG . Because most known baryon
states were discovered in elastic $\pi N$ scattering, some resonances could
have been missed because of their weak coupling to the $\pi N$ channels ar06 .
At the same time, a stronger coupling of those resonances to such channels as
$\eta N$, $\omega N$, $K\Lambda$, or $K\Sigma$ cannot be excluded, and,
therefore, an extensive study of these channels is very important in searching
for the so-called “missing” resonances Koniuk . Proof of their existence would
constitute a strong confirmation of the validity of the constituent quark
concept.

Although the $\omega$ meson is neither stable nor decaying weakly, the $\omega
N$ channel is favorable in searching for missing resonances because, owing to
vector-meson dominance (VMD) VMD , $\omega$ photoproduction on the nucleon may
be directly related to the elastic amplitude for $\omega N$ scattering. In
addition, $\omega$ photoproduction provides an “isospin filter” for the
nucleon response because $\omega N$ final states can originate only from
$N^{*}$ states with $I=1/2$, but not from $\Delta^{*}$s with $I=3/2$. The
$\omega N$ threshold region is also especially attractive in searching for new
resonances because the reaction threshold is located at the higher-energy edge
of the third resonance region, in which the Review of Particle Physics (RPP)
PDG  shows seven $N^{*}$ states with masses between 1650 and 1720 MeV, and
then there are no observed $N^{*}$ states up to 1860 MeV. It cannot be
excluded that this energy range may contain unknown $N^{*}$ resonances that
are coupled more strongly to $\omega N$ than to other meson-baryon channels.
Such a strong coupling to resonances in the near-threshold region is clearly
seen, for example, for the two most dominant channels, $\pi N$ and $\eta N$,
coupled to $\Delta(1232)3/2^{+}$ and $N(1535)1/2^{-}$, respectively.

The photoproduction of $\omega$ mesons was under extensive theoretical
discussion since the first high-statistics differential cross sections and
polarization results were provided from SAPHIR SAPHIR , covering a broad
interval in center-of-mass (c.m.) energy, up to $W=2.4$ GeV with $\sim 60$% of
the full production-angle range. Many models, including coupled channel,
effective Lagrangian, and QCD-inspired approaches Beijing ; Titov1 ; ba02 ;
Titov2 ; Giessen ; Paris , were developed, trying to explain the behavior of
the experimental data. Another high-statistics measurement of the $\gamma
p\to\omega p$ differential cross sections and spin-density matrix elements was
recently made by the CLAS Collaboration CLAS1 , covering c.m. energies from
threshold up to $W=2.84$ GeV. Their binning in c.m. energy was much finer than
from SAPHIR SAPHIR , but the production-angle coverage was narrower.
Unfortunately, for both SAPHIR and CLAS, the data obtained in the energy
region near threshold have the poorest quality.

Both SAPHIR SAPHIR  and CLAS CLAS1  experiments measured the photoproduction
of $\omega$ via its major decay mode $\omega\to\pi^{+}\pi^{-}\pi^{0}$, having
89.2% branching ratio PDG . The analyses of those experimental data were
challenging because of large background from $\gamma
p\to\pi^{+}\pi^{-}\pi^{0}p$ produced via intermediate states different from
$\gamma p\to\omega p$. In the analysis by SAPHIR SAPHIR , all data points were
obtained by individual fits of the $\omega$ peak above the
$\pi^{+}\pi^{-}\pi^{0}$ background. Their fitting procedure experienced
difficulties with describing the $\omega$ peak near threshold as a standard
Breit-Wigner (BW) shape was severely distorted by phase space, and a function
(BW with parameters of $\omega$ convoluted with two Gaussians describing the
experimental resolutions) used for higher energies did not work here. In the
analysis by CLAS CLAS1 , a so-called $Q$-factor technique was used to separate
signal and background events, allowing to fit data with the event-by-event
approach and to extract all results from their unbinned maximum-likelihood
fits.

A background contamination of $\omega$ events would be expected to improve if
$\omega$ photoproduction were measured in process $\gamma p\to\omega
p\to\pi^{0}\gamma p\to 3\gamma p$, using a radiative decay mode,
$\omega\to\pi^{0}\gamma$, with 8.28% branching ratio PDG . Other physical
processes having the three-photon final state are expected to be very small.
However, as shown in a preliminary analysis of the CBELSA/TAPS data CBELSA ,
the three-photon background can come from processes $\gamma
p\to\pi^{0}\pi^{0}p\to 4\gamma p$ and $\gamma p\to\pi^{0}\eta p\to 4\gamma p$
when one of the four final-state photons was not detected.

In this work, a new high-statistics measurement of the $\gamma p\to\omega p$
differential cross sections near threshold is presented. The results are based
on an analysis of $\sim 5\times 10^{5}$ $\omega$ mesons detected via their
$\pi^{0}\gamma$ decay mode. All data were divided into 20 (15 MeV each) bins
in the incident-photon energy, $E_{\gamma}$, and 15 angular bins, covering the
full range of the $\omega$ production angle. All results are obtained by
individual fits of the $\omega$ peak above background in each energy-angle
bin.

A partial-wave analysis with extracting spin-density matrix elements from the
present data is in progress, by using the $Q$-factor technique. As the latter
analysis is being performed by a theoretical group outside the A2
Collaboration, their results will be published later on separately.

##  II Experimental setup

The process $\gamma p\to\omega p\to\pi^{0}\gamma p\to 3\gamma p$ was measured
using the Crystal Ball (CB) CB  as a central spectrometer and TAPS TAPS ;
TAPS2  as a forward spectrometer. These detectors were installed at the
energy-tagged bremsstrahlung-photon beam produced from the electron beam of
the Mainz Microtron (MAMI) MAMI ; MAMIC . In the present experiment,
bremsstrahlung photons, produced by the 1508-MeV electrons in a 10-$\mu$m Cu
radiator and collimated by a 4-mm-diameter Pb collimator, were incident on a
5-cm-long liquid-hydrogen (LH2) target located in the center of the CB. The
energies of the incident photons were analyzed up to 1402 MeV by detecting the
postbremsstrahlung electrons in the Glasgow tagging spectrometer (or tagger)
TAGGER ; TAGGER1 ; TAGGER2 . The uncertainty in the energy of the tagged
photons is mainly determined by the width of tagger focal-plane detectors and
the energy of the MAMI electron beam used in experiments. For the MAMI energy
of 1508 MeV, such an uncertainty was about $\pm 2$ MeV.

The CB detector is a sphere consisting of 672 optically isolated NaI(Tl)
crystals, shaped as truncated triangular pyramids, which point toward the
center of the sphere. The crystals are arranged in two hemispheres that cover
93% of $4\pi$ sr, sitting outside a central spherical cavity with a radius of
25 cm, which is designed to hold the target and inner detectors. In this
experiment, TAPS was arranged in a plane consisting of 384 BaF2 counters of
hexagonal cross section. It was installed 1.5 m downstream of the CB center
and covered the full azimuthal range for polar angles from $1^{\circ}$ to
$20^{\circ}$.

The experimental trigger required the total energy deposited in the CB to
exceed $\sim$320 MeV and the number of so-called hardware clusters in the CB
(multiplicity trigger) to be two or larger. In the trigger, a hardware cluster
in the CB was a block of 16 adjacent crystals in which at least one crystal
had an energy deposit larger than 30 MeV. The TAPS information was not used in
the trigger of the present experiment.

More details on the experimental setup, its resolutions, and other conditions
during the period of the data taking (first half of 2007) are given in Refs.
etamamic ; slopemamic  and references therein.

##  III Data handling

The reaction $\gamma p\to\omega p$ was searched for in events identified as
$\gamma p\to\pi^{0}\gamma p\to 3\gamma p$ candidates, having three or four
clusters reconstructed in the CB and TAPS together by software analysis. The
cluster algorithm in software was optimized for finding a group of adjacent
crystals in which the energy was deposited by a single-photon electromagnetic
shower. This algorithm also works well for a proton cluster. The software
threshold for the cluster energy was chosen to be 12 MeV. For the $\gamma
p\to\pi^{0}\gamma p\to 3\gamma p$ candidates, the three-cluster events were
analyzed assuming that the final-state proton was not detected. This typically
happens when the outgoing proton is stopped in the material of the downstream
beam tunnel of the CB, or the proton scatters in the backward direction within
the c.m. frame, resulting in such a low kinetic energy of the proton in the
laboratory system that it is below the software cluster threshold. Thus,
including three-cluster events in the analysis is vital for measuring $\gamma
p\to\omega p$ differential cross sections at very forward production angles of
$\omega$. The fraction of $\gamma p\to\omega p\to\pi^{0}\gamma p\to 3\gamma p$
events without the detected proton varies from 2.7% at the reaction threshold
to 7.6% at the highest energy of the present experiment.

The selection of event candidates and the reconstruction of the reaction
kinematics was based on the kinematic-fit technique. Details on the kinematic-
fit parametrization of the detector information and resolutions are given in
Ref. slopemamic . As discussed there, the information for the outgoing proton,
if it is detected, is used in the kinematic fit without the proton kinetic
energy, which has large uncertainties because of the material between the
target and the crystals of the calorimeters. In addition, when energetic
protons punch through the crystals, their kinetic energy cannot be determined
from their energy deposit. For three-clusters events, all parameters of the
outgoing proton are free variables of the kinematic fit. All three- and four-
clusters events that satisfied the $\gamma p\to\pi^{0}\gamma p\to 3\gamma p$
hypothesis with a probability greater than 2% were accepted for further
analysis. The kinematic fitting for this hypothesis includes the four main
constraints, which are based on the conservation of energy and three-momentum,
and an additional constraint on the invariant mass of two outgoing photons to
have the $\pi^{0}$-meson mass. The kinematic-fit output was used to
reconstruct the kinematics of the outgoing particles. Small misidentification
of the proton with a photon for the selected $\gamma p\to\omega
p\to\pi^{0}\gamma p\to 3\gamma p$ events was observed only in the three-
clusters events and only for clusters reconstructed in TAPS. Such events were
discarded based on the time of flight between a TAPS cluster and the CB signal
with respect to the energy of the TAPS cluster.

The determination of the experimental acceptance was based on a Monte Carlo
(MC) simulation of $\gamma p\to\omega p\to\pi^{0}\gamma p$ with an isotropic
production-angular distribution and an uniform beam distribution generated
from the reaction threshold up to the maximal experimental energy. To
reproduce the energy dependence of the $\omega$ resonance shape near
production threshold, the reaction $\gamma p\to\pi^{0}\gamma p$ was generated
first as phase space. Then, the invariant mass of the $\pi^{0}\gamma$ system,
$m(\pi^{0}\gamma)$, was folded with the Breit-Wigner (BW) function, the
parameters of which were taken for the $\omega$ meson ($m=782.65$ MeV and
$\Gamma=8.49$ MeV) from the RPP PDG . This approach allowed one to properly
reproduce the folding of the BW shape with phase space. All MC events were
propagated through a GEANT (3.21) simulation of the experimental setup. To
reproduce resolutions of the experimental data, the GEANT output was subject
to additional smearing, thus allowing both the simulated and experimental data
to be analyzed in the same way. Matching the energy resolution between the
experimental and MC events was adjusted via reaching agreement in the
invariant-mass resolutions and in the kinematic-fit stretch functions (or
pulls) and probability distributions. Such an adjustment was based on the
analysis of the same data set for reactions having almost no physical
background (namely, $\gamma p\to\pi^{0}p$, $\gamma p\to\eta p\to\gamma\gamma
p$, and $\gamma p\to\eta p\to 3\pi^{0}p$ slopemamic ). After taking into
account the trigger requirements in the analysis of the MC events, the average
acceptance for the process $\gamma p\to\omega p\to\pi^{0}\gamma p$ in the A2
experimental setup was found to be about 50%. The simulation of the trigger
was adjusted with the same reactions that was used for adjusting the energy
resolution, where $\gamma p\to\pi^{0}p$ and $\gamma p\to\eta p\to\gamma\gamma
p$ are especially sensitive to the multiplicity trigger, required two hardware
clusters in the CB. The agreement of the $\gamma p\to\eta p$ differential
cross sections obtained from both the $\eta\to\gamma\gamma$ and the $\eta\to
3\pi^{0}$ decay modes was another cross-check for the correctness of the
trigger simulation. As found out, the multiplicity trigger was not important
for the present analysis, as all $\gamma p\to\omega p\to\pi^{0}\gamma p$
events that passed the requirement on the CB total energy to exceed 320 MeV
had more than one hardware cluster in the CB.

Figure 1:  $m(\pi^{0}\gamma)$ invariant-mass distributions obtained for
$E_{\gamma}=1176$ MeV and $\cos\theta=0$. (a) MC simulation of $\gamma
p\to\omega p\to\pi^{0}\gamma p$ with a Gaussian fit ($\sigma=8.2$ MeV and FWHM
= 19.3 MeV). (b) MC simulation of the background reaction $\gamma
p\to\pi^{0}\pi^{0}p$ fitted with a polynomial of order eight. (c) Measured
spectrum fitted with the sum of a Gaussian and a polynomial of order eight,
where $N_{\omega}$ is the number of $\omega$ mesons found from this fit.

As turned out, the selected experimental events with $\omega\to\pi^{0}\gamma$
decays were also contaminated with some background distributed quite smoothly
under the $\omega$ peak. Based on the MC simulations of possible background
reactions, it was found that processes $\gamma p\to\pi^{0}\pi^{0}p\to 4\gamma
p$ and $\gamma p\to\pi^{0}\eta p\to 4\gamma p$ could mimic $\gamma
p\to\pi^{0}\gamma p$ when one of the four final-state photons had not been
detected. The background from these processes mostly contaminates the four-
cluster events. Events from reaction $\gamma p\to\pi^{0}\pi^{+}n$ can mimic
$\gamma p\to\pi^{0}\gamma p$ when the outgoing neutron was not detected, and
they contaminate the three-cluster events. Nevertheless, there was no
background process found that could mimic the $\omega\to\pi^{0}\gamma$ peak.

Because the suppression of the background processes was not possible without
severe losses in $\omega\to\pi^{0}\gamma$ events themselves, and the
subtraction of the background processes was not possible without precise MC
simulations of all possible background reactions, the $\omega\to\pi^{0}\gamma$
events were measured by fitting experimental $m(\pi^{0}\gamma)$ spectra with
some function, describing the $\omega$ peak above a smooth background. To
measure the $\gamma p\to\omega p$ differential cross sections, all events were
divided into 20 incident-photon energy bins from the reaction threshold to
$E_{\gamma}=1402$ MeV. The data within each energy bin were divided into 15
identical $\cos\theta$ bins, covering the full range from -1 to 1, where
$\theta$ was the angle between the directions of the outgoing $\pi^{0}\gamma$
system and the incident photon in the c.m. frame. The number of
$\omega\to\pi^{0}\gamma$ decays observed in each energy-angle bin was
determined by an individual fit of the corresponding $m(\pi^{0}\gamma)$
spectra.

Figure 2:  Same as Fig. 1, but for $\cos\theta=0.933$. A Gaussian fit to the
$\omega$ peak results in $\sigma=12.4$ MeV (FWHM=29.2 MeV). A polynomial of
order seven used to fit the background.  Figure 3:  Same as Fig. 1, but for
$E_{\gamma}=1325$ MeV. A Gaussian fit to the $\omega$ peak results in
$\sigma=13.6$ MeV (FWHM=32.0 MeV). A polynomial of order eight used to fit the
background.

The fitting procedure for one energy-angle bin is illustrated in Figs. 1-3,
showing typical changes in the shape of the $\omega$ peak and the background
depending on $\theta$ and $E_{\gamma}$. Panels (a) in these figures depict the
$m(\pi^{0}\gamma)$ distributions for the MC simulation of $\gamma p\to\omega
p\to\pi^{0}\gamma p$ fitted with a Gaussian. The choice of the normal
distribution for the fitting procedure is motivated by the facts that the BW
shape of the $\omega$ peak is severely cut by phase space near threshold and
the $m(\pi^{0}\gamma)$ resolution strongly dominates the $\omega$-meson width
($\Gamma=8.49$ MeV PDG ), especially for the very forward production angles of
$\omega$. Because the outgoing proton is not detected for those angles, it is
treated as a missing particle in the kinematic fit, resulting in a poorer
$m(\pi^{0}\gamma)$ resolution for the three-cluster events. In the case of the
Gaussian fits shown in panels (a), the full width at half maximum (FWHM)
increases from 19.3 MeV for $E_{\gamma}=1176$ MeV and $\cos\theta=0$ to 29.2
MeV for $\cos\theta=0.93$ at the same energy, and to 32.0 MeV for
$E_{\gamma}=1325$ MeV and the same angular bin.

Panels (b) in these three figures show the $m(\pi^{0}\gamma)$ distributions
for the MC simulation of $\gamma p\to\pi^{0}\pi^{0}p$, which is one of the
background processes. The reaction itself was generated as $\gamma
p\to\pi^{0}\Delta^{+}(1232)\to\pi^{0}\pi^{0}p$. The production angular
distribution of $\gamma p\to\pi^{0}\Delta$ and the rest-frame
$\Delta^{+}(1232)\to\pi^{0}p$ decay distribution with respect to the
$\Delta$’s directions were also generated isotropically. A more precise
generation of the reaction kinematics were unnecessary as these spectra were
used only for obtaining initial values for parameters of the function
describing the experimental background.

To separate the $\omega$ signal from the background, the experimental
$m(\pi^{0}\gamma)$ distributions were fitted with the sum of a Gaussian,
describing the $\omega$ peak, and a polynomial, describing the background.
These fits are shown in panels (c). The order of the polynomial was chosen to
be sufficient for a fairly good description of the background distribution in
the range $m(\pi^{0}\gamma)>0.4$ GeV/$c^{2}$. Typically, there was no need to
use a polynomial higher than order eight. The initial values for the
polynomial coefficients were taken equal to the output parameters of the
polynomial fit to the MC simulation of $\gamma p\to\pi^{0}\pi^{0}p$. In the
fits to the experimental spectra, the centroid and width of the Gaussian were
fixed to the values obtained from the previous fits to the MC simulation for
$\gamma p\to\omega p\to\pi^{0}\gamma p$, shown in panels (a). This introduced
additional restrictions on the background function, improving the separation
of the signal and background events. As also seen in Figs. 1-3, the centroid
and width of the Gaussian obtained from fitting to the MC simulation of
$\gamma p\to\omega p\to\pi^{0}\gamma p$ suit the experimental $\omega$ peak
very well. This confirms the agreement of the energy calibration and the
detector resolution for the experimental data and the MC simulation.

The number of $\omega\to\pi^{0}\gamma$ decays in the experimental
$m(\pi^{0}\gamma)$ spectra was determined from the area under the Gaussian.
For consistency, the detection efficiency in each energy-angle bin was
obtained in the same way, i.e., based on a Gaussian fit to the MC simulation
for $\gamma p\to\omega p\to\pi^{0}\gamma p$, instead of using the number of
entries in the $m(\pi^{0}\gamma)$ spectra. This approach allowed one to
diminish the impact from some deviation of the $\omega$-peak shape from the
normal distribution used in the fits.

##  IV Experimental results

The total number of $\omega$ mesons produced in each energy-angle bin was
obtained by correcting the number of $\omega\to\pi^{0}\gamma$ decays observed
with the corresponding detection efficiency and the $\omega\to\pi^{0}\gamma$
branching ratio (8.28%) from the RPP PDG . The $\gamma p\to\omega p$
differential cross sections were obtained by taking into account the number of
protons in the target and the photon-beam flux from the tagging facility.
Statistical uncertainties of the results were calculated from the errors given
by the Gaussian fits. One contribution to the systematic uncertainty comes
from the uncertainty in the shape of the background under the $\omega$ peak
and from the deviation of the $\omega$-peak shape from the normal
distribution. This uncertainty for one energy-angle bin is independent of such
uncertainties for other energy-angle bins. The magnitude of this uncertainty,
the average of which comprises 6% of individual values in the differential
cross sections, was estimated by fulfilling several tests with the
experimental and the MC-simulation spectra. For the experimental data, the fit
was repeated with lowering the order of the polynomial used in the fitting
procedure and changing the $m(\pi^{0}\gamma)$ range fitted. For tests with the
MC simulation, the experimental distribution was replaced with the sum of the
MC simulations for $\gamma p\to\omega p\to\pi^{0}\gamma p$ and $\gamma
p\to\pi^{0}\pi^{0}p$ and fitted as experimental data, checking how well the
known number of $\omega$ events was recovered. Another contribution to the
systematic uncertainty, which is practically the same for all energy-angle
bins, comes from the determination of the detector acceptance and the photon-
beam flux; it was estimated as 5% (see Ref. etamamic  for more details on this
kind of systematic uncertainty).

Figure 4:  (Color online) Differential cross sections for $\gamma p\to\omega
p$ as a function of $\cos\theta$, where $\theta$ is the angle between the
directions of the $\omega$ meson and the incident photon in the c.m. frame.
The results of this work are shown by blue dots, SAPHIR SAPHIR  results by
open triangles, and CLAS CLAS1  results by open circles. The vertical error
bars of all data points include statistical uncertainties only. The horizontal
error bars are shown only for SAPHIR data, reflecting their nonidentical bins
in $\cos\theta$. The incident-photon energies of the data points shown for
previous experiments are within $\pm$5 MeV of $E_{\gamma}$ indicated in each
panel. Model calculations from Ref. Beijing  are shown by black short-dashed
lines and from Ref. Giessen  by magenta long-dashed lines.

In Fig. 4, the results of this work for the $\gamma p\to\omega p$ differential
cross sections are compared to previous measurements at similar energies from
SAPHIR SAPHIR  and CLAS CLAS1  and to model calculations from Refs. Beijing ;
Giessen . The vertical error bars shown for all data points in Fig. 4 include
statistical uncertainties only. As seen, the data points of this work cover
the full production-angle range and have quite small statistical
uncertainties. Thus, combining these features of the present results with
fairly small energy binning (15 MeV in $E_{\gamma}$) makes it possible to
study the threshold-region dynamics with much better accuracy than before. As
also seen, the present results are in general agreement with previous
measurements in the angular range where they overlap. Deviations in absolute
values at some energies from the results of CLAS, with general agreement in
the angular dependence at the same time, can be explained by a slight
difference in $E_{\gamma}$, which can be very important near threshold. The
results from SAPHIR for backward angles are smaller than the present and other
recent measurements.

The impact of the SAPHIR data on the models can be seen in the calculations
from the Giessen group Giessen , the predictions of which for backward angles
follow the behavior of the SAPHIR results. The quark-model calculations from
Ref. Beijing  are in good agreement with the present data at higher energies.
At lower energies, the predictions from Ref. Beijing  are larger than
experiment, especially for backward angles. Because the near-threshold
amplitudes of this quark model are dominated by the under-threshold states
$N(1720)3/2^{+}$ and $N(1680)5/2^{+}$, the comparison with the present data
indicates that the contributions of those two states could be overestimated in
Ref. Beijing .

Near-threshold cross sections of good accuracy allow the extraction of various
useful parameters, including resonance masses as well (see, for instance,
Refs. OPAL ; DELPHI ; EtaMassA2 ). In general, the total cross section for an
inelastic reaction $a~{}b\to c~{}d$ with the particle masses
$m(a)+m(b)<m(c)+m(d)$ can be written as $\sigma_{t}=(q/W)\cdot F(W^{2})$,
where $W$ is the c.m. energy and $q$ is the c.m. momentum of the final-state
particles. The factor $F(W^{2})$, not vanishing at threshold, comes from the
sum of production amplitudes squared, and $(q/W)$ from the integration over
the final-state phase space. Because $W^{2}$ is linearly related to
$E_{\gamma}$ for meson photoproduction, the value $\sigma_{t}^{2}$ as a
function of $E_{\gamma}$ reaches zero at the threshold energy
$E_{\gamma}=E_{\gamma}^{\mathrm{th}}$ without any singularity (i.e., linearly,
if the final-state $S$ wave does not vanish at threshold).

The results of this work for $\sigma_{t}^{2}(\gamma p\to\omega p)$ are shown
as a function of $E_{\gamma}$ in Fig. 5(a). In the same figure, the present
results are also compared to model calculations from Refs. Beijing ; Giessen ;
Paris  and to the results from SAPHIR SAPHIR , the angular coverage of which
(see the horizontal error bars in Fig. 4) was almost full, allowing one to
extrapolate the differential cross sections to the full range. The fit of the
present $\sigma_{t}^{2}$ data with the formula

$\sigma_{t}^{2}(E_{\gamma})=b_{1}\delta+b_{2}\delta^{2}+b_{3}\delta^{3}\,,$
(1)

with one of four free parameters included in
$\delta=E_{\gamma}-E_{\gamma}^{\mathrm{th}}$, is shown in Fig. 5(a) by a solid
red line. For the parameter $E_{\gamma}^{\mathrm{th}}$, the fit results in the
value $(1109.90\pm 0.82)$ MeV, corresponding to the mass
$m_{\omega}=(783.10\pm 0.44)$ MeV/$c^{2}$. It is in good agreement with the
RPP value $m_{\omega}=(782.65\pm 0.12)$ MeV/$c^{2}$ PDG . Although the
estimate made here for the $\omega$-meson mass cannot compete in precision
with the known RPP value, the agreement observed indicates the good quality of
the present data and the correctness of the photon-beam energy calibration,
the systematic uncertainty in which was determined as 0.5 MeV TAGGER2 .

Figure 5: (Color online) Results of this work (blue dots) for the $\gamma
p\to\omega p$ total cross sections $\sigma_{t}$ are shown in (a) for
$\sigma_{t}^{2}$ as a function of the incident-photon energy $E_{\gamma}$ and
in (b) for $\sigma_{t}$ as a function of the c.m. momentum $q$ of the final-
state particles. The $\sigma_{t}$ results from SAPHIR SAPHIR  are depicted by
open triangles. The vertical error bars represent the total uncertainties of
the results. The horizontal error bars reflect the energy binning. The red
solid line shows the fit of the present data (a) with Eq.(1) and (b) with
Eq.(2). The result from the calculation of Ref. Beijing  is shown by a black
short-dashed line, of Ref. Giessen  by a magenta long-dashed line, and of Ref.
Paris  by a green dashed line.

More traditionally, the $\sigma_{t}$ behavior of a binary inelastic reaction
near threshold can be described as a series of odd powers of $q$. The results
of this work for $\sigma_{t}(q)$ are shown in Fig. 5(b). In the energy range
under the study, the formula

$\sigma_{t}(q)=a_{1}q+a_{3}q^{3}+a_{5}q^{5}$ (2)

is enough to describe well the present results for $\sigma_{t}(q)$. The fit of
these data with Eq.(2) is shown in Fig. 5(b) by a solid red line, resulting in
$a_{1}=(4.42\pm 0.14)\cdot 10^{-2}~{}\mu b$/(MeV/$c$), $a_{3}=-(1.62\pm
0.40)\cdot 10^{-7}\mu b$/(MeV/$c$)3, and $a_{5}=-(1.14\pm 2.56)\cdot
10^{-13}\mu b$/(MeV/$c$)5. The linear term is determined here by the $S$ waves
only (with the total spin 1/2 and/or 3/2), while the contributions to the
cubic term come from both the $P$-wave amplitudes and the $W$ dependence of
the $S$-wave amplitudes, and the fifth-order term arises from the $D$ waves
and the $W$ dependences of the $S$ and the $P$ waves.

The $\sigma_{t}(\gamma p\to\omega p)$ data near threshold can also be used for
determining the $\omega N$ scattering length $\alpha_{\omega p}$, defined by
the threshold relation $d\sigma(\omega p\to\omega
p)/d\Omega|_{\mathrm{th}}=|\alpha_{\omega p}|^{2}$ (in reality, it is a
combination of two independent $S$-wave scattering lengths with total spins
1/2 and 3/2). In the VMD framework, $\alpha_{\omega p}$ appears also in
$\sigma_{t}(\gamma p\to\omega p)$ near threshold Titov

$\sigma_{t}(\gamma p\to\omega
p)|_{\mathrm{th}}=\frac{q}{k}\cdot\frac{4\alpha\pi^{2}}{\gamma^{2}}\cdot|\alpha_{\omega
p}|^{2}\,\,,$ (3)

where $k$ is the c.m. momentum of the incident photon at the $\gamma
p\to\omega p$ threshold, $\alpha$ is the fine-structure constant, and
$\gamma=8.53\pm 0.14$ is the $\gamma-\omega$ coupling, as determined from the
$\omega\to e^{+}e^{-}$ decay width PDG . Combining Eq. (3) with the $a_{1}$
value from fitting Eq. (2) to the present $\sigma_{t}(\gamma p\to\omega p)$
data results in

$|\alpha_{\omega p}|=\frac{\gamma}{2\pi}\sqrt{\frac{ka_{1}}{\alpha}}=(0.82\pm
0.03)~{}\mathrm{fm}~{},$ (4)

which should be considered just as an estimate assuming only the sequence
$\gamma\to\omega,~{}\omega p\to\omega p$. A more detailed analysis is needed,
however, to exclude contributions from
$\gamma\to\rho^{0},~{}\rho^{0}p\to\omega p$, containing in particular
$\pi^{0}$ exchange, and from a similar transition $\gamma\to\phi$. Note that
the present estimate for $|\alpha_{\omega p}|$ is within the range defined by
other $\alpha_{\omega p}$ values available in the literature:
$(-0.026+i~{}0.28)$ fm from the coupled-channel analysis of the $\omega$
production in $\pi$N and $\gamma$N interactions Giessen , $(-0.41\pm 0.05)$ fm
from the QCD sum-rule analysis KoHa , $(1.6+i~{}0.30)$ fm from the effective
Lagrangian approach based on chiral symmetry Weise , and $(-0.44+i~{}0.20)$ fm
from the coupled-channel unitary approach Lutz . The dynamical coupled-channel
analysis from Ref. Paris  yielded separate values for two scattering lengths:
$\alpha_{\omega N}^{1/2}=(-0.0454-i~{}0.0695)$ fm and $\alpha_{\omega
N}^{3/2}=(0.180-i~{}0.0597)$ fm, which need to be specially combined for
comparing with the effective scattering length obtained directly from the
present data.

Existing experimental results for the imaginary Kotulla  and the real
Friedrich  parts of the $\omega$-nucleus potential were recalculated by one of
the present authors (V. M.) into the $\omega N$ scattering length, using the
connection of the meson-nucleus optical potential to the meson-nucleon
scattering amplitude described in Ref. Friedman  (details of these
calculations are beyond the scope of the present work and will be published
separately). The obtained value, $\alpha_{\omega N}=(-0.17\pm
0.40)+i~{}(0.79\pm 0.11)$ fm, has modulus $|\alpha_{\omega N}|=(0.81\pm 0.41)$
fm that turned out to be in agreement, within the uncertainties, with the
estimate made in this work.

Availability of good-quality data on $\omega$ photoproduction near threshold
will also allow further analysis of these data for extracting contributions
from the pion-exchange and the Born nucleon diagrams. The latter diagrams
contain the coupling vertex $\omega NN$, which determines the
$\omega$-exchange contribution to $NN$ forces and was extracted earlier in the
phenomenological analysis of Ref. Ted . Thus, one can check the current
understanding of the $NN$ potential. Threshold data are important here
because, at higher energies, the Born contributions decrease and become
nonessential with respect to the Pomeron-exchange contribution.

##  V Summary and conclusions

An experimental study of $\omega$ photoproduction on the proton was conducted
by the A2 Collaboration at MAMI. The $\gamma p\to\omega p$ differential cross
sections are measured from threshold to $E_{\gamma}=1.4$ GeV with 15-MeV
binning and full production-angle coverage, improving significantly the data
available for this energy range. The quality of the present data near
threshold gives access to a variety of physical quantities that can be
extracted by studying the $\omega N$ system. In particular, our estimate for
the $\omega N$ scattering length is consistent with previous theoretical
results, and the estimate of the $\omega$-meson mass is in good agreement with
the RPP PDG  value. The present data are also expected to be invaluable for
future partial-wave and coupled-channel analyses, which could provide much
stronger constraints on the properties of nucleon states known in this energy
range and even reveal new resonances. A partial-wave analysis with extracting
spin-density matrix elements from the present data is already in progress,
being performed by a theoretical group outside the A2 Collaboration. The
results of it will be published later on separately.

## Acknowledgments

We thank T. Barnes, A. B. Gridnev, A. I. Titov, and Q. Zhao for useful remarks
and continuous interest in the paper. The authors wish to acknowledge the
excellent support of the accelerator group and operators of MAMI. This work
was supported by the Deutsche Forschungsgemeinschaft (SFB443, SFB/TR16, and
SFB1044), DFG-RFBR (Grant No. 09-02-91330), the European Community-Research
Infrastructure Activity under the FP6 “Structuring the European Research Area”
programme (Hadron Physics, Contract No. RII3-CT-2004-506078), Schweizerischer
Nationalfonds, the UK Science and Technology Facilities Council (STFC 57071/1,
50727/1), the U.S. Department of Energy (Offices of Science and Nuclear
Physics, Award Numbers DE-FG02-99-ER41110, DE-FG02-88ER40415, DE-
FG02-01-ER41194) and National Science Foundation (Grant No. PHY-1039130,
IIA-1358175), INFN (Italy), and NSERC (Canada). Ya. I. Azimov acknowledges
support by the Russian Science Foundation (Grant No. 14-22-00281). We thank
the undergraduate students of Mount Allison University and The George
Washington University for their assistance.

## References

  * (1) S. Capstick and W. Roberts, Prog. Part. Nucl. Phys. 45, S241 (2000).
  * (2) K. A. Olive et al., (Particle Data Group), Chin. Phys. C 38, 090001 (2014).
  * (3) R. A. Arndt, W. J. Briscoe, I. I. Strakovsky, and R. L. Workman, Phys. Rev. C 74, 045205 (2006).
  * (4) R. Koniuk and N. Isgur, Phys. Rev. Lett. 44, 845 (1980).
  * (5) N. M. Kroll, T. D. Lee, and B. Zumino, Phys. Rev. 157, 1376 (1967); J. J. Sakurai, Phys. Rev. Lett. 22, 981 (1969).
  * (6) J. Barth et al., Eur. Phys. J. A 18, 117 (2003).
  * (7) Q. Zhao, Phys. Rev. C 63, 025203 (2001); private communication, 2013.
  * (8) Y. Oh, A. Titov, and T. S. H. Lee, Phys. Rev. C 63, 025201 (2001).
  * (9) H. Babacan, T. Babacan, A. Gokalp, and O. Yilmaz, Eur. Phys. J. A 13, 355 (2002).
  * (10) A. I. Titov and T. S. H. Lee, Phys. Rev. C 66, 015204 (2002).
  * (11) V. Shklyar, H. Lenske, U. Mosel, and G. Penner, Phys. Rev. C 71, 055206 (2005); V. Shklyar, private communication, 2013.
  * (12) M. W. Paris, Phys. Rev. C 79, 025208 (2009).
  * (13) M. Williams et al., Phys. Rev. C 80, 065208 (2009).
  * (14) A. Willson, Ph.D. Thesis, Florida State Univ. 2013.
  * (15) A. Starostin et al., Phys. Rev. C 64, 055205 (2001).
  * (16) R. Novotny, IEEE Trans. Nucl. Sci. 38, 379 (1991).
  * (17) A. R. Gabler et al., Nucl. Instrum. Methods A 346, 168 (1994).
  * (18) H. Herminghaus et al., IEEE Trans. Nucl. Sci. 30, 3274 (1983).
  * (19) K.-H. Kaiser et al., Nucl. Instrum. Methods A 593, 159 (2008).
  * (20) I. Anthony et al., Nucl. Instrum. Methods A 310, 230 (1991).
  * (21) S. J. Hall et al., Nucl. Instrum. Methods A 368, 698 (1996).
  * (22) J. C. McGeorge et al., Eur. Phys. J. A 37, 129 (2008).
  * (23) E. F. McNicoll et al., Phys. Rev. C 82, 035208 (2010).
  * (24) S. Prakhov et al., Phys. Rev. C 79, 035204 (2009).
  * (25) K. Ackerstaff et al., Phys. Lett. B 389, 416 (1996).
  * (26) J. Abdallah et al., Eur. Phys. J. C 34, 127 (2004).
  * (27) A. Nikolaev et al., Eur. Phys. J. A 50, 58 (2014).
  * (28) A. I. Titov, T. Nakano, S. Date, and Y. Ohashi, Phys. Rev. C 76, 048202 (2007).
  * (29) Y. Koike and A. Hayashigaki, Prog. Theor. Phys. 98, 631 (1997).
  * (30) F. Klingl, T. Waas, and W. Weise, Nucl. Phys. A 650, 299 (1999).
  * (31) M. F. M. Lutz, G. Wolf, and B. Friman, Nucl. Phys. A 706, 431 (2002); Erratum, 765, 495 (2006).
  * (32) M. Kotulla et al., Phys. Rev. Lett. 100, 192302 (2008).
  * (33) S. Friedrich et al., Phys. Lett. B 736, 26 (2014).
  * (34) E. Friedman and A. Gal, Phys. Rep. 452 89 (2007).
  * (35) C. Downum, T. Barnes, J. R. Stone, and E. S. Swanson, Phys. Lett. B 638, 455 (2006).