# Ball-AR: Fostering Playful Co-Located Interaction Through Environment-
centric Physical Activity with AR

Arnav Kumar  Princeton UniversityPrinceton, NJUSA and  Andrés Monroy-Hernández
Princeton UniversityPrinceton, NJUSA

(2023)

###### Abstract.

We present Ball-AR, an augmented reality (AR) game where two players in the
same physical space attempt to hit each other with virtual dodgeballs overlaid
on the physical world. Researchers have studied AR’s potential for fostering
co-located interaction and physical activity; however, they have not
investigated the impacts of physical activity and physical environment on user
experiences and interaction. We created an AR dodgeball game centered around
encouraging physical activity and harnessing the physical environment. We then
evaluated the game with five dyads to analyze the impacts of these design
choices on the quality of gameplay and interaction between players. We found
that physical activity and the shared physical space created memorable
experiences and interactions among participants, although participants desired
a more augmented and immersive experience.

augmented reality, physical activity, physical environment, co-located,
interaction, design

††copyright: acmcopyright††journalyear: 2023††doi: XXXXXXX.XXXXXXX††ccs:
Human-centered computing Collaborative and social computing devices

Figure 1. The Ball-AR system in use. Ball-AR requires two players in the same
physical space, and utilizes surface detection and body tracking for ball
movement.

The Ball-AR system in use. Ball-AR requires two players in the same physical
space, and utilizes surface detection and body tracking for ball movement.

##  1\. Introduction

In this paper, we discuss mobile AR game design by creating Ball-AR, an AR
game inspired by dodgeball. Specifically, we study the impacts of game design
choices based on physical and social areas that we believe AR can influence
positively.

Our first motivation is to promote physical activity and encourage movement.
As a result of the COVID-19 pandemic, physical activity levels have decreased
an estimated 41% (Wilke et al., 2021). Conversely, in this same time frame, we
saw a 75% increase in gaming activity (King et al., 2020). These behavior
shifts motivated the idea to study how a game can encourage physical activity.
In addition, AR games allow the ability to move while playing, so a mobile AR
game provides a great opportunity to encourage physical activity.

The second motivation is to foster co-located interaction. Recent research has
shown that mobile devices can negatively impact communication and experiences
between people in the same space (Przybylski and Weinstein, 2013a). However,
mobile AR games can be beneficial in fostering co-located interaction since
they can facilitate communication and create fun, memorable experiences (Dagan
et al., 2022). Another benefit AR can have on co-located interaction is its
usage of the physical environment, as research has shown that the shared
physical features of an environment can induce interaction between individuals
by interrelating them through a shared space (Amran and Fuad, 2020).

We envision using AR to support physical activity in shared physical spaces.
To do so, we created Ball-AR, an AR game that builds on and extends previously
researched design choices. We foster movement by requiring players to move
quickly, encourage social interaction by requiring sharing a physical
location, and connect players physically by utilizing the physical features of
the common environment.

We evaluated how well Ball-AR deals with fostering movement, encouraging
social interaction, and utilizing the physical environment. We also study how
the design choices affect participants’ enjoyment of the game and their
interaction with their opponents.

We learned that the combination of a shared physical space and physical
movement created enjoyable interactions and that competition, hidden
information, and the physical elements of AR were effective in increasing
communication and physical activity. However, participants did not feel
connected to each other through the shared physical environment’s features and
desired more augmentation rather than utilization of just the physical
environment.

##  2\. Related Work

Recent research has shown that people are increasingly lonely and craving
interaction (Weissbourd et al., 2021). Mobile technology may add to this
disconnect, as it can negatively affect communication and interactions in the
same room (Przybylski and Weinstein, 2013b). In addition, people have become
increasingly sedentary (Wilke et al., 2021). Unfortunately, mobile technology
may also influence this, as studies have shown negative correlations between
smartphone usage and physical activity levels(School of Physical Education and
Sports, Van Yüzüncü Yıl University, Van, Turkey and Boz, 2020).

With an estimated 5 billion people owning a mobile device, it becomes
important to use these devices to encourage interaction and physical activity
(Perrin, 2021). As a result, more and more mobile technologies have emerged to
do so. Regarding fostering social interaction, video communication platforms
like Zoom and Twitch have decreased loneliness-related outcomes, and improved
social interactivity (Kaveladze et al., 2022). For physical activity, wearable
technologies such as smartwatches or smartphones with mobile apps have
positively impacted physical activity and weight control (Yen and Chiu, 2019).
However, not enough mobile technologies support both co-located interaction
and physical activity.

To build our system, we focus on using AR, due to its known affordances of
social interaction and physical activity. Previous works have shown that AR
can foster co-located interaction due to its usage of shared spaces and
embodied interaction. One previous work, Project IRL studied the influences of
different design choices and features of mobile AR games on co-located
interaction, some of which informed the design of Ball-AR (Dagan et al.,
2022). Another work, BragFish, studied specifically the impacts of shared
virtual spaces on the physical and social interactions among players (Xu et
al., 2008). Both studies noted the potential benefits of AR games in fostering
co-located interaction. However, these works did not center on combining
physical activity and co-location in AR.

In addition, other works have shown the potential for AR to encourage physical
activity. The mixed reality game VRabl immersed players in a virtual space and
explored the impacts of different design features on inducing physical
activity (Buckers et al., 2018). Game design choices from this study were used
in Ball-AR. However, this study similarly did not research how inducing
physical activity related to the quality of gameplay or interaction.

As a result, we study game features that encourage co-located interaction and
physical activity with the goal of providing insight for future mobile AR game
design. To do so, we design a mobile AR game, Ball-AR, and analyze how AR’s
usage of the shared physical space and embodied interaction can benefit co-
located experiences for players.

##  3\. Ball-AR System

Figure 2. Description of different features within Ball-AR. Many features were
designed to encourage movement or environment utilization.

Ball-AR is a multiplayer AR dodgeball game designed to be played by users in a
shared physical space. To start the game, both players load the game on their
mobile phones and stand in opposite corners of the room.

Players begin by first scanning the surfaces in the shared physical space.
This entails walking around the room and pointing their mobile device at all
physical surfaces, which are then captured by the lens. After collecting the
environment, each user stands in the corner of the room, which is then deemed
the starting spot. From this point on, each user now can throw their augmented
ball at their opponent by tapping the screen. The ball can bounce off of any
of the surfaces before hitting the opponent. If the ball hits three surfaces
without hitting the opponent, the player must collect the ball to throw it
again. If the ball hits the opponent without touching a surface, the player
gets one point. In addition, the player gets an extra point for each surface
hit before hitting the opponent (with a maximum of three surfaces). For
example, if the player hits two walls and then their opponent, their score
increases by three. The first player to reach ten points wins.

The game is designed to encourage physical activity and utilization of the
shared physical environment to study impacts on interaction. For example,
since each user only has one ball, they must physically move around the room
in order to collect it. The aspect of competition adds a sense of urgency to
the game, imploring users to rush to collect their ball for a better chance at
victory. In addition, the extra points awarded based on surfaces hit before
hitting an opponent are included to ground the users in the shared physical
space and to focus on the physical features of the environment.

Ball-AR was designed with Lens Studio and is accessible via the Snapchat app.
More specifically, the World Mesh and Body Tracking features within Lens
Studio proved very useful in scanning a physical environment’s surfaces and
determining when the ball was hitting a player.

##  4\. Evaluation

We recruited ten university students through email to play Ball-AR in pairs.
We then brought the participants to a predetermined location with ample
surfaces and no major obstructions. Each participant played a best-of-three
match of Ball-AR against another participant. Then, each participant was
interviewed individually. We conducted semi-structured interviews (see
protocol in Appendix) with participants and focused on concepts related to
overall enjoyment, physical activity, environment utilization, and quality of
interaction. The interview included questions such as ”How did physical
activity affect your overall experience?” and ”Did you make use of the ability
to bounce the ball off physical surfaces? What were your opinions on this
feature?”.

##  5\. Study Results

###  5.1. Participants felt shared physical space and movement fostered
greater interactions, though some desired a single-player experience as well

Many participants felt that being in the same room as their opponent created a
more fun experience. ”It was cool to be in the same space and it made the game
more fun to play by being in the presence of the person you’re playing
against” (P6). In addition, the aspect of both players physically running
around in the same room created enjoyable interactions between players. “The
game was kind of a frenzy because I was just running around because I didn’t
want to get hit. [P7] started laughing, probably because I was so hard to
actually hit” (P8). However, many participants voiced a desire to have a
single-player option as well to be able to play even when alone. ”Having a
game with both single-player and multiplayer would be nice. Multiplayer caters
more to playing with friends but single-player could make more usable as a
mobile game and for exercise by myself” (P3). While participants enjoyed the
physical interactions with their opponents and the shared physical space, many
still viewed mobile games as a single-player experience.

###  5.2. Hidden information increased communication and activity for
participants

Within Ball-AR, players could not see each other’s scores or see the
opponent’s ball, although they were in the same room. As a result,
participants communicated throughout the game, shouting scores and exclaiming
when they hit the opponent (sometimes to trick their opponent as well). “I
feel like the part of not knowing what other people had enhanced the
experience because it made the game really funny. We both kept yelling random
numbers and I don’t think either of us were even close to what we said” (P4).
In addition, not being able to see the opponent’s ball encouraged greater
movement. ”I had to guess which way the ball was going, so I kept moving since
I figured the ball was aimed at where I was just standing” (P5). Hiding
details of the opponent from participants created a need for participants to
communicate amongst themselves and encouraged more movement.

###  5.3. The competitive nature of the game induced physical activity among
participants and encouraged future play

While there is no timer in the game, participants felt an implicit time due to
the need to score before their opponent to win. As a result, many participants
noted that the deeming of a winner and a loser at the end of a Ball-AR game
led to greater movement and physical activity: “I kept running after my ball
to throw it again. Otherwise, I probably would have lost” (P2). Other
participants expressed their desire to play again after losing.

###  5.4. Participants felt that the blend of mobile AR and physical activity
was fitting and memorable, though potentially dangerous

Participants expressed that AR made the game more enjoyable and felt that it
allowed for features normal mobile games could not. “I think AR should be
geared towards actual real movements, not just sitting and playing with your
finger. In the game, there is a physical element and it is like a sport” (P6).
In addition, the need for movement created a memorable experience within AR.
”A lot of AR is memorable but this one had movement involved and that made it
cooler than other AR simulations where you’re just looking at a virtual space”
(P10). However, some participants noted that the movement in the game and the
small screen could potentially be dangerous. ”Someone could definitely bump
into tables and chairs and stuff if they didn’t really know the room that
well. Normally, it’s easier not to, but since you’re running and looking at a
phone, I could see that happening” (P2).

###  5.5. Participants did not focus on the physical features of the
environment and desired more augmentation

The ability to throw the ball off physical surfaces was included to ground the
players in their physical environment and to foster belonging by connecting
them through the shared space. However, most participants noted that they were
more focused on other aspects of the game rather than the shared physical
space. ”I wasn’t really that aware of my surroundings because I was mostly
looking for my ball or looking at [P2] to hit [them]” (P1). Participants also
expressed that the main use of the feature was for strategy and scoring rather
than feeling connected to a certain physical space with their opponent. In
addition, participants actually requested greater augmentation to enhance the
physical environment. “It would be cool if there were more cool things to
throw off of in the room, like some virtual objects” (P7).

##  6\. Discussion

Through these results, we can theorize at good and bad design choices for AR
games, specifically when fostering physical activity and co-located
interaction.

For physical activity, we discovered that giving each player one ball to
follow around did induce movement. In addition, the competitive nature of the
game fostered quicker movement due to the desire to win. In general, there
seemed to be a sort of expectation among participants for AR games to expand
on the capabilities of non-AR mobile games by encouraging movement. Most
participants felt that the inclusion of physical movement and activity was
fitting for AR and took advantage of AR’s physical elements, hinting at the
possibility that physical movement of some kind belongs in mobile AR games,
although with more safety precautions.

In terms of co-located interaction, the use of hidden information fostered
interesting communications among participants. In addition, most participants
enjoyed being in the same physical space as their opponent as it allowed for
these communications and created more enjoyable physical and social
interactions. However, many participants had the belief that mobile games such
as this should have some sort of single-player capability. This may be due to
participants wanting to play on their own time without restriction or may be a
result of the abundance of single-player mobile games currently available.

We also noticed that physical activity and co-located interaction intersected
greatly within the game. The encouragement of physical activity created more
memorable experiences for users and fostered physical interaction.
Participants had fun both running and seeing their opponent run frantically,
and many recalled these moments afterwards as their favorite moments of the
game. These results describe a potential correlation between physical activity
in AR games and fun physical interaction between players that we encourage
future mobile AR game designers to consider.

Lastly, many participants expressed the desire for more augmentation and
greater immersion in the shared virtual space. This speaks to the potential
preference of users to escape or augment reality while playing an AR game,
rather than just interact with it. The features of the specific shared
physical space did not make a substantial impact on the interaction or
belonging of participants, and perhaps a greater virtual presence could create
a more memorable experience for users.

##  7\. Limitations

While we determined both benefits and challenges in mobile AR game design,
there were some limitations. Our findings come from a small, homogeneous group
of participants, as all are college students with some interest in physical
activity games. Future research should explore a broader group of
participants. Furthermore, all studies were conducted in the same physical
location. Future work should also experiment with different types of physical
locations to expand this research.

##  8\. Conclusion

In this paper, we presented Ball-AR, an AR game designed to foster physical
activity, co-located interaction, and physical environment utilization. We
designed the game using both techniques from previous studies and new designs
to encourage movement, increase communication, and ground users in their
shared physical space. We then evaluated the game with five dyads. From our
results, we learned that the combination of a shared physical space and
physical movement created enjoyable interactions and that certain design
features, such as competition, hidden information, and usage of AR were
effective in increasing communication and activity. However, participants
desired more augmentation and more usage of a virtual shared space.

Future works in designing co-located AR gaming experiences could expand on the
results shown in this paper. For example, future work could study more
specific types of physical activity or study the impacts of greater
augmentation on co-located interaction. Overall, we hope future researchers
continue to explore mobile AR games and their ability to positively impact
interaction and health of players.

## References

  * (1)
  * Amran and Fuad (2020) Muhammad Farhan Amran and Achmad Hery Fuad. 2020.  The effect of public spaces’ physical features on interaction between strangers. Case study: Jurangmangu transit space. Namakkal, India, 040011.  https://doi.org/10.1063/5.0002573
  * Buckers et al. (2018) Tim Buckers, Boning Gong, Elmar Eisemann, and Stephan Lukosch. 2018\.  VRabl: stimulating physical activities through a multiplayer augmented reality sports game. In _Proceedings of the First Superhuman Sports Design Challenge on First International Symposium on Amplifying Capabilities and Competing in Mixed Realities - SHS ’18_. ACM Press, Delft, Netherlands, 1–5.  https://doi.org/10.1145/3210299.3210300
  * Dagan et al. (2022) Ella Dagan, Ana María Cárdenas Gasca, Ava Robinson, Anwar Noriega, Yu Jiang Tham, Rajan Vaish, and Andrés Monroy-Hernández. 2022\.  Project IRL: Playful Co-Located Interactions with Mobile Augmented Reality.  _Proceedings of the ACM on Human-Computer Interaction_ 6, CSCW1 (March 2022), 1–27.  https://doi.org/10.1145/3512909
  * Kaveladze et al. (2022) Benjamin T. Kaveladze, Robert R. Morris, Rosa Victoria Dimitrova-Gammeltoft, Amit Goldenberg, James J. Gross, Judd Antin, Melissa Sandgren, and Melissa C. Thomas-Hunt. 2022.  Social Interactivity in Live Video Experiences Reduces Loneliness.  _Frontiers in Digital Health_ 4 (March 2022), 859849\.  https://doi.org/10.3389/fdgth.2022.859849
  * King et al. (2020) Daniel L. King, Paul H. Delfabbro, Joel Billieux, and Marc N. Potenza. 2020.  Problematic online gaming and the COVID-19 pandemic.  _Journal of Behavioral Addictions_ 9, 2 (June 2020), 184–186.  https://doi.org/10.1556/2006.2020.00016
  * Perrin (2021) Andrew Perrin. 2021\.  _Mobile Technology and Home Broadband 2021_.  Technical Report. Pew Research Center.  https://pewresearch-org-preprod.go-vip.co/internet/2021/06/03/mobile-technology-and-home-broadband-2021/
  * Przybylski and Weinstein (2013a) Andrew K. Przybylski and Netta Weinstein. 2013a.  Can you connect with me now? How the presence of mobile communication technology influences face-to-face conversation quality.  _Journal of Social and Personal Relationships_ 30, 3 (May 2013), 237–246.  https://doi.org/10.1177/0265407512453827
  * Przybylski and Weinstein (2013b) Andrew K. Przybylski and Netta Weinstein. 2013b.  Can you connect with me now? How the presence of mobile communication technology influences face-to-face conversation quality.  _Journal of Social and Personal Relationships_ 30, 3 (May 2013), 237–246.  https://doi.org/10.1177/0265407512453827
  * School of Physical Education and Sports, Van Yüzüncü Yıl University, Van, Turkey and Boz (2020) School of Physical Education and Sports, Van Yüzüncü Yıl University, Van, Turkey and Hilal Boz. 2020.  Effects of technological device usage time on body composition, physical activity level and physical activity participation motivation in young women.  _African Educational Research Journal_ 8, 2 (Aug. 2020), 322–328.  https://doi.org/10.30918/AERJ.82.20.075
  * Weissbourd et al. (2021) Richard Weissbourd, Milena Batanova, Virginia Lovison, and Eric Torres. 2021.  _Loneliness in America: How the Pandemic Has Deepened an Epidemic of Loneliness and What We Can Do About It_.  Technical Report. Harvard University.  https://mcc.gse.harvard.edu/reports/loneliness-in-america
  * Wilke et al. (2021) Jan Wilke, Lisa Mohr, Adam S. Tenforde, Pascal Edouard, Chiara Fossati, Marcela González-Gross, Celso Sánchez Ramírez, Fernando Laiño, Benedict Tan, Julian David Pillay, Fabio Pigozzi, David Jimenez-Pavon, Bernhard Novak, Johannes Jaunig, Mandy Zhang, Mireille van Poppel, Christoph Heidt, Steffen Willwacher, Gustavo Yuki, Daniel E. Lieberman, Lutz Vogt, Evert Verhagen, Luiz Hespanhol, and Karsten Hollander. 2021\.  A Pandemic within the Pandemic? Physical Activity Levels Substantially Decreased in Countries Affected by COVID-19.  _International Journal of Environmental Research and Public Health_ 18, 5 (Feb. 2021), 2235.  https://doi.org/10.3390/ijerph18052235
  * Xu et al. (2008) Yan Xu, Jay Bolter, Blair MacIntyre, Maribeth Gandy, Sami Deen, Brian Schrank, Kim Spreen, Michael Gorbsky, Timothy White, Evan Barba, and Iulian Radu. 2008.  BragFish: exploring physical and social interaction in co-located handheld augmented reality games. In _Proceedings of the 2008 International Conference in Advances on Computer Entertainment Technology - ACE ’08_. ACM Press, Yokohama, Japan, 276\.  https://doi.org/10.1145/1501750.1501816
  * Yen and Chiu (2019) Hsin‐Yen Yen and Huei‐Ling Chiu. 2019.  The effectiveness of wearable technologies as physical activity interventions in weight control: A systematic review and meta‐analysis of randomized controlled trials.  _Obesity Reviews_ 20, 10 (Oct. 2019), 1485–1493.  https://doi.org/10.1111/obr.12909

##  Appendix A Interview Protocol

Participants followed along the protocol shown below:

  1. (1)

Each of the 10 participants was asked to meet at a predetermined location,
where they were told about the study and shown the instructions to the game.

  2. (2)

Participants were separated into five groups of two. Each group played a best-
of-three match (first to two wins) of Ball-AR.

  3. (3)

After the match ended, each participant was individually interviewed. They
were asked the following questions:

     * •

What are your thoughts on the Ball-AR experience? Would you play again?

     * •

What was your favorite feature of the game? What was your least favorite
feature of the game?

     * •

Did you enjoy the aspect of playing against another person? Would you have
preferred a single-player game?

     * •

Did you like the competitiveness of the game? Would you have preferred a
collaborative game?

     * •

Did playing the game in the same room as your opponent make the experience
more enjoyable? Would you have preferred a virtual/remote version?

     * •

Did you move at all while playing Ball-AR? How did physical movement affect
your overall experience?

     * •

Did you make use of the ability to bounce the ball off physical surfaces? What
were your opinions on this feature?

     * •

Did the use of a mobile device make the game more enjoyable or less enjoyable?
Why?

     * •

Would you have preferred sharing a ball with your opponent rather than each
having your own?

     * •

Did you encounter any dangers while playing Ball-AR?

     * •

Do you have any general suggestions for Ball-AR and its next steps?


# Predicting User Perception of Move Brilliance in Chess

Kamron Zaidi and Michael Guerzhoy
University of Toronto
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

> AI research in chess has been primarily focused on producing stronger agents
> that can maximize the probability of winning. However, there is another
> aspect to chess that has largely gone unexamined: its aesthetic appeal.
> Specifically, there exists a category of chess moves called “brilliant”
> moves. These moves are appreciated and admired by players for their high
> intellectual aesthetics. We demonstrate the first system for classifying
> chess moves as brilliant. The system uses a neural network, using the output
> of a chess engine as well as features that describe the shape of the game
> tree. The system achieves an accuracy of 79% (with 50% base-rate), a PPV of
> 83%, and an NPV of 75%. We demonstrate that what humans perceive as
> “brilliant” moves is not merely the best possible move. We show that a move
> is more likely to be predicted as brilliant, all things being equal, if a
> weaker engine considers it lower-quality (for the same rating by a stronger
> engine). Our system opens the avenues for computer chess engines to (appear
> to) display human-like brilliance, and, hence, creativity.

## Introduction

Superhuman chess AI has made significant advances in recent decades. In this
paper, we focus on human-like play, and specifically play that is
aesthetically appealing to humans, in the sense that it’s perceived as
brilliant and/or creative111“Brilliant,” “aesthetically appealing,” and
“creative” can philosophically be treated as distinct. However, in the context
of chess, those are closely related.. Chess can be thought of as an
intellectual art, as it allows for the perception of beauty (?). Moves termed
as “brilliant” are highly regarded by players for a variety of factors
relating to the aesthetic appreciation of the move. The practice of awarding
brilliancy prizes at chess tournaments for the most beautiful moves/games
gives credence to the aesthetic nature of chess and the value placed on it by
chess players (?). Aesthetics also appears outside the context of full games,
such as in chess puzzles, where aesthetics is even more central to the
experience. When annotating games, annotators ordinarily annotate brilliant
moves using exclamation points. There is a limited amount of narrow
algorithmic methods for labeling moves as “brilliant.” For example, a popular
online chess platform, chess.com, classifies “good piece sacrifices” as
brilliant, which not only misses out on many other types of brilliant moves,
but may also be prone to falsely classifying a move as brilliant (?).

In this paper, we show that it is possible to use the shape of the game tree
and the output of chess engines to predict which moves humans would label as
brilliant. We collect annotation data from humans from lichess.org, extract
features from the local game tree and the predictions of chess engines, and
learn a neural-network-based classifier to use the features to predict whether
humans would perceive the move as brilliant. We report that we can obtain an
accuracy of 79% (with 50% base-rate), a PPV of 83%, and an NPV of 75% when
classifying moves as perceived as brilliant or not. Interestingly, our
analysis shows that our neural network predicts a move to be more brilliant
if, holding the rating by a stronger chess engine constant, the weaker chess
engine predicts the move to be less good, indicating that part of the
perception of brilliance lies in the move not just being good, but also being
non-obvious.

Our main intuition is that the game tree provides insight into whether the
move is brilliant: a game tree where there are many possibilities for winning
indicates that none of the moves would be considered particularly creative or
brilliant, whereas a game tree where there is only a single and long path to
victory may indicate that finding that path would be considered creative
and/or brilliant.

## Literature Review

### Artificial Intelligence in Chess

Currently one of the top chess engines is Leela Chess Zero (lc0) (?). This
engine uses a Monte Carlo tree search (MCTS) to expand the game tree. MCTS is
guided by a neural network, which takes in a game state as input and outputs a
vector of move probabilities and a scalar evaluation estimating the
probability of winning from the game state. The body of the neural network is
a residual tower with Squeeze-and-Excitation blocks (?). At the time of
writing this, lc0 has played over 1.6 billion games against itself, learning
through self-play (?).

Despite relatively limited research towards human-level play, there is one
computer chess engine developed for playing similar to humans: Maia (?). Maia
uses the same architecture as Leela Chess Zero, however instead of training on
self-play, the weights for the neural network guiding the tree search are
instead trained on human-played games within fixed bins of player skill
ratings. The authors found that Maia can predict the move played by a human
more accurately than lc0 and can also predict when a player may make a blunder
based on the game state (?).

While Maia does model human behaviour in chess more accurately than Stockfish
(?) or lc0, none of these engines explicitly consider the aesthetic
appreciation of chess. Despite the concept of aesthetics seeming vague and
challenging to assign a numerical value to, there has been work done towards
understanding chess aesthetics and developing a metric correlated to
aesthetics.

### Chess Aesthetics

Existing research in chess aesthetics shows the existence and importance of
aesthetics in chess (?) (?) (?) (?), mostly focusing on puzzles. Previous work
is either not directly operationalizable or proposes very simple heuristics.
In contrast, we propose analyzing the shape of the game tree, which is only
made possible through the combination of using a chess engine and a learned
model.

## Methods

In this section, we describe our system for classifying moves as brilliant or
not. We collect human-annotated data, extract features from the games, and
learn a model that predicts move brilliance.

### Data Collection

We collect a set of games annotated by users from lichess.org’s (?) Study
feature. The top 624 most popular studies from Lichess were obtained on
November 13, 2023. From these, 8574 total games were extracted. 820 moves were
labelled by users as brilliant (two exclamation points) across 556 games,
which were part of 158 studies. 1637 moves were labelled as “good” (one
exclamation point). 4518 moves were labelled as other (an annotation is that’s
not “bad” is present).

### Feature Engineering

We compute features from the output of two chess engines. We consider both the
evaluation of the moves by engines and the game tree that the engines
generate. The engines generate a game tree by exploring mostly promising
moves.

We use two engines: lc0 (?), a strong chess engine trained with self-play (the
largest network, T82-768x15x24h-swa-7464000 (?) is used); and Maia, an engine
trained to output human-like moves, with an ELO of about 1900 (?) (?) (which
is quite modest).

For each move, a total of 10 search trees were generated from the board before
the move (henceforth called the parent board): once for each of the two weight
options (self-play and Maia), and once for each number of nodes evaluated
($10^{1}$, $10^{2}$, $10^{3}$, $10^{4}$, and $10^{5}$).

For each tree generated, the following two features can be calculated: is the
move of interest in the tree, and if so, did the engine evaluate it as the
best move (the move with the highest win probability from the parent board).
These features, and specifically the evolution of these features as more nodes
are added, may give insight both into the strength of a move and the
difficulty of finding it. Both of these attributes reflect on the skill of the
player of the move, hence they may be important features for identifying
brilliant moves.

We also calculate features for each subtree. A subtree is defined as a tree
rooted at a node within the original tree. There are 6 types of subtrees that
may be informative for identifying brilliant moves. First, the tree rooted at
the parent board gives information about the player’s situation before the
move, and the possible moves that could have been made. Second, the tree
rooted at the node immediately after the move of interest gives information
about the situation following the move, which can give insight into how this
move affected the game’s trajectory. The remaining 4 types of subtrees are
increasing moves, advantage moves, decreasing moves and disadvantage moves.
These are defined based on either the move’s predicted win percentage
$Q_{move}$, and the difference between the subtree’s root predicted win
percentage and the win percentage of the move: $\Delta Q=Q_{move}-Q_{root}$.
Increasing moves have $\Delta Q>0$, advantage moves have $Q_{move}>0$,
decreasing moves have $\Delta Q\leq 0$, and disadvantage moves have
$Q_{move}\leq 0$.

Since the set of moves belonging to these four types is variable, the features
are aggregated so the number of features per full tree (and hence per move
that is being classified) is constant. To do this, features are calculated for
the subtrees rooted at each increasing, advantage, decreasing and disadvantage
move possible from the parent board, excluding the move of interest. The
features for each move in each type is aggregated using mean, standard
deviation, min and max. This yields $4\times 4\times N$ features per tree from
these four types, where N is the number of features per subtree. Combined with
the $2\times N$ features from subtrees rooted at the parent board and move of
interest, and the 2 binary features calculated per full tree, there are a
total of $18\times N+2$ features per full tree. The $N$ features per subtree
are as follows: Number of increasing, advantage, decreasing, and disadvantage
moves searched from the root node (4 features), predicted win chance at the
root node, maximum predicted win chance across all possible moves from the
root, probability of search and number of times the root node was searched (2
features), maximum search probability and number of times node was searched
across all possible moves from the root (2 features), branching factor of
subtree, width at layers 1-7 (7 features), mean, standard deviation and max
width across all layers (3 features), and height of the subtree. In total,
this gives $N=22$ features per subtree, or 398 features per tree. Each parent
board/move of interest pair in the dataset each had 10 trees generated to
cover the 5 node counts ($10^{1}$ to $10^{5}$), and the 2 weight options
(self-play and Maia). So, each datum has a total of 3980 features. Note, if
the tree didn’t contain the move of interest, or if there was no move within
one of the move type subsets, a constant vector for the 22 features was used
instead. The constant vector contained 0 for all features except for the win
chance and max win chance features, which used a value of -1 instead.

Figure 1: AggReduce architecture. Features for each tree generated with a specific size and weight are processed individually in the first hidden layer, then they are aggregated in tiers. |  |
---|---|---
|  |
Figure 2: Win chance using self-play weights (top) and Maia weights (bottom)
for each move type vs number of nodes searched.

### Classifier Models

A variety of classifier models were trained and tested on the data. This
included: logistic regression, a random forest classifier, Gaussian Naïve
Bayes, k-nearest neighbors regression, SVM classifier, a fully-connected
neural network (one hidden layer with 50 units) and 4 tiered neural networks.

The tiered feedforward neural network architectures operate on the basis that
for each input board, a search tree is generated once per weight and per tree
size. As there are two weight options and five tree size options, there are
ten trees generated per board. Instead of inputting the features for all ten
trees at once, tree features can be processed either individually, combined
across both weight options, or combined across all the tree sizes. Then,
subsequent hidden layers can combine features across the categories that were
not yet considered. Specifically, four tiered architectures were tested:
PerWeight, PerSize, PerWeightPerSize and AggReduce. PerWeight’s first hidden
layer takes as input the features across all 5 tree sizes for a specific
weight. The second hidden layer takes the first layer outputs for both
weights, followed by a single output node to classify the move. PerSize
follows a similar structure, but the first hidden layer instead takes as input
the features across both weights for a specific tree size, then the second
hidden layer takes as input the first layer outputs for all 5 tree sizes.
PerWeightPerSize adds an additional tier, such that the first hidden layer
takes as input the features for a tree generated using a specific weight and
tree size. Then, the second hidden layer takes as input the first layer
outputs across both weights (still for a specific tree size), and the third
hidden layer takes the second layer outputs across all 5 tree sizes.

AggReduce applies the tiered principle, while also addressing the imbalance of
the tree features. Recall that each tree has the following features: 2 binary
features (is the move of interest in the tree, and is it the best move), and
22 features for each of the 6 types of subtrees: parent, child, increasing,
advantage, decreasing and disadvantage. The two binary features, along with
the features for the parent and child subtrees, directly characterize the game
state before and after the move of interest. The features for the remaining 4
types of subtrees characterize hypothetical scenarios about what may occur if
a different move was taken instead of the move of interest. Since the parent
tree already contains information about these moves to some degree, the
information gained from these 4 types of subtrees may be less informative for
classifying the move. However, since each of the 4 types of subtrees are
aggregated using 4 separate techniques (min, max, mean, std), they account for
$22\times 4\times 4=352$ of the total tree features ($\bar{x}_{agg}$). The
parent and child subtrees are not aggregated so they each only contribute 22
features, which together with the binary features total to $22\times 2+2=46$
key features ($\bar{x}_{key}$). This architecture maps the aggregate features
to a smaller hidden layer, allowing the network to extract key information
from the aggregate features while focusing on the key features.

The first hidden layer takes as input all the features for a tree generated by
a specific weight and tree size; both $\bar{x}_{key}$ and $\bar{x}_{agg}$.
This layer has the capability to learn key information from $\bar{x}_{agg}$,
contextualized by $\bar{x}_{key}$. The second hidden layer takes the first
hidden layers from trees generated across both weight options (still for a
specific tree size), along with $\bar{x}_{key}$ for these two trees. Finally,
the third hidden layer takes the second hidden layers generated across all 5
tree sizes, followed by a single output node to classify the move. This
architecture is shown in Figure 1.

For all neural networks, Each linear layer except for the last applies a 20%
dropout to prevent overfitting and ReLU activation applied after, and the
output has a sigmoid activation applied to constrain the classification
between 0 and 1. The networks were trained using Adam optimizer, with a
learning rate of 1e-4, weight decay of 1e-5, and early stopping with a
patience of 10 epochs.

The data was split, withholding 10% of randomly sampled data for testing. The
training set contained 744 brilliant moves, 1463 good moves, and 1444 other
moves, while the test set contained 76 brilliant moves, 174 good moves, and
156 other moves. 5-fold cross-validation was used to train and validate each
classifier on the training data; the model hyperparameter and final selection
was done based on the class-balanced accuracy averaged across each fold.
Hyperparameters were selected via grid search.

To validate the significance of the tree features, logistic regression models
were trained on various subsets of the data. These subsets are as follows:

  * •

Self-play/Maia/Both Win Chance: Using only win chance evaluation before and
after the move (and the difference), evaluated with self-play/Maia/both
weights and $10^{5}$ nodes searched.

  * •

Self-play/Maia/Both Win Chance per Number of Nodes: Using the win chance
evaluation before and after the move, evaluated with self-play/Maia/both
weights, for $10^{1}$,$10^{2}$,$10^{3}$,$10^{4}$, and $10^{5}$ nodes searched.

  * •

Is Best Move: Using only the feature encoding if the move of interest is the
best move evaluated, evaluated for both weights and all 5 tree sizes (nodes
searched).

## Results and Discussion

### Data Analysis

To verify the utility and information gained by performing the tree search
with different nodes searched, the distribution of move win percentages across
different number of nodes searched using self-play weights is shown in Fig. 2

These figures show that brilliant moves tend to start off with lower win
chance predictions when only 10 nodes are searched, then rises as more nodes
are searched. Both good and other moves start off with stronger evaluations
with 10 nodes searched, but as more nodes are searched, the brilliant moves
peak higher than the non-brilliant moves with a tighter distribution skewed
towards a high win chance.

A similar pattern is observed using Maia weights. One key distinction is that
brilliant moves appear to be very negatively evaluated when just 10 nodes are
searched, which is also the case with good moves, though to less of an effect.
Even as more nodes are evaluated, the predicted win chance for brilliant moves
remains much lower than those using self-play weights. This supports the idea
that brilliant moves are difficult to find, since the human-level chess engine
fails to recognize the strength of the move when evaluating a small number of
nodes.

### Model Evaluation

Table 1 shows the class-balanced accuracies for each model, averaged across 5
folds.

Using only the win chance from the largest tree, the class-balanced accuracy
of the logistic regression model only reached 61%, showing that more features
are necessary to characterize brilliant moves more accurately. Varying the
number of nodes is informative for classifying brilliant moves; even if the
only feature is if the move of interest is the best move, by looking at
multiple tree sizes and weights the logistic regression model improved to 64%
accuracy. By using the win chance per number of nodes, the accuracy reaches a
maximum of 68%. Using all the features available, logistic regression reaches
an accuracy of 74%, showing that the full feature set contains additional
information useful for classification, aside from just win chance.

The fully connected neural network with one hidden layer of 50 units achieves
an average cross validation accuracy of 76%. Adding a hidden layer using the
tiered architecture, both PerWeight and PerSize neural networks achieve an
accuracy of 77%. Adding another hidden layer that first maps features from
each individual tree, PerWeightPerSize achieves an accuracy of 78%. Finally,
the AggReduce architecture performed the best, achieving an average cross
validation accuracy of 79% at 21 epochs on average. The hidden layer sizes are
25, 400, 50 for the first, second and third hidden layers respectively. This
model was evaluated on the test set to quantify the final accuracy, achieving
a test accuracy of 78.60%.

When classifying moves as brilliant or not, the AggReduce architecture obtains
a true positive rate of 71%, true negative rate of 85%, a positive predictive
value of 83% and a negative predictive value of 75%. In many potential
applications of this model, e.g. in automatic game labelling, it may be of
particular interest to ensure a move labelled as brilliant is truly brilliant
(positive predictive value), and to ensure a move that is truly non-brilliant
is labelled as such (true negative rate).

### Brilliant moves more often elude the weaker engine

Further testing was done to better understand the impact of Maia trees on the
neural network predictions. For each tree generated with Maia weights, the
predicted win chance at the post-move node was set to -1 and the model output
score was recalculated. On average, this caused an increase of 0.02554
($\sigma$ = 0.0471, n = 4057) to the classification score, with a p-value of
$<$0.001. This supports the notion that one of the characterising traits of
brilliant moves is the difficulty of understanding why they are strong, since
reducing Maia’s win chance prediction increased the prediction score of the
move being brilliant.

Classifier | Accuracy
---|---
LR (Self-play Win Chance) | 61%
LR (Maia Win Chance) | 58%
LR (Both Win Chance) | 61%
LR (Self-play Win Chance per # of Nodes) | 65%
LR (Maia Win Chance per # of Nodes) | 65%
LR (Both Win Chance per # of Nodes) | 68%
LR (Is Best Move) | 64%
LR (All Features) | 74%
Random Forest Classifier | 74%
Gaussian Naïve Bayes | 65%
K-NN Classifier | 66%
SVM Classifier | 73%
Fully-connected NN (50) | 76%
PerWeight NN (25, 200) | 77%
PerSize NN (100, 50) | 77%
PerWeightPerSize NN (100, 100, 25) | 78%
AggReduce NN (25, 400, 50) | 79%
Table 1: 5-fold average class-balanced accuracy for each classifier, using the
best hyperparameters discovered through cross validation. The best discovered
NN hidden layer sizes are displayed in brackets, in order.

## Conclusions and Future Work

In this work, we have shown that it is possible to predict human perception of
the brilliance of moves in chess. We have shown that game trees provide
information about the perceived brilliance over and above how strong the move
is, and that moves that are rated highly by both a weak and a strong engine
are predicted to be less brilliant.

We have studied perception of brilliance by anonymous Lichess users. Expert
perception of brilliance might differ from perceptions of Lichess users, and
experts and amateurs both can have different tastes within the groups as well.

One can attempt to generalize our work to the perception of brilliance in
other domains where creativity has tree-search-like aspects, such as
mathematics.

## Author Contributions

KZ and MG planned the study, designed the experiments, and performed the
analysis. KZ developed the implementation. MG helped develop the
implementation.

## References

  * [Chess.com 2024] Chess.com.  2024\.  How are moves classified? what is a ’Blunder’ or ’Brilliant’ and etc?  Accessed: Jan. 22, 2024. [Online]. Available: https://support.chess.com/article/2965-how-are-moves-classified-what-is-a-blunder-or-brilliant-and-etc.
  * [Humble 1993] Humble, P. N.  1993\.  Chess as an art form.  Brit J Aesthetics 33(1):59–66.
  * [Iqbal 2006] Iqbal, A.  2006\.  Computing the aesthetics of chess.  In AAAI Workshop on Computational Aesthetics.  AAAI Press Boston, MA.
  * [Leela 2024a] Leela.  2024a.  Leela Chess Zero.  Accessed: Jan. 22, 2024. [Online]. Available: https://lczero.org/.
  * [Leela 2024b] Leela.  2024b.  LeelaChessZero.  Accessed: Apr. 14, 2024. [Online]. Available: https://github.com/LeelaChessZero/lc0.
  * [Leela 2024c] Leela.  2024c.  Neural network topology - Leela Chess Zero.  Accessed: Jan. 22, 2024. [Online]. Available: https://lczero.org/dev/backend/nn/.
  * [Levitt and Friedgood 1995] Levitt, J., and Friedgood, D.  1995\.  Secrets of Spectacular Chess.  Batsford chess library. New York: Henry Holt, 1st american ed. in edition.
  * [Lichess 2024] Lichess.  2024\.  About lichess.org • lichess.org.  Accessed: Jan. 22, 2024. [Online]. Available: https://lichess.org/about.
  * [Maia 2024] Maia.  2024\.  Maia-chess/maia-weights at master · CSSLab/maia-chess.  GitHub. Accessed: Apr. 14, 2024. [Online]. Available: https://github.com/CSSLab/maia-chess/tree/master/maia_weights.
  * [Margulies 1977] Margulies, S.  1977\.  Principles of beauty.  Psychol Rep 41(1):3–11.
  * [McIlroy-Young et al. 2020] McIlroy-Young, R.; Sen, S.; Kleinberg, J.; and Anderson, A.  2020\.  Aligning Superhuman AI with Human Behavior: Chess as a Model System.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 1677–1687.
  * [Osborne 1964] Osborne, H.  1964\.  Notes on the aesthetics of chess and the concept of intellectual beauty.  Brit J Aesthetics 4(2):160–163.
  * [Stockfish 2024] Stockfish.  2024\.  Stockfish.  Accessed: Jan. 22, 2024. [Online]. Available: https://github.com/official-stockfish/Stockfish.
  * [Walls 1997] Walls, B. P.  1997\.  Beautiful Mates: Applying Principles of Beauty to Computer Chess Heuristics.  Batsford chess library. [United States]: Dissertation.com, 1st american ed. edition.