# Playing Chess Efficiently with Transformers

Johannes Czech 1, Kristian Kersting 1, 2, 3, 4

1 Department of Computer Science, TU Darmstadt, Germany
2 Centre for Cognitive Science, TU Darmstadt, Germany
3 Hessian Center for Artificial Intelligence (hessian.AI), Darmstadt, Germany
4 German Research Center for Artificial Intelligence (DFKI), Darmstadt,
Germany

<EMAIL_ADDRESS><EMAIL_ADDRESS>0000-0002-9568-8965 0000-0002-2873-9152

###### Abstract

The Transformer architecture has established itself over the recent years in
the natural language and computer vision domain. However, it has not succeeded
in tasks where, the combination of search and predictions plays a critical
role. In this domain efficient latency is as important as a high quality
prediction. In this domain, approaches like AlphaZero are still the current
state of the art. In this work, we evaluate the classical transformer
architecture in the game of chess and improve it by presenting a new
convolutional transformer hybrid neural network architecture. This new
combination of techniques allows us to… inlineinlinetodo: inlineWhat are the
advantages using this combination, what problems of alphazero can we tackle
with transformers, which problems of transformers can be tackle with
convolutional layers? Answer this is one or two sentences. Overall, all of our
improvements to the neural network architecture result in advantage of XX Elo
compared to the AlphaZero neural network architecture.

##  1 Introduction

Transformer architectures are a type of neural network architecture used in
natural language processing (NLP) tasks such as machine translation, sentiment
analysis, and text summarization. They were introduced in 2017 by
vaswani2017attention. What distinguishes transformers from traditional
convolutional neural networks (CNNs) is the use of self-attention mechanisms.
This allows the network to weigh the importance of different elements in the
input sequence, rather than simply processing the sequence sequentially or
using a fixed size context window. As a result, transformers have achieved
state-of-the-art performance on a wide range of NLP tasks and have become a
cornerstone of NLP models. The popularity of transformers can be attributed to
their ability to handle long-range dependencies, a feature which is also very
important in the field of computer vision, where the relationships between
objects in an image can be very complex and span a large spatial area. With
transformers, this information can be effectively captured and modeled,
whereas with CNNs it can be more challenging. This is the reason nowadays
transformer models are taken over classical CNN approaches in computer vision
and other domains dosovitskiy2020image.

AlphaZero on the other hand is an architecture which combines neural networks
with Monte-Carlo tree search (MCTS) to tackle inlineinlinetodo: inlinestate
some problems AlphaZero is tackeling. AlphaZero has achieved remarkable
results in games such as chess and shogi, outperforming the strongest previous
algorithms and even challenging top human players.

inlineinlinetodo: inlineState the problem: AlphaZero is building heavily on
CNNs while also having long-range dependencies (CAP) with large spatial areas
in late game situations. We want to create an architecture that can be trained
efficiently and improve performanceinlineinlinetodo: inlineShow that
transformers alone give bad results.

AlphaZero showed huge successes because of its ability to learn from scratch,
generalize to new problems, achieve strong performance, and have implications
for decision-making and optimization. It is a powerful model for how machines
can learn to solve complex problems.

Transformers on the other hand showed us how to use attention mechanism to
improve scalability, robustness, versatility and interoperability, making is a
powerful architecture for a wide range of AI applications.

By combining the strengths of transformers and RL, it is possible to develop
powerful models for solving complex sequential decision-making problems. We
could for example use transformers in RL to model the relationships and
dependencies between sequences of observations, actions, and rewards. They can
be used to model the state representation, policy, and value function, and can
be pre-trained for transfer learning inlineinlinetodo: inlinefind sources like
Decision Transformer, Trajectory Transformer, InstructGPT, ….

However,… inlineinlinetodo: inlinewhy is it hard to apply these methods to
chess successfully? Further from a technical point of view, transformer
architectures showed good success in terms of FLOPs but has a rather low GPU
utility compared to convolutional neural networks. inlineinlinetodo:
inlinesource

In this paper we tested if one can use Transformers to tackle some of the
problems of AlphaZero, origin from using CNNs. We further tested if
Transformers can bring improvement compared to CNNs in the domain of chess and
if they can be improved in this competitive field by being combined with
convolutional layers.

Figure 1: Comparison among final RISE and efficient networks with respect to
accuracy-latency trade-off. Results are measured using three seeds. Details
can be found in Table 10

To summarize, we provide the following contributions:

  * •

We evaluated the performance of Visual Transformer in combination with the
known AlphaZero architecture

  * •

We present a convolutional transformer hybrid neural network architecture that
is superior compared to the standard AlphaZero and other alternative neural
network architecture.

  * •

We investigate the new architecture and test multiple improvements such as
squeeze excitation.

##  2 Vision Transformer vs Convolutional NN

A milestone in the short history of transformers are the so called Vision
Transformers (ViT), introduced by dosovitskiy2020image. They are a variant of
the Transformer architecture specifically designed for image processing tasks.
It uses self-attention mechanisms to learn relationships between different
parts of an image, and can process images of arbitrary size without reducing
their resolution. ViT has achieved state-of-the-art performance on several
benchmark datasets in the field of computer vision. Later, ViT has been
adapted to further improve in this domain, such as in the Mobile ViT
architecture mehta2021mobilevit and Levit structure graham2021levit.

One general disadvantage of Transformers are their latency efficiency but
recently there were some efforts made to improve performance of vision
transformers, using TensorRT and specifically adapted architectures, like Trt-
ViT xia2022trt and NextViT li2022next.

In traditional convolutional neural networks (CNNs), which are the dominant
architecture for image processing tasks, information is processed in a
hierarchical manner, with lower-level features being combined to form higher-
level features. In contrast, ViT uses a self-attention mechanism to allow the
model to learn relationships between different parts of the image, without
requiring a hierarchical processing pipeline. Further, … inlineinlinetodo:
inlinemore differences between ViT und CNNs.

Another important archtiecture for this paper is the MobileNet architecture
introduced by howard2019searching. MobileNet uses a combination of depthwise
separable convolutions and pointwise convolutions to reduce the computational
cost and memory requirements of traditional convolutional neural networks
(CNNs), while still maintaining high accuracy. MobileNets have been shown to
achieve high accuracy on a range of computer vision tasks, while being
significantly faster and more memory-efficient than traditional CNNs. This
makes them well-suited for usage within AlphaZero. Several variations of
MobileNets have been proposed, including MobileNetV2 and MobileNetV3, which
introduce additional optimizations and improvements over the original
architecture. In this work, we will use MobileNetV3. For training, we also
make use of stochastic depth huang2016deep in order to both speed up training
and improve convergence. The scaling technique that allow to generate networks
in different sizes is adapted from EfficientNet tan2019efficientnet.

inlineinlinetodo: inlineCompare CNNs and ViTs

##  3 Combining Transformers with AlphaZero

###  3.1 ViT

###  3.2 Input Representation

###  3.3 NextViT+CNN (Hier brauchen wir denke ich noch einen Namen?)

There are multiple different ways on how to integrate transformer blocks into
a convolutional neural network architecture forming a hybrid CNN structure. We
use the Next Hybrid Strategy of li2022next, where a single transformer block
is preceded by multiple convolutional blocks.

##  4 Experiments

###  4.1 Experimental Setup

For all our experiments, we use the following training hyperparmeters as shown
in Table 13. Moreover, we use three seeds for our setups to improve
statistical significance. As our database we use the KingBase Lite 2019 data
set111https://archive.org/details/KingBaseLite2019, accessed-2022-11-02.
KingBase Lite 2019 is a free chess database of over 1 million chess games
played since 2000 by human expert players with a player ELO rating greater
than 2200. Latency is measured on a RTX 2070 OC. As our loss function
definition we use a shared value policy loss with the number of plys (half-
moves) remaining as an auxiliary target:

$l=-\alpha(\text{WDL}_{\text{t}}\log\mathbf{\mathbf{WDL}}_{\text{p}})-\pi\log\mathbf{p}+\beta(\text{ply}_{\text{t}}-\text{ply}_{\text{p}})^{2}+c\|\theta\|\quad,$
(1)

where $\pi$ represents the true policy, $\mathbf{p}$ the predicted policy and
$c$ a $L_{2}$ regularization constant for the network parameters $\theta$.
Instead of expressing the value loss as a mean squared error
$\in\left[-1,+1\right]$, we define it using
$\mathbf{\mathbf{WDL}}_{\text{p}}$, a distribution over the predicted chance
of winning, drawing or losing. Details on the WDLP head can be found in
Section 4.4. Finally, $\alpha$ and $\beta$ are scalar value to adjust the
weighting of the individual loss components.

###  4.2 Preliminary Experiments

In our preliminary experiments, we investigate the performance of a plain full
transformer based neural network for standard chess. As can be seen in Table 1
these networks are both slow and achieve a relatively high loss. Therefore, we
aim to build a convolutional transformer hybrid architecture.

For this, we first search for the best convolutional base block. Next we find
the best scaling factor for depth and width. At last, we combine the
convolutional stem with the transformer block, forming a CNN transformer
hybrid.

The so called “Next Convolution Block” was compared against the ConvNext,
Transformer, PoolFormer and Uniformer block in li2022next. However, it was
missing the classical residual block that uses two 3$\times$3 convolutions as
in the standard AlphaZero network and mobile convolution block that uses group
depthwise convolution in its comparison. Therefore we evaluate in our first
experiment which of these blocks as shown in Figure LABEL:fig:conv_blocks is
performing best under the same latency constraints. We later presume using
this block as our standard convolution block. As can be seen in Table 1, the
mobile convolutional block sandler2018mobilenetv2 perform slightly better than
the classical residual block he2016deep, but the next convolutional block
li2022next performs considerably worse under the same latency constraints.

Table 1: Training results of comparing different convolution blocks in the core part of the model. The mobile convolutional block using an expansion ratio of three performs slight better than the classical residual block and significantly better than the next convolutional block. All three setups have a similar latency on GPU. Convolutional Block | # Blocks | Channels |  Combined Loss | Policy Accuracy (%) | Latency (nps)
---|---|---|---|---|---
Classical residual block he2016deep | 10 | 192 | 1.23497 $\pm$ 0.00305 | 57.500 $\pm$ 0.141 | 27,646
Mobile convolutional block sandler2018mobilenetv2 | 9 | 256 | 1.23432 $\pm$ 0.00233 | 57.533 $\pm$ 0.047 | 28,755
Next convolutional blockli2022next | 10 | 256 | 1.24112 $\pm$ 0.00092 | 57.333 $\pm$ 0.047 | 28,704
Table 2: Grid search results of using different scaling parameters according to efficient nettan2019efficientnet. A depth scaling factor of $\alpha$=1.8 and channel size scaling factor of $\beta=\sqrt{\nicefrac{{\mathbf{10}}}{{9}}}$ performs best. However, scaling along $\beta$ provides a better latency as the number of FLOPS would expect. $\mathbf{\alpha}$ | $\mathbf{\beta}$ | Depth | Channels | Combined Loss | Policy Accuracy (%) | Latency (nps)
---|---|---|---|---|---|---
1.0 | $\sqrt{2}$ | 10 | 272 | 1.21497 | 58.1 | 22,028
1.2 | $\sqrt{\nicefrac{{5}}{{3}}}$ | 12 | 248 | 1.21381 | 58.0 | 22,446
1.4 | $\sqrt{\nicefrac{{10}}{{7}}}$ | 14 | 229 | 1.19888 | 58.5 | 20,885
1.6 | $\sqrt{\nicefrac{{5}}{{4}}}$ | 16 | 215 | 1.20036 | 58.5 | 19,749
1.8 | $\mathbf{\sqrt{\nicefrac{{\mathbf{10}}}{{9}}}}$ | 18 | 202 | 1.19084 | 58.6 | 19,550
2.0 | 1 | 20 | 192 | 1.19559 | 58.4 | 19,921
Table 3: Grid search results of using adapted scaling parameters according of efficient nettan2019efficientnet. A depth scaling factor of $\alpha=1.8$ and channel size scaling factor of $\beta={\nicefrac{{10}}{{9}}}^{\nicefrac{{5}}{{8}}}$ performs best. The measured latency is similar across all configurations. $\mathbf{\alpha}$ | $\mathbf{\beta}$ | Depth | Channels | Combined Loss | Policy Accuracy (%) | Latency (nps)
---|---|---|---|---|---|---
1.0 | ${2}^{\nicefrac{{5}}{{8}}}$ | 10 | 296 | 1.21062 $\pm$ 0.00117 | 58.133 $\pm$ 0.047 | 20,328
1.2 | ${\nicefrac{{5}}{{3}}}^{\nicefrac{{5}}{{8}}}$ | 12 | 264 | 1.19977 $\pm$ 0.00263 | 58.333 $\pm$ 0.124 | 19,508
1.4 | ${\nicefrac{{10}}{{7}}}^{\nicefrac{{5}}{{8}}}$ | 14 | 240 | 1.19820 $\pm$ 0.00791 | 58.500 $\pm$ 0.216 | 20,402
1.6 | ${\nicefrac{{5}}{{4}}}^{\nicefrac{{5}}{{8}}}$ | 16 | 221 | 1.20031 $\pm$ 0.00924 | 58.400 $\pm$ 0.216 | 19,353
1.8 | $\mathbf{{\nicefrac{{\mathbf{10}}}{{9}}}^{\nicefrac{{\mathbf{5}}}{{8}}}}$ | 18 | 205 |  1.19009 $\pm$ 0.00488 |  58.667 $\pm$ 0.169 | 18,679
2.0 | 1 | 20 | 192 | 1.19039 $\pm$ 0.00740 | 58.600 $\pm$ 0.308 | 19,921
Table 4: Grid search results of different transformer block configurations. Using two transformer blocks gives the best result. # Transformer blocks | Depth | Channels | Combined Loss | Policy Accuracy (%) | Latency (nps)
---|---|---|---|---|---
0 | 18 | 205 | 1.19009 $\pm$ 0.00488 | 58.667 $\pm$ 0.169 | 18,679
1 | 16 | TODO | 1.19196 $\pm$ 0.00206 | 58.566 $\pm$ 0.169 | 18,382
2 | 15 | TODO |  1.18873 $\pm$ 0.00647 |  58.667 $\pm$ 0.124 | 18,227
3 | 14 | TODO | 1.20610 $\pm$ 0.01397 | 58.300 $\pm$ 0.432 | 18,291
4 | 12 | TODO | 1.23266 $\pm$ 0.00446 | 57.567 $\pm$ .0471 | 18,982
Table 5: Grid search results of using different $5\times 5$ kernel filter ratios. Using 50% of $5\times 5$ convolutions and 50% of $3\times 3$ convolutions gives the best result. In the case of $5\times 5$ convolutions an expansion ratio of 2.04 instead of 3.0 is used. $\mathbf{5\times 5}$ Kernel Filter Ratio (%) | Depth | Channels | Combined Loss | Policy Accuracy (%) | Latency (nps)
---|---|---|---|---|---
0 | 16 | 221 | 1.20031 $\pm$ 0.00924 | 58.400 $\pm$ 0.216 | 19,353
25 | 16 | 221 | 1.18016 $\pm$ 0.00594 | 59.000 $\pm$ 0.216 | 19,741
50 | 16 | 221 |  1.17336 $\pm$ 0.00472 |  59.199 $\pm$ 0.141 | 19,390
75 | 16 | 221 | 1.18521 $\pm$ 0.00758 | 58.766 $\pm$ 0.205 | 19,082
100 | 16 | 221 | 1.18435 $\pm$ 0.00514 | 58.833 $\pm$ 0.124 | 19,112
Table 6: Grid search results of using different squeeze excitation ratios. Using a ration of 50% gives the best result. Instead of the vanilla squeeze excitation, the more efficient squeeze excitation module by wang_eca-net_2020 is used. Squeeze Excitation Ratio (%) | Depth | Channels | Combined Loss | Policy Accuracy (%) | Latency (nps)
---|---|---|---|---|---
0 | 18 | TODO | 1.19009 $\pm$ 0.00488 | 58.667 $\pm$ 0.169 | 18,679
25 | 17 | TODO | 1.19886 $\pm$ 0.01497 | 58.366 $\pm$ 0.403 | 18,314
50 | 16 | TODO |  1.18864 $\pm$ 0.00117 |  58.666 $\pm$ 0.047 | 18,447
75 | 15 | TODO | 1.19725 $\pm$ 0.00361 | 58.533 $\pm$ 0.094 | 18,407
100 | 14 | TODO | 1.20589 $\pm$ 0.00135 | 58.233 $\pm$ 0.094 | 18,100
Table 7: Grid search results of using different ratio of the hard swish TODO activation function. Using only ReLU instead of hard swish gives the best result. Hard Swish Ratio (%) | Depth | Channels | Combined Loss | Policy Accuracy (%) | Latency (nps)
---|---|---|---|---|---
0 | 18 | 205 |  1.19009 $\pm$ 0.00488 |  58.666 $\pm$ 0.169 | 18,679
25 | 17 | TODO | 1.21490 | 57.9 | 18,933
50 | 16 | TODO | 1.22246 | 57.9 | 19,604
75 | 15 | TODO | 1.24772 | 56.9 | 20,088
100 | 14 | TODO | 1.26289 | 56.7 | 20,061
Table 8: Grid search results of using different expansion ratios for the mobile convolution block. Using an expansion ratio of 1.5 gives the best result. Expansion Ratio | Depth | Channels | Combined Loss | Policy Accuracy (%) | Latency (nps)
---|---|---|---|---|---
0.5 | 62 | 236 | 1.19824 $\pm$ 0.00210 | 58.4 $\pm$ 0.000 | 19,598
1 | 40 | 224 | 1.18850 $\pm$ 0.00084 | 58.633 $\pm$ 0.125 | 19,859
1.5 | 31 | 218 | 1.18542 $\pm$ 0.00410 | 58.733 $\pm$ 0.094 | 19,164
2 | 23 | 211 | 1.18840 $\pm$ 0.00493 | 58.700 $\pm$ 0.163 | 19,584
3 | 18 | 205 | 1.19009 $\pm$ 0.00488 | 58.667 $\pm$ 0.169 | 18,679
4 | 14 | 199 | 1.20027 $\pm$ 0.00651 | 58.367 $\pm$ 0.125 | 19,384
5 | 12 | 196 | 1.20489 $\pm$ 0.00444 | 58.367 $\pm$ 0.094 | 18,768
Table 9: Grid search results of increasing the number of expanded channels by different amounts. Incrementing expanded channels by 8 each layer works best. Expansion Channel Increment | Depth | Channels | Combined Loss | Policy Accuracy (%) | Latency (nps)
---|---|---|---|---|---
0 | 18 | 205 | 1.19009 $\pm$ 0.00488 | 58.667 $\pm$ 0.169 | 18,679
8 | 18 | 205 | 1.18740 $\pm$ 0.00243 | 58.733 $\pm$ 0.125 | 18,972
16 | 18 | 205 | 1.19038 $\pm$ 0.00336 | 58.733 $\pm$ 0.094 | 18,948
32 | 18 | 205 | 1.19422 $\pm$ 0.00489 | 58.500 $\pm$ 0.141 | 19,150
Table 10: Comparison of different neural network architectures against final RISE. Results of achieved policy accuracies are shown in Figure 1. All networks use the same input representation and loss formulation. Network Architecture | Depth | Channels | Combined Loss | Policy Accuracy (%) | Latency (nps)
---|---|---|---|---|---
Resnet (AlphaZero) | 18 | TODO | 1.16728 $\pm$ 0.00399 | 59.433 $\pm$ 0.094 | 14,652
Resnet-SE (AlphaZero) | 17 | TODO | 1.16195 $\pm$ 0.00014 | 59.633 $\pm$ 0.047 | 11,729
RISE-v2 | 16 | TODO | 1.20274 $\pm$ 0.0016 | 58.367 $\pm$ 0.047 | 25,326
LeViT | 15 | TODO | 1.25957 $\pm$ 0.00402 | 56.933 $\pm$ 0.094 | 17,271
NextViT | 14 | TODO | 1.17245 $\pm$ 0.00395 | 59.100 $\pm$ 0.082 | 18,319
ViT | 14 | TODO | 1.68660 $\pm$ 0.00144 | 47.400 $\pm$ 0.163 | 14,141
Table 11: Grid search results of using different value heads. Using a WDLP as the value head performs best. Value Head Type | Depth | Channels | Combined Loss | Policy Accuracy (%) | Value Loss | Latency (nps)
---|---|---|---|---|---|---
No WDLP | 18 | 205 | 1.19325 $\pm$ 0.00214 | 58.500 $\pm$ 0.082 | 0.4406 $\pm$ 0.4406 | 18,745
WDLP a la Forsten | 18 | 205 | 1.19007 $\pm$ 0.00511 | 58.733 $\pm$ 0.124 | 0.4356 $\pm$ 0.0006 | 18,734
Table 12: Grid search results of using different input representations. Using an adapted input representation improves both the value and policy loss. Input Representation | Depth | Channels | Combined Loss | Policy Accuracy (%) | Value Loss | Latency (nps)
---|---|---|---|---|---|---
Inputs V.1.0 | 18 | 205 | 1.19180 $\pm$ 0.00276 | 58.633 $\pm$ 0.047 | 0.4448 $\pm$ 0.0007 | 19,202
Inputs V.2.0 | 18 | 205 |  1.19009 $\pm$ 0.00489 |  58.667 $\pm$ 0.170 |  0.4371 $\pm$ 0.0002 | 18,679

After finding the best convolutional base block, we know proceed to search for
the best scaling ratios along depth and width. Here we follow the compound
scaling method as described in efficient net tan2019efficientnet.

$\quad\quad\quad\quad\quad\quad\quad\quad\text{depth:}\;d=\alpha^{\phi}\\\
\text{width:}\;w=\beta^{\phi}\\\ \text{resolution:}\;r=\gamma^{\phi}\\\
\text{s.t.}\quad\alpha\cdot\beta^{2}\cdot\gamma^{2}\approx 2\\\ \alpha\geq
1,\beta\geq 1,\gamma\geq 1$ (2)

The input resolution is on our case is fixed to a 8$\times$8 grid and we can
set $\gamma=1$. We find that scaling along width $\beta$ does not fully fit
the criteria of

$\alpha\cdot\beta^{2}\approx 2$ (3)

as can be seen in ”Latency (nps)” column of Table2. Instead, the latency
increases when increasing $\beta$ compared to $\alpha$. This is expected as
also other works zagoruyko2016wide have shown that increasing the width of a
network is more efficient on GPU than on CPU and doesn’t fully comply to the
number of floating point operations. Hence, we instead use the following
criteria

$\alpha\cdot\beta^{1.6}\approx 2$ (4)

which fits the measured latency more appropriately as can be seen in Table 3.
According to Table 3 an expansion ratio of $\alpha=1.8$ and
$\beta={\nicefrac{{10}}{{9}}}^{\nicefrac{{5}}{{8}}}$ works best. Adjustments
to the network structure either increase or decrease latency. Therefore, we
proceed to use this ratio in the other experiments to scale up or down the
network to ensure that the different configuration have a simliar latency.

In the next experiment, as shown in Table 4, we integrate the transformer
blocks of li2022next into the convolutional architecture and find the right
number of transformer blocks in relation to the convolutional blocks. We use
the hybrid setup as proposed in li2022next by putting a single transformer
block after a specified number of convolutional blocks rather than putting all
transformer blocks at the end of the network. It turns out that using two
transformer blocks per fifteen layers gives the best result.

Next we evaluate the effect of Squeeze Excitation layers on performance. Here
we employ the more efficient squeeze excitation module by wang_eca-net_2020.
Using a ratio of 50% gives the best result.

As can be seen in Table 7, relying only on the ReLU activation function is
superior compared to hard-swish. This was also recommended by li2022next for
TensorRT inference.

Afterwards, we answer the question how linearly increasing the number of
expanded channels along depth, as suggested in the pyramid network
han2017deep, affects performance. For this we test three setups with varying
degrees of increasing channels. Results are summarized in Table 9.

###  4.3 New Input Representation

Traditionally the input to the neural network is given by a stack of $8\times
8$ planes. A basic board representation for this can be found in Table
LABEL:tab:inputs_v1. In an alternative input representation, see Table
LABEL:tab:inputs_v2, we add additional features such as features to represent
the material configuration of both players and remove the color information
and current move count to avoid overfitting to this data information. In fact,
our new input definition performs better both with respect to the policy as
well as value loss. We argue that this is the case because although these new
features can be inferred by processing existing features, the prove to be
useful and implicitly increase the capacity of the network as they do not need
to be computed by the network itself. This shows that feature engineering can
still prove to be useful even in the field of deep neural networks.

###  4.4 WDLP Value Head

As proposed in other engines for the game of Go wu2019accelerating, using
auxiliary targets in the loss function during training appears to be
beneficial in chess, too. Here, Henrik Forstén suggested to use a seperate
WDL-Head 222https://github.com/LeelaChessZero/lc0/pull/635, accessed
2022-11-11 to predict the percentage ratio of winning, drawing and losing and
a Moves Left Head 333https://github.com/LeelaChessZero/lc0/pull/961, accessed
2022-11-11 to predict the number of moves until the games is over. In our case
we integrate an additional output to the value head to predict the number of
remaining plys (half-moves) until the game end and refer to the new value head
as WDLP Value Head.

One can use

$\text{value}_{output}=-\text{L}_{output}+\text{W}_{output}$ (5)

to infer the classical $\text{value}_{output}\in\left[-1,+1\right]$ from the
WDL output expression. Here, $\text{L}_{output}$ presents the probability of
losing and $\text{W}_{output}$ describes the probability of winning a game.
Table 11 verifies that using new loss formulation proves to be beneficial.

###  4.5 Final Models

Based on the intermediate results of our previous experiments, we now
construct four different model architectures in the sizes tiny, small, normal
and large. All architecture follows the best hyperparameter setup found in the
proceeding experiments:

They also make use of a channel size that is a multiple of 32 for performance
reasons. The tiny version consists of 11 blocks with a base channel size of
192 and one transformer block. The small version has 16 blocks, a base
channels size of 192 and two transformer blocks. The normal size has 19
blocks, uses a 224 base channels and three transformer blocks. The large model
specification contains 26 blocks, 224 base channels and three transformer
blocks. The final architectures are presented in Table
LABEL:tab:final_rise_architectures and their performance is show in Figure 1
and Table 10. The number of channels of all architectures are based on a
multiple of 32 due to a higher computational efficiency on GPU.

##  5 Discussion

###  5.1 Can we use Transformers to play chess?

Why do transformer not improve the result here? -¿ The state of chess is
Markovian, therefore the main advantage of transformer to model attention
across a long trajectory is not applicable. compare to: ”On Transforming
Reinforcement Learning by Transformer: The Development Trajectory”

Using history improves performance here, but arguably only due to the
attention effect. For the problem definition itself, the history is not
absolutely required.

###  5.2 Playing Strength Evaluation

Finally, in order to verify the improvement of our new model architectures, we
use the normal size mode and let it compete against the vanilla AlphaZero
model. We also train additional chess variants for both AlphaZero and our
model to see its effects in different environments. For our training data, we
use here the openly available lichess.org data set. As can be seen in
Table/Figure X, the improvement is consitent and variies from an Elo
difference from X Elo in variant Y to Z Elo in variant C. All games are played
using a time control of X min + Y sec increment. We also use opening suites to
allow a greater game variety.

##  6 Related Work

Even though transformers were primarily developed for supervised learning
settings, such as those often found in natural language processing or computer
vision, they are now also being used repeatedly in the field of reinforcement
learning.

The goal of Transformer+RL is often to model and in many cases improve the
decision-making process through the use of attention. Many RL problems can be
viewed and modelled as sequential problems (RL as OBS problem). There are
multiple possibilities for combining Transformers and RL, for example in the
area of architecture enhancement or trajectory optimization. [2212.14164] The
best-known current approaches include Trajectory Transformer and Decision
Transformer. A good overview on the topic of Tranformers in RL can be found in
the works by [2212.14164, 2301.03044]

Transformers were able to show comparable results to conventional approaches
in many RL tasks [DT Paper, Multigame Transformer Paper, …], but there are
still some open challenges.

inlineinlinetodo: inlineState Challenges, eg. performance/efficence/…

The use of Transformers in the domain of chess is not new, [ChessTransformer]
has already tried to learn chess through a transformer. However, their
approach is fundamentally different from ours. [ChessTransformer] try to learn
chess using a Large Language Model (LLM). They use the so-called PGM notation
of chess positions and approach the problem from the perspective of an NLP
problem. The results were inlineinlinetodo: inline… (can we compare the
results in some way?).

Also, Transformers and LLMs have already been used to generate comments on
chess positions to improve comprehensibility and traceability. [2212.08195]

A combination of AlphaZero and Transformers, as described in Section
LABEL:sec:combining is not known to us.

###  6.1 Current State-of-the-Art in Chess

The state-of-the-art in the domain of chess is inlineinlinetodo: inlineState
the SOTA (NNUE) and A0

##  7 Future Work

Transformers in general give interesting avenue for potential future work, e.
g. during the search or reinforcement learning procedure.

##  8 The Conclusions

In this work we presented a general neural network architecture in four
different sizes that represents a new state of the art in the chess domain and
is applicable to various other board games. We tested our architecture under
the commonly used TensorRT inference framework. Moreover, we demonstrated that
introducing transformer blocks can indeed help to slightly improve
performance, but other changes such as using 5$\times$5 convolutions have a
greater impact.

##  9 The Acknowledgements

for reviewing our manuscript, early discussions, and providing many helpful
comments and suggestions ppreciate valuable feedback and support from

Jannis Blüml

Thanks for the discussion with Hung-Zhe Lin for his input and his ideas.

Thanks for the discussion with Felix Friedrich.

##  10 Citations

##  11 Appendix A

Table 13: Hyperparameter configuration for running the experiments. Hyperparameter | Value
---|---
max learning rate | 0.07
min learning rate | 0.00001
batch size | 1024
max momentum | 0.95
min momentum | 0.8
epochs | 7
optimizer | NAG
value loss factor | 0.01
policy loss factor | 0.988
wdl loss factor | 0.01
plys to end loss factor | 0.002
stochastic depth probability | 0.05
pytorch version | 1.12.0