# End-to-End Chess Recognition

Athanasios Masouris
Delft University of Technology
Delft, The Netherlands
<EMAIL_ADDRESS>Jan van Gemert
Delft University of Technology
Delft, the Netherlands
<EMAIL_ADDRESS>
###### Abstract

Chess recognition refers to the task of identifying the chess pieces
configuration from a chessboard image. Contrary to the predominant approach
that aims to solve this task through the pipeline of chessboard detection,
square localization, and piece classification, we rely on the power of deep
learning models and introduce two novel methodologies to circumvent this
pipeline and directly predict the chessboard configuration from the entire
image. In doing so, we avoid the inherent error accumulation of the sequential
approaches and the need for intermediate annotations.

Furthermore, we introduce a new dataset, Chess Recognition Dataset (ChessReD),
specifically designed for chess recognition that consists of 10,800 images and
their corresponding annotations. In contrast to existing synthetic datasets
with limited angles, this dataset comprises a diverse collection of real
images of chess formations captured from various angles using smartphone
cameras; a sensor choice made to ensure real-world applicability. We use this
dataset to both train our model and evaluate and compare its performance to
that of the current state-of-the-art. Our approach in chess recognition on
this new benchmark dataset outperforms related approaches, achieving a board
recognition accuracy of 15.26% ($\approx$7x better than the current state-of-
the-art).

##  1 Introduction

Chess recognition from a single image poses a significant challenge in the
field of computer vision. The task requires accurate identification of each
chess piece’s type and position on a chessboard configuration. The predominant
approach [29, 15, 6, 27] is to divide the overall task into the tasks of
chessboard detection, square localization, and piece classification within
each square. However, this approach not only suffers from error accumulation
throughout the intermediate steps but also often sets restrictions to the
task, with regard to selected viewing angles [27]. In this paper, we propose a
novel approach for chess recognition that circumvents this pipeline and
directly predicts the positions of the pieces, with respect to the chessboard,
from the entire image.

One key advantage of our approach is that it does not impose any restrictions
on the input image, such as specific angles and orientations. Our model can
handle images taken from various perspectives, enabling robust chess
recognition. By leveraging a deep neural network, the model is able to extract
and use relevant visual features to efficiently predict the pieces’ type and
positions. Additionally, our approach does not require any human input beyond
the input image itself. Unlike traditional methods [21, 8, 16, 27] that rely
on user input, such as manually selecting the corners of the chessboard or
defining the player’s perspective, our approach directly learns to recognize
the chessboard and infer the piece positions strictly from the visual
information in the input image.

Furthermore, a factor that has hindered research on this particular domain is
the lack of a comprehensive real-world dataset. To this end, we introduce a
new real-world dataset specifically designed for chess recognition, Chess
Recognition Dataset (ChessReD)
111https://data.4tu.nl/datasets/99b5c721-280b-450b-b058-b2900b69a90f/2,
consisting of $10,800$ images and their corresponding annotations, that would
allow us to effectively train and evaluate our proposed approach. The dataset
comprises a diverse collection of chessboard images, covering various viewing
angles, lighting conditions, camera specifications, and piece configurations.
By providing this dedicated dataset, we enable both further research in chess
recognition algorithms and a common benchmark for researchers to evaluate
their approaches.

The results indicate that our approach in chess recognition on this new
benchmark dataset outperforms related approaches, achieving a board
recognition accuracy of 15.26% ($\approx$7x better than the current state-of-
the-art). Thus, our contributions can be summarized as follows.

  * •

We introduce the first dataset of real images for chess recognition, with
increased variability in terms of viewing angles and chess formations compared
to synthetic alternatives

  * •

We demonstrate that chess recognition can be tackled using an end-to-end
approach, with improved performance compared to sequential alternatives

##  2 Related Work

###  2.1 Early Approaches in Chess Recognition

Chess recognition has been a subject of research in the field of computer
vision, with several approaches proposed to tackle the challenges associated
mainly with detecting the chessboard, but also with recognizing the individual
pieces on top of it. Early attempts in the field primarily focused on
integrating the chess recognition task as a part of a chess-playing robotic
system [23, 21, 14, 1]. These systems detected chess moves by comparing the
previous frame, with a known game state, to the current frame. They relied on
detecting the occupied squares of the chessboard along with the colors of the
pieces occupying them. As such, these methods were only able to detect valid
chess moves and failed to detect events when two pieces of the same color were
swapped, either illegally, or by promoting pawns to another piece type.
Additionally, they worked under the assumption that the previous inferred
state is correct. Thus, in case of an erroneous move prediction, all of the
subsequent game states would be incorrect. Despite the aforementioned issues,
the same approach has been adopted by several studies [24, 12, 3, 16, 4, 11]
since, with consecutive frames obtained also from a video stream [21, 24, 12].
In this study, contrary to these approaches, we aim to develop a robust method
that does not rely on the correctness of the previous inferred state but
rather performs chess recognition from a single input image.

###  2.2 Chessboard Detection

For cases when the previous state is unknown, chess recognition from a single
image has also been the focus of studies. Same as with the previously
mentioned approaches, the first step is to employ image processing techniques
to detect the chessboard and the individual squares; a challenging task even
on its own. While it can be simplified by explicitly asking the user to select
the four corner points [21, 8, 16], modifying the chessboard [23, 1, 7]
(_e.g_. using a reference frame around the chessboard), or setting constraints
on the camera view angle (_e.g_. top-view) [23, 21, 1, 24, 12, 16, 15],
Neufeld _et al_. [17] recognized that these approaches do not represent a
general solution, where the chessboard could be in arbitrary locations or the
image taken from various camera angles. They proposed a line-based detection
method which they combined with probabilistic reasoning. However, they also
restricted the setting by expecting the camera angle to be in the range of a
human player’s perspective. Other studies have also exploited specific viewing
angles, such as the players’ perspectives [8, 3, 4, 27] or side views [7, 18].
While chessboard detection utilizing the Harris corner detection algorithm [1,
12, 11], template matching [23, 14], or flood fill [24] have been explored, in
accordance with [17], line-based chessboard detection methods have received
significant research attention [3, 4, 29, 22, 30, 7, 15]. Czyzewski _et al_.
[6] introduced an approach based on iterative heat map generation which
visualizes the probability of a chessboard being located in a sub-region of
the image. After each iteration, the four-sided area of the image containing
the highest probability values is cropped and the process is repeated until
convergence. While this method involves a great computational overhead, it is
able to detect chessboards from images taken from varied angles, with poor
quality, and regardless of the state of the actual chessboard (_e.g_. damaged
chessboard with deformed edges), with a 99.6% detection accuracy. Wölflein and
Arandjelović [27] proposed a chessboard detection method that leveraging the
geometric nature of the chessboard, utilizes a RANSAC-based algorithm to
iteratively refine the homography matrix and include all the computed
intersection points. Their method demonstrated impressive results, since it
successfully detected all of the chessboards in their validation dataset.
However, it’s worth noting that the dataset only included images with viewing
angles within the range of a player’s perspective. In our paper, we bypass the
chessboard detection task, allowing the deep learning models to internally
infer its position, and thus we do not rely on user input, or specific viewing
angles.

###  2.3 Piece Classification

Upon detection of the chessboard, the next step the aforementioned approaches
employ is piece classification. A number of techniques have been developed to
address this task, either in a 2-way approach (_i.e_. color and type), or
1-way by treating each combination of piece color and type as a separate class
(_e.g_. “white-rook”). In Matuszek _et al_. [14], the authors utilized one
classifier to determine the piece color, and then for each color they trained
a type classifier using concatenated scale-invariant feature transform (SIFT)
and kernel descriptors for features. A similar approach was used in Ding [8],
where the author employed SIFT and histogram of oriented gradients (HOG) as
feature descriptors for piece type classification with support vector machine
(SVM) classifiers. The color was subsequently detected by comparing the
binarized image of the square with that of an empty one. Danner and Kafafy [7]
and Xie _et al_. [29] argued that the lack of distinguishable texture in small
objects, such as chess pieces, leads to insufficient features obtained using
SIFT descriptors. Both studies suggested a template matching approach for
piece classification, comparing the contours of the detected pieces with
reference templates obtained from various angles. Wei _et al_. [25] proposed
an approach to recognize pieces using a depth camera and a volumetric
convolutional neural network (CNN). More recent studies [15, 6, 18, 27] follow
the 1-way approach for piece classification. They train CNNs to distinguish
between 12 or 13 classes of objects (_i.e_. six piece types in both colors and
one for empty squares), obtaining impressive results. In Czyzewski _et al_.
[6], they also leverage domain knowledge, to improve piece classification, by
utilizing a chess engine to calculate the most probable piece configurations
and clustering similar figures into groups to deduce formations based on
cardinalities. Additionally, given the variation in appearance between chess
sets, Wölflein and Arandjelović [27] proposed a novel fine-tuning process for
their piece classifier to unseen chess sets. In our paper, same as with the
chessboard detection task, the classification of the pieces is performed by
the deep learning model, without the need to train a separate piece
classifier.

###  2.4 Chess Datasets

A common problem frequently mentioned in literature [8, 15, 6, 27] is the lack
of a comprehensive chess dataset. This issue hinders not only the ability to
fairly evaluate the proposed methods in a common setting but also impedes the
deployment of deep learning end-to-end approaches that require a vast amount
of data. One proposed solution to this problem is the use of synthetic
generated data. In [25], the authors produce point cloud data using a 3D
computer-aided design (CAD) model, while Blender [5] was used to produce
synthetic image datasets from a top view camera angle [16], or the player’s
perspective [27]. In our paper, we introduce the first chess recognition
dataset of real images, without setting any of the aforementioned restrictions
regarding the viewing angles.

##  3 Chess Recognition Dataset (ChessReD)

The availability of large-scale annotated datasets is critical to the
advancement of computer vision research. In this section, we tackle a main
issue in the field of chess recognition (_i.e_. the lack of a comprehensive
dataset) by presenting a novel dataset1 specifically designed for this task.
The dataset comprises a diverse collection of images of chess formations
captured using smartphone cameras; a sensor choice made to ensure real-world
applicability.

###  3.1 Data Collection and Annotation

The dataset was collected by capturing images of chessboards with various
chess piece configurations. To guarantee the variability of those
configurations, we relied upon the chess opening theory. The Encyclopedia of
Chess Openings (ECO) classifies opening sequences into five volumes with 100
subcategories each that are uniquely identified by an ECO code. We randomly
selected 20 ECO codes from each volume. Subsequently, each code of this set
was randomly matched to an already played chess game that followed the
particular opening sequence denoted by the ECO code; thus creating a set of
100 chess games. Finally, using the move-by-move information provided by
Portable Game Notations (PGNs) that are used to record chess games, the
selected games were played out on a physical chessboard with images being
captured after each move.

Three distinct smartphone models were used to capture the images. Each model
has different camera specifications, such as resolution and sensor type, that
introduce further variability in the dataset. The images were also taken from
diverse angles, ranging from top-view to oblique angles, and from different
perspectives (_e.g_. white player perspective, side view, etc.). These
conditions simulate real-world scenarios where chessboards can be captured
from a bystander’s arbitrary point of view. Additionally, the dataset includes
images captured under different lighting conditions, with both natural and
artificial light sources introducing these variations. Most of these
variations are illustrated in the four image samples of Figure 1. Each of
those samples highlights a different challenge in chess recognition.
Occlusions between pieces occur more often in images captured from a low angle
(Fig. 1(c)) or a player’s perspective (Fig. 1(b)), while pieces are rarely
occluded in top-view images (Fig. 1(d)). However, distinct characteristics of
pieces (_e.g_. the queen’s crown) that could aid the chess recognition task
are less distinguishable in a top-view.

The dataset is accompanied by detailed annotations providing information about
the chess pieces formation in the images. Therefore, the number of annotations
for each image depends on the number of chess pieces depicted in it. There are
12 category ids in total (_i.e_. 6 piece types per color) and the chessboard
coordinates are in the form of algebraic notation strings (_e.g_. “a8”). These
annotations were automatically extracted from Forsyth-Edwards Notations (FENs)
that were available to us by the games’ PGNs. Each FEN string describes the
state of the chessboard after each move using algebraic notation for the piece
types (_e.g_. “N” is knight) , capitalization for the piece colors (_i.e_.
white pieces are denoted with uppercase letters, while black pieces with
lowercase letters), and digits to denote the number of empty squares. Thus, by
matching the captured images to the corresponding FENs, the state of the
chessboard in each image was already known and annotations could be extracted.
To further facilitate research in the chess recognition domain, we also
provide bounding-box and chessboard corner annotations for a subset of 20
chess games. An annotated sample is presented in Figure 2. The different
colors for the corner points represent the four distinct corner annotations
(_i.e_. bottom-left, bottom-right, top-left, and top-right) that are relative
to the white player’s perspective. For instance, the corner annotated with the
red color in Figure 2 is a bottom-left corner. The discrimination between
these different types of corners provides information about the orientation of
the chessboard that can be leveraged to determine the image’s perspective and
viewing angle.

(a) Corner view

(b) Player view

(c) Low angle

(d) Top view

Figure 1: Image samples from ChessReD

###  3.2 Data Statistics

In this section, we present an overview of the ChessReD statistics. As
mentioned in Section 3.1, the dataset consists of 100 chess games, each with
an arbitrary number of moves and therefore images, amounting to a total of
10,800 images being collected. The dataset was split into training,
validation, and test sets following an 60/20/20 split. Since two consecutive
images of a chess game differ only by one move, the split was performed on
game-level to ensure that quite similar images would not end up in different
sets. The split was also stratified over the three distinct smartphone cameras
(Apple iPhone 12, Huawei P40 pro, Samsung Galaxy S8) that were used to capture
the images. Table 1 presents an overview of the image statistics per
smartphone. The three smartphone cameras introduced variations to the dataset
based on the distinct characteristics of their sensors. For instance, while
the image resolution for the Huawei phone was 3072x3072, the resolution for
the remaining two models was 3024x3024.

Smartphone | Number of images
---|---
Train | Val | Test
Apple iPhone 12 | 2,146 | 851 | 638
Huawei P40 pro | 2,102 | 638 | 871
Samsung Galaxy S8 | 2,231 | 703 | 620
Total | 6,479 | 2,192 | 2,129
Table 1: Overview of the image statistics

Table 2 presents an overview of the annotations in ChessReD. In
LABEL:tab:stats-anns-pos we can see that there is a significant imbalance
between provided annotations for the piece type “Pawn” and the rest of the
types in the dataset. This was to be expected since every chess game starts
with 8 pawns in each side and only one or two of the remaining piece types.
Regarding the colors of the pieces, no imbalance is detected in the dataset.
Additionally, while annotations about the position of the pieces in algebraic
notation are available for every image in the dataset, we provide bounding box
and chessboard corner annotations only for a subset of 20 randomly selected
games from the train, validation, and test sets. For this subset we followed a
70/15/15 split stratified over the smartphone cameras, which led to a total of
14 training games (1442 images), 3 validation games (330 images), and 3 test
games (306 images) being annotated. In LABEL:tab:stats-anns-bboxes we can see
an overview of the annotation statistics for this subset, named ChessReD2K.

Figure 2: Bounding box and corner point annotations in ChessReD2K

Piece type | Number of instances
---|---
Train | Val | Test
Black | White | Black | White | Black | White
Pawn | 35,888 | 35,021 | 11,410 | 11,042 | 11,616 | 11,472
Rook | 9,317 | 9,260 | 2,605 | 2,876 | 2,992 | 3,077
Knight | 6,158 | 6,471 | 2,222 | 2,206 | 2,032 | 2,202
Bishop | 6,681 | 6,768 | 2,167 | 2,003 | 2,301 | 2,067
Queen | 4,076 | 3,996 | 1,011 | 1,013 | 1,145 | 1,109
King | 6,479 | 6,479 | 2,192 | 2,192 | 2,129 | 2,129

(a)

Piece type | Number of instances
---|---
Train | Val | Test
Black | White | Black | White | Black | White
Pawn | 8,059 | 7,653 | 1,511 | 1,625 | 1,719 | 1,624
Rook | 2,293 | 2,250 | 471 | 447 | 433 | 433
Knight | 1,276 | 1,423 | 178 | 274 | 278 | 278
Bishop | 1,578 | 1,607 | 380 | 335 | 296 | 304
Queen | 862 | 838 | 125 | 126 | 157 | 160
King | 1,442 | 1,442 | 330 | 330 | 306 | 306

(b)

Table 2: Overview of the annotation statistics

##  4 End-to-End Chess Recognition

Unlike the conventional pipeline in chess recognition that involves chessboard
detection, square localization, and chess piece classification, the focus of
this study was to develop an end-to-end approach that tackles the recognition
task utilizing only a single image as input. Thus, the developed method should
take as input an image of a chessboard and output the type and the positions
of the pieces relative to the board. To this end, we experimented with two
different solutions by treating the problem either as a multi-class multi-
label classification or as a relative object detection task.

###  4.1 Classification Approach

In the classification approach, we treat each chessboard square as a distinct
label. Since there are 64 squares in each image, and thus 64 labels, this
problem is treated as a multi-label classification task. Additionally, each
square in the chessboard is either unoccupied or occupied by one of the 12
different types of pieces (_i.e_. 6 per color) in chess. Therefore, to each
label we assign one of 13 classes (_i.e_. 12 piece types and empty),
constituting this approach multi-label multi-class classification. By
formulating this approach as such, the goal is for the model to learn the
intricate relationships and visual patterns associated with the individual
squares.

###  4.2 Relative Object Detection Approach

In addition to the multi-class multi-label classification approach, we explore
a novel technique for chess recognition which we call relative object
detection. Contrary to conventional object detection methods that predict
bounding box coordinates in terms of absolute position in the image frame, our
modified method aims to predict the x and y coordinates of the objects
relative to the chessboard grid in the image. In this manner, discrete
coordinates that align with the chessboard positions are used to provide
spatial information of its layout. For instance, the relative position (0,0)
corresponds to the chessboard square denoted by “a8” in chess algebraic
notation. Furthermore, since we only need to predict the relative coordinates,
we can omit the height and width estimation, effectively bypassing the
complexities of the bounding box size estimation.

##  5 Implementation Details

###  5.1 Classification

For the classification approach, we employed a ResNeXt [28] model. Based on
the ResNet [9] architecture, ResNeXt models constitute a powerful variant that
can learn complex representations of images. Their introduced concept of
“cardinality” (_i.e_. the number of parallel branches used in each residual
block) both enables deeper architectures with reduced computation complexity
and allows complex representations to be learned by aggregating the
information of the multiple branches. Because of these modifications, this
family of models can achieve impressive results in image classification.

For our experiments, we trained the resnext101_32d variant, which uses a
cardinality of 32 and a width (_i.e_. number of filters) of 8. This means that
each residual block in the network contains 32 parallel convolutional layers,
and each of them has a width of 8. Additionally, there are 101 layers in the
network, which amount to a total of 88.8M parameters. We trained this model
from scratch for 200 epochs, with early stopping enabled and a batch size of 8
samples, using a cross-entropy loss function. We used an Adam [10] optimizer
with a learning rate of 0.001, which was reduced to 0.0001 after the 100th
epoch. The same training recipe was used to train ResNext on all of the
datasets in our experiments (Sec. 6).

###  5.2 Relative Object Detection

As mentioned in Section 4.2, the goal is to predict a set of coordinates x and
y for the chess pieces relative to the chessboard grid in the image. Thus,
traditional object detection models that use Region Proposal Networks (RPNs)
[20] or anchor boxes [19] are not suitable for this task, since they output
absolute image coordinates. However, a single end-to-end object detection
model, like Detection Transformer (DETR) [2], that directly predicts bounding
boxes and class labels for objects in an image could be employed. DETR uses a
transformer encoder-decoder architecture, with the encoder taking as input a
feature map produced by a convolutional backbone network and the decoder
generating the final predictions using self-attention mechanisms to attend to
different parts of the feature map.

For our experiments, we attempted to train a modified version of DETR that
predicts relative object coordinates and omits the height and width dimensions
for the bounding boxes of the traditional object detection task.
ResNext101_32d was used as a backbone network for feature extraction. We set
the number of queries (_i.e_. the maximum number of objects that DETR can
detect in an image) to 32, since each chessboard can have at most 32 chess
pieces on top of it. DETR also requires a separate class for “background”,
which in our case corresponds to “empty” squares. Thus, the number of classes
that the model is trained to predict is 13 (_i.e_. 12 piece types and
background). We trained the model from scratch for a total of 800 epochs, with
early stopping enabled and a batch size of 8 samples, using DETR’s default
bipartite matching loss for set predictions, which takes into account both the
class prediction and the similarity of the predicted and ground truth
coordinates. We used an AdamW [13] optimizer with separate learning rates for
the backbone network and the encoder-decoder architecture. In particular, the
initial learning rates were set to $10^{-5}$ and $10^{-6}$ for the encoder-
decoder and backbone, respectively, and a scheduler was used to reduce both by
a factor of 10 every 300 epochs. Furthermore, gradient clipping was used with
a threshold of $0.1$.

However, the training of this modified DETR variant for chess recognition did
not yield optimal results, with the model being unable to successfully detect
chess pieces in the images of ChessReD. This issue could potentially be linked
to DETR’s inherent limitation in detecting small objects [2, 31], especially
when considering the intricacies of the dataset (_e.g_. occlusions) and the
relatively small sizes of individual pieces. Due to the unsuccessful
convergence of the DETR variant, it will not be used in the experiments of
Section 6. Nevertheless, end-to-end relative object detection with
transformers is a promising area that should be further investigated, with the
focus being on refining the model architecture ([31]) or the training
objective.

##  6 Experiments

###  6.1 Exp1: Comparison with the state-of-the-art

To the best of our knowledge, the current state-of-the-art approach in chess
recognition, namely Chesscog, was introduced in Wölflein and Arandjelović
[27]. In their experiments, Chesscog achieved a 93.86% accuracy in chess
recognition on a synthetic dataset [26] rendered in Blender[5], with a 0.23%
per-square error rate. Additionally, the authors introduced a few-shot
transfer learning approach to unseen chess sets and the system demonstrated a
88.89% accuracy and 0.17% per-square error rate, when tested on a set of
previously unseen images of chessboards. In this section, we will compare the
performance of our approach with that of Chesscog’s, both on their Blender
dataset and on our newly introduced ChessReD.

####  6.1.1 Current SOTA: Chesscog

Chesscog attempts to solve the chess recognition task using a pipeline that
involves chessboard detection, square localization, occupancy classification,
and piece classification. It leverages the geometric nature of the chessboard
to detect lines and employs a RANSAC-based algorithm to compute a projective
transformation of the board onto a regular grid. Subsequently, individual
squares are localized based on the intersection points and an occupancy
classifier is used on each individual square. Finally, the pieces on the
occupied squares are classified into one of 12 classes, using a pre-trained
piece classifier. The piece classifier is used on image patches of the squares
that are cropped based on a heuristic approach that extends the bounding boxes
based on the square’s location on the chessboard. It is also important to
mention that during inference the user must manually input the specific
player’s perspective (_i.e_. “white” or “black”) to determine the orientation
of the board.

####  6.1.2 How does the classification approach compare to Chesscog on the
synthetic Blender dataset?

First, we evaluate the performance of our classification approach on the
Blender dataset and compare it with that of Chesscog. The Blender dataset
comprises a set of 4,888 synthetic chessboard images with distinct piece
configurations, multiple lighting conditions, a limited range of viewing
angles (between 45° and 60° to the board), and images taken only from the
players’ perspectives. We trained our ResNeXt model following the recipe
described in Section 5.1 on the dataset’s training samples. Subsequently, we
evaluated our trained model’s performance on the test set. The first two
columns of Table 3 demonstrate the evaluation results for both approaches on
the Blender dataset. We use the same evaluation metrics as in Wölflein and
Arandjelović [27].

Chesscog outperforms our classification approach across all metrics. For the
percentage of boards with no mistakes, which reveals a model’s ability to
achieve perfect board recognition, Chesscog demonstrates a significant
advantage with 93.86% of boards correctly predicted, while ResNeXt achieves
this only in 39.76% of the boards. When one mistake is allowed per board
prediction, Chesscog can successfully recognize almost all of the boards, with
ResNeXt’s performance improving significantly and reaching 65.2%. Chesscog’s
superiority is also corroborated by the substantially lower mean number of
incorrect squares per board (0.15 vs. 1.19 for ResNeXt) and per-square error
rate (0.23% vs. 1.86% for ResNeXt).

| Blender Dataset | ChessReD | ChessReD*
---|---|---|---
Metric | Chesscog | ResNeXt | Chesscog | ResNeXt | Chesscog | ResNeXt
Mean incorrect squares per board | 0.15 | 1.19 | 42.87 | 3.40 | 12.96 | 3.35
Boards with no mistakes (%) | 93.86% | 39.76% | 2.30% | 15.26% | 6.69% | 15.30%
Boards with $\leq 1$ mistake (%) | 99.71% | 65.20% | 7.79% | 25.92% | 22.67% | 27.04%
Per-square error rate (%) | 0.23% | 1.86% | 73.64% | 5.31% | 39.57% | 5.24%
Table 3: Performance evaluation for Chesscog’s and our classification
approach’s (ResNeXt) predictions on the corresponding test sets. ChessReD*
represents the subset of the test images in which Chesscog could detect the
chessboard.

####  6.1.3 How does the classification approach compare to Chesscog on the
real ChessReD dataset?

In this section, we compare the performance of our approach with that of
Chesscog’s on our ChessReD. We trained our ResNeXt model, using again the
recipe of Section 5.1, and finetuned the Chesscog classifiers as mentioned in
[27], using two images of the starting position from both players’
perspectives. Furthermore, for a fair comparison we needed to take into
account that Chesscog cannot infer the orientation of the chessboard and
requires for it to be manually inputted. Since this information is not
available in our dataset, we address it by generating all possible
orientations for the detected chessboards during evaluation.

Both approaches faced increased challenges when tested on ChessReD, resulting
in a performance drop across all metrics, as seen in Table 3. While our
ResNeXt model can still demonstrate competitive results, recognizing
successfully 15.26% of boards with no mistakes and 25.92% of boards with less
than one mistake, Chesscog’s accuracy decreases significantly, achieving only
2.3% and 7.79% in these metrics, respectively. Chesscog’s performance
deterioration is also evident by its 42.87 incorrect squares per board on
average and the 73.65% per-square error rate. ResNeXt’s performance for these
metrics was 3.40 and 5.31%, respectively.

Upon further investigation, one important factor that led to Chesscog’s
performance degradation was the inaccurate results of the chessboard detection
process and the accumulation of the error throughout the pipeline. While the
limited range of angles present in the Blender dataset of the previous section
enabled Chesscog to achieve 100% accuracy in chessboard detection, the
corresponding accuracy in our dataset is 34.38%. This issue highlights the
sensitivity of the image processing algorithms employed for chessboard
detection to their hyperparameters and the necessity to finetune them across
different datasets.

To further compare the performance of both approaches, we conducted the same
evaluation without taking into account the failed chessboard detections by
Chesscog. In the last two columns of Table 3, we evaluate the performance of
both approaches on the subset of the ChessReD’s test set (denoted as
ChessReD*) consisting of the 34.38% (732) of the images in which Chesscog was
able to detect the chessboard. While Chesscog’s performance shows significant
improvement when we don’t consider those erroneous chessboard detections, it
remains inferior in comparison to the results achieved by our classification
approach across all metrics.

###  6.2 Exp2: Does the classification approach rely on chessboard marks for
recognition?

(a) With marks

(b) Without marks

Figure 4: Sample pair of images for the ablation study

One significant advantage of our approaches is that they do not require any
further input to determine the orientation of the chessboard in an image. Yet,
the visual cues used by the models to deduce the chessboard’s orientation
remain unclear. To this end, we conducted an ablation study to investigate
whether the ResNeXt model relies on specific marks of the chessboard (_e.g_.
bishop logo in Fig. 4(a)) to determine the board’s orientation and
subsequently recognize the chess configuration.

We hypothesize that the necessity of those marks for successful chess
recognition increases with the number of moves that have been made prior to
capturing the image. The intuition behind this hypothesis is that in the early
game of chess, the majority of the pieces remain in their starting position,
so determining the boards orientation poses less of a challenge, while in the
end game, only a few pieces remain on the board and they are usually far from
their starting position. To validate this hypothesis, we created a dataset
consisting of 30 test images that were randomly selected from the subset of
images that the model was able to successfully recognize in the evaluation of
Section 6.1.3. The test images were evenly distributed across three
categories: early-game, mid-game, and end-game. These categories correspond to
images that were taken when less than 30, more than 30 but less than 75, or
more than 75 moves had been made prior to capturing the images, respectively.
Subsequently, we manually removed the marks, such as the bishop logo or the
algebraic chess notation on the sides, and evaluated again the performance of
ResNeXt on this subset of 30 modified images. A sample pair of images is
illustrated in Figure 4. The model achieved an overall accuracy (_i.e_. boards
with no mistakes) of 66.6% on this subset, with a perfect recognition in the
early-game images, 60% accuracy in mid-game images, and 40% accuracy in end-
game images.

##  7 Discussion

The evaluation on the Blender dataset revealed that Chesscog outperforms our
classification approach. However, further experimentation on our newly
introduced ChessReD showcased a shift in both methods’ performances, with
ResNeXt surpassing Chesscog across all metrics. It is evident that Chesscog’s
low chessboard detection rate (34.38%), which is attributed to the diverse
angles and occlusions introduced by our dataset, significantly contributed to
that shift, while the specific range of angles used in the Blender dataset,
enabled Chesscog to successfully detect the chessboard in all cases and
achieve a remarkable end-to-end performance.

The ablation study provided significant insights into our ResNeXt model’s
reliance on specific marks for determining chessboard orientation, and
therefore chess recognition. The study confirmed our hypothesis that the
necessity of those marks increases with the number of moves made prior to
image capture. The model achieved higher accuracy in early-game images, where
most of the pieces remained in their starting positions, and lower accuracy in
end-game images, where only a few pieces were still on the board and farther
from their starting positions. While depending on such marks could be
challenging in cases where they are absent or obscured, it could prove to be
an advantage in end-game states in which even human annotators can have
trouble determining the board’s orientation without them.

##  8 Limitations

While our study sheds light on the importance of end-to-end deep learning
approaches for chess recognition, the limitations of these solutions should
also be considered. An inherent weakness of the classification approach is its
inability to recognize labels that are absent from the dataset that it was
trained on. For instance, if a specific piece/square combination was first
seen at inference time, the model would be unable to assign the corresponding
label. On the other hand, the relative object detection approach would not
encounter this issue, but as a transformer-based solution it’s difficult to
converge when trained on a small dataset. Finally, finetuning these models on
previously unseen data would require considerably more resources and compute
time compared to finetuning a simple CNN piece classifier in the sequential
approaches.

Regarding ChessReD, although including a single chess set in the images was a
design choice, this lack of diversity impedes the development of solutions
with broader applicability. Yet, it is feasible to enhance the dataset by
collecting varied data with relative positional annotations (_i.e_. FEN
strings instead of bounding boxes) from chess tournaments recordings, where
the players are obliged to annotate their every move.

##  9 Conclusion

Our experiments demonstrate the effectiveness of our classification approach
in chess recognition tasks, while also revealing Chesscog’s advantages on
certain datasets. However, with the focus being on real-world applicability,
the ChessReD dataset, consisting of real images with varied angles and
perspectives, poses a more challenging benchmark for chess recognition, and
thus the experimental results establish our approach as the state-of-the-art
method for this task. Moving forward, improving the model’s ability to
generalize by either enhancing the dataset, or incorporating domain adaptation
techniques, should be explored. Additionally, the relative object detection
approach, if converged, may constitute a more robust solution for chess
recognition, and thus requires further studying.

## References

  * [1] Nandan Banerjee, Debal Saha, Atikant Singh, and Gautam Sanyal.  A simple autonomous robotic manipulator for playing chess against any opponent in real time.  In Proceedings of the International Conference on Computational Vision and Robotics, 2011.
  * [2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * [3] Andrew Tzer-Yeu Chen, I Kevin, and Kai Wang.  Computer vision based chess playing capabilities for the baxter humanoid robot.  In 2016 2nd International Conference on Control, Automation and Robotics (ICCAR), pages 11–14. IEEE, 2016.
  * [4] Andrew Tzer-Yeu Chen and Kevin I-Kai Wang.  Robust computer vision chess analysis and interaction with a humanoid robot.  Computers, 8(1):14, 2019.
  * [5] Blender Online Community.  Blender - a 3D modelling and rendering package.  Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018.
  * [6] Maciej A Czyzewski, Artur Laskowski, and Szymon Wasik.  Chessboard and chess piece recognition with the support of neural networks.  Foundations of Computing and Decision Sciences, 45(4):257–280, 2020\.
  * [7] Cheryl Danner and Mai Kafafy.  Visual chess recognition.  URL: http://web. stanford. edu/class/ee368/Project_Spring_1415/Reports/Danner_Kafafy. pdf, 2015.
  * [8] Jialin Ding.  Chessvision: Chess board and piece recognition.  In Tech. rep. Stanford University, 2016.
  * [9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [10] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [11] Paweł Kołosowski, Adam Wolniakowski, and Kanstantsin Miatliuk.  Collaborative robot system for playing chess.  In 2020 International Conference Mechatronic Systems and Materials (MSM), pages 1–6. IEEE, 2020.
  * [12] Can Koray, Emre Sumer, V Struc, et al.  A computer vision system for chess game tracking.  In 21st Computer Vision Winter Workshop, Rimske Toplice, Slovenia, 2016.
  * [13] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [14] Cynthia Matuszek, Brian Mayton, Roberto Aimi, Marc Peter Deisenroth, Liefeng Bo, Robert Chu, Mike Kung, Louis LeGrand, Joshua R Smith, and Dieter Fox.  Gambit: An autonomous chess-playing robotic system.  In 2011 IEEE International Conference on Robotics and Automation, pages 4291–4297. IEEE, 2011.
  * [15] Anav Mehta.  Augmented reality chess analyzer (archessanalyzer): In-device inference of physical chess game positions through board segmentation and piece recognition using convolutional neural network.  arXiv preprint arXiv:2009.01649, 2020.
  * [16] Afonso de Sá Delgado Neto and Rafael Mendes Campello.  Chess position identification using pieces classification based on synthetic images generation and deep neural network fine-tuning.  In 2019 21st Symposium on Virtual and Augmented Reality (SVR), pages 152–160. IEEE, 2019.
  * [17] Jason E Neufeld and Tyson S Hall.  Probabilistic location of a populated chessboard using computer vision.  In 2010 53rd IEEE International Midwest Symposium on Circuits and Systems, pages 616–619. IEEE, 2010.
  * [18] David Mallasén Quintana, Alberto Antonio del Barrio García, and Manuel Prieto Matías.  Livechess2fen: A framework for classifying chess pieces based on cnns.  arXiv preprint arXiv:2012.06858, 2020.
  * [19] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016.
  * [20] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [21] Emir Sokic and Melita Ahic-Djokic.  Simple computer vision system for chess playing robot manipulator as a project-based learning example.  In 2008 IEEE International Symposium on Signal Processing and Information Technology, pages 75–79. IEEE, 2008.
  * [22] K.Y. Tam, J.A. Lay, and D. Levy.  Automatic grid segmentation of populated chessboard taken at a lower angle view.  In 2008 Digital Image Computing: Techniques and Applications, pages 294–299, 2008.
  * [23] David Urting and Yolande Berbers.  Marineblue: A low-cost chess robot.  In Robotics and Applications, pages 76–81, 2003.
  * [24] Victor Wang and Richard Green.  Chess move tracking using overhead rgb webcam.  In 2013 28th International Conference on Image and Vision Computing New Zealand (IVCNZ 2013), pages 299–304. IEEE, 2013.
  * [25] Yu-An Wei, Tzu-Wei Huang, Hwann-Tzong Chen, and JenChi Liu.  Chess recognition from a single depth image.  In 2017 IEEE International Conference on Multimedia and Expo (ICME), pages 931–936. IEEE, 2017.
  * [26] G Wölflein and O Arandjelovic.  Dataset of rendered chess game state images.  Open Science Framework, 2021.
  * [27] Georg Wölflein and Ognjen Arandjelović.  Determining chess game state from an image.  Journal of Imaging, 7(6):94, 2021.
  * [28] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492–1500, 2017.
  * [29] Youye Xie, Gongguo Tang, and William Hoff.  Chess piece recognition using oriented chamfer matching with a comparison to cnn.  In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 2001–2009. IEEE, 2018.
  * [30] Youye Xie, Gongguo Tang, and William Hoff.  Geometry-based populated chessboard recognition.  In Tenth International Conference on Machine Vision (ICMV 2017), volume 10696, pages 9–13. SPIE, 2018.
  * [31] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  arXiv preprint arXiv:2010.04159, 2020.