# RTHDet: Rotate Table Area and Head Detection in images

Wenxing Hu
School of Electronics & Information Engineering
Shanghai University of Electric Power
Shanghai 200090, China
<EMAIL_ADDRESS>
Minglei Tong
School of Electronics & Information Engineering
Shanghai University of Electric Power
Shanghai 200090, China
<EMAIL_ADDRESS>

###### Abstract

Traditional models and datasets primarily concentrated on horizontal table
detection, unable to accurately detect table regions and localize their head
and tail parts in rotating scenarios, thus greatly restricting the development
of table recognition in rotated contexts. Therefore, this paper presents a new
task of detecting table regions and localizing their head and tail parts in
rotation scenarios, along with the proposal of corresponding datasets,
evaluation metrics, and methods. An innovative method——Adaptively Bounded
Rotation, is introduced to address the lack of datasets for detecting rotated
table and their head-tail parts. Through this method, this study has produced
TRR360D, a rotated table detection dataset which incorporates semantic
information of table head and tail, based on the classic horizontal table
detection dataset, ICDAR2019MTD. Furthermore, the study proposes a new
evaluation metric, R360 AP, to validate the precision of rotated region
detection and head-tail localization, building on the classic evaluation
metric, AP. Following extensive literature review and rigorous
experimentation, the high-speed and high-accuracy rotated detection model,
RTMDet-S, was selected as the baseline. Furthermore, this study proposed the
RTHDet model. RTHDet defines the $D_{\\_}{r360}$ rotated rectangle angle
representation based on the baseline and applies it to the newly added AL
(Angle Loss) branch. This enhancement allows the model to locate the head and
tail of the rotated table. By applying transfer learning methods and adaptive
boundary random rotation data augmentation techniques, the AP50 (T<90) of
RTHDet has been improved from 23.7% to 88.7% compared to the baseline model.
This validates the effectiveness of RTHDet in handling the novel task of
detecting rotating table regions and accurately localizing their head and tail
parts.RTHDet is integrated into the widely-used open-source MMRotate toolkit:
https://github.com/open-mmlab/mmrotate/tree/dev-1.x/projects/RR360.

_K_ eywords Dataset $\cdot$ Object Detection $\cdot$ Table Detection $\cdot$
Rotated Detection $\cdot$ Head-Tail Detection

##  1 Introduction

The vast amount of data from the objective world is concealed in ubiquitous
paper documents and forms, such as financial statements, express waybills,
medical examination reports, and exam papers. Extracting valuable data from
these tables requires the development of theories and technologies for table
digitization and intelligence. This is crucial for enhancing efficiency in
finance, education, and logistics industries, and for accelerating their
intellectual development. However, when the paper world is converted into
digital images, various rotate, affine, perspective transformations, and
geometric distortions emerge, posing significant challenges for the
localization and analysis of table regions.

Traditional models and datasets mainly focus on the detection of horizontal
tables as shown in Fig.1 (a), failing to precisely detect table regions in
rotated scenarios as depicted in Fig.1 (b), and even more incapable of
locating the head and tail of tables as illustrated in Fig.1 (c). This greatly
restricts the development of downstream tasks in table recognition OCR.
Therefore, this paper introduces a new task for detecting table regions in
rotated scenes and locating heads and tails, and conducts in-depth research on
it. The main contributions of this paper in proposing the new task of rotated
table region detection and head-tail localization are as follows:

Figure 1: Table Detection Task: (a) Detection of Table Horizontal Regions (b)
Detection of Table Rotation Regions (c) Detection of Table Head and Tail

(1) TRR360D Dataset: In response to the lack of a dedicated dataset for
rotated table images in existing table detection research, this paper presents
a novel dataset generation method called Adaptively Bounded Rotation. Drawing
upon the ICDAR2019MTD[1] modern table detection dataset, the QBox (Quadrangle
Box) annotation format from the DOTA[2] dataset is adopted to represent
rotated boxes. By applying the Adaptively Bounded Rotation technique, the
starting point and annotation direction of the QBox are regulated, thereby
generating the TRR360D rotating table detection dataset, which inherently
carries semantic information of the table’s head and tail. This dataset
comprises 840 images with 1446 rotated box annotations and is publicly
available at https://github.com/vansinhu/TRR360D.

(2) R360 AP Evaluation Metric: To address the limitation of the traditional AP
evaluation metric, which only measures the accuracy of rotating table region
predictions but does not account for the precision of head and tail
localization, this paper introduces the R360 AP evaluation metric. Building
upon the AP metric, the R360 AP metric incorporates angle constraints to
assess the performance of both rotating table region detection and head-tail
localization accuracy.

(3) RTHDet Model: In response to the issue of existing models being unable to
distinguish between the head and tail of rotated tables, the RTHDet (Rotate
Table Head Detector) model is introduced. This model builds upon the RTMDet-S
[3] baseline model and proposes the $D_{\\_}{r360}$ rotation rectangle box
angle representation method. This method is applied to the novel AL (Angle
Loss) branch, enabling the detection of table regions and head-tail parts in
rotating scenarios. By utilizing transfer learning and adaptive boundary
random rotation data enhancement techniques, the RTHDet model achieves an
AP50(T<90) score of 88.7%, representing a substantial improvement of 65.0
percentage points compared to the RTMDet-S baseline model. This demonstrates
RTHDet’s ability to effectively detect rotating table regions and precisely
localize their head and tail parts. RTHDet is integrated into the widely-used
open-source MMRotate toolkit: https://github.com/open-
mmlab/mmrotate/tree/dev-1.x/projects/RR360.

The structure of the paper is as follows: the first section serves as an
introduction, the second section covers the related work, the third section
presents the methodology, the fourth section discusses the experimental
results, and the fifth section provides a summary of the entire paper.

##  2 Related Work

In this section, an overview of the related work on horizontal table detection
in document images is provided, categorized into stages of rule-based methods
and deep learning-based methods. The limitations of horizontal table detection
algorithms in accurately detecting rotated table regions and localizing their
head and tail are analyzed. Furthermore, a brief introduction is given to the
impact of classical and state-of-the-art methods for detecting rotated regions
and their limitations when directly applied to table detection in the field.

###  2.1 Horizontal Table Detection

####  2.1.1 Rule-Based Method

In early-stage research, rule-based methods were commonly employed. Itonori et
al. [4] addressed the issue of table detection in document images using a
rule-based method. Their approach harnessed the arrangement of text blocks and
the position of ruling lines to identify tables within documents. Chandran and
Kasturi [5] proposed an alternative method focusing on ruling lines for table
detection. Similarly, Pyreddy and Croft [6] developed a heuristics-based table
detection method that initially identified structural elements from a document
and subsequently filtered the tables.

Researchers have also developed tabular layouts and grammar for detecting
tables in documents [7]. One such method predicts tables using the correlation
of white spaces and vertical connected component analysis [8]. Another
approach, introduced by Pivk et al. [9], converts tables found in HTML
documents into a logical structure. Shigarov et al. [10] exploit metadata from
PDF files, treating each word as a text block. Their method restructures
tabular boundaries by leveraging bounding boxes for each word. For a more in-
depth understanding of these rule-based methods, readers are encouraged to
consult references [11].

While early rule-based systems can detect tables in documents with limited
patterns, they rely on manual intervention to search for optimal rules.
Furthermore, their propensity to produce generic solutions poses a
challenge.Rule-based methods are not sufficiently robust and are limited to
specific scenarios, unable to handle the challenges posed by diverse table
styles. In contrast, deep learning-based approaches, as described in the
following sections, effectively overcome these limitations in horizontal table
detection.

####  2.1.2 Deep Learning Method

With the advancement of deep learning, an increasing number of researchers
have turned to it to address horizontal table detection challenges.The
progress of object detection networks in computer vision has been shown to
have a direct correlation with advancements in table detection in document
images. Gilani et al. [12] demonstrated the applicability of Faster R-CNN [13]
to table detection in document images, treating it as an object detection
problem. Their work involved using distance transform methods to modify the
pixels in raw document images fed to the Faster R-CNN.

Building upon this foundation, Schreiber et al. [14] proposed another method
that utilized Faster R-CNN [13], in conjunction with pre-trained base
networks, such as ZFNet [14] and VGG-16 [15], to detect tables in document
images. Siddiqui et al. [16] also developed a Faster R-CNN-based approach that
implemented deformable convolutions [17] to tackle table detection with
arbitrary layouts. In addition, Reference [18] adopted a Faster R-CNN with a
corner-correlating approach to enhance the prediction of tabular boundaries in
document images.

Saha et al. [19] conducted an empirical study that revealed Mask R-CNN [20]
outperforms Faster R-CNN in detecting tables, figures, and formulas. This
finding was further corroborated by Zhong et al. [21] who used Mask R-CNN to
localize tables. Additionally, YOLO [22], SSD [23], and RetinaNet [24] have
been employed to demonstrate the advantages of closed-domain fine-tuning for
table detection in document images.

More recently, cutting-edge object detection algorithms, such as Cascade Mask
R-CNN [25], Hybrid Task Cascade (HTC) [26] and DetectoRS[27], have been
incorporated to enhance the performance of table detection systems in document
images [28, 29, 30, 31, 32]. Despite the advancements made by these prior
methods, there remains significant potential for improving the localization of
accurate tabular boundaries in scanned document images. Furthermore, existing
table detection techniques often rely on more substantial backbones or memory-
intensive deformable convolutions. Ma et al. enhanced Faster R-CNN’s accuracy
for table detection by leveraging CornerNet in their RobusTabNet [33] method,
demonstrating superior performance on various benchmarks with a basic
ResNet-18 network.

These methods primarily support the detection of horizontally aligned tables
but do not cater to rotated table detection. They also lack the capability to
differentiate between the head and tail sections of tables.

###  2.2 Rotate Table Area Detection

Traditional object detection algorithms, such as RetinaNet[24] and
CascadeRCNN[34], can only detect the horizontal bounding boxes of table
objects, and cannot detect the rotated bounding boxes of tables. In recent
years, the field of rotated object detection has also made significant
progress. Subsequently, an introduction to the representation of horizontal
objects and the representation of rotated object regions will be provided.
This will be followed by an analysis of the limitations of existing rotation
detection algorithms when applied to table region detection.

Traditional object detection algorithms predict parameters using center width
and height representation, denoted as $[x,y,w,h]$. However, in the field of
rotated table detection, an additional parameter $\theta$ is required.
Previous work on the DOTA dataset used four ways to define $\theta$:
$D_{\\_}{oc}$ shown in Figure 2, $D_{\\_}{oc}^{\prime}$ shown in Figure 3,
$D_{\\_}{le90}$ shown in Figure 4, and $D_{\\_}{le135}$ shown in Figure 5.
Although these representations can represent the rotated region of the table,
they cannot convey the semantics of the four corners of the rotated region,
nor can they represent the head and tail of the table.

Figure 2: $D_{\\_}{oc}$:the old version OpenCV definition,when $OpenCV<4.5.1$,
$angle\in[-90,0^{\circ})$, $\theta\in[-\frac{\pi}{2},0)$, and the angle
between the width and the positive x-axis is a positive acute angle. At this
time, the width edge will exchange as the angle changes. This definition comes
from the cv2.minAreaRect function in OpenCV, which returns a value in the
range $[-90^{\circ},0)$. Figure 3: $D_{\\_}{oc}^{\prime}$:the new version of
OpenCV,when $OpenCV\geq 4.5.1$,
$angle\in(0,90^{\circ}]$,$\theta\in(0,\frac{\pi}{2}]$, and the angle between
width and the positive x-axis is acute (positive), based on the
$cv2.minAreaRect$ function in OpenCV, which returns a value in the range of
$(0,90^{\circ}]$. Figure 4: $D_{\\_}{le90}$: Long Edge 90 Definition,
$angle\in[-90^{\circ},90^{\circ}]$, $\theta\in[-\frac{\pi}{2},\frac{\pi}{2}]$,
and $width>height$. Figure 5: $D_{\\_}{le135}$: Long Edge 135
Definition,$angle\in[-45^{\circ},135^{\circ}]$,
$\theta\in[-\frac{\pi}{4},\frac{3\pi}{4}]$, and $width>height$.

Therefore, the rotation region detection models in MMRotate, such as
RetinaNet-R, CascadeRCNN-R, and RTMDet-S, suffer from the limitation of angle
definition, which prevents them from accurately localizing the head and tail
parts of the rotated regions. However, the semantic information of the head
and tail parts is crucial for improving the accuracy of downstream tasks like
table recognition and OCR.In Section 3.1, a new angle representation will be
proposed and applied to existing rotation region detection models to enable
the localization of the head and tail parts. Thus, it propels the development
of downstream tasks.

##  3 Method

By conducting experiments as described in Section 4.2, the RTMDet-S model was
selected as the baseline for rotating table region detection due to its
superior speed and accuracy.The RTMDet model is an improved version of the
object detection algorithm YOLOX[35], which is very similar to YOLOX in
overall structure and consists of three parts: CSPNeXt, CSPNeXtPAFPN, and
SepBNHead. CSPNeXt and CSPNeXtPAFPN are used to extract features, and
SepBNHead is used for detection. Like YOLOX, the core module in RTMDet is also
CSPLayer, but the Basic Block is improved to CSPNeXt Block.

Figure 6: RTHDet Diagram

To address the limitation of the baseline model in localizing the head and
tail parts of rotated regions, a new angle definition method is proposed in
this section, which is applied to the AL (Angle Loss) branch of the RTMDet-S
model. The resulting model is named RTHDet, as illustrated in Figure 6.
Additionally, the Adaptively Bounded Rotation method is proposed for the
generation of the dataset in Section 4.1.2 and for random data augmentation
during the training process.

###  3.1 Definition of R360 Rotation Box Angle

The four angle definition methods mentioned in section 2.2, namely
$D_{\\_}{oc}$, $D_{\\_}{oc}^{\prime}$, $D_{\\_}{le90}$, and $D_{\\_}{le135}$,
are unable to distinguish between the head and tail of objects, such as
tables. Additionally, the angle range that can be represented is limited to
$180^{\circ}$, meaning that it cannot distinguish between 0° and -180°, as
illustrated in Figure 7. To address these limitations, this study introduces
the RBox (Rotated Bounding Box) using the R360 angle definition method,
denoted as $D_{\\_}{r360}$. The Rbox of a rotated rectangle bounding box is
defined by Equation 1, with an angle range of [-180°, 180°). Furthermore, the
corresponding QBox quadrilateral bounding box for RBox can be defined using
Equation 2.

Figure 7:  $D_{\\_}{r360}$:
$angle\in[-180^{\circ},180^{\circ})$,$\theta\in[-\pi,\pi)$

$RBox=(cx,cy,w,h,\theta)$ (1)

$QBox=[(x_{\\_}A,y_{\\_}A),(x_{\\_}B,y_{\\_}B),(x_{\\_}C,y_{\\_}C),(x_{\\_}D,y_{\\_}D)]$
(2)

####  3.1.1 QBox2RBox

As shown in Equation 3, the $cv2.minAreaRect$ function in OpenCV is able to
calculate the center point $(cx,cy)$ and the width and height $(w,h)$ of a
rotated rectangle, given a set of points. However, the resulting angle range
is limited to $angle\in(0,90^{\circ}]$, which corresponds to the angle
definition in the new version of OpenCV denoted as $D_{\\_}{oc^{\prime}}$.
This method is only capable of representing the rotated area of a table and
cannot be used to distinguish between the table head and tail.

$((cx,cy),(w,h),angle)=cv2.minAreaRect(QBox)$ (3)

Therefore, in this research, the value of $\theta$ is calculated using the
inverse trigonometric functions in numpy as shown in Equation 4. The domain of
$\theta$ is $[-\pi,\pi)$, expressed in radians.

$\theta=np.arctan2(y_{\\_}C-y_{\\_}B,x_{\\_}C-x_{\\_}B)$ (4)

####  3.1.2 RBox2QBox

The relationship between RBox and QBox is shown in Figure 8, and all the
parameters of QBox can be obtained from RBox through Equation 5, 6, 7, and 8.

Figure 8: Relationship between RBox and Qbox

$(x_{\\_}{A},y_{\\_}{A})=(cx+\frac{h}{2}sin\theta-\frac{w}{2}cos\theta,cy-\frac{h}{2}cos\theta-\frac{w}{2}sin\theta)$
(5)

$(x_{\\_}{B},y_{\\_}{B})=(cx+\frac{h}{2}sin\theta+\frac{w}{2}cos\theta,cy-\frac{h}{2}cos\theta+\frac{w}{2}sin\theta)$
(6)

$(x_{\\_}{C},y_{\\_}{C})=(cx-\frac{h}{2}sin\theta+\frac{w}{2}cos\theta,cy+\frac{h}{2}cos\theta+\frac{w}{2}sin\theta)$
(7)

$(x_{\\_}{D},y_{\\_}{D})=(cx-\frac{h}{2}sin\theta-\frac{w}{2}cos\theta,cy+\frac{h}{2}cos\theta-\frac{w}{2}sin\theta)$
(8)

###  3.2 AL Branch

The RTMDet algorithm has four models, namely L (Large), M (Medium), S (Small),
and T (Tiny). The baseline model of RTMDet in MMRotate uses the rotated IoU
loss function shown in Equation 9 to regress the rotated bounding boxes
(RBox), where RIoU is illustrated in Figure 13. However, the limitation of
$\mathcal{L}_{\\_}{Rbox}$ is that it can only learn the features of rotated
regions and cannot learn the $360^{\circ}$ rotation angle features that
include table head and tail information.

$\mathcal{L}_{\\_}{RBox}=1-RIoU$ (9)

To facilitate the learning of 360° rotation features, an Angle L1 Loss (AL)
branch was incorporated. In conjunction with the $D_{\\_}{r360}$ rotation
rectangle box angle definition method defined earlier, the AL branch employs
the loss function depicted in Equation 10 to constrain angle learning. This
adaptation enables the RTMDet model to support $360^{\circ}$ rotation table
detection and consequently head-tail location.

$\mathcal{L}_{\\_}{Angle}=\left|Angle_{\\_}{P}-Angle_{\\_}{G}\right|$ (10)

###  3.3 Adaptively Bounded Rotation

The original OpenCV rotation algorithm 1 suffers from the issue of table image
region loss, as illustrated in Figure 9 (a). To address this issue, an
adaptive boundary rotation transformation algorithm is proposed in this study.
The algorithm flow is illustrated in Algorithm 2, and the transformation
effect is displayed in Figure 9 (b).

Algorithm 1 OpenCV Original Rotation

1:image, $\theta$, points: List[(x,y)]

2:r_image, r_points: List[(x,y)]

3:h$\leftarrow$image.height

4:w$\leftarrow$image.weight

5:matrix$\leftarrow$cv2.getRotationMatrix2D(center, -angle, 1)

6:rimage$\leftarrow$cv2.warpAffine(image, matrix, (w, h))

7:pts $\leftarrow$ points.reshape([-1, 2])

8:pts $\leftarrow$ np.hstack([pts, np.ones([len(pts), 1])]).T

9:points $\leftarrow$ matrix@points

10:rpoints $\leftarrow$ [[points[0][x],points[1][x]] for x in
range(len(points[0]))]

11:return r_image, r_points

Algorithm 2 Adaptively Bounded Rotation

1:image, $\theta$, points: List[(x,y)]

2:r_image, r_points: List[(x,y)]

3:h$\leftarrow$image.height

4:w$\leftarrow$image.weight

5:matrix$\leftarrow$cv2.getRotationMatrix2D(center, -angle, 1)

6:cos = abs(matrix[0, 0])

7:sin = abs(matrix[0, 1])

8:neww = h * sin + w * cos

9:newh = h * cos + w * sin

10:matrix[0, 2]$\leftarrow$matrix[0, 2]+(neww - w) * 0.5

11:matrix[1, 2]$\leftarrow$matrix[1, 2]+ (newh - h) * 0.5

12:rimage$\leftarrow$cv2.warpAffine(image, matrix, (new_w, new_h))

13:pts $\leftarrow$ points.reshape([-1, 2])

14:pts $\leftarrow$ np.hstack([pts, np.ones([len(pts), 1])]).T

15:points $\leftarrow$ matrix@points

16:rpoints $\leftarrow$ [[points[0][x],points[1][x]] for x in
range(len(points[0]))]

17:return r_image, r_points

Figure 9: Rotation transformation

##  4 Experiment

This section is divided into three major parts. Firstly, it introduces a new
dataset and evaluation metrics specifically designed for the task of detecting
rotated table regions and localizing their head and tail parts. Secondly, it
presents the experiments conducted to select a baseline model for rotated
table region detection. Lastly, it showcases the ablation experiments
conducted to investigate the localization of table head and tail parts.

###  4.1 Dataset

In response to the scarcity and high annotation cost associated with rotated
image table detection datasets, particularly for head/tail detection, this
study introduces a method for constructing a comprehensive rotated image table
detection dataset.

####  4.1.1 ICDAR2019MTD

The ICDAR2019MTD[1] Modern Table Detection dataset was introduced at the Table
Detection and Recognition Competition of the 2019 International Conference on
Document Analysis and Recognition. Comprising 600 training images and 240
testing images, it contains 977 annotated table instances in the training set
and 449 annotated table instances in the testing set, provided in XML format.
The annotation format for this dataset adheres to the convention displayed in
Equation 11, with the top-left corner of the image serving as the starting
point and the remaining annotation points arranged counterclockwise. Figure 10
illustrates the ICDAR2019MTD visualization without any head and tail
information.

$x_{\\_}{1}\ y_{\\_}{1}\ x_{\\_}{2}\ y_{\\_}{2}\ x_{\\_}{3}\ y_{\\_}{3}\
x_{\\_}{4}\ y_{\\_}{4}$ (11)

Figure 10: ICDAR2019MTD Visualization

A notable limitation of the ICDAR2019MTD dataset is its exclusive focus on
horizontally-aligned tables, rendering it unsuitable for training table
rotation object detectors. Moreover, the four-point annotation format lacks
semantic information and does not specify the starting point as the top-left
corner of the table object. To capitalize on the capabilities of the
MMRotate[36] rotation object detection algorithm, this study converts the
original ICDAR2019MTD dataset annotations in XML format to DOTA-format[2] text
annotations, while introducing constraints and methods for generating TRR360D
annotations.

####  4.1.2 TRR360D

Figure 11: TRR360D Visualization

For the new task of detecting rotated table regions and localizing their head
and tail parts, a manual adjustment was performed on a small portion of
annotations in the ICDAR2019MTD dataset. Subsequently, the Adaptively Bounded
Rotation algorithm 2 was utilized to generate a rotated table detection
dataset, TRR360D, which contains semantic information about the table head and
tail parts.

The following provides an introduction to the format of the TRR360D dataset
and the minimal manual adjustments required for annotation. Following the DOTA
dataset annotation format, a line in the text file corresponding to the
annotation of a rotated table instance is illustrated in Equation 12. Point A
represents the top-left corner of the table, and points ABCD are arranged
clockwise. Parameter D denotes the detection difficulty of the sample, which
is uniformly defined as 0 in TRR360D.

$x_{\\_}{A}\ y_{\\_}{A}\ x_{\\_}{B}\ y_{\\_}{B}\ x_{\\_}{C}\ y_{\\_}{C}\
x_{\\_}{D}\ y_{\\_}{D}\ table\ D$ (12)

Figure 12: Adjustment of starting point labeling

The left image in Figure 12 depicts the visualization of the original
annotation for image 10497 in the ICDAR2019MTD modern table detection dataset,
where the four-point coordinates of ABCD are displayed as Equation 13. Since
point A is not located in the top-left corner of the table, the labeling
points need to be adjusted such that point A is positioned in the top-left
corner of the table, and points BCD fulfill the clockwise constraint.

$63\ 119\ 666\ 119\ 666\ 1006\ 63\ 1006\ table\ 0$ (13)

By manually adjusting the coordinates according to the order of Equation 14,
the resulting visualization is depicted in the right image of Figure 12. Now,
point A is positioned in the top-left corner of the table, and points BCD meet
the clockwise constraint.

$63\ 1006\ 63\ 119\ 666\ 119\ 666\ 1006\ table\ 0$ (14)

In the original dataset, some images do not meet the constraints.
Consequently, the starting point positions of images 10001, 10037, 10149,
10206, and 10211 in the testing set were manually adjusted. Furthermore, the
point order of images 10062, 10108, 10187, 10418, 10445, 10497, and 10537 in
the training set were manually modified to comply with the constraints of
point A being located at the top-left corner of the table and points ABCD
arranged in a clockwise manner.

Table 1: TRR360D dataset folders and annotations Folder | Description | Format | Images | Instances
---|---|---|---|---
ann_test_hbbox | Horizontal test set annotations | txt | 240 | 449
ann_test_obbox | Rotated test set annotations | txt | 240 | 449
ann_train_hbb | Horizontal training set annotations | txt | 600 | 977
ann_train_obbox | Rotated training set annotations | txt | 600 | 977

####  4.1.3 Evaluation

To address the limitations of the traditional evaluation metric AP, which only
measures the accuracy of rotation table region prediction without considering
the precision of header and footer localization, this paper introduces the
R360 AP evaluation metric. R360 AP is derived from AP by incorporating angle
constraints and is denoted as AP50(T<90), among others.

First, the concepts of $Rotate\ IoU$, $Precision$, and $Recall$ on which the
definition of AP50(T<90) relies will be introduced. Finally, a detailed
description of AP50(T<90) will be provided.

$Rotate\ IoU:$ Let the bounding boxes detected by a deep learning model be
denoted as predicted boxes $P$, and the annotated boxes in the dataset as
ground truth boxes $G$. The formula for Rotated IoU is given by Equation 15,
as illustrated in Figure 13.

Figure 13: Rotate IoU

$IoU_{\\_}{PG}=\frac{P\cap G}{P\cup G}$ (15)

The definitions of the subsequent metrics, such as $TP$, $Precision$,
$Recall$, $F1Score$, and $AP$, are all associated with the $IoU$ between the
predicted bounding boxes $P$ and the ground truth boxes $G$.

$TP$: $True\ Positive$ refers to the predicted boxes $P$ that satisfy the
conditions $IoU_{\\_}{PG}>T_{\\_}{IoU}$ and
$\left|P_{\\_}{\theta}-G_{\\_}{\theta}\right|<T_{\\_}{\theta}$. In this case,
$T_{\\_}{IoU}$ represents the IoU threshold, which is only counted once for
each ground truth box. A larger threshold implies a higher challenge for
localization accuracy. For instance, in PASCAL VOC 2007, $T_{\\_}{IoU}=0.5$.
Additionally, $T_{\\_}{\theta}$ denotes the angle threshold, with smaller
angles corresponding to greater difficulty.

$FP$: $False\ Positive$ refers to the number of predicted boxes $P$ that
satisfy $IoU\leq T_{\\_}{IoU}$ or $|\theta_{\\_}P-\theta_{\\_}G|\geq
T_{\\_}{\theta}$, or the number of redundant predicted boxes detected for the
same ground truth box. This metric is also known as the count of false
detections.

$FN$: $False\ Negative$, the number of ground truth boxes G that were not
detected by the model, also known as missed samples.

$Precision$: The ratio of the number of correctly predicted boxes to the total
number of predicted boxes.

$Precision=\frac{TP}{TP+FP}$ (16)

$Recall$: The ratio of correct predictions to the total number of ground truth
boxes is the evaluation metric for the dataset.

$Recall=\frac{TP}{TP+FN}$ (17)

$PR\ Curve:$The PR curve is a common performance evaluation metric used to
measure the performance of object detection models at different recall and
precision levels. The PR curve is plotted by calculating the recall and
precision of the object detection model at different confidence threshold
levels. The AP value is the area under the PR curve, which is usually computed
using the 11-point method for faster computation in practical implementation.

$AP50(T<90)$ refers to the area under the precision-recall (PR) curve at a
specific configuration, where the true positive (TP) condition is defined as
$IoU_{\\_}{PG}>0.5$ and $\left|P_{\\_}{\theta}-G_{\\_}{\theta}\right|<90$,
where $T_{\\_}{IoU}=0.5$ and $T_{\\_}{\theta}=90$.

$AP75(T<40)$ refers to the area under the precision-recall (PR) curve at a
specific configuration, where the true positive (TP) condition is defined as
$IoU_{\\_}{PG}>0.75$ and $\left|P_{\\_}{\theta}-G_{\\_}{\theta}\right|<40$,
where $T_{\\_}{IoU}=0.75$ and $T_{\\_}{\theta}=40$.

###  4.2 Experiment on Rotate Table Area Detection

Figure 14: Baseline Model Predict Visualization

In this section, experiments were conducted on three models: R-RetinaNet,
R-Cascade-RCNN, and RTMDet-S. The objective was to select an algorithm that
strikes a balance between accuracy and speed, which will then serve as the
baseline for implementing rotation table detection and head/tail recognition.

The results are displayed in Table 2. R-RetinaNet achieved an AP50(T<360)
score of 0.547, as depicted in the prediction outcomes in Figure 14. However,
the predictions were not precise and the speed was limited to only 13.9 FPS.

R-CascadeRCNN achieved an AP50(T<360) score of 0.788. The PR curve illustrated
in Figure 15 displayed superior performance compared to R-RetinaNet, and the
visualization of its prediction outcomes in Figure 14 (c) demonstrated that
the region predictions were predominantly accurate. Nonetheless, its speed was
a mere 7.6 FPS, which was slower than R-RetinaNet. Although the accuracy was
enhanced relative to R-RetinaNet, there was a trade-off with reduced speed.

Figure 15: Baseline P-R Curve on the TRR360D Dataset

As depicted in Figure 14 (d), Table 2, and the PR curve in Figure 15, RTMDet-S
demonstrated a level of performance that is comparable to R-CascadeRCNN in
terms of AP50(T<360) precision. Additionally, it boasted a speed of 33.7 FPS,
which is significantly faster than R-RetinaNet and R-CascadeRCNN. Based on
these results, the baseline model chosen for table rotation detection and
head-tail recognition is RTMDet-S.

Table 2: Baseline model experimental results on the TRR360D Dataset Model | AP50(T<360) | FPS
---|---|---
R-RetinaNet | 0.547 | 13.9
R-CascadeRCNN | 0.788 | 7.6
RTMDet-S | 0.788 | 33.7

###  4.3 Ablation Study on Head and Tail Location

Based on the RTMDet baseline model, this section conducted extensive
experiments on the TRR360D dataset, and the experimental results are shown in
Table 3. In Table 3, L, M, S, and T in the Rotated RTMDet comparison
experiment respectively represent the large, medium, small, and tiny models of
RTMDet. Their FPS-AP chart is shown in Figure 16, with the T series model
having the fastest average FPS of 42.0, the S series model having a relatively
fast average FPS of 34.4, the M series model having a moderate average FPS of
18.9, and the L series having the slowest average FPS of 11.5. In Table 3 AL
represents the Angle Loss branch, PT represents transfer learning, and RR
represents adaptive boundary random rotation data augmentation.

The experimental reasoning test environment is based on the Ubuntu 20.04
system, and the hardware environment used is: 12th generation Intel i7-12700
processor, 32GB RAM, NVIDIA GeForce 3060 12G GPU. Different hardware platforms
may have been used during the training phase, such as NVIDIA GeForce 3060 12G
GPU, NVIDIA Tesla V100 GPU, etc. The training used the AdamW optimizer for
32400 iterations, with a batch size set to 2.

####  4.3.1 AL Branch

Figure 16: FPS-AP50(T<90) on the TRR360D Dataset

As shown in Table 3 of the rotation RTMDet comparison experiment, the
AP50(T<360) accuracy for RTMDet-L, RTMDet-M, RTMDet-S, and RTMDet-T is 0.787,
0.786, 0.788, and 0.778, respectively. When the absolute difference between
the predicted bounding box angle and the ground truth angle is limited to less
than 90°, the AP(T<90) accuracy is 0.224, 0.217, 0.237, and 0.258,
representing a decrease of 56.3%, 56.9%, 55.1%, and 52.0%, respectively. This
indicates that the original RTMDet in MMRotate can only predict the regions of
rotated bounding boxes, but it cannot learn and predict the semantic features
of the four corner points of the rotated bounding boxes. As a result, it fails
to distinguish between the 30° and -150° rotated bounding boxes shown in
Figure 7, and can only represent a range of 180°. This highlights the
limitation of the original RTMDet-S and other related algorithms in not
supporting the localization of the head and tail of rotated table regions.

Upon integrating the AL branch into the RTMDet model, the AP50(T<90) metrics
of RTMDet-L-AL, RTMDet-M-AL, RTMDet-S-AL, and RTMDet-T-AL showed an
improvement of 24.6, 43.2, 37.4, and 31.9 percentage points, respectively,
compared to their respective counterparts RTMDet-L, RTMDet-M, RTMDet-S, and
RTMDet-T. This result highlights the effectiveness of the AL branch in
facilitating 360° rotation feature learning, enabling the detection of both
the head and tail of tables.

The visualization of the detection results also serves as evidence for the
effectiveness of the AL branch. Figure 17 shows the visualization of the
actual rotated bounding boxes for sample 10016 in the TRR360D rotated table
detection test set. Specifically, Figure 17 (a) depicts the actual rotated
bounding boxes, while Figure 17 (b) illustrates that although the RTMDet-L
model correctly predicts the table region, the RIoU loss function fails to
constrain the starting point of the table, leading to an inaccurate prediction
of point A. By integrating the AL branch into the RTMDet-L model, the starting
point prediction is effectively constrained, as shown in Figure 17 (c). While
the prediction of point A is roughly correct, it has a certain impact on the
accuracy of the predicted table region. Therefore, two methods are employed in
the following to improve the accuracy.

Table 3: Comparison of RTMdet models with different configurations on the TRR360D Dataset Model | AP50 (T<360) | AP50 (T<90) | FPS
---|---|---|---
RTMDet-L | 0.787 | 0.224 | 11.5
RTMDet-L-AL | 0.566 | 0.470 | 11.4
RTMDet-L-AL-PT | 0.793 | 0.776 | 11.6
RTMDet-L-AL-PT-RR | 0.897 | 0.896 | 11.4
RTMDet-M | 0.786 | 0.217 | 18.9
RTMDet-M-AL | 0.690 | 0.649 | 18.9
RTMDet-M-AL-PT | 0.804 | 0.798 | 19.3
RTMDet-M-AL-PT-RR | 0.896 | 0.896 | 18.8
RTMDet-S | 0.788 | 0.237 | 33.7
RTMDet-S-AL | 0.693 | 0.611 | 33.1
RTMDet-S-AL-PT | 0.792 | 0.782 | 34.9
RTMDet-S-AL-PT-RR | 0.890 | 0.887 | 36.0
RTMDet-T | 0.778 | 0.258 | 42.1
RTMDet-T-AL | 0.597 | 0.577 | 42.0
RTMDet-T-AL-PT | 0.764 | 0.677 | 44.7
RTMDet-T-AL-PT-RR | 0.805 | 0.805 | 39.3

####  4.3.2 Transfer Learning

In this study, Transfer Learning is also attempted to enhance the accuracy of
RTHDet. Table 3 presents the results of the rotating RTMDet comparison
experiment. It shows that the RTMDet-L-AL-PT, RTMDet-M-AL-PT, RTMDet-S-AL-PT,
and RTMDet-T-AL-PT models, which utilize transfer learning from the non-table
detection domain DOTA dataset, outperform their counterparts without transfer
learning, namely RTMDet-L-AL, RTMDet-M-AL, RTMDet-S-AL, and RTMDet-T-AL, in
terms of AP50(T<90) metrics. Specifically, the AP50(T<90) index for the former
models increased by 30.6, 14.9, 17.1, and 10.0 percentage points,
respectively. These results demonstrate the effectiveness of transfer learning
in the rotating table detection domain.

Figure 17: RTMDet Predict Visualization

####  4.3.3 Adaptive Boundary Random Rotation Data Augmentation

The Adaptive Boundary Rotation method 3.3 is applied to augment the training
data on the unrotated horizontal table training set of TRR360D. This approach
enriches the data and improves the model’s accuracy. Table 3 of the rotating
RTMDet comparison experiment shows that the RTMDet-L-AL-PT-RR, RTMDet-M-AL-PT-
RR, RTMDet-S-AL-PT-RR, and RTMDet-T-AL-PT-RR models, which use random rotation
data augmentation with adaptive boundaries, outperformed their counterparts
without this augmentation technique (i.e., RTMDet-L-AL-PT, RTMDet-S-AL-PT,
RTMDet-S-AL-PT, and RTMDet-T-AL-PT) in terms of the AP50(T<90) metric, with an
increase of 12.0, 9.8, 10.5, and 12.8 percentage points, respectively. The
inference visualization of the RTMDet-L-AL-PT-RR model shown in Figure 17 (d)
meets the accuracy requirements of region localization and correctly predicts
the starting point A at the top-left corner of the table. As a result, line
segment AB represents the head of the table, while line segment DC represents
the tail of the table. These results demonstrate that adaptive boundary random
rotation data augmentation effectively improves the performance of the RTHDet
model.

##  5 Conclusion

Traditional models and datasets primarily concentrated on horizontal table
detection, unable to accurately detect table regions and localize their head
and tail parts in rotating scenarios, thus greatly restricting the development
of table recognition in rotated contexts. Therefore, this paper proposed a
novel task and provided substantial innovations in detecting table regions and
localizing their head and tail parts in rotational scenarios, along with the
proposal of datasets, evaluation metrics, and methods. The Adaptively Bounded
Rotation method is proposed to generate the TRR360D dataset on the classic
ICDAR2019MTD dataset. The newly introduced R360 AP evaluation metric
significantly improved the assessment of the precision of rotated region
detection. Leveraging the $D_{\\_}{r360}$ angle definition method, Transfer
Learning, and Adaptive Boundary Rotation data augmentation on the RTMDet-S
baseline model, the proposed RTHDet remarkably improved AP50 (T<90) from 23.7%
to 88.7%, thereby attesting to its efficacy in addressing this innovative
task. This research is poised to propel the development of downstream table
recognition OCR tasks. Despite the significant progress, the current inability
to detect arbitrary quadrilateral tables in perspective scenarios remains a
notable limitation, necessitating further research to address this challenge.

## References

  * [1] Liangcai Gao, Yilun Huang, Herve Dejean, Jean Luc Meunier, Qinqin Yan, Yu Fang, Florian Kleber, and Eva Lang.  ICDAR 2019 competition on table detection and recognition (cTDaR).  In Proceedings of the International Conference on Document Analysis and Recognition, ICDAR, pages 1510–1515, 2019.
  * [2] Gui Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang.  DOTA: A Large-Scale Dataset for Object Detection in Aerial Images.  Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3974–3983, 2018.
  * [3] Chengqi Lyu, Wenwei Zhang, Haian Huang, Yue Zhou, Yudong Wang, Yanyi Liu, Shilong Zhang, and Kai Chen.  RTMDet: An Empirical Study of Designing Real-Time Object Detectors.  pages 63–65, 2022.
  * [4] Katsuhiko Itonori.  Table structure recognition based on textblock arrangement and ruled line position.  In Proceedings of 2nd International Conference on Document Analysis and Recognition (ICDAR’93), pages 765–768. IEEE, 1993.
  * [5] Surekha Chandran and Rangachar Kasturi.  Structural recognition of tabulated data.  In Proceedings of 2nd International Conference on Document Analysis and Recognition (ICDAR’93), pages 516–519. IEEE, 1993.
  * [6] Pallavi Pyreddy and W Bruce Croft.  Tintin: A system for retrieval in text tables.  In Proceedings of the second ACM international conference on Digital libraries, pages 193–200, 1997.
  * [7] E Green and M Krishnamoorthy.  Recognition of tables using table grammars.  In Proceedings of the Fourth Annual Symposium on Document Analysis and Information Retrieval, pages 261–278, 1995.
  * [8] Jianying Hu, Bell Laboratories, Lucent Technologies, Mountain Avenue, and Murray Hill.  Medium-Independent Table Detection.  In Evaluation, volume 3967, pages 1–12. International Society for Optics and Photonics, 2000.
  * [9] Aleksander Pivk, Philipp Cimiano, York Sure, Matjaz Gams, Vladislav Rajkovič, and Rudi Studer.  Transforming arbitrary tables into logical form with TARTAR.  Data \& Knowledge Engineering, 60(3):567–595, 2007.
  * [10] Alexey Shigarov, Andrey Mikhailov, and Andrey Altaev.  Configurable table structure recognition in untagged pdf documents.  In Proceedings of the 2016 ACM Symposium on Document Engineering, pages 119–122, 2016.
  * [11] David W Embley, Matthew Hurst, Daniel Lopresti, and George Nagy.  Table-processing paradigms: a research survey.  International Journal of Document Analysis and Recognition (IJDAR), 8(2):66–86, 2006.
  * [12] Azka Gilani, Shah Rukh Qasim, Imran Malik, and Faisal Shafait.  Table Detection Using Deep Learning.  In Proceedings of the International Conference on Document Analysis and Recognition, ICDAR, volume 1, pages 771–776, 2017.
  * [13] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1137–1149, 2017.
  * [14] Sebastian Schreiber, Stefan Agne, Ivo Wolf, Andreas Dengel, and Sheraz Ahmed.  Deepdesrt: Deep learning for detection and structure recognition of tables in document images.  In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 1, pages 1162–1167. IEEE, 2017.
  * [15] Karen Simonyan and Andrew Zisserman.  VERY DEEP CONVOLUTIONAL NETWORKS FOR LARGE-SCALE IMAGE RECOGNITION.  2015\.
  * [16] Shoaib Ahmed Siddiqui, Muhammad Imran Malik, Stefan Agne, Andreas Dengel, and Sheraz Ahmed.  DeCNT: Deep deformable CNN for table detection.  IEEE Access, 6:74151–74161, 2018.
  * [17] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable Convolutional Networks.  Proceedings of the IEEE International Conference on Computer Vision, 2017-Octob:764–773, 2017.
  * [18] Ningning Sun, Yuanping Zhu, and Xiaoming Hu.  Faster R-CNN based table detection combining corner locating.  In Proceedings of the International Conference on Document Analysis and Recognition, ICDAR, pages 1314–1319, 2019.
  * [19] Ranajit Saha, Ajoy Mondal, and C V Jawahar.  Graphical object detection in document images.  In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 51–58. IEEE, 2019.
  * [20] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask R-CNN.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(2):386–397, 6 2020.
  * [21] Xu Zhong, Elaheh ShafieiBavani, and Antonio Jimeno Yepes.  Image-based table recognition: data, model, and evaluation.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16, pages 564–580. Springer, 2020.
  * [22] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  arXiv preprint arXiv:1804.02767, 2018.
  * [23] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng Yang Fu, and Alexander C. Berg.  SSD: Single shot multibox detector.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 9905 LNCS:21–37, 2016.
  * [24] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [25] Zhaowei Cai and Nuno Vasconcelos.  Cascade r-cnn: Delving into high quality object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6154–6162, 2018.
  * [26] Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, and others.  Hybrid task cascade for instance segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4974–4983, 2019.
  * [27] Siyuan Qiao, Liang-Chieh Chen, and Alan Yuille.  DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution.  2020\.
  * [28] Devashish Prasad, Ayan Gadpal, Kshitij Kapadni, Manish Visave, and Kavita Sultanpure.  CascadeTabNet: An approach for end to end table detection and structure recognition from image-based documents.  In IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, volume 2020-June, pages 2439–2447, 2020.
  * [29] Madhav Agarwal, Ajoy Mondal, and C. V. Jawahar.  CDEC-Net: Composite deformable cascade network for table detection in document images.  Proceedings - International Conference on Pattern Recognition, pages 9491–9498, 2020.
  * [30] Xinyi Zheng, Douglas Burdick, Lucian Popa, Xu Zhong, and Nancy Xin Ru Wang.  Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context.  2021\.
  * [31] Danish Nazir, Khurram Azeem Hashmi, Alain Pagani, Marcus Liwicki, Didier Stricker, and Muhammad Zeshan Afzal.  Hybridtabnet: Towards better table detection in scanned document images.  Applied Sciences (Switzerland), 11(18), 2021.
  * [32] Khurram Azeem Hashmi, Alain Pagani, Marcus Liwicki, Didier Stricker, and Muhammad Zeshan Afzal.  CasTabDetectoRS : Cascade Network for Table Detection in Document Images with Recursive Feature Pyramid and Switchable Atrous Convolution.  (September):1–21, 2021.
  * [33] Chixiang Ma, Weihong Lin, Lei Sun, and Qiang Huo.  Robust table detection and structure recognition from heterogeneous document images.  Pattern Recognition, 133:109006, 2023.
  * [34] Zhaowei Cai and Nuno Vasconcelos.  Cascade R-CNN: High quality object detection and instance segmentation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(5):1483–1498, 2021.
  * [35] Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun.  YOLOX: Exceeding YOLO Series in 2021.  pages 1–7, 2021.
  * [36] Yue Zhou, Xue Yang, Gefan Zhang, Jiabao Wang, Yanyi Liu, Liping Hou, Xue Jiang, Xingzhao Liu, Junchi Yan, Chengqi Lyu, Wenwei Zhang, and Kai Chen.  MMRotate, volume 1.  2022\.