[1]Benjamin Hou

1]National Institutes of Health (NIH) Clinical Center, Bethesda MD, USA

2]Walter Reed National Military Medical Center, Bethesda MD, USA

# Enhanced Muscle and Fat Segmentation for CT-Based Body Composition Analysis:
A Comparative Study

Tejas Sudharshan Mathai  Jianfei Liu  Christopher Parnell  Ronald M. Summers
[  [

###### Abstract

Purpose: Body composition measurements from routine abdominal CT can yield
personalized risk assessments for asymptomatic and diseased patients. In
particular, attenuation and volume measures of muscle and fat are associated
with important clinical outcomes, such as cardiovascular events, fractures,
and death. This study evaluates the reliability of an Internal tool for the
segmentation of muscle and fat (subcutaneous and visceral) as compared to the
well-established public TotalSegmentator tool.

Methods: We assessed the tools across 900 CT series from the publicly
available SAROS dataset, focusing on muscle, subcutaneous fat, and visceral
fat. The Dice score was employed to assess accuracy in subcutaneous fat and
muscle segmentation. Due to the lack of ground truth segmentations for
visceral fat, Cohen’s Kappa was utilized to assess segmentation agreement
between the tools.

Results: Our Internal tool achieved a 3% higher Dice (83.8 vs. 80.8) for
subcutaneous fat and a 5% improvement (87.6 vs. 83.2) for muscle segmentation
respectively. A Wilcoxon signed-rank test revealed that our results were
statistically different with p $<$ 0.01. For visceral fat, the Cohen’s kappa
score of 0.856 indicated near-perfect agreement between the two tools. Our
internal tool also showed very strong correlations for muscle volume
(R2=0.99), muscle attenuation (R2=0.93), and subcutaneous fat volume (R2=0.99)
with a moderate correlation for subcutaneous fat attenuation (R2=0.45).

Conclusion: Our findings indicated that our Internal tool outperformed
TotalSegmentator in measuring subcutaneous fat and muscle. The high Cohen’s
Kappa score for visceral fat suggests a reliable level of agreement between
the two tools. These results demonstrate the potential of our tool in
advancing the accuracy of body composition analysis.

###### keywords:

Body Composition, CT, Segmentation, Muscle, Fat, Subcutaneous, Visceral

##  1 Introduction

The assessment of body composition, particularly the accurate segmentation of
soft tissues such as subcutaneous fat, visceral fat, and muscle, has become a
critical component in diagnostic imaging [1, 2]. Advances in computed
tomography (CT) imaging have not only facilitated detailed body composition
analysis, but also play a pivotal role in a range of medical applications,
from disease characterization to surgical planning and radiation therapies [3,
4]. This advancement in imaging technology demonstrates potential for enhanced
‘incidental’ screening and tailored risk evaluation, benefiting both
asymptomatic individuals and patients with existing medical conditions. For
instance, the distribution and volume of visceral fat are closely linked to
metabolic disorders and cardiovascular diseases, making their assessment
crucial for early intervention strategies [5]. Similarly, understanding the
balance between muscle and fat tissues is essential in evaluating nutritional
status, which is particularly relevant in conditions like obesity, sarcopenia,
and cachexia [6, 7, 1]. In sports medicine and rehabilitation, analyzing
muscle and fat distribution is crucial for creating personalized training and
recovery programs [8].

Figure 1: Example axial, coronal and sagittal slice of ``case-042— from SAROS
dataset. In the axial slice, the muscle (yellow), subcutaneous fat (red) and
abdominal cavity (green) are shown. The grey regions in the coronal/sagittal
views indicate no segmentation masks available in that area, while the streaks
in between them contain segmentations.

Similarly, in clinical research, such data significantly enhance our
understanding of various health conditions and aid in the development of
innovative treatments. This knowledge is particularly invaluable in oncology,
where it plays a key role in tailoring treatment plans and monitoring the
impact of therapies that can significantly alter body composition. Moreover,
in surgical planning, especially in reconstructive or plastic surgery, the
precise imaging of these tissues is essential for ensuring better outcomes and
guiding post-operative care [9, 10, 11]. Recent developments in automated
segmentation tools, such as TotalSegmentator [12], have shown promising
results in enhancing the efficiency and accuracy of these analyses. However,
generalized tools in medical imaging, while versatile and broadly applicable,
often do not perform with the same level of precision and efficacy as tools
that are specifically targeted or tailored to particular tasks or conditions.
The effectiveness of such tools compared to specialized solutions remains a
subject of ongoing research.

In this study, we compare the effectiveness of the public TotalSegmentator
tool against an internally developed tool for the task of muscle and fat
(subcutaneous and visceral) segmentation in CT. We hypothesized that the
internal tool developed specifically for muscle and fat segmentation would
fare better than TotalSegmentator. Through experiments on the public SAROS
dataset, we show that the internal tool fares better at the segmentation
tasks, with statistical results to corroborate our findings. Our tool has
substantial potential to be used for a broad range of clinical applications
and offers opportunities for personalized risk assessment for patients.

##  2 Materials and Methods

###  2.1 Patient Population

This study utilized deidentified data that is publicly available, thereby
obviating the need for IRB approval. The dataset employed, known as the
Sparsely Annotated Region and Organ Segmentation (SAROS) [13, 14], comprised
of 900 CT series from 882 patients, evenly divided between 450 women and 450
men. These series were randomly selected from various TCIA [14] collections.

The dataset contains CT volumes of 5mm slice thickness, with annotations
provided in NIfTI format. These annotations covers 13 semantic body regions
across 6 distinct body parts. The initial generation of annotations was
carried out using body composition analysis tools developed by Koitka et al.
[15], and subsequently reviewed and corrected by medical residents and
students on every fifth axial slice, as illustrated in Fig. 1. Slices that
were not reviewed were marked with an ‘ignore’ label of value 255. This
retrospective study concentrated on three types of soft tissues from this
dataset: subcutaneous fat, visceral fat, and muscle. However, the dataset only
includes ground truth segmentation labels for subcutaneous fat and muscle.

###  2.2 TotalSegmentator

TotalSegmentator [12] is a publicly accessible tool designed for segmenting
over 117 distinct classes in CT images. It is apt for various applications,
including organ volumetry, disease characterization, and planning for surgical
or radiation therapy. This tool was developed using a training set of 1,204 CT
examinations, encompassing a diverse array of scanners, institutions, and
protocols to ensure its versatility and robustness in different clinical
settings.

###  2.3 Internal Tool

Our internal tool leverages the 3D nnU-Net model [16], which is widely
recognized and acclaimed as the de facto standard in supervised segmentation.
The training data was acquired using a 2D dual-branch network, as described in
Liu et al. [17]. This 2D dual-branch network was initially developed to
alleviate the extensive and time-consuming annotation burden associated with
full CT volumes, enabling the generation of precise segmentations of muscle
and fat across all slices of a CT scan.

The dual-branch network features a shared encoder and two duplicate decoders.
It was trained using a combination of a few strongly labeled and a large
number of weakly labeled datasets; the strongly labeled data included manual
annotations of muscle, visceral fat, and subcutaneous fat on each CT slice.
The weak labels, generated automatically via a level-set method [18], were
prone to segmentation errors. The dual-branch network was trained through a
mixed supervision approach utilizing both strong and weak labels. Throughout
the training process, the weakly labeled data was periodically refined by the
strong decoder in a self-supervised manner. Upon completion of the dual-branch
network’s training, it was applied to all CT volumes to generate dense
annotations across all CT series. These annotations were then utilized as
training data for the 3D full-resolution nnU-Net.

###  2.4 Statistical Analysis

As previously mentioned, this retrospective study focuses on three types of
soft tissues: subcutaneous fat, visceral fat, and muscle. While both
TotalSegmentator (TS) and our Internal tool are capable of segmenting all
three tissue types, the SAROS dataset only includes ground truth labels for
subcutaneous fat and muscle. After the Internal tool and TotalSegmentator were
executed on the CT series in the dataset, the Dice coefficient was utilized to
compute the similarity between the predicted segmentations and the ground-
truth annotations. Since not all slices in the dataset were labeled, Dice
score calculation was confined to the “valid” regions of interest, which were
delineated by the body mask provided. For all analyses, slices lacking labels,
as well as background pixels, were excluded. This approach ensures that our
evaluation focused solely on the relevant anatomical areas.

After assessing the normality of the Dice score distribution, a Wilcoxon
signed-rank test was employed to determine any statistical differences. Due to
the absence of ground truth labels for visceral fat in the dataset, Cohen’s
Kappa [19] was used to evaluate the agreement between TotalSegmentator and our
internal tool in segmenting visceral fat. Cohen’s Kappa is a statistical
measure that captures the agreement between two raters, taking into account
the possibility of agreement occurring by chance. In addition, graphs
correlating the ground-truth segmentations contrasted against the predictions
were also plotted with overlaid $R^{2}$ values. Bland-Altman analysis was also
conducted through the calculation of volume differences (biases) and averages
for each structure to determine agreement. The Dice and kappa scores were
calculated using the Scikit-learn library (Version 1.3.1) in Python (Version
3.9.10). All statistical tests were performed using RStudio (Version
2023.06.1+524).

##  3 Results

Table 1 presents a direct comparison of the segmentation capabilities of
TotalSegmentator and our Internal tool, specifically focusing on subcutaneous
fat and muscle segmentation. Fig. 2 shows violin plots to visualize the
distributions of Dice Scores for both TotalSegmentator and Internal tool. Dice
scores in Table 1 are presented as means $\pm$ standard deviation, along with
the 25th and 75th percentiles (IQR), for both subcutaneous fat and muscle. For
subcutaneous fat, TotalSegmentator achieved an average score of 80.8 (±10.4)
with an IQR range of [76.7, 87.7]. In contrast, our Internal tool demonstrated
a slightly higher mean Dice Score of 83.8 (±10.9) with an IQR range of [80.7,
90.5]. With respect to muscle, TotalSegmentator attained a mean score of 83.2
(±4.6) and [80.5, 86.4] IQR. In contrast, our Internal tool outperformed it by
5% as a mean score of 87.6 (±3.3) and [85.6, 90] IQR was obtained. Notably, as
depicted in Fig. 2, the internal tool exhibits fewer outliers compared to
TotalSegmentator, particularly in muscle segmentation, indicating a more
consistent and reliable performance. These results suggest that while both
tools are effective for soft tissue segmentation, the Internal tool was
superior in segmenting both subcutaneous fat and muscle with $p<0.01$.

Table 1: Table of Dice scores: TotalSegmentator vs. Internal Tool for subcutaneous fat and muscle Segmentation. Scores are shown with mean, standard deviation, and Inter-Quartile Ranges (IQR). | Subcutaneous Fat | Muscle
---|---|---
TotalSegmentator | 80.8 $\pm$ 10.4 [76.7, 87.7] | 83.2 $\pm$ 4.6 [80.5, 86.4]
Internal | 83.8 $\pm$ 10.9 [80.7, 90.5] | 87.6 $\pm$ 3.3 [85.6, 90.0]

((a))

((b))

Figure 2: Violin plot of TotalSegmentator (yellow) vs. our internal tool (teal) for the segmentation of (a) subcutaneous fat and (b) muscle. Table 2: Table of Cohen’s Kappa scores: Agreement of TotalSegmentator and Internal tool for segmentation of subcutaneous fat, visceral fat, and muscle. Scores are shown with mean, standard deviation, and Inter-Quartile Ranges (IQR). | Cohen’s Kappa
---|---
Subcutaneous Fat | 0.874 $\pm$ 0.066 [0.854, 0.913]
Visceral Fat | 0.856 $\pm$ 0.074 [0.830, 0.906]
Muscle | 0.837 $\pm$ 0.033 [0.819, 0.861]

The kappa scores in Table 2 indicated a high level of concordance between the
two tools across all three tissue types. Fig. 3 shows $R^{2}$ correlation
plots for the volume and attenuation of the different structures respectively.
The average Hounsfield Unit (HU) of muscle attenuation for both
TotalSegmentator and our Internal tool exhibit a strong correlation with
$R^{2}$ values of 0.87 and 0.93, respectively, with our Internal tool
outperforming it by a notable margin. This is supported by the similarly
strong correlation observed with muscle volume, yielding $R^{2}$ values of
0.97 and 0.99, respectively. For subcutaneous fat, despite a significant
uncertainty in the average HU values for both tools, with $R^{2}$ values of
0.43 for TotalSegmentator and 0.45 for our Internal tool. Nevertheless, the
region was accurately segmented, with fat volume estimation showing a high
correlation, evidenced by an $R^{2}$ value of 0.99 for both tools.

((a)) TS Muscle Vol

((b)) TS Muscle Att

((c)) TS Fat Vol

((d)) TS Fat Att

((e)) Int Muscle Vol

((f)) Int Muscle Att

((g)) Int Fat Vol

((h)) Int Fat Att

Figure 3: $R^{2}$ correlation plots of the automatic segmentation results
compared against ground truth annotations. Top Row: TotalSegmentator (TS).
Bottom Row: Internal (Int) tool. L-to-R: Muscle Volume, Muscle Attenuation,
Fat Volume, Fat Attenuation.

((a)) TS Muscle

((b)) Internal Muscle

((c)) TS Fat

((d)) Internal Fat

Figure 4: Bland-Altman plots of the volume measurements between the automatic
segmentations compared against manual annotations. L-to-R: TotalSegmentator
Muscle Volume, Internal Muscle Volume, TotalSegmentator Subcutaneous Fat,
Internal Subcutaneous Fat.

Fig. 4 displays Bland-Altman plots for muscle and subcutaneous fat volume
estimation of the tools compared to the manual annotations. For both tools
measuring muscle volume, there’s a noticeable positive skew in the data. The
Internal tool demonstrated a significantly lower bias, approximately 250cm3,
in comparison to the TotalSegmentator, which exhibited a bias around 500cm3.
For the subcutaneous fat volume estimation, there is a distinct concentration
of data points on the left-hand side. The Internal tool has a slight positive
skew also with a higher bias at around +200cm3 compared to TotalSegmentator
that is around 0cm3.

((a))

((b))

((c))

((d))

((e))

((f))

((g))

((h))

((i))

((j))

((k))

((l))

Figure 5: Example segmentation of ``case-042—. Top-to-Bottom: axial, coronal,
sagittal views. L-to-R: CT image, manual annotation (ground truth),
TotalSegmentator segmentation, Internal tool segmentation. Red: Subcutaneous
Fat, Yellow: Muscle, Green: Internal Abdominal Cavity (ground truth only) /
Subcutaneous Fat, Grey: No ground truth labels. Blue arrows in the coronal
slice of TotalSegmentator segmentation shows over segmentation of subcutaneous
fat where it was correctly segmented as muscle by our Internal tool.

Fig. 5 shows and example segmentation of body composition by TotalSegmentator
and our Internal tool. In a comparison of segmentation accuracy, our internal
tool outperformed TotalSegmentator, achieving Dice scores of 0.947 for
Subcutaneous Fat and 0.884 for Muscle, compared to TotalSegmentator’s scores
of 0.919 and 0.809, respectively. Additionally, our internal tool exhibited a
robust Cohen’s Kappa score of 0.876, further demonstrating its strong
agreement compared to a popular and widely used tool. TotalSegmentator has
shown a tendency to over-segment subcutaneous fat, as indicated by the blue
arrows in Fig. 5. This is particularly evident in areas such as the muscle
between the ribs and within the pelvic cavities

##  4 Discussion and Conclusion

Through our experiments, the Internal tool achieved a 3% higher Dice (83.8 vs.
80.8) for subcutaneous fat and a 5% improvement (87.6 vs. 83.2) for muscle
segmentation respectively. The results yielded by the internal tool were
statistically different p $<$ 0.01. However, from the $R^{2}$ correlation
plots in Fig. 3 for subcutaneous fat, a significant uncertainty was seen in
the average HU values for both tools: 0.43 for TotalSegmentator and 0.45 for
our Internal tool.

The considerable standard deviation in HU values within the subcutaneous fat
layer can be attributed to its diverse composition. This layer, primarily
composed of adipocytes, also contains fibroblasts, blood vessels, nerve cells,
lymphatic vessels, immune cells, hair follicles, and sweat glands, each with
differing densities. These varying densities result in a broad spectrum of HU
values, as captured in CT scans. The contrast between the low-density
adipocytes and the higher-density components within the layer leads to the
observed variability in HU readings.

The variability can also be attributed to several other factors: the quality
and noise in CT images affecting segmentation precision, limitations in the
segmentation algorithm especially if not tailored for subcutaneous fat,
variability in fat composition and density, and the choice of thresholding in
segmentation. This complexity not only highlights the multifaceted nature of
the subcutaneous layer, but also underscores the challenge in accurately
segmenting and analyzing it using CT imaging. Despite the variations in HU,
the subcutaneous fat volume demonstrated a high correlation for both tools
with an $R^{2}$ value of 0.99, indicating accurate segmentation of the
subcutaneous fat region.

The skewness in the Bland-Altman plots in Fig. 4 suggests a tendency for the
differences between the two methods under comparison to increase as the
magnitude of the measurement decreases. Such a distribution pattern indicates
a potential systematic bias in the measurements, particularly at lower values.
For the concentration of data points on the left-hand side in the subcutaneous
fat volume estimation, the pattern indicates that the agreement between the
two methods being compared is more consistent at lower measurement values.
Such a concentration suggests that for smaller magnitudes of the variable
being measured, the two methods yield closer results, implying better
concordance in this range. However, this also raises questions about the
performance of the methods at higher values, as the relative sparsity of data
points on the right-hand side of the plot may indicate a divergence in the
methods’ readings or a limitation in the range of data sampled.

In summary, this study has demonstrated that our internal tool significantly
outperforms the more generalized TotalSegmentator in accurately segmenting
subcutaneous fat, visceral fat, and muscle in CT series. Our findings are
supported by high Dice scores and strong correlations ($R^{2}$) with manual
annotations, and is further corroborated by Bland-Altman plots demonstrating
consistent agreement and minimal bias. The enhanced accuracy and consistency
of our internal tool hold significant promise for a range of clinical
applications, such as providing improved personalized risk assessments for
patients at risk of adverse cardiovascular events and fractures.

##  5 Acknowledgments

Funding: This work was supported by the Intramural Research Program of the NIH
Clinical Center (project number 1Z01 CL040004).

Ethical approval: All procedures performed in studies involving human
participants were in accordance with the ethical standards of the
institutional and/or national research committee and the 1964 Helsinki
declaration and its later amendments or comparable ethical standards. For this
study, informed consent was not required.

Conflict of Interest: RMS receives royalties from iCAD, Philips, PingAn,
ScanMed, and Translation Holdings. His lab received research support from
PingAn. The authors have no additional conflicts of interest to declare.

## References

  * [1] Lee Matthew H., Daniel Liu, John W. Garrett, Alberto Perez, Ryan Zea, Ronald M. Summers, and Perry J. Pickhardt.  Comparing fully automated ai body composition measures derived from thin and thick slice ct image data.  Abdominal Radiology, pages 1–1, 2023.
  * [2] Alexander D. Weston, Panagiotis Korfiatis, Timothy L. Kline, Kenneth A. Philbrick, Petro Kostandy, Tomas Sakinis, Motokazu Sugimoto, Naoki Takahashi, and Bradley J. Erickson.  Automated abdominal segmentation of ct scans for body composition analysis using deep learning.  Radiology, 290(3):669–679, 2019.  PMID: 30526356.
  * [3] Sokratis Makrogiannis, Azubuike Okorie, Angelo Di Iorio, Stefania Bandinelli, and Luigi Ferrucci.  Multi-atlas segmentation and quantification of muscle, bone and subcutaneous adipose tissue in the lower leg using peripheral quantitative computed tomography.  Frontiers in Physiology, 13:951368, 2022.
  * [4] Urvi A Shah, Tarah J Ballinger, Rusha Bhandari, Christina M Dieli-Conwright, Kristin A Guertin, Elizabeth A Hibler, Faiza Kalam, Ana Elisa Lohmann, and Joseph E Ippolito.  Imaging modalities for measuring body composition in patients with cancer: opportunities and challenges.  JNCI Monographs, 2023(61):56–67, 05 2023.
  * [5] Olga Gruzdeva, Daria Borodkina, Evgenya Uchasova, Yulia Dyleva, and Olga Barbarash.  Localization of fat depots and cardiovascular risk.  Lipids in health and disease, 17(1):1–9, 2018.
  * [6] Clifton J Holmes and Susan B Racette.  The utility of body composition assessment in nutrition and clinical practice: an overview of current methodology.  Nutrients, 13(8):2493, 2021.
  * [7] Ezgi Ozen, Rada Mihaylova, Michelle Weech, Sam Kinsella, Julie A Lovegrove, and Kim G Jackson.  Association between dietary saturated fat with cardiovascular disease risk markers and body composition in healthy adults: findings from the cross-sectional bodycon study.  Nutrition & Metabolism, 19(1):1–15, 2022.
  * [8] Henning Wackerhage and Brad J Schoenfeld.  Personalized, evidence-informed training plans and exercise prescriptions for performance, fitness and health.  Sports Medicine, 51(9):1805–1813, 2021.
  * [9] Thomas Perrin, Marc Lenfant, Cyrile Boisson, Marine Bert, Patrick Rat, and Olivier Facy.  Effects of body composition profiles on oncological outcomes and postoperative intraabdominal infection following colorectal cancer surgery.  Surgery for Obesity and Related Diseases, 17(3):575–584, 2021.
  * [10] Heather Wopat, Tom Harrod, Rachel F Brem, Rebecca Kaltman, Kendall Anderson, and Kim Robien.  Body composition and chemotherapy toxicity among women treated for breast cancer: a systematic review.  Journal of Cancer Survivorship, pages 1–14, 2023.
  * [11] Gabriel Francisco Pereira Aleixo, Stephanie A Valente, Wei Wei, and Halle CF Moore.  Association of body composition and surgical outcomes in patients with early-stage breast cancer.  Breast Cancer Research and Treatment, 202(2):305–311, 2023.
  * [12] Jakob Wasserthal, Hanns-Christian Breit, Manfred T. Meyer, Maurice Pradella, Daniel Hinck, Alexander W. Sauter, Tobias Heye, Daniel T. Boll, Joshy Cyriac, Shan Yang, Michael Bach, and Martin Segeroth.  Totalsegmentator: Robust segmentation of 104 anatomic structures in ct images.  Radiology: Artificial Intelligence, 5(5):e230024, 2023.
  * [13] S. Koitka, G. Baldini, L. Kroll, N. van Landeghem, J. Haubold, M. Sung Kim, J. Kleesiek, F. Nensa, and R. Hosch.  SAROS – A large, heterogeneous, and sparsely annotated segmentation dataset on CT imaging data.  The Cancer Imaging Archive, 2023.
  * [14] K. Clark, B. Vendt, K. Smith, J. Freymann, J. Kirby, P. Koppel, S. Moore, S. Phillips, D. Maffitt, M. Pringle, L. Tarbox, and F. Prior.  The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository.  Journal of Digital Imaging, 26:1045–1057, 2013.
  * [15] Sven Koitka, Lennard Kroll, Eugen Malamutmann, Arzu Oezcelik, and Felix Nensa.  Fully-automated body composition analysis in routine CT imaging using 3d semantic segmentation convolutional neural networks.  CoRR, abs/2002.10776, 2020.
  * [16] F. Isensee, P. F. Jaeger, S. A. Kohl, J. Petersen, and K. H. Maier-Hein.  nnu-net: a self-configuring method for deep learning-based biomedical image segmentation.  Nature methods, 18(2):203–211, 2021.
  * [17] Jianfei Liu, Omid Shafaat, and Ronald M. Summers.  A dual-branch network with mixed and self-supervision for medical image segmentation: An application to segment edematous adipose tissue.  In MILLanD@MICCAI, volume 14307 of Lecture Notes in Computer Science, pages 158–167. Springer, 2023.
  * [18] J.E. Burns, J. Yao, J.J. Chen D. Chalhoub, and R.M. Summers.  A machine learning algorithm to estimate sarcopenia on abdominal ct.  Academic Radiology, 27(3):311–320, 2020.
  * [19] Jacob Cohen.  A coefficient of agreement for nominal scales.  Educational and Psychological Measurement, 20(1):37–46, 1960.


11institutetext: University at Buffalo, USA
11email: {sahmed9<EMAIL_ADDRESS>
# RealCQA: Scientific Chart Question Answering as a Test-bed for First-Order
Logic

Saleem Ahmed(✉)  0000-0001-8648-9625
Bhavin Jawade  Shubham Pandey  Srirangaraj Setlur  0000-0002-7118-9280 Venu
Govindaraju  0000-0002-5318-7409

###### Abstract

We present a comprehensive study of chart visual question-answering(QA) task,
to address the challenges faced in comprehending and extracting data from
chart visualizations within documents. Despite efforts to tackle this problem
using synthetic charts, solutions are limited by the shortage of annotated
real-world data. To fill this gap, we introduce a benchmark and dataset for
chart visual QA on real-world charts, offering a systematic analysis of the
task and a novel taxonomy for template-based chart question creation. Our
contribution includes the introduction of a new answer type, ‘list’, with both
ranked and unranked variations. Our study is conducted on a real-world chart
dataset from scientific literature, showcasing higher visual complexity
compared to other works. Our focus is on template-based QA and how it can
serve as a standard for evaluating the first-order logic capabilities of
models. The results of our experiments, conducted on a real-world out-of-
distribution dataset, provide a robust evaluation of large-scale pre-trained
models and advance the field of chart visual QA and formal logic verification
for neural networks in general. Our code and dataset is publicly available
111This is a pre-print version, accepted at ICDAR’23; https://github.com/cse-
ai-lab/RealCQA.

###### Keywords:

Charts and Document Understanding and Reasoning

##  1 Introduction

The chart question-answering[QA] task has recently received attention from a
wider community [17], [26], [8], [30], [24], [20]. While generic multi-modal
QA tasks have been studied widely, the chart-based QA task is still in its
developmental phase, especially for real-world scientific document
understanding applications.

Recent works have provided a structure for question type classification[17]
while iteratively adding complexity in chart types[8] and answer types [26].
However, no existing work fills the gap of QA on real-world charts[11] with
structured output prediction.

Figure 1: Existing datasets in chart visual QA either are fully synthetic
charts generated from synthetic data (Right sector in the above ellipse) or
synthetic charts generated from real data (left sector). None of these
datasets handle the complexity of the distribution of real-world charts found
in scientific literature. We introduce the first chart QA dataset (RealCQA) in
the third category (lower sector in the above figure) which consists of real-
world charts extracted from scientific papers along with various categories of
QA pairs. [Best viewed digitally in color].

Two main approaches for synthetic Chart-QA are: (i) considering the whole
input image as a matrix of pixels to generate output in the form of text,
answer types, etc[8], [30], [20] or (ii) first extracting tabular data by
identifying, classifying chart structural components and, then treating the
task as a table-QA task[24].

These include either numeric answers (regression task) or single-string
answers from the charts vocabulary (classification task). We further propose a
structured and unstructured list answer type task, where answers can contain
delimiter-spaced strings, where the order of strings might/or not matter. We
also include new chart types for the scatter and box plots with curated chart-
specific questions.

With the advent of representational learning over multi-modal data for
document understanding [2], [14], [35], [34], [27], [21], the task of
knowledge representation[4] and reasoning in latent space has improved
significantly from previous heuristic-driven methods used to capture
propositional logic.

Recent works have paved the way for linking $FOC_{2}$ (First Order Logic with
two variables and counting capability) with neural networks[3]. Tasks such as
learning to reason over mathematical expressions [1],[23] make a neat test bed
for NSC(Neuro-Symbolic-Computing)[5].

There has been a rich history of research in building logic-based systems, for
Theorem Proving, Conjecture Solving etc. Recent advances have seen the
efficacy of using sophisticated transformers and graph neural networks for the
purpose of mathematical reasoning over very large datasets involving millions
of intermediate logical steps[18].

One recent work claims almost a $20\%$ jump in accuracy for synthetic chart
QA, just by augmenting pretraining with mathematical reasoning [22] although
they lack a robust evaluation of the models’ reasoning capability.

To further develop models capable of formal logic in the space of document
understanding, we propose RealCQA as a robust multimodal testbed for logic and
scientific chart-based QA.

##  2 Background

We first discuss more commonly studied tasks in the literature that provide a
foundation for ChartQA. These include visual QA, document understanding, and
formal logic systems.

###  2.1 Visual QA

VQA, or Visual QA, is a task where a computer system is given an image and a
natural language question about the image and the system is expected to
generate a natural language answer[19]. VQA systems aim to mimic the ability
of humans to understand and reason about visual information and language and
to use this understanding to generate appropriate responses to questions.
Specific variations of this task include image captioning and multi-modal
retrieval.

####  2.1.1 Image Captioning

is the task where a computer system is given an input image and is expected to
generate a natural language description of the content of the image[9] . This
description should capture the main objects, actions, and events depicted in
the image, as well as the relationships between them. Image captioning systems
typically use machine learning algorithms to learn how to generate descriptive
captions from a large dataset of images and their corresponding human-
generated captions.

####  2.1.2 Multimodal Retrieval

involves retrieving images and text that are related to each other based on
their content[16]. This task involves the integration of computer vision and
natural language processing techniques and is used in a variety of
applications such as image and text search, image annotation, and automated
customer service. In image-text cross-modal retrieval, a computer system is
given a query in the form of either an image or a text and is expected to
retrieve images or texts that are related to the query. To perform image-text
cross-modal retrieval effectively, the system must be able to understand and
reason about the visual and linguistic content of both images and text and to
identify the relationships between them. This typically involves the use of
machine learning algorithms that are trained on large datasets of images and
text and their corresponding relationships.

###  2.2 Document Understanding

The field of document intelligence encompasses a broad range of tasks [6],
[31], such as localization, recognition, layout understanding, entity
recognition, and linking. In this section, we describe the downstream tasks of
document-QA, Table-QA, and Infographic-QA, which build up to Chart-QA

####  2.2.1 VQA for Document Understanding

has been explored in works such as [36] which involve document pages
comprising of tables, text and QA-pairs. The documents are sampled from
financial reports and contain lots of numbers, requiring discrete reasoning
capability to answer questions. Relational-VQA models use reasoning frameworks
based on FOL to answer questions about visual scenes. Researchers have also
explored other figure types such as Map-based QA [7]. CALM [12] proposes
extending [25] with prior knowledge reasoning, and [29] proposes models for
non-English document understanding through QA. Key requisites for Document-QA
[DQA] include (i)_Robust feature representation:_ One of the main challenges
in DQA is to effectively represent the visual and semantic content of
documents. The development of robust feature representations that capture the
relationships between objects, properties, and concepts in documents is a key
area of research in the field. (ii)_Large-scale datasets:_ Another challenge
in DQA is the lack of large-scale datasets that can be used to train and
evaluate models. The development of large-scale datasets that include a wide
variety of documents and questions is crucial for advancing the field.
(iii)_Integration of prior knowledge and context:_ In order to accurately
answer questions about documents, models must be able to effectively integrate
prior knowledge and context into their reasoning process. This requires the
development of algorithms that can reason about the relationships between
objects and concepts in a document, and that can incorporate prior knowledge
and context into the decision-making process. (iv)_Relational reasoning:_ DQA
often requires reasoning about relationships between objects and concepts in a
document. (v)_Multi-modal fusion:_ DQA requires the integration of information
from multiple modalities, including visual and semantic content. Recent works
include [25], [33], [32], [28].

####  2.2.2 Table QA

is a natural language processing (NLP) task that involves answering questions
about the information presented in tables. This task requires models to
understand the structure and content of the table, as well as the meaning of
the natural language question, in order to generate a correct answer. The
table contents are provided as text input. Recent literature in the TableQA
task include [15], [13], which present models for generating SQL queries from
natural language questions about tables.

###  2.3 Chart-VQA

We discuss two common approaches for this specific sub-area of IQA where the
input is a chart image and a corresponding query.

####  2.3.1 Semi-Structured Information Extraction (SIE)[24]

involves the following steps:(i)_Chart Text Analysis:_ Extract the tick
labels, legend, axis and chart tiles, and any other text in the image.
(ii)_Chart Structure Analysis:_ Tick association for corresponding data value
interpolation of $xy$ coordinate and the nearest tick label and legend mapping
to individual data-series components labels. (iii)_Visual Element Detection
[VED]:_ Localize the chart component (line, box, point, bar) and association
with x-tick and legend name. (iv)_Data Extraction:_ Interpolate the value
represented by each data component by using the VED module and calculate the
value from bounding ticks.

This reduces Chart-VQA to a Table-VQA task. However, this adds additional
complexity as errors are now also introduced during the data-extraction task.

####  2.3.2 Classification-Regression[20], [26]

approach has proven to be effective for chart comprehension, allowing machine
learning models to accurately classify and predict the values and trends
depicted in charts. In this school of thought, the input is directly treated
as just pixels, usually relying on the implicit representation of chart
components, plot area, visual elements, and underlying data. These features
are aggregated alongside text features of the question string where the model
learns their corresponding relations to predict either a classification
answer(string) or a regression answer(numeric). Usually, models use visual
features from a Mask-RCNN-based backbone, trained to detect chart text and
structure. These are input alongside tokenized textual queries. Answer
prediction involves predicting numeric or string type, where floats are
regressed and tokens classified.

Figure 2:  List type answers have many uses-cases specifically in chart QA but
has not been considered by existing CQA datasets. RealCQA introduces List type
QA pairs with both (i) Ranked List (ii) Unranked List. Items in a list can
consist of sets of up to 2 items.

###  2.4 Logic Order and Reasoning

We discuss formal logic, requirements for a testbed, and its applicability in
the context of Chart-QA.

####  2.4.1 Zero-order logic [ZOL]

is the basic unit of meaning, the atomic formula, a proposition that makes an
assertion, e.g. answering the root level taxonomy questions: _‘Is this chart
of type A’_ , _‘Is there a title in the chart’_ , _‘Is the dependent axis
logarithmic”_ etc. Complex statements can be formed by combining atomic
formulas using logical connectives such as ‘and’, ‘or’, and ‘not’.

####  2.4.2 First-order logic [FOL]

also known as predicate logic, is a type of formal logic that is used to study
the relationships between objects and their properties. It allows the
expression of propositions or statements that make assertions about properties
and relations, and it provides a formal language for making logical inferences
based on these assertions. In first-order logic, we have quantifiers $\forall$
(for all) and $\exists$ (there exist) to make statements about the entire
domain of discourse by involving variables that range over the objects being
discussed, e.g. a closed set of all tick labels in a chart. We can talk about
the properties of these objects or relationships between them by using 1-place
predicates or multi-place predicates, respectively, e.g. comparing data series
values at different tick locations. These predicates can be viewed as sets
where their elements are those of the domain that satisfy some property or
n-tuples that satisfy some relation, _‘Is the sum of the value of $\langle$Y
title$\rangle$ in $\langle$ i-th x tick$\rangle$ and $\langle$(i + 1)th x
tick$\rangle$ greater than the maximum value of $\langle$ title$\rangle$
across all $\langle$plural form of X title$\rangle$ ?’_

####  2.4.3 N-th order logic

uses the same quantifiers to range over predicates. This essentially allows
the quantification of sets. Using the quantifiers for elements of the sets is
provisional as required. These involve the ‘List’ type questions with
structured output, e.g. _‘Which pairs of major ticks on independent axis have
a difference greater than $\langle$i-th x tick$\rangle$ and $\langle$j-th x
tick$\rangle$, for the value of $\langle$Y title$\rangle$, arranged in the
increasing order of difference?’_. Constraining the current scope for better
evaluation, we limit this to $2^{nd}$ order logic, i.e. we create questions
with set outputs of at most 2 elements per item in the list as depicted in
Fig. 2.

####  2.4.4 A Testbed for Formal Logic

must satisfy specific requirements to ensure the correctness of the system
being tested. The first requirement is a formal specification, which should
precisely define the system’s syntax, semantics, and model-checking
algorithms. The second requirement is a set of test cases that covers all
possible scenarios and validates the system’s behavior, including its ability
to handle edge cases and exceptional conditions. The third requirement is a
repeatable, reliable, and easily extensible test harness that can accommodate
new test cases. Finally, a verification environment must be created to host
the testbed and provide necessary resources for performing the tests.

These ensure that formal logic systems are thoroughly tested and that the test
results are accurate and reliable. The exact nature of the requirements and
techniques used to meet them will vary depending on the system being tested
and the context in which it is used. Overall, rigorous testing is crucial for
establishing the correctness of logical systems and ensuring their
applicability to real-world problems. Existing experimental setups for
evaluating Chart-QA satisfy most of the above points, except the formal
specification, which we provide for a subset of our questions. These are
manually curated and verified. We will describe this in greater detail.

####  2.4.5 CQA for FOL

represents our concept of utilizing the template-based chart QA task as a
testbed of predicate logic. The innate structure of data which populates a
scientific chart aligns naturally with the previously stated formal
specification requirements. Prior research has studied VQA with charts,
however, a formal testbed has not been studied.

In FOL, sentences are written in a specific syntax and structure to allow for
precise and unambiguous representation of meaning. To translate a normal
sentence into FOL, we need to identify the objects, individuals, and
relationships described in the sentence, and express them using predicates,
variables, and logical connectives. For example, the template _Is the
difference between the value of $\langle$ Y title $\rangle$ at $\langle$ ith
x-tick $\rangle$ and $\langle$ jth x-tick $\rangle$ greater than the
difference between any two $\langle$ plural form of X-title $\rangle$ ?_ can
be converted to FOL as :

$\forall i,j,p,q:(i\neq j\neq p\neq
q)\rightarrow(|Y_{i}-Y_{j}|>|X_{p}-X_{q}|)\;$

where $\langle...\rangle$ represents variables in the template and ‘Y’ the
space of values of the dependent-variable, and ‘X’ for the independent
variable in the chart, respectively. While curating reasoning type [17]
questions, we create a subset of binary questions specifically over FOL that
are valid.

In this study, our aim is to further advance the development of the chart and
visual data parsing systems. Previous research has documented the limitations
posed by the limited availability of annotated real-world data[11]. While
absolute accuracy on a specific ChartQA dataset may not guarantee broad
generalizability, this study is a step toward establishing a comprehensive
understanding of this complex and evolving field. We believe that leveraging
the manually curated templates and structured output generated from the
semantic structure of charts presents an opportunity to effectively evaluate
the multi-modal predicate logic parsing capabilities of modern neural
networks, such as large-scale pre-trained language and layout models.

##  3 Dataset

(a) Challenge Tasks

(b) Data Distribution

Figure 3:  Train and Test Structure, Retrieval, Reasoning by answer type. For
List Type, we only curate reasoning questions for kth order FOL testing.
String/Unranked refers to a small subset of string-type retrieval or reasoning
answers where multiple equivalent conditions exist: While reading the question
string, a human would expect a single answer but multiple data series have the
same maximum/minimum etc. resulting in multiple correct single-string instance
answers.

In this section, we describe the dataset used in our study. The dataset,
called RealCQA, was created by utilizing real-world chart images and
annotations used in publicly conducted chart understanding challenges [11].
Fig.1 shows the current existing datasets in the CQA domain. The challenge
tasks around chart understanding are shown in Fig. 3, along with the annotated
data used from the publicly released train-test splits.

###  3.1 RealCQA

To generate question templates for RealCQA, we compiled templates from
previous works [8], [30], and [26]. These templates were adapted to our data
and augmented with new chart-type questions, list questions, and binary FOL
reasoning questions, forming a total of 240 templates.

The distribution of taxonomy and answer types for RealCQA is shown in Fig. 4.
We have tried to keep the templates for different answer types with equal
proportions. However, when these are used to create the actual QA pairs, the
data gets skewed depending on underlying availability. Our dataset consists of
a majority of ‘Reasoning’ type questions, as seen in previous works [17].
However, we also focus on creating binary reasoning questions that satisfy
FOL. These form a major chunk of the dataset since templates with variables
for i-th/j-th tick/data series are combinatorial in nature, and we create them
exhaustively over the closed set of objects present in the chart.

We use the ‘Structure, Retrieval, Reasoning’ taxonomy proposed in previous
works [17] to categorize our questions. However, we further demarcate them as
Types 1, 2, 3, 4, depending on their characteristics. Type-1 refers to any
questions that can be formed at the (root) level of the whole chart image,
mostly ZOL. Type-2 further refers to ZOL questions for specific chart
components, requiring the model to identify them. Type-3 and Type-4 are data
retrieval/reasoning. Each has a further specific sub-class depending on the
exact component, chart type, etc., as shown in Fig. 4.

The statistics of the dataset are shown in Fig. 5 using the previous
nomenclature of ‘Structural’, ‘Retrieval’, and ‘Reasoning’. For the List Type,
we only curate reasoning questions for $k^{th}$ order FOL testing.
String/Unranked refers to a small subset of string-type retrieval or reasoning
answers where multiple equivalent conditions exist. While reading the question
string, a human would expect a single answer, but multiple data series have
the same maximum/minimum, resulting in multiple correct single-string instance
answers. These are generally outliers.

Overall, the RealCQA dataset offers a diverse range of questions that require
various levels of chart understanding and reasoning abilities. The dataset is
publicly available and can be used for further research and evaluation in the
chart understanding domain.

Figure 4: Taxonomy and Type distribution: First pie is test set QA pairs,
second training set QA pairs and third training set templates. Legend is in
decreasing order. Best viewed digitally in color.

(a) Train Distribution

(b) Test Distribution

Figure 5:  Train and Test Structure, Retrieval, Reasoning by answer type.  Figure 6: Trend across different sampling strategies 1-5. X-axis represents each of the 9357 test-images, Y-axis the 240 templates each plotted with different colored bars at every 10th index, and Z axis shows the count of QA pairs. Table 1: Total QA pairs per sampling strategy, bold shows minimum. Answer Type | Sample 1 | Sample 2 | Sample 3 | Sample 4 | Sample 5
---|---|---|---|---|---
Total | 367139 | 322404 | 276091 | 231356 | 203735
String | 19525 | 3489 | 18548 | 2512 | 19046
Numeric | 115391 | 107153 | 93096 | 84858 | 78680
Ranked | 16389 | 13903 | 13357 | 10871 | 14228
Unranked | 44702 | 43310 | 27019 | 25627 | 25041
Binary | 171132 | 154549 | 124071 | 107488 | 66740

###  3.2 Sampling Strategies for Dataset Evaluation

For the purpose of general chart visual question answering, generating
balanced and representative datasets is of paramount importance for training
and evaluating models. However, when it comes to logic testbeds, the over-
representation of specific templates or question types is not necessarily a
disadvantage, as it allows for a more nuanced assessment of a model’s logical
reasoning capabilities. Nonetheless, it is still relevant to explore the
impact of dataset sampling on evaluation results.

To this end, we devised five different sampling strategies and evaluated their
effect on our dataset. The first strategy, exhaustive sampling, consists of
including all available question-answer pairs. The remaining strategies aim to
modify the distribution of questions per chart based on different criteria.
Specifically, the second strategy, increasing lower bound, focuses on charts
with a minimum number of questions greater than or equal to a threshold $K$.
This strategy aims to address under-represented question types, such as root
and structural questions. Conversely, the third strategy, decreasing upper
bound, selects charts with a maximum number of questions less than or equal to
a threshold $L$. This strategy is intended to address over-represented
question types, typically combinatorial binary reasoning questions. The fourth
strategy combines the effects of the second and third strategies, aiming to
remove both under and over-represented charts. Finally, the fifth strategy,
flat cap, selects a fixed number of questions per chart per template, thereby
creating a more uniform dataset.

Fig. 6 illustrates how the different sampling strategies affect the number of
questions per chart per template, while Table 1 provides the actual number of
QA-pairs per sampling. To be specific, we calculate the lower and upper 10%
for the second and third strategies, respectively. For the flat cap strategy,
we randomly select 150 QA-pairs for each template per chart.

By analyzing the impact of the different sampling strategies, we can gain a
better understanding of how removing specific sections from the test-set
affects evaluation results. These findings can be useful for modulating the
training-set as required, and ultimately for developing more robust and
accurate visual question-answering models.

###  3.3 Evaluation Metrics

In this study, we propose an evaluation metric based on the accuracy of
answers. The proposed task involves four types of answers, each with its
specific calculation method.

First, for numerical answers, we measure the accuracy of regression errors
using L2 or L1 differences or the ER-error rate. In PlotQA-D [1], we consider
a regression answer correct if it falls within $\pm 5\%$ tolerance from the
ground truth value. Second, for single string answers, we use string-matching
edit distance and count perfect matches as correct. Third, for unordered lists
of strings, we use string-matching edit distance. For each of the $K$ queries
and $M$ matches, we calculate $K\times M$ scores, and the mutually exclusive
best match is aggregated per string instance normalized by $K$. Fourth, for
ranked order lists of strings, we use the nDCG@K ranking metric, where K is
the size of the ground-truth list. nDCG is a normalized version of the DCG
(Discounted Cumulative Gain) metric, which is widely used to evaluate the
ranking quality of information retrieval systems, such as search engines and
recommendation systems. This metric assigns a relevance score to each item in
a ranked list based on the user’s preferences, and then discounts these scores
using a logarithmic function, with items appearing lower in the ranking
receiving lower scores. Lastly, for nested lists, where each item is a set, we
evaluate the results invariant of set order, but list order matters in ranked
lists.

##  4 Experiments

We benchmark multiple existing generic visual QA and chart-specific visual QA
methods on RealCQA. Fig 7 shows the generic existing architecture used for CQA
task. The model either learns visual and data features separately or in the
same shared space and then uses some fusion model to generate the final
answer. These are primarily trained on synthetic charts. Here, we evaluate
multiple baseline models that have been proposed recently including ChartQA
and CRCT. We present both the synthetic pre-training evaluation on RealCQA and
RealCQA finetuned evaluation. Below we briefly discuss the model architecture
for the baseline methods in more detail:

Figure 7: A generalized framework to represent existing methods for chart QA.

####  4.0.1 VLT5[10]

VLT5 is a state-of-the-art unified framework that leverages a multimodal text
conditioning language objective to perform different tasks within a unified
architecture. In this framework, the model learns to generate labels in the
text space based on the visual and textual inputs. In our study, we use VLT5
to perform the task of table-based question answering. Specifically, VLT5
takes as input pre-trained region-based visual features obtained from Faster-
RCNN, which was pre-trained on PlotQA [26]. These visual features, along with
textual tokens, are projected and fed through a unified bi-directional multi-
modal encoder. Additionally, a language decoder is trained in an auto-
regressive setting to perform text generation. In the textual context, we
provide a pre-extracted gold standard table of the chart as concatenated input
along with the query question.

We present the performance of VLT5 on the RealCQA test-set, which comprises
approximately 683 charts with gold-data table annotation. The evaluation is
conducted on the charts in this test-set, with a score of zero assigned to the
remaining charts. The results of this evaluation are presented in Table 2 and
Table 3, where Row 1 shows the performance of VLT5 segregated by Answer-Type
and Question-Type, respectively.

####  4.0.2 ChartQA [24]

The authors of ChartQA introduced a large-scale benchmark dataset comprising
of 9.6K human-written questions and 23.1K questions generated from human-
written chart summaries. To evaluate the effectiveness of the dataset, two
transformer-based multimodal architectures, namely VisionTapas and VLT5, were
benchmarked on ChartQA using data tables and visual features as context.

In this study, we fine-tuned the VLT5 multi-modal encoder with ChartQA visual
features that were pre-trained on PlotQA. We utilized ChartQA pre-trained Mask
RCNN visual features with VLT5 multi-modal attention in both Row 1 and Row 2
of Table 2 and Table 3, respectively. Results were segregated based on Answer-
Type and Question-Type, as presented in Table 2 and Table 3. Since the
evaluation requires a data-table, we evaluated the model on 683 charts and
assigned zero scores to the remaining QAs.

####  4.0.3 CRCT [20]

The paper proposes a novel ChartVQA approach called Classification Regression
Chart Transformer (CRCT) that aims to address the limitations of existing
methods in the field. The authors argue that the saturation of previous
methods is due to biases, oversimplification, and classification-oriented Q&A
in common datasets and benchmarks. To overcome these challenges, the CRCT
model leverages a dual-branch transformer with a chart element detector that
extracts both textual and visual information from charts. The model also
features joint processing of all textual elements in the chart to capture
inter and intra-relations between elements.

The proposed hybrid prediction head unifies classification and regression into
a single model, optimizing the end-to-end approach using multi-task learning.
For visual context, they fine-tuned a Mask-RCNN on PlotQA, while for textual
context, they used text detections and recognition output from a standard OCR
such as tesseract. We evaluated both a CRCT model fully pre-trained on the
PlotQA dataset and a CRCT model fine-tuned on RealCQA for stage 2 with pre-
trained FasterRCNN. We report the performance of both models in Row 3 and Row
4 of Table 2 and Table 3, respectively.

Table 2: Performance of existing Visual Question Answering Methods on RealCQA based on Answer Type |  Total Accuracy | String | Numeric | Rank | Unranked | Binary
---|---|---|---|---|---|---
| VLT5
---
(VG Pretrained)
0.2399 | 0.0008 | 0.0325 | 0.0106 | 0.0002 | 0.4916
VLT5 (RealCQA Finetuned) | 0.3106 | 0.3068 | 0.1487 | 0.0246 | 0.0048 | 0.5275
| CRCT
---
(PlotQA Pretrained)
0.1787 | 0.0350 | 0.0412 | 0.0015 | 0.0016 | 0.3515
| CRCT
---
(RealCQA Finetuned)
0.1880 | 0.0323 | 0.3158 | 0.0286 | 0.0124 | 0.1807
Table 3: Performance of existing Visual Question Answering Methods on Real-CQA based on Question Complexity |  Root | Structure | Retrieval | Reasoning
---|---|---|---|---
| VLT5
---
(VG Pretrained)
0.0764 | 0.1416 | 0.0765 | 0.2620
VLT5 (RealCQA Finetuned) | 0.1800 | 0.4352 | 0.5877 | 0.2937
| CRCT
---
(PlotQA Pretrained)
0.0773 | 0.2338 | 0.1168 | 0.1778
| CRCT
---
(RealCQA Finetuned)
0.0115 | 0.1497 | 0.3131 | 0.1959

###  4.1 Results

We present the quantitative results of our experiments, which are summarized
in Table 2 and Table 3. We find that the VLT5 model [10], does not perform
well on the RealCQA dataset when using pre-trained Mask RCNN visual features
and RealCQA’s gold-data table as input. The performance of 49.16% on binary-
type answers is as bad as random assignment. However, by fine-tuning the VLT5
model’s multi-modal alignment module on RealCQA with modified tokenization to
handle list-type answers, we observe significant improvements in performance
on all answer types (Table 1) and question types (Table 2). Specifically, the
performance on string type answers improves from 0.008% to 30.68%, and the
overall accuracy of QA pairs improves from 23.99% to 31.06%. Table 3 compares
the performance of the CRCT model [20] fully pre-trained on PlotQA and VG
Pretrained VLT5 on Root, Structure, and Retrieval type questions. We find that
the fully pre-trained CRCT outperforms VG Pretrained VLT5 on these question
types. However, fine-tuning the CRCT model on RealCQA leads to significant
improvements in performance on Numeric, Ranked List, and Unranked List type
answers, as shown in Table 2. Notably, fine-tuned CRCT achieves the best
performance of 31.58% on numeric type answers. Our results highlight the
importance of RealCQA as a standard test bed for evaluating chart visual QA
methods, as even models that perform well on synthetic datasets such as PlotQA
and FigureQA struggle to generalize to real-world chart distributions.

Table 4: An ablation on the performance of different models based on different sampling strategies. Here, LB - Lower Bound, UB - Upper Bound, Full - No Sampling. Data | Model |  | Total
---
Accuracy
String | Numeric | Rank | Unrank | Binary
Full |  VLT5 Pre-Trained | 0.2399 | 0.0008 | 0.0325 | 0.0106 | 0.0002 | 0.4916
LB |  | 0.0032 | 0.0252 | 0.0092 | 0.0001 | 0.5182
UB |  | 0.0008 | 0.0345 | 0.0090 | 0.0002 | 0.4947
LB+UB |  | 0.0044 | 0.0255 | 0.0069 | 0.0001 | 0.5334
150max |  | 0.0008 | 0.0403 | 0.0122 | 0.0003 | 0.4349
Full |  VLT5 (RealCQA Fine-Tuned) | 0.3106 | 0.3068 | 0.1487 | 0.0246 | 0.0048 | 0.5275
LB |  | 0.4004 | 0.1305 | 0.0201 | 0.0041 | 0.5487
UB |  | 0.3165 | 0.1585 | 0.0246 | 0.0070 | 0.5258
LB+UB |  | 0.5084 | 0.1365 | 0.0187 | 0.0061 | 0.5559
150max |  | 0.3146 | 0.1827 | 0.0283 | 0.0085 | 0.4947
Full |  CRCT Pretrained | 0.1787 | 0.0350 | 0.0412 | 0.0015 | 0.0000 | 0.3515
LB |  | 0.0224 | 0.0359 | 0.0013 | 0.0000 | 0.3473
UB |  | 0.0367 | 0.0428 | 0.0012 | 0.0000 | 0.3742
LB+UB |  | 0.0299 | 0.0363 | 0.0009 | 0.0000 | 0.3716
150max |  | 0.0359 | 0.0499 | 0.0017 | 0.0000 | 0.3601
Full |  CRCT (RealCQAFine-Tuned) | 0.1880 | 0.0323 | 0.3158 | 0.0286 | 0.0124 | 0.1807
LB |  | 0.0759 | 0.3196 | 0.0263 | 0.0102 | 0.1923
UB |  | 0.0319 | 0.3316 | 0.0327 | 0.0190 | 0.1977
LB+UB |  | 0.0903 | 0.3379 | 0.0309 | 0.0158 | 0.2170
150max |  | 0.0322 | 0.3371 | 0.0329 | 0.0220 | 0.1316

We present an ablation study that examines the impact of different sampling
strategies on the performance of our model. The results of this study are
summarized in Table 4. The study reveals that the 4th sampling strategy, which
combines both the upper and lower bounds, consistently achieves the highest
overall, string, and binary type accuracy across various experimental
settings. On the other hand, the 5th strategy, which produces the most uniform
test set, yields top accuracies for numeric and list-type answers. However,
this strategy has the smallest size in terms of overall, unranked, and binary
questions, and it attains the highest accuracy for unranked list type
questions, which are representative of Kth Order Logic questions. It is worth
noting that the 5th strategy may remove most of the challenging QA pairs,
making it less desirable for our objective.

Overall, this study highlights the importance of carefully selecting the
sampling strategy to obtain a test set that is representative of the
distribution of real-world chart visual QA. The 4th strategy appears to be a
promising choice, as it achieves high accuracy across different answer types
while maintaining a sufficient number of challenging QA pairs.

##  5 Conclusion

In addition to our contribution of curating a novel FOL-Testbed and a dataset
for the evaluation of CQA for real charts, we have also thoroughly evaluated
several state-of-the-art visual question answering models on the RealCQA
dataset. Our experiments reveal that while some models perform well on
synthetic datasets like PlotQA and FigureQA, their performance significantly
drops when tested on RealCQA, demonstrating the need for a more realistic and
challenging benchmark like RealCQA. We have shown that our proposed method,
CRCT, significantly outperforms previous models on several question types,
especially on numeric type questions. Our ablation study further highlights
the importance of sampling strategies in constructing a diverse and
representative test set.

Overall, our study emphasizes the importance of multimodal learning and
reasoning in visual question answering and provides insights into the
limitations and opportunities of current state-of-the-art models. Future work
can build on our findings by exploring more sophisticated models that
integrate text, image, and reasoning more effectively, as well as developing
new evaluation metrics that capture the full complexity of real-world chart
questions. Additionally, expanding the dataset to cover a wider range of chart
types and complexities can further improve the generalization capabilities of
visual question answering models and lead to more impactful applications in
areas such as data analysis and decision-making.

## References

  * [1] Ahmed, S., Davila, K., Setlur, S., Govindaraju, V.: Equation attention relationship network (earn) : A geometric deep metric framework for learning similar math expression embedding. In: 2020 25th International Conference on Pattern Recognition (ICPR). pp. 6282–6289 (2021). https://doi.org/10.1109/ICPR48806.2021.9412619
  * [2] Appalaraju, S., Jasani, B., Kota, B.U., Xie, Y., Manmatha, R.: Docformer: End-to-end transformer for document understanding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 993–1003 (2021)
  * [3] Barceló, P., Kostylev, E.V., Monet, M., Pérez, J., Reutter, J., Silva, J.P.: The logical expressiveness of graph neural networks. In: 8th International Conference on Learning Representations (ICLR 2020) (2020)
  * [4] Battaglia, P.W., Hamrick, J.B., Bapst, V., Sanchez-Gonzalez, A., Zambaldi, V., Malinowski, M., Tacchetti, A., Raposo, D., Santoro, A., Faulkner, R., et al.: Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261 (2018)
  * [5] Besold, T.R., d’Avila Garcez, A.S., Bader, S., Bowman, H., Domingos, P.M., Hitzler, P., Kühnberger, K., Lamb, L.C., Lowd, D., Lima, P.M.V., de Penning, L., Pinkas, G., Poon, H., Zaverucha, G.: Neural-symbolic learning and reasoning: A survey and interpretation. CoRR abs/1711.03902 (2017)
  * [6] Borchmann, Ł., Pietruszka, M., Stanislawek, T., Jurkiewicz, D., Turski, M., Szyndler, K., Graliński, F.: Due: End-to-end document understanding benchmark. In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) (2021)
  * [7] Chang, S., Palzer, D., Li, J., Fosler-Lussier, E., Xiao, N.: Mapqa: A dataset for question answering on choropleth maps. arXiv preprint arXiv:2211.08545 (2022)
  * [8] Chaudhry, R., Shekhar, S., Gupta, U., Maneriker, P., Bansal, P., Joshi, A.: Leaf-qa: Locate, encode & attend for figure question answering. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 3512–3521 (2020)
  * [9] Chen, C., Zhang, R., Kim, S., Cohen, S., Yu, T., Rossi, R., Bunescu, R.: Neural caption generation over figures. In: Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers. pp. 482–485 (2019)
  * [10] Cho, J., Lei, J., Tan, H., Bansal, M.: Unifying vision-and-language tasks via text generation. In: International Conference on Machine Learning. pp. 1931–1942. PMLR (2021)
  * [11] Davila, K., Xu, F., Ahmed, S., Mendoza, D.A., Setlur, S., Govindaraju, V.: Icpr 2022: Challenge on harvesting raw tables from infographics (chart-infographics). In: 2022 26th International Conference on Pattern Recognition (ICPR). pp. 4995–5001 (2022). https://doi.org/10.1109/ICPR56361.2022.9956289
  * [12] Du, Q., Wang, Q., Li, K., Tian, J., Xiao, L., Jin, Y.: Calm: Commen-sense knowledge augmentation for document image understanding. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 3282–3290 (2022)
  * [13] Eisenschlos, J.M., Gor, M., Müller, T., Cohen, W.W.: MATE: multi-view attention for table transformer efficiency. CoRR abs/2109.04312 (2021)
  * [14] Gu, J., Kuen, J., Morariu, V.I., Zhao, H., Jain, R., Barmpalios, N., Nenkova, A., Sun, T.: Unidoc: Unified pretraining framework for document understanding. Advances in Neural Information Processing Systems 34, 39–50 (2021)
  * [15] Herzig, J., Nowak, P.K., Müller, T., Piccinno, F., Eisenschlos, J.M.: Tapas: Weakly supervised table parsing via pre-training. arXiv preprint arXiv:2004.02349 (2020)
  * [16] Jawade, B., Mohan, D.D., Ali, N.M., Setlur, S., Govindaraju, V.: Napreg: Nouns as proxies regularization for semantically aware cross-modal embeddings. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 1135–1144 (January 2023)
  * [17] Kafle, K., Price, B., Cohen, S., Kanan, C.: Dvqa: Understanding data visualizations via question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5648–5656 (2018)
  * [18] Kaliszyk, C., Chollet, F., Szegedy, C.: Holstep: A machine learning dataset for higher-order logic theorem proving. arXiv preprint arXiv:1703.00426 (2017)
  * [19] Kodali, V., Berleant, D.: Recent, rapid advancement in visual question answering: a review. In: 2022 IEEE International Conference on Electro Information Technology (eIT). pp. 139–146 (2022). https://doi.org/10.1109/eIT53891.2022.9813988
  * [20] Levy, M., Ben-Ari, R., Lischinski, D.: Classification-regression for chart comprehension. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXVI. pp. 469–484. Springer (2022)
  * [21] Li, P., Gu, J., Kuen, J., Morariu, V.I., Zhao, H., Jain, R., Manjunatha, V., Liu, H.: Selfdoc: Self-supervised document representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5652–5660 (2021)
  * [22] Liu, F., Piccinno, F., Krichene, S., Pang, C., Lee, K., Joshi, M., Altun, Y., Collier, N., Eisenschlos, J.M.: Matcha: Enhancing visual language pretraining with math reasoning and chart derendering. arXiv preprint arXiv:2212.09662 (2022)
  * [23] Mansouri, B., Agarwal, A., Oard, D.W., Zanibbi, R.: Advancing math-aware search: The arqmath-3 lab at clef 2022. In: Advances in Information Retrieval: 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10–14, 2022, Proceedings, Part II. pp. 408–415. Springer (2022)
  * [24] Masry, A., Long, D.X., Tan, J.Q., Joty, S., Hoque, E.: Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244 (2022)
  * [25] Mathew, M., Karatzas, D., Jawahar, C.: Docvqa: A dataset for vqa on document images. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp. 2200–2209 (2021)
  * [26] Methani, N., Ganguly, P., Khapra, M.M., Kumar, P.: Plotqa: Reasoning over scientific plots. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 1527–1536 (2020)
  * [27] Powalski, R., Borchmann, Ł., Jurkiewicz, D., Dwojak, T., Pietruszka, M., Pałka, G.: Going full-tilt boogie on document understanding with text-image-layout transformer. In: Document Analysis and Recognition–ICDAR 2021: 16th International Conference, Lausanne, Switzerland, September 5–10, 2021, Proceedings, Part II 16. pp. 732–747. Springer (2021)
  * [28] Qi, L., Lv, S., Li, H., Liu, J., Zhang, Y., She, Q., Wu, H., Wang, H., Liu, T.: Dureadervis: A: A chinese dataset for open-domain document visual question answering. In: Findings of the Association for Computational Linguistics: ACL 2022\. pp. 1338–1351 (2022)
  * [29] Ščavnická, Š., Štefánik, M., Kadlčík, M., Geletka, M., Sojka, P.: Towards general document understanding through question answering. RASLAN 2022 Recent Advances in Slavonic Natural Language Processing p. 183 (2022)
  * [30] Singh, H., Shekhar, S.: Stl-cqa: Structure-based transformers with localization and encoding for chart question answering. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 3275–3284 (2020)
  * [31] Tanaka, R., Nishida, K., Yoshida, S.: Visualmrc: Machine reading comprehension on document images. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 13878–13888 (2021)
  * [32] Tito, R., Mathew, M., Jawahar, C., Valveny, E., Karatzas, D.: Icdar 2021 competition on document visual question answering. In: Document Analysis and Recognition–ICDAR 2021: 16th International Conference, Lausanne, Switzerland, September 5–10, 2021, Proceedings, Part IV 16. pp. 635–649. Springer (2021)
  * [33] Wu, X., Zheng, D., Wang, R., Sun, J., Hu, M., Feng, F., Wang, X., Jiang, H., Yang, F.: A region-based document vqa. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 4909–4920 (2022)
  * [34] Xu, Y., Li, M., Cui, L., Huang, S., Wei, F., Zhou, M.: Layoutlm: Pre-training of text and layout for document image understanding. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 1192–1200 (2020)
  * [35] Zhong, X., Tang, J., Yepes, A.J.: Publaynet: largest dataset ever for document layout analysis. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 1015–1022. IEEE (2019)
  * [36] Zhu, F., Lei, W., Feng, F., Wang, C., Zhang, H., Chua, T.S.: Towards complex document understanding by discrete reasoning. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 4857–4866 (2022)