# JDocQA: Japanese Document Question Answering
Dataset for Generative Language Models

###### Abstract

Document question answering is a task of question answering on given documents
such as reports, slides, pamphlets, and websites, and it is a truly demanding
task as paper and electronic forms of documents are so common in our society.
This is known as a quite challenging task because it requires not only text
understanding but also understanding of figures and tables, and hence visual
question answering (VQA) methods are often examined in addition to textual
approaches. We introduce Japanese Document Question Answering (JDocQA), a
large-scale document-based QA dataset, essentially requiring both visual and
textual information to answer questions, which comprises 5,504 documents in
PDF format and annotated 11,600 question-and-answer instances in Japanese.
Each QA instance includes references to the document pages and bounding boxes
for the answer clues. We incorporate multiple categories of questions and
unanswerable questions from the document for realistic question-answering
applications. We empirically evaluate the effectiveness of our dataset with
text-based large language models (LLMs) and multimodal models. Incorporating
unanswerable questions in finetuning may contribute to harnessing the so-
called hallucination generation.

Keywords: Multimodal Document Processing, Question Answering, Natural Language
Generation

JDocQA: Japanese Document Question Answering
Dataset for Generative Language Models

Eri Onami1,2, Shuhei Kurita2, Taiki Miyanishi3, Taro Watanabe1
---
1Nara Institute of Science and Technology, 2RIKEN, 3ATR
{onami.eri.ob6<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

A thorough understanding of documents that are composed of both texts and
graphical elements such as slides, reports, webpages, and pamphlets is
essential for intelligent agents that process multimedia documents and answer
some questions on such documents. Document visual understandings have been
studied to achieve joint understandings of textual and visual elements in such
documents or images, including bookcovers Mathew et al. (2021a), scene images
with characters Singh et al. (2019), webpages Tanaka et al. (2021), tables
Smock et al. (2022) and slides Tanaka et al. (2023). These datasets have
received significant attention as documents are a common form in various
industrial, public, and private sectors in the English domain. It is also
notable that the document visual question answering tasks are still quite
difficult despite its significance in industries because they heavily rely on
both textual and visual modalities as the documents often include complex
visual alignments of texts on figures, charts and illustrations. Especially in
document question answering, models are required to connect multiple
modalities to figure out answers. There are quite limited datasets in which
both visual and textual information is required to answer questions on
documents. It is also a problem that despite the significance of these tasks,
the primary focus of these datasets is limited to the English domain and
dataset constructions on other languages are still limited. As a document
question-answering task, Japanese documents have several characteristics
compared to English documents. One of the major difficulties in Japanese
document processing lies in the two official writing styles in Japanese: one
is a left-to-right horizontal style and the other is an upside-to-bottom
vertical style, which requires both writing style comprehension in the
dataset.

There has been significant progress in the generative large language models
(LLMs) and multimodal models these days. GPT-4 (OpenAI, 2023) allowing zero-
shot applications in both language related and even in multimodal tasks.
InstructBLIP Dai et al. (2023) takes both textual and image inputs and
generates texts such as image captions or visual question answering following
textual prompts. The success of LLMs also triggers the competitive development
of several publicly-available LLMs in Japanese. Instruction tuning of LLM can
improve its ability to adhere to certain domains or usage, rendering them more
suitable for particular applications rather than maintaining its ability in a
general understanding of language and limited expertise (Mishra et al., 2022;
Sanh et al., 2022). While there have been numerous attempts to fine-tune
instruction for highly technical and professional adaptation (Ouyang et al.,
2022; Wei et al., 2022), there is still a lack of adequately prepared high-
quality visual question and answering datasets that can be used for generative
language model-based question answering and particularly developed outside of
the English domain.

Figure 1: JDocQA sample question answering instances in four question
categories with the annotated bounding boxes of the supporting facts in red
color.

To address the demand for a large-scale and fully annotated Japanese document
question answering dataset, we introduce a JDocQA dataset by collecting
Japanese documents in PDF styles from open-access sources including multiple
formats of documents: slides, reports, websites and pamphlets and manually
annotating question-answer pairs on them. JDocQA consists of 11,600 question
and answer pairs on the collected 5,504 documents as references for answering
the question, four different question categories and 1,000 multi-page
questions. Each question is designed to refer to both textual and visual
components such as tables or figures by annotators. We also introduce
unanswerable questions: questions that have no answer clues in the referenced
documents. In experiments, we first present the effectiveness of finetuning
LLM with our dataset. We also suggest that incorporating these unanswerable
questions can contribute to mitigating hallucination, which is often observed
during the generation by LLMs.

## 2\. Related Work

#### Multimodal question answering datasets.

Visual question answering is a task of question-answering given visual
contexts such as images following textual queries (Malinowski and Fritz, 2014;
Antol et al., 2015). Earlier VQA studies have been not limited to images alone
but cover various forms of media such as textbook (Kembhavi et al., 2017),
recipe (Yagcioglu et al., 2018), comic book (Iyyer et al., 2016), movie
(Tapaswi et al., 2016). Among them, a document VQA, which is a task designed
for text embedded in real-world images, has attracted a lot of attention
toward a comprehensive understanding of documents from both the visual and
textual sides. Currently, some useful document VQA datasets have been
published, such as OCR-VQA (Mishra et al., 2019), TextVQA (Singh et al.,
2019), and DocVQA (Mathew et al., 2021a), VisualMRC (Tanaka et al., 2021),
WebSRC (Chen et al., 2021), and InfographicVQA (Mathew et al., 2022). Most of
these studies concentrate on the single-image VQA where each question-answer
pair has a single relevant image that always include sufficient information
for question-answering. Unlike the single-image VQA, the ability to comprehend
multiple pages or charts to answer questions is more practical for
understanding the slides and documents people read in the daily work. To
tackle such multi-image VQA, MultiModalQA (Talmor et al., 2021), MP-DocVQA
(Tito et al., 2022) and SlideVQA (Tanaka et al., 2023) concentrate on the
multi-hop and numerical reasoning while considering multimodal context similar
to previous works Rajpurkar et al. (2016); Yang et al. (2018); Dua et al.
(2019). It is also notable that in document question answering Tang et al.
(2023) proposed Universal Document Processing (UDOP), unifying vision, text,
and layout of the input document through vision-text-layout Transformer.

Category | Documents | QA | (1) Yes/No | (2) Factoid | (3) Numerical | (4) Open-ended | Multi-page | Unanswerable
---|---|---|---|---|---|---|---|---
Pamphlet | 1,715 | 4,025 | 605 | 748 | 660 | 2,012 | 46 | 671
Slide | 1,640 | 3,276 | 545 | 593 | 507 | 1,631 | 448 | 449
Report | 2,086 | 4,167 | 703 | 687 | 693 | 2,084 | 506 | 668
Website | 67 | 132 | 2 | 24 | 6 | 100 | 0 | 0
Total | 5,504 | 11,600 | 1,855 | 2,052 | 1,866 | 5,827 | 1,000 | 1,788
Table 1:  Number of document styles and question-answer pairs by the four question categories, multi-page and unanswerable questions.  Category | (1) Y/N | (2) Fact. | (3) Num. | (4) Open.
---|---|---|---|---
Context | 963.81 | 1036.63 | 1020.04 | 1017.25
Question | 67.75 | 61.26 | 60.36 | 65.44
Answer | 3.77 | 16.01 | 8.22 | 65.97
Table 2: Average character length. Dataset | #Questions | #Images | #BBoxes | Language | Multihop
---|---|---|---|---|---
OCR-VQA (Mishra et al., 2019) | 1002k | 207k | - | English | -
DocVQA (Mathew et al., 2021a) | 50k | 12k | - | English | -
InfographicVQA (Mathew et al., 2022) | 5.9k | 30k | - | English | -
MP-DocVQA (Tito et al., 2022) | 46k | 48k | - | English | ✓
SlideVQA (Tanaka et al., 2023) | 14.5k | 52k | 890k | English | ✓
JDocQA (Ours) | 11.6k | 268k | 11k | Japanese | ✓
Table 3: The comparison of the document question answering datasets.

#### Text-based question answering in Japanese.

Some related tasks of text-based question answering for Japanese have been
studied (Miyazaki and Shimizu, 2016; Yanaka and Mineshima, 2022; Takahashi et
al., 2019; Kurihara et al., 2022). Miyazaki and Shimizu (2016) created a
Japanese image captioning dataset, which is the Japanese version of the MS-
COCO captions dataset, and demonstrated that using both bilingual datasets
outperforms using monolingual ones. Yanaka and Mineshima (2022) introduced a
Japanese textual entailment dataset and highlighted that many existing models
that have focused on English do not adequately account for Japanese language
characteristics. Takahashi et al. (2019) proposed a QA dataset based on
Japanese blogs related to driving, with the aim of creating a model that can
understand the meaning of sentences or texts in Japanese. It is also notable
that Miyao and Kawazoe (2013) and relevant Todai Robot
Project111https://21robot.org/index-e.html arranged the dataset of Japanese
University entrance-exams. In math and physics subjects, their dataset
includes limited multimodal contents in DTD file format, although it doesn’t
cover general domains as of JDocQA. Among the datasets related to the Japanese
language, JGLUE (Kurihara et al., 2022) is similar to our work in terms of the
aim of the datasets. JGLUE is a large-scale natural language understanding
(NLU) benchmark purposed for the evaluation of LLM. It includes various tasks,
such as text classification, sentence pair classification, and QA to assess
Japanese comprehension. In contrast to JGLUE, our dataset offers a diverse
range of question types which can be useful for instruction tuning and
contains both Japanese text and image data, which can be used for multimodal
models. Another key difference with JGLUE is that JDocQA incorporates
unanswerable questions to help suppress hallucinations.

## 3\. Dataset

### 3.1. Task Overview and Formulation

We consider generative question answering where a model generates a textual
answer following the document context and textual question. For realistic
applications of a wide range of user questions for documents, we prepare four
categories of questions: (1) yes/no, (2) factoid, (3) numerical, and (4) open-
ended. In yes/no questions, answers are “yes” or “no.” In factoid questions,
answers are some facts, such as named entities, that typically appear in the
given documents. In numerical questions, answers are numeric values, often
including some numerals (some units, e.g., km or Japanese numerals such as “個
(objects)” and “人 (persons)”). These numeric values are written in the
documents or are calculated from other numbers in the documents. In open-ended
questions, free-form responses are required. For such questions, we aim to
assess complex comprehension abilities, such as the ability to form opinions
or brief explanations based on the provided contexts and questions. Figure 1
presents samples of these four categories of questions. All examples include
diverse images and question types related to some Japanese documents
collected. We also include unanswerable questions for each question category.

In the realistic applications of the question answering, no answers can be
found in the referenced document. Therefore, it is expected that the correct
responses for such questions are “not mentioned in the text.” The prediction
of the unanswerable questions is not addressed in previous Japanese question
answering datasets such as Kurihara et al. (2022).

Figure 2: The number of visual information categories.

### 3.2. Dataset Statistics

JDocQA dataset comprises 5,504 files and 11,600 question-and-answer pairs in
Japanese. The statistics of categorized question types are as follows: (1)
yes/no questions: 1,855, (2) factoid questions: 2,052, (3) numerical
questions: 1,866, (4) open-ended questions: 5,827. Additionally, 1,788
questions require referencing multiple pages to answer, and in 1,000 questions
the correct answer is not mentioned in the text, as shown in Table 1. Some PDF
documents contain both slide and report formats within the same documents. For
such documents, we count them in both categories of the slide and report
formats when we calculate the total number of the documents222The number of
total documents is not the sum of the number of the subcategories in Table 1
due to this counting.. Table 2 represents the average length of the context,
question, and answer in our dataset, and Figure 2 shows the category of the
visual information referenced by question or answer in our dataset. The
comparison of document question answering datasets are shown in Table 3.

### 3.3. Dataset Creation

The overall dataset creation and annotation process is presented in Figure 3.

#### PDF collection.

We gather public documents, such as, municipality pamphlets and websites, that
are created by Japanese governmental agencies or local governments. We
manually collected PDF documents from open-access resources such as Japanese
National Diet Library (NDL)’s digital collection, web archive projects
(WARP)333https://warp.ndl.go.jp/ and websites of Japanese government
ministries. We manually gathered documents such as reports, pamphlets or
websites that are published by public or quasi-public sectors, such as local
governments or public universities through WARP. We also gather Japanese
ministry documents such as slides and reports from their websites following
the government agencies’ policies. Those documents cover a wide range of
topics, for instance, economic policies, education policies, labor issues,
health and hygiene, agriculture, forestry, fisheries, culture and arts,
history, related to governmental policy or policy guidelines, as well as the
everyday affairs of local governments. These documents also include visual
elements such as figures, tables, charts, pictures, or mandala charts, complex
figures with a combination of texts and objects typically seen in the Japanese
public administrative sector’s official document. We classify these documents
into four categories, namely, pamphlet, slide, report, and website considering
the form of the documents.

#### Text Extraction & Normalization.

We extracted texts from PDF documents with PyPDF2444We also examined PyMuPDF.
However, the quality of the extracted texts was not changed greatly.. We also
notice that some PDF documents are probably created from paper scans, and we
cannot extract embedded texts from such documents. Therefore, we extracted
texts from the document page images by OCR (Optical Character Recognition) as
an alternative source. After the text extraction or OCR, we removed mistakenly
recognized symbols and emojis, or duplicated characters from texts when the
same character continuously and repeatedly appeared more than five times.

Figure 3: Annotation process.

#### Annotation Procedure.

We ask 43 annotators in total for the question-answering pairs annotation on
documents. As documents include rich textual and visual elements (e.g.,
graphs, charts, maps, illustrations, and a mix of vertical and horizontal
written text), we made question answer pairs that are related to both textual
and visual information. We ask annotators to write up two to four question-
answer annotations in each document. We also ask not to use any AI-tools such
as OpenAI ChatGPT during the annotation process. Each question is accompanied
with the supporting facts as marked in red in Figure 1 and Figure 3. We
classify a subset of questions that have multiple supporting facts in multiple
pages as multi-page questions. Multi-page questions are considerably difficult
from their single-page counterparts. For unanswerable questions, we ask
annotators to write questions that lack supporting facts in the documents,
making them impossible to answer based on the given documents.

#### Visual inputs and bounding boxes.

We prepared three types of images for visual inputs for multimodal models. The
first type of images are those of the whole page of the documents including
the annotated question answering pairs. The second type of images are those
cropped by bounding boxes on which annotators based their answers such as
tables or figures of the pages. When multiple bounding boxes are annotated to
a single question-answer pair, multiple cropped images are combined together
into a single image here. The third type of images are blank (white) images
that are used for ablation studies.

## 4\. Experiments

### 4.1. Question-Answering Task

Our dataset aims to evaluate the question answering ability following the
document contexts, including textual and visual information, and questions via
open-ended text generation. As discussed in Sec. 3.1, our dataset consists of
four forms of questions: yes/no555The chance rate of yes/no questions
including unanswerable is 61.57 when the model always marks “yes.”, factoid,
numerical, and open-ended. All of these four category questions include
unanswerable questions that are not answerable solely from the given document
file. Models are expected to generate answers in open-ended text generation in
any of these question types. Textual model inputs, or simply prompts, consist
of the embedded texts or OCR results of documents as described in Sec. 3.3 and
the questions. We also include the answer-format guidelines such as “please
answer in Yes/No form”, “please answer the fact that is referred to in the
document”, “please answer by numerical information from the document” and
“please write the answer in open-ended format” into the prompts depending on
the four question categories. For unanswerable questions, we prepare a special
answer for all question categories: “本文中に記載がありません (not mentioned in the
text).”

### 4.2. Models

We conduct experiments with both text-input models and multimodal models of
text and vision inputs. For model training, we use supervised finetuning. The
best hyperparameters are searched with train and validation sets, then the
model performance is evaluated with the best hyperparameters.

#### Models with text input.

We adapted up to 13 billion (13B) model parameter scale Japanese large
language models for experiments. We experimented with the following
representative Japanese models that take only textual inputs: rinna japanese-
gpt2-medium666https://huggingface.co/rinna/japanese-gpt2-medium, japanese-
gpt-4B-8k777https://huggingface.co/rinna/bilingual-gpt-neox-4b-8k, rinna
japanese-gpt-1B888https://huggingface.co/rinna/japanese-gpt-1b, Cyberagent
OpenCALM-7B999https://huggingface.co/cyberagent/open-calm-7b, Matsuo-Lab
weblab-10b101010https://huggingface.co/matsuo-lab/weblab-10b, PFNet
PLaMo-13B111111https://huggingface.co/pfnet/plamo-13b, Stability AI Japanese-
StableLM-Base-Alpha-7B121212https://huggingface.co/stabilityai/japanese-
stablelm-base-alpha-7b, and Stability AI Japanese-StableLM-Instruct-
Alpha-7B131313https://huggingface.co/stabilityai/japanese-stablelm-instruct-
alpha-7b. We also include multilingual large language model of Llama-2-7B
141414https://huggingface.co/meta-llama/Llama-2-7b-hf. We trained and
evaluated models with 1024 token length for fair comparisons and computational
efficiency except rinna japanese-gpt-4B-8k which is trained with 8192 tokens
at most. For analyses with longer token lengths, we train rinna japanese-
gpt-4B-8k model with 2048, 4096, and 8192 tokens.

| Validation set | Test set
---|---|---
Model | Avg. | (1) Y/N | (2) Fact. | (3) Num. | (4) Open. | Avg. | (1) Y/N | (2) Fact. | (3) Num. | (4) Open.
Evaluated with all instances.
gpt-3.5-turbo-16k | 19.86 | 47.89 | 7.85 | 7.97 | 15.75 | 20.62 | 50.29 | 7.44 | 11.11 | 13.64
gpt-4 | 17.96 | 34.73 | 9.42 | 8.51 | 19.17 | 19.47 | 43.19 | 6.51 | 11.11 | 17.07
Evaluated without “unanswerable.”
gpt-3.5-turbo-16k | 22.72 | 57.23 | 8.82 | 9.20 | 15.63 | 23.07 | 58.21 | 8.08 | 12.5 | 13.49
gpt-4 | 20.90 | 41.50 | 10.58 | 9.81 | 21.72 | 22.03 | 50.00 | 7.07 | 12.5 | 18.57
Models trained with all training instances and evaluated all instances.
rinna gpt2-medium-336M | 21.33 | 63.15 | 7.32 | 4.78 | 17.51 | 19.41 | 62.13 | 4.65 | 8.18 | 15.99
rinna gpt-1B | 23.79 | 58.42 | 10.99 | 6.38 | 22.27 | 20.46 | 59.76 | 5.58 | 8.77 | 18.13
rinna bi-4B-8k (8192 tok.) | 26.35 | 55.26 | 14.65 | 13.29 | 24.93 | 23.02 | 62.13 | 8.83 | 11.11 | 20.57
OpenCALM-7B | 21.65 | 47.36 | 14.65 | 5.31 | 20.81 | 18.33 | 43.78 | 11.62 | 9.94 | 16.03
weblab-10B | 19.20 | 46.31 | 9.42 | 7.97 | 17.13 | 16.94 | 47.92 | 10.23 | 8.18 | 13.24
PLaMo-13B | 25.79 | 55.26 | 15.18 | 15.42 | 22.92 | 20.33 | 53.84 | 10.69 | 7.01 | 18.21
StableLM Base-Al.-7B | 32.92 | 67.89 | 21.98 | 18.61 | 29.62 | 29.71 | 70.41 | 15.81 | 22.22 | 25.51
StableLM Inst.-Al.-7B | 33.80 | 67.36 | 20.94 | 20.21 | 31.39 | 29.56 | 72.78 | 16.27 | 21.05 | 24.75
Llama2-7B | 30.29 | 60.00 | 20.41 | 15.42 | 28.59 | 27.01 | 61.53 | 17.20 | 18.71 | 23.29
Models trained with all training instances while evaluated without
“unanswerable.”
rinna gpt2-medium-336M | 22.22 | 69.81 | 4.70 | 3.68 | 18.71 | 19.61 | 65.75 | 4.54 | 6.57 | 16.31
rinna gpt-1B | 22.84 | 62.26 | 7.05 | 5.52 | 21.14 | 20.18 | 64.38 | 4.04 | 8.55 | 17.40
rinna bi-4B-8k (8192 tok.) | 24.09 | 52.83 | 12.94 | 10.42 | 23.10 | 21.17 | 62.32 | 6.56 | 7.89 | 19.11
OpenCALM-7B | 20.75 | 50.31 | 12.35 | 4.90 | 19.20 | 17.53 | 46.57 | 9.59 | 8.55 | 15.10
weblab-10B | 17.74 | 45.91 | 8.82 | 6.13 | 15.34 | 16.20 | 50.68 | 9.09 | 6.57 | 12.17
PLaMo-13B | 22.66 | 54.08 | 10.58 | 9.81 | 20.76 | 18.35 | 50.68 | 8.58 | 5.26 | 16.87
StableLM Base-Al.-7B | 31.55 | 70.44 | 18.82 | 15.95 | 28.24 | 28.33 | 71.91 | 12.62 | 21.05 | 24.32
StableLM Inst.-Al.-7B | 31.74 | 69.81 | 17.05 | 15.95 | 29.55 | 28.66 | 76.02 | 12.62 | 19.07 | 24.40
Llama2-7B | 28.78 | 62.26 | 17.05 | 14.72 | 26.44 | 25.70 | 65.06 | 12.62 | 18.42 | 21.87
Models trained without “unanswerable” while evaluated with all instances.
rinna gpt2-medium-336M | 15.91 | 52.10 | 3.14 | 3.72 | 12.11 | 17.81 | 63.31 | 4.18 | 5.26 | 13.60
rinna gpt-1B | 17.66 | 45.26 | 6.80 | 2.65 | 17.04 | 17.59 | 51.47 | 6.04 | 5.26 | 15.76
rinna bi-4B-8k (8192 tok.) | 23.44 | 57.89 | 12.04 | 9.57 | 20.33 | 23.01 | 69.23 | 7.90 | 9.94 | 19.27
OpenCALM-7B | 18.94 | 42.63 | 8.90 | 3.72 | 19.44 | 16.95 | 42.01 | 7.44 | 6.43 | 16.33
weblab-10B | 20.43 | 50.00 | 9.42 | 6.91 | 18.70 | 17.96 | 52.07 | 6.51 | 8.18 | 15.34
PLaMo-13B | 22.04 | 60.00 | 7.85 | 9.57 | 18.23 | 21.11 | 64.49 | 8.83 | 11.11 | 16.31
StableLM Base-Al.-7B | 27.02 | 63.68 | 14.65 | 12.76 | 23.61 | 25.68 | 68.63 | 12.09 | 17.54 | 20.94
StableLM Inst.-Al.-7B | 27.22 | 61.57 | 15.70 | 14.36 | 23.84 | 26.25 | 70.41 | 15.34 | 16.37 | 20.74
Llama2-7B | 30.15 | 63.68 | 18.32 | 13.82 | 28.30 | 28.25 | 73.96 | 11.62 | 16.95 | 24.68
Models trained without “unanswerable” and evaluated without “unanswerable”
instances.
rinna gpt2-medium-336M | 18.75 | 62.26 | 3.52 | 4.29 | 14.33 | 20.12 | 73.28 | 4.54 | 5.92 | 15.41
rinna gpt-1B | 21.15 | 54.08 | 7.64 | 3.06 | 21.17 | 20.02 | 59.58 | 6.56 | 5.92 | 18.20
rinna bi-4B-8k (8192 tok.) | 27.63 | 69.18 | 13.52 | 11.04 | 24.26 | 25.95 | 80.13 | 8.58 | 11.18 | 21.79
OpenCALM-7B | 20.55 | 50.94 | 10.00 | 4.29 | 19.67 | 17.97 | 48.63 | 8.08 | 7.23 | 16.32
weblab-10B | 22.25 | 59.74 | 10.58 | 7.97 | 18.55 | 19.06 | 60.27 | 7.07 | 9.21 | 15.06
PLaMo-13B | 26.48 | 71.69 | 8.82 | 11.04 | 22.74 | 24.29 | 74.65 | 9.59 | 12.50 | 19.34
StableLM Base-Al.-7B | 32.30 | 76.10 | 16.47 | 14.72 | 29.15 | 29.21 | 79.45 | 13.13 | 19.73 | 24.14
StableLM Inst.-Al.-7B | 32.41 | 73.58 | 17.64 | 16.56 | 29.16 | 29.75 | 81.50 | 16.66 | 18.42 | 23.69
Llama2-7B | 33.11 | 76.10 | 20.58 | 15.95 | 28.84 | 30.57 | 85.61 | 12.62 | 19.07 | 25.47
Table 4:  Results of all finetuned models and OpenAI GPT zeroshot. Avg. is
weighted average of scores.

#### Models with multimodal input.

The purpose of JDocQA is to analyze documents with textual and visual
perceptions. To assess the impact of using both images and text on the JDocQA
dataset, we applied multimodal models that take inputs from both images and
texts. We used Stability AI Japanese-StableLM-Instruct-
Alpha-7B151515https://huggingface.co/stabilityai/japanese-stablelm-instruct-
alpha-7b, a Japanese version of InstructBLIP Dai et al. (2023); Li et al.
(2023a, c) for this purpose, as they are applicable to Japanese text and image
inputs. We trained and evaluated this model with 512 token lengths following
its max capacity. We develop three different models for with three different
visual inputs as explained in Sec. 3.3. The first model takes visual inputs of
a blank image that is always the same white image of the 800x600 pixel size as
ablation study. The second model takes an image of a whole document page that
are related to the question-answering in the annotation. These images are also
scaled to 800-pixel width. The third model take inputs of the images following
the annotated supporting facts. Following the annotated bounding boxes, we
crop the referenced regions of the page images, combine the bounding boxes and
scale them for the model visual input. As some questions have several
annotated supporting facts to their answers, the combined image may contain
more than one region of the annotated bounding boxes. All of these multimodal
models also take textual prompts that are similar to the text-input models.

#### OpenAI GPT baselines.

We also present OpenAI GPT performances as baselines. Here we use
gpt-3.5-turbo-16k and gpt-4 models161616Latest model at Oct. 9 2023.. They
take similar prompts to those of text-input models. However, as they are the
zero-shot models for our task, we observed they are quite sensitive to the
prompts. To improve their performance, we manually tune the prompts for OpenAI
GPT models. We avoid finetuning OpenAI GPT models although finetuning them may
greatly improve performances due to the following reasons. First, our purpose
is to develop local models that work on limited computational resources,
Second, the details of finetuning are unavailable for these models, and
finally due to the API cost issues.

### 4.3. Evaluation Methods

For Yes/No, factoid, and numerical questions, we used the exact match metric
after trimming trivial differences such as the presence or absence of
punctuation marks or Japanese suffix phrases such as “です (is)” by simple
rules. We have also examined the variations of exact match, such as the ratio
of whether model prediction phrases are included in the correct answer phrases
or not. However, we realize that this metric performs quite similarly to the
exact match evaluation and the difference is typically less than 10 question-
answer pairs in the validation set. For open-ended questions, answers are
typically long, e.g., the average length of answers is 65.97 characters in
Japanese, and hence the exact match does not work for evaluation. Therefore,
we used BLEU score171717https://github.com/mjpost/sacrebleu tokenized by
MeCab181818https://pypi.org/project/mecab-python3/ for automatic evaluation of
open-ended questions.

### 4.4. Experimental Settings

Our dataset includes unanswerable questions in all question categories. While
it is expected that finetuning models with unanswerable instances may harness
models to surpass illusion answers known as hallucinations, detecting
unanswerable questions is also notoriously difficult as of Rajpurkar et al.
(2018). Therefore we prepare two types of models for all base models
experimented: models finetuned with all question answering pairs including
unanswerable instances and models finetuned without unanswerable instances. In
the evaluation, we similarly prepare two separate validation and test sets:
the standard validation and test sets that consists of all question-answering
pairs, and the smaller validation and test sets where unanswerable questions
are removed.

### 4.5. Results

Table 4 presents the performance of text-input models for all question types
on valid and test splits.

#### Models trained with all instances.

We compare the first and third blocks in Table 4. They are the results of all
JDocQA instances by zero-shot and finetuned models that are trained with all
JDocQA training instances including “unanswerable questions”. We realize that
fine-tuned models outperformed gpt-3.5 and gpt-4 results especially when the
model size is larger. Next, we compare the second and fourth blocks in Table
4. They are the same models with previous block evaluations on the JDocQA
without unanswerable questions. We observe a similar tendency to all instances
evaluations, suggesting that models finetuned with unanswerable instances
perform similar performances in both answerable and unanswerable questions.
Among them, StableLM models perform best despite their parameter size of 7B.
The rinna bi-4B-8k model also performs well despite its parameter size. We
attribute this to its token length size of 8192 and will discuss this later.
We notice that (1) yes/no questions are relatively easy although they do not
have much effect on the averaged score (Avg.) that mostly follows the most
common question category of (4) open-ended.

| Test set
---|---
Model | Avg. | (1) Y/N | (2) Fact. | (3) Num. | (4) Open.
Trained all and evaluated all.
InstBLIP (blank) | 26.92 | 65.68 | 16.27 | 19.88 | 22.00
InstBLIP (img) | 27.44 | 68.63 | 15.34 | 19.88 | 22.50
InstBLIP (bbox) | 27.87 | 72.78 | 18.13 | 19.29 | 21.37
Trained all while evaluated w/o “unanswerable.”
InstBLIP (blank) | 25.12 | 65.75 | 10.60 | 17.10 | 21.68
InstBLIP (img) | 25.74 | 69.17 | 11.61 | 15.13 | 22.12
InstBLIP (bbox) | 27.99 | 78.76 | 14.14 | 17.76 | 22.16
Trained w/o “unanswerable” while evaluated all.
InstBLIP (blank) | 23.13 | 66.27 | 12.55 | 11.69 | 18.21
InstBLIP (img) | 25.01 | 71.59 | 12.09 | 16.37 | 19.19
InstBLIP (bbox) | 29.00 | 78.10 | 14.88 | 19.29 | 23.19
Trained w/o “unanswerable” and evaluated w/o “unanswerable.”
InstBLIP (blank) | 26.45 | 76.71 | 13.63 | 13.15 | 21.26
InstBLIP (img) | 28.52 | 82.87 | 13.13 | 18.42 | 22.25
InstBLIP (bbox) | 27.79 | 80.13 | 11.61 | 16.44 | 22.71
Table 5:  Results of multimodal input models. Avg. is weighted average.  | Test set
---|---
Token length | Avg. | (1) Y/N | (2) Fact. | (3) Num. | (4) Open.
Trained all and evaluated all.
2048 tokens | 20.97 | 57.39 | 10.69 | 9.94 | 17.66
4096 tokens | 21.96 | 56.21 | 9.30 | 13.45 | 19.38
8192 tokens | 23.02 | 62.13 | 8.83 | 11.11 | 20.57
Trained w/o “unanswerable” and evaluated w/o “unanswerable.”
2048 tokens | 24.57 | 72.60 | 10.10 | 9.21 | 21.18
4096 tokens | 24.26 | 67.12 | 9.09 | 11.18 | 21.90
8192 tokens | 25.95 | 80.13 | 8.58 | 11.18 | 21.79
Table 6:  Results of rinna bi-4B-8k models with different token length. Avg. is weighted average.  | Test set
---|---
Model | Pamphlet | Slide | Report | Website
Trained all and evaluated all.
rinna gpt2-med-336M | 18.62 | 16.32 | 14.57 | 3.09
rinna gpt-1B | 16.66 | 15.10 | 15.08 | 4.72
rinna bi-4B-8k (8192) | 21.81 | 16.73 | 18.41 | 3.53
OpenCALM-7B | 15.19 | 15.10 | 13.04 | 2.56
weblab-10B | 14.70 | 17.55 | 13.04 | 2.69
PLaMo-13B | 21.56 | 13.06 | 15.85 | 2.90
Base-Al.-7B | 26.96 | 20.40 | 24.04 | 4.64
Inst-Al.-7B | 27.69 | 23.26 | 23.01 | 5.71
InstBLIP-Al (blank) | 25.49 | 22.04 | 21.48 | 3.79
InstBLIP-Al (img) | 25.00 | 21.63 | 23.78 | 3.75
InstBLIP-Al (bbox) | 25.00 | 22.04 | 22.50 | 4.20
Table 7:  Detailed file-type result. “Website” is included only in test set as
an out-of-domain set.

#### Models trained without unanswerable.

As explained in Sec. 4.4, we also prepared models finetuned without
unanswerable questions in the training instances. We present the evaluation
results including and excluding unanswerable questions in the fifth and sixth
blocks in Table 4. It is quite interesting when we compare the third and fifth
block results in Table 4 as they share the same evaluation set including
unanswerable questions while the models are finetuned with and without the
unanswerable instances. Comparing the third and fifth block results, we notice
almost all models finetuned with unanswerable questions perform better than
their answerable-only finetuned counterparts in the averaged scores.
Exceptions are the OpenCALM-7B, weblab-10B and Llama2-7B models, which we will
discuss in the next paragraph. We attribute this is due to the concept of
hallucination, where models generate answers that do not appear in context
texts. We will present an example of this in the qualitative analysis
paragraph. Doping unanswerable instances may contribute to harnessing
hallucination in this experimental comparison.

#### OpenCALM-7B and weblab-10B do not predict questions as unanswerable so
much.

In the third block of Table 4, we notice interesting phenomena: OpenCALM-7B
and weblab-10B do not perform well despite their parameter size. When we
closely check these models’ outputs, we realize these models, finetuned with
all instances, predict “本文中に記載がありません (not mentioned in the text)” much less
than other models. OpenCALM-7B and weblab-10B predicts 11.9% and 13.7% of all
instances are unanswerable while other models in the third block of Table 4
predict around or more than 20%. As they are trained with the same dataset, we
suspect this is due to their pretraining. It is also notable that predicting
questions as unanswerable is a difficult task for models and often affects the
overall performance.

Figure 4: Qualitative analysis of open-ended question answering.

#### Multimodal model results.

We present the results of models with multimodal inputs models of StableLM-
InstructBLIP-Alpha in Table 5. The model performances are enhanced especially
when we use cropped images of referenced tables or figures (bbox). We also
notice that the model with black image inputs performs close to visual models
to some extent, suggesting the effectiveness of textual inputs in our task. We
also carefully note that the max token length of StableLM-InstructBLIP-Alpha
is 512, which can limit the textual understanding abilities of current
multimodal models.

#### Token length dependency.

We survey the token length effects on the performance. For this purpose, we
finetuned three models of different token lengths of rinna bi-4B-8k, e.g.,
2048, 4096, and 8192 respectively for both all training instances and without
unanswerable conditions. We present the performances in Table 6. We notice the
finetuning token length surely affects the final result, although we also
notice finetuning models with long token length are much more computationally
costly than those of short token length models, which can be the reason for
the good performance of the rinna bi-4B-8k (8192 tokens) model in Table 4.

#### Detailed Analyses of document-types.

Table 7 presents the performance comparisons between each file type. File
types are classified into Japanese pamphlets such as public relations booklets
or magazines, slides such as presentation materials, and report documents
including figures and tables. We also prepare the out-of-domain test set of
website scans where models still perform worse.

#### Qualitative Analysis.

Figure 4 presents two example instances with the models’ generations. In the
top example, we present the question, annotated answer, and generation from
three models trained with questions including unanswerable: StableLM
InstructBLIP-Alpha (img), StableLM Instruct-Alpha-7B and StableLM Base-
Alpha-7B respectively. StableLM InstructBLIP-Alpha (img) can see the
alignments of the table via visual inputs and generate reasonable descriptions
for the car. StableLM Base-Alpha also generates a similar answer while it
cannot attribute the running mileage of 123,334 presented in the table. At the
bottom of Figure 4, we presented the comparison of two finetuned models that
originate from the same pretrained model and are trained with and without
unanswerable questions. The all instances model accurately predicts that there
are no answers written in the text while the model without unanswerable
questions falsely generates the open-ended answers, causing the phenomena
known as hallucination.

Model | Human Evaluation $\uparrow$
---|---
Trained all and evaluated all.
PLaMo-13B | 1.24
StableLM Instruct-Alpha-7B | 1.49
StableLM InstructBLIP-Alpha (blank) | 1.04
StableLM InstructBLIP-Alpha (img) | 1.25
Table 8:  Human evaluation on the sampled set.

#### Human evaluation.

Finally, we performed the human evaluation on the sampled results for some
representative models. We sampled 100 open-ended questions from the test set
for this purpose. We choose two text-input models of PLaMo-13B and StableLM
Instruct-Alpha-7B. We also choose multimodal input models of StableLM
InstructBLIP-Alpha with back and image inputs. We ask an annotator to attach
scores from 0 to 2 for two criteria: whether the generated answers include the
annotated answer and whether the generated answers do not include wrong
statements as questions’ answers. The results are presented in Table 8. We
notice that StableLM Instruct-Alpha-7B outperforms PLaMo-13B and image model
outperforms its blank image counterpart.

## 5\. Conclusion

We introduced the JDocQA dataset concentrating the integration of both visual
and linguistic cues in question answering in Japanese. We incorporated
unanswerable questions from given documents, which we confirmed is effective
for harnessing the hallucinated generation to some extent in our experiments.
Our detailed evaluations revealed the effectiveness of our dataset in a wide
range of question categories from yes/no to open-ended and that the prediction
of unanswerable questions can be a clue to improve model performances,
illustrating the effectiveness of the JDocQA dataset in realistic applications
where multiple categories of questions are feasible and some extent of
questions do not have explicit written answers in the documents.

## 6\. Acknowledgments

This work was supported by JSPS Grant-in-Aid for Young Scientists (#22K17983),
JSPS Fostering Joint International Research (A) (#22KK0184) and by JST PRESTO
(#JPMJPR20C2).

## Ethical Statement & Limitation

In data collection, we gather document PDF files and webpages from Japanese
National Diet Library (NDL)’s digital collection, web archive projects (WARP),
and websites of Japanese government ministries. Administrative PDFs,
documents, pamphlets, or websites published by local governments or
universities are gathered through WARP. We carefully avoid private documents
and choose considerably public documents published by public or quasi-public
sectors for the publicity of our dataset usage. All of the documents and
webpages are publicly available online and we follow our institutional rules
to gather them. We follow our institutional rules and also consult external
advisors for data collection processes.

We assume our datasets are useful for both research and development of
generative language models and their applications for Japanese document
question answering. We also consider our dataset with unanswerable questions
can contribute to harnessing the hallucination problem of large language
models. However, this doesn’t mean that the fintuned models with unanswerable
questions do not perform hallucinations at all.

## 7\. Bibliographical References

##

  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015.  VQA: Visual Question Answering.  In _International Conference on Computer Vision (ICCV)_.
  * Bansal et al. (2020) Ankan Bansal, Yuting Zhang, and Rama Chellappa. 2020.  Visual question answering on image sets.  In _ECCV 2020_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Chen et al. (2021) Xingyu Chen, Zihan Zhao, Lu Chen, JiaBao Ji, Danyang Zhang, Ao Luo, Yuxuan Xiong, and Kai Yu. 2021.  WebSRC: A dataset for web-based structural reading comprehension.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 4173–4185, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Dai et al. (2023) Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023.  Instructblip: Towards general-purpose vision-language models with instruction tuning.
  * Dua et al. (2019) Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019.  DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2368–2378, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Iyyer et al. (2016) Mohit Iyyer, Varun Manjunatha, Anupam Guha, Yogarshi Vyas, Jordan L. Boyd-Graber, Hal Daumé, and Larry S. Davis. 2016.  The amazing mysteries of the gutter: Drawing inferences between panels in comic book narratives.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6478–6487.
  * Kembhavi et al. (2017) Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. 2017.  Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Kurihara et al. (2022) Kentaro Kurihara, Daisuke Kawahara, and Tomohide Shibata. 2022.  JGLUE: Japanese general language understanding evaluation.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 2957–2966, Marseille, France. European Language Resources Association.
  * Li et al. (2023a) Dongxu Li, Junnan Li, and Steven CH Hoi. 2023a.  Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing.  _arXiv preprint arXiv:2305.14720_.
  * Li et al. (2023b) Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Silvio Savarese, and Steven CH Hoi. 2023b.  Lavis: A one-stop library for language-vision intelligence.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)_ , pages 31–41.
  * Li et al. (2023c) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023c.  BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models.  In _ICML_.
  * Malinowski and Fritz (2014) Mateusz Malinowski and Mario Fritz. 2014.  A multi-world approach to question answering about real-world scenes based on uncertain input.  In _Advances in Neural Information Processing Systems_ , volume 27. Curran Associates, Inc.
  * Mathew et al. (2022) Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V. Jawahar. 2022.  Infographicvqa.  In _2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , pages 2582–2591.
  * Mathew et al. (2021a) Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. 2021a.  Docvqa: A dataset for vqa on document images.  In _Proceedings - 2021 IEEE Winter Conference on Applications of Computer Vision, WACV 2021_ , pages 2199–2208.
  * Mathew et al. (2021b) Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. 2021b.  _DocVQA: A dataset for VQA on document images_, Proceedings - 2021 IEEE Winter Conference on Applications of Computer Vision, WACV 2021, pages 2199–2208. Institute of Electrical and Electronics Engineers Inc.  Funding Information: We thank Amazon for supporting the annotation effort, and Dr. R. Manmatha for many useful discussions and inputs. This work is partly supported by MeitY, Government of India, the project TIN2017-89779-P, an Amazon AWS Research Award and the CERCA Programme. Publisher Copyright: © 2021 IEEE.
  * Mishra et al. (2019) Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. 2019\.  Ocr-vqa: Visual question answering by reading text in images.  In _ICDAR_.
  * Mishra et al. (2022) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022.  Cross-task generalization via natural language crowdsourcing instructions.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3470–3487, Dublin, Ireland. Association for Computational Linguistics.
  * Miyao and Kawazoe (2013) Yusuke Miyao and Ai Kawazoe. 2013.  University entrance examinations as a benchmark resource for NLP-based problem solving.  In _Proceedings of the Sixth International Joint Conference on Natural Language Processing_ , pages 1357–1365, Nagoya, Japan. Asian Federation of Natural Language Processing.
  * Miyazaki and Shimizu (2016) Takashi Miyazaki and Nobuyuki Shimizu. 2016.  Cross-lingual image caption generation.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1780–1790, Berlin, Germany. Association for Computational Linguistics.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 technical report.  Technical report.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 27730–27744. Curran Associates, Inc.
  * Powalski et al. (2021) Rafał Powalski, Łukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michał Pietruszka, and Gabriela Pałka. 2021.  Going full-tilt boogie on document understanding with text-image-layout transformer.  In _Document Analysis and Recognition–ICDAR 2021: 16th International Conference, Lausanne, Switzerland, September 5–10, 2021, Proceedings, Part II 16_ , pages 732–747. Springer.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for SQuAD.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 784–789, Melbourne, Australia. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022.  Multitask prompted training enables zero-shot task generalization.  In _International Conference on Learning Representations_.
  * Singh et al. (2019) Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019.  Towards vqa models that can read.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Smock et al. (2022) Brandon Smock, Rohith Pesala, and Robin Abraham. 2022.  Pubtables-1m: Towards comprehensive table extraction from unstructured documents.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4634–4642.
  * Takahashi et al. (2019) Norio Takahashi, Tomohide Shibata, Daisuke Kawahara, and Sadao Kurohashi. 2019.  Machine comprehension improves domain-specific Japanese predicate-argument structure analysis.  In _Proceedings of the 2nd Workshop on Machine Reading for Question Answering_ , pages 98–104, Hong Kong, China. Association for Computational Linguistics.
  * Talmor et al. (2021) Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. 2021.  Multimodalqa: complex question answering over text, tables and images.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Tanaka et al. (2023) Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. 2023.  Slidevqa: A dataset for document visual question answering on multiple images.  In _AAAI_.
  * Tanaka et al. (2021) Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. 2021.  Visualmrc: Machine reading comprehension on document images.  In _AAAI_.
  * Tang et al. (2023) Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Chao-Yue Zhang, and Mohit Bansal. 2023.  Unifying vision, text, and layout for universal document processing.  _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 19254–19264.
  * Tapaswi et al. (2016) Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2016.  MovieQA: Understanding Stories in Movies through Question-Answering.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Tito et al. (2021) Rubèn Pérez Tito, Dimosthenis Karatzas, and Ernest Valveny. 2021.  Document collection visual question answering.  In _IEEE International Conference on Document Analysis and Recognition_.
  * Tito et al. (2022) Rubèn Pérez Tito, Dimosthenis Karatzas, and Ernest Valveny. 2022.  Hierarchical multimodal transformers for multi-page docvqa.  _ArXiv_ , abs/2212.05935.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2022.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_.
  * Xu et al. (2021) Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. 2021.  LayoutLMv2: Multi-modal pre-training for visually-rich document understanding.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2579–2591, Online. Association for Computational Linguistics.
  * Xu et al. (2020) Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020.  Layoutlm: Pre-training of text and layout for document image understanding.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, KDD ’20, page 1192–1200, New York, NY, USA. Association for Computing Machinery.
  * Yagcioglu et al. (2018) Semih Yagcioglu, Aykut Erdem, Erkut Erdem, and Nazli Ikizler-Cinbis. 2018.  RecipeQA: A challenge dataset for multimodal comprehension of cooking recipes.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1358–1368, Brussels, Belgium. Association for Computational Linguistics.
  * Yanaka and Mineshima (2022) Hitomi Yanaka and Koji Mineshima. 2022.  Compositional evaluation on Japanese textual entailment and similarity.  _Transactions of the Association for Computational Linguistics_ , 10:1266–1284.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.


# Intrinsic Contribution to Non-linear Thermoelectric Effect

Pankaj Bhalla  Beijing Computational Science Research Center, Beijing, 100193,
China

###### Abstract

Irradiation of the strong light on the material leads to numerous non-linear
effects that are essential to understand the physics of excited states of the
system and for optoelectronics. Here, we study the non-linear thermoelectric
effect due to the electric and thermal fields applied on a non-centrosymmetric
system. The phenomenon arises on the Fermi surface with the transitions of
electrons from valence to conduction bands. We derive the formlism to
investigate these effects and find that the non-linearity in these effects
namely non-linear Seebeck and non-linear Peltier effects depends on the ratio
of the non-linear to the linear conductivities. The theory is tested for a
hexagonally warped and gapped topological insulator. Results show enhancement
in the longitudinal and Hall effects on increasing the warping strength while
show opposite behavior with the surface gap.

##  I Introduction

A road in pursuit of highly efficient thermoelectric materials is challenging
and fundamentally important as directly relates with the global energy crisis
issue. This makes indespensable for the researchers to puzzle out the
thermoelectric transport properties Alam and Ramakrishna (2013). These
properties decide the thermal to electrical energy conversion efficiency,
governed by the dimensionless quantity named as figure of merit $zT=\sigma
S^{2}T/\kappa$, Rowe (2006); Behnia (2015) where $T$ is an absolute
temperature, $\sigma$ is an electrical conductivity, $S$ is a Seebeck
coefficient and $\kappa=\kappa_{e}+\kappa_{\text{lattice}}$ is the sum of
electrical ($\kappa_{e}$) and lattice ($\kappa_{\text{lattice}}$) thermal
conductivities. Inovative approaches and concepts have been employed for
decades to enhance the performance of such materials and hierarchical
architecturing.Koumoto and Mori (2013); Dresselhaus  _et al._ (2007); Poudel
_et al._ (2008); Sootsman  _et al._ (2009); Tretiakov  _et al._ (2011); Pei
_et al._ (2011); Biswas  _et al._ (2012); Osterhage  _et al._ (2014); Li and
Xu (2015); Xu (2016); Xu  _et al._ (2017); Baldomir and Failde (2019)
Approaches deal with the responses to the first power of external field
provides great successGayner and Kar (2016), if the amplitude of the incident
light is small. However if the amplitude becomes sufficiently large to distort
electron’s orbits, it becomes crucial to look after the responses to higher
order fields to generate new optoelectronic devices and new quantum
technologies. In the following years, a lot of work has been done to study the
non-linear electrical effects such as shift current, second-harmonic
generation, injection current and anomalous current, resonant photovoltaic
current von Baltz and Kraut (1981); Khurgin (1994, 1995); Nastos and Sipe
(2010); Young and Rappe (2012); Inglot  _et al._ (2015); Li  _et al._ (2016a);
Wang  _et al._ (2017); Ogawa  _et al._ (2017); Kim  _et al._ (2017); Wang  _et
al._ (2016); Tan and Rappe (2016); Morimoto and Nagaosa (2016a); Khurgin
(2016); Morimoto and Nagaosa (2016b); Morimoto  _et al._ (2016); Morimoto and
Nagaosa (2016c); Hamamoto  _et al._ (2017); Cook  _et al._ (2017); Fregoso
_et al._ (2017); Zhang  _et al._ (2018); Krasnok  _et al._ (2018); Tan and
Rappe (2019); Zhang and Vignale (2018); Fregoso (2019); Liu  _et al._ (2019);
Bhalla  _et al._ (2020); Sekine and Nagaosa (2020); Ahn and Nagaosa (2020);
Wang and Qian (2020). But only few work has been devoted to the thermoelectric
effects López and Sánchez (2013); Hwang  _et al._ (2014); Gao and Xiao (2018).
To fill the gap and for better understanding about the non-linear behavior of
thermoelectric properties, we provide a theory for non-linear thermoelectric
conductivities and the related phenomenon such as Seebeck effect, Peltier
effect in the weakly non-linear regime. This has been examined for hexagonally
warped and gapped topological insulators Fu (2009).

Figure 1: Schematic view for the generation of thermoelectric effects due to
interband and intraband electronic transitions shown in the middle picture
with green dashed lines. Left diagram refers to the Seebeck effect and Right
to the Peltier effect and middle corresponds to the energy spectrum around
Dirac point having distorted Fermi surface.

Basically, the transport properties are governed by two different compartments
with one belongs to the applied field or known as field driving term and other
to scattering processes. Former part plays a significant role in the
dissipationless transport phenomenon and relates with the momentum dependent
Bloch wave functions. More specifically, the blend of interband contribution,
characterized by Berry connection that relates two neighboring momentum
dependent wave functions and intraband contribution due to the excitation of
carriers within the same band in response to the external field leads to give
intrinsic effects Culcer  _et al._ (2017). In systems having time-reversal
symmetry or inversion symmetery breaking, well known examples such as
anomalous Hall effect Nagaosa  _et al._ (2010), chiral magnetic effect Zyuzin
and Burkov (2012); Monteiro  _et al._ (2015); Jia  _et al._ (2016); Kaushik
_et al._ (2019) and negative magnetoresistance in 3D Dirac and Weyl semimetals
Li  _et al._ (2016b); Schumann  _et al._ (2017); Armitage  _et al._ (2018);
Culcer  _et al._ (2020) has been observed due to such consequences. The latter
effects due to the interactions between carriers like electron-impurity,
charged impurities, phonons, etc. are known as extrinsic effects. Here we are
mainly focus on the contribution made by intrinsic effects to thermoelectric
properties. We find that for non-centrosymmetric systems, the deviation of
thermoelectric effects from linear regime depends on the ratio of the non-
linear to the linear responses.
To exemplify our theory, we consider time-reversal invariant topological
insulator $Bi_{2}Te_{3}$ having nontrivial topological orderFu (2009); Moore
(2010); Hasan and Moore (2011); Chen  _et al._ (2011); Pollmann  _et al._
(2012). We observe that the responses relies on the Bloch band dispersion near
the Fermi surface of this material, hence get attention by thermoelectric
community Xu  _et al._ (2017); Matsushita  _et al._ (F017); Ivanov  _et al._
(2018); Wang and Jeng (2019). Detailed investigation yields vanishing first-
order Hall and second-order longitudinal response due to intraband
transitions. However, it produces non-zero first-order longitudinal
conductivity and second-order Hall conductivity. Due to the transitions
between valence and conduction bands, the first-order Hall and second-order
longitudinal response gives finite value. Here the responses depend on the
degree of distortion and the strength of gap value.
This paper is organized as follows. In Sec. I, we provide the detailed
theoretical framework to compute the thermoelectric effect. First, we give the
quantum kinetic theory and the basic structure for the generalized currents.
Later we express the non-linear Seebeck and Peltier effects in terms of the
ratio of conductivities of different order to external fields. In next
section, we apply our theory to the modelled Hamiltonian for Bi2Te3 and
demonstrate the results for electrical and thermal currents in response to the
applied fields. Finally, we summarize our study with future implications.

##  II Theoretical Framework

We embark our calculation with the quantum Liouville equation for the density
matrix Culcer  _et al._ (2017)

$\begin{array}[]{rl}&\displaystyle\frac{d\rho}{dt}+\frac{i}{\hbar}[H,\rho]=0,\end{array}$
(1)

with $H$ the total Hamiltonian, a sum of unperturbed and perturbed parts. The
unperturbed part is the band Hamiltonian $H_{0}$ and $H_{\text{pert}}=H_{F}+U$
having $H_{F}$ as an interaction with external fields such as electric,
thermal, and magnetic and $U$ as the interaction potential. Substituting this
into Eq. (1) and expanding the density matrix perturbatively into the powers
of field such as $\rho=\rho^{(0)}+\rho^{(1)}+\cdots$, where superscripts
indicate the order in the field, Eq. (1) becomes

$\begin{array}[]{rl}&\displaystyle\frac{d\rho^{(n)}}{dt}+\frac{i}{\hbar}[H_{0},\rho^{(n)}]+J(\rho^{(n)})=\mathcal{D}_{F}(\rho^{(n-1)}).\end{array}$
(2)

Here $\mathcal{D}_{F}=-i/\hbar[H_{F},\rho^{(n-1)}]$ is the field driving term
and $J(\rho^{(n)})$ is the scattering term which in the Born approximation
takes the form

$\begin{array}[]{rl}J(\rho^{(n)})&\displaystyle=\frac{1}{\hbar^{2}}\,\int^{\infty}_{0}dt^{\prime}\,\langle[U,[e^{-\frac{iH_{0}t^{\prime}}{\hbar}}Ue^{\frac{iH_{0}t^{\prime}}{\hbar}},\rho^{n}]]\rangle.\end{array}$
(3)

To solve the kinetic equation, we segregate the density matrix into diagonal,
$f_{d,{\bm{k}}}$ and off-diagonal, $f_{\text{od},{\bm{k}}}$ components using
$\rho^{(n)}=f_{d,{\bm{k}}}^{(n)}+f_{\text{od},{\bm{k}}}^{(n)}$ and express the
Liouville equation (Eq. 2) into two coupled equations. Here the diagonal part
accounts to the intraband and off-diagonal part to the interband coherence
effects. This coupling part stems from the scattering term $J$, a sum of four
parts such as
$J=J_{d}(f_{d,{\bm{k}}}^{(n)})+J_{d}(f_{\text{od},{\bm{k}}}^{(n)})+J_{\text{od}}(f_{d,{\bm{k}}}^{(n)})+J_{\text{od}}(f_{\text{od},{\bm{k}}}^{(n)})$.

$\begin{array}[]{rl}&\displaystyle\frac{df_{d,{\bm{k}}}^{(n)}}{dt}+\frac{i}{\hbar}[H_{0},f_{d,{\bm{k}}}^{(n)}]+J_{d}[f_{d,{\bm{k}}}^{(n)}]=\mathcal{D}_{F}(f_{d,{\bm{k}}}^{(n-1)})-J_{d}[f_{\text{od},{\bm{k}}}^{(n)}],\end{array}$
(4)

and

$\begin{array}[]{rl}&\displaystyle\frac{df_{\text{od},{\bm{k}}}^{(n)}}{dt}+\frac{i}{\hbar}[H_{0},f_{\text{od},{\bm{k}}}^{(n)}]+J_{\text{od}}[f_{\text{od},{\bm{k}}}^{(n)}]=\mathcal{D}_{F}(f_{\text{od},{\bm{k}}}^{(n-1)})-J_{\text{od}}[f_{\text{d},{\bm{k}}}^{(n)}].\end{array}$
(5)

The transport calculations has been performed in response to the temperature
gradient within the relaxation time approximationBhalla  _et al._ (2020).
Under this assumption, we consider
$J_{d}(f_{\text{d}},{\bm{k}}^{(n)})=f_{\text{d}},{\bm{k}}^{(n)}/\tau_{d}$, and
$J_{\text{od}}(f_{\text{od}},{\bm{k}}^{(n)})=f_{\text{od}},{\bm{k}}^{(n)}/\tau_{\text{od}}$
with $\tau$ a relaxation time. For convenience, we take $\tau_{\text{d}}$ and
$\tau_{\text{od}}$ as same parameter and treat as a constant across the Fermi
surface. With this, the solutions for diagonal and off-diagonal parts of
density matrix become

$\begin{array}[]{rl}f_{\text{d},{\bm{k}}}^{(n)}&\displaystyle=\frac{\tau}{\hbar}\frac{{\bm{\nabla}}T}{T}\cdot\frac{\partial
f_{\bm{k}}^{(n-1)}}{\partial{\bm{k}}}(\varepsilon_{\bm{k}}^{m}-\mu)-\tau
J_{d}[f_{\text{od},{\bm{k}}}^{(n)}],\\\\[8.61108pt]
f_{\text{od},{\bm{k}}}^{(n),mm^{\prime}}&\displaystyle=-i\hbar\frac{[\mathcal{D}(\rho^{(n-1)}_{\bm{k}})]^{mm^{\prime}}-[J(f_{d,{\bm{k}}}^{(n)})]^{mm^{\prime}}}{\varepsilon_{\bm{k}}^{m}-\varepsilon_{k}^{m^{\prime}}}.\end{array}$
(6)

Here $\varepsilon_{\bm{k}}$ a band dispersion, $\mu$ a chemical potential, and
the thermal driving term
$\mathcal{D}(\rho^{(n-1)}_{\bm{k}})=\frac{1}{\hbar}\frac{{\bm{\nabla}}T}{T}\cdot\frac{D(\\{H_{0},\rho^{(n-1)}\\}}{D{\bm{k}}}$
with $\\{,\\}$ an anticommutator and $\frac{DA}{D{\bm{k}}}=\frac{\partial
A}{\partial{\bm{k}}}-i[\mathcal{R}_{{\bm{k}}},A]$ a covariant derivative with
respect to the wave vector and
$\mathcal{\bm{R}}_{\bm{k}}^{mm^{\prime}}=\langle
u_{\bm{k}}^{m}|i\partial_{\bm{k}}u_{\bm{k}}^{m^{\prime}}\rangle$ a Berry
connection with $|u_{\bm{k}}^{m}\rangle$ a momentum dependent quantum Bloch
wave function having band index $m$, and $A$ is an arbitrary matrix. Due to
the absence of a commutator, there is no covariant derivative for the diagonal
part. The detailed derivation of Eq. (6) is mentioned in Appendix.

###  II.1 Generalized currents

In the presence of field, the current can be expanded in the powers of the
applied external field. Schematically, it can be expressed like

$\begin{array}[]{rl}j_{a}&\displaystyle=\sum_{b}A_{ab}F_{b}+\sum_{b,c}A_{abc}F_{b}F_{c}+\sum_{b,c,d}A_{abcd}F_{b}F_{c}F_{d}+\cdots,\end{array}$
(7)

where $F$ is an external field in an arbitrary direction indicated by
subscript, $A_{ab}$ and $A_{abc}$ are second- and third-rank tensors
respectively. The first term is the well known linear response, and the higher
order terms are non-linear responses. The even-order terms only contribute to
noncentrosymmetric systems having broken inversion symmetry such as
ferroelectrics Sturman and Fridkin (1992). In other systems, these terms
vanish. In this work, we will consider systems having breaking inversion
symmetry to calculate the conductivity tensor in response to the applied
field. Here we assume small external field, thus consider response upto the
second-order case.
From the basic definition, the electric and thermal currents are Ziman (1960)

$\begin{array}[]{rl}{\bm{J}}^{E}&\displaystyle=\frac{-e}{\hbar}\sum_{\bm{k}}{\bm{v}}_{\bm{k}}\rho_{{\bm{k}}}\\\\[8.61108pt]
{\bm{J}}^{Q}&\displaystyle=\frac{1}{2\hbar}\sum_{\bm{k}}(\varepsilon_{\bm{k}}-\mu){\bm{v}}_{\bm{k}}\rho_{{\bm{k}}}.\end{array}$
(8)

Here ${\bm{v}}_{\bm{k}}$ is a $2\times 2$ matrix and represented as

$\begin{array}[]{rl}{\bm{v}}_{\bm{k}}&\displaystyle=\left(\begin{array}[]{cc}\partial_{\bm{k}}\varepsilon_{\bm{k}}^{m}&-i\mathcal{R}_{\bm{k}}^{mm^{\prime}}(\varepsilon_{\bm{k}}^{m}-\varepsilon_{\bm{k}}^{m^{\prime}})\\\
-i\mathcal{R}_{\bm{k}}^{m^{\prime}m}(\varepsilon_{\bm{k}}^{m}-\varepsilon_{\bm{k}}^{m^{\prime}})&\partial_{\bm{k}}\varepsilon_{\bm{k}}^{m^{\prime}}\end{array}\right).\end{array}$
(9)

Using the above matrix for velocity and the density matrix, the thermal and
electrical currents are lead by three terms, mentioned below

$\begin{array}[]{rl}j_{1}&\displaystyle=\partial_{\bm{k}}\varepsilon_{\bm{k}}^{m}f_{d,{\bm{k}}}^{m},\\\\[8.61108pt]
j_{2}&\displaystyle=-Re[\mathcal{R}_{\bm{k}}^{mm^{\prime}}](\varepsilon_{\bm{k}}^{m}-\varepsilon_{\bm{k}}^{m^{\prime}})Im[f_{\text{od},{\bm{k}}}^{mm^{\prime}}],\\\\[8.61108pt]
j_{3}&\displaystyle=Im[\mathcal{R}_{\bm{k}}^{mm^{\prime}}](\varepsilon_{\bm{k}}^{m}-\varepsilon_{\bm{k}}^{m^{\prime}})Re[f_{\text{od},{\bm{k}}}^{mm^{\prime}}].\end{array}$
(10)

Last two terms survive only for $m\neq m^{\prime}$, thus describe the
interband coherence effects to responses. In the next section, we calculate
the nonlinear thermoelectric effect using these definitions.

###  II.2 Nonlinear Thermoelectric effects

The conversion of electric voltage to temperature differences and vice-versa
lead to thermoelectric effects such as Seebeck effect, Peltier effect and
Thomson effect Behnia (2015); Goldsmid (2017). The phenomenon to generate the
potential difference across the material due to the diffusion of charge
carriers along the temperature gradient is known as Seebeck coefficient,
denoted by $S$.

$\begin{array}[]{rl}S&\displaystyle=\frac{J_{Q}^{E}}{J_{E}^{E}},\end{array}$
(11)

where $J_{Q}^{E}$ is the thermal current induced by the electric field and
$J_{E}^{E}$ is the electric current.
The reverse phenomenon of the former effect means the generation of flow of
heat due to the applied electric potential is known as Peltier effect, denoted
by $\Pi$.

$\begin{array}[]{rl}\Pi&\displaystyle=\frac{J_{Q}^{Q}}{J_{E}^{E}},\end{array}$
(12)

having $J_{Q}^{Q}$ as the thermal current generated by thermal field.
Moreover, the above expressions are interconnected by Kelvin relation
$\Pi=TS$. However, these expressions are established within the linear
response regime. We generalize these expressions to higher orders in field.
Using the defintion of generalized current (Eq. 7) the above relations can be
written like

$\begin{array}[]{rl}S&\displaystyle=\frac{\alpha_{1}+\alpha_{2}{\bm{\nabla}}T+\cdots}{\sigma_{1}+\sigma_{2}{\bm{E}}+\cdots},\end{array}$
(13)

with $\alpha_{i}$, and $\sigma_{i}$ are $i^{\text{th}}$-order thermoelectric
and electrical conductivity tensors, ${\bm{E}}$ and ${\bm{\nabla}}T$ are
electric field and temperature gradient respectively.
In the weakly nonlinear regime, we expand the denominator and keep the terms
to the first power of field. Thus, we have

$\begin{array}[]{rl}&\displaystyle
S=\frac{\alpha_{1}}{\sigma_{1}}+\frac{\alpha_{2}}{\sigma_{1}}\nabla
T-\frac{\alpha_{1}\sigma_{2}}{\sigma_{1}^{2}}E\cdots\end{array}$ (14)

On substituting $S_{0}=\alpha_{1}/\sigma_{1}$ corresponding to the linear
case, the effect can be expressed like

$\begin{array}[]{rl}&\displaystyle
S=S_{0}\bigg{(}1+\frac{\alpha_{2}}{\alpha_{1}}\nabla
T-\frac{\sigma_{2}}{\sigma_{1}}E\cdots\bigg{)}.\end{array}$ (15)

Similarly, the Pelteir effect in non-linear regime is

$\begin{array}[]{rl}&\displaystyle\Pi=\Pi_{0}\bigg{(}1+\frac{\kappa_{2}}{\kappa_{1}}\nabla
T-\frac{\sigma_{2}}{\sigma_{1}}E\cdots\bigg{)},\end{array}$ (16)

having $\kappa_{i}$ an $i^{\text{th}}$-order thermal conductivity tensor and
$\Pi_{0}=\kappa_{1}/\sigma_{1}$.
From above expressions, it is evident that the ratio of the conductivities
such as the non-linear to the linear decides the conversion efficiency of
currents.
In the following subsections, we test our theoretical approach with the model
Hamiltonian and calculate the electric and thermal currents due to electric
and thermal gradient fields.

####  II.2.1 Model Hamiltonian

We consider a band Hamiltonian describing the surface states of the
topological insulatorsFu (2009); Misawa  _et al._ (2011); Xiao and Wen (2013);
Chang  _et al._ (2015); Akzyanov and Rakhmanov (2018).

$\begin{array}[]{rl}H_{0}&\displaystyle=v_{F}(k_{x}\sigma_{y}-k_{y}\sigma_{x})+\Delta\sigma_{z}+\frac{\lambda}{2}(k_{+}^{3}+k_{-}^{3})\sigma_{z},\end{array}$
(17)

where $v_{F}$ is the Fermi velocity, $\Delta$ is the surface gap, $\lambda$ is
the hexagonal warping coefficient, $\sigma_{i}$ are Pauli matrices,
$k_{\pm}=k_{x}\pm ik_{y}$ with $k_{x}$ ($k_{y}$) as x-component (y-component)
momentum of the quasiparticle. The energy spectrum associated with the
Hamiltonian is

$\begin{array}[]{rl}\varepsilon_{{\bm{k}},\pm}&\displaystyle=\pm\sqrt{v_{F}^{2}k^{2}+(\lambda
k^{3}\cos 3\theta+\Delta)^{2}},\end{array}$ (18)

having $\theta$ a polar angle to define the momentum direction in the two
dimensional surface state and $\pm$ represents band index.

The constant energy contour for band dispersion is shown in Fig. (2). The
rashba interaction term referring the first term of Eq. LABEL:eqn:bandham
shows the regular circular shape Fermi surface. This converts into the
hexagonal distorted or snowflake shape with the inclusion of warping effect or
the term proportional to the third power of the wave-vector that reduces the
infinite mirror planes to three. This scenario further show significant effect
with time-reversal symmetry breaking finite surface gap.

Figure 2: Constant energy contours to describe the surface structure of a
topological insulator. Top left: with Rashba spin-orbit effect, Top right:
Rashba and Warping effect, Bottom right: Rashba, Warping and gap effect,
Bottom left: the solid black curves are for the surface gap $\Delta=0$ eV and
the dashed green curves for $\Delta=250$ meV. Here we set $v_{F}=2.55$ eVÅand
$\lambda=150$ eVÅ3.

Corresponding to the warped and gapped Hamiltonian, the energy eigenstates are

$\begin{array}[]{rl}|u^{\pm}_{\bm{k}}\rangle&\displaystyle=\frac{1}{\sqrt{2}}\left(\begin{array}[]{c}\pm
ie^{-i\theta}\sqrt{1\pm\frac{\lambda k^{3}\cos
3\theta+\Delta}{\varepsilon_{{\bm{k}}}}}\\\ \sqrt{1\mp\frac{\lambda k^{3}\cos
3\theta+\Delta}{\varepsilon_{{\bm{k}}}}}\end{array}\right).\end{array}$ (19)

####  II.2.2 Induced electric current by thermal field

First, we will calculate the electric current generated by thermal field
${\bm{\nabla}}T$ for first-order correction to density matrices i.e. for $n=1$
in Eq. (6). We consider the gradient field in x-direction
${\bm{\nabla}}T=\nabla_{x}T\hat{x}$ and compute the generated electrical
current in $\hat{x}$ amd $\hat{y}$ directions using Eq. (8). Based on Eq. (6),
two driving terms such as field-dependent term and scattering term would give
rise to the current. Here we mainly focus on the current production due to the
field-dependent driving term that do not include interaction effects. Keeping
this in mind, we have

$\begin{array}[]{rl}J_{x,Q}^{(1),E}&\displaystyle=-\frac{e}{\hbar}\frac{\nabla_{x}T}{T}\sum_{\bm{k}}\bigg{\\{}\frac{\tau}{\hbar}(\varepsilon_{\bm{k}}-\mu)\bigg{(}\frac{\partial\varepsilon_{\bm{k}}^{+}}{\partial
k_{x}}\bigg{)}^{2}\frac{\partial
f_{\bm{k}}^{0}}{\partial\varepsilon_{\bm{k}}}\\\\[12.91663pt]
&\displaystyle+Re\bigg{(}i(\mathcal{R}_{k_{x}}^{+-})^{2}(\varepsilon_{\bm{k}}^{+}f_{\bm{k}}^{0,+}-\varepsilon_{\bm{k}}^{-}f_{\bm{k}}^{(0),-})\bigg{)}\bigg{\\}},\end{array}$
(20)
$\begin{array}[]{rl}J_{y,Q}^{(1),E}&\displaystyle=-\frac{e}{\hbar}\frac{\nabla_{x}T}{T}\sum_{\bm{k}}\bigg{\\{}\frac{\tau}{\hbar}(\varepsilon_{\bm{k}}-\mu)\frac{\partial\varepsilon_{\bm{k}}^{+}}{\partial
k_{x}}\frac{\partial\varepsilon_{\bm{k}}^{+}}{\partial k_{y}}\frac{\partial
f_{\bm{k}}^{0}}{\partial\varepsilon_{\bm{k}}}\\\\[8.61108pt]
&\displaystyle+Re\bigg{(}i\mathcal{R}_{k_{y}}^{+-}\mathcal{R}_{k_{x}}^{+-}(\varepsilon_{\bm{k}}^{+}f_{\bm{k}}^{0,+}-\varepsilon_{\bm{k}}^{-}f_{\bm{k}}^{(0),-})\bigg{)}\bigg{\\}}.\end{array}$
(21)

The first part proportional to the first power of transport time $\tau$
generates the intraband contribution and the other part gives the interband
contribution due to the involvement of the Berry connection which relates two
neighboring bands. Considering the modeled Hamiltonian discussed earlier and
assuming the linear wave-vector dependent term dominant over other terms in
the Hamiltonian, we find that for the longitudinal direction the interband
contribution vanishes and only intraband contribution produces finite value to
the induced electrical current.

$\begin{array}[]{rl}J_{x,Q}^{(1),E}&\displaystyle=\frac{e\tau}{4\pi\hbar^{2}}\bigg{\\{}\mu\log
2+T\frac{\pi^{2}}{6}\bigg{\\}}\nabla_{x}T.\end{array}$ (22)

In Hall direction, the interband part yields finite $J_{y,Q}^{(1),E}$ while
intraband gives zero result.

$\begin{array}[]{rl}J_{y,Q}^{(1),E}&\displaystyle=\frac{e}{8\pi\hbar}\frac{\Delta\log
2}{\mu}\nabla_{x}T.\end{array}$ (23)

This depicts the dependence of the Hall component of induced electric current
linearly on the surface gap and inversely to the chemical potential while the
longitudinal component comes directly proportional to $\mu$. However, the
hexagonal warping does not generate any effect on both the components.
For second-order $n=2$ case, the longitudinal component is

$\begin{array}[]{rl}J_{x,Q}^{(2),E}&\displaystyle=\frac{e}{32\pi\hbar}\frac{\Delta\lambda}{v_{F}^{2}T}(\nabla_{x}T)^{2}\bigg{(}2\log
2+\frac{\pi^{2}T}{12\mu}\bigg{)}.\end{array}$ (24)

Similarly the Hall component is

$\begin{array}[]{rl}J_{y,Q}^{(2),E}&\displaystyle=\frac{e}{32\pi\hbar}\frac{\tau\lambda}{\hbar
v_{F}^{2}T}(\nabla_{x}T)^{2}\bigg{(}\mu^{2}\log
2+\frac{\pi^{2}}{3}T\mu+\frac{9}{2}T^{2}\zeta(3)\bigg{)}.\end{array}$ (25)

Here, interband yields to the induced electrical longitudinal and intraband to
the induced electrical hall current. Also, both contributions gives directly
proportional behavior with the first power of the warping coefficient.Thus,
the warping term show significant contribution in the higher-order case.

####  II.2.3 Thermal current by temperature gradient

In the same spirit of an electrical case, we have computed the thermal
currents in response to the temperature gradient applied in the x-direction.
We find that the linear case yields

$\begin{array}[]{rl}J_{x,Q}^{(1),Q}&\displaystyle=\frac{\tau}{8\hbar^{2}}T\mu\frac{\pi}{6}\nabla_{x}T.\end{array}$
(26)
$\begin{array}[]{rl}J_{y,Q}^{(1),Q}&\displaystyle=\frac{T}{8\hbar}\frac{\Delta\pi}{12\mu}\nabla_{x}T.\end{array}$
(27)

Both components show same dependency to the surface gap and chemical potential
as the electrical current. In addition to this, these are directly
proportional to the first power of the temperature. This dependency stems from
the definition of the thermal current, contains $(\varepsilon_{\bm{k}}-\mu)$
factor. Similarly for non-linear $n=2$ case, the expressions for currents read

$\begin{array}[]{rl}J_{x,Q}^{(2),Q}&\displaystyle=\frac{1}{32\pi\hbar}\frac{\Delta\lambda}{v_{F}^{2}}(\nabla_{x}T)^{2}\bigg{(}\frac{\pi^{2}}{6}+\frac{3\zeta(3)T}{2}\bigg{)}.\end{array}$
(28)
$\begin{array}[]{rl}J_{y,Q}^{(2),Q}&\displaystyle=\frac{1}{32\pi\hbar}\frac{\tau\lambda}{\hbar
v_{F}^{2}}(\nabla_{x}T)^{2}\bigg{(}\frac{\pi^{2}\mu^{2}}{6}+9\zeta(3)T\mu+\frac{7\pi^{4}T^{2}}{30}\bigg{)}.\end{array}$
(29)

####  II.2.4 Electrical current by electric field

In the application of an external electric field ${\bm{E}}=E\hat{x}$, the
electrical current in $x$ and $y-$-directions corresponding to $n=1$ case will
be

$\begin{array}[]{rl}J_{x,E}^{(1),E}&\displaystyle=\frac{e^{2}\tau}{4\pi\hbar^{2}}\bigg{\\{}\frac{\mu}{2}+T\log
2\bigg{\\}}E_{x}.\\\\[8.61108pt]
J_{y,E}^{(1),E}&\displaystyle=\frac{e^{2}}{8\pi\hbar}\frac{\Delta}{2\mu}E_{x}.\end{array}$
(30)

Similarly for $n=2$ case,

$\begin{array}[]{rl}J_{x,E}^{(2),E}&\displaystyle=\frac{e^{3}}{32\pi\hbar}\frac{\Delta\lambda}{v_{F}^{2}T}E_{x}^{2}\bigg{(}1+\frac{T\log
2}{2\mu}\bigg{)}\\\\[8.61108pt]
J_{y,E}^{(2),E}&\displaystyle=\frac{e^{3}}{32\pi\hbar}\frac{\tau\lambda}{\hbar
v_{F}^{2}T}E_{x}^{2}\bigg{(}\frac{\mu^{2}}{2}+2T\mu\log
2+\frac{\pi^{2}T^{2}}{6}\bigg{)}.\end{array}$ (31)

Using these expressions, we analyze the non-linearity in the Seebeck effect
and the Pelteir effect in terms of the ratio of non-linear to the linear
conductivities in the next section.

##  III Numerical Analysis

(a) noonleline

(b) noonleline

Figure 3: Variation of the ratio of non-linear to linear thermoelectric
conductivities with the warping coefficient at different temperature values.
(a). Longitudinal case; (b). Hall case.

In Fig. 3, we show the results for the ratio of non-linear conductivity to the
linear conductivity $\alpha_{2}/\alpha_{1}$, as a function of warping
coefficient $\lambda$ (eVÅ3) for different low temperature values. Here we
keep the surface gap $\Delta=25$ meV, chemical potential $\mu=25$ meV and
$v_{F}=2.55$ eVÅ. We find that the ratio increases linearly with the rise of
warping coefficient. However, with the rise of temperature, the magnitude of
the conductivities ratio decreases due to the consideration of only intrinsic
contribution or the non-disorder case. On comparing the longitudinal and Hall
cases, we find that the Hall case contributes more to the variation of the
non-linear Seebeck and Peltier coefficient. We see that both the ratios gives
first-order dependence with the strength of hexagonal degree of distortion.

(a) noonleline

(b) noonleline

Figure 4: Plot for the ratio of second-order thermoelectric conductivity to
the linear one with the gap at different temperature values. (a). Longitudinal
case; (b). Hall case.

In Fig. 4, the influence of the surface gap $\Delta$ on
$\alpha_{1}/\alpha_{2}$ keeping $\lambda=150$ eVÅ3, $\mu=25$meV and
$v_{F}=2.55$ eVÅis shown. Results indicate that the larger the surface gap is,
stronger the longitudinal ratio is at all temperature values. This variation
is also consistent with the analytical results. Contrary to it, the Hall ratio
decreases with the rise of gap at all temperatures. Further with the variation
of chemical potential, we find the exponential decay in the ratio of
conductivities with the temperature. However there is an increase in the Hall
ratio as chemical potential $\mu$ increases as shown in Fig. 5. Similar
behavior can be seen in the ratio of second-order to the first-order thermal
conductivities for different parameters. Here for demonstration we have only
shown the variations of thermal conductivities ratio numerically.

(a) noonleline

(b) noonleline

Figure 5: The ratio of second-order response to the linear one with the
temperature at different chemical potential values. (a). Longitudinal case;
(b). Hall case.

The above analysis indicate that the non-linear response or the non-linear
thermoelectric effects arise due to the distorted Fermi surface by the
hexagonal warping effect. If $\lambda=0$ eVÅ3, interband and intraband second-
order response vanishes, thus the conductivities ratio approaches to zero. In
addition to this, the surface gap shifts the Fermi surface from its original
position which influences the conductivity. This has been observed in Fig. 4.
In the linear regime, the Hall conductivity varies linearly with the gap and
the longitudinal shows independent behavior with the surface gap. However, the
non-linear conductivity shows opposite behavior, means
$\alpha_{2,x}\propto\Delta$ and $\alpha_{2,y}$ shows no variation with
$\Delta$. Further, the ratio shows turnover when $\Delta$ becomes larger than
the chemical potential $\mu$ as shown in Fg. 4.
The non-linear Seebeck and Peltier coefficients, proportional to the ratio of
the non-linear to the linear responses (Eq. 15and 16) increases with gap when
the field is applied in the x-direction and the current is also measured in
the same direction. However, it reduces in the perpendicular direction
(y-direction). But warping enhances the non-linearity contribution in both
directions.
The total conducitivity, a sum of the interband and intraband contributions is
presented here in detail. Intraband processes lead to finite longitudinal
response to the first-order in an external field whether electric field or
temperature gradient is applied. Howver, interband gives Hall contribution. To
the next-order in field, the intraband yields Hall conductivity and interband
produces longitudinal one. Thus, to analyze the non-linear effects one
requires both inter and intaband participation. One important message from
this analysis is the finite contribution to the non-linear effect only if we
consider a non-zero warping and gap effects in picture. To the zero case,
longitudinal case vanishes and Hall one diverges. Hence, the both surface and
warping effects are pivotal to understand the non-linear effects. However,
with the longitudinal geometry one can get more deviation from the linear
regime with these terms and enhance the overall magnitude of the
thermoelectric effects.
We have only considered the intrinsic contribution means the contribution
stems from Berry connection and field driving terms. Disorder contribution
will come from the scattering term $J(f_{\bm{k}})$ as mentioned in the
theoretical description. Here one requires the different interactions effects
such as electron-impurity, electron-phonon, etc. and these will play a
significant role at temperature behavior of the responses. These
considerations are beyond the scope of the present study.

##  IV Conclusion

Beyond the linear response regime, we have studied the thermoelectric
properties of a topological insulator. Our formalism is based on the quantum
kinetic approach and includes both intrinsic and extrinsic responses to the
electric and thermal fields. However, we mainly focus on the intrinsic part of
the nonlinear effects. We evaluate the thermal and electric currents
expressions to the second power of the external field. We have found that the
nonlinearity arises due to the ratio of the responses from interband and
intraband processes. Thus, both the contributions play significant role to
understand the nonlinear Seebeck and Peltier effects. Furthermore, we analyzed
the warping and the surface gap effect on these properties using this
formalism and provides fundamental importance to the nonlinear effects in
topological insulators which will be beneficial to improve the technological
advances.

###### Acknowledgements.

PB acknowledges the National Key Research and Development Program of China
grant No. 2017YFA0303400 and NSAF China grant No. U1930402 for financial
support.

## References

  * Alam and Ramakrishna (2013) H. Alam and S. Ramakrishna, Nano Energy 2, 190 (2013).
  * Rowe (2006) D. M. Rowe,  _Thermoelectrics Handbook: Macro to Nano_ (Taylor and Francis Group, New York, 2006).
  * Behnia (2015) K. Behnia,  _Fundamentals of Thermoelectricity_ (Oxford University Press, 2015).
  * Koumoto and Mori (2013) K. Koumoto and T. Mori,  _Thermoelectric Nanomaterials: Materials, Design and Applications_ (Springer Series in Material Series, Vol. 182, Springer, HeidelbergOxford University Press, 2013).
  * Dresselhaus  _et al._ (2007) M. Dresselhaus, G. Chen, M. Tang, R. Yang, H. Lee, D. Wang, Z. Ren, J.-P. Fleurial, and P. Gogna, Advanced Materials 19, 1043 (2007).
  * Poudel  _et al._ (2008) B. Poudel, Q. Hao, Y. Ma, Y. Lan, A. Minnich, B. Yu, X. Yan, D. Wang, A. Muto, D. Vashaee, X. Chen, J. Liu, M. S. Dresselhaus, G. Chen, and Z. Ren, Science 320, 634 (2008), https://science.sciencemag.org/content/320/5876/634.full.pdf .
  * Sootsman  _et al._ (2009) J. Sootsman, D. Chung, and M. Kanatzidis, Angewandte Chemie International Edition 48, 8616 (2009).
  * Tretiakov  _et al._ (2011) O. A. Tretiakov, A. Abanov, and J. Sinova, Applied Physics Letters 99, 113110 (2011).
  * Pei  _et al._ (2011) Y. Pei, X. Shi, A. LaLonde, H. Wang, L. Chen, and G. J. Snyder, Nature 473, 66 (2011).
  * Biswas  _et al._ (2012) K. Biswas, J. He, I. D. Blum, C.-I. Wu, T. P. Hogan, D. N. Seidman, V. P. Dravid, and M. G. Kanatzidis, Nature 489, 414 (2012).
  * Osterhage  _et al._ (2014) H. Osterhage, J. Gooth, B. Hamdou, P. Gwozdz, R. Zierold, and K. Nielsch, Applied Physics Letters 105, 123117 (2014).
  * Li and Xu (2015) L.-L. Li and W. Xu, Chinese Physics Letters 32, 047304 (2015).
  * Xu (2016) Y. Xu, Chinese Physics B 25, 117309 (2016).
  * Xu  _et al._ (2017) N. Xu, Y. Xu, and J. Zhu, npj Quantum Materials 2, 51 (2017).
  * Baldomir and Failde (2019) D. Baldomir and D. Failde, Scientific Reports 9, 6324 (2019).
  * Gayner and Kar (2016) C. Gayner and K. K. Kar, Progress in Materials Science 83, 330 (2016).
  * von Baltz and Kraut (1981) R. von Baltz and W. Kraut, Phys. Rev. B 23, 5590 (1981).
  * Khurgin (1994) J. B. Khurgin, J. Opt. Soc. Am. B 11, 2492 (1994).
  * Khurgin (1995) J. B. Khurgin, Applied Physics Letters 67, 1113 (1995), https://doi.org/10.1063/1.114978 .
  * Nastos and Sipe (2010) F. Nastos and J. E. Sipe, Phys. Rev. B 82, 235204 (2010).
  * Young and Rappe (2012) S. M. Young and A. M. Rappe, Phys. Rev. Lett. 109, 116601 (2012).
  * Inglot  _et al._ (2015) M. Inglot, V. K. Dugaev, E. Y. Sherman, and J. Barnaś, Phys. Rev. B 91, 195428 (2015).
  * Li  _et al._ (2016a) Z. Li, Q. Liu, S. Han, T. Iitaka, H. Su, T. Tohyama, H. Jiang, Y. Dong, B. Yang, F. Zhang, Z. Yang, and S. Pan, Phys. Rev. B 93, 245125 (2016a).
  * Wang  _et al._ (2017) C. Wang, X. Liu, L. Kang, B.-L. Gu, Y. Xu, and W. Duan, Phys. Rev. B 96, 115147 (2017).
  * Ogawa  _et al._ (2017) N. Ogawa, M. Sotome, Y. Kaneko, M. Ogino, and Y. Tokura, Phys. Rev. B 96, 241203 (2017).
  * Kim  _et al._ (2017) K. W. Kim, T. Morimoto, and N. Nagaosa, Phys. Rev. B 95, 035134 (2017).
  * Wang  _et al._ (2016) Y. Wang, M. Tokman, and A. Belyanin, Phys. Rev. B 94, 195442 (2016).
  * Tan and Rappe (2016) L. Z. Tan and A. M. Rappe, Phys. Rev. Lett. 116, 237402 (2016).
  * Morimoto and Nagaosa (2016a) T. Morimoto and N. Nagaosa, Phys. Rev. B 94, 035117 (2016a).
  * Khurgin (2016) J. B. Khurgin, J. Opt. Soc. Am. B 33, C1 (2016).
  * Morimoto and Nagaosa (2016b) T. Morimoto and N. Nagaosa, Science Advances 2 (2016b), DOI: 10.1126/sciadv.1501524.
  * Morimoto  _et al._ (2016) T. Morimoto, S. Zhong, J. Orenstein, and J. E. Moore, Phys. Rev. B 94, 245121 (2016).
  * Morimoto and Nagaosa (2016c) T. Morimoto and N. Nagaosa, Science Advances 2 (2016c), 10.1126/sciadv.1501524, https://advances.sciencemag.org/content/2/5/e1501524.full.pdf .
  * Hamamoto  _et al._ (2017) K. Hamamoto, M. Ezawa, K. W. Kim, T. Morimoto, and N. Nagaosa, Phys. Rev. B 95, 224430 (2017).
  * Cook  _et al._ (2017) A. M. Cook, B. M. Fregoso, F. de Juan, S. Coh, and J. E. Moore, Nature Communications 8, 14176 (2017).
  * Fregoso  _et al._ (2017) B. M. Fregoso, T. Morimoto, and J. E. Moore, Phys. Rev. B 96, 075421 (2017).
  * Zhang  _et al._ (2018) Y. Zhang, J. van den Brink, C. Felser, and B. Yan, 2D Materials 5, 044001 (2018).
  * Krasnok  _et al._ (2018) A. Krasnok, M. Tymchenko, and A. Alù, Materials Today 21, 8 (2018).
  * Tan and Rappe (2019) L. Z. Tan and A. M. Rappe, Phys. Rev. B 100, 085102 (2019).
  * Zhang and Vignale (2018) S. S.-L. Zhang and G. Vignale, in  _Spintronics XI_, Vol. 10732, edited by H.-J. Drouhin, J.-E. Wegrowe, M. Razeghi, and H. Jaffrès, International Society for Optics and Photonics (SPIE, 2018) pp. 97 – 107.
  * Fregoso (2019) B. M. Fregoso, Phys. Rev. B 100, 064301 (2019).
  * Liu  _et al._ (2019) Y. Liu, J. Lai, P. Yu, X. Zhuo, Z. Liu, J.-H. Chen, J. Feng, and D. Sun, Nature Materials 18, 476 (2019).
  * Bhalla  _et al._ (2020) P. Bhalla, A. H. MacDonald, and D. Culcer, Phys. Rev. Lett. 124, 087402 (2020).
  * Sekine and Nagaosa (2020) A. Sekine and N. Nagaosa, Phys. Rev. B 101, 155204 (2020).
  * Ahn and Nagaosa (2020) J. Ahn and N. Nagaosa, (2020), arXiv:2006.06709 [cond-mat.mes-hall] .
  * Wang and Qian (2020) H. Wang and X. Qian, “Giant nonlinear photocurrent in $\mathcal{PT}$-symmetric magnetic topological quantum materials,” (2020), arXiv:2006.13573 [cond-mat.mtrl-sci] .
  * López and Sánchez (2013) R. López and D. Sánchez, Phys. Rev. B 88, 045129 (2013).
  * Hwang  _et al._ (2014) S.-Y. Hwang, R. López, M. Lee, and D. Sánchez, Phys. Rev. B 90, 115301 (2014).
  * Gao and Xiao (2018) Y. Gao and D. Xiao, Phys. Rev. B 98, 060402 (2018).
  * Fu (2009) L. Fu, Phys. Rev. Lett. 103, 266801 (2009).
  * Culcer  _et al._ (2017) D. Culcer, A. Sekine, and A. H. MacDonald, Phys. Rev. B 96, 035106 (2017).
  * Nagaosa  _et al._ (2010) N. Nagaosa, J. Sinova, S. Onoda, A. H. MacDonald, and N. P. Ong, Rev. Mod. Phys. 82, 1539 (2010).
  * Zyuzin and Burkov (2012) A. A. Zyuzin and A. A. Burkov, Phys. Rev. B 86, 115133 (2012).
  * Monteiro  _et al._ (2015) G. M. Monteiro, A. G. Abanov, and D. E. Kharzeev, Phys. Rev. B 92, 165109 (2015).
  * Jia  _et al._ (2016) S. Jia, S.-Y. Xu, and M. Z. Hasan, Nature Materials 15, 1140 (2016).
  * Kaushik  _et al._ (2019) S. Kaushik, D. E. Kharzeev, and E. J. Philip, Phys. Rev. B 99, 075150 (2019).
  * Li  _et al._ (2016b) H. Li, H. He, H.-Z. Lu, H. Zhang, H. Liu, R. Ma, Z. Fan, S.-Q. Shen, and J. Wang, Nature Communications 7, 10301 (2016b).
  * Schumann  _et al._ (2017) T. Schumann, M. Goyal, D. A. Kealhofer, and S. Stemmer, Phys. Rev. B 95, 241113 (2017).
  * Armitage  _et al._ (2018) N. P. Armitage, E. J. Mele, and A. Vishwanath, Rev. Mod. Phys. 90, 015001 (2018).
  * Culcer  _et al._ (2020) D. Culcer, A. C. Keser, Y. Li, and G. Tkachov, 2D Materials 7, 022007 (2020).
  * Moore (2010) J. E. Moore, Nature 464, 194 (2010).
  * Hasan and Moore (2011) M. Z. Hasan and J. E. Moore, Annual Review of Condensed Matter Physics 2, 55 (2011).
  * Chen  _et al._ (2011) X. Chen, Z.-X. Liu, and X.-G. Wen, Phys. Rev. B 84, 235141 (2011).
  * Pollmann  _et al._ (2012) F. Pollmann, E. Berg, A. M. Turner, and M. Oshikawa, Phys. Rev. B 85, 075125 (2012).
  * Matsushita  _et al._ (F017) S. Y. Matsushita, K. K. Huynh, H. Yoshino, N. H. Tu, Y. Tanabe, and K. Tanigaki, Phys. Rev. Materials 1, 054202 (2F017).
  * Ivanov  _et al._ (2018) Y. V. Ivanov, A. T. Burkov, and D. A. Pshenay-Severin, physica status solidi (b) 255, 1800020 (2018), https://onlinelibrary.wiley.com/doi/pdf/10.1002/pssb.201800020 .
  * Wang and Jeng (2019) T.-H. Wang and H.-T. Jeng, Nano Energy 66, 104092 (2019).
  * Sturman and Fridkin (1992) B. I. Sturman and V. M. Fridkin,  _The Photovoltaic and Photorefractive Effects in Noncentrosymmetric Materials_ (Gordon and Breach, Philadelphia, 1992).
  * Ziman (1960) J. M. Ziman,  _Electrons and Phonons: The Theory of Transport Phenomenon in Solids_ (Clarendon Press, Oxford, 1960).
  * Goldsmid (2017) H. J. Goldsmid, in  _The Physics of Thermoelectric Energy Conversion_, 2053-2571 (Morgan and Claypool Publishers, 2017) pp. 1–1 to 1–3.
  * Misawa  _et al._ (2011) T. Misawa, T. Yokoyama, and S. Murakami, Phys. Rev. B 84, 165407 (2011).
  * Xiao and Wen (2013) X. Xiao and W. Wen, Phys. Rev. B 88, 045442 (2013).
  * Chang  _et al._ (2015) P.-H. Chang, T. Markussen, S. Smidstrup, K. Stokbro, and B. K. Nikolić, Phys. Rev. B 92, 201406 (2015).
  * Akzyanov and Rakhmanov (2018) R. S. Akzyanov and A. L. Rakhmanov, Phys. Rev. B 97, 075421 (2018).

##  Appendix A Derivation of Eq. 6

For the diagonal case, the kinetic equation reads

$\begin{array}[]{rl}&\displaystyle\frac{df_{d,{\bm{k}}}^{(n)}}{dt}+\frac{f_{d,{\bm{k}}}}{\tau}=\mathcal{D}_{T}(f_{d,{\bm{k}}}^{(n-1)})-J_{d}[f_{\text{od},\textbf{k}}^{(n)}].\end{array}$
(32)

Solving this equation using the integrating factor, we have

$\begin{array}[]{rl}f_{d,{\bm{k}}}^{(n)}&\displaystyle=\int_{0}^{\infty}dt^{\prime}e^{-t^{\prime}/\tau}\bigg{[}\mathcal{D}_{T}(f_{d,{\bm{k}}}^{(n-1)})-J_{d}[f_{\text{od},\textbf{k}}^{(n)}]\bigg{]}.\end{array}$
(33)
$\begin{array}[]{rl}f_{d,{\bm{k}}}^{(n)}&\displaystyle=\tau\bigg{[}\mathcal{D}_{T}(f_{d,{\bm{k}}}^{(n-1)})-J_{d}[f_{\text{od},\textbf{k}}^{(n)}]\bigg{]}.\end{array}$
(34)
$\begin{array}[]{rl}f_{d,{\bm{k}}}^{(n)}&\displaystyle=\tau\bigg{[}\frac{1}{\hbar}\frac{{\bm{\nabla}}T}{T}\cdot\frac{\partial
f_{d,{\bm{k}}}^{(n-1)}}{\partial{\bm{k}}}-J_{d}[f_{\text{od},\textbf{k}}^{(n)}]\bigg{]}.\end{array}$
(35)

Similarly, for the off-diagonal case

$\begin{array}[]{rl}&\displaystyle\frac{df_{\text{od},{\bm{k}}}^{(n)}}{dt}+\frac{i}{\hbar}[H_{0},f_{\text{od},{\bm{k}}}^{(n)}]+\frac{f_{\text{od},{\bm{k}}}}{\tau}=\mathcal{D}_{T}(f_{\text{od},{\bm{k}}}^{(n-1)})-J_{\text{od}}[f_{\text{d},{\bm{k}}}^{(n)}].\end{array}$
(36)
$\begin{array}[]{rl}f_{\text{od},{\bm{k}}}^{(n)}&\displaystyle=\int_{0}^{\infty}dt^{\prime}e^{-t^{\prime}/\tau}e^{-i\varepsilon_{m{\bm{k}}}t^{\prime}/\hbar}\bigg{[}\frac{1}{\hbar}\frac{{\bm{\nabla}}T}{T}\cdot\bigg{(}\frac{\partial
f_{\text{od},{\bm{k}}}^{(n-1)}}{\partial{\bm{k}}}\\\\[8.61108pt]
&\displaystyle\quad\quad-i[\mathcal{R}_{\bm{k}},f_{{\bm{k}}}^{(n-1)}]^{mm^{\prime}}\bigg{)}-J_{\text{od}}[f_{\text{d},{\bm{k}}}^{(n)}]\bigg{]}e^{i\varepsilon_{m^{\prime}{\bm{k}}}t^{\prime}/\hbar}.\end{array}$
(37)
$\begin{array}[]{rl}f_{\text{od},{\bm{k}}}^{(n)}&\displaystyle=\frac{\hbar}{i(\varepsilon_{m{\bm{k}}}-\varepsilon_{m^{\prime}{\bm{k}}})}\bigg{[}\frac{1}{\hbar}\frac{{\bm{\nabla}}T}{T}\cdot\bigg{(}\frac{\partial
f_{\text{od},{\bm{k}}}^{(n-1)}}{\partial{\bm{k}}}\\\\[8.61108pt]
&\displaystyle\quad\quad-i[\mathcal{R}_{\bm{k}},f_{{\bm{k}}}^{(n-1)}]^{mm^{\prime}}\bigg{)}-J_{\text{od}}[f_{\text{d},{\bm{k}}}^{(n)}]\bigg{]}.\end{array}$
(38)

Here the commutation relation between $\mathcal{R}_{\bm{k}}$ and $f_{\bm{k}}$
is

$\begin{array}[]{rl}[\mathcal{R}_{\bm{k}},f_{\bm{k}}^{(n)}]^{mm^{\prime}}&\displaystyle=\sum_{m^{\prime\prime}}\langle
m|\mathcal{R}_{\bm{k}}|m^{\prime\prime}\rangle\langle
m^{\prime\prime}|f_{{\bm{k}}}|m^{\prime}\rangle\\\\[8.61108pt]
&\displaystyle\quad\quad-\langle m|f_{{\bm{k}}}|m^{\prime\prime}\rangle\langle
m^{\prime\prime}|\mathcal{R}_{\bm{k}}|m^{\prime}\rangle.\end{array}$ (39)