# T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large
Language Model Signals for Science Question Answering

Lei Wang1, Yi Hu2, Jiabang He2, Xing Xu2, Ning Liu3, Hui Liu4, Heng Tao Shen2

###### Abstract

Large Language Models (LLMs) have recently demonstrated exceptional
performance in various Natural Language Processing (NLP) tasks. They have also
shown the ability to perform chain-of-thought (CoT) reasoning to solve complex
problems. Recent studies have explored CoT reasoning in complex multimodal
scenarios, such as the science question answering task, by fine-tuning
multimodal models with high-quality human-annotated CoT rationales. However,
collecting high-quality COT rationales is usually time-consuming and costly.
Besides, the annotated rationales are hardly accurate due to the external
essential information missed. To address these issues, we propose a novel
method termed _T-SciQ_ that aims at teaching science question answering with
LLM signals. The T-SciQ approach generates high-quality CoT rationales as
teaching signals and is advanced to train much smaller models to perform CoT
reasoning in complex modalities. Additionally, we introduce a novel data
mixing strategy to produce more effective teaching data samples for simple and
complex science question answer problems. Extensive experimental results show
that our T-SciQ method achieves a new state-of-the-art performance on the
ScienceQA benchmark, with an accuracy of 96.18%. Moreover, our approach
outperforms the most powerful fine-tuned baseline by 4.5%. The code is
publicly available at https://github.com/T-SciQ/T-SciQ.

## Introduction

Scientific problem solving has recently been employed to evaluate the multi-
hop reasoning capability and interpretability of AI systems (Kembhavi et al.
2017; Sampat, Yang, and Baral 2020; Dalvi et al. 2021). However, these
datasets (Kembhavi et al. 2017; Jansen et al. 2018) suffer from limited scale.
To address this issue, Lu et al. (2022a) introduces a large-scale science
question-answering dataset across broad topics and skills called ScienceQA.
This dataset consists of 21,208 multimodal data examples associated with
questions, context, images, options, lectures, and explanations. An example is
shown in Figure 1, illustrating that a model must comprehend multimodal inputs
and incorporate external knowledge to answer scientific questions.

Figure 1: The input of a ScienceQA data example includes a question, context,
image, skill, and options. Annotations include the ground truth answer and CoT
rationale (lecture and solution). Compared to annotated CoT, LLM-generated CoT
includes greater amounts of essential external knowledge.

Recently, Large Language Models (LLMs) have shown exceptional performance in
various Natural Language Processing (NLP) tasks (Brown et al. 2020; Thoppilan
et al. 2022). Specifically, they have demonstrated the chain-of-thought (CoT)
ability to solve complex reasoning problems by using a few demonstration
examples without additional training (Wei et al. 2022a; Kojima et al. 2022;
Zhang et al. 2022). However, the existing research on CoT reasoning is mainly
limited to the language modality (Wang et al. 2022a; Zhou et al. 2022; Lu et
al. 2022b; Fu et al. 2022), with little attention paid to multimodal
scenarios, such as science question answering. To address this issue, a common
approach is to use caption models to translate visual information into the
language modality and prompt LLMs to perform CoT reasoning (Lu et al. 2022a).
However, the use of caption generation models in scientific problems may
result in significant information loss when meeting highly complex images. To
overcome this issue, Zhang et al. (2023b) proposed a framework called
Multimodal-CoT that models both language and visual modalities into a two-
stage fine-tuning process, which separates rationale generation and answer
inference.

The Multimodal-CoT method has a significant disadvantage because it relies on
the human-annotated CoT rationale to fine-tune the model. While incorporating
human-annotated CoT signals is helpful for training models to facilitate CoT
reasoning ability, it has two fundamental limitations. First, the human
annotation of CoT reasoning is time-consuming (Nye et al. 2021; Cobbe et al.
2021), particularly for complex tasks like ScienceQA, which necessitates
extensive expert knowledge to create a reasoning process for the answer.
Second, as shown in Figure 1, the annotated rationale may lack essential
external information to derive the final answer due to the limited expertise
of human annotators.

To address these issues, we propose a novel approach named _T-SciQ_ to solve
the ScienceQA task. The proposed T-SciQ framework consists of three stages:
generating teaching data, mixing teaching data, and fine-tuning. For teaching
data generation, we use a simple zero-shot instruction and a hint of the
correct answer to generate a CoT rationale for a QA data example to obtain a
QA-CoT sample. Although the model taught by QA-CoT samples excels at tackling
simple problems, it still struggles with highly complicated problems. To
overcome this challenge, we follow the zero-shot plan-and-solve prompting
(Wang et al. 2023) to generate plan-based CoT (PCoT) rationales, which
decompose complex problems into simpler subproblems to solve, to obtain QA-
PCoT teaching samples. To combine the strengths of both teaching signals, we
create a new teaching dataset called T-SciQ by mixing QA-CoT and QA-PCoT
datasets. Specifically, we use the validation set to determine whether the
PCoT teaching signal or CoT teaching signal is more appropriate for each data
example in a given skill. Then, we fine-tune the student model with teaching
data. We follow the Multimodal-CoT (Zhang et al. 2023b) to build our student
model, which consists of two-stage: rationale generation teaching and answer
inference teaching. During inference, the model trained in the first stage
generates rationales for the test data. The generated rationales are
subsequently used in the second stage to infer answers.

Experiment results on the ScienceQA benchmark show that our method surpasses
the previous state-of-the-art by a large margin. Specifically, the student
model taught by T-SciQ teaching samples outperforms the most powerful fine-
tuned baseline by 4.5%, the strongest instruction-tuning based multimodal
baseline by 5.26%, the best GPT-4 based few-shot baseline by 9.64%, and human
performance by 7.78%. To demonstrate the versatility of our teaching approach,
we additionally conduct experiments to compare Reason-Teacher (Ho, Schmid, and
Yun 2022) on six reasoning tasks. Our main contributions are summarized as
follows: 1) We propose a novel framework for generating high-quality CoT
rationale and training student models to perform CoT reasoning for the
ScienceQA task; 2) We introduce a data mixing strategy to produce effective
teaching data samples for simple and complex problems; 3) Our method achieves
a new state-of-the-art performance on the ScienceQA benchmark, surpassing all
previous models by a large margin.

Figure 2:  Key steps of our T-SciQ approach. T-SciQ consists of three stages:
(i) generating teaching data; (ii) mixing teaching data; and (iii) fine-
tuning.

## Related Work

#### Chain-of-Thought Prompting.

Recently, to solve complex reasoning tasks, Wei et al. (2022b) propose CoT
prompting by prompting large language models to generate intermediate
reasoning processes before reaching the final answer. Subsequently, a lot of
work has been proposed to further improve CoT prompting from different
aspects, including improving the quality of demonstrations (Rubin, Herzig, and
Berant 2021; Zhang et al. 2022; Fu et al. 2022; Lu et al. 2022b; He et al.
2023) and improving the quality of reasoning chains (Zhou et al. 2022; Khot et
al. 2022; Chen et al. 2022; Wang et al. 2022b, a; Li et al. 2022b; Tian et al.
2023). Zero-shot CoT (Kojima et al. 2022) elicited reasoning step by appending
a prompt like “Let’s think step by step” to the test question. Iterative
Prompting (Wang, Deng, and Sun 2022) dynamically synthesizes prompts
conditioned on the current step’s contexts. PoT Prompting (Chen et al. 2022)
writes a program as a rationale and invoked the reasoning ability of LLMs by
executing the generated program. Chameleon (Lu et al. 2023) proposed a plug-
and-play compositional reasoning framework to utilize multiple modules to
obtain high quality prompting. Our work mainly focuses on mixing different
teaching CoT rationales for different problems.

LLMs as Teachers. In recent studies, CoT reasoning is elicited in small models
using fine-tuned language models. Magister et al. (2022) benefit smaller
models through CoT distillation. Huang et al. (2022) show that LLMs can
enhance reasoning using self-generated solutions from unlabeled data. Ho,
Schmid, and Yun (2022) propose Fine-tune-CoT to leverage the capabilities of
LLMs to generate reasoning samples and teach smaller models via fine-tuning.
Distilling step-by-step (Hsieh et al. 2023) improves small model performance
using LLM rationales with less data. Multimodal-CoT (Zhang et al. 2023b) uses
two-stage fine-tuning with annotated CoT rationales and visual features to
achieve state-of-the-art results on the ScienceQA benchmark. Our work exploits
generating two types of teaching data from LLMs and mixing teaching data. We
discover that this simple method highly improves student performance in
complex multi-modality tasks, which has not yet been recognized in previous
studies on fine-tuning with CoT reasoning (Hsieh et al. 2023; Ho, Schmid, and
Yun 2022; Huang et al. 2022; Magister et al. 2022; Fu et al. 2023; Hu et al.
2023).

## Our T-SciQ Approach

### Overview

This section presents the proposed fine-tuning strategy T-SciQ, which utilizes
a LLM named SciTeacher to generate teaching data and improve the performance
of a smaller student model (SciStudent) by generated teaching data. The
proposed T-SciQ strategy comprises three components: generating teaching data,
mixing teaching data, and fine-tuning, as depicted in Figure 2. To generate
the teaching data, we leverage SciTeacher to produce CoT rationales to obtain
Question-Answer-CoT (QA-CoT) samples, and planning-based CoT rationale (PCoT)
to obtain Question-Answer-PCoT (QA-PCoT) samples. To combine the strengths of
both datasets, we create a new teaching dataset called T-SciQ by mixing QA-CoT
and QA-PCoT datasets. Specifically, we use the validation set to determine
whether the PCoT teaching signal or CoT teaching signal is more appropriate
for each data example in a given skill. We then use T-SciQ teaching samples to
fine-tune the smaller student models. In the following, we provide a detailed
description of these three components.

### Generating Teaching Data

We produce two types of data samples for teaching: QA-CoT sample with a
generated CoT rationale and QA-PCoT sample equipped with a generated PCoT
rationale.

##### QA-CoT Sample Generation.

Although using human-annotated CoT signals is valuable for training models to
elicit CoT reasoning ability, it has two inherent limitations: time-consuming
and lack of external essential information due to human annotators’ restricted
expertise.

To address these issues, we introduce a zero-shot prompting to generate high-
quality CoT rationales from LLMs. We achieve this by converting the input
training data example $X$ into a prompt, utilizing a straightforward template
that reads as follows: “Question: [$X_{q}$]. Context: [$X_{c}$]. Options:
[$X_{o}$]. Correct Answer: [$A$]. [$Instruct$]”. Here, the [$X_{q}$] slot is
for the input question, the [$X_{c}$] slot is for the input context, the
[$X_{o}$] slot contains the possible options, the [$A$] slot is for the
correct answer that can work as a hint to guide LLMs to generate a more
reliable rationale, and the [$Instruct$] slot contains instructions, i.e.,
“Please give me a detailed explanation.”, to guide LLMs to perform the task.
Note that the context may not be included for some data examples, in which
case the context slot is replaced with “N/A”. Subsequently, we feed the filled
prompt to LLMs to output a reasoning process for a given training data example
to obtain QA-CoT data $D_{\text{QA-CoT}}$.

##### QA-PCoT Sample Generation.

Although using QA-CoT samples can address issues of human-annotated CoT,
addressing highly complex problems remains a challenge. To overcome this
challenge and obtain appropriate teaching CoT rationale, we introduce a 3-step
zero-shot prompting to decompose complex problems into simpler subproblems.

Step 1: Lecture Generation. The lecture template used to generate a lecture
for a particular skill is formulated as follows: “Skill: [$S$]. QA pairs:
[$X_{q},A$] … [$Instruct$].” In this prompt, [$Instruct$] is as follows:
“based on the problems above, please give a general lecture on the [$S$] type
of question in one sentence.”. Note that many QA examples need the same skill
to be solved.

Step 2: Plan Generation. The template used to generate a plan for a specific
skill based on the generated lecture is formulated as follows: “Skill: [$S$].
Lecture: [$L$]. QA pairs: [$X_{q},A$] … [$Instruct$].”. In this prompt,
[$Instruct$] is written as follows: “Based on the lecture above and these
problems, let’s understand these problems and devise a general and brief plan
step by step to solve these problems (begin with 1, 2, 3…)”.

Step 3: Rationale Generation. The lecture and plan generated by the first two
prompts are used to generate a plan-based CoT rationale for each training
example. The rationale generation template is formulated as follows: “Skill:
[$S$]. Lecture: [$L$]. Plan: [$P$]. QA pair: [$X_{q},A$]. [$Instruct$].”. In
this prompt, [$Instruct$] is written as follows: “Based on the lecture, the
plan and the problem, please carry out the plan and solve the problem step by
step (begin with 1, 2, 3…)”. Examples of this three-step prompting can be
found in the supplementary material.

### Mixing Teaching Data

The QA-PCoT dataset is effective for teaching problem-solving skills for
complex problems, while simpler problems don’t require decomposition. In
contrast, the QA-CoT dataset is suitable for teaching problem-solving skills
for simple problems. To combine the strengths of both datasets, we create a
new teaching dataset called T-SciQ by mixing QA-CoT and QA-PCoT datasets. We
introduce a new approach that uses the validation set to determine whether the
PCoT teaching signal or CoT teaching signal is more appropriate for a data
example in a given skill.

Given a ScienceQA problem $P_{i}$ with the language input $X_{i,\textrm{la}}$
and the visual input $X_{i,\textrm{v}}$, our objective is to let an answer
generation model $F^{s}_{a}$ help identify the optimal teaching signal
$T_{i,k}$ from the possible choices $T_{i}$, i.e., CoT teaching signal
$T_{i,\text{cot}}$ or PCoT teaching signal $T_{i,\text{pcot}}$, thereby
maximizing the answer accuracy of the validation set. The answer generation
module $F^{s}_{a}$ is similar to the one described in Multimodal-CoT (Zhang et
al. 2023b). The generated answer $\hat{A}_{i}$ is produced by
$F^{s}_{a}(X_{i,\text{la}},X_{i,\text{v}},T_{i,k})$, and the number of errors
is obtained by comparing the generated answer $\hat{A}_{i}$ and the label
$A_{i}$. If the number of errors for validation samples with PCoT in a skill
is lower than that of validation samples with CoT in a skill, we select PCoT
rationale as the teaching rationale for all training data examples in this
skill. Otherwise, we select CoT rationale. The obtained teaching samples are
then used to fine-tune the student model. To train the answer generation
module, we utilize a subset of training data examples, each of which is
associated with the human-annotated teaching signal from the original
ScienceQA dataset.

### Fine-Tuning

Table 1: Main results (%) on the test set of ScienceQA. There are totally 8 classes of questions, namely natural science (NAT), social science (SOC), language science (LAN), text context (TXT), image context (IMG), no context (NO), grades 1-6 (G1-6), and grades 7-12 (G7-12). The best results are boldfaced. The improvements are shown in blue. Model | Size | NAT | SOC | LAN | TXT | IMG | NO | G1-6 | G7-12 | Avg
---|---|---|---|---|---|---|---|---|---|---
Human | - | 90.23 | 84.97 | 87.48 | 89.60 | 87.50 | 88.10 | 91.59 | 82.42 | 88.40
MCAN (Yu et al. 2019) | 95M | 56.08 | 46.23 | 58.09 | 59.43 | 51.17 | 55.40 | 51.65 | 59.72 | 54.54
Top-Down (Anderson et al. 2018) | 70M | 59.50 | 54.33 | 61.82 | 62.90 | 54.88 | 59.79 | 57.27 | 62.16 | 59.02
BAN (Kim, Jun, and Zhang 2018) | 112M | 60.88 | 46.57 | 66.64 | 62.61 | 52.60 | 65.51 | 56.83 | 63.94 | 59.37
DFAF (Gao et al. 2019) | 74M | 64.03 | 48.82 | 63.55 | 65.88 | 54.49 | 64.11 | 57.12 | 67.17 | 60.72
ViLT (Kim, Son, and Kim 2021) | 113M | 60.48 | 63.89 | 60.27 | 63.20 | 61.38 | 57.00 | 60.72 | 61.90 | 61.14
Patch-TRM (Lu et al. 2021) | 90M | 65.19 | 46.79 | 65.55 | 66.96 | 55.28 | 64.95 | 58.04 | 67.50 | 61.42
VisualBERT (Li et al. 2019) | 111M | 59.33 | 69.18 | 61.18 | 62.71 | 62.17 | 58.54 | 62.96 | 59.92 | 61.87
UnifiedQA${}_{\texttt{Base}}$ (Khashabi et al. 2020) | 223M | 68.16 | 69.18 | 74.91 | 63.78 | 61.38 | 77.84 | 72.98 | 65.00 | 70.12
LLaMa-Adapter (Zhang et al. 2023a) |  $>$7B | 84.37 | 88.30 | 84.36 | 83.72 | 80.32 | 86.90 | 85.83 | 84.05 | 85.19
LLaVA (Liu et al. 2023) |  $>$7B | 90.36 | 95.95 | 88.00 | 89.49 | 88.00 | 90.66 | 90.93 | 90.90 | 90.92
GPT-3.5 (Chen et al. 2020) |  $>$175B | 74.64 | 69.74 | 76.00 | 74.44 | 67.28 | 77.42 | 76.80 | 68.89 | 73.97
GPT-3.5 w/ CoT (Lu et al. 2022a) |  $>$175B | 75.44 | 70.87 | 78.09 | 74.68 | 67.43 | 79.93 | 78.23 | 69.68 | 75.17
ChatGPT w/ CoT (Lu et al. 2023) |  $>$175B | 78.82 | 70.98 | 83.18 | 77.37 | 67.92 | 86.13 | 80.72 | 74.03 | 78.31
GPT-4 w/ CoT (Lu et al. 2023) |  $>$175B | 84.06 | 73.45 | 87.36 | 81.87 | 70.75 | 90.73 | 84.69 | 79.10 | 82.69
Chameleon (Lu et al. 2023) |  $>$175B | 89.83 | 74.13 | 89.82 | 88.27 | 77.64 | 92.13 | 88.03 | 83.72 | 86.54
UnifiedQA-CoT${}_{\texttt{Base}}$ (Lu et al. 2022a) | 223M | 71.00 | 76.04 | 78.91 | 66.42 | 66.53 | 81.81 | 77.06 | 68.82 | 74.11
UnifiedQA-T-SciQ${}_{\texttt{Base}}$ (Ours) | 223M | 76.56 | 88.99 | 80.45 | 72.90 | 73.84 | 83.47 | 81.09 | 75.19 | 79.41
Improvement | - | +5.56 | +12.95 | +1.54 | +6.48 | +7.31 | +1.66 | +4.03 | +6.37 | +5.30
Mutimodal-CoT${}_{\texttt{Base}}$ (Zhang et al. 2023b) | 223M | 87.52 | 77.17 | 85.82 | 87.88 | 82.90 | 86.83 | 84.65 | 85.37 | 84.91
Mutimodal-T-SciQ${}_{\texttt{Base}}$ (Ours) | 223M | 91.52 | 91.45 | 92.45 | 91.94 | 90.33 | 92.26 | 92.11 | 91.10 | 91.75
Improvement | - | +4.00 | +14.28 | +6.63 | +4.06 | +7.43 | +5.43 | +7.46 | +5.73 | +6.84
Mutimodal-CoT${}_{\texttt{Large}}$ (Zhang et al. 2023b) | 738M | 95.91 | 82.00 | 90.82 | 95.26 | 88.80 | 92.89 | 92.44 | 90.31 | 91.68
Mutimodal-T-SciQ${}_{\texttt{Large}}$ (Ours) | 738M | 96.89 | 95.16 | 95.55 | 96.53 | 94.70 | 96.79 | 96.44 | 95.72 | 96.18
Improvement | - | +0.98 | +13.16 | +4.73 | +1.27 | +5.90 | +3.90 | +4.00 | +5.41 | +4.50

Our teaching follows the Multimodal-CoT (Zhang et al. 2023b) two-stage fine-
tuning framework: rationale generation teaching and answer inference teaching.

##### Rationale Generation Teaching.

In this stage, the rationale generation model $F_{r}(P_{i})$ is trained to
predict the teaching signal $T_{i}$ for a given problem $P_{i}$, where $T_{i}$
either be CoT rationale or PCoT rationale. The input of $F_{r}(P_{i})$
consists of $X_{i,\textrm{la}}^{1}$ and $X_{i,\textrm{v}}$, where
$X_{i,\textrm{la}}^{1}$ represents the language input and $X_{i,\textrm{v}}$
represents the visual input. Formally, the probability of generating rationale
$T_{i}$ can be formulated as follows:

$p(T_{i}|X_{i,\textrm{la}^{1}},X_{i,\textrm{v}})=\prod_{j=1}^{N_{T_{i}}}p_{\theta_{r}}\left(T_{i,j}\mid
X_{i,\textrm{la}}^{1},X_{i,\textrm{v}},T_{i,<j}\right),$ (1)

where $\theta_{r}$ represents learnable parameters of the rationale generation
model $F_{r}$ and $N_{T_{i}}$ is the length of $T_{i}$.

##### Answer Inference Teaching.

In the second stage, we construct the language input $X_{i,\textrm{la}}^{2}$
by appending the teaching rationale $T_{i}$ to the original language input
$X_{i,\textrm{la}}^{1}$. The new input $X^{\prime}_{i}$ is then fed to the
answer inference model to infer the final answer
$A_{i}=F_{a}(X^{\prime}_{i})$, where
$X^{\prime}_{i}=\\{X_{i,\textrm{la}}^{2},X_{i,\textrm{v}}\\}$. Formally, the
probability of generating answer $A_{i}$ can be formulated as follows:

$p(A_{i}|X^{2}_{i,\textrm{la}},X_{i,\textrm{v}})=\prod_{j=1}^{N_{A_{i}}}p_{\theta_{a}}\left(A_{i}\mid
X^{2}_{i,\textrm{la}},X_{i,\textrm{v}},A_{i,<j}\right),$ (2)

where $\theta_{a}$ represents learnable parameters in the answer inference
teaching stage.

##### Model Architecture

We utilize the Multimodal-CoT (Zhang et al. 2023b) model architecture as our
default, which employs a Transformer model (Vaswani et al. 2017) for encoding
language and a vision Transformer for encoding visual information. The gated
fusion mechanism, proposed in (Li et al. 2022a), is used to effectively
integrate the language and vision representations. Finally, a Transformer
decoder is used to generate the target output. Note that rationale generation
and answer inference share the same model but differ in the input and output.

## Experiment

Table 2: Ablation study of the impact of different signals provided by LLMs across all topics.  Model |  Avg
---|---
Multimodal-T-SciQ${}_{\texttt{Base}}$ (Mixing) | 91.75
Multimodal-T-SciQ${}_{\texttt{Base}}$ only w/ QA-CoT | 85.99
Multimodal-T-SciQ${}_{\texttt{Base}}$ only w/ QA-PCoT | 88.56
Mutimodal-CoT${}_{\texttt{Base}}$ | 84.91
Multimodal-T-SciQ${}_{\texttt{Large}}$ (Mixing) | 96.18
Multimodal-T-SciQ${}_{\texttt{Large}}$ only w/ QA-CoT | 93.44
Multimodal-T-SciQ${}_{\texttt{Large}}$ only w/ QA-PCoT | 94.11
Mutimodal-CoT${}_{\texttt{Large}}$ | 91.68

### Experimental Setup

Dataset. We evaluate our proposed method on the ScienceQA (Lu et al. 2022a)
dataset, a latest multimodal multiple-choice science question dataset
comprising 21,208 examples. ScienceQA encompasses a wide range of topics
across three distinct subjects: natural science, social science, and language
science. The dataset comprises 26 topics, 127 categories, and 379 skills that
are relevant to these three subjects. We employ the official split provided by
ScienceQA, which divides the dataset into training, validation, and test sets
with a ratio of $3$:$1$:$1$, i.e., $12,726$, $4,241$, and $4,241$ examples,
respectively. The dataset includes annotated reasoning chains for each data
example. In this work, we extract our training signals from large language
models instead of using human annotated signals.

Baselines. We provide a comparison of our proposed method with extensive
baseline methods. Specifically, we have several early VQA models, including
MCAN (Yu et al. 2019), Top-Down (Anderson et al. 2018), BAN (Kim, Jun, and
Zhang 2018), DFAF (Gao et al. 2019). These VQA baselines use the question,
context, and answer choices as textual input and the image as the visual
input. They predict a score distribution over the answer candidates using a
linear classifier. In addition, we include pre-trained text-to-text and
multimodal models such as ViLT (Kim, Son, and Kim 2021), Patch-TRM (Lu et al.
2021), and VisualBERT (Li et al. 2019), UnifiedQA (Khashabi et al. 2020), MM-
COT (Zhang et al. 2023b). These methods use pre-trained models as backbone
models and incorporate additional modules to handle multimodal signals if
necessary. We also include recent LLM-based multimodal fine-tuned baselines
such as LLaMa-Adapter (Zhang et al. 2023a) and LLaVA (Liu et al. 2023). They
use a strong open-access LLM such as LLaMa (Touvron et al. 2023) as the base
model and incorporate a vision module to model visual information. We also
include widely-used in-context learning baselines: the chain of thought (CoT)
prompting (Wei et al. 2022a), where each in-context demonstration example
comprises the input question and output annotated reasoning process. We
compare to the CoT baselines over different API-based OpenAI LLMs (OpenAI
2022, 2023), such as GPT-3.5 (GPT-3.5 w/ COT), ChatGPT (ChatGPT w/ COT), GPT-4
(GPT-4 w/ COT), and Chameleon (Lu et al. 2023). Additionally, we also compare
to the standard few-shot prompting approach using GPT-3.5 (GPT-3.5).

Evaluation Metrics. As ScienceQA is a benchmark for multiple-choice question
answering, the accuracy of the answer is evaluated by comparing the ground
truth option with the final prediction generated by the evaluated model.

Implementation Details. By default, we utilize the GPT-3.5 of text-davinci-003
version as the teacher model for our approach unless otherwise specified. To
validate the generalizability of our method, we experiment with three distinct
student models, namely UnifiedQA${}_{\texttt{Base}}$ w/ CoT (Lu et al. 2022a),
Mutimodal-CoT${}_{\texttt{Base}}$(Lu et al. 2022a), and Mutimodal-
CoT${}_{\texttt{Large}}$(Zhang et al. 2023b). These models are chosen due to
their strong performances achieved by fine-tuning with annotated reasoning
signals. To ensure fairness of comparison and effectiveness of our proposed
method, we only replace the training signals generated by our approach with
annotated signals while maintaining the same settings as the original paper.
These student models are 200$\times$ smaller than their teacher models.

### Main Results

#### T-SciQ _v.s._ Baselines.

Table 1 details the performance accuracy of baselines and student models
trained using the proposed T-SciQ signals. Mutimodal-T-
SciQ${}_{\texttt{Large}}$, which is the model architecture of Mutimodal-
CoT${}_{\texttt{Large}}$ fine-tuned with mixed teacher signals, attains an
accuracy of 96.18% and consistently outperforms all state-of-the-art methods
by a large margin for all topics across all subjects. Specifically, Mutimodal-
T-SciQ${}_{\texttt{Large}}$ outperforms the most powerful fine-tuning
baseline, Mutimodal-CoT${}_{\texttt{Large}}$, which is trained by annotated
chain-of-thought signals, by 4.5% (91.68% $\rightarrow$ 96.18%), the strongest
instruction-tuning based multimodal baseline, LLaVa, by 5.26% (90.92%
$\rightarrow$ 96.18%), the best GPT-4 based few-shot baseline, Chameleon, by
9.64% (86.54% $\rightarrow$ 96.18%), and human performance by 7.78% (88.40%
$\rightarrow$ 96.18%). This significant improvement of our proposed method
suggests that higher-quality teaching signals of planning and reasoning
provided by LLMs elicit better planning and chain-of-thought reasoning ability
in student models smaller than 1B.

#### T-SciQ with Different Base Student Models.

Instead of only using the model architecture of Mutimodal-
CoT${}_{\texttt{Large}}$ as the base student model, we evaluate different base
student models fine-tuned with mixed teaching signals: the variant UnifiedQA-
T-SciQ${}_{\texttt{Base}}$ and Mutimodal-T-SciQ${}_{\texttt{Base}}$. The
relative performance ranking between the base student model with annotated CoT
signals and the one with mixing teacher signals remains unchanged.
Specifically, UnifiedQA-T-SciQ${}_{\texttt{Base}}$ outperforms
UnifiedQA${}_{\texttt{Base}}$ w/ CoT by 5.3% (74.11% $\rightarrow$ 79.41%),
and Mutimodal-T-SciQ${}_{\texttt{Base}}$ outperforms Mutimodal-
CoT${}_{\texttt{Base}}$ by 6.84% (84.91% $\rightarrow$ 91.75%). T-SciQ still
achieves the best performance with different base student models. These
encouraging results indicate the generalizability of the proposed teaching
signals.

Table 3: Accuracy (%) of Mutimodal-T-SciQ${}_{\texttt{Base}}$ using different
visual features.

Method | T-SciQ
---|---
QA-CoT | QA-PCoT | T-SciQ
Language Only | 84.44 | 85.38 | 87.24
w/ CLIP | 86.18 | 87.41 | 90.90
w/ DETR | 85.99 | 88.56 | 91.75
w/ ResNet | 86.06 | 87.69 | 91.44

(a)

(b)

Figure 3:  Further analysis on (a) the effect of Mutimodal-T-
SciQ${}_{\texttt{Base}}$ trained with different proportion of generated data
and (b) accuracy curve of the baseline Mutimodal-CoT${}_{\texttt{Base}}$ and
our Mutimodal-T-SciQ${}_{\texttt{Base}}$ across epochs.

(a)

(b)

Figure 4:  Further analysis on (a) accuracy (%) of Mutimodal-T-
SciQ${}_{\texttt{Base}}$ with teaching signals provided by different base LLMs
and (b) error analysis of prediction for specific skills.

(a)

(b)

Figure 5:  Examples of MM-CoT (baseline) and the model trained with T-SciQ
(ours) signals for generating rationales and predicting answers. To solve
these examples, commonsense knowledge such as geographic knowledge (a) and
multi-step reasoning (b) are required.

### Further Analysis

#### Effect of Different Signals of T-SciQ.

Our approach incorporates two distinct components for teaching signals: QA-CoT
and QA-PCoT. We early show that combining these two signals (i.e., Mutimodal-
T-SciQ) yields significantly better results than using only human-annotated
CoT signals (i.e., Mutimodal-CoT) when teaching student models. In this
section, we aim to evaluate the impact of each teaching signals by testing the
performance of Mutimodal-T-SciQ${}_{\texttt{Base}}$ and Mutimodal-T-
SciQ${}_{\texttt{Large}}$ when either QA-CoT or QA-PCoT signal is removed. As
demonstrated in Table 2, we can observe a significant decrease in answering
accuracy when either teaching signal was removed. These findings indicate the
effectiveness of both proposed teaching signals. This is because 1) student
models taught by QA-CoT signals can incorporate a more extensive range of
knowledge from the open world rather than solely relying on the knowledge of
annotators and 2) student models taught by QA-PCoT signals can decompose
complex problems into several simpler sub-problems.

#### Impact of visual Features.

The choice of visual features can significantly affect the performance of
models on ScienceQA. Thus, we conduct an evaluation of three widely-used
visual features, which are CLIP (Radford et al. 2021), DETR (Carion et al.
2020), and ResNet (He et al. 2016). Both CLIP and DETR can provide patch-level
features, and DETR is designed for object detection. As for ResNet features,
we use ResNet-50 to derive visual features. Table 3 shows the results of
comparing these three visual features. Our findings suggest that incorporating
visual features yields superior performance than relying on language-only
baselines. Notably, DETR consistently outperforms the other two features in
most cases, and hence, we adopt it as the default visual feature in our main
experiments.

#### Proportion of Generated Data in Training Data.

To further compare the T-SciQ signals produced by LLMs and the annotated CoT
signals, we experiment with manipulating the proportion of these two signals
within the training data. We vary the proportion of T-SciQ signals from 0% to
100%. As demonstrated in Figure 3(a), the increasing proportion of training
data with T-SciQ signals increases performance.

#### Performance Change with Epoch.

Figure 3(b) shows the performance trends of the baseline Mutimodal-
CoT${}_{\texttt{Base}}$ and our proposed Mutimodal-T-SciQ${}_{\texttt{Base}}$
across different training epochs. Notably, our method consistently outperforms
the baseline across all epochs. We adopt a two-stage training approach similar
to the baseline Mutimodal-CoT${}_{\texttt{Base}}$, where we first train the
explanation generation module and then train the answer prediction. Hence,
like the baseline, our method exhibits relatively higher accuracy at the
initial training stages.

#### Effect of Teaching Signals Provided by Different Base LLMs.

We use the GPT-3.5 model by default, specifically the text-davinci-003
version, to generate teaching signals in the main experiment. However, other
powerful LLMs can also provide useful signals, such as the earlier version of
GPT-3.5, text-davinci-002, and the recently popular ChatGPT model. This study
explores the effectiveness of a mixture of QA-CoT signals from text-
davinci-002, text-davinci-003, or ChatGPT, and QA-PCoT signals from the above
API-based models. We conduct this experiment using the Multimodal-T-
SciQ${}_{\texttt{Base}}$. Figure 4(a) shows the comparison of the performance
of nine different mixture strategies. Our results show that even the worst
strategy, which involves a mixture of QA-CoT signal from text-davinci-003 and
QA-PCoT signal from text-davince-002, outperforms annotated CoT signal by a
significant margin. It indicates that regardless of the mixture strategy used,
LLMs can provide signals with more useful knowledge from the open world.

#### Error Analysis.

To better understand the model’s behavior trained using our proposed T-SciQ
signals, we analyze six selected skills shown in Figure 4(b). It shows the
error analysis of prediction for six specific skills (A-F), i.e., “Using guide
words”, “Comparing properties of objects”, “Reading a map: cardinal
directions”, “Identifying oceans and continents”, “How is temperature related
to thermal energy?”, and “Identifying the Thirteen Colonies”, respectively. We
can observe that training with T-SciQ signals can significantly reduce the
number of errors. Examples of skills such as “Identifying oceans and
continents” require multi-step complex reasoning that T-SciQ teaching signals
can teach. On the other hand, examples of skills such as “Reading a map:
cardinal directions” require common sense and factual knowledge from the open
world, which T-SciQ signals can also provide.

#### Case Study.

The case study compares T-SciQ and Multimodal-CoT on the ScienceQA benchmark
(Figure 5). Figure 5(a) shows cases needing geographic knowledge. Human-
annotated CoT may lack open-world information, while T-SciQ includes it.
Figure 5(b) shows a multi-step reasoning case without image input. Multimodal-
CoT errors while our model decomposes and answers correctly. These highlight
that T-SciQ is well-suited to handle problems that require open knowledge and
decomposition.

### Comparison on Other NLP Reasoning Datasets.

Table 4: Accuracy (%) on other six reasoning datasets.

Method | Aqua | Date | Shuffled | Coin | Common | Strategy
---|---|---|---|---|---|---
| Unders. | Objects | Flip | SenseQA | QA
Reason-Teacher | 24.02 | 60.36 | 64.44 | 98.67 | 56.76 | 55.02
T-SciQ | 74.80 | 89.29 | 70.28 | 98.67 | 70.76 | 76.74

To verify the versatility of our teaching approach, we additionally assess our
approach on six reasoning tasks, following Reason-Teacher (Ho, Schmid, and Yun
2022): arithmetic (Aqua (Ling et al. 2017)), symbolic (Coin Flip (Wei et al.
2022b)), commonsense (CommonSenseQA (Talmor et al. 2018), StrategyQA (Geva et
al. 2021)) reasoning, and logic (Date Understanding, Tracking Shuffled
Objects) (Geva et al. 2021). In Table 4, we compare T-SciQ to diverse
reasoning teaching signals introduced by Reason-Teacher. The results show that
our T-SciQ surpasses Reason-Teacher by a large margin in 5 out of 6 datasets.
It performs equally well in the remaining dataset, Coin Flip. These results
indicate that higher-quality teaching signals of planning and reasoning can
lead to a remarkable improvement in small student models across different
scenarios.

## Conclusion

This paper introduces a new approach named T-SciQ that utilizes large language
models’ chain-of-thought (CoT) reasoning capabilities to teach small
multimodal models for complex science question answering tasks. Our zero-shot
prompting method generates QA-CoT samples as teaching data. We also present a
3-step zero-shot prompting approach using plan-based CoT for highly complex
problems. Furthermore, our data mixture strategy combines CoT and plan-based
CoT to create a new T-SciQ teaching dataset. Empirical evaluation on ScienceQA
shows significant improvement over previous state-of-the-art baselines. Our
method overcomes the limitations of human-annotated CoT, providing a promising
approach for complex science question answering. Future work includes
exploring extensive LLMs and parameter-efficient fine-tuning with LLM
teachers.

## References

  * Anderson et al. (2018) Anderson, P.; He, X.; Buehler, C.; Teney, D.; Johnson, M.; Gould, S.; and Zhang, L. 2018.  Bottom-up and top-down attention for image captioning and visual question answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 6077–6086.
  * Brown et al. (2020) Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33: 1877–1901.
  * Carion et al. (2020) Carion, N.; Massa, F.; Synnaeve, G.; Usunier, N.; Kirillov, A.; and Zagoruyko, S. 2020.  End-to-End Object Detection with Transformers.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I_ , 213–229.
  * Chen et al. (2020) Chen, T.; Kornblith, S.; Swersky, K.; Norouzi, M.; and Hinton, G. E. 2020.  Big self-supervised models are strong semi-supervised learners.  _Advances in neural information processing systems_ , 33: 22243–22255.
  * Chen et al. (2022) Chen, W.; Ma, X.; Wang, X.; and Cohen, W. W. 2022.  Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.  _arXiv preprint arXiv:2211.12588_.
  * Cobbe et al. (2021) Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; et al. 2021.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_.
  * Dalvi et al. (2021) Dalvi, B.; Jansen, P.; Tafjord, O.; Xie, Z.; Smith, H.; Pipatanangkura, L.; and Clark, P. 2021.  Explaining answers with entailment trees.  _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Fu et al. (2023) Fu, Y.; Peng, H.; Ou, L.; Sabharwal, A.; and Khot, T. 2023.  Specializing Smaller Language Models towards Multi-Step Reasoning.  _arXiv preprint arXiv:2301.12726_.
  * Fu et al. (2022) Fu, Y.; Peng, H.; Sabharwal, A.; Clark, P.; and Khot, T. 2022.  Complexity-based prompting for multi-step reasoning.  _arXiv preprint arXiv:2210.00720_.
  * Gao et al. (2019) Gao, P.; Jiang, Z.; You, H.; Lu, P.; Hoi, S. C.; Wang, X.; and Li, H. 2019.  Dynamic fusion with intra-and inter-modality attention flow for visual question answering.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 6639–6648.
  * Geva et al. (2021) Geva, M.; Khashabi, D.; Segal, E.; Khot, T.; Roth, D.; and Berant, J. 2021.  Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies.  _Transactions of the Association for Computational Linguistics_ , 9: 346–361.
  * He et al. (2023) He, J.; Wang, L.; Hu, Y.; Liu, N.; Liu, H.; Xu, X.; and Shen, H. T. 2023.  ICL-D3IE: In-context learning with diverse demonstrations updating for document information extraction.  _arXiv preprint arXiv:2303.05063_.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep Residual Learning for Image Recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016_ , 770–778. IEEE Computer Society.
  * Ho, Schmid, and Yun (2022) Ho, N.; Schmid, L.; and Yun, S.-Y. 2022.  Large Language Models Are Reasoning Teachers.  _arXiv preprint arXiv:2212.10071_.
  * Hsieh et al. (2023) Hsieh, C.-Y.; Li, C.-L.; Yeh, C.-K.; Nakhost, H.; Fujii, Y.; Ratner, A.; Krishna, R.; Lee, C.-Y.; and Pfister, T. 2023.  Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes.  _arXiv preprint arXiv:2305.02301_.
  * Hu et al. (2023) Hu, Z.; Lan, Y.; Wang, L.; Xu, W.; Lim, E.-P.; Lee, R. K.-W.; Bing, L.; and Poria, S. 2023.  LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models.  _arXiv preprint arXiv:2304.01933_.
  * Huang et al. (2022) Huang, J.; Gu, S. S.; Hou, L.; Wu, Y.; Wang, X.; Yu, H.; and Han, J. 2022.  Large language models can self-improve.  _arXiv preprint arXiv:2210.11610_.
  * Jansen et al. (2018) Jansen, P. A.; Wainwright, E.; Marmorstein, S.; and Morrison, C. T. 2018.  Worldtree: A corpus of explanation graphs for elementary science questions supporting multi-hop inference.  _arXiv preprint arXiv:1802.03052_.
  * Kembhavi et al. (2017) Kembhavi, A.; Seo, M.; Schwenk, D.; Choi, J.; Farhadi, A.; and Hajishirzi, H. 2017\.  Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 4999–5007.
  * Khashabi et al. (2020) Khashabi, D.; Min, S.; Khot, T.; Sabharwal, A.; Tafjord, O.; Clark, P.; and Hajishirzi, H. 2020.  Unifiedqa: Crossing format boundaries with a single qa system.  _arXiv preprint arXiv:2005.00700_.
  * Khot et al. (2022) Khot, T.; Trivedi, H.; Finlayson, M.; Fu, Y.; Richardson, K.; Clark, P.; and Sabharwal, A. 2022.  Decomposed prompting: A modular approach for solving complex tasks.  _arXiv preprint arXiv:2210.02406_.
  * Kim, Jun, and Zhang (2018) Kim, J.-H.; Jun, J.; and Zhang, B.-T. 2018.  Bilinear attention networks.  _Advances in neural information processing systems_ , 31.
  * Kim, Son, and Kim (2021) Kim, W.; Son, B.; and Kim, I. 2021.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _International Conference on Machine Learning_ , 5583–5594. PMLR.
  * Kojima et al. (2022) Kojima, T.; Gu, S. S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y. 2022.  Large language models are zero-shot reasoners.  _arXiv preprint arXiv:2205.11916_.
  * Li et al. (2022a) Li, B.; Lv, C.; Zhou, Z.; Zhou, T.; Xiao, T.; Ma, A.; and Zhu, J. 2022a.  On Vision Features in Multimodal Machine Translation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 6327–6337.
  * Li et al. (2019) Li, L. H.; Yatskar, M.; Yin, D.; Hsieh, C.-J.; and Chang, K.-W. 2019.  Visualbert: A simple and performant baseline for vision and language.  _arXiv preprint arXiv:1908.03557_.
  * Li et al. (2022b) Li, Y.; Lin, Z.; Zhang, S.; Fu, Q.; Chen, B.; Lou, J.-G.; and Chen, W. 2022b.  On the advance of making language models better reasoners.  _arXiv preprint arXiv:2206.02336_.
  * Ling et al. (2017) Ling, W.; Yogatama, D.; Dyer, C.; and Blunsom, P. 2017.  Program induction by rationale generation: Learning to solve and explain algebraic word problems.  _arXiv preprint arXiv:1705.04146_.
  * Liu et al. (2023) Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_.
  * Lu et al. (2022a) Lu, P.; Mishra, S.; Xia, T.; Qiu, L.; Chang, K.-W.; Zhu, S.-C.; Tafjord, O.; Clark, P.; and Kalyan, A. 2022a.  Learn to explain: Multimodal reasoning via thought chains for science question answering.  _Advances in Neural Information Processing Systems_ , 35: 2507–2521.
  * Lu et al. (2023) Lu, P.; Peng, B.; Cheng, H.; Galley, M.; Chang, K.-W.; Wu, Y. N.; Zhu, S.-C.; and Gao, J. 2023.  Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models.  _arXiv preprint arXiv:2304.09842_.
  * Lu et al. (2022b) Lu, P.; Qiu, L.; Chang, K.-W.; Wu, Y. N.; Zhu, S.-C.; Rajpurohit, T.; Clark, P.; and Kalyan, A. 2022b.  Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning.  _arXiv preprint arXiv:2209.14610_.
  * Lu et al. (2021) Lu, P.; Qiu, L.; Chen, J.; Xia, T.; Zhao, Y.; Zhang, W.; Yu, Z.; Liang, X.; and Zhu, S.-C. 2021.  Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning.  _arXiv preprint arXiv:2110.13214_.
  * Magister et al. (2022) Magister, L. C.; Mallinson, J.; Adamek, J.; Malmi, E.; and Severyn, A. 2022.  Teaching small language models to reason.  _arXiv preprint arXiv:2212.08410_.
  * Nye et al. (2021) Nye, M.; Andreassen, A. J.; Gur-Ari, G.; Michalewski, H.; Austin, J.; Bieber, D.; Dohan, D.; Lewkowycz, A.; Bosma, M.; Luan, D.; et al. 2021.  Show your work: Scratchpads for intermediate computation with language models.  _arXiv preprint arXiv:2112.00114_.
  * OpenAI (2022) OpenAI. 2022.  Introducing chatgpt.  https://openai.com/blog/chatgpt.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  _CoRR_ , abs/2303.08774.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , 8748–8763. PMLR.
  * Rubin, Herzig, and Berant (2021) Rubin, O.; Herzig, J.; and Berant, J. 2021.  Learning to retrieve prompts for in-context learning.  _arXiv preprint arXiv:2112.08633_.
  * Sampat, Yang, and Baral (2020) Sampat, S. K.; Yang, Y.; and Baral, C. 2020.  Visuo-Lingustic Question Answering (VLQA) Challenge.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings (EMNLP)_ , 4606–4616.
  * Talmor et al. (2018) Talmor, A.; Herzig, J.; Lourie, N.; and Berant, J. 2018.  Commonsenseqa: A question answering challenge targeting commonsense knowledge.  _arXiv preprint arXiv:1811.00937_.
  * Thoppilan et al. (2022) Thoppilan, R.; De Freitas, D.; Hall, J.; Shazeer, N.; Kulshreshtha, A.; Cheng, H.-T.; Jin, A.; Bos, T.; Baker, L.; Du, Y.; et al. 2022.  Lamda: Language models for dialog applications.  _arXiv preprint arXiv:2201.08239_.
  * Tian et al. (2023) Tian, Q.; Zhu, H.; Wang, L.; Li, Y.; and Lan, Y. 2023.  R3 Prompting: Review, Rephrase and Resolve for Chain-of-Thought Reasoning in Large Language Models under Noisy Context.  _arXiv preprint arXiv:2310.16535_.
  * Touvron et al. (2023) Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017.  Attention is All you Need.  In _Advances in Neural Information Processing Systems 30_ , 5998–6008.
  * Wang, Deng, and Sun (2022) Wang, B.; Deng, X.; and Sun, H. 2022.  Iteratively prompt pre-trained language models for chain of thought.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , 2714–2730.
  * Wang et al. (2023) Wang, L.; Xu, W.; Lan, Y.; Hu, Z.; Lan, Y.; Lee, R. K.-W.; and Lim, E.-P. 2023.  Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models.  _arXiv preprint arXiv:2305.04091_.
  * Wang et al. (2022a) Wang, X.; Wei, J.; Schuurmans, D.; Le, Q.; Chi, E.; and Zhou, D. 2022a.  Rationale-augmented ensembles in language models.  _arXiv preprint arXiv:2207.00747_.
  * Wang et al. (2022b) Wang, X.; Wei, J.; Schuurmans, D.; Le, Q.; Chi, E.; and Zhou, D. 2022b.  Self-consistency improves chain of thought reasoning in language models.  _arXiv preprint arXiv:2203.11171_.
  * Wei et al. (2022a) Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Chi, E.; Le, Q.; and Zhou, D. 2022a.  Chain of Thought Prompting Elicits Reasoning in Large Language Models.  _ArXiv preprint_ , abs/2201.11903.
  * Wei et al. (2022b) Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Chi, E.; Le, Q.; and Zhou, D. 2022b.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_.
  * Yu et al. (2019) Yu, Z.; Yu, J.; Cui, Y.; Tao, D.; and Tian, Q. 2019.  Deep modular co-attention networks for visual question answering.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 6281–6290.
  * Zhang et al. (2023a) Zhang, R.; Han, J.; Zhou, A.; Hu, X.; Yan, S.; Lu, P.; Li, H.; Gao, P.; and Qiao, Y. 2023a.  Llama-adapter: Efficient fine-tuning of language models with zero-init attention.  _arXiv preprint arXiv:2303.16199_.
  * Zhang et al. (2022) Zhang, Z.; Zhang, A.; Li, M.; and Smola, A. 2022.  Automatic chain of thought prompting in large language models.  _arXiv preprint arXiv:2210.03493_.
  * Zhang et al. (2023b) Zhang, Z.; Zhang, A.; Li, M.; Zhao, H.; Karypis, G.; and Smola, A. 2023b.  Multimodal chain-of-thought reasoning in language models.  _arXiv preprint arXiv:2302.00923_.
  * Zhou et al. (2022) Zhou, D.; Schärli, N.; Hou, L.; Wei, J.; Scales, N.; Wang, X.; Schuurmans, D.; Bousquet, O.; Le, Q.; and Chi, E. 2022.  Least-to-most prompting enables complex reasoning in large language models.  _arXiv preprint arXiv:2205.10625_.

##  Appendix A More Detailed Analysis

In addition to exploring the effects of different visual features, we also
tested the performance of different backbones. Table 5 shows the results of
UnifiedQA and FLAN-T5 on our generated datasets. When using various backbone
networks, the model trained by QA-CoT data is the worst among the three
generated data types. However, it is also better than the manually annotated
data, which indicates that the manually annotated data has certain
limitations, such as redundant information, single style, etc. Furthermore,
when using the Mixture dataset of QA-CoT and QA-PCoT, the fact that all four
backbones achieved the best performances illustrates the effectiveness and
generality of our strategy.

Table 5: Accuracy (%) of using different backbones.

Method | Size | Annotated CoT | T-SciQ
---|---|---|---
QA-CoT | QA-PCoT | Mixture
UnifiedQA${}_{\texttt{Base}}$ | 223M | 84.91 | 85.99 | 88.56 | 91.75
UnifiedQA${}_{\texttt{Large}}$ | 738M | 91.68 | 93.44 | 93.54 | 96.18
FLAN-T5${}_{\texttt{Base}}$ | 248M | 85.85 | 86.87 | 89.04 | 92.33
FLAN-T5${}_{\texttt{Large}}$ | 783M | 93.02 | 93.54 | 95.68 | 96.49

##  Appendix B More Cases Analysis

To investigate the impact of different teaching signals, we conducted an
evaluation of model predictions trained on various types of data, including
both generated teaching data and manually annotated data. In particular, we
compared the test examples produced by the model trained with manually
annotated data (MM-CoT) and the QA-CoT teaching data (T-SciQ (QA-CoT)) in
Figure 6. We also displayed prediction cases of models trained on manually
annotated data (MM-CoT) and QA-PCoT teaching data (T-SciQ (QA-PCoT)) in Figure
7. Furthermore, we compared the prediction cases of models trained using our
proposed teaching data, QA-CoT and QA-PCoT, as depicted in Figure 8 and Figure
9. From the figures, it is evident that the model trained with QA-PCoT data
performs better in solving multi-step reasoning problems, while the one
trained with QA-CoT is more proficient at solving straightforward problems.

(a)  There is some redundancy in manually annotating data.

(b)  The QA-CoT data can be used to solve problems with commonsense knowledge.

Figure 6:  Examples of MM-CoT (baseline) and T-SciQ (QA-CoT) (ours) for
generating rationales and predicting answers.

(a)  T-SciQ (QA-PCoT) can solve problems by executing the plan in multiple
steps.

(b)  The QA-PCoT data can be used to solve problems with commonsense
knowledge.

Figure 7:  Examples of MM-CoT (baseline) and T-SciQ (QA-PCoT) (ours) for
generating rationales and predicting answers.

(a)  T-SciQ (QA-PCoT) can solve problems by executing the plan in multiple
steps.

(b)  T-SciQ (QA-PCoT) can solve problems by executing the plan in multiple
steps.

Figure 8:  Examples of T-SciQ (QA-CoT) (ours) and T-SciQ (QA-PCoT) (ours) for
generating rationales and predicting answers.

(a)  T-SciQ (QA-PCoT) may generate error rationale when visual features are
complicated.

(b)  T-SciQ (QA-CoT) is better than T-SciQ (QA-PCoT) on some simple problems.

Figure 9:  Examples of T-SciQ (QA-CoT) (ours) and T-SciQ (QA-PCoT) (ours) for
generating rationales and predicting answers.

##  Appendix C Data Generation Process of PCoT

In this section, we several show cases of how to generate PCoT teaching
signals. To obtain appropriate planning-based chain-of-thought rationale, we
introduce a 3-step zero-shot prompting approach that enables Language Models
to decompose complex problems into simpler subproblems that are easier to
solve.

The lecture template used to generate a lecture for a particular skill is
formulated as follows: “Skill: [$S$]. QA pairs: [$X_{q},A$] … [$Instruct$].”
In this prompt, [$Instruct$] is as follows: “based on the problems above,
please give a general lecture on the [$S$] type of question in one sentence.”.
The lecture generated for this skill will be used in the second step of
prompting, facilitating the LLM’s generation of a plan for this skill.
Examples of lecture generation are shown in Table 7.

The template used to generate a plan for a specific skill based on a lecture
is formulated as follows: “Skill: [$S$]. Lecture: [$L$]. QA pairs: [$X_{q},A$]
… [$Instruct$].”. In this prompt, [$Instruct$] is written as follows: “Based
on the lecture above and these problems, let’s understand these problems and
devise a general and brief plan step by step to solve these problems (begin
with 1, 2, 3…)”. The plan generated using this template will be utilized in
the third stage of prompting to assist the LLM in constructing a plan-based
chain-of-thought rationale for each training example. Examples are shown in
Table 8.

The lecture and plan generated by the first two prompts are used to generate a
plan-based chain-of-thought rationale for each training example. The rationale
generation template is formulated as follows: “Skill: [$S$]. Lecture: [$L$].
Plan: [$P$]. QA pairs: [$X_{q},A$] … [$Instruct$].”. In this prompt,
[$Instruct$] is written as follows: “Based on the lecture, the plan and the
problem, please carry out the plan and solve the problem step by step (begin
with 1, 2, 3…)”. This prompting method provides a strategy for problem-solving
by leveraging the expertise of the lecture and plan, and utilizing a
structured reasoning framework to solve each problem. Examples are shown in
Table 9.

Table 6: Data generation process of CoT prompting. Input: Which of these
states is farthest north?
---
Context: N/A
Options: (A) West Virginia (B) Louisiana (C) Arizona (D) Oklahoma
Correct Answer: (A) West Virginia
Please give a detailed explanation.
Solution: West Virginia is the farthest north of the four states listed. West
Virginia is located in the Appalachian region of the United States, which is
in the northeastern part of the country. Louisiana, Arizona, and Oklahoma are
all located in the southern and southwestern parts of the United States. West
Virginia is the northernmost of the four states, making it the farthest north.
Question: Identify the question that Tom and Justin’s experiment can best
answer.
Context: The passage below describes an experiment. … They compared the
distances the balls traveled when launched from a 45° angle to the distances
the balls traveled when launched from a 30° angle. Figure: a catapult for
launching ping pong balls.
Options: (A) Do ping pong balls stop rolling along the ground sooner after
being launched from a 30° angle or a 45° angle? (B) Do ping pong balls travel
farther when launched from a 30° angle compared to a 45° angle?
Correct Answer: (B) Do ping pong balls travel farther when launched from a 30°
angle compared to a 45° angle?
Please give a detailed explanation.
Solution: The question that Tom and Justin’s experiment can best answer is (B)
Do ping pong balls travel farther when launched from a 30° angle compared to a
45° angle? This is because the experiment is designed to compare the distances
the balls traveled when launched from a 45° angle to the distances the balls
traveled when launched from a 30° angle. By measuring the distances between
the catapult and the place where the ball hit the ground, Tom and Justin can
determine which angle produces the greatest distance.
Question: What does the verbal irony in this text suggest? According to Mr.
Herrera’s kids, his snoring is as quiet as a jackhammer.
Context: N/A
Options: (A) The snoring is loud. (B) The snoring occurs in bursts.
Correct Answer: (A) The snoring is loud.
Please give a detailed explanation.
Solution: Verbal irony is when someone says something that is the opposite of
what they mean. In this text, Mr. Herrera’s kids are saying that his snoring
is as quiet as a jackhammer, which implies that it is actually very loud. This
suggests that the snoring is loud, which is option (A).
Question: Which of the following could Wendy’s test show?
Context: People can use the engineering-design process to develop solutions to
problems. One step in the process is testing if a potential solution meets the
requirements of the design. … She read that giving the bacteria more nutrients
could affect the amount of insulin they produced. So, Wendy gave extra
nutrients to some of the bacteria. Then, she measured how much insulin those
bacteria produced compared to bacteria that did not get extra nutrients.
Figure: studying bacteria in a laboratory.
Options: (A) whether producing more insulin would help the bacteria grow
faster (B) whether different types of bacteria would need different nutrients
to produce insulin (C) whether she added enough nutrients to help the bacteria
produce 20% more insulin
Correct Answer: (C) whether she added enough nutrients to help the bacteria
produce 20% more insulin
Please give a detailed explanation.
Solution: The passage describes how Wendy used the engineering-design process
to test a solution to a problem. She wanted to increase the amount of insulin
that the bacteria produced by 20%, so she gave extra nutrients to some of the
bacteria and then measured how much insulin those bacteria produced compared
to bacteria that did not get extra nutrients. This testing would show whether
she added enough nutrients to help the bacteria produce 20% more insulin.
Option A is incorrect because the passage does not mention anything about the
bacteria growing faster. Option B is incorrect because the passage does not
mention anything about different types of bacteria needing different
nutrients.
Question: Does this passage describe the weather or the climate?
Context: Figure: Chicago. Chicago is known as The Windy City. But on average,
the wind there only blows at about 10 miles per hour. Hint: Weather is what
the atmosphere is like at a certain place and time. Climate is the pattern of
weather in a certain place.
Options: (A) weather (B) climate
Correct Answer: (B) climate
Please give a detailed explanation.
Solution: The passage describes the average wind speed in Chicago, which is a
pattern of weather in the city. This pattern of weather is part of the climate
of Chicago, so the correct answer is (B) climate. Climate is the average
weather conditions in a certain area over a long period of time. It is
determined by factors such as temperature, precipitation, humidity, and wind
speed. Weather, on the other hand, is the atmospheric conditions at a certain
place and time. It is determined by the same factors as climate, but it is
more localized and can change quickly.
Table 7: The process of generating lectures. Here are some problems about
”Read a map: cardinal directions”
---
Question: Which of these states is farthest north?
Context: N/A
Options: (A) West Virginia (B) Louisiana (C) Arizona (D) Oklahoma
Question: Which of these states is farthest north?
Context: N/A
Options: (A) Alabama (B) Idaho (C) South Carolina (D) Oklahoma
Question: Which of these states is farthest north?
Context: N/A
Options: (A) Maine (B) South Carolina (C) Kansas (D) Delaware
Question: Which of these states is farthest east?
Context: N/A
Options: (A) Montana (B) New Mexico (C) Arkansas (D) Nebraska
Question: Which of these states is farthest north?
Context: N/A
Options: (A) West Virginia (B) Mississippi (C) Nebraska (D) Oklahoma
Based on the problems above, please give a general lecture on the ”Read a map:
cardinal directions” type of question in one sentence.
Lecture: Reading a map to identify cardinal directions involves determining
which direction a given location is in relation to other locations on the map.
Here are some problems about ”Interpret figures of speech”
Question: What does the verbal irony in this text suggest?According to Mr.
Herrera’s kids, his snoring is as quiet as a jackhammer.
Context: N/A
Options: (A) The snoring is loud. (B) The snoring occurs in bursts.
Question: What does the metaphor in this text suggest? All religions, arts,
and sciences are branches of the same tree. 2̆014Albert Einstein
Context: N/A
Options: (A) All religions, arts, and sciences are distant from one another.
(B) All religions, arts, and sciences are related.
Question: What does the verbal irony in this text suggest? This is the best
day of my life, Mr. Hogan mumbled after his car broke down on the way to an
important job interview.
Context: N/A
Options: (A) Mr. Hogan was having a bad day. (B) Mr. Hogan liked fixing cars.
Question: What does the allusion in this text suggest?Tyler seems to have the
Midas touch. Without any special experience or training, he launched a
thriving business and then established a well-respected charity.
Context: N/A
Options: (A) Tyler is successful at all that he does. (B) Tyler has a hands-on
approach to his work.
Question: What does the idiom in this text suggest? Speak of the devil!
Whitney declared when Charlie strolled into the room.
Context: N/A
Options: (A) Whitney had just been speaking about Charlie. (B) Whitney didn’t
trust Charlie.
Based on the problems above, please give a general lecture on the ”Interpret
figures of speech” type of question in one sentence.
Lecture: Interpret figures of speech questions require the reader to identify
the figurative language used in a text and determine the meaning or
connotation it conveys.
Here are some problems about ”Animal adaptations: beaks, mouths, and necks”
Question: Which animal’s mouth is also adapted for bottom feeding?
Context: Sturgeons eat invertebrates, plants, and small fish. They are bottom
feeders. Bottom feeders find their food at the bottom of rivers, lakes, and
the ocean. The ’s mouth is located on the underside of its head and points
downward. Its mouth is adapted for bottom feeding. Figure: sturgeon.
Options: (A) discus (B) armored catfish
Question: Which bird’s beak is also adapted to tear through meat?
Context: Red-tailed hawks eat fish, mammals, and other birds. The shape of the
’s beak is adapted to tear through meat. Figure: red-tailed hawk.
Options: (A) sand martin (B) Cape vulture
Question: Which animal’s mouth is also adapted for bottom feeding?
Context: Armored catfish eat plants and small
invertebrates. They are bottom feeders. Bottom feeders find their food at the
bottom of rivers, lakes, and the ocean. The catfish’s mouth is located on the
underside of its head and points downward. Its mouth is adapted for bottom
feeding. Figure: armored catfish.
Options: (A) clown triggerfish (B) sturgeon
Question: Which bird’s beak is also adapted to get nectar out of long flowers?
Context: Green violetears live in the forests of Central and South America.
The shape of the ’s beak is adapted to get nectar out of long flowers. Figure:
green violetear.
Options: (A) ground hornbill (B) violet sabrewing
Question: …
Based on the problems above, please give a general lecture on the ”Animal
adaptations: beaks, mouths, and necks” type of question in one sentence.
Lecture: Animal adaptations such as beaks, mouths, and necks are specialized
features that enable animals to feed on different types of food, such as meat,
insects, nuts, and plant matter.
Table 8: The process of generating plans. Here are some problems about ”Read a
map: cardinal directions”
---
The lecture about ”Read a map: cardinal directions” is ”Reading a map to
identify cardinal directions involves determining which direction a given
location is in relation to other locations on the map.”
Question: Which of these states is farthest north?
Context: N/A
Options: (A) West Virginia (B) Louisiana (C) Arizona (D) Oklahoma
Question: Which of these states is farthest north?
Context: N/A
Options: (A) Alabama (B) Idaho (C) South Carolina (D) Oklahoma
Question: Which of these states is farthest north?
Context: N/A
Options: (A) Maine (B) South Carolina (C) Kansas (D) Delaware
Question: …
Based on the lecture above and these problems, let’s understand these problems
and devise a general and brief plan step by step to solve these problems
(begin with 1, 2, 3…).
Plan: 1\. Read the question carefully and identify the context. 2. Identify
the cardinal directions (north, south, east, and west) in relation to the
given locations on the map. 3. Compare the locations and determine which one
is farthest in the given direction. 4. Select the correct answer from the
given options.
Here are some problems about ”Interpret figures of speech”
The lecture about ”Interpret figures of speech” is ”Interpret figures of
speech questions require the reader to identify the figurative language used
in a text and determine the meaning or connotation it conveys.”
Question: What does the verbal irony in this text suggest?According to Mr.
Herrera’s kids, his snoring is as quiet as a jackhammer.
Context: N/A
Options: (A) The snoring is loud. (B) The snoring occurs in bursts.
Question: What does the metaphor in this text suggest? All religions, arts,
and sciences are branches of the same tree. 2̆014Albert Einstein
Context: N/A
Options: (A) All religions, arts, and sciences are distant from one another.
(B) All religions, arts, and sciences are related.
Question: What does the verbal irony in this text suggest? This is the best
day of my life, Mr. Hogan mumbled after his car broke down on the way to an
important job interview.
Context: N/A
Options: (A) Mr. Hogan was having a bad day. (B) Mr. Hogan liked fixing cars.
Question: …
Based on the lecture above and these problems, let’s understand these problems
and devise a general and brief plan step by step to solve these problems
(begin with 1, 2, 3…).
Plan: 1\. Read the text carefully and identify the figure of speech used. 2.
Analyze the context of the text to determine the meaning or connotation of the
figure of speech. 3. Choose the option that best matches the meaning or
connotation of the figure of speech.
Here are some problems about ”Animal adaptations: beaks, mouths, and necks”
The lecture about ”Animal adaptations: beaks, mouths, and necks” is ”Animal
adaptations such as beaks, mouths, and necks are specialized features that
enable animals to feed on different types of food, such as meat, insects,
nuts, and plant matter.”
Question: Which animal’s mouth is also adapted for bottom feeding?
Context: Sturgeons eat invertebrates, plants, and small fish. They are bottom
feeders. Bottom feeders find their food at the bottom of rivers, lakes, and
the ocean. The ’s mouth is located on the underside of its head and points
downward. Its mouth is adapted for bottom feeding. Figure: sturgeon.
Options: (A) discus (B) armored catfish
Question: Which bird’s beak is also adapted to tear through meat?
Context: Red-tailed hawks eat fish, mammals, and other birds. The shape of the
’s beak is adapted to tear through meat. Figure: red-tailed hawk.
Options: (A) sand martin (B) Cape vulture
Question: Which animal’s mouth is also adapted for bottom feeding?
Context: Armored catfish eat plants and small
invertebrates. They are bottom feeders. Bottom feeders find their food at the
bottom of rivers, lakes, and the ocean. The catfish’s mouth is located on the
underside of its head and points downward. Its mouth is adapted for bottom
feeding. Figure: armored catfish.
Options: (A) clown triggerfish (B) sturgeon
Question: …
Based on the lecture above and these problems, let’s understand these problems
and devise a general and brief plan step by step to solve these problems
(begin with 1, 2, 3…).
Plan: 1\. Read the lecture and understand the context of the questions. 2.
Read the question and identify the type of adaptation being asked about. 3.
Look at the figure provided and identify the animal being discussed. 4.
Research the animal to find out what type of adaptation it has. 5. Compare the
options provided and select the one that best matches the adaptation.
Table 9: Data generation of PCoT prompting. Here are is a problem about ”Read
a map: cardinal directions”
---
The lecture about ”Read a map: cardinal directions” is ”Reading a map to
identify cardinal directions involves determining which direction a given
location is in relation to other locations on the map.”
The plan to solve ”Read a map: cardinal directions” problem is ”1. Read the
question carefully and identify the context. 2. Identify the cardinal
directions (north, south, east, and west) in relation to the given locations
on the map. 3. Compare the locations and determine which one is farthest in
the given direction. 4. Select the correct answer from the given options.”
Question: Which of these states is farthest north?
Context: N/A
Options: (A) West Virginia (B) Louisiana (C) Arizona (D) Oklahoma
Correct Answer: (A) West Virginia
Based on the lecture, the plan and the problem, please carry out the plan and
solve the problem step by step (begin with 1, 2, 3…).
Solution: 1\. Read the question carefully and identify the context. The
context of the question is to determine which of the given states is farthest
north. 2. Identify the cardinal directions (north, south, east, and west) in
relation to the given locations on the map. North is the direction that is
farthest away from the equator and closest to the North Pole. South is the
direction that is farthest away from the North Pole and closest to the
equator. East is the direction that is farthest away from the Prime Meridian
and closest to the International Date Line. West is the direction that is
farthest away from the International Date Line and closest to the Prime
Meridian. 3. Compare the locations and determine which one is farthest in the
given direction. West Virginia is farthest north, followed by Louisiana,
Arizona, and Oklahoma. 4. Select the correct answer from the given options.
The correct answer is (A) West Virginia.
Here is a problem about ”Interpret figures of speech”
The lecture about ”Interpret figures of speech” is ”Interpret figures of
speech questions require the reader to identify the figurative language used
in a text and determine the meaning or connotation it conveys.”
The plan to solve ”Interpret figures of speech” problem is ”1. Read the text
carefully and identify the figure of speech used. 2. Analyze the context of
the text to determine the meaning or connotation of the figure of speech. 3.
Choose the option that best matches the meaning or connotation of the figure
of speech.”
Question: What does the verbal irony in this text suggest?According to Mr.
Herrera’s kids, his snoring is as quiet as a jackhammer.
Context: N/A
Options: (A) The snoring is loud. (B) The snoring occurs in bursts.
Correct Answer: (A) The snoring is loud.
Based on the lecture, the plan and the problem, please carry out the plan and
solve the problem step by step (begin with 1, 2, 3…).
Solution: 1\. Read the text carefully and identify the figure of speech used.
The figure of speech used in the text is verbal irony. 2. Analyze the context
of the text to determine the meaning or connotation of the figure of speech.
The context of the text suggests that the verbal irony is used to convey the
idea that Mr. Herrera’s snoring is actually very loud, despite what his kids
say. 3. Choose the option that best matches the meaning or connotation of the
figure of speech. The option that best matches the meaning or connotation of
the figure of speech is (A) The snoring is loud.
Here is a problem about ”Animal adaptations: beaks, mouths, and necks”
The lecture about ”Animal adaptations: beaks, mouths, and necks” is ”Animal
adaptations such as beaks, mouths, and necks are specialized features that
enable animals to feed on different types of food, such as meat, insects,
nuts, and plant matter.”
The plan to solve ”Animal adaptations: beaks, mouths, and necks” problem is
”1. Read the lecture and understand the context of the questions. 2. Read the
question and identify the type of adaptation being asked about. 3. Look at the
figure provided and identify the animal being discussed. 4. Research the
animal to find out what type of adaptation it has. 5. Compare the options
provided and select the one that best matches the adaptation.”
Question: Which animal’s mouth is also adapted for bottom feeding?
Context: Sturgeons eat invertebrates, plants, and small fish. They are bottom
feeders. Bottom feeders find their food at the bottom of rivers, lakes, and
the ocean. The ’s mouth is located on the underside of its head and points
downward. Its mouth is adapted for bottom feeding. Figure: sturgeon.
Options: (A) discus (B) armored catfish
Correct Answer: (B) armored catfish
Based on the lecture, the plan and the problem, please carry out the plan and
solve the problem step by step (begin with 1, 2, 3…).
Solution: 1\. Read the lecture and understand the context of the questions.
The lecture states that animal adaptations such as beaks, mouths, and necks
are specialized features that enable animals to feed on different types of
food. 2. Read the question and identify the type of adaptation being asked
about. The question is asking about an adaptation related to bottom feeding.
3. Look at the figure provided and identify the animal being discussed. The
figure provided is a sturgeon. 4. Research the animal to find out what type of
adaptation it has. Researching the sturgeon reveals that its mouth is located
on the underside of its head and points downward. This adaptation is
specifically adapted for bottom feeding. 5. Compare the options provided and
select the one that best matches the adaptation. The options provided are (A)
discus and (B) armored catfish. The armored catfish has a mouth that is
adapted for bottom feeding, making it the correct answer. Therefore, the
correct answer is (B) armored catfish.


# Machine Learning on Electronic Health Records: Models and Features Usages to
predict Medication Non-Adherence

Janssoone T Corresponding author<EMAIL_ADDRESS>Semeia, Paris, France
Bic C  Semeia, Paris, France  Kanoun D  Institut Pasteur, Paris, France
Rinder P  Semeia, Paris, France  Hornus P  Semeia, Paris, France

###### Abstract

Adherence can be defined as "the extent to which patients take their
medications as prescribed by their healthcare providers"(Osterberg and
Blaschke, 2005). World Health Organization’s
reports111http://www.who.int/chp/knowledge/publications/adherence_full_report.pdf
point out that, in developed countries, only about 50% of patients with
chronic diseases correctly follow their treatments. This severely compromises
the efficiency of long-term therapy and increases the cost of health services.
We propose in this paper different models of patient drug consumption in
breast cancer treatments. The aim of these different approaches is to predict
medication non-adherence while giving insights to doctors of the underlying
reasons of these illegitimate drop-outs. Working with oncologists, we show the
interest of Machine-Learning algorithms fined tune by the feedback of experts
to estimate a risk score of a patient’s non-adherence and thus improve support
throughout their care path.

##  1 Introduction

Fallowfield et al. (2005); O’neill and Twelves (2002) shows that patient-
administered oral medications have become more and more prevalent, increasing
the focus on adherence. A common solution is to set patient support programs
that include, for example, 1) providing patients with information and advice
2) support and coaching sessions delivered by nurses (by phone or face-to-
face), and 3) sending information to health professionals treating the
patient. These programs have been shown to be effective (Krolop et al., 2013;
Spoelstra et al., 2013). Yet, two main limitations are restraining these
interventions: 1) The use of human intervention is effective but very
expensive thus limiting their reach, 2) The use of digital technologies (e.g.
notifications and explanations) is too generic and sometimes too intrusive
(e.g. daily reminders) which leads to patients losing interest. We propose to
optimize the use of human intervention by giving an accurate evaluation of a
patient illegitimate drug drop-out. We use of machine-learning techniques on
breast cancer patients’ consumption data to evaluate this risk and fine-tune
them with insights from oncologists. These predictive models are trained on
the reimbursement data of the French Health Insurance (SNIIRAM - the French
National Health System). The goal is to categorize patients into risk classes
according to their characteristics. The long-term goal is to know the most
appropriate moments to contact them for support. Thus, people will benefit
from support adapted to their profile and their needs, and human interventions
will be reserved for the situations for which they are really necessary. This
paper first reviews previous approaches before introducing the SNIIIRAM
database. Then, the different data processing and the applied models are
described and discussed.

##  2 Related work

The phenomenon of non-adherence has been widely surveyed by medical
researchers and psychologists. For instance, DiMatteo (2004b) and Mann et al.
(2010) review many observation-based scientific publications and provide an
interesting quantitative assessment of the research conducted on the subject.
These meta-analysis give us insights about factors of non-adherence. Hence,
they underline the influence of the increasing age of patients, the treatment
complexity level (multiple drugs, injections, …) or the impact of patients’
mental health ( depressive episodes have a very negative impact on the
patient’s compliance with the prescriptions of health professionals (DiMatteo
et al., 2000)) as well as low education and, more importantly, low income
(DiMatteo, 2004a). However, Franklin et al. (2016) underline the difficulty to
use this information to predict adherence: they evaluate different approaches,
using logistic regression and boosted logistic regression, to define three
categories of adherence predictors. Hence, they show that using census
information or transaction data leads to poor prediction. However, they point
out that using adherence observations during the first month significantly
increases the accuracy of the results. This nuance on the weight of each
adherence prediction variable is confirmed in Lo-Ciganic et al. (2015). They
use random survival forests highlights to find patient specific adherence
thresholds to discriminate between hospitalization risks. Here again, the
major variables are linked to patient history and previous transactions.
Karanasiou et al. (2016) explore Machine Learning approaches to estimate the
risk of the non-adherence for a patient with Heart Failure. The database
contains general information about the patient, details about each care path,
clinical examinations and clean labels based on clinician estimations. They
prove the feasibility of this approach using a clean dedicated dataset. This
paper explores these solutions to predict the risk of an illegitimate stop
during a treatment. Our models are trained using real reimbursement records
data from the French Health System. Our goal is to explore how to use these
real, indirect and unlabeled observations to evaluate the risk of non-
adherence at key moments of a patient’s care path.

##  3 The SNIIRAM database

SNIIRAM222https://www.ameli.fr/l-assurance-maladie/statistiques-et-
publications/sniiram/finalites-du-sniiram.php is one of the largest structured
databases of health data in the world. It contains reimbursement data of the
French Health System, covering 99.8% of the French population ($\simeq$ 66
million persons). Useful data are, for example, hospitalizations, drug
purchases or contextual patient information (age, government services,
geographic information, …). The advantage of using this database is that it is
very well structured and has no-bias in term of social background due to its
universal coverage. More details can be found in Tuppin et al. (2010).
Previous work has already shown the value of massive data mining to aid
diagnosis, either by taking all the information for a "static" approach
Neumann et al. (2012), or, more recently, by also incorporating dynamic
information Morel et al. (2017). Other studies have been conducted on the
determinants of compliance, particularly for breast cancer. Our study focuses
on women’s breast cancer on part of the SNIIRAM data. The cohort of the study
consists of 50$\%$ of women (drawn randomly) who met the following criteria:
1- diagnosed with breast cancer; 2- having purchased at least one of the
following molecules for the studied period: _Anastrozole_ , _Capecitabine_ ,
_Cyclophosphamide_ , _Etoposide_ , _Everolimus_ , _Exemestane_ , _Lapatinib_ ,
_Letrozole_ , _Megestrol_ , _Melphalan_ , _Tamoxifen_ , _Toremifene_ and
_Vinorelbine_. Extraction concerns consumption between 2013 and 2015 and is
made up of three main categories: 1-Pharmacy transactions (molecule, number of
doses, date, …); 2- Hospitalizations (diagnosis, start date, end date, …);
3-Patient information (age, department, date of the diagnostic of eventual
long-term illness (referred as _ALD_) , pathologies, …)

##  4 Phase analysis study

Working with oncologists, we reworked the raw data to show the different
phases of the treatment. A phase is defined as a period of continuous intake
of a molecule or hospitalization (chemotherapy or radiotherapy), allowing the
reconstruction of a patient’s care path. A phase of treatment can end with a
legitimate stop (i.e. death, change of treatment, a serious cardiac issue,
palliative care or right censorship due to data extraction) if this event
occurs less than two months after the date of the last theoretical dose. The
date of the last theoretical dose is obtained by calculating the median
interval between two purchases of the molecule or two hospitalizations of the
same type: this median behavior is considered to conform with the drug dose.
Thus, the median time is 30 days between two box purchases of 30 doses of
Tamoxifen. The end of this period after the last purchased box corresponds to
the date of the last theoretical intake. Illegitimate stops are considered if
none of the legitimate stops occurred before the date of the last theoretical
dose. For each phase, the following data is calculated: 1- Start and end
dates, number of intakes or hospitalizations, molecule or type of
hospitalization; 2- End of treatment type (switch, death, stop, right
censorship); 3-Patient information (comorbidities, location, age, …); 4-
Interventions on the breast (mastectomy) during the three months before the
studied phase. This study focuses on predicting an illegitimate stop occurring
after a specific time length (3, 6 and 12 months) from the beginning of the
phase. For each phase, for each time period, a label indicates if an
illegitimate stop occurred or if the care path continues correctly (no stop or
a legitimate one). The data are then composed of, for the 3 months period,
51220 patients with 11.16$\%$ non-adherent, for 6 months, 44469 patients with
16.53$\%$ non-adherent, for 12 months, 34132 patients with 27.0$\%$ non-
adherent.
We compare the ability of supervised learning algorithms to predict a
legitimate stop from an illegitimate one with the information available at the
beginning of the phase. The first model is a logistic regression, trained with
a restricted number of variables according to their p-value ($<0.05$).
Logistic regression assigns a coefficient to each variable related to its
influence for the classification task, making it easy to interpret. The second
model is a decision tree trained with all variables in the dataset. Parameters
were tuned with gridsearch. Prediction function of a decision tree is easy to
read and understand. The third model is gradient boosting which, compared to
the two previous models, is harder to interpret to interpret. However, it
focuses on weak learners and can offer a better prediction. This model used
all variables in the dataset and parameters were tuned with grid search. The
last model is a multilayer perceptron (MLP). Training sets are balanced to get
an equal number of legitimate/illegitimate cases. We only use one hidden layer
as balanced dataset have quite a small size (9412 rows for the 3 months
period). We also kept all variables to compute the model. 5 folders cross-
validation was performed which results are shown in Table1. AUCs are roughly
$0.70$ with Gradient Boosting having the best performances. A Cumulative
Accuracy Profit curve (CAP or Lorenz curve) gives the second and third
measures, indicating the ability of a model to accurately spot a patient at-
risk. CAP $n\%$ is the rate of true positive classifications looking at the
$n$ highest ranked predicted risk. Gradient boosting and MLP have the best
results but these "black box methods" are difficult to explain which might
refrain their usages. These first approaches could already be applied: the
caregivers could be notified during the first medical visit and propose more
support to the patient at risk. These risk models can also be used to
underline the period with more drop-outs according to a patient’s profile and
trigger an alarm to reach them if needed.

3 months 6 months 12 months AUC CAP 20$\%$ CAP 40$\%$ AUC CAP 20$\%$ CAP
40$\%$ AUC CAP 20$\%$ CAP 40$\%$ Logistic regression 0.71 0.52 0.72 0.71 0.52
0.71 0.72 0.54 0.71 Decision tree 0.68 0.47 0.68 0.69 0.47 0.68 0.70 0.50 0.69
Gradient boosting 0.74 0.54 0.75 0.74 0.55 0.74 0.74 0.57 0.75 Multilayer
perceptron 0.71 0.52 0.72 0.71 0.53 0.72 0.72 0.55 0.71

Table 1: AUCs and CAP results of the algorithms ability to predict an
illegitimate stops in the n months after the beginning of a phase

##  5 Drug transaction study

A survival analysis, described in Janssoone et al. (2018), and a statistical
analysis of the pharmacy transactions show that illegitimate drop out rate
evolves over time and according to previous events in the patient’s care path.
This led to a more patient-centered study focusing on all transactions carried
out by a patient: every purchase in a pharmacy, every hospitalization,…For
each pharmacy transaction of a drug used for treating breast cancer
(Tamoxifen, Exemestane,…), the model predicts if the patient will continue her
treatment correctly or if an illegitimate stop might occur. Same criteria used
in the phases study are used to label pharmacy transactions. Data can,
therefore, be considered as a sequence of events (hospitalizations and
pharmacy transactions) with timestamps and a set of characteristics about the
patient. We can divide data in two categories : _"static"_ data describing the
patient (geographic and sociological information, age, …), and _dynamic_ data
about the current care-path( last drug purchases (cancer linked and general
ones), co-pathologies, …). The latter can be seen as a sequence of events on
which Recurrent Neural Networks (RNN) are known to perform well, especially
_Long short-term memory_ (LSTM Hochreiter and Schmidhuber (1997)).
Our ongoing study obtains its best results with the following architecture: 1-
an LSTM network is applied on the "dynamic" information (pharmacy transactions
and hospitalizations); 2- an MLP network on the "contextual" information
(details about the patient (geographical, financial support, …)) which are not
frequently updated in the SNIIRAM database; 3- both outputs are concatenated
and then classified through a fully-connected layer. We tested networks based
on this kind of architecture. "Dynamic" inputs are sequences of 10
observations and the output indicates if the current transaction might not be
followed by another one (indicating the risk of an illegitimate drop-out). As
10 observations are not always available for each transaction, we also test
different padding preprocessing with zeros filling or first observation
duplication. In this study, zeros filling got the best results and was
retained for our ongoing process. The underlying goal of this study is also to
gain insights into the influence of different sets of features. As it is
doable to analyze the use of static information through the MLP network, we
focus here on the dynamic data as RNN are more tricky to understand. Our
strategy is to train different network with more or fewer features data. Our
baseline just handles cancerology linked pharmacy transaction and longtime
hospitalizations. Then we add co-pathologies and other pharmacy transactions.
Yet, more dynamic information does not increase the performance as we always
obtain a score that allows targeting efficiently a patient in 82% of the
cases. The CAP curve indicates that, within the first $20\%$ of the surveyed
population ranked from the highest to the lowest risk estimated, our models
target 66% of illegitimate stops of all transactions. This is three times as
effective than the current random model used to target patients at risk, thus
validating our hypothesis that last transactions convey information to detect
an illegitimate stop.
As French Health Services have a limited number of hours to call patients for
support, our model could double their efficiency. This results could also be
applied to notify pharmacists to deliver more support when appropriate or
trigger an SMS based system to contact and motivate a high-risk patient. Yet,
the different data-usage we tried don’t show a lot of improvements in our
results. We might use deeper networks to gain a few points in AUC and CAP
curves. Our first attempts to do so resulted in too much instability in our
training looking at the standard deviation over the k-fold. This could be
explained by the need for more data to provide stable results for more complex
networks. We plan to test this hypothesis but it might take some times due to
the process of SNIIRAM data extraction (which is very regarding the respect of
privacy of the patients). We also look at other solutions such as using
Generative Adversarial Network to increase the size of data or perform
transfer-learning using other Electronic Health Records to improve the general
information about care paths of patients.

##  6 Conclusion and discussion

This paper presents our first studies on how machine-learning can be used to
help patients follow their treatment during long-term illness. We explore
several approaches applied to the French Health System’s reimbursement data to
estimate the risk of an illegitimate drug drop-out. The results obtained with
simple models on indirect observations from SNIIRAM (reimbursement data) prove
the feasibility of our process. We validate our first results with feedbacks
from oncologists and medical researchers. Our approaches also aim to be
coherent with patients’ care-paths. Using drug-phases formalism or looking
directly at pharmacy-transactions, we show that we could notify caregivers of
the potential risk of a drug drop-out at specific moments of the treatment.
This allows to provide a more efficient support at appropriate times while
avoiding stress resulting from too frequent unnecessary contacts and limiting
the waste of resources for low-risk patients.
Both studies show their abilities to predict patients’non-adherence. One main
limitation is the intelligibility of their decision process which remains hard
to interpret with "black box methods". It will be a challenge as medical
staffs want to understand the path of patients. Yet, we validate the ability
of AI to estimate the risk of non-adherence. More complex networks should
improve our efficiency. For example, we could look at several pathologies and
use some kind of domain adaptation methods to detect patterns relevant to non-
adherence. The other main challenge concerns the labeling of the data: we plan
to explore automatic labeling and anomaly discovery to find more accurate
information in our data.

## References

  * DiMatteo [2004a] M Robin DiMatteo.  Social support and patient adherence to medical treatment: a meta-analysis.  _Health psychology_ , 23(2):207, 2004a.
  * DiMatteo [2004b] M Robin DiMatteo.  Variations in patients’ adherence to medical recommendations: a quantitative review of 50 years of research.  _Medical care_ , 42(3):200–209, 2004b.
  * DiMatteo et al. [2000] M Robin DiMatteo, Heidi S Lepper, and Thomas W Croghan.  Depression is a risk factor for noncompliance with medical treatment: meta-analysis of the effects of anxiety and depression on patient adherence.  _Archives of internal medicine_ , 160(14):2101–2107, 2000.
  * Fallowfield et al. [2005] L Fallowfield, L Atkins, S Catt, A Cox, C Coxon, C Langridge, R Morris, and M Price.  Patients’ preference for administration of endocrine treatments by injection or tablets: results from a study of women with breast cancer.  _Annals of Oncology_ , 17(2):205–210, 2005.
  * Franklin et al. [2016] Jessica M Franklin, William H Shrank, Joyce Lii, Alexis K Krumme, Olga S Matlin, Troyen A Brennan, and Niteesh K Choudhry.  Observing versus predicting: Initial patterns of filling predict long-term adherence more accurately than high-dimensional modeling techniques.  _Health services research_ , 51(1):220–239, 2016\.
  * Hochreiter and Schmidhuber [1997] S. Hochreiter and J. Schmidhuber.  Long short-term memory.  _Journal of Neural computation_ , 1997.
  * Janssoone et al. [2018] T Janssoone, P Rinder, P Hornus, and D Kanoun.  Predictive patient care: Survival model to prevent medication non-adherence.  2018\.
  * Karanasiou et al. [2016] Georgia Spiridon Karanasiou, Evanthia Eleftherios Tripoliti, Theofilos Grigorios Papadopoulos, Fanis Georgios Kalatzis, Yorgos Goletsis, Katerina Kyriakos Naka, Aris Bechlioulis, Abdelhamid Errachid, and Dimitrios Ioannis Fotiadis.  Predicting adherence of patients with hf through machine learning techniques.  _Healthcare technology letters_ , 3(3):165–170, 2016.
  * Krolop et al. [2013] Linda Krolop, Yon-Dschun Ko, Peter Florian Schwindt, Claudia Schumacher, Rolf Fimmers, and Ulrich Jaehde.  Adherence management for patients with cancer taking capecitabine: a prospective two-arm cohort study.  _BMJ open_ , 3(7):e003139, 2013.
  * Lo-Ciganic et al. [2015] Wei-Hsuan Lo-Ciganic, Julie M Donohue, Joshua M Thorpe, Subashan Perera, Carolyn T Thorpe, Zachary A Marcum, and Walid F Gellad.  Using machine learning to examine medication adherence thresholds and risk of hospitalization.  _Medical care_ , 53(8):720, 2015.
  * Mann et al. [2010] Devin M Mann, Mark Woodward, Paul Muntner, Louise Falzon, and Ian Kronish.  Predictors of nonadherence to statins: a systematic review and meta-analysis.  _Annals of Pharmacotherapy_ , 44(9):1410–1421, 2010.
  * Morel et al. [2017] Maryan Morel, Emmanuel Bacry, Stéphane Gaïffas, Agathe Guilloux, and Fanny Leroy.  Convsccs: convolutional self-controlled case series model for lagged adverse event detection.  _arXiv preprint arXiv:1712.08243_ , 2017.
  * Neumann et al. [2012] A Neumann, A Weill, P Ricordeau, JP Fagot, F Alla, and H Allemand.  Pioglitazone and risk of bladder cancer among diabetic patients in france: a population-based cohort study.  _Diabetologia_ , 55(7):1953–1962, 2012.
  * O’neill and Twelves [2002] VJ O’neill and CJ Twelves.  Oral cancer treatment: developments in chemotherapy and beyond.  _British journal of cancer_ , 87(9):933, 2002\.
  * Osterberg and Blaschke [2005] L Osterberg and T Blaschke.  Adherence to medication.  _New England Journal of Medicine_ , 353(5):487–497, 2005.
  * Spoelstra et al. [2013] Sandra L Spoelstra, Barbara A Given, Charles W Given, Marcia Grant, Alla Sikorskii, Mei You, and Veronica Decker.  An intervention to improve adherence and management of symptoms for patients prescribed oral chemotherapy agents: an exploratory study.  _Cancer nursing_ , 36(1):18–28, 2013.
  * Tuppin et al. [2010] P Tuppin, L De Roquefeuil, A Weill, P Ricordeau, and Y Merlière.  French national health insurance information system and the permanent beneficiaries sample.  _Revue d’epidemiologie et de sante publique_ , 58(4):286–290, 2010.