# Learning by Asking Questions for Knowledge-based Novel Object Recognition

Kohei Uehara
The University of Tokyo
<EMAIL_ADDRESS>
&Tatsuya Harada
The University of Tokyo
RIKEN
<EMAIL_ADDRESS>

###### Abstract

In real-world object recognition, there are numerous object classes to be
recognized. Conventional image recognition based on supervised learning can
only recognize object classes that exist in the training data, and thus has
limited applicability in the real world. On the other hand, humans can
recognize novel objects by asking questions and acquiring knowledge about
them. Inspired by this, we study a framework for acquiring external knowledge
through question generation that would help the model instantly recognize
novel objects. Our pipeline consists of two components: the Object Classifier,
which performs knowledge-based object recognition, and the Question Generator,
which generates knowledge-aware questions to acquire novel knowledge. We also
propose a question generation strategy based on the confidence of the
knowledge-aware prediction of the Object Classifier. To train the Question
Generator, we construct a dataset that contains knowledge-aware questions
about objects in the images. Our experiments show that the proposed pipeline
effectively acquires knowledge about novel objects compared to several
baselines.

##  1 Introduction

Object category recognition has long been a central topic in computer vision
research. Traditionally, object recognition has been addressed by supervised
learning using a large dataset of image-label pairs Deng et al. (2009).
However, with supervised approaches, the model can only recognize a frozen set
of object classes and is not suitable for real-world object recognition, where
numerous object classes exist. Recently, image recognition methods based on
contrastive learning using image-text pair datasets have emerged Radford et
al. (2021); Jia et al. (2021). By training on hundreds of millions of image-
text pairs, these models have acquired remarkable zero-shot recognition
capabilities for a wide variety of objects. However, these models can
recognize objects that commonly appear in the pre-training dataset but are not
as effective for rare objects Shen et al. (2022). Collecting new data and
retraining the entire model to make these models recognize novel objects is
impractical considering the cost of data collection and computation.
Therefore, it is essential to develop a method that enables the model to
recognize novel objects while maintaining low data collection costs and
avoiding model retraining as much as possible.

Figure 1:  Conceptual illustration of our proposed pipeline. If the model is
confident about the predicted knowledge, question generation is performed in
confirmation mode. If the model is not confident, question generation is
performed in exploration mode.

When humans acquire knowledge about the world, asking questions and explicitly
acquiring knowledge are important skills involved Chouinard et al. (2007);
Ronfard et al. (2018). Inspired by this, we explored methods to dynamically
increase knowledge in image recognition by asking questions. This approach has
several advantages over the traditional supervised learning method: (1) it
requires only a small amount of data to acquire knowledge because the system
acquires only the knowledge it needs, and (2) it has a low data collection
cost because the system itself seeks the required data.

We propose a pipeline consisting of a knowledge-based object classifier (OC)
and a question generator (QG) for knowledge acquisition. Following previous
research on structured knowledge Ji et al. (2022), we represent knowledge as a
knowledge triplet, that is, a list of three words or phrases: head, relation,
and tail, such as $\langle$dog, IsA, mammal$\rangle$. We train the OC to
retrieve knowledge from knowledge sources, which outputs the corresponding
head in the knowledge source as the predicted object class (e.g., ⟨IsA,
mammal⟩$\,\to\,$dog). The QG model then generates questions to add new
knowledge to the knowledge source for novel object recognition. In the QG
model, we use two modes in question generation: confirmation and exploration,
as illustrated in Figure 1. First, “confirmation” is used when the unknown
object is relatively close to a known object category. For example, if the
model knows about “dog,” then a novel category “chihuahua” is considered to be
a close concept to “dog.” In this case, the model can infer reasonable
knowledge (e.g., both “chihuahua” and “dog” are a type of mammal) and ask
questions to confirm it, such as “What is the mammal on the left side of the
image?” In contrast, the “exploration” mode is used when the unknown object is
far from the existing object category (e.g., “teddy-bear” may not resemble any
known object class). In this case, the model is unable to estimate the proper
knowledge and attempts to obtain all the necessary knowledge by asking
questions (“What is the object sitting next to the dog made of?”).

Our contributions and findings can be summarized as follows:

  * •

We propose a novel pipeline to acquire knowledge about novel objects by asking
questions. We designed the OC model based on CLIP Radford et al. (2021) and
the QG model as a Transformer Vaswani et al. (2017) based text generation
model.

  * •

We built a novel dataset to train the QG model, namely, Professional K-VQG.
This dataset contains a variety of annotations such as object labels, bounding
boxes, knowledge, and knowledge-aware questions.

  * •

We compare our proposed pipeline with several baselines and show that the
knowledge acquired through question generation is effective for novel object
recognition.

##  2 Related Work

#### Novel object recognition

Increasing the number of recognizable object classes is a widely studied
problem in object recognition. A typical approach in novel object recognition
is to train a model that computes the similarity between the visual and
semantic features of objects. To compute semantic features of a novel object,
external knowledge about the object (e.g., attributes Lampert et al. (2009);
Farhadi et al. (2009); Jayaraman and Grauman (2014); Akata et al. (2016); Li
et al. (2021), class hierarchy Rohrbach et al. (2011); Wang et al. (2018), or
textual description Ba et al. (2015); Qiao et al. (2016); Reed et al. (2016);
Zareian et al. (2021)) is often used. Recently proposed vision-and-language
contrastive learning methods, such as CLIP Radford et al. (2021) or ALIGN Jia
et al. (2021), use extremely large-scale image caption data to learn the
relationship between images and their textual descriptions. With the help of
the prefix-tuning technique, these models exhibited a strong zero-shot
recognition ability. However, the abovementioned studies have a problem in
that they require either a well-prepared knowledge database on novel objects
or a large number of image-text pair datasets and appropriately designed
prompts, both of which are labor-intensive tasks for humans. In our method,
once the question generation model is trained, the model dynamically acquires
the necessary knowledge, thereby reducing human effort.

#### Learning by asking (LBA)

LBA generates questions to collect additional data to train a model. With the
development of natural language generation methods, several studies using
question generation to acquire the information necessary to solve a task
(e.g., reading comprehension Du et al. (2017); Yuan et al. (2017) or question
answering Scialom and Staiano (2020)) have been conducted. In addition, in
vision-and-language fields, LBA is applied to VQA tasks Misra et al. (2018) or
image captioning tasks Shen et al. (2019). However, our research is the first
attempt to apply LBA to a novel object recognition task, which is a more real-
world-oriented visual recognition task.

#### Visual question generation (VQG)

The first studies to address VQG used a simple method of inputting image
features into a text decoder and generating questions Mostafazadeh et al.
(2016). Subsequent research has focused on how to better control the content
of the questions to be generated. In general, the control of question
generation is achieved by providing additional information to the text decoder
in addition to the image features. The main methods include providing the
answers Li et al. (2018); Liu et al. (2018) and providing categories of the
answers Krishna et al. (2019); Uppal et al. (2021); Uehara et al. (2018). In
addition, similar to this study, another study targeted knowledge that can be
acquired through questioning Uehara and Harada (2022). They created a
knowledge-aware VQG dataset (K-VQG) using AMT and used UNITER Chen et al.
(2020), which is a state-of-the-art vision-and-language transformer, as an
encoder for images and knowledge to successfully generate questions for the
knowledge. We designed a question generation model based on their work and
created a new dataset as an extension of their dataset.

##  3 Method

Our system consists of an Object Classifier (OC) and a Question Generator
(QG). First, we describe the overall pipeline of our system (Figure 2). Then,
we describe the detail of each module in the following sections.

Figure 2:  Overall pipeline of our method. The OC model performs knowledge-
based object recognition using knowledge sources. The QG model generates
questions target the knowledge needed for novel object recognition. Answers to
the questions are provided by the Oracle Answerer and added to the knowledge
source. With the newly added knowledge, the OC model is able to recognize
novel objects.  Figure 3:  Architecture of the OC model. Based on the
knowledge encoded by BERT and the similarity calculation of the object image
encoded by CLIP, the prediction of the knowledge required for object
recognition is performed.

The OC model explicitly utilizes knowledge when object recognition is
performed. In other words, for a given image of the object, the OC retrieves
the appropriate knowledge $k=[h,\;r,\;t]\in\mathcal{K}$, where $h$, $r$, and
$t$ are the head, relation, and tail, respectively, and $\mathcal{K}$ is a
knowledge source. Here, $h$ corresponds to the object label. Thus, once the
model predicts the knowledge of the object image, it can search $\mathcal{K}$
using the predicted knowledge and obtain the object label. This design allows
the model to instantly recognize novel objects when it obtains knowledge about
them, simply by updating the knowledge source, without retraining the model.

The QG model is responsible for generating questions about the objects in the
image and acquiring knowledge that is useful for novel object recognition. To
this end, we conditioned the QG with partial knowledge, which masks part of
the knowledge.

Once the answers to the generated questions are obtained, the acquired
knowledge $\mathcal{K}^{\prime}$ is added to the model’s original knowledge
source $\mathcal{K}$. Consequently, the OC’s knowledge source is updated to
$\mathcal{K}^{+}=\mathcal{K}\;\cup\;\mathcal{K}^{\prime}$. Then, in the next
inference phase, the OC makes predictions about the knowledge and labels by
referring to the updated knowledge source $\mathcal{K}^{+}$.

###  3.1 Object Classifier

The OC model (Figure 3) predicts object-related knowledge by computing the
similarity between the object feature $\bm{f}_{o}\in\mathbb{R}^{d}$ and the
knowledge feature $\bm{f}_{k}\in\mathbb{R}^{d}$ of the associated knowledge,
that is, $p(k)=\mathrm{sim}(\bm{f}_{o},\;\bm{f}_{k})$, where $d$ is the
dimension of the object features and knowledge features.

We adopted a model based on CLIP Radford et al. (2021), which is one of the
latest visual recognition models, as the object knowledge predictor. CLIP is a
model consisting of an image encoder and text encoder, each of which can be
used to compute the similarity between images and text. The image encoder of
CLIP $f_{\theta}$ receives a cropped image $I_{\textrm{crop}}$ as input and
outputs the visual feature $\bm{f}_{o}$. Knowledge features of knowledge
$\bm{f}_{k}$ is also computed using pre-trained CLIP text encoder $f_{\phi}$.
Here, we converted the knowledge from a triplet representation (e.g., <cat,
IsA, mammal>) to the representation of a single sentence with a masked head
(e.g., “[MASK] is a mammal”) before feeding it into the text encoder. We used
cosine similarity to compute the similarity between the object and knowledge
features as follows:

$\displaystyle\bm{f}_{o}=f_{\theta}(I_{crop}),\quad\bm{f}_{k}=f_{\phi}(k)$ (1)
$\displaystyle\mathrm{sim}(\bm{f}_{o},\;\bm{f}_{k})=\frac{\bm{f}_{o}^{\top}\bm{f}_{k}}{\|\bm{f}_{o}\|\|\bm{f}_{k}\|}$
(2)

The OC model is trained to minimize the binary cross-entropy loss as follows:

$L_{\mathrm{OC}}=-\sum^{|\mathcal{K}|}_{i}\bigl{(}y_{i}\cdot\log\sigma(\mathrm{sim}(\bm{f}_{o},\;\bm{f}_{k_{i}}))+\\\
(1-y_{i})\cdot\log(1-\sigma(\mathrm{sim}(\bm{f}_{o},\;\bm{f}_{k_{i}})))\bigr{)}$
(3)

where $y_{i}\in\\{0,1\\}$ indicates the ground-truth label for the $i$-th
knowledge.

If the model successfully predicts the knowledge, then it has the relation and
tail of the knowledge about the object. Then, when inferring labels from the
predicted knowledge $\hat{k}$, we search the knowledge source $\mathcal{K}$
for knowledge that satisfies the predicted relation and tail conditions and
treat the matching head as the predicted label.

###  3.2 Question Generator

The QG model takes an image $I$ and target knowledge $k$ as input and
generates question $q$.

First, we explain how the target knowledge is determined. In knowledge
acquisition, it is important to acquire knowledge that is “appropriate” and
“useful” for recognition, that is, to acquire correct knowledge at the lowest
possible cost. Here, “low cost” implies that retraining the OC model should be
avoided as much as possible. Therefore, we propose using two different modes
of question generation: “confirmation” and “exploration.” As described in Sec.
1, the “confirmation” mode is used when the unknown object is relatively close
to a known object category, whereas the “exploration” mode is used when the
unknown object is far from the existing object category. The target knowledge
$k$ in each case is defined as follows:

$k=\left\\{\begin{array}[]{ll}[\,\texttt{MASK},\;\hat{r},\;\hat{t}\,]&\mathrm{(confirmation)}\\\
[\,\texttt{MASK},\;r^{*},\;\texttt{MASK}\,]&\mathrm{(exploration)}\end{array}\right.$
(4)

Here, $\hat{r}$ and $\hat{t}$ are the predicted relation and tail,
respectively, and $r^{*}$ is an arbitrarily chosen relation based on the
frequency of the relation in the data.

We control the policy of mode selection based on the expected value of the
utility that the model can obtain from the answer. Here, we define the policy
selection function $\pi$, which takes the value 1 for the confirmation mode
and 0 for the exploration mode. Then, we adopt a policy that maximizes the
expected utility of the model with utility function $u_{\theta}$ for the
training data $\mathcal{X}$:

$\frac{1}{|\mathcal{X}|}\sum(\pi\,u_{\theta}(\bm{f}_{o})+(1-\pi)\,u_{\theta}(\bm{f}_{o}))$
(5)

We define the utility function as the sum of the “correctness” and
“informativeness” of the expected answer. The “correctness” represents the
estimated correctness of the knowledge expected to be acquired by the answer.
For simplicity, we assume that the oracle answer should be correct and suppose
that the expected correctness is 1.0 when the mode is “exploration.” On the
other hand, when the mode is “confirmation,” the expected correctness depends
on the confidence of the model $\mathrm{conf}(\hat{k})$; thus, we set the
expected correctness as the predicted score output by the OC model.

The “informativeness” is the value representing how useful the acquired
knowledge is to the model. For the “exploration” mode, we estimate the
informativeness using the similarity between the input image feature and
target knowledge feature $\mathrm{sim}(\bm{f}_{o},\;\bm{f}_{\hat{k}})$. For
the “confirmation” mode, we use the expected value of the similarity based on
the mean similarity of the training data, i.e.,
$\mathbf{E}[I]\simeq\frac{1}{|\mathcal{X}|}\sum\mathrm{sim}(\bm{f}_{o},\;\bm{f}_{\hat{k})}$.

The utility function is expressed as follows:

$u_{\theta}(\bm{f}_{o})=\left\\{\begin{array}[]{ll}\mathrm{conf}(\hat{k})+\mathrm{sim}(\bm{f}_{o},\;\bm{f}_{\hat{k}})&\mathrm{(conf.)}\\\
1+\frac{1}{|\mathcal{X}|}\sum\mathrm{sim}(\bm{f}_{o},\;\bm{f}_{\hat{k}})&\mathrm{(exp.)}\end{array}\right.$
(6)

Once the input knowledge $k$ is determined, question generation is performed
using it as input. In the question generation model, we used the ViLT Kim et
al. (2021) encoder, which is one of the state-of-the-art vision-and-language
model, and GPT-2 Radford et al. (2019) based decoder. The ViLT encoder
$\mathrm{Enc}(\cdot)$ takes two inputs: (1) the input image $I$ and the masked
region image $I_{M}$ and (2) knowledge triplets in sentence form, i.e.,
“[MASK] is a mammal.” Here, a masked region image is one in which all pixel
values outside the target region are replaced by zero. A learnable type
embedding vector is added to the masked image and the original image,
respectively.

The model is trained using the following loss function:

$L=-\sum_{i}\log P(y_{i}\;|\;y_{<i},\;\mathrm{Enc}([I,I_{M}],\;k))$ (7)

Here, $y_{i}$ is the $i$-th word of the question and $\mathrm{Enc}(\cdot)$ is
the ViLT encoder, which produces fused feature of the visual and textual
knowledge features.

###  3.3 Oracle Answerer

Figure 4: The architecture of the oracle answerer.

Given an image and a generated question, the Oracle Answerer predicts the
answer knowledge to the question. We implement this module as a composition of
three sub-modules; (1) Head classifier, (2) Relation classifier, and (3)
Region classifier. Each module checks whether the generated question is
“valid”, and if all modules agree that the question is “valid,” Oracle
Answerer searches the oracle knowledge source and outputs the knowledge that
matches the targeted head and relation. Oracle knowledge source is a knowledge
source that merges ConceptNet Speer et al. (2017) and Atomic${}^{20}_{20}$
Hwang et al. (2021), which were used in the creation of the datset (see Sec.
4.1).

#### Head classifier

The head classifier $\mathcal{H}$ predicts the head of the target knowledge
from the generated question, i.e., $h=\mathcal{H}(I,Q)$. We implement this
module following the standard VQA methodology, that is, as a multi-class
classification problem that outputs the proper entity given an image and a
question. For this module, we fine-tuned pre-trained ViLT-VQA Kim et al.
(2021) model. This module returns “valid” if the predicted head is equal to
the object in the target region.

#### Relation classifier

The relation classifier $\mathcal{R}$ predicts the relation of the target
knowledge from the generated question, i.e., $r=\mathcal{R}(Q)$. Since this
problem can be formulated as a sentence classification problem, we use a fine-
tuned Distil-BERT Sanh et al. (2019) as the relation classifier. This module
returns “valid” if the predicted relation matches the target relation ($r$ in
the eq. 4).

#### Region classifier

The region classifier $\mathcal{G}$ predicts the target region, i.e.,
$g=\mathcal{G}(I,Q)$. We design this module as a model that, given a question
and a set of candidate regions, outputs the region that is most relevant to
the question. This problem setup is similar to that of the Referring
Expression Comprehension (RE Comprehension) Yu et al. (2016). Therefore, we
used a fine-tuned version of UNITER grounding model Chen et al. (2020), which
has achieved high performance in RE Comprehension task. This module returns
“valid” if the predicted region is sufficiently close to the target region. We
calculated the IoBB (Intersection over Bounding Box) between the predicted
region and the target region and considered two regions sufficiently close if
the value was greater than 0.4.

###  3.4 Knowledge Expansion

When an answer knowledge $k`$ is obtained for a generated question $q$ by the
model, it is added to the model’s knowledge source $\mathcal{K}$, that is,
$\mathcal{K}^{+}=\mathcal{K}\;\cup\;\\{k^{\prime}\\}_{i=1}^{M}$, where $M$ is
the number of newly acquired knowledge.

##  4 Experiments

###  4.1 Dataset

Figure 5: Word clouds for the questions and answers in the Professional K-VQG
dataset.  Figure 6: The distribution of relations in the Professional K-VQG
dataset.  Figure 7: Some examples of the Professional K-VQG dataset.

| K-VQG |  | Processional
---
K-VQG
Merged
# questions | 16,098 | 10,431 | 22,212
– head answers | 11,588 | 5,047 | 13,916
– tail answers | 4,510 | 5,384 | 8,296
# images | 13,648 | 9,494 | 9,210
# unique answers | 2,819 | 3,687 | 4,953
# unique knowledge | 6,084 | 5,242 | 7,808
# unique head | 527 | 371 | 533
# unique tail | 4,922 | 4,257 | 6,199

Table 1: Detailed statistics of the Professional K-VQG dataset.

To train the QG model, we used the K-VQG dataset Uehara and Harada (2022),
which contains knowledge-aware questions about objects in images, and the
newly created Professional K-VQG dataset. The Professional K-VQG dataset
contains knowledge-aware visual questions related to objects annotated by
professional annotators. The source of the images is the Visual Genome Krishna
et al. (2017), and the sources of knowledge are ConceptNet Speer et al. (2017)
and Atomic${}^{20}_{20}$ Hwang et al. (2021). We selected 371 object classes
from the Visual Genome dataset that were commonly included as heads in the
knowledge sources. In the annotation process, for each candidate object in the
image, we obtained the corresponding knowledge triplets from the knowledge
sources and asked the annotator to write a knowledge-aware question, with the
head or tail of the knowledge triplet as the answer. As a result, we obtained
10,431 questions for 9,210 images and 5,242 unique knowledge.

We show the word clouds and the relation distribution in the Professional
K-VQG dataset in Figure 5 and 6 to visualize several features of the dataset.
We can see that our dataset contains questions on a wide variety of topics
about food, clothing, furniture, and so on. The most and second most frequent
relations, “UsedFor” and “IsA”, account for nearly 50% and 25% of the total,
respectively, which seems a little biased, but this is due to the fact that
these relations are also frequent in our knowledge sources. We show some
examples of the Professional K-VQG dataset in Figure 7.

Because K-VQG and Professional K-VQG share the same annotation format, we
merged them to train our QG model. To use region-based image features obtained
from the Faster R-CNN Anderson et al. (2018); Ren et al. (2015) in the QG
module, we excluded samples with target regions that were not detected as
objects by the Faster R-CNN (i.e., the IoU between the detected bounding box
and the target bounding box was less than 0.5). We split the merged K-VQG into
train, query, and test sets. The train set was used to train the OC and QG
models and the Oracle Answerer. The query set was used by the OC model to
acquire new knowledge, that is, to evaluate the OC model, generate questions
to acquire new knowledge using the QG model, and provide answers using the
Oracle Answerer. The test set was used to evaluate the final performance of
the OC model using knowledge acquired from the query set. The train / query /
test splits contained 13,334 / 3,693 / 1,414 images and 16,110 / 4,561 / 1,541
questions, respectively. The number of known knowledge (i.e., knowledge
contained in the train set) was 460, and the number of novel knowledge (i.e.,
knowledge not contained in the train set) was 49. We show some detailed
statistics of the Professional K-VQG dataset in Table 1.

To train the OC model, we used a dataset consisting of object annotations in
the Visual Genome as well as the knowledge triplets obtained from the
aforementioned knowledge sources. The train and validation splits contained
52,321 / 13,080 images and 111,027 / 35,430 objects, respectively, with an
average of 52.18 knowledge for each object. We took care not to contaminate
the images and object classes in the train split with other splits in the
K-VQG dataset.

###  4.2 Training

We used the same text encoder as CLIP Radford et al. (2021) and ViT-B/32
Dosovitskiy et al. (2021) as the visual encoder in the OC model. All the
models were trained using the AdamW optimizer Kingma and Ba (2015), where the
learning rate was set to 8e-5 for the OC model and 5e-5 for the QG model. We
trained the OC model for 200 epochs and the QG model for 50 epochs with
8$\times$Tesla A100 GPUs.

###  4.3 Baselines

We compared our approach with four baselines: CLIP-Ret. In this setting, no
knowledge acquisition is performed by the QG model, and the performance of the
OC model trained with only the training set is evaluated. All Exp. / All Conf.
In these settings, the question generation policy is fixed to “exploration”
and “confirmation,” respectively. Random Policy. The question generation
policy is chosen randomly. We tested this method three times with different
random seeds.

###  4.4 Results

| Overall acc. | Known acc. | Novel acc. |  |
---|---|---|---|---|---
| zero-shot | fine-tune | zero-shot | fine-tune | zero-shot | fine-tune | #valid Q. | #knowledge
Baseline (CLIP-Ret.) | 38.9 | - | 59.9 | - | 0.0 | - | - | -
All Conf. | 56.2 | 59.7 | 65.8 | 67.5 | 38.5 | 45.3 | 9756 | 8272
All Exp. | 41.5 | 45.5 | 57.1 | 54.7 | 12.5 | 28.4 | 1057 | 8431
Random Policy | 48.9 $\pm$ 3.8 | 53.2 $\pm$ 0.39 | 60.6 $\pm$ 2.4 | 60.8 $\pm$ 0.71 | 27.4 $\pm$ 7.1 | 39.2 $\pm$ 0.56 | 2788 $\pm$ 116 | 8521 $\pm$ 169
Ours | 56.1 | 62.0 | 65.8 | 67.5 | 38.3 | 51.9 | 9864 | 8354

Table 2:  Results of the OC model. The top row shows the baseline results
without knowledge acquisition through question generation, and the subsequent
rows show the results using different question generation policies.

The main results are summarized in Table 2. We report results separately for
the “zero-shot” and “fine-tune” settings, the former in which the obtained
knowledge is simply added to the OC and tested without additional training,
and the latter in which the OC is additionally trained with the obtained
knowledge. Known acc. is the accuracy for data where the correct answers are
the objects already contained in the training data (i.e., known objects), and
Novel acc. is the accuracy for data where the correct answers are the objects
not contained in the training data (i.e., novel objects).

Overall, the proposed methods outperformed the baseline CLIP-Ret on all
metrics. In the zero-shot setting, All Conf. performed best, while Ours
performed best in the fine-tune setting. This may be due to the nature of the
confirmation mode. In the confirmation mode, the question target of the QG
model is the partial knowledge (i.e., head masked knowledge) that scored
highest in the prediction of OC. Therefore, when no additional training is
performed, the confirmation mode, which allows the acquisition of knowledge
that is known to be "useful" in advance, tends to perform better. However, in
the confirmation mode, it is not possible to obtain knowledge with a
combination of relation and tail that does not exist in the training data, and
the variety of knowledge to be obtained is limited. Thus, the number of
knowledge (#knowledge) obtained in the confirmation mode was less than in the
other modes, which may have resulted in inferior performance after the
additional training.

On the other hand, in settings All Exp. and Random Policy, the number of
knowledge acquired was larger than in Ours, but the number of valid questions
(#valid Q.) was smaller. Unlike the confirmation mode, the exploration mode is
a mode that is more likely to yield completely new knowledge. However, in the
exploration mode, less information is input to the question generation model
and the question generation performance is degraded. For these reasons, the
number of valid novel data available for training has decreased, and the
performance of the OC model cannot be adequately improved.

Figure 8:  Examples of generated questions (left) and the novel object
recognition with newly acquired knowledge (right). The left column of the
question generation results shows examples of successfully acquiring new
knowledge. In the object recognition results, both “coat” and “airplane” are
novel objects, but the model succeeds to recognize them by using the newly
acquired knowledge.

###  4.5 Qualitative Results

In Figure 8, we show several examples of the generated questions and novel
object recognition using the newly acquired knowledge. As shown in the left
column of the question generation results, the QG model generated questions
that could successfully acquire new knowledge. However, in some cases, the QG
model failed to generate valid questions. This is mainly because VQG targeting
knowledge is still a challenging task, and the model generates questions that
are not relevant to the content of the image or the targeted knowledge. We
leave further improvement of knowledge-targeted question generation as a
future issue.

##  5 Conclusion

In this study, we presented a novel method for knowledge acquisition through
VQG in object classification tasks. By using a method that appropriately
determines the target of the question, we achieved novel object recognition
based on knowledge, while reducing the cost of retraining.

A future challenge would be to study better question-targeting policies. Our
work is related to active learning, in which the main topics are to be labeled
by a human. However, when active learning is applied to complicated tasks such
as those with multimodal inputs, it is known that the performance gains are
smaller than those for random baselines Karamcheti et al. (2021). We believe
that an interesting issue to be addressed in the future is how to apply active
learning methods to our proposed task of knowledge acquisition through
question generation.

## Acknowledgments

This work was partially supported by JST AIP Acceleration Research JPMJCR20U3,
Moonshot R&D Grant Number JPMJPS2011, CREST Grant Number JPMJCR2015, JSPS
KAKENHI Grant Number JP19H01115, JP20H05556 and Basic Research Grant (Super
AI) of Institute for AI and Beyond of the University of Tokyo. We would like
to thank Yusuke Mori for their helpful discussions.

## References

  * Akata et al. (2016) Zeynep Akata, Mateusz Malinowski, Mario Fritz, and Bernt Schiele. 2016.  Multi-cue zero-shot learning with strong supervision.  In _CVPR_.
  * Anderson et al. (2018) Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018.  Bottom-up and top-down attention for image captioning and visual question answering.  In _CVPR_.
  * Ba et al. (2015) Jimmy Ba, Kevin Swersky, Sanja Fidler, and Ruslan Salakhutdinov. 2015.  Predicting deep zero-shot convolutional neural networks using textual descriptions.  In _ICCV_.
  * Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020.  UNITER: Universal image-text representation learning.  In _ECCV_.
  * Chouinard et al. (2007) Michelle M Chouinard, Paul L Harris, and Michael P Maratsos. 2007.  Children’s questions: A mechanism for cognitive development.  _Monographs of the society for research in child development_ , pages i–129.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. 2009.  Imagenet: A large-scale hierarchical image database.  pages 248–255.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _ICLR_.
  * Du et al. (2017) X. Du, Junru Shao, and Claire Cardie. 2017.  Learning to ask: Neural question generation for reading comprehension.  In _ACL_.
  * Farhadi et al. (2009) Ali Farhadi, Ian Endres, Derek Hoiem, and David Alexander Forsyth. 2009.  Describing objects by their attributes.  In _CVPR_.
  * Hwang et al. (2021) Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021.  Comet-atomic 2020: On symbolic and neural commonsense knowledge graphs.  In _AAAI_.
  * Jayaraman and Grauman (2014) Dinesh Jayaraman and Kristen Grauman. 2014.  Zero-shot recognition with unreliable attributes.  In _NIPS_.
  * Ji et al. (2022) Shaoxiong Ji, Shirui Pan, E. Cambria, Pekka Marttinen, and Philip S. Yu. 2022.  A survey on knowledge graphs: Representation, acquisition, and applications.  _IEEE Transactions on Neural Networks and Learning Systems_ , 33:494–514.
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _ICML_.
  * Karamcheti et al. (2021) Siddharth Karamcheti, Ranjay Krishna, Li Fei-Fei, and Christopher D. Manning. 2021\.  Mind your outliers! investigating the negative impact of outliers on active learning for visual question answering.  In _ACL_.
  * Kim et al. (2021) Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 5583–5594. PMLR.
  * Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _ICLR_.
  * Krishna et al. (2019) Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. 2019.  Information maximizing visual question generation.  In _CVPR_.
  * Krishna et al. (2017) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017\.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _IJCV_.
  * Lampert et al. (2009) Christoph H. Lampert, Hannes Nickisch, and Stefan Harmeling. 2009.  Learning to detect unseen object classes by between-class attribute transfer.  In _CVPR_.
  * Li et al. (2021) Shuang Li, Lichun Wang, Shaofan Wang, Dehui Kong, and Baocai Yin. 2021.  Zero-shot recognition with image attributes generation using hierarchical coupled dictionary learning.  _ACM Multimedia Asia_.
  * Li et al. (2018) Yikang Li, Nan Duan, Bolei Zhou, Xiao Chu, Wanli Ouyang, and Xiaogang Wang. 2018\.  Visual question generation as dual task of visual question answering.
  * Liu et al. (2018) Feng Liu, Tao Xiang, Timothy M. Hospedales, Wankou Yang, and Changyin Sun. 2018\.  Ivqa: Inverse visual question answering.  In _CVPR_.
  * Misra et al. (2018) Ishan Misra, Ross Girshick, Rob Fergus, Martial Hebert, Abhinav Gupta, and Laurens van der Maaten. 2018.  Learning by asking questions.  In _CVPR_.
  * Mostafazadeh et al. (2016) Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende. 2016.  Generating natural questions about an image.  In _ACL_.
  * Qiao et al. (2016) Ruizhi Qiao, Lingqiao Liu, Chunhua Shen, and Anton van den Hengel. 2016.  Less is more: Zero-shot learning from online textual documents with noise suppression.  In _CVPR_.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  In _ICML_.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Reed et al. (2016) Scott E. Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. 2016.  Learning deep representations of fine-grained visual descriptions.  In _CVPR_.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 39:1137–1149.
  * Rohrbach et al. (2011) Marcus Rohrbach, Michael Stark, and Bernt Schiele. 2011.  Evaluating knowledge transfer and zero-shot learning in a large-scale setting.  In _CVPR_.
  * Ronfard et al. (2018) Samuel Ronfard, Imac M Zambrana, Tone K Hermansen, and Deborah Kelemen. 2018.  Question-asking in childhood: A review of the literature and a framework for understanding its development.  _Developmental Review_.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  In _NeurIPS EMC2 Workshop_.
  * Scialom and Staiano (2020) Thomas Scialom and Jacopo Staiano. 2020.  Ask to learn: A study on curiosity-driven question generation.  In _COLING_.
  * Shen et al. (2022) Sheng Shen, Chunyuan Li, Xiaowei Hu, Yujia Xie, Jianwei Yang, Pengchuan Zhang, Anna Rohrbach, Zhe Gan, Lijuan Wang, Lu Yuan, Ce Liu, Kurt Keutzer, Trevor Darrell, and Jianfeng Gao. 2022.  K-lite: Learning transferable visual models with external knowledge.  _ArXiv_ , abs/2204.09222.
  * Shen et al. (2019) Tingke Shen, Amlan Kar, and Sanja Fidler. 2019.  Learning to caption images through a lifetime by asking questions.  In _ICCV_.
  * Speer et al. (2017) Robyn Speer, Joshua Chin, and Catherine Havasi. 2017.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In _AAAI_.
  * Uehara and Harada (2022) Kohei Uehara and Tatsuya Harada. 2022.  K-vqg: Knowledge-aware visual question generation for common-sense acquisition.  _ArXiv_ , abs/2203.07890.
  * Uehara et al. (2018) Kohei Uehara, Antonio Tejero-De-Pablos, Yoshitaka Ushiku, and Tatsuya Harada. 2018\.  Visual question generation for class acquisition of unknown objects.  In _ECCV_.
  * Uppal et al. (2021) Shagun Uppal, Anish Madan, Sarthak Bhagat, Yi Yu, and Rajiv Ratn Shah. 2021.  C3vqg: Category consistent cyclic visual question generation.  In _ACM Multimedia in Asia_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _NeurIPS_.
  * Wang et al. (2018) X. Wang, Yufei Ye, and Abhinav Kumar Gupta. 2018.  Zero-shot recognition via semantic embeddings and knowledge graphs.  In _CVPR_ , pages 6857–6866.
  * Yu et al. (2016) Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. 2016\.  Modeling context in referring expressions.  In _European Conference on Computer Vision_ , pages 69–85. Springer.
  * Yuan et al. (2017) Xingdi Yuan, Tong Wang, Ãaglar GülÃ§ehre, Alessandro Sordoni, Philip Bachman, Saizheng Zhang, Sandeep Subramanian, and Adam Trischler. 2017\.  Machine comprehension by text-to-text neural question generation.  In _Rep4NLP@ACL_.
  * Zareian et al. (2021) Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. 2021.  Open-vocabulary object detection using captions.  In _CVPR_.


§ ACKNOWLEDGEMENT

The authors would like to thank Thomas Scruby for enlightening discussions. SML and MM  wish to thank NTT Research for their financial and technical support. This work was supported in part by Canada’s NSERC. Research at IQC is supported in part by the Government of Canada through Innovation, Science and Economic Development Canada.

Research at Perimeter Institute is supported in part by the Government of Canada through the Department of Innovation, Science and Economic Development Canada and by the Province of Ontario through the Ministry of Colleges and Universities.
LY is supported by an Oxford - Basil Reeve Graduate Scholarship at Oriel College with the Clarendon Fund.

[1]
authorScott Aaronson &
authorDaniel Gottesman
(year2004): titleImproved simulation of
stabilizer circuits.
journalPhysical Review A
volume70(number5), p. pages052328,

[2]
authorJonas T Anderson,
authorGuillaume Duclos-Cianci &
authorDavid Poulin
(year2014): titleFault-tolerant conversion
between the steane and reed-muller quantum codes.
journalPhysical Review Letters
volume113(number8), p. pages080501,

[3]
authorFrank Arute, authorKunal
Arya, authorRyan Babbush, authorDave Bacon,
authorJoseph C Bardin,
authorRami Barends,
authorRupak Biswas,
authorSergio Boixo,
authorFernando GSL Brandao,
authorDavid A Buell et al.
(year2019): titleQuantum supremacy using a
programmable superconducting processor.
volume574(number7779), pp. pages505–510,

[4]
authorMiriam Backens
(year2014): titleThe ZX-calculus is complete
for stabilizer quantum mechanics.
journalNew Journal of Physics
volume16(number9), p. pages093021,

[5]
authorH. Bombín &
authorM. A. Martin-Delgado
(year2006): titleTopological quantum
journalPhysical Review Letters
volume97, p. pages180501,

[6]
authorH. Bombín &
authorM. A. Martin-Delgado
(year2007): titleTopological computation without
journalPhysical Review Letters
volume98, p. pages160502,

[7]
authorHéctor Bombín
(year2015): titleGauge color codes: optimal
transversal gates and gauge fixing in topological stabilizer codes.
journalNew Journal of Physics
volume17(number8), p. pages083002,

[8]
authorRobert I. Booth &
authorTitouan Carette:
titleComplete ZX-calculi for the stabiliser fragment in
odd prime dimensions, 10.4230/LIPIcs.MFCS.2022.24.
noteISSN: 1868-8969.

[9]
authorA. R. Calderbank &
authorPeter W. Shor
(year1996): titleGood quantum error-correcting
codes exist.
journalPhysical Review A volume54,
pp. pages1098–1105, 10.1103/PhysRevA.54.1098.

[10]
authorA Robert Calderbank,
authorEric M Rains,
authorPeter W Shor &
authorNeil JA Sloane
(year1997): titleQuantum error correction and
orthogonal geometry.
journalPhysical Review Letters
volume78(number3), p. pages405,

[11]
authorYudong Cao,
authorJhonathan Romero &
authorAlán Aspuru-Guzik
(year2018): titlePotential of quantum computing
for drug discovery.
journalIBM Journal of Research and Development
volume62(number6), pp. pages6–1,

[12]
authorTitouan Carette,
authorDominic Horsman &
authorSimon Perdrix
(year2019): titleSZX-calculus: scalable
graphical quantum reasoning.
In: booktitle44th International Symposium on
Mathematical Foundations of Computer Science (MFCS 2019), seriesLeibniz International Proceedings in Informatics (LIPIcs)
volume138, publisherSchloss Dagstuhl–Leibniz-Zentrum
fuer Informatik, addressDagstuhl, Germany, pp.
pages55:1–55:15, 10.4230/LIPIcs.MFCS.2019.55.

[13]
authorNicholas Chancellor,
authorAleks Kissinger,
authorJoschka Roffe,
authorStefan Zohren &
authorDominic Horsman
(year2023): titleGraphical Structures for Design
and Verification of Quantum Error Correction,

[14]
authorJulien Codsi (year2022):
titleCutting-edge graphical stabiliser decompositions for
classical simulation of quantum circuits.
Master's thesis, schoolUniversity of Oxford.

[15]
authorBob Coecke & authorRoss
Duncan (year2008):
titleInteracting quantum observables.
In: booktitleICALP, seriesLecture
Notes in Computer Science, pp. pages298–310,

[16]
authorBob Coecke & authorRoss
Duncan (year2011):
titleInteracting quantum observables: categorical algebra
and diagrammatics.
journalNew Journal of Physics
volume13(number4), p. pages043016,

[17]
authorBob Coecke & authorAleks
Kissinger (year2017):
titlePicturing quantum processes.
publisherCambridge University Press,

[18]
authorBob Coecke &
authorQuanlong Wang
(year2018): titleZX-rules for 2-qubit Clifford+T
quantum circuits.
In editorJarkko Kari &
editorIrek Ulidowski, editors: booktitleReversible Computation, publisherSpringer
International Publishing, addressCham, pp.
pages144–161, 10.1007/978-3-319-99498-7_10.

[19]
authorOliver Cole (year2022):
titleQuantum circuit optimisation through stabiliser
reduction of Pauli exponentials.
schoolMaster’s thesis, University of

[20]
authorCole Comfort (year2023):
titleThe algebra for stabilizer codes,

[21]
authorAlexander Cowtan
(year2022): titleQudit lattice surgery,

[22]
authorAlexander Cowtan &
authorSimon Burton
(year2023): titleCSS code surgery as a universal
journalarXiv preprint arXiv:2301.13738,

[23]
authorRoss Duncan &
authorMaxime Lucas
(year2014): titleVerifying the Steane code with
journalElectronic Proceedings in Theoretical
Computer Science volume171, pp. pages33–49,

[24]
authorBryan Eastin &
authorEmanuel Knill
(year2009): titleRestrictions on transversal
encoded quantum gate sets.
journalPhysical Review Letters
volume102(number11), p. pages110502,

[25]
authorKun Fang, authorJingtian
Zhao, authorXiufan Li, authorYifei Li &
authorRunyao Duan
(year2022): titleQuantum NETwork: from theory to
journalarXiv preprint arXiv:2212.01226,

[26]
authorLiam Garvie & authorRoss
Duncan (year2017):
titleVerifying the smallest interesting colour code with
journalarXiv preprint arXiv:1706.02717,

[27]
authorCraig Gidney (year2022):
titleA pair measurement surface code on pentagons,

[28]
authorCraig Gidney &
authorAustin G. Fowler
(year2019): titleEfficient magic state factories
with a catalyzed |CCZ> to 2|T> transformation.
journalQuantum volume3, p.
pages135, 10.22331/q-2019-04-30-135.

[29]
authorCraig Gidney &
authorAustin G. Fowler
(year2019): titleFlexible layout of surface code
computations using AutoCCZ states, 10.48550/arXiv.1905.08916.

[30]
authorJonathan Gorard,
authorManojna Namuduri &
authorXerxes D. Arsiwalla
(year2021): titleZX-calculus and extended
Wolfram model systems II: fast diagrammatic reasoning with an application to
quantum circuit simplification.
journalarXiv preprint arXiv:2103.15820,

[31]
authorD Gottesman (year1999):
titleThe Heisenberg representation of quantum computers.
In: booktitleGroup22: Proceedings of the XXII
International Colloquium on Group Theoretical Methods in Physics, pp.
pages32–43, 10.48550/arXiv.quant-ph/9807006.

[32]
authorDaniel Gottesman
(year1997): titleStabilizer codes and quantum
error correction.
Ph.D. thesis, schoolCalifornia Institute of Technology,

[33]
authorDaniel Gottesman
(year1998): titleTheory of fault-tolerant
quantum computation.
journalPhysical Review A volume57,
pp. pages127–137, 10.1103/PhysRevA.57.127.

[34]
authorDaniel Gottesman
(year2022): titleOpportunities and challenges in
fault-tolerant quantum computation.
journalarXiv preprint arXiv:2210.15844,

[35]
authorAmar Hadzihasanovic,
authorKang Feng Ng &
authorQuanlong Wang
(year2018): titleTwo complete axiomatisations of
pure-state qubit quantum computing.
In: booktitleACM/IEEE Symposium on Logic in
Computer Science, seriesLICS '18,
publisherAssociation for Computing Machinery,
addressNew York, NY, USA, p. pages502–511,

[36]
authorAlexander Tianlin Hu &
authorAndrey Boris Khesin
(year2022): titleImproved graph formalism for
quantum circuit simulation.
journalPhysical Review A

[37]
authorEmmanuel Jeandel,
authorSimon Perdrix &
authorRenaud Vilmart
(year2020): titleCompleteness of the
journalLogical Methods in Computer Science
volumeVolume 16, Issue 2, 10.23638/LMCS-16(2:11)2020.

[38]
authorAndrey Boris Khesin,
authorJonathan Z. Lu &
authorPeter W. Shor
(year2023): titleGraphical quantum
Clifford-encoder compilers from the ZX calculus,

[39]
authorAleks Kissinger
(year2022): titlePhase-free ZX diagrams are CSS
codes (... or how to graphically grok the surface code).
journalarXiv preprint arXiv:2204.14038,

[40]
authorAleks Kissinger &
authorJohn van de Wetering
(year2022): titleSimulating quantum circuits
with ZX-calculus reduced stabiliser decompositions.
journalQuantum Science and Technology
volume7(number4), p. pages044001,

[41]
authorEmanuel Knill &
authorRaymond Laflamme
(year1997): titleTheory of quantum
error-correcting codes.
journalPhysical Review A
volume55(number2), p. pages900,

[42]
authorEmanuel Knill,
authorRaymond Laflamme &
authorWojciech Zurek
(year1996): titleThreshold accuracy for quantum
journalarXiv preprint quant-ph/9610011,

[43]
authorDavid Kribs,
authorRaymond Laflamme &
authorDavid Poulin
(year2005): titleUnified and generalized
approach to quantum error correction.
journalPhysical Review Letters
volume94, p. pages180501,

[44]
authorAleksander Kubica &
authorMichael E. Beverland
(year2015): titleUniversal transversal gates
with color codes - a simplified approach.
journalPhysical Review A
volume91(number3), p. pages032330,

[45]
authorAleksander Kubica &
authorMichael Vasmer
(year2022): titleSingle-shot quantum error
correction with the three-dimensional subsystem toric code.
journalNature Communications
volume13(number1), p. pages6272,

[46]
authorAdrian Lehmann, authorBen
Caldwell & authorRobert Rand (year2022): titleVyZX: a
vision for verifying the ZX calculus.
journalarXiv preprint arXiv:2205.05781,

[47]
authorSebastian Leontica,
authorF Tennie &
authorT Farrow (year2021):
titleSimulating molecules on a cloud-based 5-qubit IBM-Q
universal quantum computer.
journalCommunications Physics
volume4(number1), p. pages112,

[48]
authorF. J. MacWilliams &
authorN. J. A. Sloane
(year1977): titleThe theory of error-correcting

[49]
authorTommy McElvanney &
authorMiriam Backens
(year2022): titleComplete flow-preserving
rewrite rules for MBQC patterns with Pauli measurements,

[50]
authorMichael A. Nielsen &
authorIsaac L. Chuang
(year2010): titleQuantum computation and quantum
information, edition10th anniversary ed edition.
publisherCambridge University Press,

[51]
authorAdam Paetznick &
authorBen W. Reichardt
(year2013): titleUniversal fault-tolerant
quantum computation with only transversal gates and error correction.
journalPhysical Review Letters
volume111, p. pages090505,

[52]
authorDavid Poulin (year2005):
titleStabilizer formalism for operator quantum error
journalPhysical Review Letters
volume95(number23), p. pages230504,

[53]
authorDong-Xiao Quan,
authorLi-Li Zhu,
authorChang-Xing Pei &
authorBarry C Sanders
(year2018): titleFault-tolerant conversion
between adjacent Reed–Muller quantum codes based on gauge
journalJournal of Physics A: Mathematical and
Theoretical volume51(number11), p.
pages115305, 10.1088/1751-8121/aaad13.

[54]
authorAlexis T. E. Shaw,
authorMichael J. Bremner,
authorAlexandru Paler,
authorDaniel Herr &
authorSimon J. Devitt
(year2022): titleQuantum computation on a
19-qubit wide 2d nearest neighbour qubit array,

[55]
authorPeter W. Shor
(year1995): titleScheme for reducing decoherence
in quantum computer memory.
journalPhysical Review A volume52,
pp. pagesR2493–R2496, 10.1103/PhysRevA.52.R2493.

[56]
authorWill Simmons (year2021):
titleRelating measurement patterns to circuits via Pauli
journalElectronic Proceedings in Theoretical
Computer Science volume343, pp. pages50–101,

[57]
authorAndrew Steane
(year1996): titleMultiple-particle interference
and quantum error correction.
journalProceedings of the Royal Society of
London. Series A: Mathematical, Physical and Engineering Sciences
volume452(number1954), pp.
pages2551–2577, 10.1098/rspa.1996.0136.

[58]
authorAndrew M Steane
(year1996): titleSimple quantum error-correcting
journalPhysical Review A
volume54(number6), p. pages4741,

[59]
authorAndrew M Steane
(year1999): titleQuantum Reed-Muller codes.
journalIEEE Transactions on Information Theory
volume45(number5), pp. pages1701–1703,

[60]
authorFelix Tennie & authorTim
Palmer (year2022):
titleQuantum computers for weather and climate prediction:
the good, the bad and the noisy.
journalarXiv preprint arXiv:2210.17460,

[61]
authorAlex Townsend-Teague &
authorKonstantinos Meichanetzidis
(year2022): titleClassifying complexity with the
ZX-calculus: Jones polynomials and Potts partition functions.
In: booktitleQuantum Physics and Logic,
seriesElectronic Proceedings in Theoretical Computer
Science (EPTCS 287) , addressOxford, p.

[62]
authorMichael Vasmer &
authorAleksander Kubica
(year2022): titleMorphing quantum codes.
journalPRX Quantum
volume3(number3), 10.1103/prxquantum.3.030319.

[63]
authorRenaud Vilmart
(year2019): titleA near-minimal axiomatisation
of ZX-calculus for pure qubit quantum mechanics.
In: booktitleLICS 2019, pp.
pages1–10, 10.1109/LICS.2019.8785765.

[64]
authorChristophe Vuillot,
authorLingling Lao, authorBen
Criger, authorCarmen García
Almudéver, authorKoen
Bertels & authorBarbara M Terhal (year2019): titleCode
deformation and lattice surgery are gauge fixing.
journalNew Journal of Physics
volume21(number3), p. pages033028,

[65]
authorQuanlong Wang
(year2018): titleQutrit ZX-calculus is
complete for stabilizer quantum mechanics.
journalElectronic Proceedings in Theoretical
Computer Science volume266, pp. pages58–70,

[66]
authorJohn van de Wetering
(year2020): titleZX-calculus for the working
quantum computer scientist.
journalarXiv preprint arXiv:2012.13966,

[67]
authorTheodore J. Yoder
(year2017): titleUniversal fault-tolerant
quantum computation with Bacon-Shor codes,

[68]
authorQingling Zhu, authorSirui
Cao, authorFusheng Chen, authorMing-Cheng Chen,
authorXiawei Chen,
authorTung-Hsun Chung,
authorHui Deng, authorYajie
Du, authorDaojin Fan, authorMing Gong et al.
(year2022): titleQuantum computational advantage
via 60-qubit 24-cycle random circuit sampling.
journalScience bulletin
volume67(number3), pp. pages240–245,