# Improving Pose Estimation through Contextual Activity Fusion

David Poulton  School of Computer Science and Applied Mathematics
University of the Witwatersrand
Johannesburg, South Africa
<EMAIL_ADDRESS>Richard Klein  School of Computer Science and
Applied Mathematics
University of the Witwatersrand
Johannesburg, South Africa
<EMAIL_ADDRESS>
###### Abstract

This research presents the idea of activity fusion into existing Pose
Estimation architectures to enhance their predictive ability. This is
motivated by the rise in higher level concepts found in modern machine
learning architectures, and the belief that activity context is a useful piece
of information for the problem of pose estimation. To analyse this concept we
take an existing deep learning architecture and augment it with an additional
1x1 convolution to fuse activity information into the model. We perform
evaluation and comparison on a common pose estimation dataset, and show a
performance improvement over our baseline model, especially in uncommon poses
and on typically difficult joints. Additionally, we perform an ablative
analysis to indicate that the performance improvement does in fact draw from
the activity information.

###### Index Terms:

pose estimation, computer vision, fully convolution neural networks, context
fusion

##  I Introduction

Human Pose Estimation (HPE) is a widely studied field of Machine Learning
focused on finding the joint positions of humans in an image. Initially
researchers developed models which functioned through the use of hand crafted
features, which yielded some success, however the incredible complexity of the
problem limited the viability of such methods. Some examples of pose
estimation annotations can be seen in Figure 1, composed of the predictions of
our final model on images randomly selected from the test set.

Over time, since the inception of neural networks, and specifically
Convolutional Neural Networks (CNNs), models began to shift towards fully
learned knowledge without the need for human crafted features or prior
information. CNNs enabled models to effectively handle image data due to the
nature of their design, which suited HPE and propelled model accuracies and
speeds, with networks becoming larger and more carefully structured.
Architectures such as DeepPose [17] and Stacked Hourglass [10] were
forerunners of deep learning for HPE.

The more complex structures that fell under deep learning began enabling
models to capture higher level concepts such as whole objects and bodies,
above local features or individual limbs [17, 12, 16], improving their
accuracies. This indicates that modern architectures are capable of utilising
these concepts, and designing them with this in mind can yield high performing
models. Following on from that, we predict that the use of contextual
information within pose estimators can further improve performance.

In humans, different activities generally tend to contain markedly different
poses, and in turn different poses tend to be found more amongst certain
activities, and so we choose activity as our contextual information of choice.
Our goal here is to determine whether knowing an image contains a certain
activity can drive the model to certain biases that improve its ability to
estimate poses. For example, knowing an image comes from a rugby game may
inform a pose estimator that it will likely encounter more running, diving,
and kicking, rather than sitting.

However, we also need to consider how images are categorised to activities,
which is not straightforward. Using one set of activities may be more
reasonable conceptually, where using another may better segment poses but be
less useful. For example, using “Playing soccer” is more practical than
decomposing it into “Running”, “Kicking”, and “Tackling”, even though the
latter decomposition may better separate the poses we expect to see.

Even finding an effective way to fuse this activity information into the
current deep learning networks available is not a trivial design choice due to
the complex nature of modern architectures. Above that, where to provide this
information to models requires some thought as well. We hope to address at
least some of these concerns through this research.

The remainder of the paper is structured as follows: Section II explores some
of the existing work in the field of pose estimation, as well as related
concepts of context fusion in imagery. Section III then covers the
architectural design choices for the research. Section IV provides detail on
how our chosen models were trained, followed by the results thereof in Section
V, including an ablative analysis of our method. We then provide some concepts
for future work in Section VI, followed by our conclusion in Section VII.

Figure 1: Some annotated examples from the test set, indicating the complexity
of the problem, and the large variety of poses the model is capable of
handling. The effect of having the target in the center of the image can also
be noticed in images where there are several people, even in close proximity.
Images are from the MPII dataset [2].

##  II Related Work

Early models of HPE utilised largely parts based models [13, 21], which were
successful at the time but fall well short of the accuracies enjoyed by modern
deep learning networks, and are not well suited to higher level conceptual
learning.

Figure 2: A visualisation of the original Stacked Hourglass architecture,
taken from Newell et al. [10].

Modern networks are far better suited to the task, and fall largely into one
of two categories, namely regression based or heatmap based models. DeepPose
[17] was a forerunning regression based model. It utilised a holistic method
to determine initial pose estimates, and then followed up with a series of
cascading CNNs to refine predictions on a per-joint basis.

On the other hand, models such as the Stacked Hourglass model [10] utilise
heatmaps entirely for joint predictions. The Hourglass model takes advantage
of contractions and expansions, as well as residuals, to find a good balance
between holistic, global features, and smaller, localised details in images.
Wei et al. [19] developed a similar concept of a repeated sequence of sub-CNN
modules which each produce a heatmap that is passed on to the next module,
inspired by [17]. This structure also displayed improvements in the ability of
the model to capture both global and local contextual pose information.

Some approaches also utilised model-based learning to restrict pose estimates
to realistic spaces. Sun et al. [14] implemented a bone-based representation
that allowed for learning of skeletal structure, rather than directly
predicting joint positions. They also adapted their loss functions to account
for errors specific to joints, and errors caused by misalignment of bones
between the root and current joint. Both these alterations successfully
increased accuracy over model-free methods seen before. Bourached et al. [4]
utilise a model-based generative architecture to try improve out-of-
distribution poses, enhancing the models ability to generalise, rather than
focus on more accurate predictions for known distributions.

Rather than using repeated stages of differing refinements, Chen et al. [5]
attempt to ensure realistic poses by utilising a GAN system, where adversaries
are used to determine how reasonable the generated pose and confidence maps
are. An ablative study indicates the GAN structure is indeed contributory to
accuracy improvements. Bin et al. [3] also utilise a GAN network to augment
inputs and make them easier to predict on, yielding the best performance
achieved on the MPII dataset.

As for the concept of activity fusion, it appears that the available
literature is relatively sparse. Rafi et al. [11] explores the concept of
semantic and contextual information by utilising a depth-based pose estimator
which is capable of identifying objects in the scene and using them for
context, however this only used a small set of objects. Vu et al. [18] also
uses contextual scene information in its estimation process which improved
performance, however the model is only applicable for head estimations,
whereas our focus is on full body poses.

Utilising depth information in their approach, Jalal et al. [8] extracts
skeletal information in order to produce an activity estimate, however their
activity classifications are much finer grained than those utilised here,
using classes such as walking or sitting rather than exercising. Rather than
depth, Gowda [6] uses RGB imagery and extracts poses to be used for broader
activity estimates, which is more relevant to the topic, and indicates there
is a possible relationship of significance between pose and activity.

Lan et al. [9] explores a similar concept of utilising contextual information
around an image, however in their use case it is applied to road image
segmentation rather than human poses. Their approach still yielded favourable
results, and is encouraging for our concept.

##  III Architecture

The utilised architecture was based off of the Stacked Hourglass model [10]
with some added layers for the activity fusion. The Stacked Hourglass model
was selected because of its high base accuracy, inherently flexible modular
design, and balance at finding global and local pose cues. It also maintains
the same shape of features throughout the majority of the model, specifically
$64\times 64\times 256$, making testing various fusion sites more
straightforward.

Because the Hourglass network is already designed with capturing global
context in mind, it was also of interest to see if explicitly providing the
context would have a significant impact, or if the model itself was already
capable of extracting the context in some sense.

We use the final version of the Stacked Hourglass model initially presented in
[10] as our baseline, composed of eight glasses, and utilising intermediate
supervision for training. A rough visualisation of the original network can be
seen in Figure 2.

Our baseline model still makes use of the initial down-convolution segment of
the network, as well as the intermediate bottlenecks and final remapping
convolutions, and regularisation and frequent batch normalisation. We then had
three sets of models, each composed of an ablative model and a contextual
model. The ablative models were the same as our baseline, however with a
single extra one-by-one convolution inserted at a specific point in the
network. The contextual model involved stacking the activity tensor on top of
the existing tensor in the model at the point, followed by a one-by-one
convolution, yielding the so called fused image that is then propagated
normally through the remainder of the network.

We make use of the one-by-one convolution in order to correct the number of
layers in the tensor after we have stacked our activity onto the previous
output. Throughout the majority of the network there are 256 channels in the
output of the layers, and so stacking our 21 channel activity tensor brings
the tensor to 277 channels. In order to minimise the changes needed to the
network, we then convolve the tensor back down to 256 layers. This method also
ideally allows the convolution to learn an effective mapping to merge our
context into the tensor, without having to rely on the existing layers in the
network. A diagram of this context block can be seen in Figure 4, compared to
the original hourglass block in Figure 3.

We utilise both an ablative and contextual version to verify that any possible
changes in the accuracy of the model are due to the impact of our activity
fusion, rather than the increase in the size of the model over the baseline.
Any increase in size or alteration to the flow of the network may be
significant enough to noticeably improve the network’s ability to learn
features, regardless of whether we provide activity context or not. Only
testing our contextual augmentations would not reveal the source of
improvement. This means we need to test both the contextual augmentation, as
well as an ablative version without the activity, to determine if any accuracy
changes are resultant from the one-by-one convolution itself, from the
contextual information, or from both. If our contextual augmentation
outperforms our ablative model, we can have confidence that the improvement is
owing to the context itself.

Figure 3: The original hourglass block format. The input to the module comes
from previous layer of the network, and the block produces output for the next
block as well as an intermediate heatmap prediction. Figure 4: The context
hourglass block used for our fusions. The input to the module comes from the
stacking of the activity tensor and the previous layer in the regular network.
The ablative version leaves out the activity tensor stacking.

For simplicity we refer to the three different augmented networks as A-, B-,
and C-Form, lettered from left-to-right according to their injection points,
as can be seen in Figure 5. A-Form has the fusion before the first hourglass,
B-Form before the fourth, and C-Form before the eighth. Their matching
ablative models are referred to as A-, B-, and C-Form Ablative. Each line in
the figure indicates a position where we tested our augmentation. Note that
only a single fusion point was used at a time for each model.

The decision of how to fuse the activity was made difficult by the
convolutional nature of the network, which stopped us from simply
concatenating on a one-hot encoded vector to our input. To get around this our
context took the form of a one-hot encoded activity tensor of size $64\times
64\times 21$, where 21 is the number of activities present in the dataset, and
$64\times 64$ is the shape of the image features as they move through the
network.

Figure 5: The augmented network, with lines indicating the three tested fusion
points. The input to the fusion modules come from the activity tensor and the
previous layer before the insertion point.

##  IV Training

Training was performed in a similar manner to that of the original paper [10]
for the MPII Human Pose dataset [1] only. The MPII dataset provides
approximately 25k images (2.5k of which are withheld as their own hidden test
set), leaving 22.5k images for use. Of those, approximately 18k (80%) were
used for training, with the remaining split evenly between a validation and
test set of 2.25k (10%) each.

The dataset provides a 2D ground-truth pose with joint visibility flags, an
approximate scale, and an approximate center position, among other data not
relevant to our analysis, for each person within an image. Images also have an
activity classification which is used for our context, which many other
datasets do not provide, making MPII particularly valuable. While MPII
provides more fine grained activity sub-classifications than the 21 activity
categories utilised, we did not use them as our context tensors would become
intractably large, and many of the sub-classifications had few or no samples.

Because images can contain more than one person, we follow the method of [10]
by cropping the image around the center of the target so the model knows which
person to estimate on, The images are cropped based on the provided scale and
the provided center coordinates of the target, and the crop is then scaled to
$256\times 256$. Cropped images also undergo some additional rotation (in
$[-30,30]$) and scaling (in $[0.75,1.25]$) to provide augmentation to the
dataset.

One issue encountered with the dataset was that utilising the scale provided
for each person did not always result in the full pose being in frame. The
scale value indicates a box of width 200px around the center of the target,
however even using $1.5\times$ the scale occasionally resulted in extremities
such as ankles or wrists not appearing in the final crop, which may lead to a
decrease in accuracy and increases the difficulty of identifying the already
challenging joints. Regardless, utilising the scale value as-is has been the
approach taken by several papers that utilise the MPII dataset [10, 3, 20,
15], and so we follow the same approach.

The cropping and augmentation can also result in black pixels appearing in the
final image provided to the network, as black background pixels can be
included from outside the bounds of the image. Obviously this is not very
realistic, however the complex networks are likely capable of learning to
ignore this anomaly. As a possible alternative, mirror padding could be used
when cropping, however, again, the simple background inclusion is the approach
taken by several papers [10, 3, 20, 15], and so we use the same method.

The baseline model was provided with images cropped around the center of the
person with small random rotations and scaling modifications, and a heatmap
was generated for each joint of the target pose as ground-truth for the model.
The augmented models were provided both with the image crop, and the one-hot
encoded tensor for the activity representation.

All models were trained with an MSE loss function over the ground-truth
heatmap and predicted heatmap set. The model makes heavy use of batch
normalisation [7] to improve training speed and performance. This allowed us
to use a relatively high learning rate of $2.5^{-4}$ with the RMSprop
optimiser, taken from the original Stacked Hourglass paper [10]. We utilised
our validation set to select the best model as that with the lowest validation
loss at any point of training.

Training of the models took approximately three days using an RTX3080 and a
batch size of 8, which was the largest we could achieve given the complexity
of the network. This meant a prediction time of 60ms per image. This was for
both our baseline model and our augmented model set, indicating that our
context pipeline has a negligible overhead.

##  V Results

While it does not imply an increase or decrease in accuracy, it should be
noted that results were evaluated using our own withheld test set comprised of
the released annotations, not the official MPII test set. This is due to the
now publicly available test set not having associated activities, and attempts
at getting activity annotations being unsuccessful. We utilised the common
<EMAIL_ADDRESS>metric, which represents the percentage of correct keypoints, where a
correct keypoint is defined as being within a threshold, specifically half the
normalised distance between the top of the neck and the top of the head, of
the ground-truth keypoint.

###  V-A Evaluation

Our final total accuracy for the best model, the C-Form model, was 90.3%, an
improvement of 2.7 percentage points over our baseline model’s performance of
87.6%. The contextual information was particularly impactful on some
conventionally difficult joints such as ankles, knees, and wrists, where the
contextual model improved over the baseline by 7.6%, 3.3%, and 1.5%
respectively. The accuracies of the different models can be seen in Table I.

TABLE I: Joint and total accuracies of different networks when run on our own test set. Model | Head | Neck | Torso | Pelvis | Shoulder | Elbow | Wrist | Hip | Knee | Ankle |<EMAIL_ADDRESS>
---|---|---|---|---|---|---|---|---|---|---|---
Baseline | 91.6 | 97.2 | 98.0 | 92.7 | 93.5 | 91.1 | 88.7 | 88.7 | 83.2 | 55.9 | 87.6
A-Form | 93.5 | 98.1 | 99.1 | 93.7 | 95.3 | 92.1 | 90.1 | 90.8 | 86.0 | 58.8 | 89.4
A-Form Ablative | 92.8 | 97.7 | 98.8 | 94.2 | 94.6 | 91.7 | 89.9 | 90.6 | 86.4 | 60.9 | 89.3
B-Form | 94.0 | 97.9 | 99.3 | 94.5 | 95.1 | 92.4 | 90.1 | 91.3 | 86.9 | 63.3 | 90.0
B-Form Ablative | 92.2 | 98.1 | 99.0 | 94.2 | 95.0 | 92.5 | 89.9 | 90.7 | 86.9 | 60.8 | 89.5
C-Form | 93.7 | 98.0 | 99.2 | 95.3 | 96.0 | 92.7 | 90.2 | 91.7 | 86.5 | 63.5 | 90.3
C-Form Ablative | 92.4 | 96.1 | 97.1 | 94.1 | 93.2 | 88.0 | 85.2 | 88.4 | 78.8 | 56.8 | 86.2

In terms of per-activity accuracies, our C-Form model showed noticeable
improvements on most activities, faring better in activities with more data
points. The average activity improvement was 2.5%. The apparent variance in
improvements per activity is likely due to the nature of the activities and
their compilations. For example, our model saw the largest improvement in the
“Water Activities” classification of 6.3%, where the poses are very different
from those found in other activities, with some subjects being in unusual
stances, obscured in scuba gear, or even upside down. A comparison on this
activity between the Baseline and C-Form model can be seen in Figure 7. The
“Miscellaneous” and “Bicycling” classifications on the other hand showed
minimal improvement, likely due to the relatively random grouping, and already
common poses that require minimal context, respectively. The different per-
activity accuracies can be seen in Figure 6.

Figure 6: The different activity accuracies for our baseline (blue), B-Form
Ablative (orange), and best augmented model, the C-Form model (green).
Activities are sorted left to right by the number of images in the test set,
ranging from 600+ (leftmost) to fewer than 10 (rightmost). Figure 7: Images
from our test set showcasing how the C-Form model (bottom) is better capable
of handling the relatively uncommon poses that fall under the “Water
Activities” classification, compared to the Baseline model (top).

Our C-Form model also shows consistent improvement over the baseline when
evaluating at varying PCKh thresholds, having a higher accuracy at every
value. This indicates that the performance improvement is actually caused by
overall better predictions, rather than by chance that a set of keypoints
moved within the threshold distance only for a specific value. This can be
seen in Figure 8.

Figure 8: The different accuracies when varying PCKh thresholds for our
baseline, best ablative model, and best contextual model

This indicates that our method of activity fusion may very well be useful if
our activities are structured well enough to segment very different poses and
cluster similar ones. Naturally this is in itself a challenging task, but
nevertheless means there may be room for improvement.

Interestingly, our results indicate some level of sensitivity of the method to
the fusion position. Our ablative models experienced a sharp decline in
performance towards the end of the model, with the C-Form actually performing
worse than our baseline. Alternatively, our contextual models performed
progressively better with fusions towards the end of the model. Only providing
context at the beginning of the network may be too early to enable the model
to effectively utilise the information for the final predictions, with the
signal from the context having diminished by the time it reaches the later
hourglasses.

In comparison to state of the art methods, our model seems to perform slightly
below the state of the art accuracies reported on the MPII test set [2].
Again, this is not an entirely useful comparison however, as our research was
unable to make use of the official test set, and so our true test result is
unknown. Nevertheless, the intent of the research was to indicate the
usefulness of activity fusion, which was apparent in our noticeable accuracy
improvements.

###  V-B Ablative Experiments

All of our contextual models fairly significantly outperformed the baseline
model in their overall accuracy, however we needed to perform an ablative test
to ensure the contextual information itself is actually contributory to our
results.

Our testing indicated that adding the additional one-by-one convolutions does,
in some cases, increase the performance of the network, namely the A-Form
Ablative and B-Form Ablative models. However, performance degrades in the
C-Form Ablative model, and so clearly the increase in the C-Form contextual
model’s accuracy is not only attributable to the additional convolution.

The best ablative model, the B-Form Ablative, showed a performance improvement
of 1.9% over the baseline. While such a margin may simply be caused by
variability in training, the consistent trend amongst the different
augmentation sets seems to indicate at least some level of impact by both the
architectural changes, as well as the provision of context. The overall
accuracies for each model can be seen in Table II.

TABLE II: Total accuracies for each model on our own test set. Model | Accuracy
---|---
Baseline | 87.6
A-Form | 89.4
A-Form Ablative | 89.3
B-Form | 90.0
B-Form Ablative | 89.5
C-Form | 90.3
C-Form Ablative | 86.2

It should be noted that the C-Form Ablative model also consistently performed
worse than the baseline model, which is unusual as the model could simply
learn an identity mapping and have no affect on our predictions. This drop in
accuracy could possibly be due to the degrading of gradients during back-
propagation, as the Stacked Hourglass makes extensive use of residuals, and
adding extra convolutions without skip layers may hinder this.

##  VI Future Work

While we have explored the basic concept of activity fusion in this report,
there are already numerous apparent avenues to continue analysing in various
domains.

In terms of the model utilised, this paper focused on a relatively
conceptually simple deep learning model, however in the future we could rather
make use of even newer state-of-the-art performers. Additionally we could
design our own network centered around the concept of activity fusion.

We can also perform more rigorous testing of our results, performing cross-
fold validation to account for the official test set not being usable, which
would eliminate the concern of random variance. Furthermore we could analyse
how our activity affects the features our model learns and makes use of,
should it have a significant impact.

Exploring more methods of encoding and fusing our context would also be
useful, as we only focused on straightforward one-hot encoding with
concatenation and a $1\times 1$ convolution to fuse the information.
Alternatives could involve using auto-encoders before providing the input to
the network, or other latent space methodologies. We could also investigate
different methods of fusing involving a more thorough merging strategy rather
than just a single convolution layer.

Finally we could explore the usefulness of context in other deep learning
fields. While pose estimation seems like a useful field for context provision,
there may be many others, such as image segmentation, or translation systems.

##  VII Conclusion

In this paper we explored the concept of fusing contextual activity
information into the existing Stacked Hourglass model. We show that even with
rudimentary organising of images into activities, and using straightforward
fusing methods, our method is capable of providing performance gains over a
baseline model. Above this our method introduced no significant overhead into
the training or prediction process.

Our method was capable of improving accuracy on typically difficult joints,
and is especially useful in activity classifications where poses are unusual
in comparison to the available training data. We also provide various avenues
for further exploration, and are hopeful that context fusion is a viable
addition to improving deep learning models in the field and beyond.

## Acknowledgments

We acknowledge the Centre for High Performance Computing (CHPC), South Africa,
for providing computational resources to this research project. Additionally,
this work is based on research supported in part by the National Research
Foundation of South Africa (Grant Numbers: 118075 and 117808).

## References

  * Andriluka et al. [2014a] M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele.  2d human pose estimation: New benchmark and state of the art analysis.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2014a.
  * Andriluka et al. [2014b] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele.  MPII Human Pose Dataset Results.  http://human-pose.mpi-inf.mpg.de/#results (November 2020), 2014b.
  * Bin et al. [2020] Yanrui Bin, Xuan Cao, Xinya Chen, Yanhao Ge, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, Changxin Gao, and Nong Sang.  Adversarial semantic data augmentation for human pose estimation.  In _European Conference on Computer Vision_ , pages 606–622. Springer, 2020.
  * Bourached et al. [2020] Anthony Bourached, Ryan-Rhys Griffiths, Robert Gray, Ashwani Jha, and Parashkev Nachev.  Generative model-enhanced human motion prediction.  _arXiv preprint arXiv:2010.11699_ , 2020.
  * Chen et al. [2017] Y. Chen, C. Shen, X.-S. Wei, L. Liu, and J. Yang.  Adversarial posenet: A structure-aware convolutional network for human pose estimation.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 1212–1221, 2017.
  * Gowda [2017] S. N. Gowda.  Human activity recognition using combinatorial deep belief networks.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , July 2017.
  * Ioffe and Szegedy [2015] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  _arXiv preprint arXiv:1502.03167_ , 2015.
  * Jalal et al. [2017] A. Jalal, Y.-H. Kim, Y.-J. Kim, S. Kamal, and D. Kim.  Robust human activity recognition from depth video using spatiotemporal multi-fused features.  _Pattern Recognition_ , 61:295 – 308, 2017.  ISSN 0031-3203.  doi: https://doi.org/10.1016/j.patcog.2016.08.003.  URL http://www.sciencedirect.com/science/article/pii/S0031320316302126.
  * Lan et al. [2020] Meng Lan, Yipeng Zhang, Lefei Zhang, and Bo Du.  Global context based automatic road segmentation via dilated convolutional neural network.  _Information Sciences_ , 535:156 – 171, 2020.  ISSN 0020-0255.  doi: https://doi.org/10.1016/j.ins.2020.05.062.  URL http://www.sciencedirect.com/science/article/pii/S0020025520304862.
  * Newell et al. [2016] A. Newell, K. Yang, and J. Deng.  Stacked hourglass networks for human pose estimation.  In _European conference on computer vision_ , pages 483–499. Springer, 2016.
  * Rafi et al. [2015] U. Rafi, J. Gall, and B. Leibe.  A semantic occlusion model for human pose estimation from a single depth image.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , pages 67–74, 2015.
  * Rafi et al. [2016] Umer Rafi, Bastian Leibe, Juergen Gall, and Ilya Kostrikov.  An efficient convolutional network for human pose estimation.  In _BMVC_ , volume 1, page 2, 2016.
  * Shakhnarovich et al. [2003] G. Shakhnarovich, P. Viola, and T. Darrell.  Fast pose estimation with parameter-sensitive hashing.  In _null_ , page 750. IEEE, 2003.
  * Sun et al. [2017] X. Sun, J. Shang, S. Liang, and Y. Wei.  Compositional human pose regression.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 2602–2611, 2017.
  * Tang et al. [2018] Zhiqiang Tang, Xi Peng, Shijie Geng, Lingfei Wu, Shaoting Zhang, and Dimitris Metaxas.  Quantized densely connected u-nets for efficient landmark localization.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 339–354, 2018.
  * Tompson et al. [2015] Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, and Christoph Bregler.  Efficient object localization using convolutional networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 648–656, 2015.
  * Toshev and Szegedy [2014] A. Toshev and C. Szegedy.  Deeppose: Human pose estimation via deep neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1653–1660, 2014.
  * Vu et al. [2015] T.-H. Vu, A. Osokin, and I. Laptev.  Context-aware cnns for person head detection.  In _The IEEE International Conference on Computer Vision (ICCV)_ , December 2015.
  * Wei et al. [2016] S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh.  Convolutional pose machines.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , pages 4724–4732, 2016.
  * Yang et al. [2017] Wei Yang, Shuang Li, Wanli Ouyang, Hongsheng Li, and Xiaogang Wang.  Learning feature pyramids for human pose estimation.  In _proceedings of the IEEE international conference on computer vision_ , pages 1281–1290, 2017.
  * Yang and Ramanan [2011] Y. Yang and D. Ramanan.  Articulated pose estimation with flexible mixtures-of-parts.  In _CVPR 2011_ , pages 1385–1392, 2011.


# A note on pseudofinite groups of finite centraliser dimension

Ulla Karhumäki  University of Helsinki<EMAIL_ADDRESS>
###### Abstract.

We give a structural theorem for pseudofinite groups of finite centraliser
dimension. As a corollary, we observe that there is no finitely generated
pseudofinite group of finite centraliser dimension.

The author is funded by the Finnish Science Academy grant no: 322795.

##  1\. Introduction

An infinite group is called _pseudofinite_ if every first-order sentence true
in it also holds in some finite group or, equivalently, if it is elementarily
equivalent (denoted by $\equiv$) to a non-principal ultraproduct of finite
groups.

Let $G$ be a group and $k$ be a positive integer. Then $G$ is said to be of
_centraliser dimension_ $k$ if the longest proper descending chain of
centralisers in $G$ has length $k$—this is denoted by ${\rm cd}(G)=k$. If such
an integer $k$ exists, then $G$ is called a group of _finite centraliser
dimension_.

By results of Wilson [16], the solvable radical is uniformly definable in the
class of _finite_ groups. Let ${\rm Rad}(H)$ be the solvable radical of a
finite group $H$. Then, for a pseudofinite group $G\equiv\prod_{i\in
I}G_{i}/U$, we have the well-defined definable normal subgroup
$R(G)\equiv\prod_{i\in I}{\rm Rad}(G_{i})/U$. By results of Khukhro [5], if
$G$ is of centraliser dimension $k$ then the subgroup $R(G)$ is solvable of
$k$-bounded derived length.

The _socle_ of a group $G$, denoted by ${\rm Soc}(G)$, is the subgroup
generated by all minimal normal non-trivial subgroups of $G$.

We prove the following theorem.

######  Theorem 1.1.

Let $G\equiv\prod_{i\in I}G_{i}/U$ be a pseudofinite group with ${\rm
cd}(G)=k$. Then $G$ has normal series of definable subgroups

$1\unlhd R(G)\unlhd L\unlhd G,$

where

  1. (1)

_(Khukhro[5])._ $R(G)$ is a solvable group of $k$-bounded derived length.

  2. (2)

$L$ is the full pre-image of $\overline{L}\equiv\prod_{i\in I}{\rm
Soc}(G_{i}/{\rm Rad}(G_{i}))/U$ in $G$.

  3. (3)

$\overline{L}=\overline{L}_{1}\times\cdots\times\overline{L}_{\ell}$ is a
direct product of $\ell$ many simple non-abelian groups $\overline{L}_{j}$,
each of which is either finite or isomorphic to a (twisted) Chevalley group
over a pseudofinite field. Also, $\ell$ depends only on $k$.

  4. (4)

$C_{G/R(G)}(\overline{L})=1$.

In [7], Macpherson and Tent proved that if $G$ is a _stable_ pseudofinite
group then the quotient $G/R(G)$ is finite; in this case, $R(G)$ is solvable
as stable groups are of finite centraliser dimension. In [8] they proved more
generally that if $G$ is a pseudofinite group whose theory is $NIP$ then
$G/R(G)$ is finite; here $R(G)$ needs not to be solvable. In [6], Macpherson
proved a similar theorem to Theorem 1.1$(\ref{pre-imge})$-$(\ref{socle})$ for
pseudofinite groups whose theory is $NTP_{2}$. These model-theoretic tameness
assumptions (stable, ${\rm NIP}$, ${\rm NTP_{2}}$) on $G$ are inherited by
definable quotients, such as $G/R(G)$.

The property of being of finite centraliser dimension is _not_ inherited by
(definable) quotients in general. However, if $H$ is a _locally finite_ group
of centraliser dimension $k$ then the quotient $H/{\rm Rad}(H)$ is of
$k$-bounded centraliser dimension. This fact is proven by Borovik and
Karhumäki [2] and by Buturlakin [3] in their independent proofs describing the
structure of locally finite groups of finite centraliser dimension. It
immediately follows that if $G$ is a pseudofinite group of centraliser
dimension $k$ then the quotient $G/R(G)$ is of $k$-bounded centraliser
dimension.

In the proof of Theorem 1.1 we use (the rather standard) techniques present
(at least) in [7, 8, 6, 2]. Namely, we observe that if the definable quotient
$G/R(G)$ (which is of finite centraliser dimension by [2, 3]) is infinite,
then it contain a definable normal subgroup isomorphic to a simple
pseudofinite group.

It is an open question due to Sabbagh whether finitely generated pseudofinite
groups exist. Neither solvable nor simple finitely generated pseudofinite
groups exist (see e.g. [9, Proposition 3.14]). By invoking Theorem 1.1, we
observe that no finitely generated pseudofinite group of finite centraliser
dimension exists.

######  Theorem 1.2.

There is no finitely generated pseudofinite group of finite centraliser
dimension.

This note is organised as follows. In Section 2 we give all necessary
background results on (pseudo)finite groups. Then, in Section 3, we prove
Theorem 1.1 and Theorem 1.2.

##  2\. Background results

###  2.1. Pseudofinite groups and ultraproducts

Pseudofinite groups arise in model theory111We omit the basics of model
theory; for such detail we refer the reader to [13]. as groups satisfying the
first-order properties that are true in all finite groups. Formally, the
definition goes as follows.

######  Definition 2.1.

An infinite group $G$ (resp. structure $M$) is called _pseudofinite_ if every
first-order sentence true in $G$ (resp. $M$) also holds in some finite group
(resp. structure).

The group $(\mathbb{Z},+)$ is an example of an infinite group which is _not_
pseudofinite; a first-order sentence expressing ‘if the map $x\mapsto 2x$ is
injective then it is surjective’ is true in all finite groups but false in
$(\mathbb{Z},+)$. Typical examples of pseudofinite groups are torsion-free
divisible abelian groups, infinite extraspecial groups of exponent $p>2$ and
rank $n$ and (twisted) Chevalley groups over pseudofinite fields.

######  Remark 2.2.

—

  * •

We always consider groups (resp. fields) in the pure group (resp. field)
language. Also, definable means definable possibly with parameters.

  * •

Some authors allow pseudofinite groups to be finite. We require that a
pseudofinite group is infinite. Therefore, given a pseudofinite group $G$, any
definable subgroup of $G$ is either finite or pseudofinite. Moreover, given a
definable normal subgroup $N$ of $G$, each of the groups $N$ and $G/N$ is
either finite or pseudofinite (see [9, Lemma 2.16]).

Let $\\{M_{i}:i\in I\\}$ be a family of $L$-structures, $U$ be a non-principal
ultrafilter on the index set $I$ and $M^{*}:=\prod_{i\in I}M_{i}$ be the
Cartesian product of the $M_{i}$’s. We say that a property $\star$ holds _for
almost all $i$_ if $\\{i:\star\,\,{\rm holds\,\,for}\,\,M_{i}\\}\in U$. We fix
$M=M^{*}/\sim_{U}$ for an equivalence relation $\sim_{U}$ on $M^{*}$ defined
as:

$x\sim_{U}y\,\,\,{\rm if\,\,and\,\,only\,\,if}\,\,\,\\{i\in I:x(i)=y(i)\\}\in
U,$

where $x,y\in M^{*}$, and $x(i)$ and $y(i)$ denote the $i^{th}$ coordinate of
$x$ and $y$, respectively. This is well-defined, and the resulting
$L$-structure $M=\prod_{i\in I}M_{i}/U$ is called the _ultraproduct_ of the
$L$-structures $M_{i}$ over the ultrafilter $U$.

The famous Łoś theorem (see e.g. [13, Exercise 1.2.4]) states that a first-
order expressible property is true in $M=\prod_{i\in I}M_{i}/U$ if and only if
it is true in $M_{i}$, for almost all $i$. Using this, one easily observes
that a structure is pseudofinite if and only if it is elementarily equivalent
to a non-principal ultraproduct of finite structures of increasing sizes.

####  2.1.1. Simple pseudofinite groups

In [1], Ax proved a beautiful purely algebraic characterisation for
pseudofinite fields. While no such characterisation exists for pseudofinite
groups, there is a tight correspondence between pseudofinite fields and simple
pseudofinite groups; in [15], Wilson proved that a simple pseudofinite group
is elementarily equivalent to a (twisted) Chevalley group over a pseudofinite
field. In [12, Chapter 5], Ryten proved that ‘elementarily equivalent’ can be
replaced by ‘isomorphic’ in Wilson’s result. As a (twisted) Chevalley group
over a pseudofinite field is a simple pseudofinite group [11], we have the
following classification.

######  Theorem 2.3 (Wilson [15] \+ Ryten [12]).

A pseudofinite group is simple if and only if it is isomorphic to a (twisted)
Chevalley group over a pseudofinite field.

A group is called _definably simple_ if it contains no proper non-trivial
_definable_ normal subgroups. In [14], Uğurlu proved that a definably simple
non-abelian pseudofinite group of finite centraliser dimension is simple. In
her proof, she in particular uses the following important result, which will
be useful for us too.

######  Theorem 2.4 (Ellers and Gordeev [4]).

Let $H$ be a finite simple group of (twisted) Lie type over a field with more
than $8$ elements. Then there exists a conjugacy class $C$ such that $H=CC$.

####  2.1.2. Semi-simple pseudofinite groups

Let $H$ be a _finite_ group and recall that we denote by ${\rm Rad}(H)$ the
_solvable radical_ (i.e. the maximal solvable normal subgroup) of $H$. In
[16], Wilson proved that there exists a parameter-free first-order formula
$\phi_{R}(x)$ so that $\phi_{R}(H)=\\{x:\phi(x)\\}={\rm Rad}(H)$.

Let $G\equiv\prod_{i\in I}G_{i}/U$ be pseudofinite and define $R(G)=\\{g\in
G:\phi_{R}(g)\\}$. The group $R(G)\equiv\prod_{i\in I}{\rm Rad}(G_{i})/U$ is a
definable normal subgroup of $G$ which contains all definable solvable normal
subgroups of $G$. In general, the subgroup $R(G)$ needs not to be solvable as
there may not be a common bound on the derived lengths of the finite solvable
groups ${\rm Rad}(G_{i})$. The subgroup $R(G)$ is trivial if and only if $G$
is _semi-simple_ , that is, has no non-trivial abelian normal subgroup (see
[9, Lemma 2.17]). So the notion of $R(G)$ allows one to split any pseudofinite
group $G$ to a semi-simple part $G/R(G)$ and to a ‘solvable-like’ part $R(G)$.
Note that, unlike simplicity, semi-simplicity is a first-order expressible
property by the first-order sentence $(\forall x\neq 1)(\exists
y)([x,x^{y}]\neq 1)$.

####  2.1.3. Finitely generated pseudofinite groups

The following natural question was raised by Sabbagh.

######  Question 2.5 (Sabbagh).

Are there any finitely generated pseudofinite groups?

There are several non-existence results related to Question 2.5, see [9,
Proposition 3.14]. In particular, the following in well-known.

######  Fact 2.6 (See e.g. [9, Proposition 3.14]).

Neither solvable nor simple finitely generated pseudofinite groups exist.

The following fact will be useful to us later on.

######  Fact 2.7 (Palacín [10, Lemma 2.2]).

Let $G$ be a finitely generated pseudofinite group and assume that it contains
an infinite definable definably simple non-abelian normal subgroup $N$. Then
the group $N$ is a finitely generated pseudofinite group.

###  2.2. (Pseudo)finite groups of finite centraliser dimension

Recall that a group $G$ is of centraliser dimension $k$ if the longest proper
chain of descending (equiv. ascending) centralisers has length $k$ and that
this is denoted by ${\rm cd}(G)=k$.

Clearly, one may express in a first-order way that a group is of centraliser
dimension (at most) $k$, for some _fixed_ $k$. So, by Łos’s theorem, a
pseudofinite group $G\equiv\prod_{i\in I}G_{i}/U$ is of centraliser dimension
(at most) $k$ if and only if the finite group $G_{i}$ is of centraliser
dimension (at most) $k$, for almost all $i$.

The following result by Khukhro immediately implies that the subgroup $R(G)$
of a pseudofinite group $G$ with ${\rm cd}(G)=k$ is solvable of $k$-bounded
derived length.

######  Theorem 2.8 (Khukhro [5, Theorem 2]).

Let $G$ be a group elementarily equivalent to an ultraproduct of finite
solvable groups. If $G$ is of ${\rm cd}(G)=k$ then $G$ is solvable of
$k$-bounded derived length.

Another important result for us is the following.

######  Fact 2.9 (Buturlakin [3, Proposition 2.1], see also [2]).

Let $H$ be a finite group with ${\rm cd}(H)=k$. Then the centraliser dimension
of $H/{\rm Rad}(H)$ is $k$-bounded.

It follows immediately from Fact 2.9 that if $G$ is a pseudofinite group with
${\rm cd}(G)=k$, then the definable quotient $G/R(G)$ is of $k$-bounded
centraliser dimension.

######  Remark 2.10.

When we say that a group $G$ is of _$k$ -bounded_ centraliser dimension (resp.
has $k$-bounded derived length) we mean that the finite centraliser dimension
of $G$ depends on $k$ only (resp. $G$ is solvable of derived length depending
on $k$ only).

##  3\. Proofs of Theorems 1.1 and 1.2

In this section we prove Theorems 1.1 and 1.2.

Recall that the socle ${\rm Soc}(H)$ of a group $H$ is the subgroup generated
by all minimal normal non-trivial subgroups. We say that ${\rm Soc}(H)=1$, if
$H$ has no minimal normal non-trivial subgroups. If $H\neq 1$ is a finite
group, then its socle ${\rm Soc}(H)\neq 1$ is a finite direct product of
simple finite groups.

The following easy fact is rather useful; we write down its proof for the
readers convenience.

######  Fact 3.1.

Let $G\equiv\prod_{i\in I}G_{i}/U$ be a pseudofinite group. Assume that there
is $d\in\mathbb{N}$ so that $|{\rm Soc}(G_{i}/{\rm Rad}(G_{i}))|\leqslant d$
for almost all $i$. Then $G/R(G)$ is finite.

###### Proof.

Denote $\overline{G}=G/R(G)$ and $\overline{G}_{i}=G_{i}/{\rm Rad}(G_{i})$.

The following holds for almost all $i$. We may assume that
$\overline{G}_{i}\neq 1$ and hence $1\neq|{\rm
Soc}(\overline{G}_{i})|\leqslant d$. This means that there is a common bound
on the sizes of the automorphism groups $|{\rm Aut}({\rm
Soc}(\overline{G}_{i}))|$. As the socle ${\rm Soc}(\overline{G}_{i})$ is a
characteristic subgroup of $\overline{G}_{i}$, the finite group
$\overline{G}_{i}$ embeds into ${\rm Aut}({\rm Soc}(\overline{G}_{i}))$. So
there is a common bound on the sizes $|\overline{G_{i}}|$ and hence
$\overline{G}\equiv\prod_{i\in I}\overline{G}_{i}/U$ is finite. ∎

###  3.1. Proof of Theorem 1.1

Let $G$ be a pseudofinite group of ${\rm cd}(G)=k$. Denote
$\overline{G}=G/R(G)$ and $\overline{G}_{i}=G_{i}/{\rm Rad}(G_{i})$.

Observe first that $(\ref{radical})$ holds by Theorem 2.8.

As the finite groups $\overline{G}_{i}$ are semi-simple, the socles
$\overline{S}_{i}={\rm Soc}(\overline{G}_{i})$ are finite direct products
$\overline{S}_{1_{i}}\times\cdots\times\overline{S}_{r_{i}}$ of non-abelian
simple groups $\overline{S}_{j_{i}}$. We start by observing that for almost
all $i$ the number $r_{i}$ of the components $\overline{S}_{j_{i}}$ is bounded
by some $\ell$ depending on $k$ only.

######  Claim 1.

For almost all $i$, there is $\ell\in\mathbb{N}$, depending on $k$ only, such
that $r_{i}\leqslant\ell$.

Assume contrary. Then, for almost all $i$, $r_{i}>\ell$ for all
$\ell\in\mathbb{N}$. So, by setting $j=1,\ldots,\ell$, one may choose an
element $x_{j_{i}}\in\overline{S}_{j_{i}}\setminus\\{1\\}$ so that

$C_{\overline{S}_{i}}(x_{1_{i}})>C_{\overline{S}_{i}}(x_{1_{i}},x_{2_{i}})>\ldots>C_{\overline{S}_{i}}(x_{1_{i}},x_{2_{i}},\ldots,x_{\ell_{i}})$

is a proper descending chain of centralisers in $\overline{G}_{i}$, of length
$\ell$, for an arbitrary large $\ell$. But the finite group $G_{i}$ is of
centraliser dimension $k$, and therefore, by Fact 2.9, the quotient
$\overline{G}_{i}$ is of centraliser dimension $k_{i}\in\mathbb{N}$ for some
$k_{i}$ depending on $k$. This contradiction proves that there is
$\ell\in\mathbb{N}$ so that $r_{i}\leqslant\ell$ for almost all $i$. Clearly,
the bound $\ell$ depends on $k$ only. ∎

We may now assume that for almost all $i$, the socle $\overline{S}_{i}$ is a
direct product of $\ell$ many simple non-abelian finite groups. Next we
observe that $\overline{L}\equiv\prod_{i\in I}\overline{S}_{i}/U$ is a direct
product of $\ell$ many simple non-abelian groups.

######  Claim 2.

$\overline{L}\equiv\prod_{i\in I}\overline{S}_{i}/U$ is a direct product of
$\ell$ many simple non-abelian groups $\overline{L}_{j}$, each of which is
either finite or isomorphic to a (twisted) Chevalley group over a pseudofinite
field. In particular, $\overline{L}$ is a definable normal subgroup of
$\overline{G}$.

At this point we know that, for almost all $i$, the finite group
$\overline{S}_{i}=\overline{S}_{1_{i}}\times\cdots\times\overline{S}_{\ell_{i}}$
is a product of $\ell$ many simple non-abelian groups. Let
$j\in\\{1,\ldots,\ell\\}$ and consider $\overline{L}_{j}\equiv\prod_{i\in
I}\overline{S}_{j_{i}}/U$. If $\overline{L}_{j}$ is finite then, for almost
all $i$, $\overline{S}_{j_{i}}$ is the same non-abelian finite simple group,
and hence, $\overline{L}_{j}\cong\overline{S}_{j_{i}}$ is a finite non-abelian
simple group.

Assume that $\overline{L}_{j}$ is infinite. We may assume that just one of the
three families of non-abelian finite simple groups occurs in the ultraproduct
$\overline{L}_{j}$. This family cannot be the sporadic groups as there are
only finitely many of them. One may also easily observe that this family
cannot be the alternating groups; if for almost all $i$ the finite group
$\overline{S}_{j_{i}}$ was isomorphic to some alternating group ${\rm
Alt}_{n_{i}}$ for $n_{i}\geqslant 5$, then the sizes of ${\rm Alt}_{n_{i}}$’s
were of increasing orders as $\overline{L}_{j}$ is infinite. However, as the
centralisers of products of even number of disjoint transpositions in ${\rm
Alt}_{n_{i}}$ form a proper centraliser chain of length at least
$\lfloor\frac{n_{i}}{4}\rfloor$, the ultraproduct $\overline{L}_{j}$ of
alternating groups of increasing orders would not be of finite centraliser
dimension. So $\overline{S}_{j_{i}}$ is a simple group of Lie type over a
finite field, for almost all $i$.

As there are only finitely many different Lie types, we may assume that for
almost all $i$ the groups $\overline{S}_{j_{i}}$ are of the same Lie type $X$.
We still need to ensure that the Lie ranks $n_{i}$ of $\overline{S}_{j_{i}}$’s
cannot be arbitrarily large. This follows from the structures of finite simple
groups of Lie type; as explained in [7, page 554, proof of Claim 2], if there
is no bound on the Lie ranks $n_{i}$, then $\overline{L}_{j}$ is not of finite
centraliser dimension. Therefore, for almost all $i$, the groups
$\overline{S}_{j_{i}}$ are of the same Lie type $X$ with fixed Lie rank, say
$n$, over finite fields of increasing orders. So, by Theorem 2.3,
$\overline{L}_{j}$ is simple and isomorphic to a (twisted) Chevalley group
over a pseudofinite field.

Finally, we need to ensure that each component
$\overline{L}_{j}\equiv\prod_{i\in I}\overline{S}_{j_{i}}/U$ is definable in
$\overline{G}$. Again, we may assume that $\overline{L}_{j}$ is infinite.
Then, for almost all $i$, the group $\overline{S}_{j_{i}}$ is a simple group
of Lie type over a field of more than $8$ elements. Therefore, by Theorem 2.4,
the following holds for almost all $i$. There is
$x_{j_{i}}\in\overline{S}_{j_{i}}$ such that

$\overline{S}_{j_{i}}=x_{j_{i}}^{\overline{S}_{j_{i}}}x_{j_{i}}^{\overline{S}_{j_{i}}}.$

As $\overline{S}_{j_{i}}\unlhd\overline{G}_{i}$, we have

$x_{j_{i}}^{\overline{G}_{i}}x_{j_{i}}^{\overline{G}_{i}}=\overline{S}_{j_{i}}.$

This means that $\overline{S}_{j_{i}}$ is uniformly definable in
$\overline{G}_{i}$, for almost all $i$. It follows that each component
$\overline{L}_{j}$ is a definable subgroup of $\overline{G}$. We have observed
that $\overline{L}$ is a definable subgroup of $\overline{G}$. Moreover, as
$\overline{S}_{i}\unlhd\overline{G}_{i}$, we have
$\overline{L}\unlhd\overline{G}$. ∎

At this point, setting $L$ to be the full pre-image of $\overline{L}$ in $G$,
parts $(\ref{radical})$–$(\ref{socle})$ of Theorem 1.1 are proven. We finish
the proof by observing $(\ref{centraliser})$.

######  Claim 3.

$C_{\overline{G}}(\overline{L})=1$.

We first observe that $C_{\overline{G}}(\overline{L})\cap\overline{L}=1$.
Assume contrary. Then, as centralisers are definable in a group of finite
centraliser dimension, the centraliser
$C_{\overline{L}}(\overline{L})=\overline{C}$ is a non-trivial definable
abelian normal subgroup of $\overline{L}$. Thus $\overline{C}\equiv\prod_{i\in
I}\overline{C}_{i}/U$, where $\overline{C}_{i}$ is an abelian normal subgroup
of the socle $\overline{S}_{i}$. But $\overline{S}_{i}$ is a direct product of
$\ell$ many simple non-abelian groups $\overline{L}_{j_{i}}$. That is,
$\overline{C}_{i}\cap\overline{L}_{j_{i}}=1$ for all
$j\in\\{1,\ldots,\ell\\}$; this contradiction proves that $\overline{C}=1$.

Clearly $C_{\overline{G}}(\overline{L})$ is a normal definable subgroup of
$\overline{G}$. Therefore, if $C_{\overline{G}}(\overline{L})\neq 1$, then
$C_{\overline{G}}(\overline{L})\equiv\prod_{i\in
I}\overline{K}_{i}/\mathcal{U}$, where $\overline{K}_{i}$ is a normal subgroup
of $\overline{G}_{i}$ intersecting $\overline{L}_{i}$ trivially. But this is
impossible as then $C_{\overline{G}}(\overline{L})$ would intersect
$\overline{L}$ non-trivially. ∎

Theorem 1.1 is now proven. ∎

###  3.2. Proof of Theorem 1.2

Let $G\equiv\prod_{i\in I}G_{i}/U$ be a finitely generated pseudofinite group
of finite centraliser dimension. We retain our earlier notation, that is,
$\overline{G}=G/R(G)$, $\overline{G}_{i}=G_{i}/{\rm Rad}(G_{i})$,
$\overline{S}_{i}={\rm Soc}(\overline{G}_{i})$, $\overline{L}\equiv\prod_{i\in
I}\overline{S}_{i}/U$ and $L$ is the full pre-image of $\overline{L}$ in $G$.

By Theorem 1.1, $G$ has normal series of definable subgroups

$1\unlhd R(G)\unlhd L\unlhd G,$

where $R(G)$ is solvable and $\overline{L}$ is a finite direct product of
$\ell$ many non-abelian simple groups $\overline{L}_{j}$, each of which is
either finite or pseudofinite. Moreover, $C_{\overline{G}}(\overline{L})=1$.
Therefore, the group $G$, in its action on $\overline{L}$ by conjugation,
permutes the simple subgroups $\overline{L}_{j}$ and the kernel of this
permutation action, say $G^{\circ}$, is a normal subgroup of finite index in
$G$. Without a loss of generality, we may assume that $G^{\circ}=G$ and each
$\overline{L}_{j}\unlhd\overline{G}$.

######  Claim 4.

$\overline{L}$ is finite.

Towards a contradiction, assume that $\overline{L}$ is infinite. Then, for
some $j\in\\{1,\ldots,\ell\\}$, the non-abelian simple group
$\overline{L}_{j}$ is infinite and hence isomorphic to a (twisted) Chevalley
group over a pseudofinite field. Therefore, the group $\overline{L}_{j}$ is an
infinite non-abelian (definably) simple normal definable subgroup of the
finitely generated group $\overline{G}$. So, by Fact 2.7, $\overline{L}_{j}$
is finitely generated. Now $\overline{L}_{j}$ is a finitely generated simple
pseudofinite group—this is a contradiction in light of Theorem 2.6. ∎

As $\overline{L}$ is finite there is an upper bound on the sizes
$|\overline{S}_{i}|$, for almost all $i$. Therefore, by Fact 3.1,
$\overline{G}$ is finite. This means that the subgroup $R(G)$ is a solvable
finitely generated pseudofinite group, contradictory to Theorem 2.6. Theorem
1.2 is now proven. ∎

## References

  * [1] James Ax.  The Elementary Theory of Finite Fields.  Annals of Mathematics, 88(2):239–271, 1968.
  * [2] Alexandre V. Borovik and Ulla Karhumäki.  Locally finite groups of finite centralizer dimension.  Journal of Group Theory, 22:729–740, 2019.
  * [3] Alexandre A. Buturlakin.  The structure of locally finite groups of finite $c$-dimension.  Journal of Algebra and Its Applications, 18(12), 2019.
  * [4] Erich W. Ellers and Nikolai Gordeev.  On the conjectures of J. Thompson and O. Ore.  Transactions of the American Mathematical Society, 350(9):3657–3641.
  * [5] Evgeny I. Khukhro.  On solubility of groups with bounded centralizer chains.  Glasgow Mathematical Journal, 51(1):49–54, 2009.
  * [6] Dugald Macpherson.  Model theory of finite and pseudofinite groups.  Arch. Math. Logic, 57, 2018.
  * [7] Dugald Macpherson and Katrin Tent.  Stable pseudofinite groups.  Journal of Algebra, 312:550–561, 2007.
  * [8] Dugald Macpherson and Katrin Tent.  Pseudofinite groups with NIP theory and definability in finite simple groups.  In L. Strüngmann, M. Droste, L. Fuchs, K. Tent, (eds.) Groups and Model Theory, volume 576, pages 255–267. American Mathematical Society, Providence, 2012.
  * [9] Abderezak Ould Houcine and Françoise Point.  Alternatives for pseudofinite groups.  Journal of Group Theory, 16(4):461–495, 2013.
  * [10] Daniel Palacín.  Finite rank and pseudofinite groups.  Journal of Group Theory, 21(4):583–591, 2018.
  * [11] Françoise Point.  Ultraproducts and Chevalley groups.  Archive for Mathematical Logic, 38(6):355–372, 1999.
  * [12] Mark J. Ryten.  Model Theory of Finite Difference Fields and Simple Groups.  PhD thesis, University of Leeds, 2007.
  * [13] Katrin Tent and Martin Ziegler.  A Course in Model Theory.  Lecture Notes in Logic. Cambridge University Press, 2012.
  * [14] Pınar Uğurlu.  Pseudofinite groups as fixed points in simple groups of finite Morley rank.  Journal of Pure and Applied Algebra, 217(5):892–900, 2013.
  * [15] John S. Wilson.  On simple pseudofinite groups.  Journal of the London Mathematical Society, 51(2):471–490, 1995\.
  * [16] John S. Wilson.  First-order characterization of the radical of a finite group.  Journal of Symbolic Logic, 74:1429–1435, 2009.


# MultiQG-TI: Towards Question Generation from Multi-modal Sources

Zichao Wang
Adobe Research
<EMAIL_ADDRESS>
&Richard G. Baraniuk
Rice University
<EMAIL_ADDRESS>
Work done while at Rice University.

###### Abstract

We study the new problem of automatic question generation (QG) from multi-
modal sources containing images and texts, significantly expanding the scope
of most of the existing work that focuses exclusively on QG from only textual
sources. We propose a simple solution for our new problem, called MultiQG-TI,
which enables a text-only question generator to process visual input in
addition to textual input. Specifically, we leverage an image-to-text model
and an optical character recognition model to obtain the textual description
of the image and extract any texts in the image, respectively, and then feed
them together with the input texts to the question generator. We only fine-
tune the question generator while keeping the other components fixed. On the
challenging ScienceQA dataset, we demonstrate that MultiQG-TI significantly
outperforms ChatGPT with few-shot prompting, despite having hundred-times less
trainable parameters. Additional analyses empirically confirm the necessity of
both visual and textual signals for QG and show the impact of various modeling
choices. Code is available at https://rb.gy/020tw

##  1 Introduction

Automatic question generation has the potential to enable personalized
education experiences for subjects such as reading comprehension at a large
scale Wolfe (1976); Kokku et al. (2018); Zhang et al. (2022); Kulshreshtha et
al. (2022) and improve standardized tests by reducing the costs and the test
length Burstein et al. (2021). Most, if not all, existing question generation
(QG) methods operate only on text: they take a textual paragraph Wang et al.
(2018) or story Xu et al. (2022) as input and generate a textual question.
These methods’ focus on text-based QG is limiting, because many interesting
questions can involve, or be generated from, multiple modalities such as
images, diagrams, and tables, in addition to texts Lu et al. (2022).

Figure 1: Illustration of our multi-modal question generation (QG) problem.
Given a background text and an image, our goal is to develop a model to
automatically generate a textual question based on them. Figure 2:
Illustration of the proposed MultiQG-TI methodology.

###  1.1 Contributions

In this paper, we conduct, to our knowledge, the first investigation into the
under-explored problem of multi-modal question generation (QG). Specifically,
we study the following problem: given multi-modal inputs containing both
visual (e.g., an image) and textual (e.g., a textbook paragraph) information,
we would like a model to output a textual question based on such multi-modal
input. Note that the definition of visual input is very broad, e.g., it can be
an image, a diagram, or a table in the image format. Although this multi-modal
setting (image and text as input and textual question as output) is only a
specific instance of multi-modality (one could consider using audio and video
as input to generate questions, or generating questions with images in
addition to texts), we argue that our setting is sufficiently broad and
educationally meaningful. For example, many science questions ask about
scientific phenomena, processes, and relationships commonly described in
figures, diagrams, and tables Talmor et al. (2021); Lu et al. (2022). We
believe that our problem setting, illustrated in Figure 1, is an important
first step toward more general multi-modal QG.

We propose a novel method, dubbed MultiQG-TI, for generating textual questions
from multi-modal inputs of texts and images. The idea is simple: we enable a
text-based question generator to “see” by feeding it visual information in the
form of text. Specifically, we first use an off-the-shelf image-to-text model
and an optical character recognition (OCR) model to produce a textual
description of the image and extract the texts in the image. We then fine-tune
a text-based generative model to generate a question given the input text and
the text extracted from the input image. These components are readily
available and require no or minimal fine-tuning, making MultiQG-TI easy to use
and efficient to train. Figure 2 presents a high-level overview of MultiQG-TI.

We demonstrate MultiQG-TI’s strong performance on the challenging ScienceQA
dataset Lu et al. (2022). For example, MultiQA-TI outperforms models using
only texts or only images as input, demonstrating the necessity of including
both texts and images as input in QG. MultiQA-TI also significantly
outperforms ChatGPT in the few-shot in-context learning setting, demonstrating
its competitiveness against much larger models. Finally, we analyze the
factors that impact MultiQA-TI’s performance, including the choices of image-
to-text models and the sizes of the question generator model. We also provide
generation examples to illustrate our method’s strengths and errors.

###  1.2 Related Work

#### Question generation (QG) for education.

QG models are often an integral component in personalized learning,
intelligent tutoring systems, and assessment platforms to cheaply and scalably
generate customized questions for each student Le et al. (2014); Pan et al.
(2019); Srivastava and Goodman (2021); White et al. (2022). For example, prior
research has developed models to generate a variety of questions including
those based on fairytales Xu et al. (2022); Zhao et al. (2022), factual
questions Heilman and Smith (2010); Wang et al. (2018), and math word problems
Wang et al. (2021); Liu et al. (2021). Despite the rapid progress, most
existing work focuses on textual-based QG. The exciting frontier of automatic
multi-modal QG remains under-explored.

#### Multi-modal processing with text-only models.

Our work is partially motivated by the recent line of work that demonstrate
the possibility to use text-only models to perform visual-related tasks by
feeding it text descriptors of the visual input. For example, Wang et al.
(2022) enable large language models to perform video-related tasks such as
event prediction by connecting them with image-to-text models. A few others
take a similar approach to enable text-only models to perform captioning,
reasoning, and question answering that involve videos or images Yang et al.
(2022, 2023); Hu et al. (2022). However, the utility of their approach for
multi-modal QG remains largely known.

##  2 The MultiQG-TI Methodology

We now describe the four modules in MultiQG-TI: a question generator module,
an image-to-text module, an optical character reconigion (OCR) module, and an
input formatting module.

#### The question generator module.

This module generates the question and is the only trainable module in
MultiQG-TI. We adopt a text-based question generator such that its inputs must
be all in text format. Adopting a text-based question generator enables us to
choose from a wide range of pre-trained text-based generative models, whose
training is also often more efficient than their multi-modal counterparts. In
this work, we instantiate the question generator with the recent Flan-T5 model
Chung et al. (2022) that have shown to perform strongly on new downstream
tasks when fine-tuned on limited task-specific data.

#### The image-to-text and OCR modules.

A text-based question generator cannot take any visual input. To solve this
problem, we use the image-to-text and OCR modules to interface between the
image and text modalities and extract the visual information from the image
format into a textual format appropriate as input for the text-based question
generator. In particular, we use the image-to-text module to describe the
content in the image in texts, including any objects, scenes, actions, and
events. We instantiate this module with the Flan-T5-XXL version of BLIP-2 Li
et al. (2023). While the image-to-text module extracts visually rich signals,
it often fails to recognize any text in the image. This is problematic if the
majority of the content in the image is text, such as a table. Therefore, we
complement the image-to-text module with an OCR module that specializes in
extracting the texts in the image. We instantiate the OCR module in MultiQG-TI
with PaddleOCR Du et al. (2020).

#### The input formatting module.

This module, $g$, is a simple function that concatenates the input text and
the texts from the input image into one coherent textual input for the
question generator model. There are many choices available and one can simply
perform a string join operation. In this work, we apply input formatting with
the following template: Generate a question based on the following
information. Background: {input_text}. Image: {image_description}. Texts in
image: {image_text}.. In this template, {input_text}, {image_description}, and
{image_text} are placeholders that will be replaced with the actual input
text, the output from the image-to-text model and the output from the OCR
module, respectively.

Table 1: MultiQG-TI (marked bold) significantly outperforms ChatGPT as well as
variants with a single modality input across almost all metrics.

Method BLEU METEOR ROUGE BLEURT ChatGPT 0 shot 0.014 0.264 0.209 0.448 ChatGPT
1 shot 0.021 0.298 0.208 0.434 ChatGPT 3 shot 0.063 0.332 0.266 0.449 ChatGPT
5 shot 0.088 0.346 0.301 0.464 ChatGPT 7 shot 0.089 0.342 0.307 0.460 MultiQG-
TI 0.725 0.829 0.830 0.757 \- text only 0.570 0.714 0.718 0.675 \- image only
0.714 0.817 0.813 0.760

#### Training and inference.

During training, we only update the parameters of the QG module while keeping
the other modules fixed. We use the next word prediction as the training
objective, which is commonly used in modern language model training Vaswani et
al. (2017). During inference, we proceed as follows: given an input image and
text, we first extract the text from the image using image-to-text module and
the OCR module, then format them together with the input text, and finally
feed the formatted texts to the fine-tuned QG module to generate a question.

Table 2: An example of a question in physics generated by MultiQG-TI.

Input background text Input image Magnets can pull or push on other magnets
without touching them. When magnets attract, they pull together. When magnets
repel, they push apart. These pulls and pushes are called magnetic forces.
Magnetic forces are strongest at the magnets’ poles, or ends. Every magnet has
two poles: a north pole (N) and a south pole (S). Here are some examples of
magnets. Their poles are shown in different colors and labeled. Whether a
magnet attracts or repels other magnets depends on the positions of its poles.
If opposite poles are closest to each other, the magnets attract. The magnets
in the pair below attract. If the same, or like, poles are closest to each
other, the magnets repel. The magnets in both pairs below repel. MultiQG-TI
generated question Two magnets are placed as shown. Will these magnets attract
or repel each other?

##  3 Experiments

#### Dataset.

We use the ScienceQA dataset Lu et al. (2022) throughout our experiments,
which we preprocess and split into training, validation, and test splits. All
results in this paper are reported on the test split. More details on the
dataset and preprocessing steps are in Appendix A.1.

#### Baselines.

Because there are no prior work on automatic multi-modal QG, we use off-the-
shelf model APIs and variants of MultiQG-TI as the baselines. Specifically, we
use ChatGPT API Ouyang et al. (2022) with zero-shot and in-context learning
Kaplan et al. (2020); Wei et al. (2022) with up to seven examples, each of
which is formatted exactly the same as our preprocessed data points in the
ScienceQA dataset. We also compare with MultiQG-TI with only a single modality
as input (i.e., either only text or only image).

#### evaluation.

We choose four evaluation metrics including BLEU, METEOR, ROUGE, and BLEURT,
all of which have been widely used in existing QG works. We report all
results, except for those using ChatGPT API, based on the average of 4 random,
independent runs. More details on the experiment setup, baselines, and
evaluation are in Appendices A.2 and A.3.

Figure 3: Larger model tends to result in improved QG performance across all
metrics.

###  3.1 Main quantitative results

Table 1 summarizes the main results.111For conciseness, we choose not to
report standard deviations because all of them are quite small (around
$0.002$). These results clearly show that ChatGPT fails at the multi-modal QG
task in our setting. Although its performance steadily improves with more
examples in the in-context learning setting, ChatGPT trails MultiQG-TI by a
gigantic margin. The comparison between ChatGPT and MultiQG-TI reminds one to
be cautious when using ChatGPT in specialized tasks such as multi-modal QG and
presents strong empirical evidence that a small, fine-tuned model is still
highly relevant in certain generation tasks. Table 1 also demonstrate the
benefits of including both the visual and textual information when generating
questions because MultiQG-TI outperforms its variants with only textual or
only visual input.

###  3.2 Analyses

#### The choice of question generators.

We study the impact of the model size of the QG module on the QG performance
and summarize the results in Figure 3, where “small”, “medium”, and “large”
represent the Flan-T5 variants of 80 million, 250 million, and 780 million
parameters, respectively. The figure implies that a larger model generally
leads to improved performance across all evaluation metrics. Notably, by fine-
tuning only on a few thousand training examples with a modest-sized model,
MultiQG-TI achieves high performance,222As a comparison, some of the latest QG
works achieve a BLEURT score of up to 0.67; see the results of a recent QG
competition: https://www.thequestchallenge.org/leaderboard making it appealing
for practical use and deployment in resource-constrained settings.

Table 3: A sufficiently large image-to-text model leads to better QG
performance, although the benefit of model size diminishes as the size
increases beyond 2.7 billion parameters.

ViT model bleu_4 meteor rouge bleurt ViT-GPT2 (239M) 0.671 0.79 0.785 0.733
BLIP2-OPT (2.7b) 0.744 0.843 0.843 0.770 BLIP2-OPT (6.7b) 0.743 0.842 0.841
0.773 BLIP2-Flan-T5-XXL (11b) 0.747 0.846 0.844 0.773

#### The choice of image-to-text models.

We also study the impact of the image-to-text models on the QG performance and
summarize the results in Table 3. Specifically, we compare BLIP2-Flan-T5-XXL
(11 billion parameters), the image-to-text model we use in MultiQG-TI, to
three smaller variants ranging from 239 million to 2.7 billion, and 6.7
billion parameters, respectively. We observe that QG performance improves
steadily but minimally after the model becomes larger than 2.7 billion
parameters, although the largest model still wins modestly. These results
imply that MultiQG-TI may retain the same level of competitiveness even with a
smaller off-the-shelf image-to-text model, suggesting more resource-saving
opportunities without compromising performance.

#### Qualitative examples.

We show an example generated question by MultiQG-TI in Table 2, as well as
additional ones in Appendix C. These examples further illustrates MultiQG-TI’s
capability in generating fluent, coherent, and meaningful questions from
multi-modal scientific contexts. We also provide an in-depth analyses of the
errors that MultiQG-TI makes during generation, which we defer to Appendix C
due to space constraint.

##  4 Conclusion

We have conducted a first study into automatic multi-modal QG from images and
texts. Our proposed solution, MultiQG-TI, is simple, easy-to-use, and highly
capable, as evaluated and analyzed on the ScienceQA dataset. Our work opens a
myriad of research opportunities. Some of the exciting future directions
include: 1) QG with multi-modal inputs and multi-modal outputs; 2) end-to-end
vision-language modeling approach for QG; and 3) evaluating and comparing the
pedagogical utilities of questions generated from multi-modal sources in real-
world educational scenarios.

## Acknowledgements

This work was supported by NSF grants 1842378, ONR grant N0014-20-1-2534,
AFOSR grant FA9550-22-1-0060, and a Vannevar Bush Faculty Fellowship, ONR
grant N00014-18-1-2047.

## References

  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. 2022.  Flamingo: a visual language model for few-shot learning.  In _Advances in Neural Information Processing Systems_.
  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015.  Vqa: Visual question answering.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_.
  * Burstein et al. (2021) Jill Burstein, Geoffrey T LaFlair, Antony John Kunnan, and Alina A von Davier. 2021\.  A theoretical assessment ecosystem for a digital-first assessment—the duolingo english test.  _DRR-21-04_.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.  Scaling instruction-finetuned language models.
  * Du et al. (2020) Yuning Du, Chenxia Li, Ruoyu Guo, Xiaoting Yin, Weiwei Liu, Jun Zhou, Yifan Bai, Zilin Yu, Yehua Yang, Qingqing Dang, and Haoshuang Wang. 2020.  Pp-ocr: A practical ultra lightweight ocr system.
  * He and Deng (2017) Xiaodong He and Li Deng. 2017.  Deep learning for image-to-text generation: A technical overview.  _IEEE Signal Processing Magazine_ , 34(6):109–116.
  * Heilman and Smith (2010) Michael Heilman and Noah A. Smith. 2010.  Good question! statistical ranking for question generation.  In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 609–617, Los Angeles, California. Association for Computational Linguistics.
  * Hossain et al. (2019) MD. Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratuddin, and Hamid Laga. 2019\.  A comprehensive survey of deep learning for image captioning.  _ACM Computing Surveys_ , 51(6):1–36.
  * Hu et al. (2022) Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A. Smith, and Jiebo Luo. 2022\.  Promptcap: Prompt-guided task-aware image captioning.
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of hallucination in natural language generation.  _ACM Computing Surveys_ , 55(12):1–38.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.  Scaling laws for neural language models.
  * Kokku et al. (2018) Ravi Kokku, Sharad Sundararajan, Prasenjit Dey, Renuka Sindhgatta, Satya Nitta, and Bikram Sengupta. 2018.  Augmenting classrooms with AI for personalized education.  In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE.
  * Kulshreshtha et al. (2022) Devang Kulshreshtha, Muhammad Shayan, Robert Belfer, Siva Reddy, Iulian Vlad Serban, and Ekaterina Kochmar. 2022.  Few-shot question generation for personalized feedback in intelligent tutoring systems.
  * Le et al. (2014) Nguyen-Thinh Le, Tomoko Kojiri, and Niels Pinkwart. 2014.  Automatic question generation for educational applications – the state of art.  In _Advanced Computational Methods for Knowledge Engineering_ , pages 325–338. Springer International Publishing.
  * Li et al. (2023) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.
  * Liu et al. (2021) Tianqiao Liu, Qiang Fang, Wenbiao Ding, Hang Li, Zhongqin Wu, and Zitao Liu. 2021\.  Mathematical word problem generation from commonsense knowledge graph and equations.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Lu et al. (2022) Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022.  Learn to explain: Multimodal reasoning via thought chains for science question answering.  In _The 36th Conference on Neural Information Processing Systems (NeurIPS)_.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.
  * Pan et al. (2019) Liangming Pan, Wenqiang Lei, Tat-Seng Chua, and Min-Yen Kan. 2019.  Recent advances in neural question generation.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022.  Hierarchical text-conditional image generation with clip latents.
  * Srivastava and Goodman (2021) Megha Srivastava and Noah Goodman. 2021.  Question generation for adaptive education.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_. Association for Computational Linguistics.
  * Su et al. (2022) Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. 2022\.  A contrastive framework for neural text generation.  In _Advances in Neural Information Processing Systems_.
  * Talmor et al. (2021) Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. 2021.  Multimodal{qa}: complex question answering over text, tables and images.  In _International Conference on Learning Representations_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Wang et al. (2022) Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, Shih-Fu Chang, Mohit Bansal, and Heng Ji. 2022.  Language models with image descriptors are strong few-shot video-language learners.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 8483–8497. Curran Associates, Inc.
  * Wang et al. (2021) Zichao Wang, Andrew Lan, and Richard Baraniuk. 2021.  Math word problem generation with mathematical consistency and problem context constraints.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Wang et al. (2018) Zichao Wang, Andrew S. Lan, Weili Nie, Andrew E. Waters, Phillip J. Grimaldi, and Richard G. Baraniuk. 2018.  QG-net.  In _Proceedings of the Fifth Annual ACM Conference on Learning at Scale_. ACM.
  * Wei et al. (2022) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022.  Emergent abilities of large language models.  _Transactions on Machine Learning Research_.  Survey Certification.
  * White et al. (2022) Julia White, Amy Burkhardt, Jason Yeatman, and Noah Goodman. 2022.  Automated generation of sentence reading fluency test items.  In _Proceedings of the Annual Meeting of the Cognitive Science Society_ , volume 44.
  * Wolfe (1976) John H. Wolfe. 1976.  Automatic question generation from text - an aid to independent study.  In _Proceedings of the ACM SIGCSE-SIGCUE technical symposium on Computer science and education -_. ACM Press.
  * Xu et al. (2022) Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Li, Nora Bradford, Branda Sun, Tran Hoang, Yisi Sang, Yufang Hou, Xiaojuan Ma, Diyi Yang, Nanyun Peng, Zhou Yu, and Mark Warschauer. 2022.  Fantastic questions and where to find them: FairytaleQA – an authentic dataset for narrative comprehension.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics.
  * Yang et al. (2022) Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. 2022\.  Zero-shot video question answering via frozen bidirectional language models.  In _Advances in Neural Information Processing Systems_.
  * Yang et al. (2023) Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. 2023.  Mm-react: Prompting chatgpt for multimodal reasoning and action.
  * Zhang et al. (2022) Zheng Zhang, Ying Xu, Yanhao Wang, Bingsheng Yao, Daniel Ritchie, Tongshuang Wu, Mo Yu, Dakuo Wang, and Toby Jia-Jun Li. 2022.  StoryBuddy: A human-AI collaborative chatbot for parent-child interactive storytelling with flexible parental involvement.  In _CHI Conference on Human Factors in Computing Systems_. ACM.
  * Zhao et al. (2022) Zhenjie Zhao, Yufang Hou, Dakuo Wang, Mo Yu, Chengzhong Liu, and Xiaojuan Ma. 2022\.  Educational question generation of children storybooks via question type distribution learning and event-centric summarization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics.

##  Appendix A Experiment details

###  A.1 Dataset and preprocessing

Each data point in the ScienceQA dataset contains the question text, a
background text, and an image. The total number of data points in the
ScienceQA dataset is 21,208. We refer readers to Lu et al. (2022) for more
details on the dataset. However, the background text and the image are
optionally included. As a result, not all data points contain both the
background text and the image. We only keep data points that contain all three
elements, resulting in 5,942 data points. We further randomly split them into
train, validation, and test splits, resulting in 3606/1204/1132 data points in
the train/validation/test splits, respectively. For both the remaining texts
and images, we did not perform further processing and keep them as-is before
feeding them to the MultiQG-TI components that are responsible for processing
them.

We note that the MultimodalQA dataset Talmor et al. (2021) is also an
appropriate dataset choice with rich multi-modal information beyond just texts
and images. Because our present work focuses on image and text as input
modalities, we leave more complex data modalities for QG for future work.

###  A.2 MultiQG-TI model details

#### Image-to-text generation.

We use contrastive sampling Su et al. (2022) with the following
parameters:333See this blog post for an explanation of the different
parameters that appear in contrastive sampling:
https://huggingface.co/blog/introducing-csearch $\alpha=0.6$ and $k=4$, with a
temperature of 1, n-gram penalty of 3, and minimum text description length of
30 tokens. For each given image, we sample 10 different text descriptions,
rerank them by the image-to-text model’s perplexity, and choose the best
description (with the lowest perplexity score) as the final text description
for the image, which we will then send to the QG module, together with the OCR
module’s output and the input background text.

#### QG module training.

We perform all training on a single NVIDIA Quadro RTX 8000 GPU. For all QG
module variants that we consider, we use the same training setup.
Specifically, we train it with a learning rate of $0.0003$ for 8 epochs with
early stopping if validation loss does not improve over the most recent 3
epochs. We use a batch size of 3 with a gradient accumulation step of 4,
resulting in an effective batch size of 12 (e.g., the parameters are updated
every 12 training steps). We also clip the gradients to 1 to stabilize
training. All these training procedures are standard in training text
generative models.

#### Inference and evaluation.

We use the same contrastive sampling strategy as in image-to-text generation.
Additionally, we sample 10 generated questions, rerank them by perplexity, and
fetch the best-ranked sample as the final generated question for each input
text-image pair in the test set. All evaluations are conducted on this “top-1”
setting. For each individual run, we perform the above sampling strategy with
a different seed to obtain a different set of generated questions for each
input in the test set. We then perform the same evaluation on each generated
set and then average the results, resulting in the averaged quantitative
evaluations reported in the main paper.

#### Remarks.

MultiQG-TI leverages readily available, open-source tools to solve the new
problem of multi-modal question generation. Its modular design makes it
flexible and easily adaptable, enabling one to upgrade a component when a more
capable one becomes available. Moreover, the only trainable component is the
question generator. There are many choices available for this component, any
of which can achieve competitive performance with relatively limited model
sizes, making it suitable for low-resource training settings. An end-to-end
multi-modal QG model is still methodologically interesting and we leave this
as a future work.

###  A.3 ChatGPT baseline

We use the gpt-3.5-turbo-0301 model API throughout our experiments. The system
message we give to the model at the beginning of the API call is as follows:
You are a helpful assistant. Your job is to generate a question, which
consists of a question background/context and the question itself, given the
user’s provided context information, which consists of an instruction,
background, subject, topic, and category. Your answer should be in the
following template: ’Question context: ... Question: ...’. After that, for
zero-shot QG, we send the templated input background text, OCR extracted text
from the input image, and the text description of the input image to the API,
formatted exactly as what we would do for MultiQG-TI. For few-shot QG, we
construct each example as a pair of input and output, where the input is the
templated input consisting of the input text and texts extracted from the
input image, and the output is the corresponding question text to the input
text and image. We only perform generation once for each setting and for each
input to avoid incurring higher costs of making OpenAI API calls.

#### Selecting examples for in-context learning.

We perform a basic cosine similarity search for each input context and image
pairs. Specifically, we first encode each formatted input text (recall, it
contains the input background text, the image description, and the texts in
the image) as a vector using the
SentenceTransformers.444https://www.sbert.net/. Then, for each formatted input
in the test set, we perform a similarity search, computing its cosine
similarity with every formatted input in the training set, and select up to
seven most similar formatted input as the examples to be used in prompting
ChatGPT in the few-shot in-context learning setting.

##  Appendix B Additional literature review

The MultimodalQA dataset Talmor et al. (2021) actually involves a cursory
description of generating questions from multiple sources. However, the QG
process described therein relies on human annotation, a manual process that
cannot achieve automatic QG and therefore is neither a baseline to our work
nor related to our goal of automatic QG.

Recent research has demonstrated the impressive capabilities of models that
can connect data from multiple modalities, such as generating images from
texts Ramesh et al. (2022) and vice versa He and Deng (2017). Specifically
related to our work, recent advances in vision-language models Alayrac et al.
(2022); Li et al. (2023); OpenAI (2023) enable models to converse with a user
given both texts and images. However, most demonstrated use cases of these
models are in casual dialogues Li et al. (2023), image captioning Hossain et
al. (2019), and visual question answering Antol et al. (2015). The utilities
of these models for QG remain largely unknown.

##  Appendix C Additional results

#### Additional examples of generated questions.

We provide additional generation examples in Table 4 for chemistry, physics,
and biology, respectively. These examples corroborate with the one in the main
text and demonstrate the capability of MultiQG-TI in generating reasonable
questions from image and text inputs.

Table 4: Additional examples of science questions generated by MultiQG-TI.

Input background text Input image A solution is made up of two or more
substances that are completely mixed. In a solution, solute particles are
mixed into a solvent. The solute cannot be separated from the solvent by a
filter. For example, if you stir a spoonful of salt into a cup of water, the
salt will mix into the water to make a saltwater solution. In this case, the
salt is the solute. The water is the solvent. The concentration of a solute in
a solution is a measure of the ratio of solute to solvent. Concentration can
be described in terms of particles of solute per volume of solvent.
concentration = particles of solute / volume of solvent MultiQG-TI generated
question The diagram below is a model of two solutions. Each pink ball
represents one particle of solute. Which solution has a higher concentration
of pink particles? Input background text Input image A force is a push or a
pull that acts on an object. Each force acts on an object in a certain
direction. If two forces act on an object in opposite directions, they are
called opposing forces. MultiQG-TI generated question The text below describes
a pair of opposing forces. Opposing forces act on an object in opposite
directions. Read the text. Then answer the question below. Two puppies, Rusty
and Coco, play with a toy. Think about two of the forces that act on the toy:
Coco pulls toward herself. Rusty pulls away from Coco. As Coco pulls on the
toy, what is the direction of the opposing force? Input background text Input
image Offspring genotypes: homozygous or heterozygous? How do you determine
whether an organism is homozygous or heterozygous for a gene? Look at the
alleles in the organism’s genotype for that gene. An organism with two
identical alleles for a gene is homozygous for that gene. If both alleles are
dominant, the organism is homozygous dominant for the gene. If both alleles
are recessive, the organism is homozygous recessive for the gene. An organism
with two different alleles for a gene is heterozygous for that gene. In a
Punnett square, each box represents a different outcome, or result. Each of
the four outcomes is equally likely to happen. Each box represents one way the
parents’ alleles can combine to form an offspring’s genotype. Because there
are four boxes in the Punnett square, there are four possible outcomes. An
event is a set of one or more outcomes. The probability of an event is a
measure of how likely the event is to happen. …… (truncated due to excessive
length) MultiQG-TI generated question In a group of rainbow trout, some
individuals have a greenish-brown body and others have a blue body. In this
group, the gene for the body color trait has two alleles. The allele for a
blue body (e) is recessive to the allele for a greenish- brown body (E). This
Punnett square shows a cross between two rainbow trout. What is the
probability that a rainbow trout produced by this cross will be heterozygous
for the body color gene?

#### Qualitative generation error analysis.

MultiQG-TI is not without problems. In Table 5, we provide an exemplary
erroneous generated question to illustrate the typical problems that MultiQG-
TI has when performing QG.

In our observation, there are two major sources of error. The first one comes
from the mistakes cascaded from the image-to-text model. In the example in
Table 5, the object in the image is dolerite, but the image-to-text model in
MultiQG-TI recognizes it as granite, resulting in the image description “a
black piece of granite on a white background”. As a result, the question
generator, which generates the question conditioned on the image description,
picks up the wrongly reconigized object “granite” and use it to generate a
question on granite instead of on dolerite.

The second source of error comes from hallucination, a major bottleneck
preventing language models from real-world, high-stake use scenarios Ji et al.
(2023). MultiQG-TI is not immune to this problem. In the example in Table 5,
the question generator produces the phrase “pure substance”, which is neither
a property of dolerite nor granite because both are mixtures.

These are challenging issues to tackle. For example, it is even difficult for
a non-expert to identify the object in the image in Table 5. Similarly, it is
difficult to verify the factual correctness of the generated question without
resorting to external sources such as web search and textbooks. Reducing these
errors would require improvements to the image-to-text model and mitigating
hallucination in language models, both of which remain active areas of
research.

Table 5: An illustrative erroneous question generated by MultiQG-TI. The texts
marked in red are the erroneous parts. The word “granite” results from the
image-to-text model, which incorrectly recognizes the dolerite in the image as
granite. The phrase “pure substance” is neither a property of dolerite nor
granite.

Input background text Input image Minerals are the building blocks of rocks. A rock can be made of one or more minerals. Minerals and rocks have the following properties: Property | Mineral | Rock It is a solid. | Yes | Yes It is formed in nature. | Yes | Yes It is not made by organisms. | Yes | Yes It is a pure substance. | Yes | No It has a fixed crystal structure. | Yes | No You can use these properties to tell whether a substance is a mineral, a rock, or neither. Look closely at the last three properties: Minerals and rocks are not made by organisms. Organisms make their own body parts. For example, snails and clams make their shells. Because they are made by organisms, body parts cannot be minerals or rocks. Humans are organisms too. So, substances that humans make by hand or in factories are not minerals or rocks. A mineral is a pure substance, but a rock is not. A pure substance is made of only one type of matter. Minerals are pure substances, but rocks are not. Instead, all rocks are mixtures. A mineral has a fixed crystal structure, but a rock does not. The crystal structure of a substance tells you how the atoms or molecules in the substance are arranged. Different types of minerals have different crystal structures, but all minerals have a fixed crystal structure. This means that the atoms and molecules in different pieces of the same type of mineral are always arranged the same way. However, rocks do not have a fixed crystal structure. So, the arrangement of atoms or molecules in different pieces of the same type of rock may be different! MultiQG-TI generated question Granite has the following properties: pure substance solid fixed crystal structure naturally occurring not made by living things Question: Is granite a mineral or a rock?