# Learned transform compression with optimized entropy encoding.

Magda Gregorová & Marc Desaules & Alexandros Kalousis
Geneva School of Business Administration,
HES-SO University of Applied Sciences of Western Switzerland
<EMAIL_ADDRESS>

###### Abstract

We consider the problem of learned transform compression where we learn both,
the transform as well as the probability distribution over the discrete codes.
We utilize a soft relaxation of the quantization operation to allow for back-
propagation of gradients and employ vector (rather than scalar) quantization
of the latent codes. Furthermore, we apply similar relaxation in the code
probability assignments enabling direct optimization of the code entropy. To
the best of our knowledge, this approach is completely novel. We conduct a set
of proof-of concept experiments confirming the potency of our approaches.

##  1 Introduction

We consider the problem of compressing data ${\bm{x}}\in{\mathbb{X}}$ sampled
i.i.d. according to some unknown probability measure (distribution)
${\bm{x}}\sim\mu_{x}$. We take the standard transform coding (Sayood, 2012)
approach where we first transform the data ${\bm{x}}$ by a learned non-linear
function, an encoder ${\mathcal{E}}_{\theta}:{\mathbb{X}}\to{\mathbb{Z}}$,
into some latent representation ${\bm{z}}\in{\mathbb{Z}}$. We then quantize
the transformed data ${\bm{z}}={\mathcal{E}}_{\theta}({\bm{x}})$ using a
quantization function ${\mathcal{Q}}_{\bm{E}}:{\mathbb{Z}}\to{\mathbb{C}}$
parametrized by learned embeddings ${\bm{E}}$ (codebook/ dictionary) so that
the discrete codes composed of indexes of the embedding vectors
${\bm{c}}={\mathcal{Q}}_{\bm{E}}({\bm{z}})$ can be compressed by a lossless
entropy encoding and transmitted. The received and losslessly decoded integer
codes are then used to index the embedding vectors and dequantized back to the
latent space
$\overline{{\mathcal{Q}}_{\bm{E}}}:{\mathbb{C}}\to\widehat{{\mathbb{Z}}}\subset{\mathbb{Z}}$
introducing a distortion due to mapping the codes only to the discrete subset
$\widehat{{\mathbb{Z}}}\subset{\mathbb{Z}}$ corresponding to the quantization
embeddings. The dequantized data
$\hat{{\bm{z}}}=\overline{{\mathcal{Q}}_{\bm{E}}}({\bm{c}})$ are then decoded
by a learned non-linear decoder
${\mathcal{D}}_{\phi}:\widehat{{\mathbb{Z}}}\to{\mathbb{X}}$ to obtain the
reconstructions $\hat{{\bm{x}}}={\mathcal{D}}_{\phi}(\hat{{\bm{z}}})$.

Our aim is to learn the transform (encoder/decoder) as well as the
quantization so as to minimize the expected distortion
$\mathbb{E}_{\mu_{x}}d(\mathtt{x},\hat{\mathtt{x}})$111We use
$\mathtt{x},\mathtt{z},\mathtt{c}$ for random variables and
${\bm{x}},{\bm{z}},{\bm{c}}$ for their realizations. while, at the same time,
minimizing the expected number of bits transmitted (the rate) when passing on
the discrete codes $\mathbb{E}_{\mu_{c}}l(\mathtt{c})$, where $l$ is the
length of the bit-encoding. The two competing objectives are controlled via a
hyper-parameter $\lambda$

$\displaystyle\mathcal{L}:=\underbrace{\mathbb{E}_{\mu_{x}}d(\mathtt{x},\hat{\mathtt{x}})}_{distortion}+\lambda\underbrace{\mathbb{E}_{\mu_{c}}l(\mathtt{c})}_{rate}\enspace.$
(1)

The optimal length of encoding a symbol ${\bm{c}}\sim\mu_{c}$ is determined by
Shannon’s self information222When considering bit-encoding, the $\log$ should
be with base 2 instead of the natural base for nats. $i_{c}({\bm{c}})=-\log
p_{c}({\bm{c}})$ (Cover & Thomas, 2006), where $p_{c}$ is the discrete
probability mass333The probability mass $p_{c}$ is the probability density
function of $\mu_{c}$ with respect to the counting measure
$\mu_{c}(\mathtt{c}\in{\bm{A}})=\int_{A}p_{c}\,\textnormal{d}\\#=\sum_{{\bm{a}}\in{\bm{A}}}p_{c}({\bm{a}})$,
such that
$p_{c}({\bm{a}})=p_{c}(\mathtt{c}={\bm{a}})=\mu_{c}(\mathtt{c}={\bm{a}})$. of
the distribution $\mu_{c}$. Consequently, the expected optimal description
length for the discrete code $\mathtt{c}$ can be bounded by its entropy
${\mathbb{H}}_{\mu_{c}}(\mathtt{c})=-\mathbb{E}_{\mu_{c}}\log
p_{c}(\mathtt{c})$ as
${\mathbb{H}}_{\mu_{c}}(\mathtt{c})\leq\mathbb{E}_{\mu_{c}}l(\mathtt{c})^{*}<{\mathbb{H}}_{\mu_{c}}(\mathtt{c})+1$444The
‘+1’ can be reduced by more _clever_ lossless compression strategy - out of
scope of this paper.. To minimize the rate we therefore minimize the entropy
of the discrete code ${\mathbb{H}}_{\mu_{c}}(\mathtt{c})$ so that

$\displaystyle\mathcal{L}:=\mathbb{E}_{\mu_{x}}d(\mathtt{x},\hat{\mathtt{x}})+\lambda{\mathbb{H}}_{\mu_{c}}(\mathtt{c})\enspace.$
(2)

##  2 Quantization

We employ the soft relaxation approach to quantization proposed in Agustsson
et al. (2017) simplified similarly to Mentzer et al. (2018). However, instead
of the scalar version Mentzer et al. (2018); Habibian et al. (2019) we use the
vector formulation of the quantization as in van den Oord et al. (2017) in
which the $k$ quantization centers are the learned embedding vectors
$\\{{\bm{e}}^{(j)}\\}_{j=1}^{k},\ {\bm{e}}_{i}\in{\mathbb{R}}^{m}$, columns of
the $m\times k$ embedding matrix
${\bm{E}}=[{\bm{e}}^{(1)},\ldots,{\bm{e}}^{(k)}]$.

The quantizer ${\mathcal{Q}}_{\bm{E}}$ first reshapes the transformed
data555For notational simplicity we regard the data as $d$ dimensional
vectors. In practice, these are often $(d_{1}\times d_{2})$ matrices or even
higher order tensors $(d_{1}\times\ldots\times d_{t})$. ${\bm{z}}$ can be seen
simply as their flattened version with $d=\prod_{i}d_{i}$.
${\bm{z}}\in{\mathbb{Z}}\subseteq{\mathbb{R}}^{d}$ into a $m\times d/m$ matrix
${\bm{Z}}=[{\bm{z}}^{(1)},\ldots,{\bm{z}}^{(d/m)}]$, then finds for each
column ${\bm{z}}^{(i)}\in{\mathbb{R}}^{m}$ its nearest embedding and replaces
it by the embedding vector index to output the $d/m$ dimensional vector of
discrete codes ${\bm{c}}={\mathcal{Q}}_{\bm{E}}({\bm{z}})$

$\displaystyle{\mathcal{Q}}_{\bm{E}}:\quad\hat{{\bm{z}}}^{(i)}=\operatorname*{arg\,min}_{{\bm{e}}^{(j)}}\|{\bm{z}}^{(i)}-{\bm{e}}^{(j)}\|,\qquad{c}^{(i)}=\\{j:\hat{{\bm{z}}}^{(i)}={\bm{e}}^{(j)}\\},\quad
i=1,\ldots,d/m\enspace.$ (3)

After transmission, the quantized latent representation
$\hat{{\bm{Z}}}=[\hat{{\bm{z}}}^{(1)},\ldots,\hat{{\bm{z}}}^{(d/m)}]$ is
recovered from ${\bm{c}}$ by indexing to the shared codebook ${\bm{E}}$ and
decoded $\hat{{\bm{x}}}={\mathcal{D}}_{\phi}(\hat{{\bm{z}}})$,
$\hat{{\bm{z}}}=\mathrm{flatten}(\hat{{\bm{Z}}})$. In practice, the quantized
latent $\hat{{\bm{z}}}$ can be used directly by the decoder at training in the
forward pass without triggering the ${\bm{c}}$ indexing operation.

The finite quantization operation in equation 3 is non-differentiable. To
allow for the flow of gradients back to ${\mathcal{E}}_{\theta}$ and
${\mathcal{Q}}_{\bm{E}}$ we use a differentiable soft relaxation for the
backward pass

$\displaystyle\tilde{{\bm{z}}}^{(i)}=\sum_{j}^{k}{\bm{e}}^{(j)}\mathrm{softmax}(-\sigma\|{\bm{z}}^{(i)}-{\bm{e}}^{(j)}\|)=\sum_{j}^{k}{\bm{e}}^{(j)}\frac{\exp(-\sigma\|{\bm{z}}^{(i)}-{\bm{e}}^{(j)}\|)}{\sum_{j}^{k}\exp(-\sigma\|{\bm{z}}^{(i)}-{\bm{e}}^{(j)}\|)}\enspace,$
(4)

where instead of the hard encoding $\hat{\bar{{\bm{z}}}}^{(i)}$ picking the
single nearest embedding vector, the soft $\tilde{{\bm{z}}}^{(i)}$ is a linear
combination of the embeddings weighted by their (softmaxed) distances. The
distortion loss is thus formulated as
$d({\bm{x}},\hat{{\bm{x}}})=d({\bm{x}},{\mathcal{D}}_{\phi}[\mathrm{sg}(\hat{{\bm{z}}}-\tilde{{\bm{z}}})+\tilde{{\bm{z}}}])$,
where $\mathrm{sg}$ is the stopgradient operator.

The hard/soft strategy is different from the approach of van den Oord et al.
(2017) where they use a form of straight-through gradient estimator and
dedicated codebook and commitment terms in the loss to train the embeddings
${\bm{E}}$. This is also different from Williams et al. (2020), where they use
the relaxed formulation of equation 4 for both forward and backward passes in
a fully stochastic quantization scheme aimed at preventing the mode-dropping
effect of a deterministic approximate posterior in a hierarchical vector-
quantized VAE.

##  3 Minimizing the code cross-entropy

Though the optimal lossless encoding is decided by the self-information of the
code $-\log p_{c}({\bm{c}})$, it cannot be used directly since $p_{c}$ is
unknown. Instead we replace the unknown $p_{c}$ by its estimated approximation
$q_{c}$, derive the code length $\hat{l}({\bm{c}})$ from $\hat{i}_{c}=-\log
q_{c}({\bm{c}})$, and therefore minimize the expected approximate self-
information, the cross-entropy
${\mathbb{H}}_{\mu_{c}|q_{c}}(\mathtt{c})=-\mathbb{E}_{\mu_{c}}\log
q_{c}(\mathtt{c})$. This, however, yields inefficiencies as
${\mathbb{H}}_{\mu_{c}|q_{c}}(\mathtt{c})\geq{\mathbb{H}}_{\mu_{c}}(\mathtt{c})$
due to the decomposition

$\displaystyle{\mathbb{H}}_{\mu_{c}|q_{c}}(\mathtt{c})=D_{\mathrm{KL}}(p_{c}\|q_{c})+{\mathbb{H}}_{\mu_{c}}(\mathtt{c})\enspace,$
(5)

where $D_{\mathrm{KL}}\geq 0$ is the Kullback-Leibler divergence between
$p_{c}$ and $q_{c}$ which can be interpreted as the expected additional bits
over the optimal rate ${\mathbb{H}}_{\mu_{c}}({\bm{c}})$ caused by using
$q_{c}$ instead of the true $p_{c}$. In addition to ${\mathcal{E}}_{\theta}$,
${\mathcal{D}}_{\phi}$ and ${\mathcal{Q}}_{\bm{E}}$ we shall now therefore
train also a probability estimator
${\mathcal{P}}_{\psi}:\\{{\bm{c}}\\}_{i}^{n}\to q_{c}$ by minimizing the
cross-entropy ${\mathbb{H}}_{\mu_{c}|q_{c}}(\mathtt{c})$ so that the estimated
$q_{c}$ is as close as possible to the true $p_{c}$, the $D_{\mathrm{KL}}$ is
small, and the above mentioned inefficiencies disappear.

As we cannot evaluate the cross-entropy over the unknown $\mu_{c}$, we instead
learn ${\mathcal{P}}_{\psi}$ by its empirical estimate over the sample data
which is equivalent to minimizing the negative log likelihood (NLL)

$\displaystyle\operatorname*{arg\,min}_{\psi}-\frac{1}{n}\sum_{i}^{n}\log
q_{c}(\mathtt{c}={\bm{c}}_{i}),\quad{\bm{c}}_{i}\sim\mu_{c}\enspace.$ (6)

Similar strategy has been used for example in Theis et al. (2017) and Ballé et
al. (2017) both using some form of continuous relaxation of $q_{c}$ as well as
in Mentzer et al. (2018) using an autoregressive PixelCNN as
${\mathcal{P}}_{\psi}$ to model
$q_{c}({\bm{c}})=\prod_{i}q_{c}({c}_{i}|{c}_{i-1},\ldots.{c}_{1})$.

There is one caveat to the above approach. In the minimization in equation 6
the sampling distribution $\mu_{c}$ is treated as fixed. Minimizing the cross-
entropy in such a regime minimizes the $D_{\mathrm{KL}}$ and hence the
additional bits due to $q_{c}\neq p_{c}$ but not the entropy
${\mathbb{H}}_{\mu_{c}}(\mathtt{c})$ (see equation 5) which is treated as
fixed and therefore not optimized for low rate as explained in section 2.

This may seem natural and even inevitable since the samples ${\bm{c}}$ are the
result of sampling the data ${\bm{x}}$ from the unknown yet fixed distribution
$\mu_{x}$. Yet, the distribution $\mu_{c}$ is not fixed. It is determined by
the learned transformation
${\mathcal{T}}_{{\bm{E}},\theta}={\mathcal{Q}}_{\bm{E}}\circ{\mathcal{E}}_{\theta}$
as the push-forward measure of $\mu_{x}$

$\displaystyle\mu_{c}[\mathtt{c}\in{\bm{A}}]=\mu_{c}[{\mathcal{T}}_{{\bm{E}},\theta}(\mathtt{x})\in{\bm{A}}]=\mu_{x}[\mathtt{x}\in{\mathcal{T}}_{{\bm{E}},\theta}^{-1}({\bm{A}})]\enspace,$
(7)

where ${\bm{A}}\in{\mathbb{C}}$ and ${\mathcal{T}}^{-1}$ is the inverse
image666The notation ${\mathcal{T}}^{-1}$ here should not be mistaken for an
_inverse function_ as ${\mathcal{T}}$ is generally not invertible. defined as
${\mathcal{T}}^{-1}({\bm{A}}):=\\{\mathtt{x}\in{\mathbb{X}}:{\mathcal{T}}(\mathtt{x})\in{\bm{A}}\\}$.
Changing the parameters of the encoder ${\mathcal{E}}_{\theta}$ and the
embeddings ${\bm{E}}$ will change the measure $\mu_{c}$ and hence the entropy
${\mathbb{H}}_{\mu_{c}}(\mathtt{c})$ and the cross-entropy
${\mathbb{H}}_{\mu_{c}|q_{c}}(\mathtt{c})$ even with the approximation $q_{c}$
fixed.

We therefore propose to optimise the encoder and the embeddings so as to
minimize the cross-entropy not only through learning better approximation
$q_{c}$ but also through changing $\mu_{c}$ to achieve overall lower rate.
Since the discrete sampling operation from $\mu_{c}$ is non-differentiable, we
propose to use a simple continuous soft relaxation similar the one described
in section 2. Instead of using the deterministic non-differentiable code
assignments

$\displaystyle p_{c}(\mathtt{c}^{(i)}=j)=\begin{cases}1&\text{if}\
\hat{{\bm{z}}}^{(i)}={\bm{e}}^{(j)}\\\ 0&\text{otherwise}\end{cases},\quad
i=1,\ldots,d/m$ (8)

we use the differentiable soft relaxation

$\displaystyle\hat{p}_{c}(\mathtt{c}^{(i)}=j)=\mathrm{softmax}(-\sigma\|{\bm{z}}^{(i)}-{\bm{e}}^{(j)}\|)=\frac{\exp(-\sigma\|{\bm{z}}^{(i)}-{\bm{e}}^{(j)}\|)}{\sum_{j}^{k}\exp(-\sigma\|{\bm{z}}^{(i)}-{\bm{e}}^{(j)}\|)}\enspace.$
(9)

Our final objective is the minimization of the empirical loss composed of
three terms: the distortion, the _soft_ cross-entropy, and the _hard_ cross-
entropy

$\displaystyle\widehat{\mathcal{L}}(\theta,{\bm{E}},\phi,\psi):=\frac{1}{n}\sum_{i}^{n}d({\bm{x}}_{i},\hat{{\bm{x}}}_{i})+\alpha\,s({\bm{c}}_{i})+\beta\,h({\bm{c}}_{i})\enspace.$
(10)

The distortion may be the squared error
$d({\bm{x}},\hat{{\bm{x}}})=\|{\bm{x}}-\hat{{\bm{x}}}\|^{2}$ or other
application-dependent metric (e.g. multi-scale structural similarity index for
visual perception in images). Through the distortion we optimize the
parameters of the decoder ${\mathcal{D}}_{\phi}$ and using the relaxation
described in section 2 for the backward pass also the parameters of the
encoder ${\mathcal{E}}_{\theta}$ and the quantization embeddings
${\mathcal{Q}}_{\bm{E}}$.

The _hard_ cross-entropy loss is

$\displaystyle
h({\bm{c}})=-\frac{m}{d}\sum_{j}^{d/m}\sum_{{\bm{c}}^{(j)}=1}^{k}p_{c}({\bm{c}}^{(j)})\log
q_{c}({\bm{c}}^{(j)})=-\frac{m}{d}\sum_{j}^{d/m}\log
q_{c}({\bm{c}}^{(j)})\enspace,$ (11)

where we treat the dimensions of the vector
${\bm{c}}=[{\bm{c}}^{(1)},\ldots,{\bm{c}}^{(d/m)}]$ as independent so that the
approximation $q_{c}$ can be used directly as the entropy model for the
lossless arithmetic coding (expects the elements of the messages to be sampled
i.i.d. from a single distribution). The loss simplifies to the final form due
to the 0/1 probabilities of the deterministic code-assignments in equation 8.
Through the hard cross-entropy loss we learn the parameters of the probability
model ${\mathcal{P}}_{\psi}$ outputting the $q_{c}$ distribution.

The _soft_ cross-entropy loss is

$\displaystyle
s({\bm{c}})=-\frac{m}{d}\sum_{j}^{d/m}\sum_{{\bm{c}}^{(j)}=1}^{k}\hat{p}_{c}({\bm{c}}^{(j)})\log\textrm{sg}[q_{c}({\bm{c}}^{(j)})]\enspace,$
(12)

which uses the differentiable soft relaxation $\hat{p}_{c}$ of equation 9.
This allows for back-propagating the gradients to the encoder
${\mathcal{E}}_{\theta}$ and the quantizer ${\mathcal{Q}}_{\bm{E}}$. We use
the $sg$ operator here to treat $q_{c}$ as fixed in this part of the loss
preventing further updating of the parameters of the probability model
${\mathcal{P}}_{\psi}$.

##  4 Experiments

As a proof of concept we conducted a set of experiments on the tiny 32x32
CIFAR-10 images (Krizhevsky, 2009). We use similar architecture of the encoder
${\mathcal{E}}_{\theta}$ and decoder ${\mathcal{D}}_{\phi}$ as Mentzer et al.
(2018) (without the spatial importance mapping), with the downsampling and
upsampling stride-2 convolutions with kernels of size 4, and 10 residual
blocks with skip connections between every 3. We fix the annealing parameter
$\sigma=1$ and the loss hyper-parameter $\beta=1$777In our preliminary
experiments the results were not very sensitive to $\beta$. In fact, $\beta$
influences only the speed with which the probability model
${\mathcal{P}}_{\psi}$ is trained compared to the other components of the
model updated through the other parts of the loss.. We use ADAM with default
pytorch parameters, one cycle cosine learning rate and train for 15 epochs.
The code is available at: https://bitbucket.org/dmmlgeneva/softvqae/.

We first compare the vector quantization (VQ) approach where the codebook is
composed of $m$-long vectors versus the scalar (SQ) approach where it contains
scalars $m=1$ as e.g. in Mentzer et al. (2018). By construction the VQ version
needs to transmit shorter messages for the same level of downsampling. For
example, with 8-fold downsampling to $4\times 4$ latents ${\bm{z}}$ with 8
channels the discrete codes ${\bm{c}}$ of SQ have $d/m=128$ elements. In VQ,
the channel dimension forms the rows of the matrix ${\bm{Z}}$ with $m=8$ and
the ${\bm{c}}$ messages to be encoded and transmitted have only $d/m=16$
elements. On the other hand, in the scalar version each of the 8 channels is
represented by its own code and therefore allows for more flexibility compared
to a single code for the whole vector.

Our preliminary experiments confirm the superiority of the VQ. In figure 1
left we plot the rate-distortion for comparable parts of the trade-off space.
The VQ models use 2-folds downsampling resulting in $d/m=16\times 16=256$ long
messages ${\bm{c}}$. The two curves are for embeddings (latent ${\bm{z}}$
channels) with size $m=8$ and $m=16$ respectively and the points around the
curves are the result of increasing the size of the dictionary ${\bm{E}}$ as
$k=[8,16,32,64,128]$ from left to right. The SQ models use 8-folds
downsampling with latent ${\bm{z}}$ channels 8 and 16 resulting in
$d/m=4\times 4\times 8=128$ and $d/m=4\times 4\times 8=256$ ($m=1$ here). We
observe that VQ clearly achieves better trade-offs being in the bottom-left of
the plots.

Figure 1: Rate-distortion curves. Rate is expressed in bits per pixel (bpp) of
the original images, distortion is expressed in the mean squared error (mse)
between the original and reconstructed images. See text for description.

We next confirm the effectiveness of the soft cross-entropy term $s({\bm{c}})$
in our final loss formulation in equation 10. Increasing the $\alpha$ hyper-
parameter should put more importance on the rate minimization (through the
entropy) as compared to the distortion. In the right plot of figure 1 we
compare the points $k=[8,32,128]$ of the _‘vq, 8’_ curve for values
$\alpha=[0.01,0.001,0]$ from left to right. With the highest $\alpha=0.01$,
the objective trade-off searches for the lowest rate tolerating higher
distortion. The lower the $\alpha$, the less we push for low rates which
allows for smaller distortion. This behaviour corresponds well to the expected
and desirable one where, as formulated in equation 1, we can now directly
control the trade-off between the two competing objectives by setting the
hyper-parameter $\alpha$.

In the appendix we provide examples of the learned $q_{c}$ histograms for
_‘vq, 8, k=32’_ with different values of $\alpha$ showing the concentration of
the measure into a few points in the support for high $\alpha$.

## References

  * Agustsson et al. (2017) Eirikur Agustsson, Fabian Mentzer, Michael Tschannen, Lukas Cavigelli, Radu Timofte, Luca Benini, and Luc Van Gool.  Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations.  _arXiv:1704.00648 [cs]_ , June 2017.
  * Ballé et al. (2017) Johannes Ballé, Valero Laparra, and Eero P. Simoncelli.  End-to-end Optimized Image Compression.  In _International Conferenence on Learning Representations_ , 2017.
  * Cover & Thomas (2006) Thomas M Cover and Joy A Thomas.  _Elements of Information Theory_.  Wiley, 2006.
  * Habibian et al. (2019) Amirhossein Habibian, Ties van Rozendaal, Jakub M. Tomczak, and Taco S. Cohen.  Video Compression With Rate-Distortion Autoencoders.  _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 7032–7041, October 2019.  doi: 10.1109/ICCV.2019.00713.
  * Krizhevsky (2009) Alex Krizhevsky.  Learning Multiple Layers of Features from Tiny Images.  Technical report, University of Toronto, CS, 2009.
  * Mentzer et al. (2018) Fabian Mentzer, Eirikur Agustsson, Michael Tschannen, Radu Timofte, and Luc Van Gool.  Conditional Probability Models for Deep Image Compression.  In _CVPR_ , 2018.
  * Sayood (2012) Khalid Sayood.  _Introduction to Data Compression_.  Elsevier, fourth edition, 2012.  ISBN 978-0-12-415796-5.  doi: 10.1016/B978-0-12-415796-5.00023-5.
  * Theis et al. (2017) Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár.  Lossy Image Compression with Compressive Autoencoders.  In _International Conference on Learning Representations_ , 2017.
  * van den Oord et al. (2017) Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu.  Neural Discrete Representation Learning.  In _Conference on Neural Information Processing Systems (NIPS)_ , 2017.
  * Williams et al. (2020) Will Williams, Sam Ringer, Tom Ash, John Hughes, David MacLeod, and Jamie Dougherty.  Hierarchical Quantized Autoencoders.  In _Advances in Neural Information Processing Systems_ , 2020\.

##  Appendix A Appendix

Histograms $q_{c}$ learned by the probability model ${\mathcal{P}}_{\psi}$ for
the _‘vq, 8’_ models with the number of embedding vectors $k=32$ $\approx$ the
red line in the right graph in Figure 1 with increasing
$\alpha\in\\{0,0.001,0.01\\}$. For the highest $\alpha=0.01$ the distribution
is concentrated into a few points in the support resulting in the lowest
entropy (and therefore the best rate) but highest distortion.

Figure 2: Learned historgram $q_{c}$ for $\alpha=0$ Figure 3: Learned
historgram $q_{c}$ for $\alpha=0.001$ Figure 4: Learned historgram $q_{c}$ for
$\alpha=0.01$


#  : A Personalized and Gamified eTutoring System

Antonio Bucchiarone  Fondazione Bruno KesslerTrentoItaly<EMAIL_ADDRESS>,
Tommaso Martorella  Università di PisaPisa, Italy<EMAIL_ADDRESS>and
Diego Colombo  Microsoft Corporation RedmondWA, United States
<EMAIL_ADDRESS>
###### Abstract.

The digital age is changing the role of educators and pushing for a paradigm
shift in the education system as a whole. Growing demand for general and
specialized education inside and outside classrooms is at the heart of this
rising trend. In modern, heterogeneous learning environments, the one-size-
fits-all approach is proven to be fundamentally flawed. Individualization
through adaptivity is, therefore, crucial to nurture individual potential and
address accessibility needs and neurodiversity. By formalizing a learning
framework that takes into account all these different aspects, we aim to
define and implement an open, content-agnostic, and extensible eTutoring
platform to design and consume adaptive and gamified learning experiences.
Adaptive technology supplementing teaching can extend the reach of every
teacher, making it possible to scale 1-1 learning experiences. There are many
successful existing technologies available but they come with fixed
environments that are not always suitable for the targeted audiences of the
course material. This paper presents , a system able to help teachers to
design and implement a gamified and adaptive learning paths. Through it we
address some important issues including the engagement, fairness, and
effectiveness of learning environments. We do not only propose an innovative
platform that could foster the learning process of different disciplines, but
it could also help teachers and instructors in organizing learning material in
an easy-access repository.

Adaptive Learning, Adaptive Education, 1-1 eTutoring, AI-assisted Education,
Gamification, Gamified Education.

††ccs: Software and its engineering Software design engineering††ccs: Applied
computing††ccs: Applied computing Education

##  1\. Introduction

Adaptive learning is the delivery of personalized learning experiences that
address an individual’s unique needs instead of a one-size-fits-all approach.
It may be achieved through just-in-time feedback, personalized learning paths,
ad-hoc resources, or another wide array of techniques. But why is it
important? Education is universally recognized as one of the factors with the
highest impact on society and the individual. The United Nations included
education in their 2030 Agenda for Sustainable
Development111https://www.un.org/sustainabledevelopment/education/. UNESCO
launched the Global Education
Coalition222https://en.unesco.org/covid19/educationresponse/globalcoalition in
response to the COVID-19 pandemic. The European Union created the Digital
Education Action Plan (2021-2027)333https://education.ec.europa.eu/focus-
topics/digital-education/digital-education-action-plan to foster and support
the adaptation of educational systems in the digital age. This collective
global effort is motivated by a continuously increasing technology
availability and a rising global enrolment rate. Furthermore, according to
UNESCO, higher education is the fastest-growing sector444Data from UNESCO
(IIEP, 2020) and Statista (retrieved on 29 June 2022).
https://www.statista.com/statistics/1226999/net-enrollment-rate-in-primary-
school-worldwide/ and https://www.statista.com/statistics/1227022/net-
enrollment-rate-in-secondary-school-worldwide/ , with its global enrolment
rate doubled in the last twenty years (IIEP, 2020).

Rising trends such as flexible learning pathways and micro-credentials555
https://education.ec.europa.eu/education-levels/higher-education/micro-
credentials tend toward more versatile forms of content delivery and
credential recognition to accommodate the increasing demand for specializing
training, especially in the labour market.

Among the various aspects that comprise the education field, learning is the
one that requires the most careful treatment. The inherently complex domain
lends itself to a wide range of forms and means, each with its techniques and
quirks. Learning may happen at home, at work, or even on the go; thus, it is
not limited to the classroom only nor confined to formal settings in general.
Learning activities can (and should be) tailored around the individual. This
personalization process is critical when targeting neurodiverse profiles or
students with accessibility needs. Not only is content’s form fundamental, but
delivery and additional aids are required to make a learning experience
impactful.

Educational resources are the primary means to help students in their learning
journey. They can be of various kinds such as videos, interactive tutorials,
pdf texts, images, podcasts and many others. Platforms such as Google
Classroom, Microsoft Teams, or custom Moodle deployments are common ways to
deliver content in formal educational settings, but YouTube, Wikipedia, and
platforms like Udemy can reach a broader audience in informal environments.
Furthermore, the increasing availability of Open Educational Resources
(OER)666Open Educational Resources are public domain or open licensed
educational resources https://www.unesco.org/en/communication-
information/open-solutions/open-educational-resources can help reduce
teachers’ material preparation and lower the accessibility barrier in terms of
cost and material kind.

Learning is also crucial in the industry. The continuous technological
disruptures are creating job positions in brand new
fields777https://www.linkedin.com/business/talent/blog/talent-
strategy/linkedin-most-in-demand-hard-and-soft-skills. Entirely new hard
skills are required to fit the openings. Moreover, soft skills are among the
most sought-after skills because of their lower trainability and a slower
changing pace888https://www.linkedin.com/business/talent/blog/talent-
acquisition/why-shell-pushes-hard-on-soft-skills. However, the offer seldom
matches the demand. This mismatch led to the need for upskilling and
reskilling. The former means teaching employees new, advanced and valuable
skills to match the profile required for the next step in their current career
path. The latter, instead, targets employees with a profile similar to the one
required. It consists in teaching them adjacent skills and training them for
their new, different job. Both of them are fundamental learning activities
that take place in varied (and often dissimilar) environments.

From a student’s perspective, effective teaching means 1-1
tutoring999https://www.eschoolnews.com/2022/07/26/ai-intelligent-bots/. It
allows teachers to target specific misunderstandings and necessities with
real-time feedback and explanations relevant to the student’s experiences. On
the other end of the spectrum, the most teacher-effective approach is the one-
to-many lecture, where the teacher prepares the material upfront for being
presented to a wide audience. One of this approach’s main downsides is
encouraging passive learning. A study by K.R. Koedinger et al. shows that the
”Doer Effect” is a causal association between practice and learning outcomes
and that practicing is six times more effective than reading (Koedinger et
al., 2016). Another significant drawback lies in the motivational aspect.
Active learning is more effective in learning outcomes and motivation than
passive learning (Benware and Deci, 1984). Despite that, a recent article on
PNAS by L. Deslauriers et al. highlights a negative correlation between actual
learning and the feeling of learning in the students (Deslauriers et al.,
2019). However, gamification and serious games gained consensus as tools to
motivate people to engage in beneficial activities, even if seen as
unrewarding or tedious (Deterding et al., 2011b; Bucchiarone et al., 2021;
Deterding et al., 2011a; Bassanelli et al., 2022).

The design of a learning experience should take into account all of these
different factors. On the other hand, a design framework should not make
assumptions about content type, form, delivery, and validation while still
removing any obstacle between the teacher, the student, the environment, and
the learning experience. Individual coaching is rarely feasible due to poor
scalability, whilst one-to-many general training is scalable but lacks
individualization altogether. That is the gap the Personalized and Gamified
eTutoring System - - aims to fill. Adaptive eTutoring systems support both
teachers and students by combining the benefits of individualized delivery and
manageability by leveraging software personalization, while gamification can
be used to enhance motivation through personalized rewards or cooperative and
competitive activities.

The rest of the paper is structured as follows. Section 2 presents the needed
background and the motivations that led us to realize . Section 3 introduces
its architecture and explains how it can help teachers create flexible
learning and eTutoring experiences. AI planning and gamification mechanics
needed for adaptive learning will also be discussed by the end of section 3.
Section 4 will describe a scenario where is used to assist teachers and
students in introductory statistics for a data science course. Lastly, with
Section 5 we present a glimpse into future directions and possibilities
enabled by the proposed solution.

##  2\. Background and Motivations

Adaptive learning technologies have gained traction over the last decade.
Existing solutions have been successful in both domain-specific (Prusty and
Russell, 2011) and institution-wise implementations. In 2015, the Colorado
Technical University (CTU) reported that, following their Intellipath™
adoption, the course pass rate rose sharply by 27%, and the average grade and
retention rate were also significantly affected.
CogBooks101010https://www.cogbooks.com/ conducted a pilot study with Arizona
State University (ASU), resulting in findings similar to CTU.

Smart Sparrow111111https://www.smartsparrow.com/ (recently acquired by the
industry-leading Pearson) provides a user-friendly WYSIWYG content authoring
tool to create interactive online experiences. Realizeit, Cerego and CogBooks
create content for their adaptive platforms by partnering with institutions
like ASU, the American Psychological Association, or directly with industrial
partners and customers. Each platform also provides in-depth analytics on the
students’ performance, errors, time spent learning, and other related metrics.
Those solutions solve the problem of individualization and verticality of
linear learning paths but overlook a fundamental factor: familiarity with the
tooling and the environment. Job candidates in software engineering, for
example, are expected to be somewhat comfortable with industry-leading tools
and methodologies. Therefore, not only delivering interactive exercises
directly on those tools enables richer experiences but also better prepares
the students for their future. High school electronics students, instead, may
benefit from hands-on experience with physical devices like
Arduinos121212https://www.arduino.cc/. Moreover, integrating education-ready
tools like pi-top kits131313pi-top produces high-quality educational kits for
electronics and robotics based on the Raspberry Pi. https://www.pi-top.com/ or
Swift Playgrounds141414https://www.apple.com/swift/playgrounds/ allows access
to already existing quality resources.

Novel interactive learning experiences can emerge even with the use of
available technologies. Augmented and virtual reality are still emerging, but
education and industry are already taking advantage of their benefits
(Elmqaddem, 2019). In the classroom, they are used to promote interactivity
and generate engagement. In the industry, they found use in, among the others,
remote maintenance on industrial devices, surgery training programs, aviation,
and even military equipment151515Since 2019 Microsoft and the US Army have
collaborated on using HoloLens to enhance soldiers’ situational awareness
https://news.microsoft.com/transform/u-s-army-to-use-hololens-technology-in-
high-tech-headsets-for-soldiers/. Similarly, the use of voice user interfaces
(like virtual assistants) may be a means to mimic a study companion that can
ask questions, give feedback and help with misunderstandings. Adaptive
educational technology should allow and encourage the creation of these varied
activities to prepare the students for dynamic and changing environments where
learning and adaptability are essential.

The main focus of current implementations of adaptive learning technologies is
the content personalization engine. However, although being a crucial
component of adaptive experiences, the delivery of said personalized content
happens through a custom-designed student platform. That may be a good fit for
some activities, but it is a limiting factor for others, such as those in the
aforementioned examples.

Figure 1.  architectural overview.

##  3\.

aims to provide an open, content-agnostic and extensible framework (see Figure
1 for its architecture) for designing and consuming adaptive and gamified
learning experiences. We imagine a student experience entirely tailored to
their needs and choices. For example, we think students should be able to do
some lessons and quizzes with Alexa, switch to VS Code to do some coding
activities, and then move to another frontend (i.e., Moodle) to do something
else, all without friction. That is why we exploit the flexibility of .NET
Interactive161616https://github.com/dotnet/interactive in ’s execution engine.
Students’ interactions with external tools occur through Adapters that bind
actions on the student frontend to .NET Interactive commands and bind .NET
Interactive events to a supported output (e.g. audio for Alexa) with custom
formatters.

The word ”adaptive” in adaptive learning paths means being able to adjust the
students’ needs, like assigning exercises based on their previous responses or
the capabilities of the platform they are on. This adaptation is allowed by
the collaboration between the Execution Engine and the Backend. The former
handles the students’ submissions and validates their responses, while the
latter uses the results of the validation phase to assign the next activity in
the learning path.

Figure 2. Teacher design tool for learning fragments.

The two most important aspects of a tutoring platform are simplicity in the
content creation process and content availability to students for consumption
and teachers for reuse.

Designing and realizing learning fragments for such a variegated learning
experience might seem a daunting task at first. That is why includes a
Learning Path Editor heavily focused on the teachers’ experience (see Figure
2). Its graph-like, visual-editing capabilities and integration with the rest
of the platform allow the creation of ready-to-use learning fragments with
ease, thanks to the abstractions provided. The teacher design tool is composed
of two main elements: the drawing area and the properties panel. The drawing
area is the core of the visual editing experience. Nodes can be added,
connected, and rearranged with simple clicks or drag-and-drop interactions.
Nodes and edges themselves can be interactive components that augment the
visual editing experience. The properties panel, instead, provides fine-tuning
tools to define the activities and the links between them.

To define the learning fragment, a teacher can create new activities with a
right-click on the canvas, change their type and parameters from the
properties panel, and connect them by dragging from the handle on the source
node to the handle on the destination node. By clicking on an edge, the
teacher can edit the link condition by choosing from a list of existing
abstractions or by writing their own validation code.

The tool is implemented using web technologies with
React171717https://reactjs.org/ and
TypeScript181818https://www.typescriptlang.org/ with a particular focus on
flexibility and extensibility. Its design allows it to be part of a more
comprehensive learning management tool. The drawing area and the properties
panel are different abstract views that operate on the same data differently.
Both components manipulate the same PolyglotElements (i.e. nodes and edges) by
working on the shared state via Zustand191919Zustand is a lightweight,
unopinionated state-management library for React
https://github.com/pmndrs/zustand actions.

The adaptation process necessary for effective eTutoring relies on abstract
activities: activities defined only in terms of their goals and not on
concrete exercises the students have to solve. Those abstract activities are
then replaced during the runtime refinement phase with the most suitable
composition of existing fragments that satisfies the goal decided by the AI
planner. The latter exploits the automatic generation of learning paths by
adapting an existing approach based on AI planning presented in (Bertoli et
al., 2009). The chosen fragments may have other abstract activities, so
further refinement stages may be needed.

Gamification means creating a game narrative that guides players through
increasingly complex challenges, keeping them engaged with social activities
such as group work or competitions. It means providing immediate feedback (as
expected from a game-like environment) and students taking autonomous choices
to progress down the individually decided path. Gamification is not an add-on.
Instead, gamification mechanics are fundamental to the learning path
personalization process in two ways. Not only do they keep the students
engaged, but they can also be used as tools to gain insight into the student’s
behaviour from a different perspective and thus help generate a more
personalized and engaging learning path. In order to increase engagement, the
gamification mechanics must be calibrated according to the underlying
activities. That is why gamification fragments may be composed by the AI
planner similarly to learning fragments.

##  4\. The Demonstration

The chosen scenario is part of an introductory statistics course that fits a
broader data science program. Its goal is for the student to understand the
concepts of average, median, and the difference between them. We have
identified four kinds of concrete learning activities needed to define the
scenario:

  * •

lessons

  * •

close-ended questions

  * •

quizzes

  * •

coding activities

We want students to first learn about the average, then the median, and then
how to code them. We also want students to understand the average, the median,
and the difference between them, so the fragment includes a review lesson and
an additional quiz in case the student fails the close-ended question. Once
the student has completed the ”average” section, they move to the median,
where they will face the same pattern for recovery used for the average.
Suppose that the closed-ended question requires the student to calculate the
median. Since understanding the difference is part of the goal, if the student
answers with the average value instead of the median, they should be given a
particular review lesson that highlights the differences instead of a regular
review lesson. All those different alternative paths merge before the coding
activities. While the other exercises have a known correct answer, coding
activities do not. Furthermore, to be considered correct, a program may not
only be required to have a correct output (e.g. through unit tests) but also
to have some properties on the source code itself (e.g. low cyclomatic
complexity202020https://en.wikipedia.org/wiki/Cyclomatic_complexity or proper
encapsulation212121https://en.wikipedia.org/wiki/Encapsulation_(computer_programming))
or on its execution (e.g. time or memory constraints). As evidenced by the
previous example, fragments must not pose any limit on the relationship
between inner activities. The complete fragment’s implementation is shown in
Figure 2 using the Learning Path Editor.

Figure 3. Execution Engine Lifecycle.

The student can consume learning fragments thanks to ’s Execution Engine. This
component constitutes the core of the student experience. It is responsible
for handling user interactions and validating students’ submissions. Its
straightforward lifecycle (see figure 3) is designed to exploit
Journey222222Journey is a .NET Interactive library for defining open-ended
graphs of challenges’s strengths by using it in a semi-controlled fashion.
When the assigned activity and its outgoing edges are retrieved from the
backend, they undergo a transformation phase where the activity is converted
into a Journey challenge and edges into Journey rules. This process is
essential because it allows running the validation code mentioned previously.
Once converted, the challenge is ready to be launched. The activity is ”shown”
to the student in their frontend, thanks to .NET Interactive formatters. When
the active frontend adapter produces an event, Journey intercepts it and runs
the registered rules against the student submission. then sends the result of
this validation phase back to the backend that will use this additional
knowledge to suggest the next suitable activity. At the moment of writing, the
students can execute the learning fragment with Alexa (see Figure 4) or VS
Code for coding activities (see Figure 5).

Figure 4. Alexa frontend interaction. Figure 5. VS Code frontend interaction.

##  5\. Conclusions and Future Directions

In this paper we presented an innovative eTutoring platform characterized by
its adaptivity and gamification mechanics. An overview of its architecture has
been presented together with a practical demonstration to deepen how processes
are linked and how the framework would handle the activities lifecycle. The
work done and presented here has been a good experiment to set-up a set of
research directions that we highlight in the following.

Adaptive education still remains an open research topic at the intersection of
several different domains like cognitive sciences and informatics. We have
identified some directions to explore in the future:

  * •

Abstract activities are at the core of our formalization. Extending to include
them is our highest priority. To do that, we need to include them in the
editor, design a tool for defining goals, and integrate the AI planner to
include the runtime refinement and harness the power of abstract activities.

  * •

Runtime refinement is one of the most powerful concepts in the entire
framework. However, it is most effective only if there is a substantial amount
of fragments to choose from. We think that creating an open fragment database
would benefit students (with better refinement and high-quality educational
content) and teachers who can focus on the big picture and let the platform
adjust the details.

  * •

To further extend the previous point, leveraging open educational resources
with NLP techniques for automatic content annotation would significantly
contribute to the database. This task is one of the main goals of Project
Encore232323https://grial.usal.es/encore, an ERASMUS+ project that involves
multiple universities and companies.

  * •

Allowing students to use arbitrary frontends for their tasks is indeed
challenging. This support is still experimental and needs more work to ensure
the quality of the learning experience. Furthermore, not all frontends are
suitable for all kinds of tasks: for example, using Alexa to do a coding
exercise might not be the best idea. We need to further define what ”suitable”
means and implement something similar to a content negotiation mechanism.

  * •

To easily define the fragments, provides some basic abstractions for
connections such as ”Pass/Fail”. As explained previously, this abstraction may
be correct for some exercises but not applicable to others. We want to further
explore this topic to enhance the correctness of existing fragments, validate
connections to prevent the creation of erroneous links, and define other
composable abstractions.

  * •

We plan to add support for existing learning management
systems242424https://en.wikipedia.org/wiki/Learning_management_system (e.g.
Moodle LMS) in the same modular fashion as we are doing with students’
frontends. This extension would allow incremental integration of parts of
within existing learning infrastructures and increase the adoption of adaptive
learning technologies.

  * •

The combined evolution of the engagement and learning systems is still an open
research problem. Gamification mechanics are effective only when they fit the
learning path perfectly. Usually, it is the gamification designer’s duty to
design the game narrative and the other game elements to achieve that fit.
However, the power of adaptive learning paths comes from the runtime
refinement; therefore, paths are not available upfront. The engagement system
must then evolve through an automatic calibration phase to the mechanics to
the underlying activities.

  * •

The learning path planning is crucial for effective refinement. We aim to
enhance the planner by leveraging machine learning techniques using learners’
behavioural data. Our goal is to improve contextual activity suggestions (and
hopefully feedback), considering non-standard parameters like the capabilities
of the interface the student is using.

###

## ACKNOWLEDGEMENTS

This work is supported by the Erasmus+ 101055893 project , named ”ENCORE
-ENriching Circular use of OeR for Education”, funded by the European
Commission.

## References

  * (1)
  * Bassanelli et al. (2022) Simone Bassanelli, Nicola Vasta, Antonio Bucchiarone, and Annapaola Marconi. 2022.  Gamification for behavior change: A scientometric review.  _Acta Psychologica_ 228 (2022), 103657.  https://doi.org/10.1016/j.actpsy.2022.103657
  * Benware and Deci (1984) Carl A Benware and Edward L Deci. 1984.  Quality of Learning With an Active Versus Passive Motivational Set.  _American Educational Research Journal_ 21, 4 (1984), 755–765.  https://doi.org/10.3102/00028312021004755 arXiv:https://doi.org/10.3102/00028312021004755  https://doi.org/10.3102/00028312021004755.
  * Bertoli et al. (2009) P. Bertoli, R. Kazhamiakin, M. Paolucci, M. Pistore, H. Raik, and M. Wagner. 2009\.  Control Flow Requirements for Automated Service Composition. In _Proc. ICWS’09_. 17–24.
  * Bucchiarone et al. (2021) Antonio Bucchiarone, Antonio Cicchetti, Nelly Bencomo, Enrica Loria, and Annapaola Marconi. 2021\.  Gamified and Self-Adaptive Applications for the Common Good: Research Challenges Ahead. In _2021 International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS)_. 149–155.  https://doi.org/10.1109/SEAMS51251.2021.00028 https://doi.org/10.1109/SEAMS51251.2021.00028.
  * Deslauriers et al. (2019) Louis Deslauriers, Logan S. McCarty, Kelly Miller, Kristina Callaghan, and Greg Kestin. 2019\.  Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom.  _Proceedings of the National Academy of Sciences_ 116, 39 (2019), 19251–19257.  https://doi.org/10.1073/pnas.1821936116 arXiv:https://www.pnas.org/doi/pdf/10.1073/pnas.1821936116  https://doi.org/10.1073/pnas.1821936116.
  * Deterding et al. (2011a) Sebastian Deterding, Dan Dixon, Rilla Khaled, and Lennart Nacke. 2011a.  From game design elements to gamefulness: defining” gamification”. In _Proceedings of the 15th international academic MindTrek conference: Envisioning future media environments_. 9–15.
  * Deterding et al. (2011b) Sebastian Deterding, Miguel Sicart, Lennart Nacke, Kenton O’Hara, and Dan Dixon. 2011b.  Gamification. using game-design elements in non-gaming contexts.  In _CHI’11 extended abstracts on human factors in computing systems_. 2425–2428.
  * Elmqaddem (2019) Noureddine Elmqaddem. 2019\.  Augmented Reality and Virtual Reality in Education. Myth or Reality?  _International Journal of Emerging Technologies in Learning (iJET)_ 14, 03 (2019), 234–242.  https://doi.org/10.3991/ijet.v14i03.9289 https://doi.org/10.3991/ijet.v14i03.9289.
  * IIEP (2020) UNESCO IIEP. 2020\.  SDG 4 - Policies for flexible learning pathways in higher education: taking stock of good practices internationally.  https://unesdoc.unesco.org/ark:/48223/pf0000372817.locale=en.
  * Koedinger et al. (2016) Kenneth R. Koedinger, Elizabeth A. McLaughlin, Julianna Zhuxin Jia, and Norman L. Bier. 2016\.  Is the Doer Effect a Causal Relationship? How Can We Tell and Why It’s Important. In _Proceedings of the Sixth International Conference on Learning Analytics & Knowledge_ (Edinburgh, United Kingdom) _(LAK ’16)_. Association for Computing Machinery, New York, NY, USA, 388–397.  https://doi.org/10.1145/2883851.2883957 https://doi.org/10.1145/2883851.2883957.
  * Prusty and Russell (2011) B Gangadhara Prusty and Carol Russell. 2011.  Engaging students in learning threshold concepts in engineering mechanics: adaptive eLearning tutorials. In _17th International Conference on Engineering Education (ICEE)_. 21–26.  https://www.ineer.org/Events/ICEE2011/papers/icee2011_submission_250.pdf.