# AIDX: Adaptive Inference Scheme to Mitigate State-Drift in Memristive VMM
Accelerators

Tony Liu, Amirali Amirsoleimani, Fabien Alibart, Serge Ecoffey, Dominique
Drouin, and Roman Genov Tony Liu, Amirali Amirsoleimani, and Roman Genov are
with the Department of Electrical and Computer Engineering, University of
Toronto, 10 King’s College Road, Toronto, Ontario, Canada. Fabien Alibart,
Serge Ecoffey and Dominique Drouin are with the Interdisciplinary Institute
for Technological Innovation - 3IT, University of Sherbrooke, Qc, Canada. (A.
Amirsoleimani corresponding author email: amirali.amirsoleimani@utoronto.ca).

###### Abstract

An adaptive inference method for crossbar (AIDX) is presented based on an
optimization scheme for adjusting the duration and amplitude of input voltage
pulses. AIDX minimizes the long-term effects of memristance drift on
artificial neural network accuracy. The sub-threshold behavior of memristor
has been modeled and verified by comparing with fabricated device data. The
proposed method has been evaluated by testing on different network structures
and applications, e.g., image reconstruction and classification tasks. The
results showed an average of $60\%$ improvement in convolutional neural
network (CNN) performance on CIFAR10 after $10000$ inference operations as
well as $78.6\%$ error reduction in image reconstruction.

###### Index Terms:

Memristor, Crossbar, Vector-Matrix Multiplication, Inference, State-Drift,
Neural Network.

##  I Introduction

Rresistive switching memory crossbars have emerged as potentially high-speed
and low-power accelerators for vector-matrix multiplication (VMM) [2].
However, non-idealities and defects in these platforms dramatically impact the
neural network (NN) performance and accuracy. One of the significant and not
extensively studied non-ideal phenomena is memristance drift [3] and it occurs
in different types of resistive switching memory technologies in various ways.
For instance, phase change memories (PCM) will experience increasing
resistance due to drift, even when there is no voltage applied over the cell
[4]. On the other hand, for memristors, state-drift from their programmed
state happens as a result of many repeated VMM operations which leads to the
computational accuracy degradation (Fig. 1).

Figure 1: (a) Neural network (NN) forward pass VMM performed on a memristor
crossbar. (b) Degradation of neural network weights and output accuracy over
time. (c) 3D memristor crossbar array structure. (d) Percentage change of
memristors conductance in a randomly chosen subset of NN weights done on a
simulated memristor crossbar. (e) Impact of simulated state-drift on MNIST
classification accuracy with varying number of hidden layer. (f) Sub- and
over-threshold behaviour of device transition oxide filament.

Previous studies [5, 6, 7] on memristance drift in memristor technology have
mainly been focused on high-density memory where memristors are used solely
for storage rather than computation. More recent reports on drift[8] for
computational memristor crossbars include an inline calibration approach [9]
which involves optimizing the calibration time of the memristor crossbar. By
performing polynomial fitting on the computational error data, a $21.77\%$
calibration efficiency is achieved. A closed-loop weight compensation based
solution is presented in [10] which minimizes the effects of state-drift by
increasing the computational service lifetime by $14.95\times$ and results in
approximately $70\%$ computational accuracy degradation within $1705$ read
operations. In this brief, we present an adaptive inference scheme (AIDX) as a
flexible optimization procedure that automatically adapts to existing crossbar
non-idealities and circuit parasitics and can be applied to any VMM-based
task. According to experimentally verified simulations, AIDX cuts accuracy
loss due to the device state-drift in modern convolutional neural networks
(CNN) by more than $60\%$ as well a $78.6\%$ error reduction in image
reconstruction.

##  II Preliminary

###  II-A Impact of Memristance Drift on Crossbar MAC Operations

Memristance drift is defined as the unintended small changes in memristor
conductance caused by a low-voltage read/inference operation. Ideally, for the
ideal weight distribution $(\textbf{G})$ the output current $j$-th column
$I_{j}$ is given by $I_{j}=\sum_{i}G_{ij}V_{i}$ (Fig. 1(a)). We can define the
memristance drift caused by the $k$-th inference operation as $\delta G_{k}$
and the conductance of the memristor at the $(k+1)$-th iteration as:

$G_{k+1}=G_{k}+\delta G_{k}$ (1)

The total memristance drift due to the $k$-th operation is $\Delta
G=\sum_{i}^{k}\delta G_{i}$. As such, the real output current of the $j$-th
column at the $k$-th operation is $I^{\prime}_{j}=\sum_{i}(G_{ij}+\Delta
G_{ij})V_{i}$. The current error $I-I^{\prime}$ due to memristance drift can
be quite problematic in larger crossbars because current scales with crossbar
size. However, a differential mapping scheme can prevent the build-up of
memristance drift error in very large arrays because the error in the positive
column will scale at the same rate as the negative column. Fig. 1(b)
illustrates the concept of small changes in NN weights accumulating into much
larger errors in the output layer. Fig. 1(c) illustrates 3D structure of the
network in Fig. 1(b). Fig. 1(d) shows sample heatmaps of simulated $32\times
32$ array of memristors conductance changes due to memristance drift.The
bottom row represents the bias weights of a NN and they are initially mapped
to a high-conductance state which is why it is the only row with reduction in
overall conductance. Fig. 1(e) examines the state-drift impact on MNIST
classification task for multi layer perceptron network with various number of
hidden layers and Fig. 1(f) illustrates the difference between above- and sub-
threshold memristor switching.

###  II-B Memristance drift modeling and analysis

Behaviour-based memristor models are typically used in memristor crossbar
simulations due to their simplicity and light computational load. However,
most behaviour-based models do not consider memristance drift and approximate
the internal state change due to an applied sub-threshold voltage to be zero.
To address this issue, we propose an extension to the popular VTEAM model [11]
that accounts for the minute changes in internal state due to sub-threshold
voltages. For model consistency, we adopt a similar mathematical structure in
the sub- and above-threshold region:

$\frac{dw(t)}{dt}=\begin{cases}k_{s,off}\cdot\left(\frac{v(t)}{v_{off}}\right)^{\alpha_{s,off}}\cdot
f_{s,off}(w),&\text{if }0\leq v<v_{off}\\\
k_{s,on}\cdot\left(\frac{v(t)}{v_{on}}\right)^{\alpha_{s,on}}\cdot
f_{s,on}(w),&\text{if }v_{on}<v<0\end{cases}$ (2)

Here, $v_{off}$ and $v_{on}$ represent the RESET and SET voltage thresholds
respectively. $w(t)$ is the internal state variable and is related to the
resistance $R$ of the memristor as $R(t)=R_{off}w(t)+R_{on}(1-w(t))$.
$k_{s,off}$ and $k_{s,on}$ are fitting parameters that represent the rate of
ion migration at any given applied sub-threshold voltage. Similarly,
$\alpha_{s,off}$ and $\alpha_{s,on}$ are parameters that characterize the
exponential relationship between speed of ion migration and the applied
voltage. $f_{s,on}(w)$ and $f_{s,off}(w)$ are window functions that bounds the
state between $0$ and $1$. The time derivative of the resistance can be
expressed as:

$\frac{dR(t)}{dt}=R_{off}\frac{dw(t)}{dt}-R_{on}\frac{dw(t)}{dt}$ (3)

$R_{on}$ and $R_{off}$ are low and high resistance state of device,
respectively. Cycle-to-cycle and device-to-device variations in sub-threshold
drift speed are modelled by adding 15% random Gaussian noise to $k$ and
$\alpha$ parameters. The probability density function (PDF) of $k_{on}$ is
shown in Eqn. (4) where $k_{on}$ is the ideal, fitted parameter and $x$
represents $k_{on}$ with added Gaussian noise. The PDF of the other $k$ and
$\alpha$ parameters follow the same structure as Eqn. (4).

$f_{kon}(x)=\frac{1}{\sqrt{0.3k_{on}\pi}}e^{-\frac{(x-k_{on})^{2}}{0.3k_{on}}}$
(4)

To validate our proposed model, the VTEAM extension is applied to TiOx-based
memristor device (Fig. 2(a)) data. The extended VTEAM $k$ and $\alpha$
parameters were fit using simulated annealing algorithms and gradient descent
with SET and RESET voltage thresholds of $-0.6$V and $0.6$V. Fig. 2(b-c)
illustrates that the extended VTEAM models sub-threshold memristor behaviour
much more accurately than the original VTEAM. Fig. 2(d) shows a 3D plot of how
memristor switching behavior and conductance changes with internal state $w$
and applied voltage in sub-threshold region.

Figure 2: (a) Memristor device structure used for fitting. (b) Comparison of
$I$-$V$ fitting of the extended VTEAM with experimental data and the original
model in sub-threshold region for SET operation ($k_{s,on}=-8.445\times
10^{-6}$, $\alpha_{s,on}=6$). For clarity, we extracted a best fit curve to
represent the experimental data from over the threshold and extrapolated the
curve into sub-threshold region by keeping the gradual drift trend.

(c) Fitting comparison with experimental data for RESET operation
($k_{s,off}=1.126\times 10^{-7}$, $\alpha_{s,off}=5$). (d) 3D characterization
of the extended VTEAM for the same device with respect to the internal state
variable $w$ and voltage.

##  III Methodology

###  III-A Problem and Assumptions

By formulating the issue of memristance drift as an optimization problem, we
can develop an optimization scheme to minimize accuracy degradation. With no
memristance drift, the ideal mean squared error (MSE) is
$E_{0}=\sum_{j}(y_{j}-\sum_{i}G_{ij}V_{i})^{2}$ and the real MSE at the $k$-th
operation is $E_{k}=\sum_{j}(y_{j}-\sum_{i}(G_{ij}+\Delta G_{ij})V_{i})^{2}$.
Where $V_{i}$ is the voltage applied to $i$-th row and $\Delta G_{ij}$ is the
total memristance drift of the $ij$-th memristor from its originally
programmed value. We define the error due to memristance drift $E_{Drift}$ as
the difference in MSE between the initially programmed state ($E_{0}$) and the
$k$-th inference operation ($E_{k}$). As an optimization problem, the goal is
to minimize the increase of $E_{\mathrm{Drift}}$ with respect to time. The
change in conductance due to memristance drift, $\Delta G$, can mainly be
optimized by input to voltage amplitude mapping and relative inference voltage
pulse width. Factors that cannot be easily changed such as specific memristor
characteristics and overall crossbar structure will be ignored in the
optimization procedure.

Figure 3:  (a) AIDX’s optimized parameters A and D mapped onto input voltage
pulses. (b) Proportional input inversion to balance memristance drift error
and prevent constraint violations. (c) AIDX Design flowchart. (d) AIDX
optimized parameters for duration ($D_{\mathrm{opt}}$) and amplitude
($A_{\mathrm{opt}}$) are applied to each CNN layer’s input separately. (e)
Simulated memristance drift with and without AIDX over time with 15% device-
to-device variations. To better observe the total state-drift, all devices’
initial conductance are set to $0.0052$ $\mathrm{S}$ and a positively skewed
pre-generated random sequence of voltage pulses were applied to half the
memristors and a negatively skewed pulses to the other half. (f) Percentage
change in conductance of the same simulated memristors shown in Fig. 1(d) by
utilizing AIDX.

1\. Obtain a direction $\mathbf{p}_{k}$ through solving
$B_{k}\textbf{p}_{k}=-\nabla f(\mathbf{x}_{k})$

2\. Perform $\mathrm{1D}$ Line Search to find step size $\alpha_{k}$ such that
$\alpha_{k}=argminf(x_{k}+\alpha\mathbf{p}_{k})$

3\. $\textbf{s}_{k}=\alpha_{k}\textbf{p}_{k}$

4\. $\mathbf{x}_{k+1}=\mathbf{x}_{k}+\mathbf{s}_{k}$

5\. $\textbf{y}_{k}=\nabla f(\mathbf{x}_{k+1})-\nabla f(\mathbf{x}_{k})$

6\.
$B_{k+1}=B_{k}+\frac{\mathbf{y}_{k}\mathbf{y}_{k}^{T}}{\mathbf{y}_{k}^{T}\mathbf{s}_{k}}-\frac{B_{k}\mathbf{s}_{k}\mathbf{s}_{k}^{T}B_{k}^{T}}{\mathbf{s}_{k}^{T}B_{k}\mathbf{s}_{k}}$

7\. Repeat 1-6 until $x$ converges.

Algorithm 1 BFGS Algorithm

###  III-B Optimization Methodology

We will frame the minimization of $E_{Drift}$ as an unconstrained optimization
problem where A is the input to voltage amplitude mapping and D is the
relative voltage pulse width:

$min_{A,D}E_{Drift}(\textbf{A},\textbf{D})$ (5)

D is a vector that represents the length of a positive input read pulse
relative to a negative read pulse. For instance, a given row can have a
positive read pulse of $200ns$ while the negative read pulse is only $150ns$
long. Similarly, A is a vector whose elements represent the relative inference
voltage pulse amplitude ratio for positive to negative inputs (Fig. 3a). In
summary, AIDX modifies the amplitude and duration of inference voltage pulses
to minimize memristance drift for a given task. Even the minimum allowable
voltage pulse amplitude and widths will still result in noticeable memristance
drift after many inference operations. As such, AIDX is required to minimize
aggregate memristance drift through balancing the total drift in the SET and
RESET directions. We use the popular Broyden-Fletcher-Goldfarb-Shannon (BFGS)
algorithm [12] for this optimization problem. The BFGS algorithm is a quasi-
Newton method that relies on the gradient of the objective function to find
the optimal solution. However, $E_{Drift}$ is an unknown function that can
only be evaluated, so the $\nabla E_{Drift}$ had to be approximated using
finite-difference approach. Gradient-free optimization methods like Nelder-
Mead simplex method [13] were also explored, but quasi-Newton algorithms were
most effective for this problem.

###  III-C Constraint Violations

Normally, the optimized voltage amplitude ratio A and width ratio D can be
reasonably used. However, there are certain cases where elements of the
optimal A and D are far too large or too small to be implemented practically,
typically when the device characteristics and input distribution are heavily
skewed. To address this issue, we will first frame the optimization problem
through a different lens. Let’s start with a simplified scenario of a single
memristor with input data modelled by the discrete random variable $X$ with a
probability density function (PDF) of $f(x)$. Defining the time derivative of
the internal state $w$ for a given $x$ as

$\frac{dw(x)}{dx}=g(x).$ (6)

In our sub-threshold model, $g(x)$ is the same as Eqn. (2) as $v(t)$ replaced
with $v(x)$ which represents the mapping function of input $x$ to voltage
amplitude $v(x)$. The average rate of memristance drift given input
distribution $X$ is as follows:

$E[\frac{dw(x)}{dx}]=\sum_{x}g(x)f(x).$ (7)

The optimization problem over $E_{Drift}$ can also be reframed as minimizing
$\left|E[\frac{dw(x)}{dt}]\right|$ over all memristors where the $g(x)$
parameters for each memristor is sampled according to Eqn. (4) to account for
device-to-device variations. The AIDX scheme defined so far only affects
$g(x)$, but has not yet made any adjustments related to $f(x)$. If we allow
AIDX to first optimize over $f(x)$, the issue of impractical A or D can be
circumvented. One of the only useful recoverable transformations of the input
vector $x$ is inversion through multiplying by $-1$. By inverting a random
proportion $a$ of the input data, the input PDF is transformed into
$f^{\prime}(x)$:

$f^{\prime}(x)=(1-2a)f(x),0<a<1$ (8)

Impractical A or D only occur when either
$\left|E[\frac{dw(x)}{dt}]\right|>>0$ or $\left|E[\frac{dw(x)}{dt}]\right|<<0$
before applying AIDX. As such, if we can optimize
$\left|E[\frac{dw^{\prime}(x,a)}{dt}]\right|$ over $a$,
$\left|E[\frac{dw(x)}{dt}]\right|$ will be brought close to $0$ before
optimizing A and D which will therefore prevent any constraint violations.
Fig. 3(b) illustrates our approach to constraint violations.

###  III-D General Solution Flow

As it can be seen in Fig. 3(c), during pre-processing, optimization is done in
three separate scenarios to guarantee optimal fitting parameters. Once
hardware constraint violations are resolved with input data inversion, the
input circuits e.g. digital-to-analog converters (DACs) are adjusted to fit
the optimized input to voltage signal mapping parameters. The majority of AIDX
takes place during pre-processing which only needs to be done once for any
given task. The only difference in the AIDX inference operation as compared to
a normal inference operation is to recover the intended output current from an
inverted output through multiplying by $-1$. Fig. 3(d) summarizes the general
pipeline of AIDX. Fig. 3(e) shows the evolution of memristor state due to
memristance drift for AIDX and the baseline model and Fig. 3(f) is a heatmap
of a portion of the memristor crossbar at 1000 and 10000 inference steps where
the bottom row represents the bias. While memristance drift is a phenomenon
that can cause memristors to switch in both the set and reset direction as
seen in Fig. 3(e), almost all memristors within a crossbar will typically
drift in only one direction for image-based applications. These inputs are
almost entirely positive which causes an aggregate drift in the reset
direction. Other reasons for a unidirectional aggregate drift include: the
device switching speed is not the same in the set and reset direction and most
non-biased memristors are being initialized close to the high resistive state
where the drift speed is strongly skewed in the set direction (Fig 2(d)).

##  IV Results and Discussion

In this paper, all simulations are performed using our extended VTEAM
memristor model [14] by including the effects of sub-threshold state-drift
whose parameters are fit according to the experimental data shown in Fig. 2.
We integrate this memristor model into our existing 1T1R memristor crossbar
simulation to simulate both memristance drift and crossbar non-idealities like
sneak paths and line resistance. A differential weight mapping scheme is used
where each element is mapped onto a pair of memristors where one memristor
represents positive values and the other represents negative values. In Fig.
4(a), AIDX is tested across 10 baseline tasks from the Proben1 benchmark
datasets [14]. We trained a shallow 1-hidden layer NN for all of these tasks.
While there are large variations in baseline performance across different
tasks, it should be noted that all baseline tasks ended at around the same
classification accuracy as random guessing due to some tasks having more
classification categories than others.

Figure 4:  (a) Classification accuracy comparison between baseline and AIDX
across baseline tasks from Proben1 datasets. (b) Classification accuracy
comparison across different CNN architectures on CIFAR-10 image classification
dataset. (c) Sample reconstructed MNIST images and average image
reconstruction error from baseline and AIDX-enhanced auto encoders.

To verify our solution’s effectiveness for more practical applications, we
adopted AIDX for a selection of CNN architectures on the CIFAR10 dataset. The
CNN memristor crossbar mapping scheme used is similar to the one found in
[15]. Fig. 4(b) compares the performance of 10 different CNN architectures
between AIDX and the baseline model. As compared to the shallow NNs used for
the $\mathrm{Proben1}$ datasets, the CNNs had an overall higher speed of
accuracy degradation. The worse performance of CNNs is to be expected because
of error propagation from one layer to the next amplifying the effect of
memristance drift. The error in column $j$ of the $l+1-th$ layer in a fully
connected NN is:

$E_{j,l+1}=\sum_{i}^{n}V_{i,l+1}(\sigma(E_{i,l})+\Delta G_{ij,l+1})$ (9)

Here, $n$ is the total length of the input vector and $\sigma$ is the
activation function of the $l-th$ layer. Due to the large number of parameters
in modern CNNs, BFGS optimization in AIDX is performed sequentially layer by
layer to reduce optimization time. Applying AIDX to the selected CNNs provided
consistent improvements in classification accuracy on CIFAR10. The consistent
improvement in AIDX performance across varying sizes and designs of CNNs
demonstrates the proposed method flexibility across different crossbar sizes
and structure. In addition to classification tasks, we wanted to demonstrate
AIDX’s effectiveness in a different type of memristor crossbar application.
Fig. 4(c) shows the results of image reconstruction with the MNIST dataset.
For this task, a 1-hidden layer auto-encoder with $32$ hidden units was
trained off-chip which corresponds to a $24.5\times$ compression factor. With
AIDX, the average mean squared error has improved by $78.6\%$ over the
baseline after $10000$ inference operations.

##  V Overhead Analysis and Comparison

Different state-drift mitigation techniques have been compared with two
different AIDX configurations optimized for accuracy (AIDX-A) and power
efficiency (AIDX-P) to implement a MLP network in Table 1. AIDX-A is the
baseline AIDX method discussed in previous sections while AIDX-P adds a L2
regularization terms for A and D as follows:
$min_{A,D}(E_{\mathrm{Drift}}(\textbf{A},\textbf{D})+\lambda_{1}\sum
A^{2}+\lambda_{2}\sum D^{2})$. Where $\lambda_{1}$ and $\lambda_{2}$ are
regularization constants and regularizing the voltage amplitude and width
ratios allows AIDX-P to reduce the passive crossbar power consumption. For the
sake of consistency, we use the same estimates of peripheral power consumption
as [10]. Crossbar power consumption in AIDX is computed as the average power
consumed across the memristors in one inference operation. Area overhead is
defined as the percentage increase in on-chip area required for the
memristance drift solution due to peripherals, external circuit, and other
items. Accuracy improvement is the increase in classification accuracy
provided by a solution over the baseline model in a 1-hidden layer MLP at the
end of the baseline model’s defined lifetime. Performance lifetime is defined
as the amount of time required for a system to degrade to $70\%$
classification accuracy on the MNIST dataset. We chose this metric as an axis
of comparison primarily because it is used in [10] and is easily adaptable to
the Interrupt and Benchmark method used in [9]. Scalability is a measure of
how well a memristance drift solution’s performance and overhead scales with
crossbar size and additional layers in NN applications. Time overhead is not
shown in Table 1 because there is negligible time overhead introduced by all
solutions presented as compared to their respective baseline models.

TABLE I: Comparative analysis of AIDX. Methods |  [5] |  [6] |  AIDX-A |  AIDX-P
---|---|---|---|---
Power overhead (%) |  - |  $+1.61$ |  $+3.27$ |  $\mathbf{+1.19}$
Area overhead (%) |  $0$ |  $+2.34$ |  $0$ |  $0$
Performance life-time |  $1.22\times$ |  $14.85\times$ |  $\mathbf{37.62\times}$ |  $31.41\times$
Scalability vs Baseline |  Worse |  Worse |  Better |  Better
Accuracy improvement (%) |  $8.6$ |  $37.3$ |  $\mathbf{65.7}$ |  $57.4$
Include non-idealities |  No |  No |  Yes |  Yes

##  VI Conclusion

In this paper, we propose a new inference scheme based on voltage signal
optimization called AIDX to reduce the impact of memristance drift on
memristor crossbar MAC operations. By optimizing the voltage pulse width and
amplitude input mapping, AIDX is flexible and effective across a different
range of tasks including classification and image reconstruction. AIDX
minimizes the computational error due to memristance drift. AIDX provides up
to a $60\%$ and $78.60\%$ increase in classification accuracy on the CIFAR-10
datasets and image reconstruction of MNIST dataset, respectively. In addition,
we have proposed an extension to the popular VTEAM model to more precisely
simulate memristor behaviour below the switching voltage thresholds.

## Acknowledgment

This work is supported by NSERC HIDATA project and ERC-CoG IONOS n773228.

## References

  * [1] P. Yao et al., “Fully hardware-implemented memristor convolutional neural network,” Nature, vol. 577, no. 7792, pp. 641–646, 2020.
  * [2] C. Li et al., “Efficient and self-adaptive in-situ learning in multilayer memristor neural networks,” Nature comm., vol. 9, no. 1, pp. 1–8, 2018.
  * [3] T. Chang, et al., “Short-Term Memory to Long-Term Memory Transition in a Nanoscale Memristor,” ACS Nano, vol. 5, no. 9, pp. 7669–7676, 2011.
  * [4] S. Oh, et al., ”The Impact of Resistance Drift of Phase Change Memory (PCM) Synaptic Devices on Artificial Neural Network Performance,” in IEEE Electron Device Letters, vol. 40, no. 8, pp. 1325-1328, 2019.
  * [5] S. S-. Sheu et al., ”A 4Mb embedded SLC resistive-RAM macro with 7.2 ns read-write random-access time and 160ns MLC-access capability”, 2011 IEEE Int. Solid-State Circuits Conf. (ISSCC), pp. 200-202, 2011.
  * [6] Y. Chen, et al., ”A nondestructive self-reference scheme for spin-transfer torque random access memory (STT-RAM)”, 2010 Design Autom. $\&$ Test in Europe Conf. $\&$ Exh. (DATE), pp. 148-153, 2010.
  * [7] D. Niu, et al., ”Low power memristor-based RERAM design with error correcting code”, 2012 17th Asia and South Pacific Design Autom. Conf. (ASP-DAC), pp. 79-84, 2012.
  * [8] V. Joshi, et al., ”Accurate deep neural network inference using computational phase-change memory” Nature Comm., vol. 11, no. 1, pp. 1-13, 2020.
  * [9] B. Li, et al., ”Memristor-based approximated computation,” 2013 Int. Symp. on Low Power Elec. and Design (ISLPED), pp. 242-247, 2013.
  * [10] B. Yan, et al., ”A closed-loop design to enhance weight stability of memristor based neural network chips,” 2017 IEEE/ACM Int. Conf. on Computer-Aided Design (ICCAD), pp. 541-548, 2017.
  * [11] S. Kvatinsky, et al., ”VTEAM: A General Model for Voltage-Controlled Memristors,” in IEEE Trans. on Circuits and Sys. II: Exp. Briefs, vol. 62, no. 8, pp. 786-790, 2015.
  * [12] R. Fletcher, ”Practical methods of optimization. J. Wiley $\&$ Sons, 2013.
  * [13] L. S. Nelson, “Nelder-Mead Simplex Method,” Wiley StatsRef: Statistics Reference Online, 2014.
  * [14] L. Prechelt, ”PROBEN 1-a set of benchmarks and benchmarking rules for neural network training algorithms.”, 1994.
  * [15] F. Zhang, et al, ”Mitigate Parasitic Resistance in Resistive Crossbar-based Convolutional Neural Networks.” ACM Journal on Emerg. Tech. in Computing Sys. (JETC), vol. 16, no. 3, pp. 1-20, 2020.


# Experts prefer text but videos help novices: an analysis of the utility of
multi-media content

Hayeong Song  Georgia Institute of TechnologyAtlantaUSA<EMAIL_ADDRESS>,
Jennifer Healey  Adobe ResearchSan JoseUSA<EMAIL_ADDRESS>,  Alexa Siu
Adobe ResearchSan JoseUSA<EMAIL_ADDRESS>,  Curtis Wigington  Adobe
ResearchSan JoseUSA<EMAIL_ADDRESS>and  John Stasko  Georgia Institute of
TechnologySan JoseUSA<EMAIL_ADDRESS>
(2023; 2023)

###### Abstract.

Multi-media increases engagement and is increasingly prevalent in online
content including news, web blogs, and social media, however, it may not
always be beneficial to users. To determine what types of media users actually
wanted, we conducted an exploratory study where users got to choose their own
media augmentation. Our findings showed that users desired different amounts
and types of media depending on their familiarity with the content. To further
investigate this difference, we created two versions of a media augmented
document, one designed for novices and one designed for experts. We evaluated
these prototypes in a two-way between-subject study with 48 participants and
found that while multi-media enhanced novice readers’ perception of usability
(p = .0100) and helped them with reading time (p = .0427), time on task (p=
.0156), comprehension (p = .0161), experts largely ignored multi-media and
primarily utilized text.

multi-media, document intelligence, customization, novice, expert,
augmentation, reading conditions, text, image, animated, gif, video

††copyright: acmcopyright††journalyear: 2023††doi:
10.1145/3544549.3585900††conference: Make sure to enter the correct conference
title from your rights confirmation emai; April 23–28, 2023; Hamburg,
Germany††booktitle: Hamburg ’23: ACM Transactions on Computer-Human
Interaction, April 23–28, 2023, Hamburg, Germany††price: 15.00††isbn:
978-1-4503-XXXX-X/18/06††journalyear: 2023††copyright:
rightsretained††conference: Extended Abstracts of the 2023 CHI Conference on
Human Factors in Computing Systems; April 23–28, 2023; Hamburg,
Germany††booktitle: Extended Abstracts of the 2023 CHI Conference on Human
Factors in Computing Systems (CHI EA ’23), April 23–28, 2023, Hamburg,
Germany††doi: 10.1145/3544549.3585900††isbn: 978-1-4503-9422-2/23/04††ccs:
Human-centered computing Usability testing††ccs: Human-centered computing
Empirical studies in HCI††ccs: Interaction design User interface design††ccs:
Interaction design Interaction design process and methods

Figure 1. We designed two versions of the media-augmented documents for
_novices_ and _experts_. A _text_ document (all information in text) designed
for _experts_ had in-line text augmentation and media _rich_ document (text,
images, animated gifs, and a video) designed for _novices_ had multi-media
content. We evaluated these prototypes in a two-way between-subject study with
48 participants and found that while multi-media enhanced novices’ perception
of usability and helped them with reading time, quiz time, comprehension,
experts largely ignored multi-media and primarily utilized text.

Text document has all information in text with in-line text augmentation.
Media rich document has text, image, animated gifs, and video. An expert is
showing more positive sentiment toward the text document and a novice is
showing more positive sentiment toward the media rich document.

##  1\. Introduction

We believe that in the near future documents will naturally include multi-
media content such as interactive elements, animations, and videos. Our belief
is based on the observations that text documents continue to include more
images and that information is increasingly shared through infographics,
slides, and video presentations. On the internet, the multi-media content of
digital documents such as newspapers, web blogs, and social media is
continually increasing (Byers, 2022). While it has been shown that multi-media
content increases _engagement_ , for example, tweets with video are shared ten
times more than tweets without video (Southern, 2019), it has not been well
studied whether or not multi-media content is truly _desired_ or _helpful_ to
people seeking information. We hypothesize that both future documents will
contain multi-media content and that digital documents will be able to
customize the presentation of this content differently for different people.
In this paper, we present our design process for motivating such customized
reading experiences.

To investigate how people might want a text document to be augmented, we
conducted a formative study with ten participants. In the study, we gave
people a plain text procedural document about chemistry titration lab and
asked them to search the internet for multi-media content that they thought
would help them better understand the text. We used a procedural document that
we believed might challenge many people as the case study for our
investigation. We noted a difference in both the number of searches and the
types of media retrieved based on the people’s familiarity with the topic.
From this, we hypothesized that experts would prefer less multi-media content
and novices would prefer more, particularly with respect to video content.
This led us to design a second experiment that specifically analyzed the
preferences of novice and expert users with respect to specific types of
document augmentation.

In the HCI community, research on augmenting reading interfaces has been
ongoing to extend people’s cognition and provide fluid experiences, such as by
adapting forms and contents based on user needs (Chang et al., 1998; Egan et
al., 1989; Norman, 2013). However, less work has studied and compared how
people consume multi-media features in digital documents and reading
performance based on people’s familiarity with the document content (e.g.,
novices and experts). Our work takes a fine-grained approach to study the
differences in multi-media document consumption in novices and experts where
we consider the in-line text, images, animated gifs, and videos as augmented
content. Based on our formative study our hypotheses were 1) novices will
prefer more media augmentation, particularly videos, and 2) multi-media
augmentation will decrease novices’ quiz time, increase their understanding,
and reduce their cognitive load.

To test these hypotheses, we implemented two versions of a digital procedural
document about a titration experiment: _text_ focused version and media _rich_
version. The _text_ version was designed to meet our hypothesis about expert
preferences, including only plain text and in-line text resolution for
keywords. The media _rich_ version was designed to meet our hypothesis for
novice preferences and it included multi-media augmentation. We recruited 48
participants and presented them with the two prototypes in a nested between-
subject study to evaluate the usability of tailored prototypes, track quiz
time, and understand people’s multi-media features consumption patterns. We
used surveys and interviews to collect data. Our results showed that, first,
media preferences depend on people’s familiarity with the document content.
Second, multi-media content improves both consumption time and comprehension
for people unfamiliar with the topic. Finally, multi-media inclusion is
unnecessary for people already familiar with the topic, but it does not
negatively impact reading metrics for this group. The major contributions of
this work are the study and analysis of how people consume multi-media
features in a digital procedural document and evaluation and analysis of two
customized document experiences for two different types of users.

##  2\. Related Work

Augmented Reading Interfaces. In the HCI community, foundational augmentation
for reading interfaces aimed to provide a ”fluid” experience and customized
experiences to enhance people’s cognition, such as with interactive books
(Egan et al., 1989; Norman, 2013). These HCI-introduced prototypes support
crowdsourced answers to better explain the content in web pages (Chilana et
al., 2012) or provide previews (e.g., Wikipedia preview features) to help
people to jump through the content easily for navigational affordances
(Graham, 1999; Schilit et al., 1998) and to facilitate skim-reading. Some of
the reading interfaces specifically focused on augmenting plain text with
hypertext glossaries (Zellweger et al., 1998; Conklin, 1987) or with social
annotations (Hill et al., 1992). For example, these approaches were applied in
scientific reading, providing in-text definitions for nonce words (Head et
al., 2021) and localized context (Rachatasumrit et al., 2022). This can reduce
their loads for searching materials on the internet, which can often return
inconsistent or irrelevant results. This design was employed in a digital
document reading scenario, Amazon Kindles, that shows definitions for tricky
words in space between consecutive lines (kin, 2020). Although our work was
inspired from these works (e.g., provide contextualized definitions (Head et
al., 2021)), these works mostly focused on text augmentation, and less work
has studied augmenting reading interfaces with different types of media, such
as images, animations, and videos.

Learning Differences between Novices and Experts. Prior work has studied how
learning happens among novices and experts and their different learning
requirements (Patel, 2018; Collins and Evans, 2002; Castles et al., 2018;
Council et al., 2000). People’s familiarity with the content impacts their
engagement level (Nam et al., 2020; O’Brien et al., 2016) and comprehension
(McNamara and Kintsch, 1996; Ozuru et al., 2009). For example, people
unfamiliar with the content likely learn better when a document had a coherent
flow, whereas people familiar with the content were less impacted by
coherence. Because novices and experts have different learning requirements,
this even influences people’s web searching behavior as well. For example,
readers who are versed in web search and are subject domain experts were more
successful in web searches (Lazonder et al., 2000; Hölscher and Strube, 2000)
than those who were not.

To reduce the load of searching, there has been trend in digital documents
including multi-media content, such as newspapers with videos. But because
novices and experts have different learning requirements it can impact how
people comprehend this multi-media content (Chiu et al., 2020; Candello,
2013). Prior work has also studied how learner expertise affects the
effectiveness of multi-media content (Kalyuga, 2013; Chiu and Mok, 2017;
Kalyuga, 2014). These studies showed that multi-media presentations that are
effective for low-knowledge learners lose their effectiveness for
knowledgeable learners (expertise reversal effect). Without careful design
considerations, the inclusion of multi-media content in future documents could
actually detract from the reading experience (Tversky et al., 2002). However,
it has not been well studied, if people desire multi-media content and how
they consume them when reading the digital document. Thus, we should study how
people differentially consume multi-media based on people’s familiarity with
the content and consider this as a factor when designing reading interfaces to
customize people’s experiences. This type of customization can benefit users,
which can be part of transitioning novices to experts regarding the subject
matter effectively with UI support (Cockburn et al., 2014).

In summary, there has been extensive work in designing augmented documents for
a general audience and in studying the differences in how novices and experts
read text documents, but less work has studied how novices and experts
differently consume augmented multi-media, in particular video. We were
inspired by Leake et al’s work (Leake et al., 2020), in which they studied
general multi-media content preferences and developed a system that auto-
generates slide shows from texts. It also goes beyond researching general
preferences to specific differences in multi-media consumption between expert
and novice users. For prototyping, the most similar to ours is ScholarPhi
(Head et al., 2021) because it aimed at augmenting a reading interface with
in-line text definitions. Our work expands on this scope by further augmenting
the reading interface with different richness levels of media content (images,
animated gifs, and videos).

##  3\. Formative Study

Figure 2. In our formative study, we conducted an observational study that
aimed to understand user preference for media types (e.g., text, images,
videos), when reading procedural documents.

A man reading a document is given break after each paragraph to look for
additional materials (e.g., text, image, video) to better understand the
document.

We wanted to understand how and why people would choose to read documents with
more or less media content. To discover this, we conducted a formative study
with ten participants recruited from a convenience sample. To better
understand real user preferences around media consumption and reading
behaviors related to mixed mode documents, we designed an experiment where
participants were presented with a text document where each section was
followed by a break where participants could search for additional text,
images, and video content to aid their understanding (visualized in Figure 2).
We used a procedural document that we believed might challenge many people, a
chemistry titration lab, as the case study for our investigation.

The participants were all graduate students, with backgrounds that included,
research, development, and design. Participants were recruited via flyers and
word-of-mouth. Before beginning, we obtained participants’ consent to record
the session. We introduced the study as an investigative study of how the
internet can help them better understand documents. Participants were all
familiar with internet searches. At the beginning of the session, to motivate
participants to do their best to understand the material we informed them that
there would be a quiz at the end that they would have to pass. We told
participants that they should take as much time as they needed and search for
as much material as they needed to fully understand the document. We asked
participants to “think aloud” as they read and searched. During the study, we
noted their comments, how often they searched, and what types of media they
found. After they finished reading, we presented them with the comprehension
quiz and one question regarding their satisfaction with their search results.
We followed this question with a semi-structured interview. In the interview,
we asked relatively open-ended provocation questions focused around: each
participant’s reading experience with the text-only document, their search
experience, and their preferences for different types of media.

###  3.1. Key Insights

During the interviews, we asked participants about both their reading
experience and their searching experience. When participants spoke about the
document, one key theme that emerged was how familiar they were with the
material. We began to note participants’ reading speed, the types of content
they searched for, and how easy it was for them to find the information they
needed seemed to depend on how well they already knew the material in the
document. As this insight emerged, based on participants’ comments, but not on
any specific evaluation, we coded our insights in terms of whether or not we
believed the participant to be “familiar” with the content or “unfamiliar.”
For example, if the participants stated that they were chemistry majors or
stated that they have conducted titration experiment multiples times, we coded
them as _familiar_. Whereas, if participants stated that they majored in
liberal arts and stated that they haven’t done titration experiments before,
we coded them as _unfamiliar_. We were able to clearly identify three of the
participants as familiar with (P3, P7, P8) and three of the participants as
unfamiliar (P1, P6, P10) with the material. For the remaining participants, we
were not able to make a clear determination (P2, P4, P5, P9), as they did not
clearly state their familiarity with the content. We refer to our identified
participants as the _familiar_ and _unfamiliar_ groups.

Time spent on document. _Familiar_ readers were more likely to skim through
the document as they knew most of the concepts already. P3 one of the
participants who skim-read the document stated that _“To give you some
background, I majored in chemistry when I was an undergraduate. I have done
titration experiments multiple times, so I am very familiar with titrations”_.
_Unfamiliar_ readers spent both more time reading the document and more time
searching for unfamiliar terms on the internet, as they were learning about
the materials as they read. P7 stated that _“I am really not familiar with
chemistry so this will be an interesting document. […] I don’t know what
[word] this means, I am going to look it up. Actually, I don’t know if I am
looking at the right [internet search] results. […] Because I don’t really
know these [apparatus] things, I had to look them up which took me longer to
read. Let me know if I am taking too much time.”_

Satisfaction with internet search results. _Unfamiliar_ group were less
satisfied with their internet search results, primarily because they had more
difficulty using the right search terms to find the content they wanted. P1
stated that _“I have a design background. […] Because I am not familiar with
the terms, I did not know what to look for. […] Also, internet search results
were confusing because it uses different terms interchangeably and I get
confused if this refers to that or if they are the same thing.”_ _Familiar_
readers were more satisfied with their internet search results because they
were more often only trying to recall particular information with which they
were familiar and not trying to learn concepts from scratch. P8 stated that
_“I did a lot of chemistry in undergrad, so I am already familiar with this
kind of experiment. […] I had to look up a few terms and apparatus to recall
what they were and how to use them. […] A quick formula look-up and
definitions search was helpful.”_

Desired media augmentation. _Familiar_ readers preferred texts over richer
multi-media content such as videos. Because with texts they could skim through
contents quickly by using their familiar structure. They tended not to search
for videos or watch them because they perceived it as too much effort to
search through and watch videos. P7 stated that _“I would not need a video for
this. watching them will take too much time and I did not want to stop reading
the document or get interrupted while reading. Reading is better or text is
better because I can look at it when I am unsure. […] It would be good if the
definitions or formulas are supported in the document [as text]”_. In
contrast, _unfamiliar_ readers desired and searched for multi-media content
(images and videos) because they wanted to get a holistic picture of the
document content and learn more before reading more text information. P10
stated that _“I mean, video is usually always the most helpful. Because for
this specifically, the video I watched at the end, it covered everything that
I had looked up previously. So it was really helpful. […] And more
importantly, the video explained sequential steps, which was important for
accurately performing each step, and understanding sub-steps as well, which
was not covered in the text document.”_

###  3.2. Design considerations based on user preferences

In this section, we report on user needs that were found both across all
participants and some needs that were specific to more _unfamiliar_ readers.
We call out these user needs in conjunction with design considerations that we
refer to as D1, D2, D3, and D4.

  * •

D1. Identify key words. 6 of 10 participants expressed a desire to know which
words represented items or concepts that were critical to the task. When asked
how they would like to see important words called out, some participants
suggested highlighting, underlining, boldface, color coding, or hovering over
information. P6 stated that _“Because I don’t know these, I did not know what
to search for. Because when I was reading the document, I did not want to
search everything as I did not have full context about titration. […] It will
be good to denote keywords or important words I need to understand. Then I can
try to understand those first”_.

  * •

D2. Provide contextualized definitions. 6 of 10 readers from all groups
expressed a desire to have definitions of keywords that were curated to the
context (e.g. stir plate as it is used in titration), similar to what was done
in (Head et al., 2021). This was particularly important to the _unfamiliar_
group not only because it would provide relevant information right away, but
also because without context these participants didn’t have the knowledge to
write appropriate search terms and retrieve relevant content quickly. P1
stated, _“It will be good if I could see contextualized search results. When I
searched things on the internet the search results were too general. I will
look at those images, but I did not have the context to determine if I am
seeing the right things. I felt like I needed some guidance. […] Also, some of
the results were explained differently with different terms, which made me
confused.”_ Participants from the _familiar_ group also desired in-line text
definitions to recall concepts quickly, so they don’t have to bother with
searching. P7 stated that _“While reading when I encountered words that I
forgot about, it’s kind of like fill in the blank. I would search for that
term to fill in the blank. It will be good if the document provided
definitions that were curated to this experiment so that I don’t have to look
them up”_.

  * •

D3. Make relevant media easy to include or skip. Adding the right level of
multi-media content can provide better reading experiences (Takacs et al.,
2015) by keeping people engaged and aiding concept recall. We found that
multi-media content often helped _unfamiliar_ group. All of the participants
searched for images and videos as well as text. P10 stated that _“Because I
was a beginner I did not much about the experiment and things seemed to get
technical very quickly. Especially for the procedures, videos were helpful,
like understanding the consequence of a certain step. For example, for the
endpoint, the color changes immediately with one drop of the indicator. Being
able to see that visual goal was helpful in understanding the experiment.[…] I
wanted to go back and forth with a video and texts, watch a video to
understand procedures, and text to skim and recall what was said in the video.
So if the videos are segmented that would be good and I watch videos always on
2x speed to go through them quickly. Then the images were good for looking at
specific apparatus”_.

They said, images helped them know what the apparatus looked like and videos
helped them better understand how to actually conduct the experiment. By
contrast, _familiar_ group almost exclusively searched for text information.
While they would look at images that surfaced as a result of their text
queries, they did not view any videos. _Familiar_ group stated that they
preferred text as it enabled them to quickly go through the content. When
asked specifically about alternative media types such as animated gifs, all
_unfamiliar_ group expressed a desire for animated gifs showing how to do each
step whereas only a minority of _familiar_ group said that they would like
animated gifs in general but not for this content. P4 stated that _“Animated
gifs for each step would be helpful for following steps quickly, as it does
not interrupt the reading but its kind of like a quick preview for each
step.”_

  * •

D4. Support different reading patterns. The inclusion of multi-media is often
distracting (Takacs et al., 2015; Aagaard, 2019), and _familiar_ group did not
find it valuable. Because _familiar_ group mentioned that they skimmed the
document, we hypothesized that excluding multi-media content would decrease
distraction and create a more skim-friendly reading experience for this type
of people.

##  4\. Designing Media Augmented Documents for Novices and Experts

Based on the findings of the formative study, we wanted to further investigate
designing media-augmented documents specifically for people who are either
familiar or unfamiliar with the topic. We will now refer to _familiar_ readers
as _experts_ and _unfamiliar_ readers as _novices_. To this end, we developed
two different prototypes, a _text_ focused document for _experts_ and a media
_rich_ document for _novices_ following the design insights outlined in
Section 3.2.

###  4.1. Design Process

Our prototype development followed an iterative design process. To refine our
design, we repeated the process of ideation, prototyping, and getting feedback
in several rounds. Researchers met weekly to ideate early prototype design,
then conducted informal usability testing to get feedback every two to three
weeks. In these meetings, researchers discussed what multi-media features to
support and when and how to support them in digital documents. We actively
recruited potential users with various levels of expertise to get different
perspectives. Design alternatives for augmented reading interfaces were
identified through literature review (including hypertexts (Zellweger et al.,
1998), summarization (Allahyari et al., 2017), and expansions) and evaluation
of commercial tools (including Youtube’s video timestamp and Wikipedia’s
preview).

###  4.2. Stimulus Document

We used the chemistry titration lab as our stimulus document to build our
differently augmented multi-media documents. We use this scenario as a case
study. Knowing that it might be difficult to find appropriate media with which
to augment a text document, the text chemistry titration lab was derived from
content originally in a video format. The video explains how to carry out a
10-step titration experiment including the goal of the experiment, the
apparatus needed, and the procedural steps. We transcribed the video to create
the text-only version. We then had the original video from which to derive the
other multi-media content including images, animated gifs, and videos with
timestamps. To make the two prototypes as comparable as possible, we kept the
text identical but added different media types to the _novice_ version. We
implemented these prototypes using Adobe XD and with Anima (Anima, 2022)
plugin.

Expert version. We designed an expert version as a text-focused document.
Following the design insight outlined in section 3.2 we included _keyword
underlining_ and _in-line text augmentation_ to facilitate skim reading and
quick recall of key facts. Keywords were frequently used important words in
the titration experiment. These words were underlined and users could retrieve
the tooltip of a text definition (in-line text) by hovering over the
underlined word.

Novice version. We designed a second prototype for _novice_ group augmented
with _rich_ media content, following our design insights. This version
augments the _expert_ version with _keyword underlining & in-line text
argumentation, key images describing the apparatus, animated gifs describing
each step of the procedure, and a time-stamped bookmarked video_.

##  5\. Study Design

We wanted to evaluate if our customized digital document provided the right
level of media content for each of our identified groups: _novice_ and
_experts_. We designed a task in which each group read a document and then
took a quiz to understand how different groups consume multi-media features
and measure their comprehension. We allowed people to refer to the document
during the quiz. Our hypotheses for the experiment were:

  * •

(H1): The _expert_ group will prefer _text_ version and _novice_ group will
prefer the media _rich_ version.

  * •

(H2): The _expert_ group will read faster with _text_ version.

  * •

(H3): The _novice_ group will spend more time reading the _rich_ version but
will complete the quiz faster, score higher on a quiz, and experience reduced
cognitive load.

Our study design was a two-way between-subjects study for each version of the
document. We recruited separate cohorts of _experts_ and _novices_ using a
screener survey, to experience either the media _rich_ document or the _text_
document. In total, we recruited 48 participants (24 experts and 24 novices).
Within each cohort, participants were randomly assigned to either the _text_
or media _rich_ version of the document.

Participants. We recruited 48 participants (female: 22, male: 26) from diverse
backgrounds that included engineering, education, chemistry, product
management, and design. For moderated sessions, the study took 40 minutes to
complete and we compensated participants with a $35 gift card. For unmoderated
sessions, the study took 20 minutes and we compensated participants with a $10
gift card. The moderated sessions were identical to the unmoderated sessions
except that the instructions were given verbally by the moderator and the
session was followed by a semi-structured interview. For the unmoderated
version, participants read from the instructions. For moderated sessions,
users were recruited via word of mouth and flyers and for unmoderated
sessions, they were recruited via User Testing (use, 2023) platform. To
identify novices and experts, we used a screener that asked potential
participants about their backgrounds and areas of expertise. To qualify as
experts, participants needed to answer a majority of these questions (experts
above 70 %, novices below 50 %, out of 10 questions) correctly.

Procedure. For qualified participants, we collect consent for the study. All
participants were informed that they should take as much time as they needed
to read the document to fully understand it. In the beginning, they were told
there would be a comprehension quiz given after they finished reading, but
that they would be allowed to refer to the document during the quiz.
Participants were then allowed to read the document naturally, without further
guidance (unguided reading). The full versions of each prototype are shown in
Figure 1. The comprehension quiz was given after each participant finished
reading, followed by a survey that collected quantitative measures of the
usability (SUS) (Brooke, 1996) and cognitive load (Hart, 2006). The majority
of the experiments were unmoderated, using a remote testing platform (User
Testing (use, 2023)). We randomly selected two participants in each condition,
eight total, to participate in a moderated version of the study.

##  6\. Results

In this section, we report on both objective measures of performance including
reading time, quiz time, and comprehension scores as well as subjective
measures including scores for usability and NASA TLX mental demand and effort
level. These results are summarized in Table 1 \- 4 and described in detail
below. We additionally include qualitative feedback from the semi-structured
interviews of the moderated sessions.

###  6.1. Objective measures.

We tracked reading time and quiz completion time from our recorded screen
session videos. We also noted whether or not participants watched videos in
the media _rich_ document condition. T-test results are summarized in Table 1
and 2.

Table 1. Objective Measures Conditions | Reading Time (seconds) | Quiz Time (seconds) | Quiz score
---|---|---|---
Novice,rich | M = 319.58, SD = 101.08 | M = 149.99, SD = 45.92 | M = 8.66, SD = 1.15
Novice, text | M = 213.58, SD = 137.54 | M = 299.15, SD = 191.82 | M = 6.910, SD = 2.02
Expert,rich | M = 251.41, SD = 121.86 | M = 187.53, SD = 118.85 | M = 8.83, SD = 0.71
Expert,text | M = 175.41, SD = 79.36 | M = 164.97, SD = 92.02 | M = 8.58, SD = 0.66
Table 2. T-test Objective Measures Analysis. * denotes significance Compare | Reading Time | Quiz Time | Quiz Score
---|---|---|---
Novice,rich & Novice,text | *t(22) = 2.15 , p = .04 | *t(22) = -2.61 , p = .01 | *t(22) = 2.60 , p = .01
Expert,rich & Expert,text | t(22) = 1.81 , p = .08 | t(22) = 0.51 , p = .61 | t(22) = 0.42 , p = .67
Expert, text & Novice,text | t(22) = -0.83, p = .41 | *t(22) = -2.18, p = .03 | *t(22) = 2.71, p = .01
Novice, rich & Expert, rich | t(22) = 1.49, p = .15 | t(22) = -1.02, p = .31 | t(22) = -0.42, p = .67

Reading Time. We found significant differences in reading time, for _novices_
between the media _rich_ (M = 319.58s, SD = 101.08) document and the _text_ (M
= 213.58s, SD = 137.54) document, (t(22) = 2.15, p = .04). But we did not find
this significant difference for _experts_ across conditions, (t(22) = 1.81, p
= .08). This means that _novices_ in media _rich_ condition spent
significantly longer time and consumed multi-media features in the document
compared to _novices in text_ condition.

We noted that _novices_ more frequently choose to watch the video from the
beginning to the end (10 out of 12 participants) in the media _rich_ document
and all _novices_ played the video at normal speed. Only half (6 out of 12
participants) of _experts_ chose to watch a video in the media _rich_ document
and the majority of _experts_ watched only a portion of the video.

Quiz time. We similarly found significant differences in quiz time for
_novices_ between the media _rich_ document (M = 149.99s, SD = 45.92) and the
_text_ document (M = 299.15s, SD = 191.82), (t(22) = -2.61, p=.01). But we did
not find a significant difference for _experts_ across the conditions, (t(22)
= 0.51, p = .61). This means that _novices_ benefited from the media _rich_
document as they were able to finish quizzes on time faster than _novices_ in
_text_ condition. On the other hand, _experts_ did not significantly benefit
from multi-media content.

We also found a significant difference between _experts_ and _novices_ with
respect to completing quiz for the _text_ document with _experts_ spending (M
= 164.97s, SD = 92.02) and _novices_ spending (M = 299.15s, SD = 191.82),
(t(22) = -2.18, p = .03). This means that _novices in text_ condition spent a
significantly longer time in finishing a quiz than _experts in text_
condition.

Quiz score. We again found significant differences for _novices_ between the
_text_ document (M = 6.91, SD = 2.02) and the media _rich_ document (M = 8.66,
SD = 1.15), ((t(22) = 2.60, p = .01). But we did not find significant
differences for _experts_ across the conditions, (t(22) = 0.42, p = .67). This
means that _novices_ in media _rich_ condition, who had access to multi-media
content scored significantly higher on the quiz than those in _novices in
text_ condition. In both conditions, _experts_ scored high on quizzes
regardless of having access to multi-media content. We hypothesize that this
is because _experts_ knew most of the content already and they were using the
procedural document to recall concepts.

We also found a significant difference between _experts_ and _novices_ with
respect to quiz scores for the _text_ document with _experts_ scoring (M =
8.58, SD = 0.66) and _novices_ scoring (M = 6.91, SD = 2.02), (t(22) = 2.71, p
= .01). This means that _expert in text_ condition scored significantly higher
than _novices in text_ condition.

###  6.2. Subjective Measures: Usability & Cognitive Load.

For our subjective analysis, we asked participants to take a survey that
included SUS scale questions to assess usability (Brooke, 1996) and NASA-TLX
questions to assess perceived mental demand and effort level (Hart, 2006) (see
Table 3 and 4).

Usability. Using the SUS scores, we found significant differences in usability
scores for _novices_ between the _text_ document (M = 63.95 (below average),
SD = 18.56) and the media _rich_ document (M = 83.33 (excellent), SD = 14.93),
(t(22) = 2.81, p = .01). But we did not find significant differences for
_experts_ across the condition, (t(22) = 1.52, p = .14). This means that
_novices_ in media _rich_ condition reported significantly higher usability
scores than _novices in text_ condition.

But we did not see a significant difference in perceived usability scores
across _experts in media rich_ and _experts in text_ conditions, as they
largely ignored multi-media features and were satisfied with in-line text
augmentation. We also found that reported usability scores were higher for
_experts in text_ condition (M = 75.83 (good), SD = 16.69) than _novices in
text_ (M = 63.95 (below average), SD = 18.56) condition, (t(22)=2.20,p = .03).

Mental demand & workload. We did not find a significant difference in
participants’ perceived mental demand and effort level across the conditions.

Table 3. Subjective Measures Conditions | Usability Score | Mental Demand | Effort Level
---|---|---|---
Novice,rich | M = 83.33, SD = 14.93 | M = 3.5, SD = 1.80 | M = 3.5, SD = 1.83
Novice, text | M = 63.95, SD = 18.56 | M = 4.27, SD = 2.24 | M = 3.91, SD = 2.02
Expert,rich | M = 87.5, SD = 15.73 | M = 3.18, SD = 1.40 | M = 2.83, SD = 1.11
Expert,text | M = 75.83, SD = 16.69 | M = 3.09, SD = 1.57 | M = 2.7, SD = 0.94
Table 4. T-test Subjective Measures Analysis. * denotes significance Compare | Usability Score | Mental Demand | Effort Level
---|---|---|---
Novice,rich & Novice,text | *t(22) = 2.81 , p = .01 | t(22) = -0.83 , p = .41 | t(22) = -0.52 , p = .60
Expert,rich & Expert,text | t(22) = 1.52 , p = .14 | t(22) = 0.11 , p = .91 | t(22) = -0.61 , p = .54
Expert, text & Novice,text | *t(22) = 2.20, p = .03 | (22) = -1.01, p = .32 | t(22) = -1.02, p = .31
Novice, rich & Expert, rich | t(22) = -0.66 , p = .51 | t(22) = 0.11 , p = .90 | t(22) = 1.07 , p = .29

###  6.3. Qualitative Feedback

We gathered qualitative feedback from the eight people who participated in our
moderated study. This included two participants from each of the four assigned
conditions, referred to as _novice, rich_ as NR, _novice, text_ as NT,
_expert, rich_ as ER, and _expert,text_ as ET.

Novices. _Novices_ that interacted with _text_ version (NT1-2) desired richer
media content such as images and videos that explain what the apparatus look
like. NT2 stated _“I would have liked an image or a video [in addition to
text] that explains each step in more detail. […] I think it was a bit
confusing for me to differentiate the apparatuses used in the experiment. But
I did find the tooltip definition helpful and easy to use.”_ _Novices_ that
interacted with _rich_ version found the multi-media content helpful for
understanding document content (NR1-2). NR1 said _“I found the videos helpful
to learn steps about how to conduct the titration experiment. If you noticed,
when I was taking the quiz I was going back to the animated gifs to recall
some of the contents that needed clarification.”_ NR2 stated that _“As I did
not know about this material, it was helpful to see these materials visually.
Especially, I liked the fact that the video and animated gifs were explaining
the materials I am reading exactly. […] While reading, if I get lost, I can go
watch the video or look at animated gifs to proceed. It was good to have those
information [visual representation] upfront so that I did not have to look up
those materials on my own.”_

Experts. _Experts_ that interacted with _text_ version mentioned that they had
all the information they needed and did not desire an additional layer of
media content (ET1-2). They also appreciated in-line text augmentation. ET1
stated _“I am an industrial chemistry major student, so I know how to carry
out a titration experiment, which I did multiple times. […] This document was
enough for me and had all the information I needed.[…] I don’t wish to watch a
video or something”_. ET2 stated, _“I am familiar with titration, but
sometimes I had to recall some of the concepts. And there are different types
of titration experiments, like acid-based titration and others. […] I liked
the tooltip that had definitions that explained what I was looking for
immediately, which was what I needed. And those explanations were directly
related to the context [explanations were customized to the specific type of
experiment]”_. _Experts_ that interacted with the _rich_ document did not seem
to attend to the multi-media content (ER1-2). ER1 stated _“I did think about
watching the video because it was there. But I realized the same information
was in the text and it was just easier to read. I may have looked at the
animated gifs because it was there but I mostly read text and used tooltip
when I get stuck in the document.[…] If I were to watch the video, I would
have to look for a place [video segment] that explains it and that was too
much. Also, if I start watching the video that interrupts the reading, you
know, and I did not want to stop reading”_. It was observed that both
_experts_ who interacted with the media _rich_ version in the moderated
session seemed to skim through the document. They used only a few in-line text
pop-ups to recall concepts and neither _expert_ in the moderated sessions
watched any part of the video.

##  7\. Discussion

We wanted to develop a procedural document that included the right level of
media content for all users. To do that we conducted a formative study to
understand how people consume a text-only procedural document, which
investigated what type of media content they additionally desired. Our
analysis revealed that people’s familiarity with the subject matter impacted
how they read the document and consume multi-media features. This led us to
design a second experiment that specifically targeted _novice_ and _expert_
users. We implemented two versions of a media-augmented document: _text_
document and media _rich_ document. In our second experiment, we evaluated if
these two documents provided the right level of media content for our
identified groups: _novices_ and _experts_. Our results showed that:

  * •

H1:(true) Expert group will prefer the text version and novice group will
prefer media rich version. In general, _experts_ preferred text (augmented) as
it met all of their information needs. They largely ignored the multi-media
content in the media _rich_ version as evidenced by the qualitative results of
the moderated interviews.

  * •

H2: (not true) The expert group will read faster and complete the quiz faster
with text version.

  * •

H3: (mostly true) The novice group will spend more time reading the media rich
version but will complete the quiz faster, score higher on the quiz, and
experience reduced cognitive load.

User preference for multi-media consumption. Our results showed that multi-
media content, in particular video, was not always desired and that it
depended on the user and their familiarity with the document content. We
believe that customizing the amount of multi-media content: images, animated
gifs, and videos included in documents could increase productivity and
decrease distraction. We envision that variable multi-media inclusion could be
achieved in a number of ways. One simple method would be to provide a UI
element that allows users to manually self-select media content based on their
preferences. Another method would be to identify novices and experts
automatically based on their reading history (Nam et al., 2020). Given that
many websites and e-readers collect information regarding user behaviors, we
hypothesized that a user’s familiarity with a given topic could be inferred
from this collected data, then websites could have a plug-in feature to
support the right richness level of multi-media content, such as including
more video or simply augmented text. We imagine that our work can be applied
to alternative contexts and scenarios, such as customizing the level of multi-
media support based on learner expertise in Education (Chiu et al., 2020),
such as chemistry, medicine, vocabulary learning (Balslev et al., 2005; Cocco
et al., 2021; Arndt and Woore, 2018) and for experts and novices visitors in
tourism at cultural heritage (Candello, 2013).

Dynamic multi-media augmentation. We also believe that a user’s familiarity
with the content can be dynamically assessed as the reader is consuming the
document based on reading speed or other behaviors by using predictive
statistical models (Zukerman and Albrecht, 2001; Frias-Martinez et al., 2006).
In our study, we presented two fixed content documents either with or without
multi-media content, however, this content could be included dynamically based
on real-time user behaviors. For example, when reading speed slows the
document could transform text to images (Zhu et al., 2007), text to videos to
better explain content (Zhu et al., 2007), or algorithms could even transform
the entire document into an audio-visual slide show (Leake et al., 2020).
Changing the amount of multi-media content dynamically supports automatic
customization making documents resilient to changing user preferences, such as
being able to always add content (expansion of information).

##  8\. Conclusion

We had the high-level goal of developing digital documents with the right
level of media augmentation for all users. To this end, we conducted an
observational study with real users reading a procedural document. Our study
revealed that depending on people’s familiarity with the content, people
deferentially consumed multi-media features in digital documents. To confirm
our hypothesis that the desire for multi-media content was dependent on a
people’s familiarity with the material, we next conducted a second user study
with 48 different participants where we specifically recruited _novices_ and
_experts_ in chemistry and had them read either a _text_ focused or media
_rich_ presentation of a chemistry titration lab document. Our findings showed
that experts preferred text and novices benefited from multi-media content.
That said, customizing the richness of multimedia in documents for different
types of users could increase productivity and decrease distraction.

## References

  * (1)
  * kin (2020) 2020\.  _Look Up Words, People, and Places While You Read._ https://www.amazon.com/b?ie=UTF8&node=17717476011
  * use (2023) 2023\.  _User Testing._ https://www.usertesting.com/platform?utm_source=mkto-fallback
  * Aagaard (2019) Jesper Aagaard. 2019\.  Multitasking as distraction: A conceptual analysis of media multitasking research.  _Theory & Psychology_ 29, 1 (2019), 87–99.
  * Allahyari et al. (2017) Mehdi Allahyari, Seyedamin Pouriyeh, Mehdi Assefi, Saeid Safaei, Elizabeth D Trippe, Juan B Gutierrez, and Krys Kochut. 2017.  Text summarization techniques: a brief survey.  _arXiv preprint arXiv:1707.02268_ (2017).
  * Anima (2022) Anima 2022.  Anima for Adobe XD.  https://docs.animaapp.com/v3/adobe-xd/getting-started/.
  * Arndt and Woore (2018) Henriette Arndt and Robert Woore. 2018.  Vocabulary learning from watching YouTube videos and reading blog posts.  _Language Learning and Technology_ 22, 3 (2018).
  * Balslev et al. (2005) Thomas Balslev, Willem S De Grave, Arno MM Muijtjens, and AJJA Scherpbier. 2005.  Comparison of text and video cases in a postgraduate problem-based learning format.  _Medical education_ 39, 11 (2005), 1086–1092.
  * Brooke (1996) John Brooke. 1996\.  _”SUS-A quick and dirty usability scale.” Usability evaluation in industry_.  CRC Press.  https://www.crcpress.com/product/isbn/9780748404605 ISBN: 9780748404605.
  * Byers (2022) Kyle Byers. 2022\.  _151 Content Marketing Statistics for 2022_.  https://growthbadger.com/content-marketing-statistics/
  * Candello (2013) Heloisa Candello. 2013\.  Multimedia information delivery on mobile cultural applications. In _International Conference on Universal Access in Human-Computer Interaction_. Springer, 583–592.
  * Castles et al. (2018) Anne Castles, Kathleen Rastle, and Kate Nation. 2018\.  Ending the reading wars: Reading acquisition from novice to expert.  _Psychological Science in the Public Interest_ 19, 1 (2018), 5–51.
  * Chang et al. (1998) Bay-Wei Chang, Jock D Mackinlay, Polle T Zellweger, and Takeo Igarashi. 1998.  A negotiation architecture for fluid documents. In _Proceedings of the 11th annual ACM symposium on User interface software and technology_. 123–132.
  * Chilana et al. (2012) Parmit K Chilana, Amy J Ko, and Jacob O Wobbrock. 2012\.  LemonAid: selection-based crowdsourced contextual help for web applications. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. 1549–1558.
  * Chiu et al. (2020) Thomas KF Chiu, Morris Siu-yung Jong, and Ida AC Mok. 2020\.  Does learner expertise matter when designing emotional multimedia for learners of primary school mathematics?  _Educational Technology Research and Development_ 68, 5 (2020), 2305–2320.
  * Chiu and Mok (2017) Thomas KF Chiu and Ida AC Mok. 2017.  Learner expertise and mathematics different order thinking skills in multimedia learning.  _Computers & Education_ 107 (2017), 147–164.
  * Cocco et al. (2021) Veronica Margherita Cocco, Elisa Bisagno, Gian Antonio Di Bernardo, Alessia Cadamuro, Sara Debora Riboldi, Eleonora Crapolicchio, Elena Trifiletti, Sofia Stathi, and Loris Vezzali. 2021\.  Comparing story reading and video watching as two distinct forms of vicarious contact: An experimental intervention among elementary school children.  _British Journal of Social Psychology_ 60, 1 (2021), 74–94.
  * Cockburn et al. (2014) Andy Cockburn, Carl Gutwin, Joey Scarr, and Sylvain Malacria. 2014.  Supporting novice to expert transitions in user interfaces.  _ACM Computing Surveys (CSUR)_ 47, 2 (2014), 1–36.
  * Collins and Evans (2002) Harry M Collins and Robert Evans. 2002.  The third wave of science studies: Studies of expertise and experience.  _Social studies of science_ 32, 2 (2002), 235–296.
  * Conklin (1987) Jeff Conklin. 1987\.  Hypertext: An introduction and survey.  _computer_ 20, 09 (1987), 17–41.
  * Council et al. (2000) National Research Council et al. 2000\.  _How people learn: Brain, mind, experience, and school: Expanded edition_.  National Academies Press.
  * Egan et al. (1989) Dennis E Egan, Joel R Remde, Louis M Gomez, Thomas K Landauer, Jennifer Eberhardt, and Carol C Lochbaum. 1989.  Formative design evaluation of superbook.  _ACM Transactions on information systems (TOIS)_ 7, 1 (1989), 30–57.
  * Frias-Martinez et al. (2006) Enrique Frias-Martinez, Sherry Y Chen, and Xiaohui Liu. 2006\.  Survey of data mining approaches to user modeling for adaptive hypermedia.  _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ 36, 6 (2006), 734–749.
  * Graham (1999) Jamey Graham. 1999\.  The reader’s helper: a personalized document reading environment. In _Proceedings of the SIGCHI conference on human factors in computing systems_. 481–488.
  * Hart (2006) Sandra G Hart. 2006\.  NASA-task load index (NASA-TLX); 20 years later. In _Proceedings of the human factors and ergonomics society annual meeting_ , Vol. 50. Sage publications Sage CA: Los Angeles, CA, 904–908.
  * Head et al. (2021) Andrew Head, Kyle Lo, Dongyeop Kang, Raymond Fok, Sam Skjonsberg, Daniel S Weld, and Marti A Hearst. 2021.  Augmenting scientific papers with just-in-time, position-sensitive definitions of terms and symbols. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–18.
  * Hill et al. (1992) William C Hill, James D Hollan, Dave Wroblewski, and Tim McCandless. 1992.  Edit wear and read wear. In _Proceedings of the SIGCHI conference on Human factors in computing systems_. 3–9.
  * Hölscher and Strube (2000) Christoph Hölscher and Gerhard Strube. 2000.  Web search behavior of Internet experts and newbies.  _Computer networks_ 33, 1-6 (2000), 337–346.
  * Kalyuga (2013) Slava Kalyuga. 2013\.  Effects of learner prior knowledge and working memory limitations on multimedia learning.  _Procedia-Social and Behavioral Sciences_ 83 (2013), 25–29.
  * Kalyuga (2014) Slava Kalyuga. 2014\.  The expertise reversal principle in multimedia learning.  (2014).
  * Lazonder et al. (2000) Ard W Lazonder, Harm JA Biemans, and Iwan GJH Wopereis. 2000\.  Differences between novice and experienced users in searching information on the World Wide Web.  _Journal of the American society for Information science_ 51, 6 (2000), 576–581.
  * Leake et al. (2020) Mackenzie Leake, Hijung Valentina Shin, Joy O Kim, and Maneesh Agrawala. 2020.  Generating audio-visual slideshows from text articles using word concreteness. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–11.
  * McNamara and Kintsch (1996) Danielle S McNamara and Walter Kintsch. 1996.  Learning from texts: Effects of prior knowledge and text coherence.  _Discourse processes_ 22, 3 (1996), 247–288.
  * Nam et al. (2020) Sungjin Nam, Zoya Bylinskii, Christopher Tensmeyer, Curtis Wigington, Rajiv Jain, and Tong Sun. 2020\.  Using Behavioral Interactions from a Mobile Device to Classify the Reader’s Prior Familiarity and Goal Conditions.  _arXiv preprint arXiv:2004.12016_ (2020).
  * Norman (2013) Don Norman. 2013\.  _The design of everyday things: Revised and expanded edition_.  Basic books.
  * O’Brien et al. (2016) Heather L O’Brien, Luanne Freund, and Richard Kopak. 2016\.  Investigating the role of user engagement in digital reading environments. In _Proceedings of the 2016 ACM on conference on human information interaction and retrieval_. 71–80.
  * Ozuru et al. (2009) Yasuhiro Ozuru, Kyle Dempsey, and Danielle S McNamara. 2009\.  Prior knowledge, reading skill, and text cohesion in the comprehension of science texts.  _Learning and instruction_ 19, 3 (2009), 228–242.
  * Patel (2018) Charmi Patel. 2018\.  _An Analysis of Jean Lave and Etienne Wenger’s: Situated Learning: Legitimate Peripheral Participation_.  Macat Library.
  * Rachatasumrit et al. (2022) Napol Rachatasumrit, Jonathan Bragg, Amy X Zhang, and Daniel S Weld. 2022.  CiteRead: Integrating Localized Citation Contexts into Scientific Paper Reading. In _27th International Conference on Intelligent User Interfaces_. 707–719.
  * Schilit et al. (1998) Bill N Schilit, Gene Golovchinsky, and Morgan N Price. 1998\.  Beyond paper: supporting active reading with free form digital ink annotations. In _Proceedings of the SIGCHI conference on Human factors in computing systems_. 249–256.
  * Southern (2019) Matt G. Southern. 2019\.  _Twitter Reveals Tweets With Video Generate 10X More Engagement_.  https://www.searchenginejournal.com/twitter-reveals-tweets-with-video-generate-10x-more-engagement/322523/#close
  * Takacs et al. (2015) Zsofia K Takacs, Elise K Swart, and Adriana G Bus. 2015\.  Benefits and pitfalls of multimedia and interactive features in technology-enhanced storybooks: A meta-analysis.  _Review of educational research_ 85, 4 (2015), 698–739.
  * Tversky et al. (2002) Barbara Tversky, Julie Bauer Morrison, and Mireille Betrancourt. 2002.  Animation: can it facilitate?  _International journal of human-computer studies_ 57, 4 (2002), 247–262.
  * Zellweger et al. (1998) Polle T Zellweger, Bay-Wei Chang, and Jock D Mackinlay. 1998\.  Fluid links for informed and incremental link transitions. In _Proceedings of the ninth ACM conference on Hypertext and hypermedia: links, objects, time and space—structure in hypermedia systems: links, objects, time and space—structure in hypermedia systems_. 50–57.
  * Zhu et al. (2007) Xiaojin Zhu, Andrew B Goldberg, Mohamed Eldawy, Charles R Dyer, and Bradley Strock. 2007\.  A text-to-picture synthesis system for augmenting communication. In _AAAI_ , Vol. 7. 1590–1595.
  * Zukerman and Albrecht (2001) Ingrid Zukerman and David W Albrecht. 2001.  Predictive statistical models for user modeling.  _User Modeling and User-Adapted Interaction_ 11, 1 (2001), 5–18.