scale[1]

# Enabling Learning by Teaching: Intuitive Composing of E-Learning Modules

Alexander Berntsen Stian Ellingsen Emil Henry Flakk<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In an effort to foster learning by teaching, we propose the development of a
canvas system that makes composing e-learning modules intuitive. We try to
empower and liberate non-technical module users by lowering the bar for
turning them into module authors, a bar previously set far too high. In turn,
this stimulates learning through teaching. By making a damn fine piece of
software, we furthermore make module authoring more pleasant for experienced
authors as well. We propose a system that initially enables users to easily
compose H5P modules. These modules are successively easy to share and modify.
Through gamification we encourage authors to share their work, and to improve
the works of others.

###### Contents

  1. 1 Introduction
  2. 2 The problem
  3. 3 Our idea
  4. 4 The details
  5. 5 First principles
  6. 6 Related work
  7. 7 Conclusions and further work
  8. A Design document

###### List of Figures

  1. 1 A system where the user watches a YouTube video, then reads an article on NRK, and finally does a quiz
  2. 2 A dialogue for tweaking conditional flow
  3. 3 A reward for positive behaviour
  4. 4 A user dragging a module onto the canvas to add it
  5. 5 Visualised control flow
  6. 6 Search results, filtered by type
  7. 7 An avatar helping the user

## 1\. Introduction

The ”Bottom Of the Pyramid” (BOP) is a term central to the development of
value-chains in emerging markets in developing countries. It is understood as
the largest, although poorest segment of the population. The key to economic
development lies in activating this segment of the population, enabling them
not only to use modern (digital) services, but to author them as well[1].

The inspiration for our idea comes from applying this general framework to the
field of education software. The economics of educational services are
unfavourable to those who need it the most.

We improve the status quo by offering a novel system for composing e-learning
modules that is easy to use by non-technical users, but expressive enough to
benefit power users as well.

This improvement is a stepping stone towards turning e-learning authoring into
an activity achievable by even school children. It is also a stepping stone
towards stimulating learning by teaching, a stimuli school children find
themselves largely deprived of in most situated learning environments, i.e.
schools.

## 2\. The problem

Alas, most software is written for a small subset of the population.
H5P111http://h5p.org/ is one of many novel examples of software providing
added-value to tech-savvy teachers and educational software developers.
However, the BOP would in this case be the students.

By providing the correct tooling and educational framework, we could enable
students to act not only as users of off-the-shelf educational software, but
also as authors of their own or their co-students’ learning experience. By
doing so, we could allow the student to feel some degree of pride in the
coursework. Moreover, it would allow for reinforced learning, as well as viral
effects (sharing), a major force in the modern network economy.

Such a framework would also encourage the students to make services and tools
more adapted to their particular learning situation, improving the overall
user experience. It would additionally be a valuable source of inspiration for
educational software developers and user experience designers. And if
successful, usage data would be interesting for analytics research.

Sadly, such a framework doesn’t exist. So we want to move H5P closer to being
one. In order to achieve this we provide intuitive composition of H5P modules,
and, ultimately, of all kinds of e-learning modules, including new ones.

## 3\. Our idea

We suggest developing an engaging “canvas” for authoring e-learning modules,
taking cues from RPGツクール222http://www.rpgmakerweb.com/, Game
Maker333https://www.yoyogames.com/studio, and other similar software that
succeed in take advantage of an intuitive interface with a capacious
featureset.

By “canvas” (or “scene graph”), we mean an easy-to-learn system of widgets and
composite stores (with rich content), and flows connecting them together to a
cohesive system. The widgets would initially be H5P modules.

An example of such a system might be a test leveraging media like GNU
Mediagoblin, Youtube, NRK, Twitter, and Nasjonal Digital
Læringsarena444http://ndla.no/ to showcase the issue at hand by embedding
video and articles, etc., before testing the user’s comprehension at the end
with an H5P quiz. Such a system is shown in Figure 1. Thus, the student-come-
software-developer is able to prove their comprehension of multimedia (as
mandated by the curriculum), and other students are permitted to take part in
the learning experience. Incentives for doing so might be given by the
teacher, or by arranging collaborations that invite the students to share
their best ideas.

0.1

Figure 1: A system where the user watches a YouTube video, then reads an
article on NRK, and finally does a quiz

Taking cue from Minecraft, we also propose another viable incentive: The
student is provided with ready-made canvases with actors and stores made by
other users, and encouraged to make the application perform certain actions
(“gamification”). This is akin to the redstone system found in Minecraft,
where even young users are able to construct discrete logic circuits providing
useful functionality like opening doors, or switching on lights[2].

Consequently, we have customisable flows between widgets. They may be
customised initially via simple if-expressions on the form $if$ $\chi$ $then$
$\alpha$, $else$ $\beta$, where e.g. $\chi=$ 80% completion rate, and
$\alpha=$ an H5P memory game, and $\beta=$ an H5P quiz. An example of the
conditions dialogue is shown in Figure 2.

Through use of our canvas, we eliminate the BOP by liberating and empowering
it to make its own user experience. As an added bonus, our canvas may spark
some latent creative souls, or inspire technological awareness and interest.
In our increasingly computerised society, this is in itself a noble cause.

Thus, this canvas might prove to be a force for bridging the gap between just
being a computer user, and having a promising future career in computer
software. While Computing At School555http://www.computingatschool.org.uk/
have had great success in the UK, there is as of today no readily-available
path for acquiring the advanced knowledge needed to develop modern systems
given the current education system in Norway. Grassroots organisations such as
Lær Kidsa Koding666http://www.kidsakoder.no/ are doing good work, but have yet
to strongly influence the education system. If our canvas is picked up by
prominent e-learning providers like Nasjonal digital læringsarena we can
liberate and empower users through direct action, circumventing bureaucracy.

In addition to users teaching themselves technology, they also teach
themselves the curriculum more effectively. The student becomes the teacher,
and we achieve learning by teaching, an often sought-after method of
reinforced learning. By only being e-learning module _users_ , students are
limited to learning through observation, experimentation, and (to some extent)
mistakes. Learning through teaching offers advantages not possible to fully
realise through either of these; advantages that won’t manifest if the student
is exclusively relying on an external teacher[3].

0.1

Figure 2: A dialogue for tweaking conditional flow

Our canvas makes composing e-learning modules intuitive for non-technical
users and at the same time powerful enough to entice power users and
established e-learning module authors. Consequently, the target demographic of
our canvas is not limited to the BOP, but extends to include current
e-learning module authors.

To further underline our empowering of the BOP, we make sharing of works very
simple, and make it equally simple to author derivatives (forks) and meta-
works (collections). Incentive for sharing your work, and improving or
remixing the work of others is provided through gamification of the canvas. As
an example, there may be a reward system for users whose modules are often
remixed, and for users who make an improvement to a module that then gets
integrated back into the original module itself. These rewards are then
presented in a positive and motivating way to the user, as shown in Figure 3.

0.1

Figure 3: A reward for positive behaviour

With gamification, we can also help ensure high quality modules. With a rating
system and achievements for obtaining a high rating, we realise a self-
regulating community.

Initially we aim to support authoring H5P modules, focussing our development
at H5P integration. But with a good modular design we can extend our canvas to
support other standards as well in the future. We also want to support
embedding of things like videos and news articles. Lastly, careful attention
must be paid to the application programming interface, so that it is easy for
future e-learning modules to support our system.

## 4\. The details

### 4.1. Concept

We propose a canvas system wherein the user may drag e-learning modules from a
toolbox, and drop them onto the canvas, as in Figure 4. The modules may be
empty or finished. If they are empty, then the authoring tool for said module
opens. If it is not empty, then the user may further edit it if they wish. In
addition to drag-and-drop, the user may also use an “add” button, as drag-and-
drop is not viable for universal design.

0.1

Figure 4: A user dragging a module onto the canvas to add it

To schedule data and control flow, the user drags arrows between the modules
on the canvas. These associations may be customised. “If the user finished
[module a] with a score of more than 80% correct, then direct them to [module
b]; if the user did not, then direct them to [module c]”. A simple
illustration of this is in Figure 5.

0.1

Figure 5: Visualised control flow

Compositions are modules themselves.

When a user wants to compose modules, users may search for which modules to
use. We want to encourage reuse and remixing rather than novelties.
Consequently we focus the search option before the “author new module” option.
The search option is shown in Figure 6, which also shows results filtering in
action.

0.1

Figure 6: Search results, filtered by type

As another incentive, when a user authors a new module, we track reuses and
remixes. Users may in a presentation of all their authored modules see how
many reuses and remixes have occurred, and follow links to these. If they like
a remix they may merge the changes between the remix and the original.

Compositions may be derived in whole, and further developed by another author.
Users may merge modifications to these as well. Consider an English Language
test for a school curriculum. A teacher at a different school (or the same
school next year) may want to reuse this test since they have the same
curriculum. But they might furthermore want to modify it. If the original
author finds these modifications useful, then it may merge them to the
original.

Avatars have been successfully used before to motivate children[4]. We propose
the use of an avatar for explaining the interface and pointing out problems
with the compositions, such as integrity issues, e.g. “the composition never
ends”, or continuity warnings, e.g. “this module is visited two times”,. The
avatar’s appearance may be customised. We show an example of an avatar
dialogue in Figure 7, although the avatar itself is merely a placeholder.

For our actual avatars, we have chosen otters. plaimi’s mascot is an otter, so
this decision is done partly for cohesive branding concerns, but primarily for
the same reasons it was chosen as plaimi’s mascot in the first place: the
otter is a noted playful and social creature[5]. It is furthermore considered
cute by the general population, but capable of being ferocious[6], which is
appealing for the target demographic as girls tend to prefer cute avatars,
whilst boys prefer rude avatars[7].

To encourage certain behaviour, the avatar may reward positive behaviour with
a badge, e.g. “200 remixes!”, or “remixed 200 modules!”.

0.1

Figure 7: An avatar helping the user

Punishing bad behaviour rarely works, and punishment is easily evaded. Instead
we simply prohibit bad behaviour. Chatting is done via the avatars. To
communicate with another user, you visit their profile and click on their
avatar to bring up a dial menu for constructing a query like “you should add
[module] to your composition!”, or “I like this module!”. This makes
localisation much easier since these set sentences may be translated
accurately. It also makes our system safe for children[8]. This is a recent
trend in computer games such as Journey and Hearthstone for the same reasons.
This eliminates hate speech[9], which is not acceptable with a child audience.

Users may, via their avatars as just described, encourage other users by
telling them that they like their modules. There is no way to dislike a
module, as that might be demotivating for children. Avatars as well as modules
may be favoured. The user’s favourite modules and avatars are more easily
accessible than the rest.

Users sign up with a log-on ID and password, and optionally an email address
for recovering forgotten passwords. The users name their avatars which is then
their screen names. The avatar has a name. There may be duplicate names.
People have duplicate names in real life too [citation needed], and society
hasn’t collapsed yet. When choosing a name, feedback is given if the name is
not unique.

### 4.2. Requirements

We place high usability requirements on our canvas system in order to enable
school children to use it.

Our target demographic at its largest is pretty much everyone that wants to
learn or teach something. Therefore universal design is of the utmost
importance. It is furthermore legally required in Norway[10]. More importantly
it is The Right Thing to do. Universal design is difficult with such a
graphical system, but of the utmost importance. Compromises shall err on the
side of caution and favour universal design to expressiveness.

Dealing with children has interface design implications. Tap actions should
allow some slack with bigger hit boxes than actual buttons and similar
measurements. Splash screens are bad, mkay. Turns out kids don’t have that
great an attention-span. So get to the point quick as. These things are
discussed in more detail in Section 5.

In order to encourage reuse and remixing, modules and assets are distributed
under a free licence, CC-BY-SA. This also sets a positive example for school
children to become good citizens of our society. Indeed so does the entire
project by being licensed as AGPL[11].

### 4.3. Functional Details

####  4.3.1 Frontend

For our canvas system and frontend in general we primarily use the Elm
programming language[12].

Elm is an immature language with several known shortcomings[13], but the more
popular alternative JavaScript is a mature language with even more known
shortcomings[14]. Although early research suggests that using Elm will not
magically result in a better product[15], it does allow for purely functional
programming of denotational graphical interfaces[12], which has several known
advantages for interactive graphical programs[16]. Elm itself has several
specific advantages as well — and excellent tooling to boot[17].

Being an immature language, not all libraries we could possibly wish for are
readily available. Therefore accessing JavaScript directly is still useful at
times. Thankfully, Elm comes with a sophisticated JavaScript foreign
functional interface[18]. And if it becomes necessary to use JavaScript
libraries to interact with JavaScript primitives presently outside of Elm’s
reach, we may fall-back on JavaScript for this.

####  4.3.2 Backend

For dealing with all things backend we use the Haskell programming
language[19]. Haskell is a mature and widely used programming language with
advantages similar to that of Elm, except more advantages and more powerful
advantages. Interesting advantages over Elm include laziness for
modularisation[20], typeclasses for elegant ad-hoc polymorphism[21], and an in
general more powerful and interesting type system[22]. Lastly, Haskell
seriously equips you for all cost-models of parallelism and concurrency[23].

We write property laws and test these using QuickCheck[24], a well-established
tool used in industry[25].

Haskell, being a mature language, provides us with several useful libraries to
ensure a shorter time to market.

####  4.3.3 Reuse and remix

We want to encourage reusing and remixing. We define reuse to be using someone
else’s module as-is, and remix to be using someone else’s module with
modifications.

Care must be taken not to merely give incentives for having your modules
reused and remixed by others, as this fosters a culture where being the author
of novelties is the most rewarding. Rewards should therefore be doled out more
handsomely for actually reusing and remixing than for authoring modules that
are successfully subsequently reused and remixed. This also makes sense since
the latter is accumulatively (having your module be reused and remixed is
tracked and awarded recursively) rewarding passive behaviour, whilst the
former rewards active behaviour.

### 4.4. Design document

Given our emphasis on usability, we have had a design document authored by a
user experience developer. This document includes a graphical profile, some
specific user-interface design, and some overarching design principles. It may
be found in its entirety in Appendix A.

## 5\. First principles

We wish to discuss learning itself, and how this knowledge affects our system.
A common technique used in most of these systems is spaced repetition combined
with testing, i.e. the pacing of repetition throughout time. This approach is
well-proven by research, and could be implemented both as a part of our
services (allowing module authors to apply it to their own students), or as a
part of the experience of learning the tools. An example might be: ‘After 5
days, please repeat this factoid and test the student.‘ Of course, such an
approach might prove cumbersome, and techniques like machine learning might
automate it in some meaningful fashion using a data-oriented approached.
Reminders are most effective when occurring just before the memory fades.
Successive reminders will consequently be separated at longer and longer
intervals[26].

In testing, there exists mainly three techniques for assessment:

  * •

Reading, which is the reproduction of textual representations of the fact.

  * •

Multiple choice, where one is allowed multiple alternatives, where at least
one of them should be true. Often one or more of the alternatives are ‘the odd
one out’, to test the student’s ability to discriminate irrelevant concepts.

  * •

Generation, where the user is made to produce something, for example filling
in a word. This technique is highly efficient with regards to retention,
especially when combined with quick feedback[27].

Thus, we consider generation. An example of this might be generating a
formally valid blueprint of some e-module, and asking the user what it does.
Or it might be to make the user connect the relevant parts to make it work as
desired. Thus, we foster true comprehension of the system mechanics and avoid
cramming. As noted in the literature in the Memrise review, errorful
generation might actually be beneficial, and the user should not be punished
for making mistakes. Rather, we suggest to embrace it and make failure as
smooth as possible, with quick iterations.

A big part of any learning system is the metaphor, or the user interface. The
system needs to provide some incentive for the user to perform some action.
Thus, the user interface will always be a question of demographics. We suggest
that the user interface should be amenable to at least some simplifications to
scale down to the technology-native elementary school demographic. Some
concrete suggestions are:

  * •

The user interface needs to take into consideration the rather poor motor
skills of young children[28]. Thus, tap actions should not be that precise or
responsive, but instead allow for some slack. For example, multiple taps
should not result in multiple windows, but rather provide a time buffer for
the action to launch. Also, the hit box of buttons should be greater than
their actual size.

  * •

Avoid splash screens[29], and in general get to the point as quickly as
possible. Thus, it might be better to provide a simpler, but usable interface
rather than a complete experience first-hand. E.g. one may wish to avoid modal
dialogues, as they provide children which textual choice, which they might not
understand or be interested in.

  * •

To provide some sense of continuity, and also a way of accumulating rewards,
the user constructs an avatar. This concept is known from video games, and has
already been suggested to be a viable way of familiarising children with
complicated services like search engines[4].

Unfortunately, we were not able to locate high-quality sources of research in
this area. Most sources are based on de-facto and ad-hoc solutions, with
little or no scientific basis outside of in-house usability studies. Thus, we
are potentially missing out on a great deal of useful data. On the other hand,
this means we have the unique opportunity to not only be innovators, but also
inspiring flagships.

Takeaway points, based on previous discussion and the reviews in Section 6.

  * •

Spaced learning with testing using generation fosters comprehension and
recall. We need to embrace generating errors as a way of learning.

  * •

Quick and simple feedback with additive gains facilitate user engagement and
learning greatly, and promotes attention continuity.

  * •

The system should provide a simple (preferably physical) metaphor as part of
its user story.

  * •

The system needs to use open, free standards to enhance composition and foster
a culture of reuse and experimentation.

  * •

We should strive for feature parity between different clients, to simplify the
user story and provide a consistent experience.

  * •

A strong, data-driven approach involving the users strengthens community
relations and improved quality by providing relevant feedback to create,
enhance, or QA existing material.

So the big question then becomes what do we do with all these takeaway points.
In order to provide a cohesive experience for authors as well as users we need
to take all of this into account. But cohesion traditionally proves difficult
when coupled with agnosticism — i.e. we would like to compose many — different
— kinds of modules, and different kinds of modules likely have different
approaches to authoring.

What we are eventually getting at is the need for high quality control in what
modules we allow, and likely a fair bit of customisation of the ones we do end
up allowing. Alternatively new module authoring software may be developed
altogether.

## 6\. Related work

H5P has rudimentary editing support. It is unfortunately low-level. But at the
same time it appears to be easy enough to integrate, which is ideal for our
system. At any rate, H5P does not offer high-level composing of modules, which
is our real novelty.

There does not appear to be any mature, publicly available e-learning module
authors that achieves the sophistication we would enjoy seeing, capable of
making genuinely diverse modules, from flashcards to memory games, from
quizzes to mini-games. This is another misfortune.

Intuitive composing seems virtually unheard of. And a standalone module-
agnostic composing tool appears to be a complete novelty.

However, there exist several authoring tools for the specific things we wish
to accomplish. I.e. there are flashcard authoring tools, quiz authoring tools,
etc. In this section we discuss those, as well as authoring tools for
completely different things, and unconventional ”authoring tools” like
Minecraft.

### 6.1. Evaluation of Anki

Anki is a system for spaced learning. It works by organising facts (in a wide
sense, like foreign words or parts of a map) into individual cards, which in
part constitute a deck of facts about a given topic. The user plays through
the deck by going through each card and stating how easy it was to recall the
fact. The software then replays cards depending on how good or bad the recall
was until some threshold of recollection has been met. Bad recall would result
in the card appearing more often, and vice versa. This algorithm is inspired
by the SuperMemo system written in the 1980s[30]. The intuition is that we
reinforce learning by recalling something just as we are about to forget about
it.

Users are allowed to share, amend, or extend existing decks. While Anki
primarily exists for the purpose of individual learning, the composable and
free software format allows for others to take part in the learning
experience. The software has enjoyed great popularity amongst several groups,
most notably learners of foreign languages.

Anki is available both as a desktop client and through a web interface —
AnkiWeb777https://ankiweb.net/. While the latter easily allows for online
sharing of decks, it doesn’t allow for importing them. This is because it only
synchronises with the desktop client. This highlights the challenges faced by
desktop and web clients with different capabilities, and the value of
isomorphic implementations.

AnkiWeb allows for synchronization and sharing of card decks (as files) using
a free online service. It is also able to play through the decks, but not
import them from stand-alone files.

The desktop client of Anki will therefore be the main focus of this
evaluation, as it is the only full implementation of the system. Similar
software in the same area are SuperMemo and Fluxcards.

In terms of licensing, Anki is well-aligned with our project. It is freely
licensed under the terms of the GNU AGPL v3+. Sub-components are licensed
under a variety of licenses, mostly GNU GPL and flavours of BSD/MIT licences.
All art assets are also freely licensed.

####  6.1.1 Expressiveness

Anki allows for the authoring of (decks of) flashcards using standard HTML
components with CSS styling. As a result, users are able to enrich cards with
customised formatting, sound, and video. The users are also able to record
their own voice (and repeat sounds) during playthrough. This offers an
interactive experience to the end-user. Since HTML and CSS are relatively
flexible and well-documented technologies, and is used by many other prominent
parties, this is likely a good choice by the Anki developers, that gives the
users a good balance between expressiveness and ease of use, as well as
familiarity.

####  6.1.2 Ease of use

Because the technologies used by Anki are freely available and readily
documented, they constitute an open standard. This open container format makes
Anki composable, and since the technologies are rather common (at least HTML),
users can make relatively simple changes without any major obstacles.

However, Anki is considered to have a poor user interface interaction story
for common scenarios[31].

An example of a problematic story is the rather common case of wanting to
share a deck. This usually has to be done through AnkiWeb, a free online
service, which allows the user to download new decks made by other users.
However, no such suggestion is made to the user. Thus, the social dimension of
Anki is downplayed.

Similarly, when using the software for the first time, the user is greeted by
a shallow window with a single deck, and no explanation of Anki or its
algorithm is given. The user can click a simple built-in test deck to play
through it, but this is not suggested by the software, and the deck does not
showcase Anki’s capabilities either (for example the use of rich media or
LaTeX equations).

Other examples include the main window, which has both a menu bar and some
buttons. One of the buttons suggests that the user might be able to import new
decks, but turns out to be about constructing new decks.

The most complex interface in Anki is the library view. It allows for somewhat
non-trivial querying of all the decks in use, with numerous built-in
categories and tags for distinguishing them (be it difficulty, topic, or the
decks scheduled for today). Also, this rich view is not offered in AnkiWeb, so
there’s an inherent divide in the capabilities of searching already imported
decks and looking for new ones, limiting the ability to explore.

Universal design isn’t guaranteed since the decks are made by users. There
doesn’t appear to be any way of filtering decks based on accessibility, nor
any user-interface encouragement to _do_ universal design when authoring
decks.

####  6.1.3 Takeaway points

  * •

Anki offers spaced repetition learning, which is an established and useful way
of learning.

  * •

The simple metaphor (a deck of flashcards) is intuitive and easy to explain to
users[32], and simple enough to implement.

  * •

Anki’s use of HTML elements provides a universal container format and
facilitates sharing. However, the web interface is very limited, and while it
does allow for sharing, it does not allow for importing decks, thus not
providing a full (isomorphic) user experience across platforms.

  * •

The desktop client has a problematic user interface story, and does not
provide enough directions or examples for effective use. Modal dialogues and
context switching might provide a more elegant solution for capturing common
usage patterns (compared to displaying the full set of choices and categories
available to the user, e.g. in the library view).

### 6.2. Evaluation of Memrise

Memrise888http://www.memrise.com is a flashcard-based learning system. The
cards make use of (often crowd sourced) mnemonics combined with spaced
repetition (akin to Duolingo and Anki) to augment the learning experience. It
is unique in part due to its openness regarding its techniques and its close
ties with academic researchers, who are allowed to analysis its datasets.

Memrise itself suggests three basic techniques:[26]

  * •

Elaborate encoding. I.e., the construction of cognitive structures of already
known material that new knowledge might ”attach” itself to. E.g., taxonomies
(like the colours red, white, blue) are easier to remember than a random list
(astronaut, velvet, cigar). To this end, Memrise uses crowd sourcing (with
machine-learning techniques) to suggest ”mems”, which are the aforementioned
cognitive structures to boost learning.

  * •

Choreographed testing, which test recall and comprehension. Memrise varies
between simple one-off challenges (”casa = ?”) and multiple-choice questions
to keep things interesting.

  * •

Scheduled reminders, which is their implementation of spaced learning.

Research around — or by — Memrise mostly revolves around the benefits of
testing, error generation (vis-à-vis inerrant learning), and how spacing
affects the efficacy of the learning programme.

Testing is the act of challenging retention while studying, which is
beneficial as part of the learning process. There is no consensus as to why it
works, but some explanations include that it increases the storage strength of
memory, and that it generates additional cues that creates (potentially more
efficient) routes through memory[27, p.6].

Generation (with feedback) is an active form of learning, and requires the
student to complete something, for example a sentence (”Lisa is a ___”),
before getting to know the answer. This is contrasted to reading, where the
student absorbs textual material and definitions, and multiple choice, which
amongst other things tests the ability of the student to see how the different
alternatives relate to each other (finding the odd one out etc.).

Errant learning has been thought to be detrimental to learning as people
recall their errors to a greater extent than their correct answers[27], but
this research has mostly been done with already memory-impaired populations,
which might have different requirements. Inerrant learning does not seem to be
as effective as error generation: “…” generating responses followed by
feedback is helpful to memory even when many errors are generated, compared
with inerrant studying without generation[27, p.54].

Generating errors may benefit vocabulary learning even when there are no
established associations to the items in question[27, p.54]. While completing
sentences appear to have the biggest advantage over only reading, other types
of generation also improves test scores when compared to reading by itself[33,
p.73].

Several experiments have been designed to test assumptions regarding these
models. They are mostly language-orientated (a good fit for a programming
language!), and might use techniques like archaic words (which are certain to
be unknown for most test subjects) to provide a ”clean slate” for testing
comprehension and retention of definitions. Research shows that
participations’ perception of learning is influenced by error generation, and
that participants seemingly perceive generate items as more difficult to
learn, which may lead to participants putting in more effort to learn the more
difficult items, which in turn may lead to higher retention[27].

####  6.2.1 Expressiveness

More or less any knowledge that needs to be internalised can fit within the
style of testing offered by Memrise. Users are allowed to create their own
courses, which may contain multimedia levels. For example, users may embed
videos from Vine999https://vine.co/ inside “mems” to boost retention.

####  6.2.2 Ease of use

Memrise offers an easy to use interface with a few, mostly obvious buttons.
Flow doesn’t seem to be problematic. Interactive modals guide the user through
things like creating “mems” for their own courses and so on.

####  6.2.3 Takeaway points

  * •

Generating (that is, having the student do something) is provably more
efficient than either reading or quizzing.

  * •

We need to embrace generating errors as a way of learning, and give feedback
as quickly as possible to promote retention and comprehension.

### 6.3. Evaluation of Duolingo

Duolingo101010https://www.duolingo.com is a solution that provides spaced,
paced learning. Users are presented with different challenges, usually
centered around words (for example their pronunciation), and are able to
advance through a hierarchy of lessons. Upon completion of lessons, modules,
levels etc., users are granted experience points, which can be spent buying
more attempts, in-game apparel, and so on. On completing levels, the overall
retention of the lesson is noted using a strength bar. As time goes by, this
retention decreases, and the user is prompted to re-take lessons, not
dissimilar to the system provided by Anki.

Users are able to perform simple tasks like translating individual words or
sentences, pronouncing them correctly (using a microphone), conjugating nouns
and verbs, and so on. These tasks usually come in sets, and such a set in
turns constitutes a module or level in the game. However, progression need not
be linear, and more proficient users can opt to perform tasks qualifying for
more advanced modules early on if so desired.

When performing a task, the user has a given set of hearts or HP (hit points,
an analogy from games). When they fail a task, the count is decremented. Users
can also opt for timed lessons.

####  6.3.1 Expressiveness

Duolingo is proprietary software that does not offer a way for users to author
their own assets, and instead seeks to keep them content by providing it with
the program.

Duolingo allows users to translate texts to (im)prove their language
proficiency, and uses community filtering to ascertain the quality of the
translations. This is how the service pays for itself — as a mechanical turk
for natural languages, currently partnering with CNN and Buzzfeed[34]. It also
provides language certification services[35].

Users are able to provide feedback on tasks, for example whether a translation
is reasonable or too hard at this proficiency level.

Duolingo tracks metrics, like what lessons people struggle the most with,
adopting a wholesome data-driven approach. This combined with a flexible
userbase allows them to iterate quickly on the assets provided in the
lessons[36].

But in terms of actually contributing with learning material, DuoLingo has
nothing to offer.

####  6.3.2 Ease of use

Duolingo appears to be easy to use. Quick feedback and positive reinforcement
makes it easy to start using the system. There aren’t many options available,
except for audio settings (and other preferences), which makes the user
interface easy to use.

####  6.3.3 Takeaway points

  * •

Quick and simple feedback with additive gains facilitate user engagement
greatly.

  * •

Community can be used to enhance or ensure the quality of lessons.

  * •

A data-driven approach might foster good community relations and provide more
relevant, better assets as well.

### 6.4. Evaluation of RPGツクール

For something different, let’s consider RPGツクール (RPG Tsukūru). It’s a game
authoring tool wherein users create worlds through a tile-based map editor,
and breathe life into their worlds by scripting actors, events and objects.
Users may import their graphics and scripts, or use some of the wealth of art
assets, scripted events and characters that are distributed with the program.
The series is primarily successful in Japan, with English translations being a
rather new thing.

While the main reason for us considering RPGツクール is its engaging user
interface, it has in fact been used successfully for teaching. Authoring games
in RPGツクール has proven successful in teaching mathematics[37], and games
authored in RPGツクール have been used for encouraging programming students[38].
It is no secret that authoring and playing games is a sound way of both
encouraging learners and increasing their efficacy.

RPGツクール assumes little or no experience in programming or designing games, and
can as such be said to be aimed at newbies. However, several very successful
commercial computer games by experienced developers were authored using
RPGツクール.

####  6.4.1 Expressiveness

RPGツクール is primarily aimed at making Japanese Role-Playing Games such as the
old ファイナルファンタジ (Fainaru Fantajī) or イース (Īsu) games. The user may design tile-
based 2D overhead-view maps with different types of graphics for the tiles,
representing grass, ocean, and so on. Sprite sheets are used for animating
characters. The user can script events and design a program flow through this.

Designing a map can in many ways be compared to using a bitmap editor. What
You See Is What You Get. Authoring art assets outside of the ones that are
distributed with the program is usually done with dedicated tools, such as
using a video editor for making cut scenes, or an audio editor for making
sound clips, or a bitmap editor for making sprite sheets. More recent versions
have tools such as the character generator in which sprite sheets are actually
generated by having the user choose settings such as the hair style and skin
colour of the character they wish to generate.

The scripts for events and game objects in general are very powerful. The
programming language Ruby is used in recent versions. Ruby is a fully-fledged
Turing-complete programming language. The tool provides several high-level
constructs to facilitate the scripting. Additionally there are tools for
authoring scenarios and events through graphical user interfaces, comparable
to the character generator.

####  6.4.2 Ease of use

The latest edition of RPGツクール boasts being “simple enough for a child;
powerful enough for a developer”[39]. While certainly (theoretically)
powerful, professional reviews are less than kind when discussing the user
interface[40] — just like they’ve historically been unfavourable to the
series[41].

However, interestingly, user reviews seems very positive on most sites[42, 43,
44]. And even more interestingly, most user reviews that explicitly mention
the user interface are very happy with it, calling it “clean” and “intuitive”
amongst other things.

The map editor is simple and intuitive to anyone who’s ever used a bitmap
editor of any kind. But the plethora of nested menus that needs to be endured
to do anything else is unkind to the overall user experience. Add to that a
poor overview, complex games quickly burdens the mind with the need to
remember how your pieces all fit together, if they do so at all.

It may appear user reviews focus more on virtues of the map editor, whilst
professional reviews can’t get past the troubling interface for game objects
in general.

In order to fully take advantage of the program, one arguably must learn the
Ruby programming language to master scripts. Although we can hardly criticise
RPGツクール for facilitating truly knowledgeable users, a better user-interface
design might have bridged the gap to non-programmer users.

####  6.4.3 Takeaway points

  * •

What You See Is What You Get is well-liked amongst users.

  * •

Horribly complicated nested menu systems are, uh, well, not well-liked amongst
professional reviewers…

  * •

Authoring games can be a successful way of learning.

  * •

Playing games can successfully motivate learners.

### 6.5. Evaluation of LateralGM and ENIGMA

LateralGM111111https://github.com/IsmAvatar/LateralGM is a free tool to author
and edit games. It aims to be compatible with the file formats of Game Maker
and the newer GameMaker: Studio, and has a user interface very similar to that
of Game Maker. To compile and run games, LateralGM normally interfaces with
ENIGMA121212http://enigma-dev.org/, which is a compiler and runtime system
aiming to be backwards compatible with Game Maker games.

####  6.5.1 Expressiveness

LateralGM and ENIGMA form a multi-genre game authoring tool with features that
make it easy to use for beginners, but also powerful for advanced users.
Although the scripting language has some support for 3D graphics[45], the
software mainly focuses on 2D graphics and gameplay.

LateralGM presents the game as a set of different resources, including sounds,
background images, sprites, scripts, fonts, objects, and rooms. Users may edit
sounds and images within a game using external programs.

Objects may be programmed to act on various events. Events include the “step”
event (fired once in every update cycle, e.g. 30 times per second), user
input, collision detection, timers, action-/script-triggered events, and many
more[46]. Each event can be programmed using either drag-and-drop actions or
scripts. The set of drag-and-drop actions is extensive and includes basic flow
control, and may also be extended with action libraries.

Each object can be set to use a possibly animated sprite, which causes each
instance of the object to be drawn with that sprite. This sprite may also be
used for collision detection. The “draw” event is also available for producing
more advanced graphics. Objects may also be set as invisible, appearing in the
room editor and possibly providing collision detection with the associated
sprite, but not visible during gameplay.

The room editor allows users to graphically place instances of objects, either
freely or in a configurable grid, inside a room as they will appear in the
game. The user may define multiple layers of background and foreground images,
either as whole images, possibly repeated or stretched, or in the form of
placeable tiles, using the image as a tileset.

LateralGM loads and saves games in the proprietary file formats of various
versions of Game Maker. These file formats are not officially documented, but
have been reverse-engineered by LateralGM developers[47]. While LateralGM and
ENIGMA aim to be compatible with both current and earlier versions of Game
Maker and GameMaker: Studio, several of these versions of Game Maker have
incompatibilities between themselves. Not all features are implemented in
LateralGM and ENIGMA, and development is struggling to catch up with later
versions while still being compatible with earlier versions.

####  6.5.2 Ease of use

LateralGM lets users make simple games without writing a single line of code,
using only drag-and-drop actions and graphically placing object instances in
each room. Parametres to the actions may be given as scripting language
expressions, providing a smooth learning curve from simple actions to the full
scripting language.

####  6.5.3 Takeaway points

  * •

Drag-and-drop can be used together with a written programming language for a
very smooth learning curve for non-programmers.

  * •

Trying to keep up compatibility with proprietary tools and undocumented file
formats is difficult and may take more development resources than available.

### 6.6. Evaluation of Minecraft

Minecraft131313https://minecraft.net/ is a modern open-world (sandbox) game.
It allows the user to freely place, as well as craft (potentially new) blocks
in the game. Thus, it is possible to build whatever imagination and time
allows for in the game. The game is very popular, especially with the younger
demographics, and has been noted for its ability to inspire creativity in its
users[48].

Being a virtual world, and due to its popularity and game mechanics, it has
been suggested as a tool in primary and secondary education. Partly due to its
potential for stimulating a sense of presence in the students[49], which games
in general are notoriously successful at.

In the area of situated learning, Minecraft tends to be classified as a multi-
user virtual environment, which is of particular interest to teachers and
students alike. But it is a category of situated learning which tends to
suffer from some common issues, ranging from standardisation (can others
extend it?) to time (how quickly is the user able to use the game?)[50]. Due
to its unique proposition of low cost, huge popularity, and innate creativity,
Minecraft seems to offer a novel platform for learning to take place[49].

####  6.6.1 Expressiveness

Minecraft is very expressive in allowing the user to roam freely, harvest
resources, build structures, and craft new tools. As such, there are many
modifications available to extend the game with new functionality, like energy
systems, or new entities. There is currently no official application
programming interface for developers of such extensions, and most of the
extending is done using disassembly and patching the game locally.

Minecraft offers a primitive realisation of boolean digital circuits named
“redstone”. Redstone is a resource found in the game, and might in its
harvested state be placed on the ground to carry signals. Redstone carries
either a logical low (false) or high (true) signal, which can be (de)activated
through a number of means, for example levers. Redstone can also be used in
recipes to craft new blocks like signal repeaters or switchable lights.

More notably, the redstone system in Minecraft is able to emulate the
primitives needed to construct any analogue circuit. Thus, people are able to
construct everything from simple systems that open doors, to full-on 8-bit
processors[51]. Thus, the redstone system offers an interesting value-
proposition: Mojang has succesfully fooled hundreds of thousands of children
and adults alike into understanding primitive digital circuits. By
correspondence, they understand some set of boolean algebra, making them
capable programmers as well.

####  6.6.2 Ease of use

Minecraft is readily accessible, but has a steep learning curve[48]. Systems
like TooManyItems[52] makes the game slightly more accessible by making it
easy to peruse recipes, aiding discovery. However, Minecraft itself provides
little of a set path for playing the game, save a final boss fight, which is
optional, and pretty much orthogonal to the ordinary game mechanics.

####  6.6.3 Takeaway points

  * •

There are a number of issues with using virtual worlds in situated learning.
Minecraft provides a reasonable solution to many of them.

  * •

Seemingly complicated topics (like circuits) can in fact be taught to
children, given the right incentives

### 6.7. H5P

H5P is a standard allowing for reusable, portable e-learning modules. It does
so by providing plug-ins for Drupal141414https://www.drupal.org/,
WordPress151515https://wordpress.org/, and
Joomla161616https://www.joomla.org/, along with a file format specification
for distribution, as well as standard _content types_ with built-in editors.

The value proposition of H5P is that it integrates nicely with existing
infrastructure like databases and view layers, enabling developers to focus on
developing their modules.[53] I.e., it reduces the ceremony needed to get
going. Moreover, its portability provides a much-wanted ease of deployment for
site owners, as well as a bigger market for developers.

Before deciding whether to use H5P we needed to evaluate five core issues.

####  6.7.1 Are the abstractions offered by H5P powerful enough?

At its very core, H5P offers a contract for libraries, which might function as
full-blown applications, stand-alone libraries, or application programming
interfaces for any combination of the above.

This is done in part by making the libraries document their semantics
(‘semantics.json’), i.e. the properties (data structure) that are used to set
up the application. Each property is described by its name, as well as its
type. Different things have different semantic requirements: E.g., the ‘list’
(ordered group) semantic type requires the name of a single entity in it
(‘entity’). This information is used by the generic editor to allow users to
easily customise the library.

H5P also provides a container format that manages ceremony, like loading the
appropriate CSS and JavaScript files, as well as providing useful metadata
like the version and name of the library. This format includes a JavaScript
class which libraries extend to hook into H5P. Applications are required to
implement an ‘attach’ method, which accepts the identity of the container in
which to attach the module.

Of particular interest for us is the central event dispatcher
(‘H5P.EventDispatcher’), which is used to dispatch actions within H5P[54].
Thus, like Elm’s ports[18] or Facebook React’s Flux
architecture/dispatcher[55], a synchronous interface for internal
communication is provided, which might prove useful for multi-component
systems.

Beyond this, H5P leaves the library developers plenty of room to expand. The
H5P standard is pretty much “open world” and doesn’t necessarily limit the
structure of functionality of the JavaScript libraries it wraps, enabling
developers to make arbitrarily complex software.

However, it should be noted that H5P still depends on professional developers,
and doesn’t provide primitives that necessary make any sense for ordinary
users. Therefore, the standard per se doesn’t preclude a more user-oriented
canvas system.

In conclusion, H5P doesn’t itself provide the needed abstractions necessary,
but doesn’t hinder them from being developed. We can use H5P for now.

####  6.7.2 Are the abstractions high-level enough for our purposes?

Given our emphasis on an intuitive user-interface for liberating the BOP, H5P
quite frankly doesn’t cut it for the time being. But even if H5P is not nice
enough to use (at the moment) for non-technical users turned authors, it
_does_ play nice with other software — or, at the very least, it stays out of
the way. This is _very_ desirable to us.

Once we have H5P modules “up and running”, so to speak, there’s nothing
stopping us from moving ahead with other kinds of e-learning modules. There is
furthermore nothing precluding H5P being improved in the future; we could even
conceivably do that ourselves.

The honest truth is that H5P is not good enough for our intended target
demographic. But it _is_ free software that integrates nicely and may be
improved in the future. We’ll take it.

####  6.7.3 Can we generate valid H5P modules?

Yes. Another short answer, but a more positive one at that.

As already noted, H5P doesn’t limit any system wrapped in its container
format. Therefore, any functioning JavaScript system could in principle be
embedded within a H5P module, provided it only operates on a single node (due
to the complexity and performance implications of coordinating DOM operations
on overlapping elements).

####  6.7.4 Can we import H5P modules?

Again, yes! We can! We would need to make something that would be able to
parse the semantic definitions of H5P modules, and we’d have to use this in
some meaningful way. Effectively we should enhance the authoring tool as well
to make it more appropriate for our target demographic.

####  6.7.5 Can we offer a new way of composing H5P?

H5P has no composition guarantees, and resorts to Design by Contract (DbC) for
composite systems.[56] This does nothing to ensure a uniform communication
standard, which ensures flexibility, but does not provide a uniform interface.
The specification itself admits that this is due to the lack of interfaces in
JavaScript. Thus, the contracts are more by politeness and not by obligation,
potentially creating weak implementations.

Therefore, there is obviously definite value to be provided by a system
providing more stringent, verifiable contracts between components, for example
using a more powerful type system than the one found in JavaScript. This might
in turn result in more reusable and robust applications. Also, because H5P
effectively provides no composition contracts, this system might function as a
sandbox and provide useful feedback should future versions of H5P seek to
provide more formal requirements for composite modules.

One related problem is that H5P provides no standard for sub-module
inheritance/reuse of content, as the system only has a notion of full H5P
modules. We must figure out a way for several modules to refer to the same
data.

## 7\. Conclusions and further work

Our idea is a canvas for composing e-learning modules, initially targeting
H5P. It should be intuitive to use, yet potent enough to make sufficiently
refined modules. A simple user interface is vital to construct a ladder out
from the BOP, whilst a capable set of features is important to attract users
in general. Via gamification, users are encouraged to share their modules, and
to improve the modules of others. We encourage a self-regulating community
that promotes quality through a rating system for modules.

By authoring modules, the student becomes the teacher, which leads to more
effective learning. Learning by teaching is a powerful concept that enhances
the self-efficacy of students.

The canvas system is an important step towards our goal. However, much remains
to be done. We need to achieve a cohesive authoring experience. Either by
making our own modules, or by figuring out a way to make e-learning modules
authoring agnostic and cohesive at the same time.

## References

  * [1] Coimbatore Krishna Prahalad.  The Fortune at the Bottom of the Pyramid, revised and updated 5th anniversary edition: Eradicating poverty through profits.  FT Press, 2009.
  * [2] Jeffrey Brand and Shelley Kinash.  Crafting minds in Minecraft.  2013\.
  * [3] Claudio G Cortese.  Learning through teaching.  Management Learning, 36(1):87–115, 2005.
  * [4] Tatiana Gossen, Marcus Nitsche, and Andreas Nürnberger.  Search user interface design for children: Challenges and solutions.  In EuroHCIR, pages 59–62, 2012.
  * [5] CE Gordon.  The otter in massachusetts.  Science, pages 772–775, 1908.
  * [6] Michael Belanger, Nicole Clough, Nesime Askin, Luke Tan, and Carin Wittnich.  A review of violent or fatal otter attacks.  IUCN Otter Specialist Group Bulletin, 28(1):11–16, 2011.
  * [7] Yavuz Inal, Hatice Sancar, and Kursat Cagiltay.  Children’s avatar preferences and their personalities.  In Society for Information Technology & Teacher Education International Conference, volume 2006 1, pages 4259–4266, 2006.
  * [8] Randall Sadler.  Virtual worlds: An overview and pedagogical examination.  In Bellaterra journal of teaching and learning language and literature, volume 5, pages 0001–22, 2012.
  * [9] Ben Kuchera.  Blizzard silenced Hearthstone players, and it made the game amazing.  http://www.polygon.com/2014/4/18/5625802/hearthstone-chat-Blizzard, 2014\.
  * [10] Kommunal og moderniseringsdepartementet.  Forskrift om universell utforming av informasjons- og kommunikasjonsteknologiske (ikt)-løsninger.  9, 2013.
  * [11] Free Software Foundation, Inc.  Why educational institutions should use and teach free software.  https://www.gnu.org/education/edu-why.html, 2015.  Visited 2015-09-24.
  * [12] Evan Czaplicki.  Elm: Concurrent frp for functional guis.  Senior thesis, Harvard University, 2012.
  * [13] Jeff Smits.  Extreme pong: A browser-based game implemented using functional reactive programming.
  * [14] David Flanagan.  JavaScript: the definitive guide.  ” O’Reilly Media, Inc.”, 2006.
  * [15] Simon Buist.  Extending an ide to support input device logging of programmers during the activity of user-interface programming: Analysing cognitive load.  Bachelor of Science dissertation, The University of Bath, 2014.
  * [16] Alexander Berntsen.  The quest for programming Nirvana: On programming game systems in Haskell.  2014\.
  * [17] David Kraeutmann and Philip Kindermann.  Functional reactive programming and its application in functional game programming.  2015\.
  * [18] Elm.  Interop.  http://elm-lang.org/guide/interop.  Visited 2015-09-16.
  * [19] Simon Marlow et al.  Haskell 2010 language report.  http://www.haskell.org/onlinereport/haskell2010/, 2010.
  * [20] John Hughes.  Why functional programming matters.  The computer journal, 32(2):98–107, 1989.
  * [21] Philip Wadler and Stephen Blott.  How to make ad-hoc polymorphism less ad hoc.  In Proceedings of the 16th ACM SIGPLAN-SIGACT symposium on Principles of programming languages, pages 60–76. ACM, 1989.
  * [22] Simon Peyton Jones.  Wearing the hair shirt: a retrospective on haskell.  Invited talk at POPL, 206, 2003.
  * [23] Simon Peyton Jones.  The future is parallel, the future of parallel is declarative.  2012\.
  * [24] Koen Claessen and John Hughes.  QuickCheck: a lightweight tool for random testing of Haskell programs.  Acm sigplan notices, 46(4):53–64, 2011.
  * [25] Thomas Arts, John Hughes, Joakim Johansson, and Ulf Wiger.  Testing telecoms software with quviq quickcheck.  In Proceedings of the 2006 ACM SIGPLAN workshop on Erlang, pages 2–10. ACM, 2006.
  * [26] Memrise.  Memrise science.  http://www.memrise.com/science/.  Visited 2015-10-09.
  * [27] Rosalind Potts and David R Shanks.  The benefit of generating errors during learning.  Journal of Experimental Psychology: General, 143(2):644, 2014.
  * [28] What’s important when designing a touch user interface for kids apps?  http://www.quora.com/Whats-important-when-designing-a-touch-user-interface-for-kids-apps.
  * [29] Luke Wroblewski.  Touch-based app design for toddlers.  http://www.lukew.com/ff/entry.asp?1179, 2010.
  * [30] Damien Elmes.  Anki 2.0 user manual.  http://ankisrs.net/docs/manual.html.  Visited 2015-10-09.
  * [31] Erez Zukerman.  Review: Anki helps you to learn and memorize virtually anything, for free.  http://www.pcworld.com/article/2030095/review-anki-helps-you-to-learn-and-memorize-virtually-anything-for-free.html.
  * [32] Jim Burke.  It’s all in the cards.  Voices from the Middle, 10(1):54, 2002.
  * [33] Victor Benassi, Chris Hakala, Sarah A Ambrose, Marsha Lovett, Courtney M Clark, Robert A Bjork, Chee Ha Lee, Slava Kalyuga, John AC Hattie, Gregory CR Yates, et al.  Applying science of learning in education: Infusing psychological science into the curriculum.  2014\.
  * [34] Luis.  Duolingo now translating buzzfeed and cnn.  https://www.duolingo.com/comment/954969, 2014.  Visited 2015-10-09.
  * [35] Burr Settles.  Duolingo launches its certification program to take on toefl.  http://techcrunch.com/2014/07/23/duolingo-launches-its-language-certification-program/, 2013\.
  * [36] Burr Settles.  Duolingo’s data-driven approach to education.  http://duolingo.tumblr.com/post/41960192602/duolingos-data-driven-approach-to-education, 2013\.
  * [37] Marcus Vinicius Maltempi and Maurício Rosa.  Learning vortex, games and technologies: a new approach to the teaching of mathematics.  In INTERNATIONAL CONGRESS ON MATHEMATICAL EDUCATION, volume 10. Citeseer, 2004.
  * [38] Tiffany Ralph and Tiffany Barnes.  1 the catacombs: A study on the usability of games to teach introductory programming.
  * [39] RpgMaker.org.  RPG Maker VX ace.  http://www.rpgmaker.org/downloads/rpg-maker-vx-ace-downloads/, 2015\.  Visited 2015-10-09.
  * [40] John Wizard II.  John Wizard Review RPG Maker VX Ace.  http://www.johnwizard.com/rpg-maker-vx-ace-review.php, 2015.
  * [41] Miguel Lopez.  Rpg maker review.  http://www.gamespot.com/reviews/rpg-maker-review/1900-2633694/, 2000\.
  * [42] http://www.amazon.com/RPG-Maker-VX-Ace-Pc/product-reviews/B00AA25QUM.  Visited 2015-08-31.
  * [43] http://www.amazon.com/RPG-Maker-VX-Ace-Pc/product-reviews/B00AA25QUM.  Visited 2015-08-31.
  * [44] https://steamcommunity.com/app/220700/reviews/.  Visited 2015-08-31.
  * [45] 3d graphics functions.  http://enigma-dev.org/docs/Wiki/3D_Graphics_Functions.  Visited 2015-09-01.
  * [46] Quadduc Clam, IsmAvatar and Robert B. Colton.  Event.java.  https://github.com/IsmAvatar/LateralGM/blob/dc99fbf54e586d435ad488879b399bd2df362210/org/lateralgm/resources/sub/Event.java.  Visited 2015-09-01.
  * [47] http://sourceforge.net/projects/lateralgm/files/Formats/GM/.  Visited 2015-09-01.
  * [48] Anthony Gallegos.  Minecraft review.  http://uk.ign.com/articles/2011/11/24/minecraft-review, 2011.  Visited 2015-08-31.
  * [49] Catherine Coudray.  Minecraft in higher education, beyond learning activities.
  * [50] Lisa Dawley and Chris Dede.  Situated learning in virtual worlds and immersive simulations.  In Handbook of research on educational communications and technology, pages 723–734. Springer, 2014.
  * [51] William Wyatt Earnshaw.  Minecraft reprogrammable 8-bit redstone computer.  https://www.youtube.com/watch?v=nlt0xMVoJcM, 2013.
  * [52] http://www.minecraftforum.net/forums/mapping-and-modding/minecraft-mods/1272385-toomanyitems-the-inventory-editor-and-more-1-8.  Visited 2015-08-31.
  * [53] H5P.  Why develop for h5p?  https://h5p.org/node/3150, 2015.  Visited 2015-09-10.
  * [54] H5P.  H5p api reference.  https://h5p.org/documentation/api/H5P.EventDispatcher.html, 2015\.  Visited 2015-09-10.
  * [55] Bill Fisher.  Flux: Actions and the Dispatcher.  https://facebook.github.io/react/blog/2014/07/30/flux-actions-and-the-dispatcher.html, 2014\.
  * [56] H5P.  Contracts.  https://h5p.org/documentation/for-developers/contracts-and-addons, 2015\.  Visited 2015-09-10.

##  Appendix A Design document

This is the design document authored for our system. It is authored by a user
experience expert, Lene Thuseth, as we felt the interface emphasis warranted
professional involvement.

While it certainly does not cover all of our ideas, it does showcase the core
subset of what we envision in a minimum viable product. More importantly, it
brings our user-interface research observations to life, and shows the
fundamental interaction pattern and establishes the ever intangible
“look’n’feel” of our system. The shapes, colours, fonts, and overall
presentation is given careful thought. This is all described and explained in
the document.

See pages - of pdf/LBT_DesignDocu_v3-2_1_.pdf