# FEVA: Fast Event Video Annotation Tool

Snehesh Shrestha, William Sentosatio, Huiashu Peng, Cornelia Fermüller,
Yiannis Aloimonos
The University of Maryland College Park
College Park, MD 20740
<EMAIL_ADDRESS>

###### Abstract

Video Annotation is a crucial process in computer science and social science
alike. Many video annotation tools (VATs) offer a wide range of features for
making annotation possible. We conducted an extensive survey of over 59 VATs
and interviewed interdisciplinary researchers to evaluate the usability of
VATs. Our findings suggest that most current VATs have overwhelming user
interfaces, poor interaction techniques, and difficult-to-understand features.
These often lead to longer annotation time, label inconsistencies, and user
fatigue. We introduce FEVA, a video annotation tool with streamlined
interaction techniques and a dynamic interface that makes labeling tasks easy
and fast. FEVA focuses on speed, accuracy, and simplicity to make annotation
quick, consistent, and straightforward. For example, annotators can control
the speed and direction of the video and mark the onset and the offset of a
label in real time with single key presses. In our user study, FEVA users, on
average, require 36% less interaction than the most popular annotation tools
(Advene, ANVIL, ELAN, VIA, and VIAN). The participants (N=32) rated FEVA as
more intuitive and required less mental demand. The code and demo are
available at http://www.snehesh.com/feva.

Figure 1: “Speed Label” enables you to create annotations (red rectangle) in
real-time by marking start ($st_{p}$) and end time ($et_{p}$) with a single
key press respectively. FEVA automatically adjusts these times of the event
annotation based on your reaction-time ($\Delta r$) in order to give you the
most precise intended times ($st_{i}$ and $et_{i}$).

_K_ eywords Video Annotation Tools $\cdot$ User Interface Design $\cdot$ User
Interaction

##  1 Introduction

Social scientists need to code conversations and behaviors of videotaped
interviews and experiments that quickly add up to hours of footage [1, 2, 3].
Computer scientists need datasets with appropriately labeled ground truth for
machine learning that contains video clips spanning hundreds of hours [4, 5,
6], or even thousands of hours [7]. Annotating videos is thus time-consuming
and tedious.

There are existing VATs [8, 9, 10, 11, 12, 13] that offer a range of different
features for video annotation activities. However, they often fail to meet the
need of the researchers due to the steep learning curves with complicated
features, overwhelming interfaces, and poor interaction techniques leading to
longer annotation time, inconsistencies, and user fatigue.

For example, to analyze soccer games, researchers annotate player ball
possessions, kicks, and assists. Annotating with tools that pause for the user
to name the annotation every time, needing to annotate with mouse context menu
options, and not allowing overlapping annotations make it an extremely tedious
and time-consuming task. On the contrary, coding by hand is more
straightforward. An annotator can play the video at a slower speed, use a
clicker count or a stopwatch [14] to lap the timestamps in real-time, then
enter them into a spreadsheet when completed. As a result, many annotators
still code by hand and rely on spreadsheets [15]. At scale, when you have
hours of such footage, computer scientists often outsource or crowdsource such
annotation tasks [16, 17, 18, 19]. This can raise concerns about privacy and
reliability [20]. To get around this, researchers blur faces to obfuscate
participant identity. However, this could compromise the quality of the
emotional judgment, causing inconsistencies in the results.

In this paper, we aim to design a video annotation tool that makes labeling
tasks easy and fast. We interviewed researchers from different disciplines to
understand standard practices, workflows, and tools used for video annotation.
Specifically, we interviewed 13 researchers from 5 fields (neuroscience,
behavioral psychology, film studies, and computer science). Researchers
expressed reservations with existing VATs leading to avoiding them. We further
surveyed 59 VATs (the list is available in the appendix ). We categorized
their main features and interface design choices from firsthand experience and
analyzed video tutorials for tools that are not accessible. According to the
interviews and survey results, we propose five design criteria that would
benefit video annotation activities, which are detailed in section 4:

  * •

D1. Organize space based on operational workflow

  * •

D2. Streamline high-frequency actions

  * •

D3. Use algorithmic support when possible

  * •

D4. Adopt what works and redesign what doesn’t

  * •

D5. Allow flexibility

Figure 2: The shift keys activate the fine-tuning feature. The left and right
shift keys correspond to the start and end time of the label, respectively.
Pressing the arrow key while holding down the shift key adjusts the label’s
start or end time by a single frame.

These criteria inform the design of Fast Event Video Annotation (FEVA), a
video annotation tool with streamlined interaction techniques and a dynamic
interface that makes labeling tasks easy and fast. Simplified UI and features
such as real-time labeling with reaction time adjustments (figure 1), and
precise fine-tuning mechanisms (figure 2), help annotators create a large
number of accurate labels faster than any VAT.

To evaluate FEVA, we conducted two comparative studies. In the first study, we
compared the number of inputs users need to complete a task with FEVA versus
five other event-based VATs [8, 9, 10, 12, 13]. As seen in table 3, on
average, FEVA required 36.0% fewer inputs than competing VATs to do the same
task. In the second study, we asked 34 participants to perform annotations
with one of the VATs in random order. We found that regardless of the
background, users thought FEVA was more intuitive at 88% and easier to use at
91% of the time. Users also rated FEVA as requiring lower mental demand by 46%
(p < 0.00003), caused fewer frustrations by 62% (p <0.00147), had less
physical demand by 41% (p < 0.00187), and required less effort by 34% (p <
0.00324).

In Summary, in this paper, our contributions are as follows:

  * •

a comprehensive understanding of existing VATs with interviews, tool surveys,
and pilot tests.

  * •

a list of criteria for VAT tool design

  * •

FEVA, an event-based video annotation tool that lets you annotate faster and
more accurately.

  * •

a comparative study and a user study that demonstrates FEVA as more intuitive
and requiring less effort while creating more consistent and accurate
annotations.

##  2 Related Work

In this section, we introduce event-based annotation and tools we build upon,
the workflow used in the annotation process, and finally, the user interface
and the related interaction techniques. To understand the annotation workflow
and goals, we interviewed researchers and surveyed the literature on the steps
required to complete the desired annotations. We found two primary groups of
annotators: ones that annotate with the assistance of VAT and ones that rely
on more heuristic methods.

###  2.1 Event Based Annotation

Video annotation is the process of marking regions of interest (ROI) either a)
spacial (object annotation, OA) or b) temporal (event annotation, EA). OA is
marked on a single-time instance, referred to as frame-based annotation. EA is
the period when the event occurs. OA is prevalent in the CV community for
object detection and recognition, and bounding boxes [21, 22, 23, 24], dots
[25, 12], or polygon a.k.a segmentation [26, 27, 28] or masks [25, 29, 23] are
used to mark the annotation. While some VATs also track objects [30, 31] over
time [12, 25, 22, 24], it is different from EA. EA focuses on what is
happening in the scene, what the objects are doing, or what is done to the
objects [32, 33] rather than the objects themselves. So the start and the end
time marks are used. EA includes behaviors, interactions, emotional response,
speech, and movements [34, 35, 36]. Their application range across disciplines
from computer science for activity recognition [37, 12, 38, 39, 40, 41],
psychology for behavior analysis [11, 42, 35], to journalism to track and
present stories over timelines [43]. However, events annotation can be
challenging due to the complex nature of temporal navigation, the ability to
mark at the exact desired time of sliding events, use of available features of
the tools to facilitate the annotation. Therefore, the focus of developing
FEVA was to simplify the workflow and optimize the steps for the annotation of
events to make annotation easier and faster.

###  2.2 Heuristic Workflows

Some researchers prefer not to use specialized video annotation tools. For
example, some researchers thought it was easier to use a clicker to count the
time certain events occurred. This is error-prone, less reliable, and makes it
difficult to review the records in the future. Even though some VATs can
provide a similar functionality [12] using keypresses, researchers showed
hesitation using VATs with the fear of the initial setup time and the repeated
learning curve for new coders. In another study [14], research assistants (RA)
used a stopwatch to obtain the time taken "from when the OK button was pressed
to when the device beeped to signal completion of the RR count." These clicker
counts or stopwatch intervals are recorded either by hand with pen and paper
or entered in a spreadsheet. In another workflow, an interpersonal
relationship researcher described how, from a video of two people interacting,
they asked RAs to respond to research questions set up by the researchers
while watching a video. For example, multiple RAs focus on the entire
interaction or a specific individual in the video and respond to a Likert
scale on how attentive one of the partners was when the other spoke. These are
either recorded on paper or online using Qualtrics or Google forms.

###  2.3 Video Annotation Tools Workflows

The primary workflow using a VAT does not differ from the heuristic methods
for simple coding. The main difference is the higher learning curve and a
longer setup process. However, as the coding gets more complex, heuristic
methods have a diminishing return on speed as you have more annotators. The
annotation process is much slower, needing to do a lot of steps typically
tools might facilitate manually. However, not all VAT are created equal. The
workflow design, screen layout of features, interaction steps, and techniques
differ significantly across VATs.

Computer scientists and engineers design most VATs for computer vision,
machine learning, and robotics research and applications [12, 35, 4, 16, 7].
However, there are a handful of tools specifically designed for and used by
other domains such as film studies [13, 44] where VAT allows for character
analysis and scene analysis. Journalists use VAT to annotate and synchronize
events from multiple sources to present a cohesive story [43]. In sports,
teams annotate games and significant events and move to study for their teams
[45, 46]. While most tools have different focuses, the primary annotation is
marking a binary exists or not and a temporal range of events of interest. To
this end, events annotation is the foundation that makes further work easier
or, in some cases, possible. So it is essential to have accurate annotations.
Support for synchronized multi-camera views, micro and macro visualization of
events in a timeline, and convenient searching and reviewing video features
make FEVA a favorable choice in these areas.

####  2.3.1 Layout design: Balance of features and space usage

Some tools have a lot of features with complicated workflows and many
permutations of features that can be customized and used, so the UI is packed
with small windows, buttons, and layers of menu items to get to them. These
tools take much longer to learn and get used to; however, they can be powerful
once you learn them. Due to many features, such as media player controls,
annotation controls, and visualizations, the available screen real estate can
be quite challenging. So some tools group them into smaller windows [13, 9,
8], organize them through multiple levels of menus [9, 10], or through
different operational modes [12, 10, 13]. While this helps organize the
functionality, it isn’t easy to access, and one needs a lot of practice to
remember the various steps required to use the software. While there are other
much simpler tools that serve a very niche purpose [46, 47, 44, 48, 49], and
the layout is simple, and they make excellent use of the screen real estate
for the functionality they offer. These tools are easy to learn and start to
use. However, the features are limited. Our interviews found that most
researchers’ and annotators’ needs are in between. Most features are never
used in the complex VAT, while simple VAT limits them from doing more than the
niche they offer, and they need to pair with other tools to augment the gap to
complete their needs. We designed FEVA with the motivation of creating a tool
that is simple to navigate and use but comes with features that more complex
tools offer without it being overwhelming that you have to take a course to
use a tool.

###  2.4 Adoption

The keyboard shortcuts for VIA [12] are intuitive and practical, especially
the hints that pop up based on the context is something most tools lack that
FEVA adopts as well. FEVA uses popular shortcuts that have become the standard
for media players, such as the spacebar to play or pause, ctrl+Z, and ctrl+Y
to undo and redo, etc. VIAN [13] is the only tool you don’t have to select
before dragging the label with the mouse, which is the same in FEVA. Advene
[9], ELAN [10] and VIA [12] provide alternative ways to visualize or execute
the playback option, such as continuous mode. While useful in some instances,
most annotators used the default way without changing, which could be
confusing. While some tools [12] allows one to create a label when the movie
is playing, it only expects the start time with a fixed length. Unfortunately,
most users need to return to the label and readjust them. FEVA improves upon
this interaction by allowing a second key press to mark the endpoint.

##  3 Understanding the State-Of-The-Art VATs and Their Comparisons

###  3.1 Target Users

To understand if researchers from different disciplines annotate their data,
what that entails, and what the workflow looks like, we interviewed 3
neuroscience researchers, 3 behavior psychology researchers, 2 film study
instructors, and 5 computer scientists. The interview was semi-structured to
answer the following 3 questions:

  * •

IQ1: The nature of their research involving human studies, the kinds of data
collected, and if it entails video.

  * •

IQ2: The workflow in the data collection process, post-processing of these
data, and code generation.

  * •

IQ3: The structure and workflow for annotating the videos, and how the
annotations are used after.

Post-interview, the responses were tallied and coded for technical challenges.
The interview insights (II) are as follows:

  * •

II1. There were three primary temporal annotations. 1) A binary label to mark
the presence or absence of certain events. For example, researchers were
interested in counting "how many times a person touched their face as one
indicator of how nervous the participants were." 2) A range label marks the
beginning and end of an event of interest. "Participants annotate videos for
specific moments. For instance, a couple might interact, then the researchers
have each couple member watch the video and annotate whenever a specific
thought or feeling occurred to them. They do this with both participants to
see convergence and dissonance of thoughts, feelings, goals, etc., etc." 3)
Certain kinds of labels, such as mood or scenario, lasted longer than an
action label. "For instance, we might want to compute a rating of how
responsive or caring one individual is to their relationship partner. So a
Research Assistant might watch an entire interaction, focusing on a specific
individual in the video, and then answer a question about how attentive they
are when their partner speaks." Most VATs do a poor job of supporting these
labels, so researchers had workarounds such as creating multiple label tracks,
each dedicated to a specific response.

  * •

II2: The time, effort, and cost for data annotation are exponentially high. So
any system that made some improvements to make the annotation faster and more
reliable was always a huge win. "We spend a lot of our RA hours doing these
annotations. If there were a tool that could cut the time by even an hour, I
definitely would be using that tool."

  * •

II3: Even with very explicit codes, annotated data often had low temporal
precision, so the agreement rules were relaxed. "Many RAs review and annotate
the videos. There will be slight variations in when each RA thinks a certain
event happened."

  * •

II4: Collaboration and sharing of the annotation during the annotation process
and analysis step was cumbersome and required multiple steps to be in sync
between the teams. "…the students need to refresh the page if they are working
on annotating the same clip simultaneously to see what their classmates are
writing. That is also a problem for me since I like to add my comments while
they are working (to encourage them to elaborate on points or explore a new
point)."

  * •

II5: Current VAT provided a poor interface for researchers to explore,
analyze, and search the annotated data.

  * •

II6: Crowd-sourced or AI-generated annotation often needed so much review that
it was easier and faster for researchers’ RAs to do the annotations. "So I
used the [online automatic speech recognition tool]. With this, with the
premium, I still have to go in and edit everything. The labels for the actions
and the labels for the word-for-word speech need to match up in terms of start
and end time. So creating the labels by hand is actually easier for me."

This paper focuses on II2, II3, and II5, which help shape the design decisions
made in section 4.

###  3.2 Comparing Video Annotation Tools specifications

We created an extensive list of 59 VAT from the literature as listed in
appendix B. We were able to find their websites, download links or shared open
source codes, or at the minimum online videos of either talk by the authors or
how-to videos. We cataloged typical features most software supported and
unique features and techniques distinctive to each tool. In this extensive
survey, we share the table for the narrowed-down selected five tools. We
detail the selection method in section 6.1. Two types of tables were created:

  * •

based on high-level taxonomy as shown in table 1 and

  * •

based on features as shown in table 2

SN |  Features |  Description |  Advene |  ANVIL |  ELAN |  VIA |  VIAN |  FEVA (Ours)
---|---|---|---|---|---|---|---|---
0 |  Last Updated |  Month Year |  Jun 2020 |  Mar 2019 |  Mar 2020 |  Jul 2020 |  May 2020 |  Sep 2020
1 |  SW Platform |  Cloud vs Edge |  Edge |  Edge |  Edge |  Edge |  Edge |  Cloud or Edge
|  |  Native vs Web Based |  Native |  Native |  Native |  Web Based |  Native |  Web Based
|  |  Modular vs Static |  Static |  Static |  Static |  Static |  Static |  Modular
2 |  License |  Open Source vs Proprietary |  Open Source |  Proprietary |  Open Source |  Open Source |  Open Source |  Open Source
|  |  Commercial vs Open Access |  Open Access |  Open Access |  Open Access |  Open Access |  Open Access |  Open Access
|  |  Maintained vs Outdated |  Maintained |  Maintained |  Maintained |  Maintained |  Maintained |  Maintained
3 |  Cost |  Free, low cost, vs Expensive |  Free |  Free |  Free |  Free |  Free/Low Cost |  Free
|  |  One time vs Subscription |  N/A |  N/A |  N/A |  N/A |  N/A |  N/A
4 |  Collaboration |  Single User |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  ✓
|  |  Multi-User (Simultaneous) |  ✗ |  ✗ |  ✗ |  ✗ |  ✗ |  ✗
|  |  Crowd |  ✗ |  ✗ |  ✗ |  ✗ |  ✗ |  ✗
5 |  Target Users |  Technical vs Non-Technical |  Technical |  Technical |  Technical |  Technical |  Technical |  Both
|  |  Academic vs Commercial |  Academic |  Academic |  Academic |  Academic |  Academic |  Academic
6 |  Input Type |  Image |  ✗ |  ✗ |  ✗ |  ✓ |  ✗ |  ✗
|  |  Video |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  ✓
|  |  Audio |  ✗ |  ✗ |  ✓ |  ✓ |  ✗ |  ✓
7 |  Annotation Type |  Object |  ✗ |  ✗ |  ✗ |  ✓ |  ✗ |  ✓
|  |  Action |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  ✓
|  |  Events |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  ✓
|  |  Hybrid |  ✗ |  ✗ |  ✗ |  ✓ |  ✗ |  ✓
8 |  Annotation Approach |  Manual |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  ✓
|  |  Automatic |  ✗ |  ✗ |  ✗ |  ✗ |  ✗ |  ✗
|  |  Hybrid |  ✗ |  ✗ |  ✗ |  ✗ |  ✗ |  ✗
9 |  Annotation Format |  JSON |  ✗ |  ✓ |  ✗ |  ✓ |  ✗ |  ✓
|  |  XML |  ✓ |  ✗ |  ✓ |  ✗ |  ✗ |  ✗
|  |  SQL |  ✗ |  ✗ |  ✗ |  ✗ |  ✗ |  ✗
|  |  Others? |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  ✗
Table 1: VAT taxonomy comparison table. SN | Features | Description | Advene | ANVIL | ELAN | VIA | VIAN | FEVA
---|---|---|---|---|---|---|---|---
1 | Annotation types | Object Bounding Box | ✗ | ✗ | ✗ | ✓ | ✗ | ✓
|  | Object Mask | ✗ | ✗ | ✗ | ✓ | ✗ | ✗
|  | Object Dot | ✗ | ✗ | ✗ | ✓ | ✗ | ✓
|  | Temporal Events | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
2 | Playback controls | Play Pause FF RR | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
|  | Speed +/- | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
|  | Timeline Jump | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
3 | Preview | Thumbnail Previews | ✓ | ✗ | ✗ | ✗ | ✗ | ✓
4 | Label | Multi-track | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
|  | Group tracks | ✗ | ✗ | ✗ | ✗ | ✗ | ✗
|  | User-defined Label Types | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
|  | Show/Hide/Collapse/Expand | ✗ | ✗ | ✗ | ✗ | ✗ | ✗
5 | Speed Label | Sudo-Pedal | ✗ | ✗ | ✗ | ✓ | ✗ | ✓
|  | Transcribing Pedal Support | ✗ | ✗ | ✗ | ✗ | ✗ | ✓
6 | Resize |  | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
7 | Move |  | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
8 | Add |  | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
9 | Delete |  | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
10 | Edit |  | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
11 | Import |  | ✓ | ✓ | ✓ | ✗ | ✓ | ✓
12 | Import other formats |  | ✓ | ✓ | ✓ | ✗ | ✓ | ✓
13 | Video Support | MP4 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
|  | Others | ✓ | ✓ | ✓ | ✓ | ✓ | ✗
14 | Cameras | Multi-Cam | ✓ | ✗ | ✗ | ✗ | ✗ | ✓
|  | Switch View | ✓ | ✗ | ✗ | ✗ | ✗ | ✓
|  | Instant Switch View | ✓ | ✗ | ✗ | ✗ | ✗ | ✓
15 | History | Undo/ Redo | ✓ | ✗ | ✓ | ✗ | ✓ | ✓
16 | Search | Keyword | ✓ | ✗ | ✓ | ✗ | ✓ | ✓
|  | Filter by label type | ✓ | ✗ | ✓ | ✗ | ✓ | ✓
17 | User Config | Remember/ Save | ✗ | ✗ | ✓ | ✗ | ✗ | ✓
18 | Modular/ API | Add-In Support | ✓ | ✗ | ✓ | ✗ | ✓ | ✗
|  | Full Open Source Support | ✓ | ✗ | ✓ | ✓ | ✗ | ✓
|  | Custom Layers | ✗ | ✗ | ✗ | ✓ | ✓ | ✗
|  | Custom Tracks | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
19 | Layers | Show/Hide Layers | ✗ | ✗ | ✗ | ✗ | ✓ | ✓
|  | Human Joints Keypoints Support | ✗ | ✗ | ✗ | ✗ | ✗ | ✓
|  | Human Bounding Box Support | ✗ | ✗ | ✗ | ✓ | ✗ | ✓
|  | Human Mask Support | ✗ | ✗ | ✗ | ✓ | ✗ | ✗
20 | Export Support | Video Clips | ✓ | ✗ | ✓ | ✗ | ✗ | ✓
|  | Image Frames | ✗ | ✗ | ✓ | ✗ | ✗ | ✓
|  | Closed Caption | ✓ | ✗ | ✓ | ✗ | ✗ | ✓
Table 2: VAT features comparison table. Figure 3: Example of steps needed to
annotate in 3 different VATs. The Upper left screenshot is ANVIL, the upper
right is VIAN, and the bottom screenshot is of VIA.

As a contrasting difference, here is an example of the steps required to
create a new annotation using 3 different VATs, as shown in figure 3. We make
a more thorough step-by-step comparison in the evaluation section in 6.2.

  * •

In the upper left corner is ANVIL. It requires you to double-click the
starting point and click on the endpoint time mark to select the region. Then
right-click the shaded area to select from the context menu to create and edit
the label.

  * •

On the upper right corner is VIAN. In VIAN, you right-click in the space
between the tracks and the ruler. Then select create an annotation, then
select text as the type for text annotation.

  * •

In VIA, you create the track for the particular annotation and name it ("Bird"
in this case), then play the video till the movie’s starting point and hit the
letter ’A’ to create a fixed-size annotation. You can go back and adjust the
annotation.

We surveyed the layout and the functionalities available in a number of events
annotation software by using them to annotate videos or watch videos posted by
the authors. We noticed the following:

  * •

Most VATs except for VIA [12] screen was filled with buttons, menus, and
windows. While essential features were buried in multiple levels of menu
items, there needed to be more screen real estate utilized. The new VAT
requires to be simple to understand and easy to navigate.

  * •

Observations through video instructions and pilot testing, we noticed that
only limited features and controls were needed, primarily a) project and data
management, b) annotation management, c) video navigation, d) annotations
space, and e) the tool configurations. The new VAT needs a layout such that
these features are upfront and eliminate unnecessary features or allocate them
into rarely used spaces.

  * •

The annotation visualization was limiting. An entire track was dedicated to a
single label [13], and no overlapping time labels were possible [8, 10]. Only
some [12, 9] allow overlapping time labels. However, it is difficult to
distinguish and manipulate the labels. The new VAT needs to organize the
annotations automatically and better use annotation space.

  * •

While some tools [12, 11] have good annotation UX controls to create and edit
temporal placements, the use of mouse and keyboard had to be used
interchangeably between annotation steps. Some tools [8, 9, 10] have a very
cumbersome way to move or resize annotations. Some tools provided a history
option to undo/ redo [9, 10], while others provided no way to backtrack user
mistakes or perform experimental steps. The new VAT needed simple mouse
control and default keyboard shortcuts while allowing users to redefine them.

  * •

All tools required you to stop and annotate except VIA [12], which provided
real-time annotation during video playback. However, VIA only allowed fixed-
length annotation flags requiring adjustment of the endpoint later.
Furthermore, VIA has poor visualization and lacks control for overlapping
labels. The new VAT needs a fast real-time way to continue annotating without
stopping.

##  4 Design Considerations

According to the interviews, literature survey, and the use of the tools, we
propose five design criteria that would benefit video annotation activities:

  * •

D1. Organize space based on operational workflow: Features should be laid out
in a logical flow of the workflow while not veering too far away from standard
software conventions. Features not needed in that context should not be
visible or active to free up valuable screen real estate. High-frequency
controls and features should be in the middle of the screen.

  * •

D2. Streamline high-frequency actions:  The highly repeated actions, such as
creating annotations and fine-tuning them, should be optimized for easier and
faster execution. Try to accomplish them with a single key press when
possible. Stick to a single device (i.e., do not require some steps to use the
keyboard and the other steps mouse movements, using up precious time in
transitioning between the devices.) Leverage D1 when possible to minimize user
interaction required to accomplish the task.

  * •

D3. Use algorithmic support when possible:  Whenever possible, offload the
user and rely on the algorithm to take on the burden. For example, when the
timing is concerned, consider user reaction time and adjust for the lag in the
user input from the intended time. Additionally, allowing external modules
such as movement detection, human detection, speech detection, and recognition
can offload users’ need to find and annotate the events manually. However, we
should be cautious that no matter how good machine learning modules are, no
algorithm is 100% accurate. These should be considered as only additional
assistance and not to be completely relied on for ground truth generation.

  * •

D4. Adopt what works and reinvent what doesn’t:  Instead of reinventing the
wheel, adopt from other tools what works well based on user tests or pilot
tests and not instincts and personal preferences.

  * •

D5. Allow flexibility:  While having D4, tested default input methods is
great, adding flexibility by providing redundancies with mouse context menus
and keyboard shortcuts might be more intuitive for some users. Additionally,
let users redefine the key mapping as people have personal preferences.

##  5 FEVA

Figure 4: A screenshot of FEVA, where a participant interacting with a robot,
is being annotated.

The design considerations inform the minimalist design of FEVA, with most of
the screen real estate allocated for the video and the annotation. Inspired by
the clicker/ stopwatch methods and transcribing pedal, we created the speed
label. This feature lets you create labels during video play, where a user can
press a key to mark the starting and stopping points and continue to watch,
creating multiple labels with desired lengths. The system additionally
accounts for the user’s reaction time and adjusts the marked times as seen in
figure 1.

The speed label enables users to annotate in real-time, the fine-tuning
control lets the user make frame level adjustments, and the label organizer
arranges the label without ever overlapping them for direct access. The flat
UI is responsive and aesthetically pleasing. Using F-shape design [50], the
initial workflow items are on the top left, while the most viewed items are in
the middle of the screen, and the most interacted components are at the bottom
of the screen. Keeping most used elements in the hot zones while less used
elements such as camera selection and configuration buttons are on the right-
hand side in the less noticed area, making them easily tuned out unless
needed. The UI has redundant user input for flexible and fast interaction to
cater to novice and expert users. And underlying context and configuration-
dependent UI model comes to life only when you need them. Expert users can
annotate videos faster than in real-time by using media speed control and
speed labeler without touching the mouse. Researchers can also use the zoom
feature to visualize and analyze labels at a micro or macro level. FEVA UI can
display the most number of labels on one screen without losing their meaning.

###  5.1 Workflow

To create any project, upon selecting a video, FEVA imports it and creates an
empty annotation file, also referred to as a label or the dataset file (D1,
D2). Default tracks will be loaded, which can be customized from the
configuration (D3). Users can add or remove tracks. To play or pause the
movie, users can use the media player overlay on the main video or the
keyboard shortcut ’spacebar.’ You can use your mouse scroll button to navigate
the video in the timeline. You can also use the arrow keys with or without the
’ctrl’ key to move the video by different amounts. You can click and drag the
filmstrip, roll over the global timeline and click at the desired time, or
double-click the filmstrip or the local timeline window (D1, D2, D5). You can
also change the movie playback speed. You can zoom in and out at different
time intervals using the + and - icons around the white box on the global
timeline or roll over the timeline and ctrl and scroll.

To annotate, users can right-click on the tracks and select the label type
they want to create. Users can also hit the letter ’A’ key on the keyboard to
mark the start and a second time to mark the stop. See figure 1. This is
called the speed label, as you can keep annotating without stopping the movie.
Speed annotation requires a two-pass, but in our pilot tests, the speed label
is at least 1.5x faster than the traditional methods.

Following left-to-right and top-to-bottom conventions, workflow such as
loading the project, dataset, and manipulating the video and labels are
organized in that order. The main video is at the center of the screen
occupying the most space. Below the video are the annotation tracks that
follow conventions and eyes and the hand layout of users’ gaze and action
areas. See figure 5 and figure 4.

###  5.2 View/ Layout

As seen in figure 5, ’b’ shows the project selector from which you create a
new project and import videos. The ’d’ shows your label file selector to
create, load, save, merge, import, and export labels. You can search using the
’e’ and filter labels by type. You can double-click the label from the ’2’
label list to find the corresponding label in the timeline ’j.’ You can see
the thumbnail preview in ’h’ along with the current time window ’g’ and the
global timeline ’f.’

Figure 5: FEVA Screen Default Layout black boxed areas are 1) Toolbar, 2)
Label list, main video player, and multi-camera selector, 3) Video navigation
timeline, and 4) Label tracks. Key components are a) Logo, b) Project
selection, c) toolbar icons, d) Label data file selection, e) Search bar, f)
Global progress bar timeline, g) Local timeline ruler, h) Filmstrips, i) Label
type, and j) tracks and labels.

###  5.3 Control: User input system

Every media control controlled by the mouse also has its associated keyboard
shortcuts. For flexibility and efficiency, GUI for novice users that are based
on principles of recognition rather than recall [51], and shortcuts for expert
users for faster control. Users can press the play button overlayed on the
video or use the spacebar key at any point to play or pause the video. You can
also choose to speed up or slow down. To create an annotation while the video
is in play, you can press the letter ’A’ twice to indicate the start of the
label and end the label. This can be customized from the configuration. A
blank label is created after adjusting for your reaction time. You can also
double-click an empty space on a track or right-click the tracks and select a
label type you wish to create. While these shortcuts were selected to be
consistent with existing standards from other VATs, all shortcuts can be user-
defined easily in the user configuration.

To navigate, you can use your mouse to scroll or click and drag the
filmstrips, double click the desired point in the filmstrip or the global
progress bar. You can also jump to a specific label from the label list by
double-clicking it from the label list. Users can choose what they prefer by
providing multiple redundancies with both keyboard and mouse and keyboard
shortcuts that can be re-customized by the user.

###  5.4 Model/ State: Underlying UI support system

The UI makes use of the limited screen space (x-y plain) by using the z-axis
to layer components displayed and state changes based on contexts such as if
the video is playing, if labels are selected, if labels are being edited, or
if your mouse is hovering over a component or a specific feature is enabled in
the configuration. Every area is compacted with features that feel intuitive
based on affordance users naturally would assign those components. An example
is the video player area. When the video is playing, one would only see the
video. When a mouse pointer hovers over, media controls, current time and
frame number, and layer control are displayed. From the configuration, you can
also enable showing or hiding the video, human body keypoints [52], the
bounding box of humans or objects, segmentation masks, etc., that are
extensible for researchers to customize.

##  6 Evaluation

To evaluate FEVA, we compared FEVA with existing state-of-the-art (SOTA) VAT
with two studies.

  * •

Interaction Benchmark: To evaluate the theoretical limits of how fast users
could annotate with each VAT, we counted the number of user inputs required to
perform various tasks.

  * •

User Study: To evaluate user experience based on the user’s perceived workload
with each VAT, we conducted user studies where the users provided feedback
based on their experience.

###  6.1 The State-of-the-Art VAT Selection Method

We first created a master list of highly cited VAT that we could download and
use. In this list, we only included software that supported temporal
annotation that could be downloaded, installed, and run without taking extreme
measures for practical reasons. Therefore, VCode [11], SVAT [53], and VACA
[35] could not be included. Tools that were too specific such as ToolScape
[47], HistoryTracker [46], and CASAM [54] due to missing functionalities such
as start and end time, were removed from the list. Tools focused on crowd-
sourcing such as Glance [55] and CoAT [42], were excluded as we conducted a
single-user study. Using these criteria, we could narrow down the tools to be
compared in the study. EagleView [36] was extremely unstable to run, so we
could not test them. We narrowed down to Advene [9], ANVIL [8], Elan [10], VIA
[12], and VIAN [13].

###  6.2 Interaction Benchmark

To compare the steps required to do a particular task with each VAT, we
counted the number of clicks, double-clicks, mouse movements, and keyboard key
presses and took a cumulative sum as seen in table 3. If there were multiple
ways of completing a task, we included the fastest method for that tool. For
example, if you can press Ctrl+N to create a new project (keypresses count =
2) or can move your mouse to the main file menu, click the file, move the
mouse to a new project, and click on the menu item (mouse move = 2 and mouse
clicks = 2, total = 4), then we took the lesser of the two.

Table 3 shows the 15 tasks considered for the evaluation. These included basic
setting, label creation, and manipulation tasks. We selected tasks that the
majority of the VATs could do. If a VAT missed a feature, we assigned the
worst count received by competing with the VATs. For example, [12] does not
support "undo" or "redo" and received a count of 2.

The number of inputs required in FEVA is significantly less than the SOTA, as
shown in table 3. On average, FEVA requires 36% less input than the SOTA.
Based on the T-test, FEVA required significantly less input than all tools
except VIA, which was not statistically significant.

SN | Tasks | Advene | ANVIL | ELAN | VIA | VIAN | FEVA
---|---|---|---|---|---|---|---
1 | Create a project + Import a video | 7 | 8 | 8 | 7 | 14 | 3
2 | Create a single label | 4 | 6 | 7 | 2 | 5 | 2
3 | Create multiple labels | 8 | 12 | 14 | 3 | 9 | 3
4 | Create and name label | 4 | 9 | 7 | 7 | 9 | 7
5 | Edit labels | 4 | 7 | 3 | 3 | 3 | 5
6 | Resize labels | 6 | 5 | 6 | 4 | 4 | 4
7 | Move labels | 6 | 12 | 6 | 5 | 4 | 4
8 | Change label type | 6 | 6 | 6 | 6 | 6 | 4
9 | Delete labels | 3 | 5 | 4 | 3 | 3 | 3
10 | Find labels | 3 | 3 | 5 | 5 | 3 | 2
11 | Save labels | 2 | 2 | 2 | 2 | 2 | 2
12 | Load labels | 4 | 4 | 4 | 4 | 8 | 4
13 | Navigate video | 2 | 6 | 2 | 1 | 2 | 1
14 | Play/ Pause video | 2 | 1 | 2 | 1 | 1 | 1
| Play only label video | 6 | 5 | 4 | 3 | 6 | 2
15 | Undo/ Redo | 2 | 2 | 2 | 2 | 2 | 2
| TOTAL SCORE | 69 | 93 | 82 | 58 | 81 | 49
| FEVA Faster by | 29% | 47% | 40% | 16% | 40% |
| p-value | 0.0255 | 0.0013 | 0.0149 | 0.1321 | 0.0223 |
Table 3: The list of tasks done using the fastest possible methods in each
software (shortcuts where applicable). Each number reflects a cumulative sum
of mouse clicks, double clicks, movement, and key presses. The last row shows
how much FEVA is faster than the SOTA in percent (%) and the T-test p-values.

###  6.3 User Study

To evaluate user experience, we conducted a user study where participants used
two VAT, FEVA, and one another selected SOTA VATs (Advene, ANVIL, ELAN, VIA,
and VIAN) in a round-robin fashion. We counterbalanced the order of the two
tools by alternating the order with the next participant. Due to the COVID-19
regulations, we conducted our study via Zoom remote shared screen and control
feature. This introduced some lag in the user experience. However, since both
tools were remote, we assumed that the effects of the lag on the outcome were
not significantly discriminant. Participants were given an approximate time
range where an event occurs with clear descriptions of the events to annotate,
for instance, as seen in figure 6, "between 4 minutes and 20 seconds and 4
minutes 40 seconds, please annotate bunny jump roping." Participants completed
approximately 24 tasks until they gave up on the tool. After completing all
the tasks in the first software, they filled out the NASA Task Load Index [56]
questionnaire with a 5-point Likert scale. And this was repeated with the
second tool.

####  6.3.1 Participants

We recruited 34 participants in the University community via email and social
media forums. In our study of N=32, the participants were 53% male and 47%
female, had a mean age of 30.4 +/- 5.9, with 84% not having video annotation
experience, and 66% had no experience with video editing. Two participants had
to be dropped due to zoom connection issues and are not counted in N=32.

####  6.3.2 Procedure

For this study, we trained annotators for 3 minutes by watching a short
training video that taught the basics of media controls, video timeline
navigation, and how to create and edit annotations, followed by practice
trials for each item with the research coordinator answering any questions.
They then spent another 2 minutes exploring the tool on their own. The
participants spent the next 5 minutes practicing the tasks assigned
individually by the researcher where they were allowed to ask questions. Once
they got comfortable, they were randomly given four categories of tasks shown
in the list below, with each type of task repeated at least three times. The
tasks chosen were the most fundamental and repeated tasks annotators must do
during video annotation. The tasks ranged in complexity, with some tasks
requiring combinations of the fundamental steps. For instance, some tasks
simply asked the participant to navigate the video to 2 minutes and 20
seconds. In contrast, others asked participants to annotate three consecutive
events between a specific time and name them appropriately. They were no time
limits to perform the tasks. They worked on the standard freely available "Big
Buck Bunny" video which is approximately 10 minutes long at 720p resolution.
Figure 6 demonstrates one example task. After completing all the tasks with
the first tool, the participants filled out the NASA TLX workload
questionnaire and repeated the tasks with the second VAT.

Figure 6: An example of a task where a participant was asked to annotate an
event of the bunny jump roping between 04:28 minutes mark and 04:31 minutes
mark in FEVA.

We conducted the following four categories of basic tasks during the user
study:

  * •

Navigate the video a) play/ pause the video, b) jump to a specific point in
the timeline, and c) jump to a precise point where a particular label is.

  * •

Label Creation a) Create a new label at a specific time with a specific
length, b) Create a label when a participant shows a specific behavior (e.g.,
a character yawns, eats an apple, etc.), and c) create multiple labels in a
row.

  * •

Label Content Manipulation a) Write text annotation for a label created and b)
Modify annotation text.

  * •

Label Temporal Manipulation a) Move the label by a specific number of seconds
and b) Resize the label to change its starting time or ending time to match a
specific behavior by the person in the video

####  6.3.3 Results

In this study, on average, the users felt less metal demand by 46% (p <
0.00003) with FEVA than the SOTA, less physical demand by 41% (p < 0.00187),
less effort was required by 34% (p < 0.00324), and felt less frustration by
62% (p <0.00147). The difference in the temporal demand and the performance
level indexes was not significant. We attributed this to there not being a
time limit enforced during the study, and except for one user on VIAN, where
the user gave up, all other users completed all the tasks.

FEVA |  Mental Demand |  Physical Demand |  Temporal Demand |  Performance Level |  Effort |  Frustration Level
---|---|---|---|---|---|---
MEAN (n=32) |  1.8 |  1.5 |  1.8 |  4.3 |  2.3 |  1.7

Table 4: Shows FEVA’s average score on a NASA Task Load Index with a 5-point
Likert scale.

FEVA vs. (%) |  Mental Demand |  Physical Demand |  Temporal Demand |  Performance Level |  Effort |  Frustration Level
---|---|---|---|---|---|---
Advene (n=5) |  75% |  57% |  0% |  14% |  67% |  183%
ANVIL (n=8) |  25% |  36% |  -8% |  3% |  17% |  6%
ELAN (n=8) |  29% |  42% |  13% |  12% |  38% |  83%
VAI (n=8) |  44% |  36% |  6% |  11% |  22% |  36%
VIAN (n=3) |  120% |  40% |  20% |  17% |  83% |  140%
MEAN (n=32) |  46% |  41% |  5% |  10% |  34% |  62%

Table 5: Shows how FEVA compared to the other VAT. A positive number indicates
how much people perceived FEVA to be better than other VATs in the NASA TLX
respective six dimensions, and a negative number indicates how much worse.

###  6.4 User Feedback

####  6.4.1 The Good

On average, users expressed FEVA was more intuitive 88% of the time and that
FEVA was easier to use 91% of the time. The features users liked the most were
the speed label, fine adjustments, "cooler feel," locating label and label
playback. A few users wished they could go back and change their feedback for
the first tool once they used the second tool. This was typical when they felt
they gave the first tool too high scores after using FEVA. In contrast, this
did not happen when it was the other way around. One user said the user wanted
to start a fan club and wanted to volunteer to annotate because it was "so
fun."

####  6.4.2 The bad

One user mentioned that the user preferred traditional windowed UI for serious
work. So the user thought FEVA looked too mainstream tablet app-like. Another
user stated, "while I think it was fine for me, I don’t think my mom will be
able to use either of the tools. So I gave low scores to both of them." A few
users complained that they did not like pressing the enter key to confirm the
label after editing them. Clicking elsewhere, causing the loss of what was
just typed as a cancel feature, was not popular.

####  6.4.3 The ugly

The majority of the confusion, however, was about the global and the local
timeline due to needing a clear separation and sharing the same preview
component. Participants remotely controlling the UI of the VATs over a Zoom
call on the research coordinator’s computer noticed a lag in the effect of
their actions. Some users complained that the UI did not update fast enough
due to the Zoom lags. "Maybe because I am controlling your computer through
Zoom, but a huge delay made it harder for me to resize the labels."

##  7 Implementation

###  7.1 Framework and Dependencies

We used ReactJS [57], an efficient component-based JavaScript library, and
wrote the architecture to be lightweight and responsive, so it works on most
people’s computers. The installation has only two dependencies of Python and
Flask. The front end relies on standard HTML, Javascript, ReactJS, and CSS. We
designed all the controls to optimize for performance and flexibility to
customize. We detail the UI layout breakdown in figure 5 and section 5.

###  7.2 Architecture

FEVA uses a simple server-client architecture typical for many web-based
applications. The server side runs on Python 3.5x or newer with Flask as the
webserver. The server side primarily handles servicing data (web content,
annotation data, and video streaming) when requested by the client-side
application. For FEVA, most of the modules and the design are on the client
side. We show more details of all the modules and their interaction with other
modules in the block diagram in appendix 8.

Figure 7: FEVA Client-Server Architecture Diagram. We include a more detailed
block diagram of the different modules on the client side in the appendix 8.

##  8 Discussion

This is the first version release of our tool FEVA, where we focused on
building the fundamental tool while streamlining the user interface and
interactions to make annotating events faster, intuitive, robust, and more
accurate. While these early results look promising, there are more research
questions that need to be further explored.

###  8.1 User Study

In our pilot and user study, we conducted a short-duration study. In our user
study, we assumed that 15 minutes was sufficient time for participants to
learn and practice annotating, which is how we designed the first evaluation
of comparing multiple VATs. However, we need to further our research by
conducting a longitudinal user study to understand the impact of our design on
users as they get more comfortable with the software.

###  8.2 User Input

In the input sector, we considered the keyboard and mouse/ trackpad as the
interaction devices at this stage. Still, we need to expand this to other
kinds of inputs, such as touch, speech, and gesture, to explore the potential
benefits of multi-modal methods.

###  8.3 Target Users

As more general public gets involved in the coding process, we focused this
study so anyone can participate in the coding process. Future studies to
gather feedback from seasoned coders will be valuable in understanding how
they use VAT.

###  8.4 VAT benchmarking study

In section 3 study, we counted the number of inputs as the pilot study showed
the correlation between the number of clicks and the time taken to complete a
task. A more comprehensive study should be considered, including the mouse
movements and time taken that can reflect user confusion and a more accurate
performance metric for completing the task.

###  8.5 Implicit

In section 6.3, we focused on the user’s perception to reflect their
experience. Future studies should expand to more quantitative measures
implicit in evaluating user confusion, performance, and success. We would also
like to conduct more studies to understand labeling consistency.

###  8.6 The layout

Based on user feedback, lessons learned from our observations, and the process
of comparing with existing tools, we could have done better. The multi-camera
selector layout takes up a lot of screen real estate. Many users found the
global timeline being so close to the local timeline and the thumbnail view
without any separation confusing and more challenging to get used to. We have
planned to redesign those experiences in the subsequent versions. We will
focus on several optimization opportunities in the next release to make FEVA
even faster.

###  8.7 Future Work

Beyond the incremental improvements, there were key features that we have
planned for the future:

  * •

Adaptive: All the input controls were linear. We plan to explore dynamic and
adaptive control systems to various new interaction techniques for faster
annotation and a more intuitive experience.

  * •

Extensible: An easier workflow for the open-source community to extend the
features.

  * •

AI assisted: While this had some algorithmic support, better integration with
machine learning and deep neural network models is needed. We will further
research how AI can augment the annotation process while exploring ways to
inform the users of these models’ inaccuracies, uncertainties, and inherited
bias.

  * •

Remote videos: While our internal prototypes support YouTube, there are
optimization opportunities that we need to explore before they can be used
seamlessly as an alternative to the local MPEG videos. We will also explore
other online streaming platforms.

  * •

Case study: While we are working with some research labs in evaluating the
FEVA for their video annotation [58], we want to invite other interested
research labs to try out FEVA, collaborate with us, and grow as a community to
address needs that may not have been realized by our research so far.

##  9 Conclusion

We present a new event video annotation tool with streamlined interaction
techniques and a dynamic UI, contextually visible, and active features
organized based on the workflow and usage frequency. With features like speed
labeling, users can accurately annotate videos in real-time. With simplified
onboarding and workflow, researchers can set up and start annotating videos
using minimal time. We release FEVA’s source code in GitHub for everyone to
try and further extend its features. The community can also find project
samples, tutorial videos, GitHub issues for support, and future updates on the
GitHub page. As we expand our case studies, we invite more researchers to use
FEVA or contact us if you wish to collaborate.

## Acknowledgments

We thank Chethan Parameshwara, Levi Burner, Lindsay Little, and peers from UMD
and the Perception and Robotics Group for their valuable feedback and
discussions. We extend special thanks to all our project contributors Johnny
Chiu, Rachelle Sims, John Gao, Leya Abraham, Vikram Sehgal, Swagata
Chakroborty, Lucas Stuart, and Lin Chen. The support of NSF under grant OISE
2020624 is greatly acknowledged.

## References

  * [1] Pat Broadhead.  Cooperative play and learning from nursery to year one.  Play and learning in the early years, pages 43–60, 2010.
  * [2] Grace Megumi Chen, Keith Jonathon Yoder, Barbara Lynn Ganzel, Matthew S Goodwin, and Matthew Kenneth Belmonte.  Harnessing repetitive behaviours to engage attention and learning in a novel therapy for autism: an exploratory analysis.  Frontiers in psychology, 3:12, 2012.
  * [3] Virginia K Choi, Snehesh Shrestha, Xinyue Pan, and Michele J Gelfand.  When danger strikes: A linguistic tool for tracking america’s collective response to threats.  Proc. Natl. Acad. Sci. U. S. A., 119(4), January 2022.
  * [4] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray.  The epic-kitchens dataset: Collection, challenges and baselines.  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2020.
  * [5] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  International Journal of Robotics Research (IJRR), 2013.
  * [6] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter.  Audio set: An ontology and human-labeled dataset for audio events.  In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 776–780. IEEE, 2017.
  * [7] Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan.  Youtube-8m: A large-scale video classification benchmark.  arXiv preprint arXiv:1609.08675, 2016.
  * [8] Michael Kipp.  Anvil: The video annotation research tool.  Handbook of corpus phonology, pages 420–436, 2014.
  * [9] Olivier Aubert, Yannick Prié, and Daniel Schmitt.  Advene as a tailorable hypervideo authoring tool: a case study.  In Proceedings of the 2012 ACM symposium on Document engineering, pages 79–82, 2012.
  * [10] Peter Wittenburg, Hennie Brugman, Albert Russel, Alex Klassmann, and Han Sloetjes.  Elan: a professional framework for multimodality research.  In 5th International Conference on Language Resources and Evaluation (LREC 2006), pages 1556–1559, 2006.
  * [11] Joey Hagedorn, Joshua Hailpern, and Karrie G Karahalios.  Vcode and vdata: illustrating a new framework for supporting the video annotation workflow.  In Proceedings of the working conference on Advanced visual interfaces, pages 317–321, 2008.
  * [12] Abhishek Dutta and Andrew Zisserman.  The via annotation software for images, audio and video.  In Proceedings of the 27th ACM International Conference on Multimedia, pages 2276–2279, 2019.
  * [13] Gaudenz Halter, Rafael Ballester-Ripoll, Barbara Flueckiger, and Renato Pajarola.  Vian: A visual annotation tool for film analysis.  In Computer Graphics Forum, volume 38, pages 119–129. Wiley Online Library, 2019.
  * [14] Charlotte Ward, Kevin Baker, Sarah Marks, Dawit Getachew, Tedila Habte, Cindy McWhorter, Paul Labarre, Jonathan Howard-Brand, Nathan P Miller, Hayalnesh Tarekegn, et al.  Determining the agreement between an automated respiratory rate counter and a reference standard for detecting symptoms of pneumonia in children: Protocol for a cross-sectional study in ethiopia.  JMIR Research Protocols, 9(4):e16531, 2020.
  * [15] Rob Tannen.  Feature the researcher-tool mismatch: improving the fit between user researchers and technology.  interactions, 15(5):74–78, 2008.
  * [16] Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal.  Grounding action descriptions in videos.  Transactions of the Association for Computational Linguistics, 1:25–36, 2013.
  * [17] Carl Vondrick, Donald Patterson, and Deva Ramanan.  Efficiently scaling up crowdsourced video annotation.  International journal of computer vision, 101(1):184–204, 2013\.
  * [18] Hugo Jair Escalante, Víctor Ponce-López, Jun Wan, Michael A Riegler, Baiyu Chen, Albert Clapés, Sergio Escalera, Isabelle Guyon, Xavier Baró, Pål Halvorsen, et al.  Chalearn joint contest on multimedia challenges beyond visual analysis: An overview.  In 2016 23rd international conference on pattern recognition (ICPR), pages 67–73. IEEE, 2016.
  * [19] Michael Riegler, Martha Larson, Concetto Spampinato, Pål Halvorsen, Mathias Lux, Jonas Markussen, Konstantin Pogorelov, Carsten Griwodz, and Håkon Stensland.  Right inflight? a dataset for exploring the automatic prediction of movies suitable for a watching situation.  In Proceedings of the 7th International Conference on Multimedia Systems, pages 1–6, 2016.
  * [20] Walter S Lasecki, Mitchell Gordon, Winnie Leung, Ellen Lim, Jeffrey P Bigham, and Steven P Dow.  Exploring privacy and accuracy trade-offs in crowdsourced behavioral video coding.  In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pages 1945–1954, 2015.
  * [21] Sungjoo Park and Chang Mo Yang.  Interactive video annotation tool for generating ground truth information.  In 2019 Eleventh International Conference on Ubiquitous and Future Networks (ICUFN), pages 552–554. IEEE, 2019.
  * [22] Carl Vondrick, Donald Patterson, and Deva Ramanan.  Efficiently scaling up crowdsourced video annotation.  International journal of computer vision, 101(1):184–204, 2013\.
  * [23] Simone Bianco, Gianluigi Ciocca, Paolo Napoletano, and Raimondo Schettini.  An interactive tool for manual, semi-automatic and automatic video annotation.  Computer Vision and Image Understanding, 131:88–99, 2015.
  * [24] Vott: Visual object tagging tool: An electron app for building end to end object detection models from images and videos, October 2019.
  * [25] Tewodros A Biresaw, Tahir Nawaz, James Ferryman, and Anthony I Dell.  Vitbat: Video tracking and behavior annotation tool.  In 2016 13th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pages 295–301. IEEE, 2016.
  * [26] Ajay Mishra, Yiannis Aloimonos, and Cornelia Fermüller.  Active segmentation for robotics.  In 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 3133–3139. IEEE, 2009.
  * [27] Ching L Teo, Cornelia Fermüller, and Yiannis Aloimonos.  Detection and segmentation of 2d curved reflection symmetric structures.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1644–1652, 2015.
  * [28] Francisco Barranco, Cornelia Fermüller, and Eduardo Ros.  Real-time clustering and multi-target tracking using event-based sensors.  In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5764–5769. IEEE, 2018.
  * [29] Bryan C Russell, Antonio Torralba, Kevin P Murphy, and William T Freeman.  Labelme: a database and web-based tool for image annotation.  International journal of computer vision, 77(1-3):157–173, 2008\.
  * [30] Cornelia Fermüller and Yiannis Aloimonos.  Tracking facilitates 3-d motion estimation.  Biological Cybernetics, 67(3):259–268, 1992.
  * [31] Anton Mitrokhin, Cornelia Fermüller, Chethan Parameshwara, and Yiannis Aloimonos.  Event-based moving object detection and tracking.  In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1–9. IEEE, 2018.
  * [32] Cornelia Fermüller, Fang Wang, Yezhou Yang, Konstantinos Zampogiannis, Yi Zhang, Francisco Barranco, and Michael Pfeiffer.  Prediction of manipulation actions.  International Journal of Computer Vision, 126(2):358–374, 2018\.
  * [33] Eadom Dessalene, Chinmaya Devaraj, Michael Maynord, Cornelia Fermüller, and Yiannis Aloimonos.  Forecasting action through contact representations from first person video.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [34] Stamatia Dasiopoulou, Eirini Giannakidou, Georgios Litos, Polyxeni Malasioti, and Yiannis Kompatsiaris.  A survey of semantic image and video annotation tools.  In Knowledge-driven multimedia information extraction and ontology evolution, pages 196–239. Springer, 2011.
  * [35] Brandon Burr.  Vaca: a tool for qualitative video analysis.  In CHI’06 extended abstracts on Human factors in computing systems, pages 622–627, 2006.
  * [36] Frederik Brudy, Suppachai Suwanwatcharachat, Wenyu Zhang, Steven Houben, and Nicolai Marquardt.  Eagleview: A video analysis tool for visualising and querying spatial interactions of people and devices.  In Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces, pages 61–72, 2018.
  * [37] Matthew Evanusa, Snehesh Shrestha, Vaishnavi Patil, Cornelia Fermüller, Michelle Girvan, and Yiannis Aloimonos.  Deep-readout random recurrent neural networks for real-world temporal data.  SN Comput. Sci., 3(3), May 2022.
  * [38] Yezhou Yang, Cornelia Fermüller, Anupam Guha, and John Aloimonos.  A cognitive system for human manipulation action understanding.  In The Second Annual Conference on Advances in Cognitive Systems (ACS), volume 2. Citeseer, 2013.
  * [39] Yezhou Yang, Yiannis Aloimonos, Cornelia Fermüller, and Eren Erdal Aksoy.  Learning the semantics of manipulation action.  arXiv preprint arXiv:1512.01525, 2015.
  * [40] Ching L Teo, Yezhou Yang, Hal Daumé, Cornelia Fermüller, and Yiannis Aloimonos.  Towards a watson that sees: Language-guided action recognition for robots.  In 2012 IEEE International Conference on Robotics and Automation, pages 374–381. IEEE, 2012.
  * [41] Konstantinos Zampogiannis, Yezhou Yang, Cornelia Fermüller, and Yiannis Aloimonos.  Learning the spatial semantics of manipulation actions through preposition grounding.  In 2015 IEEE international conference on robotics and automation (ICRA), pages 1389–1396. IEEE, 2015.
  * [42] Aziret Satybaldiev, Peter Hevesi, Marco Hirsch, Vitor Fortes Rey, and Paul Lukowicz.  Coat: a web-based, collaborative annotation tool.  In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers, pages 814–818, 2019.
  * [43] Dalton Bennett, Sarah Cahlan, Aaron C. Davis, and Joyce Sohyun Lee.  The crackdown before Trump’s photo op, 2020 (accessed September 15, 2020).
  * [44] Bradford Hosack.  Videoant: Extending online video annotation beyond content delivery.  TechTrends, 54(3):45–49, 2010.
  * [45] Fabio Sulser, Ivan Giangreco, and Heiko Schuldt.  Crowd-based semantic event detection and video annotation for sports videos.  In Proceedings of the 2014 International ACM Workshop on Crowdsourcing for Multimedia, pages 63–68, 2014.
  * [46] Jorge Piazentin Ono, Arvi Gjoka, Justin Salamon, Carlos Dietrich, and Claudio T Silva.  Historytracker: Minimizing human interactions in baseball game annotation.  In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1–12, 2019.
  * [47] Juho Kim.  Toolscape: enhancing the learning experience of how-to videos.  In CHI ’13 Extended Abstracts on Human Factors in Computing Systems, CHI EA ’13, pages 2707–2712, New York, NY, USA, April 2013. Association for Computing Machinery.
  * [48] Daniel Levin.  Timelinely, 2020 (accessed September 15, 2020).
  * [49] Sylvain Lagrue, Nathalie Chetcuti-Sperandio, Fabien Delorme, Chau Ma Thi, Duyen Ngo Thi, Karim Tabia, and Salem Benferhat.  An ontology web application-based annotation tool for intangible culture heritage dance videos.  In Proceedings of the 1st Workshop on Structuring and Understanding of Multimedia heritAge Contents, pages 75–81, 2019.
  * [50] Jakob Nielsen.  F-shaped pattern for reading web content, April 2006.
  * [51] Jakob Nielsen.  10 usability heuristics for user interface design, April 1994.
  * [52] Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh.  Openpose: Realtime multi-person 2d pose estimation using part affinity fields.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019\.
  * [53] Herwig Rehatschek, Werner Bailer, Helmut Neuschmied, Sandra Ober, and Horst Bischof.  A tool supporting annotation and analysis of videos.  na, 2007.
  * [54] Robert J Hendley, Russell Beale, Chris P Bowers, Christos Georgousopoulos, Charalampos Vassiliou, Petridis Sergios, Ralf Moeller, Eric Karstens, and Dimitris Spiliotopoulos.  Casam: collaborative human-machine annotation of multimedia.  Multimedia tools and applications, 70(2):1277–1308, 2014.
  * [55] Walter S Lasecki, Mitchell Gordon, Danai Koutra, Malte F Jung, Steven P Dow, and Jeffrey P Bigham.  Glance: Rapidly coding behavioral video with the crowd.  In Proceedings of the 27th annual ACM symposium on User interface software and technology, pages 551–562, 2014.
  * [56] Sandra G Hart.  Nasa-task load index (nasa-tlx); 20 years later.  In Proceedings of the human factors and ergonomics society annual meeting, volume 50, pages 904–908. Sage publications Sage CA: Los Angeles, CA, 2006.
  * [57] React – a javascript library for building user interfaces, May 2013.
  * [58] Snehesh Shrestha, Cornelia Fermüller, Tianyu Huang, Pyone Thant Win, Adam Zukerman, Chethan M. Parameshwara, and Yiannis Aloimonos.  AIMusicGuru: Music assisted human pose correction.  arXiv preprint arXiv, 2022.

##  Appendix A Online Resources

Please visit FEVA website http://www.snehesh.com/feva for more information,
code, instruction videos, samples, and any updates.

##  Appendix B List of Video Annotation Tools surveyed

  * •

A Formative Study for Record-time Manual Annotation of First-person Videos

  * •

A multi-level video annotation tool based on XML-dictionaries

  * •

A Semi-Automatic Video Annotation Tool to Generate Ground Truth for
Intelligent Video Surveillance Systems

  * •

Advene

  * •

AIBU

  * •

An innovative web-based collaborative platform for video annotation

  * •

An Ontology Web Application-based Annotation Tool for Intangible Culture
Heritage Dance Videos

  * •

Anvil

  * •

atlas.ti

  * •

Augmented Studio: Projection Mapping on Moving Body for Physiotherapy
Education

  * •

Automatic tagging of video based on voice and localization

  * •

Automatically Freezing Live Video for Annotation during Remote Collaboration

  * •

AVISA: An annotation tool for video understanding

  * •

BeaverDam: Video Annotation Tool for Computer Vision Training Labels

  * •

BEDA: Visual Analysis of Relationships between Behavioral and Physiological
Sensor Data

  * •

CASAM: collaborative human-machine annotation of multimedia

  * •

CLIPPER: Audiovisual Annotation in the Study of Physics

  * •

CoAT: A Web-based, Collaborative Annotation Tool

  * •

CoVidA: Pen-based collaborative video annotation

  * •

Crowd-Guided Ensembles: How Can We Choreograph Crowd Workers for Video
Segmentation?

  * •

CrowdSport: Crowd-based Semantic Event Detection and Video Annotation for
Sports Videos

  * •

DarkLabel

  * •

Demo: Semantic Human Activity Annotation Tool - Using Skeletonized
Surveillance Videos

  * •

EagleView: A Video Analysis Tool for Visualising and Querying Spatial
Interactions of People and Devices

  * •

Elan

  * •

Generating annotations for how-to videos using crowdsourcing

  * •

Glance: rapidly coding behavioral video with the crowd

  * •

HistoryTracker: Minimizing Human Interactions in Baseball Game Annotation

  * •

iSeg: Semi-automatic ground truth annotation in videos: An interactive tool
for polygon-based object annotation and segmentation

  * •

iVAT: An interactive tool for manual, semi-automatic and automatic video
annotation

  * •

LabelMe

  * •

Marquee: a tool for real-time video logging

  * •

MediaDiver: viewing and annotating multi-view video

  * •

MoViA: a mobile video annotation tool

  * •

MRAS: Annotations for streaming video on the web

  * •

Multimodal Video Annotation for Contemporary Dance Creation

  * •

MuLVAT: A Video Annotation Tool Based on XML-Dictionaries and Shot Clustering

  * •

NVivo

  * •

Oudjat is dedicated to the manual annotation facial expressions of
emotion(FEE)

  * •

Redesigning video analysis: an interactive ink annotation tool

  * •

Rethinking Engagement with Online News through Social and Visual Co-Annotation

  * •

Sirio, orione and pan: an integrated web system for ontology-based video
search and annotation

  * •

Stabilized Annotations for Mobile Remote Assistance

  * •

SVAT

  * •

The MESH mobile video annotation tool

  * •

Timelinely

  * •

Tool Eval: Rapid Model-Driven Annotation and Evaluation for Object Detection
in Videos

  * •

ToolScape: Enhancing the Learning Experience of How-to Videos

  * •

VACA: a tool for qualitative video analysis

  * •

VAnnotator: Annotations as multiple perspectives of video content

  * •

VATIC

  * •

VCode and VData: Illustrating a new Framework for Supporting the Video
Annotation Workflow

  * •

VIA: The VIA Annotation Software for Images, Audio and Video

  * •

ViBAT

  * •

VideoAnt

  * •

VideoJot: A Multifunctional Video Annotation Tool

  * •

VidOR: Annotating Objects and Relations in User-Generated Videos

  * •

ViTBAT: Video Tracking and Behavior Annotation Tool

  * •

VoTT

##  Appendix C FEVA Client Architecture

Figure 8: FEVA Client side block diagram of the various modules