[1]Ezequiel França dos Santos

[1]Ph.D. Student in Digital Games Development, IADE-U Creative University,
Lisbon, Portugal

2]Family Physician in Brazil, Lisbon, Portugal

# Apple Vision Pro: Comments in Healthcare

<EMAIL_ADDRESS>Vanessa Dina Palomino Castillo, MD
<EMAIL_ADDRESS>*  [

###### Abstract

This paper objectively analyzes the emerging discourse surrounding Apple
Vision Pro’s application in healthcare and medical education. Released in June
2023, Apple Vision Pro represents a significant advancement in spatial
computing, combining augmented and virtual reality to create new possibilities
in digital interaction. We aim to compile and present recent articles. We used
PubMed, IEEE Xplore, Google Scholar, and JSTOR. Non-academic publications were
excluded. The results were six commentaries, one a pre-print. All were majorly
optimistic, with one mentioning VR/AR sickness. For future research
directions, we stress the need for continued exploration of Apple Vision Pro’s
capabilities and limitations and expect expert opinions to englobe this
discussion.

###### keywords:

Apple Vision Pro, Apple Vision

##  1 Introduction

###  1.1 Context Setting

In the evolving landscape of technology, spatial computing has gradually
merged digital elements with the physical world. One of the recent
developments in this domain is Apple Vision Pro, which was introduced in June
2023[1]. This technology, blending augmented and virtual reality, represents
an incremental advancement in the field, offering new possibilities for user
interaction with digital information.

###  1.2 Purpose of the Discussion

This paper focuses on examining the discourse surrounding the Apple Vision
Pro, especially in the context of healthcare and medical education. We aim to
compile and present recent commentaries’ varied opinions and perspectives. We
will provide a straightforward summary of how this technology might be
beneficial or challenging in these particular fields, striving to maintain an
objective and clear presentation of the discussions.

##  2 Methodology

Using academic databases, including PubMed, IEEE Xplore, Google Scholar, and
JSTOR, we used the keywords: ”Apple Vision Pro” and ”Apple visionOS”.

###  2.1 Criteria for Inclusion and Exclusion of Papers

Inclusion criteria:

  * •

Publications addressing Apple Vision Pro or Apple VisionOS.

  * •

Papers published post-June 2023 for up-to-date relevance.

  * •

Works presenting empirical data, theoretical analysis, or relevant commentary
on the technology.

Exclusion criteria:

  * •

Papers not focusing on Apple Vision Pro or Apple visionOS.

  * •

Non-academic articles and unrelated editorials.

###  2.2 Results

We identified six papers that provided diverse perspectives on Apple Vision
Pro. This selection comprised comment articles, including one pre-print from
arXiv authored by a team with relevant academic expertise. Although the pre-
print was not published, its inclusion was justified by the authors’ knowledge
and insights into Apple Vision Pro’s applications.

Our goal was to cover a range of viewpoints on the technology, ensuring a
balanced representation of current state contributions from recognized
sources.

##  3 Expected Applications of Apple Vision Pro in Healthcare and Medical
Education

The key points from each commentary are summarized in Table 1, offering a
concise overview of the perspectives on Apple Vision Pro’s applications in
these critical fields.

###  3.1 Healthcare: Precision Medicine and Clinical Procedures

Egger et al. (2023): Apple Vision Pro for Healthcare: ”The Ultimate Display”?
– Entering the Wonderland of Precision Medicine. This paper, primarily a
commentary, examines Apple Vision Pro’s potential to revolutionize healthcare,
particularly in precision medicine. The authors detail the device’s
capabilities, such as its Mixed Reality (MR) headset functionality, which
combines Virtual Reality (VR) and Augmented Reality (AR) through Video See-
Through (VST) technology. They discuss its applications in enhancing clinical
procedures, surgical navigation, and patient care. The paper posits that Apple
Vision Pro could enable clinicians to overcome certain limitations of AR in
medicine, such as improving patient interaction and reducing procedural
errors. They mention the problem’s side effects, such as VR/AR sickness, and
how the improvements in computing processing and custom lenses could reduce or
solve that.

###  3.2 Medical Education and Training

Waisberg et al. (2023, Irish Journal of Medical Science): Apple Vision Pro and
why extended reality will revolutionize the future of medicine. In medical
education, this paper, primarily a commentary, explores how Apple Vision Pro’s
extended reality environment could enhance learning and training for medical
professionals. The authors discuss its potential to provide interactive,
immersive educational experiences, such as surgical simulations and
interactive learning modules, which could significantly improve medical
education and training methodologies. This article does not comment on VR/AR
sickness.

###  3.3 Ophthalmology and Vision Science

Waisberg et al. (2023, Eye): The future of ophthalmology and vision science
with the Apple Vision Pro. This study, primarily a commentary, delves into the
transformative potential of Apple Vision Pro in ophthalmology and vision
science. It discusses how the device’s advanced eye-tracking and high-
resolution displays can aid in precise vision screening and diagnostics. The
authors envision applications in patient education and immersive diagnostics,
suggesting that Apple Vision Pro could significantly impact how eye care
professionals diagnose and treat vision-related conditions. This article does
not comment on VR/AR sickness.

###  3.4 Enhancing Accessibility in Healthcare

Masalkhi et al. (2023, Annals of Biomedical Engineering): Apple Vision Pro for
Ophthalmology and Medicine. Masalkhi and colleagues explore, primarily as a
commentary, the potential of Apple Vision Pro in enhancing accessibility,
particularly for individuals with visual impairments. The paper explores the
device’s technical capabilities, particularly its superior display features.
It discusses how these features can enhance visual experiences and
accessibility in healthcare settings. The authors envision the Apple Vision
Pro as a tool that could offer groundbreaking solutions in vision screening
and assistance, opening new possibilities for inclusive care in ophthalmology.
This article does not comment on VR/AR sickness.

###  3.5 Psycological Research

Barrett et al. (2023, Behaviour and Information Technology): Exploring the
influence of audience familiarity on speaker anxiety and performance in
virtual reality and real-life presentation contexts. This research, primarily
a commentary, investigates the utilization of Apple Vision Pro in the context
of psychological research and professional development, particularly in
enhancing public speaking skills. The study highlights how virtual
environments created through the device can influence performance and anxiety
in public speaking. By simulating various audience types, the paper suggests
that Apple Vision Pro can effectively train individuals in communication
skills, offering a safe and controlled environment to practice and refine
public speaking abilities. This article does not comment on VR/AR sickness.

###  3.6 Advancement in Medical Education

Waisberg et al. (2023, Canadian Medical Education Journal): Apple Vision Pro
and the advancement of medical education with extended reality. This work,
primarily a commentary, explores the transformative impact of the Apple Vision
Pro on medical education. The authors discuss how extended reality, a
combination of virtual and augmented reality technologies, can revolutionize
traditional learning methods in the medical field. They highlight the
potential for immersive and interactive experiences that enhance understanding
of complex medical conditions, improve diagnostic skills, and provide
realistic simulations for medical training. The paper emphasizes the role of
this technology in fostering a more profound, more practical learning
experience for medical students and professionals, potentially leading to
improved patient care outcomes. This article does not comment on VR/AR
sickness.

Table 1: Summary of Results Author |  Area |  Application |  Expected Improvements
---|---|---|---
Egger et al. (2023) |  Healthcare |  Clinical procedures, patient care |  Enhanced surgical navigation, improved patient interaction
Waisberg et al. (2023, Eye) |  Ophthalmology |  Vision screening, diagnostics |  Precise diagnostics, improved patient education
Waisberg et al. (2023, Irish Journal) |  Medical Education |  Training for medical professionals |  Interactive learning, improved training methodologies
Masalkhi et al. (2023) |  Healthcare Accessibility |  Enhanced visual experiences for the visually impaired |  Improved accessibility in healthcare settings
Barrett et al. (2023) |  Psychological Research |  VR-based emotional and behavioral studies |  Enhanced research capabilities, realistic 3D rendering
Waisberg et al. (2023, Canadian Journal) |  Medical Education |  Advanced medical training |  Immersive educational experiences, enhanced diagnostic skills

##  4 Discussion

###  4.1 Challenges and Future Directions

While the papers reviewed paint an optimistic picture of Apple Vision Pro’s
applications [2] [3] [4] [5] [6] [7], they also acknowledge various challenges
and future directions. These include integrating the technology seamlessly
into existing systems (especially in healthcare), addressing privacy and data
security concerns, and ensuring the device’s accessibility and user
adaptability (e.g., in a clinical setting [5]). The current high price point
could limit accessibility, especially in resource-constrained healthcare
settings or educational institutions, although in some settings, this might
not be an issue[2].

###  4.2 VR/AR Side Effects

An essential aspect missing in most commentaries is the discussion on VR/AR
side effects, such as virtual reality sickness, which can manifest as nausea,
disorientation, and headaches. While these technologies promise revolutionary
changes, addressing their potential impact on user health and comfort is
crucial, especially in prolonged or intense usage scenarios. However, the
battery capacity does not allow extended usage periods since they endure more
or less two hours[2].

The pre-print article from arXiv[2] provides some insights into these issues,
but there is a clear need for more comprehensive research regarding Apple
Vision Pro use cases.

###  4.3 Apple Vision Pro Health Advisory

In a report dated January 19, 2024, Apple Inc. issued a health advisory for
users of Apple Vision Pro, particularly those with certain medical conditions
such as migraines or chronic headaches, dizziness or vertigo, eye or vision
conditions, such as binocular vision conditions, psychological conditions,
inner ear conditions, history of dry eyes, itchiness, or swelling of the
eyelids, skin allergies or sensitivities, seizures, balance or gait
conditions[8]. The company recommends these individuals consult with a
healthcare provider before using the device. Also, updated acknolegement to
when to stop using Apple Vision Pro: if you experience symptoms related to a
medical condition; if your medical provider confirmed that it’s safe for you
to use Apple Vision Pro, but you experience severe or persistent physical
discomfort, motion sickness, visual discomfort, skin irritation, or
psychological symptoms; if you notice swelling, itchiness, skin irritation, or
other skin reactions while using your device or after using the device[8].
Always, if symptoms persist, consult with your medical provider[8].

This advisory highlights the importance of considering individual health
conditions when using advanced technological devices in medical settings.

##  5 Final Thoughts

The potential applications of Apple Vision Pro are significant and promising.
Besides the lack of formal publications due to waiting for the Apple Vision
Pro release, the experts’ comments envision an optimistic scenario.

Future research should explore the technology’s benefits and drawbacks,
including ergonomic concerns and potential health effects like VR/AR sickness.
Understanding these dimensions will be pivotal in developing innovative and
considerate solutions for user well-being.

As we navigate the advancements in spatial computing, we must keep a balanced
view, acknowledging the opportunities and challenges presented by technologies
like Apple Vision Pro. More research and examination will guide the technology
toward a path that maximizes benefits while minimizing risks.

## Acknowledgment

The authors declare that there is no conflict of interest regarding the
publication of this paper. This research did not receive any specific grant
from funding agencies in the public, commercial, or not-for-profit sectors.

## References

  *   * Apple [2023] Apple, P.R.: Introducing Apple Vision Pro: Apple’s first spatial computer — apple.com. https://www.apple.com/newsroom/2023/06/introducing-apple-vision-pro/. [Accessed 13-01-2024] (2023)
  * Egger et al. [2023] Egger, J., Gsaxner, C., Chen, X., Bian, J., Kleesiek, J., Puladi, B.: Apple vision pro for healthcare: ”the ultimate display”? – entering the wonderland of precision medicine (2023)
  * Waisberg et al. [2023a] Waisberg, E., Ong, J., Masalkhi, M., Zaman, N., Sarker, P., Lee, A.G., Tavakkoli, A.: Apple vision pro and why extended reality will revolutionize the future of medicine. Irish Journal of Medical Science (1971 -) (2023) https://doi.org/10.1007/s11845-023-03437-z
  * Waisberg et al. [2023b] Waisberg, E., Ong, J., Masalkhi, M., Zaman, N., Sarker, P., Lee, A.G., Tavakkoli, A.: The future of ophthalmology and vision science with the Apple Vision Pro. Springer (2023). https://doi.org/10.1038/s41433-023-02688-5
  * Masalkhi et al. [2023] Masalkhi, M., Waisberg, E., Ong, J., Zaman, N., Sarker, P., Lee, A.G., Tavakkoli, A.: Apple Vision Pro for Ophthalmology and Medicine. Springer (2023). https://doi.org/10.1007/s10439-023-03283-1
  * Barrett et al. [2023] Barrett, A., Pack, A., Monteiro, D., Liang, H.N.: Exploring the influence of audience familiarity on speaker anxiety and performance in virtual reality and real-life presentation contexts. Behaviour and Information Technology (2023) https://doi.org/10.1080/0144929X.2023.2186145
  * Waisberg et al. [2023] Waisberg, E., Ong, J., Masalkhi, M., Zaman, N., Sarker, P., Lee, A.G., Tavakkoli, A.: Apple vision pro and the advancement of medical education with extended reality. Canadian Medical Education Journal (2023) https://doi.org/10.36834/cmej.77634
  * Apple Inc. [2024] Apple Inc.: Using Apple Vision Pro with certain medical conditions. https://support.apple.com/en-us/HT213935. [Accessed: 19-01-2024] (2024)


# OpusCleaner and OpusTrainer, open source toolkits for training Machine
Translation and Large language models

Nikolay Bogoychev
School of Informatics
University of Edinburgh
<EMAIL_ADDRESS>
Jelmer van der Linde
School of Informatics
University of Edinburgh
<EMAIL_ADDRESS>
Graeme Nail
School of Informatics
University of Edinburgh
<EMAIL_ADDRESS>
Barry Haddow
School of Informatics
University of Edinburgh
<EMAIL_ADDRESS>
Jaume Zaragoza-Bernabeu
Prompsit Language Engineering
SL (PLE), Spain
<EMAIL_ADDRESS>
Gema Ramírez-Sánchez
Prompsit Language Engineering
SL (PLE), Spain
<EMAIL_ADDRESS>
Lukas Weymann
Prompsit Language Engineering
SL (PLE), Spain
<EMAIL_ADDRESS>
Tudor Nicolae Mateiu
Prompsit Language Engineering
SL (PLE), Spain
<EMAIL_ADDRESS>
Jindřich Helcl
Faculty of Mathematics and Physics
Charles University, Czechia
<EMAIL_ADDRESS>
Mikko Aulamo
Department of Digital Humanities
Language Technology
University of Helsinki, Finland
<EMAIL_ADDRESS>
###### Abstract

Developing high quality machine translation systems is a labour intensive,
challenging and confusing process for newcomers to the field. We present a
pair of tools OpusCleaner and OpusTrainer that aim to simplify the process,
reduce the amount of work and lower the entry barrier for newcomers.

OpusCleaner is a data downloading, cleaning, and proprocessing toolkit. It is
designed to allow researchers to quickly download, visualise and preprocess
bilingual (or monolingual) data that comes from many different sources, each
of them with different quality, issues, and unique filtering/preprocessing
requirements.

OpusTrainer is a data scheduling and data augmenting tool aimed at building
large scale, robust machine translation systems and large language models. It
features deterministic data mixing from many different sources, on-the-fly
data augmentation and more.

Using these tools, we showcase how we can use it to create high quality
machine translation model robust to noisy user input; multilingual models and
terminology aware models.

##  1 Introduction

Machine translation is ubiquitous in modern society, however training high
quality machine translation systems is not trivial. A lot of the knowledge
about how to build high quality systems is not well defined, comes from
experience and at times may seem counter intuitive. With OpusTrainer and
OpusCleaner we aim to explicitly address the main challenges in a user
friendly manner and simplify the workload for machine translation researchers.

There are several challenges when it comes to building high quality MT
systems:

###  1.1 Data Sources

Parallel data for machine translation systems comes from many different
sources, that have widely varying quality. As an example, using Opus’s
website111https://opus.nlpl.eu/, and filtering parallel data sources for
Chinese to English, we are presented with a dozen different corpora. Here we
find that some are in traditional script, others are in simplified script, and
these may or may not have been tokenized. This is before noting any language
identification issues. In order to build a high quality translation system, we
need to first have quality data, which necessarily means auditing each corpus
manually and then deciding how to preprocess it.

###  1.2 Training schedule

High quality machine translation systems require the use of backtranslation
(Sennrich et al., 2016), usually included in the form of pretraining. Often at
the end of training models are fine tuned to in-domain data. Without a
training scheduler that supports different training stages, start-and-stop
training approach is necessary which presents challenge for automation and
increases the burden on the researcher.

###  1.3 Data Mixing

Noisy web-crawled data is useful for translation quality, but including it too
early in the training may lead to model divergence. Furthermore dirty data is
orders of magnitude more available than clean manually curated parallel data.
Without any upsampling, clean data might be overshadowed by dirty data, but
upsampling is wasteful in terms of disk space. Finally, multilingual models
require careful data mixing such that low resource language languages are not
overwhelmed by high resource ones, without a training scheduler that supports
data source mixing, this is achieved by upsampling low resource data and
carefully mixing and shuffling it in the training data.

###  1.4 Data Augmentation

Machine translation models are training on sanitised parallel data that is
usually not representative of noisy user input:

  * •

Typos are quite rare in clean data, and spellchecker is often used on web-
crawled data.

  * •

All caps and title-case text are often missing.

  * •

Emoji are basically non existent in parallel data.

  * •

Models are not trained to cope with untranslatable tokens, which should be
copied between the source and the target language.

OpusTrainer and OpusCleaner are designed to resolve the above issues, and make
it easy for a novice user to build high quality translation systems, by
explicitly setting the expectations that training data must be carefully
audited, and training data must be scheduled.

##  2 OpusCleaner

In order to address the daunting task of data cleaning, we developed
OpusCleaner, a single graphical frontend that does data downloading and
cleaning, while being modular to allow for custom modifications depending on
the language pair in question. We show screenshots of the welcome screen on
Figure 1.

Figure 1: Initial screen of OpusCleaner

###  2.1 Data Download

OpusCleaner provides seamless integration with MTData (Gowda et al., 2021) as
shown on Figures 2 and 3. Datasets can be searched by languages and then
downloaded individually, or in bulk. Basic information about each dataset
(number of lines, version, size) are shown, as well as link to the dataset
description page in Opus.

Additionally, adding one’s own custom datasets is possible.

Figure 2: Search dataset pane Figure 3: Search and download dataset with links
to dataset and basic information.

###  2.2 Data Cleaning

Once all datasets are acquired, we can navigate to the Data Tailor screen
(Figure 4) where we can label every dataset with an arbitrary label (Such as
medium or dirty) so that we can keep track of the overall quality of each
dataset.

Figure 4: Initial screen of data tailoring, as well as dataset labelling.

####  2.2.1 Filter and preview

For each dataset, we visualise a sample of 3000 sentences that includes the
first 100, the last 100 and random lines in between. From this window we can
identidy the idiosyncraticies of that dataset and add the appropriate filters
to fix them. For example, if we spot that some lines are in the wrong language
(Figure 5) we can add language identifier filter and see the result of it in
the preview window (Figure 6).

Figure 5: Initial view of dataset cleaning with some sentences obviously in
the wrong language. Figure 6: Fasttext langid filter removes lines in wrong
language.

Another example is finding mismatched punctuation on the source and the target
(Figure 7). We can then create a simple filter that fixes the issue and apply
it, see the result (Figure 8).

Figure 7: Mismatched punctuation on the source and the target. Figure 8:
Fixing mismatched punctuation.

####  2.2.2 Filters and pipelines

OpusCleaner is designed to clean data in a pipelined manner. Multiple filters
are chained where every filter receives data on stdin and outputs it on
stdout. OpusCleaner itself takes care of managing the pipeline. A typical
pipeline would have a number of filters chained up as shown on Figure 9.

Figure 9: Adding multiple filters and visualising the difference.

We support 28 built in filters with custom user filters supported by simply
providing a json configuration file that specifies path to filter executable
and optionally what arguments it should have.

####  2.2.3 Processsing all data

Once we have determined filters for every single downloaded dataset, we run a
command line utility that does batch processing of all datasets, taking care
of also cutting up files and parallelising processing. Once all processing is
done, we provide an utility to deduplicate the data but preserving the split
of datasets and then the user can proceed with training the machine
translation system.

OpusCleaner222https://github.com/hplt-project/OpusCleaner is open source,
under active development and available for free for anyone to use.

##  3 OpusTrainer

As discussed in section 1, training high quality machine translation systems
requires carefully combining parallel data from different sources and quality
levels; applying on the fly modifications to it and more.

This is challenging to achieve with neural network toolkits that make use of
static training data, because ideally we want to modify the data mixture and
potentially augment it on the fly, without having to prepare the data first
and write it to disk which is wasteful.

##### Multilingual model training

The problem is exacerbated when training many-to-many or English-to-many
multilingual models where high resource languages would often have orders of
magnitude more data than low resource languages. In order for a multilingual
model to train well in this setting, it needs to see balanced data from all
languages (Freitag and Firat, 2020). Doing this by concatenating and
upsampling data (in order to get equal amounts of data seen for all
languages), would waste multiple terabytes of disk space.

###  3.1 Data Scheduling

OpusTrainer solves this problem by streaming and mixing data from multiple
sources. OpusTrainer uses a simple yaml configuration file where the user can
declare all of their data sources and a desired mix of them for different
stages of training. OpusTrainer then reads in the data from different sources
and then outputs the desired mix to stdout. OpusTrainer is meant to be used
with neural network toolkits that support reading data from stdin such as
Marian (Junczys-Dowmunt et al., 2018), but it can also output the desired data
mix to a file, making it usable with all toolkits. An example configuration
that describes a full training run with various data mixings for different
stages of training can be seen on Figure 10.

Figure 10: OpusTrainer basic configuration defining the data scheduling for
training a model.

###  3.2 Data Augmentation

Humans are very robust to decoding noisy texts, but this can pose a major
challenge to machine translation systems due to the way we collect our
training data:

  * •

Title Case and Upper Case parallel data is quite rare in training data, and is
sometimes regularised during acquisition.

  * •

Typos are also comparatively rare in training data, because either we use
clean sources or we perform spellchecking on web crawled sources.

  * •

Emojis, which human readers expect to be copied over from the source to the
target, are not seen during training, because typically lines containing
emojis are removed from the training data at preprocessing steps.

In order to alleviate these issues, OpusTrainer provides multiple data
modifiers which can be applied on the fly, at random on the training data:

  * •

UpperCaser and TitleCaser

  * •

Typo modifier, which inserts typos in words during training

  * •

Merge modifier, which randomly merges several input sentences together to help
the model be more robust to longer sentences.

  * •

Noise modifier, that generates random sentences consisting of unicode noise,
identical on both the source and the target side. This modifier teaches the
model to copy unknown strings to the target side.

  * •

Inline Noise modifier: A more complicated version of the above that uses word
alignments in order to inject noisy unicode characters (including Emoji) in
approximately the same logical place on both the source and the target side.
This modifier teaches the model that unknown sequences of <unk> characters
should be just copied on the target side.

All of those modifiers are applied to each sentence in the training data with
a user defined probability as shown on Figure 11.

Figure 11: Different modifiers specified in YAML format to be used during
training.

###  3.3 Terminology

OpusTrainer is able to leverage word alignment information to produce
terminology augmented systems, precisely as the one described in Bogoychev and
Chen (2023). This is achieved by finding bijective word alignment mappings
between the source and the target sentences and at randomly injecting
terminology hints in the source, precisely like the one show on 12.

Where is the airport? $\leftrightarrow$ Wo ist der Flughafen?
Where is the airport __target__ Flughafen __done__? $\leftrightarrow$ Wo ist
der Flughafen?

Figure 12: Terminology augmentation in practise. During training it is hinted
that the target word Flughafen corresponds to Airport, so that at inference
when providing the model with terminology hints it will know how to
incorporate them at the output.

These terminology hints can then be used at inference time, and the model will
know how to incorporate the desired terminology hint at the target side. The
relevant training options are shown on figure 13

Figure 13: Tag modifier is used to add terminology hints to the source during
training. Values of 3% to 7% seem to work well in practise.

OpusTrainer is open source and available on GitHub,333https://github.com/hplt-
project/OpusTrainer with ample documentation and examples. OpusTrainer is
designed to be used mainly with neural network toolkits that read in training
input on stdin, as it takes care of shuffling between epochs, resuming
training and all other functions normally done by the data module of a neural
network toolkit. It can, however, also be used to write a preprocessed
training corpus on disk so toolkits that do not support reading stdin can also
make use of it.

##  4 Case study: A Robust French-English system

We highlight the use cases of data augmentation by using OpusCleaner and
OpusTrainer to train a French-English machine translation system. We define
robustness as the following criteria, which are all common concerns for real
world web text.

  * •

Accurate translation of URLs (URLs need to be copied to the target side
without any modification).

  * •

Accurate copy behaviour on OOV tokens such as emoji or snippets of foreign
language texts. The latter often occur in wikipedia, where foreign language
terms such as named entities appear alongside their local language
transliteration.

  * •

No quality loss when translating Upper Case and Title case texts compared to
normal cased text (All caps and tittle case often appear in tittles of
newspapers).

  * •

Robustness to typos (social media users).

###  4.1 Test set design

As a baseline test set we use newstest15 and we make several version of it to
more accurately measure robustness.

  * •

Title Case version of the test set

  * •

All caps version of the test set

  * •

Typo-ed version of the test set, where we insert 4 typos in each line using
the python’s typo library.444https://pypi.org/project/typo/

  * •

Emoji augmented test set where we insert random emoji in corresponding places
on the source and the target, by using precomputed word alignments in order to
place the emoji in both texts in the correct corresponding location. Example
on figure 14.

  * •

Random unicode sequence augmented test set where the random unicode sequences
are inserted in the same manner as the emoji. Example on figure 14.

Figure 14: Example cases of noise/emoji inside the source and the
corresponding target translation. We aim for our model to be able to reproduce
those at decode time.

On top of that we prepare a dataset of sentences containing URLs from the
paracrawl project. We take sentences containing exactly the same URLs on both
the source and the target, then we remove the URLs and take the top 1500
sentences according to their bicleaner-ai ("Zaragoza-Bernabeu et al., "2022")
score and reinsert the URLs.

For quality we report BLEU, but we also use several specific metrics. For the
URL test set we measure the percentage of exact matches of URLs. For datasets
with tittle case and all caps we measure as well BLEU-uncased to see how good
translation quality is, regardless of the case outputted. Finally, for
datasets with emoji and unicode sequences, we extract all of the OOV
characters and measure ChrF (Popović, 2015) on them only, so that we can see
how effective our system is at copying them to the target side.

###  4.2 Model

For training data we use all of the available French-English data accessible
through MTData (Gowda et al., 2021) and we clean it using OpusCleaner.

We split the data into four categories based on its providence and subjective
perceived quality through manual inspection:

  * •

Canonically clean datasets such as Europarl, Un are designated as clean (22M
parallel sentences).

  * •

Slightly less clean data (9M), designated as cleanish.

  * •

Not clean data, but not generated from crawled sources (16M), designated as
medium.

  * •

Web crawled data is designated as dirty (363M)

We use Marian (Junczys-Dowmunt et al., 2018) to train transformer-big Vaswani
et al. (2017) models on the training data with varying degree of data
augmentation. We train 7 different models with various additional perks, some
related to data augmentation, some not in order to show how we progressively
achieve a more robust model.

  1. 1.

Pure model

  2. 2.

\+ Sentencepiece sampling (Kudo and Richardson, 2018). Sentencepiece sampling
makes splits of words non-deterministic, potentially making unseen words
handling more robust.

  3. 3.

\+ UpperCase and LowerCase

  4. 4.

\+ typos

  5. 5.

\+ Unicode Vocabulary Fallback. Sentencepiece models can’t split OOV tokens
such as Chinese characters into subwords, but if we consider that every
character is represented by unicode bytes, we can split unseen characters such
as emoji and hanzi

  6. 6.

\+ noisy sentences

  7. 7.

\+ inline noise

Hex

###  4.3 Results

We present our results on table 1. We train 7 different systems with different
degrees of augmentation. We can see that progressively, as we add more
modifiers to the training set up, the model becomes more robust to various
sources of noisy user input. System 3 onwards have capture TitleCase and
UpperCase with relatively small performance loss compared to plain sentences.
System 5 that uses UTF fallback for OOV tokens starts capturing emoji and
other OOV tokens. Systems 6 and 7 enhance the training data with lots of noisy
examples and that leads to really good copy rate of OOV tokens to the target
side, as shown in the two ChrF columns.

| newstest15 BLEU |  |
---|---|---|---
| plain |  | TC
---
uncased
TC |  | CAPS
---
uncased
CAPS | typo | noise |  |  noise1
---
chrf
emoji |  |  emoji1
---
chrf
| url
---
BLEU
| URL only
---
precision
baseline (1) | 40 | 34.2 | 8.6 | 21.5 | 20.5 | 29.6 | 34.3 | 0 | 35.8 | 0.1 | 62.7 | 90%
\+ spm sample (2) | 39 | 36.9 | 9.1 | 29.2 | 21.2 | 30.5 | 33.4 | 0.1 | 34.7 | 0.2 | 61.4 | 87%
\+ UC/LC noise (3) | 38.4 | 37.3 | 36.3 | 34.5 | 34.5 | 29.7 | 32.9 | 0.1 | 34.3 | 0.2 | 60.9 | 87%
\+ typos (4) | 38.9 | 38 | 36.8 | 35.1 | 35.1 | 36.7 | 33.5 | 0.1 | 34.2 | 5.2 | 61.2 | 86%
\+ UTF-8 fallback (5) | 38.5 | 38 | 36.8 | 34.7 | 34.7 | 36.8 | 35.2 | 55.1 | 37 | 64.9 | 61 | 85%
\+ noise (6) | 39.6 | 39.1 | 37.9 | 35.9 | 35.9 | 37.6 | 38.9 | 87 | 38.7 | 72.3 | 61.3 | 86%
\+ inline noise (7) | 39.2 | 38.3 | 37.2 | 35.3 | 35.3 | 37.5 | 41.5 | 92 | 39.9 | 80.7 | 61.2 | 86%
Table 1: Results table
1 ChrF score was calculated on the noise/emoji only, meaning we only measure
how well our model copies just OOV tokens without considering translation
quality.

####  4.3.1 Caveats

There are some caveats that come with our test results. The more modifiers are
used, the more difficult the training data seems to be to model, and therefore
it takes more iterations through the training data to achieve convergence.
Therefore all models presented have seen different amounts of training data.
We will control for this setting in future work.

Furthermore we see slight degradation in terms of translation quality when we
add modifications to the training data on the plain test set. This suggests
that the gains we have are not entirely for free. Finally, we observe slight
deterioration on URLs. We measure only exact matches on URLs because an almost
correct URL is not useful. This regression bodes for further investigation.

##  5 Conclusion

We present a feature complete data preprocessing and data scheduling toolkit
for training machine translation systems (but also just as useful for Large
Language Models). Our tools are designed with novice and experts in mind so
that they lower the entry barrier to the field of machine translation, while
still allowing for state of the art results. Our data augmentation utilities
are crucial for producing robust machine translation systems, as well as
terminology systems (Bogoychev and Chen, 2023). Our toolkit was developed
concurrently and independently to Sotastream (Post et al., 2023) and provides
similar functionality.

## References

  * Bogoychev and Chen [2023] Nikolay Bogoychev and Pinzhen Chen.  Terminology-aware translation with constrained decoding and large language model prompting, 2023.
  * Freitag and Firat [2020] Markus Freitag and Orhan Firat.  Complete multilingual neural machine translation.  In Loïc Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Yvette Graham, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, and Matteo Negri, editors, _Proceedings of the Fifth Conference on Machine Translation_ , pages 550–560, Online, November 2020. Association for Computational Linguistics.  URL https://aclanthology.org/2020.wmt-1.66.
  * Gowda et al. [2021] Thamme Gowda, Zhao Zhang, Chris Mattmann, and Jonathan May.  Many-to-English machine translation tools, data, and pretrained models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_ , pages 306–316, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-demo.37.  URL https://aclanthology.org/2021.acl-demo.37.
  * Junczys-Dowmunt et al. [2018] Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch.  Marian: Fast neural machine translation in C++.  In _Proceedings of ACL 2018, System Demonstrations_ , pages 116–121, Melbourne, Australia, July 2018. Association for Computational Linguistics.  URL http://www.aclweb.org/anthology/P18-4020.
  * Kudo and Richardson [2018] Taku Kudo and John Richardson.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71, Brussels, Belgium, November 2018. Association for Computational Linguistics.  doi: 10.18653/v1/D18-2012.  URL https://aclanthology.org/D18-2012.
  * Popović [2015] Maja Popović.  chrF: character n-gram F-score for automatic MT evaluation.  In Ondřej Bojar, Rajan Chatterjee, Christian Federmann, Barry Haddow, Chris Hokamp, Matthias Huck, Varvara Logacheva, and Pavel Pecina, editors, _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 392–395, Lisbon, Portugal, September 2015. Association for Computational Linguistics.  doi: 10.18653/v1/W15-3049.  URL https://aclanthology.org/W15-3049.
  * Post et al. [2023] Matt Post, Thamme Gowda, Roman Grundkiewicz, Huda Khayrallah, Rohit Jain, and Marcin Junczys-Dowmunt.  Sotastream: A streaming approach to machine translation training, 2023.
  * Sennrich et al. [2016] Rico Sennrich, Barry Haddow, and Alexandra Birch.  Improving neural machine translation models with monolingual data.  In Katrin Erk and Noah A. Smith, editors, _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 86–96, Berlin, Germany, August 2016. Association for Computational Linguistics.  doi: 10.18653/v1/P16-1009.  URL https://aclanthology.org/P16-1009.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, page 6000–6010, Red Hook, NY, USA, 2017. Curran Associates Inc.  ISBN 9781510860964.
  * "Zaragoza-Bernabeu et al. ["2022"] Jaume "Zaragoza-Bernabeu, Gema Ramírez-Sánchez, Marta Bañón, and Sergio" Ortiz Rojas.  "bicleaner AI: Bicleaner goes neural".  In _"Proceedings of the Thirteenth Language Resources and Evaluation Conference"_ , pages "824–831", "Marseille, France", June "2022". "European Language Resources Association".  URL "https://aclanthology.org/2022.lrec-1.87".


# Phase Space Reconstruction Network for
Lane Intrusion Action Recognition ††thanks: $\ast$ Corresponding author:
Zhidong Deng.

Ruiwen Zhang, $\ast$Zhidong Deng, Hongsen Lin, Hongchao Lu  Institute for
Artificial Intelligence at Tsinghua University (THUAI)
State Key Laboratory of Intelligent Technology and Systems
Beijing National Research Center for Information Science and Technology
(BNRist)
Center for Intelligent Connected Vehicles and Transportation
Department of Computer Sicence and Technology, Tsinghua University, Beijing
100084, China
{zhangrw18, lin-hs20<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In a complex road traffic scene, illegal lane intrusion of pedestrians or
cyclists constitutes one of the main safety challenges in autonomous driving
application. In this paper, we propose a novel object-level phase space
reconstruction network (PSRNet) for motion time series classification, aiming
to recognize lane intrusion actions that occur 150m ahead through a monocular
camera fixed on moving vehicle. In the PSRNet, the movement of pedestrians and
cyclists, specifically viewed as an observable object-level dynamic process,
can be reconstructed as trajectories of state vectors in a latent phase space
and further characterized by a learnable Lyapunov exponent-like classifier
that indicates discrimination in terms of average exponential divergence of
state trajectories. Additionally, in order to first transform video inputs
into one-dimensional motion time series of each object, a lane width
normalization based on visual object tracking-by-detection is presented.
Extensive experiments are conducted on the THU-IntrudBehavior dataset
collected from real urban roads. The results show that our PSRNet could reach
the best accuracy of 98.0%, which remarkably exceeds existing action
recognition approaches by more than 30%.

###### Index Terms:

visual artificial intelligence, action recognition, phase space
reconstruction, autonomous driving

##  I Introduction

Figure 1: (a) Raw video frames. (b) Optical flow visualization of raw frames.
(c) Visual object detection results in raw frames. Lane intrusion actions are
only dependent on positions of moving objects $o$ (i.e., pedestrians and
cyclists) relative to central trajectory of current lanes constituted by
$l_{1}$ and $l_{2}$. (d) Resulting motion time series of each object. (e)
Normalized relative motion time series.

In recent years, autonomous driving has become a hot and promising area in
industry and academic circles. Although most of moving objects are motor
vehicles in highway scenarios, there occasionally exist some unusual
emergencies, including illegal lane intrusion of pedestrians or cyclists,
which actually constitutes one of the main safety challenges in autonomous
driving application. Focusing on this problem, we aim to recognize lane
intrusion actions that occur at least 150m ahead solely based on a monocular
camera fixed on vehicles moving at high speed.

Currently, there are many trajectory prediction methods and datasets [1, 2, 3,
4, 5] based on fixed camera. However, those methods usually do not work well
while camera is moving. It is hard to determine relative location of objects
because background changes for moving camera. Video action recognition models
[6, 7, 8, 9, 10, 11, 12] can simultaneously have extraction of spatial and
temporal features in video frames and are also exploited to recognize actions.
But unfortunately, the above-mentioned approaches are often well suited to
low-speed and close-range situations, which implies that camera is fixed or
moves slowly, and particularly moving object often occupies a large proportion
of a pixel plane in a video frame.

This paper mainly investigates high-speed and long-range scenarios in a
structured road traffic environment. In such a situation, on-board camera
moves forward at high speed with ego-vehicle, and moving objects ahead on road
usually occur about 150-250m away in front of camera. Undoubtedly, it will
bring new big challenges, such as inconsistent visual changes of moving
objects in different video frames and varying view-angles caused by vehicle
steering. Interestingly, it is readily observed from Fig. 1(b) that optical
flow may represent the motion of vehicles instead of cyclists. All the
challenges mean that pixel changes in consecutive video frames are likely to
be unable to truly reflect actual motion of all objects. Owing to the fact
that moving objects frequently occupy a small proportion of a pixel plane in a
single video frame due to being far from camera, it is very difficult to
detect such tiny moving objects and recognize their actions accurately. From
this perspective, most of the methods proposed in the past are tough to solve
intrusion action recognition problems.

For high-speed and long-range scenes, this paper first presents an object-
level lane width normalization strategy to transform lane intrusion actions of
objects including pedestrians or cyclists in video frames into relative motion
time series of objects. Through normalizing relative positions of objects with
pixel-level lane widths at current video frame, we can thus give rise to one-
dimensional motion time series of objects, which could be employed to address
the challenge caused by moving camera, i.e., pixel changes in video frames may
be unable to reflect actual actions of objects. Second, we propose a PSRNet to
classify the above motion time series and then eventually complete lane
intrusion action recognitions. Inspired by prediction methods of phase-space
reconstruction, we make use of several convolutional layers in the PSRNet to
reconstruct latent phase spaces of dynamic system embedded in observable
motion time series, which actually expresses the motion of object trajectories
in a latent state space. Following this, we set up a learnable Lyapunov
exponent-like classifier in our PSRNet, in order to learn invariant
exponential divergence characteristics in a state space for different motion
time series or dynamical processes. Note that our method does not need to use
either any navigation equipments of ego-vehicle or any high-definition maps.
Finally, we conduct extensive experiments on the lane intrusion action
recognition dataset called THU-IntrudBehavior, which is collected from real
urban roads. The experimental results show that the PSRNet reaches the best
accuracy of 98.0%, which remarkably surpasses classical action recognition
approaches by more than 30%. In short, the main contributions in this paper
are summarized as follows:

  1. 1.

Based on monocular tracking-by-detection algorithms, we present an object-
level lane width normalization strategy such that lane intrusion actions of
objects in consecutive video frames can be represented by relative motion time
series accordingly.

  2. 2.

A novel lightweight PSRNet that comprises both latent phase space
reconstruction layers and a learnable Lyapunov exponent-like classifier is
proposed to accomplish classifications of such relative motion time series.

  3. 3.

Experimental results yielded on THU-IntrudBehavior demonstrate that our PSRNet
achieves the best performance of 98.0%.

##  II Related Work

###  II-A CNN-based Video Action Recognition Approaches

In 2014, CNN was at the first time employed for large-scale video
classification [6]. In order to learn both spatial and temporal features, D.
Tran, _et al_. present C3D [13] through using 3D convolution kernels. In the
LSTM+CNN approach [14], CNN is first utilized to capture spatial features and
LSTM is then exploited for temporal feature extraction. In 2016, Feichtenhofer
_et al_. propose a two-stream fusion network [15], where several fusion
algorithms are employed to learn corresponding features between pixels. In
[16], TSN divides video into $k$ parts, then selects those parts randomly, and
fuse them by two-stream networks, which solves the problem of being unable to
capture long-term features. I3D is presented in 2017 [17] to expand 2D
convolution kernels into 3D ones and then takes advantage of pre-trained
models of image data, which significantly improves performance. But the
network architecture must be completely consistent with 2D CNN. In 2018,
R(2+1)D [10] factorizes 3D convolution filters into separate spatial and
temporal components yields and consequently it indeed reduces the amount of
parameters. Furthermore, Zhou _et al_. propose TRN [11] that randomly chooses
2, 3, and 4 frames and then employs MLPs to extract spatial features of
images. After making fusion of them, it is capable of having extraction of
temporal features more effectively. Note that TRN is usually only suitable for
actions with a strong direction. SlowFast proposed by Feichtenhofer _et al_.
in 2019 [9] contains a high-resolution slow network and a low-resolution fast
network. Those two branches can capture local features and global features,
respectively. Meanwhile, TSM [8] for video understanding is presented. In
2020, Yang _et al_. [12] propose a temporal pyramid network that the source of
features and the fusion of features jointly form a feature hierarchy for the
backbone so that it can capture action instances at various tempos.

Figure 2: The overall framework of our method.

###  II-B Time Series Prediction Based on Deep Learning

Recurrent neural networks (RNNs) can handle temporal sequence data. As one of
the most representative models in RNN, LSTM [18] has competitive performance
in long-term dependencies. GRU [19] is a variant of LSTM, which achieves the
same performance as LSTM, but it is easier to conduct calculation. TCN [20] is
just a temporal CNN model that uses both 1D causal convolution and dilated
convolution and is combined with residual networks, which can treat sequence
data in parallel and adjust receptive fields flexibly. Social GAN [1]
introduces a loss that encourages the generative network of GAN to spread its
distribution and cover the space of possible paths and designs a new pooling
mechanism that is employed to encode subtle cues for all people involved in a
scene. In [2], Zhu _et al_. propose StarNet that has a star topology with both
host network and hub one, where the hub network takes the observed
trajectories of all pedestrians and produces a comprehensive spatio-temporal
representation of all interactions in crowds. In order to integrate human
social behaviors in dynamic scenes for prediction tasks, Sun _et al_. propose
a recursive social behavior graph [3], which is demonstrated to obtain greater
expressive power and higher performance.

###  II-C Tracking-by-Detection Algorithms for Visual Objects and Lane
Markings

In general, visual object detection methods are classified into two-stage and
one-stage ones. Two-stage methods including fast R-CNN [21] and faster R-CNN
[22] are able to detect small and even tiny objects effectively but they
usually have higher computational complexity, compared to one-stage ones.
Recently, a couple of sophisticated one-stage models such as Yolo series [23,
24, 25] and SSD [26] have not only excellent detection performance but also
good real-time performance [27]. In particular, there are also large-scale
datasets for visual object detection tasks like MSCOCO [28] and PASCAL VOC
[29], which often results in effective pre-trained models being available. In
fact, visual object detection is the foundation of visual object tracking
(VOT), both of which are closely related. Based on [30, 31], Zhu _et al_.
present a distractor-aware siamese network for VOT called DaSiamRPN [32] that
adds RPN and distractor-aware module, and achieves SOTA performance with even
a speed of over 160 fps, which model parameters can be updated at runtime.
Compared to single object tracking (SOT), the research progress of multi-
object tracking (MOT) is relatively slow. Actually, DeepSort [33] proposes a
mainstream tracking-by-detection framework, which generally has better
performance in the MOT method. On the other hand, lane markings detection
plays an important role in the field of autonomous driving. A collection of
large-scale datasets on lane markings detection such as CULane [34] and
Tusimple [35] are publicly released. In recent years, as deep learning
tremendously proceeds, many lane detection algorithms with exceptional
performance like [34] and [36] continue to emerge.

##  III Mothodlogy

A novel phase space reconstruction network called PSRNet is proposed to
accurately recognize lane intrusion actions in high-speed and long-range
traffic scenes. Fig. 2 shows the overall framework of our PSRNet. In the upper
part of Fig. 2, the lane width normalization is employed to transform raw
video sequences into relative motion time series. Specifically, raw video
sequences are first fed into monocular tracking-by-detection modules to
estimate positions of both moving objects, i.e., pedestrians and cyclists, and
lane markings and accordingly, resulting positions of objects are then
normalized based on pixel lane width so as to generate the motion time series
of each object relative to central trajectory of current lane. In this case,
as shown in the lower part of Fig. 2, the PSRNet is proposed to classify the
above relative motion time series and eventually accomplish lane intrusion
action recognition. Basically, our PSRNet consists of a reconstructor and a
classifier, where a reconstructor builds latent phase spaces embedded by
motion time series through supervise learning and a classifier learns Lyapunov
exponent-like characteristics of phase spaces to make prediction of lane
intrusion actions.

###  III-A Lane Width Normalization

In this subsection, we analyze how to leverage the motion time series to
represent the lane intrusion action and discuss the effectiveness of
normalization. Especially, we explain why only a monocular camera can be
utilized to recognize lane intrusion actions that appear in the real world.

####  III-A1 Position Estimation of Objects

Using sophisticated deep learning algorithms including visual object and lane
markings detections, we can infer bounding boxes of moving objects and pixel
segmentations of current left/right lane markings. As shown in Fig. 3, the
above tracking-by-detection results are all projected onto the identical image
coordinate system. A blue line extended along the bottom line of bounding-
boxes is referred to as a horizontal baseline of current video frame. On the
horizontal baseline, we establish a one-dimensional coordinate system in order
to clearly mark positions of both moving objects and lane markings. As a
result, relative motions are evaluated in terms of this coordinate systems.
Apparently, it is the same as the $u$-axis in the image coordinate system.
Thus the point $o$ denotes the center of objects detected, which 1D
coordinates are $(u_{tl}+u_{br})/2$.

Figure 3: The visual detection results in a unified 1D coordinate system. The
black pixel points indicate the lane markings, the red box stands for the
bounding box of moving objects, and the blue line expresses the horizontal
baseline or the 1D coordinate axis. $l_{1}$, $l_{2}$, and $o$ denote positions
or 1D coordinates of the left, right lane markings, and the moving object,
respectively.

####  III-A2 Normalization Based on Lane Width

Based on deep learning methods for tracking-by-detection, we can locate moving
objects $o$ and current left/right lane markings $l_{1}$ and $l_{2}$ in the
image coordinate system. In order to express the spatial-temporal motion of
moving objects, we present an object-level normalization approach. Let the
intersection $l_{c}$ between the horizontal baseline and the current lane
central trajectory be the origin of the 1D coordinate system. The left side of
the origin is defined as negative and the right one as positive. At the same
time, the pixel lane width in current video frame is viewed as the normalized
unit length. Through normalizing with such a visual lane width in each current
frame, the 1D time series of moving objects relative to current lane central
trajectory can be calculated. It has

$\displaystyle p_{r}=\frac{u_{o}-u_{c}}{w}$ $\displaystyle u_{c}=(u_{1}$
$\displaystyle+u_{2})/2,w=\left|u_{1}-u_{2}\right|$ (1)

where $w$ indicates the pixel width of current lanes, $u_{c},u_{1},$ and
$u_{2}$, respectively, stand for the coordinates of the center point (i.e. the
origin), the left and right representatives of current lane markings near
moving objects. $o$ denotes the representative of moving objects, and $p_{r}$
expresses the normalized position of $o$ relative to the central trajectory
$l_{c}$.

####  III-A3 Action Representation by Motion Time Series

In image coordinate, there are the objects $o$ ($u_{o},v_{o}$), lane marking
$l_{1}$ ($u_{1},v_{o}$) and $l_{2}$ ($u_{2},v_{o}$). Correspondingly, in world
coordinate system as shown in Fig. 4, there are also moving objects $O$
($X_{w}^{o},Y_{w}^{o},Z_{w}^{o}$), left lane markings $L_{1}$
($X_{w}^{1},Y_{w}^{o},Z_{w}^{1}$), right lane markings $L_{2}$
($X_{w}^{2},Y_{w}^{o},Z_{w}^{2}$), and camera ($x_{c},y_{c},z_{c}$,
$\theta_{x},\theta_{y},\theta_{z}$). The principle of camera imaging geometry,
described in Eq. (15), illustrates how to project a point
($X_{w},Y_{w},Z_{w}$) in the world coordinate system onto a point ($u,v$) in
the image ones.

Figure 4: In the world coordinate system, the normalized relative positions
are identical for different view angles of camera.

$\displaystyle\centering Z_{c}\left[\begin{array}[]{c}u\\\ v\\\
1\end{array}\right]\@add@centering$
$\displaystyle=\left[\begin{array}[]{cccc}f_{x}&0&u_{0}&0\\\
0&f_{y}&v_{0}&0\\\ 0&0&1&0\end{array}\right]\left[\begin{array}[]{cc}A&T\\\
0&1\end{array}\right]\left[\begin{array}[]{c}X_{w}\\\ Y_{w}\\\ Z_{w}\\\
1\end{array}\right]$ (14) $\displaystyle A$
$\displaystyle=R_{z}(\theta_{z})R_{x}(\theta_{x})R_{y}(\theta_{y}),T=[x_{c},y_{c},z_{c}]^{T}$
(15)

where $Z_{c}$ is the object coordinates in the camera coordinate system,
$f_{x},f_{y},u_{0},v_{0}$ are the camera intrinsic parameters, and the
rotation matrix $A$ and the transformation matrix $T$ are the camera external
parameters. $R_{x}$ represents the rotation matrix of rotation angle
$\theta_{x}$, while $(x,y,z)$ denotes the location of camera in the world
coordinate system.

Suppose that within a local region, the camera only rotates $\theta_{y}$ along
the $Y$-axis and moves on the $X$-$Z$ plane, and lane markings are parallel to
the $Z$-axis. As a result, a point in the world coordinate system can be
projected onto the one in the corresponding image coordinate system, i.e.,

$\displaystyle
u=\frac{f_{u1}(X_{w})+f_{u2}(Y_{w},\theta_{y})+f_{u3}(Z_{w},\theta_{y})+f_{u4}(x_{c},z_{c})}{Z_{c}}$
(16) $\displaystyle
v=\frac{f_{v1}(X_{w})+f_{v2}(Y_{w},\theta_{y})+f_{v3}(Z_{w},\theta_{y})+f_{v4}(x_{c},z_{c})}{Z_{c}}$
(17)

where $f$ denotes a linear function. Since the monocular camera lacks the
depth information, we make a reasonable assumption that the depth ($Z_{c}$) of
both moving objects and their neighboring lane markings is almost the same. So
the line where $O$, $L_{1}$ and $L_{2}$ are located at is perpendicular to the
optical axis, as shown in Fig. 4. For objects and lane markings, $Z_{c}$ and
$v$ is equal. Thus we can have $Z_{w}^{1}$ and $Z_{w}^{2}$ below,

$\displaystyle f_{u1}(X_{w}^{o})+f_{u3}(Z_{w}^{o},\theta)$
$\displaystyle=f_{u1}(X_{w}^{1})+f_{u3}(Z_{w}^{1},\theta)$
$\displaystyle=f_{u1}(X_{w}^{2})+f_{u3}(Z_{w}^{2},\theta)$ (18)

Then we project the points $O$, $L_{1}$ and $L_{2}$ onto the ones in the image
coordinate system according to Eq. (16). Based on Eq. (1), we can get the
relative position of objects,

$\displaystyle
p_{r}=\frac{u_{o}-(u_{1}+u_{2})/2}{\left|u_{1}-u_{2}\right|}\approx\frac{2\cdot
X_{w}^{o}-(X_{w}^{1}+X_{w}^{2})}{2\cdot\left|X_{w}^{1}-X_{w}^{2}\right|}$ (19)

As shown in Fig. 4, it is easy to know that the normalized position of points
$O$ on the line $L_{1}L_{2}$ is the same as on the line2. It means that
relative position $p_{r}$ is only associated with $X_{w}^{o}$ because
$X_{w}^{1}$ and $X_{w}^{2}$ remain fixed. Therefore, lane intrusion actions
are independent of object trajectories.

###  III-B PSRNet

In general, a phase space represents the smallest possible state space of a
dynamic system. Each state vector of a dynamic system is expressed as a point
in phase space. Any time series observations can be regarded as outputs of
embedded dynamic systems with unknown system order, which is described by
phase state space representation. Accordingly, the evolution process of a
dynamic system can be depicted as a motion trajectory from the initial state
in phase space. Time series is usually analyzed in the time domain. In
particular, for the chaotic time series, many studies are carried out in
reconstructed nonlinear phase space [37, 38, 16]. Therefore, the phase space
reconstruction method plays an important role in the processing of chaotic
time series. In addition, the Lyapunov exponent is defined as the average
exponential divergence rate of adjacent motion trajectories in phase space,
which is generally viewed as one of several numerical characteristics for
identifying nonlinear chaotic motion [39].

We devise a PSRNet that comprises both a reconstructor and a classifier. The
PSRNet first reconstructs an ensemble of latent phase spaces in the
reconstructor and then learns Lyapunov exponent-like characteristics in the
classifier.

####  III-B1 Reconstruction of Latent Phase Space

Based on the idea of traditional phase space reconstruction, the proposed
PSRNet builds an $n$-order phase space to represent possible state vectors
corresponding to the motion time series. As shown in Fig. 5, considering that
the system order is unknown, the reconstructor in our PSRNet leverages $n$ CNN
modules to reconstruct $n$ dynamic systems with different system orders.
Specifically, the input of each CNN module is the $k$ observations before the
current time $t$ in the given time series data $(k=1,2,\cdots,n)$, denoted as
{$p_{t-1}$, $\cdots$, $p_{t-k}$}, so that the $k$-order phase space is
described as follows,

$\displaystyle PS_{t}^{k}=\varGamma_{k}(p_{t-1},\cdots,p_{t-k})$ (20)

where $PS_{t}^{k}$ represents the $k$-order phase space at time $t$ and
$\varGamma_{k}$ expresses the end-to-end CNN of order $k$. In essence, the
reconstructor adopts a supervised learning to accurately approximate nonlinear
mappings in nonlinear difference equations with $k$ system orders. In other
words, the identical time series observations are utilized to learn the
approximation of $k$-order difference systems. Note that the observation
$p_{t}$ itself at the current time $t$ in the given time series is employed as
the output expectation of CNN modules. Here we adopt the mean square error
(MSE) as the loss function of the reconstructor. The loss $\mathcal{L}_{1k}$
of the $k$-order CNN module is

$\displaystyle\mathcal{L}_{1k}$
$\displaystyle=\mathcal{F}_{MSE}(\widehat{p_{tk}},p_{t})$ (21)
$\displaystyle\widehat{p_{tk}}$ $\displaystyle=\mathcal{F}_{k}(PS_{t}^{k})$
(22)

where $\mathcal{F}_{MSE}$ indicates the MSE loss function and
$\mathcal{F}_{k}$ expresses the $k$-order linear layer.

The latent phase space actually expresses an ensemble of $n$ reconstructed
dynamic systems. All the inputs and outputs of the CNN modules, i.e. the
reconstructor, are aggregated to form the latent phase space of an ensemble of
dynamic systems. The ensemble is given below,

$\displaystyle PS_{t}=\mathcal{F}_{cat}(p_{t},PS_{t}^{1},\cdots,PS_{t}^{n})$
(23)

where $\mathcal{F}_{cat}$ stands for the CONCAT operation.

Figure 5: The PSRNet. The reconstructor employs the $n$ 1D-CNNs with different
system order to approximate the given time series observations by supervised
learning. All the inputs and outputs of such reconstructor are fed into the
CNN classifier for lane intrusion action recognition. $n$ is set to $4$ in
this work.

####  III-B2 Learnable Lyapunov Exponent

Afterwards, we aggregate all the inputs and outputs of the reconstructor and
feed such an ensemble $PS_{t}$ into another lightweight CNN classifier, whose
output is just the classification results of lane intrusion actions.
Heuristically, this can be considered as learning to approximate the Lyapunov
exponent-like characteristics of reconstructed latent phase spaces $PS_{t}$
and then making use of them to classify different lane intrusion actions or
corresponding motion time series.

Specifically, such Lyapunov exponent-like classifier shown in Fig. 5 includes
two convolutional layers with $3\times 3$ kernels and one fully-connected
layer with softmax, which outputs the final classification scores. The
classification results of lane intrusion actions are described as follows:

$\displaystyle\widehat{l_{p}}=\mathcal{F}_{c}(PS_{t})$ (24)

where $\mathcal{F}_{c}$ represents the Lyapunov exponent-like classifier. The
loss of such CNN classifier is defined as the cross-entropy loss that is
indicated by $\mathcal{L}_{2}$, i.e.,

$\displaystyle\mathcal{L}_{2}=\mathcal{F}_{CE}(\widehat{l_{p}},l_{p})$ (25)

where $\mathcal{F}_{CE}$ expresses the cross-entropy loss, and $l_{p}$ denotes
the label of lane intrusion actions of {$p_{0}$, $\cdots$, $p_{t}$, $p_{t+1}$,
$\cdots$}.

In a word, PSRNet employs supervised learning to model the ensemble of hybrid-
order nonlinear dynamic systems embedded by time series observations and
reconstruct corresponding latent phase spaces. Sequentially, it further learns
the Lyapunov exponent-like discrimination of phase spaces to build a
classifier in order to enhance the generalization ability of the whole PSRNet.
The overall loss $\mathcal{L}$ of the PSRNet is given by:

$\displaystyle\mathcal{L}=\lambda\cdot\sum_{k=1}^{n}\mathcal{L}_{1k}+\mathcal{L}_{2}$
(26)

where $\lambda$ indicates the weight coefficient.

##  IV Experiments

###  IV-A Data Preparation: THU-IntrudBehavior

As far as we know, there is no public dataset for lane intrusion action
recognition. In order to verify the effectiveness of our PSRNet and initially
examine the feasibility of the proposed method in real-world road traffic
environments, we built the lane intrusion action recognition dataset called
THU-IntrudBehavior. In the data collection scheme, we firmly mounted the
U-TRON telephoto lens and the FLIR camera directly in front of the driver’s
cab to clearly capture road traffic scenes within 150-250m ahead. The ego-
vehicle speed remains above 70km/h when collecting data. The video data is
acquired in three weather conditions of being sunny, cloudy, and rainy,
covering various situations. We imitate lane intrusion action on open urban
roads and structured roads in the vicinity of the Garden Expo Park, Beijing,
China. Intruders or moving objects include pedestrians and cyclists. Such THU-
IntrudBehavior is expected to be publicly released very soon.

After completing data collection, we clean and annotate all the data. We have
removal of the incomplete videos and crop relevant video clips in terms of the
presence of intrusion behaviors. Furthermore, we accomplish all the
annotations including the bounding box labels of pedestrians and cyclists, the
pixel-level labels of neighboring lane markings, and the category labels of
intrusion behaviors below: 1) intruding into current lane from left to right,
2) intruding into current lane from right to left, and 3) without any lane
intrusion. Note that we only focus on lane markings fragments near
pedestrians/cyclists in the annotation. We got 96 videos online with a total
of 10,972 pictures. It contains 154 lane intruders (i.e. pedestrians and
cyclists). In order to justly assess the accuracy and robustness of models, we
conducted the 3-fold, 5-fold, and 7-fold cross-validation experiments,
respectively, in which all the samples must be shuffled randomly.

###  IV-B Implementation Details

####  IV-B1 Object Detection and Lane Markings Detection

First, we got ready for the visual object detection model Yolov3 [25] pre-
trained using the MSCOCO dataset [28] and the lane markings detection model
SCNN [34] that was also pre-trained based on the CULane dataset [34]. Second,
we then used the THU-IntrudBehavior to make fine-tuning of the above two pre-
trained models, respectively. After those preparations were done, the video
frames were continuously fed into both the well-trained Yolov3 and SCNN
models. Accordingly, the bounding boxes of moving objects including
pedestrians and cyclists and the segmentations of current lane markings for
each frame were inferred. Finally, we combined the inferred bounding-boxes and
segmentations in a unified coordinate system, and figured out the pixel
positions of moving objects and lane markings each frame, in order to give
rise to the object-level motion time series of intruders and lane markings.

####  IV-B2 Filtering and Normalization

First, we aligned the motion time series data of each pedestrian/ cyclist and
removed all false positives using the Hungarian algorithm. Afterwards, we
adopted the Kalman filtering to smooth noises in these time series and lane
markings data. The initial state values of the Kalman filtering were assumed
to the median of the first three values in time series. Finally, the
normalized relative motion time series was computed according to Eq. (1), all
of which would be utilized as inputs of PSRNet.

####  IV-B3 PSRNet and Training Details

Specifically, we selected 24 consecutive frames of the motion time series data
as inputs to the PSRNet. In the experiment, the maximum system order of latent
phase spaces was assumed to be 4. In order to reconstruct the phase space, we
built 4 one-dimensional CNN modules to form an ensemble, whose input sizes
were 1, 2, 3, and 4, respectively. Note that each of 4 CNN modules outputs 8
channels. Meanwhile, the Lyapunov exponent-like classifier comprised two
convolutional layers with $3\times 3$ kernel size and one fully-connected
layer, where the numbers of channels in the two convolutional layers are 16
and 32, respectively. Furthermore, we select $\lambda=0.5$ in Eq. (26). In the
training algorithm, Adam optimizer was applied to train the model for 100
epochs. The initial learning rate was set to 0.001 with a mini-batch of 32.

TABLE I: Performance comparison with 3-fold
cross-validation experiments. Method | Accuracy(%)
---|---
R(2+1)D [10] | $47.9\pm 1.5$
Two-stream [7] | $52.1\pm 1.5$
TRN [11] | $57.3\pm 3.0$
LSTM-based | $96.7\pm 2.5$
GRU-based | $97.4\pm 1.1$
TCN-based | $96.1\pm 1.7$
MLP-based | $95.3\pm 1.8$
PSRNet |  98.0 $\pm$ 1.6

###  IV-C Performance Comparison

We selected R(2+1)D [10], the two-stream network [7], and TRN [11] as the
baseline models in this paper. We also compared LSTM-based, GRU-based, TCN-
based and MLP-based methods with the proposed method. In this case, all the
LSTM-based, GRU-based, and TCN-based models consist of two-layer neural
networks with the same number of channels as PSRNet, which inputs are all the
above object-level motion time series data. But the MLP-based model has one
hidden layer with 512 neurons. In particular, the 3-fold cross-validation
experiments for performance evaluation were carried out due to the relatively
small number of samples in the THUIntrudBehavior.

The experimental results are listed in Table I. Compared to 3D CNNs and two-
stream networks, the test accuracy of our PSRNet is remarkably improved, while
the LSTM-based and GRU-based methods also achieve significant improvements. In
fact, the PSRNet slightly outperforms the LSTM-based, GRU-based, TCN-based,
and MLP-based methods in terms of its lower standard deviation. All the
results demonstrate that object-level learning leads to better performance.

###  IV-D Ablation Study and Discussion

In this subsection, we completed extensive ablation experiments in order to
analyze the influence of individual components in our PSRNet method. It should
be pointed out that all experiments were done on THUIntrudBehavior based on
$k$-fold cross-validation ($k=$3, 5, and 7).

####  IV-D1 Normalization and Filtering

|
---|---
(a) Before Normalization | (b) After Normalization
Figure 6: (a) Scattered points indicate position estimations given by visual object detection algorithms. The curves represent trajectories smoothed by the Kalman filtering. (b) The red curve expresses the normalized motion time series of moving objects relative to the central trajectory and the black dashed lines denote current left/right lane markings. TABLE II: The effects of lane width normalization and filtering. Normalization | Filtering | 3-fold | 5-fold | 7-fold
---|---|---|---|---
|  | $65.8\pm 9.2$ | $65.8\pm 11.4$ | $50.0\pm 14.0$
✓ |  | $95.8\pm 4.1$ | $97.2\pm 2.8$ | $96.7\pm 2.5$
✓ | ✓ | $98.0\pm 1.6$ | $98.0\pm 1.7$ | $97.4\pm 3.3$

We study the effect of filtering and normalization through inputting object-
level relative time series into the PSRNet and making comparison of
performance, as listed in Table II. Obviously, without any filtering and
normalization, the performance of our PSRNet is basically similar to that of
the baselines [10, 7, 11]. Both filtering and normalization are crucial to
keeping spatial-temporal continuity in object-level relative time series data.
As shown in Fig. 6(a), the scattered points represent positions given by
visual object detection algorithms, while the curves indicate trajectories of
pedestrians/cyclists smoothed by the Kalman filtering. The Kalman filtering
algorithm is adopted to eliminate outliers and smooth noises, both of which
are actually caused by inevitable false positives and missed detections of any
tracking-by-detection algorithms. We believe that combination of filtering and
normalization simply leads to significant performance improvements.
Additionally, it is easy to see from Fig. 6(a) that the motion of objects
cannot be identified merely through observing the scatter points or the
curves. It is even more impossible to determine whether there are intrusion
actions. Fig. 6(b) shows the normalized smooth time series relative to the
central trajectory of current lane. Because of the normalization of position
estimations based on current lane pixel widths each frame, the adverse effect
of high-speed movements is reduced greatly. In fact, we can intuitively judge
intrusion actions of objects only based on these object-level relative motion
time series data.

TABLE III: The effects of different system orders of phase spaces. Order | 3-fold | 5-fold | 7-fold
---|---|---|---
0 | $96.3\pm 2.3$ | $96.7\pm 2.7$ | $96.4\pm 5.7$
1 | $97.2\pm 1.2$ | $96.1\pm 3.3$ | $96.7\pm 4.7$
2 | $97.3\pm 1.2$ | $96.7\pm 2.7$ | $96.7\pm 4.3$
3 | $97.3\pm 1.2$ | $97.4\pm 4.4$ | $97.2\pm 3.6$
4 | $98.0\pm 1.6$ | $98.0\pm 1.7$ | $97.4\pm 3.3$

####  IV-D2 Orders of Latent Phase Spaces

In addition, we also performed experiments of the effect of phase spaces with
different system orders on the performance of PSRNet. We used zeropadding for
lower system orders. As shown in Table III, the accuracy becomes greater as
system orders increase. In the process of reconstructing phase spaces, we
employed hybrid system orders to capture features of dynamic systems embedded
in relative motion time series. Meanwhile, all the inputs and outputs of
reconstructed latent phase spaces were aggregated as the input of the
classifier so as to further learn Lyapunov exponent-like discriminations.
Overall, the application of heuristic knowledge, including phase space and
Lyapunov exponent, makes such deep learning models capable of catching
inherent dynamic structural features more accurately, which leads to stronger
generalization ability.

####  IV-D3 Weights in the Loss Function

The weight coefficient $\lambda$ in Eq. (26) determines the relative
contribution of reconstructor and classifier on the whole action recognition
task. In general, too small weights $\lambda$ will reduce performance due to
unbalanced feature representation. Therefore, the choice of the weight
$\lambda$ needs to empirically be compromised based on trial and error. In all
the experiments, we set $\lambda=0.5$.

##  V Conclusions

This paper proposes a lightweight PSRNet for lane intrusion action recognition
based on a monocular camera without using any HD map and precise navigation
equipment. First, on the basis of visual tracking-by-detection algorithms for
moving objects including pedestrians, cyclists, and lane markings, we present
a visual lane width-based normalization policy to generate one-dimensional
relative motion time series, which can reliably express lane intrusion actions
in video. Second, we build a PSRNet to conduct such motion time series
classification. Inspired from classic phase-space reconstruction and Lyapunov
exponent methods, the PSRNet comprises several CNN layers and classifier
layers, where the former sets up a latent phase space to implicitly represent
the movement of pedestrians and cyclists. Thanks to the object-level learning
and the introduction of phase space reconstruction and Lyapunov exponent-like
classifier, our PSRNet makes it easier to recognize lane intrusion actions in
video. The experimental results show that the proposed method even achieves
98.0% accuracy with high efficiency.

## Acknowledgments

This work was supported in part by the National Key R&D Program of China under
Grant No.2017YFB1302200 and 2018YFB1600804, by TOYOTA TTAD 2020-08, by a grant
from the Institute Guo Qiang (2019GQG0002), Tsinghua University, by Alibaba
Group through Alibaba Innovative Research Program, and by ZheJiang Program in
Innovation, Entrepreneurship and Leadership Team (2018R01017).

## References

  * [1] A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi, “Social GAN: Socially acceptable trajectories with generative adversarial networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 2255–2264.
  * [2] Y. Zhu, D. Qian, D. Ren, and H. Xia, “StarNet: Pedestrian trajectory prediction using deep neural network in star topology,” _arXiv preprint arXiv:1906.01797_ , 2019.
  * [3] J. Sun, Q. Jiang, and C. Lu, “Recursive social behavior graph for trajectory prediction,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 660–669.
  * [4] S. Pellegrini, A. Ess, K. Schindler, and L. Van Gool, “You’ll never walk alone: Modeling social behavior for multi-target tracking,” in _2009 IEEE 12th International Conference on Computer Vision_. IEEE, 2009, pp. 261–268.
  * [5] A. Lerner, Y. Chrysanthou, and D. Lischinski, “Crowds by example,” in _Computer graphics forum_ , vol. 26, no. 3. Wiley Online Library, 2007, pp. 655–664.
  * [6] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei, “Large-scale video classification with convolutional neural networks,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2014, pp. 1725–1732.
  * [7] K. Simonyan and A. Zisserman, “Two-stream convolutional networks for action recognition in videos,” in _Advances in neural information processing systems_ , 2014, pp. 568–576.
  * [8] J. Lin, C. Gan, and S. Han, “TSM: Temporal shift module for efficient video understanding,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 7083–7093.
  * [9] C. Feichtenhofer, H. Fan, J. Malik, and K. He, “Slowfast networks for video recognition,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 6202–6211.
  * [10] D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun, and M. Paluri, “A closer look at spatiotemporal convolutions for action recognition,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2018, pp. 6450–6459.
  * [11] B. Zhou, A. Andonian, A. Oliva, and A. Torralba, “Temporal relational reasoning in videos,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 803–818.
  * [12] C. Yang, Y. Xu, J. Shi, B. Dai, and B. Zhou, “Temporal pyramid network for action recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 591–600.
  * [13] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3D convolutional networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 4489–4497.
  * [14] J. Yue-Hei Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, and G. Toderici, “Beyond short snippets: Deep networks for video classification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 4694–4702.
  * [15] C. Feichtenhofer, A. Pinz, and A. Zisserman, “Convolutional two-stream network fusion for video action recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1933–1941.
  * [16] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. Van Gool, “Temporal segment networks: Towards good practices for deep action recognition,” in _European conference on computer vision_. Springer, 2016, pp. 20–36.
  * [17] J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” in _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 6299–6308.
  * [18] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [19] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” _arXiv preprint arXiv:1406.1078_ , 2014.
  * [20] S. Bai, J. Z. Kolter, and V. Koltun, “An empirical evaluation of generic convolutional and recurrent networks for sequence modeling,” _arXiv preprint arXiv:1803.01271_ , 2018.
  * [21] R. Girshick, “Fast R-CNN,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1440–1448.
  * [22] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” in _Advances in neural information processing systems_ , 2015, pp. 91–99.
  * [23] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 779–788.
  * [24] J. Redmon and A. Farhadi, “Yolo9000: better, faster, stronger,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 7263–7271.
  * [25] ——, “Yolov3: An incremental improvement,” _arXiv preprint arXiv:1804.02767_ , 2018.
  * [26] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “SSD: Single shot multibox detector,” in _European conference on computer vision_. Springer, 2016, pp. 21–37.
  * [27] X. Liu, Z. Deng, and Y. Yang, “Recent progress in semantic image segmentation,” _Artificial Intelligence Review_ , vol. 52, no. 2, pp. 1089–1106, 2019.
  * [28] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [29] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” _International journal of computer vision_ , vol. 88, no. 2, pp. 303–338, 2010.
  * [30] L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. Torr, “Fully-convolutional siamese networks for object tracking,” in _European conference on computer vision_. Springer, 2016, pp. 850–865.
  * [31] M. Danelljan, A. Robinson, F. S. Khan, and M. Felsberg, “Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking,” in _European Conference on Computer Vision_ , 2016.
  * [32] Z. Zhu, Q. Wang, B. Li, W. Wu, J. Yan, and W. Hu, “Distractor-aware siamese networks for visual object tracking,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 101–117.
  * [33] N. Wojke, A. Bewley, and D. Paulus, “Simple online and realtime tracking with a deep association metric,” in _2017 IEEE international conference on image processing (ICIP)_. IEEE, 2017, pp. 3645–3649.
  * [34] X. Pan, J. Shi, P. Luo, X. Wang, and X. Tang, “Spatial as deep: Spatial cnn for traffic scene understanding,” in _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018.
  * [35] TuSimple, “Tusimple benchmark,” 2017.
  * [36] D. Neven, B. De Brabandere, S. Georgoulis, M. Proesmans, and L. Van Gool, “Towards end-to-end lane detection: an instance segmentation approach,” in _2018 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2018, pp. 286–291.
  * [37] Z. Gao and N. Jin, “Complex network from time series based on phase space reconstruction,” _Chaos: An Interdisciplinary Journal of Nonlinear Science_ , vol. 19, no. 3, p. 033137, 2009.
  * [38] G.-F. Fan, L.-L. Peng, and W.-C. Hong, “Short term load forecasting based on phase space reconstruction algorithm and bi-square kernel regression model,” _Applied Energy_ , vol. 224, pp. 13–33, 2018.
  * [39] E. B. Rozenbaum, S. Ganeshan, and V. Galitski, “Lyapunov exponent and out-of-time-ordered correlator’s growth rate in a chaotic system,” _Physical review letters_ , vol. 118, no. 8, p. 086801, 2017.