∎

11institutetext: P. H. Luz de Araujo (✉) $\cdot$ F. B. Vidal $\cdot$ T. E. de
Campos 22institutetext: Department of Computer Science, Universidade de
Brasília, 70910-900 Brasília, Brazil
22email<EMAIL_ADDRESS>
22email<EMAIL_ADDRESS>
22email<EMAIL_ADDRESS>33institutetext: A. P. G. S. de Almeida
44institutetext: Department of Mechanical Engineering, Universidade de
Brasília, 70910-900 Brasília, Brazil
44email<EMAIL_ADDRESS>55institutetext: F. A. Braz $\cdot$ N. C. da
Silva 66institutetext: Gama Faculty, Universidade de Brasília, Campus Gama -
Setor Leste - Gama - 72444-240, Brasília, Brazil
66email: {fabraz<EMAIL_ADDRESS>
# Sequence-aware multimodal page classification of Brazilian legal documents
††thanks: This preprint, which was originally written on 8 April 2021, has not
undergone peer review or any post-submission improvements or corrections. The
Version of Record of this article is published in the International Journal on
Document Analysis and Recognition (IJDAR), and is available online at
https://doi.org/10.1007/s10032-022-00406-7. The full paper can be accessed at
https://rdcu.be/cRvvV.

Pedro H. Luz de Araujo  Ana Paula G. S. de Almeida  Fabricio A. Braz  Nilton
C. da Silva  Flavio de Barros Vidal  Teofilo E. de Campos

###### Abstract

The Brazilian Supreme Court receives tens of thousands of cases each semester.
Court employees spend thousands of hours to execute the initial analysis and
classification of those cases—which takes effort away from posterior, more
complex stages of the case management workflow. In this paper, we explore
multimodal classification of documents from Brazil’s Supreme Court. We train
and evaluate our methods on a novel multimodal dataset of 6,510 lawsuits
(339,478 pages) with manual annotation assigning each page to one of six
classes. Each lawsuit is an ordered sequence of pages, which are stored both
as an image and as a corresponding text extracted through optical character
recognition. We first train two unimodal classifiers: a ResNet pre-trained on
ImageNet is fine-tuned on the images, and a convolutional network with filters
of multiple kernel sizes is trained from scratch on document texts. We use
them as extractors of visual and textual features, which are then combined
through our proposed Fusion Module. Our Fusion Module can handle missing
textual or visual input by using learned embeddings for missing data.
Moreover, we experiment with bi-directional Long Short-Term Memory (biLSTM)
networks and linear-chain conditional random fields to model the sequential
nature of the pages. The multimodal approaches outperform both textual and
visual classifiers, especially when leveraging the sequential nature of the
pages.

###### Keywords:

Multimodal page classification Document classification Legal domain Sequence
classification Portuguese language processing

##  1 Introduction

The Brazilian court system is burdened by a large number of lawsuits. In 2019,
there were 77.1 million lawsuits awaiting judgment—almost one lawsuit for
every three Brazilians. Some of these lawsuits will stay in the system for a
long time, with average processing times that can reach more than six years.
All of this contributes to raising the legal system cost: that same year,
Brazil spent about R$100 billion in expenses with the judiciary, about 25
billion dollars considering the average exchange rate in 2019
justiceSummary2020 .

Natural language processing (NLP) and machine learning (ML) techniques can
improve this scenario by enabling faster and more efficient document analysis.
Brazil’s Supreme Court receives roughly 42 thousand cases each semester, which
takes about 22 thousand hours for humans to sort through stf_victor_news .
This time could be better spent on more complex stages of the workflow, such
as those requiring legal reasoning. The cases reach the court as mostly
unstructured and unindexed PDF files of raster-scanned documents
luzDeAraujo_etal_VICTOR_LREC_2020 . Intra-class diversity and document quality
are the main challenges: the documents range from petitions and evidence to
rulings and orders, originate from different Brazilian courts and often
contain visual noise such as handwritten annotation, stamps, and stains
(Figure 1).

Ginwidth=

Figure 1: The first eight pages of a lawsuit. While the first page is clean,
the others come from an older document and contain ink stains, stamps,
handwritten signatures and other artifacts.

Therefore, our goal is to explore and evaluate methods that automatically
classify document pages by combining different sources of information. Though
previous work luzDeAraujo_etal_VICTOR_LREC_2020 ; mota_etal_cnnPet_ENIAC_2020
has examined Brazilian legal document classification, we are the first to
combine visual, textual, and sequential data to train better performing
models. Our main contributions are:

  1. 1.

a multimodal dataset of lawsuits composed of ordered document images and
corresponding texts.

  2. 2.

proposing and evaluating two mutimodal combination methods and two sequence
learning methods that leverage textual, visual and sequential information to
improve Supreme Court document classification.

  3. 3.

outperforming the state-of-the-art results on the small version of the VICTOR
dataset luzDeAraujo_etal_VICTOR_LREC_2020 .

The rest of this paper is organized as follows. In Section 2, we examine
previous work on multimodal document classification. In Section 3, we describe
the data we used to train and evaluate our models, which we describe, along
with their experimental setting, in Section 4. We then discuss the results
obtained and conclude the paper in sections 5 and 6.

##  2 Related Work

Textual and visual content are two of the four document aspects listed by Chen
and Blostein chen_and_blostein_diaSurvey_IJDAR_2007  as possible feature
sources. Image features range from fixed descriptors such as pixel density at
different locations and scales rusinol_etal_multimodalAdmDocument_IJDAR_2014
to approaches based on convolutional neural networks
jain_et_wigington_ICDAR_2019 ; engin_etal_multiModalBank_ICAIMM_2019 ;
audebert_etal_multiModalDocs_arxiv_2019 ;
wiedemann_heyer_multiModalPSS_LRE_2019 ; mota_etal_cnnPet_ENIAC_2020  such as
VGG-16 simonyan_etal_vgg16_2015  and MobileNetV2
sandler_etal_mobileNetV2_CVPR_2018 . Text features range from traditional
methods such as latent semantic analysis lsa  to pre-trained word embeddings
(e.g. Fasttext bojanowski_etal_fasttext_arxiv_2016 ) and deep learning
approaches engin_etal_multiModalBank_ICAIMM_2019 ;
audebert_etal_multiModalDocs_arxiv_2019 ;
wiedemann_heyer_multiModalPSS_LRE_2019 .

These feature modalities may be used by themselves or combined to improve
classification performance. This can happen at the feature level (early
fusion) or at the prediction level (late fusion). Rusinol et al.
rusinol_etal_multimodalAdmDocument_IJDAR_2014  experiment with both options,
trying different methods to combine predicted probabilities for late fusion
(summing, multiplying, taking the maximum and logistic regression). Jain and
Wigington jain_et_wigington_ICDAR_2019  compare a spatially-aware early fusion
method with four alternative methods of feature combination: concatenation,
addition, compact bilinear pooling and gated units. The spatially-aware fusion
underperformed simple feature combination, with concatenation, addition, and
bilinear approaches performing similarly. Engin et al.
engin_etal_multiModalBank_ICAIMM_2019  explore late and early fusion for the
classification of Turkish banking documents, finding that both outperform
unimodal methods. Mota et al. mota_etal_cnnPet_ENIAC_2020  investigate
multimodal classification of Brazilian court documents, concluding that
multimodal approaches compare favourably with unimodal ones.

Fewer works have explored incorporating sequential information. Rusinol et al.
rusinol_etal_multimodalAdmDocument_IJDAR_2014  use an $n$-gram model of the
page stream that conditions page predictions on the types of the $n-1$
previous pages to capture their sequential nature. Wiedemann and Heyer
wiedemann_heyer_multiModalPSS_LRE_2019  use as a feature of the target page
the encoding of its predecessor. Luz de Araujo et al.
luzDeAraujo_etal_VICTOR_LREC_2020  feed the predictions of a text classifier
to a linear-chain conditional random field (CRF) to jointly predict pages of
lawsuits.

In this paper, we focus on the sequentially-aware early fusion of visual and
textual features for legal document classification. To the best of our
knowledge, this is the first work that considers both visual and textual
modalities and sequential dependencies when classifying documents from the
legal domain and in Portuguese—Luz de Araujo et al.
luzDeAraujo_etal_VICTOR_LREC_2020  do not use visual features, while Mota et
al. mota_etal_cnnPet_ENIAC_2020  do not leverage sequential information.

##  3 Data

We perform our experiments on a dataset of 6,510 Extraordinary Appeals,
comprising 339,478 pages. Each instance is a lawsuit as it is received by the
Brazilian Supreme Court, before it is processed and judged. Each lawsuit
contains several different documents (petitions, rulings, orders) and is
represented as an ordered sequence of pages containing text.

This dataset is an extension of Small Victor luzDeAraujo_etal_VICTOR_LREC_2020
, which we expanded to include, in addition to textual data, the document
images. Every page in the expanded corpus is stored in at least one of two
formats. First, as text extracted through optical character recognition
smith_Tesseract_icdar2007 , with the following additional preprocessing steps:
lower-casing, removal of stop words and alphanumeric tokens, e-mail and URL
tokenization, and special tokenization of legislation references (e.g., Lei
(law) 11.419 to LEI_11419). Second, as JPEG images converted from the original
PDF files, with mean width and height of 1664 and 2322 pixels respectively.

The dataset was manually annotated with labels that assign to each page the
document class it belongs to. There are six possible classes:

  1. 1.

_Acórdão_ , for lower court decisions under review;

  2. 2.

_Recurso Extraordinário_ (RE), for appeal petitions;

  3. 3.

_Agravo de Recurso Extraordinário_ (ARE), for motions against the appeal
petition;

  4. 4.

_Despacho_ , for court orders;

  5. 5.

_Sentença_ for judgements; and

  6. 6.

_Others_ for documents not included in the previous classes.

Most of the samples contain both textual and visual sources of information,
except for 33,849 images with no corresponding text and 4 texts with no
corresponding image. We discuss our strategy for dealing with missing data
when training fusion models in Section 4.3. The corpus is divided into train,
validation and test splits containing 70%, 15%, and 15% of all suits,
respectively. Table 1 presents the number of text and image samples across
data splits and classes. Due to the nature of the data, a document may appear
more than once in a lawsuit, so we present both raw and deduplicated counts.
That said, given that the corpus has been split by lawsuits, there is no
sample intersection between splits.

Table 1: Class counts per split, showing the number of page images and text extracted through OCR. Between parentheses, the deduplicated counts. Class | Training set | Validation set | Test set
---|---|---|---
| Image | Text | Image | Text | Image | Text
Acórdão | 583 (583) | 553 (553) | 320 (314) | 299 (293) | 287 (285) | 273 (271)
ARE | 4,258 (4,220) | 2,546 (2,508) | 2,798 (2,650) | 2,149 (2,001) | 2,655 (2,537) | 1,841 (1,723)
Despacho | 361 (361) | 346 (346) | 189 (183) | 183 (177) | 199 (198) | 198 (197)
Others | 144,583 (140,786) | 134,134 (130,337) | 95,602 (91,434) | 84,104 (79,936) | 92,529 (87,902) | 85,408 (80,781)
RE | 10,225 (10,181) | 9,509 (9,465) | 6,987 (6,803) | 6,364 (6,180) | 6,386 (6,177) | 6,331 (6,122)
Sentença | 2,177 (2,177) | 2,129 (2,129) | 1,681 (1,613) | 1,636 (1,568) | 1,503 (1,478) | 1,475 (1,450)

Human agents find the first page of documents easier to classify when compared
to interior pages. This is true considering both visual and textual aspects,
since first pages contain highly informative cues, such as headers and titles.
Figure 2 compares first page and interior page samples for each class. We
validate this intuition in Section 5.5.

Ginwidth=  (a) Acórdão (b) ARE (c) Despacho (d) Others (e) Petição (f)
Sentença

Figure 2: First page (top row) and interior page samples for each class.

##  4 Methods

In this section we describe our methods for visual and textual page
classification, feature fusion and sequence learning. We also describe the
corresponding experimental settings. Unless stated otherwise, we optimise the
cross-entropy loss using Adam kingma2014adam  and mini-batches of 64 samples.
As metrics, we report arithmetic (average) and weighted by class frequencies
(weighted) means of the F1 score, defined as:

$F\textsubscript{1}=2\cdot\frac{\text{precision}\cdot\text{recall}}{\text{precision}+\text{recall}}\,,$
(1)

whereas precision and recall are defined as follows: let tp, fp and fn be the
number of true positives, false positives and false negatives, respectively.
Then,

precision $\displaystyle=\frac{\text{tp}}{\text{tp}+\text{fp}},$ (2) recall
$\displaystyle=\frac{\text{tp}}{\text{tp}+\text{fn}}\,.$ (3)

We evaluate the models using the parameters with the best average F1 score
computed on the validation set. That is, after each epoch, we only save model
parameters if the validation performance is the highest up to that point.

###  4.1 Text Classification

We use the CNN architecture described by Luz de Araujo et al.
luzDeAraujo_etal_VICTOR_LREC_2020  as the method for text classification. It
is a shallower version of the one proposed by Conneau et al. veryDeepTextClf
and works on the word level instead of the character level. Figure 3
summarises the CNN structure. To process a sample, the network takes as input
its first 500 tokens and embeds them into 100-dimensional vectors. The vectors
are then fed to three convolutional blocks composed of a convolutional layer
with three sets of 256 one-dimensional filters with different sizes (3, 4 and
5) followed by batch normalisation and max-pooling of size 2. The resulting
vectors are concatenated and fed to a max pool layer of size 50. The output is
flattened and processed by two fully connected (FC) layers, with the softmax
function producing the prediction. Dropout is applied to the output of the
first FC layer with a dropping probability of 50%.

Figure 3: The network architecture we used for text classification. Dashed
lines indicate dropout was applied.

As a strategy to deal with class imbalance, we train a variant of the CNN
model, which we call CNN-w, that weighs each sample contribution to the loss
by a factor inversely proportional to its class frequency. Let $c$ be the
number of classes and $\mathbf{w}$ a $c$-dimensional vector whose component
$i$ is the factor for class $i$. Then the factors are computed by the
following equation, as implemented in the scikit-learn library
buitinck_etal_sklearn_ECML_2013 :

$w_{i}=\frac{n}{c\cdot f_{i}}\,,$ (4)

where $n$ is the number of training samples and $f_{i}$ is the number of
samples from class $i$.

###  4.2 Image Classification

To classify document images, we fine-tune a ResNet50 he_etal_resnet_CVPR_2016
model pretrained on ImageNet russakovsky_etal_imagenet_IJCV_2015 . We first
train only the head of the model for one epoch, employing a cyclic learning
rate with cosine annealing cyclelr . Then, we train all layers for one cycle
of 6 epochs with discriminative fine-tuning ulmfit . As we did for the text
classifier, we train a variant of the model with factors inversely
proportional to class frequency: ResNet50-w.

To choose learning rates, we use the learning rate range test lrfinder . That
is, we train the model for a few iterations, starting from a low learning rate
value and increasing it after each mini-batch, plotting the loss against the
learning rates. We then pick a learning rate close to the point where the loss
starts to increase—high enough for quick learning, but not so high as to
impede learning.

###  4.3 Image and Text Combination Strategies

In this section we describe our proposed method for early fusion of visual and
textual features, a baseline method for comparison and an ablation analysis of
the fusion classifier.

####  4.3.1 Hybrid Classifier

As a baseline method that fuses visual and textual data we use a hybrid
classifier (HC) that works as follows: if textual data is available, use the
best text classifier; otherwise, use the best image classifier. The intuition
is that this approach would be at least as good as using only text data, which
better discriminates the document classes when compared with visual data.
Figure 4 illustrates the method.

Figure 4: The hybrid classifier (HC): a baseline fusion classifier that only
uses visual information if text data is not available.

####  4.3.2 Fusion Module

To fuse visual and textual data we first compute representations using the
trained text and image classifiers. As text embeddings, we extract the
3,840-dimensional activations of the flatten layer of the CNN (Fig. 3). As
image embeddings, we take the activations of the last convolutional block in
the ResNet and apply global average and global max pooling. Then we
concatenate and flatten the result, obtaining 4,096-dimensional vectors.

The pre-computed representations are concatenated and fed to a batch
normalisation layer batchnorm , followed by an FC layer with $d$ units, batch
normalisation, and a final FC layer. The softmax function produces the
predictions. Figure 5 illustrates our proposed Fusion Module (FM).

In case of missing data, when only the document image or text is available—but
not both—we experiment with two options. The first uses learnable embeddings
for missing text or image; the other, simply uses a vector of zeroes in such
cases (FM-zero).

We run preliminary experiments with one cycle of 10 epochs for each of four
configurations, varying the number of hidden units $d$ (128 or 512) and the
use of learnable embeddings or zero vectors for missing data. We then train
the model that obtained the highest average F1 score from scratch for one
cycle of 20 epochs. Learning rates are chosen using the range test.

Figure 5: Fusion Module (FM): the proposed method for early fusion of textual
and visual information. Dashed lines indicate dropout was applied. The
hyperparameter $d$ is the number of units in the first fully connected layer.

###  4.4 Sequence Classification

Given that a lawsuit is composed of an ordered series of pages, one can,
instead of classifying each page by itself, leverage the sequential nature of
the data by treating the problem as a sequence labelling task. That is, rather
than having a page and a class prediction as input and output, the sequence
classification approach outputs a sequence of class predictions, given a
sequence of input pages. We employ the IOB tagging scheme IOB_scheme  to
better leverage the sequential information: we prepend “B-” to the ground
truth of first-page samples and “I-” otherwise. For example, if a suit begins
with a RE of three pages followed by an ARE of equal length, the label
sequence would start with B-RE, I-RE, I-RE, B-ARE, I-ARE, I-ARE.

####  4.4.1 CRF postprocessing

As a baseline method for sequence classification, we save the predictions of
the FM for all samples in our data. Then we use these six-dimensional vectors
as features to train a linear-chain conditional random field (CRF) CRF .
Figure 6 illustrates this method (FM+CRF).

Figure 6: Baseline sequence classification method (FM+CRF). We feed the (pre-computed) predictions of the Fusion Module (FM) to a linear-chain CRF to jointly predict the class of each page in an Extraordinary Appeal. Table 2: Test set F1 scores (in %) of our main approaches for image, text, fusion and sequence classification. Image results are reported for the image test set; all the others, for the text test set. Class | Majority Baseline | Text | Image | Fusion | Sequence
---|---|---|---|---|---
| Text | Image | CNN | ResNet50-w | ResNet50 | FM | FM+CRF | BiLSTM-F
Acórdão | 00.00 | 00.00 | 89.96 | 18.45 | 06.78 | 90.74 | 91.56 | 88.97
ARE | 00.00 | 00.00 | 55.72 | 11.33 | 00.00 | 57.92 | 60.74 | 61.16
Despacho | 00.00 | 00.00 | 62.94 | 08.44 | 00.00 | 63.98 | 62.69 | 64.07
Others | 94.41 | 94.28 | 97.31 | 61.72 | 95.02 | 97.24 | 97.67 | 97.46
RE | 00.00 | 00.00 | 75.59 | 32.59 | 34.96 | 75.47 | 78.43 | 79.67
Sentença | 00.00 | 00.00 | 80.53 | 43.52 | 48.67 | 82.04 | 83.42 | 85.26
Average | 15.73 | 15.71 | 77.01 | 29.34 | 30.91 | 77.90 | 79.09 | 79.43
Weighted | 84.41 | 84.07 | 94.72 | 58.09 | 87.67 | 94.72 | 95.38 | 95.30
Table 3: Text classification: comparison between average and weighted by class frequencies validation set F1 scores (in %) of the different approaches. The suffix -w indicates the use of class frequency penalty weights. Method |  Average F1 |  Weighted F1
---|---|---
CNN-w | 64.24 | 90.98
CNN | 77.14 | 94.37
Table 4: Image classification: comparison between validation set F1 scores (in %) of the different approaches. The suffix -w indicates the use of class frequency penalty weights. Class | ResNet50-w | ResNet50
---|---|---
Acórdão | 17.68 | 02.50
ARE | 11.56 | 00.00
Despacho | 07.53 | 00.00
Others | 63.01 | 94.77
RE | 32.48 | 33.03
Sentença | 43.46 | 49.20
Average | 29.29 | 29.92
Weighted | 59.13 | 87.09
Table 5: Fusion Module: impact of number of hidden units and learnable embeddings for missing data on average validation set F1 scores (in %). The suffix -zero indicates the use of vector of zeros for missing data (as opposed to using learnable embeddings). Method |  Average F1
---|---
FM-512 | 74.49
FM-512-zero | 68.02
FM-128 | 75.70
FM-128-zero | 72.95
Table 6: Fusion Module ablation, comparing the test set F1 scores (in %) of the hybrid classifier and of a version of the fusion module that ignores image activations (w/o img acts), that is, always uses the missing image embedding. For the hybrid classifier, we report results using both image classifiers: with (HC-w) and without (HC) class frequency penalty. Between parentheses, the difference in performance compared with using the original fusion module (FM). Class | Text test split | Text + image test split
---|---|---
| FM | fusion w/o img acts | FM | HC-w | HC
Acórdão | 90.74 | 88.27 (-2.47) | 88.5 | 41.36 (-47.14) | 87.68 (-0.82)
ARE | 57.92 | 54.09 (-3.83) | 56.6 | 49.02 (-7.58) | 43.91 (-12.69)
Despacho | 63.98 | 62.01 (-1.97) | 63.79 | 42.71 (-21.08) | 61.85 (-1.94)
Others | 97.24 | 97.27 (+0.03) | 97.03 | 95.80 (-1.23) | 97.02 (-0.01)
RE | 75.47 | 73.26 (-2.21) | 75.05 | 72.11 (-2.94) | 75.00 (-0.05)
Sentença | 82.04 | 79.58 (-2.46) | 81.21 | 74.07 (-7.14) | 79.68 (-1.53)
Average | 77.90 | 75.74 (-2.16) | 77.03 | 62.51 (-14.52) | 74.19 (-2.84)
Weighted | 94.72 | 94.47 (-0.25) | 94.32 | 92.58 (-1.74) | 93.95 (-0.37)
Table 7: Sequence classification: comparison between average and weighted by class frequencies validation set F1 scores (in %) of the different approaches.  Method |  Average F1 |  Weighted F1
---|---|---
BiLSTM | 77.16 | 94.25
BiLSTM-CRF | 78.45 | 94.46
BiLSTM-F | 79.03 | 94.81
BiLSTM-F-CRF | 78.87 | 94.58
Table 8: Comparison of first page and not first page of a document classification performance. We report test set F1 scores (in %) for image, text, and fusion classification using as models the CNN, the ResNet50-w and the FM, respectively. Between parentheses, the number of samples. Class | Text | Image | Fusion
---|---|---|---
| First page | Not first page | First page | Not first page | First page | Not first page
Acórdão | 92.47 (199) | 83.66 (74) | 34.28 (197) | 08.24 (88) | 93.40 (199) | 77.19 (88)
ARE | 47.65 (213) | 56.74 (1,628) | 06.71 (203) | 12.10 (2,334) | 59.95 (213) | 56.28 (2,442)
Despacho | 71.54 (147) | 40.43 (51) | 12.59 (146) | 03.68 (52) | 71.81 (147) | 40.45 (52)
Others | 99.02 (25,744) | 96.58 (59,664) | 78.19 (24,193) | 54.29 (63,709) | 99.04 (25,744) | 96.26 (66,789)
RE | 74.45 (312) | 75.65 (6,019) | 18.28 (301) | 33.72 (5,876) | 75.50 (312) | 75.03 (6,074)
Sentença | 81.47 (265) | 80.32 (1,210) | 26.61 (262) | 49.71 (1,216) | 83.11 (265) | 80.78 (1,238)
Average | 77.77 (26,880) | 72.23 (68,646) | 29.44 (25,302) | 26.96 (73,275) | 80.47 (26,880) | 71.00 (76,683)
Weighted | 97.96 (26,880) | 93.46 (68,646) | 75.65 (25,302) | 51.13 (73,275) | 98.11 (26,880) | 93.00 (76,683)

####  4.4.2 BiLSTM

As an alternative method for sequence classification of pages, we use a
bidirectional long short-term (biLSTM) hochreiter1997long  layer to capture
sequential dependencies at the feature extraction level—as opposed to the
FM+CRF baseline, which only does so at the prediction level. We experiment
with two different kinds of input: the activations of the first FC layer of
the FM (128-dimensional vectors); and the concatenation of the pre-computed
image and text embeddings (7,936-dimensional vectors), obtained as described
in Section 4.3.

The network consists of a biLSTM layer with 128 units for each direction
followed by batch normalisation, dropout and an FC layer. When using
concatenated image and text embeddings as input, we first apply batch
normalisation and dropout, followed by an FC layer with 512 units.

We train four model variants:

  1. 1.

BiLSTM, which uses fusion activations as input;

  2. 2.

BiLSTM-CRF, with the same input and a CRF head on top of the described
network;

  3. 3.

BiLSTM-F, which uses concatenated image and text embeddings as input; and

  4. 4.

BiLSTM-F-CRF, with the same input and a CRF head.

Due to the memory footprint of BiLSTM-F and BiLSTM-F-CRF, we use mini-batches
of eight lawsuits when training them. All models are trained for one cycle
cyclelr  of 20 epochs. We use the range test to choose learning rates.

##  5 Results and Discussion

Table 2 exhibits the F1 scores of the best performing models, categorised by
whether they use textual (CNN), visual (ResNet50-w and ResNet50), textual and
visual (FM), or sequential (FM+CRF and BiLSTM-F) information.

All models beat majority class classifiers considering both weighted and
average F1 scores—except for ResNet50-w, whose weighted F1 score is 25.98 p.p.
lower. The models with textual data performed much better than those with only
visual information available, which is not surprising given that text content
is more discriminative than visual aspects when considering the dataset
documents—most of them are similar white pages with blocks of text (Figure 2).

Regarding fusion and sequence classification results, it is clear that each
additional information source contributed to classification metrics:
considering average F1 scores, the FM surpassed the CNN by 0.89 p.p., while
the FM+CRF and BiLSTM-F beat the FM by 1.19 and 1.53 p.p., respectively.

In the paragraphs below we will further examine the results of each category
(text, image, fusion and sequence) and perform an ablation analysis of the
Fusion Module.

###  5.1 Text Classification Results

Table 3 compares the validation performance of our approaches for text
classification. Using class frequency penalty weights to help with data
imbalance did not help: the CNN-w average and weighted scores were 12.90 and
3.39 p.p. lower than its counterpart with no such strategy. Despite using the
same architecture, we achieved better results than the ones reported by Luz de
Araujo et al luzDeAraujo_etal_VICTOR_LREC_2020 . This is probably because we
save model parameters only on validation metric improvement when training.

###  5.2 Image Classification Results

Table 4 compares the validation performance of the image classification
models.

The ResNet50 achieved higher average and weighted scores than its counterpart
that uses class frequency penalty weights. That said, since the ResNet50
scores for Acórdão, ARE and Despacho were zero or close to zero, the
ResNet50-w scores were more equally distributed across the different classes.
With the intuition that this could lead to more discriminative features, we
experiment with both models’ activations when fusing textual and visual data.

###  5.3 Image and Text Combination Results

Table 5 shows the performance of the FM trained for 10 epochs with different
hyperparameter configurations. Using learnable embeddings for missing textual
or visual data proved to be fundamental, improving average F1 scores by 6.47
and 2.75 p.p. for the models with 512 and 128 hidden units, respectively.
While the smaller model performed best, we hypothesise that with further
parameter tuning and longer training the bigger model would surpass it.

Table 6 compares the scores of the alternative fusion approaches with the ones
from the FM. All of them performed much worse, with decreases in average F1
score ranging from 2.16 to 14.52 p.p. These results signal how the increase in
performance seen by the FM is due to the fusion of data sources; not to
different training conditions or model capacity—combining visual and textual
data helps.

###  5.4 Sequence Classification

Table 7 compares the validation performance of the LSTM models. To ensure a
fair comparison to the other approaches, though we use IOB tagging scheme
during training, when reporting results we consider only the original classes.
If a given page is an ARE, for example, the predictions B-ARE and I-ARE would
both be considered correct, regardless of the position of the page in its
lawsuit.

The variants that use as input the image and text embeddings (BiLSTM-F and
BiLSTM-F-CRF) outperformed the ones that use the FM activations (BiLSTM and
BiLSTM-CRF). This suggests that it is beneficial to jointly learn how to
consider sequential dependencies and how to combine multi-modal information.
Surprisingly, the CRF layer helped the BiLSTM model, with an increase in
1.29/0.25 average/weighted F1 scores, but not the BiLSTM-F model. This may be
an artifact of our training settings, with its limited number of training
epochs.

###  5.5 First page evaluation

Table 8 shows the difference in classification performance of samples that are
the first page of a document versus those that are interior pages, considering
all levels of data availability (text, image, and fusion).

The first page sample set obtained average/weighted F1 scores 5.54/4.50,
2.48/24.52 and 9.47/5.11 p.p. higher than its complement, for the text, image
and fusion levels, respectively. These results confirm our hypothesis that the
first pages are more informative from the point of view of both textual and
visual data. Therefore, one possible improvement for page classification of
the legal documents is training under a multi-task setting that jointly learns
to classify pages and establish document boundaries.

##  6 Conclusion

In this paper, we presented a novel dataset of Brazilian lawsuits with visual
and textual data and proposed a method for sequence-aware multimodal
classification of pages from legal documents. Our proposed Fusion Module
combines visual and textual features extracted from convolutional neural
networks trained separately on image and text data. We experiment with two
approaches for sequence classification: post-processing the predictions of the
Fusion Module using a linear-chain conditional random field; and training
bidirectional LSTM models that alternatively use as input Fusion Module
activations or the concatenation of image and text embeddings. Our Fusion
Module outperformed the unimodal models, with an ablation analysis confirming
that improvement is due to the combination of modalities. We find that
learning embeddings for missing visual or textual input is much better than
using a vector of zeroes for such cases. Sequence classification of pages
brought further improvements, with the best performing model jointly learning
how to combine modalities and consider sequential dependencies.

Therefore, future work would include the end-to-end training of the full
pipeline: image and text feature extractors, Fusion Module and sequence
modelling. Moreover, it is worthwhile to explore if transformer-based
vaswani_etal_transformer_nips_2017  text encoders such as BERT bert  and T5
raffel_etal_t5_JMLR_2020  can further improve classification performance.

## Declarations

### Funding

This study was financed in part by the Coordenação de Aperfeiçoamento de
Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001. TdC received
support from Conselho Nacional de Desenvolvimento Científico e Tecnológico
(CNPq), grant PQ 314154/2018-3. We acknowledge the support of “Projeto de
Pesquisa & Desenvolvimento de aprendizado de máquina (machine learning) sobre
dados judiciais das repercussões gerais do Supremo Tribunal Federal - STF”. We
are also grateful for the support from Fundação de Apoio à Pesquisa do
Distrito Federal (FAPDF, project KnEDLe, convênio 07/2019) and Fundação de
Empreendimentos Científicos e Tecnológicos (Finatec). TdC is currently on a
leave of absence from the University of Brasilia and works at Vicon Motion
Systems, Oxford Metrics Group.

### Conflicts of interest

The authors declare that they have no conflict of interest.

### Availability of data

Data used in this work is available at http://ailab.unb.br/victor/lrec2020/.

### Code availability

Code used in this work is available at https://github.com/peluz/victor-visual-
text.

## References

  * (1) Audebert, N., Herold, C., Slimani, K., Vidal, C.: Multimodal deep networks for text and image-based document classification.  CoRR abs/1907.06370 (2019).  URL http://arxiv.org/abs/1907.06370
  * (2) Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information.  arXiv preprint arXiv:1607.04606 (2016)
  * (3) Buitinck, L., Louppe, G., Blondel, M., Pedregosa, F., Mueller, A., Grisel, O., Niculae, V., Prettenhofer, P., Gramfort, A., Grobler, J., Layton, R., VanderPlas, J., Joly, A., Holt, B., Varoquaux, G.: API design for machine learning software: experiences from the scikit-learn project.  In: ECML PKDD Workshop: Languages for Data Mining and Machine Learning, pp. 108–122 (2013)
  * (4) Chen, N., Blostein, D.: A survey of document image classification: problem statement, classifier architecture and performance evaluation.  International Journal of Document Analysis and Recognition (IJDAR) 10(1), 1–16 (2007).  DOI 10.1007/s10032-006-0020-2.  URL https://doi.org/10.1007/s10032-006-0020-2
  * (5) Conneau, A., Schwenk, H., Barrault, L., Lecun, Y.: Very deep convolutional networks for text classification.  In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pp. 1107–1116. Association for Computational Linguistics, Valencia, Spain (2017).  URL http://www.aclweb.org/anthology/E17-1104
  * (6) Deerwester, S., Dumais, S.T., Furnas, G.W., Landauer, T.K., Harshman, R.: Indexing by latent semantic analysis.  Journal of The American Society for Information Science 41(6), 391–407 (1990)
  * (7) Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding.  CoRR abs/1810.04805 (2018).  URL http://arxiv.org/abs/1810.04805
  * (8) Engin, D., Emekligil, E., Oral, B., Arslan, S., Akpınar, M.: Multimodal deep neural networks for banking document classification.  In: International Conference on Advances in Information Mining and Management, pp. 21–25 (2019)
  * (9) He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition.  In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016).  DOI 10.1109/CVPR.2016.90
  * (10) Hochreiter, S., Schmidhuber, J.: Long short-term memory.  Neural computation 9(8), 1735–1780 (1997)
  * (11) Howard, J., Ruder, S.: Universal language model fine-tuning for text classification.  In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 328–339. Association for Computational Linguistics, Melbourne, Australia (2018).  DOI 10.18653/v1/P18-1031.  URL https://www.aclweb.org/anthology/P18-1031
  * (12) Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In: Proceedings of the 32nd International Conference on Machine Learning - Volume 37, pp. 448–456. JMLR.org (2015).  URL http://proceedings.mlr.press/v37/ioffe15.html
  * (13) Jain, R., Wigington, C.: Multimodal document image classification.  In: International Conference on Document Analysis and Recognition (ICDAR), pp. 71–77 (2019).  DOI 10.1109/ICDAR.2019.00021
  * (14) Kingma, D.P., Ba, J.: Adam: A method for stochastic optmisation.  In: International Conference on Learning Representations (ICLR) (2015).  Preprint available at https://arxiv.org/abs/1412.6980
  * (15) Lafferty, J.D., Andrew, M., Pereira, F.C.N.: Conditional random fields: Probabilistic models for segmenting and labeling sequence data.  In: Proceedings of the Eighteenth International Conference on Machine Learning, ICML, pp. 282–289. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (2001)
  * (16) Luz de Araujo, P.H., de Campos, T.E., Ataides Braz, F., Correia da Silva, N.: VICTOR: a dataset for Brazilian legal documents classification.  In: Proceedings of The 12th Language Resources and Evaluation Conference (LREC), pp. 1449–1458. European Language Resources Association, Marseille, France (2020).  URL https://www.aclweb.org/anthology/2020.lrec-1.181
  * (17) Mota, C., Lima, A., Nascimento, A., Miranda, P., de Mello, R.: Classificação de páginas de petições iniciais utilizando redes neurais convolucionais multimodais.  In: Anais do XVII Encontro Nacional de Inteligência Artificial e Computacional, pp. 318–329. SBC, Porto Alegre, RS, Brasil (2020).  DOI 10.5753/eniac.2020.12139.  URL https://sol.sbc.org.br/index.php/eniac/article/view/12139
  * (18) Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer.  Journal of Machine Learning Research 21(140), 1–67 (2020).  URL http://jmlr.org/papers/v21/20-074.html
  * (19) Ramshaw, L.A., Marcus, M.P.: Text chunking using transformation-based learning.  In: Natural language processing using very large corpora, pp. 157–176. Springer (1999).  DOI 10.1007/978-94-017-2390-9˙10.  Preprint available at http://arxiv.org/abs/cmp-lg/9505040
  * (20) Rusiñol, M., Frinken, V., Karatzas, D., Bagdanov, A.D., Lladós, J.: Multimodal page classification in administrative document image streams.  International Journal on Document Analysis and Recognition (IJDAR) 17(4), 331–341 (2014).  DOI 10.1007/s10032-014-0225-8.  URL https://doi.org/10.1007/s10032-014-0225-8
  * (21) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge.  International Journal of Computer Vision (IJCV) 115(3), 211–252 (2015).  DOI 10.1007/s11263-015-0816-y
  * (22) Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: Inverted residuals and linear bottlenecks.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
  * (23) Secretaria de Comunicação Social do Conselho Nacional de Justiça: Sumário executivo do relatório justiça em números 2020 (2018).  URL https://www.cnj.jus.br/wp-content/uploads/2020/08/WEB_V2_SUMARIO_EXECUTIVO_CNJ_JN2020.pdf
  * (24) Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition.  In: International Conference on Learning Representations (2015)
  * (25) Smith, L.N.: Cyclical learning rates for training neural networks.  In: IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 464–472 (2017).  DOI 10.1109/WACV.2017.58
  * (26) Smith, L.N., Topin, N.: Super-convergence: Very fast training of neural networks using large learning rates.  CoRR abs/1708.07120 (2017).  URL http://arxiv.org/abs/1708.07120
  * (27) Smith, R.: An overview of the Tesseract OCR engine.  In: Ninth International Conference on Document Analysis and Recognition (ICDAR), vol. 2, pp. 629–633. IEEE (2007)
  * (28) Supremo Tribunal Federal: Ministra Cármen Lúcia anuncia início de funcionamento do Projeto Victor, de inteligência artificial (2018).  URL http://www.stf.jus.br/portal/cms/verNoticiaDetalhe.asp?idConteudo=388443
  * (29) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.u., Polosukhin, I.: Attention is all you need.  In: I. Guyon, U.V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Garnett (eds.) Advances in Neural Information Processing Systems 30, pp. 5998–6008. Curran Associates, Inc. (2017).  URL http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf
  * (30) Wiedemann, G., Heyer, G.: Multi-modal page stream segmentation with convolutional neural networks.  Language Resources and Evaluation (2019).  DOI 10.1007/s10579-019-09476-2


# T-Controllability of Evolution Systems having Non-instantaneous Impulses

∗Vishant Shah1 Corrorsponding Author  Kalpesh Bharvad2 Dipsha Bhadrecha3 Aashi
Maratha3 Jaita Sharma4 Dhanesh Patel5
Department of Applied Mathematics  The M. S. University of Baroda  Vadodara
India-390001
Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>jaita.sharma-
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In this manuscript, we have considered the system governed by a non-
instantaneous impulsive dynamical system of integer ordered with classical and
non-local conditions and derived sufficient conditions for the trajectory
controllability of the system on the Banach space. The conditions were
obtained through the concept of semi-group properties of operators and
Gronwall’s inequality. Finally, illustrations with the classical and non-local
conditions were also added to validate the derived conditions.

Keywords- Dynamical systems, Non-instantaneous impulses, Trajectory
Controllability, Classical conditions, Non-local conditions

AMS Subject Classification- 4K35, 34K45, 93B05, 93C25.

##  1 Introduction

Impulsive differential equations play a very important role in studying the
behavior of the phenomenon having abrupt changes in the physical problems. If
the changes are at a particular time moment then it is called instantaneous
impulsive differential equations and the changes are over the small intervals
then it is called non-instantaneous impulsive differential equations. There is
a wide range of applications of these impulsive evolution equations in all the
fields of sciences namely, physical sciences, biological science,
environmental sciences. These applications are found in the monograph [1, 2,
3] and research articles [4, 5, 6, 7, 8] and references their in.Qualitative
properties like existence and uniqueness of solution and continuity of the
solutions of instantaneous and non-instantaneous differential or integro-
differential or evolution equations are found in research articles with
initial conditions are found in research articles [9, 10, 11, 12, 13, 14, 15,
16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28] and references their in.

Controlllability is one of the important qualitative property of the systems
in which, one has to find the controllers which drives the system from given
initial state to desired final state or nearer to to the desired final state.
The notion of the modern control theory for the finite dimensional linear
system was first introduced by Kalmann. He used the concept of state space
analysis to find the controller for the finite dimensional linear systems.
Study of complete or exact cotrollability for the linear, semi-linear and non-
linear finite and infinite dimensional systems using various techniques of
operator theory is found in the monographs [29, 30, 31] and articles [32, 33,
34, 35, 36, 37] and reference their in. In the concept of complete or exact
controllability for the system one has to find the controller which steers the
system from given initial state to desire final state. But this type of
controller may not be cost effective. To overcome this situation one has to
find a controller which steers the system from given initial state to desired
final state according to our requirements and study of finding a controller
which steers the system to desired final state is called Trajectory
controllability. The concept of trajectory controllability for a finite
dimensional system was introduced by George [39] and extended to infinite
dimensional system by Chalishajar, et. al. [40] using the concept of operator
semi-group and fixed point theorems. The notion of trajectory controllability
of semi-linear system with non-instantaneous impulses with classical and non-
local conditions using the operator semi-group and Gronwall’s inequality found
in the article [41].

This manuscript established the trajectory controllability of the non-
instantaneous system:

$\displaystyle x^{\prime}(\tau)$
$\displaystyle=A(\tau)x(\tau)+F(\tau,x(\tau))+W_{1}(\tau),$
$\displaystyle\quad\tau\in[s_{k},t_{k}+1),$ $\displaystyle\ for\ all\
k=0,1,2\cdots,p$ (1.1) $\displaystyle x^{\prime}(\tau)$
$\displaystyle=A(\tau)x(\tau)+G(\tau,x(\tau))+W_{2}(\tau),$
$\displaystyle\quad\tau\in[t_{k},s_{k}),$ $\displaystyle\ for\ all\
k=1,2,\cdots,p,$

with local condition $x(0)=x_{0}$ and non-local condition $x(0)=x_{0}-h(x)$.

The structure of the article is as follows:

  * 1.

Definitions of various controllability are discussed in section (2).

  * 2.

Section (3) discussed the trajectory controlllability for the instantaneous
impulsive system with classical conditions.

  * 3.

Trajectory controllability for the instantaneous system with non-local
conditions is discussed in the section (4).

  * 4.

Finally, conclusion for the work is found in section (5)

##  2 Preliminaries

This section discusses definitions and prepositions to established trajectory
controllability of the system governed by non-instantaneous impulsive
evolution equation with classical as well nonlocal conditions.

######  Definition 2.1.

The system (1.1) is completely controllable on the interval
$\mathcal{J}=[0,T_{0}]$ if for any $x_{0},x_{1}\in\mathcal{X}$, if there exist
a control $W(\cdot)$n in $\mathcal{U}$ (control space) steers the system from
$x_{0}$ at $\tau=0$ to $x_{1}$ at $\tau=T_{0}$.

In the definition of complete controllablity, there is no information of the
path or trajectory on which the given system to be driven. Sometimes this
leads to high cost So to overcome this situation we select the path or
trajectory (having minimum cost) under which control system drives from
$x_{0}$ to $x_{1}$ over the interval $\mathcal{J}$. Searching of controller
$W(\cdot)$ in a way that the system drive from $x_{0}$ to $x_{1}$ over the
interval is called trajectory controllability of the system. Therefore,
trajectory controllability of the system is strongest amongst all types of
controllability.

######  Definition 2.2.

Let, $\mathcal{C}_{\mathcal{T}}$ be the set of all trajectories under which
the system (1.1) drives from $x_{0}$ to $x_{1}$ over the interval
$\mathcal{J}$. The system (1.1) is trajectory controllable if for any
$z\in\mathcal{C}_{\mathcal{T}}$, there is a controller
$W(\cdot)\in\mathcal{U}$ such that state of the system $x(\tau)$ drives on
prescribed trajectory $z(\tau)$. This means $x(\tau)=z(\tau)$ a.e. over the
interval $\mathcal{J}$.

##  3 T-controllability with classical conditions

Consider the system governed by the non-instantaneous impulsive evolution
equation

$\displaystyle x^{\prime}(\tau)$
$\displaystyle=A(\tau)x(\tau)+F(\tau,x(\tau))+W_{1}(\tau),\quad\tau\in[s_{k},t_{k+1})$
(3.1) $\displaystyle x^{\prime}(\tau)$
$\displaystyle=A(\tau)x(\tau)+G(\tau,x(\tau))+W_{2}(\tau),\quad\tau\in[t_{k},s_{k})$
$\displaystyle x(0)$ $\displaystyle=x_{0}$

over the interval $[0,T_{0}]$. Here, $x(\tau)$ is the state of the system lies
in Banach space $\mathcal{X}$ at any time $\tau\in[0,T_{0}]$, $A(\tau)$ at any
time $\tau$ is a linear operator on the Banach space $\mathcal{X}$,
$F,G:[0,T_{0}]\times\mathcal{X}\rightarrow\mathcal{X}$ are nonlinear
functions.

To discuss trajectory controllability of the system governed by non-
instantaneous impulsive evolution equation (3.1), we have following theorem:

######  Theorem 3.1.

If,

  * (A1)

Linear operator $A$ in the system (3.1) infinitesimal generator of $C_{0}$
semigroup.

  * (A2)

The non-linear map $F:[0,T_{0}]\times\mathcal{X}\rightarrow\mathcal{X}$ is
continuous such that there exist a non-decreasing function
$l_{F}:\mathbb{R}_{+}\rightarrow\mathbb{R}+$ and positive real number $r_{0}$
satisfying

$||F(\tau,x_{1})-F(\tau,x_{2})||\leq l_{F}(r)||x_{1}-x_{2}||,$

for all $\tau\in[0,T_{0}]$, $x_{1},x_{2}\in B_{r}(\mathcal{X})$ and $r\leq
r_{0}$.

  * (A3)

The non-linear map $G:[0,T_{0}]\times\mathcal{X}\rightarrow\mathcal{X}$ is
continuous such that there exist a non-decreasing function
$l_{G}:\mathbb{R}_{+}\rightarrow\mathbb{R}+$ and positive real number $r_{0}$
satisfying

$||G(\tau,x_{1})-G(\tau,x_{2})||\leq l_{G}(r)||x_{1}-x_{2}||,$

for all $\tau\in[0,T_{0}]$, $x_{1},x_{2}\in B_{r}(\mathcal{X})$ and $r\leq
r_{0}$.

Then, the system (3.1) is trajectory controllable over the interval
$[0,T_{0}]$.

###### Proof.

Let $u(\tau)$ be any trajectory in $\mathcal{C}_{\mathcal{\tau}}$ satisfying
$x(t^{+}_{k})=u(t^{+}_{k})$ along which the system (3.1) steered from initial
state $x_{0}$ at $\tau=0$ to desired final state $x_{1}$ at $\tau=T_{0}$.

Over the interval $[0,t_{1})$, the system (3.1) becomes:

$\displaystyle x^{\prime}(\tau)$
$\displaystyle=A(\tau)x(\tau)+F(\tau,x(\tau))+W_{1}(\tau)$ (3.2)
$\displaystyle x(0)$ $\displaystyle=x_{0}$

Consider

$W_{1}(\tau)=u^{\prime}(\tau)-A(\tau)-F(\tau,u(\tau))$

over the interval $[0,t_{1})$ in and plugging it in the (3.2) the system (3.2)
becomes

$x^{\prime}(\tau)=A(\tau)x(\tau)+F(\tau,x(\tau))+u^{\prime}(\tau)-A(\tau)-F(\tau,u(\tau))$

with initial condition $x(0)-u(0)=0$.
Choosing variable $z=x-u$ the equation system reduces to

$\displaystyle z^{\prime}(\tau)$
$\displaystyle=A(\tau)z(\tau)+F(\tau,x(\tau))-F(\tau,u(\tau))$ (3.3)
$\displaystyle z(0)$ $\displaystyle=0$

and problem of trajectory controllability of the system (3.2) is reduced to
the solvability of the system (3.3) over the interval $[0,t_{1})$. The mild
solution of the system (3.3) is given by:

$z(\tau)=\int^{\tau}_{0}\mathcal{T}(\tau-\zeta)[F(\zeta,x(\zeta))-F(\zeta,u(\zeta))]d\zeta$
(3.4)

where, $\mathcal{T}(\tau)$ is $C_{0}$ semi-group generated by linear operator
$A$ satisfying $||\mathcal{T}(\tau)||\leq M$ for some positive number $M$.
Therefore,

$\displaystyle||z(\tau)||$
$\displaystyle\leq\int^{\tau}_{0}||\mathcal{T}(\tau-\zeta)||\
||[F(\zeta,x(\zeta))-F(\zeta,u(\zeta))]||d\zeta$ $\displaystyle\leq
M\int^{\tau}_{0}l_{F}(r)||x(\zeta)-u(\zeta)||d\zeta$ $\displaystyle\leq
M\int^{\tau}_{0}l_{F}(r)||z(\zeta)||d\zeta$

and using Gronwall’s inequality, we obtain $z(\tau)=0$ over the interval
$[0,t_{1})$. Hence, $x(\tau)=u(\tau)$ for all $\tau\in[0,t_{1})$. Therefore,
the system is trajectory controllable over the interval $[0,t_{1})$.

Over the interval $[t_{1},s_{1})$, the system becomes:

$\displaystyle x^{\prime}(\tau)$
$\displaystyle=A(\tau)x(\tau)+G(\tau,x(\tau))+W_{2}(\tau),$ (3.5)
$\displaystyle x(t_{1})$ $\displaystyle=u(t_{1})$

as the state of the system satisfies $x(t_{k})=u(t_{k})$. Plugging the control
$W_{2}(\tau)=u^{\prime}(\tau)-A(\tau)u(\tau)-G(\tau,x(\tau))$ over the
interval $[t_{1},s_{1})$ in the system (3.5) the system becomes:

$x^{\prime}(\tau)-u^{\prime}(\tau)=G(\tau,x(\tau))-G(\tau,u(\tau))$

Choosing $z(\tau)=x(\tau)-u(\tau)$ we obtain

$z^{\prime}(\tau)=G(\tau,x(\tau))-G(\tau,u(\tau))$

as the value of the $z$ at $\tau=t_{1}$ is zero. Therefore, we have

$\displaystyle||z(\tau)||$
$\displaystyle\leq\int^{\tau}_{t_{1}}||\mathcal{T}(\tau-\zeta)||\
||[G(\zeta,x(\zeta))-G(\zeta,u(\zeta))]||d\zeta$ $\displaystyle\leq
M\int^{\tau}_{t_{1}}l_{G}(r)||x(\zeta)-u(\zeta)||d\zeta$ $\displaystyle\leq
M\int^{\tau}_{t_{1}}l_{G}(r)||z(\zeta)||d\zeta$

using (A3) and Gronwall’s inequality we obtain $z(\tau)=0$ for all
$\tau\in[t_{1},s_{1})$ Therefore, system (3.1) is T-Controllable over the
interval $[t_{1},s_{1})$. Moreover, $z(s_{1})=0$ as $G$ is continuous.

Continuing this process over the interval $[s_{k},t_{k+1})$ the system (3.1)
becomes:

$\displaystyle x^{\prime}(\tau)$
$\displaystyle=A(\tau)x(\tau)+F(\tau,x(\tau))+W_{1}(\tau)$ (3.6)
$\displaystyle x(s_{k})$ $\displaystyle=u(s_{k})$

Choose the control over the interval $[s_{k},t_{k+1}$ as:

$W_{1}(\tau)=u^{\prime}(\tau)-A(\tau)-F(\tau,u(\tau))$

and plugging it in the equation (3.6) we get,

$x^{\prime}(\tau)=A(\tau)x(\tau)+F(\tau,x(\tau))+u^{\prime}(\tau)-A(\tau)-F(\tau,u(\tau))$

considering $z(\tau)=x(\tau)-u(\tau)$, above expression becomes:

$\displaystyle z^{\prime}(\tau)$
$\displaystyle=A(\tau)z(\tau)+F(\tau,x(\tau))-F(\tau,u(\tau))$ (3.7)
$\displaystyle z(s_{k})$ $\displaystyle=0,$

Therefore

$\displaystyle||z(\tau)||$
$\displaystyle\leq\int^{\tau}_{s_{k}}||\mathcal{T}(\tau-\zeta)||\
||[F(\zeta,x(\zeta))-F(\zeta,u(\zeta))]||d\zeta$ $\displaystyle\leq
M\int^{\tau}_{0}l_{F}(r)||x(\zeta)-u(\zeta)||d\zeta$ $\displaystyle\leq
M\int^{\tau}_{0}l_{F}(r)||z(\zeta)||d\zeta$

and using Gronwall’s inequality, we obtain $z(\tau)=0$ over the interval
$[s_{k},t_{k+1})$. Hence, $x(\tau)=u(\tau)$ for all $\tau\in[s_{k},t_{k+1})$.
Therefore, the system is T- controllable over the interval $[s_{k},t_{k+1})$.

Over the interval $[t_{k},s_{k})$ the system becomes:

$\displaystyle x^{\prime}(\tau)$
$\displaystyle=A(\tau)x(\tau)+G(\tau,x(\tau))+W_{2}(\tau),$ (3.8)
$\displaystyle x(t_{k})$ $\displaystyle=u(t_{k})$

Plugging the control
$W_{2}(\tau)=u^{\prime}(\tau)-A(\tau)u(\tau)-G(\tau,x(\tau))$ over the
interval $[t_{k},s_{k})$ in the system (3.8) the system becomes:

$x^{\prime}(\tau)-u^{\prime}(\tau)=G(\tau,x(\tau))-G(\tau,u(\tau))$

Choosing $z(\tau)=x(\tau)-u(\tau)$ we obtain

$z^{\prime}(\tau)=G(\tau,x(\tau))-G(\tau,u(\tau))$

as the value of the $z$ at $\tau=t_{k}$ is zero. Therefore, we have

$\displaystyle||z(\tau)||$
$\displaystyle\leq\int^{\tau}_{t_{k}}||\mathcal{T}(\tau-\zeta)||\
||[G(\zeta,x(\zeta))-G(\zeta,u(\zeta))]||d\zeta$ $\displaystyle\leq
M\int^{\tau}_{t_{k}}l_{G}(r)||x(\zeta)-u(\zeta)||d\zeta$ $\displaystyle\leq
M\int^{\tau}_{t_{k}}l_{G}(r)||z(\zeta)||d\zeta$

using (A3) and Gronwall’s inequality we obtain $z(\tau)=0$ for all
$\tau\in[t_{k},s_{k})$ Therefore, system (3.1) is T-Controllable over the
interval $[t_{k},s_{k})$.

Since, the system is T-controllable over the intervals
$[0,t_{1}),[t_{1},s_{1}),[s_{k},t_{k+1})$ and $[t_{k},s_{k})$ for all $k$.
Hence, the system is controllable over entire interval $[0,T_{0}]$. This
completes the proof of the theorem. ∎

######  Example 3.1.

Let, $\mathcal{X}=L^{2}([0,\pi],\mathbb{R})$ and consider the system governed
by non-instantaneous impulsive evolution equation:

$\displaystyle\frac{\partial H(\tau,\psi)}{\partial\tau}$
$\displaystyle=\partial^{2}_{\psi}H(\tau,\psi)+F(\tau,H(\tau,\psi))+w_{1}(\tau,\psi)$
$\displaystyle\quad\tau\in[0,1/3)\cup[2/3,1],$ (3.9)
$\displaystyle\frac{\partial H(\tau,\psi)}{\partial\tau}$
$\displaystyle=\partial^{2}_{\psi}H(\tau,\psi)+G(\tau,H(\tau,\psi))+w_{2}(\tau,\psi)$
$\displaystyle\quad\tau\in[1/3,2/3),$ $\displaystyle H(\tau,0)$
$\displaystyle=0\quad H(\tau,\pi)=0$ $\displaystyle\quad\tau>0,$
$\displaystyle H(0,\psi)$ $\displaystyle=H_{0}(\psi)\quad$ $\displaystyle\quad
0<\psi<\pi,$

over the interval $[0,1]$.
Defining the operator on the space $\mathcal{X}$ as
$A(\tau)=\partial^{2}_{\psi}$, $A(\tau)$ is the infinitesimal generator of the
$C_{0}$ semigroup $\mathcal{T}(\tau)$. The representation of
$\mathcal{T}(\tau)$ is

$\mathcal{T}(\tau)z=\sum^{\infty}_{m=0}\exp(\mu_{m}\tau)<z,\phi_{m}>\phi_{m}$

where, $\phi_{m}=\sqrt{2}sin(n\psi)$ for all $m=1,2,\cdots$ is the orthonormal
basis corresponding to eigenvalue $\mu_{m}=-m^{2}$ of the operator $A$.

With this concept the equation (3.9) can be rewritten as and abstract equation
on the space $\mathcal{X}$ as

$\displaystyle x^{\prime}(\tau)$ $\displaystyle=A(\tau)+F(\tau,x)+W_{1}(\tau)$
$\displaystyle\quad\tau\in[0,1/3)\cup[2/3,1],$ (3.10) $\displaystyle
x^{\prime}(\tau)$ $\displaystyle=A(\tau)+F(\tau,x)+W_{2}(\tau)$
$\displaystyle\quad\tau\in[1/3,2/3),$ $\displaystyle x(0)$
$\displaystyle=x_{0},$

where, $x(\tau)=H(\tau,\cdot),W_{1}(\tau)=w_{1}(\tau,\psi)$ and
$W_{2}(\tau)=w_{2}(\tau,\psi)$. The system (3.10) is trajectory controllable
over the interval $[0,1]$ if the functions $F$ and $G$ satisfies the
hypotheses of the theorem.

##  4 T-controllability with non-local conditions

Consider the system governed by the non-instantaneous impulsive evolution
equation

$\displaystyle x^{\prime}(\tau)$
$\displaystyle=A(\tau)x(\tau)+F(\tau,x(\tau))+W_{1}(\tau),\quad\tau\in[s_{k},t_{k+1})$
(4.1) $\displaystyle x^{\prime}(\tau)$
$\displaystyle=A(\tau)x(\tau)+G(\tau,x(\tau))+W_{2}(\tau),\quad\tau\in[t_{k},s_{k})$
$\displaystyle x(0)$ $\displaystyle=h(x)$

over the interval $[0,T_{0}]$. Here, $x(\tau)$ is the state of the system lies
in Banach space $\mathcal{X}$ at any time $\tau\in[0,T_{0}]$, $A(\tau)$ at any
time $\tau$ is a linear operator on the Banach space $\mathcal{X}$,
$F,G:[0,T_{0}]\times\mathcal{X}\rightarrow\mathcal{X}$ are nonlinear functions
and $h:\mathcal{X}\rightarrow\mathcal{X}$ is the operator representing the
non-local conditions. The mild solution of the equation (4.1) is given by

$x(\tau)=\begin{cases}\begin{aligned}
&\mathcal{T}(\tau)h(x)+\int^{\tau}_{0}\mathcal{T}(\tau-\zeta)\big{[}F(\zeta,x(\zeta))+W_{1}(\zeta)\big{]}d\zeta,&\quad\tau\in[0,t_{1})\\\
&\mathcal{T}(\tau)x(t_{k})+\int^{\tau}_{t_{k}}\mathcal{T}(\tau-\zeta)\big{[}G(\zeta,x(\zeta))+W_{2}(\zeta)\big{]}d\zeta,,&\quad\tau\in[t_{k},s_{k})\\\
&\mathcal{T}(\tau)x(s_{k})+\int^{\tau}_{s_{k}}\mathcal{T}(\tau-\zeta)\big{[}F(\zeta,x(\zeta))+W_{1}(\zeta)\big{]}d\zeta,&\quad\tau\in[s_{k},t_{k+1}),\\\
\end{aligned}\end{cases}$ (4.2)

where, $\mathcal{T}(\tau)$ is semigroup generated by the linear operator
$A(\tau)$.

The following theorem discusses the trajectory controllability of the system
governed by the equation (4.1).

######  Theorem 4.1.

If,

  * (A1)

Linear operator $A$ in the system (3.1) infinitesimal generator of $C_{0}$
semigroup.

  * (A2)

The non-linear map $F:[0,T_{0}]\times\mathcal{X}\rightarrow\mathcal{X}$ is
continuous such that there exist a non-decreasing function
$l_{F}:\mathbb{R}_{+}\rightarrow\mathbb{R}+$ and positive real number $r_{0}$
satisfying

$||F(\tau,x_{1})-F(\tau,x_{2})||\leq l_{F}(r)||x_{1}-x_{2}||$

, for all $\tau\in[0,T_{0}]$, $x_{1},x_{2}\in B_{r}(\mathcal{X})$ and $r\leq
r_{0}$.

  * (A3)

The non-linear map $G:[0,T_{0}]\times\mathcal{X}\rightarrow\mathcal{X}$ is
continuous such that there exist a non-decreasing function
$l_{G}:\mathbb{R}_{+}\rightarrow\mathbb{R}+$ and positive real number $r_{0}$
satisfying

$||G(\tau,x_{1})-G(\tau,x_{2})||\leq l_{G}(r)||x_{1}-x_{2}||$

, for all $\tau\in[0,T_{0}]$, $x_{1},x_{2}\in B_{r}(\mathcal{X})$ and $r\leq
r_{0}$.

  * (A4)

The function $h:\mathcal{X}\rightarrow\mathcal{X}$ is Lipchitz continuous with
Lipchitz constant $0\leq l_{h}\leq 1$.

Then, the system (4.1) is trajectory controllable over the interval
$[0,T_{0}]$.

###### Proof.

Let $u(\tau)$ be any trajectory in $\mathcal{C}_{\mathcal{\tau}}$ satisfying
$x(t^{+}_{k})=u(t^{+}_{k})$ along which the system (4.1) steered from initial
state $x(0)=h(x)$ at $\tau=0$ to desired final state $x_{1}$ at $\tau=T_{0}$.

Over the interval $[0,t_{1})$, the system (4.1) becomes:

$\displaystyle x^{\prime}(\tau)$
$\displaystyle=A(\tau)x(\tau)+F(\tau,x(\tau))+W(\tau)$ (4.3) $\displaystyle
x(0)$ $\displaystyle=h(x)$

Consider

$W(\tau)=u^{\prime}(\tau)-A(\tau)-F(\tau,u(\tau))$

over the interval $[0,t_{1})$ and plugging it in the system (4.3), the system
becomes

$x^{\prime}(\tau)=A(\tau)x(\tau)+F(\tau,x(\tau))+u^{\prime}(\tau)-A(\tau)-F(\tau,u(\tau))$

with initial condition $x(0)-u(0)=h(x)-h(u)$.
Choosing variable $z=x-u$ the equation system reduces to

$\displaystyle z^{\prime}(\tau)$
$\displaystyle=A(\tau)z(\tau)+F(\tau,x(\tau))-F(\tau,u(\tau))$ (4.4)
$\displaystyle z(0)$ $\displaystyle=h(x)-h(u)$

and problem of trajectory controllability of the system (4.3) is reduced to
the solvability of the system (4.4) over the interval $[0,t_{1})$. The mild
solution of the system (4.4) is given by:

$z(\tau)=\mathcal{T}(\tau)[h(x)-h(u)]+\int^{\tau}_{0}\mathcal{T}(\tau-\zeta)[F(\zeta,x(\zeta))-F(\zeta,u(\zeta))]d\zeta$
(4.5)

where, $\mathcal{T}(\tau)$ is $C_{0}$ semigroup generated by linear operator
$A$ satisfying $||\mathcal{T}(\tau)||\leq M$ for some positive number $M$.
Therefore,

$\displaystyle||z(\tau)||$
$\displaystyle\leq||\mathcal{T}(\tau)||||h(x)-h(u)||+\int^{\tau}_{0}||\mathcal{T}(\tau-\zeta)||\
||[F(\zeta,x(\zeta))-F(\zeta,u(\zeta))]||d\zeta$ $\displaystyle\leq
Ml_{h}||x(\tau)-u(\tau)||+M\int^{\tau}_{0}l_{F}(r)||x(\zeta)-u(\zeta)||d\zeta$
$\displaystyle\leq
Ml_{h}||z(\tau)||+M\int^{\tau}_{0}l_{F}(r)||z(\zeta)||d\zeta$

This implies

$||z(\tau)||\leq\frac{Ml_{F}(r)}{1-Ml_{h}}\int^{\tau}_{0}||z(\zeta)||d\zeta$

Using Gronwall’s inequality, we obtain $z(\tau)=0$ over the interval
$[0,t_{1})$. Hence, $x(\tau)=u(\tau)$ for all $\tau\in[0,t_{1})$. Therefore,
the system (4.1) is trajectory controllable over the interval $[0,t_{1})$.

Over the interval $[t_{1},s_{1})$, the system becomes:

$\displaystyle x^{\prime}(\tau)$
$\displaystyle=A(\tau)x(\tau)+G(\tau,x(\tau))+W_{2}(\tau),$ (4.6)
$\displaystyle x(t_{1})$ $\displaystyle=u(t_{1})$

as the state of the system satisfies $x(t_{k})=u(t_{k})$. Plugging the control
$W_{2}(\tau)=u^{\prime}(\tau)-A(\tau)u(\tau)-G(\tau,x(\tau))$ over the
interval $[t_{1},s_{1})$ in the system (4.6) the system becomes:

$x^{\prime}(\tau)-u^{\prime}(\tau)=G(\tau,x(\tau))-G(\tau,u(\tau))$

Choosing $z(\tau)=x(\tau)-u(\tau)$ we obtain

$z^{\prime}(\tau)=G(\tau,x(\tau))-G(\tau,u(\tau))$

as the value of the $z$ at $\tau=t_{1}$ is zero. Therefore, we have

$\displaystyle||z(\tau)||$
$\displaystyle\leq\int^{\tau}_{t_{1}}||\mathcal{T}(\tau-\zeta)||\
||[G(\zeta,x(\zeta))-G(\zeta,u(\zeta))]||d\zeta$ $\displaystyle\leq
M\int^{\tau}_{t_{1}}l_{G}(r)||x(\zeta)-u(\zeta)||d\zeta$ $\displaystyle\leq
M\int^{\tau}_{t_{1}}l_{G}(r)||z(\zeta)||d\zeta$

using (A3) and Gronwall’s inequality we obtain $z(\tau)=0$ for all
$\tau\in[t_{1},s_{1})$ Therefore, system (4.1) is T-Controllable over the
interval $[t_{1},s_{1})$. Moreover, $z(s_{1})=0$ as $G$ is continuous.

Continuing this process over the interval $[s_{k},t_{k+1})$ the system (4.1)
becomes:

$\displaystyle x^{\prime}(\tau)$
$\displaystyle=A(\tau)x(\tau)+F(\tau,x(\tau))+W_{1}(\tau)$ (4.7)
$\displaystyle x(s_{k})$ $\displaystyle=u(s_{k})$

Choose the control over the interval $[s_{k},t_{k+1})$ as:

$W_{1}(\tau)=u^{\prime}(\tau)-A(\tau)-F(\tau,u(\tau))$

and plugging it in the equation (4.7) we get,

$x^{\prime}(\tau)=A(\tau)x(\tau)+F(\tau,x(\tau))+u^{\prime}(\tau)-A(\tau)-F(\tau,u(\tau))$

considering $z(\tau)=x(\tau)-u(\tau)$, above expression becomes:

$\displaystyle z^{\prime}(\tau)$
$\displaystyle=A(\tau)z(\tau)+F(\tau,x(\tau))-F(\tau,u(\tau))$ (4.8)
$\displaystyle z(s_{k})$ $\displaystyle=0,$

Therefore

$\displaystyle||z(\tau)||$
$\displaystyle\leq\int^{\tau}_{s_{k}}||\mathcal{T}(\tau-\zeta)||\
||[F(\zeta,x(\zeta))-F(\zeta,u(\zeta))]||d\zeta$ $\displaystyle\leq
M\int^{\tau}_{0}l_{F}(r)||x(\zeta)-u(\zeta)||d\zeta$ $\displaystyle\leq
M\int^{\tau}_{0}l_{F}(r)||z(\zeta)||d\zeta$

and using Gronwall’s inequality, we obtain $z(\tau)=0$ over the interval
$[s_{k},t_{k+1})$. Hence, $x(\tau)=u(\tau)$ for all $\tau\in[s_{k},t_{k+1})$.
Therefore, the system (4.1)is T- controllable over the interval
$[s_{k},t_{k+1})$.

Over the interval $[t_{k},s_{k})$ the system becomes:

$\displaystyle x^{\prime}(\tau)$
$\displaystyle=A(\tau)x(\tau)+G(\tau,x(\tau))+W_{2}(\tau),$ (4.9)
$\displaystyle x(t_{k})$ $\displaystyle=u(t_{k})$

Plugging the control
$W_{2}(\tau)=u^{\prime}(\tau)-A(\tau)u(\tau)-G(\tau,x(\tau))$ over the
interval $[t_{k},s_{k})$ in the system (4.9) the system becomes:

$x^{\prime}(\tau)-u^{\prime}(\tau)=G(\tau,x(\tau))-G(\tau,u(\tau))$

Choosing $z(\tau)=x(\tau)-u(\tau)$ we obtain

$z^{\prime}(\tau)=G(\tau,x(\tau))-G(\tau,u(\tau))$

as the value of the $z$ at $\tau=t_{k}$ is zero. Therefore, we have

$\displaystyle||z(\tau)||$
$\displaystyle\leq\int^{\tau}_{t_{k}}||\mathcal{T}(\tau-\zeta)||\
||[G(\zeta,x(\zeta))-G(\zeta,u(\zeta))]||d\zeta$ $\displaystyle\leq
M\int^{\tau}_{t_{k}}l_{G}(r)||x(\zeta)-u(\zeta)||d\zeta$ $\displaystyle\leq
M\int^{\tau}_{t_{k}}l_{G}(r)||z(\zeta)||d\zeta$

using (A3) and Gronwall’s inequality we obtain $z(\tau)=0$ for all
$\tau\in[t_{k},s_{k})$ Therefore, system (4.1) is T-Controllable over the
interval $[t_{k},s_{k})$.

Since, the system (4.1) is T-controllable over the intervals
$[0,t_{1}),[t_{1},s_{1}),[s_{k},t_{k+1})$ and $[t_{k},s_{k})$ for all $k$.
Hence, the system is controllable over entire interval $[0,T_{0}]$. This
completes the proof of the theorem. ∎

######  Example 4.1.

Let, $\mathcal{X}=L^{2}([0,\pi],\mathbb{R})$ and consider the system governed
by non-instantaneous impulsive evolution equation:

$\displaystyle\frac{\partial H(\tau,\psi)}{\partial\tau}$
$\displaystyle=\partial^{2}_{\psi}H(\tau,\psi)+F(\tau,H(\tau,\psi))+w_{1}(\tau,\psi)$
$\displaystyle\quad\tau\in[0,1/3)\cup[2/3,1],$ (4.10)
$\displaystyle\frac{\partial H(\tau,\psi)}{\partial\tau}$
$\displaystyle=\partial^{2}_{\psi}H(\tau,\psi)+G(\tau,H(\tau,\psi))+w_{2}(\tau,\psi)$
$\displaystyle\quad\tau\in[1/3,2/3),$ $\displaystyle H(\tau,0)$
$\displaystyle=0\quad H(\tau,\pi)=0$ $\displaystyle\quad\tau>0,$
$\displaystyle H(0,\psi)$ $\displaystyle=H(\tau,\psi)\quad$
$\displaystyle\quad 0<\psi<\pi,$

over the interval $[0,1]$. Here, $H(\tau,\psi)$ is nonlocal opertor defined by
$\sum^{n}_{i=1}\alpha_{i}H(\tau_{i},\psi)$.

Defining the operator on the space $\mathcal{X}$ as
$A(\tau)=\partial^{2}_{\psi}$, $A(\tau)$ is the infinitesimal generator of the
$C_{0}$ semigroup $\mathcal{T}(\tau)$. The representation of
$\mathcal{T}(\tau)$ is

$\mathcal{T}(\tau)z=\sum^{\infty}_{m=0}\exp(\mu_{m}\tau)<z,\phi_{m}>\phi_{m}$

where, $\phi_{m}=\sqrt{2}sin(n\psi)$ for all $m=1,2,\cdots$ is the orthonormal
basis corresponding to eigenvalue $\mu_{m}=-m^{2}$ of the operator $A$.

With this concept the equation (4.10) can be rewritten as and abstract
equation on the space $\mathcal{X}$ as

$\displaystyle x^{\prime}(\tau)$ $\displaystyle=A(\tau)+F(\tau,x)+W_{1}(\tau)$
$\displaystyle\quad\tau\in[0,1/3)\cup[2/3,1],$ (4.11) $\displaystyle x(\tau)$
$\displaystyle=A(\tau)+G(\tau,x)+W_{2}(\tau)$
$\displaystyle\quad\tau\in[1/3,2/3),$ $\displaystyle x(0)$
$\displaystyle=h(x),$

where,
$x(\tau)=H(\tau,\cdot),W_{1}(\tau)=w_{1}(\tau,\psi),W_{2}(\tau)=w_{1}(\tau,\psi)$
and $h(x)=\sum^{n}_{i=1}\alpha_{i}x(\tau_{i})$. The system (4.11) is
trajectory controllable over the interval $[0,1]$ if the functions $F,G$ and
$h$ satisfies the hypotheses of the theorem.

##  5 Conclusion

This article discussed the trajectory controllability of the system governed
by non-instantaneous impulsive evolution equation having different perturbing
force $F$ and $G$ with classical as well as non-local conditions on the Banach
space. This type of systems are useful to many physical phenomena where only
the perturbing forces were changes. Illustrations were also discussed to
validate the derived results.

## References

  * [1] V. Lakshmikantham, D. D. Bainov and P. S. Simeonov, Theory of Impulsive Differential Equations, World Scientific, Singapore, 1989.
  * [2] A. M. Samoilenko, N. A Perestyuk, Impulsive Differential Equations, World Scientific, Singapore, 1995.
  * [3] R. Agarwal, S. Hristova, D. O’Regan, Non-Instantaneous Impulses in Differential Equations, Springer International Publishing, Switzerland, 2017
  * [4] D. D. Bainov and A. B. Dishliev, Population dynamics control in regard to minimizing the time necessary for the regeneration of a biomass taken away from the population, Appl.Math.Comput. 39 (1990), 37–48.
  * [5] L. H. Erbe, H. I. Freedman, X. Z. Liu and J. H. Wu, Comparison principles for impulsive parabolic equations with applications to models of single species growth, J. Austral. Math. Soc. 32(1991), 382–400.
  * [6] M. Kirane and Yu. V. Rogovchenko, Comparison results for systems of impulse parabolic equations with applications to population dynamics, Nonlinear Anal. 28 (1997), 263–277.
  * [7] E. Joelianto, Linear impulsive differential equations for hybrid system modelling, in: 2003 European Control Conference(ECC), pp. 3335–3340, Cambridge, UK, 1–4 Sept. 2003.
  * [8] K. Dishlieva, Impulsive differential equations and applications, J. Appl. Comput. Math. 2016 (2016), 1–6.
  * [9] V. Milman and A. Myshkis, On the stability of motion in the presence of impulses, Siberian Math. J. 1 (1996), 233–237.
  * [10] Y. V. Rogovchenko, Nonlinear impulse evolution systems and applications to population models, J. Math. Anal. App. 207 (1997), 300–315.
  * [11] Y. Rogovchenko, Impulsive evolution systems: main results and new trends, Dynamics Contin. Diser. Impulsive Sys. 3(1997), 57–88.
  * [12] J. H. Liu, Nonlinear impulsive evolution equations, Dynamics Contin. Diser. Impulsive System 6 (1999), 77–85.
  * [13] H. Yang, Positive solutions for the initial value problems of impulsive evolution equations, J. Math. Res. Exposition 31(2011), 1047–1056.
  * [14] X. Xiang, Y. Peng and W. Wei, A general class of nonlinear impulsive integral differential equations and optimal control on Banach space, Discrete Cont. Dynemical Syst. 2005 (2005), 911–919.
  * [15] A. Anguraj and M. M. Arjunan, Existence and uniqueness of mild and classical solutions of impulsive evolution equations, Ele. J. Diff. Eq. 111 (2005), 1–8.
  * [16] X. Zhang, Solutions of semilinear evolution equations of mixed type in Banach spaces, Acta Anal. Funct. Appl. 11 (2009), 363–368.
  * [17] P. Sattayatham, Strongly nonlinear impulsive evolution equations and optimal control, Nonlinear Anal. 57 (2004), 1005–1020.
  * [18] J. Liang, J. H. Liu and T.-J. Xiao, Nonlocal impulsive problems for nonlinear differential equations in Banach spaces, Math.v Comput. Modell. 49 (2009), 798–804.
  * [19] Z. Fan and G. Li, Existence results for semilinear differential equations with nonlocal and impulsive conditions, J. Funct. Anal. 258 (2010), 1709–1727.
  * [20] S. Wen and S. Ji, An existence theorem of impulsive differential equations with nonlocal conditions, Int. J. Nonlinear Sci. 14 (2012), 186–192.
  * [21] Q. Tang and J. J. Nieto, Variational approach to impulsive evolution equations, Appl. Math. Lett. 36 (2014), 31–35.
  * [22] P. Chen and J. Mu, Monotone iterative method for semilinear impulsive evolution equation of mixed type in Banach space, Electron. J. Differ. Eq. 2010 (2010), 1–13.
  * [23] P. Chen and Y. Li, Mixed monotone iterative technique for a class of semilinear impulsive evolution equations in Banach spaces, Nonlinear Anal. 74 (2011), 3578–3588.
  * [24] P. Chen, Y. Li and H. Yang, Perturbation method for nonlocal impulsive evolution equations, Nonlinear Anal. Hybrid Syst. 8(2013), 22–30.
  * [25] T. Cardinali and P. Rubbioni, Mild solutions for impulsive semilinear evolution differential equation, J. Appl. Funct. Anal. 1 (2006), 303–325.
  * [26] V. Shah, J. George, J. Sharama and P. Muthukumar, Existence and Uniqueness of Classical and Mild Solutions of Generalized Impulsive Evolution Equation, Int. J. Nonlinear Sci. Numer. Simul. 19 (2018), 775–780.
  * [27] E. Hernandez, M. Pierri, D. O’Regan, On Abstract Differential Equations with Non instantaneous impulses, Topological Methods in Nonlinear Analysis, 46(2015), 1067–1088.
  * [28] A. Sood, S. Srivastava, On Stability of Differential Systems with Noninstantaneous Impulses, Mathematical Problems in Engineering, 2015 (2015), 1-5.
  * [29] D. L. Russell, Mathematics of Finite-dimensional Control Systems: Theory and Design, M. Dekker, New York, 1979.
  * [30] E. D. Sontag, Deterministic Finite Dimensional Systems, Springer, 1998.
  * [31] R. W. Brockett, Finite Dimensional Linear Systems, John Wiley & Sons, USA 1970.
  * [32] M. C. Joshi, R. K. George, Controllability of Nonlinear Systems, Numerical Functional Analysis & Optimization, 10(1989), 139-166.
  * [33] J. Klamka, Schauder’s fixed-point theorm in Nonlinear Controllability problems, Control and Cybernetics, 29(2000), 153-165.
  * [34] Z. G. Li, Y. Wen and Y. C. Soh, Analysis and design of impulsive control, IEEE Trans. Autom. Control 46(2001), 894–897.
  * [35] R. K. George, Approximate controllability of non-autonomous semilinear systems, Nonlinear Anal., 24 (1995), 1377-1393.
  * [36] J. Klamka, A. Babiarz and M. Niezabitowski, Banach fixed-point theorem in semilinear controllability problems–a survey, Bull. Polish Acad. Sci. Tech. Sci., 64 (2016), 21-35.
  * [37] J. Klamka, A. Babiarz and M. Niezabitowski, Schauder’s fixed point theorem in approximate controllability problems, Int. J. Appl. Math. Comput. Sci., 26 (2016), 263-275.
  * [38] P. Chen, X. Zhang and Y. Li, Approximate controllability of non-autonomous evolution system with nonlocal conditions, J. Dyn. Control Syst., 26 (2020), 1-16.
  * [39] R. K. George, Trajectory controllability of 1-dimensional nonlinear systems, Proceedings of the Research Seminar in honour of Professor M.N. Vasavada , Anand, India: S.P. University 1996 (pp. 43–48).
  * [40] D. N. Chalishajar, R. K. George, A. K. Nandkumaran, F. S. Acharya, Trajectory controllability of nonlinear integro-differential system, Journal of The Franklin Institute, 347(2010), 1065-1075.
  * [41] V. Shah, J. Sharma, P. H. Patel, Trajectory Controllability of Dynamical Systems with Non-instantaneous Impulses, 20(2021), 371-381.


# Hierarchical Reinforcement Learning of Locomotion Policies in Response to
Approaching Objects: A Preliminary Study

Shangqun Yu
Department of Computer Science
Brown University
Providence, RI 02906
<EMAIL_ADDRESS>
&Sreehari Rammohan
Department of Computer Science
Brown University
Providence, RI 02906
<EMAIL_ADDRESS>
Kaiyu Zheng
Department of Computer Science
Brown University
Providence, RI 02906
<EMAIL_ADDRESS>
&George Konidaris
Department of Computer Science
Brown University
Providence, RI 02906
<EMAIL_ADDRESS>

###### Abstract

Animals such as rabbits and birds can instantly generate locomotion behavior
in reaction to a dynamic, approaching object, such as a person or a rock,
despite having possibly never seen the object before and having limited
perception of the object’s properties. Recently, deep reinforcement learning
has enabled complex kinematic systems such as humanoid robots to successfully
move from point A to point B. Inspired by the observation of the innate
reactive behavior of animals in nature, we hope to extend this progress in
robot locomotion to settings where external, dynamic objects are involved
whose properties are partially observable to the robot. As a first step toward
this goal, we build a simulation environment in MuJoCo where a legged robot
must avoid getting hit by a ball moving toward it. We explore whether prior
locomotion experiences that animals typically possess benefit the learning of
a reactive control policy under a proposed hierarchical reinforcement learning
framework. Preliminary results support the claim that the learning becomes
more efficient using this hierarchical reinforcement learning method, even
when partial observability (radius-based object visibility) is taken into
account.

Keywords:

locomotion, reactive control, hierarchical reinforcement learning

##  1 Introduction

Animals have the ability to command their body—a complex kinematic
system—quickly in response to environment stimuli [Pavlov, 2010]. When being
approached by a person from an unexpected direction, a rabbit can immediately
run away, even if the rabbit has never seen a person before. When being thrown
a rock, a bird can very quickly fly away to avoid being hit, even if the rock
is moving very fast and the bird does not notice until near collision.

Our work draws a connection to both this observation of nature and the
progress of deep reinforcement learning (RL), where we noticed remarkable
progress in using deep RL for locomotion [Peng et al., 2016, 2020; Tassa et
al., 2018]. Many of the existing works in deep RL for locomotion control only
consider controlling the agent from point to point. Works that do consider
controlling an external object (e.g. dribbling) assume full observability of
the object’s properties permitted for the application of animation [Peng et
al., 2017]. Other works present empirical evidence for the viability of
training end-to-end locomotion policies directly from pixel input [Tassa et
al., 2018], but they typically use third-person images with high sample
complexity, which creates challenges for practical implementation. In this
work, we focus on learning a reactive locomotion policy, which contrasts the
settings in prior works where the agent is typically trained to proactively
complete some task.

Our goal is to extend the success of deep reinforcement learning for
locomotion control to settings where the agent must react to external objects
under partial observability. As a first step, we consider a task where the
agent must react to avoid being hit by an approaching ball (the “Dodge Ball
Task”). The only reward signal the agent gets is a negative reward when being
hit by the ball. Although reinforcement learning has achieved great
improvements in domains such as games [Mnih et al., 2015] and continuous
control for robotics [Gu et al., 2017], learning a policy in the continuous
control setting with sparse rewards is still a major challenge in RL [Li et
al., 2019]. Hierarchical Reinforcement Learning (HRL), with its structured
policy and decomposition of problems into smaller subproblems [Levy et al.,
2019], not only has shown strength under these challenging settings, but also
provides a solution to reuse low-level skill modules on different tasks [Li et
al., 2019]. Therefore, we investigate the benefits of having prior locomotion
experience for this task under an HRL framework. Our preliminary results have
shown that a two level feudal hierarchical agent with pre-trained low-level
controller can solve the task with high sample efficiency while the end to end
agent completely fails to learn with even five times the training samples.

##  2 Hierarchical Reinforcement Learning for Legged Reactive Control

A legged reactive control task can be formulated as a partially observable
sequential decision-making problem. The environment state $s$ can be broken
into two parts, that is, $s=[s_{o},s_{h}]$, where $s_{o}$ represents the fully
observable internal state of the agent, and $s_{h}$ represents the state of
the external object hidden to the agent. At each time step, the agent executes
an action $a$ to control its joint velocities, and it receives an observation,
denoted $z=[s_{o},\phi(s_{h})]$, as a function of the state as a result of the
action. The task is specified via a reward function $R(s)$. We are interested
in the standard reinforcement learning objective, where the agent needs to
maximize the cumulative reward as it interacts with the environment.

In our preliminary investigation of the reactive control setting, the
objective is to minimize the expected number of collisions $\mathbb{E}[n]$ by
developing an optimal policy $\pi^{*}(a|z)$ for an agent in a world with a
single object, which takes in the current state, and outputs joint velocities
for the robot. To this end, we develop the Dodge Ball Task, shown in the
center of Figure 1, implemented in MuJoCo. The agent (an $8$ degree of freedom
ant) must learn to dodge a ball which continuously re-spawns from different
locations before being shot at the agent along a linear path. In the
environment, we define a sparse reward function for our task based on whether
a collision happens. Negative reward is only assigned when the robot is hit by
the projectile. One episode lasts $1000$ steps and a new ball is spawned
randomly in a position that is 5 meters away from the agent and fired at the
agent at a speed of 2 meters/second every $100$ steps. An agent following the
optimal policy will be able to dodge all balls and thus will have a final
cumulative reward close to $0$.

$R(s)=\begin{cases}$-5$&\text{agent hit by ball}\\\ $0$&\text{otherwise}\\\
\end{cases}$

We propose a two level Feudal Reinforcement Learning agent shown in Figure 2.
Feudal Reinforcement Learning (Feudal RL) is a type of Hierarchical
Reinforcement Learning where a high-level controller sets a subtask that is
executed by a lower-level controller [Dayan and Hinton, 1992; Pateria et al.,
2021]. The state space $S=S_{h}\times S_{o}$ consists of the positions and
velocities of the agent’s joints as well as the position and velocity of the
projectile ball. Both high-level and low-level controllers use SAC (soft-actor
critic) [Haarnoja et al., 2018] as a base learning algorithm for the task
objective.

Figure 1: Left: The environment used to train the low-level controller. A
target location is defined randomly near the agent (every $100$ steps). The
agent will gain reward proportional to the distance it traveled toward the
goal. Center: In the dodge ball task, a single ball is spawned randomly in the
environment (every $100$ steps) and shot toward the agent along a linear
trajectory. The agent receives a $-5$ reward every time it gets hit by the
ball. Right: in the partially observable version of the dodge ball task, the
agent does not receive information about the ball until it is within the field
of view (a circle with $4$ meter radius ).

The high-level controller receives the observation $z=[s_{o},\phi(s_{h})]$ and
outputs a subgoal $g$ to the low-level controller. A subgoal is a 2D point
with coordinates relative to the agent. The low-level controller then receives
this intent along with the fully observable part of the state $s_{o}$ and
outputs an action consisting of joint velocities for the $8$-DOF agent.

Similar to how animals possess prior locomotion skills such as running or
flying, we first train the low-level controller in a different environment to
gain basic locomotion skills for the agent. This environment, shown on the
left side of Figure 1, requires the agent to move along a randomly specified
direction vector within $300$ steps, receiving reward proportional to the
distance travelled in this direction. The agent receives an observation
consisting of the internal configuration (joint positions and velocities)
along with a subgoal $g$ with the normalized direction vector. After training,
the low-level controller is able to move the agent forward along an arbitrary
direction specified by $g$.

Figure 2: The high-level controller receives a state $s$ from the environment
and outputs a subgoal $g$ to the low-level controller consisting of the
direction vector it wants the agent to move along. The low-level controller
takes in this subgoal and the internal state $s_{o}$ and outputs a primitive
action for execution in the environment.

Next, the low-level controller is ”attached” to the high-level controller in
the sense that the high-level controller outputs latent intents (similar to a
subgoal direction $g$) which are passed into the low-level controller to
compute the primitive action $a$. Note, because the low-level controller is
trained agnostic to the high-level task, this module can be re-used – all that
needs to be re-trained is the high-level controller.

##  3 Preliminary Experiments

###  3.1 Experiment Setup

We evaluate the performance of our algorithm in both fully and partially
observable settings (Figure 1). In the fully observable setting, the agent
will receive the complete information of the state $s$, including
$[s_{o},s_{h}]$. The full state of the ball $s_{h}$ is a 6D vector that
contains the 3D position (relative to the agent) and velocity of the ball
(relative to the world frame). In the partially observable setting, the
observation function $\phi_{h}$ is defined such that the agent can only ”see”
the ball once it is within a $4$-meter radius (in all other circumstances $0$
is populated for the elements corresponding to the obstacle in the low-
dimensional state $s_{h}$). We use a standard SAC end-to-end trained agent (no
low-level controller) as a baseline for comparison. Each agent is trained for
$3$ seeds (each lasting $2000$ epochs on both environments). In each epoch,
the agent collects $2000$ samples of interaction from the environment and
performs $200$ gradient update steps.

###  3.2 Results and Discussion

Our results in Figure 3 show that under sparse reward, feudal HRL agents in
both the fully observable and partially observable settings are able to learn
reactive behavior well. We observe that the high-level controller learns to
assign the direction it wants to go in as a subgoal to the low-level
controller, and the low-level controller moves the robot toward the desired
direction based on the subgoal, which allows the agent to avoid the ball. It’s
worth noting that the behavior of the HRL agent is slightly different between
the partially observable setting and fully observable setting. Under the fully
observable setting, the agent tends to react earlier to the ball when it is
approaching compared to the agent under partial observability (Figure 4). This
matches our intuition since under partial observability, the agent can not
”see” the ball until it is within $4$ meters. As a result, the HRL agent under
partial observability converges to a slightly lower average return. End-to-end
agents struggle to learn anything meaningful under the sparse reward setting
and end up with jittering policies for locomotion after extensive training.

Figure 3: All agents are trained on $3$ seeds. The results show that the HRL
agents converged within the first $250$ epochs to a cumulative reward close to
$0$, indicating that the agents are able to dodge most balls. Meanwhile, the
agents trained end-to-end in both the partially observable and fully
observable setting were not able to learn a reactive policy.

HRL

Partial-HRL

Partial-end-to-end

Figure 4: It takes less time for the HRL agent in the fully observable
environment (1st row) to react when the ball is approaching compared to the
HRL agent in the partially observable environment (2nd row), while the end-to-
end agent fails to learn an effective policy in both environments. (3rd row)

###  3.3 Conclusion and Future Work

To study the task of reactive control, we developed a ball-dodging environment
in MuJoCo that involves a subset of the challenges real world robots face such
as learning a locomotion policy and acting under partial observability. We
also presented a two level feudal hierarchical reinforcement learning
framework for the environment and empirically demonstrated significant
improvements over an end-to-end trained RL agent. We plan to extend our HRL
framework to tasks that involve a variety of kinematic systems with more
realistic assumptions of the agent’s perception capabilities in the partial
observable setting (e.g. optical flow). We also hope to increase the
complexity of approaching objects (increasing the number and changing the
dynamics) to create a more challenging domain. Eventually, we hope to realize
our framework on a real quadrupedal robot.

## References

  * Dayan and Hinton [1992] Peter Dayan and Geoffrey E Hinton.  Feudal reinforcement learning.  In S. Hanson, J. Cowan, and C. Giles, editors, _Advances in Neural Information Processing Systems_ , volume 5. Morgan-Kaufmann, 1992.
  * Gu et al. [2017] Shixiang Shane Gu, Ethan Holly, Timothy P. Lillicrap, and Sergey Levine.  Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates.  _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 3389–3396, 2017.
  * Haarnoja et al. [2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018.
  * Levy et al. [2019] Andrew Levy, George Dimitri Konidaris, Robert W. Platt, and Kate Saenko.  Learning multi-level hierarchies with hindsight.  In _International Conference on Learning Representations (ICLR)_ , 2019.
  * Li et al. [2019] Siyuan Li, Rui Wang, Minxue Tang, and Chongjie Zhang.  Hierarchical reinforcement learning with advantage-based auxiliary rewards.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 1407–1417, 2019.
  * Mnih et al. [2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis.  Human-level control through deep reinforcement learning.  _Nat._ , 518(7540):529–533, 2015.  doi: 10.1038/nature14236.
  * Pateria et al. [2021] Shubham Pateria, Budhitama Subagdja, Ah-Hwee Tan, and Chai Quek.  Hierarchical reinforcement learning: A comprehensive survey.  _ACM Comput. Surv._ , 54(5):109:1–109:35, 2021\.
  * Pavlov [2010] P Ivan Pavlov.  Conditioned reflexes: An investigation of the physiological activity of the cerebral cortex.  _Annals of neurosciences_ , 2010.
  * Peng et al. [2016] Xue Bin Peng, Glen Berseth, and Michiel Van de Panne.  Terrain-adaptive locomotion skills using deep reinforcement learning.  _ACM Transactions on Graphics (TOG)_ , 35(4):1–12, 2016.
  * Peng et al. [2017] Xue Bin Peng, Glen Berseth, KangKang Yin, and Michiel Van De Panne.  Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning.  _ACM Transactions on Graphics (TOG)_ , 36(4):1–13, 2017.
  * Peng et al. [2020] Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang-Wei Edward Lee, Jie Tan, and Sergey Levine.  Learning agile robotic locomotion skills by imitating animals.  In _Robotics: Science and Systems_ , 2020.
  * Tassa et al. [2018] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al.  Deepmind control suite.  _arXiv preprint arXiv:1801.00690_ , 2018.