# MDEAW: A Multimodal Dataset for Emotion Analysis through EDA and PPG signals
from wireless wearable low-cost off-the-shelf Devices

Arijit Nandi*  —, Fatos Xhafa*  —, Laia Subirats*  — and Santi Fort*  —
Arijit Nandi is with the Department of CS, Universitat Politècnica de
Catalunya, 08034 Barcelona, Spain and Eurecat, Centre Tecnològic de Catalunya,
08005 Barcelona, Spain (e-mail<EMAIL_ADDRESS>). Fatos Xhafa is
with the Department of CS, Universitat Politècnica de Catalunya, 08034
Barcelona (e-mail: fatos@cs.upc.edu).Laia Subirats is with Eurecat, Centre
Tecnològic de Catalunya, 08005 Barcelona, Spain and ADaS Lab, Universitat
Oberta de Catalunya, 08018 Barcelona, Spain (e-mail:
laia.subirats@eurecat.org). Santi Fort is with Eurecat, Centre Tecnològic de
Catalunya, 08005 Barcelona, Spain (e-mail: santi.fort@eurecat.org).

###### Abstract

We present MDEAW, a multimodal database consisting of Electrodermal Activity
(EDA) and Photoplethysmography (PPG) signals recorded during the exams for the
course taught by the teacher at Eurecat Academy, Sabadell, Barcelona in order
to elicit the emotional reactions to the students in a class room scenario.
Signals from 10 students were recorded along with the students self-assessment
of their affective state after each stimuli, in terms 6 basic emotion states.
All the signals were captured using portable, wearable, wireless, low-cost,
and off-the-shelf equipment that has the potential to allow the use of
affective computing methods in everyday applications. A baseline for student-
wise affect recognition using EDA and PPG-based features, as well as their
fusion, was established through ReMECS, Fed-ReMECS and Fed-ReMECS-U. These
results indicate the prospects of using low-cost devices for affective state
recognition applications. The proposed database will be made publicly
available in order to allow researchers to achieve a more thorough evaluation
of the suitability of these capturing devices for emotion state recognition
applications.

###### Index Terms:

Affective Computing, Dataset for Emotion Recognition, E-learning Dataset for
Emotion Analysis, Federated Learning, Data Streaming, Students’ Emotion State
Analysis

##  I Motivation

The main motivation of conducting this experiment at Eurecat’s Augmented
Workplace is to see how emotions affect students’ learning outcomes. The
hypothesis is that positive emotions promote good learning outcomes, while
negative emotions could be a reason for bad results. From the common sense
perspective, this hypothesis, sounds reasonable. The outcome of this study is
to formally prove the hypothesis. Also, to the best of our knowledge there is
no suitable dataset available in the emotion research domain which is related
to emotional state analysis in the real life E-learning or education context.

##  II Experimental Protocol

###  II-A Experimental environment

The experiment is conducted at a classroom of Eurecat Academy, Sabadell,
Barcelona. The classroom was a regular classroom with fully equipped
machineries and other instruments. The picture (Fig. 1) is the actual view of
the classroom where the experiment was conducted.

Figure 1: The classroom where the experiment was conducted (Eurecat Academy,
Sabadell, Barcelona).

###  II-B Learning exercise

The stimuli used in this experiment are the exams for the course taught by the
teacher at Eurecat Academy, Sabadell, Barcelona in order to elicit the
emotional reactions to the students in a class room scenario and record
Electrodermal Activity (EDA) and Photoplethysmography (PPG) data. The exam was
designed by the tutor of that course and it was divided into smaller exercises
(see Annex A for the examination formulation).

Each exercise duration was 5 minute slots and there were a total of 12 sets of
exercises, so in total the whole exam session was 84 minutes. In between each
of the 12 sets we gave 2 minutes gap to for the students to “cool down”. Also,
students were advised that their annotations would not affect their
evaluations to avoid any possible bias. In each of the 12 sets, from exercise
Set 1 to Set 7; and 11 and 12 had 4 questions in each. Exercise Set 8 to 10
had one subjective question in each. The exercises sets are added in the
additional materials for further reference.

Before starting the exam we instructed the students the whole exam pattern. In
which students were asked to give their current emotional states that they had
felt after answering each questions. For the emotion states input from
students we used 6 basic discrete emotion model
[1]111https://www.paulekman.com/universal-emotions/ developed by Paul Ekman,
widely accepted theory of basic emotions and their expressions. These emotion
labels are as follows:

  1. 1.

Sadness

  2. 2.

Happiness

  3. 3.

Fear

  4. 4.

Anger

  5. 5.

Surprise

  6. 6.

Disgust

###  II-C Students

In the exam there were 10 students. The students were all male. Their age
varies from 23 to 57 years.

###  II-D Data Acquisition

The sensors used to record the EDA and PPG data is Consensys Bundle
Development kit from Shimmer Sensing. The pictures of the sensor is in Fig 2.

Figure 2: Shimmer Sensors

Before starting the exam we put the sensors on the students shown in the
following picture (Fig. 3) for collecting the EDA and PPG data. The sensors
were put on the non active hand of each student (such as if he/she is right
handed we put the sensor on the left hand and so on). The whole experiment was
carried out in 2 sessions, in each session there were 5 students.

Figure 3: Students wearing Shimmer sensors in the classroom while taking the
exams.

The following picture (Fig. 4) is from the class while doing the experiment.

Figure 4: Students taking the exams with the senors wearing.

The data collection from multiple Shimmer sensors were done using the
Shimmer’s Consensys software, in which all the data streams are recorded and
saved. In the picture (Fig. 5), the software interface while doing the
experiment is presented.

Figure 5: Shimmer’s Consensys software interface while collecting the data.

###  II-E Data Formatting

The data recorded in the Shimmer’s Consensys software is exported into two
format, such as .csv and .mat. In each file the data is saved in the following
format (Fig. 6):

Figure 6: Default file format in Shimmer’s Consensys software.

Each .mat or .csv file contains these channels which are available in the
Shimmer sensor. From the filename shown in Fig. 6, we can see that Consensys
software saves the file in the following format:

$[Trial\;name/number,Sessionnumber,SensorUniqueID,Calibrated/notcalibrated,PC]$

The sensor recordings can be accessed by providing the channel names as dot(.)
operator, e.g. in Fig. 7.

Figure 7: Reading data from the .mat file.

The emotion labels were saved in .csv file with the file name which contains
[session name + student ID + type of data (EDA/PPG)].

###  II-F Data Set

For easy accessibility and better understandability we have extracted only the
EDA and PPG data from the data recordings saved in Shimmer’s Consensys
software. The final datatset with the EDA and PPG recordings with the emotion
labels are saved in .csv file. The folder structure of the MDEAW dataset is
shown in Fig. 8.

Figure 8: MDEAW dataset folder structure and files inside it.

Inside each csv file the EDA and PPG recordings are organised as follows:

$[Questionno,SessionNo,StudentID,EDA/PPGdata,Emotionlabels]$

The MDEAW dataset will be made publicly available.

In Fig 9, the emotion class frequencies per student is presented and in Fig.
10 the overall emotion class frequencies for the MDEAW dataset is presented.
This figures shows the class distributions of the MDEAW dataset.

Figure 9: Number of emotion class labels for individual student in MDEAW
dataset. Figure 10: Emotion class distribution in MDEAW dataset

##  III Analysis of the Data Set

###  III-A Feature Extraction

It’s important to extract features from different physiological signals (such
as EDA, PPG etc.) for retrieving relevant information, which effectively
represents emotion states. Based on the extracted features the emotion
classifier is tested and trained.

The most popular and widely used time-frequency analysis of various signals
(especially EDA, PPG, etc.) for feature extraction is Wavelet Decomposition
(WD) [2]. Its popularity and wide use are due to its localized analysis
approach(i,e. time-frequency), multi-rate filtering, and multi-scale zooming.
It is better suited for non-stationary signals (such as EDA, PPG etc.) [3].
Most frequently used wavelet base functions are Meyer WD, Morlet Mother WD,
Haar Mother WD and Daubechies WD [4]. The most frequently used features
extracted from each sub-bands of EDA and PPG are entropy, median, mean,
standard deviation, variance, 5th percentile value, 25th percentile value,
75th percentile value, 95th percentile value, root means square value, zero
crossing rate, mean crossing rate [5, 6, 7].

In our experiment we have extracted and used these features for emotion
classification. The wavelet feature extraction technique is used to extract
features from multi-modal signal streams (EDA, and PPG signals from the
dataset) in this experiment. The wavelet Daubechies 4 (Db4) is the base
function for feature extraction. Our experiment decomposes EDA and RB, into
three levels, respectively.

####  III-A1 Fusion of EDA and PPG-based features

The use of features based on multiple modalities has been shown to provide
increased classification accuracy compared to approaches based on a single
modality. In order to evaluate the performance of the combined EDA and PPG-
based features, the two feature vectors $F_{EDA}$ and $F_{PPG}$ are fused as
follows: First, the values of each feature vector are normalised in the range
[0, 1] in order to compensate for the differences in numerical range. Then,
the two normalised feature vectors $F_{EDA}$ and $F_{PPG}$ are concatenated in
the final feature vector $F_{fused}$ = [ $F_{EDA}$ $F_{PPG}$].

##  IV Emotion Recognition Results for the Data Set

For the emotion recognition results, we have used our previously developed
Real-time Multimodal Emotion Classification System (ReMECS [8]) based on Feed-
Forward Neural Network, trained in an online fashion using the Incremental
Stochastic Gradient Descent algorithm. The avg. accuracy and F1-score along
with the standard deviation (STD) are presented in TABLE I. More results can
be found in Appendix B.

TABLE I: Avg. accuracy and F1-score of real-time emotion classification from MDEAW multimodal dataset | Dataset
---
Name
| Avg.
---
Accuracy
| Avg.
---
F1-score
MDEAW | 0.9583 ($\pm$ 0.1998) | 0.9583 ($\pm$ 0.1998)

Also, the confusion matrix is presented in Fig. 11.

Figure 11: Confusion matrix showing the individual class accuracy for MDEAW
using ReMECS

##  V A New Round of the Experiment

We have planned a new round of the experiment, to extend the dataset. The
reason for conducting another round of experiment to make the MDEAW dataset
more diverse and also to increase number data samples by increasing number
participants. The diversity can help each procedure to guarantee a total good
machine learning: diversity of the training data ensures that the training
data can provide more discriminative information for the model, diversity of
the learned model (diversity in parameters of each model or diversity among
different base models) makes each parameter/model capture unique or complement
information and the diversity in inference can provide multiple choices each
of which corresponds to a specific plausible local optimal result [9]. Having
more number of diverse data samples in the MDEAW dataset will help ML/DL
models to generalize more so that the accuracy and efficiency of those
intelligent emotion models increase.

In the new round of experiment we will follow the same design pattern followed
for the first experiment for the EDA and PPG data collection from the students
but the course and the subject may be different or same based on the
availability. The reason for adding the term ”based on availability” is
because we do the experiment on real students with the real courses studied at
our Eurecat Academy.

Right now, the dataset has 10 students data, which is rather small and if we
train (offline mode training) any ML/DL model, there might be a chance that
the model will be over-fitted easily. So, by increasing the data samples in
the MDEAW dataset we can avoid the issue so that ML/DL models can learn and
generalize properly and those trained models can be used for emotion
classification from new EDA and PPG signals. Also, making the MDEAW dataset
diverse, balanced so that the ML/DL model’s accuracy can be good for emotion
classification.

## Acknowledgement

The authors would like to thank the tutors and students at Eurecat - Centro
Tecnológico de Cataluña for their participation and collaboration in the
experimental study.

This study is carried out as part of the Ph.D work of the student Arijit Nandi
and has been funded by ACCIÓ, Catalunya, Spain (Pla d’Actuació de Centres
Tecnològics 2021) under the project TutorIA for three years, from December
2019 to December 2022.

## Disclaimer

While every care has been taken to ensure the accuracy of the data included in
the MDEAW dataset,the authors and the Eurecat - Centro Tecnológico de
Cataluña, Barcelona, Spain do not provide any guaranties and disclaim all
responsibility and all liability (including without limitation, liability in
negligence) for all expenses, losses, damages (including indirect or
consequential damage) and costs which you might incur as a result of the
provided data being inaccurate or incomplete in any way and for any reason
2022, Eurecat - Centro Tecnológico de Cataluña, Barcelona, Spain.

## Contact

For any questions regarding the MDEAW database please contact:
<EMAIL_ADDRESS>Eurecat - Centro Tecnológico de Cataluña, Barcelona, Spain.

## References

  * [1] P. Ekman, “An argument for basic emotions,” _Cognition and Emotion_ , vol. 6, no. 3-4, pp. 169–200, 1992. [Online]. Available: https://doi.org/10.1080/02699939208411068
  * [2] M. R. Islam and M. Ahmad, “Wavelet analysis based classification of emotion from eeg signal,” in _Int’l Conf. on Electrical, Computer and Comm. Eng._ , 2019, pp. 1–6.
  * [3] J. Zhang, Z. Yin, P. Chen, and S. Nichele, “Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review,” _Information Fusion_ , vol. 59, pp. 103 – 126, 2020.
  * [4] A. Subasi, “Eeg signal classification using wavelet feature extraction and a mixture of expert model,” _Expert Systems with Applications_ , vol. 32, no. 4, pp. 1084 – 1093, 2007.
  * [5] P. Bota, C. Wang, A. Fred, and H. Silva, “Emotion assessment using feature fusion and decision fusion classification based on physiological data: Are we there yet?” _Sensors_ , vol. 20, no. 17, 2020.
  * [6] D. Ayata, Y. Yaslan, and E. Kamasak, Mustafa, “Emotion recognition from multimodal physiological signals for emotion aware healthcare systems,” _J. of Medical and Biological Eng._ , pp. 149–157, 2020.
  * [7] D. Ayata, Y. Yaslan, and M. Kamaşak, “Emotion recognition via random forest and galvanic skin response: Comparison of time based feature sets, window sizes and wavelet approaches,” in _Medical Technologies National Congress_ , 2016, pp. 1–4.
  * [8] A. Nandi, F. Xhafa, L. Subirats, and S. Fort, “Real-time multimodal emotion classification system in e-learning context,” in _Proceedings of the 22nd Engineering Applications of Neural Networks Conference_. Cham: Springer International Publishing, 2021, pp. 423–435.
  * [9] Z. Gong, P. Zhong, and W. Hu, “Diversity in machine learning,” _IEEE Access_ , vol. 7, pp. 64 323–64 350, 2019.
  * [10] A. Nandi and F. Xhafa, “A federated learning method for real-time emotion state classification from multi-modal streaming,” _Methods_ , vol. 204, pp. 340–347, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S104620232200072X
  * [11] A. Nandi, F. Xhafa, L. Subirats, and S. Fort, “Federated learning with exponentially weighted moving average for real-time emotion classification,” in _Proceedings of the 13th 13th International Symposium on Ambient Intelligence Conference_. Cham: Springer International Publishing, 2022, pp. 423–435.

##  Appendix A Learning Examination Exercises

##  Appendix B Further Data Analysis Results

We have applied our previously developed frameworks of Fed-ReMECS [10] and
Fed-ReMECS-U [11] with the MDEAW dataset. Herewith, we show some of the
results.

###  B-A Results from Fed-ReMECS Framework

TABLE II: Global model’s avg. accuracy in case of Fed-ReMECS [10]

No. of clients | Accuracy
---|---
5 | 0.9315 ($\pm$0.16)
10 | 0.6195 ($\pm$0.25)

The global model performance in case of Fed-ReMECS is shown in Fig. 12

Figure 12: Global model’s performance in Fed-ReMECS for different client
scenario (5 and 10 clients) using MDEAW dataset.

###  B-B Results from Fed-ReMECS-U Framework

:

TABLE III: Global model’s avg. accuracy in case of Fed-ReMECS-U [11]

No. of clients | Accuracy
---|---
5 | 0.9487 ($\pm$0.13)
10 | 0.9410 ($\pm$0.15)

The global model performance in case of Fed-ReMECS-U is shown in Fig. 13

Figure 13: Global model’s performance in Fed-ReMECS-U for different client
scenario (5 and 10 clients) using MDEAW dataset.


$\textstyle{A\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{f}$$\scriptstyle{g}$$\textstyle{H\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{a}$$\scriptstyle{c}$$\scriptstyle{x}$$\textstyle{B\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{d}$$\scriptstyle{u}$$\textstyle{G\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{y}$$\textstyle{C\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{v}$$\textstyle{F\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{b}$$\scriptstyle{m}$$\textstyle{D\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{n}$$\textstyle{E,}$

where the four edges of the outer square are exact sequences. (The sequence
$(y,m)$, as well as the other sequences forming edges of the outer square are
assumed to be exact and not necessarily short exact.) Moreover, the arrows $x$
and $v$ are surjective and injective, respectively. Prove the following:

  1. (i)

If $a$ is injective, then $y$ is injective.

  2. (ii)

If $b$ is surjective, then $u$ is surjective.

######  Exercise 6.6 (Baby Dragon Lemma).

Consider a commutative diagram,

$\textstyle{A\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{f}$$\scriptstyle{g}$$\textstyle{B\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{n}$$\scriptstyle{m}$$\textstyle{C\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{z}$$\scriptstyle{\alpha}$$\textstyle{S\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\beta}$$\textstyle{T\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{h}$$\scriptstyle{o}$$\textstyle{U\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{p}$$\scriptstyle{y}$$\textstyle{V\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{x}$$\textstyle{A^{\prime}}$$\textstyle{B^{\prime}}$$\textstyle{C^{\prime}}$

where all the diagonal sequences are exact, and certain arrows are injective
or surjective as indicated. Prove the following statements:

  1. (i)

If $\alpha$ is injective, then so is $\beta$.

  2. (ii)

If $\beta$ is surjective, then so is $\alpha$.

######  Remark 6.7.

Note that this diagram lemma holds for arbitrarily long diagrams of this type
(i.e., we may insert as many copies of the middle diamond as we wish), as in
the case of the Dragon Lemma below.

######  Exercise 6.8 (Dragon Lemma).

Consider a commutative diagram

$\textstyle{N\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{a}$$\scriptstyle{b}$$\textstyle{E\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{f}$$\scriptstyle{e}$$\textstyle{Q\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{c}$$\textstyle{P\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{d}$$\textstyle{H\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{g}$$\textstyle{G\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{h}$$\textstyle{Z\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{x_{2}}$$\scriptstyle{x_{1}}$$\textstyle{A\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{x_{3}}$$\scriptstyle{x_{4}}$$\textstyle{J\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{x_{5}}$$\scriptstyle{x_{6}}$$\textstyle{B\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{x_{n-3}}$$\scriptstyle{x_{n-4}}$$\textstyle{C\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{x_{n-2}}$$\scriptstyle{x_{n-1}}$$\textstyle{D\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{x_{n}}$$\scriptstyle{x_{n+1}}$$\textstyle{R\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{y_{1}}$$\textstyle{S\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{y_{3}}$$\scriptstyle{y_{2}}$$\textstyle{T\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{y_{4}}$$\scriptstyle{y_{5}}$$\textstyle{K\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{y_{6}}$$\textstyle{\cdots}$$\textstyle{T^{\prime}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{y_{n-4}}$$\textstyle{U\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{y_{n-3}}$$\scriptstyle{y_{n-2}}$$\textstyle{V\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{y_{n-1}}$$\scriptstyle{y_{n}}$$\textstyle{W\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{y_{n+1}}$$\textstyle{Z^{\prime}\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{i}$$\scriptstyle{j}$$\textstyle{A^{\prime}}$$\textstyle{J^{\prime}}$$\textstyle{B^{\prime}}$$\textstyle{C^{\prime}}$$\textstyle{D^{\prime}\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{o}$$\scriptstyle{m}$$\textstyle{Q^{\prime}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{k}$$\textstyle{P^{\prime}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{l}$$\textstyle{H^{\prime}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{p}$$\textstyle{G^{\prime}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{q}$$\textstyle{N^{\prime}}$$\textstyle{E^{\prime}}$

where all sequences are exact, and certain arrows are injective or surjective
as indicated. Prove the following:

  1. (i)

If $a$ and $e$ are surjective, then $y_{1}$ is injective.

  2. (ii)

If $l$ and $q$ are injective, then $x_{n+1}$ is surjective.

A version of the following diagram lemma was formulated and proved by J.
Lambek in [27] for the context of groups.

######  Exercise 6.9 (Goursat’s Theorem).

Given a commutative diagram,

$\textstyle{A\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\lambda}$$\scriptstyle{\alpha}$$\textstyle{B\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\mu}$$\scriptstyle{\beta}$$\textstyle{C\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\gamma}$$\textstyle{D\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\lambda^{\prime}}$$\textstyle{E\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\mu^{\prime}}$$\textstyle{F}$

with exact rows, prove that if $\mathsf{Ker}(\gamma\mu)$ is conormal, then:

  1. (i)

$\mathsf{Im}(\beta\lambda)\lhd\mathsf{Im}\beta\wedge\mathsf{Im}\lambda^{\prime}$
and $\mathsf{Ker}\beta\vee\mathsf{Ker}\mu\lhd\mathsf{Ker}(\gamma\mu)$, and

  2. (ii)

$(\mathsf{Im}\beta\wedge\mathsf{Im}\lambda^{\prime})/\mathsf{Im}(\beta\lambda)\approx\mathsf{Ker}(\gamma\mu)/(\mathsf{Ker}\beta\vee\mathsf{Ker}\mu)$.

Hint: use Theorem 4.21.

######  Exercise 6.10 (Generalized Snail Lemma).

Given a commutative diagram

$\textstyle{A\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\alpha}$$\scriptstyle{f}$$\scriptstyle{\gamma}$$\textstyle{B\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\beta}$$\textstyle{C\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{f_{0}^{\prime}}$$\scriptstyle{\beta^{\prime}}$$\textstyle{A_{0}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{f_{0}}$$\textstyle{B_{0},}$

if $\mathsf{Ker}\gamma$, $\mathsf{Ker}\alpha$, $\mathsf{Ker}\beta^{\prime}$
are conormal and $\mathsf{Im}\gamma$, $\mathsf{Im}\alpha$,
$\mathsf{Im}\beta^{\prime}$ are normal, then prove that the following sequence
is exact:

$\mathsf{Ker}\gamma\xrightarrow{v}\mathsf{Ker}\alpha\xrightarrow{w}\mathsf{Ker}\beta^{\prime}\xrightarrow{x}\mathsf{Coker}\gamma\xrightarrow{y}\mathsf{Coker}\alpha\xrightarrow{z}\mathsf{Coker}\beta^{\prime}.$

The Incomplete Snail Lemma (ISL, for short), first stated in the context of a
pointed regular category in [23], is a stepping stone towards the Snail Lemma
due to E. Vitale [39]. As shown in [23], in a pointed regular category, the
(Complete) Snail Lemma holds if and only if ISL holds and every kernel has a
cokernel. The main result obtained in [23] was that in this context, ISL is
equivalent to subtractivity [17]. Combining this with the results of [18, 19],
ISL is further equivalent to the Lower $3\times 3$ (in our context, that would
be Part (ii) of our Lemma 5.6), while in the context of a normal category (a
regular category where every regular epimorphism is a cokernel), it is
equivalent to the Snake Lemma, as well as the Upper $3\times 3$ Lemma (Part
(i) of our Lemma 5.6). With a suitable formulation of the Snake Lemma, it is
furthermore shown in [19] that the Snake Lemma holds in a pointed regular
category if and only if the category is normal and subtractive. For
comparison: in the same context, the Five Lemma is equivalent to the Middle
$3\times 3$ Lemma, and is in turn equivalent to _protomodularity_ in the sense
of D. Bourn [3], which is known to imply both normality and subtractivity. The
Snail Lemma was established in [23] in the very context of a pointed regular
category that is protomodular (such category is also called a _homological
category_ in [2]). In [32], the Snail Lemma was used to generalize two exact
sequences arising from a pointed functor between pointed groupoids and a
fibration of pointed groupoids respectively. This was done by replacing
groupoids by groupoids internal to a suitable category. This was generalized
even further in [15] by considering profunctors between internal groupoids.
Note that the exercise on the Generalized Snail Lemma, which yields a six term
exact sequence similar to that of the Snake Lemma, becomes equivalent to the
Snail Lemma in the presence of finite limits, which we do not require in our
context. It is also worth noting that the ‘form of subobjects’ of a
homological category is not, in general, noetherian. Indeed, this would force
the category to be semi-abelian (see [21]), while not every homological
category is semi-abelian (a well-known example is the category of topological
groups — see [2], for instance). Thus, the exercises in this section do not
directly recover the Snail Lemma and ISL in the original settings where they
have been first established (unlike in the case of the classical diagram
lemmas). Similarly, the exercises on the Spider Lemma and the Dragon Lemma do
not recover these results in the context of a subtractive regular category,
which is where they were first established.

The Spider and the Dragon Lemmas introduced in [19, 22] are in a sense the
shortest and the longest non-trivial diagram lemmas that can be proved in a
pointed regular subtractive category [17, 18], and hence in any semi-abelian
category. In fact, as shown in [19], for a pointed regular category, the
Spider Lemma is equivalent to subtractivity. It is likely that a similar fact
is true for the Dragon Lemma.

Goursat’s Theorem for the context of groups provides a one to one
correspondence between subgroups of the product of two groups and isomorphisms
between subquotients of the two groups. The homological lemma given in
Exercise 6.9 was formulated in [27] and was shown to be equivalent to
Goursat’s Theorem in the context of groups. It was shown in [27, 28] that the
connecting morphism in the Snake Lemma, the Zassenhaus-Schreier Refinement
Theorem and the Jordan-Hölder Theorem, for the context of groups, follow from
different formulations of Goursat’s Theorem.

The Salamander Lemma, which was originally formulated in [1] in the context of
abelian categories, may also be generalized to our context. The Salamander
Lemma was formulated and proved in our context in [11], where the proof was
simplified using two results (see Proposition 3.1 and Proposition 3.5 in [11])
to show the existence of the morphisms in the sequence and a simplified
condition to show exactness of these morphisms. The Salamander Lemma produces
an exact sequence of homology objects from a double complex. A number of
classical diagram lemmas, including the Four Lemma, Five Lemma, $3\times 3$
Lemma and the Snake Lemma, all follow as direct consequences of the Salamander
Lemma (see [1]) for the context of abelian categories. However, in our
context, the derivation of these classical diagram lemmas from the Salamander
Lemma does not work, since it is not guaranteed that the needed homology
objects will exist.

######  Remark 6.11.

Note that there is an error in the proof of Theorem 4.2 in [11], claiming that
the $3\times 3$ Lemma, without any additional assumptions of existence of
homology objects, follows from the Salamander Lemma.

######  Exercise 6.12 (Short Five Lemma).

Consider a commutative diagram,

$\textstyle{A\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{f}$$\scriptstyle{s}$$\textstyle{B\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{g}$$\scriptstyle{t}$$\textstyle{C\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{u}$$\textstyle{A^{\prime}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{x}$$\textstyle{B^{\prime}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{y}$$\textstyle{C^{\prime},}$

where the rows are short exact sequences. Prove the following statements:

  1. (i)

If $s$ and $u$ are injective then so is $t$.

  2. (ii)

If $s$ and $u$ are surjective, then so is $t$.

  3. (iii)

If $s$ and $u$ are isomophisms, then so is $t$.

######  Remarks 6.13.

(1) The special case where $s$ and $u$ are identities expresses that whenever
we have two short exact sequences ‘from $A$ to $C$’, either they are connected
by a map $t$ as in the diagram — in which case they are isomorphic — or no map
$t$ making the diagram commute exists. This is a historically important
ingredient towards the interpretation of cohomology via isomorphism classes of
short exact sequences; see for instance [8] where its use is implicit, or [30]
which views the lemma as a result on its own.

(2) Alternatively, specifying to the situation where $u$ is an identity while
the surjections $g$ and $y$ are split epimorphisms in the context of a pointed
category yields a version of Bourn’s protomodularity [3], one of the key
ingredients in the definition of a semi-abelian category.

Note that short exact sequences (as defined in the previous section) are
precisely the exact sequences of the form

$\textstyle{A\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{f}$$\textstyle{B\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{g}$$\textstyle{C.}$

However, in all known contexts of interest, a short exact sequence is
equivalent to an exact sequence consisting of five groups where the two outer
groups have exactly one subgroup (this is taken as the definition of a short
exact sequence for modules in [30]); in other words, an exact sequence

$\textstyle{0\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{A\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{f}$$\textstyle{B\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{g}$$\textstyle{C\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{0^{\prime},}$

where $0$, $0^{\prime}$ are groups each having exactly one subgroup — we will
refer to these as _strongly short exact sequences_. We will call a group
_trivial_ when it has exactly one subgroup.

######  Exercise 6.14.

Show that in a strongly short exact sequence, the groups $0$ and $0^{\prime}$
are necessarily isomorphic (more generally, show that any two trivial groups
connected by a zigzag of morphisms are isomorphic). Moreover, show that in
every strongly short exact sequence, the pair $f,g$ is a short exact sequence.

The equivalence of the two approaches of defining short exact sequences
follows from the following additional axiom.

######  Axiom 6.

For every group, its smallest subgroup is conormal while its largest subgroup
is normal.

######  Exercise 6.15.

Show that Axiom 6 is equivalent to requiring that a morphism $f\colon
A\rightarrow B$ is injective if and only if it forms an exact sequence

$\textstyle{0\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{A\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{f}$$\textstyle{B}$

for some trivial $0$, and, along with this, requiring that dually, $f$ is
surjective if and only if it forms an exact sequence

$\textstyle{A\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{f}$$\textstyle{B\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{0}$

for some trivial $0$. Deduce from this that under Axiom 6, every short exact
sequence completes to a strongly short exact sequence.

######  Exercise 6.16.

Formulate and prove the diagram lemmas by replacing short exact sequences with
strongly short exact sequences, and by replacing injective/surjective
morphisms with those that complete to sequences as shown in the previous
exercise (but do this, without assuming Axiom 6). Show that the version of the
Short Five Lemma where the horizontal sequences have been replaced with
strongly short exact sequences is a consequence of the Five Lemma. Establish a
similar link between the Baby Dragon Lemma and the Dragon Lemma.

######  Remark 6.17.

A context where all axioms hold except Axiom 6 is that of rings with identity
and identity-preserving ring homomorphisms, where subgroups are additive
subgroups of rings (see [12]). Conormal subgroups there are subrings
containing the identity element, while normal subgroups are ideals. The
largest additive subgroup of a ring is, of course, an ideal, but the dual
property fails: the smallest additive subgroup does not contain the identity
element. While this example shows that Axiom 6 is independent of the rest of
the axioms, it is not a particularly interesting example from the point of
view of homological diagram lemmas, since only the largest subgroups are both
normal and conormal (an ideal contains the identity element only when it is
the entire ring), and hence there is a shortage of exact sequences. We do not
know of a naturally arising example where Axiom 6 fails, but at the same time,
where there isn’t a shortage of short exact sequences.

## Acknowledgements

The first author wishes to thank Stellenbosch University for its kind
hospitality during his visits in October and November 2022. The fourth author
wishes to thank the African Institute for Mathematical Sciences (AIMS) and
Stellenbosch University for their kind hospitality and the opportunity to
participate in the _1st Workshop on Mathematical Structures_ for which the
notes [38] were prepared.

## References

  * [1] G. M. Bergman, _On diagram-chasing in double complexes_ , Theory and Applications of Categories 26(3), 2012, 60–96.
  * [2] F. Borceux and D. Bourn, _Mal’cev, protomodular, homological and semi-abelian categories_ , Mathematics and its Applications 566, Kluwer, 2004.
  * [3] D. Bourn, _Normalization Equivalence, Kernel Equivalence and Affine Categories_ , Lecture Notes in Mathematics 1488, Springer-Verlag, 1991, 43–62.
  * [4] D. Bourn, _$3\times 3$ Lemma and protomodularity_, Journal of Algebra 236, 2001, 778–795.
  * [5] D. Bourn and G. Janelidze, _Characterization of protomodular varieties of universal algebras_ , Theory and Applications of Categories 11, 2003, 143–147.
  * [6] D. A. Buchsbaum, _Exact categories and duality_ , Transactions of the American Mathematical Society 80(1), 1955, 1–34.
  * [7] S. Burris and H. P. Sankappanavar, _A course in universal algebra_ , Springer-Verlag, 1981.
  * [8] H. Cartan and S. Eilenberg, _Homological algebra_ , Princeton University Press, 1956.
  * [9] T. C. Chimpinde, _A categorical approach to Jordan-Hölder theorem_ , Structured Masters Research Project, African Institute of Mathematical Sciences, 2014.
  * [10] K. Dayaram, _Diagram lemmas of homological algebra for non-abelian group-like structures_ , MSc Thesis, University of Johannesburg, 2021.
  * [11] A. Goswami, _Salamander lemma for non-abelian group-like structures_ , Journal of Algebra and its Applications 19(2), 2020, 2050022 (12 pages).
  * [12] A. Goswami and Z. Janelidze, _Duality in non-abelian algebra IV. Duality for groups and a universal isomorphism theorem_ , Advances in Mathematics 349, 2019, 781–812.
  * [13] M. Gran, G. Kadjo, and J. Vercruysse, _A torsion theory in the category of cocommutative Hopf algebras_ , Applied Categorical Structures 24, 2016, 269–282.
  * [14] M. Grandis, _Homological algebra in strongly non-abelian settings_ , World Scientific, 2013.
  * [15] P. Jacqmin, S. Mantovani, G. Metere, and E. M. Vitale, _Bipullbacks of fractions and the snail lemma_ , Journal of Pure and Applied Algebra 223(12), 2019, 5147–5162.
  * [16] G. Janelidze, L. Márki and W. Tholen, _Semi-abelian categories_ , Journal of Pure and Applied Algebra 168, 2002, 367–386.
  * [17] Z. Janelidze, _Subtractive categories_ , Applied Categorical Structures 13, 2005, 343–350.
  * [18] Z. Janelidze, _The pointed subobject functor, 3 × 3 lemmas, and subtractivity of spans_ , Theory and Applications of Categories 23, 2010, 221–-242.
  * [19] Z. Janelidze, _An axiomatic survey of diagram lemmas for non-abelian group-like structures_ , Journal of Algebra 370, 2012, 387–401.
  * [20] Z. Janelidze, _Subobject Chasing Revisited_ , talk given at Category Theory 2014, Cambridge.
  * [21] Z. Janelidze, _On the form of subgroups in semi-abelian and regular protomodular categories_ , Applied Categorical Structures 22, 2014, 755–766.
  * [22] Z. Janelidze, _Duality for diagram chasing a la Mac Lane in non-abelian categories_ , Homology, Homotopy and Applications 18, 2016, 85–106.
  * [23] Z. Janelidze and E. Vitale, _Snail lemma in a pointed regular category_ , Journal of Pure and Applied Algebra 221, 2017, 135–143.
  * [24] Z. Janelidze and T. Weighill, _Duality in non-abelian algebra I. From cover relations to Grandis ex-2 categories_ , Theory and Applications of Categories 29(11), 2014, 315–331.
  * [25] Z. Janelidze and T. Weighill, _Duality in non-abelian algebra II. From Isbell bicategories to Grandis exact categories_ , Journal of Homotopy and Related Structures 11, 2016, 553–570.
  * [26] Z. Janelidze and T. Weighill, _Duality in non-abelian algebra III. Normal categories and 0-regular varieties_ , Algebra Universalis 77, 2017, 1–28.
  * [27] J. Lambek, _Goursat’s theorem and homological algebra_ , Canadian Mathematical Bulletin 7, 1964, 597–608.
  * [28] J. Lambek, _Goursat’s theorem and the Zassenhaus lemma_ , Canadian Journal of Mathematics 10, 1958, 45–56.
  * [29] S. Mac Lane, _Duality for groups_ , Bulletin of the American Mathematical Society 56, 1950, 485–516.
  * [30] S. Mac Lane, _Homology_ , Grundlehren der Mathematischen Wissenschaften, vol. 114, Springer, 1963.
  * [31] S. Mac Lane, _Categories for the working mathematician_ , second ed., Graduate Texts in Mathematics, vol. 5, Springer, 1998.
  * [32] S. Mantovani, G. Metere, and E. M. Vitale, _The snail lemma for internal groupoids_ , Journal of Algebra 535, 2019, 1–34.
  * [33] N. Monde, _Chasing subgroups_ , Structured Masters Research Project, African Institute of Mathematical Sciences, 2014.
  * [34] B. G. Rodrigues, _Algebraic Structures Lecture Notes_ , University of KwaZulu-Natal, 2018.
  * [35] J. Słomiński, _On the determining form of congruences in abstract algebras with equationally definable constant elements_ , Fundamenta Mathematicae 48, 1960, 325–341
  * [36] A. Ursini, _Osservazioni sulle varietá BIT_ , Bolletino della Unione Matematica Italiana 7, 1983, 205–-211.
  * [37] A. Ursini, _On subtractive varieties I_ , Algebra Universalis 31, 1994, 204–-222.
  * [38] T. Van der Linden, _Ordered sets in homological algebra_ , 1st Workshop on Mathematical Structures, African Institute of Mathematical Sciences (AIMS), 2013.
  * [39] E. M. Vitale, _The snail lemma_ , Theory and Applications of Categories 31(19), 2016, 484–501.
  * [40] F. K. van Niekerk, _Contributions to projective group theory_ , M.Sc Thesis, Stellenbosch University, 2017.
  * [41] F. K. van Niekerk, _Concrete foundations of the theory of Noetherian forms_ , PhD thesis, Stellenbosch University, 2019.
  * [42] F. K. van Niekerk, _Biproducts and commutators for noetherian forms_ , Theory and Applications of Categories 34(30), 2019, 961–992.
  * [43] T. Weighill, _Bifibrational duality in non-abelian algebra and the theory of databases_ , MSc Thesis, Stellenbosch University, 2014.
  * [44] O. Wyler, _Weakly exact categories_ , Archiv der Mathematik XVII, 1966, 9–19.
  * [45] O. Wyler, _The Zassenhaus lemma for categories_ , Archiv der Mathematik XVII, 1971, 561–569.


# Vibration Analysis of Timoshenko Beams using Isogeometric Analysis

Balakrishnan Devarajan111Department of Biomedical Engineering and Mechanics,
<EMAIL_ADDRESS>Virginia Polytechnic Institute and State University,
Blacksburg, VA 24060, USA

###### Abstract

In this paper, the finite free-form beam element is formulated by the
isogeometric approach based on the Timoshenko beam theory to investigate the
free vibration behavior of the beams. The non-uniform rational B-splines
(NURBS) functions which define the geometry of the beam are used as the basis
functions for the finite element analysis. In order to enrich the basis
functions and to increase the accuracy of the solution fields, the h-, p-, and
k-refinement techniques are implemented. The geometry and curvature of the
beams are modelled in a unique way based on NURBS. All the effects of the the
shear deformation, and the rotary inertia are taken into consideration by the
present isogeometric model. Results of the beams for non-dimensional
frequencies are compared with other available results in order to show the
accuracy and efficiency of the present isogeometric approach. From numerical
results, the present element can produce very accurate values of natural
frequencies and the mode shapes due to exact definition of the geometry. With
higher order basis functions, there is no shear locking phenomenon in very
thin beam situations. Finally, the benchmark tests described in this study are
provided as future reference solutions for Timoshenko beam vibration problem.

##  1 Nomenclature

$x,y,z$ = | coordinate system defined for the beam
---|---
$u,v,w$ = | displacements in the $x$, $y$ and $z$ directions respectively
$u_{0},v_{0}$ = | displacements in the $x$ and $y$ directions of the mid plane respectively
$\varphi$ = | rotation about the $z$ direction
$\Pi$ = | knot vector
$B$ = | strain displacement matrix
$J$ = | determinant of the jacobian
$W_{k}$ = | corresponding weight of the gauss points
$\epsilon_{i}$ = |  $i^{\text{th}}$ knot
$W(\epsilon)$ = | weighting function
$q_{i}$ = | weight of $i^{\text{th}}$ control point
$(P_{i})_{j}$ = | control point
$\text{N}^{p}_{i}$ = |  $i^{\text{th}}$ basis function of order $p$
$p$ = | order of the polynomial
$n$ = | number of control points
$E$ = | Young’s modulus

##  2 Introduction

The first meaningful observation on the beam vibration appears to be the one
conducted by Bernoulli. From his observation, [1] he discovered that the
curvature of an elastic beam is proportional to the bending moment. Later,
Leonhard Euler accepted Bernoulli’s assumption and made many advances on the
elastic curves. In engineering practice, the Euler–Bernoulli’s beam theory has
been then most frequently used to predict the natural frequencies of the
beams. However, as well-known fact, it overestimates the natural frequencies
of the beams especially for higher modes and is eligible for slender beams.
Rayleigh [1] introduced rotatory inertia effect and corrected the
overestimation of natural frequency for some problems. But in particular
problems, the introduction of shear was still required in order to improve the
accuracy of natural frequencies. In this context, Timoshenko [2] proposed a
beam theory which includes the transverse shear deformation and the rotatory
inertia effect. Since then, Timoshenko’s beam theory has been widely used in
practical engineering problems.

Khdeir and Reddy [2] derived analytical buckling solutions of refined shear
deformation beam theories and compared them with classical [3, 4] Research on
the vibration of Timoshenko beams and Mindlin plates can be divided into three
categories. Firstly, there exist exact solutions only for a very restricted
number of simple cases. Secondly, studies of semi-analytic solutions,
including the differential quadrature method [1], [2], [3] and the boundary
characteristic orthogonal polynomials [4], [5], are available. Finally, there
are the most widely used discretization methods such as the finite element
method and the finite difference method. As it is more useful to have
analytical results than to resort to numerical methods, most efforts focus on
developing efficient semi-analytic solutions.

Isogeometric analysis (IGA) developed by Hughes et al. [5] can be used to
implement multiple boundary conditions[6, 7, 8, 9, 10] as opposed to other
approximate energy method formulations. Hence, beam problems with multiple
support can be tackled using this approach. Moreover, the numerical solution
obtained by IGA is expected to give a better convergence due to its k
refinement capability as shown by Nguyen [11]. The solution does not oscillate
with an increase in the polynomial order as opposed to the polynomials used in
the conventional finite element polynomial shape functions.

As shown in the , a Timoshenko beam is considered.

Shear effects were considered in the beam formulation, as depicted by Kapoor
and Kapania [12] and Kapania and Raciti [13] using first order shear
deformation theory (Timoshenko Beam Theory [14]) where the displacements are
given by Eq. (1) and (2): -

$u(x,y)=u_{0}(x)+y\varphi(x)$ (1) $v(x,y)=v_{0}(x)$ (2)

The strain-displacement relationships are given as:-

$\varepsilon_{x}=\frac{du_{0}(x)}{dx}+y\frac{d\varphi(x)}{dx}$ (3)
$\gamma_{xy}=\varphi(x)+\frac{dv_{0}(x)}{dx}$ (4)

In IGA, mid-plane displacements and rotations can be written as:-

$u_{0}=\sum_{i=1}^{n}{N(\epsilon_{i})u_{i}};\hskip
28.45274ptv_{0}=\sum_{i=1}^{n}{N(\epsilon_{i})v_{i}};\hskip
28.45274pt\varphi=\sum_{i=1}^{n}{N(\epsilon_{i})\varphi_{i}};\hskip
28.45274ptn=\text{number\hskip 2.84544ptof\hskip 2.84544ptcontrol\hskip
2.84544ptpoints}$ (5)

Using NURBS basis functions, the in-plane extension, transverse deflection,
and the rotations at control points can be expressed as:-

$\textbf{u}_{0}=\sum_{k=1}^{n}\begin{bmatrix}N&0&0\\\ 0&N&0\\\ 0&0&N\\\
\end{bmatrix}\\\ \begin{bmatrix}u\\\ v\\\ \varphi\\\ \end{bmatrix}$ (6)

Where $\begin{bmatrix}u&v&\varphi\end{bmatrix}^{T}$ are the degrees of freedom
of $\textbf{u}_{0}$ associated with a control point $k$.
The strain displacement matrices for bending Bf, membrane Bm and transverse
shear Bc are:-

$B_{m}=\begin{bmatrix}\frac{\partial N}{\partial x}&0&0\\\ 0&0&0\\\
\end{bmatrix}$ (7)

$B_{f}=\begin{bmatrix}0&0&\frac{\partial N}{\partial x}\\\ 0&0&0\\\
\end{bmatrix}$ (8)

$B_{c}=\begin{bmatrix}0&0&0\\\ 0&\frac{\partial N}{\partial x}&N\\\
\end{bmatrix}$ (9)

The expression of stiffness matrix is given as:-

$\begin{gathered}K=b{(\sum_{i=1}^{n}{B_{m}^{T}DB_{m}}zJW+\frac{1}{2}\sum_{i=1}^{n}{B_{m}^{T}D^{B}_{f}}z^{2}JW_{k}}\\\
+{\frac{1}{2}\sum_{i=1}^{n}{B_{f}^{T}D^{B}_{m}}(z^{2}-z^{2})JW}+{\frac{1}{3}\sum_{i=1}^{n}{B_{f}^{T}DB_{f}}(z^{3}-z^{3})JW+\sum_{i=1}^{n}{B_{c}^{T}D_{c}B_{c}}(z-z)JW)}\\\
\end{gathered}$ (10)

Where b is the breadth of the beam, J is the Jacobian and

$D=\begin{bmatrix}\overline{Q}_{11}&\overline{Q}_{12}&\overline{Q}_{16}\\\
\overline{Q}_{12}&\overline{Q}_{22}&\overline{Q}_{26}\\\
\overline{Q}_{16}&\overline{Q}_{26}&\overline{Q}_{66}\\\ \end{bmatrix},\hskip
14.22636pt{D_{c}}=\begin{bmatrix}\overline{Q}_{44}&0\\\ 0&\overline{Q}_{55}\\\
\end{bmatrix}\hskip 14.22636pt$ (11)

The entries of ${K_{e}}$ (element stiffness matrix) are computed using Gauss-
Legendre quadrature numerical integration technique and the element stiffness
matrices are assembled into a global stiffness matrix $K$.

We can compute the mass matrix

$\begin{gathered}K=\rho\begin{bmatrix}N&0&0\\\ 0&N&0\\\ 0&0&N\\\
\end{bmatrix}^{T}\begin{bmatrix}h&0&0\\\ 0&h&0\\\ 0&0&h^{3}/12\\\
\end{bmatrix}\begin{bmatrix}N_{k}&0&0\\\ 0&N_{k}&0\\\ 0&0&N_{k}\\\
\end{bmatrix}\end{gathered}$ (12)

The non-dimensional natural frequencies are given by $(\widetilde{\omega})$,
defined in Eq. (14) is then evaluated by solving eigenvalue problem as.

$\\{[K]-\omega^{2}[M]\\}[\delta]=0$ (13)

$\widetilde{\omega}=\omega L^{2}\sqrt{\frac{\rho A}{EI}}$ (14)

##  3 Solution Approach

###  3.1 Isogeometric Analysis

Isogeometric approach (IGA) uses Non-Uniform Rational Basis Spline (NURBS)
representation as shown in Fig. 1 to define both geometry and the displacement
shape function.

Figure 1: A Curve Represented Using NURBS.

The NURBS basis functions can be defined as

$R_{i}^{p}(\epsilon)=\frac{N_{i,p}(\epsilon)q_{i}}{W(\epsilon)}=\frac{N_{i,p}(\epsilon)q_{i}}{\sum_{i=1}^{n}{N_{i,p}(\epsilon)}q_{i}(\epsilon)}$
(15)

$N_{i,0}(\epsilon)=\bigg{\\{}\begin{gathered}1,\epsilon_{i}\leq\epsilon\leq\epsilon_{i+1}\\\
0,otherwise\\\ \end{gathered}$ (16)

$N_{i,p}(\epsilon)=\frac{\epsilon-\epsilon_{i}}{\epsilon_{i+p}-\epsilon_{i}}N_{i,p-1}(\epsilon)+\frac{\epsilon_{i+p+1}-\epsilon}{\epsilon_{i+p+1}-\epsilon_{i+1}}N_{i+1,p-1}(\epsilon)$
(17)

Computer Aided Design (CAD) systems are based on spline basis functions, often
Non-Uniform Rational B-Splines (NURBS). In traditional finite element method,
this geometry is approximated by a piecewise lower order Lagrange/Hermite
interpolation polynomials. It has to be noted that the analysis in the current
work is restricted to straight beams and hence B-splines were used i.e. the
weights associated with all the control points are set to 1. In FEM, elements
are used to divide discretize geometry; whereas, in IGA patches are used. In
finite element analysis (FEA), the parameter space is the reference element
which is mapped into each element in the physical space. However in IGA,
parameter space is the whole patch which is further divided into elements by a
knot vector in each direction as given in Eq. (18)

$\Pi={\epsilon_{1},\epsilon_{2},...,\epsilon_{i},...,\epsilon_{n+p+1}}$ (18)

where $\epsilon_{1},\epsilon_{2},...,\epsilon_{i},...,\epsilon_{n+p+1}$ are
the knot vectors.

The length of the knot vector is given by

$\Pi=n+p+1.$ (19)

###  3.2 Refinement

The refinement in IGA analysis is of three types. The first type is the order
elevation. Analogous to p refinement in FEA, the order elevation involves
increasing the multiplicity of each knot value by one. The geometry and the
parameterization of the physical curve are not changed. However, the number of
basis functions and control points increase. Next is knot insertion. It has
similarities to h refinement in FEA, which involves splitting the elements. In
IGA, knot insertion involves adding new knot values between the existing
knots. This creates more elements, since the elements are bounded by knots of
different values. The last and most widely used refinement is the k
refinement. This has no analog in FEA and is one of the factors that make IGA
analysis very powerful. The k refinement is a combination of order elevation
and knot insertion. This is done by increasing just the multiplicity of the
first and the last knot values. The continuity across knot values is then
increased by the same number as the number of multiplicities in order
elevation (i.e. a $C^{\text{p-1}}$ continuity). The k refinement is widely
used and possesses improved computational efficiency due to lesser number of
basis functions. Mapping from reference to parametric and parametric to
physical space can be found in work by Nguyen et al. [15].

##  4 Results and Discussion

Table 2: Non-dimensionalized frequency parameter of the Timoshenko beam (pinned–pinned boundary condition, nu=0.3, alpha=5/6 Mode | CLT | h/L
---|---|---
|  | 0.002 | 0.005 | 0.01 | 0.02 | 0.05 | 0.1 | 0.2
1 | 3.14159 | 3.1417 | 3.1415 | 3.1413 | 3.1405 | 3.135 | 3.1157 | 3.0453
2 | 6.28319 | 6.2839 | 6.2828 | 6.2811 | 6.2747 | 6.2314 | 6.0907 | 5.6716
3 | 9.42478 | 9.4271 | 9.4234 | 9.4177 | 9.3964 | 9.2554 | 8.8405 | 7.8395
4 | 12.5664 | 12.5718 | 12.5632 | 12.5497 | 12.4995 | 12.1814 | 11.3431 | 9.6571
5 | 15.708 | 15.7187 | 15.7017 | 15.6755 | 15.5787 | 14.9928 | 13.6132 | 11.2221
6 | 18.8496 | 18.8682 | 18.8388 | 18.7937 | 18.6286 | 17.6812 | 15.6792 | 12.6023
7 | 21.9911 | 22.021 | 21.9742 | 21.9028 | 21.645 | 20.245 | 17.5707 | 13.0323
8 | 25.1327 | 25.1776 | 25.1075 | 25.0014 | 24.6237 | 22.6866 | 19.3144 | 13.4443
9 | 28.2743 | 28.3388 | 28.2386 | 28.0882 | 27.5614 | 25.0117 | 20.9328 | 13.8434
10 | 31.4159 | 31.5052 | 31.3672 | 31.162 | 30.4553 | 27.2271 | 22.4445 | 14.4378
Table 3: Non-dimensionalized frequency parameter of the Timoshenko beam (clamped–clamped boundary condition, nu=0.3, alpha=5/6 Mode | CLT | h/L
---|---|---
|  | 0.002 | 0.005 | 0.01 | 0.02 | 0.05 | 0.1 | 0.2
1 | 4.73004 | 4.72998 | 4.72963 | 4.72840 | 4.72350 | 4.68991 | 4.57955 | 4.24201
2 | 7.8532 | 7.9272 | 7.8877 | 7.8606 | 7.8321 | 7.7042 | 7.3314 | 6.418
3 | 10.9956 | 11.1019 | 11.0423 | 10.9991 | 10.9396 | 10.641 | 9.8563 | 8.2853
4 | 14.1372 | 14.2781 | 14.1946 | 14.1304 | 14.0223 | 13.4622 | 12.1456 | 9.9038
5 | 17.2788 | 17.4574 | 17.3452 | 17.2541 | 17.0761 | 16.1602 | 14.2327 | 11.3488
6 | 20.4204 | 20.6401 | 20.4939 | 20.3685 | 20.0964 | 18.7332 | 16.149 | 12.6403
7 | 23.5619 | 23.827 | 23.6404 | 23.4724 | 23.0791 | 21.184 | 17.9218 | 13.4567
8 | 26.7035 | 27.0187 | 26.7843 | 26.5644 | 26.0209 | 23.5185 | 19.5727 | 13.8102
9 | 29.8451 | 30.216 | 29.9254 | 29.6431 | 28.9188 | 25.7439 | 21.1189 | 14.4806
10 | 32.9867 | 33.4195 | 33.0636 | 32.7075 | 31.7707 | 27.8682 | 22.5739 | 14.9384

##  5 Conclusions and Future Work

A isogeometric analysis using NURBS basis functions is applied to the free
vibration analysis of Timoshenko beams. Because the formulation is so simple
and efficient, allowing the process of calculating weighting coefficients and
characteristic polynomials to be avoided, this method has merits over other
semi-analytic methods. Rapid convergence, good accuracy as well as the
conceptual simplicity characterize the isogeometric analysis method. The
results from this method agree with those of Bernoulli–Euler beams when the
thickness-to-length (radius) ratio is very small, however, deviate
considerably as the thickness-to-length (radius) ratio grows larger.

## References

  * Li et al. [2016] Li, Q., Devarajan, B., Zhang, X., Burgos, R., Boroyevich, D., and Raj, P., “Conceptual Design and Weight Optimization of Aircraft Power Systems with High-Peak Pulsed Power Loads,” _SAE Technical Paper Series_ , SAE International, 2016, p. 10.
  * Khdeir and Reddy [1997] Khdeir, A., and Reddy, J., “Buckling of Cross-ply Laminated Beams with Arbitrary Boundary Conditions,” _Composite Structures_ , Vol. 37, No. 1, 1997, pp. 1–3.
  * De et al. [2017] De, S., Jrad, M., Locatelli, D., Kapania, R. K., and Baker, M., “SpaRibs Geometry Parameterization for Wings with Multiple Sections using Single Design Space,” _58th AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference_ , 2017, p. 0570.
  * De et al. [2019] De, S., Jrad, M., and Kapania, R. K., “Structural Optimization of Internal Structure of Aircraft Wings with Curvilinear Spars and Ribs,” _Journal of Aircraft_ , Vol. 56, No. 2, 2019, pp. 707–718.
  * Hughes et al. [2005] Hughes, T. J., Cottrell, J. A., and Bazilevs, Y., “Isogeometric Analysis: CAD, Finite Elements, NURBS, Exact Geometry and Mesh Refinement,” _Computer Methods in Applied Mechanics and Engineering_ , Vol. 194, No. 39, 2005, pp. 4135–4195.
  * Devarajan and Kapania [2020] Devarajan, B., and Kapania, R. K., “Thermal buckling of curvilinearly stiffened laminated composite plates with cutouts using isogeometric analysis,” _Composite Structures_ , Vol. 238, 2020, p. 111881.
  * Devarajan et al. [2016] Devarajan, B., Locatelli, D., Kapania, R. K., and Meritt, R. J., “Thermo-Mechanical Analysis and Design of Threaded Fasteners,” _57th AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference_ , American Institute of Aeronautics and Astronautics, 2016, p. 0579.
  * Devarajan [2021] Devarajan, B., “Analyzing Thermal Buckling in Curvilinearly Stiffened Composite Plates with Arbitrary Shaped Cutouts Using Isogeometric Level Set Method,” , 2021.
  * Miglani et al. [2018] Miglani, J., Devarajan, B., and Kapania, R. K., “Thermal buckling analysis of periodically supported composite beams using Isogeometric analysis,” _2018 AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference_ , American Institute of Aeronautics and Astronautics, 2018, p. 1224.
  * Devarajan [2019] Devarajan, B., “Thermomechanical and Vibration Analysis of Stiffened Unitized Structures and Threaded Fasteners,” Ph.D. thesis, Virginia Tech, 2019\.
  * Nguyen [2011] Nguyen, T. N., “Isogeometric Finite Element Analysis based on Bezier Extraction of NURBS and T-Splines,” Master’s thesis, Norges teknisk-naturvitenskapelige universitet (NTNU), Fakultet for ingeniørvitenskap og teknologi, Institutt for konstruksjonsteknikk,Norway, 2011\.
  * Kapoor and Kapania [2012] Kapoor, H., and Kapania, R., “Geometrically Nonlinear NURBS Isogeometric Finite Element Analysis of Laminated Composite Plates,” _Composite Structures_ , Vol. 94, No. 12, 2012, pp. 3434–3447.
  * Kapania and Raciti [1989] Kapania, R. K., and Raciti, S., “Recent advances in analysis of laminated beams and plates, Part I: Shear effects and buckling,” _AIAA J_ , Vol. 27, No. 7, 1989, pp. 923–935.
  * Timoshenko [1922] Timoshenko, S. P., “On the Transverse Vibrations of Bars of Uniform Cross-section,” _The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science_ , Vol. 43, No. 253, 1922, pp. 125–131.
  * Nguyen et al. [2015] Nguyen, V. P., Anitescu, C., Bordas, S. P., and Rabczuk, T., “Isogeometric Analysis: An Overview and Computer Implementation Aspects,” _Mathematics and Computers in Simulation_ , Vol. 117, 2015, pp. 89–116.


# Context-Matched Collage Generation for Underwater Invertebrate Detection

R. Austin McEver, Bowen Zhang, B.S. Manjunath Austin McEver, Bowen Zhang, and
B.S. Manjunath are with University of California, Santa Barbara, 93106,
California, United States of America. Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The quality and size of training sets often limit the performance of many
state of the art object detectors. However, in many scenarios, it can be
difficult to collect images for training, not to mention the costs associated
with collecting annotations suitable for training these object detectors. For
these reasons, on challenging video datasets such as the Dataset for
Underwater Substrate and Invertebrate Analysis (DUSIA), budgets may only allow
for collecting and providing partial annotations [1]. To aid in the challenges
associated with training with limited and partial annotations, we introduce
Context Matched Collages, which leverage explicit context labels to combine
unused background examples with existing annotated data to synthesize
additional training samples that ultimately improve object detection
performance. By combining a set of our generated collage images with the
original training set, we see improved performance using three different
object detectors on DUSIA, ultimately achieving state of the art object
detection performance on the dataset.

###### Index Terms:

Object detection, data generation, data synthesis

##  I Introduction

Today’s computer vision methods largely depend on enormous datasets with many
annotated examples for each class. These sorts of datasets can be extremely
expensive to collect, especially when the data is more scientific in nature.
While any layperson can label a cat, dog, or human, the cost of labelling and
differentiating between more specific, scientific classes grow exponentially.
The cost of collecting, annotating, and analyzing scientific data is high, but
that also means that any automation or streamlining of those processes can
greatly benefit domain scientists who currently rely almost entirely on
expensive human experts.

Figure 1: Examples of the ten species that are labelled with bounding boxes in
DUSIA. YG stands for yellow gorgonian; BS, basket star; GG, gray gorgonian;
LS, laced sponge; WSpSC, white spine sea cucumber; LLS, long-legged sunflower
star; SL, squat lobster; FPU, fragile pink urchin; WSSC, white sea slipper
cucumber; RSG, red swifita gorgonian.

The Dataset for Underwater Substrate and Invertebrate Analysis (DUSIA) [1]
provides an example of a challenging, scientific dataset. DUSIA contains 10
hours of video collected in 1080p using a remotely operated vehicle (ROV) that
drives over and records the ocean floor at depths between 100 m and 400 m, and
the data within DUSIA is part of a much greater, growing collection of
hundreds of hours of unlabelled or weakly labelled videos. Marine scientists
collect these videos as part of surveys that improve their understanding of
habitats and organisms of the ocean floor. DUSIA’s videos come directly from
marine scientists working to study, understand, and survey the ocean floor.

Despite the rich content of the videos, DUSIA’s annotations are limited due to
the expense of hiring trained marine science experts to annotate video with
the level of granularity of typical computer vision datasets. The dataset
provides numerous, weak labels, which indicate timestamps at which 57
invertebrate species of interest are Counted At the Bottom Of the video Frame
(CABOF), as well as a training set with frames partially annotated with
bounding boxes for the the ten species shown in Figure 1.

CABOF labels are described in detail in the original work [1] and illustrated
via a frame by frame representation of DUSIA’s videos in Figure 2. In summary,
as the ROV traverses the ocean floor, species come into view at the top of the
frame and make their way to the bottom of the frame as the ROV and video moves
forward. Cropped example frames are shown in Figure 2 with frames going
forward in time from bottom to top. When a species individual first touches
the bottom of the frame (like the yellow gorgonian in Frame F of Figure 2),
annotators create a CABOF label with that species name and the timestamp,
which corresponds to collected GPS coordinates. This labelling gives marine
science researchers a metric for counting the number of species individuals
occurring along a narrow transect path. In Section III, we present a new use
for these CABOF labels and leverage them to try to find frames in DUSIA’s
videos where there are _no_ species.

DUSIA presents partial bounding box labels for training because collecting
full labels is preventatively expensive. These labels are partial in that
every instance of a species of interest in the training set’s frames may not
be annotated. That is, there may be some unlabelled individuals of species of
interest in the training frames.

DUSIA’s partial annotations provide an interesting challenge for today’s
computer vision methods and require new methods to solve the object detection
problems presented by the dataset. While unconventional, using computer vision
on challenging, scientific datasets opens up new possibilities for computer
vision applications. One new possibility may include generating synthetic data
to supplement and enhance small, noisy training sets.

Our contributions are as follows:

  * •

Our method leverages explicit context labels available in DUSIA to generate
new training samples that combine existing training samples with empty
background frames available in sparsely populated video areas, illustrating
that cutting bounding boxes from the training set (as opposed to cutting more
precise, segmented class instances) can serve as an effective basis for data
augmentation.

  * •

We introduce a computationally inexpensive method for leveraging DUSIA’s CABOF
labels for generating effective training samples achieving state of the art
detection results on DUSIA’s validation and test sets using multiple different
popular object detection models.

Figure 2: Diagram illustrating the method for generating Context Matched
Collages. Mine bounding boxes from training set $\mathbf{T}$, background
frames from $\mathbf{B}$, match the context, and paste boxes on to a context
matched frame from $\mathbf{B}$. See Figure 1 caption for species name
abbreviations.

##  II Related Work

Computer vision researchers have long been aware of the power of data
augmentation methods for improving training object detectors and image
classifiers, and recently, much work has gone into generating plausible
training samples via cut/paste methods. Cut/paste methods take objects of
interest, cut them from their original image, and paste them into another
training image or other type of canvas (e.g. a blank background). DeVrires et
al. [2] introduce Cutout as a method of cutting portions from images during
training to help improve performance in the image classification task, and
CutMix [3] builds upon Cutout by combining two training samples at a time by
cropping a random part of one image and pasting it on to another image.

Cut, Paste, and Learn [4] leverages separate collections of common images of
objects and typical indoor scene examples to cut object instances from the
images available for training and paste them to random background scene
images. Cutting object instances from images relies on an image segmentation
model to separate objects from their backgrounds, and then those instances are
randomly pasted on random indoor images.

Ghiasi et al. [5] perform an augmentation similar to Cut/Paste, and Learn
where they cut object instances from one image to paste on to a different,
randomly selected image. They consider indoor vs outdoor images, which they
label based on COCO’s panoptic labels. They then use these augmented images to
train an image segmentation model. In the medical domain, TumorCP [6]
leverages image segmentation labels to create additional training samples for
a segmentation network, leading to better segmentation performance.

ObjectMix leverages instance segmentation labels to augment data for action
recognition in videos by extracting object segments from two videos and
combining them to create new video samples [7]. Similarly, Continuous Copy-
Paste works to leverage instance segmentation labels to generate training
samples for training models to solve the Multi-object Tracking problem [8].

Dvornik et al. show the importance of context in cut paste methods [9]. Their
method involves training a network using both bounding box and instance
segmentation labels to generate a notion of context where the bounding box
includes pixels that are not included by the segmentation label. They train a
network to then predict possible paste locations for cut out object instances
so that objects are pasted on to images that their model predicts to be
sensible.

Our method also employs a cut-paste based method and illustrates the
importance of context in our scenario but in a few key different ways. For
one, our method does not rely on any expensive segmentation labels, which
label every pixel in an image, or segmentation models, which may segment
objects unreliably. Section IV shows that directly cutting a whole bounding
box labelled as an object of interest from one frame and pasting it to another
frame with matching context enables an improvement in object detection
performance. This is important because segmentation labels are difficult and
expensive to collect in many scenarios with challenging, scientific data like
DUSIA [1], on which we present our results.

##  III Generating Context Matched Collages

Our method for generating synthetic frames from existing ones is a simple but
powerful extension of typical cut-paste methods. We introduce novel changes to
this method that allow us to generate better training samples for our specific
DUSIA dataset. DUSIA contains 10 hours of video captured in 1080p at 30 fps,
but across all of that video footage only 8,682 partially annotated frames
contain bounding box labels suitable for supervising most of today’s object
detectors. These frames are considered partially annotated because they may
contain individuals of species of interest that are not labelled with bounding
boxes.

Still, DUSIA provides CABOF labels for the entirety of its videos. These CABOF
labels indicate the first time at which a group or individual of a species of
interest intersects with the bottom of a video frame (like the yellow
gorgonian shown in Frame F of Figure 2) such that there is a single CABOF
label for every single individual of a species of interest that touches the
bottom of a video frame in DUSIA’s videos. Our method aims to leverage these
CABOF labels in a unique way.

---
Figure 3: Examples of generated collage images. Top: bounding boxes from
images labelled Cobble/Mud pasted on to an empty, background image with the
matched Cobble/Mud label. Bottom: images and background labelled Mud. See
caption of figure 1 for species abbreviations.

Unfortunately, a training set of 8,682 frames limits the performance of the
object detectors, but collecting additional bounding box annotations is
expensive, especially given the challenging nature of DUSIA and its object
classes. DUSIA’s partial annotation scheme alleviates the annotation burden on
expensive expert annotators, but the scheme makes it very difficult to train
an effective object detection network. McEver et al. [1] demonstrate that
Negative Region Dropping (NRD) can help train Faster RCNN based models on the
partially annotated dataset, but the detector that they present is far from
perfect. We propose combining DUSIA’s original training set with Context
Matched Collages as a complementary method that enables even better detection
performance.

The first step in generating these collages is to find a set of frames from
the videos that can serve as background for the synthetic collage training
samples. Ideally, these background frames contain a minimal number of species
individuals so that the resulting collages can have few unlabelled species
individuals in them. Pasting on to these sort of background frames allows the
object detector to see more of the video, helping it generalize on the test
set. Further, by pasting known objects on to empty frames, we can generate
frames that are better supervised than some of the partially annotated
training set, as they contain fewer unlabelled species of interest.

To this end, we generate a set of frames, $\mathbf{B}$, that are unlikely to
contain species of interest. In order to do so, we leverage the Count at
Bottom of Frame (CABOF) labels, which indicate timestamps containing species
individuals, provided for DUSIA’s videos. We can therefore use frames that are
far from all CABOF labels to find all the time spans that are unlikely to
contain a species of interest.

We first initialize $\mathbf{B}$ to contain all frames in the training set.
Then, we iterate through all CABOF labels removing frames within a certain
range (e.g. a few seconds) of any CABOF label time stamp. For example, if a
CABOF label indicates that there are three fragile pink urchins at time
00:10:30, we can remove all frames ranging from 00:10:20 to 00:10:40 from
$\mathbf{B}$. Because not all species individuals that appear in the video
touch the bottom of the video frame, they do not all get a CABOF label, but
many individuals do. Additionally, since many species typically occur
together, this step helps filter out very busy parts of the video with lots of
species of interest in them, regardless of whether they touch the bottom of
the frame because it is likely that their neighboring species individuals
touch the bottom of the frame if they do not. The remaining frames in
$\mathbf{B}$ may contain a few unlabelled species of interest, but there
should be far fewer unlabelled species than in the original training frames,
which are known to be partially supervised and typically come from busy parts
of the videos that contain many intermingling groups of species of interest.

Given that nearly all frames in DUSIA have substrate labels that indicate the
substrate present on the ocean floor, we also create a map $\mathbf{S}$ that
maps each substrate combination, $s$ to the frames from $\mathbf{B}$ that
contain that substrate combination. Having this mapping allows us to match the
context (i.e. substrate label) of potential background frames and the context
of any bounding boxes we wish to paste into a new collage frame.

In order to generate the final collages, we cut bounding boxes from the
original training set and paste them on to images from $\mathbf{B}$ that have
matching substrate labels. To do so, we map all substrate combinations to a
list of bounding boxes that exist on frames with that substrate combination
label. For each substrate combination, we randomly sample boxes, and paste
them to random locations in a randomly selected image from $\mathbf{B}$ that
has matching context labels. In our case we randomly select between 1 and 15
boxes to paste on each image. We also ensure that boxes do not fully occlude
one another, though we do allow partial overlap because species individuals
often cluster closely together in the original videos.

While the generated collage images may be obviously manipulated to a human
eye, they help train a stronger object detection model by providing better
supervision, unique co-occurrences of species individuals, and more samples.
We explore these improvements in Section IV

Algorithm 1 Pseudo code for generating Context Matched Collages

$b\leftarrow$ 10 seconds $\triangleright$ buffer

$B\leftarrow$ all training frames

$C\leftarrow$ all CABOF labels

for $c\in C$ do $\triangleright$ Create set of potential background frames

$T\leftarrow c.timestamp$

$B.remove([T-b,T+b])$

end for

$L\leftarrow$ map of each substrate label to empty list

for $f\in B$ do $\triangleright$ map substrate combos to bg frames

$L[f.substrate].append(f.timestamp)$

end for

$K\leftarrow$ map of each substrate label to bounding box labels

$M\leftarrow\emptyset$ $\triangleright$ list of generated frames

while not done do

$\triangleright$ k is substrate label, O is list of boxes on substrate k

for $k,O\in K.items()$ do

$l\leftarrow L[k]$ $\triangleright$ list of bg w/ label k

$m\leftarrow random.choice(l)$ $\triangleright$ m can serve as bg

$r\leftarrow random.randint([1,MAX\\_BOXES])$

for $i\in[0,r)$ do

$o\leftarrow random.choice(O)$

$O.remove(o)$

$m.paste(o)$ $\triangleright$ paste box o randomly on bg

end for

$M.append(m)$ $\triangleright$ add Context Matched Collage

if $len(M)$ > MIN then

$done\leftarrow True$

break

end if

end for

end while

Figure 3 shows example Context Matched Collages generated by our method.

##  IV Experiments

We train multiple model architectures by combining DUSIA’s 8,682 training
frames with collages generated via our method. We refer to those 8,682
training frames as $\mathbf{T}$. We paste a maximum of 15 boxes onto each
background image. We use a buffer of 10 seconds around each CABOF label to
ensure that our background images are sufficiently empty.

In order to illustrate the importance of context matching, we generate two
collage sets. $\mathbf{M}$ contains approximately 2,000 frames generated as
described in Section III with contexts matched properly. In practice, there
are a small number of bounding box labels with substrate combinations not
present in $\mathbf{B}$. For those frames, we simply map to the nearest
substrate combination. For example, if a box’s substrate label is both mud and
cobble, we paste it to a background frame that is either mud or cobble if
there are no frames in $\mathbf{B}$ with the exact same substrate label of mud
and cobble.

The second collage set, $\mathbf{R}$, contains approximately 2,000 frames
generated in exactly the same way as $\mathbf{M}$ except the context of the
background frame and the pasted object frame are _not necessarily_ matched.
That is, the background frame is randomly selected from all of $\mathbf{B}$
rather than randomly selected from a subset of $\mathbf{B}$ with the matched
substrate combination.

When training Faster R-CNN with negative region dropping, we saw success in
first training on $\mathbf{T}+\mathbf{M}$ then lowering the learning rate to
finetune on $\mathbf{T}$. We also present results on that setting indicated by
$\mathbf{T}+\mathbf{M},\mathbf{T}$.

First, we tested the Context-Driven Detector (CDD) as proposed by McEver et
al. [1]; however, we found that the detector performed better without the
context description branch when trained with our collage augmented training
sets. We theorize this may be due to the imperfect matching of context in some
frames. Still, we found negative region dropping to help overall performance,
and we set the negative region dropping percentage, $\rho$ to 0.75 as in the
original paper, and, also following [1], we used a learning rate of 0.01 for
Faster R-CNN with Negative Region Dropping.

We also trained YOLOv5 [10] on each of our training sets to demonstrate the
impact of including our Context Matched Collages. We trained the YOLOv5 large
model from the author’s provided weights, which were pre-trained on the COCO
dataset [11]. After testing a variety of different hyperparameter settings, we
found the best performance with most of the default settings; however, we
changed the initial learning rate, $lr_{0}$, from 0.001 to 0.0001; the final
OneCycleLR [12] learning rate, $lr_{f}$, from 0.1 to 0.01; the anchor-multiple
threshold [13], $anchor_{t}$, from 4.0 to 2.0; the image rotation, $degrees$,
from 0.0 to 30.0; and the image perspective [13], $perspective$, from 0.0 to
0.001. After training using the above settings, we finetuned on $\mathbf{T}$
lowering $lr_{0}$ to 1e-6.

Finally, we tested the DEtection TRansformer (DETR) [14] to give an additional
example of a state of the art object detection framework. Starting from the
author’s provided pretrained weights, we trained DETR with minor changes to
the default settings. We used ResNet-50 as the backbone and changed learning
rate to 1e-5 and the learning rate drop to 40 epochs.

Table I shows the experimental results of all three detectors trained on the
different training sets without any collages, including the Context Matched
Collages, and the collages with random backgrounds. We evaluate our detection
results using mean Average Precision (mAP) with an intersection over union
(IOU) threshold of 0.5 following [1]. We choose this metric, widely known as
AP50, because the detection tasks in DUSIA is not sensitive to exact
localization as the detection tasks ultimately aim to aid in counting of
invertebrate species. For more in depth analysis, we also present the popular
full COCO suite [15] of evaluation metrics for our best detection model,
Faster R-CNN with Negative Region Dropping trained on
$\mathbf{T}+\mathbf{M},\mathbf{T}$ in Table II.

For our best model we also present some qualitative detection results in
Figure 4. The results for this busy frame show that the detector performs
quite well at finding objects at the edge of the frame, and it does a good job
discriminating among species. It even leaves out a few sponge species that are
not species of interest. The detector still struggles with the red swifitia
gorgonian (RSG) class, which is one of the most challenging in DUSIA, showing
some area for improvement.

Detector | Train Set | val mAP | test mAP
---|---|---|---
Context Driven Detector [1] | T | 0.524 | 0.447
DETR [14] | T | 0.534 | 0.416
DETR | T+R | 0.541 | 0.426
DETR | T+M | 0.541 | 0.446
YOLOv5 [10] | T | 0.558 | 0.452
YOLOv5 | T+R | 0.518 | 0.437
YOLOv5 | T+M | 0.558 | 0.470
Faster RCNN [16] w/ NRD [1] | T | 0.509 | 0.439
Faster RCNN w/ NRD | T+R | 0.511 | 0.419
Faster RCNN w/ NRD | T+M | 0.542 | 0.453
Faster RCNN w/ NRD | T+M, T | 0.546 | 0.482
TABLE I: Different object detectors, and their detection performance given different training sets. CDD results from [1] metric | val | test
---|---|---
AP50:95 | 0.264 | 0.221
AP50 | 0.553 | 0.482
AP75 | 0.224 | 0.174
APS | 0.017 | 0.016
APM | 0.168 | 0.165
APL | 0.335 | 0.286
TABLE II: Full COCO suite of metrics showing the performance of the best
Faster R-CNN with NRD model on both the val and test sets

For all three detectors, we see better generalizability in the model, as
evidenced by test mAP, when the model is trained with Context Matched
Collages. We also see that training with Context Matched Collages
($\mathbf{T}+\mathbf{M}$) consistently achieves better performance than
training with the the collages without context matching
($\mathbf{T}+\mathbf{R}$). YOLOv5 and Faster RCNN even see decreased
performance when trained with collages in $\mathbf{R}$ illustrating the
importance of context when detecting invertebrate species. Faster R-CNN
achieves state of the art performance on the test set when trained with
Context Matched Collages after finetuning on the original training set.
Clearly, augmenting DUSIA’s training set with Context Matched Collages leads
to better overall performance.

Figure 4: Detections from best model (top) and ground truth (bottom) for an
example frame. See figure 1 caption for species name abbreviations.

##  V Discussion

In this paper, we introduce Context Matched Collages. We mine frames
containing few species of interest, cut bounding boxes from our training set,
and paste those bounding boxes on to the mined images. This process leverages
many unused video frames and produces unique training samples that aid in
training object detectors to increase performance. We illustrate that cutting
bounding boxes, as opposed to finely segmented object instances, and pasting
them to create new training samples provides an effective augmentation for
object detectors. Further, we introduce matching the explicit context labels
of bounding boxes and the background to create collages. By augmenting the
original training set of DUSIA with these Context Matched Collages, we are
able to achieve state of the art object detection performance.

Even with these improvements, detection on DUSIA remains a challenging task,
and the detection performance still needs improvement to alleviate the manual
detection and counting of invertebrate species. The low performance on small
objects, shown as APS in Table II shows reveals an area for improvement, and
the qualitative results in Figure 4 show that measures may be taken to improve
the performance on the red swiftia gorgonian class and perhaps other specific
classes.

##  VI Acknowledgements

This research was supported in part by National Science Foundation (NSF)
award: SSI # 1664172. We would like to thank Dirk Rosen and Andy Lauermann
from Marine Applied Research & Exploration group for their video collection,
guidance, and help through this project. We would also like to thank Dr.
Robert Miller for their contributions to the project.

## References

  * [1] R. A. McEver, B. Zhang, C. Levenson, A. Iftekhar, and B. Manjunath, “Context-driven detection of invertebrate species in deep-sea video,” _arXiv preprint arXiv:2206.00718_ , 2022.
  * [2] T. DeVries and G. W. Taylor, “Improved regularization of convolutional neural networks with cutout,” _arXiv preprint arXiv:1708.04552_ , 2017.
  * [3] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. J. Yoo, “Cutmix: Regularization strategy to train strong classifiers with localizable features,” _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 6022–6031, 2019.
  * [4] D. Dwibedi, I. Misra, and M. Hebert, “Cut, paste and learn: Surprisingly easy synthesis for instance detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 1301–1310.
  * [5] G. Ghiasi, Y. Cui, A. Srinivas, R. Qian, T.-Y. Lin, E. D. Cubuk, Q. V. Le, and B. Zoph, “Simple copy-paste is a strong data augmentation method for instance segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 2918–2928.
  * [6] J. Yang, Y. Zhang, Y. Liang, Y. Zhang, L. He, and Z. He, “Tumorcp: A simple but effective object-level data augmentation for tumor segmentation,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2021, pp. 579–588.
  * [7] J. Kimata, T. Nitta, and T. Tamaki, “Objectmix: Data augmentation by copy-pasting objects in videos for action recognition,” _arXiv preprint arXiv:2204.00239_ , 2022.
  * [8] Z. Xu, A. Meng, Z. Shi, W. Yang, Z. Chen, and L. Huang, “Continuous copy-paste for one-stage multi-object tracking and segmentation,” in _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021, pp. 15 303–15 312.
  * [9] N. Dvornik, J. Mairal, and C. Schmid, “Modeling visual context is key to augmenting object detection datasets,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 364–380.
  * [10] G. Jocher, A. Stoken, J. Borovec, A. Chaurasia, and L. Changyu, “ultralytics/yolov5,” _Github Repository, YOLOv5_ , 2020.
  * [11] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _Computer Vision – ECCV 2014_ , D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, Eds. Cham: Springer International Publishing, 2014, pp. 740–755.
  * [12] L. N. Smith and N. Topin, “Super-convergence: Very fast training of neural networks using large learning rates,” in _Artificial intelligence and machine learning for multi-domain operations applications_ , vol. 11006. SPIE, 2019, pp. 369–386.
  * [13] G. Jocher, “ultralytics/yolov5: v3.1 - Bug Fixes and Performance Improvements,” https://github.com/ultralytics/yolov5, Oct. 2020. [Online]. Available: https://doi.org/10.5281/zenodo.4154370
  * [14] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _European conference on computer vision_. Springer, 2020, pp. 213–229.
  * [15] Z.-Q. Zhao, P. Zheng, S.-t. Xu, and X. Wu, “Object detection with deep learning: A review,” _IEEE transactions on neural networks and learning systems_ , vol. 30, no. 11, pp. 3212–3232, 2019.
  * [16] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” _Advances in neural information processing systems_ , vol. 28, 2015.