# Propagating Semantic Labels in Video Data

David Balaban, Justin Medich, Pranay Gosar, and Justin Hart1 1Departent of
Computer Science, The University of Texas at Austin, Austin, Texas, USA 78712
{dbalaban, jmedich, pgosar<EMAIL_ADDRESS>
###### Abstract

Semantic Segmentation combines two sub-tasks: the identification of pixel-
level image masks and the application of semantic labels to those masks.
Recently, so-called Foundation Models have been introduced; general models
trained on very large datasets which can be specialized and applied to more
specific tasks. One such model, the Segment Anything Model (SAM), performs
image segmentation. Semantic segmentation systems such as CLIPSeg and MaskRCNN
are trained on datasets of paired segments and semantic labels. Manual
labeling of custom data, however, is time-consuming. This work presents a
method for performing segmentation for objects in video. Once an object has
been found in a frame of video, the segment can then be propagated to future
frames; thus reducing manual annotation effort. The method works by combining
SAM with Structure from Motion (SfM). The video input to the system is first
reconstructed into 3D geometry using SfM. A frame of video is then segmented
using SAM. Segments identified by SAM are then projected onto the the
reconstructed 3D geometry. In subsequent video frames, the labeled 3D geometry
is reprojected into the new perspective, allowing SAM to be invoked fewer
times. System performance is evaluated, including the contributions of the SAM
and SfM components. Performance is evaluated over three main metrics:
computation time, mask IOU with manual labels, and the number of tracking
losses. Results demonstrate that the system has substantial computation time
improvements over human performance for tracking objects over video frames,
but suffers in performance.

##  I INTRODUCTION

This work presents a Semantic Label Propagation (SLP) system for labeling
large sets of images derived from video; providing a dataset suitable for
training or fine-tuning models for computer vision tasks with minimal manual
labeling. The system takes as input video from a moving camera in a static
scene and outputs a set of objects, each object corresponds to a set of binary
masks for each frame it is identified in.

Figure 1 diagrams the flow of data through of the proposed system. The
triangle on the left represents the “Video Input” to the system. The hexagon
at the top represents the output, “Objects Labeled Across Frames.” Ellipses
represent data, while squares represent sub-processes. This system begins with
a Structure from Motion (SfM) pipeline to gain an object mesh [1]. Then,
Segment Anything Model (SAM) is used to identify the distinct objects in the
mesh [2]. These objects are then tracked through all frames of video utilizing
the object mesh to enforce geometric consistency while SAM associates
previously unseen polygons to tracked objects.

Figure 1: Semantic Label Propagation Overview

To evaluate system performance, ground truth semantic labels are collected
from five videos. Volunteers use the open-source PixelAnnotationTool to
manually label objects in frame [3]. The volunteer work is spot checked by the
authors. This data will be made publicly-available upon publication of this
work, at the Texas Robotics Dataverse:
https://dataverse.tdl.org/dataverse/robotics

To demonstrate the performance contribution of each component of the semantic
label propagation pipeline, four variants are presented which vary how prompts
are selected as input to SAM:

  1. 1.

SAM-only-1.0, $k$-best prompts

  2. 2.

SAM-only-2.0, hill-climb for best prompt

  3. 3.

SfM-SAM-1.0, $k$-best SfM candidate prompts

  4. 4.

SfM-SAM-2.0, $k$-random SfM candidate prompts

These variants are fully described in Section III.

The novel contributions of this work are the use of SfM to propagate geometry
into new frames, as described in Section III-D4; enhancements of the use of
feature embeddings provided by SAM, as described in Section III-C; a direct
comparison of automatic labeling of video to volunteer manual labeling, as
described in Section IV.

##  II Related Work

Tasks such as Object Detection [4, 5], Image Captioning [6, 7], and Semantic
Segmentation [8, 9] are approaches to image labeling for the purposes of scene
understanding [10, 11]. Object Detection localizes a number of desired objects
within the image frame, typically with a bounding box for each object [4, 5].
Image Captioning gives a general description of the image, often using a text
prefix provided to a large language model [6, 7]. Semantic Segmentation
provides pixel-level annotations of the image [8, 9].

Pre-trained models have been released for people to fine-tune with custom data
[12, 13, 14], which can require an arduous amount of manual labeling. There
has been a significant effort to design models that require as few training
examples as possible [15]. One such model is CLIP, which is a joint embedding
between language and images that identifies the text which best describes an
image [16, 12]. CLIP is pre-trained with multiple datasets, which recent work
suggests leads to a lower quality of data [17]. There is a need for a method
which generates large high-quality datasets of semantically labeled images
with minimal manual labeling.

Video Object Segmentation and Tracking (VOST) refers to the coupled nature of
pixel-level annotations and object tracking, as solving one improves the
solution to the other [18]. Multiple Object Tracking (MOT) identifies the
trajectory many objects take through a scene [19].

Segment Anything Model (SAM) is a recently released vision transformer which
effectively handles the annotation piece of the VOST problem [2]. SAM segments
images on a pixel-level without semantic annotations. With SAM, a single pixel
manual label is sufficient to semantically label a whole object, and identify
that object in other frames and contexts [20]. This feature can be utilized to
track objects across many frames of a video [21].

Structure from Motion (SfM) is the problem of reconstructing a scene from
images taken at different locations, such as in a video [1]. Recent research
has studied SfM methods in both outdoor and indoor settings [22, 23]. Classic
approaches to SfM utilize optimization schemes such as Levenberg-Marquardt
[24] to jointly solve for camera poses and pixel depths, however more recent
research uses a deep learning model for this optimization [25].

Prior work on the collection of reliable datasets for training CV models with
minimal manual labeling include utilizing 3D object representations jointly
learned with object association to 2D image labels [26], and generating
datasets by combining object segmentation with optical flow methods [27].

##  III Label Propagation Procedures

Algorithm 1 describes the proposed system. The algorithm takes as input a SAM
model, a video, a positive integer parameter $F$, and additional
implementation-specific parameters described in Section III-D. A list of
objects is tracked across video frames, objects in this list are referred to
as known objects. Each known object has associated characteristic features of
which there are two types: SAM features which describe an object’s visual
appearance, and mesh faces which describe an object’s 3D geometry. One or both
of these types may be used.

At each frame, known objects are identified in frame by searching for matching
characteristic features. Searching for and updating characteristic features is
handled by the FIND_OBJECT function in Algorithm 1, described in detail in
Section III-D. This function returns a binary mask identifying the object’s
location in frame, called the object mask. The object mask and frame is
recorded as a successful label propagation.

Once all the known objects are handled, new objects are identified to add to
the list of known objects. The union of all object masks, named the seen mask,
is used to avoid searching for new objects in regions of the frame that have
been associated to known objects. The parameter $F\in\mathbb{N}$ causes the
procedure to skip $F$ frames before adding new objects for faster computation.
All object masks are identified by SAM, as described in Section III-C.

Algorithm 1 Semantic Label Propagation

0: SAM, Video, $F$, *args

1: Objs $\leftarrow\\{\\}$

2: $f\leftarrow 0$

3: for frame in Video do

4: seen_mask $\leftarrow$ empty

5: for obj in Objs do

6: obj_mask $\leftarrow$ FIND_OBJECT(frame, obj, *args)

7: if obj_mask not empty then

8: obj.append(frame, obj_mask)

9: seen_mask $\leftarrow$ seen_mask $\cup$ obj_mask

10: end if

11: end for

12: if $f\geq F$ then

13: seen_mask $\leftarrow$ Closing(seen_mask)

14: new_masks $\leftarrow$ SAM.getMasks(frame, seen_mask)

15: Objs.append(new_masks, frame)

16: $f\leftarrow 0$

17: else

18: $f\leftarrow f+1$

19: end if

20: end for

21: return Objs

###  III-A Structure from Motion

This paper uses a COLMAP to MVS implementation of SfM where COLMAP provides an
initial guess to MVS [28, 29, 30]. Given a video input of a camera moving
around a scene, COLMAP will extract distinctive features in the scene that can
be matched between frames of the video. MVS enhances this process by allowing
the SfM pipeline to estimate the depth of objects within the frames, allowing
for a more detailed 3D model. The pipeline outputs a 3D reconstruction of the
scene in the form of a mesh of polygon faces, called the scene mesh. It also
outputs the video and camera poses that capture the estimated positions and
orientations of the camera used. This output is sufficient to identify which
mesh face every pixel observes. Segmented frames outputted by SAM are then
projected onto the reconstruction which allows less invocations to SAM for
future frames.

###  III-B Segment Anything Model

SAM takes a set of $k\geq 1$ pixel locations as a prompt, and returns a binary
mask associated with each prompt to identify objects in frame. SAM identifies
masks with a joint image and feature encoding. Image encoding provides a
feature vector at each pixel location, while the prompt encoding provides a
feature vector for each prompt. The cosine similarity between feature vectors
indicates the likelihood of a prompt-pixel pair being a part of the same
segment.

###  III-C Getting Object Masks from SAM

When tracking known objects, a prompt is provided to SAM as a sample of the
object’s expected location in frame, producing an object mask. The union of
all known object masks in the current frame is the seen mask.

SAM.getMasks() searches for unseen objects, this function uses a grid of
$g$x$g$ single-pixel prompts, called the prompt grid. The seen mask is used to
filter prompts that have already been associated to known objects, with
$g^{\prime}$, $0\leq g^{\prime}\leq g^{2}$, total prompts used in each frame.
SAM will then produce $M$, $0\leq M\leq g^{\prime}$, new object masks.

The seen object mask tends to exclude borders between objects, but due to the
proximity of known objects, prompts in these regions tend to produce
duplicates or near-duplicates of these objects, leading to longer computation
times. To counter this problem, the Closing image morphology is applied to the
seen mask from the open-source software OpenCV, thereby adding the object
boarders to the seen mask [31, 32].

OpenCV’s Closing Image Morphology transforms a binary mask in an iterative
process. When Closing, at each iteration two other morphologies are applied in
succession, a dilation followed by an erosion. The dilation expands the white
space of the binary mask and the erosion expands the black space. The
resulting effect is that small regions of black space are removed while
boundaries between substantially sized white and black regions are maintained.
By applying this morphology to the seen mask, the boarders between known
objects is eliminated while unseen regions remain unaltered. [31, 32]

Figure 2 shows the effect of Closing() with a color coded filter added to a
frame of video. Blue regions show the raw seen mask, red shows the regions
added to the seen mask by Closing(), and green shows the final unseen mask.
Only the prompt grid points in the green regions (as seen in the top of Figure
2) will be used to find new objects, which significantly reduces the number of
duplicated tracked objects without sacrificing the ability to observe new
objects. In individual images, the green regions are small and appear only at
the edges of individual frames. This leads to a substantial speed-up over
running SAM on every frame of video.

Figure 2: Seen mask image morphography by color filter: blue, raw seen mask;
red, seen by image morphography; green, unseen.

###  III-D FIND_OBJECT Procedure

FIND_OBJECT() in Algorithm 1 takes as input the current frame and a desired
object to identify in frame and returns the object mask if found. It also
handles the update of characteristic features for the known object: visual,
geometric, or both. Additional parameters are required depending on which
features are used, corresponding to different function implementations.

####  III-D1 SAM Only Features

Algorithm 2 describes the FIND_OBJECT() implementation when only visual
features are used, it takes an additional integer parameter $k$ as input.
Visual features come from SAM’s image and prompt joint encoding.

The first time an object is identified, the prompt features associated with
the object mask returned by SAM are recorded as the characteristic visual
feature. These objects can be identified in future frames by maximizing the
cosine similarity of the recorded features across the image encoding.

####  III-D2 K-best approach

Pixels are scored by the cosine similarity, and the $k$-best pixels are used
to generate a prompt that identifies the object in frame. When $k>0$, all
pixels are scored, which can be an expensive process with large feature
vectors.

####  III-D3 Hill-climbing approach

When $k\leq 0$, the last prompt location is recorded along with the visual
features. This location is used as an initial guess to a discrete hill-
climbing procedure that searches for the maximum score locally. At each step
of the hill-climb, only the neighboring pixels need be scored, choosing the
best from among them as the location for the next iteration until all
neighbors score lower than the current location. This final location is used
as the single-pixel prompt.

The feature vector is updated to account for changes in lighting and viewing
angles.

Algorithm 2 SAM-only FIND_OBJECT

0: frame, obj, SAM, $k$

1: image_features $\leftarrow$ SAM.encodeImage(frame)

2: pixel_scores $\leftarrow$ DotProd(obj.features, image_features)

3: if $k>0$ then

4: prompt $\leftarrow$ kBestScores(pixel_scores)

5: else

6: prompt $\leftarrow$ hillClimb(pixel_scores, obj.last_prompt)

7: end if

8: if score at prompt $<$ threshold then

9: return none

10: end if

11: obj.last_prompt $\leftarrow$ prompt

12: obj.features $\leftarrow$ image_features.at(prompt)

13: mask $\leftarrow$ SAM.getMask(prompt, frame)

14: new_obj $\leftarrow$ (mask, frame, prompt)

15: return new_obj

####  III-D4 Using Geometric Features

Algorithm 3 describes the FIND_OBJECT implementation when geometric features
are included. This algorithm requires the integer parameter $k$, a scene mesh,
and a binary variable isRandom. This implementation requires the SfM algorithm
to be run beforehand to obtain a scene mesh and camera trajectory. The scene
mesh consists of a set of connected triangles describing the geometry of the
scene. The camera trajectory is described by the poses of the camera at each
frame. This information is sufficient to identify which pixels observe which
mesh face.

For brevity, pixel-triangle matches are included in the mesh object as part of
the setup in Algorithm 3, mesh.getObj() returns the mesh faces currently known
to be part of a known object - this is called the matching set -
mesh.getPixelMatches() maps from triangles to pixels in a given frame,
mesh.getMeshMatch() maps from pixels to triangles, and mesh.updateObj() adds
previously un-seen triangles to a known object.

Given the matching set, a SAM prompt simply needs to be selected from among
these pixels. The binary variable isRandom determines the prompt selection
methodology.

Given the object mask from SAM, the mesh object is updated by adding the
unseen mesh faces matched to the mask and removing the mesh faces associated
to obj_pixels but not in the SAM mask.

If isRandom is False, visual features are recorded in addition to geometric.
In this case, the $k$-best pixels are selected by visual feature matching as
in Section III-D1. However, only the pixels in the matching set are scored. If
isRandom is True, then $k$ pixels are selected at random from among the
matching set. In this case, visual features are not tracked and SAM does not
need to provide an image encoding.

Figure 3 shows a visual of the update step for an object in two frames. The
figure shows cropped versions of the second (top) and third (bottom) frames of
video 1. Each frame has been colored to show the update taking place to the
object. A red filter was applied to obj_pixels; a green filter was applied to
obj_mask, and a blue filter to the pixels not in the obj_mask. The resulting
colors show which pixel-mesh matches are added (green), maintained (yellow),
or discarded (purple) as part of the object. As shown, the left wheel was not
initially part of the object mesh after the first frame, added in the second
frame, persisted in the third while previously out-of-frame pixels are added.

The removal of mesh faces is designed to handle cases with imperfect scene
reconstructions so that errors in the reconstruction do not accumulate. More
precise reconstructions may not need this removal step.

Algorithm 3 SfM-SAM FIND_OBJECT

0: frame, obj, SAM, $k$, mesh, isRandom

1: obj_mesh $\leftarrow$ mesh.getObj(obj)

2: obj_pixels $\leftarrow$ mesh.getPixelMatches(frame, obj_mesh)

3: if obj_pixels is empty then

4: return empty

5: end if

6: if isRandom then

7: prompt $\leftarrow$ kRandom(obj_pixels)

8: else

9: image_features $\leftarrow$ SAM.encodeImage(frame)

10: pixel_features $\leftarrow$ image_features.at(obj_pixels)

11: pixel_scores $\leftarrow$ Dot(obj.features, pixel_features)

12: prompt $\leftarrow$ kBest(pixel_scores)

13: obj.last_prompt $\leftarrow$ prompt

14: obj.features $\leftarrow$ image_features.at(prompt)

15: end if

16: obj_mask $\leftarrow$ SAM.getMask(prompt, frame)

17: mesh_update $\leftarrow$ mesh.getMeshMatch(obj_mask, frame)

18: mesh.updateObj(obj, mesh_update)

19: new_obj $\leftarrow$ (mask, frame)

20: return new_obj

(a) Second Frame Object Update

(b) Third Frame Object Update

Figure 3: (best viewed in color) SfM-SAM object update step; green - pixel-
mesh faces added by SAM; red - pixel-mesh faces removed by SAM; yellow -
pixel-mesh maintained; blue - not part of object

##  IV Evaluation

We evaluate SLP on five videos for time efficiency, accuracy, and tracking
loss. SLP is run without any manual labels provided. Objects tracked are
objects proposed by SAM.

Time efficiency is determined by both the total run time, and the frame
processing rate. Computation is done on a machine with 4 a100 GPUs, 527821928
kB of RAM, and an Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz.

To determine accuracy, a ground truth label is needed. Each video is manually
labeled and segmented by volunteers to provide ground truth. The manual labels
are spot-checked by the authors to ensure quality with ten frames from each
video chosen at random. Each selected frame was manually labeled a second time
by the authors and quality was assessed by the Intersection over Union (IoU)
score between objects.

The IoU score is also used to assess the accuracy of SLP against ground truth.
Because no manual labels are supplied to SLP, the objects found by SAM must be
matched to the true object mask against which to compare. These matches are
determined by finding the SAM mask with the highest IoU score for each manual
mask.

A tracking loss occurs when a true label is matched to a different object than
what was matched in the previous frame. Such an event indicates that SLP
falsely determined that an object was not in frame and a new instance was
added. Tracking losses can be considered as a false negative.

False positives would be when SAM recognizes objects that are not manually
labeled. However, SAM is designed to recognize objects and component objects
independently, while the volunteers were only instructed to label full
objects. Therefore, it would not be a fair comparison to evaluate performance
with this metric against ground-truth.

The parameters $k$ and $F$ are considered for comparison of methods across
different settings. $k$ is the number of pixels used per SAM prompt when
identifying known objects in a new frame. $F$ is the number of frames skipped
before SAM searches for new objects.

To assess the significance of using geometric features, four variants of SLP
are evaluated. Each variant uses a different implementation of FIND_OBJECT()
described in Section III-D and used in Algorithm 1.

####  IV-1 SAM-only-1.0

uses Algorithm 2 with $k\geq 0$, described in Section III-D1, and does not use
any geometric features. This is the baseline algorithm.

####  IV-2 SAM-only-2.0

uses Algorithm 2 with $k=0$. This variant scores fewer pixels than SAM-
only-1.0.

####  IV-3 SfM-SAM-1.0

uses Algorithm 3 with isRandom set to False, described in Section III-D4. This
variant uses mesh faces of a scene to track objects.

####  IV-4 SfM-SAM-2.0

uses Algorithm 3 with isRandom set to True. It avoids scoring any pixels and
does not track visual features, relying only on geometric features.

##  V Results

A summary of the data-sets created by each video is shown in Table I, each row
corresponds to a different video and the column labels describe the following:

  * •

$N_{f}$ \- Number of frames in video

  * •

$T_{sfm}$ \- Time to complete SfM in minutes

  * •

$N_{m}$ \- Number of mesh faces

  * •

$T_{pxl}$ \- Time to complete pixel matching in minutes

  * •

$FN_{v}$ \- Number of unlabeled objects in spot check across all frames

  * •

$FN_{a}$ \- Number of objects added against spot check across all frames

  * •

$\overline{IoU_{s}}$ \- Average IoU across all frames and objects in spot
check

The last three columns refer to results from the spot check analysis, and the
three columns preceding them refer to the SfM setup that is done for the SfM-
SAM methods.

$N_{f}$ | $T_{sfm}$ | $N_{m}$ | $T_{pxl}$ | $FN_{v}$ | $FN_{a}$ | $\overline{IoU_{s}}$
---|---|---|---|---|---|---
281 | 23 | 558440 | 941 | 2 | 1 | 0.940
313 | 24 | 600065 | 1123 | 5 | 1 | 0.925
402 | 57 | 640527 | 1508 | 19 | 1 | 0.709
239 | 15 | 518774 | 692 | 17 | 0 | 0.784
61 | 3 | 380697 | 128 | 13 | 0 | 0.817
TABLE I: data summary for videos 1-5

###  V-A Spot Check Data

To calculate the IoU score for each object in each frame, the Author-Labeled
Object Mask (ALOM) is compared against each Volunteer-Labeled Object Mask
(VLOM), and the VLOM yielding the highest IoU score is taken as the best score
for that ALOM. Note that a VLOM might be matched to more than one ALOM if a
VLOM overlaps a few ALOM, or if there are more ALOM than VLOM. The authors
verified that the former does not happen in any spot-checked frame.
$\overline{IoU_{s}}$ is then the average best-IoU score across all ALOMs and
all ten spot-checked frames.

If there are more VLOMS than ALOMS, then the volunteer identified one or more
objects the author did not, indicating a false negative on the author’s part.
However, if there are more ALOMS than VLOMS, then the author identified more
objects than the volunteer, which indicates false nagatives made by the
volunteer. The total number of such false negatives are tracked by $FN_{a}$
and $FN_{v}$ respectively.

Some example results of IoU scores from the spot-check analysis are shown in
Figure 4. Each line represents an object’s IoU score in each spot-checked
frame. The frame ID for that video is charted on the x-axis, and the best IoU
score on the y-axis. Some lines in Figure 4 abruptly start or end because not
all objects are visible in every frame.

Figure 4: Spot check IoU scores, video 5.

An example of the discrepancy between author-labeled and volunteer-labeled
frames is shown in Figure 5. This figure shows frame 73 of video 5 with
overlays of the object masks identified by the volunteer (top) and the author
(bottom). Each object mask is color-coded by the same pallet shown in the
legend of Figure 4. The IoU scores for each object can be seen in Figure 4
with the left-most set of objects.

(a) Volunteer Labeled Image

(b) Author Labeled Image

Figure 5: (best viewed in color) comparison of labeled data (top) to author’s
spot check (bottom), frame 73, video 5

The authors labeled one frame every 20min on average, while the volunteers
labeled one frame every 10min.

###  V-B Semantic Label Propagation Data

The performance of SLP is evaluated across different parameter settings in
five videos. The best tuned parameters are presented for each of three
different performance metrics.

Table II shows the results for the parameter set which gives the fastest
computation time. The rows of the table give the implementation indicated by
the first column. The next three columns give each of the performance metrics,
total compuation time, average IoU, and total number of tracking losses. These
metrics are evaluated across all five videos used. The last two columns give
the parameter values $k$ and $F$ yielding the fastest computation time, three
values of each are tested $k=[1,2,5]$, $F=[0,1,4]$.

Table III and Table IV are organized in a similar manner, however Table III
displays the highest average IoU, and Table IV the lowest number of tracking
losses.

Implementaion | Time (min) | IoU | Tracking Loss | $k$ | $F$
---|---|---|---|---|---
SAM-only-1.0 | 260 | 0.396 | 3746 | 2 | 0
SAM-only-2.0 | 2109 | 0.323 | 1389 | 1 | 4
SfM-SAM-1.0 | 1741 | 0.188 | 5820 | 5 | 4
SfM-SAM-2.0 | 4116 | 0.222 | 8327 | 5 | 4
TABLE II: best performing implementation by time Implementaion | Time (min) | IoU | Tracking Loss | $k$ | $F$
---|---|---|---|---|---
SAM-only-1.0 | 286 | 0.396 | 3771 | 1 | 1
SAM-only-2.0 | 2188 | 0.396 | 3681 | 1 | 0
SfM-SAM-1.0 | 9200 | 0.253 | 15793 | 1 | 0
SfM-SAM-2.0 | 6765 | 0.274 | 15031 | 2 | 0
TABLE III: best performing implementation by IoU score Implementaion | Time (min) | IoU | Tracking Loss | $k$ | $F$
---|---|---|---|---|---
SAM-only-1.0 | 260 | 0.396 | 3746 | 2 | 0
SAM-only-2.0 | 2110 | 0.323 | 1389 | 1 | 4
SfM-SAM-1.0 | 2786 | 0.192 | 2196 | 2 | 4
SfM-SAM-2.0 | 4116 | 0.222 | 8327 | 5 | 4
TABLE IV: best performing implementation by tracking losses

##  VI Discussion

The fastest implementation of SLP was SAM-only-1.0 at 260min over 1296 frames
in all five video data collected. A human equivalent volunteer would have
taken 12960min to accomplish the same, while the more meticulous authors would
have taken 25,920min. The fastest computation time of SAM-only-1.0 did not
suffer on accuracy compared to the other methods. The IoU score for the
fastest time was within three decimal place rounding of the fastest IoU score
overall, although performing worse than human volunteers. SAM-only-1.0 only
had a significant disadvantage on tracking loss.

SAM-only-2.0 and SfM-SAM-1.0 showed improvement tracking losses. It is
possible this is due to SAM-only-2.0’s use of locality when searching for
known objects and the geometric features used in SfM-SAM-1.0. SfM-SAM-2.0
actually had a performance decrease, possibly because of its random prompt
selection leading to sub-optimal outcomes.

##  VII Conclusion

This work presents a system named Semantic Label Propagation for automatic
labeling of video data suitable for training a semantic segmentation model.
The system tracks visual and geometric features of objects across frames to
track segments which are part of previously identified objects. The trade-off
in computational effort versus manual effort means that this technique can be
incorporated into a tool to speed up human annotations by allowing annotators
to make a few corrections to an annotated video, rather than manually labeling
each frame; speeding up their overall effort.

## ACKNOWLEDGMENT

This work has taken place in the Living with Robots Laboratory (LWR) at UT
Austin. LWR research is supported in part by NSF (NRT-2125858 and
GCR-2219236), Cisco Research, and Army Futures Command.

## References

  * [1] Onur Özyeşil, Vladislav Voroninski, Ronen Basri and Amit Singer  “A survey of structure from motion*.”  In _Acta Numerica_ 26 Cambridge University Press, 2017, pp. 305–364
  * [2] Alexander Kirillov et al.  “Segment Anything”  In _arXiv:2304.02643_ , 2023
  * [3] Amaury Bréhéret  “Pixel Annotation Tool”, https://github.com/abreheret/PixelAnnotationTool, 2017
  * [4] Syed Sahil Abbas Zaidi et al.  “A survey of modern deep learning based object detection models”  In _Digital Signal Processing_ 126 Elsevier, 2022, pp. 103514
  * [5] Juan Terven and Diana Cordova-Esparza  “A comprehensive review of YOLO: From YOLOv1 to YOLOv8 and beyond”  In _arXiv preprint arXiv:2304.00501_ , 2023
  * [6] Matteo Stefanini et al.  “From show to tell: A survey on deep learning-based image captioning”  In _IEEE transactions on pattern analysis and machine intelligence_ 45.1 IEEE, 2022, pp. 539–559
  * [7] Junnan Li, Dongxu Li, Caiming Xiong and Steven Hoi  “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation”  In _International Conference on Machine Learning_ , 2022, pp. 12888–12900  PMLR
  * [8] Yujian Mo et al.  “Review the state-of-the-art technologies of semantic segmentation based on deep learning”  In _Neurocomputing_ 493 Elsevier, 2022, pp. 626–646
  * [9] Liulei Li et al.  “Deep hierarchical semantic segmentation”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 1246–1257
  * [10] Zhe Gan et al.  “Vision-language pre-training: Basics, recent advances, and future trends”  In _Foundations and Trends® in Computer Graphics and Vision_ 14.3–4 Now Publishers, Inc., 2022, pp. 163–352
  * [11] Muhammad Awais et al.  “Foundational models defining a new era in vision: A survey and outlook”  In _arXiv preprint arXiv:2307.13721_ , 2023
  * [12] Ron Mokady, Amir Hertz and Amit H Bermano  “Clipcap: Clip prefix for image captioning”  In _arXiv preprint arXiv:2111.09734_ , 2021
  * [13] Tausif Diwan, G Anirudh and Jitendra V Tembhurne  “Object detection using YOLO: Challenges, architectural successors, datasets and applications”  In _multimedia Tools and Applications_ 82.6 Springer, 2023, pp. 9243–9275
  * [14] Pavel Iakubovskii  “Segmentation Models”  In _GitHub repository_ GitHub, https://github.com/qubvel/segmentation_models, 2019
  * [15] Yaqing Wang, Quanming Yao, James T Kwok and Lionel M Ni  “Generalizing from a few examples: A survey on few-shot learning”  In _ACM computing surveys (csur)_ 53.3 ACM New York, NY, USA, 2020, pp. 1–34
  * [16] Alec Radford et al.  “Learning transferable visual models from natural language supervision”  In _International conference on machine learning_ , 2021, pp. 8748–8763  PMLR
  * [17] Thao Nguyen et al.  “Quality not quantity: On the interaction between dataset design and robustness of clip”  In _Advances in Neural Information Processing Systems_ 35, 2022, pp. 21455–21469
  * [18] Rui Yao et al.  “Video object segmentation and tracking: A survey”  In _ACM Transactions on Intelligent Systems and Technology (TIST)_ 11.4 ACM New York, NY, USA, 2020, pp. 1–47
  * [19] Wenhan Luo et al.  “Multiple object tracking: A literature review”  In _Artificial intelligence_ 293 Elsevier, 2021, pp. 103448
  * [20] Renrui Zhang et al.  “Personalize segment anything model with one shot”  In _arXiv preprint arXiv:2305.03048_ , 2023
  * [21] Jinyu Yang et al.  “Track anything: Segment anything meets videos”  In _arXiv preprint arXiv:2304.11968_ , 2023
  * [22] San Jiang, Cheng Jiang and Wanshou Jiang  “Efficient structure from motion for large-scale UAV images: A review and a comparison of SfM tools”  In _ISPRS Journal of Photogrammetry and Remote Sensing_ 167 Elsevier, 2020, pp. 230–251
  * [23] Giovanni Pintore et al.  “State-of-the-art in automatic 3D reconstruction of structured indoor environments”  In _Computer Graphics Forum_ 39.2, 2020, pp. 667–699  Wiley Online Library
  * [24] Jorge J Moré  “The Levenberg-Marquardt algorithm: implementation and theory”  In _Numerical analysis: proceedings of the biennial Conference held at Dundee, June 28–July 1, 1977_ , 2006, pp. 105–116  Springer
  * [25] Xingkui Wei et al.  “Deepsfm: Structure from motion via deep bundle adjustment”  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16_ , 2020, pp. 230–247  Springer
  * [26] Jiawei He et al.  “Tracking Objects with 3D Representation from Videos”  In _arXiv preprint arXiv:2306.05416_ , 2023
  * [27] Lorenzo Porzi et al.  “Learning multi-object tracking and segmentation from automatic annotations”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 6846–6855
  * [28] Dan Cernea  “OpenMVS: Multi-view stereo reconstruction library”  In _City_ 5.7, 2020
  * [29] Johannes Lutz Schönberger and Jan-Michael Frahm  “Structure-from-Motion Revisited”  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016
  * [30] Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys and Jan-Michael Frahm  “Pixelwise View Selection for Unstructured Multi-View Stereo”  In _European Conference on Computer Vision (ECCV)_ , 2016
  * [31] G. Bradski  “The OpenCV Library”  In _Dr. Dobb’s Journal of Software Tools_ , 2000
  * [32] Jean Serra  “Image analysis and mathematical morphol-ogy”  In _(No Title)_ Academic press, 1982


# Unsupervised and Interpretable Domain Adaptation to Rapidly Filter Tweets
for Emergency Services

Jitin Krishnan  Department of Computer Science
George Mason University
Fairfax, VA, USA
<EMAIL_ADDRESS>Hemant Purohit  Department of Information
Sciences & Technology
George Mason University
Fairfax, VA, USA
<EMAIL_ADDRESS>Huzefa Rangwala  Department of Computer Science
George Mason University
Fairfax, VA, USA
<EMAIL_ADDRESS>
###### Abstract

During the onset of a natural or man-made crisis event, public often share
relevant information for emergency services on social web platforms such as
Twitter. However, filtering such relevant data in real-time at scale using
social media mining is challenging due to the short noisy text, sparse
availability of relevant data, and also, practical limitations in collecting
large labeled data during an ongoing event. In this paper, we hypothesize that
unsupervised domain adaptation through multi-task learning can be a useful
framework to leverage data from past crisis events for training efficient
information filtering models during the sudden onset of a new crisis. We
present a novel method to classify relevant social posts during an ongoing
crisis without seeing any new data from this event (fully unsupervised domain
adaptation). Specifically, we construct a customized multi-task architecture
with a multi-domain discriminator for crisis analytics: multi-task domain
adversarial attention network (MT-DAAN). This model consists of dedicated
attention layers for each task to provide model interpretability; critical for
real-word applications. As deep networks struggle with sparse datasets, we
show that this can be improved by sharing a base layer for multi-task learning
and domain adversarial training. The framework is validated with the public
datasets of TREC incident streams that provide labeled Twitter posts (tweets)
with relevant classes (Priority, Factoid, Sentiment) across 10 different
crisis events such as floods and earthquakes. Evaluation of domain adaptation
for crisis events is performed by choosing one target event as the test set
and training on the rest. Our results show that the multi-task model
outperformed its single-task counterpart. For the qualitative evaluation of
interpretability, we show that the attention layer can be used as a guide to
explain the model predictions and empower emergency services for exploring
accountability of the model, by showcasing the words in a tweet that are
deemed important in the classification process. Finally, we show a practical
implication of our work by providing a use-case for the COVID-19 pandemic.

###### Index Terms:

Social Media, Crisis Analytics, Text Classification, Unsupervised Domain
Adaptation, Interpretability

Figure 1: Problem Statement: Interpretably predict labels for tweets collected
during an ongoing crisis using only the past crisis data, given a)
unavailability of labeled data in the ongoing event, and b) need for
interpretability of machine reasoning behind data filtering for emergency
managers.

##  I Introduction

During the sudden onset of a crisis situation, social media platforms such as
Twitter provide valuable information to aid crisis response organizations in
gaining real-time situational awareness [1]. Effective analysis of important
information such as affected individuals, infrastructure damage, medical
emergencies, or food and shelter needs can help emergency responders make
time-critical decisions and allocate resources in the most effective manner
[2, 3, 4].

Several machine learning systems have been deployed to help towards this
humanitarian goal of converting real-time social media streams into actionable
knowledge. Classification being the most common task, researchers have
designed models [5, 6, 7, 3] that classify tweets into various crisis-
dependent categories such as priority, affected individuals, type of damage,
type of assistance needed, usefulness of the tweet, etc. Social media streams
contain short, informal, and abbreviated content; with potential linguistic
errors and sometimes contextually ambiguous. These inherently challenging
properties of tweets make their classification task and formulation less
trivial when compared to traditional text classification tasks.

In this paper, we address two practically important and underdeveloped aspects
of current research in social media mining for crisis analytics to classify
relevant social web posts: a) a fully unsupervised domain adaptation, and b)
interpretability of predictions. A fully unsupervised domain adaptation uses
no data from the ongoing crisis to train the model. Nguyen et al., 2016 [5]
showed that their convolutional neural network (CNN) model does not require
feature engineering and performed better than the state-of-the-art methods;
one of their models being completely unsupervised [5]. Similarly, Alam et al.,
2018 [6] designed a CNN architecture with adversarial training on graph
embeddings, but utilizing unlabeled target data. Our goal is to construct an
unsupervised model that does not require any unlabeled target data with the
capability of being interpretable. We specifically address the problem of data
sparsity and limited labels by designing a multi-task classification model
with domain adversarial training; which, to the best of our knowledge, is not
explored in social media mining for crisis analytics. Another crucial
component of our model is interpretability. In prior works, when a top
performing model produces an accuracy of $78\%$, for instance, it is unclear
how trustworthy it is and what features are deemed important in the model’s
decision-making process. An interpretable model like ours can present with a
convincing evidence of which words the classifier deems important when making
a certain prediction, and helps ensure reliability for domain users, e.g.,
emergency managers.

Contributions: a) To address the problems of data sparsity and limited labels,
we construct a customized multi-task learning architecture (MT-DAAN) to filter
tweets for crisis analytics by training four different classification tasks
(c.f. examples in Fig. 3) across ten different crisis events under domain
shift. This multi-task domain adversarial model consists of dedicated
attention layers for each task for interpretability and a domain classifier
branch to promote the model to be domain-agnostic. b) We demonstrate that the
attention layers provide interpretability for the predictions made by the
classifiers; with the goal to aid emergency services in a more meaningful way.
c) We empirically validate the performance of the underlying single-task
attention-based neural network architecture by comparing it to the state-of-
the-art methods, for improving generalizability and interpretability for
domain adaptation in unsupervised tweet classification tasks in general. d)
Additionally, through experiments, we show that deep networks struggle with
small datasets, and that this can be improved by sharing the base layer for
multi-task learning and domain adversarial training.

##  II Related Work and Background

###  II-A Domain Adaptation

Domain Adaptation in text classification tasks has a long line of fruitful
research [8, 9, 10] that try to minimize the difference between the domains so
that a model trained solely on one domain is generalizable to unseen test data
from a completely different domain. With the introduction of Domain-
Adversarial training of Neural Networks (DANN) [11], many state-of-the-art
models now utilize unlabeled target data to train classifiers that are
indiscriminate toward different domains. The speciality of this architecture
is that it consists of an extra branch, which performs domain classification
using unlabeled data from different domains. Thus, both task and domain
classifiers share some bottom layers but have separate layers towards the top.
A negative gradient (gradient reversal) from the domain classifier branch is
back-propagated to promote the features at the lower layers of the network
incapable of discriminating the domains. Recent works such as Adversarial
Memory Network (AMN) [12] utilizes attention, in addition to DANN, to bring
interpretability to capture the pivots for sentiment classification.
Hierarchical Attention Network (HATN) [13] expands upon AMN by first
extracting pivots and then jointly training networks for both pivots and non-
pivots.

For filtering social web data for crisis analytics, these models do not
suffice and need customized expansions due to the following reasons: a)
Collecting and using large unlabeled target data from the new/ongoing crisis
event may not be practically viable, thus, we aim for a fully unsupervised
modeling. b) Having access to unlabeled data from multiple crisis events can
alleviate the above problem to an extent by using it to train the domain
classifier branch to push the model to be domain independent. c) Due to the
low-resource nature of the dataset, binary classifiers may miss important
lower level features that can be potentially improved by a multi-task model
that shares the lower layers of the network for all the tasks. This is also
evident from our results in Table III and IV, which show that deep models that
perform much better than simple models on Amazon reviews do not significantly
outperform them on TREC tweet dataset for crises.

###  II-B Multi-Task Learning

Multi-Task Learning (MTL) solves multiple tasks at the same time with a goal
to improve the overall generalization capability of the model [14]. Within the
context of Deep Learning, MTL is performed by sharing (or constraining) lower
level layers and using dedicated upper level layers for various tasks. A rich
overview of MTL in Deep Neural Networks is presented by Ruder (2017) [15]. MTL
has been a successful strategy over the past few years for many research
explorations such as relationship networks [16] in computer vision and Sluice
networks [17] in natural language processing. Similar problems in domain
adaptation of semantic classification and information retrieval were addressed
by jointly learning to leverage large amounts of cross-task data [18]. In low
resource datasets such as for crises, the chance of overfitting is very high.
Thus, it seems intuitively better for the model to find a shared
representation capturing different tasks and not just one, such that feature
commonalities across tasks can be exploited.

###  II-C Attention Mechanism

Attention mechanism [19], originally designed for machine translation
problems, has become one of the most successful and widely used methods in
deep learning that can look at a part of a sentence at a time like humans.
This is particularly useful because of its ability to construct a context
vector by weighing on the entire input sequence unlike previous sequence-to-
sequence models [20] that used only the last hidden state of the encoder
network (typically BiLSTM [21], LSTM [22], or GRU [23]). For example, in a
sentence, the context vector is a dot product of the word activations and
weights associated with each word; thus leading to an improved contextual
memorization, especially for long sentences. Our method incorporates such
attention mechanisms to enhance interpretability of the classifier.

##  III Methodology

###  III-A Problem Statement: Unsupervised Domain Adaptation for Crisis Tweet
Classification

Using notations in Table I, consider a set $C$ of all crisis events such as
Guatemala Earthquake or Typhoon Yolanda. The task of unsupervised domain
adaptation for crisis analytics is to train a classifier for a specific target
crisis ($c_{t}$) using labeled ($L_{C-c_{t}}$) and unlabeled ($U_{C-c_{t}}$)
data from all other crises; where $C-c_{t}$ denotes the set of all crisis
events minus the target crisis. We assume that no data record from the target
crisis is available for training. Following the traditional domain adaptation
terminology, $X_{s}$ = $L_{C-c_{t}}$ represents the labeled data from the
source domain $S$ and $Y_{s}$ = $y_{C-c_{t}}$ represents the ground truth
labels on which the classifier is trained. And, $X_{t}$ = $L_{c_{t}}$
represents the labeled data from the target domain $T$ and $Y_{t}$ =
$y_{c_{t}}$ represents the ground truth labels; both of which are only used
for testing the classifier. $X_{d}$ = $U_{C-c_{t}}$ represents the unlabeled
data from different domains minus the target. To summarize:
Input: $X_{s}$, $Y_{s}$, $X_{d}$
Output: $Y_{t}^{pred}$ $\leftarrow$ $predict(X_{t})$

Notation |  Definition
---|---
$C$ |  Set of all crisis events $\\{c_{1},c_{2},...\\}$
$L_{c_{k}}$ |  Set of labeled data from the event $c_{k}$
$y_{c_{k}}$ |  Set of ground truth labels for $L_{c_{k}}$.
$m$ |  Number of tasks (Number of bits in each label)
$U_{c_{k}}$ |  Set of unlabeled data from the event $c_{k}$
$T_{x}$ |  Number of words in a sentence
$x^{<k>}$ |  $k$-th word of a sentence
$\alpha^{<k>}$ |  attention from $k$-th word
$a^{<k>}$ |  BiLSTM activation from $k$-th word
TABLE I: Notations

###  III-B Overview

In the following sections, we describe three models: Single-Task Attention
Network (ST), Single-Task Domain Adversarial Attention Network (ST-DAAN), and
Multi-Task Domain Adversarial Attention Network (MT-DAAN). ST is the model we
adopt from [24] to build the single-task attention based baseline. ST-DAAN is
constructed on top of ST to make the model domain agnostic by performing
adversarial training using gradient reversal. Finally, MT-DAAN is constructed
on top of ST-DAAN with dedicated attention layers for each task on a shared
BiLSTM layer. This is shown in Figure 2.

Figure 2: Fully Unsupervised Domain Adaptation Set-up for Multi-Task Crisis
Tweet Classification.

###  III-C Single-Task Attention Network (ST)

We first describe the single-task attention network [24] on top of which we
build our models. This model aligns with our goals of interpretability and
unsupervised domain adaptation. This BiLSTM based model with Attention gives
us three main advantages:

  1. 1.

Unlike several existing domain adaptation methods that use unlabeled target
data to train the domain adversarial component via gradient reversal, this
method is a fully unsupervised baseline which also can be customized for
multi-task learning.

  2. 2.

The method uses attention mechanism which in turn weighs each word in a
sentence based on its importance. This can be directly utilized for
interpretability.

  3. 3.

The method also runs much faster (only a few minutes), i.e. highly useful in
crisis times, as compared to the top performing semi-supervised models such as
HATN [13] (hours).

This model [24] consists of a BiLSTM layer which produces $T_{x}$ activations,
each corresponding to a word in the sentence. These activations are passed
through dense and softmax layers and are combined by dot product to produce
the context vector $\sum_{k=1}^{T_{x}}\alpha^{<k>}a^{<k>}$, where $a^{<k>}$ is
the BiLSTM activation from $k$-th word and $\alpha^{<k>}$ is the attention
weight of $k$-th word. Sentences with words greater than $T_{x}$ are stripped
and those with words lower than $T_{x}$ are padded. This single-task ($m=1$)
attention network is the building block with which rest of the following
models are constructed. The single-task binary cross entropy loss function is
shown below.

$\footnotesize
L_{T}=-\frac{1}{N}\sum_{i=1}^{N}[y_{i}\log\hat{y_{i}}+(1-y_{i})\log(1-\hat{y_{i}})]$
(1)

where $T$ represents the task, $y$ is the true label, and $\hat{y}$ is the
predicted label.

###  III-D Single-Task Domain Adversarial Attention Network
(ST-DAAN)

To study the specific contribution of domain adversarial training, we
construct a secondary baseline over the ST architecture by constructing an
additional branch with gradient reversal layer which is represented by the
green blocks in Figure 2. This is a single-task binary classifier with $m=1$.
Domain Adversarial Training of Neural Networks (DANN) [11] was introduced with
a goal to confuse the classifier by back-propagating a negative gradient from
a separate domain classifier branch (right-most branch, as shown in Figure 2).
This makes the classifier agnostic to difference in domains. This back-
propagation is implemented using a gradient reversal layer [11] which does
nothing during the forward pass but pushes a negative gradient
($-\lambda\frac{\partial L_{d}}{\partial\theta_{f}}$) during the backward
(gradient update) pass. $L_{d}$ is the domain classification loss, $\lambda$
is the strength of the reversal, and $f$ represents the lower level layers or
features over which the negative gradient update is performed. In our
architecture, the goal is to make the BiLSTM layer indiscriminate towards
various crisis domains such that the multi-task classification does not depend
on the domain from which the tweet/sentence is coming from. The ST-DAAN loss
function is shown below.

$\footnotesize L_{T}^{\prime}=L_{T}+w_{d}L_{d}$ (2)

where $w_{d}$ is the domain adversarial loss weight. $L_{d}$ represents the
categorical cross entropy loss for multi-domain discriminator shown below.

$\footnotesize
L_{d}=-\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{|C-c_{t}|}[y_{ij}\log\hat{y_{ij}}]$
(3)

where $C-c_{t}$ is the set of all crisis events without the target event.

###  III-E Multi-Task Domain Adversarial Attention Network
(MT-DAAN)

Building on top of ST-DAAN, we construct MT-DAAN, which is intended to
classify problems with multiple tasks or labels. For each task, a dedicated
attention layer is allocated from which it predicts binary labels. The BiLSTM
layer remains exactly the same as in the single-task model but multiple
attention blocks are added for each task along with a domain classifier. In
the architecture decision process, we first investigated a multi-label
classifier where all layers are shared with the final softmax layer making
multi-label predictions. In low resource settings, constructing a multi-label
classifier using a shared architecture is challenging for two reasons: a)
jointly balancing positive and negative samples across all classes is not
trivial and potentially challenging to make it extensible when new classes
need to be added, and b) attention layer may not always produce class-specific
insights as the weights are assigned to train for the combination of labels.
On the other hand, in the multi-task architecture with separate attention
layers, it is easy to add more classes. If some classes require more training,
it is trivial to further tune a model specific to that class. More
importantly, $context^{<t_{j}>}$ vector for $j$-th task identifies the
influential words from each sentence for that specific task. The complete
architecture is shown in Figure 2. MT-DAAN loss function is shown below:

$\footnotesize L_{MT-DAAN}=\sum_{k=1}^{m}(w_{k}L_{T_{k}})+w_{d}L_{d}$ (4)

where $m$ is the number of tasks, $w_{k}$ is the loss weight and $L_{T_{k}}$
is the loss term for each task, $w_{d}$ is the domain adversarial loss weight,
and $L_{d}$ is the domain adversarial loss term.

###  III-F Model Interpretability

The output ($\alpha$) of the attention layer ($ATT$) of each task, is a
$T_{x}$-dimensional vector; $T_{x}$ being the number of words in the sentence.
The context vector ($\sum_{k=1}^{T_{x}}\alpha^{<k>}a^{<k>}$) is the product of
these attention weights and the $T_{x}$-dimensional activation ($a$) from the
$BiLSTM$ layer. $\alpha$ essentially weighs how much each word in the sentence
contributes to the classification result. Thus, $\alpha$ is the component that
is evaluated for model interpretability.

##  IV DATASETS

CRISIS EVENTS |  Total Tweets |  Vocab |  Avg #words |  P |  F |  S |  I
---|---|---|---|---|---|---|---
2012 Guatemala Earthquake |  154 |  422 |  18.74 |  104 |  108 |  12 |  15
2013 Typhoon Yolanda |  564 |  1746 |  19.47 |  249 |  46 |  119 |  51
2013 Australia Bushfire |  677 |  2102 |  20.21 |  152 |  213 |  167 |  36
2013 Boston Bombings |  535 |  1755 |  19.30 |  147 |  28 |  234 |  198
2013 Queensland Floods |  713 |  2301 |  19.08 |  293 |  54 |  173 |  215
2014 Chile Earthquake |  311 |  919 |  16.54 |  48 |  26 |  50 |  10
2014 Typhoon Hagupit |  1470 |  2893 |  15.36 |  469 |  375 |  276 |  101
2015 Nepal Earthquake |  2048 |  4026 |  13.77 |  1067 |  377 |  741 |  133
2015 Paris Attacks |  2066 |  4152 |  18.62 |  306 |  183 |  782 |  429
2018 Florida School Shooting |  1118 |  2940 |  21.40 |  329 |  64 |  206 |  70
TABLE II: TREC Dataset Statistics; Showing the number of positive samples for
each of the 4 classes. $P$=Priority, $F$=Factoid, $S$=Sentiment, and
$I$=Irrelevant.

###  IV-A TREC Dataset

TREC-IS111http://dcs.gla.ac.uk/~richardm/TREC_IS/ (Text Retrieval Conference -
Incident Streams) is a program that encourages research in information
retrieval from social media posts with the goal to improve the state-of-the-
art social media based crisis analytics solutions. We use the dataset from
2018 track proposal. Statistics of this curated dataset of Twitter downloaded
from TREC is shown in Table II. The original dataset consisted of 15 crisis
events. However, due to very low data, we trimmed the events and tasks such
that there are at least 10 positive samples for each task.

The four tasks used in our experiments are shown below:

  1. 1.

Priority: Different priority levels are assigned for each tweet: low, medium,
high, critical. We convert this into a binary classification problem where
$low=0$ and $\\{medium$, $high$, $critical\\}=1$.

  2. 2.

Factoid: ‘Factoid’ is a categorical label that represents if a tweet is
stating a fact. Eg: ‘death toll rises ...’

  3. 3.

Sentiment: ‘Sentiment’ is a categorical label that represents if a tweet
represents a sentiment. Eg: ’Worried.. Thoughts and prayers.’

  4. 4.

Irrelevant: ‘Irrelevant’ is a categorical label for tweets that do not provide
any relevant information.

###  IV-B Amazon Reviews Dataset

The standard benchmark
dataset222http://www.cs.jhu.edu/~mdredze/datasets/sentiment/ of Amazon reviews
[25] is widely used for cross-domain sentiment analysis. We chose four
domains: Books (B), Kitchen (K), DVD (D), and Electronics (E). The raw
data333https://github.com/hsqmlzno1/HATN/tree/master/raw_data, a part of
Blitzer’s original raw dataset, used in this work is from HATN [13]. This
dataset consists of $3000$ positive and $3000$ negative samples for each of
the $4$ domains. This dataset is used for two purposes: 1) to validate the
performance of the state-of-the-art methods including the single-task baseline
and 2) to compare and contrast the performance of deep models when trained
with rich versus sparse datasets.

###  IV-C COVID-19 Tweet Dataset

For the COVID-19 use-case, we use Twitter posts collected using CitizenHelper
[26] system in March 2020, for the geo-bounding box of the Washington D.C.
Metro region. These tweets were annotated by volunteers of regional Community
Emergency Response Teams (CERTs), with ‘Relevant’ label denoting how relevant
a tweet is for crisis response operations. The label values range on a scale
of $1$-$4$. We convert them into binary classes by considering values $1$ and
$2$ as $-$ve ($0$) class and values $3$ and $4$ as $+$ve ($1$) class. This
dataset consists of $4911$ tweets with $-$ve ($Relevant$=$0$) and $637$ tweets
with $+$ve ($Relevant$=$1$) classes. Following unsupervised domain adaptation
criteria, the filtering models are trained using only the TREC dataset and
evaluated on the COVID-19 tweets. For each independent run of the experiment,
a balanced subset of size $637$ for both classes is selected for testing.

S $\rightarrow$ T |  LR |  SVM |  CNN |  BiLSTM |  AMN |  HATN |  ST
---|---|---|---|---|---|---|---
B $\rightarrow$ K |  76.40 |  75.95 |  81.20 |  84.45 |  81.88 |  87.03 |  87.22
B $\rightarrow$ E |  75.53 |  74.05 |  80.44 |  84.61 |  80.55 |  85.75 |  85.51
B $\rightarrow$ D |  81.08 |  81.43 |  82.94 |  83.52 |  85.62 |  87.07 |  86.32
K $\rightarrow$ B |  76.12 |  75.78 |  78.78 |  80.67 |  79.05 |  84.88 |  81.85
K $\rightarrow$ E |  80.37 |  81.20 |  85.17 |  87.37 |  86.68 |  89.00 |  87.09
K $\rightarrow$ D |  73.32 |  74.98 |  76.41 |  78.49 |  79.50 |  84.72 |  81.13
E $\rightarrow$ B |  74.85 |  74.18 |  78.08 |  81.18 |  77.52 |  84.03 |  81.50
E $\rightarrow$ K |  81.85 |  81.85 |  86.59 |  89.00 |  87.83 |  90.08 |  89.21
E $\rightarrow$ D |  75.82 |  75.83 |  78.35 |  78.46 |  85.03 |  84.32 |  81.37
D $\rightarrow$ B |  81.17 |  82.20 |  82.26 |  84.83 |  84.53 |  87.78 |  87.02
D $\rightarrow$ K |  76.42 |  77.58 |  81.09 |  85.21 |  81.67 |  87.47 |  86.37
D $\rightarrow$ E |  72.47 |  73.68 |  79.56 |  83.66 |  80.42 |  86.32 |  85.63
AVG |  77.12 |  77.39 |  80.91 |  83.45 |  82.52 |  86.54 |  85.02
TABLE III: Performance comparison (accuracy) of various models on the standard benchmark dataset of amazon reviews. Methods in  blue do not use any unlabeled target data; hence relevant in our context. Each reported score is an average of 10 independent runs of each experiment. Target | LR | SVM | CNN | BiLSTM |  ST
---|---|---|---|---|---
Guatemala Earthquake | 60.14 | 56.76 | 60.47 | 65.54 |  59.97
Typhoon Yolanda | 65.39 | 65.97 | 63.05 | 65.49 |  65.53
Australia Bushfire | 65.61 | 63.23 | 62.10 | 60.10 |  62.44
Boston Bombings | 71.47 | 75.45 | 69.72 | 71.43 |  72.08
Queensland Floods | 65.56 | 64.81 | 64.13 | 66.01 |  66.21
Chile Earthquake | 43.09 | 37.94 | 43.37 | 35.45 |  39.23
Typhoon Hagupit | 49.86 | 46.22 | 49.21 | 54.13 |  52.61
Nepal Earthquake | 57.11 | 55.39 | 58.61 | 60.49 |  61.35
Paris Attacks | 71.43 | 71.72 | 72.50 | 72.14 |  71.31
Florida School Shooting | 58.79 | 63.02 | 58.82 | 59.71 |  60.55
AVG | 60.85 | 60.05 | 60.20 | 61.05 |  61.13
TABLE IV: Performance comparison (accuracy) of unsupervised models on TREC-
Priority (tweet) dataset showing that deep models are not strictly superior
than simpler models due to data sparsity. Each reported score is an average of
10 independent runs of each experiment. $Source$ = $Everything$ \- $Target$.

TARGET | Priority | Factoid
---|---|---
| ST | ST-DAAN | MT-DAAN | ST | ST-DAAN | MT-DAAN
| Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1
Guatemala Earthquake | 59.97 | 62.39 | 69.07 | 69.66 | 69.05 | 69.34 | 68.92 | 68.47 | 79.90 | 80.76 | 84.05 | 97.01
Typhoon Yolanda | 65.53 | 65.47 | 66.07 | 63.73 | 67.42 | 67.30 | 80.50 | 84.42 | 82.71 | 85.61 | 84.36 | 86.93
Australia Bushfire | 62.44 | 66.69 | 61.07 | 63.42 | 61.93 | 64.28 | 64.58 | 60.69 | 65.64 | 60.53 | 65.04 | 60.13
Boston Bombings | 72.08 | 74.29 | 72.34 | 73.37 | 73.80 | 74.74 | 83.10 | 88.51 | 81.42 | 85.90 | 85.82 | 88.82
Queensland Floods | 66.21 | 65.94 | 67.19 | 66.97 | 66.74 | 66.46 | 37.56 | 48.90 | 50.46 | 59.82 | 49.52 | 59.21
Chile Earthquake | 39.23 | 40.92 | 38.91 | 42.37 | 41.80 | 46.33 | 30.38 | 33.97 | 39.87 | 48.68 | 45.28 | 54.58
Typhoon Hagupit | 52.61 | 50.59 | 58.97 | 58.94 | 57.50 | 57.52 | 68.98 | 70.79 | 71.42 | 72.44 | 69.49 | 70.08
Nepal Earthquake | 61.35 | 59.44 | 60.18 | 57.80 | 61.65 | 59.49 | 74.04 | 76.08 | 80.72 | 81.00 | 81.04 | 81.02
Paris Attacks | 71.31 | 76.26 | 70.42 | 74.08 | 74.44 | 77.21 | 75.78 | 80.35 | 82.35 | 84.89 | 82.52 | 85.63
Florida School Shooting | 60.55 | 61.75 | 65.47 | 64.07 | 62.51 | 63.24 | 76.73 | 82.67 | 84.55 | 87.51 | 85.80 | 88.15
AVG | 61.13 | 62.37 | 62.97 | 63.44 | 63.68 | 64.59 | 66.06 | 69.49 | 71.90 | 74.71 | 73.29 | 77.16

TARGET | Sentiment | Irrelevant
---|---|---
| ST | ST-DAAN | MT-DAAN | ST | ST-DAAN | MT-DAAN
| Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1
Guatemala Earthquake | 96.96 | 97.03 | 96.45 | 96.68 | 96.76 | 92.73 | 89.36 | 89.03 | 91.22 | 91.06 | 93.11 | 92.73
Typhoon Yolanda | 75.81 | 77.62 | 77.54 | 79.01 | 76.82 | 78.35 | 76.05 | 79.77 | 78.49 | 80.59 | 80.46 | 82.31
Australia Bushfire | 75.95 | 77.58 | 78.80 | 79.12 | 78.54 | 78.92 | 35.42 | 47.164 | 53.78 | 65.11 | 51.76 | 63.36
Boston Bombings | 81.39 | 81.11 | 80.73 | 80.70 | 82.13 | 82.10 | 58.15 | 55.73 | 58.15 | 57.43 | 61.49 | 61.45
Queensland Floods | 81.69 | 80.39 | 81.05 | 81.39 | 81.53 | 81.32 | 65.68 | 65.36 | 67.26 | 65.72 | 67.88 | 67.27
Chile Earthquake | 92.69 | 92.91 | 93.10 | 93.21 | 93.62 | 93.68 | 75.16 | 84.98 | 80.46 | 86.38 | 80.64 | 86.56
Typhoon Hagupit | 84.98 | 85.86 | 85.15 | 86.14 | 85.43 | 86.38 | 63.21 | 75.04 | 71.50 | 78.25 | 70.22 | 77.27
Nepal Earthquake | 67.75 | 68.42 | 70.20 | 70.51 | 69.96 | 70.31 | 31.79 | 42.10 | 36.97 | 47.41 | 41.49 | 52.87
Paris Attacks | 76.01 | 76.63 | 73.65 | 73.98 | 74.47 | 74.60 | 33.91 | 35.25 | 44.52 | 48.32 | 47.17 | 51.32
Florida School Shooting | 68.77 | 71.77 | 67.06 | 70.03 | 68.14 | 71.05 | 32.66 | 40.90 | 44.22 | 55.27 | 47.64 | 58.65
AVG | 80.20 | 80.93 | 80.37 | 81.08 | 80.74 | 80.94 | 56.14 | 61.53 | 62.66 | 67.55 | 64.19 | 69.38

TABLE V: Unsupervised domain adaptation results on TREC dataset showing performance boost for Priority, Factoid, and Irrelevant tasks. However, Sentiment task did not show a significant improvement. See performance evaluation section for details. Each reported score is an average of 10 independent runs of each experiment. TARGET | Relevant
---|---
| ST | ST-DAAN | MT-DAAN
| Acc | F1 | Acc | F1 | Acc | F1
COVID-19 | 73.25 | 77.36 | 74.55 | 77.51 | 77.00 | 78.09
TABLE VI: Unsupervised domain adaptation results for COVID-19 tweets using
only the TREC dataset for training. Each reported score is an average of 10
independent runs of each experiment.

##  V Results & Discussion

We first validate the performance of the adopted unsupervised ST model [24] by
comparing it with the following standard neural network architectures and
state-of-the-art models used for domain adaption in text. We use the standard
benchmark dataset of Amazon reviews. Following the traditional domain
adaptation experimental setup, each experiment represented as S $\rightarrow$
T consists of a source domain (S) on which the model is trained and a target
domain (T) on which the model is tested. We use Keras deep learning library
for our implementations; with $T_{x}$=$200$ for Amazon reviews and $30$ for
Tweets. We use Adam optimizer with a dropout of $0.4$, maximum epoch of $50$,
early stopping patience of $3$, batch size of $32$, and validation split of
$0.15$.

  1. 1.

Simple Baselines: We construct simple baseline classifiers [27]: Logistic
Regression (LR) and Support Vector Machines (SVM). The input to these models
are constructed by aggregating the $300$-dimensional word embeddings of words
in each review.

  2. 2.

CNN: A standard Convolutional Neural Network inspired by Kim, 2014 [28] is
constructed with the following architecture:
$Word\ Embeddings(T_{x},300)\rightarrow Conv1D(128,5)$
$\rightarrow MaxPooling1D(5)$ $\rightarrow Conv1D(128,5)\\\ \rightarrow
MaxPooling1D(5)$ $\rightarrow Conv1D(128,5)\\\ \rightarrow
GlobalMaxPooling1D()\rightarrow Dense(128)$
$\rightarrow Dense(2)\rightarrow y$.
This is combined with dropouts, relu activations, and ending with softmax
activation producing labels for binary classification. State-of-the-art deep
learning methods for existing social media mining approaches of crisis
analytics [6, 5] use a similar architecture.

  3. 3.

BiLSTM: This is the bottom-most layer in Figure 2 with the activation
$a^{<T_{x}>}$ passed through the following: $Dense(10)\rightarrow
Dense(2)\rightarrow y$ also including dropouts, relu activation, and ending
with softmax.

  4. 4.

AMN and HATN: AMN [12] and HATN [13] are attention-based methods which use
gradient reversal to perform domain adversarial training on the unlabeled data
from source and target domains. HATN is an extension to AMN by adding the
hierarchical component and jointly training pivot and non-pivot networks.

Input to all the models are word
vectors444https://code.google.com/archive/p/word2vec/ [29]. The evaluation on
amazon reviews shows how well the single-task (ST) model perform when compared
to the existing top-performing domain adaptation models on benchmark dataset.
Table III shows accuracy scores on the Amazon cross-domain sentiment analysis
dataset. HATN uses unlabeled target data, gradient reversal, explicit pivot
extraction, and joint training making it a computationally expensive method.
As shown in the experimental evaluation, we use the same Amazon dataset and
GoogleNews word vectors for our experiments. ST, being unsupervised with no
need of unlabeled target data, performed competitively with an overall
accuracy of 85.02%; thus establishing a strong fully unsupervised building
block for us to build upon.

###  V-A Crisis Tweets vs Amazon Reviews

Table III and IV show that deep models struggle with small datasets such as
TREC-IS tweets. When ST model outperformed Logistic Regression by $\sim 8\%$
on the Amazon reviews dataset, the difference was only less than $1\%$ with no
statistical significance on the TREC-Priority dataset. Note that we conduct
experiments with various parameter combinations on the deep models when using
tweets. For example, $T_{x}=200$ for amazon reviews and $T_{x}=30$ for tweets
due to the difference in their average word-length. Books domain of Amazon
reviews has $182$ average number of tokens per review with a vocab size of
$105920$. On the other hand, the event with highest number of tweets in the
TREC dataset (Paris Attacks) has only 18.62 average number of tokens per tweet
with a vocab size of $4152$. This difference makes it intuitively challenging
to train deep models with several parameters that may lead the model to
memorize the entire dataset resulting in poor generalization. Multi-task
learning and domain adversarial training try to alleviate this problem by
training the shared BiLSTM layer with much more data from different tasks and
unlabeled data.

###  V-B MT-DAAN Performance Evaluation

The primary purpose of the MT-DAAN model is to show that sharing the bottom
layer of the model (i.e., shared representation) for different tasks along
with domain adversarial training can help improve the generalizability of some
of the tasks that are otherwise trained alone in the single-task model. The
experiments for MT-DAAN are setup in the same unsupervised way as for single-
task. No data from the test crisis is used for training. For example, if we
are testing our model for the event ‘Typhoon Yolanda’, no data from this
crisis is used for training. Note that the domain classifier component uses
unlabeled data only from rest of the crisis; making it a fully unsupervised
domain adaptation approach. Performance scores of the four tasks (Priority,
Factoid, Sentiment, and Irrelevant) are shown in Table V. The results show
clear performance improvement for Priority, Factoid, and Irrelevant tasks.
However, Sentiment task did not show significant improvement. We speculate
that this is because other tasks do not generalize the bottom layer enough to
boost the sentiment classification performance. These results show the
usefulness of multi-task learning as well as domain adversarial training where
different tasks in multiple domains help each other when the data is sparse
and labels are limited.

Figure 3: Examples of interpretable results using attention; darker the shade,
higher the attention. Recall that no data from the crisis-event for testing is
used for training the model. Even then, relevant keywords such as ‘police
urging’, ‘death toll rises’, ‘worried’, and ‘thoughts with people’ are
correctly picked up by the attention layers of their respective tasks.

###  V-C Word Vectors

We use fastText[30] as our word embeddings for tweets because of its sub-word
usage and the ability to create vectors for arbitrary and out-of-vocabulary
words. Although there exists many alternatives, picking the one that works
well for a specific dataset is not trivial. We conducted experiments using
four choices of word embeddings: fastText [30], GoogleNews [29], Glove [31],
and CrisisNLP [32]. Averaging over 10 crises, we received the following
accuracy scores (in %) respectively for the above word embeddings: {$80.20$,
$81.82$, $81.88$, $80.73$}. Unlike fastText, we fine-tune these pre-trained
vectors to create vectors for out-of-vocabulary words. Vectors for words that
are already in the vocabulary are locked while tuning for consistency in
evaluation. The tweet-based embeddings such as Glove or CrisisNLP did not
significantly outperform other models. Glove vectors are 200-dimensional while
the rest are 300-dimensional which makes the experiment favoring Glove word
vectors. This experiment shows that the problem of finding a strictly superior
word vector model for tweets still remains a challenging task.

Figure 4: Examples of interpretable results using attention for relevancy
prediction of COVID-19 tweets. With $77\%$ accuracy, although the highly
attended words in the ‘Relevant’ tweets provide some intuitive sense of
interpretability, the highlighted words in the ‘Irrelevant’ tweets are
somewhat ambiguous because it is unclear if those words are chosen due to
their specific or generic nature. This shows both the benefits and challenges
of unsupervised and interpretable domain adaptation.

###  V-D Interpretability: Attention Visualization

The attention weights used to create the context vector by the dot product
operation with word activations represent the interpretable layer in our
architecture. These weights represent the importance of each word in the
classification process. Some examples are shown in Figures 3 and 4. Stronger
the color intensity stronger the word attention. In the first example, ‘boston
police urging’ is the reason why the tweet is classified as $+$ve priority.
Similarly, ‘death toll rises’ in the Factoid example, ‘worried, prayers’ in
the Sentiment example, and ‘thoughts with people’ in the Irrelevant example
are clear intuitive indicators of +ve predictions. These examples show the
importance of having interpretability as a key criterion in crisis domain
adaptation tasks for social media.

To the best of our knowledge, in social media mining for crisis analytics,
there does not exist a ground truth dataset that highlights the words that
explain the labels for tweets. Using our model as a guide, we hope to build a
robust evaluation dataset as our immediate next step so that the models can be
quantitatively evaluated using robust trust-evaluation methods such as LIME
[33]. It is also crucial to note that binary classification tasks such as
sentiment analysis of Amazon reviews has a clear class divide that produces
intuitive keywords such as ‘good’, ‘excellent’, or ‘great’ for $+$ve reviews
and ‘bad’, ‘poor’, or ‘horrible’ for $-$ve reviews. However, for short texts
such as tweets shown in Figure 4, ‘relevancy’ can depend on the context and it
is unclear which keywords truly represent the examples in the ‘irrelevant’
class.

##  VI COVID-19 Use-Case

We show a practical implication of our work by applying it to COVID-19 tweets
described in Section 4.3. Our goal is to interpretably predict if a COVID-19
tweet is relevant or not; a binary classification task. The models are trained
using only the TREC dataset and evaluated on the COVID-19 tweets (a balanced
subset of size $637$ for $+$ve and $-$ve labels). We found that a combination
of ‘Priority’ and ‘Irrelevant’ labels from TREC performs better to predict
COVID-19’s ‘Relevant’ label (this can be trivially verified by constructing
two binary classifiers). We augment all three methods (ST, ST-DAAN, and MT-
DAAN) with an additional condition before label prediction:
$R_{c}=P_{t}\cap\overline{I_{t}}$, which means that a COVID-19 tweet is
‘Relevant’ only if it is predicted both ‘Priority’ = $1$ and ‘Irrelevant’ =
$0$. The scores are reported in Table VI and the attention results are shown
in Figure 4, demonstrating the effectiveness of our proposed method.

##  VII Conclusion

We presented a novel approach of unsupervised domain adaptation with multi-
task learning to classify relevant information from Twitter streams for crisis
management, while addressing the problems of data sparsity and limited labels.
We showed that a multi-task learning model that shares the lower layers of the
neural network with dedicated attention layers for each task along with a
domain classifier branch can help improve generalizability and performance of
deep models in the settings of limited data. Furthermore, we showed that using
an attention-based architecture can help in interpreting the classifier’s
predictions by highlighting the important words that justify the predictions.
We also presented an in-depth empirical analysis of the state-of-the-art
models on both benchmark dataset of Amazon reviews and TREC dataset of crisis
events. The application of our generic approach for interpretable and
unsupervised domain adaptation within a multi-task learning framework can
benefit social media mining systems in diverse domains beyond crisis
management.
Reproducibility: Source code and instructions for deployment are available at
- https://github.com/jitinkrishnan/Crisis-Tweet-Multi-Task-DA.

## References

  * [1] C. Castillo, _Big crisis data: social media in disasters and time-critical situations_. Cambridge University Press, 2016.
  * [2] M. Imran, P. Mitra, and C. Castillo, “Twitter as a lifeline: Human-annotated twitter corpora for nlp of crisis-related messages,” _arXiv preprint arXiv:1605.05894_ , 2016.
  * [3] H. Li, D. Caragea, C. Caragea, and N. Herndon, “Disaster response aided by tweet classification with a domain adaptation approach,” _Journal of Contingencies and Crisis Management_ , vol. 26, no. 1, pp. 16–27, 2018.
  * [4] S. Vieweg, C. Castillo, and M. Imran, “Integrating social media communications into the rapid assessment of sudden onset disasters,” in _International Conference on Social Informatics_. Springer, 2014, pp. 444–461.
  * [5] D. T. Nguyen, K. A. A. Mannai, S. Joty, H. Sajjad, M. Imran, and P. Mitra, “Rapid classification of crisis-related data on social networks using convolutional neural networks,” _arXiv preprint arXiv:1608.03902_ , 2016\.
  * [6] F. Alam, S. Joty, and M. Imran, “Domain adaptation with adversarial training and graph embeddings,” _arXiv preprint arXiv:1805.05151_ , 2018.
  * [7] R. Mazloom, H. Li, D. Caragea, C. Caragea, and M. Imran, “A hybrid domain adaptation approach for identifying crisis-relevant tweets,” _International Journal of Information Systems for Crisis Response and Management (IJISCRAM)_ , vol. 11, no. 2, pp. 1–19, 2019.
  * [8] J. Blitzer, R. McDonald, and F. Pereira, “Domain adaptation with structural correspondence learning,” in _Proceedings of the 2006 conference on empirical methods in natural language processing_ , 2006, pp. 120–128.
  * [9] S. J. Pan, X. Ni, J.-T. Sun, Q. Yang, and Z. Chen, “Cross-domain sentiment classification via spectral feature alignment,” in _Proceedings of the 19th international conference on World wide web_. ACM, 2010, pp. 751–760.
  * [10] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.-A. Manzagol, “Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion,” _Journal of machine learning research_ , vol. 11, no. Dec, pp. 3371–3408, 2010.
  * [11] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, “Domain-adversarial training of neural networks,” _The Journal of Machine Learning Research_ , vol. 17, no. 1, pp. 2096–2030, 2016.
  * [12] Z. Li, Y. Zhang, Y. Wei, Y. Wu, and Q. Yang, “End-to-end adversarial memory network for cross-domain sentiment classification.” in _IJCAI_ , 2017, pp. 2237–2243.
  * [13] Z. Li, Y. Wei, Y. Zhang, and Q. Yang, “Hierarchical attention transfer network for cross-domain sentiment classification,” in _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018.
  * [14] R. Caruana, “Multitask learning,” _Machine learning_ , vol. 28, no. 1, pp. 41–75, 1997.
  * [15] S. Ruder, “An overview of multi-task learning in deep neural networks,” _arXiv preprint arXiv:1706.05098_ , 2017.
  * [16] M. Long and J. Wang, “Learning multiple tasks with deep relationship networks,” _arXiv preprint arXiv:1506.02117_ , vol. 2, p. 1, 2015.
  * [17] S. Ruder12, J. Bingel, I. Augenstein, and A. Søgaard, “Sluice networks: Learning what to share between loosely related tasks,” _stat_ , vol. 1050, p. 23, 2017.
  * [18] X. Liu, J. Gao, X. He, L. Deng, K. Duh, and Y.-Y. Wang, “Representation learning using multi-task deep neural networks for semantic classification and information retrieval,” 2015.
  * [19] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” _arXiv preprint arXiv:1409.0473_ , 2014\.
  * [20] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in _Advances in neural information processing systems_ , 2014, pp. 3104–3112.
  * [21] M. Schuster and K. K. Paliwal, “Bidirectional recurrent neural networks,” _IEEE Transactions on Signal Processing_ , vol. 45, no. 11, pp. 2673–2681, 1997.
  * [22] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [23] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Gated feedback recurrent neural networks,” in _International conference on machine learning_ , 2015, pp. 2067–2075.
  * [24] J. Krishnan, H. Purohit, and H. Rangwala, “Diversity-based generalization for neural unsupervised text classification under domain shift,” _ECML-PKDD_ , 2020.
  * [25] J. Blitzer, M. Dredze, and F. Pereira, “Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification,” in _Proceedings of the 45th annual meeting of the association of computational linguistics_ , 2007, pp. 440–447.
  * [26] R. Pandey and H. Purohit, “Citizenhelper-adaptive: expert-augmented streaming analytics system for emergency services and humanitarian organizations,” in _2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM)_. IEEE, 2018, pp. 630–633.
  * [27] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine Learning in Python ,” _Journal of Machine Learning Research_ , vol. 12, pp. 2825–2830, 2011.
  * [28] Y. Kim, “Convolutional neural networks for sentence classification,” _arXiv preprint arXiv:1408.5882_ , 2014.
  * [29] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in _Advances in neural information processing systems_ , 2013, pp. 3111–3119.
  * [30] T. Mikolov, E. Grave, P. Bojanowski, C. Puhrsch, and A. Joulin, “Advances in pre-training distributed word representations,” in _Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018)_ , 2018\.
  * [31] J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in _Empirical Methods in Natural Language Processing (EMNLP)_ , 2014, pp. 1532–1543. [Online]. Available: http://www.aclweb.org/anthology/D14-1162
  * [32] M. Imran, P. Mitra, and C. Castillo, “Twitter as a lifeline: Human-annotated twitter corpora for nlp of crisis-related messages,” in _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016)_. Paris, France: European Language Resources Association (ELRA), may 2016.
  * [33] M. T. Ribeiro, S. Singh, and C. Guestrin, “” why should i trust you?” explaining the predictions of any classifier,” in _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , 2016, pp. 1135–1144.